2024-09-16T14:28:59.875 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T14:28:59.897 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T14:28:59.984 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906750 branch: reef description: rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_mon_workunits} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906750' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: reef ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(AUTH_BAD_CAPS\) mon_bind_msgr2: false sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 owner: scheduled_teuthology@teuthology priority: 931 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 1390 sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 sleep_before_teardown: 0 subset: 5319/100000 suite: rados suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 targets: smithi026.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA1oD3ch7Tmpmckzw+2+omHRyYdWDrggEpWSp4Dkyu08tBkTetF1QZQ5gtkoLKdEiFPq4Xe/beT+YWOdRkHvdEo= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIbBhEHhS8/ndwvSqja/lY74FL75GrZLIgfi1VWNkoK8DoOltkKxT6lCeEekTgwvEvThk0demR/ewuaHBduzXrU= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 20 thrash_delay: 1 - workunit: clients: client.0: - mon/pool_ops.sh - mon/crush_ops.sh - mon/osd.sh - mon/caps.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_22:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T14:28:59.984 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa; will attempt to use it 2024-09-16T14:28:59.985 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa/tasks 2024-09-16T14:28:59.985 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T14:28:59.987 INFO:teuthology.task.internal:Checking packages... 2024-09-16T14:29:00.014 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8' 2024-09-16T14:29:00.014 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T14:29:00.014 INFO:teuthology.packaging:ref: None 2024-09-16T14:29:00.014 INFO:teuthology.packaging:tag: None 2024-09-16T14:29:00.014 INFO:teuthology.packaging:branch: reef 2024-09-16T14:29:00.014 INFO:teuthology.packaging:sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T14:29:00.014 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=reef 2024-09-16T14:29:00.231 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-853-gae7f3733-1jammy 2024-09-16T14:29:00.232 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T14:29:00.238 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T14:29:00.239 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T14:29:00.291 INFO:teuthology.task.internal:Saving configuration 2024-09-16T14:29:00.303 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T14:29:00.310 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T14:29:00.336 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906750', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 14:23:06.911640', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK44yOAXZ0OOg5NZpQlshjmiv6MkvtpR83bO46B7MXf+'} 2024-09-16T14:29:00.356 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906750', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 14:23:06.910149', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE5XhZYg71n48H5hDUgcPkfJsDU5uyStxbiMWs7l/AUG'} 2024-09-16T14:29:00.356 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T14:29:00.363 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-16T14:29:00.363 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-16T14:29:00.363 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T14:29:00.424 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7f868eb250>, signals=[15]) 2024-09-16T14:29:00.424 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T14:29:00.430 INFO:teuthology.task.internal:Opening connections... 2024-09-16T14:29:00.430 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T14:29:00.431 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:29:00.513 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T14:29:00.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:29:00.592 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T14:29:00.599 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2024-09-16T14:29:00.604 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2024-09-16T14:29:00.604 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2024-09-16T14:29:00.650 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T14:29:00.650 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="Ubuntu" 2024-09-16T14:29:00.650 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="22.04" 2024-09-16T14:29:00.650 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T14:29:00.650 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_CODENAME=jammy 2024-09-16T14:29:00.650 INFO:teuthology.orchestra.run.smithi026.stdout:ID=ubuntu 2024-09-16T14:29:00.651 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE=debian 2024-09-16T14:29:00.651 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T14:29:00.651 INFO:teuthology.orchestra.run.smithi026.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T14:29:00.651 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T14:29:00.651 INFO:teuthology.orchestra.run.smithi026.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T14:29:00.651 INFO:teuthology.orchestra.run.smithi026.stdout:UBUNTU_CODENAME=jammy 2024-09-16T14:29:00.652 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2024-09-16T14:29:00.685 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-09-16T14:29:00.690 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-09-16T14:29:00.691 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-09-16T14:29:00.740 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T14:29:00.740 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="Ubuntu" 2024-09-16T14:29:00.740 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="22.04" 2024-09-16T14:29:00.740 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T14:29:00.740 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_CODENAME=jammy 2024-09-16T14:29:00.741 INFO:teuthology.orchestra.run.smithi189.stdout:ID=ubuntu 2024-09-16T14:29:00.741 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE=debian 2024-09-16T14:29:00.741 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T14:29:00.741 INFO:teuthology.orchestra.run.smithi189.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T14:29:00.741 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T14:29:00.741 INFO:teuthology.orchestra.run.smithi189.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T14:29:00.741 INFO:teuthology.orchestra.run.smithi189.stdout:UBUNTU_CODENAME=jammy 2024-09-16T14:29:00.741 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-09-16T14:29:00.769 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T14:29:00.777 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T14:29:00.894 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T14:29:00.894 DEBUG:teuthology.orchestra.run.smithi026:> test '!' -e /home/ubuntu/cephtest 2024-09-16T14:29:00.897 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-09-16T14:29:00.902 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T14:29:00.913 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T14:29:00.913 DEBUG:teuthology.orchestra.run.smithi026:> test -z $(ls -A /var/lib/ceph) 2024-09-16T14:29:00.944 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-09-16T14:29:00.960 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T14:29:01.010 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T14:29:01.023 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T14:29:01.023 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T14:29:01.024 DEBUG:teuthology.orchestra.run.smithi026:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T14:29:01.024 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T14:29:01.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T14:29:01.028 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-09-16T14:29:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T14:29:01.029 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-09-16T14:29:01.075 INFO:teuthology.orchestra.run.smithi026.stdout:5.15.0-56-generic 2024-09-16T14:29:01.075 INFO:teuthology.task.kernel:Running kernel on smithi026: 5.15.0-56-generic 2024-09-16T14:29:01.075 DEBUG:teuthology.orchestra.run.smithi026:> sudo apt-get clean 2024-09-16T14:29:01.076 INFO:teuthology.orchestra.run.smithi189.stdout:5.15.0-56-generic 2024-09-16T14:29:01.077 INFO:teuthology.task.kernel:Running kernel on smithi189: 5.15.0-56-generic 2024-09-16T14:29:01.077 DEBUG:teuthology.orchestra.run.smithi189:> sudo apt-get clean 2024-09-16T14:29:01.214 DEBUG:teuthology.orchestra.run.smithi026:> sudo apt-get update 2024-09-16T14:29:01.221 DEBUG:teuthology.orchestra.run.smithi189:> sudo apt-get update 2024-09-16T14:29:01.394 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T14:29:01.396 INFO:teuthology.orchestra.run.smithi189.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T14:29:01.396 INFO:teuthology.orchestra.run.smithi189.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T14:29:01.398 INFO:teuthology.orchestra.run.smithi026.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T14:29:01.463 INFO:teuthology.orchestra.run.smithi189.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T14:29:01.497 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T14:29:01.580 INFO:teuthology.orchestra.run.smithi026.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T14:29:01.613 INFO:teuthology.orchestra.run.smithi026.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T14:29:01.632 INFO:teuthology.orchestra.run.smithi189.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T14:29:01.703 INFO:teuthology.orchestra.run.smithi189.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T14:29:01.703 INFO:teuthology.orchestra.run.smithi189.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T14:29:01.717 INFO:teuthology.orchestra.run.smithi026.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T14:29:01.721 INFO:teuthology.orchestra.run.smithi026.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T14:29:01.723 INFO:teuthology.orchestra.run.smithi026.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T14:29:01.723 INFO:teuthology.orchestra.run.smithi026.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T14:29:01.724 INFO:teuthology.orchestra.run.smithi026.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T14:29:01.742 INFO:teuthology.orchestra.run.smithi189.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T14:29:01.744 INFO:teuthology.orchestra.run.smithi026.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T14:29:01.747 INFO:teuthology.orchestra.run.smithi026.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T14:29:01.747 INFO:teuthology.orchestra.run.smithi026.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T14:29:01.748 INFO:teuthology.orchestra.run.smithi189.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T14:29:01.748 INFO:teuthology.orchestra.run.smithi189.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T14:29:01.748 INFO:teuthology.orchestra.run.smithi189.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T14:29:01.753 INFO:teuthology.orchestra.run.smithi026.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T14:29:01.758 INFO:teuthology.orchestra.run.smithi026.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T14:29:01.759 INFO:teuthology.orchestra.run.smithi026.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T14:29:01.759 INFO:teuthology.orchestra.run.smithi026.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T14:29:01.760 INFO:teuthology.orchestra.run.smithi026.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T14:29:01.760 INFO:teuthology.orchestra.run.smithi026.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T14:29:01.763 INFO:teuthology.orchestra.run.smithi189.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T14:29:01.766 INFO:teuthology.orchestra.run.smithi189.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,025 kB] 2024-09-16T14:29:01.766 INFO:teuthology.orchestra.run.smithi189.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T14:29:01.766 INFO:teuthology.orchestra.run.smithi189.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T14:29:01.772 INFO:teuthology.orchestra.run.smithi189.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T14:29:01.776 INFO:teuthology.orchestra.run.smithi189.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T14:29:01.777 INFO:teuthology.orchestra.run.smithi026.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T14:29:01.777 INFO:teuthology.orchestra.run.smithi189.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T14:29:01.779 INFO:teuthology.orchestra.run.smithi189.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T14:29:01.818 INFO:teuthology.orchestra.run.smithi189.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T14:29:01.818 INFO:teuthology.orchestra.run.smithi189.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [353 kB] 2024-09-16T14:29:01.818 INFO:teuthology.orchestra.run.smithi189.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T14:29:01.819 INFO:teuthology.orchestra.run.smithi189.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T14:29:01.821 INFO:teuthology.orchestra.run.smithi189.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T14:29:01.821 INFO:teuthology.orchestra.run.smithi189.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,441 kB] 2024-09-16T14:29:01.836 INFO:teuthology.orchestra.run.smithi189.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T14:29:01.837 INFO:teuthology.orchestra.run.smithi189.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [420 kB] 2024-09-16T14:29:01.840 INFO:teuthology.orchestra.run.smithi189.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T14:29:01.840 INFO:teuthology.orchestra.run.smithi189.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T14:29:01.848 INFO:teuthology.orchestra.run.smithi189.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T14:29:01.858 INFO:teuthology.orchestra.run.smithi189.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T14:29:01.861 INFO:teuthology.orchestra.run.smithi189.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T14:29:01.861 INFO:teuthology.orchestra.run.smithi189.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T14:29:01.862 INFO:teuthology.orchestra.run.smithi189.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T14:29:01.872 INFO:teuthology.orchestra.run.smithi189.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T14:29:01.872 INFO:teuthology.orchestra.run.smithi189.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T14:29:01.872 INFO:teuthology.orchestra.run.smithi189.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T14:29:01.873 INFO:teuthology.orchestra.run.smithi189.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T14:29:01.873 INFO:teuthology.orchestra.run.smithi189.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T14:29:01.889 INFO:teuthology.orchestra.run.smithi189.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T14:29:01.889 INFO:teuthology.orchestra.run.smithi189.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T14:29:01.890 INFO:teuthology.orchestra.run.smithi189.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T14:29:01.890 INFO:teuthology.orchestra.run.smithi189.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T14:29:01.890 INFO:teuthology.orchestra.run.smithi189.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T14:29:01.935 INFO:teuthology.orchestra.run.smithi026.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T14:29:02.074 INFO:teuthology.orchestra.run.smithi026.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T14:29:02.269 INFO:teuthology.orchestra.run.smithi026.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,025 kB] 2024-09-16T14:29:02.428 INFO:teuthology.orchestra.run.smithi026.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [353 kB] 2024-09-16T14:29:02.441 INFO:teuthology.orchestra.run.smithi026.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T14:29:02.442 INFO:teuthology.orchestra.run.smithi026.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,441 kB] 2024-09-16T14:29:02.512 INFO:teuthology.orchestra.run.smithi026.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T14:29:02.513 INFO:teuthology.orchestra.run.smithi026.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [420 kB] 2024-09-16T14:29:02.529 INFO:teuthology.orchestra.run.smithi026.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T14:29:02.529 INFO:teuthology.orchestra.run.smithi026.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T14:29:02.551 INFO:teuthology.orchestra.run.smithi026.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T14:29:02.580 INFO:teuthology.orchestra.run.smithi026.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T14:29:02.586 INFO:teuthology.orchestra.run.smithi026.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T14:29:02.592 INFO:teuthology.orchestra.run.smithi026.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T14:29:02.592 INFO:teuthology.orchestra.run.smithi026.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T14:29:02.663 INFO:teuthology.orchestra.run.smithi026.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T14:29:02.664 INFO:teuthology.orchestra.run.smithi026.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T14:29:02.664 INFO:teuthology.orchestra.run.smithi026.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T14:29:02.665 INFO:teuthology.orchestra.run.smithi026.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T14:29:02.667 INFO:teuthology.orchestra.run.smithi026.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T14:29:02.673 INFO:teuthology.orchestra.run.smithi026.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T14:29:02.674 INFO:teuthology.orchestra.run.smithi026.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T14:29:02.674 INFO:teuthology.orchestra.run.smithi026.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T14:29:02.755 INFO:teuthology.orchestra.run.smithi026.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T14:29:02.756 INFO:teuthology.orchestra.run.smithi026.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T14:29:07.989 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 16.0 MB in 2s (7,513 kB/s) 2024-09-16T14:29:08.640 INFO:teuthology.orchestra.run.smithi026.stdout:Fetched 16.0 MB in 2s (6,994 kB/s) 2024-09-16T14:29:09.495 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2024-09-16T14:29:09.512 DEBUG:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T14:29:09.571 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2024-09-16T14:29:09.777 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2024-09-16T14:29:09.778 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2024-09-16T14:29:09.956 INFO:teuthology.orchestra.run.smithi189.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T14:29:09.956 INFO:teuthology.orchestra.run.smithi189.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T14:29:09.957 INFO:teuthology.orchestra.run.smithi189.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T14:29:09.957 INFO:teuthology.orchestra.run.smithi189.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T14:29:09.957 INFO:teuthology.orchestra.run.smithi189.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T14:29:09.957 INFO:teuthology.orchestra.run.smithi189.stdout: python2.7-minimal 2024-09-16T14:29:09.958 INFO:teuthology.orchestra.run.smithi189.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T14:29:09.958 INFO:teuthology.orchestra.run.smithi189.stdout:The following additional packages will be installed: 2024-09-16T14:29:09.959 INFO:teuthology.orchestra.run.smithi189.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T14:29:09.959 INFO:teuthology.orchestra.run.smithi189.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T14:29:09.959 INFO:teuthology.orchestra.run.smithi189.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T14:29:09.960 INFO:teuthology.orchestra.run.smithi189.stdout:Suggested packages: 2024-09-16T14:29:09.960 INFO:teuthology.orchestra.run.smithi189.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T14:29:09.961 INFO:teuthology.orchestra.run.smithi189.stdout:Recommended packages: 2024-09-16T14:29:09.961 INFO:teuthology.orchestra.run.smithi189.stdout: thermald 2024-09-16T14:29:10.006 INFO:teuthology.orchestra.run.smithi189.stdout:The following NEW packages will be installed: 2024-09-16T14:29:10.006 INFO:teuthology.orchestra.run.smithi189.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T14:29:10.006 INFO:teuthology.orchestra.run.smithi189.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T14:29:10.006 INFO:teuthology.orchestra.run.smithi189.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T14:29:10.008 INFO:teuthology.orchestra.run.smithi189.stdout:The following packages will be upgraded: 2024-09-16T14:29:10.008 INFO:teuthology.orchestra.run.smithi189.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T14:29:10.092 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2024-09-16T14:29:10.110 DEBUG:teuthology.orchestra.run.smithi026:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T14:29:10.176 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2024-09-16T14:29:10.188 INFO:teuthology.orchestra.run.smithi189.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T14:29:10.188 INFO:teuthology.orchestra.run.smithi189.stdout:Need to get 113 MB of archives. 2024-09-16T14:29:10.188 INFO:teuthology.orchestra.run.smithi189.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T14:29:10.189 INFO:teuthology.orchestra.run.smithi189.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T14:29:10.392 INFO:teuthology.orchestra.run.smithi026.stdout:Building dependency tree... 2024-09-16T14:29:10.392 INFO:teuthology.orchestra.run.smithi026.stdout:Reading state information... 2024-09-16T14:29:10.588 INFO:teuthology.orchestra.run.smithi026.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T14:29:10.588 INFO:teuthology.orchestra.run.smithi026.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T14:29:10.588 INFO:teuthology.orchestra.run.smithi026.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T14:29:10.589 INFO:teuthology.orchestra.run.smithi026.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T14:29:10.589 INFO:teuthology.orchestra.run.smithi026.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T14:29:10.589 INFO:teuthology.orchestra.run.smithi026.stdout: python2.7-minimal 2024-09-16T14:29:10.589 INFO:teuthology.orchestra.run.smithi026.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T14:29:10.591 INFO:teuthology.orchestra.run.smithi026.stdout:The following additional packages will be installed: 2024-09-16T14:29:10.591 INFO:teuthology.orchestra.run.smithi026.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T14:29:10.591 INFO:teuthology.orchestra.run.smithi026.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T14:29:10.591 INFO:teuthology.orchestra.run.smithi026.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T14:29:10.592 INFO:teuthology.orchestra.run.smithi026.stdout:Suggested packages: 2024-09-16T14:29:10.592 INFO:teuthology.orchestra.run.smithi026.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T14:29:10.592 INFO:teuthology.orchestra.run.smithi026.stdout:Recommended packages: 2024-09-16T14:29:10.593 INFO:teuthology.orchestra.run.smithi026.stdout: thermald 2024-09-16T14:29:10.648 INFO:teuthology.orchestra.run.smithi026.stdout:The following NEW packages will be installed: 2024-09-16T14:29:10.648 INFO:teuthology.orchestra.run.smithi026.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T14:29:10.648 INFO:teuthology.orchestra.run.smithi026.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T14:29:10.649 INFO:teuthology.orchestra.run.smithi026.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T14:29:10.650 INFO:teuthology.orchestra.run.smithi026.stdout:The following packages will be upgraded: 2024-09-16T14:29:10.650 INFO:teuthology.orchestra.run.smithi026.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T14:29:10.833 INFO:teuthology.orchestra.run.smithi026.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T14:29:10.833 INFO:teuthology.orchestra.run.smithi026.stdout:Need to get 113 MB of archives. 2024-09-16T14:29:10.833 INFO:teuthology.orchestra.run.smithi026.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T14:29:10.833 INFO:teuthology.orchestra.run.smithi026.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T14:29:11.593 INFO:teuthology.orchestra.run.smithi189.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T14:29:11.947 INFO:teuthology.orchestra.run.smithi189.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T14:29:12.221 INFO:teuthology.orchestra.run.smithi026.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T14:29:12.627 INFO:teuthology.orchestra.run.smithi026.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T14:29:14.176 INFO:teuthology.orchestra.run.smithi189.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T14:29:14.177 INFO:teuthology.orchestra.run.smithi189.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T14:29:14.177 INFO:teuthology.orchestra.run.smithi189.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T14:29:14.771 INFO:teuthology.orchestra.run.smithi189.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T14:29:14.799 INFO:teuthology.orchestra.run.smithi026.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T14:29:14.800 INFO:teuthology.orchestra.run.smithi026.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T14:29:14.800 INFO:teuthology.orchestra.run.smithi026.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T14:29:14.903 INFO:teuthology.orchestra.run.smithi189.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T14:29:15.293 INFO:teuthology.orchestra.run.smithi026.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T14:29:15.297 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 113 MB in 5s (23.2 MB/s) 2024-09-16T14:29:15.378 INFO:teuthology.orchestra.run.smithi026.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T14:29:15.516 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T14:29:15.803 INFO:teuthology.orchestra.run.smithi026.stdout:Fetched 113 MB in 5s (24.0 MB/s) 2024-09-16T14:29:15.973 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T14:29:17.554 INFO:teuthology.orchestra.run.smithi189.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T14:29:17.558 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T14:29:17.646 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:17.960 INFO:teuthology.orchestra.run.smithi026.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T14:29:17.964 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T14:29:18.070 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:20.114 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T14:29:20.135 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T14:29:20.138 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T14:29:20.159 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T14:29:20.225 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:20.233 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:20.684 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T14:29:20.704 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T14:29:20.733 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T14:29:20.742 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:20.753 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T14:29:20.791 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:27.570 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T14:29:27.676 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T14:29:27.688 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T14:29:27.784 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T14:29:28.046 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T14:29:28.133 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T14:29:28.150 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T14:29:28.254 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T14:29:28.454 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T14:29:28.464 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T14:29:28.495 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T14:29:28.582 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T14:29:28.605 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T14:29:28.648 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T14:29:34.498 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T14:29:34.535 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T14:29:34.573 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:34.842 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T14:29:34.879 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T14:29:34.908 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:38.314 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T14:29:38.411 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T14:29:38.752 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T14:29:38.753 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T14:29:38.874 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:38.877 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T14:29:39.000 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T14:29:39.161 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:39.253 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T14:29:39.374 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:39.516 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T14:29:39.644 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:40.493 INFO:teuthology.orchestra.run.smithi189.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T14:29:40.493 INFO:teuthology.orchestra.run.smithi189.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T14:29:40.493 INFO:teuthology.orchestra.run.smithi189.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T14:29:40.494 INFO:teuthology.orchestra.run.smithi189.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T14:29:40.655 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:40.992 INFO:teuthology.orchestra.run.smithi026.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T14:29:40.992 INFO:teuthology.orchestra.run.smithi026.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T14:29:40.992 INFO:teuthology.orchestra.run.smithi026.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T14:29:40.992 INFO:teuthology.orchestra.run.smithi026.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T14:29:41.181 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:42.043 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:42.615 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:43.916 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T14:29:44.033 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T14:29:44.159 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:44.200 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T14:29:44.247 INFO:teuthology.orchestra.run.smithi189.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T14:29:44.248 INFO:teuthology.orchestra.run.smithi189.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T14:29:44.326 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T14:29:44.460 INFO:teuthology.orchestra.run.smithi026.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T14:29:44.539 INFO:teuthology.orchestra.run.smithi026.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T14:29:44.539 INFO:teuthology.orchestra.run.smithi026.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T14:29:58.346 INFO:teuthology.orchestra.run.smithi189.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T14:29:58.346 INFO:teuthology.orchestra.run.smithi189.stdout:Sourcing file `/etc/default/grub' 2024-09-16T14:29:58.368 INFO:teuthology.orchestra.run.smithi189.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T14:29:58.395 INFO:teuthology.orchestra.run.smithi189.stdout:Generating grub configuration file ... 2024-09-16T14:29:58.674 INFO:teuthology.orchestra.run.smithi189.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T14:29:58.687 INFO:teuthology.orchestra.run.smithi189.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T14:29:58.999 INFO:teuthology.orchestra.run.smithi189.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T14:29:59.005 INFO:teuthology.orchestra.run.smithi189.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T14:29:59.070 INFO:teuthology.orchestra.run.smithi189.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T14:29:59.076 INFO:teuthology.orchestra.run.smithi189.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T14:29:59.279 INFO:teuthology.orchestra.run.smithi189.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T14:29:59.279 INFO:teuthology.orchestra.run.smithi189.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T14:29:59.279 INFO:teuthology.orchestra.run.smithi189.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T14:29:59.319 INFO:teuthology.orchestra.run.smithi189.stdout:done 2024-09-16T14:29:59.378 INFO:teuthology.orchestra.run.smithi026.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T14:29:59.379 INFO:teuthology.orchestra.run.smithi026.stdout:Sourcing file `/etc/default/grub' 2024-09-16T14:29:59.408 INFO:teuthology.orchestra.run.smithi026.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T14:29:59.433 INFO:teuthology.orchestra.run.smithi026.stdout:Generating grub configuration file ... 2024-09-16T14:29:59.715 INFO:teuthology.orchestra.run.smithi026.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T14:29:59.742 INFO:teuthology.orchestra.run.smithi026.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T14:29:59.792 DEBUG:teuthology.orchestra.run.smithi189:> dpkg -s linux-image-generic 2024-09-16T14:29:59.865 INFO:teuthology.orchestra.run.smithi189.stdout:Package: linux-image-generic 2024-09-16T14:29:59.865 INFO:teuthology.orchestra.run.smithi189.stdout:Status: install ok installed 2024-09-16T14:29:59.866 INFO:teuthology.orchestra.run.smithi189.stdout:Priority: optional 2024-09-16T14:29:59.866 INFO:teuthology.orchestra.run.smithi189.stdout:Section: kernel 2024-09-16T14:29:59.866 INFO:teuthology.orchestra.run.smithi189.stdout:Installed-Size: 21 2024-09-16T14:29:59.866 INFO:teuthology.orchestra.run.smithi189.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T14:29:59.866 INFO:teuthology.orchestra.run.smithi189.stdout:Architecture: amd64 2024-09-16T14:29:59.866 INFO:teuthology.orchestra.run.smithi189.stdout:Source: linux-meta 2024-09-16T14:29:59.866 INFO:teuthology.orchestra.run.smithi189.stdout:Version: 5.15.0.121.121 2024-09-16T14:29:59.866 INFO:teuthology.orchestra.run.smithi189.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T14:29:59.866 INFO:teuthology.orchestra.run.smithi189.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T14:29:59.866 INFO:teuthology.orchestra.run.smithi189.stdout:Recommends: thermald 2024-09-16T14:29:59.866 INFO:teuthology.orchestra.run.smithi189.stdout:Description: Generic Linux kernel image 2024-09-16T14:29:59.866 INFO:teuthology.orchestra.run.smithi189.stdout: This package will always depend on the latest generic kernel image 2024-09-16T14:29:59.866 INFO:teuthology.orchestra.run.smithi189.stdout: available. 2024-09-16T14:29:59.867 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T14:29:59.867 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T14:29:59.867 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T14:29:59.867 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T14:29:59.867 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi189.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T14:29:59.867 DEBUG:teuthology.orchestra.run.smithi189:> sudo apt-get clean 2024-09-16T14:30:00.035 DEBUG:teuthology.orchestra.run.smithi189:> sudo apt-get update 2024-09-16T14:30:00.082 INFO:teuthology.orchestra.run.smithi026.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T14:30:00.088 INFO:teuthology.orchestra.run.smithi026.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T14:30:00.150 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T14:30:00.160 INFO:teuthology.orchestra.run.smithi026.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T14:30:00.165 INFO:teuthology.orchestra.run.smithi026.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T14:30:00.272 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T14:30:00.340 INFO:teuthology.orchestra.run.smithi026.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T14:30:00.340 INFO:teuthology.orchestra.run.smithi026.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T14:30:00.340 INFO:teuthology.orchestra.run.smithi026.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T14:30:00.353 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T14:30:00.382 INFO:teuthology.orchestra.run.smithi026.stdout:done 2024-09-16T14:30:00.435 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T14:30:00.856 DEBUG:teuthology.orchestra.run.smithi026:> dpkg -s linux-image-generic 2024-09-16T14:30:00.875 INFO:teuthology.orchestra.run.smithi026.stdout:Package: linux-image-generic 2024-09-16T14:30:00.875 INFO:teuthology.orchestra.run.smithi026.stdout:Status: install ok installed 2024-09-16T14:30:00.875 INFO:teuthology.orchestra.run.smithi026.stdout:Priority: optional 2024-09-16T14:30:00.876 INFO:teuthology.orchestra.run.smithi026.stdout:Section: kernel 2024-09-16T14:30:00.876 INFO:teuthology.orchestra.run.smithi026.stdout:Installed-Size: 21 2024-09-16T14:30:00.876 INFO:teuthology.orchestra.run.smithi026.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T14:30:00.876 INFO:teuthology.orchestra.run.smithi026.stdout:Architecture: amd64 2024-09-16T14:30:00.876 INFO:teuthology.orchestra.run.smithi026.stdout:Source: linux-meta 2024-09-16T14:30:00.876 INFO:teuthology.orchestra.run.smithi026.stdout:Version: 5.15.0.121.121 2024-09-16T14:30:00.876 INFO:teuthology.orchestra.run.smithi026.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T14:30:00.876 INFO:teuthology.orchestra.run.smithi026.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T14:30:00.876 INFO:teuthology.orchestra.run.smithi026.stdout:Recommends: thermald 2024-09-16T14:30:00.876 INFO:teuthology.orchestra.run.smithi026.stdout:Description: Generic Linux kernel image 2024-09-16T14:30:00.876 INFO:teuthology.orchestra.run.smithi026.stdout: This package will always depend on the latest generic kernel image 2024-09-16T14:30:00.876 INFO:teuthology.orchestra.run.smithi026.stdout: available. 2024-09-16T14:30:00.877 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T14:30:00.877 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T14:30:00.877 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T14:30:00.878 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T14:30:00.878 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi026.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T14:30:00.878 DEBUG:teuthology.orchestra.run.smithi026:> sudo apt-get clean 2024-09-16T14:30:01.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo apt-get update 2024-09-16T14:30:01.181 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T14:30:01.303 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T14:30:01.384 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T14:30:01.465 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T14:30:01.622 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2024-09-16T14:30:01.640 DEBUG:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T14:30:01.693 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2024-09-16T14:30:01.881 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2024-09-16T14:30:01.881 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2024-09-16T14:30:02.048 INFO:teuthology.orchestra.run.smithi189.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T14:30:02.048 INFO:teuthology.orchestra.run.smithi189.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T14:30:02.048 INFO:teuthology.orchestra.run.smithi189.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T14:30:02.048 INFO:teuthology.orchestra.run.smithi189.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T14:30:02.048 INFO:teuthology.orchestra.run.smithi189.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T14:30:02.049 INFO:teuthology.orchestra.run.smithi189.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T14:30:02.049 INFO:teuthology.orchestra.run.smithi189.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T14:30:02.049 INFO:teuthology.orchestra.run.smithi189.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T14:30:02.088 INFO:teuthology.orchestra.run.smithi189.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T14:30:02.090 DEBUG:teuthology.orchestra.run.smithi189:> dpkg -s linux-image-generic 2024-09-16T14:30:02.146 INFO:teuthology.orchestra.run.smithi189.stdout:Package: linux-image-generic 2024-09-16T14:30:02.146 INFO:teuthology.orchestra.run.smithi189.stdout:Status: install ok installed 2024-09-16T14:30:02.146 INFO:teuthology.orchestra.run.smithi189.stdout:Priority: optional 2024-09-16T14:30:02.146 INFO:teuthology.orchestra.run.smithi189.stdout:Section: kernel 2024-09-16T14:30:02.146 INFO:teuthology.orchestra.run.smithi189.stdout:Installed-Size: 21 2024-09-16T14:30:02.147 INFO:teuthology.orchestra.run.smithi189.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T14:30:02.147 INFO:teuthology.orchestra.run.smithi189.stdout:Architecture: amd64 2024-09-16T14:30:02.147 INFO:teuthology.orchestra.run.smithi189.stdout:Source: linux-meta 2024-09-16T14:30:02.147 INFO:teuthology.orchestra.run.smithi189.stdout:Version: 5.15.0.121.121 2024-09-16T14:30:02.147 INFO:teuthology.orchestra.run.smithi189.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T14:30:02.147 INFO:teuthology.orchestra.run.smithi189.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T14:30:02.147 INFO:teuthology.orchestra.run.smithi189.stdout:Recommends: thermald 2024-09-16T14:30:02.147 INFO:teuthology.orchestra.run.smithi189.stdout:Description: Generic Linux kernel image 2024-09-16T14:30:02.147 INFO:teuthology.orchestra.run.smithi189.stdout: This package will always depend on the latest generic kernel image 2024-09-16T14:30:02.147 INFO:teuthology.orchestra.run.smithi189.stdout: available. 2024-09-16T14:30:02.147 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-09-16T14:30:02.191 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.871ZkEph8Z 2024-09-16T14:30:02.191 DEBUG:teuthology.orchestra.run.smithi189:> sudo cp /boot/grub/grub.cfg /tmp/tmp.871ZkEph8Z 2024-09-16T14:30:02.251 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.871ZkEph8Z 2024-09-16T14:30:02.368 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.871ZkEph8Z is 10KB 2024-09-16T14:30:02.381 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.871ZkEph8Z 2024-09-16T14:30:02.386 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T14:30:02.439 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:30:02.439 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T14:30:02.439 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T14:30:02.515 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T14:30:02.515 DEBUG:teuthology.orchestra.run.smithi189:> sudo update-grub 2024-09-16T14:30:02.683 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2024-09-16T14:30:02.701 DEBUG:teuthology.orchestra.run.smithi026:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T14:30:02.755 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2024-09-16T14:30:02.946 INFO:teuthology.orchestra.run.smithi026.stdout:Building dependency tree... 2024-09-16T14:30:02.947 INFO:teuthology.orchestra.run.smithi026.stdout:Reading state information... 2024-09-16T14:30:03.125 INFO:teuthology.orchestra.run.smithi026.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T14:30:03.125 INFO:teuthology.orchestra.run.smithi026.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T14:30:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T14:30:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T14:30:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T14:30:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T14:30:03.125 INFO:teuthology.orchestra.run.smithi026.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T14:30:03.126 INFO:teuthology.orchestra.run.smithi026.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T14:30:03.165 INFO:teuthology.orchestra.run.smithi026.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T14:30:03.167 DEBUG:teuthology.orchestra.run.smithi026:> dpkg -s linux-image-generic 2024-09-16T14:30:03.229 INFO:teuthology.orchestra.run.smithi026.stdout:Package: linux-image-generic 2024-09-16T14:30:03.229 INFO:teuthology.orchestra.run.smithi026.stdout:Status: install ok installed 2024-09-16T14:30:03.230 INFO:teuthology.orchestra.run.smithi026.stdout:Priority: optional 2024-09-16T14:30:03.230 INFO:teuthology.orchestra.run.smithi026.stdout:Section: kernel 2024-09-16T14:30:03.230 INFO:teuthology.orchestra.run.smithi026.stdout:Installed-Size: 21 2024-09-16T14:30:03.230 INFO:teuthology.orchestra.run.smithi026.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T14:30:03.230 INFO:teuthology.orchestra.run.smithi026.stdout:Architecture: amd64 2024-09-16T14:30:03.230 INFO:teuthology.orchestra.run.smithi026.stdout:Source: linux-meta 2024-09-16T14:30:03.230 INFO:teuthology.orchestra.run.smithi026.stdout:Version: 5.15.0.121.121 2024-09-16T14:30:03.230 INFO:teuthology.orchestra.run.smithi026.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T14:30:03.230 INFO:teuthology.orchestra.run.smithi026.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T14:30:03.230 INFO:teuthology.orchestra.run.smithi026.stdout:Recommends: thermald 2024-09-16T14:30:03.230 INFO:teuthology.orchestra.run.smithi026.stdout:Description: Generic Linux kernel image 2024-09-16T14:30:03.230 INFO:teuthology.orchestra.run.smithi026.stdout: This package will always depend on the latest generic kernel image 2024-09-16T14:30:03.230 INFO:teuthology.orchestra.run.smithi026.stdout: available. 2024-09-16T14:30:03.231 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-09-16T14:30:03.278 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.jynJ3Gr6sK 2024-09-16T14:30:03.278 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /boot/grub/grub.cfg /tmp/tmp.jynJ3Gr6sK 2024-09-16T14:30:03.337 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.jynJ3Gr6sK 2024-09-16T14:30:03.450 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.jynJ3Gr6sK is 10KB 2024-09-16T14:30:03.462 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.jynJ3Gr6sK 2024-09-16T14:30:03.467 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T14:30:03.521 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:30:03.521 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T14:30:03.521 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T14:30:03.587 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T14:30:03.588 DEBUG:teuthology.orchestra.run.smithi026:> sudo update-grub 2024-09-16T14:30:03.755 INFO:teuthology.orchestra.run.smithi189.stderr:Sourcing file `/etc/default/grub' 2024-09-16T14:30:03.756 INFO:teuthology.orchestra.run.smithi189.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T14:30:03.759 INFO:teuthology.orchestra.run.smithi189.stderr:Generating grub configuration file ... 2024-09-16T14:30:03.954 INFO:teuthology.orchestra.run.smithi189.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T14:30:03.959 INFO:teuthology.orchestra.run.smithi189.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T14:30:04.191 INFO:teuthology.orchestra.run.smithi189.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T14:30:04.197 INFO:teuthology.orchestra.run.smithi189.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T14:30:04.240 INFO:teuthology.orchestra.run.smithi189.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T14:30:04.245 INFO:teuthology.orchestra.run.smithi189.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T14:30:04.418 INFO:teuthology.orchestra.run.smithi189.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T14:30:04.418 INFO:teuthology.orchestra.run.smithi189.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T14:30:04.418 INFO:teuthology.orchestra.run.smithi189.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T14:30:04.444 INFO:teuthology.orchestra.run.smithi189.stderr:done 2024-09-16T14:30:04.453 DEBUG:teuthology.orchestra.run.smithi189:> sudo shutdown -r now 2024-09-16T14:30:04.863 INFO:teuthology.orchestra.run.smithi026.stderr:Sourcing file `/etc/default/grub' 2024-09-16T14:30:04.865 INFO:teuthology.orchestra.run.smithi026.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T14:30:04.867 INFO:teuthology.orchestra.run.smithi026.stderr:Generating grub configuration file ... 2024-09-16T14:30:05.170 INFO:teuthology.orchestra.run.smithi026.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T14:30:05.176 INFO:teuthology.orchestra.run.smithi026.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T14:30:05.550 INFO:teuthology.orchestra.run.smithi026.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T14:30:05.555 INFO:teuthology.orchestra.run.smithi026.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T14:30:05.600 INFO:teuthology.orchestra.run.smithi026.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T14:30:05.605 INFO:teuthology.orchestra.run.smithi026.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T14:30:05.740 INFO:teuthology.orchestra.run.smithi026.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T14:30:05.741 INFO:teuthology.orchestra.run.smithi026.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T14:30:05.741 INFO:teuthology.orchestra.run.smithi026.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T14:30:05.771 INFO:teuthology.orchestra.run.smithi026.stderr:done 2024-09-16T14:30:05.773 DEBUG:teuthology.orchestra.run.smithi026:> sudo shutdown -r now 2024-09-16T14:30:34.471 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T14:30:34.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-16T14:30:34.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:30:35.821 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T14:30:35.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-16T14:30:35.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:30:52.994 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-09-16T14:30:54.273 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-09-16T14:31:02.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-16T14:31:02.002 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:31:03.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-16T14:31:03.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:31:05.053 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-09-16T14:31:17.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-16T14:31:17.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:31:20.129 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-09-16T14:31:21.661 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-09-16T14:31:33.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-16T14:31:33.666 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:31:34.560 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-09-16T14:31:35.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-16T14:31:35.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:31:38.205 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-09-16T14:31:49.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-16T14:31:49.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:31:49.800 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-09-16T14:31:50.404 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-16T14:31:50.405 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-16T14:31:50.405 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-09-16T14:31:50.451 INFO:teuthology.orchestra.run.smithi026.stdout:5.15.0-121-generic 2024-09-16T14:31:50.451 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T14:31:50.452 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T14:31:50.452 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T14:31:51.452 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T14:31:51.453 DEBUG:teuthology.orchestra.run.smithi026:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T14:31:51.531 INFO:teuthology.orchestra.run.smithi026.stdout:ttyS1 2024-09-16T14:31:51.557 DEBUG:teuthology.parallel:result is None 2024-09-16T14:31:56.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-16T14:31:56.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:31:59.261 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-09-16T14:32:20.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-16T14:32:20.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:32:20.478 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-09-16T14:32:21.131 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-16T14:32:21.131 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-121-generic"... 2024-09-16T14:32:21.131 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-09-16T14:32:21.177 INFO:teuthology.orchestra.run.smithi189.stdout:5.15.0-121-generic 2024-09-16T14:32:21.177 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T14:32:21.177 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T14:32:21.177 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T14:32:22.178 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T14:32:22.178 DEBUG:teuthology.orchestra.run.smithi189:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T14:32:22.255 INFO:teuthology.orchestra.run.smithi189.stdout:ttyS1 2024-09-16T14:32:22.286 DEBUG:teuthology.parallel:result is None 2024-09-16T14:32:22.286 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T14:32:22.294 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T14:32:22.295 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T14:32:22.297 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T14:32:22.303 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T14:32:22.341 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T14:32:22.348 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T14:32:22.348 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T14:32:22.351 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T14:32:22.367 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T14:32:22.380 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T14:32:22.381 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T14:32:22.408 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T14:32:22.426 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T14:32:22.434 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T14:32:22.434 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T14:32:22.442 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T14:32:22.444 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T14:32:22.451 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T14:32:22.451 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T14:32:22.480 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T14:32:22.500 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T14:32:22.510 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T14:32:22.510 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T14:32:22.536 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T14:32:22.549 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T14:32:22.588 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T14:32:22.636 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:32:22.636 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T14:32:22.694 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T14:32:22.700 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T14:32:22.749 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:32:22.749 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T14:32:22.811 DEBUG:teuthology.orchestra.run.smithi026:> sudo service rsyslog restart 2024-09-16T14:32:22.814 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-09-16T14:32:22.890 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T14:32:22.898 INFO:teuthology.task.internal:Starting timer... 2024-09-16T14:32:22.898 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T14:32:22.909 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T14:32:22.918 DEBUG:teuthology.task.selinux:Excluding smithi026: OS 'ubuntu' does not support SELinux 2024-09-16T14:32:22.918 DEBUG:teuthology.task.selinux:Excluding smithi189: OS 'ubuntu' does not support SELinux 2024-09-16T14:32:22.918 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T14:32:22.918 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T14:32:22.918 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T14:32:22.918 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T14:32:22.927 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T14:32:22.940 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T14:32:22.941 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi026.front.sepia.ceph.com,smithi189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T14:39:57.399 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi026.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com')] 2024-09-16T14:39:57.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-16T14:39:57.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:39:57.487 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-09-16T14:39:57.565 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-09-16T14:39:57.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-16T14:39:57.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T14:39:57.648 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-09-16T14:39:57.725 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-16T14:39:57.725 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T14:39:57.737 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T14:39:57.738 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T14:39:57.738 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T14:39:57.741 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T14:39:57.741 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T14:39:57.770 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T14:39:57.770 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: Command line: ntpd -gq 2024-09-16T14:39:57.770 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: ---------------------------------------------------- 2024-09-16T14:39:57.770 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T14:39:57.770 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T14:39:57.770 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: corporation. Support and training for ntp-4 are 2024-09-16T14:39:57.770 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: available at https://www.nwtime.org/support 2024-09-16T14:39:57.770 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: ---------------------------------------------------- 2024-09-16T14:39:57.770 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: proto: precision = 0.070 usec (-24) 2024-09-16T14:39:57.770 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: basedate set to 2022-02-04 2024-09-16T14:39:57.770 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: gps base set to 2022-02-06 (week 2196) 2024-09-16T14:39:57.770 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T14:39:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T14:39:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T14:39:57.772 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T14:39:57.772 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T14:39:57.772 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T14:39:57.772 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T14:39:57.772 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: Listen normally on 3 ens1f0 172.21.15.26:123 2024-09-16T14:39:57.772 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: Listen normally on 4 lo [::1]:123 2024-09-16T14:39:57.772 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fea5:cac0%4]:123 2024-09-16T14:39:57.772 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:39:57 ntpd[18362]: Listening on routing socket on fd #22 for interface updates 2024-09-16T14:39:57.793 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T14:39:57.793 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: Command line: ntpd -gq 2024-09-16T14:39:57.793 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: ---------------------------------------------------- 2024-09-16T14:39:57.794 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T14:39:57.794 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T14:39:57.794 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: corporation. Support and training for ntp-4 are 2024-09-16T14:39:57.794 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: available at https://www.nwtime.org/support 2024-09-16T14:39:57.794 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: ---------------------------------------------------- 2024-09-16T14:39:57.795 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: proto: precision = 0.058 usec (-24) 2024-09-16T14:39:57.795 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: basedate set to 2022-02-04 2024-09-16T14:39:57.795 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: gps base set to 2022-02-06 (week 2196) 2024-09-16T14:39:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T14:39:57.795 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T14:39:57.796 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T14:39:57.796 INFO:teuthology.orchestra.run.smithi189.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T14:39:57.797 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T14:39:57.797 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T14:39:57.797 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T14:39:57.797 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: Listen normally on 3 enp3s0f1 172.21.15.189:123 2024-09-16T14:39:57.797 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: Listen normally on 4 lo [::1]:123 2024-09-16T14:39:57.797 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6df1%5]:123 2024-09-16T14:39:57.797 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:39:57 ntpd[18509]: Listening on routing socket on fd #22 for interface updates 2024-09-16T14:39:58.793 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:39:58 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:39:58.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:39:58 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:39:59.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:39:59 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:39:59.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:39:59 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:39:59.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:39:59 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:39:59.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:39:59 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:39:59.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:39:59 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:39:59.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:39:59 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:00.769 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:00 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:00.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:00 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:00.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:00 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:00.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:00 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:00.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:00 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:00.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:00 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:00.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:00 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:00 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:00 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:00 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:00 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:00 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:00 ntpd[18509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:00 ntpd[18509]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:00 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:00 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:00 ntpd[18509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:00 ntpd[18509]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:01.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:01 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:01.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:01 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:01.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:01 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:01.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:01 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:01.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:01 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:01.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:01 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:01.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:01 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:01.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:01 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:01.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:01 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:01.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:01 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:01.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:01 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:01.772 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:01 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:01.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:01 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:01.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:01 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:01.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:01 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:01.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:01 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:01.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:01 ntpd[18509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:01.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:01 ntpd[18509]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:02.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:02 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:02.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:02 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:02.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:02 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:02.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:02 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:02.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:02 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:02.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:02 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:02.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:02 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:02.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:02 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:02.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:02 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:02.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:02 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:02.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:02 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:02.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:02 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:02.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:02 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:02.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:02 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:02.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:02 ntpd[18509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:02.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:02 ntpd[18509]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:02.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:02 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:02.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:02 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:02.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:02 ntpd[18509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:02.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:02 ntpd[18509]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:03.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:03 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:03.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:03 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:03.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:03 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:03.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:03 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:03.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:03 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:03.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:03 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:03.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:03 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:03.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:03 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:03.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:03 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:03.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:03 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:03.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:03 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:03.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:03 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:03.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:03 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:03.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:03 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:03.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:03 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:03.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:03 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:03.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:03 ntpd[18509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:03.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:03 ntpd[18509]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:04 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:04.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:04 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:04.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:04 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:04.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:04 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:04.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:04 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:04.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:04 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:04.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:04 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:04.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:04 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:04.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:04 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:04.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:04 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:04.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:04 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:04.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:04 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:04.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:04 ntpd[18509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:04.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:04 ntpd[18509]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:04.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:04 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:04.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:04 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:04.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:04 ntpd[18509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:04.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:04 ntpd[18509]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:05.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:05 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:05.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:05 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:05.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:05 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:05.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:05 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:05.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:05 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:05.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:05 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:05.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:05 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:05.770 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:05 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:05.771 INFO:teuthology.orchestra.run.smithi026.stdout:16 Sep 14:40:05 ntpd[18362]: ntpd: time slew -0.000249 s 2024-09-16T14:40:05.771 INFO:teuthology.orchestra.run.smithi026.stdout:ntpd: time slew -0.000249s 2024-09-16T14:40:05.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:05 ntpd[18362]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T14:40:05.771 INFO:teuthology.orchestra.run.smithi026.stderr:16 Sep 14:40:05 ntpd[18362]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T14:40:05.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:05 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:05.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:05 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:05.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:05 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:05.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:05 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:05.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:05 ntpd[18509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:05.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:05 ntpd[18509]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:05.817 INFO:teuthology.orchestra.run.smithi026.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T14:40:05.817 INFO:teuthology.orchestra.run.smithi026.stdout:============================================================================== 2024-09-16T14:40:05.818 INFO:teuthology.orchestra.run.smithi026.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T14:40:05.818 INFO:teuthology.orchestra.run.smithi026.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T14:40:05.818 INFO:teuthology.orchestra.run.smithi026.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T14:40:05.818 INFO:teuthology.orchestra.run.smithi026.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T14:40:06.794 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:06 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:06.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:06 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:06.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:06 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:06.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:06 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:06.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:06 ntpd[18509]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T14:40:06.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:06 ntpd[18509]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T14:40:06.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:06 ntpd[18509]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T14:40:06.795 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:06 ntpd[18509]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T14:40:06.796 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 14:40:06 ntpd[18509]: ntpd: time slew -0.001652 s 2024-09-16T14:40:06.796 INFO:teuthology.orchestra.run.smithi189.stdout:ntpd: time slew -0.001652s 2024-09-16T14:40:06.796 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:06 ntpd[18509]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T14:40:06.797 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 14:40:06 ntpd[18509]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T14:40:06.846 INFO:teuthology.orchestra.run.smithi189.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T14:40:06.846 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================== 2024-09-16T14:40:06.846 INFO:teuthology.orchestra.run.smithi189.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T14:40:06.846 INFO:teuthology.orchestra.run.smithi189.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T14:40:06.846 INFO:teuthology.orchestra.run.smithi189.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T14:40:06.846 INFO:teuthology.orchestra.run.smithi189.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T14:40:06.846 INFO:teuthology.run_tasks:Running task install... 2024-09-16T14:40:06.855 DEBUG:teuthology.task.install:project ceph 2024-09-16T14:40:06.855 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}} 2024-09-16T14:40:06.855 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'} 2024-09-16T14:40:06.855 INFO:teuthology.task.install:Using flavor: default 2024-09-16T14:40:06.861 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T14:40:06.861 INFO:teuthology.task.install:extra packages: [] 2024-09-16T14:40:06.862 DEBUG:teuthology.orchestra.run.smithi026:> sudo apt-key list | grep Ceph 2024-09-16T14:40:06.862 DEBUG:teuthology.orchestra.run.smithi189:> sudo apt-key list | grep Ceph 2024-09-16T14:40:06.929 INFO:teuthology.orchestra.run.smithi026.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T14:40:06.942 INFO:teuthology.orchestra.run.smithi189.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T14:40:06.960 INFO:teuthology.orchestra.run.smithi026.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T14:40:06.961 INFO:teuthology.orchestra.run.smithi026.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T14:40:06.961 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T14:40:06.961 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T14:40:06.969 INFO:teuthology.orchestra.run.smithi189.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T14:40:06.969 INFO:teuthology.orchestra.run.smithi189.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T14:40:06.969 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T14:40:06.969 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T14:40:07.112 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default/ 2024-09-16T14:40:07.112 INFO:teuthology.task.install.deb:Package version is 18.2.4-845-ge4e227b3-1jammy 2024-09-16T14:40:07.115 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default/ 2024-09-16T14:40:07.115 INFO:teuthology.task.install.deb:Package version is 18.2.4-845-ge4e227b3-1jammy 2024-09-16T14:40:07.227 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:40:07.227 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T14:40:07.233 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:40:07.233 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T14:40:07.242 DEBUG:teuthology.orchestra.run.smithi026:> sudo apt-get update 2024-09-16T14:40:07.248 DEBUG:teuthology.orchestra.run.smithi189:> sudo apt-get update 2024-09-16T14:40:07.415 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T14:40:07.420 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T14:40:07.427 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T14:40:07.445 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T14:40:07.468 INFO:teuthology.orchestra.run.smithi026.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T14:40:07.472 INFO:teuthology.orchestra.run.smithi189.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T14:40:07.489 INFO:teuthology.orchestra.run.smithi026.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T14:40:07.497 INFO:teuthology.orchestra.run.smithi189.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T14:40:07.511 INFO:teuthology.orchestra.run.smithi026.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T14:40:07.518 INFO:teuthology.orchestra.run.smithi189.stdout:Ign:6 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T14:40:07.532 INFO:teuthology.orchestra.run.smithi026.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-09-16T14:40:07.538 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T14:40:07.539 INFO:teuthology.orchestra.run.smithi189.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,411 B] 2024-09-16T14:40:07.544 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T14:40:07.560 INFO:teuthology.orchestra.run.smithi026.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,411 B] 2024-09-16T14:40:07.565 INFO:teuthology.orchestra.run.smithi189.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-09-16T14:40:07.623 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T14:40:07.699 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T14:40:07.756 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 28.8 kB in 0s (88.2 kB/s) 2024-09-16T14:40:07.843 INFO:teuthology.orchestra.run.smithi026.stdout:Fetched 28.8 kB in 0s (69.6 kB/s) 2024-09-16T14:40:08.730 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2024-09-16T14:40:08.747 DEBUG:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-845-ge4e227b3-1jammy cephadm=18.2.4-845-ge4e227b3-1jammy ceph-mds=18.2.4-845-ge4e227b3-1jammy ceph-mgr=18.2.4-845-ge4e227b3-1jammy ceph-common=18.2.4-845-ge4e227b3-1jammy ceph-fuse=18.2.4-845-ge4e227b3-1jammy ceph-test=18.2.4-845-ge4e227b3-1jammy ceph-volume=18.2.4-845-ge4e227b3-1jammy radosgw=18.2.4-845-ge4e227b3-1jammy python3-rados=18.2.4-845-ge4e227b3-1jammy python3-rgw=18.2.4-845-ge4e227b3-1jammy python3-cephfs=18.2.4-845-ge4e227b3-1jammy python3-rbd=18.2.4-845-ge4e227b3-1jammy libcephfs2=18.2.4-845-ge4e227b3-1jammy libcephfs-dev=18.2.4-845-ge4e227b3-1jammy librados2=18.2.4-845-ge4e227b3-1jammy librbd1=18.2.4-845-ge4e227b3-1jammy rbd-fuse=18.2.4-845-ge4e227b3-1jammy 2024-09-16T14:40:08.801 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2024-09-16T14:40:08.870 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2024-09-16T14:40:08.887 DEBUG:teuthology.orchestra.run.smithi026:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-845-ge4e227b3-1jammy cephadm=18.2.4-845-ge4e227b3-1jammy ceph-mds=18.2.4-845-ge4e227b3-1jammy ceph-mgr=18.2.4-845-ge4e227b3-1jammy ceph-common=18.2.4-845-ge4e227b3-1jammy ceph-fuse=18.2.4-845-ge4e227b3-1jammy ceph-test=18.2.4-845-ge4e227b3-1jammy ceph-volume=18.2.4-845-ge4e227b3-1jammy radosgw=18.2.4-845-ge4e227b3-1jammy python3-rados=18.2.4-845-ge4e227b3-1jammy python3-rgw=18.2.4-845-ge4e227b3-1jammy python3-cephfs=18.2.4-845-ge4e227b3-1jammy python3-rbd=18.2.4-845-ge4e227b3-1jammy libcephfs2=18.2.4-845-ge4e227b3-1jammy libcephfs-dev=18.2.4-845-ge4e227b3-1jammy librados2=18.2.4-845-ge4e227b3-1jammy librbd1=18.2.4-845-ge4e227b3-1jammy rbd-fuse=18.2.4-845-ge4e227b3-1jammy 2024-09-16T14:40:08.958 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2024-09-16T14:40:08.986 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2024-09-16T14:40:08.988 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2024-09-16T14:40:09.171 INFO:teuthology.orchestra.run.smithi189.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T14:40:09.171 INFO:teuthology.orchestra.run.smithi189.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T14:40:09.171 INFO:teuthology.orchestra.run.smithi189.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T14:40:09.171 INFO:teuthology.orchestra.run.smithi189.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T14:40:09.172 INFO:teuthology.orchestra.run.smithi026.stdout:Building dependency tree... 2024-09-16T14:40:09.173 INFO:teuthology.orchestra.run.smithi189.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T14:40:09.173 INFO:teuthology.orchestra.run.smithi189.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T14:40:09.173 INFO:teuthology.orchestra.run.smithi189.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T14:40:09.173 INFO:teuthology.orchestra.run.smithi189.stdout:The following additional packages will be installed: 2024-09-16T14:40:09.174 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T14:40:09.174 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T14:40:09.174 INFO:teuthology.orchestra.run.smithi189.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2024-09-16T14:40:09.174 INFO:teuthology.orchestra.run.smithi189.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T14:40:09.174 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2024-09-16T14:40:09.174 INFO:teuthology.orchestra.run.smithi189.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T14:40:09.174 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T14:40:09.174 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T14:40:09.174 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T14:40:09.174 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T14:40:09.174 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T14:40:09.174 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T14:40:09.174 INFO:teuthology.orchestra.run.smithi189.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T14:40:09.174 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T14:40:09.175 INFO:teuthology.orchestra.run.smithi189.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T14:40:09.175 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T14:40:09.175 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T14:40:09.175 INFO:teuthology.orchestra.run.smithi189.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T14:40:09.175 INFO:teuthology.orchestra.run.smithi189.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T14:40:09.175 INFO:teuthology.orchestra.run.smithi189.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T14:40:09.175 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T14:40:09.175 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2024-09-16T14:40:09.176 INFO:teuthology.orchestra.run.smithi026.stdout:Reading state information... 2024-09-16T14:40:09.176 INFO:teuthology.orchestra.run.smithi189.stdout:Suggested packages: 2024-09-16T14:40:09.176 INFO:teuthology.orchestra.run.smithi189.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2024-09-16T14:40:09.176 INFO:teuthology.orchestra.run.smithi189.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2024-09-16T14:40:09.176 INFO:teuthology.orchestra.run.smithi189.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2024-09-16T14:40:09.176 INFO:teuthology.orchestra.run.smithi189.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2024-09-16T14:40:09.176 INFO:teuthology.orchestra.run.smithi189.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2024-09-16T14:40:09.176 INFO:teuthology.orchestra.run.smithi189.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2024-09-16T14:40:09.177 INFO:teuthology.orchestra.run.smithi189.stdout: python3-watchdog 2024-09-16T14:40:09.177 INFO:teuthology.orchestra.run.smithi189.stdout:Recommended packages: 2024-09-16T14:40:09.177 INFO:teuthology.orchestra.run.smithi189.stdout: btrfs-tools 2024-09-16T14:40:09.276 INFO:teuthology.orchestra.run.smithi189.stdout:The following NEW packages will be installed: 2024-09-16T14:40:09.276 INFO:teuthology.orchestra.run.smithi189.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T14:40:09.276 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T14:40:09.276 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T14:40:09.277 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2024-09-16T14:40:09.277 INFO:teuthology.orchestra.run.smithi189.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-16T14:40:09.277 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2024-09-16T14:40:09.278 INFO:teuthology.orchestra.run.smithi189.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2024-09-16T14:40:09.278 INFO:teuthology.orchestra.run.smithi189.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-16T14:40:09.278 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-16T14:40:09.278 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-09-16T14:40:09.278 INFO:teuthology.orchestra.run.smithi189.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-09-16T14:40:09.278 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-09-16T14:40:09.278 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-09-16T14:40:09.278 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-09-16T14:40:09.279 INFO:teuthology.orchestra.run.smithi189.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-16T14:40:09.279 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-09-16T14:40:09.279 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-09-16T14:40:09.279 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2024-09-16T14:40:09.279 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-09-16T14:40:09.279 INFO:teuthology.orchestra.run.smithi189.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T14:40:09.279 INFO:teuthology.orchestra.run.smithi189.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T14:40:09.279 INFO:teuthology.orchestra.run.smithi189.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T14:40:09.279 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T14:40:09.279 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-09-16T14:40:09.279 INFO:teuthology.orchestra.run.smithi189.stdout: socat xmlstarlet 2024-09-16T14:40:09.336 INFO:teuthology.orchestra.run.smithi189.stdout:0 upgraded, 109 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T14:40:09.337 INFO:teuthology.orchestra.run.smithi189.stdout:Need to get 163 MB of archives. 2024-09-16T14:40:09.337 INFO:teuthology.orchestra.run.smithi189.stdout:After this operation, 746 MB of additional disk space will be used. 2024-09-16T14:40:09.337 INFO:teuthology.orchestra.run.smithi189.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2024-09-16T14:40:09.388 INFO:teuthology.orchestra.run.smithi026.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T14:40:09.389 INFO:teuthology.orchestra.run.smithi026.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T14:40:09.389 INFO:teuthology.orchestra.run.smithi026.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T14:40:09.389 INFO:teuthology.orchestra.run.smithi026.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T14:40:09.389 INFO:teuthology.orchestra.run.smithi026.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T14:40:09.389 INFO:teuthology.orchestra.run.smithi026.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T14:40:09.390 INFO:teuthology.orchestra.run.smithi026.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T14:40:09.390 INFO:teuthology.orchestra.run.smithi026.stdout:The following additional packages will be installed: 2024-09-16T14:40:09.390 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T14:40:09.391 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T14:40:09.391 INFO:teuthology.orchestra.run.smithi026.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2024-09-16T14:40:09.391 INFO:teuthology.orchestra.run.smithi026.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T14:40:09.391 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2024-09-16T14:40:09.392 INFO:teuthology.orchestra.run.smithi026.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T14:40:09.392 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T14:40:09.392 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T14:40:09.392 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T14:40:09.392 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T14:40:09.392 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T14:40:09.392 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T14:40:09.392 INFO:teuthology.orchestra.run.smithi026.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T14:40:09.392 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T14:40:09.393 INFO:teuthology.orchestra.run.smithi026.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T14:40:09.393 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T14:40:09.393 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T14:40:09.393 INFO:teuthology.orchestra.run.smithi026.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T14:40:09.393 INFO:teuthology.orchestra.run.smithi026.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T14:40:09.393 INFO:teuthology.orchestra.run.smithi026.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T14:40:09.393 INFO:teuthology.orchestra.run.smithi026.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T14:40:09.393 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2024-09-16T14:40:09.394 INFO:teuthology.orchestra.run.smithi026.stdout:Suggested packages: 2024-09-16T14:40:09.394 INFO:teuthology.orchestra.run.smithi026.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2024-09-16T14:40:09.394 INFO:teuthology.orchestra.run.smithi026.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2024-09-16T14:40:09.394 INFO:teuthology.orchestra.run.smithi026.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2024-09-16T14:40:09.394 INFO:teuthology.orchestra.run.smithi026.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2024-09-16T14:40:09.394 INFO:teuthology.orchestra.run.smithi026.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2024-09-16T14:40:09.394 INFO:teuthology.orchestra.run.smithi026.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2024-09-16T14:40:09.395 INFO:teuthology.orchestra.run.smithi026.stdout: python3-watchdog 2024-09-16T14:40:09.395 INFO:teuthology.orchestra.run.smithi026.stdout:Recommended packages: 2024-09-16T14:40:09.395 INFO:teuthology.orchestra.run.smithi026.stdout: btrfs-tools 2024-09-16T14:40:09.395 INFO:teuthology.orchestra.run.smithi189.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T14:40:09.419 INFO:teuthology.orchestra.run.smithi189.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.4-845-ge4e227b3-1jammy [3,451 kB] 2024-09-16T14:40:09.431 INFO:teuthology.orchestra.run.smithi189.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T14:40:09.434 INFO:teuthology.orchestra.run.smithi189.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T14:40:09.449 INFO:teuthology.orchestra.run.smithi189.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T14:40:09.501 INFO:teuthology.orchestra.run.smithi189.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T14:40:09.501 INFO:teuthology.orchestra.run.smithi026.stdout:The following NEW packages will be installed: 2024-09-16T14:40:09.502 INFO:teuthology.orchestra.run.smithi026.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T14:40:09.502 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T14:40:09.502 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T14:40:09.503 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2024-09-16T14:40:09.503 INFO:teuthology.orchestra.run.smithi026.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-16T14:40:09.503 INFO:teuthology.orchestra.run.smithi026.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2024-09-16T14:40:09.503 INFO:teuthology.orchestra.run.smithi026.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2024-09-16T14:40:09.503 INFO:teuthology.orchestra.run.smithi026.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-16T14:40:09.503 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-16T14:40:09.504 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-09-16T14:40:09.504 INFO:teuthology.orchestra.run.smithi026.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-09-16T14:40:09.504 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-09-16T14:40:09.504 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-09-16T14:40:09.504 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-09-16T14:40:09.504 INFO:teuthology.orchestra.run.smithi026.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-16T14:40:09.505 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-09-16T14:40:09.505 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-09-16T14:40:09.505 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2024-09-16T14:40:09.505 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-09-16T14:40:09.505 INFO:teuthology.orchestra.run.smithi026.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T14:40:09.505 INFO:teuthology.orchestra.run.smithi026.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T14:40:09.505 INFO:teuthology.orchestra.run.smithi026.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T14:40:09.505 INFO:teuthology.orchestra.run.smithi026.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T14:40:09.505 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-09-16T14:40:09.505 INFO:teuthology.orchestra.run.smithi026.stdout: socat xmlstarlet 2024-09-16T14:40:09.506 INFO:teuthology.orchestra.run.smithi189.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T14:40:09.517 INFO:teuthology.orchestra.run.smithi189.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T14:40:09.521 INFO:teuthology.orchestra.run.smithi189.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T14:40:09.521 INFO:teuthology.orchestra.run.smithi189.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T14:40:09.522 INFO:teuthology.orchestra.run.smithi189.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T14:40:09.530 INFO:teuthology.orchestra.run.smithi189.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T14:40:09.531 INFO:teuthology.orchestra.run.smithi189.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T14:40:09.531 INFO:teuthology.orchestra.run.smithi189.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T14:40:09.532 INFO:teuthology.orchestra.run.smithi189.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T14:40:09.539 INFO:teuthology.orchestra.run.smithi189.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T14:40:09.539 INFO:teuthology.orchestra.run.smithi189.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T14:40:09.556 INFO:teuthology.orchestra.run.smithi189.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T14:40:09.556 INFO:teuthology.orchestra.run.smithi189.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T14:40:09.557 INFO:teuthology.orchestra.run.smithi189.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T14:40:09.557 INFO:teuthology.orchestra.run.smithi189.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T14:40:09.560 INFO:teuthology.orchestra.run.smithi189.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T14:40:09.561 INFO:teuthology.orchestra.run.smithi189.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T14:40:09.561 INFO:teuthology.orchestra.run.smithi189.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T14:40:09.561 INFO:teuthology.orchestra.run.smithi189.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T14:40:09.564 INFO:teuthology.orchestra.run.smithi026.stdout:0 upgraded, 109 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T14:40:09.564 INFO:teuthology.orchestra.run.smithi026.stdout:Need to get 163 MB of archives. 2024-09-16T14:40:09.564 INFO:teuthology.orchestra.run.smithi026.stdout:After this operation, 746 MB of additional disk space will be used. 2024-09-16T14:40:09.564 INFO:teuthology.orchestra.run.smithi026.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2024-09-16T14:40:09.573 INFO:teuthology.orchestra.run.smithi189.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T14:40:09.573 INFO:teuthology.orchestra.run.smithi189.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T14:40:09.590 INFO:teuthology.orchestra.run.smithi189.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T14:40:09.592 INFO:teuthology.orchestra.run.smithi189.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T14:40:09.592 INFO:teuthology.orchestra.run.smithi189.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T14:40:09.593 INFO:teuthology.orchestra.run.smithi189.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T14:40:09.598 INFO:teuthology.orchestra.run.smithi189.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T14:40:09.599 INFO:teuthology.orchestra.run.smithi189.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T14:40:09.599 INFO:teuthology.orchestra.run.smithi189.stdout:Get:35 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.4-845-ge4e227b3-1jammy [3,108 kB] 2024-09-16T14:40:09.599 INFO:teuthology.orchestra.run.smithi189.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T14:40:09.600 INFO:teuthology.orchestra.run.smithi189.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T14:40:09.607 INFO:teuthology.orchestra.run.smithi189.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T14:40:09.609 INFO:teuthology.orchestra.run.smithi189.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T14:40:09.619 INFO:teuthology.orchestra.run.smithi189.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T14:40:09.620 INFO:teuthology.orchestra.run.smithi189.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T14:40:09.621 INFO:teuthology.orchestra.run.smithi189.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T14:40:09.622 INFO:teuthology.orchestra.run.smithi026.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T14:40:09.623 INFO:teuthology.orchestra.run.smithi189.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T14:40:09.624 INFO:teuthology.orchestra.run.smithi189.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T14:40:09.625 INFO:teuthology.orchestra.run.smithi189.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T14:40:09.627 INFO:teuthology.orchestra.run.smithi189.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T14:40:09.644 INFO:teuthology.orchestra.run.smithi189.stdout:Get:47 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.4-845-ge4e227b3-1jammy [683 kB] 2024-09-16T14:40:09.651 INFO:teuthology.orchestra.run.smithi189.stdout:Get:48 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.4-845-ge4e227b3-1jammy [357 kB] 2024-09-16T14:40:09.657 INFO:teuthology.orchestra.run.smithi189.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.4-845-ge4e227b3-1jammy [32.8 kB] 2024-09-16T14:40:09.658 INFO:teuthology.orchestra.run.smithi026.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T14:40:09.658 INFO:teuthology.orchestra.run.smithi189.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.4-845-ge4e227b3-1jammy [180 kB] 2024-09-16T14:40:09.660 INFO:teuthology.orchestra.run.smithi189.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.4-845-ge4e227b3-1jammy [63.7 kB] 2024-09-16T14:40:09.661 INFO:teuthology.orchestra.run.smithi026.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T14:40:09.662 INFO:teuthology.orchestra.run.smithi189.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.4-845-ge4e227b3-1jammy [333 kB] 2024-09-16T14:40:09.668 INFO:teuthology.orchestra.run.smithi189.stdout:Get:53 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.4-845-ge4e227b3-1jammy [5,934 kB] 2024-09-16T14:40:09.676 INFO:teuthology.orchestra.run.smithi026.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T14:40:09.680 INFO:teuthology.orchestra.run.smithi189.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T14:40:09.681 INFO:teuthology.orchestra.run.smithi189.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T14:40:09.682 INFO:teuthology.orchestra.run.smithi189.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T14:40:09.682 INFO:teuthology.orchestra.run.smithi189.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T14:40:09.683 INFO:teuthology.orchestra.run.smithi189.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T14:40:09.683 INFO:teuthology.orchestra.run.smithi189.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T14:40:09.684 INFO:teuthology.orchestra.run.smithi189.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T14:40:09.686 INFO:teuthology.orchestra.run.smithi189.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T14:40:09.686 INFO:teuthology.orchestra.run.smithi189.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T14:40:09.699 INFO:teuthology.orchestra.run.smithi189.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T14:40:09.716 INFO:teuthology.orchestra.run.smithi189.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T14:40:09.716 INFO:teuthology.orchestra.run.smithi026.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.4-845-ge4e227b3-1jammy [3,451 kB] 2024-09-16T14:40:09.718 INFO:teuthology.orchestra.run.smithi189.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T14:40:09.718 INFO:teuthology.orchestra.run.smithi189.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T14:40:09.732 INFO:teuthology.orchestra.run.smithi026.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T14:40:09.733 INFO:teuthology.orchestra.run.smithi189.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T14:40:09.733 INFO:teuthology.orchestra.run.smithi189.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T14:40:09.733 INFO:teuthology.orchestra.run.smithi189.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T14:40:09.735 INFO:teuthology.orchestra.run.smithi189.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T14:40:09.735 INFO:teuthology.orchestra.run.smithi189.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T14:40:09.735 INFO:teuthology.orchestra.run.smithi189.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T14:40:09.735 INFO:teuthology.orchestra.run.smithi189.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T14:40:09.737 INFO:teuthology.orchestra.run.smithi026.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T14:40:09.748 INFO:teuthology.orchestra.run.smithi026.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T14:40:09.752 INFO:teuthology.orchestra.run.smithi189.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T14:40:09.752 INFO:teuthology.orchestra.run.smithi026.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T14:40:09.753 INFO:teuthology.orchestra.run.smithi026.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T14:40:09.754 INFO:teuthology.orchestra.run.smithi026.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T14:40:09.755 INFO:teuthology.orchestra.run.smithi189.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T14:40:09.756 INFO:teuthology.orchestra.run.smithi189.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T14:40:09.758 INFO:teuthology.orchestra.run.smithi189.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T14:40:09.762 INFO:teuthology.orchestra.run.smithi026.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T14:40:09.763 INFO:teuthology.orchestra.run.smithi026.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T14:40:09.763 INFO:teuthology.orchestra.run.smithi026.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T14:40:09.764 INFO:teuthology.orchestra.run.smithi026.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T14:40:09.767 INFO:teuthology.orchestra.run.smithi189.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.4-845-ge4e227b3-1jammy [112 kB] 2024-09-16T14:40:09.769 INFO:teuthology.orchestra.run.smithi189.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.4-845-ge4e227b3-1jammy [445 kB] 2024-09-16T14:40:09.769 INFO:teuthology.orchestra.run.smithi189.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T14:40:09.771 INFO:teuthology.orchestra.run.smithi026.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T14:40:09.772 INFO:teuthology.orchestra.run.smithi026.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T14:40:09.772 INFO:teuthology.orchestra.run.smithi026.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T14:40:09.772 INFO:teuthology.orchestra.run.smithi189.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T14:40:09.774 INFO:teuthology.orchestra.run.smithi189.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T14:40:09.775 INFO:teuthology.orchestra.run.smithi189.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.4-845-ge4e227b3-1jammy [22.0 MB] 2024-09-16T14:40:09.778 INFO:teuthology.orchestra.run.smithi189.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T14:40:09.781 INFO:teuthology.orchestra.run.smithi189.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T14:40:09.781 INFO:teuthology.orchestra.run.smithi189.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T14:40:09.786 INFO:teuthology.orchestra.run.smithi189.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T14:40:09.790 INFO:teuthology.orchestra.run.smithi026.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T14:40:09.790 INFO:teuthology.orchestra.run.smithi026.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T14:40:09.790 INFO:teuthology.orchestra.run.smithi026.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T14:40:09.793 INFO:teuthology.orchestra.run.smithi189.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T14:40:09.793 INFO:teuthology.orchestra.run.smithi189.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T14:40:09.793 INFO:teuthology.orchestra.run.smithi026.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T14:40:09.794 INFO:teuthology.orchestra.run.smithi189.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T14:40:09.794 INFO:teuthology.orchestra.run.smithi026.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T14:40:09.794 INFO:teuthology.orchestra.run.smithi026.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T14:40:09.795 INFO:teuthology.orchestra.run.smithi026.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T14:40:09.807 INFO:teuthology.orchestra.run.smithi026.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T14:40:09.807 INFO:teuthology.orchestra.run.smithi026.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T14:40:09.808 INFO:teuthology.orchestra.run.smithi026.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T14:40:09.824 INFO:teuthology.orchestra.run.smithi026.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T14:40:09.825 INFO:teuthology.orchestra.run.smithi026.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T14:40:09.825 INFO:teuthology.orchestra.run.smithi026.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T14:40:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T14:40:09.831 INFO:teuthology.orchestra.run.smithi026.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T14:40:09.832 INFO:teuthology.orchestra.run.smithi026.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T14:40:09.833 INFO:teuthology.orchestra.run.smithi026.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T14:40:09.841 INFO:teuthology.orchestra.run.smithi026.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T14:40:09.843 INFO:teuthology.orchestra.run.smithi026.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T14:40:09.844 INFO:teuthology.orchestra.run.smithi026.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T14:40:09.858 INFO:teuthology.orchestra.run.smithi026.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T14:40:09.859 INFO:teuthology.orchestra.run.smithi026.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T14:40:09.862 INFO:teuthology.orchestra.run.smithi026.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T14:40:09.863 INFO:teuthology.orchestra.run.smithi026.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T14:40:09.864 INFO:teuthology.orchestra.run.smithi026.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T14:40:09.868 INFO:teuthology.orchestra.run.smithi026.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T14:40:09.923 INFO:teuthology.orchestra.run.smithi026.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T14:40:09.924 INFO:teuthology.orchestra.run.smithi026.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T14:40:09.925 INFO:teuthology.orchestra.run.smithi026.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T14:40:09.925 INFO:teuthology.orchestra.run.smithi026.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T14:40:09.926 INFO:teuthology.orchestra.run.smithi026.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T14:40:09.935 INFO:teuthology.orchestra.run.smithi026.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T14:40:09.935 INFO:teuthology.orchestra.run.smithi026.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T14:40:09.937 INFO:teuthology.orchestra.run.smithi026.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T14:40:09.938 INFO:teuthology.orchestra.run.smithi026.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T14:40:09.942 INFO:teuthology.orchestra.run.smithi026.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T14:40:09.961 INFO:teuthology.orchestra.run.smithi026.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T14:40:09.963 INFO:teuthology.orchestra.run.smithi026.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T14:40:09.963 INFO:teuthology.orchestra.run.smithi026.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T14:40:09.980 INFO:teuthology.orchestra.run.smithi026.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T14:40:09.980 INFO:teuthology.orchestra.run.smithi026.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T14:40:09.980 INFO:teuthology.orchestra.run.smithi026.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T14:40:09.980 INFO:teuthology.orchestra.run.smithi026.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T14:40:09.981 INFO:teuthology.orchestra.run.smithi026.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T14:40:09.982 INFO:teuthology.orchestra.run.smithi026.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T14:40:09.982 INFO:teuthology.orchestra.run.smithi026.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T14:40:09.997 INFO:teuthology.orchestra.run.smithi026.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T14:40:10.001 INFO:teuthology.orchestra.run.smithi026.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T14:40:10.014 INFO:teuthology.orchestra.run.smithi026.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T14:40:10.015 INFO:teuthology.orchestra.run.smithi026.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T14:40:10.016 INFO:teuthology.orchestra.run.smithi026.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T14:40:10.020 INFO:teuthology.orchestra.run.smithi026.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T14:40:10.022 INFO:teuthology.orchestra.run.smithi026.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T14:40:10.026 INFO:teuthology.orchestra.run.smithi026.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T14:40:10.029 INFO:teuthology.orchestra.run.smithi026.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T14:40:10.030 INFO:teuthology.orchestra.run.smithi026.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T14:40:10.031 INFO:teuthology.orchestra.run.smithi026.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T14:40:10.038 INFO:teuthology.orchestra.run.smithi026.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T14:40:10.049 INFO:teuthology.orchestra.run.smithi026.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T14:40:10.050 INFO:teuthology.orchestra.run.smithi026.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T14:40:10.228 INFO:teuthology.orchestra.run.smithi189.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.4-845-ge4e227b3-1jammy [5,005 kB] 2024-09-16T14:40:10.283 INFO:teuthology.orchestra.run.smithi189.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.4-845-ge4e227b3-1jammy [241 kB] 2024-09-16T14:40:10.285 INFO:teuthology.orchestra.run.smithi189.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.4-845-ge4e227b3-1jammy [130 kB] 2024-09-16T14:40:10.287 INFO:teuthology.orchestra.run.smithi189.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.4-845-ge4e227b3-1jammy [1,494 kB] 2024-09-16T14:40:10.299 INFO:teuthology.orchestra.run.smithi189.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.4-845-ge4e227b3-1jammy [6,085 kB] 2024-09-16T14:40:10.405 INFO:teuthology.orchestra.run.smithi189.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.4-845-ge4e227b3-1jammy [22.4 MB] 2024-09-16T14:40:10.800 INFO:teuthology.orchestra.run.smithi026.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.4-845-ge4e227b3-1jammy [3,108 kB] 2024-09-16T14:40:10.929 INFO:teuthology.orchestra.run.smithi189.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.4-845-ge4e227b3-1jammy [14.1 kB] 2024-09-16T14:40:10.929 INFO:teuthology.orchestra.run.smithi189.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.4-845-ge4e227b3-1jammy [834 kB] 2024-09-16T14:40:10.941 INFO:teuthology.orchestra.run.smithi189.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.4-845-ge4e227b3-1jammy [2,159 kB] 2024-09-16T14:40:10.973 INFO:teuthology.orchestra.run.smithi189.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.4-845-ge4e227b3-1jammy [240 kB] 2024-09-16T14:40:10.976 INFO:teuthology.orchestra.run.smithi189.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.4-845-ge4e227b3-1jammy [145 kB] 2024-09-16T14:40:10.978 INFO:teuthology.orchestra.run.smithi189.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.4-845-ge4e227b3-1jammy [2,233 kB] 2024-09-16T14:40:11.002 INFO:teuthology.orchestra.run.smithi189.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.4-845-ge4e227b3-1jammy [8,625 kB] 2024-09-16T14:40:11.049 INFO:teuthology.orchestra.run.smithi026.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.4-845-ge4e227b3-1jammy [683 kB] 2024-09-16T14:40:11.146 INFO:teuthology.orchestra.run.smithi189.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.4-845-ge4e227b3-1jammy [14.3 kB] 2024-09-16T14:40:11.146 INFO:teuthology.orchestra.run.smithi189.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.4-845-ge4e227b3-1jammy [42.8 MB] 2024-09-16T14:40:11.160 INFO:teuthology.orchestra.run.smithi026.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.4-845-ge4e227b3-1jammy [357 kB] 2024-09-16T14:40:11.172 INFO:teuthology.orchestra.run.smithi026.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.4-845-ge4e227b3-1jammy [32.8 kB] 2024-09-16T14:40:11.173 INFO:teuthology.orchestra.run.smithi026.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.4-845-ge4e227b3-1jammy [180 kB] 2024-09-16T14:40:11.221 INFO:teuthology.orchestra.run.smithi026.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.4-845-ge4e227b3-1jammy [63.7 kB] 2024-09-16T14:40:11.222 INFO:teuthology.orchestra.run.smithi026.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.4-845-ge4e227b3-1jammy [333 kB] 2024-09-16T14:40:11.277 INFO:teuthology.orchestra.run.smithi026.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.4-845-ge4e227b3-1jammy [5,934 kB] 2024-09-16T14:40:11.829 INFO:teuthology.orchestra.run.smithi026.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.4-845-ge4e227b3-1jammy [112 kB] 2024-09-16T14:40:11.831 INFO:teuthology.orchestra.run.smithi026.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.4-845-ge4e227b3-1jammy [445 kB] 2024-09-16T14:40:11.884 INFO:teuthology.orchestra.run.smithi026.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.4-845-ge4e227b3-1jammy [22.0 MB] 2024-09-16T14:40:12.142 INFO:teuthology.orchestra.run.smithi189.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.4-845-ge4e227b3-1jammy [122 kB] 2024-09-16T14:40:12.209 INFO:teuthology.orchestra.run.smithi189.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.4-845-ge4e227b3-1jammy [38.3 kB] 2024-09-16T14:40:12.210 INFO:teuthology.orchestra.run.smithi189.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.4-845-ge4e227b3-1jammy [11.7 MB] 2024-09-16T14:40:12.447 INFO:teuthology.orchestra.run.smithi189.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.4-845-ge4e227b3-1jammy [93.1 kB] 2024-09-16T14:40:12.628 INFO:teuthology.orchestra.run.smithi026.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.4-845-ge4e227b3-1jammy [5,005 kB] 2024-09-16T14:40:12.703 INFO:teuthology.orchestra.run.smithi026.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.4-845-ge4e227b3-1jammy [241 kB] 2024-09-16T14:40:12.706 INFO:teuthology.orchestra.run.smithi026.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.4-845-ge4e227b3-1jammy [130 kB] 2024-09-16T14:40:12.707 INFO:teuthology.orchestra.run.smithi026.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.4-845-ge4e227b3-1jammy [1,494 kB] 2024-09-16T14:40:12.721 INFO:teuthology.orchestra.run.smithi026.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.4-845-ge4e227b3-1jammy [6,085 kB] 2024-09-16T14:40:12.801 INFO:teuthology.orchestra.run.smithi026.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.4-845-ge4e227b3-1jammy [22.4 MB] 2024-09-16T14:40:13.019 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 163 MB in 3s (51.5 MB/s) 2024-09-16T14:40:13.135 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libfmt8:amd64. 2024-09-16T14:40:13.196 INFO:teuthology.orchestra.run.smithi189.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T14:40:13.201 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2024-09-16T14:40:13.245 INFO:teuthology.orchestra.run.smithi026.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.4-845-ge4e227b3-1jammy [14.1 kB] 2024-09-16T14:40:13.245 INFO:teuthology.orchestra.run.smithi026.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.4-845-ge4e227b3-1jammy [834 kB] 2024-09-16T14:40:13.255 INFO:teuthology.orchestra.run.smithi026.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.4-845-ge4e227b3-1jammy [2,159 kB] 2024-09-16T14:40:13.280 INFO:teuthology.orchestra.run.smithi026.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.4-845-ge4e227b3-1jammy [240 kB] 2024-09-16T14:40:13.283 INFO:teuthology.orchestra.run.smithi026.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.4-845-ge4e227b3-1jammy [145 kB] 2024-09-16T14:40:13.285 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2024-09-16T14:40:13.285 INFO:teuthology.orchestra.run.smithi026.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.4-845-ge4e227b3-1jammy [2,233 kB] 2024-09-16T14:40:13.308 INFO:teuthology.orchestra.run.smithi026.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.4-845-ge4e227b3-1jammy [8,625 kB] 2024-09-16T14:40:13.433 INFO:teuthology.orchestra.run.smithi026.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.4-845-ge4e227b3-1jammy [14.3 kB] 2024-09-16T14:40:13.433 INFO:teuthology.orchestra.run.smithi026.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.4-845-ge4e227b3-1jammy [42.8 MB] 2024-09-16T14:40:13.613 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T14:40:13.639 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T14:40:13.680 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T14:40:14.008 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T14:40:14.034 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T14:40:14.066 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T14:40:14.315 INFO:teuthology.orchestra.run.smithi026.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.4-845-ge4e227b3-1jammy [122 kB] 2024-09-16T14:40:14.317 INFO:teuthology.orchestra.run.smithi026.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.4-845-ge4e227b3-1jammy [38.3 kB] 2024-09-16T14:40:14.318 INFO:teuthology.orchestra.run.smithi026.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.4-845-ge4e227b3-1jammy [11.7 MB] 2024-09-16T14:40:14.419 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T14:40:14.446 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T14:40:14.470 INFO:teuthology.orchestra.run.smithi026.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.4-845-ge4e227b3-1jammy [93.1 kB] 2024-09-16T14:40:14.486 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T14:40:14.931 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T14:40:14.957 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T14:40:15.006 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T14:40:15.043 INFO:teuthology.orchestra.run.smithi026.stdout:Fetched 163 MB in 5s (32.8 MB/s) 2024-09-16T14:40:15.173 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libfmt8:amd64. 2024-09-16T14:40:15.269 INFO:teuthology.orchestra.run.smithi026.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T14:40:15.274 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2024-09-16T14:40:15.351 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T14:40:15.372 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2024-09-16T14:40:15.374 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T14:40:15.417 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T14:40:15.708 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T14:40:15.735 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T14:40:15.771 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T14:40:15.774 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T14:40:15.797 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T14:40:15.837 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T14:40:16.078 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T14:40:16.105 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T14:40:16.136 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T14:40:16.315 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T14:40:16.341 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T14:40:16.382 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T14:40:16.456 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T14:40:16.483 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T14:40:16.523 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T14:40:16.635 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package librados2. 2024-09-16T14:40:16.661 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../008-librados2_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:16.693 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librados2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:16.968 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T14:40:16.995 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T14:40:17.043 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T14:40:17.154 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package librbd1. 2024-09-16T14:40:17.180 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../009-librbd1_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:17.221 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librbd1 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:17.438 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T14:40:17.464 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T14:40:17.504 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T14:40:17.682 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T14:40:17.708 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../010-libcephfs2_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:17.749 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libcephfs2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:17.816 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T14:40:17.842 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T14:40:17.874 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T14:40:18.009 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-rados. 2024-09-16T14:40:18.027 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../011-python3-rados_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:18.059 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-rados (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:18.210 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T14:40:18.237 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T14:40:18.277 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T14:40:18.337 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T14:40:18.363 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T14:40:18.403 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-ceph-argparse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:18.605 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package librados2. 2024-09-16T14:40:18.631 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../008-librados2_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:18.656 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T14:40:18.680 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking librados2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:18.682 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../013-python3-cephfs_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:18.723 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cephfs (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:19.042 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T14:40:19.068 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../014-python3-ceph-common_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T14:40:19.109 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-ceph-common (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:19.133 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package librbd1. 2024-09-16T14:40:19.160 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../009-librbd1_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:19.199 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking librbd1 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:19.395 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T14:40:19.418 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T14:40:19.453 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T14:40:19.610 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T14:40:19.637 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../010-libcephfs2_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:19.677 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libcephfs2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:19.714 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T14:40:19.730 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T14:40:19.764 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T14:40:19.963 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-rados. 2024-09-16T14:40:19.980 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../011-python3-rados_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:20.017 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T14:40:20.021 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-rados (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:20.032 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../017-python3-rbd_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:20.067 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-rbd (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:20.341 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T14:40:20.368 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T14:40:20.395 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T14:40:20.407 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-ceph-argparse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:20.421 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T14:40:20.462 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T14:40:20.651 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T14:40:20.667 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../013-python3-cephfs_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:20.701 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-cephfs (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:20.789 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package librgw2. 2024-09-16T14:40:20.816 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../019-librgw2_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:20.856 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librgw2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:20.979 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T14:40:21.006 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../014-python3-ceph-common_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T14:40:21.046 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-ceph-common (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:21.366 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T14:40:21.392 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T14:40:21.432 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T14:40:21.509 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T14:40:21.537 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:21.575 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-rgw (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:21.760 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T14:40:21.787 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T14:40:21.826 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T14:40:21.853 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T14:40:21.879 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T14:40:21.911 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T14:40:22.087 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T14:40:22.114 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../017-python3-rbd_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:22.154 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-rbd (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:22.231 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T14:40:22.257 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:22.297 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libradosstriper1 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:22.457 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T14:40:22.473 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T14:40:22.507 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T14:40:22.735 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-common. 2024-09-16T14:40:22.762 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../023-ceph-common_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:22.801 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-common (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:22.835 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package librgw2. 2024-09-16T14:40:22.862 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../019-librgw2_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:22.901 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking librgw2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:23.445 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T14:40:23.472 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:23.512 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-rgw (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:23.857 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T14:40:23.883 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T14:40:23.923 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T14:40:24.065 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-base. 2024-09-16T14:40:24.091 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../024-ceph-base_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:24.141 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-base (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:24.210 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T14:40:24.236 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:24.276 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libradosstriper1 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:24.679 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-common. 2024-09-16T14:40:24.706 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../023-ceph-common_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:24.728 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T14:40:24.746 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-common (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:24.754 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T14:40:24.794 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T14:40:25.131 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T14:40:25.157 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T14:40:25.197 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T14:40:25.525 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T14:40:25.552 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T14:40:25.592 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T14:40:25.887 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T14:40:25.913 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T14:40:25.953 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T14:40:26.256 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T14:40:26.283 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T14:40:26.322 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T14:40:26.327 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-base. 2024-09-16T14:40:26.354 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../024-ceph-base_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:26.403 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-base (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:26.626 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T14:40:26.652 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T14:40:26.692 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T14:40:26.953 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-portend. 2024-09-16T14:40:26.968 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2024-09-16T14:40:26.981 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T14:40:26.997 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T14:40:27.003 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T14:40:27.039 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T14:40:27.292 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T14:40:27.298 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T14:40:27.308 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T14:40:27.324 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T14:40:27.342 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T14:40:27.364 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T14:40:27.651 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T14:40:27.670 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T14:40:27.677 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T14:40:27.697 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T14:40:27.717 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T14:40:27.737 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T14:40:28.029 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T14:40:28.048 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T14:40:28.056 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T14:40:28.075 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T14:40:28.087 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T14:40:28.114 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T14:40:28.392 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T14:40:28.398 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T14:40:28.419 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T14:40:28.425 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T14:40:28.459 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T14:40:28.465 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T14:40:28.728 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T14:40:28.755 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T14:40:28.768 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T14:40:28.795 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T14:40:28.795 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T14:40:28.843 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T14:40:29.098 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-portend. 2024-09-16T14:40:29.121 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-mako. 2024-09-16T14:40:29.125 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2024-09-16T14:40:29.148 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T14:40:29.165 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T14:40:29.187 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T14:40:29.467 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T14:40:29.474 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T14:40:29.494 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T14:40:29.501 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T14:40:29.532 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T14:40:29.534 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T14:40:29.795 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T14:40:29.811 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T14:40:29.827 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T14:40:29.845 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T14:40:29.853 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T14:40:29.894 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T14:40:30.132 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T14:40:30.159 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T14:40:30.190 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T14:40:30.196 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-webob. 2024-09-16T14:40:30.223 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T14:40:30.263 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T14:40:30.493 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T14:40:30.520 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T14:40:30.552 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T14:40:30.599 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T14:40:30.626 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T14:40:30.669 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T14:40:30.838 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T14:40:30.865 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T14:40:30.904 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T14:40:30.944 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T14:40:30.970 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T14:40:31.002 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T14:40:31.199 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-mako. 2024-09-16T14:40:31.226 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T14:40:31.255 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-paste. 2024-09-16T14:40:31.265 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T14:40:31.281 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T14:40:31.322 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T14:40:31.633 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T14:40:31.648 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T14:40:31.652 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T14:40:31.679 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T14:40:31.683 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T14:40:31.719 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T14:40:31.944 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T14:40:31.970 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T14:40:31.980 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T14:40:32.006 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T14:40:32.010 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T14:40:32.038 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T14:40:32.314 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T14:40:32.333 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-webob. 2024-09-16T14:40:32.340 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T14:40:32.360 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T14:40:32.380 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T14:40:32.399 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T14:40:32.716 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T14:40:32.743 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T14:40:32.752 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T14:40:32.779 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T14:40:32.783 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T14:40:32.822 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T14:40:33.130 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T14:40:33.146 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T14:40:33.157 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T14:40:33.173 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T14:40:33.196 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T14:40:33.213 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T14:40:33.451 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-paste. 2024-09-16T14:40:33.467 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T14:40:33.482 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-idna. 2024-09-16T14:40:33.498 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../049-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T14:40:33.501 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T14:40:33.532 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T14:40:33.838 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T14:40:33.860 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T14:40:33.865 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T14:40:33.887 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T14:40:33.904 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T14:40:33.927 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T14:40:34.174 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T14:40:34.200 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T14:40:34.232 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T14:40:34.247 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-requests. 2024-09-16T14:40:34.273 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T14:40:34.313 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T14:40:34.543 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T14:40:34.570 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T14:40:34.608 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T14:40:34.610 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T14:40:34.635 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T14:40:34.674 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T14:40:34.921 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T14:40:34.948 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T14:40:34.953 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T14:40:34.968 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T14:40:34.987 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T14:40:35.011 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:35.359 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T14:40:35.386 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T14:40:35.399 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T14:40:35.425 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T14:40:35.426 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:35.466 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:35.770 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-idna. 2024-09-16T14:40:35.797 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../049-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T14:40:35.837 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T14:40:36.044 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T14:40:36.071 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../055-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T14:40:36.098 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T14:40:36.101 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T14:40:36.122 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T14:40:36.156 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T14:40:36.355 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T14:40:36.381 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../056-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T14:40:36.421 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T14:40:36.536 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-requests. 2024-09-16T14:40:36.563 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T14:40:36.643 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T14:40:36.783 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T14:40:36.810 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../057-ceph-mgr_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:36.849 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:36.954 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T14:40:36.981 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T14:40:37.021 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T14:40:37.261 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T14:40:37.282 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T14:40:37.288 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../058-ceph-mon_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:37.298 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T14:40:37.327 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mon (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:37.340 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:37.737 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T14:40:37.764 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:37.803 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:37.872 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T14:40:37.899 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../059-ceph-osd_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:37.939 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-osd (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:38.081 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T14:40:38.108 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../055-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T14:40:38.139 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T14:40:38.426 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T14:40:38.452 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../056-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T14:40:38.492 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T14:40:38.837 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T14:40:38.864 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../057-ceph-mgr_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:38.903 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:38.984 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph. 2024-09-16T14:40:38.999 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../060-ceph_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:39.050 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:39.323 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T14:40:39.350 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../058-ceph-mon_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:39.387 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T14:40:39.389 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mon (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:39.414 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../061-ceph-fuse_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:39.453 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-fuse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:39.757 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T14:40:39.784 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../062-ceph-mds_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:39.823 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mds (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:39.960 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T14:40:39.987 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../059-ceph-osd_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:40.026 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-osd (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:40.218 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package cephadm. 2024-09-16T14:40:40.245 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../063-cephadm_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:40.284 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking cephadm (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:40.612 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T14:40:40.639 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../064-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T14:40:40.679 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T14:40:40.982 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T14:40:40.997 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../065-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T14:40:41.040 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T14:40:41.113 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph. 2024-09-16T14:40:41.129 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../060-ceph_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:41.171 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:41.491 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T14:40:41.517 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../061-ceph-fuse_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:41.557 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-fuse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:42.012 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-babel. 2024-09-16T14:40:42.039 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../066-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T14:40:42.078 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T14:40:42.415 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T14:40:42.442 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../067-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T14:40:42.481 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T14:40:42.744 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T14:40:42.771 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../062-ceph-mds_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:42.810 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mds (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:43.197 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package cephadm. 2024-09-16T14:40:43.224 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../063-cephadm_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:43.263 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking cephadm (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:43.553 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T14:40:43.574 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T14:40:43.579 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../068-ceph-mgr-cephadm_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T14:40:43.601 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../064-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T14:40:43.619 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:43.641 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T14:40:43.939 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T14:40:43.966 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../069-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T14:40:44.002 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T14:40:44.005 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T14:40:44.018 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../065-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T14:40:44.061 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T14:40:44.342 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-py. 2024-09-16T14:40:44.369 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../070-python3-py_1.10.0-1_all.deb ... 2024-09-16T14:40:44.408 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T14:40:44.720 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T14:40:44.747 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../071-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T14:40:44.787 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T14:40:45.031 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-toml. 2024-09-16T14:40:45.058 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../072-python3-toml_0.10.2-1_all.deb ... 2024-09-16T14:40:45.066 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-babel. 2024-09-16T14:40:45.089 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T14:40:45.093 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../066-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T14:40:45.124 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T14:40:45.393 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T14:40:45.419 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T14:40:45.419 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../073-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T14:40:45.446 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../067-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T14:40:45.459 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T14:40:45.485 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T14:40:45.771 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T14:40:45.797 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T14:40:45.798 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../074-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T14:40:45.824 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../068-ceph-mgr-cephadm_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T14:40:45.837 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T14:40:45.863 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:46.141 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-routes. 2024-09-16T14:40:46.167 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../075-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T14:40:46.184 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T14:40:46.211 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../069-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T14:40:46.240 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T14:40:46.250 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T14:40:46.562 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-py. 2024-09-16T14:40:46.577 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T14:40:46.589 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../070-python3-py_1.10.0-1_all.deb ... 2024-09-16T14:40:46.604 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../076-ceph-mgr-dashboard_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T14:40:46.628 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T14:40:46.644 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:46.957 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T14:40:46.984 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../071-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T14:40:47.023 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T14:40:47.318 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-toml. 2024-09-16T14:40:47.345 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../072-python3-toml_0.10.2-1_all.deb ... 2024-09-16T14:40:47.384 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T14:40:47.654 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T14:40:47.681 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../073-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T14:40:47.712 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T14:40:47.833 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T14:40:47.848 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../077-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T14:40:47.882 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T14:40:48.082 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T14:40:48.110 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../074-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T14:40:48.149 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T14:40:48.402 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-routes. 2024-09-16T14:40:48.428 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T14:40:48.429 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../075-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T14:40:48.453 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../078-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T14:40:48.460 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T14:40:48.494 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T14:40:48.755 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T14:40:48.782 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../076-ceph-mgr-dashboard_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T14:40:48.813 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:48.823 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T14:40:48.850 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../079-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T14:40:48.890 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T14:40:49.209 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T14:40:49.236 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../080-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T14:40:49.276 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T14:40:49.953 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T14:40:49.981 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../081-ceph-mgr-diskprediction-local_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T14:40:50.002 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T14:40:50.019 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:50.029 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../077-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T14:40:50.068 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T14:40:50.630 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T14:40:50.658 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../078-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T14:40:50.697 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T14:40:50.989 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T14:40:51.016 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../082-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T14:40:51.055 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T14:40:51.076 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T14:40:51.103 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../079-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T14:40:51.142 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T14:40:51.395 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T14:40:51.422 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../080-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T14:40:51.443 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T14:40:51.453 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T14:40:51.470 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../083-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T14:40:51.510 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T14:40:51.880 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T14:40:51.907 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../084-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T14:40:51.947 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T14:40:52.266 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T14:40:52.294 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../085-python3-rsa_4.8-1_all.deb ... 2024-09-16T14:40:52.333 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T14:40:52.428 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T14:40:52.457 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../081-ceph-mgr-diskprediction-local_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T14:40:52.495 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:52.628 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T14:40:52.656 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../086-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T14:40:52.694 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T14:40:53.006 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T14:40:53.034 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T14:40:53.073 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T14:40:53.384 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T14:40:53.412 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../088-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T14:40:53.451 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T14:40:53.473 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T14:40:53.501 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../082-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T14:40:53.539 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T14:40:53.771 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T14:40:53.798 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../089-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T14:40:53.837 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T14:40:53.894 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T14:40:53.921 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../083-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T14:40:53.952 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T14:40:54.247 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T14:40:54.275 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../084-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T14:40:54.313 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T14:40:54.493 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T14:40:54.509 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T14:40:54.551 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:54.625 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T14:40:54.653 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../085-python3-rsa_4.8-1_all.deb ... 2024-09-16T14:40:54.691 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T14:40:54.870 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T14:40:54.898 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../091-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T14:40:54.937 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T14:40:54.962 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T14:40:54.990 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../086-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T14:40:55.020 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T14:40:55.290 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T14:40:55.318 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../092-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T14:40:55.324 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T14:40:55.348 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T14:40:55.351 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T14:40:55.390 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T14:40:55.601 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package jq. 2024-09-16T14:40:55.629 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../093-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T14:40:55.692 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T14:40:55.709 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T14:40:55.738 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../088-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T14:40:55.776 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T14:40:55.945 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package socat. 2024-09-16T14:40:55.973 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../094-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T14:40:56.011 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T14:40:56.087 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T14:40:56.115 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../089-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T14:40:56.179 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T14:40:56.315 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T14:40:56.331 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T14:40:56.373 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T14:40:56.728 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-test. 2024-09-16T14:40:56.758 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../096-ceph-test_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:56.794 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-test (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:56.851 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T14:40:56.868 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T14:40:56.909 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:57.229 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T14:40:57.257 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../091-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T14:40:57.295 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T14:40:57.640 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T14:40:57.668 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../092-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T14:40:57.706 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T14:40:57.967 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package jq. 2024-09-16T14:40:57.984 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../093-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T14:40:58.034 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T14:40:58.228 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package socat. 2024-09-16T14:40:58.245 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../094-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T14:40:58.286 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T14:40:58.598 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T14:40:58.626 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T14:40:58.664 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T14:40:58.715 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T14:40:58.742 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../097-ceph-volume_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T14:40:58.781 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-volume (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:59.044 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-test. 2024-09-16T14:40:59.073 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../096-ceph-test_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:59.076 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T14:40:59.093 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../098-libcephfs-dev_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:40:59.102 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-test (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:59.135 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libcephfs-dev (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:40:59.555 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T14:40:59.582 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../099-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T14:40:59.621 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T14:40:59.952 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T14:40:59.968 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../100-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T14:41:00.009 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T14:41:00.430 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T14:41:00.457 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T14:41:00.496 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T14:41:00.833 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T14:41:00.860 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../102-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T14:41:00.899 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T14:41:01.139 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T14:41:01.168 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../097-ceph-volume_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T14:41:01.206 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-volume (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:01.227 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T14:41:01.255 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../103-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T14:41:01.294 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T14:41:01.526 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T14:41:01.543 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../098-libcephfs-dev_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:41:01.585 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libcephfs-dev (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:01.740 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T14:41:01.768 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../104-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T14:41:01.806 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T14:41:01.988 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T14:41:02.016 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../099-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T14:41:02.155 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T14:41:02.241 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T14:41:02.269 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T14:41:02.317 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T14:41:02.551 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T14:41:02.561 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T14:41:02.577 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T14:41:02.579 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../100-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T14:41:02.619 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T14:41:02.626 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T14:41:03.054 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T14:41:03.082 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T14:41:03.120 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T14:41:03.382 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package radosgw. 2024-09-16T14:41:03.410 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../107-radosgw_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:41:03.424 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T14:41:03.448 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking radosgw (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:03.452 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../102-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T14:41:03.490 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T14:41:03.869 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T14:41:03.897 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../103-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T14:41:03.943 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T14:41:04.185 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T14:41:04.212 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../108-rbd-fuse_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:41:04.251 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking rbd-fuse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:04.397 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T14:41:04.426 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../104-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T14:41:04.455 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T14:41:04.555 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T14:41:04.750 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T14:41:04.778 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T14:41:04.806 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T14:41:04.816 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T14:41:04.932 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T14:41:05.094 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T14:41:05.122 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T14:41:05.160 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T14:41:05.194 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T14:41:05.521 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T14:41:05.889 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package radosgw. 2024-09-16T14:41:05.918 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../107-radosgw_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:41:05.956 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking radosgw (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:05.962 INFO:teuthology.orchestra.run.smithi189.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T14:41:06.243 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up cephadm (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:06.709 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T14:41:06.737 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../108-rbd-fuse_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T14:41:06.775 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking rbd-fuse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:07.043 INFO:teuthology.orchestra.run.smithi189.stdout:Adding system user cephadm....done 2024-09-16T14:41:07.055 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T14:41:07.207 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T14:41:07.313 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T14:41:07.423 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T14:41:07.458 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T14:41:07.678 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T14:41:07.701 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T14:41:07.827 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T14:41:08.049 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T14:41:08.087 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T14:41:08.321 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T14:41:08.448 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T14:41:08.464 INFO:teuthology.orchestra.run.smithi026.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T14:41:08.541 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up cephadm (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:08.749 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T14:41:09.091 INFO:teuthology.orchestra.run.smithi026.stdout:Adding system user cephadm....done 2024-09-16T14:41:09.134 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T14:41:09.247 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T14:41:09.394 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T14:41:09.515 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T14:41:09.687 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T14:41:09.758 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T14:41:09.868 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T14:41:09.938 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T14:41:10.110 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T14:41:10.181 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T14:41:10.370 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T14:41:10.496 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T14:41:10.507 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-ceph-argparse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:10.749 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T14:41:10.806 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T14:41:10.875 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T14:41:11.135 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T14:41:11.166 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T14:41:11.261 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T14:41:11.417 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T14:41:11.620 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T14:41:11.718 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T14:41:11.730 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T14:41:11.978 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T14:41:11.989 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T14:41:12.240 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T14:41:12.245 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T14:41:12.505 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-ceph-argparse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:12.533 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T14:41:12.826 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T14:41:12.990 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T14:41:13.193 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T14:41:13.241 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T14:41:13.376 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T14:41:13.453 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T14:41:13.579 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T14:41:13.705 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T14:41:13.760 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T14:41:13.903 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T14:41:13.997 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T14:41:14.171 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T14:41:14.382 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T14:41:14.438 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T14:41:14.608 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T14:41:14.740 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T14:41:14.985 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T14:41:15.278 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T14:41:15.425 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T14:41:15.621 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T14:41:15.695 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T14:41:15.821 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T14:41:15.864 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T14:41:15.947 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T14:41:15.990 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-ceph-common (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:16.239 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T14:41:16.266 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T14:41:16.558 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T14:41:16.674 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T14:41:16.925 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T14:41:17.327 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T14:41:17.653 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T14:41:17.748 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T14:41:17.874 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T14:41:18.013 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T14:41:18.108 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T14:41:18.234 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T14:41:18.264 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T14:41:18.360 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T14:41:18.365 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-ceph-common (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:18.658 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T14:41:18.670 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T14:41:18.938 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T14:41:19.017 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T14:41:19.189 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T14:41:19.306 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2024-09-16T14:41:19.408 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T14:41:19.692 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T14:41:19.952 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T14:41:20.211 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T14:41:20.290 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T14:41:20.449 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T14:41:20.462 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T14:41:20.713 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T14:41:20.750 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T14:41:20.901 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T14:41:20.981 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T14:41:21.028 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T14:41:21.082 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T14:41:21.329 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T14:41:21.341 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T14:41:21.468 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T14:41:21.597 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T14:41:21.752 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T14:41:21.839 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T14:41:21.940 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2024-09-16T14:41:22.045 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T14:41:22.066 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T14:41:22.384 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T14:41:22.446 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T14:41:22.653 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T14:41:22.912 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T14:41:22.955 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T14:41:23.190 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T14:41:23.196 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T14:41:23.300 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T14:41:23.464 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T14:41:23.684 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T14:41:23.765 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T14:41:23.916 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T14:41:23.935 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T14:41:24.195 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T14:41:24.217 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T14:41:24.338 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T14:41:24.343 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T14:41:24.645 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T14:41:24.656 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T14:41:24.946 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T14:41:24.948 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T14:41:25.197 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T14:41:25.208 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T14:41:25.334 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T14:41:25.568 INFO:teuthology.orchestra.run.smithi026.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T14:41:25.601 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T14:41:25.833 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T14:41:25.869 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T14:41:25.984 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T14:41:26.154 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T14:41:26.402 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T14:41:26.539 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T14:41:26.670 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T14:41:26.815 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T14:41:26.929 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T14:41:27.030 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T14:41:27.133 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T14:41:27.260 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librados2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:27.348 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T14:41:27.385 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librgw2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:27.520 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:27.646 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libcephfs2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:27.683 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T14:41:27.789 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libradosstriper1 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:27.976 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T14:41:27.992 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T14:41:28.118 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T14:41:28.268 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librbd1 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:28.394 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:28.395 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T14:41:28.521 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-fuse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:28.679 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T14:41:28.734 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T14:41:28.734 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T14:41:28.980 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T14:41:29.248 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libcephfs-dev (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:29.365 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-rados (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:29.365 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T14:41:29.474 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T14:41:29.666 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T14:41:30.026 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T14:41:30.153 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up librados2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:30.279 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up librgw2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:30.405 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:30.532 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libcephfs2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:30.682 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libradosstriper1 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:30.695 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-rbd (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:30.821 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up rbd-fuse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:30.918 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T14:41:30.947 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-rgw (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:31.065 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cephfs (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:31.166 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-common (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:31.237 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up librbd1 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:31.363 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:31.489 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-fuse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:31.701 INFO:teuthology.orchestra.run.smithi189.stdout:Adding group ceph....done 2024-09-16T14:41:31.708 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T14:41:31.708 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T14:41:32.044 INFO:teuthology.orchestra.run.smithi189.stdout:Adding system user ceph....done 2024-09-16T14:41:32.170 INFO:teuthology.orchestra.run.smithi189.stdout:Setting system user ceph properties....done 2024-09-16T14:41:32.174 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libcephfs-dev (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:32.189 INFO:teuthology.orchestra.run.smithi189.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T14:41:32.300 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-rados (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:32.326 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T14:41:32.426 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T14:41:32.643 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T14:41:33.109 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-test (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:33.235 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up radosgw (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:33.697 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-rbd (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:33.708 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T14:41:33.708 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T14:41:33.848 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up rbd-fuse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:34.008 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-rgw (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:34.145 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-base (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:34.158 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-cephfs (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:34.310 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-common (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:34.459 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T14:41:34.853 INFO:teuthology.orchestra.run.smithi026.stdout:Adding group ceph....done 2024-09-16T14:41:34.897 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mds (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:35.128 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T14:41:35.128 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T14:41:35.246 INFO:teuthology.orchestra.run.smithi026.stdout:Adding system user ceph....done 2024-09-16T14:41:35.414 INFO:teuthology.orchestra.run.smithi026.stdout:Setting system user ceph properties....done 2024-09-16T14:41:35.428 INFO:teuthology.orchestra.run.smithi026.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T14:41:35.574 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T14:41:35.574 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:35.788 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T14:41:35.788 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T14:41:35.928 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T14:41:36.233 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-osd (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:36.419 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-test (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:36.513 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T14:41:36.514 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T14:41:36.571 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up radosgw (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:36.968 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:37.040 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T14:41:37.040 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T14:41:37.094 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:37.254 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mon (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:37.465 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T14:41:37.465 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T14:41:37.548 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-base (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:37.902 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T14:41:38.033 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:38.350 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mds (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:38.373 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:38.499 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:38.667 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-volume (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:38.774 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T14:41:38.774 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T14:41:39.070 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T14:41:39.244 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:39.455 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T14:41:39.455 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T14:41:39.953 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-osd (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:40.149 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T14:41:40.212 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T14:41:40.212 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T14:41:40.617 INFO:teuthology.orchestra.run.smithi189.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T14:41:40.623 DEBUG:teuthology.parallel:result is None 2024-09-16T14:41:40.680 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:40.781 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:40.932 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mon (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:41.242 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T14:41:41.242 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T14:41:41.725 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:41.859 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:41.969 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:42.103 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-volume (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T14:41:42.448 INFO:teuthology.orchestra.run.smithi026.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T14:41:43.590 INFO:teuthology.orchestra.run.smithi026.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T14:41:44.422 INFO:teuthology.orchestra.run.smithi026.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T14:41:44.427 DEBUG:teuthology.parallel:result is None 2024-09-16T14:41:44.428 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T14:41:44.576 DEBUG:teuthology.orchestra.run.smithi026:> dpkg-query -W -f '${Version}' ceph 2024-09-16T14:41:44.595 INFO:teuthology.orchestra.run.smithi026.stdout:18.2.4-845-ge4e227b3-1jammy 2024-09-16T14:41:44.596 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-845-ge4e227b3-1jammy 2024-09-16T14:41:44.596 INFO:teuthology.task.install:The correct ceph version 18.2.4-845-ge4e227b3-1jammy is installed. 2024-09-16T14:41:44.598 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T14:41:44.740 DEBUG:teuthology.orchestra.run.smithi189:> dpkg-query -W -f '${Version}' ceph 2024-09-16T14:41:44.759 INFO:teuthology.orchestra.run.smithi189.stdout:18.2.4-845-ge4e227b3-1jammy 2024-09-16T14:41:44.760 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-845-ge4e227b3-1jammy 2024-09-16T14:41:44.760 INFO:teuthology.task.install:The correct ceph version 18.2.4-845-ge4e227b3-1jammy is installed. 2024-09-16T14:41:44.762 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T14:41:44.762 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:41:44.762 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T14:41:44.777 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:41:44.777 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T14:41:44.814 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T14:41:44.814 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:41:44.814 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T14:41:44.834 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T14:41:44.889 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:41:44.889 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T14:41:44.906 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T14:41:44.960 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T14:41:44.960 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:41:44.961 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T14:41:44.974 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T14:41:45.028 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:41:45.029 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T14:41:45.041 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T14:41:45.097 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T14:41:45.097 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:41:45.098 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T14:41:45.111 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T14:41:45.165 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:41:45.165 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T14:41:45.179 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T14:41:45.234 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T14:41:45.319 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T14:41:45.319 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 777 /var/log/ceph 2024-09-16T14:41:45.321 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /var/log/ceph 2024-09-16T14:41:45.334 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T14:41:45.334 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T14:41:45.378 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T14:41:45.390 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T14:41:45.390 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T14:41:45.434 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T14:41:45.455 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T14:41:45.455 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-09-16T14:41:45.455 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906750', 'branch': 'reef', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_mon_workunits}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7906750', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'mon_bind_msgr2': False, 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 931, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 1390, 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8', 'sleep_before_teardown': 0, 'subset': '5319/100000', 'suite': 'rados', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6', 'targets': {'smithi026.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA1oD3ch7Tmpmckzw+2+omHRyYdWDrggEpWSp4Dkyu08tBkTetF1QZQ5gtkoLKdEiFPq4Xe/beT+YWOdRkHvdEo=', 'smithi189.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIbBhEHhS8/ndwvSqja/lY74FL75GrZLIgfi1VWNkoK8DoOltkKxT6lCeEekTgwvEvThk0demR/ewuaHBduzXrU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 20, 'thrash_delay': 1}}, {'workunit': {'clients': {'client.0': ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_22:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T14:41:45.455 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T14:41:45.490 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T14:41:45.496 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T14:41:45.538 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T14:41:45.550 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:41:45.550 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T14:41:45.597 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T14:41:45.598 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_1 2024-09-16T14:41:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T14:41:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T14:41:45.646 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 889 Links: 1 2024-09-16T14:41:45.646 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T14:41:45.646 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-09-16 14:36:23.857194973 +0000 2024-09-16T14:41:45.646 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-09-16 14:36:23.853195009 +0000 2024-09-16T14:41:45.646 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-09-16 14:36:23.853195009 +0000 2024-09-16T14:41:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: - 2024-09-16T14:41:45.647 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T14:41:45.700 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-09-16T14:41:45.700 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-09-16T14:41:45.700 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000254419 s, 2.0 MB/s 2024-09-16T14:41:45.701 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T14:41:45.751 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_2 2024-09-16T14:41:45.798 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T14:41:45.798 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T14:41:45.798 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 901 Links: 1 2024-09-16T14:41:45.798 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T14:41:45.798 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-09-16 14:36:24.601188327 +0000 2024-09-16T14:41:45.798 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-09-16 14:36:24.345190614 +0000 2024-09-16T14:41:45.798 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-09-16 14:36:24.345190614 +0000 2024-09-16T14:41:45.798 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: - 2024-09-16T14:41:45.799 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T14:41:45.852 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-09-16T14:41:45.852 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-09-16T14:41:45.852 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000290925 s, 1.8 MB/s 2024-09-16T14:41:45.853 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T14:41:45.903 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_3 2024-09-16T14:41:45.949 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T14:41:45.949 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T14:41:45.950 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 917 Links: 1 2024-09-16T14:41:45.950 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T14:41:45.950 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-09-16 14:36:25.081184040 +0000 2024-09-16T14:41:45.950 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-09-16 14:36:24.813186433 +0000 2024-09-16T14:41:45.950 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-09-16 14:36:24.813186433 +0000 2024-09-16T14:41:45.950 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: - 2024-09-16T14:41:45.950 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T14:41:46.004 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-09-16T14:41:46.004 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-09-16T14:41:46.004 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000294931 s, 1.7 MB/s 2024-09-16T14:41:46.005 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T14:41:46.054 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_4 2024-09-16T14:41:46.101 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T14:41:46.102 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T14:41:46.102 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 934 Links: 1 2024-09-16T14:41:46.102 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T14:41:46.102 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-09-16 14:36:25.661178860 +0000 2024-09-16T14:41:46.102 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-09-16 14:36:25.417181039 +0000 2024-09-16T14:41:46.102 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-09-16 14:36:25.417181039 +0000 2024-09-16T14:41:46.102 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: - 2024-09-16T14:41:46.102 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T14:41:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-09-16T14:41:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-09-16T14:41:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000231236 s, 2.2 MB/s 2024-09-16T14:41:46.157 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T14:41:46.207 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T14:41:46.207 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:41:46.207 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T14:41:46.213 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T14:41:46.213 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-09-16T14:41:46.262 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T14:41:46.262 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T14:41:46.262 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 885 Links: 1 2024-09-16T14:41:46.262 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T14:41:46.262 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-09-16 14:36:22.039527972 +0000 2024-09-16T14:41:46.263 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-09-16 14:36:21.771530366 +0000 2024-09-16T14:41:46.263 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-09-16 14:36:21.771530366 +0000 2024-09-16T14:41:46.263 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2024-09-16T14:41:46.263 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T14:41:46.316 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-09-16T14:41:46.316 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-09-16T14:41:46.317 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000459919 s, 1.1 MB/s 2024-09-16T14:41:46.318 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T14:41:46.367 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-09-16T14:41:46.413 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T14:41:46.413 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T14:41:46.414 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 899 Links: 1 2024-09-16T14:41:46.414 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T14:41:46.414 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-09-16 14:36:22.523523648 +0000 2024-09-16T14:41:46.414 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-09-16 14:36:22.263525972 +0000 2024-09-16T14:41:46.414 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-09-16 14:36:22.263525972 +0000 2024-09-16T14:41:46.414 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2024-09-16T14:41:46.414 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T14:41:46.468 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-09-16T14:41:46.468 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-09-16T14:41:46.468 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000454397 s, 1.1 MB/s 2024-09-16T14:41:46.469 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T14:41:46.519 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-09-16T14:41:46.566 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T14:41:46.566 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T14:41:46.566 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 918 Links: 1 2024-09-16T14:41:46.566 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T14:41:46.566 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-09-16 14:36:22.743521683 +0000 2024-09-16T14:41:46.566 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-09-16 14:36:22.739521720 +0000 2024-09-16T14:41:46.566 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-09-16 14:36:22.739521720 +0000 2024-09-16T14:41:46.566 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2024-09-16T14:41:46.566 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T14:41:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-09-16T14:41:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-09-16T14:41:46.619 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000328453 s, 1.6 MB/s 2024-09-16T14:41:46.620 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T14:41:46.667 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-09-16T14:41:46.714 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T14:41:46.714 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T14:41:46.714 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 934 Links: 1 2024-09-16T14:41:46.714 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T14:41:46.714 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-09-16 14:36:23.267517003 +0000 2024-09-16T14:41:46.714 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-09-16 14:36:23.263517039 +0000 2024-09-16T14:41:46.715 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-09-16 14:36:23.263517039 +0000 2024-09-16T14:41:46.715 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2024-09-16T14:41:46.715 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T14:41:46.768 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-09-16T14:41:46.768 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-09-16T14:41:46.768 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000394161 s, 1.3 MB/s 2024-09-16T14:41:46.769 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T14:41:46.819 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T14:41:46.819 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi189.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T14:41:46.819 INFO:tasks.ceph:Generating config... 2024-09-16T14:41:46.820 INFO:tasks.ceph:[client] debug monc = 20 2024-09-16T14:41:46.821 INFO:tasks.ceph:[client] debug ms = 1 2024-09-16T14:41:46.821 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T14:41:46.821 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-16T14:41:46.821 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-16T14:41:46.821 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T14:41:46.821 INFO:tasks.ceph:[global] ms type = async 2024-09-16T14:41:46.821 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T14:41:46.821 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T14:41:46.821 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T14:41:46.821 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T14:41:46.821 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T14:41:46.821 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-16T14:41:46.822 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-16T14:41:46.822 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-16T14:41:46.822 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-16T14:41:46.822 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-16T14:41:46.822 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-16T14:41:46.822 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T14:41:46.822 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-16T14:41:46.822 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T14:41:46.822 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-16T14:41:46.822 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-16T14:41:46.822 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T14:41:46.822 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T14:41:46.822 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T14:41:46.822 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T14:41:46.823 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T14:41:46.823 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T14:41:46.823 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T14:41:46.823 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T14:41:46.823 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T14:41:46.823 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T14:41:46.823 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T14:41:46.823 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T14:41:46.823 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T14:41:46.823 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T14:41:46.823 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T14:41:46.823 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T14:41:46.824 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T14:41:46.824 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T14:41:46.824 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T14:41:46.824 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T14:41:46.860 INFO:teuthology.orchestra.run.smithi026.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T14:41:46.883 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T14:41:46.961 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T14:41:47.017 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.26:6789]'), ('mon.c', '[v1:172.21.15.26:6790]'), ('mon.b', '[v1:172.21.15.189:6789]')] 2024-09-16T14:41:47.018 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.26:6789],[v1:172.21.15.26:6790],[v1:172.21.15.189:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-16T14:41:47.019 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:41:47.019 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T14:41:47.066 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.26:6789]' --addv c '[v1:172.21.15.26:6790]' --addv b '[v1:172.21.15.189:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T14:41:47.153 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T14:41:47.153 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: generated fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:41:47.153 INFO:teuthology.orchestra.run.smithi026.stdout:setting min_mon_release = pacific 2024-09-16T14:41:47.154 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 0 2024-09-16T14:41:47.154 INFO:teuthology.orchestra.run.smithi026.stdout:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:41:47.154 INFO:teuthology.orchestra.run.smithi026.stdout:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:41:47.154 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:41:47.154 INFO:teuthology.orchestra.run.smithi026.stdout:min_mon_release 16 (pacific) 2024-09-16T14:41:47.154 INFO:teuthology.orchestra.run.smithi026.stdout:election_strategy: 3 2024-09-16T14:41:47.154 INFO:teuthology.orchestra.run.smithi026.stdout:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:41:47.154 INFO:teuthology.orchestra.run.smithi026.stdout:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:41:47.154 INFO:teuthology.orchestra.run.smithi026.stdout:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:41:47.154 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T14:41:47.155 DEBUG:teuthology.orchestra.run.smithi026:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T14:41:47.161 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID febeb2df-62e5-4150-ac3f-7a627f391606... 2024-09-16T14:41:47.161 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T14:41:47.206 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T14:41:47.230 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2024-09-16T14:41:47.230 INFO:teuthology.orchestra.run.smithi026.stdout: chdir = "" 2024-09-16T14:41:47.230 INFO:teuthology.orchestra.run.smithi026.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T14:41:47.231 INFO:teuthology.orchestra.run.smithi026.stdout: auth supported = cephx 2024-09-16T14:41:47.231 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.231 INFO:teuthology.orchestra.run.smithi026.stdout: filestore xattr use omap = true 2024-09-16T14:41:47.231 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.231 INFO:teuthology.orchestra.run.smithi026.stdout: mon clock drift allowed = 1.000 2024-09-16T14:41:47.231 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.231 INFO:teuthology.orchestra.run.smithi026.stdout: osd crush chooseleaf type = 0 2024-09-16T14:41:47.231 INFO:teuthology.orchestra.run.smithi026.stdout: auth debug = true 2024-09-16T14:41:47.231 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.231 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on old message = true 2024-09-16T14:41:47.231 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on bug = true 2024-09-16T14:41:47.231 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.231 INFO:teuthology.orchestra.run.smithi026.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T14:41:47.232 INFO:teuthology.orchestra.run.smithi026.stdout: mon pg warn max object skew = 0 2024-09-16T14:41:47.232 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.232 INFO:teuthology.orchestra.run.smithi026.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T14:41:47.232 INFO:teuthology.orchestra.run.smithi026.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T14:41:47.232 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.232 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default size = 2 2024-09-16T14:41:47.232 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.232 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow primary affinity = true 2024-09-16T14:41:47.232 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow pg remap = true 2024-09-16T14:41:47.232 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on legacy crush tunables = false 2024-09-16T14:41:47.232 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on crush straw calc version zero = false 2024-09-16T14:41:47.232 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on no sortbitwise = false 2024-09-16T14:41:47.233 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on osd down out interval zero = false 2024-09-16T14:41:47.233 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on too few osds = false 2024-09-16T14:41:47.233 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T14:41:47.233 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T14:41:47.233 INFO:teuthology.orchestra.run.smithi026.stdout: mon_allow_pool_size_one = true 2024-09-16T14:41:47.233 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.233 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T14:41:47.233 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.233 INFO:teuthology.orchestra.run.smithi026.stdout: osd default data pool replay window = 5 2024-09-16T14:41:47.233 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.233 INFO:teuthology.orchestra.run.smithi026.stdout: mon allow pool delete = true 2024-09-16T14:41:47.233 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.233 INFO:teuthology.orchestra.run.smithi026.stdout: mon cluster log file level = debug 2024-09-16T14:41:47.234 INFO:teuthology.orchestra.run.smithi026.stdout: debug asserts on shutdown = true 2024-09-16T14:41:47.234 INFO:teuthology.orchestra.run.smithi026.stdout: mon health detail to clog = false 2024-09-16T14:41:47.234 INFO:teuthology.orchestra.run.smithi026.stdout: mon host = "[v1:172.21.15.26:6789],[v1:172.21.15.26:6790],[v1:172.21.15.189:6789]" 2024-09-16T14:41:47.234 INFO:teuthology.orchestra.run.smithi026.stdout: mon client directed command retry = 5 2024-09-16T14:41:47.234 INFO:teuthology.orchestra.run.smithi026.stdout: mon election default strategy = 3 2024-09-16T14:41:47.234 INFO:teuthology.orchestra.run.smithi026.stdout: ms bind msgr2 = False 2024-09-16T14:41:47.234 INFO:teuthology.orchestra.run.smithi026.stdout: ms inject socket failures = 5000 2024-09-16T14:41:47.234 INFO:teuthology.orchestra.run.smithi026.stdout: ms type = async 2024-09-16T14:41:47.234 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:41:47.234 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.234 INFO:teuthology.orchestra.run.smithi026.stdout:[osd] 2024-09-16T14:41:47.234 INFO:teuthology.orchestra.run.smithi026.stdout: osd journal size = 100 2024-09-16T14:41:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub load threshold = 5.0 2024-09-16T14:41:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub max interval = 600 2024-09-16T14:41:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: osd mclock profile = high_recovery_ops 2024-09-16T14:41:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: osd recover clone overlap = true 2024-09-16T14:41:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: osd recovery max chunk = 1048576 2024-09-16T14:41:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug shutdown = true 2024-09-16T14:41:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug op order = true 2024-09-16T14:41:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify stray on activate = true 2024-09-16T14:41:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: osd open classes on start = true 2024-09-16T14:41:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug pg log writeout = true 2024-09-16T14:41:47.236 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.236 INFO:teuthology.orchestra.run.smithi026.stdout: osd deep scrub update digest min age = 30 2024-09-16T14:41:47.236 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.236 INFO:teuthology.orchestra.run.smithi026.stdout: osd map max advance = 10 2024-09-16T14:41:47.236 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.236 INFO:teuthology.orchestra.run.smithi026.stdout: journal zero on create = true 2024-09-16T14:41:47.236 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.236 INFO:teuthology.orchestra.run.smithi026.stdout: filestore ondisk finisher threads = 3 2024-09-16T14:41:47.236 INFO:teuthology.orchestra.run.smithi026.stdout: filestore apply finisher threads = 3 2024-09-16T14:41:47.236 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.236 INFO:teuthology.orchestra.run.smithi026.stdout: bdev debug aio = true 2024-09-16T14:41:47.236 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug misdirected ops = true 2024-09-16T14:41:47.237 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore block size = 96636764160 2024-09-16T14:41:47.237 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore compression algorithm = snappy 2024-09-16T14:41:47.237 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore compression mode = aggressive 2024-09-16T14:41:47.237 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore fsck on mount = True 2024-09-16T14:41:47.237 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore zero block detection = True 2024-09-16T14:41:47.237 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluefs = 20 2024-09-16T14:41:47.237 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluestore = 20 2024-09-16T14:41:47.237 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-09-16T14:41:47.237 INFO:teuthology.orchestra.run.smithi026.stdout: debug osd = 20 2024-09-16T14:41:47.237 INFO:teuthology.orchestra.run.smithi026.stdout: debug rocksdb = 10 2024-09-16T14:41:47.237 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T14:41:47.237 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd full ratio = 0.9 2024-09-16T14:41:47.237 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd nearfull ratio = 0.8 2024-09-16T14:41:47.238 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify cached snaps = True 2024-09-16T14:41:47.238 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify missing on start = True 2024-09-16T14:41:47.238 INFO:teuthology.orchestra.run.smithi026.stdout: osd failsafe full ratio = 0.95 2024-09-16T14:41:47.238 INFO:teuthology.orchestra.run.smithi026.stdout: osd mclock override recovery settings = True 2024-09-16T14:41:47.238 INFO:teuthology.orchestra.run.smithi026.stdout: osd objectstore = bluestore 2024-09-16T14:41:47.238 INFO:teuthology.orchestra.run.smithi026.stdout: osd op queue = debug_random 2024-09-16T14:41:47.238 INFO:teuthology.orchestra.run.smithi026.stdout: osd op queue cut off = debug_random 2024-09-16T14:41:47.238 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.238 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr] 2024-09-16T14:41:47.238 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-09-16T14:41:47.238 INFO:teuthology.orchestra.run.smithi026.stdout: debug mgr = 20 2024-09-16T14:41:47.238 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2024-09-16T14:41:47.238 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2024-09-16T14:41:47.239 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2024-09-16T14:41:47.239 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2024-09-16T14:41:47.239 INFO:teuthology.orchestra.run.smithi026.stdout: mgr/telemetry/nag = false 2024-09-16T14:41:47.239 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.239 INFO:teuthology.orchestra.run.smithi026.stdout:[mon] 2024-09-16T14:41:47.239 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-09-16T14:41:47.239 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2024-09-16T14:41:47.239 INFO:teuthology.orchestra.run.smithi026.stdout: debug paxos = 20 2024-09-16T14:41:47.239 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2024-09-16T14:41:47.239 INFO:teuthology.orchestra.run.smithi026.stdout: mon data avail warn = 5 2024-09-16T14:41:47.239 INFO:teuthology.orchestra.run.smithi026.stdout: mon mgr mkfs grace = 240 2024-09-16T14:41:47.239 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2024-09-16T14:41:47.239 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd reporter subtree level = osd 2024-09-16T14:41:47.240 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd prime pg temp = true 2024-09-16T14:41:47.240 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2024-09-16T14:41:47.240 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.240 INFO:teuthology.orchestra.run.smithi026.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T14:41:47.240 INFO:teuthology.orchestra.run.smithi026.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T14:41:47.240 INFO:teuthology.orchestra.run.smithi026.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T14:41:47.240 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.240 INFO:teuthology.orchestra.run.smithi026.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T14:41:47.240 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T14:41:47.240 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T14:41:47.240 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.240 INFO:teuthology.orchestra.run.smithi026.stdout: # 1m isn't quite enough 2024-09-16T14:41:47.241 INFO:teuthology.orchestra.run.smithi026.stdout: mon_down_mkfs_grace = 2m 2024-09-16T14:41:47.241 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.241 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_filestore_osds = false 2024-09-16T14:41:47.241 INFO:teuthology.orchestra.run.smithi026.stdout: mon min osdmap epochs = 25 2024-09-16T14:41:47.241 INFO:teuthology.orchestra.run.smithi026.stdout: mon osdmap full prune interval = 2 2024-09-16T14:41:47.241 INFO:teuthology.orchestra.run.smithi026.stdout: mon osdmap full prune min = 15 2024-09-16T14:41:47.241 INFO:teuthology.orchestra.run.smithi026.stdout: mon osdmap full prune txsize = 2 2024-09-16T14:41:47.241 INFO:teuthology.orchestra.run.smithi026.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T14:41:47.241 INFO:teuthology.orchestra.run.smithi026.stdout: mon scrub inject missing keys = 0.05 2024-09-16T14:41:47.241 INFO:teuthology.orchestra.run.smithi026.stdout: mon scrub interval = 300 2024-09-16T14:41:47.241 INFO:teuthology.orchestra.run.smithi026.stdout: paxos service trim min = 5 2024-09-16T14:41:47.241 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:41:47.241 INFO:teuthology.orchestra.run.smithi026.stdout:[client] 2024-09-16T14:41:47.242 INFO:teuthology.orchestra.run.smithi026.stdout: rgw cache enabled = true 2024-09-16T14:41:47.242 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable ops log = true 2024-09-16T14:41:47.242 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable usage log = true 2024-09-16T14:41:47.242 INFO:teuthology.orchestra.run.smithi026.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T14:41:47.242 INFO:teuthology.orchestra.run.smithi026.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T14:41:47.242 INFO:teuthology.orchestra.run.smithi026.stdout: debug monc = 20 2024-09-16T14:41:47.242 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-09-16T14:41:47.242 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.a] 2024-09-16T14:41:47.242 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.c] 2024-09-16T14:41:47.242 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.b] 2024-09-16T14:41:47.243 INFO:teuthology.orchestra.run.smithi189.stdout:[global] 2024-09-16T14:41:47.243 INFO:teuthology.orchestra.run.smithi189.stdout: chdir = "" 2024-09-16T14:41:47.243 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T14:41:47.243 INFO:teuthology.orchestra.run.smithi189.stdout: auth supported = cephx 2024-09-16T14:41:47.243 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.243 INFO:teuthology.orchestra.run.smithi189.stdout: filestore xattr use omap = true 2024-09-16T14:41:47.243 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.243 INFO:teuthology.orchestra.run.smithi189.stdout: mon clock drift allowed = 1.000 2024-09-16T14:41:47.243 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.243 INFO:teuthology.orchestra.run.smithi189.stdout: osd crush chooseleaf type = 0 2024-09-16T14:41:47.244 INFO:teuthology.orchestra.run.smithi189.stdout: auth debug = true 2024-09-16T14:41:47.244 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.244 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on old message = true 2024-09-16T14:41:47.244 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on bug = true 2024-09-16T14:41:47.244 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.244 INFO:teuthology.orchestra.run.smithi189.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T14:41:47.244 INFO:teuthology.orchestra.run.smithi189.stdout: mon pg warn max object skew = 0 2024-09-16T14:41:47.244 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.244 INFO:teuthology.orchestra.run.smithi189.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T14:41:47.244 INFO:teuthology.orchestra.run.smithi189.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T14:41:47.244 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.244 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default size = 2 2024-09-16T14:41:47.244 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.245 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow primary affinity = true 2024-09-16T14:41:47.245 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow pg remap = true 2024-09-16T14:41:47.245 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on legacy crush tunables = false 2024-09-16T14:41:47.245 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on crush straw calc version zero = false 2024-09-16T14:41:47.245 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on no sortbitwise = false 2024-09-16T14:41:47.245 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on osd down out interval zero = false 2024-09-16T14:41:47.245 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on too few osds = false 2024-09-16T14:41:47.245 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T14:41:47.245 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T14:41:47.245 INFO:teuthology.orchestra.run.smithi189.stdout: mon_allow_pool_size_one = true 2024-09-16T14:41:47.245 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.245 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T14:41:47.246 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.246 INFO:teuthology.orchestra.run.smithi189.stdout: osd default data pool replay window = 5 2024-09-16T14:41:47.246 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.246 INFO:teuthology.orchestra.run.smithi189.stdout: mon allow pool delete = true 2024-09-16T14:41:47.246 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.246 INFO:teuthology.orchestra.run.smithi189.stdout: mon cluster log file level = debug 2024-09-16T14:41:47.246 INFO:teuthology.orchestra.run.smithi189.stdout: debug asserts on shutdown = true 2024-09-16T14:41:47.246 INFO:teuthology.orchestra.run.smithi189.stdout: mon health detail to clog = false 2024-09-16T14:41:47.246 INFO:teuthology.orchestra.run.smithi189.stdout: mon host = "[v1:172.21.15.26:6789],[v1:172.21.15.26:6790],[v1:172.21.15.189:6789]" 2024-09-16T14:41:47.246 INFO:teuthology.orchestra.run.smithi189.stdout: mon client directed command retry = 5 2024-09-16T14:41:47.246 INFO:teuthology.orchestra.run.smithi189.stdout: mon election default strategy = 3 2024-09-16T14:41:47.246 INFO:teuthology.orchestra.run.smithi189.stdout: ms bind msgr2 = False 2024-09-16T14:41:47.247 INFO:teuthology.orchestra.run.smithi189.stdout: ms inject socket failures = 5000 2024-09-16T14:41:47.247 INFO:teuthology.orchestra.run.smithi189.stdout: ms type = async 2024-09-16T14:41:47.247 INFO:teuthology.orchestra.run.smithi189.stdout: fsid = febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:41:47.247 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.247 INFO:teuthology.orchestra.run.smithi189.stdout:[osd] 2024-09-16T14:41:47.247 INFO:teuthology.orchestra.run.smithi189.stdout: osd journal size = 100 2024-09-16T14:41:47.247 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.247 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub load threshold = 5.0 2024-09-16T14:41:47.247 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub max interval = 600 2024-09-16T14:41:47.247 INFO:teuthology.orchestra.run.smithi189.stdout: osd mclock profile = high_recovery_ops 2024-09-16T14:41:47.247 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.247 INFO:teuthology.orchestra.run.smithi189.stdout: osd recover clone overlap = true 2024-09-16T14:41:47.247 INFO:teuthology.orchestra.run.smithi189.stdout: osd recovery max chunk = 1048576 2024-09-16T14:41:47.248 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.248 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug shutdown = true 2024-09-16T14:41:47.248 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug op order = true 2024-09-16T14:41:47.248 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify stray on activate = true 2024-09-16T14:41:47.248 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.248 INFO:teuthology.orchestra.run.smithi189.stdout: osd open classes on start = true 2024-09-16T14:41:47.248 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug pg log writeout = true 2024-09-16T14:41:47.248 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.248 INFO:teuthology.orchestra.run.smithi189.stdout: osd deep scrub update digest min age = 30 2024-09-16T14:41:47.248 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.248 INFO:teuthology.orchestra.run.smithi189.stdout: osd map max advance = 10 2024-09-16T14:41:47.248 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.248 INFO:teuthology.orchestra.run.smithi189.stdout: journal zero on create = true 2024-09-16T14:41:47.249 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.249 INFO:teuthology.orchestra.run.smithi189.stdout: filestore ondisk finisher threads = 3 2024-09-16T14:41:47.249 INFO:teuthology.orchestra.run.smithi189.stdout: filestore apply finisher threads = 3 2024-09-16T14:41:47.249 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.249 INFO:teuthology.orchestra.run.smithi189.stdout: bdev debug aio = true 2024-09-16T14:41:47.249 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug misdirected ops = true 2024-09-16T14:41:47.249 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore block size = 96636764160 2024-09-16T14:41:47.249 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore compression algorithm = snappy 2024-09-16T14:41:47.249 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore compression mode = aggressive 2024-09-16T14:41:47.249 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore fsck on mount = True 2024-09-16T14:41:47.249 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore zero block detection = True 2024-09-16T14:41:47.249 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluefs = 20 2024-09-16T14:41:47.249 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluestore = 20 2024-09-16T14:41:47.250 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-16T14:41:47.250 INFO:teuthology.orchestra.run.smithi189.stdout: debug osd = 20 2024-09-16T14:41:47.250 INFO:teuthology.orchestra.run.smithi189.stdout: debug rocksdb = 10 2024-09-16T14:41:47.250 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T14:41:47.250 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd full ratio = 0.9 2024-09-16T14:41:47.250 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd nearfull ratio = 0.8 2024-09-16T14:41:47.250 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify cached snaps = True 2024-09-16T14:41:47.250 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify missing on start = True 2024-09-16T14:41:47.250 INFO:teuthology.orchestra.run.smithi189.stdout: osd failsafe full ratio = 0.95 2024-09-16T14:41:47.250 INFO:teuthology.orchestra.run.smithi189.stdout: osd mclock override recovery settings = True 2024-09-16T14:41:47.250 INFO:teuthology.orchestra.run.smithi189.stdout: osd objectstore = bluestore 2024-09-16T14:41:47.250 INFO:teuthology.orchestra.run.smithi189.stdout: osd op queue = debug_random 2024-09-16T14:41:47.251 INFO:teuthology.orchestra.run.smithi189.stdout: osd op queue cut off = debug_random 2024-09-16T14:41:47.251 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.251 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr] 2024-09-16T14:41:47.251 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-16T14:41:47.251 INFO:teuthology.orchestra.run.smithi189.stdout: debug mgr = 20 2024-09-16T14:41:47.251 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-09-16T14:41:47.251 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-09-16T14:41:47.251 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-09-16T14:41:47.251 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-09-16T14:41:47.251 INFO:teuthology.orchestra.run.smithi189.stdout: mgr/telemetry/nag = false 2024-09-16T14:41:47.251 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.251 INFO:teuthology.orchestra.run.smithi189.stdout:[mon] 2024-09-16T14:41:47.251 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-16T14:41:47.252 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-09-16T14:41:47.252 INFO:teuthology.orchestra.run.smithi189.stdout: debug paxos = 20 2024-09-16T14:41:47.252 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-09-16T14:41:47.252 INFO:teuthology.orchestra.run.smithi189.stdout: mon data avail warn = 5 2024-09-16T14:41:47.252 INFO:teuthology.orchestra.run.smithi189.stdout: mon mgr mkfs grace = 240 2024-09-16T14:41:47.252 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-09-16T14:41:47.252 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd reporter subtree level = osd 2024-09-16T14:41:47.252 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd prime pg temp = true 2024-09-16T14:41:47.252 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-09-16T14:41:47.252 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.252 INFO:teuthology.orchestra.run.smithi189.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T14:41:47.252 INFO:teuthology.orchestra.run.smithi189.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T14:41:47.253 INFO:teuthology.orchestra.run.smithi189.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T14:41:47.253 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.253 INFO:teuthology.orchestra.run.smithi189.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T14:41:47.253 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T14:41:47.253 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T14:41:47.253 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.253 INFO:teuthology.orchestra.run.smithi189.stdout: # 1m isn't quite enough 2024-09-16T14:41:47.253 INFO:teuthology.orchestra.run.smithi189.stdout: mon_down_mkfs_grace = 2m 2024-09-16T14:41:47.253 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.253 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_filestore_osds = false 2024-09-16T14:41:47.253 INFO:teuthology.orchestra.run.smithi189.stdout: mon min osdmap epochs = 25 2024-09-16T14:41:47.253 INFO:teuthology.orchestra.run.smithi189.stdout: mon osdmap full prune interval = 2 2024-09-16T14:41:47.253 INFO:teuthology.orchestra.run.smithi189.stdout: mon osdmap full prune min = 15 2024-09-16T14:41:47.254 INFO:teuthology.orchestra.run.smithi189.stdout: mon osdmap full prune txsize = 2 2024-09-16T14:41:47.254 INFO:teuthology.orchestra.run.smithi189.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T14:41:47.254 INFO:teuthology.orchestra.run.smithi189.stdout: mon scrub inject missing keys = 0.05 2024-09-16T14:41:47.254 INFO:teuthology.orchestra.run.smithi189.stdout: mon scrub interval = 300 2024-09-16T14:41:47.254 INFO:teuthology.orchestra.run.smithi189.stdout: paxos service trim min = 5 2024-09-16T14:41:47.254 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T14:41:47.254 INFO:teuthology.orchestra.run.smithi189.stdout:[client] 2024-09-16T14:41:47.254 INFO:teuthology.orchestra.run.smithi189.stdout: rgw cache enabled = true 2024-09-16T14:41:47.254 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable ops log = true 2024-09-16T14:41:47.254 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable usage log = true 2024-09-16T14:41:47.254 INFO:teuthology.orchestra.run.smithi189.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T14:41:47.254 INFO:teuthology.orchestra.run.smithi189.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T14:41:47.255 INFO:teuthology.orchestra.run.smithi189.stdout: debug monc = 20 2024-09-16T14:41:47.255 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-16T14:41:47.255 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.a] 2024-09-16T14:41:47.255 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.c] 2024-09-16T14:41:47.255 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.b] 2024-09-16T14:41:47.256 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T14:41:47.256 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T14:41:47.346 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T14:41:47.347 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:41:47.347 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T14:41:47.353 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:41:47.353 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T14:41:47.405 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T14:41:47.406 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:41:47.406 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T14:41:47.406 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T14:41:47.470 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:41:47.470 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T14:41:47.517 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T14:41:47.518 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:41:47.518 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T14:41:47.518 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T14:41:47.541 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:41:47.541 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T14:41:47.590 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T14:41:47.590 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T14:41:47.590 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T14:41:47.675 INFO:teuthology.orchestra.run.smithi189.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T14:41:47.689 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T14:41:47.689 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T14:41:47.689 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T14:41:47.767 INFO:teuthology.orchestra.run.smithi189.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T14:41:47.778 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T14:41:47.778 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi189.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T14:41:47.778 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T14:41:47.793 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T14:41:47.793 INFO:tasks.ceph:role: osd.0 2024-09-16T14:41:47.793 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T14:41:47.793 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T14:41:47.850 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T14:41:47.850 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T14:41:47.850 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T14:41:47.850 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T14:41:47.850 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T14:41:47.851 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-09-16T14:41:47.851 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T14:41:47.851 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T14:41:47.851 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T14:41:47.851 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T14:41:47.874 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-09-16T14:41:47.876 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-09-16T14:41:47.877 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T14:41:47.941 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T14:41:47.995 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T14:41:47.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T14:41:47.996 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T14:41:48.054 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T14:41:48.054 INFO:tasks.ceph:role: osd.1 2024-09-16T14:41:48.054 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T14:41:48.054 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T14:41:48.110 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T14:41:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T14:41:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T14:41:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T14:41:48.111 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T14:41:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-09-16T14:41:48.111 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T14:41:48.111 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T14:41:48.111 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T14:41:48.111 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T14:41:48.134 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-09-16T14:41:48.136 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-09-16T14:41:48.136 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T14:41:48.193 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T14:41:48.247 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T14:41:48.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T14:41:48.248 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T14:41:48.307 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T14:41:48.307 INFO:tasks.ceph:role: osd.2 2024-09-16T14:41:48.307 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T14:41:48.307 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T14:41:48.361 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T14:41:48.361 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T14:41:48.361 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T14:41:48.361 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T14:41:48.361 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T14:41:48.361 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-09-16T14:41:48.361 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T14:41:48.361 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T14:41:48.361 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T14:41:48.361 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T14:41:48.385 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-09-16T14:41:48.386 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-09-16T14:41:48.386 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T14:41:48.446 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T14:41:48.499 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T14:41:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T14:41:48.500 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T14:41:48.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:48.608+0000 7f44148f07c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T14:41:48.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:48.608+0000 7f44148f07c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T14:41:48.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:48.608+0000 7f44148f07c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T14:41:48.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:48.608+0000 7f44148f07c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T14:41:48.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:48.608+0000 7f44148f07c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T14:41:48.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:48.608+0000 7f44148f07c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T14:41:51.594 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T14:41:51.609 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T14:41:51.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:51.688+0000 7f1ac1a917c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T14:41:51.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:51.688+0000 7f1ac1a917c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T14:41:51.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:51.692+0000 7f1ac1a917c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T14:41:51.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:51.692+0000 7f1ac1a917c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T14:41:51.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:51.692+0000 7f1ac1a917c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T14:41:51.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:51.692+0000 7f1ac1a917c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T14:41:54.695 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T14:41:54.749 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T14:41:54.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:54.832+0000 7f86fb7c97c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T14:41:54.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:54.832+0000 7f86fb7c97c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T14:41:54.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:54.832+0000 7f86fb7c97c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T14:41:54.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:54.832+0000 7f86fb7c97c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T14:41:54.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:54.832+0000 7f86fb7c97c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T14:41:54.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:41:54.832+0000 7f86fb7c97c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T14:41:57.829 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T14:41:57.885 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T14:41:57.900 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T14:41:57.901 INFO:tasks.ceph:role: osd.3 2024-09-16T14:41:57.901 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T14:41:57.901 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T14:41:57.966 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T14:41:57.966 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T14:41:57.966 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T14:41:57.966 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T14:41:57.967 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T14:41:57.967 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-09-16T14:41:57.967 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T14:41:57.967 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T14:41:57.967 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T14:41:57.967 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T14:41:58.420 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-09-16T14:41:58.422 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-09-16T14:41:58.422 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-16T14:41:58.523 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T14:41:58.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T14:41:58.534 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T14:41:58.535 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T14:41:58.588 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T14:41:58.588 INFO:tasks.ceph:role: osd.4 2024-09-16T14:41:58.588 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T14:41:58.588 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T14:41:58.653 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T14:41:58.653 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T14:41:58.653 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T14:41:58.653 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T14:41:58.653 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T14:41:58.653 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-09-16T14:41:58.653 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T14:41:58.654 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T14:41:58.654 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T14:41:58.654 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T14:41:59.120 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-09-16T14:41:59.122 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-09-16T14:41:59.123 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-16T14:41:59.277 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T14:41:59.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T14:41:59.289 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T14:41:59.290 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T14:41:59.344 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T14:41:59.344 INFO:tasks.ceph:role: osd.5 2024-09-16T14:41:59.344 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T14:41:59.345 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T14:41:59.404 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T14:41:59.404 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T14:41:59.405 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T14:41:59.405 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T14:41:59.405 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T14:41:59.405 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-09-16T14:41:59.405 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T14:41:59.405 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T14:41:59.405 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T14:41:59.405 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T14:41:59.868 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-09-16T14:41:59.870 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-09-16T14:41:59.870 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-16T14:42:00.023 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T14:42:00.033 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T14:42:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T14:42:00.034 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T14:42:00.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:00.141+0000 7f1e05f6b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T14:42:00.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:00.141+0000 7f1e05f6b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T14:42:00.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:00.141+0000 7f1e05f6b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T14:42:00.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:00.141+0000 7f1e05f6b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T14:42:00.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:00.141+0000 7f1e05f6b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T14:42:00.145 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:00.141+0000 7f1e05f6b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T14:42:03.122 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T14:42:03.137 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T14:42:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:03.217+0000 7fc2b6fe07c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T14:42:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:03.217+0000 7fc2b6fe07c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T14:42:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:03.217+0000 7fc2b6fe07c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T14:42:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:03.217+0000 7fc2b6fe07c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T14:42:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:03.217+0000 7fc2b6fe07c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T14:42:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:03.217+0000 7fc2b6fe07c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T14:42:06.199 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T14:42:06.259 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T14:42:06.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:06.337+0000 7f175fec27c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T14:42:06.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:06.337+0000 7f175fec27c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T14:42:06.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:06.337+0000 7f175fec27c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T14:42:06.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:06.337+0000 7f175fec27c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T14:42:06.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:06.337+0000 7f175fec27c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T14:42:06.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:06.341+0000 7f175fec27c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T14:42:09.317 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T14:42:09.378 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T14:42:09.378 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:42:09.378 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T14:42:09.393 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:42:09.394 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T14:42:09.450 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:42:09.450 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T14:42:09.508 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:42:09.509 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T14:42:09.522 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:42:09.522 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T14:42:09.576 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:42:09.576 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T14:42:09.633 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:42:09.633 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T14:42:09.688 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:42:09.688 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T14:42:09.733 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T14:42:09.734 DEBUG:teuthology.orchestra.run.smithi026:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T14:42:09.736 DEBUG:teuthology.orchestra.run.smithi189:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T14:42:09.779 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.0] 2024-09-16T14:42:09.779 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCsQ+hmS699JBAAE/BCdpLPhpYpeW2qknavpQ== 2024-09-16T14:42:09.779 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.1] 2024-09-16T14:42:09.779 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCvQ+hmo+tNKRAAxiSeNZ86asNR1aGOmfdQbA== 2024-09-16T14:42:09.780 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.2] 2024-09-16T14:42:09.780 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCyQ+hmma6vMRAAHqAzhEvCHYFy6j3LaydNKA== 2024-09-16T14:42:09.780 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr.x] 2024-09-16T14:42:09.780 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCrQ+hmgA9NKBAA0wUfI6NlIvDtKM4wrm5R6g== 2024-09-16T14:42:09.780 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.3] 2024-09-16T14:42:09.780 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQC4Q+hmq9+SCBAAiT6LROOt99xxdCaB3S05Cg== 2024-09-16T14:42:09.780 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.4] 2024-09-16T14:42:09.780 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQC7Q+hmRUgYDRAAXr8vln7Whvstq6bzDltxUw== 2024-09-16T14:42:09.780 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.5] 2024-09-16T14:42:09.781 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQC+Q+hmqF5PFBAAxN8bLnGZ+hkHKswrg47leA== 2024-09-16T14:42:09.781 INFO:teuthology.orchestra.run.smithi026.stdout:[client.0] 2024-09-16T14:42:09.781 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCrQ+hmsBa/LRAASNPnIcyclVwHBfzT+H7KlQ== 2024-09-16T14:42:09.788 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.0] 2024-09-16T14:42:09.788 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQCsQ+hmS699JBAAE/BCdpLPhpYpeW2qknavpQ== 2024-09-16T14:42:09.788 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.1] 2024-09-16T14:42:09.788 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQCvQ+hmo+tNKRAAxiSeNZ86asNR1aGOmfdQbA== 2024-09-16T14:42:09.788 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.2] 2024-09-16T14:42:09.788 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQCyQ+hmma6vMRAAHqAzhEvCHYFy6j3LaydNKA== 2024-09-16T14:42:09.788 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr.x] 2024-09-16T14:42:09.788 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQCrQ+hmgA9NKBAA0wUfI6NlIvDtKM4wrm5R6g== 2024-09-16T14:42:09.788 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.3] 2024-09-16T14:42:09.788 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQC4Q+hmq9+SCBAAiT6LROOt99xxdCaB3S05Cg== 2024-09-16T14:42:09.788 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.4] 2024-09-16T14:42:09.789 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQC7Q+hmRUgYDRAAXr8vln7Whvstq6bzDltxUw== 2024-09-16T14:42:09.789 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.5] 2024-09-16T14:42:09.789 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQC+Q+hmqF5PFBAAxN8bLnGZ+hkHKswrg47leA== 2024-09-16T14:42:09.789 INFO:teuthology.orchestra.run.smithi189.stdout:[client.0] 2024-09-16T14:42:09.789 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQCrQ+hmsBa/LRAASNPnIcyclVwHBfzT+H7KlQ== 2024-09-16T14:42:09.790 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T14:42:09.827 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T14:42:09.877 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T14:42:09.914 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T14:42:09.960 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T14:42:10.006 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T14:42:10.051 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T14:42:10.053 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T14:42:10.133 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T14:42:10.138 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T14:42:10.224 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T14:42:10.226 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T14:42:10.304 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T14:42:10.307 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T14:42:10.387 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T14:42:10.447 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T14:42:10.496 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T14:42:10.497 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T14:42:10.512 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T14:42:10.605 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T14:42:10.616 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T14:42:10.671 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T14:42:10.761 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T14:42:10.815 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T14:42:10.829 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T14:42:10.923 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T14:42:10.934 DEBUG:teuthology.orchestra.run.smithi026:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T14:42:10.937 DEBUG:teuthology.orchestra.run.smithi189:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T14:42:10.981 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T14:42:10.981 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T14:42:10.982 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T14:42:10.986 INFO:tasks.ceph.mon.a:Started 2024-09-16T14:42:10.986 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T14:42:10.987 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T14:42:10.989 INFO:tasks.ceph.mon.c:Started 2024-09-16T14:42:10.989 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T14:42:10.989 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T14:42:11.026 INFO:tasks.ceph.mon.b:Started 2024-09-16T14:42:11.027 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T14:42:11.027 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T14:42:11.027 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T14:42:11.029 INFO:tasks.ceph.mgr.x:Started 2024-09-16T14:42:11.030 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T14:42:11.030 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T14:42:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa31a6bf640 10 monclient: get_monmap_and_config 2024-09-16T14:42:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa31a6bf640 10 monclient: build_initial_monmap 2024-09-16T14:42:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa31a6bf640 10 monclient: monmap: 2024-09-16T14:42:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:11.149910+0000 2024-09-16T14:42:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:11.149910+0000 2024-09-16T14:42:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:11.150 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa31a6bf640 1 Processor -- start 2024-09-16T14:42:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa31a6bf640 1 -- start start 2024-09-16T14:42:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa31a6bf640 10 monclient: init 2024-09-16T14:42:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa31a6bf640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa31a6bf640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa31a6bf640 10 monclient(hunting): picked mon.noname-c con 0x7fa31409f0a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa31a6bf640 10 monclient(hunting): picked mon.noname-b con 0x7fa31409d8e0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa31a6bf640 10 monclient(hunting): picked mon.noname-a con 0x7fa31409fb90 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa31a6bf640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31406c000 con 0x7fa31409fb90 2024-09-16T14:42:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa31a6bf640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31409cc20 con 0x7fa31409f0a0 2024-09-16T14:42:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa31a6bf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31409cf00 con 0x7fa31409d8e0 2024-09-16T14:42:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa31a6bf640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa31a6bf640 10 monclient(hunting): authenticate will time out at 955.270358s 2024-09-16T14:42:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa3137fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa31409d8e0 0x7fa314136be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:41032/0 (socket says 172.21.15.26:41032) 2024-09-16T14:42:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:11.148+0000 7fa3137fe640 1 -- 172.21.15.26:0/208070550 learned_addr learned my addr 172.21.15.26:0/208070550 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:14.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:14.148+0000 7fa311ffb640 10 monclient(hunting): tick 2024-09-16T14:42:14.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:14.148+0000 7fa311ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:42:14.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:14.148+0000 7fa311ffb640 1 monclient(hunting): continuing hunt 2024-09-16T14:42:14.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:14.148+0000 7fa311ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:42:14.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:14.148+0000 7fa311ffb640 1 -- 172.21.15.26:0/208070550 >> v1:172.21.15.26:6790/0 conn(0x7fa31409d8e0 legacy=0x7fa314136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:14.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:14.148+0000 7fa311ffb640 1 -- 172.21.15.26:0/208070550 >> v1:172.21.15.189:6789/0 conn(0x7fa31409f0a0 legacy=0x7fa31409f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:14.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:14.148+0000 7fa311ffb640 1 -- 172.21.15.26:0/208070550 >> v1:172.21.15.26:6789/0 conn(0x7fa31409fb90 legacy=0x7fa31413a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:14.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:14.148+0000 7fa311ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:14.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:14.148+0000 7fa311ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fa3000036f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:14.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:14.148+0000 7fa311ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fa31409d8e0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:14.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:14.148+0000 7fa311ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fa30000ab70 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:14.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:14.148+0000 7fa311ffb640 1 -- 172.21.15.26:0/208070550 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31409d1e0 con 0x7fa30000ab70 2024-09-16T14:42:14.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:14.148+0000 7fa311ffb640 1 -- 172.21.15.26:0/208070550 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa30000f880 con 0x7fa3000036f0 2024-09-16T14:42:14.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:14.148+0000 7fa311ffb640 1 -- 172.21.15.26:0/208070550 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa300010a90 con 0x7fa31409d8e0 2024-09-16T14:42:14.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:14.148+0000 7fa311ffb640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.148+0000 7fa311ffb640 10 monclient(hunting): tick 2024-09-16T14:42:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.148+0000 7fa311ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:42:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.148+0000 7fa311ffb640 1 monclient(hunting): continuing hunt 2024-09-16T14:42:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.148+0000 7fa311ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:42:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.148+0000 7fa311ffb640 1 -- 172.21.15.26:0/208070550 >> v1:172.21.15.26:6790/0 conn(0x7fa31409d8e0 legacy=0x7fa300009420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.148+0000 7fa311ffb640 1 -- 172.21.15.26:0/208070550 >> v1:172.21.15.189:6789/0 conn(0x7fa3000036f0 legacy=0x7fa300005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.148+0000 7fa311ffb640 1 -- 172.21.15.26:0/208070550 >> v1:172.21.15.26:6789/0 conn(0x7fa30000ab70 legacy=0x7fa30000d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.148+0000 7fa311ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:42:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.148+0000 7fa311ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fa300010e70 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.148+0000 7fa311ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fa30000fa10 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.148+0000 7fa311ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fa3000036f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.148+0000 7fa311ffb640 1 -- 172.21.15.26:0/208070550 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa300011ca0 con 0x7fa30000fa10 2024-09-16T14:42:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.148+0000 7fa311ffb640 1 -- 172.21.15.26:0/208070550 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3000105b0 con 0x7fa3000036f0 2024-09-16T14:42:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.148+0000 7fa311ffb640 1 -- 172.21.15.26:0/208070550 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa300010790 con 0x7fa300010e70 2024-09-16T14:42:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.148+0000 7fa311ffb640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:17.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.380+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199776702 0 0) 0x7fa300011ca0 con 0x7fa30000fa10 2024-09-16T14:42:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.380+0000 7fa312ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.380+0000 7fa312ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.380+0000 7fa312ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.380+0000 7fa312ffd640 10 monclient(hunting): my global_id is 4098 2024-09-16T14:42:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.380+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2fc005740 con 0x7fa30000fa10 2024-09-16T14:42:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.380+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246124563 0 0) 0x7fa3000105b0 con 0x7fa3000036f0 2024-09-16T14:42:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.380+0000 7fa312ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.380+0000 7fa312ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.380+0000 7fa312ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.380+0000 7fa312ffd640 10 monclient(hunting): my global_id is 4102 2024-09-16T14:42:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.380+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa300011ca0 con 0x7fa3000036f0 2024-09-16T14:42:17.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624749731 0 0) 0x7fa300010790 con 0x7fa300010e70 2024-09-16T14:42:17.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:17.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:17.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:17.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient(hunting): my global_id is 4103 2024-09-16T14:42:17.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3000105b0 con 0x7fa300010e70 2024-09-16T14:42:17.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1173403287 0 0) 0x7fa300011ca0 con 0x7fa3000036f0 2024-09-16T14:42:17.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 >> v1:172.21.15.26:6790/0 conn(0x7fa300010e70 legacy=0x7fa3000112b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 >> v1:172.21.15.26:6789/0 conn(0x7fa30000fa10 legacy=0x7fa30000fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 monclient: found mon.noname-c 2024-09-16T14:42:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:42:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30000a430 con 0x7fa3000036f0 2024-09-16T14:42:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa30c001920 con 0x7fa3000036f0 2024-09-16T14:42:17.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:17.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:17.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: dump: 2024-09-16T14:42:17.390 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:17.390 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:17.390 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.390 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.390 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:17.390 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:17.390 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:17.390 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:17.390 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:17.390 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:17.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa31a6bf640 5 monclient: authenticate success, global_id 4102 2024-09-16T14:42:17.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa31a6bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:17.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa318c5c640 1 -- 172.21.15.26:0/208070550 >> v1:172.21.15.189:6789/0 conn(0x7fa3000036f0 legacy=0x7fa300005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-16T14:42:17.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa318c5c640 1 -- 172.21.15.26:0/208070550 >> v1:172.21.15.189:6789/0 conn(0x7fa3000036f0 legacy=0x7fa300005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:42:17.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa318c5c640 1 --1- 172.21.15.26:0/208070550 >> v1:172.21.15.189:6789/0 conn(0x7fa3000036f0 0x7fa300005b40 :-1 s=OPENED pgs=11 cs=1 l=1).handle_message read tag failed 2024-09-16T14:42:17.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa318c5c640 1 --1- 172.21.15.26:0/208070550 >> v1:172.21.15.189:6789/0 conn(0x7fa3000036f0 0x7fa300005b40 :-1 s=OPENED pgs=11 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:42:17.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: ms_handle_reset current mon v1:172.21.15.189:6789/0 2024-09-16T14:42:17.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:17.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 >> v1:172.21.15.189:6789/0 conn(0x7fa3000036f0 legacy=0x7fa300005b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:42:17.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:42:17.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient(hunting): picked mon.c con 0x7fa300010e70 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:17.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient(hunting): picked mon.a con 0x7fa30000fa10 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:17.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient(hunting): picked mon.b con 0x7fa2fc00d2b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:17.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa300011ca0 con 0x7fa30000fa10 2024-09-16T14:42:17.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa300010790 con 0x7fa2fc00d2b0 2024-09-16T14:42:17.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa2fc013030 con 0x7fa300010e70 2024-09-16T14:42:17.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:17.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.189:6789/0 2024-09-16T14:42:17.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066511460 0 0) 0x7fa300011ca0 con 0x7fa30000fa10 2024-09-16T14:42:17.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:17.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:17.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:42:17.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 --> v1:172.21.15.26:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fa2fc0131c0 con 0x7fa30000fa10 2024-09-16T14:42:17.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 4135062253 0 0) 0x7fa2fc0131c0 con 0x7fa30000fa10 2024-09-16T14:42:17.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 >> v1:172.21.15.26:6790/0 conn(0x7fa300010e70 legacy=0x7fa2fc006050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:17.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 >> v1:172.21.15.189:6789/0 conn(0x7fa2fc00d2b0 legacy=0x7fa2fc00f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:17.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:17.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 monclient: found mon.a 2024-09-16T14:42:17.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:17.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:17.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7fa2fc0142b0 con 0x7fa30000fa10 2024-09-16T14:42:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa308001700 con 0x7fa30000fa10 2024-09-16T14:42:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa312ffd640 10 monclient: dump: 2024-09-16T14:42:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:17.395 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.384+0000 7fa31a6bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.388+0000 7fa312ffd640 1 -- 172.21.15.26:0/208070550 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3080018a0 con 0x7fa30000fa10 2024-09-16T14:42:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.388+0000 7fa312ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.388+0000 7fa31a6bf640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.388+0000 7fa31a6bf640 4 set_mon_vals no callback set 2024-09-16T14:42:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.388+0000 7fa31a6bf640 10 monclient: shutdown 2024-09-16T14:42:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.388+0000 7fa31a6bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.388+0000 7fa31a6bf640 1 -- 172.21.15.26:0/208070550 >> v1:172.21.15.26:6789/0 conn(0x7fa30000fa10 legacy=0x7fa2fc00bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.388+0000 7fa31a6bf640 1 -- 172.21.15.26:0/208070550 shutdown_connections 2024-09-16T14:42:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.388+0000 7fa31a6bf640 1 -- 172.21.15.26:0/208070550 >> 172.21.15.26:0/208070550 conn(0x7fa3140a0360 msgr2=0x7fa3140a0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.388+0000 7fa31a6bf640 1 -- 172.21.15.26:0/208070550 shutdown_connections 2024-09-16T14:42:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.388+0000 7fa31a6bf640 1 -- 172.21.15.26:0/208070550 wait complete. 2024-09-16T14:42:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa31a6bf640 10 monclient: build_initial_monmap 2024-09-16T14:42:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa31a6bf640 10 monclient: monmap: 2024-09-16T14:42:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:17.401 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa31a6bf640 1 Processor -- start 2024-09-16T14:42:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa31a6bf640 1 -- start start 2024-09-16T14:42:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa31a6bf640 10 monclient: init 2024-09-16T14:42:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa31a6bf640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa31a6bf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:42:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa31a6bf640 10 monclient(hunting): picked mon.noname-c con 0x7fa2fc00d2b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa31a6bf640 10 monclient(hunting): picked mon.noname-a con 0x7fa30000fa10 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa31a6bf640 10 monclient(hunting): picked mon.noname-b con 0x7fa300010e70 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa31a6bf640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa314097810 con 0x7fa30000fa10 2024-09-16T14:42:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa31a6bf640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31408a790 con 0x7fa300010e70 2024-09-16T14:42:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa31a6bf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3140a27f0 con 0x7fa2fc00d2b0 2024-09-16T14:42:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa31a6bf640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa31a6bf640 10 monclient(hunting): authenticate will time out at 961.515018s 2024-09-16T14:42:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa313fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa2fc00d2b0 0x7fa3141c9c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35662/0 (socket says 172.21.15.26:35662) 2024-09-16T14:42:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa313fff640 1 -- 172.21.15.26:0/3137123842 learned_addr learned my addr 172.21.15.26:0/3137123842 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa318c5c640 1 -- 172.21.15.26:0/3137123842 >> v1:172.21.15.189:6789/0 conn(0x7fa300010e70 legacy=0x7fa3141d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T14:42:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa318c5c640 1 -- 172.21.15.26:0/3137123842 >> v1:172.21.15.189:6789/0 conn(0x7fa300010e70 legacy=0x7fa3141d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:42:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa318c5c640 1 --1- 172.21.15.26:0/3137123842 >> v1:172.21.15.189:6789/0 conn(0x7fa300010e70 0x7fa3141d2360 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-16T14:42:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa318c5c640 1 --1- 172.21.15.26:0/3137123842 >> v1:172.21.15.189:6789/0 conn(0x7fa300010e70 0x7fa3141d2360 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-16T14:42:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.392+0000 7fa310ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.189:6789/0 2024-09-16T14:42:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161560199 0 0) 0x7fa3140a27f0 con 0x7fa2fc00d2b0 2024-09-16T14:42:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient(hunting): my global_id is 4109 2024-09-16T14:42:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa30000a620 con 0x7fa2fc00d2b0 2024-09-16T14:42:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026107419 0 0) 0x7fa314097810 con 0x7fa30000fa10 2024-09-16T14:42:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient(hunting): my global_id is 4107 2024-09-16T14:42:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3140a27f0 con 0x7fa30000fa10 2024-09-16T14:42:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1189025871 0 0) 0x7fa30000a620 con 0x7fa2fc00d2b0 2024-09-16T14:42:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa314097810 con 0x7fa2fc00d2b0 2024-09-16T14:42:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa3040017b0 con 0x7fa2fc00d2b0 2024-09-16T14:42:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:17.405 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa313fff640 1 -- 172.21.15.26:0/3137123842 >> v1:172.21.15.26:6790/0 conn(0x7fa2fc00d2b0 legacy=0x7fa3141c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T14:42:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa313fff640 1 -- 172.21.15.26:0/3137123842 >> v1:172.21.15.26:6790/0 conn(0x7fa2fc00d2b0 legacy=0x7fa3141c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:42:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa313fff640 1 --1- 172.21.15.26:0/3137123842 >> v1:172.21.15.26:6790/0 conn(0x7fa2fc00d2b0 0x7fa3141c9c50 :-1 s=OPENED pgs=17 cs=1 l=1).handle_message read tag failed 2024-09-16T14:42:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa313fff640 1 --1- 172.21.15.26:0/3137123842 >> v1:172.21.15.26:6790/0 conn(0x7fa2fc00d2b0 0x7fa3141c9c50 :-1 s=OPENED pgs=17 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:42:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6790/0 2024-09-16T14:42:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2833010098 0 0) 0x7fa3140a27f0 con 0x7fa30000fa10 2024-09-16T14:42:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa30000a620 con 0x7fa30000fa10 2024-09-16T14:42:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa308001470 con 0x7fa30000fa10 2024-09-16T14:42:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:17.406 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2992159624 0 0) 0x7fa30000a620 con 0x7fa30000fa10 2024-09-16T14:42:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 >> v1:172.21.15.26:6790/0 conn(0x7fa2fc00d2b0 legacy=0x7fa3141c9c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:42:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 >> v1:172.21.15.189:6789/0 conn(0x7fa300010e70 legacy=0x7fa3141d2360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:42:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:42:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 monclient: found mon.a 2024-09-16T14:42:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa31408a4b0 con 0x7fa30000fa10 2024-09-16T14:42:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient: _finish_auth 0 2024-09-16T14:42:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:42:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa31a6bf640 5 monclient: authenticate success, global_id 4107 2024-09-16T14:42:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa31a6bf640 10 monclient: _renew_subs 2024-09-16T14:42:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa31a6bf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa31a6bf640 1 -- 172.21.15.26:0/3137123842 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa314130180 con 0x7fa30000fa10 2024-09-16T14:42:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa31a6bf640 10 monclient: _renew_subs 2024-09-16T14:42:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa31a6bf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa31a6bf640 1 -- 172.21.15.26:0/3137123842 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa314130ac0 con 0x7fa30000fa10 2024-09-16T14:42:17.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa31a6bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:17.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa31a6bf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:17.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa31a6bf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:17.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa31a6bf640 1 -- 172.21.15.26:0/3137123842 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3140008e0 con 0x7fa30000fa10 2024-09-16T14:42:17.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa308001690 con 0x7fa30000fa10 2024-09-16T14:42:17.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:17.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa308004c40 con 0x7fa30000fa10 2024-09-16T14:42:17.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:17.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:17.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 10 monclient: dump: 2024-09-16T14:42:17.410 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:17.410 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:17.410 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.410 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.410 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:17.410 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:17.410 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:17.410 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:17.410 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:17.411 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:17.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa3127fc640 4 set_mon_vals no callback set 2024-09-16T14:42:17.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.396+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fa3080050a0 con 0x7fa30000fa10 2024-09-16T14:42:17.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.400+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3501550015 0 0) 0x7fa3080062c0 con 0x7fa30000fa10 2024-09-16T14:42:17.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.400+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7fa308005530 con 0x7fa30000fa10 2024-09-16T14:42:17.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.400+0000 7fa310ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:17.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.400+0000 7fa310ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:17.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.468+0000 7fa31a6bf640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T14:42:17.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.468+0000 7fa31a6bf640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T14:42:17.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.468+0000 7fa31a6bf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:17.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.468+0000 7fa31a6bf640 1 -- 172.21.15.26:0/3137123842 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fa3140008e0 con 0x7fa30000fa10 2024-09-16T14:42:17.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.472+0000 7fa310ff9640 1 -- 172.21.15.26:0/3137123842 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fa308005850 con 0x7fa30000fa10 2024-09-16T14:42:17.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.472+0000 7fa310ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T14:42:17.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.472+0000 7fa310ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-16T14:42:17.474 INFO:teuthology.orchestra.run.smithi026.stderr:adjusted tunables profile to default 2024-09-16T14:42:17.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.472+0000 7fa31a6bf640 10 monclient: shutdown 2024-09-16T14:42:17.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.472+0000 7fa31a6bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:17.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.472+0000 7fa31a6bf640 1 -- 172.21.15.26:0/3137123842 >> v1:172.21.15.26:6789/0 conn(0x7fa30000fa10 legacy=0x7fa3141cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:17.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.472+0000 7fa31a6bf640 1 -- 172.21.15.26:0/3137123842 shutdown_connections 2024-09-16T14:42:17.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.472+0000 7fa31a6bf640 1 -- 172.21.15.26:0/3137123842 >> 172.21.15.26:0/3137123842 conn(0x7fa3140a0360 msgr2=0x7fa314071e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:17.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.472+0000 7fa31a6bf640 1 -- 172.21.15.26:0/3137123842 shutdown_connections 2024-09-16T14:42:17.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:17.472+0000 7fa31a6bf640 1 -- 172.21.15.26:0/3137123842 wait complete. 2024-09-16T14:42:17.494 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T14:42:17.494 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T14:42:17.494 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:42:17.494 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T14:42:17.508 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:42:17.508 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T14:42:17.559 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-09-16T14:42:17.559 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T14:42:17.612 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:42:17.613 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T14:42:17.627 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:42:17.627 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T14:42:17.681 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:42:17.681 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T14:42:17.738 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new a4578188-6e8b-430f-b70d-1c3480eb7bae 0 2024-09-16T14:42:17.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.905+0000 7f0876a3f640 10 monclient: get_monmap_and_config 2024-09-16T14:42:17.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.905+0000 7f0876a3f640 10 monclient: build_initial_monmap 2024-09-16T14:42:17.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.905+0000 7f0876a3f640 10 monclient: monmap: 2024-09-16T14:42:17.908 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T14:42:17.908 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:17.908 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:42:17.908400+0000 2024-09-16T14:42:17.908 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:42:17.908400+0000 2024-09-16T14:42:17.908 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:17.908 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:17.908 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:17.908 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:17.908 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:17.909 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:17.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.905+0000 7f0876a3f640 1 Processor -- start 2024-09-16T14:42:17.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.905+0000 7f0876a3f640 1 -- start start 2024-09-16T14:42:17.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.905+0000 7f0876a3f640 10 monclient: init 2024-09-16T14:42:17.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.905+0000 7f0876a3f640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:17.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.905+0000 7f0876a3f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:17.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.905+0000 7f0876a3f640 10 monclient(hunting): picked mon.noname-c con 0x7f08700a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:17.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.905+0000 7f0876a3f640 10 monclient(hunting): picked mon.noname-b con 0x7f0870099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:17.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.905+0000 7f0876a3f640 10 monclient(hunting): picked mon.noname-a con 0x7f08700a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:17.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.905+0000 7f0876a3f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0870098ea0 con 0x7f08700a26f0 2024-09-16T14:42:17.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.905+0000 7f0876a3f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08700997b0 con 0x7f08700a1c00 2024-09-16T14:42:17.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.905+0000 7f0876a3f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0870099460 con 0x7f0870099b60 2024-09-16T14:42:17.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.905+0000 7f0876a3f640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:17.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.905+0000 7f0876a3f640 10 monclient(hunting): authenticate will time out at 920.023970s 2024-09-16T14:42:17.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086ffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f08700a1c00 0x7f08700a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39190/0 (socket says 172.21.15.189:39190) 2024-09-16T14:42:17.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086ffff640 1 -- 172.21.15.189:0/1183805434 learned_addr learned my addr 172.21.15.189:0/1183805434 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:42:17.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 1 -- 172.21.15.189:0/1183805434 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553495599 0 0) 0x7f08700997b0 con 0x7f08700a1c00 2024-09-16T14:42:17.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:17.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:17.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:17.916 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 10 monclient(hunting): my global_id is 4108 2024-09-16T14:42:17.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 1 -- 172.21.15.189:0/1183805434 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0858003530 con 0x7f08700a1c00 2024-09-16T14:42:17.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 1 -- 172.21.15.189:0/1183805434 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1286520526 0 0) 0x7f0858003530 con 0x7f08700a1c00 2024-09-16T14:42:17.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 1 -- 172.21.15.189:0/1183805434 >> v1:172.21.15.26:6790/0 conn(0x7f0870099b60 legacy=0x7f0870139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:17.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 1 -- 172.21.15.189:0/1183805434 >> v1:172.21.15.26:6789/0 conn(0x7f08700a26f0 legacy=0x7f087013d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:17.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:17.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 1 monclient: found mon.noname-c 2024-09-16T14:42:17.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:17.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:42:17.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 1 -- 172.21.15.189:0/1183805434 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08700a34b0 con 0x7f08700a1c00 2024-09-16T14:42:17.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 10 monclient: _finish_auth 0 2024-09-16T14:42:17.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:17.917 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:17.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 1 -- 172.21.15.189:0/1183805434 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0860001620 con 0x7f08700a1c00 2024-09-16T14:42:17.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:17.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:17.918 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 10 monclient: dump: 2024-09-16T14:42:17.918 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:17.918 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:17.918 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.918 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.918 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:17.918 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:17.918 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:17.918 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:17.918 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:17.919 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:17.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 5 monclient: authenticate success, global_id 4108 2024-09-16T14:42:17.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:17.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 1 -- 172.21.15.189:0/1183805434 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08600018d0 con 0x7f08700a1c00 2024-09-16T14:42:17.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:17.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:17.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 1 -- 172.21.15.189:0/1183805434 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0860001ca0 con 0x7f08700a1c00 2024-09-16T14:42:17.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 4 set_mon_vals no callback set 2024-09-16T14:42:17.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:17.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:17.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086effd640 10 monclient: dump: 2024-09-16T14:42:17.919 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:17.919 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:17.920 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.920 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.920 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:17.920 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:17.920 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:17.920 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:17.920 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:17.920 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:17.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 10 monclient: shutdown 2024-09-16T14:42:17.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:17.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 1 -- 172.21.15.189:0/1183805434 >> v1:172.21.15.189:6789/0 conn(0x7f08700a1c00 legacy=0x7f08700a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:17.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 1 -- 172.21.15.189:0/1183805434 shutdown_connections 2024-09-16T14:42:17.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 1 -- 172.21.15.189:0/1183805434 >> 172.21.15.189:0/1183805434 conn(0x7f08700a3d00 msgr2=0x7f08700a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:17.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 1 -- 172.21.15.189:0/1183805434 shutdown_connections 2024-09-16T14:42:17.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 1 -- 172.21.15.189:0/1183805434 wait complete. 2024-09-16T14:42:17.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 10 monclient: build_initial_monmap 2024-09-16T14:42:17.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 10 monclient: monmap: 2024-09-16T14:42:17.921 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T14:42:17.921 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:17.921 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-09-16T14:42:17.921 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-09-16T14:42:17.921 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:17.921 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-09-16T14:42:17.921 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:17.921 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:17.921 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:17.922 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:17.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 1 Processor -- start 2024-09-16T14:42:17.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 1 -- start start 2024-09-16T14:42:17.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 10 monclient: init 2024-09-16T14:42:17.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:17.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:17.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 10 monclient(hunting): picked mon.noname-c con 0x7f0870099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:17.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 10 monclient(hunting): picked mon.noname-b con 0x7f08700a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:17.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 10 monclient(hunting): picked mon.noname-a con 0x7f08700a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:17.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08700a1500 con 0x7f08700a26f0 2024-09-16T14:42:17.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087013d460 con 0x7f08700a1c00 2024-09-16T14:42:17.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087013e830 con 0x7f0870099b60 2024-09-16T14:42:17.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:17.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f0876a3f640 10 monclient(hunting): authenticate will time out at 920.027125s 2024-09-16T14:42:17.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086f7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f08700a1c00 0x7f08701cd900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39194/0 (socket says 172.21.15.189:39194) 2024-09-16T14:42:17.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086f7fe640 1 -- 172.21.15.189:0/1231266714 learned_addr learned my addr 172.21.15.189:0/1231266714 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:42:17.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441031730 0 0) 0x7f087013d460 con 0x7f08700a1c00 2024-09-16T14:42:17.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:17.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:17.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:17.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 10 monclient(hunting): my global_id is 4111 2024-09-16T14:42:17.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f085c003530 con 0x7f08700a1c00 2024-09-16T14:42:17.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3267110905 0 0) 0x7f085c003530 con 0x7f08700a1c00 2024-09-16T14:42:17.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f087013d460 con 0x7f08700a1c00 2024-09-16T14:42:17.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0864001880 con 0x7f08700a1c00 2024-09-16T14:42:17.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:17.923 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:17.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 10 monclient(hunting): dump: 2024-09-16T14:42:17.924 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:17.924 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:17.924 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.924 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.924 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:17.924 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:17.924 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:17.924 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:17.924 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:17.924 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:17.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:17.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896742741 0 0) 0x7f0870098ea0 con 0x7f08700a26f0 2024-09-16T14:42:17.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:17.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:17.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:17.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 10 monclient(hunting): my global_id is 4113 2024-09-16T14:42:17.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f085c003530 con 0x7f08700a26f0 2024-09-16T14:42:17.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.909+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878610960 0 0) 0x7f087013e830 con 0x7f0870099b60 2024-09-16T14:42:17.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:17.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:17.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:17.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 10 monclient(hunting): my global_id is 4115 2024-09-16T14:42:17.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0870098ea0 con 0x7f0870099b60 2024-09-16T14:42:17.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2980220653 0 0) 0x7f087013d460 con 0x7f08700a1c00 2024-09-16T14:42:17.925 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 >> v1:172.21.15.26:6790/0 conn(0x7f0870099b60 legacy=0x7f087013e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:17.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 >> v1:172.21.15.26:6789/0 conn(0x7f08700a26f0 legacy=0x7f08701d1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:17.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:42:17.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 1 monclient: found mon.b 2024-09-16T14:42:17.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:17.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:17.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087009e3f0 con 0x7f08700a1c00 2024-09-16T14:42:17.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 10 monclient: _finish_auth 0 2024-09-16T14:42:17.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:42:17.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:17.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f0876a3f640 5 monclient: authenticate success, global_id 4111 2024-09-16T14:42:17.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f0876a3f640 10 monclient: _renew_subs 2024-09-16T14:42:17.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f0876a3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:17.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f0876a3f640 1 -- 172.21.15.189:0/1231266714 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087013dd00 con 0x7f08700a1c00 2024-09-16T14:42:17.926 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f0876a3f640 10 monclient: _renew_subs 2024-09-16T14:42:17.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f0876a3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:17.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f0876a3f640 1 -- 172.21.15.189:0/1231266714 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f087013ba00 con 0x7f08700a1c00 2024-09-16T14:42:17.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0864001c50 con 0x7f08700a1c00 2024-09-16T14:42:17.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:17.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0864003af0 con 0x7f08700a1c00 2024-09-16T14:42:17.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:17.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:17.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086e7fc640 4 set_mon_vals no callback set 2024-09-16T14:42:17.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 10 monclient: dump: 2024-09-16T14:42:17.927 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:17.927 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:17.927 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.927 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:17.928 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:17.928 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:17.928 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:17.928 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:17.928 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:17.928 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:17.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7f0864003fd0 con 0x7f08700a1c00 2024-09-16T14:42:17.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2970693161 0 0) 0x7f08640064b0 con 0x7f08700a1c00 2024-09-16T14:42:17.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f0876a3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:17.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f0876a3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:17.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f0876a3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:17.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f0876a3f640 1 -- 172.21.15.189:0/1231266714 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0850005050 con 0x7f08700a1c00 2024-09-16T14:42:17.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7f087013ba00 con 0x7f08700a1c00 2024-09-16T14:42:17.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:17.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.913+0000 7f086cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:17.941 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:17.941+0000 7f9d7ec72180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T14:42:17.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.981+0000 7f0876a3f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a4578188-6e8b-430f-b70d-1c3480eb7bae", "id": 0}] 2024-09-16T14:42:17.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.981+0000 7f0876a3f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a4578188-6e8b-430f-b70d-1c3480eb7bae", "id": 0}] 2024-09-16T14:42:17.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.981+0000 7f0876a3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:17.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.981+0000 7f0876a3f640 1 -- 172.21.15.189:0/1231266714 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a4578188-6e8b-430f-b70d-1c3480eb7bae", "id": 0} v 0) v1 -- 0x7f0850005880 con 0x7f08700a1c00 2024-09-16T14:42:17.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.989+0000 7f086cff9640 1 -- 172.21.15.189:0/1231266714 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a4578188-6e8b-430f-b70d-1c3480eb7bae", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2670452182 0 732707515) 0x7f08640057d0 con 0x7f08700a1c00 2024-09-16T14:42:17.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.989+0000 7f086cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a4578188-6e8b-430f-b70d-1c3480eb7bae", "id": 0}] 2024-09-16T14:42:17.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.989+0000 7f086cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:17.991 INFO:teuthology.orchestra.run.smithi189.stdout:0 2024-09-16T14:42:17.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.989+0000 7f0876a3f640 10 monclient: shutdown 2024-09-16T14:42:17.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.989+0000 7f0876a3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:17.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.989+0000 7f0876a3f640 1 -- 172.21.15.189:0/1231266714 >> v1:172.21.15.189:6789/0 conn(0x7f08700a1c00 legacy=0x7f08701cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:17.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.989+0000 7f0876a3f640 1 -- 172.21.15.189:0/1231266714 shutdown_connections 2024-09-16T14:42:17.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.989+0000 7f0876a3f640 1 -- 172.21.15.189:0/1231266714 >> 172.21.15.189:0/1231266714 conn(0x7f08700a3d00 msgr2=0x7f087009f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:17.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.989+0000 7f0876a3f640 1 -- 172.21.15.189:0/1231266714 shutdown_connections 2024-09-16T14:42:17.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:17.989+0000 7f0876a3f640 1 -- 172.21.15.189:0/1231266714 wait complete. 2024-09-16T14:42:18.009 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 1a96367f-73f1-4a99-9b85-47bf8e3d4cdc 1 2024-09-16T14:42:18.124 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.121+0000 7fb9c7407640 10 monclient: get_monmap_and_config 2024-09-16T14:42:18.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.121+0000 7fb9c7407640 10 monclient: build_initial_monmap 2024-09-16T14:42:18.125 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.121+0000 7fb9c7407640 10 monclient: monmap: 2024-09-16T14:42:18.125 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T14:42:18.125 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.125 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:42:18.125442+0000 2024-09-16T14:42:18.125 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:42:18.125442+0000 2024-09-16T14:42:18.125 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:18.125 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.125 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:18.125 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:18.125 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:18.126 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 1 Processor -- start 2024-09-16T14:42:18.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 1 -- start start 2024-09-16T14:42:18.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 10 monclient: init 2024-09-16T14:42:18.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:18.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:18.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 10 monclient(hunting): picked mon.noname-c con 0x7fb9c00966a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:18.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 10 monclient(hunting): picked mon.noname-b con 0x7fb9c009be00 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:18.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 10 monclient(hunting): picked mon.noname-a con 0x7fb9c0139150 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:18.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c006c000 con 0x7fb9c0139150 2024-09-16T14:42:18.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c00998a0 con 0x7fb9c00966a0 2024-09-16T14:42:18.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c009dbc0 con 0x7fb9c009be00 2024-09-16T14:42:18.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:18.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 10 monclient(hunting): authenticate will time out at 920.241372s 2024-09-16T14:42:18.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c51a3640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb9c00966a0 0x7fb9c0096a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39200/0 (socket says 172.21.15.189:39200) 2024-09-16T14:42:18.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c51a3640 1 -- 172.21.15.189:0/1175498540 learned_addr learned my addr 172.21.15.189:0/1175498540 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:42:18.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 1 -- 172.21.15.189:0/1175498540 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810887075 0 0) 0x7fb9c00998a0 con 0x7fb9c00966a0 2024-09-16T14:42:18.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:18.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:18.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:18.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 10 monclient(hunting): my global_id is 4114 2024-09-16T14:42:18.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 1 -- 172.21.15.189:0/1175498540 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ac003530 con 0x7fb9c00966a0 2024-09-16T14:42:18.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 1 -- 172.21.15.189:0/1175498540 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3735290338 0 0) 0x7fb9ac003530 con 0x7fb9c00966a0 2024-09-16T14:42:18.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 1 -- 172.21.15.189:0/1175498540 >> v1:172.21.15.26:6790/0 conn(0x7fb9c009be00 legacy=0x7fb9c0137a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 1 -- 172.21.15.189:0/1175498540 >> v1:172.21.15.26:6789/0 conn(0x7fb9c0139150 legacy=0x7fb9c013b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:42:18.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 1 monclient: found mon.noname-c 2024-09-16T14:42:18.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:18.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 1 -- 172.21.15.189:0/1175498540 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c009c230 con 0x7fb9c00966a0 2024-09-16T14:42:18.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 10 monclient: _finish_auth 0 2024-09-16T14:42:18.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:42:18.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:18.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 5 monclient: authenticate success, global_id 4114 2024-09-16T14:42:18.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:18.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 1 -- 172.21.15.189:0/1175498540 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9bc001620 con 0x7fb9c00966a0 2024-09-16T14:42:18.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:18.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 10 monclient: dump: 2024-09-16T14:42:18.131 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.131 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.131 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.131 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.132 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.132 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.132 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.132 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.132 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.132 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:18.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 1 -- 172.21.15.189:0/1175498540 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9bc001940 con 0x7fb9c00966a0 2024-09-16T14:42:18.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:18.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:18.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 4 set_mon_vals no callback set 2024-09-16T14:42:18.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 1 -- 172.21.15.189:0/1175498540 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9bc001cd0 con 0x7fb9c00966a0 2024-09-16T14:42:18.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:18.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9b7fff640 10 monclient: dump: 2024-09-16T14:42:18.133 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.133 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.133 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.133 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.133 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.133 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.133 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.133 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.133 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.133 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 10 monclient: shutdown 2024-09-16T14:42:18.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 1 -- 172.21.15.189:0/1175498540 >> v1:172.21.15.189:6789/0 conn(0x7fb9c00966a0 legacy=0x7fb9c0096a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 1 -- 172.21.15.189:0/1175498540 shutdown_connections 2024-09-16T14:42:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 1 -- 172.21.15.189:0/1175498540 >> 172.21.15.189:0/1175498540 conn(0x7fb9c00a0360 msgr2=0x7fb9c00a0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 1 -- 172.21.15.189:0/1175498540 shutdown_connections 2024-09-16T14:42:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 1 -- 172.21.15.189:0/1175498540 wait complete. 2024-09-16T14:42:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 10 monclient: build_initial_monmap 2024-09-16T14:42:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 10 monclient: monmap: 2024-09-16T14:42:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T14:42:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-09-16T14:42:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-09-16T14:42:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:18.134 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-09-16T14:42:18.135 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:18.135 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:18.135 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:18.135 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 1 Processor -- start 2024-09-16T14:42:18.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 1 -- start start 2024-09-16T14:42:18.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.125+0000 7fb9c7407640 10 monclient: init 2024-09-16T14:42:18.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:18.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:42:18.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 10 monclient(hunting): picked mon.noname-a con 0x7fb9c00966a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:18.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 10 monclient(hunting): picked mon.noname-b con 0x7fb9c009be00 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:18.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 10 monclient(hunting): picked mon.noname-c con 0x7fb9c0139150 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:18.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c009a8f0 con 0x7fb9c00966a0 2024-09-16T14:42:18.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c0093c30 con 0x7fb9c009be00 2024-09-16T14:42:18.138 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c0071b10 con 0x7fb9c0139150 2024-09-16T14:42:18.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:18.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 10 monclient(hunting): authenticate will time out at 920.244380s 2024-09-16T14:42:18.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c49a2640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb9c009be00 0x7fb9c01cc790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39208/0 (socket says 172.21.15.189:39208) 2024-09-16T14:42:18.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c49a2640 1 -- 172.21.15.189:0/646147404 learned_addr learned my addr 172.21.15.189:0/646147404 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:42:18.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 1 -- 172.21.15.189:0/646147404 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648013910 0 0) 0x7fb9c0093c30 con 0x7fb9c009be00 2024-09-16T14:42:18.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:18.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:18.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:18.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 10 monclient(hunting): my global_id is 4117 2024-09-16T14:42:18.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 1 -- 172.21.15.189:0/646147404 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a4003530 con 0x7fb9c009be00 2024-09-16T14:42:18.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 1 -- 172.21.15.189:0/646147404 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1616276404 0 0) 0x7fb9a4003530 con 0x7fb9c009be00 2024-09-16T14:42:18.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 1 -- 172.21.15.189:0/646147404 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9c0093c30 con 0x7fb9c009be00 2024-09-16T14:42:18.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 1 -- 172.21.15.189:0/646147404 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9b00018c0 con 0x7fb9c009be00 2024-09-16T14:42:18.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:18.140 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 10 monclient(hunting): dump: 2024-09-16T14:42:18.140 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.140 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.140 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.140 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.140 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.140 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.140 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.140 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.140 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.141 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:18.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 1 -- 172.21.15.189:0/646147404 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1392604877 0 0) 0x7fb9c0093c30 con 0x7fb9c009be00 2024-09-16T14:42:18.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 1 -- 172.21.15.189:0/646147404 >> v1:172.21.15.26:6790/0 conn(0x7fb9c0139150 legacy=0x7fb9c01cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 1 -- 172.21.15.189:0/646147404 >> v1:172.21.15.26:6789/0 conn(0x7fb9c00966a0 legacy=0x7fb9c0132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:42:18.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 1 monclient: found mon.b 2024-09-16T14:42:18.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:18.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 1 -- 172.21.15.189:0/646147404 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c0071df0 con 0x7fb9c009be00 2024-09-16T14:42:18.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 10 monclient: _finish_auth 0 2024-09-16T14:42:18.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:42:18.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:18.141 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 5 monclient: authenticate success, global_id 4117 2024-09-16T14:42:18.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 10 monclient: _renew_subs 2024-09-16T14:42:18.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 1 -- 172.21.15.189:0/646147404 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c0070cb0 con 0x7fb9c009be00 2024-09-16T14:42:18.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 10 monclient: _renew_subs 2024-09-16T14:42:18.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 1 -- 172.21.15.189:0/646147404 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c0070770 con 0x7fb9c009be00 2024-09-16T14:42:18.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 1 -- 172.21.15.189:0/646147404 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9b0003620 con 0x7fb9c009be00 2024-09-16T14:42:18.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:18.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 1 -- 172.21.15.189:0/646147404 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9b00039b0 con 0x7fb9c009be00 2024-09-16T14:42:18.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:18.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b77fe640 4 set_mon_vals no callback set 2024-09-16T14:42:18.142 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 10 monclient: dump: 2024-09-16T14:42:18.143 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.143 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.143 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.143 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.143 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.143 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.143 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.143 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.143 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.143 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 1 -- 172.21.15.189:0/646147404 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fb9b0003e50 con 0x7fb9c009be00 2024-09-16T14:42:18.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9b5ffb640 1 -- 172.21.15.189:0/646147404 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2262985123 0 0) 0x7fb9b0006290 con 0x7fb9c009be00 2024-09-16T14:42:18.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:18.143 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:18.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.129+0000 7fb9c7407640 1 -- 172.21.15.189:0/646147404 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a0005050 con 0x7fb9c009be00 2024-09-16T14:42:18.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.133+0000 7fb9b5ffb640 1 -- 172.21.15.189:0/646147404 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7fb9b0004390 con 0x7fb9c009be00 2024-09-16T14:42:18.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.133+0000 7fb9b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:18.144 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.133+0000 7fb9b5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:18.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.201+0000 7fb9c7407640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1a96367f-73f1-4a99-9b85-47bf8e3d4cdc", "id": 1}] 2024-09-16T14:42:18.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.201+0000 7fb9c7407640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1a96367f-73f1-4a99-9b85-47bf8e3d4cdc", "id": 1}] 2024-09-16T14:42:18.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.201+0000 7fb9c7407640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.201+0000 7fb9c7407640 1 -- 172.21.15.189:0/646147404 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1a96367f-73f1-4a99-9b85-47bf8e3d4cdc", "id": 1} v 0) v1 -- 0x7fb9a0005880 con 0x7fb9c009be00 2024-09-16T14:42:18.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.205+0000 7fb9b5ffb640 1 -- 172.21.15.189:0/646147404 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1a96367f-73f1-4a99-9b85-47bf8e3d4cdc", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3444906713 0 940483276) 0x7fb9b00058a0 con 0x7fb9c009be00 2024-09-16T14:42:18.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.205+0000 7fb9b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1a96367f-73f1-4a99-9b85-47bf8e3d4cdc", "id": 1}] 2024-09-16T14:42:18.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.205+0000 7fb9b5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:18.208 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-09-16T14:42:18.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.205+0000 7fb9c7407640 10 monclient: shutdown 2024-09-16T14:42:18.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.205+0000 7fb9c7407640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:18.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.205+0000 7fb9c7407640 1 -- 172.21.15.189:0/646147404 >> v1:172.21.15.189:6789/0 conn(0x7fb9c009be00 legacy=0x7fb9c01cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.205+0000 7fb9c7407640 1 -- 172.21.15.189:0/646147404 shutdown_connections 2024-09-16T14:42:18.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.205+0000 7fb9c7407640 1 -- 172.21.15.189:0/646147404 >> 172.21.15.189:0/646147404 conn(0x7fb9c00a0360 msgr2=0x7fb9c0095130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:18.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.209+0000 7fb9c7407640 1 -- 172.21.15.189:0/646147404 shutdown_connections 2024-09-16T14:42:18.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.209+0000 7fb9c7407640 1 -- 172.21.15.189:0/646147404 wait complete. 2024-09-16T14:42:18.225 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 3030f43d-f848-4189-be1e-6ea367941218 2 2024-09-16T14:42:18.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 10 monclient: get_monmap_and_config 2024-09-16T14:42:18.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 10 monclient: build_initial_monmap 2024-09-16T14:42:18.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 10 monclient: monmap: 2024-09-16T14:42:18.346 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T14:42:18.346 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.347 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:42:18.347147+0000 2024-09-16T14:42:18.347 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:42:18.347147+0000 2024-09-16T14:42:18.347 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:18.347 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.347 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:18.347 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:18.347 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:18.347 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 1 Processor -- start 2024-09-16T14:42:18.350 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 1 -- start start 2024-09-16T14:42:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 10 monclient: init 2024-09-16T14:42:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:42:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 10 monclient(hunting): picked mon.noname-a con 0x7fc6240a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 10 monclient(hunting): picked mon.noname-c con 0x7fc624099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 10 monclient(hunting): picked mon.noname-b con 0x7fc6240a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc624098ea0 con 0x7fc6240a1c00 2024-09-16T14:42:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6240997b0 con 0x7fc624099b60 2024-09-16T14:42:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc624099460 con 0x7fc6240a26f0 2024-09-16T14:42:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 10 monclient(hunting): authenticate will time out at 920.462888s 2024-09-16T14:42:18.351 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc6237fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc624099b60 0x7fc624139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39216/0 (socket says 172.21.15.189:39216) 2024-09-16T14:42:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc6237fe640 1 -- 172.21.15.189:0/421971318 learned_addr learned my addr 172.21.15.189:0/421971318 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:42:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 1 -- 172.21.15.189:0/421971318 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756223690 0 0) 0x7fc6240997b0 con 0x7fc624099b60 2024-09-16T14:42:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 10 monclient(hunting): my global_id is 4120 2024-09-16T14:42:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 1 -- 172.21.15.189:0/421971318 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc60c003530 con 0x7fc624099b60 2024-09-16T14:42:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 1 -- 172.21.15.189:0/421971318 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 396863840 0 0) 0x7fc60c003530 con 0x7fc624099b60 2024-09-16T14:42:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 1 -- 172.21.15.189:0/421971318 >> v1:172.21.15.26:6790/0 conn(0x7fc6240a26f0 legacy=0x7fc62413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 1 -- 172.21.15.189:0/421971318 >> v1:172.21.15.26:6789/0 conn(0x7fc6240a1c00 legacy=0x7fc6240a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 1 monclient: found mon.noname-c 2024-09-16T14:42:18.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 1 -- 172.21.15.189:0/421971318 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6240a34b0 con 0x7fc624099b60 2024-09-16T14:42:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 1 -- 172.21.15.189:0/421971318 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc618001620 con 0x7fc624099b60 2024-09-16T14:42:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 10 monclient: dump: 2024-09-16T14:42:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.353 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.354 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.354 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.354 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.354 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.354 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 5 monclient: authenticate success, global_id 4120 2024-09-16T14:42:18.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:18.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 1 -- 172.21.15.189:0/421971318 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6180018d0 con 0x7fc624099b60 2024-09-16T14:42:18.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:18.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 1 -- 172.21.15.189:0/421971318 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc618001ca0 con 0x7fc624099b60 2024-09-16T14:42:18.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:18.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc622ffd640 10 monclient: dump: 2024-09-16T14:42:18.354 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.355 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.355 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.355 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.355 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.355 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.355 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.355 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.355 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.355 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:18.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.345+0000 7fc62a326640 4 set_mon_vals no callback set 2024-09-16T14:42:18.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 10 monclient: shutdown 2024-09-16T14:42:18.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:18.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 1 -- 172.21.15.189:0/421971318 >> v1:172.21.15.189:6789/0 conn(0x7fc624099b60 legacy=0x7fc624139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 1 -- 172.21.15.189:0/421971318 shutdown_connections 2024-09-16T14:42:18.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 1 -- 172.21.15.189:0/421971318 >> 172.21.15.189:0/421971318 conn(0x7fc6240a3d00 msgr2=0x7fc6240a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:18.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 1 -- 172.21.15.189:0/421971318 shutdown_connections 2024-09-16T14:42:18.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 1 -- 172.21.15.189:0/421971318 wait complete. 2024-09-16T14:42:18.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 10 monclient: build_initial_monmap 2024-09-16T14:42:18.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 10 monclient: monmap: 2024-09-16T14:42:18.356 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T14:42:18.356 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:18.356 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-09-16T14:42:18.356 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-09-16T14:42:18.356 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:18.356 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-09-16T14:42:18.357 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:18.357 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:18.357 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:18.357 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 1 Processor -- start 2024-09-16T14:42:18.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 1 -- start start 2024-09-16T14:42:18.357 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 10 monclient: init 2024-09-16T14:42:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 10 monclient(hunting): picked mon.noname-c con 0x7fc624099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 10 monclient(hunting): picked mon.noname-b con 0x7fc6240a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 10 monclient(hunting): picked mon.noname-a con 0x7fc6240a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc62413dbe0 con 0x7fc6240a26f0 2024-09-16T14:42:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc62413e7b0 con 0x7fc6240a1c00 2024-09-16T14:42:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc62411ee10 con 0x7fc624099b60 2024-09-16T14:42:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 10 monclient(hunting): authenticate will time out at 920.466874s 2024-09-16T14:42:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc6237fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc6240a1c00 0x7fc62413c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39228/0 (socket says 172.21.15.189:39228) 2024-09-16T14:42:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc6237fe640 1 -- 172.21.15.189:0/4115493961 learned_addr learned my addr 172.21.15.189:0/4115493961 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:42:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855130827 0 0) 0x7fc62413e7b0 con 0x7fc6240a1c00 2024-09-16T14:42:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient(hunting): my global_id is 4123 2024-09-16T14:42:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc610003530 con 0x7fc6240a1c00 2024-09-16T14:42:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3116362563 0 0) 0x7fc610003530 con 0x7fc6240a1c00 2024-09-16T14:42:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc62413e7b0 con 0x7fc6240a1c00 2024-09-16T14:42:18.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc6180015e0 con 0x7fc6240a1c00 2024-09-16T14:42:18.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:18.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:18.362 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.362 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.362 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.362 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.362 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.362 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.362 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.362 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.363 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.363 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:18.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291921689 0 0) 0x7fc62411ee10 con 0x7fc624099b60 2024-09-16T14:42:18.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:18.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:18.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:18.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient(hunting): my global_id is 4127 2024-09-16T14:42:18.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc610003530 con 0x7fc624099b60 2024-09-16T14:42:18.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1322706956 0 0) 0x7fc62413e7b0 con 0x7fc6240a1c00 2024-09-16T14:42:18.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 >> v1:172.21.15.26:6790/0 conn(0x7fc624099b60 legacy=0x7fc62413c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 >> v1:172.21.15.26:6789/0 conn(0x7fc6240a26f0 legacy=0x7fc6241d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:42:18.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 1 monclient: found mon.b 2024-09-16T14:42:18.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:18.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62409e3f0 con 0x7fc6240a1c00 2024-09-16T14:42:18.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient: _finish_auth 0 2024-09-16T14:42:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:42:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 5 monclient: authenticate success, global_id 4123 2024-09-16T14:42:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 10 monclient: _renew_subs 2024-09-16T14:42:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 1 -- 172.21.15.189:0/4115493961 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc62409f270 con 0x7fc6240a1c00 2024-09-16T14:42:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 10 monclient: _renew_subs 2024-09-16T14:42:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc62a326640 1 -- 172.21.15.189:0/4115493961 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc62411eb30 con 0x7fc6240a1c00 2024-09-16T14:42:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc618004c80 con 0x7fc6240a1c00 2024-09-16T14:42:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc618004f90 con 0x7fc6240a1c00 2024-09-16T14:42:18.365 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc621ffb640 4 set_mon_vals no callback set 2024-09-16T14:42:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 10 monclient: dump: 2024-09-16T14:42:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.366 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.367 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.367 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.367 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.367 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fc618005470 con 0x7fc6240a1c00 2024-09-16T14:42:18.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.349+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2685980870 0 0) 0x7fc618005720 con 0x7fc6240a1c00 2024-09-16T14:42:18.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.353+0000 7fc62a326640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:18.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.353+0000 7fc62a326640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:18.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.353+0000 7fc62a326640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.353+0000 7fc62a326640 1 -- 172.21.15.189:0/4115493961 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc604005050 con 0x7fc6240a1c00 2024-09-16T14:42:18.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.353+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7fc6180064c0 con 0x7fc6240a1c00 2024-09-16T14:42:18.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.353+0000 7fc620ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:18.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.353+0000 7fc620ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:18.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.421+0000 7fc62a326640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3030f43d-f848-4189-be1e-6ea367941218", "id": 2}] 2024-09-16T14:42:18.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.421+0000 7fc62a326640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3030f43d-f848-4189-be1e-6ea367941218", "id": 2}] 2024-09-16T14:42:18.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.421+0000 7fc62a326640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.421+0000 7fc62a326640 1 -- 172.21.15.189:0/4115493961 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3030f43d-f848-4189-be1e-6ea367941218", "id": 2} v 0) v1 -- 0x7fc604005880 con 0x7fc6240a1c00 2024-09-16T14:42:18.429 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:18.429+0000 7f9d7ec72180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T14:42:18.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.433+0000 7fc620ff9640 1 -- 172.21.15.189:0/4115493961 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3030f43d-f848-4189-be1e-6ea367941218", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3577061820 0 216599125) 0x7fc618006730 con 0x7fc6240a1c00 2024-09-16T14:42:18.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.433+0000 7fc620ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3030f43d-f848-4189-be1e-6ea367941218", "id": 2}] 2024-09-16T14:42:18.433 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.433+0000 7fc620ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:18.433 INFO:teuthology.orchestra.run.smithi189.stdout:2 2024-09-16T14:42:18.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.433+0000 7fc62a326640 10 monclient: shutdown 2024-09-16T14:42:18.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.433+0000 7fc62a326640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:18.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.433+0000 7fc62a326640 1 -- 172.21.15.189:0/4115493961 >> v1:172.21.15.189:6789/0 conn(0x7fc6240a1c00 legacy=0x7fc62413c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.433+0000 7fc62a326640 1 -- 172.21.15.189:0/4115493961 shutdown_connections 2024-09-16T14:42:18.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.433+0000 7fc62a326640 1 -- 172.21.15.189:0/4115493961 >> 172.21.15.189:0/4115493961 conn(0x7fc6240a3d00 msgr2=0x7fc62406bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:18.434 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.433+0000 7fc62a326640 1 -- 172.21.15.189:0/4115493961 shutdown_connections 2024-09-16T14:42:18.435 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.433+0000 7fc62a326640 1 -- 172.21.15.189:0/4115493961 wait complete. 2024-09-16T14:42:18.449 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 78731cf9-4ba7-4ce9-a483-080aa1136b65 3 2024-09-16T14:42:18.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 10 monclient: get_monmap_and_config 2024-09-16T14:42:18.557 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 10 monclient: build_initial_monmap 2024-09-16T14:42:18.558 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 10 monclient: monmap: 2024-09-16T14:42:18.558 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T14:42:18.558 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.558 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:42:18.558400+0000 2024-09-16T14:42:18.558 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:42:18.558400+0000 2024-09-16T14:42:18.558 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:18.558 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.558 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:18.558 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:18.558 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:18.558 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 1 Processor -- start 2024-09-16T14:42:18.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 1 -- start start 2024-09-16T14:42:18.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 10 monclient: init 2024-09-16T14:42:18.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:18.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:18.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 10 monclient(hunting): picked mon.noname-b con 0x7ff3b80a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:18.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 10 monclient(hunting): picked mon.noname-c con 0x7ff3b8099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:18.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 10 monclient(hunting): picked mon.noname-a con 0x7ff3b80a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:18.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b8098ea0 con 0x7ff3b80a26f0 2024-09-16T14:42:18.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b80997b0 con 0x7ff3b8099b60 2024-09-16T14:42:18.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b8099460 con 0x7ff3b80a1c00 2024-09-16T14:42:18.566 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:18.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 10 monclient(hunting): authenticate will time out at 920.674924s 2024-09-16T14:42:18.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b7fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff3b8099b60 0x7ff3b8139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39230/0 (socket says 172.21.15.189:39230) 2024-09-16T14:42:18.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b7fff640 1 -- 172.21.15.189:0/2619022266 learned_addr learned my addr 172.21.15.189:0/2619022266 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:42:18.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 1 -- 172.21.15.189:0/2619022266 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812489613 0 0) 0x7ff3b80997b0 con 0x7ff3b8099b60 2024-09-16T14:42:18.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:18.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:18.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:18.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 10 monclient(hunting): my global_id is 4126 2024-09-16T14:42:18.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 1 -- 172.21.15.189:0/2619022266 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a8003530 con 0x7ff3b8099b60 2024-09-16T14:42:18.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 1 -- 172.21.15.189:0/2619022266 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3005503291 0 0) 0x7ff3a8003530 con 0x7ff3b8099b60 2024-09-16T14:42:18.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 1 -- 172.21.15.189:0/2619022266 >> v1:172.21.15.26:6790/0 conn(0x7ff3b80a1c00 legacy=0x7ff3b80a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 1 -- 172.21.15.189:0/2619022266 >> v1:172.21.15.26:6789/0 conn(0x7ff3b80a26f0 legacy=0x7ff3b813d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:42:18.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 1 monclient: found mon.noname-c 2024-09-16T14:42:18.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:18.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 1 -- 172.21.15.189:0/2619022266 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b80a34b0 con 0x7ff3b8099b60 2024-09-16T14:42:18.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 10 monclient: _finish_auth 0 2024-09-16T14:42:18.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:42:18.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:18.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 1 -- 172.21.15.189:0/2619022266 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff3a4001620 con 0x7ff3b8099b60 2024-09-16T14:42:18.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 5 monclient: authenticate success, global_id 4126 2024-09-16T14:42:18.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:18.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 10 monclient: dump: 2024-09-16T14:42:18.569 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.569 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.569 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.569 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.569 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.569 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.569 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.569 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.569 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.569 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:18.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 1 -- 172.21.15.189:0/2619022266 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3a40018d0 con 0x7ff3b8099b60 2024-09-16T14:42:18.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:18.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 1 -- 172.21.15.189:0/2619022266 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff3a4001ca0 con 0x7ff3b8099b60 2024-09-16T14:42:18.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:18.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3bf058640 4 set_mon_vals no callback set 2024-09-16T14:42:18.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:18.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.557+0000 7ff3b77fe640 10 monclient: dump: 2024-09-16T14:42:18.570 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.570 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.570 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.570 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.570 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.570 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.570 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.571 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.571 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.571 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient: shutdown 2024-09-16T14:42:18.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:18.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 1 -- 172.21.15.189:0/2619022266 >> v1:172.21.15.189:6789/0 conn(0x7ff3b8099b60 legacy=0x7ff3b8139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 1 -- 172.21.15.189:0/2619022266 shutdown_connections 2024-09-16T14:42:18.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 1 -- 172.21.15.189:0/2619022266 >> 172.21.15.189:0/2619022266 conn(0x7ff3b80a3d00 msgr2=0x7ff3b80a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:18.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 1 -- 172.21.15.189:0/2619022266 shutdown_connections 2024-09-16T14:42:18.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 1 -- 172.21.15.189:0/2619022266 wait complete. 2024-09-16T14:42:18.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient: build_initial_monmap 2024-09-16T14:42:18.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient: monmap: 2024-09-16T14:42:18.572 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T14:42:18.572 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:18.572 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-09-16T14:42:18.572 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-09-16T14:42:18.572 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:18.572 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-09-16T14:42:18.572 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:18.572 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:18.572 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:18.572 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 1 Processor -- start 2024-09-16T14:42:18.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 1 -- start start 2024-09-16T14:42:18.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient: init 2024-09-16T14:42:18.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:18.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:18.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient(hunting): picked mon.noname-a con 0x7ff3b8099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:18.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient(hunting): picked mon.noname-c con 0x7ff3b80a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:18.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient(hunting): picked mon.noname-b con 0x7ff3b80a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:18.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b813daf0 con 0x7ff3b8099b60 2024-09-16T14:42:18.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b811ec50 con 0x7ff3b80a26f0 2024-09-16T14:42:18.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b809f290 con 0x7ff3b80a1c00 2024-09-16T14:42:18.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:18.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient(hunting): authenticate will time out at 920.677937s 2024-09-16T14:42:18.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bd5f5640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff3b80a26f0 0x7ff3b81d19c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39244/0 (socket says 172.21.15.189:39244) 2024-09-16T14:42:18.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bd5f5640 1 -- 172.21.15.189:0/4015966333 learned_addr learned my addr 172.21.15.189:0/4015966333 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:42:18.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750080924 0 0) 0x7ff3b811ec50 con 0x7ff3b80a26f0 2024-09-16T14:42:18.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:18.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:18.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:18.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient(hunting): my global_id is 4129 2024-09-16T14:42:18.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff39c003530 con 0x7ff3b80a26f0 2024-09-16T14:42:18.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2032476494 0 0) 0x7ff39c003530 con 0x7ff3b80a26f0 2024-09-16T14:42:18.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3b811ec50 con 0x7ff3b80a26f0 2024-09-16T14:42:18.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff3b0001c80 con 0x7ff3b80a26f0 2024-09-16T14:42:18.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:18.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient(hunting): dump: 2024-09-16T14:42:18.575 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.575 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.575 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.575 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.575 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.575 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.575 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.575 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.575 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.575 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:18.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434203953 0 0) 0x7ff3b809f290 con 0x7ff3b80a1c00 2024-09-16T14:42:18.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:18.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:18.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:18.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient(hunting): my global_id is 4133 2024-09-16T14:42:18.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff39c003530 con 0x7ff3b80a1c00 2024-09-16T14:42:18.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540917078 0 0) 0x7ff3b813daf0 con 0x7ff3b8099b60 2024-09-16T14:42:18.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:18.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:18.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:18.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient(hunting): my global_id is 4131 2024-09-16T14:42:18.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b809f290 con 0x7ff3b8099b60 2024-09-16T14:42:18.576 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1134559608 0 0) 0x7ff3b811ec50 con 0x7ff3b80a26f0 2024-09-16T14:42:18.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 >> v1:172.21.15.26:6790/0 conn(0x7ff3b80a1c00 legacy=0x7ff3b813c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 >> v1:172.21.15.26:6789/0 conn(0x7ff3b8099b60 legacy=0x7ff3b813c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient: _finish_hunting 0 2024-09-16T14:42:18.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 monclient: found mon.b 2024-09-16T14:42:18.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:18.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b811e690 con 0x7ff3b80a26f0 2024-09-16T14:42:18.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient: _finish_auth 0 2024-09-16T14:42:18.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient: _check_auth_tickets 2024-09-16T14:42:18.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:18.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 5 monclient: authenticate success, global_id 4129 2024-09-16T14:42:18.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient: _renew_subs 2024-09-16T14:42:18.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 1 -- 172.21.15.189:0/4015966333 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b811e970 con 0x7ff3b80a26f0 2024-09-16T14:42:18.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient: _renew_subs 2024-09-16T14:42:18.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 1 -- 172.21.15.189:0/4015966333 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b813e410 con 0x7ff3b80a26f0 2024-09-16T14:42:18.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3b0002050 con 0x7ff3b80a26f0 2024-09-16T14:42:18.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:18.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff3b0003e70 con 0x7ff3b80a26f0 2024-09-16T14:42:18.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:18.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b6ffd640 4 set_mon_vals no callback set 2024-09-16T14:42:18.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 10 monclient: dump: 2024-09-16T14:42:18.578 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.578 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.578 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.579 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.579 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.579 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.579 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.579 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.579 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.579 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7ff3b0004350 con 0x7ff3b80a26f0 2024-09-16T14:42:18.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 697874094 0 0) 0x7ff3b0006890 con 0x7ff3b80a26f0 2024-09-16T14:42:18.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:18.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:18.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.579 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.561+0000 7ff3bf058640 1 -- 172.21.15.189:0/4015966333 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff398005050 con 0x7ff3b80a26f0 2024-09-16T14:42:18.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.565+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7ff3b0006400 con 0x7ff3b80a26f0 2024-09-16T14:42:18.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.565+0000 7ff3b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:18.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.565+0000 7ff3b57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:18.616 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:18.613+0000 7f9d7ec72180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T14:42:18.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.633+0000 7ff3bf058640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "78731cf9-4ba7-4ce9-a483-080aa1136b65", "id": 3}] 2024-09-16T14:42:18.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.633+0000 7ff3bf058640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "78731cf9-4ba7-4ce9-a483-080aa1136b65", "id": 3}] 2024-09-16T14:42:18.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.633+0000 7ff3bf058640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.633+0000 7ff3bf058640 1 -- 172.21.15.189:0/4015966333 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "78731cf9-4ba7-4ce9-a483-080aa1136b65", "id": 3} v 0) v1 -- 0x7ff398005300 con 0x7ff3b80a26f0 2024-09-16T14:42:18.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.641+0000 7ff3b57fa640 1 -- 172.21.15.189:0/4015966333 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "78731cf9-4ba7-4ce9-a483-080aa1136b65", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 785832593 0 525046306) 0x7ff3b00065a0 con 0x7ff3b80a26f0 2024-09-16T14:42:18.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.641+0000 7ff3b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "78731cf9-4ba7-4ce9-a483-080aa1136b65", "id": 3}] 2024-09-16T14:42:18.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.641+0000 7ff3b57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:18.645 INFO:teuthology.orchestra.run.smithi189.stdout:3 2024-09-16T14:42:18.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.645+0000 7ff3bf058640 10 monclient: shutdown 2024-09-16T14:42:18.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.645+0000 7ff3bf058640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:18.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.645+0000 7ff3bf058640 1 -- 172.21.15.189:0/4015966333 >> v1:172.21.15.189:6789/0 conn(0x7ff3b80a26f0 legacy=0x7ff3b81d19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.645+0000 7ff3bf058640 1 -- 172.21.15.189:0/4015966333 shutdown_connections 2024-09-16T14:42:18.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.645+0000 7ff3bf058640 1 -- 172.21.15.189:0/4015966333 >> 172.21.15.189:0/4015966333 conn(0x7ff3b80a3d00 msgr2=0x7ff3b80a5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:18.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.645+0000 7ff3bf058640 1 -- 172.21.15.189:0/4015966333 shutdown_connections 2024-09-16T14:42:18.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.645+0000 7ff3bf058640 1 -- 172.21.15.189:0/4015966333 wait complete. 2024-09-16T14:42:18.661 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new d74156df-be8d-4ad7-93dc-89ac28df70dd 4 2024-09-16T14:42:18.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.769+0000 7f01194bc640 10 monclient: get_monmap_and_config 2024-09-16T14:42:18.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.769+0000 7f01194bc640 10 monclient: build_initial_monmap 2024-09-16T14:42:18.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.769+0000 7f01194bc640 10 monclient: monmap: 2024-09-16T14:42:18.773 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T14:42:18.773 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.773 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:42:18.772828+0000 2024-09-16T14:42:18.773 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:42:18.772828+0000 2024-09-16T14:42:18.773 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:18.773 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.773 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:18.773 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:18.773 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:18.773 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.769+0000 7f01194bc640 1 Processor -- start 2024-09-16T14:42:18.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.769+0000 7f01194bc640 1 -- start start 2024-09-16T14:42:18.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.769+0000 7f01194bc640 10 monclient: init 2024-09-16T14:42:18.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:18.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:18.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient(hunting): picked mon.noname-c con 0x7f01140a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:18.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient(hunting): picked mon.noname-b con 0x7f0114099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:18.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient(hunting): picked mon.noname-a con 0x7f01140a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:18.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0114098ea0 con 0x7f01140a26f0 2024-09-16T14:42:18.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01140997b0 con 0x7f01140a1c00 2024-09-16T14:42:18.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0114099460 con 0x7f0114099b60 2024-09-16T14:42:18.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:18.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient(hunting): authenticate will time out at 920.888457s 2024-09-16T14:42:18.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0112ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f01140a1c00 0x7f01140a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39252/0 (socket says 172.21.15.189:39252) 2024-09-16T14:42:18.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0112ffd640 1 -- 172.21.15.189:0/1119685839 learned_addr learned my addr 172.21.15.189:0/1119685839 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:42:18.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 1 -- 172.21.15.189:0/1119685839 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130490894 0 0) 0x7f01140997b0 con 0x7f01140a1c00 2024-09-16T14:42:18.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:18.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:18.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:18.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 10 monclient(hunting): my global_id is 4132 2024-09-16T14:42:18.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 1 -- 172.21.15.189:0/1119685839 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00fc003530 con 0x7f01140a1c00 2024-09-16T14:42:18.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 1 -- 172.21.15.189:0/1119685839 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4081716726 0 0) 0x7f00fc003530 con 0x7f01140a1c00 2024-09-16T14:42:18.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 1 -- 172.21.15.189:0/1119685839 >> v1:172.21.15.26:6790/0 conn(0x7f0114099b60 legacy=0x7f0114139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 1 -- 172.21.15.189:0/1119685839 >> v1:172.21.15.26:6789/0 conn(0x7f01140a26f0 legacy=0x7f011413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:42:18.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 1 monclient: found mon.noname-c 2024-09-16T14:42:18.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:18.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 1 -- 172.21.15.189:0/1119685839 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01140a34b0 con 0x7f01140a1c00 2024-09-16T14:42:18.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 10 monclient: _finish_auth 0 2024-09-16T14:42:18.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:42:18.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:18.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 5 monclient: authenticate success, global_id 4132 2024-09-16T14:42:18.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 1 -- 172.21.15.189:0/1119685839 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0104001620 con 0x7f01140a1c00 2024-09-16T14:42:18.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:18.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 10 monclient: dump: 2024-09-16T14:42:18.780 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.781 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.781 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.781 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.781 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.781 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.781 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.781 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.781 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.781 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:18.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 1 -- 172.21.15.189:0/1119685839 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0104001940 con 0x7f01140a1c00 2024-09-16T14:42:18.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:18.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:18.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 4 set_mon_vals no callback set 2024-09-16T14:42:18.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 1 -- 172.21.15.189:0/1119685839 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0104001cd0 con 0x7f01140a1c00 2024-09-16T14:42:18.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:18.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f0111ffb640 10 monclient: dump: 2024-09-16T14:42:18.782 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.782 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.782 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.782 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.782 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.782 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.782 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.783 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.783 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.783 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient: shutdown 2024-09-16T14:42:18.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:18.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 1 -- 172.21.15.189:0/1119685839 >> v1:172.21.15.189:6789/0 conn(0x7f01140a1c00 legacy=0x7f01140a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 1 -- 172.21.15.189:0/1119685839 shutdown_connections 2024-09-16T14:42:18.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 1 -- 172.21.15.189:0/1119685839 >> 172.21.15.189:0/1119685839 conn(0x7f01140a3d00 msgr2=0x7f01140a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:18.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 1 -- 172.21.15.189:0/1119685839 shutdown_connections 2024-09-16T14:42:18.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 1 -- 172.21.15.189:0/1119685839 wait complete. 2024-09-16T14:42:18.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient: build_initial_monmap 2024-09-16T14:42:18.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient: monmap: 2024-09-16T14:42:18.784 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T14:42:18.784 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:18.784 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-09-16T14:42:18.784 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-09-16T14:42:18.784 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:18.784 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-09-16T14:42:18.784 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:18.784 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:18.784 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:18.784 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 1 Processor -- start 2024-09-16T14:42:18.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 1 -- start start 2024-09-16T14:42:18.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient: init 2024-09-16T14:42:18.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:18.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:42:18.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient(hunting): picked mon.noname-c con 0x7f0114099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:18.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient(hunting): picked mon.noname-a con 0x7f01140a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:18.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient(hunting): picked mon.noname-b con 0x7f01140a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:18.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011409ec40 con 0x7f01140a1c00 2024-09-16T14:42:18.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011411d920 con 0x7f01140a26f0 2024-09-16T14:42:18.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011409e3f0 con 0x7f0114099b60 2024-09-16T14:42:18.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:18.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 10 monclient(hunting): authenticate will time out at 920.891265s 2024-09-16T14:42:18.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01137fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f01140a26f0 0x7f011413dd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39260/0 (socket says 172.21.15.189:39260) 2024-09-16T14:42:18.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01137fe640 1 -- 172.21.15.189:0/1339171789 learned_addr learned my addr 172.21.15.189:0/1339171789 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:42:18.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 1 -- 172.21.15.189:0/1339171789 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027079609 0 0) 0x7f011411d920 con 0x7f01140a26f0 2024-09-16T14:42:18.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:18.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:18.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:18.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 10 monclient(hunting): my global_id is 4135 2024-09-16T14:42:18.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 1 -- 172.21.15.189:0/1339171789 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0100003530 con 0x7f01140a26f0 2024-09-16T14:42:18.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 1 -- 172.21.15.189:0/1339171789 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2054525375 0 0) 0x7f0100003530 con 0x7f01140a26f0 2024-09-16T14:42:18.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 1 -- 172.21.15.189:0/1339171789 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f011411d920 con 0x7f01140a26f0 2024-09-16T14:42:18.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 1 -- 172.21.15.189:0/1339171789 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f010c001c80 con 0x7f01140a26f0 2024-09-16T14:42:18.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:18.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 10 monclient(hunting): dump: 2024-09-16T14:42:18.787 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.787 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.787 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.787 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.787 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.787 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.787 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.787 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.787 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.787 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:18.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 1 -- 172.21.15.189:0/1339171789 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 60432836 0 0) 0x7f011411d920 con 0x7f01140a26f0 2024-09-16T14:42:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 1 -- 172.21.15.189:0/1339171789 >> v1:172.21.15.26:6790/0 conn(0x7f0114099b60 legacy=0x7f011413cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 1 -- 172.21.15.189:0/1339171789 >> v1:172.21.15.26:6789/0 conn(0x7f01140a1c00 legacy=0x7f011413d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:42:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 1 monclient: found mon.b 2024-09-16T14:42:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 1 -- 172.21.15.189:0/1339171789 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011409f190 con 0x7f01140a26f0 2024-09-16T14:42:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 10 monclient: _finish_auth 0 2024-09-16T14:42:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:42:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f00fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.773+0000 7f01194bc640 5 monclient: authenticate success, global_id 4135 2024-09-16T14:42:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f01194bc640 10 monclient: _renew_subs 2024-09-16T14:42:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f01194bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f01194bc640 1 -- 172.21.15.189:0/1339171789 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f011411fc60 con 0x7f01140a26f0 2024-09-16T14:42:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f01194bc640 10 monclient: _renew_subs 2024-09-16T14:42:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f01194bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f01194bc640 1 -- 172.21.15.189:0/1339171789 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f011411f980 con 0x7f01140a26f0 2024-09-16T14:42:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f00fb7fe640 1 -- 172.21.15.189:0/1339171789 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f010c002050 con 0x7f01140a26f0 2024-09-16T14:42:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f00fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f00fb7fe640 1 -- 172.21.15.189:0/1339171789 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f010c003df0 con 0x7f01140a26f0 2024-09-16T14:42:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f00fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f00fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f01117fa640 4 set_mon_vals no callback set 2024-09-16T14:42:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f00fb7fe640 10 monclient: dump: 2024-09-16T14:42:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.790 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.790 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.790 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.790 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.790 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.790 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.790 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.790 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f00fb7fe640 1 -- 172.21.15.189:0/1339171789 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7f010c0042d0 con 0x7f01140a26f0 2024-09-16T14:42:18.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f00fb7fe640 1 -- 172.21.15.189:0/1339171789 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2397568474 0 0) 0x7f010c004580 con 0x7f01140a26f0 2024-09-16T14:42:18.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f01194bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:18.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f01194bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:18.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f01194bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f01194bc640 1 -- 172.21.15.189:0/1339171789 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f4005050 con 0x7f01140a26f0 2024-09-16T14:42:18.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f00fb7fe640 1 -- 172.21.15.189:0/1339171789 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7f010c005b50 con 0x7f01140a26f0 2024-09-16T14:42:18.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f00fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:18.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.777+0000 7f00fb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:18.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.841+0000 7f01194bc640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d74156df-be8d-4ad7-93dc-89ac28df70dd", "id": 4}] 2024-09-16T14:42:18.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.841+0000 7f01194bc640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d74156df-be8d-4ad7-93dc-89ac28df70dd", "id": 4}] 2024-09-16T14:42:18.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.841+0000 7f01194bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.841+0000 7f01194bc640 1 -- 172.21.15.189:0/1339171789 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d74156df-be8d-4ad7-93dc-89ac28df70dd", "id": 4} v 0) v1 -- 0x7f00f4005300 con 0x7f01140a26f0 2024-09-16T14:42:18.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.849+0000 7f00fb7fe640 1 -- 172.21.15.189:0/1339171789 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d74156df-be8d-4ad7-93dc-89ac28df70dd", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1028341901 0 1697012583) 0x7f010c005d70 con 0x7f01140a26f0 2024-09-16T14:42:18.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.849+0000 7f00fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d74156df-be8d-4ad7-93dc-89ac28df70dd", "id": 4}] 2024-09-16T14:42:18.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.849+0000 7f00fb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:18.851 INFO:teuthology.orchestra.run.smithi189.stdout:4 2024-09-16T14:42:18.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.849+0000 7f01194bc640 10 monclient: shutdown 2024-09-16T14:42:18.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.849+0000 7f01194bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:18.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.849+0000 7f01194bc640 1 -- 172.21.15.189:0/1339171789 >> v1:172.21.15.189:6789/0 conn(0x7f01140a26f0 legacy=0x7f011413dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.849+0000 7f01194bc640 1 -- 172.21.15.189:0/1339171789 shutdown_connections 2024-09-16T14:42:18.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.849+0000 7f01194bc640 1 -- 172.21.15.189:0/1339171789 >> 172.21.15.189:0/1339171789 conn(0x7f01140a3d00 msgr2=0x7f01140a5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:18.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.849+0000 7f01194bc640 1 -- 172.21.15.189:0/1339171789 shutdown_connections 2024-09-16T14:42:18.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.849+0000 7f01194bc640 1 -- 172.21.15.189:0/1339171789 wait complete. 2024-09-16T14:42:18.867 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new ccad9c1e-247f-4ed4-8437-6aa5c8b515d1 5 2024-09-16T14:42:18.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.977+0000 7fdcae6f3640 10 monclient: get_monmap_and_config 2024-09-16T14:42:18.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.977+0000 7fdcae6f3640 10 monclient: build_initial_monmap 2024-09-16T14:42:18.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.977+0000 7fdcae6f3640 10 monclient: monmap: 2024-09-16T14:42:18.980 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T14:42:18.980 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.981 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:42:18.981010+0000 2024-09-16T14:42:18.981 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:42:18.981010+0000 2024-09-16T14:42:18.981 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:18.981 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.981 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:18.981 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:18.981 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:18.981 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.977+0000 7fdcae6f3640 1 Processor -- start 2024-09-16T14:42:18.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.977+0000 7fdcae6f3640 1 -- start start 2024-09-16T14:42:18.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.977+0000 7fdcae6f3640 10 monclient: init 2024-09-16T14:42:18.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:18.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:18.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient(hunting): picked mon.noname-c con 0x7fdca80a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:18.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient(hunting): picked mon.noname-b con 0x7fdca8099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:18.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient(hunting): picked mon.noname-a con 0x7fdca80a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:18.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca8098ea0 con 0x7fdca80a26f0 2024-09-16T14:42:18.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca80997b0 con 0x7fdca80a1c00 2024-09-16T14:42:18.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca8099460 con 0x7fdca8099b60 2024-09-16T14:42:18.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:18.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient(hunting): authenticate will time out at 921.096634s 2024-09-16T14:42:18.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca7fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdca80a1c00 0x7fdca80a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60636/0 (socket says 172.21.15.189:60636) 2024-09-16T14:42:18.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca7fff640 1 -- 172.21.15.189:0/146379258 learned_addr learned my addr 172.21.15.189:0/146379258 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:42:18.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 1 -- 172.21.15.189:0/146379258 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891486603 0 0) 0x7fdca80997b0 con 0x7fdca80a1c00 2024-09-16T14:42:18.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:18.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:18.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:18.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 10 monclient(hunting): my global_id is 4138 2024-09-16T14:42:18.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 1 -- 172.21.15.189:0/146379258 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc90003530 con 0x7fdca80a1c00 2024-09-16T14:42:18.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 1 -- 172.21.15.189:0/146379258 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2961024515 0 0) 0x7fdc90003530 con 0x7fdca80a1c00 2024-09-16T14:42:18.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 1 -- 172.21.15.189:0/146379258 >> v1:172.21.15.26:6790/0 conn(0x7fdca8099b60 legacy=0x7fdca8139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 1 -- 172.21.15.189:0/146379258 >> v1:172.21.15.26:6789/0 conn(0x7fdca80a26f0 legacy=0x7fdca813d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:18.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 1 monclient: found mon.noname-c 2024-09-16T14:42:18.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:18.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:42:18.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 1 -- 172.21.15.189:0/146379258 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca80a34b0 con 0x7fdca80a1c00 2024-09-16T14:42:18.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:18.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:18.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:18.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 1 -- 172.21.15.189:0/146379258 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdc98001620 con 0x7fdca80a1c00 2024-09-16T14:42:18.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:18.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 10 monclient: dump: 2024-09-16T14:42:18.993 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.993 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.993 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.993 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.993 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.993 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.993 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.993 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.993 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.993 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 5 monclient: authenticate success, global_id 4138 2024-09-16T14:42:18.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:18.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 1 -- 172.21.15.189:0/146379258 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc98001940 con 0x7fdca80a1c00 2024-09-16T14:42:18.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:18.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:18.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 1 -- 172.21.15.189:0/146379258 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdc98001cd0 con 0x7fdca80a1c00 2024-09-16T14:42:18.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 4 set_mon_vals no callback set 2024-09-16T14:42:18.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:18.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca6ffd640 10 monclient: dump: 2024-09-16T14:42:18.994 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.994 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.994 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.994 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.994 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.994 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.995 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.995 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.995 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.995 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient: shutdown 2024-09-16T14:42:18.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:18.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 1 -- 172.21.15.189:0/146379258 >> v1:172.21.15.189:6789/0 conn(0x7fdca80a1c00 legacy=0x7fdca80a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 1 -- 172.21.15.189:0/146379258 shutdown_connections 2024-09-16T14:42:18.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 1 -- 172.21.15.189:0/146379258 >> 172.21.15.189:0/146379258 conn(0x7fdca80a3d00 msgr2=0x7fdca80a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:18.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 1 -- 172.21.15.189:0/146379258 shutdown_connections 2024-09-16T14:42:18.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 1 -- 172.21.15.189:0/146379258 wait complete. 2024-09-16T14:42:18.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient: build_initial_monmap 2024-09-16T14:42:18.995 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient: monmap: 2024-09-16T14:42:18.996 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T14:42:18.996 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:18.996 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-09-16T14:42:18.996 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-09-16T14:42:18.996 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:18.996 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-09-16T14:42:18.996 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:18.996 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:18.996 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:18.996 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 1 Processor -- start 2024-09-16T14:42:18.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 1 -- start start 2024-09-16T14:42:18.996 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient: init 2024-09-16T14:42:18.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:18.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:42:18.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient(hunting): picked mon.noname-b con 0x7fdca8099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:18.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient(hunting): picked mon.noname-a con 0x7fdca80a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:18.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient(hunting): picked mon.noname-c con 0x7fdca80a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:18.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca80a1500 con 0x7fdca80a1c00 2024-09-16T14:42:18.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca813d460 con 0x7fdca8099b60 2024-09-16T14:42:18.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca813e830 con 0x7fdca80a26f0 2024-09-16T14:42:18.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:18.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdcae6f3640 10 monclient(hunting): authenticate will time out at 921.099659s 2024-09-16T14:42:18.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca7fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdca8099b60 0x7fdca813e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60648/0 (socket says 172.21.15.189:60648) 2024-09-16T14:42:18.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca7fff640 1 -- 172.21.15.189:0/1391901347 learned_addr learned my addr 172.21.15.189:0/1391901347 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:42:18.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca4ff9640 1 -- 172.21.15.189:0/1391901347 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182675206 0 0) 0x7fdca813d460 con 0x7fdca8099b60 2024-09-16T14:42:18.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca4ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:18.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:18.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:18.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca4ff9640 10 monclient(hunting): my global_id is 4141 2024-09-16T14:42:18.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.981+0000 7fdca4ff9640 1 -- 172.21.15.189:0/1391901347 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc94003530 con 0x7fdca8099b60 2024-09-16T14:42:18.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 1 -- 172.21.15.189:0/1391901347 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 428107509 0 0) 0x7fdc94003530 con 0x7fdca8099b60 2024-09-16T14:42:18.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 1 -- 172.21.15.189:0/1391901347 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdca813d460 con 0x7fdca8099b60 2024-09-16T14:42:18.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 1 -- 172.21.15.189:0/1391901347 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdc980015e0 con 0x7fdca8099b60 2024-09-16T14:42:18.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:18.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:18.998 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:18.998 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:18.998 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:18.999 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.999 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:18.999 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:18.999 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:18.999 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:18.999 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:18.999 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:18.999 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:18.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:18.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 1 -- 172.21.15.189:0/1391901347 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3567037329 0 0) 0x7fdca813d460 con 0x7fdca8099b60 2024-09-16T14:42:18.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 1 -- 172.21.15.189:0/1391901347 >> v1:172.21.15.26:6790/0 conn(0x7fdca80a26f0 legacy=0x7fdca81d1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 1 -- 172.21.15.189:0/1391901347 >> v1:172.21.15.26:6789/0 conn(0x7fdca80a1c00 legacy=0x7fdca81cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:18.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:42:18.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 1 monclient: found mon.b 2024-09-16T14:42:18.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:19.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:19.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 1 -- 172.21.15.189:0/1391901347 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca809e3f0 con 0x7fdca8099b60 2024-09-16T14:42:19.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 10 monclient: _finish_auth 0 2024-09-16T14:42:19.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:42:19.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:19.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdcae6f3640 5 monclient: authenticate success, global_id 4141 2024-09-16T14:42:19.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdcae6f3640 10 monclient: _renew_subs 2024-09-16T14:42:19.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdcae6f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:19.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdcae6f3640 1 -- 172.21.15.189:0/1391901347 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca813dd00 con 0x7fdca8099b60 2024-09-16T14:42:19.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdcae6f3640 10 monclient: _renew_subs 2024-09-16T14:42:19.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdcae6f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:19.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdcae6f3640 1 -- 172.21.15.189:0/1391901347 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca813ba00 con 0x7fdca8099b60 2024-09-16T14:42:19.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 1 -- 172.21.15.189:0/1391901347 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc98001200 con 0x7fdca8099b60 2024-09-16T14:42:19.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca67fc640 4 set_mon_vals no callback set 2024-09-16T14:42:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 1 -- 172.21.15.189:0/1391901347 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdc98004c00 con 0x7fdca8099b60 2024-09-16T14:42:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 10 monclient: dump: 2024-09-16T14:42:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T14:42:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T14:42:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:19.002 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T14:42:19.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 1 -- 172.21.15.189:0/1391901347 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fdc98005060 con 0x7fdca8099b60 2024-09-16T14:42:19.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 1 -- 172.21.15.189:0/1391901347 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1447015905 0 0) 0x7fdc98006620 con 0x7fdca8099b60 2024-09-16T14:42:19.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdcae6f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:19.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdcae6f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:19.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdcae6f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:19.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdcae6f3640 1 -- 172.21.15.189:0/1391901347 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc88005050 con 0x7fdca8099b60 2024-09-16T14:42:19.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 1 -- 172.21.15.189:0/1391901347 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7fdc98006090 con 0x7fdca8099b60 2024-09-16T14:42:19.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:19.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:18.985+0000 7fdca4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:19.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:19.049+0000 7fdcae6f3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ccad9c1e-247f-4ed4-8437-6aa5c8b515d1", "id": 5}] 2024-09-16T14:42:19.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:19.049+0000 7fdcae6f3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ccad9c1e-247f-4ed4-8437-6aa5c8b515d1", "id": 5}] 2024-09-16T14:42:19.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:19.049+0000 7fdcae6f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:19.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:19.049+0000 7fdcae6f3640 1 -- 172.21.15.189:0/1391901347 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ccad9c1e-247f-4ed4-8437-6aa5c8b515d1", "id": 5} v 0) v1 -- 0x7fdc88005880 con 0x7fdca8099b60 2024-09-16T14:42:19.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:19.057+0000 7fdca4ff9640 1 -- 172.21.15.189:0/1391901347 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ccad9c1e-247f-4ed4-8437-6aa5c8b515d1", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 985355309 0 1988412176) 0x7fdc98006380 con 0x7fdca8099b60 2024-09-16T14:42:19.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:19.057+0000 7fdca4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ccad9c1e-247f-4ed4-8437-6aa5c8b515d1", "id": 5}] 2024-09-16T14:42:19.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:19.057+0000 7fdca4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:19.060 INFO:teuthology.orchestra.run.smithi189.stdout:5 2024-09-16T14:42:19.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:19.057+0000 7fdcae6f3640 10 monclient: shutdown 2024-09-16T14:42:19.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:19.057+0000 7fdcae6f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:19.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:19.057+0000 7fdcae6f3640 1 -- 172.21.15.189:0/1391901347 >> v1:172.21.15.189:6789/0 conn(0x7fdca8099b60 legacy=0x7fdca813e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:19.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:19.061+0000 7fdcae6f3640 1 -- 172.21.15.189:0/1391901347 shutdown_connections 2024-09-16T14:42:19.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:19.061+0000 7fdcae6f3640 1 -- 172.21.15.189:0/1391901347 >> 172.21.15.189:0/1391901347 conn(0x7fdca80a3d00 msgr2=0x7fdca809f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:19.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:19.061+0000 7fdcae6f3640 1 -- 172.21.15.189:0/1391901347 shutdown_connections 2024-09-16T14:42:19.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T14:42:19.061+0000 7fdcae6f3640 1 -- 172.21.15.189:0/1391901347 wait complete. 2024-09-16T14:42:19.076 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T14:42:19.076 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T14:42:19.080 INFO:tasks.ceph.osd.0:Started 2024-09-16T14:42:19.080 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T14:42:19.080 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T14:42:19.082 INFO:tasks.ceph.osd.1:Started 2024-09-16T14:42:19.083 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T14:42:19.083 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T14:42:19.085 INFO:tasks.ceph.osd.2:Started 2024-09-16T14:42:19.085 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T14:42:19.085 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T14:42:19.087 INFO:tasks.ceph.osd.3:Started 2024-09-16T14:42:19.087 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T14:42:19.087 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T14:42:19.089 INFO:tasks.ceph.osd.4:Started 2024-09-16T14:42:19.089 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T14:42:19.089 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T14:42:19.092 INFO:tasks.ceph.osd.5:Started 2024-09-16T14:42:19.092 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T14:42:19.182 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:19.181+0000 7f9d7ec72180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T14:42:19.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 10 monclient: get_monmap_and_config 2024-09-16T14:42:19.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 10 monclient: build_initial_monmap 2024-09-16T14:42:19.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 10 monclient: monmap: 2024-09-16T14:42:19.218 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:19.218 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:19.218 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:19.218332+0000 2024-09-16T14:42:19.218 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:19.218332+0000 2024-09-16T14:42:19.218 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:19.218 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:19.218 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:19.218 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:19.218 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:19.218 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 1 Processor -- start 2024-09-16T14:42:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 1 -- start start 2024-09-16T14:42:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 10 monclient: init 2024-09-16T14:42:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 10 monclient(hunting): picked mon.noname-c con 0x7f25ac0a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 10 monclient(hunting): picked mon.noname-b con 0x7f25ac0a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 10 monclient(hunting): picked mon.noname-a con 0x7f25ac0a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac0a3250 con 0x7f25ac0a2800 2024-09-16T14:42:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac09f600 con 0x7f25ac0a1d10 2024-09-16T14:42:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac0a3830 con 0x7f25ac0a0000 2024-09-16T14:42:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 10 monclient(hunting): authenticate will time out at 963.338089s 2024-09-16T14:42:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b18ab640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f25ac0a2800 0x7f25ac13d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52238/0 (socket says 172.21.15.26:52238) 2024-09-16T14:42:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b08a9640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f25ac0a0000 0x7f25ac139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35714/0 (socket says 172.21.15.26:35714) 2024-09-16T14:42:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b18ab640 1 -- 172.21.15.26:0/2943126309 learned_addr learned my addr 172.21.15.26:0/2943126309 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 1 -- 172.21.15.26:0/2943126309 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203435530 0 0) 0x7f25ac0a3250 con 0x7f25ac0a2800 2024-09-16T14:42:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient(hunting): my global_id is 4164 2024-09-16T14:42:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 1 -- 172.21.15.26:0/2943126309 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2598003530 con 0x7f25ac0a2800 2024-09-16T14:42:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 1 -- 172.21.15.26:0/2943126309 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720480017 0 0) 0x7f25ac0a3830 con 0x7f25ac0a0000 2024-09-16T14:42:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient(hunting): my global_id is 4166 2024-09-16T14:42:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 1 -- 172.21.15.26:0/2943126309 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25ac0a3250 con 0x7f25ac0a0000 2024-09-16T14:42:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 1 -- 172.21.15.26:0/2943126309 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828064820 0 0) 0x7f25ac09f600 con 0x7f25ac0a1d10 2024-09-16T14:42:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient(hunting): my global_id is 4162 2024-09-16T14:42:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 1 -- 172.21.15.26:0/2943126309 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25ac0a3830 con 0x7f25ac0a1d10 2024-09-16T14:42:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 1 -- 172.21.15.26:0/2943126309 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2248182631 0 0) 0x7f2598003530 con 0x7f25ac0a2800 2024-09-16T14:42:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 1 -- 172.21.15.26:0/2943126309 >> v1:172.21.15.26:6790/0 conn(0x7f25ac0a0000 legacy=0x7f25ac139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 1 -- 172.21.15.26:0/2943126309 >> v1:172.21.15.189:6789/0 conn(0x7f25ac0a1d10 legacy=0x7f25ac0a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:42:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 1 monclient: found mon.noname-a 2024-09-16T14:42:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:42:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 1 -- 172.21.15.26:0/2943126309 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac09ed60 con 0x7f25ac0a2800 2024-09-16T14:42:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient: _finish_auth 0 2024-09-16T14:42:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:42:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 5 monclient: authenticate success, global_id 4164 2024-09-16T14:42:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 1 -- 172.21.15.26:0/2943126309 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f25a40019a0 con 0x7f25ac0a2800 2024-09-16T14:42:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient: dump: 2024-09-16T14:42:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:19.229 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 1 -- 172.21.15.26:0/2943126309 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25a4001c80 con 0x7f25ac0a2800 2024-09-16T14:42:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 1 -- 172.21.15.26:0/2943126309 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f25a4002050 con 0x7f25ac0a2800 2024-09-16T14:42:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25a3fff640 10 monclient: dump: 2024-09-16T14:42:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:19.230 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 4 set_mon_vals no callback set 2024-09-16T14:42:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 10 monclient: shutdown 2024-09-16T14:42:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 1 -- 172.21.15.26:0/2943126309 >> v1:172.21.15.26:6789/0 conn(0x7f25ac0a2800 legacy=0x7f25ac13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 1 -- 172.21.15.26:0/2943126309 shutdown_connections 2024-09-16T14:42:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 1 -- 172.21.15.26:0/2943126309 >> 172.21.15.26:0/2943126309 conn(0x7f25ac0a7f30 msgr2=0x7f25ac0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 1 -- 172.21.15.26:0/2943126309 shutdown_connections 2024-09-16T14:42:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.216+0000 7f25b330e640 1 -- 172.21.15.26:0/2943126309 wait complete. 2024-09-16T14:42:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 monclient: build_initial_monmap 2024-09-16T14:42:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 monclient: monmap: 2024-09-16T14:42:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:19.232 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:19.232 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:19.232 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:19.232 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:19.232 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:19.232 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:19.232 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:19.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 1 Processor -- start 2024-09-16T14:42:19.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 1 -- start start 2024-09-16T14:42:19.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 monclient: init 2024-09-16T14:42:19.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:19.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:19.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 monclient(hunting): picked mon.noname-a con 0x7f25ac0a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:19.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 monclient(hunting): picked mon.noname-c con 0x7f25ac0a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:19.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 monclient(hunting): picked mon.noname-b con 0x7f25ac0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:19.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac130d40 con 0x7f25ac0a0000 2024-09-16T14:42:19.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac0a6370 con 0x7f25ac0a2800 2024-09-16T14:42:19.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac0a68a0 con 0x7f25ac0a1d10 2024-09-16T14:42:19.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:19.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 monclient(hunting): authenticate will time out at 963.341112s 2024-09-16T14:42:19.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b10aa640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f25ac0a0000 0x7f25ac12ff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52252/0 (socket says 172.21.15.26:52252) 2024-09-16T14:42:19.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b10aa640 1 -- 172.21.15.26:0/990877474 learned_addr learned my addr 172.21.15.26:0/990877474 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:19.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839618605 0 0) 0x7f25ac130d40 con 0x7f25ac0a0000 2024-09-16T14:42:19.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:19.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:19.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:19.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): my global_id is 4167 2024-09-16T14:42:19.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2590003530 con 0x7f25ac0a0000 2024-09-16T14:42:19.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727391148 0 0) 0x7f25ac0a68a0 con 0x7f25ac0a1d10 2024-09-16T14:42:19.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:19.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:19.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:19.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): my global_id is 4169 2024-09-16T14:42:19.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25ac130d40 con 0x7f25ac0a1d10 2024-09-16T14:42:19.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239973918 0 0) 0x7f25ac0a6370 con 0x7f25ac0a2800 2024-09-16T14:42:19.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:19.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:19.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:19.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): my global_id is 4165 2024-09-16T14:42:19.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25ac0a68a0 con 0x7f25ac0a2800 2024-09-16T14:42:19.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3817032905 0 0) 0x7f2590003530 con 0x7f25ac0a0000 2024-09-16T14:42:19.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25ac0a6370 con 0x7f25ac0a0000 2024-09-16T14:42:19.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f25a80019a0 con 0x7f25ac0a0000 2024-09-16T14:42:19.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:19.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:19.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): dump: 2024-09-16T14:42:19.235 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:19.235 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:19.235 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:19.235 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:19.235 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:19.236 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:19.236 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:19.236 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:19.236 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:19.236 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:19.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:19.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3502605902 0 0) 0x7f25ac130d40 con 0x7f25ac0a1d10 2024-09-16T14:42:19.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2590003530 con 0x7f25ac0a1d10 2024-09-16T14:42:19.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f259c001630 con 0x7f25ac0a1d10 2024-09-16T14:42:19.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:19.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:19.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): dump: 2024-09-16T14:42:19.236 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:19.237 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:19.237 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:19.237 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:19.237 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:19.237 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:19.237 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:19.237 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:19.237 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:19.237 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:19.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2935451462 0 0) 0x7f25ac0a68a0 con 0x7f25ac0a2800 2024-09-16T14:42:19.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25ac130d40 con 0x7f25ac0a2800 2024-09-16T14:42:19.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f25a4001f90 con 0x7f25ac0a2800 2024-09-16T14:42:19.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:19.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient(hunting): dump: 2024-09-16T14:42:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:19.238 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1813654944 0 0) 0x7f2590003530 con 0x7f25ac0a1d10 2024-09-16T14:42:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 >> v1:172.21.15.189:6789/0 conn(0x7f25ac0a2800 legacy=0x7f25ac1d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:19.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 >> v1:172.21.15.26:6789/0 conn(0x7f25ac0a0000 legacy=0x7f25ac12ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:19.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:42:19.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 monclient: found mon.c 2024-09-16T14:42:19.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:19.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:19.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac09e500 con 0x7f25ac0a1d10 2024-09-16T14:42:19.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:42:19.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:42:19.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:19.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 5 monclient: authenticate success, global_id 4169 2024-09-16T14:42:19.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 monclient: _renew_subs 2024-09-16T14:42:19.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:19.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 1 -- 172.21.15.26:0/990877474 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ac133e30 con 0x7f25ac0a1d10 2024-09-16T14:42:19.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 monclient: _renew_subs 2024-09-16T14:42:19.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:19.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f259c001ba0 con 0x7f25ac0a1d10 2024-09-16T14:42:19.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 1 -- 172.21.15.26:0/990877474 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ac134a90 con 0x7f25ac0a1d10 2024-09-16T14:42:19.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:19.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f259c0047f0 con 0x7f25ac0a1d10 2024-09-16T14:42:19.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:19.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:19.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 10 monclient: dump: 2024-09-16T14:42:19.240 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:19.240 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:19.240 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:19.240 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:19.241 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:19.241 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:19.241 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:19.241 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:19.241 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:19.241 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:19.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a37fe640 4 set_mon_vals no callback set 2024-09-16T14:42:19.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7f259c004d10 con 0x7f25ac0a1d10 2024-09-16T14:42:19.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 976301087 0 0) 0x7f259c0055b0 con 0x7f25ac0a1d10 2024-09-16T14:42:19.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:19.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:19.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:19.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.220+0000 7f25b330e640 1 -- 172.21.15.26:0/990877474 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ac0008e0 con 0x7f25ac0a1d10 2024-09-16T14:42:19.242 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:19.233+0000 7f9d7ec72180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T14:42:19.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.224+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7f259c005850 con 0x7f25ac0a1d10 2024-09-16T14:42:19.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.224+0000 7f25a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:19.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.224+0000 7f25a1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:19.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.280+0000 7f25b330e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:19.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.280+0000 7f25b330e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:19.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.280+0000 7f25b330e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:19.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.280+0000 7f25b330e640 1 -- 172.21.15.26:0/990877474 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25ac06c790 con 0x7f25ac0a1d10 2024-09-16T14:42:19.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.280+0000 7f25a1ffb640 1 -- 172.21.15.26:0/990877474 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5604 (unknown 150273500 0 3201663216) 0x7f259c0050d0 con 0x7f25ac0a1d10 2024-09-16T14:42:19.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.280+0000 7f25a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:19.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.280+0000 7f25a1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:19.284 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:42:19.284 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":8,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","created":"2024-09-16T14:42:17.361142+0000","modified":"2024-09-16T14:42:19.053975+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T14:42:19.053975+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a4578188-6e8b-430f-b70d-1c3480eb7bae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a96367f-73f1-4a99-9b85-47bf8e3d4cdc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3030f43d-f848-4189-be1e-6ea367941218","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"78731cf9-4ba7-4ce9-a483-080aa1136b65","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d74156df-be8d-4ad7-93dc-89ac28df70dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ccad9c1e-247f-4ed4-8437-6aa5c8b515d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T14:42:19.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.284+0000 7f25b330e640 10 monclient: shutdown 2024-09-16T14:42:19.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.284+0000 7f25b330e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:19.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.284+0000 7f25b330e640 1 -- 172.21.15.26:0/990877474 >> v1:172.21.15.26:6790/0 conn(0x7f25ac0a1d10 legacy=0x7f25ac130630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:19.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.284+0000 7f25b330e640 1 -- 172.21.15.26:0/990877474 shutdown_connections 2024-09-16T14:42:19.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.284+0000 7f25b330e640 1 -- 172.21.15.26:0/990877474 >> 172.21.15.26:0/990877474 conn(0x7f25ac0a7f30 msgr2=0x7f25ac06b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:19.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.284+0000 7f25b330e640 1 -- 172.21.15.26:0/990877474 shutdown_connections 2024-09-16T14:42:19.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:19.284+0000 7f25b330e640 1 -- 172.21.15.26:0/990877474 wait complete. 2024-09-16T14:42:19.288 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:19.285+0000 7f9d7ec72180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T14:42:19.301 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T14:42:19.301 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T14:42:19.364 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:19.361+0000 7f9d7ec72180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T14:42:19.416 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:19.413+0000 7f9d7ec72180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T14:42:19.493 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:19.493+0000 7f9d7ec72180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T14:42:19.544 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:19.541+0000 7f9d7ec72180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T14:42:19.678 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-09-16T14:42:19.677+0000 7efeea3e07c0 -1 Falling back to public interface 2024-09-16T14:42:19.689 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-09-16T14:42:19.689+0000 7f95327c87c0 -1 Falling back to public interface 2024-09-16T14:42:19.694 INFO:tasks.ceph.osd.3.smithi189.stderr:2024-09-16T14:42:19.693+0000 7f269c06d7c0 -1 Falling back to public interface 2024-09-16T14:42:19.694 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-16T14:42:19.692+0000 7f66916e67c0 -1 Falling back to public interface 2024-09-16T14:42:19.695 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-16T14:42:19.692+0000 7fa7c20aa7c0 -1 Falling back to public interface 2024-09-16T14:42:19.695 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-16T14:42:19.692+0000 7fa7c106b640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6800/0: (98) Address already in use 2024-09-16T14:42:19.695 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:19.693+0000 7f9d7ec72180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T14:42:19.709 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-16T14:42:19.708+0000 7f802ec017c0 -1 Falling back to public interface 2024-09-16T14:42:19.754 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:19.753+0000 7f9d7ec72180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T14:42:19.906 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T14:42:20.027 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.024+0000 7fb33d551640 10 monclient: get_monmap_and_config 2024-09-16T14:42:20.028 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.024+0000 7fb33d551640 10 monclient: build_initial_monmap 2024-09-16T14:42:20.028 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.024+0000 7fb33d551640 10 monclient: monmap: 2024-09-16T14:42:20.028 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-09-16T14:42:20.028 INFO:teuthology.misc.health.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:20.028 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-09-16T14:42:20.028420+0000 2024-09-16T14:42:20.028 INFO:teuthology.misc.health.smithi026.stderr:created 2024-09-16T14:42:20.028420+0000 2024-09-16T14:42:20.028 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:20.028 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:20.028 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:20.028 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:20.028 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:20.028 INFO:teuthology.misc.health.smithi026.stderr: 2024-09-16T14:42:20.035 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.024+0000 7fb33d551640 1 Processor -- start 2024-09-16T14:42:20.035 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.024+0000 7fb33d551640 1 -- start start 2024-09-16T14:42:20.035 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.024+0000 7fb33d551640 10 monclient: init 2024-09-16T14:42:20.035 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:20.035 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:20.035 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient(hunting): picked mon.noname-b con 0x7fb3380a1ca0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:20.035 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient(hunting): picked mon.noname-c con 0x7fb3380a2790 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:20.035 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient(hunting): picked mon.noname-a con 0x7fb3380a2ba0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:20.036 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33809efa0 con 0x7fb3380a2ba0 2024-09-16T14:42:20.036 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33809ecc0 con 0x7fb3380a2790 2024-09-16T14:42:20.036 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33809f280 con 0x7fb3380a1ca0 2024-09-16T14:42:20.036 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:20.036 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient(hunting): authenticate will time out at 964.148161s 2024-09-16T14:42:20.036 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb336ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb3380a1ca0 0x7fb3380a2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35728/0 (socket says 172.21.15.26:35728) 2024-09-16T14:42:20.036 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb336ffd640 1 -- 172.21.15.26:0/2101671274 learned_addr learned my addr 172.21.15.26:0/2101671274 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:20.036 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb3377fe640 1 --1- 172.21.15.26:0/2101671274 >> v1:172.21.15.26:6789/0 conn(0x7fb3380a2ba0 0x7fb33813d5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52256/0 (socket says 172.21.15.26:52256) 2024-09-16T14:42:20.036 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 1 -- 172.21.15.26:0/2101671274 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709933734 0 0) 0x7fb33809f280 con 0x7fb3380a1ca0 2024-09-16T14:42:20.036 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:20.036 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:20.036 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:20.037 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient(hunting): my global_id is 4172 2024-09-16T14:42:20.037 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 1 -- 172.21.15.26:0/2101671274 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb320003530 con 0x7fb3380a1ca0 2024-09-16T14:42:20.037 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 1 -- 172.21.15.26:0/2101671274 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1374608329 0 0) 0x7fb33809efa0 con 0x7fb3380a2ba0 2024-09-16T14:42:20.037 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:20.037 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:20.037 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:20.037 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient(hunting): my global_id is 4170 2024-09-16T14:42:20.037 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 1 -- 172.21.15.26:0/2101671274 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb33809f280 con 0x7fb3380a2ba0 2024-09-16T14:42:20.037 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 1 -- 172.21.15.26:0/2101671274 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2628020630 0 0) 0x7fb320003530 con 0x7fb3380a1ca0 2024-09-16T14:42:20.037 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 1 -- 172.21.15.26:0/2101671274 >> v1:172.21.15.189:6789/0 conn(0x7fb3380a2790 legacy=0x7fb338139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:20.037 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 1 -- 172.21.15.26:0/2101671274 >> v1:172.21.15.26:6789/0 conn(0x7fb3380a2ba0 legacy=0x7fb33813d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:20.037 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:42:20.038 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 1 monclient: found mon.noname-b 2024-09-16T14:42:20.038 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:20.038 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:20.038 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 1 -- 172.21.15.26:0/2101671274 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33809ff40 con 0x7fb3380a1ca0 2024-09-16T14:42:20.038 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient: _finish_auth 0 2024-09-16T14:42:20.038 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:42:20.038 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:20.038 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 1 -- 172.21.15.26:0/2101671274 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb328001620 con 0x7fb3380a1ca0 2024-09-16T14:42:20.038 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:20.038 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:20.038 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient: dump: 2024-09-16T14:42:20.038 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-09-16T14:42:20.038 INFO:teuthology.misc.health.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:20.039 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:20.039 INFO:teuthology.misc.health.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:20.039 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:20.039 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:20.039 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:20.039 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:20.039 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:20.039 INFO:teuthology.misc.health.smithi026.stderr: 2024-09-16T14:42:20.039 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 5 monclient: authenticate success, global_id 4172 2024-09-16T14:42:20.039 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:20.039 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 1 -- 172.21.15.26:0/2101671274 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb328001840 con 0x7fb3380a1ca0 2024-09-16T14:42:20.039 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:20.039 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 1 -- 172.21.15.26:0/2101671274 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb328001c10 con 0x7fb3380a1ca0 2024-09-16T14:42:20.040 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:20.040 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 4 set_mon_vals no callback set 2024-09-16T14:42:20.040 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:20.040 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:20.040 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb335ffb640 10 monclient: dump: 2024-09-16T14:42:20.040 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-09-16T14:42:20.040 INFO:teuthology.misc.health.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:20.040 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:20.040 INFO:teuthology.misc.health.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:20.040 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:20.040 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:20.040 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:20.040 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:20.041 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:20.041 INFO:teuthology.misc.health.smithi026.stderr: 2024-09-16T14:42:20.041 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient: shutdown 2024-09-16T14:42:20.041 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:20.041 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 1 -- 172.21.15.26:0/2101671274 >> v1:172.21.15.26:6790/0 conn(0x7fb3380a1ca0 legacy=0x7fb3380a2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:20.041 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 1 -- 172.21.15.26:0/2101671274 shutdown_connections 2024-09-16T14:42:20.041 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 1 -- 172.21.15.26:0/2101671274 >> 172.21.15.26:0/2101671274 conn(0x7fb3380a7ed0 msgr2=0x7fb3380a3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:20.041 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 1 -- 172.21.15.26:0/2101671274 shutdown_connections 2024-09-16T14:42:20.041 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 1 -- 172.21.15.26:0/2101671274 wait complete. 2024-09-16T14:42:20.041 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient: build_initial_monmap 2024-09-16T14:42:20.041 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient: monmap: 2024-09-16T14:42:20.041 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-09-16T14:42:20.041 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:20.042 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:20.042 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-09-16T14:42:20.042 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:20.042 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:20.042 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:20.042 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:20.042 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:20.042 INFO:teuthology.misc.health.smithi026.stderr: 2024-09-16T14:42:20.042 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 1 Processor -- start 2024-09-16T14:42:20.042 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 1 -- start start 2024-09-16T14:42:20.042 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient: init 2024-09-16T14:42:20.042 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:20.042 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:20.042 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient(hunting): picked mon.noname-b con 0x7fb3380a1ca0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:20.043 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient(hunting): picked mon.noname-c con 0x7fb3380a2790 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:20.043 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient(hunting): picked mon.noname-a con 0x7fb3380a2ba0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:20.043 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33813c970 con 0x7fb3380a2ba0 2024-09-16T14:42:20.043 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33813ae40 con 0x7fb3380a1ca0 2024-09-16T14:42:20.043 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3380a6510 con 0x7fb3380a2790 2024-09-16T14:42:20.043 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:20.043 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb33d551640 10 monclient(hunting): authenticate will time out at 964.151145s 2024-09-16T14:42:20.043 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb3367fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb3380a2790 0x7fb3381ce450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35732/0 (socket says 172.21.15.26:35732) 2024-09-16T14:42:20.043 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb3367fc640 1 -- 172.21.15.26:0/1251473451 learned_addr learned my addr 172.21.15.26:0/1251473451 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:20.043 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb3377fe640 1 --1- 172.21.15.26:0/1251473451 >> v1:172.21.15.26:6789/0 conn(0x7fb3380a2ba0 0x7fb3381d1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52270/0 (socket says 172.21.15.26:52270) 2024-09-16T14:42:20.043 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694090812 0 0) 0x7fb33813c970 con 0x7fb3380a2ba0 2024-09-16T14:42:20.043 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:20.043 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:20.044 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:20.044 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 10 monclient(hunting): my global_id is 4173 2024-09-16T14:42:20.044 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb324003530 con 0x7fb3380a2ba0 2024-09-16T14:42:20.044 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262976008 0 0) 0x7fb3380a6510 con 0x7fb3380a2790 2024-09-16T14:42:20.044 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:20.044 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:20.044 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:20.044 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 10 monclient(hunting): my global_id is 4175 2024-09-16T14:42:20.044 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb33813c970 con 0x7fb3380a2790 2024-09-16T14:42:20.044 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061613292 0 0) 0x7fb33813ae40 con 0x7fb3380a1ca0 2024-09-16T14:42:20.044 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:20.044 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:20.045 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:20.045 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 10 monclient(hunting): my global_id is 4171 2024-09-16T14:42:20.045 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3380a6510 con 0x7fb3380a1ca0 2024-09-16T14:42:20.045 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 946517465 0 0) 0x7fb324003530 con 0x7fb3380a2ba0 2024-09-16T14:42:20.045 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb33813ae40 con 0x7fb3380a2ba0 2024-09-16T14:42:20.045 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb330001480 con 0x7fb3380a2ba0 2024-09-16T14:42:20.045 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:20.045 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:20.045 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 10 monclient(hunting): dump: 2024-09-16T14:42:20.045 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-09-16T14:42:20.045 INFO:teuthology.misc.health.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:20.045 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:20.045 INFO:teuthology.misc.health.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:20.046 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:20.046 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:20.046 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:20.046 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:20.046 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:20.046 INFO:teuthology.misc.health.smithi026.stderr: 2024-09-16T14:42:20.046 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:20.046 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3578085973 0 0) 0x7fb33813c970 con 0x7fb3380a2790 2024-09-16T14:42:20.046 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb324003530 con 0x7fb3380a2790 2024-09-16T14:42:20.046 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.028+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb32c001840 con 0x7fb3380a2790 2024-09-16T14:42:20.046 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:20.046 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:20.046 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 10 monclient(hunting): dump: 2024-09-16T14:42:20.046 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-09-16T14:42:20.047 INFO:teuthology.misc.health.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:20.047 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:20.047 INFO:teuthology.misc.health.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:20.047 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:20.047 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:20.047 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:20.047 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:20.047 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:20.047 INFO:teuthology.misc.health.smithi026.stderr: 2024-09-16T14:42:20.047 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 755821576 0 0) 0x7fb33813ae40 con 0x7fb3380a2ba0 2024-09-16T14:42:20.047 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 >> v1:172.21.15.26:6790/0 conn(0x7fb3380a2790 legacy=0x7fb3381ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:20.047 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 >> v1:172.21.15.189:6789/0 conn(0x7fb3380a1ca0 legacy=0x7fb338134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:20.047 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:42:20.048 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 1 monclient: found mon.a 2024-09-16T14:42:20.048 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:20.048 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:20.048 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33809e490 con 0x7fb3380a2ba0 2024-09-16T14:42:20.048 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:42:20.048 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:42:20.048 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:20.048 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb33d551640 5 monclient: authenticate success, global_id 4173 2024-09-16T14:42:20.048 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb33d551640 10 monclient: _renew_subs 2024-09-16T14:42:20.048 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb33d551640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:20.048 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb33d551640 1 -- 172.21.15.26:0/1251473451 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb33813b790 con 0x7fb3380a2ba0 2024-09-16T14:42:20.048 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb33d551640 10 monclient: _renew_subs 2024-09-16T14:42:20.048 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb33d551640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:20.049 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb33d551640 1 -- 172.21.15.26:0/1251473451 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3381355c0 con 0x7fb3380a2ba0 2024-09-16T14:42:20.049 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb330004b70 con 0x7fb3380a2ba0 2024-09-16T14:42:20.049 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:20.049 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb330004e80 con 0x7fb3380a2ba0 2024-09-16T14:42:20.049 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:20.049 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:20.049 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb3357fa640 4 set_mon_vals no callback set 2024-09-16T14:42:20.049 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 10 monclient: dump: 2024-09-16T14:42:20.049 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-09-16T14:42:20.049 INFO:teuthology.misc.health.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:20.049 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:20.049 INFO:teuthology.misc.health.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:20.049 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:20.049 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:20.050 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:20.050 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:20.050 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:20.050 INFO:teuthology.misc.health.smithi026.stderr: 2024-09-16T14:42:20.050 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fb3300052e0 con 0x7fb3380a2ba0 2024-09-16T14:42:20.050 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 976301087 0 0) 0x7fb330006970 con 0x7fb3380a2ba0 2024-09-16T14:42:20.050 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb33d551640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:20.050 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb33d551640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:20.050 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb33d551640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:20.050 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.032+0000 7fb33d551640 1 -- 172.21.15.26:0/1251473451 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb318005050 con 0x7fb3380a2ba0 2024-09-16T14:42:20.051 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.040+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7fb330006b30 con 0x7fb3380a2ba0 2024-09-16T14:42:20.051 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.040+0000 7fb31f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:20.051 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.040+0000 7fb31f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:20.101 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.100+0000 7fb33d551640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:20.101 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.100+0000 7fb33d551640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:20.101 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.100+0000 7fb33d551640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:20.101 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.100+0000 7fb33d551640 1 -- 172.21.15.26:0/1251473451 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb318005300 con 0x7fb3380a2ba0 2024-09-16T14:42:20.101 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.100+0000 7fb31f7fe640 1 -- 172.21.15.26:0/1251473451 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5604 (unknown 150273500 0 3201663216) 0x7fb330006430 con 0x7fb3380a2ba0 2024-09-16T14:42:20.102 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.100+0000 7fb31f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:20.102 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.100+0000 7fb31f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:20.102 INFO:teuthology.misc.health.smithi026.stdout: 2024-09-16T14:42:20.102 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":8,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","created":"2024-09-16T14:42:17.361142+0000","modified":"2024-09-16T14:42:19.053975+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T14:42:19.053975+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a4578188-6e8b-430f-b70d-1c3480eb7bae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a96367f-73f1-4a99-9b85-47bf8e3d4cdc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3030f43d-f848-4189-be1e-6ea367941218","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"78731cf9-4ba7-4ce9-a483-080aa1136b65","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d74156df-be8d-4ad7-93dc-89ac28df70dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ccad9c1e-247f-4ed4-8437-6aa5c8b515d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T14:42:20.103 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.100+0000 7fb33d551640 10 monclient: shutdown 2024-09-16T14:42:20.103 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.100+0000 7fb33d551640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:20.103 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.100+0000 7fb33d551640 1 -- 172.21.15.26:0/1251473451 >> v1:172.21.15.26:6789/0 conn(0x7fb3380a2ba0 legacy=0x7fb3381d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:20.103 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.100+0000 7fb33d551640 1 -- 172.21.15.26:0/1251473451 shutdown_connections 2024-09-16T14:42:20.103 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.100+0000 7fb33d551640 1 -- 172.21.15.26:0/1251473451 >> 172.21.15.26:0/1251473451 conn(0x7fb3380a7ed0 msgr2=0x7fb3380a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:20.103 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.100+0000 7fb33d551640 1 -- 172.21.15.26:0/1251473451 shutdown_connections 2024-09-16T14:42:20.103 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:20.100+0000 7fb33d551640 1 -- 172.21.15.26:0/1251473451 wait complete. 2024-09-16T14:42:20.117 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-16T14:42:20.426 INFO:tasks.ceph.mgr.x.smithi189.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-16T14:42:20.427 INFO:tasks.ceph.mgr.x.smithi189.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-16T14:42:20.427 INFO:tasks.ceph.mgr.x.smithi189.stderr: from numpy import show_config as show_numpy_config 2024-09-16T14:42:20.487 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:20.485+0000 7f9d7ec72180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T14:42:20.668 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:20.665+0000 7f9d7ec72180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T14:42:20.834 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:20.833+0000 7f9d7ec72180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T14:42:20.903 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:20.901+0000 7f9d7ec72180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T14:42:21.050 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:21.049+0000 7f9d7ec72180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T14:42:21.675 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:21.673+0000 7f9d7ec72180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T14:42:21.749 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:21.749+0000 7f9d7ec72180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T14:42:21.928 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:21.925+0000 7f9d7ec72180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T14:42:22.000 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:21.997+0000 7f9d7ec72180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T14:42:22.286 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:22.285+0000 7f9d7ec72180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T14:42:22.348 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:22.345+0000 7f9d7ec72180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T14:42:23.382 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:23.381+0000 7f9d26222640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T14:42:23.389 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:42:23.389+0000 7f9d26222640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T14:42:23.442 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-16T14:42:23.440+0000 7fa7c20aa7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T14:42:23.500 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-09-16T14:42:23.497+0000 7efeea3e07c0 -1 osd.4 0 log_to_monitors true 2024-09-16T14:42:23.506 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-09-16T14:42:23.505+0000 7f95327c87c0 -1 osd.5 0 log_to_monitors true 2024-09-16T14:42:23.541 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-16T14:42:23.540+0000 7f66916e67c0 -1 osd.1 0 log_to_monitors true 2024-09-16T14:42:23.590 INFO:tasks.ceph.osd.3.smithi189.stderr:2024-09-16T14:42:23.589+0000 7f269c06d7c0 -1 osd.3 0 log_to_monitors true 2024-09-16T14:42:23.594 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-16T14:42:23.592+0000 7f802ec017c0 -1 osd.0 0 log_to_monitors true 2024-09-16T14:42:25.448 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-16T14:42:25.444+0000 7fa7be04f640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T14:42:25.448 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-16T14:42:25.444+0000 7f668d68b640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T14:42:25.453 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-09-16T14:42:25.453+0000 7efee6385640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T14:42:25.453 INFO:tasks.ceph.osd.3.smithi189.stderr:2024-09-16T14:42:25.453+0000 7f2698012640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T14:42:25.459 INFO:tasks.ceph.osd.3.smithi189.stderr:2024-09-16T14:42:25.457+0000 7f2693e4d640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T14:42:25.459 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-09-16T14:42:25.457+0000 7efee21c0640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T14:42:25.460 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-09-16T14:42:25.456+0000 7fa7b9e8a640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T14:42:25.460 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-09-16T14:42:25.456+0000 7f66894c6640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T14:42:26.136 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-16T14:42:26.132+0000 7f802aba6640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T14:42:26.140 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-09-16T14:42:26.136+0000 7f80229d9640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T14:42:26.162 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-09-16T14:42:26.161+0000 7f952e76d640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T14:42:26.166 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-09-16T14:42:26.165+0000 7f95265a0640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T14:42:26.723 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T14:42:26.848 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.844+0000 7f6290e42640 10 monclient: get_monmap_and_config 2024-09-16T14:42:26.848 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.844+0000 7f6290e42640 10 monclient: build_initial_monmap 2024-09-16T14:42:26.848 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.844+0000 7f6290e42640 10 monclient: monmap: 2024-09-16T14:42:26.848 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-09-16T14:42:26.848 INFO:teuthology.misc.health.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:26.848 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-09-16T14:42:26.848680+0000 2024-09-16T14:42:26.848 INFO:teuthology.misc.health.smithi026.stderr:created 2024-09-16T14:42:26.848680+0000 2024-09-16T14:42:26.849 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:26.849 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:26.849 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:26.849 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:26.849 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:26.849 INFO:teuthology.misc.health.smithi026.stderr: 2024-09-16T14:42:26.856 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.844+0000 7f6290e42640 1 Processor -- start 2024-09-16T14:42:26.856 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 1 -- start start 2024-09-16T14:42:26.856 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient: init 2024-09-16T14:42:26.856 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:26.856 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:26.856 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient(hunting): picked mon.noname-b con 0x7f628c070ad0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:26.856 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient(hunting): picked mon.noname-c con 0x7f628c09e0c0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:26.856 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient(hunting): picked mon.noname-a con 0x7f628c09f570 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:26.857 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c09eb90 con 0x7f628c09f570 2024-09-16T14:42:26.857 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c09ee70 con 0x7f628c09e0c0 2024-09-16T14:42:26.857 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c09f150 con 0x7f628c070ad0 2024-09-16T14:42:26.857 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:26.857 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient(hunting): authenticate will time out at 970.968878s 2024-09-16T14:42:26.857 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f628c070ad0 0x7f628c070eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35780/0 (socket says 172.21.15.26:35780) 2024-09-16T14:42:26.857 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628a59c640 1 -- 172.21.15.26:0/3504074154 learned_addr learned my addr 172.21.15.26:0/3504074154 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:26.857 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 1 -- 172.21.15.26:0/3504074154 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758712402 0 0) 0x7f628c09f150 con 0x7f628c070ad0 2024-09-16T14:42:26.857 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient(hunting): _negotiate 2024-09-16T14:42:26.857 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:26.858 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:26.858 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient(hunting): my global_id is 4208 2024-09-16T14:42:26.858 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 1 -- 172.21.15.26:0/3504074154 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6278003530 con 0x7f628c070ad0 2024-09-16T14:42:26.858 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 1 -- 172.21.15.26:0/3504074154 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866297579 0 0) 0x7f628c09eb90 con 0x7f628c09f570 2024-09-16T14:42:26.858 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient(hunting): _negotiate 2024-09-16T14:42:26.858 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:26.858 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:26.858 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient(hunting): my global_id is 4209 2024-09-16T14:42:26.859 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 1 -- 172.21.15.26:0/3504074154 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f628c09f150 con 0x7f628c09f570 2024-09-16T14:42:26.859 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 1 -- 172.21.15.26:0/3504074154 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 236957690 0 0) 0x7f628c09f150 con 0x7f628c09f570 2024-09-16T14:42:26.859 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 1 -- 172.21.15.26:0/3504074154 >> v1:172.21.15.26:6790/0 conn(0x7f628c070ad0 legacy=0x7f628c070eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:26.859 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 1 -- 172.21.15.26:0/3504074154 >> v1:172.21.15.189:6789/0 conn(0x7f628c09e0c0 legacy=0x7f628c139e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:26.859 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient: _finish_hunting 0 2024-09-16T14:42:26.859 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 1 monclient: found mon.noname-a 2024-09-16T14:42:26.859 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:26.859 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:42:26.859 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 1 -- 172.21.15.26:0/3504074154 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c09dcb0 con 0x7f628c09f570 2024-09-16T14:42:26.859 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient: _finish_auth 0 2024-09-16T14:42:26.859 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient: _check_auth_tickets 2024-09-16T14:42:26.859 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:26.860 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 1 -- 172.21.15.26:0/3504074154 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6284001620 con 0x7f628c09f570 2024-09-16T14:42:26.860 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 5 monclient: authenticate success, global_id 4209 2024-09-16T14:42:26.860 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:26.860 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:26.860 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:26.860 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient: dump: 2024-09-16T14:42:26.860 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-09-16T14:42:26.860 INFO:teuthology.misc.health.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:26.860 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:26.860 INFO:teuthology.misc.health.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:26.860 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:26.861 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:26.861 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:26.861 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:26.861 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:26.861 INFO:teuthology.misc.health.smithi026.stderr: 2024-09-16T14:42:26.861 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:26.861 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 1 -- 172.21.15.26:0/3504074154 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6284001840 con 0x7f628c09f570 2024-09-16T14:42:26.861 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:26.861 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:26.861 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 4 set_mon_vals no callback set 2024-09-16T14:42:26.861 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 1 -- 172.21.15.26:0/3504074154 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6284001bd0 con 0x7f628c09f570 2024-09-16T14:42:26.862 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:26.862 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:26.862 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f628959a640 10 monclient: dump: 2024-09-16T14:42:26.862 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-09-16T14:42:26.862 INFO:teuthology.misc.health.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:26.862 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:26.862 INFO:teuthology.misc.health.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:26.862 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:26.862 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:26.862 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:26.862 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:26.862 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:26.862 INFO:teuthology.misc.health.smithi026.stderr: 2024-09-16T14:42:26.862 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient: shutdown 2024-09-16T14:42:26.863 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:26.863 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 1 -- 172.21.15.26:0/3504074154 >> v1:172.21.15.26:6789/0 conn(0x7f628c09f570 legacy=0x7f628c13d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:26.863 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 1 -- 172.21.15.26:0/3504074154 shutdown_connections 2024-09-16T14:42:26.863 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 1 -- 172.21.15.26:0/3504074154 >> 172.21.15.26:0/3504074154 conn(0x7f628c09fb50 msgr2=0x7f628c09ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:26.863 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 1 -- 172.21.15.26:0/3504074154 shutdown_connections 2024-09-16T14:42:26.863 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 1 -- 172.21.15.26:0/3504074154 wait complete. 2024-09-16T14:42:26.863 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient: build_initial_monmap 2024-09-16T14:42:26.863 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient: monmap: 2024-09-16T14:42:26.863 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-09-16T14:42:26.863 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:26.863 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:26.863 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-09-16T14:42:26.863 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:26.863 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:26.864 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:26.864 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:26.864 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:26.864 INFO:teuthology.misc.health.smithi026.stderr: 2024-09-16T14:42:26.864 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 1 Processor -- start 2024-09-16T14:42:26.864 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 1 -- start start 2024-09-16T14:42:26.864 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient: init 2024-09-16T14:42:26.864 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:26.864 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:26.864 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient(hunting): picked mon.noname-b con 0x7f628c070ad0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:26.864 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.848+0000 7f6290e42640 10 monclient(hunting): picked mon.noname-c con 0x7f628c09e0c0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 10 monclient(hunting): picked mon.noname-a con 0x7f628c09f570 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c134aa0 con 0x7f628c09f570 2024-09-16T14:42:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c135910 con 0x7f628c070ad0 2024-09-16T14:42:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c09a7b0 con 0x7f628c09e0c0 2024-09-16T14:42:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 10 monclient(hunting): authenticate will time out at 970.972069s 2024-09-16T14:42:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f628ad9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f628c09f570 0x7f628c1d14b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52306/0 (socket says 172.21.15.26:52306) 2024-09-16T14:42:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f628ad9d640 1 -- 172.21.15.26:0/2278898091 learned_addr learned my addr 172.21.15.26:0/2278898091 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968941129 0 0) 0x7f628c09a7b0 con 0x7f628c09e0c0 2024-09-16T14:42:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): my global_id is 4211 2024-09-16T14:42:26.866 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f626c003530 con 0x7f628c09e0c0 2024-09-16T14:42:26.866 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471574380 0 0) 0x7f628c134aa0 con 0x7f628c09f570 2024-09-16T14:42:26.866 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:26.866 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:26.866 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:26.866 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): my global_id is 4212 2024-09-16T14:42:26.866 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f628c09a7b0 con 0x7f628c09f570 2024-09-16T14:42:26.866 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1564293412 0 0) 0x7f626c003530 con 0x7f628c09e0c0 2024-09-16T14:42:26.866 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f628c134aa0 con 0x7f628c09e0c0 2024-09-16T14:42:26.866 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f627c001b80 con 0x7f628c09e0c0 2024-09-16T14:42:26.866 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:26.866 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:26.867 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:26.867 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-09-16T14:42:26.867 INFO:teuthology.misc.health.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:26.867 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:26.867 INFO:teuthology.misc.health.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:26.867 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:26.867 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:26.867 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:26.867 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:26.867 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:26.867 INFO:teuthology.misc.health.smithi026.stderr: 2024-09-16T14:42:26.868 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:26.868 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1854818789 0 0) 0x7f628c09a7b0 con 0x7f628c09f570 2024-09-16T14:42:26.868 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f626c003530 con 0x7f628c09f570 2024-09-16T14:42:26.868 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595150325 0 0) 0x7f628c135910 con 0x7f628c070ad0 2024-09-16T14:42:26.868 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:26.868 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:26.868 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:26.868 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): my global_id is 4213 2024-09-16T14:42:26.868 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f628c09a7b0 con 0x7f628c070ad0 2024-09-16T14:42:26.868 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6284001520 con 0x7f628c09f570 2024-09-16T14:42:26.868 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:26.868 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:26.869 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:26.869 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-09-16T14:42:26.869 INFO:teuthology.misc.health.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:26.869 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:26.869 INFO:teuthology.misc.health.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:26.869 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:26.869 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:26.869 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:26.869 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:26.869 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:26.869 INFO:teuthology.misc.health.smithi026.stderr: 2024-09-16T14:42:26.869 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3564303091 0 0) 0x7f628c134aa0 con 0x7f628c09e0c0 2024-09-16T14:42:26.869 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 >> v1:172.21.15.189:6789/0 conn(0x7f628c070ad0 legacy=0x7f628c133c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 >> v1:172.21.15.26:6789/0 conn(0x7f628c09f570 legacy=0x7f628c1d14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 monclient: found mon.c 2024-09-16T14:42:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c09a1d0 con 0x7f628c09e0c0 2024-09-16T14:42:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 5 monclient: authenticate success, global_id 4211 2024-09-16T14:42:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 10 monclient: _renew_subs 2024-09-16T14:42:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 1 -- 172.21.15.26:0/2278898091 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628c137580 con 0x7f628c09e0c0 2024-09-16T14:42:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 10 monclient: _renew_subs 2024-09-16T14:42:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 1 -- 172.21.15.26:0/2278898091 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c13dd00 con 0x7f628c09e0c0 2024-09-16T14:42:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f627c003980 con 0x7f628c09e0c0 2024-09-16T14:42:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f627c003c90 con 0x7f628c09e0c0 2024-09-16T14:42:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6288d99640 4 set_mon_vals no callback set 2024-09-16T14:42:26.872 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 10 monclient: dump: 2024-09-16T14:42:26.872 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-09-16T14:42:26.872 INFO:teuthology.misc.health.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:26.872 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:26.872 INFO:teuthology.misc.health.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:26.872 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:26.872 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:26.872 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:26.872 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:26.872 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:26.872 INFO:teuthology.misc.health.smithi026.stderr: 2024-09-16T14:42:26.872 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 43216+0+0 (unknown 377793617 0 0) 0x7f627c003df0 con 0x7f628c09e0c0 2024-09-16T14:42:26.873 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:26.873 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 2767+0+0 (unknown 551773692 0 0) 0x7f627c041de0 con 0x7f628c09e0c0 2024-09-16T14:42:26.873 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:26.873 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:26.873 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f6290e42640 1 -- 172.21.15.26:0/2278898091 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6270005050 con 0x7f628c09e0c0 2024-09-16T14:42:26.873 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.852+0000 7f628a59c640 10 monclient: get_auth_request con 0x7f626c036410 auth_method 0 2024-09-16T14:42:26.873 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.856+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f627c0045f0 con 0x7f628c09e0c0 2024-09-16T14:42:26.874 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.856+0000 7f6282ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:26.874 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:26.856+0000 7f6282ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:27.023 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:27.020+0000 7f6290e42640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:27.023 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:27.020+0000 7f6290e42640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:27.023 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:27.020+0000 7f6290e42640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:27.023 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:27.020+0000 7f6290e42640 1 -- 172.21.15.26:0/2278898091 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6270005300 con 0x7f628c09e0c0 2024-09-16T14:42:27.024 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:27.020+0000 7f6282ffd640 1 -- 172.21.15.26:0/2278898091 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7056 (unknown 238102980 0 1413008740) 0x7f627c011ba0 con 0x7f628c09e0c0 2024-09-16T14:42:27.024 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:27.020+0000 7f6282ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:27.025 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:27.020+0000 7f6282ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:27.025 INFO:teuthology.misc.health.smithi026.stdout: 2024-09-16T14:42:27.025 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":11,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","created":"2024-09-16T14:42:17.361142+0000","modified":"2024-09-16T14:42:26.444878+0000","last_up_change":"2024-09-16T14:42:26.444878+0000","last_in_change":"2024-09-16T14:42:19.053975+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a4578188-6e8b-430f-b70d-1c3480eb7bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6808","nonce":3182580250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6809","nonce":3182580250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6811","nonce":3182580250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6810","nonce":3182580250}]},"public_addr":"172.21.15.26:6808/3182580250","cluster_addr":"172.21.15.26:6809/3182580250","heartbeat_back_addr":"172.21.15.26:6811/3182580250","heartbeat_front_addr":"172.21.15.26:6810/3182580250","state":["exists","up"]},{"osd":1,"uuid":"1a96367f-73f1-4a99-9b85-47bf8e3d4cdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6800","nonce":3032704211}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6801","nonce":3032704211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6805","nonce":3032704211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6803","nonce":3032704211}]},"public_addr":"172.21.15.26:6800/3032704211","cluster_addr":"172.21.15.26:6801/3032704211","heartbeat_back_addr":"172.21.15.26:6805/3032704211","heartbeat_front_addr":"172.21.15.26:6803/3032704211","state":["exists","up"]},{"osd":2,"uuid":"3030f43d-f848-4189-be1e-6ea367941218","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6802","nonce":1811454603}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6804","nonce":1811454603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6807","nonce":1811454603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6806","nonce":1811454603}]},"public_addr":"172.21.15.26:6802/1811454603","cluster_addr":"172.21.15.26:6804/1811454603","heartbeat_back_addr":"172.21.15.26:6807/1811454603","heartbeat_front_addr":"172.21.15.26:6806/1811454603","state":["exists","up"]},{"osd":3,"uuid":"78731cf9-4ba7-4ce9-a483-080aa1136b65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6808","nonce":3651837298}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6809","nonce":3651837298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6811","nonce":3651837298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6810","nonce":3651837298}]},"public_addr":"172.21.15.189:6808/3651837298","cluster_addr":"172.21.15.189:6809/3651837298","heartbeat_back_addr":"172.21.15.189:6811/3651837298","heartbeat_front_addr":"172.21.15.189:6810/3651837298","state":["exists","up"]},{"osd":4,"uuid":"d74156df-be8d-4ad7-93dc-89ac28df70dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6800","nonce":1782891661}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6801","nonce":1782891661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6803","nonce":1782891661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6802","nonce":1782891661}]},"public_addr":"172.21.15.189:6800/1782891661","cluster_addr":"172.21.15.189:6801/1782891661","heartbeat_back_addr":"172.21.15.189:6803/1782891661","heartbeat_front_addr":"172.21.15.189:6802/1782891661","state":["exists","up"]},{"osd":5,"uuid":"ccad9c1e-247f-4ed4-8437-6aa5c8b515d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6804","nonce":3793314587}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6805","nonce":3793314587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6807","nonce":3793314587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6806","nonce":3793314587}]},"public_addr":"172.21.15.189:6804/3793314587","cluster_addr":"172.21.15.189:6805/3793314587","heartbeat_back_addr":"172.21.15.189:6807/3793314587","heartbeat_front_addr":"172.21.15.189:6806/3793314587","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T14:42:27.027 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:27.024+0000 7f6290e42640 1 -- 172.21.15.26:0/2278898091 >> v1:172.21.15.189:6812/3196161582 conn(0x7f626c036410 legacy=0x7f626c0388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:27.027 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:27.024+0000 7f6290e42640 10 monclient: shutdown 2024-09-16T14:42:27.027 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:27.024+0000 7f6290e42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:27.027 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:27.024+0000 7f6290e42640 1 -- 172.21.15.26:0/2278898091 >> v1:172.21.15.26:6790/0 conn(0x7f628c09e0c0 legacy=0x7f628c134390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:27.028 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:27.024+0000 7f6290e42640 1 -- 172.21.15.26:0/2278898091 shutdown_connections 2024-09-16T14:42:27.028 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:27.024+0000 7f6290e42640 1 -- 172.21.15.26:0/2278898091 >> 172.21.15.26:0/2278898091 conn(0x7f628c09fb50 msgr2=0x7f628c0991b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:27.028 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:27.024+0000 7f6290e42640 1 -- 172.21.15.26:0/2278898091 shutdown_connections 2024-09-16T14:42:27.028 INFO:teuthology.misc.health.smithi026.stderr:2024-09-16T14:42:27.024+0000 7f6290e42640 1 -- 172.21.15.26:0/2278898091 wait complete. 2024-09-16T14:42:27.042 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-16T14:42:27.042 INFO:tasks.ceph:Creating RBD pool 2024-09-16T14:42:27.043 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T14:42:27.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.164+0000 7fbe3ebde640 10 monclient: get_monmap_and_config 2024-09-16T14:42:27.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.164+0000 7fbe3ebde640 10 monclient: build_initial_monmap 2024-09-16T14:42:27.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.164+0000 7fbe3ebde640 10 monclient: monmap: 2024-09-16T14:42:27.168 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:27.168 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:27.169 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:27.168795+0000 2024-09-16T14:42:27.169 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:27.168795+0000 2024-09-16T14:42:27.169 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:27.169 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:27.169 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:27.169 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:27.169 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:27.169 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:27.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 1 Processor -- start 2024-09-16T14:42:27.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 1 -- start start 2024-09-16T14:42:27.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 10 monclient: init 2024-09-16T14:42:27.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:27.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:27.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 10 monclient(hunting): picked mon.noname-c con 0x7fbe380a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:27.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 10 monclient(hunting): picked mon.noname-b con 0x7fbe38099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:27.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 10 monclient(hunting): picked mon.noname-a con 0x7fbe380a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:27.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe38098ea0 con 0x7fbe380a26f0 2024-09-16T14:42:27.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe380997b0 con 0x7fbe380a1c00 2024-09-16T14:42:27.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe38099460 con 0x7fbe38099b60 2024-09-16T14:42:27.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:27.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 10 monclient(hunting): authenticate will time out at 971.289446s 2024-09-16T14:42:27.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe37fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbe38099b60 0x7fbe38139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48206/0 (socket says 172.21.15.26:48206) 2024-09-16T14:42:27.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe37fff640 1 -- 172.21.15.26:0/3768454611 learned_addr learned my addr 172.21.15.26:0/3768454611 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:27.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 1 -- 172.21.15.26:0/3768454611 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201304236 0 0) 0x7fbe38099460 con 0x7fbe38099b60 2024-09-16T14:42:27.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:27.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:27.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:27.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient(hunting): my global_id is 4214 2024-09-16T14:42:27.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 1 -- 172.21.15.26:0/3768454611 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe20003530 con 0x7fbe38099b60 2024-09-16T14:42:27.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 1 -- 172.21.15.26:0/3768454611 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034258782 0 0) 0x7fbe38098ea0 con 0x7fbe380a26f0 2024-09-16T14:42:27.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:27.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:27.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:27.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient(hunting): my global_id is 4215 2024-09-16T14:42:27.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 1 -- 172.21.15.26:0/3768454611 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe38099460 con 0x7fbe380a26f0 2024-09-16T14:42:27.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 1 -- 172.21.15.26:0/3768454611 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4204443602 0 0) 0x7fbe38099460 con 0x7fbe380a26f0 2024-09-16T14:42:27.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 1 -- 172.21.15.26:0/3768454611 >> v1:172.21.15.26:6790/0 conn(0x7fbe38099b60 legacy=0x7fbe38139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:27.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 1 -- 172.21.15.26:0/3768454611 >> v1:172.21.15.189:6789/0 conn(0x7fbe380a1c00 legacy=0x7fbe380a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:27.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient: _finish_hunting 0 2024-09-16T14:42:27.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 1 monclient: found mon.noname-a 2024-09-16T14:42:27.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:27.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:42:27.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 1 -- 172.21.15.26:0/3768454611 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe380a34b0 con 0x7fbe380a26f0 2024-09-16T14:42:27.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient: _finish_auth 0 2024-09-16T14:42:27.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient: _check_auth_tickets 2024-09-16T14:42:27.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:27.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 1 -- 172.21.15.26:0/3768454611 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe30001980 con 0x7fbe380a26f0 2024-09-16T14:42:27.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:27.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:27.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient: dump: 2024-09-16T14:42:27.176 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:27.176 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:27.176 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.176 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.176 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:27.176 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:27.176 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:27.176 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:27.176 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:27.176 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 5 monclient: authenticate success, global_id 4215 2024-09-16T14:42:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 1 -- 172.21.15.26:0/3768454611 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe30001c20 con 0x7fbe380a26f0 2024-09-16T14:42:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 1 -- 172.21.15.26:0/3768454611 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe30001ff0 con 0x7fbe380a26f0 2024-09-16T14:42:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe377fe640 10 monclient: dump: 2024-09-16T14:42:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:27.178 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.178 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.178 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:27.178 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:27.178 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:27.178 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:27.178 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:27.178 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:27.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:27.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 4 set_mon_vals no callback set 2024-09-16T14:42:27.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 10 monclient: shutdown 2024-09-16T14:42:27.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:27.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 1 -- 172.21.15.26:0/3768454611 >> v1:172.21.15.26:6789/0 conn(0x7fbe380a26f0 legacy=0x7fbe3813d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:27.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 1 -- 172.21.15.26:0/3768454611 shutdown_connections 2024-09-16T14:42:27.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 1 -- 172.21.15.26:0/3768454611 >> 172.21.15.26:0/3768454611 conn(0x7fbe380a3d00 msgr2=0x7fbe380a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:27.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 1 -- 172.21.15.26:0/3768454611 shutdown_connections 2024-09-16T14:42:27.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 1 -- 172.21.15.26:0/3768454611 wait complete. 2024-09-16T14:42:27.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 10 monclient: build_initial_monmap 2024-09-16T14:42:27.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.168+0000 7fbe3ebde640 10 monclient: monmap: 2024-09-16T14:42:27.179 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:27.179 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:27.179 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:27.179 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:27.179 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:27.179 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:27.179 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:27.180 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:27.180 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:27.180 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:27.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 1 Processor -- start 2024-09-16T14:42:27.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 1 -- start start 2024-09-16T14:42:27.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 10 monclient: init 2024-09-16T14:42:27.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:27.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:27.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 10 monclient(hunting): picked mon.noname-a con 0x7fbe38099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:27.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 10 monclient(hunting): picked mon.noname-c con 0x7fbe380a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:27.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 10 monclient(hunting): picked mon.noname-b con 0x7fbe380a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:27.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe3811dc90 con 0x7fbe38099b60 2024-09-16T14:42:27.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe3809efa0 con 0x7fbe380a26f0 2024-09-16T14:42:27.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe38137920 con 0x7fbe380a1c00 2024-09-16T14:42:27.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:27.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 10 monclient(hunting): authenticate will time out at 971.292880s 2024-09-16T14:42:27.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3c97a640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbe38099b60 0x7fbe3813cd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57184/0 (socket says 172.21.15.26:57184) 2024-09-16T14:42:27.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe37fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbe380a1c00 0x7fbe3813d420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48222/0 (socket says 172.21.15.26:48222) 2024-09-16T14:42:27.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3c97a640 1 -- 172.21.15.26:0/928418293 learned_addr learned my addr 172.21.15.26:0/928418293 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:27.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620201582 0 0) 0x7fbe38137920 con 0x7fbe380a1c00 2024-09-16T14:42:27.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:27.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:27.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:27.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): my global_id is 4217 2024-09-16T14:42:27.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe24003530 con 0x7fbe380a1c00 2024-09-16T14:42:27.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979979709 0 0) 0x7fbe3811dc90 con 0x7fbe38099b60 2024-09-16T14:42:27.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:27.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:27.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:27.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): my global_id is 4218 2024-09-16T14:42:27.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe38137920 con 0x7fbe38099b60 2024-09-16T14:42:27.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1453286269 0 0) 0x7fbe24003530 con 0x7fbe380a1c00 2024-09-16T14:42:27.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe3811dc90 con 0x7fbe380a1c00 2024-09-16T14:42:27.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe2c001860 con 0x7fbe380a1c00 2024-09-16T14:42:27.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:27.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:27.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): dump: 2024-09-16T14:42:27.187 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:27.187 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:27.187 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.187 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.187 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:27.187 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:27.187 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:27.187 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:27.187 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:27.187 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:27.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:27.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3014758367 0 0) 0x7fbe38137920 con 0x7fbe38099b60 2024-09-16T14:42:27.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe24003530 con 0x7fbe38099b60 2024-09-16T14:42:27.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe28001950 con 0x7fbe38099b60 2024-09-16T14:42:27.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:27.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:27.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): dump: 2024-09-16T14:42:27.188 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:27.188 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:27.189 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.189 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.189 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:27.189 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:27.189 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:27.189 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:27.189 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:27.189 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:27.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489357050 0 0) 0x7fbe3809efa0 con 0x7fbe380a26f0 2024-09-16T14:42:27.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:27.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:27.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:27.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient(hunting): my global_id is 4219 2024-09-16T14:42:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe38137920 con 0x7fbe380a26f0 2024-09-16T14:42:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2219450402 0 0) 0x7fbe3811dc90 con 0x7fbe380a1c00 2024-09-16T14:42:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 >> v1:172.21.15.189:6789/0 conn(0x7fbe380a26f0 legacy=0x7fbe381d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 >> v1:172.21.15.26:6789/0 conn(0x7fbe38099b60 legacy=0x7fbe3813cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient: _finish_hunting 0 2024-09-16T14:42:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 monclient: found mon.c 2024-09-16T14:42:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3811f520 con 0x7fbe380a1c00 2024-09-16T14:42:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient: _finish_auth 0 2024-09-16T14:42:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient: _check_auth_tickets 2024-09-16T14:42:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 5 monclient: authenticate success, global_id 4217 2024-09-16T14:42:27.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 10 monclient: _renew_subs 2024-09-16T14:42:27.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:27.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 1 -- 172.21.15.26:0/928418293 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe3813db30 con 0x7fbe380a1c00 2024-09-16T14:42:27.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 10 monclient: _renew_subs 2024-09-16T14:42:27.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:27.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 1 -- 172.21.15.26:0/928418293 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3811e340 con 0x7fbe380a1c00 2024-09-16T14:42:27.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe2c0047f0 con 0x7fbe380a1c00 2024-09-16T14:42:27.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:27.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe2c004960 con 0x7fbe380a1c00 2024-09-16T14:42:27.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:27.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:27.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe36ffd640 4 set_mon_vals no callback set 2024-09-16T14:42:27.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 10 monclient: dump: 2024-09-16T14:42:27.192 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:27.192 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:27.192 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.192 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.192 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:27.192 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:27.192 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:27.192 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:27.192 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:27.192 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:27.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:27.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:27.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:27.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3ebde640 1 -- 172.21.15.26:0/928418293 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe18005050 con 0x7fbe380a1c00 2024-09-16T14:42:27.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 43216+0+0 (unknown 377793617 0 0) 0x7fbe2c00f3a0 con 0x7fbe380a1c00 2024-09-16T14:42:27.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 2767+0+0 (unknown 551773692 0 0) 0x7fbe2c042ae0 con 0x7fbe380a1c00 2024-09-16T14:42:27.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.172+0000 7fbe3c97a640 10 monclient: get_auth_request con 0x7fbe240363c0 auth_method 0 2024-09-16T14:42:27.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.176+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbe2c014200 con 0x7fbe380a1c00 2024-09-16T14:42:27.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.176+0000 7fbe357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:27.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.176+0000 7fbe357fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:27.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.336+0000 7fbe3ebde640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T14:42:27.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.336+0000 7fbe3ebde640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T14:42:27.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.336+0000 7fbe3ebde640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:27.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.336+0000 7fbe3ebde640 1 -- 172.21.15.26:0/928418293 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fbe18005880 con 0x7fbe380a1c00 2024-09-16T14:42:27.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.464+0000 7fbe357fa640 1 -- 172.21.15.26:0/928418293 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v12) v1 ==== 109+0+0 (unknown 3454938845 0 0) 0x7fbe2c0128a0 con 0x7fbe380a1c00 2024-09-16T14:42:27.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.464+0000 7fbe357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T14:42:27.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.464+0000 7fbe357fa640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-16T14:42:27.468 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'rbd' created 2024-09-16T14:42:27.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.468+0000 7fbe3ebde640 1 -- 172.21.15.26:0/928418293 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbe240363c0 legacy=0x7fbe24038880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:27.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.468+0000 7fbe3ebde640 10 monclient: shutdown 2024-09-16T14:42:27.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.468+0000 7fbe3ebde640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:27.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.468+0000 7fbe3ebde640 1 -- 172.21.15.26:0/928418293 >> v1:172.21.15.26:6790/0 conn(0x7fbe380a1c00 legacy=0x7fbe3813d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:27.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.468+0000 7fbe3ebde640 1 -- 172.21.15.26:0/928418293 shutdown_connections 2024-09-16T14:42:27.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.468+0000 7fbe3ebde640 1 -- 172.21.15.26:0/928418293 >> 172.21.15.26:0/928418293 conn(0x7fbe380a3d00 msgr2=0x7fbe380a0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:27.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.468+0000 7fbe3ebde640 1 -- 172.21.15.26:0/928418293 shutdown_connections 2024-09-16T14:42:27.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.468+0000 7fbe3ebde640 1 -- 172.21.15.26:0/928418293 wait complete. 2024-09-16T14:42:27.490 DEBUG:teuthology.orchestra.run.smithi026:> rbd --cluster ceph pool init rbd 2024-09-16T14:42:27.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.540+0000 7f878161de00 10 monclient: get_monmap_and_config 2024-09-16T14:42:27.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.540+0000 7f878161de00 10 monclient: build_initial_monmap 2024-09-16T14:42:27.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.540+0000 7f878161de00 10 monclient: monmap: 2024-09-16T14:42:27.544 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:27.544 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:27.544 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:27.544906+0000 2024-09-16T14:42:27.545 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:27.544906+0000 2024-09-16T14:42:27.545 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:27.545 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:27.545 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:27.545 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:27.545 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:27.545 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:27.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 1 Processor -- start 2024-09-16T14:42:27.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 1 -- start start 2024-09-16T14:42:27.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 10 monclient: init 2024-09-16T14:42:27.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 10 monclient: _reopen_session rank -1 2024-09-16T14:42:27.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:27.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 10 monclient(hunting): picked mon.noname-a con 0x55e08954be50 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:27.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 10 monclient(hunting): picked mon.noname-b con 0x55e0894bc620 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:27.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 10 monclient(hunting): picked mon.noname-c con 0x55e0895556e0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:27.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55e089351ab0 con 0x55e08954be50 2024-09-16T14:42:27.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55e08932a9c0 con 0x55e0895556e0 2024-09-16T14:42:27.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55e089337910 con 0x55e0894bc620 2024-09-16T14:42:27.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 10 monclient(hunting): _renew_subs 2024-09-16T14:42:27.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 10 monclient(hunting): authenticate will time out at 971.665439s 2024-09-16T14:42:27.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f8773fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x55e0894bc620 0x55e089553fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48230/0 (socket says 172.21.15.26:48230) 2024-09-16T14:42:27.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f8773fff640 1 -- 172.21.15.26:0/3937433540 learned_addr learned my addr 172.21.15.26:0/3937433540 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:27.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 1 -- 172.21.15.26:0/3937433540 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118466715 0 0) 0x55e089351ab0 con 0x55e08954be50 2024-09-16T14:42:27.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:27.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:27.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:27.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient(hunting): my global_id is 4221 2024-09-16T14:42:27.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 1 -- 172.21.15.26:0/3937433540 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8768001e20 con 0x55e08954be50 2024-09-16T14:42:27.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 1 -- 172.21.15.26:0/3937433540 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621123776 0 0) 0x55e089337910 con 0x55e0894bc620 2024-09-16T14:42:27.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:27.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:27.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:27.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient(hunting): my global_id is 4220 2024-09-16T14:42:27.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 1 -- 172.21.15.26:0/3937433540 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x55e089351ab0 con 0x55e0894bc620 2024-09-16T14:42:27.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 1 -- 172.21.15.26:0/3937433540 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1436565516 0 0) 0x55e089351ab0 con 0x55e0894bc620 2024-09-16T14:42:27.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 1 -- 172.21.15.26:0/3937433540 >> v1:172.21.15.189:6789/0 conn(0x55e0895556e0 legacy=0x55e089557ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:27.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 1 -- 172.21.15.26:0/3937433540 >> v1:172.21.15.26:6789/0 conn(0x55e08954be50 legacy=0x55e08954c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:27.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:42:27.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 1 monclient: found mon.noname-b 2024-09-16T14:42:27.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:27.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:27.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 1 -- 172.21.15.26:0/3937433540 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e08932ad50 con 0x55e0894bc620 2024-09-16T14:42:27.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:42:27.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:42:27.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:27.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 1 -- 172.21.15.26:0/3937433540 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8764001580 con 0x55e0894bc620 2024-09-16T14:42:27.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 5 monclient: authenticate success, global_id 4220 2024-09-16T14:42:27.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:27.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:27.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:27.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient: dump: 2024-09-16T14:42:27.551 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:27.551 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:27.551 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.551 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.552 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:27.552 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:27.552 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:27.552 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:27.552 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:27.552 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:27.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:27.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 1 -- 172.21.15.26:0/3937433540 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8764001820 con 0x55e0894bc620 2024-09-16T14:42:27.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:27.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 1 -- 172.21.15.26:0/3937433540 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8764001bf0 con 0x55e0894bc620 2024-09-16T14:42:27.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:27.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:27.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f877b7fe640 10 monclient: dump: 2024-09-16T14:42:27.553 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:27.553 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:27.553 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.553 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.553 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:27.553 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:27.553 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:27.553 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:27.553 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:27.553 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:27.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 10 monclient: get_monmap_and_config success 2024-09-16T14:42:27.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 4 set_mon_vals no callback set 2024-09-16T14:42:27.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 10 monclient: shutdown 2024-09-16T14:42:27.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:27.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 1 -- 172.21.15.26:0/3937433540 >> v1:172.21.15.26:6790/0 conn(0x55e0894bc620 legacy=0x55e089553fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:27.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 1 -- 172.21.15.26:0/3937433540 shutdown_connections 2024-09-16T14:42:27.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 1 -- 172.21.15.26:0/3937433540 >> 172.21.15.26:0/3937433540 conn(0x55e08949f1e0 msgr2=0x55e08949f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:27.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 1 -- 172.21.15.26:0/3937433540 shutdown_connections 2024-09-16T14:42:27.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.544+0000 7f878161de00 1 -- 172.21.15.26:0/3937433540 wait complete. 2024-09-16T14:42:27.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 monclient: build_initial_monmap 2024-09-16T14:42:27.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 monclient: monmap: 2024-09-16T14:42:27.558 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:27.558 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:27.558 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:27.558 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:27.558 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:27.558 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:27.558 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:27.558 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:27.558 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:27.558 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:27.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 1 Processor -- start 2024-09-16T14:42:27.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 1 -- start start 2024-09-16T14:42:27.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 monclient: init 2024-09-16T14:42:27.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 monclient: _reopen_session rank -1 2024-09-16T14:42:27.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:27.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 monclient(hunting): picked mon.noname-c con 0x55e0894bc620 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:27.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 monclient(hunting): picked mon.noname-b con 0x55e08954be50 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:27.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 monclient(hunting): picked mon.noname-a con 0x55e0895556e0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:27.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55e08932a7b0 con 0x55e0895556e0 2024-09-16T14:42:27.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55e089554840 con 0x55e08954be50 2024-09-16T14:42:27.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55e089550bb0 con 0x55e0894bc620 2024-09-16T14:42:27.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 monclient(hunting): _renew_subs 2024-09-16T14:42:27.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 monclient(hunting): authenticate will time out at 971.669505s 2024-09-16T14:42:27.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f877bfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x55e0894bc620 0x55e08954fd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48240/0 (socket says 172.21.15.26:48240) 2024-09-16T14:42:27.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f877bfff640 1 -- 172.21.15.26:0/599470966 learned_addr learned my addr 172.21.15.26:0/599470966 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:27.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941833156 0 0) 0x55e089550bb0 con 0x55e0894bc620 2024-09-16T14:42:27.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:27.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:27.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:27.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): my global_id is 4223 2024-09-16T14:42:27.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f875c001e20 con 0x55e0894bc620 2024-09-16T14:42:27.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996765678 0 0) 0x55e08932a7b0 con 0x55e0895556e0 2024-09-16T14:42:27.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:27.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:27.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:27.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): my global_id is 4224 2024-09-16T14:42:27.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x55e089550bb0 con 0x55e0895556e0 2024-09-16T14:42:27.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2150072708 0 0) 0x7f875c001e20 con 0x55e0894bc620 2024-09-16T14:42:27.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x55e08932a7b0 con 0x55e0894bc620 2024-09-16T14:42:27.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f876c0019f0 con 0x55e0894bc620 2024-09-16T14:42:27.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:27.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:27.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:27.561 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:27.561 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:27.561 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.561 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.561 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:27.561 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:27.561 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:27.561 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:27.561 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:27.561 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:27.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:27.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288037767 0 0) 0x55e08932ad50 con 0x55e08954be50 2024-09-16T14:42:27.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:27.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:27.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:27.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): my global_id is 4225 2024-09-16T14:42:27.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f875c001e20 con 0x55e08954be50 2024-09-16T14:42:27.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3183952121 0 0) 0x55e089550bb0 con 0x55e0895556e0 2024-09-16T14:42:27.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x55e08932ad50 con 0x55e0895556e0 2024-09-16T14:42:27.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8774001b00 con 0x55e0895556e0 2024-09-16T14:42:27.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:27.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:27.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:27.562 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:27.562 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:27.563 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.563 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.563 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:27.563 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:27.563 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:27.563 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:27.563 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:27.563 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:27.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2241085064 0 0) 0x55e08932a7b0 con 0x55e0894bc620 2024-09-16T14:42:27.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 >> v1:172.21.15.189:6789/0 conn(0x55e08954be50 legacy=0x55e0895504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:27.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 >> v1:172.21.15.26:6789/0 conn(0x55e0895556e0 legacy=0x55e089565fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:27.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:42:27.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 monclient: found mon.c 2024-09-16T14:42:27.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:27.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:27.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e08949dc40 con 0x55e0894bc620 2024-09-16T14:42:27.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient: _finish_auth 0 2024-09-16T14:42:27.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:42:27.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:27.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 5 monclient: authenticate success, global_id 4223 2024-09-16T14:42:27.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 monclient: _renew_subs 2024-09-16T14:42:27.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:27.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e089548f90 con 0x55e0894bc620 2024-09-16T14:42:27.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 monclient: _renew_subs 2024-09-16T14:42:27.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:27.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x55e0895520f0 con 0x55e0894bc620 2024-09-16T14:42:27.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f876c001ee0 con 0x55e0894bc620 2024-09-16T14:42:27.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:27.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f876c002270 con 0x55e0894bc620 2024-09-16T14:42:27.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:27.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:27.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 10 monclient: dump: 2024-09-16T14:42:27.565 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:27.565 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:27.565 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.565 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:27.565 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:27.565 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:27.565 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:27.565 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:27.565 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:27.566 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:27.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f877affd640 4 set_mon_vals no callback set 2024-09-16T14:42:27.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 43216+0+0 (unknown 377793617 0 0) 0x7f876c002450 con 0x55e0894bc620 2024-09-16T14:42:27.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 3109+0+0 (unknown 2275674557 0 0) 0x7f876c041820 con 0x55e0894bc620 2024-09-16T14:42:27.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T14:42:27.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T14:42:27.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:27.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.548+0000 7f878161de00 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) v1 -- 0x55e089551200 con 0x55e0894bc620 2024-09-16T14:42:27.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:27.552+0000 7f8773fff640 10 monclient: get_auth_request con 0x7f875c034cb0 auth_method 0 2024-09-16T14:42:28.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.472+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v13) v1 ==== 141+0+0 (unknown 853630634 0 0) 0x7f876c012e40 con 0x55e0894bc620 2024-09-16T14:42:28.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.472+0000 7f8778ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T14:42:28.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.472+0000 7f8778ff9640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-16T14:42:28.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.472+0000 7f878161de00 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:28.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.472+0000 7f878161de00 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.26:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x55e089352650 con 0x55e0894bc620 2024-09-16T14:42:28.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.472+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (unknown 1318990235 0 0) 0x7f876c0139a0 con 0x55e0894bc620 2024-09-16T14:42:28.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.472+0000 7f8778ff9640 10 monclient: handle_get_version_reply finishing 1 version 13 2024-09-16T14:42:28.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.472+0000 7f877affd640 10 monclient: _renew_subs 2024-09-16T14:42:28.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.472+0000 7f877affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:28.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.472+0000 7f877affd640 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=13}) v3 -- 0x7f8760002af0 con 0x55e0894bc620 2024-09-16T14:42:28.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.472+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.2 v1:172.21.15.26:6790/0 11 ==== osd_map(13..13 src has 1..13) v4 ==== 1033+0+0 (unknown 3741648649 0 0) 0x7f876c0132a0 con 0x55e0894bc620 2024-09-16T14:42:28.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.472+0000 7f878161de00 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.26:6800/3032704211 -- osd_op(unknown.0.0:1 1.3 1:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13) v8 -- 0x55e08956bd50 con 0x55e089568240 2024-09-16T14:42:28.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.472+0000 7f878139a640 10 monclient: get_auth_request con 0x55e089568240 auth_method 0 2024-09-16T14:42:28.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.488+0000 7f878139a640 1 -- 172.21.15.26:0/599470966 <== osd.1 v1:172.21.15.26:6800/3032704211 1 ==== osd_op_reply(1 rbd_trash [create] v13'1 uv1 ondisk = 0) v8 ==== 153+0+0 (unknown 96784869 0 0) 0x7f8774004420 con 0x55e089568240 2024-09-16T14:42:28.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.492+0000 7f878161de00 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.189:6800/1782891661 -- osd_op(unknown.0.0:2 1.2 1:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13) v8 -- 0x55e0895bfff0 con 0x55e0895bc3f0 2024-09-16T14:42:28.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.492+0000 7f877bfff640 10 monclient: get_auth_request con 0x55e0895bc3f0 auth_method 0 2024-09-16T14:42:28.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.492+0000 7f877bfff640 1 -- 172.21.15.26:0/599470966 <== osd.4 v1:172.21.15.189:6800/1782891661 1 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (unknown 2901837046 0 0) 0x7f876c00f4f0 con 0x55e0895bc3f0 2024-09-16T14:42:28.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.492+0000 7f878161de00 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.189:6800/1782891661 -- osd_op(unknown.0.0:3 1.2 1:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13) v8 -- 0x55e0895bae80 con 0x55e0895bc3f0 2024-09-16T14:42:28.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.492+0000 7f877bfff640 1 -- 172.21.15.26:0/599470966 <== osd.4 v1:172.21.15.189:6800/1782891661 2 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (unknown 2913369850 0 0) 0x7f876c00f4f0 con 0x55e0895bc3f0 2024-09-16T14:42:28.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.492+0000 7f877a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:28.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:28.492+0000 7f877a7fc640 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.26:6790/0 -- pool_op(create unmanaged snap pool 1 tid 4 name v0) v4 -- 0x7f8754001f30 con 0x55e0894bc620 2024-09-16T14:42:29.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:29.476+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.2 v1:172.21.15.26:6790/0 12 ==== pool_op_reply(tid 4 (0) Success v14) v1 ==== 55+0+0 (unknown 2241887195 0 0) 0x7f876c0114e0 con 0x55e0894bc620 2024-09-16T14:42:29.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:29.476+0000 7f8778ff9640 10 monclient: _renew_subs 2024-09-16T14:42:29.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:29.476+0000 7f8778ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:29.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:29.476+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=14}) v3 -- 0x55e08932a7b0 con 0x55e0894bc620 2024-09-16T14:42:29.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:29.476+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.2 v1:172.21.15.26:6790/0 13 ==== osd_map(14..14 src has 1..14) v4 ==== 1004+0+0 (unknown 3846915877 0 0) 0x7f876c00fa10 con 0x55e0894bc620 2024-09-16T14:42:29.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:29.476+0000 7f877affd640 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.189:6800/1782891661 -- osd_op(unknown.0.0:5 1.2 1:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14) v8 -- 0x7f8760005610 con 0x55e0895bc3f0 2024-09-16T14:42:29.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:29.488+0000 7f877bfff640 1 -- 172.21.15.26:0/599470966 <== osd.4 v1:172.21.15.189:6800/1782891661 3 ==== osd_op_reply(5 rbd_info [create,write 0~8] v14'1 uv1 ondisk = 0) v8 ==== 194+0+0 (unknown 3957714342 0 0) 0x7f876c00f4f0 con 0x55e0895bc3f0 2024-09-16T14:42:29.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:29.488+0000 7f877a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:29.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:29.488+0000 7f877a7fc640 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.26:6790/0 -- pool_op(delete unmanaged snap pool 1 tid 6 name v14) v4 -- 0x7f87540033c0 con 0x55e0894bc620 2024-09-16T14:42:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.548+0000 7f8772ffd640 10 monclient: tick 2024-09-16T14:42:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.548+0000 7f8772ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.548+0000 7f8772ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.548+0000 7f8772ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.572+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.2 v1:172.21.15.26:6790/0 14 ==== pool_op_reply(tid 6 (0) Success v15) v1 ==== 43+0+0 (unknown 1187503688 0 0) 0x7f876c012c60 con 0x55e0894bc620 2024-09-16T14:42:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.572+0000 7f8778ff9640 10 monclient: _renew_subs 2024-09-16T14:42:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.572+0000 7f8778ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.572+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=15}) v3 -- 0x55e089550bb0 con 0x55e0894bc620 2024-09-16T14:42:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.572+0000 7f8778ff9640 1 -- 172.21.15.26:0/599470966 <== mon.2 v1:172.21.15.26:6790/0 15 ==== osd_map(15..15 src has 1..15) v4 ==== 1024+0+0 (unknown 323372932 0 0) 0x7f876c013500 con 0x55e0894bc620 2024-09-16T14:42:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.572+0000 7f877affd640 1 -- 172.21.15.26:0/599470966 --> v1:172.21.15.189:6800/1782891661 -- osd_op(unknown.0.0:7 1.2 1:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15) v8 -- 0x7f87600059e0 con 0x55e0895bc3f0 2024-09-16T14:42:30.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.580+0000 7f877bfff640 1 -- 172.21.15.26:0/599470966 <== osd.4 v1:172.21.15.189:6800/1782891661 4 ==== osd_op_reply(7 rbd_info [write 0~19] v15'2 uv2 ondisk = 0) v8 ==== 152+0+0 (unknown 1441742725 0 0) 0x7f876c00f4f0 con 0x55e0895bc3f0 2024-09-16T14:42:30.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.580+0000 7f878161de00 1 -- 172.21.15.26:0/599470966 >> v1:172.21.15.26:6800/3032704211 conn(0x55e089568240 legacy=0x55e08956a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.580+0000 7f878161de00 1 -- 172.21.15.26:0/599470966 >> v1:172.21.15.189:6800/1782891661 conn(0x55e0895bc3f0 legacy=0x55e0895be880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.580+0000 7f878161de00 1 -- 172.21.15.26:0/599470966 >> v1:172.21.15.189:6812/3196161582 conn(0x7f875c034cb0 legacy=0x7f875c037170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.580+0000 7f878161de00 10 monclient: shutdown 2024-09-16T14:42:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.580+0000 7f878161de00 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.580+0000 7f878161de00 1 -- 172.21.15.26:0/599470966 >> v1:172.21.15.26:6790/0 conn(0x55e0894bc620 legacy=0x55e08954fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.580+0000 7f878139a640 1 -- 172.21.15.26:0/599470966 reap_dead start 2024-09-16T14:42:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.580+0000 7f878161de00 1 -- 172.21.15.26:0/599470966 shutdown_connections 2024-09-16T14:42:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.580+0000 7f878161de00 1 -- 172.21.15.26:0/599470966 >> 172.21.15.26:0/599470966 conn(0x55e08949f1e0 msgr2=0x55e089547e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.580+0000 7f878161de00 1 -- 172.21.15.26:0/599470966 shutdown_connections 2024-09-16T14:42:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.580+0000 7f878161de00 1 -- 172.21.15.26:0/599470966 wait complete. 2024-09-16T14:42:30.587 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T14:42:30.587 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T14:42:30.587 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T14:42:30.588 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T14:42:30.588 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T14:42:30.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.708+0000 7f1f2e045640 10 monclient: get_monmap_and_config 2024-09-16T14:42:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.708+0000 7f1f2e045640 10 monclient: build_initial_monmap 2024-09-16T14:42:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.708+0000 7f1f2e045640 10 monclient: monmap: 2024-09-16T14:42:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:30.712664+0000 2024-09-16T14:42:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:30.712664+0000 2024-09-16T14:42:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:30.712 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:30.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.708+0000 7f1f2e045640 1 Processor -- start 2024-09-16T14:42:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 1 -- start start 2024-09-16T14:42:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 10 monclient: init 2024-09-16T14:42:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:42:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 10 monclient(hunting): picked mon.noname-b con 0x7f1f280a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 10 monclient(hunting): picked mon.noname-a con 0x7f1f28099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 10 monclient(hunting): picked mon.noname-c con 0x7f1f280a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f28098ea0 con 0x7f1f28099b60 2024-09-16T14:42:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f280997b0 con 0x7f1f280a26f0 2024-09-16T14:42:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f28099460 con 0x7f1f280a1c00 2024-09-16T14:42:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 10 monclient(hunting): authenticate will time out at 974.832854s 2024-09-16T14:42:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f277fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1f280a1c00 0x7f1f280a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48242/0 (socket says 172.21.15.26:48242) 2024-09-16T14:42:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f277fe640 1 -- 172.21.15.26:0/1544752023 learned_addr learned my addr 172.21.15.26:0/1544752023 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f26ffd640 1 --1- 172.21.15.26:0/1544752023 >> v1:172.21.15.26:6789/0 conn(0x7f1f28099b60 0x7f1f28139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57202/0 (socket says 172.21.15.26:57202) 2024-09-16T14:42:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 1 -- 172.21.15.26:0/1544752023 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554676915 0 0) 0x7f1f28098ea0 con 0x7f1f28099b60 2024-09-16T14:42:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient(hunting): _negotiate 2024-09-16T14:42:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient(hunting): my global_id is 4236 2024-09-16T14:42:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 1 -- 172.21.15.26:0/1544752023 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f10003530 con 0x7f1f28099b60 2024-09-16T14:42:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 1 -- 172.21.15.26:0/1544752023 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601138247 0 0) 0x7f1f280997b0 con 0x7f1f280a26f0 2024-09-16T14:42:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient(hunting): _negotiate 2024-09-16T14:42:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient(hunting): my global_id is 4237 2024-09-16T14:42:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 1 -- 172.21.15.26:0/1544752023 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f28098ea0 con 0x7f1f280a26f0 2024-09-16T14:42:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 1 -- 172.21.15.26:0/1544752023 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 566857035 0 0) 0x7f1f10003530 con 0x7f1f28099b60 2024-09-16T14:42:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 1 -- 172.21.15.26:0/1544752023 >> v1:172.21.15.26:6790/0 conn(0x7f1f280a1c00 legacy=0x7f1f280a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 1 -- 172.21.15.26:0/1544752023 >> v1:172.21.15.189:6789/0 conn(0x7f1f280a26f0 legacy=0x7f1f2813d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient: _finish_hunting 0 2024-09-16T14:42:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 1 monclient: found mon.noname-a 2024-09-16T14:42:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:42:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 1 -- 172.21.15.26:0/1544752023 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f280a34b0 con 0x7f1f28099b60 2024-09-16T14:42:30.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient: _finish_auth 0 2024-09-16T14:42:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient: _check_auth_tickets 2024-09-16T14:42:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 1 -- 172.21.15.26:0/1544752023 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1f1c001620 con 0x7f1f28099b60 2024-09-16T14:42:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient: dump: 2024-09-16T14:42:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:30.723 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:30.724 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:30.724 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:30.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 5 monclient: authenticate success, global_id 4236 2024-09-16T14:42:30.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:30.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 1 -- 172.21.15.26:0/1544752023 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f1c001880 con 0x7f1f28099b60 2024-09-16T14:42:30.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:30.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 1 -- 172.21.15.26:0/1544752023 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1f1c001c50 con 0x7f1f28099b60 2024-09-16T14:42:30.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:30.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:30.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f267fc640 10 monclient: dump: 2024-09-16T14:42:30.725 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:30.725 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:30.725 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:30.725 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:30.725 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:30.725 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:30.725 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:30.725 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:30.725 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:30.725 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:30.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:30.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 4 set_mon_vals no callback set 2024-09-16T14:42:30.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 10 monclient: shutdown 2024-09-16T14:42:30.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:30.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 1 -- 172.21.15.26:0/1544752023 >> v1:172.21.15.26:6789/0 conn(0x7f1f28099b60 legacy=0x7f1f28139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:30.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 1 -- 172.21.15.26:0/1544752023 shutdown_connections 2024-09-16T14:42:30.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 1 -- 172.21.15.26:0/1544752023 >> 172.21.15.26:0/1544752023 conn(0x7f1f280a3d00 msgr2=0x7f1f280a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:30.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 1 -- 172.21.15.26:0/1544752023 shutdown_connections 2024-09-16T14:42:30.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 1 -- 172.21.15.26:0/1544752023 wait complete. 2024-09-16T14:42:30.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 10 monclient: build_initial_monmap 2024-09-16T14:42:30.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 10 monclient: monmap: 2024-09-16T14:42:30.726 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:30.726 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:30.726 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:30.726 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:30.726 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:30.726 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:30.727 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:30.727 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:30.727 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:30.727 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:30.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 1 Processor -- start 2024-09-16T14:42:30.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 1 -- start start 2024-09-16T14:42:30.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.712+0000 7f1f2e045640 10 monclient: init 2024-09-16T14:42:30.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:30.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:42:30.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 10 monclient(hunting): picked mon.noname-c con 0x7f1f28099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:30.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 10 monclient(hunting): picked mon.noname-a con 0x7f1f280a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:30.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 10 monclient(hunting): picked mon.noname-b con 0x7f1f280a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:30.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2809e3f0 con 0x7f1f280a1c00 2024-09-16T14:42:30.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2813dd40 con 0x7f1f280a26f0 2024-09-16T14:42:30.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2813abe0 con 0x7f1f28099b60 2024-09-16T14:42:30.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:30.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 10 monclient(hunting): authenticate will time out at 974.836123s 2024-09-16T14:42:30.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f277fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1f28099b60 0x7f1f28139dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48254/0 (socket says 172.21.15.26:48254) 2024-09-16T14:42:30.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f277fe640 1 -- 172.21.15.26:0/1586315760 learned_addr learned my addr 172.21.15.26:0/1586315760 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:30.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3591871952 0 0) 0x7f1f2809e3f0 con 0x7f1f280a1c00 2024-09-16T14:42:30.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:30.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:30.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:30.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient(hunting): my global_id is 4239 2024-09-16T14:42:30.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f14003530 con 0x7f1f280a1c00 2024-09-16T14:42:30.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225463709 0 0) 0x7f1f2813dd40 con 0x7f1f280a26f0 2024-09-16T14:42:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient(hunting): my global_id is 4240 2024-09-16T14:42:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f2809e3f0 con 0x7f1f280a26f0 2024-09-16T14:42:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 768250973 0 0) 0x7f1f14003530 con 0x7f1f280a1c00 2024-09-16T14:42:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f2813dd40 con 0x7f1f280a1c00 2024-09-16T14:42:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1f1c001b90 con 0x7f1f280a1c00 2024-09-16T14:42:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient(hunting): dump: 2024-09-16T14:42:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:30.730 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:30.730 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:30.730 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:30.730 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:30.730 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:30.730 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:30.730 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:30.731 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4156247726 0 0) 0x7f1f2809e3f0 con 0x7f1f280a26f0 2024-09-16T14:42:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f14003530 con 0x7f1f280a26f0 2024-09-16T14:42:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1f20001560 con 0x7f1f280a26f0 2024-09-16T14:42:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient(hunting): dump: 2024-09-16T14:42:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:30.732 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4192951852 0 0) 0x7f1f2813dd40 con 0x7f1f280a1c00 2024-09-16T14:42:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 >> v1:172.21.15.26:6790/0 conn(0x7f1f28099b60 legacy=0x7f1f28139dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 >> v1:172.21.15.189:6789/0 conn(0x7f1f280a26f0 legacy=0x7f1f281d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:42:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 monclient: found mon.a 2024-09-16T14:42:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2809eb40 con 0x7f1f280a1c00 2024-09-16T14:42:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient: _finish_auth 0 2024-09-16T14:42:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:42:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 5 monclient: authenticate success, global_id 4239 2024-09-16T14:42:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 10 monclient: _renew_subs 2024-09-16T14:42:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 1 -- 172.21.15.26:0/1586315760 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2813c650 con 0x7f1f280a1c00 2024-09-16T14:42:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 10 monclient: _renew_subs 2024-09-16T14:42:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 1 -- 172.21.15.26:0/1586315760 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2813d4a0 con 0x7f1f280a1c00 2024-09-16T14:42:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f1c001200 con 0x7f1f280a1c00 2024-09-16T14:42:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1f1c004ce0 con 0x7f1f280a1c00 2024-09-16T14:42:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f25ffb640 4 set_mon_vals no callback set 2024-09-16T14:42:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 10 monclient: dump: 2024-09-16T14:42:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:30.735 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f2e045640 1 -- 172.21.15.26:0/1586315760 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f08005050 con 0x7f1f280a1c00 2024-09-16T14:42:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 43216+0+0 (unknown 377793617 0 0) 0x7f1f1c00f720 con 0x7f1f280a1c00 2024-09-16T14:42:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f1f1c0430a0 con 0x7f1f280a1c00 2024-09-16T14:42:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.716+0000 7f1f277fe640 10 monclient: get_auth_request con 0x7f1f14036200 auth_method 0 2024-09-16T14:42:30.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.720+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1f1c014640 con 0x7f1f280a1c00 2024-09-16T14:42:30.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.720+0000 7f1f0ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:30.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.720+0000 7f1f0ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:30.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.924+0000 7f1f2e045640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-16T14:42:30.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.924+0000 7f1f2e045640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T14:42:30.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.924+0000 7f1f2e045640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:30.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.924+0000 7f1f2e045640 1 -- 172.21.15.26:0/1586315760 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f1f08005300 con 0x7f1f280a1c00 2024-09-16T14:42:30.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.928+0000 7f1f0ffff640 1 -- 172.21.15.26:0/1586315760 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+82966 (unknown 3797009233 0 33965342) 0x7f1f1c012ce0 con 0x7f1f280a1c00 2024-09-16T14:42:30.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.928+0000 7f1f0ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T14:42:30.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.928+0000 7f1f0ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:30.962 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:42:30.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.932+0000 7f1f2e045640 1 -- 172.21.15.26:0/1586315760 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1f14036200 legacy=0x7f1f140386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:30.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.932+0000 7f1f2e045640 10 monclient: shutdown 2024-09-16T14:42:30.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.932+0000 7f1f2e045640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:30.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.932+0000 7f1f2e045640 1 -- 172.21.15.26:0/1586315760 >> v1:172.21.15.26:6789/0 conn(0x7f1f280a1c00 legacy=0x7f1f2813a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:30.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.932+0000 7f1f2e045640 1 -- 172.21.15.26:0/1586315760 shutdown_connections 2024-09-16T14:42:30.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.932+0000 7f1f2e045640 1 -- 172.21.15.26:0/1586315760 >> 172.21.15.26:0/1586315760 conn(0x7f1f280a3d00 msgr2=0x7f1f280a0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:30.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.932+0000 7f1f2e045640 1 -- 172.21.15.26:0/1586315760 shutdown_connections 2024-09-16T14:42:30.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:30.932+0000 7f1f2e045640 1 -- 172.21.15.26:0/1586315760 wait complete. 2024-09-16T14:42:30.964 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":4,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6812","nonce":3196161582}]},"active_addr":"172.21.15.189:6812/3196161582","active_change":"2024-09-16T14:42:22.367882+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":4135457260}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":4278905168}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":424634794}]}]} 2024-09-16T14:42:30.966 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T14:42:30.967 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T14:42:30.967 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T14:42:31.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.080+0000 7f82e9c6d640 10 monclient: get_monmap_and_config 2024-09-16T14:42:31.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.080+0000 7f82e9c6d640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.080+0000 7f82e9c6d640 10 monclient: monmap: 2024-09-16T14:42:31.084 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.084 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.085 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:31.084502+0000 2024-09-16T14:42:31.085 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:31.084502+0000 2024-09-16T14:42:31.085 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.085 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.085 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.085 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:31.085 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:31.085 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.080+0000 7f82e9c6d640 1 Processor -- start 2024-09-16T14:42:31.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.080+0000 7f82e9c6d640 1 -- start start 2024-09-16T14:42:31.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.080+0000 7f82e9c6d640 10 monclient: init 2024-09-16T14:42:31.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:31.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 10 monclient(hunting): picked mon.noname-a con 0x7f82e40a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 10 monclient(hunting): picked mon.noname-b con 0x7f82e4099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 10 monclient(hunting): picked mon.noname-c con 0x7f82e40a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e4098ea0 con 0x7f82e40a1c00 2024-09-16T14:42:31.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e40997b0 con 0x7f82e40a26f0 2024-09-16T14:42:31.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e4099460 con 0x7f82e4099b60 2024-09-16T14:42:31.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 10 monclient(hunting): authenticate will time out at 975.204625s 2024-09-16T14:42:31.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e2ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f82e4099b60 0x7f82e4139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48260/0 (socket says 172.21.15.26:48260) 2024-09-16T14:42:31.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e37fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f82e40a1c00 0x7f82e40a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57230/0 (socket says 172.21.15.26:57230) 2024-09-16T14:42:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e2ffd640 1 -- 172.21.15.26:0/3407546609 learned_addr learned my addr 172.21.15.26:0/3407546609 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 1 -- 172.21.15.26:0/3407546609 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112217496 0 0) 0x7f82e4099460 con 0x7f82e4099b60 2024-09-16T14:42:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient(hunting): my global_id is 4235 2024-09-16T14:42:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 1 -- 172.21.15.26:0/3407546609 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d0003530 con 0x7f82e4099b60 2024-09-16T14:42:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 1 -- 172.21.15.26:0/3407546609 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065941160 0 0) 0x7f82e4098ea0 con 0x7f82e40a1c00 2024-09-16T14:42:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient(hunting): my global_id is 4242 2024-09-16T14:42:31.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 1 -- 172.21.15.26:0/3407546609 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82e4099460 con 0x7f82e40a1c00 2024-09-16T14:42:31.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 1 -- 172.21.15.26:0/3407546609 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2861812856 0 0) 0x7f82d0003530 con 0x7f82e4099b60 2024-09-16T14:42:31.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 1 -- 172.21.15.26:0/3407546609 >> v1:172.21.15.189:6789/0 conn(0x7f82e40a26f0 legacy=0x7f82e413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 1 -- 172.21.15.26:0/3407546609 >> v1:172.21.15.26:6789/0 conn(0x7f82e40a1c00 legacy=0x7f82e40a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 1 monclient: found mon.noname-b 2024-09-16T14:42:31.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 1 -- 172.21.15.26:0/3407546609 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e40a34b0 con 0x7f82e4099b60 2024-09-16T14:42:31.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 1 -- 172.21.15.26:0/3407546609 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f82cc0019a0 con 0x7f82e4099b60 2024-09-16T14:42:31.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:31.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient: dump: 2024-09-16T14:42:31.097 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.097 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.098 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.098 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.098 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.098 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.098 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.098 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.098 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.098 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 5 monclient: authenticate success, global_id 4235 2024-09-16T14:42:31.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:31.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 1 -- 172.21.15.26:0/3407546609 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82cc001c40 con 0x7f82e4099b60 2024-09-16T14:42:31.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 1 -- 172.21.15.26:0/3407546609 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f82cc002010 con 0x7f82e4099b60 2024-09-16T14:42:31.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:31.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e27fc640 10 monclient: dump: 2024-09-16T14:42:31.099 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.099 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.099 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.099 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.099 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.099 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.100 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.100 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.100 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.100 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:31.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 4 set_mon_vals no callback set 2024-09-16T14:42:31.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 10 monclient: shutdown 2024-09-16T14:42:31.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 1 -- 172.21.15.26:0/3407546609 >> v1:172.21.15.26:6790/0 conn(0x7f82e4099b60 legacy=0x7f82e4139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 1 -- 172.21.15.26:0/3407546609 shutdown_connections 2024-09-16T14:42:31.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 1 -- 172.21.15.26:0/3407546609 >> 172.21.15.26:0/3407546609 conn(0x7f82e40a3d00 msgr2=0x7f82e40a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 1 -- 172.21.15.26:0/3407546609 shutdown_connections 2024-09-16T14:42:31.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 1 -- 172.21.15.26:0/3407546609 wait complete. 2024-09-16T14:42:31.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 10 monclient: monmap: 2024-09-16T14:42:31.101 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.101 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:31.101 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:31.101 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:31.101 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.101 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:31.101 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.101 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:31.101 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:31.102 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 1 Processor -- start 2024-09-16T14:42:31.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 1 -- start start 2024-09-16T14:42:31.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.084+0000 7f82e9c6d640 10 monclient: init 2024-09-16T14:42:31.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:31.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 10 monclient(hunting): picked mon.noname-a con 0x7f82e4099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 10 monclient(hunting): picked mon.noname-c con 0x7f82e40a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 10 monclient(hunting): picked mon.noname-b con 0x7f82e40a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e411e3c0 con 0x7f82e4099b60 2024-09-16T14:42:31.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e411e700 con 0x7f82e40a26f0 2024-09-16T14:42:31.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e409f0c0 con 0x7f82e40a1c00 2024-09-16T14:42:31.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 10 monclient(hunting): authenticate will time out at 975.208388s 2024-09-16T14:42:31.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e37fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f82e4099b60 0x7f82e413cd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57240/0 (socket says 172.21.15.26:57240) 2024-09-16T14:42:31.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e37fe640 1 -- 172.21.15.26:0/3886007811 learned_addr learned my addr 172.21.15.26:0/3886007811 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e2ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f82e40a1c00 0x7f82e413d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48274/0 (socket says 172.21.15.26:48274) 2024-09-16T14:42:31.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1143837270 0 0) 0x7f82e409f0c0 con 0x7f82e40a1c00 2024-09-16T14:42:31.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): my global_id is 4238 2024-09-16T14:42:31.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82c4003530 con 0x7f82e40a1c00 2024-09-16T14:42:31.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306719643 0 0) 0x7f82e411e3c0 con 0x7f82e4099b60 2024-09-16T14:42:31.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): my global_id is 4245 2024-09-16T14:42:31.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82e409f0c0 con 0x7f82e4099b60 2024-09-16T14:42:31.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2755940008 0 0) 0x7f82c4003530 con 0x7f82e40a1c00 2024-09-16T14:42:31.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82e411e3c0 con 0x7f82e40a1c00 2024-09-16T14:42:31.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f82cc001f50 con 0x7f82e40a1c00 2024-09-16T14:42:31.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:31.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): dump: 2024-09-16T14:42:31.105 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.105 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.105 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.105 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.105 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.106 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.106 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.106 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.106 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.106 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:31.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885541524 0 0) 0x7f82e411e700 con 0x7f82e40a26f0 2024-09-16T14:42:31.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): my global_id is 4246 2024-09-16T14:42:31.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82c4003530 con 0x7f82e40a26f0 2024-09-16T14:42:31.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1906334780 0 0) 0x7f82e409f0c0 con 0x7f82e4099b60 2024-09-16T14:42:31.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82e411e700 con 0x7f82e4099b60 2024-09-16T14:42:31.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f82d4001100 con 0x7f82e4099b60 2024-09-16T14:42:31.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:31.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient(hunting): dump: 2024-09-16T14:42:31.107 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.107 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.107 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.107 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.107 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.108 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3443592809 0 0) 0x7f82e411e3c0 con 0x7f82e40a1c00 2024-09-16T14:42:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 >> v1:172.21.15.189:6789/0 conn(0x7f82e40a26f0 legacy=0x7f82e413db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 >> v1:172.21.15.26:6789/0 conn(0x7f82e4099b60 legacy=0x7f82e413cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 monclient: found mon.c 2024-09-16T14:42:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e411f870 con 0x7f82e40a1c00 2024-09-16T14:42:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 5 monclient: authenticate success, global_id 4238 2024-09-16T14:42:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 10 monclient: _renew_subs 2024-09-16T14:42:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 1 -- 172.21.15.26:0/3886007811 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e411fb50 con 0x7f82e40a1c00 2024-09-16T14:42:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 10 monclient: _renew_subs 2024-09-16T14:42:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 1 -- 172.21.15.26:0/3886007811 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e409eb70 con 0x7f82e40a1c00 2024-09-16T14:42:31.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82cc0013f0 con 0x7f82e40a1c00 2024-09-16T14:42:31.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f82cc005120 con 0x7f82e40a1c00 2024-09-16T14:42:31.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e1ffb640 4 set_mon_vals no callback set 2024-09-16T14:42:31.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:31.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 10 monclient: dump: 2024-09-16T14:42:31.110 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.110 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.110 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.111 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.111 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.111 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.111 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.111 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.111 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.111 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:31.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:31.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e9c6d640 1 -- 172.21.15.26:0/3886007811 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c8005050 con 0x7f82e40a1c00 2024-09-16T14:42:31.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 43216+0+0 (unknown 377793617 0 0) 0x7f82cc00fb60 con 0x7f82e40a1c00 2024-09-16T14:42:31.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f82cc0433f0 con 0x7f82e40a1c00 2024-09-16T14:42:31.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.088+0000 7f82e37fe640 10 monclient: get_auth_request con 0x7f82c40364b0 auth_method 0 2024-09-16T14:42:31.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.092+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f82cc014910 con 0x7f82e40a1c00 2024-09-16T14:42:31.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.092+0000 7f82dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:31.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.092+0000 7f82dbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:31.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.232+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f82cc041650 con 0x7f82e40a1c00 2024-09-16T14:42:31.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.256+0000 7f82e9c6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:31.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.256+0000 7f82e9c6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:31.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.256+0000 7f82e9c6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.256+0000 7f82e9c6d640 1 -- 172.21.15.26:0/3886007811 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82c8005300 con 0x7f82e40a1c00 2024-09-16T14:42:31.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.260+0000 7f82dbfff640 1 -- 172.21.15.26:0/3886007811 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+10842 (unknown 4221056075 0 2316752284) 0x7f82cc014910 con 0x7f82e40a1c00 2024-09-16T14:42:31.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.260+0000 7f82dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:31.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.260+0000 7f82dbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:31.262 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:42:31.262 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":15,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","created":"2024-09-16T14:42:17.361142+0000","modified":"2024-09-16T14:42:30.562817+0000","last_up_change":"2024-09-16T14:42:26.444878+0000","last_in_change":"2024-09-16T14:42:19.053975+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-16T14:42:27.380911+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-16T14:42:28.450753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a4578188-6e8b-430f-b70d-1c3480eb7bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6808","nonce":3182580250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6809","nonce":3182580250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6811","nonce":3182580250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6810","nonce":3182580250}]},"public_addr":"172.21.15.26:6808/3182580250","cluster_addr":"172.21.15.26:6809/3182580250","heartbeat_back_addr":"172.21.15.26:6811/3182580250","heartbeat_front_addr":"172.21.15.26:6810/3182580250","state":["exists","up"]},{"osd":1,"uuid":"1a96367f-73f1-4a99-9b85-47bf8e3d4cdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6800","nonce":3032704211}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6801","nonce":3032704211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6805","nonce":3032704211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6803","nonce":3032704211}]},"public_addr":"172.21.15.26:6800/3032704211","cluster_addr":"172.21.15.26:6801/3032704211","heartbeat_back_addr":"172.21.15.26:6805/3032704211","heartbeat_front_addr":"172.21.15.26:6803/3032704211","state":["exists","up"]},{"osd":2,"uuid":"3030f43d-f848-4189-be1e-6ea367941218","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6802","nonce":1811454603}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6804","nonce":1811454603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6807","nonce":1811454603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6806","nonce":1811454603}]},"public_addr":"172.21.15.26:6802/1811454603","cluster_addr":"172.21.15.26:6804/1811454603","heartbeat_back_addr":"172.21.15.26:6807/1811454603","heartbeat_front_addr":"172.21.15.26:6806/1811454603","state":["exists","up"]},{"osd":3,"uuid":"78731cf9-4ba7-4ce9-a483-080aa1136b65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6808","nonce":3651837298}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6809","nonce":3651837298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6811","nonce":3651837298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6810","nonce":3651837298}]},"public_addr":"172.21.15.189:6808/3651837298","cluster_addr":"172.21.15.189:6809/3651837298","heartbeat_back_addr":"172.21.15.189:6811/3651837298","heartbeat_front_addr":"172.21.15.189:6810/3651837298","state":["exists","up"]},{"osd":4,"uuid":"d74156df-be8d-4ad7-93dc-89ac28df70dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6800","nonce":1782891661}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6801","nonce":1782891661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6803","nonce":1782891661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6802","nonce":1782891661}]},"public_addr":"172.21.15.189:6800/1782891661","cluster_addr":"172.21.15.189:6801/1782891661","heartbeat_back_addr":"172.21.15.189:6803/1782891661","heartbeat_front_addr":"172.21.15.189:6802/1782891661","state":["exists","up"]},{"osd":5,"uuid":"ccad9c1e-247f-4ed4-8437-6aa5c8b515d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6804","nonce":3793314587}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6805","nonce":3793314587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6807","nonce":3793314587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6806","nonce":3793314587}]},"public_addr":"172.21.15.189:6804/3793314587","cluster_addr":"172.21.15.189:6805/3793314587","heartbeat_back_addr":"172.21.15.189:6807/3793314587","heartbeat_front_addr":"172.21.15.189:6806/3793314587","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.603081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.576634+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.472343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.602340+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.465228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.494237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T14:42:31.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.260+0000 7f82e9c6d640 1 -- 172.21.15.26:0/3886007811 >> v1:172.21.15.189:6812/3196161582 conn(0x7f82c40364b0 legacy=0x7f82c4038970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.260+0000 7f82e9c6d640 10 monclient: shutdown 2024-09-16T14:42:31.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.260+0000 7f82e9c6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.260+0000 7f82e9c6d640 1 -- 172.21.15.26:0/3886007811 >> v1:172.21.15.26:6790/0 conn(0x7f82e40a1c00 legacy=0x7f82e413d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.260+0000 7f82e9c6d640 1 -- 172.21.15.26:0/3886007811 shutdown_connections 2024-09-16T14:42:31.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.260+0000 7f82e9c6d640 1 -- 172.21.15.26:0/3886007811 >> 172.21.15.26:0/3886007811 conn(0x7f82e40a3d00 msgr2=0x7f82e40a0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.260+0000 7f82e9c6d640 1 -- 172.21.15.26:0/3886007811 shutdown_connections 2024-09-16T14:42:31.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.260+0000 7f82e9c6d640 1 -- 172.21.15.26:0/3886007811 wait complete. 2024-09-16T14:42:31.281 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T14:42:31.281 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T14:42:31.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87d23b7640 10 monclient: get_monmap_and_config 2024-09-16T14:42:31.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87d23b7640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87d23b7640 10 monclient: monmap: 2024-09-16T14:42:31.407 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.407 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.407 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:31.407803+0000 2024-09-16T14:42:31.407 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:31.407803+0000 2024-09-16T14:42:31.408 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.408 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.408 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.408 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:31.408 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:31.408 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87d23b7640 1 Processor -- start 2024-09-16T14:42:31.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87d23b7640 1 -- start start 2024-09-16T14:42:31.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87d23b7640 10 monclient: init 2024-09-16T14:42:31.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87d23b7640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87d23b7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:31.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87d23b7640 10 monclient(hunting): picked mon.noname-c con 0x7f87cc09a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87d23b7640 10 monclient(hunting): picked mon.noname-b con 0x7f87cc09d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87d23b7640 10 monclient(hunting): picked mon.noname-a con 0x7f87cc09b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87d23b7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc09dc60 con 0x7f87cc09b040 2024-09-16T14:42:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87d23b7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc099ae0 con 0x7f87cc09a550 2024-09-16T14:42:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87d23b7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc09b8f0 con 0x7f87cc09d1b0 2024-09-16T14:42:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87d23b7640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87d23b7640 10 monclient(hunting): authenticate will time out at 975.527868s 2024-09-16T14:42:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.404+0000 7f87cb7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f87cc09d1b0 0x7f87cc136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48290/0 (socket says 172.21.15.26:48290) 2024-09-16T14:42:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87cb7fe640 1 -- 172.21.15.26:0/3543535553 learned_addr learned my addr 172.21.15.26:0/3543535553 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 1 -- 172.21.15.26:0/3543535553 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344756660 0 0) 0x7f87cc09b8f0 con 0x7f87cc09d1b0 2024-09-16T14:42:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient(hunting): my global_id is 4241 2024-09-16T14:42:31.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 1 -- 172.21.15.26:0/3543535553 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b8003530 con 0x7f87cc09d1b0 2024-09-16T14:42:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 1 -- 172.21.15.26:0/3543535553 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085550434 0 0) 0x7f87cc09dc60 con 0x7f87cc09b040 2024-09-16T14:42:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient(hunting): my global_id is 4248 2024-09-16T14:42:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 1 -- 172.21.15.26:0/3543535553 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87cc09b8f0 con 0x7f87cc09b040 2024-09-16T14:42:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 1 -- 172.21.15.26:0/3543535553 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 197930865 0 0) 0x7f87b8003530 con 0x7f87cc09d1b0 2024-09-16T14:42:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 1 -- 172.21.15.26:0/3543535553 >> v1:172.21.15.189:6789/0 conn(0x7f87cc09a550 legacy=0x7f87cc09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 1 -- 172.21.15.26:0/3543535553 >> v1:172.21.15.26:6789/0 conn(0x7f87cc09b040 legacy=0x7f87cc13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 1 monclient: found mon.noname-b 2024-09-16T14:42:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 1 -- 172.21.15.26:0/3543535553 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87cc09bbd0 con 0x7f87cc09d1b0 2024-09-16T14:42:31.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 1 -- 172.21.15.26:0/3543535553 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f87b40019a0 con 0x7f87cc09d1b0 2024-09-16T14:42:31.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:31.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient: dump: 2024-09-16T14:42:31.421 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.421 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.421 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.421 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.421 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.422 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 5 monclient: authenticate success, global_id 4241 2024-09-16T14:42:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 1 -- 172.21.15.26:0/3543535553 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87b4001c00 con 0x7f87cc09d1b0 2024-09-16T14:42:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 1 -- 172.21.15.26:0/3543535553 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f87b4001fd0 con 0x7f87cc09d1b0 2024-09-16T14:42:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87caffd640 10 monclient: dump: 2024-09-16T14:42:31.422 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.423 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.423 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.423 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.423 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.423 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.423 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.423 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.423 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.423 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:31.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 4 set_mon_vals no callback set 2024-09-16T14:42:31.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 10 monclient: shutdown 2024-09-16T14:42:31.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 1 -- 172.21.15.26:0/3543535553 >> v1:172.21.15.26:6790/0 conn(0x7f87cc09d1b0 legacy=0x7f87cc136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 1 -- 172.21.15.26:0/3543535553 shutdown_connections 2024-09-16T14:42:31.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 1 -- 172.21.15.26:0/3543535553 >> 172.21.15.26:0/3543535553 conn(0x7f87cc09fb50 msgr2=0x7f87cc09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 1 -- 172.21.15.26:0/3543535553 shutdown_connections 2024-09-16T14:42:31.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 1 -- 172.21.15.26:0/3543535553 wait complete. 2024-09-16T14:42:31.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 10 monclient: monmap: 2024-09-16T14:42:31.424 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.424 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:31.424 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:31.424 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:31.424 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.424 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:31.424 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.425 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:31.425 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:31.425 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 1 Processor -- start 2024-09-16T14:42:31.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 1 -- start start 2024-09-16T14:42:31.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 10 monclient: init 2024-09-16T14:42:31.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:31.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 10 monclient(hunting): picked mon.noname-a con 0x7f87cc09a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 10 monclient(hunting): picked mon.noname-c con 0x7f87cc09b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 10 monclient(hunting): picked mon.noname-b con 0x7f87cc09d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc139d80 con 0x7f87cc09a550 2024-09-16T14:42:31.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc09a240 con 0x7f87cc09d1b0 2024-09-16T14:42:31.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc119f90 con 0x7f87cc09b040 2024-09-16T14:42:31.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87d23b7640 10 monclient(hunting): authenticate will time out at 975.531238s 2024-09-16T14:42:31.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87cbfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f87cc09a550 0x7f87cc131ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57268/0 (socket says 172.21.15.26:57268) 2024-09-16T14:42:31.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87cbfff640 1 -- 172.21.15.26:0/1932822071 learned_addr learned my addr 172.21.15.26:0/1932822071 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849687136 0 0) 0x7f87cc119f90 con 0x7f87cc09b040 2024-09-16T14:42:31.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 10 monclient(hunting): my global_id is 4244 2024-09-16T14:42:31.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87ac003530 con 0x7f87cc09b040 2024-09-16T14:42:31.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157537601 0 0) 0x7f87cc139d80 con 0x7f87cc09a550 2024-09-16T14:42:31.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 10 monclient(hunting): my global_id is 4251 2024-09-16T14:42:31.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87cc119f90 con 0x7f87cc09a550 2024-09-16T14:42:31.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1849392439 0 0) 0x7f87ac003530 con 0x7f87cc09b040 2024-09-16T14:42:31.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87cc139d80 con 0x7f87cc09b040 2024-09-16T14:42:31.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2300446771 0 0) 0x7f87cc119f90 con 0x7f87cc09a550 2024-09-16T14:42:31.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87ac003530 con 0x7f87cc09a550 2024-09-16T14:42:31.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f87b4001f10 con 0x7f87cc09b040 2024-09-16T14:42:31.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:31.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.408+0000 7f87c8ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:31.427 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.427 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.428 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.428 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.428 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.428 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.428 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.428 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.428 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.428 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:31.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f87bc001920 con 0x7f87cc09a550 2024-09-16T14:42:31.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:31.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:31.428 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.429 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.429 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.429 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.429 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.429 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.429 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.429 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.429 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.429 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559336482 0 0) 0x7f87cc09a240 con 0x7f87cc09d1b0 2024-09-16T14:42:31.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 10 monclient(hunting): my global_id is 4252 2024-09-16T14:42:31.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87cc119f90 con 0x7f87cc09d1b0 2024-09-16T14:42:31.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4038917436 0 0) 0x7f87cc139d80 con 0x7f87cc09b040 2024-09-16T14:42:31.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 >> v1:172.21.15.189:6789/0 conn(0x7f87cc09d1b0 legacy=0x7f87cc1cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 >> v1:172.21.15.26:6789/0 conn(0x7f87cc09a550 legacy=0x7f87cc131ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 1 monclient: found mon.c 2024-09-16T14:42:31.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87cc11a270 con 0x7f87cc09b040 2024-09-16T14:42:31.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87d23b7640 5 monclient: authenticate success, global_id 4244 2024-09-16T14:42:31.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87d23b7640 10 monclient: _renew_subs 2024-09-16T14:42:31.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87d23b7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87d23b7640 1 -- 172.21.15.26:0/1932822071 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87cc13aae0 con 0x7f87cc09b040 2024-09-16T14:42:31.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87d23b7640 10 monclient: _renew_subs 2024-09-16T14:42:31.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87d23b7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87d23b7640 1 -- 172.21.15.26:0/1932822071 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87cc1c4140 con 0x7f87cc09b040 2024-09-16T14:42:31.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87b4001550 con 0x7f87cc09b040 2024-09-16T14:42:31.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f87b4005120 con 0x7f87cc09b040 2024-09-16T14:42:31.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:31.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87ca7fc640 4 set_mon_vals no callback set 2024-09-16T14:42:31.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 10 monclient: dump: 2024-09-16T14:42:31.432 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.432 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.432 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.432 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.432 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.432 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.432 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.432 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.432 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.432 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87d23b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:31.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87d23b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:31.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87d23b7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87d23b7640 1 -- 172.21.15.26:0/1932822071 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b0005050 con 0x7f87cc09b040 2024-09-16T14:42:31.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f87b400fb90 con 0x7f87cc09b040 2024-09-16T14:42:31.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f87b40435c0 con 0x7f87cc09b040 2024-09-16T14:42:31.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.412+0000 7f87cbfff640 10 monclient: get_auth_request con 0x7f87ac0363c0 auth_method 0 2024-09-16T14:42:31.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.416+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f87b4014a00 con 0x7f87cc09b040 2024-09-16T14:42:31.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.416+0000 7f87c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:31.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.416+0000 7f87c8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:31.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.576+0000 7f87d23b7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:31.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.576+0000 7f87d23b7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:31.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.576+0000 7f87d23b7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.576+0000 7f87d23b7640 1 -- 172.21.15.26:0/1932822071 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87b0005300 con 0x7f87cc09b040 2024-09-16T14:42:31.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.580+0000 7f87c8ff9640 1 -- 172.21.15.26:0/1932822071 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+10842 (unknown 4221056075 0 2316752284) 0x7f87b40130a0 con 0x7f87cc09b040 2024-09-16T14:42:31.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.580+0000 7f87c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:31.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.580+0000 7f87c8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:31.581 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:42:31.581 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":15,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","created":"2024-09-16T14:42:17.361142+0000","modified":"2024-09-16T14:42:30.562817+0000","last_up_change":"2024-09-16T14:42:26.444878+0000","last_in_change":"2024-09-16T14:42:19.053975+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-16T14:42:27.380911+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-16T14:42:28.450753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a4578188-6e8b-430f-b70d-1c3480eb7bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6808","nonce":3182580250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6809","nonce":3182580250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6811","nonce":3182580250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6810","nonce":3182580250}]},"public_addr":"172.21.15.26:6808/3182580250","cluster_addr":"172.21.15.26:6809/3182580250","heartbeat_back_addr":"172.21.15.26:6811/3182580250","heartbeat_front_addr":"172.21.15.26:6810/3182580250","state":["exists","up"]},{"osd":1,"uuid":"1a96367f-73f1-4a99-9b85-47bf8e3d4cdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6800","nonce":3032704211}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6801","nonce":3032704211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6805","nonce":3032704211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6803","nonce":3032704211}]},"public_addr":"172.21.15.26:6800/3032704211","cluster_addr":"172.21.15.26:6801/3032704211","heartbeat_back_addr":"172.21.15.26:6805/3032704211","heartbeat_front_addr":"172.21.15.26:6803/3032704211","state":["exists","up"]},{"osd":2,"uuid":"3030f43d-f848-4189-be1e-6ea367941218","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6802","nonce":1811454603}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6804","nonce":1811454603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6807","nonce":1811454603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6806","nonce":1811454603}]},"public_addr":"172.21.15.26:6802/1811454603","cluster_addr":"172.21.15.26:6804/1811454603","heartbeat_back_addr":"172.21.15.26:6807/1811454603","heartbeat_front_addr":"172.21.15.26:6806/1811454603","state":["exists","up"]},{"osd":3,"uuid":"78731cf9-4ba7-4ce9-a483-080aa1136b65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6808","nonce":3651837298}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6809","nonce":3651837298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6811","nonce":3651837298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6810","nonce":3651837298}]},"public_addr":"172.21.15.189:6808/3651837298","cluster_addr":"172.21.15.189:6809/3651837298","heartbeat_back_addr":"172.21.15.189:6811/3651837298","heartbeat_front_addr":"172.21.15.189:6810/3651837298","state":["exists","up"]},{"osd":4,"uuid":"d74156df-be8d-4ad7-93dc-89ac28df70dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6800","nonce":1782891661}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6801","nonce":1782891661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6803","nonce":1782891661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6802","nonce":1782891661}]},"public_addr":"172.21.15.189:6800/1782891661","cluster_addr":"172.21.15.189:6801/1782891661","heartbeat_back_addr":"172.21.15.189:6803/1782891661","heartbeat_front_addr":"172.21.15.189:6802/1782891661","state":["exists","up"]},{"osd":5,"uuid":"ccad9c1e-247f-4ed4-8437-6aa5c8b515d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6804","nonce":3793314587}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6805","nonce":3793314587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6807","nonce":3793314587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6806","nonce":3793314587}]},"public_addr":"172.21.15.189:6804/3793314587","cluster_addr":"172.21.15.189:6805/3793314587","heartbeat_back_addr":"172.21.15.189:6807/3793314587","heartbeat_front_addr":"172.21.15.189:6806/3793314587","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.603081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.576634+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.472343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.602340+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.465228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.494237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T14:42:31.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.580+0000 7f87d23b7640 1 -- 172.21.15.26:0/1932822071 >> v1:172.21.15.189:6812/3196161582 conn(0x7f87ac0363c0 legacy=0x7f87ac038880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.580+0000 7f87d23b7640 10 monclient: shutdown 2024-09-16T14:42:31.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.580+0000 7f87d23b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.580+0000 7f87d23b7640 1 -- 172.21.15.26:0/1932822071 >> v1:172.21.15.26:6790/0 conn(0x7f87cc09b040 legacy=0x7f87cc1cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.580+0000 7f87d23b7640 1 -- 172.21.15.26:0/1932822071 shutdown_connections 2024-09-16T14:42:31.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.580+0000 7f87d23b7640 1 -- 172.21.15.26:0/1932822071 >> 172.21.15.26:0/1932822071 conn(0x7f87cc09fb50 msgr2=0x7f87cc09eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.580+0000 7f87d23b7640 1 -- 172.21.15.26:0/1932822071 shutdown_connections 2024-09-16T14:42:31.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.580+0000 7f87d23b7640 1 -- 172.21.15.26:0/1932822071 wait complete. 2024-09-16T14:42:31.601 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T14:42:31.601 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T14:42:31.602 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T14:42:31.602 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T14:42:31.603 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T14:42:31.603 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T14:42:31.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 10 monclient: get_monmap_and_config 2024-09-16T14:42:31.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 10 monclient: monmap: 2024-09-16T14:42:31.723 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.723 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.723 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:31.722373+0000 2024-09-16T14:42:31.723 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:31.722373+0000 2024-09-16T14:42:31.723 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.723 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.723 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.723 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:31.723 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:31.724 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 1 Processor -- start 2024-09-16T14:42:31.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 1 -- start start 2024-09-16T14:42:31.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 10 monclient: init 2024-09-16T14:42:31.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:42:31.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 10 monclient(hunting): picked mon.noname-c con 0x7f8b000a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 10 monclient(hunting): picked mon.noname-a con 0x7f8b00099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 10 monclient(hunting): picked mon.noname-b con 0x7f8b000a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b00098ea0 con 0x7f8b00099b60 2024-09-16T14:42:31.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b06b9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8b000a26f0 0x7f8b0013d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48312/0 (socket says 172.21.15.26:48312) 2024-09-16T14:42:31.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b06b9b640 1 -- 172.21.15.26:0/2012459367 learned_addr learned my addr 172.21.15.26:0/2012459367 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 1 -- 172.21.15.26:0/2012459367 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b000997b0 con 0x7f8b000a1c00 2024-09-16T14:42:31.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 1 -- 172.21.15.26:0/2012459367 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b00099460 con 0x7f8b000a26f0 2024-09-16T14:42:31.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 10 monclient(hunting): authenticate will time out at 975.842734s 2024-09-16T14:42:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 1 -- 172.21.15.26:0/2012459367 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675797555 0 0) 0x7f8b00099460 con 0x7f8b000a26f0 2024-09-16T14:42:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient(hunting): my global_id is 4247 2024-09-16T14:42:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 1 -- 172.21.15.26:0/2012459367 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aec003530 con 0x7f8b000a26f0 2024-09-16T14:42:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 1 -- 172.21.15.26:0/2012459367 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224682279 0 0) 0x7f8b00098ea0 con 0x7f8b00099b60 2024-09-16T14:42:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient(hunting): my global_id is 4254 2024-09-16T14:42:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 1 -- 172.21.15.26:0/2012459367 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b00099460 con 0x7f8b00099b60 2024-09-16T14:42:31.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 1 -- 172.21.15.26:0/2012459367 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3674323735 0 0) 0x7f8aec003530 con 0x7f8b000a26f0 2024-09-16T14:42:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 1 -- 172.21.15.26:0/2012459367 >> v1:172.21.15.189:6789/0 conn(0x7f8b000a1c00 legacy=0x7f8b000a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 1 -- 172.21.15.26:0/2012459367 >> v1:172.21.15.26:6789/0 conn(0x7f8b00099b60 legacy=0x7f8b00139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 1 monclient: found mon.noname-b 2024-09-16T14:42:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 1 -- 172.21.15.26:0/2012459367 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b000a34b0 con 0x7f8b000a26f0 2024-09-16T14:42:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 1 -- 172.21.15.26:0/2012459367 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8af8001920 con 0x7f8b000a26f0 2024-09-16T14:42:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:31.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient: dump: 2024-09-16T14:42:31.729 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.729 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.729 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.729 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.729 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.729 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.729 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.729 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.729 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.729 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 5 monclient: authenticate success, global_id 4247 2024-09-16T14:42:31.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:31.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 1 -- 172.21.15.26:0/2012459367 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8af8001b40 con 0x7f8b000a26f0 2024-09-16T14:42:31.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 1 -- 172.21.15.26:0/2012459367 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8af8001e90 con 0x7f8b000a26f0 2024-09-16T14:42:31.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:31.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b085fe640 4 set_mon_vals no callback set 2024-09-16T14:42:31.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:31.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f8b05398640 10 monclient: dump: 2024-09-16T14:42:31.730 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.730 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.730 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.730 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.730 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.730 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.731 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.731 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.731 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.731 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f58edec9640 10 monclient: get_monmap_and_config 2024-09-16T14:42:31.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f58edec9640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.720+0000 7f58edec9640 10 monclient: monmap: 2024-09-16T14:42:31.731 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.732 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.732 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:31.725397+0000 2024-09-16T14:42:31.732 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:31.725397+0000 2024-09-16T14:42:31.732 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.732 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.732 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.732 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:31.732 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:31.732 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 10 monclient: shutdown 2024-09-16T14:42:31.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 1 -- 172.21.15.26:0/2012459367 >> v1:172.21.15.26:6790/0 conn(0x7f8b000a26f0 legacy=0x7f8b0013d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58edec9640 1 Processor -- start 2024-09-16T14:42:31.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 1 -- 172.21.15.26:0/2012459367 shutdown_connections 2024-09-16T14:42:31.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 1 -- 172.21.15.26:0/2012459367 >> 172.21.15.26:0/2012459367 conn(0x7f8b000a3d00 msgr2=0x7f8b000a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58edec9640 1 -- start start 2024-09-16T14:42:31.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58edec9640 10 monclient: init 2024-09-16T14:42:31.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 1 -- 172.21.15.26:0/2012459367 shutdown_connections 2024-09-16T14:42:31.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 1 -- 172.21.15.26:0/2012459367 wait complete. 2024-09-16T14:42:31.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58edec9640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58edec9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:42:31.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58edec9640 10 monclient(hunting): picked mon.noname-a con 0x7f58e80a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58edec9640 10 monclient(hunting): picked mon.noname-c con 0x7f58e8099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58edec9640 10 monclient(hunting): picked mon.noname-b con 0x7f58e80a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58edec9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e8098ea0 con 0x7f58e80a1c00 2024-09-16T14:42:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58edec9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e80997b0 con 0x7f58e8099b60 2024-09-16T14:42:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58edec9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e8099460 con 0x7f58e80a26f0 2024-09-16T14:42:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58edec9640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58edec9640 10 monclient(hunting): authenticate will time out at 975.845325s 2024-09-16T14:42:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e77fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f58e80a1c00 0x7f58e80a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57278/0 (socket says 172.21.15.26:57278) 2024-09-16T14:42:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e77fe640 1 -- 172.21.15.26:0/2637644960 learned_addr learned my addr 172.21.15.26:0/2637644960 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 10 monclient: monmap: 2024-09-16T14:42:31.738 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.738 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:31.738 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:31.738 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:31.738 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.738 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:31.738 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.738 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:31.738 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:31.739 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 1 Processor -- start 2024-09-16T14:42:31.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 1 -- start start 2024-09-16T14:42:31.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 10 monclient: init 2024-09-16T14:42:31.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:31.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 10 monclient(hunting): picked mon.noname-a con 0x7f8b00099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b0639a640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8b00099b60 0x7f8b001380d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57292/0 (socket says 172.21.15.26:57292) 2024-09-16T14:42:31.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b0639a640 1 -- 172.21.15.26:0/3925542367 learned_addr learned my addr 172.21.15.26:0/3925542367 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 10 monclient(hunting): picked mon.noname-c con 0x7f8b000a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 10 monclient(hunting): picked mon.noname-b con 0x7f8b000a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 1 -- 172.21.15.26:0/3925542367 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0009ea90 con 0x7f8b00099b60 2024-09-16T14:42:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 1 -- 172.21.15.26:0/3925542367 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b001387e0 con 0x7f8b000a26f0 2024-09-16T14:42:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 1 -- 172.21.15.26:0/3925542367 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0009efe0 con 0x7f8b000a1c00 2024-09-16T14:42:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8b085fe640 10 monclient(hunting): authenticate will time out at 975.847148s 2024-09-16T14:42:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809521413 0 0) 0x7f8b0009ea90 con 0x7f8b00099b60 2024-09-16T14:42:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): my global_id is 4260 2024-09-16T14:42:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aec003530 con 0x7f8b00099b60 2024-09-16T14:42:31.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 1 -- 172.21.15.26:0/2637644960 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734762604 0 0) 0x7f58e8099460 con 0x7f58e80a26f0 2024-09-16T14:42:31.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient(hunting): my global_id is 4250 2024-09-16T14:42:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 1 -- 172.21.15.26:0/2637644960 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d0003530 con 0x7f58e80a26f0 2024-09-16T14:42:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 1 -- 172.21.15.26:0/2637644960 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984997015 0 0) 0x7f58e8098ea0 con 0x7f58e80a1c00 2024-09-16T14:42:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient(hunting): my global_id is 4257 2024-09-16T14:42:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 1 -- 172.21.15.26:0/2637644960 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58e8099460 con 0x7f58e80a1c00 2024-09-16T14:42:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 1 -- 172.21.15.26:0/2637644960 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402748684 0 0) 0x7f58e80997b0 con 0x7f58e8099b60 2024-09-16T14:42:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient(hunting): my global_id is 4258 2024-09-16T14:42:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 1 -- 172.21.15.26:0/2637644960 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58e8098ea0 con 0x7f58e8099b60 2024-09-16T14:42:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644726207 0 0) 0x7f8b0009efe0 con 0x7f8b000a1c00 2024-09-16T14:42:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): my global_id is 4253 2024-09-16T14:42:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b0009ea90 con 0x7f8b000a1c00 2024-09-16T14:42:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162042752 0 0) 0x7f8b001387e0 con 0x7f8b000a26f0 2024-09-16T14:42:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): my global_id is 4261 2024-09-16T14:42:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b0009efe0 con 0x7f8b000a26f0 2024-09-16T14:42:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 346227520 0 0) 0x7f8b0009ea90 con 0x7f8b000a1c00 2024-09-16T14:42:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b001387e0 con 0x7f8b000a1c00 2024-09-16T14:42:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 415368062 0 0) 0x7f8aec003530 con 0x7f8b00099b60 2024-09-16T14:42:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b0009ea90 con 0x7f8b00099b60 2024-09-16T14:42:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8af0001860 con 0x7f8b000a1c00 2024-09-16T14:42:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.746 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.746 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:31.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 1 -- 172.21.15.26:0/2637644960 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 648193767 0 0) 0x7f58d0003530 con 0x7f58e80a26f0 2024-09-16T14:42:31.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 1 -- 172.21.15.26:0/2637644960 >> v1:172.21.15.189:6789/0 conn(0x7f58e8099b60 legacy=0x7f58e8139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 1 -- 172.21.15.26:0/2637644960 >> v1:172.21.15.26:6789/0 conn(0x7f58e80a1c00 legacy=0x7f58e80a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 1 monclient: found mon.noname-b 2024-09-16T14:42:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 1 -- 172.21.15.26:0/2637644960 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e80a34b0 con 0x7f58e80a26f0 2024-09-16T14:42:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 1 -- 172.21.15.26:0/2637644960 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58e00019a0 con 0x7f58e80a26f0 2024-09-16T14:42:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58e67fc640 10 monclient: dump: 2024-09-16T14:42:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.748 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58edec9640 5 monclient: authenticate success, global_id 4250 2024-09-16T14:42:31.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f58edec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:31.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8afc0018f0 con 0x7f8b00099b60 2024-09-16T14:42:31.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:31.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.724+0000 7f8af6ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:31.750 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.750 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.750 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.750 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.750 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.750 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.750 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.750 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.750 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.750 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4190700224 0 0) 0x7f8b001387e0 con 0x7f8b000a1c00 2024-09-16T14:42:31.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 >> v1:172.21.15.189:6789/0 conn(0x7f8b000a26f0 legacy=0x7f8b001d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 >> v1:172.21.15.26:6789/0 conn(0x7f8b00099b60 legacy=0x7f8b001380d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 1 monclient: found mon.c 2024-09-16T14:42:31.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0009e3f0 con 0x7f8b000a1c00 2024-09-16T14:42:31.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8b085fe640 5 monclient: authenticate success, global_id 4253 2024-09-16T14:42:31.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8b085fe640 10 monclient: _renew_subs 2024-09-16T14:42:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8b085fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8b085fe640 1 -- 172.21.15.26:0/3925542367 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b0011fa30 con 0x7f8b000a1c00 2024-09-16T14:42:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8af0003620 con 0x7f8b000a1c00 2024-09-16T14:42:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8af0003930 con 0x7f8b000a1c00 2024-09-16T14:42:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 10 monclient: dump: 2024-09-16T14:42:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.753 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8b085fe640 10 monclient: _renew_subs 2024-09-16T14:42:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8b085fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8b085fe640 1 -- 172.21.15.26:0/3925542367 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b0011e4c0 con 0x7f8b000a1c00 2024-09-16T14:42:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af7fff640 4 set_mon_vals no callback set 2024-09-16T14:42:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f58e67fc640 1 -- 172.21.15.26:0/2637644960 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58e0001bc0 con 0x7f58e80a26f0 2024-09-16T14:42:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f58e67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f58e67fc640 1 -- 172.21.15.26:0/2637644960 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58e0001f50 con 0x7f58e80a26f0 2024-09-16T14:42:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f58e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f58e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f58e67fc640 10 monclient: dump: 2024-09-16T14:42:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.755 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.755 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.755 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.755 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f58edec9640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:31.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f58edec9640 4 set_mon_vals no callback set 2024-09-16T14:42:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8af000e3a0 con 0x7f8b000a1c00 2024-09-16T14:42:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f8af0041e40 con 0x7f8b000a1c00 2024-09-16T14:42:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8b085fe640 1 -- 172.21.15.26:0/3925542367 --> v1:172.21.15.26:6802/1811454603 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b0009e900 con 0x7f8b0009cfd0 2024-09-16T14:42:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8b0639a640 10 monclient: get_auth_request con 0x7f8aec0366c0 auth_method 0 2024-09-16T14:42:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8b06b9b640 10 monclient: get_auth_request con 0x7f8b0009cfd0 auth_method 0 2024-09-16T14:42:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f58edec9640 10 monclient: shutdown 2024-09-16T14:42:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f58edec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f58edec9640 1 -- 172.21.15.26:0/2637644960 >> v1:172.21.15.26:6790/0 conn(0x7f58e80a26f0 legacy=0x7f58e813d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f58edec9640 1 -- 172.21.15.26:0/2637644960 shutdown_connections 2024-09-16T14:42:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f58edec9640 1 -- 172.21.15.26:0/2637644960 >> 172.21.15.26:0/2637644960 conn(0x7f58e80a3d00 msgr2=0x7f58e80a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.728+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 <== osd.2 v1:172.21.15.26:6802/1811454603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f8b0009e900 con 0x7f8b0009cfd0 2024-09-16T14:42:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 1 -- 172.21.15.26:0/2637644960 shutdown_connections 2024-09-16T14:42:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 1 -- 172.21.15.26:0/2637644960 wait complete. 2024-09-16T14:42:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 10 monclient: monmap: 2024-09-16T14:42:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:31.764 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 1 Processor -- start 2024-09-16T14:42:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 1 -- start start 2024-09-16T14:42:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 10 monclient: init 2024-09-16T14:42:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:42:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 10 monclient(hunting): picked mon.noname-c con 0x7f58e8099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 10 monclient(hunting): picked mon.noname-a con 0x7f58e80a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 10 monclient(hunting): picked mon.noname-b con 0x7f58e80a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e806ee80 con 0x7f58e80a1c00 2024-09-16T14:42:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e8070710 con 0x7f58e80a26f0 2024-09-16T14:42:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e8071250 con 0x7f58e8099b60 2024-09-16T14:42:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 10 monclient(hunting): authenticate will time out at 975.857736s 2024-09-16T14:42:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58e6ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f58e80a1c00 0x7f58e81d2d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57294/0 (socket says 172.21.15.26:57294) 2024-09-16T14:42:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58e6ffd640 1 -- 172.21.15.26:0/2702460034 learned_addr learned my addr 172.21.15.26:0/2702460034 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674323568 0 0) 0x7f58e8071250 con 0x7f58e8099b60 2024-09-16T14:42:31.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient(hunting): my global_id is 4256 2024-09-16T14:42:31.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d4003530 con 0x7f58e8099b60 2024-09-16T14:42:31.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344173855 0 0) 0x7f58e806ee80 con 0x7f58e80a1c00 2024-09-16T14:42:31.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient(hunting): my global_id is 4263 2024-09-16T14:42:31.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58e8071250 con 0x7f58e80a1c00 2024-09-16T14:42:31.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3153778178 0 0) 0x7f58d4003530 con 0x7f58e8099b60 2024-09-16T14:42:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58e806ee80 con 0x7f58e8099b60 2024-09-16T14:42:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58d80017b0 con 0x7f58e8099b60 2024-09-16T14:42:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient(hunting): dump: 2024-09-16T14:42:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.767 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.768 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.768 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.768 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:31.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124563819 0 0) 0x7f58e8070710 con 0x7f58e80a26f0 2024-09-16T14:42:31.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient(hunting): my global_id is 4264 2024-09-16T14:42:31.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d4003530 con 0x7f58e80a26f0 2024-09-16T14:42:31.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3549655489 0 0) 0x7f58e8071250 con 0x7f58e80a1c00 2024-09-16T14:42:31.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58e8070710 con 0x7f58e80a1c00 2024-09-16T14:42:31.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4266149916 0 0) 0x7f58e806ee80 con 0x7f58e8099b60 2024-09-16T14:42:31.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 >> v1:172.21.15.189:6789/0 conn(0x7f58e80a26f0 legacy=0x7f58e81d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 >> v1:172.21.15.26:6789/0 conn(0x7f58e80a1c00 legacy=0x7f58e81d2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 1 monclient: found mon.c 2024-09-16T14:42:31.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e809f2d0 con 0x7f58e8099b60 2024-09-16T14:42:31.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 5 monclient: authenticate success, global_id 4256 2024-09-16T14:42:31.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 10 monclient: _renew_subs 2024-09-16T14:42:31.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 1 -- 172.21.15.26:0/2702460034 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e8070c90 con 0x7f58e8099b60 2024-09-16T14:42:31.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 10 monclient: _renew_subs 2024-09-16T14:42:31.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.736+0000 7f58edec9640 1 -- 172.21.15.26:0/2702460034 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e806f5d0 con 0x7f58e8099b60 2024-09-16T14:42:31.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.740+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58d8001ce0 con 0x7f58e8099b60 2024-09-16T14:42:31.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.740+0000 7f58edec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.740+0000 7f58edec9640 1 -- 172.21.15.26:0/2702460034 --> v1:172.21.15.26:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f58e80a3a20 con 0x7f58e8099b60 2024-09-16T14:42:31.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.740+0000 7f58cffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.740+0000 7f58e5ffb640 4 set_mon_vals no callback set 2024-09-16T14:42:31.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.740+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58d8004990 con 0x7f58e8099b60 2024-09-16T14:42:31.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.740+0000 7f58cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.740+0000 7f58cffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:31.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.740+0000 7f58cffff640 10 monclient: dump: 2024-09-16T14:42:31.771 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.771 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.771 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.772 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.772 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.772 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.772 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.772 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.772 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.772 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.740+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f58d8004bb0 con 0x7f58e8099b60 2024-09-16T14:42:31.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.740+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f58d8042ed0 con 0x7f58e8099b60 2024-09-16T14:42:31.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.740+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 --> v1:172.21.15.26:6800/3032704211 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58d4040c30 con 0x7f58d403cf80 2024-09-16T14:42:31.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.740+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (unknown 2942983318 0 0) 0x7f58d8043210 con 0x7f58e8099b60 2024-09-16T14:42:31.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.740+0000 7f58cffff640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-09-16T14:42:31.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.740+0000 7f58e6ffd640 10 monclient: get_auth_request con 0x7f58d4036480 auth_method 0 2024-09-16T14:42:31.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.740+0000 7f58e7fff640 10 monclient: get_auth_request con 0x7f58d403cf80 auth_method 0 2024-09-16T14:42:31.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.744+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 <== osd.1 v1:172.21.15.26:6800/3032704211 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f58d4040c30 con 0x7f58d403cf80 2024-09-16T14:42:31.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.744+0000 7f8b085fe640 1 -- 172.21.15.26:0/3925542367 --> v1:172.21.15.26:6802/1811454603 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8ae8000ee0 con 0x7f8b0009cfd0 2024-09-16T14:42:31.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.744+0000 7f8af6ffd640 1 -- 172.21.15.26:0/3925542367 <== osd.2 v1:172.21.15.26:6802/1811454603 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7f8ae8000ee0 con 0x7f8b0009cfd0 2024-09-16T14:42:31.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.748+0000 7f8b085fe640 1 -- 172.21.15.26:0/3925542367 >> v1:172.21.15.26:6802/1811454603 conn(0x7f8b0009cfd0 legacy=0x7f8b000609a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.748+0000 7f8b085fe640 1 -- 172.21.15.26:0/3925542367 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8aec0366c0 legacy=0x7f8aec038b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.748+0000 7f8b085fe640 10 monclient: shutdown 2024-09-16T14:42:31.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.748+0000 7f8b085fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.748+0000 7f8b085fe640 1 -- 172.21.15.26:0/3925542367 >> v1:172.21.15.26:6790/0 conn(0x7f8b000a1c00 legacy=0x7f8b001ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.748+0000 7f8b06b9b640 1 -- 172.21.15.26:0/3925542367 reap_dead start 2024-09-16T14:42:31.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.748+0000 7f8b085fe640 1 -- 172.21.15.26:0/3925542367 shutdown_connections 2024-09-16T14:42:31.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.748+0000 7f8b085fe640 1 -- 172.21.15.26:0/3925542367 >> 172.21.15.26:0/3925542367 conn(0x7f8b000a3d00 msgr2=0x7f8b000a0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.748+0000 7f8b085fe640 1 -- 172.21.15.26:0/3925542367 shutdown_connections 2024-09-16T14:42:31.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.748+0000 7f8b085fe640 1 -- 172.21.15.26:0/3925542367 wait complete. 2024-09-16T14:42:31.776 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-09-16T14:42:31.777 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T14:42:31.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.772+0000 7f58edec9640 1 -- 172.21.15.26:0/2702460034 --> v1:172.21.15.26:6800/3032704211 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f58e8070ab0 con 0x7f58d403cf80 2024-09-16T14:42:31.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.772+0000 7f58cffff640 1 -- 172.21.15.26:0/2702460034 <== osd.1 v1:172.21.15.26:6800/3032704211 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 2570787547) 0x7f58e8070ab0 con 0x7f58d403cf80 2024-09-16T14:42:31.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.772+0000 7f58edec9640 1 -- 172.21.15.26:0/2702460034 >> v1:172.21.15.26:6800/3032704211 conn(0x7f58d403cf80 legacy=0x7f58d403f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.772+0000 7f58edec9640 1 -- 172.21.15.26:0/2702460034 >> v1:172.21.15.189:6812/3196161582 conn(0x7f58d4036480 legacy=0x7f58d4038940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.772+0000 7f58edec9640 10 monclient: shutdown 2024-09-16T14:42:31.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.772+0000 7f58edec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.772+0000 7f58edec9640 1 -- 172.21.15.26:0/2702460034 >> v1:172.21.15.26:6790/0 conn(0x7f58e8099b60 legacy=0x7f58e8134530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.772+0000 7f58e7fff640 1 -- 172.21.15.26:0/2702460034 reap_dead start 2024-09-16T14:42:31.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.772+0000 7f58edec9640 1 -- 172.21.15.26:0/2702460034 shutdown_connections 2024-09-16T14:42:31.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.772+0000 7f58edec9640 1 -- 172.21.15.26:0/2702460034 >> 172.21.15.26:0/2702460034 conn(0x7f58e80a3d00 msgr2=0x7f58e80a03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.772+0000 7f58edec9640 1 -- 172.21.15.26:0/2702460034 shutdown_connections 2024-09-16T14:42:31.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.772+0000 7f58edec9640 1 -- 172.21.15.26:0/2702460034 wait complete. 2024-09-16T14:42:31.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.776+0000 7f19c255b640 10 monclient: get_monmap_and_config 2024-09-16T14:42:31.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.776+0000 7f19c255b640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.776+0000 7f19c255b640 10 monclient: monmap: 2024-09-16T14:42:31.781 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.781 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.781 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:31.781366+0000 2024-09-16T14:42:31.781 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:31.781366+0000 2024-09-16T14:42:31.781 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.781 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.782 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.782 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:31.782 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:31.782 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 1 Processor -- start 2024-09-16T14:42:31.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 1 -- start start 2024-09-16T14:42:31.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 10 monclient: init 2024-09-16T14:42:31.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:42:31.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 10 monclient(hunting): picked mon.noname-b con 0x7f19bc0a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 10 monclient(hunting): picked mon.noname-a con 0x7f19bc099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 10 monclient(hunting): picked mon.noname-c con 0x7f19bc0a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19bbfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f19bc0a1c00 0x7f19bc0a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48356/0 (socket says 172.21.15.26:48356) 2024-09-16T14:42:31.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19bb7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f19bc099b60 0x7f19bc139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57302/0 (socket says 172.21.15.26:57302) 2024-09-16T14:42:31.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19bbfff640 1 -- 172.21.15.26:0/2146188490 learned_addr learned my addr 172.21.15.26:0/2146188490 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 1 -- 172.21.15.26:0/2146188490 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19bc098ea0 con 0x7f19bc099b60 2024-09-16T14:42:31.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 1 -- 172.21.15.26:0/2146188490 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19bc0997b0 con 0x7f19bc0a26f0 2024-09-16T14:42:31.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 1 -- 172.21.15.26:0/2146188490 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19bc099460 con 0x7f19bc0a1c00 2024-09-16T14:42:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 1 -- 172.21.15.26:0/2146188490 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943072324 0 0) 0x7f19bc098ea0 con 0x7f19bc099b60 2024-09-16T14:42:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 10 monclient(hunting): authenticate will time out at 975.902051s 2024-09-16T14:42:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient(hunting): my global_id is 4266 2024-09-16T14:42:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 1 -- 172.21.15.26:0/2146188490 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19a4003530 con 0x7f19bc099b60 2024-09-16T14:42:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 1 -- 172.21.15.26:0/2146188490 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217941002 0 0) 0x7f19bc099460 con 0x7f19bc0a1c00 2024-09-16T14:42:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient(hunting): my global_id is 4259 2024-09-16T14:42:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 1 -- 172.21.15.26:0/2146188490 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19bc098ea0 con 0x7f19bc0a1c00 2024-09-16T14:42:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 1 -- 172.21.15.26:0/2146188490 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1030961556 0 0) 0x7f19a4003530 con 0x7f19bc099b60 2024-09-16T14:42:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 1 -- 172.21.15.26:0/2146188490 >> v1:172.21.15.26:6790/0 conn(0x7f19bc0a1c00 legacy=0x7f19bc0a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 1 -- 172.21.15.26:0/2146188490 >> v1:172.21.15.189:6789/0 conn(0x7f19bc0a26f0 legacy=0x7f19bc13d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 1 monclient: found mon.noname-a 2024-09-16T14:42:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:42:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 1 -- 172.21.15.26:0/2146188490 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc0a34b0 con 0x7f19bc099b60 2024-09-16T14:42:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 1 -- 172.21.15.26:0/2146188490 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19b0001620 con 0x7f19bc099b60 2024-09-16T14:42:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient: dump: 2024-09-16T14:42:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.790 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 5 monclient: authenticate success, global_id 4266 2024-09-16T14:42:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 1 -- 172.21.15.26:0/2146188490 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19b0001840 con 0x7f19bc099b60 2024-09-16T14:42:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 1 -- 172.21.15.26:0/2146188490 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19b0001c10 con 0x7f19bc099b60 2024-09-16T14:42:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 4 set_mon_vals no callback set 2024-09-16T14:42:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19baffd640 10 monclient: dump: 2024-09-16T14:42:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.792 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 10 monclient: shutdown 2024-09-16T14:42:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 1 -- 172.21.15.26:0/2146188490 >> v1:172.21.15.26:6789/0 conn(0x7f19bc099b60 legacy=0x7f19bc139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 1 -- 172.21.15.26:0/2146188490 shutdown_connections 2024-09-16T14:42:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 1 -- 172.21.15.26:0/2146188490 >> 172.21.15.26:0/2146188490 conn(0x7f19bc0a3d00 msgr2=0x7f19bc0a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 1 -- 172.21.15.26:0/2146188490 shutdown_connections 2024-09-16T14:42:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.780+0000 7f19c255b640 1 -- 172.21.15.26:0/2146188490 wait complete. 2024-09-16T14:42:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 10 monclient: monmap: 2024-09-16T14:42:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:31.794 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 1 Processor -- start 2024-09-16T14:42:31.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 1 -- start start 2024-09-16T14:42:31.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 10 monclient: init 2024-09-16T14:42:31.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:42:31.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 10 monclient(hunting): picked mon.noname-b con 0x7f19bc099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 10 monclient(hunting): picked mon.noname-a con 0x7f19bc0a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c0af8640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f19bc0a26f0 0x7f19bc1d1930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48362/0 (socket says 172.21.15.26:48362) 2024-09-16T14:42:31.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19bb7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f19bc0a1c00 0x7f19bc13a830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57316/0 (socket says 172.21.15.26:57316) 2024-09-16T14:42:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19bb7fe640 1 -- 172.21.15.26:0/2600400583 learned_addr learned my addr 172.21.15.26:0/2600400583 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 10 monclient(hunting): picked mon.noname-c con 0x7f19bc0a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 1 -- 172.21.15.26:0/2600400583 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19bc13d910 con 0x7f19bc0a1c00 2024-09-16T14:42:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 1 -- 172.21.15.26:0/2600400583 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19bc13cf60 con 0x7f19bc099b60 2024-09-16T14:42:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 1 -- 172.21.15.26:0/2600400583 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19bc13bb00 con 0x7f19bc0a26f0 2024-09-16T14:42:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579476503 0 0) 0x7f19bc13d910 con 0x7f19bc0a1c00 2024-09-16T14:42:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 10 monclient(hunting): authenticate will time out at 975.906372s 2024-09-16T14:42:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient(hunting): my global_id is 4269 2024-09-16T14:42:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19a8003530 con 0x7f19bc0a1c00 2024-09-16T14:42:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925961572 0 0) 0x7f19bc13cf60 con 0x7f19bc099b60 2024-09-16T14:42:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient(hunting): my global_id is 4270 2024-09-16T14:42:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19bc13d910 con 0x7f19bc099b60 2024-09-16T14:42:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205938166 0 0) 0x7f19bc13bb00 con 0x7f19bc0a26f0 2024-09-16T14:42:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient(hunting): my global_id is 4262 2024-09-16T14:42:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19bc13cf60 con 0x7f19bc0a26f0 2024-09-16T14:42:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4285544368 0 0) 0x7f19a8003530 con 0x7f19bc0a1c00 2024-09-16T14:42:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19bc13bb00 con 0x7f19bc0a1c00 2024-09-16T14:42:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19b0001b50 con 0x7f19bc0a1c00 2024-09-16T14:42:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.802 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.802 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.802 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.802 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:31.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1154432320 0 0) 0x7f19bc13cf60 con 0x7f19bc0a26f0 2024-09-16T14:42:31.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19a8003530 con 0x7f19bc0a26f0 2024-09-16T14:42:31.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2818834923 0 0) 0x7f19bc13bb00 con 0x7f19bc0a1c00 2024-09-16T14:42:31.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 >> v1:172.21.15.26:6790/0 conn(0x7f19bc0a26f0 legacy=0x7f19bc1d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f13ba22c640 10 monclient: get_monmap_and_config 2024-09-16T14:42:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f13ba22c640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f13ba22c640 10 monclient: monmap: 2024-09-16T14:42:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:31.788179+0000 2024-09-16T14:42:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:31.788179+0000 2024-09-16T14:42:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:31.805 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 >> v1:172.21.15.189:6789/0 conn(0x7f19bc099b60 legacy=0x7f19bc13a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 1 monclient: found mon.a 2024-09-16T14:42:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc13d630 con 0x7f19bc0a1c00 2024-09-16T14:42:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 5 monclient: authenticate success, global_id 4269 2024-09-16T14:42:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 10 monclient: _renew_subs 2024-09-16T14:42:31.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:31.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 1 -- 172.21.15.26:0/2600400583 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19bc13dbf0 con 0x7f19bc0a1c00 2024-09-16T14:42:31.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f13ba22c640 1 Processor -- start 2024-09-16T14:42:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 10 monclient: _renew_subs 2024-09-16T14:42:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f19c255b640 1 -- 172.21.15.26:0/2600400583 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19bc13af40 con 0x7f19bc0a1c00 2024-09-16T14:42:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19b0001160 con 0x7f19bc0a1c00 2024-09-16T14:42:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f19b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19b0004c40 con 0x7f19bc0a1c00 2024-09-16T14:42:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f19b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f19b8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f19b8ff9640 10 monclient: dump: 2024-09-16T14:42:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.809 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f13ba22c640 1 -- start start 2024-09-16T14:42:31.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.784+0000 7f13ba22c640 10 monclient: init 2024-09-16T14:42:31.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:42:31.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 10 monclient(hunting): picked mon.noname-b con 0x7f13b409c900 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 10 monclient(hunting): picked mon.noname-a con 0x7f13b409efd0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 10 monclient(hunting): picked mon.noname-c con 0x7f13b41371e0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b41316d0 con 0x7f13b409efd0 2024-09-16T14:42:31.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b409a6c0 con 0x7f13b41371e0 2024-09-16T14:42:31.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b409aba0 con 0x7f13b409c900 2024-09-16T14:42:31.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 10 monclient(hunting): authenticate will time out at 975.908705s 2024-09-16T14:42:31.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13b922a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f13b409c900 0x7f13b409cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48376/0 (socket says 172.21.15.26:48376) 2024-09-16T14:42:31.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13b922a640 1 -- 172.21.15.26:0/2466735607 learned_addr learned my addr 172.21.15.26:0/2466735607 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f19ba7fc640 4 set_mon_vals no callback set 2024-09-16T14:42:31.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f19b000f6b0 con 0x7f19bc0a1c00 2024-09-16T14:42:31.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f19c255b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:31.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f19c255b640 1 -- 172.21.15.26:0/2600400583 --> v1:172.21.15.26:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f199c000f20 con 0x7f19bc0a1c00 2024-09-16T14:42:31.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f19b0043020 con 0x7f19bc0a1c00 2024-09-16T14:42:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 --> v1:172.21.15.189:6800/1782891661 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19a8040b30 con 0x7f19a803ce80 2024-09-16T14:42:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (unknown 2942983318 0 0) 0x7f19b0014460 con 0x7f19bc0a1c00 2024-09-16T14:42:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f19b8ff9640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-09-16T14:42:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 1 -- 172.21.15.26:0/2466735607 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370443460 0 0) 0x7f13b409a6c0 con 0x7f13b41371e0 2024-09-16T14:42:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient(hunting): my global_id is 4273 2024-09-16T14:42:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 1 -- 172.21.15.26:0/2466735607 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a0003530 con 0x7f13b41371e0 2024-09-16T14:42:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 1 -- 172.21.15.26:0/2466735607 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490667586 0 0) 0x7f13b409aba0 con 0x7f13b409c900 2024-09-16T14:42:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient(hunting): my global_id is 4265 2024-09-16T14:42:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 1 -- 172.21.15.26:0/2466735607 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b409a6c0 con 0x7f13b409c900 2024-09-16T14:42:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 1 -- 172.21.15.26:0/2466735607 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101347906 0 0) 0x7f13b41316d0 con 0x7f13b409efd0 2024-09-16T14:42:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient(hunting): my global_id is 4272 2024-09-16T14:42:31.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f19bbfff640 10 monclient: get_auth_request con 0x7f19a80362b0 auth_method 0 2024-09-16T14:42:31.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 1 -- 172.21.15.26:0/2466735607 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b409aba0 con 0x7f13b409efd0 2024-09-16T14:42:31.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f19c0af8640 10 monclient: get_auth_request con 0x7f19a803ce80 auth_method 0 2024-09-16T14:42:31.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 1 -- 172.21.15.26:0/2466735607 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1826075278 0 0) 0x7f13b409a6c0 con 0x7f13b409c900 2024-09-16T14:42:31.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 1 -- 172.21.15.26:0/2466735607 >> v1:172.21.15.189:6789/0 conn(0x7f13b41371e0 legacy=0x7f13b41395d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 1 -- 172.21.15.26:0/2466735607 >> v1:172.21.15.26:6789/0 conn(0x7f13b409efd0 legacy=0x7f13b4135ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 1 monclient: found mon.noname-b 2024-09-16T14:42:31.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 1 -- 172.21.15.26:0/2466735607 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b409ae10 con 0x7f13b409c900 2024-09-16T14:42:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 5 monclient: authenticate success, global_id 4265 2024-09-16T14:42:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 1 -- 172.21.15.26:0/2466735607 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f13b0001600 con 0x7f13b409c900 2024-09-16T14:42:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient: dump: 2024-09-16T14:42:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.818 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f573cd29640 10 monclient: get_monmap_and_config 2024-09-16T14:42:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 1 -- 172.21.15.26:0/2466735607 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13b0001920 con 0x7f13b409c900 2024-09-16T14:42:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 1 -- 172.21.15.26:0/2466735607 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f13b0001cb0 con 0x7f13b409c900 2024-09-16T14:42:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13abfff640 10 monclient: dump: 2024-09-16T14:42:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.820 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 4 set_mon_vals no callback set 2024-09-16T14:42:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f573cd29640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f573cd29640 10 monclient: monmap: 2024-09-16T14:42:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:31.792063+0000 2024-09-16T14:42:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:31.792063+0000 2024-09-16T14:42:31.822 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.822 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.822 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.822 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:31.822 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:31.822 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f573cd29640 1 Processor -- start 2024-09-16T14:42:31.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f8ed1dbc640 10 monclient: get_monmap_and_config 2024-09-16T14:42:31.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f8ed1dbc640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f8ed1dbc640 10 monclient: monmap: 2024-09-16T14:42:31.823 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.823 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.823 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:31.792309+0000 2024-09-16T14:42:31.823 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:31.792309+0000 2024-09-16T14:42:31.823 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.823 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.823 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.824 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:31.824 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:31.824 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f573cd29640 1 -- start start 2024-09-16T14:42:31.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f573cd29640 10 monclient: init 2024-09-16T14:42:31.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f8ed1dbc640 1 Processor -- start 2024-09-16T14:42:31.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f8ed1dbc640 1 -- start start 2024-09-16T14:42:31.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f8ed1dbc640 10 monclient: init 2024-09-16T14:42:31.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 10 monclient: shutdown 2024-09-16T14:42:31.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 1 -- 172.21.15.26:0/2466735607 >> v1:172.21.15.26:6790/0 conn(0x7f13b409c900 legacy=0x7f13b409cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 1 -- 172.21.15.26:0/2466735607 shutdown_connections 2024-09-16T14:42:31.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.788+0000 7f13ba22c640 1 -- 172.21.15.26:0/2466735607 >> 172.21.15.26:0/2466735607 conn(0x7f13b409f900 msgr2=0x7f13b409fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 1 -- 172.21.15.26:0/2466735607 shutdown_connections 2024-09-16T14:42:31.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 1 -- 172.21.15.26:0/2466735607 wait complete. 2024-09-16T14:42:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f573809a550 0x7f573809a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57324/0 (socket says 172.21.15.26:57324) 2024-09-16T14:42:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573659c640 1 -- 172.21.15.26:0/2054901642 learned_addr learned my addr 172.21.15.26:0/2054901642 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 10 monclient(hunting): picked mon.noname-a con 0x7f573809a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 10 monclient(hunting): picked mon.noname-b con 0x7f573809d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 10 monclient(hunting): picked mon.noname-c con 0x7f573809b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 10 monclient(hunting): picked mon.noname-b con 0x7f8ecc133c90 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 1 -- 172.21.15.26:0/2054901642 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573809dc60 con 0x7f573809a550 2024-09-16T14:42:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 <== osd.4 v1:172.21.15.189:6800/1782891661 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f19a8040b30 con 0x7f19a803ce80 2024-09-16T14:42:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ecb7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8ecc133c90 0x7f8ecc134070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48392/0 (socket says 172.21.15.26:48392) 2024-09-16T14:42:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 1 -- 172.21.15.26:0/2054901642 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5738099ae0 con 0x7f573809b040 2024-09-16T14:42:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 1 -- 172.21.15.26:0/2054901642 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573809b8f0 con 0x7f573809d1b0 2024-09-16T14:42:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 10 monclient(hunting): authenticate will time out at 975.912739s 2024-09-16T14:42:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 10 monclient: monmap: 2024-09-16T14:42:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:31.831 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 1 -- 172.21.15.26:0/2054901642 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403570485 0 0) 0x7f573809dc60 con 0x7f573809a550 2024-09-16T14:42:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient(hunting): my global_id is 4275 2024-09-16T14:42:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 1 -- 172.21.15.26:0/2054901642 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5720005740 con 0x7f573809a550 2024-09-16T14:42:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 1 -- 172.21.15.26:0/2054901642 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197357714 0 0) 0x7f573809b8f0 con 0x7f573809d1b0 2024-09-16T14:42:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient(hunting): my global_id is 4268 2024-09-16T14:42:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 1 -- 172.21.15.26:0/2054901642 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f573809dc60 con 0x7f573809d1b0 2024-09-16T14:42:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 1 Processor -- start 2024-09-16T14:42:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 1 -- start start 2024-09-16T14:42:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 10 monclient: init 2024-09-16T14:42:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 1 -- 172.21.15.26:0/2054901642 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 368779055 0 0) 0x7f573809dc60 con 0x7f573809d1b0 2024-09-16T14:42:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 1 -- 172.21.15.26:0/2054901642 >> v1:172.21.15.189:6789/0 conn(0x7f573809b040 legacy=0x7f573813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:42:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 1 -- 172.21.15.26:0/2054901642 >> v1:172.21.15.26:6789/0 conn(0x7f573809a550 legacy=0x7f573809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 1 monclient: found mon.noname-b 2024-09-16T14:42:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 1 -- 172.21.15.26:0/2054901642 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573809bbd0 con 0x7f573809d1b0 2024-09-16T14:42:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 1 -- 172.21.15.26:0/2054901642 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f572c001620 con 0x7f573809d1b0 2024-09-16T14:42:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 5 monclient: authenticate success, global_id 4268 2024-09-16T14:42:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient: dump: 2024-09-16T14:42:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.837 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 1 -- 172.21.15.26:0/2054901642 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f572c001840 con 0x7f573809d1b0 2024-09-16T14:42:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 1 -- 172.21.15.26:0/2054901642 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f572c001bd0 con 0x7f573809d1b0 2024-09-16T14:42:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573559a640 10 monclient: dump: 2024-09-16T14:42:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.838 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 4 set_mon_vals no callback set 2024-09-16T14:42:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 10 monclient: shutdown 2024-09-16T14:42:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f573cd29640 1 -- 172.21.15.26:0/2054901642 >> v1:172.21.15.26:6790/0 conn(0x7f573809d1b0 legacy=0x7f5738136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 10 monclient(hunting): picked mon.noname-c con 0x7f13b409c900 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 10 monclient(hunting): picked mon.noname-b con 0x7f13b41371e0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 10 monclient(hunting): picked mon.noname-a con 0x7f13b41c5460 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b409a0f0 con 0x7f13b41c5460 2024-09-16T14:42:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b40a0d00 con 0x7f13b41371e0 2024-09-16T14:42:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b409c410 con 0x7f13b409c900 2024-09-16T14:42:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 10 monclient(hunting): authenticate will time out at 975.914209s 2024-09-16T14:42:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13b9a2b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f13b41c5460 0x7f13b41cd2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57348/0 (socket says 172.21.15.26:57348) 2024-09-16T14:42:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13b9a2b640 1 -- 172.21.15.26:0/1018216911 learned_addr learned my addr 172.21.15.26:0/1018216911 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367650682 0 0) 0x7f13b409a0f0 con 0x7f13b41c5460 2024-09-16T14:42:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 10 monclient(hunting): my global_id is 4281 2024-09-16T14:42:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f139800bd70 con 0x7f13b41c5460 2024-09-16T14:42:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 899051497 0 0) 0x7f139800bd70 con 0x7f13b41c5460 2024-09-16T14:42:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13b409a0f0 con 0x7f13b41c5460 2024-09-16T14:42:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f13ac001c60 con 0x7f13b41c5460 2024-09-16T14:42:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 10 monclient(hunting): dump: 2024-09-16T14:42:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.843 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3520839260 0 0) 0x7f13b409a0f0 con 0x7f13b41c5460 2024-09-16T14:42:31.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 >> v1:172.21.15.26:6790/0 conn(0x7f13b409c900 legacy=0x7f13b41c4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 >> v1:172.21.15.189:6789/0 conn(0x7f13b41371e0 legacy=0x7f13b41c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 1 monclient: found mon.a 2024-09-16T14:42:31.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:31.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b4134c90 con 0x7f13b41c5460 2024-09-16T14:42:31.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 5 monclient: authenticate success, global_id 4281 2024-09-16T14:42:31.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 10 monclient: _renew_subs 2024-09-16T14:42:31.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:31.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 1 -- 172.21.15.26:0/1018216911 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b41343f0 con 0x7f13b41c5460 2024-09-16T14:42:31.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 10 monclient: _renew_subs 2024-09-16T14:42:31.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:31.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f13ba22c640 1 -- 172.21.15.26:0/1018216911 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b4133720 con 0x7f13b41c5460 2024-09-16T14:42:31.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13ac003980 con 0x7f13b41c5460 2024-09-16T14:42:31.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f13ac003d10 con 0x7f13b41c5460 2024-09-16T14:42:31.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:31.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13aaffd640 4 set_mon_vals no callback set 2024-09-16T14:42:31.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13a9ffb640 10 monclient: dump: 2024-09-16T14:42:31.845 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.845 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.846 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13ba22c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13ba22c640 1 -- 172.21.15.26:0/1018216911 --> v1:172.21.15.26:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1394000f20 con 0x7f13b41c5460 2024-09-16T14:42:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f13ac003e70 con 0x7f13b41c5460 2024-09-16T14:42:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f13ac043490 con 0x7f13b41c5460 2024-09-16T14:42:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 --> v1:172.21.15.26:6808/3182580250 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f13980491f0 con 0x7f1398045540 2024-09-16T14:42:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (unknown 2942983318 0 0) 0x7f13ac043790 con 0x7f13b41c5460 2024-09-16T14:42:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13a9ffb640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-09-16T14:42:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13b922a640 10 monclient: get_auth_request con 0x7f139803e970 auth_method 0 2024-09-16T14:42:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13b8a29640 10 monclient: get_auth_request con 0x7f1398045540 auth_method 0 2024-09-16T14:42:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.796+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 <== osd.0 v1:172.21.15.26:6808/3182580250 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f13980491f0 con 0x7f1398045540 2024-09-16T14:42:31.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 10 monclient(hunting): picked mon.noname-c con 0x7f8ecc09f5c0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 10 monclient(hunting): picked mon.noname-a con 0x7f8ecc13b300 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc06a0d0 con 0x7f8ecc13b300 2024-09-16T14:42:31.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc09f1a0 con 0x7f8ecc09f5c0 2024-09-16T14:42:31.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ecaffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8ecc09f5c0 0x7f8ecc139be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.26:54730/0 (socket says 172.21.15.26:54730) 2024-09-16T14:42:31.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ecaffd640 1 -- 172.21.15.26:0/1049836323 learned_addr learned my addr 172.21.15.26:0/1049836323 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1049836323 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc09eec0 con 0x7f8ecc133c90 2024-09-16T14:42:31.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 10 monclient(hunting): authenticate will time out at 975.913020s 2024-09-16T14:42:31.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 1 -- 172.21.15.26:0/1049836323 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741416924 0 0) 0x7f8ecc06a0d0 con 0x7f8ecc13b300 2024-09-16T14:42:31.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient(hunting): my global_id is 4278 2024-09-16T14:42:31.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 1 -- 172.21.15.26:0/1049836323 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb4003530 con 0x7f8ecc13b300 2024-09-16T14:42:31.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 1 -- 172.21.15.26:0/1049836323 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860834778 0 0) 0x7f8ecc09f1a0 con 0x7f8ecc09f5c0 2024-09-16T14:42:31.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient(hunting): my global_id is 4276 2024-09-16T14:42:31.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 1 -- 172.21.15.26:0/1049836323 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ecc06a0d0 con 0x7f8ecc09f5c0 2024-09-16T14:42:31.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 1 -- 172.21.15.26:0/1049836323 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1099305440 0 0) 0x7f8ecc06a0d0 con 0x7f8ecc09f5c0 2024-09-16T14:42:31.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 1 -- 172.21.15.26:0/1049836323 >> v1:172.21.15.26:6790/0 conn(0x7f8ecc133c90 legacy=0x7f8ecc134070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:42:31.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 1 -- 172.21.15.26:0/1049836323 >> v1:172.21.15.26:6789/0 conn(0x7f8ecc13b300 legacy=0x7f8ecc13d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 1 monclient: found mon.noname-c 2024-09-16T14:42:31.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:42:31.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 1 -- 172.21.15.26:0/1049836323 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc09ebe0 con 0x7f8ecc09f5c0 2024-09-16T14:42:31.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 1 -- 172.21.15.26:0/1049836323 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ec0001550 con 0x7f8ecc09f5c0 2024-09-16T14:42:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient: dump: 2024-09-16T14:42:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.853 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 5 monclient: authenticate success, global_id 4276 2024-09-16T14:42:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 1 -- 172.21.15.26:0/1049836323 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ec0001770 con 0x7f8ecc09f5c0 2024-09-16T14:42:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 1 -- 172.21.15.26:0/1049836323 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ec0001b00 con 0x7f8ecc09f5c0 2024-09-16T14:42:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8eca7fc640 10 monclient: dump: 2024-09-16T14:42:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.854 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 4 set_mon_vals no callback set 2024-09-16T14:42:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 10 monclient: shutdown 2024-09-16T14:42:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1049836323 >> v1:172.21.15.189:6789/0 conn(0x7f8ecc09f5c0 legacy=0x7f8ecc139be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1049836323 shutdown_connections 2024-09-16T14:42:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.792+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1049836323 >> 172.21.15.26:0/1049836323 conn(0x7f8ecc09fb50 msgr2=0x7f8ecc09ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 1 -- 172.21.15.26:0/2054901642 shutdown_connections 2024-09-16T14:42:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 1 -- 172.21.15.26:0/2054901642 >> 172.21.15.26:0/2054901642 conn(0x7f573809fb50 msgr2=0x7f573809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f8ecb7fe640 1 --1- 172.21.15.26:0/1049836323 >> v1:172.21.15.26:6790/0 conn(0x7f8ecc133c90 0x7f8ecc134070 :-1 s=CLOSED pgs=0 cs=0 l=0).handle_server_banner_and_identify state changed while learned_addr, mark_down or replacing must be happened just now 2024-09-16T14:42:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 1 -- 172.21.15.26:0/2054901642 shutdown_connections 2024-09-16T14:42:31.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1049836323 shutdown_connections 2024-09-16T14:42:31.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 1 -- 172.21.15.26:0/2054901642 wait complete. 2024-09-16T14:42:31.857 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-09-16T14:42:31.857 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T14:42:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 10 monclient: monmap: 2024-09-16T14:42:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:31.862 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 1 Processor -- start 2024-09-16T14:42:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 1 -- start start 2024-09-16T14:42:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 10 monclient: init 2024-09-16T14:42:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:42:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 10 monclient(hunting): picked mon.noname-c con 0x7f573809a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 10 monclient(hunting): picked mon.noname-a con 0x7f573809b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 10 monclient(hunting): picked mon.noname-b con 0x7f573809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573809a240 con 0x7f573809b040 2024-09-16T14:42:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57381a6e20 con 0x7f573809d1b0 2024-09-16T14:42:31.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57381a7100 con 0x7f573809a550 2024-09-16T14:42:31.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 10 monclient(hunting): authenticate will time out at 975.922253s 2024-09-16T14:42:31.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5735d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f573809b040 0x7f57381cd9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57350/0 (socket says 172.21.15.26:57350) 2024-09-16T14:42:31.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5735d9b640 1 -- 172.21.15.26:0/1080890890 learned_addr learned my addr 172.21.15.26:0/1080890890 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657540507 0 0) 0x7f57381a7100 con 0x7f573809a550 2024-09-16T14:42:31.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): my global_id is 4271 2024-09-16T14:42:31.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5718003530 con 0x7f573809a550 2024-09-16T14:42:31.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811746023 0 0) 0x7f573809a240 con 0x7f573809b040 2024-09-16T14:42:31.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): my global_id is 4284 2024-09-16T14:42:31.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57381a7100 con 0x7f573809b040 2024-09-16T14:42:31.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095574144 0 0) 0x7f57381a6e20 con 0x7f573809d1b0 2024-09-16T14:42:31.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): my global_id is 4282 2024-09-16T14:42:31.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f573809a240 con 0x7f573809d1b0 2024-09-16T14:42:31.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3592513431 0 0) 0x7f57381a7100 con 0x7f573809b040 2024-09-16T14:42:31.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57381a6e20 con 0x7f573809b040 2024-09-16T14:42:31.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f572c001c10 con 0x7f573809b040 2024-09-16T14:42:31.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:31.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:31.866 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.866 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.866 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.866 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.866 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.866 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.866 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.866 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.866 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.867 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:31.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4186601970 0 0) 0x7f5718003530 con 0x7f573809a550 2024-09-16T14:42:31.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57381a7100 con 0x7f573809a550 2024-09-16T14:42:31.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f57280018e0 con 0x7f573809a550 2024-09-16T14:42:31.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:31.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:31.867 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.867 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.867 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.868 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 906067475 0 0) 0x7f57381a6e20 con 0x7f573809b040 2024-09-16T14:42:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 >> v1:172.21.15.26:6790/0 conn(0x7f573809a550 legacy=0x7f57381b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 >> v1:172.21.15.189:6789/0 conn(0x7f573809d1b0 legacy=0x7f57381d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 monclient: found mon.a 2024-09-16T14:42:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57381a52f0 con 0x7f573809b040 2024-09-16T14:42:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f5726ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 5 monclient: authenticate success, global_id 4284 2024-09-16T14:42:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 10 monclient: _renew_subs 2024-09-16T14:42:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 1 -- 172.21.15.26:0/1080890890 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57381a47f0 con 0x7f573809b040 2024-09-16T14:42:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 10 monclient: _renew_subs 2024-09-16T14:42:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.800+0000 7f573cd29640 1 -- 172.21.15.26:0/1080890890 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57381d29e0 con 0x7f573809b040 2024-09-16T14:42:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.804+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f572c0039d0 con 0x7f573809b040 2024-09-16T14:42:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.804+0000 7f5726ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.804+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f572c003d60 con 0x7f573809b040 2024-09-16T14:42:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.804+0000 7f5726ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.804+0000 7f5726ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.804+0000 7f5726ffd640 10 monclient: dump: 2024-09-16T14:42:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.871 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.871 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.871 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.871 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.871 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.804+0000 7f5734d99640 4 set_mon_vals no callback set 2024-09-16T14:42:31.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.804+0000 7f573cd29640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:31.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.804+0000 7f573cd29640 1 -- 172.21.15.26:0/1080890890 --> v1:172.21.15.26:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f573809c220 con 0x7f573809b040 2024-09-16T14:42:31.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.804+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f572c003ec0 con 0x7f573809b040 2024-09-16T14:42:31.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.804+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f572c043200 con 0x7f573809b040 2024-09-16T14:42:31.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.804+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 --> v1:172.21.15.189:6808/3651837298 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5718040d50 con 0x7f571803d0a0 2024-09-16T14:42:31.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.804+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (unknown 2942983318 0 0) 0x7f572c043600 con 0x7f573809b040 2024-09-16T14:42:31.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.804+0000 7f5726ffd640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-09-16T14:42:31.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.804+0000 7f573659c640 10 monclient: get_auth_request con 0x7f57180364b0 auth_method 0 2024-09-16T14:42:31.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.804+0000 7f5736d9d640 10 monclient: get_auth_request con 0x7f571803d0a0 auth_method 0 2024-09-16T14:42:31.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 <== osd.3 v1:172.21.15.189:6808/3651837298 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f5718040d50 con 0x7f571803d0a0 2024-09-16T14:42:31.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1049836323 wait complete. 2024-09-16T14:42:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ed1dbc640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ed1dbc640 10 monclient: monmap: 2024-09-16T14:42:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:31.876 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ed1dbc640 1 Processor -- start 2024-09-16T14:42:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ed1dbc640 1 -- start start 2024-09-16T14:42:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ed1dbc640 10 monclient: init 2024-09-16T14:42:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ed1dbc640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ed1dbc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:42:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ed1dbc640 10 monclient(hunting): picked mon.noname-c con 0x7f8ecc09f5c0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ed1dbc640 10 monclient(hunting): picked mon.noname-a con 0x7f8ecc13b300 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ed1dbc640 10 monclient(hunting): picked mon.noname-b con 0x7f8ecc138610 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ed1dbc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc119b20 con 0x7f8ecc13b300 2024-09-16T14:42:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ed1dbc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc09baf0 con 0x7f8ecc138610 2024-09-16T14:42:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ed1dbc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc0a2930 con 0x7f8ecc09f5c0 2024-09-16T14:42:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ed1dbc640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ed1dbc640 10 monclient(hunting): authenticate will time out at 975.930337s 2024-09-16T14:42:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ecaffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8ecc13b300 0x7f8ecc137f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57366/0 (socket says 172.21.15.26:57366) 2024-09-16T14:42:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8ecaffd640 1 -- 172.21.15.26:0/1724916240 learned_addr learned my addr 172.21.15.26:0/1724916240 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1649588193 0 0) 0x7f8ecc119b20 con 0x7f8ecc13b300 2024-09-16T14:42:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 10 monclient(hunting): my global_id is 4287 2024-09-16T14:42:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb8003530 con 0x7f8ecc13b300 2024-09-16T14:42:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651940202 0 0) 0x7f8ecc09baf0 con 0x7f8ecc138610 2024-09-16T14:42:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 10 monclient(hunting): my global_id is 4285 2024-09-16T14:42:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ecc119b20 con 0x7f8ecc138610 2024-09-16T14:42:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889883923 0 0) 0x7f8ecc0a2930 con 0x7f8ecc09f5c0 2024-09-16T14:42:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 10 monclient(hunting): my global_id is 4274 2024-09-16T14:42:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.808+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ecc09baf0 con 0x7f8ecc09f5c0 2024-09-16T14:42:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1536804066 0 0) 0x7f8ecc119b20 con 0x7f8ecc138610 2024-09-16T14:42:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ecc0a2930 con 0x7f8ecc138610 2024-09-16T14:42:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ec4001c60 con 0x7f8ecc138610 2024-09-16T14:42:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient(hunting): dump: 2024-09-16T14:42:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.881 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2083791966 0 0) 0x7f8eb8003530 con 0x7f8ecc13b300 2024-09-16T14:42:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ecc119b20 con 0x7f8ecc13b300 2024-09-16T14:42:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ec0001ac0 con 0x7f8ecc13b300 2024-09-16T14:42:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient(hunting): dump: 2024-09-16T14:42:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.882 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1299388327 0 0) 0x7f8ecc09baf0 con 0x7f8ecc09f5c0 2024-09-16T14:42:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8eb8003530 con 0x7f8ecc09f5c0 2024-09-16T14:42:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ebc0017a0 con 0x7f8ecc09f5c0 2024-09-16T14:42:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient(hunting): dump: 2024-09-16T14:42:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.883 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1073951207 0 0) 0x7f8ecc0a2930 con 0x7f8ecc138610 2024-09-16T14:42:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 >> v1:172.21.15.26:6790/0 conn(0x7f8ecc09f5c0 legacy=0x7f8ecc1377f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 >> v1:172.21.15.26:6789/0 conn(0x7f8ecc13b300 legacy=0x7f8ecc137f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 monclient: found mon.b 2024-09-16T14:42:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc09a240 con 0x7f8ecc138610 2024-09-16T14:42:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8ed1dbc640 5 monclient: authenticate success, global_id 4285 2024-09-16T14:42:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8ed1dbc640 10 monclient: _renew_subs 2024-09-16T14:42:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8ed1dbc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:31.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1724916240 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ecc119840 con 0x7f8ecc138610 2024-09-16T14:42:31.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8ed1dbc640 10 monclient: _renew_subs 2024-09-16T14:42:31.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8ed1dbc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:31.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1724916240 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ecc138aa0 con 0x7f8ecc138610 2024-09-16T14:42:31.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ec40039a0 con 0x7f8ecc138610 2024-09-16T14:42:31.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ec4003cb0 con 0x7f8ecc138610 2024-09-16T14:42:31.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:31.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8ec9ffb640 4 set_mon_vals no callback set 2024-09-16T14:42:31.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient: dump: 2024-09-16T14:42:31.885 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.885 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.886 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.886 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.886 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.886 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.886 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.886 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.886 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.886 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8ed1dbc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:31.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1724916240 --> v1:172.21.15.189:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8eac000f20 con 0x7f8ecc138610 2024-09-16T14:42:31.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8ec400e720 con 0x7f8ecc138610 2024-09-16T14:42:31.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f8ec40421f0 con 0x7f8ecc138610 2024-09-16T14:42:31.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 --> v1:172.21.15.189:6804/3793314587 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8eb8040db0 con 0x7f8eb803d100 2024-09-16T14:42:31.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f13ba22c640 1 -- 172.21.15.26:0/1018216911 --> v1:172.21.15.26:6808/3182580250 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1394002b30 con 0x7f1398045540 2024-09-16T14:42:31.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (unknown 2942983318 0 0) 0x7f8ec4013570 con 0x7f8ecc138610 2024-09-16T14:42:31.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8eb3fff640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-09-16T14:42:31.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8ecaffd640 10 monclient: get_auth_request con 0x7f8eb803d100 auth_method 0 2024-09-16T14:42:31.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.812+0000 7f8ecb7fe640 10 monclient: get_auth_request con 0x7f8eb8036510 auth_method 0 2024-09-16T14:42:31.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f13a9ffb640 1 -- 172.21.15.26:0/1018216911 <== osd.0 v1:172.21.15.26:6808/3182580250 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7f1394002b30 con 0x7f1398045540 2024-09-16T14:42:31.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f13ba22c640 1 -- 172.21.15.26:0/1018216911 >> v1:172.21.15.26:6808/3182580250 conn(0x7f1398045540 legacy=0x7f1398047980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f13ba22c640 1 -- 172.21.15.26:0/1018216911 >> v1:172.21.15.189:6812/3196161582 conn(0x7f139803e970 legacy=0x7f1398040e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f13ba22c640 10 monclient: shutdown 2024-09-16T14:42:31.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f13ba22c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f13ba22c640 1 -- 172.21.15.26:0/1018216911 >> v1:172.21.15.26:6789/0 conn(0x7f13b41c5460 legacy=0x7f13b41cd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f13b9a2b640 1 -- 172.21.15.26:0/1018216911 reap_dead start 2024-09-16T14:42:31.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f13ba22c640 1 -- 172.21.15.26:0/1018216911 shutdown_connections 2024-09-16T14:42:31.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f13ba22c640 1 -- 172.21.15.26:0/1018216911 >> 172.21.15.26:0/1018216911 conn(0x7f13b409f900 msgr2=0x7f13b40a1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== osd.5 v1:172.21.15.189:6804/3793314587 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f8eb8040db0 con 0x7f8eb803d100 2024-09-16T14:42:31.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f19c255b640 1 -- 172.21.15.26:0/2600400583 --> v1:172.21.15.189:6800/1782891661 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f199c002b90 con 0x7f19a803ce80 2024-09-16T14:42:31.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f19b8ff9640 1 -- 172.21.15.26:0/2600400583 <== osd.4 v1:172.21.15.189:6800/1782891661 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7f199c002b90 con 0x7f19a803ce80 2024-09-16T14:42:31.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f19c255b640 1 -- 172.21.15.26:0/2600400583 >> v1:172.21.15.189:6800/1782891661 conn(0x7f19a803ce80 legacy=0x7f19a803f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f19c255b640 1 -- 172.21.15.26:0/2600400583 >> v1:172.21.15.189:6812/3196161582 conn(0x7f19a80362b0 legacy=0x7f19a8038770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f19c255b640 10 monclient: shutdown 2024-09-16T14:42:31.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f19c255b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f19c255b640 1 -- 172.21.15.26:0/2600400583 >> v1:172.21.15.26:6789/0 conn(0x7f19bc0a1c00 legacy=0x7f19bc13a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f19c0af8640 1 -- 172.21.15.26:0/2600400583 reap_dead start 2024-09-16T14:42:31.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f19c255b640 1 -- 172.21.15.26:0/2600400583 shutdown_connections 2024-09-16T14:42:31.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f19c255b640 1 -- 172.21.15.26:0/2600400583 >> 172.21.15.26:0/2600400583 conn(0x7f19bc0a3d00 msgr2=0x7f19bc0a5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f19c255b640 1 -- 172.21.15.26:0/2600400583 shutdown_connections 2024-09-16T14:42:31.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.816+0000 7f19c255b640 1 -- 172.21.15.26:0/2600400583 wait complete. 2024-09-16T14:42:31.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.820+0000 7f13ba22c640 1 -- 172.21.15.26:0/1018216911 shutdown_connections 2024-09-16T14:42:31.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.820+0000 7f13ba22c640 1 -- 172.21.15.26:0/1018216911 wait complete. 2024-09-16T14:42:31.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.824+0000 7f573cd29640 1 -- 172.21.15.26:0/1080890890 --> v1:172.21.15.189:6808/3651837298 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f57381a7450 con 0x7f571803d0a0 2024-09-16T14:42:31.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.824+0000 7f5726ffd640 1 -- 172.21.15.26:0/1080890890 <== osd.3 v1:172.21.15.189:6808/3651837298 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 2570787547) 0x7f57381a7450 con 0x7f571803d0a0 2024-09-16T14:42:31.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.824+0000 7f573cd29640 1 -- 172.21.15.26:0/1080890890 >> v1:172.21.15.189:6808/3651837298 conn(0x7f571803d0a0 legacy=0x7f571803f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.824+0000 7f573cd29640 1 -- 172.21.15.26:0/1080890890 >> v1:172.21.15.189:6812/3196161582 conn(0x7f57180364b0 legacy=0x7f5718038970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.824+0000 7f573cd29640 10 monclient: shutdown 2024-09-16T14:42:31.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.824+0000 7f573cd29640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.824+0000 7f573cd29640 1 -- 172.21.15.26:0/1080890890 >> v1:172.21.15.26:6789/0 conn(0x7f573809b040 legacy=0x7f57381cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.824+0000 7f5736d9d640 1 -- 172.21.15.26:0/1080890890 reap_dead start 2024-09-16T14:42:31.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.824+0000 7f573cd29640 1 -- 172.21.15.26:0/1080890890 shutdown_connections 2024-09-16T14:42:31.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.824+0000 7f573cd29640 1 -- 172.21.15.26:0/1080890890 >> 172.21.15.26:0/1080890890 conn(0x7f573809fb50 msgr2=0x7f57380a1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.824+0000 7f573cd29640 1 -- 172.21.15.26:0/1080890890 shutdown_connections 2024-09-16T14:42:31.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.824+0000 7f573cd29640 1 -- 172.21.15.26:0/1080890890 wait complete. 2024-09-16T14:42:31.896 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-09-16T14:42:31.896 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T14:42:31.897 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-09-16T14:42:31.898 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T14:42:31.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.844+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1724916240 --> v1:172.21.15.189:6804/3793314587 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8eac002b30 con 0x7f8eb803d100 2024-09-16T14:42:31.898 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-09-16T14:42:31.899 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T14:42:31.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.844+0000 7f8eb3fff640 1 -- 172.21.15.26:0/1724916240 <== osd.5 v1:172.21.15.189:6804/3793314587 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7f8eac002b30 con 0x7f8eb803d100 2024-09-16T14:42:31.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.844+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1724916240 >> v1:172.21.15.189:6804/3793314587 conn(0x7f8eb803d100 legacy=0x7f8eb803f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.844+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1724916240 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8eb8036510 legacy=0x7f8eb80389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.844+0000 7f8ed1dbc640 10 monclient: shutdown 2024-09-16T14:42:31.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.844+0000 7f8ed1dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.844+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1724916240 >> v1:172.21.15.189:6789/0 conn(0x7f8ecc138610 legacy=0x7f8ecc1cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.844+0000 7f8ecbfff640 1 -- 172.21.15.26:0/1724916240 reap_dead start 2024-09-16T14:42:31.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.848+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1724916240 shutdown_connections 2024-09-16T14:42:31.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.848+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1724916240 >> 172.21.15.26:0/1724916240 conn(0x7f8ecc09fb50 msgr2=0x7f8ecc09a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.848+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1724916240 shutdown_connections 2024-09-16T14:42:31.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.848+0000 7f8ed1dbc640 1 -- 172.21.15.26:0/1724916240 wait complete. 2024-09-16T14:42:31.902 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-09-16T14:42:31.902 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T14:42:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d41d9a640 10 monclient: get_monmap_and_config 2024-09-16T14:42:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d41d9a640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d41d9a640 10 monclient: monmap: 2024-09-16T14:42:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:31.919872+0000 2024-09-16T14:42:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:31.919872+0000 2024-09-16T14:42:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:31.920 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:31.920 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d41d9a640 1 Processor -- start 2024-09-16T14:42:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d41d9a640 1 -- start start 2024-09-16T14:42:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d41d9a640 10 monclient: init 2024-09-16T14:42:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d41d9a640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d41d9a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d41d9a640 10 monclient(hunting): picked mon.noname-b con 0x7f8d3c0a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d41d9a640 10 monclient(hunting): picked mon.noname-c con 0x7f8d3c0a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d41d9a640 10 monclient(hunting): picked mon.noname-a con 0x7f8d3c0a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d41d9a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c0a3250 con 0x7f8d3c0a2800 2024-09-16T14:42:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d41d9a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c09f600 con 0x7f8d3c0a0000 2024-09-16T14:42:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d41d9a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c0a3830 con 0x7f8d3c0a1d10 2024-09-16T14:42:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d41d9a640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d41d9a640 10 monclient(hunting): authenticate will time out at 976.039457s 2024-09-16T14:42:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d3b7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8d3c0a1d10 0x7f8d3c0a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48430/0 (socket says 172.21.15.26:48430) 2024-09-16T14:42:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d3bfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8d3c0a2800 0x7f8d3c13d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57382/0 (socket says 172.21.15.26:57382) 2024-09-16T14:42:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d3b7fe640 1 -- 172.21.15.26:0/2692441384 learned_addr learned my addr 172.21.15.26:0/2692441384 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d3affd640 1 -- 172.21.15.26:0/2692441384 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324971918 0 0) 0x7f8d3c0a3250 con 0x7f8d3c0a2800 2024-09-16T14:42:31.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d3affd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d3affd640 10 monclient(hunting): my global_id is 4290 2024-09-16T14:42:31.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d3affd640 1 -- 172.21.15.26:0/2692441384 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d28003530 con 0x7f8d3c0a2800 2024-09-16T14:42:31.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d3affd640 1 -- 172.21.15.26:0/2692441384 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465477021 0 0) 0x7f8d3c0a3830 con 0x7f8d3c0a1d10 2024-09-16T14:42:31.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d3affd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d3affd640 10 monclient(hunting): my global_id is 4277 2024-09-16T14:42:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.916+0000 7f8d3affd640 1 -- 172.21.15.26:0/2692441384 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3c0a3250 con 0x7f8d3c0a1d10 2024-09-16T14:42:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 1 -- 172.21.15.26:0/2692441384 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3764590427 0 0) 0x7f8d28003530 con 0x7f8d3c0a2800 2024-09-16T14:42:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 1 -- 172.21.15.26:0/2692441384 >> v1:172.21.15.26:6790/0 conn(0x7f8d3c0a1d10 legacy=0x7f8d3c0a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 1 -- 172.21.15.26:0/2692441384 >> v1:172.21.15.189:6789/0 conn(0x7f8d3c0a0000 legacy=0x7f8d3c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 1 monclient: found mon.noname-a 2024-09-16T14:42:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:42:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 1 -- 172.21.15.26:0/2692441384 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c09ed60 con 0x7f8d3c0a2800 2024-09-16T14:42:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 1 -- 172.21.15.26:0/2692441384 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8d340019a0 con 0x7f8d3c0a2800 2024-09-16T14:42:31.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 5 monclient: authenticate success, global_id 4290 2024-09-16T14:42:31.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:31.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:31.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 10 monclient: dump: 2024-09-16T14:42:31.927 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.927 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.927 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.927 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.927 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.927 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.927 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.927 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.928 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 1 -- 172.21.15.26:0/2692441384 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d34001c40 con 0x7f8d3c0a2800 2024-09-16T14:42:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 1 -- 172.21.15.26:0/2692441384 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8d34002010 con 0x7f8d3c0a2800 2024-09-16T14:42:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 4 set_mon_vals no callback set 2024-09-16T14:42:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3affd640 10 monclient: dump: 2024-09-16T14:42:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.928 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.929 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.929 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.929 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.929 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.929 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.929 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.929 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.929 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 10 monclient: shutdown 2024-09-16T14:42:31.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:31.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 1 -- 172.21.15.26:0/2692441384 >> v1:172.21.15.26:6789/0 conn(0x7f8d3c0a2800 legacy=0x7f8d3c13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 1 -- 172.21.15.26:0/2692441384 shutdown_connections 2024-09-16T14:42:31.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 1 -- 172.21.15.26:0/2692441384 >> 172.21.15.26:0/2692441384 conn(0x7f8d3c0a7f30 msgr2=0x7f8d3c0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:31.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 1 -- 172.21.15.26:0/2692441384 shutdown_connections 2024-09-16T14:42:31.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 1 -- 172.21.15.26:0/2692441384 wait complete. 2024-09-16T14:42:31.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 10 monclient: build_initial_monmap 2024-09-16T14:42:31.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 10 monclient: monmap: 2024-09-16T14:42:31.935 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:31.935 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:31.935 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:31.936 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:31.936 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:31.936 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:31.936 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:31.936 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:31.936 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:31.936 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 1 Processor -- start 2024-09-16T14:42:31.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 1 -- start start 2024-09-16T14:42:31.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 10 monclient: init 2024-09-16T14:42:31.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:31.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:31.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 10 monclient(hunting): picked mon.noname-c con 0x7f8d3c0a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:31.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 10 monclient(hunting): picked mon.noname-b con 0x7f8d3c0a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:31.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 10 monclient(hunting): picked mon.noname-a con 0x7f8d3c0a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:31.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c134ce0 con 0x7f8d3c0a2800 2024-09-16T14:42:31.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c0a6460 con 0x7f8d3c0a1d10 2024-09-16T14:42:31.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c09e500 con 0x7f8d3c0a0000 2024-09-16T14:42:31.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:31.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d41d9a640 10 monclient(hunting): authenticate will time out at 976.043359s 2024-09-16T14:42:31.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3b7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8d3c0a0000 0x7f8d3c133090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48436/0 (socket says 172.21.15.26:48436) 2024-09-16T14:42:31.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d3b7fe640 1 -- 172.21.15.26:0/342145525 learned_addr learned my addr 172.21.15.26:0/342145525 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:31.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103263158 0 0) 0x7f8d3c134ce0 con 0x7f8d3c0a2800 2024-09-16T14:42:31.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d38ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d38ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d38ff9640 10 monclient(hunting): my global_id is 4293 2024-09-16T14:42:31.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d1c003530 con 0x7f8d3c0a2800 2024-09-16T14:42:31.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926976854 0 0) 0x7f8d3c09e500 con 0x7f8d3c0a0000 2024-09-16T14:42:31.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d38ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d38ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d38ff9640 10 monclient(hunting): my global_id is 4280 2024-09-16T14:42:31.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.920+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3c134ce0 con 0x7f8d3c0a0000 2024-09-16T14:42:31.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236424675 0 0) 0x7f8d3c0a6460 con 0x7f8d3c0a1d10 2024-09-16T14:42:31.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:31.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:31.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient(hunting): my global_id is 4291 2024-09-16T14:42:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3c09e500 con 0x7f8d3c0a1d10 2024-09-16T14:42:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4031576387 0 0) 0x7f8d3c134ce0 con 0x7f8d3c0a0000 2024-09-16T14:42:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d3c0a6460 con 0x7f8d3c0a0000 2024-09-16T14:42:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8d2c001860 con 0x7f8d3c0a0000 2024-09-16T14:42:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.940 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3825340393 0 0) 0x7f8d1c003530 con 0x7f8d3c0a2800 2024-09-16T14:42:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d3c134ce0 con 0x7f8d3c0a2800 2024-09-16T14:42:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8d34001f50 con 0x7f8d3c0a2800 2024-09-16T14:42:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:31.941 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.941 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.941 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.941 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.941 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.941 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.941 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.941 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.941 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.941 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2100946197 0 0) 0x7f8d3c0a6460 con 0x7f8d3c0a0000 2024-09-16T14:42:31.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 >> v1:172.21.15.189:6789/0 conn(0x7f8d3c0a1d10 legacy=0x7f8d3c1337a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 >> v1:172.21.15.26:6789/0 conn(0x7f8d3c0a2800 legacy=0x7f8d3c133eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:31.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:42:31.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 monclient: found mon.c 2024-09-16T14:42:31.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:31.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c0a6990 con 0x7f8d3c0a0000 2024-09-16T14:42:31.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient: _finish_auth 0 2024-09-16T14:42:31.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:42:31.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:31.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d41d9a640 5 monclient: authenticate success, global_id 4280 2024-09-16T14:42:31.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d41d9a640 10 monclient: _renew_subs 2024-09-16T14:42:31.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d2c0047f0 con 0x7f8d3c0a0000 2024-09-16T14:42:31.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d41d9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d41d9a640 1 -- 172.21.15.26:0/342145525 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3c136cc0 con 0x7f8d3c0a0000 2024-09-16T14:42:31.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:31.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8d2c004b00 con 0x7f8d3c0a0000 2024-09-16T14:42:31.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:31.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:31.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 10 monclient: dump: 2024-09-16T14:42:31.943 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:31.943 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:31.943 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.943 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:31.943 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:31.943 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:31.943 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:31.943 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:31.943 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:31.944 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:31.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d3a7fc640 4 set_mon_vals no callback set 2024-09-16T14:42:31.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d41d9a640 10 monclient: _renew_subs 2024-09-16T14:42:31.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d41d9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d41d9a640 1 -- 172.21.15.26:0/342145525 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3c135e30 con 0x7f8d3c0a0000 2024-09-16T14:42:31.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8d2c00f570 con 0x7f8d3c0a0000 2024-09-16T14:42:31.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d41d9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:31.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d41d9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:31.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d41d9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:31.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d41d9a640 1 -- 172.21.15.26:0/342145525 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d3c0008e0 con 0x7f8d3c0a0000 2024-09-16T14:42:31.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f8d2c043040 con 0x7f8d3c0a0000 2024-09-16T14:42:31.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.924+0000 7f8d33fff640 10 monclient: get_auth_request con 0x7f8d1c036490 auth_method 0 2024-09-16T14:42:31.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.932+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8d2c0145f0 con 0x7f8d3c0a0000 2024-09-16T14:42:31.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.932+0000 7f8d38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:31.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:31.932+0000 7f8d38ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:32.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 10 monclient: get_monmap_and_config 2024-09-16T14:42:32.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 10 monclient: build_initial_monmap 2024-09-16T14:42:32.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 10 monclient: monmap: 2024-09-16T14:42:32.030 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:32.030 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.030 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:32.030571+0000 2024-09-16T14:42:32.030 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:32.030571+0000 2024-09-16T14:42:32.030 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:32.031 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.031 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:32.031 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:32.031 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:32.031 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 1 Processor -- start 2024-09-16T14:42:32.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 1 -- start start 2024-09-16T14:42:32.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 10 monclient: init 2024-09-16T14:42:32.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc6123e640 10 monclient: get_monmap_and_config 2024-09-16T14:42:32.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc6123e640 10 monclient: build_initial_monmap 2024-09-16T14:42:32.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc6123e640 10 monclient: monmap: 2024-09-16T14:42:32.032 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:32.032 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.032 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:32.031484+0000 2024-09-16T14:42:32.032 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:32.031484+0000 2024-09-16T14:42:32.032 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:32.032 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.032 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:32.032 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:32.033 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:32.033 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:32.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:32.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 10 monclient(hunting): picked mon.noname-c con 0x7f637c0a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:32.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 10 monclient(hunting): picked mon.noname-b con 0x7f637c099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:32.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 10 monclient(hunting): picked mon.noname-a con 0x7f637c0a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:32.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637c098ea0 con 0x7f637c0a26f0 2024-09-16T14:42:32.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637c0997b0 con 0x7f637c0a1c00 2024-09-16T14:42:32.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f638085b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f637c0a26f0 0x7f637c13d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57412/0 (socket says 172.21.15.26:57412) 2024-09-16T14:42:32.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f638085b640 1 -- 172.21.15.26:0/2693688685 learned_addr learned my addr 172.21.15.26:0/2693688685 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:32.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 1 -- 172.21.15.26:0/2693688685 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637c099460 con 0x7f637c099b60 2024-09-16T14:42:32.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:32.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 10 monclient(hunting): authenticate will time out at 976.150548s 2024-09-16T14:42:32.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc6123e640 1 Processor -- start 2024-09-16T14:42:32.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc6123e640 1 -- start start 2024-09-16T14:42:32.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc6123e640 10 monclient: init 2024-09-16T14:42:32.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 1 -- 172.21.15.26:0/2693688685 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463757810 0 0) 0x7f637c098ea0 con 0x7f637c0a26f0 2024-09-16T14:42:32.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient(hunting): my global_id is 4296 2024-09-16T14:42:32.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 1 -- 172.21.15.26:0/2693688685 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6364003530 con 0x7f637c0a26f0 2024-09-16T14:42:32.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 1 -- 172.21.15.26:0/2693688685 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819176523 0 0) 0x7f637c099460 con 0x7f637c099b60 2024-09-16T14:42:32.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient(hunting): my global_id is 4283 2024-09-16T14:42:32.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 1 -- 172.21.15.26:0/2693688685 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f637c098ea0 con 0x7f637c099b60 2024-09-16T14:42:32.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 1 -- 172.21.15.26:0/2693688685 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 466201289 0 0) 0x7f6364003530 con 0x7f637c0a26f0 2024-09-16T14:42:32.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 1 -- 172.21.15.26:0/2693688685 >> v1:172.21.15.26:6790/0 conn(0x7f637c099b60 legacy=0x7f637c139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 1 -- 172.21.15.26:0/2693688685 >> v1:172.21.15.189:6789/0 conn(0x7f637c0a1c00 legacy=0x7f637c0a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:32.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 1 monclient: found mon.noname-a 2024-09-16T14:42:32.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:32.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:42:32.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 1 -- 172.21.15.26:0/2693688685 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c0a34b0 con 0x7f637c0a26f0 2024-09-16T14:42:32.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient: _finish_auth 0 2024-09-16T14:42:32.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:32.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:32.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 1 -- 172.21.15.26:0/2693688685 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f63740019a0 con 0x7f637c0a26f0 2024-09-16T14:42:32.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:32.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient: dump: 2024-09-16T14:42:32.038 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.038 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.038 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.038 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.038 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.038 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.038 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.039 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.039 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.039 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 5 monclient: authenticate success, global_id 4296 2024-09-16T14:42:32.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:32.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 1 -- 172.21.15.26:0/2693688685 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6374001c80 con 0x7f637c0a26f0 2024-09-16T14:42:32.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:32.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 1 -- 172.21.15.26:0/2693688685 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6374002050 con 0x7f637c0a26f0 2024-09-16T14:42:32.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:32.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f637affd640 10 monclient: dump: 2024-09-16T14:42:32.039 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.039 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.039 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.040 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.040 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.040 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.040 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.040 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.040 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.040 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc6123e640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:32.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc6123e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:32.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc6123e640 10 monclient(hunting): picked mon.noname-b con 0x7fdc5c0a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:32.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc6123e640 10 monclient(hunting): picked mon.noname-c con 0x7fdc5c099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:32.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc6123e640 10 monclient(hunting): picked mon.noname-a con 0x7fdc5c0a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:32.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc6123e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5c098ea0 con 0x7fdc5c0a26f0 2024-09-16T14:42:32.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc6123e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5c0997b0 con 0x7fdc5c099b60 2024-09-16T14:42:32.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc6123e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5c099460 con 0x7fdc5c0a1c00 2024-09-16T14:42:32.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc6123e640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:32.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc6123e640 10 monclient(hunting): authenticate will time out at 976.151723s 2024-09-16T14:42:32.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc5ad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fdc5c0a1c00 0x7fdc5c0a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48460/0 (socket says 172.21.15.26:48460) 2024-09-16T14:42:32.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc5b59e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdc5c0a26f0 0x7fdc5c13d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57414/0 (socket says 172.21.15.26:57414) 2024-09-16T14:42:32.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7fdc5ad9d640 1 -- 172.21.15.26:0/1980985806 learned_addr learned my addr 172.21.15.26:0/1980985806 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:32.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:32.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.028+0000 7f63822be640 4 set_mon_vals no callback set 2024-09-16T14:42:32.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 10 monclient: shutdown 2024-09-16T14:42:32.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:32.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 1 -- 172.21.15.26:0/2693688685 >> v1:172.21.15.26:6789/0 conn(0x7f637c0a26f0 legacy=0x7f637c13d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 1 -- 172.21.15.26:0/1980985806 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960557219 0 0) 0x7fdc5c099460 con 0x7fdc5c0a1c00 2024-09-16T14:42:32.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient(hunting): my global_id is 4286 2024-09-16T14:42:32.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 1 -- 172.21.15.26:0/1980985806 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc48003530 con 0x7fdc5c0a1c00 2024-09-16T14:42:32.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 1 -- 172.21.15.26:0/2693688685 shutdown_connections 2024-09-16T14:42:32.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 1 -- 172.21.15.26:0/2693688685 >> 172.21.15.26:0/2693688685 conn(0x7f637c0a3d00 msgr2=0x7f637c0a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:32.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 1 -- 172.21.15.26:0/2693688685 shutdown_connections 2024-09-16T14:42:32.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 1 -- 172.21.15.26:0/2693688685 wait complete. 2024-09-16T14:42:32.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 1 -- 172.21.15.26:0/1980985806 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093987914 0 0) 0x7fdc5c098ea0 con 0x7fdc5c0a26f0 2024-09-16T14:42:32.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient(hunting): my global_id is 4299 2024-09-16T14:42:32.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 1 -- 172.21.15.26:0/1980985806 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc5c099460 con 0x7fdc5c0a26f0 2024-09-16T14:42:32.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 1 -- 172.21.15.26:0/1980985806 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1127084384 0 0) 0x7fdc48003530 con 0x7fdc5c0a1c00 2024-09-16T14:42:32.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 1 -- 172.21.15.26:0/1980985806 >> v1:172.21.15.189:6789/0 conn(0x7fdc5c099b60 legacy=0x7fdc5c139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 1 -- 172.21.15.26:0/1980985806 >> v1:172.21.15.26:6789/0 conn(0x7fdc5c0a26f0 legacy=0x7fdc5c13d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:42:32.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 1 monclient: found mon.noname-b 2024-09-16T14:42:32.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:32.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:32.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 1 -- 172.21.15.26:0/1980985806 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c0a34b0 con 0x7fdc5c0a1c00 2024-09-16T14:42:32.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient: _finish_auth 0 2024-09-16T14:42:32.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:42:32.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:32.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 1 -- 172.21.15.26:0/1980985806 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdc44001620 con 0x7fdc5c0a1c00 2024-09-16T14:42:32.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:32.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient: dump: 2024-09-16T14:42:32.047 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.047 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.047 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.047 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.047 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.047 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.047 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.047 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.047 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.047 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc6123e640 5 monclient: authenticate success, global_id 4286 2024-09-16T14:42:32.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc6123e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:32.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 1 -- 172.21.15.26:0/1980985806 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc44001900 con 0x7fdc5c0a1c00 2024-09-16T14:42:32.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:32.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 1 -- 172.21.15.26:0/1980985806 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdc44001c90 con 0x7fdc5c0a1c00 2024-09-16T14:42:32.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:32.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc59d9b640 10 monclient: dump: 2024-09-16T14:42:32.048 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.048 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.048 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.048 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.048 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.048 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.049 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.049 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.049 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.049 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc6123e640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:32.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc6123e640 4 set_mon_vals no callback set 2024-09-16T14:42:32.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 10 monclient: build_initial_monmap 2024-09-16T14:42:32.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 10 monclient: monmap: 2024-09-16T14:42:32.049 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:32.049 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:32.050 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:32.050 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:32.050 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:32.050 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:32.050 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:32.050 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:32.050 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:32.050 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 1 Processor -- start 2024-09-16T14:42:32.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc6123e640 10 monclient: shutdown 2024-09-16T14:42:32.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc6123e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:32.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc6123e640 1 -- 172.21.15.26:0/1980985806 >> v1:172.21.15.26:6790/0 conn(0x7fdc5c0a1c00 legacy=0x7fdc5c0a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 1 -- start start 2024-09-16T14:42:32.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc6123e640 1 -- 172.21.15.26:0/1980985806 shutdown_connections 2024-09-16T14:42:32.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 10 monclient: init 2024-09-16T14:42:32.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc6123e640 1 -- 172.21.15.26:0/1980985806 >> 172.21.15.26:0/1980985806 conn(0x7fdc5c0a3d00 msgr2=0x7fdc5c0a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:32.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:32.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:42:32.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 10 monclient(hunting): picked mon.noname-a con 0x7f637c099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:32.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 10 monclient(hunting): picked mon.noname-b con 0x7f637c0a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:32.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 10 monclient(hunting): picked mon.noname-c con 0x7f637c0a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:32.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637c11ec50 con 0x7f637c099b60 2024-09-16T14:42:32.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f637bfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f637c099b60 0x7f637c13c160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57422/0 (socket says 172.21.15.26:57422) 2024-09-16T14:42:32.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f637bfff640 1 -- 172.21.15.26:0/834212658 learned_addr learned my addr 172.21.15.26:0/834212658 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:32.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 1 -- 172.21.15.26:0/834212658 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637c09f1f0 con 0x7f637c0a1c00 2024-09-16T14:42:32.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 1 -- 172.21.15.26:0/834212658 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637c09eca0 con 0x7f637c0a26f0 2024-09-16T14:42:32.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:32.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 10 monclient(hunting): authenticate will time out at 976.154481s 2024-09-16T14:42:32.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc6123e640 1 -- 172.21.15.26:0/1980985806 shutdown_connections 2024-09-16T14:42:32.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916784015 0 0) 0x7f637c11ec50 con 0x7f637c099b60 2024-09-16T14:42:32.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient(hunting): my global_id is 4302 2024-09-16T14:42:32.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6368003530 con 0x7f637c099b60 2024-09-16T14:42:32.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414067604 0 0) 0x7f637c09f1f0 con 0x7f637c0a1c00 2024-09-16T14:42:32.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient(hunting): my global_id is 4300 2024-09-16T14:42:32.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f637c11ec50 con 0x7f637c0a1c00 2024-09-16T14:42:32.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4234789746 0 0) 0x7f6368003530 con 0x7f637c099b60 2024-09-16T14:42:32.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f637c09f1f0 con 0x7f637c099b60 2024-09-16T14:42:32.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f636c001920 con 0x7f637c099b60 2024-09-16T14:42:32.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:32.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:32.056 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.056 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.056 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.056 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.056 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.056 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.057 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.057 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.057 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.057 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:32.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161206565 0 0) 0x7f637c09eca0 con 0x7f637c0a26f0 2024-09-16T14:42:32.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient(hunting): my global_id is 4289 2024-09-16T14:42:32.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6368003530 con 0x7f637c0a26f0 2024-09-16T14:42:32.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc6123e640 1 -- 172.21.15.26:0/1980985806 wait complete. 2024-09-16T14:42:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3841665271 0 0) 0x7f637c09f1f0 con 0x7f637c099b60 2024-09-16T14:42:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 >> v1:172.21.15.26:6790/0 conn(0x7f637c0a26f0 legacy=0x7f637c1d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 >> v1:172.21.15.189:6789/0 conn(0x7f637c0a1c00 legacy=0x7f637c13c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:42:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 1 monclient: found mon.a 2024-09-16T14:42:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c11e970 con 0x7f637c099b60 2024-09-16T14:42:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient: _finish_auth 0 2024-09-16T14:42:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:42:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 5 monclient: authenticate success, global_id 4302 2024-09-16T14:42:32.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 10 monclient: _renew_subs 2024-09-16T14:42:32.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:32.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f63822be640 1 -- 172.21.15.26:0/834212658 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f637c11e690 con 0x7f637c099b60 2024-09-16T14:42:32.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f636c0036a0 con 0x7f637c099b60 2024-09-16T14:42:32.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f6378ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:32.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7f637a7fc640 4 set_mon_vals no callback set 2024-09-16T14:42:32.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc6123e640 10 monclient: build_initial_monmap 2024-09-16T14:42:32.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.032+0000 7fdc6123e640 10 monclient: monmap: 2024-09-16T14:42:32.060 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:32.060 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:32.061 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:32.061 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:32.061 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:32.061 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:32.061 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:32.061 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:32.061 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:32.061 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7f63822be640 10 monclient: _renew_subs 2024-09-16T14:42:32.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f636c0039b0 con 0x7f637c099b60 2024-09-16T14:42:32.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7f63822be640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:32.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7f63822be640 1 -- 172.21.15.26:0/834212658 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f637c13e400 con 0x7f637c099b60 2024-09-16T14:42:32.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7f6378ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7f6378ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:32.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7f6378ff9640 10 monclient: dump: 2024-09-16T14:42:32.062 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.062 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.062 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.062 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.062 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.062 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.062 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.062 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.063 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.063 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 1 Processor -- start 2024-09-16T14:42:32.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 1 -- start start 2024-09-16T14:42:32.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 10 monclient: init 2024-09-16T14:42:32.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f636c003b50 con 0x7f637c099b60 2024-09-16T14:42:32.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7f63822be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:32.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:32.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:32.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 10 monclient(hunting): picked mon.noname-b con 0x7fdc5c099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:32.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 10 monclient(hunting): picked mon.noname-c con 0x7fdc5c0a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:32.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 10 monclient(hunting): picked mon.noname-a con 0x7fdc5c0a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:32.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5c126480 con 0x7fdc5c0a26f0 2024-09-16T14:42:32.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5c0a14b0 con 0x7fdc5c099b60 2024-09-16T14:42:32.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5c127050 con 0x7fdc5c0a1c00 2024-09-16T14:42:32.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:32.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 10 monclient(hunting): authenticate will time out at 976.157277s 2024-09-16T14:42:32.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7f63822be640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:32.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7f63822be640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:32.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7f63822be640 1 -- 172.21.15.26:0/834212658 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f637c0008e0 con 0x7f637c099b60 2024-09-16T14:42:32.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc5ad9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdc5c099b60 0x7fdc5c1cdb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.26:54842/0 (socket says 172.21.15.26:54842) 2024-09-16T14:42:32.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc5ad9d640 1 -- 172.21.15.26:0/2017779120 learned_addr learned my addr 172.21.15.26:0/2017779120 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:32.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 1 -- 172.21.15.26:0/2017779120 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171387874 0 0) 0x7fdc5c0a14b0 con 0x7fdc5c099b60 2024-09-16T14:42:32.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 10 monclient(hunting): my global_id is 4303 2024-09-16T14:42:32.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 1 -- 172.21.15.26:0/2017779120 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc48003530 con 0x7fdc5c099b60 2024-09-16T14:42:32.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 1 -- 172.21.15.26:0/2017779120 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2894245791 0 0) 0x7fdc48003530 con 0x7fdc5c099b60 2024-09-16T14:42:32.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 1 -- 172.21.15.26:0/2017779120 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc5c0a14b0 con 0x7fdc5c099b60 2024-09-16T14:42:32.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 1 -- 172.21.15.26:0/2017779120 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdc44001c10 con 0x7fdc5c099b60 2024-09-16T14:42:32.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:32.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 10 monclient(hunting): dump: 2024-09-16T14:42:32.069 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.069 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.069 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.069 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.069 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.069 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.069 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.069 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.069 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.069 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:32.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 1 -- 172.21.15.26:0/2017779120 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2464141434 0 0) 0x7fdc5c0a14b0 con 0x7fdc5c099b60 2024-09-16T14:42:32.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 1 -- 172.21.15.26:0/2017779120 >> v1:172.21.15.26:6790/0 conn(0x7fdc5c0a1c00 legacy=0x7fdc5c1d2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 1 -- 172.21.15.26:0/2017779120 >> v1:172.21.15.26:6789/0 conn(0x7fdc5c0a26f0 legacy=0x7fdc5c1d6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 10 monclient: _finish_hunting 0 2024-09-16T14:42:32.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 1 monclient: found mon.b 2024-09-16T14:42:32.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:32.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:32.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 1 -- 172.21.15.26:0/2017779120 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c128850 con 0x7fdc5c099b60 2024-09-16T14:42:32.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 10 monclient: _finish_auth 0 2024-09-16T14:42:32.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 10 monclient: _check_auth_tickets 2024-09-16T14:42:32.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:32.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 5 monclient: authenticate success, global_id 4303 2024-09-16T14:42:32.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 10 monclient: _renew_subs 2024-09-16T14:42:32.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:32.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 1 -- 172.21.15.26:0/2017779120 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc5c128290 con 0x7fdc5c099b60 2024-09-16T14:42:32.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 10 monclient: _renew_subs 2024-09-16T14:42:32.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:32.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.036+0000 7fdc6123e640 1 -- 172.21.15.26:0/2017779120 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc5c126dd0 con 0x7fdc5c099b60 2024-09-16T14:42:32.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.040+0000 7fdc537fe640 1 -- 172.21.15.26:0/2017779120 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc44004c00 con 0x7fdc5c099b60 2024-09-16T14:42:32.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.040+0000 7fdc537fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:32.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.040+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f636c0431c0 con 0x7f637c099b60 2024-09-16T14:42:32.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.040+0000 7f637b7fe640 10 monclient: get_auth_request con 0x7f63680363b0 auth_method 0 2024-09-16T14:42:32.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.040+0000 7fdc58d99640 4 set_mon_vals no callback set 2024-09-16T14:42:32.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.040+0000 7fdc537fe640 1 -- 172.21.15.26:0/2017779120 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdc44004f90 con 0x7fdc5c099b60 2024-09-16T14:42:32.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.040+0000 7fdc537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.040+0000 7fdc537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:32.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.040+0000 7fdc537fe640 10 monclient: dump: 2024-09-16T14:42:32.073 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.073 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.073 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.073 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.073 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.073 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.073 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.074 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.040+0000 7fdc537fe640 1 -- 172.21.15.26:0/2017779120 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fdc44005130 con 0x7fdc5c099b60 2024-09-16T14:42:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.040+0000 7fdc537fe640 1 -- 172.21.15.26:0/2017779120 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7fdc44043410 con 0x7fdc5c099b60 2024-09-16T14:42:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.040+0000 7fdc6123e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.040+0000 7fdc6123e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.040+0000 7fdc6123e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.040+0000 7fdc6123e640 1 -- 172.21.15.26:0/2017779120 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3c005050 con 0x7fdc5c099b60 2024-09-16T14:42:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.040+0000 7fdc5a59c640 10 monclient: get_auth_request con 0x7fdc480362d0 auth_method 0 2024-09-16T14:42:32.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.044+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f636c014740 con 0x7f637c099b60 2024-09-16T14:42:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.044+0000 7f6378ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.044+0000 7f6378ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.056+0000 7fdc537fe640 1 -- 172.21.15.26:0/2017779120 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fdc44014b60 con 0x7fdc5c099b60 2024-09-16T14:42:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.056+0000 7fdc537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:32.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.056+0000 7fdc537fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc3a098e640 10 monclient: get_monmap_and_config 2024-09-16T14:42:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc3a098e640 10 monclient: build_initial_monmap 2024-09-16T14:42:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc3a098e640 10 monclient: monmap: 2024-09-16T14:42:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:32.083268+0000 2024-09-16T14:42:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:32.083268+0000 2024-09-16T14:42:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:32.083 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:32.084 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc3a098e640 1 Processor -- start 2024-09-16T14:42:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc3a098e640 1 -- start start 2024-09-16T14:42:32.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc3a098e640 10 monclient: init 2024-09-16T14:42:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc3a098e640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc3a098e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:42:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc3a098e640 10 monclient(hunting): picked mon.noname-b con 0x7fc39c0a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc3a098e640 10 monclient(hunting): picked mon.noname-a con 0x7fc39c099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc3a098e640 10 monclient(hunting): picked mon.noname-c con 0x7fc39c0a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:32.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc3a098e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39c098ea0 con 0x7fc39c099b60 2024-09-16T14:42:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc399d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc39c099b60 0x7fc39c139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57444/0 (socket says 172.21.15.26:57444) 2024-09-16T14:42:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc399d9b640 1 -- 172.21.15.26:0/4167175064 learned_addr learned my addr 172.21.15.26:0/4167175064 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc3a098e640 1 -- 172.21.15.26:0/4167175064 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39c0997b0 con 0x7fc39c0a26f0 2024-09-16T14:42:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc3a098e640 1 -- 172.21.15.26:0/4167175064 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39c099460 con 0x7fc39c0a1c00 2024-09-16T14:42:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc3a098e640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.080+0000 7fc3a098e640 10 monclient(hunting): authenticate will time out at 976.203564s 2024-09-16T14:42:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 1 -- 172.21.15.26:0/4167175064 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216420824 0 0) 0x7fc39c098ea0 con 0x7fc39c099b60 2024-09-16T14:42:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient(hunting): my global_id is 4305 2024-09-16T14:42:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 1 -- 172.21.15.26:0/4167175064 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc384003530 con 0x7fc39c099b60 2024-09-16T14:42:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 1 -- 172.21.15.26:0/4167175064 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079090592 0 0) 0x7fc39c099460 con 0x7fc39c0a1c00 2024-09-16T14:42:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient(hunting): my global_id is 4292 2024-09-16T14:42:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 1 -- 172.21.15.26:0/4167175064 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc39c098ea0 con 0x7fc39c0a1c00 2024-09-16T14:42:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 1 -- 172.21.15.26:0/4167175064 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 448011218 0 0) 0x7fc39c0997b0 con 0x7fc39c0a26f0 2024-09-16T14:42:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient(hunting): my global_id is 4306 2024-09-16T14:42:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 1 -- 172.21.15.26:0/4167175064 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc39c099460 con 0x7fc39c0a26f0 2024-09-16T14:42:32.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 1 -- 172.21.15.26:0/4167175064 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3437389740 0 0) 0x7fc39c098ea0 con 0x7fc39c0a1c00 2024-09-16T14:42:32.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 1 -- 172.21.15.26:0/4167175064 >> v1:172.21.15.189:6789/0 conn(0x7fc39c0a26f0 legacy=0x7fc39c13d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 1 -- 172.21.15.26:0/4167175064 >> v1:172.21.15.26:6789/0 conn(0x7fc39c099b60 legacy=0x7fc39c139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient: _finish_hunting 0 2024-09-16T14:42:32.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 1 monclient: found mon.noname-b 2024-09-16T14:42:32.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:32.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:32.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 1 -- 172.21.15.26:0/4167175064 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c0a34b0 con 0x7fc39c0a1c00 2024-09-16T14:42:32.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient: _finish_auth 0 2024-09-16T14:42:32.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient: _check_auth_tickets 2024-09-16T14:42:32.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:32.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 1 -- 172.21.15.26:0/4167175064 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc38c0014b0 con 0x7fc39c0a1c00 2024-09-16T14:42:32.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:32.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient: dump: 2024-09-16T14:42:32.091 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.091 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.091 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.091 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.091 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.091 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.091 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.091 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.091 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.091 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 1 -- 172.21.15.26:0/4167175064 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc38c001740 con 0x7fc39c0a1c00 2024-09-16T14:42:32.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 5 monclient: authenticate success, global_id 4292 2024-09-16T14:42:32.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:32.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:32.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 1 -- 172.21.15.26:0/4167175064 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc38c001ad0 con 0x7fc39c0a1c00 2024-09-16T14:42:32.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:32.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39959a640 10 monclient: dump: 2024-09-16T14:42:32.092 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.092 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.092 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.092 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.092 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.092 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.092 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.093 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.093 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.093 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:32.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 4 set_mon_vals no callback set 2024-09-16T14:42:32.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 10 monclient: shutdown 2024-09-16T14:42:32.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:32.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 1 -- 172.21.15.26:0/4167175064 >> v1:172.21.15.26:6790/0 conn(0x7fc39c0a1c00 legacy=0x7fc39c0a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 1 -- 172.21.15.26:0/4167175064 shutdown_connections 2024-09-16T14:42:32.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 1 -- 172.21.15.26:0/4167175064 >> 172.21.15.26:0/4167175064 conn(0x7fc39c0a3d00 msgr2=0x7fc39c0a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:32.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 1 -- 172.21.15.26:0/4167175064 shutdown_connections 2024-09-16T14:42:32.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 1 -- 172.21.15.26:0/4167175064 wait complete. 2024-09-16T14:42:32.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 10 monclient: build_initial_monmap 2024-09-16T14:42:32.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 10 monclient: monmap: 2024-09-16T14:42:32.094 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:32.094 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:32.094 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:32.094 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:32.094 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:32.094 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:32.094 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:32.094 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:32.094 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:32.095 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 1 Processor -- start 2024-09-16T14:42:32.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 1 -- start start 2024-09-16T14:42:32.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 10 monclient: init 2024-09-16T14:42:32.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7f59d3532640 10 monclient: get_monmap_and_config 2024-09-16T14:42:32.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7f59d3532640 10 monclient: build_initial_monmap 2024-09-16T14:42:32.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7f59d3532640 10 monclient: monmap: 2024-09-16T14:42:32.095 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:32.088765+0000 2024-09-16T14:42:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:32.088765+0000 2024-09-16T14:42:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:32.096 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:32.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:32.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 10 monclient(hunting): picked mon.noname-a con 0x7fc39c099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:32.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 10 monclient(hunting): picked mon.noname-c con 0x7fc39c0a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:32.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 10 monclient(hunting): picked mon.noname-b con 0x7fc39c0a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:32.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39a59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc39c099b60 0x7fc39c1c9b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57446/0 (socket says 172.21.15.26:57446) 2024-09-16T14:42:32.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39c11dcc0 con 0x7fc39c099b60 2024-09-16T14:42:32.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc39a59c640 1 -- 172.21.15.26:0/1738526194 learned_addr learned my addr 172.21.15.26:0/1738526194 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:32.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 1 -- 172.21.15.26:0/1738526194 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39c11e870 con 0x7fc39c0a26f0 2024-09-16T14:42:32.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 1 -- 172.21.15.26:0/1738526194 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39c120000 con 0x7fc39c0a1c00 2024-09-16T14:42:32.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:32.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.084+0000 7fc3a098e640 10 monclient(hunting): authenticate will time out at 976.207841s 2024-09-16T14:42:32.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710358613 0 0) 0x7fc39c120000 con 0x7fc39c0a1c00 2024-09-16T14:42:32.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient(hunting): my global_id is 4295 2024-09-16T14:42:32.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc37c003530 con 0x7fc39c0a1c00 2024-09-16T14:42:32.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049984110 0 0) 0x7fc39c11dcc0 con 0x7fc39c099b60 2024-09-16T14:42:32.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient(hunting): my global_id is 4308 2024-09-16T14:42:32.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc39c120000 con 0x7fc39c099b60 2024-09-16T14:42:32.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1572622692 0 0) 0x7fc37c003530 con 0x7fc39c0a1c00 2024-09-16T14:42:32.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc39c11dcc0 con 0x7fc39c0a1c00 2024-09-16T14:42:32.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc390001860 con 0x7fc39c0a1c00 2024-09-16T14:42:32.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:32.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient(hunting): dump: 2024-09-16T14:42:32.100 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.100 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.100 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.100 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.101 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.101 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.101 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.101 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.101 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.101 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:32.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984988652 0 0) 0x7fc39c11e870 con 0x7fc39c0a26f0 2024-09-16T14:42:32.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient(hunting): my global_id is 4309 2024-09-16T14:42:32.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc37c003530 con 0x7fc39c0a26f0 2024-09-16T14:42:32.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2580682422 0 0) 0x7fc39c11dcc0 con 0x7fc39c0a1c00 2024-09-16T14:42:32.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 >> v1:172.21.15.189:6789/0 conn(0x7fc39c0a26f0 legacy=0x7fc39c1d22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 >> v1:172.21.15.26:6789/0 conn(0x7fc39c099b60 legacy=0x7fc39c1c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:32.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 monclient: found mon.c 2024-09-16T14:42:32.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:32.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:32.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c09f050 con 0x7fc39c0a1c00 2024-09-16T14:42:32.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient: _finish_auth 0 2024-09-16T14:42:32.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:32.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:32.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc3a098e640 5 monclient: authenticate success, global_id 4295 2024-09-16T14:42:32.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc3a098e640 10 monclient: _renew_subs 2024-09-16T14:42:32.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc3a098e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:32.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc3a098e640 1 -- 172.21.15.26:0/1738526194 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc39c11fa40 con 0x7fc39c0a1c00 2024-09-16T14:42:32.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc3a098e640 10 monclient: _renew_subs 2024-09-16T14:42:32.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc3a098e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:32.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc3a098e640 1 -- 172.21.15.26:0/1738526194 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc39c11e5f0 con 0x7fc39c0a1c00 2024-09-16T14:42:32.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3900048e0 con 0x7fc39c0a1c00 2024-09-16T14:42:32.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:32.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc390004bf0 con 0x7fc39c0a1c00 2024-09-16T14:42:32.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:32.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 10 monclient: dump: 2024-09-16T14:42:32.104 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.104 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.104 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.104 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.104 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.104 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.104 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.104 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.104 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.104 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc398d99640 4 set_mon_vals no callback set 2024-09-16T14:42:32.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc390004d50 con 0x7fc39c0a1c00 2024-09-16T14:42:32.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7fc390043010 con 0x7fc39c0a1c00 2024-09-16T14:42:32.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.088+0000 7fc39a59c640 10 monclient: get_auth_request con 0x7fc37c0360a0 auth_method 0 2024-09-16T14:42:32.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 1 Processor -- start 2024-09-16T14:42:32.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7fc3a098e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:32.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7fc3a098e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:32.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7fc3a098e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:32.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7fc3a098e640 1 -- 172.21.15.26:0/1738526194 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc380005050 con 0x7fc39c0a1c00 2024-09-16T14:42:32.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e375a640 10 monclient: get_monmap_and_config 2024-09-16T14:42:32.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e375a640 10 monclient: build_initial_monmap 2024-09-16T14:42:32.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e375a640 10 monclient: monmap: 2024-09-16T14:42:32.108 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:32.108 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.108 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:32.094577+0000 2024-09-16T14:42:32.108 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:32.094577+0000 2024-09-16T14:42:32.108 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:32.108 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.108 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:32.108 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:32.108 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:32.108 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e375a640 1 Processor -- start 2024-09-16T14:42:32.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e375a640 1 -- start start 2024-09-16T14:42:32.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e375a640 10 monclient: init 2024-09-16T14:42:32.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e375a640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:32.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e375a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:42:32.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e375a640 10 monclient(hunting): picked mon.noname-a con 0x7f73dc0a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:32.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e375a640 10 monclient(hunting): picked mon.noname-c con 0x7f73dc099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:32.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e375a640 10 monclient(hunting): picked mon.noname-b con 0x7f73dc0a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:32.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e375a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73dc098ea0 con 0x7f73dc0a1c00 2024-09-16T14:42:32.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e375a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73dc0997b0 con 0x7f73dc099b60 2024-09-16T14:42:32.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e1cf7640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f73dc0a26f0 0x7f73dc13d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48504/0 (socket says 172.21.15.26:48504) 2024-09-16T14:42:32.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e1cf7640 1 -- 172.21.15.26:0/3566996014 learned_addr learned my addr 172.21.15.26:0/3566996014 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:32.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e375a640 1 -- 172.21.15.26:0/3566996014 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73dc099460 con 0x7f73dc0a26f0 2024-09-16T14:42:32.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e375a640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:32.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73e375a640 10 monclient(hunting): authenticate will time out at 976.214827s 2024-09-16T14:42:32.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73d3fff640 1 -- 172.21.15.26:0/3566996014 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838026423 0 0) 0x7f73dc0997b0 con 0x7f73dc099b60 2024-09-16T14:42:32.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73d3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73d3fff640 10 monclient(hunting): my global_id is 4315 2024-09-16T14:42:32.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f73d3fff640 1 -- 172.21.15.26:0/3566996014 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73c8003530 con 0x7f73dc099b60 2024-09-16T14:42:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 1 -- start start 2024-09-16T14:42:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 10 monclient: init 2024-09-16T14:42:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 10 monclient(hunting): picked mon.noname-a con 0x7f59c4026e40 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 10 monclient(hunting): picked mon.noname-b con 0x7f59c40234c0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 10 monclient(hunting): picked mon.noname-c con 0x7f59c4023900 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c4027220 con 0x7f59c4026e40 2024-09-16T14:42:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c4025890 con 0x7f59c4023900 2024-09-16T14:42:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c4024ff0 con 0x7f59c40234c0 2024-09-16T14:42:32.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 10 monclient(hunting): authenticate will time out at 976.213006s 2024-09-16T14:42:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d2d31640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f59c4023900 0x7f59c40c1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.26:54884/0 (socket says 172.21.15.26:54884) 2024-09-16T14:42:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d2d31640 1 -- 172.21.15.26:0/770554730 learned_addr learned my addr 172.21.15.26:0/770554730 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 1 -- 172.21.15.26:0/770554730 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460328496 0 0) 0x7f59c4025890 con 0x7f59c4023900 2024-09-16T14:42:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 10 monclient(hunting): my global_id is 4312 2024-09-16T14:42:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 1 -- 172.21.15.26:0/770554730 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b8003530 con 0x7f59c4023900 2024-09-16T14:42:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 1 -- 172.21.15.26:0/770554730 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 112247696 0 0) 0x7f59b8003530 con 0x7f59c4023900 2024-09-16T14:42:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 1 -- 172.21.15.26:0/770554730 >> v1:172.21.15.26:6790/0 conn(0x7f59c40234c0 legacy=0x7f59c40be3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:42:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 1 -- 172.21.15.26:0/770554730 >> v1:172.21.15.26:6789/0 conn(0x7f59c4026e40 legacy=0x7f59c4022db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:42:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 10 monclient: _finish_hunting 0 2024-09-16T14:42:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 1 monclient: found mon.noname-c 2024-09-16T14:42:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:42:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 1 -- 172.21.15.26:0/770554730 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c40252d0 con 0x7f59c4023900 2024-09-16T14:42:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 10 monclient: _finish_auth 0 2024-09-16T14:42:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 10 monclient: _check_auth_tickets 2024-09-16T14:42:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 1 -- 172.21.15.26:0/770554730 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f59cc04e740 con 0x7f59c4023900 2024-09-16T14:42:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 10 monclient: dump: 2024-09-16T14:42:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.115 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 5 monclient: authenticate success, global_id 4312 2024-09-16T14:42:32.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 1 -- 172.21.15.26:0/770554730 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59cc04ea70 con 0x7f59c4023900 2024-09-16T14:42:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 1 -- 172.21.15.26:0/770554730 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f59cc067660 con 0x7f59c4023900 2024-09-16T14:42:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d152e640 10 monclient: dump: 2024-09-16T14:42:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.116 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.117 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 4 set_mon_vals no callback set 2024-09-16T14:42:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 10 monclient: shutdown 2024-09-16T14:42:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 1 -- 172.21.15.26:0/770554730 >> v1:172.21.15.189:6789/0 conn(0x7f59c4023900 legacy=0x7f59c40c1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 1 -- 172.21.15.26:0/770554730 shutdown_connections 2024-09-16T14:42:32.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.092+0000 7f59d3532640 1 -- 172.21.15.26:0/770554730 >> 172.21.15.26:0/770554730 conn(0x7f59c4005d50 msgr2=0x7f59c4006160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:32.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 1 -- 172.21.15.26:0/3566996014 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2790485342 0 0) 0x7f73c8003530 con 0x7f73dc099b60 2024-09-16T14:42:32.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 1 -- 172.21.15.26:0/3566996014 >> v1:172.21.15.26:6790/0 conn(0x7f73dc0a26f0 legacy=0x7f73dc13d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 1 -- 172.21.15.26:0/3566996014 >> v1:172.21.15.26:6789/0 conn(0x7f73dc0a1c00 legacy=0x7f73dc0a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:42:32.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 1 -- 172.21.15.26:0/770554730 shutdown_connections 2024-09-16T14:42:32.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 1 -- 172.21.15.26:0/770554730 wait complete. 2024-09-16T14:42:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:42:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 1 monclient: found mon.noname-c 2024-09-16T14:42:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:42:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 1 -- 172.21.15.26:0/3566996014 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73dc0a34b0 con 0x7f73dc099b60 2024-09-16T14:42:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 10 monclient: _finish_auth 0 2024-09-16T14:42:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:42:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 1 -- 172.21.15.26:0/3566996014 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f73cc001620 con 0x7f73dc099b60 2024-09-16T14:42:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 10 monclient: dump: 2024-09-16T14:42:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.122 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73e375a640 5 monclient: authenticate success, global_id 4315 2024-09-16T14:42:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73e375a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 10 monclient: build_initial_monmap 2024-09-16T14:42:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 10 monclient: monmap: 2024-09-16T14:42:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:32.123 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 1 Processor -- start 2024-09-16T14:42:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 1 -- 172.21.15.26:0/3566996014 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73cc001910 con 0x7f73dc099b60 2024-09-16T14:42:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 1 -- 172.21.15.26:0/3566996014 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f73cc001d20 con 0x7f73dc099b60 2024-09-16T14:42:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73d3fff640 10 monclient: dump: 2024-09-16T14:42:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.125 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73e375a640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73e375a640 4 set_mon_vals no callback set 2024-09-16T14:42:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 1 -- start start 2024-09-16T14:42:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 10 monclient: init 2024-09-16T14:42:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:42:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 10 monclient(hunting): picked mon.noname-a con 0x7f59c4023900 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 10 monclient(hunting): picked mon.noname-b con 0x7f59c4026e40 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 10 monclient(hunting): picked mon.noname-c con 0x7f59c401dbe0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c401cf30 con 0x7f59c4023900 2024-09-16T14:42:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c40b6420 con 0x7f59c4026e40 2024-09-16T14:42:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c4007750 con 0x7f59c401dbe0 2024-09-16T14:42:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 10 monclient(hunting): authenticate will time out at 976.217972s 2024-09-16T14:42:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73e375a640 10 monclient: shutdown 2024-09-16T14:42:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73e375a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73e375a640 1 -- 172.21.15.26:0/3566996014 >> v1:172.21.15.189:6789/0 conn(0x7f73dc099b60 legacy=0x7f73dc139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d2530640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f59c4023900 0x7f59c40b5600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57472/0 (socket says 172.21.15.26:57472) 2024-09-16T14:42:32.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d2530640 1 -- 172.21.15.26:0/3118323057 learned_addr learned my addr 172.21.15.26:0/3118323057 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:32.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73e375a640 1 -- 172.21.15.26:0/3566996014 shutdown_connections 2024-09-16T14:42:32.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f73e375a640 1 -- 172.21.15.26:0/3566996014 >> 172.21.15.26:0/3566996014 conn(0x7f73dc0a3d00 msgr2=0x7f73dc0a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:32.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 1 -- 172.21.15.26:0/3118323057 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965890775 0 0) 0x7f59c401cf30 con 0x7f59c4023900 2024-09-16T14:42:32.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 10 monclient(hunting): my global_id is 4311 2024-09-16T14:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 1 -- 172.21.15.26:0/3118323057 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b0003530 con 0x7f59c4023900 2024-09-16T14:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 1 -- 172.21.15.26:0/3118323057 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 865651088 0 0) 0x7f59b0003530 con 0x7f59c4023900 2024-09-16T14:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 1 -- 172.21.15.26:0/3118323057 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59c401cf30 con 0x7f59c4023900 2024-09-16T14:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 1 -- 172.21.15.26:0/3118323057 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f59c80018f0 con 0x7f59c4023900 2024-09-16T14:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 1 -- 172.21.15.26:0/3118323057 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3655301186 0 0) 0x7f59c401cf30 con 0x7f59c4023900 2024-09-16T14:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 1 -- 172.21.15.26:0/3118323057 >> v1:172.21.15.26:6790/0 conn(0x7f59c401dbe0 legacy=0x7f59c417a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 1 -- 172.21.15.26:0/3118323057 >> v1:172.21.15.189:6789/0 conn(0x7f59c4026e40 legacy=0x7f59c40b5d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 1 monclient: found mon.a 2024-09-16T14:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 1 -- 172.21.15.26:0/3118323057 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c40225e0 con 0x7f59c4023900 2024-09-16T14:42:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 5 monclient: authenticate success, global_id 4311 2024-09-16T14:42:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 10 monclient: _renew_subs 2024-09-16T14:42:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 1 -- 172.21.15.26:0/3118323057 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c401e030 con 0x7f59c4023900 2024-09-16T14:42:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 10 monclient: _renew_subs 2024-09-16T14:42:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.096+0000 7f59d3532640 1 -- 172.21.15.26:0/3118323057 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c401b5b0 con 0x7f59c4023900 2024-09-16T14:42:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f59c2ffd640 1 -- 172.21.15.26:0/3118323057 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59c8001c50 con 0x7f59c4023900 2024-09-16T14:42:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f59c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f59c2ffd640 1 -- 172.21.15.26:0/3118323057 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f59c8003a50 con 0x7f59c4023900 2024-09-16T14:42:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f59c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f59c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f59c2ffd640 10 monclient: dump: 2024-09-16T14:42:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.134 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f59d0d2d640 4 set_mon_vals no callback set 2024-09-16T14:42:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f59d3532640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f59c2ffd640 1 -- 172.21.15.26:0/3118323057 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f59c8003bf0 con 0x7f59c4023900 2024-09-16T14:42:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f59c2ffd640 1 -- 172.21.15.26:0/3118323057 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f59c8043120 con 0x7f59c4023900 2024-09-16T14:42:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f59d3532640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f59d3532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f59d3532640 1 -- 172.21.15.26:0/3118323057 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b4005050 con 0x7f59c4023900 2024-09-16T14:42:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 1 -- 172.21.15.26:0/3566996014 shutdown_connections 2024-09-16T14:42:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 1 -- 172.21.15.26:0/3566996014 wait complete. 2024-09-16T14:42:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 10 monclient: build_initial_monmap 2024-09-16T14:42:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 10 monclient: monmap: 2024-09-16T14:42:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:32.135 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 1 Processor -- start 2024-09-16T14:42:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f59d1d2f640 10 monclient: get_auth_request con 0x7f59b0036180 auth_method 0 2024-09-16T14:42:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 1 -- start start 2024-09-16T14:42:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 10 monclient: init 2024-09-16T14:42:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 10 monclient(hunting): picked mon.noname-c con 0x7f73dc099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 10 monclient(hunting): picked mon.noname-b con 0x7f73dc0a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 10 monclient(hunting): picked mon.noname-a con 0x7f73dc06e2e0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73dc06f4f0 con 0x7f73dc06e2e0 2024-09-16T14:42:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73dc09f270 con 0x7f73dc0a26f0 2024-09-16T14:42:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73dc070ff0 con 0x7f73dc099b60 2024-09-16T14:42:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 10 monclient(hunting): authenticate will time out at 976.221896s 2024-09-16T14:42:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc390014720 con 0x7fc39c0a1c00 2024-09-16T14:42:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7fc38affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7fc38affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e14f6640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f73dc099b60 0x7f73dc1cdb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48524/0 (socket says 172.21.15.26:48524) 2024-09-16T14:42:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e14f6640 1 -- 172.21.15.26:0/2871580983 learned_addr learned my addr 172.21.15.26:0/2871580983 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824512079 0 0) 0x7f73dc09f270 con 0x7f73dc0a26f0 2024-09-16T14:42:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient(hunting): my global_id is 4318 2024-09-16T14:42:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73c0003530 con 0x7f73dc0a26f0 2024-09-16T14:42:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447611743 0 0) 0x7f73dc070ff0 con 0x7f73dc099b60 2024-09-16T14:42:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient(hunting): my global_id is 4298 2024-09-16T14:42:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73dc09f270 con 0x7f73dc099b60 2024-09-16T14:42:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 832547986 0 0) 0x7f73c0003530 con 0x7f73dc0a26f0 2024-09-16T14:42:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73dc070ff0 con 0x7f73dc0a26f0 2024-09-16T14:42:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f73cc0015e0 con 0x7f73dc0a26f0 2024-09-16T14:42:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient(hunting): dump: 2024-09-16T14:42:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.142 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2488941868 0 0) 0x7f73dc09f270 con 0x7f73dc099b60 2024-09-16T14:42:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73c0003530 con 0x7f73dc099b60 2024-09-16T14:42:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f73d8001840 con 0x7f73dc099b60 2024-09-16T14:42:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient(hunting): dump: 2024-09-16T14:42:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.144 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 198483294 0 0) 0x7f73dc070ff0 con 0x7f73dc0a26f0 2024-09-16T14:42:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 >> v1:172.21.15.26:6790/0 conn(0x7f73dc099b60 legacy=0x7f73dc1cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 >> v1:172.21.15.26:6789/0 conn(0x7f73dc06e2e0 legacy=0x7f73dc1d6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:42:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 1 monclient: found mon.b 2024-09-16T14:42:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73dc0712d0 con 0x7f73dc0a26f0 2024-09-16T14:42:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:42:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:42:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 5 monclient: authenticate success, global_id 4318 2024-09-16T14:42:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 10 monclient: _renew_subs 2024-09-16T14:42:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 1 -- 172.21.15.26:0/2871580983 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73dc09ed20 con 0x7f73dc0a26f0 2024-09-16T14:42:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 10 monclient: _renew_subs 2024-09-16T14:42:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73e375a640 1 -- 172.21.15.26:0/2871580983 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73dc070d10 con 0x7f73dc0a26f0 2024-09-16T14:42:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73cc001200 con 0x7f73dc0a26f0 2024-09-16T14:42:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.100+0000 7f73d37fe640 4 set_mon_vals no callback set 2024-09-16T14:42:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.104+0000 7f73e375a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.104+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f73cc003b90 con 0x7f73dc0a26f0 2024-09-16T14:42:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.104+0000 7f73d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.104+0000 7f73d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.104+0000 7f73d1ffb640 10 monclient: dump: 2024-09-16T14:42:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:32.147 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.104+0000 7f73e375a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.104+0000 7f73e375a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.104+0000 7f73e375a640 1 -- 172.21.15.26:0/2871580983 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73bc005050 con 0x7f73dc0a26f0 2024-09-16T14:42:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.104+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f73cc00e680 con 0x7f73dc0a26f0 2024-09-16T14:42:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.104+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f73cc0421f0 con 0x7f73dc0a26f0 2024-09-16T14:42:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.104+0000 7f73e14f6640 10 monclient: get_auth_request con 0x7f73c0036220 auth_method 0 2024-09-16T14:42:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.104+0000 7f59c2ffd640 1 -- 172.21.15.26:0/3118323057 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f59c80146d0 con 0x7f59c4023900 2024-09-16T14:42:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.104+0000 7f59c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.104+0000 7f59c2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.116+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f73cc00fbd0 con 0x7f73dc0a26f0 2024-09-16T14:42:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.116+0000 7f73d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.116+0000 7f73d1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:32.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.176+0000 7f8d41d9a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T14:42:32.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.176+0000 7f8d41d9a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T14:42:32.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.176+0000 7f8d41d9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:32.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.176+0000 7f8d41d9a640 1 -- 172.21.15.26:0/342145525 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f8d3c06c790 con 0x7f8d3c0a0000 2024-09-16T14:42:32.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.176+0000 7f8d38ff9640 1 -- 172.21.15.26:0/342145525 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 555833778) 0x7f8d2c012a60 con 0x7f8d3c0a0000 2024-09-16T14:42:32.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.176+0000 7f8d38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T14:42:32.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.176+0000 7f8d38ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:32.178 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-09-16T14:42:32.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.176+0000 7f8d41d9a640 1 -- 172.21.15.26:0/342145525 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8d1c036490 legacy=0x7f8d1c038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.176+0000 7f8d41d9a640 10 monclient: shutdown 2024-09-16T14:42:32.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.176+0000 7f8d41d9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:32.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.176+0000 7f8d41d9a640 1 -- 172.21.15.26:0/342145525 >> v1:172.21.15.26:6790/0 conn(0x7f8d3c0a0000 legacy=0x7f8d3c133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.176+0000 7f8d41d9a640 1 -- 172.21.15.26:0/342145525 shutdown_connections 2024-09-16T14:42:32.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.176+0000 7f8d41d9a640 1 -- 172.21.15.26:0/342145525 >> 172.21.15.26:0/342145525 conn(0x7f8d3c0a7f30 msgr2=0x7f8d3c06b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:32.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.176+0000 7f8d41d9a640 1 -- 172.21.15.26:0/342145525 shutdown_connections 2024-09-16T14:42:32.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.176+0000 7f8d41d9a640 1 -- 172.21.15.26:0/342145525 wait complete. 2024-09-16T14:42:32.195 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.2 2024-09-16T14:42:32.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.256+0000 7fc3a098e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T14:42:32.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.256+0000 7fc3a098e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T14:42:32.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.256+0000 7fc3a098e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:32.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.256+0000 7fc3a098e640 1 -- 172.21.15.26:0/1738526194 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fc380005300 con 0x7fc39c0a1c00 2024-09-16T14:42:32.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.256+0000 7fc38affd640 1 -- 172.21.15.26:0/1738526194 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 555833778) 0x7fc39001cd10 con 0x7fc39c0a1c00 2024-09-16T14:42:32.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.256+0000 7fc38affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T14:42:32.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.256+0000 7fc38affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:32.260 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-09-16T14:42:32.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.260+0000 7fc3a098e640 1 -- 172.21.15.26:0/1738526194 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc37c0360a0 legacy=0x7fc37c038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.260+0000 7fc3a098e640 10 monclient: shutdown 2024-09-16T14:42:32.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.260+0000 7fc3a098e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:32.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.260+0000 7fc3a098e640 1 -- 172.21.15.26:0/1738526194 >> v1:172.21.15.26:6790/0 conn(0x7fc39c0a1c00 legacy=0x7fc39c1ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.260+0000 7fc3a098e640 1 -- 172.21.15.26:0/1738526194 shutdown_connections 2024-09-16T14:42:32.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.260+0000 7fc3a098e640 1 -- 172.21.15.26:0/1738526194 >> 172.21.15.26:0/1738526194 conn(0x7fc39c0a3d00 msgr2=0x7fc39c0a0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:32.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.260+0000 7fc3a098e640 1 -- 172.21.15.26:0/1738526194 shutdown_connections 2024-09-16T14:42:32.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.260+0000 7fc3a098e640 1 -- 172.21.15.26:0/1738526194 wait complete. 2024-09-16T14:42:32.276 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.0 2024-09-16T14:42:32.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.316+0000 7f63822be640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T14:42:32.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.316+0000 7f63822be640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T14:42:32.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.316+0000 7f63822be640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:32.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.316+0000 7f63822be640 1 -- 172.21.15.26:0/834212658 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f637c06c700 con 0x7f637c099b60 2024-09-16T14:42:32.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.316+0000 7f6378ff9640 1 -- 172.21.15.26:0/834212658 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (unknown 383520633 0 3327677263) 0x7f636c014330 con 0x7f637c099b60 2024-09-16T14:42:32.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.316+0000 7f6378ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T14:42:32.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.316+0000 7f6378ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:32.318 INFO:teuthology.orchestra.run.smithi026.stdout:47244640257 2024-09-16T14:42:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.320+0000 7f63822be640 1 -- 172.21.15.26:0/834212658 >> v1:172.21.15.189:6812/3196161582 conn(0x7f63680363b0 legacy=0x7f6368038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.320+0000 7f63822be640 10 monclient: shutdown 2024-09-16T14:42:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.320+0000 7f63822be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.320+0000 7f63822be640 1 -- 172.21.15.26:0/834212658 >> v1:172.21.15.26:6789/0 conn(0x7f637c099b60 legacy=0x7f637c13c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.320+0000 7f63822be640 1 -- 172.21.15.26:0/834212658 shutdown_connections 2024-09-16T14:42:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.320+0000 7f63822be640 1 -- 172.21.15.26:0/834212658 >> 172.21.15.26:0/834212658 conn(0x7f637c0a3d00 msgr2=0x7f637c0a0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.320+0000 7f63822be640 1 -- 172.21.15.26:0/834212658 shutdown_connections 2024-09-16T14:42:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.320+0000 7f63822be640 1 -- 172.21.15.26:0/834212658 wait complete. 2024-09-16T14:42:32.340 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.3 2024-09-16T14:42:32.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.352+0000 7fdc6123e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T14:42:32.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.352+0000 7fdc6123e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T14:42:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.352+0000 7fdc6123e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.352+0000 7fdc6123e640 1 -- 172.21.15.26:0/2017779120 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fdc3c005300 con 0x7fdc5c099b60 2024-09-16T14:42:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.352+0000 7fdc537fe640 1 -- 172.21.15.26:0/2017779120 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (unknown 1823589948 0 555833778) 0x7fdc44012fd0 con 0x7fdc5c099b60 2024-09-16T14:42:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.352+0000 7fdc537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T14:42:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.352+0000 7fdc537fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:32.355 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-09-16T14:42:32.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.356+0000 7fdc6123e640 1 -- 172.21.15.26:0/2017779120 >> v1:172.21.15.189:6812/3196161582 conn(0x7fdc480362d0 legacy=0x7fdc48038790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.356+0000 7fdc6123e640 10 monclient: shutdown 2024-09-16T14:42:32.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.356+0000 7fdc6123e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:32.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.356+0000 7fdc6123e640 1 -- 172.21.15.26:0/2017779120 >> v1:172.21.15.189:6789/0 conn(0x7fdc5c099b60 legacy=0x7fdc5c1cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.356+0000 7fdc6123e640 1 -- 172.21.15.26:0/2017779120 shutdown_connections 2024-09-16T14:42:32.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.356+0000 7fdc6123e640 1 -- 172.21.15.26:0/2017779120 >> 172.21.15.26:0/2017779120 conn(0x7fdc5c0a3d00 msgr2=0x7fdc5c09f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:32.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.356+0000 7fdc6123e640 1 -- 172.21.15.26:0/2017779120 shutdown_connections 2024-09-16T14:42:32.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.356+0000 7fdc6123e640 1 -- 172.21.15.26:0/2017779120 wait complete. 2024-09-16T14:42:32.372 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.4 2024-09-16T14:42:32.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.384+0000 7f73e375a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T14:42:32.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.384+0000 7f73e375a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T14:42:32.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.384+0000 7f73e375a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:42:32.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.384+0000 7f73e375a640 1 -- 172.21.15.26:0/2871580983 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f73bc005300 con 0x7f73dc0a26f0 2024-09-16T14:42:32.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f73d1ffb640 1 -- 172.21.15.26:0/2871580983 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (unknown 2131975755 0 555833778) 0x7f73cc011c70 con 0x7f73dc0a26f0 2024-09-16T14:42:32.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f73d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T14:42:32.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f73d1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:32.389 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-09-16T14:42:32.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f59d3532640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T14:42:32.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f59d3532640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T14:42:32.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f59d3532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:32.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f59d3532640 1 -- 172.21.15.26:0/3118323057 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f59b4005300 con 0x7f59c4023900 2024-09-16T14:42:32.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f59c2ffd640 1 -- 172.21.15.26:0/3118323057 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 3327677263) 0x7f59c8012b40 con 0x7f59c4023900 2024-09-16T14:42:32.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f59c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T14:42:32.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f59c2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:32.391 INFO:teuthology.orchestra.run.smithi026.stdout:47244640257 2024-09-16T14:42:32.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f73e375a640 1 -- 172.21.15.26:0/2871580983 >> v1:172.21.15.189:6812/3196161582 conn(0x7f73c0036220 legacy=0x7f73c00386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f73e375a640 10 monclient: shutdown 2024-09-16T14:42:32.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f73e375a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:32.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f73e375a640 1 -- 172.21.15.26:0/2871580983 >> v1:172.21.15.189:6789/0 conn(0x7f73dc0a26f0 legacy=0x7f73dc1d2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f59d3532640 1 -- 172.21.15.26:0/3118323057 >> v1:172.21.15.189:6812/3196161582 conn(0x7f59b0036180 legacy=0x7f59b0038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f59d3532640 10 monclient: shutdown 2024-09-16T14:42:32.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f73e375a640 1 -- 172.21.15.26:0/2871580983 shutdown_connections 2024-09-16T14:42:32.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f59d3532640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:32.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f59d3532640 1 -- 172.21.15.26:0/3118323057 >> v1:172.21.15.26:6789/0 conn(0x7f59c4023900 legacy=0x7f59c40b5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:32.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f73e375a640 1 -- 172.21.15.26:0/2871580983 >> 172.21.15.26:0/2871580983 conn(0x7f73dc0a3d00 msgr2=0x7f73dc0a0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:32.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f59d3532640 1 -- 172.21.15.26:0/3118323057 shutdown_connections 2024-09-16T14:42:32.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f59d3532640 1 -- 172.21.15.26:0/3118323057 >> 172.21.15.26:0/3118323057 conn(0x7f59c4005d50 msgr2=0x7f59c401f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:32.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f59d3532640 1 -- 172.21.15.26:0/3118323057 shutdown_connections 2024-09-16T14:42:32.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f59d3532640 1 -- 172.21.15.26:0/3118323057 wait complete. 2024-09-16T14:42:32.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f73e375a640 1 -- 172.21.15.26:0/2871580983 shutdown_connections 2024-09-16T14:42:32.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:32.388+0000 7f73e375a640 1 -- 172.21.15.26:0/2871580983 wait complete. 2024-09-16T14:42:32.408 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.1 2024-09-16T14:42:32.409 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.5 2024-09-16T14:42:33.196 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T14:42:33.277 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T14:42:33.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.320+0000 7fed20942640 10 monclient: get_monmap_and_config 2024-09-16T14:42:33.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.320+0000 7fed20942640 10 monclient: build_initial_monmap 2024-09-16T14:42:33.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.320+0000 7fed20942640 10 monclient: monmap: 2024-09-16T14:42:33.324 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:33.324 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.324 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:33.324882+0000 2024-09-16T14:42:33.324 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:33.324882+0000 2024-09-16T14:42:33.325 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:33.325 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.325 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:33.325 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:33.325 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:33.325 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 1 Processor -- start 2024-09-16T14:42:33.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 1 -- start start 2024-09-16T14:42:33.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient: init 2024-09-16T14:42:33.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:33.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:33.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient(hunting): picked mon.noname-b con 0x7fed1c132850 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:33.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient(hunting): picked mon.noname-c con 0x7fed1c071530 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:33.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient(hunting): picked mon.noname-a con 0x7fed1c138f40 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:33.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed1c0719a0 con 0x7fed1c138f40 2024-09-16T14:42:33.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed1c070610 con 0x7fed1c071530 2024-09-16T14:42:33.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed1c06cd70 con 0x7fed1c132850 2024-09-16T14:42:33.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:33.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient(hunting): authenticate will time out at 977.444758s 2024-09-16T14:42:33.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fed1c132850 0x7fed1c132c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48534/0 (socket says 172.21.15.26:48534) 2024-09-16T14:42:33.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1ad9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fed1c138f40 0x7fed1c13b330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57488/0 (socket says 172.21.15.26:57488) 2024-09-16T14:42:33.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1a59c640 1 -- 172.21.15.26:0/3196033209 learned_addr learned my addr 172.21.15.26:0/3196033209 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:33.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 1 -- 172.21.15.26:0/3196033209 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212665375 0 0) 0x7fed1c0719a0 con 0x7fed1c138f40 2024-09-16T14:42:33.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient(hunting): my global_id is 4314 2024-09-16T14:42:33.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 1 -- 172.21.15.26:0/3196033209 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed08003530 con 0x7fed1c138f40 2024-09-16T14:42:33.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 1 -- 172.21.15.26:0/3196033209 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471236594 0 0) 0x7fed1c06cd70 con 0x7fed1c132850 2024-09-16T14:42:33.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient(hunting): my global_id is 4301 2024-09-16T14:42:33.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 1 -- 172.21.15.26:0/3196033209 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed1c0719a0 con 0x7fed1c132850 2024-09-16T14:42:33.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 1 -- 172.21.15.26:0/3196033209 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2348027093 0 0) 0x7fed1c0719a0 con 0x7fed1c132850 2024-09-16T14:42:33.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 1 -- 172.21.15.26:0/3196033209 >> v1:172.21.15.189:6789/0 conn(0x7fed1c071530 legacy=0x7fed1c137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 1 -- 172.21.15.26:0/3196033209 >> v1:172.21.15.26:6789/0 conn(0x7fed1c138f40 legacy=0x7fed1c13b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient: _finish_hunting 0 2024-09-16T14:42:33.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 1 monclient: found mon.noname-b 2024-09-16T14:42:33.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:33.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:33.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 1 -- 172.21.15.26:0/3196033209 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1c06d6c0 con 0x7fed1c132850 2024-09-16T14:42:33.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient: _finish_auth 0 2024-09-16T14:42:33.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient: _check_auth_tickets 2024-09-16T14:42:33.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:33.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 1 -- 172.21.15.26:0/3196033209 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fed04001600 con 0x7fed1c132850 2024-09-16T14:42:33.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:33.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient: dump: 2024-09-16T14:42:33.333 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.333 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.333 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.333 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.333 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.333 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.334 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.334 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.334 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.334 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 5 monclient: authenticate success, global_id 4301 2024-09-16T14:42:33.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:33.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 1 -- 172.21.15.26:0/3196033209 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed040018b0 con 0x7fed1c132850 2024-09-16T14:42:33.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:33.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 1 -- 172.21.15.26:0/3196033209 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fed04001c80 con 0x7fed1c132850 2024-09-16T14:42:33.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:33.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 4 set_mon_vals no callback set 2024-09-16T14:42:33.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:33.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed1959a640 10 monclient: dump: 2024-09-16T14:42:33.335 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.335 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.335 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.335 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.335 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.335 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.335 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.335 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.335 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.335 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient: shutdown 2024-09-16T14:42:33.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:33.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 1 -- 172.21.15.26:0/3196033209 >> v1:172.21.15.26:6790/0 conn(0x7fed1c132850 legacy=0x7fed1c132c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 1 -- 172.21.15.26:0/3196033209 shutdown_connections 2024-09-16T14:42:33.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 1 -- 172.21.15.26:0/3196033209 >> 172.21.15.26:0/3196033209 conn(0x7fed1c0a03d0 msgr2=0x7fed1c0a07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:33.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 1 -- 172.21.15.26:0/3196033209 shutdown_connections 2024-09-16T14:42:33.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 1 -- 172.21.15.26:0/3196033209 wait complete. 2024-09-16T14:42:33.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient: build_initial_monmap 2024-09-16T14:42:33.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient: monmap: 2024-09-16T14:42:33.336 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:33.336 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:33.336 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:33.336 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:33.336 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:33.336 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:33.336 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:33.337 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:33.337 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:33.337 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 1 Processor -- start 2024-09-16T14:42:33.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 1 -- start start 2024-09-16T14:42:33.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient: init 2024-09-16T14:42:33.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:33.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:33.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient(hunting): picked mon.noname-a con 0x7fed1c071530 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:33.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient(hunting): picked mon.noname-c con 0x7fed1c132850 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:33.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient(hunting): picked mon.noname-b con 0x7fed1c138f40 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:33.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed1c138ae0 con 0x7fed1c071530 2024-09-16T14:42:33.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed1c0720a0 con 0x7fed1c138f40 2024-09-16T14:42:33.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed1c119f20 con 0x7fed1c132850 2024-09-16T14:42:33.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:33.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.324+0000 7fed20942640 10 monclient(hunting): authenticate will time out at 977.447976s 2024-09-16T14:42:33.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed19d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fed1c132850 0x7fed1c1cc300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48536/0 (socket says 172.21.15.26:48536) 2024-09-16T14:42:33.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed19d9b640 1 -- 172.21.15.26:0/2884572449 learned_addr learned my addr 172.21.15.26:0/2884572449 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:33.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698881036 0 0) 0x7fed1c119f20 con 0x7fed1c132850 2024-09-16T14:42:33.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): my global_id is 4304 2024-09-16T14:42:33.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecfc003530 con 0x7fed1c132850 2024-09-16T14:42:33.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651675923 0 0) 0x7fed1c138ae0 con 0x7fed1c071530 2024-09-16T14:42:33.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): my global_id is 4317 2024-09-16T14:42:33.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed1c119f20 con 0x7fed1c071530 2024-09-16T14:42:33.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3090277828 0 0) 0x7fecfc003530 con 0x7fed1c132850 2024-09-16T14:42:33.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed1c138ae0 con 0x7fed1c132850 2024-09-16T14:42:33.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fed0c001880 con 0x7fed1c132850 2024-09-16T14:42:33.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:33.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:33.340 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.340 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.340 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.340 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.340 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.340 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.340 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.340 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.340 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.340 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:33.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815238977 0 0) 0x7fed1c0720a0 con 0x7fed1c138f40 2024-09-16T14:42:33.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): my global_id is 4324 2024-09-16T14:42:33.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecfc003530 con 0x7fed1c138f40 2024-09-16T14:42:33.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2574093281 0 0) 0x7fed1c119f20 con 0x7fed1c071530 2024-09-16T14:42:33.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed1c0720a0 con 0x7fed1c071530 2024-09-16T14:42:33.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fed040015c0 con 0x7fed1c071530 2024-09-16T14:42:33.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:33.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:33.341 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.341 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.342 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.342 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.342 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.342 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.342 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.342 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.342 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.342 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 494618080 0 0) 0x7fed1c138ae0 con 0x7fed1c132850 2024-09-16T14:42:33.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 >> v1:172.21.15.189:6789/0 conn(0x7fed1c138f40 legacy=0x7fed1c1cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 >> v1:172.21.15.26:6789/0 conn(0x7fed1c071530 legacy=0x7fed1c1c7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:33.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 monclient: found mon.c 2024-09-16T14:42:33.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:33.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:33.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1c137c00 con 0x7fed1c132850 2024-09-16T14:42:33.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:33.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:33.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:33.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed20942640 5 monclient: authenticate success, global_id 4304 2024-09-16T14:42:33.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed20942640 10 monclient: _renew_subs 2024-09-16T14:42:33.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed20942640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:33.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed20942640 1 -- 172.21.15.26:0/2884572449 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1c137030 con 0x7fed1c132850 2024-09-16T14:42:33.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed0c003620 con 0x7fed1c132850 2024-09-16T14:42:33.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:33.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fed0c003930 con 0x7fed1c132850 2024-09-16T14:42:33.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:33.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 10 monclient: dump: 2024-09-16T14:42:33.344 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.344 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.344 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.344 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.344 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.344 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.344 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.344 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.344 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.344 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed20942640 10 monclient: _renew_subs 2024-09-16T14:42:33.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed18d99640 4 set_mon_vals no callback set 2024-09-16T14:42:33.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed20942640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:33.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed20942640 1 -- 172.21.15.26:0/2884572449 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1c11ac40 con 0x7fed1c132850 2024-09-16T14:42:33.346 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T14:42:33.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed20942640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed20942640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed20942640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:33.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed20942640 1 -- 172.21.15.26:0/2884572449 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1c09c180 con 0x7fed1c132850 2024-09-16T14:42:33.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fed0c003b10 con 0x7fed1c132850 2024-09-16T14:42:33.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7fed0c042fa0 con 0x7fed1c132850 2024-09-16T14:42:33.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.328+0000 7fed1a59c640 10 monclient: get_auth_request con 0x7fecfc036490 auth_method 0 2024-09-16T14:42:33.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.332+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fed0c0142c0 con 0x7fed1c132850 2024-09-16T14:42:33.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.332+0000 7fed12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.332+0000 7fed12ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:33.373 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T14:42:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 10 monclient: get_monmap_and_config 2024-09-16T14:42:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 10 monclient: build_initial_monmap 2024-09-16T14:42:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 10 monclient: monmap: 2024-09-16T14:42:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:33.409698+0000 2024-09-16T14:42:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:33.409698+0000 2024-09-16T14:42:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:33.410 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:33.410 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.410 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T14:42:33.410 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T14:42:33.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 1 Processor -- start 2024-09-16T14:42:33.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 1 -- start start 2024-09-16T14:42:33.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 10 monclient: init 2024-09-16T14:42:33.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:33.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:33.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 10 monclient(hunting): picked mon.noname-c con 0x7f06600a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:33.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 10 monclient(hunting): picked mon.noname-b con 0x7f0660099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:33.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 10 monclient(hunting): picked mon.noname-a con 0x7f06600a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:33.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0660098ea0 con 0x7f06600a26f0 2024-09-16T14:42:33.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06600997b0 con 0x7f06600a1c00 2024-09-16T14:42:33.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0660099460 con 0x7f0660099b60 2024-09-16T14:42:33.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:33.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 10 monclient(hunting): authenticate will time out at 977.529450s 2024-09-16T14:42:33.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f0664f40640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f06600a26f0 0x7f066013d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57510/0 (socket says 172.21.15.26:57510) 2024-09-16T14:42:33.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f0664f40640 1 -- 172.21.15.26:0/3069741515 learned_addr learned my addr 172.21.15.26:0/3069741515 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:33.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 1 -- 172.21.15.26:0/3069741515 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275102859 0 0) 0x7f0660099460 con 0x7f0660099b60 2024-09-16T14:42:33.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient(hunting): my global_id is 4307 2024-09-16T14:42:33.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 1 -- 172.21.15.26:0/3069741515 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0648003530 con 0x7f0660099b60 2024-09-16T14:42:33.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 1 -- 172.21.15.26:0/3069741515 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517210917 0 0) 0x7f0660098ea0 con 0x7f06600a26f0 2024-09-16T14:42:33.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient(hunting): my global_id is 4320 2024-09-16T14:42:33.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 1 -- 172.21.15.26:0/3069741515 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0660099460 con 0x7f06600a26f0 2024-09-16T14:42:33.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 1 -- 172.21.15.26:0/3069741515 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454879122 0 0) 0x7f06600997b0 con 0x7f06600a1c00 2024-09-16T14:42:33.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient(hunting): my global_id is 4327 2024-09-16T14:42:33.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 1 -- 172.21.15.26:0/3069741515 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0660098ea0 con 0x7f06600a1c00 2024-09-16T14:42:33.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 1 -- 172.21.15.26:0/3069741515 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2271804974 0 0) 0x7f0648003530 con 0x7f0660099b60 2024-09-16T14:42:33.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 1 -- 172.21.15.26:0/3069741515 >> v1:172.21.15.189:6789/0 conn(0x7f06600a1c00 legacy=0x7f06600a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 1 -- 172.21.15.26:0/3069741515 >> v1:172.21.15.26:6789/0 conn(0x7f06600a26f0 legacy=0x7f066013d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:33.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 1 monclient: found mon.noname-b 2024-09-16T14:42:33.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:33.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:33.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 1 -- 172.21.15.26:0/3069741515 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06600a34b0 con 0x7f0660099b60 2024-09-16T14:42:33.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient: _finish_auth 0 2024-09-16T14:42:33.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:33.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:33.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 1 -- 172.21.15.26:0/3069741515 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f06540015a0 con 0x7f0660099b60 2024-09-16T14:42:33.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:33.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient: dump: 2024-09-16T14:42:33.419 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.419 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.419 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.419 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.419 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.420 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.420 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.420 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.420 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.420 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 5 monclient: authenticate success, global_id 4307 2024-09-16T14:42:33.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:33.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 1 -- 172.21.15.26:0/3069741515 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06540017c0 con 0x7f0660099b60 2024-09-16T14:42:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 4 set_mon_vals no callback set 2024-09-16T14:42:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 1 -- 172.21.15.26:0/3069741515 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0654001b90 con 0x7f0660099b60 2024-09-16T14:42:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f065effd640 10 monclient: dump: 2024-09-16T14:42:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.422 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 10 monclient: shutdown 2024-09-16T14:42:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 1 -- 172.21.15.26:0/3069741515 >> v1:172.21.15.26:6790/0 conn(0x7f0660099b60 legacy=0x7f0660139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 1 -- 172.21.15.26:0/3069741515 shutdown_connections 2024-09-16T14:42:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 1 -- 172.21.15.26:0/3069741515 >> 172.21.15.26:0/3069741515 conn(0x7f06600a3d00 msgr2=0x7f06600a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 1 -- 172.21.15.26:0/3069741515 shutdown_connections 2024-09-16T14:42:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.408+0000 7f06669a3640 1 -- 172.21.15.26:0/3069741515 wait complete. 2024-09-16T14:42:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 10 monclient: build_initial_monmap 2024-09-16T14:42:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 10 monclient: monmap: 2024-09-16T14:42:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:33.423 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 1 Processor -- start 2024-09-16T14:42:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 1 -- start start 2024-09-16T14:42:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 10 monclient: init 2024-09-16T14:42:33.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:33.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:42:33.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 10 monclient(hunting): picked mon.noname-a con 0x7f0660099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:33.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 10 monclient(hunting): picked mon.noname-b con 0x7f06600a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:33.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 10 monclient(hunting): picked mon.noname-c con 0x7f06600a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:33.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066009e3f0 con 0x7f0660099b60 2024-09-16T14:42:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066011e680 con 0x7f06600a1c00 2024-09-16T14:42:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f0664f40640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f06600a26f0 0x7f06601d1980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48546/0 (socket says 172.21.15.26:48546) 2024-09-16T14:42:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0660099b60 0x7f0660138110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57514/0 (socket says 172.21.15.26:57514) 2024-09-16T14:42:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f0664f40640 1 -- 172.21.15.26:0/2068912580 learned_addr learned my addr 172.21.15.26:0/2068912580 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 1 -- 172.21.15.26:0/2068912580 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0660138820 con 0x7f06600a26f0 2024-09-16T14:42:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 10 monclient(hunting): authenticate will time out at 977.533401s 2024-09-16T14:42:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602452394 0 0) 0x7f066009e3f0 con 0x7f0660099b60 2024-09-16T14:42:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): my global_id is 4323 2024-09-16T14:42:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f064c003530 con 0x7f0660099b60 2024-09-16T14:42:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612588695 0 0) 0x7f0660138820 con 0x7f06600a26f0 2024-09-16T14:42:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): my global_id is 4310 2024-09-16T14:42:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f066009e3f0 con 0x7f06600a26f0 2024-09-16T14:42:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2970755164 0 0) 0x7f064c003530 con 0x7f0660099b60 2024-09-16T14:42:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0660138820 con 0x7f0660099b60 2024-09-16T14:42:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803233813 0 0) 0x7f06600a34b0 con 0x7f06600a1c00 2024-09-16T14:42:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): my global_id is 4330 2024-09-16T14:42:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f064c003530 con 0x7f06600a1c00 2024-09-16T14:42:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f06500018e0 con 0x7f0660099b60 2024-09-16T14:42:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): dump: 2024-09-16T14:42:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.431 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3609231160 0 0) 0x7f066009e3f0 con 0x7f06600a26f0 2024-09-16T14:42:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06600a34b0 con 0x7f06600a26f0 2024-09-16T14:42:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0658001ce0 con 0x7f06600a26f0 2024-09-16T14:42:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient(hunting): dump: 2024-09-16T14:42:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.432 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 329540392 0 0) 0x7f0660138820 con 0x7f0660099b60 2024-09-16T14:42:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 >> v1:172.21.15.26:6790/0 conn(0x7f06600a26f0 legacy=0x7f06601d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 >> v1:172.21.15.189:6789/0 conn(0x7f06600a1c00 legacy=0x7f06601ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:42:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 monclient: found mon.a 2024-09-16T14:42:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066011e060 con 0x7f0660099b60 2024-09-16T14:42:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient: _finish_auth 0 2024-09-16T14:42:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:42:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 5 monclient: authenticate success, global_id 4323 2024-09-16T14:42:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 10 monclient: _renew_subs 2024-09-16T14:42:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 1 -- 172.21.15.26:0/2068912580 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066009f0a0 con 0x7f0660099b60 2024-09-16T14:42:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 10 monclient: _renew_subs 2024-09-16T14:42:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 1 -- 172.21.15.26:0/2068912580 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f066011f790 con 0x7f0660099b60 2024-09-16T14:42:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0650003620 con 0x7f0660099b60 2024-09-16T14:42:33.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:33.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f06500039b0 con 0x7f0660099b60 2024-09-16T14:42:33.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:33.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065cff9640 10 monclient: dump: 2024-09-16T14:42:33.434 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.434 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.434 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.434 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.434 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.434 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.434 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.435 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.435 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.435 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f06669a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.412+0000 7f065e7fc640 4 set_mon_vals no callback set 2024-09-16T14:42:33.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.416+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0650003b10 con 0x7f0660099b60 2024-09-16T14:42:33.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.416+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f06500430c0 con 0x7f0660099b60 2024-09-16T14:42:33.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.416+0000 7f06669a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.416+0000 7f06669a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.416+0000 7f06669a3640 1 -- 172.21.15.26:0/2068912580 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06600a2d70 con 0x7f0660099b60 2024-09-16T14:42:33.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.416+0000 7f065f7fe640 10 monclient: get_auth_request con 0x7f064c036400 auth_method 0 2024-09-16T14:42:33.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.424+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0650003e40 con 0x7f0660099b60 2024-09-16T14:42:33.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.424+0000 7f065cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.424+0000 7f065cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.456+0000 7f603214f640 10 monclient: get_monmap_and_config 2024-09-16T14:42:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.456+0000 7f603214f640 10 monclient: build_initial_monmap 2024-09-16T14:42:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.456+0000 7f603214f640 10 monclient: monmap: 2024-09-16T14:42:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:33.461376+0000 2024-09-16T14:42:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:33.461376+0000 2024-09-16T14:42:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:33.462 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:33.462 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:33.462 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 1 Processor -- start 2024-09-16T14:42:33.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 1 -- start start 2024-09-16T14:42:33.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 10 monclient: init 2024-09-16T14:42:33.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:33.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 10 monclient(hunting): picked mon.noname-c con 0x7f602c0a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 10 monclient(hunting): picked mon.noname-b con 0x7f602c099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602affd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f602c099b60 0x7f602c139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48560/0 (socket says 172.21.15.26:48560) 2024-09-16T14:42:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602bfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f602c0a26f0 0x7f602c13d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57516/0 (socket says 172.21.15.26:57516) 2024-09-16T14:42:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 10 monclient(hunting): picked mon.noname-a con 0x7f602c0a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c098ea0 con 0x7f602c0a26f0 2024-09-16T14:42:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c0997b0 con 0x7f602c0a1c00 2024-09-16T14:42:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c099460 con 0x7f602c099b60 2024-09-16T14:42:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 10 monclient(hunting): authenticate will time out at 977.581493s 2024-09-16T14:42:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602affd640 1 -- 172.21.15.26:0/917797143 learned_addr learned my addr 172.21.15.26:0/917797143 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 1 -- 172.21.15.26:0/917797143 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783910779 0 0) 0x7f602c098ea0 con 0x7f602c0a26f0 2024-09-16T14:42:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient(hunting): my global_id is 4326 2024-09-16T14:42:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 1 -- 172.21.15.26:0/917797143 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6014003530 con 0x7f602c0a26f0 2024-09-16T14:42:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 1 -- 172.21.15.26:0/917797143 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990491527 0 0) 0x7f602c0997b0 con 0x7f602c0a1c00 2024-09-16T14:42:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient(hunting): my global_id is 4333 2024-09-16T14:42:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 1 -- 172.21.15.26:0/917797143 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f602c098ea0 con 0x7f602c0a1c00 2024-09-16T14:42:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 1 -- 172.21.15.26:0/917797143 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942336559 0 0) 0x7f602c099460 con 0x7f602c099b60 2024-09-16T14:42:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient(hunting): my global_id is 4313 2024-09-16T14:42:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 1 -- 172.21.15.26:0/917797143 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f602c0997b0 con 0x7f602c099b60 2024-09-16T14:42:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 1 -- 172.21.15.26:0/917797143 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1017798358 0 0) 0x7f6014003530 con 0x7f602c0a26f0 2024-09-16T14:42:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 1 -- 172.21.15.26:0/917797143 >> v1:172.21.15.26:6790/0 conn(0x7f602c099b60 legacy=0x7f602c139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 1 -- 172.21.15.26:0/917797143 >> v1:172.21.15.189:6789/0 conn(0x7f602c0a1c00 legacy=0x7f602c0a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:42:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 1 monclient: found mon.noname-a 2024-09-16T14:42:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 1 -- 172.21.15.26:0/917797143 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c0a34b0 con 0x7f602c0a26f0 2024-09-16T14:42:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:42:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:42:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 1 -- 172.21.15.26:0/917797143 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f60240019a0 con 0x7f602c0a26f0 2024-09-16T14:42:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 5 monclient: authenticate success, global_id 4326 2024-09-16T14:42:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient: dump: 2024-09-16T14:42:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.473 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 1 -- 172.21.15.26:0/917797143 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6024001c30 con 0x7f602c0a26f0 2024-09-16T14:42:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 1 -- 172.21.15.26:0/917797143 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6024001fc0 con 0x7f602c0a26f0 2024-09-16T14:42:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 4 set_mon_vals no callback set 2024-09-16T14:42:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 10 monclient: shutdown 2024-09-16T14:42:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 1 -- 172.21.15.26:0/917797143 >> v1:172.21.15.26:6789/0 conn(0x7f602c0a26f0 legacy=0x7f602c13d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f602a7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-16T14:42:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 1 -- 172.21.15.26:0/917797143 shutdown_connections 2024-09-16T14:42:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 1 -- 172.21.15.26:0/917797143 >> 172.21.15.26:0/917797143 conn(0x7f602c0a3d00 msgr2=0x7f602c0a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 1 -- 172.21.15.26:0/917797143 shutdown_connections 2024-09-16T14:42:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.460+0000 7f603214f640 1 -- 172.21.15.26:0/917797143 wait complete. 2024-09-16T14:42:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f603214f640 10 monclient: build_initial_monmap 2024-09-16T14:42:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f603214f640 10 monclient: monmap: 2024-09-16T14:42:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:33.475 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f603214f640 1 Processor -- start 2024-09-16T14:42:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f603214f640 1 -- start start 2024-09-16T14:42:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f603214f640 10 monclient: init 2024-09-16T14:42:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f603214f640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f603214f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f603214f640 10 monclient(hunting): picked mon.noname-a con 0x7f602c099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f603214f640 10 monclient(hunting): picked mon.noname-c con 0x7f602c0a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f602b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f602c099b60 0x7f602c13c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57528/0 (socket says 172.21.15.26:57528) 2024-09-16T14:42:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f602b7fe640 1 -- 172.21.15.26:0/934123906 learned_addr learned my addr 172.21.15.26:0/934123906 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f603214f640 10 monclient(hunting): picked mon.noname-b con 0x7f602c0a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f603214f640 1 -- 172.21.15.26:0/934123906 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c11ecf0 con 0x7f602c099b60 2024-09-16T14:42:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f603214f640 1 -- 172.21.15.26:0/934123906 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c09f0d0 con 0x7f602c0a26f0 2024-09-16T14:42:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f603214f640 1 -- 172.21.15.26:0/934123906 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c09eb60 con 0x7f602c0a1c00 2024-09-16T14:42:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f603214f640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428964417 0 0) 0x7f602c11ecf0 con 0x7f602c099b60 2024-09-16T14:42:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f603214f640 10 monclient(hunting): authenticate will time out at 977.586745s 2024-09-16T14:42:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): my global_id is 4329 2024-09-16T14:42:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6018003530 con 0x7f602c099b60 2024-09-16T14:42:33.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754145950 0 0) 0x7f602c09eb60 con 0x7f602c0a1c00 2024-09-16T14:42:33.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): my global_id is 4316 2024-09-16T14:42:33.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f602c11ecf0 con 0x7f602c0a1c00 2024-09-16T14:42:33.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780559243 0 0) 0x7f602c09f0d0 con 0x7f602c0a26f0 2024-09-16T14:42:33.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): my global_id is 4336 2024-09-16T14:42:33.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f602c09eb60 con 0x7f602c0a26f0 2024-09-16T14:42:33.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 397244572 0 0) 0x7f602c11ecf0 con 0x7f602c0a1c00 2024-09-16T14:42:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f602c09f0d0 con 0x7f602c0a1c00 2024-09-16T14:42:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3821578327 0 0) 0x7f6018003530 con 0x7f602c099b60 2024-09-16T14:42:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f602c11ecf0 con 0x7f602c099b60 2024-09-16T14:42:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f601c0018c0 con 0x7f602c099b60 2024-09-16T14:42:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): dump: 2024-09-16T14:42:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.479 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6020001900 con 0x7f602c0a1c00 2024-09-16T14:42:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient(hunting): dump: 2024-09-16T14:42:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.480 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.480 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.480 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.480 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.480 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2758916376 0 0) 0x7f602c11ecf0 con 0x7f602c099b60 2024-09-16T14:42:33.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 >> v1:172.21.15.26:6790/0 conn(0x7f602c0a1c00 legacy=0x7f602c13c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 >> v1:172.21.15.189:6789/0 conn(0x7f602c0a26f0 legacy=0x7f602c1d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient: _finish_hunting 0 2024-09-16T14:42:33.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 monclient: found mon.a 2024-09-16T14:42:33.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:33.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c09e3f0 con 0x7f602c099b60 2024-09-16T14:42:33.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient: _finish_auth 0 2024-09-16T14:42:33.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient: _check_auth_tickets 2024-09-16T14:42:33.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:33.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f603214f640 5 monclient: authenticate success, global_id 4329 2024-09-16T14:42:33.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f601c003600 con 0x7f602c099b60 2024-09-16T14:42:33.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:33.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f601c003990 con 0x7f602c099b60 2024-09-16T14:42:33.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:33.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6013fff640 10 monclient: dump: 2024-09-16T14:42:33.482 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.483 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.464+0000 7f6029ffb640 4 set_mon_vals no callback set 2024-09-16T14:42:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.468+0000 7f603214f640 10 monclient: _renew_subs 2024-09-16T14:42:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.468+0000 7f603214f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.468+0000 7f603214f640 1 -- 172.21.15.26:0/934123906 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f602c11ea10 con 0x7f602c099b60 2024-09-16T14:42:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.468+0000 7f603214f640 10 monclient: _renew_subs 2024-09-16T14:42:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.468+0000 7f603214f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.468+0000 7f603214f640 1 -- 172.21.15.26:0/934123906 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f602c11d850 con 0x7f602c099b60 2024-09-16T14:42:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.468+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f601c003af0 con 0x7f602c099b60 2024-09-16T14:42:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.468+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f601c043130 con 0x7f602c099b60 2024-09-16T14:42:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.468+0000 7f602affd640 10 monclient: get_auth_request con 0x7f6018036490 auth_method 0 2024-09-16T14:42:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.472+0000 7f6011ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.472+0000 7f6011ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.472+0000 7f6011ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.472+0000 7f6011ffb640 1 -- 172.21.15.26:0/934123906 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602c0a1370 con 0x7f602c099b60 2024-09-16T14:42:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.480+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f601c014670 con 0x7f602c099b60 2024-09-16T14:42:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.480+0000 7f6013fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.480+0000 7f6013fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:33.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 10 monclient: get_monmap_and_config 2024-09-16T14:42:33.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 10 monclient: build_initial_monmap 2024-09-16T14:42:33.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 10 monclient: monmap: 2024-09-16T14:42:33.510 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:33.510 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:33.510697+0000 2024-09-16T14:42:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:33.510697+0000 2024-09-16T14:42:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:33.511 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 1 Processor -- start 2024-09-16T14:42:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 1 -- start start 2024-09-16T14:42:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 10 monclient: init 2024-09-16T14:42:33.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:33.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:42:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 10 monclient(hunting): picked mon.noname-b con 0x7f62640a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 10 monclient(hunting): picked mon.noname-a con 0x7f6264099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f6268c84640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f62640a1c00 0x7f62640a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48588/0 (socket says 172.21.15.26:48588) 2024-09-16T14:42:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f6268c84640 1 -- 172.21.15.26:0/2645147893 learned_addr learned my addr 172.21.15.26:0/2645147893 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f6263fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6264099b60 0x7f6264139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57532/0 (socket says 172.21.15.26:57532) 2024-09-16T14:42:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 10 monclient(hunting): picked mon.noname-c con 0x7f62640a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 1 -- 172.21.15.26:0/2645147893 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6264098ea0 con 0x7f6264099b60 2024-09-16T14:42:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 1 -- 172.21.15.26:0/2645147893 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62640997b0 con 0x7f62640a26f0 2024-09-16T14:42:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 1 -- 172.21.15.26:0/2645147893 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6264099460 con 0x7f62640a1c00 2024-09-16T14:42:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 10 monclient(hunting): authenticate will time out at 977.631180s 2024-09-16T14:42:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 1 -- 172.21.15.26:0/2645147893 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802782603 0 0) 0x7f6264098ea0 con 0x7f6264099b60 2024-09-16T14:42:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 10 monclient(hunting): my global_id is 4332 2024-09-16T14:42:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 1 -- 172.21.15.26:0/2645147893 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6254003530 con 0x7f6264099b60 2024-09-16T14:42:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 1 -- 172.21.15.26:0/2645147893 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4248946183 0 0) 0x7f6254003530 con 0x7f6264099b60 2024-09-16T14:42:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 1 -- 172.21.15.26:0/2645147893 >> v1:172.21.15.26:6790/0 conn(0x7f62640a1c00 legacy=0x7f62640a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 1 -- 172.21.15.26:0/2645147893 >> v1:172.21.15.189:6789/0 conn(0x7f62640a26f0 legacy=0x7f626413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 10 monclient: _finish_hunting 0 2024-09-16T14:42:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 1 monclient: found mon.noname-a 2024-09-16T14:42:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 1 -- 172.21.15.26:0/2645147893 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62640a34b0 con 0x7f6264099b60 2024-09-16T14:42:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 10 monclient: _finish_auth 0 2024-09-16T14:42:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 10 monclient: _check_auth_tickets 2024-09-16T14:42:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 1 -- 172.21.15.26:0/2645147893 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6250001600 con 0x7f6264099b60 2024-09-16T14:42:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 10 monclient: dump: 2024-09-16T14:42:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.518 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 5 monclient: authenticate success, global_id 4332 2024-09-16T14:42:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 1 -- 172.21.15.26:0/2645147893 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6250001920 con 0x7f6264099b60 2024-09-16T14:42:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 1 -- 172.21.15.26:0/2645147893 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6250001600 con 0x7f6264099b60 2024-09-16T14:42:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f62637fe640 10 monclient: dump: 2024-09-16T14:42:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.519 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.508+0000 7f626aee8640 4 set_mon_vals no callback set 2024-09-16T14:42:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 10 monclient: shutdown 2024-09-16T14:42:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 1 -- 172.21.15.26:0/2645147893 >> v1:172.21.15.26:6789/0 conn(0x7f6264099b60 legacy=0x7f6264139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 1 -- 172.21.15.26:0/2645147893 shutdown_connections 2024-09-16T14:42:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 1 -- 172.21.15.26:0/2645147893 >> 172.21.15.26:0/2645147893 conn(0x7f62640a3d00 msgr2=0x7f62640a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 1 -- 172.21.15.26:0/2645147893 shutdown_connections 2024-09-16T14:42:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 1 -- 172.21.15.26:0/2645147893 wait complete. 2024-09-16T14:42:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 10 monclient: build_initial_monmap 2024-09-16T14:42:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 10 monclient: monmap: 2024-09-16T14:42:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:33.521 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 1 Processor -- start 2024-09-16T14:42:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 1 -- start start 2024-09-16T14:42:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 10 monclient: init 2024-09-16T14:42:33.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:33.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:42:33.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 10 monclient(hunting): picked mon.noname-c con 0x7f6264099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:33.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 10 monclient(hunting): picked mon.noname-a con 0x7f62640a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:33.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f6263fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f62640a1c00 0x7f626413a700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57538/0 (socket says 172.21.15.26:57538) 2024-09-16T14:42:33.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f6263fff640 1 -- 172.21.15.26:0/3601249150 learned_addr learned my addr 172.21.15.26:0/3601249150 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:33.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 10 monclient(hunting): picked mon.noname-b con 0x7f62640a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:33.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 1 -- 172.21.15.26:0/3601249150 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626409e3f0 con 0x7f62640a1c00 2024-09-16T14:42:33.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 1 -- 172.21.15.26:0/3601249150 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626413df20 con 0x7f62640a26f0 2024-09-16T14:42:33.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 1 -- 172.21.15.26:0/3601249150 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626413ae10 con 0x7f6264099b60 2024-09-16T14:42:33.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891402992 0 0) 0x7f626409e3f0 con 0x7f62640a1c00 2024-09-16T14:42:33.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:33.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f626aee8640 10 monclient(hunting): authenticate will time out at 977.635835s 2024-09-16T14:42:33.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f62617fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f62617fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f62617fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f62617fa640 10 monclient(hunting): my global_id is 4335 2024-09-16T14:42:33.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6248003530 con 0x7f62640a1c00 2024-09-16T14:42:33.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.512+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074504470 0 0) 0x7f626413ae10 con 0x7f6264099b60 2024-09-16T14:42:33.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient(hunting): my global_id is 4322 2024-09-16T14:42:33.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f626409e3f0 con 0x7f6264099b60 2024-09-16T14:42:33.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419382303 0 0) 0x7f626413df20 con 0x7f62640a26f0 2024-09-16T14:42:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient(hunting): my global_id is 4342 2024-09-16T14:42:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f626413ae10 con 0x7f62640a26f0 2024-09-16T14:42:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 730027715 0 0) 0x7f6248003530 con 0x7f62640a1c00 2024-09-16T14:42:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f626413df20 con 0x7f62640a1c00 2024-09-16T14:42:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6250001600 con 0x7f62640a1c00 2024-09-16T14:42:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient(hunting): dump: 2024-09-16T14:42:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.528 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4184408197 0 0) 0x7f626409e3f0 con 0x7f6264099b60 2024-09-16T14:42:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6248003530 con 0x7f6264099b60 2024-09-16T14:42:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6258001840 con 0x7f6264099b60 2024-09-16T14:42:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient(hunting): dump: 2024-09-16T14:42:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.529 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 610792363 0 0) 0x7f626413df20 con 0x7f62640a1c00 2024-09-16T14:42:33.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 >> v1:172.21.15.26:6790/0 conn(0x7f6264099b60 legacy=0x7f6264139ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 >> v1:172.21.15.189:6789/0 conn(0x7f62640a26f0 legacy=0x7f62641d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient: _finish_hunting 0 2024-09-16T14:42:33.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 monclient: found mon.a 2024-09-16T14:42:33.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:33.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626413c4a0 con 0x7f62640a1c00 2024-09-16T14:42:33.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient: _finish_auth 0 2024-09-16T14:42:33.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient: _check_auth_tickets 2024-09-16T14:42:33.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:33.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f626aee8640 5 monclient: authenticate success, global_id 4335 2024-09-16T14:42:33.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f626aee8640 10 monclient: _renew_subs 2024-09-16T14:42:33.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f626aee8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6250001c80 con 0x7f62640a1c00 2024-09-16T14:42:33.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f626aee8640 1 -- 172.21.15.26:0/3601249150 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626409f2c0 con 0x7f62640a1c00 2024-09-16T14:42:33.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:33.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6250004fa0 con 0x7f62640a1c00 2024-09-16T14:42:33.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:33.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 10 monclient: dump: 2024-09-16T14:42:33.531 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.531 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.531 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.531 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.531 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.532 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.532 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.532 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.532 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.532 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f6262ffd640 4 set_mon_vals no callback set 2024-09-16T14:42:33.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f625000fa10 con 0x7f62640a1c00 2024-09-16T14:42:33.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f626aee8640 10 monclient: _renew_subs 2024-09-16T14:42:33.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f626aee8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f626aee8640 1 -- 172.21.15.26:0/3601249150 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f626409edd0 con 0x7f62640a1c00 2024-09-16T14:42:33.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f6268c84640 10 monclient: get_auth_request con 0x7f62480361f0 auth_method 0 2024-09-16T14:42:33.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f62500110e0 con 0x7f62640a1c00 2024-09-16T14:42:33.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f624effd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f624effd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f624effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.516+0000 7f624effd640 1 -- 172.21.15.26:0/3601249150 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f623c005050 con 0x7f62640a1c00 2024-09-16T14:42:33.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.524+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6250010f10 con 0x7f62640a1c00 2024-09-16T14:42:33.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.524+0000 7f62617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.524+0000 7f62617fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.572+0000 7fed20942640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T14:42:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.572+0000 7fed20942640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T14:42:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.572+0000 7fed20942640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.572+0000 7fed20942640 1 -- 172.21.15.26:0/2884572449 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fed1c074390 con 0x7fed1c132850 2024-09-16T14:42:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.572+0000 7fed12ffd640 1 -- 172.21.15.26:0/2884572449 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 847495621) 0x7fed0c0140e0 con 0x7fed1c132850 2024-09-16T14:42:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.572+0000 7fed12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T14:42:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.572+0000 7fed12ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:33.576 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-09-16T14:42:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fed20942640 1 -- 172.21.15.26:0/2884572449 >> v1:172.21.15.189:6812/3196161582 conn(0x7fecfc036490 legacy=0x7fecfc038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fed20942640 10 monclient: shutdown 2024-09-16T14:42:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fed20942640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fed20942640 1 -- 172.21.15.26:0/2884572449 >> v1:172.21.15.26:6790/0 conn(0x7fed1c132850 legacy=0x7fed1c1cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fed20942640 1 -- 172.21.15.26:0/2884572449 shutdown_connections 2024-09-16T14:42:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fed20942640 1 -- 172.21.15.26:0/2884572449 >> 172.21.15.26:0/2884572449 conn(0x7fed1c0a03d0 msgr2=0x7fed1c0a2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fed20942640 1 -- 172.21.15.26:0/2884572449 shutdown_connections 2024-09-16T14:42:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fed20942640 1 -- 172.21.15.26:0/2884572449 wait complete. 2024-09-16T14:42:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fc8e1c1a640 10 monclient: get_monmap_and_config 2024-09-16T14:42:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fc8e1c1a640 10 monclient: build_initial_monmap 2024-09-16T14:42:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fc8e1c1a640 10 monclient: monmap: 2024-09-16T14:42:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:33.579727+0000 2024-09-16T14:42:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:33.579727+0000 2024-09-16T14:42:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:33.580 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fc8e1c1a640 1 Processor -- start 2024-09-16T14:42:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fc8e1c1a640 1 -- start start 2024-09-16T14:42:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fc8e1c1a640 10 monclient: init 2024-09-16T14:42:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7f71c8712640 10 monclient: get_monmap_and_config 2024-09-16T14:42:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7f71c8712640 10 monclient: build_initial_monmap 2024-09-16T14:42:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7f71c8712640 10 monclient: monmap: 2024-09-16T14:42:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:33.580859+0000 2024-09-16T14:42:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:33.580859+0000 2024-09-16T14:42:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:33.582 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fc8e1c1a640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fc8e1c1a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fc8e1c1a640 10 monclient(hunting): picked mon.noname-c con 0x7fc8dc0a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fc8e1c1a640 10 monclient(hunting): picked mon.noname-b con 0x7fc8dc099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.576+0000 7fc8e1c1a640 10 monclient(hunting): picked mon.noname-a con 0x7fc8dc0a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8e1c1a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc098ea0 con 0x7fc8dc0a26f0 2024-09-16T14:42:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8daffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc8dc099b60 0x7fc8dc139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48608/0 (socket says 172.21.15.26:48608) 2024-09-16T14:42:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8daffd640 1 -- 172.21.15.26:0/4101130715 learned_addr learned my addr 172.21.15.26:0/4101130715 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/4101130715 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc0997b0 con 0x7fc8dc0a1c00 2024-09-16T14:42:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/4101130715 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc099460 con 0x7fc8dc099b60 2024-09-16T14:42:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8e1c1a640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8e1c1a640 10 monclient(hunting): authenticate will time out at 977.700135s 2024-09-16T14:42:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 1 Processor -- start 2024-09-16T14:42:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 1 -- start start 2024-09-16T14:42:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 10 monclient: init 2024-09-16T14:42:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:42:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 10 monclient(hunting): picked mon.noname-c con 0x7f71c00a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 10 monclient(hunting): picked mon.noname-a con 0x7f71c0099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 10 monclient(hunting): picked mon.noname-b con 0x7f71c00a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c0098ea0 con 0x7f71c0099b60 2024-09-16T14:42:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 1 -- 172.21.15.26:0/4101130715 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785737419 0 0) 0x7fc8dc098ea0 con 0x7fc8dc0a26f0 2024-09-16T14:42:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient(hunting): my global_id is 4338 2024-09-16T14:42:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 1 -- 172.21.15.26:0/4101130715 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8c4003530 con 0x7fc8dc0a26f0 2024-09-16T14:42:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 1 -- 172.21.15.26:0/4101130715 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170312550 0 0) 0x7fc8dc0997b0 con 0x7fc8dc0a1c00 2024-09-16T14:42:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient(hunting): my global_id is 4345 2024-09-16T14:42:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 1 -- 172.21.15.26:0/4101130715 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8dc098ea0 con 0x7fc8dc0a1c00 2024-09-16T14:42:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 1 -- 172.21.15.26:0/4101130715 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580026987 0 0) 0x7fc8dc099460 con 0x7fc8dc099b60 2024-09-16T14:42:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient(hunting): my global_id is 4325 2024-09-16T14:42:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c00997b0 con 0x7f71c00a1c00 2024-09-16T14:42:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c0099460 con 0x7f71c00a26f0 2024-09-16T14:42:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 10 monclient(hunting): authenticate will time out at 977.701449s 2024-09-16T14:42:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c6caf640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f71c00a26f0 0x7f71c013d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48622/0 (socket says 172.21.15.26:48622) 2024-09-16T14:42:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c6caf640 1 -- 172.21.15.26:0/1880229473 learned_addr learned my addr 172.21.15.26:0/1880229473 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 1 -- 172.21.15.26:0/4101130715 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8dc0997b0 con 0x7fc8dc099b60 2024-09-16T14:42:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 1 -- 172.21.15.26:0/1880229473 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714915314 0 0) 0x7f71c0099460 con 0x7f71c00a26f0 2024-09-16T14:42:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 10 monclient(hunting): my global_id is 4328 2024-09-16T14:42:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 1 -- 172.21.15.26:0/1880229473 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71ac003530 con 0x7f71c00a26f0 2024-09-16T14:42:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 1 -- 172.21.15.26:0/4101130715 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3202775204 0 0) 0x7fc8c4003530 con 0x7fc8dc0a26f0 2024-09-16T14:42:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 1 -- 172.21.15.26:0/4101130715 >> v1:172.21.15.26:6790/0 conn(0x7fc8dc099b60 legacy=0x7fc8dc139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 1 -- 172.21.15.26:0/4101130715 >> v1:172.21.15.189:6789/0 conn(0x7fc8dc0a1c00 legacy=0x7fc8dc0a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:42:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 1 monclient: found mon.noname-a 2024-09-16T14:42:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 1 -- 172.21.15.26:0/4101130715 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc0a34b0 con 0x7fc8dc0a26f0 2024-09-16T14:42:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient: _finish_auth 0 2024-09-16T14:42:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:42:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 1 -- 172.21.15.26:0/1880229473 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2621702234 0 0) 0x7f71ac003530 con 0x7f71c00a26f0 2024-09-16T14:42:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 1 -- 172.21.15.26:0/1880229473 >> v1:172.21.15.189:6789/0 conn(0x7f71c00a1c00 legacy=0x7f71c00a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 1 -- 172.21.15.26:0/1880229473 >> v1:172.21.15.26:6789/0 conn(0x7f71c0099b60 legacy=0x7f71c0139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:42:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 10 monclient: _finish_hunting 0 2024-09-16T14:42:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 1 monclient: found mon.noname-b 2024-09-16T14:42:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 1 -- 172.21.15.26:0/1880229473 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c00a34b0 con 0x7f71c00a26f0 2024-09-16T14:42:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 10 monclient: _finish_auth 0 2024-09-16T14:42:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 10 monclient: _check_auth_tickets 2024-09-16T14:42:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 1 -- 172.21.15.26:0/1880229473 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f71b0001620 con 0x7f71c00a26f0 2024-09-16T14:42:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 10 monclient: dump: 2024-09-16T14:42:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.593 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 1 -- 172.21.15.26:0/4101130715 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc8d40019a0 con 0x7fc8dc0a26f0 2024-09-16T14:42:33.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 5 monclient: authenticate success, global_id 4328 2024-09-16T14:42:33.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8e1c1a640 5 monclient: authenticate success, global_id 4338 2024-09-16T14:42:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8e1c1a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient: dump: 2024-09-16T14:42:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.596 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8e1c1a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 1 -- 172.21.15.26:0/4101130715 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8d4001c30 con 0x7fc8dc0a26f0 2024-09-16T14:42:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8e1c1a640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8e1c1a640 4 set_mon_vals no callback set 2024-09-16T14:42:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 1 -- 172.21.15.26:0/4101130715 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc8d4001fc0 con 0x7fc8dc0a26f0 2024-09-16T14:42:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8da7fc640 10 monclient: dump: 2024-09-16T14:42:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.597 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8e1c1a640 10 monclient: shutdown 2024-09-16T14:42:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8e1c1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/4101130715 >> v1:172.21.15.26:6789/0 conn(0x7fc8dc0a26f0 legacy=0x7fc8dc13d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 1 -- 172.21.15.26:0/1880229473 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71b0001840 con 0x7f71c00a26f0 2024-09-16T14:42:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 1 -- 172.21.15.26:0/1880229473 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f71b0001c50 con 0x7f71c00a26f0 2024-09-16T14:42:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c54ac640 10 monclient: dump: 2024-09-16T14:42:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.599 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7f71c8712640 4 set_mon_vals no callback set 2024-09-16T14:42:33.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/4101130715 shutdown_connections 2024-09-16T14:42:33.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.580+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/4101130715 >> 172.21.15.26:0/4101130715 conn(0x7fc8dc0a3d00 msgr2=0x7fc8dc0a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:33.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 10 monclient: shutdown 2024-09-16T14:42:33.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:33.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 1 -- 172.21.15.26:0/1880229473 >> v1:172.21.15.26:6790/0 conn(0x7f71c00a26f0 legacy=0x7f71c013d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 1 -- 172.21.15.26:0/1880229473 shutdown_connections 2024-09-16T14:42:33.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 1 -- 172.21.15.26:0/1880229473 >> 172.21.15.26:0/1880229473 conn(0x7f71c00a3d00 msgr2=0x7f71c00a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:33.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 1 -- 172.21.15.26:0/1880229473 shutdown_connections 2024-09-16T14:42:33.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 1 -- 172.21.15.26:0/1880229473 wait complete. 2024-09-16T14:42:33.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 10 monclient: build_initial_monmap 2024-09-16T14:42:33.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 10 monclient: monmap: 2024-09-16T14:42:33.604 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:33.604 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:33.604 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:33.604 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:33.604 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:33.604 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:33.605 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 1 Processor -- start 2024-09-16T14:42:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 1 -- start start 2024-09-16T14:42:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 10 monclient: init 2024-09-16T14:42:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:42:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 10 monclient(hunting): picked mon.noname-b con 0x7f71c0099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 10 monclient(hunting): picked mon.noname-a con 0x7f71c00a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 10 monclient(hunting): picked mon.noname-c con 0x7f71c00a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:33.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c006e620 con 0x7f71c00a1c00 2024-09-16T14:42:33.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c009e3f0 con 0x7f71c0099b60 2024-09-16T14:42:33.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c006da60 con 0x7f71c00a26f0 2024-09-16T14:42:33.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:33.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c8712640 10 monclient(hunting): authenticate will time out at 977.706968s 2024-09-16T14:42:33.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c6caf640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f71c00a26f0 0x7f71c01d5970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48624/0 (socket says 172.21.15.26:48624) 2024-09-16T14:42:33.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71c6caf640 1 -- 172.21.15.26:0/366595229 learned_addr learned my addr 172.21.15.26:0/366595229 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:33.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2003958583 0 0) 0x7f71c006da60 con 0x7f71c00a26f0 2024-09-16T14:42:33.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71b6ffd640 10 monclient(hunting): my global_id is 4331 2024-09-16T14:42:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71a4003530 con 0x7f71c00a26f0 2024-09-16T14:42:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049799952 0 0) 0x7f71c009e3f0 con 0x7f71c0099b60 2024-09-16T14:42:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71b6ffd640 10 monclient(hunting): my global_id is 4351 2024-09-16T14:42:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.584+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71c006da60 con 0x7f71c0099b60 2024-09-16T14:42:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3708435428 0 0) 0x7f71a4003530 con 0x7f71c00a26f0 2024-09-16T14:42:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71c009e3f0 con 0x7f71c00a26f0 2024-09-16T14:42:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387751976 0 0) 0x7f71c006e620 con 0x7f71c00a1c00 2024-09-16T14:42:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient(hunting): my global_id is 4341 2024-09-16T14:42:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71a4003530 con 0x7f71c00a1c00 2024-09-16T14:42:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f71b0001b50 con 0x7f71c00a26f0 2024-09-16T14:42:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.609 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1240706650 0 0) 0x7f71c006da60 con 0x7f71c0099b60 2024-09-16T14:42:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71c006e620 con 0x7f71c0099b60 2024-09-16T14:42:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f71bc001780 con 0x7f71c0099b60 2024-09-16T14:42:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:33.610 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.610 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.610 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.610 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.610 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.610 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.610 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.610 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.610 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.610 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3016767529 0 0) 0x7f71c009e3f0 con 0x7f71c00a26f0 2024-09-16T14:42:33.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 >> v1:172.21.15.189:6789/0 conn(0x7f71c0099b60 legacy=0x7f71c013ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 >> v1:172.21.15.26:6789/0 conn(0x7f71c00a1c00 legacy=0x7f71c013d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 monclient: found mon.c 2024-09-16T14:42:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c009eca0 con 0x7f71c00a26f0 2024-09-16T14:42:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71c8712640 5 monclient: authenticate success, global_id 4331 2024-09-16T14:42:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71c8712640 10 monclient: _renew_subs 2024-09-16T14:42:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71c8712640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71c8712640 1 -- 172.21.15.26:0/366595229 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c006f3c0 con 0x7f71c00a26f0 2024-09-16T14:42:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71c8712640 10 monclient: _renew_subs 2024-09-16T14:42:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71c8712640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71c8712640 1 -- 172.21.15.26:0/366595229 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c009f1d0 con 0x7f71c00a26f0 2024-09-16T14:42:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71c8712640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71c8712640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71c8712640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71c8712640 1 -- 172.21.15.26:0/366595229 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c006f1a0 con 0x7f71c00a26f0 2024-09-16T14:42:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71b0004c00 con 0x7f71c00a26f0 2024-09-16T14:42:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f71b0004f10 con 0x7f71c00a26f0 2024-09-16T14:42:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 10 monclient: dump: 2024-09-16T14:42:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.613 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f71b000f980 con 0x7f71c00a26f0 2024-09-16T14:42:33.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71c4cab640 4 set_mon_vals no callback set 2024-09-16T14:42:33.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f71b0043240 con 0x7f71c00a26f0 2024-09-16T14:42:33.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.588+0000 7f71c64ae640 10 monclient: get_auth_request con 0x7f71a4036530 auth_method 0 2024-09-16T14:42:33.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/4101130715 shutdown_connections 2024-09-16T14:42:33.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/4101130715 wait complete. 2024-09-16T14:42:33.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 10 monclient: build_initial_monmap 2024-09-16T14:42:33.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 10 monclient: monmap: 2024-09-16T14:42:33.616 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:33.617 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 1 Processor -- start 2024-09-16T14:42:33.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 1 -- start start 2024-09-16T14:42:33.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 10 monclient: init 2024-09-16T14:42:33.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:33.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:42:33.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 10 monclient(hunting): picked mon.noname-a con 0x7fc8dc099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:33.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 10 monclient(hunting): picked mon.noname-b con 0x7fc8dc0a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:33.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 10 monclient(hunting): picked mon.noname-c con 0x7fc8dc0a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:33.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc09ee20 con 0x7fc8dc099b60 2024-09-16T14:42:33.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc13df50 con 0x7fc8dc0a1c00 2024-09-16T14:42:33.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc13af80 con 0x7fc8dc0a26f0 2024-09-16T14:42:33.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:33.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 10 monclient(hunting): authenticate will time out at 977.714222s 2024-09-16T14:42:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8db7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc8dc099b60 0x7fc8dc13a160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57582/0 (socket says 172.21.15.26:57582) 2024-09-16T14:42:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8db7fe640 1 -- 172.21.15.26:0/1658863492 learned_addr learned my addr 172.21.15.26:0/1658863492 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231241126 0 0) 0x7fc8dc09ee20 con 0x7fc8dc099b60 2024-09-16T14:42:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient(hunting): my global_id is 4344 2024-09-16T14:42:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8c8003530 con 0x7fc8dc099b60 2024-09-16T14:42:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537775786 0 0) 0x7fc8dc13df50 con 0x7fc8dc0a1c00 2024-09-16T14:42:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient(hunting): my global_id is 4354 2024-09-16T14:42:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8dc09ee20 con 0x7fc8dc0a1c00 2024-09-16T14:42:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1156860605 0 0) 0x7fc8c8003530 con 0x7fc8dc099b60 2024-09-16T14:42:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8dc13df50 con 0x7fc8dc099b60 2024-09-16T14:42:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc8cc001960 con 0x7fc8dc099b60 2024-09-16T14:42:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient(hunting): dump: 2024-09-16T14:42:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.621 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2352062587 0 0) 0x7fc8dc09ee20 con 0x7fc8dc0a1c00 2024-09-16T14:42:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8c8003530 con 0x7fc8dc0a1c00 2024-09-16T14:42:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc8d00017b0 con 0x7fc8dc0a1c00 2024-09-16T14:42:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient(hunting): dump: 2024-09-16T14:42:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.622 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 124695798 0 0) 0x7fc8dc13df50 con 0x7fc8dc099b60 2024-09-16T14:42:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 >> v1:172.21.15.26:6790/0 conn(0x7fc8dc0a26f0 legacy=0x7fc8dc1d1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 >> v1:172.21.15.189:6789/0 conn(0x7fc8dc0a1c00 legacy=0x7fc8dc13a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:42:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 1 monclient: found mon.a 2024-09-16T14:42:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc13bbf0 con 0x7fc8dc099b60 2024-09-16T14:42:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient: _finish_auth 0 2024-09-16T14:42:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:42:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 5 monclient: authenticate success, global_id 4344 2024-09-16T14:42:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 10 monclient: _renew_subs 2024-09-16T14:42:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/1658863492 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8dc09f390 con 0x7fc8dc099b60 2024-09-16T14:42:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 10 monclient: _renew_subs 2024-09-16T14:42:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.592+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/1658863492 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8dc13dc70 con 0x7fc8dc099b60 2024-09-16T14:42:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.596+0000 7fc8e1c1a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.596+0000 7fc8e1c1a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.596+0000 7fc8e1c1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.596+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/1658863492 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8dc09ec70 con 0x7fc8dc099b60 2024-09-16T14:42:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.600+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8cc001ca0 con 0x7fc8dc099b60 2024-09-16T14:42:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.600+0000 7fc8c3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.600+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc8cc003a70 con 0x7fc8dc099b60 2024-09-16T14:42:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.600+0000 7fc8c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.600+0000 7fc8c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.600+0000 7fc8c3fff640 10 monclient: dump: 2024-09-16T14:42:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:33.625 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.600+0000 7fc8d9ffb640 4 set_mon_vals no callback set 2024-09-16T14:42:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.600+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc8cc003bd0 con 0x7fc8dc099b60 2024-09-16T14:42:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.600+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7fc8cc043090 con 0x7fc8dc099b60 2024-09-16T14:42:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.600+0000 7fc8daffd640 10 monclient: get_auth_request con 0x7fc8c8036390 auth_method 0 2024-09-16T14:42:33.627 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.2 2024-09-16T14:42:33.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.604+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc8cc0146c0 con 0x7fc8dc099b60 2024-09-16T14:42:33.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.604+0000 7fc8c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.604+0000 7fc8c3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:33.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.604+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f71b0014760 con 0x7f71c00a26f0 2024-09-16T14:42:33.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.604+0000 7f71b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:33.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.604+0000 7f71b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:33.628 DEBUG:teuthology.parallel:result is None 2024-09-16T14:42:33.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.652+0000 7f6011ffb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T14:42:33.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.652+0000 7f6011ffb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T14:42:33.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.652+0000 7f6011ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.652+0000 7f6011ffb640 1 -- 172.21.15.26:0/934123906 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f602c0a2d70 con 0x7f602c099b60 2024-09-16T14:42:33.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.652+0000 7f6013fff640 1 -- 172.21.15.26:0/934123906 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (unknown 383520633 0 555833778) 0x7f601c014260 con 0x7f602c099b60 2024-09-16T14:42:33.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.652+0000 7f6013fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T14:42:33.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.652+0000 7f6013fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:33.656 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-09-16T14:42:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.656+0000 7f6011ffb640 1 -- 172.21.15.26:0/934123906 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6018036490 legacy=0x7f6018038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.656+0000 7f6011ffb640 10 monclient: shutdown 2024-09-16T14:42:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.656+0000 7f6011ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.656+0000 7f6011ffb640 1 -- 172.21.15.26:0/934123906 >> v1:172.21.15.26:6789/0 conn(0x7f602c099b60 legacy=0x7f602c13c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.656+0000 7f6011ffb640 1 -- 172.21.15.26:0/934123906 shutdown_connections 2024-09-16T14:42:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.656+0000 7f6011ffb640 1 -- 172.21.15.26:0/934123906 >> 172.21.15.26:0/934123906 conn(0x7f602c0a3d00 msgr2=0x7f602c0a0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.656+0000 7f6011ffb640 1 -- 172.21.15.26:0/934123906 shutdown_connections 2024-09-16T14:42:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.656+0000 7f6011ffb640 1 -- 172.21.15.26:0/934123906 wait complete. 2024-09-16T14:42:33.678 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.3 2024-09-16T14:42:33.678 DEBUG:teuthology.parallel:result is None 2024-09-16T14:42:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.704+0000 7f624effd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T14:42:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.704+0000 7f624effd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T14:42:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.704+0000 7f624effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.704+0000 7f624effd640 1 -- 172.21.15.26:0/3601249150 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f623c005300 con 0x7f62640a1c00 2024-09-16T14:42:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.704+0000 7f62617fa640 1 -- 172.21.15.26:0/3601249150 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (unknown 1823589948 0 847495621) 0x7f6250010f10 con 0x7f62640a1c00 2024-09-16T14:42:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.704+0000 7f62617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T14:42:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.704+0000 7f62617fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:33.707 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-09-16T14:42:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.704+0000 7f624effd640 1 -- 172.21.15.26:0/3601249150 >> v1:172.21.15.189:6812/3196161582 conn(0x7f62480361f0 legacy=0x7f62480386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.704+0000 7f624effd640 10 monclient: shutdown 2024-09-16T14:42:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.704+0000 7f624effd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.704+0000 7f624effd640 1 -- 172.21.15.26:0/3601249150 >> v1:172.21.15.26:6789/0 conn(0x7f62640a1c00 legacy=0x7f626413a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.704+0000 7f624effd640 1 -- 172.21.15.26:0/3601249150 shutdown_connections 2024-09-16T14:42:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.704+0000 7f624effd640 1 -- 172.21.15.26:0/3601249150 >> 172.21.15.26:0/3601249150 conn(0x7f62640a3d00 msgr2=0x7f626409f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.704+0000 7f624effd640 1 -- 172.21.15.26:0/3601249150 shutdown_connections 2024-09-16T14:42:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.704+0000 7f624effd640 1 -- 172.21.15.26:0/3601249150 wait complete. 2024-09-16T14:42:33.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.708+0000 7f06669a3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T14:42:33.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.708+0000 7f06669a3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T14:42:33.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.708+0000 7f06669a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.708+0000 7f06669a3640 1 -- 172.21.15.26:0/2068912580 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f066011f360 con 0x7f0660099b60 2024-09-16T14:42:33.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.708+0000 7f065cff9640 1 -- 172.21.15.26:0/2068912580 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 847495621) 0x7f0650014670 con 0x7f0660099b60 2024-09-16T14:42:33.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.708+0000 7f065cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T14:42:33.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.708+0000 7f065cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:33.711 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-09-16T14:42:33.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.708+0000 7f06669a3640 1 -- 172.21.15.26:0/2068912580 >> v1:172.21.15.189:6812/3196161582 conn(0x7f064c036400 legacy=0x7f064c0388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.708+0000 7f06669a3640 10 monclient: shutdown 2024-09-16T14:42:33.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.708+0000 7f06669a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:33.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.708+0000 7f06669a3640 1 -- 172.21.15.26:0/2068912580 >> v1:172.21.15.26:6789/0 conn(0x7f0660099b60 legacy=0x7f0660138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.708+0000 7f06669a3640 1 -- 172.21.15.26:0/2068912580 shutdown_connections 2024-09-16T14:42:33.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.708+0000 7f06669a3640 1 -- 172.21.15.26:0/2068912580 >> 172.21.15.26:0/2068912580 conn(0x7f06600a3d00 msgr2=0x7f06600a0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:33.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.712+0000 7f06669a3640 1 -- 172.21.15.26:0/2068912580 shutdown_connections 2024-09-16T14:42:33.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.712+0000 7f06669a3640 1 -- 172.21.15.26:0/2068912580 wait complete. 2024-09-16T14:42:33.725 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.4 2024-09-16T14:42:33.725 DEBUG:teuthology.parallel:result is None 2024-09-16T14:42:33.735 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.0 2024-09-16T14:42:33.736 DEBUG:teuthology.parallel:result is None 2024-09-16T14:42:33.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.764+0000 7fc8e1c1a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T14:42:33.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.764+0000 7fc8e1c1a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T14:42:33.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.764+0000 7fc8e1c1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:33.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.764+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/1658863492 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc8dc13b650 con 0x7fc8dc099b60 2024-09-16T14:42:33.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.764+0000 7fc8c3fff640 1 -- 172.21.15.26:0/1658863492 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 555833778) 0x7fc8cc0142b0 con 0x7fc8dc099b60 2024-09-16T14:42:33.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.764+0000 7fc8c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T14:42:33.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.764+0000 7fc8c3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:33.769 INFO:teuthology.orchestra.run.smithi026.stdout:47244640258 2024-09-16T14:42:33.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.768+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/1658863492 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc8c8036390 legacy=0x7fc8c8038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.768+0000 7fc8e1c1a640 10 monclient: shutdown 2024-09-16T14:42:33.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.768+0000 7fc8e1c1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:33.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.768+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/1658863492 >> v1:172.21.15.26:6789/0 conn(0x7fc8dc099b60 legacy=0x7fc8dc13a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.768+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/1658863492 shutdown_connections 2024-09-16T14:42:33.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.768+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/1658863492 >> 172.21.15.26:0/1658863492 conn(0x7fc8dc0a3d00 msgr2=0x7fc8dc0a0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:33.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.768+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/1658863492 shutdown_connections 2024-09-16T14:42:33.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.768+0000 7fc8e1c1a640 1 -- 172.21.15.26:0/1658863492 wait complete. 2024-09-16T14:42:33.783 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.1 2024-09-16T14:42:33.783 DEBUG:teuthology.parallel:result is None 2024-09-16T14:42:33.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.812+0000 7f71c8712640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T14:42:33.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.812+0000 7f71c8712640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T14:42:33.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.812+0000 7f71c8712640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:33.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.812+0000 7f71c8712640 1 -- 172.21.15.26:0/366595229 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f71c00738e0 con 0x7f71c00a26f0 2024-09-16T14:42:33.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.816+0000 7f71b6ffd640 1 -- 172.21.15.26:0/366595229 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (unknown 2131975755 0 847495621) 0x7f71b0012e00 con 0x7f71c00a26f0 2024-09-16T14:42:33.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.816+0000 7f71b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T14:42:33.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.816+0000 7f71b6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:33.817 INFO:teuthology.orchestra.run.smithi026.stdout:47244640259 2024-09-16T14:42:33.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.816+0000 7f71c8712640 1 -- 172.21.15.26:0/366595229 >> v1:172.21.15.189:6812/3196161582 conn(0x7f71a4036530 legacy=0x7f71a40389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.816+0000 7f71c8712640 10 monclient: shutdown 2024-09-16T14:42:33.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.816+0000 7f71c8712640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:33.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.816+0000 7f71c8712640 1 -- 172.21.15.26:0/366595229 >> v1:172.21.15.26:6790/0 conn(0x7f71c00a26f0 legacy=0x7f71c01d5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.816+0000 7f71c8712640 1 -- 172.21.15.26:0/366595229 shutdown_connections 2024-09-16T14:42:33.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.816+0000 7f71c8712640 1 -- 172.21.15.26:0/366595229 >> 172.21.15.26:0/366595229 conn(0x7f71c00a3d00 msgr2=0x7f71c00a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:33.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.816+0000 7f71c8712640 1 -- 172.21.15.26:0/366595229 shutdown_connections 2024-09-16T14:42:33.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.816+0000 7f71c8712640 1 -- 172.21.15.26:0/366595229 wait complete. 2024-09-16T14:42:33.833 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.5 2024-09-16T14:42:33.833 DEBUG:teuthology.parallel:result is None 2024-09-16T14:42:33.833 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T14:42:33.834 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T14:42:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.984+0000 7f5d4c686640 10 monclient: get_monmap_and_config 2024-09-16T14:42:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.984+0000 7f5d4c686640 10 monclient: build_initial_monmap 2024-09-16T14:42:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.984+0000 7f5d4c686640 10 monclient: monmap: 2024-09-16T14:42:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:33.988742+0000 2024-09-16T14:42:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:33.988742+0000 2024-09-16T14:42:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:33.989 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 1 Processor -- start 2024-09-16T14:42:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 1 -- start start 2024-09-16T14:42:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient: init 2024-09-16T14:42:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient(hunting): picked mon.noname-a con 0x7f5d4409a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient(hunting): picked mon.noname-b con 0x7f5d4409d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient(hunting): picked mon.noname-c con 0x7f5d4409b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4409dc60 con 0x7f5d4409a550 2024-09-16T14:42:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d44099ae0 con 0x7f5d4409b040 2024-09-16T14:42:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4409b8f0 con 0x7f5d4409d1b0 2024-09-16T14:42:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient(hunting): authenticate will time out at 978.108927s 2024-09-16T14:42:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4a422640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5d4409a550 0x7f5d4409a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57590/0 (socket says 172.21.15.26:57590) 2024-09-16T14:42:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4a422640 1 -- 172.21.15.26:0/875644342 learned_addr learned my addr 172.21.15.26:0/875644342 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 1 -- 172.21.15.26:0/875644342 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555312823 0 0) 0x7f5d4409b8f0 con 0x7f5d4409d1b0 2024-09-16T14:42:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient(hunting): my global_id is 4334 2024-09-16T14:42:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 1 -- 172.21.15.26:0/875644342 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d2c003530 con 0x7f5d4409d1b0 2024-09-16T14:42:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 1 -- 172.21.15.26:0/875644342 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414306223 0 0) 0x7f5d4409dc60 con 0x7f5d4409a550 2024-09-16T14:42:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient(hunting): _negotiate 2024-09-16T14:42:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient(hunting): my global_id is 4347 2024-09-16T14:42:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 1 -- 172.21.15.26:0/875644342 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d4409b8f0 con 0x7f5d4409a550 2024-09-16T14:42:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 1 -- 172.21.15.26:0/875644342 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3165526932 0 0) 0x7f5d2c003530 con 0x7f5d4409d1b0 2024-09-16T14:42:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 1 -- 172.21.15.26:0/875644342 >> v1:172.21.15.189:6789/0 conn(0x7f5d4409b040 legacy=0x7f5d4413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 1 -- 172.21.15.26:0/875644342 >> v1:172.21.15.26:6789/0 conn(0x7f5d4409a550 legacy=0x7f5d4409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient: _finish_hunting 0 2024-09-16T14:42:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 1 monclient: found mon.noname-b 2024-09-16T14:42:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 1 -- 172.21.15.26:0/875644342 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4409bbd0 con 0x7f5d4409d1b0 2024-09-16T14:42:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient: _finish_auth 0 2024-09-16T14:42:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient: _check_auth_tickets 2024-09-16T14:42:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 1 -- 172.21.15.26:0/875644342 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5d380019a0 con 0x7f5d4409d1b0 2024-09-16T14:42:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 5 monclient: authenticate success, global_id 4334 2024-09-16T14:42:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient: dump: 2024-09-16T14:42:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.000 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 1 -- 172.21.15.26:0/875644342 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d38001bc0 con 0x7f5d4409d1b0 2024-09-16T14:42:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 1 -- 172.21.15.26:0/875644342 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5d38001f50 con 0x7f5d4409d1b0 2024-09-16T14:42:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 4 set_mon_vals no callback set 2024-09-16T14:42:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d49420640 10 monclient: dump: 2024-09-16T14:42:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.001 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient: shutdown 2024-09-16T14:42:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 1 -- 172.21.15.26:0/875644342 >> v1:172.21.15.26:6790/0 conn(0x7f5d4409d1b0 legacy=0x7f5d44136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 1 -- 172.21.15.26:0/875644342 shutdown_connections 2024-09-16T14:42:34.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 1 -- 172.21.15.26:0/875644342 >> 172.21.15.26:0/875644342 conn(0x7f5d4409fb50 msgr2=0x7f5d4409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:34.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 1 -- 172.21.15.26:0/875644342 shutdown_connections 2024-09-16T14:42:34.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 1 -- 172.21.15.26:0/875644342 wait complete. 2024-09-16T14:42:34.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient: build_initial_monmap 2024-09-16T14:42:34.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient: monmap: 2024-09-16T14:42:34.002 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:34.002 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:34.002 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:34.002 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:34.002 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:34.003 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 1 Processor -- start 2024-09-16T14:42:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 1 -- start start 2024-09-16T14:42:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient: init 2024-09-16T14:42:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient(hunting): picked mon.noname-c con 0x7f5d4409a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient(hunting): picked mon.noname-b con 0x7f5d4409b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient(hunting): picked mon.noname-a con 0x7f5d4409d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4411bc50 con 0x7f5d4409d1b0 2024-09-16T14:42:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d44070ce0 con 0x7f5d4409b040 2024-09-16T14:42:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d440a2860 con 0x7f5d4409a550 2024-09-16T14:42:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4c686640 10 monclient(hunting): authenticate will time out at 978.111801s 2024-09-16T14:42:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4a422640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5d4409a550 0x7f5d4413a0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48648/0 (socket says 172.21.15.26:48648) 2024-09-16T14:42:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.988+0000 7f5d4a422640 1 -- 172.21.15.26:0/879528866 learned_addr learned my addr 172.21.15.26:0/879528866 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477392044 0 0) 0x7f5d4411bc50 con 0x7f5d4409d1b0 2024-09-16T14:42:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): my global_id is 4350 2024-09-16T14:42:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d30003530 con 0x7f5d4409d1b0 2024-09-16T14:42:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624553190 0 0) 0x7f5d440a2860 con 0x7f5d4409a550 2024-09-16T14:42:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): my global_id is 4337 2024-09-16T14:42:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d4411bc50 con 0x7f5d4409a550 2024-09-16T14:42:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2367753347 0 0) 0x7f5d4411bc50 con 0x7f5d4409a550 2024-09-16T14:42:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d440a2860 con 0x7f5d4409a550 2024-09-16T14:42:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2307470343 0 0) 0x7f5d30003530 con 0x7f5d4409d1b0 2024-09-16T14:42:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d4411bc50 con 0x7f5d4409d1b0 2024-09-16T14:42:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5d3c001840 con 0x7f5d4409a550 2024-09-16T14:42:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.006 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5d34001800 con 0x7f5d4409d1b0 2024-09-16T14:42:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.008 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413044400 0 0) 0x7f5d44070ce0 con 0x7f5d4409b040 2024-09-16T14:42:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient(hunting): my global_id is 4360 2024-09-16T14:42:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d30003530 con 0x7f5d4409b040 2024-09-16T14:42:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1938342882 0 0) 0x7f5d440a2860 con 0x7f5d4409a550 2024-09-16T14:42:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 >> v1:172.21.15.189:6789/0 conn(0x7f5d4409b040 legacy=0x7f5d4413a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 >> v1:172.21.15.26:6789/0 conn(0x7f5d4409d1b0 legacy=0x7f5d441cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 monclient: found mon.c 2024-09-16T14:42:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4409a240 con 0x7f5d4409a550 2024-09-16T14:42:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d4c686640 5 monclient: authenticate success, global_id 4337 2024-09-16T14:42:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d4c686640 10 monclient: _renew_subs 2024-09-16T14:42:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d4c686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d4c686640 1 -- 172.21.15.26:0/879528866 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d44119880 con 0x7f5d4409a550 2024-09-16T14:42:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d4c686640 10 monclient: _renew_subs 2024-09-16T14:42:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d4c686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d4c686640 1 -- 172.21.15.26:0/879528866 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d4411a670 con 0x7f5d4409a550 2024-09-16T14:42:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d3c0047f0 con 0x7f5d4409a550 2024-09-16T14:42:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5d3c001d90 con 0x7f5d4409a550 2024-09-16T14:42:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d48c1f640 4 set_mon_vals no callback set 2024-09-16T14:42:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 10 monclient: dump: 2024-09-16T14:42:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.011 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d4c686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d4c686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d4c686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d4c686640 1 -- 172.21.15.26:0/879528866 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d24005050 con 0x7f5d4409a550 2024-09-16T14:42:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5d3c00f490 con 0x7f5d4409a550 2024-09-16T14:42:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f5d3c0430b0 con 0x7f5d4409a550 2024-09-16T14:42:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.992+0000 7f5d49c21640 10 monclient: get_auth_request con 0x7f5d30036470 auth_method 0 2024-09-16T14:42:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.996+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5d3c010a30 con 0x7f5d4409a550 2024-09-16T14:42:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.996+0000 7f5d42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:33.996+0000 7f5d42ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:34.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.148+0000 7f5d4c686640 1 -- 172.21.15.26:0/879528866 --> v1:172.21.15.189:6812/3196161582 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5d24000ba0 con 0x7f5d30036470 2024-09-16T14:42:34.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.152+0000 7f5d42ffd640 1 -- 172.21.15.26:0/879528866 <== mgr.4105 v1:172.21.15.189:6812/3196161582 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41158 (unknown 2965378022 0 3501448901) 0x7f5d24000ba0 con 0x7f5d30036470 2024-09-16T14:42:34.154 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:42:34.154 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T14:42:34.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.152+0000 7f5d4c686640 1 -- 172.21.15.26:0/879528866 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5d30036470 legacy=0x7f5d30038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.152+0000 7f5d4c686640 10 monclient: shutdown 2024-09-16T14:42:34.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.152+0000 7f5d4c686640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:34.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.152+0000 7f5d4c686640 1 -- 172.21.15.26:0/879528866 >> v1:172.21.15.26:6790/0 conn(0x7f5d4409a550 legacy=0x7f5d4413a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.152+0000 7f5d4c686640 1 -- 172.21.15.26:0/879528866 shutdown_connections 2024-09-16T14:42:34.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.152+0000 7f5d4c686640 1 -- 172.21.15.26:0/879528866 >> 172.21.15.26:0/879528866 conn(0x7f5d4409fb50 msgr2=0x7f5d4409e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:34.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.152+0000 7f5d4c686640 1 -- 172.21.15.26:0/879528866 shutdown_connections 2024-09-16T14:42:34.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.152+0000 7f5d4c686640 1 -- 172.21.15.26:0/879528866 wait complete. 2024-09-16T14:42:34.171 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-16T14:42:32.375130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167124,"kb_used_data":896,"kb_used_omap":9,"kb_used_meta":166134,"kb_avail":566063916,"statfs":{"total":579820584960,"available":579649449984,"internally_reserved":0,"allocated":917504,"data_stored":1290382,"data_compressed":61020,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9559,"internal_metadata":170121897},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.575415+0000","last_change":"2024-09-16T14:42:30.575415+0000","last_active":"2024-09-16T14:42:30.575415+0000","last_peered":"2024-09-16T14:42:30.575415+0000","last_clean":"2024-09-16T14:42:30.575415+0000","last_became_active":"2024-09-16T14:42:28.533291+0000","last_became_peered":"2024-09-16T14:42:28.533291+0000","last_unstale":"2024-09-16T14:42:30.575415+0000","last_undegraded":"2024-09-16T14:42:30.575415+0000","last_fullsized":"2024-09-16T14:42:30.575415+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:36:22.883054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085482099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.597981+0000","last_change":"2024-09-16T14:42:30.597981+0000","last_active":"2024-09-16T14:42:30.597981+0000","last_peered":"2024-09-16T14:42:30.597981+0000","last_clean":"2024-09-16T14:42:30.597981+0000","last_became_active":"2024-09-16T14:42:28.477522+0000","last_became_peered":"2024-09-16T14:42:28.477522+0000","last_unstale":"2024-09-16T14:42:30.597981+0000","last_undegraded":"2024-09-16T14:42:30.597981+0000","last_fullsized":"2024-09-16T14:42:30.597981+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:00:15.035933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065779100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.580369+0000","last_change":"2024-09-16T14:42:30.580369+0000","last_active":"2024-09-16T14:42:30.580369+0000","last_peered":"2024-09-16T14:42:30.580369+0000","last_clean":"2024-09-16T14:42:30.580369+0000","last_became_active":"2024-09-16T14:42:28.533172+0000","last_became_peered":"2024-09-16T14:42:28.533172+0000","last_unstale":"2024-09-16T14:42:30.580369+0000","last_undegraded":"2024-09-16T14:42:30.580369+0000","last_fullsized":"2024-09-16T14:42:30.580369+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:56:43.741456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061478100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.796631+0000","last_change":"2024-09-16T14:42:30.796631+0000","last_active":"2024-09-16T14:42:30.796631+0000","last_peered":"2024-09-16T14:42:30.796631+0000","last_clean":"2024-09-16T14:42:30.796631+0000","last_became_active":"2024-09-16T14:42:28.476209+0000","last_became_peered":"2024-09-16T14:42:28.476209+0000","last_unstale":"2024-09-16T14:42:30.796631+0000","last_undegraded":"2024-09-16T14:42:30.796631+0000","last_fullsized":"2024-09-16T14:42:30.796631+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:32:31.800477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039796999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"15'117","reported_seq":160,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.819644+0000","last_change":"2024-09-16T14:42:29.481739+0000","last_active":"2024-09-16T14:42:30.819644+0000","last_peered":"2024-09-16T14:42:30.819644+0000","last_clean":"2024-09-16T14:42:30.819644+0000","last_became_active":"2024-09-16T14:42:29.480896+0000","last_became_peered":"2024-09-16T14:42:29.480896+0000","last_unstale":"2024-09-16T14:42:30.819644+0000","last_undegraded":"2024-09-16T14:42:30.819644+0000","last_fullsized":"2024-09-16T14:42:30.819644+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:28.462531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:28.462531+0000","last_clean_scrub_stamp":"2024-09-16T14:42:28.462531+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:32:11.234353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.796643+0000","last_change":"2024-09-16T14:42:30.796643+0000","last_active":"2024-09-16T14:42:30.796643+0000","last_peered":"2024-09-16T14:42:30.796643+0000","last_clean":"2024-09-16T14:42:30.796643+0000","last_became_active":"2024-09-16T14:42:28.488186+0000","last_became_peered":"2024-09-16T14:42:28.488186+0000","last_unstale":"2024-09-16T14:42:30.796643+0000","last_undegraded":"2024-09-16T14:42:30.796643+0000","last_fullsized":"2024-09-16T14:42:30.796643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:40:41.632316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042440799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"15'2","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.582824+0000","last_change":"2024-09-16T14:42:30.581886+0000","last_active":"2024-09-16T14:42:30.582824+0000","last_peered":"2024-09-16T14:42:30.582824+0000","last_clean":"2024-09-16T14:42:30.582824+0000","last_became_active":"2024-09-16T14:42:28.487188+0000","last_became_peered":"2024-09-16T14:42:28.487188+0000","last_unstale":"2024-09-16T14:42:30.582824+0000","last_undegraded":"2024-09-16T14:42:30.582824+0000","last_fullsized":"2024-09-16T14:42:30.582824+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:10:26.171186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020492470000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.575555+0000","last_change":"2024-09-16T14:42:30.575555+0000","last_active":"2024-09-16T14:42:30.575555+0000","last_peered":"2024-09-16T14:42:30.575555+0000","last_clean":"2024-09-16T14:42:30.575555+0000","last_became_active":"2024-09-16T14:42:28.484409+0000","last_became_peered":"2024-09-16T14:42:28.484409+0000","last_unstale":"2024-09-16T14:42:30.575555+0000","last_undegraded":"2024-09-16T14:42:30.575555+0000","last_fullsized":"2024-09-16T14:42:30.575555+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:16:51.119532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097689999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.598002+0000","last_change":"2024-09-16T14:42:30.598002+0000","last_active":"2024-09-16T14:42:30.598002+0000","last_peered":"2024-09-16T14:42:30.598002+0000","last_clean":"2024-09-16T14:42:30.598002+0000","last_became_active":"2024-09-16T14:42:28.477416+0000","last_became_peered":"2024-09-16T14:42:28.477416+0000","last_unstale":"2024-09-16T14:42:30.598002+0000","last_undegraded":"2024-09-16T14:42:30.598002+0000","last_fullsized":"2024-09-16T14:42:30.598002+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:49.076917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067836399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61020,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":38748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":135168,"data_stored":38767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640258,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":192512,"data_stored":567676,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":38748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640258,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":567676,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31252,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340588,"statfs":{"total":96636764160,"available":96604762112,"internally_reserved":0,"allocated":135168,"data_stored":38767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T14:42:34.172 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T14:42:34.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.284+0000 7fc00b346640 10 monclient: get_monmap_and_config 2024-09-16T14:42:34.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.284+0000 7fc00b346640 10 monclient: build_initial_monmap 2024-09-16T14:42:34.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.284+0000 7fc00b346640 10 monclient: monmap: 2024-09-16T14:42:34.288 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:34.288 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.288 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:34.289014+0000 2024-09-16T14:42:34.288 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:34.289014+0000 2024-09-16T14:42:34.289 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:34.289 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.289 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:34.289 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:34.289 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:34.289 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 1 Processor -- start 2024-09-16T14:42:34.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 1 -- start start 2024-09-16T14:42:34.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient: init 2024-09-16T14:42:34.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:34.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:42:34.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient(hunting): picked mon.noname-c con 0x7fc00409f0a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:34.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient(hunting): picked mon.noname-a con 0x7fc00409fb90 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:34.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient(hunting): picked mon.noname-b con 0x7fc00409b780 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:34.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0040669c0 con 0x7fc00409fb90 2024-09-16T14:42:34.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00409a130 con 0x7fc00409f0a0 2024-09-16T14:42:34.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00409bbb0 con 0x7fc00409b780 2024-09-16T14:42:34.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:34.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient(hunting): authenticate will time out at 978.408811s 2024-09-16T14:42:34.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc0098e3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc00409b780 0x7fc00413b160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48654/0 (socket says 172.21.15.26:48654) 2024-09-16T14:42:34.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc0098e3640 1 -- 172.21.15.26:0/2633630773 learned_addr learned my addr 172.21.15.26:0/2633630773 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:34.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 1 -- 172.21.15.26:0/2633630773 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523085565 0 0) 0x7fc00409bbb0 con 0x7fc00409b780 2024-09-16T14:42:34.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient(hunting): my global_id is 4340 2024-09-16T14:42:34.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 1 -- 172.21.15.26:0/2633630773 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbff0003530 con 0x7fc00409b780 2024-09-16T14:42:34.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 1 -- 172.21.15.26:0/2633630773 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259167199 0 0) 0x7fc0040669c0 con 0x7fc00409fb90 2024-09-16T14:42:34.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient(hunting): my global_id is 4353 2024-09-16T14:42:34.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 1 -- 172.21.15.26:0/2633630773 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc00409bbb0 con 0x7fc00409fb90 2024-09-16T14:42:34.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 1 -- 172.21.15.26:0/2633630773 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272984222 0 0) 0x7fc00409a130 con 0x7fc00409f0a0 2024-09-16T14:42:34.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient(hunting): my global_id is 4363 2024-09-16T14:42:34.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 1 -- 172.21.15.26:0/2633630773 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0040669c0 con 0x7fc00409f0a0 2024-09-16T14:42:34.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 1 -- 172.21.15.26:0/2633630773 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 239809951 0 0) 0x7fc00409bbb0 con 0x7fc00409fb90 2024-09-16T14:42:34.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 1 -- 172.21.15.26:0/2633630773 >> v1:172.21.15.26:6790/0 conn(0x7fc00409b780 legacy=0x7fc00413b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 1 -- 172.21.15.26:0/2633630773 >> v1:172.21.15.189:6789/0 conn(0x7fc00409f0a0 legacy=0x7fc00409f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient: _finish_hunting 0 2024-09-16T14:42:34.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 1 monclient: found mon.noname-a 2024-09-16T14:42:34.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:34.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:42:34.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 1 -- 172.21.15.26:0/2633630773 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc004099db0 con 0x7fc00409fb90 2024-09-16T14:42:34.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient: _finish_auth 0 2024-09-16T14:42:34.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient: _check_auth_tickets 2024-09-16T14:42:34.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:34.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 1 -- 172.21.15.26:0/2633630773 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbff4001600 con 0x7fc00409fb90 2024-09-16T14:42:34.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:34.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient: dump: 2024-09-16T14:42:34.299 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.299 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.299 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.299 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.300 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 5 monclient: authenticate success, global_id 4353 2024-09-16T14:42:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 1 -- 172.21.15.26:0/2633630773 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbff4001920 con 0x7fc00409fb90 2024-09-16T14:42:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 1 -- 172.21.15.26:0/2633630773 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbff4001cf0 con 0x7fc00409fb90 2024-09-16T14:42:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 4 set_mon_vals no callback set 2024-09-16T14:42:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fbffbfff640 10 monclient: dump: 2024-09-16T14:42:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.302 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient: shutdown 2024-09-16T14:42:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 1 -- 172.21.15.26:0/2633630773 >> v1:172.21.15.26:6789/0 conn(0x7fc00409fb90 legacy=0x7fc004137a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 1 -- 172.21.15.26:0/2633630773 shutdown_connections 2024-09-16T14:42:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 1 -- 172.21.15.26:0/2633630773 >> 172.21.15.26:0/2633630773 conn(0x7fc0040a0360 msgr2=0x7fc0040a0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 1 -- 172.21.15.26:0/2633630773 shutdown_connections 2024-09-16T14:42:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 1 -- 172.21.15.26:0/2633630773 wait complete. 2024-09-16T14:42:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient: build_initial_monmap 2024-09-16T14:42:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient: monmap: 2024-09-16T14:42:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:34.303 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 1 Processor -- start 2024-09-16T14:42:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 1 -- start start 2024-09-16T14:42:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient: init 2024-09-16T14:42:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:42:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient(hunting): picked mon.noname-b con 0x7fc00409b780 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient(hunting): picked mon.noname-a con 0x7fc00409f0a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:34.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient(hunting): picked mon.noname-c con 0x7fc00409fb90 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:34.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0040a26a0 con 0x7fc00409f0a0 2024-09-16T14:42:34.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00409a880 con 0x7fc00409b780 2024-09-16T14:42:34.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00412bcb0 con 0x7fc00409fb90 2024-09-16T14:42:34.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:34.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.288+0000 7fc00b346640 10 monclient(hunting): authenticate will time out at 978.411970s 2024-09-16T14:42:34.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fc0098e3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc00409fb90 0x7fc0041cf9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48660/0 (socket says 172.21.15.26:48660) 2024-09-16T14:42:34.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fc0098e3640 1 -- 172.21.15.26:0/471085314 learned_addr learned my addr 172.21.15.26:0/471085314 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:34.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fc0088e1640 1 --1- 172.21.15.26:0/471085314 >> v1:172.21.15.26:6789/0 conn(0x7fc00409f0a0 0x7fc0041cc270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57612/0 (socket says 172.21.15.26:57612) 2024-09-16T14:42:34.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353569118 0 0) 0x7fc00412bcb0 con 0x7fc00409fb90 2024-09-16T14:42:34.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): my global_id is 4343 2024-09-16T14:42:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfe8003530 con 0x7fc00409fb90 2024-09-16T14:42:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396645933 0 0) 0x7fc004099db0 con 0x7fc00409f0a0 2024-09-16T14:42:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): my global_id is 4356 2024-09-16T14:42:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc00412bcb0 con 0x7fc00409f0a0 2024-09-16T14:42:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2567977866 0 0) 0x7fbfe8003530 con 0x7fc00409fb90 2024-09-16T14:42:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc004099db0 con 0x7fc00409fb90 2024-09-16T14:42:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbffc001d80 con 0x7fc00409fb90 2024-09-16T14:42:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): dump: 2024-09-16T14:42:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.306 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2204523179 0 0) 0x7fc00412bcb0 con 0x7fc00409f0a0 2024-09-16T14:42:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfe8003530 con 0x7fc00409f0a0 2024-09-16T14:42:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbff4001a90 con 0x7fc00409f0a0 2024-09-16T14:42:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): dump: 2024-09-16T14:42:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.308 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966494280 0 0) 0x7fc00409a880 con 0x7fc00409b780 2024-09-16T14:42:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient(hunting): my global_id is 4366 2024-09-16T14:42:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc00412bcb0 con 0x7fc00409b780 2024-09-16T14:42:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3173325799 0 0) 0x7fc004099db0 con 0x7fc00409fb90 2024-09-16T14:42:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 >> v1:172.21.15.189:6789/0 conn(0x7fc00409b780 legacy=0x7fc004132ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 >> v1:172.21.15.26:6789/0 conn(0x7fc00409f0a0 legacy=0x7fc0041cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:42:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 monclient: found mon.c 2024-09-16T14:42:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00412bf90 con 0x7fc00409fb90 2024-09-16T14:42:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient: _finish_auth 0 2024-09-16T14:42:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:42:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fc00b346640 5 monclient: authenticate success, global_id 4343 2024-09-16T14:42:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fc00b346640 10 monclient: _renew_subs 2024-09-16T14:42:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fc00b346640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fc00b346640 1 -- 172.21.15.26:0/471085314 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc004071690 con 0x7fc00409fb90 2024-09-16T14:42:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fc00b346640 10 monclient: _renew_subs 2024-09-16T14:42:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fc00b346640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fc00b346640 1 -- 172.21.15.26:0/471085314 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc004129e50 con 0x7fc00409fb90 2024-09-16T14:42:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbffc004b70 con 0x7fc00409fb90 2024-09-16T14:42:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbffc0019b0 con 0x7fc00409fb90 2024-09-16T14:42:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbffb7fe640 4 set_mon_vals no callback set 2024-09-16T14:42:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 10 monclient: dump: 2024-09-16T14:42:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.311 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbffc00f790 con 0x7fc00409fb90 2024-09-16T14:42:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fc00b346640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fc00b346640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fc00b346640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fc00b346640 1 -- 172.21.15.26:0/471085314 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe4005050 con 0x7fc00409fb90 2024-09-16T14:42:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7fbffc043460 con 0x7fc00409fb90 2024-09-16T14:42:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.292+0000 7fc0090e2640 10 monclient: get_auth_request con 0x7fbfe80363b0 auth_method 0 2024-09-16T14:42:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.296+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbffc010d80 con 0x7fc00409fb90 2024-09-16T14:42:34.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.296+0000 7fbff9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:34.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.296+0000 7fbff9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.444+0000 7fc00b346640 1 -- 172.21.15.26:0/471085314 --> v1:172.21.15.189:6812/3196161582 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbfe4000ba0 con 0x7fbfe80363b0 2024-09-16T14:42:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.448+0000 7fbff9ffb640 1 -- 172.21.15.26:0/471085314 <== mgr.4105 v1:172.21.15.189:6812/3196161582 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41071 (unknown 2965378022 0 2295359421) 0x7fbfe4000ba0 con 0x7fbfe80363b0 2024-09-16T14:42:34.453 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:42:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T14:42:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.452+0000 7fc00b346640 1 -- 172.21.15.26:0/471085314 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbfe80363b0 legacy=0x7fbfe8038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.452+0000 7fc00b346640 10 monclient: shutdown 2024-09-16T14:42:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.452+0000 7fc00b346640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.452+0000 7fc00b346640 1 -- 172.21.15.26:0/471085314 >> v1:172.21.15.26:6790/0 conn(0x7fc00409fb90 legacy=0x7fc0041cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.452+0000 7fc00b346640 1 -- 172.21.15.26:0/471085314 shutdown_connections 2024-09-16T14:42:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.452+0000 7fc00b346640 1 -- 172.21.15.26:0/471085314 >> 172.21.15.26:0/471085314 conn(0x7fc0040a0360 msgr2=0x7fc0040a1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:34.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.452+0000 7fc00b346640 1 -- 172.21.15.26:0/471085314 shutdown_connections 2024-09-16T14:42:34.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.452+0000 7fc00b346640 1 -- 172.21.15.26:0/471085314 wait complete. 2024-09-16T14:42:34.469 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-16T14:42:34.375713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167124,"kb_used_data":896,"kb_used_omap":9,"kb_used_meta":166134,"kb_avail":566063916,"statfs":{"total":579820584960,"available":579649449984,"internally_reserved":0,"allocated":917504,"data_stored":1290382,"data_compressed":61020,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9559,"internal_metadata":170121897},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000582"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.575415+0000","last_change":"2024-09-16T14:42:30.575415+0000","last_active":"2024-09-16T14:42:30.575415+0000","last_peered":"2024-09-16T14:42:30.575415+0000","last_clean":"2024-09-16T14:42:30.575415+0000","last_became_active":"2024-09-16T14:42:28.533291+0000","last_became_peered":"2024-09-16T14:42:28.533291+0000","last_unstale":"2024-09-16T14:42:30.575415+0000","last_undegraded":"2024-09-16T14:42:30.575415+0000","last_fullsized":"2024-09-16T14:42:30.575415+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:36:22.883054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085482099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.597981+0000","last_change":"2024-09-16T14:42:30.597981+0000","last_active":"2024-09-16T14:42:30.597981+0000","last_peered":"2024-09-16T14:42:30.597981+0000","last_clean":"2024-09-16T14:42:30.597981+0000","last_became_active":"2024-09-16T14:42:28.477522+0000","last_became_peered":"2024-09-16T14:42:28.477522+0000","last_unstale":"2024-09-16T14:42:30.597981+0000","last_undegraded":"2024-09-16T14:42:30.597981+0000","last_fullsized":"2024-09-16T14:42:30.597981+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:00:15.035933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065779100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.580369+0000","last_change":"2024-09-16T14:42:30.580369+0000","last_active":"2024-09-16T14:42:30.580369+0000","last_peered":"2024-09-16T14:42:30.580369+0000","last_clean":"2024-09-16T14:42:30.580369+0000","last_became_active":"2024-09-16T14:42:28.533172+0000","last_became_peered":"2024-09-16T14:42:28.533172+0000","last_unstale":"2024-09-16T14:42:30.580369+0000","last_undegraded":"2024-09-16T14:42:30.580369+0000","last_fullsized":"2024-09-16T14:42:30.580369+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:56:43.741456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061478100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.796631+0000","last_change":"2024-09-16T14:42:30.796631+0000","last_active":"2024-09-16T14:42:30.796631+0000","last_peered":"2024-09-16T14:42:30.796631+0000","last_clean":"2024-09-16T14:42:30.796631+0000","last_became_active":"2024-09-16T14:42:28.476209+0000","last_became_peered":"2024-09-16T14:42:28.476209+0000","last_unstale":"2024-09-16T14:42:30.796631+0000","last_undegraded":"2024-09-16T14:42:30.796631+0000","last_fullsized":"2024-09-16T14:42:30.796631+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:32:31.800477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039796999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"15'117","reported_seq":160,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.819644+0000","last_change":"2024-09-16T14:42:29.481739+0000","last_active":"2024-09-16T14:42:30.819644+0000","last_peered":"2024-09-16T14:42:30.819644+0000","last_clean":"2024-09-16T14:42:30.819644+0000","last_became_active":"2024-09-16T14:42:29.480896+0000","last_became_peered":"2024-09-16T14:42:29.480896+0000","last_unstale":"2024-09-16T14:42:30.819644+0000","last_undegraded":"2024-09-16T14:42:30.819644+0000","last_fullsized":"2024-09-16T14:42:30.819644+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:28.462531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:28.462531+0000","last_clean_scrub_stamp":"2024-09-16T14:42:28.462531+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:32:11.234353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.796643+0000","last_change":"2024-09-16T14:42:30.796643+0000","last_active":"2024-09-16T14:42:30.796643+0000","last_peered":"2024-09-16T14:42:30.796643+0000","last_clean":"2024-09-16T14:42:30.796643+0000","last_became_active":"2024-09-16T14:42:28.488186+0000","last_became_peered":"2024-09-16T14:42:28.488186+0000","last_unstale":"2024-09-16T14:42:30.796643+0000","last_undegraded":"2024-09-16T14:42:30.796643+0000","last_fullsized":"2024-09-16T14:42:30.796643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:40:41.632316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042440799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"15'2","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.582824+0000","last_change":"2024-09-16T14:42:30.581886+0000","last_active":"2024-09-16T14:42:30.582824+0000","last_peered":"2024-09-16T14:42:30.582824+0000","last_clean":"2024-09-16T14:42:30.582824+0000","last_became_active":"2024-09-16T14:42:28.487188+0000","last_became_peered":"2024-09-16T14:42:28.487188+0000","last_unstale":"2024-09-16T14:42:30.582824+0000","last_undegraded":"2024-09-16T14:42:30.582824+0000","last_fullsized":"2024-09-16T14:42:30.582824+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:10:26.171186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020492470000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.575555+0000","last_change":"2024-09-16T14:42:30.575555+0000","last_active":"2024-09-16T14:42:30.575555+0000","last_peered":"2024-09-16T14:42:30.575555+0000","last_clean":"2024-09-16T14:42:30.575555+0000","last_became_active":"2024-09-16T14:42:28.484409+0000","last_became_peered":"2024-09-16T14:42:28.484409+0000","last_unstale":"2024-09-16T14:42:30.575555+0000","last_undegraded":"2024-09-16T14:42:30.575555+0000","last_fullsized":"2024-09-16T14:42:30.575555+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:16:51.119532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097689999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.598002+0000","last_change":"2024-09-16T14:42:30.598002+0000","last_active":"2024-09-16T14:42:30.598002+0000","last_peered":"2024-09-16T14:42:30.598002+0000","last_clean":"2024-09-16T14:42:30.598002+0000","last_became_active":"2024-09-16T14:42:28.477416+0000","last_became_peered":"2024-09-16T14:42:28.477416+0000","last_unstale":"2024-09-16T14:42:30.598002+0000","last_undegraded":"2024-09-16T14:42:30.598002+0000","last_fullsized":"2024-09-16T14:42:30.598002+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:49.076917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067836399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61020,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":38748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":135168,"data_stored":38767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":192512,"data_stored":567676,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":38748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":567676,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31252,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340588,"statfs":{"total":96636764160,"available":96604762112,"internally_reserved":0,"allocated":135168,"data_stored":38767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T14:42:34.471 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T14:42:34.472 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T14:42:34.472 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T14:42:34.472 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T14:42:34.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 10 monclient: get_monmap_and_config 2024-09-16T14:42:34.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 10 monclient: build_initial_monmap 2024-09-16T14:42:34.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 10 monclient: monmap: 2024-09-16T14:42:34.581 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:34.582360+0000 2024-09-16T14:42:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:34.582360+0000 2024-09-16T14:42:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:34.582 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 1 Processor -- start 2024-09-16T14:42:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 1 -- start start 2024-09-16T14:42:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 10 monclient: init 2024-09-16T14:42:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 10 monclient(hunting): picked mon.noname-c con 0x7f553c09e7a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 10 monclient(hunting): picked mon.noname-b con 0x7f553c09cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 10 monclient(hunting): picked mon.noname-a con 0x7f553c09d210 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c09aff0 con 0x7f553c09d210 2024-09-16T14:42:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c09b2d0 con 0x7f553c09e7a0 2024-09-16T14:42:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c09b5b0 con 0x7f553c09cdc0 2024-09-16T14:42:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 10 monclient(hunting): authenticate will time out at 978.702490s 2024-09-16T14:42:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553b7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f553c09cdc0 0x7f553c1370a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48668/0 (socket says 172.21.15.26:48668) 2024-09-16T14:42:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553b7fe640 1 -- 172.21.15.26:0/4023506731 learned_addr learned my addr 172.21.15.26:0/4023506731 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 1 -- 172.21.15.26:0/4023506731 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296821456 0 0) 0x7f553c09b5b0 con 0x7f553c09cdc0 2024-09-16T14:42:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient(hunting): my global_id is 4346 2024-09-16T14:42:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 1 -- 172.21.15.26:0/4023506731 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5528003530 con 0x7f553c09cdc0 2024-09-16T14:42:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 1 -- 172.21.15.26:0/4023506731 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747927465 0 0) 0x7f553c09aff0 con 0x7f553c09d210 2024-09-16T14:42:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient(hunting): my global_id is 4359 2024-09-16T14:42:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 1 -- 172.21.15.26:0/4023506731 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f553c09b5b0 con 0x7f553c09d210 2024-09-16T14:42:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 1 -- 172.21.15.26:0/4023506731 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3400428687 0 0) 0x7f5528003530 con 0x7f553c09cdc0 2024-09-16T14:42:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 1 -- 172.21.15.26:0/4023506731 >> v1:172.21.15.189:6789/0 conn(0x7f553c09e7a0 legacy=0x7f553c09da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 1 -- 172.21.15.26:0/4023506731 >> v1:172.21.15.26:6789/0 conn(0x7f553c09d210 legacy=0x7f553c13a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 1 monclient: found mon.noname-b 2024-09-16T14:42:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 1 -- 172.21.15.26:0/4023506731 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553c09b890 con 0x7f553c09cdc0 2024-09-16T14:42:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient: _finish_auth 0 2024-09-16T14:42:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 1 -- 172.21.15.26:0/4023506731 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5524001620 con 0x7f553c09cdc0 2024-09-16T14:42:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient: dump: 2024-09-16T14:42:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.589 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.589 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.589 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 5 monclient: authenticate success, global_id 4346 2024-09-16T14:42:34.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:34.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 1 -- 172.21.15.26:0/4023506731 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5524001880 con 0x7f553c09cdc0 2024-09-16T14:42:34.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:34.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 1 -- 172.21.15.26:0/4023506731 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5524001c50 con 0x7f553c09cdc0 2024-09-16T14:42:34.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:34.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f553affd640 10 monclient: dump: 2024-09-16T14:42:34.590 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.590 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.590 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.590 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.590 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.590 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.590 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.590 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.590 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.590 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:34.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 4 set_mon_vals no callback set 2024-09-16T14:42:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 10 monclient: shutdown 2024-09-16T14:42:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 1 -- 172.21.15.26:0/4023506731 >> v1:172.21.15.26:6790/0 conn(0x7f553c09cdc0 legacy=0x7f553c1370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.580+0000 7f55422b6640 1 -- 172.21.15.26:0/4023506731 shutdown_connections 2024-09-16T14:42:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 1 -- 172.21.15.26:0/4023506731 >> 172.21.15.26:0/4023506731 conn(0x7f553c09fb50 msgr2=0x7f553c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 1 -- 172.21.15.26:0/4023506731 shutdown_connections 2024-09-16T14:42:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 1 -- 172.21.15.26:0/4023506731 wait complete. 2024-09-16T14:42:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 monclient: build_initial_monmap 2024-09-16T14:42:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 monclient: monmap: 2024-09-16T14:42:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:34.592 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:34.592 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:34.592 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:34.592 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:34.592 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:34.592 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 1 Processor -- start 2024-09-16T14:42:34.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 1 -- start start 2024-09-16T14:42:34.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 monclient: init 2024-09-16T14:42:34.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:34.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:34.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 monclient(hunting): picked mon.noname-b con 0x7f553c09cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:34.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 monclient(hunting): picked mon.noname-c con 0x7f553c09d210 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:34.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 monclient(hunting): picked mon.noname-a con 0x7f553c09e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:34.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c0a2340 con 0x7f553c09e7a0 2024-09-16T14:42:34.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c11f270 con 0x7f553c09cdc0 2024-09-16T14:42:34.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c0a1f10 con 0x7f553c09d210 2024-09-16T14:42:34.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:34.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 monclient(hunting): authenticate will time out at 978.705525s 2024-09-16T14:42:34.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f553b7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f553c09d210 0x7f553c1ce3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48680/0 (socket says 172.21.15.26:48680) 2024-09-16T14:42:34.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f553b7fe640 1 -- 172.21.15.26:0/3359360048 learned_addr learned my addr 172.21.15.26:0/3359360048 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:34.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194286633 0 0) 0x7f553c0a1f10 con 0x7f553c09d210 2024-09-16T14:42:34.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): my global_id is 4349 2024-09-16T14:42:34.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f551c003530 con 0x7f553c09d210 2024-09-16T14:42:34.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1997525067 0 0) 0x7f5534001560 con 0x7f553c09e7a0 2024-09-16T14:42:34.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): my global_id is 4362 2024-09-16T14:42:34.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f553c0a1f10 con 0x7f553c09e7a0 2024-09-16T14:42:34.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3002012284 0 0) 0x7f551c003530 con 0x7f553c09d210 2024-09-16T14:42:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5534001560 con 0x7f553c09d210 2024-09-16T14:42:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5524001520 con 0x7f553c09d210 2024-09-16T14:42:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.599 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 901349713 0 0) 0x7f553c0a1f10 con 0x7f553c09e7a0 2024-09-16T14:42:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f551c003530 con 0x7f553c09e7a0 2024-09-16T14:42:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5534001c00 con 0x7f553c09e7a0 2024-09-16T14:42:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.600 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680283671 0 0) 0x7f553c11f270 con 0x7f553c09cdc0 2024-09-16T14:42:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient(hunting): my global_id is 4372 2024-09-16T14:42:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f553c0a1f10 con 0x7f553c09cdc0 2024-09-16T14:42:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3180696123 0 0) 0x7f5534001560 con 0x7f553c09d210 2024-09-16T14:42:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 >> v1:172.21.15.189:6789/0 conn(0x7f553c09cdc0 legacy=0x7f553c12ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 >> v1:172.21.15.26:6789/0 conn(0x7f553c09e7a0 legacy=0x7f553c1d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:42:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 monclient: found mon.c 2024-09-16T14:42:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553c09a240 con 0x7f553c09d210 2024-09-16T14:42:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient: _finish_auth 0 2024-09-16T14:42:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:42:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 5 monclient: authenticate success, global_id 4349 2024-09-16T14:42:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 monclient: _renew_subs 2024-09-16T14:42:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 1 -- 172.21.15.26:0/3359360048 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f553c11ff30 con 0x7f553c09d210 2024-09-16T14:42:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 monclient: _renew_subs 2024-09-16T14:42:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 1 -- 172.21.15.26:0/3359360048 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f553c1c9590 con 0x7f553c09d210 2024-09-16T14:42:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5524001180 con 0x7f553c09d210 2024-09-16T14:42:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5524004da0 con 0x7f553c09d210 2024-09-16T14:42:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f553a7fc640 4 set_mon_vals no callback set 2024-09-16T14:42:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 10 monclient: dump: 2024-09-16T14:42:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.603 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f552400f810 con 0x7f553c09d210 2024-09-16T14:42:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f55422b6640 1 -- 172.21.15.26:0/3359360048 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5520005050 con 0x7f553c09d210 2024-09-16T14:42:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f55240433c0 con 0x7f553c09d210 2024-09-16T14:42:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.584+0000 7f553bfff640 10 monclient: get_auth_request con 0x7f551c0363e0 auth_method 0 2024-09-16T14:42:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.588+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5524014800 con 0x7f553c09d210 2024-09-16T14:42:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.588+0000 7f5538ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.588+0000 7f5538ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:34.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.780+0000 7f55422b6640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-16T14:42:34.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.784+0000 7f55422b6640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-16T14:42:34.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.784+0000 7f55422b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.784+0000 7f55422b6640 1 -- 172.21.15.26:0/3359360048 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5520005880 con 0x7f553c09d210 2024-09-16T14:42:34.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.784+0000 7f5538ff9640 1 -- 172.21.15.26:0/3359360048 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f5524012ea0 con 0x7f553c09d210 2024-09-16T14:42:34.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.784+0000 7f5538ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-16T14:42:34.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.784+0000 7f5538ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:34.788 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:42:34.788 INFO:teuthology.orchestra.run.smithi026.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T14:42:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.784+0000 7f55422b6640 1 -- 172.21.15.26:0/3359360048 >> v1:172.21.15.189:6812/3196161582 conn(0x7f551c0363e0 legacy=0x7f551c0388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.784+0000 7f55422b6640 10 monclient: shutdown 2024-09-16T14:42:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.784+0000 7f55422b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.784+0000 7f55422b6640 1 -- 172.21.15.26:0/3359360048 >> v1:172.21.15.26:6790/0 conn(0x7f553c09d210 legacy=0x7f553c1ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.784+0000 7f55422b6640 1 -- 172.21.15.26:0/3359360048 shutdown_connections 2024-09-16T14:42:34.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.784+0000 7f55422b6640 1 -- 172.21.15.26:0/3359360048 >> 172.21.15.26:0/3359360048 conn(0x7f553c09fb50 msgr2=0x7f553c1b7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:34.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.784+0000 7f55422b6640 1 -- 172.21.15.26:0/3359360048 shutdown_connections 2024-09-16T14:42:34.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.784+0000 7f55422b6640 1 -- 172.21.15.26:0/3359360048 wait complete. 2024-09-16T14:42:34.801 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T14:42:34.802 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-16T14:42:34.817 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-16T14:42:34.817 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T14:42:34.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.936+0000 7f5708c57640 10 monclient: get_monmap_and_config 2024-09-16T14:42:34.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.936+0000 7f5708c57640 10 monclient: build_initial_monmap 2024-09-16T14:42:34.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.936+0000 7f5708c57640 10 monclient: monmap: 2024-09-16T14:42:34.939 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:34.939 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:34.939976+0000 2024-09-16T14:42:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:34.939976+0000 2024-09-16T14:42:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:34.940 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.936+0000 7f5708c57640 1 Processor -- start 2024-09-16T14:42:34.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 1 -- start start 2024-09-16T14:42:34.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient: init 2024-09-16T14:42:34.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:34.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:42:34.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient(hunting): picked mon.noname-c con 0x7f57040a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:34.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient(hunting): picked mon.noname-a con 0x7f57040a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:34.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient(hunting): picked mon.noname-b con 0x7f57040a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:34.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57040a3250 con 0x7f57040a0000 2024-09-16T14:42:34.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570409f600 con 0x7f57040a1d10 2024-09-16T14:42:34.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57040a3830 con 0x7f57040a2800 2024-09-16T14:42:34.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:34.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient(hunting): authenticate will time out at 979.060647s 2024-09-16T14:42:34.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5702d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f57040a2800 0x7f570413d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48696/0 (socket says 172.21.15.26:48696) 2024-09-16T14:42:34.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5702d9d640 1 -- 172.21.15.26:0/2272379394 learned_addr learned my addr 172.21.15.26:0/2272379394 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:34.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 1 -- 172.21.15.26:0/2272379394 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068518889 0 0) 0x7f57040a3250 con 0x7f57040a0000 2024-09-16T14:42:34.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient(hunting): my global_id is 4365 2024-09-16T14:42:34.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 1 -- 172.21.15.26:0/2272379394 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56ec003530 con 0x7f57040a0000 2024-09-16T14:42:34.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 1 -- 172.21.15.26:0/2272379394 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770613396 0 0) 0x7f57040a3830 con 0x7f57040a2800 2024-09-16T14:42:34.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient(hunting): my global_id is 4352 2024-09-16T14:42:34.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 1 -- 172.21.15.26:0/2272379394 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57040a3250 con 0x7f57040a2800 2024-09-16T14:42:34.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 1 -- 172.21.15.26:0/2272379394 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3796995701 0 0) 0x7f56ec003530 con 0x7f57040a0000 2024-09-16T14:42:34.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 1 -- 172.21.15.26:0/2272379394 >> v1:172.21.15.26:6790/0 conn(0x7f57040a2800 legacy=0x7f570413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 1 -- 172.21.15.26:0/2272379394 >> v1:172.21.15.189:6789/0 conn(0x7f57040a1d10 legacy=0x7f57040a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient: _finish_hunting 0 2024-09-16T14:42:34.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 1 monclient: found mon.noname-a 2024-09-16T14:42:34.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:34.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:42:34.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 1 -- 172.21.15.26:0/2272379394 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570409ed60 con 0x7f57040a0000 2024-09-16T14:42:34.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient: _finish_auth 0 2024-09-16T14:42:34.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient: _check_auth_tickets 2024-09-16T14:42:34.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:34.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 1 -- 172.21.15.26:0/2272379394 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f56f8001620 con 0x7f57040a0000 2024-09-16T14:42:34.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 5 monclient: authenticate success, global_id 4365 2024-09-16T14:42:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient: dump: 2024-09-16T14:42:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.946 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.947 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 1 -- 172.21.15.26:0/2272379394 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56f8001840 con 0x7f57040a0000 2024-09-16T14:42:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 1 -- 172.21.15.26:0/2272379394 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f56f8001c10 con 0x7f57040a0000 2024-09-16T14:42:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f570159a640 10 monclient: dump: 2024-09-16T14:42:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.947 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.948 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 4 set_mon_vals no callback set 2024-09-16T14:42:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient: shutdown 2024-09-16T14:42:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 1 -- 172.21.15.26:0/2272379394 >> v1:172.21.15.26:6789/0 conn(0x7f57040a0000 legacy=0x7f5704139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 1 -- 172.21.15.26:0/2272379394 shutdown_connections 2024-09-16T14:42:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 1 -- 172.21.15.26:0/2272379394 >> 172.21.15.26:0/2272379394 conn(0x7f57040a7f30 msgr2=0x7f57040a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 1 -- 172.21.15.26:0/2272379394 shutdown_connections 2024-09-16T14:42:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 1 -- 172.21.15.26:0/2272379394 wait complete. 2024-09-16T14:42:34.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient: build_initial_monmap 2024-09-16T14:42:34.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient: monmap: 2024-09-16T14:42:34.952 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:34.952 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:34.952 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:34.952 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:34.952 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:34.952 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:34.952 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:34.952 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:34.953 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:34.953 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 1 Processor -- start 2024-09-16T14:42:34.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 1 -- start start 2024-09-16T14:42:34.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient: init 2024-09-16T14:42:34.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:34.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:42:34.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient(hunting): picked mon.noname-b con 0x7f57040a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:34.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient(hunting): picked mon.noname-a con 0x7f57040a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:34.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient(hunting): picked mon.noname-c con 0x7f57040a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:34.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5704134490 con 0x7f57040a1d10 2024-09-16T14:42:34.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57040a6280 con 0x7f57040a0000 2024-09-16T14:42:34.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5704135830 con 0x7f57040a2800 2024-09-16T14:42:34.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:34.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5708c57640 10 monclient(hunting): authenticate will time out at 979.063794s 2024-09-16T14:42:34.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5701d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f57040a1d10 0x7f57041ce230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57646/0 (socket says 172.21.15.26:57646) 2024-09-16T14:42:34.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.940+0000 7f5701d9b640 1 -- 172.21.15.26:0/30044145 learned_addr learned my addr 172.21.15.26:0/30044145 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:34.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618138580 0 0) 0x7f5704135830 con 0x7f57040a2800 2024-09-16T14:42:34.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): my global_id is 4355 2024-09-16T14:42:34.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56e4003530 con 0x7f57040a2800 2024-09-16T14:42:34.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932286607 0 0) 0x7f5704134490 con 0x7f57040a1d10 2024-09-16T14:42:34.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): my global_id is 4368 2024-09-16T14:42:34.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5704135830 con 0x7f57040a1d10 2024-09-16T14:42:34.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3991894761 0 0) 0x7f56e4003530 con 0x7f57040a2800 2024-09-16T14:42:34.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5704134490 con 0x7f57040a2800 2024-09-16T14:42:34.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f56fc001480 con 0x7f57040a2800 2024-09-16T14:42:34.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:34.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:34.955 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.955 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.955 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.955 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.955 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.956 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.956 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.956 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.956 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.956 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:34.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3804708014 0 0) 0x7f5704135830 con 0x7f57040a1d10 2024-09-16T14:42:34.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56e4003530 con 0x7f57040a1d10 2024-09-16T14:42:34.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f56f80015e0 con 0x7f57040a1d10 2024-09-16T14:42:34.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:34.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:34.956 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.957 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.957 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.957 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.957 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.957 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.957 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.957 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.957 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.957 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773171765 0 0) 0x7f57040a6280 con 0x7f57040a0000 2024-09-16T14:42:34.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:34.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:34.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:34.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient(hunting): my global_id is 4378 2024-09-16T14:42:34.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5704135830 con 0x7f57040a0000 2024-09-16T14:42:34.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3787690653 0 0) 0x7f5704134490 con 0x7f57040a2800 2024-09-16T14:42:34.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 >> v1:172.21.15.189:6789/0 conn(0x7f57040a0000 legacy=0x7f570413d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 >> v1:172.21.15.26:6789/0 conn(0x7f57040a1d10 legacy=0x7f57041ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:34.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:34.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 monclient: found mon.c 2024-09-16T14:42:34.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:34.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57040a67b0 con 0x7f57040a2800 2024-09-16T14:42:34.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:34.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:34.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:34.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f5708c57640 5 monclient: authenticate success, global_id 4355 2024-09-16T14:42:34.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f5708c57640 10 monclient: _renew_subs 2024-09-16T14:42:34.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f5708c57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f5708c57640 1 -- 172.21.15.26:0/30044145 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570409e500 con 0x7f57040a2800 2024-09-16T14:42:34.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f5708c57640 10 monclient: _renew_subs 2024-09-16T14:42:34.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f5708c57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f5708c57640 1 -- 172.21.15.26:0/30044145 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5704134660 con 0x7f57040a2800 2024-09-16T14:42:34.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56fc004b70 con 0x7f57040a2800 2024-09-16T14:42:34.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:34.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f56fc004e80 con 0x7f57040a2800 2024-09-16T14:42:34.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:34.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:34.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 10 monclient: dump: 2024-09-16T14:42:34.959 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:34.960 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:34.960 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.960 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:34.960 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:34.960 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:34.960 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:34.960 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:34.960 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:34.960 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:34.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f5700d99640 4 set_mon_vals no callback set 2024-09-16T14:42:34.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f56fc00f8f0 con 0x7f57040a2800 2024-09-16T14:42:34.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f5708c57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:34.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f5708c57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:34.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f5708c57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:34.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f5708c57640 1 -- 172.21.15.26:0/30044145 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e8005050 con 0x7f57040a2800 2024-09-16T14:42:34.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f56fc0433a0 con 0x7f57040a2800 2024-09-16T14:42:34.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.944+0000 7f570259c640 10 monclient: get_auth_request con 0x7f56e4036440 auth_method 0 2024-09-16T14:42:34.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.948+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f56fc0148f0 con 0x7f57040a2800 2024-09-16T14:42:34.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.948+0000 7f56f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:34.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:34.948+0000 7f56f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:35.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.104+0000 7f5708c57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:35.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.104+0000 7f5708c57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:35.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.104+0000 7f5708c57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.104+0000 7f5708c57640 1 -- 172.21.15.26:0/30044145 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56e8005300 con 0x7f57040a2800 2024-09-16T14:42:35.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.104+0000 7f56f2ffd640 1 -- 172.21.15.26:0/30044145 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+10842 (unknown 4221056075 0 2316752284) 0x7f56fc012d60 con 0x7f57040a2800 2024-09-16T14:42:35.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.104+0000 7f56f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T14:42:35.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.104+0000 7f56f2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:35.106 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:42:35.106 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":15,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","created":"2024-09-16T14:42:17.361142+0000","modified":"2024-09-16T14:42:30.562817+0000","last_up_change":"2024-09-16T14:42:26.444878+0000","last_in_change":"2024-09-16T14:42:19.053975+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-16T14:42:27.380911+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-16T14:42:28.450753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a4578188-6e8b-430f-b70d-1c3480eb7bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6808","nonce":3182580250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6809","nonce":3182580250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6811","nonce":3182580250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6810","nonce":3182580250}]},"public_addr":"172.21.15.26:6808/3182580250","cluster_addr":"172.21.15.26:6809/3182580250","heartbeat_back_addr":"172.21.15.26:6811/3182580250","heartbeat_front_addr":"172.21.15.26:6810/3182580250","state":["exists","up"]},{"osd":1,"uuid":"1a96367f-73f1-4a99-9b85-47bf8e3d4cdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6800","nonce":3032704211}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6801","nonce":3032704211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6805","nonce":3032704211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6803","nonce":3032704211}]},"public_addr":"172.21.15.26:6800/3032704211","cluster_addr":"172.21.15.26:6801/3032704211","heartbeat_back_addr":"172.21.15.26:6805/3032704211","heartbeat_front_addr":"172.21.15.26:6803/3032704211","state":["exists","up"]},{"osd":2,"uuid":"3030f43d-f848-4189-be1e-6ea367941218","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6802","nonce":1811454603}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6804","nonce":1811454603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6807","nonce":1811454603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6806","nonce":1811454603}]},"public_addr":"172.21.15.26:6802/1811454603","cluster_addr":"172.21.15.26:6804/1811454603","heartbeat_back_addr":"172.21.15.26:6807/1811454603","heartbeat_front_addr":"172.21.15.26:6806/1811454603","state":["exists","up"]},{"osd":3,"uuid":"78731cf9-4ba7-4ce9-a483-080aa1136b65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6808","nonce":3651837298}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6809","nonce":3651837298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6811","nonce":3651837298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6810","nonce":3651837298}]},"public_addr":"172.21.15.189:6808/3651837298","cluster_addr":"172.21.15.189:6809/3651837298","heartbeat_back_addr":"172.21.15.189:6811/3651837298","heartbeat_front_addr":"172.21.15.189:6810/3651837298","state":["exists","up"]},{"osd":4,"uuid":"d74156df-be8d-4ad7-93dc-89ac28df70dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6800","nonce":1782891661}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6801","nonce":1782891661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6803","nonce":1782891661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6802","nonce":1782891661}]},"public_addr":"172.21.15.189:6800/1782891661","cluster_addr":"172.21.15.189:6801/1782891661","heartbeat_back_addr":"172.21.15.189:6803/1782891661","heartbeat_front_addr":"172.21.15.189:6802/1782891661","state":["exists","up"]},{"osd":5,"uuid":"ccad9c1e-247f-4ed4-8437-6aa5c8b515d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6804","nonce":3793314587}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6805","nonce":3793314587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6807","nonce":3793314587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6806","nonce":3793314587}]},"public_addr":"172.21.15.189:6804/3793314587","cluster_addr":"172.21.15.189:6805/3793314587","heartbeat_back_addr":"172.21.15.189:6807/3793314587","heartbeat_front_addr":"172.21.15.189:6806/3793314587","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.603081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.576634+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.472343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.602340+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.465228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.494237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T14:42:35.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.104+0000 7f5708c57640 1 -- 172.21.15.26:0/30044145 >> v1:172.21.15.189:6812/3196161582 conn(0x7f56e4036440 legacy=0x7f56e4038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.104+0000 7f5708c57640 10 monclient: shutdown 2024-09-16T14:42:35.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.104+0000 7f5708c57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:35.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.104+0000 7f5708c57640 1 -- 172.21.15.26:0/30044145 >> v1:172.21.15.26:6790/0 conn(0x7f57040a2800 legacy=0x7f57041d1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.104+0000 7f5708c57640 1 -- 172.21.15.26:0/30044145 shutdown_connections 2024-09-16T14:42:35.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.104+0000 7f5708c57640 1 -- 172.21.15.26:0/30044145 >> 172.21.15.26:0/30044145 conn(0x7f57040a7f30 msgr2=0x7f570406b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:35.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.108+0000 7f5708c57640 1 -- 172.21.15.26:0/30044145 shutdown_connections 2024-09-16T14:42:35.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.108+0000 7f5708c57640 1 -- 172.21.15.26:0/30044145 wait complete. 2024-09-16T14:42:35.123 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-16T14:42:27.380911+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 15, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-09-16T14:42:28.450753+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T14:42:35.124 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-16T14:42:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.236+0000 7ff79f66b640 10 monclient: get_monmap_and_config 2024-09-16T14:42:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.236+0000 7ff79f66b640 10 monclient: build_initial_monmap 2024-09-16T14:42:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.236+0000 7ff79f66b640 10 monclient: monmap: 2024-09-16T14:42:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:35.240991+0000 2024-09-16T14:42:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:35.240991+0000 2024-09-16T14:42:35.241 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:35.241 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.241 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:35.241 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:35.241 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:35.241 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 1 Processor -- start 2024-09-16T14:42:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 1 -- start start 2024-09-16T14:42:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 10 monclient: init 2024-09-16T14:42:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:42:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 10 monclient(hunting): picked mon.noname-a con 0x7ff7980a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 10 monclient(hunting): picked mon.noname-c con 0x7ff7980a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:35.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 10 monclient(hunting): picked mon.noname-b con 0x7ff7980a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7980a3250 con 0x7ff7980a1d10 2024-09-16T14:42:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff79809f600 con 0x7ff7980a0000 2024-09-16T14:42:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7980a3830 con 0x7ff7980a2800 2024-09-16T14:42:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 10 monclient(hunting): authenticate will time out at 979.361104s 2024-09-16T14:42:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79dc08640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff7980a2800 0x7ff79813d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48710/0 (socket says 172.21.15.26:48710) 2024-09-16T14:42:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79dc08640 1 -- 172.21.15.26:0/2741790056 learned_addr learned my addr 172.21.15.26:0/2741790056 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 1 -- 172.21.15.26:0/2741790056 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268234983 0 0) 0x7ff7980a3830 con 0x7ff7980a2800 2024-09-16T14:42:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:35.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient(hunting): my global_id is 4358 2024-09-16T14:42:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 1 -- 172.21.15.26:0/2741790056 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff784003530 con 0x7ff7980a2800 2024-09-16T14:42:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 1 -- 172.21.15.26:0/2741790056 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281043190 0 0) 0x7ff7980a3250 con 0x7ff7980a1d10 2024-09-16T14:42:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient(hunting): my global_id is 4371 2024-09-16T14:42:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 1 -- 172.21.15.26:0/2741790056 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7980a3830 con 0x7ff7980a1d10 2024-09-16T14:42:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 1 -- 172.21.15.26:0/2741790056 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092724725 0 0) 0x7ff79809f600 con 0x7ff7980a0000 2024-09-16T14:42:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:42:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient(hunting): my global_id is 4381 2024-09-16T14:42:35.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 1 -- 172.21.15.26:0/2741790056 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7980a3250 con 0x7ff7980a0000 2024-09-16T14:42:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 1 -- 172.21.15.26:0/2741790056 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2420610922 0 0) 0x7ff784003530 con 0x7ff7980a2800 2024-09-16T14:42:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 1 -- 172.21.15.26:0/2741790056 >> v1:172.21.15.189:6789/0 conn(0x7ff7980a0000 legacy=0x7ff798139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 1 -- 172.21.15.26:0/2741790056 >> v1:172.21.15.26:6789/0 conn(0x7ff7980a1d10 legacy=0x7ff7980a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:42:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 1 monclient: found mon.noname-b 2024-09-16T14:42:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 1 -- 172.21.15.26:0/2741790056 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79809ed60 con 0x7ff7980a2800 2024-09-16T14:42:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient: _finish_auth 0 2024-09-16T14:42:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:42:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 1 -- 172.21.15.26:0/2741790056 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff790001620 con 0x7ff7980a2800 2024-09-16T14:42:35.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient: dump: 2024-09-16T14:42:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:35.251 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 5 monclient: authenticate success, global_id 4358 2024-09-16T14:42:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 1 -- 172.21.15.26:0/2741790056 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff790001840 con 0x7ff7980a2800 2024-09-16T14:42:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 1 -- 172.21.15.26:0/2741790056 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff790001bd0 con 0x7ff7980a2800 2024-09-16T14:42:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff78ffff640 10 monclient: dump: 2024-09-16T14:42:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:35.252 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.253 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:35.253 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:35.253 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:35.253 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:35.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 4 set_mon_vals no callback set 2024-09-16T14:42:35.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 10 monclient: shutdown 2024-09-16T14:42:35.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:35.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 1 -- 172.21.15.26:0/2741790056 >> v1:172.21.15.26:6790/0 conn(0x7ff7980a2800 legacy=0x7ff79813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 1 -- 172.21.15.26:0/2741790056 shutdown_connections 2024-09-16T14:42:35.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 1 -- 172.21.15.26:0/2741790056 >> 172.21.15.26:0/2741790056 conn(0x7ff7980a7f30 msgr2=0x7ff7980a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:35.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 1 -- 172.21.15.26:0/2741790056 shutdown_connections 2024-09-16T14:42:35.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 1 -- 172.21.15.26:0/2741790056 wait complete. 2024-09-16T14:42:35.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 10 monclient: build_initial_monmap 2024-09-16T14:42:35.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 10 monclient: monmap: 2024-09-16T14:42:35.254 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:35.254 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:35.254 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:35.254 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:35.254 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:35.254 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:35.254 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:35.254 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:35.254 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:35.254 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 1 Processor -- start 2024-09-16T14:42:35.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 1 -- start start 2024-09-16T14:42:35.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.240+0000 7ff79f66b640 10 monclient: init 2024-09-16T14:42:35.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:35.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:35.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 10 monclient(hunting): picked mon.noname-b con 0x7ff7980a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:35.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 10 monclient(hunting): picked mon.noname-c con 0x7ff7980a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:35.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 10 monclient(hunting): picked mon.noname-a con 0x7ff7980a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:35.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7981320c0 con 0x7ff7980a2800 2024-09-16T14:42:35.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7981311b0 con 0x7ff7980a0000 2024-09-16T14:42:35.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff798133ec0 con 0x7ff7980a1d10 2024-09-16T14:42:35.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:35.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 10 monclient(hunting): authenticate will time out at 979.364244s 2024-09-16T14:42:35.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79cc06640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff7980a1d10 0x7ff798130aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48718/0 (socket says 172.21.15.26:48718) 2024-09-16T14:42:35.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79cc06640 1 -- 172.21.15.26:0/260675013 learned_addr learned my addr 172.21.15.26:0/260675013 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:35.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125578904 0 0) 0x7ff798133ec0 con 0x7ff7980a1d10 2024-09-16T14:42:35.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:35.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:35.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:35.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): my global_id is 4361 2024-09-16T14:42:35.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff77c003530 con 0x7ff7980a1d10 2024-09-16T14:42:35.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274663475 0 0) 0x7ff7981320c0 con 0x7ff7980a2800 2024-09-16T14:42:35.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:35.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:35.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:35.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): my global_id is 4374 2024-09-16T14:42:35.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff798133ec0 con 0x7ff7980a2800 2024-09-16T14:42:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 355014741 0 0) 0x7ff77c003530 con 0x7ff7980a1d10 2024-09-16T14:42:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7981320c0 con 0x7ff7980a1d10 2024-09-16T14:42:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff788001900 con 0x7ff7980a1d10 2024-09-16T14:42:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): dump: 2024-09-16T14:42:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:35.258 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147917674 0 0) 0x7ff7981311b0 con 0x7ff7980a0000 2024-09-16T14:42:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): my global_id is 4384 2024-09-16T14:42:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff77c003530 con 0x7ff7980a0000 2024-09-16T14:42:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2663758028 0 0) 0x7ff798133ec0 con 0x7ff7980a2800 2024-09-16T14:42:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7981311b0 con 0x7ff7980a2800 2024-09-16T14:42:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff790001520 con 0x7ff7980a2800 2024-09-16T14:42:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient(hunting): dump: 2024-09-16T14:42:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:35.259 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4208936210 0 0) 0x7ff7981320c0 con 0x7ff7980a1d10 2024-09-16T14:42:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 >> v1:172.21.15.189:6789/0 conn(0x7ff7980a0000 legacy=0x7ff798130390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 >> v1:172.21.15.26:6789/0 conn(0x7ff7980a2800 legacy=0x7ff7981d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:42:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 monclient: found mon.c 2024-09-16T14:42:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff798133be0 con 0x7ff7980a1d10 2024-09-16T14:42:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient: _finish_auth 0 2024-09-16T14:42:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:42:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 5 monclient: authenticate success, global_id 4361 2024-09-16T14:42:35.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 10 monclient: _renew_subs 2024-09-16T14:42:35.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 1 -- 172.21.15.26:0/260675013 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff798133900 con 0x7ff7980a1d10 2024-09-16T14:42:35.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 10 monclient: _renew_subs 2024-09-16T14:42:35.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 1 -- 172.21.15.26:0/260675013 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7981356b0 con 0x7ff7980a1d10 2024-09-16T14:42:35.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff788003620 con 0x7ff7980a1d10 2024-09-16T14:42:35.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:35.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff7880039b0 con 0x7ff7980a1d10 2024-09-16T14:42:35.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:35.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:35.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78f7fe640 4 set_mon_vals no callback set 2024-09-16T14:42:35.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 10 monclient: dump: 2024-09-16T14:42:35.261 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:35.262 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.262 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.262 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.262 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:35.262 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.262 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:35.262 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:35.262 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:35.262 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:35.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:35.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79f66b640 1 -- 172.21.15.26:0/260675013 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff778005050 con 0x7ff7980a1d10 2024-09-16T14:42:35.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff788003b90 con 0x7ff7980a1d10 2024-09-16T14:42:35.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7ff788043170 con 0x7ff7980a1d10 2024-09-16T14:42:35.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.244+0000 7ff79d407640 10 monclient: get_auth_request con 0x7ff77c036440 auth_method 0 2024-09-16T14:42:35.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.248+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff788014740 con 0x7ff7980a1d10 2024-09-16T14:42:35.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.248+0000 7ff78dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:35.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.248+0000 7ff78dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.408+0000 7ff79f66b640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T14:42:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.408+0000 7ff79f66b640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T14:42:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.408+0000 7ff79f66b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.408+0000 7ff79f66b640 1 -- 172.21.15.26:0/260675013 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff778005880 con 0x7ff7980a1d10 2024-09-16T14:42:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.408+0000 7ff78dffb640 1 -- 172.21.15.26:0/260675013 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) v1 ==== 92+0+10 (unknown 946793192 0 265718887) 0x7ff788012bb0 con 0x7ff7980a1d10 2024-09-16T14:42:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.408+0000 7ff78dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T14:42:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.408+0000 7ff78dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:35.410 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2024-09-16T14:42:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.408+0000 7ff79f66b640 1 -- 172.21.15.26:0/260675013 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff77c036440 legacy=0x7ff77c038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.408+0000 7ff79f66b640 10 monclient: shutdown 2024-09-16T14:42:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.408+0000 7ff79f66b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.408+0000 7ff79f66b640 1 -- 172.21.15.26:0/260675013 >> v1:172.21.15.26:6790/0 conn(0x7ff7980a1d10 legacy=0x7ff798130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.408+0000 7ff79f66b640 1 -- 172.21.15.26:0/260675013 shutdown_connections 2024-09-16T14:42:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.408+0000 7ff79f66b640 1 -- 172.21.15.26:0/260675013 >> 172.21.15.26:0/260675013 conn(0x7ff7980a7f30 msgr2=0x7ff79806b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:35.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.408+0000 7ff79f66b640 1 -- 172.21.15.26:0/260675013 shutdown_connections 2024-09-16T14:42:35.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.408+0000 7ff79f66b640 1 -- 172.21.15.26:0/260675013 wait complete. 2024-09-16T14:42:35.428 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T14:42:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.540+0000 7fe72a9f6640 10 monclient: get_monmap_and_config 2024-09-16T14:42:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.540+0000 7fe72a9f6640 10 monclient: build_initial_monmap 2024-09-16T14:42:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.540+0000 7fe72a9f6640 10 monclient: monmap: 2024-09-16T14:42:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:35.544124+0000 2024-09-16T14:42:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:35.544124+0000 2024-09-16T14:42:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:35.544 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.540+0000 7fe72a9f6640 1 Processor -- start 2024-09-16T14:42:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 1 -- start start 2024-09-16T14:42:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient: init 2024-09-16T14:42:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:35.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:42:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe724070fa0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe724131630 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe72413b130 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72409d2b0 con 0x7fe724131630 2024-09-16T14:42:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe724073ab0 con 0x7fe724070fa0 2024-09-16T14:42:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72406d0b0 con 0x7fe72413b130 2024-09-16T14:42:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient(hunting): authenticate will time out at 979.664775s 2024-09-16T14:42:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe71bfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe724131630 0x7fe724139a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57664/0 (socket says 172.21.15.26:57664) 2024-09-16T14:42:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe71bfff640 1 -- 172.21.15.26:0/2297667926 learned_addr learned my addr 172.21.15.26:0/2297667926 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 1 -- 172.21.15.26:0/2297667926 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036885630 0 0) 0x7fe72409d2b0 con 0x7fe724131630 2024-09-16T14:42:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient(hunting): my global_id is 4377 2024-09-16T14:42:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 1 -- 172.21.15.26:0/2297667926 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe710003530 con 0x7fe724131630 2024-09-16T14:42:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 1 -- 172.21.15.26:0/2297667926 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165284139 0 0) 0x7fe72406d0b0 con 0x7fe72413b130 2024-09-16T14:42:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient(hunting): my global_id is 4364 2024-09-16T14:42:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 1 -- 172.21.15.26:0/2297667926 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe72409d2b0 con 0x7fe72413b130 2024-09-16T14:42:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 1 -- 172.21.15.26:0/2297667926 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2226972286 0 0) 0x7fe710003530 con 0x7fe724131630 2024-09-16T14:42:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 1 -- 172.21.15.26:0/2297667926 >> v1:172.21.15.26:6790/0 conn(0x7fe72413b130 legacy=0x7fe72413d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 1 -- 172.21.15.26:0/2297667926 >> v1:172.21.15.189:6789/0 conn(0x7fe724070fa0 legacy=0x7fe724071380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient: _finish_hunting 0 2024-09-16T14:42:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 1 monclient: found mon.noname-a 2024-09-16T14:42:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:42:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 1 -- 172.21.15.26:0/2297667926 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72406d390 con 0x7fe724131630 2024-09-16T14:42:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient: _finish_auth 0 2024-09-16T14:42:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient: _check_auth_tickets 2024-09-16T14:42:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 1 -- 172.21.15.26:0/2297667926 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe70c001620 con 0x7fe724131630 2024-09-16T14:42:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient: dump: 2024-09-16T14:42:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:35.554 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 5 monclient: authenticate success, global_id 4377 2024-09-16T14:42:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 1 -- 172.21.15.26:0/2297667926 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe70c0018c0 con 0x7fe724131630 2024-09-16T14:42:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 1 -- 172.21.15.26:0/2297667926 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe70c001c90 con 0x7fe724131630 2024-09-16T14:42:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7237fe640 10 monclient: dump: 2024-09-16T14:42:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:35.557 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 4 set_mon_vals no callback set 2024-09-16T14:42:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient: shutdown 2024-09-16T14:42:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 1 -- 172.21.15.26:0/2297667926 >> v1:172.21.15.26:6789/0 conn(0x7fe724131630 legacy=0x7fe724139a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 1 -- 172.21.15.26:0/2297667926 shutdown_connections 2024-09-16T14:42:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 1 -- 172.21.15.26:0/2297667926 >> 172.21.15.26:0/2297667926 conn(0x7fe7240a03d0 msgr2=0x7fe7240a07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 1 -- 172.21.15.26:0/2297667926 shutdown_connections 2024-09-16T14:42:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 1 -- 172.21.15.26:0/2297667926 wait complete. 2024-09-16T14:42:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient: build_initial_monmap 2024-09-16T14:42:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient: monmap: 2024-09-16T14:42:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:35.559 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 1 Processor -- start 2024-09-16T14:42:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 1 -- start start 2024-09-16T14:42:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient: init 2024-09-16T14:42:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe724070fa0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe724131630 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe72413b130 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7241380c0 con 0x7fe72413b130 2024-09-16T14:42:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72413e860 con 0x7fe724131630 2024-09-16T14:42:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7240720c0 con 0x7fe724070fa0 2024-09-16T14:42:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe72a9f6640 10 monclient(hunting): authenticate will time out at 979.667596s 2024-09-16T14:42:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe723fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe724070fa0 0x7fe7241c61d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48730/0 (socket says 172.21.15.26:48730) 2024-09-16T14:42:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe723fff640 1 -- 172.21.15.26:0/3124932091 learned_addr learned my addr 172.21.15.26:0/3124932091 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:35.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300898192 0 0) 0x7fe7240720c0 con 0x7fe724070fa0 2024-09-16T14:42:35.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7217fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:35.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:35.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:35.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7217fa640 10 monclient(hunting): my global_id is 4367 2024-09-16T14:42:35.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe704003530 con 0x7fe724070fa0 2024-09-16T14:42:35.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015789066 0 0) 0x7fe7241380c0 con 0x7fe72413b130 2024-09-16T14:42:35.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7217fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:35.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:35.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:35.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7217fa640 10 monclient(hunting): my global_id is 4380 2024-09-16T14:42:35.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.544+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7240720c0 con 0x7fe72413b130 2024-09-16T14:42:35.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 496082064 0 0) 0x7fe704003530 con 0x7fe724070fa0 2024-09-16T14:42:35.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7241380c0 con 0x7fe724070fa0 2024-09-16T14:42:35.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe714001980 con 0x7fe724070fa0 2024-09-16T14:42:35.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:35.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:35.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient(hunting): dump: 2024-09-16T14:42:35.565 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:35.565 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.565 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.565 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.565 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:35.565 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.565 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:35.565 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:35.566 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:35.566 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:35.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2141606243 0 0) 0x7fe7240720c0 con 0x7fe72413b130 2024-09-16T14:42:35.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe704003530 con 0x7fe72413b130 2024-09-16T14:42:35.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe71c001480 con 0x7fe72413b130 2024-09-16T14:42:35.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:35.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:35.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient(hunting): dump: 2024-09-16T14:42:35.566 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:35.566 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.566 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.566 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.566 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:35.567 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146152901 0 0) 0x7fe72413e860 con 0x7fe724131630 2024-09-16T14:42:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient(hunting): my global_id is 4390 2024-09-16T14:42:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7240720c0 con 0x7fe724131630 2024-09-16T14:42:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2736174626 0 0) 0x7fe7241380c0 con 0x7fe724070fa0 2024-09-16T14:42:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 >> v1:172.21.15.189:6789/0 conn(0x7fe724131630 legacy=0x7fe7241ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 >> v1:172.21.15.26:6789/0 conn(0x7fe72413b130 legacy=0x7fe7241d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient: _finish_hunting 0 2024-09-16T14:42:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 monclient: found mon.c 2024-09-16T14:42:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe724138680 con 0x7fe724070fa0 2024-09-16T14:42:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient: _finish_auth 0 2024-09-16T14:42:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient: _check_auth_tickets 2024-09-16T14:42:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe72a9f6640 5 monclient: authenticate success, global_id 4367 2024-09-16T14:42:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe72a9f6640 10 monclient: _renew_subs 2024-09-16T14:42:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe72a9f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe72a9f6640 1 -- 172.21.15.26:0/3124932091 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe724138960 con 0x7fe724070fa0 2024-09-16T14:42:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe72a9f6640 10 monclient: _renew_subs 2024-09-16T14:42:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe72a9f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe72a9f6640 1 -- 172.21.15.26:0/3124932091 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe724137160 con 0x7fe724070fa0 2024-09-16T14:42:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe714003660 con 0x7fe724070fa0 2024-09-16T14:42:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe722ffd640 4 set_mon_vals no callback set 2024-09-16T14:42:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe7140039f0 con 0x7fe724070fa0 2024-09-16T14:42:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 10 monclient: dump: 2024-09-16T14:42:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:35.570 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe714003bd0 con 0x7fe724070fa0 2024-09-16T14:42:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe72a9f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe72a9f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe72a9f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe72a9f6640 1 -- 172.21.15.26:0/3124932091 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe708005050 con 0x7fe724070fa0 2024-09-16T14:42:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7fe7140431c0 con 0x7fe724070fa0 2024-09-16T14:42:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.548+0000 7fe71bfff640 10 monclient: get_auth_request con 0x7fe704036360 auth_method 0 2024-09-16T14:42:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.552+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe714014850 con 0x7fe724070fa0 2024-09-16T14:42:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.552+0000 7fe7217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.552+0000 7fe7217fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:35.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.712+0000 7fe72a9f6640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T14:42:35.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.712+0000 7fe72a9f6640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T14:42:35.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.712+0000 7fe72a9f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.712+0000 7fe72a9f6640 1 -- 172.21.15.26:0/3124932091 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe708005880 con 0x7fe724070fa0 2024-09-16T14:42:35.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.712+0000 7fe7217fa640 1 -- 172.21.15.26:0/3124932091 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v15) v1 ==== 93+0+10 (unknown 554344602 0 2170607528) 0x7fe714014440 con 0x7fe724070fa0 2024-09-16T14:42:35.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.712+0000 7fe7217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T14:42:35.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.712+0000 7fe7217fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:35.715 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-09-16T14:42:35.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.716+0000 7fe72a9f6640 1 -- 172.21.15.26:0/3124932091 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe704036360 legacy=0x7fe704038820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.716+0000 7fe72a9f6640 10 monclient: shutdown 2024-09-16T14:42:35.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.716+0000 7fe72a9f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:35.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.716+0000 7fe72a9f6640 1 -- 172.21.15.26:0/3124932091 >> v1:172.21.15.26:6790/0 conn(0x7fe724070fa0 legacy=0x7fe7241c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.716+0000 7fe72a9f6640 1 -- 172.21.15.26:0/3124932091 shutdown_connections 2024-09-16T14:42:35.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.716+0000 7fe72a9f6640 1 -- 172.21.15.26:0/3124932091 >> 172.21.15.26:0/3124932091 conn(0x7fe7240a03d0 msgr2=0x7fe7240700c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:35.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.716+0000 7fe72a9f6640 1 -- 172.21.15.26:0/3124932091 shutdown_connections 2024-09-16T14:42:35.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.716+0000 7fe72a9f6640 1 -- 172.21.15.26:0/3124932091 wait complete. 2024-09-16T14:42:35.732 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-16T14:42:35.732 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T14:42:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.848+0000 7f92c573d640 10 monclient: get_monmap_and_config 2024-09-16T14:42:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.848+0000 7f92c573d640 10 monclient: build_initial_monmap 2024-09-16T14:42:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.848+0000 7f92c573d640 10 monclient: monmap: 2024-09-16T14:42:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:35.852207+0000 2024-09-16T14:42:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:35.852207+0000 2024-09-16T14:42:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:35.852 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.848+0000 7f92c573d640 1 Processor -- start 2024-09-16T14:42:35.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.848+0000 7f92c573d640 1 -- start start 2024-09-16T14:42:35.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.848+0000 7f92c573d640 10 monclient: init 2024-09-16T14:42:35.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.848+0000 7f92c573d640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:35.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.848+0000 7f92c573d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:35.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.848+0000 7f92c573d640 10 monclient(hunting): picked mon.noname-c con 0x7f92c009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:35.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.848+0000 7f92c573d640 10 monclient(hunting): picked mon.noname-b con 0x7f92c0132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:35.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.848+0000 7f92c573d640 10 monclient(hunting): picked mon.noname-a con 0x7f92c0138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:35.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.848+0000 7f92c573d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c009d260 con 0x7f92c0138ff0 2024-09-16T14:42:35.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.848+0000 7f92c573d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c0132a80 con 0x7f92c009cb80 2024-09-16T14:42:35.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.848+0000 7f92c573d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c009cf60 con 0x7f92c0132650 2024-09-16T14:42:35.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.848+0000 7f92c573d640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:35.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.848+0000 7f92c573d640 10 monclient(hunting): authenticate will time out at 979.971951s 2024-09-16T14:42:35.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bf7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f92c0138ff0 0x7f92c013b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57684/0 (socket says 172.21.15.26:57684) 2024-09-16T14:42:35.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bf7fe640 1 -- 172.21.15.26:0/1115981014 learned_addr learned my addr 172.21.15.26:0/1115981014 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:35.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 1 -- 172.21.15.26:0/1115981014 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208811295 0 0) 0x7f92c009d260 con 0x7f92c0138ff0 2024-09-16T14:42:35.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:35.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:35.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:35.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient(hunting): my global_id is 4383 2024-09-16T14:42:35.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 1 -- 172.21.15.26:0/1115981014 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92ac003530 con 0x7f92c0138ff0 2024-09-16T14:42:35.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 1 -- 172.21.15.26:0/1115981014 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621430199 0 0) 0x7f92c009cf60 con 0x7f92c0132650 2024-09-16T14:42:35.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient(hunting): _negotiate 2024-09-16T14:42:35.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:35.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:35.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient(hunting): my global_id is 4370 2024-09-16T14:42:35.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 1 -- 172.21.15.26:0/1115981014 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c009d260 con 0x7f92c0132650 2024-09-16T14:42:35.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 1 -- 172.21.15.26:0/1115981014 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 397624756 0 0) 0x7f92c009d260 con 0x7f92c0132650 2024-09-16T14:42:35.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 1 -- 172.21.15.26:0/1115981014 >> v1:172.21.15.189:6789/0 conn(0x7f92c009cb80 legacy=0x7f92c00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 1 -- 172.21.15.26:0/1115981014 >> v1:172.21.15.26:6789/0 conn(0x7f92c0138ff0 legacy=0x7f92c013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient: _finish_hunting 0 2024-09-16T14:42:35.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 1 monclient: found mon.noname-b 2024-09-16T14:42:35.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:35.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 1 -- 172.21.15.26:0/1115981014 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c009f2d0 con 0x7f92c0132650 2024-09-16T14:42:35.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient: _finish_auth 0 2024-09-16T14:42:35.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient: _check_auth_tickets 2024-09-16T14:42:35.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:35.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 1 -- 172.21.15.26:0/1115981014 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f92a80015c0 con 0x7f92c0132650 2024-09-16T14:42:35.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:35.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:35.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient: dump: 2024-09-16T14:42:35.862 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:35.862 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.862 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.862 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.862 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:35.862 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.862 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:35.862 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:35.862 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:35.862 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 5 monclient: authenticate success, global_id 4370 2024-09-16T14:42:35.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:35.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 1 -- 172.21.15.26:0/1115981014 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92a8001760 con 0x7f92c0132650 2024-09-16T14:42:35.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:35.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 1 -- 172.21.15.26:0/1115981014 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f92a8001af0 con 0x7f92c0132650 2024-09-16T14:42:35.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:35.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:35.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92bdffb640 10 monclient: dump: 2024-09-16T14:42:35.863 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:35.863 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.863 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.863 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.863 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:35.863 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.864 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:35.864 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:35.864 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:35.864 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:35.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 4 set_mon_vals no callback set 2024-09-16T14:42:35.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 10 monclient: shutdown 2024-09-16T14:42:35.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:35.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 1 -- 172.21.15.26:0/1115981014 >> v1:172.21.15.26:6790/0 conn(0x7f92c0132650 legacy=0x7f92c01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 1 -- 172.21.15.26:0/1115981014 shutdown_connections 2024-09-16T14:42:35.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 1 -- 172.21.15.26:0/1115981014 >> 172.21.15.26:0/1115981014 conn(0x7f92c009fb50 msgr2=0x7f92c009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:35.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 1 -- 172.21.15.26:0/1115981014 shutdown_connections 2024-09-16T14:42:35.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 1 -- 172.21.15.26:0/1115981014 wait complete. 2024-09-16T14:42:35.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 10 monclient: build_initial_monmap 2024-09-16T14:42:35.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 10 monclient: monmap: 2024-09-16T14:42:35.865 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:35.865 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:35.865 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:35.865 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:35.865 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:35.865 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:35.865 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:35.865 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:35.865 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:35.865 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 1 Processor -- start 2024-09-16T14:42:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 1 -- start start 2024-09-16T14:42:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 10 monclient: init 2024-09-16T14:42:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:42:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 10 monclient(hunting): picked mon.noname-b con 0x7f92c009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 10 monclient(hunting): picked mon.noname-a con 0x7f92c0132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 10 monclient(hunting): picked mon.noname-c con 0x7f92c0138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c0138150 con 0x7f92c0132650 2024-09-16T14:42:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c006e450 con 0x7f92c009cb80 2024-09-16T14:42:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c0071590 con 0x7f92c0138ff0 2024-09-16T14:42:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 10 monclient(hunting): authenticate will time out at 979.974890s 2024-09-16T14:42:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92be7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f92c0132650 0x7f92c006dd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57698/0 (socket says 172.21.15.26:57698) 2024-09-16T14:42:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92be7fc640 1 -- 172.21.15.26:0/2819495827 learned_addr learned my addr 172.21.15.26:0/2819495827 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744238850 0 0) 0x7f92c009f2d0 con 0x7f92c0132650 2024-09-16T14:42:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): my global_id is 4386 2024-09-16T14:42:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a0003530 con 0x7f92c0132650 2024-09-16T14:42:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113563519 0 0) 0x7f92c0071590 con 0x7f92c0138ff0 2024-09-16T14:42:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:35.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): my global_id is 4373 2024-09-16T14:42:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c009f2d0 con 0x7f92c0138ff0 2024-09-16T14:42:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2182998752 0 0) 0x7f92a0003530 con 0x7f92c0132650 2024-09-16T14:42:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92c0071590 con 0x7f92c0132650 2024-09-16T14:42:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f92a8001420 con 0x7f92c0132650 2024-09-16T14:42:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): dump: 2024-09-16T14:42:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:35.869 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:35.869 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:35.869 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:35.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2017801522 0 0) 0x7f92c009f2d0 con 0x7f92c0138ff0 2024-09-16T14:42:35.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92a0003530 con 0x7f92c0138ff0 2024-09-16T14:42:35.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f92b80015a0 con 0x7f92c0138ff0 2024-09-16T14:42:35.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:35.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:35.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): dump: 2024-09-16T14:42:35.869 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:35.869 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.869 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.870 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.870 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:35.870 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.870 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:35.870 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:35.870 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:35.870 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888859154 0 0) 0x7f92c006e450 con 0x7f92c009cb80 2024-09-16T14:42:35.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:35.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:35.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:35.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient(hunting): my global_id is 4396 2024-09-16T14:42:35.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c009f2d0 con 0x7f92c009cb80 2024-09-16T14:42:35.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2016781213 0 0) 0x7f92a0003530 con 0x7f92c0138ff0 2024-09-16T14:42:35.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 >> v1:172.21.15.189:6789/0 conn(0x7f92c009cb80 legacy=0x7f92c006d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 >> v1:172.21.15.26:6789/0 conn(0x7f92c0132650 legacy=0x7f92c006dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:35.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:42:35.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 monclient: found mon.c 2024-09-16T14:42:35.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:35.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c013c400 con 0x7f92c0138ff0 2024-09-16T14:42:35.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient: _finish_auth 0 2024-09-16T14:42:35.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:42:35.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:35.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 5 monclient: authenticate success, global_id 4373 2024-09-16T14:42:35.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 10 monclient: _renew_subs 2024-09-16T14:42:35.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 1 -- 172.21.15.26:0/2819495827 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c0070fd0 con 0x7f92c0138ff0 2024-09-16T14:42:35.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 10 monclient: _renew_subs 2024-09-16T14:42:35.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92c573d640 1 -- 172.21.15.26:0/2819495827 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c006fb80 con 0x7f92c0138ff0 2024-09-16T14:42:35.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92b8004b70 con 0x7f92c0138ff0 2024-09-16T14:42:35.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:35.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f92b8004e80 con 0x7f92c0138ff0 2024-09-16T14:42:35.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:35.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:35.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.852+0000 7f92b77fe640 10 monclient: dump: 2024-09-16T14:42:35.872 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:35.873 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.856+0000 7f92bd7fa640 4 set_mon_vals no callback set 2024-09-16T14:42:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.856+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f92b800f8f0 con 0x7f92c0138ff0 2024-09-16T14:42:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.856+0000 7f92c573d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:35.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.856+0000 7f92c573d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:35.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.856+0000 7f92c573d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:35.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.856+0000 7f92c573d640 1 -- 172.21.15.26:0/2819495827 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a4005050 con 0x7f92c0138ff0 2024-09-16T14:42:35.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.856+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 3562+0+0 (unknown 2394778341 0 0) 0x7f92b80432f0 con 0x7f92c0138ff0 2024-09-16T14:42:35.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.856+0000 7f92beffd640 10 monclient: get_auth_request con 0x7f92a0036380 auth_method 0 2024-09-16T14:42:35.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.860+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f92b8014720 con 0x7f92c0138ff0 2024-09-16T14:42:35.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.860+0000 7f92b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:35.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:35.860+0000 7f92b77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:36.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.012+0000 7f92c573d640 1 -- 172.21.15.26:0/2819495827 --> v1:172.21.15.189:6812/3196161582 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92a4000ba0 con 0x7f92a0036380 2024-09-16T14:42:36.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.016+0000 7f92b77fe640 1 -- 172.21.15.26:0/2819495827 <== mgr.4105 v1:172.21.15.189:6812/3196161582 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41071 (unknown 2965378022 0 2295359421) 0x7f92a4000ba0 con 0x7f92a0036380 2024-09-16T14:42:36.017 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:42:36.018 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T14:42:36.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.016+0000 7f92c573d640 1 -- 172.21.15.26:0/2819495827 >> v1:172.21.15.189:6812/3196161582 conn(0x7f92a0036380 legacy=0x7f92a0038840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.016+0000 7f92c573d640 10 monclient: shutdown 2024-09-16T14:42:36.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.016+0000 7f92c573d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:36.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.016+0000 7f92c573d640 1 -- 172.21.15.26:0/2819495827 >> v1:172.21.15.26:6790/0 conn(0x7f92c0138ff0 legacy=0x7f92c01cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.016+0000 7f92c573d640 1 -- 172.21.15.26:0/2819495827 shutdown_connections 2024-09-16T14:42:36.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.016+0000 7f92c573d640 1 -- 172.21.15.26:0/2819495827 >> 172.21.15.26:0/2819495827 conn(0x7f92c009fb50 msgr2=0x7f92c0131ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:36.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.016+0000 7f92c573d640 1 -- 172.21.15.26:0/2819495827 shutdown_connections 2024-09-16T14:42:36.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.016+0000 7f92c573d640 1 -- 172.21.15.26:0/2819495827 wait complete. 2024-09-16T14:42:36.034 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-16T14:42:34.375713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167124,"kb_used_data":896,"kb_used_omap":9,"kb_used_meta":166134,"kb_avail":566063916,"statfs":{"total":579820584960,"available":579649449984,"internally_reserved":0,"allocated":917504,"data_stored":1290382,"data_compressed":61020,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9559,"internal_metadata":170121897},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000582"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.575415+0000","last_change":"2024-09-16T14:42:30.575415+0000","last_active":"2024-09-16T14:42:30.575415+0000","last_peered":"2024-09-16T14:42:30.575415+0000","last_clean":"2024-09-16T14:42:30.575415+0000","last_became_active":"2024-09-16T14:42:28.533291+0000","last_became_peered":"2024-09-16T14:42:28.533291+0000","last_unstale":"2024-09-16T14:42:30.575415+0000","last_undegraded":"2024-09-16T14:42:30.575415+0000","last_fullsized":"2024-09-16T14:42:30.575415+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:36:22.883054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085482099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.597981+0000","last_change":"2024-09-16T14:42:30.597981+0000","last_active":"2024-09-16T14:42:30.597981+0000","last_peered":"2024-09-16T14:42:30.597981+0000","last_clean":"2024-09-16T14:42:30.597981+0000","last_became_active":"2024-09-16T14:42:28.477522+0000","last_became_peered":"2024-09-16T14:42:28.477522+0000","last_unstale":"2024-09-16T14:42:30.597981+0000","last_undegraded":"2024-09-16T14:42:30.597981+0000","last_fullsized":"2024-09-16T14:42:30.597981+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:00:15.035933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065779100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.580369+0000","last_change":"2024-09-16T14:42:30.580369+0000","last_active":"2024-09-16T14:42:30.580369+0000","last_peered":"2024-09-16T14:42:30.580369+0000","last_clean":"2024-09-16T14:42:30.580369+0000","last_became_active":"2024-09-16T14:42:28.533172+0000","last_became_peered":"2024-09-16T14:42:28.533172+0000","last_unstale":"2024-09-16T14:42:30.580369+0000","last_undegraded":"2024-09-16T14:42:30.580369+0000","last_fullsized":"2024-09-16T14:42:30.580369+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:56:43.741456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061478100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.796631+0000","last_change":"2024-09-16T14:42:30.796631+0000","last_active":"2024-09-16T14:42:30.796631+0000","last_peered":"2024-09-16T14:42:30.796631+0000","last_clean":"2024-09-16T14:42:30.796631+0000","last_became_active":"2024-09-16T14:42:28.476209+0000","last_became_peered":"2024-09-16T14:42:28.476209+0000","last_unstale":"2024-09-16T14:42:30.796631+0000","last_undegraded":"2024-09-16T14:42:30.796631+0000","last_fullsized":"2024-09-16T14:42:30.796631+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:32:31.800477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039796999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"15'117","reported_seq":160,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.819644+0000","last_change":"2024-09-16T14:42:29.481739+0000","last_active":"2024-09-16T14:42:30.819644+0000","last_peered":"2024-09-16T14:42:30.819644+0000","last_clean":"2024-09-16T14:42:30.819644+0000","last_became_active":"2024-09-16T14:42:29.480896+0000","last_became_peered":"2024-09-16T14:42:29.480896+0000","last_unstale":"2024-09-16T14:42:30.819644+0000","last_undegraded":"2024-09-16T14:42:30.819644+0000","last_fullsized":"2024-09-16T14:42:30.819644+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:28.462531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:28.462531+0000","last_clean_scrub_stamp":"2024-09-16T14:42:28.462531+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:32:11.234353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.796643+0000","last_change":"2024-09-16T14:42:30.796643+0000","last_active":"2024-09-16T14:42:30.796643+0000","last_peered":"2024-09-16T14:42:30.796643+0000","last_clean":"2024-09-16T14:42:30.796643+0000","last_became_active":"2024-09-16T14:42:28.488186+0000","last_became_peered":"2024-09-16T14:42:28.488186+0000","last_unstale":"2024-09-16T14:42:30.796643+0000","last_undegraded":"2024-09-16T14:42:30.796643+0000","last_fullsized":"2024-09-16T14:42:30.796643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:40:41.632316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042440799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"15'2","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.582824+0000","last_change":"2024-09-16T14:42:30.581886+0000","last_active":"2024-09-16T14:42:30.582824+0000","last_peered":"2024-09-16T14:42:30.582824+0000","last_clean":"2024-09-16T14:42:30.582824+0000","last_became_active":"2024-09-16T14:42:28.487188+0000","last_became_peered":"2024-09-16T14:42:28.487188+0000","last_unstale":"2024-09-16T14:42:30.582824+0000","last_undegraded":"2024-09-16T14:42:30.582824+0000","last_fullsized":"2024-09-16T14:42:30.582824+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:10:26.171186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020492470000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.575555+0000","last_change":"2024-09-16T14:42:30.575555+0000","last_active":"2024-09-16T14:42:30.575555+0000","last_peered":"2024-09-16T14:42:30.575555+0000","last_clean":"2024-09-16T14:42:30.575555+0000","last_became_active":"2024-09-16T14:42:28.484409+0000","last_became_peered":"2024-09-16T14:42:28.484409+0000","last_unstale":"2024-09-16T14:42:30.575555+0000","last_undegraded":"2024-09-16T14:42:30.575555+0000","last_fullsized":"2024-09-16T14:42:30.575555+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:16:51.119532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097689999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.598002+0000","last_change":"2024-09-16T14:42:30.598002+0000","last_active":"2024-09-16T14:42:30.598002+0000","last_peered":"2024-09-16T14:42:30.598002+0000","last_clean":"2024-09-16T14:42:30.598002+0000","last_became_active":"2024-09-16T14:42:28.477416+0000","last_became_peered":"2024-09-16T14:42:28.477416+0000","last_unstale":"2024-09-16T14:42:30.598002+0000","last_undegraded":"2024-09-16T14:42:30.598002+0000","last_fullsized":"2024-09-16T14:42:30.598002+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:49.076917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067836399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61020,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":38748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":135168,"data_stored":38767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":192512,"data_stored":567676,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":38748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":567676,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31252,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340588,"statfs":{"total":96636764160,"available":96604762112,"internally_reserved":0,"allocated":135168,"data_stored":38767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T14:42:36.035 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T14:42:36.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.144+0000 7fbe16b4c640 10 monclient: get_monmap_and_config 2024-09-16T14:42:36.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.144+0000 7fbe16b4c640 10 monclient: build_initial_monmap 2024-09-16T14:42:36.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.144+0000 7fbe16b4c640 10 monclient: monmap: 2024-09-16T14:42:36.148 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:36.148 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.148 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:36.148438+0000 2024-09-16T14:42:36.148 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:36.148438+0000 2024-09-16T14:42:36.148 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:36.148 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.148 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:36.148 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:36.148 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:36.148 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.144+0000 7fbe16b4c640 1 Processor -- start 2024-09-16T14:42:36.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.144+0000 7fbe16b4c640 1 -- start start 2024-09-16T14:42:36.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.144+0000 7fbe16b4c640 10 monclient: init 2024-09-16T14:42:36.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:36.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:36.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient(hunting): picked mon.noname-c con 0x7fbe1009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient(hunting): picked mon.noname-b con 0x7fbe10132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient(hunting): picked mon.noname-a con 0x7fbe10138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1009d260 con 0x7fbe10138ff0 2024-09-16T14:42:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe10132a80 con 0x7fbe1009cb80 2024-09-16T14:42:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1009cf60 con 0x7fbe10132650 2024-09-16T14:42:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient(hunting): authenticate will time out at 980.268446s 2024-09-16T14:42:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbe10132650 0x7fbe101378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48760/0 (socket says 172.21.15.26:48760) 2024-09-16T14:42:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0ffff640 1 -- 172.21.15.26:0/623859868 learned_addr learned my addr 172.21.15.26:0/623859868 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe150e9640 1 --1- 172.21.15.26:0/623859868 >> v1:172.21.15.26:6789/0 conn(0x7fbe10138ff0 0x7fbe1013b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57700/0 (socket says 172.21.15.26:57700) 2024-09-16T14:42:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 1 -- 172.21.15.26:0/623859868 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712170230 0 0) 0x7fbe1009cf60 con 0x7fbe10132650 2024-09-16T14:42:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient(hunting): my global_id is 4376 2024-09-16T14:42:36.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 1 -- 172.21.15.26:0/623859868 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdfc003530 con 0x7fbe10132650 2024-09-16T14:42:36.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 1 -- 172.21.15.26:0/623859868 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250437338 0 0) 0x7fbe1009d260 con 0x7fbe10138ff0 2024-09-16T14:42:36.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient(hunting): my global_id is 4389 2024-09-16T14:42:36.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 1 -- 172.21.15.26:0/623859868 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe1009cf60 con 0x7fbe10138ff0 2024-09-16T14:42:36.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 1 -- 172.21.15.26:0/623859868 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1686675556 0 0) 0x7fbdfc003530 con 0x7fbe10132650 2024-09-16T14:42:36.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 1 -- 172.21.15.26:0/623859868 >> v1:172.21.15.189:6789/0 conn(0x7fbe1009cb80 legacy=0x7fbe100717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 1 -- 172.21.15.26:0/623859868 >> v1:172.21.15.26:6789/0 conn(0x7fbe10138ff0 legacy=0x7fbe1013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:42:36.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 1 monclient: found mon.noname-b 2024-09-16T14:42:36.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:36.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:36.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 1 -- 172.21.15.26:0/623859868 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1009f2d0 con 0x7fbe10132650 2024-09-16T14:42:36.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:42:36.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:42:36.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:36.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 1 -- 172.21.15.26:0/623859868 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe00001660 con 0x7fbe10132650 2024-09-16T14:42:36.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:36.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient: dump: 2024-09-16T14:42:36.157 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.158 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 5 monclient: authenticate success, global_id 4376 2024-09-16T14:42:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 1 -- 172.21.15.26:0/623859868 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe000017c0 con 0x7fbe10132650 2024-09-16T14:42:36.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:36.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:36.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 4 set_mon_vals no callback set 2024-09-16T14:42:36.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 1 -- 172.21.15.26:0/623859868 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe00001b90 con 0x7fbe10132650 2024-09-16T14:42:36.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:36.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0f7fe640 10 monclient: dump: 2024-09-16T14:42:36.159 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.160 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient: shutdown 2024-09-16T14:42:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 1 -- 172.21.15.26:0/623859868 >> v1:172.21.15.26:6790/0 conn(0x7fbe10132650 legacy=0x7fbe101378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 1 -- 172.21.15.26:0/623859868 shutdown_connections 2024-09-16T14:42:36.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 1 -- 172.21.15.26:0/623859868 >> 172.21.15.26:0/623859868 conn(0x7fbe1009fb50 msgr2=0x7fbe1009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 1 -- 172.21.15.26:0/623859868 shutdown_connections 2024-09-16T14:42:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 1 -- 172.21.15.26:0/623859868 wait complete. 2024-09-16T14:42:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient: build_initial_monmap 2024-09-16T14:42:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient: monmap: 2024-09-16T14:42:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:36.162 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 1 Processor -- start 2024-09-16T14:42:36.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 1 -- start start 2024-09-16T14:42:36.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient: init 2024-09-16T14:42:36.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:36.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:36.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient(hunting): picked mon.noname-b con 0x7fbe1009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:36.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient(hunting): picked mon.noname-c con 0x7fbe10132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:36.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient(hunting): picked mon.noname-a con 0x7fbe10138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:36.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe100a2140 con 0x7fbe10138ff0 2024-09-16T14:42:36.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe10071ac0 con 0x7fbe1009cb80 2024-09-16T14:42:36.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe100713e0 con 0x7fbe10132650 2024-09-16T14:42:36.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:36.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe16b4c640 10 monclient(hunting): authenticate will time out at 980.271481s 2024-09-16T14:42:36.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe150e9640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbe10138ff0 0x7fbe1006e530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57712/0 (socket says 172.21.15.26:57712) 2024-09-16T14:42:36.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbe10132650 0x7fbe1006de20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48768/0 (socket says 172.21.15.26:48768) 2024-09-16T14:42:36.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe150e9640 1 -- 172.21.15.26:0/4265367230 learned_addr learned my addr 172.21.15.26:0/4265367230 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:36.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721389425 0 0) 0x7fbe100a2140 con 0x7fbe10138ff0 2024-09-16T14:42:36.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0d7fa640 10 monclient(hunting): my global_id is 4392 2024-09-16T14:42:36.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf0003530 con 0x7fbe10138ff0 2024-09-16T14:42:36.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064018168 0 0) 0x7fbe100713e0 con 0x7fbe10132650 2024-09-16T14:42:36.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0d7fa640 10 monclient(hunting): my global_id is 4379 2024-09-16T14:42:36.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.148+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe100a2140 con 0x7fbe10132650 2024-09-16T14:42:36.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710976223 0 0) 0x7fbe10071ac0 con 0x7fbe1009cb80 2024-09-16T14:42:36.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient(hunting): my global_id is 4402 2024-09-16T14:42:36.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe100713e0 con 0x7fbe1009cb80 2024-09-16T14:42:36.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1272923737 0 0) 0x7fbdf0003530 con 0x7fbe10138ff0 2024-09-16T14:42:36.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe10071ac0 con 0x7fbe10138ff0 2024-09-16T14:42:36.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe080016b0 con 0x7fbe10138ff0 2024-09-16T14:42:36.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:36.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient(hunting): dump: 2024-09-16T14:42:36.165 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.165 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.165 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.165 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.165 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.165 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.165 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.165 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.165 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.165 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:36.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3294705917 0 0) 0x7fbe100a2140 con 0x7fbe10132650 2024-09-16T14:42:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdf0003530 con 0x7fbe10132650 2024-09-16T14:42:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe00001480 con 0x7fbe10132650 2024-09-16T14:42:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient(hunting): dump: 2024-09-16T14:42:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.167 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1360776548 0 0) 0x7fbe10071ac0 con 0x7fbe10138ff0 2024-09-16T14:42:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 >> v1:172.21.15.26:6790/0 conn(0x7fbe10132650 legacy=0x7fbe1006de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 >> v1:172.21.15.189:6789/0 conn(0x7fbe1009cb80 legacy=0x7fbe1006d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:42:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 monclient: found mon.a 2024-09-16T14:42:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1006fcb0 con 0x7fbe10138ff0 2024-09-16T14:42:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:42:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:42:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe16b4c640 5 monclient: authenticate success, global_id 4392 2024-09-16T14:42:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe16b4c640 10 monclient: _renew_subs 2024-09-16T14:42:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe16b4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe16b4c640 1 -- 172.21.15.26:0/4265367230 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1006f360 con 0x7fbe10138ff0 2024-09-16T14:42:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe16b4c640 10 monclient: _renew_subs 2024-09-16T14:42:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe16b4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe16b4c640 1 -- 172.21.15.26:0/4265367230 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe101d0b70 con 0x7fbe10138ff0 2024-09-16T14:42:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe08002090 con 0x7fbe10138ff0 2024-09-16T14:42:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe08004e20 con 0x7fbe10138ff0 2024-09-16T14:42:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0effd640 4 set_mon_vals no callback set 2024-09-16T14:42:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 10 monclient: dump: 2024-09-16T14:42:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.169 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe16b4c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe16b4c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:36.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe16b4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe16b4c640 1 -- 172.21.15.26:0/4265367230 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf4005050 con 0x7fbe10138ff0 2024-09-16T14:42:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbe0800f890 con 0x7fbe10138ff0 2024-09-16T14:42:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3534+0+0 (unknown 534919773 0 0) 0x7fbe080433d0 con 0x7fbe10138ff0 2024-09-16T14:42:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.152+0000 7fbe148e8640 10 monclient: get_auth_request con 0x7fbdf0036340 auth_method 0 2024-09-16T14:42:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.156+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbe08014860 con 0x7fbe10138ff0 2024-09-16T14:42:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.156+0000 7fbe0d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.156+0000 7fbe0d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:36.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.308+0000 7fbe16b4c640 1 -- 172.21.15.26:0/4265367230 --> v1:172.21.15.189:6812/3196161582 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbdf4000ba0 con 0x7fbdf0036340 2024-09-16T14:42:36.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.312+0000 7fbe0d7fa640 1 -- 172.21.15.26:0/4265367230 <== mgr.4105 v1:172.21.15.189:6812/3196161582 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41071 (unknown 2965378022 0 2948438717) 0x7fbdf4000ba0 con 0x7fbdf0036340 2024-09-16T14:42:36.315 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T14:42:36.316 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T14:42:36.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.316+0000 7fbe16b4c640 1 -- 172.21.15.26:0/4265367230 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbdf0036340 legacy=0x7fbdf0038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.316+0000 7fbe16b4c640 10 monclient: shutdown 2024-09-16T14:42:36.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.316+0000 7fbe16b4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:36.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.316+0000 7fbe16b4c640 1 -- 172.21.15.26:0/4265367230 >> v1:172.21.15.26:6789/0 conn(0x7fbe10138ff0 legacy=0x7fbe1006e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.316+0000 7fbe16b4c640 1 -- 172.21.15.26:0/4265367230 shutdown_connections 2024-09-16T14:42:36.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.316+0000 7fbe16b4c640 1 -- 172.21.15.26:0/4265367230 >> 172.21.15.26:0/4265367230 conn(0x7fbe1009fb50 msgr2=0x7fbe10131880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:36.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.316+0000 7fbe16b4c640 1 -- 172.21.15.26:0/4265367230 shutdown_connections 2024-09-16T14:42:36.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.316+0000 7fbe16b4c640 1 -- 172.21.15.26:0/4265367230 wait complete. 2024-09-16T14:42:36.332 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-16T14:42:36.073752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167124,"kb_used_data":896,"kb_used_omap":9,"kb_used_meta":166134,"kb_avail":566063916,"statfs":{"total":579820584960,"available":579649449984,"internally_reserved":0,"allocated":917504,"data_stored":1290382,"data_compressed":61020,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9559,"internal_metadata":170121897},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.698622"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.575415+0000","last_change":"2024-09-16T14:42:30.575415+0000","last_active":"2024-09-16T14:42:30.575415+0000","last_peered":"2024-09-16T14:42:30.575415+0000","last_clean":"2024-09-16T14:42:30.575415+0000","last_became_active":"2024-09-16T14:42:28.533291+0000","last_became_peered":"2024-09-16T14:42:28.533291+0000","last_unstale":"2024-09-16T14:42:30.575415+0000","last_undegraded":"2024-09-16T14:42:30.575415+0000","last_fullsized":"2024-09-16T14:42:30.575415+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:36:22.883054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085482099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.597981+0000","last_change":"2024-09-16T14:42:30.597981+0000","last_active":"2024-09-16T14:42:30.597981+0000","last_peered":"2024-09-16T14:42:30.597981+0000","last_clean":"2024-09-16T14:42:30.597981+0000","last_became_active":"2024-09-16T14:42:28.477522+0000","last_became_peered":"2024-09-16T14:42:28.477522+0000","last_unstale":"2024-09-16T14:42:30.597981+0000","last_undegraded":"2024-09-16T14:42:30.597981+0000","last_fullsized":"2024-09-16T14:42:30.597981+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:00:15.035933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065779100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.580369+0000","last_change":"2024-09-16T14:42:30.580369+0000","last_active":"2024-09-16T14:42:30.580369+0000","last_peered":"2024-09-16T14:42:30.580369+0000","last_clean":"2024-09-16T14:42:30.580369+0000","last_became_active":"2024-09-16T14:42:28.533172+0000","last_became_peered":"2024-09-16T14:42:28.533172+0000","last_unstale":"2024-09-16T14:42:30.580369+0000","last_undegraded":"2024-09-16T14:42:30.580369+0000","last_fullsized":"2024-09-16T14:42:30.580369+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:56:43.741456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061478100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.796631+0000","last_change":"2024-09-16T14:42:30.796631+0000","last_active":"2024-09-16T14:42:30.796631+0000","last_peered":"2024-09-16T14:42:30.796631+0000","last_clean":"2024-09-16T14:42:30.796631+0000","last_became_active":"2024-09-16T14:42:28.476209+0000","last_became_peered":"2024-09-16T14:42:28.476209+0000","last_unstale":"2024-09-16T14:42:30.796631+0000","last_undegraded":"2024-09-16T14:42:30.796631+0000","last_fullsized":"2024-09-16T14:42:30.796631+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:32:31.800477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039796999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"15'117","reported_seq":160,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.819644+0000","last_change":"2024-09-16T14:42:29.481739+0000","last_active":"2024-09-16T14:42:30.819644+0000","last_peered":"2024-09-16T14:42:30.819644+0000","last_clean":"2024-09-16T14:42:30.819644+0000","last_became_active":"2024-09-16T14:42:29.480896+0000","last_became_peered":"2024-09-16T14:42:29.480896+0000","last_unstale":"2024-09-16T14:42:30.819644+0000","last_undegraded":"2024-09-16T14:42:30.819644+0000","last_fullsized":"2024-09-16T14:42:30.819644+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:28.462531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:28.462531+0000","last_clean_scrub_stamp":"2024-09-16T14:42:28.462531+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:32:11.234353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.796643+0000","last_change":"2024-09-16T14:42:30.796643+0000","last_active":"2024-09-16T14:42:30.796643+0000","last_peered":"2024-09-16T14:42:30.796643+0000","last_clean":"2024-09-16T14:42:30.796643+0000","last_became_active":"2024-09-16T14:42:28.488186+0000","last_became_peered":"2024-09-16T14:42:28.488186+0000","last_unstale":"2024-09-16T14:42:30.796643+0000","last_undegraded":"2024-09-16T14:42:30.796643+0000","last_fullsized":"2024-09-16T14:42:30.796643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:40:41.632316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042440799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"15'2","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.582824+0000","last_change":"2024-09-16T14:42:30.581886+0000","last_active":"2024-09-16T14:42:30.582824+0000","last_peered":"2024-09-16T14:42:30.582824+0000","last_clean":"2024-09-16T14:42:30.582824+0000","last_became_active":"2024-09-16T14:42:28.487188+0000","last_became_peered":"2024-09-16T14:42:28.487188+0000","last_unstale":"2024-09-16T14:42:30.582824+0000","last_undegraded":"2024-09-16T14:42:30.582824+0000","last_fullsized":"2024-09-16T14:42:30.582824+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:10:26.171186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020492470000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.575555+0000","last_change":"2024-09-16T14:42:30.575555+0000","last_active":"2024-09-16T14:42:30.575555+0000","last_peered":"2024-09-16T14:42:30.575555+0000","last_clean":"2024-09-16T14:42:30.575555+0000","last_became_active":"2024-09-16T14:42:28.484409+0000","last_became_peered":"2024-09-16T14:42:28.484409+0000","last_unstale":"2024-09-16T14:42:30.575555+0000","last_undegraded":"2024-09-16T14:42:30.575555+0000","last_fullsized":"2024-09-16T14:42:30.575555+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:16:51.119532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097689999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T14:42:30.598002+0000","last_change":"2024-09-16T14:42:30.598002+0000","last_active":"2024-09-16T14:42:30.598002+0000","last_peered":"2024-09-16T14:42:30.598002+0000","last_clean":"2024-09-16T14:42:30.598002+0000","last_became_active":"2024-09-16T14:42:28.477416+0000","last_became_peered":"2024-09-16T14:42:28.477416+0000","last_unstale":"2024-09-16T14:42:30.598002+0000","last_undegraded":"2024-09-16T14:42:30.598002+0000","last_fullsized":"2024-09-16T14:42:30.598002+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:49.076917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067836399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61020,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":38748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":135168,"data_stored":38767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":192512,"data_stored":567676,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":38748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":567676,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31252,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340588,"statfs":{"total":96636764160,"available":96604762112,"internally_reserved":0,"allocated":135168,"data_stored":38767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T14:42:36.333 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-16T14:42:36.333 DEBUG:tasks.mon_thrash:Yielding 2024-09-16T14:42:36.333 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T14:42:36.347 INFO:tasks.workunit:Pulling workunits from ref ae7f37330c1a72493684eb1fcaed6f2a01195bb6 2024-09-16T14:42:36.348 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T14:42:36.348 DEBUG:teuthology.orchestra.run.smithi189:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T14:42:36.349 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-16T14:42:36.349 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726497756, revive delay: 20.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-16T14:42:36.349 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:42:36.349 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:42:36.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T14:42:36.355 INFO:teuthology.orchestra.run.smithi189.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T14:42:36.355 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T14:42:36.406 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T14:42:36.406 DEBUG:teuthology.orchestra.run.smithi189:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T14:42:36.453 INFO:tasks.workunit:timeout=3h 2024-09-16T14:42:36.453 INFO:tasks.workunit:cleanup=True 2024-09-16T14:42:36.454 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ae7f37330c1a72493684eb1fcaed6f2a01195bb6 2024-09-16T14:42:36.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 10 monclient: get_monmap_and_config 2024-09-16T14:42:36.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 10 monclient: build_initial_monmap 2024-09-16T14:42:36.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 10 monclient: monmap: 2024-09-16T14:42:36.470 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:36.470 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.470 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:36.470450+0000 2024-09-16T14:42:36.470 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:36.470450+0000 2024-09-16T14:42:36.470 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:36.470 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.470 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:36.470 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:36.470 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:36.470 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 1 Processor -- start 2024-09-16T14:42:36.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 1 -- start start 2024-09-16T14:42:36.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 10 monclient: init 2024-09-16T14:42:36.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:36.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:42:36.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 10 monclient(hunting): picked mon.noname-a con 0x7f37f809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:36.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 10 monclient(hunting): picked mon.noname-c con 0x7f37f8132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:36.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 10 monclient(hunting): picked mon.noname-b con 0x7f37f8138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:36.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f809d260 con 0x7f37f809cb80 2024-09-16T14:42:36.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f8132a80 con 0x7f37f8132650 2024-09-16T14:42:36.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f809cf60 con 0x7f37f8138ff0 2024-09-16T14:42:36.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:36.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 10 monclient(hunting): authenticate will time out at 980.590690s 2024-09-16T14:42:36.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f37f809cb80 0x7f37f80717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57724/0 (socket says 172.21.15.26:57724) 2024-09-16T14:42:36.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f659c640 1 -- 172.21.15.26:0/1070756475 learned_addr learned my addr 172.21.15.26:0/1070756475 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:36.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 1 -- 172.21.15.26:0/1070756475 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354336383 0 0) 0x7f37f809d260 con 0x7f37f809cb80 2024-09-16T14:42:36.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient(hunting): my global_id is 4395 2024-09-16T14:42:36.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 1 -- 172.21.15.26:0/1070756475 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d8003530 con 0x7f37f809cb80 2024-09-16T14:42:36.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 1 -- 172.21.15.26:0/1070756475 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261250465 0 0) 0x7f37f809cf60 con 0x7f37f8138ff0 2024-09-16T14:42:36.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient(hunting): my global_id is 4382 2024-09-16T14:42:36.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 1 -- 172.21.15.26:0/1070756475 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f809d260 con 0x7f37f8138ff0 2024-09-16T14:42:36.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 1 -- 172.21.15.26:0/1070756475 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2833669281 0 0) 0x7f37f809d260 con 0x7f37f8138ff0 2024-09-16T14:42:36.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 1 -- 172.21.15.26:0/1070756475 >> v1:172.21.15.189:6789/0 conn(0x7f37f8132650 legacy=0x7f37f81378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 1 -- 172.21.15.26:0/1070756475 >> v1:172.21.15.26:6789/0 conn(0x7f37f809cb80 legacy=0x7f37f80717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient: _finish_hunting 0 2024-09-16T14:42:36.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 1 monclient: found mon.noname-b 2024-09-16T14:42:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 1 -- 172.21.15.26:0/1070756475 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f809f2d0 con 0x7f37f8138ff0 2024-09-16T14:42:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient: _finish_auth 0 2024-09-16T14:42:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient: _check_auth_tickets 2024-09-16T14:42:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 1 -- 172.21.15.26:0/1070756475 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f37f0001660 con 0x7f37f8138ff0 2024-09-16T14:42:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient: dump: 2024-09-16T14:42:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.475 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.476 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.476 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.476 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.476 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.476 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.476 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.476 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 5 monclient: authenticate success, global_id 4382 2024-09-16T14:42:36.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:36.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 1 -- 172.21.15.26:0/1070756475 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37f0001880 con 0x7f37f8138ff0 2024-09-16T14:42:36.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:36.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 1 -- 172.21.15.26:0/1070756475 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f37f0001b90 con 0x7f37f8138ff0 2024-09-16T14:42:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37f559a640 10 monclient: dump: 2024-09-16T14:42:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.477 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:36.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.468+0000 7f37fcb92640 4 set_mon_vals no callback set 2024-09-16T14:42:36.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient: shutdown 2024-09-16T14:42:36.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:36.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 1 -- 172.21.15.26:0/1070756475 >> v1:172.21.15.26:6790/0 conn(0x7f37f8138ff0 legacy=0x7f37f813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 1 -- 172.21.15.26:0/1070756475 shutdown_connections 2024-09-16T14:42:36.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 1 -- 172.21.15.26:0/1070756475 >> 172.21.15.26:0/1070756475 conn(0x7f37f809fb50 msgr2=0x7f37f809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 1 -- 172.21.15.26:0/1070756475 shutdown_connections 2024-09-16T14:42:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 1 -- 172.21.15.26:0/1070756475 wait complete. 2024-09-16T14:42:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient: build_initial_monmap 2024-09-16T14:42:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient: monmap: 2024-09-16T14:42:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:36.482 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:36.483 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 1 Processor -- start 2024-09-16T14:42:36.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 1 -- start start 2024-09-16T14:42:36.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient: init 2024-09-16T14:42:36.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:36.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:36.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient(hunting): picked mon.noname-a con 0x7f37f809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:36.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient(hunting): picked mon.noname-c con 0x7f37f8132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:36.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient(hunting): picked mon.noname-b con 0x7f37f8138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:36.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f809b370 con 0x7f37f809cb80 2024-09-16T14:42:36.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f813c520 con 0x7f37f8138ff0 2024-09-16T14:42:36.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f809a240 con 0x7f37f8132650 2024-09-16T14:42:36.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:36.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient(hunting): authenticate will time out at 980.593801s 2024-09-16T14:42:36.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37f659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f37f809cb80 0x7f37f809a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57726/0 (socket says 172.21.15.26:57726) 2024-09-16T14:42:36.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37f659c640 1 -- 172.21.15.26:0/1613106206 learned_addr learned my addr 172.21.15.26:0/1613106206 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:36.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097443358 0 0) 0x7f37f809b370 con 0x7f37f809cb80 2024-09-16T14:42:36.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): my global_id is 4398 2024-09-16T14:42:36.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37dc003530 con 0x7f37f809cb80 2024-09-16T14:42:36.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403409382 0 0) 0x7f37f809a240 con 0x7f37f8132650 2024-09-16T14:42:36.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): my global_id is 4385 2024-09-16T14:42:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f809b370 con 0x7f37f8132650 2024-09-16T14:42:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2500590307 0 0) 0x7f37dc003530 con 0x7f37f809cb80 2024-09-16T14:42:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37f809a240 con 0x7f37f809cb80 2024-09-16T14:42:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f37e0001bb0 con 0x7f37f809cb80 2024-09-16T14:42:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): dump: 2024-09-16T14:42:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.485 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.486 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.486 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.486 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.486 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.486 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:36.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2905141631 0 0) 0x7f37f809b370 con 0x7f37f8132650 2024-09-16T14:42:36.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37dc003530 con 0x7f37f8132650 2024-09-16T14:42:36.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f37e8001800 con 0x7f37f8132650 2024-09-16T14:42:36.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:36.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): dump: 2024-09-16T14:42:36.486 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.487 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.487 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.487 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.487 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.487 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.487 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.487 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.487 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.487 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550214486 0 0) 0x7f37f813c520 con 0x7f37f8138ff0 2024-09-16T14:42:36.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient(hunting): my global_id is 4408 2024-09-16T14:42:36.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f809b370 con 0x7f37f8138ff0 2024-09-16T14:42:36.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2776153164 0 0) 0x7f37f809a240 con 0x7f37f809cb80 2024-09-16T14:42:36.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 >> v1:172.21.15.26:6790/0 conn(0x7f37f8132650 legacy=0x7f37f809ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 >> v1:172.21.15.189:6789/0 conn(0x7f37f8138ff0 legacy=0x7f37f81d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:36.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 monclient: found mon.a 2024-09-16T14:42:36.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:36.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:36.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f809b5d0 con 0x7f37f809cb80 2024-09-16T14:42:36.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:36.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:36.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:36.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 5 monclient: authenticate success, global_id 4398 2024-09-16T14:42:36.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient: _renew_subs 2024-09-16T14:42:36.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:36.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 1 -- 172.21.15.26:0/1613106206 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f806d870 con 0x7f37f809cb80 2024-09-16T14:42:36.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient: _renew_subs 2024-09-16T14:42:36.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:36.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 1 -- 172.21.15.26:0/1613106206 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f81386e0 con 0x7f37f809cb80 2024-09-16T14:42:36.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37e0002100 con 0x7f37f809cb80 2024-09-16T14:42:36.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:36.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f37e0004d90 con 0x7f37f809cb80 2024-09-16T14:42:36.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:36.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37f4d99640 4 set_mon_vals no callback set 2024-09-16T14:42:36.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 10 monclient: dump: 2024-09-16T14:42:36.490 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.490 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.490 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.490 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.490 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.490 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.490 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.490 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.490 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.490 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:36.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:36.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:36.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37fcb92640 1 -- 172.21.15.26:0/1613106206 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d0005050 con 0x7f37f809cb80 2024-09-16T14:42:36.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.472+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f37e000f800 con 0x7f37f809cb80 2024-09-16T14:42:36.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.476+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3534+0+0 (unknown 534919773 0 0) 0x7f37e0043380 con 0x7f37f809cb80 2024-09-16T14:42:36.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.476+0000 7f37f5d9b640 10 monclient: get_auth_request con 0x7f37dc036470 auth_method 0 2024-09-16T14:42:36.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.480+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f37e0010da0 con 0x7f37f809cb80 2024-09-16T14:42:36.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.480+0000 7f37eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:36.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.480+0000 7f37eeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:36.503 INFO:tasks.workunit.client.0.smithi189.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T14:42:36.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.700+0000 7f37fcb92640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:42:36.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.700+0000 7f37fcb92640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:42:36.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.700+0000 7f37fcb92640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:36.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.700+0000 7f37fcb92640 1 -- 172.21.15.26:0/1613106206 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f37d0005880 con 0x7f37f809cb80 2024-09-16T14:42:36.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.700+0000 7f37eeffd640 1 -- 172.21.15.26:0/1613106206 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1498500904) 0x7f37e0012e20 con 0x7f37f809cb80 2024-09-16T14:42:36.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.700+0000 7f37eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:42:36.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.700+0000 7f37eeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:36.702 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:42:36.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.700+0000 7f37fcb92640 1 -- 172.21.15.26:0/1613106206 >> v1:172.21.15.189:6812/3196161582 conn(0x7f37dc036470 legacy=0x7f37dc038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.700+0000 7f37fcb92640 10 monclient: shutdown 2024-09-16T14:42:36.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.700+0000 7f37fcb92640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:36.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.700+0000 7f37fcb92640 1 -- 172.21.15.26:0/1613106206 >> v1:172.21.15.26:6789/0 conn(0x7f37f809cb80 legacy=0x7f37f809a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.700+0000 7f37fcb92640 1 -- 172.21.15.26:0/1613106206 shutdown_connections 2024-09-16T14:42:36.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.700+0000 7f37fcb92640 1 -- 172.21.15.26:0/1613106206 >> 172.21.15.26:0/1613106206 conn(0x7f37f809fb50 msgr2=0x7f37f80a1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:36.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.700+0000 7f37fcb92640 1 -- 172.21.15.26:0/1613106206 shutdown_connections 2024-09-16T14:42:36.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.700+0000 7f37fcb92640 1 -- 172.21.15.26:0/1613106206 wait complete. 2024-09-16T14:42:36.721 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:42:36.721 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:42:36.721 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:42:36.721 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:42:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.840+0000 7f1fe2404640 10 monclient: get_monmap_and_config 2024-09-16T14:42:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.840+0000 7f1fe2404640 10 monclient: build_initial_monmap 2024-09-16T14:42:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.840+0000 7f1fe2404640 10 monclient: monmap: 2024-09-16T14:42:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:36.844351+0000 2024-09-16T14:42:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:36.844351+0000 2024-09-16T14:42:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:36.844 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.840+0000 7f1fe2404640 1 Processor -- start 2024-09-16T14:42:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.840+0000 7f1fe2404640 1 -- start start 2024-09-16T14:42:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.840+0000 7f1fe2404640 10 monclient: init 2024-09-16T14:42:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:42:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient(hunting): picked mon.noname-a con 0x7f1fdc09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient(hunting): picked mon.noname-c con 0x7f1fdc132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient(hunting): picked mon.noname-b con 0x7f1fdc138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fdc09d260 con 0x7f1fdc09cb80 2024-09-16T14:42:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fdc132a80 con 0x7f1fdc132650 2024-09-16T14:42:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fdc09cf60 con 0x7f1fdc138ff0 2024-09-16T14:42:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient(hunting): authenticate will time out at 980.964492s 2024-09-16T14:42:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe09a1640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1fdc138ff0 0x7f1fdc13b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48814/0 (socket says 172.21.15.26:48814) 2024-09-16T14:42:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdbfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1fdc09cb80 0x7f1fdc0717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57740/0 (socket says 172.21.15.26:57740) 2024-09-16T14:42:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe09a1640 1 -- 172.21.15.26:0/2307697599 learned_addr learned my addr 172.21.15.26:0/2307697599 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 1 -- 172.21.15.26:0/2307697599 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941395541 0 0) 0x7f1fdc09cf60 con 0x7f1fdc138ff0 2024-09-16T14:42:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient(hunting): my global_id is 4388 2024-09-16T14:42:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 1 -- 172.21.15.26:0/2307697599 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc8003530 con 0x7f1fdc138ff0 2024-09-16T14:42:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 1 -- 172.21.15.26:0/2307697599 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568461155 0 0) 0x7f1fdc09d260 con 0x7f1fdc09cb80 2024-09-16T14:42:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient(hunting): my global_id is 4401 2024-09-16T14:42:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 1 -- 172.21.15.26:0/2307697599 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fdc09cf60 con 0x7f1fdc09cb80 2024-09-16T14:42:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 1 -- 172.21.15.26:0/2307697599 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2175696299 0 0) 0x7f1fdc09cf60 con 0x7f1fdc09cb80 2024-09-16T14:42:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 1 -- 172.21.15.26:0/2307697599 >> v1:172.21.15.26:6790/0 conn(0x7f1fdc138ff0 legacy=0x7f1fdc13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 1 -- 172.21.15.26:0/2307697599 >> v1:172.21.15.189:6789/0 conn(0x7f1fdc132650 legacy=0x7f1fdc1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 1 monclient: found mon.noname-a 2024-09-16T14:42:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:42:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 1 -- 172.21.15.26:0/2307697599 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc09f2d0 con 0x7f1fdc09cb80 2024-09-16T14:42:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 1 -- 172.21.15.26:0/2307697599 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1fd4001980 con 0x7f1fdc09cb80 2024-09-16T14:42:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 5 monclient: authenticate success, global_id 4401 2024-09-16T14:42:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient: dump: 2024-09-16T14:42:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.855 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 1 -- 172.21.15.26:0/2307697599 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fd4001bb0 con 0x7f1fdc09cb80 2024-09-16T14:42:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 4 set_mon_vals no callback set 2024-09-16T14:42:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 1 -- 172.21.15.26:0/2307697599 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1fd4001f80 con 0x7f1fdc09cb80 2024-09-16T14:42:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdaffd640 10 monclient: dump: 2024-09-16T14:42:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.857 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient: shutdown 2024-09-16T14:42:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 1 -- 172.21.15.26:0/2307697599 >> v1:172.21.15.26:6789/0 conn(0x7f1fdc09cb80 legacy=0x7f1fdc0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 1 -- 172.21.15.26:0/2307697599 shutdown_connections 2024-09-16T14:42:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 1 -- 172.21.15.26:0/2307697599 >> 172.21.15.26:0/2307697599 conn(0x7f1fdc09fb50 msgr2=0x7f1fdc09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 1 -- 172.21.15.26:0/2307697599 shutdown_connections 2024-09-16T14:42:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 1 -- 172.21.15.26:0/2307697599 wait complete. 2024-09-16T14:42:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient: build_initial_monmap 2024-09-16T14:42:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient: monmap: 2024-09-16T14:42:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:36.858 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 1 Processor -- start 2024-09-16T14:42:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 1 -- start start 2024-09-16T14:42:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient: init 2024-09-16T14:42:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient(hunting): picked mon.noname-b con 0x7f1fdc09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient(hunting): picked mon.noname-c con 0x7f1fdc132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient(hunting): picked mon.noname-a con 0x7f1fdc138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fdc134a00 con 0x7f1fdc138ff0 2024-09-16T14:42:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fdc071c70 con 0x7f1fdc09cb80 2024-09-16T14:42:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fdc0a2180 con 0x7f1fdc132650 2024-09-16T14:42:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fe2404640 10 monclient(hunting): authenticate will time out at 980.967298s 2024-09-16T14:42:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdb7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1fdc132650 0x7f1fdc1342f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48826/0 (socket says 172.21.15.26:48826) 2024-09-16T14:42:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fdb7fe640 1 -- 172.21.15.26:0/665481444 learned_addr learned my addr 172.21.15.26:0/665481444 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 866205130 0 0) 0x7f1fdc0a2180 con 0x7f1fdc132650 2024-09-16T14:42:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 10 monclient(hunting): my global_id is 4391 2024-09-16T14:42:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fbc003530 con 0x7f1fdc132650 2024-09-16T14:42:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990830199 0 0) 0x7f1fdc134a00 con 0x7f1fdc138ff0 2024-09-16T14:42:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 10 monclient(hunting): my global_id is 4404 2024-09-16T14:42:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fdc0a2180 con 0x7f1fdc138ff0 2024-09-16T14:42:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 279580473 0 0) 0x7f1fbc003530 con 0x7f1fdc132650 2024-09-16T14:42:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fdc134a00 con 0x7f1fdc132650 2024-09-16T14:42:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 497577732 0 0) 0x7f1fdc0a2180 con 0x7f1fdc138ff0 2024-09-16T14:42:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fbc003530 con 0x7f1fdc138ff0 2024-09-16T14:42:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1fc40019f0 con 0x7f1fdc132650 2024-09-16T14:42:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.862 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1fcc001070 con 0x7f1fdc138ff0 2024-09-16T14:42:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.864 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.844+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203984491 0 0) 0x7f1fdc071c70 con 0x7f1fdc09cb80 2024-09-16T14:42:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient(hunting): my global_id is 4414 2024-09-16T14:42:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fdc0a2180 con 0x7f1fdc09cb80 2024-09-16T14:42:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4154663215 0 0) 0x7f1fdc134a00 con 0x7f1fdc132650 2024-09-16T14:42:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 >> v1:172.21.15.189:6789/0 conn(0x7f1fdc09cb80 legacy=0x7f1fdc133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 >> v1:172.21.15.26:6789/0 conn(0x7f1fdc138ff0 legacy=0x7f1fdc1cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:42:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 monclient: found mon.c 2024-09-16T14:42:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc09a240 con 0x7f1fdc132650 2024-09-16T14:42:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient: _finish_auth 0 2024-09-16T14:42:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:42:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fe2404640 5 monclient: authenticate success, global_id 4391 2024-09-16T14:42:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fe2404640 10 monclient: _renew_subs 2024-09-16T14:42:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fe2404640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fe2404640 1 -- 172.21.15.26:0/665481444 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fdc1372b0 con 0x7f1fdc132650 2024-09-16T14:42:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fe2404640 10 monclient: _renew_subs 2024-09-16T14:42:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fe2404640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fe2404640 1 -- 172.21.15.26:0/665481444 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fdc1357f0 con 0x7f1fdc132650 2024-09-16T14:42:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fc4001ba0 con 0x7f1fdc132650 2024-09-16T14:42:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1fc4003a70 con 0x7f1fdc132650 2024-09-16T14:42:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient: dump: 2024-09-16T14:42:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.868 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fda7fc640 4 set_mon_vals no callback set 2024-09-16T14:42:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1fc4003bd0 con 0x7f1fdc132650 2024-09-16T14:42:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fe2404640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fe2404640 1 -- 172.21.15.26:0/665481444 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1fc0006590 con 0x7f1fc0002830 2024-09-16T14:42:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3534+0+0 (unknown 534919773 0 0) 0x7f1fc40431f0 con 0x7f1fdc132650 2024-09-16T14:42:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943840481 0 0) 0x7f1fc0006590 con 0x7f1fc0002830 2024-09-16T14:42:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient(hunting): my global_id is 4407 2024-09-16T14:42:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fdc134a00 con 0x7f1fc0002830 2024-09-16T14:42:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 44595294 0 0) 0x7f1fdc134a00 con 0x7f1fc0002830 2024-09-16T14:42:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fc0006590 con 0x7f1fc0002830 2024-09-16T14:42:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1fd4001700 con 0x7f1fc0002830 2024-09-16T14:42:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fe09a1640 10 monclient: get_auth_request con 0x7f1fbc0365c0 auth_method 0 2024-09-16T14:42:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient: dump: 2024-09-16T14:42:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.870 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 316778334 0 0) 0x7f1fc0006590 con 0x7f1fc0002830 2024-09-16T14:42:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1fc000a0e0 con 0x7f1fc0002830 2024-09-16T14:42:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1fd4001920 con 0x7f1fc0002830 2024-09-16T14:42:36.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:36.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:36.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 >> v1:172.21.15.26:6789/0 conn(0x7f1fc0002830 legacy=0x7f1fc0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.848+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 >> v1:172.21.15.26:6789/0 conn(0x7f1fc0002830 legacy=0x7f1fc0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:42:36.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.852+0000 7f1fe2404640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:42:36.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.852+0000 7f1fe2404640 1 -- 172.21.15.26:0/665481444 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1fc0009d80 con 0x7f1fc00090a0 2024-09-16T14:42:36.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078450959 0 0) 0x7f1fc0009d80 con 0x7f1fc00090a0 2024-09-16T14:42:36.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:36.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:36.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:36.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 10 monclient(hunting): my global_id is 4410 2024-09-16T14:42:36.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc0006590 con 0x7f1fc00090a0 2024-09-16T14:42:36.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3992088164 0 0) 0x7f1fc0006590 con 0x7f1fc00090a0 2024-09-16T14:42:36.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fc0009d80 con 0x7f1fc00090a0 2024-09-16T14:42:36.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1fd4004f80 con 0x7f1fc00090a0 2024-09-16T14:42:36.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:36.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:36.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 10 monclient: dump: 2024-09-16T14:42:36.873 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:36.873 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.873 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.874 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:36.874 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:36.874 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.874 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:36.874 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:36.874 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:36.874 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:36.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2435744364 0 0) 0x7f1fc0009d80 con 0x7f1fc00090a0 2024-09-16T14:42:36.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1fc000c290 con 0x7f1fc00090a0 2024-09-16T14:42:36.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 2774765320) 0x7f1fd40071e0 con 0x7f1fc00090a0 2024-09-16T14:42:36.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:42:36.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:36.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 >> v1:172.21.15.26:6789/0 conn(0x7f1fc00090a0 legacy=0x7f1fc0009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fd8ff9640 1 -- 172.21.15.26:0/665481444 >> v1:172.21.15.26:6789/0 conn(0x7f1fc00090a0 legacy=0x7f1fc0009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:42:36.875 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:42:36.875 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:42:36.875 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:42:36.875 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:42:36.875 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 8, 2024-09-16T14:42:36.875 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:42:36.875 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:42:36.876 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:42:36.876 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:42:36.876 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:36.876 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 19, 2024-09-16T14:42:36.876 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:42:36.876 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:42:36.876 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:42:36.876 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:42:36.876 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:42:36.876 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:42:36.876 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:42:36.876 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:42:36.876 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:42:36.877 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:42:36.877 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:42:36.877 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:42:36.877 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:42:36.877 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:36.877 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:42:36.877 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:42:36.877 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:42:36.877 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:42:36.877 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:42:36.877 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:42:36.877 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:42:36.877 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:42:36.878 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:42:36.878 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:42:36.878 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:42:36.878 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:42:36.878 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:36.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:36.878 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:42:36.878 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:42:36.878 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:42:36.878 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:42:36.878 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:42:36.878 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:42:36.878 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:42:36.879 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:42:36.879 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:42:36.879 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:42:36.879 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:42:36.879 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:42:36.879 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:42:36.879 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:42:36.879 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:42:36.879 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:42:36.879 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:42:36.879 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:42:36.879 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:42:36.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:42:36.879 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:42:36.880 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:42:36.880 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:42:36.880 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:42:36.880 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:42:36.880 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:42:36.880 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:42:36.880 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:36.880 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:42:36.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:36.880 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:42:36.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:36.880 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:42:36.880 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:42:36.881 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:42:36.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:42:36.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:36.881 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:42:36.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:42:36.881 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:42:36.881 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:36.881 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:36.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:36.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:42:36.881 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:42:36.881 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:42:36.881 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:42:36.882 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:42:36.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:36.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:36.882 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:42:36.882 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:42:36.882 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:42:36.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:42:36.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:36.882 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:42:36.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:42:36.882 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:42:36.882 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:36.882 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:36.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:36.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:42:36.883 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:42:36.883 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:42:36.883 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:42:36.883 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:42:36.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:36.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:36.883 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:42:36.883 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:42:36.883 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:42:36.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:42:36.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:36.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:42:36.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:42:36.884 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:42:36.884 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:36.884 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:36.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:36.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:42:36.884 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:42:36.884 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:42:36.884 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:42:36.884 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:42:36.884 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:36.884 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:36.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:36.885 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:42:36.885 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:42:36.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:36.885 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:36.885 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:36.885 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:42:36.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:36.885 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:36.885 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:42:36.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:36.885 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:36.885 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:36.885 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:42:36.886 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:36.886 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:36.886 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:42:36.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:36.886 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:36.886 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:36.886 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:42:36.886 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:36.886 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:36.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:36.886 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:42:36.886 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:42:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fe2404640 1 -- 172.21.15.26:0/665481444 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1fbc0365c0 legacy=0x7f1fbc038a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fe2404640 10 monclient: shutdown 2024-09-16T14:42:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fe2404640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fe2404640 1 -- 172.21.15.26:0/665481444 >> v1:172.21.15.26:6790/0 conn(0x7f1fdc132650 legacy=0x7f1fdc1342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fe09a1640 1 -- 172.21.15.26:0/665481444 reap_dead start 2024-09-16T14:42:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fe09a1640 1 -- 172.21.15.26:0/665481444 reap_dead start 2024-09-16T14:42:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fe2404640 1 -- 172.21.15.26:0/665481444 shutdown_connections 2024-09-16T14:42:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fe2404640 1 -- 172.21.15.26:0/665481444 >> 172.21.15.26:0/665481444 conn(0x7f1fdc09fb50 msgr2=0x7f1fdc0a0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fe2404640 1 -- 172.21.15.26:0/665481444 shutdown_connections 2024-09-16T14:42:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.856+0000 7f1fe2404640 1 -- 172.21.15.26:0/665481444 wait complete. 2024-09-16T14:42:36.889 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:42:36.889 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:42:36.889 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:42:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.992+0000 7fd82e39c640 10 monclient: get_monmap_and_config 2024-09-16T14:42:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.992+0000 7fd82e39c640 10 monclient: build_initial_monmap 2024-09-16T14:42:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.992+0000 7fd82e39c640 10 monclient: monmap: 2024-09-16T14:42:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:36.996805+0000 2024-09-16T14:42:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:36.996805+0000 2024-09-16T14:42:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:36.997 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 1 Processor -- start 2024-09-16T14:42:37.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 1 -- start start 2024-09-16T14:42:37.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 10 monclient: init 2024-09-16T14:42:37.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:37.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:42:37.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 10 monclient(hunting): picked mon.noname-b con 0x7fd82809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:37.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 10 monclient(hunting): picked mon.noname-a con 0x7fd828132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:37.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 10 monclient(hunting): picked mon.noname-c con 0x7fd828138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:37.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd82809d260 con 0x7fd828132650 2024-09-16T14:42:37.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd828132a80 con 0x7fd828138ff0 2024-09-16T14:42:37.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd82809cf60 con 0x7fd82809cb80 2024-09-16T14:42:37.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:37.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 10 monclient(hunting): authenticate will time out at 981.117451s 2024-09-16T14:42:37.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd8277fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd828132650 0x7fd8281378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56458/0 (socket says 172.21.15.26:56458) 2024-09-16T14:42:37.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd8277fe640 1 -- 172.21.15.26:0/2775176081 learned_addr learned my addr 172.21.15.26:0/2775176081 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:37.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 1 -- 172.21.15.26:0/2775176081 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215460525 0 0) 0x7fd82809cf60 con 0x7fd82809cb80 2024-09-16T14:42:37.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:37.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:37.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:37.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient(hunting): my global_id is 4394 2024-09-16T14:42:37.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 1 -- 172.21.15.26:0/2775176081 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd810003530 con 0x7fd82809cb80 2024-09-16T14:42:37.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 1 -- 172.21.15.26:0/2775176081 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404125457 0 0) 0x7fd82809d260 con 0x7fd828132650 2024-09-16T14:42:37.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:37.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:37.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:37.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient(hunting): my global_id is 4413 2024-09-16T14:42:37.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 1 -- 172.21.15.26:0/2775176081 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd82809cf60 con 0x7fd828132650 2024-09-16T14:42:37.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 1 -- 172.21.15.26:0/2775176081 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1178822892 0 0) 0x7fd810003530 con 0x7fd82809cb80 2024-09-16T14:42:37.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 1 -- 172.21.15.26:0/2775176081 >> v1:172.21.15.189:6789/0 conn(0x7fd828138ff0 legacy=0x7fd82813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 1 -- 172.21.15.26:0/2775176081 >> v1:172.21.15.26:6789/0 conn(0x7fd828132650 legacy=0x7fd8281378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 1 monclient: found mon.noname-b 2024-09-16T14:42:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 1 -- 172.21.15.26:0/2775176081 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82809f2d0 con 0x7fd82809cb80 2024-09-16T14:42:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 1 -- 172.21.15.26:0/2775176081 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd818001660 con 0x7fd82809cb80 2024-09-16T14:42:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 5 monclient: authenticate success, global_id 4394 2024-09-16T14:42:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient: dump: 2024-09-16T14:42:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:37.008 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 1 -- 172.21.15.26:0/2775176081 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd818001880 con 0x7fd82809cb80 2024-09-16T14:42:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 1 -- 172.21.15.26:0/2775176081 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd818001b90 con 0x7fd82809cb80 2024-09-16T14:42:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 4 set_mon_vals no callback set 2024-09-16T14:42:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd826ffd640 10 monclient: dump: 2024-09-16T14:42:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:37.010 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 10 monclient: shutdown 2024-09-16T14:42:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 1 -- 172.21.15.26:0/2775176081 >> v1:172.21.15.26:6790/0 conn(0x7fd82809cb80 legacy=0x7fd8280717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 1 -- 172.21.15.26:0/2775176081 shutdown_connections 2024-09-16T14:42:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 1 -- 172.21.15.26:0/2775176081 >> 172.21.15.26:0/2775176081 conn(0x7fd82809fb50 msgr2=0x7fd82809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 1 -- 172.21.15.26:0/2775176081 shutdown_connections 2024-09-16T14:42:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 1 -- 172.21.15.26:0/2775176081 wait complete. 2024-09-16T14:42:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 10 monclient: build_initial_monmap 2024-09-16T14:42:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 10 monclient: monmap: 2024-09-16T14:42:37.011 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:37.011 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:37.011 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:37.011 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:37.011 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:37.011 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:37.011 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:37.011 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:37.011 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:37.011 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 1 Processor -- start 2024-09-16T14:42:37.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 1 -- start start 2024-09-16T14:42:37.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:36.996+0000 7fd82e39c640 10 monclient: init 2024-09-16T14:42:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 10 monclient(hunting): picked mon.noname-a con 0x7fd82809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 10 monclient(hunting): picked mon.noname-c con 0x7fd828132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 10 monclient(hunting): picked mon.noname-b con 0x7fd828138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd828137e20 con 0x7fd82809cb80 2024-09-16T14:42:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd828130330 con 0x7fd828138ff0 2024-09-16T14:42:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd827fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd82809cb80 0x7fd828133ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56464/0 (socket says 172.21.15.26:56464) 2024-09-16T14:42:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd827fff640 1 -- 172.21.15.26:0/3216661646 learned_addr learned my addr 172.21.15.26:0/3216661646 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd828134d10 con 0x7fd828132650 2024-09-16T14:42:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 10 monclient(hunting): authenticate will time out at 981.120508s 2024-09-16T14:42:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551138274 0 0) 0x7fd828137e20 con 0x7fd82809cb80 2024-09-16T14:42:37.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:37.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:37.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:37.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): my global_id is 4416 2024-09-16T14:42:37.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd814003530 con 0x7fd82809cb80 2024-09-16T14:42:37.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603830261 0 0) 0x7fd828134d10 con 0x7fd828132650 2024-09-16T14:42:37.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:37.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:37.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:37.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): my global_id is 4397 2024-09-16T14:42:37.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd828137e20 con 0x7fd828132650 2024-09-16T14:42:37.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 476838394 0 0) 0x7fd814003530 con 0x7fd82809cb80 2024-09-16T14:42:37.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd828134d10 con 0x7fd82809cb80 2024-09-16T14:42:37.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd818001b90 con 0x7fd82809cb80 2024-09-16T14:42:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:37.014 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1910451227 0 0) 0x7fd828137e20 con 0x7fd828132650 2024-09-16T14:42:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd814003530 con 0x7fd828132650 2024-09-16T14:42:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd81c0018e0 con 0x7fd828132650 2024-09-16T14:42:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): dump: 2024-09-16T14:42:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:37.016 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535021518 0 0) 0x7fd828130330 con 0x7fd828138ff0 2024-09-16T14:42:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): my global_id is 4420 2024-09-16T14:42:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd828137e20 con 0x7fd828138ff0 2024-09-16T14:42:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3863924844 0 0) 0x7fd828134d10 con 0x7fd82809cb80 2024-09-16T14:42:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 >> v1:172.21.15.26:6790/0 conn(0x7fd828132650 legacy=0x7fd828134600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 >> v1:172.21.15.189:6789/0 conn(0x7fd828138ff0 legacy=0x7fd8281cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:42:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 monclient: found mon.a 2024-09-16T14:42:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8281383e0 con 0x7fd82809cb80 2024-09-16T14:42:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient: _finish_auth 0 2024-09-16T14:42:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:42:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 5 monclient: authenticate success, global_id 4416 2024-09-16T14:42:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 10 monclient: _renew_subs 2024-09-16T14:42:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd82809a240 con 0x7fd82809cb80 2024-09-16T14:42:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 10 monclient: _renew_subs 2024-09-16T14:42:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd828137580 con 0x7fd82809cb80 2024-09-16T14:42:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8180011e0 con 0x7fd82809cb80 2024-09-16T14:42:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd818003ad0 con 0x7fd82809cb80 2024-09-16T14:42:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd825ffb640 4 set_mon_vals no callback set 2024-09-16T14:42:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient: dump: 2024-09-16T14:42:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:37.019 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82e39c640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd808006590 con 0x7fd808002830 2024-09-16T14:42:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd818003c30 con 0x7fd82809cb80 2024-09-16T14:42:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3534+0+0 (unknown 534919773 0 0) 0x7fd8180431c0 con 0x7fd82809cb80 2024-09-16T14:42:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618127814 0 0) 0x7fd808006590 con 0x7fd808002830 2024-09-16T14:42:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient(hunting): my global_id is 4400 2024-09-16T14:42:37.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd828134d10 con 0x7fd808002830 2024-09-16T14:42:37.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3642600876 0 0) 0x7fd828134d10 con 0x7fd808002830 2024-09-16T14:42:37.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd808006590 con 0x7fd808002830 2024-09-16T14:42:37.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd81c004b80 con 0x7fd808002830 2024-09-16T14:42:37.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:37.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:37.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient: dump: 2024-09-16T14:42:37.020 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:37.020 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:37.020 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.020 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.020 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:37.020 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:37.021 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:37.021 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:37.021 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:37.021 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd82c939640 10 monclient: get_auth_request con 0x7fd814036380 auth_method 0 2024-09-16T14:42:37.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3721947909 0 0) 0x7fd808006590 con 0x7fd808002830 2024-09-16T14:42:37.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd80800a0e0 con 0x7fd808002830 2024-09-16T14:42:37.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fd81c004ec0 con 0x7fd808002830 2024-09-16T14:42:37.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:37.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:37.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 >> v1:172.21.15.26:6790/0 conn(0x7fd808002830 legacy=0x7fd808004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.000+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 >> v1:172.21.15.26:6790/0 conn(0x7fd808002830 legacy=0x7fd808004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:42:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd82e39c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:42:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd82e39c640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd808009da0 con 0x7fd8080090c0 2024-09-16T14:42:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849726426 0 0) 0x7fd808009da0 con 0x7fd8080090c0 2024-09-16T14:42:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:42:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 10 monclient(hunting): my global_id is 4403 2024-09-16T14:42:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd808006590 con 0x7fd8080090c0 2024-09-16T14:42:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2104460031 0 0) 0x7fd808006590 con 0x7fd8080090c0 2024-09-16T14:42:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd808009da0 con 0x7fd8080090c0 2024-09-16T14:42:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd81c006ea0 con 0x7fd8080090c0 2024-09-16T14:42:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 10 monclient: dump: 2024-09-16T14:42:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:37.025 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1298431693 0 0) 0x7fd808009da0 con 0x7fd8080090c0 2024-09-16T14:42:37.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd80800c290 con 0x7fd8080090c0 2024-09-16T14:42:37.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 750398304) 0x7fd81c006690 con 0x7fd8080090c0 2024-09-16T14:42:37.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:42:37.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:37.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 >> v1:172.21.15.26:6790/0 conn(0x7fd8080090c0 legacy=0x7fd808009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd824ff9640 1 -- 172.21.15.26:0/3216661646 >> v1:172.21.15.26:6790/0 conn(0x7fd8080090c0 legacy=0x7fd808009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:42:37.026 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:42:37.026 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:42:37.026 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:42:37.026 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:42:37.026 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 8, 2024-09-16T14:42:37.026 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:42:37.026 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:42:37.026 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:42:37.026 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:42:37.026 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 19, 2024-09-16T14:42:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:42:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:42:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:42:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:42:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:42:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:42:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:42:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:42:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:42:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:42:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:42:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:42:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:42:37.028 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:37.028 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:42:37.028 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:42:37.028 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:42:37.028 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:42:37.028 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:42:37.028 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:42:37.028 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:42:37.028 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:42:37.028 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:42:37.028 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:42:37.028 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:42:37.028 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:42:37.029 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:37.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.029 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:42:37.029 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:42:37.029 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:42:37.029 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:42:37.029 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:42:37.029 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:42:37.029 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:42:37.029 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:42:37.029 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:42:37.029 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:42:37.029 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:42:37.029 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:42:37.030 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:42:37.030 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:42:37.030 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:42:37.030 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:42:37.030 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:42:37.030 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:42:37.030 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:42:37.030 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:42:37.030 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:42:37.030 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:42:37.030 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:42:37.030 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:42:37.031 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:42:37.031 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:42:37.031 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:42:37.031 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:37.031 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:42:37.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.031 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:42:37.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.031 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:42:37.031 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:42:37.032 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:42:37.032 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:42:37.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.032 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:42:37.032 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:42:37.032 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:42:37.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:37.032 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:37.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.033 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:42:37.033 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:42:37.033 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:42:37.033 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:42:37.033 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:42:37.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.033 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:42:37.033 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:42:37.033 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:42:37.034 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:42:37.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.034 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:42:37.034 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:42:37.034 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:42:37.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:37.034 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:37.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.034 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:42:37.034 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:42:37.034 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:42:37.034 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:42:37.034 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:42:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:42:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:42:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:42:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:42:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:42:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:42:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:42:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.035 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:42:37.036 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:42:37.036 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:42:37.036 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:42:37.036 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:42:37.036 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:37.036 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:37.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.036 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:42:37.036 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:42:37.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.036 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:37.036 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:37.036 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:42:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:42:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:42:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:42:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:37.037 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:42:37.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:37.038 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:37.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.038 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:42:37.038 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:42:37.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd82e39c640 1 -- 172.21.15.26:0/3216661646 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd814036380 legacy=0x7fd814038840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd82e39c640 10 monclient: shutdown 2024-09-16T14:42:37.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd82e39c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd82e39c640 1 -- 172.21.15.26:0/3216661646 >> v1:172.21.15.26:6789/0 conn(0x7fd82809cb80 legacy=0x7fd828133ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd82c939640 1 -- 172.21.15.26:0/3216661646 reap_dead start 2024-09-16T14:42:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd82c939640 1 -- 172.21.15.26:0/3216661646 reap_dead start 2024-09-16T14:42:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd82e39c640 1 -- 172.21.15.26:0/3216661646 shutdown_connections 2024-09-16T14:42:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd82e39c640 1 -- 172.21.15.26:0/3216661646 >> 172.21.15.26:0/3216661646 conn(0x7fd82809fb50 msgr2=0x7fd8281388c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd82e39c640 1 -- 172.21.15.26:0/3216661646 shutdown_connections 2024-09-16T14:42:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.008+0000 7fd82e39c640 1 -- 172.21.15.26:0/3216661646 wait complete. 2024-09-16T14:42:37.040 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:42:37.040 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:42:37.041 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:42:37.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 10 monclient: get_monmap_and_config 2024-09-16T14:42:37.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 10 monclient: build_initial_monmap 2024-09-16T14:42:37.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 10 monclient: monmap: 2024-09-16T14:42:37.146 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:37.146 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:37.146 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:37.146876+0000 2024-09-16T14:42:37.146 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:37.146876+0000 2024-09-16T14:42:37.146 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:37.146 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:37.146 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:37.147 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:37.147 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:37.147 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 1 Processor -- start 2024-09-16T14:42:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 1 -- start start 2024-09-16T14:42:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 10 monclient: init 2024-09-16T14:42:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 10 monclient(hunting): picked mon.noname-a con 0x7f8f4c0968c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 10 monclient(hunting): picked mon.noname-b con 0x7f8f4c098bc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 10 monclient(hunting): picked mon.noname-c con 0x7f8f4c096ca0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f4c06b2d0 con 0x7f8f4c0968c0 2024-09-16T14:42:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f4c097940 con 0x7f8f4c096ca0 2024-09-16T14:42:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f4c09fec0 con 0x7f8f4c098bc0 2024-09-16T14:42:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 10 monclient(hunting): authenticate will time out at 981.266844s 2024-09-16T14:42:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f49d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8f4c098bc0 0x7f8f4c09f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48476/0 (socket says 172.21.15.26:48476) 2024-09-16T14:42:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4a59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8f4c0968c0 0x7f8f4c09f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56474/0 (socket says 172.21.15.26:56474) 2024-09-16T14:42:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f49d9b640 1 -- 172.21.15.26:0/3537098095 learned_addr learned my addr 172.21.15.26:0/3537098095 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 1 -- 172.21.15.26:0/3537098095 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2759639859 0 0) 0x7f8f4c06b2d0 con 0x7f8f4c0968c0 2024-09-16T14:42:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient(hunting): _negotiate 2024-09-16T14:42:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient(hunting): my global_id is 4419 2024-09-16T14:42:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 1 -- 172.21.15.26:0/3537098095 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f34003530 con 0x7f8f4c0968c0 2024-09-16T14:42:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 1 -- 172.21.15.26:0/3537098095 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353571152 0 0) 0x7f8f4c09fec0 con 0x7f8f4c098bc0 2024-09-16T14:42:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient(hunting): _negotiate 2024-09-16T14:42:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient(hunting): my global_id is 4406 2024-09-16T14:42:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 1 -- 172.21.15.26:0/3537098095 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f4c06b2d0 con 0x7f8f4c098bc0 2024-09-16T14:42:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 1 -- 172.21.15.26:0/3537098095 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 528801898 0 0) 0x7f8f34003530 con 0x7f8f4c0968c0 2024-09-16T14:42:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 1 -- 172.21.15.26:0/3537098095 >> v1:172.21.15.26:6790/0 conn(0x7f8f4c098bc0 legacy=0x7f8f4c09f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 1 -- 172.21.15.26:0/3537098095 >> v1:172.21.15.189:6789/0 conn(0x7f8f4c096ca0 legacy=0x7f8f4c13da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient: _finish_hunting 0 2024-09-16T14:42:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 1 monclient: found mon.noname-a 2024-09-16T14:42:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:42:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 1 -- 172.21.15.26:0/3537098095 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4c097c20 con 0x7f8f4c0968c0 2024-09-16T14:42:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient: _finish_auth 0 2024-09-16T14:42:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient: _check_auth_tickets 2024-09-16T14:42:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 1 -- 172.21.15.26:0/3537098095 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8f3c001550 con 0x7f8f4c0968c0 2024-09-16T14:42:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient: dump: 2024-09-16T14:42:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:37.157 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 5 monclient: authenticate success, global_id 4419 2024-09-16T14:42:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 1 -- 172.21.15.26:0/3537098095 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f3c001830 con 0x7f8f4c0968c0 2024-09-16T14:42:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 1 -- 172.21.15.26:0/3537098095 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8f3c001c00 con 0x7f8f4c0968c0 2024-09-16T14:42:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f5088c640 4 set_mon_vals no callback set 2024-09-16T14:42:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.144+0000 7f8f4959a640 10 monclient: dump: 2024-09-16T14:42:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:37.158 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 10 monclient: shutdown 2024-09-16T14:42:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 1 -- 172.21.15.26:0/3537098095 >> v1:172.21.15.26:6789/0 conn(0x7f8f4c0968c0 legacy=0x7f8f4c09f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 1 -- 172.21.15.26:0/3537098095 shutdown_connections 2024-09-16T14:42:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 1 -- 172.21.15.26:0/3537098095 >> 172.21.15.26:0/3537098095 conn(0x7f8f4c0a0360 msgr2=0x7f8f4c0a0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 1 -- 172.21.15.26:0/3537098095 shutdown_connections 2024-09-16T14:42:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 1 -- 172.21.15.26:0/3537098095 wait complete. 2024-09-16T14:42:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 10 monclient: build_initial_monmap 2024-09-16T14:42:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 10 monclient: monmap: 2024-09-16T14:42:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:37.160 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 1 Processor -- start 2024-09-16T14:42:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 1 -- start start 2024-09-16T14:42:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 10 monclient: init 2024-09-16T14:42:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 10 monclient(hunting): picked mon.noname-b con 0x7f8f4c0968c0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 10 monclient(hunting): picked mon.noname-c con 0x7f8f4c096ca0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 10 monclient(hunting): picked mon.noname-a con 0x7f8f4c098bc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f4c091890 con 0x7f8f4c098bc0 2024-09-16T14:42:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f4c094c60 con 0x7f8f4c0968c0 2024-09-16T14:42:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f4c09a880 con 0x7f8f4c096ca0 2024-09-16T14:42:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 10 monclient(hunting): authenticate will time out at 981.269736s 2024-09-16T14:42:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f49d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8f4c096ca0 0x7f8f4c1ce860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48492/0 (socket says 172.21.15.26:48492) 2024-09-16T14:42:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f49d9b640 1 -- 172.21.15.26:0/899099855 learned_addr learned my addr 172.21.15.26:0/899099855 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301491339 0 0) 0x7f8f4c091890 con 0x7f8f4c098bc0 2024-09-16T14:42:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): my global_id is 4422 2024-09-16T14:42:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f2c003530 con 0x7f8f4c098bc0 2024-09-16T14:42:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621788496 0 0) 0x7f8f4c09a880 con 0x7f8f4c096ca0 2024-09-16T14:42:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): my global_id is 4409 2024-09-16T14:42:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f4c091890 con 0x7f8f4c096ca0 2024-09-16T14:42:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 456358957 0 0) 0x7f8f2c003530 con 0x7f8f4c098bc0 2024-09-16T14:42:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f4c09a880 con 0x7f8f4c098bc0 2024-09-16T14:42:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8f44001bc0 con 0x7f8f4c098bc0 2024-09-16T14:42:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:42:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): dump: 2024-09-16T14:42:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:37.163 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 262906725 0 0) 0x7f8f4c091890 con 0x7f8f4c096ca0 2024-09-16T14:42:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f2c003530 con 0x7f8f4c096ca0 2024-09-16T14:42:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8f400017c0 con 0x7f8f4c096ca0 2024-09-16T14:42:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): dump: 2024-09-16T14:42:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:37.164 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986431576 0 0) 0x7f8f4c094c60 con 0x7f8f4c0968c0 2024-09-16T14:42:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): my global_id is 4426 2024-09-16T14:42:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f4c091890 con 0x7f8f4c0968c0 2024-09-16T14:42:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3161377333 0 0) 0x7f8f4c09a880 con 0x7f8f4c098bc0 2024-09-16T14:42:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 >> v1:172.21.15.26:6790/0 conn(0x7f8f4c096ca0 legacy=0x7f8f4c1ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 >> v1:172.21.15.189:6789/0 conn(0x7f8f4c0968c0 legacy=0x7f8f4c1c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 monclient: found mon.a 2024-09-16T14:42:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4c072240 con 0x7f8f4c098bc0 2024-09-16T14:42:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient: _finish_auth 0 2024-09-16T14:42:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 5 monclient: authenticate success, global_id 4422 2024-09-16T14:42:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 10 monclient: _renew_subs 2024-09-16T14:42:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 1 -- 172.21.15.26:0/899099855 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4c0915b0 con 0x7f8f4c098bc0 2024-09-16T14:42:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 10 monclient: _renew_subs 2024-09-16T14:42:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:42:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 1 -- 172.21.15.26:0/899099855 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4c08f530 con 0x7f8f4c098bc0 2024-09-16T14:42:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f440020b0 con 0x7f8f4c098bc0 2024-09-16T14:42:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8f44003ba0 con 0x7f8f4c098bc0 2024-09-16T14:42:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:42:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient: dump: 2024-09-16T14:42:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:37.167 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f48d99640 4 set_mon_vals no callback set 2024-09-16T14:42:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f5088c640 1 -- 172.21.15.26:0/899099855 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8f30006590 con 0x7f8f30002830 2024-09-16T14:42:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8f4400e690 con 0x7f8f4c098bc0 2024-09-16T14:42:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3534+0+0 (unknown 534919773 0 0) 0x7f8f44042270 con 0x7f8f4c098bc0 2024-09-16T14:42:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368176177 0 0) 0x7f8f30006590 con 0x7f8f30002830 2024-09-16T14:42:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 10 monclient(hunting): my global_id is 4429 2024-09-16T14:42:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f4c09a880 con 0x7f8f30002830 2024-09-16T14:42:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.148+0000 7f8f49d9b640 10 monclient: get_auth_request con 0x7f8f2c036470 auth_method 0 2024-09-16T14:42:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.152+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 71354408 0 0) 0x7f8f4c09a880 con 0x7f8f30002830 2024-09-16T14:42:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.152+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f30006590 con 0x7f8f30002830 2024-09-16T14:42:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.152+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8f3c001b40 con 0x7f8f30002830 2024-09-16T14:42:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.152+0000 7f8f3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.152+0000 7f8f3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.152+0000 7f8f3affd640 10 monclient: dump: 2024-09-16T14:42:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:37.169 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.152+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3339781221 0 0) 0x7f8f30006590 con 0x7f8f30002830 2024-09-16T14:42:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.152+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f3000a0e0 con 0x7f8f30002830 2024-09-16T14:42:37.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.152+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8f3c0047f0 con 0x7f8f30002830 2024-09-16T14:42:37.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.152+0000 7f8f3affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.152+0000 7f8f3affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.152+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 >> v1:172.21.15.189:6789/0 conn(0x7f8f30002830 legacy=0x7f8f30004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.152+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 >> v1:172.21.15.189:6789/0 conn(0x7f8f30002830 legacy=0x7f8f30004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:42:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.156+0000 7f8f5088c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:42:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.156+0000 7f8f5088c640 1 -- 172.21.15.26:0/899099855 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8f30009da0 con 0x7f8f300090c0 2024-09-16T14:42:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.156+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445764330 0 0) 0x7f8f30009da0 con 0x7f8f300090c0 2024-09-16T14:42:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.156+0000 7f8f3affd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.156+0000 7f8f3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.156+0000 7f8f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.156+0000 7f8f3affd640 10 monclient(hunting): my global_id is 4432 2024-09-16T14:42:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.156+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f30006590 con 0x7f8f300090c0 2024-09-16T14:42:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 499837316 0 0) 0x7f8f30006590 con 0x7f8f300090c0 2024-09-16T14:42:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f30009da0 con 0x7f8f300090c0 2024-09-16T14:42:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8f3c004cf0 con 0x7f8f300090c0 2024-09-16T14:42:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f3affd640 10 monclient: dump: 2024-09-16T14:42:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:37.173 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2710975329 0 0) 0x7f8f30009da0 con 0x7f8f300090c0 2024-09-16T14:42:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8f3000c290 con 0x7f8f300090c0 2024-09-16T14:42:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 1167073946) 0x7f8f3c004f50 con 0x7f8f300090c0 2024-09-16T14:42:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f3affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:42:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f3affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 >> v1:172.21.15.189:6789/0 conn(0x7f8f300090c0 legacy=0x7f8f30009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f3affd640 1 -- 172.21.15.26:0/899099855 >> v1:172.21.15.189:6789/0 conn(0x7f8f300090c0 legacy=0x7f8f30009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:42:37.174 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:42:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:42:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:42:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:42:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 8, 2024-09-16T14:42:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:42:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:42:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:42:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:42:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 19, 2024-09-16T14:42:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:42:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:42:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:42:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:42:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:42:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:42:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:42:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:42:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:42:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:42:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:42:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:42:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:42:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:42:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:42:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:42:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:42:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:42:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:42:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:42:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:42:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:42:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:42:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:42:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:42:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:42:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:42:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:42:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:42:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:42:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:42:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:42:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:42:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:42:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:42:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:42:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:42:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:42:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:42:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:42:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:42:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:42:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:42:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:42:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:42:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:42:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:42:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:42:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:42:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:42:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:42:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:42:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:42:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:42:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:42:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:42:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:42:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:42:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:42:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:42:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:42:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:42:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:42:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:42:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:42:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:42:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:42:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:42:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:42:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:42:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:42:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:42:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:42:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:42:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:42:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:42:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:42:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:42:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:42:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:42:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:42:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:42:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:42:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:42:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:42:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:42:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:42:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:42:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:42:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:42:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.183 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:42:37.183 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:42:37.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.183 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:37.183 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:37.183 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:42:37.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:37.183 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:37.183 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:42:37.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.183 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:37.183 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:37.183 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:42:37.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:42:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:42:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:42:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:42:37.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:37.185 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:37.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:37.185 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:42:37.185 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:42:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f5088c640 1 -- 172.21.15.26:0/899099855 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8f2c036470 legacy=0x7f8f2c038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f5088c640 10 monclient: shutdown 2024-09-16T14:42:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f5088c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f5088c640 1 -- 172.21.15.26:0/899099855 >> v1:172.21.15.26:6789/0 conn(0x7f8f4c098bc0 legacy=0x7f8f4c1d1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f4ad9d640 1 -- 172.21.15.26:0/899099855 reap_dead start 2024-09-16T14:42:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f4ad9d640 1 -- 172.21.15.26:0/899099855 reap_dead start 2024-09-16T14:42:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f5088c640 1 -- 172.21.15.26:0/899099855 shutdown_connections 2024-09-16T14:42:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f5088c640 1 -- 172.21.15.26:0/899099855 >> 172.21.15.26:0/899099855 conn(0x7f8f4c0a0360 msgr2=0x7f8f4c09f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:37.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f5088c640 1 -- 172.21.15.26:0/899099855 shutdown_connections 2024-09-16T14:42:37.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:37.160+0000 7f8f5088c640 1 -- 172.21.15.26:0/899099855 wait complete. 2024-09-16T14:42:37.187 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:42:37.187 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T14:42:37.187 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-16T14:42:37.187 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T14:42:37.187 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:42:37.188 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T14:42:37.188 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:42:37.198 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T14:42:37.198 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T14:42:37.198 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T14:42:37.198 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T14:42:52.200 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T14:42:52.200 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T14:42:52.201 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:42:52.201 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:42:52.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.328+0000 7fe3fafed640 10 monclient: get_monmap_and_config 2024-09-16T14:42:52.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.328+0000 7fe3fafed640 10 monclient: build_initial_monmap 2024-09-16T14:42:52.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.328+0000 7fe3fafed640 10 monclient: monmap: 2024-09-16T14:42:52.331 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:52.331 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:52.331 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:52.331973+0000 2024-09-16T14:42:52.331 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:52.331973+0000 2024-09-16T14:42:52.331 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:52.332 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:52.332 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:52.332 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:52.332 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:52.332 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.328+0000 7fe3fafed640 1 Processor -- start 2024-09-16T14:42:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.328+0000 7fe3fafed640 1 -- start start 2024-09-16T14:42:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.328+0000 7fe3fafed640 10 monclient: init 2024-09-16T14:42:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.328+0000 7fe3fafed640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.328+0000 7fe3fafed640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.328+0000 7fe3fafed640 10 monclient(hunting): picked mon.noname-b con 0x7fe3f4134e40 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.328+0000 7fe3fafed640 10 monclient(hunting): picked mon.noname-c con 0x7fe3f4099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.328+0000 7fe3fafed640 10 monclient(hunting): picked mon.noname-a con 0x7fe3f4135220 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.328+0000 7fe3fafed640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f4099e80 con 0x7fe3f4135220 2024-09-16T14:42:52.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.328+0000 7fe3fafed640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f40719d0 con 0x7fe3f4099a70 2024-09-16T14:42:52.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.328+0000 7fe3fafed640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f409e1e0 con 0x7fe3f4134e40 2024-09-16T14:42:52.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.328+0000 7fe3fafed640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:52.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.328+0000 7fe3fafed640 10 monclient(hunting): authenticate will time out at 996.451995s 2024-09-16T14:42:52.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.332+0000 7fe3f958a640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fe3f4135220 legacy=0x7fe3f413db30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:42:52.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.332+0000 7fe3f8d89640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe3f4134e40 0x7fe3f4070bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60894/0 (socket says 172.21.15.26:60894) 2024-09-16T14:42:52.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.332+0000 7fe3f8d89640 1 -- 172.21.15.26:0/2776618712 learned_addr learned my addr 172.21.15.26:0/2776618712 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.536+0000 7fe3f958a640 1 -- 172.21.15.26:0/2776618712 >> v1:172.21.15.26:6789/0 conn(0x7fe3f4135220 legacy=0x7fe3f413db30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:42:52.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:52.940+0000 7fe3f958a640 1 -- 172.21.15.26:0/2776618712 >> v1:172.21.15.26:6789/0 conn(0x7fe3f4135220 legacy=0x7fe3f413db30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:42:53.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:53.744+0000 7fe3f958a640 1 -- 172.21.15.26:0/2776618712 >> v1:172.21.15.26:6789/0 conn(0x7fe3f4135220 legacy=0x7fe3f413db30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:42:55.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.332+0000 7fe3f27fc640 10 monclient(hunting): tick 2024-09-16T14:42:55.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.332+0000 7fe3f27fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:42:55.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.332+0000 7fe3f27fc640 1 monclient(hunting): continuing hunt 2024-09-16T14:42:55.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.332+0000 7fe3f27fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:42:55.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.332+0000 7fe3f27fc640 1 -- 172.21.15.26:0/2776618712 >> v1:172.21.15.26:6790/0 conn(0x7fe3f4134e40 legacy=0x7fe3f4070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:55.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.332+0000 7fe3f27fc640 1 -- 172.21.15.26:0/2776618712 >> v1:172.21.15.189:6789/0 conn(0x7fe3f4099a70 legacy=0x7fe3f40712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:55.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.332+0000 7fe3f27fc640 1 -- 172.21.15.26:0/2776618712 >> v1:172.21.15.26:6789/0 conn(0x7fe3f4135220 legacy=0x7fe3f413db30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:42:55.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.332+0000 7fe3f27fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:42:55.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.332+0000 7fe3f27fc640 10 monclient(hunting): picked mon.noname-a con 0x7fe3e00036d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:55.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.332+0000 7fe3f27fc640 10 monclient(hunting): picked mon.noname-c con 0x7fe3e0007360 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:55.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.332+0000 7fe3f27fc640 10 monclient(hunting): picked mon.noname-b con 0x7fe3e000b090 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:55.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.332+0000 7fe3f27fc640 1 -- 172.21.15.26:0/2776618712 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f409e7d0 con 0x7fe3e00036d0 2024-09-16T14:42:55.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.332+0000 7fe3f27fc640 1 -- 172.21.15.26:0/2776618712 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3e000fd70 con 0x7fe3e0007360 2024-09-16T14:42:55.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.332+0000 7fe3f27fc640 1 -- 172.21.15.26:0/2776618712 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3e0010f60 con 0x7fe3e000b090 2024-09-16T14:42:55.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.332+0000 7fe3f27fc640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:55.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.332+0000 7fe3f8d89640 1 -- 172.21.15.26:0/2776618712 >> v1:172.21.15.26:6789/0 conn(0x7fe3e00036d0 legacy=0x7fe3e0005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:42:55.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.536+0000 7fe3f8d89640 1 -- 172.21.15.26:0/2776618712 >> v1:172.21.15.26:6789/0 conn(0x7fe3e00036d0 legacy=0x7fe3e0005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:42:55.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:55.940+0000 7fe3f8d89640 1 -- 172.21.15.26:0/2776618712 >> v1:172.21.15.26:6789/0 conn(0x7fe3e00036d0 legacy=0x7fe3e0005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:42:56.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:56.744+0000 7fe3f8d89640 1 -- 172.21.15.26:0/2776618712 >> v1:172.21.15.26:6789/0 conn(0x7fe3e00036d0 legacy=0x7fe3e0005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:42:57.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.244+0000 7fe3f37fe640 1 -- 172.21.15.26:0/2776618712 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41671471 0 0) 0x7fe3e000fd70 con 0x7fe3e0007360 2024-09-16T14:42:57.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.244+0000 7fe3f37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:42:57.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.244+0000 7fe3f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:57.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.244+0000 7fe3f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:57.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.244+0000 7fe3f37fe640 10 monclient(hunting): my global_id is 4435 2024-09-16T14:42:57.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.244+0000 7fe3f37fe640 1 -- 172.21.15.26:0/2776618712 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3dc003530 con 0x7fe3e0007360 2024-09-16T14:42:57.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 1 -- 172.21.15.26:0/2776618712 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3422487168 0 0) 0x7fe3dc003530 con 0x7fe3e0007360 2024-09-16T14:42:57.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 1 -- 172.21.15.26:0/2776618712 >> v1:172.21.15.26:6790/0 conn(0x7fe3e000b090 legacy=0x7fe3e000d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 1 -- 172.21.15.26:0/2776618712 >> v1:172.21.15.26:6789/0 conn(0x7fe3e00036d0 legacy=0x7fe3e0005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:42:57.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:42:57.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 1 monclient: found mon.noname-c 2024-09-16T14:42:57.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:57.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:42:57.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 1 -- 172.21.15.26:0/2776618712 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e00121b0 con 0x7fe3e0007360 2024-09-16T14:42:57.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 10 monclient: _finish_auth 0 2024-09-16T14:42:57.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:42:57.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:57.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 1 -- 172.21.15.26:0/2776618712 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe3e8001520 con 0x7fe3e0007360 2024-09-16T14:42:57.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:57.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 10 monclient: dump: 2024-09-16T14:42:57.251 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.251 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.251 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.251 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.251 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.251 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.251 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.251 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.251 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.251 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3fafed640 5 monclient: authenticate success, global_id 4435 2024-09-16T14:42:57.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3fafed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 1 -- 172.21.15.26:0/2776618712 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3e80017c0 con 0x7fe3e0007360 2024-09-16T14:42:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 1 -- 172.21.15.26:0/2776618712 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe3e8001b50 con 0x7fe3e0007360 2024-09-16T14:42:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3fafed640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3fafed640 4 set_mon_vals no callback set 2024-09-16T14:42:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3f37fe640 10 monclient: dump: 2024-09-16T14:42:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.252 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.253 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.253 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.253 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.253 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.253 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3fafed640 10 monclient: shutdown 2024-09-16T14:42:57.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3fafed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:57.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3fafed640 1 -- 172.21.15.26:0/2776618712 >> v1:172.21.15.189:6789/0 conn(0x7fe3e0007360 legacy=0x7fe3e00097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3fafed640 1 -- 172.21.15.26:0/2776618712 shutdown_connections 2024-09-16T14:42:57.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3fafed640 1 -- 172.21.15.26:0/2776618712 >> 172.21.15.26:0/2776618712 conn(0x7fe3f409fb50 msgr2=0x7fe3f409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:57.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3fafed640 1 -- 172.21.15.26:0/2776618712 shutdown_connections 2024-09-16T14:42:57.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3fafed640 1 -- 172.21.15.26:0/2776618712 wait complete. 2024-09-16T14:42:57.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3fafed640 10 monclient: build_initial_monmap 2024-09-16T14:42:57.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3fafed640 10 monclient: monmap: 2024-09-16T14:42:57.257 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:57.258 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3fafed640 1 Processor -- start 2024-09-16T14:42:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3fafed640 1 -- start start 2024-09-16T14:42:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.248+0000 7fe3fafed640 10 monclient: init 2024-09-16T14:42:57.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:42:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 10 monclient(hunting): picked mon.noname-c con 0x7fe3e00036d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 10 monclient(hunting): picked mon.noname-b con 0x7fe3e0007360 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f958a640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fe3e000b090 legacy=0x7fe3f41d2040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:42:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 10 monclient(hunting): picked mon.noname-a con 0x7fe3e000b090 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f4122390 con 0x7fe3e000b090 2024-09-16T14:42:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f4122a20 con 0x7fe3e0007360 2024-09-16T14:42:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f409bd00 con 0x7fe3e00036d0 2024-09-16T14:42:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 10 monclient(hunting): authenticate will time out at 1001.373018s 2024-09-16T14:42:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f3fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe3e0007360 0x7fe3f41ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.26:44194/0 (socket says 172.21.15.26:44194) 2024-09-16T14:42:57.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f3fff640 1 -- 172.21.15.26:0/4112524958 learned_addr learned my addr 172.21.15.26:0/4112524958 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f8d89640 1 --1- 172.21.15.26:0/4112524958 >> v1:172.21.15.26:6790/0 conn(0x7fe3e00036d0 0x7fe3f41c9930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54524/0 (socket says 172.21.15.26:54524) 2024-09-16T14:42:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864485550 0 0) 0x7fe3f4122a20 con 0x7fe3e0007360 2024-09-16T14:42:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient(hunting): my global_id is 4438 2024-09-16T14:42:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3e0010470 con 0x7fe3e0007360 2024-09-16T14:42:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528617645 0 0) 0x7fe3f409bd00 con 0x7fe3e00036d0 2024-09-16T14:42:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:42:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient(hunting): my global_id is 4412 2024-09-16T14:42:57.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3f4122a20 con 0x7fe3e00036d0 2024-09-16T14:42:57.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1674985191 0 0) 0x7fe3e0010470 con 0x7fe3e0007360 2024-09-16T14:42:57.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3f409bd00 con 0x7fe3e0007360 2024-09-16T14:42:57.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe3e8001b90 con 0x7fe3e0007360 2024-09-16T14:42:57.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:57.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient(hunting): dump: 2024-09-16T14:42:57.261 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.261 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.261 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.261 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.261 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.261 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.262 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.262 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.262 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.262 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:57.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1450807606 0 0) 0x7fe3f4122a20 con 0x7fe3e00036d0 2024-09-16T14:42:57.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3e0010470 con 0x7fe3e00036d0 2024-09-16T14:42:57.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe3e4001790 con 0x7fe3e00036d0 2024-09-16T14:42:57.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:57.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient(hunting): dump: 2024-09-16T14:42:57.262 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.262 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.263 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.263 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.263 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.263 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.263 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.263 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.263 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.263 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 331500806 0 0) 0x7fe3e0010470 con 0x7fe3e00036d0 2024-09-16T14:42:57.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 >> v1:172.21.15.189:6789/0 conn(0x7fe3e0007360 legacy=0x7fe3f41ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 >> v1:172.21.15.26:6789/0 conn(0x7fe3e000b090 legacy=0x7fe3f41d2040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:42:57.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient: _finish_hunting 0 2024-09-16T14:42:57.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 monclient: found mon.c 2024-09-16T14:42:57.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:57.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:57.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f4134410 con 0x7fe3e00036d0 2024-09-16T14:42:57.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient: _finish_auth 0 2024-09-16T14:42:57.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient: _check_auth_tickets 2024-09-16T14:42:57.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:57.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 5 monclient: authenticate success, global_id 4412 2024-09-16T14:42:57.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 10 monclient: _renew_subs 2024-09-16T14:42:57.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:57.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 1 -- 172.21.15.26:0/4112524958 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f4121e50 con 0x7fe3e00036d0 2024-09-16T14:42:57.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 10 monclient: _renew_subs 2024-09-16T14:42:57.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:57.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3fafed640 1 -- 172.21.15.26:0/4112524958 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f4123f40 con 0x7fe3e00036d0 2024-09-16T14:42:57.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3e4001c60 con 0x7fe3e00036d0 2024-09-16T14:42:57.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:57.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe3e4004990 con 0x7fe3e00036d0 2024-09-16T14:42:57.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:57.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f17fa640 10 monclient: dump: 2024-09-16T14:42:57.265 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.265 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.265 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.265 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.265 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.265 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.266 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.266 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.266 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.266 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.252+0000 7fe3f27fc640 4 set_mon_vals no callback set 2024-09-16T14:42:57.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.256+0000 7fe3fafed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:57.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.256+0000 7fe3fafed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:57.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.256+0000 7fe3fafed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:57.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.256+0000 7fe3fafed640 1 -- 172.21.15.26:0/4112524958 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d4005050 con 0x7fe3e00036d0 2024-09-16T14:42:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.256+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe3e4004af0 con 0x7fe3e00036d0 2024-09-16T14:42:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.256+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3534+0+0 (unknown 534919773 0 0) 0x7fe3e4042e00 con 0x7fe3e00036d0 2024-09-16T14:42:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.256+0000 7fe3f3fff640 10 monclient: get_auth_request con 0x7fe3e00375f0 auth_method 0 2024-09-16T14:42:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.260+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe3e4014140 con 0x7fe3e00036d0 2024-09-16T14:42:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.260+0000 7fe3f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:57.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.260+0000 7fe3f17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:57.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.520+0000 7fe3fafed640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:42:57.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.520+0000 7fe3fafed640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:42:57.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.520+0000 7fe3fafed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:57.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.520+0000 7fe3fafed640 1 -- 172.21.15.26:0/4112524958 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe3d4005300 con 0x7fe3e00036d0 2024-09-16T14:42:57.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.520+0000 7fe3f17fa640 1 -- 172.21.15.26:0/4112524958 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2544840207) 0x7fe3e4013f60 con 0x7fe3e00036d0 2024-09-16T14:42:57.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.520+0000 7fe3f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:42:57.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.520+0000 7fe3f17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:57.523 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":10,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:42:57.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.524+0000 7fe3fafed640 1 -- 172.21.15.26:0/4112524958 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe3e00375f0 legacy=0x7fe3e0039ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.524+0000 7fe3fafed640 10 monclient: shutdown 2024-09-16T14:42:57.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.524+0000 7fe3fafed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:57.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.524+0000 7fe3fafed640 1 -- 172.21.15.26:0/4112524958 >> v1:172.21.15.26:6790/0 conn(0x7fe3e00036d0 legacy=0x7fe3f41c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.524+0000 7fe3fafed640 1 -- 172.21.15.26:0/4112524958 shutdown_connections 2024-09-16T14:42:57.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.524+0000 7fe3fafed640 1 -- 172.21.15.26:0/4112524958 >> 172.21.15.26:0/4112524958 conn(0x7fe3f409fb50 msgr2=0x7fe3f40a2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:57.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.524+0000 7fe3fafed640 1 -- 172.21.15.26:0/4112524958 shutdown_connections 2024-09-16T14:42:57.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.524+0000 7fe3fafed640 1 -- 172.21.15.26:0/4112524958 wait complete. 2024-09-16T14:42:57.542 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:42:57.542 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:42:57.542 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:42:57.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.660+0000 7f3ad84cb640 10 monclient: get_monmap_and_config 2024-09-16T14:42:57.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.660+0000 7f3ad84cb640 10 monclient: build_initial_monmap 2024-09-16T14:42:57.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.660+0000 7f3ad84cb640 10 monclient: monmap: 2024-09-16T14:42:57.664 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:57.664 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.665 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:57.664901+0000 2024-09-16T14:42:57.665 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:57.664901+0000 2024-09-16T14:42:57.665 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:57.665 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.665 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:57.665 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:57.665 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:57.665 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 1 Processor -- start 2024-09-16T14:42:57.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 1 -- start start 2024-09-16T14:42:57.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 10 monclient: init 2024-09-16T14:42:57.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:57.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:42:57.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 10 monclient(hunting): picked mon.noname-a con 0x7f3ad0134e40 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:57.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 10 monclient(hunting): picked mon.noname-c con 0x7f3ad0099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:57.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 10 monclient(hunting): picked mon.noname-b con 0x7f3ad0135220 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:57.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad0099e80 con 0x7f3ad0134e40 2024-09-16T14:42:57.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad00719d0 con 0x7f3ad0099a70 2024-09-16T14:42:57.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad009e1e0 con 0x7f3ad0135220 2024-09-16T14:42:57.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:57.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 10 monclient(hunting): authenticate will time out at 1001.785109s 2024-09-16T14:42:57.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad6267640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f3ad0134e40 legacy=0x7f3ad0070bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:42:57.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad6a68640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3ad0135220 0x7f3ad013db30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54528/0 (socket says 172.21.15.26:54528) 2024-09-16T14:42:57.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad6a68640 1 -- 172.21.15.26:0/1087190728 learned_addr learned my addr 172.21.15.26:0/1087190728 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:57.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 1 -- 172.21.15.26:0/1087190728 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021719043 0 0) 0x7f3ad009e1e0 con 0x7f3ad0135220 2024-09-16T14:42:57.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient(hunting): _negotiate 2024-09-16T14:42:57.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:57.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:57.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient(hunting): my global_id is 4415 2024-09-16T14:42:57.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 1 -- 172.21.15.26:0/1087190728 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3abc003530 con 0x7f3ad0135220 2024-09-16T14:42:57.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 1 -- 172.21.15.26:0/1087190728 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222484321 0 0) 0x7f3ad00719d0 con 0x7f3ad0099a70 2024-09-16T14:42:57.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient(hunting): _negotiate 2024-09-16T14:42:57.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:57.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:57.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient(hunting): my global_id is 4441 2024-09-16T14:42:57.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 1 -- 172.21.15.26:0/1087190728 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ad009e1e0 con 0x7f3ad0099a70 2024-09-16T14:42:57.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 1 -- 172.21.15.26:0/1087190728 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1694439752 0 0) 0x7f3abc003530 con 0x7f3ad0135220 2024-09-16T14:42:57.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 1 -- 172.21.15.26:0/1087190728 >> v1:172.21.15.189:6789/0 conn(0x7f3ad0099a70 legacy=0x7f3ad00712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 1 -- 172.21.15.26:0/1087190728 >> v1:172.21.15.26:6789/0 conn(0x7f3ad0134e40 legacy=0x7f3ad0070bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:42:57.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient: _finish_hunting 0 2024-09-16T14:42:57.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 1 monclient: found mon.noname-b 2024-09-16T14:42:57.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:57.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:57.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 1 -- 172.21.15.26:0/1087190728 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad009e7d0 con 0x7f3ad0135220 2024-09-16T14:42:57.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient: _finish_auth 0 2024-09-16T14:42:57.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient: _check_auth_tickets 2024-09-16T14:42:57.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:57.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 1 -- 172.21.15.26:0/1087190728 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ac80019a0 con 0x7f3ad0135220 2024-09-16T14:42:57.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:57.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient: dump: 2024-09-16T14:42:57.678 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.678 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.678 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.678 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.678 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.678 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.678 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.678 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.678 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.678 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 5 monclient: authenticate success, global_id 4415 2024-09-16T14:42:57.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:57.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 1 -- 172.21.15.26:0/1087190728 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ac8001bc0 con 0x7f3ad0135220 2024-09-16T14:42:57.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:57.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:57.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 4 set_mon_vals no callback set 2024-09-16T14:42:57.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 1 -- 172.21.15.26:0/1087190728 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ac8001ed0 con 0x7f3ad0135220 2024-09-16T14:42:57.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:57.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad5265640 10 monclient: dump: 2024-09-16T14:42:57.679 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.680 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.680 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.680 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.680 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.680 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.680 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.680 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.680 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.680 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 10 monclient: shutdown 2024-09-16T14:42:57.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:57.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 1 -- 172.21.15.26:0/1087190728 >> v1:172.21.15.26:6790/0 conn(0x7f3ad0135220 legacy=0x7f3ad013db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 1 -- 172.21.15.26:0/1087190728 shutdown_connections 2024-09-16T14:42:57.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 1 -- 172.21.15.26:0/1087190728 >> 172.21.15.26:0/1087190728 conn(0x7f3ad009fb50 msgr2=0x7f3ad009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:57.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 1 -- 172.21.15.26:0/1087190728 shutdown_connections 2024-09-16T14:42:57.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 1 -- 172.21.15.26:0/1087190728 wait complete. 2024-09-16T14:42:57.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 10 monclient: build_initial_monmap 2024-09-16T14:42:57.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 10 monclient: monmap: 2024-09-16T14:42:57.681 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:57.681 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:57.681 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:57.681 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:57.682 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:57.682 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:57.682 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:57.682 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:57.682 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:57.682 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 1 Processor -- start 2024-09-16T14:42:57.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 1 -- start start 2024-09-16T14:42:57.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.664+0000 7f3ad84cb640 10 monclient: init 2024-09-16T14:42:57.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:57.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:57.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 10 monclient(hunting): picked mon.noname-b con 0x7f3ad0099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:57.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 10 monclient(hunting): picked mon.noname-c con 0x7f3ad0134e40 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:57.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 10 monclient(hunting): picked mon.noname-a con 0x7f3ad0135220 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:57.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad0124440 con 0x7f3ad0135220 2024-09-16T14:42:57.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad0122cb0 con 0x7f3ad0099a70 2024-09-16T14:42:57.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad009c040 con 0x7f3ad0134e40 2024-09-16T14:42:57.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:57.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 10 monclient(hunting): authenticate will time out at 1001.788151s 2024-09-16T14:42:57.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad6a68640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f3ad0135220 legacy=0x7f3ad01d2190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:42:57.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad5a66640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3ad0134e40 0x7f3ad01cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54544/0 (socket says 172.21.15.26:54544) 2024-09-16T14:42:57.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad5a66640 1 -- 172.21.15.26:0/3783521545 learned_addr learned my addr 172.21.15.26:0/3783521545 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:57.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398946851 0 0) 0x7f3ad009c040 con 0x7f3ad0134e40 2024-09-16T14:42:57.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:57.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:57.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:57.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient(hunting): my global_id is 4418 2024-09-16T14:42:57.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ab4003530 con 0x7f3ad0134e40 2024-09-16T14:42:57.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048798983 0 0) 0x7f3ad0122cb0 con 0x7f3ad0099a70 2024-09-16T14:42:57.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:57.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:57.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:57.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient(hunting): my global_id is 4444 2024-09-16T14:42:57.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ad009c040 con 0x7f3ad0099a70 2024-09-16T14:42:57.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1777671521 0 0) 0x7f3ab4003530 con 0x7f3ad0134e40 2024-09-16T14:42:57.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ad0122cb0 con 0x7f3ad0134e40 2024-09-16T14:42:57.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ac00018c0 con 0x7f3ad0134e40 2024-09-16T14:42:57.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:57.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:57.685 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.685 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.686 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.686 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.686 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.686 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.686 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.686 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.686 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.686 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:57.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3257331981 0 0) 0x7f3ad0122cb0 con 0x7f3ad0134e40 2024-09-16T14:42:57.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 >> v1:172.21.15.189:6789/0 conn(0x7f3ad0099a70 legacy=0x7f3ad01c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 >> v1:172.21.15.26:6789/0 conn(0x7f3ad0135220 legacy=0x7f3ad01d2190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:42:57.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:57.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 monclient: found mon.c 2024-09-16T14:42:57.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:57.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:57.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad009a240 con 0x7f3ad0134e40 2024-09-16T14:42:57.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:57.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:57.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:57.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 5 monclient: authenticate success, global_id 4418 2024-09-16T14:42:57.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 10 monclient: _renew_subs 2024-09-16T14:42:57.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:57.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 1 -- 172.21.15.26:0/3783521545 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad01220e0 con 0x7f3ad0134e40 2024-09-16T14:42:57.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 10 monclient: _renew_subs 2024-09-16T14:42:57.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:57.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 1 -- 172.21.15.26:0/3783521545 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad01214f0 con 0x7f3ad0134e40 2024-09-16T14:42:57.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ac0001c70 con 0x7f3ad0134e40 2024-09-16T14:42:57.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:57.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ac0003a70 con 0x7f3ad0134e40 2024-09-16T14:42:57.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:57.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient: dump: 2024-09-16T14:42:57.689 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.689 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.689 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.689 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.689 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.689 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.689 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.689 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.690 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad4a64640 4 set_mon_vals no callback set 2024-09-16T14:42:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad84cb640 1 -- 172.21.15.26:0/3783521545 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3ab8006590 con 0x7f3ab8002830 2024-09-16T14:42:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3ac0003bd0 con 0x7f3ad0134e40 2024-09-16T14:42:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3534+0+0 (unknown 534919773 0 0) 0x7f3ac0043140 con 0x7f3ad0134e40 2024-09-16T14:42:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795018338 0 0) 0x7f3ab8006590 con 0x7f3ab8002830 2024-09-16T14:42:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient(hunting): my global_id is 4421 2024-09-16T14:42:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ad0122cb0 con 0x7f3ab8002830 2024-09-16T14:42:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ad6a68640 10 monclient: get_auth_request con 0x7f3ab4036130 auth_method 0 2024-09-16T14:42:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1013845905 0 0) 0x7f3ad0122cb0 con 0x7f3ab8002830 2024-09-16T14:42:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ab8006590 con 0x7f3ab8002830 2024-09-16T14:42:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3acc003ff0 con 0x7f3ab8002830 2024-09-16T14:42:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient: dump: 2024-09-16T14:42:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.692 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1590203015 0 0) 0x7f3ab8006590 con 0x7f3ab8002830 2024-09-16T14:42:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ab800a0e0 con 0x7f3ab8002830 2024-09-16T14:42:57.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f3acc005550 con 0x7f3ab8002830 2024-09-16T14:42:57.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:57.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:57.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 >> v1:172.21.15.26:6790/0 conn(0x7f3ab8002830 legacy=0x7f3ab8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.668+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 >> v1:172.21.15.26:6790/0 conn(0x7f3ab8002830 legacy=0x7f3ab8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:42:57.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ad84cb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:42:57.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ad84cb640 1 -- 172.21.15.26:0/3783521545 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3ab8009d80 con 0x7f3ab80090a0 2024-09-16T14:42:57.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573171808 0 0) 0x7f3ab8009d80 con 0x7f3ab80090a0 2024-09-16T14:42:57.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 10 monclient(hunting): my global_id is 4424 2024-09-16T14:42:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ab8006590 con 0x7f3ab80090a0 2024-09-16T14:42:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1422087270 0 0) 0x7f3ab8006590 con 0x7f3ab80090a0 2024-09-16T14:42:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ab8009d80 con 0x7f3ab80090a0 2024-09-16T14:42:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3acc004790 con 0x7f3ab80090a0 2024-09-16T14:42:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 10 monclient: dump: 2024-09-16T14:42:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.696 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.696 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.696 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.696 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.696 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.696 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.696 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.696 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.696 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 898657580 0 0) 0x7f3ab8009d80 con 0x7f3ab80090a0 2024-09-16T14:42:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3ab800c290 con 0x7f3ab80090a0 2024-09-16T14:42:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4036 (unknown 0 0 336913805) 0x7f3acc0056c0 con 0x7f3ab80090a0 2024-09-16T14:42:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:42:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 >> v1:172.21.15.26:6790/0 conn(0x7f3ab80090a0 legacy=0x7f3ab8009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ac6ffd640 1 -- 172.21.15.26:0/3783521545 >> v1:172.21.15.26:6790/0 conn(0x7f3ab80090a0 legacy=0x7f3ab8009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:42:57.697 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:42:57.697 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:42:57.698 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:42:57.698 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:42:57.698 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-09-16T14:42:57.698 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:42:57.698 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:42:57.698 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:42:57.698 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:57.698 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:42:57.698 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:42:57.698 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:42:57.699 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:42:57.699 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:42:57.699 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:42:57.699 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:42:57.699 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:42:57.699 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:42:57.699 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:42:57.699 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:42:57.699 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:42:57.699 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:42:57.699 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:42:57.699 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:57.700 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:42:57.700 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:42:57.700 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:42:57.700 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:42:57.700 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:42:57.700 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:42:57.700 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:42:57.700 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:42:57.700 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:42:57.700 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:42:57.700 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:42:57.700 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:42:57.701 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:57.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.701 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:42:57.701 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:42:57.701 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:42:57.701 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:42:57.701 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:42:57.701 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:42:57.701 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:42:57.701 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:42:57.701 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:42:57.702 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:42:57.702 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:42:57.702 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:42:57.702 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:42:57.702 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:42:57.702 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:42:57.702 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:42:57.702 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:42:57.702 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:42:57.702 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:42:57.702 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:42:57.703 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:42:57.703 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:42:57.703 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:42:57.703 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:42:57.703 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:42:57.703 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:42:57.703 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:42:57.703 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:57.703 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:42:57.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.703 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:42:57.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.704 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:42:57.704 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:42:57.704 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:42:57.704 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:42:57.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.704 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:42:57.704 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:42:57.704 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:42:57.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:57.705 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:57.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.705 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:42:57.705 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:42:57.705 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:42:57.705 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:42:57.705 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:42:57.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.705 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:42:57.705 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:42:57.706 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:42:57.706 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:42:57.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.706 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:42:57.706 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:42:57.706 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:42:57.706 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:57.706 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:57.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.706 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:42:57.707 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:42:57.707 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:42:57.707 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:42:57.707 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:42:57.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.707 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:42:57.707 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:42:57.707 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:42:57.707 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:42:57.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.707 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:42:57.708 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:42:57.708 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:42:57.708 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:57.708 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:57.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.708 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:42:57.708 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:42:57.708 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:42:57.708 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:42:57.708 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:42:57.709 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:57.709 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:57.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.709 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:42:57.709 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:42:57.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.709 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:57.709 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:57.709 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:42:57.709 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:57.709 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:57.710 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:42:57.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.710 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:57.710 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:57.710 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:42:57.710 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:57.710 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:57.710 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:42:57.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.711 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:57.711 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:57.711 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:42:57.711 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:57.711 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:57.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.711 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:42:57.711 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:42:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ad84cb640 1 -- 172.21.15.26:0/3783521545 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3ab4036130 legacy=0x7f3ab40385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ad84cb640 10 monclient: shutdown 2024-09-16T14:42:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ad84cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ad84cb640 1 -- 172.21.15.26:0/3783521545 >> v1:172.21.15.26:6790/0 conn(0x7f3ad0134e40 legacy=0x7f3ad01cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ad6a68640 1 -- 172.21.15.26:0/3783521545 reap_dead start 2024-09-16T14:42:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ad6a68640 1 -- 172.21.15.26:0/3783521545 reap_dead start 2024-09-16T14:42:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ad84cb640 1 -- 172.21.15.26:0/3783521545 shutdown_connections 2024-09-16T14:42:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ad84cb640 1 -- 172.21.15.26:0/3783521545 >> 172.21.15.26:0/3783521545 conn(0x7f3ad009fb50 msgr2=0x7f3ad00a0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ad84cb640 1 -- 172.21.15.26:0/3783521545 shutdown_connections 2024-09-16T14:42:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.676+0000 7f3ad84cb640 1 -- 172.21.15.26:0/3783521545 wait complete. 2024-09-16T14:42:57.714 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:42:57.714 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T14:42:57.714 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:42:57.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.828+0000 7f94d8282640 10 monclient: get_monmap_and_config 2024-09-16T14:42:57.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.828+0000 7f94d8282640 10 monclient: build_initial_monmap 2024-09-16T14:42:57.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.828+0000 7f94d8282640 10 monclient: monmap: 2024-09-16T14:42:57.831 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:57.831 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.831 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:42:57.832082+0000 2024-09-16T14:42:57.831 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:42:57.832082+0000 2024-09-16T14:42:57.831 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:57.832 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.832 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:57.832 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:42:57.832 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:42:57.832 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.828+0000 7f94d8282640 1 Processor -- start 2024-09-16T14:42:57.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.828+0000 7f94d8282640 1 -- start start 2024-09-16T14:42:57.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.828+0000 7f94d8282640 10 monclient: init 2024-09-16T14:42:57.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.828+0000 7f94d8282640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:57.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:42:57.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient(hunting): picked mon.noname-c con 0x7f94d009a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:57.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient(hunting): picked mon.noname-b con 0x7f94d009d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:57.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient(hunting): picked mon.noname-a con 0x7f94d009b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:57.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d009dc60 con 0x7f94d009b040 2024-09-16T14:42:57.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d0099ae0 con 0x7f94d009a550 2024-09-16T14:42:57.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d009b8f0 con 0x7f94d009d1b0 2024-09-16T14:42:57.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:57.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient(hunting): authenticate will time out at 1001.952114s 2024-09-16T14:42:57.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d681f640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f94d009b040 legacy=0x7f94d013a190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:42:57.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d581d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f94d009d1b0 0x7f94d0136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54578/0 (socket says 172.21.15.26:54578) 2024-09-16T14:42:57.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d581d640 1 -- 172.21.15.26:0/1688653068 learned_addr learned my addr 172.21.15.26:0/1688653068 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:57.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 1 -- 172.21.15.26:0/1688653068 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931864181 0 0) 0x7f94d009b8f0 con 0x7f94d009d1b0 2024-09-16T14:42:57.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 10 monclient(hunting): _negotiate 2024-09-16T14:42:57.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:57.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:57.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 10 monclient(hunting): my global_id is 4427 2024-09-16T14:42:57.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 1 -- 172.21.15.26:0/1688653068 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94bc003530 con 0x7f94d009d1b0 2024-09-16T14:42:57.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 1 -- 172.21.15.26:0/1688653068 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1845937457 0 0) 0x7f94bc003530 con 0x7f94d009d1b0 2024-09-16T14:42:57.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 1 -- 172.21.15.26:0/1688653068 >> v1:172.21.15.189:6789/0 conn(0x7f94d009a550 legacy=0x7f94d009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 1 -- 172.21.15.26:0/1688653068 >> v1:172.21.15.26:6789/0 conn(0x7f94d009b040 legacy=0x7f94d013a190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:42:57.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 10 monclient: _finish_hunting 0 2024-09-16T14:42:57.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 1 monclient: found mon.noname-b 2024-09-16T14:42:57.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:57.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:42:57.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 1 -- 172.21.15.26:0/1688653068 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d009bbd0 con 0x7f94d009d1b0 2024-09-16T14:42:57.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 10 monclient: _finish_auth 0 2024-09-16T14:42:57.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 10 monclient: _check_auth_tickets 2024-09-16T14:42:57.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:57.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 1 -- 172.21.15.26:0/1688653068 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f94c0001620 con 0x7f94d009d1b0 2024-09-16T14:42:57.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:42:57.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 10 monclient: dump: 2024-09-16T14:42:57.842 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.842 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.842 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.842 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.842 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.842 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.842 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.842 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.842 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.842 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 5 monclient: authenticate success, global_id 4427 2024-09-16T14:42:57.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:42:57.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 1 -- 172.21.15.26:0/1688653068 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94c0001940 con 0x7f94d009d1b0 2024-09-16T14:42:57.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:57.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 1 -- 172.21.15.26:0/1688653068 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f94c0001cd0 con 0x7f94d009d1b0 2024-09-16T14:42:57.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:57.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d501c640 10 monclient: dump: 2024-09-16T14:42:57.843 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.843 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.843 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.843 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.844 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.844 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.844 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.844 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.844 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.844 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient: get_monmap_and_config success 2024-09-16T14:42:57.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 4 set_mon_vals no callback set 2024-09-16T14:42:57.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient: shutdown 2024-09-16T14:42:57.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:57.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 1 -- 172.21.15.26:0/1688653068 >> v1:172.21.15.26:6790/0 conn(0x7f94d009d1b0 legacy=0x7f94d0136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 1 -- 172.21.15.26:0/1688653068 shutdown_connections 2024-09-16T14:42:57.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 1 -- 172.21.15.26:0/1688653068 >> 172.21.15.26:0/1688653068 conn(0x7f94d009fb50 msgr2=0x7f94d009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:57.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 1 -- 172.21.15.26:0/1688653068 shutdown_connections 2024-09-16T14:42:57.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 1 -- 172.21.15.26:0/1688653068 wait complete. 2024-09-16T14:42:57.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient: build_initial_monmap 2024-09-16T14:42:57.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient: monmap: 2024-09-16T14:42:57.845 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:42:57.845 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:42:57.845 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:42:57.845 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:42:57.845 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:42:57.845 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:42:57.845 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:42:57.845 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:42:57.845 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:42:57.845 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 1 Processor -- start 2024-09-16T14:42:57.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 1 -- start start 2024-09-16T14:42:57.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient: init 2024-09-16T14:42:57.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient: _reopen_session rank -1 2024-09-16T14:42:57.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:42:57.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient(hunting): picked mon.noname-a con 0x7f94d009a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:42:57.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient(hunting): picked mon.noname-c con 0x7f94d009b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:42:57.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient(hunting): picked mon.noname-b con 0x7f94d009d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:42:57.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d01a0a60 con 0x7f94d009a550 2024-09-16T14:42:57.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d00a2580 con 0x7f94d009d1b0 2024-09-16T14:42:57.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d009a240 con 0x7f94d009b040 2024-09-16T14:42:57.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient(hunting): _renew_subs 2024-09-16T14:42:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d8282640 10 monclient(hunting): authenticate will time out at 1001.955318s 2024-09-16T14:42:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d601e640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f94d009a550 legacy=0x7f94d01c90b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:42:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d581d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f94d009b040 0x7f94d01b47c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54588/0 (socket says 172.21.15.26:54588) 2024-09-16T14:42:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94d581d640 1 -- 172.21.15.26:0/980175810 learned_addr learned my addr 172.21.15.26:0/980175810 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:42:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384197393 0 0) 0x7f94d009a240 con 0x7f94d009b040 2024-09-16T14:42:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94c6ffd640 10 monclient(hunting): my global_id is 4430 2024-09-16T14:42:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94b4003530 con 0x7f94d009b040 2024-09-16T14:42:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 186984712 0 0) 0x7f94b4003530 con 0x7f94d009b040 2024-09-16T14:42:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94d009a240 con 0x7f94d009b040 2024-09-16T14:42:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f94c00015e0 con 0x7f94d009b040 2024-09-16T14:42:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:42:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94c6ffd640 10 monclient(hunting): dump: 2024-09-16T14:42:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.848 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.832+0000 7f94c6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:42:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707717129 0 0) 0x7f94d00a2580 con 0x7f94d009d1b0 2024-09-16T14:42:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient(hunting): my global_id is 4450 2024-09-16T14:42:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94b4003530 con 0x7f94d009d1b0 2024-09-16T14:42:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 900349258 0 0) 0x7f94d009a240 con 0x7f94d009b040 2024-09-16T14:42:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 >> v1:172.21.15.189:6789/0 conn(0x7f94d009d1b0 legacy=0x7f94d01d1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 >> v1:172.21.15.26:6789/0 conn(0x7f94d009a550 legacy=0x7f94d01c90b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:42:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:42:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 monclient: found mon.c 2024-09-16T14:42:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:42:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d01a0770 con 0x7f94d009b040 2024-09-16T14:42:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:42:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:42:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:42:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94d8282640 5 monclient: authenticate success, global_id 4430 2024-09-16T14:42:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94d8282640 10 monclient: _renew_subs 2024-09-16T14:42:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94d8282640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94d8282640 1 -- 172.21.15.26:0/980175810 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d019fe20 con 0x7f94d009b040 2024-09-16T14:42:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94d8282640 10 monclient: _renew_subs 2024-09-16T14:42:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94d8282640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:42:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94d8282640 1 -- 172.21.15.26:0/980175810 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d01b5430 con 0x7f94d009b040 2024-09-16T14:42:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94c0004d40 con 0x7f94d009b040 2024-09-16T14:42:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:42:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f94c00050d0 con 0x7f94d009b040 2024-09-16T14:42:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94d481b640 4 set_mon_vals no callback set 2024-09-16T14:42:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:42:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient: dump: 2024-09-16T14:42:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.852 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.852 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.852 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.852 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.852 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.852 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94d8282640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:57.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94d8282640 1 -- 172.21.15.26:0/980175810 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f94b8006590 con 0x7f94b8002830 2024-09-16T14:42:57.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f94c0005230 con 0x7f94d009b040 2024-09-16T14:42:57.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3534+0+0 (unknown 534919773 0 0) 0x7f94c00433f0 con 0x7f94d009b040 2024-09-16T14:42:57.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94d681f640 10 monclient: get_auth_request con 0x7f94b4036110 auth_method 0 2024-09-16T14:42:57.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283335933 0 0) 0x7f94b8006590 con 0x7f94b8002830 2024-09-16T14:42:57.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:57.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:57.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:57.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient(hunting): my global_id is 4453 2024-09-16T14:42:57.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94d009a240 con 0x7f94b8002830 2024-09-16T14:42:57.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 416485796 0 0) 0x7f94d009a240 con 0x7f94b8002830 2024-09-16T14:42:57.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94b8006590 con 0x7f94b8002830 2024-09-16T14:42:57.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f94cc001700 con 0x7f94b8002830 2024-09-16T14:42:57.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:57.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient: dump: 2024-09-16T14:42:57.853 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.853 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.854 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.854 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.854 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.854 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.854 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.854 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.854 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.854 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3622758935 0 0) 0x7f94b8006590 con 0x7f94b8002830 2024-09-16T14:42:57.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f94b800a0e0 con 0x7f94b8002830 2024-09-16T14:42:57.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f94cc001b50 con 0x7f94b8002830 2024-09-16T14:42:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:42:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:42:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 >> v1:172.21.15.189:6789/0 conn(0x7f94b8002830 legacy=0x7f94b8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.836+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 >> v1:172.21.15.189:6789/0 conn(0x7f94b8002830 legacy=0x7f94b8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:42:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94d8282640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:42:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94d8282640 1 -- 172.21.15.26:0/980175810 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f94b8009da0 con 0x7f94b80090c0 2024-09-16T14:42:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403917751 0 0) 0x7f94b8009da0 con 0x7f94b80090c0 2024-09-16T14:42:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:42:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:42:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:42:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 10 monclient(hunting): my global_id is 4456 2024-09-16T14:42:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94b8006590 con 0x7f94b80090c0 2024-09-16T14:42:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3482161443 0 0) 0x7f94b8006590 con 0x7f94b80090c0 2024-09-16T14:42:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94b8009da0 con 0x7f94b80090c0 2024-09-16T14:42:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f94cc003820 con 0x7f94b80090c0 2024-09-16T14:42:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:42:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:42:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 10 monclient: dump: 2024-09-16T14:42:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:42:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:42:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:42:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:42:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:42:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:42:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:42:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:42:57.858 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:42:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1122323823 0 0) 0x7f94b8009da0 con 0x7f94b80090c0 2024-09-16T14:42:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f94b800c290 con 0x7f94b80090c0 2024-09-16T14:42:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4200 (unknown 0 0 231609563) 0x7f94cc0057d0 con 0x7f94b80090c0 2024-09-16T14:42:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:42:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:42:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 >> v1:172.21.15.189:6789/0 conn(0x7f94b80090c0 legacy=0x7f94b8009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.844+0000 7f94c6ffd640 1 -- 172.21.15.26:0/980175810 >> v1:172.21.15.189:6789/0 conn(0x7f94b80090c0 legacy=0x7f94b8009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:42:57.859 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:42:57.859 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:42:57.859 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:42:57.859 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:42:57.859 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-09-16T14:42:57.859 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:42:57.859 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:42:57.859 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:42:57.859 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:57.859 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:42:57.859 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:42:57.860 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:42:57.860 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:42:57.860 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:42:57.860 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:42:57.860 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:42:57.860 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:42:57.860 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:42:57.860 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:42:57.860 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:42:57.860 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:42:57.860 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:42:57.860 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:42:57.860 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:57.860 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:42:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:42:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:42:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:42:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:42:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:42:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:42:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:42:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:42:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:42:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:42:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:42:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:42:57.861 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:42:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:42:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:42:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:42:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:42:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:42:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:42:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:42:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:42:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:42:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:42:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:42:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:42:57.862 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:42:57.863 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:42:57.863 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:42:57.863 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:42:57.863 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:42:57.863 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:42:57.863 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:42:57.863 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:42:57.863 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:42:57.863 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:42:57.863 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:42:57.863 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:42:57.863 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:42:57.863 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:57.863 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:42:57.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.864 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:42:57.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.864 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:42:57.864 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:42:57.864 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:42:57.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:42:57.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:42:57.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:42:57.864 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:42:57.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:57.864 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:57.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:42:57.865 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:42:57.865 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:42:57.865 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:42:57.865 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:42:57.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.865 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:42:57.865 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:42:57.865 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:42:57.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:42:57.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.865 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:42:57.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:42:57.866 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:42:57.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:57.866 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:57.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:42:57.866 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:42:57.866 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:42:57.866 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:42:57.866 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:42:57.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.866 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:42:57.866 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:42:57.867 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:42:57.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:42:57.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:42:57.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:42:57.867 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:42:57.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:57.867 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:57.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:42:57.867 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:42:57.867 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:42:57.867 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:42:57.868 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:42:57.868 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:57.868 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:57.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.868 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:42:57.868 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:42:57.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.868 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:57.868 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:57.868 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:42:57.868 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:57.868 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:57.868 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:42:57.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.869 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:57.869 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:57.869 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:42:57.869 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:57.869 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:57.869 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:42:57.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.869 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:57.869 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:57.869 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:42:57.869 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:57.869 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:42:57.869 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:42:57.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:42:57.870 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:42:57.870 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:42:57.870 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:42:57.870 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:42:57.870 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:42:57.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:42:57.870 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:42:57.870 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:42:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.848+0000 7f94d8282640 1 -- 172.21.15.26:0/980175810 >> v1:172.21.15.189:6812/3196161582 conn(0x7f94b4036110 legacy=0x7f94b40385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.848+0000 7f94d8282640 10 monclient: shutdown 2024-09-16T14:42:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.848+0000 7f94d8282640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:42:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.848+0000 7f94d8282640 1 -- 172.21.15.26:0/980175810 >> v1:172.21.15.26:6790/0 conn(0x7f94d009b040 legacy=0x7f94d01b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:42:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.848+0000 7f94d681f640 1 -- 172.21.15.26:0/980175810 reap_dead start 2024-09-16T14:42:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.848+0000 7f94d681f640 1 -- 172.21.15.26:0/980175810 reap_dead start 2024-09-16T14:42:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.848+0000 7f94d8282640 1 -- 172.21.15.26:0/980175810 shutdown_connections 2024-09-16T14:42:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.848+0000 7f94d8282640 1 -- 172.21.15.26:0/980175810 >> 172.21.15.26:0/980175810 conn(0x7f94d009fb50 msgr2=0x7f94d01b7740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:42:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.848+0000 7f94d8282640 1 -- 172.21.15.26:0/980175810 shutdown_connections 2024-09-16T14:42:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:42:57.848+0000 7f94d8282640 1 -- 172.21.15.26:0/980175810 wait complete. 2024-09-16T14:42:57.872 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:42:57.872 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:43:17.873 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:43:17.873 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T14:43:17.873 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T14:43:17.874 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T14:43:17.877 INFO:tasks.ceph.mon.a:Started 2024-09-16T14:43:17.877 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T14:43:17.877 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T14:43:17.878 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T14:43:30.525 INFO:tasks.workunit.client.0.smithi189.stderr:Note: switching to 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6'. 2024-09-16T14:43:30.526 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:30.526 INFO:tasks.workunit.client.0.smithi189.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T14:43:30.526 INFO:tasks.workunit.client.0.smithi189.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T14:43:30.526 INFO:tasks.workunit.client.0.smithi189.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T14:43:30.526 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:30.526 INFO:tasks.workunit.client.0.smithi189.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T14:43:30.526 INFO:tasks.workunit.client.0.smithi189.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T14:43:30.526 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:30.526 INFO:tasks.workunit.client.0.smithi189.stderr: git switch -c 2024-09-16T14:43:30.526 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:30.527 INFO:tasks.workunit.client.0.smithi189.stderr:Or undo this operation with: 2024-09-16T14:43:30.527 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:30.527 INFO:tasks.workunit.client.0.smithi189.stderr: git switch - 2024-09-16T14:43:30.527 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:30.527 INFO:tasks.workunit.client.0.smithi189.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T14:43:30.527 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:30.527 INFO:tasks.workunit.client.0.smithi189.stderr:HEAD is now at ae7f37330c1 Merge pull request #59795 from zdover23/wip-doc-2024-09-15-backport-59613-to-reef 2024-09-16T14:43:30.533 DEBUG:teuthology.orchestra.run.smithi189:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T14:43:30.581 INFO:tasks.workunit.client.0.smithi189.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T14:43:30.584 INFO:tasks.workunit.client.0.smithi189.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T14:43:30.584 INFO:tasks.workunit.client.0.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T14:43:31.362 INFO:tasks.workunit.client.0.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T14:43:31.441 INFO:tasks.workunit.client.0.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T14:43:31.491 INFO:tasks.workunit.client.0.smithi189.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T14:43:31.493 INFO:tasks.workunit.client.0.smithi189.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T14:43:31.493 INFO:tasks.workunit.client.0.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T14:43:31.539 INFO:tasks.workunit.client.0.smithi189.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T14:43:31.543 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T14:43:31.543 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T14:43:31.590 INFO:tasks.workunit:Running workunits matching mon/pool_ops.sh on client.0... 2024-09-16T14:43:31.591 INFO:tasks.workunit:Running workunit mon/pool_ops.sh... 2024-09-16T14:43:31.591 DEBUG:teuthology.orchestra.run.smithi189:workunit test mon/pool_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pool_ops.sh 2024-09-16T14:43:31.642 INFO:tasks.workunit.client.0.smithi189.stderr:+ TEST_POOL=testpool1234 2024-09-16T14:43:31.642 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create testpool1234 8 --autoscale-mode off 2024-09-16T14:43:31.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.760+0000 7fafbaf0d640 10 monclient: get_monmap_and_config 2024-09-16T14:43:31.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.760+0000 7fafbaf0d640 10 monclient: build_initial_monmap 2024-09-16T14:43:31.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.760+0000 7fafbaf0d640 10 monclient: monmap: 2024-09-16T14:43:31.765 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:31.765 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:31.765 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:31.765599+0000 2024-09-16T14:43:31.765 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:31.765599+0000 2024-09-16T14:43:31.765 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:31.765 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:31.765 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:31.765 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:31.765 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:31.765 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:31.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.764+0000 7fafbaf0d640 1 Processor -- start 2024-09-16T14:43:31.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.764+0000 7fafbaf0d640 1 -- start start 2024-09-16T14:43:31.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.764+0000 7fafbaf0d640 10 monclient: init 2024-09-16T14:43:31.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.764+0000 7fafbaf0d640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:31.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.764+0000 7fafbaf0d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:43:31.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.764+0000 7fafbaf0d640 10 monclient(hunting): picked mon.noname-a con 0x7fafb40a1a90 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:31.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.764+0000 7fafbaf0d640 10 monclient(hunting): picked mon.noname-c con 0x7fafb40a2580 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:31.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.764+0000 7fafbaf0d640 10 monclient(hunting): picked mon.noname-b con 0x7fafb409fd40 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:31.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.764+0000 7fafbaf0d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb409df00 con 0x7fafb40a1a90 2024-09-16T14:43:31.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.764+0000 7fafbaf0d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb409f0a0 con 0x7fafb40a2580 2024-09-16T14:43:31.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.764+0000 7fafbaf0d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb40a3040 con 0x7fafb409fd40 2024-09-16T14:43:31.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.764+0000 7fafbaf0d640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:31.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.764+0000 7fafbaf0d640 10 monclient(hunting): authenticate will time out at 993.881189s 2024-09-16T14:43:31.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.764+0000 7fafb3fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fafb40a2580 0x7fafb4139df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43734/0 (socket says 172.21.15.189:43734) 2024-09-16T14:43:31.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:31.764+0000 7fafb3fff640 1 -- 172.21.15.189:0/3158115464 learned_addr learned my addr 172.21.15.189:0/3158115464 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:32.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.256+0000 7fafb37fe640 1 -- 172.21.15.189:0/3158115464 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218544181 0 0) 0x7fafb409f0a0 con 0x7fafb40a2580 2024-09-16T14:43:32.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.256+0000 7fafb37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:32.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.256+0000 7fafb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:32.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.256+0000 7fafb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:32.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.256+0000 7fafb37fe640 10 monclient(hunting): my global_id is 4459 2024-09-16T14:43:32.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.256+0000 7fafb37fe640 1 -- 172.21.15.189:0/3158115464 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf9c003530 con 0x7fafb40a2580 2024-09-16T14:43:32.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafb37fe640 1 -- 172.21.15.189:0/3158115464 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1527421815 0 0) 0x7faf9c003530 con 0x7fafb40a2580 2024-09-16T14:43:32.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafb37fe640 1 -- 172.21.15.189:0/3158115464 >> v1:172.21.15.26:6790/0 conn(0x7fafb409fd40 legacy=0x7fafb413d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:43:32.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafb37fe640 1 -- 172.21.15.189:0/3158115464 >> v1:172.21.15.26:6789/0 conn(0x7fafb40a1a90 legacy=0x7fafb40a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:32.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafb37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:43:32.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafb37fe640 1 monclient: found mon.noname-c 2024-09-16T14:43:32.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:32.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafb37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:32.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafb37fe640 1 -- 172.21.15.189:0/3158115464 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb409eae0 con 0x7fafb40a2580 2024-09-16T14:43:32.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafb37fe640 10 monclient: _finish_auth 0 2024-09-16T14:43:32.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafb37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:43:32.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:32.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafb37fe640 1 -- 172.21.15.189:0/3158115464 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fafa8001620 con 0x7fafb40a2580 2024-09-16T14:43:32.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafbaf0d640 5 monclient: authenticate success, global_id 4459 2024-09-16T14:43:32.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:32.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafb37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:32.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafb37fe640 10 monclient: dump: 2024-09-16T14:43:32.263 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:32.263 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:32.263 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:32.263 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:32.264 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:32.264 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:32.264 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:32.264 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:32.264 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:32.264 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:32.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.260+0000 7fafbaf0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:32.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.264+0000 7fafb37fe640 1 -- 172.21.15.189:0/3158115464 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafa8001910 con 0x7fafb40a2580 2024-09-16T14:43:32.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.264+0000 7fafb37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:32.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.264+0000 7fafb37fe640 1 -- 172.21.15.189:0/3158115464 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fafa8001ca0 con 0x7fafb40a2580 2024-09-16T14:43:32.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.264+0000 7fafbaf0d640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:32.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.264+0000 7fafbaf0d640 4 set_mon_vals no callback set 2024-09-16T14:43:32.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.264+0000 7fafb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:32.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.264+0000 7fafb37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:32.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.264+0000 7fafb37fe640 10 monclient: dump: 2024-09-16T14:43:32.267 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:32.267 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:32.267 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:32.267 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:32.267 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:32.267 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:32.267 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:32.267 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:32.268 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:32.268 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:32.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.264+0000 7fafbaf0d640 10 monclient: shutdown 2024-09-16T14:43:32.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.264+0000 7fafbaf0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:32.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.264+0000 7fafbaf0d640 1 -- 172.21.15.189:0/3158115464 >> v1:172.21.15.189:6789/0 conn(0x7fafb40a2580 legacy=0x7fafb4139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:32.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.264+0000 7fafbaf0d640 1 -- 172.21.15.189:0/3158115464 shutdown_connections 2024-09-16T14:43:32.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.264+0000 7fafbaf0d640 1 -- 172.21.15.189:0/3158115464 >> 172.21.15.189:0/3158115464 conn(0x7fafb40a7a70 msgr2=0x7fafb40a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:32.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.264+0000 7fafbaf0d640 1 -- 172.21.15.189:0/3158115464 shutdown_connections 2024-09-16T14:43:32.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.264+0000 7fafbaf0d640 1 -- 172.21.15.189:0/3158115464 wait complete. 2024-09-16T14:43:32.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 10 monclient: build_initial_monmap 2024-09-16T14:43:32.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 10 monclient: monmap: 2024-09-16T14:43:32.269 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:32.269 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:32.269 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:32.269 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:32.270 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:32.270 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:32.270 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:32.270 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:32.270 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:32.270 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:32.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 1 Processor -- start 2024-09-16T14:43:32.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 1 -- start start 2024-09-16T14:43:32.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 10 monclient: init 2024-09-16T14:43:32.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:32.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:43:32.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 10 monclient(hunting): picked mon.noname-b con 0x7fafb409fd40 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:32.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 10 monclient(hunting): picked mon.noname-c con 0x7fafb40a1a90 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:32.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 10 monclient(hunting): picked mon.noname-a con 0x7fafb41344b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:32.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb40a3640 con 0x7fafb41344b0 2024-09-16T14:43:32.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb4121760 con 0x7fafb409fd40 2024-09-16T14:43:32.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb411fd50 con 0x7fafb40a1a90 2024-09-16T14:43:32.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:32.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 10 monclient(hunting): authenticate will time out at 994.385720s 2024-09-16T14:43:32.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb8ca9640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fafb409fd40 0x7fafb4133da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43738/0 (socket says 172.21.15.189:43738) 2024-09-16T14:43:32.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb8ca9640 1 -- 172.21.15.189:0/3564810575 learned_addr learned my addr 172.21.15.189:0/3564810575 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:32.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 1 -- 172.21.15.189:0/3564810575 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064085620 0 0) 0x7fafb4121760 con 0x7fafb409fd40 2024-09-16T14:43:32.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:43:32.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:32.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:32.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 10 monclient(hunting): my global_id is 4462 2024-09-16T14:43:32.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 1 -- 172.21.15.189:0/3564810575 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafa0003530 con 0x7fafb409fd40 2024-09-16T14:43:32.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 1 -- 172.21.15.189:0/3564810575 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2378412919 0 0) 0x7fafa0003530 con 0x7fafb409fd40 2024-09-16T14:43:32.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 1 -- 172.21.15.189:0/3564810575 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafb4121760 con 0x7fafb409fd40 2024-09-16T14:43:32.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 1 -- 172.21.15.189:0/3564810575 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fafa40018c0 con 0x7fafb409fd40 2024-09-16T14:43:32.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:32.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:32.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 10 monclient(hunting): dump: 2024-09-16T14:43:32.276 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:32.276 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:32.276 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:32.276 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:32.276 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:32.276 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:32.276 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:32.276 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:32.276 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:32.276 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:32.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:32.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 1 -- 172.21.15.189:0/3564810575 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 824590699 0 0) 0x7fafb4121760 con 0x7fafb409fd40 2024-09-16T14:43:32.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 1 -- 172.21.15.189:0/3564810575 >> v1:172.21.15.26:6790/0 conn(0x7fafb40a1a90 legacy=0x7fafb41ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:32.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 1 -- 172.21.15.189:0/3564810575 >> v1:172.21.15.26:6789/0 conn(0x7fafb41344b0 legacy=0x7fafb41d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:32.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 10 monclient: _finish_hunting 0 2024-09-16T14:43:32.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 1 monclient: found mon.b 2024-09-16T14:43:32.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:32.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:32.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 1 -- 172.21.15.189:0/3564810575 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb40a6100 con 0x7fafb409fd40 2024-09-16T14:43:32.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 10 monclient: _finish_auth 0 2024-09-16T14:43:32.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 10 monclient: _check_auth_tickets 2024-09-16T14:43:32.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 5 monclient: authenticate success, global_id 4462 2024-09-16T14:43:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 10 monclient: _renew_subs 2024-09-16T14:43:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 1 -- 172.21.15.189:0/3564810575 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb409e280 con 0x7fafb409fd40 2024-09-16T14:43:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 10 monclient: _renew_subs 2024-09-16T14:43:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafbaf0d640 1 -- 172.21.15.189:0/3564810575 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb411f420 con 0x7fafb409fd40 2024-09-16T14:43:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 1 -- 172.21.15.189:0/3564810575 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafa4001cf0 con 0x7fafb409fd40 2024-09-16T14:43:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 1 -- 172.21.15.189:0/3564810575 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fafa4003a70 con 0x7fafb409fd40 2024-09-16T14:43:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb2ffd640 4 set_mon_vals no callback set 2024-09-16T14:43:32.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.268+0000 7fafb17fa640 10 monclient: dump: 2024-09-16T14:43:32.279 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:32.279 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:32.279 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:32.279 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:32.279 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:32.279 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:32.279 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:32.279 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:32.279 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:32.279 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:32.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.272+0000 7fafb17fa640 1 -- 172.21.15.189:0/3564810575 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fafa4003bd0 con 0x7fafb409fd40 2024-09-16T14:43:32.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.272+0000 7fafbaf0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:32.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.272+0000 7fafbaf0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:32.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.272+0000 7fafbaf0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:32.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.272+0000 7fafbaf0d640 1 -- 172.21.15.189:0/3564810575 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf94005050 con 0x7fafb409fd40 2024-09-16T14:43:32.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.272+0000 7fafb17fa640 1 -- 172.21.15.189:0/3564810575 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3534+0+0 (unknown 534919773 0 0) 0x7fafa40431c0 con 0x7fafb409fd40 2024-09-16T14:43:32.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.272+0000 7fafb3fff640 10 monclient: get_auth_request con 0x7fafa00360a0 auth_method 0 2024-09-16T14:43:32.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.276+0000 7fafb17fa640 1 -- 172.21.15.189:0/3564810575 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fafa4014790 con 0x7fafb409fd40 2024-09-16T14:43:32.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.276+0000 7fafb17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:32.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.276+0000 7fafb17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:32.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.444+0000 7fafbaf0d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T14:43:32.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.444+0000 7fafbaf0d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T14:43:32.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.444+0000 7fafbaf0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:32.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:32.444+0000 7fafbaf0d640 1 -- 172.21.15.189:0/3564810575 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"} v 0) v1 -- 0x7faf94005880 con 0x7fafb409fd40 2024-09-16T14:43:32.878 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T14:43:32.879 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T14:43:32.879 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:43:32.879 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:43:33.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 10 monclient: get_monmap_and_config 2024-09-16T14:43:33.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 10 monclient: build_initial_monmap 2024-09-16T14:43:33.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 10 monclient: monmap: 2024-09-16T14:43:33.017 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:33.017 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.017 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:43:33.018092+0000 2024-09-16T14:43:33.018 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:43:33.018092+0000 2024-09-16T14:43:33.018 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:33.018 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.018 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:33.018 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:33.018 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:33.018 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.016+0000 7fafb17fa640 1 -- 172.21.15.189:0/3564810575 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}]=0 pool 'testpool1234' created v17) v1 ==== 152+0+0 (unknown 1141535973 0 0) 0x7fafa4012c00 con 0x7fafb409fd40 2024-09-16T14:43:33.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.016+0000 7fafb17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T14:43:33.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.016+0000 7fafb17fa640 10 monclient: _finish_command 2 = system:0 pool 'testpool1234' created 2024-09-16T14:43:33.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 1 Processor -- start 2024-09-16T14:43:33.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 1 -- start start 2024-09-16T14:43:33.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 10 monclient: init 2024-09-16T14:43:33.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:33.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:43:33.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 10 monclient(hunting): picked mon.noname-a con 0x7f1c180a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:33.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 10 monclient(hunting): picked mon.noname-b con 0x7f1c180a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:33.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 10 monclient(hunting): picked mon.noname-c con 0x7f1c180a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:33.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c180a3250 con 0x7f1c180a1d10 2024-09-16T14:43:33.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c1809f600 con 0x7f1c180a2800 2024-09-16T14:43:33.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c180a3830 con 0x7f1c180a0000 2024-09-16T14:43:33.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:33.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 10 monclient(hunting): authenticate will time out at 1037.138139s 2024-09-16T14:43:33.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1d7da640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1c180a0000 0x7f1c18139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40230/0 (socket says 172.21.15.26:40230) 2024-09-16T14:43:33.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1dfdb640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1c180a1d10 0x7f1c180a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51828/0 (socket says 172.21.15.26:51828) 2024-09-16T14:43:33.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1d7da640 1 -- 172.21.15.26:0/358301032 learned_addr learned my addr 172.21.15.26:0/358301032 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:33.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 1 -- 172.21.15.26:0/358301032 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103746315 0 0) 0x7f1c180a3250 con 0x7f1c180a1d10 2024-09-16T14:43:33.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient(hunting): my global_id is 74097 2024-09-16T14:43:33.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 1 -- 172.21.15.26:0/358301032 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c04003530 con 0x7f1c180a1d10 2024-09-16T14:43:33.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 1 -- 172.21.15.26:0/358301032 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555165970 0 0) 0x7f1c180a3830 con 0x7f1c180a0000 2024-09-16T14:43:33.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient(hunting): my global_id is 4433 2024-09-16T14:43:33.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 1 -- 172.21.15.26:0/358301032 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c180a3250 con 0x7f1c180a0000 2024-09-16T14:43:33.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 1 -- 172.21.15.26:0/358301032 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045141838 0 0) 0x7f1c1809f600 con 0x7f1c180a2800 2024-09-16T14:43:33.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient(hunting): my global_id is 4465 2024-09-16T14:43:33.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 1 -- 172.21.15.26:0/358301032 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c180a3830 con 0x7f1c180a2800 2024-09-16T14:43:33.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 1 -- 172.21.15.26:0/358301032 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2007303746 0 0) 0x7f1c04003530 con 0x7f1c180a1d10 2024-09-16T14:43:33.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 1 -- 172.21.15.26:0/358301032 >> v1:172.21.15.26:6790/0 conn(0x7f1c180a0000 legacy=0x7f1c18139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 1 -- 172.21.15.26:0/358301032 >> v1:172.21.15.189:6789/0 conn(0x7f1c180a2800 legacy=0x7f1c1813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient: _finish_hunting 0 2024-09-16T14:43:33.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 1 monclient: found mon.noname-a 2024-09-16T14:43:33.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:33.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:43:33.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 1 -- 172.21.15.26:0/358301032 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1809ed60 con 0x7f1c180a1d10 2024-09-16T14:43:33.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient: _finish_auth 0 2024-09-16T14:43:33.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient: _check_auth_tickets 2024-09-16T14:43:33.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:33.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 1 -- 172.21.15.26:0/358301032 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1c14001620 con 0x7f1c180a1d10 2024-09-16T14:43:33.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:43:33.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient: dump: 2024-09-16T14:43:33.026 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.026 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.026 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.026 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.026 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.026 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.026 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.026 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.026 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.026 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 5 monclient: authenticate success, global_id 74097 2024-09-16T14:43:33.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:33.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 1 -- 172.21.15.26:0/358301032 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c14001880 con 0x7f1c180a1d10 2024-09-16T14:43:33.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:33.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 1 -- 172.21.15.26:0/358301032 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1c14001c50 con 0x7f1c180a1d10 2024-09-16T14:43:33.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:33.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c2023f640 4 set_mon_vals no callback set 2024-09-16T14:43:33.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:33.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.015+0000 7f1c1cfd9640 10 monclient: dump: 2024-09-16T14:43:33.027 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.027 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.027 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.027 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.027 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.028 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 10 monclient: shutdown 2024-09-16T14:43:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 1 -- 172.21.15.26:0/358301032 >> v1:172.21.15.26:6789/0 conn(0x7f1c180a1d10 legacy=0x7f1c180a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 1 -- 172.21.15.26:0/358301032 shutdown_connections 2024-09-16T14:43:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 1 -- 172.21.15.26:0/358301032 >> 172.21.15.26:0/358301032 conn(0x7f1c180a7f30 msgr2=0x7f1c180a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 1 -- 172.21.15.26:0/358301032 shutdown_connections 2024-09-16T14:43:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 1 -- 172.21.15.26:0/358301032 wait complete. 2024-09-16T14:43:33.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 10 monclient: build_initial_monmap 2024-09-16T14:43:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 10 monclient: monmap: 2024-09-16T14:43:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:43:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:43:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:43:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:33.029 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:33.029 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 1 Processor -- start 2024-09-16T14:43:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 1 -- start start 2024-09-16T14:43:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 10 monclient: init 2024-09-16T14:43:33.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:33.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:43:33.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 10 monclient(hunting): picked mon.noname-a con 0x7f1c180a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:33.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 10 monclient(hunting): picked mon.noname-c con 0x7f1c180a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:33.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 10 monclient(hunting): picked mon.noname-b con 0x7f1c180a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:33.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c18133e60 con 0x7f1c180a0000 2024-09-16T14:43:33.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c180a6340 con 0x7f1c180a2800 2024-09-16T14:43:33.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c1809e500 con 0x7f1c180a1d10 2024-09-16T14:43:33.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:33.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 10 monclient(hunting): authenticate will time out at 1037.142363s 2024-09-16T14:43:33.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c1dfdb640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1c180a0000 0x7f1c18133040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51842/0 (socket says 172.21.15.26:51842) 2024-09-16T14:43:33.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c1dfdb640 1 -- 172.21.15.26:0/2086425510 learned_addr learned my addr 172.21.15.26:0/2086425510 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:33.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189971525 0 0) 0x7f1c18133e60 con 0x7f1c180a0000 2024-09-16T14:43:33.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): my global_id is 74100 2024-09-16T14:43:33.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bfc003530 con 0x7f1c180a0000 2024-09-16T14:43:33.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670810527 0 0) 0x7f1c1809e500 con 0x7f1c180a1d10 2024-09-16T14:43:33.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): my global_id is 4436 2024-09-16T14:43:33.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c18133e60 con 0x7f1c180a1d10 2024-09-16T14:43:33.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1578462060 0 0) 0x7f1bfc003530 con 0x7f1c180a0000 2024-09-16T14:43:33.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c1809e500 con 0x7f1c180a0000 2024-09-16T14:43:33.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1c14001560 con 0x7f1c180a0000 2024-09-16T14:43:33.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:43:33.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): dump: 2024-09-16T14:43:33.036 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.036 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.036 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.036 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.036 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.036 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.036 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.036 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.036 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.036 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:33.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4225574132 0 0) 0x7f1c18133e60 con 0x7f1c180a1d10 2024-09-16T14:43:33.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bfc003530 con 0x7f1c180a1d10 2024-09-16T14:43:33.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1c08001760 con 0x7f1c180a1d10 2024-09-16T14:43:33.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:33.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): dump: 2024-09-16T14:43:33.037 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.037 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.037 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.037 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.037 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.037 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.037 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.037 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.037 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.037 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135129402 0 0) 0x7f1c180a6340 con 0x7f1c180a2800 2024-09-16T14:43:33.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient(hunting): my global_id is 4468 2024-09-16T14:43:33.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c18133e60 con 0x7f1c180a2800 2024-09-16T14:43:33.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3194166820 0 0) 0x7f1c1809e500 con 0x7f1c180a0000 2024-09-16T14:43:33.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 >> v1:172.21.15.26:6790/0 conn(0x7f1c180a1d10 legacy=0x7f1c18133750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 >> v1:172.21.15.189:6789/0 conn(0x7f1c180a2800 legacy=0x7f1c181d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:43:33.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 monclient: found mon.a 2024-09-16T14:43:33.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:33.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:33.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c180a6870 con 0x7f1c180a0000 2024-09-16T14:43:33.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:43:33.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:43:33.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:33.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 5 monclient: authenticate success, global_id 74100 2024-09-16T14:43:33.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 10 monclient: _renew_subs 2024-09-16T14:43:33.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:33.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 1 -- 172.21.15.26:0/2086425510 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c181371d0 con 0x7f1c180a0000 2024-09-16T14:43:33.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 10 monclient: _renew_subs 2024-09-16T14:43:33.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:33.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.019+0000 7f1c2023f640 1 -- 172.21.15.26:0/2086425510 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c18136ef0 con 0x7f1c180a0000 2024-09-16T14:43:33.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.023+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c14001240 con 0x7f1c180a0000 2024-09-16T14:43:33.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.023+0000 7f1c0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:33.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.023+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1c14004de0 con 0x7f1c180a0000 2024-09-16T14:43:33.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.023+0000 7f1c0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.023+0000 7f1c0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:33.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.023+0000 7f1c0ffff640 4 set_mon_vals no callback set 2024-09-16T14:43:33.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.023+0000 7f1c0e7fc640 10 monclient: dump: 2024-09-16T14:43:33.040 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.040 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.040 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.040 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.040 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.040 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.040 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.040 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.040 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.041 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.023+0000 7f1c2023f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:33.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.023+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1c14004f80 con 0x7f1c180a0000 2024-09-16T14:43:33.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.023+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3857+0+0 (unknown 1040354461 0 0) 0x7f1c140434b0 con 0x7f1c180a0000 2024-09-16T14:43:33.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.023+0000 7f1c2023f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:33.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.023+0000 7f1c2023f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:33.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.023+0000 7f1c2023f640 1 -- 172.21.15.26:0/2086425510 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c00005050 con 0x7f1c180a0000 2024-09-16T14:43:33.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.023+0000 7f1c1d7da640 10 monclient: get_auth_request con 0x7f1bfc036440 auth_method 0 2024-09-16T14:43:33.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.027+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1c140147a0 con 0x7f1c180a0000 2024-09-16T14:43:33.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.027+0000 7f1c0e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:33.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.027+0000 7f1c0e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:33.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.116+0000 7fafbaf0d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T14:43:33.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.116+0000 7fafbaf0d640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T14:43:33.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.116+0000 7fafbaf0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:33.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.116+0000 7fafbaf0d640 1 -- 172.21.15.189:0/3564810575 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"} v 0) v1 -- 0x7faf94005370 con 0x7fafb409fd40 2024-09-16T14:43:33.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.116+0000 7fafb17fa640 1 -- 172.21.15.189:0/3564810575 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}]=0 pool 'testpool1234' already exists v17) v1 ==== 159+0+0 (unknown 1078346858 0 0) 0x7fafa4014380 con 0x7fafb409fd40 2024-09-16T14:43:33.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.116+0000 7fafb17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T14:43:33.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.116+0000 7fafb17fa640 10 monclient: _finish_command 3 = system:0 pool 'testpool1234' already exists 2024-09-16T14:43:33.120 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'testpool1234' already exists 2024-09-16T14:43:33.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.120+0000 7fafbaf0d640 1 -- 172.21.15.189:0/3564810575 >> v1:172.21.15.189:6812/3196161582 conn(0x7fafa00360a0 legacy=0x7fafa0038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.120+0000 7fafbaf0d640 10 monclient: shutdown 2024-09-16T14:43:33.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.120+0000 7fafbaf0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:33.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.120+0000 7fafbaf0d640 1 -- 172.21.15.189:0/3564810575 >> v1:172.21.15.189:6789/0 conn(0x7fafb409fd40 legacy=0x7fafb4133da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.120+0000 7fafbaf0d640 1 -- 172.21.15.189:0/3564810575 shutdown_connections 2024-09-16T14:43:33.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.120+0000 7fafbaf0d640 1 -- 172.21.15.189:0/3564810575 >> 172.21.15.189:0/3564810575 conn(0x7fafb40a7a70 msgr2=0x7fafb40a5040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:33.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.120+0000 7fafbaf0d640 1 -- 172.21.15.189:0/3564810575 shutdown_connections 2024-09-16T14:43:33.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.120+0000 7fafbaf0d640 1 -- 172.21.15.189:0/3564810575 wait complete. 2024-09-16T14:43:33.137 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set testpool1234 pg_num_min 2 2024-09-16T14:43:33.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.251+0000 7f1c2023f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:43:33.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.251+0000 7f1c2023f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:43:33.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.251+0000 7f1c2023f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:33.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.251+0000 7f1c2023f640 1 -- 172.21.15.26:0/2086425510 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1c00005880 con 0x7f1c180a0000 2024-09-16T14:43:33.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.251+0000 7f1c0e7fc640 1 -- 172.21.15.26:0/2086425510 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1235342618) 0x7f1c14012e40 con 0x7f1c180a0000 2024-09-16T14:43:33.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.251+0000 7f1c0e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:43:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.251+0000 7f1c0e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:43:33.255 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:43:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.251+0000 7f1c2023f640 1 -- 172.21.15.26:0/2086425510 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1bfc036440 legacy=0x7f1bfc038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.251+0000 7f1c2023f640 10 monclient: shutdown 2024-09-16T14:43:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.251+0000 7f1c2023f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.251+0000 7f1c2023f640 1 -- 172.21.15.26:0/2086425510 >> v1:172.21.15.26:6789/0 conn(0x7f1c180a0000 legacy=0x7f1c18133040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.251+0000 7f1c2023f640 1 -- 172.21.15.26:0/2086425510 shutdown_connections 2024-09-16T14:43:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.251+0000 7f1c2023f640 1 -- 172.21.15.26:0/2086425510 >> 172.21.15.26:0/2086425510 conn(0x7f1c180a7f30 msgr2=0x7f1c1806b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:33.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.255+0000 7f1c2023f640 1 -- 172.21.15.26:0/2086425510 shutdown_connections 2024-09-16T14:43:33.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.255+0000 7f1c2023f640 1 -- 172.21.15.26:0/2086425510 wait complete. 2024-09-16T14:43:33.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.252+0000 7f6ab476d640 10 monclient: get_monmap_and_config 2024-09-16T14:43:33.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.252+0000 7f6ab476d640 10 monclient: build_initial_monmap 2024-09-16T14:43:33.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.252+0000 7f6ab476d640 10 monclient: monmap: 2024-09-16T14:43:33.260 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:33.260 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.260 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:33.257729+0000 2024-09-16T14:43:33.260 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:33.257729+0000 2024-09-16T14:43:33.260 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:33.260 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:33.260 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:33.260 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:33.260 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:33.260 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:33.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 1 Processor -- start 2024-09-16T14:43:33.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 1 -- start start 2024-09-16T14:43:33.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 10 monclient: init 2024-09-16T14:43:33.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:33.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:43:33.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 10 monclient(hunting): picked mon.noname-c con 0x7f6aac0a19b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:33.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 10 monclient(hunting): picked mon.noname-b con 0x7f6aac0a24a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:33.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 10 monclient(hunting): picked mon.noname-a con 0x7f6aac09fc80 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:33.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aac0a00d0 con 0x7f6aac09fc80 2024-09-16T14:43:33.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aac09ea00 con 0x7f6aac0a19b0 2024-09-16T14:43:33.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aac09dde0 con 0x7f6aac0a24a0 2024-09-16T14:43:33.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:33.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 10 monclient(hunting): authenticate will time out at 995.373353s 2024-09-16T14:43:33.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab2509640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6aac0a19b0 0x7f6aac0a1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43744/0 (socket says 172.21.15.189:43744) 2024-09-16T14:43:33.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab2509640 1 -- 172.21.15.189:0/2911684801 learned_addr learned my addr 172.21.15.189:0/2911684801 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:33.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 1 -- 172.21.15.189:0/2911684801 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350693236 0 0) 0x7f6aac09ea00 con 0x7f6aac0a19b0 2024-09-16T14:43:33.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient(hunting): my global_id is 4471 2024-09-16T14:43:33.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 1 -- 172.21.15.189:0/2911684801 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a98003530 con 0x7f6aac0a19b0 2024-09-16T14:43:33.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 1 -- 172.21.15.189:0/2911684801 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735133039 0 0) 0x7f6aac0a00d0 con 0x7f6aac09fc80 2024-09-16T14:43:33.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient(hunting): my global_id is 74103 2024-09-16T14:43:33.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 1 -- 172.21.15.189:0/2911684801 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aac09ea00 con 0x7f6aac09fc80 2024-09-16T14:43:33.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 1 -- 172.21.15.189:0/2911684801 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1004971768 0 0) 0x7f6a98003530 con 0x7f6aac0a19b0 2024-09-16T14:43:33.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 1 -- 172.21.15.189:0/2911684801 >> v1:172.21.15.26:6790/0 conn(0x7f6aac0a24a0 legacy=0x7f6aac139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 1 -- 172.21.15.189:0/2911684801 >> v1:172.21.15.26:6789/0 conn(0x7f6aac09fc80 legacy=0x7f6aac13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient: _finish_hunting 0 2024-09-16T14:43:33.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 1 monclient: found mon.noname-c 2024-09-16T14:43:33.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:33.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:33.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 1 -- 172.21.15.189:0/2911684801 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aac09ece0 con 0x7f6aac0a19b0 2024-09-16T14:43:33.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient: _finish_auth 0 2024-09-16T14:43:33.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient: _check_auth_tickets 2024-09-16T14:43:33.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:33.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 1 -- 172.21.15.189:0/2911684801 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6aa8001620 con 0x7f6aac0a19b0 2024-09-16T14:43:33.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 5 monclient: authenticate success, global_id 4471 2024-09-16T14:43:33.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:33.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:33.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient: dump: 2024-09-16T14:43:33.264 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:33.265 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.265 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.265 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.265 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.265 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:33.265 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.265 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.265 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.265 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:33.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:33.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 1 -- 172.21.15.189:0/2911684801 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6aa8001900 con 0x7f6aac0a19b0 2024-09-16T14:43:33.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:33.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 1 -- 172.21.15.189:0/2911684801 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6aa8001cd0 con 0x7f6aac0a19b0 2024-09-16T14:43:33.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:33.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab1507640 10 monclient: dump: 2024-09-16T14:43:33.266 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:33.266 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.266 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.266 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.266 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.266 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:33.266 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.266 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.266 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.266 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:33.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:33.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 4 set_mon_vals no callback set 2024-09-16T14:43:33.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 10 monclient: shutdown 2024-09-16T14:43:33.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:33.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 1 -- 172.21.15.189:0/2911684801 >> v1:172.21.15.189:6789/0 conn(0x7f6aac0a19b0 legacy=0x7f6aac0a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 1 -- 172.21.15.189:0/2911684801 shutdown_connections 2024-09-16T14:43:33.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 1 -- 172.21.15.189:0/2911684801 >> 172.21.15.189:0/2911684801 conn(0x7f6aac0a3ab0 msgr2=0x7f6aac0a3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:33.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 1 -- 172.21.15.189:0/2911684801 shutdown_connections 2024-09-16T14:43:33.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 1 -- 172.21.15.189:0/2911684801 wait complete. 2024-09-16T14:43:33.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 10 monclient: build_initial_monmap 2024-09-16T14:43:33.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.256+0000 7f6ab476d640 10 monclient: monmap: 2024-09-16T14:43:33.271 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:33.271 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:33.271 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:33.271 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:33.271 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:33.271 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:33.271 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:33.271 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:33.271 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:33.271 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:33.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 1 Processor -- start 2024-09-16T14:43:33.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 1 -- start start 2024-09-16T14:43:33.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 10 monclient: init 2024-09-16T14:43:33.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:33.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:43:33.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 10 monclient(hunting): picked mon.noname-c con 0x7f6aac09fc80 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:33.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 10 monclient(hunting): picked mon.noname-b con 0x7f6aac0a19b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:33.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 10 monclient(hunting): picked mon.noname-a con 0x7f6aac0a24a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:33.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aac134690 con 0x7f6aac0a24a0 2024-09-16T14:43:33.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aac09e1a0 con 0x7f6aac0a19b0 2024-09-16T14:43:33.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aac120eb0 con 0x7f6aac09fc80 2024-09-16T14:43:33.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:33.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 10 monclient(hunting): authenticate will time out at 995.376630s 2024-09-16T14:43:33.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab1d08640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6aac0a19b0 0x7f6aac133a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43752/0 (socket says 172.21.15.189:43752) 2024-09-16T14:43:33.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab1d08640 1 -- 172.21.15.189:0/2507326579 learned_addr learned my addr 172.21.15.189:0/2507326579 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:33.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 1 -- 172.21.15.189:0/2507326579 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817584873 0 0) 0x7f6aac09e1a0 con 0x7f6aac0a19b0 2024-09-16T14:43:33.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 10 monclient(hunting): my global_id is 4474 2024-09-16T14:43:33.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 1 -- 172.21.15.189:0/2507326579 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a90003530 con 0x7f6aac0a19b0 2024-09-16T14:43:33.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 1 -- 172.21.15.189:0/2507326579 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1947320583 0 0) 0x7f6a90003530 con 0x7f6aac0a19b0 2024-09-16T14:43:33.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 1 -- 172.21.15.189:0/2507326579 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6aac09e1a0 con 0x7f6aac0a19b0 2024-09-16T14:43:33.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 1 -- 172.21.15.189:0/2507326579 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6aa4001d60 con 0x7f6aac0a19b0 2024-09-16T14:43:33.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 10 monclient(hunting): dump: 2024-09-16T14:43:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.275 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.275 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.275 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:33.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:33.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 1 -- 172.21.15.189:0/2507326579 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1518941459 0 0) 0x7f6aac09e1a0 con 0x7f6aac0a19b0 2024-09-16T14:43:33.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 1 -- 172.21.15.189:0/2507326579 >> v1:172.21.15.26:6790/0 conn(0x7f6aac09fc80 legacy=0x7f6aac1c8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 1 -- 172.21.15.189:0/2507326579 >> v1:172.21.15.26:6789/0 conn(0x7f6aac0a24a0 legacy=0x7f6aac1d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 10 monclient: _finish_hunting 0 2024-09-16T14:43:33.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 1 monclient: found mon.b 2024-09-16T14:43:33.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:33.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:33.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 1 -- 172.21.15.189:0/2507326579 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aac1213e0 con 0x7f6aac0a19b0 2024-09-16T14:43:33.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 10 monclient: _finish_auth 0 2024-09-16T14:43:33.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 10 monclient: _check_auth_tickets 2024-09-16T14:43:33.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:33.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 5 monclient: authenticate success, global_id 4474 2024-09-16T14:43:33.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 10 monclient: _renew_subs 2024-09-16T14:43:33.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:33.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 1 -- 172.21.15.189:0/2507326579 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aac0a6130 con 0x7f6aac0a19b0 2024-09-16T14:43:33.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 10 monclient: _renew_subs 2024-09-16T14:43:33.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:33.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 1 -- 172.21.15.189:0/2507326579 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aac11f0a0 con 0x7f6aac0a19b0 2024-09-16T14:43:33.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 1 -- 172.21.15.189:0/2507326579 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6aa4003c50 con 0x7f6aac0a19b0 2024-09-16T14:43:33.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:33.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 1 -- 172.21.15.189:0/2507326579 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6aa4003fe0 con 0x7f6aac0a19b0 2024-09-16T14:43:33.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:33.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab0d06640 4 set_mon_vals no callback set 2024-09-16T14:43:33.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 10 monclient: dump: 2024-09-16T14:43:33.277 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:33.277 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.277 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.277 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.277 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.277 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:33.278 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.278 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.278 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.278 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:33.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 1 -- 172.21.15.189:0/2507326579 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6aa4004140 con 0x7f6aac0a19b0 2024-09-16T14:43:33.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:33.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:33.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:33.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab476d640 1 -- 172.21.15.189:0/2507326579 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a94005050 con 0x7f6aac0a19b0 2024-09-16T14:43:33.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6a9effd640 1 -- 172.21.15.189:0/2507326579 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3857+0+0 (unknown 1040354461 0 0) 0x7f6aa4043780 con 0x7f6aac0a19b0 2024-09-16T14:43:33.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.260+0000 7f6ab2509640 10 monclient: get_auth_request con 0x7f6a90036080 auth_method 0 2024-09-16T14:43:33.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.264+0000 7f6a9effd640 1 -- 172.21.15.189:0/2507326579 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6aa4014970 con 0x7f6aac0a19b0 2024-09-16T14:43:33.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.264+0000 7f6a9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:33.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.264+0000 7f6a9effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:33.279 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:43:33.280 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:43:33.280 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:43:33.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 10 monclient: get_monmap_and_config 2024-09-16T14:43:33.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 10 monclient: build_initial_monmap 2024-09-16T14:43:33.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 10 monclient: monmap: 2024-09-16T14:43:33.401 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:33.401 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.401 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:43:33.401822+0000 2024-09-16T14:43:33.401 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:43:33.401822+0000 2024-09-16T14:43:33.401 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:33.401 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.401 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:33.402 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:33.402 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:33.402 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 1 Processor -- start 2024-09-16T14:43:33.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 1 -- start start 2024-09-16T14:43:33.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 10 monclient: init 2024-09-16T14:43:33.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:33.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:43:33.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 10 monclient(hunting): picked mon.noname-b con 0x7fa2e009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:33.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 10 monclient(hunting): picked mon.noname-a con 0x7fa2e0132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:33.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 10 monclient(hunting): picked mon.noname-c con 0x7fa2e0138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:33.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e009d260 con 0x7fa2e0132650 2024-09-16T14:43:33.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e0132a80 con 0x7fa2e0138ff0 2024-09-16T14:43:33.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e009cf60 con 0x7fa2e009cb80 2024-09-16T14:43:33.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:33.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 10 monclient(hunting): authenticate will time out at 1037.522004s 2024-09-16T14:43:33.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e574f640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa2e009cb80 0x7fa2e00717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40252/0 (socket says 172.21.15.26:40252) 2024-09-16T14:43:33.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e574f640 1 -- 172.21.15.26:0/1900700676 learned_addr learned my addr 172.21.15.26:0/1900700676 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:33.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e4f4e640 1 --1- 172.21.15.26:0/1900700676 >> v1:172.21.15.26:6789/0 conn(0x7fa2e0132650 0x7fa2e01378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51858/0 (socket says 172.21.15.26:51858) 2024-09-16T14:43:33.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 1 -- 172.21.15.26:0/1900700676 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384258432 0 0) 0x7fa2e009cf60 con 0x7fa2e009cb80 2024-09-16T14:43:33.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient(hunting): my global_id is 4442 2024-09-16T14:43:33.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 1 -- 172.21.15.26:0/1900700676 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c8003530 con 0x7fa2e009cb80 2024-09-16T14:43:33.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 1 -- 172.21.15.26:0/1900700676 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759547534 0 0) 0x7fa2e009d260 con 0x7fa2e0132650 2024-09-16T14:43:33.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient(hunting): my global_id is 74106 2024-09-16T14:43:33.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 1 -- 172.21.15.26:0/1900700676 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e009cf60 con 0x7fa2e0132650 2024-09-16T14:43:33.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 1 -- 172.21.15.26:0/1900700676 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907730851 0 0) 0x7fa2e0132a80 con 0x7fa2e0138ff0 2024-09-16T14:43:33.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient(hunting): my global_id is 4477 2024-09-16T14:43:33.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 1 -- 172.21.15.26:0/1900700676 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e009d260 con 0x7fa2e0138ff0 2024-09-16T14:43:33.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 1 -- 172.21.15.26:0/1900700676 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 653235070 0 0) 0x7fa2c8003530 con 0x7fa2e009cb80 2024-09-16T14:43:33.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 1 -- 172.21.15.26:0/1900700676 >> v1:172.21.15.189:6789/0 conn(0x7fa2e0138ff0 legacy=0x7fa2e013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 1 -- 172.21.15.26:0/1900700676 >> v1:172.21.15.26:6789/0 conn(0x7fa2e0132650 legacy=0x7fa2e01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient: _finish_hunting 0 2024-09-16T14:43:33.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 1 monclient: found mon.noname-b 2024-09-16T14:43:33.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:33.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:43:33.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 1 -- 172.21.15.26:0/1900700676 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e009f2d0 con 0x7fa2e009cb80 2024-09-16T14:43:33.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient: _finish_auth 0 2024-09-16T14:43:33.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:33.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:33.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 1 -- 172.21.15.26:0/1900700676 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa2d80019e0 con 0x7fa2e009cb80 2024-09-16T14:43:33.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient: dump: 2024-09-16T14:43:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.409 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 5 monclient: authenticate success, global_id 4442 2024-09-16T14:43:33.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:33.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 1 -- 172.21.15.26:0/1900700676 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2d8001c00 con 0x7fa2e009cb80 2024-09-16T14:43:33.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:33.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:33.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 1 -- 172.21.15.26:0/1900700676 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa2d8001fd0 con 0x7fa2e009cb80 2024-09-16T14:43:33.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 4 set_mon_vals no callback set 2024-09-16T14:43:33.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:33.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2dffff640 10 monclient: dump: 2024-09-16T14:43:33.410 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.410 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.410 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.411 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.411 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.411 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.411 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.411 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.411 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.411 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 10 monclient: shutdown 2024-09-16T14:43:33.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:33.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 1 -- 172.21.15.26:0/1900700676 >> v1:172.21.15.26:6790/0 conn(0x7fa2e009cb80 legacy=0x7fa2e00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 1 -- 172.21.15.26:0/1900700676 shutdown_connections 2024-09-16T14:43:33.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 1 -- 172.21.15.26:0/1900700676 >> 172.21.15.26:0/1900700676 conn(0x7fa2e009fb50 msgr2=0x7fa2e009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:33.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 1 -- 172.21.15.26:0/1900700676 shutdown_connections 2024-09-16T14:43:33.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.399+0000 7fa2e79b3640 1 -- 172.21.15.26:0/1900700676 wait complete. 2024-09-16T14:43:33.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 10 monclient: build_initial_monmap 2024-09-16T14:43:33.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 10 monclient: monmap: 2024-09-16T14:43:33.412 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:33.412 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:33.412 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:43:33.412 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:43:33.412 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:33.412 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:43:33.413 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:33.413 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:33.413 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:33.413 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 1 Processor -- start 2024-09-16T14:43:33.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 1 -- start start 2024-09-16T14:43:33.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 10 monclient: init 2024-09-16T14:43:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:43:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 10 monclient(hunting): picked mon.noname-b con 0x7fa2e009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 10 monclient(hunting): picked mon.noname-c con 0x7fa2e0132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 10 monclient(hunting): picked mon.noname-a con 0x7fa2e0138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e00a0e50 con 0x7fa2e0138ff0 2024-09-16T14:43:33.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e0070fc0 con 0x7fa2e009cb80 2024-09-16T14:43:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e00a2a70 con 0x7fa2e0132650 2024-09-16T14:43:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 10 monclient(hunting): authenticate will time out at 1037.525228s 2024-09-16T14:43:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e5f50640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa2e0138ff0 0x7fa2e006fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51860/0 (socket says 172.21.15.26:51860) 2024-09-16T14:43:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e4f4e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa2e0132650 0x7fa2e006f5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40258/0 (socket says 172.21.15.26:40258) 2024-09-16T14:43:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e5f50640 1 -- 172.21.15.26:0/1199942238 learned_addr learned my addr 172.21.15.26:0/1199942238 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221927674 0 0) 0x7fa2e00a2a70 con 0x7fa2e0132650 2024-09-16T14:43:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): my global_id is 4445 2024-09-16T14:43:33.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2cc003530 con 0x7fa2e0132650 2024-09-16T14:43:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247493344 0 0) 0x7fa2e00a0e50 con 0x7fa2e0138ff0 2024-09-16T14:43:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): my global_id is 74109 2024-09-16T14:43:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e00a2a70 con 0x7fa2e0138ff0 2024-09-16T14:43:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292762941 0 0) 0x7fa2e0070fc0 con 0x7fa2e009cb80 2024-09-16T14:43:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): my global_id is 4480 2024-09-16T14:43:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e00a0e50 con 0x7fa2e009cb80 2024-09-16T14:43:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3645645455 0 0) 0x7fa2e00a2a70 con 0x7fa2e0138ff0 2024-09-16T14:43:33.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2e0070fc0 con 0x7fa2e0138ff0 2024-09-16T14:43:33.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa2d0001880 con 0x7fa2e0138ff0 2024-09-16T14:43:33.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:43:33.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): dump: 2024-09-16T14:43:33.424 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.424 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.424 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.424 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.424 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.424 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.424 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.425 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.425 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.425 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:33.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 369594384 0 0) 0x7fa2cc003530 con 0x7fa2e0132650 2024-09-16T14:43:33.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2e00a2a70 con 0x7fa2e0132650 2024-09-16T14:43:33.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa2d4001960 con 0x7fa2e0132650 2024-09-16T14:43:33.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:33.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): dump: 2024-09-16T14:43:33.425 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.425 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.425 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.426 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.426 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.426 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.426 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.426 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.426 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.426 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3911202768 0 0) 0x7fa2e0070fc0 con 0x7fa2e0138ff0 2024-09-16T14:43:33.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 >> v1:172.21.15.26:6790/0 conn(0x7fa2e0132650 legacy=0x7fa2e006f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 >> v1:172.21.15.189:6789/0 conn(0x7fa2e009cb80 legacy=0x7fa2e006eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient: _finish_hunting 0 2024-09-16T14:43:33.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 monclient: found mon.a 2024-09-16T14:43:33.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:33.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:33.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e009a240 con 0x7fa2e0138ff0 2024-09-16T14:43:33.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient: _finish_auth 0 2024-09-16T14:43:33.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:43:33.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:33.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 5 monclient: authenticate success, global_id 74109 2024-09-16T14:43:33.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 10 monclient: _renew_subs 2024-09-16T14:43:33.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:33.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 1 -- 172.21.15.26:0/1199942238 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e0134ad0 con 0x7fa2e0138ff0 2024-09-16T14:43:33.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2d0003620 con 0x7fa2e0138ff0 2024-09-16T14:43:33.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:33.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa2d0003930 con 0x7fa2e0138ff0 2024-09-16T14:43:33.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient: dump: 2024-09-16T14:43:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.429 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 10 monclient: _renew_subs 2024-09-16T14:43:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 1 -- 172.21.15.26:0/1199942238 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e01d2cc0 con 0x7fa2e0138ff0 2024-09-16T14:43:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2df7fe640 4 set_mon_vals no callback set 2024-09-16T14:43:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e79b3640 1 -- 172.21.15.26:0/1199942238 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa2e00008e0 con 0x7fa2e012dee0 2024-09-16T14:43:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa2d0003a90 con 0x7fa2e0138ff0 2024-09-16T14:43:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3857+0+0 (unknown 1040354461 0 0) 0x7fa2d0043100 con 0x7fa2e0138ff0 2024-09-16T14:43:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544198518 0 0) 0x7fa2e00008e0 con 0x7fa2e012dee0 2024-09-16T14:43:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient(hunting): my global_id is 74112 2024-09-16T14:43:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e0070fc0 con 0x7fa2e012dee0 2024-09-16T14:43:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2e4f4e640 10 monclient: get_auth_request con 0x7fa2cc036490 auth_method 0 2024-09-16T14:43:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2992103516 0 0) 0x7fa2e0070fc0 con 0x7fa2e012dee0 2024-09-16T14:43:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2e00008e0 con 0x7fa2e012dee0 2024-09-16T14:43:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa2d8001d70 con 0x7fa2e012dee0 2024-09-16T14:43:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 10 monclient: dump: 2024-09-16T14:43:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.430 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.431 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 184897647 0 0) 0x7fa2e00008e0 con 0x7fa2e012dee0 2024-09-16T14:43:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.403+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2e009c330 con 0x7fa2e012dee0 2024-09-16T14:43:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.407+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa2d8002070 con 0x7fa2e012dee0 2024-09-16T14:43:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.407+0000 7fa2ddffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.407+0000 7fa2ddffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.407+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 >> v1:172.21.15.26:6789/0 conn(0x7fa2e012dee0 legacy=0x7fa2e012e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.407+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 >> v1:172.21.15.26:6789/0 conn(0x7fa2e012dee0 legacy=0x7fa2e012e2c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2e79b3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:43:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2e79b3640 1 -- 172.21.15.26:0/1199942238 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa2e012ede0 con 0x7fa2e012e9d0 2024-09-16T14:43:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356746228 0 0) 0x7fa2e012ede0 con 0x7fa2e012e9d0 2024-09-16T14:43:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 10 monclient(hunting): my global_id is 74115 2024-09-16T14:43:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e00008e0 con 0x7fa2e012e9d0 2024-09-16T14:43:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 941714198 0 0) 0x7fa2e00008e0 con 0x7fa2e012e9d0 2024-09-16T14:43:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2e012ede0 con 0x7fa2e012e9d0 2024-09-16T14:43:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa2d8005010 con 0x7fa2e012e9d0 2024-09-16T14:43:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 10 monclient: dump: 2024-09-16T14:43:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.433 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 998492912 0 0) 0x7fa2e012ede0 con 0x7fa2e012e9d0 2024-09-16T14:43:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa2e009f5b0 con 0x7fa2e012e9d0 2024-09-16T14:43:33.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 3708966580) 0x7fa2d8005310 con 0x7fa2e012e9d0 2024-09-16T14:43:33.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:43:33.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:43:33.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 >> v1:172.21.15.26:6789/0 conn(0x7fa2e012e9d0 legacy=0x7fa2e01d6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2ddffb640 1 -- 172.21.15.26:0/1199942238 >> v1:172.21.15.26:6789/0 conn(0x7fa2e012e9d0 legacy=0x7fa2e01d6e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:33.434 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:43:33.434 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:43:33.434 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:43:33.434 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:43:33.434 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 16, 2024-09-16T14:43:33.435 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:43:33.435 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:43:33.435 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:43:33.435 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:43:33.435 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:33.435 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:43:33.435 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:43:33.435 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:43:33.435 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:43:33.435 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:33.435 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:33.435 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:33.435 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:33.435 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:33.436 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:33.436 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:33.436 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:33.436 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:33.436 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:33.436 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:33.436 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:43:33.436 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:43:33.436 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:33.436 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:33.436 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:33.436 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:33.436 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:33.437 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:33.437 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:33.437 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:33.437 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:33.437 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:33.437 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.437 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:43:33.437 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:43:33.437 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:43:33.437 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:43:33.437 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:43:33.437 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:43:33.438 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:43:33.438 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:43:33.438 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:43:33.438 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:43:33.438 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:43:33.438 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:43:33.438 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:43:33.438 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:43:33.438 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:43:33.438 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:43:33.438 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:43:33.438 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:33.438 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:33.439 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:33.439 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:33.439 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:33.439 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:33.439 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:33.439 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:33.439 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:33.439 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:33.439 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:33.439 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:43:33.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.439 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:43:33.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.440 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:43:33.440 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:43:33.440 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:33.440 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:33.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.440 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:33.440 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:43:33.440 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:33.440 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.440 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.440 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:43:33.441 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:43:33.441 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:33.441 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:33.441 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:33.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.441 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:43:33.441 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:43:33.441 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:33.441 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:33.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.441 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:33.441 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:43:33.442 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:33.442 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.442 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.442 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:43:33.442 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:43:33.442 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:33.442 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:33.442 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:33.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.442 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:43:33.442 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:43:33.443 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:33.443 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:33.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.443 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:33.443 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:43:33.443 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:33.443 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.443 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.444 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:43:33.444 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:43:33.444 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:33.444 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:33.444 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:33.444 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.444 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.444 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:43:33.444 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:43:33.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.444 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:33.444 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:33.444 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:43:33.445 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.445 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:33.445 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:43:33.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.445 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:33.445 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:33.445 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:43:33.445 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.445 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.445 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:43:33.445 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:43:33.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.424+0000 7f6ab476d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T14:43:33.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.424+0000 7f6ab476d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T14:43:33.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.424+0000 7f6ab476d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:33.424+0000 7f6ab476d640 1 -- 172.21.15.189:0/2507326579 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"} v 0) v1 -- 0x7f6a94005880 con 0x7f6aac0a19b0 2024-09-16T14:43:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2e79b3640 1 -- 172.21.15.26:0/1199942238 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa2cc036490 legacy=0x7fa2cc038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2e79b3640 10 monclient: shutdown 2024-09-16T14:43:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2e79b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:33.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2e79b3640 1 -- 172.21.15.26:0/1199942238 >> v1:172.21.15.26:6789/0 conn(0x7fa2e0138ff0 legacy=0x7fa2e006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2e5f50640 1 -- 172.21.15.26:0/1199942238 reap_dead start 2024-09-16T14:43:33.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2e5f50640 1 -- 172.21.15.26:0/1199942238 reap_dead start 2024-09-16T14:43:33.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2e79b3640 1 -- 172.21.15.26:0/1199942238 shutdown_connections 2024-09-16T14:43:33.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2e79b3640 1 -- 172.21.15.26:0/1199942238 >> 172.21.15.26:0/1199942238 conn(0x7fa2e009fb50 msgr2=0x7fa2e009a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:33.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2e79b3640 1 -- 172.21.15.26:0/1199942238 shutdown_connections 2024-09-16T14:43:33.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.411+0000 7fa2e79b3640 1 -- 172.21.15.26:0/1199942238 wait complete. 2024-09-16T14:43:33.448 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:43:33.448 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:43:33.449 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:43:33.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 10 monclient: get_monmap_and_config 2024-09-16T14:43:33.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 10 monclient: build_initial_monmap 2024-09-16T14:43:33.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 10 monclient: monmap: 2024-09-16T14:43:33.562 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:33.563 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.563 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:43:33.561547+0000 2024-09-16T14:43:33.563 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:43:33.561547+0000 2024-09-16T14:43:33.563 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:33.563 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.563 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:33.563 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:33.563 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:33.563 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 1 Processor -- start 2024-09-16T14:43:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 1 -- start start 2024-09-16T14:43:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 10 monclient: init 2024-09-16T14:43:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:43:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 10 monclient(hunting): picked mon.noname-b con 0x7f1ccc09cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 10 monclient(hunting): picked mon.noname-c con 0x7f1ccc132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 10 monclient(hunting): picked mon.noname-a con 0x7f1ccc138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ccc09d260 con 0x7f1ccc138ff0 2024-09-16T14:43:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ccc132a80 con 0x7f1ccc132650 2024-09-16T14:43:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ccc09cf60 con 0x7f1ccc09cb80 2024-09-16T14:43:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 10 monclient(hunting): authenticate will time out at 1037.681528s 2024-09-16T14:43:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccbfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1ccc09cb80 0x7f1ccc0717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40264/0 (socket says 172.21.15.26:40264) 2024-09-16T14:43:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd0f4e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1ccc138ff0 0x7f1ccc13b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51898/0 (socket says 172.21.15.26:51898) 2024-09-16T14:43:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccbfff640 1 -- 172.21.15.26:0/66854573 learned_addr learned my addr 172.21.15.26:0/66854573 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 1 -- 172.21.15.26:0/66854573 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074553977 0 0) 0x7f1ccc09d260 con 0x7f1ccc138ff0 2024-09-16T14:43:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient(hunting): my global_id is 74118 2024-09-16T14:43:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 1 -- 172.21.15.26:0/66854573 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cb4003530 con 0x7f1ccc138ff0 2024-09-16T14:43:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 1 -- 172.21.15.26:0/66854573 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020366815 0 0) 0x7f1ccc09cf60 con 0x7f1ccc09cb80 2024-09-16T14:43:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient(hunting): my global_id is 4448 2024-09-16T14:43:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 1 -- 172.21.15.26:0/66854573 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ccc09d260 con 0x7f1ccc09cb80 2024-09-16T14:43:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd0f4e640 1 -- 172.21.15.26:0/66854573 >> v1:172.21.15.26:6789/0 conn(0x7f1ccc138ff0 legacy=0x7f1ccc13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T14:43:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd0f4e640 1 -- 172.21.15.26:0/66854573 >> v1:172.21.15.26:6789/0 conn(0x7f1ccc138ff0 legacy=0x7f1ccc13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:43:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd0f4e640 1 --1- 172.21.15.26:0/66854573 >> v1:172.21.15.26:6789/0 conn(0x7f1ccc138ff0 0x7f1ccc13b3e0 :-1 s=OPENED pgs=30 cs=1 l=1).handle_message read tag failed 2024-09-16T14:43:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd0f4e640 1 --1- 172.21.15.26:0/66854573 >> v1:172.21.15.26:6789/0 conn(0x7f1ccc138ff0 0x7f1ccc13b3e0 :-1 s=OPENED pgs=30 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:43:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6789/0 2024-09-16T14:43:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 1 -- 172.21.15.26:0/66854573 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3108468392 0 0) 0x7f1ccc09d260 con 0x7f1ccc09cb80 2024-09-16T14:43:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 1 -- 172.21.15.26:0/66854573 >> v1:172.21.15.189:6789/0 conn(0x7f1ccc132650 legacy=0x7f1ccc1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:43:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 1 -- 172.21.15.26:0/66854573 >> v1:172.21.15.26:6789/0 conn(0x7f1ccc138ff0 legacy=0x7f1ccc13b3e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient: _finish_hunting 0 2024-09-16T14:43:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 1 monclient: found mon.noname-b 2024-09-16T14:43:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:43:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 1 -- 172.21.15.26:0/66854573 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc09f2d0 con 0x7f1ccc09cb80 2024-09-16T14:43:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient: _finish_auth 0 2024-09-16T14:43:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient: _check_auth_tickets 2024-09-16T14:43:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 1 -- 172.21.15.26:0/66854573 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1cc0001590 con 0x7f1ccc09cb80 2024-09-16T14:43:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 5 monclient: authenticate success, global_id 4448 2024-09-16T14:43:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient: dump: 2024-09-16T14:43:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.576 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 1 -- 172.21.15.26:0/66854573 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cc0001770 con 0x7f1ccc09cb80 2024-09-16T14:43:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 1 -- 172.21.15.26:0/66854573 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1cc0001b40 con 0x7f1ccc09cb80 2024-09-16T14:43:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1ccaffd640 10 monclient: dump: 2024-09-16T14:43:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.578 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 4 set_mon_vals no callback set 2024-09-16T14:43:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 10 monclient: shutdown 2024-09-16T14:43:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 1 -- 172.21.15.26:0/66854573 >> v1:172.21.15.26:6790/0 conn(0x7f1ccc09cb80 legacy=0x7f1ccc0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 1 -- 172.21.15.26:0/66854573 shutdown_connections 2024-09-16T14:43:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 1 -- 172.21.15.26:0/66854573 >> 172.21.15.26:0/66854573 conn(0x7f1ccc09fb50 msgr2=0x7f1ccc09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 1 -- 172.21.15.26:0/66854573 shutdown_connections 2024-09-16T14:43:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 1 -- 172.21.15.26:0/66854573 wait complete. 2024-09-16T14:43:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 10 monclient: build_initial_monmap 2024-09-16T14:43:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.559+0000 7f1cd29b1640 10 monclient: monmap: 2024-09-16T14:43:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:43:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:43:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:43:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:33.579 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 1 Processor -- start 2024-09-16T14:43:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 1 -- start start 2024-09-16T14:43:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 10 monclient: init 2024-09-16T14:43:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:43:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 10 monclient(hunting): picked mon.noname-c con 0x7f1ccc09cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 10 monclient(hunting): picked mon.noname-a con 0x7f1ccc132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 10 monclient(hunting): picked mon.noname-b con 0x7f1ccc138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ccc06da30 con 0x7f1ccc132650 2024-09-16T14:43:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ccc06dd10 con 0x7f1ccc138ff0 2024-09-16T14:43:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ccc12e4a0 con 0x7f1ccc09cb80 2024-09-16T14:43:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 10 monclient(hunting): authenticate will time out at 1037.684857s 2024-09-16T14:43:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1ccbfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1ccc09cb80 0x7f1ccc12d680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40278/0 (socket says 172.21.15.26:40278) 2024-09-16T14:43:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1ccbfff640 1 -- 172.21.15.26:0/1274320793 learned_addr learned my addr 172.21.15.26:0/1274320793 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077006097 0 0) 0x7f1ccc06da30 con 0x7f1ccc132650 2024-09-16T14:43:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): my global_id is 74121 2024-09-16T14:43:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cb8003530 con 0x7f1ccc132650 2024-09-16T14:43:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165722434 0 0) 0x7f1ccc12e4a0 con 0x7f1ccc09cb80 2024-09-16T14:43:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): my global_id is 4451 2024-09-16T14:43:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ccc06da30 con 0x7f1ccc09cb80 2024-09-16T14:43:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2842726865 0 0) 0x7f1cb8003530 con 0x7f1ccc132650 2024-09-16T14:43:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ccc12e4a0 con 0x7f1ccc132650 2024-09-16T14:43:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1cbc001800 con 0x7f1ccc132650 2024-09-16T14:43:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:43:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): dump: 2024-09-16T14:43:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.583 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2517218391 0 0) 0x7f1ccc06da30 con 0x7f1ccc09cb80 2024-09-16T14:43:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cb8003530 con 0x7f1ccc09cb80 2024-09-16T14:43:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1cc0001a80 con 0x7f1ccc09cb80 2024-09-16T14:43:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): dump: 2024-09-16T14:43:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.585 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 897881442 0 0) 0x7f1ccc12e4a0 con 0x7f1ccc132650 2024-09-16T14:43:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 >> v1:172.21.15.26:6790/0 conn(0x7f1ccc09cb80 legacy=0x7f1ccc12d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 >> v1:172.21.15.189:6789/0 conn(0x7f1ccc138ff0 legacy=0x7f1ccc1d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:43:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 monclient: found mon.a 2024-09-16T14:43:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc0a2060 con 0x7f1ccc132650 2024-09-16T14:43:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient: _finish_auth 0 2024-09-16T14:43:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:43:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 5 monclient: authenticate success, global_id 74121 2024-09-16T14:43:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 10 monclient: _renew_subs 2024-09-16T14:43:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 1 -- 172.21.15.26:0/1274320793 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ccc12fe00 con 0x7f1ccc132650 2024-09-16T14:43:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 10 monclient: _renew_subs 2024-09-16T14:43:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 1 -- 172.21.15.26:0/1274320793 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ccc12f710 con 0x7f1ccc132650 2024-09-16T14:43:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cbc003600 con 0x7f1ccc132650 2024-09-16T14:43:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1cbc003910 con 0x7f1ccc132650 2024-09-16T14:43:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient: dump: 2024-09-16T14:43:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.587 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cca7fc640 4 set_mon_vals no callback set 2024-09-16T14:43:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:33.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd29b1640 1 -- 172.21.15.26:0/1274320793 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1cac006590 con 0x7f1cac002830 2024-09-16T14:43:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1cbc00e380 con 0x7f1ccc132650 2024-09-16T14:43:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3857+0+0 (unknown 1040354461 0 0) 0x7f1cbc0420e0 con 0x7f1ccc132650 2024-09-16T14:43:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235315395 0 0) 0x7f1cac006590 con 0x7f1cac002830 2024-09-16T14:43:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient(hunting): my global_id is 4454 2024-09-16T14:43:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ccc12e4a0 con 0x7f1cac002830 2024-09-16T14:43:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1155619409 0 0) 0x7f1ccc12e4a0 con 0x7f1cac002830 2024-09-16T14:43:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cac006590 con 0x7f1cac002830 2024-09-16T14:43:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1cc0004990 con 0x7f1cac002830 2024-09-16T14:43:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient: dump: 2024-09-16T14:43:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.589 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cd0f4e640 10 monclient: get_auth_request con 0x7f1cb8036220 auth_method 0 2024-09-16T14:43:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4075165425 0 0) 0x7f1cac006590 con 0x7f1cac002830 2024-09-16T14:43:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cac00a0e0 con 0x7f1cac002830 2024-09-16T14:43:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1cc0006bb0 con 0x7f1cac002830 2024-09-16T14:43:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 >> v1:172.21.15.26:6790/0 conn(0x7f1cac002830 legacy=0x7f1cac004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.563+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 >> v1:172.21.15.26:6790/0 conn(0x7f1cac002830 legacy=0x7f1cac004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cd29b1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:43:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cd29b1640 1 -- 172.21.15.26:0/1274320793 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1cac009da0 con 0x7f1cac0090c0 2024-09-16T14:43:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714108414 0 0) 0x7f1cac009da0 con 0x7f1cac0090c0 2024-09-16T14:43:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 10 monclient(hunting): my global_id is 4457 2024-09-16T14:43:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cac006590 con 0x7f1cac0090c0 2024-09-16T14:43:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2561985454 0 0) 0x7f1cac006590 con 0x7f1cac0090c0 2024-09-16T14:43:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cac009da0 con 0x7f1cac0090c0 2024-09-16T14:43:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1cc00050c0 con 0x7f1cac0090c0 2024-09-16T14:43:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 10 monclient: dump: 2024-09-16T14:43:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.593 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1366399214 0 0) 0x7f1cac009da0 con 0x7f1cac0090c0 2024-09-16T14:43:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1cac00c290 con 0x7f1cac0090c0 2024-09-16T14:43:33.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 3966491157) 0x7f1cc0006520 con 0x7f1cac0090c0 2024-09-16T14:43:33.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:43:33.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:43:33.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 >> v1:172.21.15.26:6790/0 conn(0x7f1cac0090c0 legacy=0x7f1cac009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cc8ff9640 1 -- 172.21.15.26:0/1274320793 >> v1:172.21.15.26:6790/0 conn(0x7f1cac0090c0 legacy=0x7f1cac009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:33.594 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:43:33.594 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:43:33.594 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:43:33.594 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:43:33.594 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 16, 2024-09-16T14:43:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:43:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:43:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:43:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:43:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:43:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:43:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:43:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:43:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:33.595 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:43:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:43:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:33.596 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:43:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:43:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:43:33.597 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:43:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:43:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:43:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:43:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:43:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:43:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:43:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:43:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:43:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:43:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:43:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:43:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:43:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:43:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:43:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.599 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:43:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:43:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:43:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:43:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.600 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:43:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:43:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:43:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:43:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:43:33.601 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:43:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:43:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:43:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:43:33.602 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:43:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:43:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:43:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:33.603 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:43:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:43:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:43:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:43:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.604 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:43:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:43:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:43:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.605 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:43:33.605 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:43:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cd29b1640 1 -- 172.21.15.26:0/1274320793 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1cb8036220 legacy=0x7f1cb80386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cd29b1640 10 monclient: shutdown 2024-09-16T14:43:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cd29b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cd29b1640 1 -- 172.21.15.26:0/1274320793 >> v1:172.21.15.26:6789/0 conn(0x7f1ccc132650 legacy=0x7f1ccc12dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cd0f4e640 1 -- 172.21.15.26:0/1274320793 reap_dead start 2024-09-16T14:43:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cd0f4e640 1 -- 172.21.15.26:0/1274320793 reap_dead start 2024-09-16T14:43:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cd29b1640 1 -- 172.21.15.26:0/1274320793 shutdown_connections 2024-09-16T14:43:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cd29b1640 1 -- 172.21.15.26:0/1274320793 >> 172.21.15.26:0/1274320793 conn(0x7f1ccc09fb50 msgr2=0x7f1ccc0a25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cd29b1640 1 -- 172.21.15.26:0/1274320793 shutdown_connections 2024-09-16T14:43:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.571+0000 7f1cd29b1640 1 -- 172.21.15.26:0/1274320793 wait complete. 2024-09-16T14:43:33.608 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:43:33.608 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:43:33.608 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:43:33.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 10 monclient: get_monmap_and_config 2024-09-16T14:43:33.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 10 monclient: build_initial_monmap 2024-09-16T14:43:33.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 10 monclient: monmap: 2024-09-16T14:43:33.723 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:33.723 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.723 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:43:33.722651+0000 2024-09-16T14:43:33.723 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:43:33.722651+0000 2024-09-16T14:43:33.723 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:33.724 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.724 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:33.724 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:33.724 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:33.724 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 1 Processor -- start 2024-09-16T14:43:33.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 1 -- start start 2024-09-16T14:43:33.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 10 monclient: init 2024-09-16T14:43:33.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:33.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:43:33.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 10 monclient(hunting): picked mon.noname-a con 0x7ff2500a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:33.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 10 monclient(hunting): picked mon.noname-c con 0x7ff2500a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:33.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 10 monclient(hunting): picked mon.noname-b con 0x7ff2500a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:33.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2500a3250 con 0x7ff2500a1d10 2024-09-16T14:43:33.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25009f600 con 0x7ff2500a0000 2024-09-16T14:43:33.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2500a3830 con 0x7ff2500a2800 2024-09-16T14:43:33.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:33.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 10 monclient(hunting): authenticate will time out at 1037.842710s 2024-09-16T14:43:33.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff255e92640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff2500a1d10 0x7ff2500a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51928/0 (socket says 172.21.15.26:51928) 2024-09-16T14:43:33.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff255e92640 1 -- 172.21.15.26:0/1111792456 learned_addr learned my addr 172.21.15.26:0/1111792456 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:33.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff256693640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff2500a2800 0x7ff25013d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40308/0 (socket says 172.21.15.26:40308) 2024-09-16T14:43:33.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 1 -- 172.21.15.26:0/1111792456 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070840093 0 0) 0x7ff2500a3250 con 0x7ff2500a1d10 2024-09-16T14:43:33.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient(hunting): my global_id is 74124 2024-09-16T14:43:33.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 1 -- 172.21.15.26:0/1111792456 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff23c003530 con 0x7ff2500a1d10 2024-09-16T14:43:33.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 1 -- 172.21.15.26:0/1111792456 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124270827 0 0) 0x7ff2500a3830 con 0x7ff2500a2800 2024-09-16T14:43:33.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient(hunting): my global_id is 4460 2024-09-16T14:43:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 1 -- 172.21.15.26:0/1111792456 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2500a3250 con 0x7ff2500a2800 2024-09-16T14:43:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 1 -- 172.21.15.26:0/1111792456 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4090976107 0 0) 0x7ff23c003530 con 0x7ff2500a1d10 2024-09-16T14:43:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 1 -- 172.21.15.26:0/1111792456 >> v1:172.21.15.26:6790/0 conn(0x7ff2500a2800 legacy=0x7ff25013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 1 -- 172.21.15.26:0/1111792456 >> v1:172.21.15.189:6789/0 conn(0x7ff2500a0000 legacy=0x7ff250139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient: _finish_hunting 0 2024-09-16T14:43:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 1 monclient: found mon.noname-a 2024-09-16T14:43:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:43:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 1 -- 172.21.15.26:0/1111792456 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25009ed60 con 0x7ff2500a1d10 2024-09-16T14:43:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient: _finish_auth 0 2024-09-16T14:43:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient: _check_auth_tickets 2024-09-16T14:43:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 1 -- 172.21.15.26:0/1111792456 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff240001620 con 0x7ff2500a1d10 2024-09-16T14:43:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:43:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient: dump: 2024-09-16T14:43:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.732 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.733 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 5 monclient: authenticate success, global_id 74124 2024-09-16T14:43:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 1 -- 172.21.15.26:0/1111792456 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff240001840 con 0x7ff2500a1d10 2024-09-16T14:43:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 1 -- 172.21.15.26:0/1111792456 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff240001c10 con 0x7ff2500a1d10 2024-09-16T14:43:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff2580f6640 4 set_mon_vals no callback set 2024-09-16T14:43:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:33.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.719+0000 7ff254e90640 10 monclient: dump: 2024-09-16T14:43:33.734 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.734 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.734 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.734 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.734 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.734 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.734 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.734 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.734 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.734 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 10 monclient: shutdown 2024-09-16T14:43:33.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 1 -- 172.21.15.26:0/1111792456 >> v1:172.21.15.26:6789/0 conn(0x7ff2500a1d10 legacy=0x7ff2500a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 1 -- 172.21.15.26:0/1111792456 shutdown_connections 2024-09-16T14:43:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 1 -- 172.21.15.26:0/1111792456 >> 172.21.15.26:0/1111792456 conn(0x7ff2500a7f30 msgr2=0x7ff2500a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 1 -- 172.21.15.26:0/1111792456 shutdown_connections 2024-09-16T14:43:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 1 -- 172.21.15.26:0/1111792456 wait complete. 2024-09-16T14:43:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 10 monclient: build_initial_monmap 2024-09-16T14:43:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 10 monclient: monmap: 2024-09-16T14:43:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:43:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:43:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:43:33.735 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:33.736 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 1 Processor -- start 2024-09-16T14:43:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 1 -- start start 2024-09-16T14:43:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 10 monclient: init 2024-09-16T14:43:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:43:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 10 monclient(hunting): picked mon.noname-b con 0x7ff2500a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 10 monclient(hunting): picked mon.noname-a con 0x7ff2500a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 10 monclient(hunting): picked mon.noname-c con 0x7ff2500a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:33.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff250130ee0 con 0x7ff2500a1d10 2024-09-16T14:43:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25009e500 con 0x7ff2500a0000 2024-09-16T14:43:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2500a68c0 con 0x7ff2500a2800 2024-09-16T14:43:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 10 monclient(hunting): authenticate will time out at 1037.845737s 2024-09-16T14:43:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff255691640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff2500a1d10 0x7ff2501307d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51940/0 (socket says 172.21.15.26:51940) 2024-09-16T14:43:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff255691640 1 -- 172.21.15.26:0/1492446421 learned_addr learned my addr 172.21.15.26:0/1492446421 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944429441 0 0) 0x7ff250130ee0 con 0x7ff2500a1d10 2024-09-16T14:43:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): my global_id is 74127 2024-09-16T14:43:33.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff234003530 con 0x7ff2500a1d10 2024-09-16T14:43:33.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991353908 0 0) 0x7ff2500a68c0 con 0x7ff2500a2800 2024-09-16T14:43:33.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): my global_id is 4463 2024-09-16T14:43:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff250130ee0 con 0x7ff2500a2800 2024-09-16T14:43:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4280293188 0 0) 0x7ff234003530 con 0x7ff2500a1d10 2024-09-16T14:43:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2500a68c0 con 0x7ff2500a1d10 2024-09-16T14:43:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff24c001940 con 0x7ff2500a1d10 2024-09-16T14:43:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:43:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): dump: 2024-09-16T14:43:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.739 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.740 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.740 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.740 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.740 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.740 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.740 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:33.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4259850731 0 0) 0x7ff250130ee0 con 0x7ff2500a2800 2024-09-16T14:43:33.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff234003530 con 0x7ff2500a2800 2024-09-16T14:43:33.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764888899 0 0) 0x7ff25009e500 con 0x7ff2500a0000 2024-09-16T14:43:33.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): my global_id is 4489 2024-09-16T14:43:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff250130ee0 con 0x7ff2500a0000 2024-09-16T14:43:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff248001480 con 0x7ff2500a2800 2024-09-16T14:43:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient(hunting): dump: 2024-09-16T14:43:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.741 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.742 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 509092180 0 0) 0x7ff2500a68c0 con 0x7ff2500a1d10 2024-09-16T14:43:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 >> v1:172.21.15.26:6790/0 conn(0x7ff2500a2800 legacy=0x7ff2501d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 >> v1:172.21.15.189:6789/0 conn(0x7ff2500a0000 legacy=0x7ff2501300c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient: _finish_hunting 0 2024-09-16T14:43:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 monclient: found mon.a 2024-09-16T14:43:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:33.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2500a6390 con 0x7ff2500a1d10 2024-09-16T14:43:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient: _finish_auth 0 2024-09-16T14:43:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient: _check_auth_tickets 2024-09-16T14:43:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 5 monclient: authenticate success, global_id 74127 2024-09-16T14:43:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 10 monclient: _renew_subs 2024-09-16T14:43:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 1 -- 172.21.15.26:0/1492446421 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff250131b50 con 0x7ff2500a1d10 2024-09-16T14:43:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 10 monclient: _renew_subs 2024-09-16T14:43:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 1 -- 172.21.15.26:0/1492446421 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff250132720 con 0x7ff2500a1d10 2024-09-16T14:43:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff24c003620 con 0x7ff2500a1d10 2024-09-16T14:43:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:33.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff24c003930 con 0x7ff2500a1d10 2024-09-16T14:43:33.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:33.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 10 monclient: dump: 2024-09-16T14:43:33.744 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.744 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.744 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.744 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.744 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.744 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.745 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.745 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.745 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.745 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff247fff640 4 set_mon_vals no callback set 2024-09-16T14:43:33.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:33.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff24c003a90 con 0x7ff2500a1d10 2024-09-16T14:43:33.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2580f6640 1 -- 172.21.15.26:0/1492446421 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff238006590 con 0x7ff238002830 2024-09-16T14:43:33.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.723+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3857+0+0 (unknown 1040354461 0 0) 0x7ff24c0433c0 con 0x7ff2500a1d10 2024-09-16T14:43:33.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff256693640 10 monclient: get_auth_request con 0x7ff234036440 auth_method 0 2024-09-16T14:43:33.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606673099 0 0) 0x7ff238006590 con 0x7ff238002830 2024-09-16T14:43:33.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 10 monclient(hunting): my global_id is 4492 2024-09-16T14:43:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2500a68c0 con 0x7ff238002830 2024-09-16T14:43:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3406068837 0 0) 0x7ff2500a68c0 con 0x7ff238002830 2024-09-16T14:43:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff238006590 con 0x7ff238002830 2024-09-16T14:43:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff2400011d0 con 0x7ff238002830 2024-09-16T14:43:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 10 monclient: dump: 2024-09-16T14:43:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.746 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.747 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.747 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.747 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.747 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.747 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.747 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.747 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1426551488 0 0) 0x7ff238006590 con 0x7ff238002830 2024-09-16T14:43:33.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff23800a0e0 con 0x7ff238002830 2024-09-16T14:43:33.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ff240001b10 con 0x7ff238002830 2024-09-16T14:43:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 >> v1:172.21.15.189:6789/0 conn(0x7ff238002830 legacy=0x7ff238004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.727+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 >> v1:172.21.15.189:6789/0 conn(0x7ff238002830 legacy=0x7ff238004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2580f6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:43:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2580f6640 1 -- 172.21.15.26:0/1492446421 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff238009d80 con 0x7ff2380090a0 2024-09-16T14:43:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126222084 0 0) 0x7ff238009d80 con 0x7ff2380090a0 2024-09-16T14:43:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 10 monclient(hunting): my global_id is 4495 2024-09-16T14:43:33.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff238006590 con 0x7ff2380090a0 2024-09-16T14:43:33.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3335444672 0 0) 0x7ff238006590 con 0x7ff2380090a0 2024-09-16T14:43:33.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff238009d80 con 0x7ff2380090a0 2024-09-16T14:43:33.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff240004ce0 con 0x7ff2380090a0 2024-09-16T14:43:33.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:33.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 10 monclient: dump: 2024-09-16T14:43:33.750 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.750 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.750 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.750 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.750 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.750 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.751 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.751 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.751 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.751 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3850713658 0 0) 0x7ff238009d80 con 0x7ff2380090a0 2024-09-16T14:43:33.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff23800c290 con 0x7ff2380090a0 2024-09-16T14:43:33.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 1690978978) 0x7ff240004ef0 con 0x7ff2380090a0 2024-09-16T14:43:33.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:43:33.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:43:33.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 >> v1:172.21.15.189:6789/0 conn(0x7ff2380090a0 legacy=0x7ff238009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2467fc640 1 -- 172.21.15.26:0/1492446421 >> v1:172.21.15.189:6789/0 conn(0x7ff2380090a0 legacy=0x7ff238009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:33.752 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:43:33.752 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:43:33.752 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:43:33.752 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:43:33.752 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 16, 2024-09-16T14:43:33.752 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:43:33.752 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:43:33.752 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:43:33.752 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:43:33.752 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:33.752 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:43:33.753 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:43:33.753 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:43:33.753 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:43:33.753 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:33.753 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:33.753 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:33.754 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:33.754 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:33.754 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:33.754 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:33.754 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:33.754 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:33.755 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:33.755 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:33.755 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:43:33.755 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:43:33.755 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:33.755 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:33.755 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:33.755 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:33.755 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:33.755 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:33.755 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:33.755 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:33.755 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:33.756 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:33.756 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.756 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:43:33.756 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:43:33.756 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:43:33.756 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:43:33.756 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:43:33.756 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:43:33.756 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:43:33.756 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:43:33.756 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:43:33.756 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:43:33.757 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:43:33.757 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:43:33.757 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:43:33.757 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:43:33.757 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:43:33.757 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:43:33.757 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:43:33.757 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:33.757 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:33.757 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:33.757 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:33.757 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:33.758 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:33.758 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:33.759 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:33.759 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:33.759 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:33.759 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:33.759 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:43:33.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.759 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:43:33.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.759 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:43:33.759 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:43:33.759 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:33.759 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:33.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.760 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:33.760 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:43:33.760 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:33.760 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.760 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.760 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:43:33.760 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:43:33.760 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:33.760 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:33.760 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:33.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.761 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:43:33.761 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:43:33.761 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:33.761 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:33.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.761 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:33.761 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:43:33.761 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:33.761 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.761 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.762 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:43:33.762 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:43:33.762 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:33.762 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:33.762 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:33.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.762 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:43:33.762 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:43:33.762 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:33.762 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:33.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.762 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:33.762 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:43:33.763 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:33.763 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.763 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.763 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:43:33.763 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:43:33.763 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:33.763 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:33.763 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:33.763 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.763 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.763 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:43:33.764 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:43:33.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.764 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:33.764 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:33.764 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:43:33.764 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.764 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:33.764 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:43:33.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.764 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:33.765 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:33.765 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:43:33.765 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.765 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:33.765 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:43:33.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.765 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:33.765 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:33.765 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:43:33.765 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.765 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:33.765 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:43:33.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:33.765 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:33.766 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:33.766 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:43:33.766 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:33.766 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:33.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:33.766 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:43:33.766 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:43:33.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2580f6640 1 -- 172.21.15.26:0/1492446421 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff234036440 legacy=0x7ff234038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2580f6640 10 monclient: shutdown 2024-09-16T14:43:33.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2580f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:33.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2580f6640 1 -- 172.21.15.26:0/1492446421 >> v1:172.21.15.26:6789/0 conn(0x7ff2500a1d10 legacy=0x7ff2501307d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff256693640 1 -- 172.21.15.26:0/1492446421 reap_dead start 2024-09-16T14:43:33.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff256693640 1 -- 172.21.15.26:0/1492446421 reap_dead start 2024-09-16T14:43:33.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2580f6640 1 -- 172.21.15.26:0/1492446421 shutdown_connections 2024-09-16T14:43:33.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2580f6640 1 -- 172.21.15.26:0/1492446421 >> 172.21.15.26:0/1492446421 conn(0x7ff2500a7f30 msgr2=0x7ff25006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:33.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2580f6640 1 -- 172.21.15.26:0/1492446421 shutdown_connections 2024-09-16T14:43:33.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.735+0000 7ff2580f6640 1 -- 172.21.15.26:0/1492446421 wait complete. 2024-09-16T14:43:33.768 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:43:33.769 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:43:33.769 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:43:33.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.879+0000 7fc1a7d2a640 10 monclient: get_monmap_and_config 2024-09-16T14:43:33.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.879+0000 7fc1a7d2a640 10 monclient: build_initial_monmap 2024-09-16T14:43:33.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.879+0000 7fc1a7d2a640 10 monclient: monmap: 2024-09-16T14:43:33.883 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:33.884 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.884 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:43:33.884257+0000 2024-09-16T14:43:33.884 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:43:33.884257+0000 2024-09-16T14:43:33.884 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:33.884 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.884 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:33.884 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:33.884 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:33.884 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.879+0000 7fc1a7d2a640 1 Processor -- start 2024-09-16T14:43:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.879+0000 7fc1a7d2a640 1 -- start start 2024-09-16T14:43:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.879+0000 7fc1a7d2a640 10 monclient: init 2024-09-16T14:43:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:43:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient(hunting): picked mon.noname-b con 0x7fc1a00a1ca0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient(hunting): picked mon.noname-a con 0x7fc1a00a2790 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient(hunting): picked mon.noname-c con 0x7fc1a009ff50 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a009f520 con 0x7fc1a00a2790 2024-09-16T14:43:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a00a0360 con 0x7fc1a009ff50 2024-09-16T14:43:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a009e7a0 con 0x7fc1a00a1ca0 2024-09-16T14:43:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient(hunting): authenticate will time out at 1038.004328s 2024-09-16T14:43:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a5ac6640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc1a00a1ca0 0x7fc1a00a2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40312/0 (socket says 172.21.15.26:40312) 2024-09-16T14:43:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a5ac6640 1 -- 172.21.15.26:0/97713790 learned_addr learned my addr 172.21.15.26:0/97713790 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 1 -- 172.21.15.26:0/97713790 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137290954 0 0) 0x7fc1a009e7a0 con 0x7fc1a00a1ca0 2024-09-16T14:43:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient(hunting): my global_id is 4466 2024-09-16T14:43:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 1 -- 172.21.15.26:0/97713790 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc18c003530 con 0x7fc1a00a1ca0 2024-09-16T14:43:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 1 -- 172.21.15.26:0/97713790 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602994167 0 0) 0x7fc1a009f520 con 0x7fc1a00a2790 2024-09-16T14:43:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient(hunting): my global_id is 74130 2024-09-16T14:43:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 1 -- 172.21.15.26:0/97713790 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a009e7a0 con 0x7fc1a00a2790 2024-09-16T14:43:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 1 -- 172.21.15.26:0/97713790 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1241780313 0 0) 0x7fc18c003530 con 0x7fc1a00a1ca0 2024-09-16T14:43:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 1 -- 172.21.15.26:0/97713790 >> v1:172.21.15.189:6789/0 conn(0x7fc1a009ff50 legacy=0x7fc1a013d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 1 -- 172.21.15.26:0/97713790 >> v1:172.21.15.26:6789/0 conn(0x7fc1a00a2790 legacy=0x7fc1a0139e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient: _finish_hunting 0 2024-09-16T14:43:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 1 monclient: found mon.noname-b 2024-09-16T14:43:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:43:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 1 -- 172.21.15.26:0/97713790 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a009ec80 con 0x7fc1a00a1ca0 2024-09-16T14:43:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient: _finish_auth 0 2024-09-16T14:43:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient: _check_auth_tickets 2024-09-16T14:43:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 1 -- 172.21.15.26:0/97713790 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc190001620 con 0x7fc1a00a1ca0 2024-09-16T14:43:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient: dump: 2024-09-16T14:43:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.895 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 5 monclient: authenticate success, global_id 4466 2024-09-16T14:43:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 1 -- 172.21.15.26:0/97713790 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc190001840 con 0x7fc1a00a1ca0 2024-09-16T14:43:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 1 -- 172.21.15.26:0/97713790 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc190001c10 con 0x7fc1a00a1ca0 2024-09-16T14:43:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a4ac4640 10 monclient: dump: 2024-09-16T14:43:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.896 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 4 set_mon_vals no callback set 2024-09-16T14:43:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient: shutdown 2024-09-16T14:43:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 1 -- 172.21.15.26:0/97713790 >> v1:172.21.15.26:6790/0 conn(0x7fc1a00a1ca0 legacy=0x7fc1a00a2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 1 -- 172.21.15.26:0/97713790 shutdown_connections 2024-09-16T14:43:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 1 -- 172.21.15.26:0/97713790 >> 172.21.15.26:0/97713790 conn(0x7fc1a00a3da0 msgr2=0x7fc1a00a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 1 -- 172.21.15.26:0/97713790 shutdown_connections 2024-09-16T14:43:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 1 -- 172.21.15.26:0/97713790 wait complete. 2024-09-16T14:43:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient: build_initial_monmap 2024-09-16T14:43:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient: monmap: 2024-09-16T14:43:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:43:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:43:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:43:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:33.897 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 1 Processor -- start 2024-09-16T14:43:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 1 -- start start 2024-09-16T14:43:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient: init 2024-09-16T14:43:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:43:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient(hunting): picked mon.noname-c con 0x7fc1a009ff50 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient(hunting): picked mon.noname-a con 0x7fc1a00a1ca0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient(hunting): picked mon.noname-b con 0x7fc1a00a2790 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a006bf50 con 0x7fc1a00a1ca0 2024-09-16T14:43:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a009e490 con 0x7fc1a00a2790 2024-09-16T14:43:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a0137290 con 0x7fc1a009ff50 2024-09-16T14:43:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a7d2a640 10 monclient(hunting): authenticate will time out at 1038.007609s 2024-09-16T14:43:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a52c5640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc1a00a1ca0 0x7fc1a013e0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51946/0 (socket says 172.21.15.26:51946) 2024-09-16T14:43:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1a52c5640 1 -- 172.21.15.26:0/2094655372 learned_addr learned my addr 172.21.15.26:0/2094655372 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324710488 0 0) 0x7fc1a0137290 con 0x7fc1a009ff50 2024-09-16T14:43:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1967fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1967fc640 10 monclient(hunting): my global_id is 4469 2024-09-16T14:43:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.883+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc184003530 con 0x7fc1a009ff50 2024-09-16T14:43:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696056975 0 0) 0x7fc19c001240 con 0x7fc1a00a1ca0 2024-09-16T14:43:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient(hunting): my global_id is 74133 2024-09-16T14:43:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a0137290 con 0x7fc1a00a1ca0 2024-09-16T14:43:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4084107307 0 0) 0x7fc184003530 con 0x7fc1a009ff50 2024-09-16T14:43:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc19c001240 con 0x7fc1a009ff50 2024-09-16T14:43:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc190001440 con 0x7fc1a009ff50 2024-09-16T14:43:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient(hunting): dump: 2024-09-16T14:43:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.901 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2640611567 0 0) 0x7fc1a0137290 con 0x7fc1a00a1ca0 2024-09-16T14:43:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc184003530 con 0x7fc1a00a1ca0 2024-09-16T14:43:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc19c0018e0 con 0x7fc1a00a1ca0 2024-09-16T14:43:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient(hunting): dump: 2024-09-16T14:43:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.902 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161183656 0 0) 0x7fc1a009e490 con 0x7fc1a00a2790 2024-09-16T14:43:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient(hunting): my global_id is 4501 2024-09-16T14:43:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a0137290 con 0x7fc1a00a2790 2024-09-16T14:43:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3556190145 0 0) 0x7fc19c001240 con 0x7fc1a009ff50 2024-09-16T14:43:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 >> v1:172.21.15.189:6789/0 conn(0x7fc1a00a2790 legacy=0x7fc1a01d1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 >> v1:172.21.15.26:6789/0 conn(0x7fc1a00a1ca0 legacy=0x7fc1a013e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient: _finish_hunting 0 2024-09-16T14:43:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 monclient: found mon.c 2024-09-16T14:43:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a01363a0 con 0x7fc1a009ff50 2024-09-16T14:43:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient: _finish_auth 0 2024-09-16T14:43:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient: _check_auth_tickets 2024-09-16T14:43:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1a7d2a640 5 monclient: authenticate success, global_id 4469 2024-09-16T14:43:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1a7d2a640 10 monclient: _renew_subs 2024-09-16T14:43:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1a7d2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1a7d2a640 1 -- 172.21.15.26:0/2094655372 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a0135e40 con 0x7fc1a009ff50 2024-09-16T14:43:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1a7d2a640 10 monclient: _renew_subs 2024-09-16T14:43:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1a7d2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1a7d2a640 1 -- 172.21.15.26:0/2094655372 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a013e990 con 0x7fc1a009ff50 2024-09-16T14:43:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc190004c00 con 0x7fc1a009ff50 2024-09-16T14:43:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc190004f10 con 0x7fc1a009ff50 2024-09-16T14:43:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc197fff640 4 set_mon_vals no callback set 2024-09-16T14:43:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 10 monclient: dump: 2024-09-16T14:43:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:33.905 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1a7d2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc19000f980 con 0x7fc1a009ff50 2024-09-16T14:43:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1a7d2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1a7d2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1a7d2a640 1 -- 172.21.15.26:0/2094655372 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc188005050 con 0x7fc1a009ff50 2024-09-16T14:43:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3857+0+0 (unknown 1040354461 0 0) 0x7fc1900433c0 con 0x7fc1a009ff50 2024-09-16T14:43:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.887+0000 7fc1a52c5640 10 monclient: get_auth_request con 0x7fc184036440 auth_method 0 2024-09-16T14:43:33.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.891+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc1900146d0 con 0x7fc1a009ff50 2024-09-16T14:43:33.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.891+0000 7fc1967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:33.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:33.891+0000 7fc1967fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:34.024+0000 7f6a9effd640 1 -- 172.21.15.189:0/2507326579 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}]=0 set pool 3 pg_num_min to 2 v18) v1 ==== 143+0+0 (unknown 706762250 0 0) 0x7f6aa4013010 con 0x7f6aac0a19b0 2024-09-16T14:43:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:34.024+0000 7f6a9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T14:43:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:34.024+0000 7f6a9effd640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_min to 2 2024-09-16T14:43:34.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:34.115+0000 7fc1a7d2a640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:43:34.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:34.115+0000 7fc1a7d2a640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:43:34.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:34.115+0000 7fc1a7d2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:34.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:34.115+0000 7fc1a7d2a640 1 -- 172.21.15.26:0/2094655372 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fc188005b10 con 0x7fc1a009ff50 2024-09-16T14:43:34.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:34.119+0000 7fc1967fc640 1 -- 172.21.15.26:0/2094655372 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fc190012d70 con 0x7fc1a009ff50 2024-09-16T14:43:34.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:34.119+0000 7fc1967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:43:34.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:34.119+0000 7fc1967fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:43:34.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.119+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:43:34.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.119+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=72110615,config=692619927,health=1095409094,kv=2084167376,logm=501526150}) 2024-09-16T14:43:34.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.119+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=11,logm=63} crc {auth=3027185974,config=692619927,health=1095409094,kv=1867599010,logm=1560285344}) 2024-09-16T14:43:34.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.119+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:43:34.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.119+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=72110615,config=692619927,health=1095409094,kv=2084167376,logm=501526150}) 2024-09-16T14:43:34.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.119+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=11,logm=61} crc {auth=3405927813,config=692619927,health=861809967,kv=1867599010,logm=351773904}) 2024-09-16T14:43:34.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:34.119+0000 7fc1a7d2a640 1 -- 172.21.15.26:0/2094655372 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc184036440 legacy=0x7fc184038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:34.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:34.119+0000 7fc1a7d2a640 10 monclient: shutdown 2024-09-16T14:43:34.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:34.119+0000 7fc1a7d2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:34.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:34.119+0000 7fc1a7d2a640 1 -- 172.21.15.26:0/2094655372 >> v1:172.21.15.26:6790/0 conn(0x7fc1a009ff50 legacy=0x7fc1a01c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:34.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.119+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:43:34.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:34.119+0000 7fc1a7d2a640 1 -- 172.21.15.26:0/2094655372 shutdown_connections 2024-09-16T14:43:34.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:34.119+0000 7fc1a7d2a640 1 -- 172.21.15.26:0/2094655372 >> 172.21.15.26:0/2094655372 conn(0x7fc1a00a3da0 msgr2=0x7fc1a009aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:34.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.119+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=746909997}) 2024-09-16T14:43:34.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.119+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843289425}) 2024-09-16T14:43:34.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.119+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:43:34.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.119+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=746909997}) 2024-09-16T14:43:34.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:34.119+0000 7fc1a7d2a640 1 -- 172.21.15.26:0/2094655372 shutdown_connections 2024-09-16T14:43:34.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.119+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2866171800}) 2024-09-16T14:43:34.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:34.119+0000 7fc1a7d2a640 1 -- 172.21.15.26:0/2094655372 wait complete. 2024-09-16T14:43:34.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.123+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:43:34.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.123+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3917723020}) 2024-09-16T14:43:34.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.123+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2469211954}) 2024-09-16T14:43:34.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.123+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:43:34.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.123+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3917723020}) 2024-09-16T14:43:34.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.123+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2197792332}) 2024-09-16T14:43:34.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:34.124+0000 7f6ab476d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T14:43:34.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:34.124+0000 7f6ab476d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T14:43:34.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:34.124+0000 7f6ab476d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:34.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:34.124+0000 7f6ab476d640 1 -- 172.21.15.189:0/2507326579 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"} v 0) v1 -- 0x7f6a94005340 con 0x7f6aac0a19b0 2024-09-16T14:43:34.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.123+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:43:34.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.123+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=47,mdsmap=2,mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=14} crc {logm=1242619083,mdsmap=3585761405,mgr=2509725486,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3829536687,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=2288774773,osd_snap=3794734670,osdmap=3470801164}) 2024-09-16T14:43:34.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.123+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=11} crc {logm=1253423374,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=1836305526,mon_config_key=1518680965,monmap=970648238,osd_metadata=2172984177,osd_pg_creating=2288774773,osd_snap=3794734670,osdmap=1573024951}) 2024-09-16T14:43:34.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.123+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:43:34.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.123+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=47,mdsmap=2,mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=14} crc {logm=1242619083,mdsmap=3585761405,mgr=2509725486,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3829536687,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=2288774773,osd_snap=3794734670,osdmap=3470801164}) 2024-09-16T14:43:34.129 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.123+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=48,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=7} crc {logm=4218890177,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3176896166,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=2288774773,osd_snap=3794734670,osdmap=1229949034}) 2024-09-16T14:43:34.129 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.127+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:43:34.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.127+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=25} crc {osdmap=3803770034}) 2024-09-16T14:43:34.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.127+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=24} crc {osdmap=930032791}) 2024-09-16T14:43:34.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.127+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:43:34.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.127+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=25} crc {osdmap=3803770034}) 2024-09-16T14:43:34.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:43:34.127+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=22} crc {osdmap=4125049385}) 2024-09-16T14:43:34.139 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:43:35.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.035+0000 7f6a9effd640 1 -- 172.21.15.189:0/2507326579 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}]=0 set pool 3 pg_num_min to 2 v19) v1 ==== 143+0+0 (unknown 336782373 0 0) 0x7f6aa4014790 con 0x7f6aac0a19b0 2024-09-16T14:43:35.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.035+0000 7f6a9effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T14:43:35.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.035+0000 7f6a9effd640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_min to 2 2024-09-16T14:43:35.039 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 pg_num_min to 2 2024-09-16T14:43:35.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.039+0000 7f6ab476d640 1 -- 172.21.15.189:0/2507326579 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6a90036080 legacy=0x7f6a90038540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.039+0000 7f6ab476d640 10 monclient: shutdown 2024-09-16T14:43:35.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.039+0000 7f6ab476d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:35.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.039+0000 7f6ab476d640 1 -- 172.21.15.189:0/2507326579 >> v1:172.21.15.189:6789/0 conn(0x7f6aac0a19b0 legacy=0x7f6aac133a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.039+0000 7f6ab476d640 1 -- 172.21.15.189:0/2507326579 shutdown_connections 2024-09-16T14:43:35.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.039+0000 7f6ab476d640 1 -- 172.21.15.189:0/2507326579 >> 172.21.15.189:0/2507326579 conn(0x7f6aac0a3ab0 msgr2=0x7f6aac0a4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:35.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.039+0000 7f6ab476d640 1 -- 172.21.15.189:0/2507326579 shutdown_connections 2024-09-16T14:43:35.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.039+0000 7f6ab476d640 1 -- 172.21.15.189:0/2507326579 wait complete. 2024-09-16T14:43:35.066 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 2 2024-09-16T14:43:35.066 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool get testpool1234 pg_num_min 2024-09-16T14:43:35.140 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:43:35.140 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:43:35.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.183+0000 7f76dbfb9640 10 monclient: get_monmap_and_config 2024-09-16T14:43:35.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.183+0000 7f76dbfb9640 10 monclient: build_initial_monmap 2024-09-16T14:43:35.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.183+0000 7f76dbfb9640 10 monclient: monmap: 2024-09-16T14:43:35.188 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:35.188 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.188 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:35.189181+0000 2024-09-16T14:43:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:35.189181+0000 2024-09-16T14:43:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:35.189 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:35.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 1 Processor -- start 2024-09-16T14:43:35.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 1 -- start start 2024-09-16T14:43:35.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 10 monclient: init 2024-09-16T14:43:35.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:43:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 10 monclient(hunting): picked mon.noname-c con 0x7f76d4099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 10 monclient(hunting): picked mon.noname-a con 0x7f76d41334d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 10 monclient(hunting): picked mon.noname-b con 0x7f76d4136fe0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d409b350 con 0x7f76d41334d0 2024-09-16T14:43:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d409f2e0 con 0x7f76d4099a70 2024-09-16T14:43:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d409b070 con 0x7f76d4136fe0 2024-09-16T14:43:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 10 monclient(hunting): authenticate will time out at 997.305108s 2024-09-16T14:43:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d9d55640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f76d4099a70 0x7f76d409bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43764/0 (socket says 172.21.15.189:43764) 2024-09-16T14:43:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d9d55640 1 -- 172.21.15.189:0/495815013 learned_addr learned my addr 172.21.15.189:0/495815013 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 1 -- 172.21.15.189:0/495815013 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118845515 0 0) 0x7f76d409f2e0 con 0x7f76d4099a70 2024-09-16T14:43:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 10 monclient(hunting): my global_id is 4504 2024-09-16T14:43:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 1 -- 172.21.15.189:0/495815013 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c0003530 con 0x7f76d4099a70 2024-09-16T14:43:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 1 -- 172.21.15.189:0/495815013 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 920856179 0 0) 0x7f76c0003530 con 0x7f76d4099a70 2024-09-16T14:43:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 1 -- 172.21.15.189:0/495815013 >> v1:172.21.15.26:6790/0 conn(0x7f76d4136fe0 legacy=0x7f76d41393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 1 -- 172.21.15.189:0/495815013 >> v1:172.21.15.26:6789/0 conn(0x7f76d41334d0 legacy=0x7f76d41358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 10 monclient: _finish_hunting 0 2024-09-16T14:43:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 1 monclient: found mon.noname-c 2024-09-16T14:43:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 1 -- 172.21.15.189:0/495815013 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d409b630 con 0x7f76d4099a70 2024-09-16T14:43:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 10 monclient: _finish_auth 0 2024-09-16T14:43:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 10 monclient: _check_auth_tickets 2024-09-16T14:43:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 1 -- 172.21.15.189:0/495815013 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f76d0001620 con 0x7f76d4099a70 2024-09-16T14:43:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 5 monclient: authenticate success, global_id 4504 2024-09-16T14:43:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 10 monclient: dump: 2024-09-16T14:43:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.199 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 1 -- 172.21.15.189:0/495815013 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76d00018d0 con 0x7f76d4099a70 2024-09-16T14:43:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 4 set_mon_vals no callback set 2024-09-16T14:43:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 1 -- 172.21.15.189:0/495815013 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f76d0001c20 con 0x7f76d4099a70 2024-09-16T14:43:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76d8d53640 10 monclient: dump: 2024-09-16T14:43:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.201 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 10 monclient: shutdown 2024-09-16T14:43:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 1 -- 172.21.15.189:0/495815013 >> v1:172.21.15.189:6789/0 conn(0x7f76d4099a70 legacy=0x7f76d409bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 1 -- 172.21.15.189:0/495815013 shutdown_connections 2024-09-16T14:43:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 1 -- 172.21.15.189:0/495815013 >> 172.21.15.189:0/495815013 conn(0x7f76d40a3320 msgr2=0x7f76d409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 1 -- 172.21.15.189:0/495815013 shutdown_connections 2024-09-16T14:43:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 1 -- 172.21.15.189:0/495815013 wait complete. 2024-09-16T14:43:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 10 monclient: build_initial_monmap 2024-09-16T14:43:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 10 monclient: monmap: 2024-09-16T14:43:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:35.202 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 1 Processor -- start 2024-09-16T14:43:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 1 -- start start 2024-09-16T14:43:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.187+0000 7f76dbfb9640 10 monclient: init 2024-09-16T14:43:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:43:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 10 monclient(hunting): picked mon.noname-b con 0x7f76d4099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 10 monclient(hunting): picked mon.noname-a con 0x7f76d41334d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 10 monclient(hunting): picked mon.noname-c con 0x7f76d4136fe0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d41bc5a0 con 0x7f76d41334d0 2024-09-16T14:43:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d4132d70 con 0x7f76d4099a70 2024-09-16T14:43:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d411fdc0 con 0x7f76d4136fe0 2024-09-16T14:43:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 10 monclient(hunting): authenticate will time out at 997.308208s 2024-09-16T14:43:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76d9d55640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f76d4099a70 0x7f76d4132660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43776/0 (socket says 172.21.15.189:43776) 2024-09-16T14:43:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76d9d55640 1 -- 172.21.15.189:0/892958912 learned_addr learned my addr 172.21.15.189:0/892958912 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30782150 0 0) 0x7f76d4132d70 con 0x7f76d4099a70 2024-09-16T14:43:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient(hunting): my global_id is 4507 2024-09-16T14:43:35.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b8003530 con 0x7f76d4099a70 2024-09-16T14:43:35.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2741027342 0 0) 0x7f76b8003530 con 0x7f76d4099a70 2024-09-16T14:43:35.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76d4132d70 con 0x7f76d4099a70 2024-09-16T14:43:35.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f76d00016a0 con 0x7f76d4099a70 2024-09-16T14:43:35.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:35.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient(hunting): dump: 2024-09-16T14:43:35.204 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:35.204 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.204 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.204 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.204 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.205 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:35.205 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.205 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.205 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.205 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:35.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:35.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564980716 0 0) 0x7f76d41bc5a0 con 0x7f76d41334d0 2024-09-16T14:43:35.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient(hunting): my global_id is 74139 2024-09-16T14:43:35.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b8003530 con 0x7f76d41334d0 2024-09-16T14:43:35.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37290711 0 0) 0x7f76d411fdc0 con 0x7f76d4136fe0 2024-09-16T14:43:35.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient(hunting): my global_id is 4475 2024-09-16T14:43:35.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76d41bc5a0 con 0x7f76d4136fe0 2024-09-16T14:43:35.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2965764925 0 0) 0x7f76d4132d70 con 0x7f76d4099a70 2024-09-16T14:43:35.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 >> v1:172.21.15.26:6790/0 conn(0x7f76d4136fe0 legacy=0x7f76d41cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 >> v1:172.21.15.26:6789/0 conn(0x7f76d41334d0 legacy=0x7f76d41cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:43:35.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 monclient: found mon.b 2024-09-16T14:43:35.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:35.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:35.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d40a2850 con 0x7f76d4099a70 2024-09-16T14:43:35.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient: _finish_auth 0 2024-09-16T14:43:35.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:43:35.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:35.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 5 monclient: authenticate success, global_id 4507 2024-09-16T14:43:35.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 10 monclient: _renew_subs 2024-09-16T14:43:35.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:35.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 1 -- 172.21.15.189:0/892958912 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d406b380 con 0x7f76d4099a70 2024-09-16T14:43:35.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 10 monclient: _renew_subs 2024-09-16T14:43:35.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:35.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 1 -- 172.21.15.189:0/892958912 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d411f800 con 0x7f76d4099a70 2024-09-16T14:43:35.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76d0004be0 con 0x7f76d4099a70 2024-09-16T14:43:35.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:35.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f76d0001220 con 0x7f76d4099a70 2024-09-16T14:43:35.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:35.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 10 monclient: dump: 2024-09-16T14:43:35.208 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:35.208 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.208 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.208 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.208 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.208 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:35.208 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.208 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.208 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.209 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:35.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76cbfff640 4 set_mon_vals no callback set 2024-09-16T14:43:35.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f76d0004ff0 con 0x7f76d4099a70 2024-09-16T14:43:35.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:35.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(19..19 src has 1..19) v4 ==== 3873+0+0 (unknown 73816822 0 0) 0x7f76d0043660 con 0x7f76d4099a70 2024-09-16T14:43:35.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:35.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:35.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76dbfb9640 1 -- 172.21.15.189:0/892958912 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76bc005050 con 0x7f76d4099a70 2024-09-16T14:43:35.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.191+0000 7f76d9554640 10 monclient: get_auth_request con 0x7f76b80362d0 auth_method 0 2024-09-16T14:43:35.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.195+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f76d0006660 con 0x7f76d4099a70 2024-09-16T14:43:35.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.195+0000 7f76ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:35.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.195+0000 7f76ca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:35.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.267+0000 7f0db4d30640 10 monclient: get_monmap_and_config 2024-09-16T14:43:35.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.267+0000 7f0db4d30640 10 monclient: build_initial_monmap 2024-09-16T14:43:35.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.267+0000 7f0db4d30640 10 monclient: monmap: 2024-09-16T14:43:35.271 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:35.271 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.271 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:43:35.271995+0000 2024-09-16T14:43:35.271 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:43:35.271995+0000 2024-09-16T14:43:35.272 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:35.272 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.272 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:35.272 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:35.272 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:35.272 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.267+0000 7f0db4d30640 1 Processor -- start 2024-09-16T14:43:35.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.267+0000 7f0db4d30640 1 -- start start 2024-09-16T14:43:35.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.267+0000 7f0db4d30640 10 monclient: init 2024-09-16T14:43:35.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:35.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:43:35.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 10 monclient(hunting): picked mon.noname-b con 0x7f0db009e7a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:35.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 10 monclient(hunting): picked mon.noname-a con 0x7f0db009cdc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:35.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 10 monclient(hunting): picked mon.noname-c con 0x7f0db009d210 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:35.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db009aff0 con 0x7f0db009cdc0 2024-09-16T14:43:35.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db009b2d0 con 0x7f0db009d210 2024-09-16T14:43:35.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db009b5b0 con 0x7f0db009e7a0 2024-09-16T14:43:35.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:35.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 10 monclient(hunting): authenticate will time out at 1039.392752s 2024-09-16T14:43:35.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dae59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0db009e7a0 0x7f0db009da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40328/0 (socket says 172.21.15.26:40328) 2024-09-16T14:43:35.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dae59c640 1 -- 172.21.15.26:0/2002142419 learned_addr learned my addr 172.21.15.26:0/2002142419 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:35.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 1 -- 172.21.15.26:0/2002142419 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947684371 0 0) 0x7f0db009aff0 con 0x7f0db009cdc0 2024-09-16T14:43:35.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient(hunting): my global_id is 74142 2024-09-16T14:43:35.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 1 -- 172.21.15.26:0/2002142419 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d9c003530 con 0x7f0db009cdc0 2024-09-16T14:43:35.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 1 -- 172.21.15.26:0/2002142419 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980961595 0 0) 0x7f0db009b5b0 con 0x7f0db009e7a0 2024-09-16T14:43:35.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient(hunting): my global_id is 4478 2024-09-16T14:43:35.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 1 -- 172.21.15.26:0/2002142419 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db009aff0 con 0x7f0db009e7a0 2024-09-16T14:43:35.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 1 -- 172.21.15.26:0/2002142419 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3780836908 0 0) 0x7f0d9c003530 con 0x7f0db009cdc0 2024-09-16T14:43:35.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 1 -- 172.21.15.26:0/2002142419 >> v1:172.21.15.26:6790/0 conn(0x7f0db009e7a0 legacy=0x7f0db009da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 1 -- 172.21.15.26:0/2002142419 >> v1:172.21.15.189:6789/0 conn(0x7f0db009d210 legacy=0x7f0db013a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient: _finish_hunting 0 2024-09-16T14:43:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 1 monclient: found mon.noname-a 2024-09-16T14:43:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:43:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 1 -- 172.21.15.26:0/2002142419 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db009b890 con 0x7f0db009cdc0 2024-09-16T14:43:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient: _finish_auth 0 2024-09-16T14:43:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient: _check_auth_tickets 2024-09-16T14:43:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 1 -- 172.21.15.26:0/2002142419 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0d98001620 con 0x7f0db009cdc0 2024-09-16T14:43:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:43:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient: dump: 2024-09-16T14:43:35.283 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.283 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.283 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.283 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.283 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.283 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.283 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.283 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.283 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.283 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 5 monclient: authenticate success, global_id 74142 2024-09-16T14:43:35.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:35.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 1 -- 172.21.15.26:0/2002142419 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d98001840 con 0x7f0db009cdc0 2024-09-16T14:43:35.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:35.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 1 -- 172.21.15.26:0/2002142419 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0d98001c10 con 0x7f0db009cdc0 2024-09-16T14:43:35.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:35.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 4 set_mon_vals no callback set 2024-09-16T14:43:35.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:35.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0dadd9b640 10 monclient: dump: 2024-09-16T14:43:35.284 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.284 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.284 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.284 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.284 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.285 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.285 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.285 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.285 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.285 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 10 monclient: shutdown 2024-09-16T14:43:35.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:35.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 1 -- 172.21.15.26:0/2002142419 >> v1:172.21.15.26:6789/0 conn(0x7f0db009cdc0 legacy=0x7f0db01370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 1 -- 172.21.15.26:0/2002142419 shutdown_connections 2024-09-16T14:43:35.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 1 -- 172.21.15.26:0/2002142419 >> 172.21.15.26:0/2002142419 conn(0x7f0db009fb50 msgr2=0x7f0db009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:35.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 1 -- 172.21.15.26:0/2002142419 shutdown_connections 2024-09-16T14:43:35.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 1 -- 172.21.15.26:0/2002142419 wait complete. 2024-09-16T14:43:35.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 10 monclient: build_initial_monmap 2024-09-16T14:43:35.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 10 monclient: monmap: 2024-09-16T14:43:35.286 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:35.286 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:35.286 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:43:35.286 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:43:35.286 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:35.286 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:43:35.286 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:35.286 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:35.286 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:35.286 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 1 Processor -- start 2024-09-16T14:43:35.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 1 -- start start 2024-09-16T14:43:35.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.271+0000 7f0db4d30640 10 monclient: init 2024-09-16T14:43:35.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:35.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:43:35.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 10 monclient(hunting): picked mon.noname-a con 0x7f0db009cdc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:35.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 10 monclient(hunting): picked mon.noname-b con 0x7f0db009d210 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:35.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 10 monclient(hunting): picked mon.noname-c con 0x7f0db009e7a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:35.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db011fb50 con 0x7f0db009cdc0 2024-09-16T14:43:35.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db00a1f10 con 0x7f0db009d210 2024-09-16T14:43:35.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db0121410 con 0x7f0db009e7a0 2024-09-16T14:43:35.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:35.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 10 monclient(hunting): authenticate will time out at 1039.396110s 2024-09-16T14:43:35.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0dae59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0db009cdc0 0x7f0db012eaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51960/0 (socket says 172.21.15.26:51960) 2024-09-16T14:43:35.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0dae59c640 1 -- 172.21.15.26:0/1604121163 learned_addr learned my addr 172.21.15.26:0/1604121163 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:35.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580022906 0 0) 0x7f0db011fb50 con 0x7f0db009cdc0 2024-09-16T14:43:35.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): my global_id is 74145 2024-09-16T14:43:35.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d90003530 con 0x7f0db009cdc0 2024-09-16T14:43:35.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625320801 0 0) 0x7f0db0121410 con 0x7f0db009e7a0 2024-09-16T14:43:35.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): my global_id is 4481 2024-09-16T14:43:35.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db011fb50 con 0x7f0db009e7a0 2024-09-16T14:43:35.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1103607025 0 0) 0x7f0db011fb50 con 0x7f0db009e7a0 2024-09-16T14:43:35.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0db0121410 con 0x7f0db009e7a0 2024-09-16T14:43:35.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0da8001bc0 con 0x7f0db009e7a0 2024-09-16T14:43:35.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:35.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): dump: 2024-09-16T14:43:35.289 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.289 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.289 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.289 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.289 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.289 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.290 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.290 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.290 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.290 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:35.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1556411601 0 0) 0x7f0d90003530 con 0x7f0db009cdc0 2024-09-16T14:43:35.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0db011fb50 con 0x7f0db009cdc0 2024-09-16T14:43:35.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660827104 0 0) 0x7f0db00a1f10 con 0x7f0db009d210 2024-09-16T14:43:35.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): my global_id is 4513 2024-09-16T14:43:35.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d90003530 con 0x7f0db009d210 2024-09-16T14:43:35.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0da00019e0 con 0x7f0db009cdc0 2024-09-16T14:43:35.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:35.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient(hunting): dump: 2024-09-16T14:43:35.291 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.291 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.291 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.291 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.291 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.291 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.291 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.291 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.292 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.292 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3463521543 0 0) 0x7f0db0121410 con 0x7f0db009e7a0 2024-09-16T14:43:35.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 >> v1:172.21.15.189:6789/0 conn(0x7f0db009d210 legacy=0x7f0db012f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 >> v1:172.21.15.26:6789/0 conn(0x7f0db009cdc0 legacy=0x7f0db012eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:43:35.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 monclient: found mon.c 2024-09-16T14:43:35.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:35.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:35.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db012f8c0 con 0x7f0db009e7a0 2024-09-16T14:43:35.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient: _finish_auth 0 2024-09-16T14:43:35.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:43:35.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:35.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 5 monclient: authenticate success, global_id 4481 2024-09-16T14:43:35.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 10 monclient: _renew_subs 2024-09-16T14:43:35.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:35.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 1 -- 172.21.15.26:0/1604121163 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db0130740 con 0x7f0db009e7a0 2024-09-16T14:43:35.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 10 monclient: _renew_subs 2024-09-16T14:43:35.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:35.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 1 -- 172.21.15.26:0/1604121163 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db009a240 con 0x7f0db009e7a0 2024-09-16T14:43:35.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0da80039a0 con 0x7f0db009e7a0 2024-09-16T14:43:35.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:35.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0da8003cb0 con 0x7f0db009e7a0 2024-09-16T14:43:35.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:35.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0dacd99640 4 set_mon_vals no callback set 2024-09-16T14:43:35.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 10 monclient: dump: 2024-09-16T14:43:35.294 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.294 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.294 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.294 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.294 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.294 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.294 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.294 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.294 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.294 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:35.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:35.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:35.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0db4d30640 1 -- 172.21.15.26:0/1604121163 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d94005050 con 0x7f0db009e7a0 2024-09-16T14:43:35.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0da800e720 con 0x7f0db009e7a0 2024-09-16T14:43:35.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(19..19 src has 1..19) v4 ==== 3873+0+0 (unknown 73816822 0 0) 0x7f0da8042230 con 0x7f0db009e7a0 2024-09-16T14:43:35.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.275+0000 7f0dae59c640 10 monclient: get_auth_request con 0x7f0d90036350 auth_method 0 2024-09-16T14:43:35.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.279+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0da8013600 con 0x7f0db009e7a0 2024-09-16T14:43:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.279+0000 7f0da77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.279+0000 7f0da77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.355+0000 7f76dbfb9640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T14:43:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.355+0000 7f76dbfb9640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T14:43:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.355+0000 7f76dbfb9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.355+0000 7f76dbfb9640 1 -- 172.21.15.189:0/892958912 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7f76bc005880 con 0x7f76d4099a70 2024-09-16T14:43:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.355+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=0 v19) v1 ==== 105+0+14 (unknown 2993427748 0 3052767636) 0x7f76d0013000 con 0x7f76d4099a70 2024-09-16T14:43:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.355+0000 7f76ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T14:43:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.355+0000 7f76ca7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:43:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.447+0000 7f76dbfb9640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T14:43:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.447+0000 7f76dbfb9640 10 monclient: _send_command 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T14:43:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.447+0000 7f76dbfb9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.447+0000 7f76dbfb9640 1 -- 172.21.15.189:0/892958912 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7f76bc0010a0 con 0x7f76d4099a70 2024-09-16T14:43:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.447+0000 7f76ca7fc640 1 -- 172.21.15.189:0/892958912 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=0 v19) v1 ==== 105+0+14 (unknown 2993427748 0 3052767636) 0x7f76d0014780 con 0x7f76d4099a70 2024-09-16T14:43:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.447+0000 7f76ca7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T14:43:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.447+0000 7f76ca7fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:43:35.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.451+0000 7f76dbfb9640 1 -- 172.21.15.189:0/892958912 >> v1:172.21.15.189:6812/3196161582 conn(0x7f76b80362d0 legacy=0x7f76b8038790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.451+0000 7f76dbfb9640 10 monclient: shutdown 2024-09-16T14:43:35.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.451+0000 7f76dbfb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:35.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.451+0000 7f76dbfb9640 1 -- 172.21.15.189:0/892958912 >> v1:172.21.15.189:6789/0 conn(0x7f76d4099a70 legacy=0x7f76d4132660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.451+0000 7f76dbfb9640 1 -- 172.21.15.189:0/892958912 shutdown_connections 2024-09-16T14:43:35.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.451+0000 7f76dbfb9640 1 -- 172.21.15.189:0/892958912 >> 172.21.15.189:0/892958912 conn(0x7f76d40a3320 msgr2=0x7f76d4070e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:35.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.451+0000 7f76dbfb9640 1 -- 172.21.15.189:0/892958912 shutdown_connections 2024-09-16T14:43:35.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.451+0000 7f76dbfb9640 1 -- 172.21.15.189:0/892958912 wait complete. 2024-09-16T14:43:35.469 INFO:tasks.workunit.client.0.smithi189.stdout:pg_num_min: 2 2024-09-16T14:43:35.469 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set testpool1234 pg_num_max 33 2024-09-16T14:43:35.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.491+0000 7f0db4d30640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:43:35.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.491+0000 7f0db4d30640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:43:35.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.491+0000 7f0db4d30640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:35.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.491+0000 7f0db4d30640 1 -- 172.21.15.26:0/1604121163 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0d94005880 con 0x7f0db009e7a0 2024-09-16T14:43:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.495+0000 7f0da77fe640 1 -- 172.21.15.26:0/1604121163 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1034886549) 0x7f0da8011ca0 con 0x7f0db009e7a0 2024-09-16T14:43:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.495+0000 7f0da77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:43:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.495+0000 7f0da77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:43:35.497 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:43:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.495+0000 7f0db4d30640 1 -- 172.21.15.26:0/1604121163 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0d90036350 legacy=0x7f0d90038810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.495+0000 7f0db4d30640 10 monclient: shutdown 2024-09-16T14:43:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.495+0000 7f0db4d30640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.495+0000 7f0db4d30640 1 -- 172.21.15.26:0/1604121163 >> v1:172.21.15.26:6790/0 conn(0x7f0db009e7a0 legacy=0x7f0db01d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.495+0000 7f0db4d30640 1 -- 172.21.15.26:0/1604121163 shutdown_connections 2024-09-16T14:43:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.495+0000 7f0db4d30640 1 -- 172.21.15.26:0/1604121163 >> 172.21.15.26:0/1604121163 conn(0x7f0db009fb50 msgr2=0x7f0db0131b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.495+0000 7f0db4d30640 1 -- 172.21.15.26:0/1604121163 shutdown_connections 2024-09-16T14:43:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.495+0000 7f0db4d30640 1 -- 172.21.15.26:0/1604121163 wait complete. 2024-09-16T14:43:35.514 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:43:35.514 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:43:35.514 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:43:35.515 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:43:35.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 10 monclient: get_monmap_and_config 2024-09-16T14:43:35.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 10 monclient: build_initial_monmap 2024-09-16T14:43:35.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 10 monclient: monmap: 2024-09-16T14:43:35.590 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:35.590 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.590 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:35.590965+0000 2024-09-16T14:43:35.591 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:35.590965+0000 2024-09-16T14:43:35.591 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:35.591 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:35.591 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:35.591 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:35.591 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:35.591 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:35.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 1 Processor -- start 2024-09-16T14:43:35.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 1 -- start start 2024-09-16T14:43:35.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 10 monclient: init 2024-09-16T14:43:35.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:35.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:43:35.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 10 monclient(hunting): picked mon.noname-c con 0x7f2f300a1a90 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:35.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 10 monclient(hunting): picked mon.noname-a con 0x7f2f300a2580 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:35.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 10 monclient(hunting): picked mon.noname-b con 0x7f2f3009fd40 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:35.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3009df00 con 0x7f2f300a2580 2024-09-16T14:43:35.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3009f0a0 con 0x7f2f300a1a90 2024-09-16T14:43:35.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f300a3040 con 0x7f2f3009fd40 2024-09-16T14:43:35.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:35.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 10 monclient(hunting): authenticate will time out at 997.706553s 2024-09-16T14:43:35.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f34afa640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2f300a1a90 0x7f2f300a1e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43778/0 (socket says 172.21.15.189:43778) 2024-09-16T14:43:35.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f34afa640 1 -- 172.21.15.189:0/3261010164 learned_addr learned my addr 172.21.15.189:0/3261010164 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:35.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 1 -- 172.21.15.189:0/3261010164 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299249242 0 0) 0x7f2f3009f0a0 con 0x7f2f300a1a90 2024-09-16T14:43:35.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient(hunting): my global_id is 4516 2024-09-16T14:43:35.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 1 -- 172.21.15.189:0/3261010164 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f18003530 con 0x7f2f300a1a90 2024-09-16T14:43:35.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 1 -- 172.21.15.189:0/3261010164 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544869685 0 0) 0x7f2f300a3040 con 0x7f2f3009fd40 2024-09-16T14:43:35.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient(hunting): my global_id is 4484 2024-09-16T14:43:35.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 1 -- 172.21.15.189:0/3261010164 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f3009f0a0 con 0x7f2f3009fd40 2024-09-16T14:43:35.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 1 -- 172.21.15.189:0/3261010164 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3682468 0 0) 0x7f2f18003530 con 0x7f2f300a1a90 2024-09-16T14:43:35.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 1 -- 172.21.15.189:0/3261010164 >> v1:172.21.15.26:6790/0 conn(0x7f2f3009fd40 legacy=0x7f2f3013d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 1 -- 172.21.15.189:0/3261010164 >> v1:172.21.15.26:6789/0 conn(0x7f2f300a2580 legacy=0x7f2f30139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:43:35.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 1 monclient: found mon.noname-c 2024-09-16T14:43:35.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:35.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:35.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 1 -- 172.21.15.189:0/3261010164 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3009eae0 con 0x7f2f300a1a90 2024-09-16T14:43:35.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:43:35.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:43:35.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:35.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 1 -- 172.21.15.189:0/3261010164 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2f20001620 con 0x7f2f300a1a90 2024-09-16T14:43:35.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 5 monclient: authenticate success, global_id 4516 2024-09-16T14:43:35.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:35.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:35.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient: dump: 2024-09-16T14:43:35.601 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:35.602 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.602 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.602 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.602 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.602 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:35.602 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.602 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.602 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.602 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:35.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:35.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 1 -- 172.21.15.189:0/3261010164 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f20001840 con 0x7f2f300a1a90 2024-09-16T14:43:35.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:35.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:35.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 4 set_mon_vals no callback set 2024-09-16T14:43:35.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 1 -- 172.21.15.189:0/3261010164 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2f20001b50 con 0x7f2f300a1a90 2024-09-16T14:43:35.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:35.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f2f7fe640 10 monclient: dump: 2024-09-16T14:43:35.603 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:35.603 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.603 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.603 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.603 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.603 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:35.603 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.603 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.604 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.604 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:35.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.587+0000 7f2f36d5e640 10 monclient: shutdown 2024-09-16T14:43:35.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:35.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 1 -- 172.21.15.189:0/3261010164 >> v1:172.21.15.189:6789/0 conn(0x7f2f300a1a90 legacy=0x7f2f300a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 1 -- 172.21.15.189:0/3261010164 shutdown_connections 2024-09-16T14:43:35.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 1 -- 172.21.15.189:0/3261010164 >> 172.21.15.189:0/3261010164 conn(0x7f2f300a7a70 msgr2=0x7f2f300a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:35.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 1 -- 172.21.15.189:0/3261010164 shutdown_connections 2024-09-16T14:43:35.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 1 -- 172.21.15.189:0/3261010164 wait complete. 2024-09-16T14:43:35.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 monclient: build_initial_monmap 2024-09-16T14:43:35.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 monclient: monmap: 2024-09-16T14:43:35.604 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:35.605 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:35.605 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:35.605 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:35.605 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:35.605 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:35.605 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:35.605 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:35.605 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:35.605 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:35.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 1 Processor -- start 2024-09-16T14:43:35.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 1 -- start start 2024-09-16T14:43:35.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 monclient: init 2024-09-16T14:43:35.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:35.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:43:35.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 monclient(hunting): picked mon.noname-a con 0x7f2f3009fd40 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:35.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 monclient(hunting): picked mon.noname-c con 0x7f2f300a1a90 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:35.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 monclient(hunting): picked mon.noname-b con 0x7f2f300a2580 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:35.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3009e280 con 0x7f2f3009fd40 2024-09-16T14:43:35.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f30120ac0 con 0x7f2f300a2580 2024-09-16T14:43:35.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f300a65d0 con 0x7f2f300a1a90 2024-09-16T14:43:35.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:35.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 monclient(hunting): authenticate will time out at 997.709707s 2024-09-16T14:43:35.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f352fb640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2f300a2580 0x7f2f301d1a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43792/0 (socket says 172.21.15.189:43792) 2024-09-16T14:43:35.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f352fb640 1 -- 172.21.15.189:0/2826986803 learned_addr learned my addr 172.21.15.189:0/2826986803 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:35.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267105438 0 0) 0x7f2f30120ac0 con 0x7f2f300a2580 2024-09-16T14:43:35.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): my global_id is 4519 2024-09-16T14:43:35.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f1c003530 con 0x7f2f300a2580 2024-09-16T14:43:35.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711035621 0 0) 0x7f2f300a65d0 con 0x7f2f300a1a90 2024-09-16T14:43:35.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): my global_id is 4487 2024-09-16T14:43:35.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f30120ac0 con 0x7f2f300a1a90 2024-09-16T14:43:35.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 532754158 0 0) 0x7f2f1c003530 con 0x7f2f300a2580 2024-09-16T14:43:35.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f300a65d0 con 0x7f2f300a2580 2024-09-16T14:43:35.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2f28001c60 con 0x7f2f300a2580 2024-09-16T14:43:35.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:35.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): dump: 2024-09-16T14:43:35.608 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:35.608 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.608 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.608 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.609 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.609 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:35.609 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.609 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.609 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.609 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:35.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:35.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778174410 0 0) 0x7f2f3009e280 con 0x7f2f3009fd40 2024-09-16T14:43:35.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): my global_id is 74151 2024-09-16T14:43:35.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f1c003530 con 0x7f2f3009fd40 2024-09-16T14:43:35.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3117880278 0 0) 0x7f2f30120ac0 con 0x7f2f300a1a90 2024-09-16T14:43:35.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f3009e280 con 0x7f2f300a1a90 2024-09-16T14:43:35.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2f24001880 con 0x7f2f300a1a90 2024-09-16T14:43:35.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:35.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient(hunting): dump: 2024-09-16T14:43:35.610 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:35.610 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.610 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.610 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.610 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.610 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:35.611 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.611 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.611 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.611 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:35.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1998032653 0 0) 0x7f2f300a65d0 con 0x7f2f300a2580 2024-09-16T14:43:35.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 >> v1:172.21.15.26:6790/0 conn(0x7f2f300a1a90 legacy=0x7f2f301ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 >> v1:172.21.15.26:6789/0 conn(0x7f2f3009fd40 legacy=0x7f2f30134b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:43:35.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 monclient: found mon.b 2024-09-16T14:43:35.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:35.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:35.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f30122540 con 0x7f2f300a2580 2024-09-16T14:43:35.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:43:35.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:43:35.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:35.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 5 monclient: authenticate success, global_id 4519 2024-09-16T14:43:35.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 monclient: _renew_subs 2024-09-16T14:43:35.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:35.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 1 -- 172.21.15.189:0/2826986803 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f30121ca0 con 0x7f2f300a2580 2024-09-16T14:43:35.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 monclient: _renew_subs 2024-09-16T14:43:35.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:35.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 1 -- 172.21.15.189:0/2826986803 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f30135270 con 0x7f2f300a2580 2024-09-16T14:43:35.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f28001ff0 con 0x7f2f300a2580 2024-09-16T14:43:35.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:35.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2f28003df0 con 0x7f2f300a2580 2024-09-16T14:43:35.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:35.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2effd640 4 set_mon_vals no callback set 2024-09-16T14:43:35.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 10 monclient: dump: 2024-09-16T14:43:35.613 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:35.613 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.613 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.613 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.613 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.613 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:35.613 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.614 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.614 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.614 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:35.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:35.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:35.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:35.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f36d5e640 1 -- 172.21.15.189:0/2826986803 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f10005050 con 0x7f2f300a2580 2024-09-16T14:43:35.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2f28003f50 con 0x7f2f300a2580 2024-09-16T14:43:35.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(19..19 src has 1..19) v4 ==== 3873+0+0 (unknown 73816822 0 0) 0x7f2f28043580 con 0x7f2f300a2580 2024-09-16T14:43:35.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.591+0000 7f2f34afa640 10 monclient: get_auth_request con 0x7f2f1c036460 auth_method 0 2024-09-16T14:43:35.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.599+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2f28014a10 con 0x7f2f300a2580 2024-09-16T14:43:35.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.599+0000 7f2f2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:35.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.599+0000 7f2f2d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:35.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 10 monclient: get_monmap_and_config 2024-09-16T14:43:35.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 10 monclient: build_initial_monmap 2024-09-16T14:43:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 10 monclient: monmap: 2024-09-16T14:43:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:43:35.638299+0000 2024-09-16T14:43:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:43:35.638299+0000 2024-09-16T14:43:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:35.639 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 1 Processor -- start 2024-09-16T14:43:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 1 -- start start 2024-09-16T14:43:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 10 monclient: init 2024-09-16T14:43:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:43:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 10 monclient(hunting): picked mon.noname-a con 0x7f8c5009a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 10 monclient(hunting): picked mon.noname-b con 0x7f8c5009d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 10 monclient(hunting): picked mon.noname-c con 0x7f8c5009b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5009dc60 con 0x7f8c5009a550 2024-09-16T14:43:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c50099ae0 con 0x7f8c5009b040 2024-09-16T14:43:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5009b8f0 con 0x7f8c5009d1b0 2024-09-16T14:43:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 10 monclient(hunting): authenticate will time out at 1039.758298s 2024-09-16T14:43:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c54f28640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8c5009a550 0x7f8c5009a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51974/0 (socket says 172.21.15.26:51974) 2024-09-16T14:43:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c54f28640 1 -- 172.21.15.26:0/3894441279 learned_addr learned my addr 172.21.15.26:0/3894441279 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 1 -- 172.21.15.26:0/3894441279 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431653927 0 0) 0x7f8c5009dc60 con 0x7f8c5009a550 2024-09-16T14:43:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient(hunting): my global_id is 74154 2024-09-16T14:43:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 1 -- 172.21.15.26:0/3894441279 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c38003530 con 0x7f8c5009a550 2024-09-16T14:43:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 1 -- 172.21.15.26:0/3894441279 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509857999 0 0) 0x7f8c5009b8f0 con 0x7f8c5009d1b0 2024-09-16T14:43:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient(hunting): my global_id is 4490 2024-09-16T14:43:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 1 -- 172.21.15.26:0/3894441279 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c5009dc60 con 0x7f8c5009d1b0 2024-09-16T14:43:35.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 1 -- 172.21.15.26:0/3894441279 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145363189 0 0) 0x7f8c50099ae0 con 0x7f8c5009b040 2024-09-16T14:43:35.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient(hunting): my global_id is 4522 2024-09-16T14:43:35.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 1 -- 172.21.15.26:0/3894441279 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c5009b8f0 con 0x7f8c5009b040 2024-09-16T14:43:35.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 1 -- 172.21.15.26:0/3894441279 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 666750237 0 0) 0x7f8c38003530 con 0x7f8c5009a550 2024-09-16T14:43:35.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 1 -- 172.21.15.26:0/3894441279 >> v1:172.21.15.26:6790/0 conn(0x7f8c5009d1b0 legacy=0x7f8c50136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 1 -- 172.21.15.26:0/3894441279 >> v1:172.21.15.189:6789/0 conn(0x7f8c5009b040 legacy=0x7f8c5013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:43:35.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 1 monclient: found mon.noname-a 2024-09-16T14:43:35.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:35.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:43:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 1 -- 172.21.15.26:0/3894441279 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5009bbd0 con 0x7f8c5009a550 2024-09-16T14:43:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:43:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:43:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 5 monclient: authenticate success, global_id 74154 2024-09-16T14:43:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 1 -- 172.21.15.26:0/3894441279 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8c40001620 con 0x7f8c5009a550 2024-09-16T14:43:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:43:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient: dump: 2024-09-16T14:43:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.651 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 1 -- 172.21.15.26:0/3894441279 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c40001840 con 0x7f8c5009a550 2024-09-16T14:43:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 4 set_mon_vals no callback set 2024-09-16T14:43:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 1 -- 172.21.15.26:0/3894441279 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8c40001c10 con 0x7f8c5009a550 2024-09-16T14:43:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c4f7fe640 10 monclient: dump: 2024-09-16T14:43:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.653 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 10 monclient: shutdown 2024-09-16T14:43:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 1 -- 172.21.15.26:0/3894441279 >> v1:172.21.15.26:6789/0 conn(0x7f8c5009a550 legacy=0x7f8c5009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 1 -- 172.21.15.26:0/3894441279 shutdown_connections 2024-09-16T14:43:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.635+0000 7f8c5718c640 1 -- 172.21.15.26:0/3894441279 >> 172.21.15.26:0/3894441279 conn(0x7f8c5009fb50 msgr2=0x7f8c5009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 1 -- 172.21.15.26:0/3894441279 shutdown_connections 2024-09-16T14:43:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 1 -- 172.21.15.26:0/3894441279 wait complete. 2024-09-16T14:43:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 10 monclient: build_initial_monmap 2024-09-16T14:43:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 10 monclient: monmap: 2024-09-16T14:43:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:43:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:43:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:43:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:35.654 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 1 Processor -- start 2024-09-16T14:43:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 1 -- start start 2024-09-16T14:43:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 10 monclient: init 2024-09-16T14:43:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:43:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 10 monclient(hunting): picked mon.noname-a con 0x7f8c5009a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 10 monclient(hunting): picked mon.noname-c con 0x7f8c5009b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 10 monclient(hunting): picked mon.noname-b con 0x7f8c5009d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c50139bf0 con 0x7f8c5009a550 2024-09-16T14:43:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c50139910 con 0x7f8c5009d1b0 2024-09-16T14:43:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c50137de0 con 0x7f8c5009b040 2024-09-16T14:43:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 10 monclient(hunting): authenticate will time out at 1039.761565s 2024-09-16T14:43:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8c5009b040 0x7f8c501ca6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40342/0 (socket says 172.21.15.26:40342) 2024-09-16T14:43:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c54f28640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8c5009a550 0x7f8c50131270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51982/0 (socket says 172.21.15.26:51982) 2024-09-16T14:43:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4ffff640 1 -- 172.21.15.26:0/261169070 learned_addr learned my addr 172.21.15.26:0/261169070 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231793889 0 0) 0x7f8c50139bf0 con 0x7f8c5009a550 2024-09-16T14:43:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): my global_id is 74157 2024-09-16T14:43:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c3c005740 con 0x7f8c5009a550 2024-09-16T14:43:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4132169175 0 0) 0x7f8c50137de0 con 0x7f8c5009b040 2024-09-16T14:43:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): my global_id is 4493 2024-09-16T14:43:35.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c50139bf0 con 0x7f8c5009b040 2024-09-16T14:43:35.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923796547 0 0) 0x7f8c50139910 con 0x7f8c5009d1b0 2024-09-16T14:43:35.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): my global_id is 4525 2024-09-16T14:43:35.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c50137de0 con 0x7f8c5009d1b0 2024-09-16T14:43:35.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3908546289 0 0) 0x7f8c3c005740 con 0x7f8c5009a550 2024-09-16T14:43:35.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c50139910 con 0x7f8c5009a550 2024-09-16T14:43:35.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1383076573 0 0) 0x7f8c50139bf0 con 0x7f8c5009b040 2024-09-16T14:43:35.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c3c005740 con 0x7f8c5009b040 2024-09-16T14:43:35.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8c40001b50 con 0x7f8c5009a550 2024-09-16T14:43:35.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:43:35.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): dump: 2024-09-16T14:43:35.658 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.658 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.658 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.658 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.658 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.658 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.658 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.658 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.658 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.658 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:35.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8c440019e0 con 0x7f8c5009b040 2024-09-16T14:43:35.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:35.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): dump: 2024-09-16T14:43:35.659 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.659 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.659 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.659 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.659 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.659 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.659 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.659 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.659 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.659 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4174645487 0 0) 0x7f8c3c005740 con 0x7f8c5009b040 2024-09-16T14:43:35.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 >> v1:172.21.15.189:6789/0 conn(0x7f8c5009d1b0 legacy=0x7f8c501cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 >> v1:172.21.15.26:6789/0 conn(0x7f8c5009a550 legacy=0x7f8c50131270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:43:35.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 monclient: found mon.c 2024-09-16T14:43:35.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:35.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:35.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c50137240 con 0x7f8c5009b040 2024-09-16T14:43:35.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:43:35.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:43:35.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:35.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 5 monclient: authenticate success, global_id 4493 2024-09-16T14:43:35.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 10 monclient: _renew_subs 2024-09-16T14:43:35.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:35.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 1 -- 172.21.15.26:0/261169070 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c50137490 con 0x7f8c5009b040 2024-09-16T14:43:35.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 10 monclient: _renew_subs 2024-09-16T14:43:35.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:35.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 1 -- 172.21.15.26:0/261169070 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c5009e4b0 con 0x7f8c5009b040 2024-09-16T14:43:35.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c44003600 con 0x7f8c5009b040 2024-09-16T14:43:35.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:35.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8c44003990 con 0x7f8c5009b040 2024-09-16T14:43:35.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:35.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4effd640 4 set_mon_vals no callback set 2024-09-16T14:43:35.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient: dump: 2024-09-16T14:43:35.662 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.662 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.662 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.662 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.662 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.662 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.662 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.662 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.662 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.662 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c5718c640 1 -- 172.21.15.26:0/261169070 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8c30006590 con 0x7f8c30002830 2024-09-16T14:43:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8c44003af0 con 0x7f8c5009b040 2024-09-16T14:43:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(19..19 src has 1..19) v4 ==== 3873+0+0 (unknown 73816822 0 0) 0x7f8c44043130 con 0x7f8c5009b040 2024-09-16T14:43:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281359948 0 0) 0x7f8c30006590 con 0x7f8c30002830 2024-09-16T14:43:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 10 monclient(hunting): my global_id is 74160 2024-09-16T14:43:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.639+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c3c005740 con 0x7f8c30002830 2024-09-16T14:43:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.643+0000 7f8c55729640 10 monclient: get_auth_request con 0x7f8c3c0385a0 auth_method 0 2024-09-16T14:43:35.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.643+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2105689242 0 0) 0x7f8c3c005740 con 0x7f8c30002830 2024-09-16T14:43:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.643+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c30006590 con 0x7f8c30002830 2024-09-16T14:43:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.643+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8c40004f10 con 0x7f8c30002830 2024-09-16T14:43:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.643+0000 7f8c4d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.643+0000 7f8c4d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.643+0000 7f8c4d7fa640 10 monclient: dump: 2024-09-16T14:43:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.664 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.665 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.665 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.665 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.665 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.665 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.643+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3074203726 0 0) 0x7f8c30006590 con 0x7f8c30002830 2024-09-16T14:43:35.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.643+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c3000a0e0 con 0x7f8c30002830 2024-09-16T14:43:35.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.643+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8c40005070 con 0x7f8c30002830 2024-09-16T14:43:35.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.643+0000 7f8c4d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:35.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.643+0000 7f8c4d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:35.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.643+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 >> v1:172.21.15.26:6789/0 conn(0x7f8c30002830 legacy=0x7f8c30004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.643+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 >> v1:172.21.15.26:6789/0 conn(0x7f8c30002830 legacy=0x7f8c30004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:35.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c5718c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:43:35.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c5718c640 1 -- 172.21.15.26:0/261169070 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8c30009d80 con 0x7f8c300090a0 2024-09-16T14:43:35.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647864761 0 0) 0x7f8c30009d80 con 0x7f8c300090a0 2024-09-16T14:43:35.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 10 monclient(hunting): my global_id is 74163 2024-09-16T14:43:35.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c30006590 con 0x7f8c300090a0 2024-09-16T14:43:35.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3752704693 0 0) 0x7f8c30006590 con 0x7f8c300090a0 2024-09-16T14:43:35.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c30009d80 con 0x7f8c300090a0 2024-09-16T14:43:35.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8c40005680 con 0x7f8c300090a0 2024-09-16T14:43:35.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:35.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 10 monclient: dump: 2024-09-16T14:43:35.668 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.668 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.668 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.668 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.668 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.668 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.668 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.668 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.668 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.668 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1972898627 0 0) 0x7f8c30009d80 con 0x7f8c300090a0 2024-09-16T14:43:35.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8c3000c290 con 0x7f8c300090a0 2024-09-16T14:43:35.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 444642600) 0x7f8c400058b0 con 0x7f8c300090a0 2024-09-16T14:43:35.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:43:35.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:43:35.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 >> v1:172.21.15.26:6789/0 conn(0x7f8c300090a0 legacy=0x7f8c30009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.647+0000 7f8c4d7fa640 1 -- 172.21.15.26:0/261169070 >> v1:172.21.15.26:6789/0 conn(0x7f8c300090a0 legacy=0x7f8c30009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:35.669 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:43:35.669 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:43:35.669 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:43:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:43:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 16, 2024-09-16T14:43:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:43:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:43:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:43:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:43:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:43:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:43:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:43:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:43:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:43:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:43:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:43:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:43:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:43:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:43:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:43:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:43:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:43:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:43:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:43:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:43:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:43:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:43:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:43:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:43:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:43:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:43:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:43:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:43:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:43:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:43:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:43:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:43:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:43:35.676 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:43:35.676 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:35.676 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:35.676 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:35.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.676 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:43:35.676 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:43:35.676 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:35.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:35.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:35.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:43:35.677 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:35.677 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.677 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.677 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:43:35.677 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:43:35.677 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:35.677 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:35.677 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:35.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.677 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:43:35.678 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:43:35.678 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:35.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:35.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.678 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:35.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:43:35.678 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:35.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:43:35.679 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:43:35.679 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:35.679 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:35.679 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:35.679 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.679 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.679 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:43:35.679 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:43:35.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.679 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:35.680 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:35.680 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:43:35.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.680 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:35.680 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:43:35.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.680 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:35.680 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:35.680 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:43:35.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.680 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.680 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:43:35.681 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:43:35.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.651+0000 7f8c5718c640 1 -- 172.21.15.26:0/261169070 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8c3c0385a0 legacy=0x7f8c3c03aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.651+0000 7f8c5718c640 10 monclient: shutdown 2024-09-16T14:43:35.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.651+0000 7f8c5718c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:35.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.651+0000 7f8c5718c640 1 -- 172.21.15.26:0/261169070 >> v1:172.21.15.26:6790/0 conn(0x7f8c5009b040 legacy=0x7f8c501ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.651+0000 7f8c55729640 1 -- 172.21.15.26:0/261169070 reap_dead start 2024-09-16T14:43:35.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.651+0000 7f8c55729640 1 -- 172.21.15.26:0/261169070 reap_dead start 2024-09-16T14:43:35.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.651+0000 7f8c5718c640 1 -- 172.21.15.26:0/261169070 shutdown_connections 2024-09-16T14:43:35.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.651+0000 7f8c5718c640 1 -- 172.21.15.26:0/261169070 >> 172.21.15.26:0/261169070 conn(0x7f8c5009fb50 msgr2=0x7f8c500a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:35.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.651+0000 7f8c5718c640 1 -- 172.21.15.26:0/261169070 shutdown_connections 2024-09-16T14:43:35.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.651+0000 7f8c5718c640 1 -- 172.21.15.26:0/261169070 wait complete. 2024-09-16T14:43:35.683 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:43:35.685 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:43:35.685 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:43:35.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.759+0000 7f2f36d5e640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T14:43:35.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.759+0000 7f2f36d5e640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T14:43:35.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.759+0000 7f2f36d5e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:35.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:35.759+0000 7f2f36d5e640 1 -- 172.21.15.189:0/2826986803 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"} v 0) v1 -- 0x7f2f10005880 con 0x7f2f300a2580 2024-09-16T14:43:35.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.791+0000 7f9bfb2e8640 10 monclient: get_monmap_and_config 2024-09-16T14:43:35.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.791+0000 7f9bfb2e8640 10 monclient: build_initial_monmap 2024-09-16T14:43:35.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.791+0000 7f9bfb2e8640 10 monclient: monmap: 2024-09-16T14:43:35.797 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:35.797 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.797 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:43:35.797417+0000 2024-09-16T14:43:35.797 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:43:35.797417+0000 2024-09-16T14:43:35.797 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:35.797 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.797 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:35.797 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:35.797 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:35.797 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 1 Processor -- start 2024-09-16T14:43:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 1 -- start start 2024-09-16T14:43:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 10 monclient: init 2024-09-16T14:43:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:43:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 10 monclient(hunting): picked mon.noname-c con 0x7f9bf409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 10 monclient(hunting): picked mon.noname-a con 0x7f9bf4132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 10 monclient(hunting): picked mon.noname-b con 0x7f9bf4138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf409d260 con 0x7f9bf4132650 2024-09-16T14:43:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf4132a80 con 0x7f9bf409cb80 2024-09-16T14:43:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf409cf60 con 0x7f9bf4138ff0 2024-09-16T14:43:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 10 monclient(hunting): authenticate will time out at 1039.917427s 2024-09-16T14:43:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bf8883640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9bf4132650 0x7f9bf41378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52004/0 (socket says 172.21.15.26:52004) 2024-09-16T14:43:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bf8883640 1 -- 172.21.15.26:0/695494786 learned_addr learned my addr 172.21.15.26:0/695494786 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 1 -- 172.21.15.26:0/695494786 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4026728736 0 0) 0x7f9bf409d260 con 0x7f9bf4132650 2024-09-16T14:43:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient(hunting): my global_id is 74166 2024-09-16T14:43:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 1 -- 172.21.15.26:0/695494786 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be0003530 con 0x7f9bf4132650 2024-09-16T14:43:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 1 -- 172.21.15.26:0/695494786 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177649996 0 0) 0x7f9bf409cf60 con 0x7f9bf4138ff0 2024-09-16T14:43:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient(hunting): my global_id is 4496 2024-09-16T14:43:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 1 -- 172.21.15.26:0/695494786 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf409d260 con 0x7f9bf4138ff0 2024-09-16T14:43:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 1 -- 172.21.15.26:0/695494786 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 754379067 0 0) 0x7f9be0003530 con 0x7f9bf4132650 2024-09-16T14:43:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 1 -- 172.21.15.26:0/695494786 >> v1:172.21.15.26:6790/0 conn(0x7f9bf4138ff0 legacy=0x7f9bf413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 1 -- 172.21.15.26:0/695494786 >> v1:172.21.15.189:6789/0 conn(0x7f9bf409cb80 legacy=0x7f9bf40717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient: _finish_hunting 0 2024-09-16T14:43:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 1 monclient: found mon.noname-a 2024-09-16T14:43:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:43:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 1 -- 172.21.15.26:0/695494786 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf409f2d0 con 0x7f9bf4132650 2024-09-16T14:43:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient: _finish_auth 0 2024-09-16T14:43:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 1 -- 172.21.15.26:0/695494786 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9bf0001620 con 0x7f9bf4132650 2024-09-16T14:43:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:43:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient: dump: 2024-09-16T14:43:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.811 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 5 monclient: authenticate success, global_id 74166 2024-09-16T14:43:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 1 -- 172.21.15.26:0/695494786 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bf0001850 con 0x7f9bf4132650 2024-09-16T14:43:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 1 -- 172.21.15.26:0/695494786 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9bf0001c20 con 0x7f9bf4132650 2024-09-16T14:43:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bebfff640 10 monclient: dump: 2024-09-16T14:43:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.813 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 4 set_mon_vals no callback set 2024-09-16T14:43:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 10 monclient: shutdown 2024-09-16T14:43:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 1 -- 172.21.15.26:0/695494786 >> v1:172.21.15.26:6789/0 conn(0x7f9bf4132650 legacy=0x7f9bf41378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 1 -- 172.21.15.26:0/695494786 shutdown_connections 2024-09-16T14:43:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 1 -- 172.21.15.26:0/695494786 >> 172.21.15.26:0/695494786 conn(0x7f9bf409fb50 msgr2=0x7f9bf409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 1 -- 172.21.15.26:0/695494786 shutdown_connections 2024-09-16T14:43:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 1 -- 172.21.15.26:0/695494786 wait complete. 2024-09-16T14:43:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 10 monclient: build_initial_monmap 2024-09-16T14:43:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 10 monclient: monmap: 2024-09-16T14:43:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:43:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:43:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:43:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:35.815 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.795+0000 7f9bfb2e8640 1 Processor -- start 2024-09-16T14:43:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 1 -- start start 2024-09-16T14:43:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 10 monclient: init 2024-09-16T14:43:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:43:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 10 monclient(hunting): picked mon.noname-a con 0x7f9bf409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 10 monclient(hunting): picked mon.noname-b con 0x7f9bf4132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 10 monclient(hunting): picked mon.noname-c con 0x7f9bf4138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf4070a40 con 0x7f9bf409cb80 2024-09-16T14:43:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf406ddc0 con 0x7f9bf4132650 2024-09-16T14:43:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf409b370 con 0x7f9bf4138ff0 2024-09-16T14:43:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 10 monclient(hunting): authenticate will time out at 1039.920520s 2024-09-16T14:43:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bf9084640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9bf409cb80 0x7f9bf409a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52018/0 (socket says 172.21.15.26:52018) 2024-09-16T14:43:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bf9084640 1 -- 172.21.15.26:0/2801268872 learned_addr learned my addr 172.21.15.26:0/2801268872 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392450808 0 0) 0x7f9bf409b370 con 0x7f9bf4138ff0 2024-09-16T14:43:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): my global_id is 4499 2024-09-16T14:43:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd8003530 con 0x7f9bf4138ff0 2024-09-16T14:43:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293837449 0 0) 0x7f9be40012d0 con 0x7f9bf409cb80 2024-09-16T14:43:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): my global_id is 74169 2024-09-16T14:43:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf409b370 con 0x7f9bf409cb80 2024-09-16T14:43:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2281644328 0 0) 0x7f9bd8003530 con 0x7f9bf4138ff0 2024-09-16T14:43:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9be40012d0 con 0x7f9bf4138ff0 2024-09-16T14:43:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9bec0013b0 con 0x7f9bf4138ff0 2024-09-16T14:43:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): dump: 2024-09-16T14:43:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.818 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.818 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.818 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.818 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.818 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.818 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.818 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.818 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:35.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2653002514 0 0) 0x7f9bf409b370 con 0x7f9bf409cb80 2024-09-16T14:43:35.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bd8003530 con 0x7f9bf409cb80 2024-09-16T14:43:35.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9be40019c0 con 0x7f9bf409cb80 2024-09-16T14:43:35.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:35.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): dump: 2024-09-16T14:43:35.819 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.819 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.819 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.819 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.819 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.819 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.819 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.819 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.819 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.819 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164588196 0 0) 0x7f9bf406ddc0 con 0x7f9bf4132650 2024-09-16T14:43:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): my global_id is 4531 2024-09-16T14:43:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf409b370 con 0x7f9bf4132650 2024-09-16T14:43:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3441890227 0 0) 0x7f9be40012d0 con 0x7f9bf4138ff0 2024-09-16T14:43:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 >> v1:172.21.15.189:6789/0 conn(0x7f9bf4132650 legacy=0x7f9bf409ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 >> v1:172.21.15.26:6789/0 conn(0x7f9bf409cb80 legacy=0x7f9bf409a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:43:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 monclient: found mon.c 2024-09-16T14:43:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:35.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf409bf30 con 0x7f9bf4138ff0 2024-09-16T14:43:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient: _finish_auth 0 2024-09-16T14:43:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:43:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 5 monclient: authenticate success, global_id 4499 2024-09-16T14:43:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 10 monclient: _renew_subs 2024-09-16T14:43:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 1 -- 172.21.15.26:0/2801268872 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf413c700 con 0x7f9bf4138ff0 2024-09-16T14:43:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 10 monclient: _renew_subs 2024-09-16T14:43:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 1 -- 172.21.15.26:0/2801268872 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf4071990 con 0x7f9bf4138ff0 2024-09-16T14:43:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bec001920 con 0x7f9bf4138ff0 2024-09-16T14:43:35.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9bec003b20 con 0x7f9bf4138ff0 2024-09-16T14:43:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient: dump: 2024-09-16T14:43:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.822 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.823 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9beb7fe640 4 set_mon_vals no callback set 2024-09-16T14:43:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bfb2e8640 1 -- 172.21.15.26:0/2801268872 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9bd4006590 con 0x7f9bd4002830 2024-09-16T14:43:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9bec00e590 con 0x7f9bf4138ff0 2024-09-16T14:43:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(19..19 src has 1..19) v4 ==== 3873+0+0 (unknown 73816822 0 0) 0x7f9bec042190 con 0x7f9bf4138ff0 2024-09-16T14:43:35.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412904618 0 0) 0x7f9bd4006590 con 0x7f9bd4002830 2024-09-16T14:43:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient(hunting): my global_id is 4502 2024-09-16T14:43:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be40012d0 con 0x7f9bd4002830 2024-09-16T14:43:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4140339197 0 0) 0x7f9be40012d0 con 0x7f9bd4002830 2024-09-16T14:43:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bd4006590 con 0x7f9bd4002830 2024-09-16T14:43:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9bf8883640 10 monclient: get_auth_request con 0x7f9bd8036370 auth_method 0 2024-09-16T14:43:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9be4003a70 con 0x7f9bd4002830 2024-09-16T14:43:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:35.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient: dump: 2024-09-16T14:43:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.825 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1439544322 0 0) 0x7f9bd4006590 con 0x7f9bd4002830 2024-09-16T14:43:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9bd400a0e0 con 0x7f9bd4002830 2024-09-16T14:43:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f9be40057b0 con 0x7f9bd4002830 2024-09-16T14:43:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:35.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:35.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 >> v1:172.21.15.26:6790/0 conn(0x7f9bd4002830 legacy=0x7f9bd4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.799+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 >> v1:172.21.15.26:6790/0 conn(0x7f9bd4002830 legacy=0x7f9bd4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:35.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9bfb2e8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:43:35.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9bfb2e8640 1 -- 172.21.15.26:0/2801268872 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9bd4009da0 con 0x7f9bd40090c0 2024-09-16T14:43:35.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709130970 0 0) 0x7f9bd4009da0 con 0x7f9bd40090c0 2024-09-16T14:43:35.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 10 monclient(hunting): my global_id is 4505 2024-09-16T14:43:35.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd4006590 con 0x7f9bd40090c0 2024-09-16T14:43:35.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3851882581 0 0) 0x7f9bd4006590 con 0x7f9bd40090c0 2024-09-16T14:43:35.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bd4009da0 con 0x7f9bd40090c0 2024-09-16T14:43:35.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9be4004220 con 0x7f9bd40090c0 2024-09-16T14:43:35.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:35.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 10 monclient: dump: 2024-09-16T14:43:35.828 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.828 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.828 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.829 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.829 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.829 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.829 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.829 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.829 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.829 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2872838980 0 0) 0x7f9bd4009da0 con 0x7f9bd40090c0 2024-09-16T14:43:35.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9bd400c290 con 0x7f9bd40090c0 2024-09-16T14:43:35.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 634840863) 0x7f9be4005920 con 0x7f9bd40090c0 2024-09-16T14:43:35.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:43:35.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:43:35.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 >> v1:172.21.15.26:6790/0 conn(0x7f9bd40090c0 legacy=0x7f9bd4009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9be9ffb640 1 -- 172.21.15.26:0/2801268872 >> v1:172.21.15.26:6790/0 conn(0x7f9bd40090c0 legacy=0x7f9bd4009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:35.830 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:43:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:43:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:43:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:43:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 16, 2024-09-16T14:43:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:43:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:43:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:43:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:43:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:43:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:43:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:43:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:43:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:43:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:43:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:43:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:43:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:43:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:43:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:43:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:43:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:43:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:43:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:43:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:43:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:43:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:43:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:43:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:43:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:43:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:43:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:43:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:43:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:43:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:43:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:43:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:43:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:43:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:43:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:43:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:43:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:43:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:43:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:43:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:43:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:43:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:43:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:43:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:43:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:43:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:43:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:43:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:43:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:43:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:43:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:43:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:43:35.842 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:43:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9bfb2e8640 1 -- 172.21.15.26:0/2801268872 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9bd8036370 legacy=0x7f9bd8038830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9bfb2e8640 10 monclient: shutdown 2024-09-16T14:43:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9bfb2e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9bfb2e8640 1 -- 172.21.15.26:0/2801268872 >> v1:172.21.15.26:6790/0 conn(0x7f9bf4138ff0 legacy=0x7f9bf41d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9bf9885640 1 -- 172.21.15.26:0/2801268872 reap_dead start 2024-09-16T14:43:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9bf9885640 1 -- 172.21.15.26:0/2801268872 reap_dead start 2024-09-16T14:43:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9bfb2e8640 1 -- 172.21.15.26:0/2801268872 shutdown_connections 2024-09-16T14:43:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9bfb2e8640 1 -- 172.21.15.26:0/2801268872 >> 172.21.15.26:0/2801268872 conn(0x7f9bf409fb50 msgr2=0x7f9bf413b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9bfb2e8640 1 -- 172.21.15.26:0/2801268872 shutdown_connections 2024-09-16T14:43:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.807+0000 7f9bfb2e8640 1 -- 172.21.15.26:0/2801268872 wait complete. 2024-09-16T14:43:35.845 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:43:35.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:43:35.845 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:43:35.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.951+0000 7fded6733640 10 monclient: get_monmap_and_config 2024-09-16T14:43:35.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.951+0000 7fded6733640 10 monclient: build_initial_monmap 2024-09-16T14:43:35.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.951+0000 7fded6733640 10 monclient: monmap: 2024-09-16T14:43:35.955 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:35.955 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.956 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:43:35.956053+0000 2024-09-16T14:43:35.956 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:43:35.956053+0000 2024-09-16T14:43:35.956 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:35.956 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.956 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:35.956 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:35.956 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:35.956 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.951+0000 7fded6733640 1 Processor -- start 2024-09-16T14:43:35.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.951+0000 7fded6733640 1 -- start start 2024-09-16T14:43:35.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.951+0000 7fded6733640 10 monclient: init 2024-09-16T14:43:35.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:35.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:43:35.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient(hunting): picked mon.noname-a con 0x7fded00a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:35.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient(hunting): picked mon.noname-b con 0x7fded0099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:35.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient(hunting): picked mon.noname-c con 0x7fded00a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:35.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded0098ea0 con 0x7fded00a1c00 2024-09-16T14:43:35.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded00997b0 con 0x7fded00a26f0 2024-09-16T14:43:35.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded0099460 con 0x7fded0099b60 2024-09-16T14:43:35.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:35.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient(hunting): authenticate will time out at 1040.076394s 2024-09-16T14:43:35.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdecffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fded00a1c00 0x7fded00a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52032/0 (socket says 172.21.15.26:52032) 2024-09-16T14:43:35.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdecffff640 1 -- 172.21.15.26:0/3710464318 learned_addr learned my addr 172.21.15.26:0/3710464318 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:35.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdecf7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fded0099b60 0x7fded0139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40394/0 (socket says 172.21.15.26:40394) 2024-09-16T14:43:35.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 1 -- 172.21.15.26:0/3710464318 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931877263 0 0) 0x7fded0098ea0 con 0x7fded00a1c00 2024-09-16T14:43:35.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient(hunting): my global_id is 74172 2024-09-16T14:43:35.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 1 -- 172.21.15.26:0/3710464318 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeb8003530 con 0x7fded00a1c00 2024-09-16T14:43:35.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 1 -- 172.21.15.26:0/3710464318 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586653226 0 0) 0x7fded0099460 con 0x7fded0099b60 2024-09-16T14:43:35.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient(hunting): my global_id is 4508 2024-09-16T14:43:35.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 1 -- 172.21.15.26:0/3710464318 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded0098ea0 con 0x7fded0099b60 2024-09-16T14:43:35.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 1 -- 172.21.15.26:0/3710464318 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227323811 0 0) 0x7fded00997b0 con 0x7fded00a26f0 2024-09-16T14:43:35.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient(hunting): my global_id is 4534 2024-09-16T14:43:35.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 1 -- 172.21.15.26:0/3710464318 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded0099460 con 0x7fded00a26f0 2024-09-16T14:43:35.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 1 -- 172.21.15.26:0/3710464318 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3637898188 0 0) 0x7fdeb8003530 con 0x7fded00a1c00 2024-09-16T14:43:35.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 1 -- 172.21.15.26:0/3710464318 >> v1:172.21.15.26:6790/0 conn(0x7fded0099b60 legacy=0x7fded0139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 1 -- 172.21.15.26:0/3710464318 >> v1:172.21.15.189:6789/0 conn(0x7fded00a26f0 legacy=0x7fded013d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient: _finish_hunting 0 2024-09-16T14:43:35.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 1 monclient: found mon.noname-a 2024-09-16T14:43:35.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:35.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:43:35.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 1 -- 172.21.15.26:0/3710464318 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded00a34b0 con 0x7fded00a1c00 2024-09-16T14:43:35.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient: _finish_auth 0 2024-09-16T14:43:35.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient: _check_auth_tickets 2024-09-16T14:43:35.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:35.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 1 -- 172.21.15.26:0/3710464318 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdec0001620 con 0x7fded00a1c00 2024-09-16T14:43:35.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:43:35.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient: dump: 2024-09-16T14:43:35.968 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.968 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.968 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.968 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.968 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.968 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.969 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 5 monclient: authenticate success, global_id 74172 2024-09-16T14:43:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 1 -- 172.21.15.26:0/3710464318 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdec0001840 con 0x7fded00a1c00 2024-09-16T14:43:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 1 -- 172.21.15.26:0/3710464318 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdec0001bd0 con 0x7fded00a1c00 2024-09-16T14:43:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 4 set_mon_vals no callback set 2024-09-16T14:43:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeceffd640 10 monclient: dump: 2024-09-16T14:43:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.970 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient: shutdown 2024-09-16T14:43:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 1 -- 172.21.15.26:0/3710464318 >> v1:172.21.15.26:6789/0 conn(0x7fded00a1c00 legacy=0x7fded00a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 1 -- 172.21.15.26:0/3710464318 shutdown_connections 2024-09-16T14:43:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 1 -- 172.21.15.26:0/3710464318 >> 172.21.15.26:0/3710464318 conn(0x7fded00a3d00 msgr2=0x7fded00a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 1 -- 172.21.15.26:0/3710464318 shutdown_connections 2024-09-16T14:43:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 1 -- 172.21.15.26:0/3710464318 wait complete. 2024-09-16T14:43:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient: build_initial_monmap 2024-09-16T14:43:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient: monmap: 2024-09-16T14:43:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:43:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:43:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:43:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:35.972 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 1 Processor -- start 2024-09-16T14:43:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 1 -- start start 2024-09-16T14:43:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient: init 2024-09-16T14:43:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:43:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient(hunting): picked mon.noname-a con 0x7fded0099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient(hunting): picked mon.noname-b con 0x7fded00a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient(hunting): picked mon.noname-c con 0x7fded00a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded013c610 con 0x7fded0099b60 2024-09-16T14:43:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded00a13f0 con 0x7fded00a1c00 2024-09-16T14:43:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded013b9d0 con 0x7fded00a26f0 2024-09-16T14:43:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdecffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fded0099b60 0x7fded013e1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52044/0 (socket says 172.21.15.26:52044) 2024-09-16T14:43:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fded6733640 10 monclient(hunting): authenticate will time out at 1040.079634s 2024-09-16T14:43:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdecffff640 1 -- 172.21.15.26:0/967917372 learned_addr learned my addr 172.21.15.26:0/967917372 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307819569 0 0) 0x7fded013c610 con 0x7fded0099b60 2024-09-16T14:43:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeccff9640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeccff9640 10 monclient(hunting): my global_id is 74175 2024-09-16T14:43:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdebc003530 con 0x7fded0099b60 2024-09-16T14:43:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512942884 0 0) 0x7fded013b9d0 con 0x7fded00a26f0 2024-09-16T14:43:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeccff9640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeccff9640 10 monclient(hunting): my global_id is 4511 2024-09-16T14:43:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.955+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded013c610 con 0x7fded00a26f0 2024-09-16T14:43:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2499526557 0 0) 0x7fdebc003530 con 0x7fded0099b60 2024-09-16T14:43:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fded013b9d0 con 0x7fded0099b60 2024-09-16T14:43:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1632949976 0 0) 0x7fded013c610 con 0x7fded00a26f0 2024-09-16T14:43:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdebc003530 con 0x7fded00a26f0 2024-09-16T14:43:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdec0001520 con 0x7fded0099b60 2024-09-16T14:43:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:43:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient(hunting): dump: 2024-09-16T14:43:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.976 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdec8001c40 con 0x7fded00a26f0 2024-09-16T14:43:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient(hunting): dump: 2024-09-16T14:43:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.977 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675286021 0 0) 0x7fded00a13f0 con 0x7fded00a1c00 2024-09-16T14:43:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient(hunting): my global_id is 4537 2024-09-16T14:43:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded013c610 con 0x7fded00a1c00 2024-09-16T14:43:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4141164840 0 0) 0x7fded013b9d0 con 0x7fded0099b60 2024-09-16T14:43:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 >> v1:172.21.15.26:6790/0 conn(0x7fded00a26f0 legacy=0x7fded01d1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 >> v1:172.21.15.189:6789/0 conn(0x7fded00a1c00 legacy=0x7fded01cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient: _finish_hunting 0 2024-09-16T14:43:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 monclient: found mon.a 2024-09-16T14:43:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded013d7e0 con 0x7fded0099b60 2024-09-16T14:43:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient: _finish_auth 0 2024-09-16T14:43:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient: _check_auth_tickets 2024-09-16T14:43:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fded6733640 5 monclient: authenticate success, global_id 74175 2024-09-16T14:43:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fded6733640 10 monclient: _renew_subs 2024-09-16T14:43:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fded6733640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fded6733640 1 -- 172.21.15.26:0/967917372 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded013dda0 con 0x7fded0099b60 2024-09-16T14:43:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fded6733640 10 monclient: _renew_subs 2024-09-16T14:43:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fded6733640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:43:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fded6733640 1 -- 172.21.15.26:0/967917372 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fded01d2750 con 0x7fded0099b60 2024-09-16T14:43:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdec0004b10 con 0x7fded0099b60 2024-09-16T14:43:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdec0004e20 con 0x7fded0099b60 2024-09-16T14:43:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:43:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdece7fc640 4 set_mon_vals no callback set 2024-09-16T14:43:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient: dump: 2024-09-16T14:43:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.981 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fded6733640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:35.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fded6733640 1 -- 172.21.15.26:0/967917372 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdeb0006590 con 0x7fdeb0002830 2024-09-16T14:43:35.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fdec0004f80 con 0x7fded0099b60 2024-09-16T14:43:35.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(19..19 src has 1..19) v4 ==== 3873+0+0 (unknown 73816822 0 0) 0x7fdec0043480 con 0x7fded0099b60 2024-09-16T14:43:35.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fded4cd0640 10 monclient: get_auth_request con 0x7fdebc036300 auth_method 0 2024-09-16T14:43:35.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230579715 0 0) 0x7fdeb0006590 con 0x7fdeb0002830 2024-09-16T14:43:35.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient(hunting): my global_id is 4540 2024-09-16T14:43:35.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded013b9d0 con 0x7fdeb0002830 2024-09-16T14:43:35.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 405076450 0 0) 0x7fded013b9d0 con 0x7fdeb0002830 2024-09-16T14:43:35.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdeb0006590 con 0x7fdeb0002830 2024-09-16T14:43:35.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdec40017d0 con 0x7fdeb0002830 2024-09-16T14:43:35.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:35.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient: dump: 2024-09-16T14:43:35.982 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.982 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.982 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.982 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.982 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.982 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.982 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.982 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.983 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.983 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 994476610 0 0) 0x7fdeb0006590 con 0x7fdeb0002830 2024-09-16T14:43:35.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdeb000a0e0 con 0x7fdeb0002830 2024-09-16T14:43:35.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fdec4001b40 con 0x7fdeb0002830 2024-09-16T14:43:35.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:35.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:35.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 >> v1:172.21.15.189:6789/0 conn(0x7fdeb0002830 legacy=0x7fdeb0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.959+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 >> v1:172.21.15.189:6789/0 conn(0x7fdeb0002830 legacy=0x7fdeb0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:35.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fded6733640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:43:35.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fded6733640 1 -- 172.21.15.26:0/967917372 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdeb0009d80 con 0x7fdeb00090a0 2024-09-16T14:43:35.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187025633 0 0) 0x7fdeb0009d80 con 0x7fdeb00090a0 2024-09-16T14:43:35.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 10 monclient(hunting): _negotiate 2024-09-16T14:43:35.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:35.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:35.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 10 monclient(hunting): my global_id is 4543 2024-09-16T14:43:35.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeb0006590 con 0x7fdeb00090a0 2024-09-16T14:43:35.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3980684260 0 0) 0x7fdeb0006590 con 0x7fdeb00090a0 2024-09-16T14:43:35.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdeb0009d80 con 0x7fdeb00090a0 2024-09-16T14:43:35.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdec40049e0 con 0x7fdeb00090a0 2024-09-16T14:43:35.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:35.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:35.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 10 monclient: dump: 2024-09-16T14:43:35.986 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:35.986 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:35.986 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.986 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:35.986 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:35.986 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:35.986 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:35.987 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:35.987 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:35.987 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:35.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3984015262 0 0) 0x7fdeb0009d80 con 0x7fdeb00090a0 2024-09-16T14:43:35.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdeb000c290 con 0x7fdeb00090a0 2024-09-16T14:43:35.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 464513737) 0x7fdec4004e80 con 0x7fdeb00090a0 2024-09-16T14:43:35.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:43:35.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:43:35.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 >> v1:172.21.15.189:6789/0 conn(0x7fdeb00090a0 legacy=0x7fdeb0009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:35.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.967+0000 7fdeccff9640 1 -- 172.21.15.26:0/967917372 >> v1:172.21.15.189:6789/0 conn(0x7fdeb00090a0 legacy=0x7fdeb0009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:35.988 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:43:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:43:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:43:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:43:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 16, 2024-09-16T14:43:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:43:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:43:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:43:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:43:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-09-16T14:43:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:43:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:43:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:43:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:43:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:43:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:43:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:43:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:43:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:43:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:43:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:43:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:43:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:43:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:43:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:43:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:43:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:43:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:43:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:43:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:43:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:43:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:43:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:43:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:43:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:43:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:43:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:43:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:43:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:43:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:43:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:43:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:43:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:43:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:43:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:43:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:43:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:43:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:43:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:43:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:43:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:43:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:43:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:43:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:43:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:43:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:43:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:43:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:43:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:43:36.001 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:43:36.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.971+0000 7fded6733640 1 -- 172.21.15.26:0/967917372 >> v1:172.21.15.189:6812/3196161582 conn(0x7fdebc036300 legacy=0x7fdebc0387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:36.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.971+0000 7fded6733640 10 monclient: shutdown 2024-09-16T14:43:36.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.971+0000 7fded6733640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:36.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.971+0000 7fded6733640 1 -- 172.21.15.26:0/967917372 >> v1:172.21.15.26:6789/0 conn(0x7fded0099b60 legacy=0x7fded013e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:36.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.971+0000 7fded4cd0640 1 -- 172.21.15.26:0/967917372 reap_dead start 2024-09-16T14:43:36.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.971+0000 7fded4cd0640 1 -- 172.21.15.26:0/967917372 reap_dead start 2024-09-16T14:43:36.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.971+0000 7fded6733640 1 -- 172.21.15.26:0/967917372 shutdown_connections 2024-09-16T14:43:36.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.971+0000 7fded6733640 1 -- 172.21.15.26:0/967917372 >> 172.21.15.26:0/967917372 conn(0x7fded00a3d00 msgr2=0x7fded006bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:36.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.971+0000 7fded6733640 1 -- 172.21.15.26:0/967917372 shutdown_connections 2024-09-16T14:43:36.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:35.971+0000 7fded6733640 1 -- 172.21.15.26:0/967917372 wait complete. 2024-09-16T14:43:36.004 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:43:36.004 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T14:43:36.004 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:43:36.004 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T14:43:36.004 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:43:36.004 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T14:43:36.004 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:43:36.014 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T14:43:36.014 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:43:36.014 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:43:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a30c22640 10 monclient: get_monmap_and_config 2024-09-16T14:43:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a30c22640 10 monclient: build_initial_monmap 2024-09-16T14:43:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a30c22640 10 monclient: monmap: 2024-09-16T14:43:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:43:36.167568+0000 2024-09-16T14:43:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:43:36.167568+0000 2024-09-16T14:43:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:36.167 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a30c22640 1 Processor -- start 2024-09-16T14:43:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a30c22640 1 -- start start 2024-09-16T14:43:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a30c22640 10 monclient: init 2024-09-16T14:43:36.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a30c22640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a30c22640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:43:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a30c22640 10 monclient(hunting): picked mon.noname-c con 0x7f3a2c0a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a30c22640 10 monclient(hunting): picked mon.noname-b con 0x7f3a2c0a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a30c22640 10 monclient(hunting): picked mon.noname-a con 0x7f3a2c0a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a30c22640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c0a3250 con 0x7f3a2c0a2800 2024-09-16T14:43:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a30c22640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c09f600 con 0x7f3a2c0a1d10 2024-09-16T14:43:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a30c22640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c0a3830 con 0x7f3a2c0a0000 2024-09-16T14:43:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a30c22640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a30c22640 10 monclient(hunting): authenticate will time out at 1040.287634s 2024-09-16T14:43:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a2ad9d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f3a2c0a2800 legacy=0x7f3a2c13d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a29d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3a2c0a0000 0x7f3a2c139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40408/0 (socket says 172.21.15.26:40408) 2024-09-16T14:43:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.163+0000 7f3a29d9b640 1 -- 172.21.15.26:0/3660132300 learned_addr learned my addr 172.21.15.26:0/3660132300 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:36.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 1 -- 172.21.15.26:0/3660132300 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753691565 0 0) 0x7f3a2c0a3830 con 0x7f3a2c0a0000 2024-09-16T14:43:36.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 10 monclient(hunting): _negotiate 2024-09-16T14:43:36.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:36.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:36.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 10 monclient(hunting): my global_id is 4514 2024-09-16T14:43:36.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 1 -- 172.21.15.26:0/3660132300 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a18003530 con 0x7f3a2c0a0000 2024-09-16T14:43:36.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 1 -- 172.21.15.26:0/3660132300 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 326254565 0 0) 0x7f3a18003530 con 0x7f3a2c0a0000 2024-09-16T14:43:36.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 1 -- 172.21.15.26:0/3660132300 >> v1:172.21.15.189:6789/0 conn(0x7f3a2c0a1d10 legacy=0x7f3a2c0a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:36.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 1 -- 172.21.15.26:0/3660132300 >> v1:172.21.15.26:6789/0 conn(0x7f3a2c0a2800 legacy=0x7f3a2c13d440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:36.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 10 monclient: _finish_hunting 0 2024-09-16T14:43:36.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 1 monclient: found mon.noname-b 2024-09-16T14:43:36.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:36.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:43:36.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 1 -- 172.21.15.26:0/3660132300 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2c09ed60 con 0x7f3a2c0a0000 2024-09-16T14:43:36.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 10 monclient: _finish_auth 0 2024-09-16T14:43:36.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 10 monclient: _check_auth_tickets 2024-09-16T14:43:36.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:36.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 1 -- 172.21.15.26:0/3660132300 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3a1c001620 con 0x7f3a2c0a0000 2024-09-16T14:43:36.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:36.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:36.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 10 monclient: dump: 2024-09-16T14:43:36.173 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:36.173 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:36.173 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:36.173 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:36.174 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:36.174 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:36.174 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:36.174 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:36.174 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:36.174 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:36.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 5 monclient: authenticate success, global_id 4514 2024-09-16T14:43:36.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:36.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 1 -- 172.21.15.26:0/3660132300 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a1c001840 con 0x7f3a2c0a0000 2024-09-16T14:43:36.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:36.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:36.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 4 set_mon_vals no callback set 2024-09-16T14:43:36.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 1 -- 172.21.15.26:0/3660132300 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3a1c001c10 con 0x7f3a2c0a0000 2024-09-16T14:43:36.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:36.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:36.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2959a640 10 monclient: dump: 2024-09-16T14:43:36.175 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:36.175 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:36.175 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:36.175 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:36.175 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:36.175 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:36.175 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:36.175 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:36.175 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:36.175 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:36.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 10 monclient: shutdown 2024-09-16T14:43:36.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:36.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 1 -- 172.21.15.26:0/3660132300 >> v1:172.21.15.26:6790/0 conn(0x7f3a2c0a0000 legacy=0x7f3a2c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:36.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 1 -- 172.21.15.26:0/3660132300 shutdown_connections 2024-09-16T14:43:36.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 1 -- 172.21.15.26:0/3660132300 >> 172.21.15.26:0/3660132300 conn(0x7f3a2c0a7f30 msgr2=0x7f3a2c0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:36.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 1 -- 172.21.15.26:0/3660132300 shutdown_connections 2024-09-16T14:43:36.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 1 -- 172.21.15.26:0/3660132300 wait complete. 2024-09-16T14:43:36.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 10 monclient: build_initial_monmap 2024-09-16T14:43:36.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 10 monclient: monmap: 2024-09-16T14:43:36.176 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:36.177 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:36.177 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:43:36.177 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:43:36.177 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:36.177 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:43:36.177 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:36.177 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:36.177 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:36.177 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:36.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 1 Processor -- start 2024-09-16T14:43:36.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 1 -- start start 2024-09-16T14:43:36.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 10 monclient: init 2024-09-16T14:43:36.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:36.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:43:36.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 10 monclient(hunting): picked mon.noname-c con 0x7f3a2c0a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:36.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 10 monclient(hunting): picked mon.noname-a con 0x7f3a2c0a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:36.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 10 monclient(hunting): picked mon.noname-b con 0x7f3a2c0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:36.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c131ca0 con 0x7f3a2c0a1d10 2024-09-16T14:43:36.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c130fe0 con 0x7f3a2c0a2800 2024-09-16T14:43:36.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c09e500 con 0x7f3a2c0a0000 2024-09-16T14:43:36.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:36.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a29d9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f3a2c0a1d10 legacy=0x7f3a2c1308d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:36.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 10 monclient(hunting): authenticate will time out at 1040.290952s 2024-09-16T14:43:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3a2c0a0000 0x7f3a2c1301c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40414/0 (socket says 172.21.15.26:40414) 2024-09-16T14:43:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a2a59c640 1 -- 172.21.15.26:0/654291714 learned_addr learned my addr 172.21.15.26:0/654291714 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 1 -- 172.21.15.26:0/654291714 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529002996 0 0) 0x7f3a2c09e500 con 0x7f3a2c0a0000 2024-09-16T14:43:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:43:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 10 monclient(hunting): my global_id is 4517 2024-09-16T14:43:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 1 -- 172.21.15.26:0/654291714 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a0c003530 con 0x7f3a2c0a0000 2024-09-16T14:43:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 1 -- 172.21.15.26:0/654291714 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2780687803 0 0) 0x7f3a0c003530 con 0x7f3a2c0a0000 2024-09-16T14:43:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 1 -- 172.21.15.26:0/654291714 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a2c09e500 con 0x7f3a2c0a0000 2024-09-16T14:43:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 1 -- 172.21.15.26:0/654291714 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3a14001980 con 0x7f3a2c0a0000 2024-09-16T14:43:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:36.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:36.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 10 monclient(hunting): dump: 2024-09-16T14:43:36.183 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:36.183 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:36.183 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:36.183 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:36.183 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:36.183 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:36.183 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:36.183 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:36.183 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:36.183 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:36.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:36.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 1 -- 172.21.15.26:0/654291714 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1862445822 0 0) 0x7f3a2c09e500 con 0x7f3a2c0a0000 2024-09-16T14:43:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 1 -- 172.21.15.26:0/654291714 >> v1:172.21.15.189:6789/0 conn(0x7f3a2c0a2800 legacy=0x7f3a2c1d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 1 -- 172.21.15.26:0/654291714 >> v1:172.21.15.26:6789/0 conn(0x7f3a2c0a1d10 legacy=0x7f3a2c1308d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:43:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 1 monclient: found mon.c 2024-09-16T14:43:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 1 -- 172.21.15.26:0/654291714 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2c132b60 con 0x7f3a2c0a0000 2024-09-16T14:43:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 10 monclient: _finish_auth 0 2024-09-16T14:43:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:43:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 5 monclient: authenticate success, global_id 4517 2024-09-16T14:43:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 10 monclient: _renew_subs 2024-09-16T14:43:36.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:36.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 1 -- 172.21.15.26:0/654291714 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a2c0a66d0 con 0x7f3a2c0a0000 2024-09-16T14:43:36.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 10 monclient: _renew_subs 2024-09-16T14:43:36.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:36.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a30c22640 1 -- 172.21.15.26:0/654291714 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a2c133720 con 0x7f3a2c0a0000 2024-09-16T14:43:36.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 1 -- 172.21.15.26:0/654291714 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a14001d50 con 0x7f3a2c0a0000 2024-09-16T14:43:36.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a22ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:36.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.167+0000 7f3a28d99640 4 set_mon_vals no callback set 2024-09-16T14:43:36.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.171+0000 7f3a22ffd640 1 -- 172.21.15.26:0/654291714 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3a14003a70 con 0x7f3a2c0a0000 2024-09-16T14:43:36.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.171+0000 7f3a22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:36.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.171+0000 7f3a22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:36.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.171+0000 7f3a22ffd640 10 monclient: dump: 2024-09-16T14:43:36.185 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:36.185 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:36.186 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:36.186 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:36.186 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:36.186 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:36.186 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:36.186 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:36.186 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:36.186 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:36.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.171+0000 7f3a22ffd640 1 -- 172.21.15.26:0/654291714 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3a14003c50 con 0x7f3a2c0a0000 2024-09-16T14:43:36.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.171+0000 7f3a30c22640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:36.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.171+0000 7f3a30c22640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:36.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.171+0000 7f3a30c22640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:36.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.171+0000 7f3a30c22640 1 -- 172.21.15.26:0/654291714 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a10005050 con 0x7f3a2c0a0000 2024-09-16T14:43:36.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.171+0000 7f3a22ffd640 1 -- 172.21.15.26:0/654291714 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(19..19 src has 1..19) v4 ==== 3873+0+0 (unknown 73816822 0 0) 0x7f3a14043470 con 0x7f3a2c0a0000 2024-09-16T14:43:36.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.171+0000 7f3a29d9b640 10 monclient: get_auth_request con 0x7f3a0c036000 auth_method 0 2024-09-16T14:43:36.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.175+0000 7f3a22ffd640 1 -- 172.21.15.26:0/654291714 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3a140148e0 con 0x7f3a2c0a0000 2024-09-16T14:43:36.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.175+0000 7f3a22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:36.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.175+0000 7f3a22ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:36.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.387+0000 7f3a30c22640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:43:36.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.387+0000 7f3a30c22640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:43:36.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.387+0000 7f3a30c22640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:36.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.387+0000 7f3a30c22640 1 -- 172.21.15.26:0/654291714 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3a10005880 con 0x7f3a2c0a0000 2024-09-16T14:43:36.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.391+0000 7f3a22ffd640 1 -- 172.21.15.26:0/654291714 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2237547178) 0x7f3a14012d50 con 0x7f3a2c0a0000 2024-09-16T14:43:36.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.391+0000 7f3a22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:43:36.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.391+0000 7f3a22ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:43:36.393 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:43:36.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.391+0000 7f3a30c22640 1 -- 172.21.15.26:0/654291714 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3a0c036000 legacy=0x7f3a0c0384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:36.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.391+0000 7f3a30c22640 10 monclient: shutdown 2024-09-16T14:43:36.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.391+0000 7f3a30c22640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:36.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.391+0000 7f3a30c22640 1 -- 172.21.15.26:0/654291714 >> v1:172.21.15.26:6790/0 conn(0x7f3a2c0a0000 legacy=0x7f3a2c1301c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:36.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.391+0000 7f3a30c22640 1 -- 172.21.15.26:0/654291714 shutdown_connections 2024-09-16T14:43:36.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.391+0000 7f3a30c22640 1 -- 172.21.15.26:0/654291714 >> 172.21.15.26:0/654291714 conn(0x7f3a2c0a7f30 msgr2=0x7f3a2c06b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:36.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.391+0000 7f3a30c22640 1 -- 172.21.15.26:0/654291714 shutdown_connections 2024-09-16T14:43:36.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:36.391+0000 7f3a30c22640 1 -- 172.21.15.26:0/654291714 wait complete. 2024-09-16T14:43:38.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:38.591+0000 7f2f17fff640 10 monclient: tick 2024-09-16T14:43:38.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:38.591+0000 7f2f17fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:38.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:38.591+0000 7f2f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:38.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:38.591+0000 7f2f17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:39.411 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:43:39.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ee513640 10 monclient: get_monmap_and_config 2024-09-16T14:43:39.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ee513640 10 monclient: build_initial_monmap 2024-09-16T14:43:39.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ee513640 10 monclient: monmap: 2024-09-16T14:43:39.543 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:39.543 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:39.543 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:43:39.543688+0000 2024-09-16T14:43:39.543 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:43:39.543688+0000 2024-09-16T14:43:39.543 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:39.543 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:39.544 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:39.544 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:39.544 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:39.544 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:39.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ee513640 1 Processor -- start 2024-09-16T14:43:39.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ee513640 1 -- start start 2024-09-16T14:43:39.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ee513640 10 monclient: init 2024-09-16T14:43:39.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ee513640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:39.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ee513640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:43:39.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ee513640 10 monclient(hunting): picked mon.noname-a con 0x7f82e809a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:39.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ee513640 10 monclient(hunting): picked mon.noname-c con 0x7f82e809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:39.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ee513640 10 monclient(hunting): picked mon.noname-b con 0x7f82e809b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:39.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ee513640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e809dc60 con 0x7f82e809a550 2024-09-16T14:43:39.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ee513640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e8099ae0 con 0x7f82e809d1b0 2024-09-16T14:43:39.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ee513640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e809b8f0 con 0x7f82e809b040 2024-09-16T14:43:39.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ee513640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:39.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ee513640 10 monclient(hunting): authenticate will time out at 1043.663786s 2024-09-16T14:43:39.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82e7fff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f82e809a550 legacy=0x7f82e809a930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:39.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.539+0000 7f82ecab0640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f82e809b040 0x7f82e813a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:58788/0 (socket says 172.21.15.26:58788) 2024-09-16T14:43:39.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ecab0640 1 -- 172.21.15.26:0/84791567 learned_addr learned my addr 172.21.15.26:0/84791567 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:39.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 1 -- 172.21.15.26:0/84791567 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588200773 0 0) 0x7f82e809b8f0 con 0x7f82e809b040 2024-09-16T14:43:39.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:43:39.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:39.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:39.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 10 monclient(hunting): my global_id is 4520 2024-09-16T14:43:39.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 1 -- 172.21.15.26:0/84791567 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d4005740 con 0x7f82e809b040 2024-09-16T14:43:39.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 1 -- 172.21.15.26:0/84791567 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2495739009 0 0) 0x7f82d4005740 con 0x7f82e809b040 2024-09-16T14:43:39.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 1 -- 172.21.15.26:0/84791567 >> v1:172.21.15.189:6789/0 conn(0x7f82e809d1b0 legacy=0x7f82e8136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:39.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 1 -- 172.21.15.26:0/84791567 >> v1:172.21.15.26:6789/0 conn(0x7f82e809a550 legacy=0x7f82e809a930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:39.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:43:39.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 1 monclient: found mon.noname-b 2024-09-16T14:43:39.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:39.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:43:39.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 1 -- 172.21.15.26:0/84791567 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e809bbd0 con 0x7f82e809b040 2024-09-16T14:43:39.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:43:39.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:43:39.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:39.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 1 -- 172.21.15.26:0/84791567 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f82d8001620 con 0x7f82e809b040 2024-09-16T14:43:39.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 5 monclient: authenticate success, global_id 4520 2024-09-16T14:43:39.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:39.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:39.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:39.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 10 monclient: dump: 2024-09-16T14:43:39.552 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:39.553 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:39.553 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:39.553 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:39.553 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:39.553 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:39.553 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:39.553 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:39.553 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:39.553 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:39.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:39.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 1 -- 172.21.15.26:0/84791567 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82d8001940 con 0x7f82e809b040 2024-09-16T14:43:39.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:39.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 1 -- 172.21.15.26:0/84791567 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f82d8001cd0 con 0x7f82e809b040 2024-09-16T14:43:39.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:39.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:39.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e6ffd640 10 monclient: dump: 2024-09-16T14:43:39.554 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:39.554 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:39.554 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:39.554 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:39.554 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:39.554 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:39.554 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:39.554 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:39.554 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:39.554 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:39.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:39.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 4 set_mon_vals no callback set 2024-09-16T14:43:39.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 10 monclient: shutdown 2024-09-16T14:43:39.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:39.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 1 -- 172.21.15.26:0/84791567 >> v1:172.21.15.26:6790/0 conn(0x7f82e809b040 legacy=0x7f82e813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:39.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 1 -- 172.21.15.26:0/84791567 shutdown_connections 2024-09-16T14:43:39.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 1 -- 172.21.15.26:0/84791567 >> 172.21.15.26:0/84791567 conn(0x7f82e809fb50 msgr2=0x7f82e809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:39.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 1 -- 172.21.15.26:0/84791567 shutdown_connections 2024-09-16T14:43:39.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 1 -- 172.21.15.26:0/84791567 wait complete. 2024-09-16T14:43:39.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 10 monclient: build_initial_monmap 2024-09-16T14:43:39.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 10 monclient: monmap: 2024-09-16T14:43:39.555 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:39.555 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:39.556 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:43:39.556 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:43:39.556 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:39.556 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:43:39.556 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:39.556 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:39.556 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:39.556 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:39.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 1 Processor -- start 2024-09-16T14:43:39.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 1 -- start start 2024-09-16T14:43:39.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 10 monclient: init 2024-09-16T14:43:39.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:39.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:43:39.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 10 monclient(hunting): picked mon.noname-b con 0x7f82e809a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:39.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 10 monclient(hunting): picked mon.noname-a con 0x7f82e809b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:39.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 10 monclient(hunting): picked mon.noname-c con 0x7f82e809d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:39.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e81b4580 con 0x7f82e809b040 2024-09-16T14:43:39.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e80a22d0 con 0x7f82e809a550 2024-09-16T14:43:39.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e81a7510 con 0x7f82e809d1b0 2024-09-16T14:43:39.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:39.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 10 monclient(hunting): authenticate will time out at 1043.666878s 2024-09-16T14:43:39.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e77fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f82e809b040 legacy=0x7f82e81a42f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:39.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ecab0640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f82e809d1b0 0x7f82e81d1a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:58800/0 (socket says 172.21.15.26:58800) 2024-09-16T14:43:39.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ecab0640 1 -- 172.21.15.26:0/1029906449 learned_addr learned my addr 172.21.15.26:0/1029906449 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:39.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 1 -- 172.21.15.26:0/1029906449 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710555065 0 0) 0x7f82e81a7510 con 0x7f82e809d1b0 2024-09-16T14:43:39.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:43:39.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:39.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:39.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 10 monclient(hunting): my global_id is 4523 2024-09-16T14:43:39.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 1 -- 172.21.15.26:0/1029906449 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82c8003530 con 0x7f82e809d1b0 2024-09-16T14:43:39.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 1 -- 172.21.15.26:0/1029906449 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3258395194 0 0) 0x7f82c8003530 con 0x7f82e809d1b0 2024-09-16T14:43:39.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 1 -- 172.21.15.26:0/1029906449 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82e81a7510 con 0x7f82e809d1b0 2024-09-16T14:43:39.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 1 -- 172.21.15.26:0/1029906449 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f82d80015e0 con 0x7f82e809d1b0 2024-09-16T14:43:39.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:39.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:39.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 10 monclient(hunting): dump: 2024-09-16T14:43:39.558 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:39.558 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:39.558 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:39.558 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:39.559 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:39.559 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:39.559 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:39.559 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:39.559 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:39.559 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:39.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:39.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 1 -- 172.21.15.26:0/1029906449 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2448966550 0 0) 0x7f82e81a7510 con 0x7f82e809d1b0 2024-09-16T14:43:39.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 1 -- 172.21.15.26:0/1029906449 >> v1:172.21.15.189:6789/0 conn(0x7f82e809a550 legacy=0x7f82e81a3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:39.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 1 -- 172.21.15.26:0/1029906449 >> v1:172.21.15.26:6789/0 conn(0x7f82e809b040 legacy=0x7f82e81a42f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:39.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:43:39.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 1 monclient: found mon.c 2024-09-16T14:43:39.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:39.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:39.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 1 -- 172.21.15.26:0/1029906449 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e81a77f0 con 0x7f82e809d1b0 2024-09-16T14:43:39.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 10 monclient: _finish_auth 0 2024-09-16T14:43:39.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:43:39.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:39.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 5 monclient: authenticate success, global_id 4523 2024-09-16T14:43:39.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 10 monclient: _renew_subs 2024-09-16T14:43:39.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:39.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 1 -- 172.21.15.26:0/1029906449 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e81a4a60 con 0x7f82e809d1b0 2024-09-16T14:43:39.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 10 monclient: _renew_subs 2024-09-16T14:43:39.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:39.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82ee513640 1 -- 172.21.15.26:0/1029906449 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e81a6f50 con 0x7f82e809d1b0 2024-09-16T14:43:39.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 1 -- 172.21.15.26:0/1029906449 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82d8004c50 con 0x7f82e809d1b0 2024-09-16T14:43:39.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:39.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.543+0000 7f82e67fc640 4 set_mon_vals no callback set 2024-09-16T14:43:39.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.547+0000 7f82e4ff9640 1 -- 172.21.15.26:0/1029906449 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f82d8004fe0 con 0x7f82e809d1b0 2024-09-16T14:43:39.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.547+0000 7f82e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:39.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.547+0000 7f82e4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:39.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.547+0000 7f82e4ff9640 10 monclient: dump: 2024-09-16T14:43:39.561 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:39.561 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:39.561 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:39.562 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:39.562 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:39.562 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:39.562 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:39.562 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:39.562 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:39.562 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:39.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.547+0000 7f82e4ff9640 1 -- 172.21.15.26:0/1029906449 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f82d80051c0 con 0x7f82e809d1b0 2024-09-16T14:43:39.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.547+0000 7f82ee513640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:39.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.547+0000 7f82ee513640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:39.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.547+0000 7f82ee513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:39.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.547+0000 7f82ee513640 1 -- 172.21.15.26:0/1029906449 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82cc005050 con 0x7f82e809d1b0 2024-09-16T14:43:39.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.547+0000 7f82e4ff9640 1 -- 172.21.15.26:0/1029906449 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(19..19 src has 1..19) v4 ==== 3873+0+0 (unknown 73816822 0 0) 0x7f82d8043520 con 0x7f82e809d1b0 2024-09-16T14:43:39.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.547+0000 7f82e7fff640 10 monclient: get_auth_request con 0x7f82c8035e60 auth_method 0 2024-09-16T14:43:39.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.551+0000 7f82e4ff9640 1 -- 172.21.15.26:0/1029906449 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f82d8014b00 con 0x7f82e809d1b0 2024-09-16T14:43:39.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.551+0000 7f82e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:39.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.551+0000 7f82e4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:39.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:39.591+0000 7f2f17fff640 10 monclient: tick 2024-09-16T14:43:39.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:39.591+0000 7f2f17fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:39.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:39.591+0000 7f2f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:39.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.759+0000 7f82ee513640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:43:39.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.759+0000 7f82ee513640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:43:39.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.759+0000 7f82ee513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:39.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.759+0000 7f82ee513640 1 -- 172.21.15.26:0/1029906449 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f82cc005880 con 0x7f82e809d1b0 2024-09-16T14:43:39.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.759+0000 7f82e4ff9640 1 -- 172.21.15.26:0/1029906449 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3585248395) 0x7f82d8012f70 con 0x7f82e809d1b0 2024-09-16T14:43:39.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.759+0000 7f82e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:43:39.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.759+0000 7f82e4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:43:39.763 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:43:39.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.759+0000 7f82ee513640 1 -- 172.21.15.26:0/1029906449 >> v1:172.21.15.189:6812/3196161582 conn(0x7f82c8035e60 legacy=0x7f82c8038320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:39.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.759+0000 7f82ee513640 10 monclient: shutdown 2024-09-16T14:43:39.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.759+0000 7f82ee513640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:39.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.759+0000 7f82ee513640 1 -- 172.21.15.26:0/1029906449 >> v1:172.21.15.26:6790/0 conn(0x7f82e809d1b0 legacy=0x7f82e81d1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:39.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.759+0000 7f82ee513640 1 -- 172.21.15.26:0/1029906449 shutdown_connections 2024-09-16T14:43:39.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.759+0000 7f82ee513640 1 -- 172.21.15.26:0/1029906449 >> 172.21.15.26:0/1029906449 conn(0x7f82e809fb50 msgr2=0x7f82e80a1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:39.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.763+0000 7f82ee513640 1 -- 172.21.15.26:0/1029906449 shutdown_connections 2024-09-16T14:43:39.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:39.763+0000 7f82ee513640 1 -- 172.21.15.26:0/1029906449 wait complete. 2024-09-16T14:43:40.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:40.591+0000 7f2f17fff640 10 monclient: tick 2024-09-16T14:43:40.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:40.591+0000 7f2f17fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:40.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:40.591+0000 7f2f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:41.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:41.591+0000 7f2f17fff640 10 monclient: tick 2024-09-16T14:43:41.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:41.591+0000 7f2f17fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:41.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:41.591+0000 7f2f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:42.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:42.591+0000 7f2f17fff640 10 monclient: tick 2024-09-16T14:43:42.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:42.591+0000 7f2f17fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:42.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:42.591+0000 7f2f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:42.781 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:43:42.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f4066de2640 10 monclient: get_monmap_and_config 2024-09-16T14:43:42.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f4066de2640 10 monclient: build_initial_monmap 2024-09-16T14:43:42.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f4066de2640 10 monclient: monmap: 2024-09-16T14:43:42.914 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:42.914 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:42.915 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:43:42.915043+0000 2024-09-16T14:43:42.915 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:43:42.915043+0000 2024-09-16T14:43:42.915 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:42.915 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:42.915 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:42.915 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:42.915 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:42.915 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:42.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f4066de2640 1 Processor -- start 2024-09-16T14:43:42.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f4066de2640 1 -- start start 2024-09-16T14:43:42.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f4066de2640 10 monclient: init 2024-09-16T14:43:42.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f4066de2640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:42.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f4066de2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:43:42.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f4066de2640 10 monclient(hunting): picked mon.noname-c con 0x7f406009a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:42.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f4066de2640 10 monclient(hunting): picked mon.noname-a con 0x7f406009d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:42.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f4066de2640 10 monclient(hunting): picked mon.noname-b con 0x7f406009b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:42.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f4066de2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406009dc60 con 0x7f406009d1b0 2024-09-16T14:43:42.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f4066de2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4060099ae0 con 0x7f406009a550 2024-09-16T14:43:42.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f4066de2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406009b8f0 con 0x7f406009b040 2024-09-16T14:43:42.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f4066de2640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:42.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f4066de2640 10 monclient(hunting): authenticate will time out at 1047.035150s 2024-09-16T14:43:42.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f405ffff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f406009d1b0 legacy=0x7f4060136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:42.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f406537f640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f406009b040 0x7f406013a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:58814/0 (socket says 172.21.15.26:58814) 2024-09-16T14:43:42.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:42.911+0000 7f406537f640 1 -- 172.21.15.26:0/390051077 learned_addr learned my addr 172.21.15.26:0/390051077 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:43.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:43.115+0000 7f405ffff640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.26:6789/0 conn(0x7f406009d1b0 legacy=0x7f4060136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:43.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:43.519+0000 7f405ffff640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.26:6789/0 conn(0x7f406009d1b0 legacy=0x7f4060136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:43.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:43.591+0000 7f2f17fff640 10 monclient: tick 2024-09-16T14:43:43.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:43.591+0000 7f2f17fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:43.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:43.591+0000 7f2f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:44.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:44.323+0000 7f405ffff640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.26:6789/0 conn(0x7f406009d1b0 legacy=0x7f4060136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:44.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:44.595+0000 7f2f17fff640 10 monclient: tick 2024-09-16T14:43:44.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:44.595+0000 7f2f17fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:44.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:44.595+0000 7f2f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:45.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:45.591+0000 7f2f17fff640 10 monclient: tick 2024-09-16T14:43:45.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:45.591+0000 7f2f17fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:45.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:45.591+0000 7f2f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:45.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:45.915+0000 7f405e7fc640 10 monclient(hunting): tick 2024-09-16T14:43:45.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:45.915+0000 7f405e7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:43:45.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:45.915+0000 7f405e7fc640 1 monclient(hunting): continuing hunt 2024-09-16T14:43:45.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:45.915+0000 7f405e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:43:45.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:45.915+0000 7f405e7fc640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.26:6790/0 conn(0x7f406009b040 legacy=0x7f406013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:45.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:45.915+0000 7f405e7fc640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.189:6789/0 conn(0x7f406009a550 legacy=0x7f406009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:45.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:45.915+0000 7f405e7fc640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.26:6789/0 conn(0x7f406009d1b0 legacy=0x7f4060136a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:45.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:45.915+0000 7f405e7fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:43:45.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:45.915+0000 7f405e7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f40400036d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:45.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:45.915+0000 7f405e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f4040007360 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:45.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:45.915+0000 7f405e7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f406009d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:45.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:45.915+0000 7f405e7fc640 1 -- 172.21.15.26:0/390051077 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406009bbd0 con 0x7f406009d1b0 2024-09-16T14:43:45.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:45.915+0000 7f405e7fc640 1 -- 172.21.15.26:0/390051077 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406009dc60 con 0x7f40400036d0 2024-09-16T14:43:45.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:45.915+0000 7f405e7fc640 1 -- 172.21.15.26:0/390051077 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4040010900 con 0x7f4040007360 2024-09-16T14:43:45.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:45.915+0000 7f405e7fc640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:45.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:45.915+0000 7f406537f640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.26:6789/0 conn(0x7f406009d1b0 legacy=0x7f404000d150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:46.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:46.119+0000 7f406537f640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.26:6789/0 conn(0x7f406009d1b0 legacy=0x7f404000d150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:46.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:46.523+0000 7f406537f640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.26:6789/0 conn(0x7f406009d1b0 legacy=0x7f404000d150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:46.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:46.595+0000 7f2f17fff640 10 monclient: tick 2024-09-16T14:43:46.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:46.595+0000 7f2f17fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:46.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:46.595+0000 7f2f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:47.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:47.327+0000 7f406537f640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.26:6789/0 conn(0x7f406009d1b0 legacy=0x7f404000d150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:47.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:47.595+0000 7f2f17fff640 10 monclient: tick 2024-09-16T14:43:47.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:47.595+0000 7f2f17fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:47.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:47.595+0000 7f2f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:48.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:48.595+0000 7f2f17fff640 10 monclient: tick 2024-09-16T14:43:48.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:48.595+0000 7f2f17fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:48.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:48.595+0000 7f2f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:48.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:48.595+0000 7f2f17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:48.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:48.915+0000 7f405e7fc640 10 monclient(hunting): tick 2024-09-16T14:43:48.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:48.915+0000 7f405e7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:43:48.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:48.915+0000 7f405e7fc640 1 monclient(hunting): continuing hunt 2024-09-16T14:43:48.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:48.915+0000 7f405e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:43:48.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:48.915+0000 7f405e7fc640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.26:6790/0 conn(0x7f4040007360 legacy=0x7f40400097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:48.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:48.915+0000 7f405e7fc640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.189:6789/0 conn(0x7f40400036d0 legacy=0x7f4040005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:48.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:48.915+0000 7f405e7fc640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.26:6789/0 conn(0x7f406009d1b0 legacy=0x7f404000d150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:48.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:48.915+0000 7f405e7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:43:48.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:48.915+0000 7f405e7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f4040010c40 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:48.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:48.915+0000 7f405e7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f406009d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:48.915+0000 7f405e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f4040012d70 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:48.915+0000 7f405e7fc640 1 -- 172.21.15.26:0/390051077 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4040011af0 con 0x7f406009d1b0 2024-09-16T14:43:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:48.915+0000 7f405e7fc640 1 -- 172.21.15.26:0/390051077 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406009bbd0 con 0x7f4040010c40 2024-09-16T14:43:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:48.915+0000 7f405e7fc640 1 -- 172.21.15.26:0/390051077 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4040003b00 con 0x7f4040012d70 2024-09-16T14:43:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:48.915+0000 7f405e7fc640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:48.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:48.915+0000 7f405ffff640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.26:6789/0 conn(0x7f406009d1b0 legacy=0x7f404000d150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:49.119+0000 7f405ffff640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.26:6789/0 conn(0x7f406009d1b0 legacy=0x7f404000d150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:49.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:49.523+0000 7f405ffff640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.26:6789/0 conn(0x7f406009d1b0 legacy=0x7f404000d150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:49.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:49.595+0000 7f2f17fff640 10 monclient: tick 2024-09-16T14:43:49.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:49.595+0000 7f2f17fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:49.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:49.595+0000 7f2f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:50.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 1 -- 172.21.15.26:0/390051077 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985923475 0 0) 0x7f4040003b00 con 0x7f4040012d70 2024-09-16T14:43:50.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:50.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:50.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:50.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 10 monclient(hunting): my global_id is 4526 2024-09-16T14:43:50.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 1 -- 172.21.15.26:0/390051077 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f404c003530 con 0x7f4040012d70 2024-09-16T14:43:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 1 -- 172.21.15.26:0/390051077 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768431492 0 0) 0x7f406009bbd0 con 0x7f4040010c40 2024-09-16T14:43:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 10 monclient(hunting): my global_id is 4546 2024-09-16T14:43:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 1 -- 172.21.15.26:0/390051077 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4040003b00 con 0x7f4040010c40 2024-09-16T14:43:50.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 1 -- 172.21.15.26:0/390051077 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 491151029 0 0) 0x7f4040003b00 con 0x7f4040010c40 2024-09-16T14:43:50.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.26:6790/0 conn(0x7f4040012d70 legacy=0x7f4040015160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.26:6789/0 conn(0x7f406009d1b0 legacy=0x7f404000d150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:50.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:43:50.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 1 monclient: found mon.noname-c 2024-09-16T14:43:50.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:50.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:50.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 1 -- 172.21.15.26:0/390051077 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4040004cf0 con 0x7f4040010c40 2024-09-16T14:43:50.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:43:50.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:43:50.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:50.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 1 -- 172.21.15.26:0/390051077 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f40480015a0 con 0x7f4040010c40 2024-09-16T14:43:50.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:50.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:50.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.087+0000 7f405f7fe640 10 monclient: dump: 2024-09-16T14:43:50.097 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:50.097 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.097 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.097 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.097 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:50.097 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.097 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:50.097 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:50.097 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:50.097 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f4066de2640 5 monclient: authenticate success, global_id 4546 2024-09-16T14:43:50.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f4066de2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:50.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f405f7fe640 1 -- 172.21.15.26:0/390051077 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40480017c0 con 0x7f4040010c40 2024-09-16T14:43:50.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f405f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:50.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f405f7fe640 1 -- 172.21.15.26:0/390051077 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4048001b90 con 0x7f4040010c40 2024-09-16T14:43:50.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f405f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:50.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f405f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:50.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f405f7fe640 10 monclient: dump: 2024-09-16T14:43:50.098 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:50.098 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.098 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.098 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.098 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:50.099 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f4066de2640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f4066de2640 4 set_mon_vals no callback set 2024-09-16T14:43:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f4066de2640 10 monclient: shutdown 2024-09-16T14:43:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f4066de2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f4066de2640 1 -- 172.21.15.26:0/390051077 >> v1:172.21.15.189:6789/0 conn(0x7f4040010c40 legacy=0x7f4040011080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f4066de2640 1 -- 172.21.15.26:0/390051077 shutdown_connections 2024-09-16T14:43:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f4066de2640 1 -- 172.21.15.26:0/390051077 >> 172.21.15.26:0/390051077 conn(0x7f406009fb50 msgr2=0x7f406009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:50.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f4066de2640 1 -- 172.21.15.26:0/390051077 shutdown_connections 2024-09-16T14:43:50.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f4066de2640 1 -- 172.21.15.26:0/390051077 wait complete. 2024-09-16T14:43:50.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f4066de2640 10 monclient: build_initial_monmap 2024-09-16T14:43:50.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.091+0000 7f4066de2640 10 monclient: monmap: 2024-09-16T14:43:50.103 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:50.104 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:50.104 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:43:50.104 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:43:50.104 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:50.104 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:43:50.104 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:50.104 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:50.104 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:50.105 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f4066de2640 1 Processor -- start 2024-09-16T14:43:50.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f4066de2640 1 -- start start 2024-09-16T14:43:50.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f4066de2640 10 monclient: init 2024-09-16T14:43:50.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f4066de2640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:50.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f4066de2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:43:50.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f4066de2640 10 monclient(hunting): picked mon.noname-b con 0x7f4040010c40 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:50.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f4066de2640 10 monclient(hunting): picked mon.noname-c con 0x7f4040012d70 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:50.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f4066de2640 10 monclient(hunting): picked mon.noname-a con 0x7f406009d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:50.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f4066de2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40600a2680 con 0x7f406009d1b0 2024-09-16T14:43:50.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f4066de2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40601308a0 con 0x7f4040010c40 2024-09-16T14:43:50.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f4066de2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40601219f0 con 0x7f4040012d70 2024-09-16T14:43:50.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f4066de2640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:50.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f4066de2640 10 monclient(hunting): authenticate will time out at 1054.217020s 2024-09-16T14:43:50.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f406537f640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f406009d1b0 legacy=0x7f40601d1ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:50.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f405ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4040012d70 0x7f4060133180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50960/0 (socket says 172.21.15.26:50960) 2024-09-16T14:43:50.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f405ffff640 1 -- 172.21.15.26:0/2572457930 learned_addr learned my addr 172.21.15.26:0/2572457930 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:50.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f405d7fa640 1 -- 172.21.15.26:0/2572457930 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872509930 0 0) 0x7f40601219f0 con 0x7f4040012d70 2024-09-16T14:43:50.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f405d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:43:50.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f405d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:50.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f405d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:50.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f405d7fa640 10 monclient(hunting): my global_id is 4529 2024-09-16T14:43:50.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f405d7fa640 1 -- 172.21.15.26:0/2572457930 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4040011830 con 0x7f4040012d70 2024-09-16T14:43:50.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f405d7fa640 1 -- 172.21.15.26:0/2572457930 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1210549099 0 0) 0x7f4040011830 con 0x7f4040012d70 2024-09-16T14:43:50.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f405d7fa640 1 -- 172.21.15.26:0/2572457930 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40601219f0 con 0x7f4040012d70 2024-09-16T14:43:50.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f405d7fa640 1 -- 172.21.15.26:0/2572457930 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f40500042b0 con 0x7f4040012d70 2024-09-16T14:43:50.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f405d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:50.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f405d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:50.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f405d7fa640 10 monclient(hunting): dump: 2024-09-16T14:43:50.107 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:50.107 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.107 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.107 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.107 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:50.107 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.107 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:50.107 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:50.107 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:50.108 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.095+0000 7f405d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:50.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 1 -- 172.21.15.26:0/2572457930 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4031898590 0 0) 0x7f40601219f0 con 0x7f4040012d70 2024-09-16T14:43:50.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 1 -- 172.21.15.26:0/2572457930 >> v1:172.21.15.189:6789/0 conn(0x7f4040010c40 legacy=0x7f4060132a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 1 -- 172.21.15.26:0/2572457930 >> v1:172.21.15.26:6789/0 conn(0x7f406009d1b0 legacy=0x7f40601d1ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:50.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:43:50.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 1 monclient: found mon.c 2024-09-16T14:43:50.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:50.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:50.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 1 -- 172.21.15.26:0/2572457930 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4060133890 con 0x7f4040012d70 2024-09-16T14:43:50.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:43:50.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:43:50.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:50.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f4066de2640 5 monclient: authenticate success, global_id 4529 2024-09-16T14:43:50.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f4066de2640 10 monclient: _renew_subs 2024-09-16T14:43:50.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f4066de2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:50.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f4066de2640 1 -- 172.21.15.26:0/2572457930 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f406009a240 con 0x7f4040012d70 2024-09-16T14:43:50.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f4066de2640 10 monclient: _renew_subs 2024-09-16T14:43:50.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f4066de2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:50.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f4066de2640 1 -- 172.21.15.26:0/2572457930 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40601205a0 con 0x7f4040012d70 2024-09-16T14:43:50.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 1 -- 172.21.15.26:0/2572457930 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4050004780 con 0x7f4040012d70 2024-09-16T14:43:50.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:50.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 1 -- 172.21.15.26:0/2572457930 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4050004a90 con 0x7f4040012d70 2024-09-16T14:43:50.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:50.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:50.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405effd640 4 set_mon_vals no callback set 2024-09-16T14:43:50.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 10 monclient: dump: 2024-09-16T14:43:50.110 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:50.110 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.110 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.110 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.110 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:50.110 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.110 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:50.110 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:50.110 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:50.110 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f4066de2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:50.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 1 -- 172.21.15.26:0/2572457930 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4050004c70 con 0x7f4040012d70 2024-09-16T14:43:50.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f405d7fa640 1 -- 172.21.15.26:0/2572457930 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(20..20 src has 1..20) v4 ==== 3889+0+0 (unknown 1099413130 0 0) 0x7f4050043280 con 0x7f4040012d70 2024-09-16T14:43:50.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f4066de2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:50.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f4066de2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:50.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f4066de2640 1 -- 172.21.15.26:0/2572457930 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4044005050 con 0x7f4040012d70 2024-09-16T14:43:50.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.099+0000 7f4064b7e640 10 monclient: get_auth_request con 0x7f40400379d0 auth_method 0 2024-09-16T14:43:50.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.103+0000 7f405d7fa640 1 -- 172.21.15.26:0/2572457930 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4050014740 con 0x7f4040012d70 2024-09-16T14:43:50.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.103+0000 7f405d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:50.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.103+0000 7f405d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:50.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.331+0000 7f4066de2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:43:50.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.331+0000 7f4066de2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:43:50.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.331+0000 7f4066de2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:50.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.331+0000 7f4066de2640 1 -- 172.21.15.26:0/2572457930 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4044005880 con 0x7f4040012d70 2024-09-16T14:43:50.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.331+0000 7f405d7fa640 1 -- 172.21.15.26:0/2572457930 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3645944093) 0x7f405001cd90 con 0x7f4040012d70 2024-09-16T14:43:50.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.331+0000 7f405d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:43:50.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.331+0000 7f405d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:43:50.336 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":20,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:43:50.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.335+0000 7f4066de2640 1 -- 172.21.15.26:0/2572457930 >> v1:172.21.15.189:6812/3196161582 conn(0x7f40400379d0 legacy=0x7f4040039e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.335+0000 7f4066de2640 10 monclient: shutdown 2024-09-16T14:43:50.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.335+0000 7f4066de2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:50.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.335+0000 7f4066de2640 1 -- 172.21.15.26:0/2572457930 >> v1:172.21.15.26:6790/0 conn(0x7f4040012d70 legacy=0x7f4060133180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.335+0000 7f4066de2640 1 -- 172.21.15.26:0/2572457930 shutdown_connections 2024-09-16T14:43:50.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.335+0000 7f4066de2640 1 -- 172.21.15.26:0/2572457930 >> 172.21.15.26:0/2572457930 conn(0x7f406009fb50 msgr2=0x7f40600a1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:50.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.335+0000 7f4066de2640 1 -- 172.21.15.26:0/2572457930 shutdown_connections 2024-09-16T14:43:50.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.335+0000 7f4066de2640 1 -- 172.21.15.26:0/2572457930 wait complete. 2024-09-16T14:43:50.352 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:43:50.352 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:43:50.353 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:43:50.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 10 monclient: get_monmap_and_config 2024-09-16T14:43:50.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 10 monclient: build_initial_monmap 2024-09-16T14:43:50.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 10 monclient: monmap: 2024-09-16T14:43:50.465 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:50.466 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.466 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:43:50.465904+0000 2024-09-16T14:43:50.466 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:43:50.465904+0000 2024-09-16T14:43:50.466 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:50.466 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.466 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:50.466 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:50.466 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:50.466 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 1 Processor -- start 2024-09-16T14:43:50.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 1 -- start start 2024-09-16T14:43:50.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 10 monclient: init 2024-09-16T14:43:50.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:50.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:43:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 10 monclient(hunting): picked mon.noname-b con 0x7fbad809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 10 monclient(hunting): picked mon.noname-c con 0x7fbad8132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 10 monclient(hunting): picked mon.noname-a con 0x7fbad8138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad809d260 con 0x7fbad8138ff0 2024-09-16T14:43:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad8132a80 con 0x7fbad8132650 2024-09-16T14:43:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad809cf60 con 0x7fbad809cb80 2024-09-16T14:43:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 10 monclient(hunting): authenticate will time out at 1054.586016s 2024-09-16T14:43:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad77fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fbad8138ff0 legacy=0x7fbad813b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad6ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbad809cb80 0x7fbad80717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50966/0 (socket says 172.21.15.26:50966) 2024-09-16T14:43:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad6ffd640 1 -- 172.21.15.26:0/1960980812 learned_addr learned my addr 172.21.15.26:0/1960980812 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:50.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 1 -- 172.21.15.26:0/1960980812 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637643070 0 0) 0x7fbad809cf60 con 0x7fbad809cb80 2024-09-16T14:43:50.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:50.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:50.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:50.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient(hunting): my global_id is 4532 2024-09-16T14:43:50.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 1 -- 172.21.15.26:0/1960980812 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbac4003530 con 0x7fbad809cb80 2024-09-16T14:43:50.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 1 -- 172.21.15.26:0/1960980812 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808648466 0 0) 0x7fbad8132a80 con 0x7fbad8132650 2024-09-16T14:43:50.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:50.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:50.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:50.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient(hunting): my global_id is 4552 2024-09-16T14:43:50.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 1 -- 172.21.15.26:0/1960980812 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad809cf60 con 0x7fbad8132650 2024-09-16T14:43:50.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 1 -- 172.21.15.26:0/1960980812 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2290765802 0 0) 0x7fbac4003530 con 0x7fbad809cb80 2024-09-16T14:43:50.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 1 -- 172.21.15.26:0/1960980812 >> v1:172.21.15.189:6789/0 conn(0x7fbad8132650 legacy=0x7fbad81378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 1 -- 172.21.15.26:0/1960980812 >> v1:172.21.15.26:6789/0 conn(0x7fbad8138ff0 legacy=0x7fbad813b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:50.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:43:50.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 1 monclient: found mon.noname-b 2024-09-16T14:43:50.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:50.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:43:50.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 1 -- 172.21.15.26:0/1960980812 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad809f2d0 con 0x7fbad809cb80 2024-09-16T14:43:50.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient: _finish_auth 0 2024-09-16T14:43:50.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:43:50.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:50.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 1 -- 172.21.15.26:0/1960980812 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbac8001660 con 0x7fbad809cb80 2024-09-16T14:43:50.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:50.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:50.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient: dump: 2024-09-16T14:43:50.472 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:50.472 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.472 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.472 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.472 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:50.472 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.472 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:50.472 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:50.472 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:50.472 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 5 monclient: authenticate success, global_id 4532 2024-09-16T14:43:50.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:50.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 1 -- 172.21.15.26:0/1960980812 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbac8001880 con 0x7fbad809cb80 2024-09-16T14:43:50.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:50.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 1 -- 172.21.15.26:0/1960980812 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbac8001c10 con 0x7fbad809cb80 2024-09-16T14:43:50.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:50.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 4 set_mon_vals no callback set 2024-09-16T14:43:50.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:50.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:50.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbad67fc640 10 monclient: dump: 2024-09-16T14:43:50.473 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:50.473 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.473 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.474 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.474 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:50.474 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.474 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:50.474 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:50.474 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:50.474 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 10 monclient: shutdown 2024-09-16T14:43:50.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:50.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 1 -- 172.21.15.26:0/1960980812 >> v1:172.21.15.26:6790/0 conn(0x7fbad809cb80 legacy=0x7fbad80717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 1 -- 172.21.15.26:0/1960980812 shutdown_connections 2024-09-16T14:43:50.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 1 -- 172.21.15.26:0/1960980812 >> 172.21.15.26:0/1960980812 conn(0x7fbad809fb50 msgr2=0x7fbad809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:50.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 1 -- 172.21.15.26:0/1960980812 shutdown_connections 2024-09-16T14:43:50.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.463+0000 7fbadd5bc640 1 -- 172.21.15.26:0/1960980812 wait complete. 2024-09-16T14:43:50.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 10 monclient: build_initial_monmap 2024-09-16T14:43:50.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 10 monclient: monmap: 2024-09-16T14:43:50.480 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:50.480 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:50.480 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:43:50.480 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:43:50.480 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:50.481 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:43:50.481 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:50.481 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:50.481 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:50.481 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 1 Processor -- start 2024-09-16T14:43:50.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 1 -- start start 2024-09-16T14:43:50.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 10 monclient: init 2024-09-16T14:43:50.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:50.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:43:50.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 10 monclient(hunting): picked mon.noname-a con 0x7fbad809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:50.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 10 monclient(hunting): picked mon.noname-c con 0x7fbad8132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:50.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 10 monclient(hunting): picked mon.noname-b con 0x7fbad8138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:50.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad81388d0 con 0x7fbad809cb80 2024-09-16T14:43:50.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad809a240 con 0x7fbad8138ff0 2024-09-16T14:43:50.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad813c720 con 0x7fbad8132650 2024-09-16T14:43:50.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:50.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 10 monclient(hunting): authenticate will time out at 1054.589048s 2024-09-16T14:43:50.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbad6ffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fbad809cb80 legacy=0x7fbad812dd00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:50.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbad8132650 0x7fbad812e410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50982/0 (socket says 172.21.15.26:50982) 2024-09-16T14:43:50.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacffff640 1 -- 172.21.15.26:0/4252663212 learned_addr learned my addr 172.21.15.26:0/4252663212 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:50.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633631576 0 0) 0x7fbad813c720 con 0x7fbad8132650 2024-09-16T14:43:50.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:50.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:50.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:50.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient(hunting): my global_id is 4535 2024-09-16T14:43:50.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab8003530 con 0x7fbad8132650 2024-09-16T14:43:50.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 541517151 0 0) 0x7fbab8003530 con 0x7fbad8132650 2024-09-16T14:43:50.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbad813c720 con 0x7fbad8132650 2024-09-16T14:43:50.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbac0001920 con 0x7fbad8132650 2024-09-16T14:43:50.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:50.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:50.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient(hunting): dump: 2024-09-16T14:43:50.483 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:50.483 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.483 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.483 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.483 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:50.483 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.484 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:50.484 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:50.484 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:50.484 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:50.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3389450881 0 0) 0x7fbad813c720 con 0x7fbad8132650 2024-09-16T14:43:50.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 >> v1:172.21.15.189:6789/0 conn(0x7fbad8138ff0 legacy=0x7fbad81d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 >> v1:172.21.15.26:6789/0 conn(0x7fbad809cb80 legacy=0x7fbad812dd00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:50.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:43:50.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 monclient: found mon.c 2024-09-16T14:43:50.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:50.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:50.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad8135010 con 0x7fbad8132650 2024-09-16T14:43:50.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient: _finish_auth 0 2024-09-16T14:43:50.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:43:50.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:50.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 5 monclient: authenticate success, global_id 4535 2024-09-16T14:43:50.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 10 monclient: _renew_subs 2024-09-16T14:43:50.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:50.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 1 -- 172.21.15.26:0/4252663212 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad8134770 con 0x7fbad8132650 2024-09-16T14:43:50.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 10 monclient: _renew_subs 2024-09-16T14:43:50.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:50.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 1 -- 172.21.15.26:0/4252663212 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad8134440 con 0x7fbad8132650 2024-09-16T14:43:50.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbac0001d30 con 0x7fbad8132650 2024-09-16T14:43:50.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:50.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbac0003a70 con 0x7fbad8132650 2024-09-16T14:43:50.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:50.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:50.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient: dump: 2024-09-16T14:43:50.486 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:50.486 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.486 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.486 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.486 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:50.486 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.486 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:50.486 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:50.486 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:50.486 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbad5ffb640 4 set_mon_vals no callback set 2024-09-16T14:43:50.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbac0003bd0 con 0x7fbad8132650 2024-09-16T14:43:50.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:50.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbadd5bc640 1 -- 172.21.15.26:0/4252663212 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbabc006590 con 0x7fbabc002830 2024-09-16T14:43:50.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(20..20 src has 1..20) v4 ==== 3889+0+0 (unknown 1099413130 0 0) 0x7fbac0043470 con 0x7fbad8132650 2024-09-16T14:43:50.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429076517 0 0) 0x7fbabc006590 con 0x7fbabc002830 2024-09-16T14:43:50.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:50.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:50.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:50.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient(hunting): my global_id is 4538 2024-09-16T14:43:50.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad813c720 con 0x7fbabc002830 2024-09-16T14:43:50.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2530570662 0 0) 0x7fbad813c720 con 0x7fbabc002830 2024-09-16T14:43:50.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbabc006590 con 0x7fbabc002830 2024-09-16T14:43:50.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbac8001c60 con 0x7fbabc002830 2024-09-16T14:43:50.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbad77fe640 10 monclient: get_auth_request con 0x7fbab8035e60 auth_method 0 2024-09-16T14:43:50.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:50.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:50.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient: dump: 2024-09-16T14:43:50.488 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:50.488 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.488 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.488 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.488 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:50.489 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.489 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:50.489 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:50.489 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:50.489 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1094076611 0 0) 0x7fbabc006590 con 0x7fbabc002830 2024-09-16T14:43:50.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbabc00a0e0 con 0x7fbabc002830 2024-09-16T14:43:50.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fbac80015d0 con 0x7fbabc002830 2024-09-16T14:43:50.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:50.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:50.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 >> v1:172.21.15.26:6790/0 conn(0x7fbabc002830 legacy=0x7fbabc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.467+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 >> v1:172.21.15.26:6790/0 conn(0x7fbabc002830 legacy=0x7fbabc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:50.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbadd5bc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:43:50.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbadd5bc640 1 -- 172.21.15.26:0/4252663212 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbabc009d80 con 0x7fbabc0090a0 2024-09-16T14:43:50.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938293485 0 0) 0x7fbabc009d80 con 0x7fbabc0090a0 2024-09-16T14:43:50.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:50.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:50.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:50.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 10 monclient(hunting): my global_id is 4541 2024-09-16T14:43:50.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbabc006590 con 0x7fbabc0090a0 2024-09-16T14:43:50.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1401167923 0 0) 0x7fbabc006590 con 0x7fbabc0090a0 2024-09-16T14:43:50.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbabc009d80 con 0x7fbabc0090a0 2024-09-16T14:43:50.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbac8004c80 con 0x7fbabc0090a0 2024-09-16T14:43:50.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:50.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:50.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 10 monclient: dump: 2024-09-16T14:43:50.491 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:50.491 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.491 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.491 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.491 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:50.491 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.491 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:50.491 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:50.492 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:50.492 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1795418341 0 0) 0x7fbabc009d80 con 0x7fbabc0090a0 2024-09-16T14:43:50.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbabc00c290 con 0x7fbabc0090a0 2024-09-16T14:43:50.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4036 (unknown 0 0 3621712229) 0x7fbac8004fc0 con 0x7fbabc0090a0 2024-09-16T14:43:50.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:43:50.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:43:50.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 >> v1:172.21.15.26:6790/0 conn(0x7fbabc0090a0 legacy=0x7fbabc009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbacf7fe640 1 -- 172.21.15.26:0/4252663212 >> v1:172.21.15.26:6790/0 conn(0x7fbabc0090a0 legacy=0x7fbabc009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:50.493 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:43:50.493 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:43:50.493 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:43:50.493 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:43:50.493 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 20, 2024-09-16T14:43:50.493 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:43:50.493 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:43:50.493 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:43:50.494 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:50.494 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:43:50.494 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:43:50.494 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:43:50.494 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:43:50.494 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:50.494 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:50.495 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:50.495 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:50.495 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:50.495 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:50.495 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:50.495 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:50.495 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:50.495 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:50.495 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:50.495 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:43:50.495 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:43:50.495 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:50.496 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:50.496 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:50.496 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:50.496 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:50.496 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:50.496 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:50.496 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:50.496 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:50.496 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:50.496 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:50.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.496 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:43:50.496 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:43:50.497 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:43:50.497 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:43:50.497 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:43:50.497 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:43:50.497 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:43:50.497 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:43:50.497 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:43:50.497 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:43:50.497 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:43:50.497 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:43:50.498 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:43:50.498 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:43:50.498 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:43:50.498 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:43:50.498 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:43:50.498 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:50.498 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:50.498 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:50.498 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:50.498 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:50.498 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:50.498 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:50.498 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:50.498 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:50.499 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:50.499 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:50.499 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:43:50.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.499 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:43:50.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.499 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:43:50.499 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:43:50.499 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:50.500 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:50.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.500 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:50.500 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:43:50.500 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:50.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:50.500 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:50.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.500 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:43:50.500 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:43:50.500 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:50.500 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:50.500 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:50.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.501 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:43:50.501 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:43:50.501 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:50.501 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:50.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.501 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:50.501 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:43:50.501 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:50.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:50.501 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:50.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.501 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:43:50.502 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:43:50.502 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:50.502 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:50.502 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:50.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.502 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:43:50.502 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:43:50.502 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:50.503 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:50.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.503 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:50.503 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:43:50.503 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:50.503 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:50.503 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:50.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.503 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:43:50.503 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:43:50.503 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:50.503 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:50.503 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:50.503 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:50.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:50.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.504 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:43:50.504 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:43:50.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.504 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:50.504 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:50.504 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:43:50.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:50.504 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:50.504 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:43:50.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.504 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:50.505 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:50.505 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:43:50.505 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:50.506 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:50.506 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:43:50.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.506 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:50.506 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:50.506 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:43:50.506 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:50.506 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:50.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.506 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:43:50.507 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:43:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbadd5bc640 1 -- 172.21.15.26:0/4252663212 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbab8035e60 legacy=0x7fbab8038320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbadd5bc640 10 monclient: shutdown 2024-09-16T14:43:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbadd5bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbadd5bc640 1 -- 172.21.15.26:0/4252663212 >> v1:172.21.15.26:6790/0 conn(0x7fbad8132650 legacy=0x7fbad812e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbad77fe640 1 -- 172.21.15.26:0/4252663212 reap_dead start 2024-09-16T14:43:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbad77fe640 1 -- 172.21.15.26:0/4252663212 reap_dead start 2024-09-16T14:43:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbadd5bc640 1 -- 172.21.15.26:0/4252663212 shutdown_connections 2024-09-16T14:43:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbadd5bc640 1 -- 172.21.15.26:0/4252663212 >> 172.21.15.26:0/4252663212 conn(0x7fbad809fb50 msgr2=0x7fbad809be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbadd5bc640 1 -- 172.21.15.26:0/4252663212 shutdown_connections 2024-09-16T14:43:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.475+0000 7fbadd5bc640 1 -- 172.21.15.26:0/4252663212 wait complete. 2024-09-16T14:43:50.509 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:43:50.509 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T14:43:50.510 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:43:50.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:50.591+0000 7f2f17fff640 10 monclient: tick 2024-09-16T14:43:50.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:50.591+0000 7f2f17fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:50.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:50.591+0000 7f2f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:50.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.615+0000 7f33ec2dc640 10 monclient: get_monmap_and_config 2024-09-16T14:43:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.615+0000 7f33ec2dc640 10 monclient: build_initial_monmap 2024-09-16T14:43:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.615+0000 7f33ec2dc640 10 monclient: monmap: 2024-09-16T14:43:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:43:50.621407+0000 2024-09-16T14:43:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:43:50.621407+0000 2024-09-16T14:43:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:50.621 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 1 Processor -- start 2024-09-16T14:43:50.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 1 -- start start 2024-09-16T14:43:50.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 10 monclient: init 2024-09-16T14:43:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:43:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 10 monclient(hunting): picked mon.noname-c con 0x7f33e409a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 10 monclient(hunting): picked mon.noname-a con 0x7f33e409d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 10 monclient(hunting): picked mon.noname-b con 0x7f33e409b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e409dc60 con 0x7f33e409d1b0 2024-09-16T14:43:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e4099ae0 con 0x7f33e409a550 2024-09-16T14:43:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e409b8f0 con 0x7f33e409b040 2024-09-16T14:43:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 10 monclient(hunting): authenticate will time out at 1054.741283s 2024-09-16T14:43:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9877640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f33e409d1b0 legacy=0x7f33e4136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ea879640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f33e409b040 0x7f33e413a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51014/0 (socket says 172.21.15.26:51014) 2024-09-16T14:43:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ea879640 1 -- 172.21.15.26:0/4277431879 learned_addr learned my addr 172.21.15.26:0/4277431879 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 1 -- 172.21.15.26:0/4277431879 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204736191 0 0) 0x7f33e409b8f0 con 0x7f33e409b040 2024-09-16T14:43:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 10 monclient(hunting): _negotiate 2024-09-16T14:43:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 10 monclient(hunting): my global_id is 4544 2024-09-16T14:43:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 1 -- 172.21.15.26:0/4277431879 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33d0003530 con 0x7f33e409b040 2024-09-16T14:43:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 1 -- 172.21.15.26:0/4277431879 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3479867102 0 0) 0x7f33d0003530 con 0x7f33e409b040 2024-09-16T14:43:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 1 -- 172.21.15.26:0/4277431879 >> v1:172.21.15.189:6789/0 conn(0x7f33e409a550 legacy=0x7f33e409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 1 -- 172.21.15.26:0/4277431879 >> v1:172.21.15.26:6789/0 conn(0x7f33e409d1b0 legacy=0x7f33e4136a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 10 monclient: _finish_hunting 0 2024-09-16T14:43:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 1 monclient: found mon.noname-b 2024-09-16T14:43:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:43:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 1 -- 172.21.15.26:0/4277431879 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e409bbd0 con 0x7f33e409b040 2024-09-16T14:43:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 10 monclient: _finish_auth 0 2024-09-16T14:43:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 10 monclient: _check_auth_tickets 2024-09-16T14:43:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 1 -- 172.21.15.26:0/4277431879 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f33dc0019a0 con 0x7f33e409b040 2024-09-16T14:43:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 5 monclient: authenticate success, global_id 4544 2024-09-16T14:43:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 10 monclient: dump: 2024-09-16T14:43:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:50.632 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 1 -- 172.21.15.26:0/4277431879 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33dc001cc0 con 0x7f33e409b040 2024-09-16T14:43:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 4 set_mon_vals no callback set 2024-09-16T14:43:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 1 -- 172.21.15.26:0/4277431879 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f33dc002090 con 0x7f33e409b040 2024-09-16T14:43:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33e9076640 10 monclient: dump: 2024-09-16T14:43:50.633 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:50.633 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.633 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.633 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.633 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:50.633 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.633 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:50.633 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:50.633 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:50.633 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 10 monclient: shutdown 2024-09-16T14:43:50.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:50.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 1 -- 172.21.15.26:0/4277431879 >> v1:172.21.15.26:6790/0 conn(0x7f33e409b040 legacy=0x7f33e413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 1 -- 172.21.15.26:0/4277431879 shutdown_connections 2024-09-16T14:43:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 1 -- 172.21.15.26:0/4277431879 >> 172.21.15.26:0/4277431879 conn(0x7f33e409fb50 msgr2=0x7f33e409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 1 -- 172.21.15.26:0/4277431879 shutdown_connections 2024-09-16T14:43:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 1 -- 172.21.15.26:0/4277431879 wait complete. 2024-09-16T14:43:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 10 monclient: build_initial_monmap 2024-09-16T14:43:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 10 monclient: monmap: 2024-09-16T14:43:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:43:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:43:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:43:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:43:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:50.634 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:50.635 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 1 Processor -- start 2024-09-16T14:43:50.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 1 -- start start 2024-09-16T14:43:50.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.619+0000 7f33ec2dc640 10 monclient: init 2024-09-16T14:43:50.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:50.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:43:50.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 10 monclient(hunting): picked mon.noname-b con 0x7f33e409a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:50.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 10 monclient(hunting): picked mon.noname-a con 0x7f33e409b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:50.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 10 monclient(hunting): picked mon.noname-c con 0x7f33e409d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:50.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e41388a0 con 0x7f33e409b040 2024-09-16T14:43:50.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33e9877640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f33e409b040 legacy=0x7f33e41caf00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:50.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e4137cd0 con 0x7f33e409a550 2024-09-16T14:43:50.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e4139800 con 0x7f33e409d1b0 2024-09-16T14:43:50.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:50.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 10 monclient(hunting): authenticate will time out at 1054.744257s 2024-09-16T14:43:50.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ea879640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f33e409d1b0 0x7f33e41ce630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51026/0 (socket says 172.21.15.26:51026) 2024-09-16T14:43:50.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ea879640 1 -- 172.21.15.26:0/3292381225 learned_addr learned my addr 172.21.15.26:0/3292381225 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:43:50.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081197124 0 0) 0x7f33e4139800 con 0x7f33e409d1b0 2024-09-16T14:43:50.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient(hunting): _negotiate 2024-09-16T14:43:50.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:50.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:50.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient(hunting): my global_id is 4547 2024-09-16T14:43:50.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33c8003560 con 0x7f33e409d1b0 2024-09-16T14:43:50.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3891659644 0 0) 0x7f33c8003560 con 0x7f33e409d1b0 2024-09-16T14:43:50.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33e4139800 con 0x7f33e409d1b0 2024-09-16T14:43:50.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f33dc001960 con 0x7f33e409d1b0 2024-09-16T14:43:50.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:50.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:50.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient(hunting): dump: 2024-09-16T14:43:50.637 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:50.637 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.637 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.637 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.637 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:50.637 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.637 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:50.637 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:50.637 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:50.637 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:50.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782799812 0 0) 0x7f33e4137cd0 con 0x7f33e409a550 2024-09-16T14:43:50.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient(hunting): _negotiate 2024-09-16T14:43:50.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:50.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:50.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient(hunting): my global_id is 4561 2024-09-16T14:43:50.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33c8003560 con 0x7f33e409a550 2024-09-16T14:43:50.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1195560314 0 0) 0x7f33e4139800 con 0x7f33e409d1b0 2024-09-16T14:43:50.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 >> v1:172.21.15.189:6789/0 conn(0x7f33e409a550 legacy=0x7f33e4130d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 >> v1:172.21.15.26:6789/0 conn(0x7f33e409b040 legacy=0x7f33e41caf00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:50.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient: _finish_hunting 0 2024-09-16T14:43:50.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 monclient: found mon.c 2024-09-16T14:43:50.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:50.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:50.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e4139ae0 con 0x7f33e409d1b0 2024-09-16T14:43:50.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient: _finish_auth 0 2024-09-16T14:43:50.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient: _check_auth_tickets 2024-09-16T14:43:50.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:50.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 5 monclient: authenticate success, global_id 4547 2024-09-16T14:43:50.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 10 monclient: _renew_subs 2024-09-16T14:43:50.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:50.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 1 -- 172.21.15.26:0/3292381225 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e4138ae0 con 0x7f33e409d1b0 2024-09-16T14:43:50.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 10 monclient: _renew_subs 2024-09-16T14:43:50.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:43:50.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 1 -- 172.21.15.26:0/3292381225 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e4131620 con 0x7f33e409d1b0 2024-09-16T14:43:50.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33dc005070 con 0x7f33e409d1b0 2024-09-16T14:43:50.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:50.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f33dc005400 con 0x7f33e409d1b0 2024-09-16T14:43:50.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:50.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:43:50.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33e8875640 4 set_mon_vals no callback set 2024-09-16T14:43:50.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient: dump: 2024-09-16T14:43:50.640 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:50.641 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.641 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.641 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.641 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:50.641 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.641 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:50.641 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:50.641 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:50.641 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:50.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33ec2dc640 1 -- 172.21.15.26:0/3292381225 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f33cc006590 con 0x7f33cc002830 2024-09-16T14:43:50.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f33dc005560 con 0x7f33e409d1b0 2024-09-16T14:43:50.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(20..20 src has 1..20) v4 ==== 3889+0+0 (unknown 1099413130 0 0) 0x7f33dc043850 con 0x7f33e409d1b0 2024-09-16T14:43:50.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33e9877640 10 monclient: get_auth_request con 0x7f33c8036200 auth_method 0 2024-09-16T14:43:50.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163469143 0 0) 0x7f33cc006590 con 0x7f33cc002830 2024-09-16T14:43:50.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient(hunting): _negotiate 2024-09-16T14:43:50.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:50.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:50.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient(hunting): my global_id is 4564 2024-09-16T14:43:50.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e4139800 con 0x7f33cc002830 2024-09-16T14:43:50.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3313935030 0 0) 0x7f33e4139800 con 0x7f33cc002830 2024-09-16T14:43:50.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33cc006590 con 0x7f33cc002830 2024-09-16T14:43:50.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f33e00017a0 con 0x7f33cc002830 2024-09-16T14:43:50.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:50.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:50.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 10 monclient: dump: 2024-09-16T14:43:50.643 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:50.643 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.643 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.643 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.643 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:50.643 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.643 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:50.643 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:50.644 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:50.644 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4270672507 0 0) 0x7f33cc006590 con 0x7f33cc002830 2024-09-16T14:43:50.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.623+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33cc00a0e0 con 0x7f33cc002830 2024-09-16T14:43:50.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.627+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f33e0001b50 con 0x7f33cc002830 2024-09-16T14:43:50.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.627+0000 7f33daffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:50.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.627+0000 7f33daffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:50.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.627+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 >> v1:172.21.15.189:6789/0 conn(0x7f33cc002830 legacy=0x7f33cc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.627+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 >> v1:172.21.15.189:6789/0 conn(0x7f33cc002830 legacy=0x7f33cc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:50.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.631+0000 7f33ec2dc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:43:50.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.631+0000 7f33ec2dc640 1 -- 172.21.15.26:0/3292381225 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f33cc009d80 con 0x7f33cc0090a0 2024-09-16T14:43:50.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.631+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798933351 0 0) 0x7f33cc009d80 con 0x7f33cc0090a0 2024-09-16T14:43:50.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.631+0000 7f33daffd640 10 monclient(hunting): _negotiate 2024-09-16T14:43:50.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.631+0000 7f33daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:50.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.631+0000 7f33daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:50.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.631+0000 7f33daffd640 10 monclient(hunting): my global_id is 4567 2024-09-16T14:43:50.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.631+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33cc006590 con 0x7f33cc0090a0 2024-09-16T14:43:50.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.631+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2034509723 0 0) 0x7f33cc006590 con 0x7f33cc0090a0 2024-09-16T14:43:50.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.631+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33cc009d80 con 0x7f33cc0090a0 2024-09-16T14:43:50.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.631+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f33e0003820 con 0x7f33cc0090a0 2024-09-16T14:43:50.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.631+0000 7f33daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:50.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.631+0000 7f33daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.631+0000 7f33daffd640 10 monclient: dump: 2024-09-16T14:43:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:43:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:43:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:50.647 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:43:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.631+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2818969306 0 0) 0x7f33cc009d80 con 0x7f33cc0090a0 2024-09-16T14:43:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.631+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f33cc00c290 con 0x7f33cc0090a0 2024-09-16T14:43:50.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.635+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4200 (unknown 0 0 572670662) 0x7f33e0005b00 con 0x7f33cc0090a0 2024-09-16T14:43:50.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.635+0000 7f33daffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:43:50.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.635+0000 7f33daffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:43:50.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.635+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 >> v1:172.21.15.189:6789/0 conn(0x7f33cc0090a0 legacy=0x7f33cc009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.635+0000 7f33daffd640 1 -- 172.21.15.26:0/3292381225 >> v1:172.21.15.189:6789/0 conn(0x7f33cc0090a0 legacy=0x7f33cc009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:43:50.648 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:43:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:43:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:43:50.648 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:43:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 20, 2024-09-16T14:43:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:43:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:43:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:43:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:43:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:43:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:43:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:43:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:50.649 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:43:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:43:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:50.650 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:43:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:43:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:43:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:43:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:43:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:43:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:43:50.651 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:43:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:43:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:43:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:43:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:43:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:43:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:43:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:43:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:43:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:43:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:43:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:43:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:43:50.652 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:43:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:43:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:43:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:43:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:43:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:43:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:43:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:43:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:43:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:43:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:43:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:50.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:43:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:43:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:43:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:43:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:43:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:43:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:43:50.655 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:43:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:43:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:43:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:43:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:43:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:43:50.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:43:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:43:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:43:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:43:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:43:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:43:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:43:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:43:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:43:50.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:43:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:43:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:43:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:43:50.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:43:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:43:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:43:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:43:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:43:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:43:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:43:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:43:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:43:50.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:43:50.660 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:43:50.660 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:43:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.635+0000 7f33ec2dc640 1 -- 172.21.15.26:0/3292381225 >> v1:172.21.15.189:6812/3196161582 conn(0x7f33c8036200 legacy=0x7f33c80386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.635+0000 7f33ec2dc640 10 monclient: shutdown 2024-09-16T14:43:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.635+0000 7f33ec2dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.635+0000 7f33ec2dc640 1 -- 172.21.15.26:0/3292381225 >> v1:172.21.15.26:6790/0 conn(0x7f33e409d1b0 legacy=0x7f33e41ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.635+0000 7f33ea879640 1 -- 172.21.15.26:0/3292381225 reap_dead start 2024-09-16T14:43:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.635+0000 7f33ea879640 1 -- 172.21.15.26:0/3292381225 reap_dead start 2024-09-16T14:43:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.635+0000 7f33ec2dc640 1 -- 172.21.15.26:0/3292381225 shutdown_connections 2024-09-16T14:43:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.635+0000 7f33ec2dc640 1 -- 172.21.15.26:0/3292381225 >> 172.21.15.26:0/3292381225 conn(0x7f33e409fb50 msgr2=0x7f33e4134280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.635+0000 7f33ec2dc640 1 -- 172.21.15.26:0/3292381225 shutdown_connections 2024-09-16T14:43:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:43:50.635+0000 7f33ec2dc640 1 -- 172.21.15.26:0/3292381225 wait complete. 2024-09-16T14:43:50.662 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:43:50.662 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:43:51.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:51.071+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}]=0 set pool 3 pg_num_max to 33 v21) v1 ==== 145+0+0 (unknown 127810571 0 0) 0x7f2f28012e80 con 0x7f2f300a2580 2024-09-16T14:43:51.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:51.071+0000 7f2f2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T14:43:51.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:51.071+0000 7f2f2d7fa640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_max to 33 2024-09-16T14:43:51.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:51.167+0000 7f2f36d5e640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T14:43:51.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:51.167+0000 7f2f36d5e640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T14:43:51.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:51.167+0000 7f2f36d5e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:51.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:51.167+0000 7f2f36d5e640 1 -- 172.21.15.189:0/2826986803 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"} v 0) v1 -- 0x7f2f10005340 con 0x7f2f300a2580 2024-09-16T14:43:51.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:51.595+0000 7f2f17fff640 10 monclient: tick 2024-09-16T14:43:51.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:51.595+0000 7f2f17fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:51.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:51.595+0000 7f2f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:52.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.083+0000 7f2f2d7fa640 1 -- 172.21.15.189:0/2826986803 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}]=0 set pool 3 pg_num_max to 33 v22) v1 ==== 145+0+0 (unknown 1578579183 0 0) 0x7f2f28014600 con 0x7f2f300a2580 2024-09-16T14:43:52.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.083+0000 7f2f2d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T14:43:52.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.083+0000 7f2f2d7fa640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_max to 33 2024-09-16T14:43:52.088 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 pg_num_max to 33 2024-09-16T14:43:52.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.087+0000 7f2f36d5e640 1 -- 172.21.15.189:0/2826986803 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2f1c036460 legacy=0x7f2f1c038920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:52.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.087+0000 7f2f36d5e640 10 monclient: shutdown 2024-09-16T14:43:52.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.087+0000 7f2f36d5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:52.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.087+0000 7f2f36d5e640 1 -- 172.21.15.189:0/2826986803 >> v1:172.21.15.189:6789/0 conn(0x7f2f300a2580 legacy=0x7f2f301d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:52.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.087+0000 7f2f36d5e640 1 -- 172.21.15.189:0/2826986803 shutdown_connections 2024-09-16T14:43:52.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.087+0000 7f2f36d5e640 1 -- 172.21.15.189:0/2826986803 >> 172.21.15.189:0/2826986803 conn(0x7f2f300a7a70 msgr2=0x7f2f3006b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:52.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.087+0000 7f2f36d5e640 1 -- 172.21.15.189:0/2826986803 shutdown_connections 2024-09-16T14:43:52.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.087+0000 7f2f36d5e640 1 -- 172.21.15.189:0/2826986803 wait complete. 2024-09-16T14:43:52.112 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool get testpool1234 pg_num_max 2024-09-16T14:43:52.112 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 33 2024-09-16T14:43:52.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.227+0000 7f8705776640 10 monclient: get_monmap_and_config 2024-09-16T14:43:52.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.227+0000 7f8705776640 10 monclient: build_initial_monmap 2024-09-16T14:43:52.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.227+0000 7f8705776640 10 monclient: monmap: 2024-09-16T14:43:52.232 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:52.232 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:52.232 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:52.232637+0000 2024-09-16T14:43:52.232 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:52.232637+0000 2024-09-16T14:43:52.233 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:52.233 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:52.233 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:52.233 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:52.233 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:52.233 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.227+0000 7f8705776640 1 Processor -- start 2024-09-16T14:43:52.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.227+0000 7f8705776640 1 -- start start 2024-09-16T14:43:52.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.227+0000 7f8705776640 10 monclient: init 2024-09-16T14:43:52.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:52.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:43:52.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient(hunting): picked mon.noname-c con 0x7f870009df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:52.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient(hunting): picked mon.noname-a con 0x7f870009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:52.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient(hunting): picked mon.noname-b con 0x7f870009e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:52.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870009ee20 con 0x7f870009e2f0 2024-09-16T14:43:52.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870009f100 con 0x7f870009df10 2024-09-16T14:43:52.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870009f3e0 con 0x7f870009e740 2024-09-16T14:43:52.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:52.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient(hunting): authenticate will time out at 1014.348352s 2024-09-16T14:43:52.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86feffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f870009df10 0x7f870009d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51536/0 (socket says 172.21.15.189:51536) 2024-09-16T14:43:52.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86feffd640 1 -- 172.21.15.189:0/1252738970 learned_addr learned my addr 172.21.15.189:0/1252738970 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:52.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fe7fc640 1 -- 172.21.15.189:0/1252738970 >> v1:172.21.15.26:6789/0 conn(0x7f870009e2f0 legacy=0x7f87001378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:52.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 1 -- 172.21.15.189:0/1252738970 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413618778 0 0) 0x7f870009f100 con 0x7f870009df10 2024-09-16T14:43:52.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:52.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:52.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:52.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 10 monclient(hunting): my global_id is 4570 2024-09-16T14:43:52.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 1 -- 172.21.15.189:0/1252738970 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e8003530 con 0x7f870009df10 2024-09-16T14:43:52.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 1 -- 172.21.15.189:0/1252738970 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 683185029 0 0) 0x7f86e8003530 con 0x7f870009df10 2024-09-16T14:43:52.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 1 -- 172.21.15.189:0/1252738970 >> v1:172.21.15.26:6790/0 conn(0x7f870009e740 legacy=0x7f870013b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:52.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 1 -- 172.21.15.189:0/1252738970 >> v1:172.21.15.26:6789/0 conn(0x7f870009e2f0 legacy=0x7f87001378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:52.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 10 monclient: _finish_hunting 0 2024-09-16T14:43:52.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 1 monclient: found mon.noname-c 2024-09-16T14:43:52.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:52.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 1 -- 172.21.15.189:0/1252738970 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870009c690 con 0x7f870009df10 2024-09-16T14:43:52.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 10 monclient: _finish_auth 0 2024-09-16T14:43:52.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 10 monclient: _check_auth_tickets 2024-09-16T14:43:52.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:52.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 1 -- 172.21.15.189:0/1252738970 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f86f0001620 con 0x7f870009df10 2024-09-16T14:43:52.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 5 monclient: authenticate success, global_id 4570 2024-09-16T14:43:52.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:52.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:52.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:52.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 10 monclient: dump: 2024-09-16T14:43:52.239 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:52.239 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:52.240 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.240 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.240 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:52.240 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:52.240 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:52.240 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:52.240 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:52.240 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:52.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 1 -- 172.21.15.189:0/1252738970 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86f0001940 con 0x7f870009df10 2024-09-16T14:43:52.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:52.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 4 set_mon_vals no callback set 2024-09-16T14:43:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 1 -- 172.21.15.189:0/1252738970 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f86f0001cd0 con 0x7f870009df10 2024-09-16T14:43:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86fdffb640 10 monclient: dump: 2024-09-16T14:43:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:52.242 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:52.242 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient: shutdown 2024-09-16T14:43:52.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:52.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 1 -- 172.21.15.189:0/1252738970 >> v1:172.21.15.189:6789/0 conn(0x7f870009df10 legacy=0x7f870009d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:52.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 1 -- 172.21.15.189:0/1252738970 shutdown_connections 2024-09-16T14:43:52.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 1 -- 172.21.15.189:0/1252738970 >> 172.21.15.189:0/1252738970 conn(0x7f87000a3320 msgr2=0x7f870009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:52.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 1 -- 172.21.15.189:0/1252738970 shutdown_connections 2024-09-16T14:43:52.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 1 -- 172.21.15.189:0/1252738970 wait complete. 2024-09-16T14:43:52.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient: build_initial_monmap 2024-09-16T14:43:52.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient: monmap: 2024-09-16T14:43:52.242 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:52.242 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:52.243 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:52.243 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:52.243 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:52.243 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:52.243 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:52.243 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:52.243 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:52.243 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 1 Processor -- start 2024-09-16T14:43:52.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 1 -- start start 2024-09-16T14:43:52.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient: init 2024-09-16T14:43:52.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:52.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:43:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient(hunting): picked mon.noname-a con 0x7f870009df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient(hunting): picked mon.noname-c con 0x7f870009e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient(hunting): picked mon.noname-b con 0x7f870009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870013c050 con 0x7f870009df10 2024-09-16T14:43:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870006f940 con 0x7f870009e740 2024-09-16T14:43:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8700119da0 con 0x7f870009e2f0 2024-09-16T14:43:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f8705776640 10 monclient(hunting): authenticate will time out at 1014.351965s 2024-09-16T14:43:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.231+0000 7f86feffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f870009df10 legacy=0x7f8700139aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86ff7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f870009e740 0x7f87001cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51546/0 (socket says 172.21.15.189:51546) 2024-09-16T14:43:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86ff7fe640 1 -- 172.21.15.189:0/3214935164 learned_addr learned my addr 172.21.15.189:0/3214935164 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033572715 0 0) 0x7f870006f940 con 0x7f870009e740 2024-09-16T14:43:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient(hunting): my global_id is 4573 2024-09-16T14:43:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86ec003530 con 0x7f870009e740 2024-09-16T14:43:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3433350743 0 0) 0x7f86ec003530 con 0x7f870009e740 2024-09-16T14:43:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f870006f940 con 0x7f870009e740 2024-09-16T14:43:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f86f8001c50 con 0x7f870009e740 2024-09-16T14:43:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient(hunting): dump: 2024-09-16T14:43:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:52.249 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:52.249 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.249 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.249 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:52.249 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:52.249 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:52.249 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:52.249 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:52.249 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:52.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200731514 0 0) 0x7f8700119da0 con 0x7f870009e2f0 2024-09-16T14:43:52.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:52.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:52.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:52.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient(hunting): my global_id is 4553 2024-09-16T14:43:52.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86ec003530 con 0x7f870009e2f0 2024-09-16T14:43:52.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2532648262 0 0) 0x7f870006f940 con 0x7f870009e740 2024-09-16T14:43:52.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 >> v1:172.21.15.26:6790/0 conn(0x7f870009e2f0 legacy=0x7f870013a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:52.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 >> v1:172.21.15.26:6789/0 conn(0x7f870009df10 legacy=0x7f8700139aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:52.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:43:52.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 1 monclient: found mon.b 2024-09-16T14:43:52.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:52.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870006b380 con 0x7f870009e740 2024-09-16T14:43:52.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient: _finish_auth 0 2024-09-16T14:43:52.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:43:52.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:52.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f8705776640 5 monclient: authenticate success, global_id 4573 2024-09-16T14:43:52.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f8705776640 10 monclient: _renew_subs 2024-09-16T14:43:52.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f8705776640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f8705776640 1 -- 172.21.15.189:0/3214935164 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f870011a6a0 con 0x7f870009e740 2024-09-16T14:43:52.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f8705776640 10 monclient: _renew_subs 2024-09-16T14:43:52.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f8705776640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f8705776640 1 -- 172.21.15.189:0/3214935164 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f870006ce50 con 0x7f870009e740 2024-09-16T14:43:52.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86f8002020 con 0x7f870009e740 2024-09-16T14:43:52.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:52.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f86f8003e70 con 0x7f870009e740 2024-09-16T14:43:52.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:52.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86fd7fa640 4 set_mon_vals no callback set 2024-09-16T14:43:52.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:52.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 10 monclient: dump: 2024-09-16T14:43:52.252 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:52.252 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:52.252 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.252 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.252 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:52.253 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:52.253 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:52.253 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:52.253 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:52.253 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f8705776640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:52.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f8705776640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:52.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f8705776640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f8705776640 1 -- 172.21.15.189:0/3214935164 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e0005050 con 0x7f870009e740 2024-09-16T14:43:52.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f86f8003fd0 con 0x7f870009e740 2024-09-16T14:43:52.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3889+0+0 (unknown 2825553975 0 0) 0x7f86f80436e0 con 0x7f870009e740 2024-09-16T14:43:52.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.235+0000 7f86feffd640 10 monclient: get_auth_request con 0x7f86ec0361d0 auth_method 0 2024-09-16T14:43:52.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.239+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f86f8014b40 con 0x7f870009e740 2024-09-16T14:43:52.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.239+0000 7f86e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:52.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.239+0000 7f86e77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:52.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.391+0000 7f8705776640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T14:43:52.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.391+0000 7f8705776640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T14:43:52.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.391+0000 7f8705776640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.391+0000 7f8705776640 1 -- 172.21.15.189:0/3214935164 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7f86e0005880 con 0x7f870009e740 2024-09-16T14:43:52.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.391+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=0 v22) v1 ==== 105+0+15 (unknown 1016200791 0 2995009362) 0x7f86f8012fb0 con 0x7f870009e740 2024-09-16T14:43:52.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.391+0000 7f86e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T14:43:52.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.391+0000 7f86e77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:43:52.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.483+0000 7f8705776640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T14:43:52.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.483+0000 7f8705776640 10 monclient: _send_command 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T14:43:52.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.483+0000 7f8705776640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.483+0000 7f8705776640 1 -- 172.21.15.189:0/3214935164 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7f86e0005300 con 0x7f870009e740 2024-09-16T14:43:52.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.483+0000 7f86e77fe640 1 -- 172.21.15.189:0/3214935164 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=0 v22) v1 ==== 105+0+15 (unknown 1016200791 0 2995009362) 0x7f86f8014910 con 0x7f870009e740 2024-09-16T14:43:52.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.483+0000 7f86e77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T14:43:52.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.483+0000 7f86e77fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:43:52.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.483+0000 7f8705776640 1 -- 172.21.15.189:0/3214935164 >> v1:172.21.15.189:6812/3196161582 conn(0x7f86ec0361d0 legacy=0x7f86ec038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:52.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.483+0000 7f8705776640 10 monclient: shutdown 2024-09-16T14:43:52.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.483+0000 7f8705776640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:52.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.483+0000 7f8705776640 1 -- 172.21.15.189:0/3214935164 >> v1:172.21.15.189:6789/0 conn(0x7f870009e740 legacy=0x7f87001cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:52.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.483+0000 7f8705776640 1 -- 172.21.15.189:0/3214935164 shutdown_connections 2024-09-16T14:43:52.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.483+0000 7f8705776640 1 -- 172.21.15.189:0/3214935164 >> 172.21.15.189:0/3214935164 conn(0x7f87000a3320 msgr2=0x7f87000a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:52.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.483+0000 7f8705776640 1 -- 172.21.15.189:0/3214935164 shutdown_connections 2024-09-16T14:43:52.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.483+0000 7f8705776640 1 -- 172.21.15.189:0/3214935164 wait complete. 2024-09-16T14:43:52.502 INFO:tasks.workunit.client.0.smithi189.stdout:pg_num_max: 33 2024-09-16T14:43:52.503 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_min 9 2024-09-16T14:43:52.503 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:43:52.503 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set testpool1234 pg_num_min 9 2024-09-16T14:43:52.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f040977b640 10 monclient: get_monmap_and_config 2024-09-16T14:43:52.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f040977b640 10 monclient: build_initial_monmap 2024-09-16T14:43:52.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f040977b640 10 monclient: monmap: 2024-09-16T14:43:52.619 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:52.619 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:52.619 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:52.619544+0000 2024-09-16T14:43:52.619 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:52.619544+0000 2024-09-16T14:43:52.619 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:52.619 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:52.619 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:52.620 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:52.620 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:52.620 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f040977b640 1 Processor -- start 2024-09-16T14:43:52.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f040977b640 1 -- start start 2024-09-16T14:43:52.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f040977b640 10 monclient: init 2024-09-16T14:43:52.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f040977b640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:52.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f040977b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:43:52.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f040977b640 10 monclient(hunting): picked mon.noname-c con 0x7f04040a1a90 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:52.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f040977b640 10 monclient(hunting): picked mon.noname-a con 0x7f04040a2580 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:52.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f040977b640 10 monclient(hunting): picked mon.noname-b con 0x7f040409fd40 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:52.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f040977b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f040409df00 con 0x7f04040a2580 2024-09-16T14:43:52.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f040977b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f040409f0a0 con 0x7f04040a1a90 2024-09-16T14:43:52.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f040977b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04040a3040 con 0x7f040409fd40 2024-09-16T14:43:52.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f040977b640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:52.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f040977b640 10 monclient(hunting): authenticate will time out at 1014.735199s 2024-09-16T14:43:52.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f0402ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f04040a1a90 0x7f04040a1e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51550/0 (socket says 172.21.15.189:51550) 2024-09-16T14:43:52.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f0402ffd640 1 -- 172.21.15.189:0/4126047707 learned_addr learned my addr 172.21.15.189:0/4126047707 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:52.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f04027fc640 1 -- 172.21.15.189:0/4126047707 >> v1:172.21.15.26:6789/0 conn(0x7f04040a2580 legacy=0x7f0404139df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:52.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f0401ffb640 1 -- 172.21.15.189:0/4126047707 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351546805 0 0) 0x7f040409f0a0 con 0x7f04040a1a90 2024-09-16T14:43:52.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f0401ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:52.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f0401ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:52.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f0401ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:52.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f0401ffb640 10 monclient(hunting): my global_id is 4576 2024-09-16T14:43:52.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f0401ffb640 1 -- 172.21.15.189:0/4126047707 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03ec003530 con 0x7f04040a1a90 2024-09-16T14:43:52.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f0401ffb640 1 -- 172.21.15.189:0/4126047707 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2833245231 0 0) 0x7f03ec003530 con 0x7f04040a1a90 2024-09-16T14:43:52.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.615+0000 7f0401ffb640 1 -- 172.21.15.189:0/4126047707 >> v1:172.21.15.26:6790/0 conn(0x7f040409fd40 legacy=0x7f040413d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:52.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 1 -- 172.21.15.189:0/4126047707 >> v1:172.21.15.26:6789/0 conn(0x7f04040a2580 legacy=0x7f0404139df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:52.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:43:52.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 1 monclient: found mon.noname-c 2024-09-16T14:43:52.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:52.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 1 -- 172.21.15.189:0/4126047707 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040409eae0 con 0x7f04040a1a90 2024-09-16T14:43:52.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 10 monclient: _finish_auth 0 2024-09-16T14:43:52.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:43:52.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:52.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 1 -- 172.21.15.189:0/4126047707 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f03f4001620 con 0x7f04040a1a90 2024-09-16T14:43:52.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 5 monclient: authenticate success, global_id 4576 2024-09-16T14:43:52.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:52.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:52.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:52.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 10 monclient: dump: 2024-09-16T14:43:52.626 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:52.626 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:52.626 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.626 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.626 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:52.626 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:52.626 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:52.626 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:52.626 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:52.627 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:52.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 1 -- 172.21.15.189:0/4126047707 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03f40018d0 con 0x7f04040a1a90 2024-09-16T14:43:52.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:52.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:52.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 4 set_mon_vals no callback set 2024-09-16T14:43:52.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 1 -- 172.21.15.189:0/4126047707 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f03f4001c60 con 0x7f04040a1a90 2024-09-16T14:43:52.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:52.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:52.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0401ffb640 10 monclient: dump: 2024-09-16T14:43:52.627 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:52.627 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:52.628 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.628 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.628 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:52.628 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:52.628 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:52.628 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:52.628 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:52.628 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 10 monclient: shutdown 2024-09-16T14:43:52.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:52.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 1 -- 172.21.15.189:0/4126047707 >> v1:172.21.15.189:6789/0 conn(0x7f04040a1a90 legacy=0x7f04040a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:52.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 1 -- 172.21.15.189:0/4126047707 shutdown_connections 2024-09-16T14:43:52.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 1 -- 172.21.15.189:0/4126047707 >> 172.21.15.189:0/4126047707 conn(0x7f04040a7a70 msgr2=0x7f04040a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:52.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 1 -- 172.21.15.189:0/4126047707 shutdown_connections 2024-09-16T14:43:52.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 1 -- 172.21.15.189:0/4126047707 wait complete. 2024-09-16T14:43:52.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 10 monclient: build_initial_monmap 2024-09-16T14:43:52.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 10 monclient: monmap: 2024-09-16T14:43:52.629 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:52.629 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:52.629 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:52.629 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:52.629 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:52.629 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:52.629 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:52.629 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:52.629 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:52.630 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 1 Processor -- start 2024-09-16T14:43:52.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 1 -- start start 2024-09-16T14:43:52.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 10 monclient: init 2024-09-16T14:43:52.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:52.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:43:52.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 10 monclient(hunting): picked mon.noname-b con 0x7f040409fd40 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:52.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 10 monclient(hunting): picked mon.noname-c con 0x7f04040a1a90 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:52.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 10 monclient(hunting): picked mon.noname-a con 0x7f04040a2580 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:52.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0404124210 con 0x7f04040a2580 2024-09-16T14:43:52.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f040409e280 con 0x7f040409fd40 2024-09-16T14:43:52.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0404124740 con 0x7f04040a1a90 2024-09-16T14:43:52.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:52.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 10 monclient(hunting): authenticate will time out at 1014.738679s 2024-09-16T14:43:52.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0402ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f040409fd40 0x7f0404133d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51556/0 (socket says 172.21.15.189:51556) 2024-09-16T14:43:52.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f0402ffd640 1 -- 172.21.15.189:0/3807482255 learned_addr learned my addr 172.21.15.189:0/3807482255 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:52.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f04037fe640 1 -- 172.21.15.189:0/3807482255 >> v1:172.21.15.26:6789/0 conn(0x7f04040a2580 legacy=0x7f04041d1a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:52.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 1 -- 172.21.15.189:0/3807482255 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791891224 0 0) 0x7f040409e280 con 0x7f040409fd40 2024-09-16T14:43:52.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:52.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:52.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:52.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 10 monclient(hunting): my global_id is 4579 2024-09-16T14:43:52.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 1 -- 172.21.15.189:0/3807482255 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f0003530 con 0x7f040409fd40 2024-09-16T14:43:52.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 1 -- 172.21.15.189:0/3807482255 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1429444884 0 0) 0x7f03f0003530 con 0x7f040409fd40 2024-09-16T14:43:52.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 1 -- 172.21.15.189:0/3807482255 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f040409e280 con 0x7f040409fd40 2024-09-16T14:43:52.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 1 -- 172.21.15.189:0/3807482255 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f03f40015e0 con 0x7f040409fd40 2024-09-16T14:43:52.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:52.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:52.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 10 monclient(hunting): dump: 2024-09-16T14:43:52.635 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:52.635 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:52.635 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.635 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.635 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:52.635 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:52.635 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:52.635 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:52.635 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:52.636 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:52.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 1 -- 172.21.15.189:0/3807482255 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3985291833 0 0) 0x7f040409e280 con 0x7f040409fd40 2024-09-16T14:43:52.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 1 -- 172.21.15.189:0/3807482255 >> v1:172.21.15.26:6790/0 conn(0x7f04040a1a90 legacy=0x7f04041ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:52.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 1 -- 172.21.15.189:0/3807482255 >> v1:172.21.15.26:6789/0 conn(0x7f04040a2580 legacy=0x7f04041d1a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:52.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:43:52.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 1 monclient: found mon.b 2024-09-16T14:43:52.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:52.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 1 -- 172.21.15.189:0/3807482255 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04040a6290 con 0x7f040409fd40 2024-09-16T14:43:52.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 10 monclient: _finish_auth 0 2024-09-16T14:43:52.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:43:52.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:52.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 5 monclient: authenticate success, global_id 4579 2024-09-16T14:43:52.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 10 monclient: _renew_subs 2024-09-16T14:43:52.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 1 -- 172.21.15.189:0/3807482255 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0404123ea0 con 0x7f040409fd40 2024-09-16T14:43:52.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 10 monclient: _renew_subs 2024-09-16T14:43:52.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f040977b640 1 -- 172.21.15.189:0/3807482255 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0404122fd0 con 0x7f040409fd40 2024-09-16T14:43:52.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 1 -- 172.21.15.189:0/3807482255 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03f4004c00 con 0x7f040409fd40 2024-09-16T14:43:52.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:52.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 1 -- 172.21.15.189:0/3807482255 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f03f4004f10 con 0x7f040409fd40 2024-09-16T14:43:52.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:52.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:52.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f04017fa640 4 set_mon_vals no callback set 2024-09-16T14:43:52.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.619+0000 7f03eb7fe640 10 monclient: dump: 2024-09-16T14:43:52.638 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:52.638 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:52.638 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.638 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.638 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:52.638 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:52.638 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:52.638 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:52.638 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:52.638 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.623+0000 7f03eb7fe640 1 -- 172.21.15.189:0/3807482255 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f03f400f980 con 0x7f040409fd40 2024-09-16T14:43:52.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.623+0000 7f040977b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:52.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.623+0000 7f040977b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:52.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.623+0000 7f040977b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.623+0000 7f040977b640 1 -- 172.21.15.189:0/3807482255 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e4005050 con 0x7f040409fd40 2024-09-16T14:43:52.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.623+0000 7f03eb7fe640 1 -- 172.21.15.189:0/3807482255 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3889+0+0 (unknown 2825553975 0 0) 0x7f03f4043450 con 0x7f040409fd40 2024-09-16T14:43:52.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.623+0000 7f04027fc640 10 monclient: get_auth_request con 0x7f03f0036030 auth_method 0 2024-09-16T14:43:52.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.627+0000 7f03eb7fe640 1 -- 172.21.15.189:0/3807482255 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f03f4014740 con 0x7f040409fd40 2024-09-16T14:43:52.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.627+0000 7f03eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:52.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.627+0000 7f03eb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:52.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.783+0000 7f040977b640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-16T14:43:52.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.783+0000 7f040977b640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-16T14:43:52.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.783+0000 7f040977b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.783+0000 7f040977b640 1 -- 172.21.15.189:0/3807482255 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"} v 0) v1 -- 0x7f03e4005880 con 0x7f040409fd40 2024-09-16T14:43:52.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.787+0000 7f03eb7fe640 1 -- 172.21.15.189:0/3807482255 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}]=-22 specified pg_num_min 9 > pg_num 8 v22) v1 ==== 150+0+0 (unknown 3264733481 0 0) 0x7f03f4012de0 con 0x7f040409fd40 2024-09-16T14:43:52.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.787+0000 7f03eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-16T14:43:52.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.787+0000 7f03eb7fe640 10 monclient: _finish_command 2 = mon:22 specified pg_num_min 9 > pg_num 8 2024-09-16T14:43:52.825 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: specified pg_num_min 9 > pg_num 8 2024-09-16T14:43:52.825 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:43:52.826 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_max 7 2024-09-16T14:43:52.826 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:43:52.826 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set testpool1234 pg_num_max 7 2024-09-16T14:43:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 10 monclient: get_monmap_and_config 2024-09-16T14:43:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 10 monclient: build_initial_monmap 2024-09-16T14:43:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 10 monclient: monmap: 2024-09-16T14:43:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:52.915479+0000 2024-09-16T14:43:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:52.915479+0000 2024-09-16T14:43:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:52.916 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 1 Processor -- start 2024-09-16T14:43:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 1 -- start start 2024-09-16T14:43:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 10 monclient: init 2024-09-16T14:43:52.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:52.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:43:52.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 10 monclient(hunting): picked mon.noname-a con 0x7f3ed4099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:52.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 10 monclient(hunting): picked mon.noname-b con 0x7f3ed409c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:52.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 10 monclient(hunting): picked mon.noname-c con 0x7f3ed409e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:52.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed4099e50 con 0x7f3ed4099a70 2024-09-16T14:43:52.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed409dbd0 con 0x7f3ed409e6d0 2024-09-16T14:43:52.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed409ee20 con 0x7f3ed409c550 2024-09-16T14:43:52.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:52.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 10 monclient(hunting): authenticate will time out at 1015.031114s 2024-09-16T14:43:52.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed9660640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3ed409e6d0 0x7f3ed413b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51560/0 (socket says 172.21.15.189:51560) 2024-09-16T14:43:52.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed9660640 1 -- 172.21.15.189:0/3678044966 learned_addr learned my addr 172.21.15.189:0/3678044966 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:52.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed8e5f640 1 -- 172.21.15.189:0/3678044966 >> v1:172.21.15.26:6789/0 conn(0x7f3ed4099a70 legacy=0x7f3ed409dfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:52.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 1 -- 172.21.15.189:0/3678044966 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736910756 0 0) 0x7f3ed409dbd0 con 0x7f3ed409e6d0 2024-09-16T14:43:52.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:52.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:52.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:52.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 10 monclient(hunting): my global_id is 4582 2024-09-16T14:43:52.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 1 -- 172.21.15.189:0/3678044966 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ebc003530 con 0x7f3ed409e6d0 2024-09-16T14:43:52.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 1 -- 172.21.15.189:0/3678044966 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1597540754 0 0) 0x7f3ebc003530 con 0x7f3ed409e6d0 2024-09-16T14:43:52.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 1 -- 172.21.15.189:0/3678044966 >> v1:172.21.15.26:6790/0 conn(0x7f3ed409c550 legacy=0x7f3ed41379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:52.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 1 -- 172.21.15.189:0/3678044966 >> v1:172.21.15.26:6789/0 conn(0x7f3ed4099a70 legacy=0x7f3ed409dfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:52.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:43:52.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 1 monclient: found mon.noname-c 2024-09-16T14:43:52.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:52.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 1 -- 172.21.15.189:0/3678044966 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed409f100 con 0x7f3ed409e6d0 2024-09-16T14:43:52.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 10 monclient: _finish_auth 0 2024-09-16T14:43:52.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:43:52.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:52.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 1 -- 172.21.15.189:0/3678044966 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ecc0019a0 con 0x7f3ed409e6d0 2024-09-16T14:43:52.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 5 monclient: authenticate success, global_id 4582 2024-09-16T14:43:52.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:52.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:52.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:52.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 10 monclient: dump: 2024-09-16T14:43:52.925 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:52.925 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:52.926 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.926 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.926 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:52.926 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:52.926 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:52.926 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:52.926 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:52.926 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:52.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 1 -- 172.21.15.189:0/3678044966 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ecc001c50 con 0x7f3ed409e6d0 2024-09-16T14:43:52.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3ed37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:52.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:52.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.911+0000 7f3edb0c3640 4 set_mon_vals no callback set 2024-09-16T14:43:52.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed37fe640 1 -- 172.21.15.189:0/3678044966 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ecc002020 con 0x7f3ed409e6d0 2024-09-16T14:43:52.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:52.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:52.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed37fe640 10 monclient: dump: 2024-09-16T14:43:52.927 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:52.927 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:52.927 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.927 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.927 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:52.928 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:52.928 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:52.928 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:52.928 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:52.928 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient: shutdown 2024-09-16T14:43:52.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:52.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 1 -- 172.21.15.189:0/3678044966 >> v1:172.21.15.189:6789/0 conn(0x7f3ed409e6d0 legacy=0x7f3ed413b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:52.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 1 -- 172.21.15.189:0/3678044966 shutdown_connections 2024-09-16T14:43:52.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 1 -- 172.21.15.189:0/3678044966 >> 172.21.15.189:0/3678044966 conn(0x7f3ed40a3380 msgr2=0x7f3ed409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:52.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 1 -- 172.21.15.189:0/3678044966 shutdown_connections 2024-09-16T14:43:52.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 1 -- 172.21.15.189:0/3678044966 wait complete. 2024-09-16T14:43:52.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient: build_initial_monmap 2024-09-16T14:43:52.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient: monmap: 2024-09-16T14:43:52.929 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:52.929 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:52.929 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:52.929 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:52.929 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:52.929 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:52.929 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:52.929 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:52.929 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:52.929 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 1 Processor -- start 2024-09-16T14:43:52.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 1 -- start start 2024-09-16T14:43:52.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient: init 2024-09-16T14:43:52.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:52.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:43:52.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient(hunting): picked mon.noname-b con 0x7f3ed4099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:52.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient(hunting): picked mon.noname-c con 0x7f3ed409c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:52.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient(hunting): picked mon.noname-a con 0x7f3ed409e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:52.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed409cf20 con 0x7f3ed409e6d0 2024-09-16T14:43:52.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed413c3e0 con 0x7f3ed4099a70 2024-09-16T14:43:52.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed406cd90 con 0x7f3ed409c550 2024-09-16T14:43:52.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:52.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient(hunting): authenticate will time out at 1015.033902s 2024-09-16T14:43:52.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed8e5f640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3ed4099a70 0x7f3ed4139bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51574/0 (socket says 172.21.15.189:51574) 2024-09-16T14:43:52.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed8e5f640 1 -- 172.21.15.189:0/2339211144 learned_addr learned my addr 172.21.15.189:0/2339211144 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:52.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed9660640 1 -- 172.21.15.189:0/2339211144 >> v1:172.21.15.26:6789/0 conn(0x7f3ed409e6d0 legacy=0x7f3ed413a9e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:52.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 1 -- 172.21.15.189:0/2339211144 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836946487 0 0) 0x7f3ed413c3e0 con 0x7f3ed4099a70 2024-09-16T14:43:52.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:43:52.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:52.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:52.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 10 monclient(hunting): my global_id is 4585 2024-09-16T14:43:52.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 1 -- 172.21.15.189:0/2339211144 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ec0003530 con 0x7f3ed4099a70 2024-09-16T14:43:52.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 1 -- 172.21.15.189:0/2339211144 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3329176063 0 0) 0x7f3ec0003530 con 0x7f3ed4099a70 2024-09-16T14:43:52.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 1 -- 172.21.15.189:0/2339211144 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ed413c3e0 con 0x7f3ed4099a70 2024-09-16T14:43:52.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 1 -- 172.21.15.189:0/2339211144 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ec4004190 con 0x7f3ed4099a70 2024-09-16T14:43:52.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:52.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:52.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 10 monclient(hunting): dump: 2024-09-16T14:43:52.932 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:52.932 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:52.932 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.932 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.932 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:52.932 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:52.932 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:52.932 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:52.932 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:52.932 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:52.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 1 -- 172.21.15.189:0/2339211144 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4009380391 0 0) 0x7f3ed413c3e0 con 0x7f3ed4099a70 2024-09-16T14:43:52.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 1 -- 172.21.15.189:0/2339211144 >> v1:172.21.15.26:6790/0 conn(0x7f3ed409c550 legacy=0x7f3ed413a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:52.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 1 -- 172.21.15.189:0/2339211144 >> v1:172.21.15.26:6789/0 conn(0x7f3ed409e6d0 legacy=0x7f3ed413a9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:52.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 10 monclient: _finish_hunting 0 2024-09-16T14:43:52.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 1 monclient: found mon.b 2024-09-16T14:43:52.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:52.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 1 -- 172.21.15.189:0/2339211144 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed413c180 con 0x7f3ed4099a70 2024-09-16T14:43:52.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 10 monclient: _finish_auth 0 2024-09-16T14:43:52.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 10 monclient: _check_auth_tickets 2024-09-16T14:43:52.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:52.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 5 monclient: authenticate success, global_id 4585 2024-09-16T14:43:52.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient: _renew_subs 2024-09-16T14:43:52.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 1 -- 172.21.15.189:0/2339211144 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed411ea70 con 0x7f3ed4099a70 2024-09-16T14:43:52.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient: _renew_subs 2024-09-16T14:43:52.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 1 -- 172.21.15.189:0/2339211144 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed406f970 con 0x7f3ed4099a70 2024-09-16T14:43:52.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 1 -- 172.21.15.189:0/2339211144 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ec40044b0 con 0x7f3ed4099a70 2024-09-16T14:43:52.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:52.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 1 -- 172.21.15.189:0/2339211144 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ec4004840 con 0x7f3ed4099a70 2024-09-16T14:43:52.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:52.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:52.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed2ffd640 4 set_mon_vals no callback set 2024-09-16T14:43:52.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 10 monclient: dump: 2024-09-16T14:43:52.934 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:52.934 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:52.935 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.935 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:52.935 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:52.935 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:52.935 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:52.935 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:52.935 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:52.935 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:52.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:52.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:52.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:52.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3edb0c3640 1 -- 172.21.15.189:0/2339211144 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb4005050 con 0x7f3ed4099a70 2024-09-16T14:43:52.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.915+0000 7f3ed17fa640 1 -- 172.21.15.189:0/2339211144 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3ec4004a20 con 0x7f3ed4099a70 2024-09-16T14:43:52.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.919+0000 7f3ed17fa640 1 -- 172.21.15.189:0/2339211144 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3889+0+0 (unknown 2825553975 0 0) 0x7f3ec4043300 con 0x7f3ed4099a70 2024-09-16T14:43:52.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.919+0000 7f3ed3fff640 10 monclient: get_auth_request con 0x7f3ec0036000 auth_method 0 2024-09-16T14:43:52.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.923+0000 7f3ed17fa640 1 -- 172.21.15.189:0/2339211144 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3ec40051f0 con 0x7f3ed4099a70 2024-09-16T14:43:52.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.923+0000 7f3ed17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:52.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:52.923+0000 7f3ed17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:53.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.075+0000 7f3edb0c3640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-16T14:43:53.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.075+0000 7f3edb0c3640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-16T14:43:53.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.075+0000 7f3edb0c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.075+0000 7f3edb0c3640 1 -- 172.21.15.189:0/2339211144 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"} v 0) v1 -- 0x7f3eb4005880 con 0x7f3ed4099a70 2024-09-16T14:43:53.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.079+0000 7f3ed17fa640 1 -- 172.21.15.189:0/2339211144 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}]=-22 specified pg_num_max 7 < pg_num 8 v22) v1 ==== 150+0+0 (unknown 3252560201 0 0) 0x7f3ec4012bd0 con 0x7f3ed4099a70 2024-09-16T14:43:53.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.079+0000 7f3ed17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-16T14:43:53.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.079+0000 7f3ed17fa640 10 monclient: _finish_command 2 = mon:22 specified pg_num_max 7 < pg_num 8 2024-09-16T14:43:53.083 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: specified pg_num_max 7 < pg_num 8 2024-09-16T14:43:53.088 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:43:53.088 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 1 2024-09-16T14:43:53.088 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:43:53.089 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set testpool1234 pg_num 1 2024-09-16T14:43:53.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 10 monclient: get_monmap_and_config 2024-09-16T14:43:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 10 monclient: build_initial_monmap 2024-09-16T14:43:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 10 monclient: monmap: 2024-09-16T14:43:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:53.207336+0000 2024-09-16T14:43:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:53.207336+0000 2024-09-16T14:43:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:53.208 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 1 Processor -- start 2024-09-16T14:43:53.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 1 -- start start 2024-09-16T14:43:53.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 10 monclient: init 2024-09-16T14:43:53.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:53.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:43:53.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 10 monclient(hunting): picked mon.noname-a con 0x7f1e20099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:53.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 10 monclient(hunting): picked mon.noname-b con 0x7f1e2009c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:53.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 10 monclient(hunting): picked mon.noname-c con 0x7f1e2009e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:53.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e20099e50 con 0x7f1e20099a70 2024-09-16T14:43:53.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2009dbd0 con 0x7f1e2009e6d0 2024-09-16T14:43:53.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2009ee20 con 0x7f1e2009c550 2024-09-16T14:43:53.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:53.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 10 monclient(hunting): authenticate will time out at 1015.323054s 2024-09-16T14:43:53.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1effd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f1e20099a70 legacy=0x7f1e2009dfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:53.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1f7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1e2009e6d0 0x7f1e2013b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51576/0 (socket says 172.21.15.189:51576) 2024-09-16T14:43:53.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1f7fe640 1 -- 172.21.15.189:0/429881751 learned_addr learned my addr 172.21.15.189:0/429881751 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:53.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 1 -- 172.21.15.189:0/429881751 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789032160 0 0) 0x7f1e2009dbd0 con 0x7f1e2009e6d0 2024-09-16T14:43:53.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:53.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:53.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:53.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 10 monclient(hunting): my global_id is 4588 2024-09-16T14:43:53.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 1 -- 172.21.15.189:0/429881751 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e0c003530 con 0x7f1e2009e6d0 2024-09-16T14:43:53.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 1 -- 172.21.15.189:0/429881751 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 656339800 0 0) 0x7f1e0c003530 con 0x7f1e2009e6d0 2024-09-16T14:43:53.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 1 -- 172.21.15.189:0/429881751 >> v1:172.21.15.26:6790/0 conn(0x7f1e2009c550 legacy=0x7f1e201379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:53.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 1 -- 172.21.15.189:0/429881751 >> v1:172.21.15.26:6789/0 conn(0x7f1e20099a70 legacy=0x7f1e2009dfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:53.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:43:53.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 1 monclient: found mon.noname-c 2024-09-16T14:43:53.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:53.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 1 -- 172.21.15.189:0/429881751 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2009f100 con 0x7f1e2009e6d0 2024-09-16T14:43:53.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 10 monclient: _finish_auth 0 2024-09-16T14:43:53.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:43:53.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:53.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 1 -- 172.21.15.189:0/429881751 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1e180019a0 con 0x7f1e2009e6d0 2024-09-16T14:43:53.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:53.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:53.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 10 monclient: dump: 2024-09-16T14:43:53.217 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:53.217 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:53.217 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.217 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.217 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:53.217 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:53.217 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:53.217 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:53.217 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:53.217 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 5 monclient: authenticate success, global_id 4588 2024-09-16T14:43:53.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 1 -- 172.21.15.189:0/429881751 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e18001c50 con 0x7f1e2009e6d0 2024-09-16T14:43:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 1 -- 172.21.15.189:0/429881751 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1e18002020 con 0x7f1e2009e6d0 2024-09-16T14:43:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e1dffb640 10 monclient: dump: 2024-09-16T14:43:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:53.219 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:53.219 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:53.219 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:53.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.203+0000 7f1e25477640 4 set_mon_vals no callback set 2024-09-16T14:43:53.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient: shutdown 2024-09-16T14:43:53.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:53.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 1 -- 172.21.15.189:0/429881751 >> v1:172.21.15.189:6789/0 conn(0x7f1e2009e6d0 legacy=0x7f1e2013b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:53.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 1 -- 172.21.15.189:0/429881751 shutdown_connections 2024-09-16T14:43:53.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 1 -- 172.21.15.189:0/429881751 >> 172.21.15.189:0/429881751 conn(0x7f1e200a3380 msgr2=0x7f1e2009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:53.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 1 -- 172.21.15.189:0/429881751 shutdown_connections 2024-09-16T14:43:53.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 1 -- 172.21.15.189:0/429881751 wait complete. 2024-09-16T14:43:53.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient: build_initial_monmap 2024-09-16T14:43:53.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient: monmap: 2024-09-16T14:43:53.220 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:53.220 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:53.220 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:53.220 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:53.220 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:53.220 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:53.220 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:53.220 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:53.220 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:53.220 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 1 Processor -- start 2024-09-16T14:43:53.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 1 -- start start 2024-09-16T14:43:53.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient: init 2024-09-16T14:43:53.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:53.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:43:53.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient(hunting): picked mon.noname-a con 0x7f1e20099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:53.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient(hunting): picked mon.noname-b con 0x7f1e2009c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:53.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient(hunting): picked mon.noname-c con 0x7f1e2009e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:53.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2009cf20 con 0x7f1e20099a70 2024-09-16T14:43:53.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e20125560 con 0x7f1e2009c550 2024-09-16T14:43:53.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2013c570 con 0x7f1e2009e6d0 2024-09-16T14:43:53.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:53.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient(hunting): authenticate will time out at 1015.326115s 2024-09-16T14:43:53.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e1e7fc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1e2009c550 0x7f1e2009a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51584/0 (socket says 172.21.15.189:51584) 2024-09-16T14:43:53.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e1e7fc640 1 -- 172.21.15.189:0/213182594 learned_addr learned my addr 172.21.15.189:0/213182594 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:53.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e1effd640 1 -- 172.21.15.189:0/213182594 >> v1:172.21.15.26:6789/0 conn(0x7f1e20099a70 legacy=0x7f1e2009a220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:53.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 1 -- 172.21.15.189:0/213182594 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297043775 0 0) 0x7f1e20125560 con 0x7f1e2009c550 2024-09-16T14:43:53.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:53.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:53.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:53.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 10 monclient(hunting): my global_id is 4591 2024-09-16T14:43:53.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 1 -- 172.21.15.189:0/213182594 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e00003560 con 0x7f1e2009c550 2024-09-16T14:43:53.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 1 -- 172.21.15.189:0/213182594 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3690547024 0 0) 0x7f1e00003560 con 0x7f1e2009c550 2024-09-16T14:43:53.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 1 -- 172.21.15.189:0/213182594 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e20125560 con 0x7f1e2009c550 2024-09-16T14:43:53.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 1 -- 172.21.15.189:0/213182594 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1e10001950 con 0x7f1e2009c550 2024-09-16T14:43:53.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:53.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:53.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 10 monclient(hunting): dump: 2024-09-16T14:43:53.223 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:53.223 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:53.223 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.223 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.223 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:53.223 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:53.223 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:53.223 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:53.223 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:53.223 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:53.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 1 -- 172.21.15.189:0/213182594 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 466978078 0 0) 0x7f1e20125560 con 0x7f1e2009c550 2024-09-16T14:43:53.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 1 -- 172.21.15.189:0/213182594 >> v1:172.21.15.26:6790/0 conn(0x7f1e2009e6d0 legacy=0x7f1e2009b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:53.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 1 -- 172.21.15.189:0/213182594 >> v1:172.21.15.26:6789/0 conn(0x7f1e20099a70 legacy=0x7f1e2009a220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:53.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 10 monclient: _finish_hunting 0 2024-09-16T14:43:53.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 1 monclient: found mon.b 2024-09-16T14:43:53.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:53.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 1 -- 172.21.15.189:0/213182594 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2006ce30 con 0x7f1e2009c550 2024-09-16T14:43:53.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 10 monclient: _finish_auth 0 2024-09-16T14:43:53.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 10 monclient: _check_auth_tickets 2024-09-16T14:43:53.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:53.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 5 monclient: authenticate success, global_id 4591 2024-09-16T14:43:53.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient: _renew_subs 2024-09-16T14:43:53.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 1 -- 172.21.15.189:0/213182594 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2006b320 con 0x7f1e2009c550 2024-09-16T14:43:53.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient: _renew_subs 2024-09-16T14:43:53.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 1 -- 172.21.15.189:0/213182594 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e20125280 con 0x7f1e2009c550 2024-09-16T14:43:53.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 1 -- 172.21.15.189:0/213182594 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e10001d60 con 0x7f1e2009c550 2024-09-16T14:43:53.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:53.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 1 -- 172.21.15.189:0/213182594 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1e10003a70 con 0x7f1e2009c550 2024-09-16T14:43:53.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:53.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:53.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 10 monclient: dump: 2024-09-16T14:43:53.225 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:53.225 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:53.225 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:53.226 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e1d7fa640 4 set_mon_vals no callback set 2024-09-16T14:43:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 1 -- 172.21.15.189:0/213182594 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1e10003bd0 con 0x7f1e2009c550 2024-09-16T14:43:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e25477640 1 -- 172.21.15.189:0/213182594 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e04005050 con 0x7f1e2009c550 2024-09-16T14:43:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.207+0000 7f1e177fe640 1 -- 172.21.15.189:0/213182594 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3889+0+0 (unknown 2825553975 0 0) 0x7f1e100433c0 con 0x7f1e2009c550 2024-09-16T14:43:53.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.211+0000 7f1e1effd640 10 monclient: get_auth_request con 0x7f1e00036080 auth_method 0 2024-09-16T14:43:53.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.215+0000 7f1e177fe640 1 -- 172.21.15.189:0/213182594 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1e10014820 con 0x7f1e2009c550 2024-09-16T14:43:53.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.215+0000 7f1e177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:53.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.215+0000 7f1e177fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:53.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.359+0000 7f1e25477640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-16T14:43:53.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.359+0000 7f1e25477640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-16T14:43:53.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.359+0000 7f1e25477640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.359+0000 7f1e25477640 1 -- 172.21.15.189:0/213182594 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"} v 0) v1 -- 0x7f1e04005880 con 0x7f1e2009c550 2024-09-16T14:43:53.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.363+0000 7f1e177fe640 1 -- 172.21.15.189:0/213182594 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}]=-22 specified pg_num 1 < pg_num_min 2 v22) v1 ==== 146+0+0 (unknown 939204074 0 0) 0x7f1e10012c90 con 0x7f1e2009c550 2024-09-16T14:43:53.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.363+0000 7f1e177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-16T14:43:53.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.363+0000 7f1e177fe640 10 monclient: _finish_command 2 = mon:22 specified pg_num 1 < pg_num_min 2 2024-09-16T14:43:53.367 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: specified pg_num 1 < pg_num_min 2 2024-09-16T14:43:53.371 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:43:53.371 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 44 2024-09-16T14:43:53.371 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:43:53.372 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set testpool1234 pg_num 44 2024-09-16T14:43:53.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 10 monclient: get_monmap_and_config 2024-09-16T14:43:53.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 10 monclient: build_initial_monmap 2024-09-16T14:43:53.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 10 monclient: monmap: 2024-09-16T14:43:53.490 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:53.490 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:53.490 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:53.489948+0000 2024-09-16T14:43:53.490 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:53.489948+0000 2024-09-16T14:43:53.490 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:53.490 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:53.490 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:53.490 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:53.490 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:53.490 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 1 Processor -- start 2024-09-16T14:43:53.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 1 -- start start 2024-09-16T14:43:53.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 10 monclient: init 2024-09-16T14:43:53.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:53.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:43:53.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 10 monclient(hunting): picked mon.noname-c con 0x7f496c0a19b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:53.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 10 monclient(hunting): picked mon.noname-b con 0x7f496c0a24a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:53.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 10 monclient(hunting): picked mon.noname-a con 0x7f496c09fc80 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:53.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c0a00d0 con 0x7f496c09fc80 2024-09-16T14:43:53.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c09ea00 con 0x7f496c0a19b0 2024-09-16T14:43:53.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c09dde0 con 0x7f496c0a24a0 2024-09-16T14:43:53.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:53.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 10 monclient(hunting): authenticate will time out at 1015.605566s 2024-09-16T14:43:53.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4970d47640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f496c0a19b0 0x7f496c0a1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51588/0 (socket says 172.21.15.189:51588) 2024-09-16T14:43:53.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4970d47640 1 -- 172.21.15.189:0/1691378341 learned_addr learned my addr 172.21.15.189:0/1691378341 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:53.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4971548640 1 -- 172.21.15.189:0/1691378341 >> v1:172.21.15.26:6789/0 conn(0x7f496c09fc80 legacy=0x7f496c13d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:53.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 1 -- 172.21.15.189:0/1691378341 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140003194 0 0) 0x7f496c09ea00 con 0x7f496c0a19b0 2024-09-16T14:43:53.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:53.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:53.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:53.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 10 monclient(hunting): my global_id is 4594 2024-09-16T14:43:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 1 -- 172.21.15.189:0/1691378341 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4954003530 con 0x7f496c0a19b0 2024-09-16T14:43:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 1 -- 172.21.15.189:0/1691378341 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 980217657 0 0) 0x7f4954003530 con 0x7f496c0a19b0 2024-09-16T14:43:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 1 -- 172.21.15.189:0/1691378341 >> v1:172.21.15.26:6790/0 conn(0x7f496c0a24a0 legacy=0x7f496c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:43:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 1 -- 172.21.15.189:0/1691378341 >> v1:172.21.15.26:6789/0 conn(0x7f496c09fc80 legacy=0x7f496c13d440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:43:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 1 monclient: found mon.noname-c 2024-09-16T14:43:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 1 -- 172.21.15.189:0/1691378341 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496c09ece0 con 0x7f496c0a19b0 2024-09-16T14:43:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:43:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:43:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:53.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 1 -- 172.21.15.189:0/1691378341 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f495c0019a0 con 0x7f496c0a19b0 2024-09-16T14:43:53.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 5 monclient: authenticate success, global_id 4594 2024-09-16T14:43:53.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:53.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:53.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:53.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 10 monclient: dump: 2024-09-16T14:43:53.499 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:53.499 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:53.499 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.499 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.499 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:53.499 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:53.499 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:53.500 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:53.500 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:53.500 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:53.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 1 -- 172.21.15.189:0/1691378341 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f495c001c90 con 0x7f496c0a19b0 2024-09-16T14:43:53.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:53.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:53.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 4 set_mon_vals no callback set 2024-09-16T14:43:53.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 1 -- 172.21.15.189:0/1691378341 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f495c0020a0 con 0x7f496c0a19b0 2024-09-16T14:43:53.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:53.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:53.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f496b7fe640 10 monclient: dump: 2024-09-16T14:43:53.501 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:53.501 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:53.501 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.501 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.501 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:53.501 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:53.501 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:53.501 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:53.501 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:53.501 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 10 monclient: shutdown 2024-09-16T14:43:53.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:53.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 1 -- 172.21.15.189:0/1691378341 >> v1:172.21.15.189:6789/0 conn(0x7f496c0a19b0 legacy=0x7f496c0a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:53.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 1 -- 172.21.15.189:0/1691378341 shutdown_connections 2024-09-16T14:43:53.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 1 -- 172.21.15.189:0/1691378341 >> 172.21.15.189:0/1691378341 conn(0x7f496c0a3ab0 msgr2=0x7f496c0a3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:53.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 1 -- 172.21.15.189:0/1691378341 shutdown_connections 2024-09-16T14:43:53.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 1 -- 172.21.15.189:0/1691378341 wait complete. 2024-09-16T14:43:53.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 10 monclient: build_initial_monmap 2024-09-16T14:43:53.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.487+0000 7f4972fab640 10 monclient: monmap: 2024-09-16T14:43:53.502 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:53.502 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:53.502 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:53.502 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:53.502 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:53.502 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:53.502 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:53.502 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:53.502 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:53.503 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 1 Processor -- start 2024-09-16T14:43:53.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 1 -- start start 2024-09-16T14:43:53.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 10 monclient: init 2024-09-16T14:43:53.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:53.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:43:53.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 10 monclient(hunting): picked mon.noname-b con 0x7f496c09fc80 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:53.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 10 monclient(hunting): picked mon.noname-c con 0x7f496c0a19b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:53.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 10 monclient(hunting): picked mon.noname-a con 0x7f496c0a24a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:53.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c0a63a0 con 0x7f496c0a24a0 2024-09-16T14:43:53.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c121180 con 0x7f496c09fc80 2024-09-16T14:43:53.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c11fcd0 con 0x7f496c0a19b0 2024-09-16T14:43:53.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:53.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 10 monclient(hunting): authenticate will time out at 1015.608587s 2024-09-16T14:43:53.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4970d47640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f496c09fc80 0x7f496c1c8e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51602/0 (socket says 172.21.15.189:51602) 2024-09-16T14:43:53.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4970d47640 1 -- 172.21.15.189:0/1656512720 learned_addr learned my addr 172.21.15.189:0/1656512720 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:53.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4971548640 1 -- 172.21.15.189:0/1656512720 >> v1:172.21.15.26:6789/0 conn(0x7f496c0a24a0 legacy=0x7f496c1d11a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:53.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 1 -- 172.21.15.189:0/1656512720 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53261523 0 0) 0x7f496c121180 con 0x7f496c09fc80 2024-09-16T14:43:53.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 10 monclient(hunting): _negotiate 2024-09-16T14:43:53.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:53.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:53.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 10 monclient(hunting): my global_id is 4597 2024-09-16T14:43:53.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 1 -- 172.21.15.189:0/1656512720 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4958003530 con 0x7f496c09fc80 2024-09-16T14:43:53.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 1 -- 172.21.15.189:0/1656512720 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1716876219 0 0) 0x7f4958003530 con 0x7f496c09fc80 2024-09-16T14:43:53.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 1 -- 172.21.15.189:0/1656512720 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f496c121180 con 0x7f496c09fc80 2024-09-16T14:43:53.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 1 -- 172.21.15.189:0/1656512720 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f495c001960 con 0x7f496c09fc80 2024-09-16T14:43:53.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:53.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:53.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 10 monclient(hunting): dump: 2024-09-16T14:43:53.505 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:53.505 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:53.505 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.505 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.505 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:53.505 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:53.505 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:53.505 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:53.506 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:53.506 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:53.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 1 -- 172.21.15.189:0/1656512720 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2262191265 0 0) 0x7f496c121180 con 0x7f496c09fc80 2024-09-16T14:43:53.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 1 -- 172.21.15.189:0/1656512720 >> v1:172.21.15.26:6790/0 conn(0x7f496c0a19b0 legacy=0x7f496c133aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:53.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 1 -- 172.21.15.189:0/1656512720 >> v1:172.21.15.26:6789/0 conn(0x7f496c0a24a0 legacy=0x7f496c1d11a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:53.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 10 monclient: _finish_hunting 0 2024-09-16T14:43:53.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 1 monclient: found mon.b 2024-09-16T14:43:53.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:53.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 1 -- 172.21.15.189:0/1656512720 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496c11f120 con 0x7f496c09fc80 2024-09-16T14:43:53.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 10 monclient: _finish_auth 0 2024-09-16T14:43:53.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 10 monclient: _check_auth_tickets 2024-09-16T14:43:53.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:53.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 5 monclient: authenticate success, global_id 4597 2024-09-16T14:43:53.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 10 monclient: _renew_subs 2024-09-16T14:43:53.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 1 -- 172.21.15.189:0/1656512720 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496c0a5ec0 con 0x7f496c09fc80 2024-09-16T14:43:53.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 10 monclient: _renew_subs 2024-09-16T14:43:53.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 1 -- 172.21.15.189:0/1656512720 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f496c11fa50 con 0x7f496c09fc80 2024-09-16T14:43:53.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 1 -- 172.21.15.189:0/1656512720 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f495c005050 con 0x7f496c09fc80 2024-09-16T14:43:53.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:53.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 1 -- 172.21.15.189:0/1656512720 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f495c005360 con 0x7f496c09fc80 2024-09-16T14:43:53.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:53.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f496affd640 4 set_mon_vals no callback set 2024-09-16T14:43:53.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:53.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 10 monclient: dump: 2024-09-16T14:43:53.508 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:53.508 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:53.508 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.508 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.508 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:53.508 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:53.508 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:53.508 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:53.508 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:53.508 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:53.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:53.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 1 -- 172.21.15.189:0/1656512720 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f495c0054c0 con 0x7f496c09fc80 2024-09-16T14:43:53.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f4972fab640 1 -- 172.21.15.189:0/1656512720 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f494c005050 con 0x7f496c09fc80 2024-09-16T14:43:53.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f49697fa640 1 -- 172.21.15.189:0/1656512720 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3889+0+0 (unknown 2825553975 0 0) 0x7f495c0438e0 con 0x7f496c09fc80 2024-09-16T14:43:53.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.491+0000 7f496bfff640 10 monclient: get_auth_request con 0x7f49580360d0 auth_method 0 2024-09-16T14:43:53.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.495+0000 7f49697fa640 1 -- 172.21.15.189:0/1656512720 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f495c014da0 con 0x7f496c09fc80 2024-09-16T14:43:53.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.495+0000 7f49697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:53.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.495+0000 7f49697fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:53.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.643+0000 7f4972fab640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-16T14:43:53.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.643+0000 7f4972fab640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-16T14:43:53.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.643+0000 7f4972fab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.643+0000 7f4972fab640 1 -- 172.21.15.189:0/1656512720 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"} v 0) v1 -- 0x7f494c005880 con 0x7f496c09fc80 2024-09-16T14:43:53.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.643+0000 7f49697fa640 1 -- 172.21.15.189:0/1656512720 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}]=-22 specified pg_num 44 < pg_num_max 33 v22) v1 ==== 149+0+0 (unknown 717669047 0 0) 0x7f495c013210 con 0x7f496c09fc80 2024-09-16T14:43:53.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.643+0000 7f49697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-16T14:43:53.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.643+0000 7f49697fa640 10 monclient: _finish_command 2 = mon:22 specified pg_num 44 < pg_num_max 33 2024-09-16T14:43:53.648 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: specified pg_num 44 < pg_num_max 33 2024-09-16T14:43:53.653 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:43:53.654 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set testpool1234 pg_num_min 0 2024-09-16T14:43:53.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 10 monclient: get_monmap_and_config 2024-09-16T14:43:53.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 10 monclient: build_initial_monmap 2024-09-16T14:43:53.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 10 monclient: monmap: 2024-09-16T14:43:53.771 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:53.771 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:53.771 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:53.771157+0000 2024-09-16T14:43:53.771 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:53.771157+0000 2024-09-16T14:43:53.772 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:53.772 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:53.772 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:53.772 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:53.772 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:53.772 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 1 Processor -- start 2024-09-16T14:43:53.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 1 -- start start 2024-09-16T14:43:53.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 10 monclient: init 2024-09-16T14:43:53.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:53.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:43:53.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 10 monclient(hunting): picked mon.noname-b con 0x7f1fe0131630 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:53.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 10 monclient(hunting): picked mon.noname-a con 0x7f1fe009edd0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:53.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 10 monclient(hunting): picked mon.noname-c con 0x7f1fe009e950 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:53.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe009b920 con 0x7f1fe009edd0 2024-09-16T14:43:53.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe009bc00 con 0x7f1fe009e950 2024-09-16T14:43:53.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe009cba0 con 0x7f1fe0131630 2024-09-16T14:43:53.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:53.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 10 monclient(hunting): authenticate will time out at 1015.887109s 2024-09-16T14:43:53.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe48bb640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1fe009e950 0x7f1fe013a290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51610/0 (socket says 172.21.15.189:51610) 2024-09-16T14:43:53.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe48bb640 1 -- 172.21.15.189:0/394443299 learned_addr learned my addr 172.21.15.189:0/394443299 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:53.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdf7fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f1fe009edd0 legacy=0x7f1fe0136b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:53.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 1 -- 172.21.15.189:0/394443299 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12117706 0 0) 0x7f1fe009bc00 con 0x7f1fe009e950 2024-09-16T14:43:53.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:43:53.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:53.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:53.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 10 monclient(hunting): my global_id is 4600 2024-09-16T14:43:53.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 1 -- 172.21.15.189:0/394443299 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc8003530 con 0x7f1fe009e950 2024-09-16T14:43:53.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 1 -- 172.21.15.189:0/394443299 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1997526853 0 0) 0x7f1fc8003530 con 0x7f1fe009e950 2024-09-16T14:43:53.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 1 -- 172.21.15.189:0/394443299 >> v1:172.21.15.26:6790/0 conn(0x7f1fe0131630 legacy=0x7f1fe0131a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:53.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 1 -- 172.21.15.189:0/394443299 >> v1:172.21.15.26:6789/0 conn(0x7f1fe009edd0 legacy=0x7f1fe0136b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 10 monclient: _finish_hunting 0 2024-09-16T14:43:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 1 monclient: found mon.noname-c 2024-09-16T14:43:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 1 -- 172.21.15.189:0/394443299 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe009b420 con 0x7f1fe009e950 2024-09-16T14:43:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 10 monclient: _finish_auth 0 2024-09-16T14:43:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 10 monclient: _check_auth_tickets 2024-09-16T14:43:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 5 monclient: authenticate success, global_id 4600 2024-09-16T14:43:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 1 -- 172.21.15.189:0/394443299 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1fd80019a0 con 0x7f1fe009e950 2024-09-16T14:43:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:53.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fdeffd640 10 monclient: dump: 2024-09-16T14:43:53.782 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:53.782 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:53.782 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.782 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.782 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:53.782 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:53.782 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:53.782 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:53.782 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:53.782 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.767+0000 7f1fe631e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:53.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdeffd640 1 -- 172.21.15.189:0/394443299 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fd8001cc0 con 0x7f1fe009e950 2024-09-16T14:43:53.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:53.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdeffd640 1 -- 172.21.15.189:0/394443299 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1fd8002090 con 0x7f1fe009e950 2024-09-16T14:43:53.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:53.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 4 set_mon_vals no callback set 2024-09-16T14:43:53.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:53.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:53.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdeffd640 10 monclient: dump: 2024-09-16T14:43:53.783 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:53.783 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:53.783 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.783 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.783 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:53.783 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:53.783 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:53.784 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:53.784 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:53.784 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 10 monclient: shutdown 2024-09-16T14:43:53.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:53.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 1 -- 172.21.15.189:0/394443299 >> v1:172.21.15.189:6789/0 conn(0x7f1fe009e950 legacy=0x7f1fe013a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:53.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 1 -- 172.21.15.189:0/394443299 shutdown_connections 2024-09-16T14:43:53.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 1 -- 172.21.15.189:0/394443299 >> 172.21.15.189:0/394443299 conn(0x7f1fe00a3380 msgr2=0x7f1fe009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:53.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 1 -- 172.21.15.189:0/394443299 shutdown_connections 2024-09-16T14:43:53.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 1 -- 172.21.15.189:0/394443299 wait complete. 2024-09-16T14:43:53.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 10 monclient: build_initial_monmap 2024-09-16T14:43:53.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 10 monclient: monmap: 2024-09-16T14:43:53.785 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:53.785 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:53.785 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:53.785 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:53.785 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:53.785 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:53.785 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:53.785 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:53.785 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:53.785 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 1 Processor -- start 2024-09-16T14:43:53.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 1 -- start start 2024-09-16T14:43:53.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 10 monclient: init 2024-09-16T14:43:53.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:53.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:43:53.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 10 monclient(hunting): picked mon.noname-b con 0x7f1fe009e950 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:53.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 10 monclient(hunting): picked mon.noname-a con 0x7f1fe009edd0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:53.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 10 monclient(hunting): picked mon.noname-c con 0x7f1fe0131630 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:53.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe0120360 con 0x7f1fe009edd0 2024-09-16T14:43:53.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe0131070 con 0x7f1fe009e950 2024-09-16T14:43:53.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe009a670 con 0x7f1fe0131630 2024-09-16T14:43:53.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:53.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 10 monclient(hunting): authenticate will time out at 1015.890373s 2024-09-16T14:43:53.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1fe009e950 0x7f1fe0130960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51618/0 (socket says 172.21.15.189:51618) 2024-09-16T14:43:53.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdf7fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f1fe009edd0 legacy=0x7f1fe01ce370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:53.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdffff640 1 -- 172.21.15.189:0/1014715246 learned_addr learned my addr 172.21.15.189:0/1014715246 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:53.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834676339 0 0) 0x7f1fe0131070 con 0x7f1fe009e950 2024-09-16T14:43:53.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient(hunting): _negotiate 2024-09-16T14:43:53.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:53.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:53.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient(hunting): my global_id is 4603 2024-09-16T14:43:53.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fcc003530 con 0x7f1fe009e950 2024-09-16T14:43:53.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2708402037 0 0) 0x7f1fcc003530 con 0x7f1fe009e950 2024-09-16T14:43:53.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fe0131070 con 0x7f1fe009e950 2024-09-16T14:43:53.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1fd0001940 con 0x7f1fe009e950 2024-09-16T14:43:53.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:53.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:53.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient(hunting): dump: 2024-09-16T14:43:53.787 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:53.787 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:53.787 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.788 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.788 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:53.788 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:53.788 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:53.788 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:53.788 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:53.788 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:53.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935493827 0 0) 0x7f1fe009a670 con 0x7f1fe0131630 2024-09-16T14:43:53.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient(hunting): _negotiate 2024-09-16T14:43:53.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:53.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:53.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient(hunting): my global_id is 4580 2024-09-16T14:43:53.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fcc003530 con 0x7f1fe0131630 2024-09-16T14:43:53.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1479279956 0 0) 0x7f1fe0131070 con 0x7f1fe009e950 2024-09-16T14:43:53.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 >> v1:172.21.15.26:6790/0 conn(0x7f1fe0131630 legacy=0x7f1fe01d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:53.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 >> v1:172.21.15.26:6789/0 conn(0x7f1fe009edd0 legacy=0x7f1fe01ce370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:53.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient: _finish_hunting 0 2024-09-16T14:43:53.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 1 monclient: found mon.b 2024-09-16T14:43:53.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:53.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe006b320 con 0x7f1fe009e950 2024-09-16T14:43:53.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient: _finish_auth 0 2024-09-16T14:43:53.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient: _check_auth_tickets 2024-09-16T14:43:53.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:53.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 5 monclient: authenticate success, global_id 4603 2024-09-16T14:43:53.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 10 monclient: _renew_subs 2024-09-16T14:43:53.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 1 -- 172.21.15.189:0/1014715246 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe0131280 con 0x7f1fe009e950 2024-09-16T14:43:53.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 10 monclient: _renew_subs 2024-09-16T14:43:53.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fe631e640 1 -- 172.21.15.189:0/1014715246 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe011ebd0 con 0x7f1fe009e950 2024-09-16T14:43:53.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fd0001d10 con 0x7f1fe009e950 2024-09-16T14:43:53.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:53.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1fd0003af0 con 0x7f1fe009e950 2024-09-16T14:43:53.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:53.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:53.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fde7fc640 4 set_mon_vals no callback set 2024-09-16T14:43:53.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.771+0000 7f1fdcff9640 10 monclient: dump: 2024-09-16T14:43:53.791 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:53.791 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:53.791 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.791 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:53.791 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:53.791 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:53.791 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:53.791 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:53.791 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:53.791 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:53.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.775+0000 7f1fe631e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:53.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.775+0000 7f1fe631e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:53.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.775+0000 7f1fe631e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.775+0000 7f1fe631e640 1 -- 172.21.15.189:0/1014715246 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc0005050 con 0x7f1fe009e950 2024-09-16T14:43:53.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.775+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1fd0003c50 con 0x7f1fe009e950 2024-09-16T14:43:53.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.775+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3889+0+0 (unknown 2825553975 0 0) 0x7f1fd0043360 con 0x7f1fe009e950 2024-09-16T14:43:53.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.775+0000 7f1fdf7fe640 10 monclient: get_auth_request con 0x7f1fcc0361d0 auth_method 0 2024-09-16T14:43:53.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.779+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1fd0014820 con 0x7f1fe009e950 2024-09-16T14:43:53.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.779+0000 7f1fdcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:53.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.779+0000 7f1fdcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:53.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.923+0000 7f1fe631e640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T14:43:53.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.923+0000 7f1fe631e640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T14:43:53.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.923+0000 7f1fe631e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:53.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:53.923+0000 7f1fe631e640 1 -- 172.21.15.189:0/1014715246 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"} v 0) v1 -- 0x7f1fc0005880 con 0x7f1fe009e950 2024-09-16T14:43:54.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:54.099+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}]=0 set pool 3 pg_num_min to 0 v23) v1 ==== 143+0+0 (unknown 3313309248 0 0) 0x7f1fd0012c90 con 0x7f1fe009e950 2024-09-16T14:43:54.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:54.099+0000 7f1fdcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T14:43:54.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:54.099+0000 7f1fdcff9640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_min to 0 2024-09-16T14:43:54.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:54.187+0000 7f1fe631e640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T14:43:54.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:54.187+0000 7f1fe631e640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T14:43:54.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:54.187+0000 7f1fe631e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:54.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:54.187+0000 7f1fe631e640 1 -- 172.21.15.189:0/1014715246 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"} v 0) v1 -- 0x7f1fc0005340 con 0x7f1fe009e950 2024-09-16T14:43:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.107+0000 7f1fdcff9640 1 -- 172.21.15.189:0/1014715246 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}]=0 set pool 3 pg_num_min to 0 v24) v1 ==== 143+0+0 (unknown 2342121220 0 0) 0x7f1fd0014410 con 0x7f1fe009e950 2024-09-16T14:43:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.107+0000 7f1fdcff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T14:43:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.107+0000 7f1fdcff9640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_min to 0 2024-09-16T14:43:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 pg_num_min to 0 2024-09-16T14:43:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.111+0000 7f1fe631e640 1 -- 172.21.15.189:0/1014715246 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1fcc0361d0 legacy=0x7f1fcc038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.111+0000 7f1fe631e640 10 monclient: shutdown 2024-09-16T14:43:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.111+0000 7f1fe631e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.111+0000 7f1fe631e640 1 -- 172.21.15.189:0/1014715246 >> v1:172.21.15.189:6789/0 conn(0x7f1fe009e950 legacy=0x7f1fe0130960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.111+0000 7f1fe631e640 1 -- 172.21.15.189:0/1014715246 shutdown_connections 2024-09-16T14:43:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.111+0000 7f1fe631e640 1 -- 172.21.15.189:0/1014715246 >> 172.21.15.189:0/1014715246 conn(0x7f1fe00a3380 msgr2=0x7f1fe00707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.111+0000 7f1fe631e640 1 -- 172.21.15.189:0/1014715246 shutdown_connections 2024-09-16T14:43:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.111+0000 7f1fe631e640 1 -- 172.21.15.189:0/1014715246 wait complete. 2024-09-16T14:43:55.135 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_min 2024-09-16T14:43:55.135 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:43:55.135 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool get testpool1234 pg_num_min 2024-09-16T14:43:55.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.255+0000 7f858bf60640 10 monclient: get_monmap_and_config 2024-09-16T14:43:55.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.255+0000 7f858bf60640 10 monclient: build_initial_monmap 2024-09-16T14:43:55.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.255+0000 7f858bf60640 10 monclient: monmap: 2024-09-16T14:43:55.261 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:55.261 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:55.261 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:55.260905+0000 2024-09-16T14:43:55.261 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:55.260905+0000 2024-09-16T14:43:55.261 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:55.261 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:55.262 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:55.262 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:55.262 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:55.262 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:55.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.255+0000 7f858bf60640 1 Processor -- start 2024-09-16T14:43:55.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.255+0000 7f858bf60640 1 -- start start 2024-09-16T14:43:55.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.255+0000 7f858bf60640 10 monclient: init 2024-09-16T14:43:55.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:55.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:43:55.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient(hunting): picked mon.noname-a con 0x7f858409df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:55.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient(hunting): picked mon.noname-c con 0x7f858409e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:55.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient(hunting): picked mon.noname-b con 0x7f858409e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:55.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858409ee20 con 0x7f858409df10 2024-09-16T14:43:55.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858409f100 con 0x7f858409e2f0 2024-09-16T14:43:55.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858409f3e0 con 0x7f858409e740 2024-09-16T14:43:55.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:55.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient(hunting): authenticate will time out at 1017.376618s 2024-09-16T14:43:55.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f85894fb640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f858409e2f0 0x7f85841378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51632/0 (socket says 172.21.15.189:51632) 2024-09-16T14:43:55.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f85894fb640 1 -- 172.21.15.189:0/408051202 learned_addr learned my addr 172.21.15.189:0/408051202 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:55.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8589cfc640 1 -- 172.21.15.189:0/408051202 >> v1:172.21.15.26:6789/0 conn(0x7f858409df10 legacy=0x7f858409d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:55.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 1 -- 172.21.15.189:0/408051202 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605592860 0 0) 0x7f858409f100 con 0x7f858409e2f0 2024-09-16T14:43:55.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 10 monclient(hunting): _negotiate 2024-09-16T14:43:55.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:55.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:55.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 10 monclient(hunting): my global_id is 4606 2024-09-16T14:43:55.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 1 -- 172.21.15.189:0/408051202 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8570003530 con 0x7f858409e2f0 2024-09-16T14:43:55.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 1 -- 172.21.15.189:0/408051202 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3978358455 0 0) 0x7f8570003530 con 0x7f858409e2f0 2024-09-16T14:43:55.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 1 -- 172.21.15.189:0/408051202 >> v1:172.21.15.26:6790/0 conn(0x7f858409e740 legacy=0x7f858413b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:55.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 1 -- 172.21.15.189:0/408051202 >> v1:172.21.15.26:6789/0 conn(0x7f858409df10 legacy=0x7f858409d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:55.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 10 monclient: _finish_hunting 0 2024-09-16T14:43:55.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 1 monclient: found mon.noname-c 2024-09-16T14:43:55.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:55.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:55.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 1 -- 172.21.15.189:0/408051202 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858409c690 con 0x7f858409e2f0 2024-09-16T14:43:55.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 10 monclient: _finish_auth 0 2024-09-16T14:43:55.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 10 monclient: _check_auth_tickets 2024-09-16T14:43:55.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:55.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 5 monclient: authenticate success, global_id 4606 2024-09-16T14:43:55.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 1 -- 172.21.15.189:0/408051202 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f857c001620 con 0x7f858409e2f0 2024-09-16T14:43:55.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:55.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:55.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:55.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 10 monclient: dump: 2024-09-16T14:43:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:55.273 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 1 -- 172.21.15.189:0/408051202 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f857c001940 con 0x7f858409e2f0 2024-09-16T14:43:55.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:55.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 1 -- 172.21.15.189:0/408051202 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f857c001d10 con 0x7f858409e2f0 2024-09-16T14:43:55.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:55.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 4 set_mon_vals no callback set 2024-09-16T14:43:55.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:55.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:55.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f8588cfa640 10 monclient: dump: 2024-09-16T14:43:55.274 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:55.275 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:55.275 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:55.275 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:55.275 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:55.275 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:55.275 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:55.275 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:55.275 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:55.275 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:55.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient: shutdown 2024-09-16T14:43:55.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:55.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 1 -- 172.21.15.189:0/408051202 >> v1:172.21.15.189:6789/0 conn(0x7f858409e2f0 legacy=0x7f85841378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:55.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 1 -- 172.21.15.189:0/408051202 shutdown_connections 2024-09-16T14:43:55.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 1 -- 172.21.15.189:0/408051202 >> 172.21.15.189:0/408051202 conn(0x7f85840a3320 msgr2=0x7f858409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:55.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 1 -- 172.21.15.189:0/408051202 shutdown_connections 2024-09-16T14:43:55.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 1 -- 172.21.15.189:0/408051202 wait complete. 2024-09-16T14:43:55.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient: build_initial_monmap 2024-09-16T14:43:55.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient: monmap: 2024-09-16T14:43:55.276 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:55.276 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:55.276 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:55.276 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:55.276 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:55.276 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:55.276 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:55.277 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:55.277 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:55.277 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:55.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 1 Processor -- start 2024-09-16T14:43:55.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 1 -- start start 2024-09-16T14:43:55.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient: init 2024-09-16T14:43:55.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:55.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:43:55.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient(hunting): picked mon.noname-c con 0x7f858409df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:55.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient(hunting): picked mon.noname-b con 0x7f858409e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:55.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient(hunting): picked mon.noname-a con 0x7f858409e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:55.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858411a6d0 con 0x7f858409e740 2024-09-16T14:43:55.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858406cdc0 con 0x7f858409e2f0 2024-09-16T14:43:55.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858406f940 con 0x7f858409df10 2024-09-16T14:43:55.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:55.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858bf60640 10 monclient(hunting): authenticate will time out at 1017.379659s 2024-09-16T14:43:55.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f85894fb640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f858409e2f0 0x7f858413a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51640/0 (socket says 172.21.15.189:51640) 2024-09-16T14:43:55.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f85894fb640 1 -- 172.21.15.189:0/914073533 learned_addr learned my addr 172.21.15.189:0/914073533 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:55.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f858a4fd640 1 -- 172.21.15.189:0/914073533 >> v1:172.21.15.26:6789/0 conn(0x7f858409e740 legacy=0x7f85841cf780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:55.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670980023 0 0) 0x7f858406cdc0 con 0x7f858409e2f0 2024-09-16T14:43:55.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f857a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:55.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f857a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:55.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f857a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:55.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.259+0000 7f857a7fc640 10 monclient(hunting): my global_id is 4609 2024-09-16T14:43:55.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8568003530 con 0x7f858409e2f0 2024-09-16T14:43:55.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1140099539 0 0) 0x7f8568003530 con 0x7f858409e2f0 2024-09-16T14:43:55.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f858406cdc0 con 0x7f858409e2f0 2024-09-16T14:43:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f857c0015e0 con 0x7f858409e2f0 2024-09-16T14:43:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 10 monclient(hunting): dump: 2024-09-16T14:43:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:55.280 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267682866 0 0) 0x7f858406f940 con 0x7f858409df10 2024-09-16T14:43:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:43:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 10 monclient(hunting): my global_id is 4586 2024-09-16T14:43:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8568003530 con 0x7f858409df10 2024-09-16T14:43:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 661611672 0 0) 0x7f858406cdc0 con 0x7f858409e2f0 2024-09-16T14:43:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 >> v1:172.21.15.26:6790/0 conn(0x7f858409df10 legacy=0x7f8584139c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 >> v1:172.21.15.26:6789/0 conn(0x7f858409e740 legacy=0x7f85841cf780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:43:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 1 monclient: found mon.b 2024-09-16T14:43:55.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:55.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:55.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858406b380 con 0x7f858409e2f0 2024-09-16T14:43:55.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:43:55.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:43:55.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:55.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f858bf60640 5 monclient: authenticate success, global_id 4609 2024-09-16T14:43:55.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f858bf60640 10 monclient: _renew_subs 2024-09-16T14:43:55.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f858bf60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:55.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f858bf60640 1 -- 172.21.15.189:0/914073533 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8584119e30 con 0x7f858409e2f0 2024-09-16T14:43:55.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f858bf60640 10 monclient: _renew_subs 2024-09-16T14:43:55.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f858bf60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:55.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f858bf60640 1 -- 172.21.15.189:0/914073533 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f858413c280 con 0x7f858409e2f0 2024-09-16T14:43:55.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f858bf60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:55.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f858bf60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:55.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f858bf60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:55.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f858bf60640 1 -- 172.21.15.189:0/914073533 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85840008e0 con 0x7f858409e2f0 2024-09-16T14:43:55.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f857c004cf0 con 0x7f858409e2f0 2024-09-16T14:43:55.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:55.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f857c005080 con 0x7f858409e2f0 2024-09-16T14:43:55.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:55.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:55.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 10 monclient: dump: 2024-09-16T14:43:55.282 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:55.282 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:55.282 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:55.282 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:55.283 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:55.283 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:55.283 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:55.283 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:55.283 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:55.283 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:55.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857bfff640 4 set_mon_vals no callback set 2024-09-16T14:43:55.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f857c0051e0 con 0x7f858409e2f0 2024-09-16T14:43:55.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(24..24 src has 1..24) v4 ==== 3873+0+0 (unknown 4066724946 0 0) 0x7f857c043410 con 0x7f858409e2f0 2024-09-16T14:43:55.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.263+0000 7f8589cfc640 10 monclient: get_auth_request con 0x7f85680361f0 auth_method 0 2024-09-16T14:43:55.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.267+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f857c014940 con 0x7f858409e2f0 2024-09-16T14:43:55.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.267+0000 7f857a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:55.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.267+0000 7f857a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:55.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.423+0000 7f858bf60640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T14:43:55.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.423+0000 7f858bf60640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T14:43:55.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.423+0000 7f858bf60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:55.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.423+0000 7f858bf60640 1 -- 172.21.15.189:0/914073533 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7f8584099cd0 con 0x7f858409e2f0 2024-09-16T14:43:55.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.427+0000 7f857a7fc640 1 -- 172.21.15.189:0/914073533 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=-2 option 'pg_num_min' is not set on pool 'testpool1234' v24) v1 ==== 158+0+0 (unknown 3652872207 0 0) 0x7f857c012db0 con 0x7f858409e2f0 2024-09-16T14:43:55.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.427+0000 7f857a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T14:43:55.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.427+0000 7f857a7fc640 10 monclient: _finish_command 2 = mon:2 option 'pg_num_min' is not set on pool 'testpool1234' 2024-09-16T14:43:55.430 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOENT: option 'pg_num_min' is not set on pool 'testpool1234' 2024-09-16T14:43:55.434 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:43:55.434 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set testpool1234 pg_num_max 0 2024-09-16T14:43:55.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 10 monclient: get_monmap_and_config 2024-09-16T14:43:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 10 monclient: build_initial_monmap 2024-09-16T14:43:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 10 monclient: monmap: 2024-09-16T14:43:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:55.555087+0000 2024-09-16T14:43:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:55.555087+0000 2024-09-16T14:43:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:55.555 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:55.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 1 Processor -- start 2024-09-16T14:43:55.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 1 -- start start 2024-09-16T14:43:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 10 monclient: init 2024-09-16T14:43:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:43:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 10 monclient(hunting): picked mon.noname-b con 0x7f9350099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 10 monclient(hunting): picked mon.noname-c con 0x7f935009c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 10 monclient(hunting): picked mon.noname-a con 0x7f935009e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9350099e50 con 0x7f935009e6d0 2024-09-16T14:43:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935009dbd0 con 0x7f935009c550 2024-09-16T14:43:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935009ee20 con 0x7f9350099a70 2024-09-16T14:43:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 10 monclient(hunting): authenticate will time out at 1017.670719s 2024-09-16T14:43:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9354ed0640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f935009c550 0x7f93501379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51656/0 (socket says 172.21.15.189:51656) 2024-09-16T14:43:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9354ed0640 1 -- 172.21.15.189:0/1909745717 learned_addr learned my addr 172.21.15.189:0/1909745717 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:55.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9355ed2640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f935009e6d0 legacy=0x7f935013b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:55.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 1 -- 172.21.15.189:0/1909745717 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167920076 0 0) 0x7f935009dbd0 con 0x7f935009c550 2024-09-16T14:43:55.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 10 monclient(hunting): _negotiate 2024-09-16T14:43:55.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:55.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:55.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 10 monclient(hunting): my global_id is 4612 2024-09-16T14:43:55.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 1 -- 172.21.15.189:0/1909745717 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f933c003530 con 0x7f935009c550 2024-09-16T14:43:55.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 1 -- 172.21.15.189:0/1909745717 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1232047720 0 0) 0x7f933c003530 con 0x7f935009c550 2024-09-16T14:43:55.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 1 -- 172.21.15.189:0/1909745717 >> v1:172.21.15.26:6790/0 conn(0x7f9350099a70 legacy=0x7f935009dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:55.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 1 -- 172.21.15.189:0/1909745717 >> v1:172.21.15.26:6789/0 conn(0x7f935009e6d0 legacy=0x7f935013b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:55.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 10 monclient: _finish_hunting 0 2024-09-16T14:43:55.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 1 monclient: found mon.noname-c 2024-09-16T14:43:55.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:55.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:55.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 1 -- 172.21.15.189:0/1909745717 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935009f100 con 0x7f935009c550 2024-09-16T14:43:55.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 10 monclient: _finish_auth 0 2024-09-16T14:43:55.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:55.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:55.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 1 -- 172.21.15.189:0/1909745717 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f934c001620 con 0x7f935009c550 2024-09-16T14:43:55.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 5 monclient: authenticate success, global_id 4612 2024-09-16T14:43:55.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:55.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:55.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:55.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 10 monclient: dump: 2024-09-16T14:43:55.564 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:55.564 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:55.564 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:55.565 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:55.565 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:55.565 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:55.565 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:55.565 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:55.565 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:55.565 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:55.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:55.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 1 -- 172.21.15.189:0/1909745717 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f934c0018d0 con 0x7f935009c550 2024-09-16T14:43:55.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:55.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:55.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9357935640 4 set_mon_vals no callback set 2024-09-16T14:43:55.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 1 -- 172.21.15.189:0/1909745717 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f934c001ca0 con 0x7f935009c550 2024-09-16T14:43:55.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:55.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:55.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.551+0000 7f9347fff640 10 monclient: dump: 2024-09-16T14:43:55.566 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:55.566 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:55.566 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:55.566 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:55.566 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:55.566 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:55.566 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:55.566 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:55.566 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:55.566 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:55.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient: shutdown 2024-09-16T14:43:55.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:55.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 1 -- 172.21.15.189:0/1909745717 >> v1:172.21.15.189:6789/0 conn(0x7f935009c550 legacy=0x7f93501379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:55.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 1 -- 172.21.15.189:0/1909745717 shutdown_connections 2024-09-16T14:43:55.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 1 -- 172.21.15.189:0/1909745717 >> 172.21.15.189:0/1909745717 conn(0x7f93500a3380 msgr2=0x7f935009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:55.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 1 -- 172.21.15.189:0/1909745717 shutdown_connections 2024-09-16T14:43:55.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 1 -- 172.21.15.189:0/1909745717 wait complete. 2024-09-16T14:43:55.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient: build_initial_monmap 2024-09-16T14:43:55.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient: monmap: 2024-09-16T14:43:55.567 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:55.567 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:55.567 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:55.567 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:55.568 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:55.568 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:55.568 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:55.568 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:55.568 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:55.568 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:55.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 1 Processor -- start 2024-09-16T14:43:55.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 1 -- start start 2024-09-16T14:43:55.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient: init 2024-09-16T14:43:55.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:55.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:43:55.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient(hunting): picked mon.noname-a con 0x7f9350099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:55.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient(hunting): picked mon.noname-c con 0x7f935009c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient(hunting): picked mon.noname-b con 0x7f935009e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935009cf20 con 0x7f9350099a70 2024-09-16T14:43:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935006ce30 con 0x7f935009e6d0 2024-09-16T14:43:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935006b320 con 0x7f935009c550 2024-09-16T14:43:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient(hunting): authenticate will time out at 1017.673738s 2024-09-16T14:43:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9355ed2640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f935009e6d0 0x7f93501cf700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51666/0 (socket says 172.21.15.189:51666) 2024-09-16T14:43:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9355ed2640 1 -- 172.21.15.189:0/1748704042 learned_addr learned my addr 172.21.15.189:0/1748704042 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f93556d1640 1 -- 172.21.15.189:0/1748704042 >> v1:172.21.15.26:6789/0 conn(0x7f9350099a70 legacy=0x7f9350138b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558203480 0 0) 0x7f935006ce30 con 0x7f935009e6d0 2024-09-16T14:43:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:55.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient(hunting): my global_id is 4615 2024-09-16T14:43:55.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9334003530 con 0x7f935009e6d0 2024-09-16T14:43:55.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 63020009 0 0) 0x7f9334003530 con 0x7f935009e6d0 2024-09-16T14:43:55.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f935006ce30 con 0x7f935009e6d0 2024-09-16T14:43:55.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9348004450 con 0x7f935009e6d0 2024-09-16T14:43:55.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:55.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:55.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient(hunting): dump: 2024-09-16T14:43:55.570 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:55.570 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:55.570 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:55.570 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:55.570 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:55.571 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:55.571 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:55.571 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:55.571 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:55.571 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:55.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:55.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825846320 0 0) 0x7f935006b320 con 0x7f935009c550 2024-09-16T14:43:55.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:55.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:55.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:55.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient(hunting): my global_id is 4592 2024-09-16T14:43:55.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9334003530 con 0x7f935009c550 2024-09-16T14:43:55.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1742167423 0 0) 0x7f935006ce30 con 0x7f935009e6d0 2024-09-16T14:43:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 >> v1:172.21.15.26:6790/0 conn(0x7f935009c550 legacy=0x7f9350139230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 >> v1:172.21.15.26:6789/0 conn(0x7f9350099a70 legacy=0x7f9350138b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:43:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 1 monclient: found mon.b 2024-09-16T14:43:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935013c4e0 con 0x7f935009e6d0 2024-09-16T14:43:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient: _finish_auth 0 2024-09-16T14:43:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:43:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 5 monclient: authenticate success, global_id 4615 2024-09-16T14:43:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient: _renew_subs 2024-09-16T14:43:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 1 -- 172.21.15.189:0/1748704042 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9350119620 con 0x7f935009e6d0 2024-09-16T14:43:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient: _renew_subs 2024-09-16T14:43:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 1 -- 172.21.15.189:0/1748704042 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f935013a6d0 con 0x7f935009e6d0 2024-09-16T14:43:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9348004770 con 0x7f935009e6d0 2024-09-16T14:43:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9348004b00 con 0x7f935009e6d0 2024-09-16T14:43:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f93477fe640 4 set_mon_vals no callback set 2024-09-16T14:43:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 10 monclient: dump: 2024-09-16T14:43:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:55.574 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:55.574 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:55.574 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:55.574 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:55.574 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:55.574 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:55.574 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:55.574 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:55.574 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:55.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9348004ca0 con 0x7f935009e6d0 2024-09-16T14:43:55.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:55.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:55.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:55.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9357935640 1 -- 172.21.15.189:0/1748704042 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9330005050 con 0x7f935009e6d0 2024-09-16T14:43:55.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(24..24 src has 1..24) v4 ==== 3873+0+0 (unknown 4066724946 0 0) 0x7f9348043610 con 0x7f935009e6d0 2024-09-16T14:43:55.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.555+0000 7f93556d1640 10 monclient: get_auth_request con 0x7f9334036130 auth_method 0 2024-09-16T14:43:55.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.559+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f93480054a0 con 0x7f935009e6d0 2024-09-16T14:43:55.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.559+0000 7f9345ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:55.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.559+0000 7f9345ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:55.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.723+0000 7f9357935640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T14:43:55.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.723+0000 7f9357935640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T14:43:55.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.723+0000 7f9357935640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:55.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:55.723+0000 7f9357935640 1 -- 172.21.15.189:0/1748704042 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"} v 0) v1 -- 0x7f9330005880 con 0x7f935009e6d0 2024-09-16T14:43:56.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:56.107+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}]=0 set pool 3 pg_num_max to 0 v25) v1 ==== 143+0+0 (unknown 4129009478 0 0) 0x7f9348012ef0 con 0x7f935009e6d0 2024-09-16T14:43:56.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:56.107+0000 7f9345ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T14:43:56.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:56.107+0000 7f9345ffb640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_max to 0 2024-09-16T14:43:56.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:56.207+0000 7f9357935640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T14:43:56.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:56.207+0000 7f9357935640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T14:43:56.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:56.207+0000 7f9357935640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:56.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:56.207+0000 7f9357935640 1 -- 172.21.15.189:0/1748704042 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"} v 0) v1 -- 0x7f9330005340 con 0x7f935009e6d0 2024-09-16T14:43:57.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.123+0000 7f9345ffb640 1 -- 172.21.15.189:0/1748704042 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}]=0 set pool 3 pg_num_max to 0 v26) v1 ==== 143+0+0 (unknown 3024910647 0 0) 0x7f9348014850 con 0x7f935009e6d0 2024-09-16T14:43:57.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.123+0000 7f9345ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T14:43:57.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.123+0000 7f9345ffb640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_max to 0 2024-09-16T14:43:57.129 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 pg_num_max to 0 2024-09-16T14:43:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.127+0000 7f9357935640 1 -- 172.21.15.189:0/1748704042 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9334036130 legacy=0x7f93340385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.127+0000 7f9357935640 10 monclient: shutdown 2024-09-16T14:43:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.127+0000 7f9357935640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.127+0000 7f9357935640 1 -- 172.21.15.189:0/1748704042 >> v1:172.21.15.189:6789/0 conn(0x7f935009e6d0 legacy=0x7f93501cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.127+0000 7f9357935640 1 -- 172.21.15.189:0/1748704042 shutdown_connections 2024-09-16T14:43:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.127+0000 7f9357935640 1 -- 172.21.15.189:0/1748704042 >> 172.21.15.189:0/1748704042 conn(0x7f93500a3380 msgr2=0x7f93500a0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:57.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.127+0000 7f9357935640 1 -- 172.21.15.189:0/1748704042 shutdown_connections 2024-09-16T14:43:57.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.127+0000 7f9357935640 1 -- 172.21.15.189:0/1748704042 wait complete. 2024-09-16T14:43:57.149 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_max 2024-09-16T14:43:57.149 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:43:57.149 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool get testpool1234 pg_num_max 2024-09-16T14:43:57.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.263+0000 7f9943a0c640 10 monclient: get_monmap_and_config 2024-09-16T14:43:57.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.263+0000 7f9943a0c640 10 monclient: build_initial_monmap 2024-09-16T14:43:57.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.263+0000 7f9943a0c640 10 monclient: monmap: 2024-09-16T14:43:57.269 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:57.269 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:57.269 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:57.269478+0000 2024-09-16T14:43:57.269 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:57.269478+0000 2024-09-16T14:43:57.269 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:57.269 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:57.270 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:57.270 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:57.270 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:57.270 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:57.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 1 Processor -- start 2024-09-16T14:43:57.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 1 -- start start 2024-09-16T14:43:57.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 10 monclient: init 2024-09-16T14:43:57.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:57.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:43:57.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 10 monclient(hunting): picked mon.noname-c con 0x7f993c1315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:57.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 10 monclient(hunting): picked mon.noname-a con 0x7f993c09b320 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:57.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 10 monclient(hunting): picked mon.noname-b con 0x7f993c132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:57.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993c09edd0 con 0x7f993c09b320 2024-09-16T14:43:57.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993c09bc00 con 0x7f993c1315a0 2024-09-16T14:43:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993c09e900 con 0x7f993c132090 2024-09-16T14:43:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 10 monclient(hunting): authenticate will time out at 1019.385121s 2024-09-16T14:43:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f99417a8640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f993c1315a0 0x7f993c131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51670/0 (socket says 172.21.15.189:51670) 2024-09-16T14:43:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f99417a8640 1 -- 172.21.15.189:0/1238892641 learned_addr learned my addr 172.21.15.189:0/1238892641 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9940fa7640 1 -- 172.21.15.189:0/1238892641 >> v1:172.21.15.26:6789/0 conn(0x7f993c09b320 legacy=0x7f993c136ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 1 -- 172.21.15.189:0/1238892641 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056789326 0 0) 0x7f993c09bc00 con 0x7f993c1315a0 2024-09-16T14:43:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 10 monclient(hunting): _negotiate 2024-09-16T14:43:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 10 monclient(hunting): my global_id is 4618 2024-09-16T14:43:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 1 -- 172.21.15.189:0/1238892641 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9928005740 con 0x7f993c1315a0 2024-09-16T14:43:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 1 -- 172.21.15.189:0/1238892641 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1162179939 0 0) 0x7f9928005740 con 0x7f993c1315a0 2024-09-16T14:43:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 1 -- 172.21.15.189:0/1238892641 >> v1:172.21.15.26:6790/0 conn(0x7f993c132090 legacy=0x7f993c13a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 1 -- 172.21.15.189:0/1238892641 >> v1:172.21.15.26:6789/0 conn(0x7f993c09b320 legacy=0x7f993c136ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 10 monclient: _finish_hunting 0 2024-09-16T14:43:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 1 monclient: found mon.noname-c 2024-09-16T14:43:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 1 -- 172.21.15.189:0/1238892641 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993c09b920 con 0x7f993c1315a0 2024-09-16T14:43:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 10 monclient: _finish_auth 0 2024-09-16T14:43:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 1 -- 172.21.15.189:0/1238892641 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9938001620 con 0x7f993c1315a0 2024-09-16T14:43:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 10 monclient: dump: 2024-09-16T14:43:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:57.276 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 5 monclient: authenticate success, global_id 4618 2024-09-16T14:43:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 1 -- 172.21.15.189:0/1238892641 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9938001940 con 0x7f993c1315a0 2024-09-16T14:43:57.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:57.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 1 -- 172.21.15.189:0/1238892641 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9938001cd0 con 0x7f993c1315a0 2024-09-16T14:43:57.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:57.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:57.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9933fff640 10 monclient: dump: 2024-09-16T14:43:57.277 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:57.277 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:57.277 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:57.277 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:57.278 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:57.278 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:57.278 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:57.278 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:57.278 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:57.278 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:57.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:57.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 4 set_mon_vals no callback set 2024-09-16T14:43:57.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 10 monclient: shutdown 2024-09-16T14:43:57.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:57.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 1 -- 172.21.15.189:0/1238892641 >> v1:172.21.15.189:6789/0 conn(0x7f993c1315a0 legacy=0x7f993c131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:57.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 1 -- 172.21.15.189:0/1238892641 shutdown_connections 2024-09-16T14:43:57.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 1 -- 172.21.15.189:0/1238892641 >> 172.21.15.189:0/1238892641 conn(0x7f993c0a3320 msgr2=0x7f993c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:57.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 1 -- 172.21.15.189:0/1238892641 shutdown_connections 2024-09-16T14:43:57.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 1 -- 172.21.15.189:0/1238892641 wait complete. 2024-09-16T14:43:57.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 10 monclient: build_initial_monmap 2024-09-16T14:43:57.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 10 monclient: monmap: 2024-09-16T14:43:57.279 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:57.279 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:57.279 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:57.279 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:57.279 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:57.279 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:57.279 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:57.279 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:57.279 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:57.279 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:57.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 1 Processor -- start 2024-09-16T14:43:57.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 1 -- start start 2024-09-16T14:43:57.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.267+0000 7f9943a0c640 10 monclient: init 2024-09-16T14:43:57.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:57.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:43:57.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 10 monclient(hunting): picked mon.noname-b con 0x7f993c1315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:57.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 10 monclient(hunting): picked mon.noname-c con 0x7f993c132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:57.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 10 monclient(hunting): picked mon.noname-a con 0x7f993c1c9120 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:57.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993c09cd20 con 0x7f993c1c9120 2024-09-16T14:43:57.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993c09aa50 con 0x7f993c1315a0 2024-09-16T14:43:57.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993c11ff90 con 0x7f993c132090 2024-09-16T14:43:57.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:57.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 10 monclient(hunting): authenticate will time out at 1019.388182s 2024-09-16T14:43:57.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f99417a8640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f993c1315a0 0x7f993c12fc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51684/0 (socket says 172.21.15.189:51684) 2024-09-16T14:43:57.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f99417a8640 1 -- 172.21.15.189:0/3461330071 learned_addr learned my addr 172.21.15.189:0/3461330071 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:57.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9941fa9640 1 -- 172.21.15.189:0/3461330071 >> v1:172.21.15.26:6789/0 conn(0x7f993c1c9120 legacy=0x7f993c1d1b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:57.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041847764 0 0) 0x7f993c09aa50 con 0x7f993c1315a0 2024-09-16T14:43:57.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:57.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:57.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:57.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient(hunting): my global_id is 4621 2024-09-16T14:43:57.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9920003530 con 0x7f993c1315a0 2024-09-16T14:43:57.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1907023846 0 0) 0x7f9920003530 con 0x7f993c1315a0 2024-09-16T14:43:57.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f993c09aa50 con 0x7f993c1315a0 2024-09-16T14:43:57.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f99380015e0 con 0x7f993c1315a0 2024-09-16T14:43:57.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:57.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:57.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient(hunting): dump: 2024-09-16T14:43:57.285 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:57.285 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:57.285 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:57.285 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:57.285 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:57.285 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:57.285 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:57.285 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:57.285 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:57.285 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:57.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:57.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732966301 0 0) 0x7f993c11ff90 con 0x7f993c132090 2024-09-16T14:43:57.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:57.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:57.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:57.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient(hunting): my global_id is 4598 2024-09-16T14:43:57.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9920003530 con 0x7f993c132090 2024-09-16T14:43:57.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1347810186 0 0) 0x7f993c09aa50 con 0x7f993c1315a0 2024-09-16T14:43:57.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 >> v1:172.21.15.26:6790/0 conn(0x7f993c132090 legacy=0x7f993c1ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:57.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 >> v1:172.21.15.26:6789/0 conn(0x7f993c1c9120 legacy=0x7f993c1d1b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:57.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:43:57.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 1 monclient: found mon.b 2024-09-16T14:43:57.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:57.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:57.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993c09a640 con 0x7f993c1315a0 2024-09-16T14:43:57.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient: _finish_auth 0 2024-09-16T14:43:57.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:43:57.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:57.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 5 monclient: authenticate success, global_id 4621 2024-09-16T14:43:57.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 10 monclient: _renew_subs 2024-09-16T14:43:57.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:57.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 1 -- 172.21.15.189:0/3461330071 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f993c06b380 con 0x7f993c1315a0 2024-09-16T14:43:57.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 10 monclient: _renew_subs 2024-09-16T14:43:57.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:57.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 1 -- 172.21.15.189:0/3461330071 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f993c121330 con 0x7f993c1315a0 2024-09-16T14:43:57.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9938004cd0 con 0x7f993c1315a0 2024-09-16T14:43:57.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:57.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9938004ec0 con 0x7f993c1315a0 2024-09-16T14:43:57.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:57.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:57.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f99337fe640 4 set_mon_vals no callback set 2024-09-16T14:43:57.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 10 monclient: dump: 2024-09-16T14:43:57.288 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:57.288 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:57.288 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:57.288 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:57.288 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:57.288 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:57.288 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:57.289 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9938005020 con 0x7f993c1315a0 2024-09-16T14:43:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 3857+0+0 (unknown 1926085330 0 0) 0x7f9938043590 con 0x7f993c1315a0 2024-09-16T14:43:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9943a0c640 1 -- 172.21.15.189:0/3461330071 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f991c005050 con 0x7f993c1315a0 2024-09-16T14:43:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.271+0000 7f9940fa7640 10 monclient: get_auth_request con 0x7f9920036100 auth_method 0 2024-09-16T14:43:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.275+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9938014850 con 0x7f993c1315a0 2024-09-16T14:43:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.275+0000 7f9931ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.275+0000 7f9931ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:57.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.431+0000 7f9943a0c640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T14:43:57.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.431+0000 7f9943a0c640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T14:43:57.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.431+0000 7f9943a0c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:57.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.431+0000 7f9943a0c640 1 -- 172.21.15.189:0/3461330071 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7f991c005880 con 0x7f993c1315a0 2024-09-16T14:43:57.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.431+0000 7f9931ffb640 1 -- 172.21.15.189:0/3461330071 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=-2 option 'pg_num_max' is not set on pool 'testpool1234' v26) v1 ==== 158+0+0 (unknown 1126902761 0 0) 0x7f9938014670 con 0x7f993c1315a0 2024-09-16T14:43:57.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.431+0000 7f9931ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T14:43:57.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.431+0000 7f9931ffb640 10 monclient: _finish_command 2 = mon:2 option 'pg_num_max' is not set on pool 'testpool1234' 2024-09-16T14:43:57.435 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOENT: option 'pg_num_max' is not set on pool 'testpool1234' 2024-09-16T14:43:57.438 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:43:57.438 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete testpool1234 testpool1234 --yes-i-really-really-mean-it 2024-09-16T14:43:57.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 10 monclient: get_monmap_and_config 2024-09-16T14:43:57.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 10 monclient: build_initial_monmap 2024-09-16T14:43:57.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 10 monclient: monmap: 2024-09-16T14:43:57.555 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:57.555 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:57.555 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:57.555551+0000 2024-09-16T14:43:57.555 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:57.555551+0000 2024-09-16T14:43:57.555 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:57.556 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:57.556 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:57.556 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:57.556 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:57.556 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:57.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 1 Processor -- start 2024-09-16T14:43:57.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 1 -- start start 2024-09-16T14:43:57.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 10 monclient: init 2024-09-16T14:43:57.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:57.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:43:57.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 10 monclient(hunting): picked mon.noname-a con 0x7ff490070830 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:57.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 10 monclient(hunting): picked mon.noname-b con 0x7ff490071320 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:57.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 10 monclient(hunting): picked mon.noname-c con 0x7ff490134cf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:57.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff490135740 con 0x7ff490070830 2024-09-16T14:43:57.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff490135160 con 0x7ff490134cf0 2024-09-16T14:43:57.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49009d810 con 0x7ff490071320 2024-09-16T14:43:57.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:57.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 10 monclient(hunting): authenticate will time out at 1019.671172s 2024-09-16T14:43:57.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48f7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff490134cf0 0x7ff49013d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51688/0 (socket says 172.21.15.189:51688) 2024-09-16T14:43:57.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48f7fe640 1 -- 172.21.15.189:0/3882690939 learned_addr learned my addr 172.21.15.189:0/3882690939 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48effd640 1 -- 172.21.15.189:0/3882690939 >> v1:172.21.15.26:6789/0 conn(0x7ff490070830 legacy=0x7ff490070c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 1 -- 172.21.15.189:0/3882690939 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102135639 0 0) 0x7ff490135160 con 0x7ff490134cf0 2024-09-16T14:43:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 10 monclient(hunting): my global_id is 4624 2024-09-16T14:43:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 1 -- 172.21.15.189:0/3882690939 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff478003530 con 0x7ff490134cf0 2024-09-16T14:43:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 1 -- 172.21.15.189:0/3882690939 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1947225930 0 0) 0x7ff478003530 con 0x7ff490134cf0 2024-09-16T14:43:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 1 -- 172.21.15.189:0/3882690939 >> v1:172.21.15.26:6790/0 conn(0x7ff490071320 legacy=0x7ff490139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 1 -- 172.21.15.189:0/3882690939 >> v1:172.21.15.26:6789/0 conn(0x7ff490070830 legacy=0x7ff490070c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:43:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 1 monclient: found mon.noname-c 2024-09-16T14:43:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:57.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:57.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 1 -- 172.21.15.189:0/3882690939 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49009f1c0 con 0x7ff490134cf0 2024-09-16T14:43:57.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 10 monclient: _finish_auth 0 2024-09-16T14:43:57.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:43:57.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:57.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 1 -- 172.21.15.189:0/3882690939 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff4880019a0 con 0x7ff490134cf0 2024-09-16T14:43:57.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 5 monclient: authenticate success, global_id 4624 2024-09-16T14:43:57.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:57.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:57.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:57.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff48dffb640 10 monclient: dump: 2024-09-16T14:43:57.564 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:57.564 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:57.565 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:57.565 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:57.565 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:57.565 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:57.565 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:57.565 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:57.565 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:57.565 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:57.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.551+0000 7ff4957c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:57.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff48dffb640 1 -- 172.21.15.189:0/3882690939 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff488001bc0 con 0x7ff490134cf0 2024-09-16T14:43:57.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff48dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:57.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:57.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff48dffb640 1 -- 172.21.15.189:0/3882690939 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff488001fd0 con 0x7ff490134cf0 2024-09-16T14:43:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 4 set_mon_vals no callback set 2024-09-16T14:43:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff48dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff48dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff48dffb640 10 monclient: dump: 2024-09-16T14:43:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:57.566 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient: shutdown 2024-09-16T14:43:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 1 -- 172.21.15.189:0/3882690939 >> v1:172.21.15.189:6789/0 conn(0x7ff490134cf0 legacy=0x7ff49013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 1 -- 172.21.15.189:0/3882690939 shutdown_connections 2024-09-16T14:43:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 1 -- 172.21.15.189:0/3882690939 >> 172.21.15.189:0/3882690939 conn(0x7ff4900a3380 msgr2=0x7ff49009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 1 -- 172.21.15.189:0/3882690939 shutdown_connections 2024-09-16T14:43:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 1 -- 172.21.15.189:0/3882690939 wait complete. 2024-09-16T14:43:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient: build_initial_monmap 2024-09-16T14:43:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient: monmap: 2024-09-16T14:43:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:57.568 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:57.568 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:57.568 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:57.568 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:57.568 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:57.568 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:57.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 1 Processor -- start 2024-09-16T14:43:57.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 1 -- start start 2024-09-16T14:43:57.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient: init 2024-09-16T14:43:57.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:57.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:43:57.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient(hunting): picked mon.noname-a con 0x7ff490070830 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:57.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient(hunting): picked mon.noname-c con 0x7ff490071320 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:57.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient(hunting): picked mon.noname-b con 0x7ff490134cf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:57.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49006b320 con 0x7ff490070830 2024-09-16T14:43:57.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49011e2a0 con 0x7ff490134cf0 2024-09-16T14:43:57.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff490132e10 con 0x7ff490071320 2024-09-16T14:43:57.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:57.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient(hunting): authenticate will time out at 1019.674065s 2024-09-16T14:43:57.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff48f7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff490134cf0 0x7ff4901d1a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51694/0 (socket says 172.21.15.189:51694) 2024-09-16T14:43:57.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff48f7fe640 1 -- 172.21.15.189:0/3511506339 learned_addr learned my addr 172.21.15.189:0/3511506339 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:57.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff48effd640 1 -- 172.21.15.189:0/3511506339 >> v1:172.21.15.26:6789/0 conn(0x7ff490070830 legacy=0x7ff4901308c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:57.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 1 -- 172.21.15.189:0/3511506339 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512386326 0 0) 0x7ff49011e2a0 con 0x7ff490134cf0 2024-09-16T14:43:57.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 10 monclient(hunting): _negotiate 2024-09-16T14:43:57.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:57.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:57.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 10 monclient(hunting): my global_id is 4627 2024-09-16T14:43:57.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 1 -- 172.21.15.189:0/3511506339 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff47c003530 con 0x7ff490134cf0 2024-09-16T14:43:57.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 1 -- 172.21.15.189:0/3511506339 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3305006974 0 0) 0x7ff47c003530 con 0x7ff490134cf0 2024-09-16T14:43:57.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 1 -- 172.21.15.189:0/3511506339 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff49011e2a0 con 0x7ff490134cf0 2024-09-16T14:43:57.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 1 -- 172.21.15.189:0/3511506339 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff488001ed0 con 0x7ff490134cf0 2024-09-16T14:43:57.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:57.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:57.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 10 monclient(hunting): dump: 2024-09-16T14:43:57.570 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:57.570 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:57.571 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:57.571 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:57.571 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:57.571 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:57.571 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:57.571 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:57.571 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:57.571 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:57.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:57.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 1 -- 172.21.15.189:0/3511506339 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1570754708 0 0) 0x7ff49011e2a0 con 0x7ff490134cf0 2024-09-16T14:43:57.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 1 -- 172.21.15.189:0/3511506339 >> v1:172.21.15.26:6790/0 conn(0x7ff490071320 legacy=0x7ff4901ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:57.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 1 -- 172.21.15.189:0/3511506339 >> v1:172.21.15.26:6789/0 conn(0x7ff490070830 legacy=0x7ff4901308c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:57.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 10 monclient: _finish_hunting 0 2024-09-16T14:43:57.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 1 monclient: found mon.b 2024-09-16T14:43:57.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:57.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:57.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 1 -- 172.21.15.189:0/3511506339 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4901328c0 con 0x7ff490134cf0 2024-09-16T14:43:57.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 10 monclient: _finish_auth 0 2024-09-16T14:43:57.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 10 monclient: _check_auth_tickets 2024-09-16T14:43:57.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:57.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 5 monclient: authenticate success, global_id 4627 2024-09-16T14:43:57.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient: _renew_subs 2024-09-16T14:43:57.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:57.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 1 -- 172.21.15.189:0/3511506339 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff49011dfc0 con 0x7ff490134cf0 2024-09-16T14:43:57.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient: _renew_subs 2024-09-16T14:43:57.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:57.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 1 -- 172.21.15.189:0/3511506339 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49011bf40 con 0x7ff490134cf0 2024-09-16T14:43:57.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 1 -- 172.21.15.189:0/3511506339 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff488004f80 con 0x7ff490134cf0 2024-09-16T14:43:57.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:57.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 1 -- 172.21.15.189:0/3511506339 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff488005290 con 0x7ff490134cf0 2024-09-16T14:43:57.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:57.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:57.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff48d7fa640 4 set_mon_vals no callback set 2024-09-16T14:43:57.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 10 monclient: dump: 2024-09-16T14:43:57.573 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:57.573 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:57.573 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:57.573 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:57.573 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:57.573 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:57.574 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:57.574 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:57.574 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:57.574 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:57.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:57.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:57.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:57.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4957c5640 1 -- 172.21.15.189:0/3511506339 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff470005050 con 0x7ff490134cf0 2024-09-16T14:43:57.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.555+0000 7ff4777fe640 1 -- 172.21.15.189:0/3511506339 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff48800fd00 con 0x7ff490134cf0 2024-09-16T14:43:57.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.559+0000 7ff4777fe640 1 -- 172.21.15.189:0/3511506339 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 3857+0+0 (unknown 1926085330 0 0) 0x7ff488043790 con 0x7ff490134cf0 2024-09-16T14:43:57.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.559+0000 7ff48effd640 10 monclient: get_auth_request con 0x7ff47c036050 auth_method 0 2024-09-16T14:43:57.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.563+0000 7ff4777fe640 1 -- 172.21.15.189:0/3511506339 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff488014b00 con 0x7ff490134cf0 2024-09-16T14:43:57.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.563+0000 7ff4777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:57.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.563+0000 7ff4777fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:57.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.707+0000 7ff4957c5640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T14:43:57.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.707+0000 7ff4957c5640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T14:43:57.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.707+0000 7ff4957c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:57.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:57.707+0000 7ff4957c5640 1 -- 172.21.15.189:0/3511506339 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff470005300 con 0x7ff490134cf0 2024-09-16T14:43:58.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.123+0000 7ff4777fe640 1 -- 172.21.15.189:0/3511506339 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' removed v27) v1 ==== 176+0+0 (unknown 1150842730 0 0) 0x7ff4880131a0 con 0x7ff490134cf0 2024-09-16T14:43:58.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.123+0000 7ff4777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T14:43:58.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.123+0000 7ff4777fe640 10 monclient: _finish_command 2 = system:0 pool 'testpool1234' removed 2024-09-16T14:43:58.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.222+0000 7ff4957c5640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T14:43:58.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.222+0000 7ff4957c5640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T14:43:58.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.222+0000 7ff4957c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:58.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.222+0000 7ff4957c5640 1 -- 172.21.15.189:0/3511506339 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff4700010a0 con 0x7ff490134cf0 2024-09-16T14:43:58.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.226+0000 7ff4777fe640 1 -- 172.21.15.189:0/3511506339 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' does not exist v27) v1 ==== 183+0+0 (unknown 3664109341 0 0) 0x7ff488014920 con 0x7ff490134cf0 2024-09-16T14:43:58.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.226+0000 7ff4777fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T14:43:58.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.226+0000 7ff4777fe640 10 monclient: _finish_command 3 = system:0 pool 'testpool1234' does not exist 2024-09-16T14:43:58.230 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'testpool1234' does not exist 2024-09-16T14:43:58.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.226+0000 7ff4957c5640 1 -- 172.21.15.189:0/3511506339 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff47c036050 legacy=0x7ff47c038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:58.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.226+0000 7ff4957c5640 10 monclient: shutdown 2024-09-16T14:43:58.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.226+0000 7ff4957c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:58.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.226+0000 7ff4957c5640 1 -- 172.21.15.189:0/3511506339 >> v1:172.21.15.189:6789/0 conn(0x7ff490134cf0 legacy=0x7ff4901d1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:58.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.226+0000 7ff4957c5640 1 -- 172.21.15.189:0/3511506339 shutdown_connections 2024-09-16T14:43:58.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.226+0000 7ff4957c5640 1 -- 172.21.15.189:0/3511506339 >> 172.21.15.189:0/3511506339 conn(0x7ff4900a3380 msgr2=0x7ff4900a1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:58.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.226+0000 7ff4957c5640 1 -- 172.21.15.189:0/3511506339 shutdown_connections 2024-09-16T14:43:58.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.226+0000 7ff4957c5640 1 -- 172.21.15.189:0/3511506339 wait complete. 2024-09-16T14:43:58.248 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool create foo 123 123 invalid foo-profile foo-rule 2024-09-16T14:43:58.248 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:43:58.248 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create foo 123 123 invalid foo-profile foo-rule 2024-09-16T14:43:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9db0a1f640 10 monclient: get_monmap_and_config 2024-09-16T14:43:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9db0a1f640 10 monclient: build_initial_monmap 2024-09-16T14:43:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9db0a1f640 10 monclient: monmap: 2024-09-16T14:43:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:58.371965+0000 2024-09-16T14:43:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:58.371965+0000 2024-09-16T14:43:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:58.372 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:58.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9db0a1f640 1 Processor -- start 2024-09-16T14:43:58.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9db0a1f640 1 -- start start 2024-09-16T14:43:58.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9db0a1f640 10 monclient: init 2024-09-16T14:43:58.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9db0a1f640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:58.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9db0a1f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:43:58.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9db0a1f640 10 monclient(hunting): picked mon.noname-c con 0x7f9dac0a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:58.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9db0a1f640 10 monclient(hunting): picked mon.noname-b con 0x7f9dac0a25f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:58.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9db0a1f640 10 monclient(hunting): picked mon.noname-a con 0x7f9dac13b430 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:58.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9db0a1f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dac0a30b0 con 0x7f9dac13b430 2024-09-16T14:43:58.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9db0a1f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dac0a2e30 con 0x7f9dac0a1b00 2024-09-16T14:43:58.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9db0a1f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dac09eb50 con 0x7f9dac0a25f0 2024-09-16T14:43:58.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9db0a1f640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:58.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9db0a1f640 10 monclient(hunting): authenticate will time out at 1020.487625s 2024-09-16T14:43:58.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9daa59c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9dac0a1b00 0x7f9dac0a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51698/0 (socket says 172.21.15.189:51698) 2024-09-16T14:43:58.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9daa59c640 1 -- 172.21.15.189:0/543017493 learned_addr learned my addr 172.21.15.189:0/543017493 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:58.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9daad9d640 1 -- 172.21.15.189:0/543017493 >> v1:172.21.15.26:6789/0 conn(0x7f9dac13b430 legacy=0x7f9dac13d820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:58.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.366+0000 7f9da9d9b640 1 -- 172.21.15.189:0/543017493 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125893176 0 0) 0x7f9dac0a2e30 con 0x7f9dac0a1b00 2024-09-16T14:43:58.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:43:58.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:58.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:58.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 10 monclient(hunting): my global_id is 4630 2024-09-16T14:43:58.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 1 -- 172.21.15.189:0/543017493 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d98003530 con 0x7f9dac0a1b00 2024-09-16T14:43:58.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 1 -- 172.21.15.189:0/543017493 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 128527105 0 0) 0x7f9d98003530 con 0x7f9dac0a1b00 2024-09-16T14:43:58.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 1 -- 172.21.15.189:0/543017493 >> v1:172.21.15.26:6790/0 conn(0x7f9dac0a25f0 legacy=0x7f9dac139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:58.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 1 -- 172.21.15.189:0/543017493 >> v1:172.21.15.26:6789/0 conn(0x7f9dac13b430 legacy=0x7f9dac13d820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:58.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:43:58.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 1 monclient: found mon.noname-c 2024-09-16T14:43:58.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:58.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:58.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 1 -- 172.21.15.189:0/543017493 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dac09ee30 con 0x7f9dac0a1b00 2024-09-16T14:43:58.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 10 monclient: _finish_auth 0 2024-09-16T14:43:58.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:43:58.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:58.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 1 -- 172.21.15.189:0/543017493 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9d9c001620 con 0x7f9dac0a1b00 2024-09-16T14:43:58.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 5 monclient: authenticate success, global_id 4630 2024-09-16T14:43:58.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:58.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:58.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:58.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 10 monclient: dump: 2024-09-16T14:43:58.381 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:58.382 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:58.382 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:58.382 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:58.382 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:58.382 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:58.382 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:58.382 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:58.382 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:58.382 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:58.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:58.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 1 -- 172.21.15.189:0/543017493 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d9c001940 con 0x7f9dac0a1b00 2024-09-16T14:43:58.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:58.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:58.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 4 set_mon_vals no callback set 2024-09-16T14:43:58.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 1 -- 172.21.15.189:0/543017493 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9d9c001d10 con 0x7f9dac0a1b00 2024-09-16T14:43:58.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:58.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:58.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da9d9b640 10 monclient: dump: 2024-09-16T14:43:58.383 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:58.383 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:58.383 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:58.383 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:58.383 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:58.383 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:58.383 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:58.383 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:58.383 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:58.384 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:58.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 10 monclient: shutdown 2024-09-16T14:43:58.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:58.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 1 -- 172.21.15.189:0/543017493 >> v1:172.21.15.189:6789/0 conn(0x7f9dac0a1b00 legacy=0x7f9dac0a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:58.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 1 -- 172.21.15.189:0/543017493 shutdown_connections 2024-09-16T14:43:58.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 1 -- 172.21.15.189:0/543017493 >> 172.21.15.189:0/543017493 conn(0x7f9dac0a7dd0 msgr2=0x7f9dac0a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:58.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 1 -- 172.21.15.189:0/543017493 shutdown_connections 2024-09-16T14:43:58.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 1 -- 172.21.15.189:0/543017493 wait complete. 2024-09-16T14:43:58.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 10 monclient: build_initial_monmap 2024-09-16T14:43:58.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 10 monclient: monmap: 2024-09-16T14:43:58.384 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:58.384 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:58.384 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:58.384 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:58.385 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:58.385 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:58.385 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:58.385 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:58.385 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:58.385 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:58.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 1 Processor -- start 2024-09-16T14:43:58.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 1 -- start start 2024-09-16T14:43:58.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 10 monclient: init 2024-09-16T14:43:58.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:58.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:43:58.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 10 monclient(hunting): picked mon.noname-c con 0x7f9dac0a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:58.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 10 monclient(hunting): picked mon.noname-b con 0x7f9dac0a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:58.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 10 monclient(hunting): picked mon.noname-a con 0x7f9dac13b430 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:58.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dac11e8f0 con 0x7f9dac13b430 2024-09-16T14:43:58.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dac11f770 con 0x7f9dac0a25f0 2024-09-16T14:43:58.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dac09e2f0 con 0x7f9dac0a1b00 2024-09-16T14:43:58.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:58.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 10 monclient(hunting): authenticate will time out at 1020.490576s 2024-09-16T14:43:58.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da3fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9dac0a25f0 0x7f9dac13a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51712/0 (socket says 172.21.15.189:51712) 2024-09-16T14:43:58.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da3fff640 1 -- 172.21.15.189:0/1825936983 learned_addr learned my addr 172.21.15.189:0/1825936983 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:58.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9daad9d640 1 -- 172.21.15.189:0/1825936983 >> v1:172.21.15.26:6789/0 conn(0x7f9dac13b430 legacy=0x7f9dac1d1810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:58.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 1 -- 172.21.15.189:0/1825936983 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722999343 0 0) 0x7f9dac11f770 con 0x7f9dac0a25f0 2024-09-16T14:43:58.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:43:58.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:58.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:58.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient(hunting): my global_id is 4633 2024-09-16T14:43:58.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 1 -- 172.21.15.189:0/1825936983 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d8c003530 con 0x7f9dac0a25f0 2024-09-16T14:43:58.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 1 -- 172.21.15.189:0/1825936983 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2364296858 0 0) 0x7f9d8c003530 con 0x7f9dac0a25f0 2024-09-16T14:43:58.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 1 -- 172.21.15.189:0/1825936983 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9dac11f770 con 0x7f9dac0a25f0 2024-09-16T14:43:58.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 1 -- 172.21.15.189:0/1825936983 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9d94001940 con 0x7f9dac0a25f0 2024-09-16T14:43:58.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:58.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:58.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient(hunting): dump: 2024-09-16T14:43:58.387 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:58.387 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:58.387 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:58.387 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:58.387 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:58.387 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:58.388 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:58.388 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:58.388 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:58.388 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:58.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:58.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 1 -- 172.21.15.189:0/1825936983 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890571207 0 0) 0x7f9dac09e2f0 con 0x7f9dac0a1b00 2024-09-16T14:43:58.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:43:58.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:58.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:58.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient(hunting): my global_id is 4610 2024-09-16T14:43:58.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 1 -- 172.21.15.189:0/1825936983 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d8c003530 con 0x7f9dac0a1b00 2024-09-16T14:43:58.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 1 -- 172.21.15.189:0/1825936983 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3009375750 0 0) 0x7f9dac11f770 con 0x7f9dac0a25f0 2024-09-16T14:43:58.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 1 -- 172.21.15.189:0/1825936983 >> v1:172.21.15.26:6790/0 conn(0x7f9dac0a1b00 legacy=0x7f9dac1c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:58.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 1 -- 172.21.15.189:0/1825936983 >> v1:172.21.15.26:6789/0 conn(0x7f9dac13b430 legacy=0x7f9dac1d1810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:58.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:43:58.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 1 monclient: found mon.b 2024-09-16T14:43:58.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:58.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:58.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 1 -- 172.21.15.189:0/1825936983 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dac120010 con 0x7f9dac0a25f0 2024-09-16T14:43:58.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:43:58.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:43:58.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:58.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 5 monclient: authenticate success, global_id 4633 2024-09-16T14:43:58.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 10 monclient: _renew_subs 2024-09-16T14:43:58.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:58.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 1 -- 172.21.15.189:0/1825936983 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dac0a5b40 con 0x7f9dac0a25f0 2024-09-16T14:43:58.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 10 monclient: _renew_subs 2024-09-16T14:43:58.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:58.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9db0a1f640 1 -- 172.21.15.189:0/1825936983 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dac11e600 con 0x7f9dac0a25f0 2024-09-16T14:43:58.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 1 -- 172.21.15.189:0/1825936983 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d94001d10 con 0x7f9dac0a25f0 2024-09-16T14:43:58.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:58.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 1 -- 172.21.15.189:0/1825936983 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9d94003af0 con 0x7f9dac0a25f0 2024-09-16T14:43:58.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:58.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:58.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da2ffd640 10 monclient: dump: 2024-09-16T14:43:58.390 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:58.390 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:58.390 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:58.390 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:58.390 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:58.390 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:58.391 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:58.391 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:58.391 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:58.391 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:58.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.370+0000 7f9da959a640 4 set_mon_vals no callback set 2024-09-16T14:43:58.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.374+0000 7f9da2ffd640 1 -- 172.21.15.189:0/1825936983 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9d94003c50 con 0x7f9dac0a25f0 2024-09-16T14:43:58.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.374+0000 7f9db0a1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:58.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.374+0000 7f9db0a1f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:58.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.374+0000 7f9db0a1f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:58.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.374+0000 7f9db0a1f640 1 -- 172.21.15.189:0/1825936983 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d90005050 con 0x7f9dac0a25f0 2024-09-16T14:43:58.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.374+0000 7f9da2ffd640 1 -- 172.21.15.189:0/1825936983 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(27..27 src has 1..27) v4 ==== 3506+0+0 (unknown 40144544 0 0) 0x7f9d940431e0 con 0x7f9dac0a25f0 2024-09-16T14:43:58.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.374+0000 7f9daa59c640 10 monclient: get_auth_request con 0x7f9d8c036130 auth_method 0 2024-09-16T14:43:58.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.378+0000 7f9da2ffd640 1 -- 172.21.15.189:0/1825936983 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9d94014820 con 0x7f9dac0a25f0 2024-09-16T14:43:58.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.378+0000 7f9da2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:58.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.378+0000 7f9da2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:58.538 INFO:tasks.workunit.client.0.smithi189.stderr:foo-rule not valid: foo-rule not one of 'true', 'false' 2024-09-16T14:43:58.538 INFO:tasks.workunit.client.0.smithi189.stderr:Invalid command: unused arguments: ['foo-rule'] 2024-09-16T14:43:58.538 INFO:tasks.workunit.client.0.smithi189.stderr:osd pool create [] [] [] [] [] [] [] [] [] [] [--bulk] [] [] [--yes-i-really-mean-it] : create pool 2024-09-16T14:43:58.538 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: invalid command 2024-09-16T14:43:58.542 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:43:58.542 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create foo 123 123 replicated 2024-09-16T14:43:58.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd71906640 10 monclient: get_monmap_and_config 2024-09-16T14:43:58.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd71906640 10 monclient: build_initial_monmap 2024-09-16T14:43:58.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd71906640 10 monclient: monmap: 2024-09-16T14:43:58.663 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:58.663 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:58.663 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:58.663080+0000 2024-09-16T14:43:58.663 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:58.663080+0000 2024-09-16T14:43:58.663 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:58.663 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:58.663 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:58.663 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:58.663 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:58.663 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:58.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd71906640 1 Processor -- start 2024-09-16T14:43:58.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd71906640 1 -- start start 2024-09-16T14:43:58.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd71906640 10 monclient: init 2024-09-16T14:43:58.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd71906640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:58.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd71906640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:43:58.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd71906640 10 monclient(hunting): picked mon.noname-a con 0x7fcd6c0a1a90 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:58.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd71906640 10 monclient(hunting): picked mon.noname-c con 0x7fcd6c0a2580 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:58.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd71906640 10 monclient(hunting): picked mon.noname-b con 0x7fcd6c09fd40 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:58.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd71906640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd6c09df00 con 0x7fcd6c0a1a90 2024-09-16T14:43:58.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd71906640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd6c09f0a0 con 0x7fcd6c0a2580 2024-09-16T14:43:58.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd71906640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd6c0a3040 con 0x7fcd6c09fd40 2024-09-16T14:43:58.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd71906640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:58.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd71906640 10 monclient(hunting): authenticate will time out at 1020.779631s 2024-09-16T14:43:58.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd6a7fc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcd6c0a2580 0x7fcd6c139df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51724/0 (socket says 172.21.15.189:51724) 2024-09-16T14:43:58.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd6a7fc640 1 -- 172.21.15.189:0/3145359315 learned_addr learned my addr 172.21.15.189:0/3145359315 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:58.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd6affd640 1 -- 172.21.15.189:0/3145359315 >> v1:172.21.15.26:6789/0 conn(0x7fcd6c0a1a90 legacy=0x7fcd6c0a1e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:58.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.658+0000 7fcd69ffb640 1 -- 172.21.15.189:0/3145359315 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808588708 0 0) 0x7fcd6c09f0a0 con 0x7fcd6c0a2580 2024-09-16T14:43:58.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:58.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:58.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:58.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 10 monclient(hunting): my global_id is 4636 2024-09-16T14:43:58.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 1 -- 172.21.15.189:0/3145359315 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd54003530 con 0x7fcd6c0a2580 2024-09-16T14:43:58.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 1 -- 172.21.15.189:0/3145359315 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 147209335 0 0) 0x7fcd54003530 con 0x7fcd6c0a2580 2024-09-16T14:43:58.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 1 -- 172.21.15.189:0/3145359315 >> v1:172.21.15.26:6790/0 conn(0x7fcd6c09fd40 legacy=0x7fcd6c13d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:58.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 1 -- 172.21.15.189:0/3145359315 >> v1:172.21.15.26:6789/0 conn(0x7fcd6c0a1a90 legacy=0x7fcd6c0a1e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:58.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:43:58.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 1 monclient: found mon.noname-c 2024-09-16T14:43:58.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:58.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:58.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 1 -- 172.21.15.189:0/3145359315 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c09eae0 con 0x7fcd6c0a2580 2024-09-16T14:43:58.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 10 monclient: _finish_auth 0 2024-09-16T14:43:58.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:43:58.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:58.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 1 -- 172.21.15.189:0/3145359315 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcd60001550 con 0x7fcd6c0a2580 2024-09-16T14:43:58.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 5 monclient: authenticate success, global_id 4636 2024-09-16T14:43:58.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:58.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:58.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:58.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 10 monclient: dump: 2024-09-16T14:43:58.672 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:58.672 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:58.672 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:58.672 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:58.672 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:58.673 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:58.673 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:58.673 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:58.673 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:58.673 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:58.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:58.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 1 -- 172.21.15.189:0/3145359315 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd60001800 con 0x7fcd6c0a2580 2024-09-16T14:43:58.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:58.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:58.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 1 -- 172.21.15.189:0/3145359315 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcd60001bd0 con 0x7fcd6c0a2580 2024-09-16T14:43:58.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 4 set_mon_vals no callback set 2024-09-16T14:43:58.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:58.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:58.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd69ffb640 10 monclient: dump: 2024-09-16T14:43:58.674 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:58.674 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:58.674 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:58.674 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:58.674 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:58.674 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:58.674 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:58.674 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:58.674 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:58.674 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:58.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient: shutdown 2024-09-16T14:43:58.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:58.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 1 -- 172.21.15.189:0/3145359315 >> v1:172.21.15.189:6789/0 conn(0x7fcd6c0a2580 legacy=0x7fcd6c139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:58.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 1 -- 172.21.15.189:0/3145359315 shutdown_connections 2024-09-16T14:43:58.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 1 -- 172.21.15.189:0/3145359315 >> 172.21.15.189:0/3145359315 conn(0x7fcd6c0a7a70 msgr2=0x7fcd6c0a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:58.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 1 -- 172.21.15.189:0/3145359315 shutdown_connections 2024-09-16T14:43:58.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 1 -- 172.21.15.189:0/3145359315 wait complete. 2024-09-16T14:43:58.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient: build_initial_monmap 2024-09-16T14:43:58.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient: monmap: 2024-09-16T14:43:58.675 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:58.675 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:58.675 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:58.675 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:58.675 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:58.675 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:58.675 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:58.675 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:58.676 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:58.676 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:58.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 1 Processor -- start 2024-09-16T14:43:58.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 1 -- start start 2024-09-16T14:43:58.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient: init 2024-09-16T14:43:58.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:58.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:43:58.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient(hunting): picked mon.noname-b con 0x7fcd6c09fd40 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:58.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient(hunting): picked mon.noname-a con 0x7fcd6c0a1a90 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:58.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient(hunting): picked mon.noname-c con 0x7fcd6c0a2580 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:58.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd6c120b00 con 0x7fcd6c0a1a90 2024-09-16T14:43:58.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd6c1c9e30 con 0x7fcd6c09fd40 2024-09-16T14:43:58.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd6affd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcd6c09fd40 0x7fcd6c133b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51728/0 (socket says 172.21.15.189:51728) 2024-09-16T14:43:58.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd6c09e280 con 0x7fcd6c0a2580 2024-09-16T14:43:58.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd6affd640 1 -- 172.21.15.189:0/2926822735 learned_addr learned my addr 172.21.15.189:0/2926822735 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:58.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:58.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient(hunting): authenticate will time out at 1020.782496s 2024-09-16T14:43:58.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd6a7fc640 1 -- 172.21.15.189:0/2926822735 >> v1:172.21.15.26:6789/0 conn(0x7fcd6c0a1a90 legacy=0x7fcd6c1ce200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:58.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 1 -- 172.21.15.189:0/2926822735 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981807216 0 0) 0x7fcd5c003e10 con 0x7fcd6c09fd40 2024-09-16T14:43:58.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 10 monclient(hunting): _negotiate 2024-09-16T14:43:58.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:58.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:58.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 10 monclient(hunting): my global_id is 4639 2024-09-16T14:43:58.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 1 -- 172.21.15.189:0/2926822735 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd58003530 con 0x7fcd6c09fd40 2024-09-16T14:43:58.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 1 -- 172.21.15.189:0/2926822735 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2934664934 0 0) 0x7fcd58003530 con 0x7fcd6c09fd40 2024-09-16T14:43:58.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 1 -- 172.21.15.189:0/2926822735 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd5c003e10 con 0x7fcd6c09fd40 2024-09-16T14:43:58.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 1 -- 172.21.15.189:0/2926822735 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcd5c004500 con 0x7fcd6c09fd40 2024-09-16T14:43:58.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:58.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:58.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 10 monclient(hunting): dump: 2024-09-16T14:43:58.678 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:58.678 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:58.678 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:58.678 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:58.678 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:58.678 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:58.678 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:58.678 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:58.678 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:58.679 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:58.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:58.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 1 -- 172.21.15.189:0/2926822735 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3477158530 0 0) 0x7fcd5c003e10 con 0x7fcd6c09fd40 2024-09-16T14:43:58.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 1 -- 172.21.15.189:0/2926822735 >> v1:172.21.15.26:6790/0 conn(0x7fcd6c0a2580 legacy=0x7fcd6c1d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:58.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 1 -- 172.21.15.189:0/2926822735 >> v1:172.21.15.26:6789/0 conn(0x7fcd6c0a1a90 legacy=0x7fcd6c1ce200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:58.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 10 monclient: _finish_hunting 0 2024-09-16T14:43:58.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 1 monclient: found mon.b 2024-09-16T14:43:58.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:58.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:58.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 1 -- 172.21.15.189:0/2926822735 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c0a6390 con 0x7fcd6c09fd40 2024-09-16T14:43:58.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 10 monclient: _finish_auth 0 2024-09-16T14:43:58.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:58.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:58.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 5 monclient: authenticate success, global_id 4639 2024-09-16T14:43:58.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient: _renew_subs 2024-09-16T14:43:58.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:58.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 1 -- 172.21.15.189:0/2926822735 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd6c1224d0 con 0x7fcd6c09fd40 2024-09-16T14:43:58.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient: _renew_subs 2024-09-16T14:43:58.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:58.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 1 -- 172.21.15.189:0/2926822735 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd6c120100 con 0x7fcd6c09fd40 2024-09-16T14:43:58.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 1 -- 172.21.15.189:0/2926822735 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd5c004820 con 0x7fcd6c09fd40 2024-09-16T14:43:58.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:58.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 1 -- 172.21.15.189:0/2926822735 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcd5c004bb0 con 0x7fcd6c09fd40 2024-09-16T14:43:58.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:58.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:58.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd697fa640 4 set_mon_vals no callback set 2024-09-16T14:43:58.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 10 monclient: dump: 2024-09-16T14:43:58.681 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:58.681 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:58.681 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:58.681 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:58.681 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:58.681 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:58.681 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:58.681 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:58.681 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:58.681 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:58.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:58.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:58.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:58.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd71906640 1 -- 172.21.15.189:0/2926822735 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd4c005050 con 0x7fcd6c09fd40 2024-09-16T14:43:58.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.662+0000 7fcd53fff640 1 -- 172.21.15.189:0/2926822735 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fcd5c004d10 con 0x7fcd6c09fd40 2024-09-16T14:43:58.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.666+0000 7fcd53fff640 1 -- 172.21.15.189:0/2926822735 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(27..27 src has 1..27) v4 ==== 3506+0+0 (unknown 40144544 0 0) 0x7fcd5c0434c0 con 0x7fcd6c09fd40 2024-09-16T14:43:58.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.666+0000 7fcd6a7fc640 10 monclient: get_auth_request con 0x7fcd58036050 auth_method 0 2024-09-16T14:43:58.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.670+0000 7fcd53fff640 1 -- 172.21.15.189:0/2926822735 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fcd5c0054e0 con 0x7fcd6c09fd40 2024-09-16T14:43:58.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.670+0000 7fcd53fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:58.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.670+0000 7fcd53fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:58.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.830+0000 7fcd71906640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T14:43:58.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.830+0000 7fcd71906640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T14:43:58.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.830+0000 7fcd71906640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:58.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:58.830+0000 7fcd71906640 1 -- 172.21.15.189:0/2926822735 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"} v 0) v1 -- 0x7fcd4c005880 con 0x7fcd6c09fd40 2024-09-16T14:43:59.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.146+0000 7fcd53fff640 1 -- 172.21.15.189:0/2926822735 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}]=0 pool 'foo' created v28) v1 ==== 154+0+0 (unknown 102578782 0 0) 0x7fcd5c01d0d0 con 0x7fcd6c09fd40 2024-09-16T14:43:59.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.146+0000 7fcd53fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T14:43:59.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.146+0000 7fcd53fff640 10 monclient: _finish_command 2 = system:0 pool 'foo' created 2024-09-16T14:43:59.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.254+0000 7fcd71906640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T14:43:59.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.254+0000 7fcd71906640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T14:43:59.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.254+0000 7fcd71906640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:59.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.254+0000 7fcd71906640 1 -- 172.21.15.189:0/2926822735 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"} v 0) v1 -- 0x7fcd4c005370 con 0x7fcd6c09fd40 2024-09-16T14:43:59.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.254+0000 7fcd53fff640 1 -- 172.21.15.189:0/2926822735 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}]=0 pool 'foo' already exists v28) v1 ==== 161+0+0 (unknown 3632846243 0 0) 0x7fcd5c012f50 con 0x7fcd6c09fd40 2024-09-16T14:43:59.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.254+0000 7fcd53fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T14:43:59.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.254+0000 7fcd53fff640 10 monclient: _finish_command 3 = system:0 pool 'foo' already exists 2024-09-16T14:43:59.261 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foo' already exists 2024-09-16T14:43:59.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.258+0000 7fcd71906640 1 -- 172.21.15.189:0/2926822735 >> v1:172.21.15.189:6812/3196161582 conn(0x7fcd58036050 legacy=0x7fcd58038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:59.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.258+0000 7fcd71906640 10 monclient: shutdown 2024-09-16T14:43:59.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.258+0000 7fcd71906640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:59.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.258+0000 7fcd71906640 1 -- 172.21.15.189:0/2926822735 >> v1:172.21.15.189:6789/0 conn(0x7fcd6c09fd40 legacy=0x7fcd6c133b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:59.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.258+0000 7fcd71906640 1 -- 172.21.15.189:0/2926822735 shutdown_connections 2024-09-16T14:43:59.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.258+0000 7fcd71906640 1 -- 172.21.15.189:0/2926822735 >> 172.21.15.189:0/2926822735 conn(0x7fcd6c0a7a70 msgr2=0x7fcd6c1bb300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:59.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.258+0000 7fcd71906640 1 -- 172.21.15.189:0/2926822735 shutdown_connections 2024-09-16T14:43:59.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.258+0000 7fcd71906640 1 -- 172.21.15.189:0/2926822735 wait complete. 2024-09-16T14:43:59.277 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create fooo 123 123 erasure default 2024-09-16T14:43:59.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 10 monclient: get_monmap_and_config 2024-09-16T14:43:59.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 10 monclient: build_initial_monmap 2024-09-16T14:43:59.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 10 monclient: monmap: 2024-09-16T14:43:59.401 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:59.401 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:59.402 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:43:59.401918+0000 2024-09-16T14:43:59.402 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:43:59.401918+0000 2024-09-16T14:43:59.402 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:59.402 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:59.402 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:59.402 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:43:59.402 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:43:59.402 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:59.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 1 Processor -- start 2024-09-16T14:43:59.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 1 -- start start 2024-09-16T14:43:59.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 10 monclient: init 2024-09-16T14:43:59.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:59.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:43:59.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 10 monclient(hunting): picked mon.noname-a con 0x7f554c099a20 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:59.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 10 monclient(hunting): picked mon.noname-b con 0x7f554c099e00 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:59.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 10 monclient(hunting): picked mon.noname-c con 0x7f554c13b1a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:59.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554c09a220 con 0x7f554c099a20 2024-09-16T14:43:59.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554c09f2a0 con 0x7f554c13b1a0 2024-09-16T14:43:59.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554c071c00 con 0x7f554c099e00 2024-09-16T14:43:59.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:59.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 10 monclient(hunting): authenticate will time out at 1021.517659s 2024-09-16T14:43:59.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5551afd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f554c13b1a0 0x7f554c13d590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:59374/0 (socket says 172.21.15.189:59374) 2024-09-16T14:43:59.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5551afd640 1 -- 172.21.15.189:0/3452532343 learned_addr learned my addr 172.21.15.189:0/3452532343 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:59.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f55512fc640 1 -- 172.21.15.189:0/3452532343 >> v1:172.21.15.26:6789/0 conn(0x7f554c099a20 legacy=0x7f554c134d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:59.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 1 -- 172.21.15.189:0/3452532343 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298619085 0 0) 0x7f554c09f2a0 con 0x7f554c13b1a0 2024-09-16T14:43:59.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 10 monclient(hunting): _negotiate 2024-09-16T14:43:59.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:59.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:59.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 10 monclient(hunting): my global_id is 4642 2024-09-16T14:43:59.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 1 -- 172.21.15.189:0/3452532343 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5538003530 con 0x7f554c13b1a0 2024-09-16T14:43:59.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 1 -- 172.21.15.189:0/3452532343 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1927422035 0 0) 0x7f5538003530 con 0x7f554c13b1a0 2024-09-16T14:43:59.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 1 -- 172.21.15.189:0/3452532343 >> v1:172.21.15.26:6790/0 conn(0x7f554c099e00 legacy=0x7f554c139a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:59.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 1 -- 172.21.15.189:0/3452532343 >> v1:172.21.15.26:6789/0 conn(0x7f554c099a20 legacy=0x7f554c134d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:43:59.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 10 monclient: _finish_hunting 0 2024-09-16T14:43:59.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 1 monclient: found mon.noname-c 2024-09-16T14:43:59.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:59.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:43:59.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 1 -- 172.21.15.189:0/3452532343 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c135410 con 0x7f554c13b1a0 2024-09-16T14:43:59.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 10 monclient: _finish_auth 0 2024-09-16T14:43:59.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 10 monclient: _check_auth_tickets 2024-09-16T14:43:59.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:59.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 1 -- 172.21.15.189:0/3452532343 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f55440019a0 con 0x7f554c13b1a0 2024-09-16T14:43:59.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:59.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:43:59.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 10 monclient: dump: 2024-09-16T14:43:59.411 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:59.411 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:59.411 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:59.411 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:59.411 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:59.412 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:59.412 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:59.412 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:59.412 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:59.412 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:59.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 5 monclient: authenticate success, global_id 4642 2024-09-16T14:43:59.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:43:59.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 1 -- 172.21.15.189:0/3452532343 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5544001c50 con 0x7f554c13b1a0 2024-09-16T14:43:59.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:59.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 1 -- 172.21.15.189:0/3452532343 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5544001fe0 con 0x7f554c13b1a0 2024-09-16T14:43:59.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:59.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:59.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5543fff640 10 monclient: dump: 2024-09-16T14:43:59.413 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:59.413 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:59.413 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:59.413 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:59.413 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:59.413 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:59.413 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:59.413 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:59.413 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:59.413 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:59.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 10 monclient: get_monmap_and_config success 2024-09-16T14:43:59.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 4 set_mon_vals no callback set 2024-09-16T14:43:59.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 10 monclient: shutdown 2024-09-16T14:43:59.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:43:59.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 1 -- 172.21.15.189:0/3452532343 >> v1:172.21.15.189:6789/0 conn(0x7f554c13b1a0 legacy=0x7f554c13d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:59.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 1 -- 172.21.15.189:0/3452532343 shutdown_connections 2024-09-16T14:43:59.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 1 -- 172.21.15.189:0/3452532343 >> 172.21.15.189:0/3452532343 conn(0x7f554c0a0260 msgr2=0x7f554c0a0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:43:59.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 1 -- 172.21.15.189:0/3452532343 shutdown_connections 2024-09-16T14:43:59.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 1 -- 172.21.15.189:0/3452532343 wait complete. 2024-09-16T14:43:59.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 10 monclient: build_initial_monmap 2024-09-16T14:43:59.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 10 monclient: monmap: 2024-09-16T14:43:59.414 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:43:59.414 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:43:59.414 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:43:59.414 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:43:59.414 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:43:59.414 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:43:59.415 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:43:59.415 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:43:59.415 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:43:59.415 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:59.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.398+0000 7f5553560640 1 Processor -- start 2024-09-16T14:43:59.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 1 -- start start 2024-09-16T14:43:59.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 10 monclient: init 2024-09-16T14:43:59.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 10 monclient: _reopen_session rank -1 2024-09-16T14:43:59.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:43:59.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 10 monclient(hunting): picked mon.noname-a con 0x7f554c099a20 addr v1:172.21.15.26:6789/0 2024-09-16T14:43:59.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 10 monclient(hunting): picked mon.noname-c con 0x7f554c099e00 addr v1:172.21.15.26:6790/0 2024-09-16T14:43:59.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 10 monclient(hunting): picked mon.noname-b con 0x7f554c13b1a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:43:59.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554c0a1480 con 0x7f554c099a20 2024-09-16T14:43:59.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554c13ad10 con 0x7f554c13b1a0 2024-09-16T14:43:59.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554c12bef0 con 0x7f554c099e00 2024-09-16T14:43:59.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 10 monclient(hunting): _renew_subs 2024-09-16T14:43:59.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 10 monclient(hunting): authenticate will time out at 1021.520555s 2024-09-16T14:43:59.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5551afd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f554c13b1a0 0x7f554c1d1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:59382/0 (socket says 172.21.15.189:59382) 2024-09-16T14:43:59.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5551afd640 1 -- 172.21.15.189:0/1842481162 learned_addr learned my addr 172.21.15.189:0/1842481162 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:43:59.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f55512fc640 1 -- 172.21.15.189:0/1842481162 >> v1:172.21.15.26:6789/0 conn(0x7f554c099a20 legacy=0x7f554c096660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:43:59.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 1 -- 172.21.15.189:0/1842481162 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029318277 0 0) 0x7f554c13ad10 con 0x7f554c13b1a0 2024-09-16T14:43:59.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:43:59.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:43:59.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:43:59.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 10 monclient(hunting): my global_id is 4645 2024-09-16T14:43:59.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 1 -- 172.21.15.189:0/1842481162 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5530003530 con 0x7f554c13b1a0 2024-09-16T14:43:59.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 1 -- 172.21.15.189:0/1842481162 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1905829815 0 0) 0x7f5530003530 con 0x7f554c13b1a0 2024-09-16T14:43:59.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 1 -- 172.21.15.189:0/1842481162 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f554c13ad10 con 0x7f554c13b1a0 2024-09-16T14:43:59.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 1 -- 172.21.15.189:0/1842481162 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5544001960 con 0x7f554c13b1a0 2024-09-16T14:43:59.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:59.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:43:59.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 10 monclient(hunting): dump: 2024-09-16T14:43:59.417 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:59.417 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:59.417 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:59.417 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:59.417 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:59.417 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:59.417 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:59.418 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:59.418 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:59.418 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:59.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:43:59.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 1 -- 172.21.15.189:0/1842481162 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 738836705 0 0) 0x7f554c13ad10 con 0x7f554c13b1a0 2024-09-16T14:43:59.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 1 -- 172.21.15.189:0/1842481162 >> v1:172.21.15.26:6790/0 conn(0x7f554c099e00 legacy=0x7f554c1ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:43:59.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 1 -- 172.21.15.189:0/1842481162 >> v1:172.21.15.26:6789/0 conn(0x7f554c099a20 legacy=0x7f554c096660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:43:59.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:43:59.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 1 monclient: found mon.b 2024-09-16T14:43:59.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:43:59.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:59.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 1 -- 172.21.15.189:0/1842481162 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c09a550 con 0x7f554c13b1a0 2024-09-16T14:43:59.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 10 monclient: _finish_auth 0 2024-09-16T14:43:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:43:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:43:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 5 monclient: authenticate success, global_id 4645 2024-09-16T14:43:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 10 monclient: _renew_subs 2024-09-16T14:43:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 1 -- 172.21.15.189:0/1842481162 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554c12c1d0 con 0x7f554c13b1a0 2024-09-16T14:43:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 10 monclient: _renew_subs 2024-09-16T14:43:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 1 -- 172.21.15.189:0/1842481162 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f554c12a320 con 0x7f554c13b1a0 2024-09-16T14:43:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 1 -- 172.21.15.189:0/1842481162 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5544004f80 con 0x7f554c13b1a0 2024-09-16T14:43:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:43:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 1 -- 172.21.15.189:0/1842481162 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5544005290 con 0x7f554c13b1a0 2024-09-16T14:43:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:43:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f55437fe640 4 set_mon_vals no callback set 2024-09-16T14:43:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:43:59.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 10 monclient: dump: 2024-09-16T14:43:59.420 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:43:59.420 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:43:59.420 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:59.420 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:43:59.420 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:43:59.420 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:43:59.420 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:43:59.420 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:43:59.420 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:43:59.420 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:43:59.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:59.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:59.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:59.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5553560640 1 -- 172.21.15.189:0/1842481162 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f552c005050 con 0x7f554c13b1a0 2024-09-16T14:43:59.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 1 -- 172.21.15.189:0/1842481162 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f554400fd00 con 0x7f554c13b1a0 2024-09-16T14:43:59.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f5541ffb640 1 -- 172.21.15.189:0/1842481162 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(28..28 src has 1..28) v4 ==== 3848+0+0 (unknown 4269837265 0 0) 0x7f55440437c0 con 0x7f554c13b1a0 2024-09-16T14:43:59.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.402+0000 7f55512fc640 10 monclient: get_auth_request con 0x7f5530036060 auth_method 0 2024-09-16T14:43:59.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.406+0000 7f5541ffb640 1 -- 172.21.15.189:0/1842481162 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5544014ae0 con 0x7f554c13b1a0 2024-09-16T14:43:59.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.406+0000 7f5541ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:43:59.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.406+0000 7f5541ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:43:59.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.554+0000 7f5553560640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T14:43:59.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.554+0000 7f5553560640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T14:43:59.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.554+0000 7f5553560640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:43:59.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:43:59.554+0000 7f5553560640 1 -- 172.21.15.189:0/1842481162 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"} v 0) v1 -- 0x7f552c005300 con 0x7f554c13b1a0 2024-09-16T14:44:01.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.150+0000 7f5541ffb640 1 -- 172.21.15.189:0/1842481162 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}]=0 pool 'fooo' created v30) v1 ==== 188+0+0 (unknown 4130541977 0 0) 0x7f554401d2a0 con 0x7f554c13b1a0 2024-09-16T14:44:01.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.150+0000 7f5541ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T14:44:01.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.150+0000 7f5541ffb640 10 monclient: _finish_command 2 = system:0 pool 'fooo' created 2024-09-16T14:44:01.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.270+0000 7f5553560640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T14:44:01.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.270+0000 7f5553560640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T14:44:01.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.270+0000 7f5553560640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:01.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.270+0000 7f5553560640 1 -- 172.21.15.189:0/1842481162 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"} v 0) v1 -- 0x7f552c005b60 con 0x7f554c13b1a0 2024-09-16T14:44:01.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.274+0000 7f5541ffb640 1 -- 172.21.15.189:0/1842481162 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}]=0 pool 'fooo' already exists v30) v1 ==== 195+0+0 (unknown 2547868665 0 0) 0x7f5544013180 con 0x7f554c13b1a0 2024-09-16T14:44:01.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.274+0000 7f5541ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T14:44:01.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.274+0000 7f5541ffb640 10 monclient: _finish_command 3 = system:0 pool 'fooo' already exists 2024-09-16T14:44:01.278 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fooo' already exists 2024-09-16T14:44:01.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.274+0000 7f5553560640 1 -- 172.21.15.189:0/1842481162 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5530036060 legacy=0x7f5530038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:01.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.274+0000 7f5553560640 10 monclient: shutdown 2024-09-16T14:44:01.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.274+0000 7f5553560640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:01.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.274+0000 7f5553560640 1 -- 172.21.15.189:0/1842481162 >> v1:172.21.15.189:6789/0 conn(0x7f554c13b1a0 legacy=0x7f554c1d1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:01.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.274+0000 7f5553560640 1 -- 172.21.15.189:0/1842481162 shutdown_connections 2024-09-16T14:44:01.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.274+0000 7f5553560640 1 -- 172.21.15.189:0/1842481162 >> 172.21.15.189:0/1842481162 conn(0x7f554c0a0260 msgr2=0x7f554c098510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:01.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.274+0000 7f5553560640 1 -- 172.21.15.189:0/1842481162 shutdown_connections 2024-09-16T14:44:01.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.274+0000 7f5553560640 1 -- 172.21.15.189:0/1842481162 wait complete. 2024-09-16T14:44:01.295 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create foooo 123 2024-09-16T14:44:01.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.410+0000 7f1d96eea640 10 monclient: get_monmap_and_config 2024-09-16T14:44:01.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.410+0000 7f1d96eea640 10 monclient: build_initial_monmap 2024-09-16T14:44:01.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.410+0000 7f1d96eea640 10 monclient: monmap: 2024-09-16T14:44:01.416 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:01.416 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:01.416 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:01.416278+0000 2024-09-16T14:44:01.416 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:01.416278+0000 2024-09-16T14:44:01.416 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:01.416 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:01.417 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:01.417 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:01.417 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:01.417 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:01.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.410+0000 7f1d96eea640 1 Processor -- start 2024-09-16T14:44:01.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.410+0000 7f1d96eea640 1 -- start start 2024-09-16T14:44:01.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient: init 2024-09-16T14:44:01.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:01.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:01.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient(hunting): picked mon.noname-c con 0x7f1d900a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:01.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient(hunting): picked mon.noname-a con 0x7f1d900a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:01.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient(hunting): picked mon.noname-b con 0x7f1d9013b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:01.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d900a30b0 con 0x7f1d900a25f0 2024-09-16T14:44:01.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d900a2e30 con 0x7f1d900a1b00 2024-09-16T14:44:01.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9009eb50 con 0x7f1d9013b430 2024-09-16T14:44:01.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:01.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient(hunting): authenticate will time out at 1023.532670s 2024-09-16T14:44:01.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d94c86640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1d900a1b00 0x7f1d900a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:59388/0 (socket says 172.21.15.189:59388) 2024-09-16T14:44:01.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d94c86640 1 -- 172.21.15.189:0/2228482787 learned_addr learned my addr 172.21.15.189:0/2228482787 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:01.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8ffff640 1 -- 172.21.15.189:0/2228482787 >> v1:172.21.15.26:6789/0 conn(0x7f1d900a25f0 legacy=0x7f1d90139d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:01.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 1 -- 172.21.15.189:0/2228482787 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016209224 0 0) 0x7f1d900a2e30 con 0x7f1d900a1b00 2024-09-16T14:44:01.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:01.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:01.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:01.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 10 monclient(hunting): my global_id is 4648 2024-09-16T14:44:01.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 1 -- 172.21.15.189:0/2228482787 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d80003530 con 0x7f1d900a1b00 2024-09-16T14:44:01.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 1 -- 172.21.15.189:0/2228482787 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 631701714 0 0) 0x7f1d80003530 con 0x7f1d900a1b00 2024-09-16T14:44:01.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 1 -- 172.21.15.189:0/2228482787 >> v1:172.21.15.26:6790/0 conn(0x7f1d9013b430 legacy=0x7f1d9013d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:01.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 1 -- 172.21.15.189:0/2228482787 >> v1:172.21.15.26:6789/0 conn(0x7f1d900a25f0 legacy=0x7f1d90139d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:01.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:44:01.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 1 monclient: found mon.noname-c 2024-09-16T14:44:01.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:01.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:44:01.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 1 -- 172.21.15.189:0/2228482787 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9009ee30 con 0x7f1d900a1b00 2024-09-16T14:44:01.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:44:01.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:44:01.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:01.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 5 monclient: authenticate success, global_id 4648 2024-09-16T14:44:01.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:01.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 1 -- 172.21.15.189:0/2228482787 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1d84001620 con 0x7f1d900a1b00 2024-09-16T14:44:01.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:01.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:01.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 10 monclient: dump: 2024-09-16T14:44:01.423 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:01.423 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:01.423 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:01.423 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:01.423 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:01.423 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:01.423 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:01.423 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:01.423 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:01.423 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:01.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:01.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 1 -- 172.21.15.189:0/2228482787 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d84001940 con 0x7f1d900a1b00 2024-09-16T14:44:01.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:01.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 1 -- 172.21.15.189:0/2228482787 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1d84001cd0 con 0x7f1d900a1b00 2024-09-16T14:44:01.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:01.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:01.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8f7fe640 10 monclient: dump: 2024-09-16T14:44:01.424 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:01.424 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:01.424 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:01.424 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:01.424 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:01.424 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:01.425 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:01.425 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:01.425 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:01.425 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:01.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:01.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 4 set_mon_vals no callback set 2024-09-16T14:44:01.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient: shutdown 2024-09-16T14:44:01.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:01.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 1 -- 172.21.15.189:0/2228482787 >> v1:172.21.15.189:6789/0 conn(0x7f1d900a1b00 legacy=0x7f1d900a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:01.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 1 -- 172.21.15.189:0/2228482787 shutdown_connections 2024-09-16T14:44:01.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 1 -- 172.21.15.189:0/2228482787 >> 172.21.15.189:0/2228482787 conn(0x7f1d900a7dd0 msgr2=0x7f1d900a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:01.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 1 -- 172.21.15.189:0/2228482787 shutdown_connections 2024-09-16T14:44:01.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 1 -- 172.21.15.189:0/2228482787 wait complete. 2024-09-16T14:44:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient: build_initial_monmap 2024-09-16T14:44:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient: monmap: 2024-09-16T14:44:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:01.426 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 1 Processor -- start 2024-09-16T14:44:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 1 -- start start 2024-09-16T14:44:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient: init 2024-09-16T14:44:01.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:01.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:01.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient(hunting): picked mon.noname-c con 0x7f1d900a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:01.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient(hunting): picked mon.noname-a con 0x7f1d900a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:01.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient(hunting): picked mon.noname-b con 0x7f1d9013b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:01.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9011dc70 con 0x7f1d900a25f0 2024-09-16T14:44:01.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9011ff90 con 0x7f1d9013b430 2024-09-16T14:44:01.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9009e2f0 con 0x7f1d900a1b00 2024-09-16T14:44:01.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:01.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d96eea640 10 monclient(hunting): authenticate will time out at 1023.535713s 2024-09-16T14:44:01.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d8ffff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f1d900a25f0 legacy=0x7f1d901ce860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:01.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d95487640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1d9013b430 0x7f1d901d1f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:59394/0 (socket says 172.21.15.189:59394) 2024-09-16T14:44:01.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.414+0000 7f1d95487640 1 -- 172.21.15.189:0/22343790 learned_addr learned my addr 172.21.15.189:0/22343790 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:01.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265672732 0 0) 0x7f1d9011ff90 con 0x7f1d9013b430 2024-09-16T14:44:01.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:44:01.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:01.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:01.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient(hunting): my global_id is 4651 2024-09-16T14:44:01.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d74003560 con 0x7f1d9013b430 2024-09-16T14:44:01.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2171526404 0 0) 0x7f1d74003560 con 0x7f1d9013b430 2024-09-16T14:44:01.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d9011ff90 con 0x7f1d9013b430 2024-09-16T14:44:01.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1d88001c80 con 0x7f1d9013b430 2024-09-16T14:44:01.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:01.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:01.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient(hunting): dump: 2024-09-16T14:44:01.432 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:01.432 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:01.432 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:01.432 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:01.432 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:01.432 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:01.432 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:01.432 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:01.433 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:01.433 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:01.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:01.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848377344 0 0) 0x7f1d9009e2f0 con 0x7f1d900a1b00 2024-09-16T14:44:01.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:44:01.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:01.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:01.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient(hunting): my global_id is 4625 2024-09-16T14:44:01.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d74003560 con 0x7f1d900a1b00 2024-09-16T14:44:01.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 270534932 0 0) 0x7f1d9011ff90 con 0x7f1d9013b430 2024-09-16T14:44:01.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 >> v1:172.21.15.26:6790/0 conn(0x7f1d900a1b00 legacy=0x7f1d901c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:01.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 >> v1:172.21.15.26:6789/0 conn(0x7f1d900a25f0 legacy=0x7f1d901ce860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:01.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:44:01.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 1 monclient: found mon.b 2024-09-16T14:44:01.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:01.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:01.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d900a5a30 con 0x7f1d9013b430 2024-09-16T14:44:01.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:44:01.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:01.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:01.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d96eea640 5 monclient: authenticate success, global_id 4651 2024-09-16T14:44:01.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d96eea640 10 monclient: _renew_subs 2024-09-16T14:44:01.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d96eea640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:01.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d96eea640 1 -- 172.21.15.189:0/22343790 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d9011f9d0 con 0x7f1d9013b430 2024-09-16T14:44:01.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d96eea640 10 monclient: _renew_subs 2024-09-16T14:44:01.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d96eea640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:01.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d96eea640 1 -- 172.21.15.189:0/22343790 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d9011e580 con 0x7f1d9013b430 2024-09-16T14:44:01.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d88002050 con 0x7f1d9013b430 2024-09-16T14:44:01.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:01.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1d88003e70 con 0x7f1d9013b430 2024-09-16T14:44:01.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:01.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:01.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8effd640 4 set_mon_vals no callback set 2024-09-16T14:44:01.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 10 monclient: dump: 2024-09-16T14:44:01.435 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:01.435 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:01.435 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:01.435 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:01.435 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:01.435 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:01.435 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:01.436 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:01.436 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:01.436 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:01.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d96eea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:01.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d96eea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:01.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d96eea640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:01.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d96eea640 1 -- 172.21.15.189:0/22343790 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d70005050 con 0x7f1d9013b430 2024-09-16T14:44:01.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1d88003fd0 con 0x7f1d9013b430 2024-09-16T14:44:01.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(30..30 src has 1..30) v4 ==== 4290+0+0 (unknown 2662678669 0 0) 0x7f1d88043850 con 0x7f1d9013b430 2024-09-16T14:44:01.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.418+0000 7f1d94c86640 10 monclient: get_auth_request con 0x7f1d74036200 auth_method 0 2024-09-16T14:44:01.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.422+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1d88014b60 con 0x7f1d9013b430 2024-09-16T14:44:01.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.422+0000 7f1d8d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:01.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.422+0000 7f1d8d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:01.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.586+0000 7f1d96eea640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T14:44:01.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.586+0000 7f1d96eea640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T14:44:01.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.586+0000 7f1d96eea640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:01.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:01.586+0000 7f1d96eea640 1 -- 172.21.15.189:0/22343790 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foooo", "pg_num": 123} v 0) v1 -- 0x7f1d70005880 con 0x7f1d9013b430 2024-09-16T14:44:02.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.158+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}]=0 pool 'foooo' created v31) v1 ==== 115+0+0 (unknown 2268850860 0 0) 0x7f1d88012fd0 con 0x7f1d9013b430 2024-09-16T14:44:02.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.158+0000 7f1d8d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T14:44:02.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.158+0000 7f1d8d7fa640 10 monclient: _finish_command 2 = system:0 pool 'foooo' created 2024-09-16T14:44:02.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.314+0000 7f1d96eea640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T14:44:02.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.314+0000 7f1d96eea640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T14:44:02.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.314+0000 7f1d96eea640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:02.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.314+0000 7f1d96eea640 1 -- 172.21.15.189:0/22343790 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foooo", "pg_num": 123} v 0) v1 -- 0x7f1d70005300 con 0x7f1d9013b430 2024-09-16T14:44:02.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.314+0000 7f1d8d7fa640 1 -- 172.21.15.189:0/22343790 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}]=0 pool 'foooo' already exists v31) v1 ==== 122+0+0 (unknown 4287065424 0 0) 0x7f1d88014930 con 0x7f1d9013b430 2024-09-16T14:44:02.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.314+0000 7f1d8d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T14:44:02.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.314+0000 7f1d8d7fa640 10 monclient: _finish_command 3 = system:0 pool 'foooo' already exists 2024-09-16T14:44:02.321 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foooo' already exists 2024-09-16T14:44:02.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.318+0000 7f1d96eea640 1 -- 172.21.15.189:0/22343790 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1d74036200 legacy=0x7f1d740386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:02.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.318+0000 7f1d96eea640 10 monclient: shutdown 2024-09-16T14:44:02.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.318+0000 7f1d96eea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:02.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.318+0000 7f1d96eea640 1 -- 172.21.15.189:0/22343790 >> v1:172.21.15.189:6789/0 conn(0x7f1d9013b430 legacy=0x7f1d901d1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:02.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.318+0000 7f1d96eea640 1 -- 172.21.15.189:0/22343790 shutdown_connections 2024-09-16T14:44:02.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.318+0000 7f1d96eea640 1 -- 172.21.15.189:0/22343790 >> 172.21.15.189:0/22343790 conn(0x7f1d900a7dd0 msgr2=0x7f1d9006b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:02.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.318+0000 7f1d96eea640 1 -- 172.21.15.189:0/22343790 shutdown_connections 2024-09-16T14:44:02.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.318+0000 7f1d96eea640 1 -- 172.21.15.189:0/22343790 wait complete. 2024-09-16T14:44:02.338 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create foo 123 2024-09-16T14:44:02.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a3764640 10 monclient: get_monmap_and_config 2024-09-16T14:44:02.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a3764640 10 monclient: build_initial_monmap 2024-09-16T14:44:02.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a3764640 10 monclient: monmap: 2024-09-16T14:44:02.452 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:02.452 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:02.452 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:02.451862+0000 2024-09-16T14:44:02.452 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:02.451862+0000 2024-09-16T14:44:02.452 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:02.452 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:02.452 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:02.452 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:02.452 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:02.452 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:02.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a3764640 1 Processor -- start 2024-09-16T14:44:02.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a3764640 1 -- start start 2024-09-16T14:44:02.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a3764640 10 monclient: init 2024-09-16T14:44:02.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a3764640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:02.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a3764640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:02.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a3764640 10 monclient(hunting): picked mon.noname-a con 0x7fc29c0a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:02.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a3764640 10 monclient(hunting): picked mon.noname-c con 0x7fc29c0a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:02.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a3764640 10 monclient(hunting): picked mon.noname-b con 0x7fc29c13b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:02.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a3764640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c0a30b0 con 0x7fc29c0a1b00 2024-09-16T14:44:02.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a3764640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c0a2e30 con 0x7fc29c0a25f0 2024-09-16T14:44:02.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a3764640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c09eb50 con 0x7fc29c13b430 2024-09-16T14:44:02.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a3764640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:02.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a3764640 10 monclient(hunting): authenticate will time out at 1024.567537s 2024-09-16T14:44:02.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a1500640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fc29c0a1b00 legacy=0x7fc29c0a1ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:02.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a0cff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc29c0a25f0 0x7fc29c139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:59400/0 (socket says 172.21.15.189:59400) 2024-09-16T14:44:02.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc2a0cff640 1 -- 172.21.15.189:0/3731848015 learned_addr learned my addr 172.21.15.189:0/3731848015 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:02.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.446+0000 7fc28ffff640 1 -- 172.21.15.189:0/3731848015 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259522998 0 0) 0x7fc29c0a2e30 con 0x7fc29c0a25f0 2024-09-16T14:44:02.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:02.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:02.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:02.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 10 monclient(hunting): my global_id is 4654 2024-09-16T14:44:02.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 1 -- 172.21.15.189:0/3731848015 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc288003530 con 0x7fc29c0a25f0 2024-09-16T14:44:02.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 1 -- 172.21.15.189:0/3731848015 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 814757395 0 0) 0x7fc288003530 con 0x7fc29c0a25f0 2024-09-16T14:44:02.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 1 -- 172.21.15.189:0/3731848015 >> v1:172.21.15.26:6790/0 conn(0x7fc29c13b430 legacy=0x7fc29c13d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:02.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 1 -- 172.21.15.189:0/3731848015 >> v1:172.21.15.26:6789/0 conn(0x7fc29c0a1b00 legacy=0x7fc29c0a1ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:02.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:44:02.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 1 monclient: found mon.noname-c 2024-09-16T14:44:02.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:02.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:44:02.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 1 -- 172.21.15.189:0/3731848015 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c09ee30 con 0x7fc29c0a25f0 2024-09-16T14:44:02.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 10 monclient: _finish_auth 0 2024-09-16T14:44:02.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:44:02.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:02.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 1 -- 172.21.15.189:0/3731848015 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc298001620 con 0x7fc29c0a25f0 2024-09-16T14:44:02.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 5 monclient: authenticate success, global_id 4654 2024-09-16T14:44:02.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:02.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:02.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:02.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 10 monclient: dump: 2024-09-16T14:44:02.458 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:02.458 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:02.458 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:02.458 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:02.458 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:02.459 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:02.459 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:02.459 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:02.459 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:02.459 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:02.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:02.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 1 -- 172.21.15.189:0/3731848015 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc298001940 con 0x7fc29c0a25f0 2024-09-16T14:44:02.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:02.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:02.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 4 set_mon_vals no callback set 2024-09-16T14:44:02.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 1 -- 172.21.15.189:0/3731848015 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc298001d10 con 0x7fc29c0a25f0 2024-09-16T14:44:02.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:02.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:02.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28ffff640 10 monclient: dump: 2024-09-16T14:44:02.460 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:02.460 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:02.460 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:02.460 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:02.460 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:02.460 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:02.460 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:02.460 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:02.460 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:02.460 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:02.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 10 monclient: shutdown 2024-09-16T14:44:02.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:02.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 1 -- 172.21.15.189:0/3731848015 >> v1:172.21.15.189:6789/0 conn(0x7fc29c0a25f0 legacy=0x7fc29c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:02.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 1 -- 172.21.15.189:0/3731848015 shutdown_connections 2024-09-16T14:44:02.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 1 -- 172.21.15.189:0/3731848015 >> 172.21.15.189:0/3731848015 conn(0x7fc29c0a7dd0 msgr2=0x7fc29c0a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:02.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 1 -- 172.21.15.189:0/3731848015 shutdown_connections 2024-09-16T14:44:02.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 1 -- 172.21.15.189:0/3731848015 wait complete. 2024-09-16T14:44:02.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 10 monclient: build_initial_monmap 2024-09-16T14:44:02.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 10 monclient: monmap: 2024-09-16T14:44:02.461 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:02.461 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:02.461 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:02.462 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:02.462 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:02.462 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:02.462 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:02.462 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:02.462 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:02.462 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:02.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 1 Processor -- start 2024-09-16T14:44:02.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 1 -- start start 2024-09-16T14:44:02.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 10 monclient: init 2024-09-16T14:44:02.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 10 monclient(hunting): picked mon.noname-a con 0x7fc29c0a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 10 monclient(hunting): picked mon.noname-b con 0x7fc29c0a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 10 monclient(hunting): picked mon.noname-c con 0x7fc29c13b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c0a5aa0 con 0x7fc29c0a1b00 2024-09-16T14:44:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c09e2f0 con 0x7fc29c0a25f0 2024-09-16T14:44:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c13ae40 con 0x7fc29c13b430 2024-09-16T14:44:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 10 monclient(hunting): authenticate will time out at 1024.570714s 2024-09-16T14:44:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a0cff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc29c0a25f0 0x7fc29c139ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:59414/0 (socket says 172.21.15.189:59414) 2024-09-16T14:44:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a0cff640 1 -- 172.21.15.189:0/3151932486 learned_addr learned my addr 172.21.15.189:0/3151932486 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a1500640 1 -- 172.21.15.189:0/3151932486 >> v1:172.21.15.26:6789/0 conn(0x7fc29c0a1b00 legacy=0x7fc29c139790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 1 -- 172.21.15.189:0/3151932486 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422932380 0 0) 0x7fc29c09e2f0 con 0x7fc29c0a25f0 2024-09-16T14:44:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 10 monclient(hunting): my global_id is 4657 2024-09-16T14:44:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 1 -- 172.21.15.189:0/3151932486 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc280003560 con 0x7fc29c0a25f0 2024-09-16T14:44:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 1 -- 172.21.15.189:0/3151932486 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2332800215 0 0) 0x7fc280003560 con 0x7fc29c0a25f0 2024-09-16T14:44:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 1 -- 172.21.15.189:0/3151932486 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc29c09e2f0 con 0x7fc29c0a25f0 2024-09-16T14:44:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 1 -- 172.21.15.189:0/3151932486 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc2980015e0 con 0x7fc29c0a25f0 2024-09-16T14:44:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 10 monclient(hunting): dump: 2024-09-16T14:44:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:02.467 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 1 -- 172.21.15.189:0/3151932486 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3930918529 0 0) 0x7fc29c09e2f0 con 0x7fc29c0a25f0 2024-09-16T14:44:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 1 -- 172.21.15.189:0/3151932486 >> v1:172.21.15.26:6790/0 conn(0x7fc29c13b430 legacy=0x7fc29c1d1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 1 -- 172.21.15.189:0/3151932486 >> v1:172.21.15.26:6789/0 conn(0x7fc29c0a1b00 legacy=0x7fc29c139790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:44:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 1 monclient: found mon.b 2024-09-16T14:44:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 1 -- 172.21.15.189:0/3151932486 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c11fdc0 con 0x7fc29c0a25f0 2024-09-16T14:44:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 10 monclient: _finish_auth 0 2024-09-16T14:44:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:44:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 5 monclient: authenticate success, global_id 4657 2024-09-16T14:44:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 10 monclient: _renew_subs 2024-09-16T14:44:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 1 -- 172.21.15.189:0/3151932486 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29c11f520 con 0x7fc29c0a25f0 2024-09-16T14:44:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 10 monclient: _renew_subs 2024-09-16T14:44:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc2a3764640 1 -- 172.21.15.189:0/3151932486 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29c135680 con 0x7fc29c0a25f0 2024-09-16T14:44:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 1 -- 172.21.15.189:0/3151932486 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc298004c50 con 0x7fc29c0a25f0 2024-09-16T14:44:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 1 -- 172.21.15.189:0/3151932486 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc298004fe0 con 0x7fc29c0a25f0 2024-09-16T14:44:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28f7fe640 4 set_mon_vals no callback set 2024-09-16T14:44:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.450+0000 7fc28dffb640 10 monclient: dump: 2024-09-16T14:44:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:02.471 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.454+0000 7fc2a3764640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.454+0000 7fc2a3764640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.454+0000 7fc2a3764640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.454+0000 7fc2a3764640 1 -- 172.21.15.189:0/3151932486 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc27c005050 con 0x7fc29c0a25f0 2024-09-16T14:44:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.454+0000 7fc28dffb640 1 -- 172.21.15.189:0/3151932486 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc2980051c0 con 0x7fc29c0a25f0 2024-09-16T14:44:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.454+0000 7fc28dffb640 1 -- 172.21.15.189:0/3151932486 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(31..31 src has 1..31) v4 ==== 4634+0+0 (unknown 3037888710 0 0) 0x7fc298043980 con 0x7fc29c0a25f0 2024-09-16T14:44:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.454+0000 7fc2a1500640 10 monclient: get_auth_request con 0x7fc280035ea0 auth_method 0 2024-09-16T14:44:02.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.482+0000 7fc28dffb640 1 -- 172.21.15.189:0/3151932486 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc298014c20 con 0x7fc29c0a25f0 2024-09-16T14:44:02.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.482+0000 7fc28dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:02.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.482+0000 7fc28dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:02.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.638+0000 7fc2a3764640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T14:44:02.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.638+0000 7fc2a3764640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T14:44:02.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.638+0000 7fc2a3764640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:02.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.638+0000 7fc2a3764640 1 -- 172.21.15.189:0/3151932486 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123} v 0) v1 -- 0x7fc27c005880 con 0x7fc29c0a25f0 2024-09-16T14:44:02.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.638+0000 7fc28dffb640 1 -- 172.21.15.189:0/3151932486 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}]=0 pool 'foo' already exists v31) v1 ==== 118+0+0 (unknown 3400972217 0 0) 0x7fc298013090 con 0x7fc29c0a25f0 2024-09-16T14:44:02.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.638+0000 7fc28dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T14:44:02.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.638+0000 7fc28dffb640 10 monclient: _finish_command 2 = system:0 pool 'foo' already exists 2024-09-16T14:44:02.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.734+0000 7fc2a3764640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T14:44:02.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.734+0000 7fc2a3764640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T14:44:02.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.734+0000 7fc2a3764640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:02.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.734+0000 7fc2a3764640 1 -- 172.21.15.189:0/3151932486 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123} v 0) v1 -- 0x7fc27c005340 con 0x7fc29c0a25f0 2024-09-16T14:44:02.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.734+0000 7fc28dffb640 1 -- 172.21.15.189:0/3151932486 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}]=0 pool 'foo' already exists v31) v1 ==== 118+0+0 (unknown 3400972217 0 0) 0x7fc2980149f0 con 0x7fc29c0a25f0 2024-09-16T14:44:02.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.734+0000 7fc28dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T14:44:02.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.734+0000 7fc28dffb640 10 monclient: _finish_command 3 = system:0 pool 'foo' already exists 2024-09-16T14:44:02.740 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foo' already exists 2024-09-16T14:44:02.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.738+0000 7fc2a3764640 1 -- 172.21.15.189:0/3151932486 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc280035ea0 legacy=0x7fc280038360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:02.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.738+0000 7fc2a3764640 10 monclient: shutdown 2024-09-16T14:44:02.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.738+0000 7fc2a3764640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:02.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.738+0000 7fc2a3764640 1 -- 172.21.15.189:0/3151932486 >> v1:172.21.15.189:6789/0 conn(0x7fc29c0a25f0 legacy=0x7fc29c139ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:02.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.738+0000 7fc2a3764640 1 -- 172.21.15.189:0/3151932486 shutdown_connections 2024-09-16T14:44:02.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.738+0000 7fc2a3764640 1 -- 172.21.15.189:0/3151932486 >> 172.21.15.189:0/3151932486 conn(0x7fc29c0a7dd0 msgr2=0x7fc29c09cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:02.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.738+0000 7fc2a3764640 1 -- 172.21.15.189:0/3151932486 shutdown_connections 2024-09-16T14:44:02.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.738+0000 7fc2a3764640 1 -- 172.21.15.189:0/3151932486 wait complete. 2024-09-16T14:44:02.757 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set foo size 1 --yes-i-really-mean-it 2024-09-16T14:44:02.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.874+0000 7f84dfd7e640 10 monclient: get_monmap_and_config 2024-09-16T14:44:02.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.874+0000 7f84dfd7e640 10 monclient: build_initial_monmap 2024-09-16T14:44:02.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.874+0000 7f84dfd7e640 10 monclient: monmap: 2024-09-16T14:44:02.880 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:02.880 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:02.880 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:02.879980+0000 2024-09-16T14:44:02.880 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:02.879980+0000 2024-09-16T14:44:02.880 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:02.880 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:02.880 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:02.880 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:02.880 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:02.880 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:02.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.874+0000 7f84dfd7e640 1 Processor -- start 2024-09-16T14:44:02.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.874+0000 7f84dfd7e640 1 -- start start 2024-09-16T14:44:02.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.874+0000 7f84dfd7e640 10 monclient: init 2024-09-16T14:44:02.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:02.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:02.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient(hunting): picked mon.noname-c con 0x7f84d8099730 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:02.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient(hunting): picked mon.noname-a con 0x7f84d8134c80 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:02.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient(hunting): picked mon.noname-b con 0x7f84d8099b10 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:02.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d8135710 con 0x7f84d8134c80 2024-09-16T14:44:02.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d80979d0 con 0x7f84d8099730 2024-09-16T14:44:02.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d81350e0 con 0x7f84d8099b10 2024-09-16T14:44:02.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:02.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient(hunting): authenticate will time out at 1024.996605s 2024-09-16T14:44:02.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84ddb1a640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f84d8099730 0x7f84d809ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:59424/0 (socket says 172.21.15.189:59424) 2024-09-16T14:44:02.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84ddb1a640 1 -- 172.21.15.189:0/2522834675 learned_addr learned my addr 172.21.15.189:0/2522834675 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:02.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dd319640 1 -- 172.21.15.189:0/2522834675 >> v1:172.21.15.26:6789/0 conn(0x7f84d8134c80 legacy=0x7f84d809f4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:02.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 1 -- 172.21.15.189:0/2522834675 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894605154 0 0) 0x7f84d80979d0 con 0x7f84d8099730 2024-09-16T14:44:02.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 10 monclient(hunting): _negotiate 2024-09-16T14:44:02.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:02.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:02.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 10 monclient(hunting): my global_id is 4660 2024-09-16T14:44:02.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 1 -- 172.21.15.189:0/2522834675 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84c4003530 con 0x7f84d8099730 2024-09-16T14:44:02.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 1 -- 172.21.15.189:0/2522834675 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2864195682 0 0) 0x7f84c4003530 con 0x7f84d8099730 2024-09-16T14:44:02.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 1 -- 172.21.15.189:0/2522834675 >> v1:172.21.15.26:6790/0 conn(0x7f84d8099b10 legacy=0x7f84d813db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:02.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 1 -- 172.21.15.189:0/2522834675 >> v1:172.21.15.26:6789/0 conn(0x7f84d8134c80 legacy=0x7f84d809f4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:02.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 10 monclient: _finish_hunting 0 2024-09-16T14:44:02.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 1 monclient: found mon.noname-c 2024-09-16T14:44:02.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:02.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:44:02.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 1 -- 172.21.15.189:0/2522834675 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d8097c40 con 0x7f84d8099730 2024-09-16T14:44:02.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 10 monclient: _finish_auth 0 2024-09-16T14:44:02.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 10 monclient: _check_auth_tickets 2024-09-16T14:44:02.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:02.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 5 monclient: authenticate success, global_id 4660 2024-09-16T14:44:02.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:02.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 1 -- 172.21.15.189:0/2522834675 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f84d4001620 con 0x7f84d8099730 2024-09-16T14:44:02.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:02.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:02.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 10 monclient: dump: 2024-09-16T14:44:02.886 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:02.886 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:02.887 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 1 -- 172.21.15.189:0/2522834675 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84d4001940 con 0x7f84d8099730 2024-09-16T14:44:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 1 -- 172.21.15.189:0/2522834675 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f84d4001d10 con 0x7f84d8099730 2024-09-16T14:44:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:02.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 4 set_mon_vals no callback set 2024-09-16T14:44:02.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:02.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:02.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dcb18640 10 monclient: dump: 2024-09-16T14:44:02.888 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:02.888 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:02.888 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:02.888 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:02.888 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:02.888 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:02.888 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:02.888 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:02.889 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:02.889 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:02.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient: shutdown 2024-09-16T14:44:02.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:02.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 1 -- 172.21.15.189:0/2522834675 >> v1:172.21.15.189:6789/0 conn(0x7f84d8099730 legacy=0x7f84d809ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:02.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 1 -- 172.21.15.189:0/2522834675 shutdown_connections 2024-09-16T14:44:02.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 1 -- 172.21.15.189:0/2522834675 >> 172.21.15.189:0/2522834675 conn(0x7f84d809fed0 msgr2=0x7f84d80a02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:02.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 1 -- 172.21.15.189:0/2522834675 shutdown_connections 2024-09-16T14:44:02.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 1 -- 172.21.15.189:0/2522834675 wait complete. 2024-09-16T14:44:02.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient: build_initial_monmap 2024-09-16T14:44:02.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient: monmap: 2024-09-16T14:44:02.889 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:02.889 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:02.890 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:02.890 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:02.890 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:02.890 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:02.890 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:02.890 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:02.890 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:02.890 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:02.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 1 Processor -- start 2024-09-16T14:44:02.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 1 -- start start 2024-09-16T14:44:02.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient: init 2024-09-16T14:44:02.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:02.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:02.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient(hunting): picked mon.noname-a con 0x7f84d8099730 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:02.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient(hunting): picked mon.noname-b con 0x7f84d8099b10 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:02.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient(hunting): picked mon.noname-c con 0x7f84d8134c80 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:02.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d809f4f0 con 0x7f84d8099730 2024-09-16T14:44:02.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d809a5c0 con 0x7f84d8099b10 2024-09-16T14:44:02.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d8130c30 con 0x7f84d8134c80 2024-09-16T14:44:02.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:02.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dfd7e640 10 monclient(hunting): authenticate will time out at 1024.999546s 2024-09-16T14:44:02.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84ddb1a640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f84d8099730 legacy=0x7f84d81c9920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:02.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dd319640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f84d8099b10 0x7f84d81ceb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:59432/0 (socket says 172.21.15.189:59432) 2024-09-16T14:44:02.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84dd319640 1 -- 172.21.15.189:0/4111454093 learned_addr learned my addr 172.21.15.189:0/4111454093 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:02.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588534783 0 0) 0x7f84d809a5c0 con 0x7f84d8099b10 2024-09-16T14:44:02.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:02.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:02.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:02.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84ce7fc640 10 monclient(hunting): my global_id is 4663 2024-09-16T14:44:02.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.878+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84bc003530 con 0x7f84d8099b10 2024-09-16T14:44:02.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1870694298 0 0) 0x7f84bc003530 con 0x7f84d8099b10 2024-09-16T14:44:02.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84d809a5c0 con 0x7f84d8099b10 2024-09-16T14:44:02.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f84c8004190 con 0x7f84d8099b10 2024-09-16T14:44:02.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:02.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:02.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 10 monclient(hunting): dump: 2024-09-16T14:44:02.896 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:02.896 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:02.896 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:02.896 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:02.896 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:02.896 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:02.896 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:02.897 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:02.897 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:02.897 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:02.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:02.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977974662 0 0) 0x7f84d8130c30 con 0x7f84d8134c80 2024-09-16T14:44:02.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:02.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 10 monclient(hunting): my global_id is 4637 2024-09-16T14:44:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84bc003530 con 0x7f84d8134c80 2024-09-16T14:44:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 793475568 0 0) 0x7f84d809a5c0 con 0x7f84d8099b10 2024-09-16T14:44:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 >> v1:172.21.15.26:6790/0 conn(0x7f84d8134c80 legacy=0x7f84d81d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 >> v1:172.21.15.26:6789/0 conn(0x7f84d8099730 legacy=0x7f84d81c9920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:44:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 1 monclient: found mon.b 2024-09-16T14:44:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d8130f10 con 0x7f84d8099b10 2024-09-16T14:44:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 10 monclient: _finish_auth 0 2024-09-16T14:44:02.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:44:02.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:02.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84dfd7e640 5 monclient: authenticate success, global_id 4663 2024-09-16T14:44:02.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84dfd7e640 10 monclient: _renew_subs 2024-09-16T14:44:02.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84dfd7e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:02.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84dfd7e640 1 -- 172.21.15.189:0/4111454093 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d80a25a0 con 0x7f84d8099b10 2024-09-16T14:44:02.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84dfd7e640 10 monclient: _renew_subs 2024-09-16T14:44:02.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84dfd7e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:02.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84dfd7e640 1 -- 172.21.15.189:0/4111454093 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d812ee00 con 0x7f84d8099b10 2024-09-16T14:44:02.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84c80044b0 con 0x7f84d8099b10 2024-09-16T14:44:02.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:02.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f84c8004840 con 0x7f84d8099b10 2024-09-16T14:44:02.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:02.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:02.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84cffff640 4 set_mon_vals no callback set 2024-09-16T14:44:02.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 10 monclient: dump: 2024-09-16T14:44:02.903 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:02.903 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:02.903 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:02.903 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:02.903 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:02.903 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:02.903 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:02.904 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:02.904 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:02.904 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:02.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f84c80049e0 con 0x7f84d8099b10 2024-09-16T14:44:02.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84dfd7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:02.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84dfd7e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:02.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84dfd7e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:02.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84dfd7e640 1 -- 172.21.15.189:0/4111454093 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c0005050 con 0x7f84d8099b10 2024-09-16T14:44:02.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(31..31 src has 1..31) v4 ==== 4634+0+0 (unknown 3037888710 0 0) 0x7f84c8043610 con 0x7f84d8099b10 2024-09-16T14:44:02.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.882+0000 7f84ddb1a640 10 monclient: get_auth_request con 0x7f84bc036130 auth_method 0 2024-09-16T14:44:02.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.886+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f84c80051b0 con 0x7f84d8099b10 2024-09-16T14:44:02.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.886+0000 7f84ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:02.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:02.886+0000 7f84ce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:03.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:03.042+0000 7f84dfd7e640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T14:44:03.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:03.042+0000 7f84dfd7e640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T14:44:03.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:03.042+0000 7f84dfd7e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:03.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:03.042+0000 7f84dfd7e640 1 -- 172.21.15.189:0/4111454093 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f84c0005880 con 0x7f84d8099b10 2024-09-16T14:44:03.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:03.162+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}]=0 set pool 4 size to 1 v32) v1 ==== 152+0+0 (unknown 4202350981 0 0) 0x7f84c8012c00 con 0x7f84d8099b10 2024-09-16T14:44:03.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:03.162+0000 7f84ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T14:44:03.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:03.162+0000 7f84ce7fc640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 1 2024-09-16T14:44:03.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:03.306+0000 7f84dfd7e640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T14:44:03.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:03.306+0000 7f84dfd7e640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T14:44:03.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:03.306+0000 7f84dfd7e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:03.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:03.306+0000 7f84dfd7e640 1 -- 172.21.15.189:0/4111454093 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f84c0005370 con 0x7f84d8099b10 2024-09-16T14:44:04.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.162+0000 7f84ce7fc640 1 -- 172.21.15.189:0/4111454093 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}]=0 set pool 4 size to 1 v33) v1 ==== 152+0+0 (unknown 2112903970 0 0) 0x7f84c8014560 con 0x7f84d8099b10 2024-09-16T14:44:04.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.162+0000 7f84ce7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T14:44:04.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.162+0000 7f84ce7fc640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 1 2024-09-16T14:44:04.167 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 4 size to 1 2024-09-16T14:44:04.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.170+0000 7f84dfd7e640 1 -- 172.21.15.189:0/4111454093 >> v1:172.21.15.189:6812/3196161582 conn(0x7f84bc036130 legacy=0x7f84bc0385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:04.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.170+0000 7f84dfd7e640 10 monclient: shutdown 2024-09-16T14:44:04.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.170+0000 7f84dfd7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:04.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.170+0000 7f84dfd7e640 1 -- 172.21.15.189:0/4111454093 >> v1:172.21.15.189:6789/0 conn(0x7f84d8099b10 legacy=0x7f84d81ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:04.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.170+0000 7f84dfd7e640 1 -- 172.21.15.189:0/4111454093 shutdown_connections 2024-09-16T14:44:04.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.170+0000 7f84dfd7e640 1 -- 172.21.15.189:0/4111454093 >> 172.21.15.189:0/4111454093 conn(0x7f84d809fed0 msgr2=0x7f84d80a1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:04.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.170+0000 7f84dfd7e640 1 -- 172.21.15.189:0/4111454093 shutdown_connections 2024-09-16T14:44:04.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.170+0000 7f84dfd7e640 1 -- 172.21.15.189:0/4111454093 wait complete. 2024-09-16T14:44:04.211 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_config_value foo min_size 1 2024-09-16T14:44:04.211 INFO:tasks.workunit.client.0.smithi189.stderr:+ local pool_name config_opt expected_val val 2024-09-16T14:44:04.211 INFO:tasks.workunit.client.0.smithi189.stderr:+ pool_name=foo 2024-09-16T14:44:04.212 INFO:tasks.workunit.client.0.smithi189.stderr:+ config_opt=min_size 2024-09-16T14:44:04.212 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_val=1 2024-09-16T14:44:04.212 INFO:tasks.workunit.client.0.smithi189.stderr:++ get_config_value_or_die foo min_size 2024-09-16T14:44:04.212 INFO:tasks.workunit.client.0.smithi189.stderr:++ local pool_name config_opt raw val 2024-09-16T14:44:04.212 INFO:tasks.workunit.client.0.smithi189.stderr:++ pool_name=foo 2024-09-16T14:44:04.212 INFO:tasks.workunit.client.0.smithi189.stderr:++ config_opt=min_size 2024-09-16T14:44:04.212 INFO:tasks.workunit.client.0.smithi189.stderr:+++ ceph osd pool get foo min_size 2024-09-16T14:44:04.621 INFO:tasks.workunit.client.0.smithi189.stderr:++ raw='min_size: 1' 2024-09-16T14:44:04.621 INFO:tasks.workunit.client.0.smithi189.stderr:++ [[ 0 -ne 0 ]] 2024-09-16T14:44:04.622 INFO:tasks.workunit.client.0.smithi189.stderr:+++ echo min_size: 1 2024-09-16T14:44:04.622 INFO:tasks.workunit.client.0.smithi189.stderr:+++ sed -e 's/[{} "]//g' 2024-09-16T14:44:04.623 INFO:tasks.workunit.client.0.smithi189.stderr:++ raw=min_size:1 2024-09-16T14:44:04.624 INFO:tasks.workunit.client.0.smithi189.stderr:+++ echo min_size:1 2024-09-16T14:44:04.624 INFO:tasks.workunit.client.0.smithi189.stderr:+++ cut -f2 -d: 2024-09-16T14:44:04.626 INFO:tasks.workunit.client.0.smithi189.stderr:++ val=1 2024-09-16T14:44:04.626 INFO:tasks.workunit.client.0.smithi189.stderr:++ echo 1 2024-09-16T14:44:04.627 INFO:tasks.workunit.client.0.smithi189.stderr:++ return 0 2024-09-16T14:44:04.627 INFO:tasks.workunit.client.0.smithi189.stderr:+ val=1 2024-09-16T14:44:04.627 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 1 != \1 ]] 2024-09-16T14:44:04.627 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set foo size 4 2024-09-16T14:44:04.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 10 monclient: get_monmap_and_config 2024-09-16T14:44:04.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 10 monclient: build_initial_monmap 2024-09-16T14:44:04.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 10 monclient: monmap: 2024-09-16T14:44:04.743 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:04.743 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:04.743 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:04.743266+0000 2024-09-16T14:44:04.743 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:04.743266+0000 2024-09-16T14:44:04.743 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:04.743 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:04.743 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:04.744 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:04.744 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:04.744 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 1 Processor -- start 2024-09-16T14:44:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 1 -- start start 2024-09-16T14:44:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 10 monclient: init 2024-09-16T14:44:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 10 monclient(hunting): picked mon.noname-c con 0x7f4500099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 10 monclient(hunting): picked mon.noname-a con 0x7f450009c550 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 10 monclient(hunting): picked mon.noname-b con 0x7f450009e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4500099e50 con 0x7f450009c550 2024-09-16T14:44:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450009dbd0 con 0x7f4500099a70 2024-09-16T14:44:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450009ee20 con 0x7f450009e6d0 2024-09-16T14:44:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 10 monclient(hunting): authenticate will time out at 1026.858923s 2024-09-16T14:44:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f4504dcb640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4500099a70 0x7f450009dfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:59458/0 (socket says 172.21.15.189:59458) 2024-09-16T14:44:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44fffff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f450009c550 legacy=0x7f45001379d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f4504dcb640 1 -- 172.21.15.189:0/2615975743 learned_addr learned my addr 172.21.15.189:0/2615975743 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 1 -- 172.21.15.189:0/2615975743 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702965976 0 0) 0x7f450009dbd0 con 0x7f4500099a70 2024-09-16T14:44:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 10 monclient(hunting): my global_id is 4672 2024-09-16T14:44:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 1 -- 172.21.15.189:0/2615975743 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44e8003530 con 0x7f4500099a70 2024-09-16T14:44:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 1 -- 172.21.15.189:0/2615975743 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3826640121 0 0) 0x7f44e8003530 con 0x7f4500099a70 2024-09-16T14:44:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 1 -- 172.21.15.189:0/2615975743 >> v1:172.21.15.26:6790/0 conn(0x7f450009e6d0 legacy=0x7f450013b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 1 -- 172.21.15.189:0/2615975743 >> v1:172.21.15.26:6789/0 conn(0x7f450009c550 legacy=0x7f45001379d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:44:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 1 monclient: found mon.noname-c 2024-09-16T14:44:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:44:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 1 -- 172.21.15.189:0/2615975743 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450009f100 con 0x7f4500099a70 2024-09-16T14:44:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 10 monclient: _finish_auth 0 2024-09-16T14:44:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:44:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 1 -- 172.21.15.189:0/2615975743 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f44f0001620 con 0x7f4500099a70 2024-09-16T14:44:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 10 monclient: dump: 2024-09-16T14:44:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:04.754 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 5 monclient: authenticate success, global_id 4672 2024-09-16T14:44:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 1 -- 172.21.15.189:0/2615975743 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44f00018d0 con 0x7f4500099a70 2024-09-16T14:44:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 1 -- 172.21.15.189:0/2615975743 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f44f0001c60 con 0x7f4500099a70 2024-09-16T14:44:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f450702f640 4 set_mon_vals no callback set 2024-09-16T14:44:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.738+0000 7f44ff7fe640 10 monclient: dump: 2024-09-16T14:44:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:04.755 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient: shutdown 2024-09-16T14:44:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 1 -- 172.21.15.189:0/2615975743 >> v1:172.21.15.189:6789/0 conn(0x7f4500099a70 legacy=0x7f450009dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 1 -- 172.21.15.189:0/2615975743 shutdown_connections 2024-09-16T14:44:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 1 -- 172.21.15.189:0/2615975743 >> 172.21.15.189:0/2615975743 conn(0x7f45000a3380 msgr2=0x7f450009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 1 -- 172.21.15.189:0/2615975743 shutdown_connections 2024-09-16T14:44:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 1 -- 172.21.15.189:0/2615975743 wait complete. 2024-09-16T14:44:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient: build_initial_monmap 2024-09-16T14:44:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient: monmap: 2024-09-16T14:44:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:04.757 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 1 Processor -- start 2024-09-16T14:44:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 1 -- start start 2024-09-16T14:44:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient: init 2024-09-16T14:44:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient(hunting): picked mon.noname-c con 0x7f4500099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient(hunting): picked mon.noname-a con 0x7f450009c550 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient(hunting): picked mon.noname-b con 0x7f450009e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450009cf20 con 0x7f450009c550 2024-09-16T14:44:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450006cd90 con 0x7f450009e6d0 2024-09-16T14:44:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450013b690 con 0x7f4500099a70 2024-09-16T14:44:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient(hunting): authenticate will time out at 1026.862222s 2024-09-16T14:44:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f45055cc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f450009e6d0 0x7f45001d1720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:59468/0 (socket says 172.21.15.189:59468) 2024-09-16T14:44:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f45055cc640 1 -- 172.21.15.189:0/3934722404 learned_addr learned my addr 172.21.15.189:0/3934722404 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fffff640 1 -- 172.21.15.189:0/3934722404 >> v1:172.21.15.26:6789/0 conn(0x7f450009c550 legacy=0x7f450009a900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359654546 0 0) 0x7f450006cd90 con 0x7f450009e6d0 2024-09-16T14:44:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:44:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient(hunting): my global_id is 4675 2024-09-16T14:44:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44ec003530 con 0x7f450009e6d0 2024-09-16T14:44:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3852313799 0 0) 0x7f44ec003530 con 0x7f450009e6d0 2024-09-16T14:44:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f450006cd90 con 0x7f450009e6d0 2024-09-16T14:44:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f44f8001c80 con 0x7f450009e6d0 2024-09-16T14:44:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient(hunting): dump: 2024-09-16T14:44:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:04.760 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3312887227 0 0) 0x7f450013b690 con 0x7f4500099a70 2024-09-16T14:44:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:44:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient(hunting): my global_id is 4646 2024-09-16T14:44:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44ec003530 con 0x7f4500099a70 2024-09-16T14:44:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1470891482 0 0) 0x7f450006cd90 con 0x7f450009e6d0 2024-09-16T14:44:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 >> v1:172.21.15.26:6790/0 conn(0x7f4500099a70 legacy=0x7f450009a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 >> v1:172.21.15.26:6789/0 conn(0x7f450009c550 legacy=0x7f450009a900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:44:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 1 monclient: found mon.b 2024-09-16T14:44:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450013b970 con 0x7f450009e6d0 2024-09-16T14:44:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient: _finish_auth 0 2024-09-16T14:44:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 5 monclient: authenticate success, global_id 4675 2024-09-16T14:44:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient: _renew_subs 2024-09-16T14:44:04.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:04.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 1 -- 172.21.15.189:0/3934722404 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450009b010 con 0x7f450009e6d0 2024-09-16T14:44:04.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient: _renew_subs 2024-09-16T14:44:04.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:04.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 1 -- 172.21.15.189:0/3934722404 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f450006b320 con 0x7f450009e6d0 2024-09-16T14:44:04.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44f80039e0 con 0x7f450009e6d0 2024-09-16T14:44:04.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:04.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f44f8003d70 con 0x7f450009e6d0 2024-09-16T14:44:04.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44feffd640 4 set_mon_vals no callback set 2024-09-16T14:44:04.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:04.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:04.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 10 monclient: dump: 2024-09-16T14:44:04.763 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:04.763 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:04.763 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:04.763 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:04.763 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:04.763 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:04.763 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:04.763 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:04.763 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:04.763 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:04.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:04.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:04.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:04.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f44f8003f50 con 0x7f450009e6d0 2024-09-16T14:44:04.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.742+0000 7f450702f640 1 -- 172.21.15.189:0/3934722404 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e0005050 con 0x7f450009e6d0 2024-09-16T14:44:04.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.746+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(33..33 src has 1..33) v4 ==== 4634+0+0 (unknown 601510561 0 0) 0x7f44f80439a0 con 0x7f450009e6d0 2024-09-16T14:44:04.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.746+0000 7f4504dcb640 10 monclient: get_auth_request con 0x7f44ec036130 auth_method 0 2024-09-16T14:44:04.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.750+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f44f8014b80 con 0x7f450009e6d0 2024-09-16T14:44:04.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.750+0000 7f44fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:04.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.750+0000 7f44fd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:04.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.910+0000 7f450702f640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T14:44:04.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.910+0000 7f450702f640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T14:44:04.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.910+0000 7f450702f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:04.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:04.910+0000 7f450702f640 1 -- 172.21.15.189:0/3934722404 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) v1 -- 0x7f44e0005880 con 0x7f450009e6d0 2024-09-16T14:44:05.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:05.262+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}]=0 set pool 4 size to 4 v34) v1 ==== 122+0+0 (unknown 3830054383 0 0) 0x7f44f8012ff0 con 0x7f450009e6d0 2024-09-16T14:44:05.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:05.262+0000 7f44fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T14:44:05.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:05.262+0000 7f44fd7fa640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 4 2024-09-16T14:44:05.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:05.382+0000 7f450702f640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T14:44:05.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:05.382+0000 7f450702f640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T14:44:05.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:05.382+0000 7f450702f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:05.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:05.386+0000 7f450702f640 1 -- 172.21.15.189:0/3934722404 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) v1 -- 0x7f44e0005300 con 0x7f450009e6d0 2024-09-16T14:44:06.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.262+0000 7f44fd7fa640 1 -- 172.21.15.189:0/3934722404 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}]=0 set pool 4 size to 4 v35) v1 ==== 122+0+0 (unknown 1724667936 0 0) 0x7f44f8012ff0 con 0x7f450009e6d0 2024-09-16T14:44:06.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.262+0000 7f44fd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T14:44:06.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.262+0000 7f44fd7fa640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 4 2024-09-16T14:44:06.269 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 4 size to 4 2024-09-16T14:44:06.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.274+0000 7f450702f640 1 -- 172.21.15.189:0/3934722404 >> v1:172.21.15.189:6812/3196161582 conn(0x7f44ec036130 legacy=0x7f44ec0385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:06.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.274+0000 7f450702f640 10 monclient: shutdown 2024-09-16T14:44:06.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.274+0000 7f450702f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:06.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.274+0000 7f450702f640 1 -- 172.21.15.189:0/3934722404 >> v1:172.21.15.189:6789/0 conn(0x7f450009e6d0 legacy=0x7f45001d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:06.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.274+0000 7f450702f640 1 -- 172.21.15.189:0/3934722404 shutdown_connections 2024-09-16T14:44:06.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.274+0000 7f450702f640 1 -- 172.21.15.189:0/3934722404 >> 172.21.15.189:0/3934722404 conn(0x7f45000a3380 msgr2=0x7f45000a0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:06.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.274+0000 7f450702f640 1 -- 172.21.15.189:0/3934722404 shutdown_connections 2024-09-16T14:44:06.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.278+0000 7f450702f640 1 -- 172.21.15.189:0/3934722404 wait complete. 2024-09-16T14:44:06.320 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_config_value foo min_size 2 2024-09-16T14:44:06.320 INFO:tasks.workunit.client.0.smithi189.stderr:+ local pool_name config_opt expected_val val 2024-09-16T14:44:06.320 INFO:tasks.workunit.client.0.smithi189.stderr:+ pool_name=foo 2024-09-16T14:44:06.320 INFO:tasks.workunit.client.0.smithi189.stderr:+ config_opt=min_size 2024-09-16T14:44:06.321 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_val=2 2024-09-16T14:44:06.321 INFO:tasks.workunit.client.0.smithi189.stderr:++ get_config_value_or_die foo min_size 2024-09-16T14:44:06.321 INFO:tasks.workunit.client.0.smithi189.stderr:++ local pool_name config_opt raw val 2024-09-16T14:44:06.321 INFO:tasks.workunit.client.0.smithi189.stderr:++ pool_name=foo 2024-09-16T14:44:06.321 INFO:tasks.workunit.client.0.smithi189.stderr:++ config_opt=min_size 2024-09-16T14:44:06.321 INFO:tasks.workunit.client.0.smithi189.stderr:+++ ceph osd pool get foo min_size 2024-09-16T14:44:06.735 INFO:tasks.workunit.client.0.smithi189.stderr:++ raw='min_size: 2' 2024-09-16T14:44:06.735 INFO:tasks.workunit.client.0.smithi189.stderr:++ [[ 0 -ne 0 ]] 2024-09-16T14:44:06.735 INFO:tasks.workunit.client.0.smithi189.stderr:+++ echo min_size: 2 2024-09-16T14:44:06.735 INFO:tasks.workunit.client.0.smithi189.stderr:+++ sed -e 's/[{} "]//g' 2024-09-16T14:44:06.738 INFO:tasks.workunit.client.0.smithi189.stderr:++ raw=min_size:2 2024-09-16T14:44:06.738 INFO:tasks.workunit.client.0.smithi189.stderr:+++ echo min_size:2 2024-09-16T14:44:06.738 INFO:tasks.workunit.client.0.smithi189.stderr:+++ cut -f2 -d: 2024-09-16T14:44:06.740 INFO:tasks.workunit.client.0.smithi189.stderr:++ val=2 2024-09-16T14:44:06.740 INFO:tasks.workunit.client.0.smithi189.stderr:++ echo 2 2024-09-16T14:44:06.740 INFO:tasks.workunit.client.0.smithi189.stderr:++ return 0 2024-09-16T14:44:06.740 INFO:tasks.workunit.client.0.smithi189.stderr:+ val=2 2024-09-16T14:44:06.740 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 2 != \2 ]] 2024-09-16T14:44:06.740 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set foo size 10 2024-09-16T14:44:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a3e3c640 10 monclient: get_monmap_and_config 2024-09-16T14:44:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a3e3c640 10 monclient: build_initial_monmap 2024-09-16T14:44:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a3e3c640 10 monclient: monmap: 2024-09-16T14:44:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:06.859390+0000 2024-09-16T14:44:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:06.859390+0000 2024-09-16T14:44:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:06.860 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:06.860 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:06.860 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:06.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a3e3c640 1 Processor -- start 2024-09-16T14:44:06.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a3e3c640 1 -- start start 2024-09-16T14:44:06.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a3e3c640 10 monclient: init 2024-09-16T14:44:06.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a3e3c640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:06.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a3e3c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:06.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a3e3c640 10 monclient(hunting): picked mon.noname-c con 0x7f649c0a19b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:06.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a3e3c640 10 monclient(hunting): picked mon.noname-a con 0x7f649c0a24a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:06.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a3e3c640 10 monclient(hunting): picked mon.noname-b con 0x7f649c09fc80 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a3e3c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c0a00d0 con 0x7f649c0a24a0 2024-09-16T14:44:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a3e3c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c09ea00 con 0x7f649c0a19b0 2024-09-16T14:44:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a3e3c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c09dde0 con 0x7f649c09fc80 2024-09-16T14:44:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a3e3c640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a3e3c640 10 monclient(hunting): authenticate will time out at 1028.975947s 2024-09-16T14:44:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.854+0000 7f64a1bd8640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f649c0a19b0 0x7f649c0a1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:59498/0 (socket says 172.21.15.189:59498) 2024-09-16T14:44:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a1bd8640 1 -- 172.21.15.189:0/352674519 learned_addr learned my addr 172.21.15.189:0/352674519 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a13d7640 1 -- 172.21.15.189:0/352674519 >> v1:172.21.15.26:6789/0 conn(0x7f649c0a24a0 legacy=0x7f649c139d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 1 -- 172.21.15.189:0/352674519 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754941514 0 0) 0x7f649c09ea00 con 0x7f649c0a19b0 2024-09-16T14:44:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 10 monclient(hunting): _negotiate 2024-09-16T14:44:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 10 monclient(hunting): my global_id is 4684 2024-09-16T14:44:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 1 -- 172.21.15.189:0/352674519 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6488003530 con 0x7f649c0a19b0 2024-09-16T14:44:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 1 -- 172.21.15.189:0/352674519 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1370914938 0 0) 0x7f6488003530 con 0x7f649c0a19b0 2024-09-16T14:44:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 1 -- 172.21.15.189:0/352674519 >> v1:172.21.15.26:6790/0 conn(0x7f649c09fc80 legacy=0x7f649c13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 1 -- 172.21.15.189:0/352674519 >> v1:172.21.15.26:6789/0 conn(0x7f649c0a24a0 legacy=0x7f649c139d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 10 monclient: _finish_hunting 0 2024-09-16T14:44:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 1 monclient: found mon.noname-c 2024-09-16T14:44:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:44:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 1 -- 172.21.15.189:0/352674519 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c09ece0 con 0x7f649c0a19b0 2024-09-16T14:44:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 10 monclient: _finish_auth 0 2024-09-16T14:44:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 10 monclient: _check_auth_tickets 2024-09-16T14:44:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 1 -- 172.21.15.189:0/352674519 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6498001620 con 0x7f649c0a19b0 2024-09-16T14:44:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 10 monclient: dump: 2024-09-16T14:44:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:06.867 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 5 monclient: authenticate success, global_id 4684 2024-09-16T14:44:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 1 -- 172.21.15.189:0/352674519 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6498001940 con 0x7f649c0a19b0 2024-09-16T14:44:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 4 set_mon_vals no callback set 2024-09-16T14:44:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 1 -- 172.21.15.189:0/352674519 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6498001d10 con 0x7f649c0a19b0 2024-09-16T14:44:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a0bd6640 10 monclient: dump: 2024-09-16T14:44:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:06.868 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 10 monclient: shutdown 2024-09-16T14:44:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 1 -- 172.21.15.189:0/352674519 >> v1:172.21.15.189:6789/0 conn(0x7f649c0a19b0 legacy=0x7f649c0a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 1 -- 172.21.15.189:0/352674519 shutdown_connections 2024-09-16T14:44:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 1 -- 172.21.15.189:0/352674519 >> 172.21.15.189:0/352674519 conn(0x7f649c0a3ab0 msgr2=0x7f649c0a3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 1 -- 172.21.15.189:0/352674519 shutdown_connections 2024-09-16T14:44:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 1 -- 172.21.15.189:0/352674519 wait complete. 2024-09-16T14:44:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 10 monclient: build_initial_monmap 2024-09-16T14:44:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 10 monclient: monmap: 2024-09-16T14:44:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:06.870 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:06.870 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:06.870 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:06.870 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:06.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 1 Processor -- start 2024-09-16T14:44:06.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 1 -- start start 2024-09-16T14:44:06.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 10 monclient: init 2024-09-16T14:44:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 10 monclient(hunting): picked mon.noname-b con 0x7f649c09fc80 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 10 monclient(hunting): picked mon.noname-a con 0x7f649c0a24a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 10 monclient(hunting): picked mon.noname-c con 0x7f649c1344d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c09e1a0 con 0x7f649c0a24a0 2024-09-16T14:44:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c0a6290 con 0x7f649c09fc80 2024-09-16T14:44:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c1c9790 con 0x7f649c1344d0 2024-09-16T14:44:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 10 monclient(hunting): authenticate will time out at 1028.978771s 2024-09-16T14:44:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a1bd8640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f649c09fc80 0x7f649c133dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:59512/0 (socket says 172.21.15.189:59512) 2024-09-16T14:44:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a1bd8640 1 -- 172.21.15.189:0/1858911695 learned_addr learned my addr 172.21.15.189:0/1858911695 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a13d7640 1 -- 172.21.15.189:0/1858911695 >> v1:172.21.15.26:6789/0 conn(0x7f649c0a24a0 legacy=0x7f649c1ce440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225382829 0 0) 0x7f649c0a6290 con 0x7f649c09fc80 2024-09-16T14:44:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient(hunting): my global_id is 4687 2024-09-16T14:44:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6480003530 con 0x7f649c09fc80 2024-09-16T14:44:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4197590423 0 0) 0x7f6480003530 con 0x7f649c09fc80 2024-09-16T14:44:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f649c0a6290 con 0x7f649c09fc80 2024-09-16T14:44:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6498001c50 con 0x7f649c09fc80 2024-09-16T14:44:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient(hunting): dump: 2024-09-16T14:44:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:06.876 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173692302 0 0) 0x7f649c1c9790 con 0x7f649c1344d0 2024-09-16T14:44:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient(hunting): my global_id is 4658 2024-09-16T14:44:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6480003530 con 0x7f649c1344d0 2024-09-16T14:44:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3721523954 0 0) 0x7f649c0a6290 con 0x7f649c09fc80 2024-09-16T14:44:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 >> v1:172.21.15.26:6790/0 conn(0x7f649c1344d0 legacy=0x7f649c1d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 >> v1:172.21.15.26:6789/0 conn(0x7f649c0a24a0 legacy=0x7f649c1ce440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient: _finish_hunting 0 2024-09-16T14:44:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 1 monclient: found mon.b 2024-09-16T14:44:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c121550 con 0x7f649c09fc80 2024-09-16T14:44:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient: _finish_auth 0 2024-09-16T14:44:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient: _check_auth_tickets 2024-09-16T14:44:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 5 monclient: authenticate success, global_id 4687 2024-09-16T14:44:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 10 monclient: _renew_subs 2024-09-16T14:44:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 1 -- 172.21.15.189:0/1858911695 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f649c11fb40 con 0x7f649c09fc80 2024-09-16T14:44:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 10 monclient: _renew_subs 2024-09-16T14:44:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64a3e3c640 1 -- 172.21.15.189:0/1858911695 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f649c11f1f0 con 0x7f649c09fc80 2024-09-16T14:44:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6498001180 con 0x7f649c09fc80 2024-09-16T14:44:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6498004da0 con 0x7f649c09fc80 2024-09-16T14:44:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64937fe640 4 set_mon_vals no callback set 2024-09-16T14:44:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.858+0000 7f64927fc640 10 monclient: dump: 2024-09-16T14:44:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:06.880 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.862+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f649800f810 con 0x7f649c09fc80 2024-09-16T14:44:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.862+0000 7f64a3e3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.862+0000 7f64a3e3c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.862+0000 7f64a3e3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.862+0000 7f64a3e3c640 1 -- 172.21.15.189:0/1858911695 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6484005050 con 0x7f649c09fc80 2024-09-16T14:44:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.862+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(35..35 src has 1..35) v4 ==== 4634+0+0 (unknown 1409889687 0 0) 0x7f6498043620 con 0x7f649c09fc80 2024-09-16T14:44:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.862+0000 7f64a13d7640 10 monclient: get_auth_request con 0x7f6480036270 auth_method 0 2024-09-16T14:44:06.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.866+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6498014710 con 0x7f649c09fc80 2024-09-16T14:44:06.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.866+0000 7f64927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:06.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:06.866+0000 7f64927fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:07.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:07.022+0000 7f64a3e3c640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T14:44:07.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:07.022+0000 7f64a3e3c640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T14:44:07.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:07.022+0000 7f64a3e3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:07.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:07.022+0000 7f64a3e3c640 1 -- 172.21.15.189:0/1858911695 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"} v 0) v1 -- 0x7f6484005880 con 0x7f649c09fc80 2024-09-16T14:44:07.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:07.282+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}]=0 set pool 4 size to 10 v36) v1 ==== 124+0+0 (unknown 1212883190 0 0) 0x7f6498012db0 con 0x7f649c09fc80 2024-09-16T14:44:07.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:07.282+0000 7f64927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T14:44:07.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:07.282+0000 7f64927fc640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 10 2024-09-16T14:44:07.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:07.426+0000 7f64a3e3c640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T14:44:07.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:07.426+0000 7f64a3e3c640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T14:44:07.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:07.426+0000 7f64a3e3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:07.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:07.426+0000 7f64a3e3c640 1 -- 172.21.15.189:0/1858911695 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"} v 0) v1 -- 0x7f6484005300 con 0x7f649c09fc80 2024-09-16T14:44:08.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.294+0000 7f64927fc640 1 -- 172.21.15.189:0/1858911695 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}]=0 set pool 4 size to 10 v37) v1 ==== 124+0+0 (unknown 1012676041 0 0) 0x7f6498014530 con 0x7f649c09fc80 2024-09-16T14:44:08.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.294+0000 7f64927fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T14:44:08.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.294+0000 7f64927fc640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 10 2024-09-16T14:44:08.298 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 4 size to 10 2024-09-16T14:44:08.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.310+0000 7f64a3e3c640 1 -- 172.21.15.189:0/1858911695 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6480036270 legacy=0x7f6480038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:08.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.310+0000 7f64a3e3c640 10 monclient: shutdown 2024-09-16T14:44:08.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.310+0000 7f64a3e3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:08.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.310+0000 7f64a3e3c640 1 -- 172.21.15.189:0/1858911695 >> v1:172.21.15.189:6789/0 conn(0x7f649c09fc80 legacy=0x7f649c133dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:08.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.310+0000 7f64a3e3c640 1 -- 172.21.15.189:0/1858911695 shutdown_connections 2024-09-16T14:44:08.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.310+0000 7f64a3e3c640 1 -- 172.21.15.189:0/1858911695 >> 172.21.15.189:0/1858911695 conn(0x7f649c0a3ab0 msgr2=0x7f649c0a4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:08.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.310+0000 7f64a3e3c640 1 -- 172.21.15.189:0/1858911695 shutdown_connections 2024-09-16T14:44:08.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.310+0000 7f64a3e3c640 1 -- 172.21.15.189:0/1858911695 wait complete. 2024-09-16T14:44:08.353 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_config_value foo min_size 5 2024-09-16T14:44:08.353 INFO:tasks.workunit.client.0.smithi189.stderr:+ local pool_name config_opt expected_val val 2024-09-16T14:44:08.353 INFO:tasks.workunit.client.0.smithi189.stderr:+ pool_name=foo 2024-09-16T14:44:08.353 INFO:tasks.workunit.client.0.smithi189.stderr:+ config_opt=min_size 2024-09-16T14:44:08.353 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_val=5 2024-09-16T14:44:08.353 INFO:tasks.workunit.client.0.smithi189.stderr:++ get_config_value_or_die foo min_size 2024-09-16T14:44:08.353 INFO:tasks.workunit.client.0.smithi189.stderr:++ local pool_name config_opt raw val 2024-09-16T14:44:08.353 INFO:tasks.workunit.client.0.smithi189.stderr:++ pool_name=foo 2024-09-16T14:44:08.353 INFO:tasks.workunit.client.0.smithi189.stderr:++ config_opt=min_size 2024-09-16T14:44:08.354 INFO:tasks.workunit.client.0.smithi189.stderr:+++ ceph osd pool get foo min_size 2024-09-16T14:44:08.772 INFO:tasks.workunit.client.0.smithi189.stderr:++ raw='min_size: 5' 2024-09-16T14:44:08.772 INFO:tasks.workunit.client.0.smithi189.stderr:++ [[ 0 -ne 0 ]] 2024-09-16T14:44:08.773 INFO:tasks.workunit.client.0.smithi189.stderr:+++ echo min_size: 5 2024-09-16T14:44:08.773 INFO:tasks.workunit.client.0.smithi189.stderr:+++ sed -e 's/[{} "]//g' 2024-09-16T14:44:08.774 INFO:tasks.workunit.client.0.smithi189.stderr:++ raw=min_size:5 2024-09-16T14:44:08.775 INFO:tasks.workunit.client.0.smithi189.stderr:+++ echo min_size:5 2024-09-16T14:44:08.776 INFO:tasks.workunit.client.0.smithi189.stderr:+++ cut -f2 -d: 2024-09-16T14:44:08.777 INFO:tasks.workunit.client.0.smithi189.stderr:++ val=5 2024-09-16T14:44:08.777 INFO:tasks.workunit.client.0.smithi189.stderr:++ echo 5 2024-09-16T14:44:08.777 INFO:tasks.workunit.client.0.smithi189.stderr:++ return 0 2024-09-16T14:44:08.777 INFO:tasks.workunit.client.0.smithi189.stderr:+ val=5 2024-09-16T14:44:08.778 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 5 != \5 ]] 2024-09-16T14:44:08.778 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool set foo size 0 2024-09-16T14:44:08.778 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:44:08.778 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set foo size 0 2024-09-16T14:44:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.886+0000 7f2ba207e640 10 monclient: get_monmap_and_config 2024-09-16T14:44:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.886+0000 7f2ba207e640 10 monclient: build_initial_monmap 2024-09-16T14:44:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.886+0000 7f2ba207e640 10 monclient: monmap: 2024-09-16T14:44:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:08.893519+0000 2024-09-16T14:44:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:08.893519+0000 2024-09-16T14:44:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:08.894 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 1 Processor -- start 2024-09-16T14:44:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 1 -- start start 2024-09-16T14:44:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 10 monclient: init 2024-09-16T14:44:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 10 monclient(hunting): picked mon.noname-b con 0x7f2b9c0a1a90 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 10 monclient(hunting): picked mon.noname-a con 0x7f2b9c09fdf0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 10 monclient(hunting): picked mon.noname-c con 0x7f2b9c0a2580 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c09e670 con 0x7f2b9c09fdf0 2024-09-16T14:44:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c09eb20 con 0x7f2b9c0a2580 2024-09-16T14:44:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c0a3020 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 10 monclient(hunting): authenticate will time out at 1031.009207s 2024-09-16T14:44:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b92ffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2b9c09fdf0 legacy=0x7f2b9c139d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2b9c0a2580 0x7f2b9c13d4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38808/0 (socket says 172.21.15.189:38808) 2024-09-16T14:44:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9bfff640 1 -- 172.21.15.189:0/2936993916 learned_addr learned my addr 172.21.15.189:0/2936993916 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 1 -- 172.21.15.189:0/2936993916 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412528406 0 0) 0x7f2b9c09eb20 con 0x7f2b9c0a2580 2024-09-16T14:44:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 10 monclient(hunting): my global_id is 4696 2024-09-16T14:44:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 1 -- 172.21.15.189:0/2936993916 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b88003530 con 0x7f2b9c0a2580 2024-09-16T14:44:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 1 -- 172.21.15.189:0/2936993916 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2457200625 0 0) 0x7f2b88003530 con 0x7f2b9c0a2580 2024-09-16T14:44:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 1 -- 172.21.15.189:0/2936993916 >> v1:172.21.15.26:6790/0 conn(0x7f2b9c0a1a90 legacy=0x7f2b9c0a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 1 -- 172.21.15.189:0/2936993916 >> v1:172.21.15.26:6789/0 conn(0x7f2b9c09fdf0 legacy=0x7f2b9c139d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 1 monclient: found mon.noname-c 2024-09-16T14:44:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:44:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 1 -- 172.21.15.189:0/2936993916 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c09ee00 con 0x7f2b9c0a2580 2024-09-16T14:44:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 10 monclient: _finish_auth 0 2024-09-16T14:44:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 5 monclient: authenticate success, global_id 4696 2024-09-16T14:44:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 1 -- 172.21.15.189:0/2936993916 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b940019c0 con 0x7f2b9c0a2580 2024-09-16T14:44:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 10 monclient: dump: 2024-09-16T14:44:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:08.904 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 1 -- 172.21.15.189:0/2936993916 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b94001ce0 con 0x7f2b9c0a2580 2024-09-16T14:44:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 4 set_mon_vals no callback set 2024-09-16T14:44:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 1 -- 172.21.15.189:0/2936993916 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b94002070 con 0x7f2b9c0a2580 2024-09-16T14:44:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2b9affd640 10 monclient: dump: 2024-09-16T14:44:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:08.906 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 10 monclient: shutdown 2024-09-16T14:44:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 1 -- 172.21.15.189:0/2936993916 >> v1:172.21.15.189:6789/0 conn(0x7f2b9c0a2580 legacy=0x7f2b9c13d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 1 -- 172.21.15.189:0/2936993916 shutdown_connections 2024-09-16T14:44:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 1 -- 172.21.15.189:0/2936993916 >> 172.21.15.189:0/2936993916 conn(0x7f2b9c0a7a80 msgr2=0x7f2b9c0a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 1 -- 172.21.15.189:0/2936993916 shutdown_connections 2024-09-16T14:44:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 1 -- 172.21.15.189:0/2936993916 wait complete. 2024-09-16T14:44:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 10 monclient: build_initial_monmap 2024-09-16T14:44:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 10 monclient: monmap: 2024-09-16T14:44:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:08.907 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 1 Processor -- start 2024-09-16T14:44:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 1 -- start start 2024-09-16T14:44:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 10 monclient: init 2024-09-16T14:44:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.890+0000 7f2ba207e640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:44:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 10 monclient(hunting): picked mon.noname-c con 0x7f2b9c09fdf0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 10 monclient(hunting): picked mon.noname-b con 0x7f2b9c0a1a90 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 10 monclient(hunting): picked mon.noname-a con 0x7f2b9c0a2580 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c13cd60 con 0x7f2b9c0a2580 2024-09-16T14:44:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c13c390 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c09e280 con 0x7f2b9c09fdf0 2024-09-16T14:44:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 10 monclient(hunting): authenticate will time out at 1031.012099s 2024-09-16T14:44:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b92ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2b9c0a1a90 0x7f2b9c1ce310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38822/0 (socket says 172.21.15.189:38822) 2024-09-16T14:44:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b9bfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2b9c0a2580 legacy=0x7f2b9c1d1a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b92ffd640 1 -- 172.21.15.189:0/2398242070 learned_addr learned my addr 172.21.15.189:0/2398242070 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023798907 0 0) 0x7f2b9c13c390 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient(hunting): my global_id is 4699 2024-09-16T14:44:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b7c003530 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1761785755 0 0) 0x7f2b7c003530 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b9c13c390 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b84004190 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient(hunting): dump: 2024-09-16T14:44:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:08.911 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:08.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:08.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679862686 0 0) 0x7f2b9c09e280 con 0x7f2b9c09fdf0 2024-09-16T14:44:08.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:08.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:08.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:08.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient(hunting): my global_id is 4670 2024-09-16T14:44:08.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b7c003530 con 0x7f2b9c09fdf0 2024-09-16T14:44:08.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 886927213 0 0) 0x7f2b9c13c390 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 >> v1:172.21.15.26:6790/0 conn(0x7f2b9c09fdf0 legacy=0x7f2b9c1c5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:08.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 >> v1:172.21.15.26:6789/0 conn(0x7f2b9c0a2580 legacy=0x7f2b9c1d1a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:08.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:44:08.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 1 monclient: found mon.b 2024-09-16T14:44:08.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:08.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:08.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c13ca80 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient: _finish_auth 0 2024-09-16T14:44:08.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:44:08.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:08.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 5 monclient: authenticate success, global_id 4699 2024-09-16T14:44:08.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 10 monclient: _renew_subs 2024-09-16T14:44:08.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:08.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 1 -- 172.21.15.189:0/2398242070 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9c13afc0 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 10 monclient: _renew_subs 2024-09-16T14:44:08.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:08.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 1 -- 172.21.15.189:0/2398242070 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9c13bb90 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b840044b0 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:08.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b84004840 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:08.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:08.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b9a7fc640 4 set_mon_vals no callback set 2024-09-16T14:44:08.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 10 monclient: dump: 2024-09-16T14:44:08.913 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:08.913 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:08.913 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:08.913 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:08.913 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:08.913 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:08.913 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:08.914 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:08.914 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:08.914 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:08.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2b84004a20 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:08.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:08.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:08.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2ba207e640 1 -- 172.21.15.189:0/2398242070 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b80005050 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (unknown 3430718669 0 0) 0x7f2b84043720 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.894+0000 7f2b9b7fe640 10 monclient: get_auth_request con 0x7f2b7c0360e0 auth_method 0 2024-09-16T14:44:08.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.898+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2b840148a0 con 0x7f2b9c0a1a90 2024-09-16T14:44:08.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.898+0000 7f2b98ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:08.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:08.898+0000 7f2b98ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:09.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.058+0000 7f2ba207e640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-16T14:44:09.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.058+0000 7f2ba207e640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-16T14:44:09.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.058+0000 7f2ba207e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.058+0000 7f2ba207e640 1 -- 172.21.15.189:0/2398242070 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"} v 0) v1 -- 0x7f2b80005880 con 0x7f2b9c0a1a90 2024-09-16T14:44:09.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.058+0000 7f2b98ff9640 1 -- 172.21.15.189:0/2398242070 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}]=-22 pool size must be between 1 and 10 v37) v1 ==== 136+0+0 (unknown 4196829380 0 0) 0x7f2b84012d10 con 0x7f2b9c0a1a90 2024-09-16T14:44:09.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.058+0000 7f2b98ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-16T14:44:09.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.058+0000 7f2b98ff9640 10 monclient: _finish_command 2 = mon:22 pool size must be between 1 and 10 2024-09-16T14:44:09.065 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: pool size must be between 1 and 10 2024-09-16T14:44:09.069 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:44:09.069 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool set foo size 20 2024-09-16T14:44:09.069 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:44:09.069 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set foo size 20 2024-09-16T14:44:09.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7c28cc640 10 monclient: get_monmap_and_config 2024-09-16T14:44:09.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7c28cc640 10 monclient: build_initial_monmap 2024-09-16T14:44:09.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7c28cc640 10 monclient: monmap: 2024-09-16T14:44:09.188 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:09.188 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:09.188 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:09.187934+0000 2024-09-16T14:44:09.188 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:09.187934+0000 2024-09-16T14:44:09.188 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:09.188 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:09.188 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:09.188 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:09.188 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:09.188 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7c28cc640 1 Processor -- start 2024-09-16T14:44:09.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7c28cc640 1 -- start start 2024-09-16T14:44:09.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7c28cc640 10 monclient: init 2024-09-16T14:44:09.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7c28cc640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:09.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7c28cc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:09.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7c28cc640 10 monclient(hunting): picked mon.noname-a con 0x7fa7bc099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:09.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7c28cc640 10 monclient(hunting): picked mon.noname-c con 0x7fa7bc09c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:09.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7c28cc640 10 monclient(hunting): picked mon.noname-b con 0x7fa7bc09e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:09.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7c28cc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc099e50 con 0x7fa7bc099a70 2024-09-16T14:44:09.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7c28cc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc09dbd0 con 0x7fa7bc09c550 2024-09-16T14:44:09.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7c28cc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc09ee20 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7c28cc640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7c28cc640 10 monclient(hunting): authenticate will time out at 1031.303728s 2024-09-16T14:44:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7bbfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fa7bc099a70 legacy=0x7fa7bc09dfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7bb7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa7bc09c550 0x7fa7bc1379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38830/0 (socket says 172.21.15.189:38830) 2024-09-16T14:44:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.182+0000 7fa7bb7fe640 1 -- 172.21.15.189:0/855162753 learned_addr learned my addr 172.21.15.189:0/855162753 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 1 -- 172.21.15.189:0/855162753 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657323674 0 0) 0x7fa7bc09dbd0 con 0x7fa7bc09c550 2024-09-16T14:44:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient(hunting): my global_id is 4702 2024-09-16T14:44:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 1 -- 172.21.15.189:0/855162753 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a4003530 con 0x7fa7bc09c550 2024-09-16T14:44:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 1 -- 172.21.15.189:0/855162753 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854040880 0 0) 0x7fa7bc09ee20 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient(hunting): my global_id is 4673 2024-09-16T14:44:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 1 -- 172.21.15.189:0/855162753 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7bc09dbd0 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 1 -- 172.21.15.189:0/855162753 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4173004961 0 0) 0x7fa7a4003530 con 0x7fa7bc09c550 2024-09-16T14:44:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 1 -- 172.21.15.189:0/855162753 >> v1:172.21.15.26:6790/0 conn(0x7fa7bc09e6d0 legacy=0x7fa7bc13b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 1 -- 172.21.15.189:0/855162753 >> v1:172.21.15.26:6789/0 conn(0x7fa7bc099a70 legacy=0x7fa7bc09dfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 1 monclient: found mon.noname-c 2024-09-16T14:44:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 1 -- 172.21.15.189:0/855162753 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7bc09f100 con 0x7fa7bc09c550 2024-09-16T14:44:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient: _finish_auth 0 2024-09-16T14:44:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 5 monclient: authenticate success, global_id 4702 2024-09-16T14:44:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 1 -- 172.21.15.189:0/855162753 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa7b0001620 con 0x7fa7bc09c550 2024-09-16T14:44:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient: dump: 2024-09-16T14:44:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.199 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:09.199 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:09.199 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:09.199 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:09.199 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:09.199 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:09.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 1 -- 172.21.15.189:0/855162753 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7b0001890 con 0x7fa7bc09c550 2024-09-16T14:44:09.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:09.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 1 -- 172.21.15.189:0/855162753 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa7b0001c20 con 0x7fa7bc09c550 2024-09-16T14:44:09.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:09.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 4 set_mon_vals no callback set 2024-09-16T14:44:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7baffd640 10 monclient: dump: 2024-09-16T14:44:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:09.200 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 10 monclient: shutdown 2024-09-16T14:44:09.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:09.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 1 -- 172.21.15.189:0/855162753 >> v1:172.21.15.189:6789/0 conn(0x7fa7bc09c550 legacy=0x7fa7bc1379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:09.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 1 -- 172.21.15.189:0/855162753 shutdown_connections 2024-09-16T14:44:09.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 1 -- 172.21.15.189:0/855162753 >> 172.21.15.189:0/855162753 conn(0x7fa7bc0a3380 msgr2=0x7fa7bc09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:09.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 1 -- 172.21.15.189:0/855162753 shutdown_connections 2024-09-16T14:44:09.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 1 -- 172.21.15.189:0/855162753 wait complete. 2024-09-16T14:44:09.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 10 monclient: build_initial_monmap 2024-09-16T14:44:09.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 10 monclient: monmap: 2024-09-16T14:44:09.201 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:09.201 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:09.201 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:09.202 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 1 Processor -- start 2024-09-16T14:44:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 1 -- start start 2024-09-16T14:44:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 10 monclient: init 2024-09-16T14:44:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 10 monclient(hunting): picked mon.noname-c con 0x7fa7bc099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:09.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 10 monclient(hunting): picked mon.noname-a con 0x7fa7bc09c550 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:09.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 10 monclient(hunting): picked mon.noname-b con 0x7fa7bc09e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:09.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc09cf20 con 0x7fa7bc09c550 2024-09-16T14:44:09.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc11b2f0 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc06cc70 con 0x7fa7bc099a70 2024-09-16T14:44:09.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:09.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 10 monclient(hunting): authenticate will time out at 1031.306861s 2024-09-16T14:44:09.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7bb7fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fa7bc09c550 legacy=0x7fa7bc134310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:09.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c0e69640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa7bc09e6d0 0x7fa7bc1d1720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38832/0 (socket says 172.21.15.189:38832) 2024-09-16T14:44:09.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c0e69640 1 -- 172.21.15.189:0/2144361154 learned_addr learned my addr 172.21.15.189:0/2144361154 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:09.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392838663 0 0) 0x7fa7bc11b2f0 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:09.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:09.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:09.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient(hunting): my global_id is 4705 2024-09-16T14:44:09.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a8003530 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3499778173 0 0) 0x7fa7a8003530 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7bc11b2f0 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa7b4001c60 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:09.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:09.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient(hunting): dump: 2024-09-16T14:44:09.204 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:09.204 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:09.205 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.205 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.205 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:09.205 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:09.205 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:09.205 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:09.205 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:09.205 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:09.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298881617 0 0) 0x7fa7bc06cc70 con 0x7fa7bc099a70 2024-09-16T14:44:09.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:09.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:09.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:09.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient(hunting): my global_id is 4676 2024-09-16T14:44:09.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a8003530 con 0x7fa7bc099a70 2024-09-16T14:44:09.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 453656439 0 0) 0x7fa7bc11b2f0 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 >> v1:172.21.15.26:6790/0 conn(0x7fa7bc099a70 legacy=0x7fa7bc133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:09.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 >> v1:172.21.15.26:6789/0 conn(0x7fa7bc09c550 legacy=0x7fa7bc134310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:09.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:44:09.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 1 monclient: found mon.b 2024-09-16T14:44:09.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:09.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7bc119560 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient: _finish_auth 0 2024-09-16T14:44:09.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:44:09.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:09.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 5 monclient: authenticate success, global_id 4705 2024-09-16T14:44:09.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 10 monclient: _renew_subs 2024-09-16T14:44:09.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 1 -- 172.21.15.189:0/2144361154 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7bc11b070 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 10 monclient: _renew_subs 2024-09-16T14:44:09.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7c28cc640 1 -- 172.21.15.189:0/2144361154 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7bc11b5d0 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7b4002030 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:09.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa7b4003e70 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:09.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:09.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7ba7fc640 4 set_mon_vals no callback set 2024-09-16T14:44:09.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.186+0000 7fa7b8ff9640 10 monclient: dump: 2024-09-16T14:44:09.208 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:09.208 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:09.208 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.208 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.209 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:09.209 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:09.209 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:09.209 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:09.209 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:09.209 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.190+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa7b4003fd0 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.190+0000 7fa7c28cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:09.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.190+0000 7fa7c28cc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:09.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.190+0000 7fa7c28cc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.190+0000 7fa7c28cc640 1 -- 172.21.15.189:0/2144361154 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa79c005050 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.190+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (unknown 3430718669 0 0) 0x7fa7b40439e0 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.190+0000 7fa7bbfff640 10 monclient: get_auth_request con 0x7fa7a8036130 auth_method 0 2024-09-16T14:44:09.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.194+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa7b4014b60 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.194+0000 7fa7b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:09.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.194+0000 7fa7b8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:09.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.354+0000 7fa7c28cc640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-16T14:44:09.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.354+0000 7fa7c28cc640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-16T14:44:09.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.354+0000 7fa7c28cc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.354+0000 7fa7c28cc640 1 -- 172.21.15.189:0/2144361154 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"} v 0) v1 -- 0x7fa79c005880 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.354+0000 7fa7b8ff9640 1 -- 172.21.15.189:0/2144361154 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}]=-22 pool size must be between 1 and 10 v37) v1 ==== 137+0+0 (unknown 3647764158 0 0) 0x7fa7b4012fd0 con 0x7fa7bc09e6d0 2024-09-16T14:44:09.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.354+0000 7fa7b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-16T14:44:09.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.354+0000 7fa7b8ff9640 10 monclient: _finish_command 2 = mon:22 pool size must be between 1 and 10 2024-09-16T14:44:09.360 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: pool size must be between 1 and 10 2024-09-16T14:44:09.365 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:44:09.365 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool delete foo 2024-09-16T14:44:09.366 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:44:09.366 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete foo 2024-09-16T14:44:09.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82595f9640 10 monclient: get_monmap_and_config 2024-09-16T14:44:09.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82595f9640 10 monclient: build_initial_monmap 2024-09-16T14:44:09.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82595f9640 10 monclient: monmap: 2024-09-16T14:44:09.476 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:09.476 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:09.476 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:09.476137+0000 2024-09-16T14:44:09.476 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:09.476137+0000 2024-09-16T14:44:09.476 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:09.476 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:09.476 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:09.476 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:09.477 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:09.477 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82595f9640 1 Processor -- start 2024-09-16T14:44:09.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82595f9640 1 -- start start 2024-09-16T14:44:09.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82595f9640 10 monclient: init 2024-09-16T14:44:09.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82595f9640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:09.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82595f9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:09.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82595f9640 10 monclient(hunting): picked mon.noname-b con 0x7f82541315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:09.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82595f9640 10 monclient(hunting): picked mon.noname-a con 0x7f825409b320 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:09.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82595f9640 10 monclient(hunting): picked mon.noname-c con 0x7f8254132090 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:09.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82595f9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825409edd0 con 0x7f825409b320 2024-09-16T14:44:09.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82595f9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825409bc00 con 0x7f8254132090 2024-09-16T14:44:09.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82595f9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825409e900 con 0x7f82541315a0 2024-09-16T14:44:09.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82595f9640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:09.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82595f9640 10 monclient(hunting): authenticate will time out at 1031.591760s 2024-09-16T14:44:09.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82537fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8254132090 0x7f825413a200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38848/0 (socket says 172.21.15.189:38848) 2024-09-16T14:44:09.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82537fe640 1 -- 172.21.15.189:0/2148598492 learned_addr learned my addr 172.21.15.189:0/2148598492 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:09.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.470+0000 7f82527fc640 1 -- 172.21.15.189:0/2148598492 >> v1:172.21.15.26:6789/0 conn(0x7f825409b320 legacy=0x7f8254136ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:09.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 1 -- 172.21.15.189:0/2148598492 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213629856 0 0) 0x7f825409bc00 con 0x7f8254132090 2024-09-16T14:44:09.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:09.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:09.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:09.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 10 monclient(hunting): my global_id is 4708 2024-09-16T14:44:09.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 1 -- 172.21.15.189:0/2148598492 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f823c003530 con 0x7f8254132090 2024-09-16T14:44:09.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 1 -- 172.21.15.189:0/2148598492 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4145943369 0 0) 0x7f823c003530 con 0x7f8254132090 2024-09-16T14:44:09.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 1 -- 172.21.15.189:0/2148598492 >> v1:172.21.15.26:6790/0 conn(0x7f82541315a0 legacy=0x7f8254131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:09.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 1 -- 172.21.15.189:0/2148598492 >> v1:172.21.15.26:6789/0 conn(0x7f825409b320 legacy=0x7f8254136ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:09.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:44:09.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 1 monclient: found mon.noname-c 2024-09-16T14:44:09.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:09.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 1 -- 172.21.15.189:0/2148598492 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825409b920 con 0x7f8254132090 2024-09-16T14:44:09.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 10 monclient: _finish_auth 0 2024-09-16T14:44:09.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:44:09.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:09.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 1 -- 172.21.15.189:0/2148598492 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f824c0019a0 con 0x7f8254132090 2024-09-16T14:44:09.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:09.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:09.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 10 monclient: dump: 2024-09-16T14:44:09.486 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:09.487 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:09.487 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.487 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.487 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:09.487 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:09.487 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:09.487 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:09.487 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:09.487 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 5 monclient: authenticate success, global_id 4708 2024-09-16T14:44:09.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:09.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 1 -- 172.21.15.189:0/2148598492 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f824c001cc0 con 0x7f8254132090 2024-09-16T14:44:09.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:09.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 1 -- 172.21.15.189:0/2148598492 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f824c002090 con 0x7f8254132090 2024-09-16T14:44:09.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:09.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 4 set_mon_vals no callback set 2024-09-16T14:44:09.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:09.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:09.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f8251ffb640 10 monclient: dump: 2024-09-16T14:44:09.488 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:09.488 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:09.488 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.489 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.489 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:09.489 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:09.489 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:09.489 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:09.489 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:09.489 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 10 monclient: shutdown 2024-09-16T14:44:09.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:09.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 1 -- 172.21.15.189:0/2148598492 >> v1:172.21.15.189:6789/0 conn(0x7f8254132090 legacy=0x7f825413a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:09.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 1 -- 172.21.15.189:0/2148598492 shutdown_connections 2024-09-16T14:44:09.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 1 -- 172.21.15.189:0/2148598492 >> 172.21.15.189:0/2148598492 conn(0x7f82540a3320 msgr2=0x7f825409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:09.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 1 -- 172.21.15.189:0/2148598492 shutdown_connections 2024-09-16T14:44:09.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 1 -- 172.21.15.189:0/2148598492 wait complete. 2024-09-16T14:44:09.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 10 monclient: build_initial_monmap 2024-09-16T14:44:09.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 10 monclient: monmap: 2024-09-16T14:44:09.490 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:09.490 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:09.490 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:09.490 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:09.491 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:09.491 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:09.491 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:09.491 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:09.491 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:09.491 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 1 Processor -- start 2024-09-16T14:44:09.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 1 -- start start 2024-09-16T14:44:09.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 10 monclient: init 2024-09-16T14:44:09.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:09.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:09.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 10 monclient(hunting): picked mon.noname-c con 0x7f82541315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:09.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 10 monclient(hunting): picked mon.noname-a con 0x7f8254132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:09.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 10 monclient(hunting): picked mon.noname-b con 0x7f8254130e20 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:09.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825409cd20 con 0x7f8254132090 2024-09-16T14:44:09.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82541b97e0 con 0x7f8254130e20 2024-09-16T14:44:09.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825406b380 con 0x7f82541315a0 2024-09-16T14:44:09.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:09.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 10 monclient(hunting): authenticate will time out at 1031.594891s 2024-09-16T14:44:09.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82537fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8254130e20 0x7f82541d19f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38850/0 (socket says 172.21.15.189:38850) 2024-09-16T14:44:09.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82537fe640 1 -- 172.21.15.189:0/1710554513 learned_addr learned my addr 172.21.15.189:0/1710554513 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:09.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82527fc640 1 -- 172.21.15.189:0/1710554513 >> v1:172.21.15.26:6789/0 conn(0x7f8254132090 legacy=0x7f82541ce2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:09.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 1 -- 172.21.15.189:0/1710554513 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074810746 0 0) 0x7f82541b97e0 con 0x7f8254130e20 2024-09-16T14:44:09.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:09.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:09.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:09.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 10 monclient(hunting): my global_id is 4711 2024-09-16T14:44:09.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 1 -- 172.21.15.189:0/1710554513 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8240003530 con 0x7f8254130e20 2024-09-16T14:44:09.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 1 -- 172.21.15.189:0/1710554513 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 888847062 0 0) 0x7f8240003530 con 0x7f8254130e20 2024-09-16T14:44:09.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 1 -- 172.21.15.189:0/1710554513 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82541b97e0 con 0x7f8254130e20 2024-09-16T14:44:09.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 1 -- 172.21.15.189:0/1710554513 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f824c0017c0 con 0x7f8254130e20 2024-09-16T14:44:09.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:09.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:09.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 10 monclient(hunting): dump: 2024-09-16T14:44:09.494 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:09.494 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:09.494 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.494 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.494 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:09.494 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:09.494 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:09.495 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:09.495 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:09.495 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:09.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 1 -- 172.21.15.189:0/1710554513 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 959094 0 0) 0x7f82541b97e0 con 0x7f8254130e20 2024-09-16T14:44:09.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 1 -- 172.21.15.189:0/1710554513 >> v1:172.21.15.26:6790/0 conn(0x7f82541315a0 legacy=0x7f8254130710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:09.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 1 -- 172.21.15.189:0/1710554513 >> v1:172.21.15.26:6789/0 conn(0x7f8254132090 legacy=0x7f82541ce2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:09.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:44:09.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 1 monclient: found mon.b 2024-09-16T14:44:09.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:09.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 1 -- 172.21.15.189:0/1710554513 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825411f0e0 con 0x7f8254130e20 2024-09-16T14:44:09.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:44:09.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:44:09.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f823b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:09.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 5 monclient: authenticate success, global_id 4711 2024-09-16T14:44:09.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 10 monclient: _renew_subs 2024-09-16T14:44:09.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 1 -- 172.21.15.189:0/1710554513 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f825409a770 con 0x7f8254130e20 2024-09-16T14:44:09.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 10 monclient: _renew_subs 2024-09-16T14:44:09.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.474+0000 7f82595f9640 1 -- 172.21.15.189:0/1710554513 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f825411d6d0 con 0x7f8254130e20 2024-09-16T14:44:09.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.478+0000 7f823b7fe640 1 -- 172.21.15.189:0/1710554513 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f824c004f80 con 0x7f8254130e20 2024-09-16T14:44:09.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.478+0000 7f823b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:09.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.478+0000 7f823b7fe640 1 -- 172.21.15.189:0/1710554513 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f824c005290 con 0x7f8254130e20 2024-09-16T14:44:09.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.478+0000 7f823b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:09.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.478+0000 7f823b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:09.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.478+0000 7f823b7fe640 10 monclient: dump: 2024-09-16T14:44:09.497 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:09.497 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:09.498 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.498 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.498 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:09.498 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:09.498 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:09.498 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:09.498 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:09.498 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.478+0000 7f82517fa640 4 set_mon_vals no callback set 2024-09-16T14:44:09.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.478+0000 7f82595f9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:09.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.478+0000 7f823b7fe640 1 -- 172.21.15.189:0/1710554513 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f824c00fd00 con 0x7f8254130e20 2024-09-16T14:44:09.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.478+0000 7f82595f9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:09.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.478+0000 7f82595f9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.478+0000 7f82595f9640 1 -- 172.21.15.189:0/1710554513 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8234005050 con 0x7f8254130e20 2024-09-16T14:44:09.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.478+0000 7f823b7fe640 1 -- 172.21.15.189:0/1710554513 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (unknown 3430718669 0 0) 0x7f824c043ac0 con 0x7f8254130e20 2024-09-16T14:44:09.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.478+0000 7f8252ffd640 10 monclient: get_auth_request con 0x7f8240036190 auth_method 0 2024-09-16T14:44:09.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.482+0000 7f823b7fe640 1 -- 172.21.15.189:0/1710554513 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f824c011160 con 0x7f8254130e20 2024-09-16T14:44:09.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.482+0000 7f823b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:09.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.482+0000 7f823b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:09.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.626+0000 7f82595f9640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-16T14:44:09.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.626+0000 7f82595f9640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-16T14:44:09.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.626+0000 7f82595f9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.626+0000 7f82595f9640 1 -- 172.21.15.189:0/1710554513 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo"} v 0) v1 -- 0x7f8234005300 con 0x7f8254130e20 2024-09-16T14:44:09.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.626+0000 7f823b7fe640 1 -- 172.21.15.189:0/1710554513 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo"}]=-1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. v37) v1 ==== 274+0+0 (unknown 2445739868 0 0) 0x7f824c0131b0 con 0x7f8254130e20 2024-09-16T14:44:09.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.626+0000 7f823b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-16T14:44:09.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.626+0000 7f823b7fe640 10 monclient: _finish_command 2 = mon:1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-16T14:44:09.633 INFO:tasks.workunit.client.0.smithi189.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-16T14:44:09.638 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:44:09.638 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool delete foo foo 2024-09-16T14:44:09.638 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:44:09.638 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete foo foo 2024-09-16T14:44:09.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.750+0000 7fdd8a55f640 10 monclient: get_monmap_and_config 2024-09-16T14:44:09.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.750+0000 7fdd8a55f640 10 monclient: build_initial_monmap 2024-09-16T14:44:09.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.750+0000 7fdd8a55f640 10 monclient: monmap: 2024-09-16T14:44:09.757 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:09.757 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:09.757 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:09.756208+0000 2024-09-16T14:44:09.757 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:09.756208+0000 2024-09-16T14:44:09.757 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:09.757 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:09.757 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:09.757 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:09.758 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:09.758 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.750+0000 7fdd8a55f640 1 Processor -- start 2024-09-16T14:44:09.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.750+0000 7fdd8a55f640 1 -- start start 2024-09-16T14:44:09.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.750+0000 7fdd8a55f640 10 monclient: init 2024-09-16T14:44:09.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.750+0000 7fdd8a55f640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:09.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.750+0000 7fdd8a55f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:09.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.750+0000 7fdd8a55f640 10 monclient(hunting): picked mon.noname-b con 0x7fdd8406c120 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:09.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.750+0000 7fdd8a55f640 10 monclient(hunting): picked mon.noname-a con 0x7fdd841353d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:09.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.750+0000 7fdd8a55f640 10 monclient(hunting): picked mon.noname-c con 0x7fdd840a1040 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:09.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.750+0000 7fdd8a55f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd8406c500 con 0x7fdd841353d0 2024-09-16T14:44:09.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.750+0000 7fdd8a55f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd840a2ef0 con 0x7fdd840a1040 2024-09-16T14:44:09.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.750+0000 7fdd8a55f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd8409e990 con 0x7fdd8406c120 2024-09-16T14:44:09.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.750+0000 7fdd8a55f640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:09.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.750+0000 7fdd8a55f640 10 monclient(hunting): authenticate will time out at 1031.871987s 2024-09-16T14:44:09.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd88afc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdd840a1040 0x7fdd8413d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38852/0 (socket says 172.21.15.189:38852) 2024-09-16T14:44:09.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd88afc640 1 -- 172.21.15.189:0/3788860044 learned_addr learned my addr 172.21.15.189:0/3788860044 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:09.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd837fe640 1 -- 172.21.15.189:0/3788860044 >> v1:172.21.15.26:6789/0 conn(0x7fdd841353d0 legacy=0x7fdd84139b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:09.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 1 -- 172.21.15.189:0/3788860044 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128262631 0 0) 0x7fdd840a2ef0 con 0x7fdd840a1040 2024-09-16T14:44:09.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:09.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:09.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:09.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 10 monclient(hunting): my global_id is 4714 2024-09-16T14:44:09.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 1 -- 172.21.15.189:0/3788860044 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd6c003530 con 0x7fdd840a1040 2024-09-16T14:44:09.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 1 -- 172.21.15.189:0/3788860044 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4068988574 0 0) 0x7fdd6c003530 con 0x7fdd840a1040 2024-09-16T14:44:09.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 1 -- 172.21.15.189:0/3788860044 >> v1:172.21.15.26:6790/0 conn(0x7fdd8406c120 legacy=0x7fdd84134cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:09.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 1 -- 172.21.15.189:0/3788860044 >> v1:172.21.15.26:6789/0 conn(0x7fdd841353d0 legacy=0x7fdd84139b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:09.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:09.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 1 monclient: found mon.noname-c 2024-09-16T14:44:09.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:09.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 1 -- 172.21.15.189:0/3788860044 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8409ec70 con 0x7fdd840a1040 2024-09-16T14:44:09.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 10 monclient: _finish_auth 0 2024-09-16T14:44:09.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:09.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:09.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 1 -- 172.21.15.189:0/3788860044 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdd7c0019a0 con 0x7fdd840a1040 2024-09-16T14:44:09.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:09.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:09.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 10 monclient: dump: 2024-09-16T14:44:09.772 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:09.772 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:09.772 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.772 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.772 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:09.772 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:09.772 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:09.772 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:09.773 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:09.773 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 5 monclient: authenticate success, global_id 4714 2024-09-16T14:44:09.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:09.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 1 -- 172.21.15.189:0/3788860044 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd7c001c50 con 0x7fdd840a1040 2024-09-16T14:44:09.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:09.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 1 -- 172.21.15.189:0/3788860044 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdd7c001fe0 con 0x7fdd840a1040 2024-09-16T14:44:09.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:09.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:09.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd82ffd640 10 monclient: dump: 2024-09-16T14:44:09.773 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:09.773 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:09.774 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 4 set_mon_vals no callback set 2024-09-16T14:44:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 10 monclient: shutdown 2024-09-16T14:44:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 1 -- 172.21.15.189:0/3788860044 >> v1:172.21.15.189:6789/0 conn(0x7fdd840a1040 legacy=0x7fdd8413d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 1 -- 172.21.15.189:0/3788860044 shutdown_connections 2024-09-16T14:44:09.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 1 -- 172.21.15.189:0/3788860044 >> 172.21.15.189:0/3788860044 conn(0x7fdd840a3a40 msgr2=0x7fdd840a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:09.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 1 -- 172.21.15.189:0/3788860044 shutdown_connections 2024-09-16T14:44:09.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 1 -- 172.21.15.189:0/3788860044 wait complete. 2024-09-16T14:44:09.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 10 monclient: build_initial_monmap 2024-09-16T14:44:09.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 10 monclient: monmap: 2024-09-16T14:44:09.775 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:09.775 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:09.775 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:09.775 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:09.775 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:09.775 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:09.775 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:09.775 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:09.776 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:09.776 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 1 Processor -- start 2024-09-16T14:44:09.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 1 -- start start 2024-09-16T14:44:09.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 10 monclient: init 2024-09-16T14:44:09.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:09.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:09.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 10 monclient(hunting): picked mon.noname-a con 0x7fdd8406c120 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:09.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 10 monclient(hunting): picked mon.noname-c con 0x7fdd840a1040 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:09.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 10 monclient(hunting): picked mon.noname-b con 0x7fdd841353d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:09.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd84131180 con 0x7fdd8406c120 2024-09-16T14:44:09.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd8412ff20 con 0x7fdd841353d0 2024-09-16T14:44:09.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd8409e130 con 0x7fdd840a1040 2024-09-16T14:44:09.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:09.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 10 monclient(hunting): authenticate will time out at 1031.874997s 2024-09-16T14:44:09.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd88afc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdd841353d0 0x7fdd841d1890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38868/0 (socket says 172.21.15.189:38868) 2024-09-16T14:44:09.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd88afc640 1 -- 172.21.15.189:0/1111742510 learned_addr learned my addr 172.21.15.189:0/1111742510 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:09.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd83fff640 1 -- 172.21.15.189:0/1111742510 >> v1:172.21.15.26:6789/0 conn(0x7fdd8406c120 legacy=0x7fdd8412f100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:09.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194180681 0 0) 0x7fdd8412ff20 con 0x7fdd841353d0 2024-09-16T14:44:09.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:09.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:09.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:09.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 10 monclient(hunting): my global_id is 4717 2024-09-16T14:44:09.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd70003530 con 0x7fdd841353d0 2024-09-16T14:44:09.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3941139311 0 0) 0x7fdd70003530 con 0x7fdd841353d0 2024-09-16T14:44:09.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd8412ff20 con 0x7fdd841353d0 2024-09-16T14:44:09.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdd7c001960 con 0x7fdd841353d0 2024-09-16T14:44:09.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:09.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:09.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 10 monclient(hunting): dump: 2024-09-16T14:44:09.778 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:09.778 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:09.778 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.778 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.778 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:09.778 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:09.778 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:09.779 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:09.779 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:09.779 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:09.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 498573089 0 0) 0x7fdd8412ff20 con 0x7fdd841353d0 2024-09-16T14:44:09.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 >> v1:172.21.15.26:6790/0 conn(0x7fdd840a1040 legacy=0x7fdd8412f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:09.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 >> v1:172.21.15.26:6789/0 conn(0x7fdd8406c120 legacy=0x7fdd8412f100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:09.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:44:09.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 1 monclient: found mon.b 2024-09-16T14:44:09.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:09.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd84132500 con 0x7fdd841353d0 2024-09-16T14:44:09.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 10 monclient: _finish_auth 0 2024-09-16T14:44:09.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:44:09.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:09.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 5 monclient: authenticate success, global_id 4717 2024-09-16T14:44:09.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 10 monclient: _renew_subs 2024-09-16T14:44:09.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 1 -- 172.21.15.189:0/1111742510 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd841327e0 con 0x7fdd841353d0 2024-09-16T14:44:09.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 10 monclient: _renew_subs 2024-09-16T14:44:09.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd8a55f640 1 -- 172.21.15.189:0/1111742510 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd841316c0 con 0x7fdd841353d0 2024-09-16T14:44:09.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd88afc640 1 -- 172.21.15.189:0/1111742510 >> v1:172.21.15.189:6789/0 conn(0x7fdd841353d0 legacy=0x7fdd841d1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T14:44:09.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd88afc640 1 -- 172.21.15.189:0/1111742510 >> v1:172.21.15.189:6789/0 conn(0x7fdd841353d0 legacy=0x7fdd841d1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:44:09.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd88afc640 1 --1- 172.21.15.189:0/1111742510 >> v1:172.21.15.189:6789/0 conn(0x7fdd841353d0 0x7fdd841d1890 :-1 s=OPENED pgs=251 cs=1 l=1).handle_message read tag failed 2024-09-16T14:44:09.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd88afc640 1 --1- 172.21.15.189:0/1111742510 >> v1:172.21.15.189:6789/0 conn(0x7fdd841353d0 0x7fdd841d1890 :-1 s=OPENED pgs=251 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:44:09.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 10 monclient: ms_handle_reset current mon v1:172.21.15.189:6789/0 2024-09-16T14:44:09.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:09.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 >> v1:172.21.15.189:6789/0 conn(0x7fdd841353d0 legacy=0x7fdd841d1890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:09.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.754+0000 7fdd80ff9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:44:09.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 10 monclient(hunting): picked mon.b con 0x7fdd700047d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:09.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 10 monclient(hunting): picked mon.c con 0x7fdd840a1040 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:09.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 10 monclient(hunting): picked mon.a con 0x7fdd8406c120 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:09.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdd8412ff20 con 0x7fdd8406c120 2024-09-16T14:44:09.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdd84131180 con 0x7fdd700047d0 2024-09-16T14:44:09.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdd70003530 con 0x7fdd840a1040 2024-09-16T14:44:09.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:09.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd88afc640 1 -- 172.21.15.189:0/1111742510 >> v1:172.21.15.26:6789/0 conn(0x7fdd8406c120 legacy=0x7fdd7000e2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:09.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376296847 0 0) 0x7fdd84131180 con 0x7fdd700047d0 2024-09-16T14:44:09.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:09.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:09.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:44:09.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 --> v1:172.21.15.189:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fdd70010e80 con 0x7fdd700047d0 2024-09-16T14:44:09.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 4125239439 0 0) 0x7fdd70010e80 con 0x7fdd700047d0 2024-09-16T14:44:09.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 >> v1:172.21.15.26:6790/0 conn(0x7fdd840a1040 legacy=0x7fdd7000a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:09.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 >> v1:172.21.15.26:6789/0 conn(0x7fdd8406c120 legacy=0x7fdd7000e2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:09.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:44:09.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 1 monclient: found mon.b 2024-09-16T14:44:09.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:09.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fdd70013060 con 0x7fdd700047d0 2024-09-16T14:44:09.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 10 monclient: _finish_auth 0 2024-09-16T14:44:09.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:44:09.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:09.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdd74001720 con 0x7fdd700047d0 2024-09-16T14:44:09.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:09.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:09.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 10 monclient: dump: 2024-09-16T14:44:09.783 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:09.783 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:09.783 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.784 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:09.784 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:09.784 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:09.784 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:09.784 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:09.784 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:09.784 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:09.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd740019d0 con 0x7fdd700047d0 2024-09-16T14:44:09.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:09.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd827fc640 4 set_mon_vals no callback set 2024-09-16T14:44:09.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd8a55f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:09.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd8a55f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:09.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd8a55f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd8a55f640 1 -- 172.21.15.189:0/1111742510 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd64005050 con 0x7fdd700047d0 2024-09-16T14:44:09.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fdd74001bc0 con 0x7fdd700047d0 2024-09-16T14:44:09.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 <== mon.1 v1:172.21.15.189:6789/0 6 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (unknown 3430718669 0 0) 0x7fdd74042c80 con 0x7fdd700047d0 2024-09-16T14:44:09.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.758+0000 7fdd837fe640 10 monclient: get_auth_request con 0x7fdd841353d0 auth_method 0 2024-09-16T14:44:09.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.762+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fdd74013d90 con 0x7fdd700047d0 2024-09-16T14:44:09.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.762+0000 7fdd80ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:09.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.762+0000 7fdd80ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:09.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.910+0000 7fdd8a55f640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-16T14:44:09.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.910+0000 7fdd8a55f640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-16T14:44:09.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.910+0000 7fdd8a55f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:09.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.910+0000 7fdd8a55f640 1 -- 172.21.15.189:0/1111742510 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"} v 0) v1 -- 0x7fdd64005880 con 0x7fdd700047d0 2024-09-16T14:44:09.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.910+0000 7fdd80ff9640 1 -- 172.21.15.189:0/1111742510 <== mon.1 v1:172.21.15.189:6789/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}]=-1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. v37) v1 ==== 290+0+0 (unknown 735522076 0 0) 0x7fdd74012430 con 0x7fdd700047d0 2024-09-16T14:44:09.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.910+0000 7fdd80ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-16T14:44:09.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:09.910+0000 7fdd80ff9640 10 monclient: _finish_command 2 = mon:1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-16T14:44:09.917 INFO:tasks.workunit.client.0.smithi189.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-16T14:44:09.921 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:44:09.921 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool delete foo foo --force 2024-09-16T14:44:09.921 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:44:09.921 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete foo foo --force 2024-09-16T14:44:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 10 monclient: get_monmap_and_config 2024-09-16T14:44:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 10 monclient: build_initial_monmap 2024-09-16T14:44:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 10 monclient: monmap: 2024-09-16T14:44:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:10.038820+0000 2024-09-16T14:44:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:10.038820+0000 2024-09-16T14:44:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:10.039 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 1 Processor -- start 2024-09-16T14:44:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 1 -- start start 2024-09-16T14:44:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 10 monclient: init 2024-09-16T14:44:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 10 monclient(hunting): picked mon.noname-b con 0x7f467c0a1b90 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 10 monclient(hunting): picked mon.noname-a con 0x7f467c0a2680 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 10 monclient(hunting): picked mon.noname-c con 0x7f467c099b90 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c099770 con 0x7f467c0a2680 2024-09-16T14:44:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c0a2d60 con 0x7f467c099b90 2024-09-16T14:44:10.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c098e60 con 0x7f467c0a1b90 2024-09-16T14:44:10.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:10.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 10 monclient(hunting): authenticate will time out at 1032.154248s 2024-09-16T14:44:10.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f467c099b90 0x7f467c13d5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38890/0 (socket says 172.21.15.189:38890) 2024-09-16T14:44:10.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467affd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f467c0a2680 legacy=0x7f467c139e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:10.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467bfff640 1 -- 172.21.15.189:0/3551686971 learned_addr learned my addr 172.21.15.189:0/3551686971 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:10.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 1 -- 172.21.15.189:0/3551686971 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111444841 0 0) 0x7f467c0a2d60 con 0x7f467c099b90 2024-09-16T14:44:10.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:10.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:10.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:10.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 10 monclient(hunting): my global_id is 4720 2024-09-16T14:44:10.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 1 -- 172.21.15.189:0/3551686971 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4664003530 con 0x7f467c099b90 2024-09-16T14:44:10.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 1 -- 172.21.15.189:0/3551686971 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1770549542 0 0) 0x7f4664003530 con 0x7f467c099b90 2024-09-16T14:44:10.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 1 -- 172.21.15.189:0/3551686971 >> v1:172.21.15.26:6790/0 conn(0x7f467c0a1b90 legacy=0x7f467c0a1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 1 -- 172.21.15.189:0/3551686971 >> v1:172.21.15.26:6789/0 conn(0x7f467c0a2680 legacy=0x7f467c139e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:10.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:44:10.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 1 monclient: found mon.noname-c 2024-09-16T14:44:10.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:10.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 1 -- 172.21.15.189:0/3551686971 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c099140 con 0x7f467c099b90 2024-09-16T14:44:10.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:44:10.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:44:10.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:10.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 5 monclient: authenticate success, global_id 4720 2024-09-16T14:44:10.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:10.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 1 -- 172.21.15.189:0/3551686971 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f46740019a0 con 0x7f467c099b90 2024-09-16T14:44:10.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:10.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 10 monclient: dump: 2024-09-16T14:44:10.049 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:10.049 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.049 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.050 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.050 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.050 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.050 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.050 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.050 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.050 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:10.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 1 -- 172.21.15.189:0/3551686971 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4674001cc0 con 0x7f467c099b90 2024-09-16T14:44:10.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:10.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 1 -- 172.21.15.189:0/3551686971 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4674002050 con 0x7f467c099b90 2024-09-16T14:44:10.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:10.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f467a7fc640 10 monclient: dump: 2024-09-16T14:44:10.051 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:10.051 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.051 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.051 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.051 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.051 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.051 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.052 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.052 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.052 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:10.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 4 set_mon_vals no callback set 2024-09-16T14:44:10.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 10 monclient: shutdown 2024-09-16T14:44:10.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:10.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 1 -- 172.21.15.189:0/3551686971 >> v1:172.21.15.189:6789/0 conn(0x7f467c099b90 legacy=0x7f467c13d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 1 -- 172.21.15.189:0/3551686971 shutdown_connections 2024-09-16T14:44:10.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 1 -- 172.21.15.189:0/3551686971 >> 172.21.15.189:0/3551686971 conn(0x7f467c0a7a80 msgr2=0x7f467c0a3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:10.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 1 -- 172.21.15.189:0/3551686971 shutdown_connections 2024-09-16T14:44:10.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.034+0000 7f4681b22640 1 -- 172.21.15.189:0/3551686971 wait complete. 2024-09-16T14:44:10.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 monclient: build_initial_monmap 2024-09-16T14:44:10.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 monclient: monmap: 2024-09-16T14:44:10.053 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:10.053 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:10.053 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:10.053 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:10.053 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:10.053 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:10.053 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:10.053 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:10.053 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:10.053 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 1 Processor -- start 2024-09-16T14:44:10.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 1 -- start start 2024-09-16T14:44:10.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 monclient: init 2024-09-16T14:44:10.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:10.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:10.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 monclient(hunting): picked mon.noname-c con 0x7f467c099b90 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:10.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 monclient(hunting): picked mon.noname-a con 0x7f467c0a1b90 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:10.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 monclient(hunting): picked mon.noname-b con 0x7f467c0a2680 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:10.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c09e870 con 0x7f467c0a1b90 2024-09-16T14:44:10.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c09e380 con 0x7f467c0a2680 2024-09-16T14:44:10.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c125740 con 0x7f467c099b90 2024-09-16T14:44:10.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:10.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 monclient(hunting): authenticate will time out at 1032.157055s 2024-09-16T14:44:10.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f467bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f467c0a2680 0x7f467c1d1330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38900/0 (socket says 172.21.15.189:38900) 2024-09-16T14:44:10.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f467bfff640 1 -- 172.21.15.189:0/1603521948 learned_addr learned my addr 172.21.15.189:0/1603521948 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:10.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f467affd640 1 -- 172.21.15.189:0/1603521948 >> v1:172.21.15.26:6789/0 conn(0x7f467c0a1b90 legacy=0x7f467c134d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:10.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 1 -- 172.21.15.189:0/1603521948 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279437332 0 0) 0x7f467c09e380 con 0x7f467c0a2680 2024-09-16T14:44:10.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:10.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:10.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:10.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient(hunting): my global_id is 4723 2024-09-16T14:44:10.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 1 -- 172.21.15.189:0/1603521948 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4668003530 con 0x7f467c0a2680 2024-09-16T14:44:10.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 1 -- 172.21.15.189:0/1603521948 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1534314402 0 0) 0x7f4668003530 con 0x7f467c0a2680 2024-09-16T14:44:10.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 1 -- 172.21.15.189:0/1603521948 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f467c09e380 con 0x7f467c0a2680 2024-09-16T14:44:10.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 1 -- 172.21.15.189:0/1603521948 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f46740017c0 con 0x7f467c0a2680 2024-09-16T14:44:10.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:10.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient(hunting): dump: 2024-09-16T14:44:10.056 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:10.056 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.056 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.056 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.056 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.056 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.056 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.056 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.056 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.056 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:10.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 1 -- 172.21.15.189:0/1603521948 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286402247 0 0) 0x7f467c125740 con 0x7f467c099b90 2024-09-16T14:44:10.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:10.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:10.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:10.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient(hunting): my global_id is 4691 2024-09-16T14:44:10.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 1 -- 172.21.15.189:0/1603521948 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4668003530 con 0x7f467c099b90 2024-09-16T14:44:10.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 1 -- 172.21.15.189:0/1603521948 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4160609871 0 0) 0x7f467c09e380 con 0x7f467c0a2680 2024-09-16T14:44:10.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 1 -- 172.21.15.189:0/1603521948 >> v1:172.21.15.26:6790/0 conn(0x7f467c099b90 legacy=0x7f467c1c8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 1 -- 172.21.15.189:0/1603521948 >> v1:172.21.15.26:6789/0 conn(0x7f467c0a1b90 legacy=0x7f467c134d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:10.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient: _finish_hunting 0 2024-09-16T14:44:10.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 1 monclient: found mon.b 2024-09-16T14:44:10.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:10.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 1 -- 172.21.15.189:0/1603521948 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c1238f0 con 0x7f467c0a2680 2024-09-16T14:44:10.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient: _finish_auth 0 2024-09-16T14:44:10.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient: _check_auth_tickets 2024-09-16T14:44:10.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:10.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 5 monclient: authenticate success, global_id 4723 2024-09-16T14:44:10.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 monclient: _renew_subs 2024-09-16T14:44:10.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 1 -- 172.21.15.189:0/1603521948 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467c09ede0 con 0x7f467c0a2680 2024-09-16T14:44:10.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 monclient: _renew_subs 2024-09-16T14:44:10.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 1 -- 172.21.15.189:0/1603521948 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f467c135460 con 0x7f467c0a2680 2024-09-16T14:44:10.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 1 -- 172.21.15.189:0/1603521948 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4674004fd0 con 0x7f467c0a2680 2024-09-16T14:44:10.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:10.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 1 -- 172.21.15.189:0/1603521948 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f46740052e0 con 0x7f467c0a2680 2024-09-16T14:44:10.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:10.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 10 monclient: dump: 2024-09-16T14:44:10.059 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:10.059 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.059 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.059 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.059 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.059 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.059 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.059 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.059 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.059 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4679ffb640 4 set_mon_vals no callback set 2024-09-16T14:44:10.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 1 -- 172.21.15.189:0/1603521948 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f467400fd50 con 0x7f467c0a2680 2024-09-16T14:44:10.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4663fff640 1 -- 172.21.15.189:0/1603521948 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (unknown 3430718669 0 0) 0x7f4674043a40 con 0x7f467c0a2680 2024-09-16T14:44:10.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:10.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:10.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f4681b22640 1 -- 172.21.15.189:0/1603521948 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f465c005050 con 0x7f467c0a2680 2024-09-16T14:44:10.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.038+0000 7f467b7fe640 10 monclient: get_auth_request con 0x7f46680363c0 auth_method 0 2024-09-16T14:44:10.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.042+0000 7f4663fff640 1 -- 172.21.15.189:0/1603521948 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4674014c30 con 0x7f467c0a2680 2024-09-16T14:44:10.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.042+0000 7f4663fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:10.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.042+0000 7f4663fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:10.210 INFO:tasks.workunit.client.0.smithi189.stderr:--force not valid: --force not one of 'true', 'false' 2024-09-16T14:44:10.211 INFO:tasks.workunit.client.0.smithi189.stderr:Invalid command: unused arguments: ['--force'] 2024-09-16T14:44:10.211 INFO:tasks.workunit.client.0.smithi189.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-16T14:44:10.211 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: invalid command 2024-09-16T14:44:10.213 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:44:10.213 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool delete foo fooo --yes-i-really-mean-it 2024-09-16T14:44:10.214 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:44:10.214 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete foo fooo --yes-i-really-mean-it 2024-09-16T14:44:10.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 10 monclient: get_monmap_and_config 2024-09-16T14:44:10.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 10 monclient: build_initial_monmap 2024-09-16T14:44:10.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 10 monclient: monmap: 2024-09-16T14:44:10.334 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:10.334 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.334 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:10.334501+0000 2024-09-16T14:44:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:10.334501+0000 2024-09-16T14:44:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:10.335 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 1 Processor -- start 2024-09-16T14:44:10.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 1 -- start start 2024-09-16T14:44:10.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 10 monclient: init 2024-09-16T14:44:10.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:10.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:10.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb54131630 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:10.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb5409edd0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:10.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 10 monclient(hunting): picked mon.noname-b con 0x7fcb5409e950 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:10.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5409b920 con 0x7fcb54131630 2024-09-16T14:44:10.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5409bc00 con 0x7fcb5409edd0 2024-09-16T14:44:10.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5409cba0 con 0x7fcb5409e950 2024-09-16T14:44:10.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:10.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 10 monclient(hunting): authenticate will time out at 1032.450671s 2024-09-16T14:44:10.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb537fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fcb54131630 legacy=0x7fcb54131a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:10.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb52ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcb5409edd0 0x7fcb54136b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38908/0 (socket says 172.21.15.189:38908) 2024-09-16T14:44:10.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb52ffd640 1 -- 172.21.15.189:0/1706959869 learned_addr learned my addr 172.21.15.189:0/1706959869 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:10.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 1 -- 172.21.15.189:0/1706959869 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971102357 0 0) 0x7fcb5409bc00 con 0x7fcb5409edd0 2024-09-16T14:44:10.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:10.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:10.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:10.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 10 monclient(hunting): my global_id is 4726 2024-09-16T14:44:10.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 1 -- 172.21.15.189:0/1706959869 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb3c003530 con 0x7fcb5409edd0 2024-09-16T14:44:10.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 1 -- 172.21.15.189:0/1706959869 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3255808886 0 0) 0x7fcb3c003530 con 0x7fcb5409edd0 2024-09-16T14:44:10.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 1 -- 172.21.15.189:0/1706959869 >> v1:172.21.15.26:6790/0 conn(0x7fcb5409e950 legacy=0x7fcb5413a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 1 -- 172.21.15.189:0/1706959869 >> v1:172.21.15.26:6789/0 conn(0x7fcb54131630 legacy=0x7fcb54131a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 10 monclient: _finish_hunting 0 2024-09-16T14:44:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 1 monclient: found mon.noname-c 2024-09-16T14:44:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 1 -- 172.21.15.189:0/1706959869 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5409b420 con 0x7fcb5409edd0 2024-09-16T14:44:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 10 monclient: _finish_auth 0 2024-09-16T14:44:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 10 monclient: _check_auth_tickets 2024-09-16T14:44:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 1 -- 172.21.15.189:0/1706959869 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcb48001620 con 0x7fcb5409edd0 2024-09-16T14:44:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 10 monclient: dump: 2024-09-16T14:44:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.344 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 5 monclient: authenticate success, global_id 4726 2024-09-16T14:44:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 1 -- 172.21.15.189:0/1706959869 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb48001940 con 0x7fcb5409edd0 2024-09-16T14:44:10.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:10.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 1 -- 172.21.15.189:0/1706959869 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcb48001cd0 con 0x7fcb5409edd0 2024-09-16T14:44:10.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:10.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb527fc640 10 monclient: dump: 2024-09-16T14:44:10.345 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:10.345 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.345 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.345 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.345 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.346 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.346 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.346 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.346 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.346 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:10.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 4 set_mon_vals no callback set 2024-09-16T14:44:10.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 10 monclient: shutdown 2024-09-16T14:44:10.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 1 -- 172.21.15.189:0/1706959869 >> v1:172.21.15.189:6789/0 conn(0x7fcb5409edd0 legacy=0x7fcb54136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 1 -- 172.21.15.189:0/1706959869 shutdown_connections 2024-09-16T14:44:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.330+0000 7fcb59f4d640 1 -- 172.21.15.189:0/1706959869 >> 172.21.15.189:0/1706959869 conn(0x7fcb540a3380 msgr2=0x7fcb5409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 1 -- 172.21.15.189:0/1706959869 shutdown_connections 2024-09-16T14:44:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 1 -- 172.21.15.189:0/1706959869 wait complete. 2024-09-16T14:44:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 monclient: build_initial_monmap 2024-09-16T14:44:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 monclient: monmap: 2024-09-16T14:44:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:10.348 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 1 Processor -- start 2024-09-16T14:44:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 1 -- start start 2024-09-16T14:44:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 monclient: init 2024-09-16T14:44:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 monclient(hunting): picked mon.noname-b con 0x7fcb5409e950 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb5409edd0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb54131630 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5409a8f0 con 0x7fcb5409edd0 2024-09-16T14:44:10.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5406fcc0 con 0x7fcb5409e950 2024-09-16T14:44:10.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb54070f00 con 0x7fcb54131630 2024-09-16T14:44:10.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:10.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 monclient(hunting): authenticate will time out at 1032.453449s 2024-09-16T14:44:10.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb537fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcb5409e950 0x7fcb5412fc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38912/0 (socket says 172.21.15.189:38912) 2024-09-16T14:44:10.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb537fe640 1 -- 172.21.15.189:0/109946036 learned_addr learned my addr 172.21.15.189:0/109946036 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:10.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb52ffd640 1 -- 172.21.15.189:0/109946036 >> v1:172.21.15.26:6789/0 conn(0x7fcb5409edd0 legacy=0x7fcb541ce4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:10.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 1 -- 172.21.15.189:0/109946036 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235573030 0 0) 0x7fcb5406fcc0 con 0x7fcb5409e950 2024-09-16T14:44:10.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:10.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:10.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:10.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 10 monclient(hunting): my global_id is 4729 2024-09-16T14:44:10.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 1 -- 172.21.15.189:0/109946036 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb40003530 con 0x7fcb5409e950 2024-09-16T14:44:10.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 1 -- 172.21.15.189:0/109946036 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3158028527 0 0) 0x7fcb40003530 con 0x7fcb5409e950 2024-09-16T14:44:10.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 1 -- 172.21.15.189:0/109946036 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb5406fcc0 con 0x7fcb5409e950 2024-09-16T14:44:10.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 1 -- 172.21.15.189:0/109946036 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcb44004110 con 0x7fcb5409e950 2024-09-16T14:44:10.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:10.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 10 monclient(hunting): dump: 2024-09-16T14:44:10.350 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:10.351 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.351 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.351 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.351 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.351 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.351 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.351 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.351 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.351 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:10.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 1 -- 172.21.15.189:0/109946036 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1623696876 0 0) 0x7fcb5406fcc0 con 0x7fcb5409e950 2024-09-16T14:44:10.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 1 -- 172.21.15.189:0/109946036 >> v1:172.21.15.26:6790/0 conn(0x7fcb54131630 legacy=0x7fcb541d1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 1 -- 172.21.15.189:0/109946036 >> v1:172.21.15.26:6789/0 conn(0x7fcb5409edd0 legacy=0x7fcb541ce4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:10.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:44:10.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 1 monclient: found mon.b 2024-09-16T14:44:10.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:10.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 1 -- 172.21.15.189:0/109946036 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb540711e0 con 0x7fcb5409e950 2024-09-16T14:44:10.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 10 monclient: _finish_auth 0 2024-09-16T14:44:10.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:44:10.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:10.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 5 monclient: authenticate success, global_id 4729 2024-09-16T14:44:10.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 monclient: _renew_subs 2024-09-16T14:44:10.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 1 -- 172.21.15.189:0/109946036 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5406fa40 con 0x7fcb5409e950 2024-09-16T14:44:10.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 monclient: _renew_subs 2024-09-16T14:44:10.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 1 -- 172.21.15.189:0/109946036 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb54130340 con 0x7fcb5409e950 2024-09-16T14:44:10.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 1 -- 172.21.15.189:0/109946036 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb44004430 con 0x7fcb5409e950 2024-09-16T14:44:10.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:10.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 1 -- 172.21.15.189:0/109946036 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcb440047c0 con 0x7fcb5409e950 2024-09-16T14:44:10.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:10.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 10 monclient: dump: 2024-09-16T14:44:10.353 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:10.354 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.354 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.354 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.354 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.354 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.354 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.354 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.354 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.354 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb51ffb640 4 set_mon_vals no callback set 2024-09-16T14:44:10.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:10.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:10.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb59f4d640 1 -- 172.21.15.189:0/109946036 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb34005050 con 0x7fcb5409e950 2024-09-16T14:44:10.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 1 -- 172.21.15.189:0/109946036 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fcb44004920 con 0x7fcb5409e950 2024-09-16T14:44:10.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb3bfff640 1 -- 172.21.15.189:0/109946036 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (unknown 3430718669 0 0) 0x7fcb44043570 con 0x7fcb5409e950 2024-09-16T14:44:10.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.334+0000 7fcb52ffd640 10 monclient: get_auth_request con 0x7fcb400360d0 auth_method 0 2024-09-16T14:44:10.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.338+0000 7fcb3bfff640 1 -- 172.21.15.189:0/109946036 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fcb440050f0 con 0x7fcb5409e950 2024-09-16T14:44:10.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.338+0000 7fcb3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:10.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.338+0000 7fcb3bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:10.492 INFO:tasks.workunit.client.0.smithi189.stderr:--yes-i-really-mean-it not valid: --yes-i-really-mean-it not one of 'true', 'false' 2024-09-16T14:44:10.492 INFO:tasks.workunit.client.0.smithi189.stderr:Invalid command: unused arguments: ['--yes-i-really-mean-it'] 2024-09-16T14:44:10.492 INFO:tasks.workunit.client.0.smithi189.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-16T14:44:10.492 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: invalid command 2024-09-16T14:44:10.494 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:44:10.495 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool delete foo --yes-i-really-mean-it foo 2024-09-16T14:44:10.495 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:44:10.495 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete foo --yes-i-really-mean-it foo 2024-09-16T14:44:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.594+0000 7feb3321b640 10 monclient: get_monmap_and_config 2024-09-16T14:44:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.594+0000 7feb3321b640 10 monclient: build_initial_monmap 2024-09-16T14:44:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.594+0000 7feb3321b640 10 monclient: monmap: 2024-09-16T14:44:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:10.601500+0000 2024-09-16T14:44:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:10.601500+0000 2024-09-16T14:44:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:10.602 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 1 Processor -- start 2024-09-16T14:44:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 1 -- start start 2024-09-16T14:44:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient: init 2024-09-16T14:44:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient(hunting): picked mon.noname-a con 0x7feb2c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient(hunting): picked mon.noname-b con 0x7feb2c09c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient(hunting): picked mon.noname-c con 0x7feb2c09e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c099e50 con 0x7feb2c099a70 2024-09-16T14:44:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c09dbd0 con 0x7feb2c09e6d0 2024-09-16T14:44:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c09ee20 con 0x7feb2c09c550 2024-09-16T14:44:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient(hunting): authenticate will time out at 1032.717072s 2024-09-16T14:44:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb317b8640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7feb2c09e6d0 0x7feb2c13b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38914/0 (socket says 172.21.15.189:38914) 2024-09-16T14:44:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb317b8640 1 -- 172.21.15.189:0/1735545724 learned_addr learned my addr 172.21.15.189:0/1735545724 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb30fb7640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7feb2c099a70 legacy=0x7feb2c09dfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 1 -- 172.21.15.189:0/1735545724 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609632368 0 0) 0x7feb2c09dbd0 con 0x7feb2c09e6d0 2024-09-16T14:44:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 10 monclient(hunting): my global_id is 4732 2024-09-16T14:44:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 1 -- 172.21.15.189:0/1735545724 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb14003530 con 0x7feb2c09e6d0 2024-09-16T14:44:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 1 -- 172.21.15.189:0/1735545724 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2480264753 0 0) 0x7feb14003530 con 0x7feb2c09e6d0 2024-09-16T14:44:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 1 -- 172.21.15.189:0/1735545724 >> v1:172.21.15.26:6790/0 conn(0x7feb2c09c550 legacy=0x7feb2c1379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 1 -- 172.21.15.189:0/1735545724 >> v1:172.21.15.26:6789/0 conn(0x7feb2c099a70 legacy=0x7feb2c09dfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:44:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 1 monclient: found mon.noname-c 2024-09-16T14:44:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 1 -- 172.21.15.189:0/1735545724 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c09f100 con 0x7feb2c09e6d0 2024-09-16T14:44:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:44:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:44:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 1 -- 172.21.15.189:0/1735545724 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb240019a0 con 0x7feb2c09e6d0 2024-09-16T14:44:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 10 monclient: dump: 2024-09-16T14:44:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.612 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 5 monclient: authenticate success, global_id 4732 2024-09-16T14:44:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 1 -- 172.21.15.189:0/1735545724 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb24001c50 con 0x7feb2c09e6d0 2024-09-16T14:44:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 4 set_mon_vals no callback set 2024-09-16T14:44:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 1 -- 172.21.15.189:0/1735545724 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb24002020 con 0x7feb2c09e6d0 2024-09-16T14:44:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb2b7fe640 10 monclient: dump: 2024-09-16T14:44:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.613 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient: shutdown 2024-09-16T14:44:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 1 -- 172.21.15.189:0/1735545724 >> v1:172.21.15.189:6789/0 conn(0x7feb2c09e6d0 legacy=0x7feb2c13b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 1 -- 172.21.15.189:0/1735545724 shutdown_connections 2024-09-16T14:44:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 1 -- 172.21.15.189:0/1735545724 >> 172.21.15.189:0/1735545724 conn(0x7feb2c0a3380 msgr2=0x7feb2c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 1 -- 172.21.15.189:0/1735545724 shutdown_connections 2024-09-16T14:44:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 1 -- 172.21.15.189:0/1735545724 wait complete. 2024-09-16T14:44:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient: build_initial_monmap 2024-09-16T14:44:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient: monmap: 2024-09-16T14:44:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:10.615 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 1 Processor -- start 2024-09-16T14:44:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 1 -- start start 2024-09-16T14:44:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient: init 2024-09-16T14:44:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient(hunting): picked mon.noname-a con 0x7feb2c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient(hunting): picked mon.noname-b con 0x7feb2c09c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 10 monclient(hunting): picked mon.noname-c con 0x7feb2c09e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c09cf20 con 0x7feb2c099a70 2024-09-16T14:44:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.598+0000 7feb3321b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c06f910 con 0x7feb2c09c550 2024-09-16T14:44:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb3321b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c11a5f0 con 0x7feb2c09e6d0 2024-09-16T14:44:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb3321b640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb3321b640 10 monclient(hunting): authenticate will time out at 1032.720033s 2024-09-16T14:44:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb2bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7feb2c09c550 0x7feb2c13a1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38924/0 (socket says 172.21.15.189:38924) 2024-09-16T14:44:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb2bfff640 1 -- 172.21.15.189:0/1504132447 learned_addr learned my addr 172.21.15.189:0/1504132447 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb30fb7640 1 -- 172.21.15.189:0/1504132447 >> v1:172.21.15.26:6789/0 conn(0x7feb2c099a70 legacy=0x7feb2c139aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:44:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 1 -- 172.21.15.189:0/1504132447 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118046378 0 0) 0x7feb2c06f910 con 0x7feb2c09c550 2024-09-16T14:44:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 10 monclient(hunting): _negotiate 2024-09-16T14:44:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:10.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 10 monclient(hunting): my global_id is 4735 2024-09-16T14:44:10.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 1 -- 172.21.15.189:0/1504132447 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb18003530 con 0x7feb2c09c550 2024-09-16T14:44:10.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 1 -- 172.21.15.189:0/1504132447 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2888746170 0 0) 0x7feb18003530 con 0x7feb2c09c550 2024-09-16T14:44:10.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 1 -- 172.21.15.189:0/1504132447 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb2c06f910 con 0x7feb2c09c550 2024-09-16T14:44:10.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 1 -- 172.21.15.189:0/1504132447 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb20001900 con 0x7feb2c09c550 2024-09-16T14:44:10.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:10.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 10 monclient(hunting): dump: 2024-09-16T14:44:10.618 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:10.618 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.618 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.618 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.618 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.619 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.619 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.619 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.619 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.619 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:10.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 1 -- 172.21.15.189:0/1504132447 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 329537976 0 0) 0x7feb2c06f910 con 0x7feb2c09c550 2024-09-16T14:44:10.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 1 -- 172.21.15.189:0/1504132447 >> v1:172.21.15.26:6790/0 conn(0x7feb2c09e6d0 legacy=0x7feb2c13a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 1 -- 172.21.15.189:0/1504132447 >> v1:172.21.15.26:6789/0 conn(0x7feb2c099a70 legacy=0x7feb2c139aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:10.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 10 monclient: _finish_hunting 0 2024-09-16T14:44:10.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 1 monclient: found mon.b 2024-09-16T14:44:10.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:10.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 1 -- 172.21.15.189:0/1504132447 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c06cd90 con 0x7feb2c09c550 2024-09-16T14:44:10.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 10 monclient: _finish_auth 0 2024-09-16T14:44:10.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:10.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:10.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb3321b640 5 monclient: authenticate success, global_id 4735 2024-09-16T14:44:10.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb3321b640 10 monclient: _renew_subs 2024-09-16T14:44:10.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb3321b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb3321b640 1 -- 172.21.15.189:0/1504132447 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2c06b320 con 0x7feb2c09c550 2024-09-16T14:44:10.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb3321b640 10 monclient: _renew_subs 2024-09-16T14:44:10.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb3321b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb3321b640 1 -- 172.21.15.189:0/1504132447 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2c11a310 con 0x7feb2c09c550 2024-09-16T14:44:10.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 1 -- 172.21.15.189:0/1504132447 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb20001d10 con 0x7feb2c09c550 2024-09-16T14:44:10.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:10.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 1 -- 172.21.15.189:0/1504132447 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb20003a70 con 0x7feb2c09c550 2024-09-16T14:44:10.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:10.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb2affd640 4 set_mon_vals no callback set 2024-09-16T14:44:10.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 10 monclient: dump: 2024-09-16T14:44:10.621 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:10.621 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.621 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.621 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.621 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.621 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.622 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.622 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.622 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.622 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb3321b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:10.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb3321b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:10.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb3321b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb3321b640 1 -- 172.21.15.189:0/1504132447 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0c005050 con 0x7feb2c09c550 2024-09-16T14:44:10.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 1 -- 172.21.15.189:0/1504132447 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7feb20003bd0 con 0x7feb2c09c550 2024-09-16T14:44:10.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb297fa640 1 -- 172.21.15.189:0/1504132447 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (unknown 3430718669 0 0) 0x7feb200436c0 con 0x7feb2c09c550 2024-09-16T14:44:10.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.602+0000 7feb30fb7640 10 monclient: get_auth_request con 0x7feb18036000 auth_method 0 2024-09-16T14:44:10.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.606+0000 7feb297fa640 1 -- 172.21.15.189:0/1504132447 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7feb200148a0 con 0x7feb2c09c550 2024-09-16T14:44:10.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.606+0000 7feb297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:10.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.606+0000 7feb297fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:10.664 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:44:10.664 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T14:44:10.664 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T14:44:10.664 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T14:44:10.667 INFO:tasks.ceph.mon.a:Started 2024-09-16T14:44:10.667 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:44:10.668 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:44:10.768 INFO:tasks.workunit.client.0.smithi189.stderr:foo not valid: foo not one of 'true', 'false' 2024-09-16T14:44:10.768 INFO:tasks.workunit.client.0.smithi189.stderr:Invalid command: unused arguments: ['foo'] 2024-09-16T14:44:10.768 INFO:tasks.workunit.client.0.smithi189.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-16T14:44:10.768 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: invalid command 2024-09-16T14:44:10.771 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:44:10.771 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete foooo foooo --yes-i-really-really-mean-it 2024-09-16T14:44:10.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.782+0000 7f5309f54640 10 monclient: get_monmap_and_config 2024-09-16T14:44:10.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.782+0000 7f5309f54640 10 monclient: build_initial_monmap 2024-09-16T14:44:10.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.782+0000 7f5309f54640 10 monclient: monmap: 2024-09-16T14:44:10.789 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:10.789 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.789 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:44:10.789461+0000 2024-09-16T14:44:10.789 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:44:10.789461+0000 2024-09-16T14:44:10.789 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:10.789 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:10.789 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:10.789 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:10.790 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:10.790 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:10.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.786+0000 7f5309f54640 1 Processor -- start 2024-09-16T14:44:10.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.786+0000 7f5309f54640 1 -- start start 2024-09-16T14:44:10.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.786+0000 7f5309f54640 10 monclient: init 2024-09-16T14:44:10.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.786+0000 7f5309f54640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:10.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.786+0000 7f5309f54640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:10.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.786+0000 7f5309f54640 10 monclient(hunting): picked mon.noname-a con 0x7f53040a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:10.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.786+0000 7f5309f54640 10 monclient(hunting): picked mon.noname-b con 0x7f53040a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:10.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.786+0000 7f5309f54640 10 monclient(hunting): picked mon.noname-c con 0x7f53040a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:10.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.786+0000 7f5309f54640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53040a3250 con 0x7f53040a1d10 2024-09-16T14:44:10.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.786+0000 7f5309f54640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530409f600 con 0x7f53040a2800 2024-09-16T14:44:10.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.786+0000 7f5309f54640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53040a3830 con 0x7f53040a0000 2024-09-16T14:44:10.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.786+0000 7f5309f54640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:10.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.786+0000 7f5309f54640 10 monclient(hunting): authenticate will time out at 1074.909116s 2024-09-16T14:44:10.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.786+0000 7f53037fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f53040a1d10 0x7f53040a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58048/0 (socket says 172.21.15.26:58048) 2024-09-16T14:44:10.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.786+0000 7f53037fe640 1 -- 172.21.15.26:0/2345332218 learned_addr learned my addr 172.21.15.26:0/2345332218 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:10.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.786+0000 7f5302ffd640 1 --1- 172.21.15.26:0/2345332218 >> v1:172.21.15.26:6790/0 conn(0x7f53040a0000 0x7f5304139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59626/0 (socket says 172.21.15.26:59626) 2024-09-16T14:44:10.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.794+0000 7f53027fc640 1 -- 172.21.15.26:0/2345332218 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679190168 0 0) 0x7f53040a3830 con 0x7f53040a0000 2024-09-16T14:44:10.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.794+0000 7f53027fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:10.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.794+0000 7f53027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:10.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.794+0000 7f53027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:10.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.794+0000 7f53027fc640 10 monclient(hunting): my global_id is 4706 2024-09-16T14:44:10.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.794+0000 7f53027fc640 1 -- 172.21.15.26:0/2345332218 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52ec003530 con 0x7f53040a0000 2024-09-16T14:44:10.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 1 -- 172.21.15.26:0/2345332218 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1828074920 0 0) 0x7f52ec003530 con 0x7f53040a0000 2024-09-16T14:44:10.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 1 -- 172.21.15.26:0/2345332218 >> v1:172.21.15.189:6789/0 conn(0x7f53040a2800 legacy=0x7f530413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 1 -- 172.21.15.26:0/2345332218 >> v1:172.21.15.26:6789/0 conn(0x7f53040a1d10 legacy=0x7f53040a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 10 monclient: _finish_hunting 0 2024-09-16T14:44:10.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 1 monclient: found mon.noname-b 2024-09-16T14:44:10.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:10.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:10.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 1 -- 172.21.15.26:0/2345332218 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530409ed60 con 0x7f53040a0000 2024-09-16T14:44:10.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 10 monclient: _finish_auth 0 2024-09-16T14:44:10.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 10 monclient: _check_auth_tickets 2024-09-16T14:44:10.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:10.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 1 -- 172.21.15.26:0/2345332218 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52f8001550 con 0x7f53040a0000 2024-09-16T14:44:10.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:10.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 10 monclient: dump: 2024-09-16T14:44:10.803 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:10.803 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.803 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.803 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.803 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.803 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:10.804 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.804 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.804 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.804 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:10.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 1 -- 172.21.15.26:0/2345332218 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52f8001770 con 0x7f53040a0000 2024-09-16T14:44:10.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:10.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 1 -- 172.21.15.26:0/2345332218 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52f8001b80 con 0x7f53040a0000 2024-09-16T14:44:10.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:10.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.798+0000 7f53027fc640 10 monclient: dump: 2024-09-16T14:44:10.804 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:10.804 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.805 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.805 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.805 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.805 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:10.805 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.805 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.805 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.805 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:10.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 5 monclient: authenticate success, global_id 4706 2024-09-16T14:44:10.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:10.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 4 set_mon_vals no callback set 2024-09-16T14:44:10.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 10 monclient: shutdown 2024-09-16T14:44:10.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:10.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 1 -- 172.21.15.26:0/2345332218 >> v1:172.21.15.26:6790/0 conn(0x7f53040a0000 legacy=0x7f5304139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 1 -- 172.21.15.26:0/2345332218 shutdown_connections 2024-09-16T14:44:10.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 1 -- 172.21.15.26:0/2345332218 >> 172.21.15.26:0/2345332218 conn(0x7f53040a7f30 msgr2=0x7f53040a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:10.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 1 -- 172.21.15.26:0/2345332218 shutdown_connections 2024-09-16T14:44:10.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 1 -- 172.21.15.26:0/2345332218 wait complete. 2024-09-16T14:44:10.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 10 monclient: build_initial_monmap 2024-09-16T14:44:10.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 10 monclient: monmap: 2024-09-16T14:44:10.810 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:10.810 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:10.810 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:44:10.810 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:44:10.810 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:10.811 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:44:10.811 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:10.811 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:10.811 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:10.811 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:10.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 1 Processor -- start 2024-09-16T14:44:10.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 1 -- start start 2024-09-16T14:44:10.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 10 monclient: init 2024-09-16T14:44:10.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:10.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:10.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 10 monclient(hunting): picked mon.noname-c con 0x7f53040a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:10.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 10 monclient(hunting): picked mon.noname-a con 0x7f53040a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:10.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 10 monclient(hunting): picked mon.noname-b con 0x7f53040a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:10.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5304131640 con 0x7f53040a1d10 2024-09-16T14:44:10.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5304133ee0 con 0x7f53040a2800 2024-09-16T14:44:10.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5304131000 con 0x7f53040a0000 2024-09-16T14:44:10.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:10.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 10 monclient(hunting): authenticate will time out at 1074.926415s 2024-09-16T14:44:10.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f53037fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f53040a0000 0x7f53041301e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59638/0 (socket says 172.21.15.26:59638) 2024-09-16T14:44:10.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f53037fe640 1 -- 172.21.15.26:0/1508929335 learned_addr learned my addr 172.21.15.26:0/1508929335 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:10.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 1 -- 172.21.15.26:0/1508929335 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476524151 0 0) 0x7f5304131000 con 0x7f53040a0000 2024-09-16T14:44:10.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:10.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:10.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:10.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 10 monclient(hunting): my global_id is 4709 2024-09-16T14:44:10.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 1 -- 172.21.15.26:0/1508929335 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52f0003530 con 0x7f53040a0000 2024-09-16T14:44:10.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 1 -- 172.21.15.26:0/1508929335 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1255750763 0 0) 0x7f52f0003530 con 0x7f53040a0000 2024-09-16T14:44:10.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 1 -- 172.21.15.26:0/1508929335 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5304131000 con 0x7f53040a0000 2024-09-16T14:44:10.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 1 -- 172.21.15.26:0/1508929335 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52f40018c0 con 0x7f53040a0000 2024-09-16T14:44:10.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:10.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 10 monclient(hunting): dump: 2024-09-16T14:44:10.813 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:10.813 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.813 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.813 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.814 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.814 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:10.814 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.814 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.814 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.814 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:10.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:10.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 1 -- 172.21.15.26:0/1508929335 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3222652960 0 0) 0x7f5304131000 con 0x7f53040a0000 2024-09-16T14:44:10.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 1 -- 172.21.15.26:0/1508929335 >> v1:172.21.15.189:6789/0 conn(0x7f53040a2800 legacy=0x7f53041d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 1 -- 172.21.15.26:0/1508929335 >> v1:172.21.15.26:6789/0 conn(0x7f53040a1d10 legacy=0x7f53041308f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 10 monclient: _finish_hunting 0 2024-09-16T14:44:10.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 1 monclient: found mon.c 2024-09-16T14:44:10.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:10.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:10.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 1 -- 172.21.15.26:0/1508929335 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5304131b80 con 0x7f53040a0000 2024-09-16T14:44:10.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 10 monclient: _finish_auth 0 2024-09-16T14:44:10.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 10 monclient: _check_auth_tickets 2024-09-16T14:44:10.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:10.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 5 monclient: authenticate success, global_id 4709 2024-09-16T14:44:10.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 10 monclient: _renew_subs 2024-09-16T14:44:10.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:10.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 1 -- 172.21.15.26:0/1508929335 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530409e500 con 0x7f53040a0000 2024-09-16T14:44:10.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 10 monclient: _renew_subs 2024-09-16T14:44:10.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:10.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f5309f54640 1 -- 172.21.15.26:0/1508929335 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5304133c00 con 0x7f53040a0000 2024-09-16T14:44:10.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 1 -- 172.21.15.26:0/1508929335 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52f40047f0 con 0x7f53040a0000 2024-09-16T14:44:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 1 -- 172.21.15.26:0/1508929335 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52f4004b80 con 0x7f53040a0000 2024-09-16T14:44:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f53017fa640 4 set_mon_vals no callback set 2024-09-16T14:44:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.802+0000 7f52ebfff640 10 monclient: dump: 2024-09-16T14:44:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.816 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.817 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.806+0000 7f52ebfff640 1 -- 172.21.15.26:0/1508929335 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f52f4004ce0 con 0x7f53040a0000 2024-09-16T14:44:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.806+0000 7f5309f54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.806+0000 7f5309f54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.806+0000 7f5309f54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.806+0000 7f5309f54640 1 -- 172.21.15.26:0/1508929335 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e4005050 con 0x7f53040a0000 2024-09-16T14:44:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.806+0000 7f52ebfff640 1 -- 172.21.15.26:0/1508929335 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (unknown 3430718669 0 0) 0x7f52f40435d0 con 0x7f53040a0000 2024-09-16T14:44:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.806+0000 7f5302ffd640 10 monclient: get_auth_request con 0x7f52f00360f0 auth_method 0 2024-09-16T14:44:10.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.810+0000 7f52ebfff640 1 -- 172.21.15.26:0/1508929335 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f52f40054e0 con 0x7f53040a0000 2024-09-16T14:44:10.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.810+0000 7f52ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:10.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:10.810+0000 7f52ebfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:10.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e54e20640 10 monclient: get_monmap_and_config 2024-09-16T14:44:10.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e54e20640 10 monclient: build_initial_monmap 2024-09-16T14:44:10.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e54e20640 10 monclient: monmap: 2024-09-16T14:44:10.883 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:10.884 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.884 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:10.883762+0000 2024-09-16T14:44:10.884 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:10.883762+0000 2024-09-16T14:44:10.884 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:10.884 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.884 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:10.884 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:10.884 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:10.884 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e54e20640 1 Processor -- start 2024-09-16T14:44:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e54e20640 1 -- start start 2024-09-16T14:44:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e54e20640 10 monclient: init 2024-09-16T14:44:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e54e20640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e54e20640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:44:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e54e20640 10 monclient(hunting): picked mon.noname-b con 0x7f0e50071900 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e54e20640 10 monclient(hunting): picked mon.noname-c con 0x7f0e50071ce0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e54e20640 10 monclient(hunting): picked mon.noname-a con 0x7f0e5013bab0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e54e20640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5006be30 con 0x7f0e5013bab0 2024-09-16T14:44:10.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e54e20640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e500a1310 con 0x7f0e50071ce0 2024-09-16T14:44:10.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e54e20640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5009f380 con 0x7f0e50071900 2024-09-16T14:44:10.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e54e20640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:10.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e54e20640 10 monclient(hunting): authenticate will time out at 1032.999356s 2024-09-16T14:44:10.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e4dd9b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0e50071ce0 0x7f0e5006b720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38928/0 (socket says 172.21.15.189:38928) 2024-09-16T14:44:10.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e4dd9b640 1 -- 172.21.15.189:0/359588787 learned_addr learned my addr 172.21.15.189:0/359588787 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:10.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e4d59a640 1 -- 172.21.15.189:0/359588787 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755335869 0 0) 0x7f0e500a1310 con 0x7f0e50071ce0 2024-09-16T14:44:10.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e4d59a640 10 monclient(hunting): _negotiate 2024-09-16T14:44:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e4d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e4d59a640 10 monclient(hunting): my global_id is 4738 2024-09-16T14:44:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.878+0000 7f0e4d59a640 1 -- 172.21.15.189:0/359588787 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e38003530 con 0x7f0e50071ce0 2024-09-16T14:44:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 1 -- 172.21.15.189:0/359588787 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1649184636 0 0) 0x7f0e38003530 con 0x7f0e50071ce0 2024-09-16T14:44:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 1 -- 172.21.15.189:0/359588787 >> v1:172.21.15.26:6790/0 conn(0x7f0e50071900 legacy=0x7f0e5006b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 1 -- 172.21.15.189:0/359588787 >> v1:172.21.15.26:6789/0 conn(0x7f0e5013bab0 legacy=0x7f0e5013dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:44:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 10 monclient: _finish_hunting 0 2024-09-16T14:44:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 1 monclient: found mon.noname-c 2024-09-16T14:44:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 1 -- 172.21.15.189:0/359588787 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5009cbf0 con 0x7f0e50071ce0 2024-09-16T14:44:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 10 monclient: _finish_auth 0 2024-09-16T14:44:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 10 monclient: _check_auth_tickets 2024-09-16T14:44:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 1 -- 172.21.15.189:0/359588787 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0e44001620 con 0x7f0e50071ce0 2024-09-16T14:44:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 10 monclient: dump: 2024-09-16T14:44:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.895 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 5 monclient: authenticate success, global_id 4738 2024-09-16T14:44:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 1 -- 172.21.15.189:0/359588787 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e44001910 con 0x7f0e50071ce0 2024-09-16T14:44:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 1 -- 172.21.15.189:0/359588787 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0e44001ca0 con 0x7f0e50071ce0 2024-09-16T14:44:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 4 set_mon_vals no callback set 2024-09-16T14:44:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4d59a640 10 monclient: dump: 2024-09-16T14:44:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.897 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 monclient: shutdown 2024-09-16T14:44:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 1 -- 172.21.15.189:0/359588787 >> v1:172.21.15.189:6789/0 conn(0x7f0e50071ce0 legacy=0x7f0e5006b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 1 -- 172.21.15.189:0/359588787 shutdown_connections 2024-09-16T14:44:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 1 -- 172.21.15.189:0/359588787 >> 172.21.15.189:0/359588787 conn(0x7f0e500a19b0 msgr2=0x7f0e500a1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 1 -- 172.21.15.189:0/359588787 shutdown_connections 2024-09-16T14:44:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 1 -- 172.21.15.189:0/359588787 wait complete. 2024-09-16T14:44:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 monclient: build_initial_monmap 2024-09-16T14:44:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 monclient: monmap: 2024-09-16T14:44:10.898 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:10.898 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:10.898 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:10.898 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:10.898 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:10.898 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:10.898 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:10.898 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:10.898 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:10.898 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 1 Processor -- start 2024-09-16T14:44:10.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 1 -- start start 2024-09-16T14:44:10.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 monclient: init 2024-09-16T14:44:10.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:10.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:10.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 monclient(hunting): picked mon.noname-b con 0x7f0e50071900 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:10.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 monclient(hunting): picked mon.noname-a con 0x7f0e50071ce0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:10.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 monclient(hunting): picked mon.noname-c con 0x7f0e5013bab0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:10.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e50138c90 con 0x7f0e50071ce0 2024-09-16T14:44:10.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5009c110 con 0x7f0e50071900 2024-09-16T14:44:10.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5013ad20 con 0x7f0e5013bab0 2024-09-16T14:44:10.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:10.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 monclient(hunting): authenticate will time out at 1033.002382s 2024-09-16T14:44:10.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4e59c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0e50071900 0x7f0e50137e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38944/0 (socket says 172.21.15.189:38944) 2024-09-16T14:44:10.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4e59c640 1 -- 172.21.15.189:0/261167754 learned_addr learned my addr 172.21.15.189:0/261167754 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:10.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191837357 0 0) 0x7f0e5009c110 con 0x7f0e50071900 2024-09-16T14:44:10.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:10.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:10.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:10.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient(hunting): my global_id is 4741 2024-09-16T14:44:10.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e30003530 con 0x7f0e50071900 2024-09-16T14:44:10.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2780288038 0 0) 0x7f0e30003530 con 0x7f0e50071900 2024-09-16T14:44:10.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e5009c110 con 0x7f0e50071900 2024-09-16T14:44:10.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0e48001980 con 0x7f0e50071900 2024-09-16T14:44:10.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:10.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient(hunting): dump: 2024-09-16T14:44:10.901 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:10.901 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.901 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.901 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.901 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.901 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.901 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.901 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.901 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.901 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:10.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878031869 0 0) 0x7f0e5013ad20 con 0x7f0e5013bab0 2024-09-16T14:44:10.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:10.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:10.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:10.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient(hunting): my global_id is 4715 2024-09-16T14:44:10.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e30003530 con 0x7f0e5013bab0 2024-09-16T14:44:10.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3626158597 0 0) 0x7f0e5009c110 con 0x7f0e50071900 2024-09-16T14:44:10.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 >> v1:172.21.15.26:6790/0 conn(0x7f0e5013bab0 legacy=0x7f0e501d1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 >> v1:172.21.15.26:6789/0 conn(0x7f0e50071ce0 legacy=0x7f0e50138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:10.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:10.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 1 monclient: found mon.b 2024-09-16T14:44:10.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:10.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e50139960 con 0x7f0e50071900 2024-09-16T14:44:10.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient: _finish_auth 0 2024-09-16T14:44:10.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:10.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:10.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 5 monclient: authenticate success, global_id 4741 2024-09-16T14:44:10.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 monclient: _renew_subs 2024-09-16T14:44:10.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 1 -- 172.21.15.189:0/261167754 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5013a480 con 0x7f0e50071900 2024-09-16T14:44:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e48001d50 con 0x7f0e50071900 2024-09-16T14:44:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 monclient: _renew_subs 2024-09-16T14:44:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 1 -- 172.21.15.189:0/261167754 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e50139080 con 0x7f0e50071900 2024-09-16T14:44:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0e48003af0 con 0x7f0e50071900 2024-09-16T14:44:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e3effd640 10 monclient: dump: 2024-09-16T14:44:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:10.905 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e4cd99640 4 set_mon_vals no callback set 2024-09-16T14:44:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.882+0000 7f0e54e20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.886+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0e48003cd0 con 0x7f0e50071900 2024-09-16T14:44:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.886+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (unknown 3430718669 0 0) 0x7f0e48043700 con 0x7f0e50071900 2024-09-16T14:44:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.886+0000 7f0e54e20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.886+0000 7f0e54e20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.886+0000 7f0e54e20640 1 -- 172.21.15.189:0/261167754 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e5013a8f0 con 0x7f0e50071900 2024-09-16T14:44:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.886+0000 7f0e4dd9b640 10 monclient: get_auth_request con 0x7f0e30036270 auth_method 0 2024-09-16T14:44:10.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.890+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0e480148e0 con 0x7f0e50071900 2024-09-16T14:44:10.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.890+0000 7f0e3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:10.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:10.890+0000 7f0e3effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:11.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.030+0000 7f5309f54640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:44:11.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.030+0000 7f5309f54640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:44:11.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.030+0000 7f5309f54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:11.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.030+0000 7f5309f54640 1 -- 172.21.15.26:0/1508929335 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f52e4005880 con 0x7f53040a0000 2024-09-16T14:44:11.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.030+0000 7f52ebfff640 1 -- 172.21.15.26:0/1508929335 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1737424735) 0x7f52f4029bc0 con 0x7f53040a0000 2024-09-16T14:44:11.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.030+0000 7f52ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:44:11.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.030+0000 7f52ebfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:11.037 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:44:11.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.034+0000 7f5309f54640 1 -- 172.21.15.26:0/1508929335 >> v1:172.21.15.189:6812/3196161582 conn(0x7f52f00360f0 legacy=0x7f52f00385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.034+0000 7f5309f54640 10 monclient: shutdown 2024-09-16T14:44:11.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.034+0000 7f5309f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:11.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.034+0000 7f5309f54640 1 -- 172.21.15.26:0/1508929335 >> v1:172.21.15.26:6790/0 conn(0x7f53040a0000 legacy=0x7f53041301e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.034+0000 7f5309f54640 1 -- 172.21.15.26:0/1508929335 shutdown_connections 2024-09-16T14:44:11.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.034+0000 7f5309f54640 1 -- 172.21.15.26:0/1508929335 >> 172.21.15.26:0/1508929335 conn(0x7f53040a7f30 msgr2=0x7f530406b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:11.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.034+0000 7f5309f54640 1 -- 172.21.15.26:0/1508929335 shutdown_connections 2024-09-16T14:44:11.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.034+0000 7f5309f54640 1 -- 172.21.15.26:0/1508929335 wait complete. 2024-09-16T14:44:11.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:11.042+0000 7f0e54e20640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:11.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:11.042+0000 7f0e54e20640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:11.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:11.042+0000 7f0e54e20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:11.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:11.042+0000 7f0e54e20640 1 -- 172.21.15.189:0/261167754 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0e5013a0f0 con 0x7f0e50071900 2024-09-16T14:44:11.055 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:44:11.055 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:44:11.055 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:44:11.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.178+0000 7fa85930e640 10 monclient: get_monmap_and_config 2024-09-16T14:44:11.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.178+0000 7fa85930e640 10 monclient: build_initial_monmap 2024-09-16T14:44:11.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.178+0000 7fa85930e640 10 monclient: monmap: 2024-09-16T14:44:11.183 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:11.183 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.183 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:44:11.183273+0000 2024-09-16T14:44:11.183 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:44:11.183273+0000 2024-09-16T14:44:11.183 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:11.184 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.184 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:11.184 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:11.184 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:11.184 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.178+0000 7fa85930e640 1 Processor -- start 2024-09-16T14:44:11.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.178+0000 7fa85930e640 1 -- start start 2024-09-16T14:44:11.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.178+0000 7fa85930e640 10 monclient: init 2024-09-16T14:44:11.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.178+0000 7fa85930e640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:11.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.178+0000 7fa85930e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:11.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient(hunting): picked mon.noname-a con 0x7fa8540a1ca0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:11.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient(hunting): picked mon.noname-b con 0x7fa8540a2790 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:11.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient(hunting): picked mon.noname-c con 0x7fa85409ff50 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:11.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85409f520 con 0x7fa8540a1ca0 2024-09-16T14:44:11.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8540a0360 con 0x7fa85409ff50 2024-09-16T14:44:11.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85409e7a0 con 0x7fa8540a2790 2024-09-16T14:44:11.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:11.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient(hunting): authenticate will time out at 1075.304120s 2024-09-16T14:44:11.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa852ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa8540a1ca0 0x7fa8540a2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58068/0 (socket says 172.21.15.26:58068) 2024-09-16T14:44:11.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa852ffd640 1 -- 172.21.15.26:0/310938657 learned_addr learned my addr 172.21.15.26:0/310938657 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:11.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 1 -- 172.21.15.26:0/310938657 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167771977 0 0) 0x7fa85409f520 con 0x7fa8540a1ca0 2024-09-16T14:44:11.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient(hunting): my global_id is 84099 2024-09-16T14:44:11.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 1 -- 172.21.15.26:0/310938657 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa840003530 con 0x7fa8540a1ca0 2024-09-16T14:44:11.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 1 -- 172.21.15.26:0/310938657 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777662008 0 0) 0x7fa85409e7a0 con 0x7fa8540a2790 2024-09-16T14:44:11.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient(hunting): my global_id is 4718 2024-09-16T14:44:11.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 1 -- 172.21.15.26:0/310938657 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa85409f520 con 0x7fa8540a2790 2024-09-16T14:44:11.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 1 -- 172.21.15.26:0/310938657 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967975177 0 0) 0x7fa8540a0360 con 0x7fa85409ff50 2024-09-16T14:44:11.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient(hunting): my global_id is 4744 2024-09-16T14:44:11.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 1 -- 172.21.15.26:0/310938657 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa85409e7a0 con 0x7fa85409ff50 2024-09-16T14:44:11.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 1 -- 172.21.15.26:0/310938657 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1956442864 0 0) 0x7fa85409f520 con 0x7fa8540a2790 2024-09-16T14:44:11.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 1 -- 172.21.15.26:0/310938657 >> v1:172.21.15.189:6789/0 conn(0x7fa85409ff50 legacy=0x7fa85413d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 1 -- 172.21.15.26:0/310938657 >> v1:172.21.15.26:6789/0 conn(0x7fa8540a1ca0 legacy=0x7fa8540a2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient: _finish_hunting 0 2024-09-16T14:44:11.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 1 monclient: found mon.noname-b 2024-09-16T14:44:11.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:11.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:11.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 1 -- 172.21.15.26:0/310938657 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85409ec80 con 0x7fa8540a2790 2024-09-16T14:44:11.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient: _finish_auth 0 2024-09-16T14:44:11.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient: _check_auth_tickets 2024-09-16T14:44:11.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:11.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 5 monclient: authenticate success, global_id 4718 2024-09-16T14:44:11.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:11.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 1 -- 172.21.15.26:0/310938657 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa83c001600 con 0x7fa8540a2790 2024-09-16T14:44:11.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:11.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient: dump: 2024-09-16T14:44:11.196 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.196 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.196 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.196 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.196 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.196 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.196 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.196 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.196 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.196 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:11.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 1 -- 172.21.15.26:0/310938657 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa83c001920 con 0x7fa8540a2790 2024-09-16T14:44:11.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:11.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 1 -- 172.21.15.26:0/310938657 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa83c001cf0 con 0x7fa8540a2790 2024-09-16T14:44:11.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:11.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 4 set_mon_vals no callback set 2024-09-16T14:44:11.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:11.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa8527fc640 10 monclient: dump: 2024-09-16T14:44:11.197 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.197 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.197 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.197 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.197 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.198 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.198 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.198 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.198 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.198 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient: shutdown 2024-09-16T14:44:11.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:11.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 1 -- 172.21.15.26:0/310938657 >> v1:172.21.15.26:6790/0 conn(0x7fa8540a2790 legacy=0x7fa854139e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 1 -- 172.21.15.26:0/310938657 shutdown_connections 2024-09-16T14:44:11.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 1 -- 172.21.15.26:0/310938657 >> 172.21.15.26:0/310938657 conn(0x7fa8540a3da0 msgr2=0x7fa8540a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:11.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 1 -- 172.21.15.26:0/310938657 shutdown_connections 2024-09-16T14:44:11.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 1 -- 172.21.15.26:0/310938657 wait complete. 2024-09-16T14:44:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient: build_initial_monmap 2024-09-16T14:44:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient: monmap: 2024-09-16T14:44:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:44:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:44:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:44:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:11.199 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 1 Processor -- start 2024-09-16T14:44:11.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 1 -- start start 2024-09-16T14:44:11.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient: init 2024-09-16T14:44:11.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:11.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:44:11.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient(hunting): picked mon.noname-c con 0x7fa85409ff50 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:11.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient(hunting): picked mon.noname-b con 0x7fa8540a1ca0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:11.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient(hunting): picked mon.noname-a con 0x7fa8540a2790 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:11.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8540a5180 con 0x7fa8540a2790 2024-09-16T14:44:11.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85409e490 con 0x7fa8540a1ca0 2024-09-16T14:44:11.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8541341c0 con 0x7fa85409ff50 2024-09-16T14:44:11.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa85930e640 10 monclient(hunting): authenticate will time out at 1075.307307s 2024-09-16T14:44:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa852ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa85409ff50 0x7fa85413d510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59658/0 (socket says 172.21.15.26:59658) 2024-09-16T14:44:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa852ffd640 1 -- 172.21.15.26:0/654827951 learned_addr learned my addr 172.21.15.26:0/654827951 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201753058 0 0) 0x7fa8540a5180 con 0x7fa8540a2790 2024-09-16T14:44:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa84b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa84b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa84b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa84b7fe640 10 monclient(hunting): my global_id is 84102 2024-09-16T14:44:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa834003530 con 0x7fa8540a2790 2024-09-16T14:44:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324009948 0 0) 0x7fa8541341c0 con 0x7fa85409ff50 2024-09-16T14:44:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa84b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa84b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa84b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa84b7fe640 10 monclient(hunting): my global_id is 4721 2024-09-16T14:44:11.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8540a5180 con 0x7fa85409ff50 2024-09-16T14:44:11.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.182+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1800982534 0 0) 0x7fa834003530 con 0x7fa8540a2790 2024-09-16T14:44:11.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8541341c0 con 0x7fa8540a2790 2024-09-16T14:44:11.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa84c001c40 con 0x7fa8540a2790 2024-09-16T14:44:11.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:11.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient(hunting): dump: 2024-09-16T14:44:11.202 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.202 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.202 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.203 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.203 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.203 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.203 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.203 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.203 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.203 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:11.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1428213286 0 0) 0x7fa8540a5180 con 0x7fa85409ff50 2024-09-16T14:44:11.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa834003530 con 0x7fa85409ff50 2024-09-16T14:44:11.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814469156 0 0) 0x7fa85409e490 con 0x7fa8540a1ca0 2024-09-16T14:44:11.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient(hunting): my global_id is 4747 2024-09-16T14:44:11.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8540a5180 con 0x7fa8540a1ca0 2024-09-16T14:44:11.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa844001a00 con 0x7fa85409ff50 2024-09-16T14:44:11.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:11.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient(hunting): dump: 2024-09-16T14:44:11.204 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.204 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.204 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.204 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.205 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.205 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.205 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.205 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.205 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.205 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2331662716 0 0) 0x7fa8541341c0 con 0x7fa8540a2790 2024-09-16T14:44:11.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 >> v1:172.21.15.26:6790/0 conn(0x7fa85409ff50 legacy=0x7fa85413d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 >> v1:172.21.15.189:6789/0 conn(0x7fa8540a1ca0 legacy=0x7fa8541ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:44:11.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 monclient: found mon.a 2024-09-16T14:44:11.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:11.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:11.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa854134960 con 0x7fa8540a2790 2024-09-16T14:44:11.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:44:11.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:44:11.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:11.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa85930e640 5 monclient: authenticate success, global_id 84102 2024-09-16T14:44:11.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa85930e640 10 monclient: _renew_subs 2024-09-16T14:44:11.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa85930e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:11.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa85930e640 1 -- 172.21.15.26:0/654827951 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa854136750 con 0x7fa8540a2790 2024-09-16T14:44:11.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa85930e640 10 monclient: _renew_subs 2024-09-16T14:44:11.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa85930e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:11.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa85930e640 1 -- 172.21.15.26:0/654827951 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa85413dc20 con 0x7fa8540a2790 2024-09-16T14:44:11.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa84c0039a0 con 0x7fa8540a2790 2024-09-16T14:44:11.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:11.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa84c003cb0 con 0x7fa8540a2790 2024-09-16T14:44:11.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:11.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa851ffb640 4 set_mon_vals no callback set 2024-09-16T14:44:11.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient: dump: 2024-09-16T14:44:11.207 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.207 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.207 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.207 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.207 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.208 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.208 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.208 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.208 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.208 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa85930e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:11.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa85930e640 1 -- 172.21.15.26:0/654827951 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa838006590 con 0x7fa838002830 2024-09-16T14:44:11.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa84c00e720 con 0x7fa8540a2790 2024-09-16T14:44:11.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (unknown 3430718669 0 0) 0x7fa84c042550 con 0x7fa8540a2790 2024-09-16T14:44:11.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145538502 0 0) 0x7fa838006590 con 0x7fa838002830 2024-09-16T14:44:11.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient(hunting): my global_id is 84105 2024-09-16T14:44:11.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8541341c0 con 0x7fa838002830 2024-09-16T14:44:11.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84bfff640 10 monclient: get_auth_request con 0x7fa834036420 auth_method 0 2024-09-16T14:44:11.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2614392299 0 0) 0x7fa8541341c0 con 0x7fa838002830 2024-09-16T14:44:11.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa838006590 con 0x7fa838002830 2024-09-16T14:44:11.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa844004bc0 con 0x7fa838002830 2024-09-16T14:44:11.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:11.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient: dump: 2024-09-16T14:44:11.209 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.210 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.210 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.210 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.210 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.210 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.210 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.210 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.210 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.210 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2408652919 0 0) 0x7fa838006590 con 0x7fa838002830 2024-09-16T14:44:11.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa83800a0e0 con 0x7fa838002830 2024-09-16T14:44:11.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa844004f20 con 0x7fa838002830 2024-09-16T14:44:11.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:11.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:11.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 >> v1:172.21.15.26:6789/0 conn(0x7fa838002830 legacy=0x7fa838004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.186+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 >> v1:172.21.15.26:6789/0 conn(0x7fa838002830 legacy=0x7fa838004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:11.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa85930e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:44:11.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa85930e640 1 -- 172.21.15.26:0/654827951 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa838009d80 con 0x7fa8380090a0 2024-09-16T14:44:11.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661127679 0 0) 0x7fa838009d80 con 0x7fa8380090a0 2024-09-16T14:44:11.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 10 monclient(hunting): my global_id is 84108 2024-09-16T14:44:11.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa838006590 con 0x7fa8380090a0 2024-09-16T14:44:11.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2622178472 0 0) 0x7fa838006590 con 0x7fa8380090a0 2024-09-16T14:44:11.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa838009d80 con 0x7fa8380090a0 2024-09-16T14:44:11.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa8440055c0 con 0x7fa8380090a0 2024-09-16T14:44:11.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:11.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 10 monclient: dump: 2024-09-16T14:44:11.213 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.213 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.213 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.213 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.214 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.214 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.214 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.214 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.214 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.214 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4187425512 0 0) 0x7fa838009d80 con 0x7fa8380090a0 2024-09-16T14:44:11.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa83800c290 con 0x7fa8380090a0 2024-09-16T14:44:11.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 3670355366) 0x7fa844006970 con 0x7fa8380090a0 2024-09-16T14:44:11.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:44:11.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:11.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 >> v1:172.21.15.26:6789/0 conn(0x7fa8380090a0 legacy=0x7fa838009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa84b7fe640 1 -- 172.21.15.26:0/654827951 >> v1:172.21.15.26:6789/0 conn(0x7fa8380090a0 legacy=0x7fa838009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:11.215 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:44:11.215 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:44:11.215 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:44:11.215 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:44:11.215 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 22, 2024-09-16T14:44:11.215 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:44:11.215 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:44:11.216 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:44:11.216 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:44:11.216 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:11.216 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:44:11.216 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:44:11.216 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:44:11.216 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:44:11.216 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:11.216 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:11.216 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:11.216 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:11.217 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:11.217 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:11.217 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:11.217 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:11.217 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:11.217 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:11.217 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:11.217 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:44:11.217 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:44:11.217 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:11.217 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:11.217 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:11.218 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:11.218 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:11.218 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:11.218 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:11.218 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:11.218 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:11.218 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:11.218 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.218 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:44:11.218 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:44:11.218 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:44:11.219 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:44:11.219 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:44:11.219 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:44:11.219 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:44:11.219 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:44:11.219 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:44:11.219 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:44:11.219 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:44:11.219 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:44:11.219 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:44:11.219 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:44:11.219 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:44:11.220 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:44:11.220 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:44:11.220 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:11.220 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:11.220 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:11.220 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:11.220 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:11.220 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:11.220 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:11.220 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:11.220 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:11.220 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:11.221 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:11.221 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:44:11.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.221 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:44:11.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.221 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:44:11.221 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:44:11.221 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:11.221 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:11.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.221 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:11.221 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:44:11.222 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:11.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.222 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:44:11.222 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:44:11.222 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:11.222 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:11.222 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:11.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.222 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:44:11.222 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:44:11.222 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:11.223 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:11.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.223 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:11.223 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:44:11.223 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:11.223 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.223 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.223 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:44:11.223 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:44:11.223 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:11.223 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:11.223 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:11.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.224 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:44:11.224 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:44:11.224 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:11.224 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:11.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.224 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:11.224 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:44:11.224 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:11.224 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.224 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.224 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:44:11.225 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:44:11.225 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:11.225 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:11.225 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:11.225 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.225 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.225 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:44:11.225 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:44:11.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.225 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:11.225 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:11.225 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:44:11.225 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.225 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:11.226 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:44:11.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.226 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:11.226 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:11.226 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:44:11.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.226 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.226 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:44:11.226 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:44:11.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa85930e640 1 -- 172.21.15.26:0/654827951 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa834036420 legacy=0x7fa8340388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa85930e640 10 monclient: shutdown 2024-09-16T14:44:11.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa85930e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:11.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa85930e640 1 -- 172.21.15.26:0/654827951 >> v1:172.21.15.26:6789/0 conn(0x7fa8540a2790 legacy=0x7fa8541d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa8537fe640 1 -- 172.21.15.26:0/654827951 reap_dead start 2024-09-16T14:44:11.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa8537fe640 1 -- 172.21.15.26:0/654827951 reap_dead start 2024-09-16T14:44:11.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa85930e640 1 -- 172.21.15.26:0/654827951 shutdown_connections 2024-09-16T14:44:11.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa85930e640 1 -- 172.21.15.26:0/654827951 >> 172.21.15.26:0/654827951 conn(0x7fa8540a3da0 msgr2=0x7fa8540a6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:11.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa85930e640 1 -- 172.21.15.26:0/654827951 shutdown_connections 2024-09-16T14:44:11.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.194+0000 7fa85930e640 1 -- 172.21.15.26:0/654827951 wait complete. 2024-09-16T14:44:11.229 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:44:11.229 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:44:11.229 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:44:11.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.342+0000 7f52b8044640 10 monclient: get_monmap_and_config 2024-09-16T14:44:11.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.342+0000 7f52b8044640 10 monclient: build_initial_monmap 2024-09-16T14:44:11.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.342+0000 7f52b8044640 10 monclient: monmap: 2024-09-16T14:44:11.348 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:11.348 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.349 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:44:11.348651+0000 2024-09-16T14:44:11.349 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:44:11.348651+0000 2024-09-16T14:44:11.349 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:11.349 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.349 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:11.349 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:11.349 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:11.349 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 1 Processor -- start 2024-09-16T14:44:11.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 1 -- start start 2024-09-16T14:44:11.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 10 monclient: init 2024-09-16T14:44:11.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:11.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:11.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 10 monclient(hunting): picked mon.noname-c con 0x7f52b00a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:11.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 10 monclient(hunting): picked mon.noname-a con 0x7f52b00a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:11.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 10 monclient(hunting): picked mon.noname-b con 0x7f52b00a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:11.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b00a3250 con 0x7f52b00a0000 2024-09-16T14:44:11.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b009f600 con 0x7f52b00a1d10 2024-09-16T14:44:11.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b00a3830 con 0x7f52b00a2800 2024-09-16T14:44:11.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:11.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 10 monclient(hunting): authenticate will time out at 1075.468823s 2024-09-16T14:44:11.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b55df640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f52b00a0000 0x7f52b0139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58118/0 (socket says 172.21.15.26:58118) 2024-09-16T14:44:11.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b55df640 1 -- 172.21.15.26:0/1445639580 learned_addr learned my addr 172.21.15.26:0/1445639580 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:11.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 1 -- 172.21.15.26:0/1445639580 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762418057 0 0) 0x7f52b00a3830 con 0x7f52b00a2800 2024-09-16T14:44:11.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient(hunting): my global_id is 4724 2024-09-16T14:44:11.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 1 -- 172.21.15.26:0/1445639580 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f529c003530 con 0x7f52b00a2800 2024-09-16T14:44:11.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 1 -- 172.21.15.26:0/1445639580 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900823209 0 0) 0x7f52b00a3250 con 0x7f52b00a0000 2024-09-16T14:44:11.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient(hunting): my global_id is 84111 2024-09-16T14:44:11.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 1 -- 172.21.15.26:0/1445639580 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b00a3830 con 0x7f52b00a0000 2024-09-16T14:44:11.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 1 -- 172.21.15.26:0/1445639580 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4008916126 0 0) 0x7f529c003530 con 0x7f52b00a2800 2024-09-16T14:44:11.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 1 -- 172.21.15.26:0/1445639580 >> v1:172.21.15.189:6789/0 conn(0x7f52b00a1d10 legacy=0x7f52b00a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 1 -- 172.21.15.26:0/1445639580 >> v1:172.21.15.26:6789/0 conn(0x7f52b00a0000 legacy=0x7f52b0139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient: _finish_hunting 0 2024-09-16T14:44:11.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 1 monclient: found mon.noname-b 2024-09-16T14:44:11.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:11.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:11.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 1 -- 172.21.15.26:0/1445639580 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b009ed60 con 0x7f52b00a2800 2024-09-16T14:44:11.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient: _finish_auth 0 2024-09-16T14:44:11.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient: _check_auth_tickets 2024-09-16T14:44:11.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:11.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 1 -- 172.21.15.26:0/1445639580 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52a80019a0 con 0x7f52b00a2800 2024-09-16T14:44:11.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 5 monclient: authenticate success, global_id 4724 2024-09-16T14:44:11.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:11.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:11.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient: dump: 2024-09-16T14:44:11.356 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.356 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.356 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.356 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.356 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.356 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.356 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.357 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.357 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.357 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:11.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 1 -- 172.21.15.26:0/1445639580 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52a8001c40 con 0x7f52b00a2800 2024-09-16T14:44:11.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:11.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:11.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 1 -- 172.21.15.26:0/1445639580 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52a8001fd0 con 0x7f52b00a2800 2024-09-16T14:44:11.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 4 set_mon_vals no callback set 2024-09-16T14:44:11.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b4dde640 10 monclient: dump: 2024-09-16T14:44:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.358 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 10 monclient: shutdown 2024-09-16T14:44:11.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:11.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 1 -- 172.21.15.26:0/1445639580 >> v1:172.21.15.26:6790/0 conn(0x7f52b00a2800 legacy=0x7f52b013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 1 -- 172.21.15.26:0/1445639580 shutdown_connections 2024-09-16T14:44:11.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 1 -- 172.21.15.26:0/1445639580 >> 172.21.15.26:0/1445639580 conn(0x7f52b00a7f30 msgr2=0x7f52b00a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:11.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 1 -- 172.21.15.26:0/1445639580 shutdown_connections 2024-09-16T14:44:11.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 1 -- 172.21.15.26:0/1445639580 wait complete. 2024-09-16T14:44:11.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 10 monclient: build_initial_monmap 2024-09-16T14:44:11.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 10 monclient: monmap: 2024-09-16T14:44:11.359 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:11.359 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:11.359 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:44:11.360 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:44:11.360 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:11.360 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:44:11.360 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:11.360 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:11.360 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:11.360 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 1 Processor -- start 2024-09-16T14:44:11.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 1 -- start start 2024-09-16T14:44:11.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.346+0000 7f52b8044640 10 monclient: init 2024-09-16T14:44:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 10 monclient(hunting): picked mon.noname-a con 0x7f52b00a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 10 monclient(hunting): picked mon.noname-c con 0x7f52b00a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 10 monclient(hunting): picked mon.noname-b con 0x7f52b00a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b01335d0 con 0x7f52b00a0000 2024-09-16T14:44:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b01356e0 con 0x7f52b00a2800 2024-09-16T14:44:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b0133b20 con 0x7f52b00a1d10 2024-09-16T14:44:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 10 monclient(hunting): authenticate will time out at 1075.472143s 2024-09-16T14:44:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b5de0640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f52b00a0000 0x7f52b0134150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58124/0 (socket says 172.21.15.26:58124) 2024-09-16T14:44:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b5de0640 1 -- 172.21.15.26:0/2865935598 learned_addr learned my addr 172.21.15.26:0/2865935598 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b55df640 1 --1- 172.21.15.26:0/2865935598 >> v1:172.21.15.26:6790/0 conn(0x7f52b00a1d10 0x7f52b01cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59672/0 (socket says 172.21.15.26:59672) 2024-09-16T14:44:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987131339 0 0) 0x7f52b0133b20 con 0x7f52b00a1d10 2024-09-16T14:44:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): my global_id is 4727 2024-09-16T14:44:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5294003530 con 0x7f52b00a1d10 2024-09-16T14:44:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635885166 0 0) 0x7f52b01335d0 con 0x7f52b00a0000 2024-09-16T14:44:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): my global_id is 84114 2024-09-16T14:44:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b0133b20 con 0x7f52b00a0000 2024-09-16T14:44:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 697407095 0 0) 0x7f5294003530 con 0x7f52b00a1d10 2024-09-16T14:44:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52b01335d0 con 0x7f52b00a1d10 2024-09-16T14:44:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52a00018e0 con 0x7f52b00a1d10 2024-09-16T14:44:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): dump: 2024-09-16T14:44:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.371 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3352139162 0 0) 0x7f52b0133b20 con 0x7f52b00a0000 2024-09-16T14:44:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5294003530 con 0x7f52b00a0000 2024-09-16T14:44:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52ac001900 con 0x7f52b00a0000 2024-09-16T14:44:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): dump: 2024-09-16T14:44:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.372 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464180618 0 0) 0x7f52b01356e0 con 0x7f52b00a2800 2024-09-16T14:44:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): my global_id is 4753 2024-09-16T14:44:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b0133b20 con 0x7f52b00a2800 2024-09-16T14:44:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1867729611 0 0) 0x7f52b01335d0 con 0x7f52b00a1d10 2024-09-16T14:44:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 >> v1:172.21.15.189:6789/0 conn(0x7f52b00a2800 legacy=0x7f52b01d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 >> v1:172.21.15.26:6789/0 conn(0x7f52b00a0000 legacy=0x7f52b0134150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:44:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 monclient: found mon.c 2024-09-16T14:44:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b0133e00 con 0x7f52b00a1d10 2024-09-16T14:44:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient: _finish_auth 0 2024-09-16T14:44:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:44:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 5 monclient: authenticate success, global_id 4727 2024-09-16T14:44:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 10 monclient: _renew_subs 2024-09-16T14:44:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b0133840 con 0x7f52b00a1d10 2024-09-16T14:44:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 10 monclient: _renew_subs 2024-09-16T14:44:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b0131c40 con 0x7f52b00a1d10 2024-09-16T14:44:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52a00047f0 con 0x7f52b00a1d10 2024-09-16T14:44:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52a0004b80 con 0x7f52b00a1d10 2024-09-16T14:44:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a7fff640 4 set_mon_vals no callback set 2024-09-16T14:44:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient: dump: 2024-09-16T14:44:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.376 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b8044640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5298006590 con 0x7f5298002830 2024-09-16T14:44:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791999772 0 0) 0x7f5298006590 con 0x7f5298002830 2024-09-16T14:44:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient(hunting): my global_id is 4730 2024-09-16T14:44:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b01335d0 con 0x7f5298002830 2024-09-16T14:44:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f52a0004d60 con 0x7f52b00a1d10 2024-09-16T14:44:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (unknown 3430718669 0 0) 0x7f52a0043400 con 0x7f52b00a1d10 2024-09-16T14:44:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3325555897 0 0) 0x7f52b01335d0 con 0x7f5298002830 2024-09-16T14:44:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5298006590 con 0x7f5298002830 2024-09-16T14:44:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52ac0039d0 con 0x7f5298002830 2024-09-16T14:44:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient: dump: 2024-09-16T14:44:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.378 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1354152857 0 0) 0x7f5298006590 con 0x7f5298002830 2024-09-16T14:44:11.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f529800a0e0 con 0x7f5298002830 2024-09-16T14:44:11.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52b65e1640 10 monclient: get_auth_request con 0x7f52940363e0 auth_method 0 2024-09-16T14:44:11.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f52ac0057d0 con 0x7f5298002830 2024-09-16T14:44:11.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:11.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:11.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 >> v1:172.21.15.26:6790/0 conn(0x7f5298002830 legacy=0x7f5298004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.350+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 >> v1:172.21.15.26:6790/0 conn(0x7f5298002830 legacy=0x7f5298004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:11.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52b8044640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:44:11.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52b8044640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5298009d80 con 0x7f52980090a0 2024-09-16T14:44:11.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903299423 0 0) 0x7f5298009d80 con 0x7f52980090a0 2024-09-16T14:44:11.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 10 monclient(hunting): my global_id is 4733 2024-09-16T14:44:11.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5298006590 con 0x7f52980090a0 2024-09-16T14:44:11.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1885357945 0 0) 0x7f5298006590 con 0x7f52980090a0 2024-09-16T14:44:11.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5298009d80 con 0x7f52980090a0 2024-09-16T14:44:11.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52ac004170 con 0x7f52980090a0 2024-09-16T14:44:11.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:11.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 10 monclient: dump: 2024-09-16T14:44:11.380 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.380 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.381 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.381 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.381 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.381 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.381 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.381 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.381 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.381 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 621359359 0 0) 0x7f5298009d80 con 0x7f52980090a0 2024-09-16T14:44:11.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f529800c290 con 0x7f52980090a0 2024-09-16T14:44:11.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52b5de0640 1 -- 172.21.15.26:0/2865935598 >> v1:172.21.15.26:6790/0 conn(0x7f52980090a0 legacy=0x7f5298009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T14:44:11.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52b5de0640 1 -- 172.21.15.26:0/2865935598 >> v1:172.21.15.26:6790/0 conn(0x7f52980090a0 legacy=0x7f5298009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:44:11.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52b5de0640 1 --1- 172.21.15.26:0/2865935598 >> v1:172.21.15.26:6790/0 conn(0x7f52980090a0 0x7f5298009530 :-1 s=OPENED pgs=261 cs=1 l=1).handle_message read tag failed 2024-09-16T14:44:11.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52b5de0640 1 --1- 172.21.15.26:0/2865935598 >> v1:172.21.15.26:6790/0 conn(0x7f52980090a0 0x7f5298009530 :-1 s=OPENED pgs=261 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:44:11.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 >> v1:172.21.15.26:6790/0 conn(0x7f52980090a0 legacy=0x7f5298009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:11.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 >> v1:172.21.15.26:6790/0 conn(0x7f52980090a0 legacy=0x7f5298009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:11.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5298009d80 con 0x7f52940462c0 2024-09-16T14:44:11.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043413850 0 0) 0x7f5298009d80 con 0x7f52940462c0 2024-09-16T14:44:11.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 10 monclient(hunting): my global_id is 4736 2024-09-16T14:44:11.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5298006590 con 0x7f52940462c0 2024-09-16T14:44:11.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3618909790 0 0) 0x7f5298006590 con 0x7f52940462c0 2024-09-16T14:44:11.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5298009d80 con 0x7f52940462c0 2024-09-16T14:44:11.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52ac005cd0 con 0x7f52940462c0 2024-09-16T14:44:11.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:11.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 10 monclient: dump: 2024-09-16T14:44:11.383 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.383 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.383 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.383 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.383 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.384 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.384 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.384 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.384 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.384 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1384260078 0 0) 0x7f5298009d80 con 0x7f52940462c0 2024-09-16T14:44:11.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f52ac0057d0 con 0x7f52940462c0 2024-09-16T14:44:11.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 440707670) 0x7f52ac007e90 con 0x7f52940462c0 2024-09-16T14:44:11.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:44:11.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:11.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 >> v1:172.21.15.26:6790/0 conn(0x7f52940462c0 legacy=0x7f52940466a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52a67fc640 1 -- 172.21.15.26:0/2865935598 >> v1:172.21.15.26:6790/0 conn(0x7f52940462c0 legacy=0x7f52940466a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:11.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52b65e1640 1 -- 172.21.15.26:0/2865935598 reap_dead start 2024-09-16T14:44:11.385 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:44:11.385 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:44:11.385 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:44:11.385 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:44:11.386 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 22, 2024-09-16T14:44:11.386 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:44:11.386 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:44:11.386 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:44:11.386 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:44:11.386 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:11.386 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:44:11.386 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:44:11.386 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:44:11.386 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:44:11.386 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:11.387 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:11.387 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:11.387 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:11.387 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:11.387 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:11.387 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:11.387 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:11.387 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:11.387 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:11.387 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:11.388 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:44:11.388 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:44:11.388 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:11.388 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:11.388 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:11.388 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:11.388 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:11.388 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:11.389 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:11.389 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:11.389 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:11.389 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:11.389 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.389 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:44:11.389 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:44:11.390 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:44:11.390 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:44:11.390 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:44:11.390 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:44:11.390 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:44:11.390 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:44:11.390 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:44:11.390 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:44:11.390 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:44:11.390 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:44:11.390 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:44:11.391 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:44:11.391 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:44:11.391 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:44:11.391 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:44:11.391 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:11.391 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:11.391 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:11.391 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:11.391 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:11.391 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:11.391 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:11.392 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:11.392 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:11.392 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:11.392 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:11.392 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:44:11.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.392 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:44:11.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.392 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:44:11.392 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:44:11.393 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:11.393 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:11.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.393 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:11.393 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:44:11.393 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:11.393 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.393 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.393 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:44:11.394 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:44:11.394 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:11.394 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:11.394 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:11.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.394 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:44:11.394 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:44:11.394 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:11.394 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:11.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.394 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:11.395 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:44:11.395 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:11.395 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.395 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.395 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:44:11.395 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:44:11.395 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:11.395 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:11.395 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:11.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.396 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:44:11.396 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:44:11.396 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:11.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:11.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.396 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:11.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:44:11.396 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:11.396 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.397 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:44:11.397 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:44:11.397 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:11.397 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:11.397 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:11.397 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.397 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.397 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:44:11.397 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:44:11.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.398 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:11.398 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:11.398 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:44:11.398 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.398 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:11.398 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:44:11.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.398 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:11.398 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:11.398 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:44:11.398 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.399 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:11.399 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:44:11.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.399 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:11.399 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:11.399 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:44:11.399 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.399 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.399 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:44:11.399 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:44:11.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52b8044640 1 -- 172.21.15.26:0/2865935598 >> v1:172.21.15.189:6812/3196161582 conn(0x7f52940363e0 legacy=0x7f52940388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52b8044640 10 monclient: shutdown 2024-09-16T14:44:11.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52b8044640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:11.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52b8044640 1 -- 172.21.15.26:0/2865935598 >> v1:172.21.15.26:6790/0 conn(0x7f52b00a1d10 legacy=0x7f52b01cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52b8044640 1 -- 172.21.15.26:0/2865935598 shutdown_connections 2024-09-16T14:44:11.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52b8044640 1 -- 172.21.15.26:0/2865935598 >> 172.21.15.26:0/2865935598 conn(0x7f52b00a7f30 msgr2=0x7f52b006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:11.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52b8044640 1 -- 172.21.15.26:0/2865935598 shutdown_connections 2024-09-16T14:44:11.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.358+0000 7f52b8044640 1 -- 172.21.15.26:0/2865935598 wait complete. 2024-09-16T14:44:11.402 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:44:11.402 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:44:11.402 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:44:11.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 10 monclient: get_monmap_and_config 2024-09-16T14:44:11.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 10 monclient: build_initial_monmap 2024-09-16T14:44:11.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 10 monclient: monmap: 2024-09-16T14:44:11.517 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:11.517 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.517 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:44:11.517620+0000 2024-09-16T14:44:11.517 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:44:11.517620+0000 2024-09-16T14:44:11.518 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:11.518 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.518 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:11.518 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:11.518 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:11.518 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 1 Processor -- start 2024-09-16T14:44:11.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 1 -- start start 2024-09-16T14:44:11.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 10 monclient: init 2024-09-16T14:44:11.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:11.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:44:11.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 10 monclient(hunting): picked mon.noname-c con 0x7f50300a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:11.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 10 monclient(hunting): picked mon.noname-b con 0x7f5030099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:11.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 10 monclient(hunting): picked mon.noname-a con 0x7f50300a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:11.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5030098ea0 con 0x7f50300a26f0 2024-09-16T14:44:11.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50300997b0 con 0x7f50300a1c00 2024-09-16T14:44:11.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5030099460 con 0x7f5030099b60 2024-09-16T14:44:11.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:11.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 10 monclient(hunting): authenticate will time out at 1075.637664s 2024-09-16T14:44:11.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035a78640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5030099b60 0x7f5030139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59720/0 (socket says 172.21.15.26:59720) 2024-09-16T14:44:11.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035a78640 1 -- 172.21.15.26:0/621016758 learned_addr learned my addr 172.21.15.26:0/621016758 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:11.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5036a7a640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f50300a26f0 0x7f503013d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58140/0 (socket says 172.21.15.26:58140) 2024-09-16T14:44:11.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 1 -- 172.21.15.26:0/621016758 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145826891 0 0) 0x7f5030098ea0 con 0x7f50300a26f0 2024-09-16T14:44:11.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient(hunting): my global_id is 84117 2024-09-16T14:44:11.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 1 -- 172.21.15.26:0/621016758 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f501c003530 con 0x7f50300a26f0 2024-09-16T14:44:11.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 1 -- 172.21.15.26:0/621016758 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215998335 0 0) 0x7f5030099460 con 0x7f5030099b60 2024-09-16T14:44:11.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient(hunting): my global_id is 4739 2024-09-16T14:44:11.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 1 -- 172.21.15.26:0/621016758 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5030098ea0 con 0x7f5030099b60 2024-09-16T14:44:11.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 1 -- 172.21.15.26:0/621016758 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385532592 0 0) 0x7f50300997b0 con 0x7f50300a1c00 2024-09-16T14:44:11.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient(hunting): my global_id is 4756 2024-09-16T14:44:11.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 1 -- 172.21.15.26:0/621016758 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5030099460 con 0x7f50300a1c00 2024-09-16T14:44:11.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 1 -- 172.21.15.26:0/621016758 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 186362323 0 0) 0x7f501c003530 con 0x7f50300a26f0 2024-09-16T14:44:11.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 1 -- 172.21.15.26:0/621016758 >> v1:172.21.15.26:6790/0 conn(0x7f5030099b60 legacy=0x7f5030139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 1 -- 172.21.15.26:0/621016758 >> v1:172.21.15.189:6789/0 conn(0x7f50300a1c00 legacy=0x7f50300a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient: _finish_hunting 0 2024-09-16T14:44:11.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 1 monclient: found mon.noname-a 2024-09-16T14:44:11.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:11.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:11.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 1 -- 172.21.15.26:0/621016758 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50300a34b0 con 0x7f50300a26f0 2024-09-16T14:44:11.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient: _finish_auth 0 2024-09-16T14:44:11.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient: _check_auth_tickets 2024-09-16T14:44:11.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:11.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 1 -- 172.21.15.26:0/621016758 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f50280015a0 con 0x7f50300a26f0 2024-09-16T14:44:11.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:11.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient: dump: 2024-09-16T14:44:11.530 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.530 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.530 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.531 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.531 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.531 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.531 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.531 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.531 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.531 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 5 monclient: authenticate success, global_id 84117 2024-09-16T14:44:11.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:11.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 1 -- 172.21.15.26:0/621016758 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50280017c0 con 0x7f50300a26f0 2024-09-16T14:44:11.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:11.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 1 -- 172.21.15.26:0/621016758 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5028001b90 con 0x7f50300a26f0 2024-09-16T14:44:11.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:11.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 4 set_mon_vals no callback set 2024-09-16T14:44:11.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:11.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f5035277640 10 monclient: dump: 2024-09-16T14:44:11.532 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.532 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.532 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.532 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.533 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.533 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.533 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.533 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.533 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.533 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 10 monclient: shutdown 2024-09-16T14:44:11.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:11.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 1 -- 172.21.15.26:0/621016758 >> v1:172.21.15.26:6789/0 conn(0x7f50300a26f0 legacy=0x7f503013d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 1 -- 172.21.15.26:0/621016758 shutdown_connections 2024-09-16T14:44:11.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 1 -- 172.21.15.26:0/621016758 >> 172.21.15.26:0/621016758 conn(0x7f50300a3d00 msgr2=0x7f50300a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:11.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 1 -- 172.21.15.26:0/621016758 shutdown_connections 2024-09-16T14:44:11.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 1 -- 172.21.15.26:0/621016758 wait complete. 2024-09-16T14:44:11.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 10 monclient: build_initial_monmap 2024-09-16T14:44:11.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.514+0000 7f50384dd640 10 monclient: monmap: 2024-09-16T14:44:11.534 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:11.534 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:11.534 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:44:11.534 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:44:11.534 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:11.534 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:44:11.535 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:11.535 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:11.535 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:11.535 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 1 Processor -- start 2024-09-16T14:44:11.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 1 -- start start 2024-09-16T14:44:11.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 10 monclient: init 2024-09-16T14:44:11.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:11.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:11.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 10 monclient(hunting): picked mon.noname-c con 0x7f5030099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:11.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 10 monclient(hunting): picked mon.noname-a con 0x7f50300a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:11.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 10 monclient(hunting): picked mon.noname-b con 0x7f50300a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:11.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503009e3f0 con 0x7f50300a1c00 2024-09-16T14:44:11.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503011fd10 con 0x7f50300a26f0 2024-09-16T14:44:11.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503009f100 con 0x7f5030099b60 2024-09-16T14:44:11.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:11.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 10 monclient(hunting): authenticate will time out at 1075.640871s 2024-09-16T14:44:11.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5035a78640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f50300a1c00 0x7f503013d920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58146/0 (socket says 172.21.15.26:58146) 2024-09-16T14:44:11.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5035a78640 1 -- 172.21.15.26:0/2636825870 learned_addr learned my addr 172.21.15.26:0/2636825870 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:11.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5036279640 1 --1- 172.21.15.26:0/2636825870 >> v1:172.21.15.26:6790/0 conn(0x7f5030099b60 0x7f503013d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59728/0 (socket says 172.21.15.26:59728) 2024-09-16T14:44:11.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126361497 0 0) 0x7f503009e3f0 con 0x7f50300a1c00 2024-09-16T14:44:11.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): my global_id is 84120 2024-09-16T14:44:11.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5014003530 con 0x7f50300a1c00 2024-09-16T14:44:11.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958659378 0 0) 0x7f503009f100 con 0x7f5030099b60 2024-09-16T14:44:11.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): my global_id is 4742 2024-09-16T14:44:11.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503009e3f0 con 0x7f5030099b60 2024-09-16T14:44:11.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3918332161 0 0) 0x7f5014003530 con 0x7f50300a1c00 2024-09-16T14:44:11.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f503009f100 con 0x7f50300a1c00 2024-09-16T14:44:11.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5024001ce0 con 0x7f50300a1c00 2024-09-16T14:44:11.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:11.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): dump: 2024-09-16T14:44:11.538 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.538 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.538 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.538 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.538 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.538 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.538 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.539 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.539 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.539 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:11.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2712074434 0 0) 0x7f503009e3f0 con 0x7f5030099b60 2024-09-16T14:44:11.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5014003530 con 0x7f5030099b60 2024-09-16T14:44:11.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f502c001860 con 0x7f5030099b60 2024-09-16T14:44:11.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:11.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): dump: 2024-09-16T14:44:11.539 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.540 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.540 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.540 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.540 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.540 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.540 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.540 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.540 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.540 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199424327 0 0) 0x7f503011fd10 con 0x7f50300a26f0 2024-09-16T14:44:11.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): my global_id is 4759 2024-09-16T14:44:11.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503009e3f0 con 0x7f50300a26f0 2024-09-16T14:44:11.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3377212075 0 0) 0x7f503009f100 con 0x7f50300a1c00 2024-09-16T14:44:11.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 >> v1:172.21.15.26:6790/0 conn(0x7f5030099b60 legacy=0x7f503013d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 >> v1:172.21.15.189:6789/0 conn(0x7f50300a26f0 legacy=0x7f50301d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:11.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 monclient: found mon.a 2024-09-16T14:44:11.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:11.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:11.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503009ebb0 con 0x7f50300a1c00 2024-09-16T14:44:11.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient: _finish_auth 0 2024-09-16T14:44:11.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:11.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:11.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 5 monclient: authenticate success, global_id 84120 2024-09-16T14:44:11.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 10 monclient: _renew_subs 2024-09-16T14:44:11.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:11.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 1 -- 172.21.15.26:0/2636825870 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503011fa30 con 0x7f50300a1c00 2024-09-16T14:44:11.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 10 monclient: _renew_subs 2024-09-16T14:44:11.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:11.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 1 -- 172.21.15.26:0/2636825870 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f503011d9b0 con 0x7f50300a1c00 2024-09-16T14:44:11.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5024004b70 con 0x7f50300a1c00 2024-09-16T14:44:11.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:11.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5024004e80 con 0x7f50300a1c00 2024-09-16T14:44:11.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:11.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient: dump: 2024-09-16T14:44:11.543 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.543 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.543 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.543 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.543 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.543 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.543 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.544 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.544 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.544 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5034a76640 4 set_mon_vals no callback set 2024-09-16T14:44:11.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:11.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f50384dd640 1 -- 172.21.15.26:0/2636825870 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f50300a2d70 con 0x7f503013e030 2024-09-16T14:44:11.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f502400f8f0 con 0x7f50300a1c00 2024-09-16T14:44:11.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (unknown 3430718669 0 0) 0x7f5024043890 con 0x7f50300a1c00 2024-09-16T14:44:11.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5036a7a640 10 monclient: get_auth_request con 0x7f5014036490 auth_method 0 2024-09-16T14:44:11.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121583298 0 0) 0x7f50300a2d70 con 0x7f503013e030 2024-09-16T14:44:11.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient(hunting): my global_id is 4762 2024-09-16T14:44:11.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503009f100 con 0x7f503013e030 2024-09-16T14:44:11.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2392024401 0 0) 0x7f503009f100 con 0x7f503013e030 2024-09-16T14:44:11.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50300a2d70 con 0x7f503013e030 2024-09-16T14:44:11.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f502c0037c0 con 0x7f503013e030 2024-09-16T14:44:11.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:11.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.518+0000 7f5022ffd640 10 monclient: dump: 2024-09-16T14:44:11.546 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.546 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.546 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.546 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.546 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.546 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.546 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.546 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.546 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.546 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.522+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2408892539 0 0) 0x7f50300a2d70 con 0x7f503013e030 2024-09-16T14:44:11.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.522+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f503009a5b0 con 0x7f503013e030 2024-09-16T14:44:11.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.522+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f502c0057d0 con 0x7f503013e030 2024-09-16T14:44:11.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.522+0000 7f5022ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:11.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.522+0000 7f5022ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:11.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.522+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 >> v1:172.21.15.189:6789/0 conn(0x7f503013e030 legacy=0x7f5030138d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.522+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 >> v1:172.21.15.189:6789/0 conn(0x7f503013e030 legacy=0x7f5030138d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:11.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.526+0000 7f50384dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:44:11.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.526+0000 7f50384dd640 1 -- 172.21.15.26:0/2636825870 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f503011f470 con 0x7f503002c280 2024-09-16T14:44:11.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284238449 0 0) 0x7f503011f470 con 0x7f503002c280 2024-09-16T14:44:11.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 10 monclient(hunting): my global_id is 4765 2024-09-16T14:44:11.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50300a2d70 con 0x7f503002c280 2024-09-16T14:44:11.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4077724701 0 0) 0x7f50300a2d70 con 0x7f503002c280 2024-09-16T14:44:11.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f503011f470 con 0x7f503002c280 2024-09-16T14:44:11.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f502c003e30 con 0x7f503002c280 2024-09-16T14:44:11.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:11.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 10 monclient: dump: 2024-09-16T14:44:11.550 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.550 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.551 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.551 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.552 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.552 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.552 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.552 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.552 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.552 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3078069853 0 0) 0x7f503011f470 con 0x7f503002c280 2024-09-16T14:44:11.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f503009ea20 con 0x7f503002c280 2024-09-16T14:44:11.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 2483895614) 0x7f502c004070 con 0x7f503002c280 2024-09-16T14:44:11.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:44:11.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:11.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 >> v1:172.21.15.189:6789/0 conn(0x7f503002c280 legacy=0x7f5030139440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5022ffd640 1 -- 172.21.15.26:0/2636825870 >> v1:172.21.15.189:6789/0 conn(0x7f503002c280 legacy=0x7f5030139440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:11.553 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:44:11.553 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:44:11.553 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:44:11.553 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:44:11.554 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 22, 2024-09-16T14:44:11.554 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:44:11.554 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:44:11.554 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:44:11.554 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:44:11.554 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:11.554 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:44:11.554 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:44:11.554 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:44:11.554 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:44:11.554 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:11.555 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:11.555 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:11.555 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:11.555 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:11.555 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:11.555 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:11.555 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:11.555 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:11.555 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:11.555 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:11.556 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:44:11.556 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:44:11.556 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:11.556 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:11.556 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:11.556 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:11.556 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:11.556 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:11.556 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:11.556 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:11.556 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:11.557 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:11.557 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.557 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:44:11.557 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:44:11.557 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:44:11.557 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:44:11.557 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:44:11.557 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:44:11.557 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:44:11.557 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:44:11.558 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:44:11.558 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:44:11.558 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:44:11.558 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:44:11.558 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:44:11.558 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:44:11.558 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:44:11.558 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:44:11.558 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:44:11.558 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:11.559 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:11.559 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:11.559 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:11.559 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:11.559 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:11.559 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:11.559 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:11.559 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:11.559 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:11.559 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:11.559 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:44:11.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.560 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:44:11.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.560 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:44:11.560 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:44:11.560 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:11.560 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:11.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.560 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:11.560 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:44:11.560 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:11.560 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.561 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.561 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:44:11.561 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:44:11.561 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:11.561 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:11.561 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:11.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.561 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:44:11.561 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:44:11.561 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:11.562 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:11.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.562 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:11.562 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:44:11.562 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:11.562 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.562 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.562 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:44:11.562 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:44:11.562 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:11.563 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:11.563 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:11.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.563 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:44:11.563 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:44:11.563 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:11.563 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:11.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.563 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:11.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:44:11.564 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:11.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.564 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:44:11.564 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:44:11.564 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:11.564 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:11.564 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:11.565 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.565 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.565 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:44:11.565 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:44:11.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.565 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:11.565 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:11.565 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:44:11.565 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.566 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:11.566 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:44:11.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.566 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:11.566 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:11.566 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:44:11.566 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.566 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:11.566 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:44:11.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.567 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:11.567 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:11.567 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:44:11.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.567 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:11.567 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:44:11.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:11.567 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:11.567 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:11.567 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:44:11.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:11.568 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:11.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:11.568 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:44:11.568 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:44:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f50384dd640 1 -- 172.21.15.26:0/2636825870 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5014036490 legacy=0x7f5014038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f50384dd640 10 monclient: shutdown 2024-09-16T14:44:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f50384dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f50384dd640 1 -- 172.21.15.26:0/2636825870 >> v1:172.21.15.26:6789/0 conn(0x7f50300a1c00 legacy=0x7f503013d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5036a7a640 1 -- 172.21.15.26:0/2636825870 reap_dead start 2024-09-16T14:44:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f5036a7a640 1 -- 172.21.15.26:0/2636825870 reap_dead start 2024-09-16T14:44:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f50384dd640 1 -- 172.21.15.26:0/2636825870 shutdown_connections 2024-09-16T14:44:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f50384dd640 1 -- 172.21.15.26:0/2636825870 >> 172.21.15.26:0/2636825870 conn(0x7f50300a3d00 msgr2=0x7f50300a5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f50384dd640 1 -- 172.21.15.26:0/2636825870 shutdown_connections 2024-09-16T14:44:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.530+0000 7f50384dd640 1 -- 172.21.15.26:0/2636825870 wait complete. 2024-09-16T14:44:11.570 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:44:11.570 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:44:11.571 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:44:11.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4cc9ae640 10 monclient: get_monmap_and_config 2024-09-16T14:44:11.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4cc9ae640 10 monclient: build_initial_monmap 2024-09-16T14:44:11.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4cc9ae640 10 monclient: monmap: 2024-09-16T14:44:11.687 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:11.687 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.687 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:44:11.687357+0000 2024-09-16T14:44:11.687 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:44:11.687357+0000 2024-09-16T14:44:11.687 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:11.687 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.687 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:11.687 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:11.688 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:11.688 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4cc9ae640 1 Processor -- start 2024-09-16T14:44:11.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4cc9ae640 1 -- start start 2024-09-16T14:44:11.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4cc9ae640 10 monclient: init 2024-09-16T14:44:11.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4cc9ae640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:11.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4cc9ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:11.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4cc9ae640 10 monclient(hunting): picked mon.noname-a con 0x7fe4c809a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:11.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4cc9ae640 10 monclient(hunting): picked mon.noname-c con 0x7fe4c809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:11.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4cc9ae640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c809b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:11.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4cc9ae640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c809dc60 con 0x7fe4c809a550 2024-09-16T14:44:11.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4cc9ae640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c8099ae0 con 0x7fe4c809d1b0 2024-09-16T14:44:11.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4cc9ae640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c809b8f0 con 0x7fe4c809b040 2024-09-16T14:44:11.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4cc9ae640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:11.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4cc9ae640 10 monclient(hunting): authenticate will time out at 1075.807381s 2024-09-16T14:44:11.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4c659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe4c809a550 0x7fe4c809a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58154/0 (socket says 172.21.15.26:58154) 2024-09-16T14:44:11.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4c659c640 1 -- 172.21.15.26:0/851921277 learned_addr learned my addr 172.21.15.26:0/851921277 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:11.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4c6d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe4c809b040 0x7fe4c813a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59742/0 (socket says 172.21.15.26:59742) 2024-09-16T14:44:11.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4c559a640 1 -- 172.21.15.26:0/851921277 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652181975 0 0) 0x7fe4c809dc60 con 0x7fe4c809a550 2024-09-16T14:44:11.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4c559a640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4c559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4c559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4c559a640 10 monclient(hunting): my global_id is 84123 2024-09-16T14:44:11.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4c559a640 1 -- 172.21.15.26:0/851921277 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b4003530 con 0x7fe4c809a550 2024-09-16T14:44:11.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4c559a640 1 -- 172.21.15.26:0/851921277 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512486136 0 0) 0x7fe4c809b8f0 con 0x7fe4c809b040 2024-09-16T14:44:11.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4c559a640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4c559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4c559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4c559a640 10 monclient(hunting): my global_id is 4745 2024-09-16T14:44:11.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.682+0000 7fe4c559a640 1 -- 172.21.15.26:0/851921277 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4c809dc60 con 0x7fe4c809b040 2024-09-16T14:44:11.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 1 -- 172.21.15.26:0/851921277 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1618465384 0 0) 0x7fe4c809dc60 con 0x7fe4c809b040 2024-09-16T14:44:11.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 1 -- 172.21.15.26:0/851921277 >> v1:172.21.15.189:6789/0 conn(0x7fe4c809d1b0 legacy=0x7fe4c8136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 1 -- 172.21.15.26:0/851921277 >> v1:172.21.15.26:6789/0 conn(0x7fe4c809a550 legacy=0x7fe4c809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 10 monclient: _finish_hunting 0 2024-09-16T14:44:11.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 1 monclient: found mon.noname-b 2024-09-16T14:44:11.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:11.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:11.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 1 -- 172.21.15.26:0/851921277 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c809bbd0 con 0x7fe4c809b040 2024-09-16T14:44:11.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 10 monclient: _finish_auth 0 2024-09-16T14:44:11.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 10 monclient: _check_auth_tickets 2024-09-16T14:44:11.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:11.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 1 -- 172.21.15.26:0/851921277 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe4c0001980 con 0x7fe4c809b040 2024-09-16T14:44:11.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:11.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 10 monclient: dump: 2024-09-16T14:44:11.699 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.699 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.699 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.699 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.699 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.700 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.700 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.700 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.700 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.700 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 5 monclient: authenticate success, global_id 4745 2024-09-16T14:44:11.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:11.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 1 -- 172.21.15.26:0/851921277 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4c0001c20 con 0x7fe4c809b040 2024-09-16T14:44:11.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:11.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 1 -- 172.21.15.26:0/851921277 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe4c0001ff0 con 0x7fe4c809b040 2024-09-16T14:44:11.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:11.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c559a640 10 monclient: dump: 2024-09-16T14:44:11.701 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.701 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.701 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.701 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.701 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.701 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.701 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.701 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.701 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.701 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:11.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 4 set_mon_vals no callback set 2024-09-16T14:44:11.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 10 monclient: shutdown 2024-09-16T14:44:11.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:11.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 1 -- 172.21.15.26:0/851921277 >> v1:172.21.15.26:6790/0 conn(0x7fe4c809b040 legacy=0x7fe4c813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 1 -- 172.21.15.26:0/851921277 shutdown_connections 2024-09-16T14:44:11.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 1 -- 172.21.15.26:0/851921277 >> 172.21.15.26:0/851921277 conn(0x7fe4c809fb50 msgr2=0x7fe4c809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:11.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 1 -- 172.21.15.26:0/851921277 shutdown_connections 2024-09-16T14:44:11.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 1 -- 172.21.15.26:0/851921277 wait complete. 2024-09-16T14:44:11.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 10 monclient: build_initial_monmap 2024-09-16T14:44:11.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 10 monclient: monmap: 2024-09-16T14:44:11.702 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:11.702 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:11.702 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:44:11.702 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:44:11.702 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:11.703 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:44:11.703 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:11.703 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:11.703 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:11.703 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 1 Processor -- start 2024-09-16T14:44:11.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 1 -- start start 2024-09-16T14:44:11.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 10 monclient: init 2024-09-16T14:44:11.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:11.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:11.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 10 monclient(hunting): picked mon.noname-c con 0x7fe4c809a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:11.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 10 monclient(hunting): picked mon.noname-a con 0x7fe4c809b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:11.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:11.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c809a240 con 0x7fe4c809b040 2024-09-16T14:44:11.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c8139c80 con 0x7fe4c809d1b0 2024-09-16T14:44:11.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c8119f20 con 0x7fe4c809a550 2024-09-16T14:44:11.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:11.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 10 monclient(hunting): authenticate will time out at 1075.810821s 2024-09-16T14:44:11.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c659c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe4c809a550 0x7fe4c8132080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59746/0 (socket says 172.21.15.26:59746) 2024-09-16T14:44:11.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c659c640 1 -- 172.21.15.26:0/888874932 learned_addr learned my addr 172.21.15.26:0/888874932 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:11.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987163440 0 0) 0x7fe4c8119f20 con 0x7fe4c809a550 2024-09-16T14:44:11.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): my global_id is 4748 2024-09-16T14:44:11.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a8003530 con 0x7fe4c809a550 2024-09-16T14:44:11.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695470686 0 0) 0x7fe4c809a240 con 0x7fe4c809b040 2024-09-16T14:44:11.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): my global_id is 84126 2024-09-16T14:44:11.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4c8119f20 con 0x7fe4c809b040 2024-09-16T14:44:11.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2129951068 0 0) 0x7fe4a8003530 con 0x7fe4c809a550 2024-09-16T14:44:11.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4c809a240 con 0x7fe4c809a550 2024-09-16T14:44:11.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 722302315 0 0) 0x7fe4c8119f20 con 0x7fe4c809b040 2024-09-16T14:44:11.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4a8003530 con 0x7fe4c809b040 2024-09-16T14:44:11.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe4b0001a00 con 0x7fe4c809a550 2024-09-16T14:44:11.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:11.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): dump: 2024-09-16T14:44:11.706 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.706 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.706 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.706 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.706 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.706 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.706 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.707 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.707 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.707 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:11.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe4b8001800 con 0x7fe4c809b040 2024-09-16T14:44:11.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:11.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): dump: 2024-09-16T14:44:11.707 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.707 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.707 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.707 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.707 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.708 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.708 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.708 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.708 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.708 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788333546 0 0) 0x7fe4c8139c80 con 0x7fe4c809d1b0 2024-09-16T14:44:11.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:11.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): my global_id is 4771 2024-09-16T14:44:11.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4c8119f20 con 0x7fe4c809d1b0 2024-09-16T14:44:11.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 353721847 0 0) 0x7fe4c809a240 con 0x7fe4c809a550 2024-09-16T14:44:11.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 >> v1:172.21.15.189:6789/0 conn(0x7fe4c809d1b0 legacy=0x7fe4c81cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 >> v1:172.21.15.26:6789/0 conn(0x7fe4c809b040 legacy=0x7fe4c81cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:11.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 monclient: found mon.c 2024-09-16T14:44:11.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:11.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:11.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c811a200 con 0x7fe4c809a550 2024-09-16T14:44:11.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient: _finish_auth 0 2024-09-16T14:44:11.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:11.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:11.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 5 monclient: authenticate success, global_id 4748 2024-09-16T14:44:11.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 10 monclient: _renew_subs 2024-09-16T14:44:11.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:11.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4cc9ae640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c811a4e0 con 0x7fe4c809a550 2024-09-16T14:44:11.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4b0001c50 con 0x7fe4c809a550 2024-09-16T14:44:11.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:11.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c659c640 1 -- 172.21.15.26:0/888874932 >> v1:172.21.15.26:6790/0 conn(0x7fe4c809a550 legacy=0x7fe4c8132080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T14:44:11.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c659c640 1 -- 172.21.15.26:0/888874932 >> v1:172.21.15.26:6790/0 conn(0x7fe4c809a550 legacy=0x7fe4c8132080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:44:11.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c659c640 1 --1- 172.21.15.26:0/888874932 >> v1:172.21.15.26:6790/0 conn(0x7fe4c809a550 0x7fe4c8132080 :-1 s=OPENED pgs=266 cs=1 l=1).handle_message read tag failed 2024-09-16T14:44:11.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c659c640 1 --1- 172.21.15.26:0/888874932 >> v1:172.21.15.26:6790/0 conn(0x7fe4c809a550 0x7fe4c8132080 :-1 s=OPENED pgs=266 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:44:11.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient: ms_handle_reset current mon v1:172.21.15.26:6790/0 2024-09-16T14:44:11.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:11.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 >> v1:172.21.15.26:6790/0 conn(0x7fe4c809a550 legacy=0x7fe4c8132080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:11.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4c4d99640 4 set_mon_vals no callback set 2024-09-16T14:44:11.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:11.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.686+0000 7fe4beffd640 10 monclient(hunting): picked mon.a con 0x7fe4c809b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:11.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient(hunting): picked mon.c con 0x7fe4a800cd90 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:11.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient(hunting): picked mon.b con 0x7fe4c809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:11.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe4c809a240 con 0x7fe4c809b040 2024-09-16T14:44:11.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe4a8003530 con 0x7fe4c809d1b0 2024-09-16T14:44:11.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe4c8139c80 con 0x7fe4a800cd90 2024-09-16T14:44:11.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:11.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4cc9ae640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:11.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545217577 0 0) 0x7fe4c8139c80 con 0x7fe4a800cd90 2024-09-16T14:44:11.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:44:11.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.26:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fe4a8015620 con 0x7fe4a800cd90 2024-09-16T14:44:11.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116337503 0 0) 0x7fe4c809a240 con 0x7fe4c809b040 2024-09-16T14:44:11.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:44:11.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.26:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fe4c8139c80 con 0x7fe4c809b040 2024-09-16T14:44:11.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614239969 0 0) 0x7fe4a8003530 con 0x7fe4c809d1b0 2024-09-16T14:44:11.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:11.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:11.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:44:11.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.189:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fe4c809a240 con 0x7fe4c809d1b0 2024-09-16T14:44:11.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1094527230 0 0) 0x7fe4a8015620 con 0x7fe4a800cd90 2024-09-16T14:44:11.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 >> v1:172.21.15.189:6789/0 conn(0x7fe4c809d1b0 legacy=0x7fe4a8012a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 >> v1:172.21.15.26:6789/0 conn(0x7fe4c809b040 legacy=0x7fe4a8003cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:11.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 monclient: found mon.c 2024-09-16T14:44:11.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:11.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:11.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fe4a80177a0 con 0x7fe4a800cd90 2024-09-16T14:44:11.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:11.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c813aa70 con 0x7fe4a800cd90 2024-09-16T14:44:11.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient: _finish_auth 0 2024-09-16T14:44:11.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:11.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:11.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe4b8003840 con 0x7fe4a800cd90 2024-09-16T14:44:11.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:11.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:11.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient: dump: 2024-09-16T14:44:11.714 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:11.714 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:11.714 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.714 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:11.714 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:11.714 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:11.714 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:11.714 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:11.714 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:11.714 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:11.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4b8003a10 con 0x7fe4a800cd90 2024-09-16T14:44:11.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:11.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4bffff640 4 set_mon_vals no callback set 2024-09-16T14:44:11.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4cc9ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:11.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4cc9ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:11.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4cc9ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:11.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4cc9ae640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a0005050 con 0x7fe4a800cd90 2024-09-16T14:44:11.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe4b800e510 con 0x7fe4a800cd90 2024-09-16T14:44:11.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.2 v1:172.21.15.26:6790/0 6 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (unknown 3430718669 0 0) 0x7fe4b80424f0 con 0x7fe4a800cd90 2024-09-16T14:44:11.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.690+0000 7fe4c659c640 10 monclient: get_auth_request con 0x7fe4a8045af0 auth_method 0 2024-09-16T14:44:11.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.694+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe4b800fba0 con 0x7fe4a800cd90 2024-09-16T14:44:11.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.694+0000 7fe4beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:11.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.694+0000 7fe4beffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:11.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:11.902+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}]=0 pool 'foooo' removed v38) v1 ==== 155+0+0 (unknown 4104780126 0 0) 0x7f0e48012d50 con 0x7f0e50071900 2024-09-16T14:44:11.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:11.902+0000 7f0e3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:11.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:11.902+0000 7f0e3effd640 10 monclient: _finish_command 2 = system:0 pool 'foooo' removed 2024-09-16T14:44:11.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.918+0000 7fe4cc9ae640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:44:11.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.918+0000 7fe4cc9ae640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:44:11.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.918+0000 7fe4cc9ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:11.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.918+0000 7fe4cc9ae640 1 -- 172.21.15.26:0/888874932 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fe4a0005b10 con 0x7fe4a800cd90 2024-09-16T14:44:11.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.922+0000 7fe4beffd640 1 -- 172.21.15.26:0/888874932 <== mon.2 v1:172.21.15.26:6790/0 8 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fe4b8011c40 con 0x7fe4a800cd90 2024-09-16T14:44:11.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.922+0000 7fe4beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:44:11.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.922+0000 7fe4beffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:11.927 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.922+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:44:11.928 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.922+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=10,kv=11,logm=63} crc {auth=2532116857,config=692619927,health=1281764270,kv=1867599010,logm=2090627042}) 2024-09-16T14:44:11.928 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.922+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,health=11,kv=10,logm=64} crc {auth=3049298507,health=4264675569,kv=3455456366,logm=975175189}) 2024-09-16T14:44:11.928 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.922+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:44:11.928 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.922+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=10,kv=11,logm=63} crc {auth=2532116857,config=692619927,health=1281764270,kv=1867599010,logm=2090627042}) 2024-09-16T14:44:11.928 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.922+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=12,kv=11,logm=60} crc {auth=3049298507,config=692619927,health=1921103840,kv=1867599010,logm=1280569660}) 2024-09-16T14:44:11.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.926+0000 7fe4cc9ae640 1 -- 172.21.15.26:0/888874932 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe4a8045af0 legacy=0x7fe4a8047fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.926+0000 7fe4cc9ae640 10 monclient: shutdown 2024-09-16T14:44:11.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.926+0000 7fe4cc9ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:11.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.926+0000 7fe4cc9ae640 1 -- 172.21.15.26:0/888874932 >> v1:172.21.15.26:6790/0 conn(0x7fe4a800cd90 legacy=0x7fe4a800f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:11.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.926+0000 7fe4cc9ae640 1 -- 172.21.15.26:0/888874932 shutdown_connections 2024-09-16T14:44:11.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.926+0000 7fe4cc9ae640 1 -- 172.21.15.26:0/888874932 >> 172.21.15.26:0/888874932 conn(0x7fe4c809fb50 msgr2=0x7fe4c809e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:11.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.926+0000 7fe4cc9ae640 1 -- 172.21.15.26:0/888874932 shutdown_connections 2024-09-16T14:44:11.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:11.926+0000 7fe4cc9ae640 1 -- 172.21.15.26:0/888874932 wait complete. 2024-09-16T14:44:11.931 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.926+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:44:11.931 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.926+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3165362901}) 2024-09-16T14:44:11.931 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.926+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597133513}) 2024-09-16T14:44:11.931 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.926+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:44:11.931 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.926+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3165362901}) 2024-09-16T14:44:11.931 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.926+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=249107601}) 2024-09-16T14:44:11.932 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.926+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:44:11.932 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.926+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1462708866}) 2024-09-16T14:44:11.932 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.926+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2930002997}) 2024-09-16T14:44:11.932 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.926+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:44:11.932 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.926+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1462708866}) 2024-09-16T14:44:11.933 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.926+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=283501510}) 2024-09-16T14:44:11.934 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.930+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:44:11.934 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.930+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2481094601}) 2024-09-16T14:44:11.935 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.930+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3844035286}) 2024-09-16T14:44:11.935 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.930+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:44:11.935 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.930+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2481094601}) 2024-09-16T14:44:11.935 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.930+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4092932347}) 2024-09-16T14:44:11.937 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.934+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:44:11.937 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.934+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=527830389}) 2024-09-16T14:44:11.938 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.934+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853344868}) 2024-09-16T14:44:11.938 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.934+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:44:11.938 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.934+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=527830389}) 2024-09-16T14:44:11.938 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.934+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2590652900}) 2024-09-16T14:44:11.942 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.938+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:44:11.942 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.938+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=48,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=10} crc {logm=365447058,mdsmap=1531151339,mgr=2120425906,mgr_metadata=2109538692,mgrstat=3276404662,mon_config_key=4052528784,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=3051029974,osd_snap=3794734670,osdmap=3148460013}) 2024-09-16T14:44:11.942 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.938+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=11} crc {logm=4014510137,mdsmap=1531151339,mgr=2120425906,mgr_metadata=2109538692,mgrstat=128573416,mon_config_key=1518680965,monmap=3140514009,osd_metadata=3424249799,osd_pg_creating=3051029974,osd_snap=3794734670,osdmap=4159455274}) 2024-09-16T14:44:11.943 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.938+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:44:11.943 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.938+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=48,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=10} crc {logm=365447058,mdsmap=1531151339,mgr=2120425906,mgr_metadata=2109538692,mgrstat=3276404662,mon_config_key=4052528784,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=3051029974,osd_snap=3794734670,osdmap=3148460013}) 2024-09-16T14:44:11.943 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.938+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=50,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=8} crc {logm=1711141990,mdsmap=1531151339,mgr=2539502010,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=128573416,mon_config_key=3576433448,monmap=2765269217,osd_metadata=3424249799,osd_pg_creating=3051029974,osd_snap=3794734670,osdmap=1865964509}) 2024-09-16T14:44:11.946 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.942+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:44:11.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.942+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=67} crc {osdmap=2178725927}) 2024-09-16T14:44:11.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.942+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=65} crc {osdmap=3824483917}) 2024-09-16T14:44:11.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.942+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:44:11.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.942+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=67} crc {osdmap=2178725927}) 2024-09-16T14:44:11.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:44:11.942+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=62} crc {osdmap=2289060110}) 2024-09-16T14:44:11.951 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:44:12.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.010+0000 7f0e54e20640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:12.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.010+0000 7f0e54e20640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:12.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.010+0000 7f0e54e20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:12.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.010+0000 7f0e54e20640 1 -- 172.21.15.189:0/261167754 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0e5006fc70 con 0x7f0e50071900 2024-09-16T14:44:12.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.014+0000 7f0e3effd640 1 -- 172.21.15.189:0/261167754 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}]=0 pool 'foooo' does not exist v38) v1 ==== 162+0+0 (unknown 1818529821 0 0) 0x7f0e480144d0 con 0x7f0e50071900 2024-09-16T14:44:12.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.014+0000 7f0e3effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:12.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.014+0000 7f0e3effd640 10 monclient: _finish_command 3 = system:0 pool 'foooo' does not exist 2024-09-16T14:44:12.018 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foooo' does not exist 2024-09-16T14:44:12.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.014+0000 7f0e54e20640 1 -- 172.21.15.189:0/261167754 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0e30036270 legacy=0x7f0e30038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:12.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.014+0000 7f0e54e20640 10 monclient: shutdown 2024-09-16T14:44:12.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.014+0000 7f0e54e20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:12.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.014+0000 7f0e54e20640 1 -- 172.21.15.189:0/261167754 >> v1:172.21.15.189:6789/0 conn(0x7f0e50071900 legacy=0x7f0e50137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:12.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.014+0000 7f0e54e20640 1 -- 172.21.15.189:0/261167754 shutdown_connections 2024-09-16T14:44:12.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.014+0000 7f0e54e20640 1 -- 172.21.15.189:0/261167754 >> 172.21.15.189:0/261167754 conn(0x7f0e500a19b0 msgr2=0x7f0e50136c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:12.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.014+0000 7f0e54e20640 1 -- 172.21.15.189:0/261167754 shutdown_connections 2024-09-16T14:44:12.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.014+0000 7f0e54e20640 1 -- 172.21.15.189:0/261167754 wait complete. 2024-09-16T14:44:12.035 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-16T14:44:12.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.150+0000 7f8f037cf640 10 monclient: get_monmap_and_config 2024-09-16T14:44:12.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.150+0000 7f8f037cf640 10 monclient: build_initial_monmap 2024-09-16T14:44:12.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 10 monclient: monmap: 2024-09-16T14:44:12.157 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:12.158 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:12.158 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:12.157807+0000 2024-09-16T14:44:12.158 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:12.157807+0000 2024-09-16T14:44:12.158 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:12.158 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:12.158 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:12.158 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:12.158 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:12.158 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:12.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 1 Processor -- start 2024-09-16T14:44:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 1 -- start start 2024-09-16T14:44:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 10 monclient: init 2024-09-16T14:44:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 10 monclient(hunting): picked mon.noname-a con 0x7f8efc1393b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 10 monclient(hunting): picked mon.noname-c con 0x7f8efc139790 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 10 monclient(hunting): picked mon.noname-b con 0x7f8efc09e510 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc09aad0 con 0x7f8efc1393b0 2024-09-16T14:44:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc09d030 con 0x7f8efc139790 2024-09-16T14:44:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc09edd0 con 0x7f8efc09e510 2024-09-16T14:44:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 10 monclient(hunting): authenticate will time out at 1034.273653s 2024-09-16T14:44:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f00d6a640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8efc139790 0x7f8efc09de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38946/0 (socket says 172.21.15.189:38946) 2024-09-16T14:44:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f00d6a640 1 -- 172.21.15.189:0/2752629868 learned_addr learned my addr 172.21.15.189:0/2752629868 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 1 -- 172.21.15.189:0/2752629868 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486974106 0 0) 0x7f8efc09d030 con 0x7f8efc139790 2024-09-16T14:44:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 10 monclient(hunting): my global_id is 4774 2024-09-16T14:44:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 1 -- 172.21.15.189:0/2752629868 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee8003530 con 0x7f8efc139790 2024-09-16T14:44:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 1 -- 172.21.15.189:0/2752629868 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 767699964 0 0) 0x7f8ee8003530 con 0x7f8efc139790 2024-09-16T14:44:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 1 -- 172.21.15.189:0/2752629868 >> v1:172.21.15.26:6790/0 conn(0x7f8efc09e510 legacy=0x7f8efc141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 1 -- 172.21.15.189:0/2752629868 >> v1:172.21.15.26:6789/0 conn(0x7f8efc1393b0 legacy=0x7f8efc09d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 10 monclient: _finish_hunting 0 2024-09-16T14:44:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 1 monclient: found mon.noname-c 2024-09-16T14:44:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:44:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 1 -- 172.21.15.189:0/2752629868 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc09a7f0 con 0x7f8efc139790 2024-09-16T14:44:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 10 monclient: _finish_auth 0 2024-09-16T14:44:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 10 monclient: _check_auth_tickets 2024-09-16T14:44:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 1 -- 172.21.15.189:0/2752629868 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ef80019a0 con 0x7f8efc139790 2024-09-16T14:44:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 5 monclient: authenticate success, global_id 4774 2024-09-16T14:44:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 10 monclient: dump: 2024-09-16T14:44:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:12.168 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 1 -- 172.21.15.189:0/2752629868 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ef8001c50 con 0x7f8efc139790 2024-09-16T14:44:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:12.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:12.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 4 set_mon_vals no callback set 2024-09-16T14:44:12.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 1 -- 172.21.15.189:0/2752629868 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ef8002020 con 0x7f8efc139790 2024-09-16T14:44:12.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:12.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:12.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8eeffff640 10 monclient: dump: 2024-09-16T14:44:12.169 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:12.169 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:12.169 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:12.169 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:12.169 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:12.169 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:12.170 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:12.170 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:12.170 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:12.170 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:12.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 10 monclient: shutdown 2024-09-16T14:44:12.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:12.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 1 -- 172.21.15.189:0/2752629868 >> v1:172.21.15.189:6789/0 conn(0x7f8efc139790 legacy=0x7f8efc09de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:12.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 1 -- 172.21.15.189:0/2752629868 shutdown_connections 2024-09-16T14:44:12.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 1 -- 172.21.15.189:0/2752629868 >> 172.21.15.189:0/2752629868 conn(0x7f8efc0a3380 msgr2=0x7f8efc09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:12.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 1 -- 172.21.15.189:0/2752629868 shutdown_connections 2024-09-16T14:44:12.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 1 -- 172.21.15.189:0/2752629868 wait complete. 2024-09-16T14:44:12.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 10 monclient: build_initial_monmap 2024-09-16T14:44:12.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.154+0000 7f8f037cf640 10 monclient: monmap: 2024-09-16T14:44:12.171 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:12.171 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:12.171 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:12.171 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:12.171 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:12.171 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:12.171 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:12.171 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:12.171 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:12.171 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:12.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 1 Processor -- start 2024-09-16T14:44:12.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 1 -- start start 2024-09-16T14:44:12.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 10 monclient: init 2024-09-16T14:44:12.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:12.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:12.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 10 monclient(hunting): picked mon.noname-a con 0x7f8efc09e510 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:12.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 10 monclient(hunting): picked mon.noname-c con 0x7f8efc1393b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:12.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 10 monclient(hunting): picked mon.noname-b con 0x7f8efc139790 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:12.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc1275d0 con 0x7f8efc09e510 2024-09-16T14:44:12.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc125e40 con 0x7f8efc139790 2024-09-16T14:44:12.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc137840 con 0x7f8efc1393b0 2024-09-16T14:44:12.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:12.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 10 monclient(hunting): authenticate will time out at 1034.276810s 2024-09-16T14:44:12.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f01d6c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8efc139790 0x7f8efc1d6600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38960/0 (socket says 172.21.15.189:38960) 2024-09-16T14:44:12.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f01d6c640 1 -- 172.21.15.189:0/1536175098 learned_addr learned my addr 172.21.15.189:0/1536175098 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:12.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875006499 0 0) 0x7f8efc125e40 con 0x7f8efc139790 2024-09-16T14:44:12.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:12.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:12.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:12.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient(hunting): my global_id is 4777 2024-09-16T14:44:12.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee0003560 con 0x7f8efc139790 2024-09-16T14:44:12.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 716721097 0 0) 0x7f8ee0003560 con 0x7f8efc139790 2024-09-16T14:44:12.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8efc125e40 con 0x7f8efc139790 2024-09-16T14:44:12.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ef4001900 con 0x7f8efc139790 2024-09-16T14:44:12.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:12.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:12.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient(hunting): dump: 2024-09-16T14:44:12.174 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:12.174 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:12.174 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:12.174 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:12.174 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:12.174 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:12.174 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:12.174 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:12.174 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:12.174 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:12.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:12.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616209548 0 0) 0x7f8efc137840 con 0x7f8efc1393b0 2024-09-16T14:44:12.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:12.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:12.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:12.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient(hunting): my global_id is 4754 2024-09-16T14:44:12.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee0003560 con 0x7f8efc1393b0 2024-09-16T14:44:12.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295933959 0 0) 0x7f8efc1275d0 con 0x7f8efc09e510 2024-09-16T14:44:12.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:12.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:12.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:12.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient(hunting): my global_id is 84132 2024-09-16T14:44:12.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8efc137840 con 0x7f8efc09e510 2024-09-16T14:44:12.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3037208935 0 0) 0x7f8efc125e40 con 0x7f8efc139790 2024-09-16T14:44:12.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 >> v1:172.21.15.26:6790/0 conn(0x7f8efc1393b0 legacy=0x7f8efc1d2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:12.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 >> v1:172.21.15.26:6789/0 conn(0x7f8efc09e510 legacy=0x7f8efc1cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:12.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient: _finish_hunting 0 2024-09-16T14:44:12.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 monclient: found mon.b 2024-09-16T14:44:12.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:12.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:12.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc0a2010 con 0x7f8efc139790 2024-09-16T14:44:12.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient: _finish_auth 0 2024-09-16T14:44:12.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient: _check_auth_tickets 2024-09-16T14:44:12.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:12.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 5 monclient: authenticate success, global_id 4777 2024-09-16T14:44:12.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 10 monclient: _renew_subs 2024-09-16T14:44:12.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:12.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 1 -- 172.21.15.189:0/1536175098 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8efc1272f0 con 0x7f8efc139790 2024-09-16T14:44:12.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 10 monclient: _renew_subs 2024-09-16T14:44:12.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:12.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 1 -- 172.21.15.189:0/1536175098 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8efc124630 con 0x7f8efc139790 2024-09-16T14:44:12.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ef4001cd0 con 0x7f8efc139790 2024-09-16T14:44:12.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:12.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ef4003af0 con 0x7f8efc139790 2024-09-16T14:44:12.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:12.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:12.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eef7fe640 4 set_mon_vals no callback set 2024-09-16T14:44:12.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 10 monclient: dump: 2024-09-16T14:44:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:12.178 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8ef4003cd0 con 0x7f8efc139790 2024-09-16T14:44:12.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f037cf640 1 -- 172.21.15.189:0/1536175098 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8edc005050 con 0x7f8efc139790 2024-09-16T14:44:12.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(38..38 src has 1..38) v4 ==== 4290+0+0 (unknown 4183882184 0 0) 0x7f8ef40435b0 con 0x7f8efc139790 2024-09-16T14:44:12.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.158+0000 7f8f0156b640 10 monclient: get_auth_request con 0x7f8ee0036370 auth_method 0 2024-09-16T14:44:12.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.162+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8ef40145f0 con 0x7f8efc139790 2024-09-16T14:44:12.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.162+0000 7f8eedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:12.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.162+0000 7f8eedffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:12.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.322+0000 7f8f037cf640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:12.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.322+0000 7f8f037cf640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:12.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.322+0000 7f8f037cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:12.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:12.322+0000 7f8f037cf640 1 -- 172.21.15.189:0/1536175098 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8edc005880 con 0x7f8efc139790 2024-09-16T14:44:12.951 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:44:12.952 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:44:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.078+0000 7f85235e0640 10 monclient: get_monmap_and_config 2024-09-16T14:44:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.078+0000 7f85235e0640 10 monclient: build_initial_monmap 2024-09-16T14:44:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.078+0000 7f85235e0640 10 monclient: monmap: 2024-09-16T14:44:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:44:13.083778+0000 2024-09-16T14:44:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:44:13.083778+0000 2024-09-16T14:44:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:13.084 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.078+0000 7f85235e0640 1 Processor -- start 2024-09-16T14:44:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.078+0000 7f85235e0640 1 -- start start 2024-09-16T14:44:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.078+0000 7f85235e0640 10 monclient: init 2024-09-16T14:44:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.078+0000 7f85235e0640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.078+0000 7f85235e0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.078+0000 7f85235e0640 10 monclient(hunting): picked mon.noname-b con 0x7f851c0a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.078+0000 7f85235e0640 10 monclient(hunting): picked mon.noname-a con 0x7f851c0a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.078+0000 7f85235e0640 10 monclient(hunting): picked mon.noname-c con 0x7f851c0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.078+0000 7f85235e0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851c0a3250 con 0x7f851c0a0000 2024-09-16T14:44:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.078+0000 7f85235e0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851c09f600 con 0x7f851c0a2800 2024-09-16T14:44:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.078+0000 7f85235e0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851c0a3830 con 0x7f851c0a1d10 2024-09-16T14:44:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.078+0000 7f85235e0640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.078+0000 7f85235e0640 10 monclient(hunting): authenticate will time out at 1077.203920s 2024-09-16T14:44:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8520b7b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f851c0a0000 0x7f851c139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58182/0 (socket says 172.21.15.26:58182) 2024-09-16T14:44:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8520b7b640 1 -- 172.21.15.26:0/2234679425 learned_addr learned my addr 172.21.15.26:0/2234679425 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 1 -- 172.21.15.26:0/2234679425 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347170609 0 0) 0x7f851c0a3830 con 0x7f851c0a1d10 2024-09-16T14:44:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient(hunting): my global_id is 4757 2024-09-16T14:44:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 1 -- 172.21.15.26:0/2234679425 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8508003530 con 0x7f851c0a1d10 2024-09-16T14:44:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 1 -- 172.21.15.26:0/2234679425 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814585660 0 0) 0x7f851c0a3250 con 0x7f851c0a0000 2024-09-16T14:44:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient(hunting): my global_id is 84135 2024-09-16T14:44:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 1 -- 172.21.15.26:0/2234679425 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f851c0a3830 con 0x7f851c0a0000 2024-09-16T14:44:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 1 -- 172.21.15.26:0/2234679425 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4170575997 0 0) 0x7f8508003530 con 0x7f851c0a1d10 2024-09-16T14:44:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 1 -- 172.21.15.26:0/2234679425 >> v1:172.21.15.189:6789/0 conn(0x7f851c0a2800 legacy=0x7f851c13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 1 -- 172.21.15.26:0/2234679425 >> v1:172.21.15.26:6789/0 conn(0x7f851c0a0000 legacy=0x7f851c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient: _finish_hunting 0 2024-09-16T14:44:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 1 monclient: found mon.noname-b 2024-09-16T14:44:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 1 -- 172.21.15.26:0/2234679425 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f851c09ed60 con 0x7f851c0a1d10 2024-09-16T14:44:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient: _finish_auth 0 2024-09-16T14:44:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient: _check_auth_tickets 2024-09-16T14:44:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 1 -- 172.21.15.26:0/2234679425 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8518001620 con 0x7f851c0a1d10 2024-09-16T14:44:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 5 monclient: authenticate success, global_id 4757 2024-09-16T14:44:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient: dump: 2024-09-16T14:44:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.091 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 1 -- 172.21.15.26:0/2234679425 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85180018c0 con 0x7f851c0a1d10 2024-09-16T14:44:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 1 -- 172.21.15.26:0/2234679425 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8518001c90 con 0x7f851c0a1d10 2024-09-16T14:44:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 4 set_mon_vals no callback set 2024-09-16T14:44:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8513fff640 10 monclient: dump: 2024-09-16T14:44:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.092 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 10 monclient: shutdown 2024-09-16T14:44:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 1 -- 172.21.15.26:0/2234679425 >> v1:172.21.15.26:6790/0 conn(0x7f851c0a1d10 legacy=0x7f851c0a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 1 -- 172.21.15.26:0/2234679425 shutdown_connections 2024-09-16T14:44:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 1 -- 172.21.15.26:0/2234679425 >> 172.21.15.26:0/2234679425 conn(0x7f851c0a7f30 msgr2=0x7f851c0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 1 -- 172.21.15.26:0/2234679425 shutdown_connections 2024-09-16T14:44:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 1 -- 172.21.15.26:0/2234679425 wait complete. 2024-09-16T14:44:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 10 monclient: build_initial_monmap 2024-09-16T14:44:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 10 monclient: monmap: 2024-09-16T14:44:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:44:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:44:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:44:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:13.094 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:13.094 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 1 Processor -- start 2024-09-16T14:44:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 1 -- start start 2024-09-16T14:44:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 10 monclient: init 2024-09-16T14:44:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 10 monclient(hunting): picked mon.noname-a con 0x7f851c0a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 10 monclient(hunting): picked mon.noname-c con 0x7f851c0a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 10 monclient(hunting): picked mon.noname-b con 0x7f851c0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851c0a61d0 con 0x7f851c0a0000 2024-09-16T14:44:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851c09e500 con 0x7f851c0a2800 2024-09-16T14:44:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851c136000 con 0x7f851c0a1d10 2024-09-16T14:44:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f85235e0640 10 monclient(hunting): authenticate will time out at 1077.207306s 2024-09-16T14:44:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8520b7b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f851c0a1d10 0x7f851c1ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59778/0 (socket says 172.21.15.26:59778) 2024-09-16T14:44:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f852137c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f851c0a0000 0x7f851c13d590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58184/0 (socket says 172.21.15.26:58184) 2024-09-16T14:44:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8520b7b640 1 -- 172.21.15.26:0/754329577 learned_addr learned my addr 172.21.15.26:0/754329577 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775799750 0 0) 0x7f851c136000 con 0x7f851c0a1d10 2024-09-16T14:44:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8511ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8511ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8511ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8511ffb640 10 monclient(hunting): my global_id is 4760 2024-09-16T14:44:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8500003530 con 0x7f851c0a1d10 2024-09-16T14:44:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303094185 0 0) 0x7f851c0a61d0 con 0x7f851c0a0000 2024-09-16T14:44:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8511ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8511ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8511ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8511ffb640 10 monclient(hunting): my global_id is 84138 2024-09-16T14:44:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f851c136000 con 0x7f851c0a0000 2024-09-16T14:44:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1788381419 0 0) 0x7f851c136000 con 0x7f851c0a0000 2024-09-16T14:44:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f851c0a61d0 con 0x7f851c0a0000 2024-09-16T14:44:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.082+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2803837020 0 0) 0x7f8500003530 con 0x7f851c0a1d10 2024-09-16T14:44:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f851c136000 con 0x7f851c0a1d10 2024-09-16T14:44:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8518001bd0 con 0x7f851c0a0000 2024-09-16T14:44:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient(hunting): dump: 2024-09-16T14:44:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.101 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f850c001860 con 0x7f851c0a1d10 2024-09-16T14:44:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient(hunting): dump: 2024-09-16T14:44:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.102 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360014561 0 0) 0x7f851c09e500 con 0x7f851c0a2800 2024-09-16T14:44:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient(hunting): my global_id is 4783 2024-09-16T14:44:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8500003530 con 0x7f851c0a2800 2024-09-16T14:44:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3069134640 0 0) 0x7f851c0a61d0 con 0x7f851c0a0000 2024-09-16T14:44:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 >> v1:172.21.15.26:6790/0 conn(0x7f851c0a1d10 legacy=0x7f851c1ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 >> v1:172.21.15.189:6789/0 conn(0x7f851c0a2800 legacy=0x7f851c1d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:44:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 1 monclient: found mon.a 2024-09-16T14:44:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f851c136770 con 0x7f851c0a0000 2024-09-16T14:44:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient: _finish_auth 0 2024-09-16T14:44:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:44:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:13.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f85235e0640 5 monclient: authenticate success, global_id 84138 2024-09-16T14:44:13.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f85235e0640 10 monclient: _renew_subs 2024-09-16T14:44:13.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f85235e0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:13.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f85235e0640 1 -- 172.21.15.26:0/754329577 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f851c13dca0 con 0x7f851c0a0000 2024-09-16T14:44:13.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f85235e0640 10 monclient: _renew_subs 2024-09-16T14:44:13.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f85235e0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:13.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f85235e0640 1 -- 172.21.15.26:0/754329577 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f851c135610 con 0x7f851c0a0000 2024-09-16T14:44:13.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8518003aa0 con 0x7f851c0a0000 2024-09-16T14:44:13.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:13.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8518003db0 con 0x7f851c0a0000 2024-09-16T14:44:13.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:13.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f85137fe640 4 set_mon_vals no callback set 2024-09-16T14:44:13.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 10 monclient: dump: 2024-09-16T14:44:13.105 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.105 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.106 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.106 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.106 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.106 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.106 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.106 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.106 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.106 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8518003f10 con 0x7f851c0a0000 2024-09-16T14:44:13.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f85235e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:13.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f85235e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:13.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f85235e0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:13.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f85235e0640 1 -- 172.21.15.26:0/754329577 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84fc005050 con 0x7f851c0a0000 2024-09-16T14:44:13.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(38..38 src has 1..38) v4 ==== 4290+0+0 (unknown 4183882184 0 0) 0x7f85180435a0 con 0x7f851c0a0000 2024-09-16T14:44:13.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.086+0000 7f8520b7b640 10 monclient: get_auth_request con 0x7f8500036570 auth_method 0 2024-09-16T14:44:13.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.090+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8518014700 con 0x7f851c0a0000 2024-09-16T14:44:13.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.090+0000 7f8511ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:13.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.090+0000 7f8511ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:13.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.314+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' removed v39) v1 ==== 152+0+0 (unknown 3701465616 0 0) 0x7f8ef4012c90 con 0x7f8efc139790 2024-09-16T14:44:13.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.318+0000 7f8eedffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:13.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.318+0000 7f8eedffb640 10 monclient: _finish_command 2 = system:0 pool 'fooo' removed 2024-09-16T14:44:13.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.318+0000 7f85235e0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:44:13.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.318+0000 7f85235e0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:44:13.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.318+0000 7f85235e0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:13.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.318+0000 7f85235e0640 1 -- 172.21.15.26:0/754329577 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f84fc005880 con 0x7f851c0a0000 2024-09-16T14:44:13.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.318+0000 7f8511ffb640 1 -- 172.21.15.26:0/754329577 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 327170000) 0x7f8518012da0 con 0x7f851c0a0000 2024-09-16T14:44:13.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.318+0000 7f8511ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:44:13.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.318+0000 7f8511ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:13.325 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:44:13.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.322+0000 7f85235e0640 1 -- 172.21.15.26:0/754329577 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8500036570 legacy=0x7f8500038a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.322+0000 7f85235e0640 10 monclient: shutdown 2024-09-16T14:44:13.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.322+0000 7f85235e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:13.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.322+0000 7f85235e0640 1 -- 172.21.15.26:0/754329577 >> v1:172.21.15.26:6789/0 conn(0x7f851c0a0000 legacy=0x7f851c13d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.322+0000 7f85235e0640 1 -- 172.21.15.26:0/754329577 shutdown_connections 2024-09-16T14:44:13.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.322+0000 7f85235e0640 1 -- 172.21.15.26:0/754329577 >> 172.21.15.26:0/754329577 conn(0x7f851c0a7f30 msgr2=0x7f851c13e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:13.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.322+0000 7f85235e0640 1 -- 172.21.15.26:0/754329577 shutdown_connections 2024-09-16T14:44:13.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.322+0000 7f85235e0640 1 -- 172.21.15.26:0/754329577 wait complete. 2024-09-16T14:44:13.356 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:44:13.356 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:44:13.356 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:44:13.356 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:44:13.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.430+0000 7f8f037cf640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:13.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.430+0000 7f8f037cf640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:13.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.430+0000 7f8f037cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:13.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.430+0000 7f8f037cf640 1 -- 172.21.15.189:0/1536175098 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8edc005370 con 0x7f8efc139790 2024-09-16T14:44:13.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.434+0000 7f8eedffb640 1 -- 172.21.15.189:0/1536175098 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v39) v1 ==== 159+0+0 (unknown 2048264665 0 0) 0x7f8ef4014410 con 0x7f8efc139790 2024-09-16T14:44:13.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.434+0000 7f8eedffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:13.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.434+0000 7f8eedffb640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-16T14:44:13.439 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fooo' does not exist 2024-09-16T14:44:13.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.434+0000 7f8f037cf640 1 -- 172.21.15.189:0/1536175098 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8ee0036370 legacy=0x7f8ee0038830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.434+0000 7f8f037cf640 10 monclient: shutdown 2024-09-16T14:44:13.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.434+0000 7f8f037cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:13.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.434+0000 7f8f037cf640 1 -- 172.21.15.189:0/1536175098 >> v1:172.21.15.189:6789/0 conn(0x7f8efc139790 legacy=0x7f8efc1d6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.434+0000 7f8f037cf640 1 -- 172.21.15.189:0/1536175098 shutdown_connections 2024-09-16T14:44:13.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.434+0000 7f8f037cf640 1 -- 172.21.15.189:0/1536175098 >> 172.21.15.189:0/1536175098 conn(0x7f8efc0a3380 msgr2=0x7f8efc0a2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:13.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.434+0000 7f8f037cf640 1 -- 172.21.15.189:0/1536175098 shutdown_connections 2024-09-16T14:44:13.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.434+0000 7f8f037cf640 1 -- 172.21.15.189:0/1536175098 wait complete. 2024-09-16T14:44:13.455 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-16T14:44:13.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.478+0000 7f24ccd29640 10 monclient: get_monmap_and_config 2024-09-16T14:44:13.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.478+0000 7f24ccd29640 10 monclient: build_initial_monmap 2024-09-16T14:44:13.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.478+0000 7f24ccd29640 10 monclient: monmap: 2024-09-16T14:44:13.485 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:13.485 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.486 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:44:13.485385+0000 2024-09-16T14:44:13.486 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:44:13.485385+0000 2024-09-16T14:44:13.486 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:13.486 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.486 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:13.486 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:13.486 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:13.486 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 1 Processor -- start 2024-09-16T14:44:13.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 1 -- start start 2024-09-16T14:44:13.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 10 monclient: init 2024-09-16T14:44:13.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:13.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:13.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 10 monclient(hunting): picked mon.noname-a con 0x7f24c809a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:13.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 10 monclient(hunting): picked mon.noname-b con 0x7f24c809d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:13.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 10 monclient(hunting): picked mon.noname-c con 0x7f24c809b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:13.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c809dc60 con 0x7f24c809a550 2024-09-16T14:44:13.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c8099ae0 con 0x7f24c809b040 2024-09-16T14:44:13.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c809b8f0 con 0x7f24c809d1b0 2024-09-16T14:44:13.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:13.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 10 monclient(hunting): authenticate will time out at 1077.605341s 2024-09-16T14:44:13.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f24c809a550 0x7f24c809a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58198/0 (socket says 172.21.15.26:58198) 2024-09-16T14:44:13.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c5d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f24c809d1b0 0x7f24c8136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59780/0 (socket says 172.21.15.26:59780) 2024-09-16T14:44:13.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c659c640 1 -- 172.21.15.26:0/3716794582 learned_addr learned my addr 172.21.15.26:0/3716794582 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:13.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 1 -- 172.21.15.26:0/3716794582 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849042431 0 0) 0x7f24c809b8f0 con 0x7f24c809d1b0 2024-09-16T14:44:13.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient(hunting): my global_id is 4763 2024-09-16T14:44:13.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 1 -- 172.21.15.26:0/3716794582 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24b0003530 con 0x7f24c809d1b0 2024-09-16T14:44:13.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 1 -- 172.21.15.26:0/3716794582 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822586798 0 0) 0x7f24c809dc60 con 0x7f24c809a550 2024-09-16T14:44:13.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient(hunting): my global_id is 84141 2024-09-16T14:44:13.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 1 -- 172.21.15.26:0/3716794582 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24c809b8f0 con 0x7f24c809a550 2024-09-16T14:44:13.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 1 -- 172.21.15.26:0/3716794582 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466166180 0 0) 0x7f24c8099ae0 con 0x7f24c809b040 2024-09-16T14:44:13.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient(hunting): my global_id is 4786 2024-09-16T14:44:13.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 1 -- 172.21.15.26:0/3716794582 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24c809dc60 con 0x7f24c809b040 2024-09-16T14:44:13.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 1 -- 172.21.15.26:0/3716794582 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2868228229 0 0) 0x7f24c809b8f0 con 0x7f24c809a550 2024-09-16T14:44:13.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 1 -- 172.21.15.26:0/3716794582 >> v1:172.21.15.26:6790/0 conn(0x7f24c809d1b0 legacy=0x7f24c8136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 1 -- 172.21.15.26:0/3716794582 >> v1:172.21.15.189:6789/0 conn(0x7f24c809b040 legacy=0x7f24c813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient: _finish_hunting 0 2024-09-16T14:44:13.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 1 monclient: found mon.noname-a 2024-09-16T14:44:13.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:13.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:13.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 1 -- 172.21.15.26:0/3716794582 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c809bbd0 con 0x7f24c809a550 2024-09-16T14:44:13.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient: _finish_auth 0 2024-09-16T14:44:13.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient: _check_auth_tickets 2024-09-16T14:44:13.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:13.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 5 monclient: authenticate success, global_id 84141 2024-09-16T14:44:13.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:13.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 1 -- 172.21.15.26:0/3716794582 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24b8001580 con 0x7f24c809a550 2024-09-16T14:44:13.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:13.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient: dump: 2024-09-16T14:44:13.498 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.498 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.498 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.498 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.498 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.498 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.498 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.498 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.498 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.499 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:13.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 1 -- 172.21.15.26:0/3716794582 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24b8001820 con 0x7f24c809a550 2024-09-16T14:44:13.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:13.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 1 -- 172.21.15.26:0/3716794582 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24b8001bf0 con 0x7f24c809a550 2024-09-16T14:44:13.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:13.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 4 set_mon_vals no callback set 2024-09-16T14:44:13.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:13.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24c559a640 10 monclient: dump: 2024-09-16T14:44:13.500 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.500 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.500 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.500 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.500 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.500 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.500 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.500 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.500 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.500 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 10 monclient: shutdown 2024-09-16T14:44:13.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:13.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 1 -- 172.21.15.26:0/3716794582 >> v1:172.21.15.26:6789/0 conn(0x7f24c809a550 legacy=0x7f24c809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 1 -- 172.21.15.26:0/3716794582 shutdown_connections 2024-09-16T14:44:13.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 1 -- 172.21.15.26:0/3716794582 >> 172.21.15.26:0/3716794582 conn(0x7f24c809fb50 msgr2=0x7f24c809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:13.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 1 -- 172.21.15.26:0/3716794582 shutdown_connections 2024-09-16T14:44:13.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 1 -- 172.21.15.26:0/3716794582 wait complete. 2024-09-16T14:44:13.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 10 monclient: build_initial_monmap 2024-09-16T14:44:13.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 10 monclient: monmap: 2024-09-16T14:44:13.501 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:13.501 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:13.501 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:44:13.501 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:44:13.501 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:13.501 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:44:13.501 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:13.501 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:13.502 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:13.502 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 1 Processor -- start 2024-09-16T14:44:13.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 1 -- start start 2024-09-16T14:44:13.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.482+0000 7f24ccd29640 10 monclient: init 2024-09-16T14:44:13.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:13.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:13.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 10 monclient(hunting): picked mon.noname-a con 0x7f24c809a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:13.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 10 monclient(hunting): picked mon.noname-c con 0x7f24c809b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:13.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 10 monclient(hunting): picked mon.noname-b con 0x7f24c809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:13.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c809a240 con 0x7f24c809a550 2024-09-16T14:44:13.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c81a73f0 con 0x7f24c809d1b0 2024-09-16T14:44:13.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c81b9720 con 0x7f24c809b040 2024-09-16T14:44:13.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:13.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 10 monclient(hunting): authenticate will time out at 1077.608455s 2024-09-16T14:44:13.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24c659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f24c809a550 0x7f24c81c9160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58204/0 (socket says 172.21.15.26:58204) 2024-09-16T14:44:13.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24c5d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f24c809b040 0x7f24c81b8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59782/0 (socket says 172.21.15.26:59782) 2024-09-16T14:44:13.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24c659c640 1 -- 172.21.15.26:0/3478969192 learned_addr learned my addr 172.21.15.26:0/3478969192 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:13.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24c5d9b640 1 -- 172.21.15.26:0/3478969192 >> v1:172.21.15.26:6790/0 conn(0x7f24c809b040 legacy=0x7f24c81b8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T14:44:13.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24c5d9b640 1 -- 172.21.15.26:0/3478969192 >> v1:172.21.15.26:6790/0 conn(0x7f24c809b040 legacy=0x7f24c81b8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:44:13.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24c5d9b640 1 --1- 172.21.15.26:0/3478969192 >> v1:172.21.15.26:6790/0 conn(0x7f24c809b040 0x7f24c81b8a20 :-1 s=OPENED pgs=276 cs=1 l=1).handle_message read tag failed 2024-09-16T14:44:13.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24c5d9b640 1 --1- 172.21.15.26:0/3478969192 >> v1:172.21.15.26:6790/0 conn(0x7f24c809b040 0x7f24c81b8a20 :-1 s=OPENED pgs=276 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:44:13.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6790/0 2024-09-16T14:44:13.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063725132 0 0) 0x7f24c809a240 con 0x7f24c809a550 2024-09-16T14:44:13.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): my global_id is 84144 2024-09-16T14:44:13.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24a8003530 con 0x7f24c809a550 2024-09-16T14:44:13.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940466067 0 0) 0x7f24c81a73f0 con 0x7f24c809d1b0 2024-09-16T14:44:13.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): my global_id is 4789 2024-09-16T14:44:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24c809a240 con 0x7f24c809d1b0 2024-09-16T14:44:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 491674476 0 0) 0x7f24a8003530 con 0x7f24c809a550 2024-09-16T14:44:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24c81a73f0 con 0x7f24c809a550 2024-09-16T14:44:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24b80014c0 con 0x7f24c809a550 2024-09-16T14:44:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): dump: 2024-09-16T14:44:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.506 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.506 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.506 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.506 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.506 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.506 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:13.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 705296945 0 0) 0x7f24c809a240 con 0x7f24c809d1b0 2024-09-16T14:44:13.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24a8003530 con 0x7f24c809d1b0 2024-09-16T14:44:13.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24c0001c40 con 0x7f24c809d1b0 2024-09-16T14:44:13.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:13.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): dump: 2024-09-16T14:44:13.507 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.507 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.507 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.507 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.507 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.507 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.507 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.507 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.507 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.507 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3307104202 0 0) 0x7f24c81a73f0 con 0x7f24c809a550 2024-09-16T14:44:13.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 >> v1:172.21.15.26:6790/0 conn(0x7f24c809b040 legacy=0x7f24c81b8a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:13.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 >> v1:172.21.15.189:6789/0 conn(0x7f24c809d1b0 legacy=0x7f24c81d14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:13.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 monclient: found mon.a 2024-09-16T14:44:13.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:13.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:13.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c8071080 con 0x7f24c809a550 2024-09-16T14:44:13.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:44:13.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:13.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:13.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 5 monclient: authenticate success, global_id 84144 2024-09-16T14:44:13.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 10 monclient: _renew_subs 2024-09-16T14:44:13.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:13.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 1 -- 172.21.15.26:0/3478969192 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c81a7110 con 0x7f24c809a550 2024-09-16T14:44:13.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 10 monclient: _renew_subs 2024-09-16T14:44:13.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:13.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 1 -- 172.21.15.26:0/3478969192 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c81a5090 con 0x7f24c809a550 2024-09-16T14:44:13.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24b8004aa0 con 0x7f24c809a550 2024-09-16T14:44:13.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:13.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24b8001180 con 0x7f24c809a550 2024-09-16T14:44:13.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:13.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient: dump: 2024-09-16T14:44:13.509 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.509 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.509 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.509 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.510 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.510 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.510 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.510 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.510 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.510 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24c4d99640 4 set_mon_vals no callback set 2024-09-16T14:44:13.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:13.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24ccd29640 1 -- 172.21.15.26:0/3478969192 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f24ac006590 con 0x7f24ac002830 2024-09-16T14:44:13.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f24b8005ea0 con 0x7f24c809a550 2024-09-16T14:44:13.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(39..39 src has 1..39) v4 ==== 3940+0+0 (unknown 2059123099 0 0) 0x7f24b8043490 con 0x7f24c809a550 2024-09-16T14:44:13.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907563514 0 0) 0x7f24ac006590 con 0x7f24ac002830 2024-09-16T14:44:13.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient(hunting): my global_id is 84147 2024-09-16T14:44:13.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24c81a73f0 con 0x7f24ac002830 2024-09-16T14:44:13.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24c6d9d640 10 monclient: get_auth_request con 0x7f24a8036140 auth_method 0 2024-09-16T14:44:13.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1882274558 0 0) 0x7f24c81a73f0 con 0x7f24ac002830 2024-09-16T14:44:13.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24ac006590 con 0x7f24ac002830 2024-09-16T14:44:13.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24bc001860 con 0x7f24ac002830 2024-09-16T14:44:13.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:13.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient: dump: 2024-09-16T14:44:13.511 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.511 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.511 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.512 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.512 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.512 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.512 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.512 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.512 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.512 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 502989722 0 0) 0x7f24ac006590 con 0x7f24ac002830 2024-09-16T14:44:13.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24ac00a0e0 con 0x7f24ac002830 2024-09-16T14:44:13.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f24bc001ba0 con 0x7f24ac002830 2024-09-16T14:44:13.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:13.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:13.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 >> v1:172.21.15.26:6789/0 conn(0x7f24ac002830 legacy=0x7f24ac004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.486+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 >> v1:172.21.15.26:6789/0 conn(0x7f24ac002830 legacy=0x7f24ac004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:13.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24ccd29640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:44:13.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24ccd29640 1 -- 172.21.15.26:0/3478969192 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f24ac009d80 con 0x7f24ac0090a0 2024-09-16T14:44:13.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198715012 0 0) 0x7f24ac009d80 con 0x7f24ac0090a0 2024-09-16T14:44:13.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 10 monclient(hunting): my global_id is 84150 2024-09-16T14:44:13.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24ac006590 con 0x7f24ac0090a0 2024-09-16T14:44:13.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3032980567 0 0) 0x7f24ac006590 con 0x7f24ac0090a0 2024-09-16T14:44:13.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24ac009d80 con 0x7f24ac0090a0 2024-09-16T14:44:13.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24bc004b80 con 0x7f24ac0090a0 2024-09-16T14:44:13.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:13.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 10 monclient: dump: 2024-09-16T14:44:13.515 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.515 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.515 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.515 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.515 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.515 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.515 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.515 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.515 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.516 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2038205623 0 0) 0x7f24ac009d80 con 0x7f24ac0090a0 2024-09-16T14:44:13.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f24ac00c290 con 0x7f24ac0090a0 2024-09-16T14:44:13.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 1104274975) 0x7f24bc005f30 con 0x7f24ac0090a0 2024-09-16T14:44:13.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:44:13.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:13.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 >> v1:172.21.15.26:6789/0 conn(0x7f24ac0090a0 legacy=0x7f24ac009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24b6ffd640 1 -- 172.21.15.26:0/3478969192 >> v1:172.21.15.26:6789/0 conn(0x7f24ac0090a0 legacy=0x7f24ac009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:13.516 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:44:13.516 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:44:13.517 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:44:13.517 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:44:13.517 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 22, 2024-09-16T14:44:13.517 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:44:13.517 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:44:13.517 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:44:13.517 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:44:13.517 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:13.517 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:44:13.517 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:44:13.517 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:44:13.517 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:44:13.518 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:13.518 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:13.518 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:13.518 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:13.518 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:13.518 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:13.518 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:13.518 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:13.518 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:13.518 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:13.518 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:13.518 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:44:13.518 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:44:13.518 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:44:13.519 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:44:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:44:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:44:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:44:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:44:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:44:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:44:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:44:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:44:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:44:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:44:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:44:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:44:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:44:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:44:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:44:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:44:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:44:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:44:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:44:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:44:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:44:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:44:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:44:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:44:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:13.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:44:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:44:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:44:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:44:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:44:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:44:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:44:13.525 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:44:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:44:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:44:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:44:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.526 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:13.527 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:44:13.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.527 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:13.527 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:13.527 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:44:13.527 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.527 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.527 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:44:13.527 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:44:13.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24ccd29640 1 -- 172.21.15.26:0/3478969192 >> v1:172.21.15.189:6812/3196161582 conn(0x7f24a8036140 legacy=0x7f24a8038600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24ccd29640 10 monclient: shutdown 2024-09-16T14:44:13.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24ccd29640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:13.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24ccd29640 1 -- 172.21.15.26:0/3478969192 >> v1:172.21.15.26:6789/0 conn(0x7f24c809a550 legacy=0x7f24c81c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24c6d9d640 1 -- 172.21.15.26:0/3478969192 reap_dead start 2024-09-16T14:44:13.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24c6d9d640 1 -- 172.21.15.26:0/3478969192 reap_dead start 2024-09-16T14:44:13.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24ccd29640 1 -- 172.21.15.26:0/3478969192 shutdown_connections 2024-09-16T14:44:13.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24ccd29640 1 -- 172.21.15.26:0/3478969192 >> 172.21.15.26:0/3478969192 conn(0x7f24c809fb50 msgr2=0x7f24c80a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:13.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24ccd29640 1 -- 172.21.15.26:0/3478969192 shutdown_connections 2024-09-16T14:44:13.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.494+0000 7f24ccd29640 1 -- 172.21.15.26:0/3478969192 wait complete. 2024-09-16T14:44:13.530 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:44:13.530 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:44:13.530 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:44:13.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.566+0000 7f580fe54640 10 monclient: get_monmap_and_config 2024-09-16T14:44:13.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.566+0000 7f580fe54640 10 monclient: build_initial_monmap 2024-09-16T14:44:13.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.566+0000 7f580fe54640 10 monclient: monmap: 2024-09-16T14:44:13.573 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:13.574 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.574 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:13.573754+0000 2024-09-16T14:44:13.574 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:13.573754+0000 2024-09-16T14:44:13.574 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:13.574 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:13.574 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:13.574 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:13.574 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:13.574 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:13.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 1 Processor -- start 2024-09-16T14:44:13.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 1 -- start start 2024-09-16T14:44:13.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 10 monclient: init 2024-09-16T14:44:13.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:13.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:13.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 10 monclient(hunting): picked mon.noname-a con 0x7f5808099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:13.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 10 monclient(hunting): picked mon.noname-b con 0x7f580809c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:13.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 10 monclient(hunting): picked mon.noname-c con 0x7f580809e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:13.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5808099e50 con 0x7f5808099a70 2024-09-16T14:44:13.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580809dbd0 con 0x7f580809e6d0 2024-09-16T14:44:13.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580809ee20 con 0x7f580809c550 2024-09-16T14:44:13.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:13.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 10 monclient(hunting): authenticate will time out at 1035.689451s 2024-09-16T14:44:13.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580e3f1640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f580809e6d0 0x7f580813b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38966/0 (socket says 172.21.15.189:38966) 2024-09-16T14:44:13.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580e3f1640 1 -- 172.21.15.189:0/2027558521 learned_addr learned my addr 172.21.15.189:0/2027558521 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:13.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 1 -- 172.21.15.189:0/2027558521 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237755190 0 0) 0x7f580809dbd0 con 0x7f580809e6d0 2024-09-16T14:44:13.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient(hunting): my global_id is 4792 2024-09-16T14:44:13.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 1 -- 172.21.15.189:0/2027558521 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57f4003530 con 0x7f580809e6d0 2024-09-16T14:44:13.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 1 -- 172.21.15.189:0/2027558521 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926803274 0 0) 0x7f580809ee20 con 0x7f580809c550 2024-09-16T14:44:13.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient(hunting): my global_id is 4766 2024-09-16T14:44:13.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 1 -- 172.21.15.189:0/2027558521 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f580809dbd0 con 0x7f580809c550 2024-09-16T14:44:13.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 1 -- 172.21.15.189:0/2027558521 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018209465 0 0) 0x7f5808099e50 con 0x7f5808099a70 2024-09-16T14:44:13.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient(hunting): my global_id is 84153 2024-09-16T14:44:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 1 -- 172.21.15.189:0/2027558521 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f580809ee20 con 0x7f5808099a70 2024-09-16T14:44:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 1 -- 172.21.15.189:0/2027558521 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 318587248 0 0) 0x7f57f4003530 con 0x7f580809e6d0 2024-09-16T14:44:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 1 -- 172.21.15.189:0/2027558521 >> v1:172.21.15.26:6790/0 conn(0x7f580809c550 legacy=0x7f58081379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 1 -- 172.21.15.189:0/2027558521 >> v1:172.21.15.26:6789/0 conn(0x7f5808099a70 legacy=0x7f580809dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient: _finish_hunting 0 2024-09-16T14:44:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 1 monclient: found mon.noname-c 2024-09-16T14:44:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:44:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 1 -- 172.21.15.189:0/2027558521 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580809f100 con 0x7f580809e6d0 2024-09-16T14:44:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient: _finish_auth 0 2024-09-16T14:44:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient: _check_auth_tickets 2024-09-16T14:44:13.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:13.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 1 -- 172.21.15.189:0/2027558521 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58000019a0 con 0x7f580809e6d0 2024-09-16T14:44:13.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:13.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient: dump: 2024-09-16T14:44:13.585 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:13.585 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.585 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.585 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.585 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.585 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:13.585 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.586 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.586 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.586 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:13.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 5 monclient: authenticate success, global_id 4792 2024-09-16T14:44:13.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:13.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 1 -- 172.21.15.189:0/2027558521 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5800001c10 con 0x7f580809e6d0 2024-09-16T14:44:13.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:13.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 1 -- 172.21.15.189:0/2027558521 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5800001fa0 con 0x7f580809e6d0 2024-09-16T14:44:13.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:13.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580cbee640 10 monclient: dump: 2024-09-16T14:44:13.587 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:13.587 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.587 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.587 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.587 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.587 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:13.587 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.587 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.587 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.587 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:13.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:13.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 4 set_mon_vals no callback set 2024-09-16T14:44:13.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 10 monclient: shutdown 2024-09-16T14:44:13.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:13.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 1 -- 172.21.15.189:0/2027558521 >> v1:172.21.15.189:6789/0 conn(0x7f580809e6d0 legacy=0x7f580813b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 1 -- 172.21.15.189:0/2027558521 shutdown_connections 2024-09-16T14:44:13.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 1 -- 172.21.15.189:0/2027558521 >> 172.21.15.189:0/2027558521 conn(0x7f58080a3380 msgr2=0x7f580809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:13.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 1 -- 172.21.15.189:0/2027558521 shutdown_connections 2024-09-16T14:44:13.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 1 -- 172.21.15.189:0/2027558521 wait complete. 2024-09-16T14:44:13.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 10 monclient: build_initial_monmap 2024-09-16T14:44:13.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.570+0000 7f580fe54640 10 monclient: monmap: 2024-09-16T14:44:13.588 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:13.588 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:13.588 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:13.588 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:13.589 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:13.589 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:13.589 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:13.589 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:13.589 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:13.589 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:13.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 1 Processor -- start 2024-09-16T14:44:13.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 1 -- start start 2024-09-16T14:44:13.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 10 monclient: init 2024-09-16T14:44:13.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:13.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:44:13.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 10 monclient(hunting): picked mon.noname-c con 0x7f5808099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:13.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 10 monclient(hunting): picked mon.noname-b con 0x7f580809c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:13.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 10 monclient(hunting): picked mon.noname-a con 0x7f580809e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:13.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580809cf20 con 0x7f580809e6d0 2024-09-16T14:44:13.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580809be20 con 0x7f580809c550 2024-09-16T14:44:13.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580806cf00 con 0x7f5808099a70 2024-09-16T14:44:13.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:13.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 10 monclient(hunting): authenticate will time out at 1035.692651s 2024-09-16T14:44:13.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580d3ef640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f580809c550 0x7f580809a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38974/0 (socket says 172.21.15.189:38974) 2024-09-16T14:44:13.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580d3ef640 1 -- 172.21.15.189:0/1563518999 learned_addr learned my addr 172.21.15.189:0/1563518999 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:13.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404874429 0 0) 0x7f580809be20 con 0x7f580809c550 2024-09-16T14:44:13.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient(hunting): my global_id is 4795 2024-09-16T14:44:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57ec003530 con 0x7f580809c550 2024-09-16T14:44:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345737545 0 0) 0x7f580806cf00 con 0x7f5808099a70 2024-09-16T14:44:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient(hunting): my global_id is 4769 2024-09-16T14:44:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f580809be20 con 0x7f5808099a70 2024-09-16T14:44:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3168360504 0 0) 0x7f57ec003530 con 0x7f580809c550 2024-09-16T14:44:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f580806cf00 con 0x7f580809c550 2024-09-16T14:44:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f57f8001870 con 0x7f580809c550 2024-09-16T14:44:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient(hunting): dump: 2024-09-16T14:44:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.592 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963298135 0 0) 0x7f580809cf20 con 0x7f580809e6d0 2024-09-16T14:44:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient(hunting): my global_id is 84156 2024-09-16T14:44:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57ec003530 con 0x7f580809e6d0 2024-09-16T14:44:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1922568295 0 0) 0x7f580806cf00 con 0x7f580809c550 2024-09-16T14:44:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 >> v1:172.21.15.26:6790/0 conn(0x7f5808099a70 legacy=0x7f580809a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 >> v1:172.21.15.26:6789/0 conn(0x7f580809e6d0 legacy=0x7f58081d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:44:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 monclient: found mon.b 2024-09-16T14:44:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580806faf0 con 0x7f580809c550 2024-09-16T14:44:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient: _finish_auth 0 2024-09-16T14:44:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:44:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 5 monclient: authenticate success, global_id 4795 2024-09-16T14:44:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 10 monclient: _renew_subs 2024-09-16T14:44:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580809b040 con 0x7f580809c550 2024-09-16T14:44:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 10 monclient: _renew_subs 2024-09-16T14:44:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f580806b320 con 0x7f580809c550 2024-09-16T14:44:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57f8003660 con 0x7f580809c550 2024-09-16T14:44:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f57f80039f0 con 0x7f580809c550 2024-09-16T14:44:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fffff640 4 set_mon_vals no callback set 2024-09-16T14:44:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 10 monclient: dump: 2024-09-16T14:44:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.596 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f57f8003b50 con 0x7f580809c550 2024-09-16T14:44:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580fe54640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f0005050 con 0x7f580809c550 2024-09-16T14:44:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(39..39 src has 1..39) v4 ==== 3940+0+0 (unknown 2059123099 0 0) 0x7f57f8043270 con 0x7f580809c550 2024-09-16T14:44:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.574+0000 7f580dbf0640 10 monclient: get_auth_request con 0x7f57ec036010 auth_method 0 2024-09-16T14:44:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.578+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f57f8003e80 con 0x7f580809c550 2024-09-16T14:44:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.578+0000 7f57fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.578+0000 7f57fe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:13.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.658+0000 7fbcffa90640 10 monclient: get_monmap_and_config 2024-09-16T14:44:13.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.658+0000 7fbcffa90640 10 monclient: build_initial_monmap 2024-09-16T14:44:13.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 10 monclient: monmap: 2024-09-16T14:44:13.665 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:13.665 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.665 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:44:13.665504+0000 2024-09-16T14:44:13.665 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:44:13.665504+0000 2024-09-16T14:44:13.665 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:13.666 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.666 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:13.666 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:13.666 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:13.666 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 1 Processor -- start 2024-09-16T14:44:13.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 1 -- start start 2024-09-16T14:44:13.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 10 monclient: init 2024-09-16T14:44:13.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:13.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:44:13.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 10 monclient(hunting): picked mon.noname-b con 0x7fbcf80a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:13.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 10 monclient(hunting): picked mon.noname-c con 0x7fbcf80a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:13.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 10 monclient(hunting): picked mon.noname-a con 0x7fbcf80a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:13.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf80a3250 con 0x7fbcf80a2800 2024-09-16T14:44:13.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf809f600 con 0x7fbcf80a0000 2024-09-16T14:44:13.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf80a3830 con 0x7fbcf80a1d10 2024-09-16T14:44:13.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:13.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 10 monclient(hunting): authenticate will time out at 1077.785417s 2024-09-16T14:44:13.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfd82c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbcf80a1d10 0x7fbcf80a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59790/0 (socket says 172.21.15.26:59790) 2024-09-16T14:44:13.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfd82c640 1 -- 172.21.15.26:0/492261600 learned_addr learned my addr 172.21.15.26:0/492261600 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:13.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 1 -- 172.21.15.26:0/492261600 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464267346 0 0) 0x7fbcf80a3250 con 0x7fbcf80a2800 2024-09-16T14:44:13.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient(hunting): my global_id is 84159 2024-09-16T14:44:13.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 1 -- 172.21.15.26:0/492261600 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbce4003530 con 0x7fbcf80a2800 2024-09-16T14:44:13.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 1 -- 172.21.15.26:0/492261600 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919082215 0 0) 0x7fbcf80a3830 con 0x7fbcf80a1d10 2024-09-16T14:44:13.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient(hunting): my global_id is 4772 2024-09-16T14:44:13.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 1 -- 172.21.15.26:0/492261600 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf80a3250 con 0x7fbcf80a1d10 2024-09-16T14:44:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 1 -- 172.21.15.26:0/492261600 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1337392185 0 0) 0x7fbce4003530 con 0x7fbcf80a2800 2024-09-16T14:44:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 1 -- 172.21.15.26:0/492261600 >> v1:172.21.15.26:6790/0 conn(0x7fbcf80a1d10 legacy=0x7fbcf80a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 1 -- 172.21.15.26:0/492261600 >> v1:172.21.15.189:6789/0 conn(0x7fbcf80a0000 legacy=0x7fbcf8139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient: _finish_hunting 0 2024-09-16T14:44:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 1 monclient: found mon.noname-a 2024-09-16T14:44:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 1 -- 172.21.15.26:0/492261600 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf809ed60 con 0x7fbcf80a2800 2024-09-16T14:44:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient: _finish_auth 0 2024-09-16T14:44:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient: _check_auth_tickets 2024-09-16T14:44:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 1 -- 172.21.15.26:0/492261600 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbcf00018d0 con 0x7fbcf80a2800 2024-09-16T14:44:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 5 monclient: authenticate success, global_id 84159 2024-09-16T14:44:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient: dump: 2024-09-16T14:44:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.673 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 1 -- 172.21.15.26:0/492261600 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcf0001b70 con 0x7fbcf80a2800 2024-09-16T14:44:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 1 -- 172.21.15.26:0/492261600 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbcf0001f00 con 0x7fbcf80a2800 2024-09-16T14:44:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcfc82a640 10 monclient: dump: 2024-09-16T14:44:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.673 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.674 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 4 set_mon_vals no callback set 2024-09-16T14:44:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 10 monclient: shutdown 2024-09-16T14:44:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 1 -- 172.21.15.26:0/492261600 >> v1:172.21.15.26:6789/0 conn(0x7fbcf80a2800 legacy=0x7fbcf813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 1 -- 172.21.15.26:0/492261600 shutdown_connections 2024-09-16T14:44:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 1 -- 172.21.15.26:0/492261600 >> 172.21.15.26:0/492261600 conn(0x7fbcf80a7f30 msgr2=0x7fbcf80a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:13.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 1 -- 172.21.15.26:0/492261600 shutdown_connections 2024-09-16T14:44:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 1 -- 172.21.15.26:0/492261600 wait complete. 2024-09-16T14:44:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 10 monclient: build_initial_monmap 2024-09-16T14:44:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 10 monclient: monmap: 2024-09-16T14:44:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:44:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:44:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:44:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:13.675 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.662+0000 7fbcffa90640 1 Processor -- start 2024-09-16T14:44:13.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 1 -- start start 2024-09-16T14:44:13.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 10 monclient: init 2024-09-16T14:44:13.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:13.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:44:13.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 10 monclient(hunting): picked mon.noname-c con 0x7fbcf80a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:13.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 10 monclient(hunting): picked mon.noname-b con 0x7fbcf80a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:13.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 10 monclient(hunting): picked mon.noname-a con 0x7fbcf80a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:13.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf813cd30 con 0x7fbcf80a2800 2024-09-16T14:44:13.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf813a3e0 con 0x7fbcf80a1d10 2024-09-16T14:44:13.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf80a68f0 con 0x7fbcf80a0000 2024-09-16T14:44:13.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:13.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 10 monclient(hunting): authenticate will time out at 1077.788589s 2024-09-16T14:44:13.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcfd82c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbcf80a0000 0x7fbcf81395c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59804/0 (socket says 172.21.15.26:59804) 2024-09-16T14:44:13.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcfd82c640 1 -- 172.21.15.26:0/3447627388 learned_addr learned my addr 172.21.15.26:0/3447627388 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:13.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176410799 0 0) 0x7fbcf813cd30 con 0x7fbcf80a2800 2024-09-16T14:44:13.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): my global_id is 84162 2024-09-16T14:44:13.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcdc003530 con 0x7fbcf80a2800 2024-09-16T14:44:13.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072790115 0 0) 0x7fbcf80a68f0 con 0x7fbcf80a0000 2024-09-16T14:44:13.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): my global_id is 4775 2024-09-16T14:44:13.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf813cd30 con 0x7fbcf80a0000 2024-09-16T14:44:13.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428875943 0 0) 0x7fbcf813a3e0 con 0x7fbcf80a1d10 2024-09-16T14:44:13.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): my global_id is 4801 2024-09-16T14:44:13.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf80a68f0 con 0x7fbcf80a1d10 2024-09-16T14:44:13.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2163564837 0 0) 0x7fbcdc003530 con 0x7fbcf80a2800 2024-09-16T14:44:13.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbcf813a3e0 con 0x7fbcf80a2800 2024-09-16T14:44:13.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbcf0001e80 con 0x7fbcf80a2800 2024-09-16T14:44:13.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:13.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): dump: 2024-09-16T14:44:13.684 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.684 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.684 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.684 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.684 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.684 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.684 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.684 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.684 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.684 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:13.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 405597005 0 0) 0x7fbcf813cd30 con 0x7fbcf80a0000 2024-09-16T14:44:13.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbcdc003530 con 0x7fbcf80a0000 2024-09-16T14:44:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbcf40018e0 con 0x7fbcf80a0000 2024-09-16T14:44:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): dump: 2024-09-16T14:44:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.685 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.686 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.686 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1777547043 0 0) 0x7fbcf813a3e0 con 0x7fbcf80a2800 2024-09-16T14:44:13.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 >> v1:172.21.15.26:6790/0 conn(0x7fbcf80a0000 legacy=0x7fbcf81395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 >> v1:172.21.15.189:6789/0 conn(0x7fbcf80a1d10 legacy=0x7fbcf8139cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:44:13.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 monclient: found mon.a 2024-09-16T14:44:13.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:13.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:13.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf809e500 con 0x7fbcf80a2800 2024-09-16T14:44:13.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient: _finish_auth 0 2024-09-16T14:44:13.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:44:13.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:13.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 5 monclient: authenticate success, global_id 84162 2024-09-16T14:44:13.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 10 monclient: _renew_subs 2024-09-16T14:44:13.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:13.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 1 -- 172.21.15.26:0/3447627388 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf813d2f0 con 0x7fbcf80a2800 2024-09-16T14:44:13.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 10 monclient: _renew_subs 2024-09-16T14:44:13.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:13.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 1 -- 172.21.15.26:0/3447627388 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf813af20 con 0x7fbcf80a2800 2024-09-16T14:44:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcf0001480 con 0x7fbcf80a2800 2024-09-16T14:44:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbcf0003dd0 con 0x7fbcf80a2800 2024-09-16T14:44:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbceffff640 4 set_mon_vals no callback set 2024-09-16T14:44:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient: dump: 2024-09-16T14:44:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.688 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.689 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcffa90640 1 -- 172.21.15.26:0/3447627388 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbce0006590 con 0x7fbce0002830 2024-09-16T14:44:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbcf000e840 con 0x7fbcf80a2800 2024-09-16T14:44:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(39..39 src has 1..39) v4 ==== 3940+0+0 (unknown 2059123099 0 0) 0x7fbcf0042310 con 0x7fbcf80a2800 2024-09-16T14:44:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424741159 0 0) 0x7fbce0006590 con 0x7fbce0002830 2024-09-16T14:44:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient(hunting): my global_id is 4778 2024-09-16T14:44:13.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf813a3e0 con 0x7fbce0002830 2024-09-16T14:44:13.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcfd02b640 10 monclient: get_auth_request con 0x7fbcdc0363d0 auth_method 0 2024-09-16T14:44:13.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 959503428 0 0) 0x7fbcf813a3e0 con 0x7fbce0002830 2024-09-16T14:44:13.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbce0006590 con 0x7fbce0002830 2024-09-16T14:44:13.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbcf4004990 con 0x7fbce0002830 2024-09-16T14:44:13.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:13.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient: dump: 2024-09-16T14:44:13.690 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.690 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.690 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.690 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.690 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.691 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.691 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.691 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.691 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.691 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2132614667 0 0) 0x7fbce0006590 con 0x7fbce0002830 2024-09-16T14:44:13.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbce000a0e0 con 0x7fbce0002830 2024-09-16T14:44:13.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fbcf40058e0 con 0x7fbce0002830 2024-09-16T14:44:13.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:13.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:13.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 >> v1:172.21.15.26:6790/0 conn(0x7fbce0002830 legacy=0x7fbce0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.666+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 >> v1:172.21.15.26:6790/0 conn(0x7fbce0002830 legacy=0x7fbce0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:13.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcffa90640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:44:13.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcffa90640 1 -- 172.21.15.26:0/3447627388 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbce0009d80 con 0x7fbce00090a0 2024-09-16T14:44:13.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 840355368 0 0) 0x7fbce0009d80 con 0x7fbce00090a0 2024-09-16T14:44:13.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 10 monclient(hunting): my global_id is 4781 2024-09-16T14:44:13.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbce0006590 con 0x7fbce00090a0 2024-09-16T14:44:13.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3044232959 0 0) 0x7fbce0006590 con 0x7fbce00090a0 2024-09-16T14:44:13.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbce0009d80 con 0x7fbce00090a0 2024-09-16T14:44:13.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbcf4004fd0 con 0x7fbce00090a0 2024-09-16T14:44:13.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:13.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 10 monclient: dump: 2024-09-16T14:44:13.693 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.693 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.693 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.693 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.693 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.693 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.693 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.693 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.693 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.693 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1399800363 0 0) 0x7fbce0009d80 con 0x7fbce00090a0 2024-09-16T14:44:13.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbce000c290 con 0x7fbce00090a0 2024-09-16T14:44:13.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 3556349788) 0x7fbcf4005a50 con 0x7fbce00090a0 2024-09-16T14:44:13.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:44:13.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:13.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 >> v1:172.21.15.26:6790/0 conn(0x7fbce00090a0 legacy=0x7fbce0009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcee7fc640 1 -- 172.21.15.26:0/3447627388 >> v1:172.21.15.26:6790/0 conn(0x7fbce00090a0 legacy=0x7fbce0009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:13.694 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:44:13.694 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:44:13.694 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:44:13.694 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:44:13.695 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 22, 2024-09-16T14:44:13.695 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:44:13.695 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:44:13.695 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:44:13.695 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:44:13.695 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:13.695 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:44:13.695 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:44:13.695 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:44:13.695 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:44:13.695 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:13.695 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:13.696 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:13.696 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:13.696 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:13.696 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:13.696 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:13.696 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:13.696 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:13.696 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:13.696 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:13.696 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:44:13.696 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:44:13.696 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:13.696 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:13.697 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:13.697 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:13.697 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:13.697 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:13.697 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:13.697 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:13.697 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:13.697 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:13.697 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.697 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:44:13.697 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:44:13.697 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:44:13.697 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:44:13.698 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:44:13.698 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:44:13.698 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:44:13.698 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:44:13.698 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:44:13.698 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:44:13.698 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:44:13.698 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:44:13.698 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:44:13.698 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:44:13.698 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:44:13.698 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:44:13.698 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:44:13.699 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:13.699 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:13.699 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:13.699 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:13.699 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:13.699 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:13.699 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:13.699 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:13.699 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:13.699 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:13.699 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:13.699 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:44:13.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.700 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:44:13.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.700 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:44:13.700 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:44:13.700 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:13.700 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:13.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.700 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:13.700 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:44:13.700 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:13.700 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.700 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.701 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:44:13.701 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:44:13.701 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:13.701 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:13.701 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:13.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.701 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:44:13.701 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:44:13.701 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:13.701 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:13.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.701 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:13.702 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:44:13.702 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:13.702 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.702 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.702 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:44:13.702 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:44:13.702 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:13.702 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:13.702 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:13.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.702 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:44:13.703 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:44:13.703 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:13.703 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:13.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.703 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:13.703 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:44:13.703 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:13.703 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.703 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.703 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:44:13.703 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:44:13.703 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:13.703 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:13.704 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:13.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.704 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.704 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:44:13.704 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:44:13.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.704 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:13.704 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:13.704 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:44:13.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.704 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:13.704 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:44:13.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.705 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:13.705 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:13.705 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:44:13.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.705 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:13.705 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:44:13.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.705 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:13.705 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:13.705 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:44:13.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.705 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.706 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:44:13.706 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:44:13.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcffa90640 1 -- 172.21.15.26:0/3447627388 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbcdc0363d0 legacy=0x7fbcdc038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcffa90640 10 monclient: shutdown 2024-09-16T14:44:13.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcffa90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:13.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcffa90640 1 -- 172.21.15.26:0/3447627388 >> v1:172.21.15.26:6789/0 conn(0x7fbcf80a2800 legacy=0x7fbcf81d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcfe02d640 1 -- 172.21.15.26:0/3447627388 reap_dead start 2024-09-16T14:44:13.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcfe02d640 1 -- 172.21.15.26:0/3447627388 reap_dead start 2024-09-16T14:44:13.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcffa90640 1 -- 172.21.15.26:0/3447627388 shutdown_connections 2024-09-16T14:44:13.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcffa90640 1 -- 172.21.15.26:0/3447627388 >> 172.21.15.26:0/3447627388 conn(0x7fbcf80a7f30 msgr2=0x7fbcf80a5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:13.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcffa90640 1 -- 172.21.15.26:0/3447627388 shutdown_connections 2024-09-16T14:44:13.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.674+0000 7fbcffa90640 1 -- 172.21.15.26:0/3447627388 wait complete. 2024-09-16T14:44:13.708 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:44:13.708 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:44:13.708 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:44:13.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.738+0000 7f580fe54640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:13.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.738+0000 7f580fe54640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:13.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.738+0000 7f580fe54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:44:13.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.738+0000 7f580fe54640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f57f0005880 con 0x7f580809c550 2024-09-16T14:44:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 10 monclient: get_monmap_and_config 2024-09-16T14:44:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 10 monclient: build_initial_monmap 2024-09-16T14:44:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 10 monclient: monmap: 2024-09-16T14:44:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:44:13.825561+0000 2024-09-16T14:44:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:44:13.825561+0000 2024-09-16T14:44:13.826 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:13.826 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.826 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:13.826 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:13.826 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:13.826 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 1 Processor -- start 2024-09-16T14:44:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 1 -- start start 2024-09-16T14:44:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 10 monclient: init 2024-09-16T14:44:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2e40a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2e4099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2e40a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e4098ea0 con 0x7fb2e40a1c00 2024-09-16T14:44:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e40997b0 con 0x7fb2e40a26f0 2024-09-16T14:44:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e4099460 con 0x7fb2e4099b60 2024-09-16T14:44:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 10 monclient(hunting): authenticate will time out at 1077.945611s 2024-09-16T14:44:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e3fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb2e40a1c00 0x7fb2e40a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58250/0 (socket says 172.21.15.26:58250) 2024-09-16T14:44:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e3fff640 1 -- 172.21.15.26:0/2015180066 learned_addr learned my addr 172.21.15.26:0/2015180066 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e37fe640 1 --1- 172.21.15.26:0/2015180066 >> v1:172.21.15.26:6790/0 conn(0x7fb2e4099b60 0x7fb2e4139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59844/0 (socket says 172.21.15.26:59844) 2024-09-16T14:44:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 1 -- 172.21.15.26:0/2015180066 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488849872 0 0) 0x7fb2e4098ea0 con 0x7fb2e40a1c00 2024-09-16T14:44:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient(hunting): my global_id is 84165 2024-09-16T14:44:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 1 -- 172.21.15.26:0/2015180066 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2cc003530 con 0x7fb2e40a1c00 2024-09-16T14:44:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 1 -- 172.21.15.26:0/2015180066 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917556232 0 0) 0x7fb2e4099460 con 0x7fb2e4099b60 2024-09-16T14:44:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient(hunting): my global_id is 4784 2024-09-16T14:44:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 1 -- 172.21.15.26:0/2015180066 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e4098ea0 con 0x7fb2e4099b60 2024-09-16T14:44:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 1 -- 172.21.15.26:0/2015180066 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3784294514 0 0) 0x7fb2cc003530 con 0x7fb2e40a1c00 2024-09-16T14:44:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 1 -- 172.21.15.26:0/2015180066 >> v1:172.21.15.26:6790/0 conn(0x7fb2e4099b60 legacy=0x7fb2e4139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 1 -- 172.21.15.26:0/2015180066 >> v1:172.21.15.189:6789/0 conn(0x7fb2e40a26f0 legacy=0x7fb2e413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 1 monclient: found mon.noname-a 2024-09-16T14:44:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 1 -- 172.21.15.26:0/2015180066 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e40a34b0 con 0x7fb2e40a1c00 2024-09-16T14:44:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:44:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 1 -- 172.21.15.26:0/2015180066 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb2d4001620 con 0x7fb2e40a1c00 2024-09-16T14:44:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient: dump: 2024-09-16T14:44:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.837 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.837 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.837 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.837 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.837 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 5 monclient: authenticate success, global_id 84165 2024-09-16T14:44:13.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:13.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 1 -- 172.21.15.26:0/2015180066 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2d4001840 con 0x7fb2e40a1c00 2024-09-16T14:44:13.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:13.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 1 -- 172.21.15.26:0/2015180066 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb2d4001c10 con 0x7fb2e40a1c00 2024-09-16T14:44:13.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:13.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2e2ffd640 10 monclient: dump: 2024-09-16T14:44:13.838 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.838 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.838 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.838 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.838 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.838 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.838 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.838 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.838 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.838 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:13.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 4 set_mon_vals no callback set 2024-09-16T14:44:13.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 10 monclient: shutdown 2024-09-16T14:44:13.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 1 -- 172.21.15.26:0/2015180066 >> v1:172.21.15.26:6789/0 conn(0x7fb2e40a1c00 legacy=0x7fb2e40a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 1 -- 172.21.15.26:0/2015180066 shutdown_connections 2024-09-16T14:44:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 1 -- 172.21.15.26:0/2015180066 >> 172.21.15.26:0/2015180066 conn(0x7fb2e40a3d00 msgr2=0x7fb2e40a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 1 -- 172.21.15.26:0/2015180066 shutdown_connections 2024-09-16T14:44:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 1 -- 172.21.15.26:0/2015180066 wait complete. 2024-09-16T14:44:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 10 monclient: build_initial_monmap 2024-09-16T14:44:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.822+0000 7fb2ea35d640 10 monclient: monmap: 2024-09-16T14:44:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:44:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:44:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:44:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:13.840 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 1 Processor -- start 2024-09-16T14:44:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 1 -- start start 2024-09-16T14:44:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 10 monclient: init 2024-09-16T14:44:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:44:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2e4099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2e40a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2e40a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e409ec60 con 0x7fb2e40a26f0 2024-09-16T14:44:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e409e3f0 con 0x7fb2e4099b60 2024-09-16T14:44:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e409f220 con 0x7fb2e40a1c00 2024-09-16T14:44:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 10 monclient(hunting): authenticate will time out at 1077.948702s 2024-09-16T14:44:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e37fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb2e40a1c00 0x7fb2e41ce290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59850/0 (socket says 172.21.15.26:59850) 2024-09-16T14:44:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e37fe640 1 -- 172.21.15.26:0/1539441892 learned_addr learned my addr 172.21.15.26:0/1539441892 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538411982 0 0) 0x7fb2d8001240 con 0x7fb2e40a1c00 2024-09-16T14:44:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): my global_id is 4787 2024-09-16T14:44:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d0003530 con 0x7fb2e40a1c00 2024-09-16T14:44:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507368655 0 0) 0x7fb2e409ec60 con 0x7fb2e40a26f0 2024-09-16T14:44:13.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): my global_id is 84168 2024-09-16T14:44:13.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d8001240 con 0x7fb2e40a26f0 2024-09-16T14:44:13.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2211604923 0 0) 0x7fb2d0003530 con 0x7fb2e40a1c00 2024-09-16T14:44:13.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2e409ec60 con 0x7fb2e40a1c00 2024-09-16T14:44:13.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb2d80018e0 con 0x7fb2e40a1c00 2024-09-16T14:44:13.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:13.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): dump: 2024-09-16T14:44:13.842 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.842 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.843 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.843 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.843 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.843 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.843 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.843 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.843 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.843 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:13.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2251018575 0 0) 0x7fb2d8001240 con 0x7fb2e40a26f0 2024-09-16T14:44:13.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2d0003530 con 0x7fb2e40a26f0 2024-09-16T14:44:13.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393798157 0 0) 0x7fb2e409e3f0 con 0x7fb2e4099b60 2024-09-16T14:44:13.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): my global_id is 4807 2024-09-16T14:44:13.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d8001240 con 0x7fb2e4099b60 2024-09-16T14:44:13.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb2dc001b40 con 0x7fb2e40a26f0 2024-09-16T14:44:13.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:13.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): dump: 2024-09-16T14:44:13.844 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.844 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.844 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.844 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.844 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.845 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.845 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.845 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.845 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.845 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2259078969 0 0) 0x7fb2e409ec60 con 0x7fb2e40a1c00 2024-09-16T14:44:13.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 >> v1:172.21.15.189:6789/0 conn(0x7fb2e4099b60 legacy=0x7fb2e4138180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 >> v1:172.21.15.26:6789/0 conn(0x7fb2e40a26f0 legacy=0x7fb2e41d19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:44:13.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 monclient: found mon.c 2024-09-16T14:44:13.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:13.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:13.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e4138890 con 0x7fb2e40a1c00 2024-09-16T14:44:13.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient: _finish_auth 0 2024-09-16T14:44:13.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:44:13.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:13.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 5 monclient: authenticate success, global_id 4787 2024-09-16T14:44:13.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 10 monclient: _renew_subs 2024-09-16T14:44:13.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:13.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 1 -- 172.21.15.26:0/1539441892 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e411fae0 con 0x7fb2e40a1c00 2024-09-16T14:44:13.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 10 monclient: _renew_subs 2024-09-16T14:44:13.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:13.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 1 -- 172.21.15.26:0/1539441892 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e411f800 con 0x7fb2e40a1c00 2024-09-16T14:44:13.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2d80047f0 con 0x7fb2e40a1c00 2024-09-16T14:44:13.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:13.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb2d8004b00 con 0x7fb2e40a1c00 2024-09-16T14:44:13.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:13.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e27fc640 4 set_mon_vals no callback set 2024-09-16T14:44:13.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient: dump: 2024-09-16T14:44:13.847 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.847 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.847 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.847 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.847 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.847 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.847 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.847 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.847 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.848 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:13.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2ea35d640 1 -- 172.21.15.26:0/1539441892 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb2c4006590 con 0x7fb2c4002830 2024-09-16T14:44:13.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb2d800f570 con 0x7fb2e40a1c00 2024-09-16T14:44:13.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(39..39 src has 1..39) v4 ==== 3940+0+0 (unknown 2059123099 0 0) 0x7fb2d8043180 con 0x7fb2e40a1c00 2024-09-16T14:44:13.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e88fa640 10 monclient: get_auth_request con 0x7fb2d00363d0 auth_method 0 2024-09-16T14:44:13.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028937717 0 0) 0x7fb2c4006590 con 0x7fb2c4002830 2024-09-16T14:44:13.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient(hunting): my global_id is 4810 2024-09-16T14:44:13.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e409ec60 con 0x7fb2c4002830 2024-09-16T14:44:13.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1968811217 0 0) 0x7fb2e409ec60 con 0x7fb2c4002830 2024-09-16T14:44:13.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2c4006590 con 0x7fb2c4002830 2024-09-16T14:44:13.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb2d40019b0 con 0x7fb2c4002830 2024-09-16T14:44:13.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:13.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.826+0000 7fb2e0ff9640 10 monclient: dump: 2024-09-16T14:44:13.849 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.849 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.849 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.849 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.849 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.849 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.849 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.849 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.850 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.850 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.830+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1927004564 0 0) 0x7fb2c4006590 con 0x7fb2c4002830 2024-09-16T14:44:13.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.830+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2c400a0e0 con 0x7fb2c4002830 2024-09-16T14:44:13.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.830+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fb2d4001c40 con 0x7fb2c4002830 2024-09-16T14:44:13.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.830+0000 7fb2e0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:13.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.830+0000 7fb2e0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:13.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.830+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 >> v1:172.21.15.189:6789/0 conn(0x7fb2c4002830 legacy=0x7fb2c4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.830+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 >> v1:172.21.15.189:6789/0 conn(0x7fb2c4002830 legacy=0x7fb2c4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:13.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.834+0000 7fb2ea35d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:44:13.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.834+0000 7fb2ea35d640 1 -- 172.21.15.26:0/1539441892 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb2c4009da0 con 0x7fb2c40090c0 2024-09-16T14:44:13.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189971780 0 0) 0x7fb2c4009da0 con 0x7fb2c40090c0 2024-09-16T14:44:13.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:13.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 10 monclient(hunting): my global_id is 4813 2024-09-16T14:44:13.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c4006590 con 0x7fb2c40090c0 2024-09-16T14:44:13.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2184362453 0 0) 0x7fb2c4006590 con 0x7fb2c40090c0 2024-09-16T14:44:13.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2c4009da0 con 0x7fb2c40090c0 2024-09-16T14:44:13.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb2d4004b80 con 0x7fb2c40090c0 2024-09-16T14:44:13.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:44:13.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 10 monclient: dump: 2024-09-16T14:44:13.853 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:13.853 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.853 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.853 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.853 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.853 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:13.853 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.853 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.853 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.853 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:13.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1621002146 0 0) 0x7fb2c4009da0 con 0x7fb2c40090c0 2024-09-16T14:44:13.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb2c400c290 con 0x7fb2c40090c0 2024-09-16T14:44:13.854 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:44:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:44:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:44:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:44:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 22, 2024-09-16T14:44:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:44:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:44:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:44:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:44:13.854 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-09-16T14:44:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:44:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:44:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:44:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:13.855 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:44:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:44:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:13.856 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:44:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:44:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:44:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:44:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:44:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:44:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:44:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:44:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:44:13.857 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:44:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:44:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:44:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:44:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:44:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:44:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:44:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:44:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:13.858 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:44:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:44:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:44:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:44:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.859 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:44:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:44:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:44:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.860 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:44:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:44:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:44:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:44:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:44:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:13.861 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:44:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:44:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:44:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:44:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:44:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:44:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:44:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:13.863 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:44:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:44:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:44:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:44:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:13.864 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:44:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:44:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:44:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:13.865 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:44:13.865 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:44:13.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 3949501781) 0x7fb2d4004dd0 con 0x7fb2c40090c0 2024-09-16T14:44:13.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:44:13.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:13.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 >> v1:172.21.15.189:6789/0 conn(0x7fb2c40090c0 legacy=0x7fb2c4009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e0ff9640 1 -- 172.21.15.26:0/1539441892 >> v1:172.21.15.189:6789/0 conn(0x7fb2c40090c0 legacy=0x7fb2c4009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:13.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2ea35d640 1 -- 172.21.15.26:0/1539441892 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb2d00363d0 legacy=0x7fb2d0038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2ea35d640 10 monclient: shutdown 2024-09-16T14:44:13.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2ea35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:13.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2ea35d640 1 -- 172.21.15.26:0/1539441892 >> v1:172.21.15.26:6790/0 conn(0x7fb2e40a1c00 legacy=0x7fb2e41ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e88fa640 1 -- 172.21.15.26:0/1539441892 reap_dead start 2024-09-16T14:44:13.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2e88fa640 1 -- 172.21.15.26:0/1539441892 reap_dead start 2024-09-16T14:44:13.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2ea35d640 1 -- 172.21.15.26:0/1539441892 shutdown_connections 2024-09-16T14:44:13.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2ea35d640 1 -- 172.21.15.26:0/1539441892 >> 172.21.15.26:0/1539441892 conn(0x7fb2e40a3d00 msgr2=0x7fb2e40a02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:13.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2ea35d640 1 -- 172.21.15.26:0/1539441892 shutdown_connections 2024-09-16T14:44:13.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:13.838+0000 7fb2ea35d640 1 -- 172.21.15.26:0/1539441892 wait complete. 2024-09-16T14:44:13.868 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:44:13.868 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T14:44:13.868 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-16T14:44:13.868 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T14:44:13.868 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:44:13.868 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T14:44:13.869 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:44:13.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f580d3ef640 1 -- 172.21.15.189:0/1563518999 >> v1:172.21.15.189:6789/0 conn(0x7f580809c550 legacy=0x7f580809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T14:44:13.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f580d3ef640 1 -- 172.21.15.189:0/1563518999 >> v1:172.21.15.189:6789/0 conn(0x7f580809c550 legacy=0x7f580809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:44:13.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f580d3ef640 1 --1- 172.21.15.189:0/1563518999 >> v1:172.21.15.189:6789/0 conn(0x7f580809c550 0x7f580809a930 :-1 s=OPENED pgs=282 cs=1 l=1).handle_message read tag failed 2024-09-16T14:44:13.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f580d3ef640 1 --1- 172.21.15.189:0/1563518999 >> v1:172.21.15.189:6789/0 conn(0x7f580809c550 0x7f580809a930 :-1 s=OPENED pgs=282 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:44:13.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.189:6789/0 2024-09-16T14:44:13.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:13.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 >> v1:172.21.15.189:6789/0 conn(0x7f580809c550 legacy=0x7f580809a930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:13.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:13.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient(hunting): picked mon.a con 0x7f580809e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:13.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient(hunting): picked mon.b con 0x7f57ec0429c0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:13.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient(hunting): picked mon.c con 0x7f5808099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:13.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f580806cf00 con 0x7f580809e6d0 2024-09-16T14:44:13.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f580809be20 con 0x7f57ec0429c0 2024-09-16T14:44:13.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f580809cf20 con 0x7f5808099a70 2024-09-16T14:44:13.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:13.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180779882 0 0) 0x7f580809cf20 con 0x7f5808099a70 2024-09-16T14:44:13.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:44:13.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f580e3f1640 1 -- 172.21.15.189:0/1563518999 >> v1:172.21.15.189:6789/0 conn(0x7f57ec0429c0 legacy=0x7f57ec044e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:13.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.26:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f57ec04a5f0 con 0x7f5808099a70 2024-09-16T14:44:13.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535548353 0 0) 0x7f580806cf00 con 0x7f580809e6d0 2024-09-16T14:44:13.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:13.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:13.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:44:13.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.26:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f580809cf20 con 0x7f580809e6d0 2024-09-16T14:44:13.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 3326642691 0 0) 0x7f57ec04a5f0 con 0x7f5808099a70 2024-09-16T14:44:13.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 >> v1:172.21.15.189:6789/0 conn(0x7f57ec0429c0 legacy=0x7f57ec044e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:13.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 >> v1:172.21.15.26:6789/0 conn(0x7f580809e6d0 legacy=0x7f57ec0410a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:13.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:44:13.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 1 monclient: found mon.c 2024-09-16T14:44:13.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:13.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:13.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f57ec04d680 con 0x7f5808099a70 2024-09-16T14:44:13.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:13.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:13.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f580809be20 con 0x7f5808099a70 2024-09-16T14:44:13.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient: _finish_auth 0 2024-09-16T14:44:13.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:44:13.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:13.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5804005c90 con 0x7f5808099a70 2024-09-16T14:44:13.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:13.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:13.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.874+0000 7f57fe7fc640 10 monclient: dump: 2024-09-16T14:44:13.883 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:13.883 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:13.883 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.883 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:13.883 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:13.883 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:13.883 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:13.883 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:13.884 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:13.884 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:13.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.878+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5804005af0 con 0x7f5808099a70 2024-09-16T14:44:13.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.878+0000 7f57fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:13.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.878+0000 7f57fffff640 4 set_mon_vals no callback set 2024-09-16T14:44:13.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:13.878+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5804005eb0 con 0x7f5808099a70 2024-09-16T14:44:13.889 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T14:44:13.890 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T14:44:13.890 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T14:44:13.890 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T14:44:16.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:16.574+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:16.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:16.574+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:16.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:16.574+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:16.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:16.574+0000 7f57fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:17.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:17.578+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:17.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:17.578+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:17.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:17.578+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:18.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:18.574+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:18.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:18.574+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:18.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:18.574+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:19.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:19.574+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:19.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:19.574+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:19.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:19.574+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:20.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:20.574+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:20.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:20.574+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:20.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:20.574+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:21.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:21.574+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:21.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:21.574+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:21.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:21.574+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:22.432 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T14:44:22.426+0000 7f9d20a17640 -1 calc_pg_upmaps abort due to max <= 0 2024-09-16T14:44:22.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:22.574+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:22.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:22.574+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:22.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:22.574+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:23.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:23.574+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:23.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:23.574+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:23.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:23.574+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:24.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:24.578+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:24.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:24.578+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:24.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:24.578+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:25.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:25.574+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:25.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:25.574+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:25.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:25.574+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:26.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:26.578+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:26.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:26.578+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:26.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:26.578+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:26.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:26.578+0000 7f57fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:27.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:27.574+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:27.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:27.574+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:27.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:27.574+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:28.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:28.574+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:28.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:28.574+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:28.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:28.574+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:28.891 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T14:44:28.891 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T14:44:28.892 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:44:28.892 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:44:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb757326640 10 monclient: get_monmap_and_config 2024-09-16T14:44:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb757326640 10 monclient: build_initial_monmap 2024-09-16T14:44:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb757326640 10 monclient: monmap: 2024-09-16T14:44:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:44:29.026394+0000 2024-09-16T14:44:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:44:29.026394+0000 2024-09-16T14:44:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:29.027 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:29.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb757326640 1 Processor -- start 2024-09-16T14:44:29.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb757326640 1 -- start start 2024-09-16T14:44:29.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb757326640 10 monclient: init 2024-09-16T14:44:29.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb757326640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb757326640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb757326640 10 monclient(hunting): picked mon.noname-a con 0x7fb7500a0050 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb757326640 10 monclient(hunting): picked mon.noname-c con 0x7fb7500a0b40 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb757326640 10 monclient(hunting): picked mon.noname-b con 0x7fb7500a0f50 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb757326640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75006cb40 con 0x7fb7500a0050 2024-09-16T14:44:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb757326640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75006c210 con 0x7fb7500a0b40 2024-09-16T14:44:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb757326640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75006c4f0 con 0x7fb7500a0f50 2024-09-16T14:44:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb757326640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb757326640 10 monclient(hunting): authenticate will time out at 1093.146965s 2024-09-16T14:44:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb7558c3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb7500a0f50 0x7fb75013d480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:37852/0 (socket says 172.21.15.26:37852) 2024-09-16T14:44:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb7558c3640 1 -- 172.21.15.26:0/1709462664 learned_addr learned my addr 172.21.15.26:0/1709462664 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.022+0000 7fb7548c1640 1 -- 172.21.15.26:0/1709462664 >> v1:172.21.15.189:6789/0 conn(0x7fb7500a0b40 legacy=0x7fb750139d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:29.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.226+0000 7fb7548c1640 1 -- 172.21.15.26:0/1709462664 >> v1:172.21.15.189:6789/0 conn(0x7fb7500a0b40 legacy=0x7fb750139d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:29.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:29.578+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:29.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:29.578+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:29.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:29.578+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:29.630+0000 7fb7548c1640 1 -- 172.21.15.26:0/1709462664 >> v1:172.21.15.189:6789/0 conn(0x7fb7500a0b40 legacy=0x7fb750139d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:30.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:30.434+0000 7fb7548c1640 1 -- 172.21.15.26:0/1709462664 >> v1:172.21.15.189:6789/0 conn(0x7fb7500a0b40 legacy=0x7fb750139d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:30.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:30.578+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:30.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:30.578+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:30.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:30.578+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:31.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:31.578+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:31.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:31.578+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:31.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:31.578+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:32.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.026+0000 7fb746ffd640 10 monclient(hunting): tick 2024-09-16T14:44:32.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.026+0000 7fb746ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:44:32.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.026+0000 7fb746ffd640 1 monclient(hunting): continuing hunt 2024-09-16T14:44:32.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.026+0000 7fb746ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:44:32.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.026+0000 7fb746ffd640 1 -- 172.21.15.26:0/1709462664 >> v1:172.21.15.26:6790/0 conn(0x7fb7500a0f50 legacy=0x7fb75013d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:32.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.026+0000 7fb746ffd640 1 -- 172.21.15.26:0/1709462664 >> v1:172.21.15.189:6789/0 conn(0x7fb7500a0b40 legacy=0x7fb750139d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:32.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.026+0000 7fb746ffd640 1 -- 172.21.15.26:0/1709462664 >> v1:172.21.15.26:6789/0 conn(0x7fb7500a0050 legacy=0x7fb7500a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:32.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.026+0000 7fb746ffd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:44:32.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.026+0000 7fb746ffd640 10 monclient(hunting): picked mon.noname-b con 0x7fb7340036d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:32.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.026+0000 7fb746ffd640 10 monclient(hunting): picked mon.noname-c con 0x7fb7500a0b40 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:32.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.026+0000 7fb746ffd640 10 monclient(hunting): picked mon.noname-a con 0x7fb73400aaf0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:32.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.026+0000 7fb746ffd640 1 -- 172.21.15.26:0/1709462664 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75006c7d0 con 0x7fb73400aaf0 2024-09-16T14:44:32.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.026+0000 7fb746ffd640 1 -- 172.21.15.26:0/1709462664 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75006c210 con 0x7fb7500a0b40 2024-09-16T14:44:32.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.026+0000 7fb746ffd640 1 -- 172.21.15.26:0/1709462664 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7340108a0 con 0x7fb7340036d0 2024-09-16T14:44:32.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.026+0000 7fb746ffd640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:32.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.026+0000 7fb7548c1640 1 -- 172.21.15.26:0/1709462664 >> v1:172.21.15.189:6789/0 conn(0x7fb7500a0b40 legacy=0x7fb734009380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:32.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.230+0000 7fb7548c1640 1 -- 172.21.15.26:0/1709462664 >> v1:172.21.15.189:6789/0 conn(0x7fb7500a0b40 legacy=0x7fb734009380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:32.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:32.573+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:32.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:32.573+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:32.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:32.573+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:32.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:32.634+0000 7fb7548c1640 1 -- 172.21.15.26:0/1709462664 >> v1:172.21.15.189:6789/0 conn(0x7fb7500a0b40 legacy=0x7fb734009380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:33.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.438+0000 7fb7548c1640 1 -- 172.21.15.26:0/1709462664 >> v1:172.21.15.189:6789/0 conn(0x7fb7500a0b40 legacy=0x7fb734009380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:33.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:33.573+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:33.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:33.573+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:33.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:33.573+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:33.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 1 -- 172.21.15.26:0/1709462664 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502554437 0 0) 0x7fb75006c7d0 con 0x7fb73400aaf0 2024-09-16T14:44:33.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:33.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:33.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:33.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 10 monclient(hunting): my global_id is 84171 2024-09-16T14:44:33.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 1 -- 172.21.15.26:0/1709462664 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb73c003530 con 0x7fb73400aaf0 2024-09-16T14:44:33.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 1 -- 172.21.15.26:0/1709462664 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3660311345 0 0) 0x7fb73c003530 con 0x7fb73400aaf0 2024-09-16T14:44:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 1 -- 172.21.15.26:0/1709462664 >> v1:172.21.15.26:6790/0 conn(0x7fb7340036d0 legacy=0x7fb734005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 1 -- 172.21.15.26:0/1709462664 >> v1:172.21.15.189:6789/0 conn(0x7fb7500a0b40 legacy=0x7fb734009380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 10 monclient: _finish_hunting 0 2024-09-16T14:44:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 1 monclient: found mon.noname-a 2024-09-16T14:44:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 1 -- 172.21.15.26:0/1709462664 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb734011b10 con 0x7fb73400aaf0 2024-09-16T14:44:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 10 monclient: _finish_auth 0 2024-09-16T14:44:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 10 monclient: _check_auth_tickets 2024-09-16T14:44:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 1 -- 172.21.15.26:0/1709462664 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb7480018a0 con 0x7fb73400aaf0 2024-09-16T14:44:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb757326640 5 monclient: authenticate success, global_id 84171 2024-09-16T14:44:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb757326640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 10 monclient: dump: 2024-09-16T14:44:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:33.921 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb757326640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 1 -- 172.21.15.26:0/1709462664 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb748001ac0 con 0x7fb73400aaf0 2024-09-16T14:44:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 1 -- 172.21.15.26:0/1709462664 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb748001e50 con 0x7fb73400aaf0 2024-09-16T14:44:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb747fff640 10 monclient: dump: 2024-09-16T14:44:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:33.922 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb757326640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.910+0000 7fb757326640 4 set_mon_vals no callback set 2024-09-16T14:44:33.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 10 monclient: shutdown 2024-09-16T14:44:33.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:33.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 1 -- 172.21.15.26:0/1709462664 >> v1:172.21.15.26:6789/0 conn(0x7fb73400aaf0 legacy=0x7fb73400cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 1 -- 172.21.15.26:0/1709462664 shutdown_connections 2024-09-16T14:44:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 1 -- 172.21.15.26:0/1709462664 >> 172.21.15.26:0/1709462664 conn(0x7fb7500a5540 msgr2=0x7fb7500a5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 1 -- 172.21.15.26:0/1709462664 shutdown_connections 2024-09-16T14:44:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 1 -- 172.21.15.26:0/1709462664 wait complete. 2024-09-16T14:44:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 10 monclient: build_initial_monmap 2024-09-16T14:44:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 10 monclient: monmap: 2024-09-16T14:44:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:44:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:44:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:44:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:33.930 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 1 Processor -- start 2024-09-16T14:44:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 1 -- start start 2024-09-16T14:44:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 10 monclient: init 2024-09-16T14:44:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:44:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 10 monclient(hunting): picked mon.noname-c con 0x7fb7340036d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 10 monclient(hunting): picked mon.noname-b con 0x7fb73400aaf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 10 monclient(hunting): picked mon.noname-a con 0x7fb7500a0b40 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7501a91c0 con 0x7fb7500a0b40 2024-09-16T14:44:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7500a34e0 con 0x7fb73400aaf0 2024-09-16T14:44:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75009c360 con 0x7fb7340036d0 2024-09-16T14:44:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb757326640 10 monclient(hunting): authenticate will time out at 1098.039374s 2024-09-16T14:44:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb7548c1640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fb73400aaf0 legacy=0x7fb7501ce4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb7558c3640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb7500a0b40 0x7fb7501d1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53914/0 (socket says 172.21.15.26:53914) 2024-09-16T14:44:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb7558c3640 1 -- 172.21.15.26:0/3174985634 learned_addr learned my addr 172.21.15.26:0/3174985634 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.914+0000 7fb745ffb640 1 -- 172.21.15.26:0/3174985634 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080217162 0 0) 0x7fb7501a91c0 con 0x7fb7500a0b40 2024-09-16T14:44:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:33.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 10 monclient(hunting): my global_id is 84174 2024-09-16T14:44:33.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 1 -- 172.21.15.26:0/3174985634 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb73400ff90 con 0x7fb7500a0b40 2024-09-16T14:44:33.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 1 -- 172.21.15.26:0/3174985634 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 895219847 0 0) 0x7fb73400ff90 con 0x7fb7500a0b40 2024-09-16T14:44:33.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 1 -- 172.21.15.26:0/3174985634 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7501a91c0 con 0x7fb7500a0b40 2024-09-16T14:44:33.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 1 -- 172.21.15.26:0/3174985634 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb748001dd0 con 0x7fb7500a0b40 2024-09-16T14:44:33.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:33.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:33.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 10 monclient(hunting): dump: 2024-09-16T14:44:33.932 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:33.932 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:33.932 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:33.933 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 1 -- 172.21.15.26:0/3174985634 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 226028724 0 0) 0x7fb7501a91c0 con 0x7fb7500a0b40 2024-09-16T14:44:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 1 -- 172.21.15.26:0/3174985634 >> v1:172.21.15.26:6790/0 conn(0x7fb7340036d0 legacy=0x7fb7501bcf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 1 -- 172.21.15.26:0/3174985634 >> v1:172.21.15.189:6789/0 conn(0x7fb73400aaf0 legacy=0x7fb7501ce4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:44:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 1 monclient: found mon.a 2024-09-16T14:44:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 1 -- 172.21.15.26:0/3174985634 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7501aa600 con 0x7fb7500a0b40 2024-09-16T14:44:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 10 monclient: _finish_auth 0 2024-09-16T14:44:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:44:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb757326640 5 monclient: authenticate success, global_id 84174 2024-09-16T14:44:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb757326640 10 monclient: _renew_subs 2024-09-16T14:44:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb757326640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb757326640 1 -- 172.21.15.26:0/3174985634 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7501a8ed0 con 0x7fb7500a0b40 2024-09-16T14:44:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb757326640 10 monclient: _renew_subs 2024-09-16T14:44:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb757326640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb757326640 1 -- 172.21.15.26:0/3174985634 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7500a3a10 con 0x7fb7500a0b40 2024-09-16T14:44:33.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 1 -- 172.21.15.26:0/3174985634 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7480020d0 con 0x7fb7500a0b40 2024-09-16T14:44:33.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:33.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 1 -- 172.21.15.26:0/3174985634 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb748004fc0 con 0x7fb7500a0b40 2024-09-16T14:44:33.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:33.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:33.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 10 monclient: dump: 2024-09-16T14:44:33.935 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:33.935 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:33.935 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:33.935 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:33.935 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:33.935 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:33.935 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:33.936 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:33.936 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:33.936 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:33.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb7477fe640 4 set_mon_vals no callback set 2024-09-16T14:44:33.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb757326640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:33.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb757326640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:33.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb757326640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:33.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb757326640 1 -- 172.21.15.26:0/3174985634 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75009dc60 con 0x7fb7500a0b40 2024-09-16T14:44:33.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 1 -- 172.21.15.26:0/3174985634 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb74800fa30 con 0x7fb7500a0b40 2024-09-16T14:44:33.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb745ffb640 1 -- 172.21.15.26:0/3174985634 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(39..39 src has 1..39) v4 ==== 3940+0+0 (unknown 2059123099 0 0) 0x7fb748043440 con 0x7fb7500a0b40 2024-09-16T14:44:33.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.918+0000 7fb7550c2640 10 monclient: get_auth_request con 0x7fb734036f00 auth_method 0 2024-09-16T14:44:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.926+0000 7fb745ffb640 1 -- 172.21.15.26:0/3174985634 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb748014700 con 0x7fb7500a0b40 2024-09-16T14:44:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.926+0000 7fb745ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:33.926+0000 7fb745ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:34.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.146+0000 7fb757326640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:44:34.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.146+0000 7fb757326640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:44:34.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.146+0000 7fb757326640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:34.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.146+0000 7fb757326640 1 -- 172.21.15.26:0/3174985634 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb750072560 con 0x7fb7500a0b40 2024-09-16T14:44:34.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.146+0000 7fb745ffb640 1 -- 172.21.15.26:0/3174985634 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1834267509) 0x7fb748012da0 con 0x7fb7500a0b40 2024-09-16T14:44:34.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.146+0000 7fb745ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:44:34.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.146+0000 7fb745ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:34.153 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":26,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:44:34.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.150+0000 7fb757326640 1 -- 172.21.15.26:0/3174985634 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb734036f00 legacy=0x7fb7340393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:34.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.150+0000 7fb757326640 10 monclient: shutdown 2024-09-16T14:44:34.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.150+0000 7fb757326640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:34.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.150+0000 7fb757326640 1 -- 172.21.15.26:0/3174985634 >> v1:172.21.15.26:6789/0 conn(0x7fb7500a0b40 legacy=0x7fb7501d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:34.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.150+0000 7fb757326640 1 -- 172.21.15.26:0/3174985634 shutdown_connections 2024-09-16T14:44:34.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.150+0000 7fb757326640 1 -- 172.21.15.26:0/3174985634 >> 172.21.15.26:0/3174985634 conn(0x7fb7500a5540 msgr2=0x7fb750071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:34.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.150+0000 7fb757326640 1 -- 172.21.15.26:0/3174985634 shutdown_connections 2024-09-16T14:44:34.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.150+0000 7fb757326640 1 -- 172.21.15.26:0/3174985634 wait complete. 2024-09-16T14:44:34.170 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:44:34.170 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T14:44:34.170 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:44:34.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 10 monclient: get_monmap_and_config 2024-09-16T14:44:34.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 10 monclient: build_initial_monmap 2024-09-16T14:44:34.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 10 monclient: monmap: 2024-09-16T14:44:34.283 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:34.284 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:34.284 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:44:34.282442+0000 2024-09-16T14:44:34.284 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:44:34.282442+0000 2024-09-16T14:44:34.284 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:34.284 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:34.285 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:34.285 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:34.285 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:34.285 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 1 Processor -- start 2024-09-16T14:44:34.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 1 -- start start 2024-09-16T14:44:34.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 10 monclient: init 2024-09-16T14:44:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 10 monclient(hunting): picked mon.noname-a con 0x7ffb1c09a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 10 monclient(hunting): picked mon.noname-c con 0x7ffb1c09d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 10 monclient(hunting): picked mon.noname-b con 0x7ffb1c09b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c09dc60 con 0x7ffb1c09a550 2024-09-16T14:44:34.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c099ae0 con 0x7ffb1c09d1b0 2024-09-16T14:44:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c09b8f0 con 0x7ffb1c09b040 2024-09-16T14:44:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 10 monclient(hunting): authenticate will time out at 1098.402431s 2024-09-16T14:44:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb1affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ffb1c09a550 0x7ffb1c09a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53920/0 (socket says 172.21.15.26:53920) 2024-09-16T14:44:34.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb1affd640 1 -- 172.21.15.26:0/2923028375 learned_addr learned my addr 172.21.15.26:0/2923028375 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb1a7fc640 1 -- 172.21.15.26:0/2923028375 >> v1:172.21.15.189:6789/0 conn(0x7ffb1c09d1b0 legacy=0x7ffb1c136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 1 -- 172.21.15.26:0/2923028375 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352378663 0 0) 0x7ffb1c09dc60 con 0x7ffb1c09a550 2024-09-16T14:44:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:34.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient(hunting): my global_id is 84177 2024-09-16T14:44:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 1 -- 172.21.15.26:0/2923028375 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb04003530 con 0x7ffb1c09a550 2024-09-16T14:44:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 1 -- 172.21.15.26:0/2923028375 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062150320 0 0) 0x7ffb1c09b8f0 con 0x7ffb1c09b040 2024-09-16T14:44:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:34.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient(hunting): my global_id is 4790 2024-09-16T14:44:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 1 -- 172.21.15.26:0/2923028375 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb1c09dc60 con 0x7ffb1c09b040 2024-09-16T14:44:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 1 -- 172.21.15.26:0/2923028375 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1413864369 0 0) 0x7ffb04003530 con 0x7ffb1c09a550 2024-09-16T14:44:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 1 -- 172.21.15.26:0/2923028375 >> v1:172.21.15.26:6790/0 conn(0x7ffb1c09b040 legacy=0x7ffb1c13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 1 -- 172.21.15.26:0/2923028375 >> v1:172.21.15.189:6789/0 conn(0x7ffb1c09d1b0 legacy=0x7ffb1c136a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:44:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 1 monclient: found mon.noname-a 2024-09-16T14:44:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 1 -- 172.21.15.26:0/2923028375 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c09bbd0 con 0x7ffb1c09a550 2024-09-16T14:44:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient: _finish_auth 0 2024-09-16T14:44:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:44:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 5 monclient: authenticate success, global_id 84177 2024-09-16T14:44:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 1 -- 172.21.15.26:0/2923028375 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb10001620 con 0x7ffb1c09a550 2024-09-16T14:44:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient: dump: 2024-09-16T14:44:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.306 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:34.307 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 1 -- 172.21.15.26:0/2923028375 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb10001900 con 0x7ffb1c09a550 2024-09-16T14:44:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 1 -- 172.21.15.26:0/2923028375 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb10001c90 con 0x7ffb1c09a550 2024-09-16T14:44:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:34.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb19ffb640 10 monclient: dump: 2024-09-16T14:44:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:34.308 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:34.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 4 set_mon_vals no callback set 2024-09-16T14:44:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 10 monclient: shutdown 2024-09-16T14:44:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 1 -- 172.21.15.26:0/2923028375 >> v1:172.21.15.26:6789/0 conn(0x7ffb1c09a550 legacy=0x7ffb1c09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 1 -- 172.21.15.26:0/2923028375 shutdown_connections 2024-09-16T14:44:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 1 -- 172.21.15.26:0/2923028375 >> 172.21.15.26:0/2923028375 conn(0x7ffb1c09fb50 msgr2=0x7ffb1c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 1 -- 172.21.15.26:0/2923028375 shutdown_connections 2024-09-16T14:44:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.278+0000 7ffb21631640 1 -- 172.21.15.26:0/2923028375 wait complete. 2024-09-16T14:44:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 10 monclient: build_initial_monmap 2024-09-16T14:44:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 10 monclient: monmap: 2024-09-16T14:44:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:44:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:44:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:44:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:34.310 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 1 Processor -- start 2024-09-16T14:44:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 1 -- start start 2024-09-16T14:44:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 10 monclient: init 2024-09-16T14:44:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 10 monclient(hunting): picked mon.noname-a con 0x7ffb1c09a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:34.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 10 monclient(hunting): picked mon.noname-b con 0x7ffb1c09b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 10 monclient(hunting): picked mon.noname-c con 0x7ffb1c09d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c133bf0 con 0x7ffb1c09a550 2024-09-16T14:44:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c13f180 con 0x7ffb1c09b040 2024-09-16T14:44:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c070f80 con 0x7ffb1c09d1b0 2024-09-16T14:44:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 10 monclient(hunting): authenticate will time out at 1098.405150s 2024-09-16T14:44:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb1affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ffb1c09a550 0x7ffb1c132dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53926/0 (socket says 172.21.15.26:53926) 2024-09-16T14:44:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb1affd640 1 -- 172.21.15.26:0/3218272469 learned_addr learned my addr 172.21.15.26:0/3218272469 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb1a7fc640 1 -- 172.21.15.26:0/3218272469 >> v1:172.21.15.189:6789/0 conn(0x7ffb1c09b040 legacy=0x7ffb1c1334e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969095012 0 0) 0x7ffb1c133bf0 con 0x7ffb1c09a550 2024-09-16T14:44:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:34.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): my global_id is 84180 2024-09-16T14:44:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb08003530 con 0x7ffb1c09a550 2024-09-16T14:44:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301985179 0 0) 0x7ffb1c070f80 con 0x7ffb1c09d1b0 2024-09-16T14:44:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): my global_id is 4793 2024-09-16T14:44:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb1c133bf0 con 0x7ffb1c09d1b0 2024-09-16T14:44:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 584147953 0 0) 0x7ffb08003530 con 0x7ffb1c09a550 2024-09-16T14:44:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb1c070f80 con 0x7ffb1c09a550 2024-09-16T14:44:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1482351562 0 0) 0x7ffb1c133bf0 con 0x7ffb1c09d1b0 2024-09-16T14:44:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb08003530 con 0x7ffb1c09d1b0 2024-09-16T14:44:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb100015e0 con 0x7ffb1c09a550 2024-09-16T14:44:34.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:34.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:34.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): dump: 2024-09-16T14:44:34.313 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:34.313 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:34.313 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.313 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.313 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:34.313 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:34.313 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:34.313 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:34.313 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:34.314 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:34.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb0c001c40 con 0x7ffb1c09d1b0 2024-09-16T14:44:34.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:34.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:34.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): dump: 2024-09-16T14:44:34.314 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:34.314 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:34.314 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.314 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.314 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:34.314 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:34.314 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:34.314 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:34.315 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:34.315 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 943628265 0 0) 0x7ffb1c070f80 con 0x7ffb1c09a550 2024-09-16T14:44:34.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 >> v1:172.21.15.26:6790/0 conn(0x7ffb1c09d1b0 legacy=0x7ffb1c1d1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:34.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 >> v1:172.21.15.189:6789/0 conn(0x7ffb1c09b040 legacy=0x7ffb1c1334e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:34.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient: _finish_hunting 0 2024-09-16T14:44:34.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 monclient: found mon.a 2024-09-16T14:44:34.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:34.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:34.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c119d50 con 0x7ffb1c09a550 2024-09-16T14:44:34.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient: _finish_auth 0 2024-09-16T14:44:34.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient: _check_auth_tickets 2024-09-16T14:44:34.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:34.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 5 monclient: authenticate success, global_id 84180 2024-09-16T14:44:34.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 10 monclient: _renew_subs 2024-09-16T14:44:34.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:34.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 1 -- 172.21.15.26:0/3218272469 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1c13ef00 con 0x7ffb1c09a550 2024-09-16T14:44:34.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 10 monclient: _renew_subs 2024-09-16T14:44:34.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:34.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 1 -- 172.21.15.26:0/3218272469 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1c13e5b0 con 0x7ffb1c09a550 2024-09-16T14:44:34.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb10004c70 con 0x7ffb1c09a550 2024-09-16T14:44:34.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:34.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb10005000 con 0x7ffb1c09a550 2024-09-16T14:44:34.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:34.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:34.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb197fa640 4 set_mon_vals no callback set 2024-09-16T14:44:34.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient: dump: 2024-09-16T14:44:34.317 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:34.317 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:34.317 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.317 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.317 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:34.317 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:34.317 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:34.317 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:34.317 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:34.317 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:34.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb21631640 1 -- 172.21.15.26:0/3218272469 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ffafc006590 con 0x7ffafc002830 2024-09-16T14:44:34.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ffb100051e0 con 0x7ffb1c09a550 2024-09-16T14:44:34.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(39..39 src has 1..39) v4 ==== 3940+0+0 (unknown 2059123099 0 0) 0x7ffb10043590 con 0x7ffb1c09a550 2024-09-16T14:44:34.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233875949 0 0) 0x7ffafc006590 con 0x7ffafc002830 2024-09-16T14:44:34.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:34.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:34.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:34.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient(hunting): my global_id is 84183 2024-09-16T14:44:34.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb1c070f80 con 0x7ffafc002830 2024-09-16T14:44:34.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1137429160 0 0) 0x7ffb1c070f80 con 0x7ffafc002830 2024-09-16T14:44:34.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffafc006590 con 0x7ffafc002830 2024-09-16T14:44:34.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb14003eb0 con 0x7ffafc002830 2024-09-16T14:44:34.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:34.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:34.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient: dump: 2024-09-16T14:44:34.319 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:34.319 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:34.319 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.319 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.319 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:34.319 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:34.319 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:34.319 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:34.319 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:34.319 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb1b7fe640 10 monclient: get_auth_request con 0x7ffb08036210 auth_method 0 2024-09-16T14:44:34.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1686257778 0 0) 0x7ffafc006590 con 0x7ffafc002830 2024-09-16T14:44:34.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffafc00a0e0 con 0x7ffafc002830 2024-09-16T14:44:34.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ffb140062e0 con 0x7ffafc002830 2024-09-16T14:44:34.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:34.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:34.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 >> v1:172.21.15.26:6789/0 conn(0x7ffafc002830 legacy=0x7ffafc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:34.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.282+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 >> v1:172.21.15.26:6789/0 conn(0x7ffafc002830 legacy=0x7ffafc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:34.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb21631640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:44:34.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb21631640 1 -- 172.21.15.26:0/3218272469 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ffafc009d80 con 0x7ffafc0090a0 2024-09-16T14:44:34.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544954078 0 0) 0x7ffafc009d80 con 0x7ffafc0090a0 2024-09-16T14:44:34.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:34.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:34.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 10 monclient(hunting): my global_id is 84186 2024-09-16T14:44:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffafc006590 con 0x7ffafc0090a0 2024-09-16T14:44:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3417746407 0 0) 0x7ffafc006590 con 0x7ffafc0090a0 2024-09-16T14:44:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffafc009d80 con 0x7ffafc0090a0 2024-09-16T14:44:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb14004740 con 0x7ffafc0090a0 2024-09-16T14:44:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 10 monclient: dump: 2024-09-16T14:44:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.321 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:34.322 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2927031369 0 0) 0x7ffafc009d80 con 0x7ffafc0090a0 2024-09-16T14:44:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ffafc00c290 con 0x7ffafc0090a0 2024-09-16T14:44:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4034 (unknown 0 0 3767174491) 0x7ffb14005b40 con 0x7ffafc0090a0 2024-09-16T14:44:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:44:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 >> v1:172.21.15.26:6789/0 conn(0x7ffafc0090a0 legacy=0x7ffafc009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:34.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb037fe640 1 -- 172.21.15.26:0/3218272469 >> v1:172.21.15.26:6789/0 conn(0x7ffafc0090a0 legacy=0x7ffafc009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:34.323 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:44:34.323 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:44:34.323 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:44:34.323 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:44:34.323 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 26, 2024-09-16T14:44:34.323 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:44:34.323 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:44:34.323 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:44:34.323 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:34.324 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:44:34.324 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:44:34.324 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:44:34.324 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:44:34.324 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:34.324 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:34.324 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:34.324 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:34.324 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:34.324 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:34.324 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:34.324 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:34.324 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:34.325 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:34.325 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:34.325 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:44:34.325 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:44:34.325 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:34.325 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:34.325 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:34.325 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:34.325 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:34.325 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:34.325 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:34.325 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:34.325 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:34.326 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:34.326 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:34.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.326 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:44:34.326 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:44:34.326 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:44:34.326 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:44:34.326 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:44:34.326 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:44:34.326 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:44:34.326 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:44:34.326 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:44:34.326 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:44:34.326 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:44:34.327 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:44:34.327 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:44:34.327 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:44:34.327 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:44:34.327 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:44:34.327 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:44:34.327 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:34.327 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:34.327 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:34.327 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:34.327 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:34.327 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:34.327 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:34.328 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:34.328 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:34.328 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:34.328 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:34.328 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:44:34.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.328 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:44:34.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.328 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:44:34.328 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:44:34.328 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:34.328 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:34.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.329 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:34.329 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:44:34.329 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:34.329 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:34.329 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:34.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.329 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:44:34.329 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:44:34.329 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:34.329 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:34.329 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:34.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.329 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:44:34.330 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:44:34.330 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:34.330 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:34.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.330 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:34.330 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:44:34.330 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:34.330 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:34.330 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:34.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.330 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:44:34.330 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:44:34.330 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:34.331 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:34.331 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:34.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.331 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:44:34.331 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:44:34.331 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:34.331 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:34.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.331 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:34.331 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:44:34.331 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:34.331 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:34.332 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:34.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.332 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:44:34.332 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:44:34.332 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:34.332 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:34.332 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:34.332 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:34.332 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:34.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.332 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:44:34.332 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:44:34.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.333 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:34.333 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:34.333 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:44:34.333 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:34.333 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:34.333 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:44:34.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.333 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:34.333 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:34.333 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:44:34.333 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:34.333 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:34.333 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:44:34.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.334 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:34.334 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:34.334 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:44:34.334 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:34.334 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:34.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.334 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:44:34.334 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:44:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb21631640 1 -- 172.21.15.26:0/3218272469 >> v1:172.21.15.189:6812/3196161582 conn(0x7ffb08036210 legacy=0x7ffb080386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:34.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb21631640 10 monclient: shutdown 2024-09-16T14:44:34.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb21631640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:34.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb21631640 1 -- 172.21.15.26:0/3218272469 >> v1:172.21.15.26:6789/0 conn(0x7ffb1c09a550 legacy=0x7ffb1c132dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:34.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb1b7fe640 1 -- 172.21.15.26:0/3218272469 reap_dead start 2024-09-16T14:44:34.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb1b7fe640 1 -- 172.21.15.26:0/3218272469 reap_dead start 2024-09-16T14:44:34.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb21631640 1 -- 172.21.15.26:0/3218272469 shutdown_connections 2024-09-16T14:44:34.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb21631640 1 -- 172.21.15.26:0/3218272469 >> 172.21.15.26:0/3218272469 conn(0x7ffb1c09fb50 msgr2=0x7ffb1c0a1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:34.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb21631640 1 -- 172.21.15.26:0/3218272469 shutdown_connections 2024-09-16T14:44:34.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.290+0000 7ffb21631640 1 -- 172.21.15.26:0/3218272469 wait complete. 2024-09-16T14:44:34.336 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:44:34.336 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:44:34.337 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:44:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.442+0000 7f878695d640 10 monclient: get_monmap_and_config 2024-09-16T14:44:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.442+0000 7f878695d640 10 monclient: build_initial_monmap 2024-09-16T14:44:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.442+0000 7f878695d640 10 monclient: monmap: 2024-09-16T14:44:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:44:34.448470+0000 2024-09-16T14:44:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:44:34.448470+0000 2024-09-16T14:44:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:34.449 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.442+0000 7f878695d640 1 Processor -- start 2024-09-16T14:44:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.442+0000 7f878695d640 1 -- start start 2024-09-16T14:44:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.442+0000 7f878695d640 10 monclient: init 2024-09-16T14:44:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient(hunting): picked mon.noname-b con 0x7f87800a0af0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient(hunting): picked mon.noname-a con 0x7f878009f340 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient(hunting): picked mon.noname-c con 0x7f8780072040 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87800a0ed0 con 0x7f878009f340 2024-09-16T14:44:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878009d720 con 0x7f8780072040 2024-09-16T14:44:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87800a1410 con 0x7f87800a0af0 2024-09-16T14:44:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient(hunting): authenticate will time out at 1098.568448s 2024-09-16T14:44:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f87800a0af0 0x7f8780071930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:37904/0 (socket says 172.21.15.26:37904) 2024-09-16T14:44:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877ffff640 1 -- 172.21.15.26:0/2694128241 learned_addr learned my addr 172.21.15.26:0/2694128241 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f8784efa640 1 -- 172.21.15.26:0/2694128241 >> v1:172.21.15.189:6789/0 conn(0x7f8780072040 legacy=0x7f878013d3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 1 -- 172.21.15.26:0/2694128241 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468885792 0 0) 0x7f87800a0ed0 con 0x7f878009f340 2024-09-16T14:44:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient(hunting): my global_id is 84189 2024-09-16T14:44:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 1 -- 172.21.15.26:0/2694128241 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8768003530 con 0x7f878009f340 2024-09-16T14:44:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 1 -- 172.21.15.26:0/2694128241 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534532228 0 0) 0x7f87800a1410 con 0x7f87800a0af0 2024-09-16T14:44:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient(hunting): my global_id is 4796 2024-09-16T14:44:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 1 -- 172.21.15.26:0/2694128241 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87800a0ed0 con 0x7f87800a0af0 2024-09-16T14:44:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 1 -- 172.21.15.26:0/2694128241 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3323051634 0 0) 0x7f8768003530 con 0x7f878009f340 2024-09-16T14:44:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 1 -- 172.21.15.26:0/2694128241 >> v1:172.21.15.26:6790/0 conn(0x7f87800a0af0 legacy=0x7f8780071930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 1 -- 172.21.15.26:0/2694128241 >> v1:172.21.15.189:6789/0 conn(0x7f8780072040 legacy=0x7f878013d3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:34.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:34.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 1 monclient: found mon.noname-a 2024-09-16T14:44:34.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:34.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:34.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 1 -- 172.21.15.26:0/2694128241 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8780072490 con 0x7f878009f340 2024-09-16T14:44:34.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient: _finish_auth 0 2024-09-16T14:44:34.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:34.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:34.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 1 -- 172.21.15.26:0/2694128241 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8770001620 con 0x7f878009f340 2024-09-16T14:44:34.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 5 monclient: authenticate success, global_id 84189 2024-09-16T14:44:34.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:34.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:34.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient: dump: 2024-09-16T14:44:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:34.456 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 1 -- 172.21.15.26:0/2694128241 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8770001900 con 0x7f878009f340 2024-09-16T14:44:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 1 -- 172.21.15.26:0/2694128241 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8770001c90 con 0x7f878009f340 2024-09-16T14:44:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 4 set_mon_vals no callback set 2024-09-16T14:44:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877effd640 10 monclient: dump: 2024-09-16T14:44:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:34.458 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient: shutdown 2024-09-16T14:44:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 1 -- 172.21.15.26:0/2694128241 >> v1:172.21.15.26:6789/0 conn(0x7f878009f340 legacy=0x7f8780139cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 1 -- 172.21.15.26:0/2694128241 shutdown_connections 2024-09-16T14:44:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 1 -- 172.21.15.26:0/2694128241 >> 172.21.15.26:0/2694128241 conn(0x7f87800a1be0 msgr2=0x7f87800a1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 1 -- 172.21.15.26:0/2694128241 shutdown_connections 2024-09-16T14:44:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 1 -- 172.21.15.26:0/2694128241 wait complete. 2024-09-16T14:44:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient: build_initial_monmap 2024-09-16T14:44:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient: monmap: 2024-09-16T14:44:34.459 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:44:34.459 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:34.459 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:44:34.459 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:44:34.459 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:34.459 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:44:34.459 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:34.459 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:34.459 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:34.459 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 1 Processor -- start 2024-09-16T14:44:34.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 1 -- start start 2024-09-16T14:44:34.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient: init 2024-09-16T14:44:34.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:34.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:44:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient(hunting): picked mon.noname-c con 0x7f8780072040 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient(hunting): picked mon.noname-b con 0x7f878009f340 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient(hunting): picked mon.noname-a con 0x7f87800a0af0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878009b300 con 0x7f87800a0af0 2024-09-16T14:44:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878009c2d0 con 0x7f878009f340 2024-09-16T14:44:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878009afb0 con 0x7f8780072040 2024-09-16T14:44:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f878695d640 10 monclient(hunting): authenticate will time out at 1098.571899s 2024-09-16T14:44:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8780072040 0x7f878013b0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:37906/0 (socket says 172.21.15.26:37906) 2024-09-16T14:44:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877ffff640 1 -- 172.21.15.26:0/614837189 learned_addr learned my addr 172.21.15.26:0/614837189 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:44:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.446+0000 7f877f7fe640 1 -- 172.21.15.26:0/614837189 >> v1:172.21.15.189:6789/0 conn(0x7f878009f340 legacy=0x7f878013b800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962904632 0 0) 0x7f878009afb0 con 0x7f8780072040 2024-09-16T14:44:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): my global_id is 4799 2024-09-16T14:44:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f876c003530 con 0x7f8780072040 2024-09-16T14:44:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580594547 0 0) 0x7f878009b300 con 0x7f87800a0af0 2024-09-16T14:44:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): my global_id is 84192 2024-09-16T14:44:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f878009afb0 con 0x7f87800a0af0 2024-09-16T14:44:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2631462935 0 0) 0x7f878009afb0 con 0x7f87800a0af0 2024-09-16T14:44:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f878009b300 con 0x7f87800a0af0 2024-09-16T14:44:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f87780043c0 con 0x7f87800a0af0 2024-09-16T14:44:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): dump: 2024-09-16T14:44:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.467 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:34.468 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:34.468 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:34.468 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:34.468 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:34.468 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:34.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 598712722 0 0) 0x7f876c003530 con 0x7f8780072040 2024-09-16T14:44:34.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f878009afb0 con 0x7f8780072040 2024-09-16T14:44:34.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f87740016f0 con 0x7f8780072040 2024-09-16T14:44:34.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:34.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:34.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): dump: 2024-09-16T14:44:34.469 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:34.469 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:34.469 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.469 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.469 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:34.469 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:34.469 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:34.469 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:34.469 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:34.469 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2086997482 0 0) 0x7f878009b300 con 0x7f87800a0af0 2024-09-16T14:44:34.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 >> v1:172.21.15.26:6790/0 conn(0x7f8780072040 legacy=0x7f878013b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:34.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 >> v1:172.21.15.189:6789/0 conn(0x7f878009f340 legacy=0x7f878013b800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:34.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:44:34.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 monclient: found mon.a 2024-09-16T14:44:34.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:34.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:34.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878011fec0 con 0x7f87800a0af0 2024-09-16T14:44:34.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient: _finish_auth 0 2024-09-16T14:44:34.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:44:34.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:34.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f878695d640 5 monclient: authenticate success, global_id 84192 2024-09-16T14:44:34.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f878695d640 10 monclient: _renew_subs 2024-09-16T14:44:34.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f878695d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:34.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f878695d640 1 -- 172.21.15.26:0/614837189 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878013d430 con 0x7f87800a0af0 2024-09-16T14:44:34.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f878695d640 10 monclient: _renew_subs 2024-09-16T14:44:34.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f878695d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:34.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f878695d640 1 -- 172.21.15.26:0/614837189 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f878013e880 con 0x7f87800a0af0 2024-09-16T14:44:34.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87780045e0 con 0x7f87800a0af0 2024-09-16T14:44:34.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:34.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8778004970 con 0x7f87800a0af0 2024-09-16T14:44:34.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:34.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877e7fc640 4 set_mon_vals no callback set 2024-09-16T14:44:34.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:34.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient: dump: 2024-09-16T14:44:34.472 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:34.472 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:34.472 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.472 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.472 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:34.472 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:34.472 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:34.472 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:34.472 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:34.472 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f878695d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:34.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f878695d640 1 -- 172.21.15.26:0/614837189 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8760006590 con 0x7f8760002830 2024-09-16T14:44:34.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8778004b50 con 0x7f87800a0af0 2024-09-16T14:44:34.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(39..39 src has 1..39) v4 ==== 3940+0+0 (unknown 2059123099 0 0) 0x7f8778043500 con 0x7f87800a0af0 2024-09-16T14:44:34.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872195307 0 0) 0x7f8760006590 con 0x7f8760002830 2024-09-16T14:44:34.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:34.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:34.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:34.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient(hunting): my global_id is 4802 2024-09-16T14:44:34.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f878009b300 con 0x7f8760002830 2024-09-16T14:44:34.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 596555942 0 0) 0x7f878009b300 con 0x7f8760002830 2024-09-16T14:44:34.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8760006590 con 0x7f8760002830 2024-09-16T14:44:34.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f87740047f0 con 0x7f8760002830 2024-09-16T14:44:34.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:34.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:34.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient: dump: 2024-09-16T14:44:34.474 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:34.474 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:34.474 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.474 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.474 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:34.474 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:34.474 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:34.474 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:34.474 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:34.475 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877f7fe640 10 monclient: get_auth_request con 0x7f876c036220 auth_method 0 2024-09-16T14:44:34.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1313880906 0 0) 0x7f8760006590 con 0x7f8760002830 2024-09-16T14:44:34.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f876000a0e0 con 0x7f8760002830 2024-09-16T14:44:34.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f87740058a0 con 0x7f8760002830 2024-09-16T14:44:34.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:34.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:34.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 >> v1:172.21.15.26:6790/0 conn(0x7f8760002830 legacy=0x7f8760004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:34.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.450+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 >> v1:172.21.15.26:6790/0 conn(0x7f8760002830 legacy=0x7f8760004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:34.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f878695d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:44:34.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f878695d640 1 -- 172.21.15.26:0/614837189 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8760009da0 con 0x7f87600090c0 2024-09-16T14:44:34.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755592949 0 0) 0x7f8760009da0 con 0x7f87600090c0 2024-09-16T14:44:34.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:34.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:34.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:34.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 10 monclient(hunting): my global_id is 4805 2024-09-16T14:44:34.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8760006590 con 0x7f87600090c0 2024-09-16T14:44:34.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1270772080 0 0) 0x7f8760006590 con 0x7f87600090c0 2024-09-16T14:44:34.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8760009da0 con 0x7f87600090c0 2024-09-16T14:44:34.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8774004e30 con 0x7f87600090c0 2024-09-16T14:44:34.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:34.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:34.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 10 monclient: dump: 2024-09-16T14:44:34.477 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:44:34.477 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:34.477 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.477 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:34.477 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:34.477 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:44:34.477 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:34.477 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:34.477 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:34.477 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:44:34.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1344421270 0 0) 0x7f8760009da0 con 0x7f87600090c0 2024-09-16T14:44:34.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f876000c290 con 0x7f87600090c0 2024-09-16T14:44:34.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4202 (unknown 0 0 1410788934) 0x7f8774005a10 con 0x7f87600090c0 2024-09-16T14:44:34.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:44:34.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:34.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 >> v1:172.21.15.26:6790/0 conn(0x7f87600090c0 legacy=0x7f8760009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:34.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f877cff9640 1 -- 172.21.15.26:0/614837189 >> v1:172.21.15.26:6790/0 conn(0x7f87600090c0 legacy=0x7f8760009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:44:34.478 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:44:34.478 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:44:34.478 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:44:34.478 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:44:34.479 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 26, 2024-09-16T14:44:34.479 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:44:34.479 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:44:34.479 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:44:34.479 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:34.479 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:44:34.479 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:44:34.479 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:44:34.479 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:44:34.479 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:34.479 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:34.479 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:34.480 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:34.480 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:34.480 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:34.480 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:34.480 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:34.480 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:34.480 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:34.480 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:34.481 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:44:34.481 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:44:34.481 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:34.481 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:34.481 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:34.481 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:34.481 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:34.481 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:34.481 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:34.481 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:34.481 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:34.481 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:34.481 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:34.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.482 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:44:34.482 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:44:34.482 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:44:34.482 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:44:34.482 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:44:34.482 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:44:34.482 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:44:34.482 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:44:34.483 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:44:34.483 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:44:34.483 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:44:34.483 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:44:34.483 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:44:34.483 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:44:34.483 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:44:34.483 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:44:34.483 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:44:34.483 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:44:34.483 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:44:34.483 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:44:34.483 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:44:34.484 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:44:34.484 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:44:34.484 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:44:34.484 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:44:34.484 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:44:34.484 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:44:34.484 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:34.484 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:44:34.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.484 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:44:34.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.484 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:44:34.485 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:44:34.485 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:34.485 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:34.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.485 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:34.485 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:44:34.485 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:34.485 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:34.485 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:34.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.485 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:44:34.486 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:44:34.486 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:34.486 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:34.486 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:34.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.486 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:44:34.486 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:44:34.486 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:34.486 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:34.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:34.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:44:34.487 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:34.487 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:34.487 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:34.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:44:34.487 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:44:34.487 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:34.487 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:34.487 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:34.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.488 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:44:34.488 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:44:34.488 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:44:34.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:44:34.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.488 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:44:34.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:44:34.488 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:44:34.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:34.488 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:34.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:44:34.489 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:44:34.489 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:44:34.489 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:44:34.489 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:44:34.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:34.489 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:34.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.489 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:44:34.489 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:44:34.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.489 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:34.489 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:34.489 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:44:34.490 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:34.490 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:34.490 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:44:34.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.490 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:34.490 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:34.490 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:44:34.490 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:34.490 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:34.490 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:44:34.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.490 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:34.490 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:34.491 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:44:34.491 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:34.491 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:44:34.491 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:44:34.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:44:34.491 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:44:34.491 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:44:34.491 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:44:34.491 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:44:34.491 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:44:34.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:44:34.491 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:44:34.491 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:44:34.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f878695d640 1 -- 172.21.15.26:0/614837189 >> v1:172.21.15.189:6812/3196161582 conn(0x7f876c036220 legacy=0x7f876c0386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:34.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f878695d640 10 monclient: shutdown 2024-09-16T14:44:34.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f878695d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:34.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f878695d640 1 -- 172.21.15.26:0/614837189 >> v1:172.21.15.26:6789/0 conn(0x7f87800a0af0 legacy=0x7f87801d19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:34.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f8784efa640 1 -- 172.21.15.26:0/614837189 reap_dead start 2024-09-16T14:44:34.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f8784efa640 1 -- 172.21.15.26:0/614837189 reap_dead start 2024-09-16T14:44:34.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f878695d640 1 -- 172.21.15.26:0/614837189 shutdown_connections 2024-09-16T14:44:34.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f878695d640 1 -- 172.21.15.26:0/614837189 >> 172.21.15.26:0/614837189 conn(0x7f87800a1be0 msgr2=0x7f87800a33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:34.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f878695d640 1 -- 172.21.15.26:0/614837189 shutdown_connections 2024-09-16T14:44:34.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:44:34.458+0000 7f878695d640 1 -- 172.21.15.26:0/614837189 wait complete. 2024-09-16T14:44:34.494 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:44:34.494 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:44:34.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:34.577+0000 7f57fd7fa640 10 monclient: tick 2024-09-16T14:44:34.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:34.577+0000 7f57fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:34.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:34.577+0000 7f57fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:34.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:34.917+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' removed v40) v1 ==== 149+0+0 (unknown 4274453308 0 0) 0x7f5804013f40 con 0x7f5808099a70 2024-09-16T14:44:34.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:34.917+0000 7f57fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:34.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:34.917+0000 7f57fe7fc640 10 monclient: _finish_command 2 = system:0 pool 'foo' removed 2024-09-16T14:44:35.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.041+0000 7f580fe54640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:35.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.041+0000 7f580fe54640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:35.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.041+0000 7f580fe54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:35.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.041+0000 7f580fe54640 1 -- 172.21.15.189:0/1563518999 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f57f0005330 con 0x7f5808099a70 2024-09-16T14:44:35.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.041+0000 7f57fe7fc640 1 -- 172.21.15.189:0/1563518999 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v40) v1 ==== 156+0+0 (unknown 1726847040 0 0) 0x7f580401ef70 con 0x7f5808099a70 2024-09-16T14:44:35.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.041+0000 7f57fe7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:35.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.041+0000 7f57fe7fc640 10 monclient: _finish_command 3 = system:0 pool 'foo' does not exist 2024-09-16T14:44:35.048 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foo' does not exist 2024-09-16T14:44:35.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.045+0000 7f580fe54640 1 -- 172.21.15.189:0/1563518999 >> v1:172.21.15.189:6812/3196161582 conn(0x7f57ec036010 legacy=0x7f57ec0384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:35.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.045+0000 7f580fe54640 10 monclient: shutdown 2024-09-16T14:44:35.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.045+0000 7f580fe54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:35.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.045+0000 7f580fe54640 1 -- 172.21.15.189:0/1563518999 >> v1:172.21.15.26:6790/0 conn(0x7f5808099a70 legacy=0x7f57ec048a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:35.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.045+0000 7f580fe54640 1 -- 172.21.15.189:0/1563518999 shutdown_connections 2024-09-16T14:44:35.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.045+0000 7f580fe54640 1 -- 172.21.15.189:0/1563518999 >> 172.21.15.189:0/1563518999 conn(0x7f58080a3380 msgr2=0x7f580806ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:35.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.045+0000 7f580fe54640 1 -- 172.21.15.189:0/1563518999 shutdown_connections 2024-09-16T14:44:35.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.045+0000 7f580fe54640 1 -- 172.21.15.189:0/1563518999 wait complete. 2024-09-16T14:44:35.063 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-16T14:44:35.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe968db6640 10 monclient: get_monmap_and_config 2024-09-16T14:44:35.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe968db6640 10 monclient: build_initial_monmap 2024-09-16T14:44:35.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe968db6640 10 monclient: monmap: 2024-09-16T14:44:35.179 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:35.179 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:35.179 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:35.179022+0000 2024-09-16T14:44:35.180 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:35.179022+0000 2024-09-16T14:44:35.180 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:35.180 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:35.180 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:35.180 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:35.180 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:35.180 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:35.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe968db6640 1 Processor -- start 2024-09-16T14:44:35.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe968db6640 1 -- start start 2024-09-16T14:44:35.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe968db6640 10 monclient: init 2024-09-16T14:44:35.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe968db6640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:35.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe968db6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:35.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe968db6640 10 monclient(hunting): picked mon.noname-c con 0x7fe964099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:35.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe968db6640 10 monclient(hunting): picked mon.noname-a con 0x7fe96409c550 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:35.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe968db6640 10 monclient(hunting): picked mon.noname-b con 0x7fe96409e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:35.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe968db6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe964099e50 con 0x7fe96409c550 2024-09-16T14:44:35.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe968db6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe96409dbd0 con 0x7fe964099a70 2024-09-16T14:44:35.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe968db6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe96409ee20 con 0x7fe96409e6d0 2024-09-16T14:44:35.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe968db6640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:35.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe968db6640 10 monclient(hunting): authenticate will time out at 1057.295297s 2024-09-16T14:44:35.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe96259c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fe964099a70 legacy=0x7fe96409dfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:35.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe962d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe96409e6d0 0x7fe96413b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:35966/0 (socket says 172.21.15.189:35966) 2024-09-16T14:44:35.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe962d9d640 1 -- 172.21.15.189:0/3919769964 learned_addr learned my addr 172.21.15.189:0/3919769964 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:35.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.173+0000 7fe95bfff640 1 --1- 172.21.15.189:0/3919769964 >> v1:172.21.15.26:6789/0 conn(0x7fe96409c550 0x7fe9641379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:52736/0 (socket says 172.21.15.189:52736) 2024-09-16T14:44:35.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 1 -- 172.21.15.189:0/3919769964 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610196451 0 0) 0x7fe96409ee20 con 0x7fe96409e6d0 2024-09-16T14:44:35.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:44:35.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:35.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:35.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient(hunting): my global_id is 4808 2024-09-16T14:44:35.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 1 -- 172.21.15.189:0/3919769964 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe950003530 con 0x7fe96409e6d0 2024-09-16T14:44:35.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 1 -- 172.21.15.189:0/3919769964 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325707291 0 0) 0x7fe964099e50 con 0x7fe96409c550 2024-09-16T14:44:35.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:44:35.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:35.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:35.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient(hunting): my global_id is 84195 2024-09-16T14:44:35.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 1 -- 172.21.15.189:0/3919769964 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe96409ee20 con 0x7fe96409c550 2024-09-16T14:44:35.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 1 -- 172.21.15.189:0/3919769964 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2969488553 0 0) 0x7fe950003530 con 0x7fe96409e6d0 2024-09-16T14:44:35.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 1 -- 172.21.15.189:0/3919769964 >> v1:172.21.15.189:6789/0 conn(0x7fe964099a70 legacy=0x7fe96409dfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:35.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 1 -- 172.21.15.189:0/3919769964 >> v1:172.21.15.26:6789/0 conn(0x7fe96409c550 legacy=0x7fe9641379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:35.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:44:35.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 1 monclient: found mon.noname-b 2024-09-16T14:44:35.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:35.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:35.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 1 -- 172.21.15.189:0/3919769964 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe96409f100 con 0x7fe96409e6d0 2024-09-16T14:44:35.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient: _finish_auth 0 2024-09-16T14:44:35.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:44:35.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:35.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 1 -- 172.21.15.189:0/3919769964 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe95c0018d0 con 0x7fe96409e6d0 2024-09-16T14:44:35.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:35.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:35.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient: dump: 2024-09-16T14:44:35.187 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:35.187 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:35.187 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.187 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.187 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:35.187 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:35.187 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:35.187 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:35.187 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:35.188 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:35.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 5 monclient: authenticate success, global_id 4808 2024-09-16T14:44:35.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:35.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 1 -- 172.21.15.189:0/3919769964 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe95c001b40 con 0x7fe96409e6d0 2024-09-16T14:44:35.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:35.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:35.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 4 set_mon_vals no callback set 2024-09-16T14:44:35.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 1 -- 172.21.15.189:0/3919769964 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe95c001ed0 con 0x7fe96409e6d0 2024-09-16T14:44:35.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:35.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:35.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe961d9b640 10 monclient: dump: 2024-09-16T14:44:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:35.189 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 10 monclient: shutdown 2024-09-16T14:44:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:35.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 1 -- 172.21.15.189:0/3919769964 >> v1:172.21.15.26:6790/0 conn(0x7fe96409e6d0 legacy=0x7fe96413b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:35.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 1 -- 172.21.15.189:0/3919769964 shutdown_connections 2024-09-16T14:44:35.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 1 -- 172.21.15.189:0/3919769964 >> 172.21.15.189:0/3919769964 conn(0x7fe9640a3380 msgr2=0x7fe96409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:35.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 1 -- 172.21.15.189:0/3919769964 shutdown_connections 2024-09-16T14:44:35.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 1 -- 172.21.15.189:0/3919769964 wait complete. 2024-09-16T14:44:35.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 10 monclient: build_initial_monmap 2024-09-16T14:44:35.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 10 monclient: monmap: 2024-09-16T14:44:35.190 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:35.190 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:35.190 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:35.190 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:35.190 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:35.190 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:35.191 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:35.191 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:35.191 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:35.191 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:35.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 1 Processor -- start 2024-09-16T14:44:35.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 1 -- start start 2024-09-16T14:44:35.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 10 monclient: init 2024-09-16T14:44:35.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:35.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:44:35.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 10 monclient(hunting): picked mon.noname-b con 0x7fe964099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:35.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 10 monclient(hunting): picked mon.noname-c con 0x7fe96409c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:35.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 10 monclient(hunting): picked mon.noname-a con 0x7fe96409e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:35.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe96409cf20 con 0x7fe96409e6d0 2024-09-16T14:44:35.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe96411dd40 con 0x7fe964099a70 2024-09-16T14:44:35.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe96409a750 con 0x7fe96409c550 2024-09-16T14:44:35.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:35.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe96259c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fe964099a70 legacy=0x7fe964138ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:35.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe968db6640 10 monclient(hunting): authenticate will time out at 1057.298959s 2024-09-16T14:44:35.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe962d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe96409e6d0 0x7fe9641d1720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:52744/0 (socket says 172.21.15.189:52744) 2024-09-16T14:44:35.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe962d9d640 1 -- 172.21.15.189:0/1609195620 learned_addr learned my addr 172.21.15.189:0/1609195620 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:35.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553791214 0 0) 0x7fe96409cf20 con 0x7fe96409e6d0 2024-09-16T14:44:35.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe95affd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:35.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe95affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe95affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe95affd640 10 monclient(hunting): my global_id is 84198 2024-09-16T14:44:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe944003530 con 0x7fe96409e6d0 2024-09-16T14:44:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298001253 0 0) 0x7fe96409a750 con 0x7fe96409c550 2024-09-16T14:44:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe95affd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe95affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe95affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe95affd640 10 monclient(hunting): my global_id is 4811 2024-09-16T14:44:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.177+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe96409cf20 con 0x7fe96409c550 2024-09-16T14:44:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2057733599 0 0) 0x7fe96409cf20 con 0x7fe96409c550 2024-09-16T14:44:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe96409a750 con 0x7fe96409c550 2024-09-16T14:44:35.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2719889774 0 0) 0x7fe944003530 con 0x7fe96409e6d0 2024-09-16T14:44:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe96409cf20 con 0x7fe96409e6d0 2024-09-16T14:44:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe95c001890 con 0x7fe96409e6d0 2024-09-16T14:44:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 10 monclient(hunting): dump: 2024-09-16T14:44:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:35.198 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe94c0018c0 con 0x7fe96409c550 2024-09-16T14:44:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 10 monclient(hunting): dump: 2024-09-16T14:44:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:35.199 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2065862383 0 0) 0x7fe96409a750 con 0x7fe96409c550 2024-09-16T14:44:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 >> v1:172.21.15.189:6789/0 conn(0x7fe964099a70 legacy=0x7fe964138ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 >> v1:172.21.15.26:6789/0 conn(0x7fe96409e6d0 legacy=0x7fe9641d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 1 monclient: found mon.c 2024-09-16T14:44:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe96406b320 con 0x7fe96409c550 2024-09-16T14:44:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 10 monclient: _finish_auth 0 2024-09-16T14:44:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe968db6640 5 monclient: authenticate success, global_id 4811 2024-09-16T14:44:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe968db6640 10 monclient: _renew_subs 2024-09-16T14:44:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe968db6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe968db6640 1 -- 172.21.15.189:0/1609195620 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96411da60 con 0x7fe96409c550 2024-09-16T14:44:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe968db6640 10 monclient: _renew_subs 2024-09-16T14:44:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe968db6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe968db6640 1 -- 172.21.15.189:0/1609195620 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe964134610 con 0x7fe96409c550 2024-09-16T14:44:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe94c001c50 con 0x7fe96409c550 2024-09-16T14:44:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe94c004c40 con 0x7fe96409c550 2024-09-16T14:44:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 10 monclient: dump: 2024-09-16T14:44:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:35.201 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:35.202 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe960d99640 4 set_mon_vals no callback set 2024-09-16T14:44:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe968db6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe968db6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe968db6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe968db6640 1 -- 172.21.15.189:0/1609195620 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe948005050 con 0x7fe96409c550 2024-09-16T14:44:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe94c00f6b0 con 0x7fe96409c550 2024-09-16T14:44:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(40..40 src has 1..40) v4 ==== 3598+0+0 (unknown 1780700966 0 0) 0x7fe94c043140 con 0x7fe96409c550 2024-09-16T14:44:35.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.181+0000 7fe96259c640 10 monclient: get_auth_request con 0x7fe944036210 auth_method 0 2024-09-16T14:44:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.193+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe94c0144a0 con 0x7fe96409c550 2024-09-16T14:44:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.193+0000 7fe95affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.193+0000 7fe95affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:35.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.353+0000 7fe968db6640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:35.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.353+0000 7fe968db6640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:35.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.353+0000 7fe968db6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:35.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.353+0000 7fe968db6640 1 -- 172.21.15.189:0/1609195620 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe948005880 con 0x7fe96409c550 2024-09-16T14:44:35.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.353+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v40) v1 ==== 156+0+0 (unknown 1726847040 0 0) 0x7fe94c01ccc0 con 0x7fe96409c550 2024-09-16T14:44:35.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.353+0000 7fe95affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:35.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.353+0000 7fe95affd640 10 monclient: _finish_command 2 = system:0 pool 'foo' does not exist 2024-09-16T14:44:35.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.481+0000 7fe968db6640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:35.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.481+0000 7fe968db6640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:35.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.481+0000 7fe968db6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:35.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.481+0000 7fe968db6640 1 -- 172.21.15.189:0/1609195620 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe948005370 con 0x7fe96409c550 2024-09-16T14:44:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.481+0000 7fe95affd640 1 -- 172.21.15.189:0/1609195620 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v40) v1 ==== 156+0+0 (unknown 1726847040 0 0) 0x7fe94c012b40 con 0x7fe96409c550 2024-09-16T14:44:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.481+0000 7fe95affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.481+0000 7fe95affd640 10 monclient: _finish_command 3 = system:0 pool 'foo' does not exist 2024-09-16T14:44:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foo' does not exist 2024-09-16T14:44:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.485+0000 7fe968db6640 1 -- 172.21.15.189:0/1609195620 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe944036210 legacy=0x7fe9440386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.485+0000 7fe968db6640 10 monclient: shutdown 2024-09-16T14:44:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.485+0000 7fe968db6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.485+0000 7fe968db6640 1 -- 172.21.15.189:0/1609195620 >> v1:172.21.15.26:6790/0 conn(0x7fe96409c550 legacy=0x7fe9641391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.485+0000 7fe968db6640 1 -- 172.21.15.189:0/1609195620 shutdown_connections 2024-09-16T14:44:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.485+0000 7fe968db6640 1 -- 172.21.15.189:0/1609195620 >> 172.21.15.189:0/1609195620 conn(0x7fe9640a3380 msgr2=0x7fe9640a0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.485+0000 7fe968db6640 1 -- 172.21.15.189:0/1609195620 shutdown_connections 2024-09-16T14:44:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.485+0000 7fe968db6640 1 -- 172.21.15.189:0/1609195620 wait complete. 2024-09-16T14:44:35.508 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-16T14:44:35.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 10 monclient: get_monmap_and_config 2024-09-16T14:44:35.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 10 monclient: build_initial_monmap 2024-09-16T14:44:35.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 10 monclient: monmap: 2024-09-16T14:44:35.622 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:35.622 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:35.622 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:35.622057+0000 2024-09-16T14:44:35.622 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:35.622057+0000 2024-09-16T14:44:35.622 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:35.623 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:35.623 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:35.623 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:35.623 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:35.623 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:35.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 1 Processor -- start 2024-09-16T14:44:35.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 1 -- start start 2024-09-16T14:44:35.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 10 monclient: init 2024-09-16T14:44:35.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:35.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:35.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 10 monclient(hunting): picked mon.noname-a con 0x7f9c14099620 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:35.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 10 monclient(hunting): picked mon.noname-c con 0x7f9c14131a50 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:35.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 10 monclient(hunting): picked mon.noname-b con 0x7f9c140737f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:35.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c14073bf0 con 0x7f9c14099620 2024-09-16T14:44:35.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1409bce0 con 0x7f9c14131a50 2024-09-16T14:44:35.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1409b9e0 con 0x7f9c140737f0 2024-09-16T14:44:35.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:35.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c11d9b640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f9c14131a50 legacy=0x7f9c141368b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:35.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 10 monclient(hunting): authenticate will time out at 1057.737863s 2024-09-16T14:44:35.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1259c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9c14099620 0x7f9c14099a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:52750/0 (socket says 172.21.15.189:52750) 2024-09-16T14:44:35.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c12d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9c140737f0 0x7f9c14139fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:35986/0 (socket says 172.21.15.189:35986) 2024-09-16T14:44:35.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1259c640 1 -- 172.21.15.189:0/3402477339 learned_addr learned my addr 172.21.15.189:0/3402477339 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:35.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 1 -- 172.21.15.189:0/3402477339 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635210680 0 0) 0x7f9c1409b9e0 con 0x7f9c140737f0 2024-09-16T14:44:35.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient(hunting): _negotiate 2024-09-16T14:44:35.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:35.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:35.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient(hunting): my global_id is 4814 2024-09-16T14:44:35.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 1 -- 172.21.15.189:0/3402477339 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c00003530 con 0x7f9c140737f0 2024-09-16T14:44:35.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 1 -- 172.21.15.189:0/3402477339 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976426501 0 0) 0x7f9bfc0014a0 con 0x7f9c14099620 2024-09-16T14:44:35.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient(hunting): _negotiate 2024-09-16T14:44:35.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:35.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:35.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient(hunting): my global_id is 84201 2024-09-16T14:44:35.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 1 -- 172.21.15.189:0/3402477339 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c1409b9e0 con 0x7f9c14099620 2024-09-16T14:44:35.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 1 -- 172.21.15.189:0/3402477339 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 48566716 0 0) 0x7f9c00003530 con 0x7f9c140737f0 2024-09-16T14:44:35.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 1 -- 172.21.15.189:0/3402477339 >> v1:172.21.15.189:6789/0 conn(0x7f9c14131a50 legacy=0x7f9c141368b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:35.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 1 -- 172.21.15.189:0/3402477339 >> v1:172.21.15.26:6789/0 conn(0x7f9c14099620 legacy=0x7f9c14099a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:35.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient: _finish_hunting 0 2024-09-16T14:44:35.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 1 monclient: found mon.noname-b 2024-09-16T14:44:35.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:35.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:35.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 1 -- 172.21.15.189:0/3402477339 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1409a110 con 0x7f9c140737f0 2024-09-16T14:44:35.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient: _finish_auth 0 2024-09-16T14:44:35.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient: _check_auth_tickets 2024-09-16T14:44:35.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:35.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 1 -- 172.21.15.189:0/3402477339 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9c0c001620 con 0x7f9c140737f0 2024-09-16T14:44:35.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:35.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:35.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient: dump: 2024-09-16T14:44:35.629 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:35.629 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:35.629 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.629 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.629 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:35.629 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:35.629 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:35.629 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:35.630 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:35.630 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:35.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 5 monclient: authenticate success, global_id 4814 2024-09-16T14:44:35.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:35.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 1 -- 172.21.15.189:0/3402477339 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c0c001890 con 0x7f9c140737f0 2024-09-16T14:44:35.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:35.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 1 -- 172.21.15.189:0/3402477339 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9c0c001c60 con 0x7f9c140737f0 2024-09-16T14:44:35.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:35.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c18c1c640 4 set_mon_vals no callback set 2024-09-16T14:44:35.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:35.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:35.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.617+0000 7f9c1159a640 10 monclient: dump: 2024-09-16T14:44:35.630 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:35.631 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:35.631 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.631 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.631 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:35.631 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:35.631 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:35.631 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:35.631 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:35.631 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:35.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 10 monclient: shutdown 2024-09-16T14:44:35.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:35.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 1 -- 172.21.15.189:0/3402477339 >> v1:172.21.15.26:6790/0 conn(0x7f9c140737f0 legacy=0x7f9c14139fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:35.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 1 -- 172.21.15.189:0/3402477339 shutdown_connections 2024-09-16T14:44:35.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 1 -- 172.21.15.189:0/3402477339 >> 172.21.15.189:0/3402477339 conn(0x7f9c1409fed0 msgr2=0x7f9c140a02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:35.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 1 -- 172.21.15.189:0/3402477339 shutdown_connections 2024-09-16T14:44:35.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 1 -- 172.21.15.189:0/3402477339 wait complete. 2024-09-16T14:44:35.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 10 monclient: build_initial_monmap 2024-09-16T14:44:35.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 10 monclient: monmap: 2024-09-16T14:44:35.636 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:35.636 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:35.636 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:35.636 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:35.636 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:35.636 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:35.636 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:35.636 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:35.636 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:35.636 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:35.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 1 Processor -- start 2024-09-16T14:44:35.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 1 -- start start 2024-09-16T14:44:35.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 10 monclient: init 2024-09-16T14:44:35.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:35.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:44:35.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 10 monclient(hunting): picked mon.noname-c con 0x7f9c140737f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:35.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 10 monclient(hunting): picked mon.noname-b con 0x7f9c14099620 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:35.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 10 monclient(hunting): picked mon.noname-a con 0x7f9c14131a50 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:35.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1409e480 con 0x7f9c14131a50 2024-09-16T14:44:35.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c11d9b640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f9c14099620 legacy=0x7f9c141c9f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:35.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c14098500 con 0x7f9c14099620 2024-09-16T14:44:35.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1409a5c0 con 0x7f9c140737f0 2024-09-16T14:44:35.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:35.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 10 monclient(hunting): authenticate will time out at 1057.742116s 2024-09-16T14:44:35.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c1259c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9c140737f0 0x7f9c141c50a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:35998/0 (socket says 172.21.15.189:35998) 2024-09-16T14:44:35.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c1259c640 1 -- 172.21.15.189:0/3044959558 learned_addr learned my addr 172.21.15.189:0/3044959558 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:35.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c12d9d640 1 --1- 172.21.15.189:0/3044959558 >> v1:172.21.15.26:6789/0 conn(0x7f9c14131a50 0x7f9c141cd630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:52758/0 (socket says 172.21.15.189:52758) 2024-09-16T14:44:35.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189952910 0 0) 0x7f9c1409e480 con 0x7f9c14131a50 2024-09-16T14:44:35.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:35.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:35.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:35.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient(hunting): my global_id is 84204 2024-09-16T14:44:35.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf4003530 con 0x7f9c14131a50 2024-09-16T14:44:35.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245558597 0 0) 0x7f9c1409a5c0 con 0x7f9c140737f0 2024-09-16T14:44:35.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:35.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:35.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:35.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient(hunting): my global_id is 4817 2024-09-16T14:44:35.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c1409e480 con 0x7f9c140737f0 2024-09-16T14:44:35.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1952080392 0 0) 0x7f9bf4003530 con 0x7f9c14131a50 2024-09-16T14:44:35.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c1409a5c0 con 0x7f9c14131a50 2024-09-16T14:44:35.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9c0c001ba0 con 0x7f9c14131a50 2024-09-16T14:44:35.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:35.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:35.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient(hunting): dump: 2024-09-16T14:44:35.639 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:35.639 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:35.639 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.639 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.640 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:35.640 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:35.640 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:35.640 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:35.640 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:35.640 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:35.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:35.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2517676845 0 0) 0x7f9c1409e480 con 0x7f9c140737f0 2024-09-16T14:44:35.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bf4003530 con 0x7f9c140737f0 2024-09-16T14:44:35.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9bfc001cd0 con 0x7f9c140737f0 2024-09-16T14:44:35.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:35.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:35.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient(hunting): dump: 2024-09-16T14:44:35.641 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:35.641 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:35.641 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.641 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.641 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:35.641 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:35.641 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:35.641 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:35.641 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:35.641 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:35.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2735959874 0 0) 0x7f9c1409a5c0 con 0x7f9c14131a50 2024-09-16T14:44:35.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 >> v1:172.21.15.26:6790/0 conn(0x7f9c140737f0 legacy=0x7f9c141c50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:35.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 >> v1:172.21.15.189:6789/0 conn(0x7f9c14099620 legacy=0x7f9c141c9f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:35.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:35.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 1 monclient: found mon.a 2024-09-16T14:44:35.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:35.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:35.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1412ad40 con 0x7f9c14131a50 2024-09-16T14:44:35.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient: _finish_auth 0 2024-09-16T14:44:35.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:35.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:35.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 5 monclient: authenticate success, global_id 84204 2024-09-16T14:44:35.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 10 monclient: _renew_subs 2024-09-16T14:44:35.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:35.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 1 -- 172.21.15.189:0/3044959558 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c14072340 con 0x7f9c14131a50 2024-09-16T14:44:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 10 monclient: _renew_subs 2024-09-16T14:44:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.621+0000 7f9c18c1c640 1 -- 172.21.15.189:0/3044959558 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c14129fb0 con 0x7f9c14131a50 2024-09-16T14:44:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.625+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c0c004b40 con 0x7f9c14131a50 2024-09-16T14:44:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.625+0000 7f9c0affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.625+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9c0c004e50 con 0x7f9c14131a50 2024-09-16T14:44:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.625+0000 7f9c0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.625+0000 7f9c10d99640 4 set_mon_vals no callback set 2024-09-16T14:44:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.625+0000 7f9c0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.625+0000 7f9c0affd640 10 monclient: dump: 2024-09-16T14:44:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:35.644 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:35.644 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:35.644 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:35.644 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:35.644 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:35.644 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:35.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.625+0000 7f9c18c1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:35.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.625+0000 7f9c18c1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:35.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.625+0000 7f9c18c1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:35.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.625+0000 7f9c18c1c640 1 -- 172.21.15.189:0/3044959558 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf8005050 con 0x7f9c14131a50 2024-09-16T14:44:35.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.625+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9c0c00f8c0 con 0x7f9c14131a50 2024-09-16T14:44:35.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.625+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(40..40 src has 1..40) v4 ==== 3598+0+0 (unknown 1780700966 0 0) 0x7f9c0c0431b0 con 0x7f9c14131a50 2024-09-16T14:44:35.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.625+0000 7f9c1259c640 10 monclient: get_auth_request con 0x7f9bf40362c0 auth_method 0 2024-09-16T14:44:35.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.637+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9c0c0145f0 con 0x7f9c14131a50 2024-09-16T14:44:35.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.637+0000 7f9c0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:35.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.637+0000 7f9c0affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:35.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.793+0000 7f9c18c1c640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:35.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.793+0000 7f9c18c1c640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:35.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.793+0000 7f9c18c1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:35.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.793+0000 7f9c18c1c640 1 -- 172.21.15.189:0/3044959558 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9bf8005300 con 0x7f9c14131a50 2024-09-16T14:44:35.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.797+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v40) v1 ==== 159+0+0 (unknown 2860897059 0 0) 0x7f9c0c014410 con 0x7f9c14131a50 2024-09-16T14:44:35.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.797+0000 7f9c0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:35.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.797+0000 7f9c0affd640 10 monclient: _finish_command 2 = system:0 pool 'fooo' does not exist 2024-09-16T14:44:35.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.889+0000 7f9c18c1c640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:35.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.889+0000 7f9c18c1c640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:35.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.889+0000 7f9c18c1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:35.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.889+0000 7f9c18c1c640 1 -- 172.21.15.189:0/3044959558 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9bf8001170 con 0x7f9c14131a50 2024-09-16T14:44:35.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.893+0000 7f9c0affd640 1 -- 172.21.15.189:0/3044959558 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v40) v1 ==== 159+0+0 (unknown 2860897059 0 0) 0x7f9c0c014410 con 0x7f9c14131a50 2024-09-16T14:44:35.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.893+0000 7f9c0affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:35.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.893+0000 7f9c0affd640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-16T14:44:35.899 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fooo' does not exist 2024-09-16T14:44:35.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.893+0000 7f9c18c1c640 1 -- 172.21.15.189:0/3044959558 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9bf40362c0 legacy=0x7f9bf4038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:35.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.893+0000 7f9c18c1c640 10 monclient: shutdown 2024-09-16T14:44:35.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.893+0000 7f9c18c1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.893+0000 7f9c18c1c640 1 -- 172.21.15.189:0/3044959558 >> v1:172.21.15.26:6789/0 conn(0x7f9c14131a50 legacy=0x7f9c141cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.893+0000 7f9c18c1c640 1 -- 172.21.15.189:0/3044959558 shutdown_connections 2024-09-16T14:44:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.893+0000 7f9c18c1c640 1 -- 172.21.15.189:0/3044959558 >> 172.21.15.189:0/3044959558 conn(0x7f9c1409fed0 msgr2=0x7f9c140a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.893+0000 7f9c18c1c640 1 -- 172.21.15.189:0/3044959558 shutdown_connections 2024-09-16T14:44:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:35.897+0000 7f9c18c1c640 1 -- 172.21.15.189:0/3044959558 wait complete. 2024-09-16T14:44:35.916 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-16T14:44:36.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.029+0000 7f076d456640 10 monclient: get_monmap_and_config 2024-09-16T14:44:36.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.029+0000 7f076d456640 10 monclient: build_initial_monmap 2024-09-16T14:44:36.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.029+0000 7f076d456640 10 monclient: monmap: 2024-09-16T14:44:36.036 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:36.036 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.036 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:36.035616+0000 2024-09-16T14:44:36.036 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:36.035616+0000 2024-09-16T14:44:36.036 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:36.036 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.036 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:36.036 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:36.036 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:36.036 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.029+0000 7f076d456640 1 Processor -- start 2024-09-16T14:44:36.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.029+0000 7f076d456640 1 -- start start 2024-09-16T14:44:36.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.029+0000 7f076d456640 10 monclient: init 2024-09-16T14:44:36.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:36.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:36.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient(hunting): picked mon.noname-b con 0x7f07680a1a90 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:36.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient(hunting): picked mon.noname-a con 0x7f07680a2580 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:36.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient(hunting): picked mon.noname-c con 0x7f076809fd40 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:36.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076809df00 con 0x7f07680a2580 2024-09-16T14:44:36.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076809f0a0 con 0x7f076809fd40 2024-09-16T14:44:36.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07680a3040 con 0x7f07680a1a90 2024-09-16T14:44:36.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:36.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient(hunting): authenticate will time out at 1058.152171s 2024-09-16T14:44:36.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07677fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f076809fd40 legacy=0x7f076813d520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:36.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f075ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f07680a2580 0x7f0768139df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:52774/0 (socket says 172.21.15.189:52774) 2024-09-16T14:44:36.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f075ffff640 1 -- 172.21.15.189:0/3471191577 learned_addr learned my addr 172.21.15.189:0/3471191577 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:36.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f0766ffd640 1 --1- 172.21.15.189:0/3471191577 >> v1:172.21.15.26:6790/0 conn(0x7f07680a1a90 0x7f07680a1e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:36014/0 (socket says 172.21.15.189:36014) 2024-09-16T14:44:36.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 1 -- 172.21.15.189:0/3471191577 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32841735 0 0) 0x7f07680a3040 con 0x7f07680a1a90 2024-09-16T14:44:36.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:36.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:36.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:36.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient(hunting): my global_id is 4820 2024-09-16T14:44:36.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 1 -- 172.21.15.189:0/3471191577 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0754003530 con 0x7f07680a1a90 2024-09-16T14:44:36.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 1 -- 172.21.15.189:0/3471191577 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281100479 0 0) 0x7f076809df00 con 0x7f07680a2580 2024-09-16T14:44:36.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:36.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:36.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:36.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient(hunting): my global_id is 84207 2024-09-16T14:44:36.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 1 -- 172.21.15.189:0/3471191577 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07680a3040 con 0x7f07680a2580 2024-09-16T14:44:36.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 1 -- 172.21.15.189:0/3471191577 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1045450731 0 0) 0x7f0754003530 con 0x7f07680a1a90 2024-09-16T14:44:36.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 1 -- 172.21.15.189:0/3471191577 >> v1:172.21.15.189:6789/0 conn(0x7f076809fd40 legacy=0x7f076813d520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:36.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 1 -- 172.21.15.189:0/3471191577 >> v1:172.21.15.26:6789/0 conn(0x7f07680a2580 legacy=0x7f0768139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:36.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient: _finish_hunting 0 2024-09-16T14:44:36.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 1 monclient: found mon.noname-b 2024-09-16T14:44:36.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:36.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:36.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 1 -- 172.21.15.189:0/3471191577 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076809eae0 con 0x7f07680a1a90 2024-09-16T14:44:36.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient: _finish_auth 0 2024-09-16T14:44:36.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient: _check_auth_tickets 2024-09-16T14:44:36.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:36.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 1 -- 172.21.15.189:0/3471191577 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0758001620 con 0x7f07680a1a90 2024-09-16T14:44:36.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:36.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:36.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient: dump: 2024-09-16T14:44:36.044 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:36.044 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.044 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.045 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.045 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:36.045 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.045 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:36.045 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:36.045 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:36.045 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 5 monclient: authenticate success, global_id 4820 2024-09-16T14:44:36.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:36.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 1 -- 172.21.15.189:0/3471191577 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0758001840 con 0x7f07680a1a90 2024-09-16T14:44:36.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:36.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:36.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 1 -- 172.21.15.189:0/3471191577 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0758001b50 con 0x7f07680a1a90 2024-09-16T14:44:36.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 4 set_mon_vals no callback set 2024-09-16T14:44:36.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:36.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:36.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f07667fc640 10 monclient: dump: 2024-09-16T14:44:36.046 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:36.046 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.046 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.046 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.046 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:36.046 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.046 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:36.046 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:36.046 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:36.047 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient: shutdown 2024-09-16T14:44:36.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:36.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 1 -- 172.21.15.189:0/3471191577 >> v1:172.21.15.26:6790/0 conn(0x7f07680a1a90 legacy=0x7f07680a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:36.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 1 -- 172.21.15.189:0/3471191577 shutdown_connections 2024-09-16T14:44:36.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 1 -- 172.21.15.189:0/3471191577 >> 172.21.15.189:0/3471191577 conn(0x7f07680a7a70 msgr2=0x7f07680a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:36.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 1 -- 172.21.15.189:0/3471191577 shutdown_connections 2024-09-16T14:44:36.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 1 -- 172.21.15.189:0/3471191577 wait complete. 2024-09-16T14:44:36.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient: build_initial_monmap 2024-09-16T14:44:36.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient: monmap: 2024-09-16T14:44:36.047 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:36.047 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:36.047 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:36.048 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:36.048 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:36.048 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:36.048 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:36.048 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:36.048 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:36.048 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 1 Processor -- start 2024-09-16T14:44:36.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 1 -- start start 2024-09-16T14:44:36.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient: init 2024-09-16T14:44:36.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:36.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:44:36.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient(hunting): picked mon.noname-b con 0x7f076809fd40 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:36.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient(hunting): picked mon.noname-c con 0x7f07680a1a90 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:36.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient(hunting): picked mon.noname-a con 0x7f07680a2580 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:36.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07680a64f0 con 0x7f07680a2580 2024-09-16T14:44:36.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07681c9740 con 0x7f076809fd40 2024-09-16T14:44:36.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f0766ffd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f076809fd40 legacy=0x7f0768133de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:36.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076809e280 con 0x7f07680a1a90 2024-09-16T14:44:36.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:36.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.033+0000 7f076d456640 10 monclient(hunting): authenticate will time out at 1058.155907s 2024-09-16T14:44:36.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f07677fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f07680a2580 0x7f07681d1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:52778/0 (socket says 172.21.15.189:52778) 2024-09-16T14:44:36.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f07677fe640 1 -- 172.21.15.189:0/1300710782 learned_addr learned my addr 172.21.15.189:0/1300710782 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:36.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460216850 0 0) 0x7f0760003d40 con 0x7f07680a2580 2024-09-16T14:44:36.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:36.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:36.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:36.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient(hunting): my global_id is 84210 2024-09-16T14:44:36.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0748003530 con 0x7f07680a2580 2024-09-16T14:44:36.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582488364 0 0) 0x7f076809e280 con 0x7f07680a1a90 2024-09-16T14:44:36.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:36.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:36.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:36.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient(hunting): my global_id is 4823 2024-09-16T14:44:36.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0760003d40 con 0x7f07680a1a90 2024-09-16T14:44:36.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2028802487 0 0) 0x7f0760003d40 con 0x7f07680a1a90 2024-09-16T14:44:36.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f076809e280 con 0x7f07680a1a90 2024-09-16T14:44:36.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f07500017f0 con 0x7f07680a1a90 2024-09-16T14:44:36.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:36.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:36.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient(hunting): dump: 2024-09-16T14:44:36.053 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:36.053 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.053 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.053 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.053 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:36.053 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.053 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:36.053 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:36.053 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:36.053 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:36.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1140711016 0 0) 0x7f0748003530 con 0x7f07680a2580 2024-09-16T14:44:36.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0760003d40 con 0x7f07680a2580 2024-09-16T14:44:36.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f07600043b0 con 0x7f07680a2580 2024-09-16T14:44:36.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:36.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:36.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient(hunting): dump: 2024-09-16T14:44:36.054 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:36.054 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.054 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.054 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.054 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:36.054 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.054 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:36.054 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:36.054 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:36.055 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2211583986 0 0) 0x7f076809e280 con 0x7f07680a1a90 2024-09-16T14:44:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 >> v1:172.21.15.189:6789/0 conn(0x7f076809fd40 legacy=0x7f0768133de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 >> v1:172.21.15.26:6789/0 conn(0x7f07680a2580 legacy=0x7f07681d1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:44:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 monclient: found mon.c 2024-09-16T14:44:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0768123f30 con 0x7f07680a1a90 2024-09-16T14:44:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:44:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:44:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f076d456640 5 monclient: authenticate success, global_id 4823 2024-09-16T14:44:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f076d456640 10 monclient: _renew_subs 2024-09-16T14:44:36.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f076d456640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:36.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f076d456640 1 -- 172.21.15.189:0/1300710782 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0768124210 con 0x7f07680a1a90 2024-09-16T14:44:36.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f076d456640 10 monclient: _renew_subs 2024-09-16T14:44:36.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f076d456640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:36.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f076d456640 1 -- 172.21.15.189:0/1300710782 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0768123160 con 0x7f07680a1a90 2024-09-16T14:44:36.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07500047f0 con 0x7f07680a1a90 2024-09-16T14:44:36.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:36.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f0765ffb640 4 set_mon_vals no callback set 2024-09-16T14:44:36.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0750004b00 con 0x7f07680a1a90 2024-09-16T14:44:36.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:36.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:36.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 10 monclient: dump: 2024-09-16T14:44:36.056 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:36.056 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.057 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.057 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.057 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:36.057 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.057 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:36.057 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:36.057 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:36.057 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f076d456640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:36.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f076d456640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:36.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f076d456640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:36.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f076d456640 1 -- 172.21.15.189:0/1300710782 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f074c005050 con 0x7f07680a1a90 2024-09-16T14:44:36.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f075000f570 con 0x7f07680a1a90 2024-09-16T14:44:36.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(40..40 src has 1..40) v4 ==== 3598+0+0 (unknown 1780700966 0 0) 0x7f0750043030 con 0x7f07680a1a90 2024-09-16T14:44:36.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.037+0000 7f0766ffd640 10 monclient: get_auth_request con 0x7f0748036220 auth_method 0 2024-09-16T14:44:36.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.049+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0750014330 con 0x7f07680a1a90 2024-09-16T14:44:36.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.049+0000 7f075f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:36.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.049+0000 7f075f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:36.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.229+0000 7f076d456640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:36.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.229+0000 7f076d456640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:36.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.229+0000 7f076d456640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:36.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.229+0000 7f076d456640 1 -- 172.21.15.189:0/1300710782 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f074c005300 con 0x7f07680a1a90 2024-09-16T14:44:36.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.229+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v40) v1 ==== 159+0+0 (unknown 2860897059 0 0) 0x7f0750014150 con 0x7f07680a1a90 2024-09-16T14:44:36.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.229+0000 7f075f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:36.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.229+0000 7f075f7fe640 10 monclient: _finish_command 2 = system:0 pool 'fooo' does not exist 2024-09-16T14:44:36.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.325+0000 7f076d456640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:36.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.325+0000 7f076d456640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:36.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.325+0000 7f076d456640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:36.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.325+0000 7f076d456640 1 -- 172.21.15.189:0/1300710782 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f074c001170 con 0x7f07680a1a90 2024-09-16T14:44:36.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.329+0000 7f075f7fe640 1 -- 172.21.15.189:0/1300710782 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v40) v1 ==== 159+0+0 (unknown 2860897059 0 0) 0x7f0750014e90 con 0x7f07680a1a90 2024-09-16T14:44:36.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.329+0000 7f075f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:36.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.329+0000 7f075f7fe640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-16T14:44:36.334 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fooo' does not exist 2024-09-16T14:44:36.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.329+0000 7f076d456640 1 -- 172.21.15.189:0/1300710782 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0748036220 legacy=0x7f07480386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:36.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.329+0000 7f076d456640 10 monclient: shutdown 2024-09-16T14:44:36.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.329+0000 7f076d456640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:36.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.329+0000 7f076d456640 1 -- 172.21.15.189:0/1300710782 >> v1:172.21.15.26:6790/0 conn(0x7f07680a1a90 legacy=0x7f07681ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:36.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.329+0000 7f076d456640 1 -- 172.21.15.189:0/1300710782 shutdown_connections 2024-09-16T14:44:36.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.329+0000 7f076d456640 1 -- 172.21.15.189:0/1300710782 >> 172.21.15.189:0/1300710782 conn(0x7f07680a7a70 msgr2=0x7f076806b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:36.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.329+0000 7f076d456640 1 -- 172.21.15.189:0/1300710782 shutdown_connections 2024-09-16T14:44:36.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.329+0000 7f076d456640 1 -- 172.21.15.189:0/1300710782 wait complete. 2024-09-16T14:44:36.350 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete fuggg fuggg --yes-i-really-really-mean-it 2024-09-16T14:44:36.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f23d5a640 10 monclient: get_monmap_and_config 2024-09-16T14:44:36.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f23d5a640 10 monclient: build_initial_monmap 2024-09-16T14:44:36.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f23d5a640 10 monclient: monmap: 2024-09-16T14:44:36.467 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:36.467 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.467 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:36.467091+0000 2024-09-16T14:44:36.467 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:36.467091+0000 2024-09-16T14:44:36.467 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:36.467 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.467 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:36.468 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:36.468 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:36.468 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f23d5a640 1 Processor -- start 2024-09-16T14:44:36.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f23d5a640 1 -- start start 2024-09-16T14:44:36.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f23d5a640 10 monclient: init 2024-09-16T14:44:36.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f23d5a640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:36.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f23d5a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:36.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f23d5a640 10 monclient(hunting): picked mon.noname-a con 0x7f3f1c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:36.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f23d5a640 10 monclient(hunting): picked mon.noname-b con 0x7f3f1c09c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:36.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f23d5a640 10 monclient(hunting): picked mon.noname-c con 0x7f3f1c09e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:36.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f23d5a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1c099e50 con 0x7f3f1c099a70 2024-09-16T14:44:36.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f23d5a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1c09dbd0 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f23d5a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1c09ee20 con 0x7f3f1c09c550 2024-09-16T14:44:36.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f23d5a640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:36.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f23d5a640 10 monclient(hunting): authenticate will time out at 1058.582744s 2024-09-16T14:44:36.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f222f7640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f3f1c09e6d0 legacy=0x7f3f1c13b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:36.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f212f5640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3f1c09c550 0x7f3f1c1379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:36040/0 (socket says 172.21.15.189:36040) 2024-09-16T14:44:36.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f212f5640 1 -- 172.21.15.189:0/1245606352 learned_addr learned my addr 172.21.15.189:0/1245606352 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:36.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f20af4640 1 -- 172.21.15.189:0/1245606352 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185832129 0 0) 0x7f3f1c09ee20 con 0x7f3f1c09c550 2024-09-16T14:44:36.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f20af4640 10 monclient(hunting): _negotiate 2024-09-16T14:44:36.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f20af4640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:36.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f20af4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:36.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f20af4640 10 monclient(hunting): my global_id is 4826 2024-09-16T14:44:36.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f20af4640 1 -- 172.21.15.189:0/1245606352 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f08003530 con 0x7f3f1c09c550 2024-09-16T14:44:36.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f20af4640 1 -- 172.21.15.189:0/1245606352 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418488960 0 0) 0x7f3f1c099e50 con 0x7f3f1c099a70 2024-09-16T14:44:36.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f20af4640 10 monclient(hunting): _negotiate 2024-09-16T14:44:36.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f20af4640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:36.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f20af4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:36.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f20af4640 10 monclient(hunting): my global_id is 84213 2024-09-16T14:44:36.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.461+0000 7f3f20af4640 1 -- 172.21.15.189:0/1245606352 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f1c09ee20 con 0x7f3f1c099a70 2024-09-16T14:44:36.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 1 -- 172.21.15.189:0/1245606352 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1860506280 0 0) 0x7f3f08003530 con 0x7f3f1c09c550 2024-09-16T14:44:36.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 1 -- 172.21.15.189:0/1245606352 >> v1:172.21.15.189:6789/0 conn(0x7f3f1c09e6d0 legacy=0x7f3f1c13b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:36.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 1 -- 172.21.15.189:0/1245606352 >> v1:172.21.15.26:6789/0 conn(0x7f3f1c099a70 legacy=0x7f3f1c09dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:36.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 10 monclient: _finish_hunting 0 2024-09-16T14:44:36.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 1 monclient: found mon.noname-b 2024-09-16T14:44:36.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:36.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:36.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 1 -- 172.21.15.189:0/1245606352 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1c09f100 con 0x7f3f1c09c550 2024-09-16T14:44:36.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 10 monclient: _finish_auth 0 2024-09-16T14:44:36.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 10 monclient: _check_auth_tickets 2024-09-16T14:44:36.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:36.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 1 -- 172.21.15.189:0/1245606352 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f0c001550 con 0x7f3f1c09c550 2024-09-16T14:44:36.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 5 monclient: authenticate success, global_id 4826 2024-09-16T14:44:36.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:36.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:36.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 10 monclient: dump: 2024-09-16T14:44:36.481 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:36.481 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.481 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.481 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.481 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:36.481 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.481 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:36.481 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:36.481 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:36.482 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:36.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 1 -- 172.21.15.189:0/1245606352 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f0c001770 con 0x7f3f1c09c550 2024-09-16T14:44:36.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:36.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 1 -- 172.21.15.189:0/1245606352 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f0c001ac0 con 0x7f3f1c09c550 2024-09-16T14:44:36.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:36.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:36.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f20af4640 10 monclient: dump: 2024-09-16T14:44:36.482 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:36.482 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.482 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.482 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.482 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:36.482 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.483 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:36.483 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:36.483 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:36.483 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:36.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 4 set_mon_vals no callback set 2024-09-16T14:44:36.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 10 monclient: shutdown 2024-09-16T14:44:36.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:36.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 1 -- 172.21.15.189:0/1245606352 >> v1:172.21.15.26:6790/0 conn(0x7f3f1c09c550 legacy=0x7f3f1c1379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:36.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 1 -- 172.21.15.189:0/1245606352 shutdown_connections 2024-09-16T14:44:36.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 1 -- 172.21.15.189:0/1245606352 >> 172.21.15.189:0/1245606352 conn(0x7f3f1c0a3380 msgr2=0x7f3f1c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:36.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 1 -- 172.21.15.189:0/1245606352 shutdown_connections 2024-09-16T14:44:36.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 1 -- 172.21.15.189:0/1245606352 wait complete. 2024-09-16T14:44:36.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 10 monclient: build_initial_monmap 2024-09-16T14:44:36.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 10 monclient: monmap: 2024-09-16T14:44:36.484 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:36.484 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:36.484 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:36.484 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:36.484 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:36.484 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:36.484 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:36.484 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:36.484 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:36.484 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 1 Processor -- start 2024-09-16T14:44:36.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 1 -- start start 2024-09-16T14:44:36.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 10 monclient: init 2024-09-16T14:44:36.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:36.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:44:36.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 10 monclient(hunting): picked mon.noname-c con 0x7f3f1c099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:36.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 10 monclient(hunting): picked mon.noname-b con 0x7f3f1c09c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:36.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 10 monclient(hunting): picked mon.noname-a con 0x7f3f1c09e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:36.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1c09cf20 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1c06cda0 con 0x7f3f1c09c550 2024-09-16T14:44:36.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1c06b320 con 0x7f3f1c099a70 2024-09-16T14:44:36.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:36.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f23d5a640 10 monclient(hunting): authenticate will time out at 1058.586969s 2024-09-16T14:44:36.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f212f5640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f3f1c09c550 legacy=0x7f3f1c13a290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:36.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f222f7640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3f1c09e6d0 0x7f3f1c1cf700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:52802/0 (socket says 172.21.15.189:52802) 2024-09-16T14:44:36.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f21af6640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3f1c099a70 0x7f3f1c139b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:36056/0 (socket says 172.21.15.189:36056) 2024-09-16T14:44:36.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f222f7640 1 -- 172.21.15.189:0/3434575188 learned_addr learned my addr 172.21.15.189:0/3434575188 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:36.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280540673 0 0) 0x7f3f1c09cf20 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f127fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:36.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:36.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:36.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f127fc640 10 monclient(hunting): my global_id is 84216 2024-09-16T14:44:36.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f00003560 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910851685 0 0) 0x7f3f1c06b320 con 0x7f3f1c099a70 2024-09-16T14:44:36.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f127fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:36.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:36.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:36.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f127fc640 10 monclient(hunting): my global_id is 4829 2024-09-16T14:44:36.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.465+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f1c09cf20 con 0x7f3f1c099a70 2024-09-16T14:44:36.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2058406011 0 0) 0x7f3f00003560 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f1c06b320 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4222843465 0 0) 0x7f3f1c09cf20 con 0x7f3f1c099a70 2024-09-16T14:44:36.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f00003560 con 0x7f3f1c099a70 2024-09-16T14:44:36.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f140043c0 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:36.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:36.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 10 monclient(hunting): dump: 2024-09-16T14:44:36.487 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:36.488 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.488 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.488 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.488 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:36.488 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.488 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:36.488 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:36.488 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:36.488 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:36.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f18001700 con 0x7f3f1c099a70 2024-09-16T14:44:36.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:36.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:36.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 10 monclient(hunting): dump: 2024-09-16T14:44:36.489 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:36.489 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.489 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.489 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.489 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:36.489 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.489 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:36.489 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:36.489 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:36.489 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4027589112 0 0) 0x7f3f1c06b320 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 >> v1:172.21.15.26:6790/0 conn(0x7f3f1c099a70 legacy=0x7f3f1c139b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:36.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 >> v1:172.21.15.189:6789/0 conn(0x7f3f1c09c550 legacy=0x7f3f1c13a290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:36.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 10 monclient: _finish_hunting 0 2024-09-16T14:44:36.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 1 monclient: found mon.a 2024-09-16T14:44:36.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:36.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:36.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1c11a060 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 10 monclient: _finish_auth 0 2024-09-16T14:44:36.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 10 monclient: _check_auth_tickets 2024-09-16T14:44:36.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:36.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f23d5a640 5 monclient: authenticate success, global_id 84216 2024-09-16T14:44:36.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f23d5a640 10 monclient: _renew_subs 2024-09-16T14:44:36.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f23d5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:36.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f23d5a640 1 -- 172.21.15.189:0/3434575188 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1c119d80 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f23d5a640 10 monclient: _renew_subs 2024-09-16T14:44:36.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f23d5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:36.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f23d5a640 1 -- 172.21.15.189:0/3434575188 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1c13b730 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f140045e0 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:36.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f14004970 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:36.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f137fe640 4 set_mon_vals no callback set 2024-09-16T14:44:36.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:36.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 10 monclient: dump: 2024-09-16T14:44:36.491 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:36.492 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.492 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.492 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.492 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:36.492 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.492 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:36.492 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:36.492 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:36.492 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f23d5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:36.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f23d5a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:36.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f23d5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:36.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f23d5a640 1 -- 172.21.15.189:0/3434575188 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f04005050 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3f14004b50 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(40..40 src has 1..40) v4 ==== 3598+0+0 (unknown 1780700966 0 0) 0x7f3f140432d0 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.469+0000 7f3f21af6640 10 monclient: get_auth_request con 0x7f3f00036290 auth_method 0 2024-09-16T14:44:36.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.481+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3f14014690 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.481+0000 7f3f127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:36.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.481+0000 7f3f127fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:36.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.641+0000 7f3f23d5a640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:36.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.641+0000 7f3f23d5a640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:36.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.641+0000 7f3f23d5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:36.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.641+0000 7f3f23d5a640 1 -- 172.21.15.189:0/3434575188 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3f04005300 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.641+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}]=0 pool 'fuggg' does not exist v40) v1 ==== 162+0+0 (unknown 193637622 0 0) 0x7f3f14012d30 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.641+0000 7f3f127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:36.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.641+0000 7f3f127fc640 10 monclient: _finish_command 2 = system:0 pool 'fuggg' does not exist 2024-09-16T14:44:36.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.737+0000 7f3f23d5a640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:36.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.737+0000 7f3f23d5a640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:36.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.737+0000 7f3f23d5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:36.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.737+0000 7f3f23d5a640 1 -- 172.21.15.189:0/3434575188 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3f04001170 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.737+0000 7f3f127fc640 1 -- 172.21.15.189:0/3434575188 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}]=0 pool 'fuggg' does not exist v40) v1 ==== 162+0+0 (unknown 193637622 0 0) 0x7f3f140144b0 con 0x7f3f1c09e6d0 2024-09-16T14:44:36.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.737+0000 7f3f127fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:36.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.737+0000 7f3f127fc640 10 monclient: _finish_command 3 = system:0 pool 'fuggg' does not exist 2024-09-16T14:44:36.744 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fuggg' does not exist 2024-09-16T14:44:36.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.741+0000 7f3f23d5a640 1 -- 172.21.15.189:0/3434575188 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3f00036290 legacy=0x7f3f00038750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:36.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.741+0000 7f3f23d5a640 10 monclient: shutdown 2024-09-16T14:44:36.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.741+0000 7f3f23d5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:36.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.741+0000 7f3f23d5a640 1 -- 172.21.15.189:0/3434575188 >> v1:172.21.15.26:6789/0 conn(0x7f3f1c09e6d0 legacy=0x7f3f1c1cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:36.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.741+0000 7f3f23d5a640 1 -- 172.21.15.189:0/3434575188 shutdown_connections 2024-09-16T14:44:36.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.741+0000 7f3f23d5a640 1 -- 172.21.15.189:0/3434575188 >> 172.21.15.189:0/3434575188 conn(0x7f3f1c0a3380 msgr2=0x7f3f1c0a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:36.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.741+0000 7f3f23d5a640 1 -- 172.21.15.189:0/3434575188 shutdown_connections 2024-09-16T14:44:36.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.741+0000 7f3f23d5a640 1 -- 172.21.15.189:0/3434575188 wait complete. 2024-09-16T14:44:36.761 INFO:tasks.workunit.client.0.smithi189.stdout:OK 2024-09-16T14:44:36.762 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo OK 2024-09-16T14:44:36.762 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T14:44:36.762 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T14:44:36.774 INFO:tasks.workunit:Running workunits matching mon/crush_ops.sh on client.0... 2024-09-16T14:44:36.775 INFO:tasks.workunit:Running workunit mon/crush_ops.sh... 2024-09-16T14:44:36.775 DEBUG:teuthology.orchestra.run.smithi189:workunit test mon/crush_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh 2024-09-16T14:44:36.826 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush dump 2024-09-16T14:44:36.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.929+0000 7f2bc1d2e640 10 monclient: get_monmap_and_config 2024-09-16T14:44:36.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.929+0000 7f2bc1d2e640 10 monclient: build_initial_monmap 2024-09-16T14:44:36.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.929+0000 7f2bc1d2e640 10 monclient: monmap: 2024-09-16T14:44:36.937 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:36.938 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.938 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:36.937502+0000 2024-09-16T14:44:36.938 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:36.937502+0000 2024-09-16T14:44:36.938 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:36.938 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.938 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:36.938 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:36.938 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:36.938 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 1 Processor -- start 2024-09-16T14:44:36.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 1 -- start start 2024-09-16T14:44:36.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 10 monclient: init 2024-09-16T14:44:36.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:36.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:36.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 10 monclient(hunting): picked mon.noname-c con 0x7f2bbc0a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:36.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 10 monclient(hunting): picked mon.noname-a con 0x7f2bbc0a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:36.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 10 monclient(hunting): picked mon.noname-b con 0x7f2bbc13b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:36.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc0a30b0 con 0x7f2bbc0a25f0 2024-09-16T14:44:36.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc0a2e30 con 0x7f2bbc0a1b00 2024-09-16T14:44:36.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc09eb50 con 0x7f2bbc13b430 2024-09-16T14:44:36.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:36.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbb7fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2bbc0a1b00 legacy=0x7f2bbc0a1ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:36.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 10 monclient(hunting): authenticate will time out at 1059.053254s 2024-09-16T14:44:36.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bb2ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2bbc0a25f0 0x7f2bbc139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:52804/0 (socket says 172.21.15.189:52804) 2024-09-16T14:44:36.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bb2ffd640 1 -- 172.21.15.189:0/4276101541 learned_addr learned my addr 172.21.15.189:0/4276101541 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:36.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbbfff640 1 --1- 172.21.15.189:0/4276101541 >> v1:172.21.15.26:6790/0 conn(0x7f2bbc13b430 0x7f2bbc13d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:36060/0 (socket says 172.21.15.189:36060) 2024-09-16T14:44:36.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 1 -- 172.21.15.189:0/4276101541 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50495293 0 0) 0x7f2bbc09eb50 con 0x7f2bbc13b430 2024-09-16T14:44:36.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:36.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:36.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:36.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient(hunting): my global_id is 4832 2024-09-16T14:44:36.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 1 -- 172.21.15.189:0/4276101541 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba8003530 con 0x7f2bbc13b430 2024-09-16T14:44:36.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 1 -- 172.21.15.189:0/4276101541 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674756418 0 0) 0x7f2bbc0a30b0 con 0x7f2bbc0a25f0 2024-09-16T14:44:36.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:36.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:36.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:36.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient(hunting): my global_id is 84219 2024-09-16T14:44:36.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 1 -- 172.21.15.189:0/4276101541 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bbc09eb50 con 0x7f2bbc0a25f0 2024-09-16T14:44:36.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 1 -- 172.21.15.189:0/4276101541 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1995043218 0 0) 0x7f2ba8003530 con 0x7f2bbc13b430 2024-09-16T14:44:36.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 1 -- 172.21.15.189:0/4276101541 >> v1:172.21.15.189:6789/0 conn(0x7f2bbc0a1b00 legacy=0x7f2bbc0a1ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:36.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 1 -- 172.21.15.189:0/4276101541 >> v1:172.21.15.26:6789/0 conn(0x7f2bbc0a25f0 legacy=0x7f2bbc139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:36.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:36.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 1 monclient: found mon.noname-b 2024-09-16T14:44:36.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:36.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:36.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 1 -- 172.21.15.189:0/4276101541 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc09ee30 con 0x7f2bbc13b430 2024-09-16T14:44:36.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient: _finish_auth 0 2024-09-16T14:44:36.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:36.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:36.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 5 monclient: authenticate success, global_id 4832 2024-09-16T14:44:36.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:36.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 1 -- 172.21.15.189:0/4276101541 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2bb40018d0 con 0x7f2bbc13b430 2024-09-16T14:44:36.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:36.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:36.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient: dump: 2024-09-16T14:44:36.946 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:36.946 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.946 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.946 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.946 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:36.946 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.946 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:36.946 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:36.946 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:36.946 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:36.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 1 -- 172.21.15.189:0/4276101541 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bb4001af0 con 0x7f2bbc13b430 2024-09-16T14:44:36.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:36.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:36.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 1 -- 172.21.15.189:0/4276101541 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2bb4001e80 con 0x7f2bbc13b430 2024-09-16T14:44:36.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:36.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 4 set_mon_vals no callback set 2024-09-16T14:44:36.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:36.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bbaffd640 10 monclient: dump: 2024-09-16T14:44:36.947 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:36.947 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.947 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.947 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.947 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:36.947 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.947 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:36.948 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:36.948 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:36.948 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.933+0000 7f2bc1d2e640 10 monclient: shutdown 2024-09-16T14:44:36.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:36.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 1 -- 172.21.15.189:0/4276101541 >> v1:172.21.15.26:6790/0 conn(0x7f2bbc13b430 legacy=0x7f2bbc13d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:36.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 1 -- 172.21.15.189:0/4276101541 shutdown_connections 2024-09-16T14:44:36.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 1 -- 172.21.15.189:0/4276101541 >> 172.21.15.189:0/4276101541 conn(0x7f2bbc0a7dd0 msgr2=0x7f2bbc0a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:36.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 1 -- 172.21.15.189:0/4276101541 shutdown_connections 2024-09-16T14:44:36.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 1 -- 172.21.15.189:0/4276101541 wait complete. 2024-09-16T14:44:36.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 10 monclient: build_initial_monmap 2024-09-16T14:44:36.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 10 monclient: monmap: 2024-09-16T14:44:36.952 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:36.952 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:36.952 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:36.952 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:36.952 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:36.953 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:36.953 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:36.953 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:36.953 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:36.953 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 1 Processor -- start 2024-09-16T14:44:36.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 1 -- start start 2024-09-16T14:44:36.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 10 monclient: init 2024-09-16T14:44:36.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:36.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:36.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 10 monclient(hunting): picked mon.noname-b con 0x7f2bbc0a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:36.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 10 monclient(hunting): picked mon.noname-a con 0x7f2bbc0a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:36.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 10 monclient(hunting): picked mon.noname-c con 0x7f2bbc13b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:36.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc13ae40 con 0x7f2bbc0a25f0 2024-09-16T14:44:36.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc09bc50 con 0x7f2bbc0a1b00 2024-09-16T14:44:36.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc09e2f0 con 0x7f2bbc13b430 2024-09-16T14:44:36.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:36.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bc1d2e640 10 monclient(hunting): authenticate will time out at 1059.057971s 2024-09-16T14:44:36.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bbb7fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2bbc0a1b00 legacy=0x7f2bbc099df0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:36.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bbbfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f2bbc13b430 0x7f2bbc1d1600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:36068/0 (socket says 172.21.15.189:36068) 2024-09-16T14:44:36.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb2ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2bbc0a25f0 0x7f2bbc09a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:52806/0 (socket says 172.21.15.189:52806) 2024-09-16T14:44:36.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bbbfff640 1 -- 172.21.15.189:0/1151691045 learned_addr learned my addr 172.21.15.189:0/1151691045 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:36.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558582671 0 0) 0x7f2bbc13ae40 con 0x7f2bbc0a25f0 2024-09-16T14:44:36.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:36.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:36.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:36.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 10 monclient(hunting): my global_id is 84222 2024-09-16T14:44:36.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b9c003530 con 0x7f2bbc0a25f0 2024-09-16T14:44:36.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866629534 0 0) 0x7f2bbc09e2f0 con 0x7f2bbc13b430 2024-09-16T14:44:36.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:36.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:36.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:36.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 10 monclient(hunting): my global_id is 4835 2024-09-16T14:44:36.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bbc13ae40 con 0x7f2bbc13b430 2024-09-16T14:44:36.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2762370137 0 0) 0x7f2b9c003530 con 0x7f2bbc0a25f0 2024-09-16T14:44:36.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bbc09e2f0 con 0x7f2bbc0a25f0 2024-09-16T14:44:36.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1376427007 0 0) 0x7f2bbc13ae40 con 0x7f2bbc13b430 2024-09-16T14:44:36.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b9c003530 con 0x7f2bbc13b430 2024-09-16T14:44:36.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2bb4001e00 con 0x7f2bbc13b430 2024-09-16T14:44:36.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:36.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:36.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 10 monclient(hunting): dump: 2024-09-16T14:44:36.956 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:36.956 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.956 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.956 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.956 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:36.956 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.956 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:36.956 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:36.957 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:36.957 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:36.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2ba4001960 con 0x7f2bbc0a25f0 2024-09-16T14:44:36.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:36.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:36.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.937+0000 7f2bb8ff9640 10 monclient(hunting): dump: 2024-09-16T14:44:36.957 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:36.957 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.957 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.957 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.957 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:36.957 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.958 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:36.958 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:36.958 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:36.958 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2430952968 0 0) 0x7f2b9c003530 con 0x7f2bbc13b430 2024-09-16T14:44:36.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 >> v1:172.21.15.189:6789/0 conn(0x7f2bbc0a1b00 legacy=0x7f2bbc099df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:36.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 >> v1:172.21.15.26:6789/0 conn(0x7f2bbc0a25f0 legacy=0x7f2bbc09a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:36.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:44:36.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 1 monclient: found mon.c 2024-09-16T14:44:36.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:36.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:36.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc136220 con 0x7f2bbc13b430 2024-09-16T14:44:36.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 10 monclient: _finish_auth 0 2024-09-16T14:44:36.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:44:36.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:36.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bc1d2e640 5 monclient: authenticate success, global_id 4835 2024-09-16T14:44:36.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bc1d2e640 10 monclient: _renew_subs 2024-09-16T14:44:36.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bc1d2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:36.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bc1d2e640 1 -- 172.21.15.189:0/1151691045 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bbc09c750 con 0x7f2bbc13b430 2024-09-16T14:44:36.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bc1d2e640 10 monclient: _renew_subs 2024-09-16T14:44:36.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bc1d2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:36.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bc1d2e640 1 -- 172.21.15.189:0/1151691045 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bbc13aad0 con 0x7f2bbc13b430 2024-09-16T14:44:36.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bb4004e20 con 0x7f2bbc13b430 2024-09-16T14:44:36.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:36.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2bb4005130 con 0x7f2bbc13b430 2024-09-16T14:44:36.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:36.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:36.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb9ffb640 4 set_mon_vals no callback set 2024-09-16T14:44:36.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 10 monclient: dump: 2024-09-16T14:44:36.960 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:36.960 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:36.960 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.960 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:36.960 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:36.960 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:36.960 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:36.960 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:36.960 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:36.961 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:36.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bc1d2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:36.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bc1d2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:36.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bc1d2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:36.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bc1d2e640 1 -- 172.21.15.189:0/1151691045 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba0005050 con 0x7f2bbc13b430 2024-09-16T14:44:36.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2bb400fba0 con 0x7f2bbc13b430 2024-09-16T14:44:36.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(40..40 src has 1..40) v4 ==== 3598+0+0 (unknown 1780700966 0 0) 0x7f2bb4043570 con 0x7f2bbc13b430 2024-09-16T14:44:36.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.941+0000 7f2bbb7fe640 10 monclient: get_auth_request con 0x7f2b9c0361f0 auth_method 0 2024-09-16T14:44:36.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.957+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2bb4014930 con 0x7f2bbc13b430 2024-09-16T14:44:36.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.957+0000 7f2bb8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:36.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:36.957+0000 7f2bb8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:37.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.113+0000 7f2bc1d2e640 10 start_mon_command cmd=[{"prefix": "osd crush dump"}] 2024-09-16T14:44:37.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.113+0000 7f2bc1d2e640 10 monclient: _send_command 2 [{"prefix": "osd crush dump"}] 2024-09-16T14:44:37.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.113+0000 7f2bc1d2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:37.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.113+0000 7f2bc1d2e640 1 -- 172.21.15.189:0/1151691045 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush dump"} v 0) v1 -- 0x7f2ba0005300 con 0x7f2bbc13b430 2024-09-16T14:44:37.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.113+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush dump"}]=0 v40) v1 ==== 62+0+7238 (unknown 3671628665 0 446956320) 0x7f2bb4014750 con 0x7f2bbc13b430 2024-09-16T14:44:37.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.113+0000 7f2bb8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush dump"}] 2024-09-16T14:44:37.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.113+0000 7f2bb8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:37.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.209+0000 7f2bc1d2e640 10 start_mon_command cmd=[{"prefix": "osd crush dump"}] 2024-09-16T14:44:37.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.209+0000 7f2bc1d2e640 10 monclient: _send_command 3 [{"prefix": "osd crush dump"}] 2024-09-16T14:44:37.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.209+0000 7f2bc1d2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:37.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.209+0000 7f2bc1d2e640 1 -- 172.21.15.189:0/1151691045 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush dump"} v 0) v1 -- 0x7f2ba0005540 con 0x7f2bbc13b430 2024-09-16T14:44:37.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.209+0000 7f2bb8ff9640 1 -- 172.21.15.189:0/1151691045 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush dump"}]=0 v40) v1 ==== 62+0+7238 (unknown 3671628665 0 446956320) 0x7f2bb4014930 con 0x7f2bbc13b430 2024-09-16T14:44:37.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.213+0000 7f2bb8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush dump"}] 2024-09-16T14:44:37.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.213+0000 7f2bb8ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:44:37.218 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2024-09-16T14:44:37.218 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": [ 2024-09-16T14:44:37.218 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.218 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2024-09-16T14:44:37.218 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.0", 2024-09-16T14:44:37.218 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "ssd" 2024-09-16T14:44:37.218 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.218 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.218 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2024-09-16T14:44:37.218 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.1", 2024-09-16T14:44:37.219 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "ssd" 2024-09-16T14:44:37.219 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.219 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.219 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2024-09-16T14:44:37.219 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.2", 2024-09-16T14:44:37.219 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "ssd" 2024-09-16T14:44:37.219 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.219 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.219 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 3, 2024-09-16T14:44:37.219 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.3", 2024-09-16T14:44:37.219 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "ssd" 2024-09-16T14:44:37.219 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.219 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.219 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 4, 2024-09-16T14:44:37.220 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.4", 2024-09-16T14:44:37.220 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "ssd" 2024-09-16T14:44:37.220 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.220 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.220 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 5, 2024-09-16T14:44:37.220 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.5", 2024-09-16T14:44:37.220 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "ssd" 2024-09-16T14:44:37.220 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:44:37.220 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:44:37.220 INFO:tasks.workunit.client.0.smithi189.stdout: "types": [ 2024-09-16T14:44:37.220 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.220 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 0, 2024-09-16T14:44:37.220 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd" 2024-09-16T14:44:37.221 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.221 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.221 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2024-09-16T14:44:37.221 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "host" 2024-09-16T14:44:37.221 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.221 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.221 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 2, 2024-09-16T14:44:37.221 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "chassis" 2024-09-16T14:44:37.221 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.221 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.221 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 3, 2024-09-16T14:44:37.221 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "rack" 2024-09-16T14:44:37.221 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.222 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.222 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 4, 2024-09-16T14:44:37.222 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "row" 2024-09-16T14:44:37.222 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.222 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.222 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 5, 2024-09-16T14:44:37.222 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pdu" 2024-09-16T14:44:37.222 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.222 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.222 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 6, 2024-09-16T14:44:37.222 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pod" 2024-09-16T14:44:37.222 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.222 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.223 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 7, 2024-09-16T14:44:37.223 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "room" 2024-09-16T14:44:37.223 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.223 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.223 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 8, 2024-09-16T14:44:37.223 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "datacenter" 2024-09-16T14:44:37.223 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.223 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.223 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 9, 2024-09-16T14:44:37.223 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "zone" 2024-09-16T14:44:37.223 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.223 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.223 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 10, 2024-09-16T14:44:37.223 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "region" 2024-09-16T14:44:37.224 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.224 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.224 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2024-09-16T14:44:37.224 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "root" 2024-09-16T14:44:37.224 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:44:37.224 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:44:37.224 INFO:tasks.workunit.client.0.smithi189.stdout: "buckets": [ 2024-09-16T14:44:37.224 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.224 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -1, 2024-09-16T14:44:37.224 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default", 2024-09-16T14:44:37.224 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2024-09-16T14:44:37.224 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2024-09-16T14:44:37.224 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 34560, 2024-09-16T14:44:37.225 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2024-09-16T14:44:37.225 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2024-09-16T14:44:37.225 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2024-09-16T14:44:37.225 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.225 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2024-09-16T14:44:37.225 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 17280, 2024-09-16T14:44:37.225 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2024-09-16T14:44:37.225 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.225 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.225 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -5, 2024-09-16T14:44:37.225 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 17280, 2024-09-16T14:44:37.225 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2024-09-16T14:44:37.226 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:44:37.226 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:44:37.226 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.226 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.226 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -2, 2024-09-16T14:44:37.226 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default~ssd", 2024-09-16T14:44:37.226 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2024-09-16T14:44:37.226 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2024-09-16T14:44:37.226 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 34560, 2024-09-16T14:44:37.226 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2024-09-16T14:44:37.226 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2024-09-16T14:44:37.226 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2024-09-16T14:44:37.226 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.226 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2024-09-16T14:44:37.227 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 17280, 2024-09-16T14:44:37.227 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2024-09-16T14:44:37.227 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.227 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.227 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -6, 2024-09-16T14:44:37.227 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 17280, 2024-09-16T14:44:37.227 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2024-09-16T14:44:37.227 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:44:37.227 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:44:37.227 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.227 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.227 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2024-09-16T14:44:37.227 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi026", 2024-09-16T14:44:37.228 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2024-09-16T14:44:37.228 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2024-09-16T14:44:37.228 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 17280, 2024-09-16T14:44:37.228 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2024-09-16T14:44:37.228 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2024-09-16T14:44:37.228 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2024-09-16T14:44:37.228 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.228 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2024-09-16T14:44:37.228 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T14:44:37.228 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2024-09-16T14:44:37.228 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.228 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.228 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2024-09-16T14:44:37.229 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T14:44:37.229 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2024-09-16T14:44:37.229 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.229 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.229 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2024-09-16T14:44:37.229 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T14:44:37.229 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2024-09-16T14:44:37.229 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:44:37.229 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:44:37.229 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.229 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.229 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2024-09-16T14:44:37.229 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi026~ssd", 2024-09-16T14:44:37.230 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2024-09-16T14:44:37.230 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2024-09-16T14:44:37.230 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 17280, 2024-09-16T14:44:37.230 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2024-09-16T14:44:37.230 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2024-09-16T14:44:37.230 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2024-09-16T14:44:37.230 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.230 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2024-09-16T14:44:37.230 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T14:44:37.230 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2024-09-16T14:44:37.230 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.230 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.230 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2024-09-16T14:44:37.230 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T14:44:37.231 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2024-09-16T14:44:37.231 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.231 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.231 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2024-09-16T14:44:37.231 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T14:44:37.231 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2024-09-16T14:44:37.231 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:44:37.231 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:44:37.231 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.231 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.231 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -5, 2024-09-16T14:44:37.231 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189", 2024-09-16T14:44:37.231 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2024-09-16T14:44:37.232 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2024-09-16T14:44:37.232 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 17280, 2024-09-16T14:44:37.232 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2024-09-16T14:44:37.232 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2024-09-16T14:44:37.232 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2024-09-16T14:44:37.232 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.232 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 4, 2024-09-16T14:44:37.232 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T14:44:37.232 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2024-09-16T14:44:37.232 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.232 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.233 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 5, 2024-09-16T14:44:37.233 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T14:44:37.233 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2024-09-16T14:44:37.233 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.233 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.233 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 3, 2024-09-16T14:44:37.233 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T14:44:37.233 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2024-09-16T14:44:37.233 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:44:37.233 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:44:37.233 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.233 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.233 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -6, 2024-09-16T14:44:37.234 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189~ssd", 2024-09-16T14:44:37.234 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2024-09-16T14:44:37.234 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2024-09-16T14:44:37.234 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 17280, 2024-09-16T14:44:37.234 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2024-09-16T14:44:37.234 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2024-09-16T14:44:37.234 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2024-09-16T14:44:37.234 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.234 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 4, 2024-09-16T14:44:37.235 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T14:44:37.235 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2024-09-16T14:44:37.235 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.235 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.235 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 5, 2024-09-16T14:44:37.235 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T14:44:37.235 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2024-09-16T14:44:37.235 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.235 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.235 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 3, 2024-09-16T14:44:37.235 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T14:44:37.235 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2024-09-16T14:44:37.235 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:44:37.235 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:44:37.236 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:44:37.236 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:44:37.236 INFO:tasks.workunit.client.0.smithi189.stdout: "rules": [ 2024-09-16T14:44:37.236 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.236 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 0, 2024-09-16T14:44:37.236 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "replicated_rule", 2024-09-16T14:44:37.236 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 1, 2024-09-16T14:44:37.236 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2024-09-16T14:44:37.236 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.236 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2024-09-16T14:44:37.236 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2024-09-16T14:44:37.236 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2024-09-16T14:44:37.236 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.236 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.237 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_firstn", 2024-09-16T14:44:37.237 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2024-09-16T14:44:37.237 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2024-09-16T14:44:37.237 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.237 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.237 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2024-09-16T14:44:37.237 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:44:37.237 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:44:37.237 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.237 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.237 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 1, 2024-09-16T14:44:37.237 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "erasure-code", 2024-09-16T14:44:37.237 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 3, 2024-09-16T14:44:37.238 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2024-09-16T14:44:37.238 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.238 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "set_chooseleaf_tries", 2024-09-16T14:44:37.238 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 5 2024-09-16T14:44:37.238 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.238 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.238 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "set_choose_tries", 2024-09-16T14:44:37.238 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 100 2024-09-16T14:44:37.238 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.238 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.238 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2024-09-16T14:44:37.238 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2024-09-16T14:44:37.239 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2024-09-16T14:44:37.239 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.239 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.239 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_indep", 2024-09-16T14:44:37.239 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2024-09-16T14:44:37.239 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2024-09-16T14:44:37.239 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.239 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.239 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2024-09-16T14:44:37.239 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:44:37.239 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:44:37.239 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:44:37.239 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:44:37.239 INFO:tasks.workunit.client.0.smithi189.stdout: "tunables": { 2024-09-16T14:44:37.240 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_tries": 0, 2024-09-16T14:44:37.240 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_fallback_tries": 0, 2024-09-16T14:44:37.240 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_total_tries": 50, 2024-09-16T14:44:37.240 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_descend_once": 1, 2024-09-16T14:44:37.240 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_vary_r": 1, 2024-09-16T14:44:37.240 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_stable": 1, 2024-09-16T14:44:37.240 INFO:tasks.workunit.client.0.smithi189.stdout: "straw_calc_version": 1, 2024-09-16T14:44:37.240 INFO:tasks.workunit.client.0.smithi189.stdout: "allowed_bucket_algs": 54, 2024-09-16T14:44:37.240 INFO:tasks.workunit.client.0.smithi189.stdout: "profile": "jewel", 2024-09-16T14:44:37.240 INFO:tasks.workunit.client.0.smithi189.stdout: "optimal_tunables": 1, 2024-09-16T14:44:37.240 INFO:tasks.workunit.client.0.smithi189.stdout: "legacy_tunables": 0, 2024-09-16T14:44:37.240 INFO:tasks.workunit.client.0.smithi189.stdout: "minimum_required_version": "jewel", 2024-09-16T14:44:37.240 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables": 1, 2024-09-16T14:44:37.241 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables2": 1, 2024-09-16T14:44:37.241 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v2_rules": 1, 2024-09-16T14:44:37.241 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables3": 1, 2024-09-16T14:44:37.241 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v3_rules": 0, 2024-09-16T14:44:37.241 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v4_buckets": 1, 2024-09-16T14:44:37.241 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables5": 1, 2024-09-16T14:44:37.241 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v5_rules": 0 2024-09-16T14:44:37.241 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.241 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_args": {} 2024-09-16T14:44:37.241 INFO:tasks.workunit.client.0.smithi189.stdout:} 2024-09-16T14:44:37.241 INFO:tasks.workunit.client.0.smithi189.stdout: 2024-09-16T14:44:37.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.213+0000 7f2bc1d2e640 1 -- 172.21.15.189:0/1151691045 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2b9c0361f0 legacy=0x7f2b9c0386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:37.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.213+0000 7f2bc1d2e640 10 monclient: shutdown 2024-09-16T14:44:37.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.213+0000 7f2bc1d2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:37.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.213+0000 7f2bc1d2e640 1 -- 172.21.15.189:0/1151691045 >> v1:172.21.15.26:6790/0 conn(0x7f2bbc13b430 legacy=0x7f2bbc1d1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:37.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.213+0000 7f2bc1d2e640 1 -- 172.21.15.189:0/1151691045 shutdown_connections 2024-09-16T14:44:37.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.213+0000 7f2bc1d2e640 1 -- 172.21.15.189:0/1151691045 >> 172.21.15.189:0/1151691045 conn(0x7f2bbc0a7dd0 msgr2=0x7f2bbc06b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:37.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.213+0000 7f2bc1d2e640 1 -- 172.21.15.189:0/1151691045 shutdown_connections 2024-09-16T14:44:37.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.213+0000 7f2bc1d2e640 1 -- 172.21.15.189:0/1151691045 wait complete. 2024-09-16T14:44:37.243 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule dump 2024-09-16T14:44:37.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570c3bf640 10 monclient: get_monmap_and_config 2024-09-16T14:44:37.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570c3bf640 10 monclient: build_initial_monmap 2024-09-16T14:44:37.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570c3bf640 10 monclient: monmap: 2024-09-16T14:44:37.351 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:37.351 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:37.352 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:37.351472+0000 2024-09-16T14:44:37.352 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:37.351472+0000 2024-09-16T14:44:37.352 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:37.352 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:37.352 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:37.352 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:37.352 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:37.352 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:37.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570c3bf640 1 Processor -- start 2024-09-16T14:44:37.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570c3bf640 1 -- start start 2024-09-16T14:44:37.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570c3bf640 10 monclient: init 2024-09-16T14:44:37.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570c3bf640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:37.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570c3bf640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:37.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570c3bf640 10 monclient(hunting): picked mon.noname-a con 0x7f57040a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:37.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570c3bf640 10 monclient(hunting): picked mon.noname-b con 0x7f57040a25f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:37.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570c3bf640 10 monclient(hunting): picked mon.noname-c con 0x7f570413b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:37.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570c3bf640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57040a30b0 con 0x7f57040a1b00 2024-09-16T14:44:37.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570c3bf640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57040a2e30 con 0x7f570413b430 2024-09-16T14:44:37.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570c3bf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570409eb50 con 0x7f57040a25f0 2024-09-16T14:44:37.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570c3bf640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:37.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570c3bf640 10 monclient(hunting): authenticate will time out at 1059.467189s 2024-09-16T14:44:37.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570a95c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f570413b430 legacy=0x7f570413d820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:37.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570a15b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f57040a1b00 0x7f57040a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:52808/0 (socket says 172.21.15.189:52808) 2024-09-16T14:44:37.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.345+0000 7f570a15b640 1 -- 172.21.15.189:0/2605883367 learned_addr learned my addr 172.21.15.189:0/2605883367 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:37.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 1 -- 172.21.15.189:0/2605883367 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266302940 0 0) 0x7f570409eb50 con 0x7f57040a25f0 2024-09-16T14:44:37.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient(hunting): _negotiate 2024-09-16T14:44:37.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:37.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:37.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient(hunting): my global_id is 4838 2024-09-16T14:44:37.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 1 -- 172.21.15.189:0/2605883367 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56f0003530 con 0x7f57040a25f0 2024-09-16T14:44:37.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 1 -- 172.21.15.189:0/2605883367 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125207078 0 0) 0x7f57040a30b0 con 0x7f57040a1b00 2024-09-16T14:44:37.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient(hunting): _negotiate 2024-09-16T14:44:37.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:37.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:37.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient(hunting): my global_id is 84225 2024-09-16T14:44:37.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 1 -- 172.21.15.189:0/2605883367 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f570409eb50 con 0x7f57040a1b00 2024-09-16T14:44:37.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 1 -- 172.21.15.189:0/2605883367 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4425629 0 0) 0x7f56f0003530 con 0x7f57040a25f0 2024-09-16T14:44:37.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 1 -- 172.21.15.189:0/2605883367 >> v1:172.21.15.189:6789/0 conn(0x7f570413b430 legacy=0x7f570413d820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:37.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 1 -- 172.21.15.189:0/2605883367 >> v1:172.21.15.26:6789/0 conn(0x7f57040a1b00 legacy=0x7f57040a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:37.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient: _finish_hunting 0 2024-09-16T14:44:37.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 1 monclient: found mon.noname-b 2024-09-16T14:44:37.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:37.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:37.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 1 -- 172.21.15.189:0/2605883367 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570409ee30 con 0x7f57040a25f0 2024-09-16T14:44:37.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient: _finish_auth 0 2024-09-16T14:44:37.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient: _check_auth_tickets 2024-09-16T14:44:37.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:37.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 1 -- 172.21.15.189:0/2605883367 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f56f4001620 con 0x7f57040a25f0 2024-09-16T14:44:37.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:37.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:37.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient: dump: 2024-09-16T14:44:37.363 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:37.363 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:37.363 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.363 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.363 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:37.363 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:37.363 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:37.364 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:37.364 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:37.364 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:37.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 5 monclient: authenticate success, global_id 4838 2024-09-16T14:44:37.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:37.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 1 -- 172.21.15.189:0/2605883367 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56f4001880 con 0x7f57040a25f0 2024-09-16T14:44:37.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:37.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:37.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 4 set_mon_vals no callback set 2024-09-16T14:44:37.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 1 -- 172.21.15.189:0/2605883367 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f56f4001c50 con 0x7f57040a25f0 2024-09-16T14:44:37.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:37.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:37.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f5709159640 10 monclient: dump: 2024-09-16T14:44:37.365 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:37.365 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:37.365 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.365 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.365 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:37.365 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:37.365 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:37.365 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:37.365 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:37.366 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:37.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 10 monclient: shutdown 2024-09-16T14:44:37.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:37.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 1 -- 172.21.15.189:0/2605883367 >> v1:172.21.15.26:6790/0 conn(0x7f57040a25f0 legacy=0x7f5704139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:37.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 1 -- 172.21.15.189:0/2605883367 shutdown_connections 2024-09-16T14:44:37.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 1 -- 172.21.15.189:0/2605883367 >> 172.21.15.189:0/2605883367 conn(0x7f57040a7dd0 msgr2=0x7f57040a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:37.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 1 -- 172.21.15.189:0/2605883367 shutdown_connections 2024-09-16T14:44:37.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 1 -- 172.21.15.189:0/2605883367 wait complete. 2024-09-16T14:44:37.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 10 monclient: build_initial_monmap 2024-09-16T14:44:37.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 10 monclient: monmap: 2024-09-16T14:44:37.366 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:37.366 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:37.366 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:37.367 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 1 Processor -- start 2024-09-16T14:44:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 1 -- start start 2024-09-16T14:44:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 10 monclient: init 2024-09-16T14:44:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:37.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 10 monclient(hunting): picked mon.noname-c con 0x7f57040a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:37.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 10 monclient(hunting): picked mon.noname-a con 0x7f57040a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:37.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 10 monclient(hunting): picked mon.noname-b con 0x7f570413b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:37.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57040a5a30 con 0x7f57040a25f0 2024-09-16T14:44:37.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5704137780 con 0x7f570413b430 2024-09-16T14:44:37.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570409e2f0 con 0x7f57040a1b00 2024-09-16T14:44:37.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:37.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570c3bf640 10 monclient(hunting): authenticate will time out at 1059.471492s 2024-09-16T14:44:37.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570a95c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f570413b430 legacy=0x7f570409ba40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:37.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570a15b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f57040a1b00 0x7f570409ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:36090/0 (socket says 172.21.15.189:36090) 2024-09-16T14:44:37.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.349+0000 7f570a15b640 1 -- 172.21.15.189:0/1221187619 learned_addr learned my addr 172.21.15.189:0/1221187619 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:37.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308854754 0 0) 0x7f570409e2f0 con 0x7f57040a1b00 2024-09-16T14:44:37.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:37.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:37.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:37.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient(hunting): my global_id is 4841 2024-09-16T14:44:37.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56e8003530 con 0x7f57040a1b00 2024-09-16T14:44:37.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431415673 0 0) 0x7f570409ee30 con 0x7f57040a25f0 2024-09-16T14:44:37.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:37.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:37.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:37.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient(hunting): my global_id is 84228 2024-09-16T14:44:37.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f570409e2f0 con 0x7f57040a25f0 2024-09-16T14:44:37.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2024191747 0 0) 0x7f56e8003530 con 0x7f57040a1b00 2024-09-16T14:44:37.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f570409ee30 con 0x7f57040a1b00 2024-09-16T14:44:37.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5700001380 con 0x7f57040a1b00 2024-09-16T14:44:37.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:37.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:37.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient(hunting): dump: 2024-09-16T14:44:37.370 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:37.370 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:37.370 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.370 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.370 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:37.370 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:37.370 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:37.370 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:37.370 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:37.371 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:37.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:37.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1637584701 0 0) 0x7f570409e2f0 con 0x7f57040a25f0 2024-09-16T14:44:37.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56e8003530 con 0x7f57040a25f0 2024-09-16T14:44:37.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f56f40019f0 con 0x7f57040a25f0 2024-09-16T14:44:37.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:37.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:37.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient(hunting): dump: 2024-09-16T14:44:37.371 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:37.371 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:37.371 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.371 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.371 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:37.372 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:37.372 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:37.372 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:37.372 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:37.372 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:37.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2754625668 0 0) 0x7f570409ee30 con 0x7f57040a1b00 2024-09-16T14:44:37.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 >> v1:172.21.15.189:6789/0 conn(0x7f570413b430 legacy=0x7f570409ba40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:37.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 >> v1:172.21.15.26:6789/0 conn(0x7f57040a25f0 legacy=0x7f570409b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:37.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:37.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 monclient: found mon.c 2024-09-16T14:44:37.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:37.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:37.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57041362f0 con 0x7f57040a1b00 2024-09-16T14:44:37.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient: _finish_auth 0 2024-09-16T14:44:37.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:37.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:37.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f570c3bf640 5 monclient: authenticate success, global_id 4841 2024-09-16T14:44:37.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f570c3bf640 10 monclient: _renew_subs 2024-09-16T14:44:37.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f570c3bf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:37.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f570c3bf640 1 -- 172.21.15.189:0/1221187619 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5704137360 con 0x7f57040a1b00 2024-09-16T14:44:37.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f570c3bf640 10 monclient: _renew_subs 2024-09-16T14:44:37.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f570c3bf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:37.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f570c3bf640 1 -- 172.21.15.189:0/1221187619 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f570409c780 con 0x7f57040a1b00 2024-09-16T14:44:37.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57000015e0 con 0x7f57040a1b00 2024-09-16T14:44:37.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:37.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5700004c40 con 0x7f57040a1b00 2024-09-16T14:44:37.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:37.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f5708958640 4 set_mon_vals no callback set 2024-09-16T14:44:37.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:37.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 10 monclient: dump: 2024-09-16T14:44:37.374 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:37.374 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:37.375 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.375 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.375 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:37.375 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:37.375 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:37.375 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:37.375 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:37.375 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:37.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f570c3bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:37.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f570c3bf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:37.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f570c3bf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:37.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f570c3bf640 1 -- 172.21.15.189:0/1221187619 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56ec005050 con 0x7f57040a1b00 2024-09-16T14:44:37.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f570000f6b0 con 0x7f57040a1b00 2024-09-16T14:44:37.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(40..40 src has 1..40) v4 ==== 3598+0+0 (unknown 1780700966 0 0) 0x7f5700043240 con 0x7f57040a1b00 2024-09-16T14:44:37.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.353+0000 7f570995a640 10 monclient: get_auth_request con 0x7f56e8036220 auth_method 0 2024-09-16T14:44:37.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.369+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f57000145a0 con 0x7f57040a1b00 2024-09-16T14:44:37.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.369+0000 7f56faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:37.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.369+0000 7f56faffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:37.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.517+0000 7f570c3bf640 10 start_mon_command cmd=[{"prefix": "osd crush rule dump"}] 2024-09-16T14:44:37.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.517+0000 7f570c3bf640 10 monclient: _send_command 2 [{"prefix": "osd crush rule dump"}] 2024-09-16T14:44:37.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.517+0000 7f570c3bf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:37.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.517+0000 7f570c3bf640 1 -- 172.21.15.189:0/1221187619 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rule dump"} v 0) v1 -- 0x7f56ec005300 con 0x7f57040a1b00 2024-09-16T14:44:37.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.517+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule dump"}]=0 v40) v1 ==== 67+0+1058 (unknown 1856856027 0 3068570633) 0x7f57000143c0 con 0x7f57040a1b00 2024-09-16T14:44:37.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.517+0000 7f56faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule dump"}] 2024-09-16T14:44:37.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.517+0000 7f56faffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:37.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.605+0000 7f570c3bf640 10 start_mon_command cmd=[{"prefix": "osd crush rule dump"}] 2024-09-16T14:44:37.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.605+0000 7f570c3bf640 10 monclient: _send_command 3 [{"prefix": "osd crush rule dump"}] 2024-09-16T14:44:37.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.605+0000 7f570c3bf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:37.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.605+0000 7f570c3bf640 1 -- 172.21.15.189:0/1221187619 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rule dump"} v 0) v1 -- 0x7f56ec005540 con 0x7f57040a1b00 2024-09-16T14:44:37.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.605+0000 7f56faffd640 1 -- 172.21.15.189:0/1221187619 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule dump"}]=0 v40) v1 ==== 67+0+1058 (unknown 1856856027 0 3068570633) 0x7f57000143c0 con 0x7f57040a1b00 2024-09-16T14:44:37.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.605+0000 7f56faffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule dump"}] 2024-09-16T14:44:37.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.605+0000 7f56faffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:44:37.611 INFO:tasks.workunit.client.0.smithi189.stdout:[ 2024-09-16T14:44:37.611 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.612 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 0, 2024-09-16T14:44:37.612 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "replicated_rule", 2024-09-16T14:44:37.612 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 1, 2024-09-16T14:44:37.612 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2024-09-16T14:44:37.612 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.612 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2024-09-16T14:44:37.612 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2024-09-16T14:44:37.612 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2024-09-16T14:44:37.612 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.612 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.612 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_firstn", 2024-09-16T14:44:37.612 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2024-09-16T14:44:37.612 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2024-09-16T14:44:37.613 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.613 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.613 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2024-09-16T14:44:37.613 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:44:37.613 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:44:37.613 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.613 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.613 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 1, 2024-09-16T14:44:37.613 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "erasure-code", 2024-09-16T14:44:37.613 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 3, 2024-09-16T14:44:37.613 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2024-09-16T14:44:37.613 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.613 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "set_chooseleaf_tries", 2024-09-16T14:44:37.614 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 5 2024-09-16T14:44:37.614 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.614 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.614 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "set_choose_tries", 2024-09-16T14:44:37.614 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 100 2024-09-16T14:44:37.614 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.614 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.614 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2024-09-16T14:44:37.614 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2024-09-16T14:44:37.614 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2024-09-16T14:44:37.614 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.614 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.614 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_indep", 2024-09-16T14:44:37.614 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2024-09-16T14:44:37.615 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2024-09-16T14:44:37.615 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:44:37.615 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:44:37.615 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2024-09-16T14:44:37.615 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:44:37.615 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:44:37.615 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:44:37.615 INFO:tasks.workunit.client.0.smithi189.stdout:] 2024-09-16T14:44:37.615 INFO:tasks.workunit.client.0.smithi189.stdout: 2024-09-16T14:44:37.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.605+0000 7f570c3bf640 1 -- 172.21.15.189:0/1221187619 >> v1:172.21.15.189:6812/3196161582 conn(0x7f56e8036220 legacy=0x7f56e80386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:37.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.605+0000 7f570c3bf640 10 monclient: shutdown 2024-09-16T14:44:37.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.605+0000 7f570c3bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:37.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.605+0000 7f570c3bf640 1 -- 172.21.15.189:0/1221187619 >> v1:172.21.15.26:6790/0 conn(0x7f57040a1b00 legacy=0x7f570409ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:37.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.605+0000 7f570c3bf640 1 -- 172.21.15.189:0/1221187619 shutdown_connections 2024-09-16T14:44:37.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.605+0000 7f570c3bf640 1 -- 172.21.15.189:0/1221187619 >> 172.21.15.189:0/1221187619 conn(0x7f57040a7dd0 msgr2=0x7f570406b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:37.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.609+0000 7f570c3bf640 1 -- 172.21.15.189:0/1221187619 shutdown_connections 2024-09-16T14:44:37.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.609+0000 7f570c3bf640 1 -- 172.21.15.189:0/1221187619 wait complete. 2024-09-16T14:44:37.627 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls 2024-09-16T14:44:37.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient: get_monmap_and_config 2024-09-16T14:44:37.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient: build_initial_monmap 2024-09-16T14:44:37.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient: monmap: 2024-09-16T14:44:37.740 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:37.740 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:37.740 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:37.740073+0000 2024-09-16T14:44:37.740 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:37.740073+0000 2024-09-16T14:44:37.741 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:37.741 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:37.741 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:37.741 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:37.741 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:37.741 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:37.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 1 Processor -- start 2024-09-16T14:44:37.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 1 -- start start 2024-09-16T14:44:37.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient: init 2024-09-16T14:44:37.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:37.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:37.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient(hunting): picked mon.noname-c con 0x7faa0c09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:37.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient(hunting): picked mon.noname-a con 0x7faa0c09e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:37.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient(hunting): picked mon.noname-b con 0x7faa0c09e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:37.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0c09ee20 con 0x7faa0c09e2f0 2024-09-16T14:44:37.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0c09f100 con 0x7faa0c09df10 2024-09-16T14:44:37.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0c09f3e0 con 0x7faa0c09e740 2024-09-16T14:44:37.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:37.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient(hunting): authenticate will time out at 1059.855671s 2024-09-16T14:44:37.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa11974640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7faa0c09df10 legacy=0x7faa0c09d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:37.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa12175640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7faa0c09e740 0x7faa0c13b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:36096/0 (socket says 172.21.15.189:36096) 2024-09-16T14:44:37.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa12175640 1 -- 172.21.15.189:0/1457326912 learned_addr learned my addr 172.21.15.189:0/1457326912 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:37.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa11173640 1 --1- 172.21.15.189:0/1457326912 >> v1:172.21.15.26:6789/0 conn(0x7faa0c09e2f0 0x7faa0c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:52814/0 (socket says 172.21.15.189:52814) 2024-09-16T14:44:37.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 1 -- 172.21.15.189:0/1457326912 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538032278 0 0) 0x7faa0c09ee20 con 0x7faa0c09e2f0 2024-09-16T14:44:37.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient(hunting): _negotiate 2024-09-16T14:44:37.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:37.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:37.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient(hunting): my global_id is 84231 2024-09-16T14:44:37.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 1 -- 172.21.15.189:0/1457326912 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9f8003530 con 0x7faa0c09e2f0 2024-09-16T14:44:37.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 1 -- 172.21.15.189:0/1457326912 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738596265 0 0) 0x7faa0c09f3e0 con 0x7faa0c09e740 2024-09-16T14:44:37.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient(hunting): _negotiate 2024-09-16T14:44:37.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:37.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:37.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient(hunting): my global_id is 4844 2024-09-16T14:44:37.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 1 -- 172.21.15.189:0/1457326912 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa0c09ee20 con 0x7faa0c09e740 2024-09-16T14:44:37.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 1 -- 172.21.15.189:0/1457326912 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2315217982 0 0) 0x7fa9f8003530 con 0x7faa0c09e2f0 2024-09-16T14:44:37.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 1 -- 172.21.15.189:0/1457326912 >> v1:172.21.15.26:6790/0 conn(0x7faa0c09e740 legacy=0x7faa0c13b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:37.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 1 -- 172.21.15.189:0/1457326912 >> v1:172.21.15.189:6789/0 conn(0x7faa0c09df10 legacy=0x7faa0c09d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:37.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient: _finish_hunting 0 2024-09-16T14:44:37.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 1 monclient: found mon.noname-a 2024-09-16T14:44:37.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:37.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:37.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 1 -- 172.21.15.189:0/1457326912 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0c09c690 con 0x7faa0c09e2f0 2024-09-16T14:44:37.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient: _finish_auth 0 2024-09-16T14:44:37.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient: _check_auth_tickets 2024-09-16T14:44:37.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:37.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 1 -- 172.21.15.189:0/1457326912 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faa08001620 con 0x7faa0c09e2f0 2024-09-16T14:44:37.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:37.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:37.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient: dump: 2024-09-16T14:44:37.748 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:37.748 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:37.748 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.748 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.748 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:37.748 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:37.748 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:37.748 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:37.748 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:37.748 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:37.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 5 monclient: authenticate success, global_id 84231 2024-09-16T14:44:37.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:37.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 1 -- 172.21.15.189:0/1457326912 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa08001900 con 0x7faa0c09e2f0 2024-09-16T14:44:37.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:37.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 1 -- 172.21.15.189:0/1457326912 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faa08001c90 con 0x7faa0c09e2f0 2024-09-16T14:44:37.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:37.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:37.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa10972640 10 monclient: dump: 2024-09-16T14:44:37.749 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:37.749 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:37.749 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.749 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.750 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:37.750 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:37.750 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:37.750 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:37.750 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:37.750 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:37.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:37.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 4 set_mon_vals no callback set 2024-09-16T14:44:37.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient: shutdown 2024-09-16T14:44:37.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:37.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 1 -- 172.21.15.189:0/1457326912 >> v1:172.21.15.26:6789/0 conn(0x7faa0c09e2f0 legacy=0x7faa0c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:37.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 1 -- 172.21.15.189:0/1457326912 shutdown_connections 2024-09-16T14:44:37.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 1 -- 172.21.15.189:0/1457326912 >> 172.21.15.189:0/1457326912 conn(0x7faa0c0a3320 msgr2=0x7faa0c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:37.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 1 -- 172.21.15.189:0/1457326912 shutdown_connections 2024-09-16T14:44:37.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 1 -- 172.21.15.189:0/1457326912 wait complete. 2024-09-16T14:44:37.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient: build_initial_monmap 2024-09-16T14:44:37.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient: monmap: 2024-09-16T14:44:37.751 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:37.751 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:37.751 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:37.751 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:37.751 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:37.751 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:37.751 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:37.751 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:37.751 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:37.752 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:37.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 1 Processor -- start 2024-09-16T14:44:37.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 1 -- start start 2024-09-16T14:44:37.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient: init 2024-09-16T14:44:37.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:44:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient(hunting): picked mon.noname-b con 0x7faa0c09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient(hunting): picked mon.noname-c con 0x7faa0c09e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient(hunting): picked mon.noname-a con 0x7faa0c09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0c06f9d0 con 0x7faa0c09e740 2024-09-16T14:44:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0c11a2d0 con 0x7faa0c09df10 2024-09-16T14:44:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa11974640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7faa0c09df10 legacy=0x7faa0c1399f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0c13a810 con 0x7faa0c09e2f0 2024-09-16T14:44:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa13bd8640 10 monclient(hunting): authenticate will time out at 1059.859341s 2024-09-16T14:44:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa12175640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7faa0c09e740 0x7faa0c1cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:52822/0 (socket says 172.21.15.189:52822) 2024-09-16T14:44:37.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.737+0000 7faa12175640 1 -- 172.21.15.189:0/1183635021 learned_addr learned my addr 172.21.15.189:0/1183635021 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:37.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094311082 0 0) 0x7faa0c06f9d0 con 0x7faa0c09e740 2024-09-16T14:44:37.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:37.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:37.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:37.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient(hunting): my global_id is 84234 2024-09-16T14:44:37.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9f0003560 con 0x7faa0c09e740 2024-09-16T14:44:37.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405621152 0 0) 0x7faa0c13a810 con 0x7faa0c09e2f0 2024-09-16T14:44:37.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:37.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:37.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:37.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient(hunting): my global_id is 4847 2024-09-16T14:44:37.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa0c06f9d0 con 0x7faa0c09e2f0 2024-09-16T14:44:37.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3194410339 0 0) 0x7fa9f0003560 con 0x7faa0c09e740 2024-09-16T14:44:37.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa0c13a810 con 0x7faa0c09e740 2024-09-16T14:44:37.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faa04001c00 con 0x7faa0c09e740 2024-09-16T14:44:37.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:37.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:37.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient(hunting): dump: 2024-09-16T14:44:37.757 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:37.757 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:37.757 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.757 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.757 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:37.757 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:37.757 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:37.758 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:37.758 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:37.758 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:37.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:37.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3409062190 0 0) 0x7faa0c06f9d0 con 0x7faa0c09e2f0 2024-09-16T14:44:37.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9f0003560 con 0x7faa0c09e2f0 2024-09-16T14:44:37.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faa08001c10 con 0x7faa0c09e2f0 2024-09-16T14:44:37.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:37.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:37.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient(hunting): dump: 2024-09-16T14:44:37.758 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:37.758 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:37.758 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:37.759 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 352270643 0 0) 0x7faa0c13a810 con 0x7faa0c09e740 2024-09-16T14:44:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 >> v1:172.21.15.26:6790/0 conn(0x7faa0c09e2f0 legacy=0x7faa0c13a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 >> v1:172.21.15.189:6789/0 conn(0x7faa0c09df10 legacy=0x7faa0c1399f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient: _finish_hunting 0 2024-09-16T14:44:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 monclient: found mon.a 2024-09-16T14:44:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:37.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0c06cec0 con 0x7faa0c09e740 2024-09-16T14:44:37.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient: _finish_auth 0 2024-09-16T14:44:37.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient: _check_auth_tickets 2024-09-16T14:44:37.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:37.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa13bd8640 5 monclient: authenticate success, global_id 84234 2024-09-16T14:44:37.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa13bd8640 10 monclient: _renew_subs 2024-09-16T14:44:37.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa13bd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:37.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa13bd8640 1 -- 172.21.15.189:0/1183635021 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa0c06b380 con 0x7faa0c09e740 2024-09-16T14:44:37.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa13bd8640 10 monclient: _renew_subs 2024-09-16T14:44:37.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa13bd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:37.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa13bd8640 1 -- 172.21.15.189:0/1183635021 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa0c13bf40 con 0x7faa0c09e740 2024-09-16T14:44:37.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa04004b70 con 0x7faa0c09e740 2024-09-16T14:44:37.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:37.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faa04004e80 con 0x7faa0c09e740 2024-09-16T14:44:37.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:37.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa03fff640 4 set_mon_vals no callback set 2024-09-16T14:44:37.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:37.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 10 monclient: dump: 2024-09-16T14:44:37.761 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:37.761 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:37.761 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.761 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:37.761 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:37.761 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:37.761 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:37.762 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:37.762 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:37.762 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:37.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa13bd8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:37.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa13bd8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:37.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa13bd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:37.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa13bd8640 1 -- 172.21.15.189:0/1183635021 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f4005050 con 0x7faa0c09e740 2024-09-16T14:44:37.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7faa0400f8f0 con 0x7faa0c09e740 2024-09-16T14:44:37.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(40..40 src has 1..40) v4 ==== 3598+0+0 (unknown 1780700966 0 0) 0x7faa040433e0 con 0x7faa0c09e740 2024-09-16T14:44:37.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.741+0000 7faa11974640 10 monclient: get_auth_request con 0x7fa9f00362a0 auth_method 0 2024-09-16T14:44:37.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.757+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7faa040146f0 con 0x7faa0c09e740 2024-09-16T14:44:37.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.757+0000 7faa027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:37.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.757+0000 7faa027fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:37.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.917+0000 7faa13bd8640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T14:44:37.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.917+0000 7faa13bd8640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:44:37.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.917+0000 7faa13bd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:37.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.917+0000 7faa13bd8640 1 -- 172.21.15.189:0/1183635021 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fa9f4005300 con 0x7faa0c09e740 2024-09-16T14:44:37.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.929+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v40) v1 ==== 65+0+29 (unknown 1080609712 0 852635902) 0x7faa04014510 con 0x7faa0c09e740 2024-09-16T14:44:37.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.929+0000 7faa027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:44:37.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:37.929+0000 7faa027fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:38.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.017+0000 7faa13bd8640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T14:44:38.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.017+0000 7faa13bd8640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:44:38.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.017+0000 7faa13bd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:38.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.017+0000 7faa13bd8640 1 -- 172.21.15.189:0/1183635021 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fa9f4005540 con 0x7faa0c09e740 2024-09-16T14:44:38.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.021+0000 7faa027fc640 1 -- 172.21.15.189:0/1183635021 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v40) v1 ==== 65+0+29 (unknown 1080609712 0 852635902) 0x7faa04014510 con 0x7faa0c09e740 2024-09-16T14:44:38.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.021+0000 7faa027fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:44:38.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.021+0000 7faa027fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:44:38.026 INFO:tasks.workunit.client.0.smithi189.stdout:replicated_rule 2024-09-16T14:44:38.026 INFO:tasks.workunit.client.0.smithi189.stdout:erasure-code 2024-09-16T14:44:38.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.021+0000 7faa13bd8640 1 -- 172.21.15.189:0/1183635021 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa9f00362a0 legacy=0x7fa9f0038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:38.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.021+0000 7faa13bd8640 10 monclient: shutdown 2024-09-16T14:44:38.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.021+0000 7faa13bd8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:38.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.021+0000 7faa13bd8640 1 -- 172.21.15.189:0/1183635021 >> v1:172.21.15.26:6789/0 conn(0x7faa0c09e740 legacy=0x7faa0c1cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:38.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.021+0000 7faa13bd8640 1 -- 172.21.15.189:0/1183635021 shutdown_connections 2024-09-16T14:44:38.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.021+0000 7faa13bd8640 1 -- 172.21.15.189:0/1183635021 >> 172.21.15.189:0/1183635021 conn(0x7faa0c0a3320 msgr2=0x7faa0c0a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:38.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.021+0000 7faa13bd8640 1 -- 172.21.15.189:0/1183635021 shutdown_connections 2024-09-16T14:44:38.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.021+0000 7faa13bd8640 1 -- 172.21.15.189:0/1183635021 wait complete. 2024-09-16T14:44:38.041 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule list 2024-09-16T14:44:38.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient: get_monmap_and_config 2024-09-16T14:44:38.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient: build_initial_monmap 2024-09-16T14:44:38.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient: monmap: 2024-09-16T14:44:38.160 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:38.160 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:38.160335+0000 2024-09-16T14:44:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:38.160335+0000 2024-09-16T14:44:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:38.161 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:38.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 1 Processor -- start 2024-09-16T14:44:38.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 1 -- start start 2024-09-16T14:44:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient: init 2024-09-16T14:44:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient(hunting): picked mon.noname-c con 0x7f775c099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient(hunting): picked mon.noname-a con 0x7f775c1334d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient(hunting): picked mon.noname-b con 0x7f775c136fe0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775c09b350 con 0x7f775c1334d0 2024-09-16T14:44:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775c09f2e0 con 0x7f775c099a70 2024-09-16T14:44:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7761602640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f775c099a70 legacy=0x7f775c09bd30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775c09b070 con 0x7f775c136fe0 2024-09-16T14:44:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient(hunting): authenticate will time out at 1060.276134s 2024-09-16T14:44:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7761e03640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f775c136fe0 0x7f775c1393d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:36112/0 (socket says 172.21.15.189:36112) 2024-09-16T14:44:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7760e01640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f775c1334d0 0x7f775c1358c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:52830/0 (socket says 172.21.15.189:52830) 2024-09-16T14:44:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7761e03640 1 -- 172.21.15.189:0/3933586801 learned_addr learned my addr 172.21.15.189:0/3933586801 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 1 -- 172.21.15.189:0/3933586801 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703061753 0 0) 0x7f775c09b350 con 0x7f775c1334d0 2024-09-16T14:44:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient(hunting): my global_id is 84237 2024-09-16T14:44:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 1 -- 172.21.15.189:0/3933586801 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7748003530 con 0x7f775c1334d0 2024-09-16T14:44:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 1 -- 172.21.15.189:0/3933586801 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111021552 0 0) 0x7f775c09b070 con 0x7f775c136fe0 2024-09-16T14:44:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient(hunting): my global_id is 4850 2024-09-16T14:44:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 1 -- 172.21.15.189:0/3933586801 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f775c09b350 con 0x7f775c136fe0 2024-09-16T14:44:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 1 -- 172.21.15.189:0/3933586801 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2789566260 0 0) 0x7f7748003530 con 0x7f775c1334d0 2024-09-16T14:44:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 1 -- 172.21.15.189:0/3933586801 >> v1:172.21.15.26:6790/0 conn(0x7f775c136fe0 legacy=0x7f775c1393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 1 -- 172.21.15.189:0/3933586801 >> v1:172.21.15.189:6789/0 conn(0x7f775c099a70 legacy=0x7f775c09bd30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient: _finish_hunting 0 2024-09-16T14:44:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 1 monclient: found mon.noname-a 2024-09-16T14:44:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 1 -- 172.21.15.189:0/3933586801 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c09b630 con 0x7f775c1334d0 2024-09-16T14:44:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient: _finish_auth 0 2024-09-16T14:44:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient: _check_auth_tickets 2024-09-16T14:44:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 1 -- 172.21.15.189:0/3933586801 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f774c001550 con 0x7f775c1334d0 2024-09-16T14:44:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 5 monclient: authenticate success, global_id 84237 2024-09-16T14:44:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient: dump: 2024-09-16T14:44:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:38.169 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 1 -- 172.21.15.189:0/3933586801 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f774c001770 con 0x7f775c1334d0 2024-09-16T14:44:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 4 set_mon_vals no callback set 2024-09-16T14:44:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 1 -- 172.21.15.189:0/3933586801 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f774c001a80 con 0x7f775c1334d0 2024-09-16T14:44:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7753fff640 10 monclient: dump: 2024-09-16T14:44:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:38.170 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient: shutdown 2024-09-16T14:44:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 1 -- 172.21.15.189:0/3933586801 >> v1:172.21.15.26:6789/0 conn(0x7f775c1334d0 legacy=0x7f775c1358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 1 -- 172.21.15.189:0/3933586801 shutdown_connections 2024-09-16T14:44:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 1 -- 172.21.15.189:0/3933586801 >> 172.21.15.189:0/3933586801 conn(0x7f775c0a3320 msgr2=0x7f775c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 1 -- 172.21.15.189:0/3933586801 shutdown_connections 2024-09-16T14:44:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 1 -- 172.21.15.189:0/3933586801 wait complete. 2024-09-16T14:44:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient: build_initial_monmap 2024-09-16T14:44:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient: monmap: 2024-09-16T14:44:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:38.172 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:38.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 1 Processor -- start 2024-09-16T14:44:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 1 -- start start 2024-09-16T14:44:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient: init 2024-09-16T14:44:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient(hunting): picked mon.noname-b con 0x7f775c099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient(hunting): picked mon.noname-a con 0x7f775c1334d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient(hunting): picked mon.noname-c con 0x7f775c136fe0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775c11fb70 con 0x7f775c1334d0 2024-09-16T14:44:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775c11e1a0 con 0x7f775c099a70 2024-09-16T14:44:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775c06b380 con 0x7f775c136fe0 2024-09-16T14:44:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7763866640 10 monclient(hunting): authenticate will time out at 1060.279895s 2024-09-16T14:44:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.157+0000 7f7761602640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f775c099a70 legacy=0x7f775c1c73e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7761e03640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f775c136fe0 0x7f775c1cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:36126/0 (socket says 172.21.15.189:36126) 2024-09-16T14:44:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7761e03640 1 -- 172.21.15.189:0/2639907300 learned_addr learned my addr 172.21.15.189:0/2639907300 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054337425 0 0) 0x7f775c11fb70 con 0x7f775c1334d0 2024-09-16T14:44:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient(hunting): my global_id is 84240 2024-09-16T14:44:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7740003530 con 0x7f775c1334d0 2024-09-16T14:44:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794380741 0 0) 0x7f775c06b380 con 0x7f775c136fe0 2024-09-16T14:44:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient(hunting): my global_id is 4853 2024-09-16T14:44:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f775c11fb70 con 0x7f775c136fe0 2024-09-16T14:44:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1948421032 0 0) 0x7f7740003530 con 0x7f775c1334d0 2024-09-16T14:44:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f775c06b380 con 0x7f775c1334d0 2024-09-16T14:44:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f774c001450 con 0x7f775c1334d0 2024-09-16T14:44:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient(hunting): dump: 2024-09-16T14:44:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.178 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:38.178 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:38.178 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:38.178 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:38.178 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:38.178 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:38.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:38.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4116774083 0 0) 0x7f775c11fb70 con 0x7f775c136fe0 2024-09-16T14:44:38.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7740003530 con 0x7f775c136fe0 2024-09-16T14:44:38.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7754001c00 con 0x7f775c136fe0 2024-09-16T14:44:38.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:38.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:38.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient(hunting): dump: 2024-09-16T14:44:38.178 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:38.179 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:38.179 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.179 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.179 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:38.179 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:38.179 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:38.179 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:38.179 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:38.179 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:38.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3872289967 0 0) 0x7f775c06b380 con 0x7f775c1334d0 2024-09-16T14:44:38.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 >> v1:172.21.15.26:6790/0 conn(0x7f775c136fe0 legacy=0x7f775c1cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:38.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 >> v1:172.21.15.189:6789/0 conn(0x7f775c099a70 legacy=0x7f775c1c73e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:38.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:44:38.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 monclient: found mon.a 2024-09-16T14:44:38.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:38.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:38.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c11f5b0 con 0x7f775c1334d0 2024-09-16T14:44:38.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient: _finish_auth 0 2024-09-16T14:44:38.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:44:38.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:38.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7763866640 5 monclient: authenticate success, global_id 84240 2024-09-16T14:44:38.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7763866640 10 monclient: _renew_subs 2024-09-16T14:44:38.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7763866640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:38.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7763866640 1 -- 172.21.15.189:0/2639907300 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f775c11f890 con 0x7f775c1334d0 2024-09-16T14:44:38.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7763866640 10 monclient: _renew_subs 2024-09-16T14:44:38.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7763866640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:38.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7763866640 1 -- 172.21.15.189:0/2639907300 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f775c0a2230 con 0x7f775c1334d0 2024-09-16T14:44:38.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f774c004aa0 con 0x7f775c1334d0 2024-09-16T14:44:38.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:38.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f774c004db0 con 0x7f775c1334d0 2024-09-16T14:44:38.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:38.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:38.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 10 monclient: dump: 2024-09-16T14:44:38.181 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:38.181 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:38.181 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.181 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.181 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:38.181 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:38.182 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:38.182 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:38.182 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:38.182 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:38.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7752ffd640 4 set_mon_vals no callback set 2024-09-16T14:44:38.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7763866640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:38.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7763866640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:38.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7763866640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:38.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7763866640 1 -- 172.21.15.189:0/2639907300 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773c005050 con 0x7f775c1334d0 2024-09-16T14:44:38.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f774c00f820 con 0x7f775c1334d0 2024-09-16T14:44:38.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(40..40 src has 1..40) v4 ==== 3598+0+0 (unknown 1780700966 0 0) 0x7f774c043250 con 0x7f775c1334d0 2024-09-16T14:44:38.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.161+0000 7f7761602640 10 monclient: get_auth_request con 0x7f7740036220 auth_method 0 2024-09-16T14:44:38.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.177+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f774c0145b0 con 0x7f775c1334d0 2024-09-16T14:44:38.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.177+0000 7f7751ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:38.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.177+0000 7f7751ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:38.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.333+0000 7f7763866640 10 start_mon_command cmd=[{"prefix": "osd crush rule list"}] 2024-09-16T14:44:38.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.333+0000 7f7763866640 10 monclient: _send_command 2 [{"prefix": "osd crush rule list"}] 2024-09-16T14:44:38.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.333+0000 7f7763866640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:38.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.333+0000 7f7763866640 1 -- 172.21.15.189:0/2639907300 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule list"} v 0) v1 -- 0x7f773c005300 con 0x7f775c1334d0 2024-09-16T14:44:38.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.333+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule list"}]=0 v40) v1 ==== 67+0+29 (unknown 3775985541 0 852635902) 0x7f774c0143d0 con 0x7f775c1334d0 2024-09-16T14:44:38.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.333+0000 7f7751ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule list"}] 2024-09-16T14:44:38.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.333+0000 7f7751ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.429+0000 7f7763866640 10 start_mon_command cmd=[{"prefix": "osd crush rule list"}] 2024-09-16T14:44:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.429+0000 7f7763866640 10 monclient: _send_command 3 [{"prefix": "osd crush rule list"}] 2024-09-16T14:44:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.429+0000 7f7763866640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.429+0000 7f7763866640 1 -- 172.21.15.189:0/2639907300 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule list"} v 0) v1 -- 0x7f773c005540 con 0x7f775c1334d0 2024-09-16T14:44:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.429+0000 7f7751ffb640 1 -- 172.21.15.189:0/2639907300 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule list"}]=0 v40) v1 ==== 67+0+29 (unknown 3775985541 0 852635902) 0x7f774c0143d0 con 0x7f775c1334d0 2024-09-16T14:44:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.429+0000 7f7751ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule list"}] 2024-09-16T14:44:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.429+0000 7f7751ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:44:38.435 INFO:tasks.workunit.client.0.smithi189.stdout:replicated_rule 2024-09-16T14:44:38.435 INFO:tasks.workunit.client.0.smithi189.stdout:erasure-code 2024-09-16T14:44:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.429+0000 7f7763866640 1 -- 172.21.15.189:0/2639907300 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7740036220 legacy=0x7f77400386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.429+0000 7f7763866640 10 monclient: shutdown 2024-09-16T14:44:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.429+0000 7f7763866640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.429+0000 7f7763866640 1 -- 172.21.15.189:0/2639907300 >> v1:172.21.15.26:6789/0 conn(0x7f775c1334d0 legacy=0x7f775c1cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.429+0000 7f7763866640 1 -- 172.21.15.189:0/2639907300 shutdown_connections 2024-09-16T14:44:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.429+0000 7f7763866640 1 -- 172.21.15.189:0/2639907300 >> 172.21.15.189:0/2639907300 conn(0x7f775c0a3320 msgr2=0x7f775c070cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.429+0000 7f7763866640 1 -- 172.21.15.189:0/2639907300 shutdown_connections 2024-09-16T14:44:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.429+0000 7f7763866640 1 -- 172.21.15.189:0/2639907300 wait complete. 2024-09-16T14:44:38.452 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule create-simple foo default host 2024-09-16T14:44:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.569+0000 7f62dfa18640 10 monclient: get_monmap_and_config 2024-09-16T14:44:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.569+0000 7f62dfa18640 10 monclient: build_initial_monmap 2024-09-16T14:44:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.569+0000 7f62dfa18640 10 monclient: monmap: 2024-09-16T14:44:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:38.572214+0000 2024-09-16T14:44:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:38.572214+0000 2024-09-16T14:44:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:38.573 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.569+0000 7f62dfa18640 1 Processor -- start 2024-09-16T14:44:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.569+0000 7f62dfa18640 1 -- start start 2024-09-16T14:44:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.569+0000 7f62dfa18640 10 monclient: init 2024-09-16T14:44:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.569+0000 7f62dfa18640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.569+0000 7f62dfa18640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:44:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.569+0000 7f62dfa18640 10 monclient(hunting): picked mon.noname-c con 0x7f62d8099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.569+0000 7f62dfa18640 10 monclient(hunting): picked mon.noname-b con 0x7f62d809c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.569+0000 7f62dfa18640 10 monclient(hunting): picked mon.noname-a con 0x7f62d809e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.569+0000 7f62dfa18640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d8099e50 con 0x7f62d809e6d0 2024-09-16T14:44:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.569+0000 7f62dd7b4640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f62d8099a70 legacy=0x7f62d809dfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.569+0000 7f62dfa18640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d809dbd0 con 0x7f62d8099a70 2024-09-16T14:44:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d809ee20 con 0x7f62d809c550 2024-09-16T14:44:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 10 monclient(hunting): authenticate will time out at 1060.688041s 2024-09-16T14:44:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dcfb3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f62d809c550 0x7f62d81379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:36140/0 (socket says 172.21.15.189:36140) 2024-09-16T14:44:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dcfb3640 1 -- 172.21.15.189:0/3320000449 learned_addr learned my addr 172.21.15.189:0/3320000449 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 1 -- 172.21.15.189:0/3320000449 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193479874 0 0) 0x7f62d8099e50 con 0x7f62d809e6d0 2024-09-16T14:44:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient(hunting): my global_id is 84243 2024-09-16T14:44:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 1 -- 172.21.15.189:0/3320000449 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62c4003530 con 0x7f62d809e6d0 2024-09-16T14:44:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 1 -- 172.21.15.189:0/3320000449 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233578393 0 0) 0x7f62d809ee20 con 0x7f62d809c550 2024-09-16T14:44:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient(hunting): my global_id is 4856 2024-09-16T14:44:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 1 -- 172.21.15.189:0/3320000449 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62d8099e50 con 0x7f62d809c550 2024-09-16T14:44:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 1 -- 172.21.15.189:0/3320000449 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1912223169 0 0) 0x7f62c4003530 con 0x7f62d809e6d0 2024-09-16T14:44:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 1 -- 172.21.15.189:0/3320000449 >> v1:172.21.15.26:6790/0 conn(0x7f62d809c550 legacy=0x7f62d81379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 1 -- 172.21.15.189:0/3320000449 >> v1:172.21.15.189:6789/0 conn(0x7f62d8099a70 legacy=0x7f62d809dfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient: _finish_hunting 0 2024-09-16T14:44:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 1 monclient: found mon.noname-a 2024-09-16T14:44:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 1 -- 172.21.15.189:0/3320000449 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d809f100 con 0x7f62d809e6d0 2024-09-16T14:44:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient: _finish_auth 0 2024-09-16T14:44:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient: _check_auth_tickets 2024-09-16T14:44:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 5 monclient: authenticate success, global_id 84243 2024-09-16T14:44:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 1 -- 172.21.15.189:0/3320000449 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f62d00019a0 con 0x7f62d809e6d0 2024-09-16T14:44:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient: dump: 2024-09-16T14:44:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:38.580 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 1 -- 172.21.15.189:0/3320000449 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62d0001c10 con 0x7f62d809e6d0 2024-09-16T14:44:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 1 -- 172.21.15.189:0/3320000449 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f62d0001fa0 con 0x7f62d809e6d0 2024-09-16T14:44:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 4 set_mon_vals no callback set 2024-09-16T14:44:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62cffff640 10 monclient: dump: 2024-09-16T14:44:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:38.582 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:38.582 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:38.582 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:38.582 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:38.582 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:38.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 10 monclient: shutdown 2024-09-16T14:44:38.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:38.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 1 -- 172.21.15.189:0/3320000449 >> v1:172.21.15.26:6789/0 conn(0x7f62d809e6d0 legacy=0x7f62d813b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:38.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 1 -- 172.21.15.189:0/3320000449 shutdown_connections 2024-09-16T14:44:38.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 1 -- 172.21.15.189:0/3320000449 >> 172.21.15.189:0/3320000449 conn(0x7f62d80a3380 msgr2=0x7f62d809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:38.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 1 -- 172.21.15.189:0/3320000449 shutdown_connections 2024-09-16T14:44:38.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 1 -- 172.21.15.189:0/3320000449 wait complete. 2024-09-16T14:44:38.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 10 monclient: build_initial_monmap 2024-09-16T14:44:38.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 10 monclient: monmap: 2024-09-16T14:44:38.583 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:38.583 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:38.583 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:38.583 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:38.583 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:38.583 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:38.583 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:38.583 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:38.583 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:38.584 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:38.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 1 Processor -- start 2024-09-16T14:44:38.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 1 -- start start 2024-09-16T14:44:38.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 10 monclient: init 2024-09-16T14:44:38.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:38.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:38.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 10 monclient(hunting): picked mon.noname-b con 0x7f62d8099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:38.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 10 monclient(hunting): picked mon.noname-a con 0x7f62d809c550 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:38.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 10 monclient(hunting): picked mon.noname-c con 0x7f62d809e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:38.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dd7b4640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f62d8099a70 legacy=0x7f62d81c9930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:38.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d809cf20 con 0x7f62d809c550 2024-09-16T14:44:38.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d8125d20 con 0x7f62d8099a70 2024-09-16T14:44:38.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d81243c0 con 0x7f62d809e6d0 2024-09-16T14:44:38.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:38.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dfa18640 10 monclient(hunting): authenticate will time out at 1060.691675s 2024-09-16T14:44:38.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dcfb3640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f62d809c550 0x7f62d81ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:52858/0 (socket says 172.21.15.189:52858) 2024-09-16T14:44:38.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.573+0000 7f62dcfb3640 1 -- 172.21.15.189:0/4213685240 learned_addr learned my addr 172.21.15.189:0/4213685240 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:38.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061358303 0 0) 0x7f62d809cf20 con 0x7f62d809c550 2024-09-16T14:44:38.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:38.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:38.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:38.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient(hunting): my global_id is 84246 2024-09-16T14:44:38.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62bc003530 con 0x7f62d809c550 2024-09-16T14:44:38.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305847789 0 0) 0x7f62d81243c0 con 0x7f62d809e6d0 2024-09-16T14:44:38.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:38.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:38.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:38.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient(hunting): my global_id is 4859 2024-09-16T14:44:38.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62d809cf20 con 0x7f62d809e6d0 2024-09-16T14:44:38.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3459885758 0 0) 0x7f62bc003530 con 0x7f62d809c550 2024-09-16T14:44:38.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62d81243c0 con 0x7f62d809c550 2024-09-16T14:44:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f62c8001900 con 0x7f62d809c550 2024-09-16T14:44:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient(hunting): dump: 2024-09-16T14:44:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:38.591 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:38.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:38.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 409385019 0 0) 0x7f62d809cf20 con 0x7f62d809e6d0 2024-09-16T14:44:38.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62bc003530 con 0x7f62d809e6d0 2024-09-16T14:44:38.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f62d0001f20 con 0x7f62d809e6d0 2024-09-16T14:44:38.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:38.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:38.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient(hunting): dump: 2024-09-16T14:44:38.591 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:38.591 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:38.591 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.591 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.591 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:38.592 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:38.592 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:38.592 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:38.592 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:38.592 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:38.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4279385265 0 0) 0x7f62d81243c0 con 0x7f62d809c550 2024-09-16T14:44:38.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 >> v1:172.21.15.26:6790/0 conn(0x7f62d809e6d0 legacy=0x7f62d81d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:38.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 >> v1:172.21.15.189:6789/0 conn(0x7f62d8099a70 legacy=0x7f62d81c9930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:38.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient: _finish_hunting 0 2024-09-16T14:44:38.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 monclient: found mon.a 2024-09-16T14:44:38.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:38.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:38.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d806f9a0 con 0x7f62d809c550 2024-09-16T14:44:38.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient: _finish_auth 0 2024-09-16T14:44:38.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient: _check_auth_tickets 2024-09-16T14:44:38.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:38.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62dfa18640 5 monclient: authenticate success, global_id 84246 2024-09-16T14:44:38.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62dfa18640 10 monclient: _renew_subs 2024-09-16T14:44:38.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62dfa18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:38.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62dfa18640 1 -- 172.21.15.189:0/4213685240 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d8125760 con 0x7f62d809c550 2024-09-16T14:44:38.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62dfa18640 10 monclient: _renew_subs 2024-09-16T14:44:38.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62dfa18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:38.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62dfa18640 1 -- 172.21.15.189:0/4213685240 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d806ced0 con 0x7f62d809c550 2024-09-16T14:44:38.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62c80047f0 con 0x7f62d809c550 2024-09-16T14:44:38.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:38.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cf7fe640 4 set_mon_vals no callback set 2024-09-16T14:44:38.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f62c8004b00 con 0x7f62d809c550 2024-09-16T14:44:38.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:38.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:38.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 10 monclient: dump: 2024-09-16T14:44:38.594 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:38.594 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:38.594 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.594 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:38.594 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:38.594 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:38.594 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:38.594 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:38.594 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:38.594 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:38.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62dfa18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:38.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62dfa18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:38.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62dfa18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:38.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62dfa18640 1 -- 172.21.15.189:0/4213685240 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b8005050 con 0x7f62d809c550 2024-09-16T14:44:38.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f62c800f5f0 con 0x7f62d809c550 2024-09-16T14:44:38.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(40..40 src has 1..40) v4 ==== 3598+0+0 (unknown 1780700966 0 0) 0x7f62c8043110 con 0x7f62d809c550 2024-09-16T14:44:38.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.577+0000 7f62dd7b4640 10 monclient: get_auth_request con 0x7f62bc036250 auth_method 0 2024-09-16T14:44:38.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.589+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f62c8014470 con 0x7f62d809c550 2024-09-16T14:44:38.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.589+0000 7f62cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:38.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.589+0000 7f62cdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:38.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.749+0000 7f62dfa18640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T14:44:38.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.749+0000 7f62dfa18640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T14:44:38.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.749+0000 7f62dfa18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:38.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.749+0000 7f62dfa18640 1 -- 172.21.15.189:0/4213685240 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7f62b8005300 con 0x7f62d809c550 2024-09-16T14:44:38.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.933+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 v41) v1 ==== 126+0+0 (unknown 2274706495 0 0) 0x7f62c8014290 con 0x7f62d809c550 2024-09-16T14:44:38.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.933+0000 7f62cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T14:44:38.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:38.933+0000 7f62cdffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.033+0000 7f62dfa18640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T14:44:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.033+0000 7f62dfa18640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T14:44:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.033+0000 7f62dfa18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.033+0000 7f62dfa18640 1 -- 172.21.15.189:0/4213685240 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7f62b8001150 con 0x7f62d809c550 2024-09-16T14:44:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.033+0000 7f62cdffb640 1 -- 172.21.15.189:0/4213685240 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v41) v1 ==== 149+0+0 (unknown 3457866127 0 0) 0x7f62c8014fd0 con 0x7f62d809c550 2024-09-16T14:44:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.033+0000 7f62cdffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T14:44:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.033+0000 7f62cdffb640 10 monclient: _finish_command 3 = system:0 rule foo already exists 2024-09-16T14:44:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:rule foo already exists 2024-09-16T14:44:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.037+0000 7f62dfa18640 1 -- 172.21.15.189:0/4213685240 >> v1:172.21.15.189:6812/3196161582 conn(0x7f62bc036250 legacy=0x7f62bc038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.037+0000 7f62dfa18640 10 monclient: shutdown 2024-09-16T14:44:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.037+0000 7f62dfa18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.037+0000 7f62dfa18640 1 -- 172.21.15.189:0/4213685240 >> v1:172.21.15.26:6789/0 conn(0x7f62d809c550 legacy=0x7f62d81ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.037+0000 7f62dfa18640 1 -- 172.21.15.189:0/4213685240 shutdown_connections 2024-09-16T14:44:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.037+0000 7f62dfa18640 1 -- 172.21.15.189:0/4213685240 >> 172.21.15.189:0/4213685240 conn(0x7f62d80a3380 msgr2=0x7f62d806f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.037+0000 7f62dfa18640 1 -- 172.21.15.189:0/4213685240 shutdown_connections 2024-09-16T14:44:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.037+0000 7f62dfa18640 1 -- 172.21.15.189:0/4213685240 wait complete. 2024-09-16T14:44:39.054 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule create-simple foo default host 2024-09-16T14:44:39.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.157+0000 7f7a4d6ea640 10 monclient: get_monmap_and_config 2024-09-16T14:44:39.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.157+0000 7f7a4d6ea640 10 monclient: build_initial_monmap 2024-09-16T14:44:39.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.157+0000 7f7a4d6ea640 10 monclient: monmap: 2024-09-16T14:44:39.161 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:39.161 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:39.161 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:39.160450+0000 2024-09-16T14:44:39.161 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:39.160450+0000 2024-09-16T14:44:39.161 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:39.161 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:39.161 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:39.161 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:39.161 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:39.161 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:39.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.157+0000 7f7a4d6ea640 1 Processor -- start 2024-09-16T14:44:39.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 1 -- start start 2024-09-16T14:44:39.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 10 monclient: init 2024-09-16T14:44:39.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:39.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:39.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 10 monclient(hunting): picked mon.noname-b con 0x7f7a4809c8f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:39.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 10 monclient(hunting): picked mon.noname-a con 0x7f7a4809f0d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:39.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 10 monclient(hunting): picked mon.noname-c con 0x7f7a48139100 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:39.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4809dd00 con 0x7f7a4809f0d0 2024-09-16T14:44:39.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4809e410 con 0x7f7a48139100 2024-09-16T14:44:39.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4809e6f0 con 0x7f7a4809c8f0 2024-09-16T14:44:39.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:39.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 10 monclient(hunting): authenticate will time out at 1061.277017s 2024-09-16T14:44:39.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a477fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f7a48139100 legacy=0x7f7a4813b4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:39.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a467fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7a4809f0d0 0x7f7a481379e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:32874/0 (socket says 172.21.15.189:32874) 2024-09-16T14:44:39.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a46ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7a4809c8f0 0x7f7a4809ccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:39388/0 (socket says 172.21.15.189:39388) 2024-09-16T14:44:39.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a467fc640 1 -- 172.21.15.189:0/1499327203 learned_addr learned my addr 172.21.15.189:0/1499327203 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:39.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 1 -- 172.21.15.189:0/1499327203 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497263725 0 0) 0x7f7a4809dd00 con 0x7f7a4809f0d0 2024-09-16T14:44:39.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:39.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:39.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:39.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient(hunting): my global_id is 84249 2024-09-16T14:44:39.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 1 -- 172.21.15.189:0/1499327203 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a30003530 con 0x7f7a4809f0d0 2024-09-16T14:44:39.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 1 -- 172.21.15.189:0/1499327203 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231127520 0 0) 0x7f7a4809e6f0 con 0x7f7a4809c8f0 2024-09-16T14:44:39.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:39.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:39.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:39.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient(hunting): my global_id is 4862 2024-09-16T14:44:39.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 1 -- 172.21.15.189:0/1499327203 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a4809dd00 con 0x7f7a4809c8f0 2024-09-16T14:44:39.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 1 -- 172.21.15.189:0/1499327203 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 331161347 0 0) 0x7f7a4809dd00 con 0x7f7a4809c8f0 2024-09-16T14:44:39.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 1 -- 172.21.15.189:0/1499327203 >> v1:172.21.15.189:6789/0 conn(0x7f7a48139100 legacy=0x7f7a4813b4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:39.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 1 -- 172.21.15.189:0/1499327203 >> v1:172.21.15.26:6789/0 conn(0x7f7a4809f0d0 legacy=0x7f7a481379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:39.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:44:39.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 1 monclient: found mon.noname-b 2024-09-16T14:44:39.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:39.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:39.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 1 -- 172.21.15.189:0/1499327203 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a480a1400 con 0x7f7a4809c8f0 2024-09-16T14:44:39.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient: _finish_auth 0 2024-09-16T14:44:39.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:44:39.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:39.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 1 -- 172.21.15.189:0/1499327203 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a38001600 con 0x7f7a4809c8f0 2024-09-16T14:44:39.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:39.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:39.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient: dump: 2024-09-16T14:44:39.168 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:39.168 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:39.169 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.169 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.169 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:39.169 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:39.169 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:39.169 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:39.169 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:39.169 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:39.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 5 monclient: authenticate success, global_id 4862 2024-09-16T14:44:39.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:39.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 1 -- 172.21.15.189:0/1499327203 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a380018a0 con 0x7f7a4809c8f0 2024-09-16T14:44:39.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:39.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 1 -- 172.21.15.189:0/1499327203 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a38001c70 con 0x7f7a4809c8f0 2024-09-16T14:44:39.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:39.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:39.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a45ffb640 10 monclient: dump: 2024-09-16T14:44:39.170 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:39.170 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:39.170 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.170 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.170 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:39.170 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:39.170 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:39.170 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:39.170 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:39.171 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:39.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:39.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 4 set_mon_vals no callback set 2024-09-16T14:44:39.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 10 monclient: shutdown 2024-09-16T14:44:39.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:39.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 1 -- 172.21.15.189:0/1499327203 >> v1:172.21.15.26:6790/0 conn(0x7f7a4809c8f0 legacy=0x7f7a4809ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:39.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 1 -- 172.21.15.189:0/1499327203 shutdown_connections 2024-09-16T14:44:39.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 1 -- 172.21.15.189:0/1499327203 >> 172.21.15.189:0/1499327203 conn(0x7f7a480a1920 msgr2=0x7f7a480a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:39.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 1 -- 172.21.15.189:0/1499327203 shutdown_connections 2024-09-16T14:44:39.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.161+0000 7f7a4d6ea640 1 -- 172.21.15.189:0/1499327203 wait complete. 2024-09-16T14:44:39.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 10 monclient: build_initial_monmap 2024-09-16T14:44:39.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 10 monclient: monmap: 2024-09-16T14:44:39.172 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:39.172 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:39.172 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:39.172 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:39.172 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:39.172 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:39.172 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:39.172 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:39.172 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:39.172 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:39.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 1 Processor -- start 2024-09-16T14:44:39.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 1 -- start start 2024-09-16T14:44:39.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 10 monclient: init 2024-09-16T14:44:39.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:39.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:44:39.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 10 monclient(hunting): picked mon.noname-c con 0x7f7a4809c8f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:39.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 10 monclient(hunting): picked mon.noname-b con 0x7f7a4809f0d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:39.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 10 monclient(hunting): picked mon.noname-a con 0x7f7a48139100 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:39.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a481223d0 con 0x7f7a48139100 2024-09-16T14:44:39.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a467fc640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f7a4809f0d0 legacy=0x7f7a481cc4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:39.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4806e7f0 con 0x7f7a4809f0d0 2024-09-16T14:44:39.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4809c110 con 0x7f7a4809c8f0 2024-09-16T14:44:39.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:39.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 10 monclient(hunting): authenticate will time out at 1061.281365s 2024-09-16T14:44:39.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a46ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7a4809c8f0 0x7f7a481c74e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:39402/0 (socket says 172.21.15.189:39402) 2024-09-16T14:44:39.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a46ffd640 1 -- 172.21.15.189:0/1690890720 learned_addr learned my addr 172.21.15.189:0/1690890720 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:39.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941643564 0 0) 0x7f7a481223d0 con 0x7f7a48139100 2024-09-16T14:44:39.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:39.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:39.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:39.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient(hunting): my global_id is 84252 2024-09-16T14:44:39.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a34003530 con 0x7f7a48139100 2024-09-16T14:44:39.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586248240 0 0) 0x7f7a4809c110 con 0x7f7a4809c8f0 2024-09-16T14:44:39.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:39.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:39.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:39.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient(hunting): my global_id is 4865 2024-09-16T14:44:39.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a481223d0 con 0x7f7a4809c8f0 2024-09-16T14:44:39.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1121617372 0 0) 0x7f7a34003530 con 0x7f7a48139100 2024-09-16T14:44:39.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a4809c110 con 0x7f7a48139100 2024-09-16T14:44:39.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a40004340 con 0x7f7a48139100 2024-09-16T14:44:39.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:39.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:39.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient(hunting): dump: 2024-09-16T14:44:39.179 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:39.179 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:39.179 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.180 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.180 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:39.180 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:39.180 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:39.180 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:39.180 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:39.180 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:39.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:39.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4215799806 0 0) 0x7f7a481223d0 con 0x7f7a4809c8f0 2024-09-16T14:44:39.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a34003530 con 0x7f7a4809c8f0 2024-09-16T14:44:39.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a38001bf0 con 0x7f7a4809c8f0 2024-09-16T14:44:39.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:39.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:39.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient(hunting): dump: 2024-09-16T14:44:39.181 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:39.181 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:39.181 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.181 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.181 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:39.181 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:39.181 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:39.181 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:39.181 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:39.181 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:39.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2366288277 0 0) 0x7f7a34003530 con 0x7f7a4809c8f0 2024-09-16T14:44:39.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 >> v1:172.21.15.189:6789/0 conn(0x7f7a4809f0d0 legacy=0x7f7a481cc4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:39.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 >> v1:172.21.15.26:6789/0 conn(0x7f7a48139100 legacy=0x7f7a481cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:39.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:44:39.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 1 monclient: found mon.c 2024-09-16T14:44:39.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:39.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:39.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a48120910 con 0x7f7a4809c8f0 2024-09-16T14:44:39.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:44:39.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:44:39.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:39.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 5 monclient: authenticate success, global_id 4865 2024-09-16T14:44:39.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 10 monclient: _renew_subs 2024-09-16T14:44:39.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:39.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 1 -- 172.21.15.189:0/1690890720 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a48121260 con 0x7f7a4809c8f0 2024-09-16T14:44:39.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 10 monclient: _renew_subs 2024-09-16T14:44:39.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:39.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.165+0000 7f7a4d6ea640 1 -- 172.21.15.189:0/1690890720 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a481385a0 con 0x7f7a4809c8f0 2024-09-16T14:44:39.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.169+0000 7f7a4d6ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:39.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.169+0000 7f7a4d6ea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:39.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.169+0000 7f7a4d6ea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:39.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.169+0000 7f7a4d6ea640 1 -- 172.21.15.189:0/1690890720 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4809c2a0 con 0x7f7a4809c8f0 2024-09-16T14:44:39.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.169+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a38004b40 con 0x7f7a4809c8f0 2024-09-16T14:44:39.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.169+0000 7f7a2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:39.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.169+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a38004e50 con 0x7f7a4809c8f0 2024-09-16T14:44:39.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.169+0000 7f7a2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:39.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.169+0000 7f7a2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:39.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.169+0000 7f7a2f7fe640 10 monclient: dump: 2024-09-16T14:44:39.184 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:39.184 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:39.184 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.185 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.185 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:39.185 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:39.185 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:39.185 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:39.185 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:39.185 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:39.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.169+0000 7f7a457fa640 4 set_mon_vals no callback set 2024-09-16T14:44:39.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.169+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7a3800f8c0 con 0x7f7a4809c8f0 2024-09-16T14:44:39.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.169+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(41..41 src has 15..41) v4 ==== 3657+0+0 (unknown 4128117240 0 0) 0x7f7a38043220 con 0x7f7a4809c8f0 2024-09-16T14:44:39.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.169+0000 7f7a467fc640 10 monclient: get_auth_request con 0x7f7a34036270 auth_method 0 2024-09-16T14:44:39.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.181+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7a380145a0 con 0x7f7a4809c8f0 2024-09-16T14:44:39.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.181+0000 7f7a2f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:39.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.181+0000 7f7a2f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:39.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.329+0000 7f7a4d6ea640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T14:44:39.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.329+0000 7f7a4d6ea640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T14:44:39.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.329+0000 7f7a4d6ea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:39.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.329+0000 7f7a4d6ea640 1 -- 172.21.15.189:0/1690890720 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7f7a4809dfb0 con 0x7f7a4809c8f0 2024-09-16T14:44:39.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.329+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v41) v1 ==== 149+0+0 (unknown 3457866127 0 0) 0x7f7a38012c40 con 0x7f7a4809c8f0 2024-09-16T14:44:39.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.329+0000 7f7a2f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T14:44:39.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.329+0000 7f7a2f7fe640 10 monclient: _finish_command 2 = system:0 rule foo already exists 2024-09-16T14:44:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.417+0000 7f7a4d6ea640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T14:44:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.417+0000 7f7a4d6ea640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T14:44:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.417+0000 7f7a4d6ea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.417+0000 7f7a4d6ea640 1 -- 172.21.15.189:0/1690890720 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7f7a48122840 con 0x7f7a4809c8f0 2024-09-16T14:44:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.421+0000 7f7a2f7fe640 1 -- 172.21.15.189:0/1690890720 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v41) v1 ==== 149+0+0 (unknown 3457866127 0 0) 0x7f7a380143c0 con 0x7f7a4809c8f0 2024-09-16T14:44:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.421+0000 7f7a2f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T14:44:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.421+0000 7f7a2f7fe640 10 monclient: _finish_command 3 = system:0 rule foo already exists 2024-09-16T14:44:39.423 INFO:tasks.workunit.client.0.smithi189.stderr:rule foo already exists 2024-09-16T14:44:39.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.421+0000 7f7a4d6ea640 1 -- 172.21.15.189:0/1690890720 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7a34036270 legacy=0x7f7a34038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:39.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.421+0000 7f7a4d6ea640 10 monclient: shutdown 2024-09-16T14:44:39.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.421+0000 7f7a4d6ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:39.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.421+0000 7f7a4d6ea640 1 -- 172.21.15.189:0/1690890720 >> v1:172.21.15.26:6790/0 conn(0x7f7a4809c8f0 legacy=0x7f7a481c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:39.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.421+0000 7f7a4d6ea640 1 -- 172.21.15.189:0/1690890720 shutdown_connections 2024-09-16T14:44:39.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.421+0000 7f7a4d6ea640 1 -- 172.21.15.189:0/1690890720 >> 172.21.15.189:0/1690890720 conn(0x7f7a480a1920 msgr2=0x7f7a4806fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:39.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.421+0000 7f7a4d6ea640 1 -- 172.21.15.189:0/1690890720 shutdown_connections 2024-09-16T14:44:39.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.421+0000 7f7a4d6ea640 1 -- 172.21.15.189:0/1690890720 wait complete. 2024-09-16T14:44:39.437 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule create-simple bar default host 2024-09-16T14:44:39.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.553+0000 7f7d76a0e640 10 monclient: get_monmap_and_config 2024-09-16T14:44:39.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.553+0000 7f7d76a0e640 10 monclient: build_initial_monmap 2024-09-16T14:44:39.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.553+0000 7f7d76a0e640 10 monclient: monmap: 2024-09-16T14:44:39.556 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:39.556 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:39.557 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:39.556160+0000 2024-09-16T14:44:39.557 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:39.556160+0000 2024-09-16T14:44:39.557 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:39.557 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:39.557 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:39.557 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:39.557 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:39.557 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:39.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.553+0000 7f7d76a0e640 1 Processor -- start 2024-09-16T14:44:39.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.553+0000 7f7d76a0e640 1 -- start start 2024-09-16T14:44:39.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.553+0000 7f7d76a0e640 10 monclient: init 2024-09-16T14:44:39.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:39.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:44:39.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f7d70070830 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:39.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f7d70071320 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:39.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f7d70134cf0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:39.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d70135740 con 0x7f7d70134cf0 2024-09-16T14:44:39.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d70135160 con 0x7f7d70070830 2024-09-16T14:44:39.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7009d810 con 0x7f7d70071320 2024-09-16T14:44:39.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:39.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 10 monclient(hunting): authenticate will time out at 1061.672512s 2024-09-16T14:44:39.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6ffff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f7d70070830 legacy=0x7f7d70070c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:39.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d74fab640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7d70134cf0 0x7f7d7013d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:32896/0 (socket says 172.21.15.189:32896) 2024-09-16T14:44:39.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d74fab640 1 -- 172.21.15.189:0/1957695229 learned_addr learned my addr 172.21.15.189:0/1957695229 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:39.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d677fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7d70071320 0x7f7d70139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:39406/0 (socket says 172.21.15.189:39406) 2024-09-16T14:44:39.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 1 -- 172.21.15.189:0/1957695229 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784151359 0 0) 0x7f7d70135740 con 0x7f7d70134cf0 2024-09-16T14:44:39.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:39.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:39.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:39.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient(hunting): my global_id is 84255 2024-09-16T14:44:39.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 1 -- 172.21.15.189:0/1957695229 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d5c003530 con 0x7f7d70134cf0 2024-09-16T14:44:39.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 1 -- 172.21.15.189:0/1957695229 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804783197 0 0) 0x7f7d7009d810 con 0x7f7d70071320 2024-09-16T14:44:39.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:39.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:39.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:39.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient(hunting): my global_id is 4868 2024-09-16T14:44:39.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 1 -- 172.21.15.189:0/1957695229 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d70135740 con 0x7f7d70071320 2024-09-16T14:44:39.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 1 -- 172.21.15.189:0/1957695229 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2248476718 0 0) 0x7f7d5c003530 con 0x7f7d70134cf0 2024-09-16T14:44:39.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 1 -- 172.21.15.189:0/1957695229 >> v1:172.21.15.26:6790/0 conn(0x7f7d70071320 legacy=0x7f7d70139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:39.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 1 -- 172.21.15.189:0/1957695229 >> v1:172.21.15.189:6789/0 conn(0x7f7d70070830 legacy=0x7f7d70070c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:39.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:44:39.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 1 monclient: found mon.noname-a 2024-09-16T14:44:39.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:39.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:39.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 1 -- 172.21.15.189:0/1957695229 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7009f1c0 con 0x7f7d70134cf0 2024-09-16T14:44:39.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:44:39.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:44:39.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:39.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 5 monclient: authenticate success, global_id 84255 2024-09-16T14:44:39.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 1 -- 172.21.15.189:0/1957695229 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7d680019a0 con 0x7f7d70134cf0 2024-09-16T14:44:39.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:39.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:39.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient: dump: 2024-09-16T14:44:39.567 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:39.567 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:39.568 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.568 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.568 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:39.568 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:39.569 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:39.569 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:39.569 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:39.569 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:39.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:39.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 1 -- 172.21.15.189:0/1957695229 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d68001bc0 con 0x7f7d70134cf0 2024-09-16T14:44:39.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:39.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 1 -- 172.21.15.189:0/1957695229 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7d68001f50 con 0x7f7d70134cf0 2024-09-16T14:44:39.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:39.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 4 set_mon_vals no callback set 2024-09-16T14:44:39.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:39.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:39.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d6f7fe640 10 monclient: dump: 2024-09-16T14:44:39.570 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:39.570 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:39.570 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.570 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.570 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:39.570 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:39.570 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:39.570 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:39.570 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:39.570 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:39.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 10 monclient: shutdown 2024-09-16T14:44:39.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:39.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 1 -- 172.21.15.189:0/1957695229 >> v1:172.21.15.26:6789/0 conn(0x7f7d70134cf0 legacy=0x7f7d7013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 1 -- 172.21.15.189:0/1957695229 shutdown_connections 2024-09-16T14:44:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 1 -- 172.21.15.189:0/1957695229 >> 172.21.15.189:0/1957695229 conn(0x7f7d700a3380 msgr2=0x7f7d7009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 1 -- 172.21.15.189:0/1957695229 shutdown_connections 2024-09-16T14:44:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 1 -- 172.21.15.189:0/1957695229 wait complete. 2024-09-16T14:44:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 10 monclient: build_initial_monmap 2024-09-16T14:44:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 10 monclient: monmap: 2024-09-16T14:44:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:39.572 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:39.572 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:39.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 1 Processor -- start 2024-09-16T14:44:39.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 1 -- start start 2024-09-16T14:44:39.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.557+0000 7f7d76a0e640 10 monclient: init 2024-09-16T14:44:39.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:39.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:44:39.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f7d70070830 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:39.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f7d70071320 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:39.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f7d70134cf0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:39.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7006b320 con 0x7f7d70134cf0 2024-09-16T14:44:39.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d701200c0 con 0x7f7d70070830 2024-09-16T14:44:39.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d701326e0 con 0x7f7d70071320 2024-09-16T14:44:39.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:39.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6ffff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f7d70070830 legacy=0x7f7d701316f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:39.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 10 monclient(hunting): authenticate will time out at 1061.676427s 2024-09-16T14:44:39.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d74fab640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7d70134cf0 0x7f7d701d18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:32912/0 (socket says 172.21.15.189:32912) 2024-09-16T14:44:39.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d677fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7d70071320 0x7f7d701ce1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:39420/0 (socket says 172.21.15.189:39420) 2024-09-16T14:44:39.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d74fab640 1 -- 172.21.15.189:0/4153543743 learned_addr learned my addr 172.21.15.189:0/4153543743 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:39.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422699607 0 0) 0x7f7d7006b320 con 0x7f7d70134cf0 2024-09-16T14:44:39.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:44:39.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:39.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:39.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient(hunting): my global_id is 84258 2024-09-16T14:44:39.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d50003530 con 0x7f7d70134cf0 2024-09-16T14:44:39.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277332375 0 0) 0x7f7d701326e0 con 0x7f7d70071320 2024-09-16T14:44:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:44:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient(hunting): my global_id is 4871 2024-09-16T14:44:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d7006b320 con 0x7f7d70071320 2024-09-16T14:44:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1717469642 0 0) 0x7f7d50003530 con 0x7f7d70134cf0 2024-09-16T14:44:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d701326e0 con 0x7f7d70134cf0 2024-09-16T14:44:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7d68001ed0 con 0x7f7d70134cf0 2024-09-16T14:44:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient(hunting): dump: 2024-09-16T14:44:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:39.575 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:39.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2042704265 0 0) 0x7f7d7006b320 con 0x7f7d70071320 2024-09-16T14:44:39.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d50003530 con 0x7f7d70071320 2024-09-16T14:44:39.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7d58001910 con 0x7f7d70071320 2024-09-16T14:44:39.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:39.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:39.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient(hunting): dump: 2024-09-16T14:44:39.576 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:39.576 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:39.576 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.576 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.576 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:39.576 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3128006129 0 0) 0x7f7d701326e0 con 0x7f7d70134cf0 2024-09-16T14:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 >> v1:172.21.15.26:6790/0 conn(0x7f7d70071320 legacy=0x7f7d701ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 >> v1:172.21.15.189:6789/0 conn(0x7f7d70070830 legacy=0x7f7d701316f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 monclient: found mon.a 2024-09-16T14:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d70132c10 con 0x7f7d70134cf0 2024-09-16T14:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:39.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:39.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 5 monclient: authenticate success, global_id 84258 2024-09-16T14:44:39.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 10 monclient: _renew_subs 2024-09-16T14:44:39.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:39.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 1 -- 172.21.15.189:0/4153543743 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d7011fde0 con 0x7f7d70134cf0 2024-09-16T14:44:39.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 10 monclient: _renew_subs 2024-09-16T14:44:39.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:39.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 1 -- 172.21.15.189:0/4153543743 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d7011dd60 con 0x7f7d70134cf0 2024-09-16T14:44:39.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d68004e20 con 0x7f7d70134cf0 2024-09-16T14:44:39.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:39.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7d68005130 con 0x7f7d70134cf0 2024-09-16T14:44:39.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:39.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:39.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 10 monclient: dump: 2024-09-16T14:44:39.579 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:39.579 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:39.579 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.579 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:39.579 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:39.579 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:39.579 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:39.579 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:39.579 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:39.579 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:39.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6effd640 4 set_mon_vals no callback set 2024-09-16T14:44:39.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:39.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:39.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:39.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d76a0e640 1 -- 172.21.15.189:0/4153543743 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d54005050 con 0x7f7d70134cf0 2024-09-16T14:44:39.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7d6800fba0 con 0x7f7d70134cf0 2024-09-16T14:44:39.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(41..41 src has 15..41) v4 ==== 3657+0+0 (unknown 4128117240 0 0) 0x7f7d680435f0 con 0x7f7d70134cf0 2024-09-16T14:44:39.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.561+0000 7f7d6ffff640 10 monclient: get_auth_request con 0x7f7d50036220 auth_method 0 2024-09-16T14:44:39.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.573+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7d680149d0 con 0x7f7d70134cf0 2024-09-16T14:44:39.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.573+0000 7f7d6d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:39.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.573+0000 7f7d6d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:39.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.737+0000 7f7d76a0e640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T14:44:39.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.737+0000 7f7d76a0e640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T14:44:39.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.737+0000 7f7d76a0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:39.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.737+0000 7f7d76a0e640 1 -- 172.21.15.189:0/4153543743 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"} v 0) v1 -- 0x7f7d54005300 con 0x7f7d70134cf0 2024-09-16T14:44:39.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.937+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}]=0 v42) v1 ==== 126+0+0 (unknown 3427738925 0 0) 0x7f7d680147f0 con 0x7f7d70134cf0 2024-09-16T14:44:39.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.937+0000 7f7d6d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T14:44:39.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:39.937+0000 7f7d6d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:40.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.033+0000 7f7d76a0e640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T14:44:40.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.033+0000 7f7d76a0e640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T14:44:40.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.033+0000 7f7d76a0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:40.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.033+0000 7f7d76a0e640 1 -- 172.21.15.189:0/4153543743 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"} v 0) v1 -- 0x7f7d54001150 con 0x7f7d70134cf0 2024-09-16T14:44:40.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.033+0000 7f7d6d7fa640 1 -- 172.21.15.189:0/4153543743 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}]=0 rule bar already exists v42) v1 ==== 149+0+0 (unknown 1808352653 0 0) 0x7f7d68015530 con 0x7f7d70134cf0 2024-09-16T14:44:40.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.033+0000 7f7d6d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T14:44:40.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.033+0000 7f7d6d7fa640 10 monclient: _finish_command 3 = system:0 rule bar already exists 2024-09-16T14:44:40.037 INFO:tasks.workunit.client.0.smithi189.stderr:rule bar already exists 2024-09-16T14:44:40.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.037+0000 7f7d76a0e640 1 -- 172.21.15.189:0/4153543743 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7d50036220 legacy=0x7f7d500386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:40.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.037+0000 7f7d76a0e640 10 monclient: shutdown 2024-09-16T14:44:40.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.037+0000 7f7d76a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:40.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.037+0000 7f7d76a0e640 1 -- 172.21.15.189:0/4153543743 >> v1:172.21.15.26:6789/0 conn(0x7f7d70134cf0 legacy=0x7f7d701d18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:40.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.037+0000 7f7d76a0e640 1 -- 172.21.15.189:0/4153543743 shutdown_connections 2024-09-16T14:44:40.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.037+0000 7f7d76a0e640 1 -- 172.21.15.189:0/4153543743 >> 172.21.15.189:0/4153543743 conn(0x7f7d700a3380 msgr2=0x7f7d701bbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:40.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.037+0000 7f7d76a0e640 1 -- 172.21.15.189:0/4153543743 shutdown_connections 2024-09-16T14:44:40.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.037+0000 7f7d76a0e640 1 -- 172.21.15.189:0/4153543743 wait complete. 2024-09-16T14:44:40.052 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm-device-class all 2024-09-16T14:44:40.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 10 monclient: get_monmap_and_config 2024-09-16T14:44:40.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 10 monclient: build_initial_monmap 2024-09-16T14:44:40.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 10 monclient: monmap: 2024-09-16T14:44:40.171 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:40.171 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:40.171 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:40.170650+0000 2024-09-16T14:44:40.171 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:40.170650+0000 2024-09-16T14:44:40.171 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:40.171 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:40.171 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:40.171 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:40.171 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:40.171 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:40.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 1 Processor -- start 2024-09-16T14:44:40.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 1 -- start start 2024-09-16T14:44:40.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 10 monclient: init 2024-09-16T14:44:40.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:40.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:40.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e1c0a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:40.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e1c0a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:40.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e1c13b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:40.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c0a30b0 con 0x7f6e1c0a1b00 2024-09-16T14:44:40.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c0a2e30 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c09eb50 con 0x7f6e1c13b430 2024-09-16T14:44:40.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 10 monclient(hunting): authenticate will time out at 1062.286405s 2024-09-16T14:44:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e19d9b640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f6e1c0a25f0 legacy=0x7f6e1c139d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1a59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6e1c0a1b00 0x7f6e1c0a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:32920/0 (socket says 172.21.15.189:32920) 2024-09-16T14:44:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1a59c640 1 -- 172.21.15.189:0/2079243703 learned_addr learned my addr 172.21.15.189:0/2079243703 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1ad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6e1c13b430 0x7f6e1c13d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:39434/0 (socket says 172.21.15.189:39434) 2024-09-16T14:44:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 1 -- 172.21.15.189:0/2079243703 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107482293 0 0) 0x7f6e1c09eb50 con 0x7f6e1c13b430 2024-09-16T14:44:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 10 monclient(hunting): _negotiate 2024-09-16T14:44:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 10 monclient(hunting): my global_id is 4874 2024-09-16T14:44:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 1 -- 172.21.15.189:0/2079243703 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e04003530 con 0x7f6e1c13b430 2024-09-16T14:44:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 1 -- 172.21.15.189:0/2079243703 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532745832 0 0) 0x7f6e1c0a30b0 con 0x7f6e1c0a1b00 2024-09-16T14:44:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 10 monclient(hunting): _negotiate 2024-09-16T14:44:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 10 monclient(hunting): my global_id is 84261 2024-09-16T14:44:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 1 -- 172.21.15.189:0/2079243703 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e1c09eb50 con 0x7f6e1c0a1b00 2024-09-16T14:44:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 1 -- 172.21.15.189:0/2079243703 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3996856372 0 0) 0x7f6e04003530 con 0x7f6e1c13b430 2024-09-16T14:44:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 1 -- 172.21.15.189:0/2079243703 >> v1:172.21.15.189:6789/0 conn(0x7f6e1c0a25f0 legacy=0x7f6e1c139d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 1 -- 172.21.15.189:0/2079243703 >> v1:172.21.15.26:6789/0 conn(0x7f6e1c0a1b00 legacy=0x7f6e1c0a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 10 monclient: _finish_hunting 0 2024-09-16T14:44:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 1 monclient: found mon.noname-b 2024-09-16T14:44:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 1 -- 172.21.15.189:0/2079243703 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c09ee30 con 0x7f6e1c13b430 2024-09-16T14:44:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 10 monclient: _finish_auth 0 2024-09-16T14:44:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 10 monclient: _check_auth_tickets 2024-09-16T14:44:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 1 -- 172.21.15.189:0/2079243703 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e140019a0 con 0x7f6e1c13b430 2024-09-16T14:44:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e1959a640 10 monclient: dump: 2024-09-16T14:44:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:40.179 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:40.179 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:40.180 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:40.180 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:40.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 5 monclient: authenticate success, global_id 4874 2024-09-16T14:44:40.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.169+0000 7f6e2096d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:40.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e1959a640 1 -- 172.21.15.189:0/2079243703 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e14001c80 con 0x7f6e1c13b430 2024-09-16T14:44:40.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e1959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:40.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e1959a640 1 -- 172.21.15.189:0/2079243703 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e14002010 con 0x7f6e1c13b430 2024-09-16T14:44:40.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e1959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:40.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e1959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:40.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e1959a640 10 monclient: dump: 2024-09-16T14:44:40.181 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:40.181 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:40.181 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:40.181 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:40.181 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:40.181 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:40.181 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:40.181 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:40.181 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:40.181 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:40.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:40.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 4 set_mon_vals no callback set 2024-09-16T14:44:40.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 10 monclient: shutdown 2024-09-16T14:44:40.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:40.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 1 -- 172.21.15.189:0/2079243703 >> v1:172.21.15.26:6790/0 conn(0x7f6e1c13b430 legacy=0x7f6e1c13d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:40.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 1 -- 172.21.15.189:0/2079243703 shutdown_connections 2024-09-16T14:44:40.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 1 -- 172.21.15.189:0/2079243703 >> 172.21.15.189:0/2079243703 conn(0x7f6e1c0a7dd0 msgr2=0x7f6e1c0a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:40.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 1 -- 172.21.15.189:0/2079243703 shutdown_connections 2024-09-16T14:44:40.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 1 -- 172.21.15.189:0/2079243703 wait complete. 2024-09-16T14:44:40.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 10 monclient: build_initial_monmap 2024-09-16T14:44:40.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 10 monclient: monmap: 2024-09-16T14:44:40.182 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:40.182 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:40.182 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:40.182 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:40.182 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:40.183 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:40.183 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:40.183 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:40.183 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:40.183 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:40.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 1 Processor -- start 2024-09-16T14:44:40.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 1 -- start start 2024-09-16T14:44:40.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 10 monclient: init 2024-09-16T14:44:40.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:40.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:40.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e1c0a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:40.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e1c0a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:40.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e1c13b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:40.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c11ff00 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c09e2f0 con 0x7f6e1c13b430 2024-09-16T14:44:40.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c11db70 con 0x7f6e1c0a1b00 2024-09-16T14:44:40.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:40.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e2096d640 10 monclient(hunting): authenticate will time out at 1062.290311s 2024-09-16T14:44:40.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e1ad9d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f6e1c13b430 legacy=0x7f6e1c1d1600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:40.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e19d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6e1c0a25f0 0x7f6e1c13a030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:32934/0 (socket says 172.21.15.189:32934) 2024-09-16T14:44:40.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e1a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6e1c0a1b00 0x7f6e1c139920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:39442/0 (socket says 172.21.15.189:39442) 2024-09-16T14:44:40.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e19d9b640 1 -- 172.21.15.189:0/3850930274 learned_addr learned my addr 172.21.15.189:0/3850930274 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:40.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933397089 0 0) 0x7f6e1c11ff00 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:40.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:40.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:40.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient(hunting): my global_id is 84264 2024-09-16T14:44:40.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dfc003530 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408102425 0 0) 0x7f6e1c11db70 con 0x7f6e1c0a1b00 2024-09-16T14:44:40.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:40.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:40.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:40.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient(hunting): my global_id is 4877 2024-09-16T14:44:40.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e1c11ff00 con 0x7f6e1c0a1b00 2024-09-16T14:44:40.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1236015910 0 0) 0x7f6dfc003530 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e1c11db70 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e0c004160 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:40.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:40.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient(hunting): dump: 2024-09-16T14:44:40.191 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:40.191 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:40.191 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:40.191 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:40.191 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:40.191 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:40.191 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:40.191 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:40.192 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3132931311 0 0) 0x7f6e1c11ff00 con 0x7f6e1c0a1b00 2024-09-16T14:44:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dfc003530 con 0x7f6e1c0a1b00 2024-09-16T14:44:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e100018c0 con 0x7f6e1c0a1b00 2024-09-16T14:44:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient(hunting): dump: 2024-09-16T14:44:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:40.193 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:40.193 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:40.193 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:40.193 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:40.193 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:40.193 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:40.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4066518558 0 0) 0x7f6e1c11db70 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 >> v1:172.21.15.26:6790/0 conn(0x7f6e1c0a1b00 legacy=0x7f6e1c139920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:40.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 >> v1:172.21.15.189:6789/0 conn(0x7f6e1c13b430 legacy=0x7f6e1c1d1600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:40.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:40.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 1 monclient: found mon.a 2024-09-16T14:44:40.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:40.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:40.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.173+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c0a5af0 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e0affd640 10 monclient: _finish_auth 0 2024-09-16T14:44:40.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e0affd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:40.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:40.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e2096d640 5 monclient: authenticate success, global_id 84264 2024-09-16T14:44:40.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e2096d640 10 monclient: _renew_subs 2024-09-16T14:44:40.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e2096d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:40.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e2096d640 1 -- 172.21.15.189:0/3850930274 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1c13af40 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e2096d640 10 monclient: _renew_subs 2024-09-16T14:44:40.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e2096d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:40.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e2096d640 1 -- 172.21.15.189:0/3850930274 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1c1357c0 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e0c004440 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e0affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:40.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e0c0047d0 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:40.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:40.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e18d99640 4 set_mon_vals no callback set 2024-09-16T14:44:40.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e0affd640 10 monclient: dump: 2024-09-16T14:44:40.196 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:40.196 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:40.196 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:40.196 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:40.196 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:40.196 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:40.197 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:40.197 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:40.197 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:40.197 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:40.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e2096d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:40.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e2096d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:40.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e2096d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:40.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e2096d640 1 -- 172.21.15.189:0/3850930274 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e00005050 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6e0c0049b0 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(42..42 src has 15..42) v4 ==== 3716+0+0 (unknown 3338984961 0 0) 0x7f6e0c043200 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.177+0000 7f6e1a59c640 10 monclient: get_auth_request con 0x7f6dfc036250 auth_method 0 2024-09-16T14:44:40.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.189+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6e0c0144e0 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.189+0000 7f6e0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:40.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.189+0000 7f6e0affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:40.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.349+0000 7f6e2096d640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T14:44:40.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.349+0000 7f6e2096d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T14:44:40.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.349+0000 7f6e2096d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:40.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.349+0000 7f6e2096d640 1 -- 172.21.15.189:0/3850930274 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["all"]} v 0) v1 -- 0x7f6e00005300 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.949+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["all"]}]=0 done removing class of osd(s): 0,1,2,3,4,5 v43) v1 ==== 131+0+0 (unknown 1906030773 0 0) 0x7f6e0c014300 con 0x7f6e1c0a25f0 2024-09-16T14:44:40.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.949+0000 7f6e0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T14:44:40.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:40.949+0000 7f6e0affd640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0,1,2,3,4,5 2024-09-16T14:44:41.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:41.045+0000 7f6e2096d640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T14:44:41.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:41.045+0000 7f6e2096d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T14:44:41.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:41.045+0000 7f6e2096d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:41.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:41.045+0000 7f6e2096d640 1 -- 172.21.15.189:0/3850930274 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["all"]} v 0) v1 -- 0x7f6e00005a40 con 0x7f6e1c0a25f0 2024-09-16T14:44:41.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:41.953+0000 7f6e0affd640 1 -- 172.21.15.189:0/3850930274 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["all"]}]=0 osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): v44) v1 ==== 282+0+0 (unknown 1065680302 0 0) 0x7f6e0c015040 con 0x7f6e1c0a25f0 2024-09-16T14:44:41.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:41.953+0000 7f6e0affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T14:44:41.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:41.953+0000 7f6e0affd640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2024-09-16T14:44:41.955 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2024-09-16T14:44:41.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:41.953+0000 7f6e2096d640 1 -- 172.21.15.189:0/3850930274 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6dfc036250 legacy=0x7f6dfc038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:41.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:41.953+0000 7f6e2096d640 10 monclient: shutdown 2024-09-16T14:44:41.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:41.953+0000 7f6e2096d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:41.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:41.953+0000 7f6e2096d640 1 -- 172.21.15.189:0/3850930274 >> v1:172.21.15.26:6789/0 conn(0x7f6e1c0a25f0 legacy=0x7f6e1c13a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:41.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:41.953+0000 7f6e2096d640 1 -- 172.21.15.189:0/3850930274 shutdown_connections 2024-09-16T14:44:41.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:41.953+0000 7f6e2096d640 1 -- 172.21.15.189:0/3850930274 >> 172.21.15.189:0/3850930274 conn(0x7f6e1c0a7dd0 msgr2=0x7f6e1c06b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:41.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:41.953+0000 7f6e2096d640 1 -- 172.21.15.189:0/3850930274 shutdown_connections 2024-09-16T14:44:41.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:41.953+0000 7f6e2096d640 1 -- 172.21.15.189:0/3850930274 wait complete. 2024-09-16T14:44:41.977 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush set-device-class ssd osd.0 2024-09-16T14:44:42.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 10 monclient: get_monmap_and_config 2024-09-16T14:44:42.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 10 monclient: build_initial_monmap 2024-09-16T14:44:42.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 10 monclient: monmap: 2024-09-16T14:44:42.094 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:42.094 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:42.094 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:42.093958+0000 2024-09-16T14:44:42.094 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:42.093958+0000 2024-09-16T14:44:42.094 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:42.095 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:42.095 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:42.095 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:42.095 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:42.095 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:42.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 1 Processor -- start 2024-09-16T14:44:42.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 1 -- start start 2024-09-16T14:44:42.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 10 monclient: init 2024-09-16T14:44:42.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:42.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:42.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 10 monclient(hunting): picked mon.noname-a con 0x7feff009e980 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:42.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 10 monclient(hunting): picked mon.noname-b con 0x7feff0137b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:42.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 10 monclient(hunting): picked mon.noname-c con 0x7feff013b670 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff009d4d0 con 0x7feff009e980 2024-09-16T14:44:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff009c810 con 0x7feff013b670 2024-09-16T14:44:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff009f470 con 0x7feff0137b60 2024-09-16T14:44:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 10 monclient(hunting): authenticate will time out at 1064.209594s 2024-09-16T14:44:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefeed9d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7feff013b670 legacy=0x7feff013da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefedd9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7feff0137b60 0x7feff0139f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:39452/0 (socket says 172.21.15.189:39452) 2024-09-16T14:44:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefedd9b640 1 -- 172.21.15.189:0/3192354299 learned_addr learned my addr 172.21.15.189:0/3192354299 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefee59c640 1 --1- 172.21.15.189:0/3192354299 >> v1:172.21.15.26:6789/0 conn(0x7feff009e980 0x7feff009ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:32942/0 (socket says 172.21.15.189:32942) 2024-09-16T14:44:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 1 -- 172.21.15.189:0/3192354299 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121739289 0 0) 0x7feff009d4d0 con 0x7feff009e980 2024-09-16T14:44:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient(hunting): _negotiate 2024-09-16T14:44:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:42.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient(hunting): my global_id is 84267 2024-09-16T14:44:42.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 1 -- 172.21.15.189:0/3192354299 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefdc003530 con 0x7feff009e980 2024-09-16T14:44:42.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 1 -- 172.21.15.189:0/3192354299 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428197820 0 0) 0x7feff009f470 con 0x7feff0137b60 2024-09-16T14:44:42.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient(hunting): _negotiate 2024-09-16T14:44:42.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:42.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:42.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient(hunting): my global_id is 4880 2024-09-16T14:44:42.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 1 -- 172.21.15.189:0/3192354299 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff009d4d0 con 0x7feff0137b60 2024-09-16T14:44:42.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 1 -- 172.21.15.189:0/3192354299 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3115936342 0 0) 0x7fefdc003530 con 0x7feff009e980 2024-09-16T14:44:42.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 1 -- 172.21.15.189:0/3192354299 >> v1:172.21.15.26:6790/0 conn(0x7feff0137b60 legacy=0x7feff0139f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:42.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 1 -- 172.21.15.189:0/3192354299 >> v1:172.21.15.189:6789/0 conn(0x7feff013b670 legacy=0x7feff013da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:42.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient: _finish_hunting 0 2024-09-16T14:44:42.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 1 monclient: found mon.noname-a 2024-09-16T14:44:42.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:42.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:42.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 1 -- 172.21.15.189:0/3192354299 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff006cfd0 con 0x7feff009e980 2024-09-16T14:44:42.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient: _finish_auth 0 2024-09-16T14:44:42.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient: _check_auth_tickets 2024-09-16T14:44:42.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:42.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 1 -- 172.21.15.189:0/3192354299 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fefe0001620 con 0x7feff009e980 2024-09-16T14:44:42.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 5 monclient: authenticate success, global_id 84267 2024-09-16T14:44:42.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:42.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:42.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:42.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient: dump: 2024-09-16T14:44:42.107 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:42.107 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:42.107 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:42.107 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:42.107 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:42.107 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:42.107 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:42.107 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:42.107 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:42.107 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:42.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:42.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 1 -- 172.21.15.189:0/3192354299 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefe0001900 con 0x7feff009e980 2024-09-16T14:44:42.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:42.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:42.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 4 set_mon_vals no callback set 2024-09-16T14:44:42.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 1 -- 172.21.15.189:0/3192354299 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fefe0001c90 con 0x7feff009e980 2024-09-16T14:44:42.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:42.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:42.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7fefed59a640 10 monclient: dump: 2024-09-16T14:44:42.108 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:42.108 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:42.108 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:42.109 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:42.109 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:42.109 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:42.109 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:42.109 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:42.109 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:42.109 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:42.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 10 monclient: shutdown 2024-09-16T14:44:42.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:42.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 1 -- 172.21.15.189:0/3192354299 >> v1:172.21.15.26:6789/0 conn(0x7feff009e980 legacy=0x7feff009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:42.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 1 -- 172.21.15.189:0/3192354299 shutdown_connections 2024-09-16T14:44:42.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 1 -- 172.21.15.189:0/3192354299 >> 172.21.15.189:0/3192354299 conn(0x7feff00a1a90 msgr2=0x7feff00a1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:42.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 1 -- 172.21.15.189:0/3192354299 shutdown_connections 2024-09-16T14:44:42.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.093+0000 7feff491a640 1 -- 172.21.15.189:0/3192354299 wait complete. 2024-09-16T14:44:42.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 10 monclient: build_initial_monmap 2024-09-16T14:44:42.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 10 monclient: monmap: 2024-09-16T14:44:42.110 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:42.110 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:42.110 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:42.110 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:42.110 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:42.111 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:42.111 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:42.111 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:42.111 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:42.111 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:42.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 1 Processor -- start 2024-09-16T14:44:42.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 1 -- start start 2024-09-16T14:44:42.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 10 monclient: init 2024-09-16T14:44:42.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:42.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:42.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 10 monclient(hunting): picked mon.noname-a con 0x7feff0137b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:42.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 10 monclient(hunting): picked mon.noname-b con 0x7feff013b670 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:42.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 10 monclient(hunting): picked mon.noname-c con 0x7feff01cf830 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:42.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff0070ae0 con 0x7feff0137b60 2024-09-16T14:44:42.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff0138d60 con 0x7feff013b670 2024-09-16T14:44:42.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff0139ed0 con 0x7feff01cf830 2024-09-16T14:44:42.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:42.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefedd9b640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7feff013b670 legacy=0x7feff0138650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:42.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 10 monclient(hunting): authenticate will time out at 1064.213435s 2024-09-16T14:44:42.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefee59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7feff0137b60 0x7feff0137f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:32952/0 (socket says 172.21.15.189:32952) 2024-09-16T14:44:42.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefeed9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7feff01cf830 0x7feff01d1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:39466/0 (socket says 172.21.15.189:39466) 2024-09-16T14:44:42.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefee59c640 1 -- 172.21.15.189:0/2936377203 learned_addr learned my addr 172.21.15.189:0/2936377203 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:42.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228719924 0 0) 0x7feff0070ae0 con 0x7feff0137b60 2024-09-16T14:44:42.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:42.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:42.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:42.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient(hunting): my global_id is 84270 2024-09-16T14:44:42.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd0003560 con 0x7feff0137b60 2024-09-16T14:44:42.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364176905 0 0) 0x7feff0139ed0 con 0x7feff01cf830 2024-09-16T14:44:42.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:42.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:42.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:42.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient(hunting): my global_id is 4883 2024-09-16T14:44:42.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff0070ae0 con 0x7feff01cf830 2024-09-16T14:44:42.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2977387208 0 0) 0x7fefd0003560 con 0x7feff0137b60 2024-09-16T14:44:42.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feff0139ed0 con 0x7feff0137b60 2024-09-16T14:44:42.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fefe00015e0 con 0x7feff0137b60 2024-09-16T14:44:42.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:42.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:42.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient(hunting): dump: 2024-09-16T14:44:42.114 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:42.114 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:42.114 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:42.114 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:42.114 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:42.114 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:42.114 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:42.114 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:42.115 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:42.115 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:42.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:42.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4214119292 0 0) 0x7feff0070ae0 con 0x7feff01cf830 2024-09-16T14:44:42.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefd0003560 con 0x7feff01cf830 2024-09-16T14:44:42.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fefe8004300 con 0x7feff01cf830 2024-09-16T14:44:42.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:42.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:42.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient(hunting): dump: 2024-09-16T14:44:42.115 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:42.115 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:42.115 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:42.116 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:42.116 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:42.116 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:42.116 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:42.116 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:42.116 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:42.116 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:42.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 838801232 0 0) 0x7feff0139ed0 con 0x7feff0137b60 2024-09-16T14:44:42.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 >> v1:172.21.15.26:6790/0 conn(0x7feff01cf830 legacy=0x7feff01d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:42.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 >> v1:172.21.15.189:6789/0 conn(0x7feff013b670 legacy=0x7feff0138650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:42.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:42.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 monclient: found mon.a 2024-09-16T14:44:42.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:42.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:42.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff013a1b0 con 0x7feff0137b60 2024-09-16T14:44:42.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:44:42.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:42.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:42.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 5 monclient: authenticate success, global_id 84270 2024-09-16T14:44:42.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 10 monclient: _renew_subs 2024-09-16T14:44:42.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:42.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 1 -- 172.21.15.189:0/2936377203 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff013a490 con 0x7feff0137b60 2024-09-16T14:44:42.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 10 monclient: _renew_subs 2024-09-16T14:44:42.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:42.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7feff491a640 1 -- 172.21.15.189:0/2936377203 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feff00713c0 con 0x7feff0137b60 2024-09-16T14:44:42.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefe0004b40 con 0x7feff0137b60 2024-09-16T14:44:42.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:42.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefecd99640 4 set_mon_vals no callback set 2024-09-16T14:44:42.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fefe0004e50 con 0x7feff0137b60 2024-09-16T14:44:42.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:42.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:42.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.097+0000 7fefe6ffd640 10 monclient: dump: 2024-09-16T14:44:42.118 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:42.118 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:42.119 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:42.119 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:42.119 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:42.119 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:42.119 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:42.119 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:42.119 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:42.119 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:42.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.101+0000 7feff491a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:42.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.101+0000 7feff491a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:42.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.101+0000 7feff491a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:42.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.101+0000 7feff491a640 1 -- 172.21.15.189:0/2936377203 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd4005050 con 0x7feff0137b60 2024-09-16T14:44:42.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.101+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fefe000f8c0 con 0x7feff0137b60 2024-09-16T14:44:42.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.101+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(44..44 src has 15..44) v4 ==== 3412+0+0 (unknown 2492621583 0 0) 0x7fefe00431e0 con 0x7feff0137b60 2024-09-16T14:44:42.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.101+0000 7fefedd9b640 10 monclient: get_auth_request con 0x7fefd00362f0 auth_method 0 2024-09-16T14:44:42.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.113+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fefe00146b0 con 0x7feff0137b60 2024-09-16T14:44:42.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.113+0000 7fefe6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:42.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.113+0000 7fefe6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:42.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.277+0000 7feff491a640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T14:44:42.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.277+0000 7feff491a640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T14:44:42.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.277+0000 7feff491a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:42.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.277+0000 7feff491a640 1 -- 172.21.15.189:0/2936377203 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]} v 0) v1 -- 0x7fefd4005300 con 0x7feff0137b60 2024-09-16T14:44:42.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.965+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'ssd' v45) v1 ==== 135+0+0 (unknown 208943553 0 0) 0x7fefe0012d50 con 0x7feff0137b60 2024-09-16T14:44:42.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.965+0000 7fefe6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T14:44:42.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:42.965+0000 7fefe6ffd640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'ssd' 2024-09-16T14:44:43.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:43.065+0000 7feff491a640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T14:44:43.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:43.065+0000 7feff491a640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T14:44:43.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:43.065+0000 7feff491a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:43.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:43.065+0000 7feff491a640 1 -- 172.21.15.189:0/2936377203 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]} v 0) v1 -- 0x7fefd4001170 con 0x7feff0137b60 2024-09-16T14:44:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:43.969+0000 7fefe6ffd640 1 -- 172.21.15.189:0/2936377203 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}]=0 osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' v46) v1 ==== 237+0+0 (unknown 3168923248 0 0) 0x7fefe00144d0 con 0x7feff0137b60 2024-09-16T14:44:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:43.969+0000 7fefe6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T14:44:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:43.969+0000 7fefe6ffd640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-16T14:44:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-16T14:44:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:43.969+0000 7feff491a640 1 -- 172.21.15.189:0/2936377203 >> v1:172.21.15.189:6812/3196161582 conn(0x7fefd00362f0 legacy=0x7fefd00387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:43.969+0000 7feff491a640 10 monclient: shutdown 2024-09-16T14:44:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:43.969+0000 7feff491a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:43.969+0000 7feff491a640 1 -- 172.21.15.189:0/2936377203 >> v1:172.21.15.26:6789/0 conn(0x7feff0137b60 legacy=0x7feff0137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:43.969+0000 7feff491a640 1 -- 172.21.15.189:0/2936377203 shutdown_connections 2024-09-16T14:44:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:43.969+0000 7feff491a640 1 -- 172.21.15.189:0/2936377203 >> 172.21.15.189:0/2936377203 conn(0x7feff00a1a90 msgr2=0x7feff006b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:43.973+0000 7feff491a640 1 -- 172.21.15.189:0/2936377203 shutdown_connections 2024-09-16T14:44:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:43.973+0000 7feff491a640 1 -- 172.21.15.189:0/2936377203 wait complete. 2024-09-16T14:44:43.993 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush set-device-class hdd osd.1 2024-09-16T14:44:44.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.113+0000 7eff08951640 10 monclient: get_monmap_and_config 2024-09-16T14:44:44.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.113+0000 7eff08951640 10 monclient: build_initial_monmap 2024-09-16T14:44:44.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.113+0000 7eff08951640 10 monclient: monmap: 2024-09-16T14:44:44.117 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:44.117 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:44.117 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:44.116826+0000 2024-09-16T14:44:44.117 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:44.116826+0000 2024-09-16T14:44:44.117 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:44.117 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:44.117 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:44.117 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:44.118 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:44.118 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:44.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.113+0000 7eff08951640 1 Processor -- start 2024-09-16T14:44:44.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 1 -- start start 2024-09-16T14:44:44.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 10 monclient: init 2024-09-16T14:44:44.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:44.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:44.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 10 monclient(hunting): picked mon.noname-c con 0x7eff040a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:44.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 10 monclient(hunting): picked mon.noname-a con 0x7eff040a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:44.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 10 monclient(hunting): picked mon.noname-b con 0x7eff0413b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:44.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff040a30b0 con 0x7eff040a25f0 2024-09-16T14:44:44.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff040a2e30 con 0x7eff040a1b00 2024-09-16T14:44:44.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0409eb50 con 0x7eff0413b430 2024-09-16T14:44:44.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff0259c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7eff040a1b00 legacy=0x7eff040a1ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:44.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:44.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 10 monclient(hunting): authenticate will time out at 1066.232818s 2024-09-16T14:44:44.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7efef9d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7eff040a25f0 0x7eff04139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:32954/0 (socket says 172.21.15.189:32954) 2024-09-16T14:44:44.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7efef9d9b640 1 -- 172.21.15.189:0/1651071765 learned_addr learned my addr 172.21.15.189:0/1651071765 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:44.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff02d9d640 1 --1- 172.21.15.189:0/1651071765 >> v1:172.21.15.26:6790/0 conn(0x7eff0413b430 0x7eff0413d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:39482/0 (socket says 172.21.15.189:39482) 2024-09-16T14:44:44.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 1 -- 172.21.15.189:0/1651071765 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285538984 0 0) 0x7eff040a30b0 con 0x7eff040a25f0 2024-09-16T14:44:44.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:44:44.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:44.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:44.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient(hunting): my global_id is 84273 2024-09-16T14:44:44.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 1 -- 172.21.15.189:0/1651071765 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efef0003530 con 0x7eff040a25f0 2024-09-16T14:44:44.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 1 -- 172.21.15.189:0/1651071765 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168203165 0 0) 0x7eff0409eb50 con 0x7eff0413b430 2024-09-16T14:44:44.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:44:44.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:44.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:44.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient(hunting): my global_id is 4886 2024-09-16T14:44:44.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 1 -- 172.21.15.189:0/1651071765 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff040a30b0 con 0x7eff0413b430 2024-09-16T14:44:44.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 1 -- 172.21.15.189:0/1651071765 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2782733873 0 0) 0x7efef0003530 con 0x7eff040a25f0 2024-09-16T14:44:44.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 1 -- 172.21.15.189:0/1651071765 >> v1:172.21.15.26:6790/0 conn(0x7eff0413b430 legacy=0x7eff0413d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:44.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 1 -- 172.21.15.189:0/1651071765 >> v1:172.21.15.189:6789/0 conn(0x7eff040a1b00 legacy=0x7eff040a1ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:44.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:44:44.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 1 monclient: found mon.noname-a 2024-09-16T14:44:44.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:44.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:44.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 1 -- 172.21.15.189:0/1651071765 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0409ee30 con 0x7eff040a25f0 2024-09-16T14:44:44.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient: _finish_auth 0 2024-09-16T14:44:44.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:44:44.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:44.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 1 -- 172.21.15.189:0/1651071765 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efeec001620 con 0x7eff040a25f0 2024-09-16T14:44:44.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 5 monclient: authenticate success, global_id 84273 2024-09-16T14:44:44.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:44.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:44.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:44.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient: dump: 2024-09-16T14:44:44.124 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:44.124 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:44.125 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:44.125 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:44.125 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:44.125 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:44.125 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:44.125 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:44.125 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:44.125 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:44.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:44.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 1 -- 172.21.15.189:0/1651071765 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efeec001840 con 0x7eff040a25f0 2024-09-16T14:44:44.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:44.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:44.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 1 -- 172.21.15.189:0/1651071765 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efeec001bd0 con 0x7eff040a25f0 2024-09-16T14:44:44.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:44.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 4 set_mon_vals no callback set 2024-09-16T14:44:44.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:44.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff01d9b640 10 monclient: dump: 2024-09-16T14:44:44.126 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:44.126 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:44.126 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:44.126 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:44.126 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:44.126 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:44.126 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:44.126 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:44.126 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:44.126 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:44.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 10 monclient: shutdown 2024-09-16T14:44:44.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:44.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 1 -- 172.21.15.189:0/1651071765 >> v1:172.21.15.26:6789/0 conn(0x7eff040a25f0 legacy=0x7eff04139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:44.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 1 -- 172.21.15.189:0/1651071765 shutdown_connections 2024-09-16T14:44:44.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 1 -- 172.21.15.189:0/1651071765 >> 172.21.15.189:0/1651071765 conn(0x7eff040a7dd0 msgr2=0x7eff040a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:44.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 1 -- 172.21.15.189:0/1651071765 shutdown_connections 2024-09-16T14:44:44.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 1 -- 172.21.15.189:0/1651071765 wait complete. 2024-09-16T14:44:44.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 10 monclient: build_initial_monmap 2024-09-16T14:44:44.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.117+0000 7eff08951640 10 monclient: monmap: 2024-09-16T14:44:44.131 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:44.131 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:44.131 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:44.131 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:44.131 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:44.131 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:44.131 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:44.131 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:44.132 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:44.132 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:44.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 1 Processor -- start 2024-09-16T14:44:44.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 1 -- start start 2024-09-16T14:44:44.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 10 monclient: init 2024-09-16T14:44:44.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:44.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:44.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 10 monclient(hunting): picked mon.noname-b con 0x7eff040a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:44.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 10 monclient(hunting): picked mon.noname-a con 0x7eff040a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:44.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 10 monclient(hunting): picked mon.noname-c con 0x7eff0413b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:44.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0413af40 con 0x7eff040a25f0 2024-09-16T14:44:44.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff041ca3d0 con 0x7eff040a1b00 2024-09-16T14:44:44.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff0259c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7eff040a1b00 legacy=0x7eff041c95b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:44.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0409e2f0 con 0x7eff0413b430 2024-09-16T14:44:44.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:44.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 10 monclient(hunting): authenticate will time out at 1066.237120s 2024-09-16T14:44:44.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efef9d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7eff040a25f0 0x7eff041c9cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:32956/0 (socket says 172.21.15.189:32956) 2024-09-16T14:44:44.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efef9d9b640 1 -- 172.21.15.189:0/915505040 learned_addr learned my addr 172.21.15.189:0/915505040 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:44.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158097624 0 0) 0x7eff0413af40 con 0x7eff040a25f0 2024-09-16T14:44:44.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:44.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:44.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:44.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient(hunting): my global_id is 84276 2024-09-16T14:44:44.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee4003530 con 0x7eff040a25f0 2024-09-16T14:44:44.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262487456 0 0) 0x7eff0409e2f0 con 0x7eff0413b430 2024-09-16T14:44:44.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:44.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:44.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:44.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient(hunting): my global_id is 4889 2024-09-16T14:44:44.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff0413af40 con 0x7eff0413b430 2024-09-16T14:44:44.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1884785471 0 0) 0x7efee4003530 con 0x7eff040a25f0 2024-09-16T14:44:44.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff0409e2f0 con 0x7eff040a25f0 2024-09-16T14:44:44.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efeec0015e0 con 0x7eff040a25f0 2024-09-16T14:44:44.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:44.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:44.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient(hunting): dump: 2024-09-16T14:44:44.134 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:44.134 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:44.134 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:44.135 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:44.135 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:44.135 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:44.135 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:44.135 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:44.135 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:44.135 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:44.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:44.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4282665907 0 0) 0x7eff0413af40 con 0x7eff0413b430 2024-09-16T14:44:44.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efee4003530 con 0x7eff0413b430 2024-09-16T14:44:44.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efefc001c40 con 0x7eff0413b430 2024-09-16T14:44:44.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:44.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:44.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient(hunting): dump: 2024-09-16T14:44:44.136 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:44.136 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:44.136 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:44.136 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:44.136 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:44.136 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:44.136 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:44.136 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:44.136 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:44.136 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:44.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 447158513 0 0) 0x7eff0409e2f0 con 0x7eff040a25f0 2024-09-16T14:44:44.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 >> v1:172.21.15.26:6790/0 conn(0x7eff0413b430 legacy=0x7eff041d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:44.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 >> v1:172.21.15.189:6789/0 conn(0x7eff040a1b00 legacy=0x7eff041c95b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:44.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:44:44.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 monclient: found mon.a 2024-09-16T14:44:44.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:44.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:44.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff040a5b30 con 0x7eff040a25f0 2024-09-16T14:44:44.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient: _finish_auth 0 2024-09-16T14:44:44.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:44:44.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:44.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 5 monclient: authenticate success, global_id 84276 2024-09-16T14:44:44.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 10 monclient: _renew_subs 2024-09-16T14:44:44.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:44.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 1 -- 172.21.15.189:0/915505040 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0411ff00 con 0x7eff040a25f0 2024-09-16T14:44:44.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 10 monclient: _renew_subs 2024-09-16T14:44:44.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:44.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 1 -- 172.21.15.189:0/915505040 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0411fc20 con 0x7eff040a25f0 2024-09-16T14:44:44.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efeec004aa0 con 0x7eff040a25f0 2024-09-16T14:44:44.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:44.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efeec004e30 con 0x7eff040a25f0 2024-09-16T14:44:44.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:44.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:44.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff0159a640 4 set_mon_vals no callback set 2024-09-16T14:44:44.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7efefb7fe640 10 monclient: dump: 2024-09-16T14:44:44.138 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:44.138 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:44.139 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:44.139 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:44.139 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:44.139 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:44.139 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:44.139 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:44.139 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:44.139 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:44.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:44.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:44.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:44.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.121+0000 7eff08951640 1 -- 172.21.15.189:0/915505040 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee8005050 con 0x7eff040a25f0 2024-09-16T14:44:44.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.125+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7efeec005010 con 0x7eff040a25f0 2024-09-16T14:44:44.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.125+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(46..46 src has 15..46) v4 ==== 3636+0+0 (unknown 1839952763 0 0) 0x7efeec0432c0 con 0x7eff040a25f0 2024-09-16T14:44:44.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.125+0000 7eff0259c640 10 monclient: get_auth_request con 0x7efee4036250 auth_method 0 2024-09-16T14:44:44.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.137+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7efeec0148e0 con 0x7eff040a25f0 2024-09-16T14:44:44.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.137+0000 7efefb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:44.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.137+0000 7efefb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:44.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.297+0000 7eff08951640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T14:44:44.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.297+0000 7eff08951640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T14:44:44.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.297+0000 7eff08951640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:44.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.297+0000 7eff08951640 1 -- 172.21.15.189:0/915505040 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]} v 0) v1 -- 0x7efee8005300 con 0x7eff040a25f0 2024-09-16T14:44:44.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.981+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}]=0 set osd(s) 1 to class 'hdd' v47) v1 ==== 135+0+0 (unknown 684690550 0 0) 0x7efeec0144d0 con 0x7eff040a25f0 2024-09-16T14:44:44.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.981+0000 7efefb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T14:44:44.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:44.981+0000 7efefb7fe640 10 monclient: _finish_command 2 = system:0 set osd(s) 1 to class 'hdd' 2024-09-16T14:44:45.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:45.081+0000 7eff08951640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T14:44:45.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:45.081+0000 7eff08951640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T14:44:45.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:45.081+0000 7eff08951640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:45.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:45.081+0000 7eff08951640 1 -- 172.21.15.189:0/915505040 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]} v 0) v1 -- 0x7efee8001170 con 0x7eff040a25f0 2024-09-16T14:44:45.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:45.985+0000 7efefb7fe640 1 -- 172.21.15.189:0/915505040 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}]=0 osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' v48) v1 ==== 237+0+0 (unknown 1801071739 0 0) 0x7efeec0146b0 con 0x7eff040a25f0 2024-09-16T14:44:45.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:45.985+0000 7efefb7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T14:44:45.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:45.985+0000 7efefb7fe640 10 monclient: _finish_command 3 = system:0 osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-16T14:44:45.987 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-16T14:44:45.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:45.985+0000 7eff08951640 1 -- 172.21.15.189:0/915505040 >> v1:172.21.15.189:6812/3196161582 conn(0x7efee4036250 legacy=0x7efee4038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:45.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:45.985+0000 7eff08951640 10 monclient: shutdown 2024-09-16T14:44:45.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:45.985+0000 7eff08951640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:45.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:45.985+0000 7eff08951640 1 -- 172.21.15.189:0/915505040 >> v1:172.21.15.26:6789/0 conn(0x7eff040a25f0 legacy=0x7eff041c9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:45.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:45.985+0000 7eff08951640 1 -- 172.21.15.189:0/915505040 shutdown_connections 2024-09-16T14:44:45.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:45.985+0000 7eff08951640 1 -- 172.21.15.189:0/915505040 >> 172.21.15.189:0/915505040 conn(0x7eff040a7dd0 msgr2=0x7eff0409d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:45.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:45.985+0000 7eff08951640 1 -- 172.21.15.189:0/915505040 shutdown_connections 2024-09-16T14:44:45.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:45.985+0000 7eff08951640 1 -- 172.21.15.189:0/915505040 wait complete. 2024-09-16T14:44:46.009 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule create-replicated foo-ssd default host ssd 2024-09-16T14:44:46.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd2a29640 10 monclient: get_monmap_and_config 2024-09-16T14:44:46.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd2a29640 10 monclient: build_initial_monmap 2024-09-16T14:44:46.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd2a29640 10 monclient: monmap: 2024-09-16T14:44:46.136 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:46.136 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:46.136 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:46.136133+0000 2024-09-16T14:44:46.137 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:46.136133+0000 2024-09-16T14:44:46.137 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:46.137 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:46.137 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:46.137 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:46.137 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:46.137 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:46.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd2a29640 1 Processor -- start 2024-09-16T14:44:46.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd2a29640 1 -- start start 2024-09-16T14:44:46.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd2a29640 10 monclient: init 2024-09-16T14:44:46.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd2a29640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:46.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd2a29640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:46.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd2a29640 10 monclient(hunting): picked mon.noname-a con 0x7f9fcc0a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd2a29640 10 monclient(hunting): picked mon.noname-b con 0x7f9fcc0a25f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd2a29640 10 monclient(hunting): picked mon.noname-c con 0x7f9fcc13b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd2a29640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fcc0a30b0 con 0x7f9fcc0a1b00 2024-09-16T14:44:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd2a29640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fcc0a2e30 con 0x7f9fcc13b430 2024-09-16T14:44:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd2a29640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fcc09eb50 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd0fc6640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f9fcc13b430 legacy=0x7f9fcc13d820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd2a29640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.133+0000 7f9fd2a29640 10 monclient(hunting): authenticate will time out at 1068.251975s 2024-09-16T14:44:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcb7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9fcc0a25f0 0x7f9fcc139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:39510/0 (socket says 172.21.15.189:39510) 2024-09-16T14:44:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcb7fe640 1 -- 172.21.15.189:0/3286910058 learned_addr learned my addr 172.21.15.189:0/3286910058 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 1 -- 172.21.15.189:0/3286910058 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389949732 0 0) 0x7f9fcc09eb50 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient(hunting): my global_id is 4892 2024-09-16T14:44:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 1 -- 172.21.15.189:0/3286910058 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fb4003530 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 1 -- 172.21.15.189:0/3286910058 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408024040 0 0) 0x7f9fcc0a30b0 con 0x7f9fcc0a1b00 2024-09-16T14:44:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient(hunting): my global_id is 84279 2024-09-16T14:44:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 1 -- 172.21.15.189:0/3286910058 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fcc09eb50 con 0x7f9fcc0a1b00 2024-09-16T14:44:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 1 -- 172.21.15.189:0/3286910058 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 181090325 0 0) 0x7f9fb4003530 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 1 -- 172.21.15.189:0/3286910058 >> v1:172.21.15.189:6789/0 conn(0x7f9fcc13b430 legacy=0x7f9fcc13d820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 1 -- 172.21.15.189:0/3286910058 >> v1:172.21.15.26:6789/0 conn(0x7f9fcc0a1b00 legacy=0x7f9fcc0a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:46.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 1 monclient: found mon.noname-b 2024-09-16T14:44:46.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:46.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:46.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 1 -- 172.21.15.189:0/3286910058 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fcc09ee30 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient: _finish_auth 0 2024-09-16T14:44:46.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:46.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:46.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 1 -- 172.21.15.189:0/3286910058 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9fc0001620 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:46.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:46.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient: dump: 2024-09-16T14:44:46.143 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:46.143 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:46.144 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:46.144 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:46.144 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:46.144 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:46.144 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:46.144 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:46.144 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:46.144 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:46.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 5 monclient: authenticate success, global_id 4892 2024-09-16T14:44:46.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:46.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 1 -- 172.21.15.189:0/3286910058 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fc0001840 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:46.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:46.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 4 set_mon_vals no callback set 2024-09-16T14:44:46.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 1 -- 172.21.15.189:0/3286910058 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9fc0001bd0 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:46.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:46.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fcaffd640 10 monclient: dump: 2024-09-16T14:44:46.145 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:46.145 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:46.145 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:46.145 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:46.145 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:46.145 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:46.145 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:46.145 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:46.145 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:46.146 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:46.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 10 monclient: shutdown 2024-09-16T14:44:46.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:46.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 1 -- 172.21.15.189:0/3286910058 >> v1:172.21.15.26:6790/0 conn(0x7f9fcc0a25f0 legacy=0x7f9fcc139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:46.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 1 -- 172.21.15.189:0/3286910058 shutdown_connections 2024-09-16T14:44:46.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 1 -- 172.21.15.189:0/3286910058 >> 172.21.15.189:0/3286910058 conn(0x7f9fcc0a7dd0 msgr2=0x7f9fcc0a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:46.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 1 -- 172.21.15.189:0/3286910058 shutdown_connections 2024-09-16T14:44:46.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 1 -- 172.21.15.189:0/3286910058 wait complete. 2024-09-16T14:44:46.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 10 monclient: build_initial_monmap 2024-09-16T14:44:46.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 10 monclient: monmap: 2024-09-16T14:44:46.146 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:46.146 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:46.146 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:46.147 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:46.147 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:46.147 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:46.147 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:46.147 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:46.147 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:46.147 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:46.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 1 Processor -- start 2024-09-16T14:44:46.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 1 -- start start 2024-09-16T14:44:46.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 10 monclient: init 2024-09-16T14:44:46.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:46.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:46.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 10 monclient(hunting): picked mon.noname-c con 0x7f9fcc0a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:46.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 10 monclient(hunting): picked mon.noname-a con 0x7f9fcc0a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:46.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 10 monclient(hunting): picked mon.noname-b con 0x7f9fcc13b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:46.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fcc11ff80 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fcc0a5c10 con 0x7f9fcc13b430 2024-09-16T14:44:46.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd0fc6640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f9fcc13b430 legacy=0x7f9fcc1d1600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:46.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fcc13af40 con 0x7f9fcc0a1b00 2024-09-16T14:44:46.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:46.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.137+0000 7f9fd2a29640 10 monclient(hunting): authenticate will time out at 1068.255971s 2024-09-16T14:44:46.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fcb7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9fcc0a25f0 0x7f9fcc13a1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:32982/0 (socket says 172.21.15.189:32982) 2024-09-16T14:44:46.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fcb7fe640 1 -- 172.21.15.189:0/668941580 learned_addr learned my addr 172.21.15.189:0/668941580 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:46.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728955901 0 0) 0x7f9fcc13af40 con 0x7f9fcc0a1b00 2024-09-16T14:44:46.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:46.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:46.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:46.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient(hunting): my global_id is 4895 2024-09-16T14:44:46.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fb8003530 con 0x7f9fcc0a1b00 2024-09-16T14:44:46.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197213941 0 0) 0x7f9fcc11ff80 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:46.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:46.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:46.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient(hunting): my global_id is 84282 2024-09-16T14:44:46.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fcc13af40 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 836358182 0 0) 0x7f9fcc13af40 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fcc11ff80 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9fc00047f0 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:46.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:46.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient(hunting): dump: 2024-09-16T14:44:46.154 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:46.154 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:46.154 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:46.154 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:46.154 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:46.154 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:46.154 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:46.154 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:46.154 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:46.154 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:46.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:46.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3874904352 0 0) 0x7f9fb8003530 con 0x7f9fcc0a1b00 2024-09-16T14:44:46.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fcc13af40 con 0x7f9fcc0a1b00 2024-09-16T14:44:46.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9fbc001a70 con 0x7f9fcc0a1b00 2024-09-16T14:44:46.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:46.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:46.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient(hunting): dump: 2024-09-16T14:44:46.155 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:46.155 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:46.155 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:46.155 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:46.155 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:46.155 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:46.155 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:46.155 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:46.155 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:46.155 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:46.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2570261913 0 0) 0x7f9fcc11ff80 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 >> v1:172.21.15.26:6790/0 conn(0x7f9fcc0a1b00 legacy=0x7f9fcc139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:46.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 >> v1:172.21.15.189:6789/0 conn(0x7f9fcc13b430 legacy=0x7f9fcc1d1600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:46.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:44:46.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 monclient: found mon.a 2024-09-16T14:44:46.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:46.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:46.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fcc11fca0 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient: _finish_auth 0 2024-09-16T14:44:46.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:44:46.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:46.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fd2a29640 5 monclient: authenticate success, global_id 84282 2024-09-16T14:44:46.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fd2a29640 10 monclient: _renew_subs 2024-09-16T14:44:46.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fd2a29640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:46.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fd2a29640 1 -- 172.21.15.189:0/668941580 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fcc135890 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fd2a29640 10 monclient: _renew_subs 2024-09-16T14:44:46.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fd2a29640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:46.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fd2a29640 1 -- 172.21.15.189:0/668941580 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fcc13a8b0 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fc0004c00 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:46.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9fc0004f90 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:46.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:46.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fca7fc640 4 set_mon_vals no callback set 2024-09-16T14:44:46.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 10 monclient: dump: 2024-09-16T14:44:46.158 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:46.158 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:46.158 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:46.158 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:46.158 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:46.158 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:46.158 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:46.158 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:46.158 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:46.158 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:46.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fd2a29640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:46.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fd2a29640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:46.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fd2a29640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:46.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fd2a29640 1 -- 172.21.15.189:0/668941580 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fac005050 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9fc00051b0 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(48..48 src has 15..48) v4 ==== 3868+0+0 (unknown 1598407493 0 0) 0x7f9fc0043450 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.141+0000 7f9fcbfff640 10 monclient: get_auth_request con 0x7f9fb80361e0 auth_method 0 2024-09-16T14:44:46.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.153+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9fc0014990 con 0x7f9fcc0a25f0 2024-09-16T14:44:46.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.153+0000 7f9fc8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:46.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.153+0000 7f9fc8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:46.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.317+0000 7f9fd2a29640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T14:44:46.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.317+0000 7f9fd2a29640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T14:44:46.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.317+0000 7f9fd2a29640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:46.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.317+0000 7f9fd2a29640 1 -- 172.21.15.189:0/668941580 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"} v 0) v1 -- 0x7f9fac005300 con 0x7f9fcc0a25f0 2024-09-16T14:44:47.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.997+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}]=0 v49) v1 ==== 150+0+0 (unknown 1552502280 0 0) 0x7f9fc0012e00 con 0x7f9fcc0a25f0 2024-09-16T14:44:47.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.997+0000 7f9fc8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T14:44:47.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:46.997+0000 7f9fc8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:47.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.097+0000 7f9fd2a29640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T14:44:47.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.097+0000 7f9fd2a29640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T14:44:47.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.097+0000 7f9fd2a29640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:47.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.097+0000 7f9fd2a29640 1 -- 172.21.15.189:0/668941580 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"} v 0) v1 -- 0x7f9fac0010a0 con 0x7f9fcc0a25f0 2024-09-16T14:44:47.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.101+0000 7f9fc8ff9640 1 -- 172.21.15.189:0/668941580 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}]=0 rule foo-ssd already exists v49) v1 ==== 177+0+0 (unknown 2951410334 0 0) 0x7f9fc0014580 con 0x7f9fcc0a25f0 2024-09-16T14:44:47.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.101+0000 7f9fc8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T14:44:47.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.101+0000 7f9fc8ff9640 10 monclient: _finish_command 3 = system:0 rule foo-ssd already exists 2024-09-16T14:44:47.103 INFO:tasks.workunit.client.0.smithi189.stderr:rule foo-ssd already exists 2024-09-16T14:44:47.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.101+0000 7f9fd2a29640 1 -- 172.21.15.189:0/668941580 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9fb80361e0 legacy=0x7f9fb80386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:47.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.101+0000 7f9fd2a29640 10 monclient: shutdown 2024-09-16T14:44:47.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.101+0000 7f9fd2a29640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:47.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.101+0000 7f9fd2a29640 1 -- 172.21.15.189:0/668941580 >> v1:172.21.15.26:6789/0 conn(0x7f9fcc0a25f0 legacy=0x7f9fcc13a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:47.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.101+0000 7f9fd2a29640 1 -- 172.21.15.189:0/668941580 shutdown_connections 2024-09-16T14:44:47.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.101+0000 7f9fd2a29640 1 -- 172.21.15.189:0/668941580 >> 172.21.15.189:0/668941580 conn(0x7f9fcc0a7dd0 msgr2=0x7f9fcc09cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:47.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.101+0000 7f9fd2a29640 1 -- 172.21.15.189:0/668941580 shutdown_connections 2024-09-16T14:44:47.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.101+0000 7f9fd2a29640 1 -- 172.21.15.189:0/668941580 wait complete. 2024-09-16T14:44:47.120 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule create-replicated foo-hdd default host hdd 2024-09-16T14:44:47.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 10 monclient: get_monmap_and_config 2024-09-16T14:44:47.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 10 monclient: build_initial_monmap 2024-09-16T14:44:47.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 10 monclient: monmap: 2024-09-16T14:44:47.246 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:47.246 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:47.247 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:47.245973+0000 2024-09-16T14:44:47.247 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:47.245973+0000 2024-09-16T14:44:47.247 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:47.247 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:47.247 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:47.247 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:47.247 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:47.247 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:47.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 1 Processor -- start 2024-09-16T14:44:47.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 1 -- start start 2024-09-16T14:44:47.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 10 monclient: init 2024-09-16T14:44:47.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:47.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:47.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 10 monclient(hunting): picked mon.noname-a con 0x7f40f406c120 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:47.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 10 monclient(hunting): picked mon.noname-b con 0x7f40f41353d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:47.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 10 monclient(hunting): picked mon.noname-c con 0x7f40f40a1040 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:47.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f406c500 con 0x7f40f406c120 2024-09-16T14:44:47.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f40a2ef0 con 0x7f40f40a1040 2024-09-16T14:44:47.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f409e990 con 0x7f40f41353d0 2024-09-16T14:44:47.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:47.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 10 monclient(hunting): authenticate will time out at 1069.361613s 2024-09-16T14:44:47.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f969b640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f40f40a1040 legacy=0x7f40f413d230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:47.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f3fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f40f41353d0 0x7f40f4139b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:39538/0 (socket says 172.21.15.189:39538) 2024-09-16T14:44:47.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f3fff640 1 -- 172.21.15.189:0/1557126203 learned_addr learned my addr 172.21.15.189:0/1557126203 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:47.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 1 -- 172.21.15.189:0/1557126203 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030990986 0 0) 0x7f40f409e990 con 0x7f40f41353d0 2024-09-16T14:44:47.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:47.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:47.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:47.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient(hunting): my global_id is 4898 2024-09-16T14:44:47.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 1 -- 172.21.15.189:0/1557126203 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40dc003530 con 0x7f40f41353d0 2024-09-16T14:44:47.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 1 -- 172.21.15.189:0/1557126203 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812974559 0 0) 0x7f40f406c500 con 0x7f40f406c120 2024-09-16T14:44:47.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:47.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:47.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:47.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient(hunting): my global_id is 84285 2024-09-16T14:44:47.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 1 -- 172.21.15.189:0/1557126203 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f409e990 con 0x7f40f406c120 2024-09-16T14:44:47.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 1 -- 172.21.15.189:0/1557126203 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 567101803 0 0) 0x7f40dc003530 con 0x7f40f41353d0 2024-09-16T14:44:47.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 1 -- 172.21.15.189:0/1557126203 >> v1:172.21.15.189:6789/0 conn(0x7f40f40a1040 legacy=0x7f40f413d230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:47.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 1 -- 172.21.15.189:0/1557126203 >> v1:172.21.15.26:6789/0 conn(0x7f40f406c120 legacy=0x7f40f4134cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:47.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:44:47.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 1 monclient: found mon.noname-b 2024-09-16T14:44:47.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:47.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:47.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 1 -- 172.21.15.189:0/1557126203 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f409ec70 con 0x7f40f41353d0 2024-09-16T14:44:47.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient: _finish_auth 0 2024-09-16T14:44:47.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:44:47.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:47.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 1 -- 172.21.15.189:0/1557126203 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f40e80019a0 con 0x7f40f41353d0 2024-09-16T14:44:47.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:47.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:47.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient: dump: 2024-09-16T14:44:47.258 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:47.258 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:47.258 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:47.258 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:47.258 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:47.258 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:47.258 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:47.259 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:47.259 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:47.259 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:47.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 5 monclient: authenticate success, global_id 4898 2024-09-16T14:44:47.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:47.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 1 -- 172.21.15.189:0/1557126203 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40e8001bc0 con 0x7f40f41353d0 2024-09-16T14:44:47.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:47.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 1 -- 172.21.15.189:0/1557126203 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f40e8001f10 con 0x7f40f41353d0 2024-09-16T14:44:47.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:47.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40fb0fe640 4 set_mon_vals no callback set 2024-09-16T14:44:47.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:47.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.245+0000 7f40f37fe640 10 monclient: dump: 2024-09-16T14:44:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:47.260 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 10 monclient: shutdown 2024-09-16T14:44:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 1 -- 172.21.15.189:0/1557126203 >> v1:172.21.15.26:6790/0 conn(0x7f40f41353d0 legacy=0x7f40f4139b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:47.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 1 -- 172.21.15.189:0/1557126203 shutdown_connections 2024-09-16T14:44:47.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 1 -- 172.21.15.189:0/1557126203 >> 172.21.15.189:0/1557126203 conn(0x7f40f40a3a40 msgr2=0x7f40f40a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:47.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 1 -- 172.21.15.189:0/1557126203 shutdown_connections 2024-09-16T14:44:47.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 1 -- 172.21.15.189:0/1557126203 wait complete. 2024-09-16T14:44:47.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 10 monclient: build_initial_monmap 2024-09-16T14:44:47.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 10 monclient: monmap: 2024-09-16T14:44:47.261 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:47.261 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:47.261 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:47.261 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:47.261 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:47.261 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:47.262 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:47.262 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:47.262 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:47.262 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:47.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 1 Processor -- start 2024-09-16T14:44:47.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 1 -- start start 2024-09-16T14:44:47.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 10 monclient: init 2024-09-16T14:44:47.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:47.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:47.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 10 monclient(hunting): picked mon.noname-a con 0x7f40f406c120 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:47.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 10 monclient(hunting): picked mon.noname-c con 0x7f40f40a1040 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:47.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 10 monclient(hunting): picked mon.noname-b con 0x7f40f41353d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:47.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f409e130 con 0x7f40f406c120 2024-09-16T14:44:47.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f4132590 con 0x7f40f41353d0 2024-09-16T14:44:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f412fd60 con 0x7f40f40a1040 2024-09-16T14:44:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 10 monclient(hunting): authenticate will time out at 1069.365670s 2024-09-16T14:44:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f969b640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f40f41353d0 legacy=0x7f40f41d1890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f8e9a640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f40f406c120 0x7f40f412ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33008/0 (socket says 172.21.15.189:33008) 2024-09-16T14:44:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f8e9a640 1 -- 172.21.15.189:0/1728580615 learned_addr learned my addr 172.21.15.189:0/1728580615 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f3fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f40f40a1040 0x7f40f412f650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:39554/0 (socket says 172.21.15.189:39554) 2024-09-16T14:44:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708858862 0 0) 0x7f40f412fd60 con 0x7f40f40a1040 2024-09-16T14:44:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:44:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient(hunting): my global_id is 4901 2024-09-16T14:44:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e0003530 con 0x7f40f40a1040 2024-09-16T14:44:47.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422753713 0 0) 0x7f40f409e130 con 0x7f40f406c120 2024-09-16T14:44:47.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:44:47.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:47.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:47.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient(hunting): my global_id is 84288 2024-09-16T14:44:47.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f412fd60 con 0x7f40f406c120 2024-09-16T14:44:47.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3206702323 0 0) 0x7f40e0003530 con 0x7f40f40a1040 2024-09-16T14:44:47.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40f409e130 con 0x7f40f40a1040 2024-09-16T14:44:47.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f40e80018a0 con 0x7f40f40a1040 2024-09-16T14:44:47.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:47.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:47.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient(hunting): dump: 2024-09-16T14:44:47.264 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:47.265 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:47.265 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:47.265 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:47.265 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:47.265 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:47.265 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:47.265 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:47.265 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:47.265 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:47.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:47.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1503216300 0 0) 0x7f40f412fd60 con 0x7f40f406c120 2024-09-16T14:44:47.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40e0003530 con 0x7f40f406c120 2024-09-16T14:44:47.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f40ec0019e0 con 0x7f40f406c120 2024-09-16T14:44:47.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:47.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:47.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient(hunting): dump: 2024-09-16T14:44:47.266 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:47.266 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:47.266 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:47.266 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:47.266 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:47.266 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:47.267 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:47.267 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:47.267 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:47.267 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:47.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 448220784 0 0) 0x7f40f409e130 con 0x7f40f40a1040 2024-09-16T14:44:47.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 >> v1:172.21.15.189:6789/0 conn(0x7f40f41353d0 legacy=0x7f40f41d1890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:47.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 >> v1:172.21.15.26:6789/0 conn(0x7f40f406c120 legacy=0x7f40f412ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:47.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient: _finish_hunting 0 2024-09-16T14:44:47.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 monclient: found mon.c 2024-09-16T14:44:47.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:47.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:47.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f41322b0 con 0x7f40f40a1040 2024-09-16T14:44:47.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient: _finish_auth 0 2024-09-16T14:44:47.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:47.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:47.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 5 monclient: authenticate success, global_id 4901 2024-09-16T14:44:47.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 10 monclient: _renew_subs 2024-09-16T14:44:47.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:47.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 1 -- 172.21.15.189:0/1728580615 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f4132870 con 0x7f40f40a1040 2024-09-16T14:44:47.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 10 monclient: _renew_subs 2024-09-16T14:44:47.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:47.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40fb0fe640 1 -- 172.21.15.189:0/1728580615 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f4130ce0 con 0x7f40f40a1040 2024-09-16T14:44:47.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40e8004e20 con 0x7f40f40a1040 2024-09-16T14:44:47.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:47.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f40e8005130 con 0x7f40f40a1040 2024-09-16T14:44:47.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:47.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:47.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f2ffd640 4 set_mon_vals no callback set 2024-09-16T14:44:47.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.249+0000 7f40f17fa640 10 monclient: dump: 2024-09-16T14:44:47.269 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:47.269 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:47.269 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:47.269 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:47.269 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:47.269 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:47.269 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:47.269 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:47.269 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:47.269 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:47.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.253+0000 7f40fb0fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:47.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.253+0000 7f40fb0fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:47.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.253+0000 7f40fb0fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:47.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.253+0000 7f40fb0fe640 1 -- 172.21.15.189:0/1728580615 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d4005050 con 0x7f40f40a1040 2024-09-16T14:44:47.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.253+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f40e800fba0 con 0x7f40f40a1040 2024-09-16T14:44:47.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.253+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(49..49 src has 15..49) v4 ==== 3931+0+0 (unknown 921084077 0 0) 0x7f40e80436c0 con 0x7f40f40a1040 2024-09-16T14:44:47.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.253+0000 7f40f8e9a640 10 monclient: get_auth_request con 0x7f40e0036180 auth_method 0 2024-09-16T14:44:47.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.265+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f40e8014930 con 0x7f40f40a1040 2024-09-16T14:44:47.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.265+0000 7f40f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:47.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.265+0000 7f40f17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:47.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.429+0000 7f40fb0fe640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T14:44:47.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.429+0000 7f40fb0fe640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T14:44:47.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.429+0000 7f40fb0fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:47.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.429+0000 7f40fb0fe640 1 -- 172.21.15.189:0/1728580615 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"} v 0) v1 -- 0x7f40d4005300 con 0x7f40f40a1040 2024-09-16T14:44:48.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.997+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}]=0 v50) v1 ==== 150+0+0 (unknown 3009447559 0 0) 0x7f40e8014750 con 0x7f40f40a1040 2024-09-16T14:44:48.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.997+0000 7f40f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T14:44:48.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:47.997+0000 7f40f17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:48.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.097+0000 7f40fb0fe640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T14:44:48.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.097+0000 7f40fb0fe640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T14:44:48.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.097+0000 7f40fb0fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:48.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.097+0000 7f40fb0fe640 1 -- 172.21.15.189:0/1728580615 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"} v 0) v1 -- 0x7f40d4005510 con 0x7f40f40a1040 2024-09-16T14:44:48.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.097+0000 7f40f17fa640 1 -- 172.21.15.189:0/1728580615 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}]=0 rule foo-hdd already exists v50) v1 ==== 177+0+0 (unknown 1422663505 0 0) 0x7f40e8015490 con 0x7f40f40a1040 2024-09-16T14:44:48.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.097+0000 7f40f17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T14:44:48.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.097+0000 7f40f17fa640 10 monclient: _finish_command 3 = system:0 rule foo-hdd already exists 2024-09-16T14:44:48.102 INFO:tasks.workunit.client.0.smithi189.stderr:rule foo-hdd already exists 2024-09-16T14:44:48.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.101+0000 7f40fb0fe640 1 -- 172.21.15.189:0/1728580615 >> v1:172.21.15.189:6812/3196161582 conn(0x7f40e0036180 legacy=0x7f40e0038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:48.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.101+0000 7f40fb0fe640 10 monclient: shutdown 2024-09-16T14:44:48.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.101+0000 7f40fb0fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:48.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.101+0000 7f40fb0fe640 1 -- 172.21.15.189:0/1728580615 >> v1:172.21.15.26:6790/0 conn(0x7f40f40a1040 legacy=0x7f40f412f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:48.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.101+0000 7f40fb0fe640 1 -- 172.21.15.189:0/1728580615 shutdown_connections 2024-09-16T14:44:48.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.101+0000 7f40fb0fe640 1 -- 172.21.15.189:0/1728580615 >> 172.21.15.189:0/1728580615 conn(0x7f40f40a3a40 msgr2=0x7f40f40a1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:48.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.101+0000 7f40fb0fe640 1 -- 172.21.15.189:0/1728580615 shutdown_connections 2024-09-16T14:44:48.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.101+0000 7f40fb0fe640 1 -- 172.21.15.189:0/1728580615 wait complete. 2024-09-16T14:44:48.118 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls-by-class ssd 2024-09-16T14:44:48.119 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep foo-ssd 2024-09-16T14:44:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.221+0000 7f56b3402640 10 monclient: get_monmap_and_config 2024-09-16T14:44:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.221+0000 7f56b3402640 10 monclient: build_initial_monmap 2024-09-16T14:44:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.221+0000 7f56b3402640 10 monclient: monmap: 2024-09-16T14:44:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:48.224088+0000 2024-09-16T14:44:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:48.224088+0000 2024-09-16T14:44:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:48.226 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:48.226 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:48.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.221+0000 7f56b3402640 1 Processor -- start 2024-09-16T14:44:48.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.221+0000 7f56b3402640 1 -- start start 2024-09-16T14:44:48.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.221+0000 7f56b3402640 10 monclient: init 2024-09-16T14:44:48.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.221+0000 7f56b3402640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:48.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.221+0000 7f56b3402640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:48.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.221+0000 7f56b3402640 10 monclient(hunting): picked mon.noname-a con 0x7f56ac0a1a20 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:48.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.221+0000 7f56b3402640 10 monclient(hunting): picked mon.noname-c con 0x7f56ac09fcd0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:48.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 10 monclient(hunting): picked mon.noname-b con 0x7f56ac0a2510 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:48.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ac09e520 con 0x7f56ac0a1a20 2024-09-16T14:44:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ac09ea90 con 0x7f56ac09fcd0 2024-09-16T14:44:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ac09efc0 con 0x7f56ac0a2510 2024-09-16T14:44:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 10 monclient(hunting): authenticate will time out at 1070.340084s 2024-09-16T14:44:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b099d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f56ac09fcd0 legacy=0x7f56ac139ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b119e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f56ac0a1a20 0x7f56ac0a1e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33012/0 (socket says 172.21.15.189:33012) 2024-09-16T14:44:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b119e640 1 -- 172.21.15.189:0/255030751 learned_addr learned my addr 172.21.15.189:0/255030751 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 1 -- 172.21.15.189:0/255030751 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654445910 0 0) 0x7f56ac09e520 con 0x7f56ac0a1a20 2024-09-16T14:44:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient(hunting): my global_id is 84291 2024-09-16T14:44:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 1 -- 172.21.15.189:0/255030751 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5698003530 con 0x7f56ac0a1a20 2024-09-16T14:44:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 1 -- 172.21.15.189:0/255030751 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982609271 0 0) 0x7f56ac09efc0 con 0x7f56ac0a2510 2024-09-16T14:44:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient(hunting): my global_id is 4904 2024-09-16T14:44:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 1 -- 172.21.15.189:0/255030751 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56ac09e520 con 0x7f56ac0a2510 2024-09-16T14:44:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 1 -- 172.21.15.189:0/255030751 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4172832741 0 0) 0x7f56ac09e520 con 0x7f56ac0a2510 2024-09-16T14:44:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 1 -- 172.21.15.189:0/255030751 >> v1:172.21.15.189:6789/0 conn(0x7f56ac09fcd0 legacy=0x7f56ac139ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 1 -- 172.21.15.189:0/255030751 >> v1:172.21.15.26:6789/0 conn(0x7f56ac0a1a20 legacy=0x7f56ac0a1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:44:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 1 monclient: found mon.noname-b 2024-09-16T14:44:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:48.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:48.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 1 -- 172.21.15.189:0/255030751 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ac09f2a0 con 0x7f56ac0a2510 2024-09-16T14:44:48.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient: _finish_auth 0 2024-09-16T14:44:48.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:44:48.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:48.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 5 monclient: authenticate success, global_id 4904 2024-09-16T14:44:48.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:48.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 1 -- 172.21.15.189:0/255030751 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f56a4001980 con 0x7f56ac0a2510 2024-09-16T14:44:48.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:48.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:48.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient: dump: 2024-09-16T14:44:48.233 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:48.233 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:48.234 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.234 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.234 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:48.234 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:48.234 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:48.234 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:48.234 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:48.234 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:48.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:48.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 1 -- 172.21.15.189:0/255030751 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56a4001ca0 con 0x7f56ac0a2510 2024-09-16T14:44:48.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:48.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:48.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 4 set_mon_vals no callback set 2024-09-16T14:44:48.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 1 -- 172.21.15.189:0/255030751 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f56a4002030 con 0x7f56ac0a2510 2024-09-16T14:44:48.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:48.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:48.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56a3fff640 10 monclient: dump: 2024-09-16T14:44:48.235 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:48.235 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:48.235 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.235 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.235 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:48.235 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:48.235 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:48.235 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:48.235 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:48.236 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:48.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 10 monclient: shutdown 2024-09-16T14:44:48.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:48.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 1 -- 172.21.15.189:0/255030751 >> v1:172.21.15.26:6790/0 conn(0x7f56ac0a2510 legacy=0x7f56ac13d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:48.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 1 -- 172.21.15.189:0/255030751 shutdown_connections 2024-09-16T14:44:48.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 1 -- 172.21.15.189:0/255030751 >> 172.21.15.189:0/255030751 conn(0x7f56ac0a3b20 msgr2=0x7f56ac0a3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:48.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 1 -- 172.21.15.189:0/255030751 shutdown_connections 2024-09-16T14:44:48.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 1 -- 172.21.15.189:0/255030751 wait complete. 2024-09-16T14:44:48.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 10 monclient: build_initial_monmap 2024-09-16T14:44:48.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 10 monclient: monmap: 2024-09-16T14:44:48.236 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:48.236 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:48.236 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:48.237 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:48.237 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:48.237 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:48.237 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:48.237 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:48.237 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:48.237 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:48.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 1 Processor -- start 2024-09-16T14:44:48.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 1 -- start start 2024-09-16T14:44:48.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 10 monclient: init 2024-09-16T14:44:48.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:48.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:48.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 10 monclient(hunting): picked mon.noname-b con 0x7f56ac09fcd0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:48.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 10 monclient(hunting): picked mon.noname-a con 0x7f56ac0a1a20 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:48.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 10 monclient(hunting): picked mon.noname-c con 0x7f56ac0a2510 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:48.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ac0a2d40 con 0x7f56ac0a1a20 2024-09-16T14:44:48.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ac1346e0 con 0x7f56ac09fcd0 2024-09-16T14:44:48.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ac09e210 con 0x7f56ac0a2510 2024-09-16T14:44:48.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:48.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b119e640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f56ac09fcd0 legacy=0x7f56ac1c8d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:48.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.225+0000 7f56b3402640 10 monclient(hunting): authenticate will time out at 1070.343744s 2024-09-16T14:44:48.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56b099d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f56ac0a1a20 0x7f56ac133960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33026/0 (socket says 172.21.15.189:33026) 2024-09-16T14:44:48.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56b099d640 1 -- 172.21.15.189:0/2197715384 learned_addr learned my addr 172.21.15.189:0/2197715384 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:48.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56b199f640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f56ac0a2510 0x7f56ac1d10e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:39572/0 (socket says 172.21.15.189:39572) 2024-09-16T14:44:48.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767190324 0 0) 0x7f56ac0a2d40 con 0x7f56ac0a1a20 2024-09-16T14:44:48.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:48.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:48.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:48.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient(hunting): my global_id is 84294 2024-09-16T14:44:48.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5690003530 con 0x7f56ac0a1a20 2024-09-16T14:44:48.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828108234 0 0) 0x7f56ac09e210 con 0x7f56ac0a2510 2024-09-16T14:44:48.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:48.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:48.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:48.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient(hunting): my global_id is 4907 2024-09-16T14:44:48.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56ac0a2d40 con 0x7f56ac0a2510 2024-09-16T14:44:48.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2326829461 0 0) 0x7f56ac0a2d40 con 0x7f56ac0a2510 2024-09-16T14:44:48.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56ac09e210 con 0x7f56ac0a2510 2024-09-16T14:44:48.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f56a40019a0 con 0x7f56ac0a2510 2024-09-16T14:44:48.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:48.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:48.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient(hunting): dump: 2024-09-16T14:44:48.242 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:48.242 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:48.242 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.242 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.242 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:48.243 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:48.243 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:48.243 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:48.243 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:48.243 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:48.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:48.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3885020765 0 0) 0x7f5690003530 con 0x7f56ac0a1a20 2024-09-16T14:44:48.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56ac0a2d40 con 0x7f56ac0a1a20 2024-09-16T14:44:48.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f56a8004130 con 0x7f56ac0a1a20 2024-09-16T14:44:48.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:48.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:48.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient(hunting): dump: 2024-09-16T14:44:48.243 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:48.244 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:48.244 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.244 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.244 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:48.244 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:48.244 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:48.244 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:48.244 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:48.244 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:48.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1123921465 0 0) 0x7f56ac09e210 con 0x7f56ac0a2510 2024-09-16T14:44:48.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 >> v1:172.21.15.189:6789/0 conn(0x7f56ac09fcd0 legacy=0x7f56ac1c8d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:48.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 >> v1:172.21.15.26:6789/0 conn(0x7f56ac0a1a20 legacy=0x7f56ac133960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:48.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:44:48.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 monclient: found mon.c 2024-09-16T14:44:48.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:48.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:48.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ac0a6220 con 0x7f56ac0a2510 2024-09-16T14:44:48.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:44:48.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:44:48.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:48.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56b3402640 5 monclient: authenticate success, global_id 4907 2024-09-16T14:44:48.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56b3402640 10 monclient: _renew_subs 2024-09-16T14:44:48.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56b3402640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:48.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56b3402640 1 -- 172.21.15.189:0/2197715384 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56ac120a80 con 0x7f56ac0a2510 2024-09-16T14:44:48.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56b3402640 10 monclient: _renew_subs 2024-09-16T14:44:48.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56b3402640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:48.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56b3402640 1 -- 172.21.15.189:0/2197715384 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56ac11efc0 con 0x7f56ac0a2510 2024-09-16T14:44:48.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56a4003e50 con 0x7f56ac0a2510 2024-09-16T14:44:48.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:48.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f56a40041e0 con 0x7f56ac0a2510 2024-09-16T14:44:48.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:48.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a37fe640 4 set_mon_vals no callback set 2024-09-16T14:44:48.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:48.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 10 monclient: dump: 2024-09-16T14:44:48.246 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:48.246 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:48.246 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.246 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.247 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:48.247 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:48.247 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:48.247 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:48.247 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:48.247 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:48.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56b3402640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:48.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56b3402640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:48.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56b3402640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:48.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56b3402640 1 -- 172.21.15.189:0/2197715384 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f568c005050 con 0x7f56ac0a2510 2024-09-16T14:44:48.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f56a4004340 con 0x7f56ac0a2510 2024-09-16T14:44:48.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(50..50 src has 15..50) v4 ==== 3994+0+0 (unknown 3713414589 0 0) 0x7f56a40436f0 con 0x7f56ac0a2510 2024-09-16T14:44:48.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.229+0000 7f56b119e640 10 monclient: get_auth_request con 0x7f5690036200 auth_method 0 2024-09-16T14:44:48.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.241+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f56a40149e0 con 0x7f56ac0a2510 2024-09-16T14:44:48.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.241+0000 7f56a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:48.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.241+0000 7f56a1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:48.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.389+0000 7f56b3402640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T14:44:48.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.389+0000 7f56b3402640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T14:44:48.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.389+0000 7f56b3402640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:48.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.389+0000 7f56b3402640 1 -- 172.21.15.189:0/2197715384 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7f568c005b10 con 0x7f56ac0a2510 2024-09-16T14:44:48.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.389+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v50) v1 ==== 90+0+8 (unknown 3465003168 0 905510949) 0x7f56a4014800 con 0x7f56ac0a2510 2024-09-16T14:44:48.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.389+0000 7f56a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T14:44:48.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.389+0000 7f56a1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:48.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.481+0000 7f56b3402640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T14:44:48.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.481+0000 7f56b3402640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T14:44:48.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.481+0000 7f56b3402640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:48.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.481+0000 7f56b3402640 1 -- 172.21.15.189:0/2197715384 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7f568c0010a0 con 0x7f56ac0a2510 2024-09-16T14:44:48.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.481+0000 7f56a1ffb640 1 -- 172.21.15.189:0/2197715384 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v50) v1 ==== 90+0+8 (unknown 3465003168 0 905510949) 0x7f56a4014800 con 0x7f56ac0a2510 2024-09-16T14:44:48.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.481+0000 7f56a1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T14:44:48.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.481+0000 7f56a1ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:44:48.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.485+0000 7f56b3402640 1 -- 172.21.15.189:0/2197715384 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5690036200 legacy=0x7f56900386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:48.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.485+0000 7f56b3402640 10 monclient: shutdown 2024-09-16T14:44:48.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.485+0000 7f56b3402640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:48.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.485+0000 7f56b3402640 1 -- 172.21.15.189:0/2197715384 >> v1:172.21.15.26:6790/0 conn(0x7f56ac0a2510 legacy=0x7f56ac1d10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:48.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.485+0000 7f56b3402640 1 -- 172.21.15.189:0/2197715384 shutdown_connections 2024-09-16T14:44:48.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.485+0000 7f56b3402640 1 -- 172.21.15.189:0/2197715384 >> 172.21.15.189:0/2197715384 conn(0x7f56ac0a3b20 msgr2=0x7f56ac0a4ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:48.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.485+0000 7f56b3402640 1 -- 172.21.15.189:0/2197715384 shutdown_connections 2024-09-16T14:44:48.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.485+0000 7f56b3402640 1 -- 172.21.15.189:0/2197715384 wait complete. 2024-09-16T14:44:48.499 INFO:tasks.workunit.client.0.smithi189.stdout:foo-ssd 2024-09-16T14:44:48.500 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls-by-class ssd 2024-09-16T14:44:48.500 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false grep foo-hdd 2024-09-16T14:44:48.500 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:44:48.500 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep foo-hdd 2024-09-16T14:44:48.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efffae26640 10 monclient: get_monmap_and_config 2024-09-16T14:44:48.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efffae26640 10 monclient: build_initial_monmap 2024-09-16T14:44:48.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efffae26640 10 monclient: monmap: 2024-09-16T14:44:48.616 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:48.616 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:48.616 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:48.616025+0000 2024-09-16T14:44:48.616 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:48.616025+0000 2024-09-16T14:44:48.616 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:48.617 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:48.617 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:48.617 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:48.617 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:48.617 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:48.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efffae26640 1 Processor -- start 2024-09-16T14:44:48.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efffae26640 1 -- start start 2024-09-16T14:44:48.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efffae26640 10 monclient: init 2024-09-16T14:44:48.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efffae26640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:48.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efffae26640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:48.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efffae26640 10 monclient(hunting): picked mon.noname-a con 0x7efff41318a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:48.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efffae26640 10 monclient(hunting): picked mon.noname-c con 0x7efff4134d50 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:48.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efffae26640 10 monclient(hunting): picked mon.noname-b con 0x7efff413b2b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:48.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efffae26640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff409d8a0 con 0x7efff41318a0 2024-09-16T14:44:48.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efffae26640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff409ddf0 con 0x7efff4134d50 2024-09-16T14:44:48.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efffae26640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff409e8b0 con 0x7efff413b2b0 2024-09-16T14:44:48.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efffae26640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:48.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efff3fff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7efff4134d50 legacy=0x7efff4139b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:48.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.613+0000 7efffae26640 10 monclient(hunting): authenticate will time out at 1070.731833s 2024-09-16T14:44:48.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff93c3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7efff413b2b0 0x7efff413d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:39586/0 (socket says 172.21.15.189:39586) 2024-09-16T14:44:48.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff93c3640 1 -- 172.21.15.189:0/3694354318 learned_addr learned my addr 172.21.15.189:0/3694354318 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:48.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff8bc2640 1 --1- 172.21.15.189:0/3694354318 >> v1:172.21.15.26:6789/0 conn(0x7efff41318a0 0x7efff4131c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33038/0 (socket says 172.21.15.189:33038) 2024-09-16T14:44:48.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 1 -- 172.21.15.189:0/3694354318 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409285875 0 0) 0x7efff409e8b0 con 0x7efff413b2b0 2024-09-16T14:44:48.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:48.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:48.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:48.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient(hunting): my global_id is 4910 2024-09-16T14:44:48.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 1 -- 172.21.15.189:0/3694354318 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe0003530 con 0x7efff413b2b0 2024-09-16T14:44:48.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 1 -- 172.21.15.189:0/3694354318 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018664243 0 0) 0x7efff409d8a0 con 0x7efff41318a0 2024-09-16T14:44:48.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:48.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:48.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:48.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient(hunting): my global_id is 84297 2024-09-16T14:44:48.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 1 -- 172.21.15.189:0/3694354318 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efff409e8b0 con 0x7efff41318a0 2024-09-16T14:44:48.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 1 -- 172.21.15.189:0/3694354318 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 541056274 0 0) 0x7effe0003530 con 0x7efff413b2b0 2024-09-16T14:44:48.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 1 -- 172.21.15.189:0/3694354318 >> v1:172.21.15.189:6789/0 conn(0x7efff4134d50 legacy=0x7efff4139b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:48.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 1 -- 172.21.15.189:0/3694354318 >> v1:172.21.15.26:6789/0 conn(0x7efff41318a0 legacy=0x7efff4131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:48.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:44:48.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 1 monclient: found mon.noname-b 2024-09-16T14:44:48.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:48.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:48.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 1 -- 172.21.15.189:0/3694354318 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff409eb90 con 0x7efff413b2b0 2024-09-16T14:44:48.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient: _finish_auth 0 2024-09-16T14:44:48.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:44:48.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:48.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 1 -- 172.21.15.189:0/3694354318 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7effec0019a0 con 0x7efff413b2b0 2024-09-16T14:44:48.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:48.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:48.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient: dump: 2024-09-16T14:44:48.628 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:48.628 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:48.628 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.628 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.628 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:48.628 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:48.628 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:48.629 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:48.629 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:48.629 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:48.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 5 monclient: authenticate success, global_id 4910 2024-09-16T14:44:48.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:48.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 1 -- 172.21.15.189:0/3694354318 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effec001c10 con 0x7efff413b2b0 2024-09-16T14:44:48.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:48.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:48.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 4 set_mon_vals no callback set 2024-09-16T14:44:48.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 1 -- 172.21.15.189:0/3694354318 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7effec001fa0 con 0x7efff413b2b0 2024-09-16T14:44:48.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:48.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:48.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff37fe640 10 monclient: dump: 2024-09-16T14:44:48.630 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:48.630 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:48.630 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.630 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.631 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:48.631 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:48.631 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:48.631 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:48.631 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:48.631 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:48.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 10 monclient: shutdown 2024-09-16T14:44:48.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:48.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 1 -- 172.21.15.189:0/3694354318 >> v1:172.21.15.26:6790/0 conn(0x7efff413b2b0 legacy=0x7efff413d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:48.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 1 -- 172.21.15.189:0/3694354318 shutdown_connections 2024-09-16T14:44:48.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 1 -- 172.21.15.189:0/3694354318 >> 172.21.15.189:0/3694354318 conn(0x7efff40a3320 msgr2=0x7efff409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:48.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 1 -- 172.21.15.189:0/3694354318 shutdown_connections 2024-09-16T14:44:48.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 1 -- 172.21.15.189:0/3694354318 wait complete. 2024-09-16T14:44:48.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 10 monclient: build_initial_monmap 2024-09-16T14:44:48.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 10 monclient: monmap: 2024-09-16T14:44:48.632 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:48.632 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:48.632 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:48.633 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 1 Processor -- start 2024-09-16T14:44:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 1 -- start start 2024-09-16T14:44:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 10 monclient: init 2024-09-16T14:44:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 10 monclient(hunting): picked mon.noname-c con 0x7efff41318a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:48.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 10 monclient(hunting): picked mon.noname-a con 0x7efff4134d50 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:48.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 10 monclient(hunting): picked mon.noname-b con 0x7efff413b2b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:48.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff411b790 con 0x7efff4134d50 2024-09-16T14:44:48.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff411ba70 con 0x7efff413b2b0 2024-09-16T14:44:48.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff411a2e0 con 0x7efff41318a0 2024-09-16T14:44:48.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:48.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efffae26640 10 monclient(hunting): authenticate will time out at 1070.735763s 2024-09-16T14:44:48.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.617+0000 7efff93c3640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7efff413b2b0 legacy=0x7efff41d1da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:48.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff8bc2640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7efff41318a0 0x7efff41c9690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:39600/0 (socket says 172.21.15.189:39600) 2024-09-16T14:44:48.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff8bc2640 1 -- 172.21.15.189:0/1000222693 learned_addr learned my addr 172.21.15.189:0/1000222693 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:48.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088005395 0 0) 0x7efff411b790 con 0x7efff4134d50 2024-09-16T14:44:48.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:44:48.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:48.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:48.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient(hunting): my global_id is 84300 2024-09-16T14:44:48.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd4003530 con 0x7efff4134d50 2024-09-16T14:44:48.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877572828 0 0) 0x7efff411a2e0 con 0x7efff41318a0 2024-09-16T14:44:48.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:44:48.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:48.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:48.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient(hunting): my global_id is 4913 2024-09-16T14:44:48.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efff411b790 con 0x7efff41318a0 2024-09-16T14:44:48.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3622105405 0 0) 0x7effd4003530 con 0x7efff4134d50 2024-09-16T14:44:48.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efff411a2e0 con 0x7efff4134d50 2024-09-16T14:44:48.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7effe40040d0 con 0x7efff4134d50 2024-09-16T14:44:48.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:48.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:48.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient(hunting): dump: 2024-09-16T14:44:48.636 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:48.636 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:48.636 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.636 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.636 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:48.636 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:48.636 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:48.636 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:48.636 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:48.636 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:48.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:48.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4144343610 0 0) 0x7efff411b790 con 0x7efff41318a0 2024-09-16T14:44:48.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effd4003530 con 0x7efff41318a0 2024-09-16T14:44:48.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7effdc001900 con 0x7efff41318a0 2024-09-16T14:44:48.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:48.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:48.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient(hunting): dump: 2024-09-16T14:44:48.637 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:48.637 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:48.637 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.637 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.637 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:48.637 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:48.637 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:48.638 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:48.638 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:48.638 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:48.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2881335806 0 0) 0x7efff411a2e0 con 0x7efff4134d50 2024-09-16T14:44:48.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 >> v1:172.21.15.26:6790/0 conn(0x7efff41318a0 legacy=0x7efff41c9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:48.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 >> v1:172.21.15.189:6789/0 conn(0x7efff413b2b0 legacy=0x7efff41d1da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:48.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient: _finish_hunting 0 2024-09-16T14:44:48.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 monclient: found mon.a 2024-09-16T14:44:48.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:48.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:48.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff411a060 con 0x7efff4134d50 2024-09-16T14:44:48.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient: _finish_auth 0 2024-09-16T14:44:48.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:48.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:48.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efffae26640 5 monclient: authenticate success, global_id 84300 2024-09-16T14:44:48.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efffae26640 10 monclient: _renew_subs 2024-09-16T14:44:48.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efffae26640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:48.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efffae26640 1 -- 172.21.15.189:0/1000222693 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff413b000 con 0x7efff4134d50 2024-09-16T14:44:48.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efffae26640 10 monclient: _renew_subs 2024-09-16T14:44:48.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efffae26640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:48.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efffae26640 1 -- 172.21.15.189:0/1000222693 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efff411b4b0 con 0x7efff4134d50 2024-09-16T14:44:48.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effe4004230 con 0x7efff4134d50 2024-09-16T14:44:48.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:48.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff2ffd640 4 set_mon_vals no callback set 2024-09-16T14:44:48.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7effe40045c0 con 0x7efff4134d50 2024-09-16T14:44:48.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:48.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:48.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 10 monclient: dump: 2024-09-16T14:44:48.640 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:48.640 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:48.640 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.640 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:48.640 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:48.640 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:48.640 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:48.640 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:48.640 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:48.640 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:48.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efffae26640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:48.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efffae26640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:48.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efffae26640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:48.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efffae26640 1 -- 172.21.15.189:0/1000222693 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd8005050 con 0x7efff4134d50 2024-09-16T14:44:48.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7effe4004720 con 0x7efff4134d50 2024-09-16T14:44:48.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(50..50 src has 15..50) v4 ==== 3994+0+0 (unknown 3713414589 0 0) 0x7effe40430b0 con 0x7efff4134d50 2024-09-16T14:44:48.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.621+0000 7efff8bc2640 10 monclient: get_auth_request con 0x7effd40361e0 auth_method 0 2024-09-16T14:44:48.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.633+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7effe4014220 con 0x7efff4134d50 2024-09-16T14:44:48.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.633+0000 7efff17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:48.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.633+0000 7efff17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:48.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.781+0000 7efffae26640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T14:44:48.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.781+0000 7efffae26640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T14:44:48.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.781+0000 7efffae26640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:48.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.781+0000 7efffae26640 1 -- 172.21.15.189:0/1000222693 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7effd8005b10 con 0x7efff4134d50 2024-09-16T14:44:48.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.781+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v50) v1 ==== 90+0+8 (unknown 3465003168 0 905510949) 0x7effe4014040 con 0x7efff4134d50 2024-09-16T14:44:48.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.785+0000 7efff17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T14:44:48.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.785+0000 7efff17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:48.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.873+0000 7efffae26640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T14:44:48.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.873+0000 7efffae26640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T14:44:48.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.873+0000 7efffae26640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:48.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.873+0000 7efffae26640 1 -- 172.21.15.189:0/1000222693 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7effd80010a0 con 0x7efff4134d50 2024-09-16T14:44:48.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.873+0000 7efff17fa640 1 -- 172.21.15.189:0/1000222693 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v50) v1 ==== 90+0+8 (unknown 3465003168 0 905510949) 0x7effe4014040 con 0x7efff4134d50 2024-09-16T14:44:48.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.873+0000 7efff17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T14:44:48.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.873+0000 7efff17fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:44:48.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.873+0000 7efffae26640 1 -- 172.21.15.189:0/1000222693 >> v1:172.21.15.189:6812/3196161582 conn(0x7effd40361e0 legacy=0x7effd40386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:48.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.873+0000 7efffae26640 10 monclient: shutdown 2024-09-16T14:44:48.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.873+0000 7efffae26640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:48.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.873+0000 7efffae26640 1 -- 172.21.15.189:0/1000222693 >> v1:172.21.15.26:6789/0 conn(0x7efff4134d50 legacy=0x7efff41ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:48.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.873+0000 7efffae26640 1 -- 172.21.15.189:0/1000222693 shutdown_connections 2024-09-16T14:44:48.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.873+0000 7efffae26640 1 -- 172.21.15.189:0/1000222693 >> 172.21.15.189:0/1000222693 conn(0x7efff40a3320 msgr2=0x7efff4134080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:48.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.873+0000 7efffae26640 1 -- 172.21.15.189:0/1000222693 shutdown_connections 2024-09-16T14:44:48.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:48.873+0000 7efffae26640 1 -- 172.21.15.189:0/1000222693 wait complete. 2024-09-16T14:44:48.890 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:44:48.890 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls-by-class hdd 2024-09-16T14:44:48.890 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep foo-hdd 2024-09-16T14:44:49.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15be8b0640 10 monclient: get_monmap_and_config 2024-09-16T14:44:49.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15be8b0640 10 monclient: build_initial_monmap 2024-09-16T14:44:49.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15be8b0640 10 monclient: monmap: 2024-09-16T14:44:49.008 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:49.008 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.008 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:49.008110+0000 2024-09-16T14:44:49.008 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:49.008110+0000 2024-09-16T14:44:49.009 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:49.009 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.009 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:49.009 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:49.009 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:49.009 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15be8b0640 1 Processor -- start 2024-09-16T14:44:49.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15be8b0640 1 -- start start 2024-09-16T14:44:49.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15be8b0640 10 monclient: init 2024-09-16T14:44:49.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15be8b0640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:49.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15be8b0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:44:49.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15be8b0640 10 monclient(hunting): picked mon.noname-b con 0x7f15b81318a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:49.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15be8b0640 10 monclient(hunting): picked mon.noname-c con 0x7f15b8134d50 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:49.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15be8b0640 10 monclient(hunting): picked mon.noname-a con 0x7f15b813b2b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:49.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15be8b0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b809d8a0 con 0x7f15b813b2b0 2024-09-16T14:44:49.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15be8b0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b809ddf0 con 0x7f15b8134d50 2024-09-16T14:44:49.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15be8b0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b809e8b0 con 0x7f15b81318a0 2024-09-16T14:44:49.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15be8b0640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:49.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15b77fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f15b8134d50 legacy=0x7f15b8139b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:49.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.005+0000 7f15be8b0640 10 monclient(hunting): authenticate will time out at 1071.123710s 2024-09-16T14:44:49.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.009+0000 7f15b7fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f15b81318a0 0x7f15b8131c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:46636/0 (socket says 172.21.15.189:46636) 2024-09-16T14:44:49.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.009+0000 7f15b7fff640 1 -- 172.21.15.189:0/1779944104 learned_addr learned my addr 172.21.15.189:0/1779944104 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:49.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 1 -- 172.21.15.189:0/1779944104 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869897530 0 0) 0x7f15b809e8b0 con 0x7f15b81318a0 2024-09-16T14:44:49.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:49.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:49.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:49.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient(hunting): my global_id is 4916 2024-09-16T14:44:49.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 1 -- 172.21.15.189:0/1779944104 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a0003530 con 0x7f15b81318a0 2024-09-16T14:44:49.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 1 -- 172.21.15.189:0/1779944104 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038605481 0 0) 0x7f15b809d8a0 con 0x7f15b813b2b0 2024-09-16T14:44:49.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:49.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:49.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:49.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient(hunting): my global_id is 84303 2024-09-16T14:44:49.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 1 -- 172.21.15.189:0/1779944104 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15b809e8b0 con 0x7f15b813b2b0 2024-09-16T14:44:49.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 1 -- 172.21.15.189:0/1779944104 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2437634771 0 0) 0x7f15b809e8b0 con 0x7f15b813b2b0 2024-09-16T14:44:49.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 1 -- 172.21.15.189:0/1779944104 >> v1:172.21.15.26:6790/0 conn(0x7f15b81318a0 legacy=0x7f15b8131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:49.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 1 -- 172.21.15.189:0/1779944104 >> v1:172.21.15.189:6789/0 conn(0x7f15b8134d50 legacy=0x7f15b8139b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:49.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:49.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 1 monclient: found mon.noname-a 2024-09-16T14:44:49.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:49.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:49.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 1 -- 172.21.15.189:0/1779944104 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b809eb90 con 0x7f15b813b2b0 2024-09-16T14:44:49.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:44:49.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:49.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:49.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 1 -- 172.21.15.189:0/1779944104 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f15b0001980 con 0x7f15b813b2b0 2024-09-16T14:44:49.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:49.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient: dump: 2024-09-16T14:44:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:49.021 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 5 monclient: authenticate success, global_id 84303 2024-09-16T14:44:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:49.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 1 -- 172.21.15.189:0/1779944104 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15b0001ba0 con 0x7f15b813b2b0 2024-09-16T14:44:49.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:49.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 1 -- 172.21.15.189:0/1779944104 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f15b0001ef0 con 0x7f15b813b2b0 2024-09-16T14:44:49.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:49.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 4 set_mon_vals no callback set 2024-09-16T14:44:49.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:49.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:49.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b6ffd640 10 monclient: dump: 2024-09-16T14:44:49.023 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:49.023 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.023 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.023 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.023 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:49.023 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.023 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:49.023 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:49.023 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:49.023 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 10 monclient: shutdown 2024-09-16T14:44:49.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:49.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 1 -- 172.21.15.189:0/1779944104 >> v1:172.21.15.26:6789/0 conn(0x7f15b813b2b0 legacy=0x7f15b813d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:49.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 1 -- 172.21.15.189:0/1779944104 shutdown_connections 2024-09-16T14:44:49.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 1 -- 172.21.15.189:0/1779944104 >> 172.21.15.189:0/1779944104 conn(0x7f15b80a3320 msgr2=0x7f15b809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:49.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 1 -- 172.21.15.189:0/1779944104 shutdown_connections 2024-09-16T14:44:49.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 1 -- 172.21.15.189:0/1779944104 wait complete. 2024-09-16T14:44:49.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 10 monclient: build_initial_monmap 2024-09-16T14:44:49.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 10 monclient: monmap: 2024-09-16T14:44:49.024 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:49.024 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:49.024 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:49.024 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:49.024 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:49.025 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:49.025 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:49.025 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:49.025 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:49.025 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 1 Processor -- start 2024-09-16T14:44:49.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 1 -- start start 2024-09-16T14:44:49.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 10 monclient: init 2024-09-16T14:44:49.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:49.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:44:49.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 10 monclient(hunting): picked mon.noname-c con 0x7f15b81318a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:49.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 10 monclient(hunting): picked mon.noname-b con 0x7f15b8134d50 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:49.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 10 monclient(hunting): picked mon.noname-a con 0x7f15b813b2b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b813af70 con 0x7f15b813b2b0 2024-09-16T14:44:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b806b380 con 0x7f15b8134d50 2024-09-16T14:44:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b81ca250 con 0x7f15b81318a0 2024-09-16T14:44:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15be8b0640 10 monclient(hunting): authenticate will time out at 1071.131043s 2024-09-16T14:44:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b77fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f15b8134d50 legacy=0x7f15b81c9b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b7fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f15b81318a0 0x7f15b81c9430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:46650/0 (socket says 172.21.15.189:46650) 2024-09-16T14:44:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b7fff640 1 -- 172.21.15.189:0/3465169362 learned_addr learned my addr 172.21.15.189:0/3465169362 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021324325 0 0) 0x7f15b81ca250 con 0x7f15b81318a0 2024-09-16T14:44:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b4ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b4ff9640 10 monclient(hunting): my global_id is 4919 2024-09-16T14:44:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a4003530 con 0x7f15b81318a0 2024-09-16T14:44:49.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282301476 0 0) 0x7f15b809eb90 con 0x7f15b813b2b0 2024-09-16T14:44:49.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b4ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:44:49.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:49.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:49.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.013+0000 7f15b4ff9640 10 monclient(hunting): my global_id is 84306 2024-09-16T14:44:49.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15b81ca250 con 0x7f15b813b2b0 2024-09-16T14:44:49.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4017228308 0 0) 0x7f15a4003530 con 0x7f15b81318a0 2024-09-16T14:44:49.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15b809eb90 con 0x7f15b81318a0 2024-09-16T14:44:49.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f15a8001920 con 0x7f15b81318a0 2024-09-16T14:44:49.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:49.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:49.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 10 monclient(hunting): dump: 2024-09-16T14:44:49.028 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:49.028 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.028 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.028 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.028 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:49.028 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.028 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:49.028 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:49.028 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:49.029 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:49.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3043072782 0 0) 0x7f15b81ca250 con 0x7f15b813b2b0 2024-09-16T14:44:49.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15a4003530 con 0x7f15b813b2b0 2024-09-16T14:44:49.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f15b00016e0 con 0x7f15b813b2b0 2024-09-16T14:44:49.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:49.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:49.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 10 monclient(hunting): dump: 2024-09-16T14:44:49.029 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:49.029 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.029 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.029 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.029 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:49.030 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.030 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:49.030 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:49.030 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:49.030 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2002594009 0 0) 0x7f15b809eb90 con 0x7f15b81318a0 2024-09-16T14:44:49.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 >> v1:172.21.15.189:6789/0 conn(0x7f15b8134d50 legacy=0x7f15b81c9b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:49.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 >> v1:172.21.15.26:6789/0 conn(0x7f15b813b2b0 legacy=0x7f15b81d1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:49.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:44:49.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 1 monclient: found mon.c 2024-09-16T14:44:49.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:49.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:49.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b813a9d0 con 0x7f15b81318a0 2024-09-16T14:44:49.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 10 monclient: _finish_auth 0 2024-09-16T14:44:49.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:44:49.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:49.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15be8b0640 5 monclient: authenticate success, global_id 4919 2024-09-16T14:44:49.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15be8b0640 10 monclient: _renew_subs 2024-09-16T14:44:49.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15be8b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:49.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15be8b0640 1 -- 172.21.15.189:0/3465169362 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b811b900 con 0x7f15b81318a0 2024-09-16T14:44:49.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15be8b0640 10 monclient: _renew_subs 2024-09-16T14:44:49.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15be8b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:49.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15be8b0640 1 -- 172.21.15.189:0/3465169362 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b8119530 con 0x7f15b81318a0 2024-09-16T14:44:49.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15a80047f0 con 0x7f15b81318a0 2024-09-16T14:44:49.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:49.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f15a8004b00 con 0x7f15b81318a0 2024-09-16T14:44:49.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:49.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b5ffb640 4 set_mon_vals no callback set 2024-09-16T14:44:49.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:49.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 10 monclient: dump: 2024-09-16T14:44:49.032 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:49.032 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.032 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.032 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.032 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:49.032 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.032 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:49.032 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:49.032 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:49.033 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15be8b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:49.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15be8b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:49.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15be8b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:49.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15be8b0640 1 -- 172.21.15.189:0/3465169362 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1598005050 con 0x7f15b81318a0 2024-09-16T14:44:49.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f15a800f570 con 0x7f15b81318a0 2024-09-16T14:44:49.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(50..50 src has 15..50) v4 ==== 3994+0+0 (unknown 3713414589 0 0) 0x7f15a80432c0 con 0x7f15b81318a0 2024-09-16T14:44:49.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.017+0000 7f15b77fe640 10 monclient: get_auth_request con 0x7f15a4036200 auth_method 0 2024-09-16T14:44:49.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.029+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f15a8014440 con 0x7f15b81318a0 2024-09-16T14:44:49.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.029+0000 7f15b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:49.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.029+0000 7f15b4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:49.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.193+0000 7f15be8b0640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T14:44:49.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.193+0000 7f15be8b0640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T14:44:49.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.193+0000 7f15be8b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:49.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.193+0000 7f15be8b0640 1 -- 172.21.15.189:0/3465169362 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7f1598005b10 con 0x7f15b81318a0 2024-09-16T14:44:49.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.193+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v50) v1 ==== 90+0+8 (unknown 1016402913 0 1286160932) 0x7f15a8014260 con 0x7f15b81318a0 2024-09-16T14:44:49.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.193+0000 7f15b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T14:44:49.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.193+0000 7f15b4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:49.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.289+0000 7f15be8b0640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T14:44:49.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.289+0000 7f15be8b0640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T14:44:49.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.289+0000 7f15be8b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:49.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.289+0000 7f15be8b0640 1 -- 172.21.15.189:0/3465169362 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7f15980010a0 con 0x7f15b81318a0 2024-09-16T14:44:49.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.289+0000 7f15b4ff9640 1 -- 172.21.15.189:0/3465169362 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v50) v1 ==== 90+0+8 (unknown 1016402913 0 1286160932) 0x7f15a8014440 con 0x7f15b81318a0 2024-09-16T14:44:49.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.289+0000 7f15b4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T14:44:49.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.289+0000 7f15b4ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:44:49.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.293+0000 7f15be8b0640 1 -- 172.21.15.189:0/3465169362 >> v1:172.21.15.189:6812/3196161582 conn(0x7f15a4036200 legacy=0x7f15a40386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:49.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.293+0000 7f15be8b0640 10 monclient: shutdown 2024-09-16T14:44:49.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.293+0000 7f15be8b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:49.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.293+0000 7f15be8b0640 1 -- 172.21.15.189:0/3465169362 >> v1:172.21.15.26:6790/0 conn(0x7f15b81318a0 legacy=0x7f15b81c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:49.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.293+0000 7f15be8b0640 1 -- 172.21.15.189:0/3465169362 shutdown_connections 2024-09-16T14:44:49.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.293+0000 7f15be8b0640 1 -- 172.21.15.189:0/3465169362 >> 172.21.15.189:0/3465169362 conn(0x7f15b80a3320 msgr2=0x7f15b8133170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:49.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.293+0000 7f15be8b0640 1 -- 172.21.15.189:0/3465169362 shutdown_connections 2024-09-16T14:44:49.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.293+0000 7f15be8b0640 1 -- 172.21.15.189:0/3465169362 wait complete. 2024-09-16T14:44:49.309 INFO:tasks.workunit.client.0.smithi189.stdout:foo-hdd 2024-09-16T14:44:49.310 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls-by-class hdd 2024-09-16T14:44:49.310 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false grep foo-ssd 2024-09-16T14:44:49.310 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:44:49.310 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep foo-ssd 2024-09-16T14:44:49.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.421+0000 7fb6f75d7640 10 monclient: get_monmap_and_config 2024-09-16T14:44:49.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.421+0000 7fb6f75d7640 10 monclient: build_initial_monmap 2024-09-16T14:44:49.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.421+0000 7fb6f75d7640 10 monclient: monmap: 2024-09-16T14:44:49.425 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:49.425 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.425 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:49.424343+0000 2024-09-16T14:44:49.425 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:49.424343+0000 2024-09-16T14:44:49.425 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:49.425 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.425 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:49.425 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:49.425 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:49.425 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.421+0000 7fb6f75d7640 1 Processor -- start 2024-09-16T14:44:49.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.421+0000 7fb6f75d7640 1 -- start start 2024-09-16T14:44:49.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.421+0000 7fb6f75d7640 10 monclient: init 2024-09-16T14:44:49.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.421+0000 7fb6f75d7640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:49.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.421+0000 7fb6f75d7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:44:49.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.421+0000 7fb6f75d7640 10 monclient(hunting): picked mon.noname-b con 0x7fb6f01315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:49.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.421+0000 7fb6f75d7640 10 monclient(hunting): picked mon.noname-c con 0x7fb6f009b320 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:49.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.421+0000 7fb6f75d7640 10 monclient(hunting): picked mon.noname-a con 0x7fb6f0132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:49.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.421+0000 7fb6f75d7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f009edd0 con 0x7fb6f0132090 2024-09-16T14:44:49.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.421+0000 7fb6f75d7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f009bc00 con 0x7fb6f009b320 2024-09-16T14:44:49.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.421+0000 7fb6f75d7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f009e900 con 0x7fb6f01315a0 2024-09-16T14:44:49.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.421+0000 7fb6f75d7640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:49.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.421+0000 7fb6f75d7640 10 monclient(hunting): authenticate will time out at 1071.539910s 2024-09-16T14:44:49.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f4b72640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fb6f009b320 legacy=0x7fb6f0136ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:49.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f5b74640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb6f0132090 0x7fb6f013a200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:54028/0 (socket says 172.21.15.189:54028) 2024-09-16T14:44:49.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f5373640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb6f01315a0 0x7fb6f0131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:46662/0 (socket says 172.21.15.189:46662) 2024-09-16T14:44:49.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f5b74640 1 -- 172.21.15.189:0/3772102888 learned_addr learned my addr 172.21.15.189:0/3772102888 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:49.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 1 -- 172.21.15.189:0/3772102888 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669794745 0 0) 0x7fb6f009edd0 con 0x7fb6f0132090 2024-09-16T14:44:49.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:49.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:49.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:49.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient(hunting): my global_id is 84309 2024-09-16T14:44:49.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 1 -- 172.21.15.189:0/3772102888 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6dc005740 con 0x7fb6f0132090 2024-09-16T14:44:49.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 1 -- 172.21.15.189:0/3772102888 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961594947 0 0) 0x7fb6f009e900 con 0x7fb6f01315a0 2024-09-16T14:44:49.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:49.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:49.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:49.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient(hunting): my global_id is 4922 2024-09-16T14:44:49.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 1 -- 172.21.15.189:0/3772102888 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6f009edd0 con 0x7fb6f01315a0 2024-09-16T14:44:49.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 1 -- 172.21.15.189:0/3772102888 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 799381874 0 0) 0x7fb6dc005740 con 0x7fb6f0132090 2024-09-16T14:44:49.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 1 -- 172.21.15.189:0/3772102888 >> v1:172.21.15.26:6790/0 conn(0x7fb6f01315a0 legacy=0x7fb6f0131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:49.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 1 -- 172.21.15.189:0/3772102888 >> v1:172.21.15.189:6789/0 conn(0x7fb6f009b320 legacy=0x7fb6f0136ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:49.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:44:49.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 1 monclient: found mon.noname-a 2024-09-16T14:44:49.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:49.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:49.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 1 -- 172.21.15.189:0/3772102888 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f009b920 con 0x7fb6f0132090 2024-09-16T14:44:49.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient: _finish_auth 0 2024-09-16T14:44:49.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:44:49.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:49.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 1 -- 172.21.15.189:0/3772102888 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb6e8001550 con 0x7fb6f0132090 2024-09-16T14:44:49.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:49.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:49.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient: dump: 2024-09-16T14:44:49.434 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:49.434 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.434 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.434 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.434 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:49.434 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.434 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:49.434 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:49.434 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:49.435 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 5 monclient: authenticate success, global_id 84309 2024-09-16T14:44:49.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:49.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 1 -- 172.21.15.189:0/3772102888 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6e80017b0 con 0x7fb6f0132090 2024-09-16T14:44:49.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:49.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:49.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 4 set_mon_vals no callback set 2024-09-16T14:44:49.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 1 -- 172.21.15.189:0/3772102888 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb6e8001b80 con 0x7fb6f0132090 2024-09-16T14:44:49.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:49.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:49.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6e7fff640 10 monclient: dump: 2024-09-16T14:44:49.435 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:49.435 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.436 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.436 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.436 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:49.436 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.436 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:49.436 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:49.436 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:49.436 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 10 monclient: shutdown 2024-09-16T14:44:49.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:49.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 1 -- 172.21.15.189:0/3772102888 >> v1:172.21.15.26:6789/0 conn(0x7fb6f0132090 legacy=0x7fb6f013a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:49.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 1 -- 172.21.15.189:0/3772102888 shutdown_connections 2024-09-16T14:44:49.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 1 -- 172.21.15.189:0/3772102888 >> 172.21.15.189:0/3772102888 conn(0x7fb6f00a3320 msgr2=0x7fb6f009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:49.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 1 -- 172.21.15.189:0/3772102888 shutdown_connections 2024-09-16T14:44:49.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 1 -- 172.21.15.189:0/3772102888 wait complete. 2024-09-16T14:44:49.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 10 monclient: build_initial_monmap 2024-09-16T14:44:49.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 10 monclient: monmap: 2024-09-16T14:44:49.437 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:49.437 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:49.437 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:49.437 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:49.437 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:49.437 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:49.437 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:49.437 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:49.437 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:49.438 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 1 Processor -- start 2024-09-16T14:44:49.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 1 -- start start 2024-09-16T14:44:49.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 10 monclient: init 2024-09-16T14:44:49.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:49.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:44:49.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 10 monclient(hunting): picked mon.noname-b con 0x7fb6f01315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:49.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 10 monclient(hunting): picked mon.noname-c con 0x7fb6f0132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:49.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 10 monclient(hunting): picked mon.noname-a con 0x7fb6f0131120 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:49.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f009cd20 con 0x7fb6f0131120 2024-09-16T14:44:49.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f011da80 con 0x7fb6f01315a0 2024-09-16T14:44:49.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f00a2900 con 0x7fb6f0132090 2024-09-16T14:44:49.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f5373640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fb6f01315a0 legacy=0x7fb6f01c4a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:49.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:49.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.425+0000 7fb6f75d7640 10 monclient(hunting): authenticate will time out at 1071.543992s 2024-09-16T14:44:49.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6f5b74640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb6f0131120 0x7fb6f01ccd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:54032/0 (socket says 172.21.15.189:54032) 2024-09-16T14:44:49.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6f5b74640 1 -- 172.21.15.189:0/4188388075 learned_addr learned my addr 172.21.15.189:0/4188388075 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:49.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6f4b72640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb6f0132090 0x7fb6f0130a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:46678/0 (socket says 172.21.15.189:46678) 2024-09-16T14:44:49.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815148842 0 0) 0x7fb6f009cd20 con 0x7fb6f0131120 2024-09-16T14:44:49.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:49.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:49.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:49.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient(hunting): my global_id is 84312 2024-09-16T14:44:49.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6d4003530 con 0x7fb6f0131120 2024-09-16T14:44:49.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464983437 0 0) 0x7fb6f00a2900 con 0x7fb6f0132090 2024-09-16T14:44:49.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:49.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:49.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:49.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient(hunting): my global_id is 4925 2024-09-16T14:44:49.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6f009cd20 con 0x7fb6f0132090 2024-09-16T14:44:49.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2310213687 0 0) 0x7fb6f009cd20 con 0x7fb6f0132090 2024-09-16T14:44:49.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6f00a2900 con 0x7fb6f0132090 2024-09-16T14:44:49.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb6e00040d0 con 0x7fb6f0132090 2024-09-16T14:44:49.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:49.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:49.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient(hunting): dump: 2024-09-16T14:44:49.442 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:49.442 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.442 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.443 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.443 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:49.443 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.443 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:49.443 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:49.443 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:49.443 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:49.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3757962456 0 0) 0x7fb6d4003530 con 0x7fb6f0131120 2024-09-16T14:44:49.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6f009cd20 con 0x7fb6f0131120 2024-09-16T14:44:49.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb6e8001450 con 0x7fb6f0131120 2024-09-16T14:44:49.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:49.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:49.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient(hunting): dump: 2024-09-16T14:44:49.444 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:49.444 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.444 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.444 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.444 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:49.444 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.444 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:49.444 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:49.444 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:49.444 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 549793228 0 0) 0x7fb6f00a2900 con 0x7fb6f0132090 2024-09-16T14:44:49.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 >> v1:172.21.15.189:6789/0 conn(0x7fb6f01315a0 legacy=0x7fb6f01c4a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:49.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 >> v1:172.21.15.26:6789/0 conn(0x7fb6f0131120 legacy=0x7fb6f01ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:49.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:44:49.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 monclient: found mon.c 2024-09-16T14:44:49.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:49.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:49.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f006b380 con 0x7fb6f0132090 2024-09-16T14:44:49.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient: _finish_auth 0 2024-09-16T14:44:49.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:44:49.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:49.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6f75d7640 5 monclient: authenticate success, global_id 4925 2024-09-16T14:44:49.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6f75d7640 10 monclient: _renew_subs 2024-09-16T14:44:49.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6f75d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:49.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6f75d7640 1 -- 172.21.15.189:0/4188388075 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f011f3e0 con 0x7fb6f0132090 2024-09-16T14:44:49.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6f75d7640 10 monclient: _renew_subs 2024-09-16T14:44:49.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6f75d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:49.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6f75d7640 1 -- 172.21.15.189:0/4188388075 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f009a6e0 con 0x7fb6f0132090 2024-09-16T14:44:49.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6e00046a0 con 0x7fb6f0132090 2024-09-16T14:44:49.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:49.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb6e0004a30 con 0x7fb6f0132090 2024-09-16T14:44:49.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:49.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:49.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e77fe640 4 set_mon_vals no callback set 2024-09-16T14:44:49.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 10 monclient: dump: 2024-09-16T14:44:49.446 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:49.446 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.447 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.447 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.447 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:49.447 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.447 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:49.447 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:49.447 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:49.447 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6f75d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:49.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6f75d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:49.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6f75d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:49.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6f75d7640 1 -- 172.21.15.189:0/4188388075 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d0005050 con 0x7fb6f0132090 2024-09-16T14:44:49.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb6e0004c10 con 0x7fb6f0132090 2024-09-16T14:44:49.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(50..50 src has 15..50) v4 ==== 3994+0+0 (unknown 3713414589 0 0) 0x7fb6e0043210 con 0x7fb6f0132090 2024-09-16T14:44:49.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.429+0000 7fb6f5373640 10 monclient: get_auth_request con 0x7fb6d4036220 auth_method 0 2024-09-16T14:44:49.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.441+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb6e0014750 con 0x7fb6f0132090 2024-09-16T14:44:49.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.441+0000 7fb6e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:49.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.441+0000 7fb6e5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:49.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.589+0000 7fb6f75d7640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T14:44:49.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.589+0000 7fb6f75d7640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T14:44:49.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.589+0000 7fb6f75d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:49.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.589+0000 7fb6f75d7640 1 -- 172.21.15.189:0/4188388075 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7fb6d0005880 con 0x7fb6f0132090 2024-09-16T14:44:49.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.593+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v50) v1 ==== 90+0+8 (unknown 1016402913 0 1286160932) 0x7fb6e0012bc0 con 0x7fb6f0132090 2024-09-16T14:44:49.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.593+0000 7fb6e5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T14:44:49.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.593+0000 7fb6e5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:49.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.681+0000 7fb6f75d7640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T14:44:49.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.681+0000 7fb6f75d7640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T14:44:49.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.681+0000 7fb6f75d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:49.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.681+0000 7fb6f75d7640 1 -- 172.21.15.189:0/4188388075 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7fb6d0005300 con 0x7fb6f0132090 2024-09-16T14:44:49.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.681+0000 7fb6e5ffb640 1 -- 172.21.15.189:0/4188388075 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v50) v1 ==== 90+0+8 (unknown 1016402913 0 1286160932) 0x7fb6e0014340 con 0x7fb6f0132090 2024-09-16T14:44:49.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.681+0000 7fb6e5ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T14:44:49.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.681+0000 7fb6e5ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:44:49.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.685+0000 7fb6f75d7640 1 -- 172.21.15.189:0/4188388075 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb6d4036220 legacy=0x7fb6d40386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:49.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.685+0000 7fb6f75d7640 10 monclient: shutdown 2024-09-16T14:44:49.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.685+0000 7fb6f75d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:49.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.685+0000 7fb6f75d7640 1 -- 172.21.15.189:0/4188388075 >> v1:172.21.15.26:6790/0 conn(0x7fb6f0132090 legacy=0x7fb6f0130a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:49.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.685+0000 7fb6f75d7640 1 -- 172.21.15.189:0/4188388075 shutdown_connections 2024-09-16T14:44:49.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.685+0000 7fb6f75d7640 1 -- 172.21.15.189:0/4188388075 >> 172.21.15.189:0/4188388075 conn(0x7fb6f00a3320 msgr2=0x7fb6f009d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:49.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.685+0000 7fb6f75d7640 1 -- 172.21.15.189:0/4188388075 shutdown_connections 2024-09-16T14:44:49.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.685+0000 7fb6f75d7640 1 -- 172.21.15.189:0/4188388075 wait complete. 2024-09-16T14:44:49.700 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:44:49.701 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd erasure-code-profile set ec-foo-ssd crush-device-class=ssd m=2 k=2 2024-09-16T14:44:49.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22fed6b640 10 monclient: get_monmap_and_config 2024-09-16T14:44:49.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22fed6b640 10 monclient: build_initial_monmap 2024-09-16T14:44:49.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22fed6b640 10 monclient: monmap: 2024-09-16T14:44:49.817 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:49.817 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.817 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:49.816395+0000 2024-09-16T14:44:49.817 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:49.816395+0000 2024-09-16T14:44:49.817 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:49.817 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.817 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:49.817 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:49.817 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:49.817 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22fed6b640 1 Processor -- start 2024-09-16T14:44:49.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22fed6b640 1 -- start start 2024-09-16T14:44:49.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22fed6b640 10 monclient: init 2024-09-16T14:44:49.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22fed6b640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:49.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22fed6b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:44:49.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22fed6b640 10 monclient(hunting): picked mon.noname-b con 0x7f22f8070830 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:49.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22fed6b640 10 monclient(hunting): picked mon.noname-c con 0x7f22f8071320 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:49.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22fed6b640 10 monclient(hunting): picked mon.noname-a con 0x7f22f8134cf0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:49.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22fed6b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f8135740 con 0x7f22f8134cf0 2024-09-16T14:44:49.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22fed6b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f8135160 con 0x7f22f8071320 2024-09-16T14:44:49.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22fed6b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f809d810 con 0x7f22f8070830 2024-09-16T14:44:49.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22fed6b640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:49.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22fed6b640 10 monclient(hunting): authenticate will time out at 1071.931952s 2024-09-16T14:44:49.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.813+0000 7f22f7fff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f22f8071320 legacy=0x7f22f8139d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:49.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fcb07640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f22f8070830 0x7f22f8070c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:46690/0 (socket says 172.21.15.189:46690) 2024-09-16T14:44:49.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fcb07640 1 -- 172.21.15.189:0/3361640960 learned_addr learned my addr 172.21.15.189:0/3361640960 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:49.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 1 -- 172.21.15.189:0/3361640960 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795493603 0 0) 0x7f22f8135740 con 0x7f22f8134cf0 2024-09-16T14:44:49.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:49.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:49.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:49.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient(hunting): my global_id is 84315 2024-09-16T14:44:49.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 1 -- 172.21.15.189:0/3361640960 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e0003530 con 0x7f22f8134cf0 2024-09-16T14:44:49.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 1 -- 172.21.15.189:0/3361640960 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710439145 0 0) 0x7f22f809d810 con 0x7f22f8070830 2024-09-16T14:44:49.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:49.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:49.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:49.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient(hunting): my global_id is 4928 2024-09-16T14:44:49.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 1 -- 172.21.15.189:0/3361640960 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f8135740 con 0x7f22f8070830 2024-09-16T14:44:49.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 1 -- 172.21.15.189:0/3361640960 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 659041155 0 0) 0x7f22e0003530 con 0x7f22f8134cf0 2024-09-16T14:44:49.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 1 -- 172.21.15.189:0/3361640960 >> v1:172.21.15.26:6790/0 conn(0x7f22f8070830 legacy=0x7f22f8070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:49.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 1 -- 172.21.15.189:0/3361640960 >> v1:172.21.15.189:6789/0 conn(0x7f22f8071320 legacy=0x7f22f8139d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:49.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:44:49.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 1 monclient: found mon.noname-a 2024-09-16T14:44:49.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:49.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:49.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 1 -- 172.21.15.189:0/3361640960 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f809f1c0 con 0x7f22f8134cf0 2024-09-16T14:44:49.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient: _finish_auth 0 2024-09-16T14:44:49.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:44:49.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:49.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 1 -- 172.21.15.189:0/3361640960 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22f00019a0 con 0x7f22f8134cf0 2024-09-16T14:44:49.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:49.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:49.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient: dump: 2024-09-16T14:44:49.824 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:49.824 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.824 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.824 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.825 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:49.825 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.826 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:49.826 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:49.826 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:49.826 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 5 monclient: authenticate success, global_id 84315 2024-09-16T14:44:49.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:49.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 1 -- 172.21.15.189:0/3361640960 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22f0001bc0 con 0x7f22f8134cf0 2024-09-16T14:44:49.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:49.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 1 -- 172.21.15.189:0/3361640960 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22f0001f90 con 0x7f22f8134cf0 2024-09-16T14:44:49.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:49.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 4 set_mon_vals no callback set 2024-09-16T14:44:49.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:49.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:49.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22f77fe640 10 monclient: dump: 2024-09-16T14:44:49.827 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:49.827 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.827 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.827 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.827 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:49.827 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.827 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:49.827 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:49.827 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:49.827 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 10 monclient: shutdown 2024-09-16T14:44:49.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:49.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 1 -- 172.21.15.189:0/3361640960 >> v1:172.21.15.26:6789/0 conn(0x7f22f8134cf0 legacy=0x7f22f813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:49.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 1 -- 172.21.15.189:0/3361640960 shutdown_connections 2024-09-16T14:44:49.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 1 -- 172.21.15.189:0/3361640960 >> 172.21.15.189:0/3361640960 conn(0x7f22f80a3380 msgr2=0x7f22f809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:49.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 1 -- 172.21.15.189:0/3361640960 shutdown_connections 2024-09-16T14:44:49.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 1 -- 172.21.15.189:0/3361640960 wait complete. 2024-09-16T14:44:49.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 10 monclient: build_initial_monmap 2024-09-16T14:44:49.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 10 monclient: monmap: 2024-09-16T14:44:49.828 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:49.828 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:49.828 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:49.828 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:49.829 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:49.829 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:49.829 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:49.829 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:49.829 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:49.829 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 1 Processor -- start 2024-09-16T14:44:49.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 1 -- start start 2024-09-16T14:44:49.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 10 monclient: init 2024-09-16T14:44:49.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:49.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:49.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 10 monclient(hunting): picked mon.noname-c con 0x7f22f8070830 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:49.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 10 monclient(hunting): picked mon.noname-a con 0x7f22f8071320 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:49.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 10 monclient(hunting): picked mon.noname-b con 0x7f22f8134cf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:49.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f811c840 con 0x7f22f8071320 2024-09-16T14:44:49.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f81c9e10 con 0x7f22f8134cf0 2024-09-16T14:44:49.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f8132d50 con 0x7f22f8070830 2024-09-16T14:44:49.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:49.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fed6b640 10 monclient(hunting): authenticate will time out at 1071.935705s 2024-09-16T14:44:49.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fd308640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f22f8134cf0 legacy=0x7f22f81d1920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:49.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fcb07640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f22f8070830 0x7f22f81307e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:46700/0 (socket says 172.21.15.189:46700) 2024-09-16T14:44:49.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.817+0000 7f22fcb07640 1 -- 172.21.15.189:0/1224823418 learned_addr learned my addr 172.21.15.189:0/1224823418 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:49.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734812127 0 0) 0x7f22f811c840 con 0x7f22f8071320 2024-09-16T14:44:49.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient(hunting): _negotiate 2024-09-16T14:44:49.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:49.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:49.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient(hunting): my global_id is 84318 2024-09-16T14:44:49.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e4003530 con 0x7f22f8071320 2024-09-16T14:44:49.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332134017 0 0) 0x7f22f8132d50 con 0x7f22f8070830 2024-09-16T14:44:49.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient(hunting): _negotiate 2024-09-16T14:44:49.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:49.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:49.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient(hunting): my global_id is 4931 2024-09-16T14:44:49.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f811c840 con 0x7f22f8070830 2024-09-16T14:44:49.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1075571126 0 0) 0x7f22e4003530 con 0x7f22f8071320 2024-09-16T14:44:49.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22f8132d50 con 0x7f22f8071320 2024-09-16T14:44:49.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22ec0040d0 con 0x7f22f8071320 2024-09-16T14:44:49.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:49.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:49.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient(hunting): dump: 2024-09-16T14:44:49.836 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:49.836 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.836 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.837 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.837 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:49.837 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.837 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:49.837 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:49.837 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:49.837 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:49.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 311769508 0 0) 0x7f22f811c840 con 0x7f22f8070830 2024-09-16T14:44:49.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22e4003530 con 0x7f22f8070830 2024-09-16T14:44:49.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22e80018d0 con 0x7f22f8070830 2024-09-16T14:44:49.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:49.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:49.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient(hunting): dump: 2024-09-16T14:44:49.838 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:49.838 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.838 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.838 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.838 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:49.838 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.838 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:49.838 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:49.838 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:49.838 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1043902948 0 0) 0x7f22f8132d50 con 0x7f22f8071320 2024-09-16T14:44:49.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 >> v1:172.21.15.26:6790/0 conn(0x7f22f8070830 legacy=0x7f22f81307e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:49.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 >> v1:172.21.15.189:6789/0 conn(0x7f22f8134cf0 legacy=0x7f22f81d1920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:49.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient: _finish_hunting 0 2024-09-16T14:44:49.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 monclient: found mon.a 2024-09-16T14:44:49.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:49.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:49.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f806b320 con 0x7f22f8071320 2024-09-16T14:44:49.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient: _finish_auth 0 2024-09-16T14:44:49.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient: _check_auth_tickets 2024-09-16T14:44:49.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:49.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22fed6b640 5 monclient: authenticate success, global_id 84318 2024-09-16T14:44:49.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22fed6b640 10 monclient: _renew_subs 2024-09-16T14:44:49.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22fed6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:49.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22fed6b640 1 -- 172.21.15.189:0/1224823418 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f811e200 con 0x7f22f8071320 2024-09-16T14:44:49.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22fed6b640 10 monclient: _renew_subs 2024-09-16T14:44:49.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22fed6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:49.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22fed6b640 1 -- 172.21.15.189:0/1224823418 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f811be40 con 0x7f22f8071320 2024-09-16T14:44:49.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22ec0042f0 con 0x7f22f8071320 2024-09-16T14:44:49.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:49.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f67fc640 4 set_mon_vals no callback set 2024-09-16T14:44:49.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22ec004680 con 0x7f22f8071320 2024-09-16T14:44:49.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:49.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:49.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 10 monclient: dump: 2024-09-16T14:44:49.841 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:49.841 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:49.841 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.841 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:49.841 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:49.841 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:49.841 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:49.841 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:49.841 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:49.841 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:49.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22fed6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:49.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22fed6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:49.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22fed6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:49.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22fed6b640 1 -- 172.21.15.189:0/1224823418 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22d8005050 con 0x7f22f8071320 2024-09-16T14:44:49.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f22ec004860 con 0x7f22f8071320 2024-09-16T14:44:49.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(50..50 src has 15..50) v4 ==== 3994+0+0 (unknown 3713414589 0 0) 0x7f22ec0431a0 con 0x7f22f8071320 2024-09-16T14:44:49.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.821+0000 7f22fcb07640 10 monclient: get_auth_request con 0x7f22e4036220 auth_method 0 2024-09-16T14:44:49.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.833+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f22ec014370 con 0x7f22f8071320 2024-09-16T14:44:49.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.833+0000 7f22f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:49.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.833+0000 7f22f57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.997+0000 7f22fed6b640 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T14:44:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.997+0000 7f22fed6b640 10 monclient: _send_command 2 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T14:44:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.997+0000 7f22fed6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:49.997+0000 7f22fed6b640 1 -- 172.21.15.189:0/1224823418 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]} v 0) v1 -- 0x7f22d8005300 con 0x7f22f8071320 2024-09-16T14:44:50.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.025+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}]=0 v51) v1 ==== 151+0+0 (unknown 3646192632 0 0) 0x7f22ec012a10 con 0x7f22f8071320 2024-09-16T14:44:50.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.025+0000 7f22f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T14:44:50.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.025+0000 7f22f57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:50.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.117+0000 7f22fed6b640 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T14:44:50.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.117+0000 7f22fed6b640 10 monclient: _send_command 3 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T14:44:50.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.117+0000 7f22fed6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:50.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.117+0000 7f22fed6b640 1 -- 172.21.15.189:0/1224823418 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]} v 0) v1 -- 0x7f22d80010a0 con 0x7f22f8071320 2024-09-16T14:44:50.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.117+0000 7f22f57fa640 1 -- 172.21.15.189:0/1224823418 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}]=0 v51) v1 ==== 151+0+0 (unknown 3646192632 0 0) 0x7f22ec014190 con 0x7f22f8071320 2024-09-16T14:44:50.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.117+0000 7f22f57fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T14:44:50.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.117+0000 7f22f57fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:44:50.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.121+0000 7f22fed6b640 1 -- 172.21.15.189:0/1224823418 >> v1:172.21.15.189:6812/3196161582 conn(0x7f22e4036220 legacy=0x7f22e40386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:50.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.121+0000 7f22fed6b640 10 monclient: shutdown 2024-09-16T14:44:50.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.121+0000 7f22fed6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:50.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.121+0000 7f22fed6b640 1 -- 172.21.15.189:0/1224823418 >> v1:172.21.15.26:6789/0 conn(0x7f22f8071320 legacy=0x7f22f81ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:50.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.121+0000 7f22fed6b640 1 -- 172.21.15.189:0/1224823418 shutdown_connections 2024-09-16T14:44:50.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.121+0000 7f22fed6b640 1 -- 172.21.15.189:0/1224823418 >> 172.21.15.189:0/1224823418 conn(0x7f22f80a3380 msgr2=0x7f22f81317c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:50.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.121+0000 7f22fed6b640 1 -- 172.21.15.189:0/1224823418 shutdown_connections 2024-09-16T14:44:50.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.121+0000 7f22fed6b640 1 -- 172.21.15.189:0/1224823418 wait complete. 2024-09-16T14:44:50.136 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create ec-foo 2 erasure ec-foo-ssd 2024-09-16T14:44:50.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.253+0000 7f29f40fe640 10 monclient: get_monmap_and_config 2024-09-16T14:44:50.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.253+0000 7f29f40fe640 10 monclient: build_initial_monmap 2024-09-16T14:44:50.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.253+0000 7f29f40fe640 10 monclient: monmap: 2024-09-16T14:44:50.256 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:50.257 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:50.257 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:50.256332+0000 2024-09-16T14:44:50.257 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:50.256332+0000 2024-09-16T14:44:50.257 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:50.257 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:50.257 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:50.257 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:50.257 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:50.257 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:50.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.253+0000 7f29f40fe640 1 Processor -- start 2024-09-16T14:44:50.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.253+0000 7f29f40fe640 1 -- start start 2024-09-16T14:44:50.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.253+0000 7f29f40fe640 10 monclient: init 2024-09-16T14:44:50.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:50.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:50.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 10 monclient(hunting): picked mon.noname-a con 0x7f29ec131630 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:50.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 10 monclient(hunting): picked mon.noname-c con 0x7f29ec09edd0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:50.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 10 monclient(hunting): picked mon.noname-b con 0x7f29ec09e950 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:50.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec09b920 con 0x7f29ec131630 2024-09-16T14:44:50.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec09bc00 con 0x7f29ec09edd0 2024-09-16T14:44:50.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec09cba0 con 0x7f29ec09e950 2024-09-16T14:44:50.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:50.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 10 monclient(hunting): authenticate will time out at 1072.372569s 2024-09-16T14:44:50.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f1699640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f29ec09edd0 legacy=0x7f29ec136b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:50.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f1e9a640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f29ec131630 0x7f29ec131a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:54060/0 (socket says 172.21.15.189:54060) 2024-09-16T14:44:50.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f269b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f29ec09e950 0x7f29ec13a290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:46702/0 (socket says 172.21.15.189:46702) 2024-09-16T14:44:50.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f1e9a640 1 -- 172.21.15.189:0/3257570399 learned_addr learned my addr 172.21.15.189:0/3257570399 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:50.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 1 -- 172.21.15.189:0/3257570399 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238640914 0 0) 0x7f29ec09cba0 con 0x7f29ec09e950 2024-09-16T14:44:50.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient(hunting): _negotiate 2024-09-16T14:44:50.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:50.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:50.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient(hunting): my global_id is 4934 2024-09-16T14:44:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 1 -- 172.21.15.189:0/3257570399 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29d8003530 con 0x7f29ec09e950 2024-09-16T14:44:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 1 -- 172.21.15.189:0/3257570399 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737507451 0 0) 0x7f29ec09b920 con 0x7f29ec131630 2024-09-16T14:44:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient(hunting): _negotiate 2024-09-16T14:44:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient(hunting): my global_id is 84321 2024-09-16T14:44:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 1 -- 172.21.15.189:0/3257570399 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29ec09cba0 con 0x7f29ec131630 2024-09-16T14:44:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 1 -- 172.21.15.189:0/3257570399 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 934489508 0 0) 0x7f29ec09cba0 con 0x7f29ec131630 2024-09-16T14:44:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 1 -- 172.21.15.189:0/3257570399 >> v1:172.21.15.26:6790/0 conn(0x7f29ec09e950 legacy=0x7f29ec13a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 1 -- 172.21.15.189:0/3257570399 >> v1:172.21.15.189:6789/0 conn(0x7f29ec09edd0 legacy=0x7f29ec136b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient: _finish_hunting 0 2024-09-16T14:44:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 1 monclient: found mon.noname-a 2024-09-16T14:44:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 1 -- 172.21.15.189:0/3257570399 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec09b420 con 0x7f29ec131630 2024-09-16T14:44:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient: _finish_auth 0 2024-09-16T14:44:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient: _check_auth_tickets 2024-09-16T14:44:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 1 -- 172.21.15.189:0/3257570399 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f29e8001600 con 0x7f29ec131630 2024-09-16T14:44:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 5 monclient: authenticate success, global_id 84321 2024-09-16T14:44:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient: dump: 2024-09-16T14:44:50.269 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:50.269 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:50.269 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:50.269 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:50.269 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:50.269 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:50.269 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:50.269 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:50.269 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:50.269 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:50.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:50.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 1 -- 172.21.15.189:0/3257570399 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29e8001820 con 0x7f29ec131630 2024-09-16T14:44:50.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 1 -- 172.21.15.189:0/3257570399 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f29e8001c30 con 0x7f29ec131630 2024-09-16T14:44:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 4 set_mon_vals no callback set 2024-09-16T14:44:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f0e98640 10 monclient: dump: 2024-09-16T14:44:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:50.271 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 10 monclient: shutdown 2024-09-16T14:44:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 1 -- 172.21.15.189:0/3257570399 >> v1:172.21.15.26:6789/0 conn(0x7f29ec131630 legacy=0x7f29ec131a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 1 -- 172.21.15.189:0/3257570399 shutdown_connections 2024-09-16T14:44:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 1 -- 172.21.15.189:0/3257570399 >> 172.21.15.189:0/3257570399 conn(0x7f29ec0a3380 msgr2=0x7f29ec09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 1 -- 172.21.15.189:0/3257570399 shutdown_connections 2024-09-16T14:44:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 1 -- 172.21.15.189:0/3257570399 wait complete. 2024-09-16T14:44:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 10 monclient: build_initial_monmap 2024-09-16T14:44:50.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 10 monclient: monmap: 2024-09-16T14:44:50.272 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:50.272 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:50.272 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:50.272 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:50.272 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:50.272 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:50.272 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:50.272 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:50.272 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:50.272 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:50.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 1 Processor -- start 2024-09-16T14:44:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 1 -- start start 2024-09-16T14:44:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.257+0000 7f29f40fe640 10 monclient: init 2024-09-16T14:44:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:44:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 10 monclient(hunting): picked mon.noname-a con 0x7f29ec09e950 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 10 monclient(hunting): picked mon.noname-c con 0x7f29ec09edd0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 10 monclient(hunting): picked mon.noname-b con 0x7f29ec131630 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec06b320 con 0x7f29ec09e950 2024-09-16T14:44:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec124170 con 0x7f29ec131630 2024-09-16T14:44:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec09af90 con 0x7f29ec09edd0 2024-09-16T14:44:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 10 monclient(hunting): authenticate will time out at 1072.376253s 2024-09-16T14:44:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f269b640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f29ec131630 legacy=0x7f29ec1d19c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:50.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f1699640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f29ec09edd0 0x7f29ec120ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:46712/0 (socket says 172.21.15.189:46712) 2024-09-16T14:44:50.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f1699640 1 -- 172.21.15.189:0/1856618529 learned_addr learned my addr 172.21.15.189:0/1856618529 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:50.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770733285 0 0) 0x7f29ec06b320 con 0x7f29ec09e950 2024-09-16T14:44:50.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:50.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:50.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:50.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient(hunting): my global_id is 84324 2024-09-16T14:44:50.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29d0003530 con 0x7f29ec09e950 2024-09-16T14:44:50.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687097749 0 0) 0x7f29ec09af90 con 0x7f29ec09edd0 2024-09-16T14:44:50.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:50.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:50.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:50.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient(hunting): my global_id is 4937 2024-09-16T14:44:50.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29ec06b320 con 0x7f29ec09edd0 2024-09-16T14:44:50.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2461073016 0 0) 0x7f29d0003530 con 0x7f29ec09e950 2024-09-16T14:44:50.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29ec09af90 con 0x7f29ec09e950 2024-09-16T14:44:50.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f29e8001b30 con 0x7f29ec09e950 2024-09-16T14:44:50.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:50.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:50.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient(hunting): dump: 2024-09-16T14:44:50.275 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:50.276 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:50.276 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:50.276 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:50.276 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:50.276 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:50.276 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:50.276 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:50.276 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:50.276 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:50.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:50.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 670900711 0 0) 0x7f29ec06b320 con 0x7f29ec09edd0 2024-09-16T14:44:50.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29d0003530 con 0x7f29ec09edd0 2024-09-16T14:44:50.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f29dc0041a0 con 0x7f29ec09edd0 2024-09-16T14:44:50.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:50.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:50.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient(hunting): dump: 2024-09-16T14:44:50.277 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:50.277 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:50.277 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:50.277 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:50.277 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:50.277 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:50.277 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:50.277 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:50.277 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:50.278 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:50.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3720239012 0 0) 0x7f29ec09af90 con 0x7f29ec09e950 2024-09-16T14:44:50.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 >> v1:172.21.15.26:6790/0 conn(0x7f29ec09edd0 legacy=0x7f29ec120ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:50.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 >> v1:172.21.15.189:6789/0 conn(0x7f29ec131630 legacy=0x7f29ec1d19c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:50.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient: _finish_hunting 0 2024-09-16T14:44:50.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 monclient: found mon.a 2024-09-16T14:44:50.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:50.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:50.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec130440 con 0x7f29ec09e950 2024-09-16T14:44:50.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient: _finish_auth 0 2024-09-16T14:44:50.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient: _check_auth_tickets 2024-09-16T14:44:50.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:50.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 5 monclient: authenticate success, global_id 84324 2024-09-16T14:44:50.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 10 monclient: _renew_subs 2024-09-16T14:44:50.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:50.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 1 -- 172.21.15.189:0/1856618529 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ec124790 con 0x7f29ec09e950 2024-09-16T14:44:50.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 10 monclient: _renew_subs 2024-09-16T14:44:50.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:50.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 1 -- 172.21.15.189:0/1856618529 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ec130f60 con 0x7f29ec09e950 2024-09-16T14:44:50.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29e8004c00 con 0x7f29ec09e950 2024-09-16T14:44:50.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:50.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f29e8004f10 con 0x7f29ec09e950 2024-09-16T14:44:50.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:50.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:50.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 10 monclient: dump: 2024-09-16T14:44:50.280 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:50.280 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:50.280 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:50.280 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:50.280 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:50.280 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:50.280 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:50.280 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:50.280 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:50.280 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:50.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e3fff640 4 set_mon_vals no callback set 2024-09-16T14:44:50.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:50.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:50.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:50.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f40fe640 1 -- 172.21.15.189:0/1856618529 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d4005050 con 0x7f29ec09e950 2024-09-16T14:44:50.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f29e800f980 con 0x7f29ec09e950 2024-09-16T14:44:50.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(51..51 src has 15..51) v4 ==== 4219+0+0 (unknown 780363716 0 0) 0x7f29e80434a0 con 0x7f29ec09e950 2024-09-16T14:44:50.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.261+0000 7f29f1699640 10 monclient: get_auth_request con 0x7f29d0036250 auth_method 0 2024-09-16T14:44:50.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.273+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f29e8014650 con 0x7f29ec09e950 2024-09-16T14:44:50.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.277+0000 7f29e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:50.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.277+0000 7f29e27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:50.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.433+0000 7f29f40fe640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T14:44:50.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.433+0000 7f29f40fe640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T14:44:50.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.433+0000 7f29f40fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:50.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:50.433+0000 7f29f40fe640 1 -- 172.21.15.189:0/1856618529 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"} v 0) v1 -- 0x7f29d4005300 con 0x7f29ec09e950 2024-09-16T14:44:52.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.033+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' created v53) v1 ==== 177+0+0 (unknown 3855716379 0 0) 0x7f29e8014470 con 0x7f29ec09e950 2024-09-16T14:44:52.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.033+0000 7f29e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T14:44:52.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.033+0000 7f29e27fc640 10 monclient: _finish_command 2 = system:0 pool 'ec-foo' created 2024-09-16T14:44:52.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.129+0000 7f29f40fe640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T14:44:52.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.129+0000 7f29f40fe640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T14:44:52.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.129+0000 7f29f40fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:52.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.129+0000 7f29f40fe640 1 -- 172.21.15.189:0/1856618529 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"} v 0) v1 -- 0x7f29d4000c00 con 0x7f29ec09e950 2024-09-16T14:44:52.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.133+0000 7f29e27fc640 1 -- 172.21.15.189:0/1856618529 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' already exists v53) v1 ==== 184+0+0 (unknown 3739705128 0 0) 0x7f29e80151b0 con 0x7f29ec09e950 2024-09-16T14:44:52.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.133+0000 7f29e27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T14:44:52.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.133+0000 7f29e27fc640 10 monclient: _finish_command 3 = system:0 pool 'ec-foo' already exists 2024-09-16T14:44:52.134 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'ec-foo' already exists 2024-09-16T14:44:52.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.133+0000 7f29f40fe640 1 -- 172.21.15.189:0/1856618529 >> v1:172.21.15.189:6812/3196161582 conn(0x7f29d0036250 legacy=0x7f29d0038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:52.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.133+0000 7f29f40fe640 10 monclient: shutdown 2024-09-16T14:44:52.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.133+0000 7f29f40fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:52.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.133+0000 7f29f40fe640 1 -- 172.21.15.189:0/1856618529 >> v1:172.21.15.26:6789/0 conn(0x7f29ec09e950 legacy=0x7f29ec1207b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:52.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.133+0000 7f29f40fe640 1 -- 172.21.15.189:0/1856618529 shutdown_connections 2024-09-16T14:44:52.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.133+0000 7f29f40fe640 1 -- 172.21.15.189:0/1856618529 >> 172.21.15.189:0/1856618529 conn(0x7f29ec0a3380 msgr2=0x7f29ec09d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:52.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.133+0000 7f29f40fe640 1 -- 172.21.15.189:0/1856618529 shutdown_connections 2024-09-16T14:44:52.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.133+0000 7f29f40fe640 1 -- 172.21.15.189:0/1856618529 wait complete. 2024-09-16T14:44:52.151 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool rm ec-foo ec-foo --yes-i-really-really-mean-it 2024-09-16T14:44:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9661915640 10 monclient: get_monmap_and_config 2024-09-16T14:44:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9661915640 10 monclient: build_initial_monmap 2024-09-16T14:44:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9661915640 10 monclient: monmap: 2024-09-16T14:44:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:52.271174+0000 2024-09-16T14:44:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:52.271174+0000 2024-09-16T14:44:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:52.272 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9661915640 1 Processor -- start 2024-09-16T14:44:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9661915640 1 -- start start 2024-09-16T14:44:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9661915640 10 monclient: init 2024-09-16T14:44:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9661915640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9661915640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:44:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9661915640 10 monclient(hunting): picked mon.noname-c con 0x7f965c099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9661915640 10 monclient(hunting): picked mon.noname-b con 0x7f965c09c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9661915640 10 monclient(hunting): picked mon.noname-a con 0x7f965c09e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9661915640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965c099e50 con 0x7f965c09e6d0 2024-09-16T14:44:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9661915640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965c09dbd0 con 0x7f965c099a70 2024-09-16T14:44:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9661915640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965c09ee20 con 0x7f965c09c550 2024-09-16T14:44:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9661915640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9661915640 10 monclient(hunting): authenticate will time out at 1074.386805s 2024-09-16T14:44:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f965affd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f965c099a70 legacy=0x7f965c09dfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f965b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f965c09e6d0 0x7f965c13b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:54092/0 (socket says 172.21.15.189:54092) 2024-09-16T14:44:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f965b7fe640 1 -- 172.21.15.189:0/3033761876 learned_addr learned my addr 172.21.15.189:0/3033761876 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9659ffb640 1 -- 172.21.15.189:0/3033761876 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151004202 0 0) 0x7f965c099e50 con 0x7f965c09e6d0 2024-09-16T14:44:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9659ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9659ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9659ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9659ffb640 10 monclient(hunting): my global_id is 84327 2024-09-16T14:44:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9659ffb640 1 -- 172.21.15.189:0/3033761876 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9644003530 con 0x7f965c09e6d0 2024-09-16T14:44:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9659ffb640 1 -- 172.21.15.189:0/3033761876 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226594353 0 0) 0x7f965c09ee20 con 0x7f965c09c550 2024-09-16T14:44:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9659ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9659ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9659ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9659ffb640 10 monclient(hunting): my global_id is 4940 2024-09-16T14:44:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.269+0000 7f9659ffb640 1 -- 172.21.15.189:0/3033761876 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f965c099e50 con 0x7f965c09c550 2024-09-16T14:44:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 1 -- 172.21.15.189:0/3033761876 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1239849650 0 0) 0x7f9644003530 con 0x7f965c09e6d0 2024-09-16T14:44:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 1 -- 172.21.15.189:0/3033761876 >> v1:172.21.15.26:6790/0 conn(0x7f965c09c550 legacy=0x7f965c1379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 1 -- 172.21.15.189:0/3033761876 >> v1:172.21.15.189:6789/0 conn(0x7f965c099a70 legacy=0x7f965c09dfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:44:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 1 monclient: found mon.noname-a 2024-09-16T14:44:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 1 -- 172.21.15.189:0/3033761876 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c09f100 con 0x7f965c09e6d0 2024-09-16T14:44:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 10 monclient: _finish_auth 0 2024-09-16T14:44:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:44:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 1 -- 172.21.15.189:0/3033761876 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f96540019a0 con 0x7f965c09e6d0 2024-09-16T14:44:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 5 monclient: authenticate success, global_id 84327 2024-09-16T14:44:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 10 monclient: dump: 2024-09-16T14:44:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:52.279 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:52.279 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:52.279 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:52.279 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:52.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:52.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 1 -- 172.21.15.189:0/3033761876 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9654001bc0 con 0x7f965c09e6d0 2024-09-16T14:44:52.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:52.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:52.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 4 set_mon_vals no callback set 2024-09-16T14:44:52.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 1 -- 172.21.15.189:0/3033761876 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9654001f10 con 0x7f965c09e6d0 2024-09-16T14:44:52.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:52.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:52.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9659ffb640 10 monclient: dump: 2024-09-16T14:44:52.280 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:52.280 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:52.280 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:52.280 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:52.280 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:52.280 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:52.280 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:52.280 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:52.280 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:52.280 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:52.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 10 monclient: shutdown 2024-09-16T14:44:52.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:52.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 1 -- 172.21.15.189:0/3033761876 >> v1:172.21.15.26:6789/0 conn(0x7f965c09e6d0 legacy=0x7f965c13b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:52.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 1 -- 172.21.15.189:0/3033761876 shutdown_connections 2024-09-16T14:44:52.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 1 -- 172.21.15.189:0/3033761876 >> 172.21.15.189:0/3033761876 conn(0x7f965c0a3380 msgr2=0x7f965c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:52.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 1 -- 172.21.15.189:0/3033761876 shutdown_connections 2024-09-16T14:44:52.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 1 -- 172.21.15.189:0/3033761876 wait complete. 2024-09-16T14:44:52.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 10 monclient: build_initial_monmap 2024-09-16T14:44:52.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 10 monclient: monmap: 2024-09-16T14:44:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:52.286 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 1 Processor -- start 2024-09-16T14:44:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 1 -- start start 2024-09-16T14:44:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 10 monclient: init 2024-09-16T14:44:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:44:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 10 monclient(hunting): picked mon.noname-c con 0x7f965c099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 10 monclient(hunting): picked mon.noname-a con 0x7f965c09c550 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 10 monclient(hunting): picked mon.noname-b con 0x7f965c09e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965c09cf20 con 0x7f965c09c550 2024-09-16T14:44:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965c06cce0 con 0x7f965c09e6d0 2024-09-16T14:44:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965c06b320 con 0x7f965c099a70 2024-09-16T14:44:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9661915640 10 monclient(hunting): authenticate will time out at 1074.390884s 2024-09-16T14:44:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f965b7fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f965c09e6d0 legacy=0x7f965c1d0e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f965affd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f965c099a70 0x7f965c131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:46732/0 (socket says 172.21.15.189:46732) 2024-09-16T14:44:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f965affd640 1 -- 172.21.15.189:0/1049585180 learned_addr learned my addr 172.21.15.189:0/1049585180 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219557506 0 0) 0x7f965c09cf20 con 0x7f965c09c550 2024-09-16T14:44:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9643fff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9643fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9643fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9643fff640 10 monclient(hunting): my global_id is 84330 2024-09-16T14:44:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9648003530 con 0x7f965c09c550 2024-09-16T14:44:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293842057 0 0) 0x7f965c06b320 con 0x7f965c099a70 2024-09-16T14:44:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9643fff640 10 monclient(hunting): _negotiate 2024-09-16T14:44:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9643fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9643fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9643fff640 10 monclient(hunting): my global_id is 4943 2024-09-16T14:44:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.273+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f965c09cf20 con 0x7f965c099a70 2024-09-16T14:44:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3012780444 0 0) 0x7f9648003530 con 0x7f965c09c550 2024-09-16T14:44:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f965c06b320 con 0x7f965c09c550 2024-09-16T14:44:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1813981715 0 0) 0x7f965c09cf20 con 0x7f965c099a70 2024-09-16T14:44:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9648003530 con 0x7f965c099a70 2024-09-16T14:44:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f964c0019e0 con 0x7f965c09c550 2024-09-16T14:44:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 10 monclient(hunting): dump: 2024-09-16T14:44:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:52.290 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:52.290 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:52.290 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:52.290 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:52.290 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:52.290 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:52.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:52.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f96500040e0 con 0x7f965c099a70 2024-09-16T14:44:52.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:52.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:52.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 10 monclient(hunting): dump: 2024-09-16T14:44:52.291 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:52.291 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:52.291 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:52.291 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:52.291 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:52.291 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:52.291 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:52.291 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:52.291 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:52.291 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:52.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3488203447 0 0) 0x7f965c06b320 con 0x7f965c09c550 2024-09-16T14:44:52.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 >> v1:172.21.15.26:6790/0 conn(0x7f965c099a70 legacy=0x7f965c131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:52.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 >> v1:172.21.15.189:6789/0 conn(0x7f965c09e6d0 legacy=0x7f965c1d0e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:52.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 10 monclient: _finish_hunting 0 2024-09-16T14:44:52.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 1 monclient: found mon.a 2024-09-16T14:44:52.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:52.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:52.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c13a400 con 0x7f965c09c550 2024-09-16T14:44:52.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 10 monclient: _finish_auth 0 2024-09-16T14:44:52.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 10 monclient: _check_auth_tickets 2024-09-16T14:44:52.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:52.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9661915640 5 monclient: authenticate success, global_id 84330 2024-09-16T14:44:52.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9661915640 10 monclient: _renew_subs 2024-09-16T14:44:52.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9661915640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:52.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9661915640 1 -- 172.21.15.189:0/1049585180 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f965c138ee0 con 0x7f965c09c550 2024-09-16T14:44:52.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9661915640 10 monclient: _renew_subs 2024-09-16T14:44:52.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9661915640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:52.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9661915640 1 -- 172.21.15.189:0/1049585180 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f965c139e40 con 0x7f965c09c550 2024-09-16T14:44:52.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f964c003600 con 0x7f965c09c550 2024-09-16T14:44:52.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:52.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f964c003910 con 0x7f965c09c550 2024-09-16T14:44:52.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:52.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9658ff9640 4 set_mon_vals no callback set 2024-09-16T14:44:52.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:52.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 10 monclient: dump: 2024-09-16T14:44:52.294 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:52.294 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:52.294 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:52.294 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:52.294 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:52.294 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:52.294 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:52.295 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:52.295 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:52.295 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:52.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9661915640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:52.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9661915640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:52.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9661915640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:52.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9661915640 1 -- 172.21.15.189:0/1049585180 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f963c005050 con 0x7f965c09c550 2024-09-16T14:44:52.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f964c00e380 con 0x7f965c09c550 2024-09-16T14:44:52.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(53..53 src has 15..53) v4 ==== 4660+0+0 (unknown 1422159602 0 0) 0x7f964c0422f0 con 0x7f965c09c550 2024-09-16T14:44:52.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.277+0000 7f965affd640 10 monclient: get_auth_request con 0x7f96480362a0 auth_method 0 2024-09-16T14:44:52.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.289+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f964c013220 con 0x7f965c09c550 2024-09-16T14:44:52.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.289+0000 7f9643fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:52.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.289+0000 7f9643fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:52.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.449+0000 7f9661915640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:52.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.449+0000 7f9661915640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:52.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.449+0000 7f9661915640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:52.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:52.449+0000 7f9661915640 1 -- 172.21.15.189:0/1049585180 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f963c005300 con 0x7f965c09c550 2024-09-16T14:44:53.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.049+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}]=0 pool 'ec-foo' removed v54) v1 ==== 154+0+0 (unknown 2288915467 0 0) 0x7f964c013040 con 0x7f965c09c550 2024-09-16T14:44:53.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.049+0000 7f9643fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:53.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.049+0000 7f9643fff640 10 monclient: _finish_command 2 = system:0 pool 'ec-foo' removed 2024-09-16T14:44:53.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.145+0000 7f9661915640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:53.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.145+0000 7f9661915640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:53.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.145+0000 7f9661915640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:53.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.145+0000 7f9661915640 1 -- 172.21.15.189:0/1049585180 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f963c001150 con 0x7f965c09c550 2024-09-16T14:44:53.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.145+0000 7f9643fff640 1 -- 172.21.15.189:0/1049585180 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}]=0 pool 'ec-foo' does not exist v54) v1 ==== 161+0+0 (unknown 2379256009 0 0) 0x7f964c013d80 con 0x7f965c09c550 2024-09-16T14:44:53.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.145+0000 7f9643fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T14:44:53.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.145+0000 7f9643fff640 10 monclient: _finish_command 3 = system:0 pool 'ec-foo' does not exist 2024-09-16T14:44:53.150 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'ec-foo' does not exist 2024-09-16T14:44:53.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.149+0000 7f9661915640 1 -- 172.21.15.189:0/1049585180 >> v1:172.21.15.189:6812/3196161582 conn(0x7f96480362a0 legacy=0x7f9648038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:53.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.149+0000 7f9661915640 10 monclient: shutdown 2024-09-16T14:44:53.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.149+0000 7f9661915640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:53.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.149+0000 7f9661915640 1 -- 172.21.15.189:0/1049585180 >> v1:172.21.15.26:6789/0 conn(0x7f965c09c550 legacy=0x7f965c1cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:53.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.149+0000 7f9661915640 1 -- 172.21.15.189:0/1049585180 shutdown_connections 2024-09-16T14:44:53.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.149+0000 7f9661915640 1 -- 172.21.15.189:0/1049585180 >> 172.21.15.189:0/1049585180 conn(0x7f965c0a3380 msgr2=0x7f965c0a13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:53.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.149+0000 7f9661915640 1 -- 172.21.15.189:0/1049585180 shutdown_connections 2024-09-16T14:44:53.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.149+0000 7f9661915640 1 -- 172.21.15.189:0/1049585180 wait complete. 2024-09-16T14:44:53.166 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep foo 2024-09-16T14:44:53.166 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls 2024-09-16T14:44:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f280155a640 10 monclient: get_monmap_and_config 2024-09-16T14:44:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f280155a640 10 monclient: build_initial_monmap 2024-09-16T14:44:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f280155a640 10 monclient: monmap: 2024-09-16T14:44:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:53.284313+0000 2024-09-16T14:44:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:53.284313+0000 2024-09-16T14:44:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:53.287 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f280155a640 1 Processor -- start 2024-09-16T14:44:53.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f280155a640 1 -- start start 2024-09-16T14:44:53.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f280155a640 10 monclient: init 2024-09-16T14:44:53.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f280155a640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:53.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f280155a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:53.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f280155a640 10 monclient(hunting): picked mon.noname-c con 0x7f27fc09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:53.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f280155a640 10 monclient(hunting): picked mon.noname-a con 0x7f27fc09e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:53.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f280155a640 10 monclient(hunting): picked mon.noname-b con 0x7f27fc09e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:53.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f280155a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc09ee20 con 0x7f27fc09e2f0 2024-09-16T14:44:53.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f280155a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc09f100 con 0x7f27fc09df10 2024-09-16T14:44:53.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f280155a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc09f3e0 con 0x7f27fc09e740 2024-09-16T14:44:53.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f280155a640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:53.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f280155a640 10 monclient(hunting): authenticate will time out at 1075.399948s 2024-09-16T14:44:53.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.281+0000 7f27faffd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f27fc09df10 legacy=0x7f27fc09d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:53.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27fa7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f27fc09e2f0 0x7f27fc1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:54100/0 (socket says 172.21.15.189:54100) 2024-09-16T14:44:53.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27fa7fc640 1 -- 172.21.15.189:0/893973441 learned_addr learned my addr 172.21.15.189:0/893973441 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:53.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 1 -- 172.21.15.189:0/893973441 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411346709 0 0) 0x7f27fc09ee20 con 0x7f27fc09e2f0 2024-09-16T14:44:53.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:53.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:53.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:53.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient(hunting): my global_id is 84333 2024-09-16T14:44:53.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 1 -- 172.21.15.189:0/893973441 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e4003530 con 0x7f27fc09e2f0 2024-09-16T14:44:53.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 1 -- 172.21.15.189:0/893973441 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300626015 0 0) 0x7f27fc09f3e0 con 0x7f27fc09e740 2024-09-16T14:44:53.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:44:53.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:53.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:53.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient(hunting): my global_id is 4946 2024-09-16T14:44:53.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 1 -- 172.21.15.189:0/893973441 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27fc09ee20 con 0x7f27fc09e740 2024-09-16T14:44:53.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 1 -- 172.21.15.189:0/893973441 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2202272064 0 0) 0x7f27e4003530 con 0x7f27fc09e2f0 2024-09-16T14:44:53.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 1 -- 172.21.15.189:0/893973441 >> v1:172.21.15.26:6790/0 conn(0x7f27fc09e740 legacy=0x7f27fc13b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:53.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 1 -- 172.21.15.189:0/893973441 >> v1:172.21.15.189:6789/0 conn(0x7f27fc09df10 legacy=0x7f27fc09d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:53.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:44:53.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 1 monclient: found mon.noname-a 2024-09-16T14:44:53.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:53.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:53.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 1 -- 172.21.15.189:0/893973441 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc09c690 con 0x7f27fc09e2f0 2024-09-16T14:44:53.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient: _finish_auth 0 2024-09-16T14:44:53.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:44:53.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:53.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 1 -- 172.21.15.189:0/893973441 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f27f0001620 con 0x7f27fc09e2f0 2024-09-16T14:44:53.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:53.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:53.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient: dump: 2024-09-16T14:44:53.302 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:53.303 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:53.303 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.303 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.303 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:53.303 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:53.303 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:53.303 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:53.303 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:53.303 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:53.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f280155a640 5 monclient: authenticate success, global_id 84333 2024-09-16T14:44:53.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f280155a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:53.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 1 -- 172.21.15.189:0/893973441 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27f0001840 con 0x7f27fc09e2f0 2024-09-16T14:44:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 1 -- 172.21.15.189:0/893973441 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f27f0001c10 con 0x7f27fc09e2f0 2024-09-16T14:44:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f27f9ffb640 10 monclient: dump: 2024-09-16T14:44:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:53.305 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:53.305 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:53.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f280155a640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:53.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f280155a640 4 set_mon_vals no callback set 2024-09-16T14:44:53.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f280155a640 10 monclient: shutdown 2024-09-16T14:44:53.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f280155a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:53.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f280155a640 1 -- 172.21.15.189:0/893973441 >> v1:172.21.15.26:6789/0 conn(0x7f27fc09e2f0 legacy=0x7f27fc1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:53.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f280155a640 1 -- 172.21.15.189:0/893973441 shutdown_connections 2024-09-16T14:44:53.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f280155a640 1 -- 172.21.15.189:0/893973441 >> 172.21.15.189:0/893973441 conn(0x7f27fc0a3320 msgr2=0x7f27fc09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:53.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f280155a640 1 -- 172.21.15.189:0/893973441 shutdown_connections 2024-09-16T14:44:53.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f280155a640 1 -- 172.21.15.189:0/893973441 wait complete. 2024-09-16T14:44:53.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f280155a640 10 monclient: build_initial_monmap 2024-09-16T14:44:53.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f280155a640 10 monclient: monmap: 2024-09-16T14:44:53.306 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:53.306 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:53.306 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:53.306 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:53.306 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:53.306 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:53.306 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:53.307 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f280155a640 1 Processor -- start 2024-09-16T14:44:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f280155a640 1 -- start start 2024-09-16T14:44:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.285+0000 7f280155a640 10 monclient: init 2024-09-16T14:44:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 10 monclient(hunting): picked mon.noname-b con 0x7f27fc09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 10 monclient(hunting): picked mon.noname-a con 0x7f27fc09e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 10 monclient(hunting): picked mon.noname-c con 0x7f27fc09e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc06b380 con 0x7f27fc09e2f0 2024-09-16T14:44:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc128480 con 0x7f27fc09df10 2024-09-16T14:44:53.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27faffd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f27fc09df10 legacy=0x7f27fc11c340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:53.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc1206a0 con 0x7f27fc09e740 2024-09-16T14:44:53.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:53.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 10 monclient(hunting): authenticate will time out at 1075.404320s 2024-09-16T14:44:53.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27fa7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f27fc09e2f0 0x7f27fc11ca50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:54108/0 (socket says 172.21.15.189:54108) 2024-09-16T14:44:53.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27fa7fc640 1 -- 172.21.15.189:0/1052078646 learned_addr learned my addr 172.21.15.189:0/1052078646 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:53.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429897248 0 0) 0x7f27fc06b380 con 0x7f27fc09e2f0 2024-09-16T14:44:53.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:53.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:53.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:53.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient(hunting): my global_id is 84336 2024-09-16T14:44:53.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e8003530 con 0x7f27fc09e2f0 2024-09-16T14:44:53.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485744652 0 0) 0x7f27fc1206a0 con 0x7f27fc09e740 2024-09-16T14:44:53.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:44:53.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:53.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:53.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient(hunting): my global_id is 4949 2024-09-16T14:44:53.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27fc06b380 con 0x7f27fc09e740 2024-09-16T14:44:53.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2912646084 0 0) 0x7f27e8003530 con 0x7f27fc09e2f0 2024-09-16T14:44:53.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27fc1206a0 con 0x7f27fc09e2f0 2024-09-16T14:44:53.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f27f0001b90 con 0x7f27fc09e2f0 2024-09-16T14:44:53.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:53.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:53.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient(hunting): dump: 2024-09-16T14:44:53.310 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:53.311 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:53.311 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.311 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.311 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:53.311 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:53.311 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:53.311 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:53.311 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:53.311 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:53.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:53.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 385281533 0 0) 0x7f27fc06b380 con 0x7f27fc09e740 2024-09-16T14:44:53.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27e8003530 con 0x7f27fc09e740 2024-09-16T14:44:53.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f27f40019c0 con 0x7f27fc09e740 2024-09-16T14:44:53.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:53.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:53.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient(hunting): dump: 2024-09-16T14:44:53.312 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:53.312 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:53.312 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.312 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.312 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:53.312 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:53.313 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:53.313 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:53.313 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:53.313 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:53.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 130119098 0 0) 0x7f27fc1206a0 con 0x7f27fc09e2f0 2024-09-16T14:44:53.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 >> v1:172.21.15.26:6790/0 conn(0x7f27fc09e740 legacy=0x7f27fc1cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:53.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 >> v1:172.21.15.189:6789/0 conn(0x7f27fc09df10 legacy=0x7f27fc11c340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:53.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:44:53.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 monclient: found mon.a 2024-09-16T14:44:53.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:53.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:53.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc120280 con 0x7f27fc09e2f0 2024-09-16T14:44:53.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient: _finish_auth 0 2024-09-16T14:44:53.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:44:53.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:53.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 5 monclient: authenticate success, global_id 84336 2024-09-16T14:44:53.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 10 monclient: _renew_subs 2024-09-16T14:44:53.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:53.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 1 -- 172.21.15.189:0/1052078646 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27fc06cca0 con 0x7f27fc09e2f0 2024-09-16T14:44:53.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 10 monclient: _renew_subs 2024-09-16T14:44:53.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:53.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 1 -- 172.21.15.189:0/1052078646 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27fc11d1c0 con 0x7f27fc09e2f0 2024-09-16T14:44:53.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27f0004aa0 con 0x7f27fc09e2f0 2024-09-16T14:44:53.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:53.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f27f0004db0 con 0x7f27fc09e2f0 2024-09-16T14:44:53.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27f97fa640 4 set_mon_vals no callback set 2024-09-16T14:44:53.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:53.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:53.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 10 monclient: dump: 2024-09-16T14:44:53.316 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:53.317 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:53.317 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.317 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.317 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:53.317 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:53.317 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:53.317 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:53.318 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:53.318 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:53.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:53.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:53.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:53.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f280155a640 1 -- 172.21.15.189:0/1052078646 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27dc005050 con 0x7f27fc09e2f0 2024-09-16T14:44:53.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f27f0004f90 con 0x7f27fc09e2f0 2024-09-16T14:44:53.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(54..54 src has 15..54) v4 ==== 4305+0+0 (unknown 1234261589 0 0) 0x7f27f00434e0 con 0x7f27fc09e2f0 2024-09-16T14:44:53.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.289+0000 7f27faffd640 10 monclient: get_auth_request con 0x7f27e8036250 auth_method 0 2024-09-16T14:44:53.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.301+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f27f00148a0 con 0x7f27fc09e2f0 2024-09-16T14:44:53.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.301+0000 7f27e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:53.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.301+0000 7f27e37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:53.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.457+0000 7f280155a640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T14:44:53.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.457+0000 7f280155a640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:44:53.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.457+0000 7f280155a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:53.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.457+0000 7f280155a640 1 -- 172.21.15.189:0/1052078646 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f27dc005300 con 0x7f27fc09e2f0 2024-09-16T14:44:53.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.457+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v54) v1 ==== 65+0+60 (unknown 940634683 0 1596542489) 0x7f27f00144b0 con 0x7f27fc09e2f0 2024-09-16T14:44:53.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.457+0000 7f27e37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:44:53.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.457+0000 7f27e37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:53.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.545+0000 7f280155a640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T14:44:53.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.545+0000 7f280155a640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:44:53.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.545+0000 7f280155a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:53.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.545+0000 7f280155a640 1 -- 172.21.15.189:0/1052078646 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f27dc005540 con 0x7f27fc09e2f0 2024-09-16T14:44:53.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.545+0000 7f27e37fe640 1 -- 172.21.15.189:0/1052078646 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v54) v1 ==== 65+0+60 (unknown 940634683 0 1596542489) 0x7f27f00144b0 con 0x7f27fc09e2f0 2024-09-16T14:44:53.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.545+0000 7f27e37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:44:53.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.545+0000 7f27e37fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:44:53.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.549+0000 7f280155a640 1 -- 172.21.15.189:0/1052078646 >> v1:172.21.15.189:6812/3196161582 conn(0x7f27e8036250 legacy=0x7f27e8038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:53.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.549+0000 7f280155a640 10 monclient: shutdown 2024-09-16T14:44:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.549+0000 7f280155a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.549+0000 7f280155a640 1 -- 172.21.15.189:0/1052078646 >> v1:172.21.15.26:6789/0 conn(0x7f27fc09e2f0 legacy=0x7f27fc11ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.549+0000 7f280155a640 1 -- 172.21.15.189:0/1052078646 shutdown_connections 2024-09-16T14:44:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.549+0000 7f280155a640 1 -- 172.21.15.189:0/1052078646 >> 172.21.15.189:0/1052078646 conn(0x7f27fc0a3320 msgr2=0x7f27fc12a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.549+0000 7f280155a640 1 -- 172.21.15.189:0/1052078646 shutdown_connections 2024-09-16T14:44:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.549+0000 7f280155a640 1 -- 172.21.15.189:0/1052078646 wait complete. 2024-09-16T14:44:53.565 INFO:tasks.workunit.client.0.smithi189.stdout:foo 2024-09-16T14:44:53.565 INFO:tasks.workunit.client.0.smithi189.stdout:foo-ssd 2024-09-16T14:44:53.565 INFO:tasks.workunit.client.0.smithi189.stdout:foo-hdd 2024-09-16T14:44:53.565 INFO:tasks.workunit.client.0.smithi189.stdout:ec-foo 2024-09-16T14:44:53.565 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rename foo foo-asdf 2024-09-16T14:44:53.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.673+0000 7f1b37bfa640 10 monclient: get_monmap_and_config 2024-09-16T14:44:53.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.673+0000 7f1b37bfa640 10 monclient: build_initial_monmap 2024-09-16T14:44:53.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.673+0000 7f1b37bfa640 10 monclient: monmap: 2024-09-16T14:44:53.678 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:53.678 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:53.678 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:53.677309+0000 2024-09-16T14:44:53.678 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:53.677309+0000 2024-09-16T14:44:53.678 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:53.678 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:53.678 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:53.678 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:53.678 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:53.678 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:53.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 1 Processor -- start 2024-09-16T14:44:53.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 1 -- start start 2024-09-16T14:44:53.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 10 monclient: init 2024-09-16T14:44:53.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:53.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:53.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 10 monclient(hunting): picked mon.noname-a con 0x7f1b30131630 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:53.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 10 monclient(hunting): picked mon.noname-c con 0x7f1b3009edd0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:53.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 10 monclient(hunting): picked mon.noname-b con 0x7f1b3009e950 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:53.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3009b920 con 0x7f1b30131630 2024-09-16T14:44:53.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3009bc00 con 0x7f1b3009edd0 2024-09-16T14:44:53.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3009cba0 con 0x7f1b3009e950 2024-09-16T14:44:53.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:53.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 10 monclient(hunting): authenticate will time out at 1075.792958s 2024-09-16T14:44:53.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b35195640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f1b3009edd0 legacy=0x7f1b30136b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:53.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b35996640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1b30131630 0x7f1b30131a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:54116/0 (socket says 172.21.15.189:54116) 2024-09-16T14:44:53.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b35996640 1 -- 172.21.15.189:0/1436856068 learned_addr learned my addr 172.21.15.189:0/1436856068 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:53.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b36197640 1 --1- 172.21.15.189:0/1436856068 >> v1:172.21.15.26:6790/0 conn(0x7f1b3009e950 0x7f1b3013a290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:46764/0 (socket says 172.21.15.189:46764) 2024-09-16T14:44:53.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 1 -- 172.21.15.189:0/1436856068 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584842749 0 0) 0x7f1b3009b920 con 0x7f1b30131630 2024-09-16T14:44:53.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient(hunting): _negotiate 2024-09-16T14:44:53.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:53.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:53.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient(hunting): my global_id is 84339 2024-09-16T14:44:53.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 1 -- 172.21.15.189:0/1436856068 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b1c003530 con 0x7f1b30131630 2024-09-16T14:44:53.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 1 -- 172.21.15.189:0/1436856068 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968615260 0 0) 0x7f1b3009cba0 con 0x7f1b3009e950 2024-09-16T14:44:53.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient(hunting): _negotiate 2024-09-16T14:44:53.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:53.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:53.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient(hunting): my global_id is 4952 2024-09-16T14:44:53.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 1 -- 172.21.15.189:0/1436856068 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b3009b920 con 0x7f1b3009e950 2024-09-16T14:44:53.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 1 -- 172.21.15.189:0/1436856068 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3910483049 0 0) 0x7f1b3009b920 con 0x7f1b3009e950 2024-09-16T14:44:53.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 1 -- 172.21.15.189:0/1436856068 >> v1:172.21.15.189:6789/0 conn(0x7f1b3009edd0 legacy=0x7f1b30136b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:53.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 1 -- 172.21.15.189:0/1436856068 >> v1:172.21.15.26:6789/0 conn(0x7f1b30131630 legacy=0x7f1b30131a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:53.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient: _finish_hunting 0 2024-09-16T14:44:53.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 1 monclient: found mon.noname-b 2024-09-16T14:44:53.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:53.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:44:53.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 1 -- 172.21.15.189:0/1436856068 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3009b420 con 0x7f1b3009e950 2024-09-16T14:44:53.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient: _finish_auth 0 2024-09-16T14:44:53.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient: _check_auth_tickets 2024-09-16T14:44:53.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:53.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 1 -- 172.21.15.189:0/1436856068 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b28001980 con 0x7f1b3009e950 2024-09-16T14:44:53.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:53.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:44:53.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient: dump: 2024-09-16T14:44:53.693 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:53.693 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:53.693 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.693 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.693 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:53.693 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:53.694 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:53.694 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:53.694 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:53.694 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:53.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 5 monclient: authenticate success, global_id 4952 2024-09-16T14:44:53.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:53.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 1 -- 172.21.15.189:0/1436856068 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b28001ca0 con 0x7f1b3009e950 2024-09-16T14:44:53.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:53.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 1 -- 172.21.15.189:0/1436856068 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b28002070 con 0x7f1b3009e950 2024-09-16T14:44:53.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:53.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:53.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b34994640 10 monclient: dump: 2024-09-16T14:44:53.696 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:53.697 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:53.697 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.697 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.697 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:53.697 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:53.697 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:53.697 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:53.697 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:53.697 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:53.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:53.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 4 set_mon_vals no callback set 2024-09-16T14:44:53.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 10 monclient: shutdown 2024-09-16T14:44:53.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:53.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 1 -- 172.21.15.189:0/1436856068 >> v1:172.21.15.26:6790/0 conn(0x7f1b3009e950 legacy=0x7f1b3013a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:53.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 1 -- 172.21.15.189:0/1436856068 shutdown_connections 2024-09-16T14:44:53.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 1 -- 172.21.15.189:0/1436856068 >> 172.21.15.189:0/1436856068 conn(0x7f1b300a3380 msgr2=0x7f1b3009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:53.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 1 -- 172.21.15.189:0/1436856068 shutdown_connections 2024-09-16T14:44:53.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 1 -- 172.21.15.189:0/1436856068 wait complete. 2024-09-16T14:44:53.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 10 monclient: build_initial_monmap 2024-09-16T14:44:53.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.677+0000 7f1b37bfa640 10 monclient: monmap: 2024-09-16T14:44:53.698 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:53.698 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:53.698 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:53.698 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:53.698 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:53.699 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:53.699 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:53.699 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:53.699 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:53.699 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:53.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 1 Processor -- start 2024-09-16T14:44:53.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 1 -- start start 2024-09-16T14:44:53.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 10 monclient: init 2024-09-16T14:44:53.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:53.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:44:53.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 10 monclient(hunting): picked mon.noname-b con 0x7f1b3009e950 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:53.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 10 monclient(hunting): picked mon.noname-a con 0x7f1b3009edd0 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:53.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 10 monclient(hunting): picked mon.noname-c con 0x7f1b30131630 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:53.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3011f800 con 0x7f1b3009edd0 2024-09-16T14:44:53.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b35996640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f1b3009e950 legacy=0x7f1b3011bab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:53.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b300a1e60 con 0x7f1b3009e950 2024-09-16T14:44:53.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b300a1930 con 0x7f1b30131630 2024-09-16T14:44:53.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:53.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 10 monclient(hunting): authenticate will time out at 1075.797098s 2024-09-16T14:44:53.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b35195640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1b3009edd0 0x7f1b3011c1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:54130/0 (socket says 172.21.15.189:54130) 2024-09-16T14:44:53.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b35195640 1 -- 172.21.15.189:0/1450403252 learned_addr learned my addr 172.21.15.189:0/1450403252 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:53.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b36197640 1 --1- 172.21.15.189:0/1450403252 >> v1:172.21.15.26:6790/0 conn(0x7f1b30131630 0x7f1b301ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:46772/0 (socket says 172.21.15.189:46772) 2024-09-16T14:44:53.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636010512 0 0) 0x7f1b3011f800 con 0x7f1b3009edd0 2024-09-16T14:44:53.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:53.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:53.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:53.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient(hunting): my global_id is 84342 2024-09-16T14:44:53.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b14005740 con 0x7f1b3009edd0 2024-09-16T14:44:53.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941951715 0 0) 0x7f1b300a1930 con 0x7f1b30131630 2024-09-16T14:44:53.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient(hunting): _negotiate 2024-09-16T14:44:53.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:53.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:53.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient(hunting): my global_id is 4955 2024-09-16T14:44:53.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b3011f800 con 0x7f1b30131630 2024-09-16T14:44:53.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1518306762 0 0) 0x7f1b14005740 con 0x7f1b3009edd0 2024-09-16T14:44:53.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b300a1930 con 0x7f1b3009edd0 2024-09-16T14:44:53.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b20004160 con 0x7f1b3009edd0 2024-09-16T14:44:53.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:53.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:53.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient(hunting): dump: 2024-09-16T14:44:53.702 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:53.702 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:53.702 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.702 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.702 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:53.702 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:53.702 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:53.703 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:53.703 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:53.703 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:53.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:53.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3308514108 0 0) 0x7f1b3011f800 con 0x7f1b30131630 2024-09-16T14:44:53.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b14005740 con 0x7f1b30131630 2024-09-16T14:44:53.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b28001fb0 con 0x7f1b30131630 2024-09-16T14:44:53.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:53.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:53.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient(hunting): dump: 2024-09-16T14:44:53.703 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:53.703 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:53.703 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.704 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.704 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:53.704 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:53.704 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:53.704 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:53.705 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:53.705 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:53.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1667019728 0 0) 0x7f1b300a1930 con 0x7f1b3009edd0 2024-09-16T14:44:53.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 >> v1:172.21.15.26:6790/0 conn(0x7f1b30131630 legacy=0x7f1b301ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:53.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 >> v1:172.21.15.189:6789/0 conn(0x7f1b3009e950 legacy=0x7f1b3011bab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:53.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient: _finish_hunting 0 2024-09-16T14:44:53.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 monclient: found mon.a 2024-09-16T14:44:53.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:53.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:53.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3011f520 con 0x7f1b3009edd0 2024-09-16T14:44:53.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient: _finish_auth 0 2024-09-16T14:44:53.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient: _check_auth_tickets 2024-09-16T14:44:53.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:53.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 5 monclient: authenticate success, global_id 84342 2024-09-16T14:44:53.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 10 monclient: _renew_subs 2024-09-16T14:44:53.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:53.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 1 -- 172.21.15.189:0/1450403252 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3011ddf0 con 0x7f1b3009edd0 2024-09-16T14:44:53.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 10 monclient: _renew_subs 2024-09-16T14:44:53.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:53.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 1 -- 172.21.15.189:0/1450403252 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b3011c8d0 con 0x7f1b3009edd0 2024-09-16T14:44:53.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b20004380 con 0x7f1b3009edd0 2024-09-16T14:44:53.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:53.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b20004710 con 0x7f1b3009edd0 2024-09-16T14:44:53.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b27fff640 4 set_mon_vals no callback set 2024-09-16T14:44:53.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:53.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:53.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 10 monclient: dump: 2024-09-16T14:44:53.707 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:53.707 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:53.707 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.707 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:53.707 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:53.707 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:53.707 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:53.707 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:53.707 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:53.707 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:53.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:53.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:53.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:53.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b37bfa640 1 -- 172.21.15.189:0/1450403252 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b18005050 con 0x7f1b3009edd0 2024-09-16T14:44:53.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.681+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1b200048f0 con 0x7f1b3009edd0 2024-09-16T14:44:53.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.685+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(54..54 src has 15..54) v4 ==== 4305+0+0 (unknown 1234261589 0 0) 0x7f1b200433a0 con 0x7f1b3009edd0 2024-09-16T14:44:53.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.685+0000 7f1b35996640 10 monclient: get_auth_request con 0x7f1b14038460 auth_method 0 2024-09-16T14:44:53.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.697+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1b20014430 con 0x7f1b3009edd0 2024-09-16T14:44:53.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.697+0000 7f1b267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:53.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.697+0000 7f1b267fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:53.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.845+0000 7f1b37bfa640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T14:44:53.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.845+0000 7f1b37bfa640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T14:44:53.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.845+0000 7f1b37bfa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:53.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:53.845+0000 7f1b37bfa640 1 -- 172.21.15.189:0/1450403252 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7f1b18005300 con 0x7f1b3009edd0 2024-09-16T14:44:54.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.049+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 v55) v1 ==== 110+0+0 (unknown 2755101563 0 0) 0x7f1b20012ad0 con 0x7f1b3009edd0 2024-09-16T14:44:54.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.049+0000 7f1b267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T14:44:54.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.049+0000 7f1b267fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:44:54.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.141+0000 7f1b37bfa640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T14:44:54.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.141+0000 7f1b37bfa640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T14:44:54.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.141+0000 7f1b37bfa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:44:54.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.141+0000 7f1b37bfa640 1 -- 172.21.15.189:0/1450403252 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7f1b18001130 con 0x7f1b3009edd0 2024-09-16T14:44:54.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.145+0000 7f1b267fc640 1 -- 172.21.15.189:0/1450403252 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v55) v1 ==== 139+0+0 (unknown 3897356291 0 0) 0x7f1b20014250 con 0x7f1b3009edd0 2024-09-16T14:44:54.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.145+0000 7f1b267fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T14:44:54.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.145+0000 7f1b267fc640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo-asdf' 2024-09-16T14:44:54.146 INFO:tasks.workunit.client.0.smithi189.stderr:already renamed to 'foo-asdf' 2024-09-16T14:44:54.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.145+0000 7f1b37bfa640 1 -- 172.21.15.189:0/1450403252 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1b14038460 legacy=0x7f1b1403a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:54.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.145+0000 7f1b37bfa640 10 monclient: shutdown 2024-09-16T14:44:54.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.145+0000 7f1b37bfa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:54.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.145+0000 7f1b37bfa640 1 -- 172.21.15.189:0/1450403252 >> v1:172.21.15.26:6789/0 conn(0x7f1b3009edd0 legacy=0x7f1b3011c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:54.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.145+0000 7f1b37bfa640 1 -- 172.21.15.189:0/1450403252 shutdown_connections 2024-09-16T14:44:54.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.145+0000 7f1b37bfa640 1 -- 172.21.15.189:0/1450403252 >> 172.21.15.189:0/1450403252 conn(0x7f1b300a3380 msgr2=0x7f1b3009a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:54.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.145+0000 7f1b37bfa640 1 -- 172.21.15.189:0/1450403252 shutdown_connections 2024-09-16T14:44:54.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.145+0000 7f1b37bfa640 1 -- 172.21.15.189:0/1450403252 wait complete. 2024-09-16T14:44:54.161 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rename foo foo-asdf 2024-09-16T14:44:54.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572c98e640 10 monclient: get_monmap_and_config 2024-09-16T14:44:54.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572c98e640 10 monclient: build_initial_monmap 2024-09-16T14:44:54.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572c98e640 10 monclient: monmap: 2024-09-16T14:44:54.306 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:54.306 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:54.306 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:44:54.267126+0000 2024-09-16T14:44:54.306 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:44:54.267126+0000 2024-09-16T14:44:54.307 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:54.307 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:54.307 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:54.307 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:44:54.307 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:44:54.307 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:54.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572c98e640 1 Processor -- start 2024-09-16T14:44:54.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572c98e640 1 -- start start 2024-09-16T14:44:54.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572c98e640 10 monclient: init 2024-09-16T14:44:54.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572c98e640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:54.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572c98e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:44:54.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572c98e640 10 monclient(hunting): picked mon.noname-c con 0x7f5728131630 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:54.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572c98e640 10 monclient(hunting): picked mon.noname-b con 0x7f572809edd0 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:54.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572c98e640 10 monclient(hunting): picked mon.noname-a con 0x7f572809e950 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:54.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572c98e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572809b920 con 0x7f572809e950 2024-09-16T14:44:54.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572c98e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572809bc00 con 0x7f5728131630 2024-09-16T14:44:54.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572c98e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572809cba0 con 0x7f572809edd0 2024-09-16T14:44:54.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572c98e640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:54.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572659c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f5728131630 legacy=0x7f5728131a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:54.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f572c98e640 10 monclient(hunting): authenticate will time out at 1076.383138s 2024-09-16T14:44:54.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f5726d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f572809e950 0x7f572813a290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:54142/0 (socket says 172.21.15.189:54142) 2024-09-16T14:44:54.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.265+0000 7f5726d9d640 1 -- 172.21.15.189:0/274259095 learned_addr learned my addr 172.21.15.189:0/274259095 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:54.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 1 -- 172.21.15.189:0/274259095 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92576371 0 0) 0x7f572809b920 con 0x7f572809e950 2024-09-16T14:44:54.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient(hunting): _negotiate 2024-09-16T14:44:54.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:54.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:54.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient(hunting): my global_id is 84345 2024-09-16T14:44:54.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 1 -- 172.21.15.189:0/274259095 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5710003530 con 0x7f572809e950 2024-09-16T14:44:54.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 1 -- 172.21.15.189:0/274259095 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039133601 0 0) 0x7f572809cba0 con 0x7f572809edd0 2024-09-16T14:44:54.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient(hunting): _negotiate 2024-09-16T14:44:54.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:54.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:54.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient(hunting): my global_id is 4958 2024-09-16T14:44:54.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 1 -- 172.21.15.189:0/274259095 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f572809b920 con 0x7f572809edd0 2024-09-16T14:44:54.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 1 -- 172.21.15.189:0/274259095 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2784115527 0 0) 0x7f5710003530 con 0x7f572809e950 2024-09-16T14:44:54.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 1 -- 172.21.15.189:0/274259095 >> v1:172.21.15.26:6790/0 conn(0x7f572809edd0 legacy=0x7f5728136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:54.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 1 -- 172.21.15.189:0/274259095 >> v1:172.21.15.189:6789/0 conn(0x7f5728131630 legacy=0x7f5728131a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:44:54.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient: _finish_hunting 0 2024-09-16T14:44:54.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 1 monclient: found mon.noname-a 2024-09-16T14:44:54.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:54.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:44:54.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 1 -- 172.21.15.189:0/274259095 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572809b420 con 0x7f572809e950 2024-09-16T14:44:54.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient: _finish_auth 0 2024-09-16T14:44:54.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient: _check_auth_tickets 2024-09-16T14:44:54.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:54.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 5 monclient: authenticate success, global_id 84345 2024-09-16T14:44:54.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:54.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 1 -- 172.21.15.189:0/274259095 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f57200019a0 con 0x7f572809e950 2024-09-16T14:44:54.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:54.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:44:54.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient: dump: 2024-09-16T14:44:54.320 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:54.320 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:54.320 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:54.320 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:54.320 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:54.320 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:54.321 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:54.321 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:54.321 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:54.321 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:54.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:44:54.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 1 -- 172.21.15.189:0/274259095 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5720001bc0 con 0x7f572809e950 2024-09-16T14:44:54.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:54.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 1 -- 172.21.15.189:0/274259095 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5720001f90 con 0x7f572809e950 2024-09-16T14:44:54.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:54.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:54.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572559a640 10 monclient: dump: 2024-09-16T14:44:54.322 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:54.322 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:54.322 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:54.322 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:54.322 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:54.322 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:54.322 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:54.322 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:54.322 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:54.323 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:54.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 10 monclient: get_monmap_and_config success 2024-09-16T14:44:54.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 4 set_mon_vals no callback set 2024-09-16T14:44:54.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 10 monclient: shutdown 2024-09-16T14:44:54.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:44:54.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 1 -- 172.21.15.189:0/274259095 >> v1:172.21.15.26:6789/0 conn(0x7f572809e950 legacy=0x7f572813a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:54.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 1 -- 172.21.15.189:0/274259095 shutdown_connections 2024-09-16T14:44:54.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 1 -- 172.21.15.189:0/274259095 >> 172.21.15.189:0/274259095 conn(0x7f57280a3380 msgr2=0x7f572809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:44:54.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 1 -- 172.21.15.189:0/274259095 shutdown_connections 2024-09-16T14:44:54.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 1 -- 172.21.15.189:0/274259095 wait complete. 2024-09-16T14:44:54.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 10 monclient: build_initial_monmap 2024-09-16T14:44:54.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 10 monclient: monmap: 2024-09-16T14:44:54.324 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:44:54.324 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:44:54.324 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:44:54.324 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:44:54.324 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:44:54.324 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:44:54.324 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:44:54.324 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:44:54.324 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:44:54.324 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:54.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 1 Processor -- start 2024-09-16T14:44:54.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 1 -- start start 2024-09-16T14:44:54.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 10 monclient: init 2024-09-16T14:44:54.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 10 monclient: _reopen_session rank -1 2024-09-16T14:44:54.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:44:54.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 10 monclient(hunting): picked mon.noname-a con 0x7f572809e950 addr v1:172.21.15.26:6789/0 2024-09-16T14:44:54.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 10 monclient(hunting): picked mon.noname-b con 0x7f572809edd0 addr v1:172.21.15.189:6789/0 2024-09-16T14:44:54.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 10 monclient(hunting): picked mon.noname-c con 0x7f5728131630 addr v1:172.21.15.26:6790/0 2024-09-16T14:44:54.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57280a1a30 con 0x7f572809e950 2024-09-16T14:44:54.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572811cd30 con 0x7f572809edd0 2024-09-16T14:44:54.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57280a1fd0 con 0x7f5728131630 2024-09-16T14:44:54.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 10 monclient(hunting): _renew_subs 2024-09-16T14:44:54.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f5725d9b640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f572809edd0 legacy=0x7f572811bd10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:44:54.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572c98e640 10 monclient(hunting): authenticate will time out at 1076.387057s 2024-09-16T14:44:54.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f572809e950 0x7f572811b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:54146/0 (socket says 172.21.15.189:54146) 2024-09-16T14:44:54.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f572659c640 1 -- 172.21.15.189:0/4122245945 learned_addr learned my addr 172.21.15.189:0/4122245945 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:44:54.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446367018 0 0) 0x7f57280a1a30 con 0x7f572809e950 2024-09-16T14:44:54.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f5716ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:54.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f5716ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:54.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f5716ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:54.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f5716ffd640 10 monclient(hunting): my global_id is 84348 2024-09-16T14:44:54.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5708003530 con 0x7f572809e950 2024-09-16T14:44:54.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524017731 0 0) 0x7f572809b420 con 0x7f5728131630 2024-09-16T14:44:54.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f5716ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:44:54.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f5716ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:44:54.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.269+0000 7f5716ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:44:54.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 10 monclient(hunting): my global_id is 4961 2024-09-16T14:44:54.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57280a1a30 con 0x7f5728131630 2024-09-16T14:44:54.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 996630799 0 0) 0x7f57280a1a30 con 0x7f5728131630 2024-09-16T14:44:54.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f572809b420 con 0x7f5728131630 2024-09-16T14:44:54.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5720001d30 con 0x7f5728131630 2024-09-16T14:44:54.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:54.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:44:54.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 10 monclient(hunting): dump: 2024-09-16T14:44:54.327 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:54.327 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:54.327 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:54.328 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:54.328 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:54.328 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:54.328 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:54.328 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:54.328 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:54.328 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:54.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:44:54.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2637571765 0 0) 0x7f5708003530 con 0x7f572809e950 2024-09-16T14:44:54.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57280a1a30 con 0x7f572809e950 2024-09-16T14:44:54.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f57180040e0 con 0x7f572809e950 2024-09-16T14:44:54.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:54.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:44:54.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 10 monclient(hunting): dump: 2024-09-16T14:44:54.329 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:54.329 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:54.329 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:54.329 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:54.329 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:54.329 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:54.329 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:54.329 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:54.329 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:54.329 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:54.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3316296450 0 0) 0x7f572809b420 con 0x7f5728131630 2024-09-16T14:44:54.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 >> v1:172.21.15.189:6789/0 conn(0x7f572809edd0 legacy=0x7f572811bd10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:44:54.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 >> v1:172.21.15.26:6789/0 conn(0x7f572809e950 legacy=0x7f572811b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:44:54.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:44:54.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 1 monclient: found mon.c 2024-09-16T14:44:54.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:54.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:54.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572811d280 con 0x7f5728131630 2024-09-16T14:44:54.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 10 monclient: _finish_auth 0 2024-09-16T14:44:54.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:44:54.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:54.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f572c98e640 5 monclient: authenticate success, global_id 4961 2024-09-16T14:44:54.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f572c98e640 10 monclient: _renew_subs 2024-09-16T14:44:54.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f572c98e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:54.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f572c98e640 1 -- 172.21.15.189:0/4122245945 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572806b320 con 0x7f5728131630 2024-09-16T14:44:54.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f572c98e640 10 monclient: _renew_subs 2024-09-16T14:44:54.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f572c98e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:54.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f572c98e640 1 -- 172.21.15.189:0/4122245945 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f572811c420 con 0x7f5728131630 2024-09-16T14:44:54.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5720001fc0 con 0x7f5728131630 2024-09-16T14:44:54.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:44:54.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5720004fc0 con 0x7f5728131630 2024-09-16T14:44:54.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:44:54.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:44:54.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5724d99640 4 set_mon_vals no callback set 2024-09-16T14:44:54.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 10 monclient: dump: 2024-09-16T14:44:54.332 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:44:54.332 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:44:54.332 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:54.332 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:44:54.332 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:44:54.332 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:44:54.332 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:44:54.332 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:44:54.332 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:44:54.333 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:44:54.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f572c98e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:54.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f572c98e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:54.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f572c98e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:54.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f572c98e640 1 -- 172.21.15.189:0/4122245945 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570c005050 con 0x7f5728131630 2024-09-16T14:44:54.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f572000fa30 con 0x7f5728131630 2024-09-16T14:44:54.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(55..55 src has 15..55) v4 ==== 4310+0+0 (unknown 545726176 0 0) 0x7f5720043610 con 0x7f5728131630 2024-09-16T14:44:54.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.273+0000 7f572659c640 10 monclient: get_auth_request con 0x7f5708036270 auth_method 0 2024-09-16T14:44:54.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.285+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f57200147c0 con 0x7f5728131630 2024-09-16T14:44:54.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.285+0000 7f5716ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:44:54.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.285+0000 7f5716ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:44:54.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.433+0000 7f572c98e640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T14:44:54.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.433+0000 7f572c98e640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T14:44:54.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.433+0000 7f572c98e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:54.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.433+0000 7f572c98e640 1 -- 172.21.15.189:0/4122245945 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7f570c005300 con 0x7f5728131630 2024-09-16T14:44:54.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.437+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v55) v1 ==== 139+0+0 (unknown 3897356291 0 0) 0x7f5720012e60 con 0x7f5728131630 2024-09-16T14:44:54.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.437+0000 7f5716ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T14:44:54.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.437+0000 7f5716ffd640 10 monclient: _finish_command 2 = system:0 already renamed to 'foo-asdf' 2024-09-16T14:44:54.495 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:44:54.495 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T14:44:54.495 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T14:44:54.496 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T14:44:54.498 INFO:tasks.ceph.mon.b:Started 2024-09-16T14:44:54.498 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T14:44:54.499 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T14:44:54.499 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T14:44:54.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.525+0000 7f572c98e640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T14:44:54.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.525+0000 7f572c98e640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T14:44:54.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.525+0000 7f572c98e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:44:54.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:54.525+0000 7f572c98e640 1 -- 172.21.15.189:0/4122245945 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7f570c001170 con 0x7f5728131630 2024-09-16T14:44:57.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:57.269+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:44:57.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:57.269+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:44:57.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:57.269+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:57.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:57.269+0000 7f5715ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:44:58.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:58.269+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:44:58.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:58.269+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:44:58.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:58.269+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:44:59.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:59.269+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:44:59.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:59.269+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:44:59.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:44:59.269+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:00.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:00.273+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:00.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:00.273+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:00.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:00.273+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:01.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:01.273+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:01.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:01.273+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:01.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:01.273+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:02.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:02.269+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:02.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:02.269+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:02.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:02.269+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:03.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:03.273+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:03.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:03.273+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:03.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:03.273+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:04.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:04.273+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:04.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:04.273+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:04.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:04.273+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:05.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:05.273+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:05.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:05.273+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:05.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:05.273+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:06.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:06.273+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:06.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:06.273+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:06.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:06.273+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:07.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:07.273+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:07.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:07.273+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:07.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:07.273+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:07.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:07.273+0000 7f5715ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:08.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:08.273+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:08.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:08.273+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:08.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:08.273+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:09.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:09.273+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:09.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:09.273+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:09.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:09.273+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:09.499 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T14:45:09.500 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T14:45:09.500 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:45:09.500 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:45:09.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a8360640 10 monclient: get_monmap_and_config 2024-09-16T14:45:09.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a8360640 10 monclient: build_initial_monmap 2024-09-16T14:45:09.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a8360640 10 monclient: monmap: 2024-09-16T14:45:09.632 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:09.632 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:09.632 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:09.631146+0000 2024-09-16T14:45:09.632 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:09.631146+0000 2024-09-16T14:45:09.632 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:09.633 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:09.633 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:09.633 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:09.633 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:09.633 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:09.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a8360640 1 Processor -- start 2024-09-16T14:45:09.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a8360640 1 -- start start 2024-09-16T14:45:09.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a8360640 10 monclient: init 2024-09-16T14:45:09.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a8360640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:09.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a8360640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:09.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a8360640 10 monclient(hunting): picked mon.noname-a con 0x7f68a009a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:09.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a8360640 10 monclient(hunting): picked mon.noname-b con 0x7f68a009d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:09.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a8360640 10 monclient(hunting): picked mon.noname-c con 0x7f68a009b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:09.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a8360640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a009dc60 con 0x7f68a009a550 2024-09-16T14:45:09.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a8360640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a0099ae0 con 0x7f68a009b040 2024-09-16T14:45:09.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a8360640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a009b8f0 con 0x7f68a009d1b0 2024-09-16T14:45:09.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a8360640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:09.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a8360640 10 monclient(hunting): authenticate will time out at 1133.751165s 2024-09-16T14:45:09.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a60fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f68a009a550 0x7f68a009a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:46666/0 (socket says 172.21.15.26:46666) 2024-09-16T14:45:09.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a60fc640 1 -- 172.21.15.26:0/2776980927 learned_addr learned my addr 172.21.15.26:0/2776980927 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:09.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a50fa640 1 -- 172.21.15.26:0/2776980927 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123471881 0 0) 0x7f68a009dc60 con 0x7f68a009a550 2024-09-16T14:45:09.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a50fa640 10 monclient(hunting): _negotiate 2024-09-16T14:45:09.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a50fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:09.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a50fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:09.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a50fa640 10 monclient(hunting): my global_id is 84351 2024-09-16T14:45:09.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a50fa640 1 -- 172.21.15.26:0/2776980927 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f688c003530 con 0x7f68a009a550 2024-09-16T14:45:09.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a50fa640 1 -- 172.21.15.26:0/2776980927 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2635811048 0 0) 0x7f688c003530 con 0x7f68a009a550 2024-09-16T14:45:09.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a50fa640 1 -- 172.21.15.26:0/2776980927 >> v1:172.21.15.26:6790/0 conn(0x7f68a009d1b0 legacy=0x7f68a0136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:09.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.629+0000 7f68a50fa640 1 -- 172.21.15.26:0/2776980927 >> v1:172.21.15.189:6789/0 conn(0x7f68a009b040 legacy=0x7f68a013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:09.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 10 monclient: _finish_hunting 0 2024-09-16T14:45:09.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 1 monclient: found mon.noname-a 2024-09-16T14:45:09.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:09.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:45:09.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 1 -- 172.21.15.26:0/2776980927 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a009bbd0 con 0x7f68a009a550 2024-09-16T14:45:09.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 10 monclient: _finish_auth 0 2024-09-16T14:45:09.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 10 monclient: _check_auth_tickets 2024-09-16T14:45:09.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:09.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 1 -- 172.21.15.26:0/2776980927 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f689c001620 con 0x7f68a009a550 2024-09-16T14:45:09.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:09.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:09.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 10 monclient: dump: 2024-09-16T14:45:09.643 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:09.643 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:09.643 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:09.643 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:09.643 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:09.643 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:09.643 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:09.643 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:09.643 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:09.643 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:09.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 5 monclient: authenticate success, global_id 84351 2024-09-16T14:45:09.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:09.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 1 -- 172.21.15.26:0/2776980927 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f689c0018d0 con 0x7f68a009a550 2024-09-16T14:45:09.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:09.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 1 -- 172.21.15.26:0/2776980927 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f689c001ca0 con 0x7f68a009a550 2024-09-16T14:45:09.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:09.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:09.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a50fa640 10 monclient: dump: 2024-09-16T14:45:09.644 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:09.644 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:09.644 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:09.644 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:09.644 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:09.645 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:09.645 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:09.645 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:09.645 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:09.645 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:09.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:09.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 4 set_mon_vals no callback set 2024-09-16T14:45:09.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 10 monclient: shutdown 2024-09-16T14:45:09.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:09.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 1 -- 172.21.15.26:0/2776980927 >> v1:172.21.15.26:6789/0 conn(0x7f68a009a550 legacy=0x7f68a009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:09.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 1 -- 172.21.15.26:0/2776980927 shutdown_connections 2024-09-16T14:45:09.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 1 -- 172.21.15.26:0/2776980927 >> 172.21.15.26:0/2776980927 conn(0x7f68a009fb50 msgr2=0x7f68a009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:09.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 1 -- 172.21.15.26:0/2776980927 shutdown_connections 2024-09-16T14:45:09.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 1 -- 172.21.15.26:0/2776980927 wait complete. 2024-09-16T14:45:09.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 10 monclient: build_initial_monmap 2024-09-16T14:45:09.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 10 monclient: monmap: 2024-09-16T14:45:09.646 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:09.647 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:09.647 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:09.647 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:09.647 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:09.647 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:09.647 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:09.648 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:09.648 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:09.648 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:09.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 1 Processor -- start 2024-09-16T14:45:09.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 1 -- start start 2024-09-16T14:45:09.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 10 monclient: init 2024-09-16T14:45:09.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:09.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:09.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 10 monclient(hunting): picked mon.noname-b con 0x7f68a009a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:09.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 10 monclient(hunting): picked mon.noname-c con 0x7f68a009b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:09.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 10 monclient(hunting): picked mon.noname-a con 0x7f68a009d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:09.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a006cac0 con 0x7f68a009d1b0 2024-09-16T14:45:09.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a011dd90 con 0x7f68a009a550 2024-09-16T14:45:09.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a011e070 con 0x7f68a009b040 2024-09-16T14:45:09.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:09.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 10 monclient(hunting): authenticate will time out at 1133.754454s 2024-09-16T14:45:09.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a58fb640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f68a009b040 0x7f68a01ceb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59892/0 (socket says 172.21.15.26:59892) 2024-09-16T14:45:09.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a58fb640 1 -- 172.21.15.26:0/65400475 learned_addr learned my addr 172.21.15.26:0/65400475 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:09.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676844720 0 0) 0x7f68a006cac0 con 0x7f68a009d1b0 2024-09-16T14:45:09.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:09.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:09.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:09.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient(hunting): my global_id is 84354 2024-09-16T14:45:09.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6884003530 con 0x7f68a009d1b0 2024-09-16T14:45:09.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 136227534 0 0) 0x7f6884003530 con 0x7f68a009d1b0 2024-09-16T14:45:09.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68a006cac0 con 0x7f68a009d1b0 2024-09-16T14:45:09.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249882937 0 0) 0x7f68a011dd90 con 0x7f68a009a550 2024-09-16T14:45:09.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:09.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:09.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:09.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient(hunting): my global_id is 94102 2024-09-16T14:45:09.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6884003530 con 0x7f68a009a550 2024-09-16T14:45:09.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6898001b80 con 0x7f68a009d1b0 2024-09-16T14:45:09.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:09.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:09.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient(hunting): dump: 2024-09-16T14:45:09.651 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:09.651 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:09.651 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:09.651 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:09.651 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:09.651 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:09.651 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:09.651 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:09.651 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:09.652 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4138568391 0 0) 0x7f68a006cac0 con 0x7f68a009d1b0 2024-09-16T14:45:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 >> v1:172.21.15.26:6790/0 conn(0x7f68a009b040 legacy=0x7f68a01ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 >> v1:172.21.15.189:6789/0 conn(0x7f68a009a550 legacy=0x7f68a01321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 1 monclient: found mon.a 2024-09-16T14:45:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a009a240 con 0x7f68a009d1b0 2024-09-16T14:45:09.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:09.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:09.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:09.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 5 monclient: authenticate success, global_id 84354 2024-09-16T14:45:09.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 10 monclient: _renew_subs 2024-09-16T14:45:09.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:09.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 1 -- 172.21.15.26:0/65400475 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a0070aa0 con 0x7f68a009d1b0 2024-09-16T14:45:09.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 10 monclient: _renew_subs 2024-09-16T14:45:09.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:09.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a8360640 1 -- 172.21.15.26:0/65400475 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a011dab0 con 0x7f68a009d1b0 2024-09-16T14:45:09.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6898001db0 con 0x7f68a009d1b0 2024-09-16T14:45:09.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:09.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6898003b40 con 0x7f68a009d1b0 2024-09-16T14:45:09.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:09.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:09.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f6896ffd640 10 monclient: dump: 2024-09-16T14:45:09.654 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:09.654 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:09.654 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:09.654 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:09.654 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:09.654 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:09.654 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:09.654 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:09.655 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:09.655 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:09.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.633+0000 7f68a48f9640 4 set_mon_vals no callback set 2024-09-16T14:45:09.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.637+0000 7f68a8360640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:09.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.637+0000 7f68a8360640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:09.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.637+0000 7f68a8360640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:09.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.637+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6898003d20 con 0x7f68a009d1b0 2024-09-16T14:45:09.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.637+0000 7f68a8360640 1 -- 172.21.15.26:0/65400475 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6888005050 con 0x7f68a009d1b0 2024-09-16T14:45:09.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.637+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(55..55 src has 15..55) v4 ==== 4310+0+0 (unknown 545726176 0 0) 0x7f6898042520 con 0x7f68a009d1b0 2024-09-16T14:45:09.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.637+0000 7f68a60fc640 10 monclient: get_auth_request con 0x7f6884036240 auth_method 0 2024-09-16T14:45:09.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.641+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6898004570 con 0x7f68a009d1b0 2024-09-16T14:45:09.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.641+0000 7f6896ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:09.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.641+0000 7f6896ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:09.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.861+0000 7f68a8360640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:45:09.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.861+0000 7f68a8360640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:09.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.861+0000 7f68a8360640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:09.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.861+0000 7f68a8360640 1 -- 172.21.15.26:0/65400475 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6888005880 con 0x7f68a009d1b0 2024-09-16T14:45:09.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.861+0000 7f6896ffd640 1 -- 172.21.15.26:0/65400475 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1853695393) 0x7f6898011d00 con 0x7f68a009d1b0 2024-09-16T14:45:09.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.861+0000 7f6896ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:09.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.861+0000 7f6896ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:09.868 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":28,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:45:09.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.865+0000 7f68a8360640 1 -- 172.21.15.26:0/65400475 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6884036240 legacy=0x7f6884038700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:09.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.865+0000 7f68a8360640 10 monclient: shutdown 2024-09-16T14:45:09.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.865+0000 7f68a8360640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:09.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.865+0000 7f68a8360640 1 -- 172.21.15.26:0/65400475 >> v1:172.21.15.26:6789/0 conn(0x7f68a009d1b0 legacy=0x7f68a01d2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:09.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.865+0000 7f68a8360640 1 -- 172.21.15.26:0/65400475 shutdown_connections 2024-09-16T14:45:09.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.865+0000 7f68a8360640 1 -- 172.21.15.26:0/65400475 >> 172.21.15.26:0/65400475 conn(0x7f68a009fb50 msgr2=0x7f68a009e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:09.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.865+0000 7f68a8360640 1 -- 172.21.15.26:0/65400475 shutdown_connections 2024-09-16T14:45:09.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:09.865+0000 7f68a8360640 1 -- 172.21.15.26:0/65400475 wait complete. 2024-09-16T14:45:10.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:10.273+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:10.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:10.273+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:10.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:10.273+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:11.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:11.273+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:11.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:11.273+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:11.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:11.273+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:12.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:12.272+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:12.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:12.272+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:12.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:12.272+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:12.884 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:45:13.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 10 monclient: get_monmap_and_config 2024-09-16T14:45:13.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 10 monclient: build_initial_monmap 2024-09-16T14:45:13.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 10 monclient: monmap: 2024-09-16T14:45:13.015 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:13.015 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:13.015 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:13.014818+0000 2024-09-16T14:45:13.015 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:13.014818+0000 2024-09-16T14:45:13.015 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:13.015 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:13.016 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:13.016 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:13.016 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:13.016 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 1 Processor -- start 2024-09-16T14:45:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 1 -- start start 2024-09-16T14:45:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 10 monclient: init 2024-09-16T14:45:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:45:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 10 monclient(hunting): picked mon.noname-a con 0x7f79f0134e40 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 10 monclient(hunting): picked mon.noname-c con 0x7f79f0099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 10 monclient(hunting): picked mon.noname-b con 0x7f79f0135220 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f0099e80 con 0x7f79f0134e40 2024-09-16T14:45:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f00719d0 con 0x7f79f0099a70 2024-09-16T14:45:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f009e1e0 con 0x7f79f0135220 2024-09-16T14:45:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 10 monclient(hunting): authenticate will time out at 1137.134913s 2024-09-16T14:45:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f4899640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f79f0135220 0x7f79f013db30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59896/0 (socket says 172.21.15.26:59896) 2024-09-16T14:45:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79effff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f79f0134e40 0x7f79f0070bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:46676/0 (socket says 172.21.15.26:46676) 2024-09-16T14:45:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f4899640 1 -- 172.21.15.26:0/1140113169 learned_addr learned my addr 172.21.15.26:0/1140113169 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 1 -- 172.21.15.26:0/1140113169 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274214569 0 0) 0x7f79e0001120 con 0x7f79f0134e40 2024-09-16T14:45:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 10 monclient(hunting): my global_id is 84357 2024-09-16T14:45:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 1 -- 172.21.15.26:0/1140113169 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79d0003530 con 0x7f79f0134e40 2024-09-16T14:45:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 1 -- 172.21.15.26:0/1140113169 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3012222340 0 0) 0x7f79d0003530 con 0x7f79f0134e40 2024-09-16T14:45:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 1 -- 172.21.15.26:0/1140113169 >> v1:172.21.15.26:6790/0 conn(0x7f79f0135220 legacy=0x7f79f013db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 1 -- 172.21.15.26:0/1140113169 >> v1:172.21.15.189:6789/0 conn(0x7f79f0099a70 legacy=0x7f79f00712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 1 monclient: found mon.noname-a 2024-09-16T14:45:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:45:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 1 -- 172.21.15.26:0/1140113169 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f009e7d0 con 0x7f79f0134e40 2024-09-16T14:45:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 1 -- 172.21.15.26:0/1140113169 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f79e0001610 con 0x7f79f0134e40 2024-09-16T14:45:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 10 monclient: dump: 2024-09-16T14:45:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:13.022 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 5 monclient: authenticate success, global_id 84357 2024-09-16T14:45:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79f62fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 1 -- 172.21.15.26:0/1140113169 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79e00018c0 con 0x7f79f0134e40 2024-09-16T14:45:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 1 -- 172.21.15.26:0/1140113169 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f79e0001c50 con 0x7f79f0134e40 2024-09-16T14:45:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.013+0000 7f79eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79eeffd640 10 monclient: dump: 2024-09-16T14:45:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:13.024 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 4 set_mon_vals no callback set 2024-09-16T14:45:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient: shutdown 2024-09-16T14:45:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 1 -- 172.21.15.26:0/1140113169 >> v1:172.21.15.26:6789/0 conn(0x7f79f0134e40 legacy=0x7f79f0070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 1 -- 172.21.15.26:0/1140113169 shutdown_connections 2024-09-16T14:45:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 1 -- 172.21.15.26:0/1140113169 >> 172.21.15.26:0/1140113169 conn(0x7f79f009fb50 msgr2=0x7f79f009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 1 -- 172.21.15.26:0/1140113169 shutdown_connections 2024-09-16T14:45:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 1 -- 172.21.15.26:0/1140113169 wait complete. 2024-09-16T14:45:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient: build_initial_monmap 2024-09-16T14:45:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient: monmap: 2024-09-16T14:45:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:13.026 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 1 Processor -- start 2024-09-16T14:45:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 1 -- start start 2024-09-16T14:45:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient: init 2024-09-16T14:45:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient(hunting): picked mon.noname-c con 0x7f79f0099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient(hunting): picked mon.noname-b con 0x7f79f0134e40 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient(hunting): picked mon.noname-a con 0x7f79f0135220 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f0124340 con 0x7f79f0135220 2024-09-16T14:45:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f0122e20 con 0x7f79f0134e40 2024-09-16T14:45:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f0134230 con 0x7f79f0099a70 2024-09-16T14:45:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient(hunting): authenticate will time out at 1137.138079s 2024-09-16T14:45:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f4899640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f79f0135220 0x7f79f01d22c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:46678/0 (socket says 172.21.15.26:46678) 2024-09-16T14:45:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79effff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f79f0099a70 0x7f79f01c9930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59900/0 (socket says 172.21.15.26:59900) 2024-09-16T14:45:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f4899640 1 -- 172.21.15.26:0/3389107546 learned_addr learned my addr 172.21.15.26:0/3389107546 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 1 -- 172.21.15.26:0/3389107546 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180991271 0 0) 0x7f79f0124340 con 0x7f79f0135220 2024-09-16T14:45:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 10 monclient(hunting): my global_id is 84360 2024-09-16T14:45:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 1 -- 172.21.15.26:0/3389107546 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79d4003530 con 0x7f79f0135220 2024-09-16T14:45:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 1 -- 172.21.15.26:0/3389107546 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3485966793 0 0) 0x7f79d4003530 con 0x7f79f0135220 2024-09-16T14:45:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 1 -- 172.21.15.26:0/3389107546 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79f0124340 con 0x7f79f0135220 2024-09-16T14:45:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 1 -- 172.21.15.26:0/3389107546 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f79e80017c0 con 0x7f79f0135220 2024-09-16T14:45:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 10 monclient(hunting): dump: 2024-09-16T14:45:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:13.031 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 1 -- 172.21.15.26:0/3389107546 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1067039299 0 0) 0x7f79f0124340 con 0x7f79f0135220 2024-09-16T14:45:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 1 -- 172.21.15.26:0/3389107546 >> v1:172.21.15.26:6790/0 conn(0x7f79f0099a70 legacy=0x7f79f01c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 1 -- 172.21.15.26:0/3389107546 >> v1:172.21.15.189:6789/0 conn(0x7f79f0134e40 legacy=0x7f79f01ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 10 monclient: _finish_hunting 0 2024-09-16T14:45:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 1 monclient: found mon.a 2024-09-16T14:45:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 1 -- 172.21.15.26:0/3389107546 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f009c210 con 0x7f79f0135220 2024-09-16T14:45:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 10 monclient: _finish_auth 0 2024-09-16T14:45:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 10 monclient: _check_auth_tickets 2024-09-16T14:45:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 5 monclient: authenticate success, global_id 84360 2024-09-16T14:45:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient: _renew_subs 2024-09-16T14:45:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 1 -- 172.21.15.26:0/3389107546 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f0124620 con 0x7f79f0135220 2024-09-16T14:45:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient: _renew_subs 2024-09-16T14:45:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 1 -- 172.21.15.26:0/3389107546 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f0121700 con 0x7f79f0135220 2024-09-16T14:45:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 1 -- 172.21.15.26:0/3389107546 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79e8001d30 con 0x7f79f0135220 2024-09-16T14:45:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 1 -- 172.21.15.26:0/3389107546 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f79e8004a10 con 0x7f79f0135220 2024-09-16T14:45:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ee7fc640 4 set_mon_vals no callback set 2024-09-16T14:45:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 10 monclient: dump: 2024-09-16T14:45:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:13.035 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79f62fc640 1 -- 172.21.15.26:0/3389107546 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c8005050 con 0x7f79f0135220 2024-09-16T14:45:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.017+0000 7f79ecff9640 1 -- 172.21.15.26:0/3389107546 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f79e8004bf0 con 0x7f79f0135220 2024-09-16T14:45:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.021+0000 7f79ecff9640 1 -- 172.21.15.26:0/3389107546 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(56..56 src has 15..56) v4 ==== 4310+0+0 (unknown 3213898037 0 0) 0x7f79e8043360 con 0x7f79f0135220 2024-09-16T14:45:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.021+0000 7f79effff640 10 monclient: get_auth_request con 0x7f79d4036030 auth_method 0 2024-09-16T14:45:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.025+0000 7f79ecff9640 1 -- 172.21.15.26:0/3389107546 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f79e80053f0 con 0x7f79f0135220 2024-09-16T14:45:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.025+0000 7f79ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.025+0000 7f79ecff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:13.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.237+0000 7f79f62fc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:45:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.237+0000 7f79f62fc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.237+0000 7f79f62fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.237+0000 7f79f62fc640 1 -- 172.21.15.26:0/3389107546 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f79c8005880 con 0x7f79f0135220 2024-09-16T14:45:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.237+0000 7f79ecff9640 1 -- 172.21.15.26:0/3389107546 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2254484671) 0x7f79e8012b40 con 0x7f79f0135220 2024-09-16T14:45:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.237+0000 7f79ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.237+0000 7f79ecff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:13.242 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":28,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:45:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.241+0000 7f79f62fc640 1 -- 172.21.15.26:0/3389107546 >> v1:172.21.15.189:6812/3196161582 conn(0x7f79d4036030 legacy=0x7f79d40384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.241+0000 7f79f62fc640 10 monclient: shutdown 2024-09-16T14:45:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.241+0000 7f79f62fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.241+0000 7f79f62fc640 1 -- 172.21.15.26:0/3389107546 >> v1:172.21.15.26:6789/0 conn(0x7f79f0135220 legacy=0x7f79f01d22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.241+0000 7f79f62fc640 1 -- 172.21.15.26:0/3389107546 shutdown_connections 2024-09-16T14:45:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.241+0000 7f79f62fc640 1 -- 172.21.15.26:0/3389107546 >> 172.21.15.26:0/3389107546 conn(0x7f79f009fb50 msgr2=0x7f79f00a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.241+0000 7f79f62fc640 1 -- 172.21.15.26:0/3389107546 shutdown_connections 2024-09-16T14:45:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:13.241+0000 7f79f62fc640 1 -- 172.21.15.26:0/3389107546 wait complete. 2024-09-16T14:45:13.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:13.272+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:13.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:13.272+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:13.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:13.272+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:14.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:14.272+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:14.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:14.272+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:14.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:14.272+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:15.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:15.272+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:15.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:15.272+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:15.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:15.272+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:16.258 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:45:16.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:16.272+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:16.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:16.272+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:16.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:16.272+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc7e1c640 10 monclient: get_monmap_and_config 2024-09-16T14:45:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc7e1c640 10 monclient: build_initial_monmap 2024-09-16T14:45:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc7e1c640 10 monclient: monmap: 2024-09-16T14:45:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:16.390694+0000 2024-09-16T14:45:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:16.390694+0000 2024-09-16T14:45:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:16.391 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:16.392 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:16.392 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:16.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc7e1c640 1 Processor -- start 2024-09-16T14:45:16.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc7e1c640 1 -- start start 2024-09-16T14:45:16.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc7e1c640 10 monclient: init 2024-09-16T14:45:16.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc7e1c640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:16.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc7e1c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:16.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc7e1c640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc009fe40 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:16.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc7e1c640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc00a0930 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc7e1c640 10 monclient(hunting): picked mon.noname-b con 0x7f4dc009b4a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc7e1c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc00993e0 con 0x7f4dc00a0930 2024-09-16T14:45:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc7e1c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc009ad20 con 0x7f4dc009fe40 2024-09-16T14:45:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc7e1c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc00a1920 con 0x7f4dc009b4a0 2024-09-16T14:45:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc7e1c640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc7e1c640 10 monclient(hunting): authenticate will time out at 1140.511094s 2024-09-16T14:45:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc53b7640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4dc00a0930 0x7f4dc0137b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:46680/0 (socket says 172.21.15.26:46680) 2024-09-16T14:45:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc53b7640 1 -- 172.21.15.26:0/1536132066 learned_addr learned my addr 172.21.15.26:0/1536132066 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc4bb6640 1 -- 172.21.15.26:0/1536132066 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728631219 0 0) 0x7f4dc009ad20 con 0x7f4dc009fe40 2024-09-16T14:45:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc4bb6640 10 monclient(hunting): _negotiate 2024-09-16T14:45:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc4bb6640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc4bb6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:16.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc4bb6640 10 monclient(hunting): my global_id is 94111 2024-09-16T14:45:16.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.389+0000 7f4dc4bb6640 1 -- 172.21.15.26:0/1536132066 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dac003530 con 0x7f4dc009fe40 2024-09-16T14:45:16.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 1 -- 172.21.15.26:0/1536132066 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 634491645 0 0) 0x7f4dac003530 con 0x7f4dc009fe40 2024-09-16T14:45:16.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 1 -- 172.21.15.26:0/1536132066 >> v1:172.21.15.26:6790/0 conn(0x7f4dc009b4a0 legacy=0x7f4dc013b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:16.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 1 -- 172.21.15.26:0/1536132066 >> v1:172.21.15.26:6789/0 conn(0x7f4dc00a0930 legacy=0x7f4dc0137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:16.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 10 monclient: _finish_hunting 0 2024-09-16T14:45:16.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 1 monclient: found mon.noname-c 2024-09-16T14:45:16.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:16.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:16.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 1 -- 172.21.15.26:0/1536132066 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc009a480 con 0x7f4dc009fe40 2024-09-16T14:45:16.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 10 monclient: _finish_auth 0 2024-09-16T14:45:16.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 10 monclient: _check_auth_tickets 2024-09-16T14:45:16.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:16.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 1 -- 172.21.15.26:0/1536132066 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4dbc001620 con 0x7f4dc009fe40 2024-09-16T14:45:16.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:16.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:16.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 10 monclient: dump: 2024-09-16T14:45:16.399 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:16.399 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:16.399 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:16.399 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:16.399 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:16.399 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:16.399 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:16.399 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:16.400 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:16.400 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:16.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 5 monclient: authenticate success, global_id 94111 2024-09-16T14:45:16.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:16.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 1 -- 172.21.15.26:0/1536132066 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4dbc001940 con 0x7f4dc009fe40 2024-09-16T14:45:16.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:16.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:16.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 4 set_mon_vals no callback set 2024-09-16T14:45:16.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 1 -- 172.21.15.26:0/1536132066 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4dbc001cd0 con 0x7f4dc009fe40 2024-09-16T14:45:16.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:16.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:16.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc4bb6640 10 monclient: dump: 2024-09-16T14:45:16.401 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:16.401 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:16.401 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:16.401 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:16.401 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:16.401 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:16.401 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:16.401 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:16.401 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:16.401 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:16.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 10 monclient: shutdown 2024-09-16T14:45:16.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:16.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 1 -- 172.21.15.26:0/1536132066 >> v1:172.21.15.189:6789/0 conn(0x7f4dc009fe40 legacy=0x7f4dc00a0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:16.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 1 -- 172.21.15.26:0/1536132066 shutdown_connections 2024-09-16T14:45:16.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 1 -- 172.21.15.26:0/1536132066 >> 172.21.15.26:0/1536132066 conn(0x7f4dc00a5540 msgr2=0x7f4dc00a5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:16.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 1 -- 172.21.15.26:0/1536132066 shutdown_connections 2024-09-16T14:45:16.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 1 -- 172.21.15.26:0/1536132066 wait complete. 2024-09-16T14:45:16.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 10 monclient: build_initial_monmap 2024-09-16T14:45:16.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 10 monclient: monmap: 2024-09-16T14:45:16.402 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:16.402 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:16.402 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:16.402 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:16.403 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:16.403 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:16.403 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:16.403 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:16.403 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:16.403 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:16.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 1 Processor -- start 2024-09-16T14:45:16.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 1 -- start start 2024-09-16T14:45:16.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 10 monclient: init 2024-09-16T14:45:16.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:16.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:45:16.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc009b4a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:16.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 10 monclient(hunting): picked mon.noname-b con 0x7f4dc009fe40 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:16.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc00a0930 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:16.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc006fc70 con 0x7f4dc009b4a0 2024-09-16T14:45:16.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc006f740 con 0x7f4dc009fe40 2024-09-16T14:45:16.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc006beb0 con 0x7f4dc00a0930 2024-09-16T14:45:16.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:16.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc7e1c640 10 monclient(hunting): authenticate will time out at 1140.514797s 2024-09-16T14:45:16.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc5bb8640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4dc009b4a0 0x7f4dc006b090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:46694/0 (socket says 172.21.15.26:46694) 2024-09-16T14:45:16.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4dc5bb8640 1 -- 172.21.15.26:0/183776360 learned_addr learned my addr 172.21.15.26:0/183776360 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:16.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379705422 0 0) 0x7f4dc006f740 con 0x7f4dc009fe40 2024-09-16T14:45:16.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4db67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:16.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4db67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:16.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4db67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:16.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4db67fc640 10 monclient(hunting): my global_id is 94114 2024-09-16T14:45:16.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.393+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4da4003530 con 0x7f4dc009fe40 2024-09-16T14:45:16.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 557554116 0 0) 0x7f4da4003530 con 0x7f4dc009fe40 2024-09-16T14:45:16.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4dc006f740 con 0x7f4dc009fe40 2024-09-16T14:45:16.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4db00017f0 con 0x7f4dc009fe40 2024-09-16T14:45:16.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:16.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:16.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient(hunting): dump: 2024-09-16T14:45:16.406 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:16.406 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:16.406 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:16.406 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:16.406 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:16.406 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:16.406 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:16.406 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:16.406 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:16.406 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:16.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:16.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4072316777 0 0) 0x7f4dc006f740 con 0x7f4dc009fe40 2024-09-16T14:45:16.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 >> v1:172.21.15.26:6790/0 conn(0x7f4dc00a0930 legacy=0x7f4dc01d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:16.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 >> v1:172.21.15.26:6789/0 conn(0x7f4dc009b4a0 legacy=0x7f4dc006b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:16.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:45:16.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 monclient: found mon.b 2024-09-16T14:45:16.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:16.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:16.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc011cae0 con 0x7f4dc009fe40 2024-09-16T14:45:16.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: _finish_auth 0 2024-09-16T14:45:16.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:45:16.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:16.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4dc7e1c640 5 monclient: authenticate success, global_id 94114 2024-09-16T14:45:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4dc7e1c640 10 monclient: _renew_subs 2024-09-16T14:45:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4dc7e1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4dc7e1c640 1 -- 172.21.15.26:0/183776360 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc0133700 con 0x7f4dc009fe40 2024-09-16T14:45:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4dc7e1c640 10 monclient: _renew_subs 2024-09-16T14:45:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4dc7e1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4dc7e1c640 1 -- 172.21.15.26:0/183776360 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc011c800 con 0x7f4dc009fe40 2024-09-16T14:45:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4dc53b7640 1 -- 172.21.15.26:0/183776360 >> v1:172.21.15.189:6789/0 conn(0x7f4dc009fe40 legacy=0x7f4dc006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T14:45:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4dc53b7640 1 -- 172.21.15.26:0/183776360 >> v1:172.21.15.189:6789/0 conn(0x7f4dc009fe40 legacy=0x7f4dc006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:45:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4dc53b7640 1 --1- 172.21.15.26:0/183776360 >> v1:172.21.15.189:6789/0 conn(0x7f4dc009fe40 0x7f4dc006b7a0 :-1 s=OPENED pgs=271 cs=1 l=1).handle_message read tag failed 2024-09-16T14:45:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4dc53b7640 1 --1- 172.21.15.26:0/183776360 >> v1:172.21.15.189:6789/0 conn(0x7f4dc009fe40 0x7f4dc006b7a0 :-1 s=OPENED pgs=271 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:45:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.189:6789/0 2024-09-16T14:45:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 >> v1:172.21.15.189:6789/0 conn(0x7f4dc009fe40 legacy=0x7f4dc006b7a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient(hunting): picked mon.b con 0x7f4da40047d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient(hunting): picked mon.a con 0x7f4dc009b4a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient(hunting): picked mon.c con 0x7f4dc00a0930 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4dc006f740 con 0x7f4dc009b4a0 2024-09-16T14:45:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4da4003530 con 0x7f4da40047d0 2024-09-16T14:45:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4da4011ef0 con 0x7f4dc00a0930 2024-09-16T14:45:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4dc7e1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4dc7e1c640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399681201 0 0) 0x7f4da4003530 con 0x7f4da40047d0 2024-09-16T14:45:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:45:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 --> v1:172.21.15.189:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f4da4012080 con 0x7f4da40047d0 2024-09-16T14:45:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2353717160 0 0) 0x7f4da4012080 con 0x7f4da40047d0 2024-09-16T14:45:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 >> v1:172.21.15.26:6790/0 conn(0x7f4dc00a0930 legacy=0x7f4da400e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 >> v1:172.21.15.26:6789/0 conn(0x7f4dc009b4a0 legacy=0x7f4da400a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:45:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 monclient: found mon.b 2024-09-16T14:45:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f4da4013230 con 0x7f4da40047d0 2024-09-16T14:45:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc00008e0 con 0x7f4da40047d0 2024-09-16T14:45:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da4003530 con 0x7f4da40047d0 2024-09-16T14:45:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: _finish_auth 0 2024-09-16T14:45:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:45:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4dbc001440 con 0x7f4da40047d0 2024-09-16T14:45:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: dump: 2024-09-16T14:45:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:16.415 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4dbc001660 con 0x7f4da40047d0 2024-09-16T14:45:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db77fe640 4 set_mon_vals no callback set 2024-09-16T14:45:16.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.397+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4dbc003710 con 0x7f4da40047d0 2024-09-16T14:45:16.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.401+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 <== mon.1 v1:172.21.15.189:6789/0 6 ==== osd_map(56..56 src has 15..56) v4 ==== 4310+0+0 (unknown 3213898037 0 0) 0x7f4dbc042c90 con 0x7f4da40047d0 2024-09-16T14:45:16.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.401+0000 7f4dc53b7640 10 monclient: get_auth_request con 0x7f4dc009fe40 auth_method 0 2024-09-16T14:45:16.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.413+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4dbc014150 con 0x7f4da40047d0 2024-09-16T14:45:16.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.413+0000 7f4db67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:16.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.413+0000 7f4db67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:16.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.425+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 <== mon.1 v1:172.21.15.189:6789/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4dbc0125c0 con 0x7f4da40047d0 2024-09-16T14:45:16.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.425+0000 7f4db67fc640 10 monclient: handle_mon_command_ack 1 not found 2024-09-16T14:45:16.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.629+0000 7f4dc7e1c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:45:16.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.629+0000 7f4dc7e1c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:16.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.629+0000 7f4dc7e1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:16.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.629+0000 7f4dc7e1c640 1 -- 172.21.15.26:0/183776360 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4dc00a1330 con 0x7f4da40047d0 2024-09-16T14:45:16.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.629+0000 7f4db67fc640 1 -- 172.21.15.26:0/183776360 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1251943183) 0x7f4dbc013d40 con 0x7f4da40047d0 2024-09-16T14:45:16.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.629+0000 7f4db67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:16.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.629+0000 7f4db67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:16.632 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":28,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:45:16.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.633+0000 7f4dc7e1c640 1 -- 172.21.15.26:0/183776360 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4dc009fe40 legacy=0x7f4da4044120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:16.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.633+0000 7f4dc7e1c640 10 monclient: shutdown 2024-09-16T14:45:16.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.633+0000 7f4dc7e1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:16.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.633+0000 7f4dc7e1c640 1 -- 172.21.15.26:0/183776360 >> v1:172.21.15.189:6789/0 conn(0x7f4da40047d0 legacy=0x7f4da4004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:16.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.633+0000 7f4dc7e1c640 1 -- 172.21.15.26:0/183776360 shutdown_connections 2024-09-16T14:45:16.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.633+0000 7f4dc7e1c640 1 -- 172.21.15.26:0/183776360 >> 172.21.15.26:0/183776360 conn(0x7f4dc00a5540 msgr2=0x7f4dc006fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:16.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.633+0000 7f4dc7e1c640 1 -- 172.21.15.26:0/183776360 shutdown_connections 2024-09-16T14:45:16.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:16.633+0000 7f4dc7e1c640 1 -- 172.21.15.26:0/183776360 wait complete. 2024-09-16T14:45:17.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:17.272+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:17.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:17.272+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:17.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:17.272+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:17.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:17.272+0000 7f5715ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:18.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:18.272+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:18.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:18.272+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:18.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:18.272+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:19.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:19.272+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:19.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:19.272+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:19.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:19.272+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:19.650 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:45:19.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2f057fa640 10 monclient: get_monmap_and_config 2024-09-16T14:45:19.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2f057fa640 10 monclient: build_initial_monmap 2024-09-16T14:45:19.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2f057fa640 10 monclient: monmap: 2024-09-16T14:45:19.782 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:19.783 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:19.783 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:19.782244+0000 2024-09-16T14:45:19.783 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:19.782244+0000 2024-09-16T14:45:19.783 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:19.783 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:19.783 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:19.783 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:19.783 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:19.783 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:19.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2f057fa640 1 Processor -- start 2024-09-16T14:45:19.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2f057fa640 1 -- start start 2024-09-16T14:45:19.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2f057fa640 10 monclient: init 2024-09-16T14:45:19.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2f057fa640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:19.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2f057fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:19.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2f057fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2f0009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:19.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2f057fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2f00132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:19.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2f057fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2f00138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:19.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2f057fa640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0009d260 con 0x7f2f00132650 2024-09-16T14:45:19.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2f057fa640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f00132a80 con 0x7f2f0009cb80 2024-09-16T14:45:19.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2f057fa640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0009cf60 con 0x7f2f00138ff0 2024-09-16T14:45:19.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2f057fa640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:19.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2f057fa640 10 monclient(hunting): authenticate will time out at 1143.902461s 2024-09-16T14:45:19.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2efe7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2f00132650 0x7f2f001378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48210/0 (socket says 172.21.15.26:48210) 2024-09-16T14:45:19.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2efe7fc640 1 -- 172.21.15.26:0/3603551433 learned_addr learned my addr 172.21.15.26:0/3603551433 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:19.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:19.781+0000 7f2eff7fe640 1 --1- 172.21.15.26:0/3603551433 >> v1:172.21.15.26:6790/0 conn(0x7f2f00138ff0 0x7f2f0013b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60588/0 (socket says 172.21.15.26:60588) 2024-09-16T14:45:20.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.272+0000 7f5715ffb640 10 monclient: tick 2024-09-16T14:45:20.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.272+0000 7f5715ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:20.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.272+0000 7f5715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.552+0000 7f5716ffd640 1 -- 172.21.15.189:0/4122245945 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v56) v1 ==== 139+0+0 (unknown 1681432562 0 0) 0x7f57200145e0 con 0x7f5728131630 2024-09-16T14:45:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.552+0000 7f5716ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T14:45:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.552+0000 7f5716ffd640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo-asdf' 2024-09-16T14:45:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:already renamed to 'foo-asdf' 2024-09-16T14:45:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.556+0000 7f572c98e640 1 -- 172.21.15.189:0/4122245945 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5708036270 legacy=0x7f5708038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.556+0000 7f572c98e640 10 monclient: shutdown 2024-09-16T14:45:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.556+0000 7f572c98e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.556+0000 7f572c98e640 1 -- 172.21.15.189:0/4122245945 >> v1:172.21.15.26:6790/0 conn(0x7f5728131630 legacy=0x7f57281d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.556+0000 7f572c98e640 1 -- 172.21.15.189:0/4122245945 shutdown_connections 2024-09-16T14:45:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.556+0000 7f572c98e640 1 -- 172.21.15.189:0/4122245945 >> 172.21.15.189:0/4122245945 conn(0x7f57280a3380 msgr2=0x7f57281b7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.556+0000 7f572c98e640 1 -- 172.21.15.189:0/4122245945 shutdown_connections 2024-09-16T14:45:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.556+0000 7f572c98e640 1 -- 172.21.15.189:0/4122245945 wait complete. 2024-09-16T14:45:20.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.561+0000 7f2efdffb640 1 -- 172.21.15.26:0/3603551433 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669239478 0 0) 0x7f2f0009cf60 con 0x7f2f00138ff0 2024-09-16T14:45:20.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.561+0000 7f2efdffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:20.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.561+0000 7f2efdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:20.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.561+0000 7f2efdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:20.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.561+0000 7f2efdffb640 10 monclient(hunting): my global_id is 4964 2024-09-16T14:45:20.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.561+0000 7f2efdffb640 1 -- 172.21.15.26:0/3603551433 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eec003530 con 0x7f2f00138ff0 2024-09-16T14:45:20.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.561+0000 7f2efdffb640 1 -- 172.21.15.26:0/3603551433 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 622000395 0 0) 0x7f2eec003530 con 0x7f2f00138ff0 2024-09-16T14:45:20.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 1 -- 172.21.15.26:0/3603551433 >> v1:172.21.15.189:6789/0 conn(0x7f2f0009cb80 legacy=0x7f2f000717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:20.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 1 -- 172.21.15.26:0/3603551433 >> v1:172.21.15.26:6789/0 conn(0x7f2f00132650 legacy=0x7f2f001378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:20.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 10 monclient: _finish_hunting 0 2024-09-16T14:45:20.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 1 monclient: found mon.noname-b 2024-09-16T14:45:20.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:20.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:45:20.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 1 -- 172.21.15.26:0/3603551433 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0009f2d0 con 0x7f2f00138ff0 2024-09-16T14:45:20.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 10 monclient: _finish_auth 0 2024-09-16T14:45:20.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:20.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:20.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 1 -- 172.21.15.26:0/3603551433 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2ef8001660 con 0x7f2f00138ff0 2024-09-16T14:45:20.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:20.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:20.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 10 monclient: dump: 2024-09-16T14:45:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:20.568 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:20.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2f057fa640 5 monclient: authenticate success, global_id 4964 2024-09-16T14:45:20.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2f057fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:20.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 1 -- 172.21.15.26:0/3603551433 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ef8001880 con 0x7f2f00138ff0 2024-09-16T14:45:20.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:20.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 1 -- 172.21.15.26:0/3603551433 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2ef8001c10 con 0x7f2f00138ff0 2024-09-16T14:45:20.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2f057fa640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:20.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2f057fa640 4 set_mon_vals no callback set 2024-09-16T14:45:20.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:20.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:20.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2efdffb640 10 monclient: dump: 2024-09-16T14:45:20.569 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:20.570 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:20.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2f057fa640 10 monclient: shutdown 2024-09-16T14:45:20.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2f057fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:20.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2f057fa640 1 -- 172.21.15.26:0/3603551433 >> v1:172.21.15.26:6790/0 conn(0x7f2f00138ff0 legacy=0x7f2f0013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:20.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2f057fa640 1 -- 172.21.15.26:0/3603551433 shutdown_connections 2024-09-16T14:45:20.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2f057fa640 1 -- 172.21.15.26:0/3603551433 >> 172.21.15.26:0/3603551433 conn(0x7f2f0009fb50 msgr2=0x7f2f0009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:20.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2f057fa640 1 -- 172.21.15.26:0/3603551433 shutdown_connections 2024-09-16T14:45:20.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2f057fa640 1 -- 172.21.15.26:0/3603551433 wait complete. 2024-09-16T14:45:20.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2f057fa640 10 monclient: build_initial_monmap 2024-09-16T14:45:20.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.565+0000 7f2f057fa640 10 monclient: monmap: 2024-09-16T14:45:20.576 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:20.576 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:20.577 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:20.577 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:20.577 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:20.577 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:20.577 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:20.577 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:20.577 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:20.577 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:20.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 1 Processor -- start 2024-09-16T14:45:20.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 1 -- start start 2024-09-16T14:45:20.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 10 monclient: init 2024-09-16T14:45:20.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:20.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:20.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2f0009cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:20.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2f00132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:20.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2f00138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:20.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f00138940 con 0x7f2f0009cb80 2024-09-16T14:45:20.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0009a240 con 0x7f2f00138ff0 2024-09-16T14:45:20.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0013c7d0 con 0x7f2f00132650 2024-09-16T14:45:20.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:20.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 10 monclient(hunting): authenticate will time out at 1144.689484s 2024-09-16T14:45:20.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2efeffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2f0009cb80 0x7f2f0012dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48222/0 (socket says 172.21.15.26:48222) 2024-09-16T14:45:20.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2efeffd640 1 -- 172.21.15.26:0/1049375484 learned_addr learned my addr 172.21.15.26:0/1049375484 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:20.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598269358 0 0) 0x7f2f00132a80 con 0x7f2f0009cb80 2024-09-16T14:45:20.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:45:20.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:20.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:20.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient(hunting): my global_id is 84363 2024-09-16T14:45:20.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee0003530 con 0x7f2f0009cb80 2024-09-16T14:45:20.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074905992 0 0) 0x7f2f0013c7d0 con 0x7f2f00132650 2024-09-16T14:45:20.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:45:20.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:20.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:20.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient(hunting): my global_id is 4967 2024-09-16T14:45:20.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f00132a80 con 0x7f2f00132650 2024-09-16T14:45:20.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 686781958 0 0) 0x7f2ee0003530 con 0x7f2f0009cb80 2024-09-16T14:45:20.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f0013c7d0 con 0x7f2f0009cb80 2024-09-16T14:45:20.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4056152220 0 0) 0x7f2f00132a80 con 0x7f2f00132650 2024-09-16T14:45:20.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ee0003530 con 0x7f2f00132650 2024-09-16T14:45:20.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2ee8001740 con 0x7f2f0009cb80 2024-09-16T14:45:20.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:20.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:20.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient(hunting): dump: 2024-09-16T14:45:20.581 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:20.581 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:20.581 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.581 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.581 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:20.581 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:20.581 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:20.581 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:20.581 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:20.582 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:20.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:20.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2ef0001af0 con 0x7f2f00132650 2024-09-16T14:45:20.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:20.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:20.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient(hunting): dump: 2024-09-16T14:45:20.582 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:20.582 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:20.582 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.582 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.583 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:20.583 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:20.583 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:20.583 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:20.583 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:20.583 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:20.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3899953012 0 0) 0x7f2f0013c7d0 con 0x7f2f0009cb80 2024-09-16T14:45:20.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 >> v1:172.21.15.26:6790/0 conn(0x7f2f00132650 legacy=0x7f2f0012e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:20.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 >> v1:172.21.15.189:6789/0 conn(0x7f2f00138ff0 legacy=0x7f2f001d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:20.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:45:20.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 monclient: found mon.a 2024-09-16T14:45:20.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:20.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:20.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f00135010 con 0x7f2f0009cb80 2024-09-16T14:45:20.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient: _finish_auth 0 2024-09-16T14:45:20.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:20.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:20.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 5 monclient: authenticate success, global_id 84363 2024-09-16T14:45:20.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 10 monclient: _renew_subs 2024-09-16T14:45:20.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:20.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 1 -- 172.21.15.26:0/1049375484 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0012f650 con 0x7f2f0009cb80 2024-09-16T14:45:20.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 10 monclient: _renew_subs 2024-09-16T14:45:20.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:20.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2f057fa640 1 -- 172.21.15.26:0/1049375484 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0012eac0 con 0x7f2f0009cb80 2024-09-16T14:45:20.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ee8001930 con 0x7f2f0009cb80 2024-09-16T14:45:20.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:20.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2ee80037c0 con 0x7f2f0009cb80 2024-09-16T14:45:20.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:20.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2efd7fa640 4 set_mon_vals no callback set 2024-09-16T14:45:20.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:20.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.569+0000 7f2ef77fe640 10 monclient: dump: 2024-09-16T14:45:20.586 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:20.586 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:20.586 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.586 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.586 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:20.586 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:20.586 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:20.586 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:20.586 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:20.586 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:20.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.573+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2ee8003920 con 0x7f2f0009cb80 2024-09-16T14:45:20.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.573+0000 7f2f057fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:20.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.573+0000 7f2f057fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:20.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.573+0000 7f2f057fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:20.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.573+0000 7f2f057fa640 1 -- 172.21.15.26:0/1049375484 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee4005050 con 0x7f2f0009cb80 2024-09-16T14:45:20.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.573+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(56..56 src has 15..56) v4 ==== 4310+0+0 (unknown 3213898037 0 0) 0x7f2ee8043320 con 0x7f2f0009cb80 2024-09-16T14:45:20.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.573+0000 7f2efe7fc640 10 monclient: get_auth_request con 0x7f2ee0036210 auth_method 0 2024-09-16T14:45:20.588 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rename bar bar-asdf 2024-09-16T14:45:20.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.577+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2ee8004170 con 0x7f2f0009cb80 2024-09-16T14:45:20.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.577+0000 7f2ef77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:20.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.577+0000 7f2ef77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c27304640 10 monclient: get_monmap_and_config 2024-09-16T14:45:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c27304640 10 monclient: build_initial_monmap 2024-09-16T14:45:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c27304640 10 monclient: monmap: 2024-09-16T14:45:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:20.702875+0000 2024-09-16T14:45:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:20.702875+0000 2024-09-16T14:45:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:20.704 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:20.704 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:20.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c27304640 1 Processor -- start 2024-09-16T14:45:20.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c27304640 1 -- start start 2024-09-16T14:45:20.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c27304640 10 monclient: init 2024-09-16T14:45:20.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c27304640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:20.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c27304640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:45:20.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c27304640 10 monclient(hunting): picked mon.noname-a con 0x7f8c20131630 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:20.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c27304640 10 monclient(hunting): picked mon.noname-c con 0x7f8c2009edd0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:20.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c27304640 10 monclient(hunting): picked mon.noname-b con 0x7f8c2009e950 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:20.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c27304640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2009b920 con 0x7f8c20131630 2024-09-16T14:45:20.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c27304640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2009bc00 con 0x7f8c2009edd0 2024-09-16T14:45:20.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c27304640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2009cba0 con 0x7f8c2009e950 2024-09-16T14:45:20.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c27304640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:20.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c27304640 10 monclient(hunting): authenticate will time out at 1102.819195s 2024-09-16T14:45:20.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c2489f640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8c2009edd0 0x7f8c20136b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33256/0 (socket says 172.21.15.189:33256) 2024-09-16T14:45:20.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c2489f640 1 -- 172.21.15.189:0/2290941456 learned_addr learned my addr 172.21.15.189:0/2290941456 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:20.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c17fff640 1 -- 172.21.15.189:0/2290941456 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418813445 0 0) 0x7f8c2009b920 con 0x7f8c20131630 2024-09-16T14:45:20.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c17fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:20.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c17fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:20.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:20.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c17fff640 10 monclient(hunting): my global_id is 84366 2024-09-16T14:45:20.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.700+0000 7f8c17fff640 1 -- 172.21.15.189:0/2290941456 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c0c003530 con 0x7f8c20131630 2024-09-16T14:45:20.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 1 -- 172.21.15.189:0/2290941456 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988100992 0 0) 0x7f8c2009cba0 con 0x7f8c2009e950 2024-09-16T14:45:20.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:20.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:20.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:20.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 10 monclient(hunting): my global_id is 4970 2024-09-16T14:45:20.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 1 -- 172.21.15.189:0/2290941456 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c2009b920 con 0x7f8c2009e950 2024-09-16T14:45:20.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 1 -- 172.21.15.189:0/2290941456 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2753703296 0 0) 0x7f8c0c003530 con 0x7f8c20131630 2024-09-16T14:45:20.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 1 -- 172.21.15.189:0/2290941456 >> v1:172.21.15.26:6790/0 conn(0x7f8c2009e950 legacy=0x7f8c2013a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:20.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 1 -- 172.21.15.189:0/2290941456 >> v1:172.21.15.189:6789/0 conn(0x7f8c2009edd0 legacy=0x7f8c20136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 10 monclient: _finish_hunting 0 2024-09-16T14:45:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 1 monclient: found mon.noname-a 2024-09-16T14:45:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:45:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 1 -- 172.21.15.189:0/2290941456 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c2009b420 con 0x7f8c20131630 2024-09-16T14:45:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 10 monclient: _finish_auth 0 2024-09-16T14:45:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 10 monclient: _check_auth_tickets 2024-09-16T14:45:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 1 -- 172.21.15.189:0/2290941456 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8c1c001620 con 0x7f8c20131630 2024-09-16T14:45:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 5 monclient: authenticate success, global_id 84366 2024-09-16T14:45:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 10 monclient: dump: 2024-09-16T14:45:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:20.716 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 1 -- 172.21.15.189:0/2290941456 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c1c001900 con 0x7f8c20131630 2024-09-16T14:45:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 1 -- 172.21.15.189:0/2290941456 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8c1c001c90 con 0x7f8c20131630 2024-09-16T14:45:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 4 set_mon_vals no callback set 2024-09-16T14:45:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c17fff640 10 monclient: dump: 2024-09-16T14:45:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:20.717 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 10 monclient: shutdown 2024-09-16T14:45:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 1 -- 172.21.15.189:0/2290941456 >> v1:172.21.15.26:6789/0 conn(0x7f8c20131630 legacy=0x7f8c20131a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 1 -- 172.21.15.189:0/2290941456 shutdown_connections 2024-09-16T14:45:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 1 -- 172.21.15.189:0/2290941456 >> 172.21.15.189:0/2290941456 conn(0x7f8c200a3380 msgr2=0x7f8c2009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 1 -- 172.21.15.189:0/2290941456 shutdown_connections 2024-09-16T14:45:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 1 -- 172.21.15.189:0/2290941456 wait complete. 2024-09-16T14:45:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 10 monclient: build_initial_monmap 2024-09-16T14:45:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 10 monclient: monmap: 2024-09-16T14:45:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:20.719 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 1 Processor -- start 2024-09-16T14:45:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 1 -- start start 2024-09-16T14:45:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 10 monclient: init 2024-09-16T14:45:20.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:20.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:20.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 10 monclient(hunting): picked mon.noname-a con 0x7f8c2009e950 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:20.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 10 monclient(hunting): picked mon.noname-c con 0x7f8c2009edd0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:20.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 10 monclient(hunting): picked mon.noname-b con 0x7f8c20131630 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:20.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c200a1200 con 0x7f8c2009e950 2024-09-16T14:45:20.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2006b320 con 0x7f8c20131630 2024-09-16T14:45:20.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c200a0dd0 con 0x7f8c2009edd0 2024-09-16T14:45:20.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:20.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 10 monclient(hunting): authenticate will time out at 1102.822591s 2024-09-16T14:45:20.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c258a1640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8c20131630 0x7f8c201217c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33262/0 (socket says 172.21.15.189:33262) 2024-09-16T14:45:20.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c258a1640 1 -- 172.21.15.189:0/885545010 learned_addr learned my addr 172.21.15.189:0/885545010 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:20.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041911338 0 0) 0x7f8c2009b420 con 0x7f8c2009e950 2024-09-16T14:45:20.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:20.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:20.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:20.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 10 monclient(hunting): my global_id is 84369 2024-09-16T14:45:20.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c04003530 con 0x7f8c2009e950 2024-09-16T14:45:20.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3404381513 0 0) 0x7f8c04003530 con 0x7f8c2009e950 2024-09-16T14:45:20.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c2009b420 con 0x7f8c2009e950 2024-09-16T14:45:20.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8c1c001440 con 0x7f8c2009e950 2024-09-16T14:45:20.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:20.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:20.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 10 monclient(hunting): dump: 2024-09-16T14:45:20.722 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:20.722 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:20.722 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.722 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.722 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:20.722 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:20.722 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:20.723 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:20.723 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:20.723 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:20.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:20.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048205935 0 0) 0x7f8c200a0dd0 con 0x7f8c2009edd0 2024-09-16T14:45:20.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:20.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:20.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:20.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 10 monclient(hunting): my global_id is 4973 2024-09-16T14:45:20.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c04003530 con 0x7f8c2009edd0 2024-09-16T14:45:20.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2121286603 0 0) 0x7f8c2009b420 con 0x7f8c2009e950 2024-09-16T14:45:20.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 >> v1:172.21.15.26:6790/0 conn(0x7f8c2009edd0 legacy=0x7f8c201210b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 >> v1:172.21.15.189:6789/0 conn(0x7f8c20131630 legacy=0x7f8c201217c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:45:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 1 monclient: found mon.a 2024-09-16T14:45:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c201311d0 con 0x7f8c2009e950 2024-09-16T14:45:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 10 monclient: _finish_auth 0 2024-09-16T14:45:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 5 monclient: authenticate success, global_id 84369 2024-09-16T14:45:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 10 monclient: _renew_subs 2024-09-16T14:45:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 1 -- 172.21.15.189:0/885545010 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c20124890 con 0x7f8c2009e950 2024-09-16T14:45:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 10 monclient: _renew_subs 2024-09-16T14:45:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.704+0000 7f8c27304640 1 -- 172.21.15.189:0/885545010 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c201224c0 con 0x7f8c2009e950 2024-09-16T14:45:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.708+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c1c0016c0 con 0x7f8c2009e950 2024-09-16T14:45:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.708+0000 7f8c15ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.708+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8c1c003b10 con 0x7f8c2009e950 2024-09-16T14:45:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.708+0000 7f8c15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.708+0000 7f8c15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.708+0000 7f8c16ffd640 4 set_mon_vals no callback set 2024-09-16T14:45:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.708+0000 7f8c15ffb640 10 monclient: dump: 2024-09-16T14:45:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:20.726 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.708+0000 7f8c27304640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.708+0000 7f8c27304640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.708+0000 7f8c27304640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.708+0000 7f8c27304640 1 -- 172.21.15.189:0/885545010 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c00005050 con 0x7f8c2009e950 2024-09-16T14:45:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.708+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8c1c00e580 con 0x7f8c2009e950 2024-09-16T14:45:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.708+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(56..56 src has 15..56) v4 ==== 4310+0+0 (unknown 3213898037 0 0) 0x7f8c1c042200 con 0x7f8c2009e950 2024-09-16T14:45:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.708+0000 7f8c2489f640 10 monclient: get_auth_request con 0x7f8c040362b0 auth_method 0 2024-09-16T14:45:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.716+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8c1c0132f0 con 0x7f8c2009e950 2024-09-16T14:45:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.716+0000 7f8c15ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:20.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.716+0000 7f8c15ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:20.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.805+0000 7f2f057fa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:45:20.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.805+0000 7f2f057fa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:20.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.805+0000 7f2f057fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:20.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.805+0000 7f2f057fa640 1 -- 172.21.15.26:0/1049375484 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2ee4005880 con 0x7f2f0009cb80 2024-09-16T14:45:20.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.805+0000 7f2ef77fe640 1 -- 172.21.15.26:0/1049375484 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3397463835) 0x7f2ee8012a70 con 0x7f2f0009cb80 2024-09-16T14:45:20.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.805+0000 7f2ef77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:20.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.805+0000 7f2ef77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:20.809 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":30,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:45:20.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.809+0000 7f2f057fa640 1 -- 172.21.15.26:0/1049375484 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2ee0036210 legacy=0x7f2ee00386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:20.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.809+0000 7f2f057fa640 10 monclient: shutdown 2024-09-16T14:45:20.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.809+0000 7f2f057fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:20.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.809+0000 7f2f057fa640 1 -- 172.21.15.26:0/1049375484 >> v1:172.21.15.26:6789/0 conn(0x7f2f0009cb80 legacy=0x7f2f0012dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:20.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.809+0000 7f2f057fa640 1 -- 172.21.15.26:0/1049375484 shutdown_connections 2024-09-16T14:45:20.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.809+0000 7f2f057fa640 1 -- 172.21.15.26:0/1049375484 >> 172.21.15.26:0/1049375484 conn(0x7f2f0009fb50 msgr2=0x7f2f000a21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:20.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.809+0000 7f2f057fa640 1 -- 172.21.15.26:0/1049375484 shutdown_connections 2024-09-16T14:45:20.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:20.809+0000 7f2f057fa640 1 -- 172.21.15.26:0/1049375484 wait complete. 2024-09-16T14:45:20.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.876+0000 7f8c27304640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T14:45:20.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.876+0000 7f8c27304640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T14:45:20.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.876+0000 7f8c27304640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:20.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:20.876+0000 7f8c27304640 1 -- 172.21.15.189:0/885545010 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"} v 0) v1 -- 0x7f8c00005880 con 0x7f8c2009e950 2024-09-16T14:45:21.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.616+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}]=0 v57) v1 ==== 110+0+0 (unknown 2146155625 0 0) 0x7f8c1c011990 con 0x7f8c2009e950 2024-09-16T14:45:21.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.616+0000 7f8c15ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T14:45:21.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.616+0000 7f8c15ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:21.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.716+0000 7f8c27304640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T14:45:21.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.716+0000 7f8c27304640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T14:45:21.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.716+0000 7f8c27304640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:21.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.716+0000 7f8c27304640 1 -- 172.21.15.189:0/885545010 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"} v 0) v1 -- 0x7f8c00005300 con 0x7f8c2009e950 2024-09-16T14:45:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.720+0000 7f8c15ffb640 1 -- 172.21.15.189:0/885545010 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}]=0 already renamed to 'bar-asdf' v57) v1 ==== 139+0+0 (unknown 1568429341 0 0) 0x7f8c1c013110 con 0x7f8c2009e950 2024-09-16T14:45:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.720+0000 7f8c15ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T14:45:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.720+0000 7f8c15ffb640 10 monclient: _finish_command 3 = system:0 already renamed to 'bar-asdf' 2024-09-16T14:45:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:already renamed to 'bar-asdf' 2024-09-16T14:45:21.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.720+0000 7f8c27304640 1 -- 172.21.15.189:0/885545010 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8c040362b0 legacy=0x7f8c04038770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:21.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.720+0000 7f8c27304640 10 monclient: shutdown 2024-09-16T14:45:21.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.720+0000 7f8c27304640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:21.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.720+0000 7f8c27304640 1 -- 172.21.15.189:0/885545010 >> v1:172.21.15.26:6789/0 conn(0x7f8c2009e950 legacy=0x7f8c201209a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.720+0000 7f8c27304640 1 -- 172.21.15.189:0/885545010 shutdown_connections 2024-09-16T14:45:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.720+0000 7f8c27304640 1 -- 172.21.15.189:0/885545010 >> 172.21.15.189:0/885545010 conn(0x7f8c200a3380 msgr2=0x7f8c20070810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.720+0000 7f8c27304640 1 -- 172.21.15.189:0/885545010 shutdown_connections 2024-09-16T14:45:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.720+0000 7f8c27304640 1 -- 172.21.15.189:0/885545010 wait complete. 2024-09-16T14:45:21.740 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls 2024-09-16T14:45:21.740 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep foo-asdf 2024-09-16T14:45:21.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.852+0000 7facb618e640 10 monclient: get_monmap_and_config 2024-09-16T14:45:21.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.852+0000 7facb618e640 10 monclient: build_initial_monmap 2024-09-16T14:45:21.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.852+0000 7facb618e640 10 monclient: monmap: 2024-09-16T14:45:21.857 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:21.857 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:21.857 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:21.856841+0000 2024-09-16T14:45:21.857 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:21.856841+0000 2024-09-16T14:45:21.857 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:21.857 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:21.858 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:21.858 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:21.858 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:21.858 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:21.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.852+0000 7facb618e640 1 Processor -- start 2024-09-16T14:45:21.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.852+0000 7facb618e640 1 -- start start 2024-09-16T14:45:21.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.852+0000 7facb618e640 10 monclient: init 2024-09-16T14:45:21.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:21.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:45:21.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 10 monclient(hunting): picked mon.noname-a con 0x7facb01318a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:21.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 10 monclient(hunting): picked mon.noname-c con 0x7facb0134d50 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:21.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 10 monclient(hunting): picked mon.noname-b con 0x7facb013b2b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:21.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb009d8a0 con 0x7facb01318a0 2024-09-16T14:45:21.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb009ddf0 con 0x7facb0134d50 2024-09-16T14:45:21.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb009e8b0 con 0x7facb013b2b0 2024-09-16T14:45:21.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:21.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 10 monclient(hunting): authenticate will time out at 1103.972573s 2024-09-16T14:45:21.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facaeffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7facb0134d50 0x7facb0139b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33266/0 (socket says 172.21.15.189:33266) 2024-09-16T14:45:21.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facaeffd640 1 -- 172.21.15.189:0/3857519253 learned_addr learned my addr 172.21.15.189:0/3857519253 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:21.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 1 -- 172.21.15.189:0/3857519253 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534905045 0 0) 0x7facb009e8b0 con 0x7facb013b2b0 2024-09-16T14:45:21.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:21.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:21.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:21.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient(hunting): my global_id is 4976 2024-09-16T14:45:21.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 1 -- 172.21.15.189:0/3857519253 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac9c003530 con 0x7facb013b2b0 2024-09-16T14:45:21.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 1 -- 172.21.15.189:0/3857519253 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142362365 0 0) 0x7facb009d8a0 con 0x7facb01318a0 2024-09-16T14:45:21.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:21.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:21.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:21.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient(hunting): my global_id is 84372 2024-09-16T14:45:21.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 1 -- 172.21.15.189:0/3857519253 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb009e8b0 con 0x7facb01318a0 2024-09-16T14:45:21.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 1 -- 172.21.15.189:0/3857519253 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3823463955 0 0) 0x7fac9c003530 con 0x7facb013b2b0 2024-09-16T14:45:21.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 1 -- 172.21.15.189:0/3857519253 >> v1:172.21.15.189:6789/0 conn(0x7facb0134d50 legacy=0x7facb0139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:21.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 1 -- 172.21.15.189:0/3857519253 >> v1:172.21.15.26:6789/0 conn(0x7facb01318a0 legacy=0x7facb0131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:21.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:45:21.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 1 monclient: found mon.noname-b 2024-09-16T14:45:21.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:21.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:45:21.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 1 -- 172.21.15.189:0/3857519253 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb009eb90 con 0x7facb013b2b0 2024-09-16T14:45:21.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient: _finish_auth 0 2024-09-16T14:45:21.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:45:21.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:21.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 1 -- 172.21.15.189:0/3857519253 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faca00019a0 con 0x7facb013b2b0 2024-09-16T14:45:21.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:21.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:21.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient: dump: 2024-09-16T14:45:21.872 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:21.873 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:21.873 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:21.873 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:21.873 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:21.873 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:21.873 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:21.873 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:21.873 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:21.873 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:21.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 5 monclient: authenticate success, global_id 4976 2024-09-16T14:45:21.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:21.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 1 -- 172.21.15.189:0/3857519253 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faca0001c10 con 0x7facb013b2b0 2024-09-16T14:45:21.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:21.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 1 -- 172.21.15.189:0/3857519253 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faca0001fa0 con 0x7facb013b2b0 2024-09-16T14:45:21.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:21.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:21.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facae7fc640 10 monclient: dump: 2024-09-16T14:45:21.874 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:21.874 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:21.874 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:21.874 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:21.874 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:21.874 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:21.874 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:21.874 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:21.874 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:21.874 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:21.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:21.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 4 set_mon_vals no callback set 2024-09-16T14:45:21.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 10 monclient: shutdown 2024-09-16T14:45:21.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:21.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 1 -- 172.21.15.189:0/3857519253 >> v1:172.21.15.26:6790/0 conn(0x7facb013b2b0 legacy=0x7facb013d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:21.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 1 -- 172.21.15.189:0/3857519253 shutdown_connections 2024-09-16T14:45:21.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 1 -- 172.21.15.189:0/3857519253 >> 172.21.15.189:0/3857519253 conn(0x7facb00a3320 msgr2=0x7facb009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:21.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 1 -- 172.21.15.189:0/3857519253 shutdown_connections 2024-09-16T14:45:21.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 1 -- 172.21.15.189:0/3857519253 wait complete. 2024-09-16T14:45:21.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 10 monclient: build_initial_monmap 2024-09-16T14:45:21.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 10 monclient: monmap: 2024-09-16T14:45:21.875 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:21.876 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:21.876 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:21.876 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:21.876 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:21.876 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:21.876 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:21.876 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:21.876 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:21.876 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:21.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 1 Processor -- start 2024-09-16T14:45:21.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 1 -- start start 2024-09-16T14:45:21.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.856+0000 7facb618e640 10 monclient: init 2024-09-16T14:45:21.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:21.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:21.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 10 monclient(hunting): picked mon.noname-b con 0x7facb01318a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:21.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 10 monclient(hunting): picked mon.noname-a con 0x7facb0134d50 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:21.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 10 monclient(hunting): picked mon.noname-c con 0x7facb013b2b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:21.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb011b4d0 con 0x7facb0134d50 2024-09-16T14:45:21.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb011ba00 con 0x7facb01318a0 2024-09-16T14:45:21.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb011bce0 con 0x7facb013b2b0 2024-09-16T14:45:21.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:21.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 10 monclient(hunting): authenticate will time out at 1103.976169s 2024-09-16T14:45:21.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facaf7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7facb01318a0 0x7facb0119340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33282/0 (socket says 172.21.15.189:33282) 2024-09-16T14:45:21.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facaf7fe640 1 -- 172.21.15.189:0/4229136260 learned_addr learned my addr 172.21.15.189:0/4229136260 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:21.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630842955 0 0) 0x7facb011bce0 con 0x7facb013b2b0 2024-09-16T14:45:21.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:21.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:21.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:21.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient(hunting): my global_id is 4979 2024-09-16T14:45:21.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac90003530 con 0x7facb013b2b0 2024-09-16T14:45:21.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286690706 0 0) 0x7facb011b4d0 con 0x7facb0134d50 2024-09-16T14:45:21.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:21.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:21.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:21.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient(hunting): my global_id is 84375 2024-09-16T14:45:21.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb011bce0 con 0x7facb0134d50 2024-09-16T14:45:21.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3616248333 0 0) 0x7fac90003530 con 0x7facb013b2b0 2024-09-16T14:45:21.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facb011b4d0 con 0x7facb013b2b0 2024-09-16T14:45:21.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faca0001f20 con 0x7facb013b2b0 2024-09-16T14:45:21.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:21.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:21.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient(hunting): dump: 2024-09-16T14:45:21.879 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:21.880 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:21.880 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:21.880 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:21.880 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:21.880 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:21.880 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:21.880 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:21.880 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:21.880 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:21.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:21.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2246882567 0 0) 0x7facb011bce0 con 0x7facb0134d50 2024-09-16T14:45:21.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac90003530 con 0x7facb0134d50 2024-09-16T14:45:21.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fac980016f0 con 0x7facb0134d50 2024-09-16T14:45:21.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:21.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:21.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient(hunting): dump: 2024-09-16T14:45:21.881 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:21.881 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:21.881 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:21.881 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:21.881 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:21.881 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:21.881 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:21.881 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:21.881 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:21.881 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 108502515 0 0) 0x7facb011b4d0 con 0x7facb013b2b0 2024-09-16T14:45:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 >> v1:172.21.15.189:6789/0 conn(0x7facb01318a0 legacy=0x7facb0119340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 >> v1:172.21.15.26:6789/0 conn(0x7facb0134d50 legacy=0x7facb0119a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:45:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 monclient: found mon.c 2024-09-16T14:45:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb011a160 con 0x7facb013b2b0 2024-09-16T14:45:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient: _finish_auth 0 2024-09-16T14:45:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:45:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 5 monclient: authenticate success, global_id 4979 2024-09-16T14:45:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 10 monclient: _renew_subs 2024-09-16T14:45:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:21.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 1 -- 172.21.15.189:0/4229136260 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb013a0b0 con 0x7facb013b2b0 2024-09-16T14:45:21.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 10 monclient: _renew_subs 2024-09-16T14:45:21.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:21.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 1 -- 172.21.15.189:0/4229136260 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7facb011b190 con 0x7facb013b2b0 2024-09-16T14:45:21.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faca0003c30 con 0x7facb013b2b0 2024-09-16T14:45:21.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:21.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facad7fa640 4 set_mon_vals no callback set 2024-09-16T14:45:21.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faca0003f40 con 0x7facb013b2b0 2024-09-16T14:45:21.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:21.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:21.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 10 monclient: dump: 2024-09-16T14:45:21.883 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:21.884 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:21.884 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:21.884 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:21.884 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:21.884 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:21.884 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:21.884 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:21.884 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:21.884 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:21.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:21.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:21.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:21.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facb618e640 1 -- 172.21.15.189:0/4229136260 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac94005050 con 0x7facb013b2b0 2024-09-16T14:45:21.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7faca000e9b0 con 0x7facb013b2b0 2024-09-16T14:45:21.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(57..57 src has 15..57) v4 ==== 4315+0+0 (unknown 2780822968 0 0) 0x7faca00425d0 con 0x7facb013b2b0 2024-09-16T14:45:21.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.860+0000 7facaf7fe640 10 monclient: get_auth_request con 0x7fac90036310 auth_method 0 2024-09-16T14:45:21.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.872+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7faca00136e0 con 0x7facb013b2b0 2024-09-16T14:45:21.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.872+0000 7faca7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:21.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:21.872+0000 7faca7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:22.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.032+0000 7facb618e640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:22.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.032+0000 7facb618e640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:22.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.032+0000 7facb618e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:22.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.032+0000 7facb618e640 1 -- 172.21.15.189:0/4229136260 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fac94005300 con 0x7facb013b2b0 2024-09-16T14:45:22.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.032+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v57) v1 ==== 65+0+70 (unknown 2262488454 0 3909614116) 0x7faca0013500 con 0x7facb013b2b0 2024-09-16T14:45:22.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.032+0000 7faca7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:22.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.032+0000 7faca7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:22.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.128+0000 7facb618e640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:22.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.128+0000 7facb618e640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:22.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.128+0000 7facb618e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:22.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.128+0000 7facb618e640 1 -- 172.21.15.189:0/4229136260 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fac94005540 con 0x7facb013b2b0 2024-09-16T14:45:22.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.128+0000 7faca7fff640 1 -- 172.21.15.189:0/4229136260 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v57) v1 ==== 65+0+70 (unknown 2262488454 0 3909614116) 0x7faca0013500 con 0x7facb013b2b0 2024-09-16T14:45:22.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.128+0000 7faca7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:22.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.128+0000 7faca7fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:45:22.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.128+0000 7facb618e640 1 -- 172.21.15.189:0/4229136260 >> v1:172.21.15.189:6812/3196161582 conn(0x7fac90036310 legacy=0x7fac900387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:22.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.128+0000 7facb618e640 10 monclient: shutdown 2024-09-16T14:45:22.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.128+0000 7facb618e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:22.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.128+0000 7facb618e640 1 -- 172.21.15.189:0/4229136260 >> v1:172.21.15.26:6790/0 conn(0x7facb013b2b0 legacy=0x7facb01d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:22.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.128+0000 7facb618e640 1 -- 172.21.15.189:0/4229136260 shutdown_connections 2024-09-16T14:45:22.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.128+0000 7facb618e640 1 -- 172.21.15.189:0/4229136260 >> 172.21.15.189:0/4229136260 conn(0x7facb00a3320 msgr2=0x7facb009c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:22.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.128+0000 7facb618e640 1 -- 172.21.15.189:0/4229136260 shutdown_connections 2024-09-16T14:45:22.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.128+0000 7facb618e640 1 -- 172.21.15.189:0/4229136260 wait complete. 2024-09-16T14:45:22.145 INFO:tasks.workunit.client.0.smithi189.stdout:foo-asdf 2024-09-16T14:45:22.146 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls 2024-09-16T14:45:22.146 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep bar-asdf 2024-09-16T14:45:22.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 10 monclient: get_monmap_and_config 2024-09-16T14:45:22.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 10 monclient: build_initial_monmap 2024-09-16T14:45:22.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 10 monclient: monmap: 2024-09-16T14:45:22.263 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:22.263 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:22.263 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:22.262480+0000 2024-09-16T14:45:22.263 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:22.262480+0000 2024-09-16T14:45:22.263 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:22.263 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:22.263 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:22.263 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:22.263 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:22.263 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:22.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 1 Processor -- start 2024-09-16T14:45:22.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 1 -- start start 2024-09-16T14:45:22.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 10 monclient: init 2024-09-16T14:45:22.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:22.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:45:22.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 10 monclient(hunting): picked mon.noname-a con 0x7fdbf81315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:22.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 10 monclient(hunting): picked mon.noname-c con 0x7fdbf809b320 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:22.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 10 monclient(hunting): picked mon.noname-b con 0x7fdbf8132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:22.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf809edd0 con 0x7fdbf81315a0 2024-09-16T14:45:22.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf809bc00 con 0x7fdbf809b320 2024-09-16T14:45:22.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf809e900 con 0x7fdbf8132090 2024-09-16T14:45:22.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:22.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 10 monclient(hunting): authenticate will time out at 1104.378413s 2024-09-16T14:45:22.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbfcf18640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdbf809b320 0x7fdbf8136ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33298/0 (socket says 172.21.15.189:33298) 2024-09-16T14:45:22.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbfcf18640 1 -- 172.21.15.189:0/2848924760 learned_addr learned my addr 172.21.15.189:0/2848924760 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 1 -- 172.21.15.189:0/2848924760 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853140159 0 0) 0x7fdbf809e900 con 0x7fdbf8132090 2024-09-16T14:45:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 10 monclient(hunting): my global_id is 4982 2024-09-16T14:45:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 1 -- 172.21.15.189:0/2848924760 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe4005740 con 0x7fdbf8132090 2024-09-16T14:45:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 1 -- 172.21.15.189:0/2848924760 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167642238 0 0) 0x7fdbf809edd0 con 0x7fdbf81315a0 2024-09-16T14:45:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 10 monclient(hunting): my global_id is 84378 2024-09-16T14:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 1 -- 172.21.15.189:0/2848924760 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf809e900 con 0x7fdbf81315a0 2024-09-16T14:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 1 -- 172.21.15.189:0/2848924760 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1653693844 0 0) 0x7fdbe4005740 con 0x7fdbf8132090 2024-09-16T14:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 1 -- 172.21.15.189:0/2848924760 >> v1:172.21.15.189:6789/0 conn(0x7fdbf809b320 legacy=0x7fdbf8136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 1 -- 172.21.15.189:0/2848924760 >> v1:172.21.15.26:6789/0 conn(0x7fdbf81315a0 legacy=0x7fdbf8131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 10 monclient: _finish_hunting 0 2024-09-16T14:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 1 monclient: found mon.noname-b 2024-09-16T14:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 1 -- 172.21.15.189:0/2848924760 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf809b920 con 0x7fdbf8132090 2024-09-16T14:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 10 monclient: _finish_auth 0 2024-09-16T14:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 10 monclient: _check_auth_tickets 2024-09-16T14:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 1 -- 172.21.15.189:0/2848924760 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdbf00019a0 con 0x7fdbf8132090 2024-09-16T14:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbeffff640 10 monclient: dump: 2024-09-16T14:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 5 monclient: authenticate success, global_id 4982 2024-09-16T14:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.260+0000 7fdbff97d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbeffff640 1 -- 172.21.15.189:0/2848924760 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbf0001c80 con 0x7fdbf8132090 2024-09-16T14:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbeffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbeffff640 1 -- 172.21.15.189:0/2848924760 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdbf0002050 con 0x7fdbf8132090 2024-09-16T14:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 4 set_mon_vals no callback set 2024-09-16T14:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbeffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbeffff640 10 monclient: dump: 2024-09-16T14:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 10 monclient: shutdown 2024-09-16T14:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 1 -- 172.21.15.189:0/2848924760 >> v1:172.21.15.26:6790/0 conn(0x7fdbf8132090 legacy=0x7fdbf813a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 1 -- 172.21.15.189:0/2848924760 shutdown_connections 2024-09-16T14:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 1 -- 172.21.15.189:0/2848924760 >> 172.21.15.189:0/2848924760 conn(0x7fdbf80a3320 msgr2=0x7fdbf809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 1 -- 172.21.15.189:0/2848924760 shutdown_connections 2024-09-16T14:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 1 -- 172.21.15.189:0/2848924760 wait complete. 2024-09-16T14:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 10 monclient: build_initial_monmap 2024-09-16T14:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 10 monclient: monmap: 2024-09-16T14:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 1 Processor -- start 2024-09-16T14:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 1 -- start start 2024-09-16T14:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 10 monclient: init 2024-09-16T14:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 10 monclient(hunting): picked mon.noname-b con 0x7fdbf81315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 10 monclient(hunting): picked mon.noname-a con 0x7fdbf8132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 10 monclient(hunting): picked mon.noname-c con 0x7fdbf81cade0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf809cd20 con 0x7fdbf8132090 2024-09-16T14:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf811f200 con 0x7fdbf81315a0 2024-09-16T14:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf809a5f0 con 0x7fdbf81cade0 2024-09-16T14:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 10 monclient(hunting): authenticate will time out at 1104.382180s 2024-09-16T14:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbfd719640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdbf81315a0 0x7fdbf81c4a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33306/0 (socket says 172.21.15.189:33306) 2024-09-16T14:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbfd719640 1 -- 172.21.15.189:0/1884522494 learned_addr learned my addr 172.21.15.189:0/1884522494 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483867452 0 0) 0x7fdbf809cd20 con 0x7fdbf8132090 2024-09-16T14:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient(hunting): my global_id is 84381 2024-09-16T14:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbdc003530 con 0x7fdbf8132090 2024-09-16T14:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388806183 0 0) 0x7fdbf809a5f0 con 0x7fdbf81cade0 2024-09-16T14:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient(hunting): my global_id is 4985 2024-09-16T14:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf809cd20 con 0x7fdbf81cade0 2024-09-16T14:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1306867744 0 0) 0x7fdbdc003530 con 0x7fdbf8132090 2024-09-16T14:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbf809a5f0 con 0x7fdbf8132090 2024-09-16T14:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdbe80017f0 con 0x7fdbf8132090 2024-09-16T14:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient(hunting): dump: 2024-09-16T14:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1637119151 0 0) 0x7fdbf809cd20 con 0x7fdbf81cade0 2024-09-16T14:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbdc003530 con 0x7fdbf81cade0 2024-09-16T14:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdbf0001f90 con 0x7fdbf81cade0 2024-09-16T14:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient(hunting): dump: 2024-09-16T14:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1505775479 0 0) 0x7fdbf809a5f0 con 0x7fdbf8132090 2024-09-16T14:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 >> v1:172.21.15.26:6790/0 conn(0x7fdbf81cade0 legacy=0x7fdbf81cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 >> v1:172.21.15.189:6789/0 conn(0x7fdbf81315a0 legacy=0x7fdbf81c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient: _finish_hunting 0 2024-09-16T14:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 1 monclient: found mon.a 2024-09-16T14:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf806b380 con 0x7fdbf8132090 2024-09-16T14:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient: _finish_auth 0 2024-09-16T14:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 5 monclient: authenticate success, global_id 84381 2024-09-16T14:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 10 monclient: _renew_subs 2024-09-16T14:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.264+0000 7fdbff97d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbff97d640 1 -- 172.21.15.189:0/1884522494 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf811d7f0 con 0x7fdbf8132090 2024-09-16T14:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbff97d640 10 monclient: _renew_subs 2024-09-16T14:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbff97d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbff97d640 1 -- 172.21.15.189:0/1884522494 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf8130140 con 0x7fdbf8132090 2024-09-16T14:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbe8001cc0 con 0x7fdbf8132090 2024-09-16T14:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbedffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdbe8004990 con 0x7fdbf8132090 2024-09-16T14:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbedffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbef7fe640 4 set_mon_vals no callback set 2024-09-16T14:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbedffb640 10 monclient: dump: 2024-09-16T14:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbff97d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbff97d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbff97d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:22.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbff97d640 1 -- 172.21.15.189:0/1884522494 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd8005050 con 0x7fdbf8132090 2024-09-16T14:45:22.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fdbe8004af0 con 0x7fdbf8132090 2024-09-16T14:45:22.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(57..57 src has 15..57) v4 ==== 4315+0+0 (unknown 2780822968 0 0) 0x7fdbe8043140 con 0x7fdbf8132090 2024-09-16T14:45:22.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.268+0000 7fdbfd719640 10 monclient: get_auth_request con 0x7fdbdc036310 auth_method 0 2024-09-16T14:45:22.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.280+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fdbe8014230 con 0x7fdbf8132090 2024-09-16T14:45:22.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.280+0000 7fdbedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:22.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.280+0000 7fdbedffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:22.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.440+0000 7fdbff97d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:22.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.440+0000 7fdbff97d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:22.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.440+0000 7fdbff97d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:22.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.440+0000 7fdbff97d640 1 -- 172.21.15.189:0/1884522494 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fdbd8005300 con 0x7fdbf8132090 2024-09-16T14:45:22.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.440+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v57) v1 ==== 65+0+70 (unknown 2262488454 0 3909614116) 0x7fdbe8014050 con 0x7fdbf8132090 2024-09-16T14:45:22.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.440+0000 7fdbedffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:22.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.440+0000 7fdbedffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:22.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.536+0000 7fdbff97d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:22.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.536+0000 7fdbff97d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:22.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.536+0000 7fdbff97d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:22.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.536+0000 7fdbff97d640 1 -- 172.21.15.189:0/1884522494 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fdbd8005540 con 0x7fdbf8132090 2024-09-16T14:45:22.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.536+0000 7fdbedffb640 1 -- 172.21.15.189:0/1884522494 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v57) v1 ==== 65+0+70 (unknown 2262488454 0 3909614116) 0x7fdbe8014050 con 0x7fdbf8132090 2024-09-16T14:45:22.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.536+0000 7fdbedffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:22.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.536+0000 7fdbedffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:45:22.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.536+0000 7fdbff97d640 1 -- 172.21.15.189:0/1884522494 >> v1:172.21.15.189:6812/3196161582 conn(0x7fdbdc036310 legacy=0x7fdbdc0387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:22.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.536+0000 7fdbff97d640 10 monclient: shutdown 2024-09-16T14:45:22.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.536+0000 7fdbff97d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:22.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.536+0000 7fdbff97d640 1 -- 172.21.15.189:0/1884522494 >> v1:172.21.15.26:6789/0 conn(0x7fdbf8132090 legacy=0x7fdbf812fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:22.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.536+0000 7fdbff97d640 1 -- 172.21.15.189:0/1884522494 shutdown_connections 2024-09-16T14:45:22.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.536+0000 7fdbff97d640 1 -- 172.21.15.189:0/1884522494 >> 172.21.15.189:0/1884522494 conn(0x7fdbf80a3320 msgr2=0x7fdbf809d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:22.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.536+0000 7fdbff97d640 1 -- 172.21.15.189:0/1884522494 shutdown_connections 2024-09-16T14:45:22.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:22.536+0000 7fdbff97d640 1 -- 172.21.15.189:0/1884522494 wait complete. 2024-09-16T14:45:22.555 INFO:tasks.workunit.client.0.smithi189.stdout:bar-asdf 2024-09-16T14:45:22.555 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rm foo 2024-09-16T14:45:22.555 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'does not exist' 2024-09-16T14:45:22.967 INFO:tasks.workunit.client.0.smithi189.stdout:2024-09-16T14:45:22.852+0000 7f2e5effd640 1 -- 172.21.15.189:0/2596588796 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v57) v1 ==== 103+0+0 (unknown 3293611961 0 0) 0x7f2e58012f50 con 0x7f2e681334d0 2024-09-16T14:45:22.967 INFO:tasks.workunit.client.0.smithi189.stdout:2024-09-16T14:45:22.852+0000 7f2e5effd640 10 monclient: _finish_command 2 = system:0 rule foo does not exist 2024-09-16T14:45:22.967 INFO:tasks.workunit.client.0.smithi189.stdout:2024-09-16T14:45:22.948+0000 7f2e5effd640 1 -- 172.21.15.189:0/2596588796 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v57) v1 ==== 103+0+0 (unknown 3293611961 0 0) 0x7f2e58012f50 con 0x7f2e681334d0 2024-09-16T14:45:22.967 INFO:tasks.workunit.client.0.smithi189.stdout:2024-09-16T14:45:22.948+0000 7f2e5effd640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-16T14:45:22.967 INFO:tasks.workunit.client.0.smithi189.stdout:rule foo does not exist 2024-09-16T14:45:22.968 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rm bar 2024-09-16T14:45:22.968 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'does not exist' 2024-09-16T14:45:23.356 INFO:tasks.workunit.client.0.smithi189.stdout:2024-09-16T14:45:23.248+0000 7faa16ffd640 1 -- 172.21.15.189:0/1820055201 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v57) v1 ==== 103+0+0 (unknown 2771139058 0 0) 0x7faa20011bd0 con 0x7faa2809e740 2024-09-16T14:45:23.357 INFO:tasks.workunit.client.0.smithi189.stdout:2024-09-16T14:45:23.248+0000 7faa16ffd640 10 monclient: _finish_command 2 = system:0 rule bar does not exist 2024-09-16T14:45:23.357 INFO:tasks.workunit.client.0.smithi189.stdout:2024-09-16T14:45:23.336+0000 7faa16ffd640 1 -- 172.21.15.189:0/1820055201 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v57) v1 ==== 103+0+0 (unknown 2771139058 0 0) 0x7faa20013350 con 0x7faa2809e740 2024-09-16T14:45:23.357 INFO:tasks.workunit.client.0.smithi189.stdout:2024-09-16T14:45:23.336+0000 7faa16ffd640 10 monclient: _finish_command 3 = system:0 rule bar does not exist 2024-09-16T14:45:23.357 INFO:tasks.workunit.client.0.smithi189.stdout:rule bar does not exist 2024-09-16T14:45:23.357 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rename foo-asdf foo 2024-09-16T14:45:23.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.464+0000 7f9b18ed9640 10 monclient: get_monmap_and_config 2024-09-16T14:45:23.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.464+0000 7f9b18ed9640 10 monclient: build_initial_monmap 2024-09-16T14:45:23.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.464+0000 7f9b18ed9640 10 monclient: monmap: 2024-09-16T14:45:23.470 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:23.470 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:23.470 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:23.469483+0000 2024-09-16T14:45:23.470 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:23.469483+0000 2024-09-16T14:45:23.470 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:23.470 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:23.470 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:23.470 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:23.470 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:23.470 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:23.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 1 Processor -- start 2024-09-16T14:45:23.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 1 -- start start 2024-09-16T14:45:23.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient: init 2024-09-16T14:45:23.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:23.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:23.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient(hunting): picked mon.noname-a con 0x7f9b14099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:23.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient(hunting): picked mon.noname-b con 0x7f9b1409bd30 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:23.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient(hunting): picked mon.noname-c con 0x7f9b14137500 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:23.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b14099e50 con 0x7f9b14099a70 2024-09-16T14:45:23.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b1409e510 con 0x7f9b14137500 2024-09-16T14:45:23.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b1409c2c0 con 0x7f9b1409bd30 2024-09-16T14:45:23.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:23.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient(hunting): authenticate will time out at 1105.585271s 2024-09-16T14:45:23.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b12d9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9b14137500 0x7f9b141398f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33354/0 (socket says 172.21.15.189:33354) 2024-09-16T14:45:23.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b12d9d640 1 -- 172.21.15.189:0/3712491593 learned_addr learned my addr 172.21.15.189:0/3712491593 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:23.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 1 -- 172.21.15.189:0/3712491593 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977476097 0 0) 0x7f9b1409e510 con 0x7f9b14137500 2024-09-16T14:45:23.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 10 monclient(hunting): _negotiate 2024-09-16T14:45:23.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:23.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:23.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 10 monclient(hunting): my global_id is 94117 2024-09-16T14:45:23.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 1 -- 172.21.15.189:0/3712491593 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9afc003530 con 0x7f9b14137500 2024-09-16T14:45:23.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 1 -- 172.21.15.189:0/3712491593 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1505979301 0 0) 0x7f9afc003530 con 0x7f9b14137500 2024-09-16T14:45:23.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 1 -- 172.21.15.189:0/3712491593 >> v1:172.21.15.26:6790/0 conn(0x7f9b1409bd30 legacy=0x7f9b1409de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:23.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 1 -- 172.21.15.189:0/3712491593 >> v1:172.21.15.26:6789/0 conn(0x7f9b14099a70 legacy=0x7f9b1409d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:23.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 10 monclient: _finish_hunting 0 2024-09-16T14:45:23.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 1 monclient: found mon.noname-c 2024-09-16T14:45:23.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:23.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:23.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 1 -- 172.21.15.189:0/3712491593 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1409ee60 con 0x7f9b14137500 2024-09-16T14:45:23.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 10 monclient: _finish_auth 0 2024-09-16T14:45:23.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 10 monclient: _check_auth_tickets 2024-09-16T14:45:23.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:23.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 1 -- 172.21.15.189:0/3712491593 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9b0c0019a0 con 0x7f9b14137500 2024-09-16T14:45:23.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 5 monclient: authenticate success, global_id 94117 2024-09-16T14:45:23.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:23.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:23.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:23.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 10 monclient: dump: 2024-09-16T14:45:23.480 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:23.480 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:23.480 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.480 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.480 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:23.480 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:23.480 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:23.480 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:23.480 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:23.481 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:23.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:23.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 1 -- 172.21.15.189:0/3712491593 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b0c001cc0 con 0x7f9b14137500 2024-09-16T14:45:23.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:23.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 1 -- 172.21.15.189:0/3712491593 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9b0c002050 con 0x7f9b14137500 2024-09-16T14:45:23.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:23.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:23.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b1159a640 10 monclient: dump: 2024-09-16T14:45:23.481 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:23.481 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:23.481 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.481 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.481 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:23.482 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:23.482 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:23.482 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:23.482 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:23.482 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:23.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:23.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 4 set_mon_vals no callback set 2024-09-16T14:45:23.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient: shutdown 2024-09-16T14:45:23.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:23.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 1 -- 172.21.15.189:0/3712491593 >> v1:172.21.15.189:6789/0 conn(0x7f9b14137500 legacy=0x7f9b141398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:23.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 1 -- 172.21.15.189:0/3712491593 shutdown_connections 2024-09-16T14:45:23.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 1 -- 172.21.15.189:0/3712491593 >> 172.21.15.189:0/3712491593 conn(0x7f9b140a3380 msgr2=0x7f9b1409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:23.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 1 -- 172.21.15.189:0/3712491593 shutdown_connections 2024-09-16T14:45:23.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 1 -- 172.21.15.189:0/3712491593 wait complete. 2024-09-16T14:45:23.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient: build_initial_monmap 2024-09-16T14:45:23.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient: monmap: 2024-09-16T14:45:23.483 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:23.483 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:23.483 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:23.483 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:23.483 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:23.483 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:23.483 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:23.483 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:23.483 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:23.484 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:23.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 1 Processor -- start 2024-09-16T14:45:23.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 1 -- start start 2024-09-16T14:45:23.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient: init 2024-09-16T14:45:23.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:23.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:23.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient(hunting): picked mon.noname-c con 0x7f9b14099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:23.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient(hunting): picked mon.noname-b con 0x7f9b1409bd30 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:23.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.468+0000 7f9b18ed9640 10 monclient(hunting): picked mon.noname-a con 0x7f9b14137500 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:23.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b18ed9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b140a1f30 con 0x7f9b14137500 2024-09-16T14:45:23.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b18ed9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b1411a5b0 con 0x7f9b1409bd30 2024-09-16T14:45:23.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b18ed9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b14070560 con 0x7f9b14099a70 2024-09-16T14:45:23.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b18ed9640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:23.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b18ed9640 10 monclient(hunting): authenticate will time out at 1105.588037s 2024-09-16T14:45:23.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b11d9b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9b1409bd30 0x7f9b141c96a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33362/0 (socket says 172.21.15.189:33362) 2024-09-16T14:45:23.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b11d9b640 1 -- 172.21.15.189:0/2117160984 learned_addr learned my addr 172.21.15.189:0/2117160984 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:23.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 1 -- 172.21.15.189:0/2117160984 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982141430 0 0) 0x7f9b1411a5b0 con 0x7f9b1409bd30 2024-09-16T14:45:23.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:23.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:23.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:23.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 10 monclient(hunting): my global_id is 94120 2024-09-16T14:45:23.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 1 -- 172.21.15.189:0/2117160984 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af4003530 con 0x7f9b1409bd30 2024-09-16T14:45:23.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 1 -- 172.21.15.189:0/2117160984 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1293087381 0 0) 0x7f9af4003530 con 0x7f9b1409bd30 2024-09-16T14:45:23.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 1 -- 172.21.15.189:0/2117160984 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b1411a5b0 con 0x7f9b1409bd30 2024-09-16T14:45:23.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 1 -- 172.21.15.189:0/2117160984 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9b080018b0 con 0x7f9b1409bd30 2024-09-16T14:45:23.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:23.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:23.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 10 monclient(hunting): dump: 2024-09-16T14:45:23.486 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:23.486 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:23.486 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.486 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.486 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:23.486 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:23.486 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:23.486 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:23.486 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:23.487 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:23.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:23.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 1 -- 172.21.15.189:0/2117160984 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1668679100 0 0) 0x7f9b1411a5b0 con 0x7f9b1409bd30 2024-09-16T14:45:23.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 1 -- 172.21.15.189:0/2117160984 >> v1:172.21.15.26:6790/0 conn(0x7f9b14099a70 legacy=0x7f9b1412fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:23.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 1 -- 172.21.15.189:0/2117160984 >> v1:172.21.15.26:6789/0 conn(0x7f9b14137500 legacy=0x7f9b141ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:23.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:23.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 1 monclient: found mon.b 2024-09-16T14:45:23.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:23.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:23.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 1 -- 172.21.15.189:0/2117160984 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1406b320 con 0x7f9b1409bd30 2024-09-16T14:45:23.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:23.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:23.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:23.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b18ed9640 5 monclient: authenticate success, global_id 94120 2024-09-16T14:45:23.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b18ed9640 10 monclient: _renew_subs 2024-09-16T14:45:23.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b18ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:23.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b18ed9640 1 -- 172.21.15.189:0/2117160984 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b1411a2d0 con 0x7f9b1409bd30 2024-09-16T14:45:23.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b18ed9640 10 monclient: _renew_subs 2024-09-16T14:45:23.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b18ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:23.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b18ed9640 1 -- 172.21.15.189:0/2117160984 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b14119d10 con 0x7f9b1409bd30 2024-09-16T14:45:23.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 1 -- 172.21.15.189:0/2117160984 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b08001c80 con 0x7f9b1409bd30 2024-09-16T14:45:23.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:23.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 1 -- 172.21.15.189:0/2117160984 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9b08003a70 con 0x7f9b1409bd30 2024-09-16T14:45:23.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:23.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:23.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b10d99640 4 set_mon_vals no callback set 2024-09-16T14:45:23.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 10 monclient: dump: 2024-09-16T14:45:23.489 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:23.489 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:23.489 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.489 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.489 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:23.489 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:23.489 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:23.489 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:23.489 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:23.489 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:23.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 1 -- 172.21.15.189:0/2117160984 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9b08003bd0 con 0x7f9b1409bd30 2024-09-16T14:45:23.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b02ffd640 1 -- 172.21.15.189:0/2117160984 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(57..57 src has 15..57) v4 ==== 4315+0+0 (unknown 2780822968 0 0) 0x7f9b08043490 con 0x7f9b1409bd30 2024-09-16T14:45:23.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b18ed9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:23.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b18ed9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:23.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b18ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:23.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b18ed9640 1 -- 172.21.15.189:0/2117160984 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af8005050 con 0x7f9b1409bd30 2024-09-16T14:45:23.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.472+0000 7f9b1259c640 10 monclient: get_auth_request con 0x7f9af40360b0 auth_method 0 2024-09-16T14:45:23.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.476+0000 7f9b02ffd640 1 -- 172.21.15.189:0/2117160984 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9b080043d0 con 0x7f9b1409bd30 2024-09-16T14:45:23.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.476+0000 7f9b02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:23.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.476+0000 7f9b02ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:23.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.640+0000 7f9b18ed9640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T14:45:23.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.640+0000 7f9b18ed9640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T14:45:23.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.640+0000 7f9b18ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:23.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:23.640+0000 7f9b18ed9640 1 -- 172.21.15.189:0/2117160984 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f9af8005880 con 0x7f9b1409bd30 2024-09-16T14:45:23.828 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:45:23.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 10 monclient: get_monmap_and_config 2024-09-16T14:45:23.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 10 monclient: build_initial_monmap 2024-09-16T14:45:23.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 10 monclient: monmap: 2024-09-16T14:45:23.959 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:23.959 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:23.959 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:23.958913+0000 2024-09-16T14:45:23.959 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:23.958913+0000 2024-09-16T14:45:23.960 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:23.960 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:23.960 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:23.960 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:23.960 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:23.960 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:23.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 1 Processor -- start 2024-09-16T14:45:23.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 1 -- start start 2024-09-16T14:45:23.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 10 monclient: init 2024-09-16T14:45:23.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:23.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:23.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 10 monclient(hunting): picked mon.noname-b con 0x7f976009cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:23.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 10 monclient(hunting): picked mon.noname-c con 0x7f976009e7a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:23.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 10 monclient(hunting): picked mon.noname-a con 0x7f976009d1a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:23.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f976009d5b0 con 0x7f976009d1a0 2024-09-16T14:45:23.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9760131a90 con 0x7f976009e7a0 2024-09-16T14:45:23.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f976009aff0 con 0x7f976009cdc0 2024-09-16T14:45:23.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:23.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 10 monclient(hunting): authenticate will time out at 1148.078924s 2024-09-16T14:45:23.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f976009cdc0 0x7f9760130c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60610/0 (socket says 172.21.15.26:60610) 2024-09-16T14:45:23.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975ffff640 1 -- 172.21.15.26:0/1112191308 learned_addr learned my addr 172.21.15.26:0/1112191308 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:23.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 1 -- 172.21.15.26:0/1112191308 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345915315 0 0) 0x7f976009d5b0 con 0x7f976009d1a0 2024-09-16T14:45:23.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:23.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:23.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:23.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient(hunting): my global_id is 84402 2024-09-16T14:45:23.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 1 -- 172.21.15.26:0/1112191308 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9748004640 con 0x7f976009d1a0 2024-09-16T14:45:23.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 1 -- 172.21.15.26:0/1112191308 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470345052 0 0) 0x7f976009aff0 con 0x7f976009cdc0 2024-09-16T14:45:23.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:23.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:23.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:23.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient(hunting): my global_id is 5006 2024-09-16T14:45:23.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 1 -- 172.21.15.26:0/1112191308 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f976009d5b0 con 0x7f976009cdc0 2024-09-16T14:45:23.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 1 -- 172.21.15.26:0/1112191308 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928473904 0 0) 0x7f9760131a90 con 0x7f976009e7a0 2024-09-16T14:45:23.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:23.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:23.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:23.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient(hunting): my global_id is 94123 2024-09-16T14:45:23.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 1 -- 172.21.15.26:0/1112191308 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f976009aff0 con 0x7f976009e7a0 2024-09-16T14:45:23.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 1 -- 172.21.15.26:0/1112191308 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 60806447 0 0) 0x7f9748004640 con 0x7f976009d1a0 2024-09-16T14:45:23.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 1 -- 172.21.15.26:0/1112191308 >> v1:172.21.15.26:6790/0 conn(0x7f976009cdc0 legacy=0x7f9760130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:23.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 1 -- 172.21.15.26:0/1112191308 >> v1:172.21.15.189:6789/0 conn(0x7f976009e7a0 legacy=0x7f9760131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:23.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:23.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 1 monclient: found mon.noname-a 2024-09-16T14:45:23.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:23.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:45:23.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 1 -- 172.21.15.26:0/1112191308 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976009b2d0 con 0x7f976009d1a0 2024-09-16T14:45:23.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient: _finish_auth 0 2024-09-16T14:45:23.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:23.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:23.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 1 -- 172.21.15.26:0/1112191308 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f97580019a0 con 0x7f976009d1a0 2024-09-16T14:45:23.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 5 monclient: authenticate success, global_id 84402 2024-09-16T14:45:23.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:23.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:23.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:23.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f975effd640 10 monclient: dump: 2024-09-16T14:45:23.971 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:23.971 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:23.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.971 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:23.971 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:23.971 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:23.971 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:23.971 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:23.972 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:23.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.957+0000 7f9766769640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:23.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975effd640 1 -- 172.21.15.26:0/1112191308 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9758001bc0 con 0x7f976009d1a0 2024-09-16T14:45:23.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:23.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975effd640 1 -- 172.21.15.26:0/1112191308 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9758001f10 con 0x7f976009d1a0 2024-09-16T14:45:23.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:23.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 4 set_mon_vals no callback set 2024-09-16T14:45:23.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:23.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:23.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975effd640 10 monclient: dump: 2024-09-16T14:45:23.972 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:23.972 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:23.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.973 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.973 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:23.973 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:23.973 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:23.973 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:23.973 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:23.973 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:23.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 10 monclient: shutdown 2024-09-16T14:45:23.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:23.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 1 -- 172.21.15.26:0/1112191308 >> v1:172.21.15.26:6789/0 conn(0x7f976009d1a0 legacy=0x7f9760139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:23.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 1 -- 172.21.15.26:0/1112191308 shutdown_connections 2024-09-16T14:45:23.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 1 -- 172.21.15.26:0/1112191308 >> 172.21.15.26:0/1112191308 conn(0x7f976009fb50 msgr2=0x7f976009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:23.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 1 -- 172.21.15.26:0/1112191308 shutdown_connections 2024-09-16T14:45:23.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 1 -- 172.21.15.26:0/1112191308 wait complete. 2024-09-16T14:45:23.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 10 monclient: build_initial_monmap 2024-09-16T14:45:23.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 10 monclient: monmap: 2024-09-16T14:45:23.974 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:23.974 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:23.974 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:23.974 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:23.974 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:23.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:23.974 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:23.974 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:23.975 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:23.975 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:23.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 1 Processor -- start 2024-09-16T14:45:23.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 1 -- start start 2024-09-16T14:45:23.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 10 monclient: init 2024-09-16T14:45:23.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:23.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:23.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 10 monclient(hunting): picked mon.noname-c con 0x7f976009cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:23.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 10 monclient(hunting): picked mon.noname-b con 0x7f976009d1a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:23.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 10 monclient(hunting): picked mon.noname-a con 0x7f976009e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:23.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f976011a4c0 con 0x7f976009e7a0 2024-09-16T14:45:23.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f976011a9a0 con 0x7f976009d1a0 2024-09-16T14:45:23.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f976009a240 con 0x7f976009cdc0 2024-09-16T14:45:23.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:23.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 10 monclient(hunting): authenticate will time out at 1148.082171s 2024-09-16T14:45:23.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f976009cdc0 0x7f97601196a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60622/0 (socket says 172.21.15.26:60622) 2024-09-16T14:45:23.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975ffff640 1 -- 172.21.15.26:0/241902854 learned_addr learned my addr 172.21.15.26:0/241902854 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:23.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703669290 0 0) 0x7f976011a4c0 con 0x7f976009e7a0 2024-09-16T14:45:23.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:23.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:23.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:23.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): my global_id is 84405 2024-09-16T14:45:23.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f974c003530 con 0x7f976009e7a0 2024-09-16T14:45:23.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3446118126 0 0) 0x7f976009a240 con 0x7f976009cdc0 2024-09-16T14:45:23.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:23.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:23.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:23.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): my global_id is 5009 2024-09-16T14:45:23.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f976011a4c0 con 0x7f976009cdc0 2024-09-16T14:45:23.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2868276433 0 0) 0x7f974c003530 con 0x7f976009e7a0 2024-09-16T14:45:23.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f976009a240 con 0x7f976009e7a0 2024-09-16T14:45:23.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 992943541 0 0) 0x7f976011a4c0 con 0x7f976009cdc0 2024-09-16T14:45:23.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f974c003530 con 0x7f976009cdc0 2024-09-16T14:45:23.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f97580018a0 con 0x7f976009e7a0 2024-09-16T14:45:23.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:23.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:23.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): dump: 2024-09-16T14:45:23.977 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:23.978 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:23.978 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.978 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.978 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:23.978 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:23.978 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:23.978 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:23.978 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:23.978 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:23.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:23.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f97500019e0 con 0x7f976009cdc0 2024-09-16T14:45:23.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:23.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:23.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): dump: 2024-09-16T14:45:23.979 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:23.979 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:23.979 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.979 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.979 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:23.979 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:23.979 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:23.979 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:23.979 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:23.979 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:23.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140413302 0 0) 0x7f976011a9a0 con 0x7f976009d1a0 2024-09-16T14:45:23.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:23.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient(hunting): my global_id is 94126 2024-09-16T14:45:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f976011a4c0 con 0x7f976009d1a0 2024-09-16T14:45:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2080147819 0 0) 0x7f976009a240 con 0x7f976009e7a0 2024-09-16T14:45:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 >> v1:172.21.15.26:6790/0 conn(0x7f976009cdc0 legacy=0x7f97601196a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 >> v1:172.21.15.189:6789/0 conn(0x7f976009d1a0 legacy=0x7f9760119db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:45:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 monclient: found mon.a 2024-09-16T14:45:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97600a2690 con 0x7f976009e7a0 2024-09-16T14:45:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient: _finish_auth 0 2024-09-16T14:45:23.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:45:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 5 monclient: authenticate success, global_id 84405 2024-09-16T14:45:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 10 monclient: _renew_subs 2024-09-16T14:45:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 1 -- 172.21.15.26:0/241902854 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97600a2160 con 0x7f976009e7a0 2024-09-16T14:45:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 10 monclient: _renew_subs 2024-09-16T14:45:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f9766769640 1 -- 172.21.15.26:0/241902854 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f976011cd70 con 0x7f976009e7a0 2024-09-16T14:45:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9758003c30 con 0x7f976009e7a0 2024-09-16T14:45:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9758003f40 con 0x7f976009e7a0 2024-09-16T14:45:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:23.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975cff9640 10 monclient: dump: 2024-09-16T14:45:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:23.982 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.961+0000 7f975e7fc640 4 set_mon_vals no callback set 2024-09-16T14:45:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.965+0000 7f9766769640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.965+0000 7f9766769640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:23.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.965+0000 7f9766769640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.965+0000 7f9766769640 1 -- 172.21.15.26:0/241902854 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9740005050 con 0x7f976009e7a0 2024-09-16T14:45:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.965+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f975800e9b0 con 0x7f976009e7a0 2024-09-16T14:45:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.965+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(57..57 src has 15..57) v4 ==== 4315+0+0 (unknown 2780822968 0 0) 0x7f9758042650 con 0x7f976009e7a0 2024-09-16T14:45:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.965+0000 7f975ffff640 10 monclient: get_auth_request con 0x7f974c036320 auth_method 0 2024-09-16T14:45:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.969+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f975800fe10 con 0x7f976009e7a0 2024-09-16T14:45:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.969+0000 7f975cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:23.969+0000 7f975cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:24.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.185+0000 7f9766769640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:45:24.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.185+0000 7f9766769640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:24.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.185+0000 7f9766769640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:24.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.185+0000 7f9766769640 1 -- 172.21.15.26:0/241902854 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9740005880 con 0x7f976009e7a0 2024-09-16T14:45:24.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.185+0000 7f975cff9640 1 -- 172.21.15.26:0/241902854 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3953634730) 0x7f9758011ef0 con 0x7f976009e7a0 2024-09-16T14:45:24.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.185+0000 7f975cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:24.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.185+0000 7f975cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:24.188 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:45:24.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.189+0000 7f9766769640 1 -- 172.21.15.26:0/241902854 >> v1:172.21.15.189:6812/3196161582 conn(0x7f974c036320 legacy=0x7f974c0387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.189+0000 7f9766769640 10 monclient: shutdown 2024-09-16T14:45:24.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.189+0000 7f9766769640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:24.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.189+0000 7f9766769640 1 -- 172.21.15.26:0/241902854 >> v1:172.21.15.26:6789/0 conn(0x7f976009e7a0 legacy=0x7f97601cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.189+0000 7f9766769640 1 -- 172.21.15.26:0/241902854 shutdown_connections 2024-09-16T14:45:24.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.189+0000 7f9766769640 1 -- 172.21.15.26:0/241902854 >> 172.21.15.26:0/241902854 conn(0x7f976009fb50 msgr2=0x7f97601310d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:24.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.189+0000 7f9766769640 1 -- 172.21.15.26:0/241902854 shutdown_connections 2024-09-16T14:45:24.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.189+0000 7f9766769640 1 -- 172.21.15.26:0/241902854 wait complete. 2024-09-16T14:45:24.205 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:45:24.205 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:45:24.205 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:45:24.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 10 monclient: get_monmap_and_config 2024-09-16T14:45:24.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 10 monclient: build_initial_monmap 2024-09-16T14:45:24.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 10 monclient: monmap: 2024-09-16T14:45:24.330 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:24.330 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.330 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:24.329654+0000 2024-09-16T14:45:24.330 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:24.329654+0000 2024-09-16T14:45:24.330 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:24.330 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.330 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:24.331 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:24.331 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:24.331 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 1 Processor -- start 2024-09-16T14:45:24.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 1 -- start start 2024-09-16T14:45:24.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 10 monclient: init 2024-09-16T14:45:24.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:24.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:24.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 10 monclient(hunting): picked mon.noname-c con 0x7fa6680a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:24.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 10 monclient(hunting): picked mon.noname-b con 0x7fa6680a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:24.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 10 monclient(hunting): picked mon.noname-a con 0x7fa6680a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:24.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6680a3250 con 0x7fa6680a2800 2024-09-16T14:45:24.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66809f600 con 0x7fa6680a1d10 2024-09-16T14:45:24.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6680a3830 con 0x7fa6680a0000 2024-09-16T14:45:24.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:24.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 10 monclient(hunting): authenticate will time out at 1148.449785s 2024-09-16T14:45:24.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa666d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa6680a2800 0x7fa66813d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48248/0 (socket says 172.21.15.26:48248) 2024-09-16T14:45:24.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa665d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa6680a0000 0x7fa668139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60630/0 (socket says 172.21.15.26:60630) 2024-09-16T14:45:24.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa666d9d640 1 -- 172.21.15.26:0/1296546451 learned_addr learned my addr 172.21.15.26:0/1296546451 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 1 -- 172.21.15.26:0/1296546451 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426977898 0 0) 0x7fa6680a3830 con 0x7fa6680a0000 2024-09-16T14:45:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient(hunting): my global_id is 5012 2024-09-16T14:45:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 1 -- 172.21.15.26:0/1296546451 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa650003530 con 0x7fa6680a0000 2024-09-16T14:45:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 1 -- 172.21.15.26:0/1296546451 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050314042 0 0) 0x7fa6680a3250 con 0x7fa6680a2800 2024-09-16T14:45:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient(hunting): my global_id is 84408 2024-09-16T14:45:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 1 -- 172.21.15.26:0/1296546451 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6680a3830 con 0x7fa6680a2800 2024-09-16T14:45:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 1 -- 172.21.15.26:0/1296546451 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147233684 0 0) 0x7fa66809f600 con 0x7fa6680a1d10 2024-09-16T14:45:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient(hunting): my global_id is 94129 2024-09-16T14:45:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 1 -- 172.21.15.26:0/1296546451 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6680a3250 con 0x7fa6680a1d10 2024-09-16T14:45:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 1 -- 172.21.15.26:0/1296546451 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2837670003 0 0) 0x7fa650003530 con 0x7fa6680a0000 2024-09-16T14:45:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 1 -- 172.21.15.26:0/1296546451 >> v1:172.21.15.189:6789/0 conn(0x7fa6680a1d10 legacy=0x7fa6680a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 1 -- 172.21.15.26:0/1296546451 >> v1:172.21.15.26:6789/0 conn(0x7fa6680a2800 legacy=0x7fa66813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient: _finish_hunting 0 2024-09-16T14:45:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 1 monclient: found mon.noname-b 2024-09-16T14:45:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:45:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 1 -- 172.21.15.26:0/1296546451 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66809ed60 con 0x7fa6680a0000 2024-09-16T14:45:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient: _finish_auth 0 2024-09-16T14:45:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient: _check_auth_tickets 2024-09-16T14:45:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 1 -- 172.21.15.26:0/1296546451 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa65c0015a0 con 0x7fa6680a0000 2024-09-16T14:45:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 5 monclient: authenticate success, global_id 5012 2024-09-16T14:45:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient: dump: 2024-09-16T14:45:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.343 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 1 -- 172.21.15.26:0/1296546451 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa65c0017c0 con 0x7fa6680a0000 2024-09-16T14:45:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 1 -- 172.21.15.26:0/1296546451 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa65c001b90 con 0x7fa6680a0000 2024-09-16T14:45:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66559a640 10 monclient: dump: 2024-09-16T14:45:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.344 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 4 set_mon_vals no callback set 2024-09-16T14:45:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 10 monclient: shutdown 2024-09-16T14:45:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 1 -- 172.21.15.26:0/1296546451 >> v1:172.21.15.26:6790/0 conn(0x7fa6680a0000 legacy=0x7fa668139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 1 -- 172.21.15.26:0/1296546451 shutdown_connections 2024-09-16T14:45:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 1 -- 172.21.15.26:0/1296546451 >> 172.21.15.26:0/1296546451 conn(0x7fa6680a7f30 msgr2=0x7fa6680a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 1 -- 172.21.15.26:0/1296546451 shutdown_connections 2024-09-16T14:45:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.329+0000 7fa66cf10640 1 -- 172.21.15.26:0/1296546451 wait complete. 2024-09-16T14:45:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 10 monclient: build_initial_monmap 2024-09-16T14:45:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 10 monclient: monmap: 2024-09-16T14:45:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:24.346 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:24.346 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:24.346 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:24.346 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:24.346 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:24.346 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:24.346 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 1 Processor -- start 2024-09-16T14:45:24.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 1 -- start start 2024-09-16T14:45:24.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 10 monclient: init 2024-09-16T14:45:24.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 10 monclient(hunting): picked mon.noname-a con 0x7fa6680a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 10 monclient(hunting): picked mon.noname-c con 0x7fa6680a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 10 monclient(hunting): picked mon.noname-b con 0x7fa6680a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6680a66d0 con 0x7fa6680a0000 2024-09-16T14:45:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa668131f60 con 0x7fa6680a2800 2024-09-16T14:45:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66809e500 con 0x7fa6680a1d10 2024-09-16T14:45:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 10 monclient(hunting): authenticate will time out at 1148.453065s 2024-09-16T14:45:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa6680a0000 0x7fa6681341c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48258/0 (socket says 172.21.15.26:48258) 2024-09-16T14:45:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66659c640 1 -- 172.21.15.26:0/2824048218 learned_addr learned my addr 172.21.15.26:0/2824048218 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa665d9b640 1 --1- 172.21.15.26:0/2824048218 >> v1:172.21.15.26:6790/0 conn(0x7fa6680a1d10 0x7fa6681cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60640/0 (socket says 172.21.15.26:60640) 2024-09-16T14:45:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195406257 0 0) 0x7fa6680a66d0 con 0x7fa6680a0000 2024-09-16T14:45:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): my global_id is 84411 2024-09-16T14:45:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa648003530 con 0x7fa6680a0000 2024-09-16T14:45:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255847178 0 0) 0x7fa66809e500 con 0x7fa6680a1d10 2024-09-16T14:45:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): my global_id is 5015 2024-09-16T14:45:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6680a66d0 con 0x7fa6680a1d10 2024-09-16T14:45:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3648001616 0 0) 0x7fa648003530 con 0x7fa6680a0000 2024-09-16T14:45:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa66809e500 con 0x7fa6680a0000 2024-09-16T14:45:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa658001920 con 0x7fa6680a0000 2024-09-16T14:45:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): dump: 2024-09-16T14:45:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.350 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.350 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.350 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.350 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.350 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.350 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.350 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.350 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:24.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3304199147 0 0) 0x7fa6680a66d0 con 0x7fa6680a1d10 2024-09-16T14:45:24.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa648003530 con 0x7fa6680a1d10 2024-09-16T14:45:24.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa65c001ad0 con 0x7fa6680a1d10 2024-09-16T14:45:24.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:24.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): dump: 2024-09-16T14:45:24.351 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.351 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.351 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.352 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603209973 0 0) 0x7fa668131f60 con 0x7fa6680a2800 2024-09-16T14:45:24.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): my global_id is 94132 2024-09-16T14:45:24.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6680a66d0 con 0x7fa6680a2800 2024-09-16T14:45:24.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1333051493 0 0) 0x7fa66809e500 con 0x7fa6680a0000 2024-09-16T14:45:24.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 >> v1:172.21.15.26:6790/0 conn(0x7fa6680a1d10 legacy=0x7fa6681cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 >> v1:172.21.15.189:6789/0 conn(0x7fa6680a2800 legacy=0x7fa6681d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:24.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 monclient: found mon.a 2024-09-16T14:45:24.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:24.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:24.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa668133da0 con 0x7fa6680a0000 2024-09-16T14:45:24.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:24.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:24.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:24.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 5 monclient: authenticate success, global_id 84411 2024-09-16T14:45:24.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 10 monclient: _renew_subs 2024-09-16T14:45:24.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:24.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 1 -- 172.21.15.26:0/2824048218 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa668133500 con 0x7fa6680a0000 2024-09-16T14:45:24.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 10 monclient: _renew_subs 2024-09-16T14:45:24.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:24.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 1 -- 172.21.15.26:0/2824048218 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6681d27d0 con 0x7fa6680a0000 2024-09-16T14:45:24.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa658003620 con 0x7fa6680a0000 2024-09-16T14:45:24.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:24.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa6580039b0 con 0x7fa6680a0000 2024-09-16T14:45:24.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:24.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa664d99640 4 set_mon_vals no callback set 2024-09-16T14:45:24.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient: dump: 2024-09-16T14:45:24.355 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.355 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.356 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.356 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.356 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.356 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.356 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.356 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.356 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.356 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:24.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa658003b10 con 0x7fa6680a0000 2024-09-16T14:45:24.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa66cf10640 1 -- 172.21.15.26:0/2824048218 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa64c006590 con 0x7fa64c002830 2024-09-16T14:45:24.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(57..57 src has 15..57) v4 ==== 4315+0+0 (unknown 2780822968 0 0) 0x7fa6580433a0 con 0x7fa6680a0000 2024-09-16T14:45:24.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184176470 0 0) 0x7fa64c006590 con 0x7fa64c002830 2024-09-16T14:45:24.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient(hunting): my global_id is 84414 2024-09-16T14:45:24.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa66809e500 con 0x7fa64c002830 2024-09-16T14:45:24.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa666d9d640 10 monclient: get_auth_request con 0x7fa648036490 auth_method 0 2024-09-16T14:45:24.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1165219604 0 0) 0x7fa66809e500 con 0x7fa64c002830 2024-09-16T14:45:24.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa64c006590 con 0x7fa64c002830 2024-09-16T14:45:24.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa65c004990 con 0x7fa64c002830 2024-09-16T14:45:24.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:24.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 10 monclient: dump: 2024-09-16T14:45:24.357 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.358 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.358 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.358 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.358 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.358 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.358 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.358 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.358 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.358 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2591666007 0 0) 0x7fa64c006590 con 0x7fa64c002830 2024-09-16T14:45:24.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.333+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa64c00a0e0 con 0x7fa64c002830 2024-09-16T14:45:24.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.337+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa65c004d00 con 0x7fa64c002830 2024-09-16T14:45:24.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.337+0000 7fa656ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:24.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.337+0000 7fa656ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:24.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.337+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 >> v1:172.21.15.26:6789/0 conn(0x7fa64c002830 legacy=0x7fa64c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.337+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 >> v1:172.21.15.26:6789/0 conn(0x7fa64c002830 legacy=0x7fa64c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:24.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa66cf10640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:45:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa66cf10640 1 -- 172.21.15.26:0/2824048218 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa64c009d80 con 0x7fa64c0090a0 2024-09-16T14:45:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323494002 0 0) 0x7fa64c009d80 con 0x7fa64c0090a0 2024-09-16T14:45:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 10 monclient(hunting): my global_id is 84417 2024-09-16T14:45:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa64c006590 con 0x7fa64c0090a0 2024-09-16T14:45:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3198627443 0 0) 0x7fa64c006590 con 0x7fa64c0090a0 2024-09-16T14:45:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa64c009d80 con 0x7fa64c0090a0 2024-09-16T14:45:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa65c005360 con 0x7fa64c0090a0 2024-09-16T14:45:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 10 monclient: dump: 2024-09-16T14:45:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.362 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 237200171 0 0) 0x7fa64c009d80 con 0x7fa64c0090a0 2024-09-16T14:45:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa64c00c290 con 0x7fa64c0090a0 2024-09-16T14:45:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 2177488788) 0x7fa65c0056b0 con 0x7fa64c0090a0 2024-09-16T14:45:24.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:45:24.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:24.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 >> v1:172.21.15.26:6789/0 conn(0x7fa64c0090a0 legacy=0x7fa64c009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa656ffd640 1 -- 172.21.15.26:0/2824048218 >> v1:172.21.15.26:6789/0 conn(0x7fa64c0090a0 legacy=0x7fa64c009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:24.363 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:45:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:45:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:45:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:45:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 32, 2024-09-16T14:45:24.363 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:45:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:45:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:45:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:45:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-09-16T14:45:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:45:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:45:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:45:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:24.364 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:45:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:45:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:24.365 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:45:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:45:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:45:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:45:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:45:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:45:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:45:24.366 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:45:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:45:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:45:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:45:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:45:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:45:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:45:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:45:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:45:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:45:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:24.367 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:45:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:45:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:45:24.368 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:45:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:45:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.369 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:45:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:45:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:45:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:45:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:24.370 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:45:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:45:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:45:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:45:24.371 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:45:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:45:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:45:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:45:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:24.372 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:45:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:45:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:45:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:45:24.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:45:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:45:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:45:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.375 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:45:24.375 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:45:24.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa66cf10640 1 -- 172.21.15.26:0/2824048218 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa648036490 legacy=0x7fa648038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa66cf10640 10 monclient: shutdown 2024-09-16T14:45:24.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa66cf10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:24.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa66cf10640 1 -- 172.21.15.26:0/2824048218 >> v1:172.21.15.26:6789/0 conn(0x7fa6680a0000 legacy=0x7fa6681341c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa666d9d640 1 -- 172.21.15.26:0/2824048218 reap_dead start 2024-09-16T14:45:24.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa666d9d640 1 -- 172.21.15.26:0/2824048218 reap_dead start 2024-09-16T14:45:24.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa66cf10640 1 -- 172.21.15.26:0/2824048218 shutdown_connections 2024-09-16T14:45:24.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.341+0000 7fa66cf10640 1 -- 172.21.15.26:0/2824048218 >> 172.21.15.26:0/2824048218 conn(0x7fa6680a7f30 msgr2=0x7fa66806b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:24.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.345+0000 7fa66cf10640 1 -- 172.21.15.26:0/2824048218 shutdown_connections 2024-09-16T14:45:24.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.345+0000 7fa66cf10640 1 -- 172.21.15.26:0/2824048218 wait complete. 2024-09-16T14:45:24.377 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:45:24.377 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:45:24.377 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:45:24.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 10 monclient: get_monmap_and_config 2024-09-16T14:45:24.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 10 monclient: build_initial_monmap 2024-09-16T14:45:24.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 10 monclient: monmap: 2024-09-16T14:45:24.490 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:24.491 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.491 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:24.490274+0000 2024-09-16T14:45:24.491 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:24.490274+0000 2024-09-16T14:45:24.491 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:24.491 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.491 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:24.491 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:24.491 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:24.491 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 1 Processor -- start 2024-09-16T14:45:24.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 1 -- start start 2024-09-16T14:45:24.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 10 monclient: init 2024-09-16T14:45:24.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:24.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:45:24.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 10 monclient(hunting): picked mon.noname-a con 0x7f027009a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:24.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 10 monclient(hunting): picked mon.noname-c con 0x7f027009d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:24.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 10 monclient(hunting): picked mon.noname-b con 0x7f027009b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:24.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027009dc60 con 0x7f027009a550 2024-09-16T14:45:24.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0270099ae0 con 0x7f027009d1b0 2024-09-16T14:45:24.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027009b8f0 con 0x7f027009b040 2024-09-16T14:45:24.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:24.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 10 monclient(hunting): authenticate will time out at 1148.610162s 2024-09-16T14:45:24.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f0274e5f640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f027009b040 0x7f027013a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60656/0 (socket says 172.21.15.26:60656) 2024-09-16T14:45:24.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f0274e5f640 1 -- 172.21.15.26:0/2159491069 learned_addr learned my addr 172.21.15.26:0/2159491069 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:24.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f027009a550 0x7f027009a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48272/0 (socket says 172.21.15.26:48272) 2024-09-16T14:45:24.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 1 -- 172.21.15.26:0/2159491069 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644785490 0 0) 0x7f027009b8f0 con 0x7f027009b040 2024-09-16T14:45:24.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient(hunting): my global_id is 5018 2024-09-16T14:45:24.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 1 -- 172.21.15.26:0/2159491069 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f025c003530 con 0x7f027009b040 2024-09-16T14:45:24.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 1 -- 172.21.15.26:0/2159491069 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309793944 0 0) 0x7f027009dc60 con 0x7f027009a550 2024-09-16T14:45:24.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient(hunting): my global_id is 84420 2024-09-16T14:45:24.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 1 -- 172.21.15.26:0/2159491069 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f027009b8f0 con 0x7f027009a550 2024-09-16T14:45:24.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 1 -- 172.21.15.26:0/2159491069 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3136304235 0 0) 0x7f025c003530 con 0x7f027009b040 2024-09-16T14:45:24.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 1 -- 172.21.15.26:0/2159491069 >> v1:172.21.15.189:6789/0 conn(0x7f027009d1b0 legacy=0x7f0270136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 1 -- 172.21.15.26:0/2159491069 >> v1:172.21.15.26:6789/0 conn(0x7f027009a550 legacy=0x7f027009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:45:24.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 1 monclient: found mon.noname-b 2024-09-16T14:45:24.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:24.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:45:24.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 1 -- 172.21.15.26:0/2159491069 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027009bbd0 con 0x7f027009b040 2024-09-16T14:45:24.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:45:24.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:24.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:24.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 1 -- 172.21.15.26:0/2159491069 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f02680019a0 con 0x7f027009b040 2024-09-16T14:45:24.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:24.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient: dump: 2024-09-16T14:45:24.499 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.499 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.499 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.499 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.499 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.499 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.499 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.500 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.500 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.500 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 5 monclient: authenticate success, global_id 5018 2024-09-16T14:45:24.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:24.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 1 -- 172.21.15.26:0/2159491069 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0268001c00 con 0x7f027009b040 2024-09-16T14:45:24.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:24.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 1 -- 172.21.15.26:0/2159491069 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0268001fd0 con 0x7f027009b040 2024-09-16T14:45:24.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:24.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f026f7fe640 10 monclient: dump: 2024-09-16T14:45:24.501 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.501 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.501 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.501 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.502 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.502 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.502 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.502 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.502 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.502 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:24.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 4 set_mon_vals no callback set 2024-09-16T14:45:24.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 10 monclient: shutdown 2024-09-16T14:45:24.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:24.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 1 -- 172.21.15.26:0/2159491069 >> v1:172.21.15.26:6790/0 conn(0x7f027009b040 legacy=0x7f027013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 1 -- 172.21.15.26:0/2159491069 shutdown_connections 2024-09-16T14:45:24.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 1 -- 172.21.15.26:0/2159491069 >> 172.21.15.26:0/2159491069 conn(0x7f027009fb50 msgr2=0x7f027009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:24.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 1 -- 172.21.15.26:0/2159491069 shutdown_connections 2024-09-16T14:45:24.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.489+0000 7f02768c2640 1 -- 172.21.15.26:0/2159491069 wait complete. 2024-09-16T14:45:24.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 10 monclient: build_initial_monmap 2024-09-16T14:45:24.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 10 monclient: monmap: 2024-09-16T14:45:24.504 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:24.504 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:24.504 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:24.504 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:24.504 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:24.505 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:24.505 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:24.505 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:24.505 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:24.505 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 1 Processor -- start 2024-09-16T14:45:24.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 1 -- start start 2024-09-16T14:45:24.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 10 monclient: init 2024-09-16T14:45:24.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.508+0000 7f9b02ffd640 1 -- 172.21.15.189:0/2117160984 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 v58) v1 ==== 110+0+0 (unknown 2918558110 0 0) 0x7f9b08012bc0 con 0x7f9b1409bd30 2024-09-16T14:45:24.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.508+0000 7f9b02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T14:45:24.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.508+0000 7f9b02ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:24.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:24.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:24.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 10 monclient(hunting): picked mon.noname-b con 0x7f027009a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:24.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 10 monclient(hunting): picked mon.noname-a con 0x7f027009b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:24.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 10 monclient(hunting): picked mon.noname-c con 0x7f027009d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:24.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027009a240 con 0x7f027009b040 2024-09-16T14:45:24.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02701a5f90 con 0x7f027009a550 2024-09-16T14:45:24.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02701a6c50 con 0x7f027009d1b0 2024-09-16T14:45:24.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:24.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 10 monclient(hunting): authenticate will time out at 1148.613465s 2024-09-16T14:45:24.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f0274e5f640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f027009d1b0 0x7f02701d1930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60666/0 (socket says 172.21.15.26:60666) 2024-09-16T14:45:24.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f0274e5f640 1 -- 172.21.15.26:0/3082519073 learned_addr learned my addr 172.21.15.26:0/3082519073 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:24.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f0267fff640 1 --1- 172.21.15.26:0/3082519073 >> v1:172.21.15.26:6789/0 conn(0x7f027009b040 0x7f02701ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48284/0 (socket says 172.21.15.26:48284) 2024-09-16T14:45:24.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481572712 0 0) 0x7f027009a240 con 0x7f027009b040 2024-09-16T14:45:24.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): my global_id is 84423 2024-09-16T14:45:24.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0250003530 con 0x7f027009b040 2024-09-16T14:45:24.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945046309 0 0) 0x7f02701a6c50 con 0x7f027009d1b0 2024-09-16T14:45:24.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): my global_id is 5021 2024-09-16T14:45:24.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f027009a240 con 0x7f027009d1b0 2024-09-16T14:45:24.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2402075684 0 0) 0x7f0250003530 con 0x7f027009b040 2024-09-16T14:45:24.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02701a6c50 con 0x7f027009b040 2024-09-16T14:45:24.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f02580018a0 con 0x7f027009b040 2024-09-16T14:45:24.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:24.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): dump: 2024-09-16T14:45:24.517 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.517 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.517 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.517 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.518 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.518 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.518 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.518 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.518 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.518 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:24.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4179948789 0 0) 0x7f027009a240 con 0x7f027009d1b0 2024-09-16T14:45:24.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0250003530 con 0x7f027009d1b0 2024-09-16T14:45:24.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965946019 0 0) 0x7f02701a5f90 con 0x7f027009a550 2024-09-16T14:45:24.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): my global_id is 94138 2024-09-16T14:45:24.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f027009a240 con 0x7f027009a550 2024-09-16T14:45:24.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0268001f10 con 0x7f027009d1b0 2024-09-16T14:45:24.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:24.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): dump: 2024-09-16T14:45:24.519 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.519 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.519 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.519 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.519 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.519 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.520 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.520 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.520 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.520 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 822134559 0 0) 0x7f02701a6c50 con 0x7f027009b040 2024-09-16T14:45:24.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 >> v1:172.21.15.26:6790/0 conn(0x7f027009d1b0 legacy=0x7f02701d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 >> v1:172.21.15.189:6789/0 conn(0x7f027009a550 legacy=0x7f02701b6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:45:24.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 monclient: found mon.a 2024-09-16T14:45:24.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:24.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:24.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02700a25d0 con 0x7f027009b040 2024-09-16T14:45:24.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:45:24.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:45:24.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:24.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 5 monclient: authenticate success, global_id 84423 2024-09-16T14:45:24.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 10 monclient: _renew_subs 2024-09-16T14:45:24.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:24.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 1 -- 172.21.15.26:0/3082519073 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02700a2080 con 0x7f027009b040 2024-09-16T14:45:24.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 10 monclient: _renew_subs 2024-09-16T14:45:24.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:24.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 1 -- 172.21.15.26:0/3082519073 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02701a6740 con 0x7f027009b040 2024-09-16T14:45:24.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:24.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f02768c2640 1 -- 172.21.15.26:0/3082519073 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f02701a8200 con 0x7f02701c8ee0 2024-09-16T14:45:24.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0258003620 con 0x7f027009b040 2024-09-16T14:45:24.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:24.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f02580039b0 con 0x7f027009b040 2024-09-16T14:45:24.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:24.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient: dump: 2024-09-16T14:45:24.522 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.522 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.522 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.522 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.522 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.522 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.522 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.522 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.522 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.523 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026effd640 4 set_mon_vals no callback set 2024-09-16T14:45:24.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0258003b50 con 0x7f027009b040 2024-09-16T14:45:24.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(57..57 src has 15..57) v4 ==== 4315+0+0 (unknown 2780822968 0 0) 0x7f02580432d0 con 0x7f027009b040 2024-09-16T14:45:24.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223609918 0 0) 0x7f02701a8200 con 0x7f02701c8ee0 2024-09-16T14:45:24.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 10 monclient(hunting): my global_id is 5024 2024-09-16T14:45:24.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.493+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02701a6c50 con 0x7f02701c8ee0 2024-09-16T14:45:24.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.497+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 641909206 0 0) 0x7f02701a6c50 con 0x7f02701c8ee0 2024-09-16T14:45:24.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.497+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02701a8200 con 0x7f02701c8ee0 2024-09-16T14:45:24.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.497+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0260001840 con 0x7f02701c8ee0 2024-09-16T14:45:24.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.497+0000 7f0274e5f640 10 monclient: get_auth_request con 0x7f02500362f0 auth_method 0 2024-09-16T14:45:24.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.497+0000 7f026d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.497+0000 7f026d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:24.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.497+0000 7f026d7fa640 10 monclient: dump: 2024-09-16T14:45:24.524 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.524 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.524 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.524 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.524 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.524 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.524 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.525 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.525 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.525 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.497+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 909923138 0 0) 0x7f02701a8200 con 0x7f02701c8ee0 2024-09-16T14:45:24.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.497+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f027009c220 con 0x7f02701c8ee0 2024-09-16T14:45:24.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.497+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0260001b90 con 0x7f02701c8ee0 2024-09-16T14:45:24.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.497+0000 7f026d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:24.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.497+0000 7f026d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:24.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.497+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 >> v1:172.21.15.26:6790/0 conn(0x7f02701c8ee0 legacy=0x7f027006fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.497+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 >> v1:172.21.15.26:6790/0 conn(0x7f02701c8ee0 legacy=0x7f027006fc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:24.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.501+0000 7f02768c2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:45:24.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.501+0000 7f02768c2640 1 -- 172.21.15.26:0/3082519073 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f02701c92c0 con 0x7f027002c280 2024-09-16T14:45:24.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.501+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886196978 0 0) 0x7f02701c92c0 con 0x7f027002c280 2024-09-16T14:45:24.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.501+0000 7f026d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.501+0000 7f026d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.501+0000 7f026d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.501+0000 7f026d7fa640 10 monclient(hunting): my global_id is 5027 2024-09-16T14:45:24.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.501+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02701a8200 con 0x7f027002c280 2024-09-16T14:45:24.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1072330777 0 0) 0x7f02701a8200 con 0x7f027002c280 2024-09-16T14:45:24.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02701c92c0 con 0x7f027002c280 2024-09-16T14:45:24.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0260004a50 con 0x7f027002c280 2024-09-16T14:45:24.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f026d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f026d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:24.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f026d7fa640 10 monclient: dump: 2024-09-16T14:45:24.526 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.526 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.527 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.527 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.527 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.527 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.527 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.527 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.527 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.527 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 18754337 0 0) 0x7f02701c92c0 con 0x7f027002c280 2024-09-16T14:45:24.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f02700008e0 con 0x7f027002c280 2024-09-16T14:45:24.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 4062887786) 0x7f0260004d20 con 0x7f027002c280 2024-09-16T14:45:24.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f026d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:45:24.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f026d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:24.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 >> v1:172.21.15.26:6790/0 conn(0x7f027002c280 legacy=0x7f0270070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f026d7fa640 1 -- 172.21.15.26:0/3082519073 >> v1:172.21.15.26:6790/0 conn(0x7f027002c280 legacy=0x7f0270070390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:24.528 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:45:24.528 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:45:24.528 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:45:24.528 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:45:24.528 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 32, 2024-09-16T14:45:24.528 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:45:24.528 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:45:24.528 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:45:24.528 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:45:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-09-16T14:45:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:45:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:45:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:45:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:24.529 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:45:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:45:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:24.530 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:24.531 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:24.531 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:24.531 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.531 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:45:24.531 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:45:24.531 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:45:24.531 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:45:24.531 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:45:24.531 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:45:24.531 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:45:24.531 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:45:24.531 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:45:24.531 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:45:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:45:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:45:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:45:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:45:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:45:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:45:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:45:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:24.532 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:45:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:45:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:45:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:45:24.533 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:24.534 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:24.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.534 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:24.534 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:45:24.534 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:24.534 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.534 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.534 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:45:24.534 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:45:24.534 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:24.534 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:24.534 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:24.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.535 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:45:24.535 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:45:24.535 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:24.535 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:24.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.535 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:24.535 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:45:24.535 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:24.535 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.535 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.535 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:45:24.536 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:45:24.536 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:24.536 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:24.536 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:24.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.536 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:45:24.536 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:45:24.536 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:24.536 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:24.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.536 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:24.537 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:45:24.537 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:24.537 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.537 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.537 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:45:24.537 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:45:24.537 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:24.537 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:24.537 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:24.537 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.537 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.538 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:45:24.538 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:45:24.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.538 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:24.538 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:24.538 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:45:24.538 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.538 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:24.538 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:45:24.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.538 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:24.538 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:24.539 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:45:24.539 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.539 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:24.539 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:45:24.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.539 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:24.539 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:24.539 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:45:24.539 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.539 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:24.539 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:45:24.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.539 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:24.539 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:24.540 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:45:24.540 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.540 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.540 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:45:24.540 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:45:24.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f02768c2640 1 -- 172.21.15.26:0/3082519073 >> v1:172.21.15.189:6812/3196161582 conn(0x7f02500362f0 legacy=0x7f02500387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f02768c2640 10 monclient: shutdown 2024-09-16T14:45:24.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f02768c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:24.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f02768c2640 1 -- 172.21.15.26:0/3082519073 >> v1:172.21.15.26:6789/0 conn(0x7f027009b040 legacy=0x7f02701ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f0274e5f640 1 -- 172.21.15.26:0/3082519073 reap_dead start 2024-09-16T14:45:24.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f0274e5f640 1 -- 172.21.15.26:0/3082519073 reap_dead start 2024-09-16T14:45:24.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f02768c2640 1 -- 172.21.15.26:0/3082519073 shutdown_connections 2024-09-16T14:45:24.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f02768c2640 1 -- 172.21.15.26:0/3082519073 >> 172.21.15.26:0/3082519073 conn(0x7f027009fb50 msgr2=0x7f02700a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:24.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f02768c2640 1 -- 172.21.15.26:0/3082519073 shutdown_connections 2024-09-16T14:45:24.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.505+0000 7f02768c2640 1 -- 172.21.15.26:0/3082519073 wait complete. 2024-09-16T14:45:24.542 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:45:24.542 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:45:24.542 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:45:24.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.604+0000 7f9b18ed9640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T14:45:24.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.604+0000 7f9b18ed9640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T14:45:24.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.604+0000 7f9b18ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:24.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.604+0000 7f9b18ed9640 1 -- 172.21.15.189:0/2117160984 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f9af8005ac0 con 0x7f9b1409bd30 2024-09-16T14:45:24.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.604+0000 7f9b02ffd640 1 -- 172.21.15.189:0/2117160984 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v58) v1 ==== 134+0+0 (unknown 1148654276 0 0) 0x7f9b08014750 con 0x7f9b1409bd30 2024-09-16T14:45:24.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.604+0000 7f9b02ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T14:45:24.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.604+0000 7f9b02ffd640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo' 2024-09-16T14:45:24.608 INFO:tasks.workunit.client.0.smithi189.stderr:already renamed to 'foo' 2024-09-16T14:45:24.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.604+0000 7f9b18ed9640 1 -- 172.21.15.189:0/2117160984 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9af40360b0 legacy=0x7f9af4038570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.604+0000 7f9b18ed9640 10 monclient: shutdown 2024-09-16T14:45:24.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.604+0000 7f9b18ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:24.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.604+0000 7f9b18ed9640 1 -- 172.21.15.189:0/2117160984 >> v1:172.21.15.189:6789/0 conn(0x7f9b1409bd30 legacy=0x7f9b141c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.604+0000 7f9b18ed9640 1 -- 172.21.15.189:0/2117160984 shutdown_connections 2024-09-16T14:45:24.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.604+0000 7f9b18ed9640 1 -- 172.21.15.189:0/2117160984 >> 172.21.15.189:0/2117160984 conn(0x7f9b140a3380 msgr2=0x7f9b14132ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:24.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.604+0000 7f9b18ed9640 1 -- 172.21.15.189:0/2117160984 shutdown_connections 2024-09-16T14:45:24.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.608+0000 7f9b18ed9640 1 -- 172.21.15.189:0/2117160984 wait complete. 2024-09-16T14:45:24.624 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rename foo-asdf foo 2024-09-16T14:45:24.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 10 monclient: get_monmap_and_config 2024-09-16T14:45:24.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 10 monclient: build_initial_monmap 2024-09-16T14:45:24.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 10 monclient: monmap: 2024-09-16T14:45:24.659 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:24.659 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.659 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:24.658765+0000 2024-09-16T14:45:24.659 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:24.658765+0000 2024-09-16T14:45:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:24.660 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 1 Processor -- start 2024-09-16T14:45:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 1 -- start start 2024-09-16T14:45:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 10 monclient: init 2024-09-16T14:45:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:45:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 10 monclient(hunting): picked mon.noname-a con 0x7f67d40a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 10 monclient(hunting): picked mon.noname-c con 0x7f67d40a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 10 monclient(hunting): picked mon.noname-b con 0x7f67d40a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d40a3250 con 0x7f67d40a1d10 2024-09-16T14:45:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d409f600 con 0x7f67d40a0000 2024-09-16T14:45:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d40a3830 con 0x7f67d40a2800 2024-09-16T14:45:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 10 monclient(hunting): authenticate will time out at 1148.778776s 2024-09-16T14:45:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67d97e2640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f67d40a1d10 0x7f67d40a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48290/0 (socket says 172.21.15.26:48290) 2024-09-16T14:45:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67d9fe3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f67d40a2800 0x7f67d413d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60680/0 (socket says 172.21.15.26:60680) 2024-09-16T14:45:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67d97e2640 1 -- 172.21.15.26:0/2533380083 learned_addr learned my addr 172.21.15.26:0/2533380083 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 1 -- 172.21.15.26:0/2533380083 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736774657 0 0) 0x7f67d40a3250 con 0x7f67d40a1d10 2024-09-16T14:45:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient(hunting): my global_id is 84426 2024-09-16T14:45:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 1 -- 172.21.15.26:0/2533380083 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c0003530 con 0x7f67d40a1d10 2024-09-16T14:45:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 1 -- 172.21.15.26:0/2533380083 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658464797 0 0) 0x7f67d40a3830 con 0x7f67d40a2800 2024-09-16T14:45:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient(hunting): my global_id is 5030 2024-09-16T14:45:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 1 -- 172.21.15.26:0/2533380083 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d40a3250 con 0x7f67d40a2800 2024-09-16T14:45:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 1 -- 172.21.15.26:0/2533380083 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320547916 0 0) 0x7f67d409f600 con 0x7f67d40a0000 2024-09-16T14:45:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient(hunting): my global_id is 94141 2024-09-16T14:45:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 1 -- 172.21.15.26:0/2533380083 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d40a3830 con 0x7f67d40a0000 2024-09-16T14:45:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 1 -- 172.21.15.26:0/2533380083 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3800790289 0 0) 0x7f67d40a3250 con 0x7f67d40a2800 2024-09-16T14:45:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 1 -- 172.21.15.26:0/2533380083 >> v1:172.21.15.189:6789/0 conn(0x7f67d40a0000 legacy=0x7f67d4139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 1 -- 172.21.15.26:0/2533380083 >> v1:172.21.15.26:6789/0 conn(0x7f67d40a1d10 legacy=0x7f67d40a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient: _finish_hunting 0 2024-09-16T14:45:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 1 monclient: found mon.noname-b 2024-09-16T14:45:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:45:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 1 -- 172.21.15.26:0/2533380083 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d409ed60 con 0x7f67d40a2800 2024-09-16T14:45:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient: _finish_auth 0 2024-09-16T14:45:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient: _check_auth_tickets 2024-09-16T14:45:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 5 monclient: authenticate success, global_id 5030 2024-09-16T14:45:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 1 -- 172.21.15.26:0/2533380083 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67cc001980 con 0x7f67d40a2800 2024-09-16T14:45:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient: dump: 2024-09-16T14:45:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.672 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 1 -- 172.21.15.26:0/2533380083 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67cc001ba0 con 0x7f67d40a2800 2024-09-16T14:45:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67dba46640 4 set_mon_vals no callback set 2024-09-16T14:45:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 1 -- 172.21.15.26:0/2533380083 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67cc001fb0 con 0x7f67d40a2800 2024-09-16T14:45:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.657+0000 7f67cbfff640 10 monclient: dump: 2024-09-16T14:45:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.674 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 10 monclient: shutdown 2024-09-16T14:45:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 1 -- 172.21.15.26:0/2533380083 >> v1:172.21.15.26:6790/0 conn(0x7f67d40a2800 legacy=0x7f67d413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 1 -- 172.21.15.26:0/2533380083 shutdown_connections 2024-09-16T14:45:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 1 -- 172.21.15.26:0/2533380083 >> 172.21.15.26:0/2533380083 conn(0x7f67d40a7f30 msgr2=0x7f67d40a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 1 -- 172.21.15.26:0/2533380083 shutdown_connections 2024-09-16T14:45:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 1 -- 172.21.15.26:0/2533380083 wait complete. 2024-09-16T14:45:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 10 monclient: build_initial_monmap 2024-09-16T14:45:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 10 monclient: monmap: 2024-09-16T14:45:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:24.675 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 1 Processor -- start 2024-09-16T14:45:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 1 -- start start 2024-09-16T14:45:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 10 monclient: init 2024-09-16T14:45:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 10 monclient(hunting): picked mon.noname-b con 0x7f67d40a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 10 monclient(hunting): picked mon.noname-a con 0x7f67d40a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 10 monclient(hunting): picked mon.noname-c con 0x7f67d40a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d413ad70 con 0x7f67d40a1d10 2024-09-16T14:45:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d4139f90 con 0x7f67d40a0000 2024-09-16T14:45:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d40a6650 con 0x7f67d40a2800 2024-09-16T14:45:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 10 monclient(hunting): authenticate will time out at 1148.781828s 2024-09-16T14:45:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67d9fe3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f67d40a2800 0x7f67d41d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60692/0 (socket says 172.21.15.26:60692) 2024-09-16T14:45:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67d8fe1640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f67d40a1d10 0x7f67d4139880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48304/0 (socket says 172.21.15.26:48304) 2024-09-16T14:45:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67d9fe3640 1 -- 172.21.15.26:0/1172604408 learned_addr learned my addr 172.21.15.26:0/1172604408 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892916410 0 0) 0x7f67d40a6650 con 0x7f67d40a2800 2024-09-16T14:45:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): my global_id is 5033 2024-09-16T14:45:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b8003530 con 0x7f67d40a2800 2024-09-16T14:45:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121230402 0 0) 0x7f67d413ad70 con 0x7f67d40a1d10 2024-09-16T14:45:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): my global_id is 84429 2024-09-16T14:45:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d40a6650 con 0x7f67d40a1d10 2024-09-16T14:45:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284059086 0 0) 0x7f67d4139f90 con 0x7f67d40a0000 2024-09-16T14:45:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): my global_id is 94144 2024-09-16T14:45:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d413ad70 con 0x7f67d40a0000 2024-09-16T14:45:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1587675369 0 0) 0x7f67b8003530 con 0x7f67d40a2800 2024-09-16T14:45:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67d4139f90 con 0x7f67d40a2800 2024-09-16T14:45:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3543522596 0 0) 0x7f67d40a6650 con 0x7f67d40a1d10 2024-09-16T14:45:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67b8003530 con 0x7f67d40a1d10 2024-09-16T14:45:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67cc001f70 con 0x7f67d40a2800 2024-09-16T14:45:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): dump: 2024-09-16T14:45:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.680 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67d00017a0 con 0x7f67d40a1d10 2024-09-16T14:45:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient(hunting): dump: 2024-09-16T14:45:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.682 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.682 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.682 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.682 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.682 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.682 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3416864281 0 0) 0x7f67d4139f90 con 0x7f67d40a2800 2024-09-16T14:45:24.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 >> v1:172.21.15.189:6789/0 conn(0x7f67d40a0000 legacy=0x7f67d4139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 >> v1:172.21.15.26:6789/0 conn(0x7f67d40a1d10 legacy=0x7f67d4139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:45:24.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 monclient: found mon.c 2024-09-16T14:45:24.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:24.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:24.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d413cb10 con 0x7f67d40a2800 2024-09-16T14:45:24.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient: _finish_auth 0 2024-09-16T14:45:24.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:24.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:24.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 5 monclient: authenticate success, global_id 5033 2024-09-16T14:45:24.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 10 monclient: _renew_subs 2024-09-16T14:45:24.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:24.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 1 -- 172.21.15.26:0/1172604408 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d413cdf0 con 0x7f67d40a2800 2024-09-16T14:45:24.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 10 monclient: _renew_subs 2024-09-16T14:45:24.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:24.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 1 -- 172.21.15.26:0/1172604408 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d413b4b0 con 0x7f67d40a2800 2024-09-16T14:45:24.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67cc004f30 con 0x7f67d40a2800 2024-09-16T14:45:24.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:24.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67cc005240 con 0x7f67d40a2800 2024-09-16T14:45:24.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:24.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67cb7fe640 4 set_mon_vals no callback set 2024-09-16T14:45:24.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 10 monclient: dump: 2024-09-16T14:45:24.684 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.684 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.684 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.684 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.685 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f67cc00fcb0 con 0x7f67d40a2800 2024-09-16T14:45:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67dba46640 1 -- 172.21.15.26:0/1172604408 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f67b4006590 con 0x7f67b4002830 2024-09-16T14:45:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(58..58 src has 15..58) v4 ==== 4310+0+0 (unknown 3701594958 0 0) 0x7f67cc0439a0 con 0x7f67d40a2800 2024-09-16T14:45:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.661+0000 7f67d8fe1640 10 monclient: get_auth_request con 0x7f67b8036450 auth_method 0 2024-09-16T14:45:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346770990 0 0) 0x7f67b4006590 con 0x7f67b4002830 2024-09-16T14:45:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 10 monclient(hunting): my global_id is 94147 2024-09-16T14:45:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d4139f90 con 0x7f67b4002830 2024-09-16T14:45:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 310501477 0 0) 0x7f67d4139f90 con 0x7f67b4002830 2024-09-16T14:45:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67b4006590 con 0x7f67b4002830 2024-09-16T14:45:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67c4001860 con 0x7f67b4002830 2024-09-16T14:45:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 10 monclient: dump: 2024-09-16T14:45:24.687 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.687 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.687 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.687 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.687 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.687 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.687 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.687 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.687 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.687 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 35613355 0 0) 0x7f67b4006590 con 0x7f67b4002830 2024-09-16T14:45:24.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f67b400a0e0 con 0x7f67b4002830 2024-09-16T14:45:24.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f67c4001b90 con 0x7f67b4002830 2024-09-16T14:45:24.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:24.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:24.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 >> v1:172.21.15.189:6789/0 conn(0x7f67b4002830 legacy=0x7f67b4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.665+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 >> v1:172.21.15.189:6789/0 conn(0x7f67b4002830 legacy=0x7f67b4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:24.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.669+0000 7f67dba46640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:45:24.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.669+0000 7f67dba46640 1 -- 172.21.15.26:0/1172604408 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f67b4009da0 con 0x7f67b40090c0 2024-09-16T14:45:24.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600332528 0 0) 0x7f67b4009da0 con 0x7f67b40090c0 2024-09-16T14:45:24.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 10 monclient(hunting): my global_id is 94150 2024-09-16T14:45:24.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b4006590 con 0x7f67b40090c0 2024-09-16T14:45:24.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 204512623 0 0) 0x7f67b4006590 con 0x7f67b40090c0 2024-09-16T14:45:24.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67b4009da0 con 0x7f67b40090c0 2024-09-16T14:45:24.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67c4001d90 con 0x7f67b40090c0 2024-09-16T14:45:24.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:24.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 10 monclient: dump: 2024-09-16T14:45:24.690 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.690 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.690 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.690 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.691 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.691 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.691 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.691 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.691 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.691 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1594880064 0 0) 0x7f67b4009da0 con 0x7f67b40090c0 2024-09-16T14:45:24.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f67b400c290 con 0x7f67b40090c0 2024-09-16T14:45:24.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 2911930106) 0x7f67c4005f30 con 0x7f67b40090c0 2024-09-16T14:45:24.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:45:24.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:24.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 >> v1:172.21.15.189:6789/0 conn(0x7f67b40090c0 legacy=0x7f67b4009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67c9ffb640 1 -- 172.21.15.26:0/1172604408 >> v1:172.21.15.189:6789/0 conn(0x7f67b40090c0 legacy=0x7f67b4009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:24.692 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:45:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:45:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:45:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:45:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 32, 2024-09-16T14:45:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:45:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:45:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:45:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:45:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-09-16T14:45:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:45:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:45:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:45:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:45:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:45:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:45:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:45:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:45:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:45:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:45:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:45:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:45:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:45:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:45:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:45:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:45:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:45:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:45:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:45:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:45:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:45:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:45:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:24.697 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:24.697 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:24.697 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:24.697 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:24.697 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:24.697 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:24.697 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:24.697 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:24.697 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:45:24.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.697 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:45:24.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.697 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:45:24.698 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:45:24.698 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:24.698 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:24.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.698 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:24.698 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:45:24.698 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:24.698 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.698 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.698 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:45:24.698 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:45:24.698 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:24.699 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:24.699 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:24.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.699 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:45:24.699 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:45:24.699 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:24.699 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:24.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.699 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:24.699 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:45:24.699 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:24.699 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.699 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.700 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:45:24.700 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:45:24.700 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:24.700 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:24.700 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:24.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.701 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:45:24.701 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:45:24.701 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:24.701 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:24.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.701 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:24.701 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:45:24.701 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:24.702 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.702 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.702 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:45:24.702 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:45:24.702 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:24.702 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:24.702 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:24.702 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.702 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.702 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:45:24.702 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:45:24.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.703 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:24.703 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:24.703 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:45:24.703 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.703 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:24.703 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:45:24.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:24.703 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:24.703 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:24.703 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:45:24.703 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:24.703 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:24.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:24.703 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:45:24.704 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:45:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67dba46640 1 -- 172.21.15.26:0/1172604408 >> v1:172.21.15.189:6812/3196161582 conn(0x7f67b8036450 legacy=0x7f67b8038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67dba46640 10 monclient: shutdown 2024-09-16T14:45:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67dba46640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67dba46640 1 -- 172.21.15.26:0/1172604408 >> v1:172.21.15.26:6790/0 conn(0x7f67d40a2800 legacy=0x7f67d41d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67d9fe3640 1 -- 172.21.15.26:0/1172604408 reap_dead start 2024-09-16T14:45:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67d9fe3640 1 -- 172.21.15.26:0/1172604408 reap_dead start 2024-09-16T14:45:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67dba46640 1 -- 172.21.15.26:0/1172604408 shutdown_connections 2024-09-16T14:45:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67dba46640 1 -- 172.21.15.26:0/1172604408 >> 172.21.15.26:0/1172604408 conn(0x7f67d40a7f30 msgr2=0x7f67d413e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:24.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67dba46640 1 -- 172.21.15.26:0/1172604408 shutdown_connections 2024-09-16T14:45:24.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.673+0000 7f67dba46640 1 -- 172.21.15.26:0/1172604408 wait complete. 2024-09-16T14:45:24.706 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:45:24.706 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:45:24.706 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:45:24.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd80e2640 10 monclient: get_monmap_and_config 2024-09-16T14:45:24.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd80e2640 10 monclient: build_initial_monmap 2024-09-16T14:45:24.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd80e2640 10 monclient: monmap: 2024-09-16T14:45:24.740 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:24.740 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.740 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:24.732045+0000 2024-09-16T14:45:24.740 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:24.732045+0000 2024-09-16T14:45:24.740 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:24.740 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:24.740 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:24.740 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:24.740 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:24.740 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd80e2640 1 Processor -- start 2024-09-16T14:45:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd80e2640 1 -- start start 2024-09-16T14:45:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd80e2640 10 monclient: init 2024-09-16T14:45:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd80e2640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd80e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd80e2640 10 monclient(hunting): picked mon.noname-c con 0x7f3cd0099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd80e2640 10 monclient(hunting): picked mon.noname-b con 0x7f3cd009c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd80e2640 10 monclient(hunting): picked mon.noname-a con 0x7f3cd009e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd80e2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd0099e50 con 0x7f3cd009e6d0 2024-09-16T14:45:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd80e2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd009dbd0 con 0x7f3cd0099a70 2024-09-16T14:45:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd80e2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd009ee20 con 0x7f3cd009c550 2024-09-16T14:45:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd80e2640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd80e2640 10 monclient(hunting): authenticate will time out at 1106.847705s 2024-09-16T14:45:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd5e7e640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3cd0099a70 0x7f3cd009dfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33368/0 (socket says 172.21.15.189:33368) 2024-09-16T14:45:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.728+0000 7f3cd5e7e640 1 -- 172.21.15.189:0/1686385822 learned_addr learned my addr 172.21.15.189:0/1686385822 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 1 -- 172.21.15.189:0/1686385822 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116041630 0 0) 0x7f3cd009dbd0 con 0x7f3cd0099a70 2024-09-16T14:45:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient(hunting): my global_id is 94153 2024-09-16T14:45:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 1 -- 172.21.15.189:0/1686385822 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cbc003530 con 0x7f3cd0099a70 2024-09-16T14:45:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 1 -- 172.21.15.189:0/1686385822 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802194398 0 0) 0x7f3cd0099e50 con 0x7f3cd009e6d0 2024-09-16T14:45:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient(hunting): my global_id is 84432 2024-09-16T14:45:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 1 -- 172.21.15.189:0/1686385822 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cd009dbd0 con 0x7f3cd009e6d0 2024-09-16T14:45:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 1 -- 172.21.15.189:0/1686385822 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4020664938 0 0) 0x7f3cbc003530 con 0x7f3cd0099a70 2024-09-16T14:45:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 1 -- 172.21.15.189:0/1686385822 >> v1:172.21.15.26:6790/0 conn(0x7f3cd009c550 legacy=0x7f3cd01379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 1 -- 172.21.15.189:0/1686385822 >> v1:172.21.15.26:6789/0 conn(0x7f3cd009e6d0 legacy=0x7f3cd013b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient: _finish_hunting 0 2024-09-16T14:45:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 1 monclient: found mon.noname-c 2024-09-16T14:45:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 1 -- 172.21.15.189:0/1686385822 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd009f100 con 0x7f3cd0099a70 2024-09-16T14:45:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient: _finish_auth 0 2024-09-16T14:45:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient: _check_auth_tickets 2024-09-16T14:45:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 1 -- 172.21.15.189:0/1686385822 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ccc001620 con 0x7f3cd0099a70 2024-09-16T14:45:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 5 monclient: authenticate success, global_id 94153 2024-09-16T14:45:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient: dump: 2024-09-16T14:45:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.745 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 1 -- 172.21.15.189:0/1686385822 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ccc001890 con 0x7f3cd0099a70 2024-09-16T14:45:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 1 -- 172.21.15.189:0/1686385822 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ccc001c20 con 0x7f3cd0099a70 2024-09-16T14:45:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 4 set_mon_vals no callback set 2024-09-16T14:45:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd4e7c640 10 monclient: dump: 2024-09-16T14:45:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.746 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 10 monclient: shutdown 2024-09-16T14:45:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 1 -- 172.21.15.189:0/1686385822 >> v1:172.21.15.189:6789/0 conn(0x7f3cd0099a70 legacy=0x7f3cd009dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 1 -- 172.21.15.189:0/1686385822 shutdown_connections 2024-09-16T14:45:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 1 -- 172.21.15.189:0/1686385822 >> 172.21.15.189:0/1686385822 conn(0x7f3cd00a3380 msgr2=0x7f3cd009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 1 -- 172.21.15.189:0/1686385822 shutdown_connections 2024-09-16T14:45:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 1 -- 172.21.15.189:0/1686385822 wait complete. 2024-09-16T14:45:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 10 monclient: build_initial_monmap 2024-09-16T14:45:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 10 monclient: monmap: 2024-09-16T14:45:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:24.748 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 1 Processor -- start 2024-09-16T14:45:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 1 -- start start 2024-09-16T14:45:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 10 monclient: init 2024-09-16T14:45:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 10 monclient(hunting): picked mon.noname-a con 0x7f3cd0099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 10 monclient(hunting): picked mon.noname-c con 0x7f3cd009c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 10 monclient(hunting): picked mon.noname-b con 0x7f3cd009e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd009cf20 con 0x7f3cd0099a70 2024-09-16T14:45:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd006fa40 con 0x7f3cd009e6d0 2024-09-16T14:45:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd006cec0 con 0x7f3cd009c550 2024-09-16T14:45:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 10 monclient(hunting): authenticate will time out at 1106.850464s 2024-09-16T14:45:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd667f640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3cd009e6d0 0x7f3cd01d2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33372/0 (socket says 172.21.15.189:33372) 2024-09-16T14:45:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd667f640 1 -- 172.21.15.189:0/1715778118 learned_addr learned my addr 172.21.15.189:0/1715778118 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186162210 0 0) 0x7f3cd006fa40 con 0x7f3cd009e6d0 2024-09-16T14:45:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): my global_id is 94156 2024-09-16T14:45:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cb4003560 con 0x7f3cd009e6d0 2024-09-16T14:45:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031396589 0 0) 0x7f3cd009cf20 con 0x7f3cd0099a70 2024-09-16T14:45:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): my global_id is 84435 2024-09-16T14:45:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cd006fa40 con 0x7f3cd0099a70 2024-09-16T14:45:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023831260 0 0) 0x7f3cd006cec0 con 0x7f3cd009c550 2024-09-16T14:45:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): my global_id is 5039 2024-09-16T14:45:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cd009cf20 con 0x7f3cd009c550 2024-09-16T14:45:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 853623070 0 0) 0x7f3cb4003560 con 0x7f3cd009e6d0 2024-09-16T14:45:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cd006cec0 con 0x7f3cd009e6d0 2024-09-16T14:45:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3cc8001c60 con 0x7f3cd009e6d0 2024-09-16T14:45:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): dump: 2024-09-16T14:45:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.752 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2299020069 0 0) 0x7f3cd006fa40 con 0x7f3cd0099a70 2024-09-16T14:45:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cb4003560 con 0x7f3cd0099a70 2024-09-16T14:45:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ccc001440 con 0x7f3cd0099a70 2024-09-16T14:45:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): dump: 2024-09-16T14:45:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.753 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 146010975 0 0) 0x7f3cd009cf20 con 0x7f3cd009c550 2024-09-16T14:45:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cd006fa40 con 0x7f3cd009c550 2024-09-16T14:45:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3cc0001840 con 0x7f3cd009c550 2024-09-16T14:45:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient(hunting): dump: 2024-09-16T14:45:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.755 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.755 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:24.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1991268204 0 0) 0x7f3cd006cec0 con 0x7f3cd009e6d0 2024-09-16T14:45:24.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 >> v1:172.21.15.26:6790/0 conn(0x7f3cd009c550 legacy=0x7f3cd01ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 >> v1:172.21.15.26:6789/0 conn(0x7f3cd0099a70 legacy=0x7f3cd01c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:45:24.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 monclient: found mon.b 2024-09-16T14:45:24.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:24.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:24.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd0124840 con 0x7f3cd009e6d0 2024-09-16T14:45:24.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient: _finish_auth 0 2024-09-16T14:45:24.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:45:24.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:24.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 5 monclient: authenticate success, global_id 94156 2024-09-16T14:45:24.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 10 monclient: _renew_subs 2024-09-16T14:45:24.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:24.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 1 -- 172.21.15.189:0/1715778118 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd006b320 con 0x7f3cd009e6d0 2024-09-16T14:45:24.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 10 monclient: _renew_subs 2024-09-16T14:45:24.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:24.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cd80e2640 1 -- 172.21.15.189:0/1715778118 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd0125d00 con 0x7f3cd009e6d0 2024-09-16T14:45:24.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cc8001ff0 con 0x7f3cd009e6d0 2024-09-16T14:45:24.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:24.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3cc8003df0 con 0x7f3cd009e6d0 2024-09-16T14:45:24.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:24.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc67fc640 10 monclient: dump: 2024-09-16T14:45:24.756 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:24.757 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.757 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.757 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.757 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.757 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:24.757 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.757 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.757 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.757 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:24.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.732+0000 7f3cc7fff640 4 set_mon_vals no callback set 2024-09-16T14:45:24.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.736+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3cc8003f50 con 0x7f3cd009e6d0 2024-09-16T14:45:24.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.736+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(58..58 src has 15..58) v4 ==== 4310+0+0 (unknown 3701594958 0 0) 0x7f3cc8043780 con 0x7f3cd009e6d0 2024-09-16T14:45:24.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.736+0000 7f3cd5e7e640 10 monclient: get_auth_request con 0x7f3cb40365d0 auth_method 0 2024-09-16T14:45:24.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.736+0000 7f3cd80e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:24.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.736+0000 7f3cd80e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:24.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.736+0000 7f3cd80e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:24.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.736+0000 7f3cd80e2640 1 -- 172.21.15.189:0/1715778118 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd006a0d0 con 0x7f3cd009e6d0 2024-09-16T14:45:24.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.740+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3cc8014a40 con 0x7f3cd009e6d0 2024-09-16T14:45:24.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.740+0000 7f3cc67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:24.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.740+0000 7f3cc67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a33a49640 10 monclient: get_monmap_and_config 2024-09-16T14:45:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a33a49640 10 monclient: build_initial_monmap 2024-09-16T14:45:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a33a49640 10 monclient: monmap: 2024-09-16T14:45:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:24.815420+0000 2024-09-16T14:45:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:24.815420+0000 2024-09-16T14:45:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:24.816 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a33a49640 1 Processor -- start 2024-09-16T14:45:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a33a49640 1 -- start start 2024-09-16T14:45:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a33a49640 10 monclient: init 2024-09-16T14:45:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a33a49640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a33a49640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a33a49640 10 monclient(hunting): picked mon.noname-a con 0x7f0a2c0a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a33a49640 10 monclient(hunting): picked mon.noname-b con 0x7f0a2c0a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a33a49640 10 monclient(hunting): picked mon.noname-c con 0x7f0a2c0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a33a49640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2c0a3250 con 0x7f0a2c0a1d10 2024-09-16T14:45:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a33a49640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2c09f600 con 0x7f0a2c0a2800 2024-09-16T14:45:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a33a49640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2c0a3830 con 0x7f0a2c0a0000 2024-09-16T14:45:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a33a49640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a33a49640 10 monclient(hunting): authenticate will time out at 1148.935517s 2024-09-16T14:45:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a317e5640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0a2c0a1d10 0x7f0a2c0a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48314/0 (socket says 172.21.15.26:48314) 2024-09-16T14:45:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a317e5640 1 -- 172.21.15.26:0/2767135243 learned_addr learned my addr 172.21.15.26:0/2767135243 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a23fff640 1 -- 172.21.15.26:0/2767135243 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065330879 0 0) 0x7f0a2c0a3830 con 0x7f0a2c0a0000 2024-09-16T14:45:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a23fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a23fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a23fff640 10 monclient(hunting): my global_id is 5042 2024-09-16T14:45:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a23fff640 1 -- 172.21.15.26:0/2767135243 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a18003530 con 0x7f0a2c0a0000 2024-09-16T14:45:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a23fff640 1 -- 172.21.15.26:0/2767135243 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248990693 0 0) 0x7f0a2c0a3250 con 0x7f0a2c0a1d10 2024-09-16T14:45:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a23fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a23fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.813+0000 7f0a23fff640 10 monclient(hunting): my global_id is 84438 2024-09-16T14:45:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 1 -- 172.21.15.26:0/2767135243 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a2c0a3830 con 0x7f0a2c0a1d10 2024-09-16T14:45:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 1 -- 172.21.15.26:0/2767135243 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 559883117 0 0) 0x7f0a18003530 con 0x7f0a2c0a0000 2024-09-16T14:45:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 1 -- 172.21.15.26:0/2767135243 >> v1:172.21.15.189:6789/0 conn(0x7f0a2c0a2800 legacy=0x7f0a2c13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 1 -- 172.21.15.26:0/2767135243 >> v1:172.21.15.26:6789/0 conn(0x7f0a2c0a1d10 legacy=0x7f0a2c0a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 10 monclient: _finish_hunting 0 2024-09-16T14:45:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 1 monclient: found mon.noname-b 2024-09-16T14:45:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:45:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 1 -- 172.21.15.26:0/2767135243 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2c09ed60 con 0x7f0a2c0a0000 2024-09-16T14:45:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 10 monclient: _finish_auth 0 2024-09-16T14:45:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 10 monclient: _check_auth_tickets 2024-09-16T14:45:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 1 -- 172.21.15.26:0/2767135243 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0a1c001620 con 0x7f0a2c0a0000 2024-09-16T14:45:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 10 monclient: dump: 2024-09-16T14:45:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.826 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 5 monclient: authenticate success, global_id 5042 2024-09-16T14:45:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 1 -- 172.21.15.26:0/2767135243 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a1c0018c0 con 0x7f0a2c0a0000 2024-09-16T14:45:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 1 -- 172.21.15.26:0/2767135243 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0a1c001c90 con 0x7f0a2c0a0000 2024-09-16T14:45:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 4 set_mon_vals no callback set 2024-09-16T14:45:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a23fff640 10 monclient: dump: 2024-09-16T14:45:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.828 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 10 monclient: shutdown 2024-09-16T14:45:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 1 -- 172.21.15.26:0/2767135243 >> v1:172.21.15.26:6790/0 conn(0x7f0a2c0a0000 legacy=0x7f0a2c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 1 -- 172.21.15.26:0/2767135243 shutdown_connections 2024-09-16T14:45:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 1 -- 172.21.15.26:0/2767135243 >> 172.21.15.26:0/2767135243 conn(0x7f0a2c0a7f30 msgr2=0x7f0a2c0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 1 -- 172.21.15.26:0/2767135243 shutdown_connections 2024-09-16T14:45:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 1 -- 172.21.15.26:0/2767135243 wait complete. 2024-09-16T14:45:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 10 monclient: build_initial_monmap 2024-09-16T14:45:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 10 monclient: monmap: 2024-09-16T14:45:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:24.829 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 1 Processor -- start 2024-09-16T14:45:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 1 -- start start 2024-09-16T14:45:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 10 monclient: init 2024-09-16T14:45:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 10 monclient(hunting): picked mon.noname-b con 0x7f0a2c0a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:24.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 10 monclient(hunting): picked mon.noname-c con 0x7f0a2c0a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:24.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 10 monclient(hunting): picked mon.noname-a con 0x7f0a2c0a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:24.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2c132d40 con 0x7f0a2c0a2800 2024-09-16T14:45:24.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2c09e500 con 0x7f0a2c0a0000 2024-09-16T14:45:24.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2c0a6920 con 0x7f0a2c0a1d10 2024-09-16T14:45:24.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:24.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 10 monclient(hunting): authenticate will time out at 1148.938426s 2024-09-16T14:45:24.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a31fe6640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0a2c0a2800 0x7f0a2c1d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48330/0 (socket says 172.21.15.26:48330) 2024-09-16T14:45:24.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a30fe4640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0a2c0a1d10 0x7f0a2c132630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60708/0 (socket says 172.21.15.26:60708) 2024-09-16T14:45:24.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a31fe6640 1 -- 172.21.15.26:0/2861168369 learned_addr learned my addr 172.21.15.26:0/2861168369 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:24.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697422802 0 0) 0x7f0a2c132d40 con 0x7f0a2c0a2800 2024-09-16T14:45:24.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): my global_id is 84441 2024-09-16T14:45:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a10003530 con 0x7f0a2c0a2800 2024-09-16T14:45:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873266274 0 0) 0x7f0a2c0a6920 con 0x7f0a2c0a1d10 2024-09-16T14:45:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): my global_id is 5045 2024-09-16T14:45:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a2c132d40 con 0x7f0a2c0a1d10 2024-09-16T14:45:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 875209250 0 0) 0x7f0a10003530 con 0x7f0a2c0a2800 2024-09-16T14:45:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a2c0a6920 con 0x7f0a2c0a2800 2024-09-16T14:45:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0a24001c80 con 0x7f0a2c0a2800 2024-09-16T14:45:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): dump: 2024-09-16T14:45:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.832 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545052836 0 0) 0x7f0a2c09e500 con 0x7f0a2c0a0000 2024-09-16T14:45:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): my global_id is 94162 2024-09-16T14:45:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a10003530 con 0x7f0a2c0a0000 2024-09-16T14:45:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2121344674 0 0) 0x7f0a2c132d40 con 0x7f0a2c0a1d10 2024-09-16T14:45:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a2c09e500 con 0x7f0a2c0a1d10 2024-09-16T14:45:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0a1c001bd0 con 0x7f0a2c0a1d10 2024-09-16T14:45:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient(hunting): dump: 2024-09-16T14:45:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.834 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.834 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.834 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.834 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.834 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.834 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.834 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.834 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1647037932 0 0) 0x7f0a2c0a6920 con 0x7f0a2c0a2800 2024-09-16T14:45:24.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 >> v1:172.21.15.26:6790/0 conn(0x7f0a2c0a1d10 legacy=0x7f0a2c132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 >> v1:172.21.15.189:6789/0 conn(0x7f0a2c0a0000 legacy=0x7f0a2c131f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:45:24.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 monclient: found mon.a 2024-09-16T14:45:24.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:24.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:24.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2c0a63f0 con 0x7f0a2c0a2800 2024-09-16T14:45:24.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient: _finish_auth 0 2024-09-16T14:45:24.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:24.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:24.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 5 monclient: authenticate success, global_id 84441 2024-09-16T14:45:24.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 10 monclient: _renew_subs 2024-09-16T14:45:24.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:24.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 1 -- 172.21.15.26:0/2861168369 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2c136070 con 0x7f0a2c0a2800 2024-09-16T14:45:24.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 10 monclient: _renew_subs 2024-09-16T14:45:24.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:24.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a33a49640 1 -- 172.21.15.26:0/2861168369 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a2c135d90 con 0x7f0a2c0a2800 2024-09-16T14:45:24.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a24002010 con 0x7f0a2c0a2800 2024-09-16T14:45:24.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:24.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0a24003df0 con 0x7f0a2c0a2800 2024-09-16T14:45:24.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:24.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:24.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a21ffb640 10 monclient: dump: 2024-09-16T14:45:24.836 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:24.836 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:24.836 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.836 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:24.836 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:24.836 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:24.836 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:24.837 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:24.837 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:24.837 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:24.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.817+0000 7f0a237fe640 4 set_mon_vals no callback set 2024-09-16T14:45:24.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.821+0000 7f0a33a49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:24.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.821+0000 7f0a33a49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:24.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.821+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0a24003f50 con 0x7f0a2c0a2800 2024-09-16T14:45:24.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.821+0000 7f0a33a49640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:24.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.821+0000 7f0a33a49640 1 -- 172.21.15.26:0/2861168369 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a0c005050 con 0x7f0a2c0a2800 2024-09-16T14:45:24.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.821+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(58..58 src has 15..58) v4 ==== 4310+0+0 (unknown 3701594958 0 0) 0x7f0a24043800 con 0x7f0a2c0a2800 2024-09-16T14:45:24.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.821+0000 7f0a317e5640 10 monclient: get_auth_request con 0x7f0a100363f0 auth_method 0 2024-09-16T14:45:24.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.825+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0a24014ae0 con 0x7f0a2c0a2800 2024-09-16T14:45:24.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.825+0000 7f0a21ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:24.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:24.825+0000 7f0a21ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:24.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.892+0000 7f3cd80e2640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T14:45:24.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.892+0000 7f3cd80e2640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T14:45:24.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.892+0000 7f3cd80e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:24.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.892+0000 7f3cd80e2640 1 -- 172.21.15.189:0/1715778118 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f3cd009be70 con 0x7f3cd009e6d0 2024-09-16T14:45:24.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.892+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v58) v1 ==== 134+0+0 (unknown 1148654276 0 0) 0x7f3cc8012eb0 con 0x7f3cd009e6d0 2024-09-16T14:45:24.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.892+0000 7f3cc67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T14:45:24.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.892+0000 7f3cc67fc640 10 monclient: _finish_command 2 = system:0 already renamed to 'foo' 2024-09-16T14:45:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.980+0000 7f3cd80e2640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T14:45:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.980+0000 7f3cd80e2640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T14:45:24.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.980+0000 7f3cd80e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:24.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.980+0000 7f3cd80e2640 1 -- 172.21.15.189:0/1715778118 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f3cd009b280 con 0x7f3cd009e6d0 2024-09-16T14:45:24.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.984+0000 7f3cc67fc640 1 -- 172.21.15.189:0/1715778118 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v58) v1 ==== 134+0+0 (unknown 1148654276 0 0) 0x7f3cc8014630 con 0x7f3cd009e6d0 2024-09-16T14:45:24.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.984+0000 7f3cc67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T14:45:24.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.984+0000 7f3cc67fc640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo' 2024-09-16T14:45:24.987 INFO:tasks.workunit.client.0.smithi189.stderr:already renamed to 'foo' 2024-09-16T14:45:24.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.984+0000 7f3cd80e2640 1 -- 172.21.15.189:0/1715778118 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3cb40365d0 legacy=0x7f3cb4038a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.984+0000 7f3cd80e2640 10 monclient: shutdown 2024-09-16T14:45:24.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.984+0000 7f3cd80e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:24.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.984+0000 7f3cd80e2640 1 -- 172.21.15.189:0/1715778118 >> v1:172.21.15.189:6789/0 conn(0x7f3cd009e6d0 legacy=0x7f3cd01d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:24.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.984+0000 7f3cd80e2640 1 -- 172.21.15.189:0/1715778118 shutdown_connections 2024-09-16T14:45:24.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.984+0000 7f3cd80e2640 1 -- 172.21.15.189:0/1715778118 >> 172.21.15.189:0/1715778118 conn(0x7f3cd00a3380 msgr2=0x7f3cd006d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:24.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.984+0000 7f3cd80e2640 1 -- 172.21.15.189:0/1715778118 shutdown_connections 2024-09-16T14:45:24.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:24.984+0000 7f3cd80e2640 1 -- 172.21.15.189:0/1715778118 wait complete. 2024-09-16T14:45:25.002 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rename bar-asdf bar 2024-09-16T14:45:25.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:25.041+0000 7f0a33a49640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:45:25.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:25.041+0000 7f0a33a49640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:45:25.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:25.041+0000 7f0a33a49640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:25.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:25.041+0000 7f0a33a49640 1 -- 172.21.15.26:0/2861168369 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f0a0c005b10 con 0x7f0a2c0a2800 2024-09-16T14:45:25.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:25.045+0000 7f0a21ffb640 1 -- 172.21.15.26:0/2861168369 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f0a24012f50 con 0x7f0a2c0a2800 2024-09-16T14:45:25.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:25.045+0000 7f0a21ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:45:25.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:25.045+0000 7f0a21ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:25.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.045+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.045+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=12,kv=11,logm=60} crc {auth=468672706,config=692619927,health=1535303502,kv=1867599010,logm=1956791727}) 2024-09-16T14:45:25.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.045+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=12,kv=10,logm=63} crc {auth=367045908,config=692619927,health=1535303502,kv=2084167376,logm=4273437977}) 2024-09-16T14:45:25.049 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.045+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.049 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.045+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=12,kv=11,logm=60} crc {auth=468672706,config=692619927,health=1535303502,kv=1867599010,logm=1956791727}) 2024-09-16T14:45:25.049 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.045+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=10,kv=11,logm=62} crc {auth=1789576486,config=692619927,health=1784096193,kv=1867599010,logm=2247513164}) 2024-09-16T14:45:25.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f0a33a49640 1 -- 172.21.15.26:0/2861168369 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0a100363f0 legacy=0x7f0a100388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:25.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f0a33a49640 10 monclient: shutdown 2024-09-16T14:45:25.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f0a33a49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:25.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f0a33a49640 1 -- 172.21.15.26:0/2861168369 >> v1:172.21.15.26:6789/0 conn(0x7f0a2c0a2800 legacy=0x7f0a2c1d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:25.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f0a33a49640 1 -- 172.21.15.26:0/2861168369 shutdown_connections 2024-09-16T14:45:25.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f0a33a49640 1 -- 172.21.15.26:0/2861168369 >> 172.21.15.26:0/2861168369 conn(0x7f0a2c0a7f30 msgr2=0x7f0a2c06b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:25.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f0a33a49640 1 -- 172.21.15.26:0/2861168369 shutdown_connections 2024-09-16T14:45:25.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f0a33a49640 1 -- 172.21.15.26:0/2861168369 wait complete. 2024-09-16T14:45:25.051 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.051 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=104860194}) 2024-09-16T14:45:25.051 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=363174234}) 2024-09-16T14:45:25.051 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.051 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=104860194}) 2024-09-16T14:45:25.051 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3849707279}) 2024-09-16T14:45:25.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4143930179}) 2024-09-16T14:45:25.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3004946151}) 2024-09-16T14:45:25.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4143930179}) 2024-09-16T14:45:25.053 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.049+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2766269119}) 2024-09-16T14:45:25.054 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.053+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.053+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2875957771}) 2024-09-16T14:45:25.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.053+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2647204115}) 2024-09-16T14:45:25.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.053+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.053+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2875957771}) 2024-09-16T14:45:25.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.053+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2206536723}) 2024-09-16T14:45:25.056 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.053+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.057 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.053+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=887517337}) 2024-09-16T14:45:25.057 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.053+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3547990545}) 2024-09-16T14:45:25.057 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.053+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.057 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.053+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=887517337}) 2024-09-16T14:45:25.057 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.053+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314277872}) 2024-09-16T14:45:25.058 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.057+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.059 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.057+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3577809585}) 2024-09-16T14:45:25.059 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.057+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=96467172}) 2024-09-16T14:45:25.059 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.057+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.059 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.057+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3577809585}) 2024-09-16T14:45:25.059 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.057+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3812384518}) 2024-09-16T14:45:25.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.057+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.057+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3491879151}) 2024-09-16T14:45:25.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.057+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=394975596}) 2024-09-16T14:45:25.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.057+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.057+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3491879151}) 2024-09-16T14:45:25.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.057+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1376938421}) 2024-09-16T14:45:25.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.061+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.061+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=105743594}) 2024-09-16T14:45:25.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.061+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498241765}) 2024-09-16T14:45:25.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.061+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.061+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=105743594}) 2024-09-16T14:45:25.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.061+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2976536323}) 2024-09-16T14:45:25.065 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.061+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.065 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.061+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=63,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3} crc {logm=477087470,mdsmap=1531151339,mgr=1924764865,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2789503687,mon_config_key=1518680965,monmap=970648238}) 2024-09-16T14:45:25.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.061+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=65,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=2} crc {logm=178628495,mdsmap=3140514009,mgr=273342918,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2789503687,mon_config_key=1518680965,monmap=2765269217}) 2024-09-16T14:45:25.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.061+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.061+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=63,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3} crc {logm=477087470,mdsmap=1531151339,mgr=1924764865,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2789503687,mon_config_key=1518680965,monmap=970648238}) 2024-09-16T14:45:25.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.065+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=60,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=4} crc {logm=2183796608,mdsmap=1531151339,mgr=2120425906,mgr_metadata=2109538692,mgrstat=1288188141,mon_config_key=1518680965,monmap=970648238,osd_metadata=784722430}) 2024-09-16T14:45:25.067 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:45:25.069 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.065+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.069 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.065+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=85} crc {osd_metadata=3424249799,osd_pg_creating=3600978466,osd_snap=3794734670,osdmap=3640054095}) 2024-09-16T14:45:25.069 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.065+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=76} crc {osd_metadata=3424249799,osd_pg_creating=3600978466,osd_snap=3794734670,osdmap=3628473877}) 2024-09-16T14:45:25.069 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.065+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:45:25.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.065+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=85} crc {osd_metadata=3424249799,osd_pg_creating=3600978466,osd_snap=3794734670,osdmap=3640054095}) 2024-09-16T14:45:25.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:45:25.065+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=81} crc {osd_metadata=3424249799,osd_pg_creating=3600978466,osd_snap=3794734670,osdmap=2487143242}) 2024-09-16T14:45:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 10 monclient: get_monmap_and_config 2024-09-16T14:45:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 10 monclient: build_initial_monmap 2024-09-16T14:45:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 10 monclient: monmap: 2024-09-16T14:45:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:25.122260+0000 2024-09-16T14:45:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:25.122260+0000 2024-09-16T14:45:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:25.123 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 1 Processor -- start 2024-09-16T14:45:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 1 -- start start 2024-09-16T14:45:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 10 monclient: init 2024-09-16T14:45:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 10 monclient(hunting): picked mon.noname-c con 0x7f65b80a1a90 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 10 monclient(hunting): picked mon.noname-b con 0x7f65b80a2580 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 10 monclient(hunting): picked mon.noname-a con 0x7f65b809fd40 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b809df00 con 0x7f65b809fd40 2024-09-16T14:45:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b809f0a0 con 0x7f65b80a1a90 2024-09-16T14:45:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b80a3040 con 0x7f65b80a2580 2024-09-16T14:45:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 10 monclient(hunting): authenticate will time out at 1107.238724s 2024-09-16T14:45:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bdee0640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f65b80a1a90 0x7f65b80a1e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33388/0 (socket says 172.21.15.189:33388) 2024-09-16T14:45:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bdee0640 1 -- 172.21.15.189:0/263173666 learned_addr learned my addr 172.21.15.189:0/263173666 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 1 -- 172.21.15.189:0/263173666 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2818329729 0 0) 0x7f65b809f0a0 con 0x7f65b80a1a90 2024-09-16T14:45:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient(hunting): _negotiate 2024-09-16T14:45:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient(hunting): my global_id is 94165 2024-09-16T14:45:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 1 -- 172.21.15.189:0/263173666 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65a4003530 con 0x7f65b80a1a90 2024-09-16T14:45:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 1 -- 172.21.15.189:0/263173666 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908233133 0 0) 0x7f65b80a3040 con 0x7f65b80a2580 2024-09-16T14:45:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient(hunting): _negotiate 2024-09-16T14:45:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient(hunting): my global_id is 5048 2024-09-16T14:45:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 1 -- 172.21.15.189:0/263173666 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b809f0a0 con 0x7f65b80a2580 2024-09-16T14:45:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 1 -- 172.21.15.189:0/263173666 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3778313713 0 0) 0x7f65a4003530 con 0x7f65b80a1a90 2024-09-16T14:45:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 1 -- 172.21.15.189:0/263173666 >> v1:172.21.15.26:6790/0 conn(0x7f65b80a2580 legacy=0x7f65b8139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 1 -- 172.21.15.189:0/263173666 >> v1:172.21.15.26:6789/0 conn(0x7f65b809fd40 legacy=0x7f65b813d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient: _finish_hunting 0 2024-09-16T14:45:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 1 monclient: found mon.noname-c 2024-09-16T14:45:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 1 -- 172.21.15.189:0/263173666 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b809eae0 con 0x7f65b80a1a90 2024-09-16T14:45:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient: _finish_auth 0 2024-09-16T14:45:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient: _check_auth_tickets 2024-09-16T14:45:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 1 -- 172.21.15.189:0/263173666 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f65b40019a0 con 0x7f65b80a1a90 2024-09-16T14:45:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient: dump: 2024-09-16T14:45:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:25.130 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 5 monclient: authenticate success, global_id 94165 2024-09-16T14:45:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 1 -- 172.21.15.189:0/263173666 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65b4001c10 con 0x7f65b80a1a90 2024-09-16T14:45:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 1 -- 172.21.15.189:0/263173666 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f65b4001fe0 con 0x7f65b80a1a90 2024-09-16T14:45:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65c0144640 4 set_mon_vals no callback set 2024-09-16T14:45:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.120+0000 7f65bcede640 10 monclient: dump: 2024-09-16T14:45:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:25.132 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient: shutdown 2024-09-16T14:45:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 1 -- 172.21.15.189:0/263173666 >> v1:172.21.15.189:6789/0 conn(0x7f65b80a1a90 legacy=0x7f65b80a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 1 -- 172.21.15.189:0/263173666 shutdown_connections 2024-09-16T14:45:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 1 -- 172.21.15.189:0/263173666 >> 172.21.15.189:0/263173666 conn(0x7f65b80a7a70 msgr2=0x7f65b80a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 1 -- 172.21.15.189:0/263173666 shutdown_connections 2024-09-16T14:45:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 1 -- 172.21.15.189:0/263173666 wait complete. 2024-09-16T14:45:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient: build_initial_monmap 2024-09-16T14:45:25.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient: monmap: 2024-09-16T14:45:25.133 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:25.133 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:25.133 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:25.133 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:25.133 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:25.133 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:25.133 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:25.133 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:25.133 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:25.133 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:25.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 1 Processor -- start 2024-09-16T14:45:25.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 1 -- start start 2024-09-16T14:45:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient: init 2024-09-16T14:45:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient(hunting): picked mon.noname-c con 0x7f65b809fd40 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient(hunting): picked mon.noname-b con 0x7f65b80a1a90 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient(hunting): picked mon.noname-a con 0x7f65b80a2580 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b8122a90 con 0x7f65b80a2580 2024-09-16T14:45:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b8124e60 con 0x7f65b80a1a90 2024-09-16T14:45:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b80a5d60 con 0x7f65b809fd40 2024-09-16T14:45:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient(hunting): authenticate will time out at 1107.241936s 2024-09-16T14:45:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65bd6df640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f65b80a1a90 0x7f65b81ce7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33394/0 (socket says 172.21.15.189:33394) 2024-09-16T14:45:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65bd6df640 1 -- 172.21.15.189:0/2437662044 learned_addr learned my addr 172.21.15.189:0/2437662044 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88893849 0 0) 0x7f65b8124e60 con 0x7f65b80a1a90 2024-09-16T14:45:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient(hunting): my global_id is 94168 2024-09-16T14:45:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f659c003530 con 0x7f65b80a1a90 2024-09-16T14:45:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633360103 0 0) 0x7f65b8122a90 con 0x7f65b80a2580 2024-09-16T14:45:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient(hunting): my global_id is 84447 2024-09-16T14:45:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b8124e60 con 0x7f65b80a2580 2024-09-16T14:45:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850820371 0 0) 0x7f65b80a5d60 con 0x7f65b809fd40 2024-09-16T14:45:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient(hunting): my global_id is 5051 2024-09-16T14:45:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b8122a90 con 0x7f65b809fd40 2024-09-16T14:45:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1029672443 0 0) 0x7f659c003530 con 0x7f65b80a1a90 2024-09-16T14:45:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65b80a5d60 con 0x7f65b80a1a90 2024-09-16T14:45:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f65a8001920 con 0x7f65b80a1a90 2024-09-16T14:45:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient(hunting): dump: 2024-09-16T14:45:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:25.141 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3330772178 0 0) 0x7f65b8124e60 con 0x7f65b80a2580 2024-09-16T14:45:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f659c003530 con 0x7f65b80a2580 2024-09-16T14:45:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 45242938 0 0) 0x7f65b80a5d60 con 0x7f65b80a1a90 2024-09-16T14:45:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 >> v1:172.21.15.26:6790/0 conn(0x7f65b809fd40 legacy=0x7f65b81c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 >> v1:172.21.15.26:6789/0 conn(0x7f65b80a2580 legacy=0x7f65b81d1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:45:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 monclient: found mon.b 2024-09-16T14:45:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b80a62b0 con 0x7f65b80a1a90 2024-09-16T14:45:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient: _finish_auth 0 2024-09-16T14:45:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:45:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 5 monclient: authenticate success, global_id 94168 2024-09-16T14:45:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient: _renew_subs 2024-09-16T14:45:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 1 -- 172.21.15.189:0/2437662044 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b81248a0 con 0x7f65b80a1a90 2024-09-16T14:45:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient: _renew_subs 2024-09-16T14:45:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 1 -- 172.21.15.189:0/2437662044 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b81231e0 con 0x7f65b80a1a90 2024-09-16T14:45:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65a8003660 con 0x7f65b80a1a90 2024-09-16T14:45:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f65a8003970 con 0x7f65b80a1a90 2024-09-16T14:45:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:25.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:25.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65af7fe640 4 set_mon_vals no callback set 2024-09-16T14:45:25.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 10 monclient: dump: 2024-09-16T14:45:25.144 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:25.144 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:25.144 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:25.144 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:25.144 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:25.145 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65c0144640 1 -- 172.21.15.189:0/2437662044 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a0005050 con 0x7f65b80a1a90 2024-09-16T14:45:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f65a8003ad0 con 0x7f65b80a1a90 2024-09-16T14:45:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(58..58 src has 15..58) v4 ==== 4310+0+0 (unknown 3701594958 0 0) 0x7f65a8043430 con 0x7f65b80a1a90 2024-09-16T14:45:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.124+0000 7f65bdee0640 10 monclient: get_auth_request con 0x7f659c036410 auth_method 0 2024-09-16T14:45:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.132+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f65a80146f0 con 0x7f65b80a1a90 2024-09-16T14:45:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.132+0000 7f65ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:25.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.132+0000 7f65ae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.288+0000 7f65c0144640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T14:45:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.288+0000 7f65c0144640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T14:45:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.288+0000 7f65c0144640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.288+0000 7f65c0144640 1 -- 172.21.15.189:0/2437662044 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"} v 0) v1 -- 0x7f65a0005880 con 0x7f65b80a1a90 2024-09-16T14:45:25.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.644+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}]=0 v59) v1 ==== 110+0+0 (unknown 3253558377 0 0) 0x7f65a801cce0 con 0x7f65b80a1a90 2024-09-16T14:45:25.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.644+0000 7f65ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T14:45:25.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.644+0000 7f65ae7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:25.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.744+0000 7f65c0144640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T14:45:25.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.744+0000 7f65c0144640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T14:45:25.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.744+0000 7f65c0144640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:25.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.744+0000 7f65c0144640 1 -- 172.21.15.189:0/2437662044 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"} v 0) v1 -- 0x7f65a0005300 con 0x7f65b80a1a90 2024-09-16T14:45:25.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.748+0000 7f65ae7fc640 1 -- 172.21.15.189:0/2437662044 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}]=0 already renamed to 'bar' v59) v1 ==== 134+0+0 (unknown 3573704075 0 0) 0x7f65a8012b60 con 0x7f65b80a1a90 2024-09-16T14:45:25.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.748+0000 7f65ae7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T14:45:25.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.748+0000 7f65ae7fc640 10 monclient: _finish_command 3 = system:0 already renamed to 'bar' 2024-09-16T14:45:25.821 INFO:tasks.workunit.client.0.smithi189.stderr:already renamed to 'bar' 2024-09-16T14:45:25.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.748+0000 7f65c0144640 1 -- 172.21.15.189:0/2437662044 >> v1:172.21.15.189:6812/3196161582 conn(0x7f659c036410 legacy=0x7f659c0388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:25.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.748+0000 7f65c0144640 10 monclient: shutdown 2024-09-16T14:45:25.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.748+0000 7f65c0144640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:25.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.748+0000 7f65c0144640 1 -- 172.21.15.189:0/2437662044 >> v1:172.21.15.189:6789/0 conn(0x7f65b80a1a90 legacy=0x7f65b81ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:25.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.748+0000 7f65c0144640 1 -- 172.21.15.189:0/2437662044 shutdown_connections 2024-09-16T14:45:25.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.748+0000 7f65c0144640 1 -- 172.21.15.189:0/2437662044 >> 172.21.15.189:0/2437662044 conn(0x7f65b80a7a70 msgr2=0x7f65b806b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:25.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.748+0000 7f65c0144640 1 -- 172.21.15.189:0/2437662044 shutdown_connections 2024-09-16T14:45:25.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.748+0000 7f65c0144640 1 -- 172.21.15.189:0/2437662044 wait complete. 2024-09-16T14:45:25.821 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls 2024-09-16T14:45:25.822 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false grep foo-asdf 2024-09-16T14:45:25.822 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:45:25.822 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep foo-asdf 2024-09-16T14:45:25.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f96c3f640 10 monclient: get_monmap_and_config 2024-09-16T14:45:25.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f96c3f640 10 monclient: build_initial_monmap 2024-09-16T14:45:25.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f96c3f640 10 monclient: monmap: 2024-09-16T14:45:25.879 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:25.879 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:25.879 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:25.878973+0000 2024-09-16T14:45:25.879 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:25.878973+0000 2024-09-16T14:45:25.879 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:25.879 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:25.879 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:25.880 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:25.880 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:25.880 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:25.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f96c3f640 1 Processor -- start 2024-09-16T14:45:25.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f96c3f640 1 -- start start 2024-09-16T14:45:25.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f96c3f640 10 monclient: init 2024-09-16T14:45:25.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f96c3f640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:25.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f96c3f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:25.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f96c3f640 10 monclient(hunting): picked mon.noname-a con 0x7f9f901315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:25.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f96c3f640 10 monclient(hunting): picked mon.noname-b con 0x7f9f9009b320 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:25.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f96c3f640 10 monclient(hunting): picked mon.noname-c con 0x7f9f90132090 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:25.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f96c3f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9009edd0 con 0x7f9f901315a0 2024-09-16T14:45:25.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f96c3f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9009bc00 con 0x7f9f90132090 2024-09-16T14:45:25.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f96c3f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9009e900 con 0x7f9f9009b320 2024-09-16T14:45:25.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f96c3f640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:25.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f96c3f640 10 monclient(hunting): authenticate will time out at 1107.995168s 2024-09-16T14:45:25.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f951dc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9f90132090 0x7f9f9013a200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33402/0 (socket says 172.21.15.189:33402) 2024-09-16T14:45:25.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f951dc640 1 -- 172.21.15.189:0/896607360 learned_addr learned my addr 172.21.15.189:0/896607360 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:25.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 1 -- 172.21.15.189:0/896607360 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107472640 0 0) 0x7f9f9009bc00 con 0x7f9f90132090 2024-09-16T14:45:25.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:45:25.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:25.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:25.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 10 monclient(hunting): my global_id is 94171 2024-09-16T14:45:25.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 1 -- 172.21.15.189:0/896607360 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f80003530 con 0x7f9f90132090 2024-09-16T14:45:25.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 1 -- 172.21.15.189:0/896607360 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 577903741 0 0) 0x7f9f80003530 con 0x7f9f90132090 2024-09-16T14:45:25.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 1 -- 172.21.15.189:0/896607360 >> v1:172.21.15.26:6790/0 conn(0x7f9f9009b320 legacy=0x7f9f90136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:25.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 1 -- 172.21.15.189:0/896607360 >> v1:172.21.15.26:6789/0 conn(0x7f9f901315a0 legacy=0x7f9f90131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:25.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:45:25.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 1 monclient: found mon.noname-c 2024-09-16T14:45:25.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:25.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:25.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 1 -- 172.21.15.189:0/896607360 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9009b920 con 0x7f9f90132090 2024-09-16T14:45:25.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:45:25.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:25.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:25.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 1 -- 172.21.15.189:0/896607360 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f880019a0 con 0x7f9f90132090 2024-09-16T14:45:25.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.876+0000 7f9f8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:25.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:25.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8f7fe640 10 monclient: dump: 2024-09-16T14:45:25.886 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:25.886 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:25.886 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:25.886 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:25.886 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:25.886 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:25.886 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:25.886 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:25.886 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:25.886 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:25.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 5 monclient: authenticate success, global_id 94171 2024-09-16T14:45:25.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:25.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8f7fe640 1 -- 172.21.15.189:0/896607360 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f88001cc0 con 0x7f9f90132090 2024-09-16T14:45:25.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:25.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8f7fe640 1 -- 172.21.15.189:0/896607360 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f88002050 con 0x7f9f90132090 2024-09-16T14:45:25.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:25.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 4 set_mon_vals no callback set 2024-09-16T14:45:25.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:25.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:25.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8f7fe640 10 monclient: dump: 2024-09-16T14:45:25.887 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:25.887 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:25.887 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:25.887 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:25.887 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:25.888 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:25.888 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:25.888 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:25.888 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:25.888 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:25.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 10 monclient: shutdown 2024-09-16T14:45:25.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:25.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 1 -- 172.21.15.189:0/896607360 >> v1:172.21.15.189:6789/0 conn(0x7f9f90132090 legacy=0x7f9f9013a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:25.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 1 -- 172.21.15.189:0/896607360 shutdown_connections 2024-09-16T14:45:25.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 1 -- 172.21.15.189:0/896607360 >> 172.21.15.189:0/896607360 conn(0x7f9f900a3320 msgr2=0x7f9f9009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:25.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 1 -- 172.21.15.189:0/896607360 shutdown_connections 2024-09-16T14:45:25.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 1 -- 172.21.15.189:0/896607360 wait complete. 2024-09-16T14:45:25.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 10 monclient: build_initial_monmap 2024-09-16T14:45:25.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 10 monclient: monmap: 2024-09-16T14:45:25.889 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:25.889 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:25.889 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:25.889 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:25.889 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:25.889 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:25.889 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:25.889 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:25.890 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:25.890 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:25.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 1 Processor -- start 2024-09-16T14:45:25.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 1 -- start start 2024-09-16T14:45:25.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 10 monclient: init 2024-09-16T14:45:25.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:25.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:25.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 10 monclient(hunting): picked mon.noname-c con 0x7f9f901315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:25.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 10 monclient(hunting): picked mon.noname-b con 0x7f9f90132090 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:25.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 10 monclient(hunting): picked mon.noname-a con 0x7f9f901300d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:25.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9009cd20 con 0x7f9f901300d0 2024-09-16T14:45:25.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9006b380 con 0x7f9f90132090 2024-09-16T14:45:25.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f901c5ca0 con 0x7f9f901315a0 2024-09-16T14:45:25.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:25.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 10 monclient(hunting): authenticate will time out at 1107.998455s 2024-09-16T14:45:25.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8ffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9f90132090 0x7f9f901ca0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33406/0 (socket says 172.21.15.189:33406) 2024-09-16T14:45:25.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8ffff640 1 -- 172.21.15.189:0/3364638932 learned_addr learned my addr 172.21.15.189:0/3364638932 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:25.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 1 -- 172.21.15.189:0/3364638932 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406632652 0 0) 0x7f9f9006b380 con 0x7f9f90132090 2024-09-16T14:45:25.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:45:25.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:25.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:25.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 10 monclient(hunting): my global_id is 94174 2024-09-16T14:45:25.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 1 -- 172.21.15.189:0/3364638932 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f74003530 con 0x7f9f90132090 2024-09-16T14:45:25.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 1 -- 172.21.15.189:0/3364638932 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2452655184 0 0) 0x7f9f74003530 con 0x7f9f90132090 2024-09-16T14:45:25.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 1 -- 172.21.15.189:0/3364638932 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f9006b380 con 0x7f9f90132090 2024-09-16T14:45:25.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 1 -- 172.21.15.189:0/3364638932 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f7c001980 con 0x7f9f90132090 2024-09-16T14:45:25.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:25.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:25.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 10 monclient(hunting): dump: 2024-09-16T14:45:25.895 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:25.895 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:25.895 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:25.895 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:25.895 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:25.895 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:25.895 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:25.895 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:25.895 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:25.895 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:25.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:25.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 1 -- 172.21.15.189:0/3364638932 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 966424155 0 0) 0x7f9f9006b380 con 0x7f9f90132090 2024-09-16T14:45:25.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 1 -- 172.21.15.189:0/3364638932 >> v1:172.21.15.26:6790/0 conn(0x7f9f901315a0 legacy=0x7f9f9012f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:25.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 1 -- 172.21.15.189:0/3364638932 >> v1:172.21.15.26:6789/0 conn(0x7f9f901300d0 legacy=0x7f9f901cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:25.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:45:25.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 1 monclient: found mon.b 2024-09-16T14:45:25.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:25.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:25.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 1 -- 172.21.15.189:0/3364638932 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9011d0a0 con 0x7f9f90132090 2024-09-16T14:45:25.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:45:25.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:45:25.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:25.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 5 monclient: authenticate success, global_id 94174 2024-09-16T14:45:25.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 10 monclient: _renew_subs 2024-09-16T14:45:25.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:25.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 1 -- 172.21.15.189:0/3364638932 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9009a630 con 0x7f9f90132090 2024-09-16T14:45:25.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 10 monclient: _renew_subs 2024-09-16T14:45:25.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:25.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f96c3f640 1 -- 172.21.15.189:0/3364638932 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9011b020 con 0x7f9f90132090 2024-09-16T14:45:25.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 1 -- 172.21.15.189:0/3364638932 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f7c001d50 con 0x7f9f90132090 2024-09-16T14:45:25.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:25.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 1 -- 172.21.15.189:0/3364638932 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f7c003a70 con 0x7f9f90132090 2024-09-16T14:45:25.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:25.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:25.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8effd640 4 set_mon_vals no callback set 2024-09-16T14:45:25.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.880+0000 7f9f8d7fa640 10 monclient: dump: 2024-09-16T14:45:25.897 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:25.897 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:25.898 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:25.898 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:25.898 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:25.898 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:25.898 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:25.898 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:25.898 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:25.898 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:25.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.884+0000 7f9f8d7fa640 1 -- 172.21.15.189:0/3364638932 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9f7c003bd0 con 0x7f9f90132090 2024-09-16T14:45:25.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.884+0000 7f9f96c3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:25.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.884+0000 7f9f96c3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:25.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.884+0000 7f9f96c3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:25.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.884+0000 7f9f96c3f640 1 -- 172.21.15.189:0/3364638932 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f70005050 con 0x7f9f90132090 2024-09-16T14:45:25.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.884+0000 7f9f8d7fa640 1 -- 172.21.15.189:0/3364638932 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(59..59 src has 15..59) v4 ==== 4305+0+0 (unknown 2223704037 0 0) 0x7f9f7c043500 con 0x7f9f90132090 2024-09-16T14:45:25.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.884+0000 7f9f949db640 10 monclient: get_auth_request con 0x7f9f740360f0 auth_method 0 2024-09-16T14:45:25.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.888+0000 7f9f8d7fa640 1 -- 172.21.15.189:0/3364638932 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9f7c014820 con 0x7f9f90132090 2024-09-16T14:45:25.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.888+0000 7f9f8d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:25.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:25.888+0000 7f9f8d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:26.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.044+0000 7f9f96c3f640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:26.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.044+0000 7f9f96c3f640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:26.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.044+0000 7f9f96c3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:26.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.044+0000 7f9f96c3f640 1 -- 172.21.15.189:0/3364638932 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f9f70005880 con 0x7f9f90132090 2024-09-16T14:45:26.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.044+0000 7f9f8d7fa640 1 -- 172.21.15.189:0/3364638932 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v59) v1 ==== 65+0+60 (unknown 1101553479 0 1596542489) 0x7f9f7c012c90 con 0x7f9f90132090 2024-09-16T14:45:26.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.044+0000 7f9f8d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:26.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.044+0000 7f9f8d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:26.068 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:45:26.068 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:45:26.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.140+0000 7f9f96c3f640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:26.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.140+0000 7f9f96c3f640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:26.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.140+0000 7f9f96c3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:26.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.140+0000 7f9f96c3f640 1 -- 172.21.15.189:0/3364638932 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f9f70005370 con 0x7f9f90132090 2024-09-16T14:45:26.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.140+0000 7f9f8d7fa640 1 -- 172.21.15.189:0/3364638932 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v59) v1 ==== 65+0+60 (unknown 1101553479 0 1596542489) 0x7f9f7c014820 con 0x7f9f90132090 2024-09-16T14:45:26.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.140+0000 7f9f8d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:26.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.140+0000 7f9f8d7fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:45:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.140+0000 7f9f96c3f640 1 -- 172.21.15.189:0/3364638932 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9f740360f0 legacy=0x7f9f740385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.140+0000 7f9f96c3f640 10 monclient: shutdown 2024-09-16T14:45:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.140+0000 7f9f96c3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.140+0000 7f9f96c3f640 1 -- 172.21.15.189:0/3364638932 >> v1:172.21.15.189:6789/0 conn(0x7f9f90132090 legacy=0x7f9f901ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.140+0000 7f9f96c3f640 1 -- 172.21.15.189:0/3364638932 shutdown_connections 2024-09-16T14:45:26.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.140+0000 7f9f96c3f640 1 -- 172.21.15.189:0/3364638932 >> 172.21.15.189:0/3364638932 conn(0x7f9f900a3320 msgr2=0x7f9f901307b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:26.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.140+0000 7f9f96c3f640 1 -- 172.21.15.189:0/3364638932 shutdown_connections 2024-09-16T14:45:26.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.140+0000 7f9f96c3f640 1 -- 172.21.15.189:0/3364638932 wait complete. 2024-09-16T14:45:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:45:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls 2024-09-16T14:45:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false grep bar-asdf 2024-09-16T14:45:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:45:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep bar-asdf 2024-09-16T14:45:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 10 monclient: get_monmap_and_config 2024-09-16T14:45:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 10 monclient: build_initial_monmap 2024-09-16T14:45:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 10 monclient: monmap: 2024-09-16T14:45:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:26.202725+0000 2024-09-16T14:45:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:26.202725+0000 2024-09-16T14:45:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:26.203 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.204 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:26.204 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:26.204 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:26.204 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 1 Processor -- start 2024-09-16T14:45:26.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 1 -- start start 2024-09-16T14:45:26.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 10 monclient: init 2024-09-16T14:45:26.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:26.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:26.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 10 monclient(hunting): picked mon.noname-a con 0x7fbab00a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:26.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 10 monclient(hunting): picked mon.noname-b con 0x7fbab00a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 10 monclient(hunting): picked mon.noname-c con 0x7fbab00a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab00a3250 con 0x7fbab00a1d10 2024-09-16T14:45:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab009f600 con 0x7fbab00a2800 2024-09-16T14:45:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab00a3830 con 0x7fbab00a0000 2024-09-16T14:45:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 10 monclient(hunting): authenticate will time out at 1150.322868s 2024-09-16T14:45:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaae59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbab00a1d10 0x7fbab00a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48344/0 (socket says 172.21.15.26:48344) 2024-09-16T14:45:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaae59c640 1 -- 172.21.15.26:0/1833161098 learned_addr learned my addr 172.21.15.26:0/1833161098 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaadd9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbab00a0000 0x7fbab0139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60710/0 (socket says 172.21.15.26:60710) 2024-09-16T14:45:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 1 -- 172.21.15.26:0/1833161098 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376792213 0 0) 0x7fbab00a3250 con 0x7fbab00a1d10 2024-09-16T14:45:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient(hunting): my global_id is 84456 2024-09-16T14:45:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 1 -- 172.21.15.26:0/1833161098 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba9c003530 con 0x7fbab00a1d10 2024-09-16T14:45:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 1 -- 172.21.15.26:0/1833161098 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030797003 0 0) 0x7fbab00a3830 con 0x7fbab00a0000 2024-09-16T14:45:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient(hunting): my global_id is 5060 2024-09-16T14:45:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 1 -- 172.21.15.26:0/1833161098 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab00a3250 con 0x7fbab00a0000 2024-09-16T14:45:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 1 -- 172.21.15.26:0/1833161098 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013800546 0 0) 0x7fbab009f600 con 0x7fbab00a2800 2024-09-16T14:45:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient(hunting): my global_id is 94177 2024-09-16T14:45:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 1 -- 172.21.15.26:0/1833161098 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab00a3830 con 0x7fbab00a2800 2024-09-16T14:45:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 1 -- 172.21.15.26:0/1833161098 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 197687514 0 0) 0x7fba9c003530 con 0x7fbab00a1d10 2024-09-16T14:45:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 1 -- 172.21.15.26:0/1833161098 >> v1:172.21.15.26:6790/0 conn(0x7fbab00a0000 legacy=0x7fbab0139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 1 -- 172.21.15.26:0/1833161098 >> v1:172.21.15.189:6789/0 conn(0x7fbab00a2800 legacy=0x7fbab013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient: _finish_hunting 0 2024-09-16T14:45:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 1 monclient: found mon.noname-a 2024-09-16T14:45:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:45:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 1 -- 172.21.15.26:0/1833161098 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab009ed60 con 0x7fbab00a1d10 2024-09-16T14:45:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient: _finish_auth 0 2024-09-16T14:45:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient: _check_auth_tickets 2024-09-16T14:45:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:26.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 1 -- 172.21.15.26:0/1833161098 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbaa0001620 con 0x7fbab00a1d10 2024-09-16T14:45:26.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:26.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient: dump: 2024-09-16T14:45:26.214 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.214 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.214 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.214 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.214 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.214 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.214 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.214 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.214 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.215 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 5 monclient: authenticate success, global_id 84456 2024-09-16T14:45:26.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:26.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 1 -- 172.21.15.26:0/1833161098 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbaa0001900 con 0x7fbab00a1d10 2024-09-16T14:45:26.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:26.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:26.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 1 -- 172.21.15.26:0/1833161098 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbaa0001c90 con 0x7fbab00a1d10 2024-09-16T14:45:26.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbab4cc8640 4 set_mon_vals no callback set 2024-09-16T14:45:26.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:26.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.201+0000 7fbaad59a640 10 monclient: dump: 2024-09-16T14:45:26.215 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.215 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.216 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient: shutdown 2024-09-16T14:45:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 1 -- 172.21.15.26:0/1833161098 >> v1:172.21.15.26:6789/0 conn(0x7fbab00a1d10 legacy=0x7fbab00a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 1 -- 172.21.15.26:0/1833161098 shutdown_connections 2024-09-16T14:45:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 1 -- 172.21.15.26:0/1833161098 >> 172.21.15.26:0/1833161098 conn(0x7fbab00a7f30 msgr2=0x7fbab00a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:26.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 1 -- 172.21.15.26:0/1833161098 shutdown_connections 2024-09-16T14:45:26.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 1 -- 172.21.15.26:0/1833161098 wait complete. 2024-09-16T14:45:26.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient: build_initial_monmap 2024-09-16T14:45:26.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient: monmap: 2024-09-16T14:45:26.217 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:26.217 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:26.217 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:26.217 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:26.217 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:26.217 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:26.217 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:26.218 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:26.218 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:26.218 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 1 Processor -- start 2024-09-16T14:45:26.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 1 -- start start 2024-09-16T14:45:26.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient: init 2024-09-16T14:45:26.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:26.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:26.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient(hunting): picked mon.noname-b con 0x7fbab00a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:26.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient(hunting): picked mon.noname-a con 0x7fbab00a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:26.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient(hunting): picked mon.noname-c con 0x7fbab00a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:26.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab0132af0 con 0x7fbab00a1d10 2024-09-16T14:45:26.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab0133f40 con 0x7fbab00a0000 2024-09-16T14:45:26.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab0133c60 con 0x7fbab00a2800 2024-09-16T14:45:26.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:26.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient(hunting): authenticate will time out at 1150.326102s 2024-09-16T14:45:26.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaaed9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbab00a2800 0x7fbab01d10b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60714/0 (socket says 172.21.15.26:60714) 2024-09-16T14:45:26.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaaed9d640 1 -- 172.21.15.26:0/3293461800 learned_addr learned my addr 172.21.15.26:0/3293461800 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:26.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659842842 0 0) 0x7fbab0133c60 con 0x7fbab00a2800 2024-09-16T14:45:26.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): my global_id is 5063 2024-09-16T14:45:26.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba90003530 con 0x7fbab00a2800 2024-09-16T14:45:26.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136734182 0 0) 0x7fbab0132af0 con 0x7fbab00a1d10 2024-09-16T14:45:26.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): my global_id is 84459 2024-09-16T14:45:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab0133c60 con 0x7fbab00a1d10 2024-09-16T14:45:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563563501 0 0) 0x7fbab0133f40 con 0x7fbab00a0000 2024-09-16T14:45:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): my global_id is 94180 2024-09-16T14:45:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab0132af0 con 0x7fbab00a0000 2024-09-16T14:45:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1594085258 0 0) 0x7fba90003530 con 0x7fbab00a2800 2024-09-16T14:45:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbab0133f40 con 0x7fbab00a2800 2024-09-16T14:45:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbaa8001920 con 0x7fbab00a2800 2024-09-16T14:45:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): dump: 2024-09-16T14:45:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.221 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1433534613 0 0) 0x7fbab0132af0 con 0x7fbab00a0000 2024-09-16T14:45:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba90003530 con 0x7fbab00a0000 2024-09-16T14:45:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbaa0001c10 con 0x7fbab00a0000 2024-09-16T14:45:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient(hunting): dump: 2024-09-16T14:45:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.222 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1545650366 0 0) 0x7fbab0133c60 con 0x7fbab00a1d10 2024-09-16T14:45:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbab0132af0 con 0x7fbab00a1d10 2024-09-16T14:45:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2255373916 0 0) 0x7fbab0133f40 con 0x7fbab00a2800 2024-09-16T14:45:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 >> v1:172.21.15.189:6789/0 conn(0x7fbab00a0000 legacy=0x7fbab0134360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 >> v1:172.21.15.26:6789/0 conn(0x7fbab00a1d10 legacy=0x7fbab01cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 monclient: found mon.c 2024-09-16T14:45:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab01336a0 con 0x7fbab00a2800 2024-09-16T14:45:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 5 monclient: authenticate success, global_id 5063 2024-09-16T14:45:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient: _renew_subs 2024-09-16T14:45:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 1 -- 172.21.15.26:0/3293461800 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab01339e0 con 0x7fbab00a2800 2024-09-16T14:45:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient: _renew_subs 2024-09-16T14:45:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 1 -- 172.21.15.26:0/3293461800 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab0132360 con 0x7fbab00a2800 2024-09-16T14:45:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbaa8003660 con 0x7fbab00a2800 2024-09-16T14:45:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbaa8003850 con 0x7fbab00a2800 2024-09-16T14:45:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaacd99640 4 set_mon_vals no callback set 2024-09-16T14:45:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbaa6ffd640 10 monclient: dump: 2024-09-16T14:45:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.225 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.205+0000 7fbab4cc8640 1 -- 172.21.15.26:0/3293461800 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba94005050 con 0x7fbab00a2800 2024-09-16T14:45:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.209+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbaa8003a30 con 0x7fbab00a2800 2024-09-16T14:45:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.209+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(59..59 src has 15..59) v4 ==== 4305+0+0 (unknown 2223704037 0 0) 0x7fbaa8043370 con 0x7fbab00a2800 2024-09-16T14:45:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.209+0000 7fbaae59c640 10 monclient: get_auth_request con 0x7fba900364e0 auth_method 0 2024-09-16T14:45:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.213+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbaa8004250 con 0x7fbab00a2800 2024-09-16T14:45:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.213+0000 7fbaa6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.213+0000 7fbaa6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:26.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 10 monclient: get_monmap_and_config 2024-09-16T14:45:26.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 10 monclient: build_initial_monmap 2024-09-16T14:45:26.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 10 monclient: monmap: 2024-09-16T14:45:26.271 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:26.271 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.271 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:26.270661+0000 2024-09-16T14:45:26.271 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:26.270661+0000 2024-09-16T14:45:26.271 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:26.271 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:26.271 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:26.271 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:26.271 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:26.271 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:26.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 1 Processor -- start 2024-09-16T14:45:26.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 1 -- start start 2024-09-16T14:45:26.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 10 monclient: init 2024-09-16T14:45:26.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:26.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:45:26.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 10 monclient(hunting): picked mon.noname-b con 0x7f048c0a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:26.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 10 monclient(hunting): picked mon.noname-a con 0x7f048c0a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:26.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 10 monclient(hunting): picked mon.noname-c con 0x7f048c13b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:26.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048c0a30b0 con 0x7f048c0a25f0 2024-09-16T14:45:26.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048c0a2e30 con 0x7f048c13b430 2024-09-16T14:45:26.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048c09eb50 con 0x7f048c0a1b00 2024-09-16T14:45:26.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:26.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 10 monclient(hunting): authenticate will time out at 1108.386276s 2024-09-16T14:45:26.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f048c13b430 0x7f048c13d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33408/0 (socket says 172.21.15.189:33408) 2024-09-16T14:45:26.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048bfff640 1 -- 172.21.15.189:0/508072583 learned_addr learned my addr 172.21.15.189:0/508072583 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:26.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 1 -- 172.21.15.189:0/508072583 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362898503 0 0) 0x7f048c0a2e30 con 0x7f048c13b430 2024-09-16T14:45:26.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient(hunting): my global_id is 94183 2024-09-16T14:45:26.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 1 -- 172.21.15.189:0/508072583 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0478003530 con 0x7f048c13b430 2024-09-16T14:45:26.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 1 -- 172.21.15.189:0/508072583 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61615917 0 0) 0x7f048c0a30b0 con 0x7f048c0a25f0 2024-09-16T14:45:26.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient(hunting): my global_id is 84462 2024-09-16T14:45:26.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 1 -- 172.21.15.189:0/508072583 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048c0a2e30 con 0x7f048c0a25f0 2024-09-16T14:45:26.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 1 -- 172.21.15.189:0/508072583 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229969569 0 0) 0x7f048c09eb50 con 0x7f048c0a1b00 2024-09-16T14:45:26.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient(hunting): my global_id is 5066 2024-09-16T14:45:26.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 1 -- 172.21.15.189:0/508072583 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048c0a30b0 con 0x7f048c0a1b00 2024-09-16T14:45:26.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 1 -- 172.21.15.189:0/508072583 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 138503429 0 0) 0x7f0478003530 con 0x7f048c13b430 2024-09-16T14:45:26.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 1 -- 172.21.15.189:0/508072583 >> v1:172.21.15.26:6790/0 conn(0x7f048c0a1b00 legacy=0x7f048c0a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 1 -- 172.21.15.189:0/508072583 >> v1:172.21.15.26:6789/0 conn(0x7f048c0a25f0 legacy=0x7f048c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:26.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 1 monclient: found mon.noname-c 2024-09-16T14:45:26.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:26.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:26.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 1 -- 172.21.15.189:0/508072583 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c09ee30 con 0x7f048c13b430 2024-09-16T14:45:26.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient: _finish_auth 0 2024-09-16T14:45:26.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:26.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:26.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 5 monclient: authenticate success, global_id 94183 2024-09-16T14:45:26.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 1 -- 172.21.15.189:0/508072583 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f04840019a0 con 0x7f048c13b430 2024-09-16T14:45:26.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:26.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient: dump: 2024-09-16T14:45:26.281 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:26.281 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.281 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.282 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.282 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.282 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:26.282 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.282 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.282 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.282 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:26.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:26.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 1 -- 172.21.15.189:0/508072583 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0484001bc0 con 0x7f048c13b430 2024-09-16T14:45:26.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:26.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 1 -- 172.21.15.189:0/508072583 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0484001f50 con 0x7f048c13b430 2024-09-16T14:45:26.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:26.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 4 set_mon_vals no callback set 2024-09-16T14:45:26.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:26.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f048affd640 10 monclient: dump: 2024-09-16T14:45:26.283 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:26.283 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.283 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.283 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.283 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.283 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:26.283 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.283 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.283 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.283 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:26.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 10 monclient: shutdown 2024-09-16T14:45:26.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:26.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 1 -- 172.21.15.189:0/508072583 >> v1:172.21.15.189:6789/0 conn(0x7f048c13b430 legacy=0x7f048c13d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 1 -- 172.21.15.189:0/508072583 shutdown_connections 2024-09-16T14:45:26.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.268+0000 7f0492128640 1 -- 172.21.15.189:0/508072583 >> 172.21.15.189:0/508072583 conn(0x7f048c0a7dd0 msgr2=0x7f048c0a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:26.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 1 -- 172.21.15.189:0/508072583 shutdown_connections 2024-09-16T14:45:26.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 1 -- 172.21.15.189:0/508072583 wait complete. 2024-09-16T14:45:26.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 monclient: build_initial_monmap 2024-09-16T14:45:26.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 monclient: monmap: 2024-09-16T14:45:26.284 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:26.284 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:26.284 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:26.285 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 1 Processor -- start 2024-09-16T14:45:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 1 -- start start 2024-09-16T14:45:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 monclient: init 2024-09-16T14:45:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 monclient(hunting): picked mon.noname-b con 0x7f048c0a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 monclient(hunting): picked mon.noname-a con 0x7f048c0a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 monclient(hunting): picked mon.noname-c con 0x7f048c13b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048c11fd80 con 0x7f048c0a25f0 2024-09-16T14:45:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048c120060 con 0x7f048c0a1b00 2024-09-16T14:45:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048c11e8d0 con 0x7f048c13b430 2024-09-16T14:45:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 monclient(hunting): authenticate will time out at 1108.389514s 2024-09-16T14:45:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f048b7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f048c0a1b00 0x7f048c1c9810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33410/0 (socket says 172.21.15.189:33410) 2024-09-16T14:45:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f048b7fe640 1 -- 172.21.15.189:0/569012566 learned_addr learned my addr 172.21.15.189:0/569012566 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690498391 0 0) 0x7f048c120060 con 0x7f048c0a1b00 2024-09-16T14:45:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient(hunting): my global_id is 94186 2024-09-16T14:45:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f046c003530 con 0x7f048c0a1b00 2024-09-16T14:45:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3243877866 0 0) 0x7f046c003530 con 0x7f048c0a1b00 2024-09-16T14:45:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f048c120060 con 0x7f048c0a1b00 2024-09-16T14:45:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f047c0019a0 con 0x7f048c0a1b00 2024-09-16T14:45:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient(hunting): dump: 2024-09-16T14:45:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.288 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125797602 0 0) 0x7f048c11fd80 con 0x7f048c0a25f0 2024-09-16T14:45:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient(hunting): my global_id is 84465 2024-09-16T14:45:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f046c003530 con 0x7f048c0a25f0 2024-09-16T14:45:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236830589 0 0) 0x7f048c11e8d0 con 0x7f048c13b430 2024-09-16T14:45:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient(hunting): my global_id is 5069 2024-09-16T14:45:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048c11fd80 con 0x7f048c13b430 2024-09-16T14:45:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3277941939 0 0) 0x7f048c120060 con 0x7f048c0a1b00 2024-09-16T14:45:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 >> v1:172.21.15.26:6790/0 conn(0x7f048c13b430 legacy=0x7f048c1d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 >> v1:172.21.15.26:6789/0 conn(0x7f048c0a25f0 legacy=0x7f048c1ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:45:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 monclient: found mon.b 2024-09-16T14:45:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c0a5d00 con 0x7f048c0a1b00 2024-09-16T14:45:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient: _finish_auth 0 2024-09-16T14:45:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:45:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 5 monclient: authenticate success, global_id 94186 2024-09-16T14:45:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 monclient: _renew_subs 2024-09-16T14:45:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 1 -- 172.21.15.189:0/569012566 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048c13afd0 con 0x7f048c0a1b00 2024-09-16T14:45:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 monclient: _renew_subs 2024-09-16T14:45:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 1 -- 172.21.15.189:0/569012566 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f048c11faa0 con 0x7f048c0a1b00 2024-09-16T14:45:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f047c003950 con 0x7f048c0a1b00 2024-09-16T14:45:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f047c001d90 con 0x7f048c0a1b00 2024-09-16T14:45:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f048a7fc640 4 set_mon_vals no callback set 2024-09-16T14:45:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 10 monclient: dump: 2024-09-16T14:45:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.291 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0492128640 1 -- 172.21.15.189:0/569012566 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0470005050 con 0x7f048c0a1b00 2024-09-16T14:45:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f047c003ce0 con 0x7f048c0a1b00 2024-09-16T14:45:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(59..59 src has 15..59) v4 ==== 4305+0+0 (unknown 2223704037 0 0) 0x7f047c0434e0 con 0x7f048c0a1b00 2024-09-16T14:45:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.272+0000 7f0483fff640 10 monclient: get_auth_request con 0x7f046c036340 auth_method 0 2024-09-16T14:45:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.276+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f047c014510 con 0x7f048c0a1b00 2024-09-16T14:45:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.276+0000 7f0488ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.276+0000 7f0488ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:26.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.424+0000 7f0492128640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:26.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.424+0000 7f0492128640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:26.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.424+0000 7f0492128640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:26.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.424+0000 7f0492128640 1 -- 172.21.15.189:0/569012566 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f0470005880 con 0x7f048c0a1b00 2024-09-16T14:45:26.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.424+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v59) v1 ==== 65+0+60 (unknown 1101553479 0 1596542489) 0x7f047c014330 con 0x7f048c0a1b00 2024-09-16T14:45:26.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.424+0000 7f0488ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:26.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.424+0000 7f0488ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:26.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.433+0000 7fbab4cc8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:45:26.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.433+0000 7fbab4cc8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:26.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.433+0000 7fbab4cc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:26.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.433+0000 7fbab4cc8640 1 -- 172.21.15.26:0/3293461800 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fba94005880 con 0x7fbab00a2800 2024-09-16T14:45:26.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.433+0000 7fbaa6ffd640 1 -- 172.21.15.26:0/3293461800 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 660093466) 0x7fbaa8012aa0 con 0x7fbab00a2800 2024-09-16T14:45:26.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.433+0000 7fbaa6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:26.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.433+0000 7fbaa6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:26.435 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:45:26.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.433+0000 7fbab4cc8640 1 -- 172.21.15.26:0/3293461800 >> v1:172.21.15.189:6812/3196161582 conn(0x7fba900364e0 legacy=0x7fba900389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.433+0000 7fbab4cc8640 10 monclient: shutdown 2024-09-16T14:45:26.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.433+0000 7fbab4cc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:26.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.433+0000 7fbab4cc8640 1 -- 172.21.15.26:0/3293461800 >> v1:172.21.15.26:6790/0 conn(0x7fbab00a2800 legacy=0x7fbab01d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.433+0000 7fbab4cc8640 1 -- 172.21.15.26:0/3293461800 shutdown_connections 2024-09-16T14:45:26.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.433+0000 7fbab4cc8640 1 -- 172.21.15.26:0/3293461800 >> 172.21.15.26:0/3293461800 conn(0x7fbab00a7f30 msgr2=0x7fbab00a5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:26.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.433+0000 7fbab4cc8640 1 -- 172.21.15.26:0/3293461800 shutdown_connections 2024-09-16T14:45:26.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.433+0000 7fbab4cc8640 1 -- 172.21.15.26:0/3293461800 wait complete. 2024-09-16T14:45:26.453 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:45:26.453 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:45:26.453 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:45:26.454 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:45:26.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.512+0000 7f0492128640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:26.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.512+0000 7f0492128640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:26.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.512+0000 7f0492128640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:26.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.512+0000 7f0492128640 1 -- 172.21.15.189:0/569012566 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f0470005370 con 0x7f048c0a1b00 2024-09-16T14:45:26.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.512+0000 7f0488ff9640 1 -- 172.21.15.189:0/569012566 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v59) v1 ==== 65+0+60 (unknown 1101553479 0 1596542489) 0x7f047c014510 con 0x7f048c0a1b00 2024-09-16T14:45:26.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.512+0000 7f0488ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T14:45:26.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.512+0000 7f0488ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:45:26.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.516+0000 7f0492128640 1 -- 172.21.15.189:0/569012566 >> v1:172.21.15.189:6812/3196161582 conn(0x7f046c036340 legacy=0x7f046c038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.516+0000 7f0492128640 10 monclient: shutdown 2024-09-16T14:45:26.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.516+0000 7f0492128640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:26.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.516+0000 7f0492128640 1 -- 172.21.15.189:0/569012566 >> v1:172.21.15.189:6789/0 conn(0x7f048c0a1b00 legacy=0x7f048c1c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.516+0000 7f0492128640 1 -- 172.21.15.189:0/569012566 shutdown_connections 2024-09-16T14:45:26.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.516+0000 7f0492128640 1 -- 172.21.15.189:0/569012566 >> 172.21.15.189:0/569012566 conn(0x7f048c0a7dd0 msgr2=0x7f048c06b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:26.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.516+0000 7f0492128640 1 -- 172.21.15.189:0/569012566 shutdown_connections 2024-09-16T14:45:26.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.516+0000 7f0492128640 1 -- 172.21.15.189:0/569012566 wait complete. 2024-09-16T14:45:26.534 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:45:26.534 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rm foo 2024-09-16T14:45:26.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 10 monclient: get_monmap_and_config 2024-09-16T14:45:26.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 10 monclient: build_initial_monmap 2024-09-16T14:45:26.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 10 monclient: monmap: 2024-09-16T14:45:26.570 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:26.571 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.571 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:26.570317+0000 2024-09-16T14:45:26.571 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:26.570317+0000 2024-09-16T14:45:26.571 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:26.571 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.571 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:26.571 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:26.571 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:26.571 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 1 Processor -- start 2024-09-16T14:45:26.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 1 -- start start 2024-09-16T14:45:26.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 10 monclient: init 2024-09-16T14:45:26.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:26.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:45:26.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc009a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:26.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc009d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:26.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc009b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:26.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc009dc60 con 0x7f7fc009d1b0 2024-09-16T14:45:26.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc0099ae0 con 0x7f7fc009b040 2024-09-16T14:45:26.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc009b8f0 con 0x7f7fc009a550 2024-09-16T14:45:26.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:26.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 10 monclient(hunting): authenticate will time out at 1150.690335s 2024-09-16T14:45:26.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbeffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7fc009d1b0 0x7f7fc0136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48370/0 (socket says 172.21.15.26:48370) 2024-09-16T14:45:26.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbeffd640 1 -- 172.21.15.26:0/176268652 learned_addr learned my addr 172.21.15.26:0/176268652 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:26.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 1 -- 172.21.15.26:0/176268652 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508400263 0 0) 0x7f7fc009b8f0 con 0x7f7fc009a550 2024-09-16T14:45:26.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient(hunting): my global_id is 5072 2024-09-16T14:45:26.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 1 -- 172.21.15.26:0/176268652 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa0003530 con 0x7f7fc009a550 2024-09-16T14:45:26.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 1 -- 172.21.15.26:0/176268652 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378521080 0 0) 0x7f7fc009dc60 con 0x7f7fc009d1b0 2024-09-16T14:45:26.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient(hunting): my global_id is 84468 2024-09-16T14:45:26.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 1 -- 172.21.15.26:0/176268652 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc009b8f0 con 0x7f7fc009d1b0 2024-09-16T14:45:26.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 1 -- 172.21.15.26:0/176268652 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192539724 0 0) 0x7f7fc0099ae0 con 0x7f7fc009b040 2024-09-16T14:45:26.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient(hunting): my global_id is 94189 2024-09-16T14:45:26.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 1 -- 172.21.15.26:0/176268652 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc009dc60 con 0x7f7fc009b040 2024-09-16T14:45:26.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 1 -- 172.21.15.26:0/176268652 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3590864085 0 0) 0x7f7fa0003530 con 0x7f7fc009a550 2024-09-16T14:45:26.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 1 -- 172.21.15.26:0/176268652 >> v1:172.21.15.189:6789/0 conn(0x7f7fc009b040 legacy=0x7f7fc013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 1 -- 172.21.15.26:0/176268652 >> v1:172.21.15.26:6789/0 conn(0x7f7fc009d1b0 legacy=0x7f7fc0136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:45:26.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 1 monclient: found mon.noname-b 2024-09-16T14:45:26.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:26.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:45:26.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 1 -- 172.21.15.26:0/176268652 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc009bbd0 con 0x7f7fc009a550 2024-09-16T14:45:26.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient: _finish_auth 0 2024-09-16T14:45:26.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:45:26.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:26.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 5 monclient: authenticate success, global_id 5072 2024-09-16T14:45:26.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:26.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 1 -- 172.21.15.26:0/176268652 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7fa80019a0 con 0x7f7fc009a550 2024-09-16T14:45:26.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:26.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient: dump: 2024-09-16T14:45:26.586 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.586 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.586 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.586 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.586 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.587 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.587 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.587 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.587 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.587 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:26.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 1 -- 172.21.15.26:0/176268652 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fa8001bc0 con 0x7f7fc009a550 2024-09-16T14:45:26.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:26.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:26.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fc5e86640 4 set_mon_vals no callback set 2024-09-16T14:45:26.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 1 -- 172.21.15.26:0/176268652 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7fa8001f90 con 0x7f7fc009a550 2024-09-16T14:45:26.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:26.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.569+0000 7f7fbe7fc640 10 monclient: dump: 2024-09-16T14:45:26.588 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.588 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.588 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.588 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.588 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.588 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.588 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.588 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.588 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.588 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 10 monclient: shutdown 2024-09-16T14:45:26.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:26.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 1 -- 172.21.15.26:0/176268652 >> v1:172.21.15.26:6790/0 conn(0x7f7fc009a550 legacy=0x7f7fc009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 1 -- 172.21.15.26:0/176268652 shutdown_connections 2024-09-16T14:45:26.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 1 -- 172.21.15.26:0/176268652 >> 172.21.15.26:0/176268652 conn(0x7f7fc009fb50 msgr2=0x7f7fc009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:26.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 1 -- 172.21.15.26:0/176268652 shutdown_connections 2024-09-16T14:45:26.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 1 -- 172.21.15.26:0/176268652 wait complete. 2024-09-16T14:45:26.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 10 monclient: build_initial_monmap 2024-09-16T14:45:26.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 10 monclient: monmap: 2024-09-16T14:45:26.589 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:26.589 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:26.589 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:26.589 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:26.589 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:26.589 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:26.590 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:26.590 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:26.590 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:26.590 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 1 Processor -- start 2024-09-16T14:45:26.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 1 -- start start 2024-09-16T14:45:26.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 10 monclient: init 2024-09-16T14:45:26.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:26.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:26.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc009a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:26.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc009b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:26.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc009d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:26.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc01a0980 con 0x7f7fc009b040 2024-09-16T14:45:26.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc00a2550 con 0x7f7fc009a550 2024-09-16T14:45:26.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc009a240 con 0x7f7fc009d1b0 2024-09-16T14:45:26.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:26.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 10 monclient(hunting): authenticate will time out at 1150.693668s 2024-09-16T14:45:26.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fbeffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7fc009b040 0x7f7fc01ce590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48384/0 (socket says 172.21.15.26:48384) 2024-09-16T14:45:26.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fbeffd640 1 -- 172.21.15.26:0/3869158057 learned_addr learned my addr 172.21.15.26:0/3869158057 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:26.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793523080 0 0) 0x7f7fc01a0980 con 0x7f7fc009b040 2024-09-16T14:45:26.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): my global_id is 84471 2024-09-16T14:45:26.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa4003530 con 0x7f7fc009b040 2024-09-16T14:45:26.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2072514695 0 0) 0x7f7fc009a240 con 0x7f7fc009d1b0 2024-09-16T14:45:26.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): my global_id is 5075 2024-09-16T14:45:26.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc01a0980 con 0x7f7fc009d1b0 2024-09-16T14:45:26.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755070436 0 0) 0x7f7fc00a2550 con 0x7f7fc009a550 2024-09-16T14:45:26.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): my global_id is 94192 2024-09-16T14:45:26.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc009a240 con 0x7f7fc009a550 2024-09-16T14:45:26.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 324329814 0 0) 0x7f7fa4003530 con 0x7f7fc009b040 2024-09-16T14:45:26.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fc00a2550 con 0x7f7fc009b040 2024-09-16T14:45:26.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7fb00019e0 con 0x7f7fc009b040 2024-09-16T14:45:26.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:26.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): dump: 2024-09-16T14:45:26.593 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.593 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.593 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.593 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.593 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.594 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4164695772 0 0) 0x7f7fc01a0980 con 0x7f7fc009d1b0 2024-09-16T14:45:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fa4003530 con 0x7f7fc009d1b0 2024-09-16T14:45:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7fb80018c0 con 0x7f7fc009d1b0 2024-09-16T14:45:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): dump: 2024-09-16T14:45:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.595 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.595 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.595 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.595 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.595 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.595 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.595 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.595 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.595 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 417868777 0 0) 0x7f7fc00a2550 con 0x7f7fc009b040 2024-09-16T14:45:26.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 >> v1:172.21.15.26:6790/0 conn(0x7f7fc009d1b0 legacy=0x7f7fc01d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 >> v1:172.21.15.189:6789/0 conn(0x7f7fc009a550 legacy=0x7f7fc01b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:45:26.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 monclient: found mon.a 2024-09-16T14:45:26.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:26.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:26.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc01a22e0 con 0x7f7fc009b040 2024-09-16T14:45:26.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient: _finish_auth 0 2024-09-16T14:45:26.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:45:26.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:26.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 5 monclient: authenticate success, global_id 84471 2024-09-16T14:45:26.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 10 monclient: _renew_subs 2024-09-16T14:45:26.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:26.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 1 -- 172.21.15.26:0/3869158057 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc019ff80 con 0x7f7fc009b040 2024-09-16T14:45:26.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 10 monclient: _renew_subs 2024-09-16T14:45:26.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:26.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 1 -- 172.21.15.26:0/3869158057 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc01c9680 con 0x7f7fc009b040 2024-09-16T14:45:26.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fb0001c80 con 0x7f7fc009b040 2024-09-16T14:45:26.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:26.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7fb0004c40 con 0x7f7fc009b040 2024-09-16T14:45:26.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:26.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fbdffb640 4 set_mon_vals no callback set 2024-09-16T14:45:26.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient: dump: 2024-09-16T14:45:26.597 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.597 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.597 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.598 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.598 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.598 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.598 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.598 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.598 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.598 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7fb000f6b0 con 0x7f7fc009b040 2024-09-16T14:45:26.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:26.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fc5e86640 1 -- 172.21.15.26:0/3869158057 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7f98006590 con 0x7f7f98002830 2024-09-16T14:45:26.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(59..59 src has 15..59) v4 ==== 4305+0+0 (unknown 2223704037 0 0) 0x7f7fb0043500 con 0x7f7fc009b040 2024-09-16T14:45:26.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93787472 0 0) 0x7f7f98006590 con 0x7f7f98002830 2024-09-16T14:45:26.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 10 monclient(hunting): my global_id is 84474 2024-09-16T14:45:26.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.573+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc00a2550 con 0x7f7f98002830 2024-09-16T14:45:26.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.577+0000 7f7fbffff640 10 monclient: get_auth_request con 0x7f7fa40363f0 auth_method 0 2024-09-16T14:45:26.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.577+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 317302814 0 0) 0x7f7fc00a2550 con 0x7f7f98002830 2024-09-16T14:45:26.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.577+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f98006590 con 0x7f7f98002830 2024-09-16T14:45:26.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.577+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7fa80015b0 con 0x7f7f98002830 2024-09-16T14:45:26.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.577+0000 7f7fb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.577+0000 7f7fb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:26.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.577+0000 7f7fb7fff640 10 monclient: dump: 2024-09-16T14:45:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.600 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.577+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1081291809 0 0) 0x7f7f98006590 con 0x7f7f98002830 2024-09-16T14:45:26.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.577+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f9800a0e0 con 0x7f7f98002830 2024-09-16T14:45:26.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.577+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7fa8001890 con 0x7f7f98002830 2024-09-16T14:45:26.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.577+0000 7f7fb7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:26.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.577+0000 7f7fb7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:26.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.577+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 >> v1:172.21.15.26:6789/0 conn(0x7f7f98002830 legacy=0x7f7f98004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.577+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 >> v1:172.21.15.26:6789/0 conn(0x7f7f98002830 legacy=0x7f7f98004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:26.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fc5e86640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:45:26.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fc5e86640 1 -- 172.21.15.26:0/3869158057 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7f98009da0 con 0x7f7f980090c0 2024-09-16T14:45:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323221251 0 0) 0x7f7f98009da0 con 0x7f7f980090c0 2024-09-16T14:45:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 10 monclient(hunting): my global_id is 84477 2024-09-16T14:45:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f98006590 con 0x7f7f980090c0 2024-09-16T14:45:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1061333205 0 0) 0x7f7f98006590 con 0x7f7f980090c0 2024-09-16T14:45:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f98009da0 con 0x7f7f980090c0 2024-09-16T14:45:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7fa8003d50 con 0x7f7f980090c0 2024-09-16T14:45:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 10 monclient: dump: 2024-09-16T14:45:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.604 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 792065006 0 0) 0x7f7f98009da0 con 0x7f7f980090c0 2024-09-16T14:45:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7f9800c290 con 0x7f7f980090c0 2024-09-16T14:45:26.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 351453883) 0x7f7fa8004030 con 0x7f7f980090c0 2024-09-16T14:45:26.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:45:26.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:26.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 >> v1:172.21.15.26:6789/0 conn(0x7f7f980090c0 legacy=0x7f7f98009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.581+0000 7f7fb7fff640 1 -- 172.21.15.26:0/3869158057 >> v1:172.21.15.26:6789/0 conn(0x7f7f980090c0 legacy=0x7f7f98009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:26.605 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:45:26.605 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:45:26.605 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:45:26.605 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:45:26.605 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 32, 2024-09-16T14:45:26.606 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:45:26.606 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:45:26.606 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:45:26.606 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:45:26.606 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:26.606 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-09-16T14:45:26.606 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:45:26.606 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:45:26.606 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:45:26.606 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:26.606 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:26.606 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:26.606 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:45:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:45:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:26.607 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:26.608 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:26.608 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:26.608 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:26.608 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:26.608 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.608 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:45:26.608 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:45:26.609 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:45:26.609 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:45:26.609 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:45:26.609 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:45:26.609 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:45:26.609 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:45:26.609 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:45:26.609 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:45:26.609 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:45:26.609 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:45:26.609 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:45:26.609 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:45:26.610 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:45:26.610 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:45:26.610 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:45:26.610 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:26.610 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:26.610 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:26.610 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:26.610 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:26.610 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:26.610 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:26.610 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:26.610 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:26.610 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:26.610 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:45:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:45:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:45:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:45:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:45:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.612 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:45:26.612 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:45:26.612 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:26.612 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:26.612 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:26.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.612 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:45:26.612 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:45:26.612 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:26.612 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:26.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.612 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:26.612 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:45:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:45:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:45:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:45:26.613 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:45:26.614 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:26.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:26.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.614 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:26.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:45:26.614 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:26.614 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.614 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:45:26.614 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:45:26.614 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:26.614 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:26.615 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:26.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.615 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.615 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:45:26.615 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:45:26.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.615 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:26.615 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:26.615 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:45:26.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.615 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:26.615 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:45:26.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.616 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:26.616 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:26.616 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:45:26.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.616 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:26.616 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:45:26.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.616 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:26.616 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:26.616 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:45:26.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.616 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.617 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:45:26.617 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:45:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.585+0000 7f7fc5e86640 1 -- 172.21.15.26:0/3869158057 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7fa40363f0 legacy=0x7f7fa40388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.585+0000 7f7fc5e86640 10 monclient: shutdown 2024-09-16T14:45:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.585+0000 7f7fc5e86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.585+0000 7f7fc5e86640 1 -- 172.21.15.26:0/3869158057 >> v1:172.21.15.26:6789/0 conn(0x7f7fc009b040 legacy=0x7f7fc01ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.585+0000 7f7fbffff640 1 -- 172.21.15.26:0/3869158057 reap_dead start 2024-09-16T14:45:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.585+0000 7f7fbffff640 1 -- 172.21.15.26:0/3869158057 reap_dead start 2024-09-16T14:45:26.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.585+0000 7f7fc5e86640 1 -- 172.21.15.26:0/3869158057 shutdown_connections 2024-09-16T14:45:26.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.585+0000 7f7fc5e86640 1 -- 172.21.15.26:0/3869158057 >> 172.21.15.26:0/3869158057 conn(0x7f7fc009fb50 msgr2=0x7f7fc00a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:26.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.585+0000 7f7fc5e86640 1 -- 172.21.15.26:0/3869158057 shutdown_connections 2024-09-16T14:45:26.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.585+0000 7f7fc5e86640 1 -- 172.21.15.26:0/3869158057 wait complete. 2024-09-16T14:45:26.619 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:45:26.619 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:45:26.619 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:45:26.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.644+0000 7febce9b1640 10 monclient: get_monmap_and_config 2024-09-16T14:45:26.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.644+0000 7febce9b1640 10 monclient: build_initial_monmap 2024-09-16T14:45:26.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.644+0000 7febce9b1640 10 monclient: monmap: 2024-09-16T14:45:26.649 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:26.650 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.650 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:26.649698+0000 2024-09-16T14:45:26.650 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:26.649698+0000 2024-09-16T14:45:26.650 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:26.650 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:26.650 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:26.650 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:26.650 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:26.650 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:26.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 1 Processor -- start 2024-09-16T14:45:26.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 1 -- start start 2024-09-16T14:45:26.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 10 monclient: init 2024-09-16T14:45:26.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:26.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:45:26.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 10 monclient(hunting): picked mon.noname-a con 0x7febc809df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:26.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 10 monclient(hunting): picked mon.noname-c con 0x7febc809e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:26.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 10 monclient(hunting): picked mon.noname-b con 0x7febc809e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:26.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc809ee20 con 0x7febc809df10 2024-09-16T14:45:26.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc809f100 con 0x7febc809e2f0 2024-09-16T14:45:26.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc809f3e0 con 0x7febc809e740 2024-09-16T14:45:26.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:26.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 10 monclient(hunting): authenticate will time out at 1108.765256s 2024-09-16T14:45:26.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc77fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7febc809e2f0 0x7febc81378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33412/0 (socket says 172.21.15.189:33412) 2024-09-16T14:45:26.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc77fe640 1 -- 172.21.15.189:0/730835017 learned_addr learned my addr 172.21.15.189:0/730835017 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:26.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 1 -- 172.21.15.189:0/730835017 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025304293 0 0) 0x7febc809ee20 con 0x7febc809df10 2024-09-16T14:45:26.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient(hunting): my global_id is 84480 2024-09-16T14:45:26.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 1 -- 172.21.15.189:0/730835017 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febb0003530 con 0x7febc809df10 2024-09-16T14:45:26.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 1 -- 172.21.15.189:0/730835017 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943179952 0 0) 0x7febc809f3e0 con 0x7febc809e740 2024-09-16T14:45:26.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient(hunting): my global_id is 5078 2024-09-16T14:45:26.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 1 -- 172.21.15.189:0/730835017 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc809ee20 con 0x7febc809e740 2024-09-16T14:45:26.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 1 -- 172.21.15.189:0/730835017 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 421755254 0 0) 0x7febb0003530 con 0x7febc809df10 2024-09-16T14:45:26.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 1 -- 172.21.15.189:0/730835017 >> v1:172.21.15.26:6790/0 conn(0x7febc809e740 legacy=0x7febc813b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 1 -- 172.21.15.189:0/730835017 >> v1:172.21.15.189:6789/0 conn(0x7febc809e2f0 legacy=0x7febc81378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:26.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 1 monclient: found mon.noname-a 2024-09-16T14:45:26.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:26.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:45:26.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 1 -- 172.21.15.189:0/730835017 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc809c690 con 0x7febc809df10 2024-09-16T14:45:26.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:26.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:26.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:26.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 1 -- 172.21.15.189:0/730835017 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7febc0001620 con 0x7febc809df10 2024-09-16T14:45:26.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:26.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient: dump: 2024-09-16T14:45:26.661 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:26.661 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.661 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.661 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.661 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.661 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:26.661 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.661 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.661 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.662 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:26.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 5 monclient: authenticate success, global_id 84480 2024-09-16T14:45:26.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:26.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 1 -- 172.21.15.189:0/730835017 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febc0001840 con 0x7febc809df10 2024-09-16T14:45:26.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:26.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:26.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 1 -- 172.21.15.189:0/730835017 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7febc0001bd0 con 0x7febc809df10 2024-09-16T14:45:26.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 4 set_mon_vals no callback set 2024-09-16T14:45:26.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:26.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febc6ffd640 10 monclient: dump: 2024-09-16T14:45:26.662 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:26.663 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.663 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.663 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.663 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.663 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:26.663 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.663 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.663 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.663 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:26.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 10 monclient: shutdown 2024-09-16T14:45:26.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:26.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 1 -- 172.21.15.189:0/730835017 >> v1:172.21.15.26:6789/0 conn(0x7febc809df10 legacy=0x7febc809d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 1 -- 172.21.15.189:0/730835017 shutdown_connections 2024-09-16T14:45:26.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 1 -- 172.21.15.189:0/730835017 >> 172.21.15.189:0/730835017 conn(0x7febc80a3320 msgr2=0x7febc809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:26.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 1 -- 172.21.15.189:0/730835017 shutdown_connections 2024-09-16T14:45:26.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 1 -- 172.21.15.189:0/730835017 wait complete. 2024-09-16T14:45:26.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 10 monclient: build_initial_monmap 2024-09-16T14:45:26.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 10 monclient: monmap: 2024-09-16T14:45:26.664 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:26.664 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:26.664 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:26.664 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:26.664 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:26.664 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:26.664 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:26.665 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:26.665 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:26.665 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:26.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 1 Processor -- start 2024-09-16T14:45:26.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 1 -- start start 2024-09-16T14:45:26.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.648+0000 7febce9b1640 10 monclient: init 2024-09-16T14:45:26.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:26.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:45:26.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 10 monclient(hunting): picked mon.noname-a con 0x7febc809df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:26.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 10 monclient(hunting): picked mon.noname-b con 0x7febc809e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:26.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 10 monclient(hunting): picked mon.noname-c con 0x7febc809e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:26.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc81242e0 con 0x7febc809df10 2024-09-16T14:45:26.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc806fc20 con 0x7febc809e2f0 2024-09-16T14:45:26.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc806b380 con 0x7febc809e740 2024-09-16T14:45:26.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:26.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 10 monclient(hunting): authenticate will time out at 1108.768479s 2024-09-16T14:45:26.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc77fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7febc809e2f0 0x7febc81ce990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33418/0 (socket says 172.21.15.189:33418) 2024-09-16T14:45:26.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc77fe640 1 -- 172.21.15.189:0/4253063331 learned_addr learned my addr 172.21.15.189:0/4253063331 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:26.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146204599 0 0) 0x7febc806fc20 con 0x7febc809e2f0 2024-09-16T14:45:26.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient(hunting): my global_id is 94195 2024-09-16T14:45:26.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febb4003530 con 0x7febc809e2f0 2024-09-16T14:45:26.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020801613 0 0) 0x7febc81242e0 con 0x7febc809df10 2024-09-16T14:45:26.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient(hunting): my global_id is 84483 2024-09-16T14:45:26.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc806fc20 con 0x7febc809df10 2024-09-16T14:45:26.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653335533 0 0) 0x7febc806b380 con 0x7febc809e740 2024-09-16T14:45:26.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient(hunting): my global_id is 5081 2024-09-16T14:45:26.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc81242e0 con 0x7febc809e740 2024-09-16T14:45:26.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2989487146 0 0) 0x7febb4003530 con 0x7febc809e2f0 2024-09-16T14:45:26.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febc806b380 con 0x7febc809e2f0 2024-09-16T14:45:26.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7febbc0017b0 con 0x7febc809e2f0 2024-09-16T14:45:26.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:26.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient(hunting): dump: 2024-09-16T14:45:26.668 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:26.668 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.668 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.668 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.668 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.668 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:26.668 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.668 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.668 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.668 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:26.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:26.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3427308335 0 0) 0x7febc81242e0 con 0x7febc809e740 2024-09-16T14:45:26.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febb4003530 con 0x7febc809e740 2024-09-16T14:45:26.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1720596279 0 0) 0x7febc806fc20 con 0x7febc809df10 2024-09-16T14:45:26.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febc81242e0 con 0x7febc809df10 2024-09-16T14:45:26.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2993186514 0 0) 0x7febc806b380 con 0x7febc809e2f0 2024-09-16T14:45:26.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 >> v1:172.21.15.26:6790/0 conn(0x7febc809e740 legacy=0x7febc81d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 >> v1:172.21.15.26:6789/0 conn(0x7febc809df10 legacy=0x7febc81c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:45:26.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 monclient: found mon.b 2024-09-16T14:45:26.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:26.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:26.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc8125960 con 0x7febc809e2f0 2024-09-16T14:45:26.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient: _finish_auth 0 2024-09-16T14:45:26.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:45:26.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:26.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 5 monclient: authenticate success, global_id 94195 2024-09-16T14:45:26.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 10 monclient: _renew_subs 2024-09-16T14:45:26.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:26.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 1 -- 172.21.15.189:0/4253063331 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc8125c40 con 0x7febc809e2f0 2024-09-16T14:45:26.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 10 monclient: _renew_subs 2024-09-16T14:45:26.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:26.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 1 -- 172.21.15.189:0/4253063331 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febc809be70 con 0x7febc809e2f0 2024-09-16T14:45:26.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febbc0047f0 con 0x7febc809e2f0 2024-09-16T14:45:26.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:26.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7febbc004b00 con 0x7febc809e2f0 2024-09-16T14:45:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc67fc640 4 set_mon_vals no callback set 2024-09-16T14:45:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 10 monclient: dump: 2024-09-16T14:45:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.672 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:26.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7febbc004ce0 con 0x7febc809e2f0 2024-09-16T14:45:26.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:26.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:26.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:26.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febce9b1640 1 -- 172.21.15.189:0/4253063331 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba8005050 con 0x7febc809e2f0 2024-09-16T14:45:26.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(59..59 src has 15..59) v4 ==== 4305+0+0 (unknown 2223704037 0 0) 0x7febbc0434d0 con 0x7febc809e2f0 2024-09-16T14:45:26.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.652+0000 7febc7fff640 10 monclient: get_auth_request con 0x7febb40363d0 auth_method 0 2024-09-16T14:45:26.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.656+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7febbc014790 con 0x7febc809e2f0 2024-09-16T14:45:26.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.656+0000 7febc4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:26.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.656+0000 7febc4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:26.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 10 monclient: get_monmap_and_config 2024-09-16T14:45:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 10 monclient: build_initial_monmap 2024-09-16T14:45:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 10 monclient: monmap: 2024-09-16T14:45:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:26.737733+0000 2024-09-16T14:45:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:26.737733+0000 2024-09-16T14:45:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:26.738 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:26.738 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 1 Processor -- start 2024-09-16T14:45:26.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 1 -- start start 2024-09-16T14:45:26.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 10 monclient: init 2024-09-16T14:45:26.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:26.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:26.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 10 monclient(hunting): picked mon.noname-c con 0x7f54f006c5a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:26.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 10 monclient(hunting): picked mon.noname-a con 0x7f54f00a23b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:26.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 10 monclient(hunting): picked mon.noname-b con 0x7f54f00a27c0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:26.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f006cee0 con 0x7f54f00a23b0 2024-09-16T14:45:26.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f006c980 con 0x7f54f006c5a0 2024-09-16T14:45:26.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f00a15e0 con 0x7f54f00a27c0 2024-09-16T14:45:26.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:26.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 10 monclient(hunting): authenticate will time out at 1150.857732s 2024-09-16T14:45:26.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f594a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f54f00a27c0 0x7f54f013d5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60740/0 (socket says 172.21.15.26:60740) 2024-09-16T14:45:26.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f4948640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f54f00a23b0 0x7f54f0139eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48410/0 (socket says 172.21.15.26:48410) 2024-09-16T14:45:26.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f594a640 1 -- 172.21.15.26:0/4231464245 learned_addr learned my addr 172.21.15.26:0/4231464245 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:26.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 1 -- 172.21.15.26:0/4231464245 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687848612 0 0) 0x7f54f00a15e0 con 0x7f54f00a27c0 2024-09-16T14:45:26.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient(hunting): my global_id is 5084 2024-09-16T14:45:26.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 1 -- 172.21.15.26:0/4231464245 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d8003530 con 0x7f54f00a27c0 2024-09-16T14:45:26.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 1 -- 172.21.15.26:0/4231464245 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 479356285 0 0) 0x7f54f006cee0 con 0x7f54f00a23b0 2024-09-16T14:45:26.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient(hunting): my global_id is 84486 2024-09-16T14:45:26.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 1 -- 172.21.15.26:0/4231464245 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54f00a15e0 con 0x7f54f00a23b0 2024-09-16T14:45:26.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 1 -- 172.21.15.26:0/4231464245 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 107504175 0 0) 0x7f54f00a15e0 con 0x7f54f00a23b0 2024-09-16T14:45:26.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 1 -- 172.21.15.26:0/4231464245 >> v1:172.21.15.26:6790/0 conn(0x7f54f00a27c0 legacy=0x7f54f013d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 1 -- 172.21.15.26:0/4231464245 >> v1:172.21.15.189:6789/0 conn(0x7f54f006c5a0 legacy=0x7f54f00a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient: _finish_hunting 0 2024-09-16T14:45:26.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 1 monclient: found mon.noname-a 2024-09-16T14:45:26.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:26.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:45:26.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 1 -- 172.21.15.26:0/4231464245 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f009edd0 con 0x7f54f00a23b0 2024-09-16T14:45:26.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient: _finish_auth 0 2024-09-16T14:45:26.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient: _check_auth_tickets 2024-09-16T14:45:26.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:26.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 1 -- 172.21.15.26:0/4231464245 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f54e4001900 con 0x7f54f00a23b0 2024-09-16T14:45:26.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:26.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient: dump: 2024-09-16T14:45:26.748 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.748 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.748 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.748 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.748 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.749 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.749 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.749 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.749 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.749 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 5 monclient: authenticate success, global_id 84486 2024-09-16T14:45:26.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:26.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 1 -- 172.21.15.26:0/4231464245 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54e4001ba0 con 0x7f54f00a23b0 2024-09-16T14:45:26.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:26.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 1 -- 172.21.15.26:0/4231464245 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f54e4001f70 con 0x7f54f00a23b0 2024-09-16T14:45:26.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:26.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54effff640 10 monclient: dump: 2024-09-16T14:45:26.749 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.750 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.750 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.750 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.750 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.750 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.750 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.750 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.750 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.750 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:26.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 4 set_mon_vals no callback set 2024-09-16T14:45:26.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 10 monclient: shutdown 2024-09-16T14:45:26.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:26.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 1 -- 172.21.15.26:0/4231464245 >> v1:172.21.15.26:6789/0 conn(0x7f54f00a23b0 legacy=0x7f54f0139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 1 -- 172.21.15.26:0/4231464245 shutdown_connections 2024-09-16T14:45:26.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 1 -- 172.21.15.26:0/4231464245 >> 172.21.15.26:0/4231464245 conn(0x7f54f00a3da0 msgr2=0x7f54f00a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:26.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 1 -- 172.21.15.26:0/4231464245 shutdown_connections 2024-09-16T14:45:26.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.737+0000 7f54f73ad640 1 -- 172.21.15.26:0/4231464245 wait complete. 2024-09-16T14:45:26.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 10 monclient: build_initial_monmap 2024-09-16T14:45:26.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 10 monclient: monmap: 2024-09-16T14:45:26.751 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:26.751 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:26.751 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:26.751 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:26.751 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:26.752 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:26.752 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:26.752 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:26.752 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:26.752 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 1 Processor -- start 2024-09-16T14:45:26.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 1 -- start start 2024-09-16T14:45:26.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 10 monclient: init 2024-09-16T14:45:26.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:26.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:26.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 10 monclient(hunting): picked mon.noname-c con 0x7f54f006c5a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:26.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 10 monclient(hunting): picked mon.noname-b con 0x7f54f00a23b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:26.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 10 monclient(hunting): picked mon.noname-a con 0x7f54f00a27c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:26.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f009d760 con 0x7f54f00a27c0 2024-09-16T14:45:26.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f009e490 con 0x7f54f00a23b0 2024-09-16T14:45:26.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f0136d40 con 0x7f54f006c5a0 2024-09-16T14:45:26.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:26.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 10 monclient(hunting): authenticate will time out at 1150.860931s 2024-09-16T14:45:26.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f5149640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f54f006c5a0 0x7f54f01c9230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60750/0 (socket says 172.21.15.26:60750) 2024-09-16T14:45:26.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f5149640 1 -- 172.21.15.26:0/3604833557 learned_addr learned my addr 172.21.15.26:0/3604833557 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:26.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864309608 0 0) 0x7f54e0001240 con 0x7f54f00a27c0 2024-09-16T14:45:26.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): my global_id is 84489 2024-09-16T14:45:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54dc003530 con 0x7f54f00a27c0 2024-09-16T14:45:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912062679 0 0) 0x7f54f0136d40 con 0x7f54f006c5a0 2024-09-16T14:45:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): my global_id is 5087 2024-09-16T14:45:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54e0001240 con 0x7f54f006c5a0 2024-09-16T14:45:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3502757875 0 0) 0x7f54f009e490 con 0x7f54f00a23b0 2024-09-16T14:45:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): my global_id is 94201 2024-09-16T14:45:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54f0136d40 con 0x7f54f00a23b0 2024-09-16T14:45:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2086454140 0 0) 0x7f54dc003530 con 0x7f54f00a27c0 2024-09-16T14:45:26.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54f009e490 con 0x7f54f00a27c0 2024-09-16T14:45:26.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f54e0001860 con 0x7f54f00a27c0 2024-09-16T14:45:26.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:26.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): dump: 2024-09-16T14:45:26.755 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.755 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.755 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.755 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.755 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.755 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.755 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.755 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.756 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.756 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:26.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3786217152 0 0) 0x7f54e0001240 con 0x7f54f006c5a0 2024-09-16T14:45:26.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54dc003530 con 0x7f54f006c5a0 2024-09-16T14:45:26.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 505274051 0 0) 0x7f54f009e490 con 0x7f54f00a27c0 2024-09-16T14:45:26.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 >> v1:172.21.15.26:6790/0 conn(0x7f54f006c5a0 legacy=0x7f54f01c9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 >> v1:172.21.15.189:6789/0 conn(0x7f54f00a23b0 legacy=0x7f54f013d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient: _finish_hunting 0 2024-09-16T14:45:26.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 monclient: found mon.a 2024-09-16T14:45:26.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:26.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:26.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f00a07d0 con 0x7f54f00a27c0 2024-09-16T14:45:26.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient: _finish_auth 0 2024-09-16T14:45:26.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:26.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:26.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 5 monclient: authenticate success, global_id 84489 2024-09-16T14:45:26.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 10 monclient: _renew_subs 2024-09-16T14:45:26.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:26.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 1 -- 172.21.15.26:0/3604833557 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f0135630 con 0x7f54f00a27c0 2024-09-16T14:45:26.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 10 monclient: _renew_subs 2024-09-16T14:45:26.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:45:26.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 1 -- 172.21.15.26:0/3604833557 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f0136780 con 0x7f54f00a27c0 2024-09-16T14:45:26.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54e00047f0 con 0x7f54f00a27c0 2024-09-16T14:45:26.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:26.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f54e0004b00 con 0x7f54f00a27c0 2024-09-16T14:45:26.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:26.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54ef7fe640 4 set_mon_vals no callback set 2024-09-16T14:45:26.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient: dump: 2024-09-16T14:45:26.758 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.758 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.758 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.758 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.758 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.758 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.758 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.758 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.758 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.759 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:26.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f73ad640 1 -- 172.21.15.26:0/3604833557 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f54d0006590 con 0x7f54d0002830 2024-09-16T14:45:26.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f54e000f570 con 0x7f54f00a27c0 2024-09-16T14:45:26.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(59..59 src has 15..59) v4 ==== 4305+0+0 (unknown 2223704037 0 0) 0x7f54e0043290 con 0x7f54f00a27c0 2024-09-16T14:45:26.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169899315 0 0) 0x7f54d0006590 con 0x7f54d0002830 2024-09-16T14:45:26.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient(hunting): my global_id is 5090 2024-09-16T14:45:26.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54f009e490 con 0x7f54d0002830 2024-09-16T14:45:26.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54f4948640 10 monclient: get_auth_request con 0x7f54dc036430 auth_method 0 2024-09-16T14:45:26.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 405833726 0 0) 0x7f54f009e490 con 0x7f54d0002830 2024-09-16T14:45:26.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54d0006590 con 0x7f54d0002830 2024-09-16T14:45:26.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f54e8003620 con 0x7f54d0002830 2024-09-16T14:45:26.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:26.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 10 monclient: dump: 2024-09-16T14:45:26.760 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.760 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.760 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.760 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.760 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.760 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.760 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.761 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.761 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.761 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2422242407 0 0) 0x7f54d0006590 con 0x7f54d0002830 2024-09-16T14:45:26.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.741+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54d000a0e0 con 0x7f54d0002830 2024-09-16T14:45:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.745+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f54e80057b0 con 0x7f54d0002830 2024-09-16T14:45:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.745+0000 7f54edffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.745+0000 7f54edffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:26.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.745+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 >> v1:172.21.15.26:6790/0 conn(0x7f54d0002830 legacy=0x7f54d0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.745+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 >> v1:172.21.15.26:6790/0 conn(0x7f54d0002830 legacy=0x7f54d0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54f73ad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:45:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54f73ad640 1 -- 172.21.15.26:0/3604833557 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f54d0009da0 con 0x7f54d00090c0 2024-09-16T14:45:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028212687 0 0) 0x7f54d0009da0 con 0x7f54d00090c0 2024-09-16T14:45:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 10 monclient(hunting): my global_id is 5093 2024-09-16T14:45:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d0006590 con 0x7f54d00090c0 2024-09-16T14:45:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1710246359 0 0) 0x7f54d0006590 con 0x7f54d00090c0 2024-09-16T14:45:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54d0009da0 con 0x7f54d00090c0 2024-09-16T14:45:26.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f54e8003e80 con 0x7f54d00090c0 2024-09-16T14:45:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 10 monclient: dump: 2024-09-16T14:45:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.764 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1273844475 0 0) 0x7f54d0009da0 con 0x7f54d00090c0 2024-09-16T14:45:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f54d000c290 con 0x7f54d00090c0 2024-09-16T14:45:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 2811461848) 0x7f54e8004180 con 0x7f54d00090c0 2024-09-16T14:45:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:45:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 >> v1:172.21.15.26:6790/0 conn(0x7f54d00090c0 legacy=0x7f54d0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54edffb640 1 -- 172.21.15.26:0/3604833557 >> v1:172.21.15.26:6790/0 conn(0x7f54d00090c0 legacy=0x7f54d0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:26.765 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:45:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:45:26.765 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:45:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:45:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 32, 2024-09-16T14:45:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:45:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:45:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:45:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:45:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-09-16T14:45:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:45:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:45:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:45:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:26.766 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:45:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:45:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:26.767 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:45:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:45:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:45:26.768 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:45:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:45:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:45:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:45:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:45:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:45:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:45:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:45:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:45:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:45:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:45:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:45:26.769 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:45:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:45:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:45:26.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:45:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:45:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:45:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:45:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:45:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:45:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:45:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:45:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:45:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:45:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:45:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:45:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:45:26.774 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:26.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:26.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.774 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:26.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:45:26.774 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:26.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.774 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:45:26.774 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:45:26.774 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:26.774 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:26.775 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:26.775 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.775 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.775 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:45:26.775 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:45:26.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.775 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:26.775 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:26.775 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:45:26.775 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.775 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:26.775 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:45:26.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.776 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:26.776 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:26.776 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:45:26.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.776 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:26.776 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:45:26.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.776 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:26.776 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:26.776 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:45:26.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.776 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:26.776 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:45:26.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.777 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:26.777 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:26.777 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:45:26.777 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.777 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.777 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:45:26.777 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:45:26.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54f73ad640 1 -- 172.21.15.26:0/3604833557 >> v1:172.21.15.189:6812/3196161582 conn(0x7f54dc036430 legacy=0x7f54dc0388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54f73ad640 10 monclient: shutdown 2024-09-16T14:45:26.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54f73ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:26.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54f73ad640 1 -- 172.21.15.26:0/3604833557 >> v1:172.21.15.26:6789/0 conn(0x7f54f00a27c0 legacy=0x7f54f01d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54f594a640 1 -- 172.21.15.26:0/3604833557 reap_dead start 2024-09-16T14:45:26.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54f594a640 1 -- 172.21.15.26:0/3604833557 reap_dead start 2024-09-16T14:45:26.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54f73ad640 1 -- 172.21.15.26:0/3604833557 shutdown_connections 2024-09-16T14:45:26.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54f73ad640 1 -- 172.21.15.26:0/3604833557 >> 172.21.15.26:0/3604833557 conn(0x7f54f00a3da0 msgr2=0x7f54f00a51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:26.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54f73ad640 1 -- 172.21.15.26:0/3604833557 shutdown_connections 2024-09-16T14:45:26.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.749+0000 7f54f73ad640 1 -- 172.21.15.26:0/3604833557 wait complete. 2024-09-16T14:45:26.781 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:45:26.781 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:45:26.781 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:45:26.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.820+0000 7febce9b1640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T14:45:26.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.820+0000 7febce9b1640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T14:45:26.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.820+0000 7febce9b1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:26.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:26.820+0000 7febce9b1640 1 -- 172.21.15.189:0/4253063331 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7feba8005300 con 0x7febc809e2f0 2024-09-16T14:45:26.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.893+0000 7f25b3a17640 10 monclient: get_monmap_and_config 2024-09-16T14:45:26.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.893+0000 7f25b3a17640 10 monclient: build_initial_monmap 2024-09-16T14:45:26.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.893+0000 7f25b3a17640 10 monclient: monmap: 2024-09-16T14:45:26.897 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:26.898 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.898 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:26.897328+0000 2024-09-16T14:45:26.898 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:26.897328+0000 2024-09-16T14:45:26.898 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:26.898 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.898 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:26.898 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:26.898 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:26.898 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 1 Processor -- start 2024-09-16T14:45:26.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 1 -- start start 2024-09-16T14:45:26.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 10 monclient: init 2024-09-16T14:45:26.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:26.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:26.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 10 monclient(hunting): picked mon.noname-c con 0x7f25ac09a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:26.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 10 monclient(hunting): picked mon.noname-a con 0x7f25ac09d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:26.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 10 monclient(hunting): picked mon.noname-b con 0x7f25ac09b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:26.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac09dc60 con 0x7f25ac09d1b0 2024-09-16T14:45:26.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac099ae0 con 0x7f25ac09a550 2024-09-16T14:45:26.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac09b8f0 con 0x7f25ac09b040 2024-09-16T14:45:26.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:26.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 10 monclient(hunting): authenticate will time out at 1151.017113s 2024-09-16T14:45:26.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b0fb2640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f25ac09d1b0 0x7f25ac136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54516/0 (socket says 172.21.15.26:54516) 2024-09-16T14:45:26.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b0fb2640 1 -- 172.21.15.26:0/1322492017 learned_addr learned my addr 172.21.15.26:0/1322492017 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:26.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b1fb4640 1 -- 172.21.15.26:0/1322492017 >> v1:172.21.15.26:6790/0 conn(0x7f25ac09b040 legacy=0x7f25ac13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-16T14:45:26.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b1fb4640 1 -- 172.21.15.26:0/1322492017 >> v1:172.21.15.26:6790/0 conn(0x7f25ac09b040 legacy=0x7f25ac13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:45:26.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b1fb4640 1 --1- 172.21.15.26:0/1322492017 >> v1:172.21.15.26:6790/0 conn(0x7f25ac09b040 0x7f25ac13a190 :-1 s=OPENED pgs=418 cs=1 l=1).handle_message read tag failed 2024-09-16T14:45:26.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b1fb4640 1 --1- 172.21.15.26:0/1322492017 >> v1:172.21.15.26:6790/0 conn(0x7f25ac09b040 0x7f25ac13a190 :-1 s=OPENED pgs=418 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:45:26.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6790/0 2024-09-16T14:45:26.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 1 -- 172.21.15.26:0/1322492017 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670439806 0 0) 0x7f25ac09dc60 con 0x7f25ac09d1b0 2024-09-16T14:45:26.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient(hunting): my global_id is 84492 2024-09-16T14:45:26.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 1 -- 172.21.15.26:0/1322492017 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2598005740 con 0x7f25ac09d1b0 2024-09-16T14:45:26.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 1 -- 172.21.15.26:0/1322492017 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031033887 0 0) 0x7f25ac099ae0 con 0x7f25ac09a550 2024-09-16T14:45:26.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient(hunting): my global_id is 94204 2024-09-16T14:45:26.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 1 -- 172.21.15.26:0/1322492017 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25ac09dc60 con 0x7f25ac09a550 2024-09-16T14:45:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 1 -- 172.21.15.26:0/1322492017 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2705852082 0 0) 0x7f2598005740 con 0x7f25ac09d1b0 2024-09-16T14:45:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 1 -- 172.21.15.26:0/1322492017 >> v1:172.21.15.26:6790/0 conn(0x7f25ac09b040 legacy=0x7f25ac13a190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 1 -- 172.21.15.26:0/1322492017 >> v1:172.21.15.189:6789/0 conn(0x7f25ac09a550 legacy=0x7f25ac09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:45:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 1 monclient: found mon.noname-a 2024-09-16T14:45:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:45:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 1 -- 172.21.15.26:0/1322492017 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac09bbd0 con 0x7f25ac09d1b0 2024-09-16T14:45:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient: _finish_auth 0 2024-09-16T14:45:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:45:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 1 -- 172.21.15.26:0/1322492017 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f259c001620 con 0x7f25ac09d1b0 2024-09-16T14:45:26.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:26.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient: dump: 2024-09-16T14:45:26.908 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.908 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.908 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.908 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.908 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.908 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.908 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.908 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.908 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.908 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 5 monclient: authenticate success, global_id 84492 2024-09-16T14:45:26.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:26.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 1 -- 172.21.15.26:0/1322492017 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f259c001900 con 0x7f25ac09d1b0 2024-09-16T14:45:26.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:26.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:26.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 1 -- 172.21.15.26:0/1322492017 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f259c001c90 con 0x7f25ac09d1b0 2024-09-16T14:45:26.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 4 set_mon_vals no callback set 2024-09-16T14:45:26.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:45:26.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25a3fff640 10 monclient: dump: 2024-09-16T14:45:26.909 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.909 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.909 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.910 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.910 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.910 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.910 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.910 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.910 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.910 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 10 monclient: shutdown 2024-09-16T14:45:26.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:26.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 1 -- 172.21.15.26:0/1322492017 >> v1:172.21.15.26:6789/0 conn(0x7f25ac09d1b0 legacy=0x7f25ac136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 1 -- 172.21.15.26:0/1322492017 shutdown_connections 2024-09-16T14:45:26.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 1 -- 172.21.15.26:0/1322492017 >> 172.21.15.26:0/1322492017 conn(0x7f25ac09fb50 msgr2=0x7f25ac09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:26.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 1 -- 172.21.15.26:0/1322492017 shutdown_connections 2024-09-16T14:45:26.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 1 -- 172.21.15.26:0/1322492017 wait complete. 2024-09-16T14:45:26.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 10 monclient: build_initial_monmap 2024-09-16T14:45:26.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 10 monclient: monmap: 2024-09-16T14:45:26.911 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:26.911 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:26.911 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:26.911 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:26.911 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:26.911 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:26.911 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:26.911 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:26.911 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:26.912 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.897+0000 7f25b3a17640 1 Processor -- start 2024-09-16T14:45:26.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 1 -- start start 2024-09-16T14:45:26.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 10 monclient: init 2024-09-16T14:45:26.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:26.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:26.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 10 monclient(hunting): picked mon.noname-b con 0x7f25ac09a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:26.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 10 monclient(hunting): picked mon.noname-a con 0x7f25ac09b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:26.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 10 monclient(hunting): picked mon.noname-c con 0x7f25ac09d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:26.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac0a2710 con 0x7f25ac09b040 2024-09-16T14:45:26.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac09a240 con 0x7f25ac09a550 2024-09-16T14:45:26.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac1a2f80 con 0x7f25ac09d1b0 2024-09-16T14:45:26.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:26.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 10 monclient(hunting): authenticate will time out at 1151.020521s 2024-09-16T14:45:26.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b0fb2640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f25ac09b040 0x7f25ac1ca110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54528/0 (socket says 172.21.15.26:54528) 2024-09-16T14:45:26.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b0fb2640 1 -- 172.21.15.26:0/2205830456 learned_addr learned my addr 172.21.15.26:0/2205830456 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:26.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535454380 0 0) 0x7f25ac0a2710 con 0x7f25ac09b040 2024-09-16T14:45:26.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): my global_id is 84495 2024-09-16T14:45:26.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2590003530 con 0x7f25ac09b040 2024-09-16T14:45:26.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789874695 0 0) 0x7f25ac1a2f80 con 0x7f25ac09d1b0 2024-09-16T14:45:26.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): my global_id is 5096 2024-09-16T14:45:26.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25ac0a2710 con 0x7f25ac09d1b0 2024-09-16T14:45:26.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2021989458 0 0) 0x7f2590003530 con 0x7f25ac09b040 2024-09-16T14:45:26.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25ac1a2f80 con 0x7f25ac09b040 2024-09-16T14:45:26.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f259c001660 con 0x7f25ac09b040 2024-09-16T14:45:26.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:45:26.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): dump: 2024-09-16T14:45:26.914 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.914 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.914 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.915 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.915 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.915 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.915 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.915 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.915 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.915 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:26.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3423211026 0 0) 0x7f25ac0a2710 con 0x7f25ac09d1b0 2024-09-16T14:45:26.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2590003530 con 0x7f25ac09d1b0 2024-09-16T14:45:26.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f25a4001830 con 0x7f25ac09d1b0 2024-09-16T14:45:26.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:26.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): dump: 2024-09-16T14:45:26.916 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.916 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.916 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.916 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.916 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.916 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.916 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.916 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.916 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.916 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670563705 0 0) 0x7f25ac09a240 con 0x7f25ac09a550 2024-09-16T14:45:26.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): my global_id is 94207 2024-09-16T14:45:26.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25ac0a2710 con 0x7f25ac09a550 2024-09-16T14:45:26.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2428894929 0 0) 0x7f2590003530 con 0x7f25ac09d1b0 2024-09-16T14:45:26.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 >> v1:172.21.15.189:6789/0 conn(0x7f25ac09a550 legacy=0x7f25ac1b4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 >> v1:172.21.15.26:6789/0 conn(0x7f25ac09b040 legacy=0x7f25ac1ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:45:26.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 monclient: found mon.c 2024-09-16T14:45:26.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:26.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:26.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac1a3260 con 0x7f25ac09d1b0 2024-09-16T14:45:26.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:45:26.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:26.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:26.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 5 monclient: authenticate success, global_id 5096 2024-09-16T14:45:26.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 10 monclient: _renew_subs 2024-09-16T14:45:26.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:26.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 1 -- 172.21.15.26:0/2205830456 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ac1a1850 con 0x7f25ac09d1b0 2024-09-16T14:45:26.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 10 monclient: _renew_subs 2024-09-16T14:45:26.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:26.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 1 -- 172.21.15.26:0/2205830456 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ac1b4fa0 con 0x7f25ac09d1b0 2024-09-16T14:45:26.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25a4003620 con 0x7f25ac09d1b0 2024-09-16T14:45:26.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:26.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f25a4003930 con 0x7f25ac09d1b0 2024-09-16T14:45:26.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:26.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient: dump: 2024-09-16T14:45:26.919 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.919 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.919 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.919 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.919 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.919 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.919 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.919 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.919 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.919 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a37fe640 4 set_mon_vals no callback set 2024-09-16T14:45:26.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:26.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b3a17640 1 -- 172.21.15.26:0/2205830456 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f258c006590 con 0x7f258c002830 2024-09-16T14:45:26.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f25a4003a90 con 0x7f25ac09d1b0 2024-09-16T14:45:26.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(59..59 src has 15..59) v4 ==== 4305+0+0 (unknown 2223704037 0 0) 0x7f25a4043240 con 0x7f25ac09d1b0 2024-09-16T14:45:26.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25b0fb2640 10 monclient: get_auth_request con 0x7f2590036410 auth_method 0 2024-09-16T14:45:26.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177373683 0 0) 0x7f258c006590 con 0x7f258c002830 2024-09-16T14:45:26.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient(hunting): my global_id is 94210 2024-09-16T14:45:26.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2590003530 con 0x7f258c002830 2024-09-16T14:45:26.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 443960581 0 0) 0x7f2590003530 con 0x7f258c002830 2024-09-16T14:45:26.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f258c006590 con 0x7f258c002830 2024-09-16T14:45:26.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f25a8001a90 con 0x7f258c002830 2024-09-16T14:45:26.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.901+0000 7f25a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:26.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.905+0000 7f25a1ffb640 10 monclient: dump: 2024-09-16T14:45:26.923 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.923 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.923 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.923 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.923 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.923 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.923 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.923 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.923 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.923 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.905+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1964187148 0 0) 0x7f258c006590 con 0x7f258c002830 2024-09-16T14:45:26.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.905+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f258c00a0e0 con 0x7f258c002830 2024-09-16T14:45:26.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.905+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f25a8001e00 con 0x7f258c002830 2024-09-16T14:45:26.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.905+0000 7f25a1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:26.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.905+0000 7f25a1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:26.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.905+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 >> v1:172.21.15.189:6789/0 conn(0x7f258c002830 legacy=0x7f258c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.905+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 >> v1:172.21.15.189:6789/0 conn(0x7f258c002830 legacy=0x7f258c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:26.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.909+0000 7f25b3a17640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:45:26.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.909+0000 7f25b3a17640 1 -- 172.21.15.26:0/2205830456 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f258c009d80 con 0x7f258c0090a0 2024-09-16T14:45:26.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361113594 0 0) 0x7f258c009d80 con 0x7f258c0090a0 2024-09-16T14:45:26.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:26.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:26.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:26.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 10 monclient(hunting): my global_id is 94213 2024-09-16T14:45:26.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f258c006590 con 0x7f258c0090a0 2024-09-16T14:45:26.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 626256158 0 0) 0x7f258c006590 con 0x7f258c0090a0 2024-09-16T14:45:26.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f258c009d80 con 0x7f258c0090a0 2024-09-16T14:45:26.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f25a8003b20 con 0x7f258c0090a0 2024-09-16T14:45:26.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:26.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:26.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 10 monclient: dump: 2024-09-16T14:45:26.925 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:26.925 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:26.925 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.925 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:26.925 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:26.925 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:26.925 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:26.925 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:26.925 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:26.925 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1883651382 0 0) 0x7f258c009d80 con 0x7f258c0090a0 2024-09-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f258c00c290 con 0x7f258c0090a0 2024-09-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 1778541926) 0x7f25a8004f30 con 0x7f258c0090a0 2024-09-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 >> v1:172.21.15.189:6789/0 conn(0x7f258c0090a0 legacy=0x7f258c009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25a1ffb640 1 -- 172.21.15.26:0/2205830456 >> v1:172.21.15.189:6789/0 conn(0x7f258c0090a0 legacy=0x7f258c009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:45:26.927 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 32, 2024-09-16T14:45:26.927 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:45:26.927 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:45:26.927 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:45:26.927 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:45:26.927 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:26.927 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-09-16T14:45:26.927 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:45:26.927 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:45:26.927 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:45:26.927 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:26.927 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:26.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:26.928 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:26.928 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:26.928 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:26.928 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:26.928 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:26.928 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:26.928 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:26.928 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:26.928 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:45:26.928 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:45:26.928 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:26.928 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:26.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:26.929 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:26.929 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:26.929 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:26.929 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:26.929 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:26.929 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:26.929 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:26.929 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.929 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:45:26.929 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:45:26.929 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:45:26.929 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:45:26.929 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:45:26.930 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:45:26.930 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:45:26.930 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:45:26.930 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:45:26.930 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:45:26.930 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:45:26.930 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:45:26.930 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:45:26.930 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:45:26.930 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:45:26.930 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:45:26.930 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:45:26.930 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:26.931 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:26.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:26.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:26.931 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:26.931 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:26.931 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:26.931 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:26.931 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:26.931 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:26.931 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:26.931 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:45:26.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:45:26.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.932 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:45:26.932 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:45:26.932 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:26.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:26.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:26.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:45:26.932 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:26.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:45:26.932 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:45:26.932 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:26.933 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:26.933 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:26.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.933 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:45:26.933 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:45:26.933 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:26.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:26.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.933 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:26.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:45:26.933 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:26.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.933 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:45:26.934 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:45:26.934 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:26.934 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:26.934 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:26.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.934 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:45:26.934 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:45:26.934 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:26.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:26.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:26.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:45:26.935 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:26.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.935 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:45:26.935 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:45:26.935 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:26.935 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:26.935 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:26.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.935 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.936 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:45:26.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:45:26.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.936 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:26.936 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:26.936 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:45:26.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.936 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:26.936 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:45:26.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:26.936 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:26.936 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:26.936 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:45:26.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:26.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:26.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:26.937 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:45:26.937 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:45:26.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25b3a17640 1 -- 172.21.15.26:0/2205830456 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2590036410 legacy=0x7f25900388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25b3a17640 10 monclient: shutdown 2024-09-16T14:45:26.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25b3a17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:26.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25b3a17640 1 -- 172.21.15.26:0/2205830456 >> v1:172.21.15.26:6790/0 conn(0x7f25ac09d1b0 legacy=0x7f25ac1cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:26.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25b1fb4640 1 -- 172.21.15.26:0/2205830456 reap_dead start 2024-09-16T14:45:26.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25b1fb4640 1 -- 172.21.15.26:0/2205830456 reap_dead start 2024-09-16T14:45:26.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25b3a17640 1 -- 172.21.15.26:0/2205830456 shutdown_connections 2024-09-16T14:45:26.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25b3a17640 1 -- 172.21.15.26:0/2205830456 >> 172.21.15.26:0/2205830456 conn(0x7f25ac09fb50 msgr2=0x7f25ac0a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:26.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25b3a17640 1 -- 172.21.15.26:0/2205830456 shutdown_connections 2024-09-16T14:45:26.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:26.913+0000 7f25b3a17640 1 -- 172.21.15.26:0/2205830456 wait complete. 2024-09-16T14:45:26.939 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:45:26.939 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T14:45:26.939 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:45:26.939 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T14:45:26.940 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:45:26.940 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T14:45:26.940 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:45:26.951 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T14:45:26.951 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:45:26.951 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:45:27.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dd0119640 10 monclient: get_monmap_and_config 2024-09-16T14:45:27.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dd0119640 10 monclient: build_initial_monmap 2024-09-16T14:45:27.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dd0119640 10 monclient: monmap: 2024-09-16T14:45:27.107 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:27.107 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:27.107 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:27.107216+0000 2024-09-16T14:45:27.108 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:27.107216+0000 2024-09-16T14:45:27.108 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:27.108 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:27.108 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:27.108 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:27.108 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:27.108 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:27.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dd0119640 1 Processor -- start 2024-09-16T14:45:27.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dd0119640 1 -- start start 2024-09-16T14:45:27.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dd0119640 10 monclient: init 2024-09-16T14:45:27.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dd0119640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:27.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dd0119640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:45:27.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dd0119640 10 monclient(hunting): picked mon.noname-b con 0x7f8dc8134e40 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:27.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dd0119640 10 monclient(hunting): picked mon.noname-a con 0x7f8dc8099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:27.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dd0119640 10 monclient(hunting): picked mon.noname-c con 0x7f8dc8135220 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:27.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dd0119640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc8099e80 con 0x7f8dc8099a70 2024-09-16T14:45:27.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dd0119640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc80719d0 con 0x7f8dc8135220 2024-09-16T14:45:27.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dd0119640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc809e1e0 con 0x7f8dc8134e40 2024-09-16T14:45:27.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dd0119640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:27.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dd0119640 10 monclient(hunting): authenticate will time out at 1151.227275s 2024-09-16T14:45:27.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dcd6b4640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f8dc8099a70 legacy=0x7f8dc80712c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:27.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dcdeb5640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8dc8134e40 0x7f8dc8070bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:42186/0 (socket says 172.21.15.26:42186) 2024-09-16T14:45:27.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dcdeb5640 1 -- 172.21.15.26:0/1213255549 learned_addr learned my addr 172.21.15.26:0/1213255549 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:27.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dcceb3640 1 -- 172.21.15.26:0/1213255549 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726503554 0 0) 0x7f8dc809e1e0 con 0x7f8dc8134e40 2024-09-16T14:45:27.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dcceb3640 10 monclient(hunting): _negotiate 2024-09-16T14:45:27.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dcceb3640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:27.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dcceb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:27.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dcceb3640 10 monclient(hunting): my global_id is 5099 2024-09-16T14:45:27.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.105+0000 7f8dcceb3640 1 -- 172.21.15.26:0/1213255549 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8db0003530 con 0x7f8dc8134e40 2024-09-16T14:45:27.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 1 -- 172.21.15.26:0/1213255549 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689578622 0 0) 0x7f8dc80719d0 con 0x7f8dc8135220 2024-09-16T14:45:27.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 10 monclient(hunting): _negotiate 2024-09-16T14:45:27.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:27.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:27.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 10 monclient(hunting): my global_id is 94216 2024-09-16T14:45:27.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 1 -- 172.21.15.26:0/1213255549 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dc809e1e0 con 0x7f8dc8135220 2024-09-16T14:45:27.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 1 -- 172.21.15.26:0/1213255549 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1837664730 0 0) 0x7f8db0003530 con 0x7f8dc8134e40 2024-09-16T14:45:27.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 1 -- 172.21.15.26:0/1213255549 >> v1:172.21.15.189:6789/0 conn(0x7f8dc8135220 legacy=0x7f8dc813db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:27.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 1 -- 172.21.15.26:0/1213255549 >> v1:172.21.15.26:6789/0 conn(0x7f8dc8099a70 legacy=0x7f8dc80712c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:27.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 10 monclient: _finish_hunting 0 2024-09-16T14:45:27.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 1 monclient: found mon.noname-b 2024-09-16T14:45:27.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:27.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:45:27.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 1 -- 172.21.15.26:0/1213255549 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc809e7d0 con 0x7f8dc8134e40 2024-09-16T14:45:27.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 10 monclient: _finish_auth 0 2024-09-16T14:45:27.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 10 monclient: _check_auth_tickets 2024-09-16T14:45:27.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:27.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 1 -- 172.21.15.26:0/1213255549 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8db8001620 con 0x7f8dc8134e40 2024-09-16T14:45:27.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:27.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:27.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 10 monclient: dump: 2024-09-16T14:45:27.115 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:27.115 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:27.115 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:27.115 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:27.115 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:27.115 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:27.115 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:27.115 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:27.115 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:27.115 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:27.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 5 monclient: authenticate success, global_id 5099 2024-09-16T14:45:27.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:27.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 1 -- 172.21.15.26:0/1213255549 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8db8001890 con 0x7f8dc8134e40 2024-09-16T14:45:27.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:27.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 1 -- 172.21.15.26:0/1213255549 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8db8001c20 con 0x7f8dc8134e40 2024-09-16T14:45:27.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:27.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:27.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcceb3640 10 monclient: dump: 2024-09-16T14:45:27.116 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:27.116 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:27.116 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:27.116 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:27.116 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:27.116 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:27.116 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:27.116 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:27.116 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:27.117 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:27.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:27.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 4 set_mon_vals no callback set 2024-09-16T14:45:27.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 10 monclient: shutdown 2024-09-16T14:45:27.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:27.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 1 -- 172.21.15.26:0/1213255549 >> v1:172.21.15.26:6790/0 conn(0x7f8dc8134e40 legacy=0x7f8dc8070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:27.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 1 -- 172.21.15.26:0/1213255549 shutdown_connections 2024-09-16T14:45:27.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 1 -- 172.21.15.26:0/1213255549 >> 172.21.15.26:0/1213255549 conn(0x7f8dc809fb50 msgr2=0x7f8dc809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:27.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 1 -- 172.21.15.26:0/1213255549 shutdown_connections 2024-09-16T14:45:27.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 1 -- 172.21.15.26:0/1213255549 wait complete. 2024-09-16T14:45:27.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 10 monclient: build_initial_monmap 2024-09-16T14:45:27.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 10 monclient: monmap: 2024-09-16T14:45:27.117 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:27.118 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:27.118 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:27.118 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:27.118 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:27.118 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:27.118 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:27.118 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:27.118 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:27.118 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:27.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 1 Processor -- start 2024-09-16T14:45:27.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 1 -- start start 2024-09-16T14:45:27.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 10 monclient: init 2024-09-16T14:45:27.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:27.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:27.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 10 monclient(hunting): picked mon.noname-a con 0x7f8dc8099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:27.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 10 monclient(hunting): picked mon.noname-c con 0x7f8dc8135220 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:27.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 10 monclient(hunting): picked mon.noname-b con 0x7f8dc8121490 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:27.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc8124330 con 0x7f8dc8099a70 2024-09-16T14:45:27.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc81340b0 con 0x7f8dc8121490 2024-09-16T14:45:27.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc809c1d0 con 0x7f8dc8135220 2024-09-16T14:45:27.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:27.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcdeb5640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f8dc8099a70 legacy=0x7f8dc81c9930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:27.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 10 monclient(hunting): authenticate will time out at 1151.230519s 2024-09-16T14:45:27.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcd6b4640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8dc8135220 0x7f8dc81ce980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:42196/0 (socket says 172.21.15.26:42196) 2024-09-16T14:45:27.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dcd6b4640 1 -- 172.21.15.26:0/1445207303 learned_addr learned my addr 172.21.15.26:0/1445207303 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:27.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227372108 0 0) 0x7f8dc809c1d0 con 0x7f8dc8135220 2024-09-16T14:45:27.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:27.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:27.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:27.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient(hunting): my global_id is 5102 2024-09-16T14:45:27.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8db4003530 con 0x7f8dc8135220 2024-09-16T14:45:27.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1751088031 0 0) 0x7f8db4003530 con 0x7f8dc8135220 2024-09-16T14:45:27.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8dc809c1d0 con 0x7f8dc8135220 2024-09-16T14:45:27.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8dbc0044e0 con 0x7f8dc8135220 2024-09-16T14:45:27.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:27.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:27.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient(hunting): dump: 2024-09-16T14:45:27.123 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:27.123 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:27.123 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:27.123 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:27.123 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:27.123 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:27.124 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:27.124 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:27.124 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:27.124 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:27.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:27.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940579066 0 0) 0x7f8dc81340b0 con 0x7f8dc8121490 2024-09-16T14:45:27.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:27.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:27.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:27.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient(hunting): my global_id is 94219 2024-09-16T14:45:27.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8db4003530 con 0x7f8dc8121490 2024-09-16T14:45:27.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3968755521 0 0) 0x7f8dc809c1d0 con 0x7f8dc8135220 2024-09-16T14:45:27.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 >> v1:172.21.15.189:6789/0 conn(0x7f8dc8121490 legacy=0x7f8dc81d20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:27.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 >> v1:172.21.15.26:6789/0 conn(0x7f8dc8099a70 legacy=0x7f8dc81c9930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:27.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:45:27.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 1 monclient: found mon.c 2024-09-16T14:45:27.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:27.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:27.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc809a240 con 0x7f8dc8135220 2024-09-16T14:45:27.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient: _finish_auth 0 2024-09-16T14:45:27.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:45:27.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:27.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 5 monclient: authenticate success, global_id 5102 2024-09-16T14:45:27.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 10 monclient: _renew_subs 2024-09-16T14:45:27.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:27.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 1 -- 172.21.15.26:0/1445207303 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc8124050 con 0x7f8dc8135220 2024-09-16T14:45:27.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 10 monclient: _renew_subs 2024-09-16T14:45:27.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:27.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dd0119640 1 -- 172.21.15.26:0/1445207303 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc81ca040 con 0x7f8dc8135220 2024-09-16T14:45:27.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dbc004700 con 0x7f8dc8135220 2024-09-16T14:45:27.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:27.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8dbc004a90 con 0x7f8dc8135220 2024-09-16T14:45:27.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:27.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:27.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc7fff640 4 set_mon_vals no callback set 2024-09-16T14:45:27.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.109+0000 7f8dc67fc640 10 monclient: dump: 2024-09-16T14:45:27.126 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:27.126 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:27.126 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:27.127 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:27.127 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:27.127 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:27.127 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:27.127 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:27.127 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:27.127 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:27.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.113+0000 7f8dd0119640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:27.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.113+0000 7f8dd0119640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:27.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.113+0000 7f8dd0119640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:27.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.113+0000 7f8dd0119640 1 -- 172.21.15.26:0/1445207303 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da8005050 con 0x7f8dc8135220 2024-09-16T14:45:27.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.113+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8dbc004c70 con 0x7f8dc8135220 2024-09-16T14:45:27.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.113+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(59..59 src has 15..59) v4 ==== 4305+0+0 (unknown 2223704037 0 0) 0x7f8dbc043780 con 0x7f8dc8135220 2024-09-16T14:45:27.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.113+0000 7f8dcdeb5640 10 monclient: get_auth_request con 0x7f8db4036180 auth_method 0 2024-09-16T14:45:27.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.117+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8dbc005470 con 0x7f8dc8135220 2024-09-16T14:45:27.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.117+0000 7f8dc67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:27.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.117+0000 7f8dc67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:27.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.329+0000 7f8dd0119640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:45:27.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.329+0000 7f8dd0119640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:27.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.329+0000 7f8dd0119640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:27.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.329+0000 7f8dd0119640 1 -- 172.21.15.26:0/1445207303 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8da8005880 con 0x7f8dc8135220 2024-09-16T14:45:27.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.329+0000 7f8dc67fc640 1 -- 172.21.15.26:0/1445207303 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 62536884) 0x7f8dbc012eb0 con 0x7f8dc8135220 2024-09-16T14:45:27.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.329+0000 7f8dc67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:27.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.329+0000 7f8dc67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:27.332 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:45:27.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.333+0000 7f8dd0119640 1 -- 172.21.15.26:0/1445207303 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8db4036180 legacy=0x7f8db4038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:27.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.333+0000 7f8dd0119640 10 monclient: shutdown 2024-09-16T14:45:27.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.333+0000 7f8dd0119640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:27.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.333+0000 7f8dd0119640 1 -- 172.21.15.26:0/1445207303 >> v1:172.21.15.26:6790/0 conn(0x7f8dc8135220 legacy=0x7f8dc81ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:27.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.333+0000 7f8dd0119640 1 -- 172.21.15.26:0/1445207303 shutdown_connections 2024-09-16T14:45:27.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.333+0000 7f8dd0119640 1 -- 172.21.15.26:0/1445207303 >> 172.21.15.26:0/1445207303 conn(0x7f8dc809fb50 msgr2=0x7f8dc8099350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:27.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.333+0000 7f8dd0119640 1 -- 172.21.15.26:0/1445207303 shutdown_connections 2024-09-16T14:45:27.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:27.333+0000 7f8dd0119640 1 -- 172.21.15.26:0/1445207303 wait complete. 2024-09-16T14:45:29.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:29.652+0000 7febaf7fe640 10 monclient: tick 2024-09-16T14:45:29.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:29.652+0000 7febaf7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:29.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:29.652+0000 7febaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:29.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:29.652+0000 7febaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:30.349 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:45:30.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.476+0000 7f6643610640 10 monclient: get_monmap_and_config 2024-09-16T14:45:30.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.476+0000 7f6643610640 10 monclient: build_initial_monmap 2024-09-16T14:45:30.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.476+0000 7f6643610640 10 monclient: monmap: 2024-09-16T14:45:30.481 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:30.481 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:30.481 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:30.480970+0000 2024-09-16T14:45:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:30.480970+0000 2024-09-16T14:45:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:30.482 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:30.482 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:30.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 1 Processor -- start 2024-09-16T14:45:30.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 1 -- start start 2024-09-16T14:45:30.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 10 monclient: init 2024-09-16T14:45:30.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:30.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:45:30.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 10 monclient(hunting): picked mon.noname-b con 0x7f663c134d00 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:30.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 10 monclient(hunting): picked mon.noname-a con 0x7f663c09a1c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:30.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 10 monclient(hunting): picked mon.noname-c con 0x7f663c1357f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:30.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663c06ca10 con 0x7f663c09a1c0 2024-09-16T14:45:30.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663c0998e0 con 0x7f663c1357f0 2024-09-16T14:45:30.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663c099040 con 0x7f663c134d00 2024-09-16T14:45:30.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:30.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 10 monclient(hunting): authenticate will time out at 1154.601042s 2024-09-16T14:45:30.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6640bab640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f663c09a1c0 legacy=0x7f663c13a230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:30.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f66413ac640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f663c134d00 0x7f663c1350e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:42204/0 (socket says 172.21.15.26:42204) 2024-09-16T14:45:30.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f66413ac640 1 -- 172.21.15.26:0/3099341621 learned_addr learned my addr 172.21.15.26:0/3099341621 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:30.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 1 -- 172.21.15.26:0/3099341621 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3670610490 0 0) 0x7f663c099040 con 0x7f663c134d00 2024-09-16T14:45:30.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:30.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:30.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:30.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 10 monclient(hunting): my global_id is 5105 2024-09-16T14:45:30.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 1 -- 172.21.15.26:0/3099341621 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6628003530 con 0x7f663c134d00 2024-09-16T14:45:30.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 1 -- 172.21.15.26:0/3099341621 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 199964697 0 0) 0x7f6628003530 con 0x7f663c134d00 2024-09-16T14:45:30.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 1 -- 172.21.15.26:0/3099341621 >> v1:172.21.15.189:6789/0 conn(0x7f663c1357f0 legacy=0x7f663c13d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:30.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 1 -- 172.21.15.26:0/3099341621 >> v1:172.21.15.26:6789/0 conn(0x7f663c09a1c0 legacy=0x7f663c13a230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:30.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 10 monclient: _finish_hunting 0 2024-09-16T14:45:30.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 1 monclient: found mon.noname-b 2024-09-16T14:45:30.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:30.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:45:30.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 1 -- 172.21.15.26:0/3099341621 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c099320 con 0x7f663c134d00 2024-09-16T14:45:30.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 10 monclient: _finish_auth 0 2024-09-16T14:45:30.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 10 monclient: _check_auth_tickets 2024-09-16T14:45:30.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:30.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 1 -- 172.21.15.26:0/3099341621 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6638001620 con 0x7f663c134d00 2024-09-16T14:45:30.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:30.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:30.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 10 monclient: dump: 2024-09-16T14:45:30.488 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:30.488 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:30.488 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:30.488 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:30.488 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:30.488 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:30.488 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:30.488 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:30.488 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:30.488 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:30.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 5 monclient: authenticate success, global_id 5105 2024-09-16T14:45:30.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:30.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 1 -- 172.21.15.26:0/3099341621 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6638001940 con 0x7f663c134d00 2024-09-16T14:45:30.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:30.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:30.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 4 set_mon_vals no callback set 2024-09-16T14:45:30.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 1 -- 172.21.15.26:0/3099341621 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6638001d10 con 0x7f663c134d00 2024-09-16T14:45:30.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:30.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:30.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6633fff640 10 monclient: dump: 2024-09-16T14:45:30.489 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:30.489 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:30.489 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:30.489 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:30.489 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:30.490 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:30.490 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:30.490 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:30.490 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:30.490 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:30.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 10 monclient: shutdown 2024-09-16T14:45:30.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:30.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 1 -- 172.21.15.26:0/3099341621 >> v1:172.21.15.26:6790/0 conn(0x7f663c134d00 legacy=0x7f663c1350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:30.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 1 -- 172.21.15.26:0/3099341621 shutdown_connections 2024-09-16T14:45:30.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 1 -- 172.21.15.26:0/3099341621 >> 172.21.15.26:0/3099341621 conn(0x7f663c0a0da0 msgr2=0x7f663c0a11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:30.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 1 -- 172.21.15.26:0/3099341621 shutdown_connections 2024-09-16T14:45:30.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 1 -- 172.21.15.26:0/3099341621 wait complete. 2024-09-16T14:45:30.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 10 monclient: build_initial_monmap 2024-09-16T14:45:30.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.480+0000 7f6643610640 10 monclient: monmap: 2024-09-16T14:45:30.491 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:30.491 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:30.491 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:30.491 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:30.491 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:30.491 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:30.491 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:30.491 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:30.491 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:30.491 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:30.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 1 Processor -- start 2024-09-16T14:45:30.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 1 -- start start 2024-09-16T14:45:30.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 10 monclient: init 2024-09-16T14:45:30.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 10 monclient(hunting): picked mon.noname-c con 0x7f663c09a1c0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 10 monclient(hunting): picked mon.noname-b con 0x7f663c134d00 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 10 monclient(hunting): picked mon.noname-a con 0x7f663c1357f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663c09b2c0 con 0x7f663c1357f0 2024-09-16T14:45:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663c122cd0 con 0x7f663c134d00 2024-09-16T14:45:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6641bad640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f663c1357f0 legacy=0x7f663c1d1e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663c09e290 con 0x7f663c09a1c0 2024-09-16T14:45:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 10 monclient(hunting): authenticate will time out at 1154.604705s 2024-09-16T14:45:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f66413ac640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f663c09a1c0 0x7f663c134130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:42208/0 (socket says 172.21.15.26:42208) 2024-09-16T14:45:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f66413ac640 1 -- 172.21.15.26:0/3472418611 learned_addr learned my addr 172.21.15.26:0/3472418611 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:30.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 1 -- 172.21.15.26:0/3472418611 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858102724 0 0) 0x7f663c09e290 con 0x7f663c09a1c0 2024-09-16T14:45:30.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:30.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:30.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:30.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 10 monclient(hunting): my global_id is 5108 2024-09-16T14:45:30.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 1 -- 172.21.15.26:0/3472418611 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6620003530 con 0x7f663c09a1c0 2024-09-16T14:45:30.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 1 -- 172.21.15.26:0/3472418611 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2892200977 0 0) 0x7f6620003530 con 0x7f663c09a1c0 2024-09-16T14:45:30.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 1 -- 172.21.15.26:0/3472418611 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f663c09e290 con 0x7f663c09a1c0 2024-09-16T14:45:30.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 1 -- 172.21.15.26:0/3472418611 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6638001620 con 0x7f663c09a1c0 2024-09-16T14:45:30.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:30.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:30.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 10 monclient(hunting): dump: 2024-09-16T14:45:30.497 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:30.497 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:30.497 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:30.497 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:30.497 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:30.497 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:30.497 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:30.497 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:30.497 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:30.497 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:30.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 1 -- 172.21.15.26:0/3472418611 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2332150208 0 0) 0x7f663c09e290 con 0x7f663c09a1c0 2024-09-16T14:45:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 1 -- 172.21.15.26:0/3472418611 >> v1:172.21.15.189:6789/0 conn(0x7f663c134d00 legacy=0x7f663c1ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 1 -- 172.21.15.26:0/3472418611 >> v1:172.21.15.26:6789/0 conn(0x7f663c1357f0 legacy=0x7f663c1d1e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:45:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 1 monclient: found mon.c 2024-09-16T14:45:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 1 -- 172.21.15.26:0/3472418611 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c0a31c0 con 0x7f663c09a1c0 2024-09-16T14:45:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 10 monclient: _finish_auth 0 2024-09-16T14:45:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 5 monclient: authenticate success, global_id 5108 2024-09-16T14:45:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 10 monclient: _renew_subs 2024-09-16T14:45:30.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:30.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 1 -- 172.21.15.26:0/3472418611 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f663c124840 con 0x7f663c09a1c0 2024-09-16T14:45:30.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 10 monclient: _renew_subs 2024-09-16T14:45:30.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:30.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 1 -- 172.21.15.26:0/3472418611 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f663c134840 con 0x7f663c09a1c0 2024-09-16T14:45:30.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 1 -- 172.21.15.26:0/3472418611 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6638004c00 con 0x7f663c09a1c0 2024-09-16T14:45:30.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:30.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 1 -- 172.21.15.26:0/3472418611 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6638004f90 con 0x7f663c09a1c0 2024-09-16T14:45:30.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:30.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:30.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f66337fe640 4 set_mon_vals no callback set 2024-09-16T14:45:30.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 10 monclient: dump: 2024-09-16T14:45:30.499 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:30.499 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:30.500 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:30.500 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:30.500 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:30.500 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:30.500 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:30.500 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:30.500 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:30.500 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:30.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:30.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:30.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 1 -- 172.21.15.26:0/3472418611 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f66380050f0 con 0x7f663c09a1c0 2024-09-16T14:45:30.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:30.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6643610640 1 -- 172.21.15.26:0/3472418611 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c005050 con 0x7f663c09a1c0 2024-09-16T14:45:30.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6631ffb640 1 -- 172.21.15.26:0/3472418611 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(59..59 src has 15..59) v4 ==== 4305+0+0 (unknown 2223704037 0 0) 0x7f6638043670 con 0x7f663c09a1c0 2024-09-16T14:45:30.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.484+0000 7f6640bab640 10 monclient: get_auth_request con 0x7f66200360a0 auth_method 0 2024-09-16T14:45:30.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.488+0000 7f6631ffb640 1 -- 172.21.15.26:0/3472418611 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6638014a50 con 0x7f663c09a1c0 2024-09-16T14:45:30.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.488+0000 7f6631ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:30.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.488+0000 7f6631ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:30.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:30.652+0000 7febaf7fe640 10 monclient: tick 2024-09-16T14:45:30.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:30.652+0000 7febaf7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:30.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:30.652+0000 7febaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:30.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.704+0000 7f6643610640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:45:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.704+0000 7f6643610640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.704+0000 7f6643610640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.704+0000 7f6643610640 1 -- 172.21.15.26:0/3472418611 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f661c005880 con 0x7f663c09a1c0 2024-09-16T14:45:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.708+0000 7f6631ffb640 1 -- 172.21.15.26:0/3472418611 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3477449476) 0x7f6638012ec0 con 0x7f663c09a1c0 2024-09-16T14:45:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.708+0000 7f6631ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.708+0000 7f6631ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:30.711 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:45:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.708+0000 7f6643610640 1 -- 172.21.15.26:0/3472418611 >> v1:172.21.15.189:6812/3196161582 conn(0x7f66200360a0 legacy=0x7f6620038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.708+0000 7f6643610640 10 monclient: shutdown 2024-09-16T14:45:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.708+0000 7f6643610640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.708+0000 7f6643610640 1 -- 172.21.15.26:0/3472418611 >> v1:172.21.15.26:6790/0 conn(0x7f663c09a1c0 legacy=0x7f663c134130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.708+0000 7f6643610640 1 -- 172.21.15.26:0/3472418611 shutdown_connections 2024-09-16T14:45:30.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.708+0000 7f6643610640 1 -- 172.21.15.26:0/3472418611 >> 172.21.15.26:0/3472418611 conn(0x7f663c0a0da0 msgr2=0x7f663c0a27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:30.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.708+0000 7f6643610640 1 -- 172.21.15.26:0/3472418611 shutdown_connections 2024-09-16T14:45:30.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:30.708+0000 7f6643610640 1 -- 172.21.15.26:0/3472418611 wait complete. 2024-09-16T14:45:31.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:31.652+0000 7febaf7fe640 10 monclient: tick 2024-09-16T14:45:31.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:31.652+0000 7febaf7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:31.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:31.652+0000 7febaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:32.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:32.652+0000 7febaf7fe640 10 monclient: tick 2024-09-16T14:45:32.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:32.652+0000 7febaf7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:32.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:32.652+0000 7febaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:33.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:33.652+0000 7febaf7fe640 10 monclient: tick 2024-09-16T14:45:33.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:33.652+0000 7febaf7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:33.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:33.652+0000 7febaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:33.729 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:45:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.856+0000 7f8451560640 10 monclient: get_monmap_and_config 2024-09-16T14:45:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.856+0000 7f8451560640 10 monclient: build_initial_monmap 2024-09-16T14:45:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.856+0000 7f8451560640 10 monclient: monmap: 2024-09-16T14:45:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:33.860337+0000 2024-09-16T14:45:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:33.860337+0000 2024-09-16T14:45:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:33.861 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.856+0000 7f8451560640 1 Processor -- start 2024-09-16T14:45:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.856+0000 7f8451560640 1 -- start start 2024-09-16T14:45:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.856+0000 7f8451560640 10 monclient: init 2024-09-16T14:45:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.860+0000 7f8451560640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.860+0000 7f8451560640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:45:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.860+0000 7f8451560640 10 monclient(hunting): picked mon.noname-b con 0x7f844c134e40 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.860+0000 7f8451560640 10 monclient(hunting): picked mon.noname-a con 0x7f844c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.860+0000 7f8451560640 10 monclient(hunting): picked mon.noname-c con 0x7f844c135220 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.860+0000 7f8451560640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c099e80 con 0x7f844c099a70 2024-09-16T14:45:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.860+0000 7f8451560640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c0719d0 con 0x7f844c135220 2024-09-16T14:45:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.860+0000 7f8451560640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c09e1e0 con 0x7f844c134e40 2024-09-16T14:45:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.860+0000 7f8451560640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.860+0000 7f8451560640 10 monclient(hunting): authenticate will time out at 1157.980396s 2024-09-16T14:45:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.860+0000 7f8443fff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f844c0712c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.860+0000 7f844affd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f844c134e40 0x7f844c070bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:42212/0 (socket says 172.21.15.26:42212) 2024-09-16T14:45:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:33.860+0000 7f844affd640 1 -- 172.21.15.26:0/2779934760 learned_addr learned my addr 172.21.15.26:0/2779934760 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:34.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:34.064+0000 7f8443fff640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f844c0712c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:34.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:34.468+0000 7f8443fff640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f844c0712c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:34.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:34.652+0000 7febaf7fe640 10 monclient: tick 2024-09-16T14:45:34.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:34.652+0000 7febaf7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:34.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:34.652+0000 7febaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:35.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:35.272+0000 7f8443fff640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f844c0712c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:35.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:35.652+0000 7febaf7fe640 10 monclient: tick 2024-09-16T14:45:35.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:35.652+0000 7febaf7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:35.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:35.652+0000 7febaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:36.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:36.652+0000 7febaf7fe640 10 monclient: tick 2024-09-16T14:45:36.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:36.652+0000 7febaf7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:36.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:36.652+0000 7febaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:36.860+0000 7f84497fa640 10 monclient(hunting): tick 2024-09-16T14:45:36.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:36.860+0000 7f84497fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:45:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:36.860+0000 7f84497fa640 1 monclient(hunting): continuing hunt 2024-09-16T14:45:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:36.860+0000 7f84497fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:45:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:36.860+0000 7f84497fa640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6790/0 conn(0x7f844c134e40 legacy=0x7f844c070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:36.860+0000 7f84497fa640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.189:6789/0 conn(0x7f844c135220 legacy=0x7f844c13db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:36.860+0000 7f84497fa640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f844c0712c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:36.860+0000 7f84497fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:45:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:36.860+0000 7f84497fa640 10 monclient(hunting): picked mon.noname-b con 0x7f84300036d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:36.860+0000 7f84497fa640 10 monclient(hunting): picked mon.noname-a con 0x7f844c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:36.860+0000 7f84497fa640 10 monclient(hunting): picked mon.noname-c con 0x7f844c135220 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:36.860+0000 7f84497fa640 1 -- 172.21.15.26:0/2779934760 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c09e7d0 con 0x7f844c099a70 2024-09-16T14:45:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:36.860+0000 7f84497fa640 1 -- 172.21.15.26:0/2779934760 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f843000f330 con 0x7f844c135220 2024-09-16T14:45:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:36.860+0000 7f84497fa640 1 -- 172.21.15.26:0/2779934760 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8430010540 con 0x7f84300036d0 2024-09-16T14:45:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:36.860+0000 7f84497fa640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:36.860+0000 7f8443fff640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f84300093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:37.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:37.064+0000 7f8443fff640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f84300093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:37.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:37.468+0000 7f8443fff640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f84300093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:37.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:37.652+0000 7febaf7fe640 10 monclient: tick 2024-09-16T14:45:37.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:37.652+0000 7febaf7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:37.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:37.652+0000 7febaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:38.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:38.272+0000 7f8443fff640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f84300093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:38.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:38.652+0000 7febaf7fe640 10 monclient: tick 2024-09-16T14:45:38.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:38.652+0000 7febaf7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:38.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:38.652+0000 7febaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:39.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:39.652+0000 7febaf7fe640 10 monclient: tick 2024-09-16T14:45:39.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:39.652+0000 7febaf7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:39.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:39.652+0000 7febaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:39.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:39.652+0000 7febaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:39.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:39.860+0000 7f84497fa640 10 monclient(hunting): tick 2024-09-16T14:45:39.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:39.860+0000 7f84497fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:45:39.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:39.860+0000 7f84497fa640 1 monclient(hunting): continuing hunt 2024-09-16T14:45:39.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:39.860+0000 7f84497fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:45:39.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:39.860+0000 7f84497fa640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6790/0 conn(0x7f84300036d0 legacy=0x7f8430005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:39.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:39.860+0000 7f84497fa640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.189:6789/0 conn(0x7f844c135220 legacy=0x7f843000cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:39.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:39.860+0000 7f84497fa640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f84300093b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:39.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:39.860+0000 7f84497fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:39.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:39.860+0000 7f84497fa640 10 monclient(hunting): picked mon.noname-c con 0x7f843000f5c0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:39.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:39.860+0000 7f84497fa640 10 monclient(hunting): picked mon.noname-a con 0x7f844c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:39.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:39.860+0000 7f84497fa640 10 monclient(hunting): picked mon.noname-b con 0x7f84300036d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:39.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:39.860+0000 7f84497fa640 1 -- 172.21.15.26:0/2779934760 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84300117b0 con 0x7f844c099a70 2024-09-16T14:45:39.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:39.860+0000 7f84497fa640 1 -- 172.21.15.26:0/2779934760 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c09e7d0 con 0x7f843000f5c0 2024-09-16T14:45:39.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:39.860+0000 7f84497fa640 1 -- 172.21.15.26:0/2779934760 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84300102b0 con 0x7f84300036d0 2024-09-16T14:45:39.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:39.860+0000 7f84497fa640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:39.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:39.860+0000 7f8443fff640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f84300093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:40.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:40.064+0000 7f8443fff640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f84300093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:40.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:40.468+0000 7f8443fff640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f84300093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:40.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:40.652+0000 7febaf7fe640 10 monclient: tick 2024-09-16T14:45:40.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:40.652+0000 7febaf7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:40.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:40.652+0000 7febaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:41.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.272+0000 7f8443fff640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f84300093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:41.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:41.652+0000 7febaf7fe640 10 monclient: tick 2024-09-16T14:45:41.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:41.652+0000 7febaf7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:41.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:41.652+0000 7febaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.704+0000 7f844a7fc640 1 -- 172.21.15.26:0/2779934760 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211878032 0 0) 0x7f844c09e7d0 con 0x7f843000f5c0 2024-09-16T14:45:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.704+0000 7f844a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.704+0000 7f844a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.704+0000 7f844a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.704+0000 7f844a7fc640 10 monclient(hunting): my global_id is 94222 2024-09-16T14:45:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.704+0000 7f844a7fc640 1 -- 172.21.15.26:0/2779934760 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f842c003530 con 0x7f843000f5c0 2024-09-16T14:45:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.704+0000 7f844a7fc640 1 -- 172.21.15.26:0/2779934760 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788358745 0 0) 0x7f84300102b0 con 0x7f84300036d0 2024-09-16T14:45:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.704+0000 7f844a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.704+0000 7f844a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.704+0000 7f844a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.704+0000 7f844a7fc640 10 monclient(hunting): my global_id is 5111 2024-09-16T14:45:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.704+0000 7f844a7fc640 1 -- 172.21.15.26:0/2779934760 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f844c09e7d0 con 0x7f84300036d0 2024-09-16T14:45:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 1 -- 172.21.15.26:0/2779934760 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1875050750 0 0) 0x7f844c09e7d0 con 0x7f84300036d0 2024-09-16T14:45:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.189:6789/0 conn(0x7f843000f5c0 legacy=0x7f843000fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f84300093b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:45:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 1 monclient: found mon.noname-b 2024-09-16T14:45:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:45:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 1 -- 172.21.15.26:0/2779934760 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8430004b20 con 0x7f84300036d0 2024-09-16T14:45:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:45:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:45:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 1 -- 172.21.15.26:0/2779934760 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f84440019a0 con 0x7f84300036d0 2024-09-16T14:45:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 10 monclient: dump: 2024-09-16T14:45:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:41.712 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f8451560640 5 monclient: authenticate success, global_id 5111 2024-09-16T14:45:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f8451560640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 1 -- 172.21.15.26:0/2779934760 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8444001b90 con 0x7f84300036d0 2024-09-16T14:45:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 1 -- 172.21.15.26:0/2779934760 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8444001fa0 con 0x7f84300036d0 2024-09-16T14:45:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f844a7fc640 10 monclient: dump: 2024-09-16T14:45:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:41.714 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f8451560640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.708+0000 7f8451560640 4 set_mon_vals no callback set 2024-09-16T14:45:41.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 10 monclient: shutdown 2024-09-16T14:45:41.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:41.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 1 -- 172.21.15.26:0/2779934760 >> v1:172.21.15.26:6790/0 conn(0x7f84300036d0 legacy=0x7f843000cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:41.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 1 -- 172.21.15.26:0/2779934760 shutdown_connections 2024-09-16T14:45:41.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 1 -- 172.21.15.26:0/2779934760 >> 172.21.15.26:0/2779934760 conn(0x7f844c09fb50 msgr2=0x7f844c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 1 -- 172.21.15.26:0/2779934760 shutdown_connections 2024-09-16T14:45:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 1 -- 172.21.15.26:0/2779934760 wait complete. 2024-09-16T14:45:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 10 monclient: build_initial_monmap 2024-09-16T14:45:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 10 monclient: monmap: 2024-09-16T14:45:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:41.717 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 1 Processor -- start 2024-09-16T14:45:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 1 -- start start 2024-09-16T14:45:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 10 monclient: init 2024-09-16T14:45:41.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:41.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:41.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 10 monclient(hunting): picked mon.noname-b con 0x7f84300036d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:41.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 10 monclient(hunting): picked mon.noname-c con 0x7f843000f5c0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:41.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 10 monclient(hunting): picked mon.noname-a con 0x7f844c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:41.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c124030 con 0x7f844c099a70 2024-09-16T14:45:41.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c124310 con 0x7f84300036d0 2024-09-16T14:45:41.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c122b80 con 0x7f843000f5c0 2024-09-16T14:45:41.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:41.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8451560640 10 monclient(hunting): authenticate will time out at 1165.835644s 2024-09-16T14:45:41.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f844b7fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f844c1d2040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:41.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8443fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f843000f5c0 0x7f844c1ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35770/0 (socket says 172.21.15.26:35770) 2024-09-16T14:45:41.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.712+0000 7f8443fff640 1 -- 172.21.15.26:0/3220713785 learned_addr learned my addr 172.21.15.26:0/3220713785 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:41.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273821989 0 0) 0x7f844c122b80 con 0x7f843000f5c0 2024-09-16T14:45:41.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient(hunting): _negotiate 2024-09-16T14:45:41.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:41.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:41.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient(hunting): my global_id is 5114 2024-09-16T14:45:41.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8430003400 con 0x7f843000f5c0 2024-09-16T14:45:41.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742877999 0 0) 0x7f844c124310 con 0x7f84300036d0 2024-09-16T14:45:41.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient(hunting): _negotiate 2024-09-16T14:45:41.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:41.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:41.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient(hunting): my global_id is 94225 2024-09-16T14:45:41.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f844c122b80 con 0x7f84300036d0 2024-09-16T14:45:41.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3212348249 0 0) 0x7f8430003400 con 0x7f843000f5c0 2024-09-16T14:45:41.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f844c124310 con 0x7f843000f5c0 2024-09-16T14:45:41.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f843c004450 con 0x7f843000f5c0 2024-09-16T14:45:41.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:41.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:41.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient(hunting): dump: 2024-09-16T14:45:41.723 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:41.723 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:41.723 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:41.723 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:41.723 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:41.723 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:41.723 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:41.724 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:41.724 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:41.724 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:41.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:41.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3292092748 0 0) 0x7f844c122b80 con 0x7f84300036d0 2024-09-16T14:45:41.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8430003400 con 0x7f84300036d0 2024-09-16T14:45:41.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f84340016f0 con 0x7f84300036d0 2024-09-16T14:45:41.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:41.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:41.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient(hunting): dump: 2024-09-16T14:45:41.724 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:41.725 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:41.725 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:41.725 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:41.725 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:41.725 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:41.725 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:41.725 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:41.725 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:41.725 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:41.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1187085977 0 0) 0x7f844c124310 con 0x7f843000f5c0 2024-09-16T14:45:41.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 >> v1:172.21.15.189:6789/0 conn(0x7f84300036d0 legacy=0x7f844c1c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:41.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 >> v1:172.21.15.26:6789/0 conn(0x7f844c099a70 legacy=0x7f844c1d2040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient: _finish_hunting 0 2024-09-16T14:45:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 monclient: found mon.c 2024-09-16T14:45:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c133fd0 con 0x7f843000f5c0 2024-09-16T14:45:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient: _finish_auth 0 2024-09-16T14:45:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f8451560640 5 monclient: authenticate success, global_id 5114 2024-09-16T14:45:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f8451560640 10 monclient: _renew_subs 2024-09-16T14:45:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f8451560640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f8451560640 1 -- 172.21.15.26:0/3220713785 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844c09c180 con 0x7f843000f5c0 2024-09-16T14:45:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f8451560640 10 monclient: _renew_subs 2024-09-16T14:45:41.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f8451560640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:41.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f8451560640 1 -- 172.21.15.26:0/3220713785 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c123d50 con 0x7f843000f5c0 2024-09-16T14:45:41.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f843c004920 con 0x7f843000f5c0 2024-09-16T14:45:41.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:41.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f843c004c30 con 0x7f843000f5c0 2024-09-16T14:45:41.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:41.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:41.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84497fa640 4 set_mon_vals no callback set 2024-09-16T14:45:41.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 10 monclient: dump: 2024-09-16T14:45:41.727 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:41.727 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:41.727 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:41.727 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:41.727 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:41.728 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f8451560640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f8451560640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f8451560640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f8451560640 1 -- 172.21.15.26:0/3220713785 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8424005050 con 0x7f843000f5c0 2024-09-16T14:45:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f843c004d90 con 0x7f843000f5c0 2024-09-16T14:45:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(59..59 src has 15..59) v4 ==== 4305+0+0 (unknown 2223704037 0 0) 0x7f843c043400 con 0x7f843000f5c0 2024-09-16T14:45:41.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.716+0000 7f844affd640 10 monclient: get_auth_request con 0x7f8430037340 auth_method 0 2024-09-16T14:45:41.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.720+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f843c014470 con 0x7f843000f5c0 2024-09-16T14:45:41.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.720+0000 7f84437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:41.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.720+0000 7f84437fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:41.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.948+0000 7f8451560640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:45:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.948+0000 7f8451560640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.948+0000 7f8451560640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.948+0000 7f8451560640 1 -- 172.21.15.26:0/3220713785 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8424005300 con 0x7f843000f5c0 2024-09-16T14:45:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.952+0000 7f84437fe640 1 -- 172.21.15.26:0/3220713785 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2122657139) 0x7f843c014290 con 0x7f843000f5c0 2024-09-16T14:45:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.952+0000 7f84437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:45:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.952+0000 7f84437fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:41.955 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":36,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:45:41.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.952+0000 7f8451560640 1 -- 172.21.15.26:0/3220713785 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8430037340 legacy=0x7f8430039800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:41.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.952+0000 7f8451560640 10 monclient: shutdown 2024-09-16T14:45:41.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.952+0000 7f8451560640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:41.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.952+0000 7f8451560640 1 -- 172.21.15.26:0/3220713785 >> v1:172.21.15.26:6790/0 conn(0x7f843000f5c0 legacy=0x7f844c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:41.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.952+0000 7f8451560640 1 -- 172.21.15.26:0/3220713785 shutdown_connections 2024-09-16T14:45:41.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.952+0000 7f8451560640 1 -- 172.21.15.26:0/3220713785 >> 172.21.15.26:0/3220713785 conn(0x7f844c09fb50 msgr2=0x7f844c0a0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:41.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.952+0000 7f8451560640 1 -- 172.21.15.26:0/3220713785 shutdown_connections 2024-09-16T14:45:41.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:41.952+0000 7f8451560640 1 -- 172.21.15.26:0/3220713785 wait complete. 2024-09-16T14:45:41.974 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:45:41.974 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:45:41.975 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:45:42.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 10 monclient: get_monmap_and_config 2024-09-16T14:45:42.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 10 monclient: build_initial_monmap 2024-09-16T14:45:42.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 10 monclient: monmap: 2024-09-16T14:45:42.094 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:42.094 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.094 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:42.093801+0000 2024-09-16T14:45:42.094 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:42.093801+0000 2024-09-16T14:45:42.094 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:42.094 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:42.094 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:42.094 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:42.095 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:42.095 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:42.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 1 Processor -- start 2024-09-16T14:45:42.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 1 -- start start 2024-09-16T14:45:42.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 10 monclient: init 2024-09-16T14:45:42.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:42.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:42.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 10 monclient(hunting): picked mon.noname-c con 0x7fd4b80a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:42.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 10 monclient(hunting): picked mon.noname-b con 0x7fd4b8099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:42.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 10 monclient(hunting): picked mon.noname-a con 0x7fd4b80a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:42.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b8098ea0 con 0x7fd4b80a26f0 2024-09-16T14:45:42.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b80997b0 con 0x7fd4b80a1c00 2024-09-16T14:45:42.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b8099460 con 0x7fd4b8099b60 2024-09-16T14:45:42.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:42.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 10 monclient(hunting): authenticate will time out at 1166.214097s 2024-09-16T14:45:42.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4bcb58640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fd4b80a26f0 legacy=0x7fd4b813d330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:42.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b77fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd4b8099b60 0x7fd4b8139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35782/0 (socket says 172.21.15.26:35782) 2024-09-16T14:45:42.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b77fe640 1 -- 172.21.15.26:0/2958462196 learned_addr learned my addr 172.21.15.26:0/2958462196 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:42.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 1 -- 172.21.15.26:0/2958462196 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11800460 0 0) 0x7fd4b8099460 con 0x7fd4b8099b60 2024-09-16T14:45:42.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:42.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:42.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:42.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient(hunting): my global_id is 5117 2024-09-16T14:45:42.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 1 -- 172.21.15.26:0/2958462196 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4a4003530 con 0x7fd4b8099b60 2024-09-16T14:45:42.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 1 -- 172.21.15.26:0/2958462196 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611048939 0 0) 0x7fd4b80997b0 con 0x7fd4b80a1c00 2024-09-16T14:45:42.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:42.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:42.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:42.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient(hunting): my global_id is 94228 2024-09-16T14:45:42.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 1 -- 172.21.15.26:0/2958462196 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4b8099460 con 0x7fd4b80a1c00 2024-09-16T14:45:42.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 1 -- 172.21.15.26:0/2958462196 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2602080174 0 0) 0x7fd4a4003530 con 0x7fd4b8099b60 2024-09-16T14:45:42.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 1 -- 172.21.15.26:0/2958462196 >> v1:172.21.15.189:6789/0 conn(0x7fd4b80a1c00 legacy=0x7fd4b80a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 1 -- 172.21.15.26:0/2958462196 >> v1:172.21.15.26:6789/0 conn(0x7fd4b80a26f0 legacy=0x7fd4b813d330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 1 monclient: found mon.noname-b 2024-09-16T14:45:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:45:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 1 -- 172.21.15.26:0/2958462196 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b80a34b0 con 0x7fd4b8099b60 2024-09-16T14:45:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 1 -- 172.21.15.26:0/2958462196 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd4a0001620 con 0x7fd4b8099b60 2024-09-16T14:45:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient: dump: 2024-09-16T14:45:42.101 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:42.102 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.102 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.102 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.102 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:42.102 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:42.102 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:42.102 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:42.102 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:42.102 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:42.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 5 monclient: authenticate success, global_id 5117 2024-09-16T14:45:42.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:42.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 1 -- 172.21.15.26:0/2958462196 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4a0001900 con 0x7fd4b8099b60 2024-09-16T14:45:42.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:42.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 1 -- 172.21.15.26:0/2958462196 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd4a0001c90 con 0x7fd4b8099b60 2024-09-16T14:45:42.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:42.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:42.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4b6ffd640 10 monclient: dump: 2024-09-16T14:45:42.103 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:42.103 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.103 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.103 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.103 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:42.103 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:42.103 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:42.103 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:42.103 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:42.103 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:42.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 4 set_mon_vals no callback set 2024-09-16T14:45:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 10 monclient: shutdown 2024-09-16T14:45:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 1 -- 172.21.15.26:0/2958462196 >> v1:172.21.15.26:6790/0 conn(0x7fd4b8099b60 legacy=0x7fd4b8139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 1 -- 172.21.15.26:0/2958462196 shutdown_connections 2024-09-16T14:45:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 1 -- 172.21.15.26:0/2958462196 >> 172.21.15.26:0/2958462196 conn(0x7fd4b80a3d00 msgr2=0x7fd4b80a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 1 -- 172.21.15.26:0/2958462196 shutdown_connections 2024-09-16T14:45:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.092+0000 7fd4be5bb640 1 -- 172.21.15.26:0/2958462196 wait complete. 2024-09-16T14:45:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 10 monclient: build_initial_monmap 2024-09-16T14:45:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 10 monclient: monmap: 2024-09-16T14:45:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:42.104 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:42.105 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:42.105 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:42.105 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:42.105 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:42.105 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:42.105 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:42.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 1 Processor -- start 2024-09-16T14:45:42.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 1 -- start start 2024-09-16T14:45:42.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 10 monclient: init 2024-09-16T14:45:42.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:42.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:42.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 10 monclient(hunting): picked mon.noname-c con 0x7fd4b8099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:42.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 10 monclient(hunting): picked mon.noname-b con 0x7fd4b80a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:42.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 10 monclient(hunting): picked mon.noname-a con 0x7fd4b80a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b813c7c0 con 0x7fd4b80a26f0 2024-09-16T14:45:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b809e3f0 con 0x7fd4b80a1c00 2024-09-16T14:45:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b813dda0 con 0x7fd4b8099b60 2024-09-16T14:45:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4bcb58640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fd4b80a26f0 legacy=0x7fd4b81d1930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 10 monclient(hunting): authenticate will time out at 1166.217445s 2024-09-16T14:45:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b7fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd4b8099b60 0x7fd4b8139e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35792/0 (socket says 172.21.15.26:35792) 2024-09-16T14:45:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b7fff640 1 -- 172.21.15.26:0/1022901675 learned_addr learned my addr 172.21.15.26:0/1022901675 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504858251 0 0) 0x7fd4b813dda0 con 0x7fd4b8099b60 2024-09-16T14:45:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:42.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient(hunting): my global_id is 5120 2024-09-16T14:45:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd498003530 con 0x7fd4b8099b60 2024-09-16T14:45:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 601864944 0 0) 0x7fd498003530 con 0x7fd4b8099b60 2024-09-16T14:45:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4b813dda0 con 0x7fd4b8099b60 2024-09-16T14:45:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd4a8001920 con 0x7fd4b8099b60 2024-09-16T14:45:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient(hunting): dump: 2024-09-16T14:45:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:42.112 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:42.113 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:42.113 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:42.113 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:42.113 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:42.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:42.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044450873 0 0) 0x7fd4b809e3f0 con 0x7fd4b80a1c00 2024-09-16T14:45:42.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:42.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:42.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:42.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient(hunting): my global_id is 94231 2024-09-16T14:45:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd498003530 con 0x7fd4b80a1c00 2024-09-16T14:45:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1414538687 0 0) 0x7fd4b813dda0 con 0x7fd4b8099b60 2024-09-16T14:45:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 >> v1:172.21.15.189:6789/0 conn(0x7fd4b80a1c00 legacy=0x7fd4b813a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 >> v1:172.21.15.26:6789/0 conn(0x7fd4b80a26f0 legacy=0x7fd4b81d1930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:45:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 monclient: found mon.c 2024-09-16T14:45:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b813ac90 con 0x7fd4b8099b60 2024-09-16T14:45:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient: _finish_auth 0 2024-09-16T14:45:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:45:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:42.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 5 monclient: authenticate success, global_id 5120 2024-09-16T14:45:42.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 10 monclient: _renew_subs 2024-09-16T14:45:42.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:42.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 1 -- 172.21.15.26:0/1022901675 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b813c320 con 0x7fd4b8099b60 2024-09-16T14:45:42.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 10 monclient: _renew_subs 2024-09-16T14:45:42.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:42.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 1 -- 172.21.15.26:0/1022901675 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b809ed00 con 0x7fd4b8099b60 2024-09-16T14:45:42.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4a8001cf0 con 0x7fd4b8099b60 2024-09-16T14:45:42.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:42.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd4a8003af0 con 0x7fd4b8099b60 2024-09-16T14:45:42.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:42.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient: dump: 2024-09-16T14:45:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:42.116 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b5ffb640 4 set_mon_vals no callback set 2024-09-16T14:45:42.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:42.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4be5bb640 1 -- 172.21.15.26:0/1022901675 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd49c006590 con 0x7fd49c002830 2024-09-16T14:45:42.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd4a8003c50 con 0x7fd4b8099b60 2024-09-16T14:45:42.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(59..59 src has 15..59) v4 ==== 4305+0+0 (unknown 2223704037 0 0) 0x7fd4a8043540 con 0x7fd4b8099b60 2024-09-16T14:45:42.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321325318 0 0) 0x7fd49c006590 con 0x7fd49c002830 2024-09-16T14:45:42.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:42.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:42.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:42.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 10 monclient(hunting): my global_id is 5123 2024-09-16T14:45:42.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4b813dda0 con 0x7fd49c002830 2024-09-16T14:45:42.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4bcb58640 10 monclient: get_auth_request con 0x7fd498036130 auth_method 0 2024-09-16T14:45:42.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1088328185 0 0) 0x7fd4b813dda0 con 0x7fd49c002830 2024-09-16T14:45:42.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.096+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd49c006590 con 0x7fd49c002830 2024-09-16T14:45:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.100+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd4a0001cd0 con 0x7fd49c002830 2024-09-16T14:45:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.100+0000 7fd4b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.100+0000 7fd4b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.100+0000 7fd4b4ff9640 10 monclient: dump: 2024-09-16T14:45:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:42.118 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:42.119 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.100+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3470366904 0 0) 0x7fd49c006590 con 0x7fd49c002830 2024-09-16T14:45:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.100+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd49c00a0e0 con 0x7fd49c002830 2024-09-16T14:45:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.100+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fd4a0001610 con 0x7fd49c002830 2024-09-16T14:45:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.100+0000 7fd4b4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.100+0000 7fd4b4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.100+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 >> v1:172.21.15.26:6790/0 conn(0x7fd49c002830 legacy=0x7fd49c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.100+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 >> v1:172.21.15.26:6790/0 conn(0x7fd49c002830 legacy=0x7fd49c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4be5bb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:45:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4be5bb640 1 -- 172.21.15.26:0/1022901675 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd49c009da0 con 0x7fd49c0090c0 2024-09-16T14:45:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72747563 0 0) 0x7fd49c009da0 con 0x7fd49c0090c0 2024-09-16T14:45:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:42.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 10 monclient(hunting): my global_id is 5126 2024-09-16T14:45:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd49c006590 con 0x7fd49c0090c0 2024-09-16T14:45:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2115482694 0 0) 0x7fd49c006590 con 0x7fd49c0090c0 2024-09-16T14:45:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd49c009da0 con 0x7fd49c0090c0 2024-09-16T14:45:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd4a0004df0 con 0x7fd49c0090c0 2024-09-16T14:45:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 10 monclient: dump: 2024-09-16T14:45:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.120 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:42.121 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4120927454 0 0) 0x7fd49c009da0 con 0x7fd49c0090c0 2024-09-16T14:45:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd49c00c290 con 0x7fd49c0090c0 2024-09-16T14:45:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4202 (unknown 0 0 1696142349) 0x7fd4a0005010 con 0x7fd49c0090c0 2024-09-16T14:45:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:45:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 >> v1:172.21.15.26:6790/0 conn(0x7fd49c0090c0 legacy=0x7fd49c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4b4ff9640 1 -- 172.21.15.26:0/1022901675 >> v1:172.21.15.26:6790/0 conn(0x7fd49c0090c0 legacy=0x7fd49c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:42.122 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:45:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:45:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:45:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:45:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 36, 2024-09-16T14:45:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:45:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:45:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:45:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:45:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:45:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:45:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:45:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:45:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:45:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:45:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:45:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:45:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:45:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:45:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:45:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:45:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:45:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:45:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:45:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:45:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:45:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:45:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:45:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:45:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:45:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:45:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:45:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:45:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:45:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:45:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:45:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:45:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:45:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:45:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:45:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:45:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:45:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:45:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:45:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:45:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:45:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:45:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:45:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:45:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:45:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:45:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:45:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:45:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:45:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:45:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:45:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:45:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:45:42.133 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:45:42.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.104+0000 7fd4be5bb640 1 -- 172.21.15.26:0/1022901675 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd498036130 legacy=0x7fd4980385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.108+0000 7fd4be5bb640 10 monclient: shutdown 2024-09-16T14:45:42.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.108+0000 7fd4be5bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:42.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.108+0000 7fd4be5bb640 1 -- 172.21.15.26:0/1022901675 >> v1:172.21.15.26:6790/0 conn(0x7fd4b8099b60 legacy=0x7fd4b8139e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.108+0000 7fd4bcb58640 1 -- 172.21.15.26:0/1022901675 reap_dead start 2024-09-16T14:45:42.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.108+0000 7fd4bcb58640 1 -- 172.21.15.26:0/1022901675 reap_dead start 2024-09-16T14:45:42.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.108+0000 7fd4be5bb640 1 -- 172.21.15.26:0/1022901675 shutdown_connections 2024-09-16T14:45:42.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.108+0000 7fd4be5bb640 1 -- 172.21.15.26:0/1022901675 >> 172.21.15.26:0/1022901675 conn(0x7fd4b80a3d00 msgr2=0x7fd4b80a0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:42.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.108+0000 7fd4be5bb640 1 -- 172.21.15.26:0/1022901675 shutdown_connections 2024-09-16T14:45:42.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.108+0000 7fd4be5bb640 1 -- 172.21.15.26:0/1022901675 wait complete. 2024-09-16T14:45:42.135 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:45:42.135 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T14:45:42.135 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:45:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 10 monclient: get_monmap_and_config 2024-09-16T14:45:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 10 monclient: build_initial_monmap 2024-09-16T14:45:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 10 monclient: monmap: 2024-09-16T14:45:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:45:42.249971+0000 2024-09-16T14:45:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:45:42.249971+0000 2024-09-16T14:45:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:42.251 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:42.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 1 Processor -- start 2024-09-16T14:45:42.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 1 -- start start 2024-09-16T14:45:42.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 10 monclient: init 2024-09-16T14:45:42.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:42.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:42.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 10 monclient(hunting): picked mon.noname-c con 0x7fa8c409cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:42.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 10 monclient(hunting): picked mon.noname-a con 0x7fa8c409e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:42.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 10 monclient(hunting): picked mon.noname-b con 0x7fa8c409d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:42.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c409d5b0 con 0x7fa8c409e7a0 2024-09-16T14:45:42.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c4131a90 con 0x7fa8c409cdc0 2024-09-16T14:45:42.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c409aff0 con 0x7fa8c409d1a0 2024-09-16T14:45:42.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:42.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 10 monclient(hunting): authenticate will time out at 1166.369791s 2024-09-16T14:45:42.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c37fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fa8c409e7a0 legacy=0x7fa8c4131380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:42.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c88b3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa8c409d1a0 0x7fa8c4139fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35818/0 (socket says 172.21.15.26:35818) 2024-09-16T14:45:42.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c88b3640 1 -- 172.21.15.26:0/850895456 learned_addr learned my addr 172.21.15.26:0/850895456 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:42.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 1 -- 172.21.15.26:0/850895456 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593068358 0 0) 0x7fa8c409aff0 con 0x7fa8c409d1a0 2024-09-16T14:45:42.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:42.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:42.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:42.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 10 monclient(hunting): my global_id is 5129 2024-09-16T14:45:42.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 1 -- 172.21.15.26:0/850895456 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8ac003530 con 0x7fa8c409d1a0 2024-09-16T14:45:42.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 1 -- 172.21.15.26:0/850895456 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1687143402 0 0) 0x7fa8ac003530 con 0x7fa8c409d1a0 2024-09-16T14:45:42.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 1 -- 172.21.15.26:0/850895456 >> v1:172.21.15.189:6789/0 conn(0x7fa8c409cdc0 legacy=0x7fa8c4130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 1 -- 172.21.15.26:0/850895456 >> v1:172.21.15.26:6789/0 conn(0x7fa8c409e7a0 legacy=0x7fa8c4131380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:42.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:42.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 1 monclient: found mon.noname-b 2024-09-16T14:45:42.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:42.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:45:42.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 1 -- 172.21.15.26:0/850895456 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c409b2d0 con 0x7fa8c409d1a0 2024-09-16T14:45:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 1 -- 172.21.15.26:0/850895456 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa8bc0019a0 con 0x7fa8c409d1a0 2024-09-16T14:45:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 5 monclient: authenticate success, global_id 5129 2024-09-16T14:45:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 10 monclient: dump: 2024-09-16T14:45:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:42.261 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 1 -- 172.21.15.26:0/850895456 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8bc001c50 con 0x7fa8c409d1a0 2024-09-16T14:45:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 1 -- 172.21.15.26:0/850895456 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa8bc001fe0 con 0x7fa8c409d1a0 2024-09-16T14:45:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:42.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 4 set_mon_vals no callback set 2024-09-16T14:45:42.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:42.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:42.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8c2ffd640 10 monclient: dump: 2024-09-16T14:45:42.262 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:42.262 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.262 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.262 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.262 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:42.262 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:42.262 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:42.262 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:42.262 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:42.262 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:42.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 10 monclient: shutdown 2024-09-16T14:45:42.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:42.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 1 -- 172.21.15.26:0/850895456 >> v1:172.21.15.26:6790/0 conn(0x7fa8c409d1a0 legacy=0x7fa8c4139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 1 -- 172.21.15.26:0/850895456 shutdown_connections 2024-09-16T14:45:42.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 1 -- 172.21.15.26:0/850895456 >> 172.21.15.26:0/850895456 conn(0x7fa8c409fb50 msgr2=0x7fa8c409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:42.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 1 -- 172.21.15.26:0/850895456 shutdown_connections 2024-09-16T14:45:42.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.248+0000 7fa8ca316640 1 -- 172.21.15.26:0/850895456 wait complete. 2024-09-16T14:45:42.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 10 monclient: build_initial_monmap 2024-09-16T14:45:42.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 10 monclient: monmap: 2024-09-16T14:45:42.263 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:45:42.263 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:42.263 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:45:42.263 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:45:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:45:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:42.264 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 1 Processor -- start 2024-09-16T14:45:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 1 -- start start 2024-09-16T14:45:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 10 monclient: init 2024-09-16T14:45:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 10 monclient(hunting): picked mon.noname-a con 0x7fa8c409cdc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 10 monclient(hunting): picked mon.noname-c con 0x7fa8c409d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 10 monclient(hunting): picked mon.noname-b con 0x7fa8c409e7a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c411e260 con 0x7fa8c409cdc0 2024-09-16T14:45:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c3fff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fa8c409cdc0 legacy=0x7fa8c412de30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c409a240 con 0x7fa8c409e7a0 2024-09-16T14:45:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c40712a0 con 0x7fa8c409d1a0 2024-09-16T14:45:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 10 monclient(hunting): authenticate will time out at 1166.372983s 2024-09-16T14:45:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c37fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa8c409d1a0 0x7fa8c41ce650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35822/0 (socket says 172.21.15.26:35822) 2024-09-16T14:45:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c37fe640 1 -- 172.21.15.26:0/2908022538 learned_addr learned my addr 172.21.15.26:0/2908022538 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:45:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326267300 0 0) 0x7fa8c40712a0 con 0x7fa8c409d1a0 2024-09-16T14:45:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient(hunting): my global_id is 5132 2024-09-16T14:45:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b0003530 con 0x7fa8c409d1a0 2024-09-16T14:45:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3121546019 0 0) 0x7fa8b0003530 con 0x7fa8c409d1a0 2024-09-16T14:45:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8c40712a0 con 0x7fa8c409d1a0 2024-09-16T14:45:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa8b8004160 con 0x7fa8c409d1a0 2024-09-16T14:45:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient(hunting): dump: 2024-09-16T14:45:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:42.267 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77310918 0 0) 0x7fa8c409a240 con 0x7fa8c409e7a0 2024-09-16T14:45:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient(hunting): my global_id is 94237 2024-09-16T14:45:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b0003530 con 0x7fa8c409e7a0 2024-09-16T14:45:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2630679278 0 0) 0x7fa8c40712a0 con 0x7fa8c409d1a0 2024-09-16T14:45:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 >> v1:172.21.15.189:6789/0 conn(0x7fa8c409e7a0 legacy=0x7fa8c41d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 >> v1:172.21.15.26:6789/0 conn(0x7fa8c409cdc0 legacy=0x7fa8c412de30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:45:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 monclient: found mon.c 2024-09-16T14:45:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c411df90 con 0x7fa8c409d1a0 2024-09-16T14:45:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient: _finish_auth 0 2024-09-16T14:45:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:45:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 5 monclient: authenticate success, global_id 5132 2024-09-16T14:45:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 10 monclient: _renew_subs 2024-09-16T14:45:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 1 -- 172.21.15.26:0/2908022538 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c40a0f30 con 0x7fa8c409d1a0 2024-09-16T14:45:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 10 monclient: _renew_subs 2024-09-16T14:45:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:45:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 1 -- 172.21.15.26:0/2908022538 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c41303f0 con 0x7fa8c409d1a0 2024-09-16T14:45:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8b8004410 con 0x7fa8c409d1a0 2024-09-16T14:45:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa8b80047a0 con 0x7fa8c409d1a0 2024-09-16T14:45:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:45:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c27fc640 4 set_mon_vals no callback set 2024-09-16T14:45:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient: dump: 2024-09-16T14:45:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:42.274 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8ca316640 1 -- 172.21.15.26:0/2908022538 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa8a4006590 con 0x7fa8a4002830 2024-09-16T14:45:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa8b8004900 con 0x7fa8c409d1a0 2024-09-16T14:45:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(59..59 src has 15..59) v4 ==== 4305+0+0 (unknown 2223704037 0 0) 0x7fa8b80434b0 con 0x7fa8c409d1a0 2024-09-16T14:45:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c88b3640 10 monclient: get_auth_request con 0x7fa8b0036130 auth_method 0 2024-09-16T14:45:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652208603 0 0) 0x7fa8a4006590 con 0x7fa8a4002830 2024-09-16T14:45:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient(hunting): my global_id is 94240 2024-09-16T14:45:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c40712a0 con 0x7fa8a4002830 2024-09-16T14:45:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2031582261 0 0) 0x7fa8c40712a0 con 0x7fa8a4002830 2024-09-16T14:45:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8a4006590 con 0x7fa8a4002830 2024-09-16T14:45:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa8b4001790 con 0x7fa8a4002830 2024-09-16T14:45:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.252+0000 7fa8c0ff9640 10 monclient: dump: 2024-09-16T14:45:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.275 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.276 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:42.276 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:42.276 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:42.276 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:42.276 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:42.276 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:42.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.256+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1185620769 0 0) 0x7fa8a4006590 con 0x7fa8a4002830 2024-09-16T14:45:42.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.256+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa8a400a0e0 con 0x7fa8a4002830 2024-09-16T14:45:42.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.256+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa8b4001b40 con 0x7fa8a4002830 2024-09-16T14:45:42.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.256+0000 7fa8c0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:42.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.256+0000 7fa8c0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:42.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.256+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 >> v1:172.21.15.189:6789/0 conn(0x7fa8a4002830 legacy=0x7fa8a4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.256+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 >> v1:172.21.15.189:6789/0 conn(0x7fa8a4002830 legacy=0x7fa8a4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:42.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.260+0000 7fa8ca316640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:45:42.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.260+0000 7fa8ca316640 1 -- 172.21.15.26:0/2908022538 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa8a4009da0 con 0x7fa8a40090c0 2024-09-16T14:45:42.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041505817 0 0) 0x7fa8a4009da0 con 0x7fa8a40090c0 2024-09-16T14:45:42.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:42.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:42.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:42.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 10 monclient(hunting): my global_id is 94243 2024-09-16T14:45:42.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a4006590 con 0x7fa8a40090c0 2024-09-16T14:45:42.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3429621453 0 0) 0x7fa8a4006590 con 0x7fa8a40090c0 2024-09-16T14:45:42.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8a4009da0 con 0x7fa8a40090c0 2024-09-16T14:45:42.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa8b4004b30 con 0x7fa8a40090c0 2024-09-16T14:45:42.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:42.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:42.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 10 monclient: dump: 2024-09-16T14:45:42.279 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:45:42.279 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.279 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.279 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.279 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:45:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:42.280 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:45:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 882009549 0 0) 0x7fa8a4009da0 con 0x7fa8a40090c0 2024-09-16T14:45:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa8a400c290 con 0x7fa8a40090c0 2024-09-16T14:45:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3868 (unknown 0 0 1534447970) 0x7fa8b4005f30 con 0x7fa8a40090c0 2024-09-16T14:45:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:45:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 >> v1:172.21.15.189:6789/0 conn(0x7fa8a40090c0 legacy=0x7fa8a4009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c0ff9640 1 -- 172.21.15.26:0/2908022538 >> v1:172.21.15.189:6789/0 conn(0x7fa8a40090c0 legacy=0x7fa8a4009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:45:42.281 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:45:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:45:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:45:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:45:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 36, 2024-09-16T14:45:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:45:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:45:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:45:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:45:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:45:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:45:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:45:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:45:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:45:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:45:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:45:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:45:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:45:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:45:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:45:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:45:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:45:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:45:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:45:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:45:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:45:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:45:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:45:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:45:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:45:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:45:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:45:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:45:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:45:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:45:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:45:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:45:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:45:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:45:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:45:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:45:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:45:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:45:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:45:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:45:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:45:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:45:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:45:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:45:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:45:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:45:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:45:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:45:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:45:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:45:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:45:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:45:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:45:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:45:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:45:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:45:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:45:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:45:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:45:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:45:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:45:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:45:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:45:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:45:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:45:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:45:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:45:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:45:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:45:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:45:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:45:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:45:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:45:42.291 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:45:42.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8ca316640 1 -- 172.21.15.26:0/2908022538 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa8b0036130 legacy=0x7fa8b00385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8ca316640 10 monclient: shutdown 2024-09-16T14:45:42.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8ca316640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:42.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8ca316640 1 -- 172.21.15.26:0/2908022538 >> v1:172.21.15.26:6790/0 conn(0x7fa8c409d1a0 legacy=0x7fa8c41ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c88b3640 1 -- 172.21.15.26:0/2908022538 reap_dead start 2024-09-16T14:45:42.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8c88b3640 1 -- 172.21.15.26:0/2908022538 reap_dead start 2024-09-16T14:45:42.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8ca316640 1 -- 172.21.15.26:0/2908022538 shutdown_connections 2024-09-16T14:45:42.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8ca316640 1 -- 172.21.15.26:0/2908022538 >> 172.21.15.26:0/2908022538 conn(0x7fa8c409fb50 msgr2=0x7fa8c41b9ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:42.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8ca316640 1 -- 172.21.15.26:0/2908022538 shutdown_connections 2024-09-16T14:45:42.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:45:42.264+0000 7fa8ca316640 1 -- 172.21.15.26:0/2908022538 wait complete. 2024-09-16T14:45:42.294 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:45:42.294 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:45:42.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.652+0000 7febaf7fe640 10 monclient: tick 2024-09-16T14:45:42.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.652+0000 7febaf7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:42.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.652+0000 7febaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:42.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.692+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 v60) v1 ==== 80+0+0 (unknown 928933629 0 0) 0x7febbc012c00 con 0x7febc809e2f0 2024-09-16T14:45:42.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.692+0000 7febc4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T14:45:42.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.692+0000 7febc4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:42.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.792+0000 7febce9b1640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T14:45:42.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.792+0000 7febce9b1640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T14:45:42.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.792+0000 7febce9b1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:42.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.792+0000 7febce9b1640 1 -- 172.21.15.189:0/4253063331 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7feba8005850 con 0x7febc809e2f0 2024-09-16T14:45:42.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.792+0000 7febc4ff9640 1 -- 172.21.15.189:0/4253063331 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v60) v1 ==== 103+0+0 (unknown 2158068389 0 0) 0x7febbc014380 con 0x7febc809e2f0 2024-09-16T14:45:42.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.792+0000 7febc4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T14:45:42.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.792+0000 7febc4ff9640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-16T14:45:42.797 INFO:tasks.workunit.client.0.smithi189.stderr:rule foo does not exist 2024-09-16T14:45:42.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.796+0000 7febce9b1640 1 -- 172.21.15.189:0/4253063331 >> v1:172.21.15.189:6812/3196161582 conn(0x7febb40363d0 legacy=0x7febb4038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.796+0000 7febce9b1640 10 monclient: shutdown 2024-09-16T14:45:42.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.796+0000 7febce9b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:42.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.796+0000 7febce9b1640 1 -- 172.21.15.189:0/4253063331 >> v1:172.21.15.189:6789/0 conn(0x7febc809e2f0 legacy=0x7febc81ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.796+0000 7febce9b1640 1 -- 172.21.15.189:0/4253063331 shutdown_connections 2024-09-16T14:45:42.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.796+0000 7febce9b1640 1 -- 172.21.15.189:0/4253063331 >> 172.21.15.189:0/4253063331 conn(0x7febc80a3320 msgr2=0x7febc806d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:42.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.796+0000 7febce9b1640 1 -- 172.21.15.189:0/4253063331 shutdown_connections 2024-09-16T14:45:42.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.796+0000 7febce9b1640 1 -- 172.21.15.189:0/4253063331 wait complete. 2024-09-16T14:45:42.814 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rm foo 2024-09-16T14:45:42.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.928+0000 7f880fdce640 10 monclient: get_monmap_and_config 2024-09-16T14:45:42.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.928+0000 7f880fdce640 10 monclient: build_initial_monmap 2024-09-16T14:45:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.928+0000 7f880fdce640 10 monclient: monmap: 2024-09-16T14:45:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:42.933381+0000 2024-09-16T14:45:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:42.933381+0000 2024-09-16T14:45:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:42.934 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:42.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 1 Processor -- start 2024-09-16T14:45:42.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 1 -- start start 2024-09-16T14:45:42.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 10 monclient: init 2024-09-16T14:45:42.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 10 monclient(hunting): picked mon.noname-c con 0x7f88081315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 10 monclient(hunting): picked mon.noname-b con 0x7f880809b320 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 10 monclient(hunting): picked mon.noname-a con 0x7f8808132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f880809edd0 con 0x7f8808132090 2024-09-16T14:45:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f880809bc00 con 0x7f88081315a0 2024-09-16T14:45:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f880809e900 con 0x7f880809b320 2024-09-16T14:45:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 10 monclient(hunting): authenticate will time out at 1125.049158s 2024-09-16T14:45:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880db6a640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f88081315a0 0x7f8808131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50262/0 (socket says 172.21.15.189:50262) 2024-09-16T14:45:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880db6a640 1 -- 172.21.15.189:0/3748937508 learned_addr learned my addr 172.21.15.189:0/3748937508 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880e36b640 1 -- 172.21.15.189:0/3748937508 >> v1:172.21.15.26:6789/0 conn(0x7f8808132090 legacy=0x7f880813a200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 1 -- 172.21.15.189:0/3748937508 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654823825 0 0) 0x7f880809bc00 con 0x7f88081315a0 2024-09-16T14:45:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 10 monclient(hunting): _negotiate 2024-09-16T14:45:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 10 monclient(hunting): my global_id is 94246 2024-09-16T14:45:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 1 -- 172.21.15.189:0/3748937508 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f0003530 con 0x7f88081315a0 2024-09-16T14:45:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 1 -- 172.21.15.189:0/3748937508 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 368223890 0 0) 0x7f87f0003530 con 0x7f88081315a0 2024-09-16T14:45:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 1 -- 172.21.15.189:0/3748937508 >> v1:172.21.15.26:6790/0 conn(0x7f880809b320 legacy=0x7f8808136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 1 -- 172.21.15.189:0/3748937508 >> v1:172.21.15.26:6789/0 conn(0x7f8808132090 legacy=0x7f880813a200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 10 monclient: _finish_hunting 0 2024-09-16T14:45:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 1 monclient: found mon.noname-c 2024-09-16T14:45:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 1 -- 172.21.15.189:0/3748937508 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f880809b920 con 0x7f88081315a0 2024-09-16T14:45:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 10 monclient: _finish_auth 0 2024-09-16T14:45:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 10 monclient: _check_auth_tickets 2024-09-16T14:45:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 5 monclient: authenticate success, global_id 94246 2024-09-16T14:45:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 1 -- 172.21.15.189:0/3748937508 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f87f80019a0 con 0x7f88081315a0 2024-09-16T14:45:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 10 monclient: dump: 2024-09-16T14:45:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:42.944 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 1 -- 172.21.15.189:0/3748937508 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87f8001cc0 con 0x7f88081315a0 2024-09-16T14:45:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 1 -- 172.21.15.189:0/3748937508 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f87f8002050 con 0x7f88081315a0 2024-09-16T14:45:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880cb68640 10 monclient: dump: 2024-09-16T14:45:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:42.946 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 4 set_mon_vals no callback set 2024-09-16T14:45:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 10 monclient: shutdown 2024-09-16T14:45:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 1 -- 172.21.15.189:0/3748937508 >> v1:172.21.15.189:6789/0 conn(0x7f88081315a0 legacy=0x7f8808131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 1 -- 172.21.15.189:0/3748937508 shutdown_connections 2024-09-16T14:45:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 1 -- 172.21.15.189:0/3748937508 >> 172.21.15.189:0/3748937508 conn(0x7f88080a3320 msgr2=0x7f880809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:42.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 1 -- 172.21.15.189:0/3748937508 shutdown_connections 2024-09-16T14:45:42.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 1 -- 172.21.15.189:0/3748937508 wait complete. 2024-09-16T14:45:42.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 10 monclient: build_initial_monmap 2024-09-16T14:45:42.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 10 monclient: monmap: 2024-09-16T14:45:42.947 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:42.947 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:42.947 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:42.947 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:42.947 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:42.947 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:42.947 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:42.948 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:42.948 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:42.948 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:42.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 1 Processor -- start 2024-09-16T14:45:42.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 1 -- start start 2024-09-16T14:45:42.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.932+0000 7f880fdce640 10 monclient: init 2024-09-16T14:45:42.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:42.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:42.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 10 monclient(hunting): picked mon.noname-b con 0x7f88081315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:42.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 10 monclient(hunting): picked mon.noname-a con 0x7f8808132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:42.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 10 monclient(hunting): picked mon.noname-c con 0x7f88081c4d70 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:42.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f880809cd20 con 0x7f8808132090 2024-09-16T14:45:42.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f880806b380 con 0x7f88081315a0 2024-09-16T14:45:42.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f880811ce20 con 0x7f88081c4d70 2024-09-16T14:45:42.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:42.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880db6a640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f88081315a0 0x7f880812f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50268/0 (socket says 172.21.15.189:50268) 2024-09-16T14:45:42.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880db6a640 1 -- 172.21.15.189:0/2764466713 learned_addr learned my addr 172.21.15.189:0/2764466713 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:42.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 10 monclient(hunting): authenticate will time out at 1125.052578s 2024-09-16T14:45:42.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880d369640 1 -- 172.21.15.189:0/2764466713 >> v1:172.21.15.26:6789/0 conn(0x7f8808132090 legacy=0x7f88081c9e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:42.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188343532 0 0) 0x7f880806b380 con 0x7f88081315a0 2024-09-16T14:45:42.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:42.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:42.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:42.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient(hunting): my global_id is 94249 2024-09-16T14:45:42.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f4003530 con 0x7f88081315a0 2024-09-16T14:45:42.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3664321310 0 0) 0x7f87f4003530 con 0x7f88081315a0 2024-09-16T14:45:42.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f880806b380 con 0x7f88081315a0 2024-09-16T14:45:42.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f87f8001960 con 0x7f88081315a0 2024-09-16T14:45:42.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:42.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:42.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient(hunting): dump: 2024-09-16T14:45:42.950 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:42.950 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.950 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.950 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.950 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:42.951 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:42.951 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:42.951 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:42.951 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:42.951 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:42.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:42.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530905176 0 0) 0x7f880811ce20 con 0x7f88081c4d70 2024-09-16T14:45:42.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:42.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:42.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:42.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient(hunting): my global_id is 5138 2024-09-16T14:45:42.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f4003530 con 0x7f88081c4d70 2024-09-16T14:45:42.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 506098245 0 0) 0x7f880806b380 con 0x7f88081315a0 2024-09-16T14:45:42.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 >> v1:172.21.15.26:6790/0 conn(0x7f88081c4d70 legacy=0x7f88081cd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:42.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 >> v1:172.21.15.26:6789/0 conn(0x7f8808132090 legacy=0x7f88081c9e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:42.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient: _finish_hunting 0 2024-09-16T14:45:42.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 1 monclient: found mon.b 2024-09-16T14:45:42.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:42.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:42.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f880809aa20 con 0x7f88081315a0 2024-09-16T14:45:42.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient: _finish_auth 0 2024-09-16T14:45:42.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient: _check_auth_tickets 2024-09-16T14:45:42.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:42.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 5 monclient: authenticate success, global_id 94249 2024-09-16T14:45:42.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 10 monclient: _renew_subs 2024-09-16T14:45:42.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:42.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 1 -- 172.21.15.189:0/2764466713 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f880809a5f0 con 0x7f88081315a0 2024-09-16T14:45:42.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 10 monclient: _renew_subs 2024-09-16T14:45:42.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:42.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 1 -- 172.21.15.189:0/2764466713 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f880811f180 con 0x7f88081315a0 2024-09-16T14:45:42.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87f80014e0 con 0x7f88081315a0 2024-09-16T14:45:42.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:42.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f87f80051a0 con 0x7f88081315a0 2024-09-16T14:45:42.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:42.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:42.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 10 monclient: dump: 2024-09-16T14:45:42.954 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:42.954 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:42.954 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.954 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:42.954 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:42.954 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:42.954 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:42.954 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:42.955 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:42.955 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:42.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f8807fff640 4 set_mon_vals no callback set 2024-09-16T14:45:42.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f87f8005300 con 0x7f88081315a0 2024-09-16T14:45:42.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:42.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:42.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:42.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880fdce640 1 -- 172.21.15.189:0/2764466713 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e8005050 con 0x7f88081315a0 2024-09-16T14:45:42.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(60..60 src has 15..60) v4 ==== 4250+0+0 (unknown 1167516382 0 0) 0x7f87f8043970 con 0x7f88081315a0 2024-09-16T14:45:42.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.936+0000 7f880d369640 10 monclient: get_auth_request con 0x7f87f40360e0 auth_method 0 2024-09-16T14:45:42.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.940+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f87f8014b00 con 0x7f88081315a0 2024-09-16T14:45:42.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.940+0000 7f88067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:42.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:42.940+0000 7f88067fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:43.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.092+0000 7f880fdce640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T14:45:43.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.092+0000 7f880fdce640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T14:45:43.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.092+0000 7f880fdce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:43.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.092+0000 7f880fdce640 1 -- 172.21.15.189:0/2764466713 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7f87e8005300 con 0x7f88081315a0 2024-09-16T14:45:43.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.096+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v60) v1 ==== 103+0+0 (unknown 2158068389 0 0) 0x7f87f8014920 con 0x7f88081315a0 2024-09-16T14:45:43.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.096+0000 7f88067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T14:45:43.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.096+0000 7f88067fc640 10 monclient: _finish_command 2 = system:0 rule foo does not exist 2024-09-16T14:45:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.184+0000 7f880fdce640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T14:45:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.184+0000 7f880fdce640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T14:45:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.184+0000 7f880fdce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.184+0000 7f880fdce640 1 -- 172.21.15.189:0/2764466713 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7f87e8005850 con 0x7f88081315a0 2024-09-16T14:45:43.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.184+0000 7f88067fc640 1 -- 172.21.15.189:0/2764466713 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v60) v1 ==== 103+0+0 (unknown 2158068389 0 0) 0x7f87f8014920 con 0x7f88081315a0 2024-09-16T14:45:43.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.184+0000 7f88067fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T14:45:43.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.184+0000 7f88067fc640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-16T14:45:43.189 INFO:tasks.workunit.client.0.smithi189.stderr:rule foo does not exist 2024-09-16T14:45:43.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.188+0000 7f880fdce640 1 -- 172.21.15.189:0/2764466713 >> v1:172.21.15.189:6812/3196161582 conn(0x7f87f40360e0 legacy=0x7f87f40385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:43.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.188+0000 7f880fdce640 10 monclient: shutdown 2024-09-16T14:45:43.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.188+0000 7f880fdce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:43.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.188+0000 7f880fdce640 1 -- 172.21.15.189:0/2764466713 >> v1:172.21.15.189:6789/0 conn(0x7f88081315a0 legacy=0x7f880812f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:43.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.188+0000 7f880fdce640 1 -- 172.21.15.189:0/2764466713 shutdown_connections 2024-09-16T14:45:43.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.188+0000 7f880fdce640 1 -- 172.21.15.189:0/2764466713 >> 172.21.15.189:0/2764466713 conn(0x7f88080a3320 msgr2=0x7f88080a1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:43.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.188+0000 7f880fdce640 1 -- 172.21.15.189:0/2764466713 shutdown_connections 2024-09-16T14:45:43.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.188+0000 7f880fdce640 1 -- 172.21.15.189:0/2764466713 wait complete. 2024-09-16T14:45:43.204 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rm bar 2024-09-16T14:45:43.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7aa4bc5640 10 monclient: get_monmap_and_config 2024-09-16T14:45:43.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7aa4bc5640 10 monclient: build_initial_monmap 2024-09-16T14:45:43.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7aa4bc5640 10 monclient: monmap: 2024-09-16T14:45:43.320 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:43.320 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:43.320 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:43.320156+0000 2024-09-16T14:45:43.320 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:43.320156+0000 2024-09-16T14:45:43.321 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:43.321 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:43.321 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:43.321 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:43.321 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:43.321 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:43.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7aa4bc5640 1 Processor -- start 2024-09-16T14:45:43.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7aa4bc5640 1 -- start start 2024-09-16T14:45:43.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7aa4bc5640 10 monclient: init 2024-09-16T14:45:43.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7aa4bc5640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:43.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7aa4bc5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:43.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7aa4bc5640 10 monclient(hunting): picked mon.noname-a con 0x7f7aa009df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:43.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7aa4bc5640 10 monclient(hunting): picked mon.noname-b con 0x7f7aa009e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:43.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7aa4bc5640 10 monclient(hunting): picked mon.noname-c con 0x7f7aa009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:43.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7aa4bc5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa009ee20 con 0x7f7aa009df10 2024-09-16T14:45:43.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7aa4bc5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa009f100 con 0x7f7aa009e740 2024-09-16T14:45:43.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7aa4bc5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa009f3e0 con 0x7f7aa009e2f0 2024-09-16T14:45:43.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7aa4bc5640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:43.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7aa4bc5640 10 monclient(hunting): authenticate will time out at 1125.435687s 2024-09-16T14:45:43.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7a9ed9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7aa009e740 0x7f7aa013b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50276/0 (socket says 172.21.15.189:50276) 2024-09-16T14:45:43.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7a9ed9d640 1 -- 172.21.15.189:0/4188936520 learned_addr learned my addr 172.21.15.189:0/4188936520 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:43.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.316+0000 7f7a9e59c640 1 -- 172.21.15.189:0/4188936520 >> v1:172.21.15.26:6789/0 conn(0x7f7aa009df10 legacy=0x7f7aa009d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:43.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 1 -- 172.21.15.189:0/4188936520 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984994032 0 0) 0x7f7aa009f100 con 0x7f7aa009e740 2024-09-16T14:45:43.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 10 monclient(hunting): _negotiate 2024-09-16T14:45:43.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:43.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:43.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 10 monclient(hunting): my global_id is 94252 2024-09-16T14:45:43.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 1 -- 172.21.15.189:0/4188936520 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a8c003530 con 0x7f7aa009e740 2024-09-16T14:45:43.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 1 -- 172.21.15.189:0/4188936520 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2355606097 0 0) 0x7f7a8c003530 con 0x7f7aa009e740 2024-09-16T14:45:43.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 1 -- 172.21.15.189:0/4188936520 >> v1:172.21.15.26:6790/0 conn(0x7f7aa009e2f0 legacy=0x7f7aa01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 1 -- 172.21.15.189:0/4188936520 >> v1:172.21.15.26:6789/0 conn(0x7f7aa009df10 legacy=0x7f7aa009d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 10 monclient: _finish_hunting 0 2024-09-16T14:45:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 1 monclient: found mon.noname-c 2024-09-16T14:45:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 1 -- 172.21.15.189:0/4188936520 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa009c690 con 0x7f7aa009e740 2024-09-16T14:45:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 10 monclient: _finish_auth 0 2024-09-16T14:45:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 10 monclient: _check_auth_tickets 2024-09-16T14:45:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 1 -- 172.21.15.189:0/4188936520 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a980019a0 con 0x7f7aa009e740 2024-09-16T14:45:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 5 monclient: authenticate success, global_id 94252 2024-09-16T14:45:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:43.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:43.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:43.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 10 monclient: dump: 2024-09-16T14:45:43.331 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:43.331 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:43.331 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:43.331 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:43.331 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:43.331 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:43.331 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:43.331 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:43.331 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:43.331 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:43.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:43.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 1 -- 172.21.15.189:0/4188936520 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a98001bc0 con 0x7f7aa009e740 2024-09-16T14:45:43.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:43.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 1 -- 172.21.15.189:0/4188936520 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a98001fd0 con 0x7f7aa009e740 2024-09-16T14:45:43.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:43.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:43.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9d59a640 10 monclient: dump: 2024-09-16T14:45:43.332 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:43.332 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:43.332 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:43.332 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:43.333 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:43.333 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:43.333 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:43.333 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:43.333 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:43.333 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:43.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:43.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 4 set_mon_vals no callback set 2024-09-16T14:45:43.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient: shutdown 2024-09-16T14:45:43.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:43.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 1 -- 172.21.15.189:0/4188936520 >> v1:172.21.15.189:6789/0 conn(0x7f7aa009e740 legacy=0x7f7aa013b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:43.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 1 -- 172.21.15.189:0/4188936520 shutdown_connections 2024-09-16T14:45:43.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 1 -- 172.21.15.189:0/4188936520 >> 172.21.15.189:0/4188936520 conn(0x7f7aa00a3320 msgr2=0x7f7aa009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:43.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 1 -- 172.21.15.189:0/4188936520 shutdown_connections 2024-09-16T14:45:43.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 1 -- 172.21.15.189:0/4188936520 wait complete. 2024-09-16T14:45:43.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient: build_initial_monmap 2024-09-16T14:45:43.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient: monmap: 2024-09-16T14:45:43.334 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:43.334 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:43.334 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:43.334 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:43.334 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:43.334 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:43.334 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:43.334 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:43.334 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:43.335 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:43.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 1 Processor -- start 2024-09-16T14:45:43.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 1 -- start start 2024-09-16T14:45:43.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient: init 2024-09-16T14:45:43.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:43.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:45:43.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient(hunting): picked mon.noname-a con 0x7f7aa009df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:43.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient(hunting): picked mon.noname-b con 0x7f7aa009e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:43.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient(hunting): picked mon.noname-c con 0x7f7aa009e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:43.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa006f9f0 con 0x7f7aa009df10 2024-09-16T14:45:43.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa006cf90 con 0x7f7aa009e2f0 2024-09-16T14:45:43.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa006b380 con 0x7f7aa009e740 2024-09-16T14:45:43.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:43.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient(hunting): authenticate will time out at 1125.438488s 2024-09-16T14:45:43.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9dd9b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7aa009e2f0 0x7f7aa01ce990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50286/0 (socket says 172.21.15.189:50286) 2024-09-16T14:45:43.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9dd9b640 1 -- 172.21.15.189:0/1775367595 learned_addr learned my addr 172.21.15.189:0/1775367595 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:43.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9e59c640 1 -- 172.21.15.189:0/1775367595 >> v1:172.21.15.26:6789/0 conn(0x7f7aa009df10 legacy=0x7f7aa01c99b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:43.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 1 -- 172.21.15.189:0/1775367595 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259034108 0 0) 0x7f7aa006cf90 con 0x7f7aa009e2f0 2024-09-16T14:45:43.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:43.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:43.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:43.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 10 monclient(hunting): my global_id is 94255 2024-09-16T14:45:43.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 1 -- 172.21.15.189:0/1775367595 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a80003530 con 0x7f7aa009e2f0 2024-09-16T14:45:43.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 1 -- 172.21.15.189:0/1775367595 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2270312984 0 0) 0x7f7a80003530 con 0x7f7aa009e2f0 2024-09-16T14:45:43.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 1 -- 172.21.15.189:0/1775367595 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7aa006cf90 con 0x7f7aa009e2f0 2024-09-16T14:45:43.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 1 -- 172.21.15.189:0/1775367595 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a90001980 con 0x7f7aa009e2f0 2024-09-16T14:45:43.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:43.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:43.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 10 monclient(hunting): dump: 2024-09-16T14:45:43.337 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:43.337 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:43.337 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:43.337 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:43.337 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:43.337 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:43.337 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:43.337 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:43.337 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:43.338 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:43.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:43.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 1 -- 172.21.15.189:0/1775367595 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3586664146 0 0) 0x7f7aa006cf90 con 0x7f7aa009e2f0 2024-09-16T14:45:43.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 1 -- 172.21.15.189:0/1775367595 >> v1:172.21.15.26:6790/0 conn(0x7f7aa009e740 legacy=0x7f7aa01d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:43.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 1 -- 172.21.15.189:0/1775367595 >> v1:172.21.15.26:6789/0 conn(0x7f7aa009df10 legacy=0x7f7aa01c99b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:43.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:43.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 1 monclient: found mon.b 2024-09-16T14:45:43.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:43.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:43.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 1 -- 172.21.15.189:0/1775367595 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa01258b0 con 0x7f7aa009e2f0 2024-09-16T14:45:43.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:43.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:43.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:43.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 5 monclient: authenticate success, global_id 94255 2024-09-16T14:45:43.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient: _renew_subs 2024-09-16T14:45:43.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:43.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 1 -- 172.21.15.189:0/1775367595 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa0125b90 con 0x7f7aa009e2f0 2024-09-16T14:45:43.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient: _renew_subs 2024-09-16T14:45:43.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:43.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 1 -- 172.21.15.189:0/1775367595 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa013c4d0 con 0x7f7aa009e2f0 2024-09-16T14:45:43.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 1 -- 172.21.15.189:0/1775367595 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a90001d50 con 0x7f7aa009e2f0 2024-09-16T14:45:43.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:43.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 1 -- 172.21.15.189:0/1775367595 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a90003a70 con 0x7f7aa009e2f0 2024-09-16T14:45:43.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:43.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:43.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a9cd99640 4 set_mon_vals no callback set 2024-09-16T14:45:43.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7a96ffd640 10 monclient: dump: 2024-09-16T14:45:43.340 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:43.340 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:43.340 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:43.340 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:43.340 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:43.340 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:43.340 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:43.340 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:43.340 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:43.341 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:43.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:43.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:43.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:43.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.320+0000 7f7aa4bc5640 1 -- 172.21.15.189:0/1775367595 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a84005050 con 0x7f7aa009e2f0 2024-09-16T14:45:43.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.324+0000 7f7a96ffd640 1 -- 172.21.15.189:0/1775367595 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7a90003bd0 con 0x7f7aa009e2f0 2024-09-16T14:45:43.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.324+0000 7f7a96ffd640 1 -- 172.21.15.189:0/1775367595 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(60..60 src has 15..60) v4 ==== 4250+0+0 (unknown 1167516382 0 0) 0x7f7a90043530 con 0x7f7aa009e2f0 2024-09-16T14:45:43.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.324+0000 7f7a9e59c640 10 monclient: get_auth_request con 0x7f7a80036050 auth_method 0 2024-09-16T14:45:43.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.328+0000 7f7a96ffd640 1 -- 172.21.15.189:0/1775367595 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7a90014830 con 0x7f7aa009e2f0 2024-09-16T14:45:43.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.328+0000 7f7a96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:43.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.328+0000 7f7a96ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:43.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.484+0000 7f7aa4bc5640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T14:45:43.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.484+0000 7f7aa4bc5640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T14:45:43.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.484+0000 7f7aa4bc5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:43.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.484+0000 7f7aa4bc5640 1 -- 172.21.15.189:0/1775367595 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "bar"} v 0) v1 -- 0x7f7a84005300 con 0x7f7aa009e2f0 2024-09-16T14:45:43.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.692+0000 7f7a96ffd640 1 -- 172.21.15.189:0/1775367595 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 v61) v1 ==== 80+0+0 (unknown 2655826462 0 0) 0x7f7a90012ca0 con 0x7f7aa009e2f0 2024-09-16T14:45:43.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.692+0000 7f7a96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T14:45:43.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.692+0000 7f7a96ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:43.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.784+0000 7f7aa4bc5640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T14:45:43.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.784+0000 7f7aa4bc5640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T14:45:43.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.784+0000 7f7aa4bc5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:43.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.784+0000 7f7aa4bc5640 1 -- 172.21.15.189:0/1775367595 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "bar"} v 0) v1 -- 0x7f7a84005850 con 0x7f7aa009e2f0 2024-09-16T14:45:43.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.788+0000 7f7a96ffd640 1 -- 172.21.15.189:0/1775367595 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v61) v1 ==== 103+0+0 (unknown 1563074952 0 0) 0x7f7a90014600 con 0x7f7aa009e2f0 2024-09-16T14:45:43.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.788+0000 7f7a96ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T14:45:43.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.788+0000 7f7a96ffd640 10 monclient: _finish_command 3 = system:0 rule bar does not exist 2024-09-16T14:45:43.791 INFO:tasks.workunit.client.0.smithi189.stderr:rule bar does not exist 2024-09-16T14:45:43.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.788+0000 7f7aa4bc5640 1 -- 172.21.15.189:0/1775367595 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7a80036050 legacy=0x7f7a80038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:43.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.788+0000 7f7aa4bc5640 10 monclient: shutdown 2024-09-16T14:45:43.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.788+0000 7f7aa4bc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:43.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.788+0000 7f7aa4bc5640 1 -- 172.21.15.189:0/1775367595 >> v1:172.21.15.189:6789/0 conn(0x7f7aa009e2f0 legacy=0x7f7aa01ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:43.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.788+0000 7f7aa4bc5640 1 -- 172.21.15.189:0/1775367595 shutdown_connections 2024-09-16T14:45:43.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.788+0000 7f7aa4bc5640 1 -- 172.21.15.189:0/1775367595 >> 172.21.15.189:0/1775367595 conn(0x7f7aa00a3320 msgr2=0x7f7aa00a1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:43.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.788+0000 7f7aa4bc5640 1 -- 172.21.15.189:0/1775367595 shutdown_connections 2024-09-16T14:45:43.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.788+0000 7f7aa4bc5640 1 -- 172.21.15.189:0/1775367595 wait complete. 2024-09-16T14:45:43.806 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create pinning_pool 1 2024-09-16T14:45:43.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.920+0000 7f4b245ca640 10 monclient: get_monmap_and_config 2024-09-16T14:45:43.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.920+0000 7f4b245ca640 10 monclient: build_initial_monmap 2024-09-16T14:45:43.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.920+0000 7f4b245ca640 10 monclient: monmap: 2024-09-16T14:45:43.924 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:43.925 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:43.925 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:43.924360+0000 2024-09-16T14:45:43.925 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:43.924360+0000 2024-09-16T14:45:43.925 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:43.925 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:43.925 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:43.925 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:43.925 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:43.925 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:43.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.920+0000 7f4b245ca640 1 Processor -- start 2024-09-16T14:45:43.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 1 -- start start 2024-09-16T14:45:43.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient: init 2024-09-16T14:45:43.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:43.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:45:43.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient(hunting): picked mon.noname-a con 0x7f4b1c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:43.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient(hunting): picked mon.noname-c con 0x7f4b1c09ff00 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:43.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient(hunting): picked mon.noname-b con 0x7f4b1c0a0310 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:43.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b1c09b380 con 0x7f4b1c099a70 2024-09-16T14:45:43.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b1c099e50 con 0x7f4b1c09ff00 2024-09-16T14:45:43.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b1c09f2e0 con 0x7f4b1c0a0310 2024-09-16T14:45:43.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:43.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient(hunting): authenticate will time out at 1126.040809s 2024-09-16T14:45:43.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b22366640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f4b1c099a70 legacy=0x7f4b1c09bd30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:43.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21b65640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4b1c09ff00 0x7f4b1c1365f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50296/0 (socket says 172.21.15.189:50296) 2024-09-16T14:45:43.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21b65640 1 -- 172.21.15.189:0/3468636146 learned_addr learned my addr 172.21.15.189:0/3468636146 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:43.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 1 -- 172.21.15.189:0/3468636146 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396552884 0 0) 0x7f4b1c099e50 con 0x7f4b1c09ff00 2024-09-16T14:45:43.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 10 monclient(hunting): _negotiate 2024-09-16T14:45:43.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:43.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:43.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 10 monclient(hunting): my global_id is 94258 2024-09-16T14:45:43.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 1 -- 172.21.15.189:0/3468636146 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b08003530 con 0x7f4b1c09ff00 2024-09-16T14:45:43.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 1 -- 172.21.15.189:0/3468636146 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3094012555 0 0) 0x7f4b08003530 con 0x7f4b1c09ff00 2024-09-16T14:45:43.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 1 -- 172.21.15.189:0/3468636146 >> v1:172.21.15.26:6790/0 conn(0x7f4b1c0a0310 legacy=0x7f4b1c139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:43.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 1 -- 172.21.15.189:0/3468636146 >> v1:172.21.15.26:6789/0 conn(0x7f4b1c099a70 legacy=0x7f4b1c09bd30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:43.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 10 monclient: _finish_hunting 0 2024-09-16T14:45:43.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 1 monclient: found mon.noname-c 2024-09-16T14:45:43.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:43.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:43.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 1 -- 172.21.15.189:0/3468636146 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1c09b070 con 0x7f4b1c09ff00 2024-09-16T14:45:43.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 10 monclient: _finish_auth 0 2024-09-16T14:45:43.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 10 monclient: _check_auth_tickets 2024-09-16T14:45:43.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:43.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 1 -- 172.21.15.189:0/3468636146 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4b0c001620 con 0x7f4b1c09ff00 2024-09-16T14:45:43.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 5 monclient: authenticate success, global_id 94258 2024-09-16T14:45:43.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:43.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:43.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:43.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 10 monclient: dump: 2024-09-16T14:45:43.932 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:43.932 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:43.932 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:43.932 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:43.932 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:43.932 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:43.932 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:43.932 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:43.932 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:43.932 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:43.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:43.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 1 -- 172.21.15.189:0/3468636146 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b0c001940 con 0x7f4b1c09ff00 2024-09-16T14:45:43.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:43.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 1 -- 172.21.15.189:0/3468636146 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4b0c001d10 con 0x7f4b1c09ff00 2024-09-16T14:45:43.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:43.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 4 set_mon_vals no callback set 2024-09-16T14:45:43.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:43.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:43.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b21364640 10 monclient: dump: 2024-09-16T14:45:43.933 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:43.933 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:43.933 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:43.933 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:43.933 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:43.933 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:43.934 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:43.934 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:43.934 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:43.934 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:43.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient: shutdown 2024-09-16T14:45:43.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:43.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 1 -- 172.21.15.189:0/3468636146 >> v1:172.21.15.189:6789/0 conn(0x7f4b1c09ff00 legacy=0x7f4b1c1365f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:43.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 1 -- 172.21.15.189:0/3468636146 shutdown_connections 2024-09-16T14:45:43.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 1 -- 172.21.15.189:0/3468636146 >> 172.21.15.189:0/3468636146 conn(0x7f4b1c0a3320 msgr2=0x7f4b1c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:43.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 1 -- 172.21.15.189:0/3468636146 shutdown_connections 2024-09-16T14:45:43.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 1 -- 172.21.15.189:0/3468636146 wait complete. 2024-09-16T14:45:43.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient: build_initial_monmap 2024-09-16T14:45:43.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient: monmap: 2024-09-16T14:45:43.935 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:43.935 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:43.935 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:43.935 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:43.935 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:43.935 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:43.935 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:43.935 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:43.935 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:43.935 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:43.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 1 Processor -- start 2024-09-16T14:45:43.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 1 -- start start 2024-09-16T14:45:43.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient: init 2024-09-16T14:45:43.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:43.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:43.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient(hunting): picked mon.noname-a con 0x7f4b1c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:43.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient(hunting): picked mon.noname-c con 0x7f4b1c09ff00 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:43.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient(hunting): picked mon.noname-b con 0x7f4b1c0a0310 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:43.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b1c06b380 con 0x7f4b1c099a70 2024-09-16T14:45:43.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b1c120840 con 0x7f4b1c0a0310 2024-09-16T14:45:43.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b1c0a2020 con 0x7f4b1c09ff00 2024-09-16T14:45:43.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:43.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b245ca640 10 monclient(hunting): authenticate will time out at 1126.043813s 2024-09-16T14:45:43.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b22366640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f4b1c099a70 legacy=0x7f4b1c11d990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:43.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b22b67640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4b1c0a0310 0x7f4b1c1d1b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50298/0 (socket says 172.21.15.189:50298) 2024-09-16T14:45:43.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.924+0000 7f4b22b67640 1 -- 172.21.15.189:0/290190374 learned_addr learned my addr 172.21.15.189:0/290190374 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:43.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2017075653 0 0) 0x7f4b1c120840 con 0x7f4b1c0a0310 2024-09-16T14:45:43.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:43.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:43.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:43.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient(hunting): my global_id is 94261 2024-09-16T14:45:43.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b00003530 con 0x7f4b1c0a0310 2024-09-16T14:45:43.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 210449447 0 0) 0x7f4b00003530 con 0x7f4b1c0a0310 2024-09-16T14:45:43.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b1c120840 con 0x7f4b1c0a0310 2024-09-16T14:45:43.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4b14001c40 con 0x7f4b1c0a0310 2024-09-16T14:45:43.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:43.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:43.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient(hunting): dump: 2024-09-16T14:45:43.941 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:43.941 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:43.941 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:43.941 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:43.941 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:43.941 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:43.941 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:43.941 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:43.942 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:43.942 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:43.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:43.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520434354 0 0) 0x7f4b1c0a2020 con 0x7f4b1c09ff00 2024-09-16T14:45:43.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:43.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:43.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:43.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient(hunting): my global_id is 5150 2024-09-16T14:45:43.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b00003530 con 0x7f4b1c09ff00 2024-09-16T14:45:43.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2586611638 0 0) 0x7f4b1c120840 con 0x7f4b1c0a0310 2024-09-16T14:45:43.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 >> v1:172.21.15.26:6790/0 conn(0x7f4b1c09ff00 legacy=0x7f4b1c11e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:43.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 >> v1:172.21.15.26:6789/0 conn(0x7f4b1c099a70 legacy=0x7f4b1c11d990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:43.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:43.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 1 monclient: found mon.b 2024-09-16T14:45:43.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:43.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:43.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1c0a1ad0 con 0x7f4b1c0a0310 2024-09-16T14:45:43.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:43.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:43.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:43.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b245ca640 5 monclient: authenticate success, global_id 94261 2024-09-16T14:45:43.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b245ca640 10 monclient: _renew_subs 2024-09-16T14:45:43.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b245ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:43.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b245ca640 1 -- 172.21.15.189:0/290190374 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1c11f430 con 0x7f4b1c0a0310 2024-09-16T14:45:43.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b245ca640 10 monclient: _renew_subs 2024-09-16T14:45:43.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b245ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:43.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b245ca640 1 -- 172.21.15.189:0/290190374 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1c11e980 con 0x7f4b1c0a0310 2024-09-16T14:45:43.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b14002010 con 0x7f4b1c0a0310 2024-09-16T14:45:43.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:43.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4b14003e70 con 0x7f4b1c0a0310 2024-09-16T14:45:43.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:43.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:43.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b20b63640 4 set_mon_vals no callback set 2024-09-16T14:45:43.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 10 monclient: dump: 2024-09-16T14:45:43.944 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:43.944 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:43.944 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:43.944 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:43.944 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:43.945 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:43.945 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:43.945 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:43.945 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:43.945 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:43.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4b14003fd0 con 0x7f4b1c0a0310 2024-09-16T14:45:43.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b245ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:43.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b245ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:43.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b245ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:43.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b245ca640 1 -- 172.21.15.189:0/290190374 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b04005050 con 0x7f4b1c0a0310 2024-09-16T14:45:43.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(61..61 src has 15..61) v4 ==== 4195+0+0 (unknown 2518494908 0 0) 0x7f4b140437f0 con 0x7f4b1c0a0310 2024-09-16T14:45:43.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.928+0000 7f4b22366640 10 monclient: get_auth_request con 0x7f4b000361d0 auth_method 0 2024-09-16T14:45:43.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.932+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4b14014b20 con 0x7f4b1c0a0310 2024-09-16T14:45:43.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.932+0000 7f4b12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:43.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:43.932+0000 7f4b12ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:44.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.088+0000 7f4b245ca640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T14:45:44.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.088+0000 7f4b245ca640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T14:45:44.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.088+0000 7f4b245ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:44.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.088+0000 7f4b245ca640 1 -- 172.21.15.189:0/290190374 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1} v 0) v1 -- 0x7f4b04005880 con 0x7f4b1c0a0310 2024-09-16T14:45:44.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.708+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}]=0 pool 'pinning_pool' created v62) v1 ==== 127+0+0 (unknown 3251069011 0 0) 0x7f4b14012f90 con 0x7f4b1c0a0310 2024-09-16T14:45:44.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.708+0000 7f4b12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T14:45:44.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.708+0000 7f4b12ffd640 10 monclient: _finish_command 2 = system:0 pool 'pinning_pool' created 2024-09-16T14:45:44.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.800+0000 7f4b245ca640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T14:45:44.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.800+0000 7f4b245ca640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T14:45:44.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.800+0000 7f4b245ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:44.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.800+0000 7f4b245ca640 1 -- 172.21.15.189:0/290190374 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1} v 0) v1 -- 0x7f4b04005340 con 0x7f4b1c0a0310 2024-09-16T14:45:44.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.804+0000 7f4b12ffd640 1 -- 172.21.15.189:0/290190374 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}]=0 pool 'pinning_pool' already exists v62) v1 ==== 134+0+0 (unknown 4256851748 0 0) 0x7f4b140148f0 con 0x7f4b1c0a0310 2024-09-16T14:45:44.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.804+0000 7f4b12ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T14:45:44.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.804+0000 7f4b12ffd640 10 monclient: _finish_command 3 = system:0 pool 'pinning_pool' already exists 2024-09-16T14:45:44.806 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'pinning_pool' already exists 2024-09-16T14:45:44.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.804+0000 7f4b245ca640 1 -- 172.21.15.189:0/290190374 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4b000361d0 legacy=0x7f4b00038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:44.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.804+0000 7f4b245ca640 10 monclient: shutdown 2024-09-16T14:45:44.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.804+0000 7f4b245ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:44.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.804+0000 7f4b245ca640 1 -- 172.21.15.189:0/290190374 >> v1:172.21.15.189:6789/0 conn(0x7f4b1c0a0310 legacy=0x7f4b1c1d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:44.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.804+0000 7f4b245ca640 1 -- 172.21.15.189:0/290190374 shutdown_connections 2024-09-16T14:45:44.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.804+0000 7f4b245ca640 1 -- 172.21.15.189:0/290190374 >> 172.21.15.189:0/290190374 conn(0x7f4b1c0a3320 msgr2=0x7f4b1c1b86f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:44.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.804+0000 7f4b245ca640 1 -- 172.21.15.189:0/290190374 shutdown_connections 2024-09-16T14:45:44.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.804+0000 7f4b245ca640 1 -- 172.21.15.189:0/290190374 wait complete. 2024-09-16T14:45:44.822 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush rule rm replicated_rule 2024-09-16T14:45:44.822 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:45:44.822 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rm replicated_rule 2024-09-16T14:45:44.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.936+0000 7fe1d1144640 10 monclient: get_monmap_and_config 2024-09-16T14:45:44.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.936+0000 7fe1d1144640 10 monclient: build_initial_monmap 2024-09-16T14:45:44.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.936+0000 7fe1d1144640 10 monclient: monmap: 2024-09-16T14:45:44.940 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:44.940 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:44.940 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:44.940148+0000 2024-09-16T14:45:44.940 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:44.940148+0000 2024-09-16T14:45:44.941 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:44.941 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:44.941 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:44.941 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:44.941 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:44.941 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:44.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.936+0000 7fe1d1144640 1 Processor -- start 2024-09-16T14:45:44.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.936+0000 7fe1d1144640 1 -- start start 2024-09-16T14:45:44.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.936+0000 7fe1d1144640 10 monclient: init 2024-09-16T14:45:44.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.936+0000 7fe1d1144640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:44.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.936+0000 7fe1d1144640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:44.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.936+0000 7fe1d1144640 10 monclient(hunting): picked mon.noname-c con 0x7fe1cc09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:44.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.936+0000 7fe1d1144640 10 monclient(hunting): picked mon.noname-b con 0x7fe1cc09e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:44.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.936+0000 7fe1d1144640 10 monclient(hunting): picked mon.noname-a con 0x7fe1cc09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:44.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.936+0000 7fe1d1144640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1cc09ee20 con 0x7fe1cc09e740 2024-09-16T14:45:44.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.936+0000 7fe1d1144640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1cc09f100 con 0x7fe1cc09df10 2024-09-16T14:45:44.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.936+0000 7fe1d1144640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1cc09f3e0 con 0x7fe1cc09e2f0 2024-09-16T14:45:44.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.936+0000 7fe1d1144640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:44.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.936+0000 7fe1d1144640 10 monclient(hunting): authenticate will time out at 1127.055905s 2024-09-16T14:45:44.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1cb59e640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fe1cc09e740 legacy=0x7fe1cc13b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:44.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1cad9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe1cc09df10 0x7fe1cc09d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50310/0 (socket says 172.21.15.189:50310) 2024-09-16T14:45:44.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1cad9d640 1 -- 172.21.15.189:0/2925135155 learned_addr learned my addr 172.21.15.189:0/2925135155 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:44.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 1 -- 172.21.15.189:0/2925135155 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634947383 0 0) 0x7fe1cc09f100 con 0x7fe1cc09df10 2024-09-16T14:45:44.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:45:44.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:44.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:44.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 10 monclient(hunting): my global_id is 94264 2024-09-16T14:45:44.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 1 -- 172.21.15.189:0/2925135155 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1b8003530 con 0x7fe1cc09df10 2024-09-16T14:45:44.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 1 -- 172.21.15.189:0/2925135155 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1536403137 0 0) 0x7fe1b8003530 con 0x7fe1cc09df10 2024-09-16T14:45:44.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 1 -- 172.21.15.189:0/2925135155 >> v1:172.21.15.26:6790/0 conn(0x7fe1cc09e2f0 legacy=0x7fe1cc1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:44.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 1 -- 172.21.15.189:0/2925135155 >> v1:172.21.15.26:6789/0 conn(0x7fe1cc09e740 legacy=0x7fe1cc13b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:44.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:45:44.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 1 monclient: found mon.noname-c 2024-09-16T14:45:44.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:44.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:44.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 1 -- 172.21.15.189:0/2925135155 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1cc09c690 con 0x7fe1cc09df10 2024-09-16T14:45:44.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 10 monclient: _finish_auth 0 2024-09-16T14:45:44.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:45:44.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:44.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 5 monclient: authenticate success, global_id 94264 2024-09-16T14:45:44.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:44.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 1 -- 172.21.15.189:0/2925135155 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe1b4001620 con 0x7fe1cc09df10 2024-09-16T14:45:44.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:44.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:44.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 10 monclient: dump: 2024-09-16T14:45:44.950 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:44.950 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:44.950 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:44.951 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:44.951 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:44.951 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:44.951 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:44.951 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:44.951 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:44.951 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:44.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:44.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 1 -- 172.21.15.189:0/2925135155 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1b4001940 con 0x7fe1cc09df10 2024-09-16T14:45:44.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:44.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:44.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 1 -- 172.21.15.189:0/2925135155 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe1b4001d10 con 0x7fe1cc09df10 2024-09-16T14:45:44.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 4 set_mon_vals no callback set 2024-09-16T14:45:44.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:44.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:44.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c9d9b640 10 monclient: dump: 2024-09-16T14:45:44.952 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:44.952 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:44.952 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:44.952 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:44.952 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:44.952 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:44.952 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:44.952 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:44.952 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:44.952 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:44.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 10 monclient: shutdown 2024-09-16T14:45:44.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:44.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 1 -- 172.21.15.189:0/2925135155 >> v1:172.21.15.189:6789/0 conn(0x7fe1cc09df10 legacy=0x7fe1cc09d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:44.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 1 -- 172.21.15.189:0/2925135155 shutdown_connections 2024-09-16T14:45:44.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 1 -- 172.21.15.189:0/2925135155 >> 172.21.15.189:0/2925135155 conn(0x7fe1cc0a3320 msgr2=0x7fe1cc09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:44.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 1 -- 172.21.15.189:0/2925135155 shutdown_connections 2024-09-16T14:45:44.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 1 -- 172.21.15.189:0/2925135155 wait complete. 2024-09-16T14:45:44.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 10 monclient: build_initial_monmap 2024-09-16T14:45:44.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 10 monclient: monmap: 2024-09-16T14:45:44.953 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:44.953 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:44.953 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:44.953 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:44.953 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:44.954 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:44.954 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:44.954 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:44.954 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:44.954 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:44.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 1 Processor -- start 2024-09-16T14:45:44.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 1 -- start start 2024-09-16T14:45:44.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 10 monclient: init 2024-09-16T14:45:44.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:44.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:44.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 10 monclient(hunting): picked mon.noname-a con 0x7fe1cc09df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:44.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 10 monclient(hunting): picked mon.noname-c con 0x7fe1cc09e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:44.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 10 monclient(hunting): picked mon.noname-b con 0x7fe1cc09e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:44.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1cc070610 con 0x7fe1cc09df10 2024-09-16T14:45:44.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1cc09aa30 con 0x7fe1cc09e740 2024-09-16T14:45:44.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1cc06b380 con 0x7fe1cc09e2f0 2024-09-16T14:45:44.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:44.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 10 monclient(hunting): authenticate will time out at 1127.059078s 2024-09-16T14:45:44.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1cad9d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fe1cc09df10 legacy=0x7fe1cc1c99b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:44.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1cb59e640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe1cc09e740 0x7fe1cc1d20c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50322/0 (socket says 172.21.15.189:50322) 2024-09-16T14:45:44.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1cb59e640 1 -- 172.21.15.189:0/2031809272 learned_addr learned my addr 172.21.15.189:0/2031809272 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:44.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 1 -- 172.21.15.189:0/2031809272 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402201547 0 0) 0x7fe1cc09aa30 con 0x7fe1cc09e740 2024-09-16T14:45:44.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:45:44.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:44.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:44.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 10 monclient(hunting): my global_id is 94267 2024-09-16T14:45:44.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 1 -- 172.21.15.189:0/2031809272 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1ac003530 con 0x7fe1cc09e740 2024-09-16T14:45:44.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 1 -- 172.21.15.189:0/2031809272 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1376584258 0 0) 0x7fe1ac003530 con 0x7fe1cc09e740 2024-09-16T14:45:44.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 1 -- 172.21.15.189:0/2031809272 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1cc09aa30 con 0x7fe1cc09e740 2024-09-16T14:45:44.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 1 -- 172.21.15.189:0/2031809272 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe1c4004400 con 0x7fe1cc09e740 2024-09-16T14:45:44.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:44.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:44.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 10 monclient(hunting): dump: 2024-09-16T14:45:44.956 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:44.956 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:44.956 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:44.956 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:44.956 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:44.956 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:44.957 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:44.957 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:44.957 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:44.957 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:44.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:44.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 1 -- 172.21.15.189:0/2031809272 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2478766360 0 0) 0x7fe1cc09aa30 con 0x7fe1cc09e740 2024-09-16T14:45:44.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 1 -- 172.21.15.189:0/2031809272 >> v1:172.21.15.26:6790/0 conn(0x7fe1cc09e2f0 legacy=0x7fe1cc1ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:44.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 1 -- 172.21.15.189:0/2031809272 >> v1:172.21.15.26:6789/0 conn(0x7fe1cc09df10 legacy=0x7fe1cc1c99b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:44.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:45:44.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 1 monclient: found mon.b 2024-09-16T14:45:44.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:44.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:44.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 1 -- 172.21.15.189:0/2031809272 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1cc06eb50 con 0x7fe1cc09e740 2024-09-16T14:45:44.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 10 monclient: _finish_auth 0 2024-09-16T14:45:44.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:44.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:44.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 5 monclient: authenticate success, global_id 94267 2024-09-16T14:45:44.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 10 monclient: _renew_subs 2024-09-16T14:45:44.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:44.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 1 -- 172.21.15.189:0/2031809272 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1cc06f4a0 con 0x7fe1cc09e740 2024-09-16T14:45:44.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 10 monclient: _renew_subs 2024-09-16T14:45:44.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:44.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1d1144640 1 -- 172.21.15.189:0/2031809272 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1cc06f940 con 0x7fe1cc09e740 2024-09-16T14:45:44.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.940+0000 7fe1c37fe640 1 -- 172.21.15.189:0/2031809272 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1c4004620 con 0x7fe1cc09e740 2024-09-16T14:45:44.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.944+0000 7fe1c37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:44.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.944+0000 7fe1c37fe640 1 -- 172.21.15.189:0/2031809272 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe1c40049b0 con 0x7fe1cc09e740 2024-09-16T14:45:44.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.944+0000 7fe1c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:44.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.944+0000 7fe1c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:44.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.944+0000 7fe1c8d99640 4 set_mon_vals no callback set 2024-09-16T14:45:44.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.944+0000 7fe1c37fe640 10 monclient: dump: 2024-09-16T14:45:44.959 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:44.959 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:44.960 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:44.960 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:44.960 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:44.960 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:44.960 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:44.960 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:44.960 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:44.960 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:44.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.944+0000 7fe1c37fe640 1 -- 172.21.15.189:0/2031809272 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe1c4004b10 con 0x7fe1cc09e740 2024-09-16T14:45:44.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.944+0000 7fe1d1144640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:44.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.944+0000 7fe1d1144640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:44.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.944+0000 7fe1d1144640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:44.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.944+0000 7fe1d1144640 1 -- 172.21.15.189:0/2031809272 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b0005050 con 0x7fe1cc09e740 2024-09-16T14:45:44.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.944+0000 7fe1c37fe640 1 -- 172.21.15.189:0/2031809272 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(62..62 src has 15..62) v4 ==== 4546+0+0 (unknown 2351852164 0 0) 0x7fe1c4043780 con 0x7fe1cc09e740 2024-09-16T14:45:44.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.944+0000 7fe1cad9d640 10 monclient: get_auth_request con 0x7fe1ac0360a0 auth_method 0 2024-09-16T14:45:44.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.948+0000 7fe1c37fe640 1 -- 172.21.15.189:0/2031809272 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe1c4005310 con 0x7fe1cc09e740 2024-09-16T14:45:44.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.948+0000 7fe1c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:44.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:44.948+0000 7fe1c37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:45.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.108+0000 7fe1d1144640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-16T14:45:45.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.108+0000 7fe1d1144640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-16T14:45:45.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.108+0000 7fe1d1144640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:45.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.108+0000 7fe1d1144640 1 -- 172.21.15.189:0/2031809272 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "replicated_rule"} v 0) v1 -- 0x7fe1b0005880 con 0x7fe1cc09e740 2024-09-16T14:45:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.112+0000 7fe1c37fe640 1 -- 172.21.15.189:0/2031809272 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "replicated_rule"}]=-16 crush rule replicated_rule (0) is in use v62) v1 ==== 132+0+0 (unknown 3064424850 0 0) 0x7fe1c401cf40 con 0x7fe1cc09e740 2024-09-16T14:45:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.112+0000 7fe1c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-16T14:45:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.112+0000 7fe1c37fe640 10 monclient: _finish_command 2 = mon:16 crush rule replicated_rule (0) is in use 2024-09-16T14:45:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:Error EBUSY: crush rule replicated_rule (0) is in use 2024-09-16T14:45:45.120 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:45:45.120 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool rm pinning_pool pinning_pool --yes-i-really-really-mean-it 2024-09-16T14:45:45.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 10 monclient: get_monmap_and_config 2024-09-16T14:45:45.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 10 monclient: build_initial_monmap 2024-09-16T14:45:45.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 10 monclient: monmap: 2024-09-16T14:45:45.234 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:45.234 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:45.234 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:45.234088+0000 2024-09-16T14:45:45.234 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:45.234088+0000 2024-09-16T14:45:45.234 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:45.235 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:45.235 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:45.235 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:45.235 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:45.235 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:45.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 1 Processor -- start 2024-09-16T14:45:45.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 1 -- start start 2024-09-16T14:45:45.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 10 monclient: init 2024-09-16T14:45:45.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:45.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:45.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 10 monclient(hunting): picked mon.noname-c con 0x7f1a2c099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:45.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 10 monclient(hunting): picked mon.noname-b con 0x7f1a2c09c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:45.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 10 monclient(hunting): picked mon.noname-a con 0x7f1a2c09e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:45.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c099e50 con 0x7f1a2c09e6d0 2024-09-16T14:45:45.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c09dbd0 con 0x7f1a2c099a70 2024-09-16T14:45:45.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c09ee20 con 0x7f1a2c09c550 2024-09-16T14:45:45.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:45.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 10 monclient(hunting): authenticate will time out at 1127.349706s 2024-09-16T14:45:45.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a321f9640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1a2c099a70 0x7f1a2c09dfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50326/0 (socket says 172.21.15.189:50326) 2024-09-16T14:45:45.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a329fa640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f1a2c09e6d0 legacy=0x7f1a2c13b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:45.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a321f9640 1 -- 172.21.15.189:0/3315913698 learned_addr learned my addr 172.21.15.189:0/3315913698 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:45.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 1 -- 172.21.15.189:0/3315913698 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495319283 0 0) 0x7f1a2c09dbd0 con 0x7f1a2c099a70 2024-09-16T14:45:45.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 10 monclient(hunting): _negotiate 2024-09-16T14:45:45.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:45.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:45.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 10 monclient(hunting): my global_id is 94270 2024-09-16T14:45:45.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 1 -- 172.21.15.189:0/3315913698 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a18003530 con 0x7f1a2c099a70 2024-09-16T14:45:45.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 1 -- 172.21.15.189:0/3315913698 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2393345904 0 0) 0x7f1a18003530 con 0x7f1a2c099a70 2024-09-16T14:45:45.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 1 -- 172.21.15.189:0/3315913698 >> v1:172.21.15.26:6790/0 conn(0x7f1a2c09c550 legacy=0x7f1a2c1379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:45.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 1 -- 172.21.15.189:0/3315913698 >> v1:172.21.15.26:6789/0 conn(0x7f1a2c09e6d0 legacy=0x7f1a2c13b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:45.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 10 monclient: _finish_hunting 0 2024-09-16T14:45:45.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 1 monclient: found mon.noname-c 2024-09-16T14:45:45.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:45.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:45.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 1 -- 172.21.15.189:0/3315913698 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c09f100 con 0x7f1a2c099a70 2024-09-16T14:45:45.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 10 monclient: _finish_auth 0 2024-09-16T14:45:45.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 10 monclient: _check_auth_tickets 2024-09-16T14:45:45.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:45.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 5 monclient: authenticate success, global_id 94270 2024-09-16T14:45:45.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:45.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 1 -- 172.21.15.189:0/3315913698 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1a1c001620 con 0x7f1a2c099a70 2024-09-16T14:45:45.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:45.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:45.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 10 monclient: dump: 2024-09-16T14:45:45.244 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:45.244 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:45.245 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:45.245 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:45.245 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:45.245 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:45.245 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:45.245 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:45.245 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:45.245 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:45.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:45.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 1 -- 172.21.15.189:0/3315913698 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a1c0018d0 con 0x7f1a2c099a70 2024-09-16T14:45:45.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:45.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 1 -- 172.21.15.189:0/3315913698 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1a1c001c60 con 0x7f1a2c099a70 2024-09-16T14:45:45.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:45.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 4 set_mon_vals no callback set 2024-09-16T14:45:45.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:45.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:45.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a311f7640 10 monclient: dump: 2024-09-16T14:45:45.246 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:45.246 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:45.246 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:45.246 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:45.246 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:45.246 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:45.246 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:45.246 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:45.247 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:45.247 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:45.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 10 monclient: shutdown 2024-09-16T14:45:45.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:45.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 1 -- 172.21.15.189:0/3315913698 >> v1:172.21.15.189:6789/0 conn(0x7f1a2c099a70 legacy=0x7f1a2c09dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:45.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 1 -- 172.21.15.189:0/3315913698 shutdown_connections 2024-09-16T14:45:45.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 1 -- 172.21.15.189:0/3315913698 >> 172.21.15.189:0/3315913698 conn(0x7f1a2c0a3380 msgr2=0x7f1a2c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:45.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 1 -- 172.21.15.189:0/3315913698 shutdown_connections 2024-09-16T14:45:45.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 1 -- 172.21.15.189:0/3315913698 wait complete. 2024-09-16T14:45:45.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 10 monclient: build_initial_monmap 2024-09-16T14:45:45.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.232+0000 7f1a3445d640 10 monclient: monmap: 2024-09-16T14:45:45.247 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:45.247 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:45.247 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:45.248 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:45.248 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:45.248 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:45.248 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:45.248 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:45.248 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:45.248 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:45.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 1 Processor -- start 2024-09-16T14:45:45.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 1 -- start start 2024-09-16T14:45:45.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 10 monclient: init 2024-09-16T14:45:45.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:45.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 10 monclient(hunting): picked mon.noname-c con 0x7f1a2c099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 10 monclient(hunting): picked mon.noname-b con 0x7f1a2c09c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 10 monclient(hunting): picked mon.noname-a con 0x7f1a2c09e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c09cf20 con 0x7f1a2c09e6d0 2024-09-16T14:45:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c06b320 con 0x7f1a2c09c550 2024-09-16T14:45:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c13b740 con 0x7f1a2c099a70 2024-09-16T14:45:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 10 monclient(hunting): authenticate will time out at 1127.352729s 2024-09-16T14:45:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a329fa640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f1a2c09e6d0 legacy=0x7f1a2c1d1720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a319f8640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1a2c09c550 0x7f1a2c09a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50328/0 (socket says 172.21.15.189:50328) 2024-09-16T14:45:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a319f8640 1 -- 172.21.15.189:0/125689978 learned_addr learned my addr 172.21.15.189:0/125689978 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 1 -- 172.21.15.189:0/125689978 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027689969 0 0) 0x7f1a2c06b320 con 0x7f1a2c09c550 2024-09-16T14:45:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:45.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:45.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:45.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 10 monclient(hunting): my global_id is 94273 2024-09-16T14:45:45.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 1 -- 172.21.15.189:0/125689978 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a10003530 con 0x7f1a2c09c550 2024-09-16T14:45:45.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 1 -- 172.21.15.189:0/125689978 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 10922716 0 0) 0x7f1a10003530 con 0x7f1a2c09c550 2024-09-16T14:45:45.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 1 -- 172.21.15.189:0/125689978 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a2c06b320 con 0x7f1a2c09c550 2024-09-16T14:45:45.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 1 -- 172.21.15.189:0/125689978 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1a28001880 con 0x7f1a2c09c550 2024-09-16T14:45:45.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:45.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:45.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 10 monclient(hunting): dump: 2024-09-16T14:45:45.250 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:45.250 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:45.250 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:45.251 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:45.251 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:45.251 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:45.251 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:45.251 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:45.251 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:45.251 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:45.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:45.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 1 -- 172.21.15.189:0/125689978 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2913601004 0 0) 0x7f1a2c06b320 con 0x7f1a2c09c550 2024-09-16T14:45:45.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 1 -- 172.21.15.189:0/125689978 >> v1:172.21.15.26:6790/0 conn(0x7f1a2c099a70 legacy=0x7f1a2c09a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:45.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 1 -- 172.21.15.189:0/125689978 >> v1:172.21.15.26:6789/0 conn(0x7f1a2c09e6d0 legacy=0x7f1a2c1d1720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:45.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:45.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 1 monclient: found mon.b 2024-09-16T14:45:45.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:45.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:45.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 1 -- 172.21.15.189:0/125689978 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c09b090 con 0x7f1a2c09c550 2024-09-16T14:45:45.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:45.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:45.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:45.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 5 monclient: authenticate success, global_id 94273 2024-09-16T14:45:45.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 10 monclient: _renew_subs 2024-09-16T14:45:45.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:45.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 1 -- 172.21.15.189:0/125689978 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2c13aa60 con 0x7f1a2c09c550 2024-09-16T14:45:45.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 10 monclient: _renew_subs 2024-09-16T14:45:45.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:45.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 1 -- 172.21.15.189:0/125689978 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2c09bd80 con 0x7f1a2c09c550 2024-09-16T14:45:45.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 1 -- 172.21.15.189:0/125689978 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a28001c50 con 0x7f1a2c09c550 2024-09-16T14:45:45.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:45.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 1 -- 172.21.15.189:0/125689978 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1a28003a70 con 0x7f1a2c09c550 2024-09-16T14:45:45.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:45.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:45.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 10 monclient: dump: 2024-09-16T14:45:45.253 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:45.253 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:45.253 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:45.253 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:45.253 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:45.254 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:45.254 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:45.254 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:45.254 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:45.254 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:45.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a309f6640 4 set_mon_vals no callback set 2024-09-16T14:45:45.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 1 -- 172.21.15.189:0/125689978 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1a28003bd0 con 0x7f1a2c09c550 2024-09-16T14:45:45.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:45.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:45.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:45.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a3445d640 1 -- 172.21.15.189:0/125689978 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a14005050 con 0x7f1a2c09c550 2024-09-16T14:45:45.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a22ffd640 1 -- 172.21.15.189:0/125689978 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(62..62 src has 15..62) v4 ==== 4546+0+0 (unknown 2351852164 0 0) 0x7f1a280434f0 con 0x7f1a2c09c550 2024-09-16T14:45:45.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.236+0000 7f1a321f9640 10 monclient: get_auth_request con 0x7f1a10035fe0 auth_method 0 2024-09-16T14:45:45.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.240+0000 7f1a22ffd640 1 -- 172.21.15.189:0/125689978 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1a280146c0 con 0x7f1a2c09c550 2024-09-16T14:45:45.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.240+0000 7f1a22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:45.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.240+0000 7f1a22ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:45.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.400+0000 7f1a3445d640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T14:45:45.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.400+0000 7f1a3445d640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T14:45:45.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.400+0000 7f1a3445d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:45.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.400+0000 7f1a3445d640 1 -- 172.21.15.189:0/125689978 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1a14005300 con 0x7f1a2c09c550 2024-09-16T14:45:45.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.712+0000 7f1a22ffd640 1 -- 172.21.15.189:0/125689978 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}]=0 pool 'pinning_pool' removed v63) v1 ==== 172+0+0 (unknown 4052887156 0 0) 0x7f1a28012b30 con 0x7f1a2c09c550 2024-09-16T14:45:45.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.712+0000 7f1a22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T14:45:45.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.712+0000 7f1a22ffd640 10 monclient: _finish_command 2 = system:0 pool 'pinning_pool' removed 2024-09-16T14:45:45.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.808+0000 7f1a3445d640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T14:45:45.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.808+0000 7f1a3445d640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T14:45:45.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.808+0000 7f1a3445d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:45.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.808+0000 7f1a3445d640 1 -- 172.21.15.189:0/125689978 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1a140010a0 con 0x7f1a2c09c550 2024-09-16T14:45:45.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.812+0000 7f1a22ffd640 1 -- 172.21.15.189:0/125689978 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}]=0 pool 'pinning_pool' does not exist v63) v1 ==== 179+0+0 (unknown 3391167355 0 0) 0x7f1a28012b30 con 0x7f1a2c09c550 2024-09-16T14:45:45.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.812+0000 7f1a22ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T14:45:45.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.812+0000 7f1a22ffd640 10 monclient: _finish_command 3 = system:0 pool 'pinning_pool' does not exist 2024-09-16T14:45:45.815 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'pinning_pool' does not exist 2024-09-16T14:45:45.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.812+0000 7f1a3445d640 1 -- 172.21.15.189:0/125689978 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1a10035fe0 legacy=0x7f1a100384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:45.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.812+0000 7f1a3445d640 10 monclient: shutdown 2024-09-16T14:45:45.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.812+0000 7f1a3445d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:45.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.812+0000 7f1a3445d640 1 -- 172.21.15.189:0/125689978 >> v1:172.21.15.189:6789/0 conn(0x7f1a2c09c550 legacy=0x7f1a2c09a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:45.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.812+0000 7f1a3445d640 1 -- 172.21.15.189:0/125689978 shutdown_connections 2024-09-16T14:45:45.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.812+0000 7f1a3445d640 1 -- 172.21.15.189:0/125689978 >> 172.21.15.189:0/125689978 conn(0x7f1a2c0a3380 msgr2=0x7f1a2c0a0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:45.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.812+0000 7f1a3445d640 1 -- 172.21.15.189:0/125689978 shutdown_connections 2024-09-16T14:45:45.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.812+0000 7f1a3445d640 1 -- 172.21.15.189:0/125689978 wait complete. 2024-09-16T14:45:45.831 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush add-bucket foo osd 2024-09-16T14:45:45.831 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:45:45.831 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket foo osd 2024-09-16T14:45:45.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.940+0000 7f034536a640 10 monclient: get_monmap_and_config 2024-09-16T14:45:45.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.940+0000 7f034536a640 10 monclient: build_initial_monmap 2024-09-16T14:45:45.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.940+0000 7f034536a640 10 monclient: monmap: 2024-09-16T14:45:45.945 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:45.946 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:45.946 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:45.945452+0000 2024-09-16T14:45:45.946 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:45.945452+0000 2024-09-16T14:45:45.946 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:45.946 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:45.946 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:45.946 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:45.946 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:45.946 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:45.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 1 Processor -- start 2024-09-16T14:45:45.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 1 -- start start 2024-09-16T14:45:45.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient: init 2024-09-16T14:45:45.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:45.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:45:45.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient(hunting): picked mon.noname-b con 0x7f0340099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:45.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient(hunting): picked mon.noname-a con 0x7f03401334d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:45.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient(hunting): picked mon.noname-c con 0x7f0340136fe0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:45.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034009b350 con 0x7f03401334d0 2024-09-16T14:45:45.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034009f2e0 con 0x7f0340136fe0 2024-09-16T14:45:45.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034009b070 con 0x7f0340099a70 2024-09-16T14:45:45.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:45.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient(hunting): authenticate will time out at 1128.061023s 2024-09-16T14:45:45.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033f7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0340136fe0 0x7f03401393d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50336/0 (socket says 172.21.15.189:50336) 2024-09-16T14:45:45.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033f7fe640 1 -- 172.21.15.189:0/3880325255 learned_addr learned my addr 172.21.15.189:0/3880325255 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:45.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f0337fff640 1 -- 172.21.15.189:0/3880325255 >> v1:172.21.15.26:6789/0 conn(0x7f03401334d0 legacy=0x7f03401358c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:45.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 1 -- 172.21.15.189:0/3880325255 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713779812 0 0) 0x7f034009f2e0 con 0x7f0340136fe0 2024-09-16T14:45:45.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:45.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:45.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:45.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 10 monclient(hunting): my global_id is 94276 2024-09-16T14:45:45.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 1 -- 172.21.15.189:0/3880325255 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f032c003450 con 0x7f0340136fe0 2024-09-16T14:45:45.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 1 -- 172.21.15.189:0/3880325255 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 145221417 0 0) 0x7f032c003450 con 0x7f0340136fe0 2024-09-16T14:45:45.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 1 -- 172.21.15.189:0/3880325255 >> v1:172.21.15.26:6790/0 conn(0x7f0340099a70 legacy=0x7f034009bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:45.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 1 -- 172.21.15.189:0/3880325255 >> v1:172.21.15.26:6789/0 conn(0x7f03401334d0 legacy=0x7f03401358c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:45.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:45:45.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 1 monclient: found mon.noname-c 2024-09-16T14:45:45.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:45.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:45.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 1 -- 172.21.15.189:0/3880325255 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034009b630 con 0x7f0340136fe0 2024-09-16T14:45:45.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:45:45.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:45:45.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:45.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 1 -- 172.21.15.189:0/3880325255 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f03380019a0 con 0x7f0340136fe0 2024-09-16T14:45:45.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:45.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:45.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 10 monclient: dump: 2024-09-16T14:45:45.954 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:45.955 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:45.955 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:45.955 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:45.955 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:45.955 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:45.955 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:45.955 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:45.955 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:45.955 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:45.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 5 monclient: authenticate success, global_id 94276 2024-09-16T14:45:45.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:45.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 1 -- 172.21.15.189:0/3880325255 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0338001c50 con 0x7f0340136fe0 2024-09-16T14:45:45.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:45.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:45.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 4 set_mon_vals no callback set 2024-09-16T14:45:45.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 1 -- 172.21.15.189:0/3880325255 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0338002020 con 0x7f0340136fe0 2024-09-16T14:45:45.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:45.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:45.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033e7fc640 10 monclient: dump: 2024-09-16T14:45:45.956 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:45.956 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:45.956 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:45.956 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:45.956 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:45.956 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:45.956 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:45.957 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:45.957 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:45.957 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:45.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient: shutdown 2024-09-16T14:45:45.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:45.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 1 -- 172.21.15.189:0/3880325255 >> v1:172.21.15.189:6789/0 conn(0x7f0340136fe0 legacy=0x7f03401393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:45.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 1 -- 172.21.15.189:0/3880325255 shutdown_connections 2024-09-16T14:45:45.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 1 -- 172.21.15.189:0/3880325255 >> 172.21.15.189:0/3880325255 conn(0x7f03400a3320 msgr2=0x7f034009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:45.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 1 -- 172.21.15.189:0/3880325255 shutdown_connections 2024-09-16T14:45:45.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 1 -- 172.21.15.189:0/3880325255 wait complete. 2024-09-16T14:45:45.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient: build_initial_monmap 2024-09-16T14:45:45.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient: monmap: 2024-09-16T14:45:45.957 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:45.958 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:45.958 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:45.958 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:45.958 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:45.958 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:45.958 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:45.958 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:45.958 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:45.958 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:45.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 1 Processor -- start 2024-09-16T14:45:45.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 1 -- start start 2024-09-16T14:45:45.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient: init 2024-09-16T14:45:45.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:45.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:45:45.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient(hunting): picked mon.noname-a con 0x7f0340099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:45.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient(hunting): picked mon.noname-b con 0x7f03401334d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:45.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient(hunting): picked mon.noname-c con 0x7f0340136fe0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:45.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034011d920 con 0x7f0340099a70 2024-09-16T14:45:45.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0340136320 con 0x7f03401334d0 2024-09-16T14:45:45.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03400a1b20 con 0x7f0340136fe0 2024-09-16T14:45:45.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:45.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f034536a640 10 monclient(hunting): authenticate will time out at 1128.063582s 2024-09-16T14:45:45.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f0337fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f03401334d0 0x7f03401ca3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50342/0 (socket says 172.21.15.189:50342) 2024-09-16T14:45:45.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f0337fff640 1 -- 172.21.15.189:0/459864143 learned_addr learned my addr 172.21.15.189:0/459864143 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:45.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f033effd640 1 -- 172.21.15.189:0/459864143 >> v1:172.21.15.26:6789/0 conn(0x7f0340099a70 legacy=0x7f03401c53c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:45.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f03377fe640 1 -- 172.21.15.189:0/459864143 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827777155 0 0) 0x7f0340136320 con 0x7f03401334d0 2024-09-16T14:45:45.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f03377fe640 10 monclient(hunting): _negotiate 2024-09-16T14:45:45.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f03377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:45.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f03377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:45.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f03377fe640 10 monclient(hunting): my global_id is 94279 2024-09-16T14:45:45.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.944+0000 7f03377fe640 1 -- 172.21.15.189:0/459864143 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0320003560 con 0x7f03401334d0 2024-09-16T14:45:45.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 1 -- 172.21.15.189:0/459864143 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1119535274 0 0) 0x7f0320003560 con 0x7f03401334d0 2024-09-16T14:45:45.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 1 -- 172.21.15.189:0/459864143 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0340136320 con 0x7f03401334d0 2024-09-16T14:45:45.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 1 -- 172.21.15.189:0/459864143 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0328004190 con 0x7f03401334d0 2024-09-16T14:45:45.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:45.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:45.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 10 monclient(hunting): dump: 2024-09-16T14:45:45.961 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:45.961 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:45.961 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:45.961 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:45.961 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:45.961 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:45.961 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:45.961 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:45.962 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:45.962 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:45.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:45.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 1 -- 172.21.15.189:0/459864143 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1499325010 0 0) 0x7f0340136320 con 0x7f03401334d0 2024-09-16T14:45:45.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 1 -- 172.21.15.189:0/459864143 >> v1:172.21.15.26:6790/0 conn(0x7f0340136fe0 legacy=0x7f03401cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:45.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 1 -- 172.21.15.189:0/459864143 >> v1:172.21.15.26:6789/0 conn(0x7f0340099a70 legacy=0x7f03401c53c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:45.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 10 monclient: _finish_hunting 0 2024-09-16T14:45:45.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 1 monclient: found mon.b 2024-09-16T14:45:45.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:45.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:45.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 1 -- 172.21.15.189:0/459864143 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034006b380 con 0x7f03401334d0 2024-09-16T14:45:45.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 10 monclient: _finish_auth 0 2024-09-16T14:45:45.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:45.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:45.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f034536a640 5 monclient: authenticate success, global_id 94279 2024-09-16T14:45:45.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f034536a640 10 monclient: _renew_subs 2024-09-16T14:45:45.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f034536a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:45.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f034536a640 1 -- 172.21.15.189:0/459864143 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034011d640 con 0x7f03401334d0 2024-09-16T14:45:45.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f034536a640 10 monclient: _renew_subs 2024-09-16T14:45:45.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f034536a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:45.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f034536a640 1 -- 172.21.15.189:0/459864143 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0340133250 con 0x7f03401334d0 2024-09-16T14:45:45.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 1 -- 172.21.15.189:0/459864143 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0328004480 con 0x7f03401334d0 2024-09-16T14:45:45.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:45.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 1 -- 172.21.15.189:0/459864143 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0328004790 con 0x7f03401334d0 2024-09-16T14:45:45.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:45.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:45.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f033dffb640 4 set_mon_vals no callback set 2024-09-16T14:45:45.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 10 monclient: dump: 2024-09-16T14:45:45.964 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:45.964 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:45.964 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:45.964 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:45.964 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:45.964 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:45.964 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:45.964 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:45.964 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:45.964 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:45.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f034536a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:45.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f034536a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:45.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f034536a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:45.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f034536a640 1 -- 172.21.15.189:0/459864143 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0324005050 con 0x7f03401334d0 2024-09-16T14:45:45.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 1 -- 172.21.15.189:0/459864143 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f03280048f0 con 0x7f03401334d0 2024-09-16T14:45:45.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f03377fe640 1 -- 172.21.15.189:0/459864143 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(63..63 src has 15..63) v4 ==== 4195+0+0 (unknown 1959505487 0 0) 0x7f0328043470 con 0x7f03401334d0 2024-09-16T14:45:45.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.948+0000 7f033effd640 10 monclient: get_auth_request con 0x7f0320036080 auth_method 0 2024-09-16T14:45:45.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.952+0000 7f03377fe640 1 -- 172.21.15.189:0/459864143 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f03280050c0 con 0x7f03401334d0 2024-09-16T14:45:45.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.952+0000 7f03377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:45.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:45.952+0000 7f03377fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:46.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.100+0000 7f034536a640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-16T14:45:46.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.100+0000 7f034536a640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-16T14:45:46.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.100+0000 7f034536a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:46.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.100+0000 7f034536a640 1 -- 172.21.15.189:0/459864143 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"} v 0) v1 -- 0x7f0324005880 con 0x7f03401334d0 2024-09-16T14:45:46.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.104+0000 7f03377fe640 1 -- 172.21.15.189:0/459864143 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}]=-22 type 'osd' is for devices, not buckets v63) v1 ==== 136+0+0 (unknown 2396622767 0 0) 0x7f032801cd90 con 0x7f03401334d0 2024-09-16T14:45:46.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.104+0000 7f03377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-16T14:45:46.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.104+0000 7f03377fe640 10 monclient: _finish_command 2 = mon:22 type 'osd' is for devices, not buckets 2024-09-16T14:45:46.106 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: type 'osd' is for devices, not buckets 2024-09-16T14:45:46.110 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:45:46.110 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket foo root 2024-09-16T14:45:46.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.216+0000 7fda47585640 10 monclient: get_monmap_and_config 2024-09-16T14:45:46.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.216+0000 7fda47585640 10 monclient: build_initial_monmap 2024-09-16T14:45:46.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.216+0000 7fda47585640 10 monclient: monmap: 2024-09-16T14:45:46.222 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:46.222 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:46.222 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:46.221561+0000 2024-09-16T14:45:46.222 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:46.221561+0000 2024-09-16T14:45:46.222 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:46.222 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:46.222 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:46.222 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:46.222 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:46.222 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:46.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 1 Processor -- start 2024-09-16T14:45:46.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 1 -- start start 2024-09-16T14:45:46.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 10 monclient: init 2024-09-16T14:45:46.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:46.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:46.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 10 monclient(hunting): picked mon.noname-a con 0x7fda4009df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:46.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 10 monclient(hunting): picked mon.noname-b con 0x7fda4009e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:46.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 10 monclient(hunting): picked mon.noname-c con 0x7fda4009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:46.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4009ee20 con 0x7fda4009df10 2024-09-16T14:45:46.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4009f100 con 0x7fda4009e740 2024-09-16T14:45:46.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4009f3e0 con 0x7fda4009e2f0 2024-09-16T14:45:46.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:46.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 10 monclient(hunting): authenticate will time out at 1128.337328s 2024-09-16T14:45:46.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda45b22640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fda4009e740 0x7fda4013b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50344/0 (socket says 172.21.15.189:50344) 2024-09-16T14:45:46.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda45b22640 1 -- 172.21.15.189:0/741808107 learned_addr learned my addr 172.21.15.189:0/741808107 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:46.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda45321640 1 -- 172.21.15.189:0/741808107 >> v1:172.21.15.26:6789/0 conn(0x7fda4009df10 legacy=0x7fda4009d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:46.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 1 -- 172.21.15.189:0/741808107 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640997933 0 0) 0x7fda4009f100 con 0x7fda4009e740 2024-09-16T14:45:46.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:46.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:46.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:46.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 10 monclient(hunting): my global_id is 94282 2024-09-16T14:45:46.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 1 -- 172.21.15.189:0/741808107 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda2c003530 con 0x7fda4009e740 2024-09-16T14:45:46.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 1 -- 172.21.15.189:0/741808107 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3181577040 0 0) 0x7fda2c003530 con 0x7fda4009e740 2024-09-16T14:45:46.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 1 -- 172.21.15.189:0/741808107 >> v1:172.21.15.26:6790/0 conn(0x7fda4009e2f0 legacy=0x7fda401378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:46.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 1 -- 172.21.15.189:0/741808107 >> v1:172.21.15.26:6789/0 conn(0x7fda4009df10 legacy=0x7fda4009d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:46.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 10 monclient: _finish_hunting 0 2024-09-16T14:45:46.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 1 monclient: found mon.noname-c 2024-09-16T14:45:46.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:46.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:46.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 1 -- 172.21.15.189:0/741808107 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4009c690 con 0x7fda4009e740 2024-09-16T14:45:46.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 10 monclient: _finish_auth 0 2024-09-16T14:45:46.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 10 monclient: _check_auth_tickets 2024-09-16T14:45:46.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:46.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 1 -- 172.21.15.189:0/741808107 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fda380019a0 con 0x7fda4009e740 2024-09-16T14:45:46.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:46.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:46.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 10 monclient: dump: 2024-09-16T14:45:46.232 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:46.232 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:46.232 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:46.232 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:46.232 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:46.232 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:46.232 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:46.232 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:46.232 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:46.232 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:46.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 5 monclient: authenticate success, global_id 94282 2024-09-16T14:45:46.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:46.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 1 -- 172.21.15.189:0/741808107 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda38001cc0 con 0x7fda4009e740 2024-09-16T14:45:46.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:46.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 1 -- 172.21.15.189:0/741808107 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fda38002090 con 0x7fda4009e740 2024-09-16T14:45:46.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:46.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:46.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 4 set_mon_vals no callback set 2024-09-16T14:45:46.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:46.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda37fff640 10 monclient: dump: 2024-09-16T14:45:46.233 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:46.233 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:46.233 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:46.233 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:46.234 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:46.234 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:46.234 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:46.234 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:46.234 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:46.234 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:46.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 10 monclient: shutdown 2024-09-16T14:45:46.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:46.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 1 -- 172.21.15.189:0/741808107 >> v1:172.21.15.189:6789/0 conn(0x7fda4009e740 legacy=0x7fda4013b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:46.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 1 -- 172.21.15.189:0/741808107 shutdown_connections 2024-09-16T14:45:46.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 1 -- 172.21.15.189:0/741808107 >> 172.21.15.189:0/741808107 conn(0x7fda400a3320 msgr2=0x7fda4009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:46.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 1 -- 172.21.15.189:0/741808107 shutdown_connections 2024-09-16T14:45:46.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 1 -- 172.21.15.189:0/741808107 wait complete. 2024-09-16T14:45:46.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 10 monclient: build_initial_monmap 2024-09-16T14:45:46.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 10 monclient: monmap: 2024-09-16T14:45:46.235 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:46.235 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:46.235 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:46.235 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:46.235 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:46.235 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:46.235 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:46.235 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:46.235 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:46.235 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:46.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 1 Processor -- start 2024-09-16T14:45:46.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 1 -- start start 2024-09-16T14:45:46.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.220+0000 7fda47585640 10 monclient: init 2024-09-16T14:45:46.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:46.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 10 monclient(hunting): picked mon.noname-a con 0x7fda4009df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 10 monclient(hunting): picked mon.noname-c con 0x7fda4009e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 10 monclient(hunting): picked mon.noname-b con 0x7fda4009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4006cdc0 con 0x7fda4009df10 2024-09-16T14:45:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4006b380 con 0x7fda4009e740 2024-09-16T14:45:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda401227c0 con 0x7fda4009e2f0 2024-09-16T14:45:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 10 monclient(hunting): authenticate will time out at 1128.340113s 2024-09-16T14:45:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda45321640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fda4009df10 legacy=0x7fda401c99b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda45b22640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fda4009e740 0x7fda401d20c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50346/0 (socket says 172.21.15.189:50346) 2024-09-16T14:45:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda45b22640 1 -- 172.21.15.189:0/4072004346 learned_addr learned my addr 172.21.15.189:0/4072004346 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 1 -- 172.21.15.189:0/4072004346 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332600426 0 0) 0x7fda4006b380 con 0x7fda4009e740 2024-09-16T14:45:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:46.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:46.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:46.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 10 monclient(hunting): my global_id is 94285 2024-09-16T14:45:46.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 1 -- 172.21.15.189:0/4072004346 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda24003530 con 0x7fda4009e740 2024-09-16T14:45:46.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 1 -- 172.21.15.189:0/4072004346 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2417971960 0 0) 0x7fda24003530 con 0x7fda4009e740 2024-09-16T14:45:46.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 1 -- 172.21.15.189:0/4072004346 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda4006b380 con 0x7fda4009e740 2024-09-16T14:45:46.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 1 -- 172.21.15.189:0/4072004346 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fda38001960 con 0x7fda4009e740 2024-09-16T14:45:46.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:46.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:46.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 10 monclient(hunting): dump: 2024-09-16T14:45:46.237 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:46.237 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:46.237 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:46.238 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:46.238 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:46.238 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:46.238 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:46.238 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:46.238 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:46.238 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:46.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:46.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 1 -- 172.21.15.189:0/4072004346 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2461141294 0 0) 0x7fda4006b380 con 0x7fda4009e740 2024-09-16T14:45:46.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 1 -- 172.21.15.189:0/4072004346 >> v1:172.21.15.26:6790/0 conn(0x7fda4009e2f0 legacy=0x7fda401ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:46.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 1 -- 172.21.15.189:0/4072004346 >> v1:172.21.15.26:6789/0 conn(0x7fda4009df10 legacy=0x7fda401c99b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:46.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:45:46.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 1 monclient: found mon.b 2024-09-16T14:45:46.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:46.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:46.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 1 -- 172.21.15.189:0/4072004346 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda40121370 con 0x7fda4009e740 2024-09-16T14:45:46.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 10 monclient: _finish_auth 0 2024-09-16T14:45:46.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:46.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:46.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 5 monclient: authenticate success, global_id 94285 2024-09-16T14:45:46.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 10 monclient: _renew_subs 2024-09-16T14:45:46.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:46.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 1 -- 172.21.15.189:0/4072004346 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4009bec0 con 0x7fda4009e740 2024-09-16T14:45:46.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 10 monclient: _renew_subs 2024-09-16T14:45:46.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:46.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 1 -- 172.21.15.189:0/4072004346 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4009b230 con 0x7fda4009e740 2024-09-16T14:45:46.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 1 -- 172.21.15.189:0/4072004346 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda38004f80 con 0x7fda4009e740 2024-09-16T14:45:46.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:46.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 1 -- 172.21.15.189:0/4072004346 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fda38005310 con 0x7fda4009e740 2024-09-16T14:45:46.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:46.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:46.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 10 monclient: dump: 2024-09-16T14:45:46.240 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:46.240 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:46.240 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:46.240 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:46.240 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:46.240 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:46.240 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:46.240 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:46.240 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:46.240 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:46.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda377fe640 4 set_mon_vals no callback set 2024-09-16T14:45:46.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 1 -- 172.21.15.189:0/4072004346 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fda38005470 con 0x7fda4009e740 2024-09-16T14:45:46.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:46.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:46.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:46.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda47585640 1 -- 172.21.15.189:0/4072004346 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda20005050 con 0x7fda4009e740 2024-09-16T14:45:46.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda35ffb640 1 -- 172.21.15.189:0/4072004346 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(63..63 src has 15..63) v4 ==== 4195+0+0 (unknown 1959505487 0 0) 0x7fda38043970 con 0x7fda4009e740 2024-09-16T14:45:46.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.224+0000 7fda45321640 10 monclient: get_auth_request con 0x7fda24035fe0 auth_method 0 2024-09-16T14:45:46.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.228+0000 7fda35ffb640 1 -- 172.21.15.189:0/4072004346 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fda38014dc0 con 0x7fda4009e740 2024-09-16T14:45:46.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.228+0000 7fda35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:46.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.228+0000 7fda35ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:46.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.376+0000 7fda47585640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T14:45:46.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.376+0000 7fda47585640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T14:45:46.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.376+0000 7fda47585640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:46.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.376+0000 7fda47585640 1 -- 172.21.15.189:0/4072004346 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "root"} v 0) v1 -- 0x7fda20005880 con 0x7fda4009e740 2024-09-16T14:45:46.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.724+0000 7fda35ffb640 1 -- 172.21.15.189:0/4072004346 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}]=0 added bucket foo type root to crush map v64) v1 ==== 138+0+0 (unknown 1156879197 0 0) 0x7fda38013230 con 0x7fda4009e740 2024-09-16T14:45:46.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.724+0000 7fda35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T14:45:46.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.724+0000 7fda35ffb640 10 monclient: _finish_command 2 = system:0 added bucket foo type root to crush map 2024-09-16T14:45:46.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.824+0000 7fda47585640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T14:45:46.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.824+0000 7fda47585640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T14:45:46.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.824+0000 7fda47585640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:46.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.824+0000 7fda47585640 1 -- 172.21.15.189:0/4072004346 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "root"} v 0) v1 -- 0x7fda20005340 con 0x7fda4009e740 2024-09-16T14:45:46.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.824+0000 7fda35ffb640 1 -- 172.21.15.189:0/4072004346 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}]=0 bucket 'foo' already exists v64) v1 ==== 126+0+0 (unknown 1014795778 0 0) 0x7fda38014dc0 con 0x7fda4009e740 2024-09-16T14:45:46.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.824+0000 7fda35ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T14:45:46.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.824+0000 7fda35ffb640 10 monclient: _finish_command 3 = system:0 bucket 'foo' already exists 2024-09-16T14:45:46.830 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'foo' already exists 2024-09-16T14:45:46.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.828+0000 7fda47585640 1 -- 172.21.15.189:0/4072004346 >> v1:172.21.15.189:6812/3196161582 conn(0x7fda24035fe0 legacy=0x7fda240384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:46.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.828+0000 7fda47585640 10 monclient: shutdown 2024-09-16T14:45:46.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.828+0000 7fda47585640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:46.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.828+0000 7fda47585640 1 -- 172.21.15.189:0/4072004346 >> v1:172.21.15.189:6789/0 conn(0x7fda4009e740 legacy=0x7fda401d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:46.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.828+0000 7fda47585640 1 -- 172.21.15.189:0/4072004346 shutdown_connections 2024-09-16T14:45:46.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.828+0000 7fda47585640 1 -- 172.21.15.189:0/4072004346 >> 172.21.15.189:0/4072004346 conn(0x7fda400a3320 msgr2=0x7fda4012a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:46.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.828+0000 7fda47585640 1 -- 172.21.15.189:0/4072004346 shutdown_connections 2024-09-16T14:45:46.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.828+0000 7fda47585640 1 -- 172.21.15.189:0/4072004346 wait complete. 2024-09-16T14:45:46.844 INFO:tasks.workunit.client.0.smithi189.stderr:++ ceph osd create 2024-09-16T14:45:46.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.948+0000 7f3285890640 10 monclient: get_monmap_and_config 2024-09-16T14:45:46.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.948+0000 7f3285890640 10 monclient: build_initial_monmap 2024-09-16T14:45:46.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 10 monclient: monmap: 2024-09-16T14:45:46.954 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:46.954 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:46.954 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:46.953798+0000 2024-09-16T14:45:46.954 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:46.953798+0000 2024-09-16T14:45:46.954 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:46.954 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:46.954 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:46.955 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:46.955 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:46.955 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:46.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 1 Processor -- start 2024-09-16T14:45:46.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 1 -- start start 2024-09-16T14:45:46.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 10 monclient: init 2024-09-16T14:45:46.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:46.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:46.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 10 monclient(hunting): picked mon.noname-a con 0x7f3280073590 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:46.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 10 monclient(hunting): picked mon.noname-b con 0x7f3280073970 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:46.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 10 monclient(hunting): picked mon.noname-c con 0x7f328013b2a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:46.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3280098200 con 0x7f3280073590 2024-09-16T14:45:46.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32800718f0 con 0x7f328013b2a0 2024-09-16T14:45:46.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3280135080 con 0x7f3280073970 2024-09-16T14:45:46.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:46.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 10 monclient(hunting): authenticate will time out at 1129.069590s 2024-09-16T14:45:46.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327effd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f3280073590 legacy=0x7f32800998f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:46.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327f7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f328013b2a0 0x7f328013d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50356/0 (socket says 172.21.15.189:50356) 2024-09-16T14:45:46.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327f7fe640 1 -- 172.21.15.189:0/1446230628 learned_addr learned my addr 172.21.15.189:0/1446230628 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:46.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 1 -- 172.21.15.189:0/1446230628 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754030309 0 0) 0x7f32800718f0 con 0x7f328013b2a0 2024-09-16T14:45:46.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:46.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:46.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:46.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 10 monclient(hunting): my global_id is 94288 2024-09-16T14:45:46.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 1 -- 172.21.15.189:0/1446230628 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3268003530 con 0x7f328013b2a0 2024-09-16T14:45:46.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 1 -- 172.21.15.189:0/1446230628 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3260964423 0 0) 0x7f3268003530 con 0x7f328013b2a0 2024-09-16T14:45:46.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 1 -- 172.21.15.189:0/1446230628 >> v1:172.21.15.26:6790/0 conn(0x7f3280073970 legacy=0x7f3280139b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:46.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 1 -- 172.21.15.189:0/1446230628 >> v1:172.21.15.26:6789/0 conn(0x7f3280073590 legacy=0x7f32800998f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:46.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:45:46.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 1 monclient: found mon.noname-c 2024-09-16T14:45:46.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:46.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:46.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 1 -- 172.21.15.189:0/1446230628 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32800973d0 con 0x7f328013b2a0 2024-09-16T14:45:46.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 10 monclient: _finish_auth 0 2024-09-16T14:45:46.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:46.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:46.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 1 -- 172.21.15.189:0/1446230628 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3270001620 con 0x7f328013b2a0 2024-09-16T14:45:46.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:46.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:46.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 10 monclient: dump: 2024-09-16T14:45:46.965 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:46.965 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:46.965 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:46.965 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:46.965 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:46.965 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:46.966 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:46.966 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:46.966 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:46.966 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:46.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 5 monclient: authenticate success, global_id 94288 2024-09-16T14:45:46.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:46.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 1 -- 172.21.15.189:0/1446230628 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3270001940 con 0x7f328013b2a0 2024-09-16T14:45:46.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:46.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 1 -- 172.21.15.189:0/1446230628 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3270001d10 con 0x7f328013b2a0 2024-09-16T14:45:46.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:46.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 4 set_mon_vals no callback set 2024-09-16T14:45:46.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:46.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:46.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f327dffb640 10 monclient: dump: 2024-09-16T14:45:46.967 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:46.967 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:46.967 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:46.967 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:46.967 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:46.967 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:46.967 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:46.967 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:46.967 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:46.967 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:46.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 10 monclient: shutdown 2024-09-16T14:45:46.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:46.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 1 -- 172.21.15.189:0/1446230628 >> v1:172.21.15.189:6789/0 conn(0x7f328013b2a0 legacy=0x7f328013d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:46.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 1 -- 172.21.15.189:0/1446230628 shutdown_connections 2024-09-16T14:45:46.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 1 -- 172.21.15.189:0/1446230628 >> 172.21.15.189:0/1446230628 conn(0x7f32800a0030 msgr2=0x7f32800a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:46.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 1 -- 172.21.15.189:0/1446230628 shutdown_connections 2024-09-16T14:45:46.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 1 -- 172.21.15.189:0/1446230628 wait complete. 2024-09-16T14:45:46.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 10 monclient: build_initial_monmap 2024-09-16T14:45:46.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 10 monclient: monmap: 2024-09-16T14:45:46.968 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:46.968 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:46.968 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:46.968 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:46.968 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:46.968 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:46.969 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:46.969 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:46.969 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:46.969 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:46.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 1 Processor -- start 2024-09-16T14:45:46.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 1 -- start start 2024-09-16T14:45:46.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.952+0000 7f3285890640 10 monclient: init 2024-09-16T14:45:46.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:46.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:46.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 10 monclient(hunting): picked mon.noname-b con 0x7f3280073590 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:46.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 10 monclient(hunting): picked mon.noname-c con 0x7f3280073970 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:46.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 10 monclient(hunting): picked mon.noname-a con 0x7f328013b2a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:46.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328009a550 con 0x7f328013b2a0 2024-09-16T14:45:46.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328012b3c0 con 0x7f3280073590 2024-09-16T14:45:46.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328012b6a0 con 0x7f3280073970 2024-09-16T14:45:46.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:46.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 10 monclient(hunting): authenticate will time out at 1129.072307s 2024-09-16T14:45:46.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f327effd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3280073590 0x7f3280096310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50362/0 (socket says 172.21.15.189:50362) 2024-09-16T14:45:46.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f327effd640 1 -- 172.21.15.189:0/4040888343 learned_addr learned my addr 172.21.15.189:0/4040888343 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:46.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f327f7fe640 1 -- 172.21.15.189:0/4040888343 >> v1:172.21.15.26:6789/0 conn(0x7f328013b2a0 legacy=0x7f32801d1fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:46.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 1 -- 172.21.15.189:0/4040888343 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450953955 0 0) 0x7f328012b3c0 con 0x7f3280073590 2024-09-16T14:45:46.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:46.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:46.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:46.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 10 monclient(hunting): my global_id is 94291 2024-09-16T14:45:46.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 1 -- 172.21.15.189:0/4040888343 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f326c003530 con 0x7f3280073590 2024-09-16T14:45:46.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 1 -- 172.21.15.189:0/4040888343 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1811849543 0 0) 0x7f326c003530 con 0x7f3280073590 2024-09-16T14:45:46.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 1 -- 172.21.15.189:0/4040888343 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f328012b3c0 con 0x7f3280073590 2024-09-16T14:45:46.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 1 -- 172.21.15.189:0/4040888343 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3278004510 con 0x7f3280073590 2024-09-16T14:45:46.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:46.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:46.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 10 monclient(hunting): dump: 2024-09-16T14:45:46.971 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:46.971 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:46.971 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:46.971 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:46.971 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:46.971 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:46.971 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:46.971 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:46.972 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:46.972 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:46.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:46.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 1 -- 172.21.15.189:0/4040888343 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2921588148 0 0) 0x7f328012b3c0 con 0x7f3280073590 2024-09-16T14:45:46.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 1 -- 172.21.15.189:0/4040888343 >> v1:172.21.15.26:6790/0 conn(0x7f3280073970 legacy=0x7f32801ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:46.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 1 -- 172.21.15.189:0/4040888343 >> v1:172.21.15.26:6789/0 conn(0x7f328013b2a0 legacy=0x7f32801d1fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:46.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 10 monclient: _finish_hunting 0 2024-09-16T14:45:46.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 1 monclient: found mon.b 2024-09-16T14:45:46.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:46.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:46.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 1 -- 172.21.15.189:0/4040888343 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328012b980 con 0x7f3280073590 2024-09-16T14:45:46.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 10 monclient: _finish_auth 0 2024-09-16T14:45:46.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 10 monclient: _check_auth_tickets 2024-09-16T14:45:46.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:46.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 5 monclient: authenticate success, global_id 94291 2024-09-16T14:45:46.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 10 monclient: _renew_subs 2024-09-16T14:45:46.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:46.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 1 -- 172.21.15.189:0/4040888343 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3280129ca0 con 0x7f3280073590 2024-09-16T14:45:46.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 10 monclient: _renew_subs 2024-09-16T14:45:46.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:46.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 1 -- 172.21.15.189:0/4040888343 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f328012a2c0 con 0x7f3280073590 2024-09-16T14:45:46.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 1 -- 172.21.15.189:0/4040888343 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3278004800 con 0x7f3280073590 2024-09-16T14:45:46.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:46.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 1 -- 172.21.15.189:0/4040888343 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3278004b90 con 0x7f3280073590 2024-09-16T14:45:46.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:46.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:46.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f327d7fa640 4 set_mon_vals no callback set 2024-09-16T14:45:46.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 10 monclient: dump: 2024-09-16T14:45:46.974 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:46.974 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:46.974 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:46.974 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:46.974 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:46.974 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:46.974 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:46.974 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:46.974 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:46.974 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:46.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 1 -- 172.21.15.189:0/4040888343 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3278004db0 con 0x7f3280073590 2024-09-16T14:45:46.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:46.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:46.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:46.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3285890640 1 -- 172.21.15.189:0/4040888343 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3260005050 con 0x7f3280073590 2024-09-16T14:45:46.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f3267fff640 1 -- 172.21.15.189:0/4040888343 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(64..64 src has 15..64) v4 ==== 4222+0+0 (unknown 1518984211 0 0) 0x7f32780438f0 con 0x7f3280073590 2024-09-16T14:45:46.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.956+0000 7f327e7fc640 10 monclient: get_auth_request con 0x7f326c036000 auth_method 0 2024-09-16T14:45:46.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.960+0000 7f3267fff640 1 -- 172.21.15.189:0/4040888343 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3278014c10 con 0x7f3280073590 2024-09-16T14:45:46.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.960+0000 7f3267fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:46.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:46.960+0000 7f3267fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:47.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:47.120+0000 7f3285890640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T14:45:47.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:47.120+0000 7f3285890640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T14:45:47.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:47.120+0000 7f3285890640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:47.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:47.120+0000 7f3285890640 1 -- 172.21.15.189:0/4040888343 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f3260005880 con 0x7f3280073590 2024-09-16T14:45:47.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:47.728+0000 7f3267fff640 1 -- 172.21.15.189:0/4040888343 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v65) v1 ==== 58+0+2 (unknown 1587384094 0 1113811849) 0x7f3278013080 con 0x7f3280073590 2024-09-16T14:45:47.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:47.728+0000 7f3267fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T14:45:47.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:47.728+0000 7f3267fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:47.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:47.832+0000 7f3285890640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T14:45:47.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:47.832+0000 7f3285890640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T14:45:47.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:47.832+0000 7f3285890640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:47.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:47.832+0000 7f3285890640 1 -- 172.21.15.189:0/4040888343 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f3260005370 con 0x7f3280073590 2024-09-16T14:45:48.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.740+0000 7f3267fff640 1 -- 172.21.15.189:0/4040888343 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v66) v1 ==== 58+0+2 (unknown 2230295283 0 1371665406) 0x7f3278014800 con 0x7f3280073590 2024-09-16T14:45:48.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.740+0000 7f3267fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T14:45:48.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.740+0000 7f3267fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:45:48.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.744+0000 7f3285890640 1 -- 172.21.15.189:0/4040888343 >> v1:172.21.15.189:6812/3196161582 conn(0x7f326c036000 legacy=0x7f326c0384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:48.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.744+0000 7f3285890640 10 monclient: shutdown 2024-09-16T14:45:48.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.744+0000 7f3285890640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:48.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.744+0000 7f3285890640 1 -- 172.21.15.189:0/4040888343 >> v1:172.21.15.189:6789/0 conn(0x7f3280073590 legacy=0x7f3280096310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:48.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.744+0000 7f3285890640 1 -- 172.21.15.189:0/4040888343 shutdown_connections 2024-09-16T14:45:48.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.744+0000 7f3285890640 1 -- 172.21.15.189:0/4040888343 >> 172.21.15.189:0/4040888343 conn(0x7f32800a0030 msgr2=0x7f32800a20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:48.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.744+0000 7f3285890640 1 -- 172.21.15.189:0/4040888343 shutdown_connections 2024-09-16T14:45:48.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.744+0000 7f3285890640 1 -- 172.21.15.189:0/4040888343 wait complete. 2024-09-16T14:45:48.768 INFO:tasks.workunit.client.0.smithi189.stderr:+ o1=7 2024-09-16T14:45:48.769 INFO:tasks.workunit.client.0.smithi189.stderr:++ ceph osd create 2024-09-16T14:45:48.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.880+0000 7f46533b9640 10 monclient: get_monmap_and_config 2024-09-16T14:45:48.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.880+0000 7f46533b9640 10 monclient: build_initial_monmap 2024-09-16T14:45:48.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.880+0000 7f46533b9640 10 monclient: monmap: 2024-09-16T14:45:48.886 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:48.886 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:48.886 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:48.885393+0000 2024-09-16T14:45:48.886 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:48.885393+0000 2024-09-16T14:45:48.886 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:48.886 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:48.886 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:48.886 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:48.886 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:48.887 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:48.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 1 Processor -- start 2024-09-16T14:45:48.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 1 -- start start 2024-09-16T14:45:48.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient: init 2024-09-16T14:45:48.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:48.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:48.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient(hunting): picked mon.noname-b con 0x7f464c09df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:48.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient(hunting): picked mon.noname-c con 0x7f464c09e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:48.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient(hunting): picked mon.noname-a con 0x7f464c09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:48.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464c09ee20 con 0x7f464c09e740 2024-09-16T14:45:48.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464c09f100 con 0x7f464c09e2f0 2024-09-16T14:45:48.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464c09f3e0 con 0x7f464c09df10 2024-09-16T14:45:48.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:48.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient(hunting): authenticate will time out at 1131.001156s 2024-09-16T14:45:48.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f4650954640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f464c09e2f0 0x7f464c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50376/0 (socket says 172.21.15.189:50376) 2024-09-16T14:45:48.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f4650954640 1 -- 172.21.15.189:0/374876289 learned_addr learned my addr 172.21.15.189:0/374876289 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:48.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f4651956640 1 -- 172.21.15.189:0/374876289 >> v1:172.21.15.26:6789/0 conn(0x7f464c09e740 legacy=0x7f464c13b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:48.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 1 -- 172.21.15.189:0/374876289 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707957546 0 0) 0x7f464c09f100 con 0x7f464c09e2f0 2024-09-16T14:45:48.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:48.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:48.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:48.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 10 monclient(hunting): my global_id is 94294 2024-09-16T14:45:48.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 1 -- 172.21.15.189:0/374876289 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4638003530 con 0x7f464c09e2f0 2024-09-16T14:45:48.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 1 -- 172.21.15.189:0/374876289 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1508003986 0 0) 0x7f4638003530 con 0x7f464c09e2f0 2024-09-16T14:45:48.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 1 -- 172.21.15.189:0/374876289 >> v1:172.21.15.26:6790/0 conn(0x7f464c09df10 legacy=0x7f464c09d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:48.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 1 -- 172.21.15.189:0/374876289 >> v1:172.21.15.26:6789/0 conn(0x7f464c09e740 legacy=0x7f464c13b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:48.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:45:48.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 1 monclient: found mon.noname-c 2024-09-16T14:45:48.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:48.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:48.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 1 -- 172.21.15.189:0/374876289 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c09c690 con 0x7f464c09e2f0 2024-09-16T14:45:48.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 10 monclient: _finish_auth 0 2024-09-16T14:45:48.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:45:48.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:48.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 1 -- 172.21.15.189:0/374876289 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4640001620 con 0x7f464c09e2f0 2024-09-16T14:45:48.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:48.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:48.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 10 monclient: dump: 2024-09-16T14:45:48.896 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:48.896 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:48.896 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:48.896 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:48.897 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:48.897 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:48.897 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:48.897 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:48.897 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:48.897 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:48.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 5 monclient: authenticate success, global_id 94294 2024-09-16T14:45:48.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:48.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 1 -- 172.21.15.189:0/374876289 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4640001940 con 0x7f464c09e2f0 2024-09-16T14:45:48.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:48.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 1 -- 172.21.15.189:0/374876289 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4640001d10 con 0x7f464c09e2f0 2024-09-16T14:45:48.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:48.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:48.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 4 set_mon_vals no callback set 2024-09-16T14:45:48.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:48.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f463ffff640 10 monclient: dump: 2024-09-16T14:45:48.898 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:48.898 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:48.898 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:48.898 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:48.898 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:48.898 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:48.899 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:48.899 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:48.899 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:48.899 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:48.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient: shutdown 2024-09-16T14:45:48.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:48.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 1 -- 172.21.15.189:0/374876289 >> v1:172.21.15.189:6789/0 conn(0x7f464c09e2f0 legacy=0x7f464c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:48.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 1 -- 172.21.15.189:0/374876289 shutdown_connections 2024-09-16T14:45:48.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 1 -- 172.21.15.189:0/374876289 >> 172.21.15.189:0/374876289 conn(0x7f464c0a3320 msgr2=0x7f464c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:48.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 1 -- 172.21.15.189:0/374876289 shutdown_connections 2024-09-16T14:45:48.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 1 -- 172.21.15.189:0/374876289 wait complete. 2024-09-16T14:45:48.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient: build_initial_monmap 2024-09-16T14:45:48.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient: monmap: 2024-09-16T14:45:48.900 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:48.900 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:48.900 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:48.900 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:48.900 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:48.900 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:48.900 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:48.900 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:48.900 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:48.900 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:48.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 1 Processor -- start 2024-09-16T14:45:48.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 1 -- start start 2024-09-16T14:45:48.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient: init 2024-09-16T14:45:48.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:48.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:45:48.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient(hunting): picked mon.noname-c con 0x7f464c09df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:48.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient(hunting): picked mon.noname-a con 0x7f464c09e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:48.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient(hunting): picked mon.noname-b con 0x7f464c09e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:48.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464c13ab80 con 0x7f464c09e2f0 2024-09-16T14:45:48.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464c06cea0 con 0x7f464c09e740 2024-09-16T14:45:48.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464c06f970 con 0x7f464c09df10 2024-09-16T14:45:48.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:48.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.884+0000 7f46533b9640 10 monclient(hunting): authenticate will time out at 1131.003985s 2024-09-16T14:45:48.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f4650954640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f464c09e2f0 legacy=0x7f464c09a900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:48.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f4651956640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f464c09e740 0x7f464c1d17a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50386/0 (socket says 172.21.15.189:50386) 2024-09-16T14:45:48.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f4651956640 1 -- 172.21.15.189:0/3434881756 learned_addr learned my addr 172.21.15.189:0/3434881756 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:48.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368896746 0 0) 0x7f464c06cea0 con 0x7f464c09e740 2024-09-16T14:45:48.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:48.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:48.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:48.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient(hunting): my global_id is 94297 2024-09-16T14:45:48.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4630003530 con 0x7f464c09e740 2024-09-16T14:45:48.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3662870024 0 0) 0x7f4630003530 con 0x7f464c09e740 2024-09-16T14:45:48.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f464c06cea0 con 0x7f464c09e740 2024-09-16T14:45:48.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4644004400 con 0x7f464c09e740 2024-09-16T14:45:48.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:48.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:48.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient(hunting): dump: 2024-09-16T14:45:48.902 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:48.902 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:48.902 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:48.902 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:48.903 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:48.903 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:48.903 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:48.903 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:48.903 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:48.903 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:48.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:48.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145816580 0 0) 0x7f464c06f970 con 0x7f464c09df10 2024-09-16T14:45:48.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:48.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:48.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:48.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient(hunting): my global_id is 5183 2024-09-16T14:45:48.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4630003530 con 0x7f464c09df10 2024-09-16T14:45:48.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1303838219 0 0) 0x7f464c06cea0 con 0x7f464c09e740 2024-09-16T14:45:48.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 >> v1:172.21.15.26:6790/0 conn(0x7f464c09df10 legacy=0x7f464c09a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:48.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 >> v1:172.21.15.26:6789/0 conn(0x7f464c09e2f0 legacy=0x7f464c09a900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:48.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:45:48.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 1 monclient: found mon.b 2024-09-16T14:45:48.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:48.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:48.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c13b370 con 0x7f464c09e740 2024-09-16T14:45:48.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient: _finish_auth 0 2024-09-16T14:45:48.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:48.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:48.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f46533b9640 5 monclient: authenticate success, global_id 94297 2024-09-16T14:45:48.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f46533b9640 10 monclient: _renew_subs 2024-09-16T14:45:48.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f46533b9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:48.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f46533b9640 1 -- 172.21.15.189:0/3434881756 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f464c09b010 con 0x7f464c09e740 2024-09-16T14:45:48.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f46533b9640 10 monclient: _renew_subs 2024-09-16T14:45:48.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f46533b9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:48.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f46533b9640 1 -- 172.21.15.189:0/3434881756 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f464c13b090 con 0x7f464c09e740 2024-09-16T14:45:48.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4644004720 con 0x7f464c09e740 2024-09-16T14:45:48.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:48.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4644004ab0 con 0x7f464c09e740 2024-09-16T14:45:48.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:48.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:48.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f46533b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:48.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 10 monclient: dump: 2024-09-16T14:45:48.905 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:48.906 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:48.906 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:48.906 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:48.906 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:48.906 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:48.906 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:48.906 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:48.906 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:48.906 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:48.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463f7fe640 4 set_mon_vals no callback set 2024-09-16T14:45:48.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4644004c50 con 0x7f464c09e740 2024-09-16T14:45:48.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(66..66 src has 15..66) v4 ==== 4452+0+0 (unknown 3486522517 0 0) 0x7f46440437b0 con 0x7f464c09e740 2024-09-16T14:45:48.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f46533b9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:48.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f46533b9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:48.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f46533b9640 1 -- 172.21.15.189:0/3434881756 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464c0008e0 con 0x7f464c09e740 2024-09-16T14:45:48.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.888+0000 7f4651155640 10 monclient: get_auth_request con 0x7f4630036160 auth_method 0 2024-09-16T14:45:48.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.892+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4644005450 con 0x7f464c09e740 2024-09-16T14:45:48.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.892+0000 7f463dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:48.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:48.892+0000 7f463dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:49.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:49.048+0000 7f46533b9640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T14:45:49.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:49.048+0000 7f46533b9640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T14:45:49.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:49.048+0000 7f46533b9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:49.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:49.048+0000 7f46533b9640 1 -- 172.21.15.189:0/3434881756 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f464c139b40 con 0x7f464c09e740 2024-09-16T14:45:49.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:49.512+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v67) v1 ==== 58+0+2 (unknown 832193543 0 3065575683) 0x7f4644012f80 con 0x7f464c09e740 2024-09-16T14:45:49.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:49.512+0000 7f463dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T14:45:49.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:49.512+0000 7f463dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:49.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:49.616+0000 7f46533b9640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T14:45:49.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:49.616+0000 7f46533b9640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T14:45:49.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:49.616+0000 7f46533b9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:49.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:49.616+0000 7f46533b9640 1 -- 172.21.15.189:0/3434881756 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f464c099cd0 con 0x7f464c09e740 2024-09-16T14:45:50.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.516+0000 7f463dffb640 1 -- 172.21.15.189:0/3434881756 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v68) v1 ==== 58+0+2 (unknown 904382424 0 2769969524) 0x7f4644014700 con 0x7f464c09e740 2024-09-16T14:45:50.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.516+0000 7f463dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T14:45:50.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.516+0000 7f463dffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:45:50.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.516+0000 7f46533b9640 1 -- 172.21.15.189:0/3434881756 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4630036160 legacy=0x7f4630038620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:50.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.516+0000 7f46533b9640 10 monclient: shutdown 2024-09-16T14:45:50.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.516+0000 7f46533b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:50.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.516+0000 7f46533b9640 1 -- 172.21.15.189:0/3434881756 >> v1:172.21.15.189:6789/0 conn(0x7f464c09e740 legacy=0x7f464c1d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:50.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.520+0000 7f46533b9640 1 -- 172.21.15.189:0/3434881756 shutdown_connections 2024-09-16T14:45:50.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.520+0000 7f46533b9640 1 -- 172.21.15.189:0/3434881756 >> 172.21.15.189:0/3434881756 conn(0x7f464c0a3320 msgr2=0x7f464c06e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:50.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.520+0000 7f46533b9640 1 -- 172.21.15.189:0/3434881756 shutdown_connections 2024-09-16T14:45:50.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.520+0000 7f46533b9640 1 -- 172.21.15.189:0/3434881756 wait complete. 2024-09-16T14:45:50.539 INFO:tasks.workunit.client.0.smithi189.stderr:+ o2=9 2024-09-16T14:45:50.539 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add 7 1 host=host1 root=foo 2024-09-16T14:45:50.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a8630640 10 monclient: get_monmap_and_config 2024-09-16T14:45:50.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a8630640 10 monclient: build_initial_monmap 2024-09-16T14:45:50.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a8630640 10 monclient: monmap: 2024-09-16T14:45:50.660 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:50.660 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:50.660 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:50.659877+0000 2024-09-16T14:45:50.660 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:50.659877+0000 2024-09-16T14:45:50.660 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:50.661 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:50.661 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:50.661 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:50.661 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:50.661 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:50.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a8630640 1 Processor -- start 2024-09-16T14:45:50.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a8630640 1 -- start start 2024-09-16T14:45:50.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a8630640 10 monclient: init 2024-09-16T14:45:50.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a8630640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:50.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a8630640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:45:50.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a8630640 10 monclient(hunting): picked mon.noname-a con 0x7f96a0099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:50.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a8630640 10 monclient(hunting): picked mon.noname-c con 0x7f96a009c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:50.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a8630640 10 monclient(hunting): picked mon.noname-b con 0x7f96a009e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:50.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a8630640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a0099e50 con 0x7f96a0099a70 2024-09-16T14:45:50.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a8630640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a009dbd0 con 0x7f96a009c550 2024-09-16T14:45:50.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a8630640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a009ee20 con 0x7f96a009e6d0 2024-09-16T14:45:50.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a8630640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:50.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a8630640 10 monclient(hunting): authenticate will time out at 1132.775584s 2024-09-16T14:45:50.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a5bcb640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f96a009c550 0x7f96a01379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46964/0 (socket says 172.21.15.189:46964) 2024-09-16T14:45:50.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a5bcb640 1 -- 172.21.15.189:0/3934125217 learned_addr learned my addr 172.21.15.189:0/3934125217 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:50.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.656+0000 7f96a63cc640 1 -- 172.21.15.189:0/3934125217 >> v1:172.21.15.26:6789/0 conn(0x7f96a0099a70 legacy=0x7f96a009dfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:50.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 1 -- 172.21.15.189:0/3934125217 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057472835 0 0) 0x7f96a009dbd0 con 0x7f96a009c550 2024-09-16T14:45:50.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 10 monclient(hunting): _negotiate 2024-09-16T14:45:50.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:50.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:50.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 10 monclient(hunting): my global_id is 94300 2024-09-16T14:45:50.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 1 -- 172.21.15.189:0/3934125217 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f968c003530 con 0x7f96a009c550 2024-09-16T14:45:50.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 1 -- 172.21.15.189:0/3934125217 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 794010501 0 0) 0x7f968c003530 con 0x7f96a009c550 2024-09-16T14:45:50.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 1 -- 172.21.15.189:0/3934125217 >> v1:172.21.15.26:6790/0 conn(0x7f96a009e6d0 legacy=0x7f96a013b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:50.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 1 -- 172.21.15.189:0/3934125217 >> v1:172.21.15.26:6789/0 conn(0x7f96a0099a70 legacy=0x7f96a009dfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:50.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 10 monclient: _finish_hunting 0 2024-09-16T14:45:50.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 1 monclient: found mon.noname-c 2024-09-16T14:45:50.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:50.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:50.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 1 -- 172.21.15.189:0/3934125217 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a009f100 con 0x7f96a009c550 2024-09-16T14:45:50.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 10 monclient: _finish_auth 0 2024-09-16T14:45:50.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 10 monclient: _check_auth_tickets 2024-09-16T14:45:50.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:50.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 5 monclient: authenticate success, global_id 94300 2024-09-16T14:45:50.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:50.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 1 -- 172.21.15.189:0/3934125217 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9690001620 con 0x7f96a009c550 2024-09-16T14:45:50.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:50.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:50.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 10 monclient: dump: 2024-09-16T14:45:50.671 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:50.671 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:50.671 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:50.671 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:50.671 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:50.671 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:50.671 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:50.671 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:50.672 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:50.672 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:50.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:50.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 1 -- 172.21.15.189:0/3934125217 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96900018d0 con 0x7f96a009c550 2024-09-16T14:45:50.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:50.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 1 -- 172.21.15.189:0/3934125217 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9690001ca0 con 0x7f96a009c550 2024-09-16T14:45:50.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:50.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 4 set_mon_vals no callback set 2024-09-16T14:45:50.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:50.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:50.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a53ca640 10 monclient: dump: 2024-09-16T14:45:50.672 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:50.672 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:50.673 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:50.673 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:50.673 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:50.673 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:50.673 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:50.673 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:50.673 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:50.673 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:50.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 10 monclient: shutdown 2024-09-16T14:45:50.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:50.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 1 -- 172.21.15.189:0/3934125217 >> v1:172.21.15.189:6789/0 conn(0x7f96a009c550 legacy=0x7f96a01379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:50.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 1 -- 172.21.15.189:0/3934125217 shutdown_connections 2024-09-16T14:45:50.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 1 -- 172.21.15.189:0/3934125217 >> 172.21.15.189:0/3934125217 conn(0x7f96a00a3380 msgr2=0x7f96a009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:50.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 1 -- 172.21.15.189:0/3934125217 shutdown_connections 2024-09-16T14:45:50.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 1 -- 172.21.15.189:0/3934125217 wait complete. 2024-09-16T14:45:50.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 10 monclient: build_initial_monmap 2024-09-16T14:45:50.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 10 monclient: monmap: 2024-09-16T14:45:50.674 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:50.674 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:50.674 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:50.674 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:50.674 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:50.674 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:50.674 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:50.674 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:50.675 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:50.675 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:50.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 1 Processor -- start 2024-09-16T14:45:50.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 1 -- start start 2024-09-16T14:45:50.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 10 monclient: init 2024-09-16T14:45:50.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:50.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:50.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 10 monclient(hunting): picked mon.noname-a con 0x7f96a0099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:50.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 10 monclient(hunting): picked mon.noname-c con 0x7f96a009c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:50.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 10 monclient(hunting): picked mon.noname-b con 0x7f96a009e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:50.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a009cf20 con 0x7f96a0099a70 2024-09-16T14:45:50.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a0139ec0 con 0x7f96a009e6d0 2024-09-16T14:45:50.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a009b6c0 con 0x7f96a009c550 2024-09-16T14:45:50.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:50.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 10 monclient(hunting): authenticate will time out at 1132.778550s 2024-09-16T14:45:50.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a6bcd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f96a009e6d0 0x7f96a01d1720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46970/0 (socket says 172.21.15.189:46970) 2024-09-16T14:45:50.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a6bcd640 1 -- 172.21.15.189:0/3892980216 learned_addr learned my addr 172.21.15.189:0/3892980216 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:50.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a63cc640 1 -- 172.21.15.189:0/3892980216 >> v1:172.21.15.26:6789/0 conn(0x7f96a0099a70 legacy=0x7f96a009a1f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:50.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338632799 0 0) 0x7f96a0139ec0 con 0x7f96a009e6d0 2024-09-16T14:45:50.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:50.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:50.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:50.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient(hunting): my global_id is 94303 2024-09-16T14:45:50.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9684003530 con 0x7f96a009e6d0 2024-09-16T14:45:50.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2510716208 0 0) 0x7f9684003530 con 0x7f96a009e6d0 2024-09-16T14:45:50.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96a0139ec0 con 0x7f96a009e6d0 2024-09-16T14:45:50.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9698001c40 con 0x7f96a009e6d0 2024-09-16T14:45:50.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:50.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:50.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient(hunting): dump: 2024-09-16T14:45:50.677 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:50.677 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:50.677 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:50.677 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:50.677 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:50.677 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:50.678 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:50.678 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:50.678 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:50.678 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:50.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:50.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733558956 0 0) 0x7f96a009b6c0 con 0x7f96a009c550 2024-09-16T14:45:50.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:50.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:50.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:50.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient(hunting): my global_id is 5189 2024-09-16T14:45:50.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9684003530 con 0x7f96a009c550 2024-09-16T14:45:50.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 294244391 0 0) 0x7f96a0139ec0 con 0x7f96a009e6d0 2024-09-16T14:45:50.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 >> v1:172.21.15.26:6790/0 conn(0x7f96a009c550 legacy=0x7f96a009a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:50.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 >> v1:172.21.15.26:6789/0 conn(0x7f96a0099a70 legacy=0x7f96a009a1f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:50.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:50.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 1 monclient: found mon.b 2024-09-16T14:45:50.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:50.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:50.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a013b820 con 0x7f96a009e6d0 2024-09-16T14:45:50.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:50.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:50.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:50.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 5 monclient: authenticate success, global_id 94303 2024-09-16T14:45:50.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 10 monclient: _renew_subs 2024-09-16T14:45:50.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:50.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 1 -- 172.21.15.189:0/3892980216 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a009b010 con 0x7f96a009e6d0 2024-09-16T14:45:50.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 10 monclient: _renew_subs 2024-09-16T14:45:50.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:50.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a8630640 1 -- 172.21.15.189:0/3892980216 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a006f8f0 con 0x7f96a009e6d0 2024-09-16T14:45:50.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9698002010 con 0x7f96a009e6d0 2024-09-16T14:45:50.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:50.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f96a4bc9640 4 set_mon_vals no callback set 2024-09-16T14:45:50.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9698003e70 con 0x7f96a009e6d0 2024-09-16T14:45:50.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:50.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:50.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.660+0000 7f9696ffd640 10 monclient: dump: 2024-09-16T14:45:50.681 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:50.681 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:50.681 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:50.681 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:50.681 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:50.681 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:50.681 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:50.681 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:50.682 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:50.682 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:50.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.664+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9698004050 con 0x7f96a009e6d0 2024-09-16T14:45:50.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.664+0000 7f96a8630640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:50.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.664+0000 7f96a8630640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:50.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.664+0000 7f96a8630640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:50.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.664+0000 7f96a8630640 1 -- 172.21.15.189:0/3892980216 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9688005050 con 0x7f96a009e6d0 2024-09-16T14:45:50.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.664+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(68..68 src has 15..68) v4 ==== 4682+0+0 (unknown 1382858372 0 0) 0x7f9698043950 con 0x7f96a009e6d0 2024-09-16T14:45:50.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.664+0000 7f96a63cc640 10 monclient: get_auth_request con 0x7f96840360e0 auth_method 0 2024-09-16T14:45:50.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.668+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9698014950 con 0x7f96a009e6d0 2024-09-16T14:45:50.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.668+0000 7f9696ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:50.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.668+0000 7f9696ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:50.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.832+0000 7f96a8630640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T14:45:50.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.832+0000 7f96a8630640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T14:45:50.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.832+0000 7f96a8630640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:50.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:50.832+0000 7f96a8630640 1 -- 172.21.15.189:0/3892980216 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f9688005880 con 0x7f96a009e6d0 2024-09-16T14:45:51.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.748+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 add item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo} to crush map v69) v1 ==== 203+0+0 (unknown 1034282244 0 0) 0x7f9698012ff0 con 0x7f96a009e6d0 2024-09-16T14:45:51.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.748+0000 7f9696ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T14:45:51.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.748+0000 7f9696ffd640 10 monclient: _finish_command 2 = system:0 add item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo} to crush map 2024-09-16T14:45:51.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.844+0000 7f96a8630640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T14:45:51.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.844+0000 7f96a8630640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T14:45:51.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.844+0000 7f96a8630640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:51.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.844+0000 7f96a8630640 1 -- 172.21.15.189:0/3892980216 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f9688005300 con 0x7f96a009e6d0 2024-09-16T14:45:51.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.848+0000 7f9696ffd640 1 -- 172.21.15.189:0/3892980216 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v69) v1 ==== 201+0+0 (unknown 2184306352 0 0) 0x7f9698014770 con 0x7f96a009e6d0 2024-09-16T14:45:51.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.848+0000 7f9696ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T14:45:51.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.848+0000 7f9696ffd640 10 monclient: _finish_command 3 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T14:45:51.850 INFO:tasks.workunit.client.0.smithi189.stderr:set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T14:45:51.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.848+0000 7f96a8630640 1 -- 172.21.15.189:0/3892980216 >> v1:172.21.15.189:6812/3196161582 conn(0x7f96840360e0 legacy=0x7f96840385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:51.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.848+0000 7f96a8630640 10 monclient: shutdown 2024-09-16T14:45:51.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.848+0000 7f96a8630640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:51.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.848+0000 7f96a8630640 1 -- 172.21.15.189:0/3892980216 >> v1:172.21.15.189:6789/0 conn(0x7f96a009e6d0 legacy=0x7f96a01d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:51.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.848+0000 7f96a8630640 1 -- 172.21.15.189:0/3892980216 shutdown_connections 2024-09-16T14:45:51.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.848+0000 7f96a8630640 1 -- 172.21.15.189:0/3892980216 >> 172.21.15.189:0/3892980216 conn(0x7f96a00a3380 msgr2=0x7f96a00a0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:51.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.848+0000 7f96a8630640 1 -- 172.21.15.189:0/3892980216 shutdown_connections 2024-09-16T14:45:51.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.848+0000 7f96a8630640 1 -- 172.21.15.189:0/3892980216 wait complete. 2024-09-16T14:45:51.866 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add 7 1 host=host1 root=foo 2024-09-16T14:45:51.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eb55be640 10 monclient: get_monmap_and_config 2024-09-16T14:45:51.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eb55be640 10 monclient: build_initial_monmap 2024-09-16T14:45:51.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eb55be640 10 monclient: monmap: 2024-09-16T14:45:51.987 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:51.987 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:51.987 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:51.986883+0000 2024-09-16T14:45:51.987 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:51.986883+0000 2024-09-16T14:45:51.987 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:51.987 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:51.987 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:51.987 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:51.988 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:51.988 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:51.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eb55be640 1 Processor -- start 2024-09-16T14:45:51.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eb55be640 1 -- start start 2024-09-16T14:45:51.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eb55be640 10 monclient: init 2024-09-16T14:45:51.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eb55be640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:51.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eb55be640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:51.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eb55be640 10 monclient(hunting): picked mon.noname-b con 0x7f9eb0131630 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:51.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eb55be640 10 monclient(hunting): picked mon.noname-c con 0x7f9eb009edd0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:51.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eb55be640 10 monclient(hunting): picked mon.noname-a con 0x7f9eb009e950 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:51.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eb55be640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb009b920 con 0x7f9eb009e950 2024-09-16T14:45:51.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eb55be640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb009bc00 con 0x7f9eb009edd0 2024-09-16T14:45:51.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eb55be640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb009cba0 con 0x7f9eb0131630 2024-09-16T14:45:51.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eb55be640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:51.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eb55be640 10 monclient(hunting): authenticate will time out at 1134.103370s 2024-09-16T14:45:51.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9ea7fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9eb009edd0 0x7f9eb0136b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46974/0 (socket says 172.21.15.189:46974) 2024-09-16T14:45:51.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eaf7fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f9eb009e950 legacy=0x7f9eb013a290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:51.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9ea7fff640 1 -- 172.21.15.189:0/2903437356 learned_addr learned my addr 172.21.15.189:0/2903437356 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:51.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eae7fc640 1 -- 172.21.15.189:0/2903437356 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345043241 0 0) 0x7f9eb009bc00 con 0x7f9eb009edd0 2024-09-16T14:45:51.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eae7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:51.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:51.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:51.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eae7fc640 10 monclient(hunting): my global_id is 94306 2024-09-16T14:45:51.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.984+0000 7f9eae7fc640 1 -- 172.21.15.189:0/2903437356 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e9c003530 con 0x7f9eb009edd0 2024-09-16T14:45:51.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 1 -- 172.21.15.189:0/2903437356 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3412860892 0 0) 0x7f9e9c003530 con 0x7f9eb009edd0 2024-09-16T14:45:51.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 1 -- 172.21.15.189:0/2903437356 >> v1:172.21.15.26:6790/0 conn(0x7f9eb0131630 legacy=0x7f9eb0131a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:51.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 1 -- 172.21.15.189:0/2903437356 >> v1:172.21.15.26:6789/0 conn(0x7f9eb009e950 legacy=0x7f9eb013a290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:51.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:45:51.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 1 monclient: found mon.noname-c 2024-09-16T14:45:51.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:51.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:51.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 1 -- 172.21.15.189:0/2903437356 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb009b420 con 0x7f9eb009edd0 2024-09-16T14:45:51.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 10 monclient: _finish_auth 0 2024-09-16T14:45:51.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:45:51.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:51.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 5 monclient: authenticate success, global_id 94306 2024-09-16T14:45:51.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:51.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 1 -- 172.21.15.189:0/2903437356 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e98001620 con 0x7f9eb009edd0 2024-09-16T14:45:51.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:51.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:51.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 10 monclient: dump: 2024-09-16T14:45:51.994 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:51.994 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:51.994 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:51.994 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:51.994 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:51.994 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:51.994 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:51.995 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:51.995 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:51.995 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:51.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:51.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 1 -- 172.21.15.189:0/2903437356 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e98001940 con 0x7f9eb009edd0 2024-09-16T14:45:51.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:51.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 1 -- 172.21.15.189:0/2903437356 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e98001cd0 con 0x7f9eb009edd0 2024-09-16T14:45:51.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:51.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:51.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eae7fc640 10 monclient: dump: 2024-09-16T14:45:51.995 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:51.995 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:51.995 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:51.995 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:51.996 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:51.996 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:51.996 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:51.996 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:51.996 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:51.996 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:51.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:51.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 4 set_mon_vals no callback set 2024-09-16T14:45:51.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient: shutdown 2024-09-16T14:45:51.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:51.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 1 -- 172.21.15.189:0/2903437356 >> v1:172.21.15.189:6789/0 conn(0x7f9eb009edd0 legacy=0x7f9eb0136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:51.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 1 -- 172.21.15.189:0/2903437356 shutdown_connections 2024-09-16T14:45:51.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 1 -- 172.21.15.189:0/2903437356 >> 172.21.15.189:0/2903437356 conn(0x7f9eb00a3380 msgr2=0x7f9eb009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:51.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 1 -- 172.21.15.189:0/2903437356 shutdown_connections 2024-09-16T14:45:51.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 1 -- 172.21.15.189:0/2903437356 wait complete. 2024-09-16T14:45:51.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient: build_initial_monmap 2024-09-16T14:45:51.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient: monmap: 2024-09-16T14:45:51.997 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:51.997 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:51.997 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:51.997 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:51.998 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:51.998 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:51.998 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:51.998 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:51.998 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:51.998 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:51.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 1 Processor -- start 2024-09-16T14:45:51.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 1 -- start start 2024-09-16T14:45:51.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient: init 2024-09-16T14:45:52.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:52.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:52.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient(hunting): picked mon.noname-a con 0x7f9eb009e950 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:52.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient(hunting): picked mon.noname-c con 0x7f9eb009edd0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:52.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient(hunting): picked mon.noname-b con 0x7f9eb0131630 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:52.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb011f6b0 con 0x7f9eb009e950 2024-09-16T14:45:52.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb011f910 con 0x7f9eb0131630 2024-09-16T14:45:52.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb00a1da0 con 0x7f9eb009edd0 2024-09-16T14:45:52.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:52.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient(hunting): authenticate will time out at 1134.106425s 2024-09-16T14:45:52.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eaeffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f9eb009e950 legacy=0x7f9eb011e890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:52.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eaf7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9eb0131630 0x7f9eb01cf9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46982/0 (socket says 172.21.15.189:46982) 2024-09-16T14:45:52.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eaf7fe640 1 -- 172.21.15.189:0/3359175825 learned_addr learned my addr 172.21.15.189:0/3359175825 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:52.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179266627 0 0) 0x7f9eb011f910 con 0x7f9eb0131630 2024-09-16T14:45:52.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:45:52.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:52.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:52.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient(hunting): my global_id is 94309 2024-09-16T14:45:52.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e90003530 con 0x7f9eb0131630 2024-09-16T14:45:52.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 817158363 0 0) 0x7f9e90003530 con 0x7f9eb0131630 2024-09-16T14:45:52.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9eb011f910 con 0x7f9eb0131630 2024-09-16T14:45:52.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9ea80043d0 con 0x7f9eb0131630 2024-09-16T14:45:52.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:52.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:52.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient(hunting): dump: 2024-09-16T14:45:52.003 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:52.004 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:52.004 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:52.004 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:52.004 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:52.004 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:52.004 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:52.004 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:52.004 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:52.004 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:52.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:52.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831577584 0 0) 0x7f9eb00a1da0 con 0x7f9eb009edd0 2024-09-16T14:45:52.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:45:52.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:52.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:52.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient(hunting): my global_id is 5195 2024-09-16T14:45:52.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e90003530 con 0x7f9eb009edd0 2024-09-16T14:45:52.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2257509981 0 0) 0x7f9eb011f910 con 0x7f9eb0131630 2024-09-16T14:45:52.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 >> v1:172.21.15.26:6790/0 conn(0x7f9eb009edd0 legacy=0x7f9eb011efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:52.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 >> v1:172.21.15.26:6789/0 conn(0x7f9eb009e950 legacy=0x7f9eb011e890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:52.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:45:52.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 1 monclient: found mon.b 2024-09-16T14:45:52.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:52.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:52.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb006b320 con 0x7f9eb0131630 2024-09-16T14:45:52.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient: _finish_auth 0 2024-09-16T14:45:52.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:52.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:52.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 5 monclient: authenticate success, global_id 94309 2024-09-16T14:45:52.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient: _renew_subs 2024-09-16T14:45:52.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:52.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 1 -- 172.21.15.189:0/3359175825 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb0121480 con 0x7f9eb0131630 2024-09-16T14:45:52.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient: _renew_subs 2024-09-16T14:45:52.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:52.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 1 -- 172.21.15.189:0/3359175825 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb011fd50 con 0x7f9eb0131630 2024-09-16T14:45:52.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ea8004680 con 0x7f9eb0131630 2024-09-16T14:45:52.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:52.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9ea8004a10 con 0x7f9eb0131630 2024-09-16T14:45:52.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:52.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:52.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eadffb640 4 set_mon_vals no callback set 2024-09-16T14:45:52.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9ea77fe640 10 monclient: dump: 2024-09-16T14:45:52.007 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:52.007 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:52.007 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:52.007 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:52.007 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:52.007 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:52.008 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:52.008 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:52.008 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:52.008 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:52.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:52.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:52.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:52.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.988+0000 7f9eb55be640 1 -- 172.21.15.189:0/3359175825 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e94005050 con 0x7f9eb0131630 2024-09-16T14:45:52.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.992+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9ea8004b70 con 0x7f9eb0131630 2024-09-16T14:45:52.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.992+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(69..69 src has 15..69) v4 ==== 4964+0+0 (unknown 100915928 0 0) 0x7f9ea80439f0 con 0x7f9eb0131630 2024-09-16T14:45:52.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.992+0000 7f9eaeffd640 10 monclient: get_auth_request con 0x7f9e90036130 auth_method 0 2024-09-16T14:45:52.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.996+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9ea8005370 con 0x7f9eb0131630 2024-09-16T14:45:52.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.996+0000 7f9ea77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:52.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:51.996+0000 7f9ea77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:52.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.144+0000 7f9eb55be640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T14:45:52.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.144+0000 7f9eb55be640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T14:45:52.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.144+0000 7f9eb55be640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:52.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.144+0000 7f9eb55be640 1 -- 172.21.15.189:0/3359175825 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f9e94005880 con 0x7f9eb0131630 2024-09-16T14:45:52.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.144+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v69) v1 ==== 201+0+0 (unknown 2184306352 0 0) 0x7f9ea8012e40 con 0x7f9eb0131630 2024-09-16T14:45:52.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.144+0000 7f9ea77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T14:45:52.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.144+0000 7f9ea77fe640 10 monclient: _finish_command 2 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T14:45:52.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.232+0000 7f9eb55be640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T14:45:52.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.232+0000 7f9eb55be640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T14:45:52.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.232+0000 7f9eb55be640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:52.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.232+0000 7f9eb55be640 1 -- 172.21.15.189:0/3359175825 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f9e94005300 con 0x7f9eb0131630 2024-09-16T14:45:52.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.236+0000 7f9ea77fe640 1 -- 172.21.15.189:0/3359175825 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v69) v1 ==== 201+0+0 (unknown 2184306352 0 0) 0x7f9ea80147a0 con 0x7f9eb0131630 2024-09-16T14:45:52.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.236+0000 7f9ea77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T14:45:52.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.236+0000 7f9ea77fe640 10 monclient: _finish_command 3 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T14:45:52.239 INFO:tasks.workunit.client.0.smithi189.stderr:set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T14:45:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.236+0000 7f9eb55be640 1 -- 172.21.15.189:0/3359175825 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9e90036130 legacy=0x7f9e900385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.236+0000 7f9eb55be640 10 monclient: shutdown 2024-09-16T14:45:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.236+0000 7f9eb55be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.236+0000 7f9eb55be640 1 -- 172.21.15.189:0/3359175825 >> v1:172.21.15.189:6789/0 conn(0x7f9eb0131630 legacy=0x7f9eb01cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.236+0000 7f9eb55be640 1 -- 172.21.15.189:0/3359175825 shutdown_connections 2024-09-16T14:45:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.236+0000 7f9eb55be640 1 -- 172.21.15.189:0/3359175825 >> 172.21.15.189:0/3359175825 conn(0x7f9eb00a3380 msgr2=0x7f9eb009a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.236+0000 7f9eb55be640 1 -- 172.21.15.189:0/3359175825 shutdown_connections 2024-09-16T14:45:52.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.236+0000 7f9eb55be640 1 -- 172.21.15.189:0/3359175825 wait complete. 2024-09-16T14:45:52.254 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add 9 1 host=host2 root=foo 2024-09-16T14:45:52.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6f18f6640 10 monclient: get_monmap_and_config 2024-09-16T14:45:52.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6f18f6640 10 monclient: build_initial_monmap 2024-09-16T14:45:52.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6f18f6640 10 monclient: monmap: 2024-09-16T14:45:52.372 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:52.372 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:52.373 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:52.372209+0000 2024-09-16T14:45:52.373 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:52.372209+0000 2024-09-16T14:45:52.373 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:52.373 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:52.373 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:52.373 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:52.373 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:52.373 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:52.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6f18f6640 1 Processor -- start 2024-09-16T14:45:52.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6f18f6640 1 -- start start 2024-09-16T14:45:52.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6f18f6640 10 monclient: init 2024-09-16T14:45:52.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6f18f6640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:52.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6f18f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:52.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6f18f6640 10 monclient(hunting): picked mon.noname-a con 0x7ff6ec099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:52.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6f18f6640 10 monclient(hunting): picked mon.noname-b con 0x7ff6ec09c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:52.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6f18f6640 10 monclient(hunting): picked mon.noname-c con 0x7ff6ec09e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:52.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6f18f6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec099e50 con 0x7ff6ec099a70 2024-09-16T14:45:52.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6f18f6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec09dbd0 con 0x7ff6ec09e6d0 2024-09-16T14:45:52.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6f18f6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec09ee20 con 0x7ff6ec09c550 2024-09-16T14:45:52.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6f18f6640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:52.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6f18f6640 10 monclient(hunting): authenticate will time out at 1134.487865s 2024-09-16T14:45:52.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.368+0000 7ff6eaffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7ff6ec099a70 legacy=0x7ff6ec09dfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:52.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6eb7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff6ec09e6d0 0x7ff6ec13b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46986/0 (socket says 172.21.15.189:46986) 2024-09-16T14:45:52.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6eb7fe640 1 -- 172.21.15.189:0/3991277908 learned_addr learned my addr 172.21.15.189:0/3991277908 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:52.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 1 -- 172.21.15.189:0/3991277908 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8870803 0 0) 0x7ff6ec09dbd0 con 0x7ff6ec09e6d0 2024-09-16T14:45:52.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:52.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:52.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:52.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 10 monclient(hunting): my global_id is 94312 2024-09-16T14:45:52.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 1 -- 172.21.15.189:0/3991277908 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d4003530 con 0x7ff6ec09e6d0 2024-09-16T14:45:52.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 1 -- 172.21.15.189:0/3991277908 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3041294608 0 0) 0x7ff6d4003530 con 0x7ff6ec09e6d0 2024-09-16T14:45:52.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 1 -- 172.21.15.189:0/3991277908 >> v1:172.21.15.26:6790/0 conn(0x7ff6ec09c550 legacy=0x7ff6ec1379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:52.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 1 -- 172.21.15.189:0/3991277908 >> v1:172.21.15.26:6789/0 conn(0x7ff6ec099a70 legacy=0x7ff6ec09dfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:52.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:45:52.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 1 monclient: found mon.noname-c 2024-09-16T14:45:52.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:52.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:52.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 1 -- 172.21.15.189:0/3991277908 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ec09f100 con 0x7ff6ec09e6d0 2024-09-16T14:45:52.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 10 monclient: _finish_auth 0 2024-09-16T14:45:52.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:52.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:52.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 1 -- 172.21.15.189:0/3991277908 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6e40019a0 con 0x7ff6ec09e6d0 2024-09-16T14:45:52.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:52.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:52.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 10 monclient: dump: 2024-09-16T14:45:52.383 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:52.383 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:52.383 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:52.383 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:52.383 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:52.383 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:52.383 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:52.383 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:52.383 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:52.383 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:52.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 5 monclient: authenticate success, global_id 94312 2024-09-16T14:45:52.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:52.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 1 -- 172.21.15.189:0/3991277908 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6e4001c50 con 0x7ff6ec09e6d0 2024-09-16T14:45:52.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:52.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 1 -- 172.21.15.189:0/3991277908 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6e4001fe0 con 0x7ff6ec09e6d0 2024-09-16T14:45:52.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:52.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:52.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e9ffb640 10 monclient: dump: 2024-09-16T14:45:52.384 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:52.384 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:52.384 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:52.384 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:52.384 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:52.384 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:52.384 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:52.385 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:52.385 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:52.385 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:52.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:52.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 4 set_mon_vals no callback set 2024-09-16T14:45:52.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 10 monclient: shutdown 2024-09-16T14:45:52.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:52.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 1 -- 172.21.15.189:0/3991277908 >> v1:172.21.15.189:6789/0 conn(0x7ff6ec09e6d0 legacy=0x7ff6ec13b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:52.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 1 -- 172.21.15.189:0/3991277908 shutdown_connections 2024-09-16T14:45:52.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 1 -- 172.21.15.189:0/3991277908 >> 172.21.15.189:0/3991277908 conn(0x7ff6ec0a3380 msgr2=0x7ff6ec09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:52.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 1 -- 172.21.15.189:0/3991277908 shutdown_connections 2024-09-16T14:45:52.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 1 -- 172.21.15.189:0/3991277908 wait complete. 2024-09-16T14:45:52.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 10 monclient: build_initial_monmap 2024-09-16T14:45:52.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 10 monclient: monmap: 2024-09-16T14:45:52.386 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:52.386 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:52.386 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:52.386 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:52.386 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:52.386 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:52.386 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:52.386 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:52.386 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:52.387 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:52.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 1 Processor -- start 2024-09-16T14:45:52.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 1 -- start start 2024-09-16T14:45:52.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 10 monclient: init 2024-09-16T14:45:52.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:52.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:52.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 10 monclient(hunting): picked mon.noname-c con 0x7ff6ec099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:52.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 10 monclient(hunting): picked mon.noname-b con 0x7ff6ec09c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:52.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 10 monclient(hunting): picked mon.noname-a con 0x7ff6ec09e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:52.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec09cf20 con 0x7ff6ec09e6d0 2024-09-16T14:45:52.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec1220b0 con 0x7ff6ec09c550 2024-09-16T14:45:52.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec06b320 con 0x7ff6ec099a70 2024-09-16T14:45:52.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:52.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 10 monclient(hunting): authenticate will time out at 1134.490838s 2024-09-16T14:45:52.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6ea7fc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff6ec09c550 0x7ff6ec1201c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46992/0 (socket says 172.21.15.189:46992) 2024-09-16T14:45:52.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6ea7fc640 1 -- 172.21.15.189:0/3386569526 learned_addr learned my addr 172.21.15.189:0/3386569526 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:52.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6eb7fe640 1 -- 172.21.15.189:0/3386569526 >> v1:172.21.15.26:6789/0 conn(0x7ff6ec09e6d0 legacy=0x7ff6ec1208d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:52.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748955757 0 0) 0x7ff6ec1220b0 con 0x7ff6ec09c550 2024-09-16T14:45:52.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:52.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:52.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:52.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient(hunting): my global_id is 94315 2024-09-16T14:45:52.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d8003530 con 0x7ff6ec09c550 2024-09-16T14:45:52.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3636675237 0 0) 0x7ff6d8003530 con 0x7ff6ec09c550 2024-09-16T14:45:52.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6ec1220b0 con 0x7ff6ec09c550 2024-09-16T14:45:52.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6dc001950 con 0x7ff6ec09c550 2024-09-16T14:45:52.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:52.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:52.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient(hunting): dump: 2024-09-16T14:45:52.389 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:52.389 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:52.389 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:52.389 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:52.389 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:52.389 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:52.389 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:52.389 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:52.389 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:52.390 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:52.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:52.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278781826 0 0) 0x7ff6ec06b320 con 0x7ff6ec099a70 2024-09-16T14:45:52.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:52.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:52.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:52.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient(hunting): my global_id is 5201 2024-09-16T14:45:52.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d8003530 con 0x7ff6ec099a70 2024-09-16T14:45:52.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1352328886 0 0) 0x7ff6ec1220b0 con 0x7ff6ec09c550 2024-09-16T14:45:52.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 >> v1:172.21.15.26:6790/0 conn(0x7ff6ec099a70 legacy=0x7ff6ec11fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:52.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 >> v1:172.21.15.26:6789/0 conn(0x7ff6ec09e6d0 legacy=0x7ff6ec1208d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:52.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:45:52.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 1 monclient: found mon.b 2024-09-16T14:45:52.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:52.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:52.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ec06ce10 con 0x7ff6ec09c550 2024-09-16T14:45:52.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient: _finish_auth 0 2024-09-16T14:45:52.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:45:52.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:52.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 5 monclient: authenticate success, global_id 94315 2024-09-16T14:45:52.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 10 monclient: _renew_subs 2024-09-16T14:45:52.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:52.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 1 -- 172.21.15.189:0/3386569526 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6ec06f940 con 0x7ff6ec09c550 2024-09-16T14:45:52.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 10 monclient: _renew_subs 2024-09-16T14:45:52.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:52.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6f18f6640 1 -- 172.21.15.189:0/3386569526 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6ec123170 con 0x7ff6ec09c550 2024-09-16T14:45:52.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6dc001d20 con 0x7ff6ec09c550 2024-09-16T14:45:52.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:52.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6dc003af0 con 0x7ff6ec09c550 2024-09-16T14:45:52.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:52.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:52.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6e97fa640 4 set_mon_vals no callback set 2024-09-16T14:45:52.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.372+0000 7ff6d3fff640 10 monclient: dump: 2024-09-16T14:45:52.392 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:52.392 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:52.392 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:52.392 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:52.392 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:52.392 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:52.392 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:52.393 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:52.393 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:52.393 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:52.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.376+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff6dc003cd0 con 0x7ff6ec09c550 2024-09-16T14:45:52.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.376+0000 7ff6f18f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:52.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.376+0000 7ff6f18f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:52.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.376+0000 7ff6f18f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:52.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.376+0000 7ff6f18f6640 1 -- 172.21.15.189:0/3386569526 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6cc005050 con 0x7ff6ec09c550 2024-09-16T14:45:52.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.376+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(69..69 src has 15..69) v4 ==== 4964+0+0 (unknown 100915928 0 0) 0x7ff6dc043870 con 0x7ff6ec09c550 2024-09-16T14:45:52.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.376+0000 7ff6eaffd640 10 monclient: get_auth_request con 0x7ff6d80361d0 auth_method 0 2024-09-16T14:45:52.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.380+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff6dc014670 con 0x7ff6ec09c550 2024-09-16T14:45:52.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.380+0000 7ff6d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:52.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.380+0000 7ff6d3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:52.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.532+0000 7ff6f18f6640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T14:45:52.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.532+0000 7ff6f18f6640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T14:45:52.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.532+0000 7ff6f18f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:52.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.532+0000 7ff6f18f6640 1 -- 172.21.15.189:0/3386569526 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7ff6cc005880 con 0x7ff6ec09c550 2024-09-16T14:45:52.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.752+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 add item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo} to crush map v70) v1 ==== 203+0+0 (unknown 934216173 0 0) 0x7ff6dc014490 con 0x7ff6ec09c550 2024-09-16T14:45:52.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.752+0000 7ff6d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T14:45:52.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.752+0000 7ff6d3fff640 10 monclient: _finish_command 2 = system:0 add item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo} to crush map 2024-09-16T14:45:52.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.844+0000 7ff6f18f6640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T14:45:52.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.844+0000 7ff6f18f6640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T14:45:52.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.844+0000 7ff6f18f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:52.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.844+0000 7ff6f18f6640 1 -- 172.21.15.189:0/3386569526 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7ff6cc005ac0 con 0x7ff6ec09c550 2024-09-16T14:45:52.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.848+0000 7ff6d3fff640 1 -- 172.21.15.189:0/3386569526 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v70) v1 ==== 201+0+0 (unknown 257434170 0 0) 0x7ff6dc014ad0 con 0x7ff6ec09c550 2024-09-16T14:45:52.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.848+0000 7ff6d3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T14:45:52.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.848+0000 7ff6d3fff640 10 monclient: _finish_command 3 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T14:45:52.850 INFO:tasks.workunit.client.0.smithi189.stderr:set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T14:45:52.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.848+0000 7ff6f18f6640 1 -- 172.21.15.189:0/3386569526 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff6d80361d0 legacy=0x7ff6d8038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:52.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.848+0000 7ff6f18f6640 10 monclient: shutdown 2024-09-16T14:45:52.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.848+0000 7ff6f18f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:52.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.848+0000 7ff6f18f6640 1 -- 172.21.15.189:0/3386569526 >> v1:172.21.15.189:6789/0 conn(0x7ff6ec09c550 legacy=0x7ff6ec1201c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:52.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.848+0000 7ff6f18f6640 1 -- 172.21.15.189:0/3386569526 shutdown_connections 2024-09-16T14:45:52.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.848+0000 7ff6f18f6640 1 -- 172.21.15.189:0/3386569526 >> 172.21.15.189:0/3386569526 conn(0x7ff6ec0a3380 msgr2=0x7ff6ec06feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:52.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.848+0000 7ff6f18f6640 1 -- 172.21.15.189:0/3386569526 shutdown_connections 2024-09-16T14:45:52.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.848+0000 7ff6f18f6640 1 -- 172.21.15.189:0/3386569526 wait complete. 2024-09-16T14:45:52.866 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add 9 1 host=host2 root=foo 2024-09-16T14:45:52.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.980+0000 7ff86473d640 10 monclient: get_monmap_and_config 2024-09-16T14:45:52.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.980+0000 7ff86473d640 10 monclient: build_initial_monmap 2024-09-16T14:45:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.980+0000 7ff86473d640 10 monclient: monmap: 2024-09-16T14:45:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:52.984639+0000 2024-09-16T14:45:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:52.984639+0000 2024-09-16T14:45:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:52.985 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:52.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.980+0000 7ff86473d640 1 Processor -- start 2024-09-16T14:45:52.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.980+0000 7ff86473d640 1 -- start start 2024-09-16T14:45:52.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.980+0000 7ff86473d640 10 monclient: init 2024-09-16T14:45:52.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:52.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:52.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient(hunting): picked mon.noname-c con 0x7ff85c0a1a90 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:52.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient(hunting): picked mon.noname-b con 0x7ff85c0a2580 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:52.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient(hunting): picked mon.noname-a con 0x7ff85c09fd40 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:52.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c09df00 con 0x7ff85c09fd40 2024-09-16T14:45:52.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c09f0a0 con 0x7ff85c0a1a90 2024-09-16T14:45:52.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c0a3040 con 0x7ff85c0a2580 2024-09-16T14:45:52.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:52.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient(hunting): authenticate will time out at 1135.100539s 2024-09-16T14:45:52.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff862cda640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7ff85c09fd40 legacy=0x7ff85c13d520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:52.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8624d9640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff85c0a1a90 0x7ff85c0a1e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47006/0 (socket says 172.21.15.189:47006) 2024-09-16T14:45:52.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8624d9640 1 -- 172.21.15.189:0/1587968390 learned_addr learned my addr 172.21.15.189:0/1587968390 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:52.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 1 -- 172.21.15.189:0/1587968390 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968807017 0 0) 0x7ff85c09f0a0 con 0x7ff85c0a1a90 2024-09-16T14:45:52.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 10 monclient(hunting): _negotiate 2024-09-16T14:45:52.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:52.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 10 monclient(hunting): my global_id is 94318 2024-09-16T14:45:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 1 -- 172.21.15.189:0/1587968390 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff848003530 con 0x7ff85c0a1a90 2024-09-16T14:45:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 1 -- 172.21.15.189:0/1587968390 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2783505312 0 0) 0x7ff848003530 con 0x7ff85c0a1a90 2024-09-16T14:45:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 1 -- 172.21.15.189:0/1587968390 >> v1:172.21.15.26:6790/0 conn(0x7ff85c0a2580 legacy=0x7ff85c139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 1 -- 172.21.15.189:0/1587968390 >> v1:172.21.15.26:6789/0 conn(0x7ff85c09fd40 legacy=0x7ff85c13d520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 10 monclient: _finish_hunting 0 2024-09-16T14:45:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 1 monclient: found mon.noname-c 2024-09-16T14:45:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 1 -- 172.21.15.189:0/1587968390 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85c09eae0 con 0x7ff85c0a1a90 2024-09-16T14:45:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 10 monclient: _finish_auth 0 2024-09-16T14:45:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 10 monclient: _check_auth_tickets 2024-09-16T14:45:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 1 -- 172.21.15.189:0/1587968390 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff858001620 con 0x7ff85c0a1a90 2024-09-16T14:45:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 10 monclient: dump: 2024-09-16T14:45:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:52.992 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:52.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 5 monclient: authenticate success, global_id 94318 2024-09-16T14:45:52.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:52.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 1 -- 172.21.15.189:0/1587968390 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8580018d0 con 0x7ff85c0a1a90 2024-09-16T14:45:52.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:52.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:52.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 4 set_mon_vals no callback set 2024-09-16T14:45:52.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 1 -- 172.21.15.189:0/1587968390 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff858001c60 con 0x7ff85c0a1a90 2024-09-16T14:45:52.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:52.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:52.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8614d7640 10 monclient: dump: 2024-09-16T14:45:52.992 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:52.993 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:52.993 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:52.993 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:52.993 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:52.993 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:52.993 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:52.993 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:52.993 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:52.993 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:52.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient: shutdown 2024-09-16T14:45:52.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:52.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 1 -- 172.21.15.189:0/1587968390 >> v1:172.21.15.189:6789/0 conn(0x7ff85c0a1a90 legacy=0x7ff85c0a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:52.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 1 -- 172.21.15.189:0/1587968390 shutdown_connections 2024-09-16T14:45:52.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 1 -- 172.21.15.189:0/1587968390 >> 172.21.15.189:0/1587968390 conn(0x7ff85c0a7a70 msgr2=0x7ff85c0a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:52.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 1 -- 172.21.15.189:0/1587968390 shutdown_connections 2024-09-16T14:45:52.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 1 -- 172.21.15.189:0/1587968390 wait complete. 2024-09-16T14:45:52.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient: build_initial_monmap 2024-09-16T14:45:52.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient: monmap: 2024-09-16T14:45:52.994 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:52.994 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:52.994 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:52.994 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:52.994 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:52.994 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:52.994 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:52.994 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:52.995 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:52.995 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:52.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 1 Processor -- start 2024-09-16T14:45:52.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 1 -- start start 2024-09-16T14:45:52.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient: init 2024-09-16T14:45:52.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:52.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:52.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient(hunting): picked mon.noname-a con 0x7ff85c09fd40 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:52.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient(hunting): picked mon.noname-c con 0x7ff85c0a1a90 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:52.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient(hunting): picked mon.noname-b con 0x7ff85c0a2580 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:52.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c1246a0 con 0x7ff85c09fd40 2024-09-16T14:45:52.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c09e280 con 0x7ff85c0a2580 2024-09-16T14:45:52.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c0a63c0 con 0x7ff85c0a1a90 2024-09-16T14:45:52.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:52.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff86473d640 10 monclient(hunting): authenticate will time out at 1135.103514s 2024-09-16T14:45:52.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff8624d9640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7ff85c09fd40 legacy=0x7ff85c133cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:52.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff862cda640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff85c0a2580 0x7ff85c1d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47022/0 (socket says 172.21.15.189:47022) 2024-09-16T14:45:52.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff862cda640 1 -- 172.21.15.189:0/4189299617 learned_addr learned my addr 172.21.15.189:0/4189299617 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:52.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966945931 0 0) 0x7ff85c09e280 con 0x7ff85c0a2580 2024-09-16T14:45:53.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff852ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:53.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff852ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:53.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:53.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff852ffd640 10 monclient(hunting): my global_id is 94321 2024-09-16T14:45:53.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.984+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff840003530 con 0x7ff85c0a2580 2024-09-16T14:45:53.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2075312596 0 0) 0x7ff840003530 con 0x7ff85c0a2580 2024-09-16T14:45:53.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff85c09e280 con 0x7ff85c0a2580 2024-09-16T14:45:53.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff854004450 con 0x7ff85c0a2580 2024-09-16T14:45:53.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:53.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:53.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 10 monclient(hunting): dump: 2024-09-16T14:45:53.000 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:53.001 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:53.001 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:53.001 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:53.001 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:53.001 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:53.001 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:53.001 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:53.001 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:53.001 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:53.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:53.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334679020 0 0) 0x7ff85c0a63c0 con 0x7ff85c0a1a90 2024-09-16T14:45:53.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:53.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:53.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:53.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 10 monclient(hunting): my global_id is 5207 2024-09-16T14:45:53.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff840003530 con 0x7ff85c0a1a90 2024-09-16T14:45:53.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4080646523 0 0) 0x7ff85c09e280 con 0x7ff85c0a2580 2024-09-16T14:45:53.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 >> v1:172.21.15.26:6790/0 conn(0x7ff85c0a1a90 legacy=0x7ff85c1ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:53.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 >> v1:172.21.15.26:6789/0 conn(0x7ff85c09fd40 legacy=0x7ff85c133cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:53.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:53.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 1 monclient: found mon.b 2024-09-16T14:45:53.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:53.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:53.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85c122a20 con 0x7ff85c0a2580 2024-09-16T14:45:53.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:53.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:53.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:53.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff86473d640 5 monclient: authenticate success, global_id 94321 2024-09-16T14:45:53.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff86473d640 10 monclient: _renew_subs 2024-09-16T14:45:53.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff86473d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:53.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff86473d640 1 -- 172.21.15.189:0/4189299617 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85c123e00 con 0x7ff85c0a2580 2024-09-16T14:45:53.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff86473d640 10 monclient: _renew_subs 2024-09-16T14:45:53.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff86473d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:53.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff86473d640 1 -- 172.21.15.189:0/4189299617 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85c1c91a0 con 0x7ff85c0a2580 2024-09-16T14:45:53.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff854004770 con 0x7ff85c0a2580 2024-09-16T14:45:53.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:53.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff854004b00 con 0x7ff85c0a2580 2024-09-16T14:45:53.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:53.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:53.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff853fff640 4 set_mon_vals no callback set 2024-09-16T14:45:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 10 monclient: dump: 2024-09-16T14:45:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:53.004 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff86473d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff86473d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:53.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff86473d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:53.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff854004c60 con 0x7ff85c0a2580 2024-09-16T14:45:53.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff86473d640 1 -- 172.21.15.189:0/4189299617 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff844005050 con 0x7ff85c0a2580 2024-09-16T14:45:53.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(70..70 src has 15..70) v4 ==== 5208+0+0 (unknown 4277616870 0 0) 0x7ff854043b10 con 0x7ff85c0a2580 2024-09-16T14:45:53.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.988+0000 7ff8624d9640 10 monclient: get_auth_request con 0x7ff840036130 auth_method 0 2024-09-16T14:45:53.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.992+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff854005460 con 0x7ff85c0a2580 2024-09-16T14:45:53.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.992+0000 7ff852ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:53.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:52.992+0000 7ff852ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:53.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.152+0000 7ff86473d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T14:45:53.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.152+0000 7ff86473d640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T14:45:53.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.152+0000 7ff86473d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:53.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.152+0000 7ff86473d640 1 -- 172.21.15.189:0/4189299617 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7ff844005880 con 0x7ff85c0a2580 2024-09-16T14:45:53.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.152+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v70) v1 ==== 201+0+0 (unknown 257434170 0 0) 0x7ff854012f20 con 0x7ff85c0a2580 2024-09-16T14:45:53.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.152+0000 7ff852ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T14:45:53.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.152+0000 7ff852ffd640 10 monclient: _finish_command 2 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T14:45:53.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.248+0000 7ff86473d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T14:45:53.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.248+0000 7ff86473d640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T14:45:53.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.248+0000 7ff86473d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:53.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.248+0000 7ff86473d640 1 -- 172.21.15.189:0/4189299617 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7ff844005300 con 0x7ff85c0a2580 2024-09-16T14:45:53.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.248+0000 7ff852ffd640 1 -- 172.21.15.189:0/4189299617 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v70) v1 ==== 201+0+0 (unknown 257434170 0 0) 0x7ff8540146a0 con 0x7ff85c0a2580 2024-09-16T14:45:53.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.248+0000 7ff852ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T14:45:53.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.248+0000 7ff852ffd640 10 monclient: _finish_command 3 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T14:45:53.254 INFO:tasks.workunit.client.0.smithi189.stderr:set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T14:45:53.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.252+0000 7ff86473d640 1 -- 172.21.15.189:0/4189299617 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff840036130 legacy=0x7ff8400385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:53.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.252+0000 7ff86473d640 10 monclient: shutdown 2024-09-16T14:45:53.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.252+0000 7ff86473d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:53.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.252+0000 7ff86473d640 1 -- 172.21.15.189:0/4189299617 >> v1:172.21.15.189:6789/0 conn(0x7ff85c0a2580 legacy=0x7ff85c1d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:53.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.252+0000 7ff86473d640 1 -- 172.21.15.189:0/4189299617 shutdown_connections 2024-09-16T14:45:53.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.252+0000 7ff86473d640 1 -- 172.21.15.189:0/4189299617 >> 172.21.15.189:0/4189299617 conn(0x7ff85c0a7a70 msgr2=0x7ff85c0a6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:53.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.252+0000 7ff86473d640 1 -- 172.21.15.189:0/4189299617 shutdown_connections 2024-09-16T14:45:53.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.252+0000 7ff86473d640 1 -- 172.21.15.189:0/4189299617 wait complete. 2024-09-16T14:45:53.269 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket bar root 2024-09-16T14:45:53.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 10 monclient: get_monmap_and_config 2024-09-16T14:45:53.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 10 monclient: build_initial_monmap 2024-09-16T14:45:53.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 10 monclient: monmap: 2024-09-16T14:45:53.386 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:53.386 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:53.386 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:53.385869+0000 2024-09-16T14:45:53.386 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:53.385869+0000 2024-09-16T14:45:53.386 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:53.386 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:53.386 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:53.386 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:53.386 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:53.386 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 1 Processor -- start 2024-09-16T14:45:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 1 -- start start 2024-09-16T14:45:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 10 monclient: init 2024-09-16T14:45:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 10 monclient(hunting): picked mon.noname-a con 0x7fe07c06c120 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 10 monclient(hunting): picked mon.noname-b con 0x7fe07c1353d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 10 monclient(hunting): picked mon.noname-c con 0x7fe07c0a1040 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07c06c500 con 0x7fe07c06c120 2024-09-16T14:45:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07c0a2ef0 con 0x7fe07c0a1040 2024-09-16T14:45:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07c09e990 con 0x7fe07c1353d0 2024-09-16T14:45:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 10 monclient(hunting): authenticate will time out at 1135.501590s 2024-09-16T14:45:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07bfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fe07c06c120 legacy=0x7fe07c134cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe08099d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe07c0a1040 0x7fe07c13d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47038/0 (socket says 172.21.15.189:47038) 2024-09-16T14:45:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe08099d640 1 -- 172.21.15.189:0/3639592700 learned_addr learned my addr 172.21.15.189:0/3639592700 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 1 -- 172.21.15.189:0/3639592700 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532951012 0 0) 0x7fe07c0a2ef0 con 0x7fe07c0a1040 2024-09-16T14:45:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 10 monclient(hunting): my global_id is 94324 2024-09-16T14:45:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 1 -- 172.21.15.189:0/3639592700 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe068003530 con 0x7fe07c0a1040 2024-09-16T14:45:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 1 -- 172.21.15.189:0/3639592700 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4096325352 0 0) 0x7fe068003530 con 0x7fe07c0a1040 2024-09-16T14:45:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 1 -- 172.21.15.189:0/3639592700 >> v1:172.21.15.26:6790/0 conn(0x7fe07c1353d0 legacy=0x7fe07c139b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 1 -- 172.21.15.189:0/3639592700 >> v1:172.21.15.26:6789/0 conn(0x7fe07c06c120 legacy=0x7fe07c134cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 1 monclient: found mon.noname-c 2024-09-16T14:45:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 1 -- 172.21.15.189:0/3639592700 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c09ec70 con 0x7fe07c0a1040 2024-09-16T14:45:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 10 monclient: _finish_auth 0 2024-09-16T14:45:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 1 -- 172.21.15.189:0/3639592700 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe064001620 con 0x7fe07c0a1040 2024-09-16T14:45:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 5 monclient: authenticate success, global_id 94324 2024-09-16T14:45:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 10 monclient: dump: 2024-09-16T14:45:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:53.397 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 1 -- 172.21.15.189:0/3639592700 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0640018d0 con 0x7fe07c0a1040 2024-09-16T14:45:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 4 set_mon_vals no callback set 2024-09-16T14:45:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 1 -- 172.21.15.189:0/3639592700 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe064001ca0 con 0x7fe07c0a1040 2024-09-16T14:45:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe07affd640 10 monclient: dump: 2024-09-16T14:45:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:53.399 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 10 monclient: shutdown 2024-09-16T14:45:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 1 -- 172.21.15.189:0/3639592700 >> v1:172.21.15.189:6789/0 conn(0x7fe07c0a1040 legacy=0x7fe07c13d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 1 -- 172.21.15.189:0/3639592700 shutdown_connections 2024-09-16T14:45:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 1 -- 172.21.15.189:0/3639592700 >> 172.21.15.189:0/3639592700 conn(0x7fe07c0a3a40 msgr2=0x7fe07c0a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 1 -- 172.21.15.189:0/3639592700 shutdown_connections 2024-09-16T14:45:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 1 -- 172.21.15.189:0/3639592700 wait complete. 2024-09-16T14:45:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 10 monclient: build_initial_monmap 2024-09-16T14:45:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.384+0000 7fe082400640 10 monclient: monmap: 2024-09-16T14:45:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:53.400 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 1 Processor -- start 2024-09-16T14:45:53.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 1 -- start start 2024-09-16T14:45:53.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 10 monclient: init 2024-09-16T14:45:53.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:53.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:45:53.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 10 monclient(hunting): picked mon.noname-c con 0x7fe07c06c120 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:53.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 10 monclient(hunting): picked mon.noname-a con 0x7fe07c0a1040 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:53.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 10 monclient(hunting): picked mon.noname-b con 0x7fe07c1353d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:53.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07c131570 con 0x7fe07c0a1040 2024-09-16T14:45:53.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07c0a52c0 con 0x7fe07c1353d0 2024-09-16T14:45:53.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07c09e130 con 0x7fe07c06c120 2024-09-16T14:45:53.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:53.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 10 monclient(hunting): authenticate will time out at 1135.504690s 2024-09-16T14:45:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe07b7fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fe07c0a1040 legacy=0x7fe07c12e490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe08099d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe07c1353d0 0x7fe07c12eba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47040/0 (socket says 172.21.15.189:47040) 2024-09-16T14:45:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe08099d640 1 -- 172.21.15.189:0/1545693498 learned_addr learned my addr 172.21.15.189:0/1545693498 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599434543 0 0) 0x7fe07c0a52c0 con 0x7fe07c1353d0 2024-09-16T14:45:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient(hunting): my global_id is 94327 2024-09-16T14:45:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe05c003530 con 0x7fe07c1353d0 2024-09-16T14:45:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 724191479 0 0) 0x7fe05c003530 con 0x7fe07c1353d0 2024-09-16T14:45:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe07c0a52c0 con 0x7fe07c1353d0 2024-09-16T14:45:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe064001be0 con 0x7fe07c1353d0 2024-09-16T14:45:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:53.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:53.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient(hunting): dump: 2024-09-16T14:45:53.403 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:53.403 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:53.403 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:53.403 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:53.403 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:53.403 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:53.403 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:53.403 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:53.403 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:53.403 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:53.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:53.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798147798 0 0) 0x7fe07c09e130 con 0x7fe07c06c120 2024-09-16T14:45:53.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:53.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:53.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:53.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient(hunting): my global_id is 5213 2024-09-16T14:45:53.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe05c003530 con 0x7fe07c06c120 2024-09-16T14:45:53.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3173250542 0 0) 0x7fe07c0a52c0 con 0x7fe07c1353d0 2024-09-16T14:45:53.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 >> v1:172.21.15.26:6790/0 conn(0x7fe07c06c120 legacy=0x7fe07c12dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:53.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 >> v1:172.21.15.26:6789/0 conn(0x7fe07c0a1040 legacy=0x7fe07c12e490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:53.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:45:53.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 1 monclient: found mon.b 2024-09-16T14:45:53.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:53.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c133710 con 0x7fe07c1353d0 2024-09-16T14:45:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient: _finish_auth 0 2024-09-16T14:45:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:45:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 5 monclient: authenticate success, global_id 94327 2024-09-16T14:45:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 10 monclient: _renew_subs 2024-09-16T14:45:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 1 -- 172.21.15.189:0/1545693498 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07c131b30 con 0x7fe07c1353d0 2024-09-16T14:45:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 10 monclient: _renew_subs 2024-09-16T14:45:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 1 -- 172.21.15.189:0/1545693498 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07c12f7e0 con 0x7fe07c1353d0 2024-09-16T14:45:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe064004c80 con 0x7fe07c1353d0 2024-09-16T14:45:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:53.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe064005010 con 0x7fe07c1353d0 2024-09-16T14:45:53.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:53.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:53.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe07a7fc640 4 set_mon_vals no callback set 2024-09-16T14:45:53.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 10 monclient: dump: 2024-09-16T14:45:53.406 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:53.406 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:53.406 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:53.406 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:53.406 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:53.406 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:53.406 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:53.406 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:53.407 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:53.407 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:53.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:53.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:53.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:53.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe082400640 1 -- 172.21.15.189:0/1545693498 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe060005050 con 0x7fe07c1353d0 2024-09-16T14:45:53.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe064005170 con 0x7fe07c1353d0 2024-09-16T14:45:53.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(70..70 src has 15..70) v4 ==== 5208+0+0 (unknown 4277616870 0 0) 0x7fe064043a50 con 0x7fe07c1353d0 2024-09-16T14:45:53.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.388+0000 7fe07bfff640 10 monclient: get_auth_request con 0x7fe05c036160 auth_method 0 2024-09-16T14:45:53.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.392+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe064014a50 con 0x7fe07c1353d0 2024-09-16T14:45:53.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.392+0000 7fe078ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:53.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.392+0000 7fe078ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:53.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.556+0000 7fe082400640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T14:45:53.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.556+0000 7fe082400640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T14:45:53.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.556+0000 7fe082400640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:53.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.556+0000 7fe082400640 1 -- 172.21.15.189:0/1545693498 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7fe060005880 con 0x7fe07c1353d0 2024-09-16T14:45:53.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.768+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 added bucket bar type root to crush map v71) v1 ==== 138+0+0 (unknown 1697214064 0 0) 0x7fe064012ec0 con 0x7fe07c1353d0 2024-09-16T14:45:53.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.768+0000 7fe078ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T14:45:53.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.768+0000 7fe078ff9640 10 monclient: _finish_command 2 = system:0 added bucket bar type root to crush map 2024-09-16T14:45:53.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.860+0000 7fe082400640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T14:45:53.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.860+0000 7fe082400640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T14:45:53.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.860+0000 7fe082400640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:53.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.860+0000 7fe082400640 1 -- 172.21.15.189:0/1545693498 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7fe060005340 con 0x7fe07c1353d0 2024-09-16T14:45:53.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.860+0000 7fe078ff9640 1 -- 172.21.15.189:0/1545693498 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v71) v1 ==== 126+0+0 (unknown 3211882315 0 0) 0x7fe064014640 con 0x7fe07c1353d0 2024-09-16T14:45:53.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.860+0000 7fe078ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T14:45:53.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.860+0000 7fe078ff9640 10 monclient: _finish_command 3 = system:0 bucket 'bar' already exists 2024-09-16T14:45:53.865 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'bar' already exists 2024-09-16T14:45:53.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.864+0000 7fe082400640 1 -- 172.21.15.189:0/1545693498 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe05c036160 legacy=0x7fe05c038620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:53.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.864+0000 7fe082400640 10 monclient: shutdown 2024-09-16T14:45:53.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.864+0000 7fe082400640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:53.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.864+0000 7fe082400640 1 -- 172.21.15.189:0/1545693498 >> v1:172.21.15.189:6789/0 conn(0x7fe07c1353d0 legacy=0x7fe07c12eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:53.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.864+0000 7fe082400640 1 -- 172.21.15.189:0/1545693498 shutdown_connections 2024-09-16T14:45:53.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.864+0000 7fe082400640 1 -- 172.21.15.189:0/1545693498 >> 172.21.15.189:0/1545693498 conn(0x7fe07c0a3a40 msgr2=0x7fe07c0a1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:53.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.864+0000 7fe082400640 1 -- 172.21.15.189:0/1545693498 shutdown_connections 2024-09-16T14:45:53.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.864+0000 7fe082400640 1 -- 172.21.15.189:0/1545693498 wait complete. 2024-09-16T14:45:53.881 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket bar root 2024-09-16T14:45:53.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.992+0000 7fbca0fe8640 10 monclient: get_monmap_and_config 2024-09-16T14:45:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.992+0000 7fbca0fe8640 10 monclient: build_initial_monmap 2024-09-16T14:45:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.992+0000 7fbca0fe8640 10 monclient: monmap: 2024-09-16T14:45:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:53.997722+0000 2024-09-16T14:45:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:53.997722+0000 2024-09-16T14:45:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:53.999 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:53.999 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:54.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 1 Processor -- start 2024-09-16T14:45:54.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 1 -- start start 2024-09-16T14:45:54.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 10 monclient: init 2024-09-16T14:45:54.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:54.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:54.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 10 monclient(hunting): picked mon.noname-b con 0x7fbc9c1318b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:54.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 10 monclient(hunting): picked mon.noname-c con 0x7fbc9c06a630 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:54.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 10 monclient(hunting): picked mon.noname-a con 0x7fbc9c13b160 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:54.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c0993c0 con 0x7fbc9c13b160 2024-09-16T14:45:54.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c099150 con 0x7fbc9c06a630 2024-09-16T14:45:54.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c1323a0 con 0x7fbc9c1318b0 2024-09-16T14:45:54.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:54.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 10 monclient(hunting): authenticate will time out at 1136.113578s 2024-09-16T14:45:54.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc99d9b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbc9c06a630 0x7fbc9c139a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47048/0 (socket says 172.21.15.189:47048) 2024-09-16T14:45:54.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc99d9b640 1 -- 172.21.15.189:0/3495526700 learned_addr learned my addr 172.21.15.189:0/3495526700 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:54.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9ad9d640 1 -- 172.21.15.189:0/3495526700 >> v1:172.21.15.26:6789/0 conn(0x7fbc9c13b160 legacy=0x7fbc9c13d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:54.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 1 -- 172.21.15.189:0/3495526700 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303838608 0 0) 0x7fbc9c099150 con 0x7fbc9c06a630 2024-09-16T14:45:54.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 10 monclient(hunting): _negotiate 2024-09-16T14:45:54.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:54.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:54.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 10 monclient(hunting): my global_id is 94330 2024-09-16T14:45:54.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 1 -- 172.21.15.189:0/3495526700 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc84003530 con 0x7fbc9c06a630 2024-09-16T14:45:54.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 1 -- 172.21.15.189:0/3495526700 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1144669304 0 0) 0x7fbc84003530 con 0x7fbc9c06a630 2024-09-16T14:45:54.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 1 -- 172.21.15.189:0/3495526700 >> v1:172.21.15.26:6790/0 conn(0x7fbc9c1318b0 legacy=0x7fbc9c131c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:54.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 1 -- 172.21.15.189:0/3495526700 >> v1:172.21.15.26:6789/0 conn(0x7fbc9c13b160 legacy=0x7fbc9c13d550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:54.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 10 monclient: _finish_hunting 0 2024-09-16T14:45:54.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 1 monclient: found mon.noname-c 2024-09-16T14:45:54.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:54.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:54.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 1 -- 172.21.15.189:0/3495526700 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c09a360 con 0x7fbc9c06a630 2024-09-16T14:45:54.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 10 monclient: _finish_auth 0 2024-09-16T14:45:54.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 10 monclient: _check_auth_tickets 2024-09-16T14:45:54.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:54.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 1 -- 172.21.15.189:0/3495526700 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbc90001620 con 0x7fbc9c06a630 2024-09-16T14:45:54.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 5 monclient: authenticate success, global_id 94330 2024-09-16T14:45:54.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:54.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:54.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:54.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 10 monclient: dump: 2024-09-16T14:45:54.008 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:54.008 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:54.008 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:54.008 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:54.008 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:54.009 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:54.009 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:54.009 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:54.009 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:54.009 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:54.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:54.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 1 -- 172.21.15.189:0/3495526700 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc900018d0 con 0x7fbc9c06a630 2024-09-16T14:45:54.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:54.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:54.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 1 -- 172.21.15.189:0/3495526700 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbc90001ca0 con 0x7fbc9c06a630 2024-09-16T14:45:54.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 4 set_mon_vals no callback set 2024-09-16T14:45:54.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:54.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbc9959a640 10 monclient: dump: 2024-09-16T14:45:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:54.010 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 10 monclient: shutdown 2024-09-16T14:45:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 1 -- 172.21.15.189:0/3495526700 >> v1:172.21.15.189:6789/0 conn(0x7fbc9c06a630 legacy=0x7fbc9c139a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 1 -- 172.21.15.189:0/3495526700 shutdown_connections 2024-09-16T14:45:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 1 -- 172.21.15.189:0/3495526700 >> 172.21.15.189:0/3495526700 conn(0x7fbc9c0a4420 msgr2=0x7fbc9c0a08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 1 -- 172.21.15.189:0/3495526700 shutdown_connections 2024-09-16T14:45:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 1 -- 172.21.15.189:0/3495526700 wait complete. 2024-09-16T14:45:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 10 monclient: build_initial_monmap 2024-09-16T14:45:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 10 monclient: monmap: 2024-09-16T14:45:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:54.012 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:53.996+0000 7fbca0fe8640 1 Processor -- start 2024-09-16T14:45:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 1 -- start start 2024-09-16T14:45:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 10 monclient: init 2024-09-16T14:45:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:45:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 10 monclient(hunting): picked mon.noname-a con 0x7fbc9c06a630 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 10 monclient(hunting): picked mon.noname-b con 0x7fbc9c1318b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 10 monclient(hunting): picked mon.noname-c con 0x7fbc9c13b160 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c11bf50 con 0x7fbc9c06a630 2024-09-16T14:45:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c13a680 con 0x7fbc9c1318b0 2024-09-16T14:45:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c13ad30 con 0x7fbc9c13b160 2024-09-16T14:45:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 10 monclient(hunting): authenticate will time out at 1136.116577s 2024-09-16T14:45:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc9a59c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fbc9c06a630 legacy=0x7fbc9c1301f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc99d9b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbc9c1318b0 0x7fbc9c1ce730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47062/0 (socket says 172.21.15.189:47062) 2024-09-16T14:45:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc99d9b640 1 -- 172.21.15.189:0/1083306269 learned_addr learned my addr 172.21.15.189:0/1083306269 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 1 -- 172.21.15.189:0/1083306269 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316795888 0 0) 0x7fbc9c13a680 con 0x7fbc9c1318b0 2024-09-16T14:45:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 10 monclient(hunting): my global_id is 94333 2024-09-16T14:45:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 1 -- 172.21.15.189:0/1083306269 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc7c003530 con 0x7fbc9c1318b0 2024-09-16T14:45:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 1 -- 172.21.15.189:0/1083306269 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3374793740 0 0) 0x7fbc7c003530 con 0x7fbc9c1318b0 2024-09-16T14:45:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 1 -- 172.21.15.189:0/1083306269 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc9c13a680 con 0x7fbc9c1318b0 2024-09-16T14:45:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 1 -- 172.21.15.189:0/1083306269 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbc90001be0 con 0x7fbc9c1318b0 2024-09-16T14:45:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 10 monclient(hunting): dump: 2024-09-16T14:45:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:54.015 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 1 -- 172.21.15.189:0/1083306269 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 859384752 0 0) 0x7fbc9c13a680 con 0x7fbc9c1318b0 2024-09-16T14:45:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 1 -- 172.21.15.189:0/1083306269 >> v1:172.21.15.26:6790/0 conn(0x7fbc9c13b160 legacy=0x7fbc9c1d1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 1 -- 172.21.15.189:0/1083306269 >> v1:172.21.15.26:6789/0 conn(0x7fbc9c06a630 legacy=0x7fbc9c1301f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 1 monclient: found mon.b 2024-09-16T14:45:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 1 -- 172.21.15.189:0/1083306269 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c0a3040 con 0x7fbc9c1318b0 2024-09-16T14:45:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 10 monclient: _finish_auth 0 2024-09-16T14:45:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 5 monclient: authenticate success, global_id 94333 2024-09-16T14:45:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 10 monclient: _renew_subs 2024-09-16T14:45:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 1 -- 172.21.15.189:0/1083306269 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc9c13a100 con 0x7fbc9c1318b0 2024-09-16T14:45:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 10 monclient: _renew_subs 2024-09-16T14:45:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 1 -- 172.21.15.189:0/1083306269 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9c11a4d0 con 0x7fbc9c1318b0 2024-09-16T14:45:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 1 -- 172.21.15.189:0/1083306269 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc90001200 con 0x7fbc9c1318b0 2024-09-16T14:45:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 1 -- 172.21.15.189:0/1083306269 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbc90004e20 con 0x7fbc9c1318b0 2024-09-16T14:45:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc98d99640 4 set_mon_vals no callback set 2024-09-16T14:45:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 10 monclient: dump: 2024-09-16T14:45:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:54.017 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 1 -- 172.21.15.189:0/1083306269 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbc90004f80 con 0x7fbc9c1318b0 2024-09-16T14:45:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbca0fe8640 1 -- 172.21.15.189:0/1083306269 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc80005050 con 0x7fbc9c1318b0 2024-09-16T14:45:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc8affd640 1 -- 172.21.15.189:0/1083306269 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(71..71 src has 15..71) v4 ==== 5235+0+0 (unknown 1665813604 0 0) 0x7fbc90043b40 con 0x7fbc9c1318b0 2024-09-16T14:45:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.000+0000 7fbc9a59c640 10 monclient: get_auth_request con 0x7fbc7c036010 auth_method 0 2024-09-16T14:45:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.004+0000 7fbc8affd640 1 -- 172.21.15.189:0/1083306269 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbc900065a0 con 0x7fbc9c1318b0 2024-09-16T14:45:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.004+0000 7fbc8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:54.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.004+0000 7fbc8affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:54.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.168+0000 7fbca0fe8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T14:45:54.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.168+0000 7fbca0fe8640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T14:45:54.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.168+0000 7fbca0fe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:54.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.168+0000 7fbca0fe8640 1 -- 172.21.15.189:0/1083306269 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7fbc80005880 con 0x7fbc9c1318b0 2024-09-16T14:45:54.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.172+0000 7fbc8affd640 1 -- 172.21.15.189:0/1083306269 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v71) v1 ==== 126+0+0 (unknown 3211882315 0 0) 0x7fbc90012f40 con 0x7fbc9c1318b0 2024-09-16T14:45:54.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.172+0000 7fbc8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T14:45:54.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.172+0000 7fbc8affd640 10 monclient: _finish_command 2 = system:0 bucket 'bar' already exists 2024-09-16T14:45:54.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.268+0000 7fbca0fe8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T14:45:54.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.268+0000 7fbca0fe8640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T14:45:54.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.268+0000 7fbca0fe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:54.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.268+0000 7fbca0fe8640 1 -- 172.21.15.189:0/1083306269 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7fbc80005340 con 0x7fbc9c1318b0 2024-09-16T14:45:54.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.268+0000 7fbc8affd640 1 -- 172.21.15.189:0/1083306269 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v71) v1 ==== 126+0+0 (unknown 3211882315 0 0) 0x7fbc900146c0 con 0x7fbc9c1318b0 2024-09-16T14:45:54.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.268+0000 7fbc8affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T14:45:54.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.268+0000 7fbc8affd640 10 monclient: _finish_command 3 = system:0 bucket 'bar' already exists 2024-09-16T14:45:54.272 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'bar' already exists 2024-09-16T14:45:54.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.272+0000 7fbca0fe8640 1 -- 172.21.15.189:0/1083306269 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbc7c036010 legacy=0x7fbc7c0384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:54.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.272+0000 7fbca0fe8640 10 monclient: shutdown 2024-09-16T14:45:54.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.272+0000 7fbca0fe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:54.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.272+0000 7fbca0fe8640 1 -- 172.21.15.189:0/1083306269 >> v1:172.21.15.189:6789/0 conn(0x7fbc9c1318b0 legacy=0x7fbc9c1ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:54.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.272+0000 7fbca0fe8640 1 -- 172.21.15.189:0/1083306269 shutdown_connections 2024-09-16T14:45:54.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.272+0000 7fbca0fe8640 1 -- 172.21.15.189:0/1083306269 >> 172.21.15.189:0/1083306269 conn(0x7fbc9c0a4420 msgr2=0x7fbc9c0a2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:54.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.272+0000 7fbca0fe8640 1 -- 172.21.15.189:0/1083306269 shutdown_connections 2024-09-16T14:45:54.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.272+0000 7fbca0fe8640 1 -- 172.21.15.189:0/1083306269 wait complete. 2024-09-16T14:45:54.289 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush link host1 root=bar 2024-09-16T14:45:54.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.404+0000 7f7b58385640 10 monclient: get_monmap_and_config 2024-09-16T14:45:54.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.404+0000 7f7b58385640 10 monclient: build_initial_monmap 2024-09-16T14:45:54.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.404+0000 7f7b58385640 10 monclient: monmap: 2024-09-16T14:45:54.408 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:54.408 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:54.408 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:54.408044+0000 2024-09-16T14:45:54.408 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:54.408044+0000 2024-09-16T14:45:54.408 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:54.409 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:54.409 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:54.409 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:54.409 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:54.409 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:54.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.404+0000 7f7b58385640 1 Processor -- start 2024-09-16T14:45:54.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.404+0000 7f7b58385640 1 -- start start 2024-09-16T14:45:54.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.404+0000 7f7b58385640 10 monclient: init 2024-09-16T14:45:54.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:54.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:54.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient(hunting): picked mon.noname-b con 0x7f7b5009e890 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:54.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient(hunting): picked mon.noname-c con 0x7f7b500a13d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:54.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient(hunting): picked mon.noname-a con 0x7f7b5013aa10 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:54.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5009f380 con 0x7f7b5013aa10 2024-09-16T14:45:54.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5009d4b0 con 0x7f7b500a13d0 2024-09-16T14:45:54.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b500a11d0 con 0x7f7b5009e890 2024-09-16T14:45:54.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:54.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient(hunting): authenticate will time out at 1136.524510s 2024-09-16T14:45:54.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b56922640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f7b5013aa10 legacy=0x7f7b5013ce00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:54.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b55920640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7b500a13d0 0x7f7b5009cda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47064/0 (socket says 172.21.15.189:47064) 2024-09-16T14:45:54.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b55920640 1 -- 172.21.15.189:0/3735122553 learned_addr learned my addr 172.21.15.189:0/3735122553 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 1 -- 172.21.15.189:0/3735122553 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266955373 0 0) 0x7f7b40001120 con 0x7f7b500a13d0 2024-09-16T14:45:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 10 monclient(hunting): _negotiate 2024-09-16T14:45:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 10 monclient(hunting): my global_id is 94336 2024-09-16T14:45:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 1 -- 172.21.15.189:0/3735122553 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b3c003530 con 0x7f7b500a13d0 2024-09-16T14:45:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 1 -- 172.21.15.189:0/3735122553 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4098418160 0 0) 0x7f7b3c003530 con 0x7f7b500a13d0 2024-09-16T14:45:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 1 -- 172.21.15.189:0/3735122553 >> v1:172.21.15.26:6790/0 conn(0x7f7b5009e890 legacy=0x7f7b5009ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 1 -- 172.21.15.189:0/3735122553 >> v1:172.21.15.26:6789/0 conn(0x7f7b5013aa10 legacy=0x7f7b5013ce00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 10 monclient: _finish_hunting 0 2024-09-16T14:45:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 1 monclient: found mon.noname-c 2024-09-16T14:45:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 1 -- 172.21.15.189:0/3735122553 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5009c3c0 con 0x7f7b500a13d0 2024-09-16T14:45:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 10 monclient: _finish_auth 0 2024-09-16T14:45:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 10 monclient: _check_auth_tickets 2024-09-16T14:45:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 1 -- 172.21.15.189:0/3735122553 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7b40001610 con 0x7f7b500a13d0 2024-09-16T14:45:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 5 monclient: authenticate success, global_id 94336 2024-09-16T14:45:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 10 monclient: dump: 2024-09-16T14:45:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:54.416 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:54.416 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:54.416 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:54.416 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:54.416 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:54.416 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:54.416 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:54.416 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:54.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:54.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 1 -- 172.21.15.189:0/3735122553 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b40001930 con 0x7f7b500a13d0 2024-09-16T14:45:54.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 1 -- 172.21.15.189:0/3735122553 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7b40001d00 con 0x7f7b500a13d0 2024-09-16T14:45:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 4 set_mon_vals no callback set 2024-09-16T14:45:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b5511f640 10 monclient: dump: 2024-09-16T14:45:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:54.418 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient: shutdown 2024-09-16T14:45:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 1 -- 172.21.15.189:0/3735122553 >> v1:172.21.15.189:6789/0 conn(0x7f7b500a13d0 legacy=0x7f7b5009cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 1 -- 172.21.15.189:0/3735122553 shutdown_connections 2024-09-16T14:45:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 1 -- 172.21.15.189:0/3735122553 >> 172.21.15.189:0/3735122553 conn(0x7f7b500a1920 msgr2=0x7f7b500a1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 1 -- 172.21.15.189:0/3735122553 shutdown_connections 2024-09-16T14:45:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 1 -- 172.21.15.189:0/3735122553 wait complete. 2024-09-16T14:45:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient: build_initial_monmap 2024-09-16T14:45:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient: monmap: 2024-09-16T14:45:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:54.419 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 1 Processor -- start 2024-09-16T14:45:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 1 -- start start 2024-09-16T14:45:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient: init 2024-09-16T14:45:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient(hunting): picked mon.noname-c con 0x7f7b5009e890 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient(hunting): picked mon.noname-b con 0x7f7b500a13d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient(hunting): picked mon.noname-a con 0x7f7b5013aa10 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5009e0b0 con 0x7f7b5013aa10 2024-09-16T14:45:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5009db80 con 0x7f7b500a13d0 2024-09-16T14:45:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b50136c70 con 0x7f7b5009e890 2024-09-16T14:45:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b58385640 10 monclient(hunting): authenticate will time out at 1136.527778s 2024-09-16T14:45:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b56922640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f7b5013aa10 legacy=0x7f7b501d1790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b55920640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7b500a13d0 0x7f7b5009a310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47066/0 (socket says 172.21.15.189:47066) 2024-09-16T14:45:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.408+0000 7f7b55920640 1 -- 172.21.15.189:0/3731658545 learned_addr learned my addr 172.21.15.189:0/3731658545 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159204750 0 0) 0x7f7b5009db80 con 0x7f7b500a13d0 2024-09-16T14:45:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient(hunting): my global_id is 94339 2024-09-16T14:45:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b34003530 con 0x7f7b500a13d0 2024-09-16T14:45:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1706215834 0 0) 0x7f7b34003530 con 0x7f7b500a13d0 2024-09-16T14:45:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b5009db80 con 0x7f7b500a13d0 2024-09-16T14:45:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7b40001710 con 0x7f7b500a13d0 2024-09-16T14:45:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient(hunting): dump: 2024-09-16T14:45:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:54.426 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432903377 0 0) 0x7f7b50136c70 con 0x7f7b5009e890 2024-09-16T14:45:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient(hunting): my global_id is 5225 2024-09-16T14:45:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b34003530 con 0x7f7b5009e890 2024-09-16T14:45:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2689202201 0 0) 0x7f7b5009db80 con 0x7f7b500a13d0 2024-09-16T14:45:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 >> v1:172.21.15.26:6790/0 conn(0x7f7b5009e890 legacy=0x7f7b50099c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 >> v1:172.21.15.26:6789/0 conn(0x7f7b5013aa10 legacy=0x7f7b501d1790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 1 monclient: found mon.b 2024-09-16T14:45:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5009aa20 con 0x7f7b500a13d0 2024-09-16T14:45:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b58385640 5 monclient: authenticate success, global_id 94339 2024-09-16T14:45:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b58385640 10 monclient: _renew_subs 2024-09-16T14:45:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b58385640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b58385640 1 -- 172.21.15.189:0/3731658545 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b5009c110 con 0x7f7b500a13d0 2024-09-16T14:45:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b58385640 10 monclient: _renew_subs 2024-09-16T14:45:54.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b58385640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:54.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b58385640 1 -- 172.21.15.189:0/3731658545 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b50136990 con 0x7f7b500a13d0 2024-09-16T14:45:54.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b40004cf0 con 0x7f7b500a13d0 2024-09-16T14:45:54.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:54.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7b40005080 con 0x7f7b500a13d0 2024-09-16T14:45:54.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:54.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:54.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b5491e640 4 set_mon_vals no callback set 2024-09-16T14:45:54.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 10 monclient: dump: 2024-09-16T14:45:54.428 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:54.428 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:54.428 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:54.428 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:54.429 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:54.429 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:54.429 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:54.429 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:54.429 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:54.429 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:54.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b58385640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:54.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b58385640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:54.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b58385640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:54.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b58385640 1 -- 172.21.15.189:0/3731658545 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b38005050 con 0x7f7b500a13d0 2024-09-16T14:45:54.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7b40005260 con 0x7f7b500a13d0 2024-09-16T14:45:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(71..71 src has 15..71) v4 ==== 5235+0+0 (unknown 1665813604 0 0) 0x7f7b40043b90 con 0x7f7b500a13d0 2024-09-16T14:45:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.412+0000 7f7b56121640 10 monclient: get_auth_request con 0x7f7b340361d0 auth_method 0 2024-09-16T14:45:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.416+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7b400149a0 con 0x7f7b500a13d0 2024-09-16T14:45:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.416+0000 7f7b46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.416+0000 7f7b46ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.580+0000 7f7b58385640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T14:45:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.580+0000 7f7b58385640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T14:45:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.580+0000 7f7b58385640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:54.580+0000 7f7b58385640 1 -- 172.21.15.189:0/3731658545 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7f7b38005880 con 0x7f7b500a13d0 2024-09-16T14:45:55.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.520+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 linked item id -11 name 'host1' to location {root=bar} in crush map v72) v1 ==== 168+0+0 (unknown 3323837898 0 0) 0x7f7b40013040 con 0x7f7b500a13d0 2024-09-16T14:45:55.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.520+0000 7f7b46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T14:45:55.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.520+0000 7f7b46ffd640 10 monclient: _finish_command 2 = system:0 linked item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T14:45:55.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.624+0000 7f7b58385640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T14:45:55.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.624+0000 7f7b58385640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T14:45:55.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.624+0000 7f7b58385640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:55.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.624+0000 7f7b58385640 1 -- 172.21.15.189:0/3731658545 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7f7b38005300 con 0x7f7b500a13d0 2024-09-16T14:45:55.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.624+0000 7f7b46ffd640 1 -- 172.21.15.189:0/3731658545 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v72) v1 ==== 177+0+0 (unknown 214602497 0 0) 0x7f7b400147c0 con 0x7f7b500a13d0 2024-09-16T14:45:55.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.624+0000 7f7b46ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T14:45:55.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.624+0000 7f7b46ffd640 10 monclient: _finish_command 3 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T14:45:55.628 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T14:45:55.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.628+0000 7f7b58385640 1 -- 172.21.15.189:0/3731658545 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7b340361d0 legacy=0x7f7b34038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:55.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.628+0000 7f7b58385640 10 monclient: shutdown 2024-09-16T14:45:55.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.628+0000 7f7b58385640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:55.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.628+0000 7f7b58385640 1 -- 172.21.15.189:0/3731658545 >> v1:172.21.15.189:6789/0 conn(0x7f7b500a13d0 legacy=0x7f7b5009a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:55.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.628+0000 7f7b58385640 1 -- 172.21.15.189:0/3731658545 shutdown_connections 2024-09-16T14:45:55.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.628+0000 7f7b58385640 1 -- 172.21.15.189:0/3731658545 >> 172.21.15.189:0/3731658545 conn(0x7f7b500a1920 msgr2=0x7f7b50070590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:55.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.628+0000 7f7b58385640 1 -- 172.21.15.189:0/3731658545 shutdown_connections 2024-09-16T14:45:55.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.628+0000 7f7b58385640 1 -- 172.21.15.189:0/3731658545 wait complete. 2024-09-16T14:45:55.644 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush link host1 root=bar 2024-09-16T14:45:55.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.756+0000 7f1c0d7ed640 10 monclient: get_monmap_and_config 2024-09-16T14:45:55.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.756+0000 7f1c0d7ed640 10 monclient: build_initial_monmap 2024-09-16T14:45:55.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.756+0000 7f1c0d7ed640 10 monclient: monmap: 2024-09-16T14:45:55.760 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:55.760 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:55.760 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:55.760080+0000 2024-09-16T14:45:55.760 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:55.760080+0000 2024-09-16T14:45:55.760 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:55.761 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:55.761 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:55.761 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:55.761 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:55.761 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:55.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.756+0000 7f1c0d7ed640 1 Processor -- start 2024-09-16T14:45:55.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.756+0000 7f1c0d7ed640 1 -- start start 2024-09-16T14:45:55.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.756+0000 7f1c0d7ed640 10 monclient: init 2024-09-16T14:45:55.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:55.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:55.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient(hunting): picked mon.noname-c con 0x7f1c0809df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:55.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient(hunting): picked mon.noname-b con 0x7f1c0809e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:55.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient(hunting): picked mon.noname-a con 0x7f1c0809e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:55.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0809ee20 con 0x7f1c0809e740 2024-09-16T14:45:55.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0809f100 con 0x7f1c0809df10 2024-09-16T14:45:55.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0809f3e0 con 0x7f1c0809e2f0 2024-09-16T14:45:55.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:55.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient(hunting): authenticate will time out at 1137.876347s 2024-09-16T14:45:55.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c06ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1c0809df10 0x7f1c0809d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47070/0 (socket says 172.21.15.189:47070) 2024-09-16T14:45:55.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c06ffd640 1 -- 172.21.15.189:0/238092142 learned_addr learned my addr 172.21.15.189:0/238092142 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:55.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c077fe640 1 -- 172.21.15.189:0/238092142 >> v1:172.21.15.26:6789/0 conn(0x7f1c0809e740 legacy=0x7f1c0813b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:55.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 1 -- 172.21.15.189:0/238092142 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453548680 0 0) 0x7f1c0809f100 con 0x7f1c0809df10 2024-09-16T14:45:55.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:55.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:55.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:55.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 10 monclient(hunting): my global_id is 94342 2024-09-16T14:45:55.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 1 -- 172.21.15.189:0/238092142 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf0003530 con 0x7f1c0809df10 2024-09-16T14:45:55.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 1 -- 172.21.15.189:0/238092142 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 480349142 0 0) 0x7f1bf0003530 con 0x7f1c0809df10 2024-09-16T14:45:55.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 1 -- 172.21.15.189:0/238092142 >> v1:172.21.15.26:6790/0 conn(0x7f1c0809e2f0 legacy=0x7f1c081378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:55.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 1 -- 172.21.15.189:0/238092142 >> v1:172.21.15.26:6789/0 conn(0x7f1c0809e740 legacy=0x7f1c0813b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:55.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:45:55.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 1 monclient: found mon.noname-c 2024-09-16T14:45:55.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:55.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:55.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 1 -- 172.21.15.189:0/238092142 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0809c690 con 0x7f1c0809df10 2024-09-16T14:45:55.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 10 monclient: _finish_auth 0 2024-09-16T14:45:55.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:55.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:55.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 1 -- 172.21.15.189:0/238092142 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1bf80019a0 con 0x7f1c0809df10 2024-09-16T14:45:55.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:55.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:55.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 10 monclient: dump: 2024-09-16T14:45:55.771 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:55.771 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:55.771 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:55.771 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:55.771 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:55.771 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:55.772 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:55.772 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:55.772 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:55.772 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:55.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 5 monclient: authenticate success, global_id 94342 2024-09-16T14:45:55.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:55.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 1 -- 172.21.15.189:0/238092142 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bf8001cc0 con 0x7f1c0809df10 2024-09-16T14:45:55.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:55.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:55.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 4 set_mon_vals no callback set 2024-09-16T14:45:55.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 1 -- 172.21.15.189:0/238092142 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1bf8002090 con 0x7f1c0809df10 2024-09-16T14:45:55.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:55.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:55.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c05ffb640 10 monclient: dump: 2024-09-16T14:45:55.773 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:55.773 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:55.773 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:55.773 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:55.773 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:55.773 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:55.773 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:55.773 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:55.774 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:55.774 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:55.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient: shutdown 2024-09-16T14:45:55.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:55.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 1 -- 172.21.15.189:0/238092142 >> v1:172.21.15.189:6789/0 conn(0x7f1c0809df10 legacy=0x7f1c0809d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:55.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 1 -- 172.21.15.189:0/238092142 shutdown_connections 2024-09-16T14:45:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 1 -- 172.21.15.189:0/238092142 >> 172.21.15.189:0/238092142 conn(0x7f1c080a3320 msgr2=0x7f1c0809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 1 -- 172.21.15.189:0/238092142 shutdown_connections 2024-09-16T14:45:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 1 -- 172.21.15.189:0/238092142 wait complete. 2024-09-16T14:45:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient: build_initial_monmap 2024-09-16T14:45:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient: monmap: 2024-09-16T14:45:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:55.776 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:55.776 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:55.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 1 Processor -- start 2024-09-16T14:45:55.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 1 -- start start 2024-09-16T14:45:55.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient: init 2024-09-16T14:45:55.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:55.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:55.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient(hunting): picked mon.noname-a con 0x7f1c0809df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:55.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient(hunting): picked mon.noname-c con 0x7f1c0809e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:55.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient(hunting): picked mon.noname-b con 0x7f1c0809e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:55.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0813ab80 con 0x7f1c0809df10 2024-09-16T14:45:55.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0806cea0 con 0x7f1c0809e740 2024-09-16T14:45:55.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0806f970 con 0x7f1c0809e2f0 2024-09-16T14:45:55.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:55.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient(hunting): authenticate will time out at 1137.879261s 2024-09-16T14:45:55.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c06ffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f1c0809df10 legacy=0x7f1c0809a1f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:55.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c077fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1c0809e740 0x7f1c081d17a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47086/0 (socket says 172.21.15.189:47086) 2024-09-16T14:45:55.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c077fe640 1 -- 172.21.15.189:0/4058720468 learned_addr learned my addr 172.21.15.189:0/4058720468 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:55.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 1 -- 172.21.15.189:0/4058720468 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936892936 0 0) 0x7f1c0806cea0 con 0x7f1c0809e740 2024-09-16T14:45:55.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:45:55.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:55.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:55.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 10 monclient(hunting): my global_id is 94345 2024-09-16T14:45:55.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 1 -- 172.21.15.189:0/4058720468 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf4003530 con 0x7f1c0809e740 2024-09-16T14:45:55.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 1 -- 172.21.15.189:0/4058720468 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2208554348 0 0) 0x7f1bf4003530 con 0x7f1c0809e740 2024-09-16T14:45:55.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 1 -- 172.21.15.189:0/4058720468 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c0806cea0 con 0x7f1c0809e740 2024-09-16T14:45:55.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 1 -- 172.21.15.189:0/4058720468 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1c00004080 con 0x7f1c0809e740 2024-09-16T14:45:55.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:55.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:55.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 10 monclient(hunting): dump: 2024-09-16T14:45:55.778 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:55.778 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:55.778 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:55.778 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:55.778 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:55.778 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:55.778 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:55.778 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:55.778 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:55.779 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:55.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:55.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 1 -- 172.21.15.189:0/4058720468 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 799801578 0 0) 0x7f1c0806cea0 con 0x7f1c0809e740 2024-09-16T14:45:55.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 1 -- 172.21.15.189:0/4058720468 >> v1:172.21.15.26:6790/0 conn(0x7f1c0809e2f0 legacy=0x7f1c0809a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:55.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 1 -- 172.21.15.189:0/4058720468 >> v1:172.21.15.26:6789/0 conn(0x7f1c0809df10 legacy=0x7f1c0809a1f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:55.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:45:55.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 1 monclient: found mon.b 2024-09-16T14:45:55.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:55.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:55.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 1 -- 172.21.15.189:0/4058720468 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0813b370 con 0x7f1c0809e740 2024-09-16T14:45:55.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 10 monclient: _finish_auth 0 2024-09-16T14:45:55.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:45:55.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1bef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:55.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 5 monclient: authenticate success, global_id 94345 2024-09-16T14:45:55.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.760+0000 7f1c0d7ed640 10 monclient: _renew_subs 2024-09-16T14:45:55.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1c0d7ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:55.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1c0d7ed640 1 -- 172.21.15.189:0/4058720468 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0809b010 con 0x7f1c0809e740 2024-09-16T14:45:55.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1c0d7ed640 10 monclient: _renew_subs 2024-09-16T14:45:55.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1c0d7ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:55.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1c0d7ed640 1 -- 172.21.15.189:0/4058720468 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0813b090 con 0x7f1c0809e740 2024-09-16T14:45:55.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1bef7fe640 1 -- 172.21.15.189:0/4058720468 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c000043a0 con 0x7f1c0809e740 2024-09-16T14:45:55.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1bef7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:55.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1bef7fe640 1 -- 172.21.15.189:0/4058720468 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1c00004730 con 0x7f1c0809e740 2024-09-16T14:45:55.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1bef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:55.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1bef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:55.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1c057fa640 4 set_mon_vals no callback set 2024-09-16T14:45:55.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1bef7fe640 10 monclient: dump: 2024-09-16T14:45:55.781 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:55.781 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:55.781 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:55.781 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:55.781 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:55.781 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:55.781 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:55.781 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:55.781 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:55.781 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:55.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1bef7fe640 1 -- 172.21.15.189:0/4058720468 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1c00004910 con 0x7f1c0809e740 2024-09-16T14:45:55.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1c0d7ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:55.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1c0d7ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:55.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1c0d7ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:55.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1c0d7ed640 1 -- 172.21.15.189:0/4058720468 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be8005050 con 0x7f1c0809e740 2024-09-16T14:45:55.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1bef7fe640 1 -- 172.21.15.189:0/4058720468 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(72..72 src has 15..72) v4 ==== 5361+0+0 (unknown 4174954110 0 0) 0x7f1c000438a0 con 0x7f1c0809e740 2024-09-16T14:45:55.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.764+0000 7f1c06ffd640 10 monclient: get_auth_request con 0x7f1bf4035fe0 auth_method 0 2024-09-16T14:45:55.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.768+0000 7f1bef7fe640 1 -- 172.21.15.189:0/4058720468 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1c00014740 con 0x7f1c0809e740 2024-09-16T14:45:55.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.768+0000 7f1bef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:55.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.768+0000 7f1bef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.932+0000 7f1c0d7ed640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T14:45:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.932+0000 7f1c0d7ed640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T14:45:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.932+0000 7f1c0d7ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.932+0000 7f1c0d7ed640 1 -- 172.21.15.189:0/4058720468 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7f1be8005880 con 0x7f1c0809e740 2024-09-16T14:45:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.936+0000 7f1bef7fe640 1 -- 172.21.15.189:0/4058720468 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v72) v1 ==== 177+0+0 (unknown 214602497 0 0) 0x7f1c00012bb0 con 0x7f1c0809e740 2024-09-16T14:45:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.936+0000 7f1bef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T14:45:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:55.936+0000 7f1bef7fe640 10 monclient: _finish_command 2 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T14:45:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.028+0000 7f1c0d7ed640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T14:45:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.028+0000 7f1c0d7ed640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T14:45:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.028+0000 7f1c0d7ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.028+0000 7f1c0d7ed640 1 -- 172.21.15.189:0/4058720468 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7f1be8005300 con 0x7f1c0809e740 2024-09-16T14:45:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.032+0000 7f1bef7fe640 1 -- 172.21.15.189:0/4058720468 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v72) v1 ==== 177+0+0 (unknown 214602497 0 0) 0x7f1c00014330 con 0x7f1c0809e740 2024-09-16T14:45:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.032+0000 7f1bef7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T14:45:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.032+0000 7f1bef7fe640 10 monclient: _finish_command 3 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T14:45:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T14:45:56.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.032+0000 7f1c0d7ed640 1 -- 172.21.15.189:0/4058720468 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1bf4035fe0 legacy=0x7f1bf40384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:56.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.032+0000 7f1c0d7ed640 10 monclient: shutdown 2024-09-16T14:45:56.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.032+0000 7f1c0d7ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.032+0000 7f1c0d7ed640 1 -- 172.21.15.189:0/4058720468 >> v1:172.21.15.189:6789/0 conn(0x7f1c0809e740 legacy=0x7f1c081d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.032+0000 7f1c0d7ed640 1 -- 172.21.15.189:0/4058720468 shutdown_connections 2024-09-16T14:45:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.032+0000 7f1c0d7ed640 1 -- 172.21.15.189:0/4058720468 >> 172.21.15.189:0/4058720468 conn(0x7f1c080a3320 msgr2=0x7f1c0806e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.032+0000 7f1c0d7ed640 1 -- 172.21.15.189:0/4058720468 shutdown_connections 2024-09-16T14:45:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.032+0000 7f1c0d7ed640 1 -- 172.21.15.189:0/4058720468 wait complete. 2024-09-16T14:45:56.050 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush link host2 root=bar 2024-09-16T14:45:56.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa388567640 10 monclient: get_monmap_and_config 2024-09-16T14:45:56.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa388567640 10 monclient: build_initial_monmap 2024-09-16T14:45:56.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa388567640 10 monclient: monmap: 2024-09-16T14:45:56.239 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:56.239 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:56.239 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:56.164178+0000 2024-09-16T14:45:56.240 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:56.164178+0000 2024-09-16T14:45:56.240 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:56.240 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:56.240 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:56.240 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:56.241 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:56.241 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:56.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa388567640 1 Processor -- start 2024-09-16T14:45:56.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa388567640 1 -- start start 2024-09-16T14:45:56.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa388567640 10 monclient: init 2024-09-16T14:45:56.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa388567640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:56.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa388567640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:56.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa388567640 10 monclient(hunting): picked mon.noname-c con 0x7fa3801315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:56.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa388567640 10 monclient(hunting): picked mon.noname-b con 0x7fa38009b320 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:56.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa388567640 10 monclient(hunting): picked mon.noname-a con 0x7fa380132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:56.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa388567640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38009edd0 con 0x7fa380132090 2024-09-16T14:45:56.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa388567640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38009bc00 con 0x7fa3801315a0 2024-09-16T14:45:56.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa388567640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38009e900 con 0x7fa38009b320 2024-09-16T14:45:56.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa388567640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:56.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa388567640 10 monclient(hunting): authenticate will time out at 1138.279799s 2024-09-16T14:45:56.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa386303640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa3801315a0 0x7fa380131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47092/0 (socket says 172.21.15.189:47092) 2024-09-16T14:45:56.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa386303640 1 -- 172.21.15.189:0/1973604249 learned_addr learned my addr 172.21.15.189:0/1973604249 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:56.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.160+0000 7fa386b04640 1 -- 172.21.15.189:0/1973604249 >> v1:172.21.15.26:6789/0 conn(0x7fa380132090 legacy=0x7fa38013a200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:56.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 1 -- 172.21.15.189:0/1973604249 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575453571 0 0) 0x7fa38009bc00 con 0x7fa3801315a0 2024-09-16T14:45:56.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 10 monclient(hunting): _negotiate 2024-09-16T14:45:56.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:56.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:56.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 10 monclient(hunting): my global_id is 94348 2024-09-16T14:45:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 1 -- 172.21.15.189:0/1973604249 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa36c003530 con 0x7fa3801315a0 2024-09-16T14:45:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 1 -- 172.21.15.189:0/1973604249 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1431638116 0 0) 0x7fa36c003530 con 0x7fa3801315a0 2024-09-16T14:45:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 1 -- 172.21.15.189:0/1973604249 >> v1:172.21.15.26:6790/0 conn(0x7fa38009b320 legacy=0x7fa380136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 1 -- 172.21.15.189:0/1973604249 >> v1:172.21.15.26:6789/0 conn(0x7fa380132090 legacy=0x7fa38013a200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 10 monclient: _finish_hunting 0 2024-09-16T14:45:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 1 monclient: found mon.noname-c 2024-09-16T14:45:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 1 -- 172.21.15.189:0/1973604249 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38009b920 con 0x7fa3801315a0 2024-09-16T14:45:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 10 monclient: _finish_auth 0 2024-09-16T14:45:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 10 monclient: _check_auth_tickets 2024-09-16T14:45:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 5 monclient: authenticate success, global_id 94348 2024-09-16T14:45:56.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:56.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 1 -- 172.21.15.189:0/1973604249 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa37c001620 con 0x7fa3801315a0 2024-09-16T14:45:56.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:56.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:56.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 10 monclient: dump: 2024-09-16T14:45:56.251 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:56.251 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:56.251 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:56.251 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:56.251 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:56.251 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:56.251 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:56.252 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:56.252 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:56.252 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:56.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:56.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 1 -- 172.21.15.189:0/1973604249 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa37c001940 con 0x7fa3801315a0 2024-09-16T14:45:56.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:56.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:56.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 4 set_mon_vals no callback set 2024-09-16T14:45:56.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 1 -- 172.21.15.189:0/1973604249 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa37c001cd0 con 0x7fa3801315a0 2024-09-16T14:45:56.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385301640 10 monclient: dump: 2024-09-16T14:45:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:56.253 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:56.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 10 monclient: shutdown 2024-09-16T14:45:56.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:56.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 1 -- 172.21.15.189:0/1973604249 >> v1:172.21.15.189:6789/0 conn(0x7fa3801315a0 legacy=0x7fa380131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:56.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 1 -- 172.21.15.189:0/1973604249 shutdown_connections 2024-09-16T14:45:56.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 1 -- 172.21.15.189:0/1973604249 >> 172.21.15.189:0/1973604249 conn(0x7fa3800a3320 msgr2=0x7fa38009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:56.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 1 -- 172.21.15.189:0/1973604249 shutdown_connections 2024-09-16T14:45:56.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 1 -- 172.21.15.189:0/1973604249 wait complete. 2024-09-16T14:45:56.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 10 monclient: build_initial_monmap 2024-09-16T14:45:56.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 10 monclient: monmap: 2024-09-16T14:45:56.255 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:56.255 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:56.255 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:56.255 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:56.255 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:56.255 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:56.255 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:56.255 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:56.255 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:56.255 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:56.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 1 Processor -- start 2024-09-16T14:45:56.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 1 -- start start 2024-09-16T14:45:56.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 10 monclient: init 2024-09-16T14:45:56.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:56.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:56.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 10 monclient(hunting): picked mon.noname-c con 0x7fa3801315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:56.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 10 monclient(hunting): picked mon.noname-b con 0x7fa380132090 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:56.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 10 monclient(hunting): picked mon.noname-a con 0x7fa380130130 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:56.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38009cd20 con 0x7fa380130130 2024-09-16T14:45:56.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38009aac0 con 0x7fa380132090 2024-09-16T14:45:56.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38006b380 con 0x7fa3801315a0 2024-09-16T14:45:56.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:56.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 10 monclient(hunting): authenticate will time out at 1138.282912s 2024-09-16T14:45:56.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385b02640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa380132090 0x7fa3801ce2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47104/0 (socket says 172.21.15.189:47104) 2024-09-16T14:45:56.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa385b02640 1 -- 172.21.15.189:0/3047525105 learned_addr learned my addr 172.21.15.189:0/3047525105 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:56.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa386b04640 1 -- 172.21.15.189:0/3047525105 >> v1:172.21.15.26:6789/0 conn(0x7fa380130130 legacy=0x7fa3801d1a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:56.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705489306 0 0) 0x7fa38009aac0 con 0x7fa380132090 2024-09-16T14:45:56.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:56.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:56.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:56.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient(hunting): my global_id is 94351 2024-09-16T14:45:56.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa364003530 con 0x7fa380132090 2024-09-16T14:45:56.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1266817068 0 0) 0x7fa364003530 con 0x7fa380132090 2024-09-16T14:45:56.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa38009aac0 con 0x7fa380132090 2024-09-16T14:45:56.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa370001940 con 0x7fa380132090 2024-09-16T14:45:56.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:56.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:56.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient(hunting): dump: 2024-09-16T14:45:56.258 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:56.258 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:56.258 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:56.258 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:56.258 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:56.258 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:56.258 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:56.258 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:56.259 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:56.259 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:56.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:56.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683670811 0 0) 0x7fa38006b380 con 0x7fa3801315a0 2024-09-16T14:45:56.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:56.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:56.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:56.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient(hunting): my global_id is 5237 2024-09-16T14:45:56.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa364003530 con 0x7fa3801315a0 2024-09-16T14:45:56.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1314458273 0 0) 0x7fa38009aac0 con 0x7fa380132090 2024-09-16T14:45:56.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 >> v1:172.21.15.26:6790/0 conn(0x7fa3801315a0 legacy=0x7fa38012fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:56.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 >> v1:172.21.15.26:6789/0 conn(0x7fa380130130 legacy=0x7fa3801d1a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:56.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:56.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 1 monclient: found mon.b 2024-09-16T14:45:56.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:56.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:56.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38011f070 con 0x7fa380132090 2024-09-16T14:45:56.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:56.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:56.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:56.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 5 monclient: authenticate success, global_id 94351 2024-09-16T14:45:56.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 10 monclient: _renew_subs 2024-09-16T14:45:56.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:56.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 1 -- 172.21.15.189:0/3047525105 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa380121160 con 0x7fa380132090 2024-09-16T14:45:56.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 10 monclient: _renew_subs 2024-09-16T14:45:56.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:56.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa388567640 1 -- 172.21.15.189:0/3047525105 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38011f7c0 con 0x7fa380132090 2024-09-16T14:45:56.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa370001d10 con 0x7fa380132090 2024-09-16T14:45:56.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:56.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa370003af0 con 0x7fa380132090 2024-09-16T14:45:56.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:56.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:56.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa384b00640 4 set_mon_vals no callback set 2024-09-16T14:45:56.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.164+0000 7fa376ffd640 10 monclient: dump: 2024-09-16T14:45:56.263 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:56.263 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:56.263 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:56.263 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:56.263 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:56.264 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:56.264 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:56.264 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:56.264 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:56.264 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:56.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.168+0000 7fa388567640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:56.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.168+0000 7fa388567640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:56.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.168+0000 7fa388567640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:56.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.168+0000 7fa388567640 1 -- 172.21.15.189:0/3047525105 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa368005050 con 0x7fa380132090 2024-09-16T14:45:56.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.168+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa370003c50 con 0x7fa380132090 2024-09-16T14:45:56.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.168+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(72..72 src has 15..72) v4 ==== 5361+0+0 (unknown 4174954110 0 0) 0x7fa370043980 con 0x7fa380132090 2024-09-16T14:45:56.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.168+0000 7fa386303640 10 monclient: get_auth_request con 0x7fa364036180 auth_method 0 2024-09-16T14:45:56.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.172+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa370014820 con 0x7fa380132090 2024-09-16T14:45:56.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.172+0000 7fa376ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:56.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.172+0000 7fa376ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:56.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.320+0000 7fa388567640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T14:45:56.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.320+0000 7fa388567640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T14:45:56.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.320+0000 7fa388567640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:56.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.320+0000 7fa388567640 1 -- 172.21.15.189:0/3047525105 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7fa368005880 con 0x7fa380132090 2024-09-16T14:45:56.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.784+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 linked item id -16 name 'host2' to location {root=bar} in crush map v73) v1 ==== 168+0+0 (unknown 864546005 0 0) 0x7fa370012c90 con 0x7fa380132090 2024-09-16T14:45:56.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.784+0000 7fa376ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T14:45:56.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.784+0000 7fa376ffd640 10 monclient: _finish_command 2 = system:0 linked item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T14:45:56.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.880+0000 7fa388567640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T14:45:56.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.880+0000 7fa388567640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T14:45:56.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.880+0000 7fa388567640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:56.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.880+0000 7fa388567640 1 -- 172.21.15.189:0/3047525105 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7fa368005300 con 0x7fa380132090 2024-09-16T14:45:56.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.880+0000 7fa376ffd640 1 -- 172.21.15.189:0/3047525105 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v73) v1 ==== 177+0+0 (unknown 3725627425 0 0) 0x7fa370014410 con 0x7fa380132090 2024-09-16T14:45:56.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.880+0000 7fa376ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T14:45:56.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.880+0000 7fa376ffd640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T14:45:56.884 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T14:45:56.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.884+0000 7fa388567640 1 -- 172.21.15.189:0/3047525105 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa364036180 legacy=0x7fa364038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:56.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.884+0000 7fa388567640 10 monclient: shutdown 2024-09-16T14:45:56.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.884+0000 7fa388567640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:56.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.884+0000 7fa388567640 1 -- 172.21.15.189:0/3047525105 >> v1:172.21.15.189:6789/0 conn(0x7fa380132090 legacy=0x7fa3801ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:56.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.884+0000 7fa388567640 1 -- 172.21.15.189:0/3047525105 shutdown_connections 2024-09-16T14:45:56.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.884+0000 7fa388567640 1 -- 172.21.15.189:0/3047525105 >> 172.21.15.189:0/3047525105 conn(0x7fa3800a3320 msgr2=0x7fa3800a1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:56.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.884+0000 7fa388567640 1 -- 172.21.15.189:0/3047525105 shutdown_connections 2024-09-16T14:45:56.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:56.884+0000 7fa388567640 1 -- 172.21.15.189:0/3047525105 wait complete. 2024-09-16T14:45:56.901 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush link host2 root=bar 2024-09-16T14:45:57.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f7353b70640 10 monclient: get_monmap_and_config 2024-09-16T14:45:57.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f7353b70640 10 monclient: build_initial_monmap 2024-09-16T14:45:57.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f7353b70640 10 monclient: monmap: 2024-09-16T14:45:57.016 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:57.017 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:57.017 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:57.016123+0000 2024-09-16T14:45:57.017 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:57.016123+0000 2024-09-16T14:45:57.017 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:57.017 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:57.017 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:57.017 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:57.017 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:57.017 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f7353b70640 1 Processor -- start 2024-09-16T14:45:57.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f7353b70640 1 -- start start 2024-09-16T14:45:57.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f7353b70640 10 monclient: init 2024-09-16T14:45:57.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f7353b70640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:57.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f7353b70640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:57.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f7353b70640 10 monclient(hunting): picked mon.noname-c con 0x7f734c1315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:57.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f7353b70640 10 monclient(hunting): picked mon.noname-b con 0x7f734c09b320 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:57.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f7353b70640 10 monclient(hunting): picked mon.noname-a con 0x7f734c132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:57.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f7353b70640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c09edd0 con 0x7f734c132090 2024-09-16T14:45:57.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f7353b70640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c09bc00 con 0x7f734c1315a0 2024-09-16T14:45:57.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f7353b70640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c09e900 con 0x7f734c09b320 2024-09-16T14:45:57.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f7353b70640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:57.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f7353b70640 10 monclient(hunting): authenticate will time out at 1139.131844s 2024-09-16T14:45:57.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f735190c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f734c1315a0 0x7f734c131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47110/0 (socket says 172.21.15.189:47110) 2024-09-16T14:45:57.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f735190c640 1 -- 172.21.15.189:0/171607940 learned_addr learned my addr 172.21.15.189:0/171607940 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:57.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.012+0000 7f735210d640 1 -- 172.21.15.189:0/171607940 >> v1:172.21.15.26:6789/0 conn(0x7f734c132090 legacy=0x7f734c13a200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:57.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 1 -- 172.21.15.189:0/171607940 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085125575 0 0) 0x7f734c09bc00 con 0x7f734c1315a0 2024-09-16T14:45:57.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 10 monclient(hunting): _negotiate 2024-09-16T14:45:57.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:57.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:57.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 10 monclient(hunting): my global_id is 94354 2024-09-16T14:45:57.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 1 -- 172.21.15.189:0/171607940 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7334003530 con 0x7f734c1315a0 2024-09-16T14:45:57.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 1 -- 172.21.15.189:0/171607940 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2855427458 0 0) 0x7f7334003530 con 0x7f734c1315a0 2024-09-16T14:45:57.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 1 -- 172.21.15.189:0/171607940 >> v1:172.21.15.26:6790/0 conn(0x7f734c09b320 legacy=0x7f734c136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:57.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 1 -- 172.21.15.189:0/171607940 >> v1:172.21.15.26:6789/0 conn(0x7f734c132090 legacy=0x7f734c13a200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:57.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 10 monclient: _finish_hunting 0 2024-09-16T14:45:57.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 1 monclient: found mon.noname-c 2024-09-16T14:45:57.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:57.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 1 -- 172.21.15.189:0/171607940 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c09b920 con 0x7f734c1315a0 2024-09-16T14:45:57.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 10 monclient: _finish_auth 0 2024-09-16T14:45:57.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 10 monclient: _check_auth_tickets 2024-09-16T14:45:57.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:57.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 1 -- 172.21.15.189:0/171607940 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f73440019a0 con 0x7f734c1315a0 2024-09-16T14:45:57.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 5 monclient: authenticate success, global_id 94354 2024-09-16T14:45:57.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:57.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:57.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:57.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 10 monclient: dump: 2024-09-16T14:45:57.027 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:57.027 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:57.027 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.027 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.027 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:57.027 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:57.027 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:57.027 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:57.028 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:57.028 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:57.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 1 -- 172.21.15.189:0/171607940 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7344001cc0 con 0x7f734c1315a0 2024-09-16T14:45:57.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:57.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 1 -- 172.21.15.189:0/171607940 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7344002050 con 0x7f734c1315a0 2024-09-16T14:45:57.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:57.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:57.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735090a640 10 monclient: dump: 2024-09-16T14:45:57.028 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:57.028 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:57.028 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.029 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.029 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:57.029 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:57.029 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:57.029 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:57.029 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:57.029 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:57.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 4 set_mon_vals no callback set 2024-09-16T14:45:57.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 10 monclient: shutdown 2024-09-16T14:45:57.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:57.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 1 -- 172.21.15.189:0/171607940 >> v1:172.21.15.189:6789/0 conn(0x7f734c1315a0 legacy=0x7f734c131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:57.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 1 -- 172.21.15.189:0/171607940 shutdown_connections 2024-09-16T14:45:57.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 1 -- 172.21.15.189:0/171607940 >> 172.21.15.189:0/171607940 conn(0x7f734c0a3320 msgr2=0x7f734c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 1 -- 172.21.15.189:0/171607940 shutdown_connections 2024-09-16T14:45:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 1 -- 172.21.15.189:0/171607940 wait complete. 2024-09-16T14:45:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 10 monclient: build_initial_monmap 2024-09-16T14:45:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 10 monclient: monmap: 2024-09-16T14:45:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:57.031 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:57.031 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 1 Processor -- start 2024-09-16T14:45:57.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 1 -- start start 2024-09-16T14:45:57.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 10 monclient: init 2024-09-16T14:45:57.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:57.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:45:57.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 10 monclient(hunting): picked mon.noname-c con 0x7f734c1315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:57.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 10 monclient(hunting): picked mon.noname-a con 0x7f734c132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:57.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 10 monclient(hunting): picked mon.noname-b con 0x7f734c1c8fb0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:57.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c09cd20 con 0x7f734c132090 2024-09-16T14:45:57.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c0a0d80 con 0x7f734c1c8fb0 2024-09-16T14:45:57.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c1bd3a0 con 0x7f734c1315a0 2024-09-16T14:45:57.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:57.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 10 monclient(hunting): authenticate will time out at 1139.134610s 2024-09-16T14:45:57.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735110b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f734c132090 legacy=0x7f734c1ce2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:57.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735210d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f734c1c8fb0 0x7f734c1d1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47126/0 (socket says 172.21.15.189:47126) 2024-09-16T14:45:57.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f735210d640 1 -- 172.21.15.189:0/2708608248 learned_addr learned my addr 172.21.15.189:0/2708608248 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:57.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 1 -- 172.21.15.189:0/2708608248 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562740515 0 0) 0x7f734c0a0d80 con 0x7f734c1c8fb0 2024-09-16T14:45:57.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:45:57.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:57.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:57.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 10 monclient(hunting): my global_id is 94357 2024-09-16T14:45:57.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 1 -- 172.21.15.189:0/2708608248 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7338003530 con 0x7f734c1c8fb0 2024-09-16T14:45:57.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 1 -- 172.21.15.189:0/2708608248 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 511521230 0 0) 0x7f7338003530 con 0x7f734c1c8fb0 2024-09-16T14:45:57.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 1 -- 172.21.15.189:0/2708608248 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f734c0a0d80 con 0x7f734c1c8fb0 2024-09-16T14:45:57.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 1 -- 172.21.15.189:0/2708608248 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f733c0040d0 con 0x7f734c1c8fb0 2024-09-16T14:45:57.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:57.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:57.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 10 monclient(hunting): dump: 2024-09-16T14:45:57.034 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:57.035 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:57.035 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.035 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.035 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:57.035 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:57.035 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:57.035 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:57.035 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:57.035 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:57.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 1 -- 172.21.15.189:0/2708608248 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4171714823 0 0) 0x7f734c0a0d80 con 0x7f734c1c8fb0 2024-09-16T14:45:57.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 1 -- 172.21.15.189:0/2708608248 >> v1:172.21.15.26:6790/0 conn(0x7f734c1315a0 legacy=0x7f734c12fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:57.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 1 -- 172.21.15.189:0/2708608248 >> v1:172.21.15.26:6789/0 conn(0x7f734c132090 legacy=0x7f734c1ce2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:57.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:45:57.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 1 monclient: found mon.b 2024-09-16T14:45:57.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:57.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 1 -- 172.21.15.189:0/2708608248 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c1c9440 con 0x7f734c1c8fb0 2024-09-16T14:45:57.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:45:57.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:45:57.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:57.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 5 monclient: authenticate success, global_id 94357 2024-09-16T14:45:57.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 10 monclient: _renew_subs 2024-09-16T14:45:57.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 1 -- 172.21.15.189:0/2708608248 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f734c1bc1d0 con 0x7f734c1c8fb0 2024-09-16T14:45:57.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 10 monclient: _renew_subs 2024-09-16T14:45:57.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f7353b70640 1 -- 172.21.15.189:0/2708608248 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f734c1bbf50 con 0x7f734c1c8fb0 2024-09-16T14:45:57.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 1 -- 172.21.15.189:0/2708608248 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f733c004380 con 0x7f734c1c8fb0 2024-09-16T14:45:57.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:57.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 1 -- 172.21.15.189:0/2708608248 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f733c004710 con 0x7f734c1c8fb0 2024-09-16T14:45:57.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:57.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:57.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734bfff640 4 set_mon_vals no callback set 2024-09-16T14:45:57.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.016+0000 7f734a7fc640 10 monclient: dump: 2024-09-16T14:45:57.038 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:57.038 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:57.038 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.038 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.038 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:57.038 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:57.038 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:57.038 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:57.038 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:57.038 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.020+0000 7f734a7fc640 1 -- 172.21.15.189:0/2708608248 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f733c0048f0 con 0x7f734c1c8fb0 2024-09-16T14:45:57.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.020+0000 7f7353b70640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:57.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.020+0000 7f7353b70640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:57.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.020+0000 7f7353b70640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.020+0000 7f7353b70640 1 -- 172.21.15.189:0/2708608248 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732c005050 con 0x7f734c1c8fb0 2024-09-16T14:45:57.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.020+0000 7f734a7fc640 1 -- 172.21.15.189:0/2708608248 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(73..73 src has 15..73) v4 ==== 5385+0+0 (unknown 4089813478 0 0) 0x7f733c043900 con 0x7f734c1c8fb0 2024-09-16T14:45:57.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.020+0000 7f735190c640 10 monclient: get_auth_request con 0x7f7338036010 auth_method 0 2024-09-16T14:45:57.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.024+0000 7f734a7fc640 1 -- 172.21.15.189:0/2708608248 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f733c005140 con 0x7f734c1c8fb0 2024-09-16T14:45:57.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.024+0000 7f734a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:57.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.024+0000 7f734a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:57.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.172+0000 7f7353b70640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T14:45:57.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.172+0000 7f7353b70640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T14:45:57.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.172+0000 7f7353b70640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.172+0000 7f7353b70640 1 -- 172.21.15.189:0/2708608248 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f732c005880 con 0x7f734c1c8fb0 2024-09-16T14:45:57.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.176+0000 7f734a7fc640 1 -- 172.21.15.189:0/2708608248 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v73) v1 ==== 177+0+0 (unknown 3725627425 0 0) 0x7f733c012ba0 con 0x7f734c1c8fb0 2024-09-16T14:45:57.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.176+0000 7f734a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T14:45:57.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.176+0000 7f734a7fc640 10 monclient: _finish_command 2 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T14:45:57.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.268+0000 7f7353b70640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T14:45:57.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.268+0000 7f7353b70640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T14:45:57.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.268+0000 7f7353b70640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.268+0000 7f7353b70640 1 -- 172.21.15.189:0/2708608248 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f732c005300 con 0x7f734c1c8fb0 2024-09-16T14:45:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.272+0000 7f734a7fc640 1 -- 172.21.15.189:0/2708608248 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v73) v1 ==== 177+0+0 (unknown 3725627425 0 0) 0x7f733c014320 con 0x7f734c1c8fb0 2024-09-16T14:45:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.272+0000 7f734a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T14:45:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.272+0000 7f734a7fc640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T14:45:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T14:45:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.272+0000 7f7353b70640 1 -- 172.21.15.189:0/2708608248 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7338036010 legacy=0x7f73380384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.272+0000 7f7353b70640 10 monclient: shutdown 2024-09-16T14:45:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.272+0000 7f7353b70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.272+0000 7f7353b70640 1 -- 172.21.15.189:0/2708608248 >> v1:172.21.15.189:6789/0 conn(0x7f734c1c8fb0 legacy=0x7f734c1d1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.272+0000 7f7353b70640 1 -- 172.21.15.189:0/2708608248 shutdown_connections 2024-09-16T14:45:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.272+0000 7f7353b70640 1 -- 172.21.15.189:0/2708608248 >> 172.21.15.189:0/2708608248 conn(0x7f734c0a3320 msgr2=0x7f734c0a1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.272+0000 7f7353b70640 1 -- 172.21.15.189:0/2708608248 shutdown_connections 2024-09-16T14:45:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.272+0000 7f7353b70640 1 -- 172.21.15.189:0/2708608248 wait complete. 2024-09-16T14:45:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T14:45:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c osd.7 2024-09-16T14:45:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 2 2024-09-16T14:45:57.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd8294640 10 monclient: get_monmap_and_config 2024-09-16T14:45:57.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd8294640 10 monclient: build_initial_monmap 2024-09-16T14:45:57.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd8294640 10 monclient: monmap: 2024-09-16T14:45:57.400 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:57.400 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:57.400 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:57.400023+0000 2024-09-16T14:45:57.400 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:57.400023+0000 2024-09-16T14:45:57.400 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:57.401 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:57.401 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:57.401 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:57.401 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:57.401 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd8294640 1 Processor -- start 2024-09-16T14:45:57.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd8294640 1 -- start start 2024-09-16T14:45:57.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd8294640 10 monclient: init 2024-09-16T14:45:57.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd8294640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:57.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd8294640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:45:57.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd8294640 10 monclient(hunting): picked mon.noname-b con 0x7f4cd0099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:57.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd8294640 10 monclient(hunting): picked mon.noname-a con 0x7f4cd01334d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:57.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd8294640 10 monclient(hunting): picked mon.noname-c con 0x7f4cd0136fe0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:57.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd8294640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd009b350 con 0x7f4cd01334d0 2024-09-16T14:45:57.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd8294640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd009f2e0 con 0x7f4cd0136fe0 2024-09-16T14:45:57.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd8294640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd009b070 con 0x7f4cd0099a70 2024-09-16T14:45:57.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd8294640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:57.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd8294640 10 monclient(hunting): authenticate will time out at 1139.515733s 2024-09-16T14:45:57.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd582f640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f4cd01334d0 legacy=0x7f4cd01358c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:57.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd6831640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4cd0136fe0 0x7f4cd01393d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47138/0 (socket says 172.21.15.189:47138) 2024-09-16T14:45:57.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.396+0000 7f4cd6831640 1 -- 172.21.15.189:0/2906970799 learned_addr learned my addr 172.21.15.189:0/2906970799 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:57.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 1 -- 172.21.15.189:0/2906970799 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611657211 0 0) 0x7f4cd009f2e0 con 0x7f4cd0136fe0 2024-09-16T14:45:57.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 10 monclient(hunting): _negotiate 2024-09-16T14:45:57.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:57.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 10 monclient(hunting): my global_id is 94360 2024-09-16T14:45:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 1 -- 172.21.15.189:0/2906970799 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cbc003530 con 0x7f4cd0136fe0 2024-09-16T14:45:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 1 -- 172.21.15.189:0/2906970799 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2332608658 0 0) 0x7f4cbc003530 con 0x7f4cd0136fe0 2024-09-16T14:45:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 1 -- 172.21.15.189:0/2906970799 >> v1:172.21.15.26:6790/0 conn(0x7f4cd0099a70 legacy=0x7f4cd009bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 1 -- 172.21.15.189:0/2906970799 >> v1:172.21.15.26:6789/0 conn(0x7f4cd01334d0 legacy=0x7f4cd01358c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 10 monclient: _finish_hunting 0 2024-09-16T14:45:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 1 monclient: found mon.noname-c 2024-09-16T14:45:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 1 -- 172.21.15.189:0/2906970799 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd009b630 con 0x7f4cd0136fe0 2024-09-16T14:45:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 10 monclient: _finish_auth 0 2024-09-16T14:45:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 10 monclient: _check_auth_tickets 2024-09-16T14:45:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:57.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 1 -- 172.21.15.189:0/2906970799 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4cc80019a0 con 0x7f4cd0136fe0 2024-09-16T14:45:57.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 5 monclient: authenticate success, global_id 94360 2024-09-16T14:45:57.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:57.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:57.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:57.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 10 monclient: dump: 2024-09-16T14:45:57.410 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:57.410 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:57.410 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.410 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.410 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:57.410 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:57.411 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:57.411 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:57.411 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:57.411 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:57.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 1 -- 172.21.15.189:0/2906970799 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cc8001c50 con 0x7f4cd0136fe0 2024-09-16T14:45:57.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:57.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 1 -- 172.21.15.189:0/2906970799 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4cc8002020 con 0x7f4cd0136fe0 2024-09-16T14:45:57.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:57.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:57.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd502e640 10 monclient: dump: 2024-09-16T14:45:57.411 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:57.411 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:57.411 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.412 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.412 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:57.412 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:57.412 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:57.412 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:57.412 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:57.412 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:57.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 4 set_mon_vals no callback set 2024-09-16T14:45:57.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 10 monclient: shutdown 2024-09-16T14:45:57.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:57.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 1 -- 172.21.15.189:0/2906970799 >> v1:172.21.15.189:6789/0 conn(0x7f4cd0136fe0 legacy=0x7f4cd01393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:57.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 1 -- 172.21.15.189:0/2906970799 shutdown_connections 2024-09-16T14:45:57.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 1 -- 172.21.15.189:0/2906970799 >> 172.21.15.189:0/2906970799 conn(0x7f4cd00a3320 msgr2=0x7f4cd009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:57.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 1 -- 172.21.15.189:0/2906970799 shutdown_connections 2024-09-16T14:45:57.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 1 -- 172.21.15.189:0/2906970799 wait complete. 2024-09-16T14:45:57.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 10 monclient: build_initial_monmap 2024-09-16T14:45:57.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 10 monclient: monmap: 2024-09-16T14:45:57.413 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:57.413 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:57.413 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:57.413 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:57.413 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:57.413 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:57.413 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:57.413 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:57.414 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:57.414 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 1 Processor -- start 2024-09-16T14:45:57.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 1 -- start start 2024-09-16T14:45:57.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 10 monclient: init 2024-09-16T14:45:57.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:57.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:57.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 10 monclient(hunting): picked mon.noname-c con 0x7f4cd0099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:57.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 10 monclient(hunting): picked mon.noname-b con 0x7f4cd01334d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:57.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 10 monclient(hunting): picked mon.noname-a con 0x7f4cd0136fe0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:57.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd0136be0 con 0x7f4cd0136fe0 2024-09-16T14:45:57.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd006b380 con 0x7f4cd01334d0 2024-09-16T14:45:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd0121df0 con 0x7f4cd0099a70 2024-09-16T14:45:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 10 monclient(hunting): authenticate will time out at 1139.518562s 2024-09-16T14:45:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd582f640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4cd01334d0 0x7f4cd01326f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47154/0 (socket says 172.21.15.189:47154) 2024-09-16T14:45:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd582f640 1 -- 172.21.15.189:0/3508762705 learned_addr learned my addr 172.21.15.189:0/3508762705 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd6831640 1 -- 172.21.15.189:0/3508762705 >> v1:172.21.15.26:6789/0 conn(0x7f4cd0136fe0 legacy=0x7f4cd01d15e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 1 -- 172.21.15.189:0/3508762705 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266613147 0 0) 0x7f4cd006b380 con 0x7f4cd01334d0 2024-09-16T14:45:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 10 monclient(hunting): my global_id is 94363 2024-09-16T14:45:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 1 -- 172.21.15.189:0/3508762705 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb4003530 con 0x7f4cd01334d0 2024-09-16T14:45:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 1 -- 172.21.15.189:0/3508762705 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3697046301 0 0) 0x7f4cb4003530 con 0x7f4cd01334d0 2024-09-16T14:45:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 1 -- 172.21.15.189:0/3508762705 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cd006b380 con 0x7f4cd01334d0 2024-09-16T14:45:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 1 -- 172.21.15.189:0/3508762705 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4ccc004190 con 0x7f4cd01334d0 2024-09-16T14:45:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 10 monclient(hunting): dump: 2024-09-16T14:45:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:57.417 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:57.417 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:57.417 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:57.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 1 -- 172.21.15.189:0/3508762705 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 572002309 0 0) 0x7f4cd006b380 con 0x7f4cd01334d0 2024-09-16T14:45:57.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 1 -- 172.21.15.189:0/3508762705 >> v1:172.21.15.26:6790/0 conn(0x7f4cd0099a70 legacy=0x7f4cd01c9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:57.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 1 -- 172.21.15.189:0/3508762705 >> v1:172.21.15.26:6789/0 conn(0x7f4cd0136fe0 legacy=0x7f4cd01d15e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:57.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:57.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 1 monclient: found mon.b 2024-09-16T14:45:57.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:57.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 1 -- 172.21.15.189:0/3508762705 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd0121830 con 0x7f4cd01334d0 2024-09-16T14:45:57.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:57.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:57.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:57.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 5 monclient: authenticate success, global_id 94363 2024-09-16T14:45:57.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 10 monclient: _renew_subs 2024-09-16T14:45:57.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 1 -- 172.21.15.189:0/3508762705 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd0135440 con 0x7f4cd01334d0 2024-09-16T14:45:57.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 10 monclient: _renew_subs 2024-09-16T14:45:57.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd8294640 1 -- 172.21.15.189:0/3508762705 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd0132e00 con 0x7f4cd01334d0 2024-09-16T14:45:57.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 1 -- 172.21.15.189:0/3508762705 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ccc0043b0 con 0x7f4cd01334d0 2024-09-16T14:45:57.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:57.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 1 -- 172.21.15.189:0/3508762705 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4ccc004740 con 0x7f4cd01334d0 2024-09-16T14:45:57.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:57.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:57.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 10 monclient: dump: 2024-09-16T14:45:57.419 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:57.419 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:57.419 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.419 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.419 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:57.419 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:57.419 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:57.419 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:57.419 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:57.419 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cd482d640 4 set_mon_vals no callback set 2024-09-16T14:45:57.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.400+0000 7f4cc6ffd640 1 -- 172.21.15.189:0/3508762705 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4ccc004920 con 0x7f4cd01334d0 2024-09-16T14:45:57.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.404+0000 7f4cd8294640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:57.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.404+0000 7f4cd8294640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:57.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.404+0000 7f4cd8294640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.404+0000 7f4cc6ffd640 1 -- 172.21.15.189:0/3508762705 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(73..73 src has 15..73) v4 ==== 5385+0+0 (unknown 4089813478 0 0) 0x7f4ccc0438c0 con 0x7f4cd01334d0 2024-09-16T14:45:57.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.404+0000 7f4cd8294640 1 -- 172.21.15.189:0/3508762705 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb8005050 con 0x7f4cd01334d0 2024-09-16T14:45:57.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.404+0000 7f4cd6030640 10 monclient: get_auth_request con 0x7f4cb4036080 auth_method 0 2024-09-16T14:45:57.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.408+0000 7f4cc6ffd640 1 -- 172.21.15.189:0/3508762705 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4ccc014750 con 0x7f4cd01334d0 2024-09-16T14:45:57.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.408+0000 7f4cc6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:57.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.408+0000 7f4cc6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:57.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.556+0000 7f4cd8294640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:45:57.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.556+0000 7f4cd8294640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T14:45:57.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.556+0000 7f4cd8294640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.556+0000 7f4cd8294640 1 -- 172.21.15.189:0/3508762705 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f4cb8005b10 con 0x7f4cd01334d0 2024-09-16T14:45:57.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.556+0000 7f4cc6ffd640 1 -- 172.21.15.189:0/3508762705 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v73) v1 ==== 56+0+1474 (unknown 4132081119 0 260673783) 0x7f4ccc012bc0 con 0x7f4cd01334d0 2024-09-16T14:45:57.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.556+0000 7f4cc6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T14:45:57.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.556+0000 7f4cc6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.644+0000 7f4cd8294640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:45:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.644+0000 7f4cd8294640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T14:45:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.644+0000 7f4cd8294640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.644+0000 7f4cd8294640 1 -- 172.21.15.189:0/3508762705 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f4cb8005850 con 0x7f4cd01334d0 2024-09-16T14:45:57.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.644+0000 7f4cc6ffd640 1 -- 172.21.15.189:0/3508762705 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v73) v1 ==== 56+0+1474 (unknown 4132081119 0 260673783) 0x7f4ccc014340 con 0x7f4cd01334d0 2024-09-16T14:45:57.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.644+0000 7f4cc6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T14:45:57.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.644+0000 7f4cc6ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:45:57.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.644+0000 7f4cd8294640 1 -- 172.21.15.189:0/3508762705 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4cb4036080 legacy=0x7f4cb4038540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:57.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.644+0000 7f4cd8294640 10 monclient: shutdown 2024-09-16T14:45:57.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.644+0000 7f4cd8294640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:57.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.644+0000 7f4cd8294640 1 -- 172.21.15.189:0/3508762705 >> v1:172.21.15.189:6789/0 conn(0x7f4cd01334d0 legacy=0x7f4cd01326f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:57.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.644+0000 7f4cd8294640 1 -- 172.21.15.189:0/3508762705 shutdown_connections 2024-09-16T14:45:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.644+0000 7f4cd8294640 1 -- 172.21.15.189:0/3508762705 >> 172.21.15.189:0/3508762705 conn(0x7f4cd00a3320 msgr2=0x7f4cd0070ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.644+0000 7f4cd8294640 1 -- 172.21.15.189:0/3508762705 shutdown_connections 2024-09-16T14:45:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.644+0000 7f4cd8294640 1 -- 172.21.15.189:0/3508762705 wait complete. 2024-09-16T14:45:57.663 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T14:45:57.663 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c host1 2024-09-16T14:45:57.663 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 2 2024-09-16T14:45:57.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.772+0000 7f4a6e675640 10 monclient: get_monmap_and_config 2024-09-16T14:45:57.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.772+0000 7f4a6e675640 10 monclient: build_initial_monmap 2024-09-16T14:45:57.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.772+0000 7f4a6e675640 10 monclient: monmap: 2024-09-16T14:45:57.777 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:57.777 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:57.777 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:57.776425+0000 2024-09-16T14:45:57.777 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:57.776425+0000 2024-09-16T14:45:57.777 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:57.777 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:57.777 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:57.777 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:57.777 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:57.777 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.772+0000 7f4a6e675640 1 Processor -- start 2024-09-16T14:45:57.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 1 -- start start 2024-09-16T14:45:57.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient: init 2024-09-16T14:45:57.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:57.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:57.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient(hunting): picked mon.noname-c con 0x7f4a681315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:57.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient(hunting): picked mon.noname-b con 0x7f4a6809b320 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:57.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient(hunting): picked mon.noname-a con 0x7f4a68132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:57.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6809edd0 con 0x7f4a68132090 2024-09-16T14:45:57.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6809bc00 con 0x7f4a681315a0 2024-09-16T14:45:57.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6809e900 con 0x7f4a6809b320 2024-09-16T14:45:57.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:57.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient(hunting): authenticate will time out at 1139.892801s 2024-09-16T14:45:57.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a67fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4a681315a0 0x7f4a68131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47166/0 (socket says 172.21.15.189:47166) 2024-09-16T14:45:57.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a67fff640 1 -- 172.21.15.189:0/957246022 learned_addr learned my addr 172.21.15.189:0/957246022 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:57.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6cc12640 1 -- 172.21.15.189:0/957246022 >> v1:172.21.15.26:6789/0 conn(0x7f4a68132090 legacy=0x7f4a6813a200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:57.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 1 -- 172.21.15.189:0/957246022 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191203244 0 0) 0x7f4a6809bc00 con 0x7f4a681315a0 2024-09-16T14:45:57.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:57.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:57.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:57.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 10 monclient(hunting): my global_id is 94366 2024-09-16T14:45:57.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 1 -- 172.21.15.189:0/957246022 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a50003530 con 0x7f4a681315a0 2024-09-16T14:45:57.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 1 -- 172.21.15.189:0/957246022 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1574109523 0 0) 0x7f4a50003530 con 0x7f4a681315a0 2024-09-16T14:45:57.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 1 -- 172.21.15.189:0/957246022 >> v1:172.21.15.26:6790/0 conn(0x7f4a6809b320 legacy=0x7f4a68136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:57.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 1 -- 172.21.15.189:0/957246022 >> v1:172.21.15.26:6789/0 conn(0x7f4a68132090 legacy=0x7f4a6813a200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:57.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:57.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 1 monclient: found mon.noname-c 2024-09-16T14:45:57.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:57.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 1 -- 172.21.15.189:0/957246022 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6809b920 con 0x7f4a681315a0 2024-09-16T14:45:57.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:57.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:57.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:57.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 5 monclient: authenticate success, global_id 94366 2024-09-16T14:45:57.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:57.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 1 -- 172.21.15.189:0/957246022 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4a58001620 con 0x7f4a681315a0 2024-09-16T14:45:57.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:57.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:57.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 10 monclient: dump: 2024-09-16T14:45:57.783 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:57.783 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:57.783 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.784 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.784 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:57.784 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:57.784 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:57.784 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:57.784 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:57.784 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:57.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 1 -- 172.21.15.189:0/957246022 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a58001940 con 0x7f4a681315a0 2024-09-16T14:45:57.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:57.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:57.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 4 set_mon_vals no callback set 2024-09-16T14:45:57.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 1 -- 172.21.15.189:0/957246022 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4a58001d10 con 0x7f4a681315a0 2024-09-16T14:45:57.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:57.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:57.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a66ffd640 10 monclient: dump: 2024-09-16T14:45:57.785 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:57.785 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:57.785 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.785 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.785 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:57.785 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:57.785 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:57.785 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:57.785 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:57.785 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient: shutdown 2024-09-16T14:45:57.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:57.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 1 -- 172.21.15.189:0/957246022 >> v1:172.21.15.189:6789/0 conn(0x7f4a681315a0 legacy=0x7f4a68131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:57.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 1 -- 172.21.15.189:0/957246022 shutdown_connections 2024-09-16T14:45:57.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 1 -- 172.21.15.189:0/957246022 >> 172.21.15.189:0/957246022 conn(0x7f4a680a3320 msgr2=0x7f4a6809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:57.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 1 -- 172.21.15.189:0/957246022 shutdown_connections 2024-09-16T14:45:57.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 1 -- 172.21.15.189:0/957246022 wait complete. 2024-09-16T14:45:57.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient: build_initial_monmap 2024-09-16T14:45:57.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient: monmap: 2024-09-16T14:45:57.790 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:57.790 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:57.790 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:57.790 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:57.790 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:57.790 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:57.790 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:57.790 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:57.790 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:57.790 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 1 Processor -- start 2024-09-16T14:45:57.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 1 -- start start 2024-09-16T14:45:57.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient: init 2024-09-16T14:45:57.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:57.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:45:57.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient(hunting): picked mon.noname-c con 0x7f4a681315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:57.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient(hunting): picked mon.noname-a con 0x7f4a68132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:57.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient(hunting): picked mon.noname-b con 0x7f4a68130130 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:57.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6809cd20 con 0x7f4a68132090 2024-09-16T14:45:57.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6809aac0 con 0x7f4a68130130 2024-09-16T14:45:57.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6806b380 con 0x7f4a681315a0 2024-09-16T14:45:57.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:57.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.776+0000 7f4a6e675640 10 monclient(hunting): authenticate will time out at 1139.895931s 2024-09-16T14:45:57.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a677fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f4a68132090 legacy=0x7f4a681ce2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:57.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a6cc12640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4a68130130 0x7f4a681d1a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47172/0 (socket says 172.21.15.189:47172) 2024-09-16T14:45:57.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a6cc12640 1 -- 172.21.15.189:0/3305286645 learned_addr learned my addr 172.21.15.189:0/3305286645 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:57.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 1 -- 172.21.15.189:0/3305286645 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609263349 0 0) 0x7f4a6809aac0 con 0x7f4a68130130 2024-09-16T14:45:57.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:45:57.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:57.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:57.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 10 monclient(hunting): my global_id is 94369 2024-09-16T14:45:57.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 1 -- 172.21.15.189:0/3305286645 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a54003530 con 0x7f4a68130130 2024-09-16T14:45:57.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 1 -- 172.21.15.189:0/3305286645 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1381567710 0 0) 0x7f4a54003530 con 0x7f4a68130130 2024-09-16T14:45:57.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 1 -- 172.21.15.189:0/3305286645 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a6809aac0 con 0x7f4a68130130 2024-09-16T14:45:57.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 1 -- 172.21.15.189:0/3305286645 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4a600043c0 con 0x7f4a68130130 2024-09-16T14:45:57.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:57.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:57.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 10 monclient(hunting): dump: 2024-09-16T14:45:57.793 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:57.793 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:57.793 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.793 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.793 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:57.793 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:57.793 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:57.793 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:57.793 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:57.793 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:57.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 1 -- 172.21.15.189:0/3305286645 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3446343071 0 0) 0x7f4a6809aac0 con 0x7f4a68130130 2024-09-16T14:45:57.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 1 -- 172.21.15.189:0/3305286645 >> v1:172.21.15.26:6790/0 conn(0x7f4a681315a0 legacy=0x7f4a6812fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:57.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 1 -- 172.21.15.189:0/3305286645 >> v1:172.21.15.26:6789/0 conn(0x7f4a68132090 legacy=0x7f4a681ce2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:57.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:45:57.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 1 monclient: found mon.b 2024-09-16T14:45:57.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:57.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 1 -- 172.21.15.189:0/3305286645 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6811f070 con 0x7f4a68130130 2024-09-16T14:45:57.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 10 monclient: _finish_auth 0 2024-09-16T14:45:57.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:45:57.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:57.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a6e675640 5 monclient: authenticate success, global_id 94369 2024-09-16T14:45:57.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a6e675640 10 monclient: _renew_subs 2024-09-16T14:45:57.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a6e675640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a6e675640 1 -- 172.21.15.189:0/3305286645 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a68121160 con 0x7f4a68130130 2024-09-16T14:45:57.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a6e675640 10 monclient: _renew_subs 2024-09-16T14:45:57.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a6e675640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a6e675640 1 -- 172.21.15.189:0/3305286645 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6811f7c0 con 0x7f4a68130130 2024-09-16T14:45:57.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 1 -- 172.21.15.189:0/3305286645 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a600046e0 con 0x7f4a68130130 2024-09-16T14:45:57.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:57.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 1 -- 172.21.15.189:0/3305286645 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4a60004a70 con 0x7f4a68130130 2024-09-16T14:45:57.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:57.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a667fc640 4 set_mon_vals no callback set 2024-09-16T14:45:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 10 monclient: dump: 2024-09-16T14:45:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:57.797 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:57.797 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:57.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 1 -- 172.21.15.189:0/3305286645 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4a60004c50 con 0x7f4a68130130 2024-09-16T14:45:57.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a6e675640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:57.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a6e675640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:57.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a6e675640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a6e675640 1 -- 172.21.15.189:0/3305286645 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a48005050 con 0x7f4a68130130 2024-09-16T14:45:57.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a64ff9640 1 -- 172.21.15.189:0/3305286645 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(73..73 src has 15..73) v4 ==== 5385+0+0 (unknown 4089813478 0 0) 0x7f4a60043c20 con 0x7f4a68130130 2024-09-16T14:45:57.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.780+0000 7f4a67fff640 10 monclient: get_auth_request con 0x7f4a54036000 auth_method 0 2024-09-16T14:45:57.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.784+0000 7f4a64ff9640 1 -- 172.21.15.189:0/3305286645 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4a60014ab0 con 0x7f4a68130130 2024-09-16T14:45:57.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.784+0000 7f4a64ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:57.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.784+0000 7f4a64ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:57.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.948+0000 7f4a6e675640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:45:57.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.948+0000 7f4a6e675640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T14:45:57.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.948+0000 7f4a6e675640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:57.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.948+0000 7f4a6e675640 1 -- 172.21.15.189:0/3305286645 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f4a48005b10 con 0x7f4a68130130 2024-09-16T14:45:57.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.948+0000 7f4a64ff9640 1 -- 172.21.15.189:0/3305286645 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v73) v1 ==== 56+0+1474 (unknown 4132081119 0 260673783) 0x7f4a60012f20 con 0x7f4a68130130 2024-09-16T14:45:57.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.948+0000 7f4a64ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T14:45:57.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:57.948+0000 7f4a64ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.048+0000 7f4a6e675640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:45:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.048+0000 7f4a6e675640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T14:45:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.048+0000 7f4a6e675640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.048+0000 7f4a6e675640 1 -- 172.21.15.189:0/3305286645 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f4a48005850 con 0x7f4a68130130 2024-09-16T14:45:58.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.048+0000 7f4a64ff9640 1 -- 172.21.15.189:0/3305286645 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v73) v1 ==== 56+0+1474 (unknown 4132081119 0 260673783) 0x7f4a600146a0 con 0x7f4a68130130 2024-09-16T14:45:58.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.048+0000 7f4a64ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T14:45:58.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.048+0000 7f4a64ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:45:58.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.048+0000 7f4a6e675640 1 -- 172.21.15.189:0/3305286645 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4a54036000 legacy=0x7f4a540384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:58.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.048+0000 7f4a6e675640 10 monclient: shutdown 2024-09-16T14:45:58.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.048+0000 7f4a6e675640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:58.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.048+0000 7f4a6e675640 1 -- 172.21.15.189:0/3305286645 >> v1:172.21.15.189:6789/0 conn(0x7f4a68130130 legacy=0x7f4a681d1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:58.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.048+0000 7f4a6e675640 1 -- 172.21.15.189:0/3305286645 shutdown_connections 2024-09-16T14:45:58.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.048+0000 7f4a6e675640 1 -- 172.21.15.189:0/3305286645 >> 172.21.15.189:0/3305286645 conn(0x7f4a680a3320 msgr2=0x7f4a680a1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:58.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.048+0000 7f4a6e675640 1 -- 172.21.15.189:0/3305286645 shutdown_connections 2024-09-16T14:45:58.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.048+0000 7f4a6e675640 1 -- 172.21.15.189:0/3305286645 wait complete. 2024-09-16T14:45:58.068 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T14:45:58.068 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c osd.9 2024-09-16T14:45:58.068 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 2 2024-09-16T14:45:58.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.172+0000 7f9cc33da640 10 monclient: get_monmap_and_config 2024-09-16T14:45:58.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.172+0000 7f9cc33da640 10 monclient: build_initial_monmap 2024-09-16T14:45:58.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.172+0000 7f9cc33da640 10 monclient: monmap: 2024-09-16T14:45:58.177 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:58.177 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:58.178 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:58.177325+0000 2024-09-16T14:45:58.178 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:58.177325+0000 2024-09-16T14:45:58.178 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:58.178 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:58.178 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:58.178 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:58.178 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:58.178 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 1 Processor -- start 2024-09-16T14:45:58.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 1 -- start start 2024-09-16T14:45:58.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 10 monclient: init 2024-09-16T14:45:58.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:58.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:45:58.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 10 monclient(hunting): picked mon.noname-b con 0x7f9cbc1318a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:58.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 10 monclient(hunting): picked mon.noname-c con 0x7f9cbc134d50 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:58.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 10 monclient(hunting): picked mon.noname-a con 0x7f9cbc13b2b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:58.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cbc09d8a0 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cbc09ddf0 con 0x7f9cbc134d50 2024-09-16T14:45:58.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cbc09e8b0 con 0x7f9cbc1318a0 2024-09-16T14:45:58.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:58.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 10 monclient(hunting): authenticate will time out at 1140.292828s 2024-09-16T14:45:58.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc0975640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9cbc134d50 0x7f9cbc139b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47186/0 (socket says 172.21.15.189:47186) 2024-09-16T14:45:58.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc0975640 1 -- 172.21.15.189:0/3075153109 learned_addr learned my addr 172.21.15.189:0/3075153109 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:58.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc1977640 1 -- 172.21.15.189:0/3075153109 >> v1:172.21.15.26:6789/0 conn(0x7f9cbc13b2b0 legacy=0x7f9cbc13d6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:58.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 1 -- 172.21.15.189:0/3075153109 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881869019 0 0) 0x7f9cbc09ddf0 con 0x7f9cbc134d50 2024-09-16T14:45:58.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:58.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:58.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:58.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 10 monclient(hunting): my global_id is 94372 2024-09-16T14:45:58.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 1 -- 172.21.15.189:0/3075153109 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ca8003530 con 0x7f9cbc134d50 2024-09-16T14:45:58.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 1 -- 172.21.15.189:0/3075153109 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4203503275 0 0) 0x7f9ca8003530 con 0x7f9cbc134d50 2024-09-16T14:45:58.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 1 -- 172.21.15.189:0/3075153109 >> v1:172.21.15.26:6790/0 conn(0x7f9cbc1318a0 legacy=0x7f9cbc131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:58.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 1 -- 172.21.15.189:0/3075153109 >> v1:172.21.15.26:6789/0 conn(0x7f9cbc13b2b0 legacy=0x7f9cbc13d6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:58.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:45:58.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 1 monclient: found mon.noname-c 2024-09-16T14:45:58.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:58.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 1 -- 172.21.15.189:0/3075153109 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cbc09eb90 con 0x7f9cbc134d50 2024-09-16T14:45:58.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 10 monclient: _finish_auth 0 2024-09-16T14:45:58.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:45:58.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:58.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 1 -- 172.21.15.189:0/3075153109 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9cac001620 con 0x7f9cbc134d50 2024-09-16T14:45:58.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 5 monclient: authenticate success, global_id 94372 2024-09-16T14:45:58.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:58.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:58.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:58.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 10 monclient: dump: 2024-09-16T14:45:58.188 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:58.188 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:58.188 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.188 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.188 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:58.188 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:58.189 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:58.189 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:58.189 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:58.189 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:58.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 1 -- 172.21.15.189:0/3075153109 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cac0018d0 con 0x7f9cbc134d50 2024-09-16T14:45:58.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:58.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:58.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 1 -- 172.21.15.189:0/3075153109 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9cac001c60 con 0x7f9cbc134d50 2024-09-16T14:45:58.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 4 set_mon_vals no callback set 2024-09-16T14:45:58.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:58.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:58.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cb3fff640 10 monclient: dump: 2024-09-16T14:45:58.190 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:58.190 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:58.190 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.190 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.190 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:58.190 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:58.190 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:58.190 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:58.190 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:58.191 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 10 monclient: shutdown 2024-09-16T14:45:58.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:58.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 1 -- 172.21.15.189:0/3075153109 >> v1:172.21.15.189:6789/0 conn(0x7f9cbc134d50 legacy=0x7f9cbc139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:58.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 1 -- 172.21.15.189:0/3075153109 shutdown_connections 2024-09-16T14:45:58.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 1 -- 172.21.15.189:0/3075153109 >> 172.21.15.189:0/3075153109 conn(0x7f9cbc0a3320 msgr2=0x7f9cbc09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:58.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 1 -- 172.21.15.189:0/3075153109 shutdown_connections 2024-09-16T14:45:58.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 1 -- 172.21.15.189:0/3075153109 wait complete. 2024-09-16T14:45:58.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 10 monclient: build_initial_monmap 2024-09-16T14:45:58.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 10 monclient: monmap: 2024-09-16T14:45:58.191 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:58.192 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:58.192 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:58.192 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:58.192 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:58.192 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:58.192 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:58.192 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:58.192 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:58.192 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 1 Processor -- start 2024-09-16T14:45:58.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 1 -- start start 2024-09-16T14:45:58.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.176+0000 7f9cc33da640 10 monclient: init 2024-09-16T14:45:58.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:58.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:45:58.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 10 monclient(hunting): picked mon.noname-c con 0x7f9cbc1318a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:58.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 10 monclient(hunting): picked mon.noname-a con 0x7f9cbc134d50 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:58.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 10 monclient(hunting): picked mon.noname-b con 0x7f9cbc13b2b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:58.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cbc119680 con 0x7f9cbc134d50 2024-09-16T14:45:58.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cbc06b380 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cbc11b9e0 con 0x7f9cbc1318a0 2024-09-16T14:45:58.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:58.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 10 monclient(hunting): authenticate will time out at 1140.296204s 2024-09-16T14:45:58.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc1977640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9cbc13b2b0 0x7f9cbc1d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47190/0 (socket says 172.21.15.189:47190) 2024-09-16T14:45:58.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc1977640 1 -- 172.21.15.189:0/983949453 learned_addr learned my addr 172.21.15.189:0/983949453 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:58.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc0975640 1 -- 172.21.15.189:0/983949453 >> v1:172.21.15.26:6789/0 conn(0x7f9cbc134d50 legacy=0x7f9cbc1ce670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:58.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3591999104 0 0) 0x7f9cbc06b380 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:58.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:58.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:58.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient(hunting): my global_id is 94375 2024-09-16T14:45:58.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ca0003530 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1118984839 0 0) 0x7f9ca0003530 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cbc06b380 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9cb4004390 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:58.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:58.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient(hunting): dump: 2024-09-16T14:45:58.195 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:58.195 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:58.195 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.195 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.195 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:58.195 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:58.195 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:58.196 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:58.196 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:58.196 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:58.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383811271 0 0) 0x7f9cbc11b9e0 con 0x7f9cbc1318a0 2024-09-16T14:45:58.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:58.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:58.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:58.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient(hunting): my global_id is 5258 2024-09-16T14:45:58.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ca0003530 con 0x7f9cbc1318a0 2024-09-16T14:45:58.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3120963938 0 0) 0x7f9cbc06b380 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 >> v1:172.21.15.26:6790/0 conn(0x7f9cbc1318a0 legacy=0x7f9cbc1c9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:58.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 >> v1:172.21.15.26:6789/0 conn(0x7f9cbc134d50 legacy=0x7f9cbc1ce670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:58.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:45:58.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 1 monclient: found mon.b 2024-09-16T14:45:58.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:58.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cbc0a1e70 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:45:58.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:58.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:58.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 5 monclient: authenticate success, global_id 94375 2024-09-16T14:45:58.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 10 monclient: _renew_subs 2024-09-16T14:45:58.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 1 -- 172.21.15.189:0/983949453 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cbc13ac60 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 10 monclient: _renew_subs 2024-09-16T14:45:58.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 1 -- 172.21.15.189:0/983949453 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cbc11b700 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cb40046b0 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:58.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9cb4004a40 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:58.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb37fe640 4 set_mon_vals no callback set 2024-09-16T14:45:58.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:58.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 10 monclient: dump: 2024-09-16T14:45:58.199 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:58.199 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:58.199 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.199 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.199 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:58.199 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:58.199 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:58.199 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:58.199 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:58.200 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:58.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:58.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc33da640 1 -- 172.21.15.189:0/983949453 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9c005050 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9cb4004c20 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(73..73 src has 15..73) v4 ==== 5385+0+0 (unknown 4089813478 0 0) 0x7f9cb4043c30 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.180+0000 7f9cc1176640 10 monclient: get_auth_request con 0x7f9ca00360e0 auth_method 0 2024-09-16T14:45:58.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.184+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9cb4014ac0 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.184+0000 7f9cb1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:58.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.184+0000 7f9cb1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:58.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.332+0000 7f9cc33da640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:45:58.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.332+0000 7f9cc33da640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T14:45:58.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.332+0000 7f9cc33da640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.332+0000 7f9cc33da640 1 -- 172.21.15.189:0/983949453 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f9c9c005b10 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.332+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v73) v1 ==== 56+0+1474 (unknown 4132081119 0 260673783) 0x7f9cb4012f30 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.332+0000 7f9cb1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T14:45:58.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.332+0000 7f9cb1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:58.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.428+0000 7f9cc33da640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:45:58.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.428+0000 7f9cc33da640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T14:45:58.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.428+0000 7f9cc33da640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.428+0000 7f9cc33da640 1 -- 172.21.15.189:0/983949453 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f9c9c0055a0 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.428+0000 7f9cb1ffb640 1 -- 172.21.15.189:0/983949453 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v73) v1 ==== 56+0+1474 (unknown 4132081119 0 260673783) 0x7f9cb40146b0 con 0x7f9cbc13b2b0 2024-09-16T14:45:58.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.428+0000 7f9cb1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T14:45:58.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.428+0000 7f9cb1ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:45:58.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.428+0000 7f9cc33da640 1 -- 172.21.15.189:0/983949453 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9ca00360e0 legacy=0x7f9ca00385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:58.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.428+0000 7f9cc33da640 10 monclient: shutdown 2024-09-16T14:45:58.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.428+0000 7f9cc33da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:58.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.428+0000 7f9cc33da640 1 -- 172.21.15.189:0/983949453 >> v1:172.21.15.189:6789/0 conn(0x7f9cbc13b2b0 legacy=0x7f9cbc1d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:58.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.432+0000 7f9cc33da640 1 -- 172.21.15.189:0/983949453 shutdown_connections 2024-09-16T14:45:58.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.432+0000 7f9cc33da640 1 -- 172.21.15.189:0/983949453 >> 172.21.15.189:0/983949453 conn(0x7f9cbc0a3320 msgr2=0x7f9cbc134140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:58.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.432+0000 7f9cc33da640 1 -- 172.21.15.189:0/983949453 shutdown_connections 2024-09-16T14:45:58.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.432+0000 7f9cc33da640 1 -- 172.21.15.189:0/983949453 wait complete. 2024-09-16T14:45:58.448 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T14:45:58.448 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c host2 2024-09-16T14:45:58.449 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 2 2024-09-16T14:45:58.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.556+0000 7f75d080d640 10 monclient: get_monmap_and_config 2024-09-16T14:45:58.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.556+0000 7f75d080d640 10 monclient: build_initial_monmap 2024-09-16T14:45:58.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.556+0000 7f75d080d640 10 monclient: monmap: 2024-09-16T14:45:58.561 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:58.561 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:58.561 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:58.561214+0000 2024-09-16T14:45:58.561 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:58.561214+0000 2024-09-16T14:45:58.561 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:58.561 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:58.562 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:58.562 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:58.562 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:58.562 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 1 Processor -- start 2024-09-16T14:45:58.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 1 -- start start 2024-09-16T14:45:58.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient: init 2024-09-16T14:45:58.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:58.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:45:58.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient(hunting): picked mon.noname-b con 0x7f75cc0a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:58.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient(hunting): picked mon.noname-a con 0x7f75cc0a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:58.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient(hunting): picked mon.noname-c con 0x7f75cc13b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:58.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc0a30b0 con 0x7f75cc0a25f0 2024-09-16T14:45:58.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc0a2e30 con 0x7f75cc13b430 2024-09-16T14:45:58.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc09eb50 con 0x7f75cc0a1b00 2024-09-16T14:45:58.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:58.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient(hunting): authenticate will time out at 1140.676803s 2024-09-16T14:45:58.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c3fff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f75cc0a25f0 legacy=0x7f75cc139d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:58.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75cad9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f75cc13b430 0x7f75cc13d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47204/0 (socket says 172.21.15.189:47204) 2024-09-16T14:45:58.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75cad9d640 1 -- 172.21.15.189:0/2522061028 learned_addr learned my addr 172.21.15.189:0/2522061028 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:58.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 1 -- 172.21.15.189:0/2522061028 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328244420 0 0) 0x7f75cc0a2e30 con 0x7f75cc13b430 2024-09-16T14:45:58.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:45:58.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:58.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:58.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 10 monclient(hunting): my global_id is 94378 2024-09-16T14:45:58.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 1 -- 172.21.15.189:0/2522061028 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b8003530 con 0x7f75cc13b430 2024-09-16T14:45:58.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 1 -- 172.21.15.189:0/2522061028 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3179454662 0 0) 0x7f75b8003530 con 0x7f75cc13b430 2024-09-16T14:45:58.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 1 -- 172.21.15.189:0/2522061028 >> v1:172.21.15.26:6790/0 conn(0x7f75cc0a1b00 legacy=0x7f75cc0a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:58.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 1 -- 172.21.15.189:0/2522061028 >> v1:172.21.15.26:6789/0 conn(0x7f75cc0a25f0 legacy=0x7f75cc139d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:58.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:45:58.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 1 monclient: found mon.noname-c 2024-09-16T14:45:58.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:58.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 1 -- 172.21.15.189:0/2522061028 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc09ee30 con 0x7f75cc13b430 2024-09-16T14:45:58.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 10 monclient: _finish_auth 0 2024-09-16T14:45:58.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:45:58.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:58.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 1 -- 172.21.15.189:0/2522061028 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f75c40019a0 con 0x7f75cc13b430 2024-09-16T14:45:58.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 5 monclient: authenticate success, global_id 94378 2024-09-16T14:45:58.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:58.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:58.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 10 monclient: dump: 2024-09-16T14:45:58.568 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:58.568 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:58.568 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.568 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.568 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:58.568 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:58.568 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:58.568 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:58.568 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:58.568 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:58.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 1 -- 172.21.15.189:0/2522061028 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75c4001cc0 con 0x7f75cc13b430 2024-09-16T14:45:58.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:58.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:58.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 4 set_mon_vals no callback set 2024-09-16T14:45:58.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 1 -- 172.21.15.189:0/2522061028 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f75c4002090 con 0x7f75cc13b430 2024-09-16T14:45:58.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:58.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:58.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c9d9b640 10 monclient: dump: 2024-09-16T14:45:58.569 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:58.569 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:58.569 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.569 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.569 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:58.569 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:58.570 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:58.570 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:58.570 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:58.570 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient: shutdown 2024-09-16T14:45:58.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:58.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 1 -- 172.21.15.189:0/2522061028 >> v1:172.21.15.189:6789/0 conn(0x7f75cc13b430 legacy=0x7f75cc13d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:58.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 1 -- 172.21.15.189:0/2522061028 shutdown_connections 2024-09-16T14:45:58.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 1 -- 172.21.15.189:0/2522061028 >> 172.21.15.189:0/2522061028 conn(0x7f75cc0a7dd0 msgr2=0x7f75cc0a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:58.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 1 -- 172.21.15.189:0/2522061028 shutdown_connections 2024-09-16T14:45:58.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 1 -- 172.21.15.189:0/2522061028 wait complete. 2024-09-16T14:45:58.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient: build_initial_monmap 2024-09-16T14:45:58.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient: monmap: 2024-09-16T14:45:58.570 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:58.571 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:58.571 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:58.571 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:58.571 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:58.571 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:58.571 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:58.571 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:58.571 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:58.571 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 1 Processor -- start 2024-09-16T14:45:58.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 1 -- start start 2024-09-16T14:45:58.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient: init 2024-09-16T14:45:58.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:58.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:45:58.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient(hunting): picked mon.noname-a con 0x7f75cc0a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:58.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient(hunting): picked mon.noname-c con 0x7f75cc0a25f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:58.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient(hunting): picked mon.noname-b con 0x7f75cc13b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:58.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc11fc10 con 0x7f75cc0a1b00 2024-09-16T14:45:58.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc11fef0 con 0x7f75cc13b430 2024-09-16T14:45:58.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc1ca3d0 con 0x7f75cc0a25f0 2024-09-16T14:45:58.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:58.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75d080d640 10 monclient(hunting): authenticate will time out at 1140.679667s 2024-09-16T14:45:58.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75cad9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f75cc13b430 0x7f75cc1d1810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47220/0 (socket says 172.21.15.189:47220) 2024-09-16T14:45:58.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75cad9d640 1 -- 172.21.15.189:0/265954658 learned_addr learned my addr 172.21.15.189:0/265954658 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:58.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75ca59c640 1 -- 172.21.15.189:0/265954658 >> v1:172.21.15.26:6789/0 conn(0x7f75cc0a1b00 legacy=0x7f75cc1c95b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:58.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c2ffd640 1 -- 172.21.15.189:0/265954658 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 678946813 0 0) 0x7f75cc11fef0 con 0x7f75cc13b430 2024-09-16T14:45:58.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:58.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:58.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:58.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.560+0000 7f75c2ffd640 10 monclient(hunting): my global_id is 94381 2024-09-16T14:45:58.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 1 -- 172.21.15.189:0/265954658 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75ac003530 con 0x7f75cc13b430 2024-09-16T14:45:58.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 1 -- 172.21.15.189:0/265954658 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2851867267 0 0) 0x7f75ac003530 con 0x7f75cc13b430 2024-09-16T14:45:58.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 1 -- 172.21.15.189:0/265954658 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75cc11fef0 con 0x7f75cc13b430 2024-09-16T14:45:58.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 1 -- 172.21.15.189:0/265954658 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f75c4001960 con 0x7f75cc13b430 2024-09-16T14:45:58.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:58.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:58.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 10 monclient(hunting): dump: 2024-09-16T14:45:58.576 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:58.576 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:58.576 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.576 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.577 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:58.577 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:58.577 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:58.577 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:58.577 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:58.577 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:58.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 1 -- 172.21.15.189:0/265954658 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3473533950 0 0) 0x7f75cc11fef0 con 0x7f75cc13b430 2024-09-16T14:45:58.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 1 -- 172.21.15.189:0/265954658 >> v1:172.21.15.26:6790/0 conn(0x7f75cc0a25f0 legacy=0x7f75cc1c9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:58.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 1 -- 172.21.15.189:0/265954658 >> v1:172.21.15.26:6789/0 conn(0x7f75cc0a1b00 legacy=0x7f75cc1c95b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:58.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:58.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 1 monclient: found mon.b 2024-09-16T14:45:58.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:58.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 1 -- 172.21.15.189:0/265954658 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc135820 con 0x7f75cc13b430 2024-09-16T14:45:58.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:58.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:58.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:58.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75d080d640 5 monclient: authenticate success, global_id 94381 2024-09-16T14:45:58.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75d080d640 10 monclient: _renew_subs 2024-09-16T14:45:58.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75d080d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75d080d640 1 -- 172.21.15.189:0/265954658 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75cc11e980 con 0x7f75cc13b430 2024-09-16T14:45:58.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75d080d640 10 monclient: _renew_subs 2024-09-16T14:45:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75d080d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75d080d640 1 -- 172.21.15.189:0/265954658 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75cc09e350 con 0x7f75cc13b430 2024-09-16T14:45:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 1 -- 172.21.15.189:0/265954658 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75c4001580 con 0x7f75cc13b430 2024-09-16T14:45:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 1 -- 172.21.15.189:0/265954658 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f75c40051a0 con 0x7f75cc13b430 2024-09-16T14:45:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c959a640 4 set_mon_vals no callback set 2024-09-16T14:45:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 10 monclient: dump: 2024-09-16T14:45:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.580 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:58.580 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:58.580 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:58.580 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:58.580 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:58.580 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75d080d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:58.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75d080d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:58.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75d080d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 1 -- 172.21.15.189:0/265954658 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f75c4005380 con 0x7f75cc13b430 2024-09-16T14:45:58.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75d080d640 1 -- 172.21.15.189:0/265954658 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b0005050 con 0x7f75cc13b430 2024-09-16T14:45:58.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75c2ffd640 1 -- 172.21.15.189:0/265954658 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(73..73 src has 15..73) v4 ==== 5385+0+0 (unknown 4089813478 0 0) 0x7f75c4043ed0 con 0x7f75cc13b430 2024-09-16T14:45:58.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.564+0000 7f75ca59c640 10 monclient: get_auth_request con 0x7f75ac036000 auth_method 0 2024-09-16T14:45:58.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.568+0000 7f75c2ffd640 1 -- 172.21.15.189:0/265954658 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f75c4014c50 con 0x7f75cc13b430 2024-09-16T14:45:58.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.568+0000 7f75c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:58.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.568+0000 7f75c2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:58.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.716+0000 7f75d080d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:45:58.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.716+0000 7f75d080d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T14:45:58.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.716+0000 7f75d080d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.716+0000 7f75d080d640 1 -- 172.21.15.189:0/265954658 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f75b0005b10 con 0x7f75cc13b430 2024-09-16T14:45:58.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.716+0000 7f75c2ffd640 1 -- 172.21.15.189:0/265954658 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v73) v1 ==== 56+0+1474 (unknown 4132081119 0 260673783) 0x7f75c40132f0 con 0x7f75cc13b430 2024-09-16T14:45:58.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.716+0000 7f75c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T14:45:58.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.716+0000 7f75c2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:45:58.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.812+0000 7f75d080d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:45:58.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.812+0000 7f75d080d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T14:45:58.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.812+0000 7f75d080d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.812+0000 7f75d080d640 1 -- 172.21.15.189:0/265954658 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f75b0005850 con 0x7f75cc13b430 2024-09-16T14:45:58.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.812+0000 7f75c2ffd640 1 -- 172.21.15.189:0/265954658 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v73) v1 ==== 56+0+1474 (unknown 4132081119 0 260673783) 0x7f75c40132f0 con 0x7f75cc13b430 2024-09-16T14:45:58.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.812+0000 7f75c2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T14:45:58.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.812+0000 7f75c2ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:45:58.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.812+0000 7f75d080d640 1 -- 172.21.15.189:0/265954658 >> v1:172.21.15.189:6812/3196161582 conn(0x7f75ac036000 legacy=0x7f75ac0384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:58.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.812+0000 7f75d080d640 10 monclient: shutdown 2024-09-16T14:45:58.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.812+0000 7f75d080d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:58.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.812+0000 7f75d080d640 1 -- 172.21.15.189:0/265954658 >> v1:172.21.15.189:6789/0 conn(0x7f75cc13b430 legacy=0x7f75cc1d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:58.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.812+0000 7f75d080d640 1 -- 172.21.15.189:0/265954658 shutdown_connections 2024-09-16T14:45:58.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.812+0000 7f75d080d640 1 -- 172.21.15.189:0/265954658 >> 172.21.15.189:0/265954658 conn(0x7f75cc0a7dd0 msgr2=0x7f75cc09d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:58.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.812+0000 7f75d080d640 1 -- 172.21.15.189:0/265954658 shutdown_connections 2024-09-16T14:45:58.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.812+0000 7f75d080d640 1 -- 172.21.15.189:0/265954658 wait complete. 2024-09-16T14:45:58.831 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush rm host1 foo 2024-09-16T14:45:58.831 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:45:58.831 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm host1 foo 2024-09-16T14:45:58.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe7719e4640 10 monclient: get_monmap_and_config 2024-09-16T14:45:58.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe7719e4640 10 monclient: build_initial_monmap 2024-09-16T14:45:58.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe7719e4640 10 monclient: monmap: 2024-09-16T14:45:58.944 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:58.944 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:58.944 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:58.944099+0000 2024-09-16T14:45:58.944 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:58.944099+0000 2024-09-16T14:45:58.945 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:58.945 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:58.945 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:58.945 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:58.945 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:58.945 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe7719e4640 1 Processor -- start 2024-09-16T14:45:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe7719e4640 1 -- start start 2024-09-16T14:45:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe7719e4640 10 monclient: init 2024-09-16T14:45:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe7719e4640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe7719e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe7719e4640 10 monclient(hunting): picked mon.noname-c con 0x7fe76c09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe7719e4640 10 monclient(hunting): picked mon.noname-a con 0x7fe76c09e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe7719e4640 10 monclient(hunting): picked mon.noname-b con 0x7fe76c09e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe7719e4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76c09ee20 con 0x7fe76c09e2f0 2024-09-16T14:45:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe7719e4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76c09f100 con 0x7fe76c09df10 2024-09-16T14:45:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe7719e4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76c09f3e0 con 0x7fe76c09e740 2024-09-16T14:45:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe7719e4640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe7719e4640 10 monclient(hunting): authenticate will time out at 1141.059810s 2024-09-16T14:45:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.940+0000 7fe76a7fc640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fe76c09e2f0 legacy=0x7fe76c1378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe76affd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe76c09df10 0x7fe76c09d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58036/0 (socket says 172.21.15.189:58036) 2024-09-16T14:45:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe76affd640 1 -- 172.21.15.189:0/3993760184 learned_addr learned my addr 172.21.15.189:0/3993760184 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 1 -- 172.21.15.189:0/3993760184 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978498375 0 0) 0x7fe76c09f100 con 0x7fe76c09df10 2024-09-16T14:45:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:45:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 10 monclient(hunting): my global_id is 94384 2024-09-16T14:45:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 1 -- 172.21.15.189:0/3993760184 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe754003530 con 0x7fe76c09df10 2024-09-16T14:45:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 1 -- 172.21.15.189:0/3993760184 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3432448393 0 0) 0x7fe754003530 con 0x7fe76c09df10 2024-09-16T14:45:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 1 -- 172.21.15.189:0/3993760184 >> v1:172.21.15.26:6790/0 conn(0x7fe76c09e740 legacy=0x7fe76c13b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 1 -- 172.21.15.189:0/3993760184 >> v1:172.21.15.26:6789/0 conn(0x7fe76c09e2f0 legacy=0x7fe76c1378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:45:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 1 monclient: found mon.noname-c 2024-09-16T14:45:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 1 -- 172.21.15.189:0/3993760184 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c09c690 con 0x7fe76c09df10 2024-09-16T14:45:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 10 monclient: _finish_auth 0 2024-09-16T14:45:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:45:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 1 -- 172.21.15.189:0/3993760184 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe760001620 con 0x7fe76c09df10 2024-09-16T14:45:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 5 monclient: authenticate success, global_id 94384 2024-09-16T14:45:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 10 monclient: dump: 2024-09-16T14:45:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:58.955 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 1 -- 172.21.15.189:0/3993760184 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe760001940 con 0x7fe76c09df10 2024-09-16T14:45:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 4 set_mon_vals no callback set 2024-09-16T14:45:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 1 -- 172.21.15.189:0/3993760184 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe760001cd0 con 0x7fe76c09df10 2024-09-16T14:45:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe769ffb640 10 monclient: dump: 2024-09-16T14:45:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:58.956 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 10 monclient: shutdown 2024-09-16T14:45:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 1 -- 172.21.15.189:0/3993760184 >> v1:172.21.15.189:6789/0 conn(0x7fe76c09df10 legacy=0x7fe76c09d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 1 -- 172.21.15.189:0/3993760184 shutdown_connections 2024-09-16T14:45:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 1 -- 172.21.15.189:0/3993760184 >> 172.21.15.189:0/3993760184 conn(0x7fe76c0a3320 msgr2=0x7fe76c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 1 -- 172.21.15.189:0/3993760184 shutdown_connections 2024-09-16T14:45:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 1 -- 172.21.15.189:0/3993760184 wait complete. 2024-09-16T14:45:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 10 monclient: build_initial_monmap 2024-09-16T14:45:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 10 monclient: monmap: 2024-09-16T14:45:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:58.958 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 1 Processor -- start 2024-09-16T14:45:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 1 -- start start 2024-09-16T14:45:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 10 monclient: init 2024-09-16T14:45:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:45:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 10 monclient(hunting): picked mon.noname-b con 0x7fe76c09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 10 monclient(hunting): picked mon.noname-c con 0x7fe76c09e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 10 monclient(hunting): picked mon.noname-a con 0x7fe76c09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76c06b380 con 0x7fe76c09e740 2024-09-16T14:45:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76c06ce70 con 0x7fe76c09df10 2024-09-16T14:45:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76c06f940 con 0x7fe76c09e2f0 2024-09-16T14:45:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 10 monclient(hunting): authenticate will time out at 1141.062926s 2024-09-16T14:45:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe76affd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe76c09df10 0x7fe76c1c99b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58040/0 (socket says 172.21.15.189:58040) 2024-09-16T14:45:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe76affd640 1 -- 172.21.15.189:0/669538007 learned_addr learned my addr 172.21.15.189:0/669538007 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe76b7fe640 1 -- 172.21.15.189:0/669538007 >> v1:172.21.15.26:6789/0 conn(0x7fe76c09e740 legacy=0x7fe76c1d20c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 1 -- 172.21.15.189:0/669538007 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756435461 0 0) 0x7fe76c06ce70 con 0x7fe76c09df10 2024-09-16T14:45:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 10 monclient(hunting): _negotiate 2024-09-16T14:45:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 10 monclient(hunting): my global_id is 94387 2024-09-16T14:45:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 1 -- 172.21.15.189:0/669538007 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe758003560 con 0x7fe76c09df10 2024-09-16T14:45:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 1 -- 172.21.15.189:0/669538007 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 102080051 0 0) 0x7fe758003560 con 0x7fe76c09df10 2024-09-16T14:45:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 1 -- 172.21.15.189:0/669538007 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe76c06ce70 con 0x7fe76c09df10 2024-09-16T14:45:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 1 -- 172.21.15.189:0/669538007 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe7600015e0 con 0x7fe76c09df10 2024-09-16T14:45:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 10 monclient(hunting): dump: 2024-09-16T14:45:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:58.960 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 1 -- 172.21.15.189:0/669538007 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 778265061 0 0) 0x7fe76c06ce70 con 0x7fe76c09df10 2024-09-16T14:45:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 1 -- 172.21.15.189:0/669538007 >> v1:172.21.15.26:6790/0 conn(0x7fe76c09e2f0 legacy=0x7fe76c1ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 1 -- 172.21.15.189:0/669538007 >> v1:172.21.15.26:6789/0 conn(0x7fe76c09e740 legacy=0x7fe76c1d20c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 10 monclient: _finish_hunting 0 2024-09-16T14:45:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 1 monclient: found mon.b 2024-09-16T14:45:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 1 -- 172.21.15.189:0/669538007 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c11fc90 con 0x7fe76c09df10 2024-09-16T14:45:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 10 monclient: _finish_auth 0 2024-09-16T14:45:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 10 monclient: _check_auth_tickets 2024-09-16T14:45:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe753fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 5 monclient: authenticate success, global_id 94387 2024-09-16T14:45:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 10 monclient: _renew_subs 2024-09-16T14:45:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 1 -- 172.21.15.189:0/669538007 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76c11f9b0 con 0x7fe76c09df10 2024-09-16T14:45:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 10 monclient: _renew_subs 2024-09-16T14:45:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.944+0000 7fe7719e4640 1 -- 172.21.15.189:0/669538007 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76c09b230 con 0x7fe76c09df10 2024-09-16T14:45:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.948+0000 7fe753fff640 1 -- 172.21.15.189:0/669538007 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe760004c50 con 0x7fe76c09df10 2024-09-16T14:45:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.948+0000 7fe753fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.948+0000 7fe753fff640 1 -- 172.21.15.189:0/669538007 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe760004fe0 con 0x7fe76c09df10 2024-09-16T14:45:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.948+0000 7fe753fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.948+0000 7fe753fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.948+0000 7fe753fff640 10 monclient: dump: 2024-09-16T14:45:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:58.963 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.948+0000 7fe7719e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.948+0000 7fe753fff640 1 -- 172.21.15.189:0/669538007 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe76000fa50 con 0x7fe76c09df10 2024-09-16T14:45:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.948+0000 7fe768ff9640 4 set_mon_vals no callback set 2024-09-16T14:45:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.948+0000 7fe753fff640 1 -- 172.21.15.189:0/669538007 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(73..73 src has 15..73) v4 ==== 5385+0+0 (unknown 4089813478 0 0) 0x7fe760043aa0 con 0x7fe76c09df10 2024-09-16T14:45:58.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.948+0000 7fe7719e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:58.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.948+0000 7fe7719e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:58.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.948+0000 7fe7719e4640 1 -- 172.21.15.189:0/669538007 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76c0008e0 con 0x7fe76c09df10 2024-09-16T14:45:58.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.948+0000 7fe76a7fc640 10 monclient: get_auth_request con 0x7fe7580361b0 auth_method 0 2024-09-16T14:45:58.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.952+0000 7fe753fff640 1 -- 172.21.15.189:0/669538007 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe760014a70 con 0x7fe76c09df10 2024-09-16T14:45:58.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.952+0000 7fe753fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:58.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:58.952+0000 7fe753fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:59.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.100+0000 7fe7719e4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-16T14:45:59.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.100+0000 7fe7719e4640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-16T14:45:59.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.100+0000 7fe7719e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:59.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.100+0000 7fe7719e4640 1 -- 172.21.15.189:0/669538007 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7fe76c09b4e0 con 0x7fe76c09df10 2024-09-16T14:45:59.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.104+0000 7fe753fff640 1 -- 172.21.15.189:0/669538007 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}]=-39 (39) Directory not empty v73) v1 ==== 120+0+0 (unknown 2381247391 0 0) 0x7fe760012ee0 con 0x7fe76c09df10 2024-09-16T14:45:59.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.104+0000 7fe753fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-16T14:45:59.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.104+0000 7fe753fff640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-16T14:45:59.106 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-16T14:45:59.109 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:45:59.110 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush unlink host1 foo 2024-09-16T14:45:59.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.224+0000 7f35f4b0d640 10 monclient: get_monmap_and_config 2024-09-16T14:45:59.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.224+0000 7f35f4b0d640 10 monclient: build_initial_monmap 2024-09-16T14:45:59.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.224+0000 7f35f4b0d640 10 monclient: monmap: 2024-09-16T14:45:59.228 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:59.228 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:59.229 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:45:59.228164+0000 2024-09-16T14:45:59.229 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:45:59.228164+0000 2024-09-16T14:45:59.229 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:59.229 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:59.229 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:59.229 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:45:59.229 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:45:59.229 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:59.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.224+0000 7f35f4b0d640 1 Processor -- start 2024-09-16T14:45:59.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.224+0000 7f35f4b0d640 1 -- start start 2024-09-16T14:45:59.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.224+0000 7f35f4b0d640 10 monclient: init 2024-09-16T14:45:59.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:59.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:45:59.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 10 monclient(hunting): picked mon.noname-c con 0x7f35f01315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:59.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 10 monclient(hunting): picked mon.noname-a con 0x7f35f009b320 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:59.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 10 monclient(hunting): picked mon.noname-b con 0x7f35f0132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:59.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f009edd0 con 0x7f35f009b320 2024-09-16T14:45:59.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f009bc00 con 0x7f35f01315a0 2024-09-16T14:45:59.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f009e900 con 0x7f35f0132090 2024-09-16T14:45:59.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:59.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 10 monclient(hunting): authenticate will time out at 1141.344596s 2024-09-16T14:45:59.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35ee59c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f35f01315a0 0x7f35f0131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58042/0 (socket says 172.21.15.189:58042) 2024-09-16T14:45:59.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35ee59c640 1 -- 172.21.15.189:0/3874529242 learned_addr learned my addr 172.21.15.189:0/3874529242 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:59.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35e7fff640 1 -- 172.21.15.189:0/3874529242 >> v1:172.21.15.26:6789/0 conn(0x7f35f009b320 legacy=0x7f35f0136ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:59.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 1 -- 172.21.15.189:0/3874529242 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464110478 0 0) 0x7f35f009bc00 con 0x7f35f01315a0 2024-09-16T14:45:59.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:45:59.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:59.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:59.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 10 monclient(hunting): my global_id is 94390 2024-09-16T14:45:59.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 1 -- 172.21.15.189:0/3874529242 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35dc003530 con 0x7f35f01315a0 2024-09-16T14:45:59.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 1 -- 172.21.15.189:0/3874529242 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1276220318 0 0) 0x7f35dc003530 con 0x7f35f01315a0 2024-09-16T14:45:59.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 1 -- 172.21.15.189:0/3874529242 >> v1:172.21.15.26:6790/0 conn(0x7f35f0132090 legacy=0x7f35f013a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:59.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 1 -- 172.21.15.189:0/3874529242 >> v1:172.21.15.26:6789/0 conn(0x7f35f009b320 legacy=0x7f35f0136ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:45:59.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 10 monclient: _finish_hunting 0 2024-09-16T14:45:59.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 1 monclient: found mon.noname-c 2024-09-16T14:45:59.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:59.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:45:59.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 1 -- 172.21.15.189:0/3874529242 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f009b920 con 0x7f35f01315a0 2024-09-16T14:45:59.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 10 monclient: _finish_auth 0 2024-09-16T14:45:59.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 10 monclient: _check_auth_tickets 2024-09-16T14:45:59.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:59.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 1 -- 172.21.15.189:0/3874529242 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f35e0001620 con 0x7f35f01315a0 2024-09-16T14:45:59.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:59.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:45:59.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 10 monclient: dump: 2024-09-16T14:45:59.241 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:59.241 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:59.241 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:59.241 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:59.241 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:59.241 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:59.241 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:59.242 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 5 monclient: authenticate success, global_id 94390 2024-09-16T14:45:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:45:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 1 -- 172.21.15.189:0/3874529242 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35e0001940 con 0x7f35f01315a0 2024-09-16T14:45:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 1 -- 172.21.15.189:0/3874529242 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f35e0001cd0 con 0x7f35f01315a0 2024-09-16T14:45:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 10 monclient: get_monmap_and_config success 2024-09-16T14:45:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 4 set_mon_vals no callback set 2024-09-16T14:45:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:59.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35edd9b640 10 monclient: dump: 2024-09-16T14:45:59.243 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:59.243 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:59.243 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:59.243 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:59.243 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:59.243 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:59.243 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:59.243 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:59.243 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:59.244 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:59.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 10 monclient: shutdown 2024-09-16T14:45:59.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:59.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 1 -- 172.21.15.189:0/3874529242 >> v1:172.21.15.189:6789/0 conn(0x7f35f01315a0 legacy=0x7f35f0131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:59.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 1 -- 172.21.15.189:0/3874529242 shutdown_connections 2024-09-16T14:45:59.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 1 -- 172.21.15.189:0/3874529242 >> 172.21.15.189:0/3874529242 conn(0x7f35f00a3320 msgr2=0x7f35f009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:59.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 1 -- 172.21.15.189:0/3874529242 shutdown_connections 2024-09-16T14:45:59.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 1 -- 172.21.15.189:0/3874529242 wait complete. 2024-09-16T14:45:59.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 10 monclient: build_initial_monmap 2024-09-16T14:45:59.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 10 monclient: monmap: 2024-09-16T14:45:59.244 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:45:59.245 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:45:59.245 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:45:59.245 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:45:59.245 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:45:59.245 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:45:59.245 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:45:59.245 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:45:59.245 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:45:59.245 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:59.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 1 Processor -- start 2024-09-16T14:45:59.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 1 -- start start 2024-09-16T14:45:59.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 10 monclient: init 2024-09-16T14:45:59.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 10 monclient: _reopen_session rank -1 2024-09-16T14:45:59.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:45:59.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 10 monclient(hunting): picked mon.noname-c con 0x7f35f01315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:45:59.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.228+0000 7f35f4b0d640 10 monclient(hunting): picked mon.noname-a con 0x7f35f0132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:45:59.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 10 monclient(hunting): picked mon.noname-b con 0x7f35f0131090 addr v1:172.21.15.189:6789/0 2024-09-16T14:45:59.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f009cd20 con 0x7f35f0132090 2024-09-16T14:45:59.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f006b380 con 0x7f35f0131090 2024-09-16T14:45:59.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f00a18f0 con 0x7f35f01315a0 2024-09-16T14:45:59.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 10 monclient(hunting): _renew_subs 2024-09-16T14:45:59.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 10 monclient(hunting): authenticate will time out at 1141.348107s 2024-09-16T14:45:59.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e7fff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f35f0132090 legacy=0x7f35f0130980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:45:59.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35eed9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f35f0131090 0x7f35f01cdfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58058/0 (socket says 172.21.15.189:58058) 2024-09-16T14:45:59.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35eed9d640 1 -- 172.21.15.189:0/452357322 learned_addr learned my addr 172.21.15.189:0/452357322 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:45:59.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410658582 0 0) 0x7f35f006b380 con 0x7f35f0131090 2024-09-16T14:45:59.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:59.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:59.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:59.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient(hunting): my global_id is 94393 2024-09-16T14:45:59.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d0003530 con 0x7f35f0131090 2024-09-16T14:45:59.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1405890429 0 0) 0x7f35d0003530 con 0x7f35f0131090 2024-09-16T14:45:59.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35f006b380 con 0x7f35f0131090 2024-09-16T14:45:59.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f35e8001c80 con 0x7f35f0131090 2024-09-16T14:45:59.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:59.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:45:59.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient(hunting): dump: 2024-09-16T14:45:59.247 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:59.247 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:59.247 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:59.248 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:59.248 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:59.248 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:59.248 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:59.248 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:59.248 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:59.248 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:59.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:45:59.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045318595 0 0) 0x7f35f00a18f0 con 0x7f35f01315a0 2024-09-16T14:45:59.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:45:59.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:45:59.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:45:59.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient(hunting): my global_id is 5276 2024-09-16T14:45:59.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d0003530 con 0x7f35f01315a0 2024-09-16T14:45:59.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2122559761 0 0) 0x7f35f006b380 con 0x7f35f0131090 2024-09-16T14:45:59.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 >> v1:172.21.15.26:6790/0 conn(0x7f35f01315a0 legacy=0x7f35f01c5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:59.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 >> v1:172.21.15.26:6789/0 conn(0x7f35f0132090 legacy=0x7f35f0130980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:45:59.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:45:59.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 1 monclient: found mon.b 2024-09-16T14:45:59.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:45:59.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:59.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f0120360 con 0x7f35f0131090 2024-09-16T14:45:59.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:45:59.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:45:59.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:45:59.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 5 monclient: authenticate success, global_id 94393 2024-09-16T14:45:59.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 10 monclient: _renew_subs 2024-09-16T14:45:59.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:59.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 1 -- 172.21.15.189:0/452357322 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f011e950 con 0x7f35f0131090 2024-09-16T14:45:59.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 10 monclient: _renew_subs 2024-09-16T14:45:59.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:59.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 1 -- 172.21.15.189:0/452357322 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f0120080 con 0x7f35f0131090 2024-09-16T14:45:59.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35e8002050 con 0x7f35f0131090 2024-09-16T14:45:59.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:45:59.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35ed59a640 4 set_mon_vals no callback set 2024-09-16T14:45:59.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f35e8003e70 con 0x7f35f0131090 2024-09-16T14:45:59.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:45:59.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:45:59.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 10 monclient: dump: 2024-09-16T14:45:59.251 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:45:59.251 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:45:59.251 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:59.251 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:45:59.251 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:45:59.251 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:45:59.251 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:45:59.251 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:45:59.251 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:45:59.251 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:45:59.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:59.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f35e8004050 con 0x7f35f0131090 2024-09-16T14:45:59.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:59.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:59.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35f4b0d640 1 -- 172.21.15.189:0/452357322 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d4005050 con 0x7f35f0131090 2024-09-16T14:45:59.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(73..73 src has 15..73) v4 ==== 5385+0+0 (unknown 4089813478 0 0) 0x7f35e8043d40 con 0x7f35f0131090 2024-09-16T14:45:59.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.232+0000 7f35ee59c640 10 monclient: get_auth_request con 0x7f35d00360e0 auth_method 0 2024-09-16T14:45:59.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.236+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f35e80149e0 con 0x7f35f0131090 2024-09-16T14:45:59.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.236+0000 7f35e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:45:59.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.236+0000 7f35e6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:45:59.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.400+0000 7f35f4b0d640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T14:45:59.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.400+0000 7f35f4b0d640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T14:45:59.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.400+0000 7f35f4b0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:59.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.400+0000 7f35f4b0d640 1 -- 172.21.15.189:0/452357322 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f35d4005880 con 0x7f35f0131090 2024-09-16T14:45:59.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.796+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 removed item id -11 name 'host1' from crush map v74) v1 ==== 147+0+0 (unknown 2678557947 0 0) 0x7f35e8013080 con 0x7f35f0131090 2024-09-16T14:45:59.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.796+0000 7f35e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T14:45:59.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.796+0000 7f35e6ffd640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-16T14:45:59.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.900+0000 7f35f4b0d640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T14:45:59.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.900+0000 7f35f4b0d640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T14:45:59.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.900+0000 7f35f4b0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:45:59.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.900+0000 7f35f4b0d640 1 -- 172.21.15.189:0/452357322 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f35d4005300 con 0x7f35f0131090 2024-09-16T14:45:59.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.900+0000 7f35e6ffd640 1 -- 172.21.15.189:0/452357322 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v74) v1 ==== 141+0+0 (unknown 3819635062 0 0) 0x7f35e8014800 con 0x7f35f0131090 2024-09-16T14:45:59.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.900+0000 7f35e6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T14:45:59.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.900+0000 7f35e6ffd640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-16T14:45:59.904 INFO:tasks.workunit.client.0.smithi189.stderr:item -11 does not appear in that position 2024-09-16T14:45:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.904+0000 7f35f4b0d640 1 -- 172.21.15.189:0/452357322 >> v1:172.21.15.189:6812/3196161582 conn(0x7f35d00360e0 legacy=0x7f35d00385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.904+0000 7f35f4b0d640 10 monclient: shutdown 2024-09-16T14:45:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.904+0000 7f35f4b0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:45:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.904+0000 7f35f4b0d640 1 -- 172.21.15.189:0/452357322 >> v1:172.21.15.189:6789/0 conn(0x7f35f0131090 legacy=0x7f35f01cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:45:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.904+0000 7f35f4b0d640 1 -- 172.21.15.189:0/452357322 shutdown_connections 2024-09-16T14:45:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.904+0000 7f35f4b0d640 1 -- 172.21.15.189:0/452357322 >> 172.21.15.189:0/452357322 conn(0x7f35f00a3320 msgr2=0x7f35f00a2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:45:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.904+0000 7f35f4b0d640 1 -- 172.21.15.189:0/452357322 shutdown_connections 2024-09-16T14:45:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:45:59.904+0000 7f35f4b0d640 1 -- 172.21.15.189:0/452357322 wait complete. 2024-09-16T14:45:59.920 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush unlink host1 foo 2024-09-16T14:46:00.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.032+0000 7ff2cd8c0640 10 monclient: get_monmap_and_config 2024-09-16T14:46:00.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.032+0000 7ff2cd8c0640 10 monclient: build_initial_monmap 2024-09-16T14:46:00.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.032+0000 7ff2cd8c0640 10 monclient: monmap: 2024-09-16T14:46:00.037 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:00.037 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:00.037 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:00.036923+0000 2024-09-16T14:46:00.037 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:00.036923+0000 2024-09-16T14:46:00.037 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:00.037 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:00.037 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:00.038 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:00.038 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:00.038 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 1 Processor -- start 2024-09-16T14:46:00.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 1 -- start start 2024-09-16T14:46:00.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 10 monclient: init 2024-09-16T14:46:00.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:00.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:00.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 10 monclient(hunting): picked mon.noname-c con 0x7ff2c80969c0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:00.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 10 monclient(hunting): picked mon.noname-a con 0x7ff2c8096da0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:00.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 10 monclient(hunting): picked mon.noname-b con 0x7ff2c8138fe0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:00.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c8099ad0 con 0x7ff2c8096da0 2024-09-16T14:46:00.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c809efe0 con 0x7ff2c80969c0 2024-09-16T14:46:00.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c809b4c0 con 0x7ff2c8138fe0 2024-09-16T14:46:00.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:00.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 10 monclient(hunting): authenticate will time out at 1142.153363s 2024-09-16T14:46:00.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c6ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff2c80969c0 0x7ff2c809adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58072/0 (socket says 172.21.15.189:58072) 2024-09-16T14:46:00.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2be7fc640 1 -- 172.21.15.189:0/4203116054 >> v1:172.21.15.26:6789/0 conn(0x7ff2c8096da0 legacy=0x7ff2c81378c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:46:00.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c6ffd640 1 -- 172.21.15.189:0/4203116054 learned_addr learned my addr 172.21.15.189:0/4203116054 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:00.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 1 -- 172.21.15.189:0/4203116054 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982114217 0 0) 0x7ff2c809efe0 con 0x7ff2c80969c0 2024-09-16T14:46:00.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:00.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:00.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:00.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient(hunting): my global_id is 94396 2024-09-16T14:46:00.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 1 -- 172.21.15.189:0/4203116054 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b4003530 con 0x7ff2c80969c0 2024-09-16T14:46:00.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 1 -- 172.21.15.189:0/4203116054 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845774324 0 0) 0x7ff2c809b4c0 con 0x7ff2c8138fe0 2024-09-16T14:46:00.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:00.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:00.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:00.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient(hunting): my global_id is 5279 2024-09-16T14:46:00.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 1 -- 172.21.15.189:0/4203116054 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2c809efe0 con 0x7ff2c8138fe0 2024-09-16T14:46:00.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 1 -- 172.21.15.189:0/4203116054 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3400681937 0 0) 0x7ff2b4003530 con 0x7ff2c80969c0 2024-09-16T14:46:00.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 1 -- 172.21.15.189:0/4203116054 >> v1:172.21.15.26:6790/0 conn(0x7ff2c8138fe0 legacy=0x7ff2c813b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:00.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 1 -- 172.21.15.189:0/4203116054 >> v1:172.21.15.26:6789/0 conn(0x7ff2c8096da0 legacy=0x7ff2c81378c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:00.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:46:00.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 1 monclient: found mon.noname-c 2024-09-16T14:46:00.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:00.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 1 -- 172.21.15.189:0/4203116054 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c80997f0 con 0x7ff2c80969c0 2024-09-16T14:46:00.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient: _finish_auth 0 2024-09-16T14:46:00.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:46:00.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:00.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 1 -- 172.21.15.189:0/4203116054 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff2b8001620 con 0x7ff2c80969c0 2024-09-16T14:46:00.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:00.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:00.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient: dump: 2024-09-16T14:46:00.047 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:00.048 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:00.048 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.048 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.048 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:00.048 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:00.048 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:00.048 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:00.048 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:00.048 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 5 monclient: authenticate success, global_id 94396 2024-09-16T14:46:00.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:00.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 1 -- 172.21.15.189:0/4203116054 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2b8001900 con 0x7ff2c80969c0 2024-09-16T14:46:00.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:00.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:00.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 1 -- 172.21.15.189:0/4203116054 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff2b8001c90 con 0x7ff2c80969c0 2024-09-16T14:46:00.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:00.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 4 set_mon_vals no callback set 2024-09-16T14:46:00.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:00.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2c67fc640 10 monclient: dump: 2024-09-16T14:46:00.049 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:00.049 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:00.049 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.049 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.049 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:00.049 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:00.050 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:00.050 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:00.050 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:00.050 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 10 monclient: shutdown 2024-09-16T14:46:00.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:00.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 1 -- 172.21.15.189:0/4203116054 >> v1:172.21.15.189:6789/0 conn(0x7ff2c80969c0 legacy=0x7ff2c809adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:00.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 1 -- 172.21.15.189:0/4203116054 shutdown_connections 2024-09-16T14:46:00.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 1 -- 172.21.15.189:0/4203116054 >> 172.21.15.189:0/4203116054 conn(0x7ff2c80a02d0 msgr2=0x7ff2c80a06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:00.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 1 -- 172.21.15.189:0/4203116054 shutdown_connections 2024-09-16T14:46:00.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 1 -- 172.21.15.189:0/4203116054 wait complete. 2024-09-16T14:46:00.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 10 monclient: build_initial_monmap 2024-09-16T14:46:00.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.036+0000 7ff2cd8c0640 10 monclient: monmap: 2024-09-16T14:46:00.051 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:00.051 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:00.051 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:00.051 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:00.051 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:00.051 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:00.051 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:00.051 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:00.051 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:00.051 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 1 Processor -- start 2024-09-16T14:46:00.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 1 -- start start 2024-09-16T14:46:00.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 10 monclient: init 2024-09-16T14:46:00.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:00.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:46:00.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 10 monclient(hunting): picked mon.noname-a con 0x7ff2c80969c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:00.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 10 monclient(hunting): picked mon.noname-b con 0x7ff2c8096da0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:00.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 10 monclient(hunting): picked mon.noname-c con 0x7ff2c8138fe0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:00.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c813c4f0 con 0x7ff2c80969c0 2024-09-16T14:46:00.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c8099230 con 0x7ff2c8096da0 2024-09-16T14:46:00.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c809a5c0 con 0x7ff2c8138fe0 2024-09-16T14:46:00.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:00.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 10 monclient(hunting): authenticate will time out at 1142.156661s 2024-09-16T14:46:00.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c6ffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7ff2c80969c0 legacy=0x7ff2c80973d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:46:00.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2be7fc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff2c8096da0 0x7ff2c8097ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58076/0 (socket says 172.21.15.189:58076) 2024-09-16T14:46:00.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2be7fc640 1 -- 172.21.15.189:0/2541257829 learned_addr learned my addr 172.21.15.189:0/2541257829 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:00.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245237004 0 0) 0x7ff2c8099230 con 0x7ff2c8096da0 2024-09-16T14:46:00.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:00.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:00.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:00.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient(hunting): my global_id is 94399 2024-09-16T14:46:00.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2a8003530 con 0x7ff2c8096da0 2024-09-16T14:46:00.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2709821343 0 0) 0x7ff2a8003530 con 0x7ff2c8096da0 2024-09-16T14:46:00.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2c8099230 con 0x7ff2c8096da0 2024-09-16T14:46:00.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff2b0004190 con 0x7ff2c8096da0 2024-09-16T14:46:00.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:00.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:00.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient(hunting): dump: 2024-09-16T14:46:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:00.054 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181901348 0 0) 0x7ff2c809a5c0 con 0x7ff2c8138fe0 2024-09-16T14:46:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:00.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:00.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:00.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient(hunting): my global_id is 5282 2024-09-16T14:46:00.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2a8003530 con 0x7ff2c8138fe0 2024-09-16T14:46:00.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1564523598 0 0) 0x7ff2c8099230 con 0x7ff2c8096da0 2024-09-16T14:46:00.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 >> v1:172.21.15.26:6790/0 conn(0x7ff2c8138fe0 legacy=0x7ff2c81d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:00.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 >> v1:172.21.15.26:6789/0 conn(0x7ff2c80969c0 legacy=0x7ff2c80973d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:00.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:46:00.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 1 monclient: found mon.b 2024-09-16T14:46:00.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:00.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c8096180 con 0x7ff2c8096da0 2024-09-16T14:46:00.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient: _finish_auth 0 2024-09-16T14:46:00.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:46:00.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:00.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 5 monclient: authenticate success, global_id 94399 2024-09-16T14:46:00.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 10 monclient: _renew_subs 2024-09-16T14:46:00.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 1 -- 172.21.15.189:0/2541257829 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c806fc40 con 0x7ff2c8096da0 2024-09-16T14:46:00.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 10 monclient: _renew_subs 2024-09-16T14:46:00.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 1 -- 172.21.15.189:0/2541257829 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c8098250 con 0x7ff2c8096da0 2024-09-16T14:46:00.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2b00044b0 con 0x7ff2c8096da0 2024-09-16T14:46:00.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:00.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff2b0004840 con 0x7ff2c8096da0 2024-09-16T14:46:00.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:00.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:00.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c5ffb640 4 set_mon_vals no callback set 2024-09-16T14:46:00.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 10 monclient: dump: 2024-09-16T14:46:00.057 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:00.057 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:00.057 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.057 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.057 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:00.057 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:00.057 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:00.057 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:00.057 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:00.058 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2cd8c0640 1 -- 172.21.15.189:0/2541257829 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2ac005050 con 0x7ff2c8096da0 2024-09-16T14:46:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff2b00049a0 con 0x7ff2c8096da0 2024-09-16T14:46:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(74..74 src has 15..74) v4 ==== 5377+0+0 (unknown 3577921172 0 0) 0x7ff2b00438f0 con 0x7ff2c8096da0 2024-09-16T14:46:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.040+0000 7ff2c6ffd640 10 monclient: get_auth_request con 0x7ff2a8036130 auth_method 0 2024-09-16T14:46:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.044+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff2b0005170 con 0x7ff2c8096da0 2024-09-16T14:46:00.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.044+0000 7ff2c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:00.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.044+0000 7ff2c4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:00.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.208+0000 7ff2cd8c0640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T14:46:00.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.208+0000 7ff2cd8c0640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T14:46:00.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.208+0000 7ff2cd8c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.208+0000 7ff2cd8c0640 1 -- 172.21.15.189:0/2541257829 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7ff2ac005880 con 0x7ff2c8096da0 2024-09-16T14:46:00.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.212+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v74) v1 ==== 141+0+0 (unknown 3819635062 0 0) 0x7ff2b001cdd0 con 0x7ff2c8096da0 2024-09-16T14:46:00.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.212+0000 7ff2c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T14:46:00.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.212+0000 7ff2c4ff9640 10 monclient: _finish_command 2 = system:0 item -11 does not appear in that position 2024-09-16T14:46:00.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.304+0000 7ff2cd8c0640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T14:46:00.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.304+0000 7ff2cd8c0640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T14:46:00.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.304+0000 7ff2cd8c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.304+0000 7ff2cd8c0640 1 -- 172.21.15.189:0/2541257829 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7ff2ac005300 con 0x7ff2c8096da0 2024-09-16T14:46:00.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.308+0000 7ff2c4ff9640 1 -- 172.21.15.189:0/2541257829 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v74) v1 ==== 141+0+0 (unknown 3819635062 0 0) 0x7ff2b0012c50 con 0x7ff2c8096da0 2024-09-16T14:46:00.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.308+0000 7ff2c4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T14:46:00.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.308+0000 7ff2c4ff9640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-16T14:46:00.310 INFO:tasks.workunit.client.0.smithi189.stderr:item -11 does not appear in that position 2024-09-16T14:46:00.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.308+0000 7ff2cd8c0640 1 -- 172.21.15.189:0/2541257829 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff2a8036130 legacy=0x7ff2a80385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:00.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.308+0000 7ff2cd8c0640 10 monclient: shutdown 2024-09-16T14:46:00.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.308+0000 7ff2cd8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:00.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.308+0000 7ff2cd8c0640 1 -- 172.21.15.189:0/2541257829 >> v1:172.21.15.189:6789/0 conn(0x7ff2c8096da0 legacy=0x7ff2c8097ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:00.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.308+0000 7ff2cd8c0640 1 -- 172.21.15.189:0/2541257829 shutdown_connections 2024-09-16T14:46:00.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.308+0000 7ff2cd8c0640 1 -- 172.21.15.189:0/2541257829 >> 172.21.15.189:0/2541257829 conn(0x7ff2c80a02d0 msgr2=0x7ff2c8094430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:00.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.308+0000 7ff2cd8c0640 1 -- 172.21.15.189:0/2541257829 shutdown_connections 2024-09-16T14:46:00.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.308+0000 7ff2cd8c0640 1 -- 172.21.15.189:0/2541257829 wait complete. 2024-09-16T14:46:00.327 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T14:46:00.327 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 1 2024-09-16T14:46:00.327 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c host1 2024-09-16T14:46:00.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.432+0000 7f56273d5640 10 monclient: get_monmap_and_config 2024-09-16T14:46:00.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.432+0000 7f56273d5640 10 monclient: build_initial_monmap 2024-09-16T14:46:00.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.432+0000 7f56273d5640 10 monclient: monmap: 2024-09-16T14:46:00.438 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:00.438 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:00.438 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:00.437480+0000 2024-09-16T14:46:00.438 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:00.437480+0000 2024-09-16T14:46:00.438 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:00.438 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:00.438 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:00.438 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:00.438 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:00.438 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 1 Processor -- start 2024-09-16T14:46:00.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 1 -- start start 2024-09-16T14:46:00.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 10 monclient: init 2024-09-16T14:46:00.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:00.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:00.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 10 monclient(hunting): picked mon.noname-c con 0x7f562009df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:00.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 10 monclient(hunting): picked mon.noname-a con 0x7f562009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:00.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 10 monclient(hunting): picked mon.noname-b con 0x7f562009e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:00.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562009ee20 con 0x7f562009e2f0 2024-09-16T14:46:00.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562009f100 con 0x7f562009df10 2024-09-16T14:46:00.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562009f3e0 con 0x7f562009e740 2024-09-16T14:46:00.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:00.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 10 monclient(hunting): authenticate will time out at 1142.553303s 2024-09-16T14:46:00.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5625171640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f562009df10 0x7f562009d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58086/0 (socket says 172.21.15.189:58086) 2024-09-16T14:46:00.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5625171640 1 -- 172.21.15.189:0/3786242372 learned_addr learned my addr 172.21.15.189:0/3786242372 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:00.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5624970640 1 -- 172.21.15.189:0/3786242372 >> v1:172.21.15.26:6789/0 conn(0x7f562009e2f0 legacy=0x7f56201378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:46:00.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 1 -- 172.21.15.189:0/3786242372 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986473827 0 0) 0x7f562009f100 con 0x7f562009df10 2024-09-16T14:46:00.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:00.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:00.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:00.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 10 monclient(hunting): my global_id is 94402 2024-09-16T14:46:00.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 1 -- 172.21.15.189:0/3786242372 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f560c003530 con 0x7f562009df10 2024-09-16T14:46:00.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 1 -- 172.21.15.189:0/3786242372 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2641759252 0 0) 0x7f560c003530 con 0x7f562009df10 2024-09-16T14:46:00.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 1 -- 172.21.15.189:0/3786242372 >> v1:172.21.15.26:6790/0 conn(0x7f562009e740 legacy=0x7f562013b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:00.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 1 -- 172.21.15.189:0/3786242372 >> v1:172.21.15.26:6789/0 conn(0x7f562009e2f0 legacy=0x7f56201378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:00.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:00.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 1 monclient: found mon.noname-c 2024-09-16T14:46:00.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:00.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 1 -- 172.21.15.189:0/3786242372 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562009c690 con 0x7f562009df10 2024-09-16T14:46:00.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 10 monclient: _finish_auth 0 2024-09-16T14:46:00.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:00.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:00.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 1 -- 172.21.15.189:0/3786242372 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f561c001620 con 0x7f562009df10 2024-09-16T14:46:00.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 5 monclient: authenticate success, global_id 94402 2024-09-16T14:46:00.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:00.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:00.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:00.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 10 monclient: dump: 2024-09-16T14:46:00.448 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:00.448 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:00.448 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.448 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.448 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:00.448 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:00.448 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:00.448 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:00.449 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:00.449 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:00.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 1 -- 172.21.15.189:0/3786242372 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f561c001940 con 0x7f562009df10 2024-09-16T14:46:00.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:00.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 1 -- 172.21.15.189:0/3786242372 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f561c001cd0 con 0x7f562009df10 2024-09-16T14:46:00.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:00.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 4 set_mon_vals no callback set 2024-09-16T14:46:00.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:00.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:00.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f5617fff640 10 monclient: dump: 2024-09-16T14:46:00.449 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:00.449 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:00.450 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.450 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.450 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:00.450 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:00.450 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:00.450 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:00.450 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:00.450 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 10 monclient: shutdown 2024-09-16T14:46:00.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:00.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 1 -- 172.21.15.189:0/3786242372 >> v1:172.21.15.189:6789/0 conn(0x7f562009df10 legacy=0x7f562009d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:00.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 1 -- 172.21.15.189:0/3786242372 shutdown_connections 2024-09-16T14:46:00.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 1 -- 172.21.15.189:0/3786242372 >> 172.21.15.189:0/3786242372 conn(0x7f56200a3320 msgr2=0x7f562009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:00.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 1 -- 172.21.15.189:0/3786242372 shutdown_connections 2024-09-16T14:46:00.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 1 -- 172.21.15.189:0/3786242372 wait complete. 2024-09-16T14:46:00.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 10 monclient: build_initial_monmap 2024-09-16T14:46:00.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 10 monclient: monmap: 2024-09-16T14:46:00.451 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:00.451 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:00.451 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:00.451 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:00.451 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:00.451 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:00.452 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:00.452 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:00.452 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:00.452 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 1 Processor -- start 2024-09-16T14:46:00.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 1 -- start start 2024-09-16T14:46:00.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.436+0000 7f56273d5640 10 monclient: init 2024-09-16T14:46:00.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:00.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:00.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 10 monclient(hunting): picked mon.noname-b con 0x7f562009df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:00.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 10 monclient(hunting): picked mon.noname-c con 0x7f562009e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:00.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 10 monclient(hunting): picked mon.noname-a con 0x7f562009e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:00.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56201203f0 con 0x7f562009e740 2024-09-16T14:46:00.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562006cd50 con 0x7f562009df10 2024-09-16T14:46:00.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562006b380 con 0x7f562009e2f0 2024-09-16T14:46:00.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:00.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 10 monclient(hunting): authenticate will time out at 1142.556394s 2024-09-16T14:46:00.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5625171640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f562009df10 0x7f562011c590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58100/0 (socket says 172.21.15.189:58100) 2024-09-16T14:46:00.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5625171640 1 -- 172.21.15.189:0/2392961641 learned_addr learned my addr 172.21.15.189:0/2392961641 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:00.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5625972640 1 -- 172.21.15.189:0/2392961641 >> v1:172.21.15.26:6789/0 conn(0x7f562009e740 legacy=0x7f562011d3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:46:00.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 1 -- 172.21.15.189:0/2392961641 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748911204 0 0) 0x7f562006cd50 con 0x7f562009df10 2024-09-16T14:46:00.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:00.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:00.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:00.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 10 monclient(hunting): my global_id is 94405 2024-09-16T14:46:00.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 1 -- 172.21.15.189:0/2392961641 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5604003530 con 0x7f562009df10 2024-09-16T14:46:00.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 1 -- 172.21.15.189:0/2392961641 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3461184087 0 0) 0x7f5604003530 con 0x7f562009df10 2024-09-16T14:46:00.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 1 -- 172.21.15.189:0/2392961641 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f562006cd50 con 0x7f562009df10 2024-09-16T14:46:00.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 1 -- 172.21.15.189:0/2392961641 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f561c0015e0 con 0x7f562009df10 2024-09-16T14:46:00.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:00.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:00.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 10 monclient(hunting): dump: 2024-09-16T14:46:00.455 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:00.455 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:00.455 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.455 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.455 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:00.456 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:00.456 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:00.456 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:00.456 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:00.456 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:00.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 1 -- 172.21.15.189:0/2392961641 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3027216362 0 0) 0x7f562006cd50 con 0x7f562009df10 2024-09-16T14:46:00.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 1 -- 172.21.15.189:0/2392961641 >> v1:172.21.15.26:6790/0 conn(0x7f562009e2f0 legacy=0x7f562011cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:00.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 1 -- 172.21.15.189:0/2392961641 >> v1:172.21.15.26:6789/0 conn(0x7f562009e740 legacy=0x7f562011d3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:00.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:46:00.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 1 monclient: found mon.b 2024-09-16T14:46:00.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:00.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 1 -- 172.21.15.189:0/2392961641 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562011edf0 con 0x7f562009df10 2024-09-16T14:46:00.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 10 monclient: _finish_auth 0 2024-09-16T14:46:00.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:00.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:00.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 5 monclient: authenticate success, global_id 94405 2024-09-16T14:46:00.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 10 monclient: _renew_subs 2024-09-16T14:46:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 1 -- 172.21.15.189:0/2392961641 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562011fb50 con 0x7f562009df10 2024-09-16T14:46:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 10 monclient: _renew_subs 2024-09-16T14:46:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 1 -- 172.21.15.189:0/2392961641 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f562011e850 con 0x7f562009df10 2024-09-16T14:46:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 1 -- 172.21.15.189:0/2392961641 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f561c0011d0 con 0x7f562009df10 2024-09-16T14:46:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 1 -- 172.21.15.189:0/2392961641 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f561c004e70 con 0x7f562009df10 2024-09-16T14:46:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 10 monclient: dump: 2024-09-16T14:46:00.459 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:00.460 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:00.460 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.460 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.460 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:00.460 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:00.460 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:00.460 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:00.461 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:00.461 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56177fe640 4 set_mon_vals no callback set 2024-09-16T14:46:00.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:00.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:00.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 1 -- 172.21.15.189:0/2392961641 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f561c004fd0 con 0x7f562009df10 2024-09-16T14:46:00.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f56273d5640 1 -- 172.21.15.189:0/2392961641 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5600005050 con 0x7f562009df10 2024-09-16T14:46:00.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5615ffb640 1 -- 172.21.15.189:0/2392961641 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(74..74 src has 15..74) v4 ==== 5377+0+0 (unknown 3577921172 0 0) 0x7f561c043b50 con 0x7f562009df10 2024-09-16T14:46:00.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.440+0000 7f5624970640 10 monclient: get_auth_request con 0x7f5604036050 auth_method 0 2024-09-16T14:46:00.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.444+0000 7f5615ffb640 1 -- 172.21.15.189:0/2392961641 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f561c012f10 con 0x7f562009df10 2024-09-16T14:46:00.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.444+0000 7f5615ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:00.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.444+0000 7f5615ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:00.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.604+0000 7f56273d5640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:46:00.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.604+0000 7f56273d5640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T14:46:00.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.604+0000 7f56273d5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.604+0000 7f56273d5640 1 -- 172.21.15.189:0/2392961641 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f5600005b10 con 0x7f562009df10 2024-09-16T14:46:00.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.604+0000 7f5615ffb640 1 -- 172.21.15.189:0/2392961641 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1340 (unknown 1366924582 0 779692950) 0x7f561c0153d0 con 0x7f562009df10 2024-09-16T14:46:00.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.604+0000 7f5615ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T14:46:00.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.604+0000 7f5615ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:00.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.692+0000 7f56273d5640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:46:00.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.692+0000 7f56273d5640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T14:46:00.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.692+0000 7f56273d5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.692+0000 7f56273d5640 1 -- 172.21.15.189:0/2392961641 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f56000055a0 con 0x7f562009df10 2024-09-16T14:46:00.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.696+0000 7f5615ffb640 1 -- 172.21.15.189:0/2392961641 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1340 (unknown 1366924582 0 779692950) 0x7f561c0153d0 con 0x7f562009df10 2024-09-16T14:46:00.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.696+0000 7f5615ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T14:46:00.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.696+0000 7f5615ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:46:00.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.696+0000 7f56273d5640 1 -- 172.21.15.189:0/2392961641 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5604036050 legacy=0x7f5604038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:00.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.696+0000 7f56273d5640 10 monclient: shutdown 2024-09-16T14:46:00.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.696+0000 7f56273d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:00.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.696+0000 7f56273d5640 1 -- 172.21.15.189:0/2392961641 >> v1:172.21.15.189:6789/0 conn(0x7f562009df10 legacy=0x7f562011c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:00.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.696+0000 7f56273d5640 1 -- 172.21.15.189:0/2392961641 shutdown_connections 2024-09-16T14:46:00.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.696+0000 7f56273d5640 1 -- 172.21.15.189:0/2392961641 >> 172.21.15.189:0/2392961641 conn(0x7f56200a3320 msgr2=0x7f5620128640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:00.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.696+0000 7f56273d5640 1 -- 172.21.15.189:0/2392961641 shutdown_connections 2024-09-16T14:46:00.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.696+0000 7f56273d5640 1 -- 172.21.15.189:0/2392961641 wait complete. 2024-09-16T14:46:00.713 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush rm foo 2024-09-16T14:46:00.713 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:46:00.713 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm foo 2024-09-16T14:46:00.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.820+0000 7ffb49d38640 10 monclient: get_monmap_and_config 2024-09-16T14:46:00.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.820+0000 7ffb49d38640 10 monclient: build_initial_monmap 2024-09-16T14:46:00.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.820+0000 7ffb49d38640 10 monclient: monmap: 2024-09-16T14:46:00.825 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:00.825 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:00.825 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:00.824293+0000 2024-09-16T14:46:00.825 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:00.824293+0000 2024-09-16T14:46:00.826 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:00.826 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:00.826 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:00.826 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:00.826 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:00.827 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.820+0000 7ffb49d38640 1 Processor -- start 2024-09-16T14:46:00.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.820+0000 7ffb49d38640 1 -- start start 2024-09-16T14:46:00.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.820+0000 7ffb49d38640 10 monclient: init 2024-09-16T14:46:00.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:00.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:00.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient(hunting): picked mon.noname-c con 0x7ffb4409df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:00.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient(hunting): picked mon.noname-a con 0x7ffb4409e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:00.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient(hunting): picked mon.noname-b con 0x7ffb4409e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:00.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4409ee20 con 0x7ffb4409e2f0 2024-09-16T14:46:00.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4409f100 con 0x7ffb4409df10 2024-09-16T14:46:00.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4409f3e0 con 0x7ffb4409e740 2024-09-16T14:46:00.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:00.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient(hunting): authenticate will time out at 1142.940188s 2024-09-16T14:46:00.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb42ffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7ffb4409e2f0 legacy=0x7ffb441378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:46:00.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb437fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffb4409df10 0x7ffb4409d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58106/0 (socket says 172.21.15.189:58106) 2024-09-16T14:46:00.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb437fe640 1 -- 172.21.15.189:0/203706994 learned_addr learned my addr 172.21.15.189:0/203706994 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:00.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 1 -- 172.21.15.189:0/203706994 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332894834 0 0) 0x7ffb4409f100 con 0x7ffb4409df10 2024-09-16T14:46:00.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:00.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:00.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:00.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 10 monclient(hunting): my global_id is 94408 2024-09-16T14:46:00.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 1 -- 172.21.15.189:0/203706994 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb2c003530 con 0x7ffb4409df10 2024-09-16T14:46:00.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 1 -- 172.21.15.189:0/203706994 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3628711934 0 0) 0x7ffb2c003530 con 0x7ffb4409df10 2024-09-16T14:46:00.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 1 -- 172.21.15.189:0/203706994 >> v1:172.21.15.26:6790/0 conn(0x7ffb4409e740 legacy=0x7ffb4413b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:00.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 1 -- 172.21.15.189:0/203706994 >> v1:172.21.15.26:6789/0 conn(0x7ffb4409e2f0 legacy=0x7ffb441378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:00.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 10 monclient: _finish_hunting 0 2024-09-16T14:46:00.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 1 monclient: found mon.noname-c 2024-09-16T14:46:00.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:00.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 1 -- 172.21.15.189:0/203706994 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4409c690 con 0x7ffb4409df10 2024-09-16T14:46:00.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 10 monclient: _finish_auth 0 2024-09-16T14:46:00.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 10 monclient: _check_auth_tickets 2024-09-16T14:46:00.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:00.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 1 -- 172.21.15.189:0/203706994 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb34001620 con 0x7ffb4409df10 2024-09-16T14:46:00.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 5 monclient: authenticate success, global_id 94408 2024-09-16T14:46:00.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:00.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:00.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:00.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 10 monclient: dump: 2024-09-16T14:46:00.838 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:00.838 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:00.838 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.838 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.838 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:00.839 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:00.839 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:00.839 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:00.839 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:00.839 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:00.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 1 -- 172.21.15.189:0/203706994 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb34001940 con 0x7ffb4409df10 2024-09-16T14:46:00.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:00.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:00.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 1 -- 172.21.15.189:0/203706994 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb34001d10 con 0x7ffb4409df10 2024-09-16T14:46:00.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 4 set_mon_vals no callback set 2024-09-16T14:46:00.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:00.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:00.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb427fc640 10 monclient: dump: 2024-09-16T14:46:00.840 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:00.840 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:00.840 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.841 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.841 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:00.841 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:00.841 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:00.841 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:00.841 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:00.841 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient: shutdown 2024-09-16T14:46:00.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:00.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 1 -- 172.21.15.189:0/203706994 >> v1:172.21.15.189:6789/0 conn(0x7ffb4409df10 legacy=0x7ffb4409d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:00.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 1 -- 172.21.15.189:0/203706994 shutdown_connections 2024-09-16T14:46:00.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 1 -- 172.21.15.189:0/203706994 >> 172.21.15.189:0/203706994 conn(0x7ffb440a3320 msgr2=0x7ffb4409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:00.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 1 -- 172.21.15.189:0/203706994 shutdown_connections 2024-09-16T14:46:00.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 1 -- 172.21.15.189:0/203706994 wait complete. 2024-09-16T14:46:00.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient: build_initial_monmap 2024-09-16T14:46:00.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient: monmap: 2024-09-16T14:46:00.842 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:00.842 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:00.842 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:00.842 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:00.843 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:00.843 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:00.843 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:00.843 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:00.843 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:00.843 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 1 Processor -- start 2024-09-16T14:46:00.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 1 -- start start 2024-09-16T14:46:00.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient: init 2024-09-16T14:46:00.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:00.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:00.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient(hunting): picked mon.noname-c con 0x7ffb4409df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:00.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient(hunting): picked mon.noname-b con 0x7ffb4409e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:00.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient(hunting): picked mon.noname-a con 0x7ffb4409e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:00.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4413c050 con 0x7ffb4409e740 2024-09-16T14:46:00.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4406f940 con 0x7ffb4409e2f0 2024-09-16T14:46:00.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb44119da0 con 0x7ffb4409df10 2024-09-16T14:46:00.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:00.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient(hunting): authenticate will time out at 1142.942768s 2024-09-16T14:46:00.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb42ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffb4409e2f0 0x7ffb4413a1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58120/0 (socket says 172.21.15.189:58120) 2024-09-16T14:46:00.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb42ffd640 1 -- 172.21.15.189:0/1937352587 learned_addr learned my addr 172.21.15.189:0/1937352587 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:00.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb43fff640 1 -- 172.21.15.189:0/1937352587 >> v1:172.21.15.26:6789/0 conn(0x7ffb4409e740 legacy=0x7ffb441cf780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:46:00.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561863938 0 0) 0x7ffb4406f940 con 0x7ffb4409e2f0 2024-09-16T14:46:00.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:00.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:00.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:00.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient(hunting): my global_id is 94411 2024-09-16T14:46:00.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb30003530 con 0x7ffb4409e2f0 2024-09-16T14:46:00.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2505581256 0 0) 0x7ffb30003530 con 0x7ffb4409e2f0 2024-09-16T14:46:00.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb4406f940 con 0x7ffb4409e2f0 2024-09-16T14:46:00.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb38004190 con 0x7ffb4409e2f0 2024-09-16T14:46:00.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:00.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:00.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient(hunting): dump: 2024-09-16T14:46:00.847 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:00.849 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896497397 0 0) 0x7ffb44119da0 con 0x7ffb4409df10 2024-09-16T14:46:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:00.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:00.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient(hunting): my global_id is 5294 2024-09-16T14:46:00.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb30003530 con 0x7ffb4409df10 2024-09-16T14:46:00.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 905016705 0 0) 0x7ffb4406f940 con 0x7ffb4409e2f0 2024-09-16T14:46:00.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 >> v1:172.21.15.26:6790/0 conn(0x7ffb4409df10 legacy=0x7ffb44139aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:00.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 >> v1:172.21.15.26:6789/0 conn(0x7ffb4409e740 legacy=0x7ffb441cf780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 1 monclient: found mon.b 2024-09-16T14:46:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4406b380 con 0x7ffb4409e2f0 2024-09-16T14:46:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient: _finish_auth 0 2024-09-16T14:46:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 5 monclient: authenticate success, global_id 94411 2024-09-16T14:46:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient: _renew_subs 2024-09-16T14:46:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 1 -- 172.21.15.189:0/1937352587 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4411a6a0 con 0x7ffb4409e2f0 2024-09-16T14:46:00.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient: _renew_subs 2024-09-16T14:46:00.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb49d38640 1 -- 172.21.15.189:0/1937352587 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4406ce50 con 0x7ffb4409e2f0 2024-09-16T14:46:00.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb380044b0 con 0x7ffb4409e2f0 2024-09-16T14:46:00.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:00.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb38004840 con 0x7ffb4409e2f0 2024-09-16T14:46:00.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:00.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:00.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb41ffb640 4 set_mon_vals no callback set 2024-09-16T14:46:00.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.824+0000 7ffb2bfff640 10 monclient: dump: 2024-09-16T14:46:00.852 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:00.852 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:00.852 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:00.853 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.828+0000 7ffb49d38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.828+0000 7ffb49d38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.828+0000 7ffb49d38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.828+0000 7ffb49d38640 1 -- 172.21.15.189:0/1937352587 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb24005050 con 0x7ffb4409e2f0 2024-09-16T14:46:00.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.828+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ffb38004a20 con 0x7ffb4409e2f0 2024-09-16T14:46:00.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.828+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(74..74 src has 15..74) v4 ==== 5377+0+0 (unknown 3577921172 0 0) 0x7ffb38043a10 con 0x7ffb4409e2f0 2024-09-16T14:46:00.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.828+0000 7ffb437fe640 10 monclient: get_auth_request con 0x7ffb30036100 auth_method 0 2024-09-16T14:46:00.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.832+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ffb380148a0 con 0x7ffb4409e2f0 2024-09-16T14:46:00.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.832+0000 7ffb2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:00.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.832+0000 7ffb2bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.980+0000 7ffb49d38640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.980+0000 7ffb49d38640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.980+0000 7ffb49d38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.980+0000 7ffb49d38640 1 -- 172.21.15.189:0/1937352587 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7ffb24005300 con 0x7ffb4409e2f0 2024-09-16T14:46:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.984+0000 7ffb2bfff640 1 -- 172.21.15.189:0/1937352587 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=-39 (39) Directory not empty v74) v1 ==== 99+0+0 (unknown 3192285291 0 0) 0x7ffb38012d10 con 0x7ffb4409e2f0 2024-09-16T14:46:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.984+0000 7ffb2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:00.984+0000 7ffb2bfff640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-16T14:46:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-16T14:46:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:46:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush rm bar 2024-09-16T14:46:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:46:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm bar 2024-09-16T14:46:01.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.088+0000 7fe040c5a640 10 monclient: get_monmap_and_config 2024-09-16T14:46:01.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.088+0000 7fe040c5a640 10 monclient: build_initial_monmap 2024-09-16T14:46:01.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.088+0000 7fe040c5a640 10 monclient: monmap: 2024-09-16T14:46:01.092 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:01.093 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:01.093 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:01.092316+0000 2024-09-16T14:46:01.093 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:01.092316+0000 2024-09-16T14:46:01.093 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:01.093 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:01.093 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:01.093 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:01.093 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:01.093 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:01.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.088+0000 7fe040c5a640 1 Processor -- start 2024-09-16T14:46:01.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.088+0000 7fe040c5a640 1 -- start start 2024-09-16T14:46:01.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.088+0000 7fe040c5a640 10 monclient: init 2024-09-16T14:46:01.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.088+0000 7fe040c5a640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:01.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.088+0000 7fe040c5a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:01.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.088+0000 7fe040c5a640 10 monclient(hunting): picked mon.noname-a con 0x7fe03c1315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.088+0000 7fe040c5a640 10 monclient(hunting): picked mon.noname-b con 0x7fe03c09b320 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.088+0000 7fe040c5a640 10 monclient(hunting): picked mon.noname-c con 0x7fe03c132090 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.088+0000 7fe040c5a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe03c09edd0 con 0x7fe03c1315a0 2024-09-16T14:46:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.088+0000 7fe040c5a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe03c09bc00 con 0x7fe03c132090 2024-09-16T14:46:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.088+0000 7fe040c5a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe03c09e900 con 0x7fe03c09b320 2024-09-16T14:46:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.088+0000 7fe040c5a640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.088+0000 7fe040c5a640 10 monclient(hunting): authenticate will time out at 1143.207918s 2024-09-16T14:46:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03a59c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fe03c1315a0 legacy=0x7fe03c131980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:46:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03ad9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe03c132090 0x7fe03c13a200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58136/0 (socket says 172.21.15.189:58136) 2024-09-16T14:46:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03ad9d640 1 -- 172.21.15.189:0/2775376446 learned_addr learned my addr 172.21.15.189:0/2775376446 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 1 -- 172.21.15.189:0/2775376446 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417301051 0 0) 0x7fe03c09bc00 con 0x7fe03c132090 2024-09-16T14:46:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 10 monclient(hunting): _negotiate 2024-09-16T14:46:01.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:01.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:01.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 10 monclient(hunting): my global_id is 94414 2024-09-16T14:46:01.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 1 -- 172.21.15.189:0/2775376446 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe024003530 con 0x7fe03c132090 2024-09-16T14:46:01.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 1 -- 172.21.15.189:0/2775376446 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2454198551 0 0) 0x7fe024003530 con 0x7fe03c132090 2024-09-16T14:46:01.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 1 -- 172.21.15.189:0/2775376446 >> v1:172.21.15.26:6790/0 conn(0x7fe03c09b320 legacy=0x7fe03c136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:01.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 1 -- 172.21.15.189:0/2775376446 >> v1:172.21.15.26:6789/0 conn(0x7fe03c1315a0 legacy=0x7fe03c131980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:01.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 10 monclient: _finish_hunting 0 2024-09-16T14:46:01.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 1 monclient: found mon.noname-c 2024-09-16T14:46:01.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:01.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:01.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 1 -- 172.21.15.189:0/2775376446 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03c09b920 con 0x7fe03c132090 2024-09-16T14:46:01.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 10 monclient: _finish_auth 0 2024-09-16T14:46:01.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 10 monclient: _check_auth_tickets 2024-09-16T14:46:01.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:01.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 5 monclient: authenticate success, global_id 94414 2024-09-16T14:46:01.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 1 -- 172.21.15.189:0/2775376446 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe0340019a0 con 0x7fe03c132090 2024-09-16T14:46:01.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:01.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:01.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:01.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 10 monclient: dump: 2024-09-16T14:46:01.103 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:01.103 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:01.103 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:01.103 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:01.104 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:01.104 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:01.104 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:01.104 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:01.104 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:01.104 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:01.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:01.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 1 -- 172.21.15.189:0/2775376446 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe034001cc0 con 0x7fe03c132090 2024-09-16T14:46:01.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:01.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:01.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 4 set_mon_vals no callback set 2024-09-16T14:46:01.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 1 -- 172.21.15.189:0/2775376446 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe034002090 con 0x7fe03c132090 2024-09-16T14:46:01.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:01.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:01.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03959a640 10 monclient: dump: 2024-09-16T14:46:01.105 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:01.106 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:01.106 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:01.106 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:01.106 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:01.106 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:01.106 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:01.106 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:01.106 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:01.106 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:01.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 10 monclient: shutdown 2024-09-16T14:46:01.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:01.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 1 -- 172.21.15.189:0/2775376446 >> v1:172.21.15.189:6789/0 conn(0x7fe03c132090 legacy=0x7fe03c13a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:01.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 1 -- 172.21.15.189:0/2775376446 shutdown_connections 2024-09-16T14:46:01.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 1 -- 172.21.15.189:0/2775376446 >> 172.21.15.189:0/2775376446 conn(0x7fe03c0a3320 msgr2=0x7fe03c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:01.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 1 -- 172.21.15.189:0/2775376446 shutdown_connections 2024-09-16T14:46:01.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 1 -- 172.21.15.189:0/2775376446 wait complete. 2024-09-16T14:46:01.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 10 monclient: build_initial_monmap 2024-09-16T14:46:01.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 10 monclient: monmap: 2024-09-16T14:46:01.107 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:01.107 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:01.107 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:01.108 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:01.108 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:01.108 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:01.108 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:01.108 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:01.108 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:01.108 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:01.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 1 Processor -- start 2024-09-16T14:46:01.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 1 -- start start 2024-09-16T14:46:01.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 10 monclient: init 2024-09-16T14:46:01.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:01.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:01.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 10 monclient(hunting): picked mon.noname-b con 0x7fe03c1315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:01.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 10 monclient(hunting): picked mon.noname-c con 0x7fe03c132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:01.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 10 monclient(hunting): picked mon.noname-a con 0x7fe03c1caf20 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:01.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe03c09cd20 con 0x7fe03c1caf20 2024-09-16T14:46:01.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe03c09a6d0 con 0x7fe03c1315a0 2024-09-16T14:46:01.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe03c0a0d80 con 0x7fe03c132090 2024-09-16T14:46:01.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:01.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 10 monclient(hunting): authenticate will time out at 1143.210776s 2024-09-16T14:46:01.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03a59c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe03c1315a0 0x7fe03c1c4bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58144/0 (socket says 172.21.15.189:58144) 2024-09-16T14:46:01.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03a59c640 1 -- 172.21.15.189:0/388964263 learned_addr learned my addr 172.21.15.189:0/388964263 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:01.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe03ad9d640 1 -- 172.21.15.189:0/388964263 >> v1:172.21.15.26:6789/0 conn(0x7fe03c1caf20 legacy=0x7fe03c1cd310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:46:01.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 1 -- 172.21.15.189:0/388964263 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001890899 0 0) 0x7fe03c09a6d0 con 0x7fe03c1315a0 2024-09-16T14:46:01.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:01.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:01.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:01.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 10 monclient(hunting): my global_id is 94417 2024-09-16T14:46:01.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 1 -- 172.21.15.189:0/388964263 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe01c003530 con 0x7fe03c1315a0 2024-09-16T14:46:01.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 1 -- 172.21.15.189:0/388964263 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1270772281 0 0) 0x7fe01c003530 con 0x7fe03c1315a0 2024-09-16T14:46:01.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 1 -- 172.21.15.189:0/388964263 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe03c09a6d0 con 0x7fe03c1315a0 2024-09-16T14:46:01.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 1 -- 172.21.15.189:0/388964263 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe02c004190 con 0x7fe03c1315a0 2024-09-16T14:46:01.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:01.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:01.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 10 monclient(hunting): dump: 2024-09-16T14:46:01.111 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:01.112 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:01.112 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:01.112 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:01.112 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:01.112 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:01.112 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:01.112 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:01.112 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:01.112 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:01.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:01.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 1 -- 172.21.15.189:0/388964263 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2076608163 0 0) 0x7fe03c09a6d0 con 0x7fe03c1315a0 2024-09-16T14:46:01.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 1 -- 172.21.15.189:0/388964263 >> v1:172.21.15.26:6790/0 conn(0x7fe03c132090 legacy=0x7fe03c130990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:01.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 1 -- 172.21.15.189:0/388964263 >> v1:172.21.15.26:6789/0 conn(0x7fe03c1caf20 legacy=0x7fe03c1cd310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:01.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:01.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 1 monclient: found mon.b 2024-09-16T14:46:01.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:01.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:01.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 1 -- 172.21.15.189:0/388964263 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03c06b380 con 0x7fe03c1315a0 2024-09-16T14:46:01.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 10 monclient: _finish_auth 0 2024-09-16T14:46:01.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:01.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:01.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 5 monclient: authenticate success, global_id 94417 2024-09-16T14:46:01.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 10 monclient: _renew_subs 2024-09-16T14:46:01.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:01.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 1 -- 172.21.15.189:0/388964263 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03c11f3c0 con 0x7fe03c1315a0 2024-09-16T14:46:01.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 10 monclient: _renew_subs 2024-09-16T14:46:01.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:01.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe040c5a640 1 -- 172.21.15.189:0/388964263 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03c11d000 con 0x7fe03c1315a0 2024-09-16T14:46:01.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 1 -- 172.21.15.189:0/388964263 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe02c004480 con 0x7fe03c1315a0 2024-09-16T14:46:01.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:01.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 1 -- 172.21.15.189:0/388964263 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe02c004810 con 0x7fe03c1315a0 2024-09-16T14:46:01.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:01.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:01.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe038d99640 4 set_mon_vals no callback set 2024-09-16T14:46:01.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.092+0000 7fe02affd640 10 monclient: dump: 2024-09-16T14:46:01.115 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:01.115 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:01.115 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:01.115 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:01.116 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:01.116 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:01.116 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:01.116 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:01.116 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:01.116 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:01.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.096+0000 7fe040c5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:01.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.096+0000 7fe040c5a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:01.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.096+0000 7fe040c5a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:01.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.096+0000 7fe02affd640 1 -- 172.21.15.189:0/388964263 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe02c0049f0 con 0x7fe03c1315a0 2024-09-16T14:46:01.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.096+0000 7fe040c5a640 1 -- 172.21.15.189:0/388964263 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe020005050 con 0x7fe03c1315a0 2024-09-16T14:46:01.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.096+0000 7fe02affd640 1 -- 172.21.15.189:0/388964263 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(74..74 src has 15..74) v4 ==== 5377+0+0 (unknown 3577921172 0 0) 0x7fe02c0439c0 con 0x7fe03c1315a0 2024-09-16T14:46:01.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.096+0000 7fe039d9b640 10 monclient: get_auth_request con 0x7fe01c036000 auth_method 0 2024-09-16T14:46:01.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.100+0000 7fe02affd640 1 -- 172.21.15.189:0/388964263 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe02c014850 con 0x7fe03c1315a0 2024-09-16T14:46:01.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.100+0000 7fe02affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:01.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.100+0000 7fe02affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:01.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.260+0000 7fe040c5a640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T14:46:01.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.260+0000 7fe040c5a640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T14:46:01.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.260+0000 7fe040c5a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:01.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.260+0000 7fe040c5a640 1 -- 172.21.15.189:0/388964263 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7fe020005300 con 0x7fe03c1315a0 2024-09-16T14:46:01.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.264+0000 7fe02affd640 1 -- 172.21.15.189:0/388964263 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=-39 (39) Directory not empty v74) v1 ==== 99+0+0 (unknown 527356516 0 0) 0x7fe02c012cc0 con 0x7fe03c1315a0 2024-09-16T14:46:01.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.264+0000 7fe02affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T14:46:01.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.264+0000 7fe02affd640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-16T14:46:01.266 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-16T14:46:01.270 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:46:01.270 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush unlink host1 bar 2024-09-16T14:46:01.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.380+0000 7f5a09507640 10 monclient: get_monmap_and_config 2024-09-16T14:46:01.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.380+0000 7f5a09507640 10 monclient: build_initial_monmap 2024-09-16T14:46:01.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.380+0000 7f5a09507640 10 monclient: monmap: 2024-09-16T14:46:01.388 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:01.388 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:01.389 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:01.384982+0000 2024-09-16T14:46:01.389 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:01.384982+0000 2024-09-16T14:46:01.389 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:01.389 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:01.389 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:01.389 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:01.389 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:01.389 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:01.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.380+0000 7f5a09507640 1 Processor -- start 2024-09-16T14:46:01.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.380+0000 7f5a09507640 1 -- start start 2024-09-16T14:46:01.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.380+0000 7f5a09507640 10 monclient: init 2024-09-16T14:46:01.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:01.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient(hunting): picked mon.noname-a con 0x7f5a041318a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient(hunting): picked mon.noname-b con 0x7f5a04134d50 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient(hunting): picked mon.noname-c con 0x7f5a0413b2b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0409d8a0 con 0x7f5a041318a0 2024-09-16T14:46:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0409ddf0 con 0x7f5a0413b2b0 2024-09-16T14:46:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0409e8b0 con 0x7f5a04134d50 2024-09-16T14:46:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient(hunting): authenticate will time out at 1143.500549s 2024-09-16T14:46:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a037fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5a0413b2b0 0x7f5a0413d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58152/0 (socket says 172.21.15.189:58152) 2024-09-16T14:46:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a037fe640 1 -- 172.21.15.189:0/4139325661 learned_addr learned my addr 172.21.15.189:0/4139325661 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a02ffd640 1 -- 172.21.15.189:0/4139325661 >> v1:172.21.15.26:6789/0 conn(0x7f5a041318a0 legacy=0x7f5a04131c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:46:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 1 -- 172.21.15.189:0/4139325661 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208938472 0 0) 0x7f5a0409ddf0 con 0x7f5a0413b2b0 2024-09-16T14:46:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 10 monclient(hunting): my global_id is 94420 2024-09-16T14:46:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 1 -- 172.21.15.189:0/4139325661 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59ec003530 con 0x7f5a0413b2b0 2024-09-16T14:46:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 1 -- 172.21.15.189:0/4139325661 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1712794737 0 0) 0x7f59ec003530 con 0x7f5a0413b2b0 2024-09-16T14:46:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 1 -- 172.21.15.189:0/4139325661 >> v1:172.21.15.26:6790/0 conn(0x7f5a04134d50 legacy=0x7f5a04139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 1 -- 172.21.15.189:0/4139325661 >> v1:172.21.15.26:6789/0 conn(0x7f5a041318a0 legacy=0x7f5a04131c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:46:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 1 monclient: found mon.noname-c 2024-09-16T14:46:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 1 -- 172.21.15.189:0/4139325661 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0409eb90 con 0x7f5a0413b2b0 2024-09-16T14:46:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 10 monclient: _finish_auth 0 2024-09-16T14:46:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 1 -- 172.21.15.189:0/4139325661 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f59fc0019a0 con 0x7f5a0413b2b0 2024-09-16T14:46:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 10 monclient: dump: 2024-09-16T14:46:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:01.393 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 5 monclient: authenticate success, global_id 94420 2024-09-16T14:46:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 1 -- 172.21.15.189:0/4139325661 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59fc001c50 con 0x7f5a0413b2b0 2024-09-16T14:46:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 1 -- 172.21.15.189:0/4139325661 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f59fc001fe0 con 0x7f5a0413b2b0 2024-09-16T14:46:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a01ffb640 10 monclient: dump: 2024-09-16T14:46:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:01.395 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 4 set_mon_vals no callback set 2024-09-16T14:46:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient: shutdown 2024-09-16T14:46:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 1 -- 172.21.15.189:0/4139325661 >> v1:172.21.15.189:6789/0 conn(0x7f5a0413b2b0 legacy=0x7f5a0413d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 1 -- 172.21.15.189:0/4139325661 shutdown_connections 2024-09-16T14:46:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 1 -- 172.21.15.189:0/4139325661 >> 172.21.15.189:0/4139325661 conn(0x7f5a040a3320 msgr2=0x7f5a0409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 1 -- 172.21.15.189:0/4139325661 shutdown_connections 2024-09-16T14:46:01.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 1 -- 172.21.15.189:0/4139325661 wait complete. 2024-09-16T14:46:01.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient: build_initial_monmap 2024-09-16T14:46:01.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient: monmap: 2024-09-16T14:46:01.396 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:01.396 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:01.396 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:01.396 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:01.396 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:01.396 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:01.397 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:01.397 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:01.397 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:01.397 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:01.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 1 Processor -- start 2024-09-16T14:46:01.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 1 -- start start 2024-09-16T14:46:01.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient: init 2024-09-16T14:46:01.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:01.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:01.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient(hunting): picked mon.noname-c con 0x7f5a041318a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient(hunting): picked mon.noname-b con 0x7f5a04134d50 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient(hunting): picked mon.noname-a con 0x7f5a0413b2b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0411ba80 con 0x7f5a0413b2b0 2024-09-16T14:46:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0406b380 con 0x7f5a04134d50 2024-09-16T14:46:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0413abd0 con 0x7f5a041318a0 2024-09-16T14:46:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient(hunting): authenticate will time out at 1143.503258s 2024-09-16T14:46:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a027fc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5a04134d50 0x7f5a04139c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58158/0 (socket says 172.21.15.189:58158) 2024-09-16T14:46:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a027fc640 1 -- 172.21.15.189:0/3185272138 learned_addr learned my addr 172.21.15.189:0/3185272138 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a037fe640 1 -- 172.21.15.189:0/3185272138 >> v1:172.21.15.26:6789/0 conn(0x7f5a0413b2b0 legacy=0x7f5a041d1480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:46:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 1 -- 172.21.15.189:0/3185272138 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473763734 0 0) 0x7f5a0406b380 con 0x7f5a04134d50 2024-09-16T14:46:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 10 monclient(hunting): my global_id is 94423 2024-09-16T14:46:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 1 -- 172.21.15.189:0/3185272138 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59f0003530 con 0x7f5a04134d50 2024-09-16T14:46:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 1 -- 172.21.15.189:0/3185272138 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3746589328 0 0) 0x7f59f0003530 con 0x7f5a04134d50 2024-09-16T14:46:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 1 -- 172.21.15.189:0/3185272138 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a0406b380 con 0x7f5a04134d50 2024-09-16T14:46:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 1 -- 172.21.15.189:0/3185272138 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f59f8001940 con 0x7f5a04134d50 2024-09-16T14:46:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 10 monclient(hunting): dump: 2024-09-16T14:46:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:01.403 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 1 -- 172.21.15.189:0/3185272138 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3834567836 0 0) 0x7f5a0406b380 con 0x7f5a04134d50 2024-09-16T14:46:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 1 -- 172.21.15.189:0/3185272138 >> v1:172.21.15.26:6790/0 conn(0x7f5a041318a0 legacy=0x7f5a04139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 1 -- 172.21.15.189:0/3185272138 >> v1:172.21.15.26:6789/0 conn(0x7f5a0413b2b0 legacy=0x7f5a041d1480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 1 monclient: found mon.b 2024-09-16T14:46:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 1 -- 172.21.15.189:0/3185272138 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a040a1cd0 con 0x7f5a04134d50 2024-09-16T14:46:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f59eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 5 monclient: authenticate success, global_id 94423 2024-09-16T14:46:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient: _renew_subs 2024-09-16T14:46:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.384+0000 7f5a09507640 1 -- 172.21.15.189:0/3185272138 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0411b1e0 con 0x7f5a04134d50 2024-09-16T14:46:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f5a09507640 10 monclient: _renew_subs 2024-09-16T14:46:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f5a09507640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f5a09507640 1 -- 172.21.15.189:0/3185272138 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a04119900 con 0x7f5a04134d50 2024-09-16T14:46:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f59eb7fe640 1 -- 172.21.15.189:0/3185272138 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59f8003620 con 0x7f5a04134d50 2024-09-16T14:46:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f59eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f59eb7fe640 1 -- 172.21.15.189:0/3185272138 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f59f80039b0 con 0x7f5a04134d50 2024-09-16T14:46:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f59eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f59eb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f59eb7fe640 10 monclient: dump: 2024-09-16T14:46:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:01.406 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f5a017fa640 4 set_mon_vals no callback set 2024-09-16T14:46:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f59eb7fe640 1 -- 172.21.15.189:0/3185272138 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f59f8003bd0 con 0x7f5a04134d50 2024-09-16T14:46:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f59eb7fe640 1 -- 172.21.15.189:0/3185272138 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(74..74 src has 15..74) v4 ==== 5377+0+0 (unknown 3577921172 0 0) 0x7f59f80438e0 con 0x7f5a04134d50 2024-09-16T14:46:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f5a02ffd640 10 monclient: get_auth_request con 0x7f59f0036060 auth_method 0 2024-09-16T14:46:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f5a09507640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f5a09507640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f5a09507640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.388+0000 7f5a09507640 1 -- 172.21.15.189:0/3185272138 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a040008e0 con 0x7f5a04134d50 2024-09-16T14:46:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.392+0000 7f59eb7fe640 1 -- 172.21.15.189:0/3185272138 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f59f80147d0 con 0x7f5a04134d50 2024-09-16T14:46:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.392+0000 7f59eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.392+0000 7f59eb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:01.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.552+0000 7f5a09507640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T14:46:01.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.552+0000 7f5a09507640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T14:46:01.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.552+0000 7f5a09507640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:01.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.552+0000 7f5a09507640 1 -- 172.21.15.189:0/3185272138 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"} v 0) v1 -- 0x7f5a0413a3b0 con 0x7f5a04134d50 2024-09-16T14:46:01.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.812+0000 7f59eb7fe640 1 -- 172.21.15.189:0/3185272138 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}]=0 removed item id -11 name 'host1' from crush map v75) v1 ==== 147+0+0 (unknown 2832240248 0 0) 0x7f59f80143c0 con 0x7f5a04134d50 2024-09-16T14:46:01.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.812+0000 7f59eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T14:46:01.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.812+0000 7f59eb7fe640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-16T14:46:01.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.916+0000 7f5a09507640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T14:46:01.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.916+0000 7f5a09507640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T14:46:01.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.916+0000 7f5a09507640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:01.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.916+0000 7f5a09507640 1 -- 172.21.15.189:0/3185272138 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"} v 0) v1 -- 0x7f5a0413a5e0 con 0x7f5a04134d50 2024-09-16T14:46:01.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.916+0000 7f59eb7fe640 1 -- 172.21.15.189:0/3185272138 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}]=0 item -11 does not appear in that position v75) v1 ==== 141+0+0 (unknown 2262321512 0 0) 0x7f59f80145a0 con 0x7f5a04134d50 2024-09-16T14:46:01.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.916+0000 7f59eb7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T14:46:01.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.916+0000 7f59eb7fe640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-16T14:46:01.920 INFO:tasks.workunit.client.0.smithi189.stderr:item -11 does not appear in that position 2024-09-16T14:46:01.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.920+0000 7f5a09507640 1 -- 172.21.15.189:0/3185272138 >> v1:172.21.15.189:6812/3196161582 conn(0x7f59f0036060 legacy=0x7f59f0038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:01.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.920+0000 7f5a09507640 10 monclient: shutdown 2024-09-16T14:46:01.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.920+0000 7f5a09507640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:01.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.920+0000 7f5a09507640 1 -- 172.21.15.189:0/3185272138 >> v1:172.21.15.189:6789/0 conn(0x7f5a04134d50 legacy=0x7f5a04139c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:01.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.920+0000 7f5a09507640 1 -- 172.21.15.189:0/3185272138 shutdown_connections 2024-09-16T14:46:01.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.920+0000 7f5a09507640 1 -- 172.21.15.189:0/3185272138 >> 172.21.15.189:0/3185272138 conn(0x7f5a040a3320 msgr2=0x7f5a04133440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:01.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.920+0000 7f5a09507640 1 -- 172.21.15.189:0/3185272138 shutdown_connections 2024-09-16T14:46:01.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:01.920+0000 7f5a09507640 1 -- 172.21.15.189:0/3185272138 wait complete. 2024-09-16T14:46:01.938 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T14:46:01.938 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 1 2024-09-16T14:46:01.938 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c host1 2024-09-16T14:46:02.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4680b64640 10 monclient: get_monmap_and_config 2024-09-16T14:46:02.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4680b64640 10 monclient: build_initial_monmap 2024-09-16T14:46:02.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4680b64640 10 monclient: monmap: 2024-09-16T14:46:02.052 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:02.052 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.052 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:02.051750+0000 2024-09-16T14:46:02.052 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:02.051750+0000 2024-09-16T14:46:02.052 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:02.052 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:02.052 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:02.053 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:02.053 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:02.053 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:02.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4680b64640 1 Processor -- start 2024-09-16T14:46:02.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4680b64640 1 -- start start 2024-09-16T14:46:02.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4680b64640 10 monclient: init 2024-09-16T14:46:02.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4680b64640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:02.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4680b64640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:02.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4680b64640 10 monclient(hunting): picked mon.noname-c con 0x7f467c1315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:02.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4680b64640 10 monclient(hunting): picked mon.noname-a con 0x7f467c09b320 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:02.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4680b64640 10 monclient(hunting): picked mon.noname-b con 0x7f467c132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:02.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4680b64640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c09edd0 con 0x7f467c09b320 2024-09-16T14:46:02.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4680b64640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c09bc00 con 0x7f467c1315a0 2024-09-16T14:46:02.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4680b64640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c09e900 con 0x7f467c132090 2024-09-16T14:46:02.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4680b64640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:02.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4680b64640 10 monclient(hunting): authenticate will time out at 1144.167608s 2024-09-16T14:46:02.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f467a59c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f467c1315a0 0x7f467c131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58170/0 (socket says 172.21.15.189:58170) 2024-09-16T14:46:02.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f467a59c640 1 -- 172.21.15.189:0/105582016 learned_addr learned my addr 172.21.15.189:0/105582016 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:02.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f4679d9b640 1 -- 172.21.15.189:0/105582016 >> v1:172.21.15.26:6789/0 conn(0x7f467c09b320 legacy=0x7f467c136ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:46:02.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f467959a640 1 -- 172.21.15.189:0/105582016 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497125482 0 0) 0x7f467c09bc00 con 0x7f467c1315a0 2024-09-16T14:46:02.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f467959a640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f467959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f467959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.048+0000 7f467959a640 10 monclient(hunting): my global_id is 94426 2024-09-16T14:46:02.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 1 -- 172.21.15.189:0/105582016 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4664003530 con 0x7f467c1315a0 2024-09-16T14:46:02.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 1 -- 172.21.15.189:0/105582016 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 645990159 0 0) 0x7f4664003530 con 0x7f467c1315a0 2024-09-16T14:46:02.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 1 -- 172.21.15.189:0/105582016 >> v1:172.21.15.26:6790/0 conn(0x7f467c132090 legacy=0x7f467c13a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 1 -- 172.21.15.189:0/105582016 >> v1:172.21.15.26:6789/0 conn(0x7f467c09b320 legacy=0x7f467c136ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:02.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 10 monclient: _finish_hunting 0 2024-09-16T14:46:02.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 1 monclient: found mon.noname-c 2024-09-16T14:46:02.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:02.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:02.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 1 -- 172.21.15.189:0/105582016 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c09b920 con 0x7f467c1315a0 2024-09-16T14:46:02.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 10 monclient: _finish_auth 0 2024-09-16T14:46:02.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 10 monclient: _check_auth_tickets 2024-09-16T14:46:02.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:02.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 5 monclient: authenticate success, global_id 94426 2024-09-16T14:46:02.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:02.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 1 -- 172.21.15.189:0/105582016 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f466c001620 con 0x7f467c1315a0 2024-09-16T14:46:02.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:02.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 10 monclient: dump: 2024-09-16T14:46:02.070 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:02.070 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.070 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.070 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.071 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.071 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:02.071 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.071 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.072 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.072 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:02.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:02.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 1 -- 172.21.15.189:0/105582016 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f466c001940 con 0x7f467c1315a0 2024-09-16T14:46:02.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:02.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 1 -- 172.21.15.189:0/105582016 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f466c001cd0 con 0x7f467c1315a0 2024-09-16T14:46:02.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:02.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 4 set_mon_vals no callback set 2024-09-16T14:46:02.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:02.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467959a640 10 monclient: dump: 2024-09-16T14:46:02.072 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:02.073 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.073 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.073 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.073 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.073 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:02.073 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.073 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.073 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.073 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:02.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient: shutdown 2024-09-16T14:46:02.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:02.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 1 -- 172.21.15.189:0/105582016 >> v1:172.21.15.189:6789/0 conn(0x7f467c1315a0 legacy=0x7f467c131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 1 -- 172.21.15.189:0/105582016 shutdown_connections 2024-09-16T14:46:02.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 1 -- 172.21.15.189:0/105582016 >> 172.21.15.189:0/105582016 conn(0x7f467c0a3320 msgr2=0x7f467c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:02.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 1 -- 172.21.15.189:0/105582016 shutdown_connections 2024-09-16T14:46:02.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 1 -- 172.21.15.189:0/105582016 wait complete. 2024-09-16T14:46:02.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient: build_initial_monmap 2024-09-16T14:46:02.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient: monmap: 2024-09-16T14:46:02.074 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:02.074 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:02.074 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:02.074 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:02.075 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:02.075 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:02.075 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:02.075 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:02.075 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:02.075 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:02.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 1 Processor -- start 2024-09-16T14:46:02.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 1 -- start start 2024-09-16T14:46:02.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient: init 2024-09-16T14:46:02.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:02.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:02.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient(hunting): picked mon.noname-a con 0x7f467c1315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:02.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient(hunting): picked mon.noname-c con 0x7f467c132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:02.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient(hunting): picked mon.noname-b con 0x7f467c12ffd0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:02.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c09cd20 con 0x7f467c1315a0 2024-09-16T14:46:02.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c09e1d0 con 0x7f467c12ffd0 2024-09-16T14:46:02.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c06b380 con 0x7f467c132090 2024-09-16T14:46:02.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:02.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient(hunting): authenticate will time out at 1144.170316s 2024-09-16T14:46:02.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467ad9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f467c12ffd0 0x7f467c1cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58178/0 (socket says 172.21.15.189:58178) 2024-09-16T14:46:02.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467a59c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f467c1315a0 legacy=0x7f467c12f8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:46:02.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f467ad9d640 1 -- 172.21.15.189:0/3664617523 learned_addr learned my addr 172.21.15.189:0/3664617523 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:02.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 1 -- 172.21.15.189:0/3664617523 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943527641 0 0) 0x7f467c09e1d0 con 0x7f467c12ffd0 2024-09-16T14:46:02.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 10 monclient(hunting): my global_id is 94429 2024-09-16T14:46:02.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 1 -- 172.21.15.189:0/3664617523 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f465c003530 con 0x7f467c12ffd0 2024-09-16T14:46:02.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 1 -- 172.21.15.189:0/3664617523 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3970579855 0 0) 0x7f465c003530 con 0x7f467c12ffd0 2024-09-16T14:46:02.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 1 -- 172.21.15.189:0/3664617523 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f467c09e1d0 con 0x7f467c12ffd0 2024-09-16T14:46:02.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 1 -- 172.21.15.189:0/3664617523 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4674001c40 con 0x7f467c12ffd0 2024-09-16T14:46:02.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:02.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 10 monclient(hunting): dump: 2024-09-16T14:46:02.078 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:02.078 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.078 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.078 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.078 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.078 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:02.078 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.078 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.078 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.078 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:02.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:02.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 1 -- 172.21.15.189:0/3664617523 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 117557893 0 0) 0x7f467c09e1d0 con 0x7f467c12ffd0 2024-09-16T14:46:02.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 1 -- 172.21.15.189:0/3664617523 >> v1:172.21.15.26:6790/0 conn(0x7f467c132090 legacy=0x7f467c1cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 1 -- 172.21.15.189:0/3664617523 >> v1:172.21.15.26:6789/0 conn(0x7f467c1315a0 legacy=0x7f467c12f8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:02.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:02.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 1 monclient: found mon.b 2024-09-16T14:46:02.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:02.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:02.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 1 -- 172.21.15.189:0/3664617523 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c0a1ab0 con 0x7f467c12ffd0 2024-09-16T14:46:02.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 10 monclient: _finish_auth 0 2024-09-16T14:46:02.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:02.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:02.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 5 monclient: authenticate success, global_id 94429 2024-09-16T14:46:02.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient: _renew_subs 2024-09-16T14:46:02.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:02.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 1 -- 172.21.15.189:0/3664617523 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467c11d9f0 con 0x7f467c12ffd0 2024-09-16T14:46:02.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient: _renew_subs 2024-09-16T14:46:02.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 1 -- 172.21.15.189:0/3664617523 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4674002010 con 0x7f467c12ffd0 2024-09-16T14:46:02.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:02.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 1 -- 172.21.15.189:0/3664617523 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f467c11cb00 con 0x7f467c12ffd0 2024-09-16T14:46:02.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:02.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 1 -- 172.21.15.189:0/3664617523 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4674003df0 con 0x7f467c12ffd0 2024-09-16T14:46:02.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:02.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 10 monclient: dump: 2024-09-16T14:46:02.080 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:02.081 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.081 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.081 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.081 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.081 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:02.081 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.081 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.081 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.081 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:02.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4678d99640 4 set_mon_vals no callback set 2024-09-16T14:46:02.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:02.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:02.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:02.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f466affd640 1 -- 172.21.15.189:0/3664617523 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4674003fd0 con 0x7f467c12ffd0 2024-09-16T14:46:02.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.052+0000 7f4680b64640 1 -- 172.21.15.189:0/3664617523 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f467c0008e0 con 0x7f467c12ffd0 2024-09-16T14:46:02.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.056+0000 7f466affd640 1 -- 172.21.15.189:0/3664617523 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(75..75 src has 15..75) v4 ==== 5337+0+0 (unknown 2993449477 0 0) 0x7f4674043cc0 con 0x7f467c12ffd0 2024-09-16T14:46:02.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.056+0000 7f467a59c640 10 monclient: get_auth_request con 0x7f465c036010 auth_method 0 2024-09-16T14:46:02.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.060+0000 7f466affd640 1 -- 172.21.15.189:0/3664617523 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f46740148f0 con 0x7f467c12ffd0 2024-09-16T14:46:02.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.060+0000 7f466affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:02.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.060+0000 7f466affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:02.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.216+0000 7f4680b64640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:46:02.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.216+0000 7f4680b64640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T14:46:02.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.216+0000 7f4680b64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:02.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.216+0000 7f4680b64640 1 -- 172.21.15.189:0/3664617523 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f467c1c71e0 con 0x7f467c12ffd0 2024-09-16T14:46:02.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.216+0000 7f466affd640 1 -- 172.21.15.189:0/3664617523 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v75) v1 ==== 56+0+1340 (unknown 3476242654 0 181529295) 0x7f4674014710 con 0x7f467c12ffd0 2024-09-16T14:46:02.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.216+0000 7f466affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T14:46:02.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.216+0000 7f466affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:02.295 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:46:02.295 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T14:46:02.295 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T14:46:02.295 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T14:46:02.298 INFO:tasks.ceph.mon.a:Started 2024-09-16T14:46:02.298 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:46:02.298 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:46:02.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.304+0000 7f4680b64640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:46:02.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.304+0000 7f4680b64640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T14:46:02.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.304+0000 7f4680b64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:02.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.304+0000 7f4680b64640 1 -- 172.21.15.189:0/3664617523 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f467c09b320 con 0x7f467c12ffd0 2024-09-16T14:46:02.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.304+0000 7f466affd640 1 -- 172.21.15.189:0/3664617523 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v75) v1 ==== 56+0+1340 (unknown 3476242654 0 181529295) 0x7f4674014710 con 0x7f467c12ffd0 2024-09-16T14:46:02.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.304+0000 7f466affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T14:46:02.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.304+0000 7f466affd640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:46:02.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.308+0000 7f4680b64640 1 -- 172.21.15.189:0/3664617523 >> v1:172.21.15.189:6812/3196161582 conn(0x7f465c036010 legacy=0x7f465c0384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.308+0000 7f4680b64640 10 monclient: shutdown 2024-09-16T14:46:02.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.308+0000 7f4680b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:02.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.308+0000 7f4680b64640 1 -- 172.21.15.189:0/3664617523 >> v1:172.21.15.189:6789/0 conn(0x7f467c12ffd0 legacy=0x7f467c1cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.308+0000 7f4680b64640 1 -- 172.21.15.189:0/3664617523 shutdown_connections 2024-09-16T14:46:02.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.308+0000 7f4680b64640 1 -- 172.21.15.189:0/3664617523 >> 172.21.15.189:0/3664617523 conn(0x7f467c0a3320 msgr2=0x7f467c130760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:02.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.308+0000 7f4680b64640 1 -- 172.21.15.189:0/3664617523 shutdown_connections 2024-09-16T14:46:02.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.308+0000 7f4680b64640 1 -- 172.21.15.189:0/3664617523 wait complete. 2024-09-16T14:46:02.323 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm osd.7 host1 2024-09-16T14:46:02.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 10 monclient: get_monmap_and_config 2024-09-16T14:46:02.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 10 monclient: build_initial_monmap 2024-09-16T14:46:02.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 10 monclient: monmap: 2024-09-16T14:46:02.437 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:02.438 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.438 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:46:02.429633+0000 2024-09-16T14:46:02.438 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:46:02.429633+0000 2024-09-16T14:46:02.438 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:02.438 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.438 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:02.438 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:02.438 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:02.438 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 1 Processor -- start 2024-09-16T14:46:02.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 1 -- start start 2024-09-16T14:46:02.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 10 monclient: init 2024-09-16T14:46:02.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:02.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:02.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 10 monclient(hunting): picked mon.noname-b con 0x7f9e080a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:02.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 10 monclient(hunting): picked mon.noname-c con 0x7f9e080a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:02.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 10 monclient(hunting): picked mon.noname-a con 0x7f9e080a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:02.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e080a3250 con 0x7f9e080a2800 2024-09-16T14:46:02.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0809f600 con 0x7f9e080a0000 2024-09-16T14:46:02.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e080a3830 con 0x7f9e080a1d10 2024-09-16T14:46:02.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:02.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 10 monclient(hunting): authenticate will time out at 1186.549285s 2024-09-16T14:46:02.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0e2f4640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9e080a1d10 0x7f9e080a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43268/0 (socket says 172.21.15.26:43268) 2024-09-16T14:46:02.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0e2f4640 1 -- 172.21.15.26:0/1885904515 learned_addr learned my addr 172.21.15.26:0/1885904515 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:02.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 1 -- 172.21.15.26:0/1885904515 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365303826 0 0) 0x7f9e080a3830 con 0x7f9e080a1d10 2024-09-16T14:46:02.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient(hunting): my global_id is 5315 2024-09-16T14:46:02.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 1 -- 172.21.15.26:0/1885904515 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df8003530 con 0x7f9e080a1d10 2024-09-16T14:46:02.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 1 -- 172.21.15.26:0/1885904515 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009142915 0 0) 0x7f9e0809f600 con 0x7f9e080a0000 2024-09-16T14:46:02.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient(hunting): my global_id is 94432 2024-09-16T14:46:02.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 1 -- 172.21.15.26:0/1885904515 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e080a3830 con 0x7f9e080a0000 2024-09-16T14:46:02.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 1 -- 172.21.15.26:0/1885904515 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2080013168 0 0) 0x7f9df8003530 con 0x7f9e080a1d10 2024-09-16T14:46:02.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 1 -- 172.21.15.26:0/1885904515 >> v1:172.21.15.189:6789/0 conn(0x7f9e080a0000 legacy=0x7f9e08139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 1 -- 172.21.15.26:0/1885904515 >> v1:172.21.15.26:6789/0 conn(0x7f9e080a2800 legacy=0x7f9e0813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient: _finish_hunting 0 2024-09-16T14:46:02.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 1 monclient: found mon.noname-b 2024-09-16T14:46:02.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:02.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:46:02.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 1 -- 172.21.15.26:0/1885904515 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0809ed60 con 0x7f9e080a1d10 2024-09-16T14:46:02.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient: _finish_auth 0 2024-09-16T14:46:02.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient: _check_auth_tickets 2024-09-16T14:46:02.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:02.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 1 -- 172.21.15.26:0/1885904515 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9df4001620 con 0x7f9e080a1d10 2024-09-16T14:46:02.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 5 monclient: authenticate success, global_id 5315 2024-09-16T14:46:02.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:02.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:02.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient: dump: 2024-09-16T14:46:02.442 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.442 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.442 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.442 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.442 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.442 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.442 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.442 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.443 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.443 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:02.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 1 -- 172.21.15.26:0/1885904515 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9df4001880 con 0x7f9e080a1d10 2024-09-16T14:46:02.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:02.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 1 -- 172.21.15.26:0/1885904515 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9df4001c50 con 0x7f9e080a1d10 2024-09-16T14:46:02.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:02.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e0d2f2640 10 monclient: dump: 2024-09-16T14:46:02.443 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.443 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.443 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.443 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.444 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.444 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.444 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.444 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.444 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.444 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:02.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 4 set_mon_vals no callback set 2024-09-16T14:46:02.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 10 monclient: shutdown 2024-09-16T14:46:02.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:02.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 1 -- 172.21.15.26:0/1885904515 >> v1:172.21.15.26:6790/0 conn(0x7f9e080a1d10 legacy=0x7f9e080a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 1 -- 172.21.15.26:0/1885904515 shutdown_connections 2024-09-16T14:46:02.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 1 -- 172.21.15.26:0/1885904515 >> 172.21.15.26:0/1885904515 conn(0x7f9e080a7f30 msgr2=0x7f9e080a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:02.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 1 -- 172.21.15.26:0/1885904515 shutdown_connections 2024-09-16T14:46:02.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 1 -- 172.21.15.26:0/1885904515 wait complete. 2024-09-16T14:46:02.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 10 monclient: build_initial_monmap 2024-09-16T14:46:02.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.428+0000 7f9e10558640 10 monclient: monmap: 2024-09-16T14:46:02.445 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:02.445 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:02.445 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:46:02.445 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:46:02.445 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:02.446 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:46:02.446 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:02.446 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:02.446 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:02.446 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 1 Processor -- start 2024-09-16T14:46:02.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 1 -- start start 2024-09-16T14:46:02.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 10 monclient: init 2024-09-16T14:46:02.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:02.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:02.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 10 monclient(hunting): picked mon.noname-b con 0x7f9e080a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:02.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 10 monclient(hunting): picked mon.noname-c con 0x7f9e080a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:02.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 10 monclient(hunting): picked mon.noname-a con 0x7f9e080a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:02.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0813b6e0 con 0x7f9e080a2800 2024-09-16T14:46:02.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0813a530 con 0x7f9e080a0000 2024-09-16T14:46:02.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e08139e60 con 0x7f9e080a1d10 2024-09-16T14:46:02.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:02.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 10 monclient(hunting): authenticate will time out at 1186.552819s 2024-09-16T14:46:02.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e0daf3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9e080a1d10 0x7f9e08139750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43274/0 (socket says 172.21.15.26:43274) 2024-09-16T14:46:02.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e0eaf5640 1 --1- 172.21.15.26:0/754975886 >> v1:172.21.15.26:6789/0 conn(0x7f9e080a2800 0x7f9e081d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42946/0 (socket says 172.21.15.26:42946) 2024-09-16T14:46:02.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e0daf3640 1 -- 172.21.15.26:0/754975886 learned_addr learned my addr 172.21.15.26:0/754975886 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:02.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39923230 0 0) 0x7f9e08139e60 con 0x7f9e080a1d10 2024-09-16T14:46:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient(hunting): my global_id is 5318 2024-09-16T14:46:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dec003560 con 0x7f9e080a1d10 2024-09-16T14:46:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721746677 0 0) 0x7f9e0813b6e0 con 0x7f9e080a2800 2024-09-16T14:46:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient(hunting): my global_id is 104097 2024-09-16T14:46:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e08139e60 con 0x7f9e080a2800 2024-09-16T14:46:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1000071323 0 0) 0x7f9dec003560 con 0x7f9e080a1d10 2024-09-16T14:46:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e0813b6e0 con 0x7f9e080a1d10 2024-09-16T14:46:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2836431765 0 0) 0x7f9e08139e60 con 0x7f9e080a2800 2024-09-16T14:46:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9dec003560 con 0x7f9e080a2800 2024-09-16T14:46:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9dfc0018f0 con 0x7f9e080a1d10 2024-09-16T14:46:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient(hunting): dump: 2024-09-16T14:46:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.450 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e00001ab0 con 0x7f9e080a2800 2024-09-16T14:46:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient(hunting): dump: 2024-09-16T14:46:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.451 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2888523008 0 0) 0x7f9e0813b6e0 con 0x7f9e080a1d10 2024-09-16T14:46:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 >> v1:172.21.15.189:6789/0 conn(0x7f9e080a0000 legacy=0x7f9e08139040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 >> v1:172.21.15.26:6789/0 conn(0x7f9e080a2800 legacy=0x7f9e081d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 monclient: found mon.c 2024-09-16T14:46:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:46:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0809e500 con 0x7f9e080a1d10 2024-09-16T14:46:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient: _finish_auth 0 2024-09-16T14:46:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 5 monclient: authenticate success, global_id 5318 2024-09-16T14:46:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 10 monclient: _renew_subs 2024-09-16T14:46:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:46:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 1 -- 172.21.15.26:0/754975886 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e080a65c0 con 0x7f9e080a1d10 2024-09-16T14:46:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 10 monclient: _renew_subs 2024-09-16T14:46:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:46:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 1 -- 172.21.15.26:0/754975886 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0813c7a0 con 0x7f9e080a1d10 2024-09-16T14:46:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9dfc001c60 con 0x7f9e080a1d10 2024-09-16T14:46:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9dfc003a50 con 0x7f9e080a1d10 2024-09-16T14:46:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e0caf1640 4 set_mon_vals no callback set 2024-09-16T14:46:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 10 monclient: dump: 2024-09-16T14:46:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.454 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9dfc003c30 con 0x7f9e080a1d10 2024-09-16T14:46:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:46:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e10558640 1 -- 172.21.15.26:0/754975886 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df0005050 con 0x7f9e080a1d10 2024-09-16T14:46:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(75..75 src has 15..75) v4 ==== 5337+0+0 (unknown 2993449477 0 0) 0x7f9dfc043840 con 0x7f9e080a1d10 2024-09-16T14:46:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.432+0000 7f9e0e2f4640 10 monclient: get_auth_request con 0x7f9dec036300 auth_method 0 2024-09-16T14:46:02.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.436+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9dfc014700 con 0x7f9e080a1d10 2024-09-16T14:46:02.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.436+0000 7f9e06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:02.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.436+0000 7f9e06ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:02.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.436+0000 7f3c230d9640 10 monclient: get_monmap_and_config 2024-09-16T14:46:02.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.436+0000 7f3c230d9640 10 monclient: build_initial_monmap 2024-09-16T14:46:02.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.436+0000 7f3c230d9640 10 monclient: monmap: 2024-09-16T14:46:02.464 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:02.464 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.464 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:02.440582+0000 2024-09-16T14:46:02.464 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:02.440582+0000 2024-09-16T14:46:02.464 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:02.466 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 1 Processor -- start 2024-09-16T14:46:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 1 -- start start 2024-09-16T14:46:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient: init 2024-09-16T14:46:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient(hunting): picked mon.noname-c con 0x7f3c1c06c120 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient(hunting): picked mon.noname-a con 0x7f3c1c1353d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient(hunting): picked mon.noname-b con 0x7f3c1c0a1040 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c06c500 con 0x7f3c1c1353d0 2024-09-16T14:46:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c0a2ef0 con 0x7f3c1c06c120 2024-09-16T14:46:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c09e990 con 0x7f3c1c0a1040 2024-09-16T14:46:02.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient(hunting): authenticate will time out at 1144.556853s 2024-09-16T14:46:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c20e75640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3c1c06c120 0x7f3c1c134cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58194/0 (socket says 172.21.15.189:58194) 2024-09-16T14:46:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c20e75640 1 -- 172.21.15.189:0/279043480 learned_addr learned my addr 172.21.15.189:0/279043480 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 1 -- 172.21.15.189:0/279043480 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800237618 0 0) 0x7f3c1c0a2ef0 con 0x7f3c1c06c120 2024-09-16T14:46:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient(hunting): my global_id is 94435 2024-09-16T14:46:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 1 -- 172.21.15.189:0/279043480 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c0c003530 con 0x7f3c1c06c120 2024-09-16T14:46:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 1 -- 172.21.15.189:0/279043480 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579969193 0 0) 0x7f3c1c06c500 con 0x7f3c1c1353d0 2024-09-16T14:46:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient(hunting): my global_id is 104100 2024-09-16T14:46:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 1 -- 172.21.15.189:0/279043480 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c1c0a2ef0 con 0x7f3c1c1353d0 2024-09-16T14:46:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 1 -- 172.21.15.189:0/279043480 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175095373 0 0) 0x7f3c1c09e990 con 0x7f3c1c0a1040 2024-09-16T14:46:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient(hunting): my global_id is 5321 2024-09-16T14:46:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 1 -- 172.21.15.189:0/279043480 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c1c06c500 con 0x7f3c1c0a1040 2024-09-16T14:46:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 1 -- 172.21.15.189:0/279043480 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3467678216 0 0) 0x7f3c0c003530 con 0x7f3c1c06c120 2024-09-16T14:46:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 1 -- 172.21.15.189:0/279043480 >> v1:172.21.15.26:6790/0 conn(0x7f3c1c0a1040 legacy=0x7f3c1c13d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 1 -- 172.21.15.189:0/279043480 >> v1:172.21.15.26:6789/0 conn(0x7f3c1c1353d0 legacy=0x7f3c1c139b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 1 monclient: found mon.noname-c 2024-09-16T14:46:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 1 -- 172.21.15.189:0/279043480 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c09ec70 con 0x7f3c1c06c120 2024-09-16T14:46:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 1 -- 172.21.15.189:0/279043480 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3c10001620 con 0x7f3c1c06c120 2024-09-16T14:46:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient: dump: 2024-09-16T14:46:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.470 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 5 monclient: authenticate success, global_id 94435 2024-09-16T14:46:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 1 -- 172.21.15.189:0/279043480 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c10001890 con 0x7f3c1c06c120 2024-09-16T14:46:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 4 set_mon_vals no callback set 2024-09-16T14:46:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 1 -- 172.21.15.189:0/279043480 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3c10001c20 con 0x7f3c1c06c120 2024-09-16T14:46:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1b7fe640 10 monclient: dump: 2024-09-16T14:46:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.472 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.472 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.472 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.472 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:02.472 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.472 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.472 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.472 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:02.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient: shutdown 2024-09-16T14:46:02.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:02.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 1 -- 172.21.15.189:0/279043480 >> v1:172.21.15.189:6789/0 conn(0x7f3c1c06c120 legacy=0x7f3c1c134cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 1 -- 172.21.15.189:0/279043480 shutdown_connections 2024-09-16T14:46:02.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 1 -- 172.21.15.189:0/279043480 >> 172.21.15.189:0/279043480 conn(0x7f3c1c0a3a40 msgr2=0x7f3c1c0a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:02.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 1 -- 172.21.15.189:0/279043480 shutdown_connections 2024-09-16T14:46:02.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 1 -- 172.21.15.189:0/279043480 wait complete. 2024-09-16T14:46:02.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient: build_initial_monmap 2024-09-16T14:46:02.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient: monmap: 2024-09-16T14:46:02.473 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:02.473 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:02.473 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:02.473 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:02.473 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:02.474 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:02.474 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:02.474 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:02.474 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:02.474 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:02.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 1 Processor -- start 2024-09-16T14:46:02.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 1 -- start start 2024-09-16T14:46:02.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient: init 2024-09-16T14:46:02.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:02.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:02.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient(hunting): picked mon.noname-c con 0x7f3c1c06c120 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:02.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient(hunting): picked mon.noname-b con 0x7f3c1c0a1040 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:02.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient(hunting): picked mon.noname-a con 0x7f3c1c1353d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:02.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c09ce60 con 0x7f3c1c1353d0 2024-09-16T14:46:02.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c132a70 con 0x7f3c1c0a1040 2024-09-16T14:46:02.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c12fc20 con 0x7f3c1c06c120 2024-09-16T14:46:02.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:02.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c230d9640 10 monclient(hunting): authenticate will time out at 1144.559756s 2024-09-16T14:46:02.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3c1c0a1040 0x7f3c1c12f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58196/0 (socket says 172.21.15.189:58196) 2024-09-16T14:46:02.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.440+0000 7f3c1bfff640 1 -- 172.21.15.189:0/1694266085 learned_addr learned my addr 172.21.15.189:0/1694266085 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:02.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726684791 0 0) 0x7f3c1c132a70 con 0x7f3c1c0a1040 2024-09-16T14:46:02.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): my global_id is 94438 2024-09-16T14:46:02.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c00003530 con 0x7f3c1c0a1040 2024-09-16T14:46:02.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758692464 0 0) 0x7f3c1c09ce60 con 0x7f3c1c1353d0 2024-09-16T14:46:02.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): my global_id is 104103 2024-09-16T14:46:02.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c1c132a70 con 0x7f3c1c1353d0 2024-09-16T14:46:02.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377212526 0 0) 0x7f3c1c12fc20 con 0x7f3c1c06c120 2024-09-16T14:46:02.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): my global_id is 5324 2024-09-16T14:46:02.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c1c09ce60 con 0x7f3c1c06c120 2024-09-16T14:46:02.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2802673433 0 0) 0x7f3c00003530 con 0x7f3c1c0a1040 2024-09-16T14:46:02.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c1c12fc20 con 0x7f3c1c0a1040 2024-09-16T14:46:02.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3c080018c0 con 0x7f3c1c0a1040 2024-09-16T14:46:02.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:02.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): dump: 2024-09-16T14:46:02.477 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:02.477 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.477 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.478 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.478 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.478 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:02.478 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.478 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.478 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.478 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:02.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:02.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 158688066 0 0) 0x7f3c1c132a70 con 0x7f3c1c1353d0 2024-09-16T14:46:02.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c00003530 con 0x7f3c1c1353d0 2024-09-16T14:46:02.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3c14001c40 con 0x7f3c1c1353d0 2024-09-16T14:46:02.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:02.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): dump: 2024-09-16T14:46:02.479 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:02.479 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.479 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.479 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.479 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.479 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:02.479 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.479 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.479 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.479 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3412316576 0 0) 0x7f3c1c09ce60 con 0x7f3c1c06c120 2024-09-16T14:46:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c1c132a70 con 0x7f3c1c06c120 2024-09-16T14:46:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3c100015e0 con 0x7f3c1c06c120 2024-09-16T14:46:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient(hunting): dump: 2024-09-16T14:46:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.481 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1552656767 0 0) 0x7f3c1c12fc20 con 0x7f3c1c0a1040 2024-09-16T14:46:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 >> v1:172.21.15.26:6790/0 conn(0x7f3c1c06c120 legacy=0x7f3c1c12ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 >> v1:172.21.15.26:6789/0 conn(0x7f3c1c1353d0 legacy=0x7f3c1c1d1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient: _finish_hunting 0 2024-09-16T14:46:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 monclient: found mon.b 2024-09-16T14:46:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c0a5670 con 0x7f3c1c0a1040 2024-09-16T14:46:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient: _finish_auth 0 2024-09-16T14:46:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient: _check_auth_tickets 2024-09-16T14:46:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c230d9640 5 monclient: authenticate success, global_id 94438 2024-09-16T14:46:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c230d9640 10 monclient: _renew_subs 2024-09-16T14:46:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c230d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c230d9640 1 -- 172.21.15.189:0/1694266085 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c1c09e130 con 0x7f3c1c0a1040 2024-09-16T14:46:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c230d9640 10 monclient: _renew_subs 2024-09-16T14:46:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c230d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c230d9640 1 -- 172.21.15.189:0/1694266085 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1c131140 con 0x7f3c1c0a1040 2024-09-16T14:46:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c08001cb0 con 0x7f3c1c0a1040 2024-09-16T14:46:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3c08003a70 con 0x7f3c1c0a1040 2024-09-16T14:46:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 10 monclient: dump: 2024-09-16T14:46:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.483 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c1affd640 4 set_mon_vals no callback set 2024-09-16T14:46:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3c08003bd0 con 0x7f3c1c0a1040 2024-09-16T14:46:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(75..75 src has 15..75) v4 ==== 5337+0+0 (unknown 2993449477 0 0) 0x7f3c080437d0 con 0x7f3c1c0a1040 2024-09-16T14:46:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c230d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c230d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c230d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c230d9640 1 -- 172.21.15.189:0/1694266085 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bfc005050 con 0x7f3c1c0a1040 2024-09-16T14:46:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.444+0000 7f3c20e75640 10 monclient: get_auth_request con 0x7f3c00036470 auth_method 0 2024-09-16T14:46:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.448+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3c08014690 con 0x7f3c1c0a1040 2024-09-16T14:46:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.448+0000 7f3c197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.448+0000 7f3c197fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:02.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.600+0000 7f3c230d9640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T14:46:02.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.600+0000 7f3c230d9640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T14:46:02.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.600+0000 7f3c230d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:02.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:02.600+0000 7f3c230d9640 1 -- 172.21.15.189:0/1694266085 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"} v 0) v1 -- 0x7f3bfc005880 con 0x7f3c1c0a1040 2024-09-16T14:46:02.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.656+0000 7f9e10558640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:46:02.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.656+0000 7f9e10558640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:46:02.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.656+0000 7f9e10558640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:46:02.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.656+0000 7f9e10558640 1 -- 172.21.15.26:0/754975886 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9df0005880 con 0x7f9e080a1d10 2024-09-16T14:46:02.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.656+0000 7f9e06ffd640 1 -- 172.21.15.26:0/754975886 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1276175008) 0x7f9dfc012b70 con 0x7f9e080a1d10 2024-09-16T14:46:02.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.656+0000 7f9e06ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:46:02.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.656+0000 7f9e06ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:02.661 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:46:02.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.660+0000 7f9e10558640 1 -- 172.21.15.26:0/754975886 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9dec036300 legacy=0x7f9dec0387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.660+0000 7f9e10558640 10 monclient: shutdown 2024-09-16T14:46:02.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.660+0000 7f9e10558640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:02.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.660+0000 7f9e10558640 1 -- 172.21.15.26:0/754975886 >> v1:172.21.15.26:6790/0 conn(0x7f9e080a1d10 legacy=0x7f9e08139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.660+0000 7f9e10558640 1 -- 172.21.15.26:0/754975886 shutdown_connections 2024-09-16T14:46:02.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.660+0000 7f9e10558640 1 -- 172.21.15.26:0/754975886 >> 172.21.15.26:0/754975886 conn(0x7f9e080a7f30 msgr2=0x7f9e0813e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:02.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.660+0000 7f9e10558640 1 -- 172.21.15.26:0/754975886 shutdown_connections 2024-09-16T14:46:02.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.660+0000 7f9e10558640 1 -- 172.21.15.26:0/754975886 wait complete. 2024-09-16T14:46:02.678 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:46:02.678 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:46:02.678 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:46:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 10 monclient: get_monmap_and_config 2024-09-16T14:46:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 10 monclient: build_initial_monmap 2024-09-16T14:46:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 10 monclient: monmap: 2024-09-16T14:46:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:46:02.798674+0000 2024-09-16T14:46:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:46:02.798674+0000 2024-09-16T14:46:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:02.799 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 1 Processor -- start 2024-09-16T14:46:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 1 -- start start 2024-09-16T14:46:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 10 monclient: init 2024-09-16T14:46:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 10 monclient(hunting): picked mon.noname-b con 0x7f84a809a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 10 monclient(hunting): picked mon.noname-a con 0x7f84a809d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 10 monclient(hunting): picked mon.noname-c con 0x7f84a809b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a809dc60 con 0x7f84a809d1b0 2024-09-16T14:46:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a8099ae0 con 0x7f84a809b040 2024-09-16T14:46:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a809b8f0 con 0x7f84a809a550 2024-09-16T14:46:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 10 monclient(hunting): authenticate will time out at 1186.918886s 2024-09-16T14:46:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a77fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f84a809a550 0x7f84a809a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43276/0 (socket says 172.21.15.26:43276) 2024-09-16T14:46:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a77fe640 1 -- 172.21.15.26:0/4290029910 learned_addr learned my addr 172.21.15.26:0/4290029910 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a6ffd640 1 --1- 172.21.15.26:0/4290029910 >> v1:172.21.15.26:6789/0 conn(0x7f84a809d1b0 0x7f84a8136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42962/0 (socket says 172.21.15.26:42962) 2024-09-16T14:46:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 1 -- 172.21.15.26:0/4290029910 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857654892 0 0) 0x7f84a809b8f0 con 0x7f84a809a550 2024-09-16T14:46:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient(hunting): my global_id is 5327 2024-09-16T14:46:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 1 -- 172.21.15.26:0/4290029910 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8494003530 con 0x7f84a809a550 2024-09-16T14:46:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 1 -- 172.21.15.26:0/4290029910 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008497714 0 0) 0x7f84a809dc60 con 0x7f84a809d1b0 2024-09-16T14:46:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient(hunting): my global_id is 104106 2024-09-16T14:46:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 1 -- 172.21.15.26:0/4290029910 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a809b8f0 con 0x7f84a809d1b0 2024-09-16T14:46:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 1 -- 172.21.15.26:0/4290029910 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3061817045 0 0) 0x7f8494003530 con 0x7f84a809a550 2024-09-16T14:46:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 1 -- 172.21.15.26:0/4290029910 >> v1:172.21.15.189:6789/0 conn(0x7f84a809b040 legacy=0x7f84a813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 1 -- 172.21.15.26:0/4290029910 >> v1:172.21.15.26:6789/0 conn(0x7f84a809d1b0 legacy=0x7f84a8136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:46:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 1 monclient: found mon.noname-b 2024-09-16T14:46:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:46:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 1 -- 172.21.15.26:0/4290029910 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a809bbd0 con 0x7f84a809a550 2024-09-16T14:46:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient: _finish_auth 0 2024-09-16T14:46:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:46:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 1 -- 172.21.15.26:0/4290029910 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f84980019a0 con 0x7f84a809a550 2024-09-16T14:46:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient: dump: 2024-09-16T14:46:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.807 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 5 monclient: authenticate success, global_id 5327 2024-09-16T14:46:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 1 -- 172.21.15.26:0/4290029910 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8498001c40 con 0x7f84a809a550 2024-09-16T14:46:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 1 -- 172.21.15.26:0/4290029910 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8498002010 con 0x7f84a809a550 2024-09-16T14:46:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84adf6d640 4 set_mon_vals no callback set 2024-09-16T14:46:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.796+0000 7f84a67fc640 10 monclient: dump: 2024-09-16T14:46:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.808 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 10 monclient: shutdown 2024-09-16T14:46:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 1 -- 172.21.15.26:0/4290029910 >> v1:172.21.15.26:6790/0 conn(0x7f84a809a550 legacy=0x7f84a809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 1 -- 172.21.15.26:0/4290029910 shutdown_connections 2024-09-16T14:46:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 1 -- 172.21.15.26:0/4290029910 >> 172.21.15.26:0/4290029910 conn(0x7f84a809fb50 msgr2=0x7f84a809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 1 -- 172.21.15.26:0/4290029910 shutdown_connections 2024-09-16T14:46:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 1 -- 172.21.15.26:0/4290029910 wait complete. 2024-09-16T14:46:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 10 monclient: build_initial_monmap 2024-09-16T14:46:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 10 monclient: monmap: 2024-09-16T14:46:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:46:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:46:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:46:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:02.810 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 1 Processor -- start 2024-09-16T14:46:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 1 -- start start 2024-09-16T14:46:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 10 monclient: init 2024-09-16T14:46:02.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:02.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:02.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 10 monclient(hunting): picked mon.noname-a con 0x7f84a809a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:02.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 10 monclient(hunting): picked mon.noname-c con 0x7f84a809b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:02.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 10 monclient(hunting): picked mon.noname-b con 0x7f84a809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:02.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a811dbb0 con 0x7f84a809a550 2024-09-16T14:46:02.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a811e760 con 0x7f84a809d1b0 2024-09-16T14:46:02.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a8071020 con 0x7f84a809b040 2024-09-16T14:46:02.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:02.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 10 monclient(hunting): authenticate will time out at 1186.922023s 2024-09-16T14:46:02.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84a6ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f84a809b040 0x7f84a806fcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43284/0 (socket says 172.21.15.26:43284) 2024-09-16T14:46:02.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84a6ffd640 1 -- 172.21.15.26:0/720060173 learned_addr learned my addr 172.21.15.26:0/720060173 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:02.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84a77fe640 1 --1- 172.21.15.26:0/720060173 >> v1:172.21.15.26:6789/0 conn(0x7f84a809a550 0x7f84a806f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42972/0 (socket says 172.21.15.26:42972) 2024-09-16T14:46:02.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591591372 0 0) 0x7f84a8071020 con 0x7f84a809b040 2024-09-16T14:46:02.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): my global_id is 5330 2024-09-16T14:46:02.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8488003530 con 0x7f84a809b040 2024-09-16T14:46:02.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465208753 0 0) 0x7f84a811dbb0 con 0x7f84a809a550 2024-09-16T14:46:02.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): my global_id is 104109 2024-09-16T14:46:02.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a8071020 con 0x7f84a809a550 2024-09-16T14:46:02.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3756745393 0 0) 0x7f84a8071020 con 0x7f84a809a550 2024-09-16T14:46:02.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84a811dbb0 con 0x7f84a809a550 2024-09-16T14:46:02.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8498001db0 con 0x7f84a809a550 2024-09-16T14:46:02.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:46:02.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): dump: 2024-09-16T14:46:02.819 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.819 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.819 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.819 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.819 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.819 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.819 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.819 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.819 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.819 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:02.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2220981369 0 0) 0x7f8488003530 con 0x7f84a809b040 2024-09-16T14:46:02.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84a8071020 con 0x7f84a809b040 2024-09-16T14:46:02.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8490001860 con 0x7f84a809b040 2024-09-16T14:46:02.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:02.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): dump: 2024-09-16T14:46:02.820 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.820 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.820 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.820 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.820 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.820 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.820 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.820 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.821 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.821 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442556933 0 0) 0x7f84a811e760 con 0x7f84a809d1b0 2024-09-16T14:46:02.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient(hunting): my global_id is 94444 2024-09-16T14:46:02.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8488003530 con 0x7f84a809d1b0 2024-09-16T14:46:02.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2518786900 0 0) 0x7f84a811dbb0 con 0x7f84a809a550 2024-09-16T14:46:02.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 >> v1:172.21.15.26:6790/0 conn(0x7f84a809b040 legacy=0x7f84a806fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 >> v1:172.21.15.189:6789/0 conn(0x7f84a809d1b0 legacy=0x7f84a81d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:02.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 monclient: found mon.a 2024-09-16T14:46:02.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:02.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:02.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a811ed20 con 0x7f84a809a550 2024-09-16T14:46:02.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient: _finish_auth 0 2024-09-16T14:46:02.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:02.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:02.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 5 monclient: authenticate success, global_id 104109 2024-09-16T14:46:02.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 10 monclient: _renew_subs 2024-09-16T14:46:02.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:02.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 1 -- 172.21.15.26:0/720060173 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a811f060 con 0x7f84a809a550 2024-09-16T14:46:02.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 10 monclient: _renew_subs 2024-09-16T14:46:02.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:02.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 1 -- 172.21.15.26:0/720060173 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a8071970 con 0x7f84a809a550 2024-09-16T14:46:02.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84980015c0 con 0x7f84a809a550 2024-09-16T14:46:02.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:02.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84a5ffb640 4 set_mon_vals no callback set 2024-09-16T14:46:02.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8498004f80 con 0x7f84a809a550 2024-09-16T14:46:02.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:02.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 10 monclient: dump: 2024-09-16T14:46:02.823 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.823 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.823 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.823 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.823 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.823 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.823 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.824 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.824 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.824 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:02.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f84adf6d640 1 -- 172.21.15.26:0/720060173 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f848c006590 con 0x7f848c002830 2024-09-16T14:46:02.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.800+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f849800f9f0 con 0x7f84a809a550 2024-09-16T14:46:02.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(75..75 src has 15..75) v4 ==== 5337+0+0 (unknown 2993449477 0 0) 0x7f8498043b30 con 0x7f84a809a550 2024-09-16T14:46:02.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199295038 0 0) 0x7f848c006590 con 0x7f848c002830 2024-09-16T14:46:02.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 10 monclient(hunting): my global_id is 104112 2024-09-16T14:46:02.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a811dbb0 con 0x7f848c002830 2024-09-16T14:46:02.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f84a7fff640 10 monclient: get_auth_request con 0x7f8488036880 auth_method 0 2024-09-16T14:46:02.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2160814621 0 0) 0x7f84a811dbb0 con 0x7f848c002830 2024-09-16T14:46:02.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f848c006590 con 0x7f848c002830 2024-09-16T14:46:02.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8490004990 con 0x7f848c002830 2024-09-16T14:46:02.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:02.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 10 monclient: dump: 2024-09-16T14:46:02.825 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.825 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.825 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.825 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.825 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.825 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.826 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 510683488 0 0) 0x7f848c006590 con 0x7f848c002830 2024-09-16T14:46:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f848c00a0e0 con 0x7f848c002830 2024-09-16T14:46:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8490007480 con 0x7f848c002830 2024-09-16T14:46:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 >> v1:172.21.15.26:6789/0 conn(0x7f848c002830 legacy=0x7f848c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.804+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 >> v1:172.21.15.26:6789/0 conn(0x7f848c002830 legacy=0x7f848c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:46:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.808+0000 7f84adf6d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:46:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.808+0000 7f84adf6d640 1 -- 172.21.15.26:0/720060173 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f848c009da0 con 0x7f848c0090c0 2024-09-16T14:46:02.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.808+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784459937 0 0) 0x7f848c009da0 con 0x7f848c0090c0 2024-09-16T14:46:02.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.808+0000 7f849ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.808+0000 7f849ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.808+0000 7f849ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.808+0000 7f849ffff640 10 monclient(hunting): my global_id is 104115 2024-09-16T14:46:02.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.808+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f848c006590 con 0x7f848c0090c0 2024-09-16T14:46:02.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.808+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1547620626 0 0) 0x7f848c006590 con 0x7f848c0090c0 2024-09-16T14:46:02.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.808+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f848c009da0 con 0x7f848c0090c0 2024-09-16T14:46:02.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.808+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8490005490 con 0x7f848c0090c0 2024-09-16T14:46:02.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.808+0000 7f849ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.808+0000 7f849ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:02.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.808+0000 7f849ffff640 10 monclient: dump: 2024-09-16T14:46:02.827 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.827 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.828 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.828 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.828 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.828 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.828 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.828 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.828 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.828 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.808+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2999232111 0 0) 0x7f848c009da0 con 0x7f848c0090c0 2024-09-16T14:46:02.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.808+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f848c00c290 con 0x7f848c0090c0 2024-09-16T14:46:02.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.812+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 2352556306) 0x7f84900075f0 con 0x7f848c0090c0 2024-09-16T14:46:02.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.812+0000 7f849ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:46:02.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.812+0000 7f849ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:02.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.812+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 >> v1:172.21.15.26:6789/0 conn(0x7f848c0090c0 legacy=0x7f848c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.812+0000 7f849ffff640 1 -- 172.21.15.26:0/720060173 >> v1:172.21.15.26:6789/0 conn(0x7f848c0090c0 legacy=0x7f848c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:46:02.829 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:46:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:46:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:46:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:46:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 38, 2024-09-16T14:46:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:46:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:46:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:46:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:46:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:46:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:46:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:46:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:46:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:46:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:46:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:46:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:46:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:46:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:46:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:46:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:46:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:46:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:46:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:46:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:46:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:46:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:46:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:46:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:46:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:46:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:46:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:46:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:46:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:46:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:46:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:46:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:46:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:46:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:46:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:46:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:46:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:46:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:46:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:46:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:46:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:46:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:46:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:46:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:46:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:46:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:46:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:46:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:46:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:46:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:46:02.840 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:46:02.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.812+0000 7f84adf6d640 1 -- 172.21.15.26:0/720060173 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8488036880 legacy=0x7f8488038d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.812+0000 7f84adf6d640 10 monclient: shutdown 2024-09-16T14:46:02.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.812+0000 7f84adf6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:02.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.812+0000 7f84adf6d640 1 -- 172.21.15.26:0/720060173 >> v1:172.21.15.26:6789/0 conn(0x7f84a809a550 legacy=0x7f84a806f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.812+0000 7f84a7fff640 1 -- 172.21.15.26:0/720060173 reap_dead start 2024-09-16T14:46:02.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.812+0000 7f84a7fff640 1 -- 172.21.15.26:0/720060173 reap_dead start 2024-09-16T14:46:02.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.812+0000 7f84adf6d640 1 -- 172.21.15.26:0/720060173 shutdown_connections 2024-09-16T14:46:02.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.812+0000 7f84adf6d640 1 -- 172.21.15.26:0/720060173 >> 172.21.15.26:0/720060173 conn(0x7f84a809fb50 msgr2=0x7f84a813a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:02.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.812+0000 7f84adf6d640 1 -- 172.21.15.26:0/720060173 shutdown_connections 2024-09-16T14:46:02.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.812+0000 7f84adf6d640 1 -- 172.21.15.26:0/720060173 wait complete. 2024-09-16T14:46:02.843 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:46:02.843 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:46:02.843 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:46:02.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 10 monclient: get_monmap_and_config 2024-09-16T14:46:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 10 monclient: build_initial_monmap 2024-09-16T14:46:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 10 monclient: monmap: 2024-09-16T14:46:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:46:02.953586+0000 2024-09-16T14:46:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:46:02.953586+0000 2024-09-16T14:46:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.956 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:02.956 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:02.956 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:02.956 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 1 Processor -- start 2024-09-16T14:46:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 1 -- start start 2024-09-16T14:46:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 10 monclient: init 2024-09-16T14:46:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 10 monclient(hunting): picked mon.noname-c con 0x7f99e80a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 10 monclient(hunting): picked mon.noname-b con 0x7f99e80a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 10 monclient(hunting): picked mon.noname-a con 0x7f99e80a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e80a3250 con 0x7f99e80a2800 2024-09-16T14:46:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e809f600 con 0x7f99e80a1d10 2024-09-16T14:46:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e80a3830 con 0x7f99e80a0000 2024-09-16T14:46:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 10 monclient(hunting): authenticate will time out at 1187.073259s 2024-09-16T14:46:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ec9ab640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f99e80a0000 0x7f99e8139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43286/0 (socket says 172.21.15.26:43286) 2024-09-16T14:46:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ec9ab640 1 -- 172.21.15.26:0/2029746390 learned_addr learned my addr 172.21.15.26:0/2029746390 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 1 -- 172.21.15.26:0/2029746390 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384327744 0 0) 0x7f99e80a3830 con 0x7f99e80a0000 2024-09-16T14:46:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient(hunting): my global_id is 5333 2024-09-16T14:46:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 1 -- 172.21.15.26:0/2029746390 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d4003530 con 0x7f99e80a0000 2024-09-16T14:46:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 1 -- 172.21.15.26:0/2029746390 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715610748 0 0) 0x7f99e80a3250 con 0x7f99e80a2800 2024-09-16T14:46:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient(hunting): my global_id is 104118 2024-09-16T14:46:02.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 1 -- 172.21.15.26:0/2029746390 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99e80a3830 con 0x7f99e80a2800 2024-09-16T14:46:02.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 1 -- 172.21.15.26:0/2029746390 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3197786275 0 0) 0x7f99e80a3830 con 0x7f99e80a2800 2024-09-16T14:46:02.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 1 -- 172.21.15.26:0/2029746390 >> v1:172.21.15.26:6790/0 conn(0x7f99e80a0000 legacy=0x7f99e8139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 1 -- 172.21.15.26:0/2029746390 >> v1:172.21.15.189:6789/0 conn(0x7f99e80a1d10 legacy=0x7f99e80a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:02.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 1 monclient: found mon.noname-a 2024-09-16T14:46:02.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:02.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:46:02.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 1 -- 172.21.15.26:0/2029746390 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e809ed60 con 0x7f99e80a2800 2024-09-16T14:46:02.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient: _finish_auth 0 2024-09-16T14:46:02.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:02.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 1 -- 172.21.15.26:0/2029746390 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f99e0001980 con 0x7f99e80a2800 2024-09-16T14:46:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:46:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient: dump: 2024-09-16T14:46:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.980 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 5 monclient: authenticate success, global_id 104118 2024-09-16T14:46:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 1 -- 172.21.15.26:0/2029746390 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99e0001bb0 con 0x7f99e80a2800 2024-09-16T14:46:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 1 -- 172.21.15.26:0/2029746390 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f99e0001f80 con 0x7f99e80a2800 2024-09-16T14:46:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 4 set_mon_vals no callback set 2024-09-16T14:46:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99dffff640 10 monclient: dump: 2024-09-16T14:46:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.982 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.982 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.982 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.982 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.982 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.983 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.983 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.983 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.983 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 10 monclient: shutdown 2024-09-16T14:46:02.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:02.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 1 -- 172.21.15.26:0/2029746390 >> v1:172.21.15.26:6789/0 conn(0x7f99e80a2800 legacy=0x7f99e813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 1 -- 172.21.15.26:0/2029746390 shutdown_connections 2024-09-16T14:46:02.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 1 -- 172.21.15.26:0/2029746390 >> 172.21.15.26:0/2029746390 conn(0x7f99e80a7f30 msgr2=0x7f99e80a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:02.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 1 -- 172.21.15.26:0/2029746390 shutdown_connections 2024-09-16T14:46:02.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 1 -- 172.21.15.26:0/2029746390 wait complete. 2024-09-16T14:46:02.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 10 monclient: build_initial_monmap 2024-09-16T14:46:02.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 10 monclient: monmap: 2024-09-16T14:46:02.984 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:02.985 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:02.985 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:46:02.985 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:46:02.985 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:02.985 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:46:02.985 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:02.986 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:02.986 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:02.986 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 1 Processor -- start 2024-09-16T14:46:02.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 1 -- start start 2024-09-16T14:46:02.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.952+0000 7f99ef410640 10 monclient: init 2024-09-16T14:46:02.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:02.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:02.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 10 monclient(hunting): picked mon.noname-c con 0x7f99e80a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:02.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 10 monclient(hunting): picked mon.noname-b con 0x7f99e80a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:02.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 10 monclient(hunting): picked mon.noname-a con 0x7f99e80a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:02.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e80a67f0 con 0x7f99e80a2800 2024-09-16T14:46:02.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e81350e0 con 0x7f99e80a1d10 2024-09-16T14:46:02.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e80a62a0 con 0x7f99e80a0000 2024-09-16T14:46:02.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 10 monclient(hunting): authenticate will time out at 1187.076367s 2024-09-16T14:46:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ed1ac640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f99e80a0000 0x7f99e813d4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43302/0 (socket says 172.21.15.26:43302) 2024-09-16T14:46:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ed1ac640 1 -- 172.21.15.26:0/1505210630 learned_addr learned my addr 172.21.15.26:0/1505210630 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846242284 0 0) 0x7f99e80a62a0 con 0x7f99e80a0000 2024-09-16T14:46:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): my global_id is 5336 2024-09-16T14:46:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99cc003530 con 0x7f99e80a0000 2024-09-16T14:46:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028993094 0 0) 0x7f99e80a67f0 con 0x7f99e80a2800 2024-09-16T14:46:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): my global_id is 104121 2024-09-16T14:46:02.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99e80a62a0 con 0x7f99e80a2800 2024-09-16T14:46:02.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 953136810 0 0) 0x7f99cc003530 con 0x7f99e80a0000 2024-09-16T14:46:02.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99e80a67f0 con 0x7f99e80a0000 2024-09-16T14:46:02.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f99e4001920 con 0x7f99e80a0000 2024-09-16T14:46:02.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:02.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): dump: 2024-09-16T14:46:02.993 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.993 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.994 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.994 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.994 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.994 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.994 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.994 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.994 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.994 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:02.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1562599006 0 0) 0x7f99e80a62a0 con 0x7f99e80a2800 2024-09-16T14:46:02.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99cc003530 con 0x7f99e80a2800 2024-09-16T14:46:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f99e0001ec0 con 0x7f99e80a2800 2024-09-16T14:46:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): dump: 2024-09-16T14:46:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:02.996 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653788236 0 0) 0x7f99e81350e0 con 0x7f99e80a1d10 2024-09-16T14:46:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): my global_id is 94450 2024-09-16T14:46:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99e80a62a0 con 0x7f99e80a1d10 2024-09-16T14:46:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4235490640 0 0) 0x7f99e80a67f0 con 0x7f99e80a0000 2024-09-16T14:46:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 >> v1:172.21.15.189:6789/0 conn(0x7f99e80a1d10 legacy=0x7f99e81ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 >> v1:172.21.15.26:6789/0 conn(0x7f99e80a2800 legacy=0x7f99e81d1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient: _finish_hunting 0 2024-09-16T14:46:02.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 monclient: found mon.c 2024-09-16T14:46:02.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:02.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:46:02.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e809e500 con 0x7f99e80a0000 2024-09-16T14:46:02.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient: _finish_auth 0 2024-09-16T14:46:02.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:02.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:02.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 5 monclient: authenticate success, global_id 5336 2024-09-16T14:46:02.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 10 monclient: _renew_subs 2024-09-16T14:46:02.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:46:02.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 1 -- 172.21.15.26:0/1505210630 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e8135590 con 0x7f99e80a0000 2024-09-16T14:46:02.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 10 monclient: _renew_subs 2024-09-16T14:46:02.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:46:02.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 1 -- 172.21.15.26:0/1505210630 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e81357d0 con 0x7f99e80a0000 2024-09-16T14:46:02.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99e4003660 con 0x7f99e80a0000 2024-09-16T14:46:02.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:02.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f99e40039f0 con 0x7f99e80a0000 2024-09-16T14:46:02.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:02.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:02.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient: dump: 2024-09-16T14:46:02.999 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:02.999 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:02.999 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.999 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:02.999 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.000 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.000 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.000 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.000 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.000 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99df7fe640 4 set_mon_vals no callback set 2024-09-16T14:46:03.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:03.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ef410640 1 -- 172.21.15.26:0/1505210630 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f99c8006590 con 0x7f99c8002830 2024-09-16T14:46:03.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f99e4003b50 con 0x7f99e80a0000 2024-09-16T14:46:03.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(75..75 src has 15..75) v4 ==== 5337+0+0 (unknown 2993449477 0 0) 0x7f99e4043780 con 0x7f99e80a0000 2024-09-16T14:46:03.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2184853731 0 0) 0x7f99c8006590 con 0x7f99c8002830 2024-09-16T14:46:03.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient(hunting): my global_id is 5339 2024-09-16T14:46:03.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99e80a67f0 con 0x7f99c8002830 2024-09-16T14:46:03.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4146620023 0 0) 0x7f99e80a67f0 con 0x7f99c8002830 2024-09-16T14:46:03.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ed9ad640 10 monclient: get_auth_request con 0x7f99cc036360 auth_method 0 2024-09-16T14:46:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99c8006590 con 0x7f99c8002830 2024-09-16T14:46:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f99d8001670 con 0x7f99c8002830 2024-09-16T14:46:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient: dump: 2024-09-16T14:46:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.003 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1215617839 0 0) 0x7f99c8006590 con 0x7f99c8002830 2024-09-16T14:46:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99c800a0e0 con 0x7f99c8002830 2024-09-16T14:46:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f99d8001d10 con 0x7f99c8002830 2024-09-16T14:46:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 >> v1:172.21.15.26:6790/0 conn(0x7f99c8002830 legacy=0x7f99c8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.956+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 >> v1:172.21.15.26:6790/0 conn(0x7f99c8002830 legacy=0x7f99c8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:46:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ef410640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:46:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ef410640 1 -- 172.21.15.26:0/1505210630 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f99c8009da0 con 0x7f99c80090c0 2024-09-16T14:46:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564811022 0 0) 0x7f99c8009da0 con 0x7f99c80090c0 2024-09-16T14:46:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 10 monclient(hunting): my global_id is 5342 2024-09-16T14:46:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99c8006590 con 0x7f99c80090c0 2024-09-16T14:46:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3736291441 0 0) 0x7f99c8006590 con 0x7f99c80090c0 2024-09-16T14:46:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99c8009da0 con 0x7f99c80090c0 2024-09-16T14:46:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f99d8004b80 con 0x7f99c80090c0 2024-09-16T14:46:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 10 monclient: dump: 2024-09-16T14:46:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.005 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4024374411 0 0) 0x7f99c8009da0 con 0x7f99c80090c0 2024-09-16T14:46:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f99c800c290 con 0x7f99c80090c0 2024-09-16T14:46:03.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 826153043) 0x7f99d8004df0 con 0x7f99c80090c0 2024-09-16T14:46:03.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:46:03.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:03.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 >> v1:172.21.15.26:6790/0 conn(0x7f99c80090c0 legacy=0x7f99c8009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ddffb640 1 -- 172.21.15.26:0/1505210630 >> v1:172.21.15.26:6790/0 conn(0x7f99c80090c0 legacy=0x7f99c8009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:46:03.006 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:46:03.006 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:46:03.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:46:03.007 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:46:03.007 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 38, 2024-09-16T14:46:03.007 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:46:03.007 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:46:03.007 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:46:03.007 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:46:03.007 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:03.007 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:46:03.007 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:46:03.007 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:46:03.007 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:46:03.007 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:03.007 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:03.008 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:03.008 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:03.008 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:03.008 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:03.008 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:03.008 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:03.008 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:03.008 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:03.008 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:03.008 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:46:03.008 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:46:03.008 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:03.008 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:03.009 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:03.009 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:03.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:03.009 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:03.009 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:03.009 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:03.009 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:03.009 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:03.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:03.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.009 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:46:03.009 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:46:03.009 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:46:03.010 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:46:03.010 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:46:03.010 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:46:03.010 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:46:03.010 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:46:03.010 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:46:03.010 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:46:03.010 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:46:03.010 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:46:03.010 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:46:03.010 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:46:03.010 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:46:03.010 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:46:03.010 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:46:03.011 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:03.011 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:03.011 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:03.011 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:03.011 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:03.011 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:03.011 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:03.011 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:03.011 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:03.011 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:03.011 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:03.011 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:46:03.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.011 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:46:03.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:46:03.012 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:46:03.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:03.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:03.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.012 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:03.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:46:03.012 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:03.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:03.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:03.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:46:03.013 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:46:03.013 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:03.013 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:03.013 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:03.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:46:03.013 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:46:03.013 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:03.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:03.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.013 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:03.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:46:03.014 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:03.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:03.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:03.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:46:03.014 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:46:03.014 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:03.014 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:03.014 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:03.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:46:03.014 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:46:03.015 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:03.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:03.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.015 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:03.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:46:03.015 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:03.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:03.015 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:03.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:46:03.015 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:46:03.015 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:03.015 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:03.016 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:03.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:03.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:03.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.016 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:46:03.016 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:46:03.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.016 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:03.016 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:03.016 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:46:03.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:03.016 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:03.017 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:46:03.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.017 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:03.017 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:03.017 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:46:03.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:03.017 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:03.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:46:03.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.017 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:03.017 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:03.017 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:46:03.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:46:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:46:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:46:03.018 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:46:03.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ef410640 1 -- 172.21.15.26:0/1505210630 >> v1:172.21.15.189:6812/3196161582 conn(0x7f99cc036360 legacy=0x7f99cc038820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ef410640 10 monclient: shutdown 2024-09-16T14:46:03.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ef410640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:03.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ef410640 1 -- 172.21.15.26:0/1505210630 >> v1:172.21.15.26:6790/0 conn(0x7f99e80a0000 legacy=0x7f99e813d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ed9ad640 1 -- 172.21.15.26:0/1505210630 reap_dead start 2024-09-16T14:46:03.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ed9ad640 1 -- 172.21.15.26:0/1505210630 reap_dead start 2024-09-16T14:46:03.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ef410640 1 -- 172.21.15.26:0/1505210630 shutdown_connections 2024-09-16T14:46:03.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ef410640 1 -- 172.21.15.26:0/1505210630 >> 172.21.15.26:0/1505210630 conn(0x7f99e80a7f30 msgr2=0x7f99e80a52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:03.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ef410640 1 -- 172.21.15.26:0/1505210630 shutdown_connections 2024-09-16T14:46:03.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:02.964+0000 7f99ef410640 1 -- 172.21.15.26:0/1505210630 wait complete. 2024-09-16T14:46:03.021 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:46:03.021 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:46:03.021 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:46:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 10 monclient: get_monmap_and_config 2024-09-16T14:46:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 10 monclient: build_initial_monmap 2024-09-16T14:46:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 10 monclient: monmap: 2024-09-16T14:46:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:46:03.130335+0000 2024-09-16T14:46:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:46:03.130335+0000 2024-09-16T14:46:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:03.131 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 1 Processor -- start 2024-09-16T14:46:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 1 -- start start 2024-09-16T14:46:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 10 monclient: init 2024-09-16T14:46:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 10 monclient(hunting): picked mon.noname-b con 0x7fee340a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 10 monclient(hunting): picked mon.noname-a con 0x7fee340a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 10 monclient(hunting): picked mon.noname-c con 0x7fee340a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee340a3250 con 0x7fee340a0000 2024-09-16T14:46:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3409f600 con 0x7fee340a2800 2024-09-16T14:46:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee340a3830 con 0x7fee340a1d10 2024-09-16T14:46:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 10 monclient(hunting): authenticate will time out at 1187.250479s 2024-09-16T14:46:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee39d40640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fee340a1d10 0x7fee340a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43326/0 (socket says 172.21.15.26:43326) 2024-09-16T14:46:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee39d40640 1 -- 172.21.15.26:0/2942576113 learned_addr learned my addr 172.21.15.26:0/2942576113 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3953f640 1 --1- 172.21.15.26:0/2942576113 >> v1:172.21.15.26:6789/0 conn(0x7fee340a0000 0x7fee34139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43026/0 (socket says 172.21.15.26:43026) 2024-09-16T14:46:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 1 -- 172.21.15.26:0/2942576113 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904023462 0 0) 0x7fee340a3830 con 0x7fee340a1d10 2024-09-16T14:46:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient(hunting): my global_id is 5345 2024-09-16T14:46:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 1 -- 172.21.15.26:0/2942576113 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee20003530 con 0x7fee340a1d10 2024-09-16T14:46:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 1 -- 172.21.15.26:0/2942576113 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498488842 0 0) 0x7fee340a3250 con 0x7fee340a0000 2024-09-16T14:46:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient(hunting): my global_id is 104124 2024-09-16T14:46:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 1 -- 172.21.15.26:0/2942576113 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee340a3830 con 0x7fee340a0000 2024-09-16T14:46:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 1 -- 172.21.15.26:0/2942576113 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1260714809 0 0) 0x7fee20003530 con 0x7fee340a1d10 2024-09-16T14:46:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 1 -- 172.21.15.26:0/2942576113 >> v1:172.21.15.189:6789/0 conn(0x7fee340a2800 legacy=0x7fee3413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 1 -- 172.21.15.26:0/2942576113 >> v1:172.21.15.26:6789/0 conn(0x7fee340a0000 legacy=0x7fee34139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient: _finish_hunting 0 2024-09-16T14:46:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 1 monclient: found mon.noname-b 2024-09-16T14:46:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:46:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 1 -- 172.21.15.26:0/2942576113 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3409ed60 con 0x7fee340a1d10 2024-09-16T14:46:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient: _finish_auth 0 2024-09-16T14:46:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient: _check_auth_tickets 2024-09-16T14:46:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 1 -- 172.21.15.26:0/2942576113 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fee24001620 con 0x7fee340a1d10 2024-09-16T14:46:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient: dump: 2024-09-16T14:46:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.138 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 5 monclient: authenticate success, global_id 5345 2024-09-16T14:46:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 1 -- 172.21.15.26:0/2942576113 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee24001840 con 0x7fee340a1d10 2024-09-16T14:46:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 4 set_mon_vals no callback set 2024-09-16T14:46:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 1 -- 172.21.15.26:0/2942576113 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fee24001c10 con 0x7fee340a1d10 2024-09-16T14:46:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee38d3e640 10 monclient: dump: 2024-09-16T14:46:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.139 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 10 monclient: shutdown 2024-09-16T14:46:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 1 -- 172.21.15.26:0/2942576113 >> v1:172.21.15.26:6790/0 conn(0x7fee340a1d10 legacy=0x7fee340a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 1 -- 172.21.15.26:0/2942576113 shutdown_connections 2024-09-16T14:46:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 1 -- 172.21.15.26:0/2942576113 >> 172.21.15.26:0/2942576113 conn(0x7fee340a7f30 msgr2=0x7fee340a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 1 -- 172.21.15.26:0/2942576113 shutdown_connections 2024-09-16T14:46:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.128+0000 7fee3bfa4640 1 -- 172.21.15.26:0/2942576113 wait complete. 2024-09-16T14:46:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 10 monclient: build_initial_monmap 2024-09-16T14:46:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 10 monclient: monmap: 2024-09-16T14:46:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:46:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:46:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:46:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:03.141 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 1 Processor -- start 2024-09-16T14:46:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 1 -- start start 2024-09-16T14:46:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 10 monclient: init 2024-09-16T14:46:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 10 monclient(hunting): picked mon.noname-a con 0x7fee340a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 10 monclient(hunting): picked mon.noname-c con 0x7fee340a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 10 monclient(hunting): picked mon.noname-b con 0x7fee340a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3409e500 con 0x7fee340a0000 2024-09-16T14:46:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee340a67f0 con 0x7fee340a2800 2024-09-16T14:46:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee340a62a0 con 0x7fee340a1d10 2024-09-16T14:46:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 10 monclient(hunting): authenticate will time out at 1187.253182s 2024-09-16T14:46:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3953f640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fee340a1d10 0x7fee341338b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43328/0 (socket says 172.21.15.26:43328) 2024-09-16T14:46:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3953f640 1 -- 172.21.15.26:0/1985031264 learned_addr learned my addr 172.21.15.26:0/1985031264 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268925607 0 0) 0x7fee340a62a0 con 0x7fee340a1d10 2024-09-16T14:46:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): my global_id is 5348 2024-09-16T14:46:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee18003530 con 0x7fee340a1d10 2024-09-16T14:46:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462654216 0 0) 0x7fee3409e500 con 0x7fee340a0000 2024-09-16T14:46:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): my global_id is 104127 2024-09-16T14:46:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee340a62a0 con 0x7fee340a0000 2024-09-16T14:46:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 210122473 0 0) 0x7fee18003530 con 0x7fee340a1d10 2024-09-16T14:46:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee3409e500 con 0x7fee340a1d10 2024-09-16T14:46:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fee30001370 con 0x7fee340a1d10 2024-09-16T14:46:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): dump: 2024-09-16T14:46:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.149 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4230172930 0 0) 0x7fee340a62a0 con 0x7fee340a0000 2024-09-16T14:46:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee18003530 con 0x7fee340a0000 2024-09-16T14:46:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fee24001440 con 0x7fee340a0000 2024-09-16T14:46:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): dump: 2024-09-16T14:46:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:03.149 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.150 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.150 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.150 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.150 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.150 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.150 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.150 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.150 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212961042 0 0) 0x7fee340a67f0 con 0x7fee340a2800 2024-09-16T14:46:03.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): my global_id is 94456 2024-09-16T14:46:03.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee340a62a0 con 0x7fee340a2800 2024-09-16T14:46:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2942796502 0 0) 0x7fee3409e500 con 0x7fee340a1d10 2024-09-16T14:46:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 >> v1:172.21.15.189:6789/0 conn(0x7fee340a2800 legacy=0x7fee34133fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 >> v1:172.21.15.26:6789/0 conn(0x7fee340a0000 legacy=0x7fee341331a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:46:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 monclient: found mon.c 2024-09-16T14:46:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:46:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee34135e00 con 0x7fee340a1d10 2024-09-16T14:46:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:46:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:46:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 5 monclient: authenticate success, global_id 5348 2024-09-16T14:46:03.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 10 monclient: _renew_subs 2024-09-16T14:46:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:46:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 1 -- 172.21.15.26:0/1985031264 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee34134df0 con 0x7fee340a1d10 2024-09-16T14:46:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 10 monclient: _renew_subs 2024-09-16T14:46:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:46:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 1 -- 172.21.15.26:0/1985031264 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee34137360 con 0x7fee340a1d10 2024-09-16T14:46:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee30001ca0 con 0x7fee340a1d10 2024-09-16T14:46:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fee30004aa0 con 0x7fee340a1d10 2024-09-16T14:46:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2bfff640 4 set_mon_vals no callback set 2024-09-16T14:46:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:03.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient: dump: 2024-09-16T14:46:03.153 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:03.153 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.153 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.153 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.153 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.153 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.153 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.153 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.153 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.153 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:03.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3bfa4640 1 -- 172.21.15.26:0/1985031264 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fee1c006590 con 0x7fee1c002830 2024-09-16T14:46:03.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fee3000f510 con 0x7fee340a1d10 2024-09-16T14:46:03.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(75..75 src has 15..75) v4 ==== 5337+0+0 (unknown 2993449477 0 0) 0x7fee300438f0 con 0x7fee340a1d10 2024-09-16T14:46:03.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093917087 0 0) 0x7fee1c006590 con 0x7fee1c002830 2024-09-16T14:46:03.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 10 monclient(hunting): my global_id is 94459 2024-09-16T14:46:03.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee3409e500 con 0x7fee1c002830 2024-09-16T14:46:03.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.132+0000 7fee3a541640 10 monclient: get_auth_request con 0x7fee18036380 auth_method 0 2024-09-16T14:46:03.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.136+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 363961336 0 0) 0x7fee3409e500 con 0x7fee1c002830 2024-09-16T14:46:03.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.136+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee1c006590 con 0x7fee1c002830 2024-09-16T14:46:03.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.136+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fee24004ce0 con 0x7fee1c002830 2024-09-16T14:46:03.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.136+0000 7fee2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.136+0000 7fee2a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:03.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.136+0000 7fee2a7fc640 10 monclient: dump: 2024-09-16T14:46:03.155 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:03.155 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.155 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.155 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.155 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.155 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.155 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.155 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.155 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.155 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.136+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1809936231 0 0) 0x7fee1c006590 con 0x7fee1c002830 2024-09-16T14:46:03.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.136+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fee1c00a0e0 con 0x7fee1c002830 2024-09-16T14:46:03.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.136+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fee24007480 con 0x7fee1c002830 2024-09-16T14:46:03.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.136+0000 7fee2a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:03.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.136+0000 7fee2a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:03.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.136+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 >> v1:172.21.15.189:6789/0 conn(0x7fee1c002830 legacy=0x7fee1c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.136+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 >> v1:172.21.15.189:6789/0 conn(0x7fee1c002830 legacy=0x7fee1c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:46:03.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.140+0000 7fee3bfa4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:46:03.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.140+0000 7fee3bfa4640 1 -- 172.21.15.26:0/1985031264 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fee1c009da0 con 0x7fee1c0090c0 2024-09-16T14:46:03.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658686892 0 0) 0x7fee1c009da0 con 0x7fee1c0090c0 2024-09-16T14:46:03.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 10 monclient(hunting): my global_id is 94462 2024-09-16T14:46:03.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee1c006590 con 0x7fee1c0090c0 2024-09-16T14:46:03.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3560242919 0 0) 0x7fee1c006590 con 0x7fee1c0090c0 2024-09-16T14:46:03.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee1c009da0 con 0x7fee1c0090c0 2024-09-16T14:46:03.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fee240053f0 con 0x7fee1c0090c0 2024-09-16T14:46:03.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:03.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 10 monclient: dump: 2024-09-16T14:46:03.157 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:03.157 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.157 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.157 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.157 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.157 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.157 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.158 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.158 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.158 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1295173539 0 0) 0x7fee1c009da0 con 0x7fee1c0090c0 2024-09-16T14:46:03.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fee1c00c290 con 0x7fee1c0090c0 2024-09-16T14:46:03.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 3565227078) 0x7fee240075f0 con 0x7fee1c0090c0 2024-09-16T14:46:03.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:46:03.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:03.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 >> v1:172.21.15.189:6789/0 conn(0x7fee1c0090c0 legacy=0x7fee1c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1985031264 >> v1:172.21.15.189:6789/0 conn(0x7fee1c0090c0 legacy=0x7fee1c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:46:03.159 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:46:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:46:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:46:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:46:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 38, 2024-09-16T14:46:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:46:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:46:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:46:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:46:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:46:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:46:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:46:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:46:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:46:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:46:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:46:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:46:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:46:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:46:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:46:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:46:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:46:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:46:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:46:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:46:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:46:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:46:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:46:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:46:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:46:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:46:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:46:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:46:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:46:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:46:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:46:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:46:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:46:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:46:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:46:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:46:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:46:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:46:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:46:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:46:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:46:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:46:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:46:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:46:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:46:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:46:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:46:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:46:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:03.170 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:03.170 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:46:03.170 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:03.170 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:03.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:03.170 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:46:03.170 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:46:03.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee3bfa4640 1 -- 172.21.15.26:0/1985031264 >> v1:172.21.15.189:6812/3196161582 conn(0x7fee18036380 legacy=0x7fee18038840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee3bfa4640 10 monclient: shutdown 2024-09-16T14:46:03.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee3bfa4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee3bfa4640 1 -- 172.21.15.26:0/1985031264 >> v1:172.21.15.26:6790/0 conn(0x7fee340a1d10 legacy=0x7fee341338b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee3a541640 1 -- 172.21.15.26:0/1985031264 reap_dead start 2024-09-16T14:46:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee3a541640 1 -- 172.21.15.26:0/1985031264 reap_dead start 2024-09-16T14:46:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee3bfa4640 1 -- 172.21.15.26:0/1985031264 shutdown_connections 2024-09-16T14:46:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee3bfa4640 1 -- 172.21.15.26:0/1985031264 >> 172.21.15.26:0/1985031264 conn(0x7fee340a7f30 msgr2=0x7fee3406b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee3bfa4640 1 -- 172.21.15.26:0/1985031264 shutdown_connections 2024-09-16T14:46:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.144+0000 7fee3bfa4640 1 -- 172.21.15.26:0/1985031264 wait complete. 2024-09-16T14:46:03.172 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:46:03.172 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:46:03.173 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:46:03.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.280+0000 7f00bc3bf640 10 monclient: get_monmap_and_config 2024-09-16T14:46:03.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.280+0000 7f00bc3bf640 10 monclient: build_initial_monmap 2024-09-16T14:46:03.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.280+0000 7f00bc3bf640 10 monclient: monmap: 2024-09-16T14:46:03.284 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:03.285 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.285 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:46:03.284200+0000 2024-09-16T14:46:03.285 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:46:03.284200+0000 2024-09-16T14:46:03.285 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:03.285 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.285 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:03.285 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:03.285 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:03.285 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.280+0000 7f00bc3bf640 1 Processor -- start 2024-09-16T14:46:03.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.280+0000 7f00bc3bf640 1 -- start start 2024-09-16T14:46:03.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.280+0000 7f00bc3bf640 10 monclient: init 2024-09-16T14:46:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient(hunting): picked mon.noname-c con 0x7f00b409a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient(hunting): picked mon.noname-b con 0x7f00b409d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient(hunting): picked mon.noname-a con 0x7f00b409b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b409dc60 con 0x7f00b409b040 2024-09-16T14:46:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b4099ae0 con 0x7f00b409a550 2024-09-16T14:46:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b409b8f0 con 0x7f00b409d1b0 2024-09-16T14:46:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient(hunting): authenticate will time out at 1187.404735s 2024-09-16T14:46:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b995a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f00b409d1b0 0x7f00b4136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43332/0 (socket says 172.21.15.26:43332) 2024-09-16T14:46:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b995a640 1 -- 172.21.15.26:0/3110980005 learned_addr learned my addr 172.21.15.26:0/3110980005 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 1 -- 172.21.15.26:0/3110980005 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239073683 0 0) 0x7f00b409b8f0 con 0x7f00b409d1b0 2024-09-16T14:46:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient(hunting): my global_id is 5351 2024-09-16T14:46:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 1 -- 172.21.15.26:0/3110980005 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00a0005740 con 0x7f00b409d1b0 2024-09-16T14:46:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 1 -- 172.21.15.26:0/3110980005 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364858393 0 0) 0x7f00b409dc60 con 0x7f00b409b040 2024-09-16T14:46:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient(hunting): my global_id is 104130 2024-09-16T14:46:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 1 -- 172.21.15.26:0/3110980005 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00b409b8f0 con 0x7f00b409b040 2024-09-16T14:46:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 1 -- 172.21.15.26:0/3110980005 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1239665089 0 0) 0x7f00a0005740 con 0x7f00b409d1b0 2024-09-16T14:46:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 1 -- 172.21.15.26:0/3110980005 >> v1:172.21.15.189:6789/0 conn(0x7f00b409a550 legacy=0x7f00b409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 1 -- 172.21.15.26:0/3110980005 >> v1:172.21.15.26:6789/0 conn(0x7f00b409b040 legacy=0x7f00b413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient: _finish_hunting 0 2024-09-16T14:46:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 1 monclient: found mon.noname-b 2024-09-16T14:46:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:46:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 1 -- 172.21.15.26:0/3110980005 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b409bbd0 con 0x7f00b409d1b0 2024-09-16T14:46:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient: _finish_auth 0 2024-09-16T14:46:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient: _check_auth_tickets 2024-09-16T14:46:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 5 monclient: authenticate success, global_id 5351 2024-09-16T14:46:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 1 -- 172.21.15.26:0/3110980005 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f00a4001620 con 0x7f00b409d1b0 2024-09-16T14:46:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient: dump: 2024-09-16T14:46:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.292 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 1 -- 172.21.15.26:0/3110980005 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00a40018c0 con 0x7f00b409d1b0 2024-09-16T14:46:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 1 -- 172.21.15.26:0/3110980005 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f00a4001c90 con 0x7f00b409d1b0 2024-09-16T14:46:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 4 set_mon_vals no callback set 2024-09-16T14:46:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00b9159640 10 monclient: dump: 2024-09-16T14:46:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.294 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient: shutdown 2024-09-16T14:46:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 1 -- 172.21.15.26:0/3110980005 >> v1:172.21.15.26:6790/0 conn(0x7f00b409d1b0 legacy=0x7f00b4136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 1 -- 172.21.15.26:0/3110980005 shutdown_connections 2024-09-16T14:46:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 1 -- 172.21.15.26:0/3110980005 >> 172.21.15.26:0/3110980005 conn(0x7f00b409fb50 msgr2=0x7f00b409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 1 -- 172.21.15.26:0/3110980005 shutdown_connections 2024-09-16T14:46:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 1 -- 172.21.15.26:0/3110980005 wait complete. 2024-09-16T14:46:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient: build_initial_monmap 2024-09-16T14:46:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient: monmap: 2024-09-16T14:46:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:46:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:46:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:46:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:03.295 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 1 Processor -- start 2024-09-16T14:46:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 1 -- start start 2024-09-16T14:46:03.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient: init 2024-09-16T14:46:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:46:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient(hunting): picked mon.noname-a con 0x7f00b409a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient(hunting): picked mon.noname-b con 0x7f00b409b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient(hunting): picked mon.noname-c con 0x7f00b409d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b41a34e0 con 0x7f00b409a550 2024-09-16T14:46:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b40a2630 con 0x7f00b409b040 2024-09-16T14:46:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b409a240 con 0x7f00b409d1b0 2024-09-16T14:46:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.284+0000 7f00bc3bf640 10 monclient(hunting): authenticate will time out at 1187.407992s 2024-09-16T14:46:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00ba15b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f00b409a550 0x7f00b41b4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43058/0 (socket says 172.21.15.26:43058) 2024-09-16T14:46:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00ba15b640 1 -- 172.21.15.26:0/3144512766 learned_addr learned my addr 172.21.15.26:0/3144512766 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988901535 0 0) 0x7f00b41a34e0 con 0x7f00b409a550 2024-09-16T14:46:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): my global_id is 104133 2024-09-16T14:46:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0098003530 con 0x7f00b409a550 2024-09-16T14:46:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006340222 0 0) 0x7f00b409a240 con 0x7f00b409d1b0 2024-09-16T14:46:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): my global_id is 5354 2024-09-16T14:46:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00b41a34e0 con 0x7f00b409d1b0 2024-09-16T14:46:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 519269374 0 0) 0x7f0098003530 con 0x7f00b409a550 2024-09-16T14:46:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00b409a240 con 0x7f00b409a550 2024-09-16T14:46:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f00b0001940 con 0x7f00b409a550 2024-09-16T14:46:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:46:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): dump: 2024-09-16T14:46:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.302 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1065836504 0 0) 0x7f00b41a34e0 con 0x7f00b409d1b0 2024-09-16T14:46:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0098003530 con 0x7f00b409d1b0 2024-09-16T14:46:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f00ac001be0 con 0x7f00b409d1b0 2024-09-16T14:46:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): dump: 2024-09-16T14:46:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.303 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873111112 0 0) 0x7f00b40a2630 con 0x7f00b409b040 2024-09-16T14:46:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient(hunting): my global_id is 94468 2024-09-16T14:46:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00b41a34e0 con 0x7f00b409b040 2024-09-16T14:46:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2013696008 0 0) 0x7f00b409a240 con 0x7f00b409a550 2024-09-16T14:46:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 >> v1:172.21.15.26:6790/0 conn(0x7f00b409d1b0 legacy=0x7f00b41cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 >> v1:172.21.15.189:6789/0 conn(0x7f00b409b040 legacy=0x7f00b41ca210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 monclient: found mon.a 2024-09-16T14:46:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b41a1ad0 con 0x7f00b409a550 2024-09-16T14:46:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient: _finish_auth 0 2024-09-16T14:46:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00bc3bf640 5 monclient: authenticate success, global_id 104133 2024-09-16T14:46:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00bc3bf640 10 monclient: _renew_subs 2024-09-16T14:46:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00bc3bf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:03.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00bc3bf640 1 -- 172.21.15.26:0/3144512766 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b41a2c40 con 0x7f00b409a550 2024-09-16T14:46:03.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00bc3bf640 10 monclient: _renew_subs 2024-09-16T14:46:03.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00bc3bf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:03.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00bc3bf640 1 -- 172.21.15.26:0/3144512766 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b41c2410 con 0x7f00b409a550 2024-09-16T14:46:03.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00b0003620 con 0x7f00b409a550 2024-09-16T14:46:03.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:03.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f00b0003930 con 0x7f00b409a550 2024-09-16T14:46:03.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:03.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 10 monclient: dump: 2024-09-16T14:46:03.305 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:03.305 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.305 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.306 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.306 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.306 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:03.306 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.306 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.306 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.306 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:03.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00abfff640 4 set_mon_vals no callback set 2024-09-16T14:46:03.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f00b0003a90 con 0x7f00b409a550 2024-09-16T14:46:03.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00bc3bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:03.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(75..75 src has 15..75) v4 ==== 5337+0+0 (unknown 2993449477 0 0) 0x7f00b0043840 con 0x7f00b409a550 2024-09-16T14:46:03.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00bc3bf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:03.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00bc3bf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:03.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00bc3bf640 1 -- 172.21.15.26:0/3144512766 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f009c005050 con 0x7f00b409a550 2024-09-16T14:46:03.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.288+0000 7f00b995a640 10 monclient: get_auth_request con 0x7f0098036490 auth_method 0 2024-09-16T14:46:03.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.292+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f00b0004290 con 0x7f00b409a550 2024-09-16T14:46:03.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.292+0000 7f00aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:03.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.292+0000 7f00aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:03.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.436+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}]=0 removed item id 7 name 'osd.7' from crush map v76) v1 ==== 143+0+0 (unknown 1772838655 0 0) 0x7f3c08012b00 con 0x7f3c1c0a1040 2024-09-16T14:46:03.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.436+0000 7f3c197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T14:46:03.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.436+0000 7f3c197fa640 10 monclient: _finish_command 2 = system:0 removed item id 7 name 'osd.7' from crush map 2024-09-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.504+0000 7f00bc3bf640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.504+0000 7f00bc3bf640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.504+0000 7f00bc3bf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:03.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.504+0000 7f00bc3bf640 1 -- 172.21.15.26:0/3144512766 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f009c005b10 con 0x7f00b409a550 2024-09-16T14:46:03.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.508+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.508+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=2,health=12,kv=9,logm=61} crc {auth=197722518,config=692619927,health=1361790972,kv=2627315910,logm=1535267029}) 2024-09-16T14:46:03.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.508+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=2679250430,config=692619927,health=312797148,kv=263502946,logm=2148204366}) 2024-09-16T14:46:03.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.508+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.508+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=2,health=12,kv=9,logm=61} crc {auth=197722518,config=692619927,health=1361790972,kv=2627315910,logm=1535267029}) 2024-09-16T14:46:03.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.508+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=12,kv=11,logm=59} crc {auth=1492242079,config=692619927,health=1361790972,kv=1867599010,logm=41101570}) 2024-09-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.508+0000 7f00aaffd640 1 -- 172.21.15.26:0/3144512766 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f00b0012b10 con 0x7f00b409a550 2024-09-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.508+0000 7f00aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:46:03.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.508+0000 7f00aaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f00bc3bf640 1 -- 172.21.15.26:0/3144512766 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0098036490 legacy=0x7f0098038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f00bc3bf640 10 monclient: shutdown 2024-09-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f00bc3bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f00bc3bf640 1 -- 172.21.15.26:0/3144512766 >> v1:172.21.15.26:6789/0 conn(0x7f00b409a550 legacy=0x7f00b41b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f00bc3bf640 1 -- 172.21.15.26:0/3144512766 shutdown_connections 2024-09-16T14:46:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f00bc3bf640 1 -- 172.21.15.26:0/3144512766 >> 172.21.15.26:0/3144512766 conn(0x7f00b409fb50 msgr2=0x7f00b40a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:03.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1715294278}) 2024-09-16T14:46:03.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2526960212}) 2024-09-16T14:46:03.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1715294278}) 2024-09-16T14:46:03.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=342822073}) 2024-09-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f00bc3bf640 1 -- 172.21.15.26:0/3144512766 shutdown_connections 2024-09-16T14:46:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f00bc3bf640 1 -- 172.21.15.26:0/3144512766 wait complete. 2024-09-16T14:46:03.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1810779093}) 2024-09-16T14:46:03.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835555535}) 2024-09-16T14:46:03.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1810779093}) 2024-09-16T14:46:03.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.512+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3428096108}) 2024-09-16T14:46:03.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.516+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.516+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1722956900}) 2024-09-16T14:46:03.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.516+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1609768146}) 2024-09-16T14:46:03.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.516+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.516+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1722956900}) 2024-09-16T14:46:03.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.516+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1037946579}) 2024-09-16T14:46:03.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.516+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.516+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=599252640}) 2024-09-16T14:46:03.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.516+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770547712}) 2024-09-16T14:46:03.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.516+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.516+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=599252640}) 2024-09-16T14:46:03.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.516+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1007221645}) 2024-09-16T14:46:03.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.520+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.520+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1403757252}) 2024-09-16T14:46:03.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.520+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3259588494}) 2024-09-16T14:46:03.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.520+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.520+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1403757252}) 2024-09-16T14:46:03.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.520+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1839613740}) 2024-09-16T14:46:03.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.520+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.520+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334884690}) 2024-09-16T14:46:03.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.520+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245084708}) 2024-09-16T14:46:03.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.520+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.520+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334884690}) 2024-09-16T14:46:03.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.520+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2429321686}) 2024-09-16T14:46:03.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.524+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.524+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3444047219}) 2024-09-16T14:46:03.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.524+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209975331}) 2024-09-16T14:46:03.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.524+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.524+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3444047219}) 2024-09-16T14:46:03.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.524+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3483076445}) 2024-09-16T14:46:03.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.524+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.524+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2298963908}) 2024-09-16T14:46:03.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.524+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3420373883}) 2024-09-16T14:46:03.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.524+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.524+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2298963908}) 2024-09-16T14:46:03.528 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.524+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=354020642}) 2024-09-16T14:46:03.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.528+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.530 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.528+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2649610830}) 2024-09-16T14:46:03.530 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.528+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1343795744}) 2024-09-16T14:46:03.530 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.528+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.530 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.528+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2649610830}) 2024-09-16T14:46:03.530 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.528+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1999193777}) 2024-09-16T14:46:03.531 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:46:03.532 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.528+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.532 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.528+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3833494826}) 2024-09-16T14:46:03.532 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.528+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=697445626}) 2024-09-16T14:46:03.532 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.528+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.532 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.528+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3833494826}) 2024-09-16T14:46:03.532 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.528+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3553708974}) 2024-09-16T14:46:03.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.532+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.532+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=60,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,osd_metadata=2} crc {logm=30870591,mdsmap=1531151339,mgr=916073340,mgr_metadata=2109538692,mgrstat=1353170003,mon_config_key=270382713,monmap=970648238,osd_metadata=1100806521}) 2024-09-16T14:46:03.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.532+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=59,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=2} crc {logm=2620609885,mdsmap=1531151339,mgr=4284542101,mgr_command_descs=794974419,mgrstat=1353170003,mon_config_key=1518680965,monmap=970648238,osd_metadata=1100806521}) 2024-09-16T14:46:03.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.532+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.532+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=60,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,osd_metadata=2} crc {logm=30870591,mdsmap=1531151339,mgr=916073340,mgr_metadata=2109538692,mgrstat=1353170003,mon_config_key=270382713,monmap=970648238,osd_metadata=1100806521}) 2024-09-16T14:46:03.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.532+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=61,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=2} crc {logm=485273502,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=1353170003,mon_config_key=1518680965,monmap=2765269217}) 2024-09-16T14:46:03.537 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.532+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.537 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.532+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=93} crc {osd_metadata=3759610743,osd_pg_creating=2344302655,osd_snap=3794734670,osdmap=4108733011}) 2024-09-16T14:46:03.537 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.532+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osd_snap=2,osdmap=94} crc {osd_metadata=2530836999,osd_pg_creating=2344302655,osd_snap=3794734670,osdmap=1730987388}) 2024-09-16T14:46:03.537 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.532+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.537 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.532+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=93} crc {osd_metadata=3759610743,osd_pg_creating=2344302655,osd_snap=3794734670,osdmap=4108733011}) 2024-09-16T14:46:03.537 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.532+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=93} crc {osd_metadata=3759610743,osd_pg_creating=2344302655,osd_snap=3794734670,osdmap=1583534737}) 2024-09-16T14:46:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.536+0000 7f3c230d9640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T14:46:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.536+0000 7f3c230d9640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T14:46:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.536+0000 7f3c230d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.536+0000 7f3c230d9640 1 -- 172.21.15.189:0/1694266085 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"} v 0) v1 -- 0x7f3bfc005300 con 0x7f3c1c0a1040 2024-09-16T14:46:03.542 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.536+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.542 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.536+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=16} crc {osdmap=490030920}) 2024-09-16T14:46:03.542 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.536+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=14} crc {osdmap=3584982167}) 2024-09-16T14:46:03.542 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.536+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:46:03.542 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.536+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=16} crc {osdmap=490030920}) 2024-09-16T14:46:03.542 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:46:03.536+0000 7f2d60adf640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=14} crc {osdmap=1298118301}) 2024-09-16T14:46:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.540+0000 7f3c197fa640 1 -- 172.21.15.189:0/1694266085 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}]=0 device 'osd.7' does not appear in the crush map v76) v1 ==== 145+0+0 (unknown 3881301127 0 0) 0x7f3c08014280 con 0x7f3c1c0a1040 2024-09-16T14:46:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.540+0000 7f3c197fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T14:46:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.540+0000 7f3c197fa640 10 monclient: _finish_command 3 = system:0 device 'osd.7' does not appear in the crush map 2024-09-16T14:46:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:device 'osd.7' does not appear in the crush map 2024-09-16T14:46:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.540+0000 7f3c230d9640 1 -- 172.21.15.189:0/1694266085 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3c00036470 legacy=0x7f3c00038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.540+0000 7f3c230d9640 10 monclient: shutdown 2024-09-16T14:46:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.540+0000 7f3c230d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.540+0000 7f3c230d9640 1 -- 172.21.15.189:0/1694266085 >> v1:172.21.15.189:6789/0 conn(0x7f3c1c0a1040 legacy=0x7f3c1c12f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.540+0000 7f3c230d9640 1 -- 172.21.15.189:0/1694266085 shutdown_connections 2024-09-16T14:46:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.540+0000 7f3c230d9640 1 -- 172.21.15.189:0/1694266085 >> 172.21.15.189:0/1694266085 conn(0x7f3c1c0a3a40 msgr2=0x7f3c1c06b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.540+0000 7f3c230d9640 1 -- 172.21.15.189:0/1694266085 shutdown_connections 2024-09-16T14:46:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.540+0000 7f3c230d9640 1 -- 172.21.15.189:0/1694266085 wait complete. 2024-09-16T14:46:03.560 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm host1 2024-09-16T14:46:03.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f212d29e640 10 monclient: get_monmap_and_config 2024-09-16T14:46:03.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f212d29e640 10 monclient: build_initial_monmap 2024-09-16T14:46:03.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f212d29e640 10 monclient: monmap: 2024-09-16T14:46:03.680 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:03.680 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.680 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:03.679703+0000 2024-09-16T14:46:03.680 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:03.679703+0000 2024-09-16T14:46:03.680 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:03.680 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:03.680 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:03.680 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:03.680 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:03.680 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:03.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f212d29e640 1 Processor -- start 2024-09-16T14:46:03.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f212d29e640 1 -- start start 2024-09-16T14:46:03.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f212d29e640 10 monclient: init 2024-09-16T14:46:03.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f212d29e640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:03.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f212d29e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:03.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f212d29e640 10 monclient(hunting): picked mon.noname-c con 0x7f212809df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:03.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f212d29e640 10 monclient(hunting): picked mon.noname-a con 0x7f212809e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:03.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f212d29e640 10 monclient(hunting): picked mon.noname-b con 0x7f212809e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:03.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f212d29e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f212809ee20 con 0x7f212809e2f0 2024-09-16T14:46:03.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f212d29e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f212809f100 con 0x7f212809df10 2024-09-16T14:46:03.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f212d29e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f212809f3e0 con 0x7f212809e740 2024-09-16T14:46:03.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f212d29e640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:03.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f212d29e640 10 monclient(hunting): authenticate will time out at 1145.795336s 2024-09-16T14:46:03.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f2126ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f212809df10 0x7f212809d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58200/0 (socket says 172.21.15.189:58200) 2024-09-16T14:46:03.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f2126ffd640 1 -- 172.21.15.189:0/539149808 learned_addr learned my addr 172.21.15.189:0/539149808 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:03.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f2125ffb640 1 -- 172.21.15.189:0/539149808 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363661263 0 0) 0x7f212809f100 con 0x7f212809df10 2024-09-16T14:46:03.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f2125ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f2125ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f2125ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f2125ffb640 10 monclient(hunting): my global_id is 94471 2024-09-16T14:46:03.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.676+0000 7f2125ffb640 1 -- 172.21.15.189:0/539149808 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2110003530 con 0x7f212809df10 2024-09-16T14:46:03.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 1 -- 172.21.15.189:0/539149808 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1440140827 0 0) 0x7f2110003530 con 0x7f212809df10 2024-09-16T14:46:03.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 1 -- 172.21.15.189:0/539149808 >> v1:172.21.15.26:6790/0 conn(0x7f212809e740 legacy=0x7f212813b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 1 -- 172.21.15.189:0/539149808 >> v1:172.21.15.26:6789/0 conn(0x7f212809e2f0 legacy=0x7f21281378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:46:03.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 1 monclient: found mon.noname-c 2024-09-16T14:46:03.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:03.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:03.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 1 -- 172.21.15.189:0/539149808 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212809c690 con 0x7f212809df10 2024-09-16T14:46:03.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 10 monclient: _finish_auth 0 2024-09-16T14:46:03.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:03.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:03.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 1 -- 172.21.15.189:0/539149808 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2118001620 con 0x7f212809df10 2024-09-16T14:46:03.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:03.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 10 monclient: dump: 2024-09-16T14:46:03.690 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:03.690 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.690 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.690 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.690 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.690 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:03.690 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.690 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.690 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.690 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:03.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 5 monclient: authenticate success, global_id 94471 2024-09-16T14:46:03.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:03.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 1 -- 172.21.15.189:0/539149808 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2118001940 con 0x7f212809df10 2024-09-16T14:46:03.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:03.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:03.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 4 set_mon_vals no callback set 2024-09-16T14:46:03.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 1 -- 172.21.15.189:0/539149808 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2118001cd0 con 0x7f212809df10 2024-09-16T14:46:03.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:03.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f2125ffb640 10 monclient: dump: 2024-09-16T14:46:03.691 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:03.691 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.691 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.691 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.691 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.691 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:03.692 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.692 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.692 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.692 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:03.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 10 monclient: shutdown 2024-09-16T14:46:03.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:03.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 1 -- 172.21.15.189:0/539149808 >> v1:172.21.15.189:6789/0 conn(0x7f212809df10 legacy=0x7f212809d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 1 -- 172.21.15.189:0/539149808 shutdown_connections 2024-09-16T14:46:03.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 1 -- 172.21.15.189:0/539149808 >> 172.21.15.189:0/539149808 conn(0x7f21280a3320 msgr2=0x7f212809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:03.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 1 -- 172.21.15.189:0/539149808 shutdown_connections 2024-09-16T14:46:03.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 1 -- 172.21.15.189:0/539149808 wait complete. 2024-09-16T14:46:03.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 10 monclient: build_initial_monmap 2024-09-16T14:46:03.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 10 monclient: monmap: 2024-09-16T14:46:03.692 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:03.693 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:03.693 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:03.693 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:03.693 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:03.693 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:03.693 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:03.693 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:03.693 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:03.693 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:03.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 1 Processor -- start 2024-09-16T14:46:03.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 1 -- start start 2024-09-16T14:46:03.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 10 monclient: init 2024-09-16T14:46:03.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:03.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:03.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 10 monclient(hunting): picked mon.noname-c con 0x7f212809df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:03.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 10 monclient(hunting): picked mon.noname-a con 0x7f212809e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:03.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 10 monclient(hunting): picked mon.noname-b con 0x7f212809e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:03.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f212809be70 con 0x7f212809e2f0 2024-09-16T14:46:03.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f212809b040 con 0x7f212809e740 2024-09-16T14:46:03.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f212806cf20 con 0x7f212809df10 2024-09-16T14:46:03.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:03.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 10 monclient(hunting): authenticate will time out at 1145.798496s 2024-09-16T14:46:03.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f21277fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f212809e740 0x7f21281d17a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58214/0 (socket says 172.21.15.189:58214) 2024-09-16T14:46:03.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f21277fe640 1 -- 172.21.15.189:0/2071368919 learned_addr learned my addr 172.21.15.189:0/2071368919 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:03.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611036394 0 0) 0x7f212809b040 con 0x7f212809e740 2024-09-16T14:46:03.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient(hunting): my global_id is 94474 2024-09-16T14:46:03.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2114003530 con 0x7f212809e740 2024-09-16T14:46:03.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2815753404 0 0) 0x7f2114003530 con 0x7f212809e740 2024-09-16T14:46:03.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f212809b040 con 0x7f212809e740 2024-09-16T14:46:03.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2120001c40 con 0x7f212809e740 2024-09-16T14:46:03.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:03.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient(hunting): dump: 2024-09-16T14:46:03.695 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:03.695 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.695 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.696 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.696 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.696 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:03.696 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.696 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.696 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.696 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:03.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:03.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997326324 0 0) 0x7f212809be70 con 0x7f212809e2f0 2024-09-16T14:46:03.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient(hunting): my global_id is 104139 2024-09-16T14:46:03.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2114003530 con 0x7f212809e2f0 2024-09-16T14:46:03.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192144790 0 0) 0x7f212806cf20 con 0x7f212809df10 2024-09-16T14:46:03.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:03.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:03.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:03.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient(hunting): my global_id is 5360 2024-09-16T14:46:03.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f212809be70 con 0x7f212809df10 2024-09-16T14:46:03.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 747426770 0 0) 0x7f212809b040 con 0x7f212809e740 2024-09-16T14:46:03.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 >> v1:172.21.15.26:6790/0 conn(0x7f212809df10 legacy=0x7f212809a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 >> v1:172.21.15.26:6789/0 conn(0x7f212809e2f0 legacy=0x7f212809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:03.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:03.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 1 monclient: found mon.b 2024-09-16T14:46:03.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:03.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:03.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212806b380 con 0x7f212809e740 2024-09-16T14:46:03.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:03.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:03.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:03.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 5 monclient: authenticate success, global_id 94474 2024-09-16T14:46:03.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 10 monclient: _renew_subs 2024-09-16T14:46:03.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:03.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 1 -- 172.21.15.189:0/2071368919 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f212813a440 con 0x7f212809e740 2024-09-16T14:46:03.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 10 monclient: _renew_subs 2024-09-16T14:46:03.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:03.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f212d29e640 1 -- 172.21.15.189:0/2071368919 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f212813aaf0 con 0x7f212809e740 2024-09-16T14:46:03.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2120002050 con 0x7f212809e740 2024-09-16T14:46:03.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:03.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2120003e70 con 0x7f212809e740 2024-09-16T14:46:03.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:03.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:03.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f21257fa640 4 set_mon_vals no callback set 2024-09-16T14:46:03.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.680+0000 7f210f7fe640 10 monclient: dump: 2024-09-16T14:46:03.699 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:03.699 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:03.699 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.699 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:03.699 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:03.699 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:03.700 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:03.700 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:03.700 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:03.700 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:03.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.684+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2120004050 con 0x7f212809e740 2024-09-16T14:46:03.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.684+0000 7f212d29e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:03.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.684+0000 7f212d29e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:03.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.684+0000 7f212d29e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:03.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.684+0000 7f212d29e640 1 -- 172.21.15.189:0/2071368919 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2108005050 con 0x7f212809e740 2024-09-16T14:46:03.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.684+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(76..76 src has 15..76) v4 ==== 5316+0+0 (unknown 4206865450 0 0) 0x7f2120043b50 con 0x7f212809e740 2024-09-16T14:46:03.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.684+0000 7f2126ffd640 10 monclient: get_auth_request con 0x7f21140362f0 auth_method 0 2024-09-16T14:46:03.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.688+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2120014930 con 0x7f212809e740 2024-09-16T14:46:03.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.688+0000 7f210f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:03.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.688+0000 7f210f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:03.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.852+0000 7f212d29e640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T14:46:03.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.852+0000 7f212d29e640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T14:46:03.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.852+0000 7f212d29e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:03.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:03.852+0000 7f212d29e640 1 -- 172.21.15.189:0/2071368919 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host1"} v 0) v1 -- 0x7f2108005300 con 0x7f212809e740 2024-09-16T14:46:04.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.516+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1"}]=0 removed item id -11 name 'host1' from crush map v77) v1 ==== 124+0+0 (unknown 3515914486 0 0) 0x7f212003ce00 con 0x7f212809e740 2024-09-16T14:46:04.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.516+0000 7f210f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T14:46:04.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.516+0000 7f210f7fe640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-16T14:46:04.532 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:46:04.532 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:46:04.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.620+0000 7f212d29e640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T14:46:04.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.620+0000 7f212d29e640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T14:46:04.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.620+0000 7f212d29e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:04.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.620+0000 7f212d29e640 1 -- 172.21.15.189:0/2071368919 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host1"} v 0) v1 -- 0x7f21080023c0 con 0x7f212809e740 2024-09-16T14:46:04.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.620+0000 7f210f7fe640 1 -- 172.21.15.189:0/2071368919 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1"}]=0 device 'host1' does not appear in the crush map v77) v1 ==== 124+0+0 (unknown 1240852503 0 0) 0x7f2120012fd0 con 0x7f212809e740 2024-09-16T14:46:04.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.620+0000 7f210f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T14:46:04.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.620+0000 7f210f7fe640 10 monclient: _finish_command 3 = system:0 device 'host1' does not appear in the crush map 2024-09-16T14:46:04.624 INFO:tasks.workunit.client.0.smithi189.stderr:device 'host1' does not appear in the crush map 2024-09-16T14:46:04.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.624+0000 7f212d29e640 1 -- 172.21.15.189:0/2071368919 >> v1:172.21.15.189:6812/3196161582 conn(0x7f21140362f0 legacy=0x7f21140387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:04.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.624+0000 7f212d29e640 10 monclient: shutdown 2024-09-16T14:46:04.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.624+0000 7f212d29e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:04.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.624+0000 7f212d29e640 1 -- 172.21.15.189:0/2071368919 >> v1:172.21.15.189:6789/0 conn(0x7f212809e740 legacy=0x7f21281d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:04.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.624+0000 7f212d29e640 1 -- 172.21.15.189:0/2071368919 shutdown_connections 2024-09-16T14:46:04.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.624+0000 7f212d29e640 1 -- 172.21.15.189:0/2071368919 >> 172.21.15.189:0/2071368919 conn(0x7f21280a3320 msgr2=0x7f21280a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:04.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.624+0000 7f212d29e640 1 -- 172.21.15.189:0/2071368919 shutdown_connections 2024-09-16T14:46:04.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.624+0000 7f212d29e640 1 -- 172.21.15.189:0/2071368919 wait complete. 2024-09-16T14:46:04.641 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T14:46:04.641 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 0 2024-09-16T14:46:04.641 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c host1 2024-09-16T14:46:04.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.652+0000 7fb87eab7640 10 monclient: get_monmap_and_config 2024-09-16T14:46:04.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.652+0000 7fb87eab7640 10 monclient: build_initial_monmap 2024-09-16T14:46:04.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.652+0000 7fb87eab7640 10 monclient: monmap: 2024-09-16T14:46:04.656 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:04.656 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:04.656 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:46:04.655360+0000 2024-09-16T14:46:04.656 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:46:04.655360+0000 2024-09-16T14:46:04.656 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:04.656 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:04.656 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:04.656 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:04.656 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:04.656 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:04.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.652+0000 7fb87eab7640 1 Processor -- start 2024-09-16T14:46:04.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.652+0000 7fb87eab7640 1 -- start start 2024-09-16T14:46:04.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.652+0000 7fb87eab7640 10 monclient: init 2024-09-16T14:46:04.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:04.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:04.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient(hunting): picked mon.noname-c con 0x7fb87809a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:04.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient(hunting): picked mon.noname-a con 0x7fb87809d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:04.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient(hunting): picked mon.noname-b con 0x7fb87809b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:04.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87809dc60 con 0x7fb87809d1b0 2024-09-16T14:46:04.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb878099ae0 con 0x7fb87809a550 2024-09-16T14:46:04.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87809b8f0 con 0x7fb87809b040 2024-09-16T14:46:04.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:04.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient(hunting): authenticate will time out at 1188.776175s 2024-09-16T14:46:04.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87d054640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb87809b040 0x7fb87813a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43346/0 (socket says 172.21.15.26:43346) 2024-09-16T14:46:04.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87d054640 1 -- 172.21.15.26:0/3766893694 learned_addr learned my addr 172.21.15.26:0/3766893694 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:04.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 1 -- 172.21.15.26:0/3766893694 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524367655 0 0) 0x7fb87809dc60 con 0x7fb87809d1b0 2024-09-16T14:46:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient(hunting): my global_id is 104142 2024-09-16T14:46:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 1 -- 172.21.15.26:0/3766893694 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb860003530 con 0x7fb87809d1b0 2024-09-16T14:46:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 1 -- 172.21.15.26:0/3766893694 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986857186 0 0) 0x7fb87809b8f0 con 0x7fb87809b040 2024-09-16T14:46:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient(hunting): my global_id is 5363 2024-09-16T14:46:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 1 -- 172.21.15.26:0/3766893694 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb87809dc60 con 0x7fb87809b040 2024-09-16T14:46:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 1 -- 172.21.15.26:0/3766893694 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002179488 0 0) 0x7fb878099ae0 con 0x7fb87809a550 2024-09-16T14:46:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:04.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:04.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:04.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient(hunting): my global_id is 94477 2024-09-16T14:46:04.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 1 -- 172.21.15.26:0/3766893694 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb87809b8f0 con 0x7fb87809a550 2024-09-16T14:46:04.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 1 -- 172.21.15.26:0/3766893694 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1959312665 0 0) 0x7fb860003530 con 0x7fb87809d1b0 2024-09-16T14:46:04.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 1 -- 172.21.15.26:0/3766893694 >> v1:172.21.15.26:6790/0 conn(0x7fb87809b040 legacy=0x7fb87813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:04.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 1 -- 172.21.15.26:0/3766893694 >> v1:172.21.15.189:6789/0 conn(0x7fb87809a550 legacy=0x7fb87809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:04.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:04.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 1 monclient: found mon.noname-a 2024-09-16T14:46:04.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:04.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:46:04.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 1 -- 172.21.15.26:0/3766893694 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87809bbd0 con 0x7fb87809d1b0 2024-09-16T14:46:04.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:04.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:04.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:04.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 1 -- 172.21.15.26:0/3766893694 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb86c001620 con 0x7fb87809d1b0 2024-09-16T14:46:04.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 5 monclient: authenticate success, global_id 104142 2024-09-16T14:46:04.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:04.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:04.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:46:04.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient: dump: 2024-09-16T14:46:04.668 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:04.668 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:04.668 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.668 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.668 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:04.668 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:04.669 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:04.669 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:04.669 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:04.669 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:04.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:04.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 1 -- 172.21.15.26:0/3766893694 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb86c001900 con 0x7fb87809d1b0 2024-09-16T14:46:04.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:04.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 1 -- 172.21.15.26:0/3766893694 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb86c001cd0 con 0x7fb87809d1b0 2024-09-16T14:46:04.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:04.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:04.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8777fe640 10 monclient: dump: 2024-09-16T14:46:04.670 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:04.670 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:04.670 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.670 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.670 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:04.670 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:04.670 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:04.670 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:04.671 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:04.671 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:04.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:04.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 4 set_mon_vals no callback set 2024-09-16T14:46:04.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient: shutdown 2024-09-16T14:46:04.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:04.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 1 -- 172.21.15.26:0/3766893694 >> v1:172.21.15.26:6789/0 conn(0x7fb87809d1b0 legacy=0x7fb878136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:04.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 1 -- 172.21.15.26:0/3766893694 shutdown_connections 2024-09-16T14:46:04.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 1 -- 172.21.15.26:0/3766893694 >> 172.21.15.26:0/3766893694 conn(0x7fb87809fb50 msgr2=0x7fb87809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:04.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 1 -- 172.21.15.26:0/3766893694 shutdown_connections 2024-09-16T14:46:04.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 1 -- 172.21.15.26:0/3766893694 wait complete. 2024-09-16T14:46:04.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient: build_initial_monmap 2024-09-16T14:46:04.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient: monmap: 2024-09-16T14:46:04.672 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:04.672 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:04.672 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:46:04.672 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:46:04.672 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:04.672 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:46:04.672 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:04.672 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:04.672 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:04.672 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:04.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 1 Processor -- start 2024-09-16T14:46:04.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 1 -- start start 2024-09-16T14:46:04.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient: init 2024-09-16T14:46:04.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:04.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:04.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient(hunting): picked mon.noname-b con 0x7fb87809a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:04.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient(hunting): picked mon.noname-a con 0x7fb87809b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:04.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient(hunting): picked mon.noname-c con 0x7fb87809d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:04.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8781a5d40 con 0x7fb87809b040 2024-09-16T14:46:04.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8781a4890 con 0x7fb87809a550 2024-09-16T14:46:04.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8781b6fb0 con 0x7fb87809d1b0 2024-09-16T14:46:04.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:04.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87eab7640 10 monclient(hunting): authenticate will time out at 1188.779265s 2024-09-16T14:46:04.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87d054640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb87809d1b0 0x7fb8781cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43362/0 (socket says 172.21.15.26:43362) 2024-09-16T14:46:04.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb87d054640 1 -- 172.21.15.26:0/4284721812 learned_addr learned my addr 172.21.15.26:0/4284721812 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:04.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb877fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb87809b040 0x7fb8781ca250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43084/0 (socket says 172.21.15.26:43084) 2024-09-16T14:46:04.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572122544 0 0) 0x7fb8781b6fb0 con 0x7fb87809d1b0 2024-09-16T14:46:04.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:04.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:04.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:04.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 10 monclient(hunting): my global_id is 5366 2024-09-16T14:46:04.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb864003530 con 0x7fb87809d1b0 2024-09-16T14:46:04.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811899736 0 0) 0x7fb8781a5d40 con 0x7fb87809b040 2024-09-16T14:46:04.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:04.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:04.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:04.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 10 monclient(hunting): my global_id is 104145 2024-09-16T14:46:04.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8781b6fb0 con 0x7fb87809b040 2024-09-16T14:46:04.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604731453 0 0) 0x7fb8781a4890 con 0x7fb87809a550 2024-09-16T14:46:04.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:04.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:04.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:04.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 10 monclient(hunting): my global_id is 94480 2024-09-16T14:46:04.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8781a5d40 con 0x7fb87809a550 2024-09-16T14:46:04.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3987218945 0 0) 0x7fb864003530 con 0x7fb87809d1b0 2024-09-16T14:46:04.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8781a4890 con 0x7fb87809d1b0 2024-09-16T14:46:04.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb870001c10 con 0x7fb87809d1b0 2024-09-16T14:46:04.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:04.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:04.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 10 monclient(hunting): dump: 2024-09-16T14:46:04.675 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:04.675 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:04.675 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.675 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.675 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:04.676 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:04.676 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:04.676 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:04.676 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:04.676 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:04.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:04.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.656+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2747654914 0 0) 0x7fb8781b6fb0 con 0x7fb87809b040 2024-09-16T14:46:04.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb864003530 con 0x7fb87809b040 2024-09-16T14:46:04.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb86c001c10 con 0x7fb87809b040 2024-09-16T14:46:04.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:04.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:04.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 10 monclient(hunting): dump: 2024-09-16T14:46:04.676 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:04.677 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:04.677 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.677 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.677 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:04.677 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:04.677 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:04.677 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:04.677 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:04.677 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:04.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 131772002 0 0) 0x7fb8781a5d40 con 0x7fb87809a550 2024-09-16T14:46:04.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8781b6fb0 con 0x7fb87809a550 2024-09-16T14:46:04.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb868001920 con 0x7fb87809a550 2024-09-16T14:46:04.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:04.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:04.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 10 monclient(hunting): dump: 2024-09-16T14:46:04.678 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:04.678 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:04.678 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.678 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.678 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:04.678 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:04.678 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:04.678 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:04.678 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:04.678 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:04.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3760454019 0 0) 0x7fb864003530 con 0x7fb87809b040 2024-09-16T14:46:04.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 >> v1:172.21.15.26:6790/0 conn(0x7fb87809d1b0 legacy=0x7fb8781cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:04.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 >> v1:172.21.15.189:6789/0 conn(0x7fb87809a550 legacy=0x7fb8781b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:04.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 10 monclient: _finish_hunting 0 2024-09-16T14:46:04.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 1 monclient: found mon.a 2024-09-16T14:46:04.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:04.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:04.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8780a2640 con 0x7fb87809b040 2024-09-16T14:46:04.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 10 monclient: _finish_auth 0 2024-09-16T14:46:04.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 10 monclient: _check_auth_tickets 2024-09-16T14:46:04.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:04.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb87eab7640 5 monclient: authenticate success, global_id 104145 2024-09-16T14:46:04.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb87eab7640 10 monclient: _renew_subs 2024-09-16T14:46:04.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb87eab7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:04.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb87eab7640 1 -- 172.21.15.26:0/4284721812 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8781a5a60 con 0x7fb87809b040 2024-09-16T14:46:04.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb87eab7640 10 monclient: _renew_subs 2024-09-16T14:46:04.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb87eab7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:04.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb87eab7640 1 -- 172.21.15.26:0/4284721812 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8781a3cc0 con 0x7fb87809b040 2024-09-16T14:46:04.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb86c0039b0 con 0x7fb87809b040 2024-09-16T14:46:04.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:04.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb86c003cc0 con 0x7fb87809b040 2024-09-16T14:46:04.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:04.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:04.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 10 monclient: dump: 2024-09-16T14:46:04.680 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:04.680 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:04.681 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.681 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.681 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:04.681 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:04.681 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:04.681 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:04.681 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:04.681 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:04.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb876ffd640 4 set_mon_vals no callback set 2024-09-16T14:46:04.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb86c003e20 con 0x7fb87809b040 2024-09-16T14:46:04.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(77..77 src has 15..77) v4 ==== 5181+0+0 (unknown 3765235196 0 0) 0x7fb86c043860 con 0x7fb87809b040 2024-09-16T14:46:04.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb87eab7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:04.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb87eab7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:04.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb87eab7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:04.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb87eab7640 1 -- 172.21.15.26:0/4284721812 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb858005050 con 0x7fb87809b040 2024-09-16T14:46:04.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.660+0000 7fb87c853640 10 monclient: get_auth_request con 0x7fb8640364f0 auth_method 0 2024-09-16T14:46:04.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.664+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb86c014660 con 0x7fb87809b040 2024-09-16T14:46:04.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.664+0000 7fb8757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:04.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.664+0000 7fb8757fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:04.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.736+0000 7fd6f7f63640 10 monclient: get_monmap_and_config 2024-09-16T14:46:04.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.736+0000 7fd6f7f63640 10 monclient: build_initial_monmap 2024-09-16T14:46:04.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.736+0000 7fd6f7f63640 10 monclient: monmap: 2024-09-16T14:46:04.742 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:04.742 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:04.742 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:04.741588+0000 2024-09-16T14:46:04.742 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:04.741588+0000 2024-09-16T14:46:04.742 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:04.742 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:04.742 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:04.743 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:04.743 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:04.743 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:04.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 1 Processor -- start 2024-09-16T14:46:04.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 1 -- start start 2024-09-16T14:46:04.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient: init 2024-09-16T14:46:04.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:04.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:04.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient(hunting): picked mon.noname-c con 0x7fd6f009df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:04.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient(hunting): picked mon.noname-b con 0x7fd6f009e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:04.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient(hunting): picked mon.noname-a con 0x7fd6f009e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:04.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f009ee20 con 0x7fd6f009e740 2024-09-16T14:46:04.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f009f100 con 0x7fd6f009df10 2024-09-16T14:46:04.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f009f3e0 con 0x7fd6f009e2f0 2024-09-16T14:46:04.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:04.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient(hunting): authenticate will time out at 1146.857143s 2024-09-16T14:46:04.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f5cff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd6f009df10 0x7fd6f009d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58218/0 (socket says 172.21.15.189:58218) 2024-09-16T14:46:04.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f5cff640 1 -- 172.21.15.189:0/2225635000 learned_addr learned my addr 172.21.15.189:0/2225635000 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:04.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 1 -- 172.21.15.189:0/2225635000 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898541221 0 0) 0x7fd6f009f100 con 0x7fd6f009df10 2024-09-16T14:46:04.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:04.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:04.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:04.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient(hunting): my global_id is 94483 2024-09-16T14:46:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 1 -- 172.21.15.189:0/2225635000 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6dc003530 con 0x7fd6f009df10 2024-09-16T14:46:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 1 -- 172.21.15.189:0/2225635000 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660921877 0 0) 0x7fd6f009f3e0 con 0x7fd6f009e2f0 2024-09-16T14:46:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient(hunting): my global_id is 5369 2024-09-16T14:46:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 1 -- 172.21.15.189:0/2225635000 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6f009f100 con 0x7fd6f009e2f0 2024-09-16T14:46:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 1 -- 172.21.15.189:0/2225635000 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3154893536 0 0) 0x7fd6dc003530 con 0x7fd6f009df10 2024-09-16T14:46:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 1 -- 172.21.15.189:0/2225635000 >> v1:172.21.15.26:6790/0 conn(0x7fd6f009e2f0 legacy=0x7fd6f01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 1 -- 172.21.15.189:0/2225635000 >> v1:172.21.15.26:6789/0 conn(0x7fd6f009e740 legacy=0x7fd6f013b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 1 monclient: found mon.noname-c 2024-09-16T14:46:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 1 -- 172.21.15.189:0/2225635000 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f009c690 con 0x7fd6f009df10 2024-09-16T14:46:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient: _finish_auth 0 2024-09-16T14:46:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 1 -- 172.21.15.189:0/2225635000 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd6e0001620 con 0x7fd6f009df10 2024-09-16T14:46:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 5 monclient: authenticate success, global_id 94483 2024-09-16T14:46:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient: dump: 2024-09-16T14:46:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:04.752 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 1 -- 172.21.15.189:0/2225635000 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6e0001840 con 0x7fd6f009df10 2024-09-16T14:46:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 4 set_mon_vals no callback set 2024-09-16T14:46:04.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 1 -- 172.21.15.189:0/2225635000 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd6e0001bd0 con 0x7fd6f009df10 2024-09-16T14:46:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f4cfd640 10 monclient: dump: 2024-09-16T14:46:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:04.753 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:04.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient: shutdown 2024-09-16T14:46:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 1 -- 172.21.15.189:0/2225635000 >> v1:172.21.15.189:6789/0 conn(0x7fd6f009df10 legacy=0x7fd6f009d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 1 -- 172.21.15.189:0/2225635000 shutdown_connections 2024-09-16T14:46:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 1 -- 172.21.15.189:0/2225635000 >> 172.21.15.189:0/2225635000 conn(0x7fd6f00a3320 msgr2=0x7fd6f009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 1 -- 172.21.15.189:0/2225635000 shutdown_connections 2024-09-16T14:46:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 1 -- 172.21.15.189:0/2225635000 wait complete. 2024-09-16T14:46:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient: build_initial_monmap 2024-09-16T14:46:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient: monmap: 2024-09-16T14:46:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:04.755 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 1 Processor -- start 2024-09-16T14:46:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 1 -- start start 2024-09-16T14:46:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient: init 2024-09-16T14:46:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient(hunting): picked mon.noname-c con 0x7fd6f009df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient(hunting): picked mon.noname-b con 0x7fd6f009e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient(hunting): picked mon.noname-a con 0x7fd6f009e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:04.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f006b380 con 0x7fd6f009e740 2024-09-16T14:46:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f011b940 con 0x7fd6f009e2f0 2024-09-16T14:46:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f013ba30 con 0x7fd6f009df10 2024-09-16T14:46:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.740+0000 7fd6f7f63640 10 monclient(hunting): authenticate will time out at 1146.859953s 2024-09-16T14:46:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6f54fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd6f009e2f0 0x7fd6f01ce990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58230/0 (socket says 172.21.15.189:58230) 2024-09-16T14:46:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6f54fe640 1 -- 172.21.15.189:0/3763104601 learned_addr learned my addr 172.21.15.189:0/3763104601 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 1 -- 172.21.15.189:0/3763104601 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252925549 0 0) 0x7fd6f011b940 con 0x7fd6f009e2f0 2024-09-16T14:46:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 10 monclient(hunting): my global_id is 94486 2024-09-16T14:46:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 1 -- 172.21.15.189:0/3763104601 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6d4003530 con 0x7fd6f009e2f0 2024-09-16T14:46:04.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 1 -- 172.21.15.189:0/3763104601 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2297851232 0 0) 0x7fd6d4003530 con 0x7fd6f009e2f0 2024-09-16T14:46:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 1 -- 172.21.15.189:0/3763104601 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6f011b940 con 0x7fd6f009e2f0 2024-09-16T14:46:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 1 -- 172.21.15.189:0/3763104601 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd6ec001d00 con 0x7fd6f009e2f0 2024-09-16T14:46:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 10 monclient(hunting): dump: 2024-09-16T14:46:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:04.757 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:04.758 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 1 -- 172.21.15.189:0/3763104601 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1412528665 0 0) 0x7fd6f011b940 con 0x7fd6f009e2f0 2024-09-16T14:46:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 1 -- 172.21.15.189:0/3763104601 >> v1:172.21.15.26:6790/0 conn(0x7fd6f009df10 legacy=0x7fd6f01c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 1 -- 172.21.15.189:0/3763104601 >> v1:172.21.15.26:6789/0 conn(0x7fd6f009e740 legacy=0x7fd6f01d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:46:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 1 monclient: found mon.b 2024-09-16T14:46:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 1 -- 172.21.15.189:0/3763104601 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f009aba0 con 0x7fd6f009e2f0 2024-09-16T14:46:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 10 monclient: _finish_auth 0 2024-09-16T14:46:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:46:04.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6f7f63640 5 monclient: authenticate success, global_id 94486 2024-09-16T14:46:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6f7f63640 10 monclient: _renew_subs 2024-09-16T14:46:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6f7f63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6f7f63640 1 -- 172.21.15.189:0/3763104601 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f011b660 con 0x7fd6f009e2f0 2024-09-16T14:46:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6f7f63640 10 monclient: _renew_subs 2024-09-16T14:46:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6f7f63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6f7f63640 1 -- 172.21.15.189:0/3763104601 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f01195e0 con 0x7fd6f009e2f0 2024-09-16T14:46:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 1 -- 172.21.15.189:0/3763104601 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6ec0020b0 con 0x7fd6f009e2f0 2024-09-16T14:46:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 1 -- 172.21.15.189:0/3763104601 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd6ec003df0 con 0x7fd6f009e2f0 2024-09-16T14:46:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e7fff640 4 set_mon_vals no callback set 2024-09-16T14:46:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:04.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 10 monclient: dump: 2024-09-16T14:46:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:04.760 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6f7f63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6f7f63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6f7f63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:04.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6f7f63640 1 -- 172.21.15.189:0/3763104601 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d8005050 con 0x7fd6f009e2f0 2024-09-16T14:46:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 1 -- 172.21.15.189:0/3763104601 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd6ec003f50 con 0x7fd6f009e2f0 2024-09-16T14:46:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6e67fc640 1 -- 172.21.15.189:0/3763104601 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(77..77 src has 15..77) v4 ==== 5181+0+0 (unknown 3765235196 0 0) 0x7fd6ec043be0 con 0x7fd6f009e2f0 2024-09-16T14:46:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.744+0000 7fd6f5cff640 10 monclient: get_auth_request con 0x7fd6d40360f0 auth_method 0 2024-09-16T14:46:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.748+0000 7fd6e67fc640 1 -- 172.21.15.189:0/3763104601 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd6ec014b40 con 0x7fd6f009e2f0 2024-09-16T14:46:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.748+0000 7fd6e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.748+0000 7fd6e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:04.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.880+0000 7fb87eab7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:46:04.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.880+0000 7fb87eab7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:46:04.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.880+0000 7fb87eab7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:04.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.880+0000 7fb87eab7640 1 -- 172.21.15.26:0/4284721812 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb858005880 con 0x7fb87809b040 2024-09-16T14:46:04.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.880+0000 7fb8757fa640 1 -- 172.21.15.26:0/4284721812 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 941500975) 0x7fb86c01cdb0 con 0x7fb87809b040 2024-09-16T14:46:04.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.880+0000 7fb8757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:46:04.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.880+0000 7fb8757fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:04.885 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:46:04.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.884+0000 7fb87eab7640 1 -- 172.21.15.26:0/4284721812 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb8640364f0 legacy=0x7fb8640389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:04.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.884+0000 7fb87eab7640 10 monclient: shutdown 2024-09-16T14:46:04.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.884+0000 7fb87eab7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:04.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.884+0000 7fb87eab7640 1 -- 172.21.15.26:0/4284721812 >> v1:172.21.15.26:6789/0 conn(0x7fb87809b040 legacy=0x7fb8781ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:04.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.884+0000 7fb87eab7640 1 -- 172.21.15.26:0/4284721812 shutdown_connections 2024-09-16T14:46:04.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.884+0000 7fb87eab7640 1 -- 172.21.15.26:0/4284721812 >> 172.21.15.26:0/4284721812 conn(0x7fb87809fb50 msgr2=0x7fb8780a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:04.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.884+0000 7fb87eab7640 1 -- 172.21.15.26:0/4284721812 shutdown_connections 2024-09-16T14:46:04.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:04.884+0000 7fb87eab7640 1 -- 172.21.15.26:0/4284721812 wait complete. 2024-09-16T14:46:04.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.896+0000 7fd6f7f63640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:46:04.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.896+0000 7fd6f7f63640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T14:46:04.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.896+0000 7fd6f7f63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:04.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.896+0000 7fd6f7f63640 1 -- 172.21.15.189:0/3763104601 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fd6d8005b10 con 0x7fd6f009e2f0 2024-09-16T14:46:04.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.896+0000 7fd6e67fc640 1 -- 172.21.15.189:0/3763104601 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v77) v1 ==== 56+0+1273 (unknown 2226722781 0 2699675046) 0x7fd6ec012fb0 con 0x7fd6f009e2f0 2024-09-16T14:46:04.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.896+0000 7fd6e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T14:46:04.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.896+0000 7fd6e67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:04.902 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:46:04.902 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:46:04.902 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:46:04.903 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:46:04.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.984+0000 7fd6f7f63640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:46:04.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.984+0000 7fd6f7f63640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T14:46:04.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.984+0000 7fd6f7f63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:04.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.984+0000 7fd6f7f63640 1 -- 172.21.15.189:0/3763104601 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fd6d80055a0 con 0x7fd6f009e2f0 2024-09-16T14:46:04.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.984+0000 7fd6e67fc640 1 -- 172.21.15.189:0/3763104601 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v77) v1 ==== 56+0+1273 (unknown 2226722781 0 2699675046) 0x7fd6ec014730 con 0x7fd6f009e2f0 2024-09-16T14:46:04.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.984+0000 7fd6e67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T14:46:04.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.984+0000 7fd6e67fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:46:04.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.988+0000 7fd6f7f63640 1 -- 172.21.15.189:0/3763104601 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd6d40360f0 legacy=0x7fd6d40385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:04.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.988+0000 7fd6f7f63640 10 monclient: shutdown 2024-09-16T14:46:04.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.988+0000 7fd6f7f63640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:04.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.988+0000 7fd6f7f63640 1 -- 172.21.15.189:0/3763104601 >> v1:172.21.15.189:6789/0 conn(0x7fd6f009e2f0 legacy=0x7fd6f01ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:04.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.988+0000 7fd6f7f63640 1 -- 172.21.15.189:0/3763104601 shutdown_connections 2024-09-16T14:46:04.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.988+0000 7fd6f7f63640 1 -- 172.21.15.189:0/3763104601 >> 172.21.15.189:0/3763104601 conn(0x7fd6f00a3320 msgr2=0x7fd6f013bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:04.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.988+0000 7fd6f7f63640 1 -- 172.21.15.189:0/3763104601 shutdown_connections 2024-09-16T14:46:04.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:04.988+0000 7fd6f7f63640 1 -- 172.21.15.189:0/3763104601 wait complete. 2024-09-16T14:46:05.004 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd tree-from host1 2024-09-16T14:46:05.004 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:46:05.005 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree-from host1 2024-09-16T14:46:05.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.020+0000 7fb9c02a8640 10 monclient: get_monmap_and_config 2024-09-16T14:46:05.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.020+0000 7fb9c02a8640 10 monclient: build_initial_monmap 2024-09-16T14:46:05.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.020+0000 7fb9c02a8640 10 monclient: monmap: 2024-09-16T14:46:05.025 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:05.025 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.025 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:46:05.024683+0000 2024-09-16T14:46:05.025 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:46:05.024683+0000 2024-09-16T14:46:05.025 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:05.026 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.026 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:05.026 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:05.026 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:05.026 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.020+0000 7fb9c02a8640 1 Processor -- start 2024-09-16T14:46:05.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 1 -- start start 2024-09-16T14:46:05.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient: init 2024-09-16T14:46:05.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:05.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:05.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb9b8134e40 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:05.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb9b8099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:05.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb9b8135220 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:05.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b8099e80 con 0x7fb9b8135220 2024-09-16T14:46:05.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b80719d0 con 0x7fb9b8134e40 2024-09-16T14:46:05.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b809e1e0 con 0x7fb9b8099a70 2024-09-16T14:46:05.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:05.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient(hunting): authenticate will time out at 1189.144769s 2024-09-16T14:46:05.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9be845640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb9b8135220 0x7fb9b813db30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43096/0 (socket says 172.21.15.26:43096) 2024-09-16T14:46:05.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9be845640 1 -- 172.21.15.26:0/2511545916 learned_addr learned my addr 172.21.15.26:0/2511545916 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:05.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 1 -- 172.21.15.26:0/2511545916 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005515693 0 0) 0x7fb9b00014a0 con 0x7fb9b8135220 2024-09-16T14:46:05.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient(hunting): my global_id is 104154 2024-09-16T14:46:05.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 1 -- 172.21.15.26:0/2511545916 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a4003530 con 0x7fb9b8135220 2024-09-16T14:46:05.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 1 -- 172.21.15.26:0/2511545916 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780777165 0 0) 0x7fb9b809e1e0 con 0x7fb9b8099a70 2024-09-16T14:46:05.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient(hunting): my global_id is 5375 2024-09-16T14:46:05.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 1 -- 172.21.15.26:0/2511545916 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b00014a0 con 0x7fb9b8099a70 2024-09-16T14:46:05.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 1 -- 172.21.15.26:0/2511545916 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1908176349 0 0) 0x7fb9b80719d0 con 0x7fb9b8134e40 2024-09-16T14:46:05.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient(hunting): my global_id is 94489 2024-09-16T14:46:05.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 1 -- 172.21.15.26:0/2511545916 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b809e1e0 con 0x7fb9b8134e40 2024-09-16T14:46:05.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 1 -- 172.21.15.26:0/2511545916 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 392256691 0 0) 0x7fb9a4003530 con 0x7fb9b8135220 2024-09-16T14:46:05.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 1 -- 172.21.15.26:0/2511545916 >> v1:172.21.15.26:6790/0 conn(0x7fb9b8099a70 legacy=0x7fb9b80712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 1 -- 172.21.15.26:0/2511545916 >> v1:172.21.15.189:6789/0 conn(0x7fb9b8134e40 legacy=0x7fb9b8070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient: _finish_hunting 0 2024-09-16T14:46:05.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 1 monclient: found mon.noname-a 2024-09-16T14:46:05.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:05.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:46:05.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 1 -- 172.21.15.26:0/2511545916 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b809e7d0 con 0x7fb9b8135220 2024-09-16T14:46:05.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient: _finish_auth 0 2024-09-16T14:46:05.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient: _check_auth_tickets 2024-09-16T14:46:05.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:05.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 1 -- 172.21.15.26:0/2511545916 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9b0001990 con 0x7fb9b8135220 2024-09-16T14:46:05.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:46:05.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient: dump: 2024-09-16T14:46:05.038 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.038 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.038 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.038 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.038 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.038 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.038 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.038 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.038 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.038 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 5 monclient: authenticate success, global_id 104154 2024-09-16T14:46:05.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 1 -- 172.21.15.26:0/2511545916 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9b0001b80 con 0x7fb9b8135220 2024-09-16T14:46:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 1 -- 172.21.15.26:0/2511545916 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9b0001f50 con 0x7fb9b8135220 2024-09-16T14:46:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd042640 10 monclient: dump: 2024-09-16T14:46:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.040 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.040 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.040 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:05.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 4 set_mon_vals no callback set 2024-09-16T14:46:05.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient: shutdown 2024-09-16T14:46:05.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:05.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 1 -- 172.21.15.26:0/2511545916 >> v1:172.21.15.26:6789/0 conn(0x7fb9b8135220 legacy=0x7fb9b813db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 1 -- 172.21.15.26:0/2511545916 shutdown_connections 2024-09-16T14:46:05.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 1 -- 172.21.15.26:0/2511545916 >> 172.21.15.26:0/2511545916 conn(0x7fb9b809fb50 msgr2=0x7fb9b809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:05.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 1 -- 172.21.15.26:0/2511545916 shutdown_connections 2024-09-16T14:46:05.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 1 -- 172.21.15.26:0/2511545916 wait complete. 2024-09-16T14:46:05.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient: build_initial_monmap 2024-09-16T14:46:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient: monmap: 2024-09-16T14:46:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:46:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:46:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:46:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:05.042 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 1 Processor -- start 2024-09-16T14:46:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 1 -- start start 2024-09-16T14:46:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient: init 2024-09-16T14:46:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb9b8099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb9b8134e40 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb9b8135220 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b81240e0 con 0x7fb9b8134e40 2024-09-16T14:46:05.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b8122300 con 0x7fb9b8099a70 2024-09-16T14:46:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b809c1a0 con 0x7fb9b8135220 2024-09-16T14:46:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9c02a8640 10 monclient(hunting): authenticate will time out at 1189.147905s 2024-09-16T14:46:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd843640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb9b8134e40 0x7fb9b81ce980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43108/0 (socket says 172.21.15.26:43108) 2024-09-16T14:46:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.024+0000 7fb9bd843640 1 -- 172.21.15.26:0/2812403554 learned_addr learned my addr 172.21.15.26:0/2812403554 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925946020 0 0) 0x7fb9b81240e0 con 0x7fb9b8134e40 2024-09-16T14:46:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): my global_id is 104157 2024-09-16T14:46:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb99c003530 con 0x7fb9b8134e40 2024-09-16T14:46:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549178980 0 0) 0x7fb9b8122300 con 0x7fb9b8099a70 2024-09-16T14:46:05.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): my global_id is 94492 2024-09-16T14:46:05.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b81240e0 con 0x7fb9b8099a70 2024-09-16T14:46:05.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540249573 0 0) 0x7fb9b809c1a0 con 0x7fb9b8135220 2024-09-16T14:46:05.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): my global_id is 5378 2024-09-16T14:46:05.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b8122300 con 0x7fb9b8135220 2024-09-16T14:46:05.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3798899141 0 0) 0x7fb99c003530 con 0x7fb9b8134e40 2024-09-16T14:46:05.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9b809c1a0 con 0x7fb9b8134e40 2024-09-16T14:46:05.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9a80018d0 con 0x7fb9b8134e40 2024-09-16T14:46:05.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:46:05.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): dump: 2024-09-16T14:46:05.046 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.046 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.046 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.046 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.046 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.046 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.046 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.047 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.047 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.047 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:05.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3223664620 0 0) 0x7fb9b8122300 con 0x7fb9b8135220 2024-09-16T14:46:05.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb99c003530 con 0x7fb9b8135220 2024-09-16T14:46:05.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1319450426 0 0) 0x7fb9b81240e0 con 0x7fb9b8099a70 2024-09-16T14:46:05.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9b8122300 con 0x7fb9b8099a70 2024-09-16T14:46:05.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9b0001cf0 con 0x7fb9b8135220 2024-09-16T14:46:05.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:05.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): dump: 2024-09-16T14:46:05.048 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.048 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.048 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.048 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.048 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.048 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.048 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.048 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.048 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.049 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9b4001840 con 0x7fb9b8099a70 2024-09-16T14:46:05.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:05.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): dump: 2024-09-16T14:46:05.049 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.049 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.049 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.049 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.049 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.049 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.049 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.049 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.050 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.050 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1302271311 0 0) 0x7fb9b809c1a0 con 0x7fb9b8134e40 2024-09-16T14:46:05.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 >> v1:172.21.15.26:6790/0 conn(0x7fb9b8135220 legacy=0x7fb9b81d20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 >> v1:172.21.15.189:6789/0 conn(0x7fb9b8099a70 legacy=0x7fb9b81c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:05.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 monclient: found mon.a 2024-09-16T14:46:05.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:05.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:05.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b8121d80 con 0x7fb9b8134e40 2024-09-16T14:46:05.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient: _finish_auth 0 2024-09-16T14:46:05.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:05.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:05.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9c02a8640 5 monclient: authenticate success, global_id 104157 2024-09-16T14:46:05.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9c02a8640 10 monclient: _renew_subs 2024-09-16T14:46:05.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9c02a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:05.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9c02a8640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b8122950 con 0x7fb9b8134e40 2024-09-16T14:46:05.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9c02a8640 10 monclient: _renew_subs 2024-09-16T14:46:05.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9c02a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:05.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9c02a8640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b8123b20 con 0x7fb9b8134e40 2024-09-16T14:46:05.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9a8003600 con 0x7fb9b8134e40 2024-09-16T14:46:05.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:05.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9a8003910 con 0x7fb9b8134e40 2024-09-16T14:46:05.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:05.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9bc841640 4 set_mon_vals no callback set 2024-09-16T14:46:05.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient: dump: 2024-09-16T14:46:05.052 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.052 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.052 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.053 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.053 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.053 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.053 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.053 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.053 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.053 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9c02a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:05.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9c02a8640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb9a0006590 con 0x7fb9a0002830 2024-09-16T14:46:05.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb9a8003a70 con 0x7fb9b8134e40 2024-09-16T14:46:05.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(77..77 src has 15..77) v4 ==== 5181+0+0 (unknown 3765235196 0 0) 0x7fb9a8043650 con 0x7fb9b8134e40 2024-09-16T14:46:05.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251343013 0 0) 0x7fb9a0006590 con 0x7fb9a0002830 2024-09-16T14:46:05.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient(hunting): my global_id is 104160 2024-09-16T14:46:05.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b809c1a0 con 0x7fb9a0002830 2024-09-16T14:46:05.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9be845640 10 monclient: get_auth_request con 0x7fb99c036540 auth_method 0 2024-09-16T14:46:05.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3329978628 0 0) 0x7fb9b809c1a0 con 0x7fb9a0002830 2024-09-16T14:46:05.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9a0006590 con 0x7fb9a0002830 2024-09-16T14:46:05.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9b4003620 con 0x7fb9a0002830 2024-09-16T14:46:05.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:05.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient: dump: 2024-09-16T14:46:05.055 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.055 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.055 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.055 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.055 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.055 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.056 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.056 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.056 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.056 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1728103547 0 0) 0x7fb9a0006590 con 0x7fb9a0002830 2024-09-16T14:46:05.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9a000a0e0 con 0x7fb9a0002830 2024-09-16T14:46:05.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fb9b40057c0 con 0x7fb9a0002830 2024-09-16T14:46:05.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:05.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:05.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 >> v1:172.21.15.26:6789/0 conn(0x7fb9a0002830 legacy=0x7fb9a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.028+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 >> v1:172.21.15.26:6789/0 conn(0x7fb9a0002830 legacy=0x7fb9a0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:46:05.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9c02a8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:46:05.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9c02a8640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb9a0009d80 con 0x7fb9a00090a0 2024-09-16T14:46:05.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337926424 0 0) 0x7fb9a0009d80 con 0x7fb9a00090a0 2024-09-16T14:46:05.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 10 monclient(hunting): my global_id is 104163 2024-09-16T14:46:05.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a0006590 con 0x7fb9a00090a0 2024-09-16T14:46:05.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2700862455 0 0) 0x7fb9a0006590 con 0x7fb9a00090a0 2024-09-16T14:46:05.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9a0009d80 con 0x7fb9a00090a0 2024-09-16T14:46:05.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9b4003e40 con 0x7fb9a00090a0 2024-09-16T14:46:05.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:05.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 10 monclient: dump: 2024-09-16T14:46:05.060 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.060 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.060 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.060 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.060 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.060 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.060 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.060 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.060 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.060 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1688219892 0 0) 0x7fb9a0009d80 con 0x7fb9a00090a0 2024-09-16T14:46:05.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb9a000c290 con 0x7fb9a00090a0 2024-09-16T14:46:05.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 388883115) 0x7fb9b4005290 con 0x7fb9a00090a0 2024-09-16T14:46:05.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:46:05.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:05.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 >> v1:172.21.15.26:6789/0 conn(0x7fb9a00090a0 legacy=0x7fb9a0009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9aeffd640 1 -- 172.21.15.26:0/2812403554 >> v1:172.21.15.26:6789/0 conn(0x7fb9a00090a0 legacy=0x7fb9a0009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:46:05.061 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:46:05.061 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:46:05.062 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:46:05.062 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:46:05.062 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 38, 2024-09-16T14:46:05.062 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:46:05.062 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:46:05.062 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:46:05.062 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:46:05.062 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:05.062 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:46:05.062 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:46:05.062 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:46:05.062 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:46:05.063 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:05.063 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:05.063 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:05.063 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:05.063 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:05.063 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:05.063 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:05.063 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:05.063 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:05.063 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:05.063 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:05.063 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:46:05.064 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:46:05.064 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:05.064 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:05.064 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:05.064 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:05.064 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:05.064 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:05.064 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:05.064 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:05.064 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:05.064 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:05.064 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.065 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:46:05.065 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:46:05.065 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:46:05.065 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:46:05.065 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:46:05.065 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:46:05.065 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:46:05.065 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:46:05.065 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:46:05.065 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:46:05.065 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:46:05.065 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:46:05.065 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:46:05.065 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:46:05.066 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:46:05.066 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:46:05.066 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:46:05.066 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:05.066 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:05.066 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:05.066 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:05.066 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:05.066 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:05.066 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:05.066 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:05.066 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:05.066 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:05.067 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:05.067 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:46:05.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.067 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:46:05.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.067 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:46:05.067 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:46:05.067 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:05.067 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:05.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.067 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:05.067 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:46:05.067 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:05.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.068 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.068 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:46:05.068 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:46:05.068 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:05.068 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:05.068 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:05.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.068 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:46:05.068 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:46:05.068 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:05.069 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:05.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.069 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:05.069 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:46:05.069 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:05.069 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.069 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.069 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:46:05.069 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:46:05.069 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:05.069 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:05.069 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:05.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.070 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:46:05.070 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:46:05.070 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:05.070 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:05.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.070 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:05.070 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:46:05.070 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:05.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.070 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.070 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:46:05.071 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:46:05.071 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:05.071 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:05.071 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:05.071 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.071 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.071 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:46:05.071 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:46:05.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.071 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:05.071 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:05.072 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:46:05.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.072 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:05.072 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:46:05.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.072 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:05.072 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:05.072 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:46:05.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.072 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.072 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:46:05.072 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:46:05.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9c02a8640 1 -- 172.21.15.26:0/2812403554 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb99c036540 legacy=0x7fb99c038a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9c02a8640 10 monclient: shutdown 2024-09-16T14:46:05.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9c02a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:05.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9c02a8640 1 -- 172.21.15.26:0/2812403554 >> v1:172.21.15.26:6789/0 conn(0x7fb9b8134e40 legacy=0x7fb9b81ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9be845640 1 -- 172.21.15.26:0/2812403554 reap_dead start 2024-09-16T14:46:05.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9be845640 1 -- 172.21.15.26:0/2812403554 reap_dead start 2024-09-16T14:46:05.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9c02a8640 1 -- 172.21.15.26:0/2812403554 shutdown_connections 2024-09-16T14:46:05.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9c02a8640 1 -- 172.21.15.26:0/2812403554 >> 172.21.15.26:0/2812403554 conn(0x7fb9b809fb50 msgr2=0x7fb9b80a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:05.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9c02a8640 1 -- 172.21.15.26:0/2812403554 shutdown_connections 2024-09-16T14:46:05.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.036+0000 7fb9c02a8640 1 -- 172.21.15.26:0/2812403554 wait complete. 2024-09-16T14:46:05.075 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:46:05.075 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:46:05.075 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:46:05.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 10 monclient: get_monmap_and_config 2024-09-16T14:46:05.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 10 monclient: build_initial_monmap 2024-09-16T14:46:05.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 10 monclient: monmap: 2024-09-16T14:46:05.114 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:05.114 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.114 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:05.114025+0000 2024-09-16T14:46:05.114 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:05.114025+0000 2024-09-16T14:46:05.114 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:05.114 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:05.115 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:05.115 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:05.115 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:05.115 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 1 Processor -- start 2024-09-16T14:46:05.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 1 -- start start 2024-09-16T14:46:05.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 10 monclient: init 2024-09-16T14:46:05.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:05.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:46:05.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 10 monclient(hunting): picked mon.noname-a con 0x7feac809df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:05.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 10 monclient(hunting): picked mon.noname-c con 0x7feac809e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:05.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 10 monclient(hunting): picked mon.noname-b con 0x7feac809e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:05.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac809ee20 con 0x7feac809df10 2024-09-16T14:46:05.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac809f100 con 0x7feac809e2f0 2024-09-16T14:46:05.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac809f3e0 con 0x7feac809e740 2024-09-16T14:46:05.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:05.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 10 monclient(hunting): authenticate will time out at 1147.229688s 2024-09-16T14:46:05.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacc865640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7feac809e2f0 0x7feac81378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58236/0 (socket says 172.21.15.189:58236) 2024-09-16T14:46:05.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacc865640 1 -- 172.21.15.189:0/3779934684 learned_addr learned my addr 172.21.15.189:0/3779934684 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:05.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 1 -- 172.21.15.189:0/3779934684 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444236374 0 0) 0x7feac809f100 con 0x7feac809e2f0 2024-09-16T14:46:05.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient(hunting): my global_id is 94495 2024-09-16T14:46:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 1 -- 172.21.15.189:0/3779934684 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab0003530 con 0x7feac809e2f0 2024-09-16T14:46:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 1 -- 172.21.15.189:0/3779934684 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348128713 0 0) 0x7feac809ee20 con 0x7feac809df10 2024-09-16T14:46:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient(hunting): my global_id is 104166 2024-09-16T14:46:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 1 -- 172.21.15.189:0/3779934684 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feac809f100 con 0x7feac809df10 2024-09-16T14:46:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 1 -- 172.21.15.189:0/3779934684 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200712898 0 0) 0x7feac809f3e0 con 0x7feac809e740 2024-09-16T14:46:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient(hunting): my global_id is 5381 2024-09-16T14:46:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 1 -- 172.21.15.189:0/3779934684 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feac809ee20 con 0x7feac809e740 2024-09-16T14:46:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 1 -- 172.21.15.189:0/3779934684 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 611495543 0 0) 0x7feab0003530 con 0x7feac809e2f0 2024-09-16T14:46:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 1 -- 172.21.15.189:0/3779934684 >> v1:172.21.15.26:6790/0 conn(0x7feac809e740 legacy=0x7feac813b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 1 -- 172.21.15.189:0/3779934684 >> v1:172.21.15.26:6789/0 conn(0x7feac809df10 legacy=0x7feac809d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 1 monclient: found mon.noname-c 2024-09-16T14:46:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 1 -- 172.21.15.189:0/3779934684 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac809c690 con 0x7feac809e2f0 2024-09-16T14:46:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient: _finish_auth 0 2024-09-16T14:46:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 1 -- 172.21.15.189:0/3779934684 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feabc001620 con 0x7feac809e2f0 2024-09-16T14:46:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 5 monclient: authenticate success, global_id 94495 2024-09-16T14:46:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient: dump: 2024-09-16T14:46:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.127 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 1 -- 172.21.15.189:0/3779934684 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feabc001900 con 0x7feac809e2f0 2024-09-16T14:46:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 1 -- 172.21.15.189:0/3779934684 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feabc001c90 con 0x7feac809e2f0 2024-09-16T14:46:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 4 set_mon_vals no callback set 2024-09-16T14:46:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feac7fff640 10 monclient: dump: 2024-09-16T14:46:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.128 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 10 monclient: shutdown 2024-09-16T14:46:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 1 -- 172.21.15.189:0/3779934684 >> v1:172.21.15.189:6789/0 conn(0x7feac809e2f0 legacy=0x7feac81378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 1 -- 172.21.15.189:0/3779934684 shutdown_connections 2024-09-16T14:46:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 1 -- 172.21.15.189:0/3779934684 >> 172.21.15.189:0/3779934684 conn(0x7feac80a3320 msgr2=0x7feac809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 1 -- 172.21.15.189:0/3779934684 shutdown_connections 2024-09-16T14:46:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 1 -- 172.21.15.189:0/3779934684 wait complete. 2024-09-16T14:46:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 10 monclient: build_initial_monmap 2024-09-16T14:46:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.112+0000 7feacf2ca640 10 monclient: monmap: 2024-09-16T14:46:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:05.130 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 1 Processor -- start 2024-09-16T14:46:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 1 -- start start 2024-09-16T14:46:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 10 monclient: init 2024-09-16T14:46:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 10 monclient(hunting): picked mon.noname-c con 0x7feac809df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 10 monclient(hunting): picked mon.noname-b con 0x7feac809e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 10 monclient(hunting): picked mon.noname-a con 0x7feac809e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac811a640 con 0x7feac809e740 2024-09-16T14:46:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac813bba0 con 0x7feac809e2f0 2024-09-16T14:46:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac806fab0 con 0x7feac809df10 2024-09-16T14:46:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 10 monclient(hunting): authenticate will time out at 1147.232789s 2024-09-16T14:46:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacc865640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7feac809e2f0 0x7feac813a3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58238/0 (socket says 172.21.15.189:58238) 2024-09-16T14:46:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacc865640 1 -- 172.21.15.189:0/1530591573 learned_addr learned my addr 172.21.15.189:0/1530591573 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982976735 0 0) 0x7feac813bba0 con 0x7feac809e2f0 2024-09-16T14:46:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): my global_id is 94498 2024-09-16T14:46:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab4003530 con 0x7feac809e2f0 2024-09-16T14:46:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428102492 0 0) 0x7feac811a640 con 0x7feac809e740 2024-09-16T14:46:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): my global_id is 104169 2024-09-16T14:46:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feac813bba0 con 0x7feac809e740 2024-09-16T14:46:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310433969 0 0) 0x7feac806fab0 con 0x7feac809df10 2024-09-16T14:46:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): my global_id is 5384 2024-09-16T14:46:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feac811a640 con 0x7feac809df10 2024-09-16T14:46:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1231929942 0 0) 0x7feab4003530 con 0x7feac809e2f0 2024-09-16T14:46:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feac806fab0 con 0x7feac809e2f0 2024-09-16T14:46:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feabc0015e0 con 0x7feac809e2f0 2024-09-16T14:46:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): dump: 2024-09-16T14:46:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.133 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 464972811 0 0) 0x7feac813bba0 con 0x7feac809e740 2024-09-16T14:46:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feab4003530 con 0x7feac809e740 2024-09-16T14:46:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feab80018c0 con 0x7feac809e740 2024-09-16T14:46:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient(hunting): dump: 2024-09-16T14:46:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.135 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1087997046 0 0) 0x7feac811a640 con 0x7feac809df10 2024-09-16T14:46:05.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feac813bba0 con 0x7feac809df10 2024-09-16T14:46:05.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 120432968 0 0) 0x7feac806fab0 con 0x7feac809e2f0 2024-09-16T14:46:05.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 >> v1:172.21.15.26:6790/0 conn(0x7feac809df10 legacy=0x7feac8139cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 >> v1:172.21.15.26:6789/0 conn(0x7feac809e740 legacy=0x7feac81cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:46:05.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 monclient: found mon.b 2024-09-16T14:46:05.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:05.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac806d020 con 0x7feac809e2f0 2024-09-16T14:46:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient: _finish_auth 0 2024-09-16T14:46:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 5 monclient: authenticate success, global_id 94498 2024-09-16T14:46:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 10 monclient: _renew_subs 2024-09-16T14:46:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 1 -- 172.21.15.189:0/1530591573 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac806b380 con 0x7feac809e2f0 2024-09-16T14:46:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 10 monclient: _renew_subs 2024-09-16T14:46:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 1 -- 172.21.15.189:0/1530591573 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feac813b680 con 0x7feac809e2f0 2024-09-16T14:46:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feabc004c00 con 0x7feac809e2f0 2024-09-16T14:46:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feabc004f10 con 0x7feac809e2f0 2024-09-16T14:46:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac77fe640 4 set_mon_vals no callback set 2024-09-16T14:46:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 10 monclient: dump: 2024-09-16T14:46:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.138 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7feabc00f980 con 0x7feac809e2f0 2024-09-16T14:46:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacf2ca640 1 -- 172.21.15.189:0/1530591573 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa8005050 con 0x7feac809e2f0 2024-09-16T14:46:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(77..77 src has 15..77) v4 ==== 5181+0+0 (unknown 3765235196 0 0) 0x7feabc0439b0 con 0x7feac809e2f0 2024-09-16T14:46:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.116+0000 7feacd066640 10 monclient: get_auth_request con 0x7feab4036450 auth_method 0 2024-09-16T14:46:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.120+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7feabc014740 con 0x7feac809e2f0 2024-09-16T14:46:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.120+0000 7feac5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.120+0000 7feac5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:05.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 10 monclient: get_monmap_and_config 2024-09-16T14:46:05.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 10 monclient: build_initial_monmap 2024-09-16T14:46:05.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 10 monclient: monmap: 2024-09-16T14:46:05.186 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:05.186 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.186 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:46:05.185793+0000 2024-09-16T14:46:05.186 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:46:05.185793+0000 2024-09-16T14:46:05.186 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:05.186 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.187 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:05.187 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:05.187 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:05.187 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 1 Processor -- start 2024-09-16T14:46:05.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 1 -- start start 2024-09-16T14:46:05.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 10 monclient: init 2024-09-16T14:46:05.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:05.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:05.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 10 monclient(hunting): picked mon.noname-a con 0x7fa2a8096600 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:05.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 10 monclient(hunting): picked mon.noname-b con 0x7fa2a809d580 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:05.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 10 monclient(hunting): picked mon.noname-c con 0x7fa2a8137600 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:05.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a809bed0 con 0x7fa2a8096600 2024-09-16T14:46:05.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a809cc00 con 0x7fa2a8137600 2024-09-16T14:46:05.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a809f7d0 con 0x7fa2a809d580 2024-09-16T14:46:05.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:05.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 10 monclient(hunting): authenticate will time out at 1189.305845s 2024-09-16T14:46:05.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a5d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa2a809d580 0x7fa2a8135ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43390/0 (socket says 172.21.15.26:43390) 2024-09-16T14:46:05.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a5d9b640 1 -- 172.21.15.26:0/3289932291 learned_addr learned my addr 172.21.15.26:0/3289932291 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:05.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 1 -- 172.21.15.26:0/3289932291 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258217544 0 0) 0x7fa2a809f7d0 con 0x7fa2a809d580 2024-09-16T14:46:05.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient(hunting): my global_id is 5387 2024-09-16T14:46:05.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 1 -- 172.21.15.26:0/3289932291 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa290003530 con 0x7fa2a809d580 2024-09-16T14:46:05.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 1 -- 172.21.15.26:0/3289932291 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742871974 0 0) 0x7fa2a809bed0 con 0x7fa2a8096600 2024-09-16T14:46:05.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient(hunting): my global_id is 104172 2024-09-16T14:46:05.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 1 -- 172.21.15.26:0/3289932291 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2a809f7d0 con 0x7fa2a8096600 2024-09-16T14:46:05.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 1 -- 172.21.15.26:0/3289932291 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4071542785 0 0) 0x7fa290003530 con 0x7fa2a809d580 2024-09-16T14:46:05.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 1 -- 172.21.15.26:0/3289932291 >> v1:172.21.15.189:6789/0 conn(0x7fa2a8137600 legacy=0x7fa2a81399f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 1 -- 172.21.15.26:0/3289932291 >> v1:172.21.15.26:6789/0 conn(0x7fa2a8096600 legacy=0x7fa2a80969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient: _finish_hunting 0 2024-09-16T14:46:05.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 1 monclient: found mon.noname-b 2024-09-16T14:46:05.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:05.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:46:05.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 1 -- 172.21.15.26:0/3289932291 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a809c2f0 con 0x7fa2a809d580 2024-09-16T14:46:05.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient: _finish_auth 0 2024-09-16T14:46:05.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient: _check_auth_tickets 2024-09-16T14:46:05.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:05.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 1 -- 172.21.15.26:0/3289932291 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa29c001620 con 0x7fa2a809d580 2024-09-16T14:46:05.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 5 monclient: authenticate success, global_id 5387 2024-09-16T14:46:05.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:05.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:05.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient: dump: 2024-09-16T14:46:05.194 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.194 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.194 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.194 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.194 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.194 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.195 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 1 -- 172.21.15.26:0/3289932291 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa29c001900 con 0x7fa2a809d580 2024-09-16T14:46:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 1 -- 172.21.15.26:0/3289932291 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa29c001cd0 con 0x7fa2a809d580 2024-09-16T14:46:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2a559a640 10 monclient: dump: 2024-09-16T14:46:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.196 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.196 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.196 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.196 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.196 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.196 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.196 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:05.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 4 set_mon_vals no callback set 2024-09-16T14:46:05.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 10 monclient: shutdown 2024-09-16T14:46:05.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:05.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 1 -- 172.21.15.26:0/3289932291 >> v1:172.21.15.26:6790/0 conn(0x7fa2a809d580 legacy=0x7fa2a8135ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 1 -- 172.21.15.26:0/3289932291 shutdown_connections 2024-09-16T14:46:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 1 -- 172.21.15.26:0/3289932291 >> 172.21.15.26:0/3289932291 conn(0x7fa2a80a0360 msgr2=0x7fa2a80a0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 1 -- 172.21.15.26:0/3289932291 shutdown_connections 2024-09-16T14:46:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.184+0000 7fa2acd34640 1 -- 172.21.15.26:0/3289932291 wait complete. 2024-09-16T14:46:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 10 monclient: build_initial_monmap 2024-09-16T14:46:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 10 monclient: monmap: 2024-09-16T14:46:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:46:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:46:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:46:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:05.197 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:05.198 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:05.198 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 1 Processor -- start 2024-09-16T14:46:05.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 1 -- start start 2024-09-16T14:46:05.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 10 monclient: init 2024-09-16T14:46:05.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:05.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:46:05.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 10 monclient(hunting): picked mon.noname-a con 0x7fa2a8096600 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:05.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 10 monclient(hunting): picked mon.noname-b con 0x7fa2a809d580 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:05.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 10 monclient(hunting): picked mon.noname-c con 0x7fa2a8137600 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:05.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a808ce90 con 0x7fa2a8096600 2024-09-16T14:46:05.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a808b700 con 0x7fa2a809d580 2024-09-16T14:46:05.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a8094c00 con 0x7fa2a8137600 2024-09-16T14:46:05.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:05.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 10 monclient(hunting): authenticate will time out at 1189.309456s 2024-09-16T14:46:05.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2a659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa2a8096600 0x7fa2a8093640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43144/0 (socket says 172.21.15.26:43144) 2024-09-16T14:46:05.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2a659c640 1 -- 172.21.15.26:0/1398470693 learned_addr learned my addr 172.21.15.26:0/1398470693 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:05.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185746159 0 0) 0x7fa2a808ce90 con 0x7fa2a8096600 2024-09-16T14:46:05.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): my global_id is 104175 2024-09-16T14:46:05.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa288003530 con 0x7fa2a8096600 2024-09-16T14:46:05.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674630710 0 0) 0x7fa2a8094c00 con 0x7fa2a8137600 2024-09-16T14:46:05.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): my global_id is 5390 2024-09-16T14:46:05.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2a808ce90 con 0x7fa2a8137600 2024-09-16T14:46:05.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 708389432 0 0) 0x7fa288003530 con 0x7fa2a8096600 2024-09-16T14:46:05.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2a8094c00 con 0x7fa2a8096600 2024-09-16T14:46:05.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa2a0001ca0 con 0x7fa2a8096600 2024-09-16T14:46:05.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:46:05.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): dump: 2024-09-16T14:46:05.206 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.206 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.206 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.206 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.206 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.206 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.206 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.206 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.206 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.206 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:05.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062340602 0 0) 0x7fa2a808b700 con 0x7fa2a809d580 2024-09-16T14:46:05.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): my global_id is 94504 2024-09-16T14:46:05.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa288003530 con 0x7fa2a809d580 2024-09-16T14:46:05.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 538683211 0 0) 0x7fa2a808ce90 con 0x7fa2a8137600 2024-09-16T14:46:05.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2a808b700 con 0x7fa2a8137600 2024-09-16T14:46:05.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa298001780 con 0x7fa2a8137600 2024-09-16T14:46:05.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:05.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): dump: 2024-09-16T14:46:05.207 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.208 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.208 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.208 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.208 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.208 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.208 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.208 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.208 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.208 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1816225051 0 0) 0x7fa2a8094c00 con 0x7fa2a8096600 2024-09-16T14:46:05.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 >> v1:172.21.15.26:6790/0 conn(0x7fa2a8137600 legacy=0x7fa2a81cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 >> v1:172.21.15.189:6789/0 conn(0x7fa2a809d580 legacy=0x7fa2a81c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:05.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 monclient: found mon.a 2024-09-16T14:46:05.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:05.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:05.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a809a880 con 0x7fa2a8096600 2024-09-16T14:46:05.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient: _finish_auth 0 2024-09-16T14:46:05.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:05.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:05.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 5 monclient: authenticate success, global_id 104175 2024-09-16T14:46:05.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 10 monclient: _renew_subs 2024-09-16T14:46:05.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:05.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 1 -- 172.21.15.26:0/1398470693 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a8093d80 con 0x7fa2a8096600 2024-09-16T14:46:05.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 10 monclient: _renew_subs 2024-09-16T14:46:05.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:05.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 1 -- 172.21.15.26:0/1398470693 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a8137300 con 0x7fa2a8096600 2024-09-16T14:46:05.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2a0002030 con 0x7fa2a8096600 2024-09-16T14:46:05.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:05.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa2a0003df0 con 0x7fa2a8096600 2024-09-16T14:46:05.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:05.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient: dump: 2024-09-16T14:46:05.210 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.210 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.210 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.210 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.210 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.211 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.211 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.211 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.211 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.211 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2a4d99640 4 set_mon_vals no callback set 2024-09-16T14:46:05.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:05.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2acd34640 1 -- 172.21.15.26:0/1398470693 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa28c006590 con 0x7fa28c002830 2024-09-16T14:46:05.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa2a0003f50 con 0x7fa2a8096600 2024-09-16T14:46:05.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(77..77 src has 15..77) v4 ==== 5181+0+0 (unknown 3765235196 0 0) 0x7fa2a0043b40 con 0x7fa2a8096600 2024-09-16T14:46:05.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998090819 0 0) 0x7fa28c006590 con 0x7fa28c002830 2024-09-16T14:46:05.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient(hunting): my global_id is 5393 2024-09-16T14:46:05.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2a8094c00 con 0x7fa28c002830 2024-09-16T14:46:05.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3095797366 0 0) 0x7fa2a8094c00 con 0x7fa28c002830 2024-09-16T14:46:05.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa28c006590 con 0x7fa28c002830 2024-09-16T14:46:05.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa29c0016e0 con 0x7fa28c002830 2024-09-16T14:46:05.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:05.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 10 monclient: dump: 2024-09-16T14:46:05.212 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.212 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.212 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.212 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.213 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.213 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.213 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.213 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.213 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.213 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa2a6d9d640 10 monclient: get_auth_request con 0x7fa288038650 auth_method 0 2024-09-16T14:46:05.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2508263268 0 0) 0x7fa28c006590 con 0x7fa28c002830 2024-09-16T14:46:05.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.188+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa28c00a0e0 con 0x7fa28c002830 2024-09-16T14:46:05.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.192+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa29c004950 con 0x7fa28c002830 2024-09-16T14:46:05.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.192+0000 7fa296ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:05.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.192+0000 7fa296ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:05.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.192+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 >> v1:172.21.15.26:6790/0 conn(0x7fa28c002830 legacy=0x7fa28c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.192+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 >> v1:172.21.15.26:6790/0 conn(0x7fa28c002830 legacy=0x7fa28c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:46:05.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa2acd34640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:46:05.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa2acd34640 1 -- 172.21.15.26:0/1398470693 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa28c009d80 con 0x7fa28c0090a0 2024-09-16T14:46:05.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620812781 0 0) 0x7fa28c009d80 con 0x7fa28c0090a0 2024-09-16T14:46:05.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 10 monclient(hunting): my global_id is 5396 2024-09-16T14:46:05.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa28c006590 con 0x7fa28c0090a0 2024-09-16T14:46:05.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 757330713 0 0) 0x7fa28c006590 con 0x7fa28c0090a0 2024-09-16T14:46:05.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa28c009d80 con 0x7fa28c0090a0 2024-09-16T14:46:05.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa29c004e50 con 0x7fa28c0090a0 2024-09-16T14:46:05.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:05.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 10 monclient: dump: 2024-09-16T14:46:05.215 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.215 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.215 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.215 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.215 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.215 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.215 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.215 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.215 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.216 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3752665893 0 0) 0x7fa28c009d80 con 0x7fa28c0090a0 2024-09-16T14:46:05.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa28c00c290 con 0x7fa28c0090a0 2024-09-16T14:46:05.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 2226253094) 0x7fa29c007050 con 0x7fa28c0090a0 2024-09-16T14:46:05.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:46:05.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:05.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 >> v1:172.21.15.26:6790/0 conn(0x7fa28c0090a0 legacy=0x7fa28c009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa296ffd640 1 -- 172.21.15.26:0/1398470693 >> v1:172.21.15.26:6790/0 conn(0x7fa28c0090a0 legacy=0x7fa28c009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:46:05.217 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:46:05.217 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:46:05.217 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:46:05.217 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:46:05.217 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 38, 2024-09-16T14:46:05.218 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:46:05.218 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:46:05.220 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:46:05.220 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:46:05.220 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:05.220 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:46:05.220 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:46:05.220 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:46:05.220 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:46:05.220 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:05.220 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:05.220 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:05.220 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:46:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:46:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:05.221 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:05.222 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:05.222 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:05.222 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:05.222 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:05.222 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:05.222 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:05.222 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.222 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:46:05.222 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:46:05.222 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:46:05.222 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:46:05.223 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:46:05.223 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:46:05.223 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:46:05.223 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:46:05.223 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:46:05.223 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:46:05.223 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:46:05.223 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:46:05.223 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:46:05.223 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:46:05.223 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:46:05.223 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:46:05.224 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:46:05.224 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:05.224 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:05.224 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:05.224 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:05.224 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:05.224 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:05.224 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:05.224 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:05.224 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:05.224 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:05.224 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:05.224 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:46:05.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.225 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:46:05.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.225 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:46:05.225 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:46:05.225 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:05.225 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:05.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.225 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:05.225 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:46:05.225 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:05.225 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.225 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.226 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:46:05.226 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:46:05.226 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:05.226 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:05.226 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:05.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.226 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:46:05.226 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:46:05.226 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:05.226 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:05.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.226 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:05.227 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:46:05.227 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:05.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.227 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.227 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:46:05.227 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:46:05.227 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:05.227 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:05.227 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:05.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.227 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:46:05.228 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:46:05.228 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:05.228 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:05.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.228 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:05.228 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:46:05.228 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:05.228 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.228 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.228 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:46:05.228 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:46:05.228 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:05.229 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:05.229 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:05.229 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.229 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.229 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:46:05.229 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:46:05.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.229 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:05.229 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:05.229 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:46:05.229 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.229 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:05.230 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:46:05.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.230 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:05.230 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:05.230 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:46:05.230 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.230 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:05.230 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:46:05.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.230 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:05.230 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:05.230 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:46:05.230 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.231 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:05.231 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:46:05.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.231 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:05.231 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:05.231 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:46:05.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.231 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.231 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:46:05.231 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:46:05.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa2acd34640 1 -- 172.21.15.26:0/1398470693 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa288038650 legacy=0x7fa28803ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa2acd34640 10 monclient: shutdown 2024-09-16T14:46:05.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa2acd34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:05.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa2acd34640 1 -- 172.21.15.26:0/1398470693 >> v1:172.21.15.26:6789/0 conn(0x7fa2a8096600 legacy=0x7fa2a8093640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa2a6d9d640 1 -- 172.21.15.26:0/1398470693 reap_dead start 2024-09-16T14:46:05.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa2a6d9d640 1 -- 172.21.15.26:0/1398470693 reap_dead start 2024-09-16T14:46:05.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa2acd34640 1 -- 172.21.15.26:0/1398470693 shutdown_connections 2024-09-16T14:46:05.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa2acd34640 1 -- 172.21.15.26:0/1398470693 >> 172.21.15.26:0/1398470693 conn(0x7fa2a80a0360 msgr2=0x7fa2a80995a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:05.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa2acd34640 1 -- 172.21.15.26:0/1398470693 shutdown_connections 2024-09-16T14:46:05.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.196+0000 7fa2acd34640 1 -- 172.21.15.26:0/1398470693 wait complete. 2024-09-16T14:46:05.234 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:46:05.234 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:46:05.234 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:46:05.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.280+0000 7feacf2ca640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-16T14:46:05.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.280+0000 7feacf2ca640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-16T14:46:05.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.280+0000 7feacf2ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.280+0000 7feacf2ca640 1 -- 172.21.15.189:0/1530591573 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree-from", "bucket": "host1"} v 0) v1 -- 0x7feaa8005300 con 0x7feac809e2f0 2024-09-16T14:46:05.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.280+0000 7feac5ffb640 1 -- 172.21.15.189:0/1530591573 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host1"}]=-2 bucket 'host1' does not exist v77) v1 ==== 109+0+0 (unknown 1570759592 0 0) 0x7feabc012de0 con 0x7feac809e2f0 2024-09-16T14:46:05.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.280+0000 7feac5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-16T14:46:05.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.280+0000 7feac5ffb640 10 monclient: _finish_command 2 = mon:2 bucket 'host1' does not exist 2024-09-16T14:46:05.284 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOENT: bucket 'host1' does not exist 2024-09-16T14:46:05.288 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:46:05.288 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree-from host2 2024-09-16T14:46:05.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.352+0000 7f6e6e374640 10 monclient: get_monmap_and_config 2024-09-16T14:46:05.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.352+0000 7f6e6e374640 10 monclient: build_initial_monmap 2024-09-16T14:46:05.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.352+0000 7f6e6e374640 10 monclient: monmap: 2024-09-16T14:46:05.357 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:05.357 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.357 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:46:05.356300+0000 2024-09-16T14:46:05.357 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:46:05.356300+0000 2024-09-16T14:46:05.357 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:05.357 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.357 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:05.357 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:05.357 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:05.357 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.352+0000 7f6e6e374640 1 Processor -- start 2024-09-16T14:46:05.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.352+0000 7f6e6e374640 1 -- start start 2024-09-16T14:46:05.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.352+0000 7f6e6e374640 10 monclient: init 2024-09-16T14:46:05.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:05.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:05.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient(hunting): picked mon.noname-c con 0x7f6e6809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:05.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient(hunting): picked mon.noname-a con 0x7f6e68132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:05.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient(hunting): picked mon.noname-b con 0x7f6e68138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:05.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6809d260 con 0x7f6e68132650 2024-09-16T14:46:05.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e68132a80 con 0x7f6e6809cb80 2024-09-16T14:46:05.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6809cf60 con 0x7f6e68138ff0 2024-09-16T14:46:05.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:05.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient(hunting): authenticate will time out at 1189.476359s 2024-09-16T14:46:05.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e677fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6e68132650 0x7f6e681378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43158/0 (socket says 172.21.15.26:43158) 2024-09-16T14:46:05.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e677fe640 1 -- 172.21.15.26:0/4191652865 learned_addr learned my addr 172.21.15.26:0/4191652865 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:05.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 1 -- 172.21.15.26:0/4191652865 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110353099 0 0) 0x7f6e6809cf60 con 0x7f6e68138ff0 2024-09-16T14:46:05.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient(hunting): my global_id is 5399 2024-09-16T14:46:05.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 1 -- 172.21.15.26:0/4191652865 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e54003530 con 0x7f6e68138ff0 2024-09-16T14:46:05.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 1 -- 172.21.15.26:0/4191652865 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727151056 0 0) 0x7f6e6809d260 con 0x7f6e68132650 2024-09-16T14:46:05.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient(hunting): my global_id is 104178 2024-09-16T14:46:05.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 1 -- 172.21.15.26:0/4191652865 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e6809cf60 con 0x7f6e68132650 2024-09-16T14:46:05.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 1 -- 172.21.15.26:0/4191652865 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 581795795 0 0) 0x7f6e54003530 con 0x7f6e68138ff0 2024-09-16T14:46:05.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 1 -- 172.21.15.26:0/4191652865 >> v1:172.21.15.189:6789/0 conn(0x7f6e6809cb80 legacy=0x7f6e680717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 1 -- 172.21.15.26:0/4191652865 >> v1:172.21.15.26:6789/0 conn(0x7f6e68132650 legacy=0x7f6e681378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:05.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 1 monclient: found mon.noname-b 2024-09-16T14:46:05.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:05.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:46:05.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 1 -- 172.21.15.26:0/4191652865 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6809f2d0 con 0x7f6e68138ff0 2024-09-16T14:46:05.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient: _finish_auth 0 2024-09-16T14:46:05.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:05.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:05.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 1 -- 172.21.15.26:0/4191652865 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e58001660 con 0x7f6e68138ff0 2024-09-16T14:46:05.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:05.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient: dump: 2024-09-16T14:46:05.369 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.369 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.369 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.369 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.369 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.370 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.370 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.370 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.370 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.370 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 5 monclient: authenticate success, global_id 5399 2024-09-16T14:46:05.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:05.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 1 -- 172.21.15.26:0/4191652865 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e58001880 con 0x7f6e68138ff0 2024-09-16T14:46:05.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:05.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 1 -- 172.21.15.26:0/4191652865 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e58001bd0 con 0x7f6e68138ff0 2024-09-16T14:46:05.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:05.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 4 set_mon_vals no callback set 2024-09-16T14:46:05.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:05.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e66ffd640 10 monclient: dump: 2024-09-16T14:46:05.371 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.371 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.371 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.371 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.371 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.371 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.371 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.371 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.372 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.372 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient: shutdown 2024-09-16T14:46:05.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:05.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 1 -- 172.21.15.26:0/4191652865 >> v1:172.21.15.26:6790/0 conn(0x7f6e68138ff0 legacy=0x7f6e6813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 1 -- 172.21.15.26:0/4191652865 shutdown_connections 2024-09-16T14:46:05.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 1 -- 172.21.15.26:0/4191652865 >> 172.21.15.26:0/4191652865 conn(0x7f6e6809fb50 msgr2=0x7f6e6809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:05.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 1 -- 172.21.15.26:0/4191652865 shutdown_connections 2024-09-16T14:46:05.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 1 -- 172.21.15.26:0/4191652865 wait complete. 2024-09-16T14:46:05.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient: build_initial_monmap 2024-09-16T14:46:05.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient: monmap: 2024-09-16T14:46:05.372 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:05.373 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:05.373 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:46:05.373 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:46:05.373 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:05.373 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:46:05.373 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:05.373 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:05.373 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:05.373 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 1 Processor -- start 2024-09-16T14:46:05.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 1 -- start start 2024-09-16T14:46:05.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient: init 2024-09-16T14:46:05.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:05.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:05.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient(hunting): picked mon.noname-b con 0x7f6e6809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:05.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient(hunting): picked mon.noname-c con 0x7f6e68132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:05.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient(hunting): picked mon.noname-a con 0x7f6e68138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:05.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6809b370 con 0x7f6e68138ff0 2024-09-16T14:46:05.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6809a240 con 0x7f6e6809cb80 2024-09-16T14:46:05.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e68071b50 con 0x7f6e68132650 2024-09-16T14:46:05.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:05.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e6e374640 10 monclient(hunting): authenticate will time out at 1189.479687s 2024-09-16T14:46:05.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e677fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6e68132650 0x7f6e6809ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43432/0 (socket says 172.21.15.26:43432) 2024-09-16T14:46:05.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e677fe640 1 -- 172.21.15.26:0/1214697994 learned_addr learned my addr 172.21.15.26:0/1214697994 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:05.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.356+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237680299 0 0) 0x7f6e68071b50 con 0x7f6e68132650 2024-09-16T14:46:05.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): my global_id is 5402 2024-09-16T14:46:05.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e48003530 con 0x7f6e68132650 2024-09-16T14:46:05.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844866219 0 0) 0x7f6e6809b370 con 0x7f6e68138ff0 2024-09-16T14:46:05.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): my global_id is 104181 2024-09-16T14:46:05.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e68071b50 con 0x7f6e68138ff0 2024-09-16T14:46:05.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2244676250 0 0) 0x7f6e48003530 con 0x7f6e68132650 2024-09-16T14:46:05.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e6809b370 con 0x7f6e68132650 2024-09-16T14:46:05.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e500019d0 con 0x7f6e68132650 2024-09-16T14:46:05.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:05.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): dump: 2024-09-16T14:46:05.377 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.377 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.377 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.377 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.377 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.377 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.377 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.377 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.377 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.377 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:05.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1380862448 0 0) 0x7f6e68071b50 con 0x7f6e68138ff0 2024-09-16T14:46:05.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e48003530 con 0x7f6e68138ff0 2024-09-16T14:46:05.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e58001560 con 0x7f6e68138ff0 2024-09-16T14:46:05.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:05.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): dump: 2024-09-16T14:46:05.378 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.378 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.378 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.378 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.378 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.378 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.379 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.379 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.379 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.379 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156520380 0 0) 0x7f6e6809a240 con 0x7f6e6809cb80 2024-09-16T14:46:05.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): my global_id is 94510 2024-09-16T14:46:05.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e68071b50 con 0x7f6e6809cb80 2024-09-16T14:46:05.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1422005335 0 0) 0x7f6e6809b370 con 0x7f6e68132650 2024-09-16T14:46:05.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 >> v1:172.21.15.189:6789/0 conn(0x7f6e6809cb80 legacy=0x7f6e6809a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 >> v1:172.21.15.26:6789/0 conn(0x7f6e68138ff0 legacy=0x7f6e681d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:46:05.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 monclient: found mon.c 2024-09-16T14:46:05.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:05.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:46:05.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6809c340 con 0x7f6e68132650 2024-09-16T14:46:05.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient: _finish_auth 0 2024-09-16T14:46:05.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:46:05.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:05.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e6e374640 5 monclient: authenticate success, global_id 5402 2024-09-16T14:46:05.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e6e374640 10 monclient: _renew_subs 2024-09-16T14:46:05.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e6e374640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:46:05.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e6e374640 1 -- 172.21.15.26:0/1214697994 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e68138730 con 0x7f6e68132650 2024-09-16T14:46:05.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e6e374640 10 monclient: _renew_subs 2024-09-16T14:46:05.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e6e374640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:46:05.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e6e374640 1 -- 172.21.15.26:0/1214697994 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6806da20 con 0x7f6e68132650 2024-09-16T14:46:05.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e500047f0 con 0x7f6e68132650 2024-09-16T14:46:05.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:05.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e50004b00 con 0x7f6e68132650 2024-09-16T14:46:05.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:46:05.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient: dump: 2024-09-16T14:46:05.381 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.381 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.381 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.381 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.382 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.382 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.382 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.382 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.382 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.382 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e667fc640 4 set_mon_vals no callback set 2024-09-16T14:46:05.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6e5000f570 con 0x7f6e68132650 2024-09-16T14:46:05.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e6e374640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:05.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e6e374640 1 -- 172.21.15.26:0/1214697994 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6e4c006590 con 0x7f6e4c002830 2024-09-16T14:46:05.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(77..77 src has 15..77) v4 ==== 5181+0+0 (unknown 3765235196 0 0) 0x7f6e50043790 con 0x7f6e68132650 2024-09-16T14:46:05.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e6c911640 10 monclient: get_auth_request con 0x7f6e480363d0 auth_method 0 2024-09-16T14:46:05.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174005442 0 0) 0x7f6e4c006590 con 0x7f6e4c002830 2024-09-16T14:46:05.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient(hunting): my global_id is 94513 2024-09-16T14:46:05.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e6809b370 con 0x7f6e4c002830 2024-09-16T14:46:05.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1132700362 0 0) 0x7f6e6809b370 con 0x7f6e4c002830 2024-09-16T14:46:05.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e4c006590 con 0x7f6e4c002830 2024-09-16T14:46:05.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e60001a70 con 0x7f6e4c002830 2024-09-16T14:46:05.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:05.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient: dump: 2024-09-16T14:46:05.383 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.384 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.384 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.384 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.384 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.384 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.384 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.384 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.384 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.384 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2111489605 0 0) 0x7f6e4c006590 con 0x7f6e4c002830 2024-09-16T14:46:05.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e4c00a0e0 con 0x7f6e4c002830 2024-09-16T14:46:05.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6e60002050 con 0x7f6e4c002830 2024-09-16T14:46:05.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:05.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:05.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 >> v1:172.21.15.189:6789/0 conn(0x7f6e4c002830 legacy=0x7f6e4c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.360+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 >> v1:172.21.15.189:6789/0 conn(0x7f6e4c002830 legacy=0x7f6e4c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:46:05.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e6e374640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:46:05.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e6e374640 1 -- 172.21.15.26:0/1214697994 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6e4c009d80 con 0x7f6e4c0090a0 2024-09-16T14:46:05.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171426286 0 0) 0x7f6e4c009d80 con 0x7f6e4c0090a0 2024-09-16T14:46:05.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 10 monclient(hunting): my global_id is 94516 2024-09-16T14:46:05.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e4c006590 con 0x7f6e4c0090a0 2024-09-16T14:46:05.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3256622196 0 0) 0x7f6e4c006590 con 0x7f6e4c0090a0 2024-09-16T14:46:05.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e4c009d80 con 0x7f6e4c0090a0 2024-09-16T14:46:05.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e60003d10 con 0x7f6e4c0090a0 2024-09-16T14:46:05.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:05.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 10 monclient: dump: 2024-09-16T14:46:05.387 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:05.387 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.387 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.387 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.387 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.387 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:05.387 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.388 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.388 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.388 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:05.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1661761708 0 0) 0x7f6e4c009d80 con 0x7f6e4c0090a0 2024-09-16T14:46:05.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6e4c00c290 con 0x7f6e4c0090a0 2024-09-16T14:46:05.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 320073690) 0x7f6e60004f30 con 0x7f6e4c0090a0 2024-09-16T14:46:05.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:46:05.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:05.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 >> v1:172.21.15.189:6789/0 conn(0x7f6e4c0090a0 legacy=0x7f6e4c009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e64ff9640 1 -- 172.21.15.26:0/1214697994 >> v1:172.21.15.189:6789/0 conn(0x7f6e4c0090a0 legacy=0x7f6e4c009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:46:05.389 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:46:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:46:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:46:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:46:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 38, 2024-09-16T14:46:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:46:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:46:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:46:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:46:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:46:05.389 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:46:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:46:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:46:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:05.390 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:46:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:46:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:46:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:46:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:46:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:46:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:46:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:46:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:46:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:46:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:46:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:46:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:46:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:46:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:46:05.392 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:46:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:46:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:46:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:46:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:05.393 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:46:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:46:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:46:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:46:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:46:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:05.394 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:46:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:46:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:05.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:46:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:46:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:05.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:46:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:46:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:46:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:46:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:46:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:05.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:46:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:46:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:46:05.399 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:46:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:46:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:46:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.400 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:46:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:46:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:05.401 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:46:05.401 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:46:05.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e6e374640 1 -- 172.21.15.26:0/1214697994 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6e480363d0 legacy=0x7f6e48038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e6e374640 10 monclient: shutdown 2024-09-16T14:46:05.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e6e374640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:05.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e6e374640 1 -- 172.21.15.26:0/1214697994 >> v1:172.21.15.26:6790/0 conn(0x7f6e68132650 legacy=0x7f6e6809ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.368+0000 7f6e6c911640 1 -- 172.21.15.26:0/1214697994 reap_dead start 2024-09-16T14:46:05.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.372+0000 7f6e6c911640 1 -- 172.21.15.26:0/1214697994 reap_dead start 2024-09-16T14:46:05.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.372+0000 7f6e6e374640 1 -- 172.21.15.26:0/1214697994 shutdown_connections 2024-09-16T14:46:05.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.372+0000 7f6e6e374640 1 -- 172.21.15.26:0/1214697994 >> 172.21.15.26:0/1214697994 conn(0x7f6e6809fb50 msgr2=0x7f6e681319c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:05.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.372+0000 7f6e6e374640 1 -- 172.21.15.26:0/1214697994 shutdown_connections 2024-09-16T14:46:05.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:05.372+0000 7f6e6e374640 1 -- 172.21.15.26:0/1214697994 wait complete. 2024-09-16T14:46:05.404 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:46:05.404 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T14:46:05.404 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-16T14:46:05.404 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T14:46:05.405 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T14:46:05.405 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T14:46:05.405 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:46:05.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 10 monclient: get_monmap_and_config 2024-09-16T14:46:05.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 10 monclient: build_initial_monmap 2024-09-16T14:46:05.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 10 monclient: monmap: 2024-09-16T14:46:05.417 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:05.417 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.417 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:05.401892+0000 2024-09-16T14:46:05.417 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:05.401892+0000 2024-09-16T14:46:05.417 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:05.417 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:05.417 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:05.417 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:05.418 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:05.418 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 1 Processor -- start 2024-09-16T14:46:05.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 1 -- start start 2024-09-16T14:46:05.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 10 monclient: init 2024-09-16T14:46:05.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:05.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:05.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 10 monclient(hunting): picked mon.noname-b con 0x7f861406c120 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:05.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 10 monclient(hunting): picked mon.noname-a con 0x7f86141353d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:05.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 10 monclient(hunting): picked mon.noname-c con 0x7f86140a1040 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:05.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861406c500 con 0x7f86141353d0 2024-09-16T14:46:05.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86140a2ef0 con 0x7f86140a1040 2024-09-16T14:46:05.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861409e990 con 0x7f861406c120 2024-09-16T14:46:05.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:05.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 10 monclient(hunting): authenticate will time out at 1147.517635s 2024-09-16T14:46:05.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f86199bb640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f86140a1040 0x7f861413d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58240/0 (socket says 172.21.15.189:58240) 2024-09-16T14:46:05.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f86199bb640 1 -- 172.21.15.189:0/580062943 learned_addr learned my addr 172.21.15.189:0/580062943 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:05.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 1 -- 172.21.15.189:0/580062943 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651878866 0 0) 0x7f86140a2ef0 con 0x7f86140a1040 2024-09-16T14:46:05.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient(hunting): my global_id is 94519 2024-09-16T14:46:05.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 1 -- 172.21.15.189:0/580062943 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8600003530 con 0x7f86140a1040 2024-09-16T14:46:05.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 1 -- 172.21.15.189:0/580062943 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236013685 0 0) 0x7f861406c500 con 0x7f86141353d0 2024-09-16T14:46:05.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient(hunting): my global_id is 104184 2024-09-16T14:46:05.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 1 -- 172.21.15.189:0/580062943 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86140a2ef0 con 0x7f86141353d0 2024-09-16T14:46:05.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 1 -- 172.21.15.189:0/580062943 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355920831 0 0) 0x7f861409e990 con 0x7f861406c120 2024-09-16T14:46:05.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient(hunting): my global_id is 5405 2024-09-16T14:46:05.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 1 -- 172.21.15.189:0/580062943 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f861406c500 con 0x7f861406c120 2024-09-16T14:46:05.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 1 -- 172.21.15.189:0/580062943 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2349330762 0 0) 0x7f8600003530 con 0x7f86140a1040 2024-09-16T14:46:05.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 1 -- 172.21.15.189:0/580062943 >> v1:172.21.15.26:6790/0 conn(0x7f861406c120 legacy=0x7f8614134cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 1 -- 172.21.15.189:0/580062943 >> v1:172.21.15.26:6789/0 conn(0x7f86141353d0 legacy=0x7f8614139b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:05.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 1 monclient: found mon.noname-c 2024-09-16T14:46:05.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:05.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 1 -- 172.21.15.189:0/580062943 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861409ec70 con 0x7f86140a1040 2024-09-16T14:46:05.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient: _finish_auth 0 2024-09-16T14:46:05.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:05.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:05.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 1 -- 172.21.15.189:0/580062943 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8610001620 con 0x7f86140a1040 2024-09-16T14:46:05.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 5 monclient: authenticate success, global_id 94519 2024-09-16T14:46:05.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:05.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:05.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient: dump: 2024-09-16T14:46:05.423 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:05.424 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.424 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.424 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.424 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.424 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:05.424 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.424 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.424 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.424 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:05.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 1 -- 172.21.15.189:0/580062943 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8610001890 con 0x7f86140a1040 2024-09-16T14:46:05.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:05.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:05.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 4 set_mon_vals no callback set 2024-09-16T14:46:05.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 1 -- 172.21.15.189:0/580062943 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8610001c20 con 0x7f86140a1040 2024-09-16T14:46:05.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:05.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f860bfff640 10 monclient: dump: 2024-09-16T14:46:05.425 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:05.425 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.425 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.425 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.425 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.425 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:05.425 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.426 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.426 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.426 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 10 monclient: shutdown 2024-09-16T14:46:05.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:05.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 1 -- 172.21.15.189:0/580062943 >> v1:172.21.15.189:6789/0 conn(0x7f86140a1040 legacy=0x7f861413d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 1 -- 172.21.15.189:0/580062943 shutdown_connections 2024-09-16T14:46:05.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 1 -- 172.21.15.189:0/580062943 >> 172.21.15.189:0/580062943 conn(0x7f86140a3a40 msgr2=0x7f86140a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:05.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 1 -- 172.21.15.189:0/580062943 shutdown_connections 2024-09-16T14:46:05.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 1 -- 172.21.15.189:0/580062943 wait complete. 2024-09-16T14:46:05.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 10 monclient: build_initial_monmap 2024-09-16T14:46:05.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.400+0000 7f861b41e640 10 monclient: monmap: 2024-09-16T14:46:05.427 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:05.427 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:05.427 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:05.427 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:05.427 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:05.427 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:05.427 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:05.428 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:05.428 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:05.428 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 1 Processor -- start 2024-09-16T14:46:05.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 1 -- start start 2024-09-16T14:46:05.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 10 monclient: init 2024-09-16T14:46:05.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:05.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:05.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 10 monclient(hunting): picked mon.noname-b con 0x7f861406c120 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:05.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 10 monclient(hunting): picked mon.noname-a con 0x7f86140a1040 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:05.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 10 monclient(hunting): picked mon.noname-c con 0x7f86141353d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:05.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86140a56c0 con 0x7f86140a1040 2024-09-16T14:46:05.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861409e130 con 0x7f861406c120 2024-09-16T14:46:05.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86141325b0 con 0x7f86141353d0 2024-09-16T14:46:05.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:05.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 10 monclient(hunting): authenticate will time out at 1147.520724s 2024-09-16T14:46:05.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f86191ba640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f861406c120 0x7f861412ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58244/0 (socket says 172.21.15.189:58244) 2024-09-16T14:46:05.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f86191ba640 1 -- 172.21.15.189:0/2444837755 learned_addr learned my addr 172.21.15.189:0/2444837755 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:05.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227463623 0 0) 0x7f861409e130 con 0x7f861406c120 2024-09-16T14:46:05.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient(hunting): my global_id is 94522 2024-09-16T14:46:05.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f8003530 con 0x7f861406c120 2024-09-16T14:46:05.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392065912 0 0) 0x7f86140a56c0 con 0x7f86140a1040 2024-09-16T14:46:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient(hunting): my global_id is 104187 2024-09-16T14:46:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f861409e130 con 0x7f86140a1040 2024-09-16T14:46:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1928131827 0 0) 0x7f85f8003530 con 0x7f861406c120 2024-09-16T14:46:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86140a56c0 con 0x7f861406c120 2024-09-16T14:46:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f860c001d20 con 0x7f861406c120 2024-09-16T14:46:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient(hunting): dump: 2024-09-16T14:46:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.431 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.431 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.431 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.431 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:05.431 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.431 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.431 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.431 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:05.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2016023182 0 0) 0x7f86140a56c0 con 0x7f861406c120 2024-09-16T14:46:05.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 >> v1:172.21.15.26:6790/0 conn(0x7f86141353d0 legacy=0x7f86141d1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 >> v1:172.21.15.26:6789/0 conn(0x7f86140a1040 legacy=0x7f861412f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:46:05.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 1 monclient: found mon.b 2024-09-16T14:46:05.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:05.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8614132b70 con 0x7f861406c120 2024-09-16T14:46:05.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient: _finish_auth 0 2024-09-16T14:46:05.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:05.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:05.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 5 monclient: authenticate success, global_id 94522 2024-09-16T14:46:05.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 10 monclient: _renew_subs 2024-09-16T14:46:05.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 1 -- 172.21.15.189:0/2444837755 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f861412fd40 con 0x7f861406c120 2024-09-16T14:46:05.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 10 monclient: _renew_subs 2024-09-16T14:46:05.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 1 -- 172.21.15.189:0/2444837755 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f861409ce60 con 0x7f861406c120 2024-09-16T14:46:05.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f860c003c50 con 0x7f861406c120 2024-09-16T14:46:05.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:05.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f860c002110 con 0x7f861406c120 2024-09-16T14:46:05.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f860b7fe640 4 set_mon_vals no callback set 2024-09-16T14:46:05.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:05.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 10 monclient: dump: 2024-09-16T14:46:05.434 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:05.434 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.434 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.434 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.434 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.434 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:05.434 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.434 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.434 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.434 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f860c003fe0 con 0x7f861406c120 2024-09-16T14:46:05.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:05.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:05.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f861b41e640 1 -- 172.21.15.189:0/2444837755 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f4005050 con 0x7f861406c120 2024-09-16T14:46:05.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(77..77 src has 15..77) v4 ==== 5181+0+0 (unknown 3765235196 0 0) 0x7f860c043c10 con 0x7f861406c120 2024-09-16T14:46:05.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.404+0000 7f86189b9640 10 monclient: get_auth_request con 0x7f85f8035f30 auth_method 0 2024-09-16T14:46:05.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.412+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f860c014b90 con 0x7f861406c120 2024-09-16T14:46:05.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.412+0000 7f8609ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:05.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.412+0000 7f8609ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:05.436 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T14:46:05.436 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T14:46:05.436 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T14:46:05.437 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T14:46:05.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.572+0000 7f861b41e640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T14:46:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.572+0000 7f861b41e640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T14:46:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.572+0000 7f861b41e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.572+0000 7f861b41e640 1 -- 172.21.15.189:0/2444837755 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree-from", "bucket": "host2"} v 0) v1 -- 0x7f85f4005300 con 0x7f861406c120 2024-09-16T14:46:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.572+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host2"}]=0 v77) v1 ==== 80+0+177 (unknown 3405347542 0 2845383371) 0x7f860c013000 con 0x7f861406c120 2024-09-16T14:46:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.572+0000 7f8609ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T14:46:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.572+0000 7f8609ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:05.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.660+0000 7f861b41e640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T14:46:05.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.660+0000 7f861b41e640 10 monclient: _send_command 3 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T14:46:05.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.660+0000 7f861b41e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.660+0000 7f861b41e640 1 -- 172.21.15.189:0/2444837755 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree-from", "bucket": "host2"} v 0) v1 -- 0x7f85f40055a0 con 0x7f861406c120 2024-09-16T14:46:05.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.660+0000 7f8609ffb640 1 -- 172.21.15.189:0/2444837755 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host2"}]=0 v77) v1 ==== 80+0+177 (unknown 3405347542 0 2845383371) 0x7f860c014780 con 0x7f861406c120 2024-09-16T14:46:05.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.660+0000 7f8609ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T14:46:05.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.660+0000 7f8609ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:46:05.664 INFO:tasks.workunit.client.0.smithi189.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-16T14:46:05.664 INFO:tasks.workunit.client.0.smithi189.stdout:-16 1.00000 host host2 2024-09-16T14:46:05.664 INFO:tasks.workunit.client.0.smithi189.stdout: 9 1.00000 osd.9 down 1.00000 1.00000 2024-09-16T14:46:05.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.660+0000 7f861b41e640 1 -- 172.21.15.189:0/2444837755 >> v1:172.21.15.189:6812/3196161582 conn(0x7f85f8035f30 legacy=0x7f85f80383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.660+0000 7f861b41e640 10 monclient: shutdown 2024-09-16T14:46:05.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.660+0000 7f861b41e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:05.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.660+0000 7f861b41e640 1 -- 172.21.15.189:0/2444837755 >> v1:172.21.15.189:6789/0 conn(0x7f861406c120 legacy=0x7f861412ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.664+0000 7f861b41e640 1 -- 172.21.15.189:0/2444837755 shutdown_connections 2024-09-16T14:46:05.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.664+0000 7f861b41e640 1 -- 172.21.15.189:0/2444837755 >> 172.21.15.189:0/2444837755 conn(0x7f86140a3a40 msgr2=0x7f861406b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:05.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.664+0000 7f861b41e640 1 -- 172.21.15.189:0/2444837755 shutdown_connections 2024-09-16T14:46:05.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.664+0000 7f861b41e640 1 -- 172.21.15.189:0/2444837755 wait complete. 2024-09-16T14:46:05.679 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd tree-from osd.9 2024-09-16T14:46:05.679 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:46:05.679 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree-from osd.9 2024-09-16T14:46:05.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.788+0000 7fc1151e0640 10 monclient: get_monmap_and_config 2024-09-16T14:46:05.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.788+0000 7fc1151e0640 10 monclient: build_initial_monmap 2024-09-16T14:46:05.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient: monmap: 2024-09-16T14:46:05.794 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:05.794 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.794 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:05.793779+0000 2024-09-16T14:46:05.794 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:05.793779+0000 2024-09-16T14:46:05.794 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:05.794 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:05.794 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:05.794 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:05.794 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:05.795 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 1 Processor -- start 2024-09-16T14:46:05.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 1 -- start start 2024-09-16T14:46:05.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient: init 2024-09-16T14:46:05.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:05.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:46:05.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient(hunting): picked mon.noname-a con 0x7fc1101315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:05.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient(hunting): picked mon.noname-c con 0x7fc11009b320 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:05.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient(hunting): picked mon.noname-b con 0x7fc110132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:05.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11009edd0 con 0x7fc1101315a0 2024-09-16T14:46:05.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11009bc00 con 0x7fc11009b320 2024-09-16T14:46:05.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11009e900 con 0x7fc110132090 2024-09-16T14:46:05.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:05.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient(hunting): authenticate will time out at 1147.909289s 2024-09-16T14:46:05.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10e59c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc11009b320 0x7fc110136ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58344/0 (socket says 172.21.15.189:58344) 2024-09-16T14:46:05.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10e59c640 1 -- 172.21.15.189:0/700429023 learned_addr learned my addr 172.21.15.189:0/700429023 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:05.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10f59e640 1 -- 172.21.15.189:0/700429023 >> v1:172.21.15.26:6790/0 conn(0x7fc110132090 legacy=0x7fc11013a200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:05.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 1 -- 172.21.15.189:0/700429023 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054306720 0 0) 0x7fc11009bc00 con 0x7fc11009b320 2024-09-16T14:46:05.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 10 monclient(hunting): my global_id is 94525 2024-09-16T14:46:05.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 1 -- 172.21.15.189:0/700429023 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0fc003530 con 0x7fc11009b320 2024-09-16T14:46:05.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 1 -- 172.21.15.189:0/700429023 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 641665657 0 0) 0x7fc0fc003530 con 0x7fc11009b320 2024-09-16T14:46:05.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 1 -- 172.21.15.189:0/700429023 >> v1:172.21.15.26:6790/0 conn(0x7fc110132090 legacy=0x7fc11013a200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:05.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 1 -- 172.21.15.189:0/700429023 >> v1:172.21.15.26:6789/0 conn(0x7fc1101315a0 legacy=0x7fc110131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:46:05.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 10 monclient: _finish_hunting 0 2024-09-16T14:46:05.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 1 monclient: found mon.noname-c 2024-09-16T14:46:05.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:05.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 1 -- 172.21.15.189:0/700429023 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11009b920 con 0x7fc11009b320 2024-09-16T14:46:05.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 10 monclient: _finish_auth 0 2024-09-16T14:46:05.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 10 monclient: _check_auth_tickets 2024-09-16T14:46:05.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:05.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 1 -- 172.21.15.189:0/700429023 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc100001620 con 0x7fc11009b320 2024-09-16T14:46:05.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 5 monclient: authenticate success, global_id 94525 2024-09-16T14:46:05.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:05.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:05.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 10 monclient: dump: 2024-09-16T14:46:05.803 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:05.803 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.803 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.803 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.803 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.803 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:05.804 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.804 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.804 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.804 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:05.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 1 -- 172.21.15.189:0/700429023 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc100001910 con 0x7fc11009b320 2024-09-16T14:46:05.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:05.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 1 -- 172.21.15.189:0/700429023 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc100001d20 con 0x7fc11009b320 2024-09-16T14:46:05.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:05.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 4 set_mon_vals no callback set 2024-09-16T14:46:05.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:05.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc10dd9b640 10 monclient: dump: 2024-09-16T14:46:05.805 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:05.805 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.805 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.805 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.805 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.805 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:05.805 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.805 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.805 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.805 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient: shutdown 2024-09-16T14:46:05.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:05.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 1 -- 172.21.15.189:0/700429023 >> v1:172.21.15.189:6789/0 conn(0x7fc11009b320 legacy=0x7fc110136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 1 -- 172.21.15.189:0/700429023 shutdown_connections 2024-09-16T14:46:05.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 1 -- 172.21.15.189:0/700429023 >> 172.21.15.189:0/700429023 conn(0x7fc1100a3320 msgr2=0x7fc11009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:05.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 1 -- 172.21.15.189:0/700429023 shutdown_connections 2024-09-16T14:46:05.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 1 -- 172.21.15.189:0/700429023 wait complete. 2024-09-16T14:46:05.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient: build_initial_monmap 2024-09-16T14:46:05.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient: monmap: 2024-09-16T14:46:05.806 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:05.806 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:05.806 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:05.806 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:05.806 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:05.806 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:05.807 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:05.807 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:05.807 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:05.807 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 1 Processor -- start 2024-09-16T14:46:05.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 1 -- start start 2024-09-16T14:46:05.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient: init 2024-09-16T14:46:05.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:05.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:05.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient(hunting): picked mon.noname-b con 0x7fc1101315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:05.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient(hunting): picked mon.noname-a con 0x7fc110132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:05.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient(hunting): picked mon.noname-c con 0x7fc1101c4ae0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:05.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11009cd20 con 0x7fc110132090 2024-09-16T14:46:05.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11011dbb0 con 0x7fc1101315a0 2024-09-16T14:46:05.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1100a2870 con 0x7fc1101c4ae0 2024-09-16T14:46:05.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:05.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.792+0000 7fc1151e0640 10 monclient(hunting): authenticate will time out at 1147.911958s 2024-09-16T14:46:05.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc10ed9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc1101315a0 0x7fc110130900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58354/0 (socket says 172.21.15.189:58354) 2024-09-16T14:46:05.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc10ed9d640 1 -- 172.21.15.189:0/712645401 learned_addr learned my addr 172.21.15.189:0/712645401 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:05.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc10f59e640 1 -- 172.21.15.189:0/712645401 >> v1:172.21.15.26:6790/0 conn(0x7fc1101c4ae0 legacy=0x7fc1101cd2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:05.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 1 -- 172.21.15.189:0/712645401 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856249268 0 0) 0x7fc11011dbb0 con 0x7fc1101315a0 2024-09-16T14:46:05.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:05.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:05.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:05.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 10 monclient(hunting): my global_id is 94528 2024-09-16T14:46:05.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 1 -- 172.21.15.189:0/712645401 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0f0003530 con 0x7fc1101315a0 2024-09-16T14:46:05.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 1 -- 172.21.15.189:0/712645401 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4036055339 0 0) 0x7fc0f0003530 con 0x7fc1101315a0 2024-09-16T14:46:05.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 1 -- 172.21.15.189:0/712645401 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc11011dbb0 con 0x7fc1101315a0 2024-09-16T14:46:05.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 1 -- 172.21.15.189:0/712645401 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc0f80018f0 con 0x7fc1101315a0 2024-09-16T14:46:05.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:05.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 10 monclient(hunting): dump: 2024-09-16T14:46:05.809 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:05.809 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.809 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.809 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.809 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.809 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:05.810 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.810 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.810 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.810 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:05.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 1 -- 172.21.15.189:0/712645401 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4148240387 0 0) 0x7fc11011dbb0 con 0x7fc1101315a0 2024-09-16T14:46:05.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 1 -- 172.21.15.189:0/712645401 >> v1:172.21.15.26:6790/0 conn(0x7fc1101c4ae0 legacy=0x7fc1101cd2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:05.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 1 -- 172.21.15.189:0/712645401 >> v1:172.21.15.26:6789/0 conn(0x7fc110132090 legacy=0x7fc1101c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:05.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:05.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 1 monclient: found mon.b 2024-09-16T14:46:05.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:05.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 1 -- 172.21.15.189:0/712645401 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1101c57c0 con 0x7fc1101315a0 2024-09-16T14:46:05.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:05.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:05.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:05.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1151e0640 5 monclient: authenticate success, global_id 94528 2024-09-16T14:46:05.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1151e0640 10 monclient: _renew_subs 2024-09-16T14:46:05.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1151e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1151e0640 1 -- 172.21.15.189:0/712645401 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11011f050 con 0x7fc1101315a0 2024-09-16T14:46:05.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1151e0640 10 monclient: _renew_subs 2024-09-16T14:46:05.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1151e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1151e0640 1 -- 172.21.15.189:0/712645401 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11011d8c0 con 0x7fc1101315a0 2024-09-16T14:46:05.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 1 -- 172.21.15.189:0/712645401 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0f8001d20 con 0x7fc1101315a0 2024-09-16T14:46:05.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:05.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 1 -- 172.21.15.189:0/712645401 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc0f8003a50 con 0x7fc1101315a0 2024-09-16T14:46:05.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:05.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:05.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc10d59a640 4 set_mon_vals no callback set 2024-09-16T14:46:05.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 10 monclient: dump: 2024-09-16T14:46:05.812 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:05.812 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:05.812 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.812 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:05.812 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:05.812 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:05.813 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:05.813 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:05.813 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:05.813 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:05.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 1 -- 172.21.15.189:0/712645401 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc0f8003c30 con 0x7fc1101315a0 2024-09-16T14:46:05.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1151e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:05.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1151e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:05.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1151e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1151e0640 1 -- 172.21.15.189:0/712645401 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f4005050 con 0x7fc1101315a0 2024-09-16T14:46:05.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc1077fe640 1 -- 172.21.15.189:0/712645401 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(77..77 src has 15..77) v4 ==== 5181+0+0 (unknown 3765235196 0 0) 0x7fc0f8043900 con 0x7fc1101315a0 2024-09-16T14:46:05.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.796+0000 7fc10e59c640 10 monclient: get_auth_request con 0x7fc0f0035fe0 auth_method 0 2024-09-16T14:46:05.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.800+0000 7fc1077fe640 1 -- 172.21.15.189:0/712645401 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc0f8014630 con 0x7fc1101315a0 2024-09-16T14:46:05.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.800+0000 7fc1077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:05.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.800+0000 7fc1077fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:05.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.948+0000 7fc1151e0640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-16T14:46:05.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.948+0000 7fc1151e0640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-16T14:46:05.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.948+0000 7fc1151e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:05.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.948+0000 7fc1151e0640 1 -- 172.21.15.189:0/712645401 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree-from", "bucket": "osd.9"} v 0) v1 -- 0x7fc0f4005300 con 0x7fc1101315a0 2024-09-16T14:46:05.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.952+0000 7fc1077fe640 1 -- 172.21.15.189:0/712645401 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "osd.9"}]=-22 "osd.9" is not a bucket v77) v1 ==== 103+0+0 (unknown 1856690693 0 0) 0x7fc0f8012cd0 con 0x7fc1101315a0 2024-09-16T14:46:05.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.952+0000 7fc1077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-16T14:46:05.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:05.952+0000 7fc1077fe640 10 monclient: _finish_command 2 = mon:22 "osd.9" is not a bucket 2024-09-16T14:46:05.955 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: "osd.9" is not a bucket 2024-09-16T14:46:05.959 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:46:05.959 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush rm bar 2024-09-16T14:46:05.959 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:46:05.959 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm bar 2024-09-16T14:46:06.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 10 monclient: get_monmap_and_config 2024-09-16T14:46:06.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 10 monclient: build_initial_monmap 2024-09-16T14:46:06.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 10 monclient: monmap: 2024-09-16T14:46:06.075 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:06.075 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:06.075 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:06.074927+0000 2024-09-16T14:46:06.075 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:06.074927+0000 2024-09-16T14:46:06.075 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:06.075 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:06.075 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:06.076 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:06.076 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:06.076 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:06.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 1 Processor -- start 2024-09-16T14:46:06.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 1 -- start start 2024-09-16T14:46:06.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 10 monclient: init 2024-09-16T14:46:06.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:06.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:06.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 10 monclient(hunting): picked mon.noname-b con 0x7efe5c0a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:06.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 10 monclient(hunting): picked mon.noname-c con 0x7efe5c0a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:06.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 10 monclient(hunting): picked mon.noname-a con 0x7efe5c13b430 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:06.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c0a30b0 con 0x7efe5c13b430 2024-09-16T14:46:06.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c0a2e30 con 0x7efe5c0a25f0 2024-09-16T14:46:06.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c09eb50 con 0x7efe5c0a1b00 2024-09-16T14:46:06.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:06.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 10 monclient(hunting): authenticate will time out at 1148.190539s 2024-09-16T14:46:06.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe5a7fc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efe5c0a25f0 0x7efe5c139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58362/0 (socket says 172.21.15.189:58362) 2024-09-16T14:46:06.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe5a7fc640 1 -- 172.21.15.189:0/737832893 learned_addr learned my addr 172.21.15.189:0/737832893 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:06.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe5affd640 1 -- 172.21.15.189:0/737832893 >> v1:172.21.15.26:6790/0 conn(0x7efe5c0a1b00 legacy=0x7efe5c0a1ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:06.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 1 -- 172.21.15.189:0/737832893 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204747038 0 0) 0x7efe5c0a2e30 con 0x7efe5c0a25f0 2024-09-16T14:46:06.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:06.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:06.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:06.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 10 monclient(hunting): my global_id is 94531 2024-09-16T14:46:06.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 1 -- 172.21.15.189:0/737832893 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe48003530 con 0x7efe5c0a25f0 2024-09-16T14:46:06.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 1 -- 172.21.15.189:0/737832893 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 831592078 0 0) 0x7efe48003530 con 0x7efe5c0a25f0 2024-09-16T14:46:06.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 1 -- 172.21.15.189:0/737832893 >> v1:172.21.15.26:6790/0 conn(0x7efe5c0a1b00 legacy=0x7efe5c0a1ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:06.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 1 -- 172.21.15.189:0/737832893 >> v1:172.21.15.26:6789/0 conn(0x7efe5c13b430 legacy=0x7efe5c13d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:46:06.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:46:06.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 1 monclient: found mon.noname-c 2024-09-16T14:46:06.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:06.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:06.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 1 -- 172.21.15.189:0/737832893 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5c09ee30 con 0x7efe5c0a25f0 2024-09-16T14:46:06.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 10 monclient: _finish_auth 0 2024-09-16T14:46:06.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:06.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:06.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 1 -- 172.21.15.189:0/737832893 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efe54001620 con 0x7efe5c0a25f0 2024-09-16T14:46:06.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 5 monclient: authenticate success, global_id 94531 2024-09-16T14:46:06.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:06.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:06.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:06.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 10 monclient: dump: 2024-09-16T14:46:06.086 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:06.086 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:06.086 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:06.087 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:06.087 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:06.087 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:06.087 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:06.087 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:06.087 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:06.087 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:06.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:06.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 1 -- 172.21.15.189:0/737832893 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe54001910 con 0x7efe5c0a25f0 2024-09-16T14:46:06.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:06.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 1 -- 172.21.15.189:0/737832893 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efe54001d20 con 0x7efe5c0a25f0 2024-09-16T14:46:06.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:06.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:06.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe59ffb640 10 monclient: dump: 2024-09-16T14:46:06.088 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:06.088 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:06.088 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:06.088 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:06.088 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:06.089 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:06.089 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:06.089 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:06.089 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:06.089 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:06.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:06.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 4 set_mon_vals no callback set 2024-09-16T14:46:06.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 10 monclient: shutdown 2024-09-16T14:46:06.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:06.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 1 -- 172.21.15.189:0/737832893 >> v1:172.21.15.189:6789/0 conn(0x7efe5c0a25f0 legacy=0x7efe5c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:06.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 1 -- 172.21.15.189:0/737832893 shutdown_connections 2024-09-16T14:46:06.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.072+0000 7efe61831640 1 -- 172.21.15.189:0/737832893 >> 172.21.15.189:0/737832893 conn(0x7efe5c0a7dd0 msgr2=0x7efe5c0a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:06.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 1 -- 172.21.15.189:0/737832893 shutdown_connections 2024-09-16T14:46:06.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 1 -- 172.21.15.189:0/737832893 wait complete. 2024-09-16T14:46:06.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 monclient: build_initial_monmap 2024-09-16T14:46:06.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 monclient: monmap: 2024-09-16T14:46:06.090 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:06.090 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:06.090 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:06.090 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:06.090 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:06.090 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:06.091 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:06.091 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:06.091 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:06.091 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:06.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 1 Processor -- start 2024-09-16T14:46:06.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 1 -- start start 2024-09-16T14:46:06.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 monclient: init 2024-09-16T14:46:06.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:06.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:06.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 monclient(hunting): picked mon.noname-a con 0x7efe5c0a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:06.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 monclient(hunting): picked mon.noname-c con 0x7efe5c0a25f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:06.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 monclient(hunting): picked mon.noname-b con 0x7efe5c13b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:06.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c09d820 con 0x7efe5c0a1b00 2024-09-16T14:46:06.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c09e2f0 con 0x7efe5c13b430 2024-09-16T14:46:06.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c09d340 con 0x7efe5c0a25f0 2024-09-16T14:46:06.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:06.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 monclient(hunting): authenticate will time out at 1148.193469s 2024-09-16T14:46:06.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe5b7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efe5c13b430 0x7efe5c13a450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58366/0 (socket says 172.21.15.189:58366) 2024-09-16T14:46:06.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe5b7fe640 1 -- 172.21.15.189:0/2760172023 learned_addr learned my addr 172.21.15.189:0/2760172023 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:06.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe5a7fc640 1 -- 172.21.15.189:0/2760172023 >> v1:172.21.15.26:6790/0 conn(0x7efe5c0a25f0 legacy=0x7efe5c139d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:06.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 1 -- 172.21.15.189:0/2760172023 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230389914 0 0) 0x7efe5c09e2f0 con 0x7efe5c13b430 2024-09-16T14:46:06.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:06.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:06.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:06.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 10 monclient(hunting): my global_id is 94534 2024-09-16T14:46:06.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 1 -- 172.21.15.189:0/2760172023 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe3c003530 con 0x7efe5c13b430 2024-09-16T14:46:06.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 1 -- 172.21.15.189:0/2760172023 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3054519221 0 0) 0x7efe3c003530 con 0x7efe5c13b430 2024-09-16T14:46:06.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 1 -- 172.21.15.189:0/2760172023 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe5c09e2f0 con 0x7efe5c13b430 2024-09-16T14:46:06.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 1 -- 172.21.15.189:0/2760172023 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efe44001870 con 0x7efe5c13b430 2024-09-16T14:46:06.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:06.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:06.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 10 monclient(hunting): dump: 2024-09-16T14:46:06.094 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:06.094 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:06.094 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:06.094 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:06.094 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:06.094 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:06.094 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:06.094 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:06.094 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:06.095 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:06.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:06.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 1 -- 172.21.15.189:0/2760172023 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2565489932 0 0) 0x7efe5c09e2f0 con 0x7efe5c13b430 2024-09-16T14:46:06.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 1 -- 172.21.15.189:0/2760172023 >> v1:172.21.15.26:6790/0 conn(0x7efe5c0a25f0 legacy=0x7efe5c139d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:06.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 1 -- 172.21.15.189:0/2760172023 >> v1:172.21.15.26:6789/0 conn(0x7efe5c0a1b00 legacy=0x7efe5c139630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:06.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:06.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 1 monclient: found mon.b 2024-09-16T14:46:06.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:06.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:06.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 1 -- 172.21.15.189:0/2760172023 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5c1356f0 con 0x7efe5c13b430 2024-09-16T14:46:06.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 10 monclient: _finish_auth 0 2024-09-16T14:46:06.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:06.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:06.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 5 monclient: authenticate success, global_id 94534 2024-09-16T14:46:06.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 monclient: _renew_subs 2024-09-16T14:46:06.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:06.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 1 -- 172.21.15.189:0/2760172023 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5c11fd20 con 0x7efe5c13b430 2024-09-16T14:46:06.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 monclient: _renew_subs 2024-09-16T14:46:06.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:06.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 1 -- 172.21.15.189:0/2760172023 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5c11d950 con 0x7efe5c13b430 2024-09-16T14:46:06.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 1 -- 172.21.15.189:0/2760172023 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe44001ca0 con 0x7efe5c13b430 2024-09-16T14:46:06.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:06.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 1 -- 172.21.15.189:0/2760172023 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efe44003a50 con 0x7efe5c13b430 2024-09-16T14:46:06.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:06.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:06.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe597fa640 4 set_mon_vals no callback set 2024-09-16T14:46:06.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 10 monclient: dump: 2024-09-16T14:46:06.097 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:06.098 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:06.098 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:06.098 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:06.098 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:06.098 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:06.098 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:06.098 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:06.098 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:06.099 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:06.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 1 -- 172.21.15.189:0/2760172023 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7efe44003bb0 con 0x7efe5c13b430 2024-09-16T14:46:06.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:06.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:06.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:06.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe61831640 1 -- 172.21.15.189:0/2760172023 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe40005050 con 0x7efe5c13b430 2024-09-16T14:46:06.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe53fff640 1 -- 172.21.15.189:0/2760172023 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(77..77 src has 15..77) v4 ==== 5181+0+0 (unknown 3765235196 0 0) 0x7efe440437d0 con 0x7efe5c13b430 2024-09-16T14:46:06.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.076+0000 7efe5affd640 10 monclient: get_auth_request con 0x7efe3c035e70 auth_method 0 2024-09-16T14:46:06.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.080+0000 7efe53fff640 1 -- 172.21.15.189:0/2760172023 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7efe44014730 con 0x7efe5c13b430 2024-09-16T14:46:06.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.080+0000 7efe53fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:06.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.080+0000 7efe53fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:06.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.248+0000 7efe61831640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T14:46:06.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.248+0000 7efe61831640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T14:46:06.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.248+0000 7efe61831640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:06.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:06.248+0000 7efe61831640 1 -- 172.21.15.189:0/2760172023 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7efe40005300 con 0x7efe5c13b430 2024-09-16T14:46:09.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:09.075+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:09.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:09.075+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:09.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:09.075+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:09.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:09.075+0000 7efe52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:10.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:10.075+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:10.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:10.075+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:10.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:10.075+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:11.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:11.075+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:11.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:11.075+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:11.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:11.075+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:12.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:12.075+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:12.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:12.075+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:12.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:12.075+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:13.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:13.075+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:13.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:13.075+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:13.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:13.075+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:14.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:14.075+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:14.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:14.075+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:14.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:14.075+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:15.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:15.079+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:15.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:15.079+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:15.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:15.079+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:16.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:16.075+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:16.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:16.075+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:16.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:16.075+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:17.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:17.075+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:17.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:17.075+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:17.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:17.075+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:18.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:18.075+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:18.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:18.075+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:18.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:18.075+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:19.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:19.079+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:19.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:19.079+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:19.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:19.079+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:19.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:19.079+0000 7efe52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:20.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:20.075+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:20.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:20.075+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:20.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:20.075+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:20.438 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T14:46:20.438 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T14:46:20.438 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:46:20.438 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:46:20.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd959de9640 10 monclient: get_monmap_and_config 2024-09-16T14:46:20.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd959de9640 10 monclient: build_initial_monmap 2024-09-16T14:46:20.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd959de9640 10 monclient: monmap: 2024-09-16T14:46:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:46:20.567359+0000 2024-09-16T14:46:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:46:20.567359+0000 2024-09-16T14:46:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:20.568 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:20.568 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:20.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd959de9640 1 Processor -- start 2024-09-16T14:46:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd959de9640 1 -- start start 2024-09-16T14:46:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd959de9640 10 monclient: init 2024-09-16T14:46:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd959de9640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd959de9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:46:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd959de9640 10 monclient(hunting): picked mon.noname-a con 0x7fd9540a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd959de9640 10 monclient(hunting): picked mon.noname-c con 0x7fd9540a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd959de9640 10 monclient(hunting): picked mon.noname-b con 0x7fd9540a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd959de9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9540a3250 con 0x7fd9540a1d10 2024-09-16T14:46:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd959de9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95409f600 con 0x7fd9540a0000 2024-09-16T14:46:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd959de9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9540a3830 con 0x7fd9540a2800 2024-09-16T14:46:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd959de9640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:20.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd959de9640 10 monclient(hunting): authenticate will time out at 1204.687454s 2024-09-16T14:46:20.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd9537fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd9540a1d10 0x7fd9540a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34018/0 (socket says 172.21.15.26:34018) 2024-09-16T14:46:20.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd9537fe640 1 -- 172.21.15.26:0/1044199355 learned_addr learned my addr 172.21.15.26:0/1044199355 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:20.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.564+0000 7fd953fff640 1 -- 172.21.15.26:0/1044199355 >> v1:172.21.15.26:6790/0 conn(0x7fd9540a2800 legacy=0x7fd95413d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:20.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:20.768+0000 7fd953fff640 1 -- 172.21.15.26:0/1044199355 >> v1:172.21.15.26:6790/0 conn(0x7fd9540a2800 legacy=0x7fd95413d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:21.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:21.075+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:21.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:21.075+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:21.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:21.075+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:21.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:21.172+0000 7fd953fff640 1 -- 172.21.15.26:0/1044199355 >> v1:172.21.15.26:6790/0 conn(0x7fd9540a2800 legacy=0x7fd95413d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:21.976+0000 7fd953fff640 1 -- 172.21.15.26:0/1044199355 >> v1:172.21.15.26:6790/0 conn(0x7fd9540a2800 legacy=0x7fd95413d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:22.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:22.079+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:22.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:22.079+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:22.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:22.079+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:23.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:23.075+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:23.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:23.075+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:23.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:23.075+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.564+0000 7fd9517fa640 10 monclient(hunting): tick 2024-09-16T14:46:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.564+0000 7fd9517fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:46:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.564+0000 7fd9517fa640 1 monclient(hunting): continuing hunt 2024-09-16T14:46:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.564+0000 7fd9517fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:46:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.564+0000 7fd9517fa640 1 -- 172.21.15.26:0/1044199355 >> v1:172.21.15.26:6790/0 conn(0x7fd9540a2800 legacy=0x7fd95413d440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.564+0000 7fd9517fa640 1 -- 172.21.15.26:0/1044199355 >> v1:172.21.15.189:6789/0 conn(0x7fd9540a0000 legacy=0x7fd954139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.564+0000 7fd9517fa640 1 -- 172.21.15.26:0/1044199355 >> v1:172.21.15.26:6789/0 conn(0x7fd9540a1d10 legacy=0x7fd9540a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.564+0000 7fd9517fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.564+0000 7fd9517fa640 10 monclient(hunting): picked mon.noname-a con 0x7fd9340036d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.564+0000 7fd9517fa640 10 monclient(hunting): picked mon.noname-b con 0x7fd9540a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.564+0000 7fd9517fa640 10 monclient(hunting): picked mon.noname-c con 0x7fd9540a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.564+0000 7fd952ffd640 1 -- 172.21.15.26:0/1044199355 >> v1:172.21.15.26:6790/0 conn(0x7fd9540a2800 legacy=0x7fd9340093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.564+0000 7fd9517fa640 1 -- 172.21.15.26:0/1044199355 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95409ed60 con 0x7fd9340036d0 2024-09-16T14:46:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.568+0000 7fd9517fa640 1 -- 172.21.15.26:0/1044199355 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9540a3830 con 0x7fd9540a0000 2024-09-16T14:46:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.568+0000 7fd9517fa640 1 -- 172.21.15.26:0/1044199355 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd934010380 con 0x7fd9540a2800 2024-09-16T14:46:23.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.568+0000 7fd9517fa640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:23.768+0000 7fd952ffd640 1 -- 172.21.15.26:0/1044199355 >> v1:172.21.15.26:6790/0 conn(0x7fd9540a2800 legacy=0x7fd9340093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:24.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:24.075+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:24.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:24.075+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:24.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:24.075+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:24.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:24.172+0000 7fd952ffd640 1 -- 172.21.15.26:0/1044199355 >> v1:172.21.15.26:6790/0 conn(0x7fd9540a2800 legacy=0x7fd9340093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:24.976+0000 7fd952ffd640 1 -- 172.21.15.26:0/1044199355 >> v1:172.21.15.26:6790/0 conn(0x7fd9540a2800 legacy=0x7fd9340093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.075+0000 7efe52ffd640 10 monclient: tick 2024-09-16T14:46:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.075+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.075+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.484+0000 7fd9527fc640 1 -- 172.21.15.26:0/1044199355 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158966206 0 0) 0x7fd9540a3830 con 0x7fd9540a0000 2024-09-16T14:46:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.484+0000 7fd9527fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.484+0000 7fd9527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.484+0000 7fd9527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.484+0000 7fd9527fc640 10 monclient(hunting): my global_id is 94537 2024-09-16T14:46:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.484+0000 7fd9527fc640 1 -- 172.21.15.26:0/1044199355 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd940003530 con 0x7fd9540a0000 2024-09-16T14:46:25.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.483+0000 7efe53fff640 1 -- 172.21.15.189:0/2760172023 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=-39 (39) Directory not empty v77) v1 ==== 99+0+0 (unknown 2202415905 0 0) 0x7efe4401cd20 con 0x7efe5c13b430 2024-09-16T14:46:25.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.483+0000 7efe53fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T14:46:25.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.483+0000 7efe53fff640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-16T14:46:25.491 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-16T14:46:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.484+0000 7fd9527fc640 1 -- 172.21.15.26:0/1044199355 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4037950263 0 0) 0x7fd95409ed60 con 0x7fd9340036d0 2024-09-16T14:46:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.484+0000 7fd9527fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.484+0000 7fd9527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.484+0000 7fd9527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.484+0000 7fd9527fc640 10 monclient(hunting): my global_id is 104190 2024-09-16T14:46:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.484+0000 7fd9527fc640 1 -- 172.21.15.26:0/1044199355 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9540a3830 con 0x7fd9340036d0 2024-09-16T14:46:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 1 -- 172.21.15.26:0/1044199355 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2251122348 0 0) 0x7fd9540a3830 con 0x7fd9340036d0 2024-09-16T14:46:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 1 -- 172.21.15.26:0/1044199355 >> v1:172.21.15.26:6790/0 conn(0x7fd9540a2800 legacy=0x7fd9340093f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 1 -- 172.21.15.26:0/1044199355 >> v1:172.21.15.189:6789/0 conn(0x7fd9540a0000 legacy=0x7fd93400cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 10 monclient: _finish_hunting 0 2024-09-16T14:46:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 1 monclient: found mon.noname-a 2024-09-16T14:46:25.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:25.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:46:25.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 1 -- 172.21.15.26:0/1044199355 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd934011590 con 0x7fd9340036d0 2024-09-16T14:46:25.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 10 monclient: _finish_auth 0 2024-09-16T14:46:25.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 10 monclient: _check_auth_tickets 2024-09-16T14:46:25.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:25.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 1 -- 172.21.15.26:0/1044199355 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd944001a20 con 0x7fd9340036d0 2024-09-16T14:46:25.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:46:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 10 monclient: dump: 2024-09-16T14:46:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:25.498 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd959de9640 5 monclient: authenticate success, global_id 104190 2024-09-16T14:46:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd959de9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 1 -- 172.21.15.26:0/1044199355 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd944001c10 con 0x7fd9340036d0 2024-09-16T14:46:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd959de9640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 1 -- 172.21.15.26:0/1044199355 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd944001fe0 con 0x7fd9340036d0 2024-09-16T14:46:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd959de9640 4 set_mon_vals no callback set 2024-09-16T14:46:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd9527fc640 10 monclient: dump: 2024-09-16T14:46:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:25.499 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd959de9640 10 monclient: shutdown 2024-09-16T14:46:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd959de9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd959de9640 1 -- 172.21.15.26:0/1044199355 >> v1:172.21.15.26:6789/0 conn(0x7fd9340036d0 legacy=0x7fd934005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:25.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd959de9640 1 -- 172.21.15.26:0/1044199355 shutdown_connections 2024-09-16T14:46:25.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd959de9640 1 -- 172.21.15.26:0/1044199355 >> 172.21.15.26:0/1044199355 conn(0x7fd9540a7f30 msgr2=0x7fd9540a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:25.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd959de9640 1 -- 172.21.15.26:0/1044199355 shutdown_connections 2024-09-16T14:46:25.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd959de9640 1 -- 172.21.15.26:0/1044199355 wait complete. 2024-09-16T14:46:25.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd959de9640 10 monclient: build_initial_monmap 2024-09-16T14:46:25.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.488+0000 7fd959de9640 10 monclient: monmap: 2024-09-16T14:46:25.500 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:25.500 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:25.500 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:46:25.500 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:46:25.500 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:25.500 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:46:25.501 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:25.501 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:25.501 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:25.501 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:25.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd959de9640 1 Processor -- start 2024-09-16T14:46:25.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd959de9640 1 -- start start 2024-09-16T14:46:25.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd959de9640 10 monclient: init 2024-09-16T14:46:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:46:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush unlink host2 2024-09-16T14:46:25.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd959de9640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:25.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd959de9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:46:25.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd959de9640 10 monclient(hunting): picked mon.noname-a con 0x7fd9340036d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:25.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd959de9640 10 monclient(hunting): picked mon.noname-b con 0x7fd9540a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:25.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd959de9640 10 monclient(hunting): picked mon.noname-c con 0x7fd9540a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:25.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd959de9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9540a6560 con 0x7fd9340036d0 2024-09-16T14:46:25.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd953fff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fd9540a2800 legacy=0x7fd9541d1a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:25.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd959de9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd954121730 con 0x7fd9540a0000 2024-09-16T14:46:25.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd959de9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95409e500 con 0x7fd9540a2800 2024-09-16T14:46:25.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd959de9640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:25.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd9537fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd9340036d0 0x7fd954133d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34042/0 (socket says 172.21.15.26:34042) 2024-09-16T14:46:25.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd959de9640 10 monclient(hunting): authenticate will time out at 1209.614285s 2024-09-16T14:46:25.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd9537fe640 1 -- 172.21.15.26:0/2467699804 learned_addr learned my addr 172.21.15.26:0/2467699804 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:25.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117554116 0 0) 0x7fd9540a6560 con 0x7fd9340036d0 2024-09-16T14:46:25.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:25.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:25.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:25.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 10 monclient(hunting): my global_id is 104193 2024-09-16T14:46:25.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9340052e0 con 0x7fd9340036d0 2024-09-16T14:46:25.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576079100 0 0) 0x7fd954121730 con 0x7fd9540a0000 2024-09-16T14:46:25.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:25.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:25.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:25.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 10 monclient(hunting): my global_id is 94540 2024-09-16T14:46:25.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9540a6560 con 0x7fd9540a0000 2024-09-16T14:46:25.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3882272138 0 0) 0x7fd9540a6560 con 0x7fd9540a0000 2024-09-16T14:46:25.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd954121730 con 0x7fd9540a0000 2024-09-16T14:46:25.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd93c001790 con 0x7fd9540a0000 2024-09-16T14:46:25.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:25.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:25.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 10 monclient(hunting): dump: 2024-09-16T14:46:25.509 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:25.509 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:25.509 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.509 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.509 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:25.510 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:25.510 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:25.510 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:25.510 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:25.510 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:25.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.492+0000 7fd94bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:25.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2535321871 0 0) 0x7fd954121730 con 0x7fd9540a0000 2024-09-16T14:46:25.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 >> v1:172.21.15.26:6790/0 conn(0x7fd9540a2800 legacy=0x7fd9541d1a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:25.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 >> v1:172.21.15.26:6789/0 conn(0x7fd9340036d0 legacy=0x7fd954133d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:25.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:25.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 1 monclient: found mon.b 2024-09-16T14:46:25.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:25.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:25.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd954134ad0 con 0x7fd9540a0000 2024-09-16T14:46:25.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 10 monclient: _finish_auth 0 2024-09-16T14:46:25.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:25.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:25.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd959de9640 5 monclient: authenticate success, global_id 94540 2024-09-16T14:46:25.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd959de9640 10 monclient: _renew_subs 2024-09-16T14:46:25.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd959de9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:25.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd959de9640 1 -- 172.21.15.26:0/2467699804 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95411fd20 con 0x7fd9540a0000 2024-09-16T14:46:25.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd959de9640 10 monclient: _renew_subs 2024-09-16T14:46:25.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd959de9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:25.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd959de9640 1 -- 172.21.15.26:0/2467699804 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95411f3d0 con 0x7fd9540a0000 2024-09-16T14:46:25.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd93c001d60 con 0x7fd9540a0000 2024-09-16T14:46:25.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:25.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd93c0037c0 con 0x7fd9540a0000 2024-09-16T14:46:25.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:25.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:25.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 10 monclient: dump: 2024-09-16T14:46:25.512 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:25.512 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:25.512 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.513 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.513 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:25.513 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:25.513 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:25.513 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:25.513 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:25.513 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:25.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd951ffb640 4 set_mon_vals no callback set 2024-09-16T14:46:25.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd93c0039e0 con 0x7fd9540a0000 2024-09-16T14:46:25.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.496+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(77..77 src has 15..77) v4 ==== 5181+0+0 (unknown 3765235196 0 0) 0x7fd93c0436c0 con 0x7fd9540a0000 2024-09-16T14:46:25.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.500+0000 7fd9537fe640 10 monclient: get_auth_request con 0x7fd934036980 auth_method 0 2024-09-16T14:46:25.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.500+0000 7fd959de9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:25.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.500+0000 7fd959de9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.500+0000 7fd959de9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.500+0000 7fd959de9640 1 -- 172.21.15.26:0/2467699804 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd938005050 con 0x7fd9540a0000 2024-09-16T14:46:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.504+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd93c004200 con 0x7fd9540a0000 2024-09-16T14:46:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.504+0000 7fd94bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.504+0000 7fd94bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:25.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.611+0000 7f735f8f8640 10 monclient: get_monmap_and_config 2024-09-16T14:46:25.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.611+0000 7f735f8f8640 10 monclient: build_initial_monmap 2024-09-16T14:46:25.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.611+0000 7f735f8f8640 10 monclient: monmap: 2024-09-16T14:46:25.616 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:25.617 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:25.617 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:25.616565+0000 2024-09-16T14:46:25.617 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:25.616565+0000 2024-09-16T14:46:25.617 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:25.617 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:25.617 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:25.617 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:25.617 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:25.617 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:25.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.611+0000 7f735f8f8640 1 Processor -- start 2024-09-16T14:46:25.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.611+0000 7f735f8f8640 1 -- start start 2024-09-16T14:46:25.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.611+0000 7f735f8f8640 10 monclient: init 2024-09-16T14:46:25.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:25.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:25.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient(hunting): picked mon.noname-b con 0x7f73581318a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:25.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient(hunting): picked mon.noname-c con 0x7f7358134d50 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:25.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient(hunting): picked mon.noname-a con 0x7f735813b2b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:25.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735809d8a0 con 0x7f735813b2b0 2024-09-16T14:46:25.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735809ddf0 con 0x7f7358134d50 2024-09-16T14:46:25.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735809e8b0 con 0x7f73581318a0 2024-09-16T14:46:25.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:25.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient(hunting): authenticate will time out at 1167.732193s 2024-09-16T14:46:25.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735d694640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f73581318a0 legacy=0x7f7358131c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:25.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735ce93640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7358134d50 0x7f7358139b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39384/0 (socket says 172.21.15.189:39384) 2024-09-16T14:46:25.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735ce93640 1 -- 172.21.15.189:0/71656118 learned_addr learned my addr 172.21.15.189:0/71656118 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:25.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 1 -- 172.21.15.189:0/71656118 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733599712 0 0) 0x7f735809ddf0 con 0x7f7358134d50 2024-09-16T14:46:25.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:25.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:25.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:25.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 10 monclient(hunting): my global_id is 94543 2024-09-16T14:46:25.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 1 -- 172.21.15.189:0/71656118 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7344003530 con 0x7f7358134d50 2024-09-16T14:46:25.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 1 -- 172.21.15.189:0/71656118 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3549763939 0 0) 0x7f7344003530 con 0x7f7358134d50 2024-09-16T14:46:25.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 1 -- 172.21.15.189:0/71656118 >> v1:172.21.15.26:6790/0 conn(0x7f73581318a0 legacy=0x7f7358131c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:25.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 1 -- 172.21.15.189:0/71656118 >> v1:172.21.15.26:6789/0 conn(0x7f735813b2b0 legacy=0x7f735813d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:25.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:25.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 1 monclient: found mon.noname-c 2024-09-16T14:46:25.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:25.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:25.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 1 -- 172.21.15.189:0/71656118 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735809eb90 con 0x7f7358134d50 2024-09-16T14:46:25.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 10 monclient: _finish_auth 0 2024-09-16T14:46:25.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:25.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:25.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 5 monclient: authenticate success, global_id 94543 2024-09-16T14:46:25.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 1 -- 172.21.15.189:0/71656118 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7348001620 con 0x7f7358134d50 2024-09-16T14:46:25.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:25.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:25.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 10 monclient: dump: 2024-09-16T14:46:25.623 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:25.623 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:25.623 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.624 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.624 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:25.624 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:25.624 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:25.624 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:25.624 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:25.624 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:25.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:25.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 1 -- 172.21.15.189:0/71656118 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73480018d0 con 0x7f7358134d50 2024-09-16T14:46:25.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:25.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 1 -- 172.21.15.189:0/71656118 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7348001ca0 con 0x7f7358134d50 2024-09-16T14:46:25.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:25.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 4 set_mon_vals no callback set 2024-09-16T14:46:25.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:25.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:25.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734ffff640 10 monclient: dump: 2024-09-16T14:46:25.625 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:25.625 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:25.625 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.625 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.625 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:25.625 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:25.625 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:25.625 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:25.625 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:25.625 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:25.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient: shutdown 2024-09-16T14:46:25.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:25.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 1 -- 172.21.15.189:0/71656118 >> v1:172.21.15.189:6789/0 conn(0x7f7358134d50 legacy=0x7f7358139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:25.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 1 -- 172.21.15.189:0/71656118 shutdown_connections 2024-09-16T14:46:25.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 1 -- 172.21.15.189:0/71656118 >> 172.21.15.189:0/71656118 conn(0x7f73580a3320 msgr2=0x7f735809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:25.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 1 -- 172.21.15.189:0/71656118 shutdown_connections 2024-09-16T14:46:25.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 1 -- 172.21.15.189:0/71656118 wait complete. 2024-09-16T14:46:25.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient: build_initial_monmap 2024-09-16T14:46:25.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient: monmap: 2024-09-16T14:46:25.626 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:25.626 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:25.626 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:25.626 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:25.626 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:25.627 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:25.627 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:25.627 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:25.627 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:25.627 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:25.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 1 Processor -- start 2024-09-16T14:46:25.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 1 -- start start 2024-09-16T14:46:25.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient: init 2024-09-16T14:46:25.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:25.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:46:25.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient(hunting): picked mon.noname-a con 0x7f73581318a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:25.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient(hunting): picked mon.noname-b con 0x7f7358134d50 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:25.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient(hunting): picked mon.noname-c con 0x7f735813b2b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:25.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735811b940 con 0x7f73581318a0 2024-09-16T14:46:25.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735806b380 con 0x7f7358134d50 2024-09-16T14:46:25.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73580a1c60 con 0x7f735813b2b0 2024-09-16T14:46:25.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:25.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735f8f8640 10 monclient(hunting): authenticate will time out at 1167.735643s 2024-09-16T14:46:25.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735ce93640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7358134d50 0x7f7358139960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39388/0 (socket says 172.21.15.189:39388) 2024-09-16T14:46:25.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735ce93640 1 -- 172.21.15.189:0/656627782 learned_addr learned my addr 172.21.15.189:0/656627782 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:25.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f735de95640 1 -- 172.21.15.189:0/656627782 >> v1:172.21.15.26:6790/0 conn(0x7f735813b2b0 legacy=0x7f73581d1480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:25.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.615+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130753192 0 0) 0x7f735806b380 con 0x7f7358134d50 2024-09-16T14:46:25.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:25.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:25.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:25.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient(hunting): my global_id is 94546 2024-09-16T14:46:25.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f733c003530 con 0x7f7358134d50 2024-09-16T14:46:25.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2521142550 0 0) 0x7f733c003530 con 0x7f7358134d50 2024-09-16T14:46:25.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f735806b380 con 0x7f7358134d50 2024-09-16T14:46:25.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436857265 0 0) 0x7f735811b940 con 0x7f73581318a0 2024-09-16T14:46:25.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:25.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:25.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:25.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient(hunting): my global_id is 104199 2024-09-16T14:46:25.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f733c003530 con 0x7f73581318a0 2024-09-16T14:46:25.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7348001be0 con 0x7f7358134d50 2024-09-16T14:46:25.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:25.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:25.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient(hunting): dump: 2024-09-16T14:46:25.633 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:25.633 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:25.633 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.633 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.633 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:25.633 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:25.633 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:25.633 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:25.634 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:25.634 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:25.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:25.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2979343070 0 0) 0x7f735806b380 con 0x7f7358134d50 2024-09-16T14:46:25.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 >> v1:172.21.15.26:6790/0 conn(0x7f735813b2b0 legacy=0x7f73581d1480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:25.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 >> v1:172.21.15.26:6789/0 conn(0x7f73581318a0 legacy=0x7f7358139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:25.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:46:25.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 1 monclient: found mon.b 2024-09-16T14:46:25.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:25.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:25.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735813ab60 con 0x7f7358134d50 2024-09-16T14:46:25.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient: _finish_auth 0 2024-09-16T14:46:25.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:25.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:25.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f735f8f8640 5 monclient: authenticate success, global_id 94546 2024-09-16T14:46:25.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f735f8f8640 10 monclient: _renew_subs 2024-09-16T14:46:25.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f735f8f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:25.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f735f8f8640 1 -- 172.21.15.189:0/656627782 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7358119570 con 0x7f7358134d50 2024-09-16T14:46:25.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f735f8f8640 10 monclient: _renew_subs 2024-09-16T14:46:25.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f735f8f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:25.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f735f8f8640 1 -- 172.21.15.189:0/656627782 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f735813a070 con 0x7f7358134d50 2024-09-16T14:46:25.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7348001200 con 0x7f7358134d50 2024-09-16T14:46:25.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:25.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734f7fe640 4 set_mon_vals no callback set 2024-09-16T14:46:25.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7348004da0 con 0x7f7358134d50 2024-09-16T14:46:25.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:25.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:25.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 10 monclient: dump: 2024-09-16T14:46:25.637 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:25.637 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:25.637 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.637 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.637 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:25.637 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:25.637 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:25.637 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:25.637 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:25.637 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:25.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7348004fc0 con 0x7f7358134d50 2024-09-16T14:46:25.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f735f8f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:25.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f735f8f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:25.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f735f8f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:25.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f735f8f8640 1 -- 172.21.15.189:0/656627782 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7338005050 con 0x7f7358134d50 2024-09-16T14:46:25.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(77..77 src has 15..77) v4 ==== 5181+0+0 (unknown 3765235196 0 0) 0x7f7348043b10 con 0x7f7358134d50 2024-09-16T14:46:25.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.619+0000 7f735d694640 10 monclient: get_auth_request con 0x7f733c0360e0 auth_method 0 2024-09-16T14:46:25.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.623+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7348006630 con 0x7f7358134d50 2024-09-16T14:46:25.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.623+0000 7f734dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:25.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.623+0000 7f734dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:25.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.720+0000 7fd959de9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:46:25.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.720+0000 7fd959de9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:46:25.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.720+0000 7fd959de9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:25.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.720+0000 7fd959de9640 1 -- 172.21.15.26:0/2467699804 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd938005880 con 0x7fd9540a0000 2024-09-16T14:46:25.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.720+0000 7fd94bfff640 1 -- 172.21.15.26:0/2467699804 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 223403484) 0x7fd93c012a90 con 0x7fd9540a0000 2024-09-16T14:46:25.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.720+0000 7fd94bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:46:25.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.720+0000 7fd94bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:25.725 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":40,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:46:25.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.724+0000 7fd959de9640 1 -- 172.21.15.26:0/2467699804 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd934036980 legacy=0x7fd934038e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:25.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.724+0000 7fd959de9640 10 monclient: shutdown 2024-09-16T14:46:25.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.724+0000 7fd959de9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:25.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.724+0000 7fd959de9640 1 -- 172.21.15.26:0/2467699804 >> v1:172.21.15.189:6789/0 conn(0x7fd9540a0000 legacy=0x7fd9541ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:25.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.724+0000 7fd959de9640 1 -- 172.21.15.26:0/2467699804 shutdown_connections 2024-09-16T14:46:25.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.724+0000 7fd959de9640 1 -- 172.21.15.26:0/2467699804 >> 172.21.15.26:0/2467699804 conn(0x7fd9540a7f30 msgr2=0x7fd954134e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:25.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.724+0000 7fd959de9640 1 -- 172.21.15.26:0/2467699804 shutdown_connections 2024-09-16T14:46:25.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.724+0000 7fd959de9640 1 -- 172.21.15.26:0/2467699804 wait complete. 2024-09-16T14:46:25.742 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:46:25.742 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T14:46:25.742 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:46:25.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.787+0000 7f735f8f8640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T14:46:25.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.787+0000 7f735f8f8640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T14:46:25.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.787+0000 7f735f8f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:25.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:25.787+0000 7f735f8f8640 1 -- 172.21.15.189:0/656627782 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host2"} v 0) v1 -- 0x7f7338005300 con 0x7f7358134d50 2024-09-16T14:46:25.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.856+0000 7f76ac444640 10 monclient: get_monmap_and_config 2024-09-16T14:46:25.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.856+0000 7f76ac444640 10 monclient: build_initial_monmap 2024-09-16T14:46:25.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.856+0000 7f76ac444640 10 monclient: monmap: 2024-09-16T14:46:25.861 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:25.861 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:25.861 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:46:25.861158+0000 2024-09-16T14:46:25.861 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:46:25.861158+0000 2024-09-16T14:46:25.862 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:25.862 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:25.863 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:25.863 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:25.863 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:25.863 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:25.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 1 Processor -- start 2024-09-16T14:46:25.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 1 -- start start 2024-09-16T14:46:25.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 10 monclient: init 2024-09-16T14:46:25.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:25.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:25.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 10 monclient(hunting): picked mon.noname-a con 0x7f76a40a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:25.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 10 monclient(hunting): picked mon.noname-b con 0x7f76a4099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:25.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 10 monclient(hunting): picked mon.noname-c con 0x7f76a40a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:25.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a4098ea0 con 0x7f76a40a1c00 2024-09-16T14:46:25.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a40997b0 con 0x7f76a40a26f0 2024-09-16T14:46:25.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a4099460 con 0x7f76a4099b60 2024-09-16T14:46:25.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:25.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 10 monclient(hunting): authenticate will time out at 1209.981176s 2024-09-16T14:46:25.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a99df640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f76a4099b60 legacy=0x7f76a4139c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:25.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76aa1e0640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f76a40a1c00 0x7f76a40a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34050/0 (socket says 172.21.15.26:34050) 2024-09-16T14:46:25.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76aa1e0640 1 -- 172.21.15.26:0/1992845542 learned_addr learned my addr 172.21.15.26:0/1992845542 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:25.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 1 -- 172.21.15.26:0/1992845542 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155907603 0 0) 0x7f76a4098ea0 con 0x7f76a40a1c00 2024-09-16T14:46:25.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 10 monclient(hunting): _negotiate 2024-09-16T14:46:25.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:25.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:25.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 10 monclient(hunting): my global_id is 104202 2024-09-16T14:46:25.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 1 -- 172.21.15.26:0/1992845542 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7690003530 con 0x7f76a40a1c00 2024-09-16T14:46:25.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 1 -- 172.21.15.26:0/1992845542 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 487903504 0 0) 0x7f7690003530 con 0x7f76a40a1c00 2024-09-16T14:46:25.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 1 -- 172.21.15.26:0/1992845542 >> v1:172.21.15.26:6790/0 conn(0x7f76a4099b60 legacy=0x7f76a4139c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:25.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 1 -- 172.21.15.26:0/1992845542 >> v1:172.21.15.189:6789/0 conn(0x7f76a40a26f0 legacy=0x7f76a413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:25.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 10 monclient: _finish_hunting 0 2024-09-16T14:46:25.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 1 monclient: found mon.noname-a 2024-09-16T14:46:25.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:25.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:46:25.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 1 -- 172.21.15.26:0/1992845542 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a40a34b0 con 0x7f76a40a1c00 2024-09-16T14:46:25.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 10 monclient: _finish_auth 0 2024-09-16T14:46:25.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 10 monclient: _check_auth_tickets 2024-09-16T14:46:25.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:25.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 1 -- 172.21.15.26:0/1992845542 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f76a0001620 con 0x7f76a40a1c00 2024-09-16T14:46:25.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:25.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:46:25.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 10 monclient: dump: 2024-09-16T14:46:25.873 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:25.873 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:25.873 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.873 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.873 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:25.873 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:25.874 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:25.874 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:25.874 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:25.874 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:25.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 5 monclient: authenticate success, global_id 104202 2024-09-16T14:46:25.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:25.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 1 -- 172.21.15.26:0/1992845542 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76a0001940 con 0x7f76a40a1c00 2024-09-16T14:46:25.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:25.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 1 -- 172.21.15.26:0/1992845542 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f76a0001cd0 con 0x7f76a40a1c00 2024-09-16T14:46:25.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:25.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:25.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76a91de640 10 monclient: dump: 2024-09-16T14:46:25.874 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:25.875 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:25.875 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.875 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.875 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:25.875 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:25.875 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:25.875 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:25.875 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:25.875 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:25.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:25.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 4 set_mon_vals no callback set 2024-09-16T14:46:25.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 10 monclient: shutdown 2024-09-16T14:46:25.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:25.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 1 -- 172.21.15.26:0/1992845542 >> v1:172.21.15.26:6789/0 conn(0x7f76a40a1c00 legacy=0x7f76a40a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:25.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 1 -- 172.21.15.26:0/1992845542 shutdown_connections 2024-09-16T14:46:25.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 1 -- 172.21.15.26:0/1992845542 >> 172.21.15.26:0/1992845542 conn(0x7f76a40a3d00 msgr2=0x7f76a40a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:25.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 1 -- 172.21.15.26:0/1992845542 shutdown_connections 2024-09-16T14:46:25.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 1 -- 172.21.15.26:0/1992845542 wait complete. 2024-09-16T14:46:25.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 10 monclient: build_initial_monmap 2024-09-16T14:46:25.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 10 monclient: monmap: 2024-09-16T14:46:25.876 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:25.876 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:25.876 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:46:25.876 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:46:25.876 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:25.876 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:46:25.876 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:25.877 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:25.877 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:25.877 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:25.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 1 Processor -- start 2024-09-16T14:46:25.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 1 -- start start 2024-09-16T14:46:25.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.860+0000 7f76ac444640 10 monclient: init 2024-09-16T14:46:25.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:25.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:46:25.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 10 monclient(hunting): picked mon.noname-a con 0x7f76a4099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:25.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 10 monclient(hunting): picked mon.noname-b con 0x7f76a40a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:25.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 10 monclient(hunting): picked mon.noname-c con 0x7f76a40a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:25.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a409f1a0 con 0x7f76a4099b60 2024-09-16T14:46:25.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a409ebe0 con 0x7f76a40a1c00 2024-09-16T14:46:25.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a409e3f0 con 0x7f76a40a26f0 2024-09-16T14:46:25.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:25.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 10 monclient(hunting): authenticate will time out at 1209.984330s 2024-09-16T14:46:25.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76aa9e1640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f76a40a26f0 legacy=0x7f76a41d1980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:25.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76aa1e0640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f76a4099b60 0x7f76a41370b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34052/0 (socket says 172.21.15.26:34052) 2024-09-16T14:46:25.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76aa1e0640 1 -- 172.21.15.26:0/19302641 learned_addr learned my addr 172.21.15.26:0/19302641 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:25.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948321678 0 0) 0x7f76a40a34b0 con 0x7f76a4099b60 2024-09-16T14:46:25.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:25.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:25.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:25.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient(hunting): my global_id is 104205 2024-09-16T14:46:25.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7688003530 con 0x7f76a4099b60 2024-09-16T14:46:25.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1172747616 0 0) 0x7f7688003530 con 0x7f76a4099b60 2024-09-16T14:46:25.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76a40a34b0 con 0x7f76a4099b60 2024-09-16T14:46:25.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f76a0001440 con 0x7f76a4099b60 2024-09-16T14:46:25.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:25.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:46:25.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient(hunting): dump: 2024-09-16T14:46:25.880 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:25.880 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:25.880 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.880 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.880 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:25.880 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:25.880 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:25.881 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:25.881 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:25.881 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:25.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:25.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071933311 0 0) 0x7f76a409ebe0 con 0x7f76a40a1c00 2024-09-16T14:46:25.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:25.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:25.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:25.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient(hunting): my global_id is 94552 2024-09-16T14:46:25.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7688003530 con 0x7f76a40a1c00 2024-09-16T14:46:25.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2513234208 0 0) 0x7f76a40a34b0 con 0x7f76a4099b60 2024-09-16T14:46:25.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 >> v1:172.21.15.26:6790/0 conn(0x7f76a40a26f0 legacy=0x7f76a41d1980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:25.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 >> v1:172.21.15.189:6789/0 conn(0x7f76a40a1c00 legacy=0x7f76a41ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:25.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:25.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 monclient: found mon.a 2024-09-16T14:46:25.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:25.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:25.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a411ea10 con 0x7f76a4099b60 2024-09-16T14:46:25.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient: _finish_auth 0 2024-09-16T14:46:25.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:25.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:25.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 5 monclient: authenticate success, global_id 104205 2024-09-16T14:46:25.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 10 monclient: _renew_subs 2024-09-16T14:46:25.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:25.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 1 -- 172.21.15.26:0/19302641 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a411e170 con 0x7f76a4099b60 2024-09-16T14:46:25.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 10 monclient: _renew_subs 2024-09-16T14:46:25.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:25.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 1 -- 172.21.15.26:0/19302641 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a41377c0 con 0x7f76a4099b60 2024-09-16T14:46:25.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76a0001210 con 0x7f76a4099b60 2024-09-16T14:46:25.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:25.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f76a0004c50 con 0x7f76a4099b60 2024-09-16T14:46:25.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:25.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:25.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76a89dd640 4 set_mon_vals no callback set 2024-09-16T14:46:25.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient: dump: 2024-09-16T14:46:25.884 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:25.884 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:25.885 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.885 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.885 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:25.885 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:25.885 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:25.885 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:25.885 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:25.885 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:25.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:25.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76ac444640 1 -- 172.21.15.26:0/19302641 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f768c006590 con 0x7f768c002830 2024-09-16T14:46:25.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f76a000f6c0 con 0x7f76a4099b60 2024-09-16T14:46:25.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(77..77 src has 15..77) v4 ==== 5181+0+0 (unknown 3765235196 0 0) 0x7f76a0043760 con 0x7f76a4099b60 2024-09-16T14:46:25.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534593679 0 0) 0x7f768c006590 con 0x7f768c002830 2024-09-16T14:46:25.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:25.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:25.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:25.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient(hunting): my global_id is 104208 2024-09-16T14:46:25.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a40a34b0 con 0x7f768c002830 2024-09-16T14:46:25.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f76aa9e1640 10 monclient: get_auth_request con 0x7f7688036180 auth_method 0 2024-09-16T14:46:25.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 720184294 0 0) 0x7f76a40a34b0 con 0x7f768c002830 2024-09-16T14:46:25.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f768c006590 con 0x7f768c002830 2024-09-16T14:46:25.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7694004040 con 0x7f768c002830 2024-09-16T14:46:25.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:25.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:25.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient: dump: 2024-09-16T14:46:25.887 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:25.888 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:25.888 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.888 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.888 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:25.888 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:25.888 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:25.888 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:25.888 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:25.888 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:25.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 736574126 0 0) 0x7f768c006590 con 0x7f768c002830 2024-09-16T14:46:25.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f768c00a0e0 con 0x7f768c002830 2024-09-16T14:46:25.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7694006450 con 0x7f768c002830 2024-09-16T14:46:25.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:25.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:25.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 >> v1:172.21.15.26:6789/0 conn(0x7f768c002830 legacy=0x7f768c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:25.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.864+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 >> v1:172.21.15.26:6789/0 conn(0x7f768c002830 legacy=0x7f768c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:46:25.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f76ac444640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:46:25.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f76ac444640 1 -- 172.21.15.26:0/19302641 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f768c009d80 con 0x7f768c0090a0 2024-09-16T14:46:25.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505726061 0 0) 0x7f768c009d80 con 0x7f768c0090a0 2024-09-16T14:46:25.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:25.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:25.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:25.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 10 monclient(hunting): my global_id is 104211 2024-09-16T14:46:25.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f768c006590 con 0x7f768c0090a0 2024-09-16T14:46:25.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 480849959 0 0) 0x7f768c006590 con 0x7f768c0090a0 2024-09-16T14:46:25.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f768c009d80 con 0x7f768c0090a0 2024-09-16T14:46:25.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7694004a40 con 0x7f768c0090a0 2024-09-16T14:46:25.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:25.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:25.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 10 monclient: dump: 2024-09-16T14:46:25.892 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:25.892 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:25.892 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.892 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:25.892 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:25.892 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:25.893 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:25.893 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:25.893 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:25.893 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:25.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1264748005 0 0) 0x7f768c009d80 con 0x7f768c0090a0 2024-09-16T14:46:25.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f768c00c290 con 0x7f768c0090a0 2024-09-16T14:46:25.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4200 (unknown 0 0 291056545) 0x7f7694005ef0 con 0x7f768c0090a0 2024-09-16T14:46:25.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:46:25.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:25.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 >> v1:172.21.15.26:6789/0 conn(0x7f768c0090a0 legacy=0x7f768c009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:25.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f769affd640 1 -- 172.21.15.26:0/19302641 >> v1:172.21.15.26:6789/0 conn(0x7f768c0090a0 legacy=0x7f768c009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:46:25.894 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:46:25.894 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:46:25.894 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:46:25.894 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:46:25.894 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 40, 2024-09-16T14:46:25.894 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:46:25.894 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:46:25.894 INFO:teuthology.orchestra.run.smithi026.stdout: 1 2024-09-16T14:46:25.894 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:25.894 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:46:25.895 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:46:25.895 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:46:25.895 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:46:25.895 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:25.895 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:25.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:25.895 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:25.895 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:25.895 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:25.895 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:25.895 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:25.895 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:25.896 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:25.896 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:25.896 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:46:25.896 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:46:25.896 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:25.896 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:25.896 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:25.896 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:25.896 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:25.896 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:25.896 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:25.897 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:25.897 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:25.897 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:25.897 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:25.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:25.897 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:46:25.897 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:46:25.897 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:46:25.897 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:46:25.897 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:46:25.897 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:46:25.897 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:46:25.897 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:46:25.898 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:46:25.898 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:46:25.898 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:46:25.898 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:46:25.898 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:46:25.898 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:46:25.898 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:46:25.898 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:46:25.898 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:46:25.898 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:25.898 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:25.898 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:25.898 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:25.899 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:25.899 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:25.899 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:25.899 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:25.899 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:25.899 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:25.899 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:25.899 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:46:25.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:25.899 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:46:25.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:25.899 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:46:25.899 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:46:25.900 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:25.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:25.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:25.900 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:25.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:46:25.900 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:25.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:25.900 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:25.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:25.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:46:25.900 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:46:25.900 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:25.900 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:25.901 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:25.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:25.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:25.901 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:46:25.901 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:46:25.901 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:25.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:25.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:25.901 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:25.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:46:25.901 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:25.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:25.901 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:25.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:25.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:46:25.902 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:46:25.902 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:25.902 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:25.902 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:25.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:25.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:25.902 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:46:25.902 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:46:25.902 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:25.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:25.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:25.902 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:25.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:46:25.903 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:25.903 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:25.903 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:25.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:25.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:46:25.903 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:46:25.903 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:25.903 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:25.903 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:25.903 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:25.903 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:25.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:25.904 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:46:25.904 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:46:25.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:25.904 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:25.904 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:25.904 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:46:25.904 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:25.904 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:25.904 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:46:25.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:25.904 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:25.904 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:25.905 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:46:25.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:25.905 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:25.905 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:46:25.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:25.905 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:25.905 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:25.905 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:46:25.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:25.905 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:25.905 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:46:25.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:25.906 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:25.906 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:25.906 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:46:25.906 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:25.906 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:25.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:25.906 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:46:25.906 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:46:25.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f76ac444640 1 -- 172.21.15.26:0/19302641 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7688036180 legacy=0x7f7688038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:25.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f76ac444640 10 monclient: shutdown 2024-09-16T14:46:25.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f76ac444640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:25.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f76ac444640 1 -- 172.21.15.26:0/19302641 >> v1:172.21.15.26:6789/0 conn(0x7f76a4099b60 legacy=0x7f76a41370b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:25.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f76aa9e1640 1 -- 172.21.15.26:0/19302641 reap_dead start 2024-09-16T14:46:25.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f76aa9e1640 1 -- 172.21.15.26:0/19302641 reap_dead start 2024-09-16T14:46:25.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f76ac444640 1 -- 172.21.15.26:0/19302641 shutdown_connections 2024-09-16T14:46:25.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f76ac444640 1 -- 172.21.15.26:0/19302641 >> 172.21.15.26:0/19302641 conn(0x7f76a40a3d00 msgr2=0x7f76a40a01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:25.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f76ac444640 1 -- 172.21.15.26:0/19302641 shutdown_connections 2024-09-16T14:46:25.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:25.872+0000 7f76ac444640 1 -- 172.21.15.26:0/19302641 wait complete. 2024-09-16T14:46:25.908 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:46:25.909 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T14:46:25.909 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:46:26.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.016+0000 7f369695a640 10 monclient: get_monmap_and_config 2024-09-16T14:46:26.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.016+0000 7f369695a640 10 monclient: build_initial_monmap 2024-09-16T14:46:26.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.016+0000 7f369695a640 10 monclient: monmap: 2024-09-16T14:46:26.021 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:26.021 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:26.021 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:46:26.021023+0000 2024-09-16T14:46:26.021 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:46:26.021023+0000 2024-09-16T14:46:26.022 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:26.022 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:26.022 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:26.022 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:26.022 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:26.022 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:26.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 1 Processor -- start 2024-09-16T14:46:26.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 1 -- start start 2024-09-16T14:46:26.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 10 monclient: init 2024-09-16T14:46:26.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:26.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:26.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 10 monclient(hunting): picked mon.noname-b con 0x7f369009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:26.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 10 monclient(hunting): picked mon.noname-a con 0x7f3690132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:26.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 10 monclient(hunting): picked mon.noname-c con 0x7f3690138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:26.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369009d260 con 0x7f3690132650 2024-09-16T14:46:26.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3690132a80 con 0x7f3690138ff0 2024-09-16T14:46:26.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369009cf60 con 0x7f369009cb80 2024-09-16T14:46:26.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:26.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 10 monclient(hunting): authenticate will time out at 1210.140997s 2024-09-16T14:46:26.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368ffff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f369009cb80 legacy=0x7f36900717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:26.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368f7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3690132650 0x7f36901378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34070/0 (socket says 172.21.15.26:34070) 2024-09-16T14:46:26.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368f7fe640 1 -- 172.21.15.26:0/2986776430 learned_addr learned my addr 172.21.15.26:0/2986776430 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:26.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 1 -- 172.21.15.26:0/2986776430 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035532260 0 0) 0x7f369009d260 con 0x7f3690132650 2024-09-16T14:46:26.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:26.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:26.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:26.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient(hunting): my global_id is 104214 2024-09-16T14:46:26.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 1 -- 172.21.15.26:0/2986776430 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3678003530 con 0x7f3690132650 2024-09-16T14:46:26.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 1 -- 172.21.15.26:0/2986776430 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424065659 0 0) 0x7f3690132a80 con 0x7f3690138ff0 2024-09-16T14:46:26.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:26.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:26.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:26.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient(hunting): my global_id is 94555 2024-09-16T14:46:26.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 1 -- 172.21.15.26:0/2986776430 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f369009d260 con 0x7f3690138ff0 2024-09-16T14:46:26.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 1 -- 172.21.15.26:0/2986776430 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2807266454 0 0) 0x7f3678003530 con 0x7f3690132650 2024-09-16T14:46:26.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 1 -- 172.21.15.26:0/2986776430 >> v1:172.21.15.26:6790/0 conn(0x7f369009cb80 legacy=0x7f36900717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:26.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 1 -- 172.21.15.26:0/2986776430 >> v1:172.21.15.189:6789/0 conn(0x7f3690138ff0 legacy=0x7f369013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:26.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:26.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 1 monclient: found mon.noname-a 2024-09-16T14:46:26.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:26.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:46:26.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 1 -- 172.21.15.26:0/2986776430 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369009f2d0 con 0x7f3690132650 2024-09-16T14:46:26.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient: _finish_auth 0 2024-09-16T14:46:26.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:26.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:26.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 1 -- 172.21.15.26:0/2986776430 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3680001620 con 0x7f3690132650 2024-09-16T14:46:26.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 5 monclient: authenticate success, global_id 104214 2024-09-16T14:46:26.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:26.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:26.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:46:26.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient: dump: 2024-09-16T14:46:26.033 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:26.033 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:26.034 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:26.034 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:26.034 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:26.034 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:26.034 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:26.034 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:26.034 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:26.034 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:26.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:26.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 1 -- 172.21.15.26:0/2986776430 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3680001890 con 0x7f3690132650 2024-09-16T14:46:26.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:26.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 1 -- 172.21.15.26:0/2986776430 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3680001c20 con 0x7f3690132650 2024-09-16T14:46:26.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:26.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:26.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f368effd640 10 monclient: dump: 2024-09-16T14:46:26.035 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:26.035 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:26.035 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:26.035 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:26.035 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:26.035 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:26.035 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:26.035 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:26.035 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:26.035 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:26.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:26.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 4 set_mon_vals no callback set 2024-09-16T14:46:26.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 10 monclient: shutdown 2024-09-16T14:46:26.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:26.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 1 -- 172.21.15.26:0/2986776430 >> v1:172.21.15.26:6789/0 conn(0x7f3690132650 legacy=0x7f36901378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:26.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 1 -- 172.21.15.26:0/2986776430 shutdown_connections 2024-09-16T14:46:26.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 1 -- 172.21.15.26:0/2986776430 >> 172.21.15.26:0/2986776430 conn(0x7f369009fb50 msgr2=0x7f369009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:26.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 1 -- 172.21.15.26:0/2986776430 shutdown_connections 2024-09-16T14:46:26.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 1 -- 172.21.15.26:0/2986776430 wait complete. 2024-09-16T14:46:26.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 10 monclient: build_initial_monmap 2024-09-16T14:46:26.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 10 monclient: monmap: 2024-09-16T14:46:26.037 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:46:26.037 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:26.037 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:46:26.037 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:46:26.037 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:26.037 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:46:26.037 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:26.037 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:26.037 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:26.037 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:26.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 1 Processor -- start 2024-09-16T14:46:26.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 1 -- start start 2024-09-16T14:46:26.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.020+0000 7f369695a640 10 monclient: init 2024-09-16T14:46:26.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:26.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:46:26.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 10 monclient(hunting): picked mon.noname-a con 0x7f369009cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:26.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 10 monclient(hunting): picked mon.noname-b con 0x7f3690132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:26.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 10 monclient(hunting): picked mon.noname-c con 0x7f3690138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:26.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3690135640 con 0x7f369009cb80 2024-09-16T14:46:26.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36900a2950 con 0x7f3690132650 2024-09-16T14:46:26.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369013c6b0 con 0x7f3690138ff0 2024-09-16T14:46:26.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:26.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 10 monclient(hunting): authenticate will time out at 1210.144407s 2024-09-16T14:46:26.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f3694ef7640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f3690138ff0 legacy=0x7f369012eda0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:26.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f369009cb80 0x7f369012df80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34084/0 (socket says 172.21.15.26:34084) 2024-09-16T14:46:26.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368ffff640 1 -- 172.21.15.26:0/2257453823 learned_addr learned my addr 172.21.15.26:0/2257453823 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:46:26.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 448287739 0 0) 0x7f3690135640 con 0x7f369009cb80 2024-09-16T14:46:26.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:26.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:26.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:26.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient(hunting): my global_id is 104217 2024-09-16T14:46:26.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f367c003530 con 0x7f369009cb80 2024-09-16T14:46:26.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2938452821 0 0) 0x7f367c003530 con 0x7f369009cb80 2024-09-16T14:46:26.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3690135640 con 0x7f369009cb80 2024-09-16T14:46:26.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3684004160 con 0x7f369009cb80 2024-09-16T14:46:26.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:26.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:46:26.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient(hunting): dump: 2024-09-16T14:46:26.040 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:26.040 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:26.040 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:26.040 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:26.040 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:26.040 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:26.040 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:26.040 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:26.040 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:26.040 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:26.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:26.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592352248 0 0) 0x7f36900a2950 con 0x7f3690132650 2024-09-16T14:46:26.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:26.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:26.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:26.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient(hunting): my global_id is 94558 2024-09-16T14:46:26.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f367c003530 con 0x7f3690132650 2024-09-16T14:46:26.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2712518277 0 0) 0x7f3690135640 con 0x7f369009cb80 2024-09-16T14:46:26.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 >> v1:172.21.15.26:6790/0 conn(0x7f3690138ff0 legacy=0x7f369012eda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:26.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 >> v1:172.21.15.189:6789/0 conn(0x7f3690132650 legacy=0x7f369012e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:26.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:46:26.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 monclient: found mon.a 2024-09-16T14:46:26.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:26.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:26.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369009a240 con 0x7f369009cb80 2024-09-16T14:46:26.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient: _finish_auth 0 2024-09-16T14:46:26.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:46:26.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:26.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 5 monclient: authenticate success, global_id 104217 2024-09-16T14:46:26.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 10 monclient: _renew_subs 2024-09-16T14:46:26.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:26.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 1 -- 172.21.15.26:0/2257453823 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3690135360 con 0x7f369009cb80 2024-09-16T14:46:26.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 10 monclient: _renew_subs 2024-09-16T14:46:26.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:46:26.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 1 -- 172.21.15.26:0/2257453823 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3690134060 con 0x7f369009cb80 2024-09-16T14:46:26.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3684004480 con 0x7f369009cb80 2024-09-16T14:46:26.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:26.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368e7fc640 4 set_mon_vals no callback set 2024-09-16T14:46:26.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3684004810 con 0x7f369009cb80 2024-09-16T14:46:26.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:26.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:46:26.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient: dump: 2024-09-16T14:46:26.043 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:26.043 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:26.043 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:26.043 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:26.043 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:26.043 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:26.043 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:26.044 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:26.044 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:26.044 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:26.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:26.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f369695a640 1 -- 172.21.15.26:0/2257453823 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3670006590 con 0x7f3670002830 2024-09-16T14:46:26.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f36840049f0 con 0x7f369009cb80 2024-09-16T14:46:26.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(77..77 src has 15..77) v4 ==== 5181+0+0 (unknown 3765235196 0 0) 0x7f3684043940 con 0x7f369009cb80 2024-09-16T14:46:26.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077976435 0 0) 0x7f3670006590 con 0x7f3670002830 2024-09-16T14:46:26.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:26.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:26.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:26.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient(hunting): my global_id is 94561 2024-09-16T14:46:26.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3690135640 con 0x7f3670002830 2024-09-16T14:46:26.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f3694ef7640 10 monclient: get_auth_request con 0x7f367c0360e0 auth_method 0 2024-09-16T14:46:26.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2243394198 0 0) 0x7f3690135640 con 0x7f3670002830 2024-09-16T14:46:26.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3670006590 con 0x7f3670002830 2024-09-16T14:46:26.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3680001440 con 0x7f3670002830 2024-09-16T14:46:26.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:26.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:26.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 10 monclient: dump: 2024-09-16T14:46:26.045 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:26.045 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:26.045 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:26.045 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:26.046 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:26.046 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:26.046 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:26.046 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:26.046 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:26.046 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:26.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2854655669 0 0) 0x7f3670006590 con 0x7f3670002830 2024-09-16T14:46:26.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.024+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f367000a0e0 con 0x7f3670002830 2024-09-16T14:46:26.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.028+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f3680004880 con 0x7f3670002830 2024-09-16T14:46:26.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.028+0000 7f368cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:26.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.028+0000 7f368cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:26.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.028+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 >> v1:172.21.15.189:6789/0 conn(0x7f3670002830 legacy=0x7f3670004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:26.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.028+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 >> v1:172.21.15.189:6789/0 conn(0x7f3670002830 legacy=0x7f3670004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:46:26.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.032+0000 7f369695a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:46:26.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.032+0000 7f369695a640 1 -- 172.21.15.26:0/2257453823 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3670009d80 con 0x7f36700090a0 2024-09-16T14:46:26.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.032+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217899101 0 0) 0x7f3670009d80 con 0x7f36700090a0 2024-09-16T14:46:26.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.032+0000 7f368cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:26.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.032+0000 7f368cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:26.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.032+0000 7f368cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:26.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.032+0000 7f368cff9640 10 monclient(hunting): my global_id is 94564 2024-09-16T14:46:26.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.032+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3670006590 con 0x7f36700090a0 2024-09-16T14:46:26.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 423225378 0 0) 0x7f3670006590 con 0x7f36700090a0 2024-09-16T14:46:26.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3670009d80 con 0x7f36700090a0 2024-09-16T14:46:26.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3680004d80 con 0x7f36700090a0 2024-09-16T14:46:26.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f368cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:26.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f368cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:26.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f368cff9640 10 monclient: dump: 2024-09-16T14:46:26.049 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:46:26.049 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:26.049 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:26.049 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:26.050 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:26.050 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:46:26.050 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:26.050 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:26.050 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:26.050 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:46:26.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1680201104 0 0) 0x7f3670009d80 con 0x7f36700090a0 2024-09-16T14:46:26.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f367000c290 con 0x7f36700090a0 2024-09-16T14:46:26.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3870 (unknown 0 0 1800800535) 0x7f3680005f30 con 0x7f36700090a0 2024-09-16T14:46:26.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f368cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:46:26.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f368cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:26.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 >> v1:172.21.15.189:6789/0 conn(0x7f36700090a0 legacy=0x7f3670009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:26.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f368cff9640 1 -- 172.21.15.26:0/2257453823 >> v1:172.21.15.189:6789/0 conn(0x7f36700090a0 legacy=0x7f3670009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:46:26.051 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:46:26.051 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:46:26.051 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:46:26.051 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:46:26.052 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 40, 2024-09-16T14:46:26.052 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:46:26.052 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:46:26.052 INFO:teuthology.orchestra.run.smithi026.stdout: 1 2024-09-16T14:46:26.052 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:26.052 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:46:26.052 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:46:26.052 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:46:26.052 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:46:26.052 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:26.052 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:26.053 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:26.053 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:26.053 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:26.053 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:26.053 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:26.053 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:26.053 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:26.053 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:26.053 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:26.053 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:46:26.053 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:46:26.053 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:26.057 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:26.057 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:26.057 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:26.057 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:26.057 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:26.058 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:26.058 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:26.058 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:26.058 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:26.058 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:26.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:26.058 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:46:26.058 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:46:26.058 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:46:26.058 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:46:26.058 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:46:26.058 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:46:26.059 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:46:26.059 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:46:26.059 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:46:26.059 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:46:26.059 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:46:26.059 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:46:26.059 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:46:26.059 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:46:26.059 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:46:26.059 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:46:26.059 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:46:26.059 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:46:26.059 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:46:26.060 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:46:26.060 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:46:26.060 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:46:26.060 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:46:26.060 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:46:26.060 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:46:26.060 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:46:26.060 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:46:26.060 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:26.060 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:46:26.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:26.060 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:46:26.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:26.061 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:46:26.061 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:46:26.061 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:26.061 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:26.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:26.061 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:26.061 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:46:26.061 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:26.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:26.061 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:26.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:26.061 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:46:26.061 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:46:26.061 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:26.062 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:26.062 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:26.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:26.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:26.062 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:46:26.062 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:46:26.062 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:26.062 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:26.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:26.062 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:26.062 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:46:26.062 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:26.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:26.063 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:26.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:26.063 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:46:26.063 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:46:26.063 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:26.063 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:26.063 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:26.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:26.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:26.063 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:46:26.063 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:46:26.063 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:46:26.063 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:46:26.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:26.064 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:46:26.064 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:46:26.064 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:46:26.064 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:26.064 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:26.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:26.064 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:46:26.064 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:46:26.064 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:46:26.064 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:46:26.064 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:46:26.064 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:26.065 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:26.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:26.065 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:46:26.065 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:46:26.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:26.065 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:26.065 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:26.065 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:46:26.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:26.065 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:46:26.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:46:26.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:46:26.065 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:46:26.066 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:46:26.066 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:46:26.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:46:26.066 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:46:26.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:46:26.066 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:46:26.066 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:46:26.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f369695a640 1 -- 172.21.15.26:0/2257453823 >> v1:172.21.15.189:6812/3196161582 conn(0x7f367c0360e0 legacy=0x7f367c0385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:26.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f369695a640 10 monclient: shutdown 2024-09-16T14:46:26.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f369695a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:26.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f369695a640 1 -- 172.21.15.26:0/2257453823 >> v1:172.21.15.26:6789/0 conn(0x7f369009cb80 legacy=0x7f369012df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:26.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f3694ef7640 1 -- 172.21.15.26:0/2257453823 reap_dead start 2024-09-16T14:46:26.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f3694ef7640 1 -- 172.21.15.26:0/2257453823 reap_dead start 2024-09-16T14:46:26.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f369695a640 1 -- 172.21.15.26:0/2257453823 shutdown_connections 2024-09-16T14:46:26.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f369695a640 1 -- 172.21.15.26:0/2257453823 >> 172.21.15.26:0/2257453823 conn(0x7f369009fb50 msgr2=0x7f3690070bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:26.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f369695a640 1 -- 172.21.15.26:0/2257453823 shutdown_connections 2024-09-16T14:46:26.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:46:26.036+0000 7f369695a640 1 -- 172.21.15.26:0/2257453823 wait complete. 2024-09-16T14:46:26.068 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:46:26.069 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:46:26.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:26.551+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v78) v1 ==== 128+0+0 (unknown 2690968142 0 0) 0x7f7348014720 con 0x7f7358134d50 2024-09-16T14:46:26.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:26.551+0000 7f734dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T14:46:26.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:26.551+0000 7f734dffb640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'host2' from crush map 2024-09-16T14:46:26.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:26.655+0000 7f735f8f8640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T14:46:26.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:26.655+0000 7f735f8f8640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T14:46:26.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:26.655+0000 7f735f8f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:26.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:26.655+0000 7f735f8f8640 1 -- 172.21.15.189:0/656627782 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host2"} v 0) v1 -- 0x7f73380055a0 con 0x7f7358134d50 2024-09-16T14:46:27.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.555+0000 7f734dffb640 1 -- 172.21.15.189:0/656627782 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v79) v1 ==== 128+0+0 (unknown 3388351724 0 0) 0x7f7348014900 con 0x7f7358134d50 2024-09-16T14:46:27.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.555+0000 7f734dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T14:46:27.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.555+0000 7f734dffb640 10 monclient: _finish_command 3 = system:0 removed item id -16 name 'host2' from crush map 2024-09-16T14:46:27.560 INFO:tasks.workunit.client.0.smithi189.stderr:removed item id -16 name 'host2' from crush map 2024-09-16T14:46:27.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.559+0000 7f735f8f8640 1 -- 172.21.15.189:0/656627782 >> v1:172.21.15.189:6812/3196161582 conn(0x7f733c0360e0 legacy=0x7f733c0385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:27.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.559+0000 7f735f8f8640 10 monclient: shutdown 2024-09-16T14:46:27.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.559+0000 7f735f8f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:27.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.559+0000 7f735f8f8640 1 -- 172.21.15.189:0/656627782 >> v1:172.21.15.189:6789/0 conn(0x7f7358134d50 legacy=0x7f7358139960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:27.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.559+0000 7f735f8f8640 1 -- 172.21.15.189:0/656627782 shutdown_connections 2024-09-16T14:46:27.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.559+0000 7f735f8f8640 1 -- 172.21.15.189:0/656627782 >> 172.21.15.189:0/656627782 conn(0x7f73580a3320 msgr2=0x7f7358134250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:27.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.559+0000 7f735f8f8640 1 -- 172.21.15.189:0/656627782 shutdown_connections 2024-09-16T14:46:27.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.559+0000 7f735f8f8640 1 -- 172.21.15.189:0/656627782 wait complete. 2024-09-16T14:46:27.580 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket host-for-test host root=root-for-test rack=rack-for-test 2024-09-16T14:46:27.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.699+0000 7f534eaee640 10 monclient: get_monmap_and_config 2024-09-16T14:46:27.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.699+0000 7f534eaee640 10 monclient: build_initial_monmap 2024-09-16T14:46:27.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.699+0000 7f534eaee640 10 monclient: monmap: 2024-09-16T14:46:27.704 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:27.704 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:27.704 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:27.704195+0000 2024-09-16T14:46:27.704 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:27.704195+0000 2024-09-16T14:46:27.704 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:27.704 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:27.705 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:27.705 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:27.705 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:27.705 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:27.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.699+0000 7f534eaee640 1 Processor -- start 2024-09-16T14:46:27.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.699+0000 7f534eaee640 1 -- start start 2024-09-16T14:46:27.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.699+0000 7f534eaee640 10 monclient: init 2024-09-16T14:46:27.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.699+0000 7f534eaee640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:27.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.699+0000 7f534eaee640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:27.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.699+0000 7f534eaee640 10 monclient(hunting): picked mon.noname-b con 0x7f5348070830 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:27.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.699+0000 7f534eaee640 10 monclient(hunting): picked mon.noname-a con 0x7f5348071320 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:27.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.699+0000 7f534eaee640 10 monclient(hunting): picked mon.noname-c con 0x7f5348134cf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:27.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.699+0000 7f534eaee640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5348135740 con 0x7f5348071320 2024-09-16T14:46:27.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.699+0000 7f534eaee640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5348135160 con 0x7f5348134cf0 2024-09-16T14:46:27.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.699+0000 7f534eaee640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f534809d810 con 0x7f5348070830 2024-09-16T14:46:27.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.699+0000 7f534eaee640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:27.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.699+0000 7f534eaee640 10 monclient(hunting): authenticate will time out at 1169.819912s 2024-09-16T14:46:27.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534c88a640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f5348070830 legacy=0x7f5348070c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:27.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534d08b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5348134cf0 0x7f534813d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39394/0 (socket says 172.21.15.189:39394) 2024-09-16T14:46:27.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534d08b640 1 -- 172.21.15.189:0/680271323 learned_addr learned my addr 172.21.15.189:0/680271323 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:27.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 1 -- 172.21.15.189:0/680271323 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436324923 0 0) 0x7f5348135160 con 0x7f5348134cf0 2024-09-16T14:46:27.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:27.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:27.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:27.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 10 monclient(hunting): my global_id is 94567 2024-09-16T14:46:27.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 1 -- 172.21.15.189:0/680271323 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5330003530 con 0x7f5348134cf0 2024-09-16T14:46:27.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 1 -- 172.21.15.189:0/680271323 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1242831829 0 0) 0x7f5330003530 con 0x7f5348134cf0 2024-09-16T14:46:27.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 1 -- 172.21.15.189:0/680271323 >> v1:172.21.15.26:6790/0 conn(0x7f5348070830 legacy=0x7f5348070c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:27.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 1 -- 172.21.15.189:0/680271323 >> v1:172.21.15.26:6789/0 conn(0x7f5348071320 legacy=0x7f5348139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:27.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:27.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 1 monclient: found mon.noname-c 2024-09-16T14:46:27.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:27.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:27.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 1 -- 172.21.15.189:0/680271323 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534809f1c0 con 0x7f5348134cf0 2024-09-16T14:46:27.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:27.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:27.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:27.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 1 -- 172.21.15.189:0/680271323 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f53400019a0 con 0x7f5348134cf0 2024-09-16T14:46:27.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:27.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:27.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 10 monclient: dump: 2024-09-16T14:46:27.711 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:27.711 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:27.711 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:27.711 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:27.711 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:27.711 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:27.711 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:27.711 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:27.711 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:27.711 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:27.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 5 monclient: authenticate success, global_id 94567 2024-09-16T14:46:27.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:27.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 1 -- 172.21.15.189:0/680271323 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5340001cc0 con 0x7f5348134cf0 2024-09-16T14:46:27.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:27.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 1 -- 172.21.15.189:0/680271323 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5340002090 con 0x7f5348134cf0 2024-09-16T14:46:27.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:27.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:27.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53477fe640 10 monclient: dump: 2024-09-16T14:46:27.712 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:27.712 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:27.712 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:27.712 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:27.712 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:27.712 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:27.712 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:27.713 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:27.713 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:27.713 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:27.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:27.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 4 set_mon_vals no callback set 2024-09-16T14:46:27.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 10 monclient: shutdown 2024-09-16T14:46:27.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:27.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 1 -- 172.21.15.189:0/680271323 >> v1:172.21.15.189:6789/0 conn(0x7f5348134cf0 legacy=0x7f534813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:27.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 1 -- 172.21.15.189:0/680271323 shutdown_connections 2024-09-16T14:46:27.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 1 -- 172.21.15.189:0/680271323 >> 172.21.15.189:0/680271323 conn(0x7f53480a3380 msgr2=0x7f534809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:27.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 1 -- 172.21.15.189:0/680271323 shutdown_connections 2024-09-16T14:46:27.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 1 -- 172.21.15.189:0/680271323 wait complete. 2024-09-16T14:46:27.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 10 monclient: build_initial_monmap 2024-09-16T14:46:27.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 10 monclient: monmap: 2024-09-16T14:46:27.714 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:27.714 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:27.714 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:27.714 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:27.714 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:27.714 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:27.714 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:27.715 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:27.715 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:27.715 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:27.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 1 Processor -- start 2024-09-16T14:46:27.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 1 -- start start 2024-09-16T14:46:27.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 10 monclient: init 2024-09-16T14:46:27.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:27.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:27.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 10 monclient(hunting): picked mon.noname-b con 0x7f5348070830 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:27.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 10 monclient(hunting): picked mon.noname-a con 0x7f5348071320 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:27.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 10 monclient(hunting): picked mon.noname-c con 0x7f5348134cf0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:27.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f534811e290 con 0x7f5348071320 2024-09-16T14:46:27.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5348132c10 con 0x7f5348070830 2024-09-16T14:46:27.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53481326e0 con 0x7f5348134cf0 2024-09-16T14:46:27.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:27.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 10 monclient(hunting): authenticate will time out at 1169.822889s 2024-09-16T14:46:27.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534c88a640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5348070830 0x7f53481308b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39402/0 (socket says 172.21.15.189:39402) 2024-09-16T14:46:27.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534c88a640 1 -- 172.21.15.189:0/1994240408 learned_addr learned my addr 172.21.15.189:0/1994240408 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:27.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534d08b640 1 -- 172.21.15.189:0/1994240408 >> v1:172.21.15.26:6790/0 conn(0x7f5348134cf0 legacy=0x7f53481d1a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:27.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 1 -- 172.21.15.189:0/1994240408 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122365723 0 0) 0x7f5348132c10 con 0x7f5348070830 2024-09-16T14:46:27.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:27.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:27.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:27.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 10 monclient(hunting): my global_id is 94570 2024-09-16T14:46:27.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 1 -- 172.21.15.189:0/1994240408 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5334003530 con 0x7f5348070830 2024-09-16T14:46:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 1 -- 172.21.15.189:0/1994240408 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 770839350 0 0) 0x7f5334003530 con 0x7f5348070830 2024-09-16T14:46:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 1 -- 172.21.15.189:0/1994240408 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5348132c10 con 0x7f5348070830 2024-09-16T14:46:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 1 -- 172.21.15.189:0/1994240408 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5338004190 con 0x7f5348070830 2024-09-16T14:46:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 10 monclient(hunting): dump: 2024-09-16T14:46:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:27.721 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:27.721 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:27.721 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:27.721 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:27.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:27.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 1 -- 172.21.15.189:0/1994240408 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2608811012 0 0) 0x7f5348132c10 con 0x7f5348070830 2024-09-16T14:46:27.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 1 -- 172.21.15.189:0/1994240408 >> v1:172.21.15.26:6790/0 conn(0x7f5348134cf0 legacy=0x7f53481d1a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:27.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 1 -- 172.21.15.189:0/1994240408 >> v1:172.21.15.26:6789/0 conn(0x7f5348071320 legacy=0x7f53481ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:27.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 10 monclient: _finish_hunting 0 2024-09-16T14:46:27.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 1 monclient: found mon.b 2024-09-16T14:46:27.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:27.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:27.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 1 -- 172.21.15.189:0/1994240408 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534806b320 con 0x7f5348070830 2024-09-16T14:46:27.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 10 monclient: _finish_auth 0 2024-09-16T14:46:27.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 10 monclient: _check_auth_tickets 2024-09-16T14:46:27.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:27.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 5 monclient: authenticate success, global_id 94570 2024-09-16T14:46:27.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 10 monclient: _renew_subs 2024-09-16T14:46:27.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:27.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 1 -- 172.21.15.189:0/1994240408 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f534811dfb0 con 0x7f5348070830 2024-09-16T14:46:27.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 10 monclient: _renew_subs 2024-09-16T14:46:27.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:27.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f534eaee640 1 -- 172.21.15.189:0/1994240408 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f534811bf30 con 0x7f5348070830 2024-09-16T14:46:27.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 1 -- 172.21.15.189:0/1994240408 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53380044b0 con 0x7f5348070830 2024-09-16T14:46:27.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:27.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 1 -- 172.21.15.189:0/1994240408 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5338004840 con 0x7f5348070830 2024-09-16T14:46:27.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:27.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:27.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53467fc640 4 set_mon_vals no callback set 2024-09-16T14:46:27.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.703+0000 7f53457fa640 10 monclient: dump: 2024-09-16T14:46:27.724 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:27.724 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:27.724 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:27.724 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:27.724 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:27.724 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:27.724 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:27.724 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:27.724 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:27.724 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:27.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.707+0000 7f53457fa640 1 -- 172.21.15.189:0/1994240408 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5338004a20 con 0x7f5348070830 2024-09-16T14:46:27.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.707+0000 7f534eaee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:27.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.707+0000 7f534eaee640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:27.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.707+0000 7f534eaee640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:27.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.707+0000 7f534eaee640 1 -- 172.21.15.189:0/1994240408 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5328005050 con 0x7f5348070830 2024-09-16T14:46:27.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.707+0000 7f53457fa640 1 -- 172.21.15.189:0/1994240408 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(79..79 src has 15..79) v4 ==== 5133+0+0 (unknown 1444272615 0 0) 0x7f53380438b0 con 0x7f5348070830 2024-09-16T14:46:27.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.707+0000 7f5347fff640 10 monclient: get_auth_request con 0x7f5334036010 auth_method 0 2024-09-16T14:46:27.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.711+0000 7f53457fa640 1 -- 172.21.15.189:0/1994240408 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5338014840 con 0x7f5348070830 2024-09-16T14:46:27.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.711+0000 7f53457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:27.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.711+0000 7f53457fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:27.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.871+0000 7f534eaee640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T14:46:27.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.871+0000 7f534eaee640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T14:46:27.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.871+0000 7f534eaee640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:27.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:27.871+0000 7f534eaee640 1 -- 172.21.15.189:0/1994240408 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]} v 0) v1 -- 0x7f5328005300 con 0x7f5348070830 2024-09-16T14:46:28.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.567+0000 7f53457fa640 1 -- 172.21.15.189:0/1994240408 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}]=0 added bucket host-for-test type host to location {rack=rack-for-test,root=root-for-test} v80) v1 ==== 251+0+0 (unknown 3118541439 0 0) 0x7f5338012cb0 con 0x7f5348070830 2024-09-16T14:46:28.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.567+0000 7f53457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T14:46:28.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.567+0000 7f53457fa640 10 monclient: _finish_command 2 = system:0 added bucket host-for-test type host to location {rack=rack-for-test,root=root-for-test} 2024-09-16T14:46:28.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.663+0000 7f534eaee640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T14:46:28.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.663+0000 7f534eaee640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T14:46:28.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.663+0000 7f534eaee640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:28.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.663+0000 7f534eaee640 1 -- 172.21.15.189:0/1994240408 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]} v 0) v1 -- 0x7f5328005b60 con 0x7f5348070830 2024-09-16T14:46:28.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.663+0000 7f53457fa640 1 -- 172.21.15.189:0/1994240408 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}]=0 bucket 'host-for-test' already exists v80) v1 ==== 200+0+0 (unknown 1413236190 0 0) 0x7f5338014430 con 0x7f5348070830 2024-09-16T14:46:28.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.663+0000 7f53457fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T14:46:28.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.663+0000 7f53457fa640 10 monclient: _finish_command 3 = system:0 bucket 'host-for-test' already exists 2024-09-16T14:46:28.669 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'host-for-test' already exists 2024-09-16T14:46:28.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.667+0000 7f534eaee640 1 -- 172.21.15.189:0/1994240408 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5334036010 legacy=0x7f53340384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:28.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.667+0000 7f534eaee640 10 monclient: shutdown 2024-09-16T14:46:28.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.667+0000 7f534eaee640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:28.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.667+0000 7f534eaee640 1 -- 172.21.15.189:0/1994240408 >> v1:172.21.15.189:6789/0 conn(0x7f5348070830 legacy=0x7f53481308b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:28.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.667+0000 7f534eaee640 1 -- 172.21.15.189:0/1994240408 shutdown_connections 2024-09-16T14:46:28.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.667+0000 7f534eaee640 1 -- 172.21.15.189:0/1994240408 >> 172.21.15.189:0/1994240408 conn(0x7f53480a3380 msgr2=0x7f5348131760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:28.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.667+0000 7f534eaee640 1 -- 172.21.15.189:0/1994240408 shutdown_connections 2024-09-16T14:46:28.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.667+0000 7f534eaee640 1 -- 172.21.15.189:0/1994240408 wait complete. 2024-09-16T14:46:28.686 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T14:46:28.686 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep host-for-test 2024-09-16T14:46:28.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126f388640 10 monclient: get_monmap_and_config 2024-09-16T14:46:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126f388640 10 monclient: build_initial_monmap 2024-09-16T14:46:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126f388640 10 monclient: monmap: 2024-09-16T14:46:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:28.799896+0000 2024-09-16T14:46:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:28.799896+0000 2024-09-16T14:46:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:28.801 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:28.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126f388640 1 Processor -- start 2024-09-16T14:46:28.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126f388640 1 -- start start 2024-09-16T14:46:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126f388640 10 monclient: init 2024-09-16T14:46:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126f388640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126f388640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126f388640 10 monclient(hunting): picked mon.noname-c con 0x7f12681318a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126f388640 10 monclient(hunting): picked mon.noname-a con 0x7f1268134d50 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126f388640 10 monclient(hunting): picked mon.noname-b con 0x7f126813b2b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126f388640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126809d8a0 con 0x7f1268134d50 2024-09-16T14:46:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126f388640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126809ddf0 con 0x7f12681318a0 2024-09-16T14:46:28.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126f388640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126809e8b0 con 0x7f126813b2b0 2024-09-16T14:46:28.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126f388640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:28.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126f388640 10 monclient(hunting): authenticate will time out at 1170.915559s 2024-09-16T14:46:28.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126d124640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f12681318a0 0x7f1268131c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39412/0 (socket says 172.21.15.189:39412) 2024-09-16T14:46:28.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126d124640 1 -- 172.21.15.189:0/2103820278 learned_addr learned my addr 172.21.15.189:0/2103820278 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:28.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f126d925640 1 -- 172.21.15.189:0/2103820278 >> v1:172.21.15.26:6790/0 conn(0x7f126813b2b0 legacy=0x7f126813d6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:28.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f125ffff640 1 -- 172.21.15.189:0/2103820278 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2980234228 0 0) 0x7f126809ddf0 con 0x7f12681318a0 2024-09-16T14:46:28.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f125ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:28.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f125ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:28.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f125ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:28.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.795+0000 7f125ffff640 10 monclient(hunting): my global_id is 94573 2024-09-16T14:46:28.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 1 -- 172.21.15.189:0/2103820278 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1254003530 con 0x7f12681318a0 2024-09-16T14:46:28.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 1 -- 172.21.15.189:0/2103820278 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3151620779 0 0) 0x7f1254003530 con 0x7f12681318a0 2024-09-16T14:46:28.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 1 -- 172.21.15.189:0/2103820278 >> v1:172.21.15.26:6790/0 conn(0x7f126813b2b0 legacy=0x7f126813d6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:28.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 1 -- 172.21.15.189:0/2103820278 >> v1:172.21.15.26:6789/0 conn(0x7f1268134d50 legacy=0x7f1268139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:28.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:28.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 1 monclient: found mon.noname-c 2024-09-16T14:46:28.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:28.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:28.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 1 -- 172.21.15.189:0/2103820278 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126809eb90 con 0x7f12681318a0 2024-09-16T14:46:28.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 10 monclient: _finish_auth 0 2024-09-16T14:46:28.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:28.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:28.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 5 monclient: authenticate success, global_id 94573 2024-09-16T14:46:28.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:28.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 1 -- 172.21.15.189:0/2103820278 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1264001620 con 0x7f12681318a0 2024-09-16T14:46:28.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:28.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:28.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 10 monclient: dump: 2024-09-16T14:46:28.816 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:28.816 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:28.816 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:28.816 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:28.817 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:28.817 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:28.817 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:28.817 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:28.817 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:28.818 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:28.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:28.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 1 -- 172.21.15.189:0/2103820278 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12640018d0 con 0x7f12681318a0 2024-09-16T14:46:28.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:28.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:28.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 4 set_mon_vals no callback set 2024-09-16T14:46:28.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 1 -- 172.21.15.189:0/2103820278 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1264001c60 con 0x7f12681318a0 2024-09-16T14:46:28.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:28.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:28.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125ffff640 10 monclient: dump: 2024-09-16T14:46:28.820 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:28.820 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:28.820 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:28.821 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:28.821 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:28.821 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:28.822 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:28.822 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:28.823 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:28.823 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:28.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 10 monclient: shutdown 2024-09-16T14:46:28.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:28.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 1 -- 172.21.15.189:0/2103820278 >> v1:172.21.15.189:6789/0 conn(0x7f12681318a0 legacy=0x7f1268131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:28.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 1 -- 172.21.15.189:0/2103820278 shutdown_connections 2024-09-16T14:46:28.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 1 -- 172.21.15.189:0/2103820278 >> 172.21.15.189:0/2103820278 conn(0x7f12680a3320 msgr2=0x7f126809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:28.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 1 -- 172.21.15.189:0/2103820278 shutdown_connections 2024-09-16T14:46:28.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 1 -- 172.21.15.189:0/2103820278 wait complete. 2024-09-16T14:46:28.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 10 monclient: build_initial_monmap 2024-09-16T14:46:28.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 10 monclient: monmap: 2024-09-16T14:46:28.824 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:28.825 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:28.825 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:28.825 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:28.825 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:28.825 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:28.825 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:28.825 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:28.825 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:28.825 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:28.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 1 Processor -- start 2024-09-16T14:46:28.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 1 -- start start 2024-09-16T14:46:28.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 10 monclient: init 2024-09-16T14:46:28.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:28.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:28.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 10 monclient(hunting): picked mon.noname-a con 0x7f12681318a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:28.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 10 monclient(hunting): picked mon.noname-c con 0x7f1268134d50 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:28.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 10 monclient(hunting): picked mon.noname-b con 0x7f126813b2b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:28.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126811a1d0 con 0x7f12681318a0 2024-09-16T14:46:28.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126813ac40 con 0x7f126813b2b0 2024-09-16T14:46:28.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126806b380 con 0x7f1268134d50 2024-09-16T14:46:28.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:28.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 10 monclient(hunting): authenticate will time out at 1170.918671s 2024-09-16T14:46:28.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126d925640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f126813b2b0 0x7f12681d1e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39420/0 (socket says 172.21.15.189:39420) 2024-09-16T14:46:28.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126d925640 1 -- 172.21.15.189:0/3097919445 learned_addr learned my addr 172.21.15.189:0/3097919445 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:28.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126c923640 1 -- 172.21.15.189:0/3097919445 >> v1:172.21.15.26:6790/0 conn(0x7f1268134d50 legacy=0x7f12681ce750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:28.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12832097 0 0) 0x7f126813ac40 con 0x7f126813b2b0 2024-09-16T14:46:28.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:28.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:28.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:28.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient(hunting): my global_id is 94576 2024-09-16T14:46:28.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f124c003530 con 0x7f126813b2b0 2024-09-16T14:46:28.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3442948507 0 0) 0x7f124c003530 con 0x7f126813b2b0 2024-09-16T14:46:28.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f126813ac40 con 0x7f126813b2b0 2024-09-16T14:46:28.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1260004420 con 0x7f126813b2b0 2024-09-16T14:46:28.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:28.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:28.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient(hunting): dump: 2024-09-16T14:46:28.828 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:28.828 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:28.828 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:28.828 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:28.828 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:28.828 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:28.829 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:28.829 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:28.829 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:28.829 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:28.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:28.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835860644 0 0) 0x7f126811a1d0 con 0x7f12681318a0 2024-09-16T14:46:28.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:28.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:28.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:28.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient(hunting): my global_id is 104226 2024-09-16T14:46:28.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f124c003530 con 0x7f12681318a0 2024-09-16T14:46:28.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3020539260 0 0) 0x7f126813ac40 con 0x7f126813b2b0 2024-09-16T14:46:28.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 >> v1:172.21.15.26:6790/0 conn(0x7f1268134d50 legacy=0x7f12681ce750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:28.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 >> v1:172.21.15.26:6789/0 conn(0x7f12681318a0 legacy=0x7f12681c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:28.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:46:28.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 1 monclient: found mon.b 2024-09-16T14:46:28.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:28.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:28.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12680a1d20 con 0x7f126813b2b0 2024-09-16T14:46:28.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient: _finish_auth 0 2024-09-16T14:46:28.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:28.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:28.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 5 monclient: authenticate success, global_id 94576 2024-09-16T14:46:28.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 10 monclient: _renew_subs 2024-09-16T14:46:28.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:28.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 1 -- 172.21.15.189:0/3097919445 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126811b850 con 0x7f126813b2b0 2024-09-16T14:46:28.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 10 monclient: _renew_subs 2024-09-16T14:46:28.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:28.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f126f388640 1 -- 172.21.15.189:0/3097919445 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f126811a5c0 con 0x7f126813b2b0 2024-09-16T14:46:28.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1260004740 con 0x7f126813b2b0 2024-09-16T14:46:28.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:28.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1260004ad0 con 0x7f126813b2b0 2024-09-16T14:46:28.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:28.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:28.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125f7fe640 4 set_mon_vals no callback set 2024-09-16T14:46:28.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.799+0000 7f125dffb640 10 monclient: dump: 2024-09-16T14:46:28.832 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:28.832 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:28.832 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:28.833 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:28.833 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:28.833 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:28.833 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:28.833 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:28.833 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:28.833 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:28.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.803+0000 7f126f388640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:28.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.803+0000 7f126f388640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:28.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.803+0000 7f126f388640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:28.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.803+0000 7f126f388640 1 -- 172.21.15.189:0/3097919445 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1248005050 con 0x7f126813b2b0 2024-09-16T14:46:28.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.803+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1260004c70 con 0x7f126813b2b0 2024-09-16T14:46:28.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.803+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(80..80 src has 15..80) v4 ==== 5658+0+0 (unknown 3739197995 0 0) 0x7f1260043cd0 con 0x7f126813b2b0 2024-09-16T14:46:28.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.803+0000 7f126d124640 10 monclient: get_auth_request con 0x7f124c036130 auth_method 0 2024-09-16T14:46:28.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.807+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1260005470 con 0x7f126813b2b0 2024-09-16T14:46:28.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.807+0000 7f125dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:28.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.807+0000 7f125dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:28.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.967+0000 7f126f388640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:46:28.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.967+0000 7f126f388640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T14:46:28.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.967+0000 7f126f388640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:28.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.967+0000 7f126f388640 1 -- 172.21.15.189:0/3097919445 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f1248005b10 con 0x7f126813b2b0 2024-09-16T14:46:28.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.967+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v80) v1 ==== 56+0+1500 (unknown 1137015514 0 2924518876) 0x7f1260012ec0 con 0x7f126813b2b0 2024-09-16T14:46:28.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.967+0000 7f125dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T14:46:28.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:28.967+0000 7f125dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:29.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.059+0000 7f126f388640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:46:29.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.059+0000 7f126f388640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T14:46:29.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.059+0000 7f126f388640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.059+0000 7f126f388640 1 -- 172.21.15.189:0/3097919445 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f12480055a0 con 0x7f126813b2b0 2024-09-16T14:46:29.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.059+0000 7f125dffb640 1 -- 172.21.15.189:0/3097919445 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v80) v1 ==== 56+0+1500 (unknown 1137015514 0 2924518876) 0x7f1260014820 con 0x7f126813b2b0 2024-09-16T14:46:29.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.059+0000 7f125dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T14:46:29.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.059+0000 7f125dffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:46:29.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.063+0000 7f126f388640 1 -- 172.21.15.189:0/3097919445 >> v1:172.21.15.189:6812/3196161582 conn(0x7f124c036130 legacy=0x7f124c0385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:29.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.063+0000 7f126f388640 10 monclient: shutdown 2024-09-16T14:46:29.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.063+0000 7f126f388640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:29.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.063+0000 7f126f388640 1 -- 172.21.15.189:0/3097919445 >> v1:172.21.15.189:6789/0 conn(0x7f126813b2b0 legacy=0x7f12681d1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:29.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.063+0000 7f126f388640 1 -- 172.21.15.189:0/3097919445 shutdown_connections 2024-09-16T14:46:29.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.063+0000 7f126f388640 1 -- 172.21.15.189:0/3097919445 >> 172.21.15.189:0/3097919445 conn(0x7f12680a3320 msgr2=0x7f1268134110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:29.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.063+0000 7f126f388640 1 -- 172.21.15.189:0/3097919445 shutdown_connections 2024-09-16T14:46:29.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.063+0000 7f126f388640 1 -- 172.21.15.189:0/3097919445 wait complete. 2024-09-16T14:46:29.081 INFO:tasks.workunit.client.0.smithi189.stdout:-11 0 host host-for-test 2024-09-16T14:46:29.082 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T14:46:29.082 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep rack-for-test 2024-09-16T14:46:29.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bdcd72640 10 monclient: get_monmap_and_config 2024-09-16T14:46:29.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bdcd72640 10 monclient: build_initial_monmap 2024-09-16T14:46:29.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bdcd72640 10 monclient: monmap: 2024-09-16T14:46:29.196 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:29.196 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:29.196 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:29.195948+0000 2024-09-16T14:46:29.196 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:29.195948+0000 2024-09-16T14:46:29.196 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:29.196 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:29.197 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:29.197 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:29.197 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:29.197 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bdcd72640 1 Processor -- start 2024-09-16T14:46:29.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bdcd72640 1 -- start start 2024-09-16T14:46:29.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bdcd72640 10 monclient: init 2024-09-16T14:46:29.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bdcd72640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:29.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bdcd72640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:29.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bdcd72640 10 monclient(hunting): picked mon.noname-b con 0x7f3bd809df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:29.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bdcd72640 10 monclient(hunting): picked mon.noname-c con 0x7f3bd809e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:29.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bdcd72640 10 monclient(hunting): picked mon.noname-a con 0x7f3bd809e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:29.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bdcd72640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd809ee20 con 0x7f3bd809e740 2024-09-16T14:46:29.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bdcd72640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd809f100 con 0x7f3bd809e2f0 2024-09-16T14:46:29.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bdcd72640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd809f3e0 con 0x7f3bd809df10 2024-09-16T14:46:29.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bdcd72640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:29.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bdcd72640 10 monclient(hunting): authenticate will time out at 1171.311751s 2024-09-16T14:46:29.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bd659c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f3bd809df10 legacy=0x7f3bd809d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:29.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bd5d9b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3bd809e2f0 0x7f3bd81378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43600/0 (socket says 172.21.15.189:43600) 2024-09-16T14:46:29.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.191+0000 7f3bd5d9b640 1 -- 172.21.15.189:0/1506823761 learned_addr learned my addr 172.21.15.189:0/1506823761 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:29.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 1 -- 172.21.15.189:0/1506823761 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517872080 0 0) 0x7f3bd809f100 con 0x7f3bd809e2f0 2024-09-16T14:46:29.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 10 monclient(hunting): _negotiate 2024-09-16T14:46:29.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:29.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:29.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 10 monclient(hunting): my global_id is 94579 2024-09-16T14:46:29.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 1 -- 172.21.15.189:0/1506823761 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bc4003530 con 0x7f3bd809e2f0 2024-09-16T14:46:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 1 -- 172.21.15.189:0/1506823761 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 514445316 0 0) 0x7f3bc4003530 con 0x7f3bd809e2f0 2024-09-16T14:46:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 1 -- 172.21.15.189:0/1506823761 >> v1:172.21.15.26:6790/0 conn(0x7f3bd809df10 legacy=0x7f3bd809d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 1 -- 172.21.15.189:0/1506823761 >> v1:172.21.15.26:6789/0 conn(0x7f3bd809e740 legacy=0x7f3bd813b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 10 monclient: _finish_hunting 0 2024-09-16T14:46:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 1 monclient: found mon.noname-c 2024-09-16T14:46:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 1 -- 172.21.15.189:0/1506823761 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd809c690 con 0x7f3bd809e2f0 2024-09-16T14:46:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 10 monclient: _finish_auth 0 2024-09-16T14:46:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 10 monclient: _check_auth_tickets 2024-09-16T14:46:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 1 -- 172.21.15.189:0/1506823761 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3bc0001620 con 0x7f3bd809e2f0 2024-09-16T14:46:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 5 monclient: authenticate success, global_id 94579 2024-09-16T14:46:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 10 monclient: dump: 2024-09-16T14:46:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:29.209 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:29.209 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:29.209 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:29.209 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:29.209 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:29.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 1 -- 172.21.15.189:0/1506823761 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bc0001940 con 0x7f3bd809e2f0 2024-09-16T14:46:29.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:29.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:29.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 4 set_mon_vals no callback set 2024-09-16T14:46:29.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 1 -- 172.21.15.189:0/1506823761 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3bc0001d10 con 0x7f3bd809e2f0 2024-09-16T14:46:29.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:29.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:29.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd559a640 10 monclient: dump: 2024-09-16T14:46:29.210 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:29.210 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:29.210 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.210 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.210 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:29.210 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:29.211 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:29.211 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:29.211 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:29.211 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 10 monclient: shutdown 2024-09-16T14:46:29.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:29.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 1 -- 172.21.15.189:0/1506823761 >> v1:172.21.15.189:6789/0 conn(0x7f3bd809e2f0 legacy=0x7f3bd81378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:29.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 1 -- 172.21.15.189:0/1506823761 shutdown_connections 2024-09-16T14:46:29.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 1 -- 172.21.15.189:0/1506823761 >> 172.21.15.189:0/1506823761 conn(0x7f3bd80a3320 msgr2=0x7f3bd809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:29.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 1 -- 172.21.15.189:0/1506823761 shutdown_connections 2024-09-16T14:46:29.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 1 -- 172.21.15.189:0/1506823761 wait complete. 2024-09-16T14:46:29.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 10 monclient: build_initial_monmap 2024-09-16T14:46:29.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 10 monclient: monmap: 2024-09-16T14:46:29.212 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:29.212 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:29.212 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:29.212 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:29.212 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:29.212 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:29.212 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:29.213 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:29.213 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:29.213 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 1 Processor -- start 2024-09-16T14:46:29.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 1 -- start start 2024-09-16T14:46:29.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 10 monclient: init 2024-09-16T14:46:29.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:29.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:29.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 10 monclient(hunting): picked mon.noname-c con 0x7f3bd809df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:29.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 10 monclient(hunting): picked mon.noname-b con 0x7f3bd809e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:29.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 10 monclient(hunting): picked mon.noname-a con 0x7f3bd809e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:29.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd81353e0 con 0x7f3bd809e740 2024-09-16T14:46:29.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd806c900 con 0x7f3bd809e2f0 2024-09-16T14:46:29.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd81359a0 con 0x7f3bd809df10 2024-09-16T14:46:29.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 10 monclient(hunting): authenticate will time out at 1171.314731s 2024-09-16T14:46:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd5d9b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3bd809e2f0 0x7f3bd8132240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43602/0 (socket says 172.21.15.189:43602) 2024-09-16T14:46:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd5d9b640 1 -- 172.21.15.189:0/2053517097 learned_addr learned my addr 172.21.15.189:0/2053517097 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd659c640 1 -- 172.21.15.189:0/2053517097 >> v1:172.21.15.26:6790/0 conn(0x7f3bd809df10 legacy=0x7f3bd8131b30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342198473 0 0) 0x7f3bd806c900 con 0x7f3bd809e2f0 2024-09-16T14:46:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient(hunting): my global_id is 94582 2024-09-16T14:46:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bb8003530 con 0x7f3bd809e2f0 2024-09-16T14:46:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1566075757 0 0) 0x7f3bb8003530 con 0x7f3bd809e2f0 2024-09-16T14:46:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bd806c900 con 0x7f3bd809e2f0 2024-09-16T14:46:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3bc00015e0 con 0x7f3bd809e2f0 2024-09-16T14:46:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient(hunting): dump: 2024-09-16T14:46:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:29.217 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109759632 0 0) 0x7f3bd81353e0 con 0x7f3bd809e740 2024-09-16T14:46:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient(hunting): my global_id is 104232 2024-09-16T14:46:29.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bb8003530 con 0x7f3bd809e740 2024-09-16T14:46:29.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1137825200 0 0) 0x7f3bd806c900 con 0x7f3bd809e2f0 2024-09-16T14:46:29.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 >> v1:172.21.15.26:6790/0 conn(0x7f3bd809df10 legacy=0x7f3bd8131b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:29.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 >> v1:172.21.15.26:6789/0 conn(0x7f3bd809e740 legacy=0x7f3bd81cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:29.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:29.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 1 monclient: found mon.b 2024-09-16T14:46:29.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:29.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd806b380 con 0x7f3bd809e2f0 2024-09-16T14:46:29.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient: _finish_auth 0 2024-09-16T14:46:29.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:29.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:29.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 5 monclient: authenticate success, global_id 94582 2024-09-16T14:46:29.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 10 monclient: _renew_subs 2024-09-16T14:46:29.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 1 -- 172.21.15.189:0/2053517097 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd813b630 con 0x7f3bd809e2f0 2024-09-16T14:46:29.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 10 monclient: _renew_subs 2024-09-16T14:46:29.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bdcd72640 1 -- 172.21.15.189:0/2053517097 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd81335d0 con 0x7f3bd809e2f0 2024-09-16T14:46:29.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bc0004cf0 con 0x7f3bd809e2f0 2024-09-16T14:46:29.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:29.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3bc0001220 con 0x7f3bd809e2f0 2024-09-16T14:46:29.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:29.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:29.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bd4d99640 4 set_mon_vals no callback set 2024-09-16T14:46:29.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.195+0000 7f3bceffd640 10 monclient: dump: 2024-09-16T14:46:29.222 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:29.222 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:29.222 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.222 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.222 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:29.222 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:29.222 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:29.222 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:29.222 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:29.222 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.199+0000 7f3bdcd72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:29.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.199+0000 7f3bdcd72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:29.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.199+0000 7f3bdcd72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.199+0000 7f3bdcd72640 1 -- 172.21.15.189:0/2053517097 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bbc005050 con 0x7f3bd809e2f0 2024-09-16T14:46:29.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.199+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3bc0005100 con 0x7f3bd809e2f0 2024-09-16T14:46:29.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.199+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(80..80 src has 15..80) v4 ==== 5658+0+0 (unknown 3739197995 0 0) 0x7f3bc0043d40 con 0x7f3bd809e2f0 2024-09-16T14:46:29.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.199+0000 7f3bd659c640 10 monclient: get_auth_request con 0x7f3bb80360e0 auth_method 0 2024-09-16T14:46:29.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.203+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3bc0014be0 con 0x7f3bd809e2f0 2024-09-16T14:46:29.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.203+0000 7f3bceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:29.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.203+0000 7f3bceffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:29.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.363+0000 7f3bdcd72640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:46:29.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.363+0000 7f3bdcd72640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T14:46:29.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.363+0000 7f3bdcd72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.363+0000 7f3bdcd72640 1 -- 172.21.15.189:0/2053517097 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f3bbc005b10 con 0x7f3bd809e2f0 2024-09-16T14:46:29.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.363+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v80) v1 ==== 56+0+1500 (unknown 1137015514 0 2924518876) 0x7f3bc0013050 con 0x7f3bd809e2f0 2024-09-16T14:46:29.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.363+0000 7f3bceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T14:46:29.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.363+0000 7f3bceffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:29.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.455+0000 7f3bdcd72640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:46:29.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.455+0000 7f3bdcd72640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T14:46:29.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.455+0000 7f3bdcd72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.455+0000 7f3bdcd72640 1 -- 172.21.15.189:0/2053517097 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f3bbc0055a0 con 0x7f3bd809e2f0 2024-09-16T14:46:29.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.455+0000 7f3bceffd640 1 -- 172.21.15.189:0/2053517097 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v80) v1 ==== 56+0+1500 (unknown 1137015514 0 2924518876) 0x7f3bc00149b0 con 0x7f3bd809e2f0 2024-09-16T14:46:29.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.455+0000 7f3bceffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T14:46:29.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.455+0000 7f3bceffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:46:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.459+0000 7f3bdcd72640 1 -- 172.21.15.189:0/2053517097 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3bb80360e0 legacy=0x7f3bb80385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.459+0000 7f3bdcd72640 10 monclient: shutdown 2024-09-16T14:46:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.459+0000 7f3bdcd72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.459+0000 7f3bdcd72640 1 -- 172.21.15.189:0/2053517097 >> v1:172.21.15.189:6789/0 conn(0x7f3bd809e2f0 legacy=0x7f3bd8132240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.459+0000 7f3bdcd72640 1 -- 172.21.15.189:0/2053517097 shutdown_connections 2024-09-16T14:46:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.459+0000 7f3bdcd72640 1 -- 172.21.15.189:0/2053517097 >> 172.21.15.189:0/2053517097 conn(0x7f3bd80a3320 msgr2=0x7f3bd812b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:29.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.459+0000 7f3bdcd72640 1 -- 172.21.15.189:0/2053517097 shutdown_connections 2024-09-16T14:46:29.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.459+0000 7f3bdcd72640 1 -- 172.21.15.189:0/2053517097 wait complete. 2024-09-16T14:46:29.477 INFO:tasks.workunit.client.0.smithi189.stdout:-12 0 rack rack-for-test 2024-09-16T14:46:29.477 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep root-for-test 2024-09-16T14:46:29.477 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T14:46:29.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.583+0000 7f36410b7640 10 monclient: get_monmap_and_config 2024-09-16T14:46:29.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.583+0000 7f36410b7640 10 monclient: build_initial_monmap 2024-09-16T14:46:29.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.583+0000 7f36410b7640 10 monclient: monmap: 2024-09-16T14:46:29.589 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:29.590 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:29.590 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:29.589676+0000 2024-09-16T14:46:29.590 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:29.589676+0000 2024-09-16T14:46:29.590 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:29.590 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:29.590 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:29.590 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:29.590 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:29.590 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 1 Processor -- start 2024-09-16T14:46:29.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 1 -- start start 2024-09-16T14:46:29.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 10 monclient: init 2024-09-16T14:46:29.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:29.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:46:29.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 10 monclient(hunting): picked mon.noname-a con 0x7f363c073590 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:29.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 10 monclient(hunting): picked mon.noname-c con 0x7f363c1348a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:29.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 10 monclient(hunting): picked mon.noname-b con 0x7f363c1383b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:29.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c073970 con 0x7f363c073590 2024-09-16T14:46:29.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c131ec0 con 0x7f363c1348a0 2024-09-16T14:46:29.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c099710 con 0x7f363c1383b0 2024-09-16T14:46:29.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:29.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 10 monclient(hunting): authenticate will time out at 1171.705544s 2024-09-16T14:46:29.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f363b59e640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f363c1383b0 legacy=0x7f363c13a7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:29.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f363a59c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f363c1348a0 0x7f363c136c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43608/0 (socket says 172.21.15.189:43608) 2024-09-16T14:46:29.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f363a59c640 1 -- 172.21.15.189:0/2755117856 learned_addr learned my addr 172.21.15.189:0/2755117856 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:29.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 1 -- 172.21.15.189:0/2755117856 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566000461 0 0) 0x7f363c131ec0 con 0x7f363c1348a0 2024-09-16T14:46:29.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:46:29.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:29.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:29.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 10 monclient(hunting): my global_id is 94585 2024-09-16T14:46:29.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 1 -- 172.21.15.189:0/2755117856 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3624003530 con 0x7f363c1348a0 2024-09-16T14:46:29.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 1 -- 172.21.15.189:0/2755117856 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 419954616 0 0) 0x7f3624003530 con 0x7f363c1348a0 2024-09-16T14:46:29.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 1 -- 172.21.15.189:0/2755117856 >> v1:172.21.15.26:6790/0 conn(0x7f363c1383b0 legacy=0x7f363c13a7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:29.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 1 -- 172.21.15.189:0/2755117856 >> v1:172.21.15.26:6789/0 conn(0x7f363c073590 legacy=0x7f363c09cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:29.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:46:29.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 1 monclient: found mon.noname-c 2024-09-16T14:46:29.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:29.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 1 -- 172.21.15.189:0/2755117856 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c0999f0 con 0x7f363c1348a0 2024-09-16T14:46:29.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 10 monclient: _finish_auth 0 2024-09-16T14:46:29.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:46:29.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:29.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 5 monclient: authenticate success, global_id 94585 2024-09-16T14:46:29.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 1 -- 172.21.15.189:0/2755117856 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f36300019a0 con 0x7f363c1348a0 2024-09-16T14:46:29.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:29.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:29.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 10 monclient: dump: 2024-09-16T14:46:29.596 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:29.596 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:29.596 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.597 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.597 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:29.597 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:29.597 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:29.597 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:29.597 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:29.597 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:29.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 1 -- 172.21.15.189:0/2755117856 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3630001cc0 con 0x7f363c1348a0 2024-09-16T14:46:29.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:29.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 1 -- 172.21.15.189:0/2755117856 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3630002050 con 0x7f363c1348a0 2024-09-16T14:46:29.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:29.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:29.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f3639d9b640 10 monclient: dump: 2024-09-16T14:46:29.598 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:29.598 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:29.598 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.598 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.598 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:29.598 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:29.598 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:29.598 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:29.598 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:29.598 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:29.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 4 set_mon_vals no callback set 2024-09-16T14:46:29.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 10 monclient: shutdown 2024-09-16T14:46:29.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:29.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 1 -- 172.21.15.189:0/2755117856 >> v1:172.21.15.189:6789/0 conn(0x7f363c1348a0 legacy=0x7f363c136c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:29.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 1 -- 172.21.15.189:0/2755117856 shutdown_connections 2024-09-16T14:46:29.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 1 -- 172.21.15.189:0/2755117856 >> 172.21.15.189:0/2755117856 conn(0x7f363c0a0030 msgr2=0x7f363c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:29.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 1 -- 172.21.15.189:0/2755117856 shutdown_connections 2024-09-16T14:46:29.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 1 -- 172.21.15.189:0/2755117856 wait complete. 2024-09-16T14:46:29.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 10 monclient: build_initial_monmap 2024-09-16T14:46:29.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.587+0000 7f36410b7640 10 monclient: monmap: 2024-09-16T14:46:29.599 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:29.599 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:29.599 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:29.599 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:29.599 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:29.600 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:29.600 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:29.600 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:29.600 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:29.600 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 1 Processor -- start 2024-09-16T14:46:29.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 1 -- start start 2024-09-16T14:46:29.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 10 monclient: init 2024-09-16T14:46:29.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:29.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:29.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 10 monclient(hunting): picked mon.noname-b con 0x7f363c073590 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:29.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 10 monclient(hunting): picked mon.noname-a con 0x7f363c1348a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:29.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 10 monclient(hunting): picked mon.noname-c con 0x7f363c1383b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:29.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c12a130 con 0x7f363c1348a0 2024-09-16T14:46:29.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c137cb0 con 0x7f363c073590 2024-09-16T14:46:29.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c1380e0 con 0x7f363c1383b0 2024-09-16T14:46:29.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:29.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 10 monclient(hunting): authenticate will time out at 1171.708527s 2024-09-16T14:46:29.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f363ad9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f363c073590 0x7f363c1c97c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43612/0 (socket says 172.21.15.189:43612) 2024-09-16T14:46:29.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f363ad9d640 1 -- 172.21.15.189:0/3724890339 learned_addr learned my addr 172.21.15.189:0/3724890339 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:29.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f363b59e640 1 -- 172.21.15.189:0/3724890339 >> v1:172.21.15.26:6790/0 conn(0x7f363c1383b0 legacy=0x7f363c1d1f40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:29.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 1 -- 172.21.15.189:0/3724890339 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339922052 0 0) 0x7f363c137cb0 con 0x7f363c073590 2024-09-16T14:46:29.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:29.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:29.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:29.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 10 monclient(hunting): my global_id is 94588 2024-09-16T14:46:29.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 1 -- 172.21.15.189:0/3724890339 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3628003530 con 0x7f363c073590 2024-09-16T14:46:29.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 1 -- 172.21.15.189:0/3724890339 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 612554810 0 0) 0x7f3628003530 con 0x7f363c073590 2024-09-16T14:46:29.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 1 -- 172.21.15.189:0/3724890339 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f363c137cb0 con 0x7f363c073590 2024-09-16T14:46:29.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 1 -- 172.21.15.189:0/3724890339 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f362c0018e0 con 0x7f363c073590 2024-09-16T14:46:29.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:29.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:29.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 10 monclient(hunting): dump: 2024-09-16T14:46:29.605 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:29.605 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:29.605 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.605 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.605 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:29.605 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:29.606 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:29.606 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:29.606 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:29.606 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:29.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 1 -- 172.21.15.189:0/3724890339 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3822450922 0 0) 0x7f363c137cb0 con 0x7f363c073590 2024-09-16T14:46:29.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 1 -- 172.21.15.189:0/3724890339 >> v1:172.21.15.26:6790/0 conn(0x7f363c1383b0 legacy=0x7f363c1d1f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:29.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 1 -- 172.21.15.189:0/3724890339 >> v1:172.21.15.26:6789/0 conn(0x7f363c1348a0 legacy=0x7f363c1ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:29.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:29.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 1 monclient: found mon.b 2024-09-16T14:46:29.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:29.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 1 -- 172.21.15.189:0/3724890339 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c09a550 con 0x7f363c073590 2024-09-16T14:46:29.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:29.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:29.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:29.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 5 monclient: authenticate success, global_id 94588 2024-09-16T14:46:29.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 10 monclient: _renew_subs 2024-09-16T14:46:29.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 1 -- 172.21.15.189:0/3724890339 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f363c12bc90 con 0x7f363c073590 2024-09-16T14:46:29.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 10 monclient: _renew_subs 2024-09-16T14:46:29.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 1 -- 172.21.15.189:0/3724890339 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f363c129e00 con 0x7f363c073590 2024-09-16T14:46:29.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 1 -- 172.21.15.189:0/3724890339 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f362c001cb0 con 0x7f363c073590 2024-09-16T14:46:29.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:29.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 1 -- 172.21.15.189:0/3724890339 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f362c003a70 con 0x7f363c073590 2024-09-16T14:46:29.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:29.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f363959a640 4 set_mon_vals no callback set 2024-09-16T14:46:29.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:29.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 10 monclient: dump: 2024-09-16T14:46:29.608 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:29.608 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:29.608 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.608 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.608 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:29.608 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:29.608 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:29.608 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:29.608 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:29.608 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 1 -- 172.21.15.189:0/3724890339 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f362c003c50 con 0x7f363c073590 2024-09-16T14:46:29.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:29.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:29.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36410b7640 1 -- 172.21.15.189:0/3724890339 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361c005050 con 0x7f363c073590 2024-09-16T14:46:29.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f36237fe640 1 -- 172.21.15.189:0/3724890339 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(81..81 src has 15..81) v4 ==== 5658+0+0 (unknown 3493545696 0 0) 0x7f362c043a90 con 0x7f363c073590 2024-09-16T14:46:29.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.591+0000 7f363a59c640 10 monclient: get_auth_request con 0x7f3628036050 auth_method 0 2024-09-16T14:46:29.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.595+0000 7f36237fe640 1 -- 172.21.15.189:0/3724890339 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f362c014810 con 0x7f363c073590 2024-09-16T14:46:29.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.595+0000 7f36237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:29.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.595+0000 7f36237fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:29.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.743+0000 7f36410b7640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:46:29.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.743+0000 7f36410b7640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T14:46:29.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.743+0000 7f36410b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.743+0000 7f36410b7640 1 -- 172.21.15.189:0/3724890339 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f361c005b10 con 0x7f363c073590 2024-09-16T14:46:29.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.743+0000 7f36237fe640 1 -- 172.21.15.189:0/3724890339 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (unknown 3717179170 0 2924518876) 0x7f362c0145e0 con 0x7f363c073590 2024-09-16T14:46:29.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.743+0000 7f36237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T14:46:29.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.743+0000 7f36237fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:29.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.839+0000 7f36410b7640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:46:29.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.839+0000 7f36410b7640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T14:46:29.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.839+0000 7f36410b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.839+0000 7f36410b7640 1 -- 172.21.15.189:0/3724890339 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f361c0055a0 con 0x7f363c073590 2024-09-16T14:46:29.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.839+0000 7f36237fe640 1 -- 172.21.15.189:0/3724890339 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (unknown 3717179170 0 2924518876) 0x7f362c015140 con 0x7f363c073590 2024-09-16T14:46:29.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.839+0000 7f36237fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T14:46:29.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.839+0000 7f36237fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:46:29.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.839+0000 7f36410b7640 1 -- 172.21.15.189:0/3724890339 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3628036050 legacy=0x7f3628038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:29.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.839+0000 7f36410b7640 10 monclient: shutdown 2024-09-16T14:46:29.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.839+0000 7f36410b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:29.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.839+0000 7f36410b7640 1 -- 172.21.15.189:0/3724890339 >> v1:172.21.15.189:6789/0 conn(0x7f363c073590 legacy=0x7f363c1c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:29.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.839+0000 7f36410b7640 1 -- 172.21.15.189:0/3724890339 shutdown_connections 2024-09-16T14:46:29.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.839+0000 7f36410b7640 1 -- 172.21.15.189:0/3724890339 >> 172.21.15.189:0/3724890339 conn(0x7f363c0a0030 msgr2=0x7f363c09ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:29.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.839+0000 7f36410b7640 1 -- 172.21.15.189:0/3724890339 shutdown_connections 2024-09-16T14:46:29.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.839+0000 7f36410b7640 1 -- 172.21.15.189:0/3724890339 wait complete. 2024-09-16T14:46:29.860 INFO:tasks.workunit.client.0.smithi189.stdout:-14 0 root root-for-test 2024-09-16T14:46:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm host-for-test 2024-09-16T14:46:29.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 10 monclient: get_monmap_and_config 2024-09-16T14:46:29.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 10 monclient: build_initial_monmap 2024-09-16T14:46:29.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 10 monclient: monmap: 2024-09-16T14:46:29.975 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:29.975 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:29.975 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:29.974464+0000 2024-09-16T14:46:29.975 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:29.974464+0000 2024-09-16T14:46:29.975 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:29.975 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:29.975 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:29.975 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:29.975 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:29.975 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 1 Processor -- start 2024-09-16T14:46:29.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 1 -- start start 2024-09-16T14:46:29.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 10 monclient: init 2024-09-16T14:46:29.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:29.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:29.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 10 monclient(hunting): picked mon.noname-a con 0x7fd34c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:29.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 10 monclient(hunting): picked mon.noname-b con 0x7fd34c1334d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:29.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 10 monclient(hunting): picked mon.noname-c con 0x7fd34c136fe0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:29.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c09b350 con 0x7fd34c099a70 2024-09-16T14:46:29.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c09f2e0 con 0x7fd34c136fe0 2024-09-16T14:46:29.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c09b070 con 0x7fd34c1334d0 2024-09-16T14:46:29.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:29.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 10 monclient(hunting): authenticate will time out at 1172.090600s 2024-09-16T14:46:29.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3511e6640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fd34c1334d0 legacy=0x7fd34c1358c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:29.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3521e8640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd34c136fe0 0x7fd34c1393d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43626/0 (socket says 172.21.15.189:43626) 2024-09-16T14:46:29.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3521e8640 1 -- 172.21.15.189:0/4175300882 learned_addr learned my addr 172.21.15.189:0/4175300882 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:29.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 1 -- 172.21.15.189:0/4175300882 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108568094 0 0) 0x7fd34c09f2e0 con 0x7fd34c136fe0 2024-09-16T14:46:29.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 10 monclient(hunting): _negotiate 2024-09-16T14:46:29.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:29.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:29.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 10 monclient(hunting): my global_id is 94591 2024-09-16T14:46:29.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 1 -- 172.21.15.189:0/4175300882 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd338003450 con 0x7fd34c136fe0 2024-09-16T14:46:29.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 1 -- 172.21.15.189:0/4175300882 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 411178461 0 0) 0x7fd338003450 con 0x7fd34c136fe0 2024-09-16T14:46:29.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 1 -- 172.21.15.189:0/4175300882 >> v1:172.21.15.26:6790/0 conn(0x7fd34c1334d0 legacy=0x7fd34c1358c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:29.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 1 -- 172.21.15.189:0/4175300882 >> v1:172.21.15.26:6789/0 conn(0x7fd34c099a70 legacy=0x7fd34c09bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:29.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 10 monclient: _finish_hunting 0 2024-09-16T14:46:29.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 1 monclient: found mon.noname-c 2024-09-16T14:46:29.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:29.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 1 -- 172.21.15.189:0/4175300882 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c09b630 con 0x7fd34c136fe0 2024-09-16T14:46:29.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 10 monclient: _finish_auth 0 2024-09-16T14:46:29.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 10 monclient: _check_auth_tickets 2024-09-16T14:46:29.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:29.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 1 -- 172.21.15.189:0/4175300882 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd3440019a0 con 0x7fd34c136fe0 2024-09-16T14:46:29.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 5 monclient: authenticate success, global_id 94591 2024-09-16T14:46:29.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:29.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:29.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:29.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 10 monclient: dump: 2024-09-16T14:46:29.986 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:29.986 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:29.986 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.986 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.986 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:29.986 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:29.986 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:29.986 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:29.986 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:29.986 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:29.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 1 -- 172.21.15.189:0/4175300882 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd344001c50 con 0x7fd34c136fe0 2024-09-16T14:46:29.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:29.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 1 -- 172.21.15.189:0/4175300882 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd344001fe0 con 0x7fd34c136fe0 2024-09-16T14:46:29.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:29.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 4 set_mon_vals no callback set 2024-09-16T14:46:29.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:29.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:29.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd3509e5640 10 monclient: dump: 2024-09-16T14:46:29.987 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:29.987 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:29.987 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.987 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.987 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:29.988 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:29.988 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:29.988 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:29.988 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:29.988 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 10 monclient: shutdown 2024-09-16T14:46:29.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:29.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.971+0000 7fd353c4b640 1 -- 172.21.15.189:0/4175300882 >> v1:172.21.15.189:6789/0 conn(0x7fd34c136fe0 legacy=0x7fd34c1393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:29.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 1 -- 172.21.15.189:0/4175300882 shutdown_connections 2024-09-16T14:46:29.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 1 -- 172.21.15.189:0/4175300882 >> 172.21.15.189:0/4175300882 conn(0x7fd34c0a3320 msgr2=0x7fd34c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:29.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 1 -- 172.21.15.189:0/4175300882 shutdown_connections 2024-09-16T14:46:29.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 1 -- 172.21.15.189:0/4175300882 wait complete. 2024-09-16T14:46:29.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 monclient: build_initial_monmap 2024-09-16T14:46:29.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 monclient: monmap: 2024-09-16T14:46:29.989 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:29.989 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:29.989 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:29.989 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:29.990 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:29.990 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:29.990 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:29.990 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:29.990 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:29.990 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 1 Processor -- start 2024-09-16T14:46:29.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 1 -- start start 2024-09-16T14:46:29.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 monclient: init 2024-09-16T14:46:29.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:29.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:46:29.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 monclient(hunting): picked mon.noname-a con 0x7fd34c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:29.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 monclient(hunting): picked mon.noname-b con 0x7fd34c1334d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:29.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 monclient(hunting): picked mon.noname-c con 0x7fd34c136fe0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:29.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c06d240 con 0x7fd34c099a70 2024-09-16T14:46:29.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c0a1b10 con 0x7fd34c1334d0 2024-09-16T14:46:29.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c1362b0 con 0x7fd34c136fe0 2024-09-16T14:46:29.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:29.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 monclient(hunting): authenticate will time out at 1172.093636s 2024-09-16T14:46:29.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3511e6640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd34c1334d0 0x7fd34c06cb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43638/0 (socket says 172.21.15.189:43638) 2024-09-16T14:46:29.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3511e6640 1 -- 172.21.15.189:0/4161824275 learned_addr learned my addr 172.21.15.189:0/4161824275 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:29.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3521e8640 1 -- 172.21.15.189:0/4161824275 >> v1:172.21.15.26:6790/0 conn(0x7fd34c136fe0 legacy=0x7fd34c1cd4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:29.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 1 -- 172.21.15.189:0/4161824275 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489915452 0 0) 0x7fd34c0a1b10 con 0x7fd34c1334d0 2024-09-16T14:46:29.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:29.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:29.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:29.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 10 monclient(hunting): my global_id is 94594 2024-09-16T14:46:29.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 1 -- 172.21.15.189:0/4161824275 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd330003530 con 0x7fd34c1334d0 2024-09-16T14:46:29.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 1 -- 172.21.15.189:0/4161824275 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3562641744 0 0) 0x7fd330003530 con 0x7fd34c1334d0 2024-09-16T14:46:29.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 1 -- 172.21.15.189:0/4161824275 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd34c0a1b10 con 0x7fd34c1334d0 2024-09-16T14:46:29.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 1 -- 172.21.15.189:0/4161824275 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd33c0041e0 con 0x7fd34c1334d0 2024-09-16T14:46:29.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:29.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:29.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 10 monclient(hunting): dump: 2024-09-16T14:46:29.993 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:29.993 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:29.993 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.993 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.993 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:29.993 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:29.993 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:29.993 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:29.993 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:29.994 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:29.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 1 -- 172.21.15.189:0/4161824275 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3075509662 0 0) 0x7fd34c0a1b10 con 0x7fd34c1334d0 2024-09-16T14:46:29.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 1 -- 172.21.15.189:0/4161824275 >> v1:172.21.15.26:6790/0 conn(0x7fd34c136fe0 legacy=0x7fd34c1cd4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:29.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 1 -- 172.21.15.189:0/4161824275 >> v1:172.21.15.26:6789/0 conn(0x7fd34c099a70 legacy=0x7fd34c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:29.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 10 monclient: _finish_hunting 0 2024-09-16T14:46:29.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 1 monclient: found mon.b 2024-09-16T14:46:29.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:29.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 1 -- 172.21.15.189:0/4161824275 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c06b380 con 0x7fd34c1334d0 2024-09-16T14:46:29.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 10 monclient: _finish_auth 0 2024-09-16T14:46:29.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 10 monclient: _check_auth_tickets 2024-09-16T14:46:29.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:29.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 5 monclient: authenticate success, global_id 94594 2024-09-16T14:46:29.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 monclient: _renew_subs 2024-09-16T14:46:29.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 1 -- 172.21.15.189:0/4161824275 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34c06ef20 con 0x7fd34c1334d0 2024-09-16T14:46:29.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 monclient: _renew_subs 2024-09-16T14:46:29.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 1 -- 172.21.15.189:0/4161824275 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34c06d790 con 0x7fd34c1334d0 2024-09-16T14:46:29.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 1 -- 172.21.15.189:0/4161824275 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd33c004500 con 0x7fd34c1334d0 2024-09-16T14:46:29.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:29.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 1 -- 172.21.15.189:0/4161824275 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd33c004890 con 0x7fd34c1334d0 2024-09-16T14:46:29.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:29.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:29.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 10 monclient: dump: 2024-09-16T14:46:29.996 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:29.996 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:29.996 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.996 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:29.996 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:29.996 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:29.996 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:29.996 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:29.997 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:29.997 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:29.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd343fff640 4 set_mon_vals no callback set 2024-09-16T14:46:29.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:29.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:29.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:29.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 1 -- 172.21.15.189:0/4161824275 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd33c004a70 con 0x7fd34c1334d0 2024-09-16T14:46:29.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd353c4b640 1 -- 172.21.15.189:0/4161824275 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd334005050 con 0x7fd34c1334d0 2024-09-16T14:46:29.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3427fc640 1 -- 172.21.15.189:0/4161824275 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(81..81 src has 15..81) v4 ==== 5658+0+0 (unknown 3493545696 0 0) 0x7fd33c043af0 con 0x7fd34c1334d0 2024-09-16T14:46:29.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.975+0000 7fd3519e7640 10 monclient: get_auth_request con 0x7fd330035fe0 auth_method 0 2024-09-16T14:46:29.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.979+0000 7fd3427fc640 1 -- 172.21.15.189:0/4161824275 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd33c014870 con 0x7fd34c1334d0 2024-09-16T14:46:29.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.979+0000 7fd3427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:29.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:29.979+0000 7fd3427fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:30.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.131+0000 7fd353c4b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T14:46:30.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.131+0000 7fd353c4b640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T14:46:30.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.131+0000 7fd353c4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:30.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.131+0000 7fd353c4b640 1 -- 172.21.15.189:0/4161824275 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host-for-test"} v 0) v1 -- 0x7fd334005300 con 0x7fd34c1334d0 2024-09-16T14:46:30.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.595+0000 7fd3427fc640 1 -- 172.21.15.189:0/4161824275 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host-for-test"}]=0 removed item id -11 name 'host-for-test' from crush map v82) v1 ==== 140+0+0 (unknown 1658408989 0 0) 0x7fd33c012ce0 con 0x7fd34c1334d0 2024-09-16T14:46:30.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.595+0000 7fd3427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T14:46:30.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.595+0000 7fd3427fc640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host-for-test' from crush map 2024-09-16T14:46:30.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.695+0000 7fd353c4b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T14:46:30.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.695+0000 7fd353c4b640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T14:46:30.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.695+0000 7fd353c4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:30.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.695+0000 7fd353c4b640 1 -- 172.21.15.189:0/4161824275 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host-for-test"} v 0) v1 -- 0x7fd3340055a0 con 0x7fd34c1334d0 2024-09-16T14:46:30.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.699+0000 7fd3427fc640 1 -- 172.21.15.189:0/4161824275 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host-for-test"}]=0 device 'host-for-test' does not appear in the crush map v82) v1 ==== 140+0+0 (unknown 2378121819 0 0) 0x7fd33c014640 con 0x7fd34c1334d0 2024-09-16T14:46:30.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.699+0000 7fd3427fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T14:46:30.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.699+0000 7fd3427fc640 10 monclient: _finish_command 3 = system:0 device 'host-for-test' does not appear in the crush map 2024-09-16T14:46:30.702 INFO:tasks.workunit.client.0.smithi189.stderr:device 'host-for-test' does not appear in the crush map 2024-09-16T14:46:30.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.699+0000 7fd353c4b640 1 -- 172.21.15.189:0/4161824275 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd330035fe0 legacy=0x7fd3300384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:30.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.699+0000 7fd353c4b640 10 monclient: shutdown 2024-09-16T14:46:30.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.699+0000 7fd353c4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:30.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.699+0000 7fd353c4b640 1 -- 172.21.15.189:0/4161824275 >> v1:172.21.15.189:6789/0 conn(0x7fd34c1334d0 legacy=0x7fd34c06cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:30.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.699+0000 7fd353c4b640 1 -- 172.21.15.189:0/4161824275 shutdown_connections 2024-09-16T14:46:30.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.699+0000 7fd353c4b640 1 -- 172.21.15.189:0/4161824275 >> 172.21.15.189:0/4161824275 conn(0x7fd34c0a3320 msgr2=0x7fd34c09d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:30.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.699+0000 7fd353c4b640 1 -- 172.21.15.189:0/4161824275 shutdown_connections 2024-09-16T14:46:30.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.699+0000 7fd353c4b640 1 -- 172.21.15.189:0/4161824275 wait complete. 2024-09-16T14:46:30.718 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm rack-for-test 2024-09-16T14:46:30.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 10 monclient: get_monmap_and_config 2024-09-16T14:46:30.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 10 monclient: build_initial_monmap 2024-09-16T14:46:30.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 10 monclient: monmap: 2024-09-16T14:46:30.831 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:30.831 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:30.831 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:30.830811+0000 2024-09-16T14:46:30.831 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:30.830811+0000 2024-09-16T14:46:30.831 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:30.831 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:30.831 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:30.831 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:30.832 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:30.832 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:30.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 1 Processor -- start 2024-09-16T14:46:30.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 1 -- start start 2024-09-16T14:46:30.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 10 monclient: init 2024-09-16T14:46:30.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:30.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:30.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 10 monclient(hunting): picked mon.noname-b con 0x7f15a809df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:30.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 10 monclient(hunting): picked mon.noname-c con 0x7f15a809e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:30.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 10 monclient(hunting): picked mon.noname-a con 0x7f15a809e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:30.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a809ee20 con 0x7f15a809e740 2024-09-16T14:46:30.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a809f100 con 0x7f15a809e2f0 2024-09-16T14:46:30.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a809f3e0 con 0x7f15a809df10 2024-09-16T14:46:30.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:30.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 10 monclient(hunting): authenticate will time out at 1172.946505s 2024-09-16T14:46:30.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a6ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f15a809e2f0 0x7f15a81378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43650/0 (socket says 172.21.15.189:43650) 2024-09-16T14:46:30.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a6ffd640 1 -- 172.21.15.189:0/4032029603 learned_addr learned my addr 172.21.15.189:0/4032029603 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:30.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a77fe640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f15a809df10 legacy=0x7f15a809d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:30.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 1 -- 172.21.15.189:0/4032029603 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514552488 0 0) 0x7f15a809f100 con 0x7f15a809e2f0 2024-09-16T14:46:30.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:30.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:30.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:30.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 10 monclient(hunting): my global_id is 94597 2024-09-16T14:46:30.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 1 -- 172.21.15.189:0/4032029603 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1590003530 con 0x7f15a809e2f0 2024-09-16T14:46:30.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 1 -- 172.21.15.189:0/4032029603 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 287495756 0 0) 0x7f1590003530 con 0x7f15a809e2f0 2024-09-16T14:46:30.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 1 -- 172.21.15.189:0/4032029603 >> v1:172.21.15.26:6790/0 conn(0x7f15a809df10 legacy=0x7f15a809d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:30.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 1 -- 172.21.15.189:0/4032029603 >> v1:172.21.15.26:6789/0 conn(0x7f15a809e740 legacy=0x7f15a813b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:30.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:46:30.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 1 monclient: found mon.noname-c 2024-09-16T14:46:30.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:30.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:30.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 1 -- 172.21.15.189:0/4032029603 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a809c690 con 0x7f15a809e2f0 2024-09-16T14:46:30.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 10 monclient: _finish_auth 0 2024-09-16T14:46:30.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:46:30.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:30.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 1 -- 172.21.15.189:0/4032029603 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1598001620 con 0x7f15a809e2f0 2024-09-16T14:46:30.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:30.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:30.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 10 monclient: dump: 2024-09-16T14:46:30.842 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:30.842 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:30.842 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:30.842 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:30.842 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:30.842 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:30.842 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:30.842 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:30.842 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:30.842 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:30.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 5 monclient: authenticate success, global_id 94597 2024-09-16T14:46:30.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:30.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 1 -- 172.21.15.189:0/4032029603 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1598001940 con 0x7f15a809e2f0 2024-09-16T14:46:30.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:30.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 1 -- 172.21.15.189:0/4032029603 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1598001d10 con 0x7f15a809e2f0 2024-09-16T14:46:30.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:30.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:30.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15a67fc640 10 monclient: dump: 2024-09-16T14:46:30.843 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:30.843 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:30.843 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:30.843 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:30.843 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:30.843 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:30.843 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:30.843 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:30.843 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:30.844 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 4 set_mon_vals no callback set 2024-09-16T14:46:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 10 monclient: shutdown 2024-09-16T14:46:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 1 -- 172.21.15.189:0/4032029603 >> v1:172.21.15.189:6789/0 conn(0x7f15a809e2f0 legacy=0x7f15a81378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 1 -- 172.21.15.189:0/4032029603 shutdown_connections 2024-09-16T14:46:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 1 -- 172.21.15.189:0/4032029603 >> 172.21.15.189:0/4032029603 conn(0x7f15a80a3320 msgr2=0x7f15a809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 1 -- 172.21.15.189:0/4032029603 shutdown_connections 2024-09-16T14:46:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.827+0000 7f15ae06b640 1 -- 172.21.15.189:0/4032029603 wait complete. 2024-09-16T14:46:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 monclient: build_initial_monmap 2024-09-16T14:46:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 monclient: monmap: 2024-09-16T14:46:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:30.845 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 1 Processor -- start 2024-09-16T14:46:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 1 -- start start 2024-09-16T14:46:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 monclient: init 2024-09-16T14:46:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:30.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:30.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 monclient(hunting): picked mon.noname-a con 0x7f15a809df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:30.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 monclient(hunting): picked mon.noname-c con 0x7f15a809e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:30.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 monclient(hunting): picked mon.noname-b con 0x7f15a809e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:30.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a81c8550 con 0x7f15a809df10 2024-09-16T14:46:30.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a81222b0 con 0x7f15a809e740 2024-09-16T14:46:30.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a806b380 con 0x7f15a809e2f0 2024-09-16T14:46:30.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:30.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 monclient(hunting): authenticate will time out at 1172.949329s 2024-09-16T14:46:30.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15a7fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f15a809e740 0x7f15a81cf990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43654/0 (socket says 172.21.15.189:43654) 2024-09-16T14:46:30.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15a6ffd640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f15a809e2f0 legacy=0x7f15a81c7e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:30.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15a7fff640 1 -- 172.21.15.189:0/2201223886 learned_addr learned my addr 172.21.15.189:0/2201223886 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:30.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 1 -- 172.21.15.189:0/2201223886 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360646388 0 0) 0x7f15a81222b0 con 0x7f15a809e740 2024-09-16T14:46:30.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:30.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:30.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:30.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 10 monclient(hunting): my global_id is 94600 2024-09-16T14:46:30.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 1 -- 172.21.15.189:0/2201223886 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1594003530 con 0x7f15a809e740 2024-09-16T14:46:30.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 1 -- 172.21.15.189:0/2201223886 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4067390202 0 0) 0x7f1594003530 con 0x7f15a809e740 2024-09-16T14:46:30.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 1 -- 172.21.15.189:0/2201223886 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15a81222b0 con 0x7f15a809e740 2024-09-16T14:46:30.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 1 -- 172.21.15.189:0/2201223886 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f15a0001c50 con 0x7f15a809e740 2024-09-16T14:46:30.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:30.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:30.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 10 monclient(hunting): dump: 2024-09-16T14:46:30.847 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:30.847 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:30.848 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:30.848 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:30.848 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:30.848 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:30.848 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:30.848 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:30.848 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:30.848 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:30.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:30.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 1 -- 172.21.15.189:0/2201223886 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1586446730 0 0) 0x7f15a81222b0 con 0x7f15a809e740 2024-09-16T14:46:30.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 1 -- 172.21.15.189:0/2201223886 >> v1:172.21.15.26:6790/0 conn(0x7f15a809e2f0 legacy=0x7f15a81c7e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:30.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 1 -- 172.21.15.189:0/2201223886 >> v1:172.21.15.26:6789/0 conn(0x7f15a809df10 legacy=0x7f15a81c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:30.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 1 monclient: found mon.b 2024-09-16T14:46:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 1 -- 172.21.15.189:0/2201223886 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a806cdd0 con 0x7f15a809e740 2024-09-16T14:46:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 10 monclient: _finish_auth 0 2024-09-16T14:46:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 5 monclient: authenticate success, global_id 94600 2024-09-16T14:46:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 monclient: _renew_subs 2024-09-16T14:46:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 1 -- 172.21.15.189:0/2201223886 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a8123710 con 0x7f15a809e740 2024-09-16T14:46:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 monclient: _renew_subs 2024-09-16T14:46:30.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:30.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 1 -- 172.21.15.189:0/2201223886 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a8122e70 con 0x7f15a809e740 2024-09-16T14:46:30.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:30.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:30.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:30.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15ae06b640 1 -- 172.21.15.189:0/2201223886 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a8123060 con 0x7f15a809e740 2024-09-16T14:46:30.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 1 -- 172.21.15.189:0/2201223886 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15a0002020 con 0x7f15a809e740 2024-09-16T14:46:30.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:30.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 1 -- 172.21.15.189:0/2201223886 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f15a0003df0 con 0x7f15a809e740 2024-09-16T14:46:30.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:30.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:30.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 10 monclient: dump: 2024-09-16T14:46:30.850 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:30.851 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:30.851 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:30.851 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:30.851 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:30.851 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:30.851 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:30.851 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:30.851 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:30.851 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:30.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15a5ffb640 4 set_mon_vals no callback set 2024-09-16T14:46:30.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 1 -- 172.21.15.189:0/2201223886 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f15a0003f50 con 0x7f15a809e740 2024-09-16T14:46:30.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f158ffff640 1 -- 172.21.15.189:0/2201223886 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(82..82 src has 15..82) v4 ==== 5475+0+0 (unknown 2774079880 0 0) 0x7f15a0043b90 con 0x7f15a809e740 2024-09-16T14:46:30.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.831+0000 7f15a77fe640 10 monclient: get_auth_request con 0x7f1594036060 auth_method 0 2024-09-16T14:46:30.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.835+0000 7f158ffff640 1 -- 172.21.15.189:0/2201223886 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f15a0043e60 con 0x7f15a809e740 2024-09-16T14:46:30.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.835+0000 7f158ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:30.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.835+0000 7f158ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:30.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.987+0000 7f15ae06b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T14:46:30.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.987+0000 7f15ae06b640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T14:46:30.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.987+0000 7f15ae06b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:30.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:30.987+0000 7f15ae06b640 1 -- 172.21.15.189:0/2201223886 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "rack-for-test"} v 0) v1 -- 0x7f15a811c170 con 0x7f15a809e740 2024-09-16T14:46:31.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.599+0000 7f158ffff640 1 -- 172.21.15.189:0/2201223886 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "rack-for-test"}]=0 removed item id -12 name 'rack-for-test' from crush map v83) v1 ==== 140+0+0 (unknown 293873728 0 0) 0x7f15a0012e90 con 0x7f15a809e740 2024-09-16T14:46:31.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.599+0000 7f158ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T14:46:31.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.599+0000 7f158ffff640 10 monclient: _finish_command 2 = system:0 removed item id -12 name 'rack-for-test' from crush map 2024-09-16T14:46:31.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.695+0000 7f15ae06b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T14:46:31.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.695+0000 7f15ae06b640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T14:46:31.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.695+0000 7f15ae06b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:31.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.695+0000 7f15ae06b640 1 -- 172.21.15.189:0/2201223886 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "rack-for-test"} v 0) v1 -- 0x7f15a81213b0 con 0x7f15a809e740 2024-09-16T14:46:31.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.695+0000 7f158ffff640 1 -- 172.21.15.189:0/2201223886 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "rack-for-test"}]=0 device 'rack-for-test' does not appear in the crush map v83) v1 ==== 140+0+0 (unknown 32484356 0 0) 0x7f15a0015350 con 0x7f15a809e740 2024-09-16T14:46:31.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.695+0000 7f158ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T14:46:31.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.695+0000 7f158ffff640 10 monclient: _finish_command 3 = system:0 device 'rack-for-test' does not appear in the crush map 2024-09-16T14:46:31.700 INFO:tasks.workunit.client.0.smithi189.stderr:device 'rack-for-test' does not appear in the crush map 2024-09-16T14:46:31.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.695+0000 7f15ae06b640 1 -- 172.21.15.189:0/2201223886 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1594036060 legacy=0x7f1594038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:31.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.695+0000 7f15ae06b640 10 monclient: shutdown 2024-09-16T14:46:31.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.695+0000 7f15ae06b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:31.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.699+0000 7f15ae06b640 1 -- 172.21.15.189:0/2201223886 >> v1:172.21.15.189:6789/0 conn(0x7f15a809e740 legacy=0x7f15a81cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:31.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.699+0000 7f15ae06b640 1 -- 172.21.15.189:0/2201223886 shutdown_connections 2024-09-16T14:46:31.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.699+0000 7f15ae06b640 1 -- 172.21.15.189:0/2201223886 >> 172.21.15.189:0/2201223886 conn(0x7f15a80a3320 msgr2=0x7f15a8119340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:31.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.699+0000 7f15ae06b640 1 -- 172.21.15.189:0/2201223886 shutdown_connections 2024-09-16T14:46:31.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.699+0000 7f15ae06b640 1 -- 172.21.15.189:0/2201223886 wait complete. 2024-09-16T14:46:31.716 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm root-for-test 2024-09-16T14:46:31.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.827+0000 7fdb640f5640 10 monclient: get_monmap_and_config 2024-09-16T14:46:31.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.827+0000 7fdb640f5640 10 monclient: build_initial_monmap 2024-09-16T14:46:31.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.827+0000 7fdb640f5640 10 monclient: monmap: 2024-09-16T14:46:31.833 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:31.833 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:31.833 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:31.832655+0000 2024-09-16T14:46:31.833 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:31.832655+0000 2024-09-16T14:46:31.833 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:31.833 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:31.833 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:31.833 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:31.833 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:31.833 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:31.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 1 Processor -- start 2024-09-16T14:46:31.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 1 -- start start 2024-09-16T14:46:31.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 10 monclient: init 2024-09-16T14:46:31.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:31.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:31.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 10 monclient(hunting): picked mon.noname-b con 0x7fdb5c132b80 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:31.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 10 monclient(hunting): picked mon.noname-a con 0x7fdb5c09f290 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:31.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 10 monclient(hunting): picked mon.noname-c con 0x7fdb5c139190 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:31.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c133670 con 0x7fdb5c09f290 2024-09-16T14:46:31.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c071400 con 0x7fdb5c139190 2024-09-16T14:46:31.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c09d2c0 con 0x7fdb5c132b80 2024-09-16T14:46:31.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:31.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 10 monclient(hunting): authenticate will time out at 1173.949147s 2024-09-16T14:46:31.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb62692640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdb5c139190 0x7fdb5c13b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43660/0 (socket says 172.21.15.189:43660) 2024-09-16T14:46:31.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb62692640 1 -- 172.21.15.189:0/2566973817 learned_addr learned my addr 172.21.15.189:0/2566973817 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:31.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb61e91640 1 -- 172.21.15.189:0/2566973817 >> v1:172.21.15.26:6790/0 conn(0x7fdb5c132b80 legacy=0x7fdb5c132f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:31.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 1 -- 172.21.15.189:0/2566973817 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293271780 0 0) 0x7fdb5c071400 con 0x7fdb5c139190 2024-09-16T14:46:31.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 10 monclient(hunting): _negotiate 2024-09-16T14:46:31.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:31.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:31.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 10 monclient(hunting): my global_id is 94603 2024-09-16T14:46:31.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 1 -- 172.21.15.189:0/2566973817 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb48003530 con 0x7fdb5c139190 2024-09-16T14:46:31.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 1 -- 172.21.15.189:0/2566973817 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2075166450 0 0) 0x7fdb48003530 con 0x7fdb5c139190 2024-09-16T14:46:31.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 1 -- 172.21.15.189:0/2566973817 >> v1:172.21.15.26:6790/0 conn(0x7fdb5c132b80 legacy=0x7fdb5c132f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:31.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 1 -- 172.21.15.189:0/2566973817 >> v1:172.21.15.26:6789/0 conn(0x7fdb5c09f290 legacy=0x7fdb5c137a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:31.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 10 monclient: _finish_hunting 0 2024-09-16T14:46:31.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 1 monclient: found mon.noname-c 2024-09-16T14:46:31.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:31.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:31.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 1 -- 172.21.15.189:0/2566973817 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c09cd00 con 0x7fdb5c139190 2024-09-16T14:46:31.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 10 monclient: _finish_auth 0 2024-09-16T14:46:31.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 10 monclient: _check_auth_tickets 2024-09-16T14:46:31.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:31.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 1 -- 172.21.15.189:0/2566973817 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdb540019a0 con 0x7fdb5c139190 2024-09-16T14:46:31.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:31.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:31.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 10 monclient: dump: 2024-09-16T14:46:31.840 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:31.840 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:31.840 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:31.840 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:31.840 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:31.840 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:31.840 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:31.840 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:31.840 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:31.840 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:31.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 5 monclient: authenticate success, global_id 94603 2024-09-16T14:46:31.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:31.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 1 -- 172.21.15.189:0/2566973817 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb54001cc0 con 0x7fdb5c139190 2024-09-16T14:46:31.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:31.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 1 -- 172.21.15.189:0/2566973817 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdb54002050 con 0x7fdb5c139190 2024-09-16T14:46:31.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:31.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 4 set_mon_vals no callback set 2024-09-16T14:46:31.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:31.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:31.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb60e8f640 10 monclient: dump: 2024-09-16T14:46:31.841 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:31.842 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:31.842 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:31.842 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:31.842 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:31.842 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:31.842 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:31.842 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:31.842 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:31.842 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:31.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 10 monclient: shutdown 2024-09-16T14:46:31.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:31.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 1 -- 172.21.15.189:0/2566973817 >> v1:172.21.15.189:6789/0 conn(0x7fdb5c139190 legacy=0x7fdb5c13b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:31.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 1 -- 172.21.15.189:0/2566973817 shutdown_connections 2024-09-16T14:46:31.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 1 -- 172.21.15.189:0/2566973817 >> 172.21.15.189:0/2566973817 conn(0x7fdb5c09fd20 msgr2=0x7fdb5c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:31.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 1 -- 172.21.15.189:0/2566973817 shutdown_connections 2024-09-16T14:46:31.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 1 -- 172.21.15.189:0/2566973817 wait complete. 2024-09-16T14:46:31.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 10 monclient: build_initial_monmap 2024-09-16T14:46:31.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 10 monclient: monmap: 2024-09-16T14:46:31.847 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:31.847 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:31.848 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:31.848 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:31.848 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:31.848 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:31.848 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:31.848 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:31.848 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:31.848 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:31.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 1 Processor -- start 2024-09-16T14:46:31.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 1 -- start start 2024-09-16T14:46:31.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.831+0000 7fdb640f5640 10 monclient: init 2024-09-16T14:46:31.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:31.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:31.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 10 monclient(hunting): picked mon.noname-b con 0x7fdb5c09f290 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:31.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 10 monclient(hunting): picked mon.noname-a con 0x7fdb5c132b80 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:31.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 10 monclient(hunting): picked mon.noname-c con 0x7fdb5c139190 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:31.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c11ed20 con 0x7fdb5c132b80 2024-09-16T14:46:31.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c0a1f60 con 0x7fdb5c09f290 2024-09-16T14:46:31.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c09b370 con 0x7fdb5c139190 2024-09-16T14:46:31.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:31.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 10 monclient(hunting): authenticate will time out at 1173.952120s 2024-09-16T14:46:31.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb61e91640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdb5c09f290 0x7fdb5c09a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43676/0 (socket says 172.21.15.189:43676) 2024-09-16T14:46:31.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb61e91640 1 -- 172.21.15.189:0/1704421910 learned_addr learned my addr 172.21.15.189:0/1704421910 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:31.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb62692640 1 -- 172.21.15.189:0/1704421910 >> v1:172.21.15.26:6790/0 conn(0x7fdb5c139190 legacy=0x7fdb5c1d1590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:31.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 1 -- 172.21.15.189:0/1704421910 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287766592 0 0) 0x7fdb5c0a1f60 con 0x7fdb5c09f290 2024-09-16T14:46:31.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:31.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:31.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:31.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 10 monclient(hunting): my global_id is 94606 2024-09-16T14:46:31.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 1 -- 172.21.15.189:0/1704421910 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb40003530 con 0x7fdb5c09f290 2024-09-16T14:46:31.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 1 -- 172.21.15.189:0/1704421910 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3163599967 0 0) 0x7fdb40003530 con 0x7fdb5c09f290 2024-09-16T14:46:31.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 1 -- 172.21.15.189:0/1704421910 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb5c0a1f60 con 0x7fdb5c09f290 2024-09-16T14:46:31.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 1 -- 172.21.15.189:0/1704421910 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdb580041e0 con 0x7fdb5c09f290 2024-09-16T14:46:31.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:31.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:31.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 10 monclient(hunting): dump: 2024-09-16T14:46:31.850 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:31.851 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:31.851 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:31.851 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:31.851 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:31.851 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:31.851 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:31.851 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:31.851 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:31.851 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:31.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:31.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 1 -- 172.21.15.189:0/1704421910 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1798178158 0 0) 0x7fdb5c0a1f60 con 0x7fdb5c09f290 2024-09-16T14:46:31.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 1 -- 172.21.15.189:0/1704421910 >> v1:172.21.15.26:6790/0 conn(0x7fdb5c139190 legacy=0x7fdb5c1d1590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:31.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 1 -- 172.21.15.189:0/1704421910 >> v1:172.21.15.26:6789/0 conn(0x7fdb5c132b80 legacy=0x7fdb5c09ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:31.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 10 monclient: _finish_hunting 0 2024-09-16T14:46:31.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 1 monclient: found mon.b 2024-09-16T14:46:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 1 -- 172.21.15.189:0/1704421910 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c09b970 con 0x7fdb5c09f290 2024-09-16T14:46:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 10 monclient: _finish_auth 0 2024-09-16T14:46:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 10 monclient: _check_auth_tickets 2024-09-16T14:46:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 5 monclient: authenticate success, global_id 94606 2024-09-16T14:46:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 10 monclient: _renew_subs 2024-09-16T14:46:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 1 -- 172.21.15.189:0/1704421910 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c138b50 con 0x7fdb5c09f290 2024-09-16T14:46:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 10 monclient: _renew_subs 2024-09-16T14:46:31.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:31.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 1 -- 172.21.15.189:0/1704421910 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c11f000 con 0x7fdb5c09f290 2024-09-16T14:46:31.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 1 -- 172.21.15.189:0/1704421910 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb58004500 con 0x7fdb5c09f290 2024-09-16T14:46:31.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:31.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 1 -- 172.21.15.189:0/1704421910 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdb58004890 con 0x7fdb5c09f290 2024-09-16T14:46:31.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:31.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:31.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 10 monclient: dump: 2024-09-16T14:46:31.854 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:31.854 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:31.855 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:31.855 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:31.855 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:31.855 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:31.855 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:31.855 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:31.855 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:31.855 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:31.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb53fff640 4 set_mon_vals no callback set 2024-09-16T14:46:31.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 1 -- 172.21.15.189:0/1704421910 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fdb58004a70 con 0x7fdb5c09f290 2024-09-16T14:46:31.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:31.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:31.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:31.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb640f5640 1 -- 172.21.15.189:0/1704421910 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb44005050 con 0x7fdb5c09f290 2024-09-16T14:46:31.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb527fc640 1 -- 172.21.15.189:0/1704421910 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(83..83 src has 15..83) v4 ==== 5292+0+0 (unknown 1829918342 0 0) 0x7fdb58043980 con 0x7fdb5c09f290 2024-09-16T14:46:31.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.835+0000 7fdb61690640 10 monclient: get_auth_request con 0x7fdb40036000 auth_method 0 2024-09-16T14:46:31.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.839+0000 7fdb527fc640 1 -- 172.21.15.189:0/1704421910 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fdb58014870 con 0x7fdb5c09f290 2024-09-16T14:46:31.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.839+0000 7fdb527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:31.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.839+0000 7fdb527fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:32.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.999+0000 7fdb640f5640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T14:46:32.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.999+0000 7fdb640f5640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T14:46:32.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.999+0000 7fdb640f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:32.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:31.999+0000 7fdb640f5640 1 -- 172.21.15.189:0/1704421910 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "root-for-test"} v 0) v1 -- 0x7fdb44005300 con 0x7fdb5c09f290 2024-09-16T14:46:32.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.611+0000 7fdb527fc640 1 -- 172.21.15.189:0/1704421910 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "root-for-test"}]=0 removed item id -14 name 'root-for-test' from crush map v84) v1 ==== 140+0+0 (unknown 4032571887 0 0) 0x7fdb58012ce0 con 0x7fdb5c09f290 2024-09-16T14:46:32.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.611+0000 7fdb527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T14:46:32.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.611+0000 7fdb527fc640 10 monclient: _finish_command 2 = system:0 removed item id -14 name 'root-for-test' from crush map 2024-09-16T14:46:32.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.707+0000 7fdb640f5640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T14:46:32.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.707+0000 7fdb640f5640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T14:46:32.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.707+0000 7fdb640f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:32.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.707+0000 7fdb640f5640 1 -- 172.21.15.189:0/1704421910 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "root-for-test"} v 0) v1 -- 0x7fdb440055a0 con 0x7fdb5c09f290 2024-09-16T14:46:32.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.711+0000 7fdb527fc640 1 -- 172.21.15.189:0/1704421910 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "root-for-test"}]=0 device 'root-for-test' does not appear in the crush map v84) v1 ==== 140+0+0 (unknown 3603915341 0 0) 0x7fdb58014640 con 0x7fdb5c09f290 2024-09-16T14:46:32.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.711+0000 7fdb527fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T14:46:32.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.711+0000 7fdb527fc640 10 monclient: _finish_command 3 = system:0 device 'root-for-test' does not appear in the crush map 2024-09-16T14:46:32.714 INFO:tasks.workunit.client.0.smithi189.stderr:device 'root-for-test' does not appear in the crush map 2024-09-16T14:46:32.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.711+0000 7fdb640f5640 1 -- 172.21.15.189:0/1704421910 >> v1:172.21.15.189:6812/3196161582 conn(0x7fdb40036000 legacy=0x7fdb400384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:32.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.711+0000 7fdb640f5640 10 monclient: shutdown 2024-09-16T14:46:32.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.711+0000 7fdb640f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:32.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.711+0000 7fdb640f5640 1 -- 172.21.15.189:0/1704421910 >> v1:172.21.15.189:6789/0 conn(0x7fdb5c09f290 legacy=0x7fdb5c09a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:32.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.711+0000 7fdb640f5640 1 -- 172.21.15.189:0/1704421910 shutdown_connections 2024-09-16T14:46:32.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.711+0000 7fdb640f5640 1 -- 172.21.15.189:0/1704421910 >> 172.21.15.189:0/1704421910 conn(0x7fdb5c09fd20 msgr2=0x7fdb5c070980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:32.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.711+0000 7fdb640f5640 1 -- 172.21.15.189:0/1704421910 shutdown_connections 2024-09-16T14:46:32.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.711+0000 7fdb640f5640 1 -- 172.21.15.189:0/1704421910 wait complete. 2024-09-16T14:46:32.730 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule create-simple foo-rule foo host firstn 2024-09-16T14:46:32.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.847+0000 7f74f4b17640 10 monclient: get_monmap_and_config 2024-09-16T14:46:32.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.847+0000 7f74f4b17640 10 monclient: build_initial_monmap 2024-09-16T14:46:32.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.847+0000 7f74f4b17640 10 monclient: monmap: 2024-09-16T14:46:32.853 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:32.853 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:32.853 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:32.852868+0000 2024-09-16T14:46:32.853 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:32.852868+0000 2024-09-16T14:46:32.854 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:32.854 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:32.854 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:32.854 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:32.854 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:32.854 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:32.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.847+0000 7f74f4b17640 1 Processor -- start 2024-09-16T14:46:32.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.847+0000 7f74f4b17640 1 -- start start 2024-09-16T14:46:32.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.847+0000 7f74f4b17640 10 monclient: init 2024-09-16T14:46:32.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:32.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:32.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient(hunting): picked mon.noname-b con 0x7f74f009df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:32.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient(hunting): picked mon.noname-a con 0x7f74f009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:32.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient(hunting): picked mon.noname-c con 0x7f74f009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:32.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f009ee20 con 0x7f74f009e2f0 2024-09-16T14:46:32.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f009f100 con 0x7f74f009e740 2024-09-16T14:46:32.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f009f3e0 con 0x7f74f009df10 2024-09-16T14:46:32.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:32.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient(hunting): authenticate will time out at 1174.968635s 2024-09-16T14:46:32.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ee59c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f74f009df10 legacy=0x7f74f009d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:32.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74eed9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f74f009e740 0x7f74f013b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43678/0 (socket says 172.21.15.189:43678) 2024-09-16T14:46:32.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74eed9d640 1 -- 172.21.15.189:0/2340551808 learned_addr learned my addr 172.21.15.189:0/2340551808 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:32.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 1 -- 172.21.15.189:0/2340551808 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975170808 0 0) 0x7f74f009f100 con 0x7f74f009e740 2024-09-16T14:46:32.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 10 monclient(hunting): _negotiate 2024-09-16T14:46:32.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:32.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:32.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 10 monclient(hunting): my global_id is 94609 2024-09-16T14:46:32.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 1 -- 172.21.15.189:0/2340551808 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d8003530 con 0x7f74f009e740 2024-09-16T14:46:32.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 1 -- 172.21.15.189:0/2340551808 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1741355067 0 0) 0x7f74d8003530 con 0x7f74f009e740 2024-09-16T14:46:32.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 1 -- 172.21.15.189:0/2340551808 >> v1:172.21.15.26:6790/0 conn(0x7f74f009df10 legacy=0x7f74f009d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:32.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 1 -- 172.21.15.189:0/2340551808 >> v1:172.21.15.26:6789/0 conn(0x7f74f009e2f0 legacy=0x7f74f01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:32.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 10 monclient: _finish_hunting 0 2024-09-16T14:46:32.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 1 monclient: found mon.noname-c 2024-09-16T14:46:32.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:32.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:32.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 1 -- 172.21.15.189:0/2340551808 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f009c690 con 0x7f74f009e740 2024-09-16T14:46:32.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 10 monclient: _finish_auth 0 2024-09-16T14:46:32.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 10 monclient: _check_auth_tickets 2024-09-16T14:46:32.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:32.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 1 -- 172.21.15.189:0/2340551808 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74e80019a0 con 0x7f74f009e740 2024-09-16T14:46:32.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:32.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:32.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 10 monclient: dump: 2024-09-16T14:46:32.863 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:32.863 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:32.863 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:32.863 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:32.863 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:32.864 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:32.864 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:32.864 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:32.864 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:32.864 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:32.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 5 monclient: authenticate success, global_id 94609 2024-09-16T14:46:32.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:32.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 1 -- 172.21.15.189:0/2340551808 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74e8001cc0 con 0x7f74f009e740 2024-09-16T14:46:32.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:32.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:32.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 4 set_mon_vals no callback set 2024-09-16T14:46:32.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 1 -- 172.21.15.189:0/2340551808 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74e8002090 con 0x7f74f009e740 2024-09-16T14:46:32.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:32.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:32.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ed59a640 10 monclient: dump: 2024-09-16T14:46:32.865 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:32.865 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:32.865 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:32.865 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:32.865 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:32.865 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:32.865 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:32.865 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:32.865 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:32.865 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:32.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient: shutdown 2024-09-16T14:46:32.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:32.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 1 -- 172.21.15.189:0/2340551808 >> v1:172.21.15.189:6789/0 conn(0x7f74f009e740 legacy=0x7f74f013b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:32.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 1 -- 172.21.15.189:0/2340551808 shutdown_connections 2024-09-16T14:46:32.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 1 -- 172.21.15.189:0/2340551808 >> 172.21.15.189:0/2340551808 conn(0x7f74f00a3320 msgr2=0x7f74f009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:32.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 1 -- 172.21.15.189:0/2340551808 shutdown_connections 2024-09-16T14:46:32.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 1 -- 172.21.15.189:0/2340551808 wait complete. 2024-09-16T14:46:32.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient: build_initial_monmap 2024-09-16T14:46:32.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient: monmap: 2024-09-16T14:46:32.866 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:32.866 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:32.866 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:32.866 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:32.866 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:32.866 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:32.866 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:32.867 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:32.867 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:32.867 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:32.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 1 Processor -- start 2024-09-16T14:46:32.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 1 -- start start 2024-09-16T14:46:32.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient: init 2024-09-16T14:46:32.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:32.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:32.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient(hunting): picked mon.noname-c con 0x7f74f009df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:32.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient(hunting): picked mon.noname-b con 0x7f74f009e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:32.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient(hunting): picked mon.noname-a con 0x7f74f009e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:32.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f006ce70 con 0x7f74f009e740 2024-09-16T14:46:32.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f006b380 con 0x7f74f009e2f0 2024-09-16T14:46:32.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f013b440 con 0x7f74f009df10 2024-09-16T14:46:32.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:32.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74f4b17640 10 monclient(hunting): authenticate will time out at 1174.971489s 2024-09-16T14:46:32.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74ee59c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f74f009df10 legacy=0x7f74f009a270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:32.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74edd9b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f74f009e2f0 0x7f74f009a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43680/0 (socket says 172.21.15.189:43680) 2024-09-16T14:46:32.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74edd9b640 1 -- 172.21.15.189:0/3144722694 learned_addr learned my addr 172.21.15.189:0/3144722694 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:32.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014962728 0 0) 0x7f74f006b380 con 0x7f74f009e2f0 2024-09-16T14:46:32.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74deffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:32.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:32.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:32.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74deffd640 10 monclient(hunting): my global_id is 94612 2024-09-16T14:46:32.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.851+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d0003530 con 0x7f74f009e2f0 2024-09-16T14:46:32.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1955483456 0 0) 0x7f74d0003530 con 0x7f74f009e2f0 2024-09-16T14:46:32.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74f006b380 con 0x7f74f009e2f0 2024-09-16T14:46:32.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74e4001940 con 0x7f74f009e2f0 2024-09-16T14:46:32.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:32.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:32.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 10 monclient(hunting): dump: 2024-09-16T14:46:32.869 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:32.869 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:32.869 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:32.869 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:32.869 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:32.870 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:32.870 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:32.870 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:32.870 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:32.870 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:32.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:32.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758020994 0 0) 0x7f74f006ce70 con 0x7f74f009e740 2024-09-16T14:46:32.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:32.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:32.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:32.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 10 monclient(hunting): my global_id is 104259 2024-09-16T14:46:32.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d0003530 con 0x7f74f009e740 2024-09-16T14:46:32.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2222171982 0 0) 0x7f74f006b380 con 0x7f74f009e2f0 2024-09-16T14:46:32.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 >> v1:172.21.15.26:6790/0 conn(0x7f74f009df10 legacy=0x7f74f009a270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:32.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 >> v1:172.21.15.26:6789/0 conn(0x7f74f009e740 legacy=0x7f74f01d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:32.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:32.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 1 monclient: found mon.b 2024-09-16T14:46:32.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:32.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:32.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f013b720 con 0x7f74f009e2f0 2024-09-16T14:46:32.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 10 monclient: _finish_auth 0 2024-09-16T14:46:32.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:32.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:32.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74f4b17640 5 monclient: authenticate success, global_id 94612 2024-09-16T14:46:32.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74f4b17640 10 monclient: _renew_subs 2024-09-16T14:46:32.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74f4b17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:32.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74f4b17640 1 -- 172.21.15.189:0/3144722694 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f009b090 con 0x7f74f009e2f0 2024-09-16T14:46:32.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74f4b17640 10 monclient: _renew_subs 2024-09-16T14:46:32.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74f4b17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:32.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74f4b17640 1 -- 172.21.15.189:0/3144722694 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f006f920 con 0x7f74f009e2f0 2024-09-16T14:46:32.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74e4001d10 con 0x7f74f009e2f0 2024-09-16T14:46:32.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:32.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74e4003af0 con 0x7f74f009e2f0 2024-09-16T14:46:32.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:32.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:32.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74ecd99640 4 set_mon_vals no callback set 2024-09-16T14:46:32.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 10 monclient: dump: 2024-09-16T14:46:32.872 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:32.873 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:32.873 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:32.873 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:32.873 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:32.873 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:32.873 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:32.873 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:32.873 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:32.873 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:32.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74f4b17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:32.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f74e4003c50 con 0x7f74f009e2f0 2024-09-16T14:46:32.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74f4b17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:32.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74f4b17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:32.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74f4b17640 1 -- 172.21.15.189:0/3144722694 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d4005050 con 0x7f74f009e2f0 2024-09-16T14:46:32.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(84..84 src has 15..84) v4 ==== 5133+0+0 (unknown 23320314 0 0) 0x7f74e4043930 con 0x7f74f009e2f0 2024-09-16T14:46:32.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.855+0000 7f74ee59c640 10 monclient: get_auth_request con 0x7f74d00360e0 auth_method 0 2024-09-16T14:46:32.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.859+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f74e4014630 con 0x7f74f009e2f0 2024-09-16T14:46:32.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.859+0000 7f74deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:32.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:32.859+0000 7f74deffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:33.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.015+0000 7f74f4b17640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T14:46:33.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.015+0000 7f74f4b17640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T14:46:33.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.015+0000 7f74f4b17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:33.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.015+0000 7f74f4b17640 1 -- 172.21.15.189:0/3144722694 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"} v 0) v1 -- 0x7f74d4005300 con 0x7f74f009e2f0 2024-09-16T14:46:33.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.623+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}]=0 v85) v1 ==== 145+0+0 (unknown 3680813466 0 0) 0x7f74e4012cd0 con 0x7f74f009e2f0 2024-09-16T14:46:33.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.623+0000 7f74deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T14:46:33.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.623+0000 7f74deffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:33.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.715+0000 7f74f4b17640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T14:46:33.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.715+0000 7f74f4b17640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T14:46:33.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.715+0000 7f74f4b17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:33.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.715+0000 7f74f4b17640 1 -- 172.21.15.189:0/3144722694 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"} v 0) v1 -- 0x7f74d40010a0 con 0x7f74f009e2f0 2024-09-16T14:46:33.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.715+0000 7f74deffd640 1 -- 172.21.15.189:0/3144722694 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}]=0 rule foo-rule already exists v85) v1 ==== 173+0+0 (unknown 1950840413 0 0) 0x7f74e4014450 con 0x7f74f009e2f0 2024-09-16T14:46:33.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.715+0000 7f74deffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T14:46:33.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.715+0000 7f74deffd640 10 monclient: _finish_command 3 = system:0 rule foo-rule already exists 2024-09-16T14:46:33.721 INFO:tasks.workunit.client.0.smithi189.stderr:rule foo-rule already exists 2024-09-16T14:46:33.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.719+0000 7f74f4b17640 1 -- 172.21.15.189:0/3144722694 >> v1:172.21.15.189:6812/3196161582 conn(0x7f74d00360e0 legacy=0x7f74d00385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:33.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.719+0000 7f74f4b17640 10 monclient: shutdown 2024-09-16T14:46:33.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.719+0000 7f74f4b17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:33.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.719+0000 7f74f4b17640 1 -- 172.21.15.189:0/3144722694 >> v1:172.21.15.189:6789/0 conn(0x7f74f009e2f0 legacy=0x7f74f009a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:33.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.719+0000 7f74f4b17640 1 -- 172.21.15.189:0/3144722694 shutdown_connections 2024-09-16T14:46:33.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.719+0000 7f74f4b17640 1 -- 172.21.15.189:0/3144722694 >> 172.21.15.189:0/3144722694 conn(0x7f74f00a3320 msgr2=0x7f74f00a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:33.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.719+0000 7f74f4b17640 1 -- 172.21.15.189:0/3144722694 shutdown_connections 2024-09-16T14:46:33.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.719+0000 7f74f4b17640 1 -- 172.21.15.189:0/3144722694 wait complete. 2024-09-16T14:46:33.738 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush rm foo 2024-09-16T14:46:33.738 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:46:33.738 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm foo 2024-09-16T14:46:33.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.851+0000 7fe6a25dd640 10 monclient: get_monmap_and_config 2024-09-16T14:46:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.851+0000 7fe6a25dd640 10 monclient: build_initial_monmap 2024-09-16T14:46:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.851+0000 7fe6a25dd640 10 monclient: monmap: 2024-09-16T14:46:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:33.856573+0000 2024-09-16T14:46:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:33.856573+0000 2024-09-16T14:46:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:33.857 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:33.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 1 Processor -- start 2024-09-16T14:46:33.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 1 -- start start 2024-09-16T14:46:33.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 10 monclient: init 2024-09-16T14:46:33.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:33.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:33.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 10 monclient(hunting): picked mon.noname-c con 0x7fe69c073590 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:33.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 10 monclient(hunting): picked mon.noname-a con 0x7fe69c1348a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:33.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 10 monclient(hunting): picked mon.noname-b con 0x7fe69c1383b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:33.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69c073970 con 0x7fe69c1348a0 2024-09-16T14:46:33.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69c131ec0 con 0x7fe69c073590 2024-09-16T14:46:33.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69c099710 con 0x7fe69c1383b0 2024-09-16T14:46:33.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:33.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 10 monclient(hunting): authenticate will time out at 1175.973157s 2024-09-16T14:46:33.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe69c073590 0x7fe69c09cab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43688/0 (socket says 172.21.15.189:43688) 2024-09-16T14:46:33.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69bfff640 1 -- 172.21.15.189:0/1419558594 learned_addr learned my addr 172.21.15.189:0/1419558594 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:33.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a0b7a640 1 -- 172.21.15.189:0/1419558594 >> v1:172.21.15.26:6790/0 conn(0x7fe69c1383b0 legacy=0x7fe69c13a7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:33.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 1 -- 172.21.15.189:0/1419558594 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266003319 0 0) 0x7fe69c131ec0 con 0x7fe69c073590 2024-09-16T14:46:33.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:33.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:33.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:33.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 10 monclient(hunting): my global_id is 94615 2024-09-16T14:46:33.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 1 -- 172.21.15.189:0/1419558594 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe684003530 con 0x7fe69c073590 2024-09-16T14:46:33.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 1 -- 172.21.15.189:0/1419558594 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 462097968 0 0) 0x7fe684003530 con 0x7fe69c073590 2024-09-16T14:46:33.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 1 -- 172.21.15.189:0/1419558594 >> v1:172.21.15.26:6790/0 conn(0x7fe69c1383b0 legacy=0x7fe69c13a7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:33.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 1 -- 172.21.15.189:0/1419558594 >> v1:172.21.15.26:6789/0 conn(0x7fe69c1348a0 legacy=0x7fe69c136c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:33.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:33.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 1 monclient: found mon.noname-c 2024-09-16T14:46:33.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:33.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:33.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 1 -- 172.21.15.189:0/1419558594 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c0999f0 con 0x7fe69c073590 2024-09-16T14:46:33.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 10 monclient: _finish_auth 0 2024-09-16T14:46:33.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:33.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:33.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 1 -- 172.21.15.189:0/1419558594 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe68c001620 con 0x7fe69c073590 2024-09-16T14:46:33.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 5 monclient: authenticate success, global_id 94615 2024-09-16T14:46:33.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:33.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:33.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:33.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 10 monclient: dump: 2024-09-16T14:46:33.868 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:33.868 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:33.868 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:33.868 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:33.868 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:33.868 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:33.868 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:33.868 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:33.868 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:33.868 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:33.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:33.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 1 -- 172.21.15.189:0/1419558594 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe68c001940 con 0x7fe69c073590 2024-09-16T14:46:33.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:33.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:33.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 4 set_mon_vals no callback set 2024-09-16T14:46:33.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 1 -- 172.21.15.189:0/1419558594 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe68c001cd0 con 0x7fe69c073590 2024-09-16T14:46:33.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:33.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:33.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe69affd640 10 monclient: dump: 2024-09-16T14:46:33.869 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:33.869 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:33.869 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:33.870 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:33.870 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:33.870 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:33.870 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:33.870 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:33.870 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:33.870 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:33.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 10 monclient: shutdown 2024-09-16T14:46:33.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:33.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 1 -- 172.21.15.189:0/1419558594 >> v1:172.21.15.189:6789/0 conn(0x7fe69c073590 legacy=0x7fe69c09cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:33.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 1 -- 172.21.15.189:0/1419558594 shutdown_connections 2024-09-16T14:46:33.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 1 -- 172.21.15.189:0/1419558594 >> 172.21.15.189:0/1419558594 conn(0x7fe69c0a0030 msgr2=0x7fe69c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:33.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 1 -- 172.21.15.189:0/1419558594 shutdown_connections 2024-09-16T14:46:33.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 1 -- 172.21.15.189:0/1419558594 wait complete. 2024-09-16T14:46:33.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 10 monclient: build_initial_monmap 2024-09-16T14:46:33.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 10 monclient: monmap: 2024-09-16T14:46:33.871 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:33.871 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:33.871 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:33.871 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:33.871 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:33.871 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:33.871 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:33.871 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:33.872 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:33.872 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:33.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 1 Processor -- start 2024-09-16T14:46:33.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 1 -- start start 2024-09-16T14:46:33.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.855+0000 7fe6a25dd640 10 monclient: init 2024-09-16T14:46:33.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:33.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:33.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 10 monclient(hunting): picked mon.noname-a con 0x7fe69c073590 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:33.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 10 monclient(hunting): picked mon.noname-c con 0x7fe69c1348a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:33.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 10 monclient(hunting): picked mon.noname-b con 0x7fe69c1383b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:33.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69c06fb10 con 0x7fe69c073590 2024-09-16T14:46:33.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69c099120 con 0x7fe69c1383b0 2024-09-16T14:46:33.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69c06fd40 con 0x7fe69c1348a0 2024-09-16T14:46:33.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:33.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 10 monclient(hunting): authenticate will time out at 1175.976149s 2024-09-16T14:46:33.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a0b7a640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe69c1383b0 0x7fe69c1cd990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43690/0 (socket says 172.21.15.189:43690) 2024-09-16T14:46:33.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a0b7a640 1 -- 172.21.15.189:0/3279813976 learned_addr learned my addr 172.21.15.189:0/3279813976 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:33.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe69b7fe640 1 -- 172.21.15.189:0/3279813976 >> v1:172.21.15.26:6790/0 conn(0x7fe69c1348a0 legacy=0x7fe69c0967c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:33.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628865175 0 0) 0x7fe69c099120 con 0x7fe69c1383b0 2024-09-16T14:46:33.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:33.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:33.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:33.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient(hunting): my global_id is 94618 2024-09-16T14:46:33.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe688003530 con 0x7fe69c1383b0 2024-09-16T14:46:33.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 850940451 0 0) 0x7fe688003530 con 0x7fe69c1383b0 2024-09-16T14:46:33.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe69c099120 con 0x7fe69c1383b0 2024-09-16T14:46:33.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe694004450 con 0x7fe69c1383b0 2024-09-16T14:46:33.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:33.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:33.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient(hunting): dump: 2024-09-16T14:46:33.876 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:33.876 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:33.877 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:33.877 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:33.877 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:33.877 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:33.877 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:33.877 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:33.877 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:33.877 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:33.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:33.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512015056 0 0) 0x7fe69c06fb10 con 0x7fe69c073590 2024-09-16T14:46:33.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:33.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:33.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:33.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient(hunting): my global_id is 104265 2024-09-16T14:46:33.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe688003530 con 0x7fe69c073590 2024-09-16T14:46:33.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1983695528 0 0) 0x7fe69c099120 con 0x7fe69c1383b0 2024-09-16T14:46:33.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 >> v1:172.21.15.26:6790/0 conn(0x7fe69c1348a0 legacy=0x7fe69c0967c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:33.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 >> v1:172.21.15.26:6789/0 conn(0x7fe69c073590 legacy=0x7fe69c0960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:33.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:46:33.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 1 monclient: found mon.b 2024-09-16T14:46:33.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:33.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:33.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c071590 con 0x7fe69c1383b0 2024-09-16T14:46:33.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient: _finish_auth 0 2024-09-16T14:46:33.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:46:33.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:33.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 5 monclient: authenticate success, global_id 94618 2024-09-16T14:46:33.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 10 monclient: _renew_subs 2024-09-16T14:46:33.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:33.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 1 -- 172.21.15.189:0/3279813976 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe69c071ac0 con 0x7fe69c1383b0 2024-09-16T14:46:33.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 10 monclient: _renew_subs 2024-09-16T14:46:33.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:33.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 1 -- 172.21.15.189:0/3279813976 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe69c0701a0 con 0x7fe69c1383b0 2024-09-16T14:46:33.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe694004770 con 0x7fe69c1383b0 2024-09-16T14:46:33.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:33.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe694004b00 con 0x7fe69c1383b0 2024-09-16T14:46:33.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:33.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:33.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe69a7fc640 4 set_mon_vals no callback set 2024-09-16T14:46:33.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 10 monclient: dump: 2024-09-16T14:46:33.883 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:33.883 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:33.883 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:33.883 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:33.883 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:33.883 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:33.883 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:33.883 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:33.883 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:33.883 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:33.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:33.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:33.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:33.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe6a25dd640 1 -- 172.21.15.189:0/3279813976 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe67c005050 con 0x7fe69c1383b0 2024-09-16T14:46:33.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe694004c60 con 0x7fe69c1383b0 2024-09-16T14:46:33.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(85..85 src has 15..85) v4 ==== 5193+0+0 (unknown 2313085344 0 0) 0x7fe694043b60 con 0x7fe69c1383b0 2024-09-16T14:46:33.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.859+0000 7fe69bfff640 10 monclient: get_auth_request con 0x7fe688036130 auth_method 0 2024-09-16T14:46:33.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.863+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe694005460 con 0x7fe69c1383b0 2024-09-16T14:46:33.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.863+0000 7fe698ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:33.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:33.863+0000 7fe698ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:34.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.043+0000 7fe6a25dd640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:34.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.043+0000 7fe6a25dd640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:34.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.043+0000 7fe6a25dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:34.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.043+0000 7fe6a25dd640 1 -- 172.21.15.189:0/3279813976 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fe67c005300 con 0x7fe69c1383b0 2024-09-16T14:46:34.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.043+0000 7fe698ff9640 1 -- 172.21.15.189:0/3279813976 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=-16 (16) Device or resource busy v85) v1 ==== 103+0+0 (unknown 1496711894 0 0) 0x7fe694012f20 con 0x7fe69c1383b0 2024-09-16T14:46:34.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.043+0000 7fe698ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:34.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.043+0000 7fe698ff9640 10 monclient: _finish_command 2 = mon:16 (16) Device or resource busy 2024-09-16T14:46:34.049 INFO:tasks.workunit.client.0.smithi189.stderr:Error EBUSY: (16) Device or resource busy 2024-09-16T14:46:34.051 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:46:34.051 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rm foo-rule 2024-09-16T14:46:34.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.167+0000 7f19e3735640 10 monclient: get_monmap_and_config 2024-09-16T14:46:34.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.167+0000 7f19e3735640 10 monclient: build_initial_monmap 2024-09-16T14:46:34.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.167+0000 7f19e3735640 10 monclient: monmap: 2024-09-16T14:46:34.172 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:34.172 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:34.172 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:34.172343+0000 2024-09-16T14:46:34.173 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:34.172343+0000 2024-09-16T14:46:34.173 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:34.173 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:34.173 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:34.173 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:34.173 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:34.173 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:34.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.167+0000 7f19e3735640 1 Processor -- start 2024-09-16T14:46:34.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.167+0000 7f19e3735640 1 -- start start 2024-09-16T14:46:34.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.167+0000 7f19e3735640 10 monclient: init 2024-09-16T14:46:34.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.167+0000 7f19e3735640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:34.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.167+0000 7f19e3735640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:34.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.167+0000 7f19e3735640 10 monclient(hunting): picked mon.noname-c con 0x7f19dc1318a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:34.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.167+0000 7f19e3735640 10 monclient(hunting): picked mon.noname-b con 0x7f19dc134d50 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:34.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.167+0000 7f19e3735640 10 monclient(hunting): picked mon.noname-a con 0x7f19dc13b2b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:34.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.167+0000 7f19e3735640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc09d8a0 con 0x7f19dc13b2b0 2024-09-16T14:46:34.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.167+0000 7f19e3735640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc09ddf0 con 0x7f19dc1318a0 2024-09-16T14:46:34.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.167+0000 7f19e3735640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc09e8b0 con 0x7f19dc134d50 2024-09-16T14:46:34.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.167+0000 7f19e3735640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:34.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.167+0000 7f19e3735640 10 monclient(hunting): authenticate will time out at 1176.287925s 2024-09-16T14:46:34.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e14d1640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f19dc1318a0 0x7f19dc131c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43692/0 (socket says 172.21.15.189:43692) 2024-09-16T14:46:34.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e14d1640 1 -- 172.21.15.189:0/1431351919 learned_addr learned my addr 172.21.15.189:0/1431351919 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:34.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e0cd0640 1 -- 172.21.15.189:0/1431351919 >> v1:172.21.15.26:6790/0 conn(0x7f19dc134d50 legacy=0x7f19dc139b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:34.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 1 -- 172.21.15.189:0/1431351919 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314442583 0 0) 0x7f19dc09ddf0 con 0x7f19dc1318a0 2024-09-16T14:46:34.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:34.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:34.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:34.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 10 monclient(hunting): my global_id is 94621 2024-09-16T14:46:34.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 1 -- 172.21.15.189:0/1431351919 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c8003530 con 0x7f19dc1318a0 2024-09-16T14:46:34.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 1 -- 172.21.15.189:0/1431351919 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 335805067 0 0) 0x7f19c8003530 con 0x7f19dc1318a0 2024-09-16T14:46:34.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 1 -- 172.21.15.189:0/1431351919 >> v1:172.21.15.26:6790/0 conn(0x7f19dc134d50 legacy=0x7f19dc139b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:34.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 1 -- 172.21.15.189:0/1431351919 >> v1:172.21.15.26:6789/0 conn(0x7f19dc13b2b0 legacy=0x7f19dc13d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:34.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:34.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 1 monclient: found mon.noname-c 2024-09-16T14:46:34.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:34.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:34.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 1 -- 172.21.15.189:0/1431351919 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc09eb90 con 0x7f19dc1318a0 2024-09-16T14:46:34.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 10 monclient: _finish_auth 0 2024-09-16T14:46:34.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:34.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:34.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 1 -- 172.21.15.189:0/1431351919 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19d8001620 con 0x7f19dc1318a0 2024-09-16T14:46:34.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:34.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:34.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 10 monclient: dump: 2024-09-16T14:46:34.182 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:34.182 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:34.182 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:34.183 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:34.183 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:34.183 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:34.183 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:34.183 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:34.183 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:34.183 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:34.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 5 monclient: authenticate success, global_id 94621 2024-09-16T14:46:34.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:34.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 1 -- 172.21.15.189:0/1431351919 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19d80018d0 con 0x7f19dc1318a0 2024-09-16T14:46:34.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:34.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 1 -- 172.21.15.189:0/1431351919 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19d8001ca0 con 0x7f19dc1318a0 2024-09-16T14:46:34.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:34.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:34.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d3fff640 10 monclient: dump: 2024-09-16T14:46:34.184 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:34.184 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:34.184 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:34.184 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:34.184 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:34.184 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:34.184 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:34.184 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:34.184 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:34.184 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:34.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:34.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 4 set_mon_vals no callback set 2024-09-16T14:46:34.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 10 monclient: shutdown 2024-09-16T14:46:34.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:34.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 1 -- 172.21.15.189:0/1431351919 >> v1:172.21.15.189:6789/0 conn(0x7f19dc1318a0 legacy=0x7f19dc131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:34.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 1 -- 172.21.15.189:0/1431351919 shutdown_connections 2024-09-16T14:46:34.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 1 -- 172.21.15.189:0/1431351919 >> 172.21.15.189:0/1431351919 conn(0x7f19dc0a3320 msgr2=0x7f19dc09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:34.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 1 -- 172.21.15.189:0/1431351919 shutdown_connections 2024-09-16T14:46:34.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 1 -- 172.21.15.189:0/1431351919 wait complete. 2024-09-16T14:46:34.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 10 monclient: build_initial_monmap 2024-09-16T14:46:34.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 10 monclient: monmap: 2024-09-16T14:46:34.185 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:34.185 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:34.186 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:34.186 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:34.186 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:34.186 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:34.186 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:34.186 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:34.186 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:34.186 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:34.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 1 Processor -- start 2024-09-16T14:46:34.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 1 -- start start 2024-09-16T14:46:34.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 10 monclient: init 2024-09-16T14:46:34.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:34.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:34.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 10 monclient(hunting): picked mon.noname-c con 0x7f19dc1318a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:34.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 10 monclient(hunting): picked mon.noname-a con 0x7f19dc134d50 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:34.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 10 monclient(hunting): picked mon.noname-b con 0x7f19dc13b2b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:34.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc11a3c0 con 0x7f19dc134d50 2024-09-16T14:46:34.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc06b380 con 0x7f19dc13b2b0 2024-09-16T14:46:34.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc11a160 con 0x7f19dc1318a0 2024-09-16T14:46:34.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:34.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 10 monclient(hunting): authenticate will time out at 1176.290879s 2024-09-16T14:46:34.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e14d1640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f19dc1318a0 legacy=0x7f19dc119340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:34.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e1cd2640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f19dc13b2b0 0x7f19dc1d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43698/0 (socket says 172.21.15.189:43698) 2024-09-16T14:46:34.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e1cd2640 1 -- 172.21.15.189:0/2558717509 learned_addr learned my addr 172.21.15.189:0/2558717509 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:34.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 1 -- 172.21.15.189:0/2558717509 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628330248 0 0) 0x7f19dc06b380 con 0x7f19dc13b2b0 2024-09-16T14:46:34.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:34.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:34.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:34.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 10 monclient(hunting): my global_id is 94624 2024-09-16T14:46:34.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 1 -- 172.21.15.189:0/2558717509 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c0003530 con 0x7f19dc13b2b0 2024-09-16T14:46:34.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 1 -- 172.21.15.189:0/2558717509 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1326956353 0 0) 0x7f19c0003530 con 0x7f19dc13b2b0 2024-09-16T14:46:34.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 1 -- 172.21.15.189:0/2558717509 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19dc06b380 con 0x7f19dc13b2b0 2024-09-16T14:46:34.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 1 -- 172.21.15.189:0/2558717509 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19d4001c50 con 0x7f19dc13b2b0 2024-09-16T14:46:34.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:34.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:34.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 10 monclient(hunting): dump: 2024-09-16T14:46:34.188 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:34.188 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:34.189 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:34.189 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:34.189 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:34.189 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:34.189 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:34.189 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:34.189 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:34.189 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:34.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:34.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 1 -- 172.21.15.189:0/2558717509 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3770041739 0 0) 0x7f19dc06b380 con 0x7f19dc13b2b0 2024-09-16T14:46:34.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 1 -- 172.21.15.189:0/2558717509 >> v1:172.21.15.26:6790/0 conn(0x7f19dc1318a0 legacy=0x7f19dc119340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:34.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 1 -- 172.21.15.189:0/2558717509 >> v1:172.21.15.26:6789/0 conn(0x7f19dc134d50 legacy=0x7f19dc119a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:34.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:46:34.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 1 monclient: found mon.b 2024-09-16T14:46:34.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:34.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:34.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 1 -- 172.21.15.189:0/2558717509 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc0a1cd0 con 0x7f19dc13b2b0 2024-09-16T14:46:34.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:46:34.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:34.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:34.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 5 monclient: authenticate success, global_id 94624 2024-09-16T14:46:34.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 10 monclient: _renew_subs 2024-09-16T14:46:34.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:34.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 1 -- 172.21.15.189:0/2558717509 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19dc13ac40 con 0x7f19dc13b2b0 2024-09-16T14:46:34.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 10 monclient: _renew_subs 2024-09-16T14:46:34.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:34.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.171+0000 7f19e3735640 1 -- 172.21.15.189:0/2558717509 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19dc11bd90 con 0x7f19dc13b2b0 2024-09-16T14:46:34.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.175+0000 7f19d1ffb640 1 -- 172.21.15.189:0/2558717509 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19d40039a0 con 0x7f19dc13b2b0 2024-09-16T14:46:34.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.175+0000 7f19d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:34.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.175+0000 7f19d1ffb640 1 -- 172.21.15.189:0/2558717509 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19d4003cb0 con 0x7f19dc13b2b0 2024-09-16T14:46:34.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.175+0000 7f19d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:34.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.175+0000 7f19d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:34.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.175+0000 7f19d1ffb640 10 monclient: dump: 2024-09-16T14:46:34.192 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:34.192 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:34.192 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:34.192 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:34.192 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:34.192 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:34.192 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:34.192 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:34.193 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:34.193 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:34.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.175+0000 7f19e3735640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:34.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.175+0000 7f19d1ffb640 1 -- 172.21.15.189:0/2558717509 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f19d4003e50 con 0x7f19dc13b2b0 2024-09-16T14:46:34.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.175+0000 7f19d37fe640 4 set_mon_vals no callback set 2024-09-16T14:46:34.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.175+0000 7f19d1ffb640 1 -- 172.21.15.189:0/2558717509 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(85..85 src has 15..85) v4 ==== 5193+0+0 (unknown 2313085344 0 0) 0x7f19d4043a70 con 0x7f19dc13b2b0 2024-09-16T14:46:34.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.175+0000 7f19e3735640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:34.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.175+0000 7f19e3735640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:34.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.175+0000 7f19e3735640 1 -- 172.21.15.189:0/2558717509 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19dc06a0d0 con 0x7f19dc13b2b0 2024-09-16T14:46:34.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.175+0000 7f19e14d1640 10 monclient: get_auth_request con 0x7f19c00360f0 auth_method 0 2024-09-16T14:46:34.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.179+0000 7f19d1ffb640 1 -- 172.21.15.189:0/2558717509 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f19d4014ab0 con 0x7f19dc13b2b0 2024-09-16T14:46:34.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.179+0000 7f19d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:34.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.179+0000 7f19d1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:34.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.343+0000 7f19e3735640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T14:46:34.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.343+0000 7f19e3735640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T14:46:34.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.343+0000 7f19e3735640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:34.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.343+0000 7f19e3735640 1 -- 172.21.15.189:0/2558717509 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo-rule"} v 0) v1 -- 0x7f19dc139f80 con 0x7f19dc13b2b0 2024-09-16T14:46:34.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.635+0000 7f19d1ffb640 1 -- 172.21.15.189:0/2558717509 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo-rule"}]=0 v86) v1 ==== 85+0+0 (unknown 3330723634 0 0) 0x7f19d40146a0 con 0x7f19dc13b2b0 2024-09-16T14:46:34.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.635+0000 7f19d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T14:46:34.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.635+0000 7f19d1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:34.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.735+0000 7f19e3735640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T14:46:34.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.735+0000 7f19e3735640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T14:46:34.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.735+0000 7f19e3735640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:34.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.735+0000 7f19e3735640 1 -- 172.21.15.189:0/2558717509 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo-rule"} v 0) v1 -- 0x7f19dc139680 con 0x7f19dc13b2b0 2024-09-16T14:46:34.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.735+0000 7f19d1ffb640 1 -- 172.21.15.189:0/2558717509 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo-rule"}]=0 rule foo-rule does not exist v86) v1 ==== 113+0+0 (unknown 3789652312 0 0) 0x7f19d4014880 con 0x7f19dc13b2b0 2024-09-16T14:46:34.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.735+0000 7f19d1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T14:46:34.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.735+0000 7f19d1ffb640 10 monclient: _finish_command 3 = system:0 rule foo-rule does not exist 2024-09-16T14:46:34.740 INFO:tasks.workunit.client.0.smithi189.stderr:rule foo-rule does not exist 2024-09-16T14:46:34.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.739+0000 7f19e3735640 1 -- 172.21.15.189:0/2558717509 >> v1:172.21.15.189:6812/3196161582 conn(0x7f19c00360f0 legacy=0x7f19c00385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:34.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.739+0000 7f19e3735640 10 monclient: shutdown 2024-09-16T14:46:34.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.739+0000 7f19e3735640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:34.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.739+0000 7f19e3735640 1 -- 172.21.15.189:0/2558717509 >> v1:172.21.15.189:6789/0 conn(0x7f19dc13b2b0 legacy=0x7f19dc1d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:34.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.739+0000 7f19e3735640 1 -- 172.21.15.189:0/2558717509 shutdown_connections 2024-09-16T14:46:34.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.739+0000 7f19e3735640 1 -- 172.21.15.189:0/2558717509 >> 172.21.15.189:0/2558717509 conn(0x7f19dc0a3320 msgr2=0x7f19dc1340a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:34.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.739+0000 7f19e3735640 1 -- 172.21.15.189:0/2558717509 shutdown_connections 2024-09-16T14:46:34.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.739+0000 7f19e3735640 1 -- 172.21.15.189:0/2558717509 wait complete. 2024-09-16T14:46:34.756 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm bar 2024-09-16T14:46:34.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8cac5640 10 monclient: get_monmap_and_config 2024-09-16T14:46:34.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8cac5640 10 monclient: build_initial_monmap 2024-09-16T14:46:34.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8cac5640 10 monclient: monmap: 2024-09-16T14:46:34.872 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:34.872 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:34.872 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:34.872119+0000 2024-09-16T14:46:34.872 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:34.872119+0000 2024-09-16T14:46:34.872 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:34.873 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:34.873 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:34.873 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:34.873 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:34.873 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:34.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8cac5640 1 Processor -- start 2024-09-16T14:46:34.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8cac5640 1 -- start start 2024-09-16T14:46:34.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8cac5640 10 monclient: init 2024-09-16T14:46:34.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8cac5640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:34.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8cac5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:34.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8cac5640 10 monclient(hunting): picked mon.noname-b con 0x7f3a8809df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:34.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8cac5640 10 monclient(hunting): picked mon.noname-a con 0x7f3a8809e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:34.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8cac5640 10 monclient(hunting): picked mon.noname-c con 0x7f3a8809e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:34.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8cac5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8809ee20 con 0x7f3a8809e2f0 2024-09-16T14:46:34.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8cac5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8809f100 con 0x7f3a8809e740 2024-09-16T14:46:34.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8cac5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8809f3e0 con 0x7f3a8809df10 2024-09-16T14:46:34.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8cac5640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:34.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8cac5640 10 monclient(hunting): authenticate will time out at 1176.987685s 2024-09-16T14:46:34.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a86d9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3a8809e740 0x7f3a8813b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43710/0 (socket says 172.21.15.189:43710) 2024-09-16T14:46:34.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a86d9d640 1 -- 172.21.15.189:0/615205955 learned_addr learned my addr 172.21.15.189:0/615205955 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:34.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.867+0000 7f3a8659c640 1 -- 172.21.15.189:0/615205955 >> v1:172.21.15.26:6790/0 conn(0x7f3a8809df10 legacy=0x7f3a8809d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:34.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 1 -- 172.21.15.189:0/615205955 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954508816 0 0) 0x7f3a8809f100 con 0x7f3a8809e740 2024-09-16T14:46:34.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 10 monclient(hunting): _negotiate 2024-09-16T14:46:34.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:34.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:34.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 10 monclient(hunting): my global_id is 94627 2024-09-16T14:46:34.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 1 -- 172.21.15.189:0/615205955 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a74003530 con 0x7f3a8809e740 2024-09-16T14:46:34.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 1 -- 172.21.15.189:0/615205955 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2595913459 0 0) 0x7f3a74003530 con 0x7f3a8809e740 2024-09-16T14:46:34.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 1 -- 172.21.15.189:0/615205955 >> v1:172.21.15.26:6790/0 conn(0x7f3a8809df10 legacy=0x7f3a8809d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:34.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 1 -- 172.21.15.189:0/615205955 >> v1:172.21.15.26:6789/0 conn(0x7f3a8809e2f0 legacy=0x7f3a881378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:34.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 10 monclient: _finish_hunting 0 2024-09-16T14:46:34.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 1 monclient: found mon.noname-c 2024-09-16T14:46:34.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:34.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:34.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 1 -- 172.21.15.189:0/615205955 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8809c690 con 0x7f3a8809e740 2024-09-16T14:46:34.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 10 monclient: _finish_auth 0 2024-09-16T14:46:34.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 10 monclient: _check_auth_tickets 2024-09-16T14:46:34.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:34.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 1 -- 172.21.15.189:0/615205955 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3a800019a0 con 0x7f3a8809e740 2024-09-16T14:46:34.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:34.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:34.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 10 monclient: dump: 2024-09-16T14:46:34.882 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:34.882 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:34.882 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:34.882 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:34.882 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:34.882 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:34.882 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:34.882 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:34.882 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:34.882 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:34.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 5 monclient: authenticate success, global_id 94627 2024-09-16T14:46:34.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:34.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 1 -- 172.21.15.189:0/615205955 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a80001cc0 con 0x7f3a8809e740 2024-09-16T14:46:34.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:34.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:34.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 1 -- 172.21.15.189:0/615205955 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3a80002050 con 0x7f3a8809e740 2024-09-16T14:46:34.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 4 set_mon_vals no callback set 2024-09-16T14:46:34.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:34.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:34.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8559a640 10 monclient: dump: 2024-09-16T14:46:34.883 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:34.883 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:34.883 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:34.884 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:34.884 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:34.884 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:34.884 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:34.884 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:34.884 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:34.884 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:34.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 10 monclient: shutdown 2024-09-16T14:46:34.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:34.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 1 -- 172.21.15.189:0/615205955 >> v1:172.21.15.189:6789/0 conn(0x7f3a8809e740 legacy=0x7f3a8813b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:34.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 1 -- 172.21.15.189:0/615205955 shutdown_connections 2024-09-16T14:46:34.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 1 -- 172.21.15.189:0/615205955 >> 172.21.15.189:0/615205955 conn(0x7f3a880a3320 msgr2=0x7f3a8809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 1 -- 172.21.15.189:0/615205955 shutdown_connections 2024-09-16T14:46:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 1 -- 172.21.15.189:0/615205955 wait complete. 2024-09-16T14:46:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 10 monclient: build_initial_monmap 2024-09-16T14:46:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 10 monclient: monmap: 2024-09-16T14:46:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:34.886 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:34.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 1 Processor -- start 2024-09-16T14:46:34.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 1 -- start start 2024-09-16T14:46:34.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 10 monclient: init 2024-09-16T14:46:34.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:34.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:34.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 10 monclient(hunting): picked mon.noname-b con 0x7f3a8809df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:34.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 10 monclient(hunting): picked mon.noname-a con 0x7f3a8809e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:34.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 10 monclient(hunting): picked mon.noname-c con 0x7f3a8809e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:34.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a881c4230 con 0x7f3a8809e2f0 2024-09-16T14:46:34.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a881332f0 con 0x7f3a8809df10 2024-09-16T14:46:34.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8806ce70 con 0x7f3a8809e740 2024-09-16T14:46:34.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 10 monclient(hunting): authenticate will time out at 1176.990745s 2024-09-16T14:46:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8659c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3a8809df10 0x7f3a88132be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43720/0 (socket says 172.21.15.189:43720) 2024-09-16T14:46:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8659c640 1 -- 172.21.15.189:0/570379402 learned_addr learned my addr 172.21.15.189:0/570379402 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a86d9d640 1 -- 172.21.15.189:0/570379402 >> v1:172.21.15.26:6790/0 conn(0x7f3a8809e740 legacy=0x7f3a881cf7c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706735736 0 0) 0x7f3a881332f0 con 0x7f3a8809df10 2024-09-16T14:46:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient(hunting): my global_id is 94630 2024-09-16T14:46:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a68003530 con 0x7f3a8809df10 2024-09-16T14:46:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 884375588 0 0) 0x7f3a68003530 con 0x7f3a8809df10 2024-09-16T14:46:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a881332f0 con 0x7f3a8809df10 2024-09-16T14:46:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3a78004190 con 0x7f3a8809df10 2024-09-16T14:46:34.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:34.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:34.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient(hunting): dump: 2024-09-16T14:46:34.888 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:34.888 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:34.888 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:34.888 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:34.888 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:34.888 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:34.888 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:34.888 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:34.888 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:34.888 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:34.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:34.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473209397 0 0) 0x7f3a8809ee20 con 0x7f3a8809e2f0 2024-09-16T14:46:34.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:34.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:34.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:34.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient(hunting): my global_id is 104277 2024-09-16T14:46:34.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a68003530 con 0x7f3a8809e2f0 2024-09-16T14:46:34.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2174592901 0 0) 0x7f3a881332f0 con 0x7f3a8809df10 2024-09-16T14:46:34.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 >> v1:172.21.15.26:6790/0 conn(0x7f3a8809e740 legacy=0x7f3a881cf7c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:34.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 >> v1:172.21.15.26:6789/0 conn(0x7f3a8809e2f0 legacy=0x7f3a881cc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:34.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:34.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 1 monclient: found mon.b 2024-09-16T14:46:34.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:34.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:34.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8806b380 con 0x7f3a8809df10 2024-09-16T14:46:34.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient: _finish_auth 0 2024-09-16T14:46:34.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:34.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:34.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 5 monclient: authenticate success, global_id 94630 2024-09-16T14:46:34.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 10 monclient: _renew_subs 2024-09-16T14:46:34.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:34.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 1 -- 172.21.15.189:0/570379402 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8813b530 con 0x7f3a8809df10 2024-09-16T14:46:34.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 10 monclient: _renew_subs 2024-09-16T14:46:34.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:34.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a8cac5640 1 -- 172.21.15.189:0/570379402 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a8813c320 con 0x7f3a8809df10 2024-09-16T14:46:34.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a78004440 con 0x7f3a8809df10 2024-09-16T14:46:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a84d99640 4 set_mon_vals no callback set 2024-09-16T14:46:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3a780047d0 con 0x7f3a8809df10 2024-09-16T14:46:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.871+0000 7f3a7effd640 10 monclient: dump: 2024-09-16T14:46:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:34.892 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:34.892 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:34.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.875+0000 7f3a8cac5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:34.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.875+0000 7f3a8cac5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:34.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.875+0000 7f3a8cac5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:34.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.875+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3a780049b0 con 0x7f3a8809df10 2024-09-16T14:46:34.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.875+0000 7f3a8cac5640 1 -- 172.21.15.189:0/570379402 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6c005050 con 0x7f3a8809df10 2024-09-16T14:46:34.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.875+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(86..86 src has 60..86) v4 ==== 5133+0+0 (unknown 2885907247 0 0) 0x7f3a780438f0 con 0x7f3a8809df10 2024-09-16T14:46:34.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.875+0000 7f3a85d9b640 10 monclient: get_auth_request con 0x7f3a680360e0 auth_method 0 2024-09-16T14:46:34.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.879+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3a780051d0 con 0x7f3a8809df10 2024-09-16T14:46:34.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.879+0000 7f3a7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:34.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:34.879+0000 7f3a7effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:35.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.035+0000 7f3a8cac5640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T14:46:35.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.035+0000 7f3a8cac5640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T14:46:35.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.035+0000 7f3a8cac5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:35.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.035+0000 7f3a8cac5640 1 -- 172.21.15.189:0/570379402 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f3a6c005300 con 0x7f3a8809df10 2024-09-16T14:46:35.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.635+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=0 removed item id -19 name 'bar' from crush map v87) v1 ==== 120+0+0 (unknown 643381365 0 0) 0x7f3a78012c90 con 0x7f3a8809df10 2024-09-16T14:46:35.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.635+0000 7f3a7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T14:46:35.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.635+0000 7f3a7effd640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'bar' from crush map 2024-09-16T14:46:35.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.731+0000 7f3a8cac5640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T14:46:35.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.731+0000 7f3a8cac5640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T14:46:35.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.731+0000 7f3a8cac5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:35.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.731+0000 7f3a8cac5640 1 -- 172.21.15.189:0/570379402 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f3a6c0055a0 con 0x7f3a8809df10 2024-09-16T14:46:35.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.731+0000 7f3a7effd640 1 -- 172.21.15.189:0/570379402 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=0 device 'bar' does not appear in the crush map v87) v1 ==== 120+0+0 (unknown 3475446210 0 0) 0x7f3a78014410 con 0x7f3a8809df10 2024-09-16T14:46:35.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.731+0000 7f3a7effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T14:46:35.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.731+0000 7f3a7effd640 10 monclient: _finish_command 3 = system:0 device 'bar' does not appear in the crush map 2024-09-16T14:46:35.737 INFO:tasks.workunit.client.0.smithi189.stderr:device 'bar' does not appear in the crush map 2024-09-16T14:46:35.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.735+0000 7f3a8cac5640 1 -- 172.21.15.189:0/570379402 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3a680360e0 legacy=0x7f3a680385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:35.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.735+0000 7f3a8cac5640 10 monclient: shutdown 2024-09-16T14:46:35.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.735+0000 7f3a8cac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:35.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.735+0000 7f3a8cac5640 1 -- 172.21.15.189:0/570379402 >> v1:172.21.15.189:6789/0 conn(0x7f3a8809df10 legacy=0x7f3a88132be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:35.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.735+0000 7f3a8cac5640 1 -- 172.21.15.189:0/570379402 shutdown_connections 2024-09-16T14:46:35.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.735+0000 7f3a8cac5640 1 -- 172.21.15.189:0/570379402 >> 172.21.15.189:0/570379402 conn(0x7f3a880a3320 msgr2=0x7f3a880a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:35.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.735+0000 7f3a8cac5640 1 -- 172.21.15.189:0/570379402 shutdown_connections 2024-09-16T14:46:35.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.735+0000 7f3a8cac5640 1 -- 172.21.15.189:0/570379402 wait complete. 2024-09-16T14:46:35.753 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm foo 2024-09-16T14:46:35.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.859+0000 7fa0b9863640 10 monclient: get_monmap_and_config 2024-09-16T14:46:35.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.859+0000 7fa0b9863640 10 monclient: build_initial_monmap 2024-09-16T14:46:35.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.859+0000 7fa0b9863640 10 monclient: monmap: 2024-09-16T14:46:35.864 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:35.864 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:35.865 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:35.864427+0000 2024-09-16T14:46:35.865 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:35.864427+0000 2024-09-16T14:46:35.865 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:35.865 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:35.865 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:35.865 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:35.865 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:35.865 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:35.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.859+0000 7fa0b9863640 1 Processor -- start 2024-09-16T14:46:35.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.859+0000 7fa0b9863640 1 -- start start 2024-09-16T14:46:35.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.859+0000 7fa0b9863640 10 monclient: init 2024-09-16T14:46:35.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.859+0000 7fa0b9863640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:35.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.859+0000 7fa0b9863640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:35.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.859+0000 7fa0b9863640 10 monclient(hunting): picked mon.noname-c con 0x7fa0b41315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:35.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.859+0000 7fa0b9863640 10 monclient(hunting): picked mon.noname-b con 0x7fa0b409b320 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:35.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.859+0000 7fa0b9863640 10 monclient(hunting): picked mon.noname-a con 0x7fa0b4132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:35.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.859+0000 7fa0b9863640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b409edd0 con 0x7fa0b4132090 2024-09-16T14:46:35.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b409bc00 con 0x7fa0b41315a0 2024-09-16T14:46:35.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b409e900 con 0x7fa0b409b320 2024-09-16T14:46:35.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:35.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient(hunting): authenticate will time out at 1177.980039s 2024-09-16T14:46:35.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0abfff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fa0b409b320 legacy=0x7fa0b4136ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:35.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b2ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa0b41315a0 0x7fa0b4131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43734/0 (socket says 172.21.15.189:43734) 2024-09-16T14:46:35.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b2ffd640 1 -- 172.21.15.189:0/1466267909 learned_addr learned my addr 172.21.15.189:0/1466267909 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:35.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 1 -- 172.21.15.189:0/1466267909 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436154721 0 0) 0x7fa0b409bc00 con 0x7fa0b41315a0 2024-09-16T14:46:35.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:35.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:35.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:35.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 10 monclient(hunting): my global_id is 94633 2024-09-16T14:46:35.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 1 -- 172.21.15.189:0/1466267909 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0a0003530 con 0x7fa0b41315a0 2024-09-16T14:46:35.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 1 -- 172.21.15.189:0/1466267909 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2749478795 0 0) 0x7fa0a0003530 con 0x7fa0b41315a0 2024-09-16T14:46:35.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 1 -- 172.21.15.189:0/1466267909 >> v1:172.21.15.26:6790/0 conn(0x7fa0b409b320 legacy=0x7fa0b4136ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:35.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 1 -- 172.21.15.189:0/1466267909 >> v1:172.21.15.26:6789/0 conn(0x7fa0b4132090 legacy=0x7fa0b413a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:35.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 10 monclient: _finish_hunting 0 2024-09-16T14:46:35.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 1 monclient: found mon.noname-c 2024-09-16T14:46:35.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:35.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:35.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 1 -- 172.21.15.189:0/1466267909 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b409b920 con 0x7fa0b41315a0 2024-09-16T14:46:35.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 10 monclient: _finish_auth 0 2024-09-16T14:46:35.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 10 monclient: _check_auth_tickets 2024-09-16T14:46:35.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:35.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 1 -- 172.21.15.189:0/1466267909 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa0a4001620 con 0x7fa0b41315a0 2024-09-16T14:46:35.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 5 monclient: authenticate success, global_id 94633 2024-09-16T14:46:35.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:35.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:35.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:35.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 10 monclient: dump: 2024-09-16T14:46:35.875 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:35.875 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:35.875 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:35.875 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:35.875 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:35.875 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:35.875 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:35.875 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:35.875 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:35.875 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:35.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:35.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 1 -- 172.21.15.189:0/1466267909 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0a4001940 con 0x7fa0b41315a0 2024-09-16T14:46:35.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:35.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 1 -- 172.21.15.189:0/1466267909 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa0a4001cd0 con 0x7fa0b41315a0 2024-09-16T14:46:35.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:35.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 4 set_mon_vals no callback set 2024-09-16T14:46:35.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:35.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:35.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b27fc640 10 monclient: dump: 2024-09-16T14:46:35.876 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:35.876 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:35.876 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:35.876 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:35.877 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:35.877 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:35.877 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:35.877 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:35.877 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:35.877 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:35.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient: shutdown 2024-09-16T14:46:35.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:35.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 1 -- 172.21.15.189:0/1466267909 >> v1:172.21.15.189:6789/0 conn(0x7fa0b41315a0 legacy=0x7fa0b4131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:35.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 1 -- 172.21.15.189:0/1466267909 shutdown_connections 2024-09-16T14:46:35.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 1 -- 172.21.15.189:0/1466267909 >> 172.21.15.189:0/1466267909 conn(0x7fa0b40a3320 msgr2=0x7fa0b409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:35.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 1 -- 172.21.15.189:0/1466267909 shutdown_connections 2024-09-16T14:46:35.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 1 -- 172.21.15.189:0/1466267909 wait complete. 2024-09-16T14:46:35.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient: build_initial_monmap 2024-09-16T14:46:35.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient: monmap: 2024-09-16T14:46:35.878 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:35.878 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:35.878 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:35.878 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:35.878 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:35.878 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:35.878 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:35.878 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:35.878 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:35.878 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:35.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 1 Processor -- start 2024-09-16T14:46:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 1 -- start start 2024-09-16T14:46:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient: init 2024-09-16T14:46:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient(hunting): picked mon.noname-a con 0x7fa0b41315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient(hunting): picked mon.noname-c con 0x7fa0b4132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient(hunting): picked mon.noname-b con 0x7fa0b41c9100 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b409cd20 con 0x7fa0b41315a0 2024-09-16T14:46:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b411eb60 con 0x7fa0b41c9100 2024-09-16T14:46:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b409a660 con 0x7fa0b4132090 2024-09-16T14:46:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient(hunting): authenticate will time out at 1177.982703s 2024-09-16T14:46:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b37fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa0b41c9100 0x7fa0b41d1b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43738/0 (socket says 172.21.15.189:43738) 2024-09-16T14:46:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0abfff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fa0b4132090 legacy=0x7fa0b41ce420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b37fe640 1 -- 172.21.15.189:0/581003275 learned_addr learned my addr 172.21.15.189:0/581003275 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 1 -- 172.21.15.189:0/581003275 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029338961 0 0) 0x7fa0b411eb60 con 0x7fa0b41c9100 2024-09-16T14:46:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 10 monclient(hunting): my global_id is 94636 2024-09-16T14:46:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 1 -- 172.21.15.189:0/581003275 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa094003530 con 0x7fa0b41c9100 2024-09-16T14:46:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 1 -- 172.21.15.189:0/581003275 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3873787576 0 0) 0x7fa094003530 con 0x7fa0b41c9100 2024-09-16T14:46:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 1 -- 172.21.15.189:0/581003275 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0b411eb60 con 0x7fa0b41c9100 2024-09-16T14:46:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 1 -- 172.21.15.189:0/581003275 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa0ac001c80 con 0x7fa0b41c9100 2024-09-16T14:46:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:35.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 10 monclient(hunting): dump: 2024-09-16T14:46:35.881 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:35.881 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:35.881 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:35.881 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:35.881 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:35.881 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:35.881 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:35.881 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:35.881 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:35.881 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:35.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:35.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 1 -- 172.21.15.189:0/581003275 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1601827058 0 0) 0x7fa0b411eb60 con 0x7fa0b41c9100 2024-09-16T14:46:35.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 1 -- 172.21.15.189:0/581003275 >> v1:172.21.15.26:6790/0 conn(0x7fa0b4132090 legacy=0x7fa0b41ce420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:35.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 1 -- 172.21.15.189:0/581003275 >> v1:172.21.15.26:6789/0 conn(0x7fa0b41315a0 legacy=0x7fa0b4130ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:35.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:46:35.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 1 monclient: found mon.b 2024-09-16T14:46:35.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:35.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:35.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 1 -- 172.21.15.189:0/581003275 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b40a27d0 con 0x7fa0b41c9100 2024-09-16T14:46:35.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 10 monclient: _finish_auth 0 2024-09-16T14:46:35.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:46:35.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:35.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 5 monclient: authenticate success, global_id 94636 2024-09-16T14:46:35.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient: _renew_subs 2024-09-16T14:46:35.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:35.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 1 -- 172.21.15.189:0/581003275 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b406b380 con 0x7fa0b41c9100 2024-09-16T14:46:35.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient: _renew_subs 2024-09-16T14:46:35.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:35.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.863+0000 7fa0b9863640 1 -- 172.21.15.189:0/581003275 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b41c9b60 con 0x7fa0b41c9100 2024-09-16T14:46:35.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.867+0000 7fa0b9863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:35.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.867+0000 7fa0b9863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:35.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.867+0000 7fa0b9863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:35.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.867+0000 7fa0b0ff9640 1 -- 172.21.15.189:0/581003275 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0ac002050 con 0x7fa0b41c9100 2024-09-16T14:46:35.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.867+0000 7fa0b9863640 1 -- 172.21.15.189:0/581003275 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b41b8ed0 con 0x7fa0b41c9100 2024-09-16T14:46:35.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.867+0000 7fa0b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:35.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.867+0000 7fa0b0ff9640 1 -- 172.21.15.189:0/581003275 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa0ac003df0 con 0x7fa0b41c9100 2024-09-16T14:46:35.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.867+0000 7fa0b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:35.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.867+0000 7fa0b0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:35.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.867+0000 7fa0b0ff9640 10 monclient: dump: 2024-09-16T14:46:35.884 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:35.884 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:35.884 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:35.884 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:35.884 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:35.884 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:35.884 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:35.884 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:35.884 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:35.884 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:35.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.867+0000 7fa0b1ffb640 4 set_mon_vals no callback set 2024-09-16T14:46:35.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.867+0000 7fa0b0ff9640 1 -- 172.21.15.189:0/581003275 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa0ac003f50 con 0x7fa0b41c9100 2024-09-16T14:46:35.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.867+0000 7fa0b0ff9640 1 -- 172.21.15.189:0/581003275 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(87..87 src has 60..87) v4 ==== 5004+0+0 (unknown 2797814553 0 0) 0x7fa0ac043a90 con 0x7fa0b41c9100 2024-09-16T14:46:35.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.867+0000 7fa0b2ffd640 10 monclient: get_auth_request con 0x7fa0940360a0 auth_method 0 2024-09-16T14:46:35.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.871+0000 7fa0b0ff9640 1 -- 172.21.15.189:0/581003275 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa0ac014b00 con 0x7fa0b41c9100 2024-09-16T14:46:35.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.871+0000 7fa0b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:35.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:35.871+0000 7fa0b0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:36.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.019+0000 7fa0b9863640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:36.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.019+0000 7fa0b9863640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:36.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.019+0000 7fa0b9863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:36.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.019+0000 7fa0b9863640 1 -- 172.21.15.189:0/581003275 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fa0b406d980 con 0x7fa0b41c9100 2024-09-16T14:46:36.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.651+0000 7fa0b0ff9640 1 -- 172.21.15.189:0/581003275 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 removed item id -10 name 'foo' from crush map v88) v1 ==== 120+0+0 (unknown 4062457904 0 0) 0x7fa0ac012f70 con 0x7fa0b41c9100 2024-09-16T14:46:36.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.651+0000 7fa0b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:36.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.651+0000 7fa0b0ff9640 10 monclient: _finish_command 2 = system:0 removed item id -10 name 'foo' from crush map 2024-09-16T14:46:36.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.747+0000 7fa0b9863640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:36.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.747+0000 7fa0b9863640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:36.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.747+0000 7fa0b9863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:36.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.747+0000 7fa0b9863640 1 -- 172.21.15.189:0/581003275 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fa0b40008e0 con 0x7fa0b41c9100 2024-09-16T14:46:36.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.747+0000 7fa0b0ff9640 1 -- 172.21.15.189:0/581003275 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 device 'foo' does not appear in the crush map v88) v1 ==== 120+0+0 (unknown 3543709723 0 0) 0x7fa0ac0146f0 con 0x7fa0b41c9100 2024-09-16T14:46:36.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.747+0000 7fa0b0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:36.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.747+0000 7fa0b0ff9640 10 monclient: _finish_command 3 = system:0 device 'foo' does not appear in the crush map 2024-09-16T14:46:36.753 INFO:tasks.workunit.client.0.smithi189.stderr:device 'foo' does not appear in the crush map 2024-09-16T14:46:36.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.751+0000 7fa0b9863640 1 -- 172.21.15.189:0/581003275 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa0940360a0 legacy=0x7fa094038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.751+0000 7fa0b9863640 10 monclient: shutdown 2024-09-16T14:46:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.751+0000 7fa0b9863640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.751+0000 7fa0b9863640 1 -- 172.21.15.189:0/581003275 >> v1:172.21.15.189:6789/0 conn(0x7fa0b41c9100 legacy=0x7fa0b41d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.751+0000 7fa0b9863640 1 -- 172.21.15.189:0/581003275 shutdown_connections 2024-09-16T14:46:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.751+0000 7fa0b9863640 1 -- 172.21.15.189:0/581003275 >> 172.21.15.189:0/581003275 conn(0x7fa0b40a3320 msgr2=0x7fa0b40707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.751+0000 7fa0b9863640 1 -- 172.21.15.189:0/581003275 shutdown_connections 2024-09-16T14:46:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.751+0000 7fa0b9863640 1 -- 172.21.15.189:0/581003275 wait complete. 2024-09-16T14:46:36.768 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm osd.9 host2 2024-09-16T14:46:36.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 10 monclient: get_monmap_and_config 2024-09-16T14:46:36.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 10 monclient: build_initial_monmap 2024-09-16T14:46:36.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 10 monclient: monmap: 2024-09-16T14:46:36.886 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:36.886 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:36.886 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:36.886088+0000 2024-09-16T14:46:36.886 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:36.886088+0000 2024-09-16T14:46:36.886 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:36.886 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:36.886 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:36.886 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:36.886 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:36.887 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:36.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 1 Processor -- start 2024-09-16T14:46:36.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 1 -- start start 2024-09-16T14:46:36.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 10 monclient: init 2024-09-16T14:46:36.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:36.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:36.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 10 monclient(hunting): picked mon.noname-b con 0x7f90dc06c120 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:36.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 10 monclient(hunting): picked mon.noname-c con 0x7f90dc1353d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:36.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 10 monclient(hunting): picked mon.noname-a con 0x7f90dc0a1040 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:36.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90dc06c500 con 0x7f90dc0a1040 2024-09-16T14:46:36.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90dc0a2ef0 con 0x7f90dc1353d0 2024-09-16T14:46:36.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90dc09e990 con 0x7f90dc06c120 2024-09-16T14:46:36.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:36.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 10 monclient(hunting): authenticate will time out at 1179.001740s 2024-09-16T14:46:36.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90dad9d640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f90dc06c120 legacy=0x7f90dc134cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:36.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90da59c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f90dc1353d0 0x7f90dc139b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43748/0 (socket says 172.21.15.189:43748) 2024-09-16T14:46:36.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90da59c640 1 -- 172.21.15.189:0/2358961296 learned_addr learned my addr 172.21.15.189:0/2358961296 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:36.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 1 -- 172.21.15.189:0/2358961296 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590444264 0 0) 0x7f90dc0a2ef0 con 0x7f90dc1353d0 2024-09-16T14:46:36.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:46:36.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:36.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:36.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 10 monclient(hunting): my global_id is 94639 2024-09-16T14:46:36.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 1 -- 172.21.15.189:0/2358961296 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90c4003530 con 0x7f90dc1353d0 2024-09-16T14:46:36.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 1 -- 172.21.15.189:0/2358961296 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 745592322 0 0) 0x7f90c4003530 con 0x7f90dc1353d0 2024-09-16T14:46:36.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 1 -- 172.21.15.189:0/2358961296 >> v1:172.21.15.26:6790/0 conn(0x7f90dc06c120 legacy=0x7f90dc134cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:36.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 1 -- 172.21.15.189:0/2358961296 >> v1:172.21.15.26:6789/0 conn(0x7f90dc0a1040 legacy=0x7f90dc13d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:36.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:46:36.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 1 monclient: found mon.noname-c 2024-09-16T14:46:36.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:36.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:36.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 1 -- 172.21.15.189:0/2358961296 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc09ec70 con 0x7f90dc1353d0 2024-09-16T14:46:36.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 10 monclient: _finish_auth 0 2024-09-16T14:46:36.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:46:36.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:36.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 1 -- 172.21.15.189:0/2358961296 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f90d0001620 con 0x7f90dc1353d0 2024-09-16T14:46:36.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 5 monclient: authenticate success, global_id 94639 2024-09-16T14:46:36.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:36.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:36.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:36.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 10 monclient: dump: 2024-09-16T14:46:36.897 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:36.897 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:36.897 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:36.897 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:36.897 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:36.897 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:36.897 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:36.897 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:36.897 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:36.897 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:36.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:36.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 1 -- 172.21.15.189:0/2358961296 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90d00018d0 con 0x7f90dc1353d0 2024-09-16T14:46:36.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:36.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:36.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 1 -- 172.21.15.189:0/2358961296 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f90d0001ca0 con 0x7f90dc1353d0 2024-09-16T14:46:36.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 4 set_mon_vals no callback set 2024-09-16T14:46:36.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:36.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:36.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90d9d9b640 10 monclient: dump: 2024-09-16T14:46:36.898 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:36.898 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:36.898 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:36.898 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:36.898 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:36.899 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:36.899 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:36.899 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:36.899 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:36.899 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:36.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 10 monclient: shutdown 2024-09-16T14:46:36.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:36.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 1 -- 172.21.15.189:0/2358961296 >> v1:172.21.15.189:6789/0 conn(0x7f90dc1353d0 legacy=0x7f90dc139b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:36.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 1 -- 172.21.15.189:0/2358961296 shutdown_connections 2024-09-16T14:46:36.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 1 -- 172.21.15.189:0/2358961296 >> 172.21.15.189:0/2358961296 conn(0x7f90dc0a3a40 msgr2=0x7f90dc0a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:36.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 1 -- 172.21.15.189:0/2358961296 shutdown_connections 2024-09-16T14:46:36.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.883+0000 7f90e11ad640 1 -- 172.21.15.189:0/2358961296 wait complete. 2024-09-16T14:46:36.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 monclient: build_initial_monmap 2024-09-16T14:46:36.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 monclient: monmap: 2024-09-16T14:46:36.900 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:36.900 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:36.900 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:36.900 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:36.900 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:36.900 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:36.900 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:36.900 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:36.900 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:36.900 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:36.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 1 Processor -- start 2024-09-16T14:46:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 1 -- start start 2024-09-16T14:46:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 monclient: init 2024-09-16T14:46:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 monclient(hunting): picked mon.noname-a con 0x7f90dc06c120 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 monclient(hunting): picked mon.noname-c con 0x7f90dc0a1040 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 monclient(hunting): picked mon.noname-b con 0x7f90dc1353d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90dc13a930 con 0x7f90dc06c120 2024-09-16T14:46:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90dc13a540 con 0x7f90dc1353d0 2024-09-16T14:46:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90dc0a53c0 con 0x7f90dc0a1040 2024-09-16T14:46:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 monclient(hunting): authenticate will time out at 1179.004917s 2024-09-16T14:46:36.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90da59c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f90dc0a1040 legacy=0x7f90dc138600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:36.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90db59e640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f90dc1353d0 0x7f90dc1d1890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43764/0 (socket says 172.21.15.189:43764) 2024-09-16T14:46:36.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90db59e640 1 -- 172.21.15.189:0/1761915612 learned_addr learned my addr 172.21.15.189:0/1761915612 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:36.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 1 -- 172.21.15.189:0/1761915612 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676405797 0 0) 0x7f90dc13a540 con 0x7f90dc1353d0 2024-09-16T14:46:36.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:36.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:36.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:36.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 10 monclient(hunting): my global_id is 94642 2024-09-16T14:46:36.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 1 -- 172.21.15.189:0/1761915612 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90c8003530 con 0x7f90dc1353d0 2024-09-16T14:46:36.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 1 -- 172.21.15.189:0/1761915612 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3136248807 0 0) 0x7f90c8003530 con 0x7f90dc1353d0 2024-09-16T14:46:36.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 1 -- 172.21.15.189:0/1761915612 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90dc13a540 con 0x7f90dc1353d0 2024-09-16T14:46:36.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 1 -- 172.21.15.189:0/1761915612 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f90d4001c40 con 0x7f90dc1353d0 2024-09-16T14:46:36.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:36.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:36.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 10 monclient(hunting): dump: 2024-09-16T14:46:36.903 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:36.903 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:36.903 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:36.903 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:36.903 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:36.903 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:36.903 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:36.903 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:36.903 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:36.903 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:36.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:36.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 1 -- 172.21.15.189:0/1761915612 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1577231460 0 0) 0x7f90dc13a540 con 0x7f90dc1353d0 2024-09-16T14:46:36.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 1 -- 172.21.15.189:0/1761915612 >> v1:172.21.15.26:6790/0 conn(0x7f90dc0a1040 legacy=0x7f90dc138600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:36.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 1 -- 172.21.15.189:0/1761915612 >> v1:172.21.15.26:6789/0 conn(0x7f90dc06c120 legacy=0x7f90dc137ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:36.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:36.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 1 monclient: found mon.b 2024-09-16T14:46:36.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:36.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:36.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 1 -- 172.21.15.189:0/1761915612 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc138d10 con 0x7f90dc1353d0 2024-09-16T14:46:36.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:36.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:36.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:36.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 5 monclient: authenticate success, global_id 94642 2024-09-16T14:46:36.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 monclient: _renew_subs 2024-09-16T14:46:36.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:36.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 1 -- 172.21.15.189:0/1761915612 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90dc09cc10 con 0x7f90dc1353d0 2024-09-16T14:46:36.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 monclient: _renew_subs 2024-09-16T14:46:36.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:36.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 1 -- 172.21.15.189:0/1761915612 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90dc13bbc0 con 0x7f90dc1353d0 2024-09-16T14:46:36.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 1 -- 172.21.15.189:0/1761915612 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90d40039a0 con 0x7f90dc1353d0 2024-09-16T14:46:36.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:36.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 1 -- 172.21.15.189:0/1761915612 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f90d4003cb0 con 0x7f90dc1353d0 2024-09-16T14:46:36.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:36.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:36.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 10 monclient: dump: 2024-09-16T14:46:36.906 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:36.906 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:36.906 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:36.906 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:36.906 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:36.906 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:36.906 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:36.906 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:36.906 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:36.906 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:36.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90d959a640 4 set_mon_vals no callback set 2024-09-16T14:46:36.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:36.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:36.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:36.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90e11ad640 1 -- 172.21.15.189:0/1761915612 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90bc005050 con 0x7f90dc1353d0 2024-09-16T14:46:36.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 1 -- 172.21.15.189:0/1761915612 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f90d4003e10 con 0x7f90dc1353d0 2024-09-16T14:46:36.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90c37fe640 1 -- 172.21.15.189:0/1761915612 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(88..88 src has 60..88) v4 ==== 4875+0+0 (unknown 4069088217 0 0) 0x7f90d40438e0 con 0x7f90dc1353d0 2024-09-16T14:46:36.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.887+0000 7f90dad9d640 10 monclient: get_auth_request con 0x7f90c80360d0 auth_method 0 2024-09-16T14:46:36.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.895+0000 7f90c37fe640 1 -- 172.21.15.189:0/1761915612 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f90d4004610 con 0x7f90dc1353d0 2024-09-16T14:46:36.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.895+0000 7f90c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:36.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:36.895+0000 7f90c37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:37.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.051+0000 7f90e11ad640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T14:46:37.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.051+0000 7f90e11ad640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T14:46:37.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.051+0000 7f90e11ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:37.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.051+0000 7f90e11ad640 1 -- 172.21.15.189:0/1761915612 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"} v 0) v1 -- 0x7f90bc005880 con 0x7f90dc1353d0 2024-09-16T14:46:37.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.651+0000 7f90c37fe640 1 -- 172.21.15.189:0/1761915612 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}]=0 removed item id 9 name 'osd.9' from crush map v89) v1 ==== 143+0+0 (unknown 142172526 0 0) 0x7f90d4012e60 con 0x7f90dc1353d0 2024-09-16T14:46:37.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.651+0000 7f90c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T14:46:37.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.651+0000 7f90c37fe640 10 monclient: _finish_command 2 = system:0 removed item id 9 name 'osd.9' from crush map 2024-09-16T14:46:37.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.747+0000 7f90e11ad640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T14:46:37.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.747+0000 7f90e11ad640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T14:46:37.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.747+0000 7f90e11ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:37.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.747+0000 7f90e11ad640 1 -- 172.21.15.189:0/1761915612 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"} v 0) v1 -- 0x7f90bc005300 con 0x7f90dc1353d0 2024-09-16T14:46:37.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.747+0000 7f90c37fe640 1 -- 172.21.15.189:0/1761915612 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}]=0 device 'osd.9' does not appear in the crush map v89) v1 ==== 145+0+0 (unknown 659029593 0 0) 0x7f90d40145e0 con 0x7f90dc1353d0 2024-09-16T14:46:37.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.747+0000 7f90c37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T14:46:37.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.747+0000 7f90c37fe640 10 monclient: _finish_command 3 = system:0 device 'osd.9' does not appear in the crush map 2024-09-16T14:46:37.753 INFO:tasks.workunit.client.0.smithi189.stderr:device 'osd.9' does not appear in the crush map 2024-09-16T14:46:37.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.751+0000 7f90e11ad640 1 -- 172.21.15.189:0/1761915612 >> v1:172.21.15.189:6812/3196161582 conn(0x7f90c80360d0 legacy=0x7f90c8038590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:37.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.751+0000 7f90e11ad640 10 monclient: shutdown 2024-09-16T14:46:37.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.751+0000 7f90e11ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.751+0000 7f90e11ad640 1 -- 172.21.15.189:0/1761915612 >> v1:172.21.15.189:6789/0 conn(0x7f90dc1353d0 legacy=0x7f90dc1d1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.751+0000 7f90e11ad640 1 -- 172.21.15.189:0/1761915612 shutdown_connections 2024-09-16T14:46:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.751+0000 7f90e11ad640 1 -- 172.21.15.189:0/1761915612 >> 172.21.15.189:0/1761915612 conn(0x7f90dc0a3a40 msgr2=0x7f90dc09d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.751+0000 7f90e11ad640 1 -- 172.21.15.189:0/1761915612 shutdown_connections 2024-09-16T14:46:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.751+0000 7f90e11ad640 1 -- 172.21.15.189:0/1761915612 wait complete. 2024-09-16T14:46:37.769 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm host2 2024-09-16T14:46:37.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.879+0000 7f9e9d897640 10 monclient: get_monmap_and_config 2024-09-16T14:46:37.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.879+0000 7f9e9d897640 10 monclient: build_initial_monmap 2024-09-16T14:46:37.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.879+0000 7f9e9d897640 10 monclient: monmap: 2024-09-16T14:46:37.884 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:37.884 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:37.885 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:37.884602+0000 2024-09-16T14:46:37.885 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:37.884602+0000 2024-09-16T14:46:37.885 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:37.885 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:37.885 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:37.885 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:37.885 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:37.885 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:37.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 1 Processor -- start 2024-09-16T14:46:37.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 1 -- start start 2024-09-16T14:46:37.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 10 monclient: init 2024-09-16T14:46:37.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:37.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:37.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 10 monclient(hunting): picked mon.noname-b con 0x7f9e9812dbb0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:37.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 10 monclient(hunting): picked mon.noname-c con 0x7f9e9809fa40 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:37.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 10 monclient(hunting): picked mon.noname-a con 0x7f9e98137730 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:37.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e98098940 con 0x7f9e98137730 2024-09-16T14:46:37.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e98098e80 con 0x7f9e9809fa40 2024-09-16T14:46:37.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e98099160 con 0x7f9e9812dbb0 2024-09-16T14:46:37.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:37.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 10 monclient(hunting): authenticate will time out at 1180.001064s 2024-09-16T14:46:37.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e967fc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9e9809fa40 0x7f9e98136010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43774/0 (socket says 172.21.15.189:43774) 2024-09-16T14:46:37.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e967fc640 1 -- 172.21.15.189:0/1588695638 learned_addr learned my addr 172.21.15.189:0/1588695638 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:37.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e96ffd640 1 -- 172.21.15.189:0/1588695638 >> v1:172.21.15.26:6790/0 conn(0x7f9e9812dbb0 legacy=0x7f9e9812df90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:37.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 1 -- 172.21.15.189:0/1588695638 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535948579 0 0) 0x7f9e98098e80 con 0x7f9e9809fa40 2024-09-16T14:46:37.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:37.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:37.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:37.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 10 monclient(hunting): my global_id is 94645 2024-09-16T14:46:37.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 1 -- 172.21.15.189:0/1588695638 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e80003530 con 0x7f9e9809fa40 2024-09-16T14:46:37.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 1 -- 172.21.15.189:0/1588695638 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2451906155 0 0) 0x7f9e80003530 con 0x7f9e9809fa40 2024-09-16T14:46:37.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 1 -- 172.21.15.189:0/1588695638 >> v1:172.21.15.26:6790/0 conn(0x7f9e9812dbb0 legacy=0x7f9e9812df90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:37.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 1 -- 172.21.15.189:0/1588695638 >> v1:172.21.15.26:6789/0 conn(0x7f9e98137730 legacy=0x7f9e98139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:37.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:46:37.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 1 monclient: found mon.noname-c 2024-09-16T14:46:37.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:37.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:37.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 1 -- 172.21.15.189:0/1588695638 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e98099440 con 0x7f9e9809fa40 2024-09-16T14:46:37.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 10 monclient: _finish_auth 0 2024-09-16T14:46:37.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:37.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:37.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 1 -- 172.21.15.189:0/1588695638 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e8c001620 con 0x7f9e9809fa40 2024-09-16T14:46:37.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:37.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:37.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 10 monclient: dump: 2024-09-16T14:46:37.898 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:37.899 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:37.899 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:37.899 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:37.899 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:37.899 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:37.899 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:37.899 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:37.899 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:37.899 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:37.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 5 monclient: authenticate success, global_id 94645 2024-09-16T14:46:37.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:37.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 1 -- 172.21.15.189:0/1588695638 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e8c001940 con 0x7f9e9809fa40 2024-09-16T14:46:37.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:37.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 1 -- 172.21.15.189:0/1588695638 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e8c001cd0 con 0x7f9e9809fa40 2024-09-16T14:46:37.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:37.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 4 set_mon_vals no callback set 2024-09-16T14:46:37.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:37.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:37.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e95ffb640 10 monclient: dump: 2024-09-16T14:46:37.901 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:37.901 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:37.901 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:37.901 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:37.901 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:37.901 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:37.902 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:37.902 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:37.902 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:37.902 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:37.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 10 monclient: shutdown 2024-09-16T14:46:37.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:37.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 1 -- 172.21.15.189:0/1588695638 >> v1:172.21.15.189:6789/0 conn(0x7f9e9809fa40 legacy=0x7f9e98136010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:37.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 1 -- 172.21.15.189:0/1588695638 shutdown_connections 2024-09-16T14:46:37.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 1 -- 172.21.15.189:0/1588695638 >> 172.21.15.189:0/1588695638 conn(0x7f9e980a0030 msgr2=0x7f9e980a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:37.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 1 -- 172.21.15.189:0/1588695638 shutdown_connections 2024-09-16T14:46:37.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 1 -- 172.21.15.189:0/1588695638 wait complete. 2024-09-16T14:46:37.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 10 monclient: build_initial_monmap 2024-09-16T14:46:37.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 10 monclient: monmap: 2024-09-16T14:46:37.903 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:37.904 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:37.904 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:37.904 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:37.904 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:37.904 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:37.904 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:37.905 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:37.905 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:37.905 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:37.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 1 Processor -- start 2024-09-16T14:46:37.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 1 -- start start 2024-09-16T14:46:37.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 10 monclient: init 2024-09-16T14:46:37.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:37.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.883+0000 7f9e9d897640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:37.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 10 monclient(hunting): picked mon.noname-b con 0x7f9e9809fa40 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:37.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 10 monclient(hunting): picked mon.noname-c con 0x7f9e9812dbb0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:37.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 10 monclient(hunting): picked mon.noname-a con 0x7f9e98137730 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:37.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9809a360 con 0x7f9e98137730 2024-09-16T14:46:37.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9809ad80 con 0x7f9e9809fa40 2024-09-16T14:46:37.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9812d490 con 0x7f9e9812dbb0 2024-09-16T14:46:37.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:37.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 10 monclient(hunting): authenticate will time out at 1180.004058s 2024-09-16T14:46:37.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e96ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9e9809fa40 0x7f9e980943e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:43786/0 (socket says 172.21.15.189:43786) 2024-09-16T14:46:37.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e96ffd640 1 -- 172.21.15.189:0/4052735321 learned_addr learned my addr 172.21.15.189:0/4052735321 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:37.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e967fc640 1 -- 172.21.15.189:0/4052735321 >> v1:172.21.15.26:6790/0 conn(0x7f9e9812dbb0 legacy=0x7f9e981ca020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:37.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 1 -- 172.21.15.189:0/4052735321 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401004643 0 0) 0x7f9e9809ad80 con 0x7f9e9809fa40 2024-09-16T14:46:37.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:37.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:37.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:37.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 10 monclient(hunting): my global_id is 94648 2024-09-16T14:46:37.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 1 -- 172.21.15.189:0/4052735321 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e84003530 con 0x7f9e9809fa40 2024-09-16T14:46:37.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 1 -- 172.21.15.189:0/4052735321 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2038909686 0 0) 0x7f9e84003530 con 0x7f9e9809fa40 2024-09-16T14:46:37.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 1 -- 172.21.15.189:0/4052735321 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e9809ad80 con 0x7f9e9809fa40 2024-09-16T14:46:37.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 1 -- 172.21.15.189:0/4052735321 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e88004190 con 0x7f9e9809fa40 2024-09-16T14:46:37.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:37.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:37.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 10 monclient(hunting): dump: 2024-09-16T14:46:37.909 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:37.909 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:37.909 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:37.909 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:37.909 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:37.909 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:37.909 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:37.909 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:37.909 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:37.909 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:37.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:37.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 1 -- 172.21.15.189:0/4052735321 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1770810534 0 0) 0x7f9e9809ad80 con 0x7f9e9809fa40 2024-09-16T14:46:37.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 1 -- 172.21.15.189:0/4052735321 >> v1:172.21.15.26:6790/0 conn(0x7f9e9812dbb0 legacy=0x7f9e981ca020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:37.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 1 -- 172.21.15.189:0/4052735321 >> v1:172.21.15.26:6789/0 conn(0x7f9e98137730 legacy=0x7f9e981cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:37.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:37.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 1 monclient: found mon.b 2024-09-16T14:46:37.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:37.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:37.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 1 -- 172.21.15.189:0/4052735321 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9809a550 con 0x7f9e9809fa40 2024-09-16T14:46:37.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 10 monclient: _finish_auth 0 2024-09-16T14:46:37.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:37.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:37.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 5 monclient: authenticate success, global_id 94648 2024-09-16T14:46:37.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 10 monclient: _renew_subs 2024-09-16T14:46:37.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:37.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 1 -- 172.21.15.189:0/4052735321 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9809b5c0 con 0x7f9e9809fa40 2024-09-16T14:46:37.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 10 monclient: _renew_subs 2024-09-16T14:46:37.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:37.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 1 -- 172.21.15.189:0/4052735321 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9809a8a0 con 0x7f9e9809fa40 2024-09-16T14:46:37.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 1 -- 172.21.15.189:0/4052735321 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e880044b0 con 0x7f9e9809fa40 2024-09-16T14:46:37.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:37.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 1 -- 172.21.15.189:0/4052735321 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e88004840 con 0x7f9e9809fa40 2024-09-16T14:46:37.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:37.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:37.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e957fa640 4 set_mon_vals no callback set 2024-09-16T14:46:37.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 10 monclient: dump: 2024-09-16T14:46:37.912 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:37.912 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:37.913 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 1 -- 172.21.15.189:0/4052735321 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9e880049a0 con 0x7f9e9809fa40 2024-09-16T14:46:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e9d897640 1 -- 172.21.15.189:0/4052735321 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e78005050 con 0x7f9e9809fa40 2024-09-16T14:46:37.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e7ffff640 1 -- 172.21.15.189:0/4052735321 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(89..89 src has 60..89) v4 ==== 4854+0+0 (unknown 2741193929 0 0) 0x7f9e880436a0 con 0x7f9e9809fa40 2024-09-16T14:46:37.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.887+0000 7f9e967fc640 10 monclient: get_auth_request con 0x7f9e84036050 auth_method 0 2024-09-16T14:46:37.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.891+0000 7f9e7ffff640 1 -- 172.21.15.189:0/4052735321 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9e88005170 con 0x7f9e9809fa40 2024-09-16T14:46:37.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.891+0000 7f9e7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:37.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:37.891+0000 7f9e7ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:38.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.051+0000 7f9e9d897640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T14:46:38.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.051+0000 7f9e9d897640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T14:46:38.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.051+0000 7f9e9d897640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:38.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.051+0000 7f9e9d897640 1 -- 172.21.15.189:0/4052735321 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host2"} v 0) v1 -- 0x7f9e78005300 con 0x7f9e9809fa40 2024-09-16T14:46:38.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.663+0000 7f9e7ffff640 1 -- 172.21.15.189:0/4052735321 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v90) v1 ==== 124+0+0 (unknown 911159847 0 0) 0x7f9e88012bb0 con 0x7f9e9809fa40 2024-09-16T14:46:38.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.663+0000 7f9e7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T14:46:38.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.663+0000 7f9e7ffff640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'host2' from crush map 2024-09-16T14:46:38.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.767+0000 7f9e9d897640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T14:46:38.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.767+0000 7f9e9d897640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T14:46:38.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.767+0000 7f9e9d897640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:38.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.767+0000 7f9e9d897640 1 -- 172.21.15.189:0/4052735321 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host2"} v 0) v1 -- 0x7f9e780055a0 con 0x7f9e9809fa40 2024-09-16T14:46:38.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.767+0000 7f9e7ffff640 1 -- 172.21.15.189:0/4052735321 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host2"}]=0 device 'host2' does not appear in the crush map v90) v1 ==== 124+0+0 (unknown 652536795 0 0) 0x7f9e88014330 con 0x7f9e9809fa40 2024-09-16T14:46:38.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.767+0000 7f9e7ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T14:46:38.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.767+0000 7f9e7ffff640 10 monclient: _finish_command 3 = system:0 device 'host2' does not appear in the crush map 2024-09-16T14:46:38.773 INFO:tasks.workunit.client.0.smithi189.stderr:device 'host2' does not appear in the crush map 2024-09-16T14:46:38.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.771+0000 7f9e9d897640 1 -- 172.21.15.189:0/4052735321 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9e84036050 legacy=0x7f9e84038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:38.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.771+0000 7f9e9d897640 10 monclient: shutdown 2024-09-16T14:46:38.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.771+0000 7f9e9d897640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:38.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.771+0000 7f9e9d897640 1 -- 172.21.15.189:0/4052735321 >> v1:172.21.15.189:6789/0 conn(0x7f9e9809fa40 legacy=0x7f9e980943e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:38.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.771+0000 7f9e9d897640 1 -- 172.21.15.189:0/4052735321 shutdown_connections 2024-09-16T14:46:38.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.771+0000 7f9e9d897640 1 -- 172.21.15.189:0/4052735321 >> 172.21.15.189:0/4052735321 conn(0x7f9e980a0030 msgr2=0x7f9e980a2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:38.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.771+0000 7f9e9d897640 1 -- 172.21.15.189:0/4052735321 shutdown_connections 2024-09-16T14:46:38.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.771+0000 7f9e9d897640 1 -- 172.21.15.189:0/4052735321 wait complete. 2024-09-16T14:46:38.789 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket foo host 2024-09-16T14:46:38.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.903+0000 7fbca6560640 10 monclient: get_monmap_and_config 2024-09-16T14:46:38.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.903+0000 7fbca6560640 10 monclient: build_initial_monmap 2024-09-16T14:46:38.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.903+0000 7fbca6560640 10 monclient: monmap: 2024-09-16T14:46:38.909 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:38.909 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:38.909 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:38.909117+0000 2024-09-16T14:46:38.909 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:38.909117+0000 2024-09-16T14:46:38.910 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:38.910 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:38.910 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:38.910 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:38.910 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:38.910 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:38.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 1 Processor -- start 2024-09-16T14:46:38.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 1 -- start start 2024-09-16T14:46:38.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient: init 2024-09-16T14:46:38.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:38.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:38.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient(hunting): picked mon.noname-c con 0x7fbca01315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:38.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient(hunting): picked mon.noname-b con 0x7fbca009b320 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient(hunting): picked mon.noname-a con 0x7fbca0132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca009edd0 con 0x7fbca0132090 2024-09-16T14:46:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca009bc00 con 0x7fbca01315a0 2024-09-16T14:46:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca009e900 con 0x7fbca009b320 2024-09-16T14:46:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient(hunting): authenticate will time out at 1181.024997s 2024-09-16T14:46:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9ffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbca01315a0 0x7fbca0131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35406/0 (socket says 172.21.15.189:35406) 2024-09-16T14:46:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9ffff640 1 -- 172.21.15.189:0/4007161518 learned_addr learned my addr 172.21.15.189:0/4007161518 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc977fe640 1 -- 172.21.15.189:0/4007161518 >> v1:172.21.15.26:6790/0 conn(0x7fbca009b320 legacy=0x7fbca0136ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 1 -- 172.21.15.189:0/4007161518 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050487605 0 0) 0x7fbca009bc00 con 0x7fbca01315a0 2024-09-16T14:46:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 10 monclient(hunting): my global_id is 94651 2024-09-16T14:46:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 1 -- 172.21.15.189:0/4007161518 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc8c003530 con 0x7fbca01315a0 2024-09-16T14:46:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 1 -- 172.21.15.189:0/4007161518 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2763180669 0 0) 0x7fbc8c003530 con 0x7fbca01315a0 2024-09-16T14:46:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 1 -- 172.21.15.189:0/4007161518 >> v1:172.21.15.26:6790/0 conn(0x7fbca009b320 legacy=0x7fbca0136ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 1 -- 172.21.15.189:0/4007161518 >> v1:172.21.15.26:6789/0 conn(0x7fbca0132090 legacy=0x7fbca013a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 1 monclient: found mon.noname-c 2024-09-16T14:46:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 1 -- 172.21.15.189:0/4007161518 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca009b920 con 0x7fbca01315a0 2024-09-16T14:46:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:38.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 1 -- 172.21.15.189:0/4007161518 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbc90001620 con 0x7fbca01315a0 2024-09-16T14:46:38.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 5 monclient: authenticate success, global_id 94651 2024-09-16T14:46:38.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:38.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:38.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:38.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 10 monclient: dump: 2024-09-16T14:46:38.919 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:38.920 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:38.920 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:38.920 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:38.920 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:38.920 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:38.920 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:38.920 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:38.920 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:38.920 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:38.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:38.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 1 -- 172.21.15.189:0/4007161518 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc90001940 con 0x7fbca01315a0 2024-09-16T14:46:38.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:38.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:38.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 1 -- 172.21.15.189:0/4007161518 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbc90001d10 con 0x7fbca01315a0 2024-09-16T14:46:38.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 4 set_mon_vals no callback set 2024-09-16T14:46:38.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:38.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:38.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbc9f7fe640 10 monclient: dump: 2024-09-16T14:46:38.921 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:38.921 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:38.921 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:38.922 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient: shutdown 2024-09-16T14:46:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 1 -- 172.21.15.189:0/4007161518 >> v1:172.21.15.189:6789/0 conn(0x7fbca01315a0 legacy=0x7fbca0131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 1 -- 172.21.15.189:0/4007161518 shutdown_connections 2024-09-16T14:46:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 1 -- 172.21.15.189:0/4007161518 >> 172.21.15.189:0/4007161518 conn(0x7fbca00a3320 msgr2=0x7fbca009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 1 -- 172.21.15.189:0/4007161518 shutdown_connections 2024-09-16T14:46:38.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 1 -- 172.21.15.189:0/4007161518 wait complete. 2024-09-16T14:46:38.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient: build_initial_monmap 2024-09-16T14:46:38.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient: monmap: 2024-09-16T14:46:38.923 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:38.923 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:38.923 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:38.923 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:38.923 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:38.923 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:38.923 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:38.923 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:38.924 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:38.924 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:38.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 1 Processor -- start 2024-09-16T14:46:38.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 1 -- start start 2024-09-16T14:46:38.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient: init 2024-09-16T14:46:38.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:38.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:38.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient(hunting): picked mon.noname-b con 0x7fbca01315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:38.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient(hunting): picked mon.noname-c con 0x7fbca0132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:38.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient(hunting): picked mon.noname-a con 0x7fbca0130fe0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:38.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca009cd20 con 0x7fbca0130fe0 2024-09-16T14:46:38.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca011c930 con 0x7fbca01315a0 2024-09-16T14:46:38.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca011e290 con 0x7fbca0132090 2024-09-16T14:46:38.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:38.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.907+0000 7fbca6560640 10 monclient(hunting): authenticate will time out at 1181.027940s 2024-09-16T14:46:38.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9ffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbca01315a0 0x7fbca01c6eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35422/0 (socket says 172.21.15.189:35422) 2024-09-16T14:46:38.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9ffff640 1 -- 172.21.15.189:0/1351008756 learned_addr learned my addr 172.21.15.189:0/1351008756 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:38.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc977fe640 1 -- 172.21.15.189:0/1351008756 >> v1:172.21.15.26:6790/0 conn(0x7fbca0132090 legacy=0x7fbca01308d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:38.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708967125 0 0) 0x7fbca011c930 con 0x7fbca01315a0 2024-09-16T14:46:38.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:38.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:38.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:38.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient(hunting): my global_id is 94654 2024-09-16T14:46:38.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc80003530 con 0x7fbca01315a0 2024-09-16T14:46:38.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 80651804 0 0) 0x7fbc80003530 con 0x7fbca01315a0 2024-09-16T14:46:38.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbca011c930 con 0x7fbca01315a0 2024-09-16T14:46:38.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbc900015e0 con 0x7fbca01315a0 2024-09-16T14:46:38.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:38.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:38.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient(hunting): dump: 2024-09-16T14:46:38.929 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:38.929 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:38.929 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:38.929 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:38.929 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:38.929 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:38.929 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:38.929 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:38.930 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:38.930 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:38.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:38.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135311324 0 0) 0x7fbca009cd20 con 0x7fbca0130fe0 2024-09-16T14:46:38.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:38.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:38.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:38.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient(hunting): my global_id is 104301 2024-09-16T14:46:38.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc80003530 con 0x7fbca0130fe0 2024-09-16T14:46:38.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 132301697 0 0) 0x7fbca011c930 con 0x7fbca01315a0 2024-09-16T14:46:38.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 >> v1:172.21.15.26:6790/0 conn(0x7fbca0132090 legacy=0x7fbca01308d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:38.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 >> v1:172.21.15.26:6789/0 conn(0x7fbca0130fe0 legacy=0x7fbca01cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:38.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:46:38.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 1 monclient: found mon.b 2024-09-16T14:46:38.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:38.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:38.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca009a660 con 0x7fbca01315a0 2024-09-16T14:46:38.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:46:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:46:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbca6560640 5 monclient: authenticate success, global_id 94654 2024-09-16T14:46:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbca6560640 10 monclient: _renew_subs 2024-09-16T14:46:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbca6560640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbca6560640 1 -- 172.21.15.189:0/1351008756 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca006b380 con 0x7fbca01315a0 2024-09-16T14:46:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbca6560640 10 monclient: _renew_subs 2024-09-16T14:46:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbca6560640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbca6560640 1 -- 172.21.15.189:0/1351008756 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca011dfb0 con 0x7fbca01315a0 2024-09-16T14:46:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc90001160 con 0x7fbca01315a0 2024-09-16T14:46:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbc90004e20 con 0x7fbca01315a0 2024-09-16T14:46:38.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:38.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:38.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9effd640 4 set_mon_vals no callback set 2024-09-16T14:46:38.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 10 monclient: dump: 2024-09-16T14:46:38.933 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:38.933 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:38.933 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:38.933 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:38.933 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:38.933 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:38.934 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbca6560640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbca6560640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbca6560640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbca6560640 1 -- 172.21.15.189:0/1351008756 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc84005050 con 0x7fbca01315a0 2024-09-16T14:46:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbc90005000 con 0x7fbca01315a0 2024-09-16T14:46:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(90..90 src has 60..90) v4 ==== 4719+0+0 (unknown 4243801124 0 0) 0x7fbc900437c0 con 0x7fbca01315a0 2024-09-16T14:46:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.911+0000 7fbc977fe640 10 monclient: get_auth_request con 0x7fbc80036100 auth_method 0 2024-09-16T14:46:38.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.915+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbc90014a90 con 0x7fbca01315a0 2024-09-16T14:46:38.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.915+0000 7fbc9d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:38.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:38.915+0000 7fbc9d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:39.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.063+0000 7fbca6560640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T14:46:39.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.063+0000 7fbca6560640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T14:46:39.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.063+0000 7fbca6560640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:39.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.063+0000 7fbca6560640 1 -- 172.21.15.189:0/1351008756 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "host"} v 0) v1 -- 0x7fbc84005880 con 0x7fbca01315a0 2024-09-16T14:46:39.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.679+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}]=0 added bucket foo type host to crush map v91) v1 ==== 138+0+0 (unknown 1810020012 0 0) 0x7fbc90012f00 con 0x7fbca01315a0 2024-09-16T14:46:39.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.679+0000 7fbc9d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T14:46:39.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.679+0000 7fbc9d7fa640 10 monclient: _finish_command 2 = system:0 added bucket foo type host to crush map 2024-09-16T14:46:39.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.779+0000 7fbca6560640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T14:46:39.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.779+0000 7fbca6560640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T14:46:39.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.779+0000 7fbca6560640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:39.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.779+0000 7fbca6560640 1 -- 172.21.15.189:0/1351008756 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "host"} v 0) v1 -- 0x7fbc84005340 con 0x7fbca01315a0 2024-09-16T14:46:39.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.779+0000 7fbc9d7fa640 1 -- 172.21.15.189:0/1351008756 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}]=0 bucket 'foo' already exists v91) v1 ==== 126+0+0 (unknown 2753640235 0 0) 0x7fbc90014860 con 0x7fbca01315a0 2024-09-16T14:46:39.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.779+0000 7fbc9d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T14:46:39.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.779+0000 7fbc9d7fa640 10 monclient: _finish_command 3 = system:0 bucket 'foo' already exists 2024-09-16T14:46:39.783 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'foo' already exists 2024-09-16T14:46:39.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.779+0000 7fbca6560640 1 -- 172.21.15.189:0/1351008756 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbc80036100 legacy=0x7fbc800385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:39.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.779+0000 7fbca6560640 10 monclient: shutdown 2024-09-16T14:46:39.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.779+0000 7fbca6560640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:39.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.779+0000 7fbca6560640 1 -- 172.21.15.189:0/1351008756 >> v1:172.21.15.189:6789/0 conn(0x7fbca01315a0 legacy=0x7fbca01c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:39.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.779+0000 7fbca6560640 1 -- 172.21.15.189:0/1351008756 shutdown_connections 2024-09-16T14:46:39.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.779+0000 7fbca6560640 1 -- 172.21.15.189:0/1351008756 >> 172.21.15.189:0/1351008756 conn(0x7fbca00a3320 msgr2=0x7fbca00707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:39.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.779+0000 7fbca6560640 1 -- 172.21.15.189:0/1351008756 shutdown_connections 2024-09-16T14:46:39.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.779+0000 7fbca6560640 1 -- 172.21.15.189:0/1351008756 wait complete. 2024-09-16T14:46:39.799 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush move foo root=default rack=localrack 2024-09-16T14:46:39.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.907+0000 7f0f62bbd640 10 monclient: get_monmap_and_config 2024-09-16T14:46:39.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.907+0000 7f0f62bbd640 10 monclient: build_initial_monmap 2024-09-16T14:46:39.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.907+0000 7f0f62bbd640 10 monclient: monmap: 2024-09-16T14:46:39.913 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:39.913 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:39.913 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:39.913369+0000 2024-09-16T14:46:39.913 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:39.913369+0000 2024-09-16T14:46:39.914 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:39.914 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:39.914 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:39.914 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:39.914 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:39.914 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:39.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 1 Processor -- start 2024-09-16T14:46:39.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 1 -- start start 2024-09-16T14:46:39.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient: init 2024-09-16T14:46:39.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:39.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:46:39.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient(hunting): picked mon.noname-a con 0x7f0f5c0a1a90 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:39.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient(hunting): picked mon.noname-c con 0x7f0f5c0a2580 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:39.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient(hunting): picked mon.noname-b con 0x7f0f5c09fd40 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:39.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5c09df00 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5c09f0a0 con 0x7f0f5c0a2580 2024-09-16T14:46:39.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5c0a3040 con 0x7f0f5c09fd40 2024-09-16T14:46:39.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:39.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient(hunting): authenticate will time out at 1182.028972s 2024-09-16T14:46:39.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0f5c0a2580 0x7f0f5c139df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35428/0 (socket says 172.21.15.189:35428) 2024-09-16T14:46:39.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5bfff640 1 -- 172.21.15.189:0/956348719 learned_addr learned my addr 172.21.15.189:0/956348719 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:39.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f6115a640 1 -- 172.21.15.189:0/956348719 >> v1:172.21.15.26:6790/0 conn(0x7f0f5c09fd40 legacy=0x7f0f5c13d520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:39.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 1 -- 172.21.15.189:0/956348719 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899784133 0 0) 0x7f0f5c09f0a0 con 0x7f0f5c0a2580 2024-09-16T14:46:39.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:39.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:39.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:39.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 10 monclient(hunting): my global_id is 94657 2024-09-16T14:46:39.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 1 -- 172.21.15.189:0/956348719 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f4c003530 con 0x7f0f5c0a2580 2024-09-16T14:46:39.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 1 -- 172.21.15.189:0/956348719 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 529517709 0 0) 0x7f0f4c003530 con 0x7f0f5c0a2580 2024-09-16T14:46:39.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 1 -- 172.21.15.189:0/956348719 >> v1:172.21.15.26:6790/0 conn(0x7f0f5c09fd40 legacy=0x7f0f5c13d520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:39.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 1 -- 172.21.15.189:0/956348719 >> v1:172.21.15.26:6789/0 conn(0x7f0f5c0a1a90 legacy=0x7f0f5c0a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:39.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:39.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 1 monclient: found mon.noname-c 2024-09-16T14:46:39.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:39.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:39.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 1 -- 172.21.15.189:0/956348719 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c09eae0 con 0x7f0f5c0a2580 2024-09-16T14:46:39.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:39.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:39.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:39.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 1 -- 172.21.15.189:0/956348719 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0f48001620 con 0x7f0f5c0a2580 2024-09-16T14:46:39.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 5 monclient: authenticate success, global_id 94657 2024-09-16T14:46:39.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:39.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:39.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:39.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 10 monclient: dump: 2024-09-16T14:46:39.924 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:39.924 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:39.924 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:39.924 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:39.924 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:39.924 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:39.924 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:39.924 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:39.924 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:39.925 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:39.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:39.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 1 -- 172.21.15.189:0/956348719 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f480018d0 con 0x7f0f5c0a2580 2024-09-16T14:46:39.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:39.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 1 -- 172.21.15.189:0/956348719 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0f48001c60 con 0x7f0f5c0a2580 2024-09-16T14:46:39.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:39.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:39.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5b7fe640 10 monclient: dump: 2024-09-16T14:46:39.925 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:39.925 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:39.925 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:39.925 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:39.925 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:39.926 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:39.926 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:39.926 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:39.926 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:39.926 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:39.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:39.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 4 set_mon_vals no callback set 2024-09-16T14:46:39.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient: shutdown 2024-09-16T14:46:39.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:39.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 1 -- 172.21.15.189:0/956348719 >> v1:172.21.15.189:6789/0 conn(0x7f0f5c0a2580 legacy=0x7f0f5c139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:39.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 1 -- 172.21.15.189:0/956348719 shutdown_connections 2024-09-16T14:46:39.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 1 -- 172.21.15.189:0/956348719 >> 172.21.15.189:0/956348719 conn(0x7f0f5c0a7a70 msgr2=0x7f0f5c0a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:39.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 1 -- 172.21.15.189:0/956348719 shutdown_connections 2024-09-16T14:46:39.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 1 -- 172.21.15.189:0/956348719 wait complete. 2024-09-16T14:46:39.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient: build_initial_monmap 2024-09-16T14:46:39.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient: monmap: 2024-09-16T14:46:39.927 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:39.927 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:39.927 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:39.927 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:39.927 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:39.927 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:39.927 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:39.927 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:39.927 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:39.927 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:39.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 1 Processor -- start 2024-09-16T14:46:39.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 1 -- start start 2024-09-16T14:46:39.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient: init 2024-09-16T14:46:39.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:39.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:39.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient(hunting): picked mon.noname-c con 0x7f0f5c09fd40 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:39.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient(hunting): picked mon.noname-b con 0x7f0f5c0a1a90 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:39.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient(hunting): picked mon.noname-a con 0x7f0f5c0a2580 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:39.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5c09e280 con 0x7f0f5c0a2580 2024-09-16T14:46:39.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5c122870 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5c0a6480 con 0x7f0f5c09fd40 2024-09-16T14:46:39.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:39.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f62bbd640 10 monclient(hunting): authenticate will time out at 1182.031733s 2024-09-16T14:46:39.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f60959640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f0f5c09fd40 legacy=0x7f0f5c133cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:39.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0f5c0a1a90 0x7f0f5c1ce310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35434/0 (socket says 172.21.15.189:35434) 2024-09-16T14:46:39.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.911+0000 7f0f5bfff640 1 -- 172.21.15.189:0/419182308 learned_addr learned my addr 172.21.15.189:0/419182308 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:39.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050222013 0 0) 0x7f0f5c122870 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:39.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:39.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:39.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient(hunting): my global_id is 94660 2024-09-16T14:46:39.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f40003530 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1602168059 0 0) 0x7f0f40003530 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f5c122870 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0f48001be0 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:39.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:39.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient(hunting): dump: 2024-09-16T14:46:39.930 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:39.930 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:39.930 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:39.930 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:39.930 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:39.930 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:39.930 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:39.930 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:39.930 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:39.931 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:39.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:39.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816877389 0 0) 0x7f0f5c09e280 con 0x7f0f5c0a2580 2024-09-16T14:46:39.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:39.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:39.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:39.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient(hunting): my global_id is 104307 2024-09-16T14:46:39.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f40003530 con 0x7f0f5c0a2580 2024-09-16T14:46:39.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1308110083 0 0) 0x7f0f5c122870 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 >> v1:172.21.15.26:6790/0 conn(0x7f0f5c09fd40 legacy=0x7f0f5c133cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:39.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 >> v1:172.21.15.26:6789/0 conn(0x7f0f5c0a2580 legacy=0x7f0f5c1d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:39.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient: _finish_hunting 0 2024-09-16T14:46:39.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 1 monclient: found mon.b 2024-09-16T14:46:39.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:39.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:39.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c1246c0 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient: _finish_auth 0 2024-09-16T14:46:39.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient: _check_auth_tickets 2024-09-16T14:46:39.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:39.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f62bbd640 5 monclient: authenticate success, global_id 94660 2024-09-16T14:46:39.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f62bbd640 10 monclient: _renew_subs 2024-09-16T14:46:39.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f62bbd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:39.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f62bbd640 1 -- 172.21.15.189:0/419182308 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5c123e20 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f62bbd640 10 monclient: _renew_subs 2024-09-16T14:46:39.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f62bbd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:39.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f62bbd640 1 -- 172.21.15.189:0/419182308 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5c1c93e0 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f48001210 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:39.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0f48004da0 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:39.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:39.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f5affd640 4 set_mon_vals no callback set 2024-09-16T14:46:39.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 10 monclient: dump: 2024-09-16T14:46:39.933 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:39.933 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:39.933 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:39.933 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:39.934 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:39.934 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:39.934 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:39.934 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:39.934 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:39.934 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:39.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f62bbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:39.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0f4800f890 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f62bbd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:39.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f62bbd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:39.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f62bbd640 1 -- 172.21.15.189:0/419182308 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3c005050 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(91..91 src has 65..91) v4 ==== 4746+0+0 (unknown 2280409212 0 0) 0x7f0f480438d0 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.915+0000 7f0f60959640 10 monclient: get_auth_request con 0x7f0f400360e0 auth_method 0 2024-09-16T14:46:39.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.919+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0f48010ed0 con 0x7f0f5c0a1a90 2024-09-16T14:46:39.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.919+0000 7f0f597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:39.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:39.919+0000 7f0f597fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:40.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.067+0000 7f0f62bbd640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T14:46:40.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.067+0000 7f0f62bbd640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T14:46:40.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.067+0000 7f0f62bbd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:40.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.067+0000 7f0f62bbd640 1 -- 172.21.15.189:0/419182308 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]} v 0) v1 -- 0x7f0f3c005880 con 0x7f0f5c0a1a90 2024-09-16T14:46:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.691+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}]=0 moved item id -10 name 'foo' to location {rack=localrack,root=default} in crush map v92) v1 ==== 204+0+0 (unknown 1464787419 0 0) 0x7f0f48012f10 con 0x7f0f5c0a1a90 2024-09-16T14:46:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.691+0000 7f0f597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T14:46:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.691+0000 7f0f597fa640 10 monclient: _finish_command 2 = system:0 moved item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-16T14:46:40.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.787+0000 7f0f62bbd640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T14:46:40.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.787+0000 7f0f62bbd640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T14:46:40.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.787+0000 7f0f62bbd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:40.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.787+0000 7f0f62bbd640 1 -- 172.21.15.189:0/419182308 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]} v 0) v1 -- 0x7f0f3c005300 con 0x7f0f5c0a1a90 2024-09-16T14:46:40.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.787+0000 7f0f597fa640 1 -- 172.21.15.189:0/419182308 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}]=0 no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map v92) v1 ==== 214+0+0 (unknown 4238273258 0 0) 0x7f0f48014690 con 0x7f0f5c0a1a90 2024-09-16T14:46:40.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.787+0000 7f0f597fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T14:46:40.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.787+0000 7f0f597fa640 10 monclient: _finish_command 3 = system:0 no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-16T14:46:40.792 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-16T14:46:40.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.791+0000 7f0f62bbd640 1 -- 172.21.15.189:0/419182308 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0f400360e0 legacy=0x7f0f400385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:40.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.791+0000 7f0f62bbd640 10 monclient: shutdown 2024-09-16T14:46:40.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.791+0000 7f0f62bbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:40.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.791+0000 7f0f62bbd640 1 -- 172.21.15.189:0/419182308 >> v1:172.21.15.189:6789/0 conn(0x7f0f5c0a1a90 legacy=0x7f0f5c1ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:40.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.791+0000 7f0f62bbd640 1 -- 172.21.15.189:0/419182308 shutdown_connections 2024-09-16T14:46:40.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.791+0000 7f0f62bbd640 1 -- 172.21.15.189:0/419182308 >> 172.21.15.189:0/419182308 conn(0x7f0f5c0a7a70 msgr2=0x7f0f5c135380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:40.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.791+0000 7f0f62bbd640 1 -- 172.21.15.189:0/419182308 shutdown_connections 2024-09-16T14:46:40.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.791+0000 7f0f62bbd640 1 -- 172.21.15.189:0/419182308 wait complete. 2024-09-16T14:46:40.808 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush create-or-move osd.7 1.0 root=default 2024-09-16T14:46:40.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.923+0000 7fe3f5e81640 10 monclient: get_monmap_and_config 2024-09-16T14:46:40.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.923+0000 7fe3f5e81640 10 monclient: build_initial_monmap 2024-09-16T14:46:40.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.923+0000 7fe3f5e81640 10 monclient: monmap: 2024-09-16T14:46:40.929 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:40.930 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:40.930 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:40.929580+0000 2024-09-16T14:46:40.930 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:40.929580+0000 2024-09-16T14:46:40.930 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:40.930 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:40.930 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:40.930 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:40.930 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:40.930 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:40.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 1 Processor -- start 2024-09-16T14:46:40.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 1 -- start start 2024-09-16T14:46:40.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 10 monclient: init 2024-09-16T14:46:40.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:40.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:46:40.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 10 monclient(hunting): picked mon.noname-a con 0x7fe3f0099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:40.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 10 monclient(hunting): picked mon.noname-c con 0x7fe3f009c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:40.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 10 monclient(hunting): picked mon.noname-b con 0x7fe3f009e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:40.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f0099e50 con 0x7fe3f0099a70 2024-09-16T14:46:40.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f009dbd0 con 0x7fe3f009c550 2024-09-16T14:46:40.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f009ee20 con 0x7fe3f009e6d0 2024-09-16T14:46:40.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:40.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 10 monclient(hunting): authenticate will time out at 1183.045399s 2024-09-16T14:46:40.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3effff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fe3f009e6d0 legacy=0x7fe3f013b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:40.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3e6ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe3f009c550 0x7fe3f01379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35436/0 (socket says 172.21.15.189:35436) 2024-09-16T14:46:40.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3e6ffd640 1 -- 172.21.15.189:0/824649137 learned_addr learned my addr 172.21.15.189:0/824649137 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:40.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 1 -- 172.21.15.189:0/824649137 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793770398 0 0) 0x7fe3f009dbd0 con 0x7fe3f009c550 2024-09-16T14:46:40.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:40.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:40.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:40.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 10 monclient(hunting): my global_id is 94663 2024-09-16T14:46:40.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 1 -- 172.21.15.189:0/824649137 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3dc003530 con 0x7fe3f009c550 2024-09-16T14:46:40.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 1 -- 172.21.15.189:0/824649137 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 340666752 0 0) 0x7fe3dc003530 con 0x7fe3f009c550 2024-09-16T14:46:40.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 1 -- 172.21.15.189:0/824649137 >> v1:172.21.15.26:6790/0 conn(0x7fe3f009e6d0 legacy=0x7fe3f013b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:40.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 1 -- 172.21.15.189:0/824649137 >> v1:172.21.15.26:6789/0 conn(0x7fe3f0099a70 legacy=0x7fe3f009dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:40.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:40.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 1 monclient: found mon.noname-c 2024-09-16T14:46:40.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:40.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:40.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 1 -- 172.21.15.189:0/824649137 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f009f100 con 0x7fe3f009c550 2024-09-16T14:46:40.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 10 monclient: _finish_auth 0 2024-09-16T14:46:40.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:40.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:40.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 1 -- 172.21.15.189:0/824649137 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe3d8001620 con 0x7fe3f009c550 2024-09-16T14:46:40.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:40.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:40.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 10 monclient: dump: 2024-09-16T14:46:40.939 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:40.939 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:40.939 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:40.939 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:40.939 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:40.940 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:40.940 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:40.940 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:40.940 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:40.940 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:40.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 5 monclient: authenticate success, global_id 94663 2024-09-16T14:46:40.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:40.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 1 -- 172.21.15.189:0/824649137 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3d80018d0 con 0x7fe3f009c550 2024-09-16T14:46:40.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:40.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 1 -- 172.21.15.189:0/824649137 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe3d8001c60 con 0x7fe3f009c550 2024-09-16T14:46:40.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:40.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:40.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3eeffd640 10 monclient: dump: 2024-09-16T14:46:40.941 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:40.941 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:40.941 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:40.941 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:40.941 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:40.941 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:40.941 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:40.941 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:40.941 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:40.941 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:40.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:40.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 4 set_mon_vals no callback set 2024-09-16T14:46:40.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 10 monclient: shutdown 2024-09-16T14:46:40.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:40.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 1 -- 172.21.15.189:0/824649137 >> v1:172.21.15.189:6789/0 conn(0x7fe3f009c550 legacy=0x7fe3f01379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:40.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 1 -- 172.21.15.189:0/824649137 shutdown_connections 2024-09-16T14:46:40.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 1 -- 172.21.15.189:0/824649137 >> 172.21.15.189:0/824649137 conn(0x7fe3f00a3380 msgr2=0x7fe3f009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:40.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 1 -- 172.21.15.189:0/824649137 shutdown_connections 2024-09-16T14:46:40.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 1 -- 172.21.15.189:0/824649137 wait complete. 2024-09-16T14:46:40.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 10 monclient: build_initial_monmap 2024-09-16T14:46:40.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 10 monclient: monmap: 2024-09-16T14:46:40.942 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:40.942 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:40.942 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:40.942 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:40.942 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:40.943 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:40.943 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:40.943 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:40.943 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:40.943 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:40.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 1 Processor -- start 2024-09-16T14:46:40.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 1 -- start start 2024-09-16T14:46:40.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.927+0000 7fe3f5e81640 10 monclient: init 2024-09-16T14:46:40.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:40.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:40.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 10 monclient(hunting): picked mon.noname-c con 0x7fe3f0099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:40.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 10 monclient(hunting): picked mon.noname-b con 0x7fe3f009c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:40.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 10 monclient(hunting): picked mon.noname-a con 0x7fe3f009e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:40.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f009cf20 con 0x7fe3f009e6d0 2024-09-16T14:46:40.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f0133120 con 0x7fe3f009c550 2024-09-16T14:46:40.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f013b290 con 0x7fe3f0099a70 2024-09-16T14:46:40.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:40.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 10 monclient(hunting): authenticate will time out at 1183.048222s 2024-09-16T14:46:40.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3e6ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe3f009c550 0x7fe3f009a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35452/0 (socket says 172.21.15.189:35452) 2024-09-16T14:46:40.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3e6ffd640 1 -- 172.21.15.189:0/2898094585 learned_addr learned my addr 172.21.15.189:0/2898094585 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:40.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ef7fe640 1 -- 172.21.15.189:0/2898094585 >> v1:172.21.15.26:6790/0 conn(0x7fe3f0099a70 legacy=0x7fe3f009a220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:40.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470463230 0 0) 0x7fe3f0133120 con 0x7fe3f009c550 2024-09-16T14:46:40.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:40.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:40.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:40.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient(hunting): my global_id is 94666 2024-09-16T14:46:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3d0003530 con 0x7fe3f009c550 2024-09-16T14:46:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1930826147 0 0) 0x7fe3d0003530 con 0x7fe3f009c550 2024-09-16T14:46:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3f0133120 con 0x7fe3f009c550 2024-09-16T14:46:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe3d8001640 con 0x7fe3f009c550 2024-09-16T14:46:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient(hunting): dump: 2024-09-16T14:46:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:40.946 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:40.946 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:40.946 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:40.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:40.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513886123 0 0) 0x7fe3f009cf20 con 0x7fe3f009e6d0 2024-09-16T14:46:40.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:40.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:40.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:40.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient(hunting): my global_id is 104313 2024-09-16T14:46:40.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3d0003530 con 0x7fe3f009e6d0 2024-09-16T14:46:40.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 334822369 0 0) 0x7fe3f0133120 con 0x7fe3f009c550 2024-09-16T14:46:40.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 >> v1:172.21.15.26:6790/0 conn(0x7fe3f0099a70 legacy=0x7fe3f009a220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:40.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 >> v1:172.21.15.26:6789/0 conn(0x7fe3f009e6d0 legacy=0x7fe3f01d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:40.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient: _finish_hunting 0 2024-09-16T14:46:40.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 1 monclient: found mon.b 2024-09-16T14:46:40.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:40.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:40.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f009b040 con 0x7fe3f009c550 2024-09-16T14:46:40.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient: _finish_auth 0 2024-09-16T14:46:40.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient: _check_auth_tickets 2024-09-16T14:46:40.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:40.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 5 monclient: authenticate success, global_id 94666 2024-09-16T14:46:40.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 10 monclient: _renew_subs 2024-09-16T14:46:40.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:40.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 1 -- 172.21.15.189:0/2898094585 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f013afb0 con 0x7fe3f009c550 2024-09-16T14:46:40.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 10 monclient: _renew_subs 2024-09-16T14:46:40.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:40.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 1 -- 172.21.15.189:0/2898094585 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f009be70 con 0x7fe3f009c550 2024-09-16T14:46:40.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3d8004c00 con 0x7fe3f009c550 2024-09-16T14:46:40.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:40.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe3d8004f10 con 0x7fe3f009c550 2024-09-16T14:46:40.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:40.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:40.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ee7fc640 4 set_mon_vals no callback set 2024-09-16T14:46:40.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 10 monclient: dump: 2024-09-16T14:46:40.948 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:40.948 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:40.949 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:40.949 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:40.949 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:40.949 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:40.949 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:40.949 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:40.949 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:40.949 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:40.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:40.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:40.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:40.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3f5e81640 1 -- 172.21.15.189:0/2898094585 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d4005050 con 0x7fe3f009c550 2024-09-16T14:46:40.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe3d800fa00 con 0x7fe3f009c550 2024-09-16T14:46:40.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(92..92 src has 65..92) v4 ==== 5043+0+0 (unknown 3640320179 0 0) 0x7fe3d8043b00 con 0x7fe3f009c550 2024-09-16T14:46:40.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.931+0000 7fe3ef7fe640 10 monclient: get_auth_request con 0x7fe3d0036110 auth_method 0 2024-09-16T14:46:40.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.935+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe3d8014970 con 0x7fe3f009c550 2024-09-16T14:46:40.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.935+0000 7fe3ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:40.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:40.935+0000 7fe3ecff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:41.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.099+0000 7fe3f5e81640 10 start_mon_command cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T14:46:41.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.099+0000 7fe3f5e81640 10 monclient: _send_command 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T14:46:41.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.099+0000 7fe3f5e81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:41.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.099+0000 7fe3f5e81640 1 -- 172.21.15.189:0/2898094585 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]} v 0) v1 -- 0x7fe3d4005880 con 0x7fe3f009c550 2024-09-16T14:46:41.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.691+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}]=0 create-or-move updating item name 'osd.7' weight 1 at location {root=default} to crush map v93) v1 ==== 212+0+0 (unknown 2234209046 0 0) 0x7fe3d8013010 con 0x7fe3f009c550 2024-09-16T14:46:41.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.691+0000 7fe3ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T14:46:41.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.691+0000 7fe3ecff9640 10 monclient: _finish_command 2 = system:0 create-or-move updating item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-16T14:46:41.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.791+0000 7fe3f5e81640 10 start_mon_command cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T14:46:41.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.791+0000 7fe3f5e81640 10 monclient: _send_command 3 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T14:46:41.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.791+0000 7fe3f5e81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:41.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.791+0000 7fe3f5e81640 1 -- 172.21.15.189:0/2898094585 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]} v 0) v1 -- 0x7fe3d4005300 con 0x7fe3f009c550 2024-09-16T14:46:41.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.795+0000 7fe3ecff9640 1 -- 172.21.15.189:0/2898094585 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}]=0 create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map v93) v1 ==== 211+0+0 (unknown 1298689654 0 0) 0x7fe3d8014790 con 0x7fe3f009c550 2024-09-16T14:46:41.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.795+0000 7fe3ecff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T14:46:41.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.795+0000 7fe3ecff9640 10 monclient: _finish_command 3 = system:0 create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-16T14:46:41.798 INFO:tasks.workunit.client.0.smithi189.stderr:create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-16T14:46:41.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.795+0000 7fe3f5e81640 1 -- 172.21.15.189:0/2898094585 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe3d0036110 legacy=0x7fe3d00385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:41.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.795+0000 7fe3f5e81640 10 monclient: shutdown 2024-09-16T14:46:41.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.795+0000 7fe3f5e81640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:41.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.795+0000 7fe3f5e81640 1 -- 172.21.15.189:0/2898094585 >> v1:172.21.15.189:6789/0 conn(0x7fe3f009c550 legacy=0x7fe3f009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:41.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.795+0000 7fe3f5e81640 1 -- 172.21.15.189:0/2898094585 shutdown_connections 2024-09-16T14:46:41.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.795+0000 7fe3f5e81640 1 -- 172.21.15.189:0/2898094585 >> 172.21.15.189:0/2898094585 conn(0x7fe3f00a3380 msgr2=0x7fe3f00a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:41.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.795+0000 7fe3f5e81640 1 -- 172.21.15.189:0/2898094585 shutdown_connections 2024-09-16T14:46:41.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.795+0000 7fe3f5e81640 1 -- 172.21.15.189:0/2898094585 wait complete. 2024-09-16T14:46:41.814 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush move osd.7 host=foo 2024-09-16T14:46:41.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.923+0000 7f5a75156640 10 monclient: get_monmap_and_config 2024-09-16T14:46:41.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.923+0000 7f5a75156640 10 monclient: build_initial_monmap 2024-09-16T14:46:41.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.923+0000 7f5a75156640 10 monclient: monmap: 2024-09-16T14:46:41.928 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:41.928 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:41.928527+0000 2024-09-16T14:46:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:41.928527+0000 2024-09-16T14:46:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:41.929 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:41.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.923+0000 7f5a75156640 1 Processor -- start 2024-09-16T14:46:41.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 1 -- start start 2024-09-16T14:46:41.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient: init 2024-09-16T14:46:41.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:41.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:41.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient(hunting): picked mon.noname-c con 0x7f5a7009df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:41.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient(hunting): picked mon.noname-b con 0x7f5a7009e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:41.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient(hunting): picked mon.noname-a con 0x7f5a7009e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:41.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7009ee20 con 0x7f5a7009e740 2024-09-16T14:46:41.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7009f100 con 0x7f5a7009df10 2024-09-16T14:46:41.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7009f3e0 con 0x7f5a7009e2f0 2024-09-16T14:46:41.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:41.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient(hunting): authenticate will time out at 1184.044726s 2024-09-16T14:46:41.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6e59c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f5a7009e2f0 legacy=0x7f5a701378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:41.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6ed9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5a7009df10 0x7f5a7009d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35468/0 (socket says 172.21.15.189:35468) 2024-09-16T14:46:41.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6ed9d640 1 -- 172.21.15.189:0/2005089888 learned_addr learned my addr 172.21.15.189:0/2005089888 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:41.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 1 -- 172.21.15.189:0/2005089888 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108469368 0 0) 0x7f5a7009f100 con 0x7f5a7009df10 2024-09-16T14:46:41.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:46:41.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:41.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:41.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 10 monclient(hunting): my global_id is 94669 2024-09-16T14:46:41.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 1 -- 172.21.15.189:0/2005089888 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a58003530 con 0x7f5a7009df10 2024-09-16T14:46:41.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 1 -- 172.21.15.189:0/2005089888 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 635917851 0 0) 0x7f5a58003530 con 0x7f5a7009df10 2024-09-16T14:46:41.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 1 -- 172.21.15.189:0/2005089888 >> v1:172.21.15.26:6790/0 conn(0x7f5a7009e2f0 legacy=0x7f5a701378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:41.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 1 -- 172.21.15.189:0/2005089888 >> v1:172.21.15.26:6789/0 conn(0x7f5a7009e740 legacy=0x7f5a7013b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:41.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 10 monclient: _finish_hunting 0 2024-09-16T14:46:41.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 1 monclient: found mon.noname-c 2024-09-16T14:46:41.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:41.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:41.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 1 -- 172.21.15.189:0/2005089888 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7009c690 con 0x7f5a7009df10 2024-09-16T14:46:41.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 10 monclient: _finish_auth 0 2024-09-16T14:46:41.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 10 monclient: _check_auth_tickets 2024-09-16T14:46:41.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:41.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 1 -- 172.21.15.189:0/2005089888 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5a600019a0 con 0x7f5a7009df10 2024-09-16T14:46:41.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:41.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:41.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 10 monclient: dump: 2024-09-16T14:46:41.939 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:41.939 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:41.939 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:41.939 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:41.939 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:41.940 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:41.940 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:41.940 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:41.940 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:41.940 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:41.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 5 monclient: authenticate success, global_id 94669 2024-09-16T14:46:41.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:41.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 1 -- 172.21.15.189:0/2005089888 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a60001cc0 con 0x7f5a7009df10 2024-09-16T14:46:41.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:41.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 1 -- 172.21.15.189:0/2005089888 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5a60002090 con 0x7f5a7009df10 2024-09-16T14:46:41.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:41.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:41.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6dd9b640 10 monclient: dump: 2024-09-16T14:46:41.941 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:41.941 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:41.941 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:41.941 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:41.941 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:41.941 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:41.941 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:41.941 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:41.941 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:41.941 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:41.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:41.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 4 set_mon_vals no callback set 2024-09-16T14:46:41.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient: shutdown 2024-09-16T14:46:41.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:41.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 1 -- 172.21.15.189:0/2005089888 >> v1:172.21.15.189:6789/0 conn(0x7f5a7009df10 legacy=0x7f5a7009d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:41.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 1 -- 172.21.15.189:0/2005089888 shutdown_connections 2024-09-16T14:46:41.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 1 -- 172.21.15.189:0/2005089888 >> 172.21.15.189:0/2005089888 conn(0x7f5a700a3320 msgr2=0x7f5a7009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:41.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 1 -- 172.21.15.189:0/2005089888 shutdown_connections 2024-09-16T14:46:41.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 1 -- 172.21.15.189:0/2005089888 wait complete. 2024-09-16T14:46:41.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient: build_initial_monmap 2024-09-16T14:46:41.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient: monmap: 2024-09-16T14:46:41.942 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:41.942 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:41.943 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:41.943 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:41.943 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:41.943 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:41.943 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:41.943 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:41.943 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:41.943 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:41.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 1 Processor -- start 2024-09-16T14:46:41.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 1 -- start start 2024-09-16T14:46:41.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient: init 2024-09-16T14:46:41.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:41.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:41.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient(hunting): picked mon.noname-c con 0x7f5a7009df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:41.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient(hunting): picked mon.noname-a con 0x7f5a7009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:41.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient(hunting): picked mon.noname-b con 0x7f5a7009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:41.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7006b380 con 0x7f5a7009e2f0 2024-09-16T14:46:41.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7012c5a0 con 0x7f5a7009e740 2024-09-16T14:46:41.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7006cb00 con 0x7f5a7009df10 2024-09-16T14:46:41.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:41.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a75156640 10 monclient(hunting): authenticate will time out at 1184.047878s 2024-09-16T14:46:41.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6ed9d640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f5a7009df10 legacy=0x7f5a7012b780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:41.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6f59e640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5a7009e740 0x7f5a701cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35470/0 (socket says 172.21.15.189:35470) 2024-09-16T14:46:41.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.927+0000 7f5a6f59e640 1 -- 172.21.15.189:0/684212246 learned_addr learned my addr 172.21.15.189:0/684212246 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:41.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 1 -- 172.21.15.189:0/684212246 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709962659 0 0) 0x7f5a7012c5a0 con 0x7f5a7009e740 2024-09-16T14:46:41.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:41.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:41.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:41.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 10 monclient(hunting): my global_id is 94672 2024-09-16T14:46:41.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 1 -- 172.21.15.189:0/684212246 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a5c003530 con 0x7f5a7009e740 2024-09-16T14:46:41.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 1 -- 172.21.15.189:0/684212246 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4124098072 0 0) 0x7f5a5c003530 con 0x7f5a7009e740 2024-09-16T14:46:41.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 1 -- 172.21.15.189:0/684212246 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a7012c5a0 con 0x7f5a7009e740 2024-09-16T14:46:41.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 1 -- 172.21.15.189:0/684212246 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5a680018a0 con 0x7f5a7009e740 2024-09-16T14:46:41.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:41.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:41.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 10 monclient(hunting): dump: 2024-09-16T14:46:41.946 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:41.946 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:41.946 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:41.946 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:41.946 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:41.946 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:41.946 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:41.946 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:41.946 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:41.947 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:41.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:41.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 1 -- 172.21.15.189:0/684212246 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 323660936 0 0) 0x7f5a7012c5a0 con 0x7f5a7009e740 2024-09-16T14:46:41.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 1 -- 172.21.15.189:0/684212246 >> v1:172.21.15.26:6790/0 conn(0x7f5a7009df10 legacy=0x7f5a7012b780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:41.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 1 -- 172.21.15.189:0/684212246 >> v1:172.21.15.26:6789/0 conn(0x7f5a7009e2f0 legacy=0x7f5a7012be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:41.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:41.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 1 monclient: found mon.b 2024-09-16T14:46:41.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:41.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:41.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 1 -- 172.21.15.189:0/684212246 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7012edc0 con 0x7f5a7009e740 2024-09-16T14:46:41.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:41.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:41.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:41.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a75156640 5 monclient: authenticate success, global_id 94672 2024-09-16T14:46:41.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a75156640 10 monclient: _renew_subs 2024-09-16T14:46:41.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a75156640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:41.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a75156640 1 -- 172.21.15.189:0/684212246 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7012c9f0 con 0x7f5a7009e740 2024-09-16T14:46:41.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a75156640 10 monclient: _renew_subs 2024-09-16T14:46:41.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a75156640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:41.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a75156640 1 -- 172.21.15.189:0/684212246 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7012d890 con 0x7f5a7009e740 2024-09-16T14:46:41.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 1 -- 172.21.15.189:0/684212246 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a68003620 con 0x7f5a7009e740 2024-09-16T14:46:41.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:41.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 1 -- 172.21.15.189:0/684212246 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5a680039b0 con 0x7f5a7009e740 2024-09-16T14:46:41.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:41.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:41.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 10 monclient: dump: 2024-09-16T14:46:41.949 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:41.949 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:41.949 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:41.949 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:41.949 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:41.949 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:41.949 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:41.949 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:41.949 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:41.949 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:41.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a6d59a640 4 set_mon_vals no callback set 2024-09-16T14:46:41.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 1 -- 172.21.15.189:0/684212246 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5a68003b10 con 0x7f5a7009e740 2024-09-16T14:46:41.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a75156640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:41.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a75156640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:41.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a75156640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:41.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a75156640 1 -- 172.21.15.189:0/684212246 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a50005050 con 0x7f5a7009e740 2024-09-16T14:46:41.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a577fe640 1 -- 172.21.15.189:0/684212246 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(93..93 src has 65..93) v4 ==== 5064+0+0 (unknown 963460647 0 0) 0x7f5a680435f0 con 0x7f5a7009e740 2024-09-16T14:46:41.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.931+0000 7f5a6ed9d640 10 monclient: get_auth_request con 0x7f5a5c036050 auth_method 0 2024-09-16T14:46:41.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.935+0000 7f5a577fe640 1 -- 172.21.15.189:0/684212246 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5a68014600 con 0x7f5a7009e740 2024-09-16T14:46:41.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.935+0000 7f5a577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:41.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:41.935+0000 7f5a577fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:42.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.091+0000 7f5a75156640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T14:46:42.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.091+0000 7f5a75156640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T14:46:42.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.091+0000 7f5a75156640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:42.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.091+0000 7f5a75156640 1 -- 172.21.15.189:0/684212246 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]} v 0) v1 -- 0x7f5a50005880 con 0x7f5a7009e740 2024-09-16T14:46:42.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.699+0000 7f5a577fe640 1 -- 172.21.15.189:0/684212246 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}]=0 moved item id 7 name 'osd.7' to location {host=foo} in crush map v94) v1 ==== 165+0+0 (unknown 995047785 0 0) 0x7f5a68012a70 con 0x7f5a7009e740 2024-09-16T14:46:42.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.699+0000 7f5a577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T14:46:42.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.699+0000 7f5a577fe640 10 monclient: _finish_command 2 = system:0 moved item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-16T14:46:42.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.807+0000 7f5a75156640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T14:46:42.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.807+0000 7f5a75156640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T14:46:42.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.807+0000 7f5a75156640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:42.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.807+0000 7f5a75156640 1 -- 172.21.15.189:0/684212246 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]} v 0) v1 -- 0x7f5a50005300 con 0x7f5a7009e740 2024-09-16T14:46:42.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.807+0000 7f5a577fe640 1 -- 172.21.15.189:0/684212246 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}]=0 no need to move item id 7 name 'osd.7' to location {host=foo} in crush map v94) v1 ==== 175+0+0 (unknown 2455362110 0 0) 0x7f5a680141f0 con 0x7f5a7009e740 2024-09-16T14:46:42.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.807+0000 7f5a577fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T14:46:42.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.807+0000 7f5a577fe640 10 monclient: _finish_command 3 = system:0 no need to move item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-16T14:46:42.812 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-16T14:46:42.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.811+0000 7f5a75156640 1 -- 172.21.15.189:0/684212246 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5a5c036050 legacy=0x7f5a5c038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:42.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.811+0000 7f5a75156640 10 monclient: shutdown 2024-09-16T14:46:42.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.811+0000 7f5a75156640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:42.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.811+0000 7f5a75156640 1 -- 172.21.15.189:0/684212246 >> v1:172.21.15.189:6789/0 conn(0x7f5a7009e740 legacy=0x7f5a701cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:42.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.811+0000 7f5a75156640 1 -- 172.21.15.189:0/684212246 shutdown_connections 2024-09-16T14:46:42.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.811+0000 7f5a75156640 1 -- 172.21.15.189:0/684212246 >> 172.21.15.189:0/684212246 conn(0x7f5a700a3320 msgr2=0x7f5a7009d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:42.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.811+0000 7f5a75156640 1 -- 172.21.15.189:0/684212246 shutdown_connections 2024-09-16T14:46:42.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.811+0000 7f5a75156640 1 -- 172.21.15.189:0/684212246 wait complete. 2024-09-16T14:46:42.828 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd find osd.7 2024-09-16T14:46:42.828 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep host 2024-09-16T14:46:42.828 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep foo 2024-09-16T14:46:42.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient: get_monmap_and_config 2024-09-16T14:46:42.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient: build_initial_monmap 2024-09-16T14:46:42.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient: monmap: 2024-09-16T14:46:42.926 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:42.926 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:42.926 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:42.925828+0000 2024-09-16T14:46:42.926 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:42.925828+0000 2024-09-16T14:46:42.926 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:42.926 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:42.926 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:42.926 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:42.926 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:42.926 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:42.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 1 Processor -- start 2024-09-16T14:46:42.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 1 -- start start 2024-09-16T14:46:42.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient: init 2024-09-16T14:46:42.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:42.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:42.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient(hunting): picked mon.noname-a con 0x7fa13c1315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:42.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient(hunting): picked mon.noname-b con 0x7fa13c09b320 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:42.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient(hunting): picked mon.noname-c con 0x7fa13c132090 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:42.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c09edd0 con 0x7fa13c1315a0 2024-09-16T14:46:42.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c09bc00 con 0x7fa13c132090 2024-09-16T14:46:42.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c09e900 con 0x7fa13c09b320 2024-09-16T14:46:42.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:42.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient(hunting): authenticate will time out at 1185.041237s 2024-09-16T14:46:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa141311640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa13c132090 0x7fa13c13a200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35486/0 (socket says 172.21.15.189:35486) 2024-09-16T14:46:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa141311640 1 -- 172.21.15.189:0/2307254174 learned_addr learned my addr 172.21.15.189:0/2307254174 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13bfff640 1 -- 172.21.15.189:0/2307254174 >> v1:172.21.15.26:6790/0 conn(0x7fa13c09b320 legacy=0x7fa13c136ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 1 -- 172.21.15.189:0/2307254174 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44421095 0 0) 0x7fa13c09bc00 con 0x7fa13c132090 2024-09-16T14:46:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 10 monclient(hunting): my global_id is 94675 2024-09-16T14:46:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 1 -- 172.21.15.189:0/2307254174 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa128003530 con 0x7fa13c132090 2024-09-16T14:46:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 1 -- 172.21.15.189:0/2307254174 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2101006858 0 0) 0x7fa128003530 con 0x7fa13c132090 2024-09-16T14:46:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 1 -- 172.21.15.189:0/2307254174 >> v1:172.21.15.26:6790/0 conn(0x7fa13c09b320 legacy=0x7fa13c136ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 1 -- 172.21.15.189:0/2307254174 >> v1:172.21.15.26:6789/0 conn(0x7fa13c1315a0 legacy=0x7fa13c131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:42.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:42.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 1 monclient: found mon.noname-c 2024-09-16T14:46:42.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:42.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:42.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 1 -- 172.21.15.189:0/2307254174 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c09b920 con 0x7fa13c132090 2024-09-16T14:46:42.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:42.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:42.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:42.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 5 monclient: authenticate success, global_id 94675 2024-09-16T14:46:42.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:42.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 1 -- 172.21.15.189:0/2307254174 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa134001620 con 0x7fa13c132090 2024-09-16T14:46:42.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:42.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:42.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 10 monclient: dump: 2024-09-16T14:46:42.936 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:42.936 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:42.936 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:42.936 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:42.936 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:42.936 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:42.936 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:42.936 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:42.936 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:42.936 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:42.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:42.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 1 -- 172.21.15.189:0/2307254174 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa134001940 con 0x7fa13c132090 2024-09-16T14:46:42.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:42.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 1 -- 172.21.15.189:0/2307254174 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa134001d10 con 0x7fa13c132090 2024-09-16T14:46:42.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:42.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 4 set_mon_vals no callback set 2024-09-16T14:46:42.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:42.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:42.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa13b7fe640 10 monclient: dump: 2024-09-16T14:46:42.937 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:42.937 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:42.937 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:42.937 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:42.937 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:42.937 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:42.938 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:42.938 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:42.938 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:42.938 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:42.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient: shutdown 2024-09-16T14:46:42.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:42.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 1 -- 172.21.15.189:0/2307254174 >> v1:172.21.15.189:6789/0 conn(0x7fa13c132090 legacy=0x7fa13c13a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:42.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 1 -- 172.21.15.189:0/2307254174 shutdown_connections 2024-09-16T14:46:42.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 1 -- 172.21.15.189:0/2307254174 >> 172.21.15.189:0/2307254174 conn(0x7fa13c0a3320 msgr2=0x7fa13c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:42.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 1 -- 172.21.15.189:0/2307254174 shutdown_connections 2024-09-16T14:46:42.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 1 -- 172.21.15.189:0/2307254174 wait complete. 2024-09-16T14:46:42.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient: build_initial_monmap 2024-09-16T14:46:42.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient: monmap: 2024-09-16T14:46:42.939 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:42.939 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:42.939 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:42.939 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:42.939 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:42.939 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:42.939 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:42.939 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:42.939 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:42.939 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:42.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 1 Processor -- start 2024-09-16T14:46:42.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 1 -- start start 2024-09-16T14:46:42.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient: init 2024-09-16T14:46:42.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:42.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:46:42.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient(hunting): picked mon.noname-b con 0x7fa13c1315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:42.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient(hunting): picked mon.noname-a con 0x7fa13c132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:42.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient(hunting): picked mon.noname-c con 0x7fa13c12c650 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:42.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c06b380 con 0x7fa13c132090 2024-09-16T14:46:42.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c12ed30 con 0x7fa13c1315a0 2024-09-16T14:46:42.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c12f310 con 0x7fa13c12c650 2024-09-16T14:46:42.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:42.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.923+0000 7fa142d74640 10 monclient(hunting): authenticate will time out at 1185.043925s 2024-09-16T14:46:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa140b10640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa13c1315a0 0x7fa13c129800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35492/0 (socket says 172.21.15.189:35492) 2024-09-16T14:46:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa140b10640 1 -- 172.21.15.189:0/3143575455 learned_addr learned my addr 172.21.15.189:0/3143575455 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa141311640 1 -- 172.21.15.189:0/3143575455 >> v1:172.21.15.26:6790/0 conn(0x7fa13c12c650 legacy=0x7fa13c1cf5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065866731 0 0) 0x7fa13c12ed30 con 0x7fa13c1315a0 2024-09-16T14:46:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient(hunting): my global_id is 94678 2024-09-16T14:46:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa11c003530 con 0x7fa13c1315a0 2024-09-16T14:46:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3724786000 0 0) 0x7fa11c003530 con 0x7fa13c1315a0 2024-09-16T14:46:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa13c12ed30 con 0x7fa13c1315a0 2024-09-16T14:46:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa12c001cc0 con 0x7fa13c1315a0 2024-09-16T14:46:42.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient(hunting): dump: 2024-09-16T14:46:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:42.942 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985565101 0 0) 0x7fa13c06b380 con 0x7fa13c132090 2024-09-16T14:46:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient(hunting): my global_id is 104325 2024-09-16T14:46:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa11c003530 con 0x7fa13c132090 2024-09-16T14:46:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 900233874 0 0) 0x7fa13c12ed30 con 0x7fa13c1315a0 2024-09-16T14:46:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 >> v1:172.21.15.26:6790/0 conn(0x7fa13c12c650 legacy=0x7fa13c1cf5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 >> v1:172.21.15.26:6789/0 conn(0x7fa13c132090 legacy=0x7fa13c1cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient: _finish_hunting 0 2024-09-16T14:46:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 1 monclient: found mon.b 2024-09-16T14:46:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c130250 con 0x7fa13c1315a0 2024-09-16T14:46:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient: _finish_auth 0 2024-09-16T14:46:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient: _check_auth_tickets 2024-09-16T14:46:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa142d74640 5 monclient: authenticate success, global_id 94678 2024-09-16T14:46:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa142d74640 10 monclient: _renew_subs 2024-09-16T14:46:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa142d74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa142d74640 1 -- 172.21.15.189:0/3143575455 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa13c12fcf0 con 0x7fa13c1315a0 2024-09-16T14:46:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa142d74640 10 monclient: _renew_subs 2024-09-16T14:46:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa142d74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa142d74640 1 -- 172.21.15.189:0/3143575455 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa13c129f10 con 0x7fa13c1315a0 2024-09-16T14:46:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa12c003a20 con 0x7fa13c1315a0 2024-09-16T14:46:42.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa12c003db0 con 0x7fa13c1315a0 2024-09-16T14:46:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 10 monclient: dump: 2024-09-16T14:46:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:42.946 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa13affd640 4 set_mon_vals no callback set 2024-09-16T14:46:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa142d74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa142d74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa142d74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa142d74640 1 -- 172.21.15.189:0/3143575455 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa120005050 con 0x7fa13c1315a0 2024-09-16T14:46:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa12c003f90 con 0x7fa13c1315a0 2024-09-16T14:46:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(94..94 src has 65..94) v4 ==== 5238+0+0 (unknown 3191735785 0 0) 0x7fa12c043c50 con 0x7fa13c1315a0 2024-09-16T14:46:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.927+0000 7fa13bfff640 10 monclient: get_auth_request con 0x7fa11c0361d0 auth_method 0 2024-09-16T14:46:42.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.931+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa12c014b70 con 0x7fa13c1315a0 2024-09-16T14:46:42.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.931+0000 7fa1397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:42.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:42.931+0000 7fa1397fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:43.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.075+0000 7fa142d74640 10 start_mon_command cmd=[{"prefix": "osd find", "id": 7}] 2024-09-16T14:46:43.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.075+0000 7fa142d74640 10 monclient: _send_command 2 [{"prefix": "osd find", "id": 7}] 2024-09-16T14:46:43.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.075+0000 7fa142d74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:43.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.075+0000 7fa142d74640 1 -- 172.21.15.189:0/3143575455 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd find", "id": 7} v 0) v1 -- 0x7fa120005880 con 0x7fa13c1315a0 2024-09-16T14:46:43.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.079+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd find", "id": 7}]=0 v94) v1 ==== 65+0+226 (unknown 188148463 0 2157906930) 0x7fa12c014940 con 0x7fa13c1315a0 2024-09-16T14:46:43.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.079+0000 7fa1397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd find", "id": 7}] 2024-09-16T14:46:43.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.079+0000 7fa1397fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:43.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.163+0000 7fa142d74640 10 start_mon_command cmd=[{"prefix": "osd find", "id": 7}] 2024-09-16T14:46:43.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.163+0000 7fa142d74640 10 monclient: _send_command 3 [{"prefix": "osd find", "id": 7}] 2024-09-16T14:46:43.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.163+0000 7fa142d74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:43.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.163+0000 7fa142d74640 1 -- 172.21.15.189:0/3143575455 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd find", "id": 7} v 0) v1 -- 0x7fa120005370 con 0x7fa13c1315a0 2024-09-16T14:46:43.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.163+0000 7fa1397fa640 1 -- 172.21.15.189:0/3143575455 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd find", "id": 7}]=0 v94) v1 ==== 65+0+226 (unknown 188148463 0 2157906930) 0x7fa12c0154a0 con 0x7fa13c1315a0 2024-09-16T14:46:43.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.163+0000 7fa1397fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd find", "id": 7}] 2024-09-16T14:46:43.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.163+0000 7fa1397fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:46:43.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.167+0000 7fa142d74640 1 -- 172.21.15.189:0/3143575455 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa11c0361d0 legacy=0x7fa11c038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:43.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.167+0000 7fa142d74640 10 monclient: shutdown 2024-09-16T14:46:43.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.167+0000 7fa142d74640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:43.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.167+0000 7fa142d74640 1 -- 172.21.15.189:0/3143575455 >> v1:172.21.15.189:6789/0 conn(0x7fa13c1315a0 legacy=0x7fa13c129800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:43.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.167+0000 7fa142d74640 1 -- 172.21.15.189:0/3143575455 shutdown_connections 2024-09-16T14:46:43.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.167+0000 7fa142d74640 1 -- 172.21.15.189:0/3143575455 >> 172.21.15.189:0/3143575455 conn(0x7fa13c0a3320 msgr2=0x7fa13c13ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:43.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.167+0000 7fa142d74640 1 -- 172.21.15.189:0/3143575455 shutdown_connections 2024-09-16T14:46:43.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.167+0000 7fa142d74640 1 -- 172.21.15.189:0/3143575455 wait complete. 2024-09-16T14:46:43.184 INFO:tasks.workunit.client.0.smithi189.stdout: "host": "foo", 2024-09-16T14:46:43.185 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm osd.7 2024-09-16T14:46:43.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776ee0f640 10 monclient: get_monmap_and_config 2024-09-16T14:46:43.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776ee0f640 10 monclient: build_initial_monmap 2024-09-16T14:46:43.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776ee0f640 10 monclient: monmap: 2024-09-16T14:46:43.299 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:43.299 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:43.299 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:43.298977+0000 2024-09-16T14:46:43.299 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:43.298977+0000 2024-09-16T14:46:43.299 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:43.299 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:43.299 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:43.300 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:43.300 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:43.300 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:43.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776ee0f640 1 Processor -- start 2024-09-16T14:46:43.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776ee0f640 1 -- start start 2024-09-16T14:46:43.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776ee0f640 10 monclient: init 2024-09-16T14:46:43.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776ee0f640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:43.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776ee0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:43.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776ee0f640 10 monclient(hunting): picked mon.noname-c con 0x7f77680a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:43.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776ee0f640 10 monclient(hunting): picked mon.noname-b con 0x7f77680a25f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:43.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776ee0f640 10 monclient(hunting): picked mon.noname-a con 0x7f776813b430 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:43.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776ee0f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680a30b0 con 0x7f776813b430 2024-09-16T14:46:43.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776ee0f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680a2e30 con 0x7f77680a1b00 2024-09-16T14:46:43.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776ee0f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776809eb50 con 0x7f77680a25f0 2024-09-16T14:46:43.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776ee0f640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:43.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776ee0f640 10 monclient(hunting): authenticate will time out at 1185.415443s 2024-09-16T14:46:43.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776cbab640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f77680a1b00 0x7f77680a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35498/0 (socket says 172.21.15.189:35498) 2024-09-16T14:46:43.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f7767fff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f77680a25f0 legacy=0x7f7768139d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:43.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f776cbab640 1 -- 172.21.15.189:0/1345448142 learned_addr learned my addr 172.21.15.189:0/1345448142 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:43.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f77677fe640 1 -- 172.21.15.189:0/1345448142 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81502730 0 0) 0x7f77680a2e30 con 0x7f77680a1b00 2024-09-16T14:46:43.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f77677fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:43.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f77677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:43.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f77677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:43.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f77677fe640 10 monclient(hunting): my global_id is 94681 2024-09-16T14:46:43.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.295+0000 7f77677fe640 1 -- 172.21.15.189:0/1345448142 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7750003530 con 0x7f77680a1b00 2024-09-16T14:46:43.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 1 -- 172.21.15.189:0/1345448142 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 653929815 0 0) 0x7f7750003530 con 0x7f77680a1b00 2024-09-16T14:46:43.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 1 -- 172.21.15.189:0/1345448142 >> v1:172.21.15.26:6790/0 conn(0x7f77680a25f0 legacy=0x7f7768139d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:43.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 1 -- 172.21.15.189:0/1345448142 >> v1:172.21.15.26:6789/0 conn(0x7f776813b430 legacy=0x7f776813d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:43.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:43.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 1 monclient: found mon.noname-c 2024-09-16T14:46:43.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:43.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:43.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 1 -- 172.21.15.189:0/1345448142 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776809ee30 con 0x7f77680a1b00 2024-09-16T14:46:43.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:43.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:43.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:43.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 1 -- 172.21.15.189:0/1345448142 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f77580019a0 con 0x7f77680a1b00 2024-09-16T14:46:43.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:43.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:43.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 10 monclient: dump: 2024-09-16T14:46:43.305 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:43.305 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:43.305 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:43.306 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:43.306 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:43.306 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:43.306 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:43.306 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:43.306 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:43.306 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:43.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 5 monclient: authenticate success, global_id 94681 2024-09-16T14:46:43.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:43.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 1 -- 172.21.15.189:0/1345448142 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7758001cc0 con 0x7f77680a1b00 2024-09-16T14:46:43.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:43.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:43.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 1 -- 172.21.15.189:0/1345448142 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7758002050 con 0x7f77680a1b00 2024-09-16T14:46:43.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 4 set_mon_vals no callback set 2024-09-16T14:46:43.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:43.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:43.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77677fe640 10 monclient: dump: 2024-09-16T14:46:43.307 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:43.307 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:43.307 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:43.307 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:43.307 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:43.307 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:43.307 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:43.307 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:43.307 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:43.307 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:43.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 10 monclient: shutdown 2024-09-16T14:46:43.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:43.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 1 -- 172.21.15.189:0/1345448142 >> v1:172.21.15.189:6789/0 conn(0x7f77680a1b00 legacy=0x7f77680a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:43.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 1 -- 172.21.15.189:0/1345448142 shutdown_connections 2024-09-16T14:46:43.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 1 -- 172.21.15.189:0/1345448142 >> 172.21.15.189:0/1345448142 conn(0x7f77680a7dd0 msgr2=0x7f77680a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:43.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 1 -- 172.21.15.189:0/1345448142 shutdown_connections 2024-09-16T14:46:43.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 1 -- 172.21.15.189:0/1345448142 wait complete. 2024-09-16T14:46:43.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 10 monclient: build_initial_monmap 2024-09-16T14:46:43.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 10 monclient: monmap: 2024-09-16T14:46:43.308 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:43.308 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:43.308 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:43.309 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:43.309 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:43.309 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:43.309 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:43.309 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:43.309 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:43.309 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:43.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 1 Processor -- start 2024-09-16T14:46:43.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 1 -- start start 2024-09-16T14:46:43.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 10 monclient: init 2024-09-16T14:46:43.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:43.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:43.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 10 monclient(hunting): picked mon.noname-c con 0x7f77680a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:43.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 10 monclient(hunting): picked mon.noname-a con 0x7f77680a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:43.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 10 monclient(hunting): picked mon.noname-b con 0x7f776813b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:43.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776811ff20 con 0x7f77680a25f0 2024-09-16T14:46:43.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680a59e0 con 0x7f776813b430 2024-09-16T14:46:43.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680a5350 con 0x7f77680a1b00 2024-09-16T14:46:43.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:43.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 10 monclient(hunting): authenticate will time out at 1185.418528s 2024-09-16T14:46:43.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776cbab640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f77680a1b00 legacy=0x7f77681c9810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:43.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776d3ac640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f776813b430 0x7f77681d1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35506/0 (socket says 172.21.15.189:35506) 2024-09-16T14:46:43.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776d3ac640 1 -- 172.21.15.189:0/2264719531 learned_addr learned my addr 172.21.15.189:0/2264719531 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:43.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463486406 0 0) 0x7f77680a59e0 con 0x7f776813b430 2024-09-16T14:46:43.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:43.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:43.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:43.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient(hunting): my global_id is 94684 2024-09-16T14:46:43.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7754003530 con 0x7f776813b430 2024-09-16T14:46:43.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 76339001 0 0) 0x7f7754003530 con 0x7f776813b430 2024-09-16T14:46:43.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77680a59e0 con 0x7f776813b430 2024-09-16T14:46:43.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7760001900 con 0x7f776813b430 2024-09-16T14:46:43.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:43.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:43.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient(hunting): dump: 2024-09-16T14:46:43.314 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:43.315 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930372290 0 0) 0x7f776811ff20 con 0x7f77680a25f0 2024-09-16T14:46:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:43.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient(hunting): my global_id is 104331 2024-09-16T14:46:43.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7754003530 con 0x7f77680a25f0 2024-09-16T14:46:43.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 882828773 0 0) 0x7f77680a59e0 con 0x7f776813b430 2024-09-16T14:46:43.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 >> v1:172.21.15.26:6790/0 conn(0x7f77680a1b00 legacy=0x7f77681c9810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:43.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 >> v1:172.21.15.26:6789/0 conn(0x7f77680a25f0 legacy=0x7f77681ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:43.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient: _finish_hunting 0 2024-09-16T14:46:43.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 1 monclient: found mon.b 2024-09-16T14:46:43.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:43.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:43.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776811fc40 con 0x7f776813b430 2024-09-16T14:46:43.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient: _finish_auth 0 2024-09-16T14:46:43.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient: _check_auth_tickets 2024-09-16T14:46:43.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 5 monclient: authenticate success, global_id 94684 2024-09-16T14:46:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 10 monclient: _renew_subs 2024-09-16T14:46:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 1 -- 172.21.15.189:0/2264719531 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776811e510 con 0x7f776813b430 2024-09-16T14:46:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 10 monclient: _renew_subs 2024-09-16T14:46:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f776ee0f640 1 -- 172.21.15.189:0/2264719531 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f776813a600 con 0x7f776813b430 2024-09-16T14:46:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7760001cd0 con 0x7f776813b430 2024-09-16T14:46:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7760003af0 con 0x7f776813b430 2024-09-16T14:46:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f77657fa640 10 monclient: dump: 2024-09-16T14:46:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:43.318 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:43.318 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:43.318 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:43.318 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:43.318 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:43.318 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:43.318 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:43.318 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:43.318 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:43.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.299+0000 7f7766ffd640 4 set_mon_vals no callback set 2024-09-16T14:46:43.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.303+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7760003c50 con 0x7f776813b430 2024-09-16T14:46:43.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.303+0000 7f776ee0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:43.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.303+0000 7f776ee0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:43.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.303+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(94..94 src has 65..94) v4 ==== 5238+0+0 (unknown 3191735785 0 0) 0x7f77600438e0 con 0x7f776813b430 2024-09-16T14:46:43.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.303+0000 7f776ee0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:43.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.303+0000 7f776ee0f640 1 -- 172.21.15.189:0/2264719531 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7748005050 con 0x7f776813b430 2024-09-16T14:46:43.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.303+0000 7f776cbab640 10 monclient: get_auth_request con 0x7f7754036130 auth_method 0 2024-09-16T14:46:43.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.307+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7760012c70 con 0x7f776813b430 2024-09-16T14:46:43.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.307+0000 7f77657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:43.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.307+0000 7f77657fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:43.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.471+0000 7f776ee0f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T14:46:43.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.471+0000 7f776ee0f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T14:46:43.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.471+0000 7f776ee0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:43.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.471+0000 7f776ee0f640 1 -- 172.21.15.189:0/2264719531 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.7"} v 0) v1 -- 0x7f7748005300 con 0x7f776813b430 2024-09-16T14:46:43.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.715+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7"}]=0 removed item id 7 name 'osd.7' from crush map v95) v1 ==== 122+0+0 (unknown 3479235143 0 0) 0x7f77600147c0 con 0x7f776813b430 2024-09-16T14:46:43.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.715+0000 7f77657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T14:46:43.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.715+0000 7f77657fa640 10 monclient: _finish_command 2 = system:0 removed item id 7 name 'osd.7' from crush map 2024-09-16T14:46:43.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.819+0000 7f776ee0f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T14:46:43.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.819+0000 7f776ee0f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T14:46:43.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.819+0000 7f776ee0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:43.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.819+0000 7f776ee0f640 1 -- 172.21.15.189:0/2264719531 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.7"} v 0) v1 -- 0x7f77480023c0 con 0x7f776813b430 2024-09-16T14:46:43.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.819+0000 7f77657fa640 1 -- 172.21.15.189:0/2264719531 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7"}]=0 device 'osd.7' does not appear in the crush map v95) v1 ==== 124+0+0 (unknown 157926631 0 0) 0x7f77600147c0 con 0x7f776813b430 2024-09-16T14:46:43.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.819+0000 7f77657fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T14:46:43.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.819+0000 7f77657fa640 10 monclient: _finish_command 3 = system:0 device 'osd.7' does not appear in the crush map 2024-09-16T14:46:43.823 INFO:tasks.workunit.client.0.smithi189.stderr:device 'osd.7' does not appear in the crush map 2024-09-16T14:46:43.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.819+0000 7f776ee0f640 1 -- 172.21.15.189:0/2264719531 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7754036130 legacy=0x7f77540385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:43.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.819+0000 7f776ee0f640 10 monclient: shutdown 2024-09-16T14:46:43.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.819+0000 7f776ee0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:43.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.819+0000 7f776ee0f640 1 -- 172.21.15.189:0/2264719531 >> v1:172.21.15.189:6789/0 conn(0x7f776813b430 legacy=0x7f77681d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:43.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.819+0000 7f776ee0f640 1 -- 172.21.15.189:0/2264719531 shutdown_connections 2024-09-16T14:46:43.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.819+0000 7f776ee0f640 1 -- 172.21.15.189:0/2264719531 >> 172.21.15.189:0/2264719531 conn(0x7f77680a7dd0 msgr2=0x7f776806b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:43.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.819+0000 7f776ee0f640 1 -- 172.21.15.189:0/2264719531 shutdown_connections 2024-09-16T14:46:43.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.819+0000 7f776ee0f640 1 -- 172.21.15.189:0/2264719531 wait complete. 2024-09-16T14:46:43.839 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm osd.9 2024-09-16T14:46:43.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.947+0000 7ffac8c20640 10 monclient: get_monmap_and_config 2024-09-16T14:46:43.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.947+0000 7ffac8c20640 10 monclient: build_initial_monmap 2024-09-16T14:46:43.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.947+0000 7ffac8c20640 10 monclient: monmap: 2024-09-16T14:46:43.952 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:43.952 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:43.952 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:43.952390+0000 2024-09-16T14:46:43.953 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:43.952390+0000 2024-09-16T14:46:43.953 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:43.953 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:43.953 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:43.953 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:43.953 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:43.953 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:43.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.947+0000 7ffac8c20640 1 Processor -- start 2024-09-16T14:46:43.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.947+0000 7ffac8c20640 1 -- start start 2024-09-16T14:46:43.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.947+0000 7ffac8c20640 10 monclient: init 2024-09-16T14:46:43.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.947+0000 7ffac8c20640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:43.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.947+0000 7ffac8c20640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:46:43.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.947+0000 7ffac8c20640 10 monclient(hunting): picked mon.noname-a con 0x7ffac409df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:43.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.947+0000 7ffac8c20640 10 monclient(hunting): picked mon.noname-c con 0x7ffac409e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:43.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.947+0000 7ffac8c20640 10 monclient(hunting): picked mon.noname-b con 0x7ffac409e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:43.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.947+0000 7ffac8c20640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac409ee20 con 0x7ffac409df10 2024-09-16T14:46:43.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac409f100 con 0x7ffac409e2f0 2024-09-16T14:46:43.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac409f3e0 con 0x7ffac409e740 2024-09-16T14:46:43.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:43.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 10 monclient(hunting): authenticate will time out at 1186.068056s 2024-09-16T14:46:43.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac2d9d640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7ffac409e740 legacy=0x7ffac413b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:43.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac1d9b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffac409e2f0 0x7ffac41378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35508/0 (socket says 172.21.15.189:35508) 2024-09-16T14:46:43.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac1d9b640 1 -- 172.21.15.189:0/480471871 learned_addr learned my addr 172.21.15.189:0/480471871 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:43.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 1 -- 172.21.15.189:0/480471871 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183953893 0 0) 0x7ffac409f100 con 0x7ffac409e2f0 2024-09-16T14:46:43.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 10 monclient(hunting): _negotiate 2024-09-16T14:46:43.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:43.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:43.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 10 monclient(hunting): my global_id is 94687 2024-09-16T14:46:43.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 1 -- 172.21.15.189:0/480471871 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaac003530 con 0x7ffac409e2f0 2024-09-16T14:46:43.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 1 -- 172.21.15.189:0/480471871 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3328530897 0 0) 0x7ffaac003530 con 0x7ffac409e2f0 2024-09-16T14:46:43.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 1 -- 172.21.15.189:0/480471871 >> v1:172.21.15.26:6790/0 conn(0x7ffac409e740 legacy=0x7ffac413b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:43.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 1 -- 172.21.15.189:0/480471871 >> v1:172.21.15.26:6789/0 conn(0x7ffac409df10 legacy=0x7ffac409d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:43.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 10 monclient: _finish_hunting 0 2024-09-16T14:46:43.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 1 monclient: found mon.noname-c 2024-09-16T14:46:43.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:43.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:43.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 1 -- 172.21.15.189:0/480471871 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac409c690 con 0x7ffac409e2f0 2024-09-16T14:46:43.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 10 monclient: _finish_auth 0 2024-09-16T14:46:43.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 10 monclient: _check_auth_tickets 2024-09-16T14:46:43.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:43.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 1 -- 172.21.15.189:0/480471871 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffab4001620 con 0x7ffac409e2f0 2024-09-16T14:46:43.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 5 monclient: authenticate success, global_id 94687 2024-09-16T14:46:43.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:43.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:43.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:43.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 10 monclient: dump: 2024-09-16T14:46:43.964 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:43.965 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:43.965 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:43.965 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:43.965 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:43.965 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:43.965 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:43.966 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:43.966 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:43.966 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:43.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:43.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 1 -- 172.21.15.189:0/480471871 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffab40018c0 con 0x7ffac409e2f0 2024-09-16T14:46:43.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:43.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 1 -- 172.21.15.189:0/480471871 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffab4001c90 con 0x7ffac409e2f0 2024-09-16T14:46:43.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:43.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 4 set_mon_vals no callback set 2024-09-16T14:46:43.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:43.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:43.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac159a640 10 monclient: dump: 2024-09-16T14:46:43.967 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:43.967 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:43.967 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:43.967 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:43.967 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:43.967 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:43.967 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:43.967 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:43.967 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:43.967 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:43.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 10 monclient: shutdown 2024-09-16T14:46:43.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:43.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 1 -- 172.21.15.189:0/480471871 >> v1:172.21.15.189:6789/0 conn(0x7ffac409e2f0 legacy=0x7ffac41378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:43.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 1 -- 172.21.15.189:0/480471871 shutdown_connections 2024-09-16T14:46:43.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 1 -- 172.21.15.189:0/480471871 >> 172.21.15.189:0/480471871 conn(0x7ffac40a3320 msgr2=0x7ffac409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:43.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 1 -- 172.21.15.189:0/480471871 shutdown_connections 2024-09-16T14:46:43.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 1 -- 172.21.15.189:0/480471871 wait complete. 2024-09-16T14:46:43.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 10 monclient: build_initial_monmap 2024-09-16T14:46:43.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 10 monclient: monmap: 2024-09-16T14:46:43.968 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:43.969 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:43.969 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:43.969 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:43.969 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:43.969 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:43.970 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:43.970 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:43.970 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:43.970 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:43.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 1 Processor -- start 2024-09-16T14:46:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 1 -- start start 2024-09-16T14:46:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 10 monclient: init 2024-09-16T14:46:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 10 monclient(hunting): picked mon.noname-b con 0x7ffac409df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 10 monclient(hunting): picked mon.noname-c con 0x7ffac409e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 10 monclient(hunting): picked mon.noname-a con 0x7ffac409e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac406cc70 con 0x7ffac409e740 2024-09-16T14:46:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac411d170 con 0x7ffac409df10 2024-09-16T14:46:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac411dff0 con 0x7ffac409e2f0 2024-09-16T14:46:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac8c20640 10 monclient(hunting): authenticate will time out at 1186.071670s 2024-09-16T14:46:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac259c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffac409df10 0x7ffac411c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35514/0 (socket says 172.21.15.189:35514) 2024-09-16T14:46:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac259c640 1 -- 172.21.15.189:0/2870224049 learned_addr learned my addr 172.21.15.189:0/2870224049 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.951+0000 7ffac1d9b640 1 -- 172.21.15.189:0/2870224049 >> v1:172.21.15.26:6790/0 conn(0x7ffac409e2f0 legacy=0x7ffac411ca60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114815158 0 0) 0x7ffac411d170 con 0x7ffac409df10 2024-09-16T14:46:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient(hunting): my global_id is 94690 2024-09-16T14:46:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa4003530 con 0x7ffac409df10 2024-09-16T14:46:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1994665064 0 0) 0x7ffaa4003530 con 0x7ffac409df10 2024-09-16T14:46:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffac411d170 con 0x7ffac409df10 2024-09-16T14:46:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffab80019f0 con 0x7ffac409df10 2024-09-16T14:46:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient(hunting): dump: 2024-09-16T14:46:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:43.974 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539962463 0 0) 0x7ffac406cc70 con 0x7ffac409e740 2024-09-16T14:46:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient(hunting): my global_id is 104337 2024-09-16T14:46:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa4003530 con 0x7ffac409e740 2024-09-16T14:46:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2824020972 0 0) 0x7ffac411d170 con 0x7ffac409df10 2024-09-16T14:46:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 >> v1:172.21.15.26:6790/0 conn(0x7ffac409e2f0 legacy=0x7ffac411ca60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 >> v1:172.21.15.26:6789/0 conn(0x7ffac409e740 legacy=0x7ffac41cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 1 monclient: found mon.b 2024-09-16T14:46:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac406b380 con 0x7ffac409df10 2024-09-16T14:46:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:46:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffac8c20640 5 monclient: authenticate success, global_id 94690 2024-09-16T14:46:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffac8c20640 10 monclient: _renew_subs 2024-09-16T14:46:43.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffac8c20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:43.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffac8c20640 1 -- 172.21.15.189:0/2870224049 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac4120070 con 0x7ffac409df10 2024-09-16T14:46:43.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffac8c20640 10 monclient: _renew_subs 2024-09-16T14:46:43.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffac8c20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:43.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffac8c20640 1 -- 172.21.15.189:0/2870224049 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac411fab0 con 0x7ffac409df10 2024-09-16T14:46:43.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffab8003950 con 0x7ffac409df10 2024-09-16T14:46:43.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:43.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffab8003c60 con 0x7ffac409df10 2024-09-16T14:46:43.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffac0d99640 4 set_mon_vals no callback set 2024-09-16T14:46:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 10 monclient: dump: 2024-09-16T14:46:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:43.978 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:43.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ffab8003dc0 con 0x7ffac409df10 2024-09-16T14:46:43.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffac8c20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:43.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffac8c20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:43.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffac8c20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:43.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffac8c20640 1 -- 172.21.15.189:0/2870224049 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa8005050 con 0x7ffac409df10 2024-09-16T14:46:43.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(95..95 src has 65..95) v4 ==== 5130+0+0 (unknown 3745819097 0 0) 0x7ffab80438f0 con 0x7ffac409df10 2024-09-16T14:46:43.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.955+0000 7ffac1d9b640 10 monclient: get_auth_request con 0x7ffaa40360e0 auth_method 0 2024-09-16T14:46:43.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.959+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ffab8014650 con 0x7ffac409df10 2024-09-16T14:46:43.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.959+0000 7ffab2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:43.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:43.959+0000 7ffab2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:44.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.107+0000 7ffac8c20640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T14:46:44.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.107+0000 7ffac8c20640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T14:46:44.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.107+0000 7ffac8c20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:44.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.107+0000 7ffac8c20640 1 -- 172.21.15.189:0/2870224049 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.9"} v 0) v1 -- 0x7ffaa8005300 con 0x7ffac409df10 2024-09-16T14:46:44.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.107+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9"}]=0 device 'osd.9' does not appear in the crush map v95) v1 ==== 124+0+0 (unknown 1574543391 0 0) 0x7ffab8012cf0 con 0x7ffac409df10 2024-09-16T14:46:44.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.107+0000 7ffab2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T14:46:44.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.107+0000 7ffab2ffd640 10 monclient: _finish_command 2 = system:0 device 'osd.9' does not appear in the crush map 2024-09-16T14:46:44.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.199+0000 7ffac8c20640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T14:46:44.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.199+0000 7ffac8c20640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T14:46:44.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.199+0000 7ffac8c20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:44.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.199+0000 7ffac8c20640 1 -- 172.21.15.189:0/2870224049 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.9"} v 0) v1 -- 0x7ffaa80023c0 con 0x7ffac409df10 2024-09-16T14:46:44.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.199+0000 7ffab2ffd640 1 -- 172.21.15.189:0/2870224049 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9"}]=0 device 'osd.9' does not appear in the crush map v95) v1 ==== 124+0+0 (unknown 1574543391 0 0) 0x7ffab8013360 con 0x7ffac409df10 2024-09-16T14:46:44.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.199+0000 7ffab2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T14:46:44.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.199+0000 7ffab2ffd640 10 monclient: _finish_command 3 = system:0 device 'osd.9' does not appear in the crush map 2024-09-16T14:46:44.204 INFO:tasks.workunit.client.0.smithi189.stderr:device 'osd.9' does not appear in the crush map 2024-09-16T14:46:44.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.199+0000 7ffac8c20640 1 -- 172.21.15.189:0/2870224049 >> v1:172.21.15.189:6812/3196161582 conn(0x7ffaa40360e0 legacy=0x7ffaa40385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:44.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.199+0000 7ffac8c20640 10 monclient: shutdown 2024-09-16T14:46:44.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.199+0000 7ffac8c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:44.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.199+0000 7ffac8c20640 1 -- 172.21.15.189:0/2870224049 >> v1:172.21.15.189:6789/0 conn(0x7ffac409df10 legacy=0x7ffac411c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:44.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.203+0000 7ffac8c20640 1 -- 172.21.15.189:0/2870224049 shutdown_connections 2024-09-16T14:46:44.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.203+0000 7ffac8c20640 1 -- 172.21.15.189:0/2870224049 >> 172.21.15.189:0/2870224049 conn(0x7ffac40a3320 msgr2=0x7ffac409d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:44.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.203+0000 7ffac8c20640 1 -- 172.21.15.189:0/2870224049 shutdown_connections 2024-09-16T14:46:44.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.203+0000 7ffac8c20640 1 -- 172.21.15.189:0/2870224049 wait complete. 2024-09-16T14:46:44.219 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm foo 2024-09-16T14:46:44.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9f15a640 10 monclient: get_monmap_and_config 2024-09-16T14:46:44.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9f15a640 10 monclient: build_initial_monmap 2024-09-16T14:46:44.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9f15a640 10 monclient: monmap: 2024-09-16T14:46:44.335 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:44.335 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:44.336 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:44.335336+0000 2024-09-16T14:46:44.336 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:44.335336+0000 2024-09-16T14:46:44.336 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:44.336 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:44.336 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:44.336 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:44.336 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:44.336 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:44.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9f15a640 1 Processor -- start 2024-09-16T14:46:44.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9f15a640 1 -- start start 2024-09-16T14:46:44.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9f15a640 10 monclient: init 2024-09-16T14:46:44.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9f15a640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:44.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9f15a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:44.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9f15a640 10 monclient(hunting): picked mon.noname-c con 0x7fea980993f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:44.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9f15a640 10 monclient(hunting): picked mon.noname-b con 0x7fea98131b70 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:44.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9f15a640 10 monclient(hunting): picked mon.noname-a con 0x7fea981380d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:44.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9f15a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea98073ba0 con 0x7fea981380d0 2024-09-16T14:46:44.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9f15a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea98099ee0 con 0x7fea980993f0 2024-09-16T14:46:44.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9f15a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea9809cc80 con 0x7fea98131b70 2024-09-16T14:46:44.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9f15a640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:44.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9f15a640 10 monclient(hunting): authenticate will time out at 1186.451737s 2024-09-16T14:46:44.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9cef6640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fea980993f0 0x7fea980997d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35524/0 (socket says 172.21.15.189:35524) 2024-09-16T14:46:44.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea9cef6640 1 -- 172.21.15.189:0/1168970664 learned_addr learned my addr 172.21.15.189:0/1168970664 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:44.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.331+0000 7fea97fff640 1 -- 172.21.15.189:0/1168970664 >> v1:172.21.15.26:6790/0 conn(0x7fea98131b70 legacy=0x7fea981369b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:44.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 1 -- 172.21.15.189:0/1168970664 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825235392 0 0) 0x7fea98099ee0 con 0x7fea980993f0 2024-09-16T14:46:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 10 monclient(hunting): my global_id is 94693 2024-09-16T14:46:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 1 -- 172.21.15.189:0/1168970664 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea88005740 con 0x7fea980993f0 2024-09-16T14:46:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 1 -- 172.21.15.189:0/1168970664 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 579827245 0 0) 0x7fea88005740 con 0x7fea980993f0 2024-09-16T14:46:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 1 -- 172.21.15.189:0/1168970664 >> v1:172.21.15.26:6790/0 conn(0x7fea98131b70 legacy=0x7fea981369b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 1 -- 172.21.15.189:0/1168970664 >> v1:172.21.15.26:6789/0 conn(0x7fea981380d0 legacy=0x7fea9813a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 1 monclient: found mon.noname-c 2024-09-16T14:46:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 1 -- 172.21.15.189:0/1168970664 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea9809cf60 con 0x7fea980993f0 2024-09-16T14:46:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 5 monclient: authenticate success, global_id 94693 2024-09-16T14:46:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 1 -- 172.21.15.189:0/1168970664 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fea84001620 con 0x7fea980993f0 2024-09-16T14:46:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 10 monclient: dump: 2024-09-16T14:46:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:44.347 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 1 -- 172.21.15.189:0/1168970664 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea84001940 con 0x7fea980993f0 2024-09-16T14:46:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 4 set_mon_vals no callback set 2024-09-16T14:46:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 1 -- 172.21.15.189:0/1168970664 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fea84001cd0 con 0x7fea980993f0 2024-09-16T14:46:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea977fe640 10 monclient: dump: 2024-09-16T14:46:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:44.349 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 10 monclient: shutdown 2024-09-16T14:46:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 1 -- 172.21.15.189:0/1168970664 >> v1:172.21.15.189:6789/0 conn(0x7fea980993f0 legacy=0x7fea980997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 1 -- 172.21.15.189:0/1168970664 shutdown_connections 2024-09-16T14:46:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 1 -- 172.21.15.189:0/1168970664 >> 172.21.15.189:0/1168970664 conn(0x7fea9809fed0 msgr2=0x7fea980a02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 1 -- 172.21.15.189:0/1168970664 shutdown_connections 2024-09-16T14:46:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 1 -- 172.21.15.189:0/1168970664 wait complete. 2024-09-16T14:46:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 10 monclient: build_initial_monmap 2024-09-16T14:46:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 10 monclient: monmap: 2024-09-16T14:46:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:44.351 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:44.351 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:44.351 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:44.351 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:44.351 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:44.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 1 Processor -- start 2024-09-16T14:46:44.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 1 -- start start 2024-09-16T14:46:44.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 10 monclient: init 2024-09-16T14:46:44.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:44.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:44.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 10 monclient(hunting): picked mon.noname-c con 0x7fea980993f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:44.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 10 monclient(hunting): picked mon.noname-a con 0x7fea98131b70 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:44.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 10 monclient(hunting): picked mon.noname-b con 0x7fea981380d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:44.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea98137af0 con 0x7fea98131b70 2024-09-16T14:46:44.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea9809a5c0 con 0x7fea981380d0 2024-09-16T14:46:44.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea98129cd0 con 0x7fea980993f0 2024-09-16T14:46:44.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:44.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 10 monclient(hunting): authenticate will time out at 1186.454629s 2024-09-16T14:46:44.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9cef6640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fea980993f0 legacy=0x7fea980704f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:44.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9d6f7640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fea981380d0 0x7fea981cd290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35530/0 (socket says 172.21.15.189:35530) 2024-09-16T14:46:44.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9d6f7640 1 -- 172.21.15.189:0/2756597071 learned_addr learned my addr 172.21.15.189:0/2756597071 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:44.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387063269 0 0) 0x7fea9809a5c0 con 0x7fea981380d0 2024-09-16T14:46:44.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:44.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:44.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:44.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient(hunting): my global_id is 94696 2024-09-16T14:46:44.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea7c003530 con 0x7fea981380d0 2024-09-16T14:46:44.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 687588095 0 0) 0x7fea7c003530 con 0x7fea981380d0 2024-09-16T14:46:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea9809a5c0 con 0x7fea981380d0 2024-09-16T14:46:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fea90001c50 con 0x7fea981380d0 2024-09-16T14:46:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient(hunting): dump: 2024-09-16T14:46:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:44.355 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587356500 0 0) 0x7fea98137af0 con 0x7fea98131b70 2024-09-16T14:46:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient(hunting): my global_id is 104343 2024-09-16T14:46:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea7c003530 con 0x7fea98131b70 2024-09-16T14:46:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3583629761 0 0) 0x7fea9809a5c0 con 0x7fea981380d0 2024-09-16T14:46:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 >> v1:172.21.15.26:6790/0 conn(0x7fea980993f0 legacy=0x7fea980704f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 >> v1:172.21.15.26:6789/0 conn(0x7fea98131b70 legacy=0x7fea98070c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient: _finish_hunting 0 2024-09-16T14:46:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 1 monclient: found mon.b 2024-09-16T14:46:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea98129430 con 0x7fea981380d0 2024-09-16T14:46:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient: _finish_auth 0 2024-09-16T14:46:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient: _check_auth_tickets 2024-09-16T14:46:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 5 monclient: authenticate success, global_id 94696 2024-09-16T14:46:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 10 monclient: _renew_subs 2024-09-16T14:46:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 1 -- 172.21.15.189:0/2756597071 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea98129710 con 0x7fea981380d0 2024-09-16T14:46:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 10 monclient: _renew_subs 2024-09-16T14:46:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea9f15a640 1 -- 172.21.15.189:0/2756597071 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea98071b40 con 0x7fea981380d0 2024-09-16T14:46:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea900039e0 con 0x7fea981380d0 2024-09-16T14:46:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fea90003cf0 con 0x7fea981380d0 2024-09-16T14:46:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea96ffd640 4 set_mon_vals no callback set 2024-09-16T14:46:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.335+0000 7fea957fa640 10 monclient: dump: 2024-09-16T14:46:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:44.358 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.339+0000 7fea9f15a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.339+0000 7fea9f15a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.339+0000 7fea9f15a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.339+0000 7fea9f15a640 1 -- 172.21.15.189:0/2756597071 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea78005050 con 0x7fea981380d0 2024-09-16T14:46:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.339+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fea90003e50 con 0x7fea981380d0 2024-09-16T14:46:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.339+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(95..95 src has 65..95) v4 ==== 5130+0+0 (unknown 3745819097 0 0) 0x7fea90043ab0 con 0x7fea981380d0 2024-09-16T14:46:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.339+0000 7fea9cef6640 10 monclient: get_auth_request con 0x7fea7c036180 auth_method 0 2024-09-16T14:46:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.343+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fea90014a40 con 0x7fea981380d0 2024-09-16T14:46:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.343+0000 7fea957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.343+0000 7fea957fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:44.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.503+0000 7fea9f15a640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:44.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.503+0000 7fea9f15a640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:44.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.503+0000 7fea9f15a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:44.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.503+0000 7fea9f15a640 1 -- 172.21.15.189:0/2756597071 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fea78005300 con 0x7fea981380d0 2024-09-16T14:46:44.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.591+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 removed item id -10 name 'foo' from crush map v96) v1 ==== 120+0+0 (unknown 3939064113 0 0) 0x7fea90012eb0 con 0x7fea981380d0 2024-09-16T14:46:44.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.591+0000 7fea957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:44.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.591+0000 7fea957fa640 10 monclient: _finish_command 2 = system:0 removed item id -10 name 'foo' from crush map 2024-09-16T14:46:44.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.691+0000 7fea9f15a640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:44.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.691+0000 7fea9f15a640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:44.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.691+0000 7fea9f15a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:44.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.691+0000 7fea9f15a640 1 -- 172.21.15.189:0/2756597071 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fea780055a0 con 0x7fea981380d0 2024-09-16T14:46:44.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.691+0000 7fea957fa640 1 -- 172.21.15.189:0/2756597071 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 device 'foo' does not appear in the crush map v96) v1 ==== 120+0+0 (unknown 3419787546 0 0) 0x7fea90014630 con 0x7fea981380d0 2024-09-16T14:46:44.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.691+0000 7fea957fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T14:46:44.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.691+0000 7fea957fa640 10 monclient: _finish_command 3 = system:0 device 'foo' does not appear in the crush map 2024-09-16T14:46:44.699 INFO:tasks.workunit.client.0.smithi189.stderr:device 'foo' does not appear in the crush map 2024-09-16T14:46:44.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.695+0000 7fea9f15a640 1 -- 172.21.15.189:0/2756597071 >> v1:172.21.15.189:6812/3196161582 conn(0x7fea7c036180 legacy=0x7fea7c038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:44.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.695+0000 7fea9f15a640 10 monclient: shutdown 2024-09-16T14:46:44.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.695+0000 7fea9f15a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:44.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.695+0000 7fea9f15a640 1 -- 172.21.15.189:0/2756597071 >> v1:172.21.15.189:6789/0 conn(0x7fea981380d0 legacy=0x7fea981cd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:44.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.695+0000 7fea9f15a640 1 -- 172.21.15.189:0/2756597071 shutdown_connections 2024-09-16T14:46:44.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.695+0000 7fea9f15a640 1 -- 172.21.15.189:0/2756597071 >> 172.21.15.189:0/2756597071 conn(0x7fea9809fed0 msgr2=0x7fea980a12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:44.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.695+0000 7fea9f15a640 1 -- 172.21.15.189:0/2756597071 shutdown_connections 2024-09-16T14:46:44.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.695+0000 7fea9f15a640 1 -- 172.21.15.189:0/2756597071 wait complete. 2024-09-16T14:46:44.713 INFO:tasks.workunit.client.0.smithi189.stderr:++ ceph osd create 2024-09-16T14:46:44.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f42043d1640 10 monclient: get_monmap_and_config 2024-09-16T14:46:44.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f42043d1640 10 monclient: build_initial_monmap 2024-09-16T14:46:44.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f42043d1640 10 monclient: monmap: 2024-09-16T14:46:44.824 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:44.824 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:44.824 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:44.823868+0000 2024-09-16T14:46:44.824 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:44.823868+0000 2024-09-16T14:46:44.824 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:44.824 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:44.824 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:44.824 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:44.825 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:44.825 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:44.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f42043d1640 1 Processor -- start 2024-09-16T14:46:44.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f42043d1640 1 -- start start 2024-09-16T14:46:44.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f42043d1640 10 monclient: init 2024-09-16T14:46:44.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f42043d1640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:44.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f42043d1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:44.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f42043d1640 10 monclient(hunting): picked mon.noname-c con 0x7f41fc09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:44.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f42043d1640 10 monclient(hunting): picked mon.noname-b con 0x7f41fc09e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:44.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f42043d1640 10 monclient(hunting): picked mon.noname-a con 0x7f41fc09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:44.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f42043d1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc09ee20 con 0x7f41fc09e740 2024-09-16T14:46:44.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f42043d1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc09f100 con 0x7f41fc09df10 2024-09-16T14:46:44.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f42043d1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc09f3e0 con 0x7f41fc09e2f0 2024-09-16T14:46:44.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f42043d1640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:44.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f42043d1640 10 monclient(hunting): authenticate will time out at 1186.939629s 2024-09-16T14:46:44.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f420196c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f41fc09e2f0 legacy=0x7f41fc1378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:44.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f420216d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f41fc09df10 0x7f41fc09d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35536/0 (socket says 172.21.15.189:35536) 2024-09-16T14:46:44.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.819+0000 7f420216d640 1 -- 172.21.15.189:0/4072740252 learned_addr learned my addr 172.21.15.189:0/4072740252 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:44.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 1 -- 172.21.15.189:0/4072740252 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685751646 0 0) 0x7f41fc09f100 con 0x7f41fc09df10 2024-09-16T14:46:44.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient(hunting): _negotiate 2024-09-16T14:46:44.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:44.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:44.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient(hunting): my global_id is 94699 2024-09-16T14:46:44.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 1 -- 172.21.15.189:0/4072740252 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41e8003530 con 0x7f41fc09df10 2024-09-16T14:46:44.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 1 -- 172.21.15.189:0/4072740252 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897977826 0 0) 0x7f41fc09ee20 con 0x7f41fc09e740 2024-09-16T14:46:44.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient(hunting): _negotiate 2024-09-16T14:46:44.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:44.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:44.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient(hunting): my global_id is 104346 2024-09-16T14:46:44.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 1 -- 172.21.15.189:0/4072740252 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41fc09f100 con 0x7f41fc09e740 2024-09-16T14:46:44.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 1 -- 172.21.15.189:0/4072740252 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 979484890 0 0) 0x7f41e8003530 con 0x7f41fc09df10 2024-09-16T14:46:44.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 1 -- 172.21.15.189:0/4072740252 >> v1:172.21.15.26:6790/0 conn(0x7f41fc09e2f0 legacy=0x7f41fc1378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:46:44.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 1 -- 172.21.15.189:0/4072740252 >> v1:172.21.15.26:6789/0 conn(0x7f41fc09e740 legacy=0x7f41fc13b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:44.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient: _finish_hunting 0 2024-09-16T14:46:44.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 1 monclient: found mon.noname-c 2024-09-16T14:46:44.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:44.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:44.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 1 -- 172.21.15.189:0/4072740252 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc09c690 con 0x7f41fc09df10 2024-09-16T14:46:44.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient: _finish_auth 0 2024-09-16T14:46:44.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient: _check_auth_tickets 2024-09-16T14:46:44.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:44.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 1 -- 172.21.15.189:0/4072740252 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f41ec001620 con 0x7f41fc09df10 2024-09-16T14:46:44.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:44.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:44.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient: dump: 2024-09-16T14:46:44.835 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:44.835 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:44.835 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:44.835 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:44.835 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:44.835 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:44.835 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:44.835 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:44.835 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:44.835 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:44.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 5 monclient: authenticate success, global_id 94699 2024-09-16T14:46:44.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:44.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 1 -- 172.21.15.189:0/4072740252 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41ec001900 con 0x7f41fc09df10 2024-09-16T14:46:44.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:44.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:44.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 4 set_mon_vals no callback set 2024-09-16T14:46:44.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 1 -- 172.21.15.189:0/4072740252 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f41ec001c90 con 0x7f41fc09df10 2024-09-16T14:46:44.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:44.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:44.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420116b640 10 monclient: dump: 2024-09-16T14:46:44.836 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:44.836 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:44.836 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:44.837 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:44.837 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:44.837 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:44.837 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:44.837 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:44.837 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:44.837 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:44.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 10 monclient: shutdown 2024-09-16T14:46:44.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:44.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 1 -- 172.21.15.189:0/4072740252 >> v1:172.21.15.189:6789/0 conn(0x7f41fc09df10 legacy=0x7f41fc09d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:44.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 1 -- 172.21.15.189:0/4072740252 shutdown_connections 2024-09-16T14:46:44.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 1 -- 172.21.15.189:0/4072740252 >> 172.21.15.189:0/4072740252 conn(0x7f41fc0a3320 msgr2=0x7f41fc09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:44.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 1 -- 172.21.15.189:0/4072740252 shutdown_connections 2024-09-16T14:46:44.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 1 -- 172.21.15.189:0/4072740252 wait complete. 2024-09-16T14:46:44.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 10 monclient: build_initial_monmap 2024-09-16T14:46:44.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 10 monclient: monmap: 2024-09-16T14:46:44.838 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:44.838 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:44.838 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:44.838 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:44.838 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:44.838 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:44.838 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:44.838 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:44.838 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:44.838 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:44.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 1 Processor -- start 2024-09-16T14:46:44.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 1 -- start start 2024-09-16T14:46:44.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 10 monclient: init 2024-09-16T14:46:44.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:44.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:44.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 10 monclient(hunting): picked mon.noname-b con 0x7f41fc09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:44.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 10 monclient(hunting): picked mon.noname-c con 0x7f41fc09e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:44.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 10 monclient(hunting): picked mon.noname-a con 0x7f41fc09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:44.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc06cdc0 con 0x7f41fc09e740 2024-09-16T14:46:44.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc06b380 con 0x7f41fc09df10 2024-09-16T14:46:44.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc121d50 con 0x7f41fc09e2f0 2024-09-16T14:46:44.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:44.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 10 monclient(hunting): authenticate will time out at 1186.942699s 2024-09-16T14:46:44.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420216d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f41fc09df10 0x7f41fc09a1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35546/0 (socket says 172.21.15.189:35546) 2024-09-16T14:46:44.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420216d640 1 -- 172.21.15.189:0/2136877205 learned_addr learned my addr 172.21.15.189:0/2136877205 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:44.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420196c640 1 -- 172.21.15.189:0/2136877205 >> v1:172.21.15.26:6790/0 conn(0x7f41fc09e2f0 legacy=0x7f41fc09a900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:46:44.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119340891 0 0) 0x7f41fc06b380 con 0x7f41fc09df10 2024-09-16T14:46:44.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:44.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:44.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:44.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient(hunting): my global_id is 94702 2024-09-16T14:46:44.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41e0003530 con 0x7f41fc09df10 2024-09-16T14:46:44.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1537081453 0 0) 0x7f41e0003530 con 0x7f41fc09df10 2024-09-16T14:46:44.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41fc06b380 con 0x7f41fc09df10 2024-09-16T14:46:44.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f41ec0015e0 con 0x7f41fc09df10 2024-09-16T14:46:44.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:44.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:44.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient(hunting): dump: 2024-09-16T14:46:44.841 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:44.841 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:44.841 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:44.841 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:44.841 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:44.841 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:44.841 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:44.841 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:44.842 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:44.842 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:44.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:44.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306608235 0 0) 0x7f41fc09f100 con 0x7f41fc09e740 2024-09-16T14:46:44.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:44.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:44.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:44.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient(hunting): my global_id is 104349 2024-09-16T14:46:44.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41e0003530 con 0x7f41fc09e740 2024-09-16T14:46:44.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3915252155 0 0) 0x7f41fc06b380 con 0x7f41fc09df10 2024-09-16T14:46:44.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 >> v1:172.21.15.26:6790/0 conn(0x7f41fc09e2f0 legacy=0x7f41fc09a900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:46:44.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 >> v1:172.21.15.26:6789/0 conn(0x7f41fc09e740 legacy=0x7f41fc09b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:44.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:44.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 1 monclient: found mon.b 2024-09-16T14:46:44.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:44.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:44.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc09bcb0 con 0x7f41fc09df10 2024-09-16T14:46:44.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:46:44.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:44.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:44.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 5 monclient: authenticate success, global_id 94702 2024-09-16T14:46:44.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 10 monclient: _renew_subs 2024-09-16T14:46:44.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:44.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 1 -- 172.21.15.189:0/2136877205 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41fc13c380 con 0x7f41fc09df10 2024-09-16T14:46:44.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 10 monclient: _renew_subs 2024-09-16T14:46:44.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:44.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f42043d1640 1 -- 172.21.15.189:0/2136877205 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41fc1d27b0 con 0x7f41fc09df10 2024-09-16T14:46:44.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41ec004c80 con 0x7f41fc09df10 2024-09-16T14:46:44.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:44.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f41ec005010 con 0x7f41fc09df10 2024-09-16T14:46:44.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:44.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:44.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f420096a640 4 set_mon_vals no callback set 2024-09-16T14:46:44.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.823+0000 7f41f2ffd640 10 monclient: dump: 2024-09-16T14:46:44.844 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:44.844 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:44.844 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:44.844 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:44.845 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:44.845 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:44.845 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:44.845 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:44.845 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:44.845 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:44.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.827+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f41ec0051f0 con 0x7f41fc09df10 2024-09-16T14:46:44.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.827+0000 7f42043d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:44.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.827+0000 7f42043d1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:44.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.827+0000 7f42043d1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:44.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.827+0000 7f42043d1640 1 -- 172.21.15.189:0/2136877205 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e4005050 con 0x7f41fc09df10 2024-09-16T14:46:44.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.827+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(96..96 src has 65..96) v4 ==== 4890+0+0 (unknown 769994835 0 0) 0x7f41ec043a30 con 0x7f41fc09df10 2024-09-16T14:46:44.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.827+0000 7f420196c640 10 monclient: get_auth_request con 0x7f41e00360e0 auth_method 0 2024-09-16T14:46:44.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.831+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f41ec014940 con 0x7f41fc09df10 2024-09-16T14:46:44.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.831+0000 7f41f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:44.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.831+0000 7f41f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:45.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.999+0000 7f42043d1640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T14:46:45.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.999+0000 7f42043d1640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T14:46:45.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.999+0000 7f42043d1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:45.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:44.999+0000 7f42043d1640 1 -- 172.21.15.189:0/2136877205 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f41e4005880 con 0x7f41fc09df10 2024-09-16T14:46:45.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:45.603+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v97) v1 ==== 58+0+3 (unknown 2995259560 0 520826459) 0x7f41ec012fe0 con 0x7f41fc09df10 2024-09-16T14:46:45.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:45.603+0000 7f41f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T14:46:45.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:45.603+0000 7f41f2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:45.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:45.711+0000 7f42043d1640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T14:46:45.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:45.711+0000 7f42043d1640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T14:46:45.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:45.711+0000 7f42043d1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:45.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:45.711+0000 7f42043d1640 1 -- 172.21.15.189:0/2136877205 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f41e4005370 con 0x7f41fc09df10 2024-09-16T14:46:46.069 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T14:46:46.069 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T14:46:46.069 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T14:46:46.070 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T14:46:46.073 INFO:tasks.ceph.mon.c:Started 2024-09-16T14:46:46.073 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T14:46:46.073 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T14:46:46.073 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T14:46:47.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:47.827+0000 7f41f1ffb640 10 monclient: tick 2024-09-16T14:46:47.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:47.827+0000 7f41f1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:47.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:47.827+0000 7f41f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:47.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:47.827+0000 7f41f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:48.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:48.823+0000 7f41f1ffb640 10 monclient: tick 2024-09-16T14:46:48.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:48.823+0000 7f41f1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:48.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:48.823+0000 7f41f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:49.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:49.823+0000 7f41f1ffb640 10 monclient: tick 2024-09-16T14:46:49.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:49.823+0000 7f41f1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:49.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:49.823+0000 7f41f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:50.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:50.823+0000 7f41f1ffb640 10 monclient: tick 2024-09-16T14:46:50.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:50.823+0000 7f41f1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:50.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:50.823+0000 7f41f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:51.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.495+0000 7f41f2ffd640 1 -- 172.21.15.189:0/2136877205 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v98) v1 ==== 58+0+3 (unknown 1761221957 0 212448812) 0x7f41ec014760 con 0x7f41fc09df10 2024-09-16T14:46:51.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.495+0000 7f41f2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T14:46:51.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.495+0000 7f41f2ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:46:51.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.499+0000 7f42043d1640 1 -- 172.21.15.189:0/2136877205 >> v1:172.21.15.189:6812/3196161582 conn(0x7f41e00360e0 legacy=0x7f41e00385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:51.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.499+0000 7f42043d1640 10 monclient: shutdown 2024-09-16T14:46:51.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.499+0000 7f42043d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:51.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.499+0000 7f42043d1640 1 -- 172.21.15.189:0/2136877205 >> v1:172.21.15.189:6789/0 conn(0x7f41fc09df10 legacy=0x7f41fc09a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:51.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.499+0000 7f42043d1640 1 -- 172.21.15.189:0/2136877205 shutdown_connections 2024-09-16T14:46:51.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.499+0000 7f42043d1640 1 -- 172.21.15.189:0/2136877205 >> 172.21.15.189:0/2136877205 conn(0x7f41fc0a3320 msgr2=0x7f41fc128640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:51.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.499+0000 7f42043d1640 1 -- 172.21.15.189:0/2136877205 shutdown_connections 2024-09-16T14:46:51.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.499+0000 7f42043d1640 1 -- 172.21.15.189:0/2136877205 wait complete. 2024-09-16T14:46:51.520 INFO:tasks.workunit.client.0.smithi189.stderr:+ o3=11 2024-09-16T14:46:51.521 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add 11 123 root=default 2024-09-16T14:46:51.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.635+0000 7f896e5c3640 10 monclient: get_monmap_and_config 2024-09-16T14:46:51.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.635+0000 7f896e5c3640 10 monclient: build_initial_monmap 2024-09-16T14:46:51.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 10 monclient: monmap: 2024-09-16T14:46:51.642 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:51.642 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:51.642 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:51.641774+0000 2024-09-16T14:46:51.642 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:51.641774+0000 2024-09-16T14:46:51.642 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:51.642 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:51.642 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:51.642 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:51.642 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:51.642 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:51.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 1 Processor -- start 2024-09-16T14:46:51.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 1 -- start start 2024-09-16T14:46:51.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 10 monclient: init 2024-09-16T14:46:51.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:51.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:51.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 10 monclient(hunting): picked mon.noname-a con 0x7f8968099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:51.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 10 monclient(hunting): picked mon.noname-b con 0x7f896809c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:51.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 10 monclient(hunting): picked mon.noname-c con 0x7f896809e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:51.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8968099e50 con 0x7f8968099a70 2024-09-16T14:46:51.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896809dbd0 con 0x7f896809e6d0 2024-09-16T14:46:51.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896809ee20 con 0x7f896809c550 2024-09-16T14:46:51.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:51.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 10 monclient(hunting): authenticate will time out at 1193.757559s 2024-09-16T14:46:51.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896cb60640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f896809e6d0 0x7f896813b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39994/0 (socket says 172.21.15.189:39994) 2024-09-16T14:46:51.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896cb60640 1 -- 172.21.15.189:0/1307541250 learned_addr learned my addr 172.21.15.189:0/1307541250 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:51.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 1 -- 172.21.15.189:0/1307541250 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71553281 0 0) 0x7f896809dbd0 con 0x7f896809e6d0 2024-09-16T14:46:51.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:46:51.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:51.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:51.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 10 monclient(hunting): my global_id is 94705 2024-09-16T14:46:51.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 1 -- 172.21.15.189:0/1307541250 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8954003530 con 0x7f896809e6d0 2024-09-16T14:46:51.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 1 -- 172.21.15.189:0/1307541250 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1135095575 0 0) 0x7f8954003530 con 0x7f896809e6d0 2024-09-16T14:46:51.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 1 -- 172.21.15.189:0/1307541250 >> v1:172.21.15.26:6790/0 conn(0x7f896809c550 legacy=0x7f89681379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:51.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 1 -- 172.21.15.189:0/1307541250 >> v1:172.21.15.26:6789/0 conn(0x7f8968099a70 legacy=0x7f896809dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:46:51.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:46:51.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 1 monclient: found mon.noname-c 2024-09-16T14:46:51.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:51.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:51.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 1 -- 172.21.15.189:0/1307541250 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896809f100 con 0x7f896809e6d0 2024-09-16T14:46:51.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 10 monclient: _finish_auth 0 2024-09-16T14:46:51.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:46:51.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:51.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 1 -- 172.21.15.189:0/1307541250 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f89600019a0 con 0x7f896809e6d0 2024-09-16T14:46:51.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:51.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:51.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 10 monclient: dump: 2024-09-16T14:46:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:51.652 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 5 monclient: authenticate success, global_id 94705 2024-09-16T14:46:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 1 -- 172.21.15.189:0/1307541250 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8960001bc0 con 0x7f896809e6d0 2024-09-16T14:46:51.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:51.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:51.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 4 set_mon_vals no callback set 2024-09-16T14:46:51.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 1 -- 172.21.15.189:0/1307541250 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8960001f50 con 0x7f896809e6d0 2024-09-16T14:46:51.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:51.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:51.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f8966ffd640 10 monclient: dump: 2024-09-16T14:46:51.653 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:51.653 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:51.653 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:51.653 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:51.653 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:51.653 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:51.653 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:51.654 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:51.654 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:51.654 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:51.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 10 monclient: shutdown 2024-09-16T14:46:51.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:51.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 1 -- 172.21.15.189:0/1307541250 >> v1:172.21.15.189:6789/0 conn(0x7f896809e6d0 legacy=0x7f896813b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:51.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 1 -- 172.21.15.189:0/1307541250 shutdown_connections 2024-09-16T14:46:51.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 1 -- 172.21.15.189:0/1307541250 >> 172.21.15.189:0/1307541250 conn(0x7f89680a3380 msgr2=0x7f896809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:51.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 1 -- 172.21.15.189:0/1307541250 shutdown_connections 2024-09-16T14:46:51.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 1 -- 172.21.15.189:0/1307541250 wait complete. 2024-09-16T14:46:51.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 10 monclient: build_initial_monmap 2024-09-16T14:46:51.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.639+0000 7f896e5c3640 10 monclient: monmap: 2024-09-16T14:46:51.654 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:51.655 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:51.655 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:51.655 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:51.655 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:51.655 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:51.655 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:51.655 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:51.655 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:51.655 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:51.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 1 Processor -- start 2024-09-16T14:46:51.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 1 -- start start 2024-09-16T14:46:51.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 10 monclient: init 2024-09-16T14:46:51.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:51.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:51.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 10 monclient(hunting): picked mon.noname-a con 0x7f8968099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:51.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 10 monclient(hunting): picked mon.noname-c con 0x7f896809c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:51.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 10 monclient(hunting): picked mon.noname-b con 0x7f896809e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:51.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896809cf20 con 0x7f8968099a70 2024-09-16T14:46:51.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896809a850 con 0x7f896809e6d0 2024-09-16T14:46:51.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896806b320 con 0x7f896809c550 2024-09-16T14:46:51.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:51.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 10 monclient(hunting): authenticate will time out at 1193.760768s 2024-09-16T14:46:51.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896cb60640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f896809e6d0 0x7f89681d1720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40010/0 (socket says 172.21.15.189:40010) 2024-09-16T14:46:51.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896cb60640 1 -- 172.21.15.189:0/3897998774 learned_addr learned my addr 172.21.15.189:0/3897998774 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:51.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 1 -- 172.21.15.189:0/3897998774 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953747708 0 0) 0x7f896809a850 con 0x7f896809e6d0 2024-09-16T14:46:51.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:46:51.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:51.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:51.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 10 monclient(hunting): my global_id is 94708 2024-09-16T14:46:51.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 1 -- 172.21.15.189:0/3897998774 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8948003530 con 0x7f896809e6d0 2024-09-16T14:46:51.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 1 -- 172.21.15.189:0/3897998774 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 299555060 0 0) 0x7f8948003530 con 0x7f896809e6d0 2024-09-16T14:46:51.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 1 -- 172.21.15.189:0/3897998774 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f896809a850 con 0x7f896809e6d0 2024-09-16T14:46:51.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 1 -- 172.21.15.189:0/3897998774 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8960001960 con 0x7f896809e6d0 2024-09-16T14:46:51.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:51.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:51.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 10 monclient(hunting): dump: 2024-09-16T14:46:51.660 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:51.660 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:51.660 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:51.660 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:51.660 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:51.660 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:51.660 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:51.660 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:51.660 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:51.661 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:51.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:51.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 1 -- 172.21.15.189:0/3897998774 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3386946458 0 0) 0x7f896809a850 con 0x7f896809e6d0 2024-09-16T14:46:51.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 1 -- 172.21.15.189:0/3897998774 >> v1:172.21.15.26:6790/0 conn(0x7f896809c550 legacy=0x7f8968138fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:51.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 1 -- 172.21.15.189:0/3897998774 >> v1:172.21.15.26:6789/0 conn(0x7f8968099a70 legacy=0x7f89681388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:51.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:46:51.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 1 monclient: found mon.b 2024-09-16T14:46:51.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:51.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:51.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 1 -- 172.21.15.189:0/3897998774 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896813a220 con 0x7f896809e6d0 2024-09-16T14:46:51.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 10 monclient: _finish_auth 0 2024-09-16T14:46:51.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:46:51.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:51.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 5 monclient: authenticate success, global_id 94708 2024-09-16T14:46:51.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 10 monclient: _renew_subs 2024-09-16T14:46:51.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:51.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 1 -- 172.21.15.189:0/3897998774 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f896813ab70 con 0x7f896809e6d0 2024-09-16T14:46:51.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 10 monclient: _renew_subs 2024-09-16T14:46:51.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:51.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f896e5c3640 1 -- 172.21.15.189:0/3897998774 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89681398c0 con 0x7f896809e6d0 2024-09-16T14:46:51.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 1 -- 172.21.15.189:0/3897998774 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8960004e20 con 0x7f896809e6d0 2024-09-16T14:46:51.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:51.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 1 -- 172.21.15.189:0/3897998774 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8960005130 con 0x7f896809e6d0 2024-09-16T14:46:51.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:51.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:51.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f89667fc640 4 set_mon_vals no callback set 2024-09-16T14:46:51.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.643+0000 7f8964ff9640 10 monclient: dump: 2024-09-16T14:46:51.663 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:51.663 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:51.663 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:51.663 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:51.663 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:51.663 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:51.663 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:51.663 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:51.663 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:51.663 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:51.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.647+0000 7f8964ff9640 1 -- 172.21.15.189:0/3897998774 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8960005290 con 0x7f896809e6d0 2024-09-16T14:46:51.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.647+0000 7f896e5c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:51.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.647+0000 7f896e5c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:51.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.647+0000 7f896e5c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:51.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.647+0000 7f896e5c3640 1 -- 172.21.15.189:0/3897998774 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f894c005050 con 0x7f896809e6d0 2024-09-16T14:46:51.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.647+0000 7f8964ff9640 1 -- 172.21.15.189:0/3897998774 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(98..98 src has 65..98) v4 ==== 5120+0+0 (unknown 2684827438 0 0) 0x7f8960043c50 con 0x7f896809e6d0 2024-09-16T14:46:51.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.647+0000 7f8967fff640 10 monclient: get_auth_request con 0x7f89480360f0 auth_method 0 2024-09-16T14:46:51.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.651+0000 7f8964ff9640 1 -- 172.21.15.189:0/3897998774 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8960014c50 con 0x7f896809e6d0 2024-09-16T14:46:51.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.651+0000 7f8964ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:51.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.651+0000 7f8964ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:51.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.803+0000 7f896e5c3640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T14:46:51.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.803+0000 7f896e5c3640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T14:46:51.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.803+0000 7f896e5c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:51.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:51.803+0000 7f896e5c3640 1 -- 172.21.15.189:0/3897998774 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]} v 0) v1 -- 0x7f894c005880 con 0x7f896809e6d0 2024-09-16T14:46:52.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.651+0000 7f8964ff9640 1 -- 172.21.15.189:0/3897998774 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}]=0 add item id 11 name 'osd.11' weight 123 at location {root=default} to crush map v99) v1 ==== 193+0+0 (unknown 1204463796 0 0) 0x7f89600130c0 con 0x7f896809e6d0 2024-09-16T14:46:52.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.651+0000 7f8964ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T14:46:52.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.651+0000 7f8964ff9640 10 monclient: _finish_command 2 = system:0 add item id 11 name 'osd.11' weight 123 at location {root=default} to crush map 2024-09-16T14:46:52.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.751+0000 7f896e5c3640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T14:46:52.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.751+0000 7f896e5c3640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T14:46:52.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.751+0000 7f896e5c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:52.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.751+0000 7f896e5c3640 1 -- 172.21.15.189:0/3897998774 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]} v 0) v1 -- 0x7f894c005300 con 0x7f896809e6d0 2024-09-16T14:46:52.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.751+0000 7f8964ff9640 1 -- 172.21.15.189:0/3897998774 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}]=0 set item id 11 name 'osd.11' weight 123 at location {root=default}: no change v99) v1 ==== 191+0+0 (unknown 1675519427 0 0) 0x7f8960014840 con 0x7f896809e6d0 2024-09-16T14:46:52.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.751+0000 7f8964ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T14:46:52.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.751+0000 7f8964ff9640 10 monclient: _finish_command 3 = system:0 set item id 11 name 'osd.11' weight 123 at location {root=default}: no change 2024-09-16T14:46:52.756 INFO:tasks.workunit.client.0.smithi189.stderr:set item id 11 name 'osd.11' weight 123 at location {root=default}: no change 2024-09-16T14:46:52.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.755+0000 7f896e5c3640 1 -- 172.21.15.189:0/3897998774 >> v1:172.21.15.189:6812/3196161582 conn(0x7f89480360f0 legacy=0x7f89480385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:52.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.755+0000 7f896e5c3640 10 monclient: shutdown 2024-09-16T14:46:52.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.755+0000 7f896e5c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:52.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.755+0000 7f896e5c3640 1 -- 172.21.15.189:0/3897998774 >> v1:172.21.15.189:6789/0 conn(0x7f896809e6d0 legacy=0x7f89681d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:52.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.755+0000 7f896e5c3640 1 -- 172.21.15.189:0/3897998774 shutdown_connections 2024-09-16T14:46:52.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.755+0000 7f896e5c3640 1 -- 172.21.15.189:0/3897998774 >> 172.21.15.189:0/3897998774 conn(0x7f89680a3380 msgr2=0x7f89680a0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:52.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.755+0000 7f896e5c3640 1 -- 172.21.15.189:0/3897998774 shutdown_connections 2024-09-16T14:46:52.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.755+0000 7f896e5c3640 1 -- 172.21.15.189:0/3897998774 wait complete. 2024-09-16T14:46:52.773 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T14:46:52.773 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep osd.11 2024-09-16T14:46:52.773 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 123 2024-09-16T14:46:52.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.879+0000 7f600d5b7640 10 monclient: get_monmap_and_config 2024-09-16T14:46:52.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.879+0000 7f600d5b7640 10 monclient: build_initial_monmap 2024-09-16T14:46:52.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.879+0000 7f600d5b7640 10 monclient: monmap: 2024-09-16T14:46:52.885 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:52.886 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:52.886 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:52.885533+0000 2024-09-16T14:46:52.886 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:52.885533+0000 2024-09-16T14:46:52.886 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:52.886 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:52.886 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:52.886 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:52.886 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:52.886 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:52.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 1 Processor -- start 2024-09-16T14:46:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 1 -- start start 2024-09-16T14:46:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 10 monclient: init 2024-09-16T14:46:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:46:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 10 monclient(hunting): picked mon.noname-a con 0x7f60081318a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 10 monclient(hunting): picked mon.noname-c con 0x7f6008134d50 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 10 monclient(hunting): picked mon.noname-b con 0x7f600813b2b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600809d8a0 con 0x7f60081318a0 2024-09-16T14:46:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600809ddf0 con 0x7f6008134d50 2024-09-16T14:46:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600809e8b0 con 0x7f600813b2b0 2024-09-16T14:46:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 10 monclient(hunting): authenticate will time out at 1195.001440s 2024-09-16T14:46:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f60067fc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6008134d50 0x7f6008139b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40014/0 (socket says 172.21.15.189:40014) 2024-09-16T14:46:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f60067fc640 1 -- 172.21.15.189:0/3725790856 learned_addr learned my addr 172.21.15.189:0/3725790856 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 1 -- 172.21.15.189:0/3725790856 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28285560 0 0) 0x7f600809ddf0 con 0x7f6008134d50 2024-09-16T14:46:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 10 monclient(hunting): my global_id is 94711 2024-09-16T14:46:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 1 -- 172.21.15.189:0/3725790856 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff0003530 con 0x7f6008134d50 2024-09-16T14:46:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 1 -- 172.21.15.189:0/3725790856 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3761787827 0 0) 0x7f5ff0003530 con 0x7f6008134d50 2024-09-16T14:46:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 1 -- 172.21.15.189:0/3725790856 >> v1:172.21.15.26:6790/0 conn(0x7f600813b2b0 legacy=0x7f600813d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 1 -- 172.21.15.189:0/3725790856 >> v1:172.21.15.26:6789/0 conn(0x7f60081318a0 legacy=0x7f6008131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:46:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:46:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 1 monclient: found mon.noname-c 2024-09-16T14:46:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 1 -- 172.21.15.189:0/3725790856 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600809eb90 con 0x7f6008134d50 2024-09-16T14:46:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 10 monclient: _finish_auth 0 2024-09-16T14:46:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 1 -- 172.21.15.189:0/3725790856 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5ffc001620 con 0x7f6008134d50 2024-09-16T14:46:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 5 monclient: authenticate success, global_id 94711 2024-09-16T14:46:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:52.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:52.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 10 monclient: dump: 2024-09-16T14:46:52.897 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:52.897 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:52.897 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:52.897 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:52.897 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:52.897 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:52.898 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:52.898 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:52.898 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:52.898 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:52.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:52.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 1 -- 172.21.15.189:0/3725790856 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ffc001910 con 0x7f6008134d50 2024-09-16T14:46:52.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:52.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:52.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 1 -- 172.21.15.189:0/3725790856 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5ffc001ca0 con 0x7f6008134d50 2024-09-16T14:46:52.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 4 set_mon_vals no callback set 2024-09-16T14:46:52.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:52.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:52.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f6005ffb640 10 monclient: dump: 2024-09-16T14:46:52.898 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:52.899 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 10 monclient: shutdown 2024-09-16T14:46:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:52.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 1 -- 172.21.15.189:0/3725790856 >> v1:172.21.15.189:6789/0 conn(0x7f6008134d50 legacy=0x7f6008139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:52.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 1 -- 172.21.15.189:0/3725790856 shutdown_connections 2024-09-16T14:46:52.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 1 -- 172.21.15.189:0/3725790856 >> 172.21.15.189:0/3725790856 conn(0x7f60080a3320 msgr2=0x7f600809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:52.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 1 -- 172.21.15.189:0/3725790856 shutdown_connections 2024-09-16T14:46:52.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 1 -- 172.21.15.189:0/3725790856 wait complete. 2024-09-16T14:46:52.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 10 monclient: build_initial_monmap 2024-09-16T14:46:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 10 monclient: monmap: 2024-09-16T14:46:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:52.902 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 1 Processor -- start 2024-09-16T14:46:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 1 -- start start 2024-09-16T14:46:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.883+0000 7f600d5b7640 10 monclient: init 2024-09-16T14:46:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 10 monclient(hunting): picked mon.noname-c con 0x7f60081318a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 10 monclient(hunting): picked mon.noname-a con 0x7f6008134d50 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 10 monclient(hunting): picked mon.noname-b con 0x7f600813b2b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600811b760 con 0x7f6008134d50 2024-09-16T14:46:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600811ba40 con 0x7f600813b2b0 2024-09-16T14:46:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600811a2b0 con 0x7f60081318a0 2024-09-16T14:46:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 10 monclient(hunting): authenticate will time out at 1195.004515s 2024-09-16T14:46:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f60077fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f600813b2b0 0x7f60081d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40028/0 (socket says 172.21.15.189:40028) 2024-09-16T14:46:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f60077fe640 1 -- 172.21.15.189:0/2435989636 learned_addr learned my addr 172.21.15.189:0/2435989636 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 1 -- 172.21.15.189:0/2435989636 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503473295 0 0) 0x7f600811ba40 con 0x7f600813b2b0 2024-09-16T14:46:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 10 monclient(hunting): my global_id is 94714 2024-09-16T14:46:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 1 -- 172.21.15.189:0/2435989636 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff4003530 con 0x7f600813b2b0 2024-09-16T14:46:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 1 -- 172.21.15.189:0/2435989636 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2026755533 0 0) 0x7f5ff4003530 con 0x7f600813b2b0 2024-09-16T14:46:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 1 -- 172.21.15.189:0/2435989636 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f600811ba40 con 0x7f600813b2b0 2024-09-16T14:46:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 1 -- 172.21.15.189:0/2435989636 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6000001c00 con 0x7f600813b2b0 2024-09-16T14:46:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 10 monclient(hunting): dump: 2024-09-16T14:46:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:52.905 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:52.905 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:52.905 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:52.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:52.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 1 -- 172.21.15.189:0/2435989636 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3708966734 0 0) 0x7f600811ba40 con 0x7f600813b2b0 2024-09-16T14:46:52.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 1 -- 172.21.15.189:0/2435989636 >> v1:172.21.15.26:6790/0 conn(0x7f60081318a0 legacy=0x7f60081c9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:52.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 1 -- 172.21.15.189:0/2435989636 >> v1:172.21.15.26:6789/0 conn(0x7f6008134d50 legacy=0x7f60081ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:52.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:52.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 1 monclient: found mon.b 2024-09-16T14:46:52.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:52.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:52.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 1 -- 172.21.15.189:0/2435989636 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60080a1d00 con 0x7f600813b2b0 2024-09-16T14:46:52.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:52.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:52.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:52.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 5 monclient: authenticate success, global_id 94714 2024-09-16T14:46:52.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 10 monclient: _renew_subs 2024-09-16T14:46:52.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:52.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 1 -- 172.21.15.189:0/2435989636 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600813ac40 con 0x7f600813b2b0 2024-09-16T14:46:52.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 10 monclient: _renew_subs 2024-09-16T14:46:52.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:52.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 1 -- 172.21.15.189:0/2435989636 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f600811b480 con 0x7f600813b2b0 2024-09-16T14:46:52.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 1 -- 172.21.15.189:0/2435989636 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6000002010 con 0x7f600813b2b0 2024-09-16T14:46:52.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:52.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 1 -- 172.21.15.189:0/2435989636 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6000003df0 con 0x7f600813b2b0 2024-09-16T14:46:52.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:52.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:52.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f60057fa640 4 set_mon_vals no callback set 2024-09-16T14:46:52.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 10 monclient: dump: 2024-09-16T14:46:52.907 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:52.907 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:52.907 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:52.907 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:52.907 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:52.907 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:52.907 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:52.907 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:52.908 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:52.908 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:52.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 1 -- 172.21.15.189:0/2435989636 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6000003fd0 con 0x7f600813b2b0 2024-09-16T14:46:52.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:52.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:52.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:52.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f600d5b7640 1 -- 172.21.15.189:0/2435989636 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe8005050 con 0x7f600813b2b0 2024-09-16T14:46:52.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f5fef7fe640 1 -- 172.21.15.189:0/2435989636 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(99..99 src has 65..99) v4 ==== 5142+0+0 (unknown 2621744354 0 0) 0x7f6000043bd0 con 0x7f600813b2b0 2024-09-16T14:46:52.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.887+0000 7f6006ffd640 10 monclient: get_auth_request con 0x7f5ff4036010 auth_method 0 2024-09-16T14:46:52.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.891+0000 7f5fef7fe640 1 -- 172.21.15.189:0/2435989636 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f60000148c0 con 0x7f600813b2b0 2024-09-16T14:46:52.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.891+0000 7f5fef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:52.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:52.891+0000 7f5fef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:53.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.039+0000 7f600d5b7640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:46:53.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.039+0000 7f600d5b7640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T14:46:53.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.039+0000 7f600d5b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:53.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.039+0000 7f600d5b7640 1 -- 172.21.15.189:0/2435989636 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f5fe8005b10 con 0x7f600813b2b0 2024-09-16T14:46:53.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.039+0000 7f5fef7fe640 1 -- 172.21.15.189:0/2435989636 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v99) v1 ==== 56+0+1173 (unknown 3011264473 0 3649659468) 0x7f60000146e0 con 0x7f600813b2b0 2024-09-16T14:46:53.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.039+0000 7f5fef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T14:46:53.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.039+0000 7f5fef7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:53.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.127+0000 7f600d5b7640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:46:53.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.127+0000 7f600d5b7640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T14:46:53.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.127+0000 7f600d5b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:53.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.127+0000 7f600d5b7640 1 -- 172.21.15.189:0/2435989636 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f5fe80055a0 con 0x7f600813b2b0 2024-09-16T14:46:53.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.127+0000 7f5fef7fe640 1 -- 172.21.15.189:0/2435989636 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v99) v1 ==== 56+0+1173 (unknown 3011264473 0 3649659468) 0x7f60000148c0 con 0x7f600813b2b0 2024-09-16T14:46:53.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.127+0000 7f5fef7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T14:46:53.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.127+0000 7f5fef7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:46:53.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.131+0000 7f600d5b7640 1 -- 172.21.15.189:0/2435989636 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5ff4036010 legacy=0x7f5ff40384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:53.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.131+0000 7f600d5b7640 10 monclient: shutdown 2024-09-16T14:46:53.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.131+0000 7f600d5b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:53.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.131+0000 7f600d5b7640 1 -- 172.21.15.189:0/2435989636 >> v1:172.21.15.189:6789/0 conn(0x7f600813b2b0 legacy=0x7f60081d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:53.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.131+0000 7f600d5b7640 1 -- 172.21.15.189:0/2435989636 shutdown_connections 2024-09-16T14:46:53.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.131+0000 7f600d5b7640 1 -- 172.21.15.189:0/2435989636 >> 172.21.15.189:0/2435989636 conn(0x7f60080a3320 msgr2=0x7f6008133370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:53.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.131+0000 7f600d5b7640 1 -- 172.21.15.189:0/2435989636 shutdown_connections 2024-09-16T14:46:53.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.131+0000 7f600d5b7640 1 -- 172.21.15.189:0/2435989636 wait complete. 2024-09-16T14:46:53.148 INFO:tasks.workunit.client.0.smithi189.stdout: 11 123.00000 osd.11 down 1.00000 1.00000 2024-09-16T14:46:53.148 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush reweight osd.11 113 2024-09-16T14:46:53.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.259+0000 7fbd757ca640 10 monclient: get_monmap_and_config 2024-09-16T14:46:53.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.259+0000 7fbd757ca640 10 monclient: build_initial_monmap 2024-09-16T14:46:53.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.259+0000 7fbd757ca640 10 monclient: monmap: 2024-09-16T14:46:53.264 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:53.264 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:53.265 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:53.264386+0000 2024-09-16T14:46:53.265 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:53.264386+0000 2024-09-16T14:46:53.265 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:53.265 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:53.265 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:53.265 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:53.265 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:53.265 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:53.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.259+0000 7fbd757ca640 1 Processor -- start 2024-09-16T14:46:53.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.259+0000 7fbd757ca640 1 -- start start 2024-09-16T14:46:53.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.259+0000 7fbd757ca640 10 monclient: init 2024-09-16T14:46:53.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.259+0000 7fbd757ca640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:53.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.259+0000 7fbd757ca640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:53.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.259+0000 7fbd757ca640 10 monclient(hunting): picked mon.noname-b con 0x7fbd7009f740 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:53.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.259+0000 7fbd757ca640 10 monclient(hunting): picked mon.noname-a con 0x7fbd700a0230 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:53.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.259+0000 7fbd757ca640 10 monclient(hunting): picked mon.noname-c con 0x7fbd70099d60 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:53.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.259+0000 7fbd757ca640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd70099660 con 0x7fbd700a0230 2024-09-16T14:46:53.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.259+0000 7fbd757ca640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd70099380 con 0x7fbd70099d60 2024-09-16T14:46:53.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.259+0000 7fbd757ca640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd700990a0 con 0x7fbd7009f740 2024-09-16T14:46:53.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.259+0000 7fbd757ca640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:53.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.259+0000 7fbd757ca640 10 monclient(hunting): authenticate will time out at 1195.379870s 2024-09-16T14:46:53.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6f7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbd70099d60 0x7fbd7013d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40036/0 (socket says 172.21.15.189:40036) 2024-09-16T14:46:53.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6f7fe640 1 -- 172.21.15.189:0/447093632 learned_addr learned my addr 172.21.15.189:0/447093632 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:53.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 1 -- 172.21.15.189:0/447093632 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420786645 0 0) 0x7fbd70099380 con 0x7fbd70099d60 2024-09-16T14:46:53.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:53.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:53.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:53.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 10 monclient(hunting): my global_id is 94717 2024-09-16T14:46:53.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 1 -- 172.21.15.189:0/447093632 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd58003530 con 0x7fbd70099d60 2024-09-16T14:46:53.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 1 -- 172.21.15.189:0/447093632 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2458582393 0 0) 0x7fbd58003530 con 0x7fbd70099d60 2024-09-16T14:46:53.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 1 -- 172.21.15.189:0/447093632 >> v1:172.21.15.26:6790/0 conn(0x7fbd7009f740 legacy=0x7fbd7009fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:53.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 1 -- 172.21.15.189:0/447093632 >> v1:172.21.15.26:6789/0 conn(0x7fbd700a0230 legacy=0x7fbd70139d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:46:53.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:46:53.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 1 monclient: found mon.noname-c 2024-09-16T14:46:53.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:53.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:53.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 1 -- 172.21.15.189:0/447093632 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd700a11a0 con 0x7fbd70099d60 2024-09-16T14:46:53.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 10 monclient: _finish_auth 0 2024-09-16T14:46:53.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:53.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:53.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 1 -- 172.21.15.189:0/447093632 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbd680019a0 con 0x7fbd70099d60 2024-09-16T14:46:53.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:53.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:53.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 10 monclient: dump: 2024-09-16T14:46:53.275 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:53.275 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:53.275 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:53.275 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:53.275 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:53.275 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:53.275 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:53.275 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:53.275 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:53.275 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:53.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 5 monclient: authenticate success, global_id 94717 2024-09-16T14:46:53.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:53.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 1 -- 172.21.15.189:0/447093632 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd68001c90 con 0x7fbd70099d60 2024-09-16T14:46:53.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:53.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 1 -- 172.21.15.189:0/447093632 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbd680020a0 con 0x7fbd70099d60 2024-09-16T14:46:53.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:53.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:53.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6dffb640 10 monclient: dump: 2024-09-16T14:46:53.276 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:53.276 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:53.276 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:53.276 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:53.276 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:53.276 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:53.276 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:53.276 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:53.277 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:53.277 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:53.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:53.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 4 set_mon_vals no callback set 2024-09-16T14:46:53.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 10 monclient: shutdown 2024-09-16T14:46:53.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:53.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 1 -- 172.21.15.189:0/447093632 >> v1:172.21.15.189:6789/0 conn(0x7fbd70099d60 legacy=0x7fbd7013d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:53.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 1 -- 172.21.15.189:0/447093632 shutdown_connections 2024-09-16T14:46:53.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 1 -- 172.21.15.189:0/447093632 >> 172.21.15.189:0/447093632 conn(0x7fbd700a4390 msgr2=0x7fbd700a0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:53.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 1 -- 172.21.15.189:0/447093632 shutdown_connections 2024-09-16T14:46:53.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 1 -- 172.21.15.189:0/447093632 wait complete. 2024-09-16T14:46:53.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 10 monclient: build_initial_monmap 2024-09-16T14:46:53.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 10 monclient: monmap: 2024-09-16T14:46:53.282 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:53.282 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:53.282 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:53.282 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:53.282 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:53.282 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:53.282 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:53.282 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:53.282 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:53.282 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:53.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 1 Processor -- start 2024-09-16T14:46:53.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 1 -- start start 2024-09-16T14:46:53.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 10 monclient: init 2024-09-16T14:46:53.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:53.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:53.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 10 monclient(hunting): picked mon.noname-b con 0x7fbd70099d60 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:53.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 10 monclient(hunting): picked mon.noname-c con 0x7fbd7009f740 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:53.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 10 monclient(hunting): picked mon.noname-a con 0x7fbd700a0230 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:53.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd70121740 con 0x7fbd700a0230 2024-09-16T14:46:53.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd7009b710 con 0x7fbd70099d60 2024-09-16T14:46:53.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd7009b100 con 0x7fbd7009f740 2024-09-16T14:46:53.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:53.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 10 monclient(hunting): authenticate will time out at 1195.382826s 2024-09-16T14:46:53.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6effd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbd70099d60 0x7fbd7011d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40052/0 (socket says 172.21.15.189:40052) 2024-09-16T14:46:53.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6effd640 1 -- 172.21.15.189:0/746244082 learned_addr learned my addr 172.21.15.189:0/746244082 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:53.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338766947 0 0) 0x7fbd7009b710 con 0x7fbd70099d60 2024-09-16T14:46:53.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:53.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:53.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:53.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient(hunting): my global_id is 94720 2024-09-16T14:46:53.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd5c003560 con 0x7fbd70099d60 2024-09-16T14:46:53.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3881736625 0 0) 0x7fbd5c003560 con 0x7fbd70099d60 2024-09-16T14:46:53.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd7009b710 con 0x7fbd70099d60 2024-09-16T14:46:53.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbd60001940 con 0x7fbd70099d60 2024-09-16T14:46:53.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:53.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient(hunting): dump: 2024-09-16T14:46:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:53.285 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377060685 0 0) 0x7fbd7009b100 con 0x7fbd7009f740 2024-09-16T14:46:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient(hunting): my global_id is 114110 2024-09-16T14:46:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd5c003560 con 0x7fbd7009f740 2024-09-16T14:46:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2013583884 0 0) 0x7fbd7009b710 con 0x7fbd70099d60 2024-09-16T14:46:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 >> v1:172.21.15.26:6790/0 conn(0x7fbd7009f740 legacy=0x7fbd7011ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 >> v1:172.21.15.26:6789/0 conn(0x7fbd700a0230 legacy=0x7fbd701d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 1 monclient: found mon.b 2024-09-16T14:46:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:53.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd70120ea0 con 0x7fbd70099d60 2024-09-16T14:46:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 5 monclient: authenticate success, global_id 94720 2024-09-16T14:46:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 10 monclient: _renew_subs 2024-09-16T14:46:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 1 -- 172.21.15.189:0/746244082 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd70121180 con 0x7fbd70099d60 2024-09-16T14:46:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 10 monclient: _renew_subs 2024-09-16T14:46:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd757ca640 1 -- 172.21.15.189:0/746244082 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd70130fa0 con 0x7fbd70099d60 2024-09-16T14:46:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd60001d50 con 0x7fbd70099d60 2024-09-16T14:46:53.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:53.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbd60003af0 con 0x7fbd70099d60 2024-09-16T14:46:53.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:53.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:53.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd577fe640 10 monclient: dump: 2024-09-16T14:46:53.288 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:53.288 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:53.288 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:53.288 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:53.288 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:53.288 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:53.288 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:53.288 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:53.289 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:53.289 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:53.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.263+0000 7fbd6d7fa640 4 set_mon_vals no callback set 2024-09-16T14:46:53.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.267+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbd60003cd0 con 0x7fbd70099d60 2024-09-16T14:46:53.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.267+0000 7fbd757ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:53.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.267+0000 7fbd757ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:53.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.267+0000 7fbd757ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:53.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.267+0000 7fbd757ca640 1 -- 172.21.15.189:0/746244082 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd50005050 con 0x7fbd70099d60 2024-09-16T14:46:53.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.267+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(99..99 src has 65..99) v4 ==== 5142+0+0 (unknown 2621744354 0 0) 0x7fbd60043950 con 0x7fbd70099d60 2024-09-16T14:46:53.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.267+0000 7fbd6e7fc640 10 monclient: get_auth_request con 0x7fbd5c036200 auth_method 0 2024-09-16T14:46:53.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.271+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbd60014640 con 0x7fbd70099d60 2024-09-16T14:46:53.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.271+0000 7fbd577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:53.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.271+0000 7fbd577fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.435+0000 7fbd757ca640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T14:46:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.435+0000 7fbd757ca640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T14:46:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.435+0000 7fbd757ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.435+0000 7fbd757ca640 1 -- 172.21.15.189:0/746244082 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0} v 0) v1 -- 0x7fbd50005880 con 0x7fbd70099d60 2024-09-16T14:46:53.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.655+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}]=0 reweighted item id 11 name 'osd.11' to 113 in crush map v100) v1 ==== 156+0+0 (unknown 1860376160 0 0) 0x7fbd60012ce0 con 0x7fbd70099d60 2024-09-16T14:46:53.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.655+0000 7fbd577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T14:46:53.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.655+0000 7fbd577fe640 10 monclient: _finish_command 2 = system:0 reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-16T14:46:53.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.755+0000 7fbd757ca640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T14:46:53.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.755+0000 7fbd757ca640 10 monclient: _send_command 3 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T14:46:53.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.755+0000 7fbd757ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:53.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:53.755+0000 7fbd757ca640 1 -- 172.21.15.189:0/746244082 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0} v 0) v1 -- 0x7fbd50005510 con 0x7fbd70099d60 2024-09-16T14:46:54.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.663+0000 7fbd577fe640 1 -- 172.21.15.189:0/746244082 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}]=0 reweighted item id 11 name 'osd.11' to 113 in crush map v101) v1 ==== 156+0+0 (unknown 1361214950 0 0) 0x7fbd60014460 con 0x7fbd70099d60 2024-09-16T14:46:54.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.663+0000 7fbd577fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T14:46:54.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.663+0000 7fbd577fe640 10 monclient: _finish_command 3 = system:0 reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-16T14:46:54.669 INFO:tasks.workunit.client.0.smithi189.stderr:reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-16T14:46:54.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.667+0000 7fbd757ca640 1 -- 172.21.15.189:0/746244082 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbd5c036200 legacy=0x7fbd5c0386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:54.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.667+0000 7fbd757ca640 10 monclient: shutdown 2024-09-16T14:46:54.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.667+0000 7fbd757ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:54.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.667+0000 7fbd757ca640 1 -- 172.21.15.189:0/746244082 >> v1:172.21.15.189:6789/0 conn(0x7fbd70099d60 legacy=0x7fbd7011d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:54.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.667+0000 7fbd757ca640 1 -- 172.21.15.189:0/746244082 shutdown_connections 2024-09-16T14:46:54.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.667+0000 7fbd757ca640 1 -- 172.21.15.189:0/746244082 >> 172.21.15.189:0/746244082 conn(0x7fbd700a4390 msgr2=0x7fbd700a3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:54.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.667+0000 7fbd757ca640 1 -- 172.21.15.189:0/746244082 shutdown_connections 2024-09-16T14:46:54.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.667+0000 7fbd757ca640 1 -- 172.21.15.189:0/746244082 wait complete. 2024-09-16T14:46:54.691 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush reweight osd.11 123456 2024-09-16T14:46:54.692 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:46:54.692 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush reweight osd.11 123456 2024-09-16T14:46:54.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.803+0000 7f935f8c0640 10 monclient: get_monmap_and_config 2024-09-16T14:46:54.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.803+0000 7f935f8c0640 10 monclient: build_initial_monmap 2024-09-16T14:46:54.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.803+0000 7f935f8c0640 10 monclient: monmap: 2024-09-16T14:46:54.809 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:54.809 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:54.809 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:54.808925+0000 2024-09-16T14:46:54.809 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:54.808925+0000 2024-09-16T14:46:54.810 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:54.810 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:54.810 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:54.810 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:54.810 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:54.810 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:54.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.803+0000 7f935f8c0640 1 Processor -- start 2024-09-16T14:46:54.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 1 -- start start 2024-09-16T14:46:54.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient: init 2024-09-16T14:46:54.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:54.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:54.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient(hunting): picked mon.noname-b con 0x7f9358099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:54.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient(hunting): picked mon.noname-a con 0x7f935809ff00 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:54.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient(hunting): picked mon.noname-c con 0x7f93580a0310 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:54.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935809b380 con 0x7f935809ff00 2024-09-16T14:46:54.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9358099e50 con 0x7f93580a0310 2024-09-16T14:46:54.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935809f2e0 con 0x7f9358099a70 2024-09-16T14:46:54.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:54.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient(hunting): authenticate will time out at 1196.924825s 2024-09-16T14:46:54.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935de5d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f93580a0310 0x7f9358139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40060/0 (socket says 172.21.15.189:40060) 2024-09-16T14:46:54.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935de5d640 1 -- 172.21.15.189:0/1588252914 learned_addr learned my addr 172.21.15.189:0/1588252914 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:54.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 1 -- 172.21.15.189:0/1588252914 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554845351 0 0) 0x7f9358099e50 con 0x7f93580a0310 2024-09-16T14:46:54.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:54.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:54.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:54.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 10 monclient(hunting): my global_id is 94723 2024-09-16T14:46:54.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 1 -- 172.21.15.189:0/1588252914 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9344003530 con 0x7f93580a0310 2024-09-16T14:46:54.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 1 -- 172.21.15.189:0/1588252914 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1058495389 0 0) 0x7f9344003530 con 0x7f93580a0310 2024-09-16T14:46:54.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 1 -- 172.21.15.189:0/1588252914 >> v1:172.21.15.26:6790/0 conn(0x7f9358099a70 legacy=0x7f935809bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 1 -- 172.21.15.189:0/1588252914 >> v1:172.21.15.26:6789/0 conn(0x7f935809ff00 legacy=0x7f93581365f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:46:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 1 monclient: found mon.noname-c 2024-09-16T14:46:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 1 -- 172.21.15.189:0/1588252914 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935809b070 con 0x7f93580a0310 2024-09-16T14:46:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 10 monclient: _finish_auth 0 2024-09-16T14:46:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 1 -- 172.21.15.189:0/1588252914 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f93500019a0 con 0x7f93580a0310 2024-09-16T14:46:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 10 monclient: dump: 2024-09-16T14:46:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:54.819 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 5 monclient: authenticate success, global_id 94723 2024-09-16T14:46:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 1 -- 172.21.15.189:0/1588252914 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9350001c90 con 0x7f93580a0310 2024-09-16T14:46:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 1 -- 172.21.15.189:0/1588252914 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f93500020a0 con 0x7f93580a0310 2024-09-16T14:46:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f934ffff640 10 monclient: dump: 2024-09-16T14:46:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:54.821 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 4 set_mon_vals no callback set 2024-09-16T14:46:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient: shutdown 2024-09-16T14:46:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 1 -- 172.21.15.189:0/1588252914 >> v1:172.21.15.189:6789/0 conn(0x7f93580a0310 legacy=0x7f9358139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 1 -- 172.21.15.189:0/1588252914 shutdown_connections 2024-09-16T14:46:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 1 -- 172.21.15.189:0/1588252914 >> 172.21.15.189:0/1588252914 conn(0x7f93580a3320 msgr2=0x7f935809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 1 -- 172.21.15.189:0/1588252914 shutdown_connections 2024-09-16T14:46:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 1 -- 172.21.15.189:0/1588252914 wait complete. 2024-09-16T14:46:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient: build_initial_monmap 2024-09-16T14:46:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient: monmap: 2024-09-16T14:46:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:54.822 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 1 Processor -- start 2024-09-16T14:46:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 1 -- start start 2024-09-16T14:46:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient: init 2024-09-16T14:46:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient(hunting): picked mon.noname-c con 0x7f9358099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient(hunting): picked mon.noname-a con 0x7f935809ff00 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient(hunting): picked mon.noname-b con 0x7f93580a0310 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9358119dd0 con 0x7f935809ff00 2024-09-16T14:46:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935813a670 con 0x7f93580a0310 2024-09-16T14:46:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935806b380 con 0x7f9358099a70 2024-09-16T14:46:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.807+0000 7f935f8c0640 10 monclient(hunting): authenticate will time out at 1196.927917s 2024-09-16T14:46:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f935de5d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f93580a0310 0x7f93581cf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40068/0 (socket says 172.21.15.189:40068) 2024-09-16T14:46:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f935de5d640 1 -- 172.21.15.189:0/3266179058 learned_addr learned my addr 172.21.15.189:0/3266179058 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 1 -- 172.21.15.189:0/3266179058 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103773893 0 0) 0x7f935813a670 con 0x7f93580a0310 2024-09-16T14:46:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 10 monclient(hunting): my global_id is 94726 2024-09-16T14:46:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 1 -- 172.21.15.189:0/3266179058 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f933c003530 con 0x7f93580a0310 2024-09-16T14:46:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 1 -- 172.21.15.189:0/3266179058 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3855558928 0 0) 0x7f933c003530 con 0x7f93580a0310 2024-09-16T14:46:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 1 -- 172.21.15.189:0/3266179058 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f935813a670 con 0x7f93580a0310 2024-09-16T14:46:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 1 -- 172.21.15.189:0/3266179058 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f93500017c0 con 0x7f93580a0310 2024-09-16T14:46:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 10 monclient(hunting): dump: 2024-09-16T14:46:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:54.825 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 1 -- 172.21.15.189:0/3266179058 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3933119650 0 0) 0x7f935813a670 con 0x7f93580a0310 2024-09-16T14:46:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 1 -- 172.21.15.189:0/3266179058 >> v1:172.21.15.26:6790/0 conn(0x7f9358099a70 legacy=0x7f9358139850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 1 -- 172.21.15.189:0/3266179058 >> v1:172.21.15.26:6789/0 conn(0x7f935809ff00 legacy=0x7f9358139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:46:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 1 monclient: found mon.b 2024-09-16T14:46:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 1 -- 172.21.15.189:0/3266179058 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93580a1fd0 con 0x7f93580a0310 2024-09-16T14:46:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 10 monclient: _finish_auth 0 2024-09-16T14:46:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f935f8c0640 5 monclient: authenticate success, global_id 94726 2024-09-16T14:46:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f935f8c0640 10 monclient: _renew_subs 2024-09-16T14:46:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f935f8c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:54.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f935f8c0640 1 -- 172.21.15.189:0/3266179058 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935811b6c0 con 0x7f93580a0310 2024-09-16T14:46:54.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f935f8c0640 10 monclient: _renew_subs 2024-09-16T14:46:54.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f935f8c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:54.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f935f8c0640 1 -- 172.21.15.189:0/3266179058 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9358070450 con 0x7f93580a0310 2024-09-16T14:46:54.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 1 -- 172.21.15.189:0/3266179058 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9350005050 con 0x7f93580a0310 2024-09-16T14:46:54.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:54.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 1 -- 172.21.15.189:0/3266179058 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9350005360 con 0x7f93580a0310 2024-09-16T14:46:54.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:54.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:54.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 10 monclient: dump: 2024-09-16T14:46:54.827 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:54.827 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:54.827 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:54.828 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:54.828 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:54.828 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:54.828 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:54.828 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:54.828 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:54.828 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:54.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934f7fe640 4 set_mon_vals no callback set 2024-09-16T14:46:54.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 1 -- 172.21.15.189:0/3266179058 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f93500054c0 con 0x7f93580a0310 2024-09-16T14:46:54.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f935f8c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:54.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f935f8c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:54.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f935f8c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:54.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f935f8c0640 1 -- 172.21.15.189:0/3266179058 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9338005050 con 0x7f93580a0310 2024-09-16T14:46:54.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f934dffb640 1 -- 172.21.15.189:0/3266179058 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(101..101 src has 73..101) v4 ==== 5142+0+0 (unknown 1808928000 0 0) 0x7f9350043d90 con 0x7f93580a0310 2024-09-16T14:46:54.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.811+0000 7f935d65c640 10 monclient: get_auth_request con 0x7f933c036030 auth_method 0 2024-09-16T14:46:54.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.815+0000 7f934dffb640 1 -- 172.21.15.189:0/3266179058 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9350014e30 con 0x7f93580a0310 2024-09-16T14:46:54.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.815+0000 7f934dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:54.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.815+0000 7f934dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:54.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.979+0000 7f935f8c0640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-16T14:46:54.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.979+0000 7f935f8c0640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-16T14:46:54.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.979+0000 7f935f8c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:54.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.979+0000 7f935f8c0640 1 -- 172.21.15.189:0/3266179058 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0} v 0) v1 -- 0x7f9338005880 con 0x7f93580a0310 2024-09-16T14:46:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.979+0000 7f934dffb640 1 -- 172.21.15.189:0/3266179058 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}]=-75 (75) Value too large for defined data type v101) v1 ==== 146+0+0 (unknown 1716379202 0 0) 0x7f93500132a0 con 0x7f93580a0310 2024-09-16T14:46:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.979+0000 7f934dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-16T14:46:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:54.979+0000 7f934dffb640 10 monclient: _finish_command 2 = mon:75 (75) Value too large for defined data type 2024-09-16T14:46:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2024-09-16T14:46:54.987 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:46:54.987 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T14:46:54.987 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep osd.11 2024-09-16T14:46:54.987 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 113 2024-09-16T14:46:55.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.095+0000 7fcf5fa63640 10 monclient: get_monmap_and_config 2024-09-16T14:46:55.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.095+0000 7fcf5fa63640 10 monclient: build_initial_monmap 2024-09-16T14:46:55.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.095+0000 7fcf5fa63640 10 monclient: monmap: 2024-09-16T14:46:55.101 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:55.102 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:55.102 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:55.101520+0000 2024-09-16T14:46:55.102 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:55.101520+0000 2024-09-16T14:46:55.102 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:55.102 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:55.102 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:55.102 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:55.102 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:55.102 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 1 Processor -- start 2024-09-16T14:46:55.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 1 -- start start 2024-09-16T14:46:55.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 10 monclient: init 2024-09-16T14:46:55.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:55.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:55.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 10 monclient(hunting): picked mon.noname-b con 0x7fcf58099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:55.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 10 monclient(hunting): picked mon.noname-c con 0x7fcf5809ff00 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 10 monclient(hunting): picked mon.noname-a con 0x7fcf580a0310 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf5809b380 con 0x7fcf580a0310 2024-09-16T14:46:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf58099e50 con 0x7fcf5809ff00 2024-09-16T14:46:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf5809f2e0 con 0x7fcf58099a70 2024-09-16T14:46:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 10 monclient(hunting): authenticate will time out at 1197.217525s 2024-09-16T14:46:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5cffe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcf5809ff00 0x7fcf581365f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40074/0 (socket says 172.21.15.189:40074) 2024-09-16T14:46:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5cffe640 1 -- 172.21.15.189:0/3225007547 learned_addr learned my addr 172.21.15.189:0/3225007547 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 1 -- 172.21.15.189:0/3225007547 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052897841 0 0) 0x7fcf58099e50 con 0x7fcf5809ff00 2024-09-16T14:46:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 10 monclient(hunting): my global_id is 94729 2024-09-16T14:46:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 1 -- 172.21.15.189:0/3225007547 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf44003530 con 0x7fcf5809ff00 2024-09-16T14:46:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 1 -- 172.21.15.189:0/3225007547 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 394910697 0 0) 0x7fcf44003530 con 0x7fcf5809ff00 2024-09-16T14:46:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 1 -- 172.21.15.189:0/3225007547 >> v1:172.21.15.26:6790/0 conn(0x7fcf58099a70 legacy=0x7fcf5809bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 1 -- 172.21.15.189:0/3225007547 >> v1:172.21.15.26:6789/0 conn(0x7fcf580a0310 legacy=0x7fcf58139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:46:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 1 monclient: found mon.noname-c 2024-09-16T14:46:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 1 -- 172.21.15.189:0/3225007547 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5809b070 con 0x7fcf5809ff00 2024-09-16T14:46:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 10 monclient: _finish_auth 0 2024-09-16T14:46:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 5 monclient: authenticate success, global_id 94729 2024-09-16T14:46:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 1 -- 172.21.15.189:0/3225007547 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcf48001620 con 0x7fcf5809ff00 2024-09-16T14:46:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 10 monclient: dump: 2024-09-16T14:46:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:55.108 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 1 -- 172.21.15.189:0/3225007547 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf48001910 con 0x7fcf5809ff00 2024-09-16T14:46:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 4 set_mon_vals no callback set 2024-09-16T14:46:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 1 -- 172.21.15.189:0/3225007547 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcf48001d20 con 0x7fcf5809ff00 2024-09-16T14:46:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf4ffff640 10 monclient: dump: 2024-09-16T14:46:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:55.110 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 10 monclient: shutdown 2024-09-16T14:46:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 1 -- 172.21.15.189:0/3225007547 >> v1:172.21.15.189:6789/0 conn(0x7fcf5809ff00 legacy=0x7fcf581365f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 1 -- 172.21.15.189:0/3225007547 shutdown_connections 2024-09-16T14:46:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 1 -- 172.21.15.189:0/3225007547 >> 172.21.15.189:0/3225007547 conn(0x7fcf580a3320 msgr2=0x7fcf5809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 1 -- 172.21.15.189:0/3225007547 shutdown_connections 2024-09-16T14:46:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 1 -- 172.21.15.189:0/3225007547 wait complete. 2024-09-16T14:46:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 10 monclient: build_initial_monmap 2024-09-16T14:46:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 10 monclient: monmap: 2024-09-16T14:46:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:55.112 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.099+0000 7fcf5fa63640 1 Processor -- start 2024-09-16T14:46:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 1 -- start start 2024-09-16T14:46:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 10 monclient: init 2024-09-16T14:46:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 10 monclient(hunting): picked mon.noname-b con 0x7fcf58099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 10 monclient(hunting): picked mon.noname-c con 0x7fcf5809ff00 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 10 monclient(hunting): picked mon.noname-a con 0x7fcf580a0310 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf5813a6a0 con 0x7fcf580a0310 2024-09-16T14:46:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf58125d10 con 0x7fcf58099a70 2024-09-16T14:46:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf581258f0 con 0x7fcf5809ff00 2024-09-16T14:46:55.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:55.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 10 monclient(hunting): authenticate will time out at 1197.220592s 2024-09-16T14:46:55.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5d7ff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcf58099a70 0x7fcf581c97d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40082/0 (socket says 172.21.15.189:40082) 2024-09-16T14:46:55.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5d7ff640 1 -- 172.21.15.189:0/640527661 learned_addr learned my addr 172.21.15.189:0/640527661 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:55.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298624855 0 0) 0x7fcf58125d10 con 0x7fcf58099a70 2024-09-16T14:46:55.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:55.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:55.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:55.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient(hunting): my global_id is 94732 2024-09-16T14:46:55.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf3c003530 con 0x7fcf58099a70 2024-09-16T14:46:55.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3920880673 0 0) 0x7fcf3c003530 con 0x7fcf58099a70 2024-09-16T14:46:55.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf58125d10 con 0x7fcf58099a70 2024-09-16T14:46:55.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcf54001940 con 0x7fcf58099a70 2024-09-16T14:46:55.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:55.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:55.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient(hunting): dump: 2024-09-16T14:46:55.117 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:55.117 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:55.117 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.117 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.117 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:55.117 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:55.117 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:55.117 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:55.117 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:55.117 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:55.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388946872 0 0) 0x7fcf581258f0 con 0x7fcf5809ff00 2024-09-16T14:46:55.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:55.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:55.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:55.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient(hunting): my global_id is 114119 2024-09-16T14:46:55.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf3c003530 con 0x7fcf5809ff00 2024-09-16T14:46:55.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1558126388 0 0) 0x7fcf58125d10 con 0x7fcf58099a70 2024-09-16T14:46:55.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 >> v1:172.21.15.26:6790/0 conn(0x7fcf5809ff00 legacy=0x7fcf581ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:55.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 >> v1:172.21.15.26:6789/0 conn(0x7fcf580a0310 legacy=0x7fcf581d1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:55.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:46:55.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 1 monclient: found mon.b 2024-09-16T14:46:55.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:55.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf58070570 con 0x7fcf58099a70 2024-09-16T14:46:55.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient: _finish_auth 0 2024-09-16T14:46:55.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:55.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:55.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 5 monclient: authenticate success, global_id 94732 2024-09-16T14:46:55.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 10 monclient: _renew_subs 2024-09-16T14:46:55.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 1 -- 172.21.15.189:0/640527661 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf580a20f0 con 0x7fcf58099a70 2024-09-16T14:46:55.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 10 monclient: _renew_subs 2024-09-16T14:46:55.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 1 -- 172.21.15.189:0/640527661 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf5806e000 con 0x7fcf58099a70 2024-09-16T14:46:55.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf54001d50 con 0x7fcf58099a70 2024-09-16T14:46:55.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:55.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcf54003af0 con 0x7fcf58099a70 2024-09-16T14:46:55.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:55.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4f7fe640 4 set_mon_vals no callback set 2024-09-16T14:46:55.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:55.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 10 monclient: dump: 2024-09-16T14:46:55.120 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:55.120 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:55.120 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.120 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.120 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:55.120 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:55.120 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:55.120 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:55.121 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:55.121 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:55.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:55.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5fa63640 1 -- 172.21.15.189:0/640527661 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf38005050 con 0x7fcf58099a70 2024-09-16T14:46:55.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fcf54003c50 con 0x7fcf58099a70 2024-09-16T14:46:55.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(101..101 src has 73..101) v4 ==== 5142+0+0 (unknown 1808928000 0 0) 0x7fcf54043990 con 0x7fcf58099a70 2024-09-16T14:46:55.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.103+0000 7fcf5cffe640 10 monclient: get_auth_request con 0x7fcf3c036130 auth_method 0 2024-09-16T14:46:55.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.107+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fcf54004470 con 0x7fcf58099a70 2024-09-16T14:46:55.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.107+0000 7fcf4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:55.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.107+0000 7fcf4dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:55.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.267+0000 7fcf5fa63640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:46:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.267+0000 7fcf5fa63640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T14:46:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.267+0000 7fcf5fa63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.267+0000 7fcf5fa63640 1 -- 172.21.15.189:0/640527661 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fcf38005b10 con 0x7fcf58099a70 2024-09-16T14:46:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.271+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v101) v1 ==== 56+0+1173 (unknown 4176898266 0 2657860669) 0x7fcf54012d80 con 0x7fcf58099a70 2024-09-16T14:46:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.271+0000 7fcf4dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T14:46:55.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.271+0000 7fcf4dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:55.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.355+0000 7fcf5fa63640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:46:55.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.355+0000 7fcf5fa63640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T14:46:55.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.355+0000 7fcf5fa63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.355+0000 7fcf5fa63640 1 -- 172.21.15.189:0/640527661 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fcf380055a0 con 0x7fcf58099a70 2024-09-16T14:46:55.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.355+0000 7fcf4dffb640 1 -- 172.21.15.189:0/640527661 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v101) v1 ==== 56+0+1173 (unknown 4176898266 0 2657860669) 0x7fcf54014500 con 0x7fcf58099a70 2024-09-16T14:46:55.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.355+0000 7fcf4dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T14:46:55.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.355+0000 7fcf4dffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:46:55.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.359+0000 7fcf5fa63640 1 -- 172.21.15.189:0/640527661 >> v1:172.21.15.189:6812/3196161582 conn(0x7fcf3c036130 legacy=0x7fcf3c0385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:55.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.359+0000 7fcf5fa63640 10 monclient: shutdown 2024-09-16T14:46:55.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.359+0000 7fcf5fa63640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:55.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.359+0000 7fcf5fa63640 1 -- 172.21.15.189:0/640527661 >> v1:172.21.15.189:6789/0 conn(0x7fcf58099a70 legacy=0x7fcf581c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:55.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.359+0000 7fcf5fa63640 1 -- 172.21.15.189:0/640527661 shutdown_connections 2024-09-16T14:46:55.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.359+0000 7fcf5fa63640 1 -- 172.21.15.189:0/640527661 >> 172.21.15.189:0/640527661 conn(0x7fcf580a3320 msgr2=0x7fcf5809bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:55.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.359+0000 7fcf5fa63640 1 -- 172.21.15.189:0/640527661 shutdown_connections 2024-09-16T14:46:55.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.359+0000 7fcf5fa63640 1 -- 172.21.15.189:0/640527661 wait complete. 2024-09-16T14:46:55.376 INFO:tasks.workunit.client.0.smithi189.stdout: 11 113.00000 osd.11 down 1.00000 1.00000 2024-09-16T14:46:55.377 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm osd.11 2024-09-16T14:46:55.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 10 monclient: get_monmap_and_config 2024-09-16T14:46:55.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 10 monclient: build_initial_monmap 2024-09-16T14:46:55.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 10 monclient: monmap: 2024-09-16T14:46:55.491 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:55.491 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:55.491 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:55.490930+0000 2024-09-16T14:46:55.491 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:55.490930+0000 2024-09-16T14:46:55.491 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:55.491 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:55.491 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:55.491 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:55.491 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:55.491 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 1 Processor -- start 2024-09-16T14:46:55.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 1 -- start start 2024-09-16T14:46:55.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 10 monclient: init 2024-09-16T14:46:55.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:55.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:55.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 10 monclient(hunting): picked mon.noname-c con 0x7fe2d006c1e0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:55.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 10 monclient(hunting): picked mon.noname-b con 0x7fe2d00a1140 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:55.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 10 monclient(hunting): picked mon.noname-a con 0x7fe2d0135360 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:55.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d006c5c0 con 0x7fe2d0135360 2024-09-16T14:46:55.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d009e4d0 con 0x7fe2d006c1e0 2024-09-16T14:46:55.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d009de70 con 0x7fe2d00a1140 2024-09-16T14:46:55.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:55.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 10 monclient(hunting): authenticate will time out at 1197.606721s 2024-09-16T14:46:55.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2cf7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe2d006c1e0 0x7fe2d0134c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40094/0 (socket says 172.21.15.189:40094) 2024-09-16T14:46:55.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2cf7fe640 1 -- 172.21.15.189:0/2141492703 learned_addr learned my addr 172.21.15.189:0/2141492703 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:55.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 1 -- 172.21.15.189:0/2141492703 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711572541 0 0) 0x7fe2d009e4d0 con 0x7fe2d006c1e0 2024-09-16T14:46:55.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:55.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:55.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:55.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 10 monclient(hunting): my global_id is 94735 2024-09-16T14:46:55.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 1 -- 172.21.15.189:0/2141492703 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b8003530 con 0x7fe2d006c1e0 2024-09-16T14:46:55.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 1 -- 172.21.15.189:0/2141492703 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3204798327 0 0) 0x7fe2b8003530 con 0x7fe2d006c1e0 2024-09-16T14:46:55.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 1 -- 172.21.15.189:0/2141492703 >> v1:172.21.15.26:6790/0 conn(0x7fe2d00a1140 legacy=0x7fe2d0139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:55.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 1 -- 172.21.15.189:0/2141492703 >> v1:172.21.15.26:6789/0 conn(0x7fe2d0135360 legacy=0x7fe2d013d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:46:55.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:46:55.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 1 monclient: found mon.noname-c 2024-09-16T14:46:55.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:55.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 1 -- 172.21.15.189:0/2141492703 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d009e990 con 0x7fe2d006c1e0 2024-09-16T14:46:55.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 10 monclient: _finish_auth 0 2024-09-16T14:46:55.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:46:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 5 monclient: authenticate success, global_id 94735 2024-09-16T14:46:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 1 -- 172.21.15.189:0/2141492703 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2c0001620 con 0x7fe2d006c1e0 2024-09-16T14:46:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 10 monclient: dump: 2024-09-16T14:46:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:55.501 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:55.501 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:55.501 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:55.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 1 -- 172.21.15.189:0/2141492703 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2c0001910 con 0x7fe2d006c1e0 2024-09-16T14:46:55.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:55.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:55.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 1 -- 172.21.15.189:0/2141492703 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2c0001ca0 con 0x7fe2d006c1e0 2024-09-16T14:46:55.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2d5c8f640 4 set_mon_vals no callback set 2024-09-16T14:46:55.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:55.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:55.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.487+0000 7fe2ce7fc640 10 monclient: dump: 2024-09-16T14:46:55.501 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:55.502 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:55.502 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.502 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.502 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:55.502 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:55.502 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:55.502 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:55.502 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:55.502 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient: shutdown 2024-09-16T14:46:55.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:55.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 1 -- 172.21.15.189:0/2141492703 >> v1:172.21.15.189:6789/0 conn(0x7fe2d006c1e0 legacy=0x7fe2d0134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:55.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 1 -- 172.21.15.189:0/2141492703 shutdown_connections 2024-09-16T14:46:55.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 1 -- 172.21.15.189:0/2141492703 >> 172.21.15.189:0/2141492703 conn(0x7fe2d00a3a40 msgr2=0x7fe2d00a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:55.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 1 -- 172.21.15.189:0/2141492703 shutdown_connections 2024-09-16T14:46:55.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 1 -- 172.21.15.189:0/2141492703 wait complete. 2024-09-16T14:46:55.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient: build_initial_monmap 2024-09-16T14:46:55.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient: monmap: 2024-09-16T14:46:55.503 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:55.503 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:55.503 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:55.503 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:55.503 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:55.503 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:55.503 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:55.503 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:55.504 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:55.504 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 1 Processor -- start 2024-09-16T14:46:55.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 1 -- start start 2024-09-16T14:46:55.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient: init 2024-09-16T14:46:55.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:55.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:55.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient(hunting): picked mon.noname-b con 0x7fe2d006c1e0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:55.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient(hunting): picked mon.noname-c con 0x7fe2d00a1140 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:55.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient(hunting): picked mon.noname-a con 0x7fe2d0135360 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:55.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d01311d0 con 0x7fe2d0135360 2024-09-16T14:46:55.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d009e130 con 0x7fe2d006c1e0 2024-09-16T14:46:55.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d00a5430 con 0x7fe2d00a1140 2024-09-16T14:46:55.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:55.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient(hunting): authenticate will time out at 1197.609793s 2024-09-16T14:46:55.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2cf7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe2d006c1e0 0x7fe2d012dc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40108/0 (socket says 172.21.15.189:40108) 2024-09-16T14:46:55.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2cf7fe640 1 -- 172.21.15.189:0/585349018 learned_addr learned my addr 172.21.15.189:0/585349018 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:55.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 1 -- 172.21.15.189:0/585349018 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129226420 0 0) 0x7fe2d009e130 con 0x7fe2d006c1e0 2024-09-16T14:46:55.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:55.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:55.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:55.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 10 monclient(hunting): my global_id is 94738 2024-09-16T14:46:55.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 1 -- 172.21.15.189:0/585349018 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2bc003530 con 0x7fe2d006c1e0 2024-09-16T14:46:55.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 1 -- 172.21.15.189:0/585349018 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3439140459 0 0) 0x7fe2bc003530 con 0x7fe2d006c1e0 2024-09-16T14:46:55.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 1 -- 172.21.15.189:0/585349018 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2d009e130 con 0x7fe2d006c1e0 2024-09-16T14:46:55.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 1 -- 172.21.15.189:0/585349018 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2c00015e0 con 0x7fe2d006c1e0 2024-09-16T14:46:55.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:55.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:55.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 10 monclient(hunting): dump: 2024-09-16T14:46:55.506 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:55.506 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:55.506 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.506 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.506 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:55.506 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:55.507 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:55.507 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:55.507 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:55.507 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:55.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 1 -- 172.21.15.189:0/585349018 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4173561473 0 0) 0x7fe2d009e130 con 0x7fe2d006c1e0 2024-09-16T14:46:55.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 1 -- 172.21.15.189:0/585349018 >> v1:172.21.15.26:6790/0 conn(0x7fe2d00a1140 legacy=0x7fe2d012e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:55.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 1 -- 172.21.15.189:0/585349018 >> v1:172.21.15.26:6789/0 conn(0x7fe2d0135360 legacy=0x7fe2d01d1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:55.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:55.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 1 monclient: found mon.b 2024-09-16T14:46:55.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:55.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 1 -- 172.21.15.189:0/585349018 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d012ea70 con 0x7fe2d006c1e0 2024-09-16T14:46:55.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 10 monclient: _finish_auth 0 2024-09-16T14:46:55.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:55.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:55.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 5 monclient: authenticate success, global_id 94738 2024-09-16T14:46:55.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient: _renew_subs 2024-09-16T14:46:55.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 1 -- 172.21.15.189:0/585349018 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d0132db0 con 0x7fe2d006c1e0 2024-09-16T14:46:55.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient: _renew_subs 2024-09-16T14:46:55.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 1 -- 172.21.15.189:0/585349018 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d012ee00 con 0x7fe2d006c1e0 2024-09-16T14:46:55.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 1 -- 172.21.15.189:0/585349018 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2c0004c00 con 0x7fe2d006c1e0 2024-09-16T14:46:55.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:55.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 1 -- 172.21.15.189:0/585349018 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2c0004f90 con 0x7fe2d006c1e0 2024-09-16T14:46:55.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:55.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:55.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2cdffb640 4 set_mon_vals no callback set 2024-09-16T14:46:55.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 10 monclient: dump: 2024-09-16T14:46:55.509 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:55.509 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:55.509 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.509 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.509 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:55.509 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:55.509 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:55.510 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:55.510 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:55.510 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:55.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:55.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2d5c8f640 1 -- 172.21.15.189:0/585349018 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b0005050 con 0x7fe2d006c1e0 2024-09-16T14:46:55.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 1 -- 172.21.15.189:0/585349018 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe2c0005170 con 0x7fe2d006c1e0 2024-09-16T14:46:55.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.491+0000 7fe2b7fff640 1 -- 172.21.15.189:0/585349018 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(101..101 src has 73..101) v4 ==== 5142+0+0 (unknown 1808928000 0 0) 0x7fe2c0043b60 con 0x7fe2d006c1e0 2024-09-16T14:46:55.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.495+0000 7fe2ceffd640 10 monclient: get_auth_request con 0x7fe2bc035fe0 auth_method 0 2024-09-16T14:46:55.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.499+0000 7fe2b7fff640 1 -- 172.21.15.189:0/585349018 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe2c0014c00 con 0x7fe2d006c1e0 2024-09-16T14:46:55.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.499+0000 7fe2b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:55.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.499+0000 7fe2b7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:55.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.659+0000 7fe2d5c8f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T14:46:55.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.659+0000 7fe2d5c8f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T14:46:55.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.659+0000 7fe2d5c8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.659+0000 7fe2d5c8f640 1 -- 172.21.15.189:0/585349018 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.11"} v 0) v1 -- 0x7fe2b0005300 con 0x7fe2d006c1e0 2024-09-16T14:46:55.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.671+0000 7fe2b7fff640 1 -- 172.21.15.189:0/585349018 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.11"}]=0 removed item id 11 name 'osd.11' from crush map v102) v1 ==== 125+0+0 (unknown 1807744277 0 0) 0x7fe2c0013070 con 0x7fe2d006c1e0 2024-09-16T14:46:55.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.671+0000 7fe2b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T14:46:55.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.671+0000 7fe2b7fff640 10 monclient: _finish_command 2 = system:0 removed item id 11 name 'osd.11' from crush map 2024-09-16T14:46:55.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.767+0000 7fe2d5c8f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T14:46:55.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.767+0000 7fe2d5c8f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T14:46:55.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.767+0000 7fe2d5c8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.767+0000 7fe2d5c8f640 1 -- 172.21.15.189:0/585349018 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.11"} v 0) v1 -- 0x7fe2b00055a0 con 0x7fe2d006c1e0 2024-09-16T14:46:55.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.767+0000 7fe2b7fff640 1 -- 172.21.15.189:0/585349018 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.11"}]=0 device 'osd.11' does not appear in the crush map v102) v1 ==== 126+0+0 (unknown 4119740108 0 0) 0x7fe2c00147f0 con 0x7fe2d006c1e0 2024-09-16T14:46:55.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.767+0000 7fe2b7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T14:46:55.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.767+0000 7fe2b7fff640 10 monclient: _finish_command 3 = system:0 device 'osd.11' does not appear in the crush map 2024-09-16T14:46:55.774 INFO:tasks.workunit.client.0.smithi189.stderr:device 'osd.11' does not appear in the crush map 2024-09-16T14:46:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.771+0000 7fe2d5c8f640 1 -- 172.21.15.189:0/585349018 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe2bc035fe0 legacy=0x7fe2bc0384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.771+0000 7fe2d5c8f640 10 monclient: shutdown 2024-09-16T14:46:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.771+0000 7fe2d5c8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.771+0000 7fe2d5c8f640 1 -- 172.21.15.189:0/585349018 >> v1:172.21.15.189:6789/0 conn(0x7fe2d006c1e0 legacy=0x7fe2d012dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.771+0000 7fe2d5c8f640 1 -- 172.21.15.189:0/585349018 shutdown_connections 2024-09-16T14:46:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.771+0000 7fe2d5c8f640 1 -- 172.21.15.189:0/585349018 >> 172.21.15.189:0/585349018 conn(0x7fe2d00a3a40 msgr2=0x7fe2d00a58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.771+0000 7fe2d5c8f640 1 -- 172.21.15.189:0/585349018 shutdown_connections 2024-09-16T14:46:55.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.771+0000 7fe2d5c8f640 1 -- 172.21.15.189:0/585349018 wait complete. 2024-09-16T14:46:55.790 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd rm osd.11 2024-09-16T14:46:55.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 10 monclient: get_monmap_and_config 2024-09-16T14:46:55.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 10 monclient: build_initial_monmap 2024-09-16T14:46:55.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 10 monclient: monmap: 2024-09-16T14:46:55.899 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:55.899 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:55.899 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:55.899002+0000 2024-09-16T14:46:55.899 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:55.899002+0000 2024-09-16T14:46:55.899 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:55.900 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:55.900 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:55.900 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:55.900 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:55.900 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 1 Processor -- start 2024-09-16T14:46:55.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 1 -- start start 2024-09-16T14:46:55.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 10 monclient: init 2024-09-16T14:46:55.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:55.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:55.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 10 monclient(hunting): picked mon.noname-a con 0x7fbc0409df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:55.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 10 monclient(hunting): picked mon.noname-b con 0x7fbc0409e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:55.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 10 monclient(hunting): picked mon.noname-c con 0x7fbc0409e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:55.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0409ee20 con 0x7fbc0409df10 2024-09-16T14:46:55.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0409f100 con 0x7fbc0409e740 2024-09-16T14:46:55.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0409f3e0 con 0x7fbc0409e2f0 2024-09-16T14:46:55.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:55.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 10 monclient(hunting): authenticate will time out at 1198.015194s 2024-09-16T14:46:55.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc09459640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbc0409e740 0x7fbc0413b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40124/0 (socket says 172.21.15.189:40124) 2024-09-16T14:46:55.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc09459640 1 -- 172.21.15.189:0/3755467271 learned_addr learned my addr 172.21.15.189:0/3755467271 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:55.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 1 -- 172.21.15.189:0/3755467271 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149945986 0 0) 0x7fbc0409f100 con 0x7fbc0409e740 2024-09-16T14:46:55.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 10 monclient(hunting): _negotiate 2024-09-16T14:46:55.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:55.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:55.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 10 monclient(hunting): my global_id is 94741 2024-09-16T14:46:55.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 1 -- 172.21.15.189:0/3755467271 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbec003530 con 0x7fbc0409e740 2024-09-16T14:46:55.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 1 -- 172.21.15.189:0/3755467271 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2052333800 0 0) 0x7fbbec003530 con 0x7fbc0409e740 2024-09-16T14:46:55.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 1 -- 172.21.15.189:0/3755467271 >> v1:172.21.15.26:6790/0 conn(0x7fbc0409e2f0 legacy=0x7fbc041378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:55.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 1 -- 172.21.15.189:0/3755467271 >> v1:172.21.15.26:6789/0 conn(0x7fbc0409df10 legacy=0x7fbc0409d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:46:55.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 10 monclient: _finish_hunting 0 2024-09-16T14:46:55.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 1 monclient: found mon.noname-c 2024-09-16T14:46:55.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:55.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 1 -- 172.21.15.189:0/3755467271 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0409c690 con 0x7fbc0409e740 2024-09-16T14:46:55.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 10 monclient: _finish_auth 0 2024-09-16T14:46:55.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 10 monclient: _check_auth_tickets 2024-09-16T14:46:55.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:55.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 1 -- 172.21.15.189:0/3755467271 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbbfc0019a0 con 0x7fbc0409e740 2024-09-16T14:46:55.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:55.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:55.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc037fe640 10 monclient: dump: 2024-09-16T14:46:55.917 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:55.917 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:55.917 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.917 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.918 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:55.918 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:55.918 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:55.918 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:55.918 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:55.918 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 5 monclient: authenticate success, global_id 94741 2024-09-16T14:46:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.895+0000 7fbc0aebc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc037fe640 1 -- 172.21.15.189:0/3755467271 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbfc001c90 con 0x7fbc0409e740 2024-09-16T14:46:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc037fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc037fe640 1 -- 172.21.15.189:0/3755467271 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbbfc0020a0 con 0x7fbc0409e740 2024-09-16T14:46:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 4 set_mon_vals no callback set 2024-09-16T14:46:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:55.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:55.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc037fe640 10 monclient: dump: 2024-09-16T14:46:55.920 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:55.920 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:55.920 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.920 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.920 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:55.920 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:55.921 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:55.921 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:55.921 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:55.921 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient: shutdown 2024-09-16T14:46:55.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:55.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 1 -- 172.21.15.189:0/3755467271 >> v1:172.21.15.189:6789/0 conn(0x7fbc0409e740 legacy=0x7fbc0413b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:55.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 1 -- 172.21.15.189:0/3755467271 shutdown_connections 2024-09-16T14:46:55.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 1 -- 172.21.15.189:0/3755467271 >> 172.21.15.189:0/3755467271 conn(0x7fbc040a3320 msgr2=0x7fbc0409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:55.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 1 -- 172.21.15.189:0/3755467271 shutdown_connections 2024-09-16T14:46:55.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 1 -- 172.21.15.189:0/3755467271 wait complete. 2024-09-16T14:46:55.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient: build_initial_monmap 2024-09-16T14:46:55.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient: monmap: 2024-09-16T14:46:55.922 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:55.922 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:55.923 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:55.923 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:55.923 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:55.923 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:55.923 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:55.923 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:55.923 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:55.924 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 1 Processor -- start 2024-09-16T14:46:55.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 1 -- start start 2024-09-16T14:46:55.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient: init 2024-09-16T14:46:55.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient(hunting): picked mon.noname-a con 0x7fbc0409df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient(hunting): picked mon.noname-c con 0x7fbc0409e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient(hunting): picked mon.noname-b con 0x7fbc0409e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc041238a0 con 0x7fbc0409df10 2024-09-16T14:46:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0406f9d0 con 0x7fbc0409e740 2024-09-16T14:46:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0406cf90 con 0x7fbc0409e2f0 2024-09-16T14:46:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient(hunting): authenticate will time out at 1198.018001s 2024-09-16T14:46:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc09459640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbc0409e740 0x7fbc04120850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40132/0 (socket says 172.21.15.189:40132) 2024-09-16T14:46:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc09459640 1 -- 172.21.15.189:0/4215032231 learned_addr learned my addr 172.21.15.189:0/4215032231 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 1 -- 172.21.15.189:0/4215032231 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442898454 0 0) 0x7fbc0406f9d0 con 0x7fbc0409e740 2024-09-16T14:46:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 10 monclient(hunting): _negotiate 2024-09-16T14:46:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 10 monclient(hunting): my global_id is 94744 2024-09-16T14:46:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 1 -- 172.21.15.189:0/4215032231 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf0003530 con 0x7fbc0409e740 2024-09-16T14:46:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 1 -- 172.21.15.189:0/4215032231 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2274420220 0 0) 0x7fbbf0003530 con 0x7fbc0409e740 2024-09-16T14:46:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 1 -- 172.21.15.189:0/4215032231 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc0406f9d0 con 0x7fbc0409e740 2024-09-16T14:46:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 1 -- 172.21.15.189:0/4215032231 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbbfc0017c0 con 0x7fbc0409e740 2024-09-16T14:46:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 10 monclient(hunting): dump: 2024-09-16T14:46:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:55.933 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:55.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 1 -- 172.21.15.189:0/4215032231 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3312456725 0 0) 0x7fbc0406f9d0 con 0x7fbc0409e740 2024-09-16T14:46:55.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 1 -- 172.21.15.189:0/4215032231 >> v1:172.21.15.26:6790/0 conn(0x7fbc0409e2f0 legacy=0x7fbc04120140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:55.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 1 -- 172.21.15.189:0/4215032231 >> v1:172.21.15.26:6789/0 conn(0x7fbc0409df10 legacy=0x7fbc0411fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:55.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 10 monclient: _finish_hunting 0 2024-09-16T14:46:55.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 1 monclient: found mon.b 2024-09-16T14:46:55.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:55.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 1 -- 172.21.15.189:0/4215032231 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0406b380 con 0x7fbc0409e740 2024-09-16T14:46:55.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 10 monclient: _finish_auth 0 2024-09-16T14:46:55.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 10 monclient: _check_auth_tickets 2024-09-16T14:46:55.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:55.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 5 monclient: authenticate success, global_id 94744 2024-09-16T14:46:55.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient: _renew_subs 2024-09-16T14:46:55.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 1 -- 172.21.15.189:0/4215032231 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc041235c0 con 0x7fbc0409e740 2024-09-16T14:46:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient: _renew_subs 2024-09-16T14:46:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 1 -- 172.21.15.189:0/4215032231 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc04121540 con 0x7fbc0409e740 2024-09-16T14:46:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 1 -- 172.21.15.189:0/4215032231 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbfc005000 con 0x7fbc0409e740 2024-09-16T14:46:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 1 -- 172.21.15.189:0/4215032231 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbbfc005310 con 0x7fbc0409e740 2024-09-16T14:46:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 10 monclient: dump: 2024-09-16T14:46:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:55.937 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc02ffd640 4 set_mon_vals no callback set 2024-09-16T14:46:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 1 -- 172.21.15.189:0/4215032231 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbbfc0054f0 con 0x7fbc0409e740 2024-09-16T14:46:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc0aebc640 1 -- 172.21.15.189:0/4215032231 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbe4005050 con 0x7fbc0409e740 2024-09-16T14:46:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc017fa640 1 -- 172.21.15.189:0/4215032231 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(102..102 src has 73..102) v4 ==== 5120+0+0 (unknown 2506992667 0 0) 0x7fbbfc043de0 con 0x7fbc0409e740 2024-09-16T14:46:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.899+0000 7fbc08c58640 10 monclient: get_auth_request con 0x7fbbf0036030 auth_method 0 2024-09-16T14:46:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.903+0000 7fbc017fa640 1 -- 172.21.15.189:0/4215032231 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbbfc014de0 con 0x7fbc0409e740 2024-09-16T14:46:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.903+0000 7fbc017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:55.903+0000 7fbc017fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:56.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.063+0000 7fbc0aebc640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T14:46:56.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.063+0000 7fbc0aebc640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T14:46:56.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.063+0000 7fbc0aebc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:56.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.063+0000 7fbc0aebc640 1 -- 172.21.15.189:0/4215032231 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.11"]} v 0) v1 -- 0x7fbbe4005880 con 0x7fbc0409e740 2024-09-16T14:46:56.066 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:46:56.063+0000 7f5c67484640 -1 mon.b@1(leader).osd e102 definitely_dead 0 2024-09-16T14:46:56.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.683+0000 7fbc017fa640 1 -- 172.21.15.189:0/4215032231 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.11"]}]=0 removed osd.11 v103) v1 ==== 87+0+0 (unknown 1647965412 0 0) 0x7fbbfc0149d0 con 0x7fbc0409e740 2024-09-16T14:46:56.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.683+0000 7fbc017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T14:46:56.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.683+0000 7fbc017fa640 10 monclient: _finish_command 2 = system:0 removed osd.11 2024-09-16T14:46:56.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.783+0000 7fbc0aebc640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T14:46:56.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.783+0000 7fbc0aebc640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T14:46:56.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.783+0000 7fbc0aebc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:56.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.783+0000 7fbc0aebc640 1 -- 172.21.15.189:0/4215032231 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.11"]} v 0) v1 -- 0x7fbbe4005300 con 0x7fbc0409e740 2024-09-16T14:46:56.790 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:46:56.783+0000 7f5c67484640 -1 mon.b@1(leader).osd e103 definitely_dead 0 2024-09-16T14:46:56.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.787+0000 7fbc017fa640 1 -- 172.21.15.189:0/4215032231 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.11"]}]=0 osd.11 does not exist. v103) v1 ==== 96+0+0 (unknown 131915146 0 0) 0x7fbbfc014bb0 con 0x7fbc0409e740 2024-09-16T14:46:56.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.787+0000 7fbc017fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T14:46:56.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.787+0000 7fbc017fa640 10 monclient: _finish_command 3 = system:0 osd.11 does not exist. 2024-09-16T14:46:56.791 INFO:tasks.workunit.client.0.smithi189.stderr:osd.11 does not exist. 2024-09-16T14:46:56.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.787+0000 7fbc0aebc640 1 -- 172.21.15.189:0/4215032231 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbbf0036030 legacy=0x7fbbf00384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:56.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.787+0000 7fbc0aebc640 10 monclient: shutdown 2024-09-16T14:46:56.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.787+0000 7fbc0aebc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:56.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.787+0000 7fbc0aebc640 1 -- 172.21.15.189:0/4215032231 >> v1:172.21.15.189:6789/0 conn(0x7fbc0409e740 legacy=0x7fbc04120850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:56.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.787+0000 7fbc0aebc640 1 -- 172.21.15.189:0/4215032231 shutdown_connections 2024-09-16T14:46:56.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.787+0000 7fbc0aebc640 1 -- 172.21.15.189:0/4215032231 >> 172.21.15.189:0/4215032231 conn(0x7fbc040a3320 msgr2=0x7fbc0406e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:56.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.787+0000 7fbc0aebc640 1 -- 172.21.15.189:0/4215032231 shutdown_connections 2024-09-16T14:46:56.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.787+0000 7fbc0aebc640 1 -- 172.21.15.189:0/4215032231 wait complete. 2024-09-16T14:46:56.807 INFO:tasks.workunit.client.0.smithi189.stderr:++ ceph osd create 2024-09-16T14:46:56.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.915+0000 7f07c5830640 10 monclient: get_monmap_and_config 2024-09-16T14:46:56.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.915+0000 7f07c5830640 10 monclient: build_initial_monmap 2024-09-16T14:46:56.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.915+0000 7f07c5830640 10 monclient: monmap: 2024-09-16T14:46:56.920 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:56.920 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:56.920 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:56.920217+0000 2024-09-16T14:46:56.920 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:56.920217+0000 2024-09-16T14:46:56.920 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:56.921 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:56.921 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:56.921 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:56.921 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:56.921 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:56.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.915+0000 7f07c5830640 1 Processor -- start 2024-09-16T14:46:56.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.915+0000 7f07c5830640 1 -- start start 2024-09-16T14:46:56.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.915+0000 7f07c5830640 10 monclient: init 2024-09-16T14:46:56.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:56.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:46:56.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient(hunting): picked mon.noname-a con 0x7f07c00a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:56.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient(hunting): picked mon.noname-b con 0x7f07c00a25f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:56.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient(hunting): picked mon.noname-c con 0x7f07c013b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:56.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c00a30b0 con 0x7f07c00a1b00 2024-09-16T14:46:56.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c00a2e30 con 0x7f07c013b430 2024-09-16T14:46:56.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c009eb50 con 0x7f07c00a25f0 2024-09-16T14:46:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient(hunting): authenticate will time out at 1199.036361s 2024-09-16T14:46:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bf7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f07c013b430 0x7f07c013d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40142/0 (socket says 172.21.15.189:40142) 2024-09-16T14:46:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bf7fe640 1 -- 172.21.15.189:0/687527572 learned_addr learned my addr 172.21.15.189:0/687527572 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 1 -- 172.21.15.189:0/687527572 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794968862 0 0) 0x7f07c00a2e30 con 0x7f07c013b430 2024-09-16T14:46:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 10 monclient(hunting): _negotiate 2024-09-16T14:46:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 10 monclient(hunting): my global_id is 94747 2024-09-16T14:46:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 1 -- 172.21.15.189:0/687527572 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07a8003530 con 0x7f07c013b430 2024-09-16T14:46:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 1 -- 172.21.15.189:0/687527572 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 733420440 0 0) 0x7f07a8003530 con 0x7f07c013b430 2024-09-16T14:46:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 1 -- 172.21.15.189:0/687527572 >> v1:172.21.15.26:6790/0 conn(0x7f07c00a25f0 legacy=0x7f07c0139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 1 -- 172.21.15.189:0/687527572 >> v1:172.21.15.26:6789/0 conn(0x7f07c00a1b00 legacy=0x7f07c00a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:46:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 10 monclient: _finish_hunting 0 2024-09-16T14:46:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 1 monclient: found mon.noname-c 2024-09-16T14:46:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:56.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:56.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 1 -- 172.21.15.189:0/687527572 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c009ee30 con 0x7f07c013b430 2024-09-16T14:46:56.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 10 monclient: _finish_auth 0 2024-09-16T14:46:56.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 10 monclient: _check_auth_tickets 2024-09-16T14:46:56.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:56.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 1 -- 172.21.15.189:0/687527572 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f07b80019a0 con 0x7f07c013b430 2024-09-16T14:46:56.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:56.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:56.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 10 monclient: dump: 2024-09-16T14:46:56.938 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:56.938 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:56.938 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:56.939 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:56.939 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:56.939 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:56.939 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:56.939 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:56.940 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:56.940 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:56.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 5 monclient: authenticate success, global_id 94747 2024-09-16T14:46:56.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:56.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 1 -- 172.21.15.189:0/687527572 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07b8001c90 con 0x7f07c013b430 2024-09-16T14:46:56.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 1 -- 172.21.15.189:0/687527572 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f07b80020a0 con 0x7f07c013b430 2024-09-16T14:46:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bdffb640 10 monclient: dump: 2024-09-16T14:46:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:56.942 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 4 set_mon_vals no callback set 2024-09-16T14:46:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient: shutdown 2024-09-16T14:46:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 1 -- 172.21.15.189:0/687527572 >> v1:172.21.15.189:6789/0 conn(0x7f07c013b430 legacy=0x7f07c013d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 1 -- 172.21.15.189:0/687527572 shutdown_connections 2024-09-16T14:46:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 1 -- 172.21.15.189:0/687527572 >> 172.21.15.189:0/687527572 conn(0x7f07c00a7dd0 msgr2=0x7f07c00a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 1 -- 172.21.15.189:0/687527572 shutdown_connections 2024-09-16T14:46:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 1 -- 172.21.15.189:0/687527572 wait complete. 2024-09-16T14:46:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient: build_initial_monmap 2024-09-16T14:46:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient: monmap: 2024-09-16T14:46:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:56.944 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 1 Processor -- start 2024-09-16T14:46:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 1 -- start start 2024-09-16T14:46:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient: init 2024-09-16T14:46:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:46:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient(hunting): picked mon.noname-c con 0x7f07c00a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient(hunting): picked mon.noname-a con 0x7f07c00a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient(hunting): picked mon.noname-b con 0x7f07c013b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c0120060 con 0x7f07c00a25f0 2024-09-16T14:46:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c00a5ad0 con 0x7f07c013b430 2024-09-16T14:46:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c009e2f0 con 0x7f07c00a1b00 2024-09-16T14:46:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07c5830640 10 monclient(hunting): authenticate will time out at 1199.039424s 2024-09-16T14:46:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bf7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f07c013b430 0x7f07c01d1810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40156/0 (socket says 172.21.15.189:40156) 2024-09-16T14:46:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07bf7fe640 1 -- 172.21.15.189:0/2153873885 learned_addr learned my addr 172.21.15.189:0/2153873885 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448015426 0 0) 0x7f07c00a5ad0 con 0x7f07c013b430 2024-09-16T14:46:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07a7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07a7fff640 10 monclient(hunting): my global_id is 94750 2024-09-16T14:46:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.919+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07ac003530 con 0x7f07c013b430 2024-09-16T14:46:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3739826447 0 0) 0x7f07ac003530 con 0x7f07c013b430 2024-09-16T14:46:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07c00a5ad0 con 0x7f07c013b430 2024-09-16T14:46:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571621939 0 0) 0x7f07c009e2f0 con 0x7f07c00a1b00 2024-09-16T14:46:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:46:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 10 monclient(hunting): my global_id is 114134 2024-09-16T14:46:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07ac003530 con 0x7f07c00a1b00 2024-09-16T14:46:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f07b80017c0 con 0x7f07c013b430 2024-09-16T14:46:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 10 monclient(hunting): dump: 2024-09-16T14:46:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:56.947 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2779967583 0 0) 0x7f07c00a5ad0 con 0x7f07c013b430 2024-09-16T14:46:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 >> v1:172.21.15.26:6790/0 conn(0x7f07c00a1b00 legacy=0x7f07c01c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 >> v1:172.21.15.26:6789/0 conn(0x7f07c00a25f0 legacy=0x7f07c013a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:46:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 1 monclient: found mon.b 2024-09-16T14:46:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c011e650 con 0x7f07c013b430 2024-09-16T14:46:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 10 monclient: _finish_auth 0 2024-09-16T14:46:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:46:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07c5830640 5 monclient: authenticate success, global_id 94750 2024-09-16T14:46:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07c5830640 10 monclient: _renew_subs 2024-09-16T14:46:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07c5830640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07c5830640 1 -- 172.21.15.189:0/2153873885 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c011dd00 con 0x7f07c013b430 2024-09-16T14:46:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07c5830640 10 monclient: _renew_subs 2024-09-16T14:46:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07c5830640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07c5830640 1 -- 172.21.15.189:0/2153873885 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c01ca0f0 con 0x7f07c013b430 2024-09-16T14:46:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07b8005050 con 0x7f07c013b430 2024-09-16T14:46:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f07b8005360 con 0x7f07c013b430 2024-09-16T14:46:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07bd7fa640 4 set_mon_vals no callback set 2024-09-16T14:46:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 10 monclient: dump: 2024-09-16T14:46:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:56.950 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:56.950 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:56.950 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:56.950 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:56.951 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f07b80054c0 con 0x7f07c013b430 2024-09-16T14:46:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07c5830640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07c5830640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07c5830640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07c5830640 1 -- 172.21.15.189:0/2153873885 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a0005050 con 0x7f07c013b430 2024-09-16T14:46:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(103..103 src has 73..103) v4 ==== 5168+0+0 (unknown 2839347669 0 0) 0x7f07b8043d20 con 0x7f07c013b430 2024-09-16T14:46:56.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.923+0000 7f07beffd640 10 monclient: get_auth_request con 0x7f07ac036220 auth_method 0 2024-09-16T14:46:56.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.927+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f07b8014e30 con 0x7f07c013b430 2024-09-16T14:46:56.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.927+0000 7f07a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:56.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:56.927+0000 7f07a7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:57.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:57.099+0000 7f07c5830640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T14:46:57.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:57.099+0000 7f07c5830640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T14:46:57.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:57.099+0000 7f07c5830640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:57.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:57.099+0000 7f07c5830640 1 -- 172.21.15.189:0/2153873885 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f07a0005880 con 0x7f07c013b430 2024-09-16T14:46:57.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:57.687+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v104) v1 ==== 58+0+3 (unknown 3188624585 0 212448812) 0x7f07b80132a0 con 0x7f07c013b430 2024-09-16T14:46:57.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:57.687+0000 7f07a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T14:46:57.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:57.687+0000 7f07a7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:57.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:57.795+0000 7f07c5830640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T14:46:57.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:57.795+0000 7f07c5830640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T14:46:57.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:57.795+0000 7f07c5830640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:57.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:57.795+0000 7f07c5830640 1 -- 172.21.15.189:0/2153873885 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f07a0005370 con 0x7f07c013b430 2024-09-16T14:46:58.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.699+0000 7f07a7fff640 1 -- 172.21.15.189:0/2153873885 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v105) v1 ==== 58+0+3 (unknown 192631357 0 944643765) 0x7f07b8014a20 con 0x7f07c013b430 2024-09-16T14:46:58.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.703+0000 7f07a7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T14:46:58.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.703+0000 7f07a7fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:46:58.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.703+0000 7f07c5830640 1 -- 172.21.15.189:0/2153873885 >> v1:172.21.15.189:6812/3196161582 conn(0x7f07ac036220 legacy=0x7f07ac0386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:58.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.703+0000 7f07c5830640 10 monclient: shutdown 2024-09-16T14:46:58.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.703+0000 7f07c5830640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:58.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.703+0000 7f07c5830640 1 -- 172.21.15.189:0/2153873885 >> v1:172.21.15.189:6789/0 conn(0x7f07c013b430 legacy=0x7f07c01d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:58.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.703+0000 7f07c5830640 1 -- 172.21.15.189:0/2153873885 shutdown_connections 2024-09-16T14:46:58.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.703+0000 7f07c5830640 1 -- 172.21.15.189:0/2153873885 >> 172.21.15.189:0/2153873885 conn(0x7f07c00a7dd0 msgr2=0x7f07c006b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:58.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.703+0000 7f07c5830640 1 -- 172.21.15.189:0/2153873885 shutdown_connections 2024-09-16T14:46:58.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.703+0000 7f07c5830640 1 -- 172.21.15.189:0/2153873885 wait complete. 2024-09-16T14:46:58.725 INFO:tasks.workunit.client.0.smithi189.stderr:+ o4=12 2024-09-16T14:46:58.725 INFO:tasks.workunit.client.0.smithi189.stderr:++ ceph osd create 2024-09-16T14:46:58.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.835+0000 7fd223d53640 10 monclient: get_monmap_and_config 2024-09-16T14:46:58.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.835+0000 7fd223d53640 10 monclient: build_initial_monmap 2024-09-16T14:46:58.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.835+0000 7fd223d53640 10 monclient: monmap: 2024-09-16T14:46:58.842 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:58.842 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:58.842 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:46:58.841286+0000 2024-09-16T14:46:58.842 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:46:58.841286+0000 2024-09-16T14:46:58.842 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:58.842 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:58.843 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:58.843 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:46:58.843 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:46:58.843 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:58.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 1 Processor -- start 2024-09-16T14:46:58.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 1 -- start start 2024-09-16T14:46:58.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient: init 2024-09-16T14:46:58.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:58.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:46:58.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient(hunting): picked mon.noname-b con 0x7fd21c1318a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:58.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient(hunting): picked mon.noname-c con 0x7fd21c134d50 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:58.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient(hunting): picked mon.noname-a con 0x7fd21c13b2b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:58.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd21c09d8a0 con 0x7fd21c13b2b0 2024-09-16T14:46:58.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd21c09ddf0 con 0x7fd21c134d50 2024-09-16T14:46:58.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd21c09e8b0 con 0x7fd21c1318a0 2024-09-16T14:46:58.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:58.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient(hunting): authenticate will time out at 1200.956898s 2024-09-16T14:46:58.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd2212ee640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd21c134d50 0x7fd21c139b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40158/0 (socket says 172.21.15.189:40158) 2024-09-16T14:46:58.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd2212ee640 1 -- 172.21.15.189:0/3380960318 learned_addr learned my addr 172.21.15.189:0/3380960318 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:58.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 1 -- 172.21.15.189:0/3380960318 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947378477 0 0) 0x7fd21c09ddf0 con 0x7fd21c134d50 2024-09-16T14:46:58.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 10 monclient(hunting): _negotiate 2024-09-16T14:46:58.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:58.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:58.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 10 monclient(hunting): my global_id is 94753 2024-09-16T14:46:58.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 1 -- 172.21.15.189:0/3380960318 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd208003530 con 0x7fd21c134d50 2024-09-16T14:46:58.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 1 -- 172.21.15.189:0/3380960318 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3952639397 0 0) 0x7fd208003530 con 0x7fd21c134d50 2024-09-16T14:46:58.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 1 -- 172.21.15.189:0/3380960318 >> v1:172.21.15.26:6790/0 conn(0x7fd21c1318a0 legacy=0x7fd21c131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:58.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 1 -- 172.21.15.189:0/3380960318 >> v1:172.21.15.26:6789/0 conn(0x7fd21c13b2b0 legacy=0x7fd21c13d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:46:58.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 10 monclient: _finish_hunting 0 2024-09-16T14:46:58.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 1 monclient: found mon.noname-c 2024-09-16T14:46:58.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:58.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:46:58.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 1 -- 172.21.15.189:0/3380960318 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21c09eb90 con 0x7fd21c134d50 2024-09-16T14:46:58.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 10 monclient: _finish_auth 0 2024-09-16T14:46:58.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 10 monclient: _check_auth_tickets 2024-09-16T14:46:58.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:58.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 1 -- 172.21.15.189:0/3380960318 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd20c001620 con 0x7fd21c134d50 2024-09-16T14:46:58.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 5 monclient: authenticate success, global_id 94753 2024-09-16T14:46:58.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:58.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:58.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:46:58.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 10 monclient: dump: 2024-09-16T14:46:58.855 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:58.856 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:58.856 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:58.856 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:58.856 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:58.856 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:58.856 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:58.856 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:58.856 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:58.856 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:58.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:46:58.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 1 -- 172.21.15.189:0/3380960318 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd20c001910 con 0x7fd21c134d50 2024-09-16T14:46:58.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:58.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient: get_monmap_and_config success 2024-09-16T14:46:58.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 4 set_mon_vals no callback set 2024-09-16T14:46:58.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 1 -- 172.21.15.189:0/3380960318 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd20c001ca0 con 0x7fd21c134d50 2024-09-16T14:46:58.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:58.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:58.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd220aed640 10 monclient: dump: 2024-09-16T14:46:58.858 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:58.859 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:58.859 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:58.859 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:58.859 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:58.859 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:58.859 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:58.859 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:58.859 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:58.860 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:58.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient: shutdown 2024-09-16T14:46:58.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:46:58.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 1 -- 172.21.15.189:0/3380960318 >> v1:172.21.15.189:6789/0 conn(0x7fd21c134d50 legacy=0x7fd21c139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:58.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 1 -- 172.21.15.189:0/3380960318 shutdown_connections 2024-09-16T14:46:58.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 1 -- 172.21.15.189:0/3380960318 >> 172.21.15.189:0/3380960318 conn(0x7fd21c0a3320 msgr2=0x7fd21c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:46:58.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 1 -- 172.21.15.189:0/3380960318 shutdown_connections 2024-09-16T14:46:58.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 1 -- 172.21.15.189:0/3380960318 wait complete. 2024-09-16T14:46:58.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient: build_initial_monmap 2024-09-16T14:46:58.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient: monmap: 2024-09-16T14:46:58.861 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:46:58.861 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:46:58.861 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:46:58.861 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:46:58.861 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:46:58.861 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:46:58.861 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:46:58.861 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:46:58.861 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:46:58.862 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:58.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 1 Processor -- start 2024-09-16T14:46:58.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 1 -- start start 2024-09-16T14:46:58.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient: init 2024-09-16T14:46:58.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient: _reopen_session rank -1 2024-09-16T14:46:58.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:46:58.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient(hunting): picked mon.noname-b con 0x7fd21c1318a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:46:58.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient(hunting): picked mon.noname-c con 0x7fd21c134d50 addr v1:172.21.15.26:6790/0 2024-09-16T14:46:58.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient(hunting): picked mon.noname-a con 0x7fd21c13b2b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:46:58.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd21c0a1d50 con 0x7fd21c13b2b0 2024-09-16T14:46:58.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd21c13abd0 con 0x7fd21c1318a0 2024-09-16T14:46:58.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd21c06b380 con 0x7fd21c134d50 2024-09-16T14:46:58.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient(hunting): _renew_subs 2024-09-16T14:46:58.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.839+0000 7fd223d53640 10 monclient(hunting): authenticate will time out at 1200.959961s 2024-09-16T14:46:58.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd221aef640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd21c1318a0 0x7fd21c119340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:40162/0 (socket says 172.21.15.189:40162) 2024-09-16T14:46:58.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd221aef640 1 -- 172.21.15.189:0/1836010001 learned_addr learned my addr 172.21.15.189:0/1836010001 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:46:58.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 1 -- 172.21.15.189:0/1836010001 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149948900 0 0) 0x7fd21c13abd0 con 0x7fd21c1318a0 2024-09-16T14:46:58.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 10 monclient(hunting): _negotiate 2024-09-16T14:46:58.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:46:58.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:46:58.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 10 monclient(hunting): my global_id is 94756 2024-09-16T14:46:58.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 1 -- 172.21.15.189:0/1836010001 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd200003530 con 0x7fd21c1318a0 2024-09-16T14:46:58.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 1 -- 172.21.15.189:0/1836010001 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2007710101 0 0) 0x7fd200003530 con 0x7fd21c1318a0 2024-09-16T14:46:58.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 1 -- 172.21.15.189:0/1836010001 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd21c13abd0 con 0x7fd21c1318a0 2024-09-16T14:46:58.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 1 -- 172.21.15.189:0/1836010001 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd218001940 con 0x7fd21c1318a0 2024-09-16T14:46:58.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:58.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:46:58.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 10 monclient(hunting): dump: 2024-09-16T14:46:58.865 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:58.865 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:58.865 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:58.865 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:58.865 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:58.865 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:58.866 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:58.866 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:58.866 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:58.866 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:58.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:46:58.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 1 -- 172.21.15.189:0/1836010001 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1948589312 0 0) 0x7fd21c13abd0 con 0x7fd21c1318a0 2024-09-16T14:46:58.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 1 -- 172.21.15.189:0/1836010001 >> v1:172.21.15.26:6790/0 conn(0x7fd21c134d50 legacy=0x7fd21c119a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:58.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 1 -- 172.21.15.189:0/1836010001 >> v1:172.21.15.26:6789/0 conn(0x7fd21c13b2b0 legacy=0x7fd21c1d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:46:58.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 10 monclient: _finish_hunting 0 2024-09-16T14:46:58.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 1 monclient: found mon.b 2024-09-16T14:46:58.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:46:58.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:58.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 1 -- 172.21.15.189:0/1836010001 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21c11bbb0 con 0x7fd21c1318a0 2024-09-16T14:46:58.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 10 monclient: _finish_auth 0 2024-09-16T14:46:58.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 10 monclient: _check_auth_tickets 2024-09-16T14:46:58.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:46:58.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd223d53640 5 monclient: authenticate success, global_id 94756 2024-09-16T14:46:58.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd223d53640 10 monclient: _renew_subs 2024-09-16T14:46:58.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd223d53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:58.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd223d53640 1 -- 172.21.15.189:0/1836010001 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd21c11a160 con 0x7fd21c1318a0 2024-09-16T14:46:58.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd223d53640 10 monclient: _renew_subs 2024-09-16T14:46:58.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd223d53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:58.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd223d53640 1 -- 172.21.15.189:0/1836010001 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd21c11b8d0 con 0x7fd21c1318a0 2024-09-16T14:46:58.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 1 -- 172.21.15.189:0/1836010001 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd218001c80 con 0x7fd21c1318a0 2024-09-16T14:46:58.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:46:58.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 1 -- 172.21.15.189:0/1836010001 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd218003a70 con 0x7fd21c1318a0 2024-09-16T14:46:58.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:46:58.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:46:58.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd213fff640 4 set_mon_vals no callback set 2024-09-16T14:46:58.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 10 monclient: dump: 2024-09-16T14:46:58.868 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:46:58.869 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:46:58.869 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:58.869 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:46:58.869 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:46:58.869 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:46:58.869 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:46:58.869 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:46:58.869 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:46:58.869 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:46:58.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 1 -- 172.21.15.189:0/1836010001 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd218003bd0 con 0x7fd21c1318a0 2024-09-16T14:46:58.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd223d53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:58.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd223d53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:58.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd223d53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:58.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd223d53640 1 -- 172.21.15.189:0/1836010001 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd204005050 con 0x7fd21c1318a0 2024-09-16T14:46:58.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2127fc640 1 -- 172.21.15.189:0/1836010001 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(105..105 src has 73..105) v4 ==== 5287+0+0 (unknown 2249535202 0 0) 0x7fd218043840 con 0x7fd21c1318a0 2024-09-16T14:46:58.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.843+0000 7fd2212ee640 10 monclient: get_auth_request con 0x7fd200036080 auth_method 0 2024-09-16T14:46:58.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.847+0000 7fd2127fc640 1 -- 172.21.15.189:0/1836010001 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd218014730 con 0x7fd21c1318a0 2024-09-16T14:46:58.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.847+0000 7fd2127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:46:58.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:58.847+0000 7fd2127fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:46:59.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:59.015+0000 7fd223d53640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T14:46:59.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:59.015+0000 7fd223d53640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T14:46:59.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:59.015+0000 7fd223d53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:59.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:59.015+0000 7fd223d53640 1 -- 172.21.15.189:0/1836010001 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fd204005880 con 0x7fd21c1318a0 2024-09-16T14:46:59.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:59.711+0000 7fd2127fc640 1 -- 172.21.15.189:0/1836010001 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v106) v1 ==== 58+0+3 (unknown 3507046352 0 736921282) 0x7fd218012ba0 con 0x7fd21c1318a0 2024-09-16T14:46:59.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:59.711+0000 7fd2127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T14:46:59.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:59.711+0000 7fd2127fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:46:59.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:59.819+0000 7fd223d53640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T14:46:59.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:59.819+0000 7fd223d53640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T14:46:59.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:59.819+0000 7fd223d53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:46:59.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:46:59.819+0000 7fd223d53640 1 -- 172.21.15.189:0/1836010001 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fd204005370 con 0x7fd21c1318a0 2024-09-16T14:47:00.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.711+0000 7fd2127fc640 1 -- 172.21.15.189:0/1836010001 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v107) v1 ==== 58+0+3 (unknown 1685887268 0 1367428999) 0x7fd218014320 con 0x7fd21c1318a0 2024-09-16T14:47:00.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.711+0000 7fd2127fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T14:47:00.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.711+0000 7fd2127fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:00.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.715+0000 7fd223d53640 1 -- 172.21.15.189:0/1836010001 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd200036080 legacy=0x7fd200038540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:00.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.715+0000 7fd223d53640 10 monclient: shutdown 2024-09-16T14:47:00.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.715+0000 7fd223d53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:00.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.715+0000 7fd223d53640 1 -- 172.21.15.189:0/1836010001 >> v1:172.21.15.189:6789/0 conn(0x7fd21c1318a0 legacy=0x7fd21c119340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:00.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.715+0000 7fd223d53640 1 -- 172.21.15.189:0/1836010001 shutdown_connections 2024-09-16T14:47:00.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.715+0000 7fd223d53640 1 -- 172.21.15.189:0/1836010001 >> 172.21.15.189:0/1836010001 conn(0x7fd21c0a3320 msgr2=0x7fd21c133340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:00.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.715+0000 7fd223d53640 1 -- 172.21.15.189:0/1836010001 shutdown_connections 2024-09-16T14:47:00.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.715+0000 7fd223d53640 1 -- 172.21.15.189:0/1836010001 wait complete. 2024-09-16T14:47:00.735 INFO:tasks.workunit.client.0.smithi189.stderr:+ o5=14 2024-09-16T14:47:00.735 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add 12 123 root=default host=foobaz 2024-09-16T14:47:00.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.855+0000 7fc8d3540640 10 monclient: get_monmap_and_config 2024-09-16T14:47:00.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.855+0000 7fc8d3540640 10 monclient: build_initial_monmap 2024-09-16T14:47:00.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.855+0000 7fc8d3540640 10 monclient: monmap: 2024-09-16T14:47:00.861 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:00.861 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:00.861 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:00.860973+0000 2024-09-16T14:47:00.861 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:00.860973+0000 2024-09-16T14:47:00.861 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:00.861 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:00.861 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:00.861 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:00.861 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:00.862 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:00.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 1 Processor -- start 2024-09-16T14:47:00.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 1 -- start start 2024-09-16T14:47:00.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 10 monclient: init 2024-09-16T14:47:00.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:00.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:00.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 10 monclient(hunting): picked mon.noname-b con 0x7fc8cc099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:00.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 10 monclient(hunting): picked mon.noname-c con 0x7fc8cc09c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:00.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 10 monclient(hunting): picked mon.noname-a con 0x7fc8cc09e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:00.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc099e50 con 0x7fc8cc09e6d0 2024-09-16T14:47:00.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc09dbd0 con 0x7fc8cc09c550 2024-09-16T14:47:00.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc09ee20 con 0x7fc8cc099a70 2024-09-16T14:47:00.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:00.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 10 monclient(hunting): authenticate will time out at 1202.977215s 2024-09-16T14:47:00.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d0adb640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc8cc09c550 0x7fc8cc1379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42500/0 (socket says 172.21.15.189:42500) 2024-09-16T14:47:00.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d0adb640 1 -- 172.21.15.189:0/2936036934 learned_addr learned my addr 172.21.15.189:0/2936036934 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:00.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 1 -- 172.21.15.189:0/2936036934 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796149777 0 0) 0x7fc8cc09dbd0 con 0x7fc8cc09c550 2024-09-16T14:47:00.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:00.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:00.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:00.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 10 monclient(hunting): my global_id is 94759 2024-09-16T14:47:00.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 1 -- 172.21.15.189:0/2936036934 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b8003530 con 0x7fc8cc09c550 2024-09-16T14:47:00.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 1 -- 172.21.15.189:0/2936036934 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3079313657 0 0) 0x7fc8b8003530 con 0x7fc8cc09c550 2024-09-16T14:47:00.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 1 -- 172.21.15.189:0/2936036934 >> v1:172.21.15.26:6790/0 conn(0x7fc8cc099a70 legacy=0x7fc8cc09dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:00.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 1 -- 172.21.15.189:0/2936036934 >> v1:172.21.15.26:6789/0 conn(0x7fc8cc09e6d0 legacy=0x7fc8cc13b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:47:00.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:47:00.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 1 monclient: found mon.noname-c 2024-09-16T14:47:00.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:00.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:00.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 1 -- 172.21.15.189:0/2936036934 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc09f100 con 0x7fc8cc09c550 2024-09-16T14:47:00.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 10 monclient: _finish_auth 0 2024-09-16T14:47:00.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:47:00.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:00.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 1 -- 172.21.15.189:0/2936036934 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc8bc001620 con 0x7fc8cc09c550 2024-09-16T14:47:00.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 5 monclient: authenticate success, global_id 94759 2024-09-16T14:47:00.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:00.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:00.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:00.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 10 monclient: dump: 2024-09-16T14:47:00.867 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:00.867 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:00.867 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:00.867 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:00.867 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:00.867 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:00.867 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:00.867 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:00.867 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:00.867 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:00.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:00.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 1 -- 172.21.15.189:0/2936036934 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8bc001910 con 0x7fc8cc09c550 2024-09-16T14:47:00.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:00.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 1 -- 172.21.15.189:0/2936036934 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc8bc001ca0 con 0x7fc8cc09c550 2024-09-16T14:47:00.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:00.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:00.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8c3fff640 10 monclient: dump: 2024-09-16T14:47:00.868 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:00.868 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:00.868 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:00.868 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:00.868 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:00.869 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:00.869 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:00.869 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:00.869 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:00.869 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:00.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:00.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 4 set_mon_vals no callback set 2024-09-16T14:47:00.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 10 monclient: shutdown 2024-09-16T14:47:00.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:00.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 1 -- 172.21.15.189:0/2936036934 >> v1:172.21.15.189:6789/0 conn(0x7fc8cc09c550 legacy=0x7fc8cc1379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:00.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 1 -- 172.21.15.189:0/2936036934 shutdown_connections 2024-09-16T14:47:00.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 1 -- 172.21.15.189:0/2936036934 >> 172.21.15.189:0/2936036934 conn(0x7fc8cc0a3380 msgr2=0x7fc8cc09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:00.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 1 -- 172.21.15.189:0/2936036934 shutdown_connections 2024-09-16T14:47:00.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 1 -- 172.21.15.189:0/2936036934 wait complete. 2024-09-16T14:47:00.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 10 monclient: build_initial_monmap 2024-09-16T14:47:00.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 10 monclient: monmap: 2024-09-16T14:47:00.873 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:00.873 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:00.873 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:00.874 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:00.874 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:00.874 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:00.874 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:00.874 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:00.874 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:00.874 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:00.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 1 Processor -- start 2024-09-16T14:47:00.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 1 -- start start 2024-09-16T14:47:00.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.859+0000 7fc8d3540640 10 monclient: init 2024-09-16T14:47:00.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:00.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:00.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 10 monclient(hunting): picked mon.noname-c con 0x7fc8cc099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:00.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 10 monclient(hunting): picked mon.noname-b con 0x7fc8cc09c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:00.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 10 monclient(hunting): picked mon.noname-a con 0x7fc8cc09e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:00.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc09cf20 con 0x7fc8cc09e6d0 2024-09-16T14:47:00.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc06cdd0 con 0x7fc8cc09c550 2024-09-16T14:47:00.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc06b320 con 0x7fc8cc099a70 2024-09-16T14:47:00.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:00.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 10 monclient(hunting): authenticate will time out at 1202.980303s 2024-09-16T14:47:00.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d0adb640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc8cc09c550 0x7fc8cc09a900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42508/0 (socket says 172.21.15.189:42508) 2024-09-16T14:47:00.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d0adb640 1 -- 172.21.15.189:0/4252826934 learned_addr learned my addr 172.21.15.189:0/4252826934 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:00.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 1 -- 172.21.15.189:0/4252826934 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 386876411 0 0) 0x7fc8cc06cdd0 con 0x7fc8cc09c550 2024-09-16T14:47:00.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:00.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:00.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:00.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 10 monclient(hunting): my global_id is 94762 2024-09-16T14:47:00.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 1 -- 172.21.15.189:0/4252826934 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b0003530 con 0x7fc8cc09c550 2024-09-16T14:47:00.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 1 -- 172.21.15.189:0/4252826934 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2040299066 0 0) 0x7fc8b0003530 con 0x7fc8cc09c550 2024-09-16T14:47:00.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 1 -- 172.21.15.189:0/4252826934 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8cc06cdd0 con 0x7fc8cc09c550 2024-09-16T14:47:00.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 1 -- 172.21.15.189:0/4252826934 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc8bc0015e0 con 0x7fc8cc09c550 2024-09-16T14:47:00.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:00.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:00.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 10 monclient(hunting): dump: 2024-09-16T14:47:00.877 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:00.877 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:00.877 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:00.877 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:00.877 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:00.877 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:00.877 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:00.877 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:00.877 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:00.877 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:00.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:00.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 1 -- 172.21.15.189:0/4252826934 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 41420274 0 0) 0x7fc8cc06cdd0 con 0x7fc8cc09c550 2024-09-16T14:47:00.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 1 -- 172.21.15.189:0/4252826934 >> v1:172.21.15.26:6790/0 conn(0x7fc8cc099a70 legacy=0x7fc8cc09a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:00.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 1 -- 172.21.15.189:0/4252826934 >> v1:172.21.15.26:6789/0 conn(0x7fc8cc09e6d0 legacy=0x7fc8cc1d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:00.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:47:00.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 1 monclient: found mon.b 2024-09-16T14:47:00.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:00.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:00.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 1 -- 172.21.15.189:0/4252826934 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc09b5c0 con 0x7fc8cc09c550 2024-09-16T14:47:00.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:47:00.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:00.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:00.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 5 monclient: authenticate success, global_id 94762 2024-09-16T14:47:00.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 10 monclient: _renew_subs 2024-09-16T14:47:00.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:00.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 1 -- 172.21.15.189:0/4252826934 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8cc13b960 con 0x7fc8cc09c550 2024-09-16T14:47:00.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 10 monclient: _renew_subs 2024-09-16T14:47:00.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:00.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 1 -- 172.21.15.189:0/4252826934 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8cc13b620 con 0x7fc8cc09c550 2024-09-16T14:47:00.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 1 -- 172.21.15.189:0/4252826934 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8bc004c00 con 0x7fc8cc09c550 2024-09-16T14:47:00.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:00.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 1 -- 172.21.15.189:0/4252826934 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc8bc004f90 con 0x7fc8cc09c550 2024-09-16T14:47:00.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:00.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:00.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c37fe640 4 set_mon_vals no callback set 2024-09-16T14:47:00.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 10 monclient: dump: 2024-09-16T14:47:00.879 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:00.879 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:00.880 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:00.880 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:00.880 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:00.880 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:00.880 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:00.880 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:00.880 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:00.880 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:00.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:00.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:00.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:00.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 1 -- 172.21.15.189:0/4252826934 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc8bc005170 con 0x7fc8cc09c550 2024-09-16T14:47:00.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d3540640 1 -- 172.21.15.189:0/4252826934 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ac005050 con 0x7fc8cc09c550 2024-09-16T14:47:00.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8c1ffb640 1 -- 172.21.15.189:0/4252826934 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(107..107 src has 73..107) v4 ==== 5525+0+0 (unknown 3170555406 0 0) 0x7fc8bc043ce0 con 0x7fc8cc09c550 2024-09-16T14:47:00.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.863+0000 7fc8d12dc640 10 monclient: get_auth_request con 0x7fc8b0036010 auth_method 0 2024-09-16T14:47:00.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.867+0000 7fc8c1ffb640 1 -- 172.21.15.189:0/4252826934 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc8bc014ba0 con 0x7fc8cc09c550 2024-09-16T14:47:00.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.867+0000 7fc8c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:00.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:00.867+0000 7fc8c1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:01.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.027+0000 7fc8d3540640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T14:47:01.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.027+0000 7fc8d3540640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T14:47:01.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.027+0000 7fc8d3540640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:01.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.027+0000 7fc8d3540640 1 -- 172.21.15.189:0/4252826934 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7fc8ac005880 con 0x7fc8cc09c550 2024-09-16T14:47:01.074 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T14:47:01.074 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T14:47:01.074 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:47:01.075 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:47:01.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdb0e02640 10 monclient: get_monmap_and_config 2024-09-16T14:47:01.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdb0e02640 10 monclient: build_initial_monmap 2024-09-16T14:47:01.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdb0e02640 10 monclient: monmap: 2024-09-16T14:47:01.215 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:01.215 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.215 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:01.215203+0000 2024-09-16T14:47:01.215 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:01.215203+0000 2024-09-16T14:47:01.215 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:01.215 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.216 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:01.216 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:01.216 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:01.216 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdb0e02640 1 Processor -- start 2024-09-16T14:47:01.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdb0e02640 1 -- start start 2024-09-16T14:47:01.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdb0e02640 10 monclient: init 2024-09-16T14:47:01.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdb0e02640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:01.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdb0e02640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:01.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdb0e02640 10 monclient(hunting): picked mon.noname-c con 0x7fcdac09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:01.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdb0e02640 10 monclient(hunting): picked mon.noname-b con 0x7fcdac132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:01.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdb0e02640 10 monclient(hunting): picked mon.noname-a con 0x7fcdac138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:01.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdb0e02640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdac09d260 con 0x7fcdac138ff0 2024-09-16T14:47:01.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdb0e02640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdac132a80 con 0x7fcdac09cb80 2024-09-16T14:47:01.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdb0e02640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdac09cf60 con 0x7fcdac132650 2024-09-16T14:47:01.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdb0e02640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:01.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdb0e02640 10 monclient(hunting): authenticate will time out at 1245.335295s 2024-09-16T14:47:01.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdaad9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fcdac138ff0 0x7fcdac13b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38458/0 (socket says 172.21.15.26:38458) 2024-09-16T14:47:01.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcda9d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fcdac132650 0x7fcdac1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35366/0 (socket says 172.21.15.26:35366) 2024-09-16T14:47:01.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcdaad9d640 1 -- 172.21.15.26:0/2733912099 learned_addr learned my addr 172.21.15.26:0/2733912099 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:01.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcda959a640 1 -- 172.21.15.26:0/2733912099 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675816347 0 0) 0x7fcdac09cf60 con 0x7fcdac132650 2024-09-16T14:47:01.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcda959a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcda959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcda959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcda959a640 10 monclient(hunting): my global_id is 114149 2024-09-16T14:47:01.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcda959a640 1 -- 172.21.15.26:0/2733912099 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd98003530 con 0x7fcdac132650 2024-09-16T14:47:01.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcda959a640 1 -- 172.21.15.26:0/2733912099 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524946514 0 0) 0x7fcdac09d260 con 0x7fcdac138ff0 2024-09-16T14:47:01.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcda959a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcda959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcda959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcda959a640 10 monclient(hunting): my global_id is 104352 2024-09-16T14:47:01.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.211+0000 7fcda959a640 1 -- 172.21.15.26:0/2733912099 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdac09cf60 con 0x7fcdac138ff0 2024-09-16T14:47:01.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 1 -- 172.21.15.26:0/2733912099 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 734942430 0 0) 0x7fcd98003530 con 0x7fcdac132650 2024-09-16T14:47:01.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 1 -- 172.21.15.26:0/2733912099 >> v1:172.21.15.189:6789/0 conn(0x7fcdac09cb80 legacy=0x7fcdac0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 1 -- 172.21.15.26:0/2733912099 >> v1:172.21.15.26:6789/0 conn(0x7fcdac138ff0 legacy=0x7fcdac13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 10 monclient: _finish_hunting 0 2024-09-16T14:47:01.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 1 monclient: found mon.noname-b 2024-09-16T14:47:01.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:01.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:47:01.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 1 -- 172.21.15.26:0/2733912099 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac09f2d0 con 0x7fcdac132650 2024-09-16T14:47:01.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 10 monclient: _finish_auth 0 2024-09-16T14:47:01.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 10 monclient: _check_auth_tickets 2024-09-16T14:47:01.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:01.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 1 -- 172.21.15.26:0/2733912099 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcd94001590 con 0x7fcdac132650 2024-09-16T14:47:01.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 5 monclient: authenticate success, global_id 114149 2024-09-16T14:47:01.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:01.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:01.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 10 monclient: dump: 2024-09-16T14:47:01.226 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.226 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.226 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.226 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.226 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.226 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.226 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.226 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.226 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.226 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:01.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 1 -- 172.21.15.26:0/2733912099 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd94001770 con 0x7fcdac132650 2024-09-16T14:47:01.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:01.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:01.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 1 -- 172.21.15.26:0/2733912099 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcd94001b40 con 0x7fcdac132650 2024-09-16T14:47:01.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 4 set_mon_vals no callback set 2024-09-16T14:47:01.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:01.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda959a640 10 monclient: dump: 2024-09-16T14:47:01.227 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.227 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.227 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.227 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.227 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.227 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.227 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.227 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.228 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.228 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 10 monclient: shutdown 2024-09-16T14:47:01.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:01.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 1 -- 172.21.15.26:0/2733912099 >> v1:172.21.15.26:6790/0 conn(0x7fcdac132650 legacy=0x7fcdac1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 1 -- 172.21.15.26:0/2733912099 shutdown_connections 2024-09-16T14:47:01.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 1 -- 172.21.15.26:0/2733912099 >> 172.21.15.26:0/2733912099 conn(0x7fcdac09fb50 msgr2=0x7fcdac09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:01.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 1 -- 172.21.15.26:0/2733912099 shutdown_connections 2024-09-16T14:47:01.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 1 -- 172.21.15.26:0/2733912099 wait complete. 2024-09-16T14:47:01.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 10 monclient: build_initial_monmap 2024-09-16T14:47:01.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 10 monclient: monmap: 2024-09-16T14:47:01.228 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:01.228 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:01.229 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:01.229 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:01.229 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:01.229 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:01.229 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:01.229 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:01.229 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:01.229 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 1 Processor -- start 2024-09-16T14:47:01.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 1 -- start start 2024-09-16T14:47:01.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 10 monclient: init 2024-09-16T14:47:01.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:01.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:01.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 10 monclient(hunting): picked mon.noname-a con 0x7fcdac09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:01.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 10 monclient(hunting): picked mon.noname-b con 0x7fcdac132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:01.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 10 monclient(hunting): picked mon.noname-c con 0x7fcdac138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:01.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdac138150 con 0x7fcdac09cb80 2024-09-16T14:47:01.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdac06e450 con 0x7fcdac132650 2024-09-16T14:47:01.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdac071590 con 0x7fcdac138ff0 2024-09-16T14:47:01.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:01.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 10 monclient(hunting): authenticate will time out at 1245.338341s 2024-09-16T14:47:01.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdaad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fcdac138ff0 0x7fcdac1cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35378/0 (socket says 172.21.15.26:35378) 2024-09-16T14:47:01.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdaad9d640 1 -- 172.21.15.26:0/1898724831 learned_addr learned my addr 172.21.15.26:0/1898724831 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:01.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdaa59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fcdac09cb80 0x7fcdac06d630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38462/0 (socket says 172.21.15.26:38462) 2024-09-16T14:47:01.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946022013 0 0) 0x7fcdac071590 con 0x7fcdac138ff0 2024-09-16T14:47:01.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): my global_id is 114152 2024-09-16T14:47:01.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd8c003530 con 0x7fcdac138ff0 2024-09-16T14:47:01.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700703733 0 0) 0x7fcdac132a80 con 0x7fcdac09cb80 2024-09-16T14:47:01.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): my global_id is 104355 2024-09-16T14:47:01.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdac071590 con 0x7fcdac09cb80 2024-09-16T14:47:01.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2350939896 0 0) 0x7fcd8c003530 con 0x7fcdac138ff0 2024-09-16T14:47:01.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdac132a80 con 0x7fcdac138ff0 2024-09-16T14:47:01.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294156124 0 0) 0x7fcdac06e450 con 0x7fcdac132650 2024-09-16T14:47:01.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): my global_id is 94768 2024-09-16T14:47:01.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd8c003530 con 0x7fcdac132650 2024-09-16T14:47:01.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcda4001a00 con 0x7fcdac138ff0 2024-09-16T14:47:01.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:01.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): dump: 2024-09-16T14:47:01.232 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.232 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.232 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.232 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.233 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.233 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.233 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.233 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.233 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.233 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:01.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1191618194 0 0) 0x7fcdac071590 con 0x7fcdac09cb80 2024-09-16T14:47:01.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdac06e450 con 0x7fcdac09cb80 2024-09-16T14:47:01.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcd9c001980 con 0x7fcdac09cb80 2024-09-16T14:47:01.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:01.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient(hunting): dump: 2024-09-16T14:47:01.234 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.234 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.234 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.234 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.234 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.234 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.234 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.234 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.234 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.234 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2322955405 0 0) 0x7fcdac132a80 con 0x7fcdac138ff0 2024-09-16T14:47:01.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 >> v1:172.21.15.189:6789/0 conn(0x7fcdac132650 legacy=0x7fcdac06dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 >> v1:172.21.15.26:6789/0 conn(0x7fcdac09cb80 legacy=0x7fcdac06d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:01.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 monclient: found mon.c 2024-09-16T14:47:01.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:01.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:01.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac13c400 con 0x7fcdac138ff0 2024-09-16T14:47:01.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:47:01.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:01.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:01.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 5 monclient: authenticate success, global_id 114152 2024-09-16T14:47:01.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 10 monclient: _renew_subs 2024-09-16T14:47:01.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:01.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 1 -- 172.21.15.26:0/1898724831 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdac070fd0 con 0x7fcdac138ff0 2024-09-16T14:47:01.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 10 monclient: _renew_subs 2024-09-16T14:47:01.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:01.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcdb0e02640 1 -- 172.21.15.26:0/1898724831 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdac06fb80 con 0x7fcdac138ff0 2024-09-16T14:47:01.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcda40047f0 con 0x7fcdac138ff0 2024-09-16T14:47:01.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:01.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcda4004b00 con 0x7fcdac138ff0 2024-09-16T14:47:01.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:01.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda2ffd640 10 monclient: dump: 2024-09-16T14:47:01.236 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.236 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.236 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.237 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.237 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.237 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.237 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.237 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.237 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.237 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.215+0000 7fcda8d99640 4 set_mon_vals no callback set 2024-09-16T14:47:01.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.219+0000 7fcdb0e02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:01.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.219+0000 7fcdb0e02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:01.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.219+0000 7fcdb0e02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:01.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.219+0000 7fcdb0e02640 1 -- 172.21.15.26:0/1898724831 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd90005050 con 0x7fcdac138ff0 2024-09-16T14:47:01.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.219+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fcda400f570 con 0x7fcdac138ff0 2024-09-16T14:47:01.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.219+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(107..107 src has 73..107) v4 ==== 5525+0+0 (unknown 3170555406 0 0) 0x7fcda4043880 con 0x7fcdac138ff0 2024-09-16T14:47:01.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.219+0000 7fcdaa59c640 10 monclient: get_auth_request con 0x7fcd8c036490 auth_method 0 2024-09-16T14:47:01.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.223+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fcda4010ac0 con 0x7fcdac138ff0 2024-09-16T14:47:01.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.223+0000 7fcda2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:01.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.223+0000 7fcda2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:01.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.443+0000 7fcdb0e02640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:47:01.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.443+0000 7fcdb0e02640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:01.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.443+0000 7fcdb0e02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:01.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.443+0000 7fcdb0e02640 1 -- 172.21.15.26:0/1898724831 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcd90005880 con 0x7fcdac138ff0 2024-09-16T14:47:01.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.443+0000 7fcda2ffd640 1 -- 172.21.15.26:0/1898724831 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1552464170) 0x7fcda4012ba0 con 0x7fcdac138ff0 2024-09-16T14:47:01.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.443+0000 7fcda2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:01.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.443+0000 7fcda2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:01.449 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:47:01.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.443+0000 7fcdb0e02640 1 -- 172.21.15.26:0/1898724831 >> v1:172.21.15.189:6812/3196161582 conn(0x7fcd8c036490 legacy=0x7fcd8c038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.443+0000 7fcdb0e02640 10 monclient: shutdown 2024-09-16T14:47:01.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.443+0000 7fcdb0e02640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:01.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.443+0000 7fcdb0e02640 1 -- 172.21.15.26:0/1898724831 >> v1:172.21.15.26:6790/0 conn(0x7fcdac138ff0 legacy=0x7fcdac1cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.443+0000 7fcdb0e02640 1 -- 172.21.15.26:0/1898724831 shutdown_connections 2024-09-16T14:47:01.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.443+0000 7fcdb0e02640 1 -- 172.21.15.26:0/1898724831 >> 172.21.15.26:0/1898724831 conn(0x7fcdac09fb50 msgr2=0x7fcdac131ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:01.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.443+0000 7fcdb0e02640 1 -- 172.21.15.26:0/1898724831 shutdown_connections 2024-09-16T14:47:01.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.443+0000 7fcdb0e02640 1 -- 172.21.15.26:0/1898724831 wait complete. 2024-09-16T14:47:01.465 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:47:01.465 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:47:01.465 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:47:01.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.579+0000 7f9681d25640 10 monclient: get_monmap_and_config 2024-09-16T14:47:01.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.579+0000 7f9681d25640 10 monclient: build_initial_monmap 2024-09-16T14:47:01.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.579+0000 7f9681d25640 10 monclient: monmap: 2024-09-16T14:47:01.585 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:01.586 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.586 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:01.585383+0000 2024-09-16T14:47:01.586 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:01.585383+0000 2024-09-16T14:47:01.586 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:01.586 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.586 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:01.586 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:01.586 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:01.586 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 1 Processor -- start 2024-09-16T14:47:01.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 1 -- start start 2024-09-16T14:47:01.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 10 monclient: init 2024-09-16T14:47:01.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:01.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:01.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 10 monclient(hunting): picked mon.noname-a con 0x7f967c09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:01.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 10 monclient(hunting): picked mon.noname-c con 0x7f967c132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:01.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 10 monclient(hunting): picked mon.noname-b con 0x7f967c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:01.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967c09d260 con 0x7f967c09cb80 2024-09-16T14:47:01.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967c132a80 con 0x7f967c132650 2024-09-16T14:47:01.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967c09cf60 con 0x7f967c138ff0 2024-09-16T14:47:01.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:01.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 10 monclient(hunting): authenticate will time out at 1245.705454s 2024-09-16T14:47:01.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f967c09cb80 0x7f967c0717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38468/0 (socket says 172.21.15.26:38468) 2024-09-16T14:47:01.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967b7fe640 1 -- 172.21.15.26:0/2206146631 learned_addr learned my addr 172.21.15.26:0/2206146631 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:01.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 1 -- 172.21.15.26:0/2206146631 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962374236 0 0) 0x7f967c09d260 con 0x7f967c09cb80 2024-09-16T14:47:01.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient(hunting): my global_id is 104358 2024-09-16T14:47:01.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 1 -- 172.21.15.26:0/2206146631 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9668003530 con 0x7f967c09cb80 2024-09-16T14:47:01.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 1 -- 172.21.15.26:0/2206146631 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617970319 0 0) 0x7f967c09cf60 con 0x7f967c138ff0 2024-09-16T14:47:01.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient(hunting): my global_id is 114155 2024-09-16T14:47:01.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 1 -- 172.21.15.26:0/2206146631 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f967c09d260 con 0x7f967c138ff0 2024-09-16T14:47:01.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 1 -- 172.21.15.26:0/2206146631 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4131742466 0 0) 0x7f9668003530 con 0x7f967c09cb80 2024-09-16T14:47:01.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 1 -- 172.21.15.26:0/2206146631 >> v1:172.21.15.26:6790/0 conn(0x7f967c138ff0 legacy=0x7f967c13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 1 -- 172.21.15.26:0/2206146631 >> v1:172.21.15.189:6789/0 conn(0x7f967c132650 legacy=0x7f967c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:01.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 1 monclient: found mon.noname-a 2024-09-16T14:47:01.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:01.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:47:01.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 1 -- 172.21.15.26:0/2206146631 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c09f2d0 con 0x7f967c09cb80 2024-09-16T14:47:01.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:01.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 1 -- 172.21.15.26:0/2206146631 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f966c0019a0 con 0x7f967c09cb80 2024-09-16T14:47:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient: dump: 2024-09-16T14:47:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.598 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.599 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 5 monclient: authenticate success, global_id 104358 2024-09-16T14:47:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 1 -- 172.21.15.26:0/2206146631 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f966c001bd0 con 0x7f967c09cb80 2024-09-16T14:47:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 1 -- 172.21.15.26:0/2206146631 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f966c001f60 con 0x7f967c09cb80 2024-09-16T14:47:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 4 set_mon_vals no callback set 2024-09-16T14:47:01.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f967a7fc640 10 monclient: dump: 2024-09-16T14:47:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.600 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 10 monclient: shutdown 2024-09-16T14:47:01.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:01.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 1 -- 172.21.15.26:0/2206146631 >> v1:172.21.15.26:6789/0 conn(0x7f967c09cb80 legacy=0x7f967c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 1 -- 172.21.15.26:0/2206146631 shutdown_connections 2024-09-16T14:47:01.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 1 -- 172.21.15.26:0/2206146631 >> 172.21.15.26:0/2206146631 conn(0x7f967c09fb50 msgr2=0x7f967c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:01.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 1 -- 172.21.15.26:0/2206146631 shutdown_connections 2024-09-16T14:47:01.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 1 -- 172.21.15.26:0/2206146631 wait complete. 2024-09-16T14:47:01.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 10 monclient: build_initial_monmap 2024-09-16T14:47:01.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 10 monclient: monmap: 2024-09-16T14:47:01.601 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:01.601 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:01.601 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:01.601 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:01.602 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:01.602 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:01.602 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:01.602 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:01.602 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:01.602 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.583+0000 7f9681d25640 1 Processor -- start 2024-09-16T14:47:01.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 1 -- start start 2024-09-16T14:47:01.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 10 monclient: init 2024-09-16T14:47:01.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:01.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:01.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 10 monclient(hunting): picked mon.noname-a con 0x7f967c09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:01.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 10 monclient(hunting): picked mon.noname-b con 0x7f967c132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:01.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 10 monclient(hunting): picked mon.noname-c con 0x7f967c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:01.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967c13c520 con 0x7f967c09cb80 2024-09-16T14:47:01.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967c0701b0 con 0x7f967c132650 2024-09-16T14:47:01.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967c1320c0 con 0x7f967c138ff0 2024-09-16T14:47:01.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:01.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 10 monclient(hunting): authenticate will time out at 1245.708578s 2024-09-16T14:47:01.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f967b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f967c09cb80 0x7f967c06d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38482/0 (socket says 172.21.15.26:38482) 2024-09-16T14:47:01.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f967b7fe640 1 -- 172.21.15.26:0/4150067399 learned_addr learned my addr 172.21.15.26:0/4150067399 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:01.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631288222 0 0) 0x7f967c13c520 con 0x7f967c09cb80 2024-09-16T14:47:01.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): my global_id is 104361 2024-09-16T14:47:01.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f965c003530 con 0x7f967c09cb80 2024-09-16T14:47:01.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442940951 0 0) 0x7f967c1320c0 con 0x7f967c138ff0 2024-09-16T14:47:01.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): my global_id is 114158 2024-09-16T14:47:01.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f967c13c520 con 0x7f967c138ff0 2024-09-16T14:47:01.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1526158551 0 0) 0x7f965c003530 con 0x7f967c09cb80 2024-09-16T14:47:01.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f967c1320c0 con 0x7f967c09cb80 2024-09-16T14:47:01.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f966c001fa0 con 0x7f967c09cb80 2024-09-16T14:47:01.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:01.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): dump: 2024-09-16T14:47:01.605 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.605 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.605 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.605 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.605 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.605 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.605 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.606 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.606 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.606 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:01.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2710013715 0 0) 0x7f967c13c520 con 0x7f967c138ff0 2024-09-16T14:47:01.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f965c003530 con 0x7f967c138ff0 2024-09-16T14:47:01.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9674001920 con 0x7f967c138ff0 2024-09-16T14:47:01.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:01.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): dump: 2024-09-16T14:47:01.606 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.606 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.606 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.606 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.607 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565927304 0 0) 0x7f967c0701b0 con 0x7f967c132650 2024-09-16T14:47:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): my global_id is 94774 2024-09-16T14:47:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f967c13c520 con 0x7f967c132650 2024-09-16T14:47:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1350110308 0 0) 0x7f967c1320c0 con 0x7f967c09cb80 2024-09-16T14:47:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 >> v1:172.21.15.26:6790/0 conn(0x7f967c138ff0 legacy=0x7f967c06e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 >> v1:172.21.15.189:6789/0 conn(0x7f967c132650 legacy=0x7f967c06ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient: _finish_hunting 0 2024-09-16T14:47:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 monclient: found mon.a 2024-09-16T14:47:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c06f2b0 con 0x7f967c09cb80 2024-09-16T14:47:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient: _finish_auth 0 2024-09-16T14:47:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient: _check_auth_tickets 2024-09-16T14:47:01.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 5 monclient: authenticate success, global_id 104361 2024-09-16T14:47:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 10 monclient: _renew_subs 2024-09-16T14:47:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 1 -- 172.21.15.26:0/4150067399 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967c06fc00 con 0x7f967c09cb80 2024-09-16T14:47:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 10 monclient: _renew_subs 2024-09-16T14:47:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 1 -- 172.21.15.26:0/4150067399 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f967c1d0b70 con 0x7f967c09cb80 2024-09-16T14:47:01.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f966c001540 con 0x7f967c09cb80 2024-09-16T14:47:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f966c005040 con 0x7f967c09cb80 2024-09-16T14:47:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9679ffb640 4 set_mon_vals no callback set 2024-09-16T14:47:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient: dump: 2024-09-16T14:47:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.611 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.612 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9681d25640 1 -- 172.21.15.26:0/4150067399 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9660006590 con 0x7f9660002830 2024-09-16T14:47:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f966c00fab0 con 0x7f967c09cb80 2024-09-16T14:47:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(107..107 src has 73..107) v4 ==== 5525+0+0 (unknown 3170555406 0 0) 0x7f966c043e00 con 0x7f967c09cb80 2024-09-16T14:47:01.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995211850 0 0) 0x7f9660006590 con 0x7f9660002830 2024-09-16T14:47:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient(hunting): my global_id is 104364 2024-09-16T14:47:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f967c1320c0 con 0x7f9660002830 2024-09-16T14:47:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f967bfff640 10 monclient: get_auth_request con 0x7f965c036390 auth_method 0 2024-09-16T14:47:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1591663241 0 0) 0x7f967c1320c0 con 0x7f9660002830 2024-09-16T14:47:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9660006590 con 0x7f9660002830 2024-09-16T14:47:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f96640017a0 con 0x7f9660002830 2024-09-16T14:47:01.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:01.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient: dump: 2024-09-16T14:47:01.614 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.614 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.615 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.615 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.615 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.615 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.615 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.615 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.615 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.615 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3361520350 0 0) 0x7f9660006590 con 0x7f9660002830 2024-09-16T14:47:01.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f966000a0e0 con 0x7f9660002830 2024-09-16T14:47:01.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f9664001be0 con 0x7f9660002830 2024-09-16T14:47:01.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:01.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:01.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 >> v1:172.21.15.26:6789/0 conn(0x7f9660002830 legacy=0x7f9660004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.587+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 >> v1:172.21.15.26:6789/0 conn(0x7f9660002830 legacy=0x7f9660004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:01.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9681d25640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:47:01.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9681d25640 1 -- 172.21.15.26:0/4150067399 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9660009d80 con 0x7f96600090a0 2024-09-16T14:47:01.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057752763 0 0) 0x7f9660009d80 con 0x7f96600090a0 2024-09-16T14:47:01.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 10 monclient(hunting): my global_id is 104367 2024-09-16T14:47:01.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9660006590 con 0x7f96600090a0 2024-09-16T14:47:01.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2301650032 0 0) 0x7f9660006590 con 0x7f96600090a0 2024-09-16T14:47:01.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9660009d80 con 0x7f96600090a0 2024-09-16T14:47:01.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9664003a50 con 0x7f96600090a0 2024-09-16T14:47:01.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:01.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 10 monclient: dump: 2024-09-16T14:47:01.618 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.619 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.619 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.619 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.619 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.619 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.619 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.619 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.619 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.619 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2861851500 0 0) 0x7f9660009d80 con 0x7f96600090a0 2024-09-16T14:47:01.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f966000c290 con 0x7f96600090a0 2024-09-16T14:47:01.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 3094095662) 0x7f9664001960 con 0x7f96600090a0 2024-09-16T14:47:01.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:47:01.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:01.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 >> v1:172.21.15.26:6789/0 conn(0x7f96600090a0 legacy=0x7f9660009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9673fff640 1 -- 172.21.15.26:0/4150067399 >> v1:172.21.15.26:6789/0 conn(0x7f96600090a0 legacy=0x7f9660009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:01.620 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:47:01.620 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:01.620 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:01.620 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:47:01.620 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 44, 2024-09-16T14:47:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:47:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:47:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:47:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:47:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:47:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:47:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:47:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:01.621 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:47:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:47:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:01.622 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:47:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:47:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:47:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:47:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:47:01.623 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:47:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:47:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:47:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:47:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:47:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:47:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:47:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:47:01.624 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:47:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:47:01.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:47:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:47:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:47:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:47:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:01.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:47:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:47:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:47:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:01.628 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:47:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:01.629 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:47:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:47:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:47:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:47:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:47:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:47:01.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:47:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:47:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:47:01.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.633 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:47:01.633 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:47:01.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9681d25640 1 -- 172.21.15.26:0/4150067399 >> v1:172.21.15.189:6812/3196161582 conn(0x7f965c036390 legacy=0x7f965c038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9681d25640 10 monclient: shutdown 2024-09-16T14:47:01.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9681d25640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:01.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9681d25640 1 -- 172.21.15.26:0/4150067399 >> v1:172.21.15.26:6789/0 conn(0x7f967c09cb80 legacy=0x7f967c06d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f967bfff640 1 -- 172.21.15.26:0/4150067399 reap_dead start 2024-09-16T14:47:01.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f967bfff640 1 -- 172.21.15.26:0/4150067399 reap_dead start 2024-09-16T14:47:01.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9681d25640 1 -- 172.21.15.26:0/4150067399 shutdown_connections 2024-09-16T14:47:01.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9681d25640 1 -- 172.21.15.26:0/4150067399 >> 172.21.15.26:0/4150067399 conn(0x7f967c09fb50 msgr2=0x7f967c130890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:01.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9681d25640 1 -- 172.21.15.26:0/4150067399 shutdown_connections 2024-09-16T14:47:01.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.595+0000 7f9681d25640 1 -- 172.21.15.26:0/4150067399 wait complete. 2024-09-16T14:47:01.637 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:47:01.637 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:47:01.637 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:47:01.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.727+0000 7fc8c1ffb640 1 -- 172.21.15.189:0/4252826934 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 add item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default} to crush map v108) v1 ==== 220+0+0 (unknown 458349866 0 0) 0x7fc8bc013010 con 0x7fc8cc09c550 2024-09-16T14:47:01.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.727+0000 7fc8c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T14:47:01.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.727+0000 7fc8c1ffb640 10 monclient: _finish_command 2 = system:0 add item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default} to crush map 2024-09-16T14:47:01.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.759+0000 7f82a7d62640 10 monclient: get_monmap_and_config 2024-09-16T14:47:01.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.759+0000 7f82a7d62640 10 monclient: build_initial_monmap 2024-09-16T14:47:01.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.759+0000 7f82a7d62640 10 monclient: monmap: 2024-09-16T14:47:01.764 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:01.764 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.764 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:01.763971+0000 2024-09-16T14:47:01.764 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:01.763971+0000 2024-09-16T14:47:01.764 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:01.764 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.764 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:01.764 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:01.765 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:01.765 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.759+0000 7f82a7d62640 1 Processor -- start 2024-09-16T14:47:01.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.759+0000 7f82a7d62640 1 -- start start 2024-09-16T14:47:01.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.759+0000 7f82a7d62640 10 monclient: init 2024-09-16T14:47:01.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.759+0000 7f82a7d62640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:01.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.759+0000 7f82a7d62640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:01.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.759+0000 7f82a7d62640 10 monclient(hunting): picked mon.noname-b con 0x7f82a00a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:01.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.759+0000 7f82a7d62640 10 monclient(hunting): picked mon.noname-a con 0x7f82a00a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:01.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.759+0000 7f82a7d62640 10 monclient(hunting): picked mon.noname-c con 0x7f82a00a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:01.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.759+0000 7f82a7d62640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a00a3250 con 0x7f82a00a0000 2024-09-16T14:47:01.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.759+0000 7f82a7d62640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a009f600 con 0x7f82a00a2800 2024-09-16T14:47:01.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.759+0000 7f82a7d62640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a00a3830 con 0x7f82a00a1d10 2024-09-16T14:47:01.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:01.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 10 monclient(hunting): authenticate will time out at 1245.884056s 2024-09-16T14:47:01.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a5afe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f82a00a1d10 0x7f82a00a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35402/0 (socket says 172.21.15.26:35402) 2024-09-16T14:47:01.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a5afe640 1 -- 172.21.15.26:0/1070004766 learned_addr learned my addr 172.21.15.26:0/1070004766 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:01.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 1 -- 172.21.15.26:0/1070004766 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971526849 0 0) 0x7f82a00a3830 con 0x7f82a00a1d10 2024-09-16T14:47:01.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient(hunting): my global_id is 114161 2024-09-16T14:47:01.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 1 -- 172.21.15.26:0/1070004766 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f828c003530 con 0x7f82a00a1d10 2024-09-16T14:47:01.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 1 -- 172.21.15.26:0/1070004766 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935296486 0 0) 0x7f82a00a3250 con 0x7f82a00a0000 2024-09-16T14:47:01.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient(hunting): my global_id is 104370 2024-09-16T14:47:01.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 1 -- 172.21.15.26:0/1070004766 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a00a3830 con 0x7f82a00a0000 2024-09-16T14:47:01.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 1 -- 172.21.15.26:0/1070004766 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1633832706 0 0) 0x7f828c003530 con 0x7f82a00a1d10 2024-09-16T14:47:01.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 1 -- 172.21.15.26:0/1070004766 >> v1:172.21.15.189:6789/0 conn(0x7f82a00a2800 legacy=0x7f82a013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 1 -- 172.21.15.26:0/1070004766 >> v1:172.21.15.26:6789/0 conn(0x7f82a00a0000 legacy=0x7f82a0139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:01.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 1 monclient: found mon.noname-b 2024-09-16T14:47:01.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:01.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:47:01.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 1 -- 172.21.15.26:0/1070004766 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a009ed60 con 0x7f82a00a1d10 2024-09-16T14:47:01.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient: _finish_auth 0 2024-09-16T14:47:01.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:01.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:01.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 1 -- 172.21.15.26:0/1070004766 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f82980019a0 con 0x7f82a00a1d10 2024-09-16T14:47:01.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:01.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient: dump: 2024-09-16T14:47:01.775 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.775 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.776 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.776 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.776 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.776 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.776 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.776 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.776 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.776 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 5 monclient: authenticate success, global_id 114161 2024-09-16T14:47:01.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:01.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 1 -- 172.21.15.26:0/1070004766 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8298001c00 con 0x7f82a00a1d10 2024-09-16T14:47:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 1 -- 172.21.15.26:0/1070004766 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8298001fd0 con 0x7f82a00a1d10 2024-09-16T14:47:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 4 set_mon_vals no callback set 2024-09-16T14:47:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a4afc640 10 monclient: dump: 2024-09-16T14:47:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.777 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.778 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.778 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.778 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.778 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.778 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 10 monclient: shutdown 2024-09-16T14:47:01.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:01.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 1 -- 172.21.15.26:0/1070004766 >> v1:172.21.15.26:6790/0 conn(0x7f82a00a1d10 legacy=0x7f82a00a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 1 -- 172.21.15.26:0/1070004766 shutdown_connections 2024-09-16T14:47:01.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 1 -- 172.21.15.26:0/1070004766 >> 172.21.15.26:0/1070004766 conn(0x7f82a00a7f30 msgr2=0x7f82a00a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:01.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 1 -- 172.21.15.26:0/1070004766 shutdown_connections 2024-09-16T14:47:01.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 1 -- 172.21.15.26:0/1070004766 wait complete. 2024-09-16T14:47:01.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 10 monclient: build_initial_monmap 2024-09-16T14:47:01.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 10 monclient: monmap: 2024-09-16T14:47:01.779 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:01.779 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:01.779 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:01.779 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:01.779 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:01.779 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:01.779 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:01.779 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:01.779 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:01.779 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 1 Processor -- start 2024-09-16T14:47:01.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 1 -- start start 2024-09-16T14:47:01.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 10 monclient: init 2024-09-16T14:47:01.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:01.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:01.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 10 monclient(hunting): picked mon.noname-a con 0x7f82a00a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:01.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 10 monclient(hunting): picked mon.noname-c con 0x7f82a00a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:01.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 10 monclient(hunting): picked mon.noname-b con 0x7f82a00a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:01.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a00a6380 con 0x7f82a00a0000 2024-09-16T14:47:01.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a0136d60 con 0x7f82a00a2800 2024-09-16T14:47:01.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a0133c50 con 0x7f82a00a1d10 2024-09-16T14:47:01.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:01.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a7d62640 10 monclient(hunting): authenticate will time out at 1245.887109s 2024-09-16T14:47:01.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a5afe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f82a00a0000 0x7f82a0132e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38504/0 (socket says 172.21.15.26:38504) 2024-09-16T14:47:01.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82a5afe640 1 -- 172.21.15.26:0/1351994089 learned_addr learned my addr 172.21.15.26:0/1351994089 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:01.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248313842 0 0) 0x7f82a0133c50 con 0x7f82a00a1d10 2024-09-16T14:47:01.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): my global_id is 114164 2024-09-16T14:47:01.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8284003530 con 0x7f82a00a1d10 2024-09-16T14:47:01.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854012487 0 0) 0x7f82a00a6380 con 0x7f82a00a0000 2024-09-16T14:47:01.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): my global_id is 104373 2024-09-16T14:47:01.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a0133c50 con 0x7f82a00a0000 2024-09-16T14:47:01.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4095399789 0 0) 0x7f8284003530 con 0x7f82a00a1d10 2024-09-16T14:47:01.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82a00a6380 con 0x7f82a00a1d10 2024-09-16T14:47:01.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084686624 0 0) 0x7f82a0136d60 con 0x7f82a00a2800 2024-09-16T14:47:01.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): my global_id is 94780 2024-09-16T14:47:01.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8284003530 con 0x7f82a00a2800 2024-09-16T14:47:01.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f82900019e0 con 0x7f82a00a1d10 2024-09-16T14:47:01.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:01.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): dump: 2024-09-16T14:47:01.783 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.783 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.783 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.783 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.783 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.783 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.784 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.784 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.784 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.784 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:01.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4081256283 0 0) 0x7f82a0133c50 con 0x7f82a00a0000 2024-09-16T14:47:01.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82a0136d60 con 0x7f82a00a0000 2024-09-16T14:47:01.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f82980018a0 con 0x7f82a00a0000 2024-09-16T14:47:01.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:01.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.763+0000 7f82967fc640 10 monclient(hunting): dump: 2024-09-16T14:47:01.785 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.785 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.785 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.785 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.785 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.785 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.785 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.785 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.785 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.785 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3810638333 0 0) 0x7f82a00a6380 con 0x7f82a00a1d10 2024-09-16T14:47:01.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 >> v1:172.21.15.189:6789/0 conn(0x7f82a00a2800 legacy=0x7f82a01d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 >> v1:172.21.15.26:6789/0 conn(0x7f82a00a0000 legacy=0x7f82a0132e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:01.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 monclient: found mon.c 2024-09-16T14:47:01.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:01.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:01.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a00a68b0 con 0x7f82a00a1d10 2024-09-16T14:47:01.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:01.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:01.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:01.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82a7d62640 5 monclient: authenticate success, global_id 114164 2024-09-16T14:47:01.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82a7d62640 10 monclient: _renew_subs 2024-09-16T14:47:01.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82a7d62640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:01.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82a7d62640 1 -- 172.21.15.26:0/1351994089 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a0134ad0 con 0x7f82a00a1d10 2024-09-16T14:47:01.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82a7d62640 10 monclient: _renew_subs 2024-09-16T14:47:01.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82a7d62640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:01.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82a7d62640 1 -- 172.21.15.26:0/1351994089 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a0136a80 con 0x7f82a00a1d10 2024-09-16T14:47:01.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8290003600 con 0x7f82a00a1d10 2024-09-16T14:47:01.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:01.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8290003910 con 0x7f82a00a1d10 2024-09-16T14:47:01.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:01.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f8297fff640 4 set_mon_vals no callback set 2024-09-16T14:47:01.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient: dump: 2024-09-16T14:47:01.788 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.788 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.788 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.788 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.788 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.788 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.788 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.788 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.789 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.789 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82a7d62640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:01.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82a7d62640 1 -- 172.21.15.26:0/1351994089 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8288006590 con 0x7f8288002830 2024-09-16T14:47:01.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f829000e380 con 0x7f82a00a1d10 2024-09-16T14:47:01.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(108..108 src has 73..108) v4 ==== 5709+0+0 (unknown 2728102717 0 0) 0x7f8290042660 con 0x7f82a00a1d10 2024-09-16T14:47:01.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336163800 0 0) 0x7f8288006590 con 0x7f8288002830 2024-09-16T14:47:01.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient(hunting): my global_id is 114167 2024-09-16T14:47:01.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a00a6380 con 0x7f8288002830 2024-09-16T14:47:01.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 589943594 0 0) 0x7f82a00a6380 con 0x7f8288002830 2024-09-16T14:47:01.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8288006590 con 0x7f8288002830 2024-09-16T14:47:01.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8298005120 con 0x7f8288002830 2024-09-16T14:47:01.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82a62ff640 10 monclient: get_auth_request con 0x7f8284036420 auth_method 0 2024-09-16T14:47:01.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:01.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient: dump: 2024-09-16T14:47:01.790 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.790 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.790 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.790 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.791 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.791 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.791 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.791 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.791 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.791 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2323493412 0 0) 0x7f8288006590 con 0x7f8288002830 2024-09-16T14:47:01.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f828800a0e0 con 0x7f8288002830 2024-09-16T14:47:01.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8298007510 con 0x7f8288002830 2024-09-16T14:47:01.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:01.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:01.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 >> v1:172.21.15.26:6790/0 conn(0x7f8288002830 legacy=0x7f8288004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.767+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 >> v1:172.21.15.26:6790/0 conn(0x7f8288002830 legacy=0x7f8288004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:01.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82a7d62640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:47:01.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82a7d62640 1 -- 172.21.15.26:0/1351994089 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8288009da0 con 0x7f82880090c0 2024-09-16T14:47:01.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247617863 0 0) 0x7f8288009da0 con 0x7f82880090c0 2024-09-16T14:47:01.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 10 monclient(hunting): my global_id is 114170 2024-09-16T14:47:01.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8288006590 con 0x7f82880090c0 2024-09-16T14:47:01.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 439056906 0 0) 0x7f8288006590 con 0x7f82880090c0 2024-09-16T14:47:01.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8288009da0 con 0x7f82880090c0 2024-09-16T14:47:01.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8298005a00 con 0x7f82880090c0 2024-09-16T14:47:01.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:01.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 10 monclient: dump: 2024-09-16T14:47:01.794 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.794 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.795 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.795 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.795 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.795 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.795 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.795 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.795 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.795 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2932323524 0 0) 0x7f8288009da0 con 0x7f82880090c0 2024-09-16T14:47:01.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f828800c290 con 0x7f82880090c0 2024-09-16T14:47:01.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 446964059) 0x7f8298006e50 con 0x7f82880090c0 2024-09-16T14:47:01.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:47:01.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:01.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 >> v1:172.21.15.26:6790/0 conn(0x7f82880090c0 legacy=0x7f8288009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82967fc640 1 -- 172.21.15.26:0/1351994089 >> v1:172.21.15.26:6790/0 conn(0x7f82880090c0 legacy=0x7f8288009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:01.796 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:47:01.796 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:01.796 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:01.796 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:47:01.797 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 44, 2024-09-16T14:47:01.797 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:47:01.797 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:47:01.797 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:47:01.797 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:47:01.797 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:01.797 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:47:01.797 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:01.797 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:47:01.797 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:47:01.797 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:01.798 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:01.798 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:01.798 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:01.798 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:01.798 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:01.798 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:01.798 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:01.798 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:01.798 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:01.798 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:01.798 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:47:01.799 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:47:01.799 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:01.799 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:01.799 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:01.799 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:01.799 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:01.799 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:01.799 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:01.799 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:01.799 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:01.799 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:01.800 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.800 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:47:01.800 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:47:01.800 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:47:01.800 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:47:01.800 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:47:01.800 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:47:01.800 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:01.800 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:01.800 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:47:01.801 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:47:01.801 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:47:01.801 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:47:01.801 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:47:01.801 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:47:01.801 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:47:01.801 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:01.801 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:47:01.801 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:01.801 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:01.801 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:01.802 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:01.802 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:01.802 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:01.802 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:01.802 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:01.802 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:01.802 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:01.802 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:01.802 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:47:01.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.802 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:47:01.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.803 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:01.803 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:01.803 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:01.803 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:01.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.803 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:01.803 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:47:01.803 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:01.803 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.803 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.803 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:47:01.804 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:47:01.804 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:01.804 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:01.804 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:01.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.804 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:01.804 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:01.804 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:01.804 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:01.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.804 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:01.805 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:47:01.805 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:01.805 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.805 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.805 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:47:01.805 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:47:01.805 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:01.805 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:01.805 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:01.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.806 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:01.806 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:01.806 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:01.806 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:01.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.806 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:01.806 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:47:01.806 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:01.806 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.806 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.806 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:47:01.807 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:47:01.807 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:01.807 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:01.807 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:01.807 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.807 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.807 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:47:01.807 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:47:01.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.807 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:01.807 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:01.808 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:01.808 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.808 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:01.808 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:47:01.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.808 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:01.808 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:01.808 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:47:01.808 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.808 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.808 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:47:01.809 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:47:01.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82a7d62640 1 -- 172.21.15.26:0/1351994089 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8284036420 legacy=0x7f82840388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82a7d62640 10 monclient: shutdown 2024-09-16T14:47:01.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82a7d62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:01.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82a7d62640 1 -- 172.21.15.26:0/1351994089 >> v1:172.21.15.26:6790/0 conn(0x7f82a00a1d10 legacy=0x7f82a0133540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82a62ff640 1 -- 172.21.15.26:0/1351994089 reap_dead start 2024-09-16T14:47:01.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82a62ff640 1 -- 172.21.15.26:0/1351994089 reap_dead start 2024-09-16T14:47:01.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82a7d62640 1 -- 172.21.15.26:0/1351994089 shutdown_connections 2024-09-16T14:47:01.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82a7d62640 1 -- 172.21.15.26:0/1351994089 >> 172.21.15.26:0/1351994089 conn(0x7f82a00a7f30 msgr2=0x7f82a013e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:01.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82a7d62640 1 -- 172.21.15.26:0/1351994089 shutdown_connections 2024-09-16T14:47:01.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.775+0000 7f82a7d62640 1 -- 172.21.15.26:0/1351994089 wait complete. 2024-09-16T14:47:01.811 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:47:01.811 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:47:01.811 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:47:01.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.827+0000 7fc8d3540640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T14:47:01.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.827+0000 7fc8d3540640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T14:47:01.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.827+0000 7fc8d3540640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:01.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.827+0000 7fc8d3540640 1 -- 172.21.15.189:0/4252826934 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7fc8ac005300 con 0x7fc8cc09c550 2024-09-16T14:47:01.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.827+0000 7fc8c1ffb640 1 -- 172.21.15.189:0/4252826934 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change v108) v1 ==== 218+0+0 (unknown 176011899 0 0) 0x7fc8bc014790 con 0x7fc8cc09c550 2024-09-16T14:47:01.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.827+0000 7fc8c1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T14:47:01.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.827+0000 7fc8c1ffb640 10 monclient: _finish_command 3 = system:0 set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change 2024-09-16T14:47:01.832 INFO:tasks.workunit.client.0.smithi189.stderr:set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change 2024-09-16T14:47:01.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.831+0000 7fc8d3540640 1 -- 172.21.15.189:0/4252826934 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc8b0036010 legacy=0x7fc8b00384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.831+0000 7fc8d3540640 10 monclient: shutdown 2024-09-16T14:47:01.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.831+0000 7fc8d3540640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:01.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.831+0000 7fc8d3540640 1 -- 172.21.15.189:0/4252826934 >> v1:172.21.15.189:6789/0 conn(0x7fc8cc09c550 legacy=0x7fc8cc09a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.831+0000 7fc8d3540640 1 -- 172.21.15.189:0/4252826934 shutdown_connections 2024-09-16T14:47:01.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.831+0000 7fc8d3540640 1 -- 172.21.15.189:0/4252826934 >> 172.21.15.189:0/4252826934 conn(0x7fc8cc0a3380 msgr2=0x7fc8cc0a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:01.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.831+0000 7fc8d3540640 1 -- 172.21.15.189:0/4252826934 shutdown_connections 2024-09-16T14:47:01.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.831+0000 7fc8d3540640 1 -- 172.21.15.189:0/4252826934 wait complete. 2024-09-16T14:47:01.849 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add 14 123 root=default host=foobaz 2024-09-16T14:47:01.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 10 monclient: get_monmap_and_config 2024-09-16T14:47:01.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 10 monclient: build_initial_monmap 2024-09-16T14:47:01.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 10 monclient: monmap: 2024-09-16T14:47:01.930 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:01.930 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.930 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:01.930294+0000 2024-09-16T14:47:01.930 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:01.930294+0000 2024-09-16T14:47:01.930 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:01.931 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.931 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:01.931 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:01.931 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:01.931 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 1 Processor -- start 2024-09-16T14:47:01.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 1 -- start start 2024-09-16T14:47:01.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 10 monclient: init 2024-09-16T14:47:01.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:01.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:01.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 10 monclient(hunting): picked mon.noname-c con 0x7fdfa40a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:01.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 10 monclient(hunting): picked mon.noname-a con 0x7fdfa40a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:01.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 10 monclient(hunting): picked mon.noname-b con 0x7fdfa40a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:01.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa40a3250 con 0x7fdfa40a0000 2024-09-16T14:47:01.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa409f600 con 0x7fdfa40a1d10 2024-09-16T14:47:01.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa40a3830 con 0x7fdfa40a2800 2024-09-16T14:47:01.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:01.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 10 monclient(hunting): authenticate will time out at 1246.050483s 2024-09-16T14:47:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa9108640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fdfa40a2800 0x7fdfa413d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35426/0 (socket says 172.21.15.26:35426) 2024-09-16T14:47:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa9108640 1 -- 172.21.15.26:0/3770847038 learned_addr learned my addr 172.21.15.26:0/3770847038 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa3fff640 1 --1- 172.21.15.26:0/3770847038 >> v1:172.21.15.26:6789/0 conn(0x7fdfa40a0000 0x7fdfa4139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38506/0 (socket says 172.21.15.26:38506) 2024-09-16T14:47:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 1 -- 172.21.15.26:0/3770847038 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759490932 0 0) 0x7fdfa40a3830 con 0x7fdfa40a2800 2024-09-16T14:47:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient(hunting): my global_id is 114173 2024-09-16T14:47:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 1 -- 172.21.15.26:0/3770847038 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf8c003530 con 0x7fdfa40a2800 2024-09-16T14:47:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 1 -- 172.21.15.26:0/3770847038 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298964362 0 0) 0x7fdfa40a3250 con 0x7fdfa40a0000 2024-09-16T14:47:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient(hunting): my global_id is 104376 2024-09-16T14:47:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 1 -- 172.21.15.26:0/3770847038 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa40a3830 con 0x7fdfa40a0000 2024-09-16T14:47:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 1 -- 172.21.15.26:0/3770847038 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362882634 0 0) 0x7fdfa409f600 con 0x7fdfa40a1d10 2024-09-16T14:47:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient(hunting): my global_id is 94783 2024-09-16T14:47:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 1 -- 172.21.15.26:0/3770847038 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa40a3250 con 0x7fdfa40a1d10 2024-09-16T14:47:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 1 -- 172.21.15.26:0/3770847038 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1027307621 0 0) 0x7fdfa40a3830 con 0x7fdfa40a0000 2024-09-16T14:47:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 1 -- 172.21.15.26:0/3770847038 >> v1:172.21.15.26:6790/0 conn(0x7fdfa40a2800 legacy=0x7fdfa413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 1 -- 172.21.15.26:0/3770847038 >> v1:172.21.15.189:6789/0 conn(0x7fdfa40a1d10 legacy=0x7fdfa40a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:01.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 1 monclient: found mon.noname-a 2024-09-16T14:47:01.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:01.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:47:01.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 1 -- 172.21.15.26:0/3770847038 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa409ed60 con 0x7fdfa40a0000 2024-09-16T14:47:01.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:01.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:01.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:01.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 1 -- 172.21.15.26:0/3770847038 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdf98001600 con 0x7fdfa40a0000 2024-09-16T14:47:01.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 5 monclient: authenticate success, global_id 104376 2024-09-16T14:47:01.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:01.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:01.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient: dump: 2024-09-16T14:47:01.943 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.943 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.943 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.943 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.943 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.943 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.943 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.943 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.943 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.943 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:01.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 1 -- 172.21.15.26:0/3770847038 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf98001820 con 0x7fdfa40a0000 2024-09-16T14:47:01.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 1 -- 172.21.15.26:0/3770847038 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdf98001bf0 con 0x7fdfa40a0000 2024-09-16T14:47:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfaab6b640 4 set_mon_vals no callback set 2024-09-16T14:47:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.927+0000 7fdfa37fe640 10 monclient: dump: 2024-09-16T14:47:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.944 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.945 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 10 monclient: shutdown 2024-09-16T14:47:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 1 -- 172.21.15.26:0/3770847038 >> v1:172.21.15.26:6789/0 conn(0x7fdfa40a0000 legacy=0x7fdfa4139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 1 -- 172.21.15.26:0/3770847038 shutdown_connections 2024-09-16T14:47:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 1 -- 172.21.15.26:0/3770847038 >> 172.21.15.26:0/3770847038 conn(0x7fdfa40a7f30 msgr2=0x7fdfa40a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 1 -- 172.21.15.26:0/3770847038 shutdown_connections 2024-09-16T14:47:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 1 -- 172.21.15.26:0/3770847038 wait complete. 2024-09-16T14:47:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 10 monclient: build_initial_monmap 2024-09-16T14:47:01.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 10 monclient: monmap: 2024-09-16T14:47:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:01.946 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 1 Processor -- start 2024-09-16T14:47:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 1 -- start start 2024-09-16T14:47:01.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 10 monclient: init 2024-09-16T14:47:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 10 monclient(hunting): picked mon.noname-b con 0x7fdfa40a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 10 monclient(hunting): picked mon.noname-c con 0x7fdfa40a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 10 monclient(hunting): picked mon.noname-a con 0x7fdfa40a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa4139e30 con 0x7fdfa40a2800 2024-09-16T14:47:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa40a65b0 con 0x7fdfa40a0000 2024-09-16T14:47:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa409e500 con 0x7fdfa40a1d10 2024-09-16T14:47:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa9108640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdfa40a2800 0x7fdfa41d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38508/0 (socket says 172.21.15.26:38508) 2024-09-16T14:47:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa9108640 1 -- 172.21.15.26:0/3832223572 learned_addr learned my addr 172.21.15.26:0/3832223572 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:01.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 10 monclient(hunting): authenticate will time out at 1246.053977s 2024-09-16T14:47:01.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603598937 0 0) 0x7fdfa4139e30 con 0x7fdfa40a2800 2024-09-16T14:47:01.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): my global_id is 104379 2024-09-16T14:47:01.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf90003530 con 0x7fdfa40a2800 2024-09-16T14:47:01.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513919753 0 0) 0x7fdfa409e500 con 0x7fdfa40a1d10 2024-09-16T14:47:01.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): my global_id is 114176 2024-09-16T14:47:01.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa4139e30 con 0x7fdfa40a1d10 2024-09-16T14:47:01.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422980443 0 0) 0x7fdfa40a65b0 con 0x7fdfa40a0000 2024-09-16T14:47:01.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): my global_id is 94786 2024-09-16T14:47:01.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa409e500 con 0x7fdfa40a0000 2024-09-16T14:47:01.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4129224422 0 0) 0x7fdf90003530 con 0x7fdfa40a2800 2024-09-16T14:47:01.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfa40a65b0 con 0x7fdfa40a2800 2024-09-16T14:47:01.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 137523827 0 0) 0x7fdfa4139e30 con 0x7fdfa40a1d10 2024-09-16T14:47:01.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf90003530 con 0x7fdfa40a1d10 2024-09-16T14:47:01.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdf9c001be0 con 0x7fdfa40a2800 2024-09-16T14:47:01.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:01.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): dump: 2024-09-16T14:47:01.950 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.950 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.950 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.950 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.950 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.950 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.950 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.950 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.950 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.950 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:01.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdf98001b30 con 0x7fdfa40a1d10 2024-09-16T14:47:01.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:01.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient(hunting): dump: 2024-09-16T14:47:01.951 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.951 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.951 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.951 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.951 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.951 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.951 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.951 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.952 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.952 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 562777363 0 0) 0x7fdfa40a65b0 con 0x7fdfa40a2800 2024-09-16T14:47:01.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 >> v1:172.21.15.26:6790/0 conn(0x7fdfa40a1d10 legacy=0x7fdfa4139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 >> v1:172.21.15.189:6789/0 conn(0x7fdfa40a0000 legacy=0x7fdfa4139010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient: _finish_hunting 0 2024-09-16T14:47:01.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 monclient: found mon.a 2024-09-16T14:47:01.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:01.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:01.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa413b580 con 0x7fdfa40a2800 2024-09-16T14:47:01.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient: _finish_auth 0 2024-09-16T14:47:01.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:01.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:01.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 5 monclient: authenticate success, global_id 104379 2024-09-16T14:47:01.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 10 monclient: _renew_subs 2024-09-16T14:47:01.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:01.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 1 -- 172.21.15.26:0/3832223572 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa413ac60 con 0x7fdfa40a2800 2024-09-16T14:47:01.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf9c004b70 con 0x7fdfa40a2800 2024-09-16T14:47:01.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:01.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdf9c004e80 con 0x7fdfa40a2800 2024-09-16T14:47:01.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 10 monclient: _renew_subs 2024-09-16T14:47:01.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:01.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 1 -- 172.21.15.26:0/3832223572 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa413ba50 con 0x7fdfa40a2800 2024-09-16T14:47:01.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa2ffd640 4 set_mon_vals no callback set 2024-09-16T14:47:01.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:01.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfa17fa640 10 monclient: dump: 2024-09-16T14:47:01.954 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.954 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.954 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.954 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.954 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.954 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.954 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.954 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.954 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.954 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:01.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.931+0000 7fdfaab6b640 1 -- 172.21.15.26:0/3832223572 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdfa402c7d0 con 0x7fdfa402c3f0 2024-09-16T14:47:01.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fdf9c00f8f0 con 0x7fdfa40a2800 2024-09-16T14:47:01.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(108..108 src has 73..108) v4 ==== 5709+0+0 (unknown 2728102717 0 0) 0x7fdf9c043c50 con 0x7fdfa40a2800 2024-09-16T14:47:01.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024154936 0 0) 0x7fdfa402c7d0 con 0x7fdfa402c3f0 2024-09-16T14:47:01.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 10 monclient(hunting): my global_id is 94789 2024-09-16T14:47:01.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa40a65b0 con 0x7fdfa402c3f0 2024-09-16T14:47:01.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa3fff640 10 monclient: get_auth_request con 0x7fdf90036370 auth_method 0 2024-09-16T14:47:01.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3704571789 0 0) 0x7fdfa40a65b0 con 0x7fdfa402c3f0 2024-09-16T14:47:01.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfa402c7d0 con 0x7fdfa402c3f0 2024-09-16T14:47:01.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdf94001770 con 0x7fdfa402c3f0 2024-09-16T14:47:01.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:01.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 10 monclient: dump: 2024-09-16T14:47:01.956 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.956 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.956 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.956 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.956 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.956 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.956 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.956 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.956 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.956 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2965465539 0 0) 0x7fdfa402c7d0 con 0x7fdfa402c3f0 2024-09-16T14:47:01.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdfa413d3e0 con 0x7fdfa402c3f0 2024-09-16T14:47:01.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fdf94001af0 con 0x7fdfa402c3f0 2024-09-16T14:47:01.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:01.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:01.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 >> v1:172.21.15.189:6789/0 conn(0x7fdfa402c3f0 legacy=0x7fdfa4061160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.935+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 >> v1:172.21.15.189:6789/0 conn(0x7fdfa402c3f0 legacy=0x7fdfa4061160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:01.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfaab6b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:47:01.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfaab6b640 1 -- 172.21.15.26:0/3832223572 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdfa41358f0 con 0x7fdfa4135510 2024-09-16T14:47:01.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001000914 0 0) 0x7fdfa41358f0 con 0x7fdfa4135510 2024-09-16T14:47:01.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 10 monclient(hunting): my global_id is 94792 2024-09-16T14:47:01.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa402c7d0 con 0x7fdfa4135510 2024-09-16T14:47:01.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3852519213 0 0) 0x7fdfa402c7d0 con 0x7fdfa4135510 2024-09-16T14:47:01.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfa41358f0 con 0x7fdfa4135510 2024-09-16T14:47:01.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdf940039b0 con 0x7fdfa4135510 2024-09-16T14:47:01.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:01.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 10 monclient: dump: 2024-09-16T14:47:01.959 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:01.959 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.959 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.959 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.959 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.960 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:01.960 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.960 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.960 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.960 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:01.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2774316367 0 0) 0x7fdfa41358f0 con 0x7fdfa4135510 2024-09-16T14:47:01.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdfa413c720 con 0x7fdfa4135510 2024-09-16T14:47:01.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3881 (unknown 0 0 2846600621) 0x7fdf94005b70 con 0x7fdfa4135510 2024-09-16T14:47:01.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:47:01.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:01.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 >> v1:172.21.15.189:6789/0 conn(0x7fdfa4135510 legacy=0x7fdfa409d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa17fa640 1 -- 172.21.15.26:0/3832223572 >> v1:172.21.15.189:6789/0 conn(0x7fdfa4135510 legacy=0x7fdfa409d200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:01.961 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:47:01.961 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:01.961 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:01.961 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:47:01.961 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 44, 2024-09-16T14:47:01.961 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:47:01.961 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:47:01.961 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:47:01.961 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:47:01.961 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:01.962 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:47:01.962 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:01.962 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:47:01.962 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:47:01.962 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:01.962 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:01.962 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:01.962 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:01.962 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:01.962 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:01.962 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:01.962 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:01.962 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:01.963 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:01.963 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:01.963 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:47:01.963 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:47:01.963 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:01.963 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:01.963 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:01.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:01.963 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:01.963 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:01.963 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:01.963 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:01.964 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:01.964 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:01.964 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.964 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:47:01.964 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:47:01.964 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:47:01.964 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:47:01.964 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:47:01.964 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:47:01.964 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:01.964 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:01.964 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:47:01.965 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:47:01.965 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:47:01.965 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:47:01.965 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:47:01.965 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:47:01.965 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:47:01.965 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:01.965 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:47:01.965 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:01.965 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:01.965 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:01.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:01.965 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:01.965 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:01.966 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:01.966 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:01.966 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:01.966 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:01.966 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:01.966 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:47:01.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.966 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:47:01.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:01.966 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:01.966 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:01.966 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:01.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.967 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:01.967 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:47:01.967 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:01.967 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.967 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.967 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:47:01.967 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:47:01.967 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:01.967 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:01.967 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:01.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.968 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:01.968 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:01.968 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:01.968 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:01.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.968 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:01.968 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:47:01.968 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:01.968 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.968 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.969 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:47:01.969 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:47:01.969 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:01.969 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:01.969 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:01.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.969 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:01.969 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:01.969 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:01.969 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:01.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.969 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:01.969 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:47:01.970 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:01.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.970 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.970 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:47:01.970 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:47:01.970 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:01.970 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:01.970 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:01.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.970 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.970 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:47:01.970 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:47:01.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.971 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:01.971 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:01.971 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:01.971 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.971 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:01.971 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:47:01.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:01.971 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:01.971 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:01.971 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:47:01.971 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:01.971 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:01.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:01.972 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:47:01.972 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:47:01.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfaab6b640 1 -- 172.21.15.26:0/3832223572 >> v1:172.21.15.189:6812/3196161582 conn(0x7fdf90036370 legacy=0x7fdf90038830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfaab6b640 10 monclient: shutdown 2024-09-16T14:47:01.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfaab6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:01.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfaab6b640 1 -- 172.21.15.26:0/3832223572 >> v1:172.21.15.26:6789/0 conn(0x7fdfa40a2800 legacy=0x7fdfa41d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa9108640 1 -- 172.21.15.26:0/3832223572 reap_dead start 2024-09-16T14:47:01.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfa9108640 1 -- 172.21.15.26:0/3832223572 reap_dead start 2024-09-16T14:47:01.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfaab6b640 1 -- 172.21.15.26:0/3832223572 shutdown_connections 2024-09-16T14:47:01.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfaab6b640 1 -- 172.21.15.26:0/3832223572 >> 172.21.15.26:0/3832223572 conn(0x7fdfa40a7f30 msgr2=0x7fdfa413e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:01.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfaab6b640 1 -- 172.21.15.26:0/3832223572 shutdown_connections 2024-09-16T14:47:01.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:01.943+0000 7fdfaab6b640 1 -- 172.21.15.26:0/3832223572 wait complete. 2024-09-16T14:47:01.974 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:47:01.974 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:47:01.974 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:47:01.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.963+0000 7f4a5f1ac640 10 monclient: get_monmap_and_config 2024-09-16T14:47:01.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.963+0000 7f4a5f1ac640 10 monclient: build_initial_monmap 2024-09-16T14:47:01.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.963+0000 7f4a5f1ac640 10 monclient: monmap: 2024-09-16T14:47:01.981 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:01.981 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.981 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:01.968364+0000 2024-09-16T14:47:01.981 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:01.968364+0000 2024-09-16T14:47:01.981 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:01.981 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:01.982 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.963+0000 7f4a5f1ac640 1 Processor -- start 2024-09-16T14:47:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.963+0000 7f4a5f1ac640 1 -- start start 2024-09-16T14:47:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.963+0000 7f4a5f1ac640 10 monclient: init 2024-09-16T14:47:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.963+0000 7f4a5f1ac640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.963+0000 7f4a5f1ac640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.963+0000 7f4a5f1ac640 10 monclient(hunting): picked mon.noname-b con 0x7f4a58099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.963+0000 7f4a5f1ac640 10 monclient(hunting): picked mon.noname-c con 0x7f4a5809c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.963+0000 7f4a5f1ac640 10 monclient(hunting): picked mon.noname-a con 0x7f4a5809e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.963+0000 7f4a5f1ac640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a58099e50 con 0x7f4a5809e6d0 2024-09-16T14:47:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.963+0000 7f4a5f1ac640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5809dbd0 con 0x7f4a5809c550 2024-09-16T14:47:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.963+0000 7f4a5f1ac640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5809ee20 con 0x7f4a58099a70 2024-09-16T14:47:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.963+0000 7f4a5f1ac640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.963+0000 7f4a5f1ac640 10 monclient(hunting): authenticate will time out at 1204.083945s 2024-09-16T14:47:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a57fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4a5809c550 0x7f4a581379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42516/0 (socket says 172.21.15.189:42516) 2024-09-16T14:47:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a57fff640 1 -- 172.21.15.189:0/2072034544 learned_addr learned my addr 172.21.15.189:0/2072034544 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 1 -- 172.21.15.189:0/2072034544 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709351677 0 0) 0x7f4a5809dbd0 con 0x7f4a5809c550 2024-09-16T14:47:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient(hunting): my global_id is 94795 2024-09-16T14:47:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 1 -- 172.21.15.189:0/2072034544 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a44003530 con 0x7f4a5809c550 2024-09-16T14:47:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 1 -- 172.21.15.189:0/2072034544 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 160729481 0 0) 0x7f4a58099e50 con 0x7f4a5809e6d0 2024-09-16T14:47:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient(hunting): my global_id is 104382 2024-09-16T14:47:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 1 -- 172.21.15.189:0/2072034544 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a5809dbd0 con 0x7f4a5809e6d0 2024-09-16T14:47:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 1 -- 172.21.15.189:0/2072034544 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917468924 0 0) 0x7f4a5809ee20 con 0x7f4a58099a70 2024-09-16T14:47:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient(hunting): my global_id is 114179 2024-09-16T14:47:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 1 -- 172.21.15.189:0/2072034544 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a58099e50 con 0x7f4a58099a70 2024-09-16T14:47:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 1 -- 172.21.15.189:0/2072034544 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1072459945 0 0) 0x7f4a44003530 con 0x7f4a5809c550 2024-09-16T14:47:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 1 -- 172.21.15.189:0/2072034544 >> v1:172.21.15.26:6790/0 conn(0x7f4a58099a70 legacy=0x7f4a5809dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 1 -- 172.21.15.189:0/2072034544 >> v1:172.21.15.26:6789/0 conn(0x7f4a5809e6d0 legacy=0x7f4a5813b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 1 monclient: found mon.noname-c 2024-09-16T14:47:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 1 -- 172.21.15.189:0/2072034544 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5809f100 con 0x7f4a5809c550 2024-09-16T14:47:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 1 -- 172.21.15.189:0/2072034544 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4a40001620 con 0x7f4a5809c550 2024-09-16T14:47:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 5 monclient: authenticate success, global_id 94795 2024-09-16T14:47:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient: dump: 2024-09-16T14:47:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.986 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 1 -- 172.21.15.189:0/2072034544 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a40001890 con 0x7f4a5809c550 2024-09-16T14:47:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 1 -- 172.21.15.189:0/2072034544 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4a40001c20 con 0x7f4a5809c550 2024-09-16T14:47:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 4 set_mon_vals no callback set 2024-09-16T14:47:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a577fe640 10 monclient: dump: 2024-09-16T14:47:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.988 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 10 monclient: shutdown 2024-09-16T14:47:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 1 -- 172.21.15.189:0/2072034544 >> v1:172.21.15.189:6789/0 conn(0x7f4a5809c550 legacy=0x7f4a581379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 1 -- 172.21.15.189:0/2072034544 shutdown_connections 2024-09-16T14:47:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 1 -- 172.21.15.189:0/2072034544 >> 172.21.15.189:0/2072034544 conn(0x7f4a580a3380 msgr2=0x7f4a5809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 1 -- 172.21.15.189:0/2072034544 shutdown_connections 2024-09-16T14:47:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 1 -- 172.21.15.189:0/2072034544 wait complete. 2024-09-16T14:47:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 10 monclient: build_initial_monmap 2024-09-16T14:47:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 10 monclient: monmap: 2024-09-16T14:47:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:01.989 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 1 Processor -- start 2024-09-16T14:47:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 1 -- start start 2024-09-16T14:47:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 10 monclient: init 2024-09-16T14:47:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 10 monclient(hunting): picked mon.noname-b con 0x7f4a58099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 10 monclient(hunting): picked mon.noname-a con 0x7f4a5809c550 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 10 monclient(hunting): picked mon.noname-c con 0x7f4a5809e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5809cf20 con 0x7f4a5809c550 2024-09-16T14:47:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5806b320 con 0x7f4a58099a70 2024-09-16T14:47:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5806fb10 con 0x7f4a5809e6d0 2024-09-16T14:47:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 10 monclient(hunting): authenticate will time out at 1204.086742s 2024-09-16T14:47:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5cf48640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4a58099a70 0x7f4a5811fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42528/0 (socket says 172.21.15.189:42528) 2024-09-16T14:47:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5cf48640 1 -- 172.21.15.189:0/4190906234 learned_addr learned my addr 172.21.15.189:0/4190906234 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094440934 0 0) 0x7f4a5806b320 con 0x7f4a58099a70 2024-09-16T14:47:01.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient(hunting): my global_id is 94798 2024-09-16T14:47:01.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a38003530 con 0x7f4a58099a70 2024-09-16T14:47:01.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187164671 0 0) 0x7f4a5809cf20 con 0x7f4a5809c550 2024-09-16T14:47:01.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient(hunting): my global_id is 104385 2024-09-16T14:47:01.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a5806b320 con 0x7f4a5809c550 2024-09-16T14:47:01.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049310508 0 0) 0x7f4a5806fb10 con 0x7f4a5809e6d0 2024-09-16T14:47:01.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:01.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:01.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:01.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient(hunting): my global_id is 114182 2024-09-16T14:47:01.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a5809cf20 con 0x7f4a5809e6d0 2024-09-16T14:47:01.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3430527391 0 0) 0x7f4a38003530 con 0x7f4a58099a70 2024-09-16T14:47:01.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a5806fb10 con 0x7f4a58099a70 2024-09-16T14:47:01.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4a480019a0 con 0x7f4a58099a70 2024-09-16T14:47:01.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:01.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient(hunting): dump: 2024-09-16T14:47:01.992 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:01.992 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.993 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3767939079 0 0) 0x7f4a5806fb10 con 0x7f4a58099a70 2024-09-16T14:47:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 >> v1:172.21.15.26:6790/0 conn(0x7f4a5809e6d0 legacy=0x7f4a58120920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 >> v1:172.21.15.26:6789/0 conn(0x7f4a5809c550 legacy=0x7f4a58120210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient: _finish_hunting 0 2024-09-16T14:47:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 1 monclient: found mon.b 2024-09-16T14:47:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a58123a70 con 0x7f4a58099a70 2024-09-16T14:47:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient: _finish_auth 0 2024-09-16T14:47:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 5 monclient: authenticate success, global_id 94798 2024-09-16T14:47:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 10 monclient: _renew_subs 2024-09-16T14:47:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 1 -- 172.21.15.189:0/4190906234 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5806cf70 con 0x7f4a58099a70 2024-09-16T14:47:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 10 monclient: _renew_subs 2024-09-16T14:47:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:01.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a5f1ac640 1 -- 172.21.15.189:0/4190906234 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a58122060 con 0x7f4a58099a70 2024-09-16T14:47:01.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a48001d50 con 0x7f4a58099a70 2024-09-16T14:47:01.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:01.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4a48003a70 con 0x7f4a58099a70 2024-09-16T14:47:01.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:01.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:01.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a557fa640 10 monclient: dump: 2024-09-16T14:47:01.995 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:01.995 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:01.995 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.995 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:01.995 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:01.996 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:01.996 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:01.996 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:01.996 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:01.996 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:01.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.967+0000 7f4a56ffd640 4 set_mon_vals no callback set 2024-09-16T14:47:01.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.971+0000 7f4a5f1ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:01.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.971+0000 7f4a5f1ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:01.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.971+0000 7f4a5f1ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:01.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.971+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4a48003c50 con 0x7f4a58099a70 2024-09-16T14:47:01.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.971+0000 7f4a5f1ac640 1 -- 172.21.15.189:0/4190906234 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a3c005050 con 0x7f4a58099a70 2024-09-16T14:47:01.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.971+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(108..108 src has 73..108) v4 ==== 5709+0+0 (unknown 2728102717 0 0) 0x7f4a48043b20 con 0x7f4a58099a70 2024-09-16T14:47:01.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.971+0000 7f4a57fff640 10 monclient: get_auth_request con 0x7f4a38036340 auth_method 0 2024-09-16T14:47:01.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.975+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4a48014870 con 0x7f4a58099a70 2024-09-16T14:47:01.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.975+0000 7f4a557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:01.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:01.975+0000 7f4a557fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:02.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.107+0000 7fbeefab0640 10 monclient: get_monmap_and_config 2024-09-16T14:47:02.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.107+0000 7fbeefab0640 10 monclient: build_initial_monmap 2024-09-16T14:47:02.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.107+0000 7fbeefab0640 10 monclient: monmap: 2024-09-16T14:47:02.112 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:02.113 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:02.113 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:02.112442+0000 2024-09-16T14:47:02.113 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:02.112442+0000 2024-09-16T14:47:02.113 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:02.113 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:02.113 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:02.114 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:02.114 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:02.114 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:02.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 1 Processor -- start 2024-09-16T14:47:02.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 1 -- start start 2024-09-16T14:47:02.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 10 monclient: init 2024-09-16T14:47:02.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:02.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:02.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 10 monclient(hunting): picked mon.noname-c con 0x7fbee809a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:02.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 10 monclient(hunting): picked mon.noname-a con 0x7fbee809d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:02.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 10 monclient(hunting): picked mon.noname-b con 0x7fbee809b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:02.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee809dc60 con 0x7fbee809d1b0 2024-09-16T14:47:02.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee8099ae0 con 0x7fbee809a550 2024-09-16T14:47:02.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee809b8f0 con 0x7fbee809b040 2024-09-16T14:47:02.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:02.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 10 monclient(hunting): authenticate will time out at 1246.232960s 2024-09-16T14:47:02.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeed04b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbee809d1b0 0x7fbee8136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38512/0 (socket says 172.21.15.26:38512) 2024-09-16T14:47:02.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeed04b640 1 -- 172.21.15.26:0/3941703418 learned_addr learned my addr 172.21.15.26:0/3941703418 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:02.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 1 -- 172.21.15.26:0/3941703418 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072077844 0 0) 0x7fbee809b8f0 con 0x7fbee809b040 2024-09-16T14:47:02.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:02.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:02.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:02.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient(hunting): my global_id is 114185 2024-09-16T14:47:02.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 1 -- 172.21.15.26:0/3941703418 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed4005740 con 0x7fbee809b040 2024-09-16T14:47:02.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 1 -- 172.21.15.26:0/3941703418 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921583563 0 0) 0x7fbee809dc60 con 0x7fbee809d1b0 2024-09-16T14:47:02.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:02.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:02.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:02.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient(hunting): my global_id is 104388 2024-09-16T14:47:02.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 1 -- 172.21.15.26:0/3941703418 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbee809b8f0 con 0x7fbee809d1b0 2024-09-16T14:47:02.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 1 -- 172.21.15.26:0/3941703418 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4074136645 0 0) 0x7fbed4005740 con 0x7fbee809b040 2024-09-16T14:47:02.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 1 -- 172.21.15.26:0/3941703418 >> v1:172.21.15.189:6789/0 conn(0x7fbee809a550 legacy=0x7fbee809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:02.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 1 -- 172.21.15.26:0/3941703418 >> v1:172.21.15.26:6789/0 conn(0x7fbee809d1b0 legacy=0x7fbee8136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:02.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient: _finish_hunting 0 2024-09-16T14:47:02.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 1 monclient: found mon.noname-b 2024-09-16T14:47:02.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:02.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:47:02.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 1 -- 172.21.15.26:0/3941703418 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee809bbd0 con 0x7fbee809b040 2024-09-16T14:47:02.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient: _finish_auth 0 2024-09-16T14:47:02.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient: _check_auth_tickets 2024-09-16T14:47:02.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:02.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 1 -- 172.21.15.26:0/3941703418 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbee00019a0 con 0x7fbee809b040 2024-09-16T14:47:02.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:02.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:02.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient: dump: 2024-09-16T14:47:02.126 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:02.126 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:02.126 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:02.126 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:02.126 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:02.126 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:02.126 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:02.126 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:02.126 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:02.126 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:02.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 5 monclient: authenticate success, global_id 114185 2024-09-16T14:47:02.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:02.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 1 -- 172.21.15.26:0/3941703418 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbee0001c00 con 0x7fbee809b040 2024-09-16T14:47:02.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:02.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:02.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 1 -- 172.21.15.26:0/3941703418 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbee0001fd0 con 0x7fbee809b040 2024-09-16T14:47:02.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 4 set_mon_vals no callback set 2024-09-16T14:47:02.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:02.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:02.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeec84a640 10 monclient: dump: 2024-09-16T14:47:02.127 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:02.127 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:02.127 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:02.127 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:02.128 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:02.128 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:02.128 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:02.128 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:02.128 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:02.128 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:02.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 10 monclient: shutdown 2024-09-16T14:47:02.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:02.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 1 -- 172.21.15.26:0/3941703418 >> v1:172.21.15.26:6790/0 conn(0x7fbee809b040 legacy=0x7fbee813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:02.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 1 -- 172.21.15.26:0/3941703418 shutdown_connections 2024-09-16T14:47:02.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 1 -- 172.21.15.26:0/3941703418 >> 172.21.15.26:0/3941703418 conn(0x7fbee809fb50 msgr2=0x7fbee809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:02.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 1 -- 172.21.15.26:0/3941703418 shutdown_connections 2024-09-16T14:47:02.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 1 -- 172.21.15.26:0/3941703418 wait complete. 2024-09-16T14:47:02.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 10 monclient: build_initial_monmap 2024-09-16T14:47:02.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.111+0000 7fbeefab0640 10 monclient: monmap: 2024-09-16T14:47:02.129 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:02.129 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:02.129 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:02.129 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:02.129 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:02.129 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:02.129 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:02.129 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:02.129 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:02.129 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:02.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 1 Processor -- start 2024-09-16T14:47:02.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 1 -- start start 2024-09-16T14:47:02.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 10 monclient: init 2024-09-16T14:47:02.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:02.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:02.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 10 monclient(hunting): picked mon.noname-b con 0x7fbee809a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:02.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 10 monclient(hunting): picked mon.noname-c con 0x7fbee809b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:02.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 10 monclient(hunting): picked mon.noname-a con 0x7fbee809d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:02.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee80a25d0 con 0x7fbee809d1b0 2024-09-16T14:47:02.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee809a240 con 0x7fbee809a550 2024-09-16T14:47:02.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee81a4c30 con 0x7fbee809b040 2024-09-16T14:47:02.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:02.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 10 monclient(hunting): authenticate will time out at 1246.236818s 2024-09-16T14:47:02.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeee04d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbee809d1b0 0x7fbee81cd390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38520/0 (socket says 172.21.15.26:38520) 2024-09-16T14:47:02.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeee04d640 1 -- 172.21.15.26:0/3958875778 learned_addr learned my addr 172.21.15.26:0/3958875778 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:02.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278869861 0 0) 0x7fbee80a25d0 con 0x7fbee809d1b0 2024-09-16T14:47:02.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:02.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:02.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:02.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): my global_id is 104391 2024-09-16T14:47:02.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbecc003530 con 0x7fbee809d1b0 2024-09-16T14:47:02.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007800052 0 0) 0x7fbee81a4c30 con 0x7fbee809b040 2024-09-16T14:47:02.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:02.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:02.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:02.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): my global_id is 114188 2024-09-16T14:47:02.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbee80a25d0 con 0x7fbee809b040 2024-09-16T14:47:02.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429300791 0 0) 0x7fbee809a240 con 0x7fbee809a550 2024-09-16T14:47:02.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:02.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:02.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:02.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): my global_id is 94804 2024-09-16T14:47:02.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbee81a4c30 con 0x7fbee809a550 2024-09-16T14:47:02.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 805499859 0 0) 0x7fbecc003530 con 0x7fbee809d1b0 2024-09-16T14:47:02.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbee809a240 con 0x7fbee809d1b0 2024-09-16T14:47:02.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3854977921 0 0) 0x7fbee80a25d0 con 0x7fbee809b040 2024-09-16T14:47:02.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbecc003530 con 0x7fbee809b040 2024-09-16T14:47:02.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbee0001f10 con 0x7fbee809d1b0 2024-09-16T14:47:02.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:02.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:02.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): dump: 2024-09-16T14:47:02.133 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:02.133 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:02.133 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:02.133 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:02.133 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:02.133 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:02.133 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:02.133 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:02.133 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:02.133 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:02.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:02.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbed80019e0 con 0x7fbee809b040 2024-09-16T14:47:02.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:02.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:02.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient(hunting): dump: 2024-09-16T14:47:02.134 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:02.134 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:02.134 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:02.134 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:02.134 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:02.134 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:02.134 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:02.134 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:02.134 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:02.134 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:02.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2416788323 0 0) 0x7fbee809a240 con 0x7fbee809d1b0 2024-09-16T14:47:02.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 >> v1:172.21.15.26:6790/0 conn(0x7fbee809b040 legacy=0x7fbee81c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:02.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 >> v1:172.21.15.189:6789/0 conn(0x7fbee809a550 legacy=0x7fbee81b77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:02.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:02.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 monclient: found mon.a 2024-09-16T14:47:02.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:02.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:02.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee80a2080 con 0x7fbee809d1b0 2024-09-16T14:47:02.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:02.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:02.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:02.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 5 monclient: authenticate success, global_id 104391 2024-09-16T14:47:02.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 10 monclient: _renew_subs 2024-09-16T14:47:02.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:02.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 1 -- 172.21.15.26:0/3958875778 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee81a61a0 con 0x7fbee809d1b0 2024-09-16T14:47:02.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 10 monclient: _renew_subs 2024-09-16T14:47:02.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:02.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 1 -- 172.21.15.26:0/3958875778 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee81b7ee0 con 0x7fbee809d1b0 2024-09-16T14:47:02.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbee0001540 con 0x7fbee809d1b0 2024-09-16T14:47:02.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:02.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbee0003f10 con 0x7fbee809d1b0 2024-09-16T14:47:02.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:02.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:02.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbedffff640 4 set_mon_vals no callback set 2024-09-16T14:47:02.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 10 monclient: dump: 2024-09-16T14:47:02.136 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:02.137 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:02.137 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:02.137 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:02.137 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:02.137 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:02.137 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:02.137 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:02.137 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:02.137 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:02.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:02.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:02.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:02.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeefab0640 1 -- 172.21.15.26:0/3958875778 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed0005050 con 0x7fbee809d1b0 2024-09-16T14:47:02.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbee0004070 con 0x7fbee809d1b0 2024-09-16T14:47:02.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(108..108 src has 73..108) v4 ==== 5709+0+0 (unknown 2728102717 0 0) 0x7fbee0043c80 con 0x7fbee809d1b0 2024-09-16T14:47:02.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.115+0000 7fbeed84c640 10 monclient: get_auth_request con 0x7fbecc036430 auth_method 0 2024-09-16T14:47:02.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.123+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbee0014ba0 con 0x7fbee809d1b0 2024-09-16T14:47:02.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.123+0000 7fbede7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:02.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.123+0000 7fbede7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:02.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.135+0000 7f4a5f1ac640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T14:47:02.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.135+0000 7f4a5f1ac640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T14:47:02.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.135+0000 7f4a5f1ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:02.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.135+0000 7f4a5f1ac640 1 -- 172.21.15.189:0/4190906234 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7f4a3c005880 con 0x7f4a58099a70 2024-09-16T14:47:02.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.347+0000 7fbeefab0640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:47:02.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.347+0000 7fbeefab0640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:47:02.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.347+0000 7fbeefab0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:02.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.347+0000 7fbeefab0640 1 -- 172.21.15.26:0/3958875778 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fbed0005b10 con 0x7fbee809d1b0 2024-09-16T14:47:02.355 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.351+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.355 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.351+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=11,kv=11,logm=60} crc {auth=3235009390,config=692619927,health=2580715980,kv=1867599010,logm=499811951}) 2024-09-16T14:47:02.355 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.351+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=1694425078,config=692619927,health=3837507830,kv=3455456366,logm=3452437100}) 2024-09-16T14:47:02.355 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.351+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.356 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.351+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=11,kv=11,logm=60} crc {auth=3235009390,config=692619927,health=2580715980,kv=1867599010,logm=499811951}) 2024-09-16T14:47:02.356 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.351+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=2,health=12,kv=11,logm=59} crc {auth=279358716,config=692619927,health=3837507830,kv=1867599010,logm=3494355996}) 2024-09-16T14:47:02.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.351+0000 7fbede7fc640 1 -- 172.21.15.26:0/3958875778 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fbee0014790 con 0x7fbee809d1b0 2024-09-16T14:47:02.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.351+0000 7fbede7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:47:02.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.351+0000 7fbede7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:02.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.351+0000 7fbeefab0640 1 -- 172.21.15.26:0/3958875778 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbecc036430 legacy=0x7fbecc0388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:02.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.351+0000 7fbeefab0640 10 monclient: shutdown 2024-09-16T14:47:02.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.351+0000 7fbeefab0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:02.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.351+0000 7fbeefab0640 1 -- 172.21.15.26:0/3958875778 >> v1:172.21.15.26:6789/0 conn(0x7fbee809d1b0 legacy=0x7fbee81cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:02.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.351+0000 7fbeefab0640 1 -- 172.21.15.26:0/3958875778 shutdown_connections 2024-09-16T14:47:02.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.351+0000 7fbeefab0640 1 -- 172.21.15.26:0/3958875778 >> 172.21.15.26:0/3958875778 conn(0x7fbee809fb50 msgr2=0x7fbee80a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:02.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.351+0000 7fbeefab0640 1 -- 172.21.15.26:0/3958875778 shutdown_connections 2024-09-16T14:47:02.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:02.351+0000 7fbeefab0640 1 -- 172.21.15.26:0/3958875778 wait complete. 2024-09-16T14:47:02.358 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.351+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.358 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.351+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262062067}) 2024-09-16T14:47:02.358 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.351+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063132983}) 2024-09-16T14:47:02.358 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.355+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.359 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.355+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262062067}) 2024-09-16T14:47:02.359 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.355+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3791695875}) 2024-09-16T14:47:02.360 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.355+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.360 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.355+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1897214814}) 2024-09-16T14:47:02.360 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.355+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1524258071}) 2024-09-16T14:47:02.361 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.355+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.361 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.355+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1897214814}) 2024-09-16T14:47:02.361 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.355+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=624276086}) 2024-09-16T14:47:02.362 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.359+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.362 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.359+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3526599279}) 2024-09-16T14:47:02.362 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.359+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3973264324}) 2024-09-16T14:47:02.362 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.359+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.362 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.359+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3526599279}) 2024-09-16T14:47:02.362 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.359+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2910992873}) 2024-09-16T14:47:02.364 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.359+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.364 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.359+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2884265587}) 2024-09-16T14:47:02.364 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.359+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3308823008}) 2024-09-16T14:47:02.365 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.359+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.365 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.359+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2884265587}) 2024-09-16T14:47:02.365 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.359+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679543686}) 2024-09-16T14:47:02.367 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.363+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.367 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.363+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2084313384}) 2024-09-16T14:47:02.367 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.363+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=326226654}) 2024-09-16T14:47:02.367 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.363+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.367 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.363+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2084313384}) 2024-09-16T14:47:02.367 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.363+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2394526726}) 2024-09-16T14:47:02.369 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.363+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.369 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.363+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=852669288}) 2024-09-16T14:47:02.369 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.363+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2965729780}) 2024-09-16T14:47:02.369 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.363+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.369 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.363+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=852669288}) 2024-09-16T14:47:02.369 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.363+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3715719870}) 2024-09-16T14:47:02.371 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.367+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.371 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.367+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4129888332}) 2024-09-16T14:47:02.371 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.367+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2107470138}) 2024-09-16T14:47:02.371 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.367+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.371 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.367+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4129888332}) 2024-09-16T14:47:02.371 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.367+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3267108959}) 2024-09-16T14:47:02.372 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:47:02.373 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.367+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.373 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.367+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3141472555}) 2024-09-16T14:47:02.373 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.367+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103112104}) 2024-09-16T14:47:02.373 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.367+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.373 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.367+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3141472555}) 2024-09-16T14:47:02.373 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.367+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4042139849}) 2024-09-16T14:47:02.374 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.371+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.375 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.371+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1671692819}) 2024-09-16T14:47:02.375 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.371+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=797255040}) 2024-09-16T14:47:02.375 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.371+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.375 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.371+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1671692819}) 2024-09-16T14:47:02.375 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.371+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3977959277}) 2024-09-16T14:47:02.377 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.371+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.377 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.371+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1045343412}) 2024-09-16T14:47:02.377 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.371+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2307330012}) 2024-09-16T14:47:02.377 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.371+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.377 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.371+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1045343412}) 2024-09-16T14:47:02.377 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.371+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4153415008}) 2024-09-16T14:47:02.378 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.375+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.379 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.375+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1809905809}) 2024-09-16T14:47:02.379 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.375+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013424649}) 2024-09-16T14:47:02.379 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.375+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.379 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.375+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1809905809}) 2024-09-16T14:47:02.379 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.375+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2078091894}) 2024-09-16T14:47:02.380 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.375+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.380 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.375+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3827368843}) 2024-09-16T14:47:02.380 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.375+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2072959382}) 2024-09-16T14:47:02.381 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.375+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.381 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.375+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3827368843}) 2024-09-16T14:47:02.381 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.375+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1692330890}) 2024-09-16T14:47:02.382 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.379+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.382 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.379+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=793841594}) 2024-09-16T14:47:02.383 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.379+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554371060}) 2024-09-16T14:47:02.383 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.379+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.383 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.379+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=793841594}) 2024-09-16T14:47:02.383 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.379+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1323270023}) 2024-09-16T14:47:02.384 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.379+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.384 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.379+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792609263}) 2024-09-16T14:47:02.384 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.379+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3557671211}) 2024-09-16T14:47:02.384 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.379+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.384 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.379+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792609263}) 2024-09-16T14:47:02.385 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.379+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3566139499}) 2024-09-16T14:47:02.386 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.383+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.386 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.383+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=2264074537,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3469501306}) 2024-09-16T14:47:02.386 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.383+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=82,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=3223810287,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3211780814}) 2024-09-16T14:47:02.387 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.383+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.387 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.383+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=2264074537,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3469501306}) 2024-09-16T14:47:02.387 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.383+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=5} crc {logm=1879027095,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgrstat=3307229296}) 2024-09-16T14:47:02.394 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.391+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.394 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.391+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=4,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=73} crc {mgrstat=1547895545,mon_config_key=1000370824,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=652501384,osd_snap=3794734670,osdmap=3388722920}) 2024-09-16T14:47:02.394 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.391+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=5,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=73} crc {mgrstat=2345543184,mon_config_key=1518680965,monmap=2765269217,osd_metadata=2466911662,osd_pg_creating=652501384,osd_snap=3794734670,osdmap=1437112867}) 2024-09-16T14:47:02.394 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.391+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:02.394 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.391+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=4,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=73} crc {mgrstat=1547895545,mon_config_key=1000370824,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=652501384,osd_snap=3794734670,osdmap=3388722920}) 2024-09-16T14:47:02.394 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T14:47:02.391+0000 7f5c67484640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=5,mon_config_key=9,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=73} crc {mgrstat=2345543184,mon_config_key=510897897,monmap=2765269217,osd_metadata=3424249799,osd_pg_creating=652501385,osd_snap=3794734670,osdmap=2249797834}) 2024-09-16T14:47:02.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.739+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 add item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default} to crush map v109) v1 ==== 220+0+0 (unknown 21260919 0 0) 0x7f4a48014460 con 0x7f4a58099a70 2024-09-16T14:47:02.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.739+0000 7f4a557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T14:47:02.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.739+0000 7f4a557fa640 10 monclient: _finish_command 2 = system:0 add item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default} to crush map 2024-09-16T14:47:02.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.843+0000 7f4a5f1ac640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T14:47:02.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.843+0000 7f4a5f1ac640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T14:47:02.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.843+0000 7f4a5f1ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:02.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.843+0000 7f4a5f1ac640 1 -- 172.21.15.189:0/4190906234 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7f4a3c005300 con 0x7f4a58099a70 2024-09-16T14:47:02.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.843+0000 7f4a557fa640 1 -- 172.21.15.189:0/4190906234 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change v109) v1 ==== 218+0+0 (unknown 3186741685 0 0) 0x7f4a48014640 con 0x7f4a58099a70 2024-09-16T14:47:02.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.843+0000 7f4a557fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T14:47:02.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.843+0000 7f4a557fa640 10 monclient: _finish_command 3 = system:0 set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change 2024-09-16T14:47:02.847 INFO:tasks.workunit.client.0.smithi189.stderr:set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change 2024-09-16T14:47:02.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.843+0000 7f4a5f1ac640 1 -- 172.21.15.189:0/4190906234 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4a38036340 legacy=0x7f4a38038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:02.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.843+0000 7f4a5f1ac640 10 monclient: shutdown 2024-09-16T14:47:02.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.843+0000 7f4a5f1ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:02.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.843+0000 7f4a5f1ac640 1 -- 172.21.15.189:0/4190906234 >> v1:172.21.15.189:6789/0 conn(0x7f4a58099a70 legacy=0x7f4a5811fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:02.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.843+0000 7f4a5f1ac640 1 -- 172.21.15.189:0/4190906234 shutdown_connections 2024-09-16T14:47:02.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.843+0000 7f4a5f1ac640 1 -- 172.21.15.189:0/4190906234 >> 172.21.15.189:0/4190906234 conn(0x7f4a580a3380 msgr2=0x7f4a58070060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:02.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.843+0000 7f4a5f1ac640 1 -- 172.21.15.189:0/4190906234 shutdown_connections 2024-09-16T14:47:02.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.843+0000 7f4a5f1ac640 1 -- 172.21.15.189:0/4190906234 wait complete. 2024-09-16T14:47:02.864 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T14:47:02.865 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep osd.12 2024-09-16T14:47:02.865 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 123 2024-09-16T14:47:02.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.971+0000 7f49a0581640 10 monclient: get_monmap_and_config 2024-09-16T14:47:02.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.971+0000 7f49a0581640 10 monclient: build_initial_monmap 2024-09-16T14:47:02.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.971+0000 7f49a0581640 10 monclient: monmap: 2024-09-16T14:47:02.977 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:02.977 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:02.977 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:02.977427+0000 2024-09-16T14:47:02.978 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:02.977427+0000 2024-09-16T14:47:02.978 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:02.978 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:02.978 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:02.978 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:02.978 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:02.978 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:02.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 1 Processor -- start 2024-09-16T14:47:02.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 1 -- start start 2024-09-16T14:47:02.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 10 monclient: init 2024-09-16T14:47:02.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:02.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:02.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 10 monclient(hunting): picked mon.noname-b con 0x7f49980a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:02.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 10 monclient(hunting): picked mon.noname-a con 0x7f49980a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:02.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 10 monclient(hunting): picked mon.noname-c con 0x7f499813b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:02.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49980a30b0 con 0x7f49980a25f0 2024-09-16T14:47:02.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49980a2e30 con 0x7f499813b430 2024-09-16T14:47:02.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499809eb50 con 0x7f49980a1b00 2024-09-16T14:47:02.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:02.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 10 monclient(hunting): authenticate will time out at 1205.093403s 2024-09-16T14:47:02.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499eb1e640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f499813b430 0x7f499813d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42544/0 (socket says 172.21.15.189:42544) 2024-09-16T14:47:02.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499eb1e640 1 -- 172.21.15.189:0/3362680981 learned_addr learned my addr 172.21.15.189:0/3362680981 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:02.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 1 -- 172.21.15.189:0/3362680981 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460691318 0 0) 0x7f49980a2e30 con 0x7f499813b430 2024-09-16T14:47:02.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 10 monclient(hunting): _negotiate 2024-09-16T14:47:02.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:02.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:02.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 10 monclient(hunting): my global_id is 94807 2024-09-16T14:47:02.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 1 -- 172.21.15.189:0/3362680981 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4984003530 con 0x7f499813b430 2024-09-16T14:47:02.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 1 -- 172.21.15.189:0/3362680981 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 286140879 0 0) 0x7f4984003530 con 0x7f499813b430 2024-09-16T14:47:02.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 1 -- 172.21.15.189:0/3362680981 >> v1:172.21.15.26:6790/0 conn(0x7f49980a1b00 legacy=0x7f49980a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:02.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 1 -- 172.21.15.189:0/3362680981 >> v1:172.21.15.26:6789/0 conn(0x7f49980a25f0 legacy=0x7f4998139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:02.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 10 monclient: _finish_hunting 0 2024-09-16T14:47:02.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 1 monclient: found mon.noname-c 2024-09-16T14:47:02.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:02.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:02.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 1 -- 172.21.15.189:0/3362680981 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499809ee30 con 0x7f499813b430 2024-09-16T14:47:02.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 10 monclient: _finish_auth 0 2024-09-16T14:47:02.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 10 monclient: _check_auth_tickets 2024-09-16T14:47:02.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:02.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 1 -- 172.21.15.189:0/3362680981 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f49900019a0 con 0x7f499813b430 2024-09-16T14:47:02.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:02.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:02.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 10 monclient: dump: 2024-09-16T14:47:02.984 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:02.984 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:02.984 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:02.985 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:02.985 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:02.985 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:02.985 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:02.985 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:02.985 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:02.985 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:02.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 5 monclient: authenticate success, global_id 94807 2024-09-16T14:47:02.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:02.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 1 -- 172.21.15.189:0/3362680981 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4990001cc0 con 0x7f499813b430 2024-09-16T14:47:02.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:02.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 1 -- 172.21.15.189:0/3362680981 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4990002050 con 0x7f499813b430 2024-09-16T14:47:02.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:02.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:02.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f499d31b640 10 monclient: dump: 2024-09-16T14:47:02.986 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:02.986 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:02.986 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:02.986 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:02.986 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:02.986 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:02.986 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:02.986 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:02.986 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:02.986 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:02.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:02.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 4 set_mon_vals no callback set 2024-09-16T14:47:02.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 10 monclient: shutdown 2024-09-16T14:47:02.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:02.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 1 -- 172.21.15.189:0/3362680981 >> v1:172.21.15.189:6789/0 conn(0x7f499813b430 legacy=0x7f499813d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:02.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 1 -- 172.21.15.189:0/3362680981 shutdown_connections 2024-09-16T14:47:02.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 1 -- 172.21.15.189:0/3362680981 >> 172.21.15.189:0/3362680981 conn(0x7f49980a7dd0 msgr2=0x7f49980a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:02.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 1 -- 172.21.15.189:0/3362680981 shutdown_connections 2024-09-16T14:47:02.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 1 -- 172.21.15.189:0/3362680981 wait complete. 2024-09-16T14:47:02.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 10 monclient: build_initial_monmap 2024-09-16T14:47:02.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.975+0000 7f49a0581640 10 monclient: monmap: 2024-09-16T14:47:02.987 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:02.988 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:02.988 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:02.988 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:02.988 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:02.988 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:02.988 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:02.989 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:02.989 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:02.989 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:02.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 1 Processor -- start 2024-09-16T14:47:02.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 1 -- start start 2024-09-16T14:47:02.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 10 monclient: init 2024-09-16T14:47:02.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:02.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:02.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 10 monclient(hunting): picked mon.noname-b con 0x7f49980a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:02.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 10 monclient(hunting): picked mon.noname-c con 0x7f49980a25f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:02.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 10 monclient(hunting): picked mon.noname-a con 0x7f499813b430 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:02.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4998120170 con 0x7f499813b430 2024-09-16T14:47:02.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499811e570 con 0x7f49980a1b00 2024-09-16T14:47:02.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499809e2f0 con 0x7f49980a25f0 2024-09-16T14:47:02.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:02.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 10 monclient(hunting): authenticate will time out at 1205.097032s 2024-09-16T14:47:02.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f499e31d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f49980a1b00 0x7f499811d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42552/0 (socket says 172.21.15.189:42552) 2024-09-16T14:47:02.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f499e31d640 1 -- 172.21.15.189:0/2143038897 learned_addr learned my addr 172.21.15.189:0/2143038897 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:02.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713760115 0 0) 0x7f499811e570 con 0x7f49980a1b00 2024-09-16T14:47:02.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:02.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:02.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:02.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient(hunting): my global_id is 94810 2024-09-16T14:47:02.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f497c003530 con 0x7f49980a1b00 2024-09-16T14:47:02.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 709952482 0 0) 0x7f497c003530 con 0x7f49980a1b00 2024-09-16T14:47:02.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f499811e570 con 0x7f49980a1b00 2024-09-16T14:47:02.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4988001980 con 0x7f49980a1b00 2024-09-16T14:47:02.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:02.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:02.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient(hunting): dump: 2024-09-16T14:47:02.995 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:02.995 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:02.996 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:02.996 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:02.996 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:02.996 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:02.996 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:02.996 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:02.996 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:02.996 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:02.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:02.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890519761 0 0) 0x7f4998120170 con 0x7f499813b430 2024-09-16T14:47:02.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:02.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:02.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:02.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient(hunting): my global_id is 104397 2024-09-16T14:47:02.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f497c003530 con 0x7f499813b430 2024-09-16T14:47:02.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186111898 0 0) 0x7f499809e2f0 con 0x7f49980a25f0 2024-09-16T14:47:02.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:02.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:02.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:02.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient(hunting): my global_id is 114194 2024-09-16T14:47:02.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4998120170 con 0x7f49980a25f0 2024-09-16T14:47:02.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3260756567 0 0) 0x7f499811e570 con 0x7f49980a1b00 2024-09-16T14:47:02.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 >> v1:172.21.15.26:6790/0 conn(0x7f49980a25f0 legacy=0x7f499811de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:02.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 >> v1:172.21.15.26:6789/0 conn(0x7f499813b430 legacy=0x7f49981d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:02.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:02.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 monclient: found mon.b 2024-09-16T14:47:02.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:02.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:02.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4998135570 con 0x7f49980a1b00 2024-09-16T14:47:02.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient: _finish_auth 0 2024-09-16T14:47:02.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:02.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:02.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 5 monclient: authenticate success, global_id 94810 2024-09-16T14:47:02.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 10 monclient: _renew_subs 2024-09-16T14:47:02.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:02.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 1 -- 172.21.15.189:0/2143038897 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499813aaf0 con 0x7f49980a1b00 2024-09-16T14:47:02.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 10 monclient: _renew_subs 2024-09-16T14:47:02.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:02.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 1 -- 172.21.15.189:0/2143038897 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f499811e9e0 con 0x7f49980a1b00 2024-09-16T14:47:02.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4988001d50 con 0x7f49980a1b00 2024-09-16T14:47:03.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:03.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4988003af0 con 0x7f49980a1b00 2024-09-16T14:47:03.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f499cb1a640 4 set_mon_vals no callback set 2024-09-16T14:47:03.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:03.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 10 monclient: dump: 2024-09-16T14:47:03.001 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:03.001 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.001 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.001 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.001 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.001 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:03.001 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.001 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.001 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.001 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:03.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:03.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:03.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:03.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f49a0581640 1 -- 172.21.15.189:0/2143038897 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4980005050 con 0x7f49980a1b00 2024-09-16T14:47:03.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4988003cd0 con 0x7f49980a1b00 2024-09-16T14:47:03.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(109..109 src has 73..109) v4 ==== 5789+0+0 (unknown 733532602 0 0) 0x7f4988043b20 con 0x7f49980a1b00 2024-09-16T14:47:03.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.979+0000 7f499db1c640 10 monclient: get_auth_request con 0x7f497c036340 auth_method 0 2024-09-16T14:47:03.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.983+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4988014650 con 0x7f49980a1b00 2024-09-16T14:47:03.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.983+0000 7f498effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:03.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:02.983+0000 7f498effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:03.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.147+0000 7f49a0581640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:47:03.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.147+0000 7f49a0581640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T14:47:03.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.147+0000 7f49a0581640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:03.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.147+0000 7f49a0581640 1 -- 172.21.15.189:0/2143038897 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f4980005b10 con 0x7f49980a1b00 2024-09-16T14:47:03.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.147+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v109) v1 ==== 56+0+1449 (unknown 487677150 0 2709867379) 0x7f4988012cf0 con 0x7f49980a1b00 2024-09-16T14:47:03.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.147+0000 7f498effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T14:47:03.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.147+0000 7f498effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:03.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.247+0000 7f49a0581640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:47:03.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.247+0000 7f49a0581640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T14:47:03.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.247+0000 7f49a0581640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:03.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.247+0000 7f49a0581640 1 -- 172.21.15.189:0/2143038897 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f49800055a0 con 0x7f49980a1b00 2024-09-16T14:47:03.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.247+0000 7f498effd640 1 -- 172.21.15.189:0/2143038897 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v109) v1 ==== 56+0+1449 (unknown 487677150 0 2709867379) 0x7f4988014470 con 0x7f49980a1b00 2024-09-16T14:47:03.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.247+0000 7f498effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T14:47:03.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.247+0000 7f498effd640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:03.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.251+0000 7f49a0581640 1 -- 172.21.15.189:0/2143038897 >> v1:172.21.15.189:6812/3196161582 conn(0x7f497c036340 legacy=0x7f497c038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.251+0000 7f49a0581640 10 monclient: shutdown 2024-09-16T14:47:03.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.251+0000 7f49a0581640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:03.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.251+0000 7f49a0581640 1 -- 172.21.15.189:0/2143038897 >> v1:172.21.15.189:6789/0 conn(0x7f49980a1b00 legacy=0x7f499811d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.251+0000 7f49a0581640 1 -- 172.21.15.189:0/2143038897 shutdown_connections 2024-09-16T14:47:03.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.251+0000 7f49a0581640 1 -- 172.21.15.189:0/2143038897 >> 172.21.15.189:0/2143038897 conn(0x7f49980a7dd0 msgr2=0x7f499806b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:03.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.251+0000 7f49a0581640 1 -- 172.21.15.189:0/2143038897 shutdown_connections 2024-09-16T14:47:03.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.251+0000 7f49a0581640 1 -- 172.21.15.189:0/2143038897 wait complete. 2024-09-16T14:47:03.269 INFO:tasks.workunit.client.0.smithi189.stdout: 12 123.00000 osd.12 down 1.00000 1.00000 2024-09-16T14:47:03.270 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T14:47:03.270 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep osd.14 2024-09-16T14:47:03.270 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 123 2024-09-16T14:47:03.373 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:47:03.373 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:47:03.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.375+0000 7ffbe566d640 10 monclient: get_monmap_and_config 2024-09-16T14:47:03.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.375+0000 7ffbe566d640 10 monclient: build_initial_monmap 2024-09-16T14:47:03.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.375+0000 7ffbe566d640 10 monclient: monmap: 2024-09-16T14:47:03.381 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:03.381 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.381 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:03.381082+0000 2024-09-16T14:47:03.381 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:03.381082+0000 2024-09-16T14:47:03.381 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:03.381 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:03.381 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:03.381 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:03.382 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:03.382 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:03.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 1 Processor -- start 2024-09-16T14:47:03.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 1 -- start start 2024-09-16T14:47:03.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 10 monclient: init 2024-09-16T14:47:03.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:03.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:03.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 10 monclient(hunting): picked mon.noname-c con 0x7ffbe0096490 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:03.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 10 monclient(hunting): picked mon.noname-a con 0x7ffbe0096870 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:03.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 10 monclient(hunting): picked mon.noname-b con 0x7ffbe0096cb0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:03.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe009f070 con 0x7ffbe0096870 2024-09-16T14:47:03.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe01334a0 con 0x7ffbe0096490 2024-09-16T14:47:03.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe0099890 con 0x7ffbe0096cb0 2024-09-16T14:47:03.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:03.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 10 monclient(hunting): authenticate will time out at 1205.497333s 2024-09-16T14:47:03.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbdeffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffbe0096490 0x7ffbe0132d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42564/0 (socket says 172.21.15.189:42564) 2024-09-16T14:47:03.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbdeffd640 1 -- 172.21.15.189:0/3026901765 learned_addr learned my addr 172.21.15.189:0/3026901765 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:03.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 1 -- 172.21.15.189:0/3026901765 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932045610 0 0) 0x7ffbe01334a0 con 0x7ffbe0096490 2024-09-16T14:47:03.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 10 monclient(hunting): my global_id is 94813 2024-09-16T14:47:03.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 1 -- 172.21.15.189:0/3026901765 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbc8003530 con 0x7ffbe0096490 2024-09-16T14:47:03.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 1 -- 172.21.15.189:0/3026901765 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 726420794 0 0) 0x7ffbc8003530 con 0x7ffbe0096490 2024-09-16T14:47:03.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 1 -- 172.21.15.189:0/3026901765 >> v1:172.21.15.26:6790/0 conn(0x7ffbe0096cb0 legacy=0x7ffbe013b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 1 -- 172.21.15.189:0/3026901765 >> v1:172.21.15.26:6789/0 conn(0x7ffbe0096870 legacy=0x7ffbe0137a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 10 monclient: _finish_hunting 0 2024-09-16T14:47:03.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 1 monclient: found mon.noname-c 2024-09-16T14:47:03.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:03.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:03.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 1 -- 172.21.15.189:0/3026901765 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe0099b70 con 0x7ffbe0096490 2024-09-16T14:47:03.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 10 monclient: _finish_auth 0 2024-09-16T14:47:03.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:03.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:03.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 5 monclient: authenticate success, global_id 94813 2024-09-16T14:47:03.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:03.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 1 -- 172.21.15.189:0/3026901765 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffbd0001620 con 0x7ffbe0096490 2024-09-16T14:47:03.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:03.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 10 monclient: dump: 2024-09-16T14:47:03.388 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:03.388 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.389 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.389 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.389 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.389 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:03.389 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.389 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.389 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.389 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:03.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:03.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 1 -- 172.21.15.189:0/3026901765 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbd0001940 con 0x7ffbe0096490 2024-09-16T14:47:03.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:03.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 1 -- 172.21.15.189:0/3026901765 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffbd0001d10 con 0x7ffbe0096490 2024-09-16T14:47:03.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:03.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 4 set_mon_vals no callback set 2024-09-16T14:47:03.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:03.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbddffb640 10 monclient: dump: 2024-09-16T14:47:03.390 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:03.390 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.391 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.391 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.391 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.391 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:03.391 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.391 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.391 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.391 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:03.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 10 monclient: shutdown 2024-09-16T14:47:03.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:03.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 1 -- 172.21.15.189:0/3026901765 >> v1:172.21.15.189:6789/0 conn(0x7ffbe0096490 legacy=0x7ffbe0132d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 1 -- 172.21.15.189:0/3026901765 shutdown_connections 2024-09-16T14:47:03.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 1 -- 172.21.15.189:0/3026901765 >> 172.21.15.189:0/3026901765 conn(0x7ffbe00a02a0 msgr2=0x7ffbe00a06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:03.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 1 -- 172.21.15.189:0/3026901765 shutdown_connections 2024-09-16T14:47:03.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 1 -- 172.21.15.189:0/3026901765 wait complete. 2024-09-16T14:47:03.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 10 monclient: build_initial_monmap 2024-09-16T14:47:03.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 10 monclient: monmap: 2024-09-16T14:47:03.392 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:03.392 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:03.392 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:03.392 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:03.392 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:03.392 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:03.393 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:03.393 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:03.393 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:03.393 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:03.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 1 Processor -- start 2024-09-16T14:47:03.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 1 -- start start 2024-09-16T14:47:03.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.379+0000 7ffbe566d640 10 monclient: init 2024-09-16T14:47:03.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:03.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:03.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 10 monclient(hunting): picked mon.noname-b con 0x7ffbe0096490 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:03.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 10 monclient(hunting): picked mon.noname-a con 0x7ffbe0096870 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:03.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 10 monclient(hunting): picked mon.noname-c con 0x7ffbe0096cb0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:03.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe009b6f0 con 0x7ffbe0096870 2024-09-16T14:47:03.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe009a5c0 con 0x7ffbe0096490 2024-09-16T14:47:03.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe009b2a0 con 0x7ffbe0096cb0 2024-09-16T14:47:03.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:03.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 10 monclient(hunting): authenticate will time out at 1205.500288s 2024-09-16T14:47:03.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbdeffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffbe0096490 0x7ffbe0097630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42576/0 (socket says 172.21.15.189:42576) 2024-09-16T14:47:03.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbdeffd640 1 -- 172.21.15.189:0/2279467820 learned_addr learned my addr 172.21.15.189:0/2279467820 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 1 -- 172.21.15.189:0/2279467820 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285629997 0 0) 0x7ffbe009a5c0 con 0x7ffbe0096490 2024-09-16T14:47:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 10 monclient(hunting): my global_id is 94816 2024-09-16T14:47:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 1 -- 172.21.15.189:0/2279467820 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbcc003530 con 0x7ffbe0096490 2024-09-16T14:47:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 1 -- 172.21.15.189:0/2279467820 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1004813848 0 0) 0x7ffbcc003530 con 0x7ffbe0096490 2024-09-16T14:47:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 1 -- 172.21.15.189:0/2279467820 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbe009a5c0 con 0x7ffbe0096490 2024-09-16T14:47:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 1 -- 172.21.15.189:0/2279467820 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffbd00015e0 con 0x7ffbe0096490 2024-09-16T14:47:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 10 monclient(hunting): dump: 2024-09-16T14:47:03.399 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:03.399 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.399 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.399 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.399 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.399 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:03.399 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.399 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.399 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.399 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:03.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:03.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 1 -- 172.21.15.189:0/2279467820 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4215935034 0 0) 0x7ffbe009a5c0 con 0x7ffbe0096490 2024-09-16T14:47:03.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 1 -- 172.21.15.189:0/2279467820 >> v1:172.21.15.26:6790/0 conn(0x7ffbe0096cb0 legacy=0x7ffbe01d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 1 -- 172.21.15.189:0/2279467820 >> v1:172.21.15.26:6789/0 conn(0x7ffbe0096870 legacy=0x7ffbe0097d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:03.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 1 monclient: found mon.b 2024-09-16T14:47:03.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:03.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:03.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 1 -- 172.21.15.189:0/2279467820 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe013b990 con 0x7ffbe0096490 2024-09-16T14:47:03.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:03.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:03.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:03.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 5 monclient: authenticate success, global_id 94816 2024-09-16T14:47:03.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 10 monclient: _renew_subs 2024-09-16T14:47:03.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:03.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 1 -- 172.21.15.189:0/2279467820 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe013bf50 con 0x7ffbe0096490 2024-09-16T14:47:03.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 10 monclient: _renew_subs 2024-09-16T14:47:03.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:03.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 1 -- 172.21.15.189:0/2279467820 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe0099310 con 0x7ffbe0096490 2024-09-16T14:47:03.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 1 -- 172.21.15.189:0/2279467820 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbd0004c50 con 0x7ffbe0096490 2024-09-16T14:47:03.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:03.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 1 -- 172.21.15.189:0/2279467820 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffbd0004fe0 con 0x7ffbe0096490 2024-09-16T14:47:03.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:03.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbdd7fa640 4 set_mon_vals no callback set 2024-09-16T14:47:03.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 10 monclient: dump: 2024-09-16T14:47:03.402 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:03.403 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.403 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.403 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.403 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.403 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:03.403 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.403 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.403 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.403 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:03.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 1 -- 172.21.15.189:0/2279467820 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ffbd0005140 con 0x7ffbe0096490 2024-09-16T14:47:03.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:03.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:03.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:03.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbe566d640 1 -- 172.21.15.189:0/2279467820 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc0005050 con 0x7ffbe0096490 2024-09-16T14:47:03.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbc77fe640 1 -- 172.21.15.189:0/2279467820 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(109..109 src has 73..109) v4 ==== 5789+0+0 (unknown 733532602 0 0) 0x7ffbd0043cf0 con 0x7ffbe0096490 2024-09-16T14:47:03.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.383+0000 7ffbde7fc640 10 monclient: get_auth_request con 0x7ffbcc0360a0 auth_method 0 2024-09-16T14:47:03.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.387+0000 7ffbc77fe640 1 -- 172.21.15.189:0/2279467820 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ffbd0006730 con 0x7ffbe0096490 2024-09-16T14:47:03.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.387+0000 7ffbc77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:03.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.387+0000 7ffbc77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:03.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e89f8640 10 monclient: get_monmap_and_config 2024-09-16T14:47:03.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e89f8640 10 monclient: build_initial_monmap 2024-09-16T14:47:03.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e89f8640 10 monclient: monmap: 2024-09-16T14:47:03.507 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:03.507 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.507 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:03.506873+0000 2024-09-16T14:47:03.507 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:03.506873+0000 2024-09-16T14:47:03.507 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:03.507 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:03.508 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:03.508 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:03.508 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:03.508 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e89f8640 1 Processor -- start 2024-09-16T14:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e89f8640 1 -- start start 2024-09-16T14:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e89f8640 10 monclient: init 2024-09-16T14:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e89f8640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e89f8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e89f8640 10 monclient(hunting): picked mon.noname-c con 0x7f18e409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e89f8640 10 monclient(hunting): picked mon.noname-a con 0x7f18e4132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e89f8640 10 monclient(hunting): picked mon.noname-b con 0x7f18e4138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e89f8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e409d260 con 0x7f18e4132650 2024-09-16T14:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e89f8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e4132a80 con 0x7f18e409cb80 2024-09-16T14:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e89f8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e409cf60 con 0x7f18e4138ff0 2024-09-16T14:47:03.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e89f8640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:03.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e89f8640 10 monclient(hunting): authenticate will time out at 1247.627534s 2024-09-16T14:47:03.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e2d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f18e4138ff0 0x7f18e413b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35450/0 (socket says 172.21.15.26:35450) 2024-09-16T14:47:03.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.503+0000 7f18e2d9d640 1 -- 172.21.15.26:0/2971295903 learned_addr learned my addr 172.21.15.26:0/2971295903 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:03.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 1 -- 172.21.15.26:0/2971295903 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329818619 0 0) 0x7f18e409cf60 con 0x7f18e4138ff0 2024-09-16T14:47:03.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient(hunting): my global_id is 114203 2024-09-16T14:47:03.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 1 -- 172.21.15.26:0/2971295903 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18cc003530 con 0x7f18e4138ff0 2024-09-16T14:47:03.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 1 -- 172.21.15.26:0/2971295903 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518868936 0 0) 0x7f18e409d260 con 0x7f18e4132650 2024-09-16T14:47:03.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient(hunting): my global_id is 104406 2024-09-16T14:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 1 -- 172.21.15.26:0/2971295903 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e409cf60 con 0x7f18e4132650 2024-09-16T14:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 1 -- 172.21.15.26:0/2971295903 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 470305389 0 0) 0x7f18e409cf60 con 0x7f18e4132650 2024-09-16T14:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 1 -- 172.21.15.26:0/2971295903 >> v1:172.21.15.26:6790/0 conn(0x7f18e4138ff0 legacy=0x7f18e413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 1 -- 172.21.15.26:0/2971295903 >> v1:172.21.15.189:6789/0 conn(0x7f18e409cb80 legacy=0x7f18e40717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient: _finish_hunting 0 2024-09-16T14:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 1 monclient: found mon.noname-a 2024-09-16T14:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 1 -- 172.21.15.26:0/2971295903 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e409f2d0 con 0x7f18e4132650 2024-09-16T14:47:03.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient: _finish_auth 0 2024-09-16T14:47:03.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient: _check_auth_tickets 2024-09-16T14:47:03.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:03.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 1 -- 172.21.15.26:0/2971295903 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f18d80014b0 con 0x7f18e4132650 2024-09-16T14:47:03.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:03.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient: dump: 2024-09-16T14:47:03.518 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:03.518 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.518 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.518 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.518 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.518 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.519 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 5 monclient: authenticate success, global_id 104406 2024-09-16T14:47:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 1 -- 172.21.15.26:0/2971295903 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18d80016e0 con 0x7f18e4132650 2024-09-16T14:47:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 1 -- 172.21.15.26:0/2971295903 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f18d8001ab0 con 0x7f18e4132650 2024-09-16T14:47:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e159a640 10 monclient: dump: 2024-09-16T14:47:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:03.519 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.520 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.520 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.520 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.520 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:03.520 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.520 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.520 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.520 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:03.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:03.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 4 set_mon_vals no callback set 2024-09-16T14:47:03.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 10 monclient: shutdown 2024-09-16T14:47:03.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 1 -- 172.21.15.26:0/2971295903 >> v1:172.21.15.26:6789/0 conn(0x7f18e4132650 legacy=0x7f18e41378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 1 -- 172.21.15.26:0/2971295903 shutdown_connections 2024-09-16T14:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 1 -- 172.21.15.26:0/2971295903 >> 172.21.15.26:0/2971295903 conn(0x7f18e409fb50 msgr2=0x7f18e409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 1 -- 172.21.15.26:0/2971295903 shutdown_connections 2024-09-16T14:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 1 -- 172.21.15.26:0/2971295903 wait complete. 2024-09-16T14:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 10 monclient: build_initial_monmap 2024-09-16T14:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 10 monclient: monmap: 2024-09-16T14:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 1 Processor -- start 2024-09-16T14:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 1 -- start start 2024-09-16T14:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 10 monclient: init 2024-09-16T14:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 10 monclient(hunting): picked mon.noname-a con 0x7f18e409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 10 monclient(hunting): picked mon.noname-b con 0x7f18e4132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 10 monclient(hunting): picked mon.noname-c con 0x7f18e4138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e40a2140 con 0x7f18e409cb80 2024-09-16T14:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e4071b70 con 0x7f18e4132650 2024-09-16T14:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e409a240 con 0x7f18e4138ff0 2024-09-16T14:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 10 monclient(hunting): authenticate will time out at 1247.630674s 2024-09-16T14:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e259c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f18e409cb80 0x7f18e4138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38540/0 (socket says 172.21.15.26:38540) 2024-09-16T14:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e259c640 1 -- 172.21.15.26:0/1790700586 learned_addr learned my addr 172.21.15.26:0/1790700586 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140649068 0 0) 0x7f18e409a240 con 0x7f18e4138ff0 2024-09-16T14:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): my global_id is 114206 2024-09-16T14:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c4003530 con 0x7f18e4138ff0 2024-09-16T14:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327773228 0 0) 0x7f18e40a2140 con 0x7f18e409cb80 2024-09-16T14:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): my global_id is 104409 2024-09-16T14:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e409a240 con 0x7f18e409cb80 2024-09-16T14:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 261188918 0 0) 0x7f18e409a240 con 0x7f18e409cb80 2024-09-16T14:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18e40a2140 con 0x7f18e409cb80 2024-09-16T14:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f18d4001c00 con 0x7f18e409cb80 2024-09-16T14:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): dump: 2024-09-16T14:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2137309312 0 0) 0x7f18c4003530 con 0x7f18e4138ff0 2024-09-16T14:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18e409a240 con 0x7f18e4138ff0 2024-09-16T14:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958027678 0 0) 0x7f18e4071b70 con 0x7f18e4132650 2024-09-16T14:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): my global_id is 94822 2024-09-16T14:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c4003530 con 0x7f18e4132650 2024-09-16T14:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f18dc001260 con 0x7f18e4138ff0 2024-09-16T14:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient(hunting): dump: 2024-09-16T14:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1402924816 0 0) 0x7f18e40a2140 con 0x7f18e409cb80 2024-09-16T14:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 >> v1:172.21.15.26:6790/0 conn(0x7f18e4138ff0 legacy=0x7f18e41cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 >> v1:172.21.15.189:6789/0 conn(0x7f18e4132650 legacy=0x7f18e41cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 monclient: found mon.a 2024-09-16T14:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e406f130 con 0x7f18e409cb80 2024-09-16T14:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 5 monclient: authenticate success, global_id 104409 2024-09-16T14:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 10 monclient: _renew_subs 2024-09-16T14:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 1 -- 172.21.15.26:0/1790700586 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e40711b0 con 0x7f18e409cb80 2024-09-16T14:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 10 monclient: _renew_subs 2024-09-16T14:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e89f8640 1 -- 172.21.15.26:0/1790700586 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e4137e00 con 0x7f18e409cb80 2024-09-16T14:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18d40039a0 con 0x7f18e409cb80 2024-09-16T14:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f18d4003d30 con 0x7f18e409cb80 2024-09-16T14:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18d2ffd640 10 monclient: dump: 2024-09-16T14:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.530 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.507+0000 7f18e0d99640 4 set_mon_vals no callback set 2024-09-16T14:47:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.511+0000 7f18e89f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.511+0000 7f18e89f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.511+0000 7f18e89f8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.511+0000 7f18e89f8640 1 -- 172.21.15.26:0/1790700586 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c8005050 con 0x7f18e409cb80 2024-09-16T14:47:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.511+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f18d4003e90 con 0x7f18e409cb80 2024-09-16T14:47:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.511+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(109..109 src has 73..109) v4 ==== 5789+0+0 (unknown 733532602 0 0) 0x7f18d4043bf0 con 0x7f18e409cb80 2024-09-16T14:47:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.511+0000 7f18e1d9b640 10 monclient: get_auth_request con 0x7f18c4036400 auth_method 0 2024-09-16T14:47:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.515+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f18d4004640 con 0x7f18e409cb80 2024-09-16T14:47:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.515+0000 7f18d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.515+0000 7f18d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:03.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.547+0000 7ffbe566d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:47:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.547+0000 7ffbe566d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T14:47:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.547+0000 7ffbe566d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.547+0000 7ffbe566d640 1 -- 172.21.15.189:0/2279467820 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7ffbc0005b10 con 0x7ffbe0096490 2024-09-16T14:47:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.547+0000 7ffbc77fe640 1 -- 172.21.15.189:0/2279467820 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v109) v1 ==== 56+0+1449 (unknown 487677150 0 2709867379) 0x7ffbd0012f80 con 0x7ffbe0096490 2024-09-16T14:47:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.547+0000 7ffbc77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T14:47:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.547+0000 7ffbc77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:03.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.639+0000 7ffbe566d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:47:03.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.639+0000 7ffbe566d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T14:47:03.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.639+0000 7ffbe566d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:03.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.639+0000 7ffbe566d640 1 -- 172.21.15.189:0/2279467820 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7ffbc0005850 con 0x7ffbe0096490 2024-09-16T14:47:03.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.639+0000 7ffbc77fe640 1 -- 172.21.15.189:0/2279467820 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v109) v1 ==== 56+0+1449 (unknown 487677150 0 2709867379) 0x7ffbd0014700 con 0x7ffbe0096490 2024-09-16T14:47:03.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.639+0000 7ffbc77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T14:47:03.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.639+0000 7ffbc77fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:03.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.639+0000 7ffbe566d640 1 -- 172.21.15.189:0/2279467820 >> v1:172.21.15.189:6812/3196161582 conn(0x7ffbcc0360a0 legacy=0x7ffbcc038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.639+0000 7ffbe566d640 10 monclient: shutdown 2024-09-16T14:47:03.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.639+0000 7ffbe566d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:03.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.639+0000 7ffbe566d640 1 -- 172.21.15.189:0/2279467820 >> v1:172.21.15.189:6789/0 conn(0x7ffbe0096490 legacy=0x7ffbe0097630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.639+0000 7ffbe566d640 1 -- 172.21.15.189:0/2279467820 shutdown_connections 2024-09-16T14:47:03.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.639+0000 7ffbe566d640 1 -- 172.21.15.189:0/2279467820 >> 172.21.15.189:0/2279467820 conn(0x7ffbe00a02a0 msgr2=0x7ffbe00a1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:03.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.639+0000 7ffbe566d640 1 -- 172.21.15.189:0/2279467820 shutdown_connections 2024-09-16T14:47:03.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.639+0000 7ffbe566d640 1 -- 172.21.15.189:0/2279467820 wait complete. 2024-09-16T14:47:03.659 INFO:tasks.workunit.client.0.smithi189.stdout: 14 123.00000 osd.14 down 1.00000 1.00000 2024-09-16T14:47:03.659 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush reweight-subtree foobaz 155 2024-09-16T14:47:03.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.735+0000 7f18e89f8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.735+0000 7f18e89f8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.735+0000 7f18e89f8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.735+0000 7f18e89f8640 1 -- 172.21.15.26:0/1790700586 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f18c8005880 con 0x7f18e409cb80 2024-09-16T14:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.735+0000 7f18d2ffd640 1 -- 172.21.15.26:0/1790700586 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 679899557) 0x7f18d4012dc0 con 0x7f18e409cb80 2024-09-16T14:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.735+0000 7f18d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:03.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.735+0000 7f18d2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:03.742 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:47:03.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.739+0000 7f18e89f8640 1 -- 172.21.15.26:0/1790700586 >> v1:172.21.15.189:6812/3196161582 conn(0x7f18c4036400 legacy=0x7f18c40388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.739+0000 7f18e89f8640 10 monclient: shutdown 2024-09-16T14:47:03.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.739+0000 7f18e89f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:03.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.739+0000 7f18e89f8640 1 -- 172.21.15.26:0/1790700586 >> v1:172.21.15.26:6789/0 conn(0x7f18e409cb80 legacy=0x7f18e4138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.739+0000 7f18e89f8640 1 -- 172.21.15.26:0/1790700586 shutdown_connections 2024-09-16T14:47:03.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.739+0000 7f18e89f8640 1 -- 172.21.15.26:0/1790700586 >> 172.21.15.26:0/1790700586 conn(0x7f18e409fb50 msgr2=0x7f18e4131a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:03.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.739+0000 7f18e89f8640 1 -- 172.21.15.26:0/1790700586 shutdown_connections 2024-09-16T14:47:03.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.739+0000 7f18e89f8640 1 -- 172.21.15.26:0/1790700586 wait complete. 2024-09-16T14:47:03.763 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:47:03.763 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:47:03.763 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:47:03.764 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:47:03.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.771+0000 7f5900168640 10 monclient: get_monmap_and_config 2024-09-16T14:47:03.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.771+0000 7f5900168640 10 monclient: build_initial_monmap 2024-09-16T14:47:03.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.771+0000 7f5900168640 10 monclient: monmap: 2024-09-16T14:47:03.776 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:03.776 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.776 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:03.776511+0000 2024-09-16T14:47:03.776 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:03.776511+0000 2024-09-16T14:47:03.777 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:03.777 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:03.777 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:03.777 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:03.777 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:03.777 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:03.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.771+0000 7f5900168640 1 Processor -- start 2024-09-16T14:47:03.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 1 -- start start 2024-09-16T14:47:03.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient: init 2024-09-16T14:47:03.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:03.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:03.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient(hunting): picked mon.noname-c con 0x7f58f81318a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:03.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient(hunting): picked mon.noname-a con 0x7f58f8134d50 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:03.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient(hunting): picked mon.noname-b con 0x7f58f813b2b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:03.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f809d8a0 con 0x7f58f8134d50 2024-09-16T14:47:03.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f809ddf0 con 0x7f58f81318a0 2024-09-16T14:47:03.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f809e8b0 con 0x7f58f813b2b0 2024-09-16T14:47:03.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:03.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient(hunting): authenticate will time out at 1205.892741s 2024-09-16T14:47:03.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fdf04640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f58f81318a0 0x7f58f8131c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42578/0 (socket says 172.21.15.189:42578) 2024-09-16T14:47:03.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fdf04640 1 -- 172.21.15.189:0/3394936712 learned_addr learned my addr 172.21.15.189:0/3394936712 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:03.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 1 -- 172.21.15.189:0/3394936712 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529965999 0 0) 0x7f58f809ddf0 con 0x7f58f81318a0 2024-09-16T14:47:03.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient(hunting): my global_id is 94825 2024-09-16T14:47:03.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 1 -- 172.21.15.189:0/3394936712 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58e4003530 con 0x7f58f81318a0 2024-09-16T14:47:03.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 1 -- 172.21.15.189:0/3394936712 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330928305 0 0) 0x7f58f809d8a0 con 0x7f58f8134d50 2024-09-16T14:47:03.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient(hunting): my global_id is 104412 2024-09-16T14:47:03.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 1 -- 172.21.15.189:0/3394936712 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58f809ddf0 con 0x7f58f8134d50 2024-09-16T14:47:03.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 1 -- 172.21.15.189:0/3394936712 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202658298 0 0) 0x7f58f809e8b0 con 0x7f58f813b2b0 2024-09-16T14:47:03.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient(hunting): my global_id is 114209 2024-09-16T14:47:03.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 1 -- 172.21.15.189:0/3394936712 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58f809d8a0 con 0x7f58f813b2b0 2024-09-16T14:47:03.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 1 -- 172.21.15.189:0/3394936712 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3942179867 0 0) 0x7f58e4003530 con 0x7f58f81318a0 2024-09-16T14:47:03.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 1 -- 172.21.15.189:0/3394936712 >> v1:172.21.15.26:6790/0 conn(0x7f58f813b2b0 legacy=0x7f58f813d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 1 -- 172.21.15.189:0/3394936712 >> v1:172.21.15.26:6789/0 conn(0x7f58f8134d50 legacy=0x7f58f8139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient: _finish_hunting 0 2024-09-16T14:47:03.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 1 monclient: found mon.noname-c 2024-09-16T14:47:03.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:03.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:03.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 1 -- 172.21.15.189:0/3394936712 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f809eb90 con 0x7f58f81318a0 2024-09-16T14:47:03.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient: _finish_auth 0 2024-09-16T14:47:03.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient: _check_auth_tickets 2024-09-16T14:47:03.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:03.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 5 monclient: authenticate success, global_id 94825 2024-09-16T14:47:03.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 1 -- 172.21.15.189:0/3394936712 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58f40015a0 con 0x7f58f81318a0 2024-09-16T14:47:03.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:03.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:03.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient: dump: 2024-09-16T14:47:03.788 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:03.788 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.788 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.788 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.788 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.789 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 1 -- 172.21.15.189:0/3394936712 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58f40017c0 con 0x7f58f81318a0 2024-09-16T14:47:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 1 -- 172.21.15.189:0/3394936712 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58f4001ad0 con 0x7f58f81318a0 2024-09-16T14:47:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 4 set_mon_vals no callback set 2024-09-16T14:47:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:03.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fcf02640 10 monclient: dump: 2024-09-16T14:47:03.790 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:03.790 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.790 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.790 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.790 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.790 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:03.790 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.790 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.790 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.790 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:03.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient: shutdown 2024-09-16T14:47:03.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:03.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 1 -- 172.21.15.189:0/3394936712 >> v1:172.21.15.189:6789/0 conn(0x7f58f81318a0 legacy=0x7f58f8131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 1 -- 172.21.15.189:0/3394936712 shutdown_connections 2024-09-16T14:47:03.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 1 -- 172.21.15.189:0/3394936712 >> 172.21.15.189:0/3394936712 conn(0x7f58f80a3320 msgr2=0x7f58f809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:03.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 1 -- 172.21.15.189:0/3394936712 shutdown_connections 2024-09-16T14:47:03.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 1 -- 172.21.15.189:0/3394936712 wait complete. 2024-09-16T14:47:03.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient: build_initial_monmap 2024-09-16T14:47:03.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient: monmap: 2024-09-16T14:47:03.791 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:03.791 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:03.791 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:03.791 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:03.791 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:03.791 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:03.792 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:03.792 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:03.792 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:03.792 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:03.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 1 Processor -- start 2024-09-16T14:47:03.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 1 -- start start 2024-09-16T14:47:03.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient: init 2024-09-16T14:47:03.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:03.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:03.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient(hunting): picked mon.noname-b con 0x7f58f81318a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:03.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient(hunting): picked mon.noname-a con 0x7f58f8134d50 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:03.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient(hunting): picked mon.noname-c con 0x7f58f813b2b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:03.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f806b380 con 0x7f58f8134d50 2024-09-16T14:47:03.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f811b990 con 0x7f58f81318a0 2024-09-16T14:47:03.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f811a160 con 0x7f58f813b2b0 2024-09-16T14:47:03.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:03.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f5900168640 10 monclient(hunting): authenticate will time out at 1205.895571s 2024-09-16T14:47:03.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fdf04640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f58f81318a0 0x7f58f8119340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:42586/0 (socket says 172.21.15.189:42586) 2024-09-16T14:47:03.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58fdf04640 1 -- 172.21.15.189:0/1821466252 learned_addr learned my addr 172.21.15.189:0/1821466252 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:03.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316428045 0 0) 0x7f58f811b990 con 0x7f58f81318a0 2024-09-16T14:47:03.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58ee7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58ee7fc640 10 monclient(hunting): my global_id is 94828 2024-09-16T14:47:03.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58dc003530 con 0x7f58f81318a0 2024-09-16T14:47:03.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.775+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128095436 0 0) 0x7f58f811a160 con 0x7f58f813b2b0 2024-09-16T14:47:03.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): my global_id is 114212 2024-09-16T14:47:03.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58f811b990 con 0x7f58f813b2b0 2024-09-16T14:47:03.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243302425 0 0) 0x7f58f806b380 con 0x7f58f8134d50 2024-09-16T14:47:03.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): my global_id is 104415 2024-09-16T14:47:03.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58f811a160 con 0x7f58f8134d50 2024-09-16T14:47:03.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3370015005 0 0) 0x7f58dc003530 con 0x7f58f81318a0 2024-09-16T14:47:03.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58f806b380 con 0x7f58f81318a0 2024-09-16T14:47:03.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58f4001520 con 0x7f58f81318a0 2024-09-16T14:47:03.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:03.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): dump: 2024-09-16T14:47:03.795 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:03.795 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.795 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.795 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.795 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.795 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:03.795 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.795 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.795 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.796 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:03.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:03.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 370446032 0 0) 0x7f58f811b990 con 0x7f58f813b2b0 2024-09-16T14:47:03.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58dc003530 con 0x7f58f813b2b0 2024-09-16T14:47:03.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58f0001c40 con 0x7f58f813b2b0 2024-09-16T14:47:03.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:03.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): dump: 2024-09-16T14:47:03.796 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:03.796 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.796 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.797 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.797 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.797 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:03.797 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.797 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.797 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.797 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:03.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1017502891 0 0) 0x7f58f811a160 con 0x7f58f8134d50 2024-09-16T14:47:03.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58f811b990 con 0x7f58f8134d50 2024-09-16T14:47:03.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58e80018c0 con 0x7f58f8134d50 2024-09-16T14:47:03.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:03.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient(hunting): dump: 2024-09-16T14:47:03.798 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:03.798 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.798 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.798 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.798 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.798 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:03.798 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.798 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.798 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.799 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:03.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2852235444 0 0) 0x7f58f806b380 con 0x7f58f81318a0 2024-09-16T14:47:03.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 >> v1:172.21.15.26:6790/0 conn(0x7f58f813b2b0 legacy=0x7f58f81d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 >> v1:172.21.15.26:6789/0 conn(0x7f58f8134d50 legacy=0x7f58f8119a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:03.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 monclient: found mon.b 2024-09-16T14:47:03.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:03.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:03.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f80a1ec0 con 0x7f58f81318a0 2024-09-16T14:47:03.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:03.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:03.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:03.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f5900168640 5 monclient: authenticate success, global_id 94828 2024-09-16T14:47:03.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f5900168640 10 monclient: _renew_subs 2024-09-16T14:47:03.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f5900168640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:03.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f5900168640 1 -- 172.21.15.189:0/1821466252 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f813acf0 con 0x7f58f81318a0 2024-09-16T14:47:03.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f5900168640 10 monclient: _renew_subs 2024-09-16T14:47:03.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f5900168640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:03.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f5900168640 1 -- 172.21.15.189:0/1821466252 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f811b3d0 con 0x7f58f81318a0 2024-09-16T14:47:03.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58f4004aa0 con 0x7f58f81318a0 2024-09-16T14:47:03.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:03.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58f4004db0 con 0x7f58f81318a0 2024-09-16T14:47:03.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58effff640 4 set_mon_vals no callback set 2024-09-16T14:47:03.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:03.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 10 monclient: dump: 2024-09-16T14:47:03.801 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:03.801 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.801 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.801 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.801 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.801 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:03.801 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.801 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.801 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.801 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:03.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f58f4004f10 con 0x7f58f81318a0 2024-09-16T14:47:03.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f5900168640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:03.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f5900168640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:03.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f5900168640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:03.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f5900168640 1 -- 172.21.15.189:0/1821466252 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e0005050 con 0x7f58f81318a0 2024-09-16T14:47:03.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(110..110 src has 73..110) v4 ==== 5789+0+0 (unknown 4244852878 0 0) 0x7f58f4043b60 con 0x7f58f81318a0 2024-09-16T14:47:03.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.779+0000 7f58fd703640 10 monclient: get_auth_request con 0x7f58dc036420 auth_method 0 2024-09-16T14:47:03.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.783+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f58f40148c0 con 0x7f58f81318a0 2024-09-16T14:47:03.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.783+0000 7f58ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:03.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.783+0000 7f58ee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:03.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 10 monclient: get_monmap_and_config 2024-09-16T14:47:03.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 10 monclient: build_initial_monmap 2024-09-16T14:47:03.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 10 monclient: monmap: 2024-09-16T14:47:03.886 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:03.886 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.886 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:03.885891+0000 2024-09-16T14:47:03.886 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:03.885891+0000 2024-09-16T14:47:03.886 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:03.886 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:03.886 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:03.886 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:03.886 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:03.887 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:03.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 1 Processor -- start 2024-09-16T14:47:03.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 1 -- start start 2024-09-16T14:47:03.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 10 monclient: init 2024-09-16T14:47:03.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:03.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:03.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 10 monclient(hunting): picked mon.noname-c con 0x7ff09009a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 10 monclient(hunting): picked mon.noname-a con 0x7ff09009d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 10 monclient(hunting): picked mon.noname-b con 0x7ff09009b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09009dc60 con 0x7ff09009d1b0 2024-09-16T14:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff090099ae0 con 0x7ff09009a550 2024-09-16T14:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09009b8f0 con 0x7ff09009b040 2024-09-16T14:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 10 monclient(hunting): authenticate will time out at 1248.005901s 2024-09-16T14:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08f7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff09009d1b0 0x7ff090136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38552/0 (socket says 172.21.15.26:38552) 2024-09-16T14:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08f7fe640 1 -- 172.21.15.26:0/1783442364 learned_addr learned my addr 172.21.15.26:0/1783442364 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff094a12640 1 --1- 172.21.15.26:0/1783442364 >> v1:172.21.15.26:6790/0 conn(0x7ff09009b040 0x7ff09013a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35478/0 (socket says 172.21.15.26:35478) 2024-09-16T14:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 1 -- 172.21.15.26:0/1783442364 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155047244 0 0) 0x7ff09009dc60 con 0x7ff09009d1b0 2024-09-16T14:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient(hunting): my global_id is 104418 2024-09-16T14:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 1 -- 172.21.15.26:0/1783442364 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff07c003530 con 0x7ff09009d1b0 2024-09-16T14:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 1 -- 172.21.15.26:0/1783442364 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808767960 0 0) 0x7ff09009b8f0 con 0x7ff09009b040 2024-09-16T14:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient(hunting): my global_id is 114215 2024-09-16T14:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 1 -- 172.21.15.26:0/1783442364 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff09009dc60 con 0x7ff09009b040 2024-09-16T14:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 1 -- 172.21.15.26:0/1783442364 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 512899953 0 0) 0x7ff07c003530 con 0x7ff09009d1b0 2024-09-16T14:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 1 -- 172.21.15.26:0/1783442364 >> v1:172.21.15.26:6790/0 conn(0x7ff09009b040 legacy=0x7ff09013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 1 -- 172.21.15.26:0/1783442364 >> v1:172.21.15.189:6789/0 conn(0x7ff09009a550 legacy=0x7ff09009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 1 monclient: found mon.noname-a 2024-09-16T14:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 1 -- 172.21.15.26:0/1783442364 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09009bbd0 con 0x7ff09009d1b0 2024-09-16T14:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient: _finish_auth 0 2024-09-16T14:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 5 monclient: authenticate success, global_id 104418 2024-09-16T14:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 1 -- 172.21.15.26:0/1783442364 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff0780019a0 con 0x7ff09009d1b0 2024-09-16T14:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient: dump: 2024-09-16T14:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 1 -- 172.21.15.26:0/1783442364 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff078001c40 con 0x7ff09009d1b0 2024-09-16T14:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 1 -- 172.21.15.26:0/1783442364 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff078002010 con 0x7ff09009d1b0 2024-09-16T14:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 4 set_mon_vals no callback set 2024-09-16T14:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff08effd640 10 monclient: dump: 2024-09-16T14:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 10 monclient: shutdown 2024-09-16T14:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 1 -- 172.21.15.26:0/1783442364 >> v1:172.21.15.26:6789/0 conn(0x7ff09009d1b0 legacy=0x7ff090136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 1 -- 172.21.15.26:0/1783442364 shutdown_connections 2024-09-16T14:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 1 -- 172.21.15.26:0/1783442364 >> 172.21.15.26:0/1783442364 conn(0x7ff09009fb50 msgr2=0x7ff09009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 1 -- 172.21.15.26:0/1783442364 shutdown_connections 2024-09-16T14:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.883+0000 7ff096475640 1 -- 172.21.15.26:0/1783442364 wait complete. 2024-09-16T14:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 10 monclient: build_initial_monmap 2024-09-16T14:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 10 monclient: monmap: 2024-09-16T14:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 1 Processor -- start 2024-09-16T14:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 1 -- start start 2024-09-16T14:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 10 monclient: init 2024-09-16T14:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 10 monclient(hunting): picked mon.noname-a con 0x7ff09009a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 10 monclient(hunting): picked mon.noname-c con 0x7ff09009b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 10 monclient(hunting): picked mon.noname-b con 0x7ff09009d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09009a240 con 0x7ff09009a550 2024-09-16T14:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0901399a0 con 0x7ff09009d1b0 2024-09-16T14:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09013a380 con 0x7ff09009b040 2024-09-16T14:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 10 monclient(hunting): authenticate will time out at 1248.009071s 2024-09-16T14:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff09009a550 0x7ff09006c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38562/0 (socket says 172.21.15.26:38562) 2024-09-16T14:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08ffff640 1 -- 172.21.15.26:0/3195492655 learned_addr learned my addr 172.21.15.26:0/3195492655 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236172118 0 0) 0x7ff09013a380 con 0x7ff09009b040 2024-09-16T14:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): my global_id is 114218 2024-09-16T14:47:03.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff070003530 con 0x7ff09009b040 2024-09-16T14:47:03.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843465244 0 0) 0x7ff09009a240 con 0x7ff09009a550 2024-09-16T14:47:03.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): my global_id is 104421 2024-09-16T14:47:03.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff09013a380 con 0x7ff09009a550 2024-09-16T14:47:03.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2647339602 0 0) 0x7ff070003530 con 0x7ff09009b040 2024-09-16T14:47:03.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff09009a240 con 0x7ff09009b040 2024-09-16T14:47:03.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff078001f50 con 0x7ff09009b040 2024-09-16T14:47:03.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:03.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): dump: 2024-09-16T14:47:03.905 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:03.905 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.905 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.905 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.905 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.905 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:03.905 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.905 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.905 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.905 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:03.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:03.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4167476851 0 0) 0x7ff09013a380 con 0x7ff09009a550 2024-09-16T14:47:03.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff070003530 con 0x7ff09009a550 2024-09-16T14:47:03.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff0800019d0 con 0x7ff09009a550 2024-09-16T14:47:03.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:03.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): dump: 2024-09-16T14:47:03.906 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:03.908 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.908 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.908 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.908 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.908 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:03.908 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.908 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.908 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545916011 0 0) 0x7ff0901399a0 con 0x7ff09009d1b0 2024-09-16T14:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): my global_id is 94834 2024-09-16T14:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff09013a380 con 0x7ff09009d1b0 2024-09-16T14:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3518089431 0 0) 0x7ff09009a240 con 0x7ff09009b040 2024-09-16T14:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 >> v1:172.21.15.189:6789/0 conn(0x7ff09009d1b0 legacy=0x7ff0901d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 >> v1:172.21.15.26:6789/0 conn(0x7ff09009a550 legacy=0x7ff09006c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 monclient: found mon.c 2024-09-16T14:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09006d4f0 con 0x7ff09009b040 2024-09-16T14:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient: _finish_auth 0 2024-09-16T14:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 5 monclient: authenticate success, global_id 114218 2024-09-16T14:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 10 monclient: _renew_subs 2024-09-16T14:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 1 -- 172.21.15.26:0/3195492655 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0901380d0 con 0x7ff09009b040 2024-09-16T14:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 10 monclient: _renew_subs 2024-09-16T14:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 1 -- 172.21.15.26:0/3195492655 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff09006e370 con 0x7ff09009b040 2024-09-16T14:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff078004f80 con 0x7ff09009b040 2024-09-16T14:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff078001580 con 0x7ff09009b040 2024-09-16T14:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient: dump: 2024-09-16T14:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08e7fc640 4 set_mon_vals no callback set 2024-09-16T14:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff07800fba0 con 0x7ff09009b040 2024-09-16T14:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(110..110 src has 73..110) v4 ==== 5789+0+0 (unknown 4244852878 0 0) 0x7ff078043ca0 con 0x7ff09009b040 2024-09-16T14:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff096475640 1 -- 172.21.15.26:0/3195492655 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff09002c7d0 con 0x7ff09002c3f0 2024-09-16T14:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08ffff640 10 monclient: get_auth_request con 0x7ff070036470 auth_method 0 2024-09-16T14:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385366594 0 0) 0x7ff09002c7d0 con 0x7ff09002c3f0 2024-09-16T14:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient(hunting): my global_id is 104424 2024-09-16T14:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff09009a240 con 0x7ff09002c3f0 2024-09-16T14:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2426916062 0 0) 0x7ff09009a240 con 0x7ff09002c3f0 2024-09-16T14:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff09002c7d0 con 0x7ff09002c3f0 2024-09-16T14:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff088001770 con 0x7ff09002c3f0 2024-09-16T14:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 10 monclient: dump: 2024-09-16T14:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 964031891 0 0) 0x7ff09002c7d0 con 0x7ff09002c3f0 2024-09-16T14:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.887+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff09009c220 con 0x7ff09002c3f0 2024-09-16T14:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.891+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ff088001a30 con 0x7ff09002c3f0 2024-09-16T14:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.891+0000 7ff08cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.891+0000 7ff08cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.891+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 >> v1:172.21.15.26:6789/0 conn(0x7ff09002c3f0 legacy=0x7ff090061160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.891+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 >> v1:172.21.15.26:6789/0 conn(0x7ff09002c3f0 legacy=0x7ff090061160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff096475640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff096475640 1 -- 172.21.15.26:0/3195492655 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff090132730 con 0x7ff090132350 2024-09-16T14:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108173242 0 0) 0x7ff090132730 con 0x7ff090132350 2024-09-16T14:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 10 monclient(hunting): my global_id is 104427 2024-09-16T14:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff09002c7d0 con 0x7ff090132350 2024-09-16T14:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 524481630 0 0) 0x7ff09002c7d0 con 0x7ff090132350 2024-09-16T14:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff090132730 con 0x7ff090132350 2024-09-16T14:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff088004990 con 0x7ff090132350 2024-09-16T14:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 10 monclient: dump: 2024-09-16T14:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 660090319 0 0) 0x7ff090132730 con 0x7ff090132350 2024-09-16T14:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff090135190 con 0x7ff090132350 2024-09-16T14:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 2008629854) 0x7ff088004c10 con 0x7ff090132350 2024-09-16T14:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 >> v1:172.21.15.26:6789/0 conn(0x7ff090132350 legacy=0x7ff090134a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff08cff9640 1 -- 172.21.15.26:0/3195492655 >> v1:172.21.15.26:6789/0 conn(0x7ff090132350 legacy=0x7ff090134a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:03.920 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:47:03.920 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:03.920 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:03.920 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:47:03.920 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 44, 2024-09-16T14:47:03.920 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:47:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:47:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:47:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:47:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:47:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:47:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:47:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:47:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:47:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:47:03.932 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:47:03.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff096475640 1 -- 172.21.15.26:0/3195492655 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff070036470 legacy=0x7ff070038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff096475640 10 monclient: shutdown 2024-09-16T14:47:03.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff096475640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:03.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff096475640 1 -- 172.21.15.26:0/3195492655 >> v1:172.21.15.26:6790/0 conn(0x7ff09009b040 legacy=0x7ff09006cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:03.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff094a12640 1 -- 172.21.15.26:0/3195492655 reap_dead start 2024-09-16T14:47:03.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff096475640 1 -- 172.21.15.26:0/3195492655 shutdown_connections 2024-09-16T14:47:03.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff096475640 1 -- 172.21.15.26:0/3195492655 >> 172.21.15.26:0/3195492655 conn(0x7ff09009fb50 msgr2=0x7ff090070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:03.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff096475640 1 -- 172.21.15.26:0/3195492655 shutdown_connections 2024-09-16T14:47:03.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:03.895+0000 7ff096475640 1 -- 172.21.15.26:0/3195492655 wait complete. 2024-09-16T14:47:03.934 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:47:03.934 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:47:03.934 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:47:03.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.931+0000 7f5900168640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T14:47:03.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.931+0000 7f5900168640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T14:47:03.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.935+0000 7f5900168640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:03.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:03.935+0000 7f5900168640 1 -- 172.21.15.189:0/1821466252 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0} v 0) v1 -- 0x7f58e0005880 con 0x7f58f81318a0 2024-09-16T14:47:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 10 monclient: get_monmap_and_config 2024-09-16T14:47:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 10 monclient: build_initial_monmap 2024-09-16T14:47:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 10 monclient: monmap: 2024-09-16T14:47:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:04.050430+0000 2024-09-16T14:47:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:04.050430+0000 2024-09-16T14:47:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:04.051 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 1 Processor -- start 2024-09-16T14:47:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 1 -- start start 2024-09-16T14:47:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 10 monclient: init 2024-09-16T14:47:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 10 monclient(hunting): picked mon.noname-b con 0x7f04f00a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 10 monclient(hunting): picked mon.noname-c con 0x7f04f00a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 10 monclient(hunting): picked mon.noname-a con 0x7f04f00a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f00a3250 con 0x7f04f00a2800 2024-09-16T14:47:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f009f600 con 0x7f04f00a0000 2024-09-16T14:47:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f00a3830 con 0x7f04f00a1d10 2024-09-16T14:47:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 10 monclient(hunting): authenticate will time out at 1248.170487s 2024-09-16T14:47:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04eed9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f04f00a1d10 0x7f04f00a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35498/0 (socket says 172.21.15.26:35498) 2024-09-16T14:47:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04eed9d640 1 -- 172.21.15.26:0/2739844398 learned_addr learned my addr 172.21.15.26:0/2739844398 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 1 -- 172.21.15.26:0/2739844398 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524967327 0 0) 0x7f04f00a3830 con 0x7f04f00a1d10 2024-09-16T14:47:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:47:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient(hunting): my global_id is 114221 2024-09-16T14:47:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 1 -- 172.21.15.26:0/2739844398 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d8003530 con 0x7f04f00a1d10 2024-09-16T14:47:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 1 -- 172.21.15.26:0/2739844398 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164583612 0 0) 0x7f04f00a3250 con 0x7f04f00a2800 2024-09-16T14:47:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:47:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient(hunting): my global_id is 104430 2024-09-16T14:47:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 1 -- 172.21.15.26:0/2739844398 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f00a3830 con 0x7f04f00a2800 2024-09-16T14:47:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 1 -- 172.21.15.26:0/2739844398 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2789600270 0 0) 0x7f04d8003530 con 0x7f04f00a1d10 2024-09-16T14:47:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 1 -- 172.21.15.26:0/2739844398 >> v1:172.21.15.189:6789/0 conn(0x7f04f00a0000 legacy=0x7f04f0139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 1 -- 172.21.15.26:0/2739844398 >> v1:172.21.15.26:6789/0 conn(0x7f04f00a2800 legacy=0x7f04f013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient: _finish_hunting 0 2024-09-16T14:47:04.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 1 monclient: found mon.noname-b 2024-09-16T14:47:04.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:04.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:47:04.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 1 -- 172.21.15.26:0/2739844398 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f009ed60 con 0x7f04f00a1d10 2024-09-16T14:47:04.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient: _finish_auth 0 2024-09-16T14:47:04.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient: _check_auth_tickets 2024-09-16T14:47:04.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:04.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 1 -- 172.21.15.26:0/2739844398 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f04e0001620 con 0x7f04f00a1d10 2024-09-16T14:47:04.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:04.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient: dump: 2024-09-16T14:47:04.062 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.062 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.062 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.062 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.062 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.062 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.062 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.062 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.062 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.062 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 5 monclient: authenticate success, global_id 114221 2024-09-16T14:47:04.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:04.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 1 -- 172.21.15.26:0/2739844398 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04e0001900 con 0x7f04f00a1d10 2024-09-16T14:47:04.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:04.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 1 -- 172.21.15.26:0/2739844398 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f04e0001cd0 con 0x7f04f00a1d10 2024-09-16T14:47:04.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:04.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04edd9b640 10 monclient: dump: 2024-09-16T14:47:04.063 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.063 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.063 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.063 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.063 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.063 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.063 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.063 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.064 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.064 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:04.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.047+0000 7f04f5120640 4 set_mon_vals no callback set 2024-09-16T14:47:04.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 10 monclient: shutdown 2024-09-16T14:47:04.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:04.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 1 -- 172.21.15.26:0/2739844398 >> v1:172.21.15.26:6790/0 conn(0x7f04f00a1d10 legacy=0x7f04f00a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 1 -- 172.21.15.26:0/2739844398 shutdown_connections 2024-09-16T14:47:04.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 1 -- 172.21.15.26:0/2739844398 >> 172.21.15.26:0/2739844398 conn(0x7f04f00a7f30 msgr2=0x7f04f00a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:04.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 1 -- 172.21.15.26:0/2739844398 shutdown_connections 2024-09-16T14:47:04.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 1 -- 172.21.15.26:0/2739844398 wait complete. 2024-09-16T14:47:04.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 10 monclient: build_initial_monmap 2024-09-16T14:47:04.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 10 monclient: monmap: 2024-09-16T14:47:04.065 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:04.065 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:04.065 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:04.065 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:04.065 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:04.065 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:04.065 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:04.065 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:04.065 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:04.065 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 1 Processor -- start 2024-09-16T14:47:04.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 1 -- start start 2024-09-16T14:47:04.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 10 monclient: init 2024-09-16T14:47:04.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:04.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:04.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 10 monclient(hunting): picked mon.noname-b con 0x7f04f00a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:04.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 10 monclient(hunting): picked mon.noname-a con 0x7f04f00a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:04.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 10 monclient(hunting): picked mon.noname-c con 0x7f04f00a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:04.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f01366a0 con 0x7f04f00a1d10 2024-09-16T14:47:04.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f01349d0 con 0x7f04f00a0000 2024-09-16T14:47:04.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f00a62f0 con 0x7f04f00a2800 2024-09-16T14:47:04.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:04.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 10 monclient(hunting): authenticate will time out at 1248.173678s 2024-09-16T14:47:04.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04ee59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f04f00a1d10 0x7f04f01c8f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38594/0 (socket says 172.21.15.26:38594) 2024-09-16T14:47:04.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04ee59c640 1 -- 172.21.15.26:0/1725263257 learned_addr learned my addr 172.21.15.26:0/1725263257 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:04.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984424709 0 0) 0x7f04f01366a0 con 0x7f04f00a1d10 2024-09-16T14:47:04.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:04.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:04.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:04.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): my global_id is 104433 2024-09-16T14:47:04.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04dc003530 con 0x7f04f00a1d10 2024-09-16T14:47:04.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980127849 0 0) 0x7f04f00a62f0 con 0x7f04f00a2800 2024-09-16T14:47:04.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:04.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:04.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:04.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): my global_id is 114224 2024-09-16T14:47:04.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f01366a0 con 0x7f04f00a2800 2024-09-16T14:47:04.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2182145730 0 0) 0x7f04dc003530 con 0x7f04f00a1d10 2024-09-16T14:47:04.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04f00a62f0 con 0x7f04f00a1d10 2024-09-16T14:47:04.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f04e4001880 con 0x7f04f00a1d10 2024-09-16T14:47:04.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:04.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): dump: 2024-09-16T14:47:04.068 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.068 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.068 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.068 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.068 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.068 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.068 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.068 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.068 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.069 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:04.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2001504006 0 0) 0x7f04f01366a0 con 0x7f04f00a2800 2024-09-16T14:47:04.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04dc003530 con 0x7f04f00a2800 2024-09-16T14:47:04.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f04e80019c0 con 0x7f04f00a2800 2024-09-16T14:47:04.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:04.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): dump: 2024-09-16T14:47:04.069 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.069 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.069 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.069 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.070 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.070 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.070 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.070 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.070 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.070 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906613295 0 0) 0x7f04f01349d0 con 0x7f04f00a0000 2024-09-16T14:47:04.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:04.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:04.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:04.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): my global_id is 94840 2024-09-16T14:47:04.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f01366a0 con 0x7f04f00a0000 2024-09-16T14:47:04.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2377374177 0 0) 0x7f04f00a62f0 con 0x7f04f00a1d10 2024-09-16T14:47:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 >> v1:172.21.15.26:6790/0 conn(0x7f04f00a2800 legacy=0x7f04f01d12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 >> v1:172.21.15.189:6789/0 conn(0x7f04f00a0000 legacy=0x7f04f013d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 monclient: found mon.a 2024-09-16T14:47:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f009e500 con 0x7f04f00a1d10 2024-09-16T14:47:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 5 monclient: authenticate success, global_id 104433 2024-09-16T14:47:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 10 monclient: _renew_subs 2024-09-16T14:47:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:04.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 1 -- 172.21.15.26:0/1725263257 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f0136020 con 0x7f04f00a1d10 2024-09-16T14:47:04.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 10 monclient: _renew_subs 2024-09-16T14:47:04.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:04.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 1 -- 172.21.15.26:0/1725263257 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f01c96a0 con 0x7f04f00a1d10 2024-09-16T14:47:04.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04e40036a0 con 0x7f04f00a1d10 2024-09-16T14:47:04.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:04.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f04e4003a30 con 0x7f04f00a1d10 2024-09-16T14:47:04.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:04.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04ed59a640 4 set_mon_vals no callback set 2024-09-16T14:47:04.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient: dump: 2024-09-16T14:47:04.072 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.073 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04f5120640 1 -- 172.21.15.26:0/1725263257 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f04d0006590 con 0x7f04d0002830 2024-09-16T14:47:04.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f04e4003b90 con 0x7f04f00a1d10 2024-09-16T14:47:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(110..110 src has 73..110) v4 ==== 5789+0+0 (unknown 4244852878 0 0) 0x7f04e40438e0 con 0x7f04f00a1d10 2024-09-16T14:47:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542653299 0 0) 0x7f04d0006590 con 0x7f04d0002830 2024-09-16T14:47:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.051+0000 7f04d77fe640 10 monclient(hunting): my global_id is 114227 2024-09-16T14:47:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.055+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f00a62f0 con 0x7f04d0002830 2024-09-16T14:47:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.055+0000 7f04ef59e640 10 monclient: get_auth_request con 0x7f04dc0363f0 auth_method 0 2024-09-16T14:47:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.055+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 118406597 0 0) 0x7f04f00a62f0 con 0x7f04d0002830 2024-09-16T14:47:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.055+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04d0006590 con 0x7f04d0002830 2024-09-16T14:47:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.055+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f04e00015e0 con 0x7f04d0002830 2024-09-16T14:47:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.055+0000 7f04d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.055+0000 7f04d77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.055+0000 7f04d77fe640 10 monclient: dump: 2024-09-16T14:47:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.075 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.055+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3094593651 0 0) 0x7f04d0006590 con 0x7f04d0002830 2024-09-16T14:47:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.055+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04d000a0e0 con 0x7f04d0002830 2024-09-16T14:47:04.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.055+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f04e0001bc0 con 0x7f04d0002830 2024-09-16T14:47:04.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.055+0000 7f04d77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:04.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.055+0000 7f04d77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:04.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.055+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 >> v1:172.21.15.26:6790/0 conn(0x7f04d0002830 legacy=0x7f04d0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.055+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 >> v1:172.21.15.26:6790/0 conn(0x7f04d0002830 legacy=0x7f04d0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:04.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04f5120640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:47:04.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04f5120640 1 -- 172.21.15.26:0/1725263257 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f04d0009da0 con 0x7f04d00090c0 2024-09-16T14:47:04.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984442693 0 0) 0x7f04d0009da0 con 0x7f04d00090c0 2024-09-16T14:47:04.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:04.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:04.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:04.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 10 monclient(hunting): my global_id is 114230 2024-09-16T14:47:04.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d0006590 con 0x7f04d00090c0 2024-09-16T14:47:04.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 95779731 0 0) 0x7f04d0006590 con 0x7f04d00090c0 2024-09-16T14:47:04.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04d0009da0 con 0x7f04d00090c0 2024-09-16T14:47:04.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f04e0004b40 con 0x7f04d00090c0 2024-09-16T14:47:04.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:04.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 10 monclient: dump: 2024-09-16T14:47:04.078 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.078 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.079 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.079 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.079 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.079 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.079 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.079 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.079 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.079 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2973393053 0 0) 0x7f04d0009da0 con 0x7f04d00090c0 2024-09-16T14:47:04.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f04d000c290 con 0x7f04d00090c0 2024-09-16T14:47:04.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 3711353543) 0x7f04e0005f30 con 0x7f04d00090c0 2024-09-16T14:47:04.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:47:04.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:04.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.059+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 >> v1:172.21.15.26:6790/0 conn(0x7f04d00090c0 legacy=0x7f04d0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.063+0000 7f04d77fe640 1 -- 172.21.15.26:0/1725263257 >> v1:172.21.15.26:6790/0 conn(0x7f04d00090c0 legacy=0x7f04d0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:04.080 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:47:04.080 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:04.080 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:04.080 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:47:04.080 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 44, 2024-09-16T14:47:04.080 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:47:04.080 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:47:04.080 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:47:04.081 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:47:04.081 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:04.081 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:47:04.081 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:04.081 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:47:04.081 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:47:04.081 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:04.081 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:04.081 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:04.081 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:04.081 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:04.081 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:04.081 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:04.082 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:04.082 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:04.082 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:04.082 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:04.082 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:47:04.082 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:47:04.082 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:04.082 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:04.082 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:04.082 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:04.082 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:04.082 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:04.082 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:04.083 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:04.083 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:04.083 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:04.083 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:04.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.083 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:47:04.083 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:47:04.083 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:47:04.083 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:47:04.083 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:47:04.083 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:47:04.083 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:04.083 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:47:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:47:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:47:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:47:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:47:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:47:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:47:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:47:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:04.084 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:47:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:47:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:04.085 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:47:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:47:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:47:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:04.086 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:47:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:04.087 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:47:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:47:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:04.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:47:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:47:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:47:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:04.089 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:47:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:47:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:47:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:04.090 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.091 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:47:04.091 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:47:04.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.063+0000 7f04f5120640 1 -- 172.21.15.26:0/1725263257 >> v1:172.21.15.189:6812/3196161582 conn(0x7f04dc0363f0 legacy=0x7f04dc0388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.063+0000 7f04f5120640 10 monclient: shutdown 2024-09-16T14:47:04.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.063+0000 7f04f5120640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:04.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.063+0000 7f04f5120640 1 -- 172.21.15.26:0/1725263257 >> v1:172.21.15.26:6789/0 conn(0x7f04f00a1d10 legacy=0x7f04f01c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.063+0000 7f04ef59e640 1 -- 172.21.15.26:0/1725263257 reap_dead start 2024-09-16T14:47:04.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.063+0000 7f04ef59e640 1 -- 172.21.15.26:0/1725263257 reap_dead start 2024-09-16T14:47:04.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.063+0000 7f04f5120640 1 -- 172.21.15.26:0/1725263257 shutdown_connections 2024-09-16T14:47:04.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.063+0000 7f04f5120640 1 -- 172.21.15.26:0/1725263257 >> 172.21.15.26:0/1725263257 conn(0x7f04f00a7f30 msgr2=0x7f04f006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:04.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.063+0000 7f04f5120640 1 -- 172.21.15.26:0/1725263257 shutdown_connections 2024-09-16T14:47:04.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.063+0000 7f04f5120640 1 -- 172.21.15.26:0/1725263257 wait complete. 2024-09-16T14:47:04.093 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:47:04.093 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:47:04.093 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:47:04.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.203+0000 7f2859da9640 10 monclient: get_monmap_and_config 2024-09-16T14:47:04.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.203+0000 7f2859da9640 10 monclient: build_initial_monmap 2024-09-16T14:47:04.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.203+0000 7f2859da9640 10 monclient: monmap: 2024-09-16T14:47:04.209 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:04.209 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.210 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:04.209439+0000 2024-09-16T14:47:04.210 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:04.209439+0000 2024-09-16T14:47:04.210 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:04.210 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.210 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:04.210 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:04.210 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:04.210 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 1 Processor -- start 2024-09-16T14:47:04.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 1 -- start start 2024-09-16T14:47:04.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 10 monclient: init 2024-09-16T14:47:04.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:04.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:04.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 10 monclient(hunting): picked mon.noname-b con 0x7f28541339d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:04.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 10 monclient(hunting): picked mon.noname-c con 0x7f285409dcc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:04.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 10 monclient(hunting): picked mon.noname-a con 0x7f285413b120 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:04.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2854099e40 con 0x7f285413b120 2024-09-16T14:47:04.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28541344c0 con 0x7f285409dcc0 2024-09-16T14:47:04.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285409e0e0 con 0x7f28541339d0 2024-09-16T14:47:04.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:04.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 10 monclient(hunting): authenticate will time out at 1248.329707s 2024-09-16T14:47:04.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28537fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f28541339d0 0x7f2854133db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35528/0 (socket says 172.21.15.26:35528) 2024-09-16T14:47:04.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28537fe640 1 -- 172.21.15.26:0/951380002 learned_addr learned my addr 172.21.15.26:0/951380002 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:04.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 1 -- 172.21.15.26:0/951380002 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015188902 0 0) 0x7f2854099e40 con 0x7f285413b120 2024-09-16T14:47:04.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:04.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:04.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:04.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient(hunting): my global_id is 104436 2024-09-16T14:47:04.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 1 -- 172.21.15.26:0/951380002 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2840003530 con 0x7f285413b120 2024-09-16T14:47:04.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 1 -- 172.21.15.26:0/951380002 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015548939 0 0) 0x7f285409e0e0 con 0x7f28541339d0 2024-09-16T14:47:04.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:04.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:04.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:04.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient(hunting): my global_id is 114233 2024-09-16T14:47:04.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 1 -- 172.21.15.26:0/951380002 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2854099e40 con 0x7f28541339d0 2024-09-16T14:47:04.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 1 -- 172.21.15.26:0/951380002 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1825945730 0 0) 0x7f2840003530 con 0x7f285413b120 2024-09-16T14:47:04.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 1 -- 172.21.15.26:0/951380002 >> v1:172.21.15.26:6790/0 conn(0x7f28541339d0 legacy=0x7f2854133db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 1 -- 172.21.15.26:0/951380002 >> v1:172.21.15.189:6789/0 conn(0x7f285409dcc0 legacy=0x7f2854139a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:04.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 1 monclient: found mon.noname-a 2024-09-16T14:47:04.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:04.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:47:04.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 1 -- 172.21.15.26:0/951380002 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2854099b90 con 0x7f285413b120 2024-09-16T14:47:04.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:04.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:04.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:04.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 5 monclient: authenticate success, global_id 104436 2024-09-16T14:47:04.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 1 -- 172.21.15.26:0/951380002 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f284c001620 con 0x7f285413b120 2024-09-16T14:47:04.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:04.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient: dump: 2024-09-16T14:47:04.217 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.217 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.217 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.217 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.217 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.217 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.217 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.217 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.218 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.218 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:04.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 1 -- 172.21.15.26:0/951380002 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f284c0017d0 con 0x7f285413b120 2024-09-16T14:47:04.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:04.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 1 -- 172.21.15.26:0/951380002 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f284c001b60 con 0x7f285413b120 2024-09-16T14:47:04.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:04.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 4 set_mon_vals no callback set 2024-09-16T14:47:04.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:04.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f28527fc640 10 monclient: dump: 2024-09-16T14:47:04.219 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.219 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.219 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.219 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.219 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.219 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.219 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.219 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.219 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.220 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 10 monclient: shutdown 2024-09-16T14:47:04.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:04.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 1 -- 172.21.15.26:0/951380002 >> v1:172.21.15.26:6789/0 conn(0x7f285413b120 legacy=0x7f285413d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 1 -- 172.21.15.26:0/951380002 shutdown_connections 2024-09-16T14:47:04.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 1 -- 172.21.15.26:0/951380002 >> 172.21.15.26:0/951380002 conn(0x7f285409f900 msgr2=0x7f285409fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:04.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 1 -- 172.21.15.26:0/951380002 shutdown_connections 2024-09-16T14:47:04.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.207+0000 7f2859da9640 1 -- 172.21.15.26:0/951380002 wait complete. 2024-09-16T14:47:04.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 10 monclient: build_initial_monmap 2024-09-16T14:47:04.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 10 monclient: monmap: 2024-09-16T14:47:04.221 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:04.221 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:04.221 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:04.221 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:04.221 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:04.221 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:04.221 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:04.221 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:04.221 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:04.222 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 1 Processor -- start 2024-09-16T14:47:04.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 1 -- start start 2024-09-16T14:47:04.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 10 monclient: init 2024-09-16T14:47:04.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:04.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:04.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 10 monclient(hunting): picked mon.noname-b con 0x7f285409dcc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:04.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 10 monclient(hunting): picked mon.noname-c con 0x7f28541339d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:04.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 10 monclient(hunting): picked mon.noname-a con 0x7f285413b120 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:04.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285411d8e0 con 0x7f285413b120 2024-09-16T14:47:04.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285411dbc0 con 0x7f285409dcc0 2024-09-16T14:47:04.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285411c430 con 0x7f28541339d0 2024-09-16T14:47:04.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:04.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 10 monclient(hunting): authenticate will time out at 1248.333043s 2024-09-16T14:47:04.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2852ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f28541339d0 0x7f28541ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35534/0 (socket says 172.21.15.26:35534) 2024-09-16T14:47:04.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2852ffd640 1 -- 172.21.15.26:0/3451095399 learned_addr learned my addr 172.21.15.26:0/3451095399 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:04.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851655652 0 0) 0x7f285411c430 con 0x7f28541339d0 2024-09-16T14:47:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): my global_id is 114236 2024-09-16T14:47:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2834003560 con 0x7f28541339d0 2024-09-16T14:47:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273787321 0 0) 0x7f285411d8e0 con 0x7f285413b120 2024-09-16T14:47:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:04.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:04.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): my global_id is 104439 2024-09-16T14:47:04.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f285411c430 con 0x7f285413b120 2024-09-16T14:47:04.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4241624104 0 0) 0x7f2834003560 con 0x7f28541339d0 2024-09-16T14:47:04.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f285411d8e0 con 0x7f28541339d0 2024-09-16T14:47:04.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3269833912 0 0) 0x7f285411c430 con 0x7f285413b120 2024-09-16T14:47:04.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2834003560 con 0x7f285413b120 2024-09-16T14:47:04.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f283c001d50 con 0x7f28541339d0 2024-09-16T14:47:04.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:04.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): dump: 2024-09-16T14:47:04.232 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.232 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.232 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.232 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.232 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.232 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.232 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.232 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.232 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.232 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:04.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f284c001ae0 con 0x7f285413b120 2024-09-16T14:47:04.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:04.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): dump: 2024-09-16T14:47:04.233 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.233 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.233 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.233 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.233 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.233 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.233 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.233 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.233 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.233 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189976359 0 0) 0x7f285411dbc0 con 0x7f285409dcc0 2024-09-16T14:47:04.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:04.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:04.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:04.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): my global_id is 94846 2024-09-16T14:47:04.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f285411c430 con 0x7f285409dcc0 2024-09-16T14:47:04.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 723572245 0 0) 0x7f285411d8e0 con 0x7f28541339d0 2024-09-16T14:47:04.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 >> v1:172.21.15.189:6789/0 conn(0x7f285409dcc0 legacy=0x7f28541c9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 >> v1:172.21.15.26:6789/0 conn(0x7f285413b120 legacy=0x7f28541d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:47:04.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 monclient: found mon.c 2024-09-16T14:47:04.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:04.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:04.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285409a8a0 con 0x7f28541339d0 2024-09-16T14:47:04.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient: _finish_auth 0 2024-09-16T14:47:04.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:47:04.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:04.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 5 monclient: authenticate success, global_id 114236 2024-09-16T14:47:04.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 10 monclient: _renew_subs 2024-09-16T14:47:04.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:04.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 1 -- 172.21.15.26:0/3451095399 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f285409a390 con 0x7f28541339d0 2024-09-16T14:47:04.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 10 monclient: _renew_subs 2024-09-16T14:47:04.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:04.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 1 -- 172.21.15.26:0/3451095399 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f285411b8c0 con 0x7f28541339d0 2024-09-16T14:47:04.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:04.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2859da9640 1 -- 172.21.15.26:0/3451095399 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f285411b090 con 0x7f285411acb0 2024-09-16T14:47:04.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f283c002080 con 0x7f28541339d0 2024-09-16T14:47:04.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:04.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f283c003df0 con 0x7f28541339d0 2024-09-16T14:47:04.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:04.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient: dump: 2024-09-16T14:47:04.236 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.236 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.236 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.236 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.236 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.236 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.236 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.236 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.237 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.237 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2851ffb640 4 set_mon_vals no callback set 2024-09-16T14:47:04.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f283c003f90 con 0x7f28541339d0 2024-09-16T14:47:04.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(110..110 src has 73..110) v4 ==== 5789+0+0 (unknown 4244852878 0 0) 0x7f283c043de0 con 0x7f28541339d0 2024-09-16T14:47:04.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056672588 0 0) 0x7f285411b090 con 0x7f285411acb0 2024-09-16T14:47:04.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:04.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:04.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:04.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 10 monclient(hunting): my global_id is 94849 2024-09-16T14:47:04.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f285411d8e0 con 0x7f285411acb0 2024-09-16T14:47:04.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f2853fff640 10 monclient: get_auth_request con 0x7f28340363f0 auth_method 0 2024-09-16T14:47:04.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.211+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1108507156 0 0) 0x7f285411d8e0 con 0x7f285411acb0 2024-09-16T14:47:04.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.215+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f285411b090 con 0x7f285411acb0 2024-09-16T14:47:04.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.215+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2844000fc0 con 0x7f285411acb0 2024-09-16T14:47:04.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.215+0000 7f284bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.215+0000 7f284bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:04.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.215+0000 7f284bfff640 10 monclient: dump: 2024-09-16T14:47:04.238 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.238 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.238 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.238 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.238 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.238 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.238 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.238 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.238 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.239 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.215+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4190149430 0 0) 0x7f285411b090 con 0x7f285411acb0 2024-09-16T14:47:04.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.215+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f285411c270 con 0x7f285411acb0 2024-09-16T14:47:04.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.215+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2844001340 con 0x7f285411acb0 2024-09-16T14:47:04.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.215+0000 7f284bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:04.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.215+0000 7f284bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:04.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.215+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 >> v1:172.21.15.189:6789/0 conn(0x7f285411acb0 legacy=0x7f28541377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.215+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 >> v1:172.21.15.189:6789/0 conn(0x7f285411acb0 legacy=0x7f28541377b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:04.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.219+0000 7f2859da9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:47:04.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.219+0000 7f2859da9640 1 -- 172.21.15.26:0/3451095399 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f285406a180 con 0x7f28541c9c10 2024-09-16T14:47:04.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.219+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176308907 0 0) 0x7f285406a180 con 0x7f28541c9c10 2024-09-16T14:47:04.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.219+0000 7f284bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:04.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.219+0000 7f284bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:04.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.219+0000 7f284bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:04.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.219+0000 7f284bfff640 10 monclient(hunting): my global_id is 94852 2024-09-16T14:47:04.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.219+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f285411b090 con 0x7f28541c9c10 2024-09-16T14:47:04.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2578364663 0 0) 0x7f285411b090 con 0x7f28541c9c10 2024-09-16T14:47:04.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f285406a180 con 0x7f28541c9c10 2024-09-16T14:47:04.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2844004990 con 0x7f28541c9c10 2024-09-16T14:47:04.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f284bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f284bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:04.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f284bfff640 10 monclient: dump: 2024-09-16T14:47:04.240 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.240 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.240 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.240 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.241 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.241 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.241 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.241 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.241 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.241 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3574690620 0 0) 0x7f285406a180 con 0x7f28541c9c10 2024-09-16T14:47:04.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f285409f6b0 con 0x7f28541c9c10 2024-09-16T14:47:04.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3881 (unknown 0 0 2350293686) 0x7f2844004cd0 con 0x7f28541c9c10 2024-09-16T14:47:04.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f284bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:47:04.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f284bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:04.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 >> v1:172.21.15.189:6789/0 conn(0x7f28541c9c10 legacy=0x7f2854137ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f284bfff640 1 -- 172.21.15.26:0/3451095399 >> v1:172.21.15.189:6789/0 conn(0x7f28541c9c10 legacy=0x7f2854137ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:04.242 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:47:04.242 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:04.242 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:04.242 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:47:04.242 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 44, 2024-09-16T14:47:04.242 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:47:04.242 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:47:04.242 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:47:04.242 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:47:04.242 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:04.243 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-09-16T14:47:04.243 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:04.243 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:47:04.243 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:47:04.243 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:04.243 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:04.243 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:04.243 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:04.243 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:04.243 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:04.243 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:04.243 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:04.243 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:04.244 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:04.244 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:04.244 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:47:04.244 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:47:04.244 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:04.244 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:04.244 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:04.244 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:04.244 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:04.244 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:04.244 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:04.244 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:04.244 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:04.244 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:04.245 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:04.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.245 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:47:04.245 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:47:04.245 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:47:04.245 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:47:04.245 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:47:04.245 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:47:04.245 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:04.245 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:04.245 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:47:04.245 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:47:04.245 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:47:04.246 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:47:04.246 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:47:04.246 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:47:04.246 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:47:04.246 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:04.246 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:47:04.246 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:04.246 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:04.246 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:04.246 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:04.246 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:04.246 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:04.246 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:04.246 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:04.247 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:04.247 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:04.247 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:04.247 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:47:04.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.247 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:47:04.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:04.247 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:04.247 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:04.247 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:04.247 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:04.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:04.247 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:04.248 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:47:04.248 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:04.248 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:04.248 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:04.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.248 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:47:04.248 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:47:04.248 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:04.248 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:04.248 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:04.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:04.248 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:04.248 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:04.249 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:04.249 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:04.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:04.249 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:04.249 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:47:04.249 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:04.249 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:04.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:04.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.249 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:47:04.249 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:47:04.249 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:04.249 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:04.250 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:04.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:04.250 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:04.250 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:04.250 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:04.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:04.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:04.250 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:04.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:47:04.251 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:04.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:04.251 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:04.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:47:04.251 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:47:04.251 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:04.251 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:04.251 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:04.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:04.251 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:04.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.252 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:47:04.252 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:47:04.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:04.252 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:04.252 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:04.252 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:04.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:04.252 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:04.252 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:47:04.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:04.253 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:04.253 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:04.253 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:47:04.253 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:04.253 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:04.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:04.253 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:47:04.253 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:47:04.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f2859da9640 1 -- 172.21.15.26:0/3451095399 >> v1:172.21.15.189:6812/3196161582 conn(0x7f28340363f0 legacy=0x7f28340388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f2859da9640 10 monclient: shutdown 2024-09-16T14:47:04.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f2859da9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:04.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f2859da9640 1 -- 172.21.15.26:0/3451095399 >> v1:172.21.15.26:6790/0 conn(0x7f28541339d0 legacy=0x7f28541ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f2853fff640 1 -- 172.21.15.26:0/3451095399 reap_dead start 2024-09-16T14:47:04.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f2853fff640 1 -- 172.21.15.26:0/3451095399 reap_dead start 2024-09-16T14:47:04.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f2859da9640 1 -- 172.21.15.26:0/3451095399 shutdown_connections 2024-09-16T14:47:04.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f2859da9640 1 -- 172.21.15.26:0/3451095399 >> 172.21.15.26:0/3451095399 conn(0x7f285409f900 msgr2=0x7f28540a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:04.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f2859da9640 1 -- 172.21.15.26:0/3451095399 shutdown_connections 2024-09-16T14:47:04.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.223+0000 7f2859da9640 1 -- 172.21.15.26:0/3451095399 wait complete. 2024-09-16T14:47:04.256 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:47:04.257 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T14:47:04.257 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:47:04.257 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T14:47:04.257 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:47:04.257 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T14:47:04.257 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:47:04.270 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T14:47:04.270 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:47:04.271 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:47:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 10 monclient: get_monmap_and_config 2024-09-16T14:47:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 10 monclient: build_initial_monmap 2024-09-16T14:47:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 10 monclient: monmap: 2024-09-16T14:47:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:04.426503+0000 2024-09-16T14:47:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:04.426503+0000 2024-09-16T14:47:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:04.427 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 1 Processor -- start 2024-09-16T14:47:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 1 -- start start 2024-09-16T14:47:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 10 monclient: init 2024-09-16T14:47:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 10 monclient(hunting): picked mon.noname-a con 0x7fc7d409a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 10 monclient(hunting): picked mon.noname-b con 0x7fc7d409d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 10 monclient(hunting): picked mon.noname-c con 0x7fc7d409b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d409dc60 con 0x7fc7d409a550 2024-09-16T14:47:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d4099ae0 con 0x7fc7d409b040 2024-09-16T14:47:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d409b8f0 con 0x7fc7d409d1b0 2024-09-16T14:47:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 10 monclient(hunting): authenticate will time out at 1248.546758s 2024-09-16T14:47:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d2d9d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fc7d409a550 legacy=0x7fc7d409a930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7cbfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc7d409d1b0 0x7fc7d4136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35578/0 (socket says 172.21.15.26:35578) 2024-09-16T14:47:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7cbfff640 1 -- 172.21.15.26:0/3918848927 learned_addr learned my addr 172.21.15.26:0/3918848927 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 1 -- 172.21.15.26:0/3918848927 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785766127 0 0) 0x7fc7d409b8f0 con 0x7fc7d409d1b0 2024-09-16T14:47:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 10 monclient(hunting): _negotiate 2024-09-16T14:47:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 10 monclient(hunting): my global_id is 114239 2024-09-16T14:47:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 1 -- 172.21.15.26:0/3918848927 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c0005740 con 0x7fc7d409d1b0 2024-09-16T14:47:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 1 -- 172.21.15.26:0/3918848927 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4151763482 0 0) 0x7fc7c0005740 con 0x7fc7d409d1b0 2024-09-16T14:47:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 1 -- 172.21.15.26:0/3918848927 >> v1:172.21.15.189:6789/0 conn(0x7fc7d409b040 legacy=0x7fc7d413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 1 -- 172.21.15.26:0/3918848927 >> v1:172.21.15.26:6789/0 conn(0x7fc7d409a550 legacy=0x7fc7d409a930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:04.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 10 monclient: _finish_hunting 0 2024-09-16T14:47:04.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 1 monclient: found mon.noname-b 2024-09-16T14:47:04.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:04.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:47:04.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 1 -- 172.21.15.26:0/3918848927 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d409bbd0 con 0x7fc7d409d1b0 2024-09-16T14:47:04.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 10 monclient: _finish_auth 0 2024-09-16T14:47:04.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 10 monclient: _check_auth_tickets 2024-09-16T14:47:04.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:04.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 1 -- 172.21.15.26:0/3918848927 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc7bc001620 con 0x7fc7d409d1b0 2024-09-16T14:47:04.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 5 monclient: authenticate success, global_id 114239 2024-09-16T14:47:04.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:04.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:04.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 10 monclient: dump: 2024-09-16T14:47:04.437 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.437 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.437 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.437 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.437 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.437 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.437 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.437 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.437 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.437 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:04.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 1 -- 172.21.15.26:0/3918848927 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7bc001940 con 0x7fc7d409d1b0 2024-09-16T14:47:04.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:04.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:04.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 1 -- 172.21.15.26:0/3918848927 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc7bc001cd0 con 0x7fc7d409d1b0 2024-09-16T14:47:04.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d909c640 4 set_mon_vals no callback set 2024-09-16T14:47:04.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:04.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.423+0000 7fc7d259c640 10 monclient: dump: 2024-09-16T14:47:04.438 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.438 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.438 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.438 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.438 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.439 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.439 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.439 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.439 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.439 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient: shutdown 2024-09-16T14:47:04.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:04.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 1 -- 172.21.15.26:0/3918848927 >> v1:172.21.15.26:6790/0 conn(0x7fc7d409d1b0 legacy=0x7fc7d4136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 1 -- 172.21.15.26:0/3918848927 shutdown_connections 2024-09-16T14:47:04.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 1 -- 172.21.15.26:0/3918848927 >> 172.21.15.26:0/3918848927 conn(0x7fc7d409fb50 msgr2=0x7fc7d409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:04.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 1 -- 172.21.15.26:0/3918848927 shutdown_connections 2024-09-16T14:47:04.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 1 -- 172.21.15.26:0/3918848927 wait complete. 2024-09-16T14:47:04.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient: build_initial_monmap 2024-09-16T14:47:04.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient: monmap: 2024-09-16T14:47:04.440 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:04.440 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:04.440 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:04.440 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:04.440 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:04.440 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:04.440 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:04.440 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:04.440 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:04.440 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 1 Processor -- start 2024-09-16T14:47:04.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 1 -- start start 2024-09-16T14:47:04.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient: init 2024-09-16T14:47:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient(hunting): picked mon.noname-a con 0x7fc7d409a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient(hunting): picked mon.noname-c con 0x7fc7d409b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient(hunting): picked mon.noname-b con 0x7fc7d409d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d409a240 con 0x7fc7d409a550 2024-09-16T14:47:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d2d9d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fc7d409a550 legacy=0x7fc7d41b46f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d41b5580 con 0x7fc7d409d1b0 2024-09-16T14:47:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d40a2480 con 0x7fc7d409b040 2024-09-16T14:47:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient(hunting): authenticate will time out at 1248.549836s 2024-09-16T14:47:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cbfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc7d409b040 0x7fc7d41c9ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35586/0 (socket says 172.21.15.26:35586) 2024-09-16T14:47:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cbfff640 1 -- 172.21.15.26:0/3978598470 learned_addr learned my addr 172.21.15.26:0/3978598470 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 1 -- 172.21.15.26:0/3978598470 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620233007 0 0) 0x7fc7d40a2480 con 0x7fc7d409b040 2024-09-16T14:47:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 10 monclient(hunting): my global_id is 114242 2024-09-16T14:47:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 1 -- 172.21.15.26:0/3978598470 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b4003530 con 0x7fc7d409b040 2024-09-16T14:47:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 1 -- 172.21.15.26:0/3978598470 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 355085920 0 0) 0x7fc7b4003530 con 0x7fc7d409b040 2024-09-16T14:47:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 1 -- 172.21.15.26:0/3978598470 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7d40a2480 con 0x7fc7d409b040 2024-09-16T14:47:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 1 -- 172.21.15.26:0/3978598470 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc7bc0015e0 con 0x7fc7d409b040 2024-09-16T14:47:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 10 monclient(hunting): dump: 2024-09-16T14:47:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.443 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 1 -- 172.21.15.26:0/3978598470 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3983377566 0 0) 0x7fc7d40a2480 con 0x7fc7d409b040 2024-09-16T14:47:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 1 -- 172.21.15.26:0/3978598470 >> v1:172.21.15.189:6789/0 conn(0x7fc7d409d1b0 legacy=0x7fc7d41cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 1 -- 172.21.15.26:0/3978598470 >> v1:172.21.15.26:6789/0 conn(0x7fc7d409a550 legacy=0x7fc7d41b46f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 1 monclient: found mon.c 2024-09-16T14:47:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 1 -- 172.21.15.26:0/3978598470 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d41a30c0 con 0x7fc7d409b040 2024-09-16T14:47:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 5 monclient: authenticate success, global_id 114242 2024-09-16T14:47:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient: _renew_subs 2024-09-16T14:47:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 1 -- 172.21.15.26:0/3978598470 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d41a16b0 con 0x7fc7d409b040 2024-09-16T14:47:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient: _renew_subs 2024-09-16T14:47:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 1 -- 172.21.15.26:0/3978598470 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d41a0d60 con 0x7fc7d409b040 2024-09-16T14:47:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 1 -- 172.21.15.26:0/3978598470 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7bc004c00 con 0x7fc7d409b040 2024-09-16T14:47:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 1 -- 172.21.15.26:0/3978598470 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc7bc004f10 con 0x7fc7d409b040 2024-09-16T14:47:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d159a640 4 set_mon_vals no callback set 2024-09-16T14:47:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 10 monclient: dump: 2024-09-16T14:47:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:04.446 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 1 -- 172.21.15.26:0/3978598470 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc7bc005070 con 0x7fc7d409b040 2024-09-16T14:47:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7d909c640 1 -- 172.21.15.26:0/3978598470 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b8005050 con 0x7fc7d409b040 2024-09-16T14:47:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.427+0000 7fc7cb7fe640 1 -- 172.21.15.26:0/3978598470 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(110..110 src has 73..110) v4 ==== 5789+0+0 (unknown 4244852878 0 0) 0x7fc7bc043bf0 con 0x7fc7d409b040 2024-09-16T14:47:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.431+0000 7fc7d2d9d640 10 monclient: get_auth_request con 0x7fc7b4036050 auth_method 0 2024-09-16T14:47:04.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.435+0000 7fc7cb7fe640 1 -- 172.21.15.26:0/3978598470 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc7bc014950 con 0x7fc7d409b040 2024-09-16T14:47:04.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.435+0000 7fc7cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:04.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.435+0000 7fc7cb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:04.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.651+0000 7fc7d909c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:47:04.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.651+0000 7fc7d909c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:04.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.651+0000 7fc7d909c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:04.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.651+0000 7fc7d909c640 1 -- 172.21.15.26:0/3978598470 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc7b8005880 con 0x7fc7d409b040 2024-09-16T14:47:04.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.651+0000 7fc7cb7fe640 1 -- 172.21.15.26:0/3978598470 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2423729818) 0x7fc7bc012dc0 con 0x7fc7d409b040 2024-09-16T14:47:04.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.651+0000 7fc7cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:04.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.651+0000 7fc7cb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:04.656 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:47:04.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.655+0000 7fc7d909c640 1 -- 172.21.15.26:0/3978598470 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc7b4036050 legacy=0x7fc7b4038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.655+0000 7fc7d909c640 10 monclient: shutdown 2024-09-16T14:47:04.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.655+0000 7fc7d909c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:04.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.655+0000 7fc7d909c640 1 -- 172.21.15.26:0/3978598470 >> v1:172.21.15.26:6790/0 conn(0x7fc7d409b040 legacy=0x7fc7d41c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:04.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.655+0000 7fc7d909c640 1 -- 172.21.15.26:0/3978598470 shutdown_connections 2024-09-16T14:47:04.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.655+0000 7fc7d909c640 1 -- 172.21.15.26:0/3978598470 >> 172.21.15.26:0/3978598470 conn(0x7fc7d409fb50 msgr2=0x7fc7d41b7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:04.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.655+0000 7fc7d909c640 1 -- 172.21.15.26:0/3978598470 shutdown_connections 2024-09-16T14:47:04.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:04.655+0000 7fc7d909c640 1 -- 172.21.15.26:0/3978598470 wait complete. 2024-09-16T14:47:06.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:06.775+0000 7f58ed7fa640 10 monclient: tick 2024-09-16T14:47:06.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:06.775+0000 7f58ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:06.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:06.775+0000 7f58ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:06.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:06.775+0000 7f58ed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:07.674 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:47:07.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:07.774+0000 7f58ed7fa640 10 monclient: tick 2024-09-16T14:47:07.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:07.774+0000 7f58ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:07.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:07.774+0000 7f58ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:07.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0b49fb640 10 monclient: get_monmap_and_config 2024-09-16T14:47:07.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0b49fb640 10 monclient: build_initial_monmap 2024-09-16T14:47:07.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0b49fb640 10 monclient: monmap: 2024-09-16T14:47:07.803 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:07.803 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:07.804 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:07.803516+0000 2024-09-16T14:47:07.804 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:07.803516+0000 2024-09-16T14:47:07.804 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:07.804 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:07.804 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:07.804 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:07.804 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:07.804 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:07.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0b49fb640 1 Processor -- start 2024-09-16T14:47:07.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0b49fb640 1 -- start start 2024-09-16T14:47:07.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0b49fb640 10 monclient: init 2024-09-16T14:47:07.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0b49fb640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:07.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0b49fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:07.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0b49fb640 10 monclient(hunting): picked mon.noname-c con 0x7ff0b009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:07.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0b49fb640 10 monclient(hunting): picked mon.noname-b con 0x7ff0b0132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:07.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0b49fb640 10 monclient(hunting): picked mon.noname-a con 0x7ff0b0138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:07.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0b49fb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b009d260 con 0x7ff0b0138ff0 2024-09-16T14:47:07.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0b49fb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b0132a80 con 0x7ff0b009cb80 2024-09-16T14:47:07.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0b49fb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b009cf60 con 0x7ff0b0132650 2024-09-16T14:47:07.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0b49fb640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:07.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0aed9d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7ff0b0138ff0 legacy=0x7ff0b013b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:07.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0b49fb640 10 monclient(hunting): authenticate will time out at 1251.923789s 2024-09-16T14:47:07.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0add9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff0b0132650 0x7ff0b01378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:44034/0 (socket says 172.21.15.26:44034) 2024-09-16T14:47:07.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.799+0000 7ff0add9b640 1 -- 172.21.15.26:0/2464880600 learned_addr learned my addr 172.21.15.26:0/2464880600 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:07.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 1 -- 172.21.15.26:0/2464880600 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338359236 0 0) 0x7ff0b0132a80 con 0x7ff0b009cb80 2024-09-16T14:47:07.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:07.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:07.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:07.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 10 monclient(hunting): my global_id is 94861 2024-09-16T14:47:07.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 1 -- 172.21.15.26:0/2464880600 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff09c003530 con 0x7ff0b009cb80 2024-09-16T14:47:07.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 1 -- 172.21.15.26:0/2464880600 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 257826492 0 0) 0x7ff09c003530 con 0x7ff0b009cb80 2024-09-16T14:47:07.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 1 -- 172.21.15.26:0/2464880600 >> v1:172.21.15.26:6790/0 conn(0x7ff0b0132650 legacy=0x7ff0b01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:07.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 1 -- 172.21.15.26:0/2464880600 >> v1:172.21.15.26:6789/0 conn(0x7ff0b0138ff0 legacy=0x7ff0b013b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:07.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 10 monclient: _finish_hunting 0 2024-09-16T14:47:07.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 1 monclient: found mon.noname-c 2024-09-16T14:47:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 1 -- 172.21.15.26:0/2464880600 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b009f2d0 con 0x7ff0b009cb80 2024-09-16T14:47:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 10 monclient: _finish_auth 0 2024-09-16T14:47:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 10 monclient: _check_auth_tickets 2024-09-16T14:47:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 5 monclient: authenticate success, global_id 94861 2024-09-16T14:47:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 1 -- 172.21.15.26:0/2464880600 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff0a00019a0 con 0x7ff0b009cb80 2024-09-16T14:47:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 10 monclient: dump: 2024-09-16T14:47:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:07.812 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 1 -- 172.21.15.26:0/2464880600 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0a0001c50 con 0x7ff0b009cb80 2024-09-16T14:47:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 4 set_mon_vals no callback set 2024-09-16T14:47:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 1 -- 172.21.15.26:0/2464880600 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff0a0001fe0 con 0x7ff0b009cb80 2024-09-16T14:47:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ad59a640 10 monclient: dump: 2024-09-16T14:47:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:07.814 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 10 monclient: shutdown 2024-09-16T14:47:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 1 -- 172.21.15.26:0/2464880600 >> v1:172.21.15.189:6789/0 conn(0x7ff0b009cb80 legacy=0x7ff0b00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 1 -- 172.21.15.26:0/2464880600 shutdown_connections 2024-09-16T14:47:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 1 -- 172.21.15.26:0/2464880600 >> 172.21.15.26:0/2464880600 conn(0x7ff0b009fb50 msgr2=0x7ff0b009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 1 -- 172.21.15.26:0/2464880600 shutdown_connections 2024-09-16T14:47:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 1 -- 172.21.15.26:0/2464880600 wait complete. 2024-09-16T14:47:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 10 monclient: build_initial_monmap 2024-09-16T14:47:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 10 monclient: monmap: 2024-09-16T14:47:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:07.815 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 1 Processor -- start 2024-09-16T14:47:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 1 -- start start 2024-09-16T14:47:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 10 monclient: init 2024-09-16T14:47:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 10 monclient(hunting): picked mon.noname-c con 0x7ff0b0132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 10 monclient(hunting): picked mon.noname-a con 0x7ff0b0138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 10 monclient(hunting): picked mon.noname-b con 0x7ff0b0138950 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b0071260 con 0x7ff0b0138ff0 2024-09-16T14:47:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b006e3e0 con 0x7ff0b0138950 2024-09-16T14:47:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b006fd40 con 0x7ff0b0132650 2024-09-16T14:47:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0b49fb640 10 monclient(hunting): authenticate will time out at 1251.927362s 2024-09-16T14:47:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0add9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7ff0b0138ff0 legacy=0x7ff0b006dcd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ae59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff0b0132650 0x7ff0b006d5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:44038/0 (socket says 172.21.15.26:44038) 2024-09-16T14:47:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.803+0000 7ff0ae59c640 1 -- 172.21.15.26:0/1291082179 learned_addr learned my addr 172.21.15.26:0/1291082179 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 1 -- 172.21.15.26:0/1291082179 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96397326 0 0) 0x7ff0b006e3e0 con 0x7ff0b0138950 2024-09-16T14:47:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 10 monclient(hunting): my global_id is 94864 2024-09-16T14:47:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 1 -- 172.21.15.26:0/1291082179 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff090003530 con 0x7ff0b0138950 2024-09-16T14:47:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 1 -- 172.21.15.26:0/1291082179 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3023895600 0 0) 0x7ff090003530 con 0x7ff0b0138950 2024-09-16T14:47:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 1 -- 172.21.15.26:0/1291082179 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0b006e3e0 con 0x7ff0b0138950 2024-09-16T14:47:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 1 -- 172.21.15.26:0/1291082179 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff0a8003f40 con 0x7ff0b0138950 2024-09-16T14:47:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 10 monclient(hunting): dump: 2024-09-16T14:47:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:07.819 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 1 -- 172.21.15.26:0/1291082179 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2432563003 0 0) 0x7ff0b006e3e0 con 0x7ff0b0138950 2024-09-16T14:47:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 1 -- 172.21.15.26:0/1291082179 >> v1:172.21.15.26:6790/0 conn(0x7ff0b0132650 legacy=0x7ff0b006d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 1 -- 172.21.15.26:0/1291082179 >> v1:172.21.15.26:6789/0 conn(0x7ff0b0138ff0 legacy=0x7ff0b006dcd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 1 monclient: found mon.b 2024-09-16T14:47:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 1 -- 172.21.15.26:0/1291082179 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b013c400 con 0x7ff0b0138950 2024-09-16T14:47:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:47:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0b49fb640 5 monclient: authenticate success, global_id 94864 2024-09-16T14:47:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0b49fb640 10 monclient: _renew_subs 2024-09-16T14:47:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0b49fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0b49fb640 1 -- 172.21.15.26:0/1291082179 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b0070f80 con 0x7ff0b0138950 2024-09-16T14:47:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0b49fb640 10 monclient: _renew_subs 2024-09-16T14:47:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0b49fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0b49fb640 1 -- 172.21.15.26:0/1291082179 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b006f170 con 0x7ff0b0138950 2024-09-16T14:47:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 1 -- 172.21.15.26:0/1291082179 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0a8004410 con 0x7ff0b0138950 2024-09-16T14:47:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 1 -- 172.21.15.26:0/1291082179 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff0a8004720 con 0x7ff0b0138950 2024-09-16T14:47:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a7fff640 4 set_mon_vals no callback set 2024-09-16T14:47:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 10 monclient: dump: 2024-09-16T14:47:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:07.821 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:07.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0b49fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:07.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0b49fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0b49fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0b49fb640 1 -- 172.21.15.26:0/1291082179 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff094005050 con 0x7ff0b0138950 2024-09-16T14:47:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 1 -- 172.21.15.26:0/1291082179 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff0a8004900 con 0x7ff0b0138950 2024-09-16T14:47:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0a6ffd640 1 -- 172.21.15.26:0/1291082179 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(110..110 src has 73..110) v4 ==== 5789+0+0 (unknown 4244852878 0 0) 0x7ff0a80436b0 con 0x7ff0b0138950 2024-09-16T14:47:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.807+0000 7ff0ae59c640 10 monclient: get_auth_request con 0x7ff0900360a0 auth_method 0 2024-09-16T14:47:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.819+0000 7ff0a6ffd640 1 -- 172.21.15.26:0/1291082179 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff0a80144c0 con 0x7ff0b0138950 2024-09-16T14:47:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.819+0000 7ff0a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:07.819+0000 7ff0a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:08.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:08.039+0000 7ff0b49fb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:47:08.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:08.039+0000 7ff0b49fb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:08.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:08.039+0000 7ff0b49fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:08.039+0000 7ff0b49fb640 1 -- 172.21.15.26:0/1291082179 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff094005880 con 0x7ff0b0138950 2024-09-16T14:47:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:08.039+0000 7ff0a6ffd640 1 -- 172.21.15.26:0/1291082179 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3231441083) 0x7ff0a8012930 con 0x7ff0b0138950 2024-09-16T14:47:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:08.039+0000 7ff0a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:08.039+0000 7ff0a6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:08.045 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:47:08.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:08.043+0000 7ff0b49fb640 1 -- 172.21.15.26:0/1291082179 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff0900360a0 legacy=0x7ff090038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:08.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:08.043+0000 7ff0b49fb640 10 monclient: shutdown 2024-09-16T14:47:08.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:08.043+0000 7ff0b49fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:08.043+0000 7ff0b49fb640 1 -- 172.21.15.26:0/1291082179 >> v1:172.21.15.189:6789/0 conn(0x7ff0b0138950 legacy=0x7ff0b01cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:08.043+0000 7ff0b49fb640 1 -- 172.21.15.26:0/1291082179 shutdown_connections 2024-09-16T14:47:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:08.043+0000 7ff0b49fb640 1 -- 172.21.15.26:0/1291082179 >> 172.21.15.26:0/1291082179 conn(0x7ff0b009fb50 msgr2=0x7ff0b0130750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:08.043+0000 7ff0b49fb640 1 -- 172.21.15.26:0/1291082179 shutdown_connections 2024-09-16T14:47:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:08.043+0000 7ff0b49fb640 1 -- 172.21.15.26:0/1291082179 wait complete. 2024-09-16T14:47:08.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:08.778+0000 7f58ed7fa640 10 monclient: tick 2024-09-16T14:47:08.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:08.778+0000 7f58ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:08.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:08.778+0000 7f58ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:09.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:09.778+0000 7f58ed7fa640 10 monclient: tick 2024-09-16T14:47:09.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:09.778+0000 7f58ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:09.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:09.778+0000 7f58ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:10.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:10.778+0000 7f58ed7fa640 10 monclient: tick 2024-09-16T14:47:10.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:10.778+0000 7f58ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:10.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:10.778+0000 7f58ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:11.064 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:47:11.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b97bc640 10 monclient: get_monmap_and_config 2024-09-16T14:47:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b97bc640 10 monclient: build_initial_monmap 2024-09-16T14:47:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b97bc640 10 monclient: monmap: 2024-09-16T14:47:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:11.198842+0000 2024-09-16T14:47:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:11.198842+0000 2024-09-16T14:47:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:11.199 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:11.199 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:11.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b97bc640 1 Processor -- start 2024-09-16T14:47:11.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b97bc640 1 -- start start 2024-09-16T14:47:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b97bc640 10 monclient: init 2024-09-16T14:47:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b97bc640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b97bc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b97bc640 10 monclient(hunting): picked mon.noname-c con 0x7f40b409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b97bc640 10 monclient(hunting): picked mon.noname-b con 0x7f40b4132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b97bc640 10 monclient(hunting): picked mon.noname-a con 0x7f40b4138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b97bc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b409d260 con 0x7f40b4138ff0 2024-09-16T14:47:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b97bc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b4132a80 con 0x7f40b409cb80 2024-09-16T14:47:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b97bc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b409cf60 con 0x7f40b4132650 2024-09-16T14:47:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b97bc640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b97bc640 10 monclient(hunting): authenticate will time out at 1255.318878s 2024-09-16T14:47:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b37fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f40b4138ff0 legacy=0x7f40b413b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:11.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b27fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f40b4132650 0x7f40b41378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:44054/0 (socket says 172.21.15.26:44054) 2024-09-16T14:47:11.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.195+0000 7f40b27fc640 1 -- 172.21.15.26:0/1669121055 learned_addr learned my addr 172.21.15.26:0/1669121055 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:11.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.399+0000 7f40b37fe640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6789/0 conn(0x7f40b4138ff0 legacy=0x7f40b413b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:11.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:11.778+0000 7f58ed7fa640 10 monclient: tick 2024-09-16T14:47:11.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:11.778+0000 7f58ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:11.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:11.778+0000 7f58ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:11.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:11.803+0000 7f40b37fe640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6789/0 conn(0x7f40b4138ff0 legacy=0x7f40b413b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:12.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:12.607+0000 7f40b37fe640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6789/0 conn(0x7f40b4138ff0 legacy=0x7f40b413b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:12.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:12.778+0000 7f58ed7fa640 10 monclient: tick 2024-09-16T14:47:12.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:12.778+0000 7f58ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:12.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:12.778+0000 7f58ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:13.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:13.778+0000 7f58ed7fa640 10 monclient: tick 2024-09-16T14:47:13.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:13.778+0000 7f58ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:13.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:13.778+0000 7f58ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:14.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.195+0000 7f40b0ff9640 10 monclient(hunting): tick 2024-09-16T14:47:14.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.195+0000 7f40b0ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:47:14.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.195+0000 7f40b0ff9640 1 monclient(hunting): continuing hunt 2024-09-16T14:47:14.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.195+0000 7f40b0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:47:14.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.195+0000 7f40b0ff9640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6790/0 conn(0x7f40b4132650 legacy=0x7f40b41378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:14.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.195+0000 7f40b0ff9640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.189:6789/0 conn(0x7f40b409cb80 legacy=0x7f40b40717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:14.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.195+0000 7f40b0ff9640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6789/0 conn(0x7f40b4138ff0 legacy=0x7f40b413b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:14.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.195+0000 7f40b0ff9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:14.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.195+0000 7f40b0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f40a00036d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:14.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.195+0000 7f40b0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f40a0007360 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:14.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.195+0000 7f40b0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f40b409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:14.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.195+0000 7f40b0ff9640 1 -- 172.21.15.26:0/1669121055 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b409f2d0 con 0x7f40a00036d0 2024-09-16T14:47:14.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.195+0000 7f40b0ff9640 1 -- 172.21.15.26:0/1669121055 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b409d260 con 0x7f40b409cb80 2024-09-16T14:47:14.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.195+0000 7f40b0ff9640 1 -- 172.21.15.26:0/1669121055 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40a00108b0 con 0x7f40a0007360 2024-09-16T14:47:14.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.195+0000 7f40b0ff9640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:14.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.195+0000 7f40b2ffd640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6789/0 conn(0x7f40a00036d0 legacy=0x7f40a0005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:14.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.399+0000 7f40b2ffd640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6789/0 conn(0x7f40a00036d0 legacy=0x7f40a0005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:14.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:14.778+0000 7f58ed7fa640 10 monclient: tick 2024-09-16T14:47:14.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:14.778+0000 7f58ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:14.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:14.778+0000 7f58ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:14.803+0000 7f40b2ffd640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6789/0 conn(0x7f40a00036d0 legacy=0x7f40a0005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:15.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:15.607+0000 7f40b2ffd640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6789/0 conn(0x7f40a00036d0 legacy=0x7f40a0005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:15.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:15.778+0000 7f58ed7fa640 10 monclient: tick 2024-09-16T14:47:15.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:15.778+0000 7f58ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:15.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:15.778+0000 7f58ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:16.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:16.778+0000 7f58ed7fa640 10 monclient: tick 2024-09-16T14:47:16.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:16.778+0000 7f58ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:16.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:16.778+0000 7f58ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:16.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:16.778+0000 7f58ed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:17.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.195+0000 7f40b0ff9640 10 monclient(hunting): tick 2024-09-16T14:47:17.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.195+0000 7f40b0ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:47:17.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.195+0000 7f40b0ff9640 1 monclient(hunting): continuing hunt 2024-09-16T14:47:17.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.195+0000 7f40b0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:47:17.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.195+0000 7f40b0ff9640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6790/0 conn(0x7f40a0007360 legacy=0x7f40a00097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:17.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.195+0000 7f40b0ff9640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.189:6789/0 conn(0x7f40b409cb80 legacy=0x7f40a000d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:17.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.195+0000 7f40b0ff9640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6789/0 conn(0x7f40a00036d0 legacy=0x7f40a0005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:17.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.195+0000 7f40b0ff9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:17.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.195+0000 7f40b0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f40a000f930 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:17.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.195+0000 7f40b0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f40b409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:17.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.195+0000 7f40b0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f40a0007360 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:17.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.195+0000 7f40b0ff9640 1 -- 172.21.15.26:0/1669121055 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40a0011b20 con 0x7f40a000f930 2024-09-16T14:47:17.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.195+0000 7f40b0ff9640 1 -- 172.21.15.26:0/1669121055 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b409f2d0 con 0x7f40b409cb80 2024-09-16T14:47:17.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.195+0000 7f40b0ff9640 1 -- 172.21.15.26:0/1669121055 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40a0010600 con 0x7f40a0007360 2024-09-16T14:47:17.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.195+0000 7f40b0ff9640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:17.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.195+0000 7f40b2ffd640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6789/0 conn(0x7f40a000f930 legacy=0x7f40a000fd70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.399+0000 7f40b2ffd640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6789/0 conn(0x7f40a000f930 legacy=0x7f40a000fd70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:17.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:17.782+0000 7f58ed7fa640 10 monclient: tick 2024-09-16T14:47:17.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:17.782+0000 7f58ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:17.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:17.782+0000 7f58ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:17.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:17.803+0000 7f40b2ffd640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6789/0 conn(0x7f40a000f930 legacy=0x7f40a000fd70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:18.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.607+0000 7f40b2ffd640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6789/0 conn(0x7f40a000f930 legacy=0x7f40a000fd70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:18.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:18.778+0000 7f58ed7fa640 10 monclient: tick 2024-09-16T14:47:18.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:18.778+0000 7f58ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:18.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:18.778+0000 7f58ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:18.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.807+0000 7f40b1ffb640 1 -- 172.21.15.26:0/1669121055 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969756416 0 0) 0x7f40b409f2d0 con 0x7f40b409cb80 2024-09-16T14:47:18.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.807+0000 7f40b1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:18.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.807+0000 7f40b1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:18.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.807+0000 7f40b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:18.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.807+0000 7f40b1ffb640 10 monclient(hunting): my global_id is 94867 2024-09-16T14:47:18.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.807+0000 7f40b1ffb640 1 -- 172.21.15.26:0/1669121055 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f409c003530 con 0x7f40b409cb80 2024-09-16T14:47:18.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.811+0000 7f40b1ffb640 1 -- 172.21.15.26:0/1669121055 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230929115 0 0) 0x7f40a0010600 con 0x7f40a0007360 2024-09-16T14:47:18.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.811+0000 7f40b1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:18.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.811+0000 7f40b1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:18.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.811+0000 7f40b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:18.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.811+0000 7f40b1ffb640 10 monclient(hunting): my global_id is 114245 2024-09-16T14:47:18.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.811+0000 7f40b1ffb640 1 -- 172.21.15.26:0/1669121055 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b409f2d0 con 0x7f40a0007360 2024-09-16T14:47:18.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 1 -- 172.21.15.26:0/1669121055 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2555057796 0 0) 0x7f409c003530 con 0x7f40b409cb80 2024-09-16T14:47:18.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6790/0 conn(0x7f40a0007360 legacy=0x7f40a00097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:18.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.26:6789/0 conn(0x7f40a000f930 legacy=0x7f40a000fd70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:18.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:47:18.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 1 monclient: found mon.noname-c 2024-09-16T14:47:18.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:18.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:18.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 1 -- 172.21.15.26:0/1669121055 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a0004b20 con 0x7f40b409cb80 2024-09-16T14:47:18.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:47:18.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:18.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:18.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 1 -- 172.21.15.26:0/1669121055 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f40a8001520 con 0x7f40b409cb80 2024-09-16T14:47:18.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:18.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:18.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 10 monclient: dump: 2024-09-16T14:47:18.824 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:18.824 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:18.824 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:18.824 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:18.824 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:18.824 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:18.824 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:18.824 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:18.824 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:18.824 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:18.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b97bc640 5 monclient: authenticate success, global_id 94867 2024-09-16T14:47:18.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b97bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:18.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 1 -- 172.21.15.26:0/1669121055 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40a8001700 con 0x7f40b409cb80 2024-09-16T14:47:18.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:18.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 1 -- 172.21.15.26:0/1669121055 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f40a8001ad0 con 0x7f40b409cb80 2024-09-16T14:47:18.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:18.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:18.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b1ffb640 10 monclient: dump: 2024-09-16T14:47:18.825 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:18.825 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:18.825 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:18.825 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:18.825 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:18.825 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:18.825 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:18.825 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:18.826 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:18.826 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:18.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b97bc640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:18.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b97bc640 4 set_mon_vals no callback set 2024-09-16T14:47:18.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b97bc640 10 monclient: shutdown 2024-09-16T14:47:18.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b97bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:18.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b97bc640 1 -- 172.21.15.26:0/1669121055 >> v1:172.21.15.189:6789/0 conn(0x7f40b409cb80 legacy=0x7f40a000d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:18.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b97bc640 1 -- 172.21.15.26:0/1669121055 shutdown_connections 2024-09-16T14:47:18.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b97bc640 1 -- 172.21.15.26:0/1669121055 >> 172.21.15.26:0/1669121055 conn(0x7f40b409fb50 msgr2=0x7f40b409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:18.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b97bc640 1 -- 172.21.15.26:0/1669121055 shutdown_connections 2024-09-16T14:47:18.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b97bc640 1 -- 172.21.15.26:0/1669121055 wait complete. 2024-09-16T14:47:18.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b97bc640 10 monclient: build_initial_monmap 2024-09-16T14:47:18.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b97bc640 10 monclient: monmap: 2024-09-16T14:47:18.831 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:18.831 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:18.831 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:18.831 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:18.831 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:18.831 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:18.831 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:18.831 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:18.832 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:18.832 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:18.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b97bc640 1 Processor -- start 2024-09-16T14:47:18.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b97bc640 1 -- start start 2024-09-16T14:47:18.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.819+0000 7f40b97bc640 10 monclient: init 2024-09-16T14:47:18.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:18.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:18.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 10 monclient(hunting): picked mon.noname-b con 0x7f40a0007360 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:18.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 10 monclient(hunting): picked mon.noname-c con 0x7f40a000f930 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:18.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 10 monclient(hunting): picked mon.noname-a con 0x7f40b409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:18.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b41386b0 con 0x7f40b409cb80 2024-09-16T14:47:18.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b40a2920 con 0x7f40a0007360 2024-09-16T14:47:18.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b409a240 con 0x7f40a000f930 2024-09-16T14:47:18.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:18.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 10 monclient(hunting): authenticate will time out at 1262.944681s 2024-09-16T14:47:18.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b37fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f40b409cb80 legacy=0x7f40b41d2190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:18.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b27fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f40a000f930 0x7f40b41cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49328/0 (socket says 172.21.15.26:49328) 2024-09-16T14:47:18.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b27fc640 1 -- 172.21.15.26:0/2746573196 learned_addr learned my addr 172.21.15.26:0/2746573196 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:18.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568809143 0 0) 0x7f40b409a240 con 0x7f40a000f930 2024-09-16T14:47:18.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:18.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:18.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:18.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient(hunting): my global_id is 114248 2024-09-16T14:47:18.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40a0010600 con 0x7f40a000f930 2024-09-16T14:47:18.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289244080 0 0) 0x7f40b40a2920 con 0x7f40a0007360 2024-09-16T14:47:18.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:18.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:18.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:18.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient(hunting): my global_id is 94870 2024-09-16T14:47:18.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b409a240 con 0x7f40a0007360 2024-09-16T14:47:18.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3107269965 0 0) 0x7f40a0010600 con 0x7f40a000f930 2024-09-16T14:47:18.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40b40a2920 con 0x7f40a000f930 2024-09-16T14:47:18.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f40a8001a10 con 0x7f40a000f930 2024-09-16T14:47:18.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:18.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:18.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient(hunting): dump: 2024-09-16T14:47:18.838 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:18.838 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:18.838 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:18.838 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:18.838 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:18.838 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:18.838 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:18.838 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:18.838 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:18.838 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3821563358 0 0) 0x7f40b40a2920 con 0x7f40a000f930 2024-09-16T14:47:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 >> v1:172.21.15.189:6789/0 conn(0x7f40a0007360 legacy=0x7f40b4137c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 >> v1:172.21.15.26:6789/0 conn(0x7f40b409cb80 legacy=0x7f40b41d2190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 1 monclient: found mon.c 2024-09-16T14:47:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b4136760 con 0x7f40a000f930 2024-09-16T14:47:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 5 monclient: authenticate success, global_id 114248 2024-09-16T14:47:18.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 10 monclient: _renew_subs 2024-09-16T14:47:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 1 -- 172.21.15.26:0/2746573196 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b4135ec0 con 0x7f40a000f930 2024-09-16T14:47:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 10 monclient: _renew_subs 2024-09-16T14:47:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 1 -- 172.21.15.26:0/2746573196 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b41c70a0 con 0x7f40a000f930 2024-09-16T14:47:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40a8001cd0 con 0x7f40a000f930 2024-09-16T14:47:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f40a8004c40 con 0x7f40a000f930 2024-09-16T14:47:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b17fa640 4 set_mon_vals no callback set 2024-09-16T14:47:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 10 monclient: dump: 2024-09-16T14:47:18.840 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:18.841 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:18.841 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:18.841 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:18.841 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:18.841 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:18.841 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:18.841 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:18.841 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:18.841 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:18.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f40a800f6b0 con 0x7f40a000f930 2024-09-16T14:47:18.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:18.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:18.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:18.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b97bc640 1 -- 172.21.15.26:0/2746573196 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4094005050 con 0x7f40a000f930 2024-09-16T14:47:18.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(111..111 src has 73..111) v4 ==== 5789+0+0 (unknown 2962608226 0 0) 0x7f40a8043960 con 0x7f40a000f930 2024-09-16T14:47:18.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.823+0000 7f40b2ffd640 10 monclient: get_auth_request con 0x7f40a0037490 auth_method 0 2024-09-16T14:47:18.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.831+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f40a8010b10 con 0x7f40a000f930 2024-09-16T14:47:18.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.831+0000 7f409b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:18.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:18.831+0000 7f409b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:19.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.059+0000 7f40b97bc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:47:19.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.059+0000 7f40b97bc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:19.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.059+0000 7f40b97bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:19.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.059+0000 7f40b97bc640 1 -- 172.21.15.26:0/2746573196 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4094005880 con 0x7f40a000f930 2024-09-16T14:47:19.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.059+0000 7f409b7fe640 1 -- 172.21.15.26:0/2746573196 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 629885942) 0x7f40a8014310 con 0x7f40a000f930 2024-09-16T14:47:19.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.059+0000 7f409b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:19.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.059+0000 7f409b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:19.072 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":46,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:47:19.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.063+0000 7f40b97bc640 1 -- 172.21.15.26:0/2746573196 >> v1:172.21.15.189:6812/3196161582 conn(0x7f40a0037490 legacy=0x7f40a0039950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:19.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.063+0000 7f40b97bc640 10 monclient: shutdown 2024-09-16T14:47:19.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.063+0000 7f40b97bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:19.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.063+0000 7f40b97bc640 1 -- 172.21.15.26:0/2746573196 >> v1:172.21.15.26:6790/0 conn(0x7f40a000f930 legacy=0x7f40b41cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:19.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.063+0000 7f40b97bc640 1 -- 172.21.15.26:0/2746573196 shutdown_connections 2024-09-16T14:47:19.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.063+0000 7f40b97bc640 1 -- 172.21.15.26:0/2746573196 >> 172.21.15.26:0/2746573196 conn(0x7f40b409fb50 msgr2=0x7f40b40a16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:19.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.063+0000 7f40b97bc640 1 -- 172.21.15.26:0/2746573196 shutdown_connections 2024-09-16T14:47:19.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.063+0000 7f40b97bc640 1 -- 172.21.15.26:0/2746573196 wait complete. 2024-09-16T14:47:19.086 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:47:19.086 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:47:19.087 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:47:19.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 10 monclient: get_monmap_and_config 2024-09-16T14:47:19.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 10 monclient: build_initial_monmap 2024-09-16T14:47:19.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 10 monclient: monmap: 2024-09-16T14:47:19.206 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:19.206 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:19.206 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:19.206052+0000 2024-09-16T14:47:19.206 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:19.206052+0000 2024-09-16T14:47:19.206 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:19.206 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:19.206 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:19.206 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:19.206 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:19.206 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:19.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 1 Processor -- start 2024-09-16T14:47:19.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 1 -- start start 2024-09-16T14:47:19.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 10 monclient: init 2024-09-16T14:47:19.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:19.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:19.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 10 monclient(hunting): picked mon.noname-a con 0x7f696c0a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:19.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 10 monclient(hunting): picked mon.noname-c con 0x7f696c0a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:19.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 10 monclient(hunting): picked mon.noname-b con 0x7f696c0a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:19.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c0a3250 con 0x7f696c0a1d10 2024-09-16T14:47:19.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c09f600 con 0x7f696c0a0000 2024-09-16T14:47:19.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c0a3830 con 0x7f696c0a2800 2024-09-16T14:47:19.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:19.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 10 monclient(hunting): authenticate will time out at 1263.326093s 2024-09-16T14:47:19.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f69709f9640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f696c0a1d10 legacy=0x7f696c0a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:19.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f69711fa640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f696c0a2800 0x7f696c13d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49334/0 (socket says 172.21.15.26:49334) 2024-09-16T14:47:19.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f69711fa640 1 -- 172.21.15.26:0/3894928474 learned_addr learned my addr 172.21.15.26:0/3894928474 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:19.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 1 -- 172.21.15.26:0/3894928474 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853419350 0 0) 0x7f696c0a3830 con 0x7f696c0a2800 2024-09-16T14:47:19.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:19.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:19.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:19.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient(hunting): my global_id is 114251 2024-09-16T14:47:19.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 1 -- 172.21.15.26:0/3894928474 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6954003530 con 0x7f696c0a2800 2024-09-16T14:47:19.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 1 -- 172.21.15.26:0/3894928474 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029855377 0 0) 0x7f696c09f600 con 0x7f696c0a0000 2024-09-16T14:47:19.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:19.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:19.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:19.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient(hunting): my global_id is 94873 2024-09-16T14:47:19.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 1 -- 172.21.15.26:0/3894928474 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f696c0a3830 con 0x7f696c0a0000 2024-09-16T14:47:19.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 1 -- 172.21.15.26:0/3894928474 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 236140281 0 0) 0x7f6954003530 con 0x7f696c0a2800 2024-09-16T14:47:19.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 1 -- 172.21.15.26:0/3894928474 >> v1:172.21.15.189:6789/0 conn(0x7f696c0a0000 legacy=0x7f696c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:19.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 1 -- 172.21.15.26:0/3894928474 >> v1:172.21.15.26:6789/0 conn(0x7f696c0a1d10 legacy=0x7f696c0a20f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:19.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:19.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 1 monclient: found mon.noname-b 2024-09-16T14:47:19.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:19.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:47:19.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 1 -- 172.21.15.26:0/3894928474 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c09ed60 con 0x7f696c0a2800 2024-09-16T14:47:19.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:19.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:19.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:19.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 1 -- 172.21.15.26:0/3894928474 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6964001620 con 0x7f696c0a2800 2024-09-16T14:47:19.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:19.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:19.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient: dump: 2024-09-16T14:47:19.212 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:19.212 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:19.212 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.212 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.212 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:19.212 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:19.212 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:19.213 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:19.213 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:19.213 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:19.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 5 monclient: authenticate success, global_id 114251 2024-09-16T14:47:19.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:19.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 1 -- 172.21.15.26:0/3894928474 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6964001900 con 0x7f696c0a2800 2024-09-16T14:47:19.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:19.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 1 -- 172.21.15.26:0/3894928474 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6964001c90 con 0x7f696c0a2800 2024-09-16T14:47:19.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:19.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 4 set_mon_vals no callback set 2024-09-16T14:47:19.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:19.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:19.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f696b7fe640 10 monclient: dump: 2024-09-16T14:47:19.213 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:19.214 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:19.214 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.214 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.214 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:19.214 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:19.214 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:19.214 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:19.214 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:19.214 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:19.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 10 monclient: shutdown 2024-09-16T14:47:19.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:19.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 1 -- 172.21.15.26:0/3894928474 >> v1:172.21.15.26:6790/0 conn(0x7f696c0a2800 legacy=0x7f696c13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:19.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 1 -- 172.21.15.26:0/3894928474 shutdown_connections 2024-09-16T14:47:19.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 1 -- 172.21.15.26:0/3894928474 >> 172.21.15.26:0/3894928474 conn(0x7f696c0a7f30 msgr2=0x7f696c0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:19.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 1 -- 172.21.15.26:0/3894928474 shutdown_connections 2024-09-16T14:47:19.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.203+0000 7f6972c5d640 1 -- 172.21.15.26:0/3894928474 wait complete. 2024-09-16T14:47:19.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 10 monclient: build_initial_monmap 2024-09-16T14:47:19.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 10 monclient: monmap: 2024-09-16T14:47:19.219 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:19.219 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:19.219 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:19.219 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:19.219 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:19.219 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:19.219 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:19.219 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:19.219 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:19.219 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:19.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 1 Processor -- start 2024-09-16T14:47:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 1 -- start start 2024-09-16T14:47:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 10 monclient: init 2024-09-16T14:47:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 10 monclient(hunting): picked mon.noname-c con 0x7f696c0a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 10 monclient(hunting): picked mon.noname-a con 0x7f696c0a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 10 monclient(hunting): picked mon.noname-b con 0x7f696c0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c133cc0 con 0x7f696c0a1d10 2024-09-16T14:47:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c09e570 con 0x7f696c0a2800 2024-09-16T14:47:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c131290 con 0x7f696c0a0000 2024-09-16T14:47:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 10 monclient(hunting): authenticate will time out at 1263.329400s 2024-09-16T14:47:19.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f696bfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f696c0a1d10 legacy=0x7f696c130b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69709f9640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f696c0a0000 0x7f696c130470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49338/0 (socket says 172.21.15.26:49338) 2024-09-16T14:47:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69709f9640 1 -- 172.21.15.26:0/406517086 learned_addr learned my addr 172.21.15.26:0/406517086 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873257953 0 0) 0x7f696c131290 con 0x7f696c0a0000 2024-09-16T14:47:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient(hunting): my global_id is 114254 2024-09-16T14:47:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6958003530 con 0x7f696c0a0000 2024-09-16T14:47:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 417414347 0 0) 0x7f6958003530 con 0x7f696c0a0000 2024-09-16T14:47:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f696c131290 con 0x7f696c0a0000 2024-09-16T14:47:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f695c004160 con 0x7f696c0a0000 2024-09-16T14:47:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:19.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:19.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient(hunting): dump: 2024-09-16T14:47:19.222 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:19.222 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:19.222 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.222 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.222 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:19.222 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:19.222 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:19.222 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:19.222 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:19.222 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:19.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:19.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173794322 0 0) 0x7f696c09ed60 con 0x7f696c0a2800 2024-09-16T14:47:19.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:19.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:19.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:19.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient(hunting): my global_id is 94876 2024-09-16T14:47:19.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6958003530 con 0x7f696c0a2800 2024-09-16T14:47:19.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2875859091 0 0) 0x7f696c131290 con 0x7f696c0a0000 2024-09-16T14:47:19.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 >> v1:172.21.15.189:6789/0 conn(0x7f696c0a2800 legacy=0x7f696c1d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:19.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 >> v1:172.21.15.26:6789/0 conn(0x7f696c0a1d10 legacy=0x7f696c130b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:19.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient: _finish_hunting 0 2024-09-16T14:47:19.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 monclient: found mon.c 2024-09-16T14:47:19.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:19.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:19.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c132a40 con 0x7f696c0a0000 2024-09-16T14:47:19.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient: _finish_auth 0 2024-09-16T14:47:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 5 monclient: authenticate success, global_id 114254 2024-09-16T14:47:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 10 monclient: _renew_subs 2024-09-16T14:47:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 1 -- 172.21.15.26:0/406517086 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c133700 con 0x7f696c0a0000 2024-09-16T14:47:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 10 monclient: _renew_subs 2024-09-16T14:47:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 1 -- 172.21.15.26:0/406517086 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c131de0 con 0x7f696c0a0000 2024-09-16T14:47:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f695c004480 con 0x7f696c0a0000 2024-09-16T14:47:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f695c004810 con 0x7f696c0a0000 2024-09-16T14:47:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:19.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f696a7fc640 4 set_mon_vals no callback set 2024-09-16T14:47:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient: dump: 2024-09-16T14:47:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:19.225 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:19.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f6972c5d640 1 -- 172.21.15.26:0/406517086 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f694c006590 con 0x7f694c002830 2024-09-16T14:47:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f695c004970 con 0x7f696c0a0000 2024-09-16T14:47:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(111..111 src has 73..111) v4 ==== 5789+0+0 (unknown 2962608226 0 0) 0x7f695c043ac0 con 0x7f696c0a0000 2024-09-16T14:47:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705011108 0 0) 0x7f694c006590 con 0x7f694c002830 2024-09-16T14:47:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient(hunting): my global_id is 114257 2024-09-16T14:47:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f696c131290 con 0x7f694c002830 2024-09-16T14:47:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1534692465 0 0) 0x7f696c131290 con 0x7f694c002830 2024-09-16T14:47:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f694c006590 con 0x7f694c002830 2024-09-16T14:47:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f69600019f0 con 0x7f694c002830 2024-09-16T14:47:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:19.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 10 monclient: dump: 2024-09-16T14:47:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:19.227 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69711fa640 10 monclient: get_auth_request con 0x7f6958036130 auth_method 0 2024-09-16T14:47:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2741946810 0 0) 0x7f694c006590 con 0x7f694c002830 2024-09-16T14:47:19.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.207+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f694c00a0e0 con 0x7f694c002830 2024-09-16T14:47:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.211+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6960001df0 con 0x7f694c002830 2024-09-16T14:47:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.211+0000 7f69697fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.211+0000 7f69697fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.211+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 >> v1:172.21.15.26:6790/0 conn(0x7f694c002830 legacy=0x7f694c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.211+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 >> v1:172.21.15.26:6790/0 conn(0x7f694c002830 legacy=0x7f694c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f6972c5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:47:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f6972c5d640 1 -- 172.21.15.26:0/406517086 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f694c009da0 con 0x7f694c0090c0 2024-09-16T14:47:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024536774 0 0) 0x7f694c009da0 con 0x7f694c0090c0 2024-09-16T14:47:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:19.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 10 monclient(hunting): my global_id is 114260 2024-09-16T14:47:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f694c006590 con 0x7f694c0090c0 2024-09-16T14:47:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2420202077 0 0) 0x7f694c006590 con 0x7f694c0090c0 2024-09-16T14:47:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f694c009da0 con 0x7f694c0090c0 2024-09-16T14:47:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6960003b40 con 0x7f694c0090c0 2024-09-16T14:47:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 10 monclient: dump: 2024-09-16T14:47:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:19.231 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2070233302 0 0) 0x7f694c009da0 con 0x7f694c0090c0 2024-09-16T14:47:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f694c00c290 con 0x7f694c0090c0 2024-09-16T14:47:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4202 (unknown 0 0 1483867670) 0x7f6960005b50 con 0x7f694c0090c0 2024-09-16T14:47:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:47:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 >> v1:172.21.15.26:6790/0 conn(0x7f694c0090c0 legacy=0x7f694c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.215+0000 7f69697fa640 1 -- 172.21.15.26:0/406517086 >> v1:172.21.15.26:6790/0 conn(0x7f694c0090c0 legacy=0x7f694c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:19.232 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:47:19.232 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:19.232 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:19.232 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:47:19.232 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 46, 2024-09-16T14:47:19.232 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:47:19.232 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:47:19.232 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:47:19.232 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:19.232 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:47:19.232 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:19.232 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:47:19.233 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:47:19.233 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:19.233 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:19.233 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:19.233 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:19.233 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:19.233 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:19.233 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:19.233 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:19.233 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:19.233 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:19.233 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:19.233 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:47:19.234 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:47:19.234 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:19.234 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:19.234 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:19.234 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:19.234 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:19.234 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:19.234 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:19.234 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:19.234 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:19.235 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:19.235 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:19.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.235 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:47:19.235 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:47:19.235 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:47:19.235 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:47:19.235 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:47:19.235 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:47:19.235 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:19.235 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:19.235 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:47:19.235 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:47:19.236 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:47:19.236 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:47:19.236 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:47:19.236 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:47:19.236 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:47:19.236 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:19.236 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:47:19.236 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:19.236 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:19.236 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:19.236 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:19.236 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:19.236 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:19.236 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:19.237 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:19.237 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:19.237 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:19.237 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:19.237 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:47:19.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.237 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:47:19.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.237 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:19.237 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:19.237 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:19.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:19.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.237 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:19.238 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:47:19.238 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:19.238 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:19.238 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:19.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.238 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:47:19.238 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:47:19.238 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:19.238 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:19.238 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:19.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.238 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:19.239 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:19.239 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:19.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:19.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.239 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:19.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:47:19.239 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:19.239 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:19.239 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:19.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:47:19.239 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:47:19.239 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:19.239 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:47:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:19.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.241 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:47:19.241 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:47:19.241 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:19.241 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:19.241 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:19.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:19.241 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:19.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.241 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:47:19.241 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:47:19.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.241 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:19.241 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:19.241 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:19.242 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:19.242 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:19.242 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:47:19.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.242 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:19.242 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:19.242 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:47:19.242 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:19.242 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:19.242 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:47:19.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.242 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:19.242 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:19.242 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:47:19.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:19.243 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:19.243 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:47:19.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.244 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:19.244 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:19.244 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:19.244 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:19.245 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:19.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.245 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:47:19.245 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:47:19.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.219+0000 7f6972c5d640 1 -- 172.21.15.26:0/406517086 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6958036130 legacy=0x7f69580385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:19.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.219+0000 7f6972c5d640 10 monclient: shutdown 2024-09-16T14:47:19.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.219+0000 7f6972c5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:19.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.219+0000 7f6972c5d640 1 -- 172.21.15.26:0/406517086 >> v1:172.21.15.26:6790/0 conn(0x7f696c0a0000 legacy=0x7f696c130470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:19.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.219+0000 7f69711fa640 1 -- 172.21.15.26:0/406517086 reap_dead start 2024-09-16T14:47:19.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.219+0000 7f69711fa640 1 -- 172.21.15.26:0/406517086 reap_dead start 2024-09-16T14:47:19.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.219+0000 7f6972c5d640 1 -- 172.21.15.26:0/406517086 shutdown_connections 2024-09-16T14:47:19.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.219+0000 7f6972c5d640 1 -- 172.21.15.26:0/406517086 >> 172.21.15.26:0/406517086 conn(0x7f696c0a7f30 msgr2=0x7f696c06b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:19.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.219+0000 7f6972c5d640 1 -- 172.21.15.26:0/406517086 shutdown_connections 2024-09-16T14:47:19.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.219+0000 7f6972c5d640 1 -- 172.21.15.26:0/406517086 wait complete. 2024-09-16T14:47:19.247 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:47:19.247 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T14:47:19.247 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:47:19.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 10 monclient: get_monmap_and_config 2024-09-16T14:47:19.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 10 monclient: build_initial_monmap 2024-09-16T14:47:19.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 10 monclient: monmap: 2024-09-16T14:47:19.414 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:19.414 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:19.414 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:19.414393+0000 2024-09-16T14:47:19.415 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:19.414393+0000 2024-09-16T14:47:19.415 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:19.415 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:19.415 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:19.415 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:19.415 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:19.415 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:19.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 1 Processor -- start 2024-09-16T14:47:19.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 1 -- start start 2024-09-16T14:47:19.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 10 monclient: init 2024-09-16T14:47:19.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:19.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:19.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 10 monclient(hunting): picked mon.noname-b con 0x7f584c09cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:19.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 10 monclient(hunting): picked mon.noname-a con 0x7f584c132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:19.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 10 monclient(hunting): picked mon.noname-c con 0x7f584c138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:19.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584c09d260 con 0x7f584c132650 2024-09-16T14:47:19.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584c132a80 con 0x7f584c138ff0 2024-09-16T14:47:19.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584c09cf60 con 0x7f584c09cb80 2024-09-16T14:47:19.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:19.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 10 monclient(hunting): authenticate will time out at 1263.534447s 2024-09-16T14:47:19.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5849d9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f584c132650 legacy=0x7f584c1378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:19.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f584c09cb80 0x7f584c0717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49362/0 (socket says 172.21.15.26:49362) 2024-09-16T14:47:19.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584a59c640 1 -- 172.21.15.26:0/1343652462 learned_addr learned my addr 172.21.15.26:0/1343652462 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:19.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 1 -- 172.21.15.26:0/1343652462 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351007861 0 0) 0x7f584c09cf60 con 0x7f584c09cb80 2024-09-16T14:47:19.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:19.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:19.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:19.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 10 monclient(hunting): my global_id is 114263 2024-09-16T14:47:19.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 1 -- 172.21.15.26:0/1343652462 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5834003530 con 0x7f584c09cb80 2024-09-16T14:47:19.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 1 -- 172.21.15.26:0/1343652462 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2834386923 0 0) 0x7f5834003530 con 0x7f584c09cb80 2024-09-16T14:47:19.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 1 -- 172.21.15.26:0/1343652462 >> v1:172.21.15.189:6789/0 conn(0x7f584c138ff0 legacy=0x7f584c13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:19.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 1 -- 172.21.15.26:0/1343652462 >> v1:172.21.15.26:6789/0 conn(0x7f584c132650 legacy=0x7f584c1378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:19.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 10 monclient: _finish_hunting 0 2024-09-16T14:47:19.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 1 monclient: found mon.noname-b 2024-09-16T14:47:19.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:19.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:47:19.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 1 -- 172.21.15.26:0/1343652462 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584c09f2d0 con 0x7f584c09cb80 2024-09-16T14:47:19.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 10 monclient: _finish_auth 0 2024-09-16T14:47:19.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 10 monclient: _check_auth_tickets 2024-09-16T14:47:19.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:19.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 1 -- 172.21.15.26:0/1343652462 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f583c001660 con 0x7f584c09cb80 2024-09-16T14:47:19.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:19.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:19.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 10 monclient: dump: 2024-09-16T14:47:19.424 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:19.424 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:19.424 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.424 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.424 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:19.424 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:19.424 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:19.424 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:19.424 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:19.424 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:19.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 5 monclient: authenticate success, global_id 114263 2024-09-16T14:47:19.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:19.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 1 -- 172.21.15.26:0/1343652462 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f583c0018c0 con 0x7f584c09cb80 2024-09-16T14:47:19.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:19.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:19.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 4 set_mon_vals no callback set 2024-09-16T14:47:19.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 1 -- 172.21.15.26:0/1343652462 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f583c001c50 con 0x7f584c09cb80 2024-09-16T14:47:19.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:19.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:19.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f584959a640 10 monclient: dump: 2024-09-16T14:47:19.425 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:19.425 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:19.425 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.426 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.426 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:19.426 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:19.426 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:19.426 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:19.426 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:19.426 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:19.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 10 monclient: shutdown 2024-09-16T14:47:19.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:19.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 1 -- 172.21.15.26:0/1343652462 >> v1:172.21.15.26:6790/0 conn(0x7f584c09cb80 legacy=0x7f584c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:19.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 1 -- 172.21.15.26:0/1343652462 shutdown_connections 2024-09-16T14:47:19.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.411+0000 7f5850ba5640 1 -- 172.21.15.26:0/1343652462 >> 172.21.15.26:0/1343652462 conn(0x7f584c09fb50 msgr2=0x7f584c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:19.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 1 -- 172.21.15.26:0/1343652462 shutdown_connections 2024-09-16T14:47:19.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 1 -- 172.21.15.26:0/1343652462 wait complete. 2024-09-16T14:47:19.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 10 monclient: build_initial_monmap 2024-09-16T14:47:19.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 10 monclient: monmap: 2024-09-16T14:47:19.427 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:19.427 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:19.427 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:19.427 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:19.427 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:19.427 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:19.427 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:19.427 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:19.427 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:19.428 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:19.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 1 Processor -- start 2024-09-16T14:47:19.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 1 -- start start 2024-09-16T14:47:19.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 10 monclient: init 2024-09-16T14:47:19.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:19.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:19.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 10 monclient(hunting): picked mon.noname-c con 0x7f584c09cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:19.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 10 monclient(hunting): picked mon.noname-a con 0x7f584c132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:19.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 10 monclient(hunting): picked mon.noname-b con 0x7f584c138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:19.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584c12f490 con 0x7f584c132650 2024-09-16T14:47:19.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584c134b50 con 0x7f584c138ff0 2024-09-16T14:47:19.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5849d9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f584c132650 legacy=0x7f584c12e160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:19.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f584a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f584c09cb80 0x7f584c12da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49376/0 (socket says 172.21.15.26:49376) 2024-09-16T14:47:19.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584c12f9a0 con 0x7f584c09cb80 2024-09-16T14:47:19.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f584a59c640 1 -- 172.21.15.26:0/2923498501 learned_addr learned my addr 172.21.15.26:0/2923498501 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:19.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:19.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 10 monclient(hunting): authenticate will time out at 1263.537664s 2024-09-16T14:47:19.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075413385 0 0) 0x7f584c12f9a0 con 0x7f584c09cb80 2024-09-16T14:47:19.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:19.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:19.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:19.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 10 monclient(hunting): my global_id is 114266 2024-09-16T14:47:19.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f582c003530 con 0x7f584c09cb80 2024-09-16T14:47:19.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 492955859 0 0) 0x7f582c003530 con 0x7f584c09cb80 2024-09-16T14:47:19.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f584c12f9a0 con 0x7f584c09cb80 2024-09-16T14:47:19.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f583c001620 con 0x7f584c09cb80 2024-09-16T14:47:19.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:19.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:19.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 10 monclient(hunting): dump: 2024-09-16T14:47:19.430 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:19.430 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:19.430 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.430 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.430 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:19.430 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:19.430 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:19.431 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:19.431 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:19.431 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:19.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:19.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3384019234 0 0) 0x7f584c12f9a0 con 0x7f584c09cb80 2024-09-16T14:47:19.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 >> v1:172.21.15.189:6789/0 conn(0x7f584c138ff0 legacy=0x7f584c1d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:19.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 >> v1:172.21.15.26:6789/0 conn(0x7f584c132650 legacy=0x7f584c12e160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:19.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:19.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 1 monclient: found mon.c 2024-09-16T14:47:19.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:19.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:19.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584c134e30 con 0x7f584c09cb80 2024-09-16T14:47:19.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 10 monclient: _finish_auth 0 2024-09-16T14:47:19.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:19.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:19.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 5 monclient: authenticate success, global_id 114266 2024-09-16T14:47:19.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 10 monclient: _renew_subs 2024-09-16T14:47:19.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:19.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 1 -- 172.21.15.26:0/2923498501 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f584c13c7d0 con 0x7f584c09cb80 2024-09-16T14:47:19.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 10 monclient: _renew_subs 2024-09-16T14:47:19.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:19.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 1 -- 172.21.15.26:0/2923498501 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f584c09a240 con 0x7f584c09cb80 2024-09-16T14:47:19.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f583c001240 con 0x7f584c09cb80 2024-09-16T14:47:19.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:19.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f583c004c40 con 0x7f584c09cb80 2024-09-16T14:47:19.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:19.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:19.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 10 monclient: dump: 2024-09-16T14:47:19.433 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:19.433 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:19.433 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.433 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.433 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:19.433 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:19.433 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:19.433 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:19.433 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:19.433 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:19.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5848d99640 4 set_mon_vals no callback set 2024-09-16T14:47:19.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f583c00f6b0 con 0x7f584c09cb80 2024-09-16T14:47:19.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:19.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f5850ba5640 1 -- 172.21.15.26:0/2923498501 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5830006590 con 0x7f5830002830 2024-09-16T14:47:19.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(111..111 src has 73..111) v4 ==== 5789+0+0 (unknown 2962608226 0 0) 0x7f583c043b30 con 0x7f584c09cb80 2024-09-16T14:47:19.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.415+0000 7f584ad9d640 10 monclient: get_auth_request con 0x7f582c036090 auth_method 0 2024-09-16T14:47:19.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307290481 0 0) 0x7f5830006590 con 0x7f5830002830 2024-09-16T14:47:19.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:19.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:19.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:19.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 10 monclient(hunting): my global_id is 94885 2024-09-16T14:47:19.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f584c12f9a0 con 0x7f5830002830 2024-09-16T14:47:19.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 87540953 0 0) 0x7f584c12f9a0 con 0x7f5830002830 2024-09-16T14:47:19.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5830006590 con 0x7f5830002830 2024-09-16T14:47:19.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5840003f70 con 0x7f5830002830 2024-09-16T14:47:19.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:19.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:19.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 10 monclient: dump: 2024-09-16T14:47:19.437 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:19.437 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:19.437 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.437 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.437 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:19.438 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:19.438 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:19.438 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:19.438 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:19.438 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:19.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2364928539 0 0) 0x7f5830006590 con 0x7f5830002830 2024-09-16T14:47:19.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f583000a0e0 con 0x7f5830002830 2024-09-16T14:47:19.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5840006270 con 0x7f5830002830 2024-09-16T14:47:19.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:19.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:19.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 >> v1:172.21.15.189:6789/0 conn(0x7f5830002830 legacy=0x7f5830004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:19.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.419+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 >> v1:172.21.15.189:6789/0 conn(0x7f5830002830 legacy=0x7f5830004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:19.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f5850ba5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:47:19.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f5850ba5640 1 -- 172.21.15.26:0/2923498501 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5830009da0 con 0x7f58300090c0 2024-09-16T14:47:19.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716865999 0 0) 0x7f5830009da0 con 0x7f58300090c0 2024-09-16T14:47:19.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:19.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:19.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:19.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 10 monclient(hunting): my global_id is 94888 2024-09-16T14:47:19.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5830006590 con 0x7f58300090c0 2024-09-16T14:47:19.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 343281834 0 0) 0x7f5830006590 con 0x7f58300090c0 2024-09-16T14:47:19.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5830009da0 con 0x7f58300090c0 2024-09-16T14:47:19.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5840005830 con 0x7f58300090c0 2024-09-16T14:47:19.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:19.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:19.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 10 monclient: dump: 2024-09-16T14:47:19.440 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:19.440 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:19.440 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.440 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:19.440 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:19.440 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:19.440 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:19.440 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:19.440 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:19.440 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:19.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1011085258 0 0) 0x7f5830009da0 con 0x7f58300090c0 2024-09-16T14:47:19.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f583000c290 con 0x7f58300090c0 2024-09-16T14:47:19.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4034 (unknown 0 0 1523663247) 0x7f5840005b60 con 0x7f58300090c0 2024-09-16T14:47:19.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:47:19.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:19.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 >> v1:172.21.15.189:6789/0 conn(0x7f58300090c0 legacy=0x7f5830009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:19.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f583affd640 1 -- 172.21.15.26:0/2923498501 >> v1:172.21.15.189:6789/0 conn(0x7f58300090c0 legacy=0x7f5830009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:19.441 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:47:19.442 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:19.442 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:19.442 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:47:19.442 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 46, 2024-09-16T14:47:19.442 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:47:19.442 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:47:19.442 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:47:19.442 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:19.442 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:47:19.442 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:19.442 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:47:19.442 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:47:19.442 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:47:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:47:19.443 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:47:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:47:19.444 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:47:19.445 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:47:19.445 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:47:19.445 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:47:19.445 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:19.445 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:19.445 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:47:19.445 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:47:19.445 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:47:19.445 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:47:19.449 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:47:19.449 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:47:19.449 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:47:19.449 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:19.450 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:47:19.450 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:19.450 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:19.450 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:19.451 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:19.451 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:19.451 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:19.451 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:19.451 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:19.452 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:19.452 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:19.452 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:19.452 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:47:19.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.452 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:47:19.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.452 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:19.452 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:19.453 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:19.453 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:19.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.453 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:19.453 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:47:19.453 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:19.453 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:19.453 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:19.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.454 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:47:19.454 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:47:19.454 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:19.454 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:19.454 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:19.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.455 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:19.455 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:19.455 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:19.455 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:19.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.455 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:19.455 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:47:19.456 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:19.456 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:19.456 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:19.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.456 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:47:19.456 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:47:19.457 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:19.457 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:19.457 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:19.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.457 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:19.457 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:19.457 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:19.458 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:19.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.458 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:19.458 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:47:19.458 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:19.458 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:19.458 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:19.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.459 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:47:19.459 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:47:19.459 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:19.459 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:19.459 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:19.459 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:19.460 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:19.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.460 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:47:19.460 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:47:19.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.460 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:19.460 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:19.461 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:19.461 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:19.461 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:19.461 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:47:19.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.461 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:19.461 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:19.462 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:19.462 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:19.462 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:19.462 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:47:19.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:19.462 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:19.462 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:19.463 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:47:19.463 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:19.463 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:19.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:19.463 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:47:19.464 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:47:19.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f5850ba5640 1 -- 172.21.15.26:0/2923498501 >> v1:172.21.15.189:6812/3196161582 conn(0x7f582c036090 legacy=0x7f582c038550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:19.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f5850ba5640 10 monclient: shutdown 2024-09-16T14:47:19.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f5850ba5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:19.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f5850ba5640 1 -- 172.21.15.26:0/2923498501 >> v1:172.21.15.26:6790/0 conn(0x7f584c09cb80 legacy=0x7f584c12da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:19.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f584ad9d640 1 -- 172.21.15.26:0/2923498501 reap_dead start 2024-09-16T14:47:19.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f584ad9d640 1 -- 172.21.15.26:0/2923498501 reap_dead start 2024-09-16T14:47:19.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f5850ba5640 1 -- 172.21.15.26:0/2923498501 shutdown_connections 2024-09-16T14:47:19.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f5850ba5640 1 -- 172.21.15.26:0/2923498501 >> 172.21.15.26:0/2923498501 conn(0x7f584c09fb50 msgr2=0x7f584c09be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:19.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f5850ba5640 1 -- 172.21.15.26:0/2923498501 shutdown_connections 2024-09-16T14:47:19.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:19.427+0000 7f5850ba5640 1 -- 172.21.15.26:0/2923498501 wait complete. 2024-09-16T14:47:19.468 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:47:19.468 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:47:19.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:19.782+0000 7f58ed7fa640 10 monclient: tick 2024-09-16T14:47:19.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:19.782+0000 7f58ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:19.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:19.782+0000 7f58ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:19.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:19.818+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}]=0 reweighted subtree id -10 name 'foobaz' to 155 in crush map v112) v1 ==== 168+0+0 (unknown 435931192 0 0) 0x7f58f4012d30 con 0x7f58f81318a0 2024-09-16T14:47:19.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:19.818+0000 7f58ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T14:47:19.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:19.818+0000 7f58ee7fc640 10 monclient: _finish_command 2 = system:0 reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-16T14:47:19.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:19.922+0000 7f5900168640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T14:47:19.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:19.922+0000 7f5900168640 10 monclient: _send_command 3 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T14:47:19.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:19.922+0000 7f5900168640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:19.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:19.922+0000 7f5900168640 1 -- 172.21.15.189:0/1821466252 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0} v 0) v1 -- 0x7f58e0005390 con 0x7f58f81318a0 2024-09-16T14:47:20.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.782+0000 7f58ed7fa640 10 monclient: tick 2024-09-16T14:47:20.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.782+0000 7f58ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:20.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.782+0000 7f58ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:20.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.822+0000 7f58ee7fc640 1 -- 172.21.15.189:0/1821466252 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}]=0 reweighted subtree id -10 name 'foobaz' to 155 in crush map v113) v1 ==== 168+0+0 (unknown 3273076486 0 0) 0x7f58f40144b0 con 0x7f58f81318a0 2024-09-16T14:47:20.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.822+0000 7f58ee7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T14:47:20.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.822+0000 7f58ee7fc640 10 monclient: _finish_command 3 = system:0 reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-16T14:47:20.827 INFO:tasks.workunit.client.0.smithi189.stderr:reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-16T14:47:20.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.826+0000 7f5900168640 1 -- 172.21.15.189:0/1821466252 >> v1:172.21.15.189:6812/3196161582 conn(0x7f58dc036420 legacy=0x7f58dc0388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:20.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.826+0000 7f5900168640 10 monclient: shutdown 2024-09-16T14:47:20.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.826+0000 7f5900168640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:20.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.826+0000 7f5900168640 1 -- 172.21.15.189:0/1821466252 >> v1:172.21.15.189:6789/0 conn(0x7f58f81318a0 legacy=0x7f58f8119340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:20.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.826+0000 7f5900168640 1 -- 172.21.15.189:0/1821466252 shutdown_connections 2024-09-16T14:47:20.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.826+0000 7f5900168640 1 -- 172.21.15.189:0/1821466252 >> 172.21.15.189:0/1821466252 conn(0x7f58f80a3320 msgr2=0x7f58f8133560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:20.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.826+0000 7f5900168640 1 -- 172.21.15.189:0/1821466252 shutdown_connections 2024-09-16T14:47:20.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.826+0000 7f5900168640 1 -- 172.21.15.189:0/1821466252 wait complete. 2024-09-16T14:47:20.848 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush reweight-subtree foobaz 123456 2024-09-16T14:47:20.849 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:47:20.849 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush reweight-subtree foobaz 123456 2024-09-16T14:47:20.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.962+0000 7f509302a640 10 monclient: get_monmap_and_config 2024-09-16T14:47:20.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.962+0000 7f509302a640 10 monclient: build_initial_monmap 2024-09-16T14:47:20.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.962+0000 7f509302a640 10 monclient: monmap: 2024-09-16T14:47:20.968 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:20.968 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:20.968 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:20.968502+0000 2024-09-16T14:47:20.969 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:20.968502+0000 2024-09-16T14:47:20.969 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:20.969 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:20.969 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:20.969 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:20.969 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:20.969 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:20.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.962+0000 7f509302a640 1 Processor -- start 2024-09-16T14:47:20.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.962+0000 7f509302a640 1 -- start start 2024-09-16T14:47:20.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.962+0000 7f509302a640 10 monclient: init 2024-09-16T14:47:20.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:20.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:20.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient(hunting): picked mon.noname-b con 0x7f508c09df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:20.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient(hunting): picked mon.noname-c con 0x7f508c09e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:20.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient(hunting): picked mon.noname-a con 0x7f508c09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:20.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508c09ee20 con 0x7f508c09e740 2024-09-16T14:47:20.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508c09f100 con 0x7f508c09e2f0 2024-09-16T14:47:20.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508c09f3e0 con 0x7f508c09df10 2024-09-16T14:47:20.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:20.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient(hunting): authenticate will time out at 1223.084649s 2024-09-16T14:47:20.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f50915c7640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f508c09e740 legacy=0x7f508c13b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:20.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f508c09e2f0 0x7f508c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46346/0 (socket says 172.21.15.189:46346) 2024-09-16T14:47:20.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508bfff640 1 -- 172.21.15.189:0/3296937696 learned_addr learned my addr 172.21.15.189:0/3296937696 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:20.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 1 -- 172.21.15.189:0/3296937696 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156850378 0 0) 0x7f508c09f100 con 0x7f508c09e2f0 2024-09-16T14:47:20.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:20.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:20.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:20.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 10 monclient(hunting): my global_id is 94891 2024-09-16T14:47:20.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 1 -- 172.21.15.189:0/3296937696 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5074003530 con 0x7f508c09e2f0 2024-09-16T14:47:20.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 1 -- 172.21.15.189:0/3296937696 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2977431528 0 0) 0x7f5074003530 con 0x7f508c09e2f0 2024-09-16T14:47:20.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 1 -- 172.21.15.189:0/3296937696 >> v1:172.21.15.26:6790/0 conn(0x7f508c09df10 legacy=0x7f508c09d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:20.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 1 -- 172.21.15.189:0/3296937696 >> v1:172.21.15.26:6789/0 conn(0x7f508c09e740 legacy=0x7f508c13b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:20.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:20.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 1 monclient: found mon.noname-c 2024-09-16T14:47:20.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:20.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:20.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 1 -- 172.21.15.189:0/3296937696 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508c09c690 con 0x7f508c09e2f0 2024-09-16T14:47:20.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:20.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:20.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:20.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 1 -- 172.21.15.189:0/3296937696 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5080001620 con 0x7f508c09e2f0 2024-09-16T14:47:20.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 5 monclient: authenticate success, global_id 94891 2024-09-16T14:47:20.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:20.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:20.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:20.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 10 monclient: dump: 2024-09-16T14:47:20.975 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:20.975 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:20.975 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:20.975 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:20.975 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:20.975 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:20.975 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:20.975 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:20.975 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:20.975 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:20.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:20.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 1 -- 172.21.15.189:0/3296937696 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5080001940 con 0x7f508c09e2f0 2024-09-16T14:47:20.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:20.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 1 -- 172.21.15.189:0/3296937696 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5080001d10 con 0x7f508c09e2f0 2024-09-16T14:47:20.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:20.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 4 set_mon_vals no callback set 2024-09-16T14:47:20.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:20.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:20.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508b7fe640 10 monclient: dump: 2024-09-16T14:47:20.976 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:20.976 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:20.976 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:20.976 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:20.976 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:20.976 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:20.977 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:20.977 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:20.977 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:20.977 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:20.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient: shutdown 2024-09-16T14:47:20.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:20.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 1 -- 172.21.15.189:0/3296937696 >> v1:172.21.15.189:6789/0 conn(0x7f508c09e2f0 legacy=0x7f508c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:20.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 1 -- 172.21.15.189:0/3296937696 shutdown_connections 2024-09-16T14:47:20.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 1 -- 172.21.15.189:0/3296937696 >> 172.21.15.189:0/3296937696 conn(0x7f508c0a3320 msgr2=0x7f508c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:20.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 1 -- 172.21.15.189:0/3296937696 shutdown_connections 2024-09-16T14:47:20.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 1 -- 172.21.15.189:0/3296937696 wait complete. 2024-09-16T14:47:20.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient: build_initial_monmap 2024-09-16T14:47:20.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient: monmap: 2024-09-16T14:47:20.978 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:20.978 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:20.978 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:20.978 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:20.978 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:20.978 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:20.978 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:20.978 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:20.978 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:20.978 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:20.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 1 Processor -- start 2024-09-16T14:47:20.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 1 -- start start 2024-09-16T14:47:20.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient: init 2024-09-16T14:47:20.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:20.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:20.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient(hunting): picked mon.noname-b con 0x7f508c09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:20.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient(hunting): picked mon.noname-a con 0x7f508c09e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:20.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient(hunting): picked mon.noname-c con 0x7f508c09e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:20.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508c06cea0 con 0x7f508c09e2f0 2024-09-16T14:47:20.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508c119e50 con 0x7f508c09df10 2024-09-16T14:47:20.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508c1324a0 con 0x7f508c09e740 2024-09-16T14:47:20.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:20.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f509302a640 10 monclient(hunting): authenticate will time out at 1223.087571s 2024-09-16T14:47:20.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f5090dc6640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f508c09df10 0x7f508c139b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46356/0 (socket says 172.21.15.189:46356) 2024-09-16T14:47:20.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f5090dc6640 1 -- 172.21.15.189:0/1522459997 learned_addr learned my addr 172.21.15.189:0/1522459997 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:20.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f508bfff640 1 -- 172.21.15.189:0/1522459997 >> v1:172.21.15.26:6789/0 conn(0x7f508c09e2f0 legacy=0x7f508c13a230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:20.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274444154 0 0) 0x7f508c119e50 con 0x7f508c09df10 2024-09-16T14:47:20.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f50897fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:20.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f50897fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:20.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f50897fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:20.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f50897fa640 10 monclient(hunting): my global_id is 94894 2024-09-16T14:47:20.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.966+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5078003530 con 0x7f508c09df10 2024-09-16T14:47:20.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1190092982 0 0) 0x7f5078003530 con 0x7f508c09df10 2024-09-16T14:47:20.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f508c119e50 con 0x7f508c09df10 2024-09-16T14:47:20.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f507c001940 con 0x7f508c09df10 2024-09-16T14:47:20.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:20.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:20.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 10 monclient(hunting): dump: 2024-09-16T14:47:20.984 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:20.984 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:20.984 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:20.984 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:20.984 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:20.984 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:20.984 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:20.985 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:20.985 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:20.985 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:20.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:20.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059651067 0 0) 0x7f508c1324a0 con 0x7f508c09e740 2024-09-16T14:47:20.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:20.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:20.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:20.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 10 monclient(hunting): my global_id is 114272 2024-09-16T14:47:20.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5078003530 con 0x7f508c09e740 2024-09-16T14:47:20.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 571160148 0 0) 0x7f508c119e50 con 0x7f508c09df10 2024-09-16T14:47:20.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 >> v1:172.21.15.26:6790/0 conn(0x7f508c09e740 legacy=0x7f508c1cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:20.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 >> v1:172.21.15.26:6789/0 conn(0x7f508c09e2f0 legacy=0x7f508c13a230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:20.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 10 monclient: _finish_hunting 0 2024-09-16T14:47:20.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 1 monclient: found mon.b 2024-09-16T14:47:20.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:20.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:20.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508c11a660 con 0x7f508c09df10 2024-09-16T14:47:20.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 10 monclient: _finish_auth 0 2024-09-16T14:47:20.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:20.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:20.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f509302a640 5 monclient: authenticate success, global_id 94894 2024-09-16T14:47:20.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f509302a640 10 monclient: _renew_subs 2024-09-16T14:47:20.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f509302a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:20.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f509302a640 1 -- 172.21.15.189:0/1522459997 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508c13a940 con 0x7f508c09df10 2024-09-16T14:47:20.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f509302a640 10 monclient: _renew_subs 2024-09-16T14:47:20.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f509302a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:20.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f509302a640 1 -- 172.21.15.189:0/1522459997 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f508c13b690 con 0x7f508c09df10 2024-09-16T14:47:20.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f509302a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:20.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f509302a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:20.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f509302a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:20.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f509302a640 1 -- 172.21.15.189:0/1522459997 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f508c11a4a0 con 0x7f508c09df10 2024-09-16T14:47:20.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f507c001c80 con 0x7f508c09df10 2024-09-16T14:47:20.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:20.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f507c003950 con 0x7f508c09df10 2024-09-16T14:47:20.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:20.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:20.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 10 monclient: dump: 2024-09-16T14:47:20.987 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:20.987 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:20.987 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:20.988 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:20.988 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:20.988 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:20.988 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:20.988 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:20.988 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:20.988 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:20.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f508affd640 4 set_mon_vals no callback set 2024-09-16T14:47:20.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f507c003b30 con 0x7f508c09df10 2024-09-16T14:47:20.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(113..113 src has 73..113) v4 ==== 5789+0+0 (unknown 2981862269 0 0) 0x7f507c0438f0 con 0x7f508c09df10 2024-09-16T14:47:20.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.970+0000 7f508bfff640 10 monclient: get_auth_request con 0x7f5078036110 auth_method 0 2024-09-16T14:47:20.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.974+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f507c014590 con 0x7f508c09df10 2024-09-16T14:47:20.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.974+0000 7f50897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:20.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:20.974+0000 7f50897fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:21.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.138+0000 7f509302a640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-16T14:47:21.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.138+0000 7f509302a640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-16T14:47:21.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.138+0000 7f509302a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:21.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.138+0000 7f509302a640 1 -- 172.21.15.189:0/1522459997 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0} v 0) v1 -- 0x7f508c06a0d0 con 0x7f508c09df10 2024-09-16T14:47:21.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.138+0000 7f50897fa640 1 -- 172.21.15.189:0/1522459997 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}]=-75 (75) Value too large for defined data type v113) v1 ==== 154+0+0 (unknown 2757362208 0 0) 0x7f507c012a00 con 0x7f508c09df10 2024-09-16T14:47:21.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.138+0000 7f50897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-16T14:47:21.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.138+0000 7f50897fa640 10 monclient: _finish_command 2 = mon:75 (75) Value too large for defined data type 2024-09-16T14:47:21.145 INFO:tasks.workunit.client.0.smithi189.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2024-09-16T14:47:21.150 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:47:21.151 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T14:47:21.151 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep osd.12 2024-09-16T14:47:21.151 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 155 2024-09-16T14:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc91208e640 10 monclient: get_monmap_and_config 2024-09-16T14:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc91208e640 10 monclient: build_initial_monmap 2024-09-16T14:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc91208e640 10 monclient: monmap: 2024-09-16T14:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:21.265 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:21.265 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:21.264035+0000 2024-09-16T14:47:21.265 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:21.264035+0000 2024-09-16T14:47:21.265 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:21.266 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:21.266 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:21.266 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:21.266 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:21.266 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:21.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc91208e640 1 Processor -- start 2024-09-16T14:47:21.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc91208e640 1 -- start start 2024-09-16T14:47:21.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc91208e640 10 monclient: init 2024-09-16T14:47:21.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc91208e640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:21.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc91208e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:21.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc91208e640 10 monclient(hunting): picked mon.noname-c con 0x7fc90c099a20 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:21.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc91208e640 10 monclient(hunting): picked mon.noname-b con 0x7fc90c099e00 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:21.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc91208e640 10 monclient(hunting): picked mon.noname-a con 0x7fc90c13b1a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:21.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc91208e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90c09a220 con 0x7fc90c13b1a0 2024-09-16T14:47:21.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc91208e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90c09f2a0 con 0x7fc90c099a20 2024-09-16T14:47:21.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc91208e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90c071c00 con 0x7fc90c099e00 2024-09-16T14:47:21.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc91208e640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:21.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc91208e640 10 monclient(hunting): authenticate will time out at 1223.379769s 2024-09-16T14:47:21.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc90b7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc90c099a20 0x7fc90c134d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46370/0 (socket says 172.21.15.189:46370) 2024-09-16T14:47:21.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc90b7fe640 1 -- 172.21.15.189:0/2166254138 learned_addr learned my addr 172.21.15.189:0/2166254138 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:21.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.258+0000 7fc90bfff640 1 -- 172.21.15.189:0/2166254138 >> v1:172.21.15.26:6789/0 conn(0x7fc90c13b1a0 legacy=0x7fc90c13d590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:21.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 1 -- 172.21.15.189:0/2166254138 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950255799 0 0) 0x7fc90c09f2a0 con 0x7fc90c099a20 2024-09-16T14:47:21.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:21.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:21.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:21.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 10 monclient(hunting): my global_id is 94897 2024-09-16T14:47:21.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 1 -- 172.21.15.189:0/2166254138 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f4003530 con 0x7fc90c099a20 2024-09-16T14:47:21.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 1 -- 172.21.15.189:0/2166254138 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1495810772 0 0) 0x7fc8f4003530 con 0x7fc90c099a20 2024-09-16T14:47:21.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 1 -- 172.21.15.189:0/2166254138 >> v1:172.21.15.26:6790/0 conn(0x7fc90c099e00 legacy=0x7fc90c139a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:21.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 1 -- 172.21.15.189:0/2166254138 >> v1:172.21.15.26:6789/0 conn(0x7fc90c13b1a0 legacy=0x7fc90c13d590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:21.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:21.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 1 monclient: found mon.noname-c 2024-09-16T14:47:21.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:21.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:21.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 1 -- 172.21.15.189:0/2166254138 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90c135410 con 0x7fc90c099a20 2024-09-16T14:47:21.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:21.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:21.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:21.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 1 -- 172.21.15.189:0/2166254138 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc8fc001620 con 0x7fc90c099a20 2024-09-16T14:47:21.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 5 monclient: authenticate success, global_id 94897 2024-09-16T14:47:21.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:21.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:21.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:21.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 10 monclient: dump: 2024-09-16T14:47:21.286 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:21.286 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:21.286 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:21.286 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:21.287 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:21.287 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:21.287 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:21.287 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:21.287 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:21.287 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:21.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:21.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 1 -- 172.21.15.189:0/2166254138 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8fc0018d0 con 0x7fc90c099a20 2024-09-16T14:47:21.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:21.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:21.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 4 set_mon_vals no callback set 2024-09-16T14:47:21.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 1 -- 172.21.15.189:0/2166254138 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc8fc001ca0 con 0x7fc90c099a20 2024-09-16T14:47:21.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:21.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:21.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90a7fc640 10 monclient: dump: 2024-09-16T14:47:21.288 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:21.288 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:21.288 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:21.288 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:21.288 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:21.288 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:21.288 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:21.288 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:21.289 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:21.289 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:21.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 10 monclient: shutdown 2024-09-16T14:47:21.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:21.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 1 -- 172.21.15.189:0/2166254138 >> v1:172.21.15.189:6789/0 conn(0x7fc90c099a20 legacy=0x7fc90c134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:21.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 1 -- 172.21.15.189:0/2166254138 shutdown_connections 2024-09-16T14:47:21.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 1 -- 172.21.15.189:0/2166254138 >> 172.21.15.189:0/2166254138 conn(0x7fc90c0a0260 msgr2=0x7fc90c0a0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:21.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 1 -- 172.21.15.189:0/2166254138 shutdown_connections 2024-09-16T14:47:21.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 1 -- 172.21.15.189:0/2166254138 wait complete. 2024-09-16T14:47:21.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 10 monclient: build_initial_monmap 2024-09-16T14:47:21.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 10 monclient: monmap: 2024-09-16T14:47:21.289 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:21.290 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:21.290 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:21.290 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:21.290 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:21.290 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:21.290 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:21.290 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:21.290 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:21.290 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:21.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 1 Processor -- start 2024-09-16T14:47:21.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 1 -- start start 2024-09-16T14:47:21.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 10 monclient: init 2024-09-16T14:47:21.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:21.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:21.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 10 monclient(hunting): picked mon.noname-c con 0x7fc90c099a20 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:21.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 10 monclient(hunting): picked mon.noname-a con 0x7fc90c099e00 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:21.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 10 monclient(hunting): picked mon.noname-b con 0x7fc90c13b1a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:21.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90c09a550 con 0x7fc90c099e00 2024-09-16T14:47:21.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90c12b780 con 0x7fc90c13b1a0 2024-09-16T14:47:21.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90c12ba60 con 0x7fc90c099a20 2024-09-16T14:47:21.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:21.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 10 monclient(hunting): authenticate will time out at 1223.382897s 2024-09-16T14:47:21.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc90c13b1a0 0x7fc90c1d1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46384/0 (socket says 172.21.15.189:46384) 2024-09-16T14:47:21.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90bfff640 1 -- 172.21.15.189:0/2933853593 learned_addr learned my addr 172.21.15.189:0/2933853593 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:21.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc90affd640 1 -- 172.21.15.189:0/2933853593 >> v1:172.21.15.26:6789/0 conn(0x7fc90c099e00 legacy=0x7fc90c1ce7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:21.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76720375 0 0) 0x7fc90c12b780 con 0x7fc90c13b1a0 2024-09-16T14:47:21.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:21.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:21.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:21.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient(hunting): my global_id is 94900 2024-09-16T14:47:21.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f8003530 con 0x7fc90c13b1a0 2024-09-16T14:47:21.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1134359032 0 0) 0x7fc8f8003530 con 0x7fc90c13b1a0 2024-09-16T14:47:21.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc90c12b780 con 0x7fc90c13b1a0 2024-09-16T14:47:21.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc9040043d0 con 0x7fc90c13b1a0 2024-09-16T14:47:21.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:21.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:21.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient(hunting): dump: 2024-09-16T14:47:21.292 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:21.292 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:21.293 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:21.293 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:21.293 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:21.293 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:21.293 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:21.293 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:21.293 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:21.293 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:21.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:21.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448366211 0 0) 0x7fc90c12ba60 con 0x7fc90c099a20 2024-09-16T14:47:21.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:21.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:21.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:21.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient(hunting): my global_id is 114278 2024-09-16T14:47:21.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f8003530 con 0x7fc90c099a20 2024-09-16T14:47:21.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3206991024 0 0) 0x7fc90c12b780 con 0x7fc90c13b1a0 2024-09-16T14:47:21.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 >> v1:172.21.15.26:6790/0 conn(0x7fc90c099a20 legacy=0x7fc90c096310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:21.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 >> v1:172.21.15.26:6789/0 conn(0x7fc90c099e00 legacy=0x7fc90c1ce7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:21.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:47:21.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 1 monclient: found mon.b 2024-09-16T14:47:21.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:21.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:21.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90c12a920 con 0x7fc90c13b1a0 2024-09-16T14:47:21.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient: _finish_auth 0 2024-09-16T14:47:21.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:47:21.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:21.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 5 monclient: authenticate success, global_id 94900 2024-09-16T14:47:21.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 10 monclient: _renew_subs 2024-09-16T14:47:21.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:21.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 1 -- 172.21.15.189:0/2933853593 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90c129dc0 con 0x7fc90c13b1a0 2024-09-16T14:47:21.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 10 monclient: _renew_subs 2024-09-16T14:47:21.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:21.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc91208e640 1 -- 172.21.15.189:0/2933853593 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90c12a620 con 0x7fc90c13b1a0 2024-09-16T14:47:21.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9040046c0 con 0x7fc90c13b1a0 2024-09-16T14:47:21.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:21.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc9040049d0 con 0x7fc90c13b1a0 2024-09-16T14:47:21.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:21.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:21.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc909ffb640 4 set_mon_vals no callback set 2024-09-16T14:47:21.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.262+0000 7fc8f3fff640 10 monclient: dump: 2024-09-16T14:47:21.296 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:21.296 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:21.296 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:21.296 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:21.296 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:21.296 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:21.296 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:21.296 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:21.296 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:21.296 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:21.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.266+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc904004bf0 con 0x7fc90c13b1a0 2024-09-16T14:47:21.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.266+0000 7fc91208e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:21.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.266+0000 7fc91208e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:21.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.266+0000 7fc91208e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:21.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.266+0000 7fc91208e640 1 -- 172.21.15.189:0/2933853593 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ec005050 con 0x7fc90c13b1a0 2024-09-16T14:47:21.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.266+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(114..114 src has 73..114) v4 ==== 5789+0+0 (unknown 1728059977 0 0) 0x7fc904043cb0 con 0x7fc90c13b1a0 2024-09-16T14:47:21.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.266+0000 7fc90b7fe640 10 monclient: get_auth_request con 0x7fc8f80360e0 auth_method 0 2024-09-16T14:47:21.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.270+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc9040053f0 con 0x7fc90c13b1a0 2024-09-16T14:47:21.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.270+0000 7fc8f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:21.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.270+0000 7fc8f3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:21.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.426+0000 7fc91208e640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:47:21.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.426+0000 7fc91208e640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T14:47:21.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.426+0000 7fc91208e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:21.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.426+0000 7fc91208e640 1 -- 172.21.15.189:0/2933853593 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fc8ec005b10 con 0x7fc90c13b1a0 2024-09-16T14:47:21.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.426+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v114) v1 ==== 56+0+1449 (unknown 3809803480 0 3599791968) 0x7fc904012e20 con 0x7fc90c13b1a0 2024-09-16T14:47:21.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.426+0000 7fc8f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T14:47:21.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.426+0000 7fc8f3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:21.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.514+0000 7fc91208e640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:47:21.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.514+0000 7fc91208e640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T14:47:21.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.514+0000 7fc91208e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:21.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.514+0000 7fc91208e640 1 -- 172.21.15.189:0/2933853593 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fc8ec0055a0 con 0x7fc90c13b1a0 2024-09-16T14:47:21.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.514+0000 7fc8f3fff640 1 -- 172.21.15.189:0/2933853593 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v114) v1 ==== 56+0+1449 (unknown 3809803480 0 3599791968) 0x7fc9040145a0 con 0x7fc90c13b1a0 2024-09-16T14:47:21.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.514+0000 7fc8f3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T14:47:21.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.514+0000 7fc8f3fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:21.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.518+0000 7fc91208e640 1 -- 172.21.15.189:0/2933853593 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc8f80360e0 legacy=0x7fc8f80385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:21.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.518+0000 7fc91208e640 10 monclient: shutdown 2024-09-16T14:47:21.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.518+0000 7fc91208e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:21.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.518+0000 7fc91208e640 1 -- 172.21.15.189:0/2933853593 >> v1:172.21.15.189:6789/0 conn(0x7fc90c13b1a0 legacy=0x7fc90c1d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:21.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.518+0000 7fc91208e640 1 -- 172.21.15.189:0/2933853593 shutdown_connections 2024-09-16T14:47:21.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.518+0000 7fc91208e640 1 -- 172.21.15.189:0/2933853593 >> 172.21.15.189:0/2933853593 conn(0x7fc90c0a0260 msgr2=0x7fc90c098510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:21.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.518+0000 7fc91208e640 1 -- 172.21.15.189:0/2933853593 shutdown_connections 2024-09-16T14:47:21.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.518+0000 7fc91208e640 1 -- 172.21.15.189:0/2933853593 wait complete. 2024-09-16T14:47:21.535 INFO:tasks.workunit.client.0.smithi189.stdout: 12 155.00000 osd.12 down 1.00000 1.00000 2024-09-16T14:47:21.536 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T14:47:21.536 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 155 2024-09-16T14:47:21.536 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep osd.14 2024-09-16T14:47:21.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.642+0000 7f38c79ab640 10 monclient: get_monmap_and_config 2024-09-16T14:47:21.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.642+0000 7f38c79ab640 10 monclient: build_initial_monmap 2024-09-16T14:47:21.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.642+0000 7f38c79ab640 10 monclient: monmap: 2024-09-16T14:47:21.649 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:21.649 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:21.649 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:21.648695+0000 2024-09-16T14:47:21.649 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:21.648695+0000 2024-09-16T14:47:21.649 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:21.649 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:21.649 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:21.649 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:21.649 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:21.650 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:21.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.642+0000 7f38c79ab640 1 Processor -- start 2024-09-16T14:47:21.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.642+0000 7f38c79ab640 1 -- start start 2024-09-16T14:47:21.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.642+0000 7f38c79ab640 10 monclient: init 2024-09-16T14:47:21.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:21.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:21.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient(hunting): picked mon.noname-b con 0x7f38c0099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:21.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient(hunting): picked mon.noname-a con 0x7f38c01334d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:21.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient(hunting): picked mon.noname-c con 0x7f38c0136fe0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:21.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c009b350 con 0x7f38c01334d0 2024-09-16T14:47:21.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c009f2e0 con 0x7f38c0136fe0 2024-09-16T14:47:21.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c009b070 con 0x7f38c0099a70 2024-09-16T14:47:21.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:21.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient(hunting): authenticate will time out at 1223.764396s 2024-09-16T14:47:21.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c4f46640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f38c01334d0 legacy=0x7f38c01358c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:21.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c5f48640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f38c0136fe0 0x7f38c01393d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46398/0 (socket says 172.21.15.189:46398) 2024-09-16T14:47:21.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c5f48640 1 -- 172.21.15.189:0/2223934160 learned_addr learned my addr 172.21.15.189:0/2223934160 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:21.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 1 -- 172.21.15.189:0/2223934160 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372279874 0 0) 0x7f38c009f2e0 con 0x7f38c0136fe0 2024-09-16T14:47:21.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:21.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:21.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:21.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 10 monclient(hunting): my global_id is 94903 2024-09-16T14:47:21.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 1 -- 172.21.15.189:0/2223934160 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38ac003530 con 0x7f38c0136fe0 2024-09-16T14:47:21.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 1 -- 172.21.15.189:0/2223934160 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3580390029 0 0) 0x7f38ac003530 con 0x7f38c0136fe0 2024-09-16T14:47:21.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 1 -- 172.21.15.189:0/2223934160 >> v1:172.21.15.26:6790/0 conn(0x7f38c0099a70 legacy=0x7f38c009bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:21.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 1 -- 172.21.15.189:0/2223934160 >> v1:172.21.15.26:6789/0 conn(0x7f38c01334d0 legacy=0x7f38c01358c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:21.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:47:21.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 1 monclient: found mon.noname-c 2024-09-16T14:47:21.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:21.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:21.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 1 -- 172.21.15.189:0/2223934160 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c009b630 con 0x7f38c0136fe0 2024-09-16T14:47:21.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 10 monclient: _finish_auth 0 2024-09-16T14:47:21.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:47:21.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:21.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 1 -- 172.21.15.189:0/2223934160 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f38b80019a0 con 0x7f38c0136fe0 2024-09-16T14:47:21.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 5 monclient: authenticate success, global_id 94903 2024-09-16T14:47:21.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:21.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:21.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:21.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 10 monclient: dump: 2024-09-16T14:47:21.659 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:21.659 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:21.659 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:21.659 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:21.659 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:21.659 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:21.660 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:21.660 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:21.660 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:21.660 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:21.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:21.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 1 -- 172.21.15.189:0/2223934160 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38b8001c50 con 0x7f38c0136fe0 2024-09-16T14:47:21.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:21.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 1 -- 172.21.15.189:0/2223934160 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f38b8001fe0 con 0x7f38c0136fe0 2024-09-16T14:47:21.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:21.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:21.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b7fff640 10 monclient: dump: 2024-09-16T14:47:21.661 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:21.662 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:21.662 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:21.662 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:21.662 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:21.662 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:21.662 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:21.662 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:21.662 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:21.662 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:21.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:21.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 4 set_mon_vals no callback set 2024-09-16T14:47:21.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient: shutdown 2024-09-16T14:47:21.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:21.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 1 -- 172.21.15.189:0/2223934160 >> v1:172.21.15.189:6789/0 conn(0x7f38c0136fe0 legacy=0x7f38c01393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:21.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 1 -- 172.21.15.189:0/2223934160 shutdown_connections 2024-09-16T14:47:21.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 1 -- 172.21.15.189:0/2223934160 >> 172.21.15.189:0/2223934160 conn(0x7f38c00a3320 msgr2=0x7f38c009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:21.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 1 -- 172.21.15.189:0/2223934160 shutdown_connections 2024-09-16T14:47:21.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 1 -- 172.21.15.189:0/2223934160 wait complete. 2024-09-16T14:47:21.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient: build_initial_monmap 2024-09-16T14:47:21.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient: monmap: 2024-09-16T14:47:21.663 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:21.663 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:21.663 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:21.664 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:21.664 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:21.664 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:21.664 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:21.664 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:21.664 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:21.664 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:21.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 1 Processor -- start 2024-09-16T14:47:21.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 1 -- start start 2024-09-16T14:47:21.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient: init 2024-09-16T14:47:21.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:21.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:21.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient(hunting): picked mon.noname-b con 0x7f38c0099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:21.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient(hunting): picked mon.noname-c con 0x7f38c01334d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:21.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient(hunting): picked mon.noname-a con 0x7f38c0136fe0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:21.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c011fb30 con 0x7f38c0136fe0 2024-09-16T14:47:21.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c011e410 con 0x7f38c0099a70 2024-09-16T14:47:21.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c00a1b20 con 0x7f38c01334d0 2024-09-16T14:47:21.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:21.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient(hunting): authenticate will time out at 1223.767195s 2024-09-16T14:47:21.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c5747640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f38c0099a70 0x7f38c01c73e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46406/0 (socket says 172.21.15.189:46406) 2024-09-16T14:47:21.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c5747640 1 -- 172.21.15.189:0/3316184322 learned_addr learned my addr 172.21.15.189:0/3316184322 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:21.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c5f48640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f38c0136fe0 legacy=0x7f38c01cfaf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:21.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 1 -- 172.21.15.189:0/3316184322 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196126399 0 0) 0x7f38c011e410 con 0x7f38c0099a70 2024-09-16T14:47:21.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:21.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:21.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:21.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 10 monclient(hunting): my global_id is 94906 2024-09-16T14:47:21.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 1 -- 172.21.15.189:0/3316184322 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a4003560 con 0x7f38c0099a70 2024-09-16T14:47:21.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 1 -- 172.21.15.189:0/3316184322 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1141800045 0 0) 0x7f38a4003560 con 0x7f38c0099a70 2024-09-16T14:47:21.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 1 -- 172.21.15.189:0/3316184322 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38c011e410 con 0x7f38c0099a70 2024-09-16T14:47:21.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 1 -- 172.21.15.189:0/3316184322 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f38bc001940 con 0x7f38c0099a70 2024-09-16T14:47:21.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:21.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:21.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 10 monclient(hunting): dump: 2024-09-16T14:47:21.669 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:21.669 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:21.669 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:21.669 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:21.669 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:21.669 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:21.669 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:21.669 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:21.670 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:21.670 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:21.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:21.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 1 -- 172.21.15.189:0/3316184322 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3727650456 0 0) 0x7f38c011e410 con 0x7f38c0099a70 2024-09-16T14:47:21.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 1 -- 172.21.15.189:0/3316184322 >> v1:172.21.15.26:6790/0 conn(0x7f38c01334d0 legacy=0x7f38c01cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:21.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 1 -- 172.21.15.189:0/3316184322 >> v1:172.21.15.26:6789/0 conn(0x7f38c0136fe0 legacy=0x7f38c01cfaf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:21.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:47:21.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 1 monclient: found mon.b 2024-09-16T14:47:21.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:21.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:21.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 1 -- 172.21.15.189:0/3316184322 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c011f850 con 0x7f38c0099a70 2024-09-16T14:47:21.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 10 monclient: _finish_auth 0 2024-09-16T14:47:21.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:21.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:21.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 5 monclient: authenticate success, global_id 94906 2024-09-16T14:47:21.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient: _renew_subs 2024-09-16T14:47:21.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:21.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.646+0000 7f38c79ab640 1 -- 172.21.15.189:0/3316184322 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c011d7d0 con 0x7f38c0099a70 2024-09-16T14:47:21.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38c79ab640 10 monclient: _renew_subs 2024-09-16T14:47:21.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38c79ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:21.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38c79ab640 1 -- 172.21.15.189:0/3316184322 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c0136380 con 0x7f38c0099a70 2024-09-16T14:47:21.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38b5ffb640 1 -- 172.21.15.189:0/3316184322 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38bc001d10 con 0x7f38c0099a70 2024-09-16T14:47:21.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:21.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38b5ffb640 1 -- 172.21.15.189:0/3316184322 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f38bc003a70 con 0x7f38c0099a70 2024-09-16T14:47:21.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:21.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38b6ffd640 4 set_mon_vals no callback set 2024-09-16T14:47:21.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:21.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38b5ffb640 10 monclient: dump: 2024-09-16T14:47:21.673 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:21.673 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:21.673 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:21.673 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:21.673 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:21.673 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:21.674 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:21.674 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:21.674 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:21.674 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:21.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38b5ffb640 1 -- 172.21.15.189:0/3316184322 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f38bc003bd0 con 0x7f38c0099a70 2024-09-16T14:47:21.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38c79ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:21.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38c79ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:21.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38c79ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:21.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38c79ab640 1 -- 172.21.15.189:0/3316184322 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a0005050 con 0x7f38c0099a70 2024-09-16T14:47:21.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38b5ffb640 1 -- 172.21.15.189:0/3316184322 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(114..114 src has 73..114) v4 ==== 5789+0+0 (unknown 1728059977 0 0) 0x7f38bc043aa0 con 0x7f38c0099a70 2024-09-16T14:47:21.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.650+0000 7f38c4f46640 10 monclient: get_auth_request con 0x7f38a4036080 auth_method 0 2024-09-16T14:47:21.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.654+0000 7f38b5ffb640 1 -- 172.21.15.189:0/3316184322 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f38bc014740 con 0x7f38c0099a70 2024-09-16T14:47:21.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.654+0000 7f38b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:21.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.654+0000 7f38b5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:21.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.806+0000 7f38c79ab640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:47:21.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.806+0000 7f38c79ab640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T14:47:21.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.806+0000 7f38c79ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:21.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.806+0000 7f38c79ab640 1 -- 172.21.15.189:0/3316184322 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f38a0005b10 con 0x7f38c0099a70 2024-09-16T14:47:21.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.810+0000 7f38b5ffb640 1 -- 172.21.15.189:0/3316184322 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v114) v1 ==== 56+0+1449 (unknown 3809803480 0 3599791968) 0x7f38bc014740 con 0x7f38c0099a70 2024-09-16T14:47:21.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.810+0000 7f38b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T14:47:21.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.810+0000 7f38b5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:21.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.898+0000 7f38c79ab640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T14:47:21.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.898+0000 7f38c79ab640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T14:47:21.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.898+0000 7f38c79ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:21.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.898+0000 7f38c79ab640 1 -- 172.21.15.189:0/3316184322 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f38a00055a0 con 0x7f38c0099a70 2024-09-16T14:47:21.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.898+0000 7f38b5ffb640 1 -- 172.21.15.189:0/3316184322 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v114) v1 ==== 56+0+1449 (unknown 3809803480 0 3599791968) 0x7f38bc015070 con 0x7f38c0099a70 2024-09-16T14:47:21.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.898+0000 7f38b5ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T14:47:21.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.898+0000 7f38b5ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:21.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.902+0000 7f38c79ab640 1 -- 172.21.15.189:0/3316184322 >> v1:172.21.15.189:6812/3196161582 conn(0x7f38a4036080 legacy=0x7f38a4038540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:21.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.902+0000 7f38c79ab640 10 monclient: shutdown 2024-09-16T14:47:21.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.902+0000 7f38c79ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:21.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.902+0000 7f38c79ab640 1 -- 172.21.15.189:0/3316184322 >> v1:172.21.15.189:6789/0 conn(0x7f38c0099a70 legacy=0x7f38c01c73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:21.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.902+0000 7f38c79ab640 1 -- 172.21.15.189:0/3316184322 shutdown_connections 2024-09-16T14:47:21.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.902+0000 7f38c79ab640 1 -- 172.21.15.189:0/3316184322 >> 172.21.15.189:0/3316184322 conn(0x7f38c00a3320 msgr2=0x7f38c006c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:21.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.902+0000 7f38c79ab640 1 -- 172.21.15.189:0/3316184322 shutdown_connections 2024-09-16T14:47:21.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:21.902+0000 7f38c79ab640 1 -- 172.21.15.189:0/3316184322 wait complete. 2024-09-16T14:47:21.919 INFO:tasks.workunit.client.0.smithi189.stdout: 14 155.00000 osd.14 down 1.00000 1.00000 2024-09-16T14:47:21.920 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm osd.12 2024-09-16T14:47:22.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 10 monclient: get_monmap_and_config 2024-09-16T14:47:22.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 10 monclient: build_initial_monmap 2024-09-16T14:47:22.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 10 monclient: monmap: 2024-09-16T14:47:22.035 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:22.035 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:22.036 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:22.035509+0000 2024-09-16T14:47:22.036 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:22.035509+0000 2024-09-16T14:47:22.036 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:22.036 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:22.036 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:22.036 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:22.036 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:22.036 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:22.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 1 Processor -- start 2024-09-16T14:47:22.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 1 -- start start 2024-09-16T14:47:22.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 10 monclient: init 2024-09-16T14:47:22.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:22.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:22.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 10 monclient(hunting): picked mon.noname-c con 0x7f85c01315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:22.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 10 monclient(hunting): picked mon.noname-a con 0x7f85c009b320 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:22.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 10 monclient(hunting): picked mon.noname-b con 0x7f85c0132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:22.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c009edd0 con 0x7f85c009b320 2024-09-16T14:47:22.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c009bc00 con 0x7f85c01315a0 2024-09-16T14:47:22.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c009e900 con 0x7f85c0132090 2024-09-16T14:47:22.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:22.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 10 monclient(hunting): authenticate will time out at 1224.151227s 2024-09-16T14:47:22.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bed9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f85c01315a0 0x7f85c0131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46414/0 (socket says 172.21.15.189:46414) 2024-09-16T14:47:22.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bed9d640 1 -- 172.21.15.189:0/3763174119 learned_addr learned my addr 172.21.15.189:0/3763174119 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:22.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85be59c640 1 -- 172.21.15.189:0/3763174119 >> v1:172.21.15.26:6789/0 conn(0x7f85c009b320 legacy=0x7f85c0136ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:22.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 1 -- 172.21.15.189:0/3763174119 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359758378 0 0) 0x7f85c009bc00 con 0x7f85c01315a0 2024-09-16T14:47:22.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:47:22.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:22.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:22.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 10 monclient(hunting): my global_id is 94909 2024-09-16T14:47:22.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 1 -- 172.21.15.189:0/3763174119 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85a8003530 con 0x7f85c01315a0 2024-09-16T14:47:22.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 1 -- 172.21.15.189:0/3763174119 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2096798406 0 0) 0x7f85a8003530 con 0x7f85c01315a0 2024-09-16T14:47:22.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 1 -- 172.21.15.189:0/3763174119 >> v1:172.21.15.26:6790/0 conn(0x7f85c0132090 legacy=0x7f85c013a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:22.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 1 -- 172.21.15.189:0/3763174119 >> v1:172.21.15.26:6789/0 conn(0x7f85c009b320 legacy=0x7f85c0136ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:22.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 10 monclient: _finish_hunting 0 2024-09-16T14:47:22.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 1 monclient: found mon.noname-c 2024-09-16T14:47:22.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:22.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:22.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 1 -- 172.21.15.189:0/3763174119 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c009b920 con 0x7f85c01315a0 2024-09-16T14:47:22.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 10 monclient: _finish_auth 0 2024-09-16T14:47:22.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 10 monclient: _check_auth_tickets 2024-09-16T14:47:22.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:22.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 1 -- 172.21.15.189:0/3763174119 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f85b0001620 con 0x7f85c01315a0 2024-09-16T14:47:22.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 5 monclient: authenticate success, global_id 94909 2024-09-16T14:47:22.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:22.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:22.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:22.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 10 monclient: dump: 2024-09-16T14:47:22.045 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:22.045 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:22.045 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:22.045 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:22.045 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:22.045 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:22.045 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 1 -- 172.21.15.189:0/3763174119 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85b0001940 con 0x7f85c01315a0 2024-09-16T14:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 1 -- 172.21.15.189:0/3763174119 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f85b0001cd0 con 0x7f85c01315a0 2024-09-16T14:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85c503f640 4 set_mon_vals no callback set 2024-09-16T14:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.030+0000 7f85bdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85bdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85bdd9b640 10 monclient: dump: 2024-09-16T14:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient: shutdown 2024-09-16T14:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 1 -- 172.21.15.189:0/3763174119 >> v1:172.21.15.189:6789/0 conn(0x7f85c01315a0 legacy=0x7f85c0131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 1 -- 172.21.15.189:0/3763174119 shutdown_connections 2024-09-16T14:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 1 -- 172.21.15.189:0/3763174119 >> 172.21.15.189:0/3763174119 conn(0x7f85c00a3320 msgr2=0x7f85c009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 1 -- 172.21.15.189:0/3763174119 shutdown_connections 2024-09-16T14:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 1 -- 172.21.15.189:0/3763174119 wait complete. 2024-09-16T14:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient: build_initial_monmap 2024-09-16T14:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient: monmap: 2024-09-16T14:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:22.049 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:22.049 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:22.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 1 Processor -- start 2024-09-16T14:47:22.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 1 -- start start 2024-09-16T14:47:22.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient: init 2024-09-16T14:47:22.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:22.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:22.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient(hunting): picked mon.noname-a con 0x7f85c01315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:22.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient(hunting): picked mon.noname-b con 0x7f85c0132090 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:22.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient(hunting): picked mon.noname-c con 0x7f85c0130f90 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:22.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c009cd20 con 0x7f85c01315a0 2024-09-16T14:47:22.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c011e240 con 0x7f85c0132090 2024-09-16T14:47:22.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c00a0eb0 con 0x7f85c0130f90 2024-09-16T14:47:22.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:22.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient(hunting): authenticate will time out at 1224.154072s 2024-09-16T14:47:22.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85bed9d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f85c01315a0 legacy=0x7f85c0130880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:22.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85be59c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f85c0132090 0x7f85c01cc0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46424/0 (socket says 172.21.15.189:46424) 2024-09-16T14:47:22.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85be59c640 1 -- 172.21.15.189:0/3158889828 learned_addr learned my addr 172.21.15.189:0/3158889828 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:22.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 1 -- 172.21.15.189:0/3158889828 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986766342 0 0) 0x7f85c011e240 con 0x7f85c0132090 2024-09-16T14:47:22.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:22.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:22.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:22.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 10 monclient(hunting): my global_id is 94912 2024-09-16T14:47:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 1 -- 172.21.15.189:0/3158889828 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85ac003530 con 0x7f85c0132090 2024-09-16T14:47:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 1 -- 172.21.15.189:0/3158889828 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1209591017 0 0) 0x7f85ac003530 con 0x7f85c0132090 2024-09-16T14:47:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 1 -- 172.21.15.189:0/3158889828 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85c011e240 con 0x7f85c0132090 2024-09-16T14:47:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 1 -- 172.21.15.189:0/3158889828 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f85b4004190 con 0x7f85c0132090 2024-09-16T14:47:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 10 monclient(hunting): dump: 2024-09-16T14:47:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 1 -- 172.21.15.189:0/3158889828 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2917910596 0 0) 0x7f85c011e240 con 0x7f85c0132090 2024-09-16T14:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 1 -- 172.21.15.189:0/3158889828 >> v1:172.21.15.26:6790/0 conn(0x7f85c0130f90 legacy=0x7f85c01cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 1 -- 172.21.15.189:0/3158889828 >> v1:172.21.15.26:6789/0 conn(0x7f85c01315a0 legacy=0x7f85c0130880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 1 monclient: found mon.b 2024-09-16T14:47:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 1 -- 172.21.15.189:0/3158889828 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c006b380 con 0x7f85c0132090 2024-09-16T14:47:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 5 monclient: authenticate success, global_id 94912 2024-09-16T14:47:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient: _renew_subs 2024-09-16T14:47:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 1 -- 172.21.15.189:0/3158889828 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c009a7f0 con 0x7f85c0132090 2024-09-16T14:47:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient: _renew_subs 2024-09-16T14:47:22.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:22.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 1 -- 172.21.15.189:0/3158889828 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c011be70 con 0x7f85c0132090 2024-09-16T14:47:22.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 1 -- 172.21.15.189:0/3158889828 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85b4004480 con 0x7f85c0132090 2024-09-16T14:47:22.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:22.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 1 -- 172.21.15.189:0/3158889828 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f85b4004790 con 0x7f85c0132090 2024-09-16T14:47:22.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:22.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:22.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85bd59a640 4 set_mon_vals no callback set 2024-09-16T14:47:22.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 10 monclient: dump: 2024-09-16T14:47:22.054 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:22.054 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:22.054 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:22.054 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:22.055 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:22.055 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:22.055 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:22.055 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:22.055 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:22.055 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:22.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:22.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:22.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:22.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85a77fe640 1 -- 172.21.15.189:0/3158889828 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f85b4004970 con 0x7f85c0132090 2024-09-16T14:47:22.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.034+0000 7f85c503f640 1 -- 172.21.15.189:0/3158889828 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a0005050 con 0x7f85c0132090 2024-09-16T14:47:22.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.038+0000 7f85a77fe640 1 -- 172.21.15.189:0/3158889828 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(114..114 src has 73..114) v4 ==== 5789+0+0 (unknown 1728059977 0 0) 0x7f85b4043ad0 con 0x7f85c0132090 2024-09-16T14:47:22.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.038+0000 7f85bed9d640 10 monclient: get_auth_request con 0x7f85ac036030 auth_method 0 2024-09-16T14:47:22.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.042+0000 7f85a77fe640 1 -- 172.21.15.189:0/3158889828 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f85b4005190 con 0x7f85c0132090 2024-09-16T14:47:22.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.042+0000 7f85a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:22.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.042+0000 7f85a77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:22.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.202+0000 7f85c503f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T14:47:22.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.202+0000 7f85c503f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T14:47:22.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.202+0000 7f85c503f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:22.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.202+0000 7f85c503f640 1 -- 172.21.15.189:0/3158889828 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) v1 -- 0x7f85a0005300 con 0x7f85c0132090 2024-09-16T14:47:22.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.838+0000 7f85a77fe640 1 -- 172.21.15.189:0/3158889828 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.12"}]=0 removed item id 12 name 'osd.12' from crush map v115) v1 ==== 125+0+0 (unknown 4113640734 0 0) 0x7f85b4012c40 con 0x7f85c0132090 2024-09-16T14:47:22.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.838+0000 7f85a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T14:47:22.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.838+0000 7f85a77fe640 10 monclient: _finish_command 2 = system:0 removed item id 12 name 'osd.12' from crush map 2024-09-16T14:47:22.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.938+0000 7f85c503f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T14:47:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.938+0000 7f85c503f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T14:47:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.938+0000 7f85c503f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.938+0000 7f85c503f640 1 -- 172.21.15.189:0/3158889828 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) v1 -- 0x7f85a00055a0 con 0x7f85c0132090 2024-09-16T14:47:22.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.942+0000 7f85a77fe640 1 -- 172.21.15.189:0/3158889828 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.12"}]=0 device 'osd.12' does not appear in the crush map v115) v1 ==== 126+0+0 (unknown 2394366590 0 0) 0x7f85b40143c0 con 0x7f85c0132090 2024-09-16T14:47:22.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.942+0000 7f85a77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T14:47:22.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.942+0000 7f85a77fe640 10 monclient: _finish_command 3 = system:0 device 'osd.12' does not appear in the crush map 2024-09-16T14:47:22.946 INFO:tasks.workunit.client.0.smithi189.stderr:device 'osd.12' does not appear in the crush map 2024-09-16T14:47:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.942+0000 7f85c503f640 1 -- 172.21.15.189:0/3158889828 >> v1:172.21.15.189:6812/3196161582 conn(0x7f85ac036030 legacy=0x7f85ac0384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.942+0000 7f85c503f640 10 monclient: shutdown 2024-09-16T14:47:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.942+0000 7f85c503f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.942+0000 7f85c503f640 1 -- 172.21.15.189:0/3158889828 >> v1:172.21.15.189:6789/0 conn(0x7f85c0132090 legacy=0x7f85c01cc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.942+0000 7f85c503f640 1 -- 172.21.15.189:0/3158889828 shutdown_connections 2024-09-16T14:47:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.942+0000 7f85c503f640 1 -- 172.21.15.189:0/3158889828 >> 172.21.15.189:0/3158889828 conn(0x7f85c00a3320 msgr2=0x7f85c0070640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.942+0000 7f85c503f640 1 -- 172.21.15.189:0/3158889828 shutdown_connections 2024-09-16T14:47:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:22.942+0000 7f85c503f640 1 -- 172.21.15.189:0/3158889828 wait complete. 2024-09-16T14:47:22.963 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm osd.14 2024-09-16T14:47:23.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 10 monclient: get_monmap_and_config 2024-09-16T14:47:23.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 10 monclient: build_initial_monmap 2024-09-16T14:47:23.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 10 monclient: monmap: 2024-09-16T14:47:23.083 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:23.083 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:23.083 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:23.083471+0000 2024-09-16T14:47:23.084 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:23.083471+0000 2024-09-16T14:47:23.084 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:23.084 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:23.084 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:23.084 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:23.084 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:23.084 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:23.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 1 Processor -- start 2024-09-16T14:47:23.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 1 -- start start 2024-09-16T14:47:23.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 10 monclient: init 2024-09-16T14:47:23.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:23.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:23.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 10 monclient(hunting): picked mon.noname-a con 0x7f34e809df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:23.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 10 monclient(hunting): picked mon.noname-c con 0x7f34e809e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:23.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 10 monclient(hunting): picked mon.noname-b con 0x7f34e809e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:23.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e809ee20 con 0x7f34e809df10 2024-09-16T14:47:23.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e809f100 con 0x7f34e809e2f0 2024-09-16T14:47:23.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e809f3e0 con 0x7f34e809e740 2024-09-16T14:47:23.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:23.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 10 monclient(hunting): authenticate will time out at 1225.199049s 2024-09-16T14:47:23.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e6ffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f34e809df10 legacy=0x7f34e809d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:23.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34dffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f34e809e2f0 0x7f34e81378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46440/0 (socket says 172.21.15.189:46440) 2024-09-16T14:47:23.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34dffff640 1 -- 172.21.15.189:0/2060376488 learned_addr learned my addr 172.21.15.189:0/2060376488 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:23.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 1 -- 172.21.15.189:0/2060376488 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473385140 0 0) 0x7f34e809f100 con 0x7f34e809e2f0 2024-09-16T14:47:23.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:23.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:23.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:23.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 10 monclient(hunting): my global_id is 94915 2024-09-16T14:47:23.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 1 -- 172.21.15.189:0/2060376488 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d4003530 con 0x7f34e809e2f0 2024-09-16T14:47:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 1 -- 172.21.15.189:0/2060376488 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1609080628 0 0) 0x7f34d4003530 con 0x7f34e809e2f0 2024-09-16T14:47:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 1 -- 172.21.15.189:0/2060376488 >> v1:172.21.15.26:6790/0 conn(0x7f34e809e740 legacy=0x7f34e813b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 1 -- 172.21.15.189:0/2060376488 >> v1:172.21.15.26:6789/0 conn(0x7f34e809df10 legacy=0x7f34e809d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 1 monclient: found mon.noname-c 2024-09-16T14:47:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 1 -- 172.21.15.189:0/2060376488 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e809c690 con 0x7f34e809e2f0 2024-09-16T14:47:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 1 -- 172.21.15.189:0/2060376488 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34d0001620 con 0x7f34e809e2f0 2024-09-16T14:47:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 5 monclient: authenticate success, global_id 94915 2024-09-16T14:47:23.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:23.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:23.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:23.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 10 monclient: dump: 2024-09-16T14:47:23.093 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:23.093 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:23.093 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:23.093 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:23.093 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:23.093 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:23.093 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:23.093 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:23.093 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:23.094 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:23.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34ed4b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:23.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.078+0000 7f34e67fc640 1 -- 172.21.15.189:0/2060376488 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34d0001940 con 0x7f34e809e2f0 2024-09-16T14:47:23.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34e67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:23.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:23.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 4 set_mon_vals no callback set 2024-09-16T14:47:23.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34e67fc640 1 -- 172.21.15.189:0/2060376488 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34d0001cd0 con 0x7f34e809e2f0 2024-09-16T14:47:23.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:23.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:23.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34e67fc640 10 monclient: dump: 2024-09-16T14:47:23.094 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:23.094 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:23.094 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:23.095 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:23.095 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:23.095 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:23.095 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:23.095 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:23.095 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:23.095 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:23.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient: shutdown 2024-09-16T14:47:23.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:23.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 1 -- 172.21.15.189:0/2060376488 >> v1:172.21.15.189:6789/0 conn(0x7f34e809e2f0 legacy=0x7f34e81378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:23.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 1 -- 172.21.15.189:0/2060376488 shutdown_connections 2024-09-16T14:47:23.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 1 -- 172.21.15.189:0/2060376488 >> 172.21.15.189:0/2060376488 conn(0x7f34e80a3320 msgr2=0x7f34e809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:23.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 1 -- 172.21.15.189:0/2060376488 shutdown_connections 2024-09-16T14:47:23.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 1 -- 172.21.15.189:0/2060376488 wait complete. 2024-09-16T14:47:23.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient: build_initial_monmap 2024-09-16T14:47:23.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient: monmap: 2024-09-16T14:47:23.096 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:23.096 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:23.096 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:23.096 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:23.096 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:23.096 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:23.096 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:23.096 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:23.096 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:23.096 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:23.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 1 Processor -- start 2024-09-16T14:47:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 1 -- start start 2024-09-16T14:47:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient: init 2024-09-16T14:47:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient(hunting): picked mon.noname-b con 0x7f34e809df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient(hunting): picked mon.noname-a con 0x7f34e809e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient(hunting): picked mon.noname-c con 0x7f34e809e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e81284a0 con 0x7f34e809e2f0 2024-09-16T14:47:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e806cdc0 con 0x7f34e809df10 2024-09-16T14:47:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e806b380 con 0x7f34e809e740 2024-09-16T14:47:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient(hunting): authenticate will time out at 1225.202126s 2024-09-16T14:47:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34dffff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f34e809e2f0 legacy=0x7f34e809ab90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:23.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34e6ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f34e809df10 0x7f34e809a480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46456/0 (socket says 172.21.15.189:46456) 2024-09-16T14:47:23.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34e6ffd640 1 -- 172.21.15.189:0/1651573324 learned_addr learned my addr 172.21.15.189:0/1651573324 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:23.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281736924 0 0) 0x7f34e806cdc0 con 0x7f34e809df10 2024-09-16T14:47:23.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:23.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:23.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:23.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient(hunting): my global_id is 94918 2024-09-16T14:47:23.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34c8003560 con 0x7f34e809df10 2024-09-16T14:47:23.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2602688192 0 0) 0x7f34c8003560 con 0x7f34e809df10 2024-09-16T14:47:23.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34e806cdc0 con 0x7f34e809df10 2024-09-16T14:47:23.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34d8004190 con 0x7f34e809df10 2024-09-16T14:47:23.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:23.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:23.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient(hunting): dump: 2024-09-16T14:47:23.099 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:23.099 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:23.099 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:23.099 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:23.099 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:23.099 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:23.099 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:23.099 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:23.099 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:23.099 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:23.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:23.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424835923 0 0) 0x7f34e806b380 con 0x7f34e809e740 2024-09-16T14:47:23.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:23.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:23.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:23.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient(hunting): my global_id is 114296 2024-09-16T14:47:23.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34c8003560 con 0x7f34e809e740 2024-09-16T14:47:23.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3983604264 0 0) 0x7f34e806cdc0 con 0x7f34e809df10 2024-09-16T14:47:23.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 >> v1:172.21.15.26:6790/0 conn(0x7f34e809e740 legacy=0x7f34e81d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:23.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 >> v1:172.21.15.26:6789/0 conn(0x7f34e809e2f0 legacy=0x7f34e809ab90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:23.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:23.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 1 monclient: found mon.b 2024-09-16T14:47:23.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:23.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:23.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e8122640 con 0x7f34e809df10 2024-09-16T14:47:23.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:23.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:23.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:23.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 5 monclient: authenticate success, global_id 94918 2024-09-16T14:47:23.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient: _renew_subs 2024-09-16T14:47:23.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:23.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 1 -- 172.21.15.189:0/1651573324 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e81211f0 con 0x7f34e809df10 2024-09-16T14:47:23.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient: _renew_subs 2024-09-16T14:47:23.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:23.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 1 -- 172.21.15.189:0/1651573324 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e809bec0 con 0x7f34e809df10 2024-09-16T14:47:23.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34d80044b0 con 0x7f34e809df10 2024-09-16T14:47:23.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:23.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34d8004840 con 0x7f34e809df10 2024-09-16T14:47:23.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:23.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:23.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34e5ffb640 4 set_mon_vals no callback set 2024-09-16T14:47:23.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 10 monclient: dump: 2024-09-16T14:47:23.102 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:23.102 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:23.102 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:23.102 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:23.102 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:23.102 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:23.102 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:23.102 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:23.102 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:23.102 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:23.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:23.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:23.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:23.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f34d8004a20 con 0x7f34e809df10 2024-09-16T14:47:23.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.082+0000 7f34ed4b0640 1 -- 172.21.15.189:0/1651573324 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34cc005050 con 0x7f34e809df10 2024-09-16T14:47:23.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.086+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(115..115 src has 73..115) v4 ==== 5767+0+0 (unknown 3819318646 0 0) 0x7f34d8043b90 con 0x7f34e809df10 2024-09-16T14:47:23.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.086+0000 7f34dffff640 10 monclient: get_auth_request con 0x7f34c8035f70 auth_method 0 2024-09-16T14:47:23.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.090+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f34d80148a0 con 0x7f34e809df10 2024-09-16T14:47:23.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.090+0000 7f34df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:23.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.090+0000 7f34df7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:23.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.238+0000 7f34ed4b0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T14:47:23.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.238+0000 7f34ed4b0640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T14:47:23.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.238+0000 7f34ed4b0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:23.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.238+0000 7f34ed4b0640 1 -- 172.21.15.189:0/1651573324 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.14"} v 0) v1 -- 0x7f34cc005300 con 0x7f34e809df10 2024-09-16T14:47:23.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.846+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.14"}]=0 removed item id 14 name 'osd.14' from crush map v116) v1 ==== 125+0+0 (unknown 2333909030 0 0) 0x7f34d8012d10 con 0x7f34e809df10 2024-09-16T14:47:23.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.846+0000 7f34df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T14:47:23.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.846+0000 7f34df7fe640 10 monclient: _finish_command 2 = system:0 removed item id 14 name 'osd.14' from crush map 2024-09-16T14:47:23.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.950+0000 7f34ed4b0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T14:47:23.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.950+0000 7f34ed4b0640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T14:47:23.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.950+0000 7f34ed4b0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:23.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.950+0000 7f34ed4b0640 1 -- 172.21.15.189:0/1651573324 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.14"} v 0) v1 -- 0x7f34cc0055a0 con 0x7f34e809df10 2024-09-16T14:47:23.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.954+0000 7f34df7fe640 1 -- 172.21.15.189:0/1651573324 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.14"}]=0 device 'osd.14' does not appear in the crush map v116) v1 ==== 126+0+0 (unknown 4114815694 0 0) 0x7f34d8014670 con 0x7f34e809df10 2024-09-16T14:47:23.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.954+0000 7f34df7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T14:47:23.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.954+0000 7f34df7fe640 10 monclient: _finish_command 3 = system:0 device 'osd.14' does not appear in the crush map 2024-09-16T14:47:23.959 INFO:tasks.workunit.client.0.smithi189.stderr:device 'osd.14' does not appear in the crush map 2024-09-16T14:47:23.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.954+0000 7f34ed4b0640 1 -- 172.21.15.189:0/1651573324 >> v1:172.21.15.189:6812/3196161582 conn(0x7f34c8035f70 legacy=0x7f34c8038430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:23.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.954+0000 7f34ed4b0640 10 monclient: shutdown 2024-09-16T14:47:23.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.954+0000 7f34ed4b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:23.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.954+0000 7f34ed4b0640 1 -- 172.21.15.189:0/1651573324 >> v1:172.21.15.189:6789/0 conn(0x7f34e809df10 legacy=0x7f34e809a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:23.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.954+0000 7f34ed4b0640 1 -- 172.21.15.189:0/1651573324 shutdown_connections 2024-09-16T14:47:23.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.954+0000 7f34ed4b0640 1 -- 172.21.15.189:0/1651573324 >> 172.21.15.189:0/1651573324 conn(0x7f34e80a3320 msgr2=0x7f34e812a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:23.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.954+0000 7f34ed4b0640 1 -- 172.21.15.189:0/1651573324 shutdown_connections 2024-09-16T14:47:23.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:23.954+0000 7f34ed4b0640 1 -- 172.21.15.189:0/1651573324 wait complete. 2024-09-16T14:47:23.975 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd rm osd.12 2024-09-16T14:47:24.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 10 monclient: get_monmap_and_config 2024-09-16T14:47:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 10 monclient: build_initial_monmap 2024-09-16T14:47:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 10 monclient: monmap: 2024-09-16T14:47:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:24.091004+0000 2024-09-16T14:47:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:24.091004+0000 2024-09-16T14:47:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:24.091 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:24.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 1 Processor -- start 2024-09-16T14:47:24.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 1 -- start start 2024-09-16T14:47:24.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 10 monclient: init 2024-09-16T14:47:24.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:24.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:24.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 10 monclient(hunting): picked mon.noname-c con 0x7f9a2809df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:24.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 10 monclient(hunting): picked mon.noname-b con 0x7f9a2809e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:24.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 10 monclient(hunting): picked mon.noname-a con 0x7f9a2809e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:24.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2809ee20 con 0x7f9a2809e740 2024-09-16T14:47:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2809f100 con 0x7f9a2809df10 2024-09-16T14:47:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2809f3e0 con 0x7f9a2809e2f0 2024-09-16T14:47:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 10 monclient(hunting): authenticate will time out at 1226.206755s 2024-09-16T14:47:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a26ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9a2809df10 0x7f9a2809d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46458/0 (socket says 172.21.15.189:46458) 2024-09-16T14:47:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a26ffd640 1 -- 172.21.15.189:0/4172307422 learned_addr learned my addr 172.21.15.189:0/4172307422 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a277fe640 1 -- 172.21.15.189:0/4172307422 >> v1:172.21.15.26:6789/0 conn(0x7f9a2809e740 legacy=0x7f9a2813b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 1 -- 172.21.15.189:0/4172307422 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259048053 0 0) 0x7f9a2809f100 con 0x7f9a2809df10 2024-09-16T14:47:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 10 monclient(hunting): my global_id is 94921 2024-09-16T14:47:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 1 -- 172.21.15.189:0/4172307422 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a10003530 con 0x7f9a2809df10 2024-09-16T14:47:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 1 -- 172.21.15.189:0/4172307422 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4145625389 0 0) 0x7f9a10003530 con 0x7f9a2809df10 2024-09-16T14:47:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 1 -- 172.21.15.189:0/4172307422 >> v1:172.21.15.26:6790/0 conn(0x7f9a2809e2f0 legacy=0x7f9a281378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 1 -- 172.21.15.189:0/4172307422 >> v1:172.21.15.26:6789/0 conn(0x7f9a2809e740 legacy=0x7f9a2813b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:47:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 1 monclient: found mon.noname-c 2024-09-16T14:47:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 1 -- 172.21.15.189:0/4172307422 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2809c690 con 0x7f9a2809df10 2024-09-16T14:47:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 10 monclient: _finish_auth 0 2024-09-16T14:47:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 5 monclient: authenticate success, global_id 94921 2024-09-16T14:47:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 1 -- 172.21.15.189:0/4172307422 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9a200018d0 con 0x7f9a2809df10 2024-09-16T14:47:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 10 monclient: dump: 2024-09-16T14:47:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:24.102 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:24.102 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:24.102 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:24.102 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:24.102 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:24.102 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:24.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:24.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 1 -- 172.21.15.189:0/4172307422 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a20001bf0 con 0x7f9a2809df10 2024-09-16T14:47:24.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:24.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:24.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a2d495640 4 set_mon_vals no callback set 2024-09-16T14:47:24.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 1 -- 172.21.15.189:0/4172307422 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9a20001fc0 con 0x7f9a2809df10 2024-09-16T14:47:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.086+0000 7f9a25ffb640 10 monclient: dump: 2024-09-16T14:47:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:24.104 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient: shutdown 2024-09-16T14:47:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 1 -- 172.21.15.189:0/4172307422 >> v1:172.21.15.189:6789/0 conn(0x7f9a2809df10 legacy=0x7f9a2809d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 1 -- 172.21.15.189:0/4172307422 shutdown_connections 2024-09-16T14:47:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 1 -- 172.21.15.189:0/4172307422 >> 172.21.15.189:0/4172307422 conn(0x7f9a280a3320 msgr2=0x7f9a2809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 1 -- 172.21.15.189:0/4172307422 shutdown_connections 2024-09-16T14:47:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 1 -- 172.21.15.189:0/4172307422 wait complete. 2024-09-16T14:47:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient: build_initial_monmap 2024-09-16T14:47:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient: monmap: 2024-09-16T14:47:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:24.106 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 1 Processor -- start 2024-09-16T14:47:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 1 -- start start 2024-09-16T14:47:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient: init 2024-09-16T14:47:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient(hunting): picked mon.noname-c con 0x7f9a2809df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient(hunting): picked mon.noname-a con 0x7f9a2809e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient(hunting): picked mon.noname-b con 0x7f9a2809e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a28121bb0 con 0x7f9a2809e2f0 2024-09-16T14:47:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a281ca570 con 0x7f9a2809e740 2024-09-16T14:47:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2806f9d0 con 0x7f9a2809df10 2024-09-16T14:47:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient(hunting): authenticate will time out at 1226.209539s 2024-09-16T14:47:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a277fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9a2809e740 0x7f9a281d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46464/0 (socket says 172.21.15.189:46464) 2024-09-16T14:47:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a277fe640 1 -- 172.21.15.189:0/706423159 learned_addr learned my addr 172.21.15.189:0/706423159 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a267fc640 1 -- 172.21.15.189:0/706423159 >> v1:172.21.15.26:6789/0 conn(0x7f9a2809e2f0 legacy=0x7f9a281c9e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81213115 0 0) 0x7f9a281ca570 con 0x7f9a2809e740 2024-09-16T14:47:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient(hunting): my global_id is 94924 2024-09-16T14:47:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a14003530 con 0x7f9a2809e740 2024-09-16T14:47:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4031349449 0 0) 0x7f9a14003530 con 0x7f9a2809e740 2024-09-16T14:47:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a281ca570 con 0x7f9a2809e740 2024-09-16T14:47:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9a18004080 con 0x7f9a2809e740 2024-09-16T14:47:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient(hunting): dump: 2024-09-16T14:47:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:24.109 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:24.109 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:24.109 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:24.109 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:24.109 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:24.109 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:24.109 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:24.109 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:24.109 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:24.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:24.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572521826 0 0) 0x7f9a2806f9d0 con 0x7f9a2809df10 2024-09-16T14:47:24.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:24.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:24.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:24.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient(hunting): my global_id is 114299 2024-09-16T14:47:24.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a14003530 con 0x7f9a2809df10 2024-09-16T14:47:24.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 166380363 0 0) 0x7f9a281ca570 con 0x7f9a2809e740 2024-09-16T14:47:24.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 >> v1:172.21.15.26:6790/0 conn(0x7f9a2809df10 legacy=0x7f9a281c9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:24.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 >> v1:172.21.15.26:6789/0 conn(0x7f9a2809e2f0 legacy=0x7f9a281c9e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:24.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:24.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 1 monclient: found mon.b 2024-09-16T14:47:24.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:24.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:24.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2806cea0 con 0x7f9a2809e740 2024-09-16T14:47:24.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:24.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:24.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:24.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 5 monclient: authenticate success, global_id 94924 2024-09-16T14:47:24.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient: _renew_subs 2024-09-16T14:47:24.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:24.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 1 -- 172.21.15.189:0/706423159 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2806b380 con 0x7f9a2809e740 2024-09-16T14:47:24.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient: _renew_subs 2024-09-16T14:47:24.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:24.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 1 -- 172.21.15.189:0/706423159 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a281201a0 con 0x7f9a2809e740 2024-09-16T14:47:24.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a180043a0 con 0x7f9a2809e740 2024-09-16T14:47:24.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:24.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9a18004730 con 0x7f9a2809e740 2024-09-16T14:47:24.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:24.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:24.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 10 monclient: dump: 2024-09-16T14:47:24.112 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:24.113 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:24.113 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:24.113 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:24.113 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:24.113 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:24.113 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:24.113 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:24.113 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:24.113 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:24.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a257fa640 4 set_mon_vals no callback set 2024-09-16T14:47:24.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:24.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:24.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:24.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9a18004890 con 0x7f9a2809e740 2024-09-16T14:47:24.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a2d495640 1 -- 172.21.15.189:0/706423159 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a08005050 con 0x7f9a2809e740 2024-09-16T14:47:24.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(116..116 src has 73..116) v4 ==== 5745+0+0 (unknown 3281127532 0 0) 0x7f9a180439c0 con 0x7f9a2809e740 2024-09-16T14:47:24.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.090+0000 7f9a26ffd640 10 monclient: get_auth_request con 0x7f9a14036130 auth_method 0 2024-09-16T14:47:24.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.094+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9a18005090 con 0x7f9a2809e740 2024-09-16T14:47:24.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.094+0000 7f9a0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:24.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.094+0000 7f9a0f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:24.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.262+0000 7f9a2d495640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T14:47:24.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.262+0000 7f9a2d495640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T14:47:24.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.262+0000 7f9a2d495640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:24.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.262+0000 7f9a2d495640 1 -- 172.21.15.189:0/706423159 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.12"]} v 0) v1 -- 0x7f9a08005880 con 0x7f9a2809e740 2024-09-16T14:47:24.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:24.267+0000 7fc29af71640 -1 mon.c@2(leader).osd e116 definitely_dead 0 2024-09-16T14:47:24.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.854+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.12"]}]=0 removed osd.12 v117) v1 ==== 87+0+0 (unknown 4165419023 0 0) 0x7f9a18012b50 con 0x7f9a2809e740 2024-09-16T14:47:24.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.854+0000 7f9a0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T14:47:24.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.854+0000 7f9a0f7fe640 10 monclient: _finish_command 2 = system:0 removed osd.12 2024-09-16T14:47:24.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.966+0000 7f9a2d495640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T14:47:24.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.966+0000 7f9a2d495640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T14:47:24.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.966+0000 7f9a2d495640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:24.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.966+0000 7f9a2d495640 1 -- 172.21.15.189:0/706423159 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.12"]} v 0) v1 -- 0x7f9a08005370 con 0x7f9a2809e740 2024-09-16T14:47:24.971 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:24.967+0000 7fc29af71640 -1 mon.c@2(leader).osd e117 definitely_dead 0 2024-09-16T14:47:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.966+0000 7f9a0f7fe640 1 -- 172.21.15.189:0/706423159 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.12"]}]=0 osd.12 does not exist. v117) v1 ==== 96+0+0 (unknown 3191329385 0 0) 0x7f9a180142d0 con 0x7f9a2809e740 2024-09-16T14:47:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.966+0000 7f9a0f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T14:47:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.966+0000 7f9a0f7fe640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-16T14:47:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:osd.12 does not exist. 2024-09-16T14:47:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.970+0000 7f9a2d495640 1 -- 172.21.15.189:0/706423159 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9a14036130 legacy=0x7f9a140385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.970+0000 7f9a2d495640 10 monclient: shutdown 2024-09-16T14:47:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.970+0000 7f9a2d495640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.970+0000 7f9a2d495640 1 -- 172.21.15.189:0/706423159 >> v1:172.21.15.189:6789/0 conn(0x7f9a2809e740 legacy=0x7f9a281d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.970+0000 7f9a2d495640 1 -- 172.21.15.189:0/706423159 shutdown_connections 2024-09-16T14:47:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.970+0000 7f9a2d495640 1 -- 172.21.15.189:0/706423159 >> 172.21.15.189:0/706423159 conn(0x7f9a280a3320 msgr2=0x7f9a2806e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.970+0000 7f9a2d495640 1 -- 172.21.15.189:0/706423159 shutdown_connections 2024-09-16T14:47:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:24.970+0000 7f9a2d495640 1 -- 172.21.15.189:0/706423159 wait complete. 2024-09-16T14:47:24.988 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd rm osd.14 2024-09-16T14:47:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 10 monclient: get_monmap_and_config 2024-09-16T14:47:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 10 monclient: build_initial_monmap 2024-09-16T14:47:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 10 monclient: monmap: 2024-09-16T14:47:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:25.106205+0000 2024-09-16T14:47:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:25.106205+0000 2024-09-16T14:47:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:25.107 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:25.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 1 Processor -- start 2024-09-16T14:47:25.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 1 -- start start 2024-09-16T14:47:25.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 10 monclient: init 2024-09-16T14:47:25.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:25.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 10 monclient(hunting): picked mon.noname-c con 0x7f16bc1315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 10 monclient(hunting): picked mon.noname-b con 0x7f16bc09b320 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 10 monclient(hunting): picked mon.noname-a con 0x7f16bc132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc09edd0 con 0x7f16bc132090 2024-09-16T14:47:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc09bc00 con 0x7f16bc1315a0 2024-09-16T14:47:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc09e900 con 0x7f16bc09b320 2024-09-16T14:47:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 10 monclient(hunting): authenticate will time out at 1227.221931s 2024-09-16T14:47:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c2af3640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f16bc132090 legacy=0x7f16bc13a200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c22f2640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f16bc1315a0 0x7f16bc131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46470/0 (socket says 172.21.15.189:46470) 2024-09-16T14:47:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c22f2640 1 -- 172.21.15.189:0/1998933821 learned_addr learned my addr 172.21.15.189:0/1998933821 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 1 -- 172.21.15.189:0/1998933821 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853734782 0 0) 0x7f16bc09bc00 con 0x7f16bc1315a0 2024-09-16T14:47:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 10 monclient(hunting): _negotiate 2024-09-16T14:47:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 10 monclient(hunting): my global_id is 94927 2024-09-16T14:47:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 1 -- 172.21.15.189:0/1998933821 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a8003530 con 0x7f16bc1315a0 2024-09-16T14:47:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 1 -- 172.21.15.189:0/1998933821 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 934107216 0 0) 0x7f16a8003530 con 0x7f16bc1315a0 2024-09-16T14:47:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 1 -- 172.21.15.189:0/1998933821 >> v1:172.21.15.26:6790/0 conn(0x7f16bc09b320 legacy=0x7f16bc136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 1 -- 172.21.15.189:0/1998933821 >> v1:172.21.15.26:6789/0 conn(0x7f16bc132090 legacy=0x7f16bc13a200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 10 monclient: _finish_hunting 0 2024-09-16T14:47:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 1 monclient: found mon.noname-c 2024-09-16T14:47:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 1 -- 172.21.15.189:0/1998933821 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc09b920 con 0x7f16bc1315a0 2024-09-16T14:47:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 10 monclient: _finish_auth 0 2024-09-16T14:47:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 10 monclient: _check_auth_tickets 2024-09-16T14:47:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 5 monclient: authenticate success, global_id 94927 2024-09-16T14:47:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 1 -- 172.21.15.189:0/1998933821 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f16b8001620 con 0x7f16bc1315a0 2024-09-16T14:47:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 10 monclient: dump: 2024-09-16T14:47:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:25.117 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 1 -- 172.21.15.189:0/1998933821 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16b8001940 con 0x7f16bc1315a0 2024-09-16T14:47:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 1 -- 172.21.15.189:0/1998933821 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f16b8001cd0 con 0x7f16bc1315a0 2024-09-16T14:47:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c12f0640 10 monclient: dump: 2024-09-16T14:47:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:25.118 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 4 set_mon_vals no callback set 2024-09-16T14:47:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 10 monclient: shutdown 2024-09-16T14:47:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 1 -- 172.21.15.189:0/1998933821 >> v1:172.21.15.189:6789/0 conn(0x7f16bc1315a0 legacy=0x7f16bc131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 1 -- 172.21.15.189:0/1998933821 shutdown_connections 2024-09-16T14:47:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 1 -- 172.21.15.189:0/1998933821 >> 172.21.15.189:0/1998933821 conn(0x7f16bc0a3320 msgr2=0x7f16bc09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 1 -- 172.21.15.189:0/1998933821 shutdown_connections 2024-09-16T14:47:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.102+0000 7f16c4556640 1 -- 172.21.15.189:0/1998933821 wait complete. 2024-09-16T14:47:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 monclient: build_initial_monmap 2024-09-16T14:47:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 monclient: monmap: 2024-09-16T14:47:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:25.120 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 1 Processor -- start 2024-09-16T14:47:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 1 -- start start 2024-09-16T14:47:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 monclient: init 2024-09-16T14:47:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 monclient(hunting): picked mon.noname-a con 0x7f16bc1315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 monclient(hunting): picked mon.noname-b con 0x7f16bc132090 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 monclient(hunting): picked mon.noname-c con 0x7f16bc12fe70 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc09cd20 con 0x7f16bc1315a0 2024-09-16T14:47:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc0a2120 con 0x7f16bc132090 2024-09-16T14:47:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc06b380 con 0x7f16bc12fe70 2024-09-16T14:47:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 monclient(hunting): authenticate will time out at 1227.225216s 2024-09-16T14:47:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c22f2640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f16bc1315a0 legacy=0x7f16bc12f760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c1af1640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f16bc132090 0x7f16bc1caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46476/0 (socket says 172.21.15.189:46476) 2024-09-16T14:47:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c1af1640 1 -- 172.21.15.189:0/756007085 learned_addr learned my addr 172.21.15.189:0/756007085 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807987975 0 0) 0x7f16bc0a2120 con 0x7f16bc132090 2024-09-16T14:47:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient(hunting): my global_id is 94930 2024-09-16T14:47:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a0003530 con 0x7f16bc132090 2024-09-16T14:47:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2005738910 0 0) 0x7f16a0003530 con 0x7f16bc132090 2024-09-16T14:47:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16bc0a2120 con 0x7f16bc132090 2024-09-16T14:47:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f16ac001940 con 0x7f16bc132090 2024-09-16T14:47:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient(hunting): dump: 2024-09-16T14:47:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:25.123 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720378814 0 0) 0x7f16bc06b380 con 0x7f16bc12fe70 2024-09-16T14:47:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient(hunting): my global_id is 114305 2024-09-16T14:47:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a0003530 con 0x7f16bc12fe70 2024-09-16T14:47:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 19198425 0 0) 0x7f16bc0a2120 con 0x7f16bc132090 2024-09-16T14:47:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 >> v1:172.21.15.26:6790/0 conn(0x7f16bc12fe70 legacy=0x7f16bc1ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 >> v1:172.21.15.26:6789/0 conn(0x7f16bc1315a0 legacy=0x7f16bc12f760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 1 monclient: found mon.b 2024-09-16T14:47:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc0a1c10 con 0x7f16bc132090 2024-09-16T14:47:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:47:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 5 monclient: authenticate success, global_id 94930 2024-09-16T14:47:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 monclient: _renew_subs 2024-09-16T14:47:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 1 -- 172.21.15.189:0/756007085 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16bc11ade0 con 0x7f16bc132090 2024-09-16T14:47:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 monclient: _renew_subs 2024-09-16T14:47:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 1 -- 172.21.15.189:0/756007085 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16bc11d120 con 0x7f16bc132090 2024-09-16T14:47:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16ac001c80 con 0x7f16bc132090 2024-09-16T14:47:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f16ac003af0 con 0x7f16bc132090 2024-09-16T14:47:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c0aef640 4 set_mon_vals no callback set 2024-09-16T14:47:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 10 monclient: dump: 2024-09-16T14:47:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:25.127 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f16ac003c50 con 0x7f16bc132090 2024-09-16T14:47:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c4556640 1 -- 172.21.15.189:0/756007085 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a4005050 con 0x7f16bc132090 2024-09-16T14:47:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(117..117 src has 73..117) v4 ==== 5687+0+0 (unknown 1428450909 0 0) 0x7f16ac043a20 con 0x7f16bc132090 2024-09-16T14:47:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.106+0000 7f16c22f2640 10 monclient: get_auth_request con 0x7f16a00383e0 auth_method 0 2024-09-16T14:47:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.110+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f16ac011140 con 0x7f16bc132090 2024-09-16T14:47:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.110+0000 7f16b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.110+0000 7f16b2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:25.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.278+0000 7f16c4556640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T14:47:25.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.278+0000 7f16c4556640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T14:47:25.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.278+0000 7f16c4556640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:25.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.278+0000 7f16c4556640 1 -- 172.21.15.189:0/756007085 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.14"]} v 0) v1 -- 0x7f16a4005880 con 0x7f16bc132090 2024-09-16T14:47:25.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:25.279+0000 7fc29af71640 -1 mon.c@2(leader).osd e117 definitely_dead 0 2024-09-16T14:47:25.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.858+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.14"]}]=0 removed osd.14 v118) v1 ==== 87+0+0 (unknown 1613309980 0 0) 0x7f16ac012bf0 con 0x7f16bc132090 2024-09-16T14:47:25.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.858+0000 7f16b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T14:47:25.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.858+0000 7f16b2ffd640 10 monclient: _finish_command 2 = system:0 removed osd.14 2024-09-16T14:47:25.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.966+0000 7f16c4556640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T14:47:25.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.966+0000 7f16c4556640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T14:47:25.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.966+0000 7f16c4556640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:25.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.966+0000 7f16c4556640 1 -- 172.21.15.189:0/756007085 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.14"]} v 0) v1 -- 0x7f16a4005370 con 0x7f16bc132090 2024-09-16T14:47:25.973 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:25.971+0000 7fc29af71640 -1 mon.c@2(leader).osd e118 definitely_dead 0 2024-09-16T14:47:25.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.970+0000 7f16b2ffd640 1 -- 172.21.15.189:0/756007085 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.14"]}]=0 osd.14 does not exist. v118) v1 ==== 96+0+0 (unknown 2118422257 0 0) 0x7f16ac014370 con 0x7f16bc132090 2024-09-16T14:47:25.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.970+0000 7f16b2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T14:47:25.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.970+0000 7f16b2ffd640 10 monclient: _finish_command 3 = system:0 osd.14 does not exist. 2024-09-16T14:47:25.974 INFO:tasks.workunit.client.0.smithi189.stderr:osd.14 does not exist. 2024-09-16T14:47:25.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.970+0000 7f16c4556640 1 -- 172.21.15.189:0/756007085 >> v1:172.21.15.189:6812/3196161582 conn(0x7f16a00383e0 legacy=0x7f16a003a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:25.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.970+0000 7f16c4556640 10 monclient: shutdown 2024-09-16T14:47:25.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.970+0000 7f16c4556640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:25.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.970+0000 7f16c4556640 1 -- 172.21.15.189:0/756007085 >> v1:172.21.15.189:6789/0 conn(0x7f16bc132090 legacy=0x7f16bc1caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:25.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.970+0000 7f16c4556640 1 -- 172.21.15.189:0/756007085 shutdown_connections 2024-09-16T14:47:25.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.970+0000 7f16c4556640 1 -- 172.21.15.189:0/756007085 >> 172.21.15.189:0/756007085 conn(0x7f16bc0a3320 msgr2=0x7f16bc130770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:25.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.970+0000 7f16c4556640 1 -- 172.21.15.189:0/756007085 shutdown_connections 2024-09-16T14:47:25.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:25.970+0000 7f16c4556640 1 -- 172.21.15.189:0/756007085 wait complete. 2024-09-16T14:47:25.992 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd set-require-min-compat-client luminous 2024-09-16T14:47:26.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6bdd1640 10 monclient: get_monmap_and_config 2024-09-16T14:47:26.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6bdd1640 10 monclient: build_initial_monmap 2024-09-16T14:47:26.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6bdd1640 10 monclient: monmap: 2024-09-16T14:47:26.107 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:26.107 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:26.107 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:26.107288+0000 2024-09-16T14:47:26.107 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:26.107288+0000 2024-09-16T14:47:26.107 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:26.107 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:26.107 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:26.108 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:26.108 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:26.108 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:26.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6bdd1640 1 Processor -- start 2024-09-16T14:47:26.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6bdd1640 1 -- start start 2024-09-16T14:47:26.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6bdd1640 10 monclient: init 2024-09-16T14:47:26.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6bdd1640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:26.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6bdd1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:26.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6bdd1640 10 monclient(hunting): picked mon.noname-a con 0x7fac64096600 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:26.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6bdd1640 10 monclient(hunting): picked mon.noname-c con 0x7fac6409bf10 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:26.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6bdd1640 10 monclient(hunting): picked mon.noname-b con 0x7fac64138ae0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:26.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6bdd1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac64099c50 con 0x7fac64096600 2024-09-16T14:47:26.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6bdd1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac64099f30 con 0x7fac6409bf10 2024-09-16T14:47:26.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6bdd1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6409a210 con 0x7fac64138ae0 2024-09-16T14:47:26.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6bdd1640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:26.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6bdd1640 10 monclient(hunting): authenticate will time out at 1228.223961s 2024-09-16T14:47:26.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.102+0000 7fac6936c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fac6409bf10 0x7fac6409f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46484/0 (socket says 172.21.15.189:46484) 2024-09-16T14:47:26.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6936c640 1 -- 172.21.15.189:0/2561798555 learned_addr learned my addr 172.21.15.189:0/2561798555 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:26.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac69b6d640 1 -- 172.21.15.189:0/2561798555 >> v1:172.21.15.26:6789/0 conn(0x7fac64096600 legacy=0x7fac640969e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:26.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 1 -- 172.21.15.189:0/2561798555 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957787110 0 0) 0x7fac64099f30 con 0x7fac6409bf10 2024-09-16T14:47:26.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 10 monclient(hunting): _negotiate 2024-09-16T14:47:26.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:26.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:26.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 10 monclient(hunting): my global_id is 94933 2024-09-16T14:47:26.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 1 -- 172.21.15.189:0/2561798555 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac50003530 con 0x7fac6409bf10 2024-09-16T14:47:26.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 1 -- 172.21.15.189:0/2561798555 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 302911187 0 0) 0x7fac50003530 con 0x7fac6409bf10 2024-09-16T14:47:26.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 1 -- 172.21.15.189:0/2561798555 >> v1:172.21.15.26:6790/0 conn(0x7fac64138ae0 legacy=0x7fac6413aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:26.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 1 -- 172.21.15.189:0/2561798555 >> v1:172.21.15.26:6789/0 conn(0x7fac64096600 legacy=0x7fac640969e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:26.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 10 monclient: _finish_hunting 0 2024-09-16T14:47:26.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 1 monclient: found mon.noname-c 2024-09-16T14:47:26.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:26.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:26.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 1 -- 172.21.15.189:0/2561798555 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6413f620 con 0x7fac6409bf10 2024-09-16T14:47:26.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 10 monclient: _finish_auth 0 2024-09-16T14:47:26.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 10 monclient: _check_auth_tickets 2024-09-16T14:47:26.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:26.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 1 -- 172.21.15.189:0/2561798555 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fac54001620 con 0x7fac6409bf10 2024-09-16T14:47:26.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 5 monclient: authenticate success, global_id 94933 2024-09-16T14:47:26.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:26.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:26.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:26.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 10 monclient: dump: 2024-09-16T14:47:26.118 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:26.118 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:26.118 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:26.118 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:26.118 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:26.118 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:26.118 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:26.119 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:26.119 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:26.119 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:26.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:26.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 1 -- 172.21.15.189:0/2561798555 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac54001940 con 0x7fac6409bf10 2024-09-16T14:47:26.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:26.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 1 -- 172.21.15.189:0/2561798555 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fac54001d10 con 0x7fac6409bf10 2024-09-16T14:47:26.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:26.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:26.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac68b6b640 10 monclient: dump: 2024-09-16T14:47:26.119 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:26.119 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:26.119 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:26.120 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:26.120 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:26.120 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:26.120 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:26.120 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:26.120 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:26.120 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:26.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:26.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 4 set_mon_vals no callback set 2024-09-16T14:47:26.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 10 monclient: shutdown 2024-09-16T14:47:26.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:26.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 1 -- 172.21.15.189:0/2561798555 >> v1:172.21.15.189:6789/0 conn(0x7fac6409bf10 legacy=0x7fac6409f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:26.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 1 -- 172.21.15.189:0/2561798555 shutdown_connections 2024-09-16T14:47:26.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 1 -- 172.21.15.189:0/2561798555 >> 172.21.15.189:0/2561798555 conn(0x7fac640a0540 msgr2=0x7fac6409de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:26.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 1 -- 172.21.15.189:0/2561798555 shutdown_connections 2024-09-16T14:47:26.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 1 -- 172.21.15.189:0/2561798555 wait complete. 2024-09-16T14:47:26.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 10 monclient: build_initial_monmap 2024-09-16T14:47:26.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 10 monclient: monmap: 2024-09-16T14:47:26.121 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:26.121 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:26.121 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:26.121 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:26.122 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:26.122 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:26.122 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:26.122 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:26.122 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:26.122 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:26.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 1 Processor -- start 2024-09-16T14:47:26.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 1 -- start start 2024-09-16T14:47:26.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 10 monclient: init 2024-09-16T14:47:26.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:26.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:26.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 10 monclient(hunting): picked mon.noname-a con 0x7fac64096600 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:26.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 10 monclient(hunting): picked mon.noname-c con 0x7fac6409bf10 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:26.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 10 monclient(hunting): picked mon.noname-b con 0x7fac64138ae0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:26.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6413f080 con 0x7fac64096600 2024-09-16T14:47:26.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6413ed30 con 0x7fac64138ae0 2024-09-16T14:47:26.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6409a630 con 0x7fac6409bf10 2024-09-16T14:47:26.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:26.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 10 monclient(hunting): authenticate will time out at 1228.226861s 2024-09-16T14:47:26.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac69b6d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fac64096600 legacy=0x7fac641c7750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:26.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6a36e640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fac64138ae0 0x7fac641cfe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46492/0 (socket says 172.21.15.189:46492) 2024-09-16T14:47:26.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6a36e640 1 -- 172.21.15.189:0/34174244 learned_addr learned my addr 172.21.15.189:0/34174244 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:26.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519016228 0 0) 0x7fac6413ed30 con 0x7fac64138ae0 2024-09-16T14:47:26.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:26.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:26.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:26.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient(hunting): my global_id is 94936 2024-09-16T14:47:26.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac48003530 con 0x7fac64138ae0 2024-09-16T14:47:26.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 895125104 0 0) 0x7fac48003530 con 0x7fac64138ae0 2024-09-16T14:47:26.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac6413ed30 con 0x7fac64138ae0 2024-09-16T14:47:26.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fac5c0018c0 con 0x7fac64138ae0 2024-09-16T14:47:26.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:26.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:26.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient(hunting): dump: 2024-09-16T14:47:26.124 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:26.124 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:26.124 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:26.125 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:26.125 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:26.125 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:26.125 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:26.125 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:26.125 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:26.125 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:26.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:26.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346263179 0 0) 0x7fac6409a630 con 0x7fac6409bf10 2024-09-16T14:47:26.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:26.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:26.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:26.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient(hunting): my global_id is 114311 2024-09-16T14:47:26.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac48003530 con 0x7fac6409bf10 2024-09-16T14:47:26.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1030005112 0 0) 0x7fac6413ed30 con 0x7fac64138ae0 2024-09-16T14:47:26.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 >> v1:172.21.15.26:6790/0 conn(0x7fac6409bf10 legacy=0x7fac641cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:26.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 >> v1:172.21.15.26:6789/0 conn(0x7fac64096600 legacy=0x7fac641c7750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:26.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:26.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 1 monclient: found mon.b 2024-09-16T14:47:26.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:26.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:26.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6408a800 con 0x7fac64138ae0 2024-09-16T14:47:26.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:26.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:26.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:26.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 5 monclient: authenticate success, global_id 94936 2024-09-16T14:47:26.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 10 monclient: _renew_subs 2024-09-16T14:47:26.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:26.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 1 -- 172.21.15.189:0/34174244 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac64070d70 con 0x7fac64138ae0 2024-09-16T14:47:26.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 10 monclient: _renew_subs 2024-09-16T14:47:26.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:26.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac6bdd1640 1 -- 172.21.15.189:0/34174244 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac64070210 con 0x7fac64138ae0 2024-09-16T14:47:26.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac5c001c90 con 0x7fac64138ae0 2024-09-16T14:47:26.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:26.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fac5c003af0 con 0x7fac64138ae0 2024-09-16T14:47:26.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:26.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:26.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5bfff640 4 set_mon_vals no callback set 2024-09-16T14:47:26.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.106+0000 7fac5a7fc640 10 monclient: dump: 2024-09-16T14:47:26.128 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:26.128 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:26.128 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:26.128 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:26.128 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:26.128 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:26.128 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:26.128 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:26.128 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:26.128 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:26.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.110+0000 7fac6bdd1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:26.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.110+0000 7fac6bdd1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:26.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.110+0000 7fac6bdd1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:26.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.110+0000 7fac6bdd1640 1 -- 172.21.15.189:0/34174244 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac4c005050 con 0x7fac64138ae0 2024-09-16T14:47:26.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.110+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fac5c003c50 con 0x7fac64138ae0 2024-09-16T14:47:26.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.110+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(118..118 src has 73..118) v4 ==== 5687+0+0 (unknown 3637030096 0 0) 0x7fac5c043900 con 0x7fac64138ae0 2024-09-16T14:47:26.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.110+0000 7fac69b6d640 10 monclient: get_auth_request con 0x7fac480361d0 auth_method 0 2024-09-16T14:47:26.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.114+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fac5c0147a0 con 0x7fac64138ae0 2024-09-16T14:47:26.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.114+0000 7fac5a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:26.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.114+0000 7fac5a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:26.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.270+0000 7fac6bdd1640 10 start_mon_command cmd=[{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T14:47:26.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.270+0000 7fac6bdd1640 10 monclient: _send_command 2 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T14:47:26.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.270+0000 7fac6bdd1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:26.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:26.270+0000 7fac6bdd1640 1 -- 172.21.15.189:0/34174244 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "luminous"} v 0) v1 -- 0x7fac4c005880 con 0x7fac64138ae0 2024-09-16T14:47:27.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:27.178+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "luminous"}]=0 set require_min_compat_client to luminous v120) v1 ==== 145+0+0 (unknown 728945408 0 0) 0x7fac5c012c10 con 0x7fac64138ae0 2024-09-16T14:47:27.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:27.178+0000 7fac5a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T14:47:27.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:27.178+0000 7fac5a7fc640 10 monclient: _finish_command 2 = system:0 set require_min_compat_client to luminous 2024-09-16T14:47:27.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:27.282+0000 7fac6bdd1640 10 start_mon_command cmd=[{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T14:47:27.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:27.282+0000 7fac6bdd1640 10 monclient: _send_command 3 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T14:47:27.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:27.282+0000 7fac6bdd1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:27.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:27.282+0000 7fac6bdd1640 1 -- 172.21.15.189:0/34174244 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "luminous"} v 0) v1 -- 0x7fac4c005300 con 0x7fac64138ae0 2024-09-16T14:47:28.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.178+0000 7fac5a7fc640 1 -- 172.21.15.189:0/34174244 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "luminous"}]=0 set require_min_compat_client to luminous v121) v1 ==== 145+0+0 (unknown 3769501427 0 0) 0x7fac5c014570 con 0x7fac64138ae0 2024-09-16T14:47:28.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.178+0000 7fac5a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T14:47:28.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.178+0000 7fac5a7fc640 10 monclient: _finish_command 3 = system:0 set require_min_compat_client to luminous 2024-09-16T14:47:28.186 INFO:tasks.workunit.client.0.smithi189.stderr:set require_min_compat_client to luminous 2024-09-16T14:47:28.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.186+0000 7fac6bdd1640 1 -- 172.21.15.189:0/34174244 >> v1:172.21.15.189:6812/3196161582 conn(0x7fac480361d0 legacy=0x7fac48038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:28.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.186+0000 7fac6bdd1640 10 monclient: shutdown 2024-09-16T14:47:28.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.186+0000 7fac6bdd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:28.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.186+0000 7fac6bdd1640 1 -- 172.21.15.189:0/34174244 >> v1:172.21.15.189:6789/0 conn(0x7fac64138ae0 legacy=0x7fac641cfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:28.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.186+0000 7fac6bdd1640 1 -- 172.21.15.189:0/34174244 shutdown_connections 2024-09-16T14:47:28.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.186+0000 7fac6bdd1640 1 -- 172.21.15.189:0/34174244 >> 172.21.15.189:0/34174244 conn(0x7fac640a0540 msgr2=0x7fac6409a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:28.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.186+0000 7fac6bdd1640 1 -- 172.21.15.189:0/34174244 shutdown_connections 2024-09-16T14:47:28.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.186+0000 7fac6bdd1640 1 -- 172.21.15.189:0/34174244 wait complete. 2024-09-16T14:47:28.210 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set dump 2024-09-16T14:47:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.322+0000 7f1503604640 10 monclient: get_monmap_and_config 2024-09-16T14:47:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.322+0000 7f1503604640 10 monclient: build_initial_monmap 2024-09-16T14:47:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.322+0000 7f1503604640 10 monclient: monmap: 2024-09-16T14:47:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:28.328742+0000 2024-09-16T14:47:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:28.328742+0000 2024-09-16T14:47:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:28.329 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 1 Processor -- start 2024-09-16T14:47:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 1 -- start start 2024-09-16T14:47:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 10 monclient: init 2024-09-16T14:47:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 10 monclient(hunting): picked mon.noname-c con 0x7f14fc09f740 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 10 monclient(hunting): picked mon.noname-a con 0x7f14fc0a0230 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 10 monclient(hunting): picked mon.noname-b con 0x7f14fc099d60 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14fc099660 con 0x7f14fc0a0230 2024-09-16T14:47:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14fc099380 con 0x7f14fc09f740 2024-09-16T14:47:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14fc0990a0 con 0x7f14fc099d60 2024-09-16T14:47:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 10 monclient(hunting): authenticate will time out at 1230.445196s 2024-09-16T14:47:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f15013a0640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f14fc09f740 0x7f14fc09fb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46500/0 (socket says 172.21.15.189:46500) 2024-09-16T14:47:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f15013a0640 1 -- 172.21.15.189:0/2240917364 learned_addr learned my addr 172.21.15.189:0/2240917364 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1500b9f640 1 -- 172.21.15.189:0/2240917364 >> v1:172.21.15.26:6789/0 conn(0x7f14fc0a0230 legacy=0x7f14fc139d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:28.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 1 -- 172.21.15.189:0/2240917364 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776094972 0 0) 0x7f14fc099380 con 0x7f14fc09f740 2024-09-16T14:47:28.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:28.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:28.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 10 monclient(hunting): my global_id is 94939 2024-09-16T14:47:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 1 -- 172.21.15.189:0/2240917364 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e8003530 con 0x7f14fc09f740 2024-09-16T14:47:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 1 -- 172.21.15.189:0/2240917364 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2758408243 0 0) 0x7f14e8003530 con 0x7f14fc09f740 2024-09-16T14:47:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 1 -- 172.21.15.189:0/2240917364 >> v1:172.21.15.26:6790/0 conn(0x7f14fc099d60 legacy=0x7f14fc13d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 1 -- 172.21.15.189:0/2240917364 >> v1:172.21.15.26:6789/0 conn(0x7f14fc0a0230 legacy=0x7f14fc139d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:47:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 1 monclient: found mon.noname-c 2024-09-16T14:47:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 1 -- 172.21.15.189:0/2240917364 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14fc0a11a0 con 0x7f14fc09f740 2024-09-16T14:47:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 10 monclient: _finish_auth 0 2024-09-16T14:47:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:47:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 1 -- 172.21.15.189:0/2240917364 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f14f8001620 con 0x7f14fc09f740 2024-09-16T14:47:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 10 monclient: dump: 2024-09-16T14:47:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:28.339 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 5 monclient: authenticate success, global_id 94939 2024-09-16T14:47:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 1 -- 172.21.15.189:0/2240917364 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14f8001940 con 0x7f14fc09f740 2024-09-16T14:47:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 1 -- 172.21.15.189:0/2240917364 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f14f8001d10 con 0x7f14fc09f740 2024-09-16T14:47:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 4 set_mon_vals no callback set 2024-09-16T14:47:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f14f3fff640 10 monclient: dump: 2024-09-16T14:47:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:28.340 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 10 monclient: shutdown 2024-09-16T14:47:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 1 -- 172.21.15.189:0/2240917364 >> v1:172.21.15.189:6789/0 conn(0x7f14fc09f740 legacy=0x7f14fc09fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 1 -- 172.21.15.189:0/2240917364 shutdown_connections 2024-09-16T14:47:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 1 -- 172.21.15.189:0/2240917364 >> 172.21.15.189:0/2240917364 conn(0x7f14fc0a4390 msgr2=0x7f14fc0a0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 1 -- 172.21.15.189:0/2240917364 shutdown_connections 2024-09-16T14:47:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 1 -- 172.21.15.189:0/2240917364 wait complete. 2024-09-16T14:47:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 10 monclient: build_initial_monmap 2024-09-16T14:47:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.326+0000 7f1503604640 10 monclient: monmap: 2024-09-16T14:47:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:28.342 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 1 Processor -- start 2024-09-16T14:47:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 1 -- start start 2024-09-16T14:47:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 10 monclient: init 2024-09-16T14:47:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 10 monclient(hunting): picked mon.noname-a con 0x7f14fc099d60 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 10 monclient(hunting): picked mon.noname-b con 0x7f14fc09f740 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 10 monclient(hunting): picked mon.noname-c con 0x7f14fc0a0230 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14fc09b7a0 con 0x7f14fc099d60 2024-09-16T14:47:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14fc11fe80 con 0x7f14fc09f740 2024-09-16T14:47:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f15013a0640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f14fc099d60 legacy=0x7f14fc130a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14fc09b100 con 0x7f14fc0a0230 2024-09-16T14:47:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 10 monclient(hunting): authenticate will time out at 1230.448959s 2024-09-16T14:47:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1500b9f640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f14fc09f740 0x7f14fc1ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46514/0 (socket says 172.21.15.189:46514) 2024-09-16T14:47:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1500b9f640 1 -- 172.21.15.189:0/194641496 learned_addr learned my addr 172.21.15.189:0/194641496 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309245213 0 0) 0x7f14fc11fe80 con 0x7f14fc09f740 2024-09-16T14:47:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient(hunting): my global_id is 94942 2024-09-16T14:47:28.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e0003560 con 0x7f14fc09f740 2024-09-16T14:47:28.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2905092094 0 0) 0x7f14e0003560 con 0x7f14fc09f740 2024-09-16T14:47:28.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14fc11fe80 con 0x7f14fc09f740 2024-09-16T14:47:28.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f14ec004190 con 0x7f14fc09f740 2024-09-16T14:47:28.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:28.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:28.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient(hunting): dump: 2024-09-16T14:47:28.344 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:28.344 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:28.344 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:28.344 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:28.344 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:28.345 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:28.345 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:28.345 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:28.345 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:28.345 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:28.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:28.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332321088 0 0) 0x7f14fc09b100 con 0x7f14fc0a0230 2024-09-16T14:47:28.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:28.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:28.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:28.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient(hunting): my global_id is 114317 2024-09-16T14:47:28.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e0003560 con 0x7f14fc0a0230 2024-09-16T14:47:28.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4078512711 0 0) 0x7f14fc11fe80 con 0x7f14fc09f740 2024-09-16T14:47:28.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 >> v1:172.21.15.26:6790/0 conn(0x7f14fc0a0230 legacy=0x7f14fc1d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:28.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 >> v1:172.21.15.26:6789/0 conn(0x7f14fc099d60 legacy=0x7f14fc130a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:28.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:47:28.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 1 monclient: found mon.b 2024-09-16T14:47:28.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:28.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:28.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14fc121480 con 0x7f14fc09f740 2024-09-16T14:47:28.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:47:28.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:28.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:28.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 5 monclient: authenticate success, global_id 94942 2024-09-16T14:47:28.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 10 monclient: _renew_subs 2024-09-16T14:47:28.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:28.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 1 -- 172.21.15.189:0/194641496 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14fc120be0 con 0x7f14fc09f740 2024-09-16T14:47:28.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 10 monclient: _renew_subs 2024-09-16T14:47:28.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:28.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 1 -- 172.21.15.189:0/194641496 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14fc1311a0 con 0x7f14fc09f740 2024-09-16T14:47:28.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14ec0044b0 con 0x7f14fc09f740 2024-09-16T14:47:28.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:28.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f14ec004840 con 0x7f14fc09f740 2024-09-16T14:47:28.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:28.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:28.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f37fe640 4 set_mon_vals no callback set 2024-09-16T14:47:28.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 10 monclient: dump: 2024-09-16T14:47:28.347 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:28.347 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:28.347 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:28.348 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f1503604640 1 -- 172.21.15.189:0/194641496 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14dc005050 con 0x7f14fc09f740 2024-09-16T14:47:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f14ec004a20 con 0x7f14fc09f740 2024-09-16T14:47:28.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(121..121 src has 73..121) v4 ==== 5687+0+0 (unknown 1766820235 0 0) 0x7f14ec043b40 con 0x7f14fc09f740 2024-09-16T14:47:28.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.330+0000 7f15013a0640 10 monclient: get_auth_request con 0x7f14e0036130 auth_method 0 2024-09-16T14:47:28.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.338+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f14ec0148a0 con 0x7f14fc09f740 2024-09-16T14:47:28.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.338+0000 7f14f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:28.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.338+0000 7f14f1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:28.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.510+0000 7f1503604640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-16T14:47:28.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.510+0000 7f1503604640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T14:47:28.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.510+0000 7f1503604640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:28.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.510+0000 7f1503604640 1 -- 172.21.15.189:0/194641496 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7f14dc005880 con 0x7f14fc09f740 2024-09-16T14:47:28.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.510+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v121) v1 ==== 73+0+3 (unknown 3238285775 0 2423995997) 0x7f14ec012d10 con 0x7f14fc09f740 2024-09-16T14:47:28.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.510+0000 7f14f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T14:47:28.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.510+0000 7f14f1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:28.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.614+0000 7f1503604640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-16T14:47:28.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.614+0000 7f1503604640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T14:47:28.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.614+0000 7f1503604640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:28.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.614+0000 7f1503604640 1 -- 172.21.15.189:0/194641496 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7f14dc005370 con 0x7f14fc09f740 2024-09-16T14:47:28.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.614+0000 7f14f1ffb640 1 -- 172.21.15.189:0/194641496 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v121) v1 ==== 73+0+3 (unknown 3238285775 0 2423995997) 0x7f14ec014670 con 0x7f14fc09f740 2024-09-16T14:47:28.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.614+0000 7f14f1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T14:47:28.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.614+0000 7f14f1ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:28.620 INFO:tasks.workunit.client.0.smithi189.stdout:{} 2024-09-16T14:47:28.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.618+0000 7f1503604640 1 -- 172.21.15.189:0/194641496 >> v1:172.21.15.189:6812/3196161582 conn(0x7f14e0036130 legacy=0x7f14e00385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:28.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.618+0000 7f1503604640 10 monclient: shutdown 2024-09-16T14:47:28.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.618+0000 7f1503604640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:28.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.618+0000 7f1503604640 1 -- 172.21.15.189:0/194641496 >> v1:172.21.15.189:6789/0 conn(0x7f14fc09f740 legacy=0x7f14fc1ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:28.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.618+0000 7f1503604640 1 -- 172.21.15.189:0/194641496 shutdown_connections 2024-09-16T14:47:28.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.618+0000 7f1503604640 1 -- 172.21.15.189:0/194641496 >> 172.21.15.189:0/194641496 conn(0x7f14fc0a4390 msgr2=0x7f14fc0a3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:28.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.618+0000 7f1503604640 1 -- 172.21.15.189:0/194641496 shutdown_connections 2024-09-16T14:47:28.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.618+0000 7f1503604640 1 -- 172.21.15.189:0/194641496 wait complete. 2024-09-16T14:47:28.637 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set ls 2024-09-16T14:47:28.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.746+0000 7f8af1f26640 10 monclient: get_monmap_and_config 2024-09-16T14:47:28.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.746+0000 7f8af1f26640 10 monclient: build_initial_monmap 2024-09-16T14:47:28.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.746+0000 7f8af1f26640 10 monclient: monmap: 2024-09-16T14:47:28.752 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:28.753 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:28.753 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:28.752600+0000 2024-09-16T14:47:28.753 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:28.752600+0000 2024-09-16T14:47:28.753 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:28.753 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:28.753 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:28.753 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:28.753 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:28.753 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:28.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.746+0000 7f8af1f26640 1 Processor -- start 2024-09-16T14:47:28.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.746+0000 7f8af1f26640 1 -- start start 2024-09-16T14:47:28.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.746+0000 7f8af1f26640 10 monclient: init 2024-09-16T14:47:28.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:28.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:28.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient(hunting): picked mon.noname-c con 0x7f8aec1315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:28.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient(hunting): picked mon.noname-b con 0x7f8aec09b320 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:28.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient(hunting): picked mon.noname-a con 0x7f8aec132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:28.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aec09edd0 con 0x7f8aec132090 2024-09-16T14:47:28.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aec09bc00 con 0x7f8aec1315a0 2024-09-16T14:47:28.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aec09e900 con 0x7f8aec09b320 2024-09-16T14:47:28.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:28.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient(hunting): authenticate will time out at 1230.868345s 2024-09-16T14:47:28.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeb7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8aec1315a0 0x7f8aec131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46528/0 (socket says 172.21.15.189:46528) 2024-09-16T14:47:28.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeb7fe640 1 -- 172.21.15.189:0/192487455 learned_addr learned my addr 172.21.15.189:0/192487455 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:28.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aebfff640 1 -- 172.21.15.189:0/192487455 >> v1:172.21.15.26:6789/0 conn(0x7f8aec132090 legacy=0x7f8aec13a200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:28.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 1 -- 172.21.15.189:0/192487455 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202123526 0 0) 0x7f8aec09bc00 con 0x7f8aec1315a0 2024-09-16T14:47:28.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:28.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:28.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:28.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient(hunting): my global_id is 94945 2024-09-16T14:47:28.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 1 -- 172.21.15.189:0/192487455 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8acc003530 con 0x7f8aec1315a0 2024-09-16T14:47:28.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 1 -- 172.21.15.189:0/192487455 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72082083 0 0) 0x7f8aec09e900 con 0x7f8aec09b320 2024-09-16T14:47:28.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:28.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:28.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:28.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient(hunting): my global_id is 114320 2024-09-16T14:47:28.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 1 -- 172.21.15.189:0/192487455 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aec09bc00 con 0x7f8aec09b320 2024-09-16T14:47:28.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 1 -- 172.21.15.189:0/192487455 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4005303242 0 0) 0x7f8acc003530 con 0x7f8aec1315a0 2024-09-16T14:47:28.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 1 -- 172.21.15.189:0/192487455 >> v1:172.21.15.26:6790/0 conn(0x7f8aec09b320 legacy=0x7f8aec136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:28.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 1 -- 172.21.15.189:0/192487455 >> v1:172.21.15.26:6789/0 conn(0x7f8aec132090 legacy=0x7f8aec13a200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:28.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:28.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 1 monclient: found mon.noname-c 2024-09-16T14:47:28.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:28.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:28.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 1 -- 172.21.15.189:0/192487455 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec09b920 con 0x7f8aec1315a0 2024-09-16T14:47:28.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient: _finish_auth 0 2024-09-16T14:47:28.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:28.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:28.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 1 -- 172.21.15.189:0/192487455 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ad4001620 con 0x7f8aec1315a0 2024-09-16T14:47:28.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 5 monclient: authenticate success, global_id 94945 2024-09-16T14:47:28.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:28.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:28.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:28.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient: dump: 2024-09-16T14:47:28.763 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:28.763 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:28.763 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:28.763 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:28.764 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:28.764 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:28.764 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:28.764 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:28.764 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:28.764 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:28.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:28.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 1 -- 172.21.15.189:0/192487455 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ad4001900 con 0x7f8aec1315a0 2024-09-16T14:47:28.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:28.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:28.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 4 set_mon_vals no callback set 2024-09-16T14:47:28.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 1 -- 172.21.15.189:0/192487455 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ad4001c90 con 0x7f8aec1315a0 2024-09-16T14:47:28.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:28.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:28.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeaffd640 10 monclient: dump: 2024-09-16T14:47:28.765 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:28.765 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:28.765 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:28.765 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:28.765 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:28.765 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:28.765 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:28.765 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:28.765 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:28.765 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:28.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient: shutdown 2024-09-16T14:47:28.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:28.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 1 -- 172.21.15.189:0/192487455 >> v1:172.21.15.189:6789/0 conn(0x7f8aec1315a0 legacy=0x7f8aec131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:28.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 1 -- 172.21.15.189:0/192487455 shutdown_connections 2024-09-16T14:47:28.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 1 -- 172.21.15.189:0/192487455 >> 172.21.15.189:0/192487455 conn(0x7f8aec0a3320 msgr2=0x7f8aec09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:28.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 1 -- 172.21.15.189:0/192487455 shutdown_connections 2024-09-16T14:47:28.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 1 -- 172.21.15.189:0/192487455 wait complete. 2024-09-16T14:47:28.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient: build_initial_monmap 2024-09-16T14:47:28.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient: monmap: 2024-09-16T14:47:28.766 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:28.766 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:28.766 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:28.766 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:28.766 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:28.767 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:28.767 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:28.767 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:28.767 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:28.767 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:28.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 1 Processor -- start 2024-09-16T14:47:28.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 1 -- start start 2024-09-16T14:47:28.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient: init 2024-09-16T14:47:28.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:28.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:28.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient(hunting): picked mon.noname-b con 0x7f8aec132090 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:28.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient(hunting): picked mon.noname-c con 0x7f8aec1302f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:28.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient(hunting): picked mon.noname-a con 0x7f8aec1c8f90 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:28.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aec09cd20 con 0x7f8aec1c8f90 2024-09-16T14:47:28.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aec0a0df0 con 0x7f8aec132090 2024-09-16T14:47:28.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aec06fcb0 con 0x7f8aec1302f0 2024-09-16T14:47:28.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:28.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8af1f26640 10 monclient(hunting): authenticate will time out at 1230.871723s 2024-09-16T14:47:28.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeb7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8aec132090 0x7f8aec12fbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46540/0 (socket says 172.21.15.189:46540) 2024-09-16T14:47:28.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aeb7fe640 1 -- 172.21.15.189:0/4075330695 learned_addr learned my addr 172.21.15.189:0/4075330695 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:28.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.750+0000 7f8aebfff640 1 -- 172.21.15.189:0/4075330695 >> v1:172.21.15.26:6789/0 conn(0x7f8aec1c8f90 legacy=0x7f8aec1d19e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:28.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895854339 0 0) 0x7f8aec0a0df0 con 0x7f8aec132090 2024-09-16T14:47:28.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:47:28.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:28.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:28.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient(hunting): my global_id is 94948 2024-09-16T14:47:28.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad0003530 con 0x7f8aec132090 2024-09-16T14:47:28.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2863788282 0 0) 0x7f8ad0003530 con 0x7f8aec132090 2024-09-16T14:47:28.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8aec0a0df0 con 0x7f8aec132090 2024-09-16T14:47:28.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ad40015e0 con 0x7f8aec132090 2024-09-16T14:47:28.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:28.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:28.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient(hunting): dump: 2024-09-16T14:47:28.769 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:28.769 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:28.769 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:28.769 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:28.769 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:28.769 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:28.770 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:28.770 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:28.770 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:28.770 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:28.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:28.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241435332 0 0) 0x7f8aec06fcb0 con 0x7f8aec1302f0 2024-09-16T14:47:28.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:47:28.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:28.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:28.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient(hunting): my global_id is 114323 2024-09-16T14:47:28.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad0003530 con 0x7f8aec1302f0 2024-09-16T14:47:28.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1717383055 0 0) 0x7f8aec0a0df0 con 0x7f8aec132090 2024-09-16T14:47:28.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 >> v1:172.21.15.26:6790/0 conn(0x7f8aec1302f0 legacy=0x7f8aec1ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:28.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 >> v1:172.21.15.26:6789/0 conn(0x7f8aec1c8f90 legacy=0x7f8aec1d19e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:28.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:47:28.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 1 monclient: found mon.b 2024-09-16T14:47:28.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:28.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:28.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec06f0e0 con 0x7f8aec132090 2024-09-16T14:47:28.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient: _finish_auth 0 2024-09-16T14:47:28.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:47:28.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:28.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8af1f26640 5 monclient: authenticate success, global_id 94948 2024-09-16T14:47:28.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8af1f26640 10 monclient: _renew_subs 2024-09-16T14:47:28.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8af1f26640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:28.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8af1f26640 1 -- 172.21.15.189:0/4075330695 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aec070e80 con 0x7f8aec132090 2024-09-16T14:47:28.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8af1f26640 10 monclient: _renew_subs 2024-09-16T14:47:28.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8af1f26640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:28.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8af1f26640 1 -- 172.21.15.189:0/4075330695 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aec06fa30 con 0x7f8aec132090 2024-09-16T14:47:28.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ad4004c80 con 0x7f8aec132090 2024-09-16T14:47:28.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:28.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ad4005010 con 0x7f8aec132090 2024-09-16T14:47:28.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:28.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8aea7fc640 4 set_mon_vals no callback set 2024-09-16T14:47:28.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:28.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 10 monclient: dump: 2024-09-16T14:47:28.772 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:28.772 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:28.773 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:28.773 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:28.773 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:28.773 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:28.773 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:28.773 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:28.773 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:28.773 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:28.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8ad4005170 con 0x7f8aec132090 2024-09-16T14:47:28.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8af1f26640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:28.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8af1f26640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:28.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8af1f26640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:28.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8af1f26640 1 -- 172.21.15.189:0/4075330695 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac4005050 con 0x7f8aec132090 2024-09-16T14:47:28.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(121..121 src has 73..121) v4 ==== 5687+0+0 (unknown 1766820235 0 0) 0x7f8ad4043be0 con 0x7f8aec132090 2024-09-16T14:47:28.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.754+0000 7f8ae2ffd640 10 monclient: get_auth_request con 0x7f8ad00361d0 auth_method 0 2024-09-16T14:47:28.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.758+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8ad4014a10 con 0x7f8aec132090 2024-09-16T14:47:28.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.758+0000 7f8ae8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:28.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.758+0000 7f8ae8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:28.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.926+0000 7f8af1f26640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:28.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.926+0000 7f8af1f26640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:28.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.926+0000 7f8af1f26640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:28.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.926+0000 7f8af1f26640 1 -- 172.21.15.189:0/4075330695 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f8ac4005880 con 0x7f8aec132090 2024-09-16T14:47:28.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.926+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v121) v1 ==== 71+0+0 (unknown 2941357928 0 0) 0x7f8ad4012e80 con 0x7f8aec132090 2024-09-16T14:47:28.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.926+0000 7f8ae8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:28.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:28.926+0000 7f8ae8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.022+0000 7f8af1f26640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.022+0000 7f8af1f26640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.022+0000 7f8af1f26640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.022+0000 7f8af1f26640 1 -- 172.21.15.189:0/4075330695 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f8ac4005370 con 0x7f8aec132090 2024-09-16T14:47:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.022+0000 7f8ae8ff9640 1 -- 172.21.15.189:0/4075330695 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v121) v1 ==== 71+0+0 (unknown 2941357928 0 0) 0x7f8ad40147e0 con 0x7f8aec132090 2024-09-16T14:47:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.022+0000 7f8ae8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.022+0000 7f8ae8ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.026+0000 7f8af1f26640 1 -- 172.21.15.189:0/4075330695 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8ad00361d0 legacy=0x7f8ad0038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.026+0000 7f8af1f26640 10 monclient: shutdown 2024-09-16T14:47:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.026+0000 7f8af1f26640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.026+0000 7f8af1f26640 1 -- 172.21.15.189:0/4075330695 >> v1:172.21.15.189:6789/0 conn(0x7f8aec132090 legacy=0x7f8aec12fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.026+0000 7f8af1f26640 1 -- 172.21.15.189:0/4075330695 shutdown_connections 2024-09-16T14:47:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.026+0000 7f8af1f26640 1 -- 172.21.15.189:0/4075330695 >> 172.21.15.189:0/4075330695 conn(0x7f8aec0a3320 msgr2=0x7f8aec0a21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.026+0000 7f8af1f26640 1 -- 172.21.15.189:0/4075330695 shutdown_connections 2024-09-16T14:47:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.026+0000 7f8af1f26640 1 -- 172.21.15.189:0/4075330695 wait complete. 2024-09-16T14:47:29.044 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T14:47:29.044 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:47:29.044 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T14:47:29.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.158+0000 7fd363a9d640 10 monclient: get_monmap_and_config 2024-09-16T14:47:29.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.158+0000 7fd363a9d640 10 monclient: build_initial_monmap 2024-09-16T14:47:29.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.158+0000 7fd363a9d640 10 monclient: monmap: 2024-09-16T14:47:29.164 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:29.164 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:29.164 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:29.164136+0000 2024-09-16T14:47:29.164 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:29.164136+0000 2024-09-16T14:47:29.164 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:29.164 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:29.164 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:29.164 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:29.164 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:29.165 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:29.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.158+0000 7fd363a9d640 1 Processor -- start 2024-09-16T14:47:29.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.158+0000 7fd363a9d640 1 -- start start 2024-09-16T14:47:29.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.158+0000 7fd363a9d640 10 monclient: init 2024-09-16T14:47:29.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:29.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:29.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd35c131630 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:29.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd35c09edd0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:29.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd35c09e950 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:29.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c09b920 con 0x7fd35c09e950 2024-09-16T14:47:29.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c09bc00 con 0x7fd35c131630 2024-09-16T14:47:29.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c09cba0 con 0x7fd35c09edd0 2024-09-16T14:47:29.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:29.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient(hunting): authenticate will time out at 1231.280324s 2024-09-16T14:47:29.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd36203a640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fd35c09e950 legacy=0x7fd35c13a290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:29.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd361839640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd35c131630 0x7fd35c131a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54596/0 (socket says 172.21.15.189:54596) 2024-09-16T14:47:29.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd361839640 1 -- 172.21.15.189:0/2213771185 learned_addr learned my addr 172.21.15.189:0/2213771185 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:29.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 1 -- 172.21.15.189:0/2213771185 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516959470 0 0) 0x7fd35c09bc00 con 0x7fd35c131630 2024-09-16T14:47:29.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 10 monclient(hunting): _negotiate 2024-09-16T14:47:29.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:29.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:29.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 10 monclient(hunting): my global_id is 94951 2024-09-16T14:47:29.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 1 -- 172.21.15.189:0/2213771185 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd348003530 con 0x7fd35c131630 2024-09-16T14:47:29.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 1 -- 172.21.15.189:0/2213771185 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2696984963 0 0) 0x7fd348003530 con 0x7fd35c131630 2024-09-16T14:47:29.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 1 -- 172.21.15.189:0/2213771185 >> v1:172.21.15.26:6790/0 conn(0x7fd35c09edd0 legacy=0x7fd35c136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:29.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 1 -- 172.21.15.189:0/2213771185 >> v1:172.21.15.26:6789/0 conn(0x7fd35c09e950 legacy=0x7fd35c13a290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 10 monclient: _finish_hunting 0 2024-09-16T14:47:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 1 monclient: found mon.noname-c 2024-09-16T14:47:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 1 -- 172.21.15.189:0/2213771185 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c09b420 con 0x7fd35c131630 2024-09-16T14:47:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 10 monclient: _finish_auth 0 2024-09-16T14:47:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 10 monclient: _check_auth_tickets 2024-09-16T14:47:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 1 -- 172.21.15.189:0/2213771185 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd358001620 con 0x7fd35c131630 2024-09-16T14:47:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 5 monclient: authenticate success, global_id 94951 2024-09-16T14:47:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 10 monclient: dump: 2024-09-16T14:47:29.171 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:29.171 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:29.171 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:29.171 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:29.171 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:29.171 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:29.171 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:29.171 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:29.171 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:29.171 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:29.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:29.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 1 -- 172.21.15.189:0/2213771185 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd358001940 con 0x7fd35c131630 2024-09-16T14:47:29.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:29.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 1 -- 172.21.15.189:0/2213771185 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd358001cd0 con 0x7fd35c131630 2024-09-16T14:47:29.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:29.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 4 set_mon_vals no callback set 2024-09-16T14:47:29.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:29.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:29.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd360837640 10 monclient: dump: 2024-09-16T14:47:29.172 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:29.172 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:29.172 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:29.172 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:29.172 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:29.172 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:29.172 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:29.173 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:29.173 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:29.173 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:29.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient: shutdown 2024-09-16T14:47:29.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:29.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 1 -- 172.21.15.189:0/2213771185 >> v1:172.21.15.189:6789/0 conn(0x7fd35c131630 legacy=0x7fd35c131a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:29.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 1 -- 172.21.15.189:0/2213771185 shutdown_connections 2024-09-16T14:47:29.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 1 -- 172.21.15.189:0/2213771185 >> 172.21.15.189:0/2213771185 conn(0x7fd35c0a3380 msgr2=0x7fd35c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:29.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 1 -- 172.21.15.189:0/2213771185 shutdown_connections 2024-09-16T14:47:29.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 1 -- 172.21.15.189:0/2213771185 wait complete. 2024-09-16T14:47:29.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient: build_initial_monmap 2024-09-16T14:47:29.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient: monmap: 2024-09-16T14:47:29.173 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:29.174 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:29.174 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:29.174 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:29.174 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:29.174 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:29.174 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:29.174 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:29.174 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:29.174 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:29.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 1 Processor -- start 2024-09-16T14:47:29.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 1 -- start start 2024-09-16T14:47:29.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient: init 2024-09-16T14:47:29.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:29.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:29.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd35c09e950 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:29.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd35c09edd0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:29.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd35c131630 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:29.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c122990 con 0x7fd35c09e950 2024-09-16T14:47:29.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c122740 con 0x7fd35c131630 2024-09-16T14:47:29.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c0a0c80 con 0x7fd35c09edd0 2024-09-16T14:47:29.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:29.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd363a9d640 10 monclient(hunting): authenticate will time out at 1231.283541s 2024-09-16T14:47:29.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd361839640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fd35c09e950 legacy=0x7fd35c11ec10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:29.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd36203a640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd35c131630 0x7fd35c1d19c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54600/0 (socket says 172.21.15.189:54600) 2024-09-16T14:47:29.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd36203a640 1 -- 172.21.15.189:0/373502558 learned_addr learned my addr 172.21.15.189:0/373502558 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:29.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710705954 0 0) 0x7fd35c122740 con 0x7fd35c131630 2024-09-16T14:47:29.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd3527fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:29.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd3527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:29.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd3527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:29.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd3527fc640 10 monclient(hunting): my global_id is 94954 2024-09-16T14:47:29.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.162+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd340003530 con 0x7fd35c131630 2024-09-16T14:47:29.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2036086044 0 0) 0x7fd340003530 con 0x7fd35c131630 2024-09-16T14:47:29.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd35c122740 con 0x7fd35c131630 2024-09-16T14:47:29.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd3540043d0 con 0x7fd35c131630 2024-09-16T14:47:29.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:29.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:29.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 10 monclient(hunting): dump: 2024-09-16T14:47:29.179 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:29.179 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:29.179 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:29.180 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:29.180 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:29.180 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:29.180 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:29.180 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:29.180 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:29.180 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:29.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:29.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371936726 0 0) 0x7fd35c0a0c80 con 0x7fd35c09edd0 2024-09-16T14:47:29.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:29.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:29.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:29.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 10 monclient(hunting): my global_id is 114329 2024-09-16T14:47:29.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd340003530 con 0x7fd35c09edd0 2024-09-16T14:47:29.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 765053315 0 0) 0x7fd35c122740 con 0x7fd35c131630 2024-09-16T14:47:29.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 >> v1:172.21.15.26:6790/0 conn(0x7fd35c09edd0 legacy=0x7fd35c11f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:29.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 >> v1:172.21.15.26:6789/0 conn(0x7fd35c09e950 legacy=0x7fd35c11ec10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:29.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:29.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 1 monclient: found mon.b 2024-09-16T14:47:29.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:29.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:29.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c06b320 con 0x7fd35c131630 2024-09-16T14:47:29.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:29.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:29.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:29.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd363a9d640 5 monclient: authenticate success, global_id 94954 2024-09-16T14:47:29.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd363a9d640 10 monclient: _renew_subs 2024-09-16T14:47:29.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd363a9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:29.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd363a9d640 1 -- 172.21.15.189:0/373502558 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35c121540 con 0x7fd35c131630 2024-09-16T14:47:29.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd363a9d640 10 monclient: _renew_subs 2024-09-16T14:47:29.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd363a9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:29.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd363a9d640 1 -- 172.21.15.189:0/373502558 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35c121200 con 0x7fd35c131630 2024-09-16T14:47:29.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3540045f0 con 0x7fd35c131630 2024-09-16T14:47:29.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:29.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd354004980 con 0x7fd35c131630 2024-09-16T14:47:29.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:29.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:29.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd353fff640 4 set_mon_vals no callback set 2024-09-16T14:47:29.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 10 monclient: dump: 2024-09-16T14:47:29.182 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:29.183 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:29.183 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:29.183 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:29.183 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:29.183 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:29.183 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:29.183 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:29.183 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:29.183 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:29.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd354004ae0 con 0x7fd35c131630 2024-09-16T14:47:29.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd363a9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:29.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd363a9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:29.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd363a9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:29.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(121..121 src has 73..121) v4 ==== 5687+0+0 (unknown 1766820235 0 0) 0x7fd354043c00 con 0x7fd35c131630 2024-09-16T14:47:29.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd363a9d640 1 -- 172.21.15.189:0/373502558 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd344005050 con 0x7fd35c131630 2024-09-16T14:47:29.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.166+0000 7fd361839640 10 monclient: get_auth_request con 0x7fd340036220 auth_method 0 2024-09-16T14:47:29.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.170+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd354014960 con 0x7fd35c131630 2024-09-16T14:47:29.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.170+0000 7fd3527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:29.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.170+0000 7fd3527fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.330+0000 7fd363a9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T14:47:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.330+0000 7fd363a9d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T14:47:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.330+0000 7fd363a9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.330+0000 7fd363a9d640 1 -- 172.21.15.189:0/373502558 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7fd344005880 con 0x7fd35c131630 2024-09-16T14:47:29.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.334+0000 7fd3527fc640 1 -- 172.21.15.189:0/373502558 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=-2 pool 'fooset' not found v121) v1 ==== 152+0+0 (unknown 870129431 0 0) 0x7fd35401cf50 con 0x7fd35c131630 2024-09-16T14:47:29.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.334+0000 7fd3527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T14:47:29.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.334+0000 7fd3527fc640 10 monclient: _finish_command 2 = mon:2 pool 'fooset' not found 2024-09-16T14:47:29.337 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOENT: pool 'fooset' not found 2024-09-16T14:47:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:47:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create fooset 8 2024-09-16T14:47:29.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.450+0000 7f9052dbe640 10 monclient: get_monmap_and_config 2024-09-16T14:47:29.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.450+0000 7f9052dbe640 10 monclient: build_initial_monmap 2024-09-16T14:47:29.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.450+0000 7f9052dbe640 10 monclient: monmap: 2024-09-16T14:47:29.456 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:29.456 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:29.456 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:29.456229+0000 2024-09-16T14:47:29.456 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:29.456229+0000 2024-09-16T14:47:29.456 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:29.456 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:29.456 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:29.456 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:29.457 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:29.457 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:29.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.450+0000 7f9052dbe640 1 Processor -- start 2024-09-16T14:47:29.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.450+0000 7f9052dbe640 1 -- start start 2024-09-16T14:47:29.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.450+0000 7f9052dbe640 10 monclient: init 2024-09-16T14:47:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient(hunting): picked mon.noname-b con 0x7f904c09df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient(hunting): picked mon.noname-c con 0x7f904c09e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient(hunting): picked mon.noname-a con 0x7f904c09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c09ee20 con 0x7f904c09e740 2024-09-16T14:47:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c09f100 con 0x7f904c09e2f0 2024-09-16T14:47:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c09f3e0 con 0x7f904c09df10 2024-09-16T14:47:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient(hunting): authenticate will time out at 1231.572163s 2024-09-16T14:47:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f904c09e2f0 0x7f904c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54614/0 (socket says 172.21.15.189:54614) 2024-09-16T14:47:29.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904bfff640 1 -- 172.21.15.189:0/1330711178 learned_addr learned my addr 172.21.15.189:0/1330711178 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:29.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f905135b640 1 -- 172.21.15.189:0/1330711178 >> v1:172.21.15.26:6789/0 conn(0x7f904c09e740 legacy=0x7f904c13b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:29.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 1 -- 172.21.15.189:0/1330711178 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578962025 0 0) 0x7f904c09f100 con 0x7f904c09e2f0 2024-09-16T14:47:29.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:29.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:29.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:29.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 10 monclient(hunting): my global_id is 94957 2024-09-16T14:47:29.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 1 -- 172.21.15.189:0/1330711178 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9034003530 con 0x7f904c09e2f0 2024-09-16T14:47:29.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 1 -- 172.21.15.189:0/1330711178 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2284285828 0 0) 0x7f9034003530 con 0x7f904c09e2f0 2024-09-16T14:47:29.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 1 -- 172.21.15.189:0/1330711178 >> v1:172.21.15.26:6790/0 conn(0x7f904c09df10 legacy=0x7f904c09d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:29.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 1 -- 172.21.15.189:0/1330711178 >> v1:172.21.15.26:6789/0 conn(0x7f904c09e740 legacy=0x7f904c13b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:29.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:29.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 1 monclient: found mon.noname-c 2024-09-16T14:47:29.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:29.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:29.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 1 -- 172.21.15.189:0/1330711178 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c09c690 con 0x7f904c09e2f0 2024-09-16T14:47:29.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:29.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:29.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:29.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 1 -- 172.21.15.189:0/1330711178 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f903c001620 con 0x7f904c09e2f0 2024-09-16T14:47:29.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:29.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:29.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 10 monclient: dump: 2024-09-16T14:47:29.466 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:29.466 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:29.466 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:29.466 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:29.466 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:29.466 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:29.466 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:29.466 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:29.466 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:29.466 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:29.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 5 monclient: authenticate success, global_id 94957 2024-09-16T14:47:29.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:29.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 1 -- 172.21.15.189:0/1330711178 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f903c001940 con 0x7f904c09e2f0 2024-09-16T14:47:29.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:29.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 1 -- 172.21.15.189:0/1330711178 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f903c001d10 con 0x7f904c09e2f0 2024-09-16T14:47:29.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:29.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 4 set_mon_vals no callback set 2024-09-16T14:47:29.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:29.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:29.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904b7fe640 10 monclient: dump: 2024-09-16T14:47:29.467 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:29.468 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:29.468 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:29.468 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:29.468 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:29.468 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:29.468 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:29.468 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:29.468 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:29.468 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:29.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient: shutdown 2024-09-16T14:47:29.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:29.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 1 -- 172.21.15.189:0/1330711178 >> v1:172.21.15.189:6789/0 conn(0x7f904c09e2f0 legacy=0x7f904c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:29.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 1 -- 172.21.15.189:0/1330711178 shutdown_connections 2024-09-16T14:47:29.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 1 -- 172.21.15.189:0/1330711178 >> 172.21.15.189:0/1330711178 conn(0x7f904c0a3320 msgr2=0x7f904c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:29.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 1 -- 172.21.15.189:0/1330711178 shutdown_connections 2024-09-16T14:47:29.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 1 -- 172.21.15.189:0/1330711178 wait complete. 2024-09-16T14:47:29.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient: build_initial_monmap 2024-09-16T14:47:29.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient: monmap: 2024-09-16T14:47:29.469 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:29.469 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:29.469 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:29.469 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:29.469 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:29.469 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:29.469 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:29.470 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:29.470 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:29.470 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:29.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 1 Processor -- start 2024-09-16T14:47:29.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 1 -- start start 2024-09-16T14:47:29.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient: init 2024-09-16T14:47:29.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:29.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:29.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient(hunting): picked mon.noname-c con 0x7f904c09df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:29.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient(hunting): picked mon.noname-a con 0x7f904c09e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:29.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient(hunting): picked mon.noname-b con 0x7f904c09e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:29.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c09bd30 con 0x7f904c09e2f0 2024-09-16T14:47:29.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c06b380 con 0x7f904c09e740 2024-09-16T14:47:29.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c139e50 con 0x7f904c09df10 2024-09-16T14:47:29.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:29.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient(hunting): authenticate will time out at 1231.575076s 2024-09-16T14:47:29.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f905135b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f904c09e740 0x7f904c1d17a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54626/0 (socket says 172.21.15.189:54626) 2024-09-16T14:47:29.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f905135b640 1 -- 172.21.15.189:0/2936002534 learned_addr learned my addr 172.21.15.189:0/2936002534 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:29.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904bfff640 1 -- 172.21.15.189:0/2936002534 >> v1:172.21.15.26:6789/0 conn(0x7f904c09e2f0 legacy=0x7f904c09a900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:29.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 1 -- 172.21.15.189:0/2936002534 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239403583 0 0) 0x7f904c06b380 con 0x7f904c09e740 2024-09-16T14:47:29.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:29.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:29.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:29.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 10 monclient(hunting): my global_id is 94960 2024-09-16T14:47:29.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 1 -- 172.21.15.189:0/2936002534 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9038003530 con 0x7f904c09e740 2024-09-16T14:47:29.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 1 -- 172.21.15.189:0/2936002534 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4023032858 0 0) 0x7f9038003530 con 0x7f904c09e740 2024-09-16T14:47:29.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 1 -- 172.21.15.189:0/2936002534 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f904c06b380 con 0x7f904c09e740 2024-09-16T14:47:29.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 1 -- 172.21.15.189:0/2936002534 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9044004400 con 0x7f904c09e740 2024-09-16T14:47:29.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:29.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:29.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 10 monclient(hunting): dump: 2024-09-16T14:47:29.472 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:29.472 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:29.472 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:29.472 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:29.472 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:29.472 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:29.472 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:29.473 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:29.473 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:29.473 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:29.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:29.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 1 -- 172.21.15.189:0/2936002534 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3703318433 0 0) 0x7f904c06b380 con 0x7f904c09e740 2024-09-16T14:47:29.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 1 -- 172.21.15.189:0/2936002534 >> v1:172.21.15.26:6790/0 conn(0x7f904c09df10 legacy=0x7f904c09a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:29.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 1 -- 172.21.15.189:0/2936002534 >> v1:172.21.15.26:6789/0 conn(0x7f904c09e2f0 legacy=0x7f904c09a900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:29.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 10 monclient: _finish_hunting 0 2024-09-16T14:47:29.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 1 monclient: found mon.b 2024-09-16T14:47:29.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:29.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:29.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 1 -- 172.21.15.189:0/2936002534 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c06cea0 con 0x7f904c09e740 2024-09-16T14:47:29.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 10 monclient: _finish_auth 0 2024-09-16T14:47:29.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:29.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:29.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 5 monclient: authenticate success, global_id 94960 2024-09-16T14:47:29.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient: _renew_subs 2024-09-16T14:47:29.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:29.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 1 -- 172.21.15.189:0/2936002534 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904c09b010 con 0x7f904c09e740 2024-09-16T14:47:29.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient: _renew_subs 2024-09-16T14:47:29.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:29.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f9052dbe640 1 -- 172.21.15.189:0/2936002534 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f904c06fa30 con 0x7f904c09e740 2024-09-16T14:47:29.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 1 -- 172.21.15.189:0/2936002534 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9044004720 con 0x7f904c09e740 2024-09-16T14:47:29.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:29.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 1 -- 172.21.15.189:0/2936002534 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9044004ab0 con 0x7f904c09e740 2024-09-16T14:47:29.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:29.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:29.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f904affd640 4 set_mon_vals no callback set 2024-09-16T14:47:29.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.454+0000 7f90497fa640 10 monclient: dump: 2024-09-16T14:47:29.475 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:29.475 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:29.475 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:29.475 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:29.475 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:29.475 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:29.475 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:29.475 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:29.475 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:29.475 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:29.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.458+0000 7f9052dbe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:29.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.458+0000 7f9052dbe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:29.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.458+0000 7f9052dbe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:29.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.458+0000 7f90497fa640 1 -- 172.21.15.189:0/2936002534 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9044004c10 con 0x7f904c09e740 2024-09-16T14:47:29.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.458+0000 7f9052dbe640 1 -- 172.21.15.189:0/2936002534 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f902c005050 con 0x7f904c09e740 2024-09-16T14:47:29.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.458+0000 7f90497fa640 1 -- 172.21.15.189:0/2936002534 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(121..121 src has 73..121) v4 ==== 5687+0+0 (unknown 1766820235 0 0) 0x7f9044043c60 con 0x7f904c09e740 2024-09-16T14:47:29.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.458+0000 7f9050b5a640 10 monclient: get_auth_request con 0x7f9038035fe0 auth_method 0 2024-09-16T14:47:29.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.462+0000 7f90497fa640 1 -- 172.21.15.189:0/2936002534 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9044005410 con 0x7f904c09e740 2024-09-16T14:47:29.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.462+0000 7f90497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:29.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.462+0000 7f90497fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:29.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.606+0000 7f9052dbe640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T14:47:29.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.606+0000 7f9052dbe640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T14:47:29.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.606+0000 7f9052dbe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:29.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:29.606+0000 7f9052dbe640 1 -- 172.21.15.189:0/2936002534 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "fooset", "pg_num": 8} v 0) v1 -- 0x7f902c005880 con 0x7f904c09e740 2024-09-16T14:47:30.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.198+0000 7f90497fa640 1 -- 172.21.15.189:0/2936002534 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}]=0 pool 'fooset' created v122) v1 ==== 115+0+0 (unknown 3012195055 0 0) 0x7f9044012e70 con 0x7f904c09e740 2024-09-16T14:47:30.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.198+0000 7f90497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T14:47:30.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.198+0000 7f90497fa640 10 monclient: _finish_command 2 = system:0 pool 'fooset' created 2024-09-16T14:47:30.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.290+0000 7f9052dbe640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T14:47:30.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.290+0000 7f9052dbe640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T14:47:30.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.290+0000 7f9052dbe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:30.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.290+0000 7f9052dbe640 1 -- 172.21.15.189:0/2936002534 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "fooset", "pg_num": 8} v 0) v1 -- 0x7f902c005300 con 0x7f904c09e740 2024-09-16T14:47:30.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.294+0000 7f90497fa640 1 -- 172.21.15.189:0/2936002534 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}]=0 pool 'fooset' already exists v122) v1 ==== 122+0+0 (unknown 694839052 0 0) 0x7f90440145f0 con 0x7f904c09e740 2024-09-16T14:47:30.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.294+0000 7f90497fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T14:47:30.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.294+0000 7f90497fa640 10 monclient: _finish_command 3 = system:0 pool 'fooset' already exists 2024-09-16T14:47:30.299 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fooset' already exists 2024-09-16T14:47:30.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.298+0000 7f9052dbe640 1 -- 172.21.15.189:0/2936002534 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9038035fe0 legacy=0x7f90380384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:30.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.298+0000 7f9052dbe640 10 monclient: shutdown 2024-09-16T14:47:30.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.298+0000 7f9052dbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:30.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.298+0000 7f9052dbe640 1 -- 172.21.15.189:0/2936002534 >> v1:172.21.15.189:6789/0 conn(0x7f904c09e740 legacy=0x7f904c1d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:30.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.298+0000 7f9052dbe640 1 -- 172.21.15.189:0/2936002534 shutdown_connections 2024-09-16T14:47:30.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.298+0000 7f9052dbe640 1 -- 172.21.15.189:0/2936002534 >> 172.21.15.189:0/2936002534 conn(0x7f904c0a3320 msgr2=0x7f904c1335e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:30.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.298+0000 7f9052dbe640 1 -- 172.21.15.189:0/2936002534 shutdown_connections 2024-09-16T14:47:30.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.298+0000 7f9052dbe640 1 -- 172.21.15.189:0/2936002534 wait complete. 2024-09-16T14:47:30.316 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create barset 8 2024-09-16T14:47:30.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.426+0000 7fbe72d49640 10 monclient: get_monmap_and_config 2024-09-16T14:47:30.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.426+0000 7fbe72d49640 10 monclient: build_initial_monmap 2024-09-16T14:47:30.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.426+0000 7fbe72d49640 10 monclient: monmap: 2024-09-16T14:47:30.434 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:30.434 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:30.434 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:30.433666+0000 2024-09-16T14:47:30.434 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:30.433666+0000 2024-09-16T14:47:30.434 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:30.434 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:30.434 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:30.434 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:30.434 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:30.434 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:30.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 1 Processor -- start 2024-09-16T14:47:30.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 1 -- start start 2024-09-16T14:47:30.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 10 monclient: init 2024-09-16T14:47:30.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:30.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:30.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 10 monclient(hunting): picked mon.noname-c con 0x7fbe6c099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:30.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 10 monclient(hunting): picked mon.noname-a con 0x7fbe6c1334d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:30.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 10 monclient(hunting): picked mon.noname-b con 0x7fbe6c136fe0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:30.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6c09b350 con 0x7fbe6c1334d0 2024-09-16T14:47:30.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6c09f2e0 con 0x7fbe6c099a70 2024-09-16T14:47:30.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6c09b070 con 0x7fbe6c136fe0 2024-09-16T14:47:30.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:30.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 10 monclient(hunting): authenticate will time out at 1232.549398s 2024-09-16T14:47:30.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe70ae5640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbe6c099a70 0x7fbe6c09bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54628/0 (socket says 172.21.15.189:54628) 2024-09-16T14:47:30.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe70ae5640 1 -- 172.21.15.189:0/3830917877 learned_addr learned my addr 172.21.15.189:0/3830917877 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:30.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6bfff640 1 -- 172.21.15.189:0/3830917877 >> v1:172.21.15.26:6789/0 conn(0x7fbe6c1334d0 legacy=0x7fbe6c1358c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:30.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 1 -- 172.21.15.189:0/3830917877 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894622869 0 0) 0x7fbe6c09f2e0 con 0x7fbe6c099a70 2024-09-16T14:47:30.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:30.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:30.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:30.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 10 monclient(hunting): my global_id is 94963 2024-09-16T14:47:30.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 1 -- 172.21.15.189:0/3830917877 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe54003530 con 0x7fbe6c099a70 2024-09-16T14:47:30.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 1 -- 172.21.15.189:0/3830917877 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 853814884 0 0) 0x7fbe54003530 con 0x7fbe6c099a70 2024-09-16T14:47:30.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 1 -- 172.21.15.189:0/3830917877 >> v1:172.21.15.26:6790/0 conn(0x7fbe6c136fe0 legacy=0x7fbe6c1393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:30.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 1 -- 172.21.15.189:0/3830917877 >> v1:172.21.15.26:6789/0 conn(0x7fbe6c1334d0 legacy=0x7fbe6c1358c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 1 monclient: found mon.noname-c 2024-09-16T14:47:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 1 -- 172.21.15.189:0/3830917877 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6c09b630 con 0x7fbe6c099a70 2024-09-16T14:47:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 1 -- 172.21.15.189:0/3830917877 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe64001620 con 0x7fbe6c099a70 2024-09-16T14:47:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 5 monclient: authenticate success, global_id 94963 2024-09-16T14:47:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:30.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:30.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:30.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 10 monclient: dump: 2024-09-16T14:47:30.445 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:30.445 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:30.445 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:30.445 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:30.445 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:30.445 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:30.445 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:30.445 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:30.445 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:30.446 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:30.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:30.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 1 -- 172.21.15.189:0/3830917877 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe640018d0 con 0x7fbe6c099a70 2024-09-16T14:47:30.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:30.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 1 -- 172.21.15.189:0/3830917877 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe64001ca0 con 0x7fbe6c099a70 2024-09-16T14:47:30.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:30.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:30.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe6b7fe640 10 monclient: dump: 2024-09-16T14:47:30.446 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:30.446 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:30.446 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:30.446 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:30.447 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:30.447 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:30.447 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:30.447 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:30.447 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:30.447 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:30.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:30.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 4 set_mon_vals no callback set 2024-09-16T14:47:30.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 10 monclient: shutdown 2024-09-16T14:47:30.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:30.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 1 -- 172.21.15.189:0/3830917877 >> v1:172.21.15.189:6789/0 conn(0x7fbe6c099a70 legacy=0x7fbe6c09bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:30.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 1 -- 172.21.15.189:0/3830917877 shutdown_connections 2024-09-16T14:47:30.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 1 -- 172.21.15.189:0/3830917877 >> 172.21.15.189:0/3830917877 conn(0x7fbe6c0a3320 msgr2=0x7fbe6c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:30.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 1 -- 172.21.15.189:0/3830917877 shutdown_connections 2024-09-16T14:47:30.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.430+0000 7fbe72d49640 1 -- 172.21.15.189:0/3830917877 wait complete. 2024-09-16T14:47:30.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 monclient: build_initial_monmap 2024-09-16T14:47:30.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 monclient: monmap: 2024-09-16T14:47:30.448 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:30.448 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:30.448 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:30.448 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:30.448 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:30.448 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:30.448 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:30.449 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:30.449 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:30.449 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:30.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 1 Processor -- start 2024-09-16T14:47:30.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 1 -- start start 2024-09-16T14:47:30.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 monclient: init 2024-09-16T14:47:30.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:30.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:30.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 monclient(hunting): picked mon.noname-b con 0x7fbe6c099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:30.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 monclient(hunting): picked mon.noname-a con 0x7fbe6c1334d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:30.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 monclient(hunting): picked mon.noname-c con 0x7fbe6c136fe0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:30.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6c0710b0 con 0x7fbe6c1334d0 2024-09-16T14:47:30.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6c11c180 con 0x7fbe6c099a70 2024-09-16T14:47:30.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6c0a2190 con 0x7fbe6c136fe0 2024-09-16T14:47:30.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:30.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 monclient(hunting): authenticate will time out at 1232.553091s 2024-09-16T14:47:30.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe6bfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fbe6c1334d0 legacy=0x7fbe6c1cc230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:30.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe70ae5640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbe6c099a70 0x7fbe6c132770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54632/0 (socket says 172.21.15.189:54632) 2024-09-16T14:47:30.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe70ae5640 1 -- 172.21.15.189:0/996562167 learned_addr learned my addr 172.21.15.189:0/996562167 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:30.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747911516 0 0) 0x7fbe6c11c180 con 0x7fbe6c099a70 2024-09-16T14:47:30.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:30.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:30.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:30.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient(hunting): my global_id is 94966 2024-09-16T14:47:30.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe58003530 con 0x7fbe6c099a70 2024-09-16T14:47:30.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2428111248 0 0) 0x7fbe58003530 con 0x7fbe6c099a70 2024-09-16T14:47:30.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe6c11c180 con 0x7fbe6c099a70 2024-09-16T14:47:30.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe64001be0 con 0x7fbe6c099a70 2024-09-16T14:47:30.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:30.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:30.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient(hunting): dump: 2024-09-16T14:47:30.451 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:30.451 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:30.451 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:30.451 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:30.452 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:30.452 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:30.452 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:30.452 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:30.452 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:30.452 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:30.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:30.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015291761 0 0) 0x7fbe6c0a2190 con 0x7fbe6c136fe0 2024-09-16T14:47:30.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:30.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:30.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:30.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient(hunting): my global_id is 114341 2024-09-16T14:47:30.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe58003530 con 0x7fbe6c136fe0 2024-09-16T14:47:30.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2587091223 0 0) 0x7fbe6c11c180 con 0x7fbe6c099a70 2024-09-16T14:47:30.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 >> v1:172.21.15.26:6790/0 conn(0x7fbe6c136fe0 legacy=0x7fbe6c1cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:30.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 >> v1:172.21.15.26:6789/0 conn(0x7fbe6c1334d0 legacy=0x7fbe6c1cc230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:30.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient: _finish_hunting 0 2024-09-16T14:47:30.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 1 monclient: found mon.b 2024-09-16T14:47:30.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:30.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:30.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6c06b380 con 0x7fbe6c099a70 2024-09-16T14:47:30.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient: _finish_auth 0 2024-09-16T14:47:30.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:30.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:30.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 5 monclient: authenticate success, global_id 94966 2024-09-16T14:47:30.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 monclient: _renew_subs 2024-09-16T14:47:30.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:30.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 1 -- 172.21.15.189:0/996562167 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe6c11b850 con 0x7fbe6c099a70 2024-09-16T14:47:30.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 monclient: _renew_subs 2024-09-16T14:47:30.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:30.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 1 -- 172.21.15.189:0/996562167 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe6c119820 con 0x7fbe6c099a70 2024-09-16T14:47:30.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe64004c80 con 0x7fbe6c099a70 2024-09-16T14:47:30.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:30.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe64005010 con 0x7fbe6c099a70 2024-09-16T14:47:30.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:30.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:30.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe6affd640 4 set_mon_vals no callback set 2024-09-16T14:47:30.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 10 monclient: dump: 2024-09-16T14:47:30.455 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:30.455 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:30.455 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:30.455 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:30.455 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:30.455 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:30.455 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:30.455 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:30.455 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:30.455 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:30.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbe64005170 con 0x7fbe6c099a70 2024-09-16T14:47:30.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:30.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:30.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:30.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe72d49640 1 -- 172.21.15.189:0/996562167 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe4c005050 con 0x7fbe6c099a70 2024-09-16T14:47:30.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(122..122 src has 73..122) v4 ==== 6032+0+0 (unknown 560862622 0 0) 0x7fbe64043dc0 con 0x7fbe6c099a70 2024-09-16T14:47:30.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.434+0000 7fbe6bfff640 10 monclient: get_auth_request con 0x7fbe580361d0 auth_method 0 2024-09-16T14:47:30.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.438+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbe64014af0 con 0x7fbe6c099a70 2024-09-16T14:47:30.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.438+0000 7fbe697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:30.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.438+0000 7fbe697fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:30.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.602+0000 7fbe72d49640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T14:47:30.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.602+0000 7fbe72d49640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T14:47:30.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.602+0000 7fbe72d49640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:30.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:30.602+0000 7fbe72d49640 1 -- 172.21.15.189:0/996562167 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "barset", "pg_num": 8} v 0) v1 -- 0x7fbe4c005880 con 0x7fbe6c099a70 2024-09-16T14:47:31.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.174+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}]=0 pool 'barset' created v123) v1 ==== 115+0+0 (unknown 817224582 0 0) 0x7fbe64012f60 con 0x7fbe6c099a70 2024-09-16T14:47:31.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.174+0000 7fbe697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T14:47:31.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.174+0000 7fbe697fa640 10 monclient: _finish_command 2 = system:0 pool 'barset' created 2024-09-16T14:47:31.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.274+0000 7fbe72d49640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T14:47:31.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.274+0000 7fbe72d49640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T14:47:31.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.274+0000 7fbe72d49640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:31.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.274+0000 7fbe72d49640 1 -- 172.21.15.189:0/996562167 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "barset", "pg_num": 8} v 0) v1 -- 0x7fbe4c005300 con 0x7fbe6c099a70 2024-09-16T14:47:31.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.274+0000 7fbe697fa640 1 -- 172.21.15.189:0/996562167 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}]=0 pool 'barset' already exists v123) v1 ==== 122+0+0 (unknown 171313487 0 0) 0x7fbe640148c0 con 0x7fbe6c099a70 2024-09-16T14:47:31.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.274+0000 7fbe697fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T14:47:31.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.274+0000 7fbe697fa640 10 monclient: _finish_command 3 = system:0 pool 'barset' already exists 2024-09-16T14:47:31.279 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'barset' already exists 2024-09-16T14:47:31.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.278+0000 7fbe72d49640 1 -- 172.21.15.189:0/996562167 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbe580361d0 legacy=0x7fbe58038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:31.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.278+0000 7fbe72d49640 10 monclient: shutdown 2024-09-16T14:47:31.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.278+0000 7fbe72d49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:31.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.278+0000 7fbe72d49640 1 -- 172.21.15.189:0/996562167 >> v1:172.21.15.189:6789/0 conn(0x7fbe6c099a70 legacy=0x7fbe6c132770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:31.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.278+0000 7fbe72d49640 1 -- 172.21.15.189:0/996562167 shutdown_connections 2024-09-16T14:47:31.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.278+0000 7fbe72d49640 1 -- 172.21.15.189:0/996562167 >> 172.21.15.189:0/996562167 conn(0x7fbe6c0a3320 msgr2=0x7fbe6c1338b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:31.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.278+0000 7fbe72d49640 1 -- 172.21.15.189:0/996562167 shutdown_connections 2024-09-16T14:47:31.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.278+0000 7fbe72d49640 1 -- 172.21.15.189:0/996562167 wait complete. 2024-09-16T14:47:31.297 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set barset size 3 2024-09-16T14:47:31.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.398+0000 7f8aa67f2640 10 monclient: get_monmap_and_config 2024-09-16T14:47:31.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.398+0000 7f8aa67f2640 10 monclient: build_initial_monmap 2024-09-16T14:47:31.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.398+0000 7f8aa67f2640 10 monclient: monmap: 2024-09-16T14:47:31.405 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:31.405 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:31.405 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:31.404793+0000 2024-09-16T14:47:31.405 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:31.404793+0000 2024-09-16T14:47:31.405 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:31.405 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:31.405 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:31.405 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:31.405 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:31.405 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:31.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.398+0000 7f8aa67f2640 1 Processor -- start 2024-09-16T14:47:31.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.398+0000 7f8aa67f2640 1 -- start start 2024-09-16T14:47:31.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.398+0000 7f8aa67f2640 10 monclient: init 2024-09-16T14:47:31.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:31.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:31.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient(hunting): picked mon.noname-c con 0x7f8aa0131630 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:31.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient(hunting): picked mon.noname-b con 0x7f8aa009edd0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:31.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient(hunting): picked mon.noname-a con 0x7f8aa009e950 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:31.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa009b920 con 0x7f8aa009e950 2024-09-16T14:47:31.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa009bc00 con 0x7f8aa0131630 2024-09-16T14:47:31.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa009cba0 con 0x7f8aa009edd0 2024-09-16T14:47:31.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:31.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient(hunting): authenticate will time out at 1233.520394s 2024-09-16T14:47:31.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9ffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8aa0131630 0x7f8aa0131a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54648/0 (socket says 172.21.15.189:54648) 2024-09-16T14:47:31.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9ffff640 1 -- 172.21.15.189:0/3326199292 learned_addr learned my addr 172.21.15.189:0/3326199292 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:31.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa4d8f640 1 -- 172.21.15.189:0/3326199292 >> v1:172.21.15.26:6789/0 conn(0x7f8aa009e950 legacy=0x7f8aa013a290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:31.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 1 -- 172.21.15.189:0/3326199292 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366444499 0 0) 0x7f8aa009bc00 con 0x7f8aa0131630 2024-09-16T14:47:31.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:31.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:31.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:31.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 10 monclient(hunting): my global_id is 94969 2024-09-16T14:47:31.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 1 -- 172.21.15.189:0/3326199292 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a88003530 con 0x7f8aa0131630 2024-09-16T14:47:31.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 1 -- 172.21.15.189:0/3326199292 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3146853383 0 0) 0x7f8a88003530 con 0x7f8aa0131630 2024-09-16T14:47:31.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 1 -- 172.21.15.189:0/3326199292 >> v1:172.21.15.26:6790/0 conn(0x7f8aa009edd0 legacy=0x7f8aa0136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:47:31.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 1 -- 172.21.15.189:0/3326199292 >> v1:172.21.15.26:6789/0 conn(0x7f8aa009e950 legacy=0x7f8aa013a290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:31.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:31.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 1 monclient: found mon.noname-c 2024-09-16T14:47:31.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:31.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:31.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 1 -- 172.21.15.189:0/3326199292 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa009b420 con 0x7f8aa0131630 2024-09-16T14:47:31.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 10 monclient: _finish_auth 0 2024-09-16T14:47:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 1 -- 172.21.15.189:0/3326199292 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8a90001620 con 0x7f8aa0131630 2024-09-16T14:47:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 10 monclient: dump: 2024-09-16T14:47:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:31.415 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:31.415 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:31.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 5 monclient: authenticate success, global_id 94969 2024-09-16T14:47:31.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:31.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 1 -- 172.21.15.189:0/3326199292 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a90001910 con 0x7f8aa0131630 2024-09-16T14:47:31.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:31.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 1 -- 172.21.15.189:0/3326199292 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8a90001d20 con 0x7f8aa0131630 2024-09-16T14:47:31.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:31.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:31.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9effd640 10 monclient: dump: 2024-09-16T14:47:31.415 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:31.415 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:31.415 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:31.416 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:31.416 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:31.416 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:31.416 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:31.416 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:31.416 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:31.416 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:31.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:31.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 4 set_mon_vals no callback set 2024-09-16T14:47:31.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient: shutdown 2024-09-16T14:47:31.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:31.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 1 -- 172.21.15.189:0/3326199292 >> v1:172.21.15.189:6789/0 conn(0x7f8aa0131630 legacy=0x7f8aa0131a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:31.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 1 -- 172.21.15.189:0/3326199292 shutdown_connections 2024-09-16T14:47:31.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 1 -- 172.21.15.189:0/3326199292 >> 172.21.15.189:0/3326199292 conn(0x7f8aa00a3380 msgr2=0x7f8aa009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:31.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 1 -- 172.21.15.189:0/3326199292 shutdown_connections 2024-09-16T14:47:31.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 1 -- 172.21.15.189:0/3326199292 wait complete. 2024-09-16T14:47:31.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient: build_initial_monmap 2024-09-16T14:47:31.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient: monmap: 2024-09-16T14:47:31.417 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:31.417 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:31.417 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:31.417 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:31.417 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:31.417 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:31.417 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:31.417 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:31.417 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:31.418 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:31.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 1 Processor -- start 2024-09-16T14:47:31.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 1 -- start start 2024-09-16T14:47:31.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient: init 2024-09-16T14:47:31.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:31.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:31.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient(hunting): picked mon.noname-c con 0x7f8aa009e950 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:31.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient(hunting): picked mon.noname-b con 0x7f8aa009edd0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:31.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient(hunting): picked mon.noname-a con 0x7f8aa0131630 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:31.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa006b320 con 0x7f8aa0131630 2024-09-16T14:47:31.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa011e0d0 con 0x7f8aa009edd0 2024-09-16T14:47:31.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa009ad70 con 0x7f8aa009e950 2024-09-16T14:47:31.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:31.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa67f2640 10 monclient(hunting): authenticate will time out at 1233.523347s 2024-09-16T14:47:31.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9f7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8aa009edd0 0x7f8aa011c170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54662/0 (socket says 172.21.15.189:54662) 2024-09-16T14:47:31.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9f7fe640 1 -- 172.21.15.189:0/3016984450 learned_addr learned my addr 172.21.15.189:0/3016984450 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:31.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8aa4d8f640 1 -- 172.21.15.189:0/3016984450 >> v1:172.21.15.26:6789/0 conn(0x7f8aa0131630 legacy=0x7f8aa01cd3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:31.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9cff9640 1 -- 172.21.15.189:0/3016984450 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939226794 0 0) 0x7f8aa011e0d0 con 0x7f8aa009edd0 2024-09-16T14:47:31.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:47:31.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:31.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:31.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9cff9640 10 monclient(hunting): my global_id is 94972 2024-09-16T14:47:31.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9cff9640 1 -- 172.21.15.189:0/3016984450 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a8c003530 con 0x7f8aa009edd0 2024-09-16T14:47:31.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9cff9640 1 -- 172.21.15.189:0/3016984450 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2519890287 0 0) 0x7f8a8c003530 con 0x7f8aa009edd0 2024-09-16T14:47:31.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9cff9640 1 -- 172.21.15.189:0/3016984450 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8aa011e0d0 con 0x7f8aa009edd0 2024-09-16T14:47:31.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9cff9640 1 -- 172.21.15.189:0/3016984450 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8a94004010 con 0x7f8aa009edd0 2024-09-16T14:47:31.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:31.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:31.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9cff9640 10 monclient(hunting): dump: 2024-09-16T14:47:31.420 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:31.420 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:31.420 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:31.420 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:31.420 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:31.420 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:31.420 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:31.420 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:31.420 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:31.420 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:31.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.402+0000 7f8a9cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:31.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 1 -- 172.21.15.189:0/3016984450 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3009401150 0 0) 0x7f8aa011e0d0 con 0x7f8aa009edd0 2024-09-16T14:47:31.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 1 -- 172.21.15.189:0/3016984450 >> v1:172.21.15.26:6790/0 conn(0x7f8aa009e950 legacy=0x7f8aa011ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:31.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 1 -- 172.21.15.189:0/3016984450 >> v1:172.21.15.26:6789/0 conn(0x7f8aa0131630 legacy=0x7f8aa01cd3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:31.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:47:31.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 1 monclient: found mon.b 2024-09-16T14:47:31.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:31.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:31.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 1 -- 172.21.15.189:0/3016984450 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa00a0c60 con 0x7f8aa009edd0 2024-09-16T14:47:31.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 10 monclient: _finish_auth 0 2024-09-16T14:47:31.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:47:31.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:31.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8aa67f2640 5 monclient: authenticate success, global_id 94972 2024-09-16T14:47:31.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8aa67f2640 10 monclient: _renew_subs 2024-09-16T14:47:31.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8aa67f2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:31.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8aa67f2640 1 -- 172.21.15.189:0/3016984450 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa011f440 con 0x7f8aa009edd0 2024-09-16T14:47:31.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8aa67f2640 10 monclient: _renew_subs 2024-09-16T14:47:31.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8aa67f2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:31.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8aa67f2640 1 -- 172.21.15.189:0/3016984450 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa011dd10 con 0x7f8aa009edd0 2024-09-16T14:47:31.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 1 -- 172.21.15.189:0/3016984450 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a94004440 con 0x7f8aa009edd0 2024-09-16T14:47:31.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:31.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 1 -- 172.21.15.189:0/3016984450 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8a94001330 con 0x7f8aa009edd0 2024-09-16T14:47:31.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:31.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9e7fc640 4 set_mon_vals no callback set 2024-09-16T14:47:31.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:31.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 10 monclient: dump: 2024-09-16T14:47:31.422 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:31.423 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:31.423 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:31.423 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:31.423 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:31.423 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:31.423 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:31.423 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:31.423 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:31.423 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:31.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8aa67f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:31.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8aa67f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:31.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8aa67f2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:31.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8aa67f2640 1 -- 172.21.15.189:0/3016984450 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a80005050 con 0x7f8aa009edd0 2024-09-16T14:47:31.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 1 -- 172.21.15.189:0/3016984450 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8a94001510 con 0x7f8aa009edd0 2024-09-16T14:47:31.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9cff9640 1 -- 172.21.15.189:0/3016984450 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(123..123 src has 73..123) v4 ==== 6377+0+0 (unknown 3975865481 0 0) 0x7f8a94043d70 con 0x7f8aa009edd0 2024-09-16T14:47:31.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.406+0000 7f8a9ffff640 10 monclient: get_auth_request con 0x7f8a8c036010 auth_method 0 2024-09-16T14:47:31.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.410+0000 7f8a9cff9640 1 -- 172.21.15.189:0/3016984450 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8a94014630 con 0x7f8aa009edd0 2024-09-16T14:47:31.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.410+0000 7f8a9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:31.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.410+0000 7f8a9cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:31.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.566+0000 7f8aa67f2640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T14:47:31.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.566+0000 7f8aa67f2640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T14:47:31.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.566+0000 7f8aa67f2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:31.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:31.566+0000 7f8aa67f2640 1 -- 172.21.15.189:0/3016984450 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"} v 0) v1 -- 0x7f8a80005880 con 0x7f8aa009edd0 2024-09-16T14:47:32.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:32.174+0000 7f8a9cff9640 1 -- 172.21.15.189:0/3016984450 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}]=0 set pool 10 size to 3 v124) v1 ==== 126+0+0 (unknown 1038236803 0 0) 0x7f8a94014450 con 0x7f8aa009edd0 2024-09-16T14:47:32.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:32.174+0000 7f8a9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T14:47:32.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:32.174+0000 7f8a9cff9640 10 monclient: _finish_command 2 = system:0 set pool 10 size to 3 2024-09-16T14:47:32.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:32.278+0000 7f8aa67f2640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T14:47:32.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:32.278+0000 7f8aa67f2640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T14:47:32.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:32.278+0000 7f8aa67f2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:32.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:32.278+0000 7f8aa67f2640 1 -- 172.21.15.189:0/3016984450 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"} v 0) v1 -- 0x7f8a80005300 con 0x7f8aa009edd0 2024-09-16T14:47:33.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.178+0000 7f8a9cff9640 1 -- 172.21.15.189:0/3016984450 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}]=0 set pool 10 size to 3 v125) v1 ==== 126+0+0 (unknown 3212477337 0 0) 0x7f8a94014450 con 0x7f8aa009edd0 2024-09-16T14:47:33.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.178+0000 7f8a9cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T14:47:33.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.178+0000 7f8a9cff9640 10 monclient: _finish_command 3 = system:0 set pool 10 size to 3 2024-09-16T14:47:33.185 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 10 size to 3 2024-09-16T14:47:33.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.182+0000 7f8aa67f2640 1 -- 172.21.15.189:0/3016984450 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8a8c036010 legacy=0x7f8a8c0384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:33.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.182+0000 7f8aa67f2640 10 monclient: shutdown 2024-09-16T14:47:33.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.182+0000 7f8aa67f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:33.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.182+0000 7f8aa67f2640 1 -- 172.21.15.189:0/3016984450 >> v1:172.21.15.189:6789/0 conn(0x7f8aa009edd0 legacy=0x7f8aa011c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:33.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.182+0000 7f8aa67f2640 1 -- 172.21.15.189:0/3016984450 shutdown_connections 2024-09-16T14:47:33.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.182+0000 7f8aa67f2640 1 -- 172.21.15.189:0/3016984450 >> 172.21.15.189:0/3016984450 conn(0x7f8aa00a3380 msgr2=0x7f8aa009d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:33.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.186+0000 7f8aa67f2640 1 -- 172.21.15.189:0/3016984450 shutdown_connections 2024-09-16T14:47:33.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.186+0000 7f8aa67f2640 1 -- 172.21.15.189:0/3016984450 wait complete. 2024-09-16T14:47:33.209 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T14:47:33.209 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:47:33.209 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T14:47:33.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 10 monclient: get_monmap_and_config 2024-09-16T14:47:33.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 10 monclient: build_initial_monmap 2024-09-16T14:47:33.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 10 monclient: monmap: 2024-09-16T14:47:33.331 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:33.331 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:33.331 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:33.330850+0000 2024-09-16T14:47:33.331 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:33.330850+0000 2024-09-16T14:47:33.331 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:33.331 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:33.331 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:33.331 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:33.331 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:33.331 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:33.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 1 Processor -- start 2024-09-16T14:47:33.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 1 -- start start 2024-09-16T14:47:33.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 10 monclient: init 2024-09-16T14:47:33.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:33.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:33.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 10 monclient(hunting): picked mon.noname-a con 0x7fd76c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:33.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 10 monclient(hunting): picked mon.noname-b con 0x7fd76c09c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:33.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 10 monclient(hunting): picked mon.noname-c con 0x7fd76c09e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:33.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76c099e50 con 0x7fd76c099a70 2024-09-16T14:47:33.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76c09dbd0 con 0x7fd76c09e6d0 2024-09-16T14:47:33.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76c09ee20 con 0x7fd76c09c550 2024-09-16T14:47:33.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:33.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 10 monclient(hunting): authenticate will time out at 1235.446945s 2024-09-16T14:47:33.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76ad9d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fd76c099a70 legacy=0x7fd76c09dfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:33.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76b59e640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd76c09e6d0 0x7fd76c13b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54674/0 (socket says 172.21.15.189:54674) 2024-09-16T14:47:33.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76b59e640 1 -- 172.21.15.189:0/2736582513 learned_addr learned my addr 172.21.15.189:0/2736582513 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:33.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 1 -- 172.21.15.189:0/2736582513 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 196466604 0 0) 0x7fd76c09dbd0 con 0x7fd76c09e6d0 2024-09-16T14:47:33.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 10 monclient(hunting): _negotiate 2024-09-16T14:47:33.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:33.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:33.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 10 monclient(hunting): my global_id is 94975 2024-09-16T14:47:33.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 1 -- 172.21.15.189:0/2736582513 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd758003530 con 0x7fd76c09e6d0 2024-09-16T14:47:33.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 1 -- 172.21.15.189:0/2736582513 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2070095317 0 0) 0x7fd758003530 con 0x7fd76c09e6d0 2024-09-16T14:47:33.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 1 -- 172.21.15.189:0/2736582513 >> v1:172.21.15.26:6790/0 conn(0x7fd76c09c550 legacy=0x7fd76c1379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:33.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 1 -- 172.21.15.189:0/2736582513 >> v1:172.21.15.26:6789/0 conn(0x7fd76c099a70 legacy=0x7fd76c09dfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:33.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 10 monclient: _finish_hunting 0 2024-09-16T14:47:33.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 1 monclient: found mon.noname-c 2024-09-16T14:47:33.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:33.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:33.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 1 -- 172.21.15.189:0/2736582513 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c09f100 con 0x7fd76c09e6d0 2024-09-16T14:47:33.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 10 monclient: _finish_auth 0 2024-09-16T14:47:33.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 10 monclient: _check_auth_tickets 2024-09-16T14:47:33.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:33.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 5 monclient: authenticate success, global_id 94975 2024-09-16T14:47:33.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:33.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 1 -- 172.21.15.189:0/2736582513 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd7640019a0 con 0x7fd76c09e6d0 2024-09-16T14:47:33.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:33.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:33.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 10 monclient: dump: 2024-09-16T14:47:33.345 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:33.345 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:33.345 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:33.345 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:33.345 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:33.345 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:33.345 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:33.345 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:33.345 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:33.345 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:33.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:33.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 1 -- 172.21.15.189:0/2736582513 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd764001c50 con 0x7fd76c09e6d0 2024-09-16T14:47:33.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:33.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 1 -- 172.21.15.189:0/2736582513 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd764001fe0 con 0x7fd76c09e6d0 2024-09-16T14:47:33.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:33.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd7710d6640 4 set_mon_vals no callback set 2024-09-16T14:47:33.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:33.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:33.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.326+0000 7fd76a59c640 10 monclient: dump: 2024-09-16T14:47:33.346 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:33.346 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:33.346 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:33.346 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:33.346 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:33.346 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:33.346 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:33.347 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:33.347 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:33.347 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:33.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient: shutdown 2024-09-16T14:47:33.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:33.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 1 -- 172.21.15.189:0/2736582513 >> v1:172.21.15.189:6789/0 conn(0x7fd76c09e6d0 legacy=0x7fd76c13b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:33.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 1 -- 172.21.15.189:0/2736582513 shutdown_connections 2024-09-16T14:47:33.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 1 -- 172.21.15.189:0/2736582513 >> 172.21.15.189:0/2736582513 conn(0x7fd76c0a3380 msgr2=0x7fd76c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:33.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 1 -- 172.21.15.189:0/2736582513 shutdown_connections 2024-09-16T14:47:33.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 1 -- 172.21.15.189:0/2736582513 wait complete. 2024-09-16T14:47:33.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient: build_initial_monmap 2024-09-16T14:47:33.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient: monmap: 2024-09-16T14:47:33.347 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:33.348 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:33.348 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:33.348 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:33.348 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:33.348 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:33.348 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:33.348 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:33.348 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:33.348 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:33.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 1 Processor -- start 2024-09-16T14:47:33.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 1 -- start start 2024-09-16T14:47:33.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient: init 2024-09-16T14:47:33.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:33.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:33.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient(hunting): picked mon.noname-c con 0x7fd76c099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:33.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient(hunting): picked mon.noname-a con 0x7fd76c09c550 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:33.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient(hunting): picked mon.noname-b con 0x7fd76c09e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:33.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76c09cf20 con 0x7fd76c09c550 2024-09-16T14:47:33.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76c13b280 con 0x7fd76c09e6d0 2024-09-16T14:47:33.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76c1194c0 con 0x7fd76c099a70 2024-09-16T14:47:33.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:33.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient(hunting): authenticate will time out at 1235.449962s 2024-09-16T14:47:33.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd763fff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fd76c09c550 legacy=0x7fd76c1cbee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:33.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd76b59e640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd76c09e6d0 0x7fd76c1cf610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54680/0 (socket says 172.21.15.189:54680) 2024-09-16T14:47:33.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd76b59e640 1 -- 172.21.15.189:0/294205619 learned_addr learned my addr 172.21.15.189:0/294205619 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:33.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068221517 0 0) 0x7fd76c13b280 con 0x7fd76c09e6d0 2024-09-16T14:47:33.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:33.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:33.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:33.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient(hunting): my global_id is 94978 2024-09-16T14:47:33.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd74c003530 con 0x7fd76c09e6d0 2024-09-16T14:47:33.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3244516620 0 0) 0x7fd74c003530 con 0x7fd76c09e6d0 2024-09-16T14:47:33.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd76c13b280 con 0x7fd76c09e6d0 2024-09-16T14:47:33.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd764001960 con 0x7fd76c09e6d0 2024-09-16T14:47:33.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:33.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:33.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient(hunting): dump: 2024-09-16T14:47:33.350 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:33.350 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:33.351 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:33.351 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:33.351 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:33.351 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:33.351 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:33.351 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:33.351 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:33.351 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:33.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:33.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460961288 0 0) 0x7fd76c1194c0 con 0x7fd76c099a70 2024-09-16T14:47:33.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:33.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:33.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:33.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient(hunting): my global_id is 114347 2024-09-16T14:47:33.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd74c003530 con 0x7fd76c099a70 2024-09-16T14:47:33.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2111153815 0 0) 0x7fd76c13b280 con 0x7fd76c09e6d0 2024-09-16T14:47:33.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 >> v1:172.21.15.26:6790/0 conn(0x7fd76c099a70 legacy=0x7fd76c131b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:33.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 >> v1:172.21.15.26:6789/0 conn(0x7fd76c09c550 legacy=0x7fd76c1cbee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:33.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:33.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 1 monclient: found mon.b 2024-09-16T14:47:33.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:33.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:33.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c13af90 con 0x7fd76c09e6d0 2024-09-16T14:47:33.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:33.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:33.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:33.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 5 monclient: authenticate success, global_id 94978 2024-09-16T14:47:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient: _renew_subs 2024-09-16T14:47:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 1 -- 172.21.15.189:0/294205619 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76c13b540 con 0x7fd76c09e6d0 2024-09-16T14:47:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient: _renew_subs 2024-09-16T14:47:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 1 -- 172.21.15.189:0/294205619 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76c132650 con 0x7fd76c09e6d0 2024-09-16T14:47:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd764004f80 con 0x7fd76c09e6d0 2024-09-16T14:47:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd764005290 con 0x7fd76c09e6d0 2024-09-16T14:47:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd769d9b640 4 set_mon_vals no callback set 2024-09-16T14:47:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 10 monclient: dump: 2024-09-16T14:47:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:33.354 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7710d6640 1 -- 172.21.15.189:0/294205619 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd750005050 con 0x7fd76c09e6d0 2024-09-16T14:47:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.330+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd76400fd00 con 0x7fd76c09e6d0 2024-09-16T14:47:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.334+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(125..125 src has 73..125) v4 ==== 6377+0+0 (unknown 3363505439 0 0) 0x7fd764044290 con 0x7fd76c09e6d0 2024-09-16T14:47:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.334+0000 7fd76ad9d640 10 monclient: get_auth_request con 0x7fd74c036120 auth_method 0 2024-09-16T14:47:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.338+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd764014c30 con 0x7fd76c09e6d0 2024-09-16T14:47:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.338+0000 7fd7637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.338+0000 7fd7637fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:33.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.502+0000 7fd7710d6640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T14:47:33.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.502+0000 7fd7710d6640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T14:47:33.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.502+0000 7fd7710d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:33.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.502+0000 7fd7710d6640 1 -- 172.21.15.189:0/294205619 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7fd750005880 con 0x7fd76c09e6d0 2024-09-16T14:47:33.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.506+0000 7fd7637fe640 1 -- 172.21.15.189:0/294205619 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=-2 no weight-set for pool 'fooset' v125) v1 ==== 160+0+0 (unknown 3111324029 0 0) 0x7fd7640132d0 con 0x7fd76c09e6d0 2024-09-16T14:47:33.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.506+0000 7fd7637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T14:47:33.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.506+0000 7fd7637fe640 10 monclient: _finish_command 2 = mon:2 no weight-set for pool 'fooset' 2024-09-16T14:47:33.511 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOENT: no weight-set for pool 'fooset' 2024-09-16T14:47:33.515 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:47:33.515 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set create fooset flat 2024-09-16T14:47:33.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f1367162640 10 monclient: get_monmap_and_config 2024-09-16T14:47:33.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f1367162640 10 monclient: build_initial_monmap 2024-09-16T14:47:33.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f1367162640 10 monclient: monmap: 2024-09-16T14:47:33.635 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:33.635 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:33.635 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:33.635192+0000 2024-09-16T14:47:33.635 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:33.635192+0000 2024-09-16T14:47:33.635 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:33.636 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:33.636 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:33.636 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:33.636 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:33.636 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:33.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f1367162640 1 Processor -- start 2024-09-16T14:47:33.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f1367162640 1 -- start start 2024-09-16T14:47:33.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f1367162640 10 monclient: init 2024-09-16T14:47:33.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f1367162640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:33.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f1367162640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:33.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f1367162640 10 monclient(hunting): picked mon.noname-b con 0x7f1360131920 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:33.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f1367162640 10 monclient(hunting): picked mon.noname-a con 0x7f136006a650 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:33.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f1367162640 10 monclient(hunting): picked mon.noname-c con 0x7f136013b180 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:33.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f1367162640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136006a480 con 0x7f136006a650 2024-09-16T14:47:33.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f1367162640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13600993c0 con 0x7f136013b180 2024-09-16T14:47:33.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f1367162640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13600990e0 con 0x7f1360131920 2024-09-16T14:47:33.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f1367162640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:33.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f1367162640 10 monclient(hunting): authenticate will time out at 1235.751737s 2024-09-16T14:47:33.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f135ffff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f136006a650 legacy=0x7f1360139a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:33.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.630+0000 7f13656ff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f136013b180 0x7f136013d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54694/0 (socket says 172.21.15.189:54694) 2024-09-16T14:47:33.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f13656ff640 1 -- 172.21.15.189:0/4011250748 learned_addr learned my addr 172.21.15.189:0/4011250748 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:33.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 1 -- 172.21.15.189:0/4011250748 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748124883 0 0) 0x7f13600993c0 con 0x7f136013b180 2024-09-16T14:47:33.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:33.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:33.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:33.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 10 monclient(hunting): my global_id is 94981 2024-09-16T14:47:33.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 1 -- 172.21.15.189:0/4011250748 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1350003530 con 0x7f136013b180 2024-09-16T14:47:33.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 1 -- 172.21.15.189:0/4011250748 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 220966463 0 0) 0x7f1350003530 con 0x7f136013b180 2024-09-16T14:47:33.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 1 -- 172.21.15.189:0/4011250748 >> v1:172.21.15.26:6790/0 conn(0x7f1360131920 legacy=0x7f1360131d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:33.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 1 -- 172.21.15.189:0/4011250748 >> v1:172.21.15.26:6789/0 conn(0x7f136006a650 legacy=0x7f1360139a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:33.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:33.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 1 monclient: found mon.noname-c 2024-09-16T14:47:33.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:33.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:33.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 1 -- 172.21.15.189:0/4011250748 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13600996a0 con 0x7f136013b180 2024-09-16T14:47:33.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:33.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:33.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:33.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 5 monclient: authenticate success, global_id 94981 2024-09-16T14:47:33.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:33.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 1 -- 172.21.15.189:0/4011250748 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f13580019a0 con 0x7f136013b180 2024-09-16T14:47:33.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:33.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:33.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 10 monclient: dump: 2024-09-16T14:47:33.648 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:33.648 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:33.648 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:33.648 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:33.648 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:33.649 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:33.649 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:33.649 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:33.649 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:33.649 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:33.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:33.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 1 -- 172.21.15.189:0/4011250748 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1358001c50 con 0x7f136013b180 2024-09-16T14:47:33.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:33.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:33.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 1 -- 172.21.15.189:0/4011250748 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1358002020 con 0x7f136013b180 2024-09-16T14:47:33.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:33.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 4 set_mon_vals no callback set 2024-09-16T14:47:33.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:33.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135f7fe640 10 monclient: dump: 2024-09-16T14:47:33.650 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:33.650 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:33.651 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:33.651 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:33.651 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:33.651 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:33.651 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:33.651 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:33.651 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:33.651 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:33.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 10 monclient: shutdown 2024-09-16T14:47:33.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:33.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 1 -- 172.21.15.189:0/4011250748 >> v1:172.21.15.189:6789/0 conn(0x7f136013b180 legacy=0x7f136013d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:33.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 1 -- 172.21.15.189:0/4011250748 shutdown_connections 2024-09-16T14:47:33.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 1 -- 172.21.15.189:0/4011250748 >> 172.21.15.189:0/4011250748 conn(0x7f13600a4400 msgr2=0x7f13600a08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:33.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 1 -- 172.21.15.189:0/4011250748 shutdown_connections 2024-09-16T14:47:33.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 1 -- 172.21.15.189:0/4011250748 wait complete. 2024-09-16T14:47:33.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 10 monclient: build_initial_monmap 2024-09-16T14:47:33.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 10 monclient: monmap: 2024-09-16T14:47:33.652 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:33.652 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:33.653 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:33.653 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:33.653 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:33.653 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:33.653 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:33.653 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:33.653 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:33.653 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:33.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 1 Processor -- start 2024-09-16T14:47:33.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 1 -- start start 2024-09-16T14:47:33.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 10 monclient: init 2024-09-16T14:47:33.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:33.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:33.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 10 monclient(hunting): picked mon.noname-a con 0x7f136006a650 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:33.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 10 monclient(hunting): picked mon.noname-b con 0x7f1360131920 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:33.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 10 monclient(hunting): picked mon.noname-c con 0x7f136013b180 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:33.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13600a0040 con 0x7f136006a650 2024-09-16T14:47:33.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136009f8d0 con 0x7f1360131920 2024-09-16T14:47:33.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136009b100 con 0x7f136013b180 2024-09-16T14:47:33.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:33.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 10 monclient(hunting): authenticate will time out at 1235.754764s 2024-09-16T14:47:33.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135ffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1360131920 0x7f136013a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54710/0 (socket says 172.21.15.189:54710) 2024-09-16T14:47:33.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135ffff640 1 -- 172.21.15.189:0/1262610459 learned_addr learned my addr 172.21.15.189:0/1262610459 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:33.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1364efe640 1 -- 172.21.15.189:0/1262610459 >> v1:172.21.15.26:6789/0 conn(0x7f136006a650 legacy=0x7f13601c8ee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:33.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 1 -- 172.21.15.189:0/1262610459 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858479927 0 0) 0x7f136009f8d0 con 0x7f1360131920 2024-09-16T14:47:33.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:33.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:33.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:33.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 10 monclient(hunting): my global_id is 94984 2024-09-16T14:47:33.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 1 -- 172.21.15.189:0/1262610459 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1344003530 con 0x7f1360131920 2024-09-16T14:47:33.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 1 -- 172.21.15.189:0/1262610459 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3745521642 0 0) 0x7f1344003530 con 0x7f1360131920 2024-09-16T14:47:33.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 1 -- 172.21.15.189:0/1262610459 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f136009f8d0 con 0x7f1360131920 2024-09-16T14:47:33.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 1 -- 172.21.15.189:0/1262610459 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f134c004190 con 0x7f1360131920 2024-09-16T14:47:33.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:33.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:33.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 10 monclient(hunting): dump: 2024-09-16T14:47:33.657 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:33.657 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:33.657 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:33.657 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:33.657 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:33.657 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:33.657 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:33.657 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:33.657 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:33.657 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:33.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:33.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 1 -- 172.21.15.189:0/1262610459 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1159965611 0 0) 0x7f136009f8d0 con 0x7f1360131920 2024-09-16T14:47:33.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 1 -- 172.21.15.189:0/1262610459 >> v1:172.21.15.26:6790/0 conn(0x7f136013b180 legacy=0x7f13601d1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:33.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 1 -- 172.21.15.189:0/1262610459 >> v1:172.21.15.26:6789/0 conn(0x7f136006a650 legacy=0x7f13601c8ee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:33.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:47:33.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 1 monclient: found mon.b 2024-09-16T14:47:33.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:33.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:33.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 1 -- 172.21.15.189:0/1262610459 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1360136a60 con 0x7f1360131920 2024-09-16T14:47:33.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:47:33.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:33.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:33.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 5 monclient: authenticate success, global_id 94984 2024-09-16T14:47:33.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 10 monclient: _renew_subs 2024-09-16T14:47:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 1 -- 172.21.15.189:0/1262610459 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1360138520 con 0x7f1360131920 2024-09-16T14:47:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 10 monclient: _renew_subs 2024-09-16T14:47:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f1367162640 1 -- 172.21.15.189:0/1262610459 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f136013aa00 con 0x7f1360131920 2024-09-16T14:47:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 1 -- 172.21.15.189:0/1262610459 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f134c004440 con 0x7f1360131920 2024-09-16T14:47:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 1 -- 172.21.15.189:0/1262610459 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f134c0047d0 con 0x7f1360131920 2024-09-16T14:47:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135effd640 4 set_mon_vals no callback set 2024-09-16T14:47:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.634+0000 7f135d7fa640 10 monclient: dump: 2024-09-16T14:47:33.660 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:33.660 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:33.660 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:33.660 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:33.660 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:33.660 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:33.660 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:33.660 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:33.660 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:33.660 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:33.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.638+0000 7f1367162640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:33.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.638+0000 7f1367162640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:33.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.638+0000 7f1367162640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:33.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.638+0000 7f1367162640 1 -- 172.21.15.189:0/1262610459 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1340005050 con 0x7f1360131920 2024-09-16T14:47:33.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.638+0000 7f135d7fa640 1 -- 172.21.15.189:0/1262610459 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f134c004930 con 0x7f1360131920 2024-09-16T14:47:33.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.638+0000 7f135d7fa640 1 -- 172.21.15.189:0/1262610459 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(125..125 src has 73..125) v4 ==== 6377+0+0 (unknown 3363505439 0 0) 0x7f134c043ca0 con 0x7f1360131920 2024-09-16T14:47:33.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.638+0000 7f1364efe640 10 monclient: get_auth_request con 0x7f1344035e40 auth_method 0 2024-09-16T14:47:33.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.642+0000 7f135d7fa640 1 -- 172.21.15.189:0/1262610459 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f134c005100 con 0x7f1360131920 2024-09-16T14:47:33.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.642+0000 7f135d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:33.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.642+0000 7f135d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:33.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.806+0000 7f1367162640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T14:47:33.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.806+0000 7f1367162640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T14:47:33.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.806+0000 7f1367162640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:33.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:33.806+0000 7f1367162640 1 -- 172.21.15.189:0/1262610459 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"} v 0) v1 -- 0x7f1340005880 con 0x7f1360131920 2024-09-16T14:47:34.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.182+0000 7f135d7fa640 1 -- 172.21.15.189:0/1262610459 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}]=0 v126) v1 ==== 109+0+0 (unknown 2608208192 0 0) 0x7f134c012b60 con 0x7f1360131920 2024-09-16T14:47:34.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.182+0000 7f135d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T14:47:34.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.182+0000 7f135d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.286+0000 7f1367162640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T14:47:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.286+0000 7f1367162640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T14:47:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.286+0000 7f1367162640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.286+0000 7f1367162640 1 -- 172.21.15.189:0/1262610459 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"} v 0) v1 -- 0x7f1340005300 con 0x7f1360131920 2024-09-16T14:47:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.290+0000 7f135d7fa640 1 -- 172.21.15.189:0/1262610459 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}]=0 weight-set for pool 'fooset' already created v126) v1 ==== 153+0+0 (unknown 1886876454 0 0) 0x7f134c0142e0 con 0x7f1360131920 2024-09-16T14:47:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.290+0000 7f135d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T14:47:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.290+0000 7f135d7fa640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'fooset' already created 2024-09-16T14:47:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:weight-set for pool 'fooset' already created 2024-09-16T14:47:34.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.290+0000 7f1367162640 1 -- 172.21.15.189:0/1262610459 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1344035e40 legacy=0x7f1344038300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:34.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.290+0000 7f1367162640 10 monclient: shutdown 2024-09-16T14:47:34.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.290+0000 7f1367162640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:34.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.290+0000 7f1367162640 1 -- 172.21.15.189:0/1262610459 >> v1:172.21.15.189:6789/0 conn(0x7f1360131920 legacy=0x7f136013a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:34.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.290+0000 7f1367162640 1 -- 172.21.15.189:0/1262610459 shutdown_connections 2024-09-16T14:47:34.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.290+0000 7f1367162640 1 -- 172.21.15.189:0/1262610459 >> 172.21.15.189:0/1262610459 conn(0x7f13600a4400 msgr2=0x7f136009cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:34.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.290+0000 7f1367162640 1 -- 172.21.15.189:0/1262610459 shutdown_connections 2024-09-16T14:47:34.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.290+0000 7f1367162640 1 -- 172.21.15.189:0/1262610459 wait complete. 2024-09-16T14:47:34.312 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set create barset positional 2024-09-16T14:47:34.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.426+0000 7f42e0e96640 10 monclient: get_monmap_and_config 2024-09-16T14:47:34.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.426+0000 7f42e0e96640 10 monclient: build_initial_monmap 2024-09-16T14:47:34.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.426+0000 7f42e0e96640 10 monclient: monmap: 2024-09-16T14:47:34.433 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:34.434 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:34.434 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:34.433551+0000 2024-09-16T14:47:34.434 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:34.433551+0000 2024-09-16T14:47:34.434 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:34.434 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:34.434 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:34.434 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:34.434 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:34.434 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 1 Processor -- start 2024-09-16T14:47:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 1 -- start start 2024-09-16T14:47:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 10 monclient: init 2024-09-16T14:47:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 10 monclient(hunting): picked mon.noname-a con 0x7f42dc0a1a90 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 10 monclient(hunting): picked mon.noname-c con 0x7f42dc0a2580 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 10 monclient(hunting): picked mon.noname-b con 0x7f42dc09fd40 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42dc09df00 con 0x7f42dc0a1a90 2024-09-16T14:47:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42dc09f0a0 con 0x7f42dc0a2580 2024-09-16T14:47:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42dc0a3040 con 0x7f42dc09fd40 2024-09-16T14:47:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 10 monclient(hunting): authenticate will time out at 1236.549531s 2024-09-16T14:47:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42da59c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f42dc0a1a90 legacy=0x7f42dc0a1e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d3fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f42dc0a2580 0x7f42dc139df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54722/0 (socket says 172.21.15.189:54722) 2024-09-16T14:47:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d3fff640 1 -- 172.21.15.189:0/3265848496 learned_addr learned my addr 172.21.15.189:0/3265848496 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 1 -- 172.21.15.189:0/3265848496 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676416897 0 0) 0x7f42dc09f0a0 con 0x7f42dc0a2580 2024-09-16T14:47:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:47:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 10 monclient(hunting): my global_id is 94987 2024-09-16T14:47:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 1 -- 172.21.15.189:0/3265848496 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42c8003530 con 0x7f42dc0a2580 2024-09-16T14:47:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 1 -- 172.21.15.189:0/3265848496 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 362130225 0 0) 0x7f42c8003530 con 0x7f42dc0a2580 2024-09-16T14:47:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 1 -- 172.21.15.189:0/3265848496 >> v1:172.21.15.26:6790/0 conn(0x7f42dc09fd40 legacy=0x7f42dc13d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 1 -- 172.21.15.189:0/3265848496 >> v1:172.21.15.26:6789/0 conn(0x7f42dc0a1a90 legacy=0x7f42dc0a1e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:47:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 1 monclient: found mon.noname-c 2024-09-16T14:47:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 1 -- 172.21.15.189:0/3265848496 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42dc09eae0 con 0x7f42dc0a2580 2024-09-16T14:47:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 10 monclient: _finish_auth 0 2024-09-16T14:47:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:47:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 1 -- 172.21.15.189:0/3265848496 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f42c4001620 con 0x7f42dc0a2580 2024-09-16T14:47:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 5 monclient: authenticate success, global_id 94987 2024-09-16T14:47:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 10 monclient: dump: 2024-09-16T14:47:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:34.444 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:34.444 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:34.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:34.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 1 -- 172.21.15.189:0/3265848496 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42c40018d0 con 0x7f42dc0a2580 2024-09-16T14:47:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 1 -- 172.21.15.189:0/3265848496 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f42c4001c60 con 0x7f42dc0a2580 2024-09-16T14:47:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42d9d9b640 10 monclient: dump: 2024-09-16T14:47:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:34.446 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 4 set_mon_vals no callback set 2024-09-16T14:47:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 10 monclient: shutdown 2024-09-16T14:47:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 1 -- 172.21.15.189:0/3265848496 >> v1:172.21.15.189:6789/0 conn(0x7f42dc0a2580 legacy=0x7f42dc139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 1 -- 172.21.15.189:0/3265848496 shutdown_connections 2024-09-16T14:47:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 1 -- 172.21.15.189:0/3265848496 >> 172.21.15.189:0/3265848496 conn(0x7f42dc0a7a70 msgr2=0x7f42dc0a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 1 -- 172.21.15.189:0/3265848496 shutdown_connections 2024-09-16T14:47:34.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 1 -- 172.21.15.189:0/3265848496 wait complete. 2024-09-16T14:47:34.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 10 monclient: build_initial_monmap 2024-09-16T14:47:34.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.430+0000 7f42e0e96640 10 monclient: monmap: 2024-09-16T14:47:34.447 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:34.447 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:34.447 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:34.447 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:34.447 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:34.447 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:34.448 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 1 Processor -- start 2024-09-16T14:47:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 1 -- start start 2024-09-16T14:47:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 10 monclient: init 2024-09-16T14:47:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 10 monclient(hunting): picked mon.noname-b con 0x7f42dc09fd40 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 10 monclient(hunting): picked mon.noname-a con 0x7f42dc0a1a90 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 10 monclient(hunting): picked mon.noname-c con 0x7f42dc0a2580 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42dc1242f0 con 0x7f42dc0a1a90 2024-09-16T14:47:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42dc0a6340 con 0x7f42dc09fd40 2024-09-16T14:47:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42dc09e280 con 0x7f42dc0a2580 2024-09-16T14:47:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 10 monclient(hunting): authenticate will time out at 1236.552626s 2024-09-16T14:47:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42da59c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f42dc09fd40 0x7f42dc133ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54728/0 (socket says 172.21.15.189:54728) 2024-09-16T14:47:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42da59c640 1 -- 172.21.15.189:0/3706910800 learned_addr learned my addr 172.21.15.189:0/3706910800 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d3fff640 1 -- 172.21.15.189:0/3706910800 >> v1:172.21.15.26:6789/0 conn(0x7f42dc0a1a90 legacy=0x7f42dc1ce4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389183717 0 0) 0x7f42dc0a6340 con 0x7f42dc09fd40 2024-09-16T14:47:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient(hunting): my global_id is 94990 2024-09-16T14:47:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42bc003530 con 0x7f42dc09fd40 2024-09-16T14:47:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 114020839 0 0) 0x7f42bc003530 con 0x7f42dc09fd40 2024-09-16T14:47:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42dc0a6340 con 0x7f42dc09fd40 2024-09-16T14:47:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f42cc0041e0 con 0x7f42dc09fd40 2024-09-16T14:47:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient(hunting): dump: 2024-09-16T14:47:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:34.451 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741178631 0 0) 0x7f42dc09e280 con 0x7f42dc0a2580 2024-09-16T14:47:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient(hunting): my global_id is 114359 2024-09-16T14:47:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42bc003530 con 0x7f42dc0a2580 2024-09-16T14:47:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3156452545 0 0) 0x7f42dc0a6340 con 0x7f42dc09fd40 2024-09-16T14:47:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 >> v1:172.21.15.26:6790/0 conn(0x7f42dc0a2580 legacy=0x7f42dc1d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 >> v1:172.21.15.26:6789/0 conn(0x7f42dc0a1a90 legacy=0x7f42dc1ce4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 1 monclient: found mon.b 2024-09-16T14:47:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42dc123320 con 0x7f42dc09fd40 2024-09-16T14:47:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:47:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 5 monclient: authenticate success, global_id 94990 2024-09-16T14:47:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 10 monclient: _renew_subs 2024-09-16T14:47:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 1 -- 172.21.15.189:0/3706910800 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42dc124010 con 0x7f42dc09fd40 2024-09-16T14:47:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 10 monclient: _renew_subs 2024-09-16T14:47:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 1 -- 172.21.15.189:0/3706910800 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42dc1c66f0 con 0x7f42dc09fd40 2024-09-16T14:47:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42cc004500 con 0x7f42dc09fd40 2024-09-16T14:47:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f42cc004890 con 0x7f42dc09fd40 2024-09-16T14:47:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 10 monclient: dump: 2024-09-16T14:47:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:34.454 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:34.454 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:34.454 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:34.454 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:34.454 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:34.454 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:34.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d959a640 4 set_mon_vals no callback set 2024-09-16T14:47:34.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:34.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:34.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:34.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f42cc004a70 con 0x7f42dc09fd40 2024-09-16T14:47:34.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42e0e96640 1 -- 172.21.15.189:0/3706910800 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c0005050 con 0x7f42dc09fd40 2024-09-16T14:47:34.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(126..126 src has 73..126) v4 ==== 6709+0+0 (unknown 3158201439 0 0) 0x7f42cc043e50 con 0x7f42dc09fd40 2024-09-16T14:47:34.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.434+0000 7f42d3fff640 10 monclient: get_auth_request con 0x7f42bc0360e0 auth_method 0 2024-09-16T14:47:34.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.438+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f42cc0148f0 con 0x7f42dc09fd40 2024-09-16T14:47:34.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.438+0000 7f42d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:34.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.438+0000 7f42d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:34.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.606+0000 7f42e0e96640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T14:47:34.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.606+0000 7f42e0e96640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T14:47:34.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.606+0000 7f42e0e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:34.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:34.606+0000 7f42e0e96640 1 -- 172.21.15.189:0/3706910800 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) v1 -- 0x7f42c0005880 con 0x7f42dc09fd40 2024-09-16T14:47:35.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.230+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}]=0 v127) v1 ==== 115+0+0 (unknown 2100378261 0 0) 0x7f42cc012d60 con 0x7f42dc09fd40 2024-09-16T14:47:35.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.230+0000 7f42d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T14:47:35.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.230+0000 7f42d2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:35.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.334+0000 7f42e0e96640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T14:47:35.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.334+0000 7f42e0e96640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T14:47:35.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.334+0000 7f42e0e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:35.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.334+0000 7f42e0e96640 1 -- 172.21.15.189:0/3706910800 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) v1 -- 0x7f42c0005300 con 0x7f42dc09fd40 2024-09-16T14:47:35.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.334+0000 7f42d2ffd640 1 -- 172.21.15.189:0/3706910800 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}]=0 weight-set for pool 'barset' already created v127) v1 ==== 159+0+0 (unknown 3428678846 0 0) 0x7f42cc0146c0 con 0x7f42dc09fd40 2024-09-16T14:47:35.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.334+0000 7f42d2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T14:47:35.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.334+0000 7f42d2ffd640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'barset' already created 2024-09-16T14:47:35.341 INFO:tasks.workunit.client.0.smithi189.stderr:weight-set for pool 'barset' already created 2024-09-16T14:47:35.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.338+0000 7f42e0e96640 1 -- 172.21.15.189:0/3706910800 >> v1:172.21.15.189:6812/3196161582 conn(0x7f42bc0360e0 legacy=0x7f42bc0385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:35.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.338+0000 7f42e0e96640 10 monclient: shutdown 2024-09-16T14:47:35.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.338+0000 7f42e0e96640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:35.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.338+0000 7f42e0e96640 1 -- 172.21.15.189:0/3706910800 >> v1:172.21.15.189:6789/0 conn(0x7f42dc09fd40 legacy=0x7f42dc133ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:35.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.338+0000 7f42e0e96640 1 -- 172.21.15.189:0/3706910800 shutdown_connections 2024-09-16T14:47:35.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.338+0000 7f42e0e96640 1 -- 172.21.15.189:0/3706910800 >> 172.21.15.189:0/3706910800 conn(0x7f42dc0a7a70 msgr2=0x7f42dc134a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:35.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.338+0000 7f42e0e96640 1 -- 172.21.15.189:0/3706910800 shutdown_connections 2024-09-16T14:47:35.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.338+0000 7f42e0e96640 1 -- 172.21.15.189:0/3706910800 wait complete. 2024-09-16T14:47:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set ls 2024-09-16T14:47:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep fooset 2024-09-16T14:47:35.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 10 monclient: get_monmap_and_config 2024-09-16T14:47:35.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 10 monclient: build_initial_monmap 2024-09-16T14:47:35.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 10 monclient: monmap: 2024-09-16T14:47:35.475 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:35.475 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:35.475 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:35.475323+0000 2024-09-16T14:47:35.475 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:35.475323+0000 2024-09-16T14:47:35.475 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:35.475 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:35.476 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:35.476 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:35.476 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:35.476 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:35.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 1 Processor -- start 2024-09-16T14:47:35.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 1 -- start start 2024-09-16T14:47:35.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 10 monclient: init 2024-09-16T14:47:35.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:35.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:35.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 10 monclient(hunting): picked mon.noname-a con 0x7fc0c00a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:35.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 10 monclient(hunting): picked mon.noname-b con 0x7fc0c00a25f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:35.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 10 monclient(hunting): picked mon.noname-c con 0x7fc0c013b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:35.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c00a30b0 con 0x7fc0c00a1b00 2024-09-16T14:47:35.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c00a2e30 con 0x7fc0c013b430 2024-09-16T14:47:35.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c009eb50 con 0x7fc0c00a25f0 2024-09-16T14:47:35.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:35.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 10 monclient(hunting): authenticate will time out at 1237.591253s 2024-09-16T14:47:35.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c5963640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc0c013b430 0x7fc0c013d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54736/0 (socket says 172.21.15.189:54736) 2024-09-16T14:47:35.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c5963640 1 -- 172.21.15.189:0/3830985086 learned_addr learned my addr 172.21.15.189:0/3830985086 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:35.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c5162640 1 -- 172.21.15.189:0/3830985086 >> v1:172.21.15.26:6789/0 conn(0x7fc0c00a1b00 legacy=0x7fc0c00a1ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:35.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 1 -- 172.21.15.189:0/3830985086 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902798322 0 0) 0x7fc0c00a2e30 con 0x7fc0c013b430 2024-09-16T14:47:35.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:35.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:35.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:35.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 10 monclient(hunting): my global_id is 94993 2024-09-16T14:47:35.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 1 -- 172.21.15.189:0/3830985086 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0ac003530 con 0x7fc0c013b430 2024-09-16T14:47:35.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 1 -- 172.21.15.189:0/3830985086 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1193494712 0 0) 0x7fc0ac003530 con 0x7fc0c013b430 2024-09-16T14:47:35.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 1 -- 172.21.15.189:0/3830985086 >> v1:172.21.15.26:6790/0 conn(0x7fc0c00a25f0 legacy=0x7fc0c0139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:35.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 1 -- 172.21.15.189:0/3830985086 >> v1:172.21.15.26:6789/0 conn(0x7fc0c00a1b00 legacy=0x7fc0c00a1ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:35.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:47:35.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 1 monclient: found mon.noname-c 2024-09-16T14:47:35.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:35.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:35.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 1 -- 172.21.15.189:0/3830985086 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c009ee30 con 0x7fc0c013b430 2024-09-16T14:47:35.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 10 monclient: _finish_auth 0 2024-09-16T14:47:35.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:47:35.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:35.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0c73c6640 5 monclient: authenticate success, global_id 94993 2024-09-16T14:47:35.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 1 -- 172.21.15.189:0/3830985086 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc0b80019a0 con 0x7fc0c013b430 2024-09-16T14:47:35.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.470+0000 7fc0b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:35.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:35.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b7fff640 10 monclient: dump: 2024-09-16T14:47:35.485 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:35.485 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:35.485 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:35.485 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:35.485 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:35.486 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:35.486 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:35.486 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:35.486 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:35.486 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:35.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:35.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b7fff640 1 -- 172.21.15.189:0/3830985086 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0b8001cc0 con 0x7fc0c013b430 2024-09-16T14:47:35.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:35.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:35.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b7fff640 1 -- 172.21.15.189:0/3830985086 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc0b8002050 con 0x7fc0c013b430 2024-09-16T14:47:35.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 4 set_mon_vals no callback set 2024-09-16T14:47:35.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:35.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:35.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b7fff640 10 monclient: dump: 2024-09-16T14:47:35.487 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:35.487 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:35.487 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:35.487 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:35.487 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:35.487 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:35.487 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:35.488 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:35.488 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:35.488 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:35.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient: shutdown 2024-09-16T14:47:35.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:35.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 1 -- 172.21.15.189:0/3830985086 >> v1:172.21.15.189:6789/0 conn(0x7fc0c013b430 legacy=0x7fc0c013d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:35.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 1 -- 172.21.15.189:0/3830985086 shutdown_connections 2024-09-16T14:47:35.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 1 -- 172.21.15.189:0/3830985086 >> 172.21.15.189:0/3830985086 conn(0x7fc0c00a7dd0 msgr2=0x7fc0c00a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:35.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 1 -- 172.21.15.189:0/3830985086 shutdown_connections 2024-09-16T14:47:35.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 1 -- 172.21.15.189:0/3830985086 wait complete. 2024-09-16T14:47:35.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient: build_initial_monmap 2024-09-16T14:47:35.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient: monmap: 2024-09-16T14:47:35.488 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:35.489 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:35.489 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:35.489 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:35.489 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:35.489 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:35.489 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:35.489 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:35.489 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:35.489 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:35.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 1 Processor -- start 2024-09-16T14:47:35.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 1 -- start start 2024-09-16T14:47:35.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient: init 2024-09-16T14:47:35.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient(hunting): picked mon.noname-b con 0x7fc0c00a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient(hunting): picked mon.noname-a con 0x7fc0c00a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient(hunting): picked mon.noname-c con 0x7fc0c013b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c00a5aa0 con 0x7fc0c00a25f0 2024-09-16T14:47:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c011db70 con 0x7fc0c00a1b00 2024-09-16T14:47:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c009e2f0 con 0x7fc0c013b430 2024-09-16T14:47:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient(hunting): authenticate will time out at 1237.594175s 2024-09-16T14:47:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c5162640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc0c00a1b00 0x7fc0c01399a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54748/0 (socket says 172.21.15.189:54748) 2024-09-16T14:47:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c5162640 1 -- 172.21.15.189:0/2023272820 learned_addr learned my addr 172.21.15.189:0/2023272820 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c4961640 1 -- 172.21.15.189:0/2023272820 >> v1:172.21.15.26:6789/0 conn(0x7fc0c00a25f0 legacy=0x7fc0c013a0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:35.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 1 -- 172.21.15.189:0/2023272820 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180591143 0 0) 0x7fc0c011db70 con 0x7fc0c00a1b00 2024-09-16T14:47:35.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:35.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:35.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:35.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 10 monclient(hunting): my global_id is 94996 2024-09-16T14:47:35.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 1 -- 172.21.15.189:0/2023272820 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a4003560 con 0x7fc0c00a1b00 2024-09-16T14:47:35.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 1 -- 172.21.15.189:0/2023272820 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2528634943 0 0) 0x7fc0a4003560 con 0x7fc0c00a1b00 2024-09-16T14:47:35.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 1 -- 172.21.15.189:0/2023272820 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0c011db70 con 0x7fc0c00a1b00 2024-09-16T14:47:35.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 1 -- 172.21.15.189:0/2023272820 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc0bc0040c0 con 0x7fc0c00a1b00 2024-09-16T14:47:35.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:35.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:35.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 10 monclient(hunting): dump: 2024-09-16T14:47:35.491 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:35.492 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:35.492 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:35.492 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:35.492 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:35.492 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:35.492 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:35.492 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:35.492 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:35.493 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 1 -- 172.21.15.189:0/2023272820 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2234728799 0 0) 0x7fc0c011db70 con 0x7fc0c00a1b00 2024-09-16T14:47:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 1 -- 172.21.15.189:0/2023272820 >> v1:172.21.15.26:6790/0 conn(0x7fc0c013b430 legacy=0x7fc0c01d1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 1 -- 172.21.15.189:0/2023272820 >> v1:172.21.15.26:6789/0 conn(0x7fc0c00a25f0 legacy=0x7fc0c013a0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:47:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 1 monclient: found mon.b 2024-09-16T14:47:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 1 -- 172.21.15.189:0/2023272820 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c013ae10 con 0x7fc0c00a1b00 2024-09-16T14:47:35.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 10 monclient: _finish_auth 0 2024-09-16T14:47:35.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:35.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:35.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 5 monclient: authenticate success, global_id 94996 2024-09-16T14:47:35.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient: _renew_subs 2024-09-16T14:47:35.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:35.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 1 -- 172.21.15.189:0/2023272820 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c011f660 con 0x7fc0c00a1b00 2024-09-16T14:47:35.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient: _renew_subs 2024-09-16T14:47:35.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:35.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 1 -- 172.21.15.189:0/2023272820 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c01357c0 con 0x7fc0c00a1b00 2024-09-16T14:47:35.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 1 -- 172.21.15.189:0/2023272820 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0bc0043e0 con 0x7fc0c00a1b00 2024-09-16T14:47:35.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:35.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 1 -- 172.21.15.189:0/2023272820 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc0bc004770 con 0x7fc0c00a1b00 2024-09-16T14:47:35.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:35.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:35.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b77fe640 4 set_mon_vals no callback set 2024-09-16T14:47:35.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 10 monclient: dump: 2024-09-16T14:47:35.496 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:35.496 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:35.496 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:35.496 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:35.496 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:35.496 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:35.496 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:35.497 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:35.497 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:35.497 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:35.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:35.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:35.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:35.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0c73c6640 1 -- 172.21.15.189:0/2023272820 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a0005050 con 0x7fc0c00a1b00 2024-09-16T14:47:35.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.474+0000 7fc0b5ffb640 1 -- 172.21.15.189:0/2023272820 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc0bc0048d0 con 0x7fc0c00a1b00 2024-09-16T14:47:35.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.478+0000 7fc0b5ffb640 1 -- 172.21.15.189:0/2023272820 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(127..127 src has 73..127) v4 ==== 7321+0+0 (unknown 2265503266 0 0) 0x7fc0bc043fa0 con 0x7fc0c00a1b00 2024-09-16T14:47:35.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.478+0000 7fc0c4961640 10 monclient: get_auth_request con 0x7fc0a4035f60 auth_method 0 2024-09-16T14:47:35.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.482+0000 7fc0b5ffb640 1 -- 172.21.15.189:0/2023272820 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc0bc0050a0 con 0x7fc0c00a1b00 2024-09-16T14:47:35.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.482+0000 7fc0b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:35.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.482+0000 7fc0b5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:35.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.642+0000 7fc0c73c6640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:35.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.642+0000 7fc0c73c6640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:35.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.642+0000 7fc0c73c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:35.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.642+0000 7fc0c73c6640 1 -- 172.21.15.189:0/2023272820 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fc0a0005880 con 0x7fc0c00a1b00 2024-09-16T14:47:35.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.646+0000 7fc0b5ffb640 1 -- 172.21.15.189:0/2023272820 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v127) v1 ==== 71+0+14 (unknown 4067154654 0 3003798303) 0x7fc0bc01cc90 con 0x7fc0c00a1b00 2024-09-16T14:47:35.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.646+0000 7fc0b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:35.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.646+0000 7fc0b5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:35.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.738+0000 7fc0c73c6640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:35.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.738+0000 7fc0c73c6640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:35.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.738+0000 7fc0c73c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:35.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.738+0000 7fc0c73c6640 1 -- 172.21.15.189:0/2023272820 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fc0a0005370 con 0x7fc0c00a1b00 2024-09-16T14:47:35.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.738+0000 7fc0b5ffb640 1 -- 172.21.15.189:0/2023272820 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v127) v1 ==== 71+0+14 (unknown 4067154654 0 3003798303) 0x7fc0bc012b10 con 0x7fc0c00a1b00 2024-09-16T14:47:35.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.738+0000 7fc0b5ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:35.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.738+0000 7fc0b5ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:35.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.742+0000 7fc0c73c6640 1 -- 172.21.15.189:0/2023272820 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc0a4035f60 legacy=0x7fc0a4038420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:35.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.742+0000 7fc0c73c6640 10 monclient: shutdown 2024-09-16T14:47:35.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.742+0000 7fc0c73c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:35.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.742+0000 7fc0c73c6640 1 -- 172.21.15.189:0/2023272820 >> v1:172.21.15.189:6789/0 conn(0x7fc0c00a1b00 legacy=0x7fc0c01399a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:35.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.742+0000 7fc0c73c6640 1 -- 172.21.15.189:0/2023272820 shutdown_connections 2024-09-16T14:47:35.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.742+0000 7fc0c73c6640 1 -- 172.21.15.189:0/2023272820 >> 172.21.15.189:0/2023272820 conn(0x7fc0c00a7dd0 msgr2=0x7fc0c009cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:35.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.742+0000 7fc0c73c6640 1 -- 172.21.15.189:0/2023272820 shutdown_connections 2024-09-16T14:47:35.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.742+0000 7fc0c73c6640 1 -- 172.21.15.189:0/2023272820 wait complete. 2024-09-16T14:47:35.760 INFO:tasks.workunit.client.0.smithi189.stdout:fooset 2024-09-16T14:47:35.761 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep barset 2024-09-16T14:47:35.761 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set ls 2024-09-16T14:47:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 10 monclient: get_monmap_and_config 2024-09-16T14:47:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 10 monclient: build_initial_monmap 2024-09-16T14:47:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 10 monclient: monmap: 2024-09-16T14:47:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:35.879538+0000 2024-09-16T14:47:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:35.879538+0000 2024-09-16T14:47:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:35.880 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:35.880 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:35.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 1 Processor -- start 2024-09-16T14:47:35.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 1 -- start start 2024-09-16T14:47:35.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 10 monclient: init 2024-09-16T14:47:35.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:35.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:35.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 10 monclient(hunting): picked mon.noname-b con 0x7f254009df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:35.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 10 monclient(hunting): picked mon.noname-a con 0x7f254009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:35.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 10 monclient(hunting): picked mon.noname-c con 0x7f254009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:35.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254009ee20 con 0x7f254009e2f0 2024-09-16T14:47:35.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254009f100 con 0x7f254009e740 2024-09-16T14:47:35.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254009f3e0 con 0x7f254009df10 2024-09-16T14:47:35.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:35.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 10 monclient(hunting): authenticate will time out at 1237.995169s 2024-09-16T14:47:35.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2544979640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f254009e2f0 legacy=0x7f25401378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:35.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f254597b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f254009e740 0x7f254013b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54754/0 (socket says 172.21.15.189:54754) 2024-09-16T14:47:35.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f254597b640 1 -- 172.21.15.189:0/1857291898 learned_addr learned my addr 172.21.15.189:0/1857291898 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:35.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 1 -- 172.21.15.189:0/1857291898 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336689464 0 0) 0x7f254009f100 con 0x7f254009e740 2024-09-16T14:47:35.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:35.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:35.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:35.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 10 monclient(hunting): my global_id is 94999 2024-09-16T14:47:35.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 1 -- 172.21.15.189:0/1857291898 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f252c003530 con 0x7f254009e740 2024-09-16T14:47:35.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 1 -- 172.21.15.189:0/1857291898 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2952100494 0 0) 0x7f252c003530 con 0x7f254009e740 2024-09-16T14:47:35.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 1 -- 172.21.15.189:0/1857291898 >> v1:172.21.15.26:6790/0 conn(0x7f254009df10 legacy=0x7f254009d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:35.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 1 -- 172.21.15.189:0/1857291898 >> v1:172.21.15.26:6789/0 conn(0x7f254009e2f0 legacy=0x7f25401378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:35.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 10 monclient: _finish_hunting 0 2024-09-16T14:47:35.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 1 monclient: found mon.noname-c 2024-09-16T14:47:35.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:35.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:35.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 1 -- 172.21.15.189:0/1857291898 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254009c690 con 0x7f254009e740 2024-09-16T14:47:35.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 10 monclient: _finish_auth 0 2024-09-16T14:47:35.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 10 monclient: _check_auth_tickets 2024-09-16T14:47:35.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:35.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 5 monclient: authenticate success, global_id 94999 2024-09-16T14:47:35.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 1 -- 172.21.15.189:0/1857291898 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f25380019a0 con 0x7f254009e740 2024-09-16T14:47:35.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:35.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:35.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:35.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 10 monclient: dump: 2024-09-16T14:47:35.888 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:35.888 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:35.889 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:35.889 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:35.889 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:35.889 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:35.889 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:35.889 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:35.889 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:35.889 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:35.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:35.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 1 -- 172.21.15.189:0/1857291898 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2538001cc0 con 0x7f254009e740 2024-09-16T14:47:35.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:35.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:35.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f25473de640 4 set_mon_vals no callback set 2024-09-16T14:47:35.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.874+0000 7f2537fff640 1 -- 172.21.15.189:0/1857291898 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2538002090 con 0x7f254009e740 2024-09-16T14:47:35.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2537fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:35.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2537fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:35.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2537fff640 10 monclient: dump: 2024-09-16T14:47:35.890 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:35.890 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:35.890 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:35.890 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:35.890 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:35.890 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:35.891 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:35.891 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:35.891 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:35.891 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:35.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient: shutdown 2024-09-16T14:47:35.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:35.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 1 -- 172.21.15.189:0/1857291898 >> v1:172.21.15.189:6789/0 conn(0x7f254009e740 legacy=0x7f254013b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:35.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 1 -- 172.21.15.189:0/1857291898 shutdown_connections 2024-09-16T14:47:35.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 1 -- 172.21.15.189:0/1857291898 >> 172.21.15.189:0/1857291898 conn(0x7f25400a3320 msgr2=0x7f254009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:35.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 1 -- 172.21.15.189:0/1857291898 shutdown_connections 2024-09-16T14:47:35.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 1 -- 172.21.15.189:0/1857291898 wait complete. 2024-09-16T14:47:35.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient: build_initial_monmap 2024-09-16T14:47:35.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient: monmap: 2024-09-16T14:47:35.893 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:35.893 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:35.893 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:35.894 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:35.894 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:35.894 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:35.894 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:35.894 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:35.894 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:35.895 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:35.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 1 Processor -- start 2024-09-16T14:47:35.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 1 -- start start 2024-09-16T14:47:35.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient: init 2024-09-16T14:47:35.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:35.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:35.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient(hunting): picked mon.noname-a con 0x7f254009df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:35.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient(hunting): picked mon.noname-c con 0x7f254009e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:35.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient(hunting): picked mon.noname-b con 0x7f254009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:35.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254011a660 con 0x7f254009df10 2024-09-16T14:47:35.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254011a0a0 con 0x7f254009e740 2024-09-16T14:47:35.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254011a380 con 0x7f254009e2f0 2024-09-16T14:47:35.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:35.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient(hunting): authenticate will time out at 1237.998033s 2024-09-16T14:47:35.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f254597b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f254009e740 0x7f25401cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54758/0 (socket says 172.21.15.189:54758) 2024-09-16T14:47:35.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f254597b640 1 -- 172.21.15.189:0/3097662633 learned_addr learned my addr 172.21.15.189:0/3097662633 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:35.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f254517a640 1 -- 172.21.15.189:0/3097662633 >> v1:172.21.15.26:6789/0 conn(0x7f254009df10 legacy=0x7f2540139c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:35.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 1 -- 172.21.15.189:0/3097662633 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644193252 0 0) 0x7f254011a0a0 con 0x7f254009e740 2024-09-16T14:47:35.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:35.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:35.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:35.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 10 monclient(hunting): my global_id is 95002 2024-09-16T14:47:35.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 1 -- 172.21.15.189:0/3097662633 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2524003530 con 0x7f254009e740 2024-09-16T14:47:35.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 1 -- 172.21.15.189:0/3097662633 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 650071831 0 0) 0x7f2524003530 con 0x7f254009e740 2024-09-16T14:47:35.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 1 -- 172.21.15.189:0/3097662633 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f254011a0a0 con 0x7f254009e740 2024-09-16T14:47:35.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 1 -- 172.21.15.189:0/3097662633 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2538001960 con 0x7f254009e740 2024-09-16T14:47:35.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:35.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:35.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 10 monclient(hunting): dump: 2024-09-16T14:47:35.899 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:35.900 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:35.900 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:35.900 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:35.900 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:35.900 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:35.900 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:35.900 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:35.901 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:35.901 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:35.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:35.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 1 -- 172.21.15.189:0/3097662633 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1844162772 0 0) 0x7f254011a0a0 con 0x7f254009e740 2024-09-16T14:47:35.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 1 -- 172.21.15.189:0/3097662633 >> v1:172.21.15.26:6790/0 conn(0x7f254009e2f0 legacy=0x7f254013a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:35.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 1 -- 172.21.15.189:0/3097662633 >> v1:172.21.15.26:6789/0 conn(0x7f254009df10 legacy=0x7f2540139c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:35.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:47:35.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 1 monclient: found mon.b 2024-09-16T14:47:35.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:35.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:35.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 1 -- 172.21.15.189:0/3097662633 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254006f930 con 0x7f254009e740 2024-09-16T14:47:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 10 monclient: _finish_auth 0 2024-09-16T14:47:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 5 monclient: authenticate success, global_id 95002 2024-09-16T14:47:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient: _renew_subs 2024-09-16T14:47:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 1 -- 172.21.15.189:0/3097662633 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254013bb70 con 0x7f254009e740 2024-09-16T14:47:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient: _renew_subs 2024-09-16T14:47:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 1 -- 172.21.15.189:0/3097662633 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f254013bfe0 con 0x7f254009e740 2024-09-16T14:47:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 1 -- 172.21.15.189:0/3097662633 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2538004f80 con 0x7f254009e740 2024-09-16T14:47:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 1 -- 172.21.15.189:0/3097662633 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2538005290 con 0x7f254009e740 2024-09-16T14:47:35.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:35.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:35.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 10 monclient: dump: 2024-09-16T14:47:35.903 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:35.903 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:35.903 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:35.903 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:35.903 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:35.903 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:35.903 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:35.903 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:35.903 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:35.903 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:35.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25377fe640 4 set_mon_vals no callback set 2024-09-16T14:47:35.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:35.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f2535ffb640 1 -- 172.21.15.189:0/3097662633 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f253800fd00 con 0x7f254009e740 2024-09-16T14:47:35.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:35.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:35.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.878+0000 7f25473de640 1 -- 172.21.15.189:0/3097662633 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2520005050 con 0x7f254009e740 2024-09-16T14:47:35.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.882+0000 7f2535ffb640 1 -- 172.21.15.189:0/3097662633 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(127..127 src has 73..127) v4 ==== 7321+0+0 (unknown 2265503266 0 0) 0x7f2538044480 con 0x7f254009e740 2024-09-16T14:47:35.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.882+0000 7f254517a640 10 monclient: get_auth_request con 0x7f2524036030 auth_method 0 2024-09-16T14:47:35.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.886+0000 7f2535ffb640 1 -- 172.21.15.189:0/3097662633 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2538014a90 con 0x7f254009e740 2024-09-16T14:47:35.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.886+0000 7f2535ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:35.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:35.886+0000 7f2535ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:36.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.050+0000 7f25473de640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.050+0000 7f25473de640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.050+0000 7f25473de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.050+0000 7f25473de640 1 -- 172.21.15.189:0/3097662633 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f2520005300 con 0x7f254009e740 2024-09-16T14:47:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.050+0000 7f2535ffb640 1 -- 172.21.15.189:0/3097662633 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v127) v1 ==== 71+0+14 (unknown 4067154654 0 3003798303) 0x7f2538013130 con 0x7f254009e740 2024-09-16T14:47:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.050+0000 7f2535ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:36.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.050+0000 7f2535ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:36.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.150+0000 7f25473de640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:36.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.150+0000 7f25473de640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:36.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.150+0000 7f25473de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:36.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.150+0000 7f25473de640 1 -- 172.21.15.189:0/3097662633 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f25200055a0 con 0x7f254009e740 2024-09-16T14:47:36.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.154+0000 7f2535ffb640 1 -- 172.21.15.189:0/3097662633 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v127) v1 ==== 71+0+14 (unknown 4067154654 0 3003798303) 0x7f2538013130 con 0x7f254009e740 2024-09-16T14:47:36.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.154+0000 7f2535ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:36.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.154+0000 7f2535ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:36.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.154+0000 7f25473de640 1 -- 172.21.15.189:0/3097662633 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2524036030 legacy=0x7f25240384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:36.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.154+0000 7f25473de640 10 monclient: shutdown 2024-09-16T14:47:36.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.154+0000 7f25473de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:36.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.154+0000 7f25473de640 1 -- 172.21.15.189:0/3097662633 >> v1:172.21.15.189:6789/0 conn(0x7f254009e740 legacy=0x7f25401cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:36.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.154+0000 7f25473de640 1 -- 172.21.15.189:0/3097662633 shutdown_connections 2024-09-16T14:47:36.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.154+0000 7f25473de640 1 -- 172.21.15.189:0/3097662633 >> 172.21.15.189:0/3097662633 conn(0x7f25400a3320 msgr2=0x7f25400a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:36.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.154+0000 7f25473de640 1 -- 172.21.15.189:0/3097662633 shutdown_connections 2024-09-16T14:47:36.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.154+0000 7f25473de640 1 -- 172.21.15.189:0/3097662633 wait complete. 2024-09-16T14:47:36.174 INFO:tasks.workunit.client.0.smithi189.stdout:barset 2024-09-16T14:47:36.174 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set dump 2024-09-16T14:47:36.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.286+0000 7f85861ab640 10 monclient: get_monmap_and_config 2024-09-16T14:47:36.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.286+0000 7f85861ab640 10 monclient: build_initial_monmap 2024-09-16T14:47:36.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.286+0000 7f85861ab640 10 monclient: monmap: 2024-09-16T14:47:36.292 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:36.292 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:36.292 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:36.291947+0000 2024-09-16T14:47:36.292 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:36.291947+0000 2024-09-16T14:47:36.292 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:36.292 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:36.292 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:36.292 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:36.292 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:36.292 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:36.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.286+0000 7f85861ab640 1 Processor -- start 2024-09-16T14:47:36.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.286+0000 7f85861ab640 1 -- start start 2024-09-16T14:47:36.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.286+0000 7f85861ab640 10 monclient: init 2024-09-16T14:47:36.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:36.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:36.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient(hunting): picked mon.noname-b con 0x7f858009df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:36.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient(hunting): picked mon.noname-a con 0x7f858009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:36.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient(hunting): picked mon.noname-c con 0x7f858009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:36.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858009ee20 con 0x7f858009e2f0 2024-09-16T14:47:36.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858009f100 con 0x7f858009e740 2024-09-16T14:47:36.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858009f3e0 con 0x7f858009df10 2024-09-16T14:47:36.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:36.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient(hunting): authenticate will time out at 1238.408204s 2024-09-16T14:47:36.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857ffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f858009e740 0x7f858013b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54774/0 (socket says 172.21.15.189:54774) 2024-09-16T14:47:36.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857ffff640 1 -- 172.21.15.189:0/2953792855 learned_addr learned my addr 172.21.15.189:0/2953792855 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:36.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857effd640 1 -- 172.21.15.189:0/2953792855 >> v1:172.21.15.26:6789/0 conn(0x7f858009e2f0 legacy=0x7f85801378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:36.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 1 -- 172.21.15.189:0/2953792855 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15273507 0 0) 0x7f858009f100 con 0x7f858009e740 2024-09-16T14:47:36.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:36.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:36.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:36.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 10 monclient(hunting): my global_id is 95005 2024-09-16T14:47:36.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 1 -- 172.21.15.189:0/2953792855 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8568003530 con 0x7f858009e740 2024-09-16T14:47:36.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 1 -- 172.21.15.189:0/2953792855 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2321325458 0 0) 0x7f8568003530 con 0x7f858009e740 2024-09-16T14:47:36.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 1 -- 172.21.15.189:0/2953792855 >> v1:172.21.15.26:6790/0 conn(0x7f858009df10 legacy=0x7f858009d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:36.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 1 -- 172.21.15.189:0/2953792855 >> v1:172.21.15.26:6789/0 conn(0x7f858009e2f0 legacy=0x7f85801378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:36.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:36.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 1 monclient: found mon.noname-c 2024-09-16T14:47:36.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 1 -- 172.21.15.189:0/2953792855 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858009c690 con 0x7f858009e740 2024-09-16T14:47:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 5 monclient: authenticate success, global_id 95005 2024-09-16T14:47:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 1 -- 172.21.15.189:0/2953792855 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f85780019a0 con 0x7f858009e740 2024-09-16T14:47:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 10 monclient: dump: 2024-09-16T14:47:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:36.303 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 1 -- 172.21.15.189:0/2953792855 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8578001cc0 con 0x7f858009e740 2024-09-16T14:47:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 1 -- 172.21.15.189:0/2953792855 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8578002050 con 0x7f858009e740 2024-09-16T14:47:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:36.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 4 set_mon_vals no callback set 2024-09-16T14:47:36.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:36.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:36.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857e7fc640 10 monclient: dump: 2024-09-16T14:47:36.304 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:36.304 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:36.304 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:36.304 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:36.304 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:36.304 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:36.304 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:36.304 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:36.304 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:36.305 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:36.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient: shutdown 2024-09-16T14:47:36.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:36.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 1 -- 172.21.15.189:0/2953792855 >> v1:172.21.15.189:6789/0 conn(0x7f858009e740 legacy=0x7f858013b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:36.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 1 -- 172.21.15.189:0/2953792855 shutdown_connections 2024-09-16T14:47:36.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 1 -- 172.21.15.189:0/2953792855 >> 172.21.15.189:0/2953792855 conn(0x7f85800a3320 msgr2=0x7f858009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:36.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 1 -- 172.21.15.189:0/2953792855 shutdown_connections 2024-09-16T14:47:36.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 1 -- 172.21.15.189:0/2953792855 wait complete. 2024-09-16T14:47:36.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient: build_initial_monmap 2024-09-16T14:47:36.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient: monmap: 2024-09-16T14:47:36.305 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:36.305 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:36.305 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:36.306 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:36.306 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:36.306 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:36.306 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:36.306 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:36.306 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:36.306 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:36.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 1 Processor -- start 2024-09-16T14:47:36.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 1 -- start start 2024-09-16T14:47:36.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient: init 2024-09-16T14:47:36.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:36.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:36.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient(hunting): picked mon.noname-b con 0x7f858009df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:36.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient(hunting): picked mon.noname-c con 0x7f858009e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:36.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient(hunting): picked mon.noname-a con 0x7f858009e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:36.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858011a660 con 0x7f858009e740 2024-09-16T14:47:36.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858011a0a0 con 0x7f858009df10 2024-09-16T14:47:36.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858011a380 con 0x7f858009e2f0 2024-09-16T14:47:36.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:36.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient(hunting): authenticate will time out at 1238.410904s 2024-09-16T14:47:36.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857f7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f858009df10 0x7f8580139c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54788/0 (socket says 172.21.15.189:54788) 2024-09-16T14:47:36.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857f7fe640 1 -- 172.21.15.189:0/3935045177 learned_addr learned my addr 172.21.15.189:0/3935045177 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:36.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857ffff640 1 -- 172.21.15.189:0/3935045177 >> v1:172.21.15.26:6789/0 conn(0x7f858009e740 legacy=0x7f85801cf780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:36.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 1 -- 172.21.15.189:0/3935045177 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999779074 0 0) 0x7f858011a0a0 con 0x7f858009df10 2024-09-16T14:47:36.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:36.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:36.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:36.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 10 monclient(hunting): my global_id is 95008 2024-09-16T14:47:36.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 1 -- 172.21.15.189:0/3935045177 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f856c003530 con 0x7f858009df10 2024-09-16T14:47:36.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 1 -- 172.21.15.189:0/3935045177 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 237252200 0 0) 0x7f856c003530 con 0x7f858009df10 2024-09-16T14:47:36.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 1 -- 172.21.15.189:0/3935045177 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f858011a0a0 con 0x7f858009df10 2024-09-16T14:47:36.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 1 -- 172.21.15.189:0/3935045177 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8570001980 con 0x7f858009df10 2024-09-16T14:47:36.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:36.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:36.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 10 monclient(hunting): dump: 2024-09-16T14:47:36.309 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:36.309 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:36.309 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:36.309 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:36.309 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:36.309 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:36.309 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:36.309 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:36.310 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:36.310 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:36.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:36.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 1 -- 172.21.15.189:0/3935045177 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2748421955 0 0) 0x7f858011a0a0 con 0x7f858009df10 2024-09-16T14:47:36.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 1 -- 172.21.15.189:0/3935045177 >> v1:172.21.15.26:6790/0 conn(0x7f858009e2f0 legacy=0x7f858013a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:36.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 1 -- 172.21.15.189:0/3935045177 >> v1:172.21.15.26:6789/0 conn(0x7f858009e740 legacy=0x7f85801cf780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:36.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 10 monclient: _finish_hunting 0 2024-09-16T14:47:36.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 1 monclient: found mon.b 2024-09-16T14:47:36.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:36.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:36.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 1 -- 172.21.15.189:0/3935045177 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858006f930 con 0x7f858009df10 2024-09-16T14:47:36.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 10 monclient: _finish_auth 0 2024-09-16T14:47:36.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 10 monclient: _check_auth_tickets 2024-09-16T14:47:36.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:36.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 5 monclient: authenticate success, global_id 95008 2024-09-16T14:47:36.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient: _renew_subs 2024-09-16T14:47:36.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:36.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 1 -- 172.21.15.189:0/3935045177 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f858013bb70 con 0x7f858009df10 2024-09-16T14:47:36.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient: _renew_subs 2024-09-16T14:47:36.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:36.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f85861ab640 1 -- 172.21.15.189:0/3935045177 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f858013bfe0 con 0x7f858009df10 2024-09-16T14:47:36.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 1 -- 172.21.15.189:0/3935045177 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8570001d50 con 0x7f858009df10 2024-09-16T14:47:36.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:36.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 1 -- 172.21.15.189:0/3935045177 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8570003a70 con 0x7f858009df10 2024-09-16T14:47:36.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:36.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:36.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f857dffb640 4 set_mon_vals no callback set 2024-09-16T14:47:36.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.290+0000 7f8567fff640 10 monclient: dump: 2024-09-16T14:47:36.312 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:36.312 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:36.312 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:36.312 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:36.312 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:36.313 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:36.313 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:36.313 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:36.313 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:36.313 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:36.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.294+0000 7f85861ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:36.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.294+0000 7f85861ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:36.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.294+0000 7f85861ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:36.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.294+0000 7f8567fff640 1 -- 172.21.15.189:0/3935045177 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8570003bd0 con 0x7f858009df10 2024-09-16T14:47:36.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.294+0000 7f85861ab640 1 -- 172.21.15.189:0/3935045177 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8560005050 con 0x7f858009df10 2024-09-16T14:47:36.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.294+0000 7f8567fff640 1 -- 172.21.15.189:0/3935045177 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(127..127 src has 73..127) v4 ==== 7321+0+0 (unknown 2265503266 0 0) 0x7f85700440f0 con 0x7f858009df10 2024-09-16T14:47:36.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.294+0000 7f857effd640 10 monclient: get_auth_request con 0x7f856c036050 auth_method 0 2024-09-16T14:47:36.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.298+0000 7f8567fff640 1 -- 172.21.15.189:0/3935045177 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8570014790 con 0x7f858009df10 2024-09-16T14:47:36.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.298+0000 7f8567fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:36.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.298+0000 7f8567fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:36.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.462+0000 7f85861ab640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-16T14:47:36.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.462+0000 7f85861ab640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T14:47:36.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.462+0000 7f85861ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:36.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.462+0000 7f85861ab640 1 -- 172.21.15.189:0/3935045177 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7f8560005880 con 0x7f858009df10 2024-09-16T14:47:36.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.462+0000 7f8567fff640 1 -- 172.21.15.189:0/3935045177 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v127) v1 ==== 73+0+6748 (unknown 2006088562 0 3407865348) 0x7f8570012c00 con 0x7f858009df10 2024-09-16T14:47:36.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.462+0000 7f8567fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T14:47:36.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.462+0000 7f8567fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:36.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.558+0000 7f85861ab640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-16T14:47:36.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.558+0000 7f85861ab640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T14:47:36.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.558+0000 7f85861ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:36.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.558+0000 7f85861ab640 1 -- 172.21.15.189:0/3935045177 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7f85600053a0 con 0x7f858009df10 2024-09-16T14:47:36.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.558+0000 7f8567fff640 1 -- 172.21.15.189:0/3935045177 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v127) v1 ==== 73+0+6748 (unknown 2006088562 0 3407865348) 0x7f8570014380 con 0x7f858009df10 2024-09-16T14:47:36.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.558+0000 7f8567fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T14:47:36.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.558+0000 7f8567fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:36.563 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2024-09-16T14:47:36.563 INFO:tasks.workunit.client.0.smithi189.stdout: "9": [ 2024-09-16T14:47:36.563 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.563 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -1, 2024-09-16T14:47:36.563 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.564 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.564 INFO:tasks.workunit.client.0.smithi189.stdout: 0.263671875, 2024-09-16T14:47:36.564 INFO:tasks.workunit.client.0.smithi189.stdout: 0.263671875, 2024-09-16T14:47:36.564 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.564 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T14:47:36.564 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.564 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.564 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.564 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.564 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -2, 2024-09-16T14:47:36.564 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.564 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.565 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T14:47:36.565 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.565 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.565 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.565 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.565 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -3, 2024-09-16T14:47:36.565 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.565 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.565 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.565 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.565 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T14:47:36.565 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.566 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.566 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.566 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.566 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -4, 2024-09-16T14:47:36.566 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.566 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T14:47:36.566 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.566 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.566 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.566 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -5, 2024-09-16T14:47:36.566 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.566 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.566 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.567 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.567 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T14:47:36.567 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.567 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.567 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.567 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.567 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -6, 2024-09-16T14:47:36.567 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.567 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.567 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.567 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.567 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.567 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T14:47:36.568 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.568 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.568 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.568 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.568 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -7, 2024-09-16T14:47:36.568 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.568 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.568 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T14:47:36.568 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.568 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.568 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.568 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.568 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -8, 2024-09-16T14:47:36.569 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.569 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T14:47:36.569 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.569 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.569 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.569 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -9, 2024-09-16T14:47:36.569 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.569 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.569 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.569 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.569 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.569 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T14:47:36.570 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.570 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.570 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.570 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.570 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -10, 2024-09-16T14:47:36.570 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.570 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T14:47:36.570 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.570 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.570 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.570 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -11, 2024-09-16T14:47:36.570 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.570 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T14:47:36.570 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.571 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.571 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.571 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -12, 2024-09-16T14:47:36.572 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.572 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T14:47:36.572 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.572 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.572 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.572 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -13, 2024-09-16T14:47:36.572 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.572 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T14:47:36.572 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.572 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.572 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.573 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -14, 2024-09-16T14:47:36.573 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.573 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T14:47:36.573 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.573 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.573 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.573 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -15, 2024-09-16T14:47:36.573 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.573 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T14:47:36.574 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.574 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:47:36.574 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:47:36.574 INFO:tasks.workunit.client.0.smithi189.stdout: "10": [ 2024-09-16T14:47:36.574 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.574 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -1, 2024-09-16T14:47:36.574 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.574 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.574 INFO:tasks.workunit.client.0.smithi189.stdout: 0.263671875, 2024-09-16T14:47:36.574 INFO:tasks.workunit.client.0.smithi189.stdout: 0.263671875, 2024-09-16T14:47:36.574 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.574 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T14:47:36.574 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:47:36.574 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.575 INFO:tasks.workunit.client.0.smithi189.stdout: 0.263671875, 2024-09-16T14:47:36.575 INFO:tasks.workunit.client.0.smithi189.stdout: 0.263671875, 2024-09-16T14:47:36.575 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.575 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T14:47:36.575 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:47:36.575 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.575 INFO:tasks.workunit.client.0.smithi189.stdout: 0.263671875, 2024-09-16T14:47:36.575 INFO:tasks.workunit.client.0.smithi189.stdout: 0.263671875, 2024-09-16T14:47:36.575 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.575 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T14:47:36.575 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.575 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.575 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.576 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.576 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -2, 2024-09-16T14:47:36.576 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.576 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.576 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T14:47:36.576 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:47:36.576 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.576 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T14:47:36.576 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:47:36.576 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.576 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T14:47:36.576 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.576 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.577 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.577 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.577 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -3, 2024-09-16T14:47:36.577 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.577 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.577 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.577 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.577 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T14:47:36.577 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:47:36.577 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.577 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.577 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.577 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T14:47:36.577 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:47:36.578 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.578 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.578 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.578 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T14:47:36.578 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.578 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.578 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.578 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.578 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -4, 2024-09-16T14:47:36.578 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.578 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T14:47:36.578 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T14:47:36.578 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T14:47:36.579 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.579 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.579 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.579 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -5, 2024-09-16T14:47:36.579 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.579 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.579 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.579 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.579 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T14:47:36.579 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:47:36.579 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.579 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.579 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.579 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T14:47:36.580 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:47:36.580 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.580 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.580 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.580 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T14:47:36.580 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.580 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.580 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.580 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.580 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -6, 2024-09-16T14:47:36.580 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.580 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.580 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.581 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.581 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.581 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T14:47:36.581 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:47:36.581 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.581 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.581 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.581 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.581 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T14:47:36.581 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:47:36.581 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.581 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.581 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.581 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.582 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T14:47:36.582 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.582 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.582 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.582 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.582 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -7, 2024-09-16T14:47:36.582 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.582 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.582 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T14:47:36.582 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:47:36.582 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.582 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T14:47:36.582 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:47:36.582 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.583 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T14:47:36.583 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.583 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.583 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.583 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.583 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -8, 2024-09-16T14:47:36.583 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.583 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T14:47:36.583 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T14:47:36.583 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T14:47:36.583 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.583 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.583 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.583 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -9, 2024-09-16T14:47:36.584 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.584 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.584 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.584 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.584 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.584 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T14:47:36.584 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:47:36.584 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.584 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.584 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.584 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.584 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T14:47:36.584 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T14:47:36.585 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T14:47:36.585 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T14:47:36.585 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.585 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T14:47:36.585 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T14:47:36.585 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.585 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.585 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.585 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.585 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -10, 2024-09-16T14:47:36.585 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.585 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T14:47:36.585 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T14:47:36.585 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T14:47:36.586 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.586 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.586 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.586 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -11, 2024-09-16T14:47:36.586 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.586 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T14:47:36.586 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T14:47:36.586 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T14:47:36.586 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.586 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.586 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.586 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -12, 2024-09-16T14:47:36.586 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.586 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T14:47:36.587 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T14:47:36.587 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T14:47:36.587 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.587 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.587 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.587 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -13, 2024-09-16T14:47:36.587 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.587 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T14:47:36.587 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T14:47:36.587 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T14:47:36.587 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.587 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.587 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.588 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -14, 2024-09-16T14:47:36.588 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.588 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T14:47:36.588 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T14:47:36.588 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T14:47:36.588 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.588 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T14:47:36.588 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T14:47:36.588 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -15, 2024-09-16T14:47:36.588 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T14:47:36.588 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T14:47:36.588 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T14:47:36.588 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T14:47:36.588 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.589 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T14:47:36.589 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T14:47:36.589 INFO:tasks.workunit.client.0.smithi189.stdout:} 2024-09-16T14:47:36.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.558+0000 7f85861ab640 1 -- 172.21.15.189:0/3935045177 >> v1:172.21.15.189:6812/3196161582 conn(0x7f856c036050 legacy=0x7f856c038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:36.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.558+0000 7f85861ab640 10 monclient: shutdown 2024-09-16T14:47:36.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.558+0000 7f85861ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:36.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.558+0000 7f85861ab640 1 -- 172.21.15.189:0/3935045177 >> v1:172.21.15.189:6789/0 conn(0x7f858009df10 legacy=0x7f8580139c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:36.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.558+0000 7f85861ab640 1 -- 172.21.15.189:0/3935045177 shutdown_connections 2024-09-16T14:47:36.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.558+0000 7f85861ab640 1 -- 172.21.15.189:0/3935045177 >> 172.21.15.189:0/3935045177 conn(0x7f85800a3320 msgr2=0x7f85800a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:36.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.558+0000 7f85861ab640 1 -- 172.21.15.189:0/3935045177 shutdown_connections 2024-09-16T14:47:36.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.558+0000 7f85861ab640 1 -- 172.21.15.189:0/3935045177 wait complete. 2024-09-16T14:47:36.591 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T14:47:36.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.690+0000 7f845bd90640 10 monclient: get_monmap_and_config 2024-09-16T14:47:36.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.690+0000 7f845bd90640 10 monclient: build_initial_monmap 2024-09-16T14:47:36.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.690+0000 7f845bd90640 10 monclient: monmap: 2024-09-16T14:47:36.696 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:36.696 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:36.696 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:36.696233+0000 2024-09-16T14:47:36.696 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:36.696233+0000 2024-09-16T14:47:36.696 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:36.696 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:36.697 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:36.697 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:36.697 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:36.697 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:36.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.690+0000 7f845bd90640 1 Processor -- start 2024-09-16T14:47:36.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.690+0000 7f845bd90640 1 -- start start 2024-09-16T14:47:36.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.690+0000 7f845bd90640 10 monclient: init 2024-09-16T14:47:36.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:36.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:36.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient(hunting): picked mon.noname-c con 0x7f8454131630 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:36.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient(hunting): picked mon.noname-a con 0x7f845409edd0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:36.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient(hunting): picked mon.noname-b con 0x7f845409e950 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:36.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845409b920 con 0x7f845409edd0 2024-09-16T14:47:36.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845409bc00 con 0x7f8454131630 2024-09-16T14:47:36.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845409cba0 con 0x7f845409e950 2024-09-16T14:47:36.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:36.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient(hunting): authenticate will time out at 1238.812630s 2024-09-16T14:47:36.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8459b2c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8454131630 0x7f8454131a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54796/0 (socket says 172.21.15.189:54796) 2024-09-16T14:47:36.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8459b2c640 1 -- 172.21.15.189:0/2835212179 learned_addr learned my addr 172.21.15.189:0/2835212179 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:36.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845932b640 1 -- 172.21.15.189:0/2835212179 >> v1:172.21.15.26:6789/0 conn(0x7f845409edd0 legacy=0x7f8454136b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:36.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 1 -- 172.21.15.189:0/2835212179 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776689682 0 0) 0x7f845409bc00 con 0x7f8454131630 2024-09-16T14:47:36.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:36.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:36.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:36.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 10 monclient(hunting): my global_id is 95011 2024-09-16T14:47:36.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 1 -- 172.21.15.189:0/2835212179 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8440003530 con 0x7f8454131630 2024-09-16T14:47:36.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 1 -- 172.21.15.189:0/2835212179 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2924953152 0 0) 0x7f8440003530 con 0x7f8454131630 2024-09-16T14:47:36.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 1 -- 172.21.15.189:0/2835212179 >> v1:172.21.15.26:6790/0 conn(0x7f845409e950 legacy=0x7f845413a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:36.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 1 -- 172.21.15.189:0/2835212179 >> v1:172.21.15.26:6789/0 conn(0x7f845409edd0 legacy=0x7f8454136b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:36.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 10 monclient: _finish_hunting 0 2024-09-16T14:47:36.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 1 monclient: found mon.noname-c 2024-09-16T14:47:36.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:36.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:36.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 1 -- 172.21.15.189:0/2835212179 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845409b420 con 0x7f8454131630 2024-09-16T14:47:36.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 10 monclient: _finish_auth 0 2024-09-16T14:47:36.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 10 monclient: _check_auth_tickets 2024-09-16T14:47:36.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:36.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 1 -- 172.21.15.189:0/2835212179 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8450001620 con 0x7f8454131630 2024-09-16T14:47:36.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:36.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:36.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 10 monclient: dump: 2024-09-16T14:47:36.708 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:36.708 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:36.708 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:36.708 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:36.708 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:36.708 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:36.708 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:36.708 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:36.708 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:36.708 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:36.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 5 monclient: authenticate success, global_id 95011 2024-09-16T14:47:36.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:36.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 1 -- 172.21.15.189:0/2835212179 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8450001940 con 0x7f8454131630 2024-09-16T14:47:36.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:36.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 1 -- 172.21.15.189:0/2835212179 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8450001d10 con 0x7f8454131630 2024-09-16T14:47:36.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:36.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:36.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f8458b2a640 10 monclient: dump: 2024-09-16T14:47:36.709 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:36.709 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:36.709 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:36.710 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:36.710 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:36.710 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:36.710 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:36.710 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:36.710 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:36.710 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:36.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:36.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 4 set_mon_vals no callback set 2024-09-16T14:47:36.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient: shutdown 2024-09-16T14:47:36.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:36.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 1 -- 172.21.15.189:0/2835212179 >> v1:172.21.15.189:6789/0 conn(0x7f8454131630 legacy=0x7f8454131a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:36.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 1 -- 172.21.15.189:0/2835212179 shutdown_connections 2024-09-16T14:47:36.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 1 -- 172.21.15.189:0/2835212179 >> 172.21.15.189:0/2835212179 conn(0x7f84540a3380 msgr2=0x7f845409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:36.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 1 -- 172.21.15.189:0/2835212179 shutdown_connections 2024-09-16T14:47:36.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 1 -- 172.21.15.189:0/2835212179 wait complete. 2024-09-16T14:47:36.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient: build_initial_monmap 2024-09-16T14:47:36.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient: monmap: 2024-09-16T14:47:36.711 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:36.711 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:36.711 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:36.711 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:36.711 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:36.712 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:36.712 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:36.712 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:36.712 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:36.712 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:36.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 1 Processor -- start 2024-09-16T14:47:36.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 1 -- start start 2024-09-16T14:47:36.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient: init 2024-09-16T14:47:36.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:36.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:36.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient(hunting): picked mon.noname-c con 0x7f845409e950 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:36.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient(hunting): picked mon.noname-b con 0x7f845409edd0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:36.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient(hunting): picked mon.noname-a con 0x7f8454131630 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:36.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845406b320 con 0x7f8454131630 2024-09-16T14:47:36.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84541305f0 con 0x7f845409edd0 2024-09-16T14:47:36.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845413e820 con 0x7f845409e950 2024-09-16T14:47:36.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:36.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845bd90640 10 monclient(hunting): authenticate will time out at 1238.815576s 2024-09-16T14:47:36.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845932b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f845409edd0 0x7f845413e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54806/0 (socket says 172.21.15.189:54806) 2024-09-16T14:47:36.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845932b640 1 -- 172.21.15.189:0/3206980300 learned_addr learned my addr 172.21.15.189:0/3206980300 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:36.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f845a32d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f8454131630 legacy=0x7f84541d1990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:36.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f844a7fc640 1 -- 172.21.15.189:0/3206980300 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855847969 0 0) 0x7f84541305f0 con 0x7f845409edd0 2024-09-16T14:47:36.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f844a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:36.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f844a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:36.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f844a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:36.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f844a7fc640 10 monclient(hunting): my global_id is 95014 2024-09-16T14:47:36.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.694+0000 7f844a7fc640 1 -- 172.21.15.189:0/3206980300 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8438003530 con 0x7f845409edd0 2024-09-16T14:47:36.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 1 -- 172.21.15.189:0/3206980300 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2242702116 0 0) 0x7f8438003530 con 0x7f845409edd0 2024-09-16T14:47:36.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 1 -- 172.21.15.189:0/3206980300 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84541305f0 con 0x7f845409edd0 2024-09-16T14:47:36.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 1 -- 172.21.15.189:0/3206980300 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8444004190 con 0x7f845409edd0 2024-09-16T14:47:36.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:36.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:36.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 10 monclient(hunting): dump: 2024-09-16T14:47:36.714 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:36.714 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:36.714 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:36.715 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:36.715 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:36.715 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:36.715 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:36.715 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:36.715 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:36.715 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:36.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:36.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 1 -- 172.21.15.189:0/3206980300 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 58514100 0 0) 0x7f84541305f0 con 0x7f845409edd0 2024-09-16T14:47:36.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 1 -- 172.21.15.189:0/3206980300 >> v1:172.21.15.26:6790/0 conn(0x7f845409e950 legacy=0x7f845413da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:36.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 1 -- 172.21.15.189:0/3206980300 >> v1:172.21.15.26:6789/0 conn(0x7f8454131630 legacy=0x7f84541d1990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:36.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:36.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 1 monclient: found mon.b 2024-09-16T14:47:36.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:36.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:36.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 1 -- 172.21.15.189:0/3206980300 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845412f140 con 0x7f845409edd0 2024-09-16T14:47:36.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:36.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:36.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:36.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f845bd90640 5 monclient: authenticate success, global_id 95014 2024-09-16T14:47:36.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f845bd90640 10 monclient: _renew_subs 2024-09-16T14:47:36.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f845bd90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:36.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f845bd90640 1 -- 172.21.15.189:0/3206980300 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845409aa00 con 0x7f845409edd0 2024-09-16T14:47:36.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f845bd90640 10 monclient: _renew_subs 2024-09-16T14:47:36.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f845bd90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:36.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f845bd90640 1 -- 172.21.15.189:0/3206980300 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8454130310 con 0x7f845409edd0 2024-09-16T14:47:36.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 1 -- 172.21.15.189:0/3206980300 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8444004440 con 0x7f845409edd0 2024-09-16T14:47:36.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:36.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 1 -- 172.21.15.189:0/3206980300 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f84440047d0 con 0x7f845409edd0 2024-09-16T14:47:36.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:36.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:36.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844bfff640 4 set_mon_vals no callback set 2024-09-16T14:47:36.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 10 monclient: dump: 2024-09-16T14:47:36.718 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:36.718 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:36.718 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:36.718 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:36.718 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:36.718 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:36.718 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:36.718 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:36.719 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:36.719 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:36.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f845bd90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:36.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f845bd90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:36.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f845bd90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:36.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 1 -- 172.21.15.189:0/3206980300 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f84440049b0 con 0x7f845409edd0 2024-09-16T14:47:36.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f845bd90640 1 -- 172.21.15.189:0/3206980300 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f843c005050 con 0x7f845409edd0 2024-09-16T14:47:36.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f844a7fc640 1 -- 172.21.15.189:0/3206980300 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(127..127 src has 73..127) v4 ==== 7321+0+0 (unknown 2265503266 0 0) 0x7f8444044110 con 0x7f845409edd0 2024-09-16T14:47:36.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.698+0000 7f8459b2c640 10 monclient: get_auth_request con 0x7f8438035fe0 auth_method 0 2024-09-16T14:47:36.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.702+0000 7f844a7fc640 1 -- 172.21.15.189:0/3206980300 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f84440051d0 con 0x7f845409edd0 2024-09-16T14:47:36.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.702+0000 7f844a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:36.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.702+0000 7f844a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:36.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.866+0000 7f845bd90640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T14:47:36.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.866+0000 7f845bd90640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T14:47:36.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.866+0000 7f845bd90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:36.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:36.866+0000 7f845bd90640 1 -- 172.21.15.189:0/3206980300 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f843c005880 con 0x7f845409edd0 2024-09-16T14:47:37.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:37.238+0000 7f844a7fc640 1 -- 172.21.15.189:0/3206980300 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=0 v128) v1 ==== 129+0+0 (unknown 2858930871 0 0) 0x7f8444012c30 con 0x7f845409edd0 2024-09-16T14:47:37.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:37.238+0000 7f844a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T14:47:37.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:37.238+0000 7f844a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:37.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:37.346+0000 7f845bd90640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T14:47:37.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:37.346+0000 7f845bd90640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T14:47:37.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:37.346+0000 7f845bd90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:37.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:37.346+0000 7f845bd90640 1 -- 172.21.15.189:0/3206980300 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f843c005370 con 0x7f845409edd0 2024-09-16T14:47:38.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.250+0000 7f844a7fc640 1 -- 172.21.15.189:0/3206980300 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=0 v129) v1 ==== 129+0+0 (unknown 3917382952 0 0) 0x7f84440143b0 con 0x7f845409edd0 2024-09-16T14:47:38.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.250+0000 7f844a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T14:47:38.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.250+0000 7f844a7fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:38.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.254+0000 7f845bd90640 1 -- 172.21.15.189:0/3206980300 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8438035fe0 legacy=0x7f84380384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:38.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.254+0000 7f845bd90640 10 monclient: shutdown 2024-09-16T14:47:38.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.254+0000 7f845bd90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:38.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.254+0000 7f845bd90640 1 -- 172.21.15.189:0/3206980300 >> v1:172.21.15.189:6789/0 conn(0x7f845409edd0 legacy=0x7f845413e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:38.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.254+0000 7f845bd90640 1 -- 172.21.15.189:0/3206980300 shutdown_connections 2024-09-16T14:47:38.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.254+0000 7f845bd90640 1 -- 172.21.15.189:0/3206980300 >> 172.21.15.189:0/3206980300 conn(0x7f84540a3380 msgr2=0x7f845409d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:38.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.254+0000 7f845bd90640 1 -- 172.21.15.189:0/3206980300 shutdown_connections 2024-09-16T14:47:38.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.254+0000 7f845bd90640 1 -- 172.21.15.189:0/3206980300 wait complete. 2024-09-16T14:47:38.279 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 .9 2024-09-16T14:47:38.279 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:47:38.279 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 .9 2024-09-16T14:47:38.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 10 monclient: get_monmap_and_config 2024-09-16T14:47:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 10 monclient: build_initial_monmap 2024-09-16T14:47:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 10 monclient: monmap: 2024-09-16T14:47:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:38.406211+0000 2024-09-16T14:47:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:38.406211+0000 2024-09-16T14:47:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:38.407 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:38.407 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 1 Processor -- start 2024-09-16T14:47:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 1 -- start start 2024-09-16T14:47:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 10 monclient: init 2024-09-16T14:47:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 10 monclient(hunting): picked mon.noname-a con 0x7f44540a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 10 monclient(hunting): picked mon.noname-c con 0x7f44540a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 10 monclient(hunting): picked mon.noname-b con 0x7f445413b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44540a30b0 con 0x7f44540a1b00 2024-09-16T14:47:38.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44540a2e30 con 0x7f44540a25f0 2024-09-16T14:47:38.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445409eb50 con 0x7f445413b430 2024-09-16T14:47:38.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:38.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 10 monclient(hunting): authenticate will time out at 1240.522620s 2024-09-16T14:47:38.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f4459253640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f44540a1b00 legacy=0x7f44540a1ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:38.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f4458a52640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f44540a25f0 0x7f4454139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54808/0 (socket says 172.21.15.189:54808) 2024-09-16T14:47:38.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f4458a52640 1 -- 172.21.15.189:0/4185675691 learned_addr learned my addr 172.21.15.189:0/4185675691 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:38.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 1 -- 172.21.15.189:0/4185675691 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202512756 0 0) 0x7f44540a2e30 con 0x7f44540a25f0 2024-09-16T14:47:38.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:38.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:38.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:38.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 10 monclient(hunting): my global_id is 95017 2024-09-16T14:47:38.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 1 -- 172.21.15.189:0/4185675691 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4440003530 con 0x7f44540a25f0 2024-09-16T14:47:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 1 -- 172.21.15.189:0/4185675691 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4119586869 0 0) 0x7f4440003530 con 0x7f44540a25f0 2024-09-16T14:47:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 1 -- 172.21.15.189:0/4185675691 >> v1:172.21.15.26:6790/0 conn(0x7f445413b430 legacy=0x7f445413d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 1 -- 172.21.15.189:0/4185675691 >> v1:172.21.15.26:6789/0 conn(0x7f44540a1b00 legacy=0x7f44540a1ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:47:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 1 monclient: found mon.noname-c 2024-09-16T14:47:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 1 -- 172.21.15.189:0/4185675691 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445409ee30 con 0x7f44540a25f0 2024-09-16T14:47:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 10 monclient: _finish_auth 0 2024-09-16T14:47:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:47:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 5 monclient: authenticate success, global_id 95017 2024-09-16T14:47:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 1 -- 172.21.15.189:0/4185675691 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4444001620 con 0x7f44540a25f0 2024-09-16T14:47:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 10 monclient: dump: 2024-09-16T14:47:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:38.417 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:38.417 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:38.417 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:38.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:38.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 1 -- 172.21.15.189:0/4185675691 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4444001940 con 0x7f44540a25f0 2024-09-16T14:47:38.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:38.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:38.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f445b4b7640 4 set_mon_vals no callback set 2024-09-16T14:47:38.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 1 -- 172.21.15.189:0/4185675691 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4444001d10 con 0x7f44540a25f0 2024-09-16T14:47:38.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:38.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:38.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.402+0000 7f444bfff640 10 monclient: dump: 2024-09-16T14:47:38.417 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:38.418 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:38.418 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:38.418 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:38.418 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:38.418 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:38.418 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:38.418 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:38.418 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:38.418 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:38.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient: shutdown 2024-09-16T14:47:38.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:38.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 1 -- 172.21.15.189:0/4185675691 >> v1:172.21.15.189:6789/0 conn(0x7f44540a25f0 legacy=0x7f4454139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:38.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 1 -- 172.21.15.189:0/4185675691 shutdown_connections 2024-09-16T14:47:38.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 1 -- 172.21.15.189:0/4185675691 >> 172.21.15.189:0/4185675691 conn(0x7f44540a7dd0 msgr2=0x7f44540a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:38.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 1 -- 172.21.15.189:0/4185675691 shutdown_connections 2024-09-16T14:47:38.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 1 -- 172.21.15.189:0/4185675691 wait complete. 2024-09-16T14:47:38.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient: build_initial_monmap 2024-09-16T14:47:38.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient: monmap: 2024-09-16T14:47:38.419 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:38.419 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:38.419 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:38.419 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:38.419 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:38.419 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:38.419 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:38.420 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:38.420 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:38.420 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:38.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 1 Processor -- start 2024-09-16T14:47:38.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 1 -- start start 2024-09-16T14:47:38.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient: init 2024-09-16T14:47:38.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:38.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient(hunting): picked mon.noname-b con 0x7f44540a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient(hunting): picked mon.noname-a con 0x7f44540a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient(hunting): picked mon.noname-c con 0x7f445413b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44540a5aa0 con 0x7f44540a25f0 2024-09-16T14:47:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445409e2f0 con 0x7f44540a1b00 2024-09-16T14:47:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445413ae40 con 0x7f445413b430 2024-09-16T14:47:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient(hunting): authenticate will time out at 1240.525705s 2024-09-16T14:47:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4459253640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f44540a1b00 0x7f44541395e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54822/0 (socket says 172.21.15.189:54822) 2024-09-16T14:47:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4458a52640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f44540a25f0 legacy=0x7f4454139cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4459253640 1 -- 172.21.15.189:0/2867329955 learned_addr learned my addr 172.21.15.189:0/2867329955 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832036343 0 0) 0x7f445409e2f0 con 0x7f44540a1b00 2024-09-16T14:47:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient(hunting): my global_id is 95020 2024-09-16T14:47:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4438003560 con 0x7f44540a1b00 2024-09-16T14:47:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3892733533 0 0) 0x7f4438003560 con 0x7f44540a1b00 2024-09-16T14:47:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f445409e2f0 con 0x7f44540a1b00 2024-09-16T14:47:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4450004190 con 0x7f44540a1b00 2024-09-16T14:47:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient(hunting): dump: 2024-09-16T14:47:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:38.424 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279932271 0 0) 0x7f445413ae40 con 0x7f445413b430 2024-09-16T14:47:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient(hunting): my global_id is 114389 2024-09-16T14:47:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4438003560 con 0x7f445413b430 2024-09-16T14:47:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3179991736 0 0) 0x7f445409e2f0 con 0x7f44540a1b00 2024-09-16T14:47:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 >> v1:172.21.15.26:6790/0 conn(0x7f445413b430 legacy=0x7f445413a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 >> v1:172.21.15.26:6789/0 conn(0x7f44540a25f0 legacy=0x7f4454139cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:47:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 1 monclient: found mon.b 2024-09-16T14:47:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44541357a0 con 0x7f44540a1b00 2024-09-16T14:47:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient: _finish_auth 0 2024-09-16T14:47:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 5 monclient: authenticate success, global_id 95020 2024-09-16T14:47:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient: _renew_subs 2024-09-16T14:47:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 1 -- 172.21.15.189:0/2867329955 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f445411fdd0 con 0x7f44540a1b00 2024-09-16T14:47:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient: _renew_subs 2024-09-16T14:47:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 1 -- 172.21.15.189:0/2867329955 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44541d2610 con 0x7f44540a1b00 2024-09-16T14:47:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44500044b0 con 0x7f44540a1b00 2024-09-16T14:47:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4450004840 con 0x7f44540a1b00 2024-09-16T14:47:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 10 monclient: dump: 2024-09-16T14:47:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:38.427 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f444affd640 4 set_mon_vals no callback set 2024-09-16T14:47:38.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:38.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:38.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:38.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f445b4b7640 1 -- 172.21.15.189:0/2867329955 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4434005050 con 0x7f44540a1b00 2024-09-16T14:47:38.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f44500049e0 con 0x7f44540a1b00 2024-09-16T14:47:38.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(129..129 src has 73..129) v4 ==== 7321+0+0 (unknown 195063541 0 0) 0x7f4450044090 con 0x7f44540a1b00 2024-09-16T14:47:38.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.406+0000 7f4458a52640 10 monclient: get_auth_request con 0x7f4438036060 auth_method 0 2024-09-16T14:47:38.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.414+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f44500051b0 con 0x7f44540a1b00 2024-09-16T14:47:38.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.414+0000 7f4449ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:38.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.414+0000 7f4449ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:38.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.582+0000 7f445b4b7640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-16T14:47:38.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.582+0000 7f445b4b7640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-16T14:47:38.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.582+0000 7f445b4b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:38.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.582+0000 7f445b4b7640 1 -- 172.21.15.189:0/2867329955 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]} v 0) v1 -- 0x7f4434005880 con 0x7f44540a1b00 2024-09-16T14:47:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.586+0000 7f4449ffb640 1 -- 172.21.15.189:0/2867329955 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}]=-22 must specify exact 1 weight values v129) v1 ==== 168+0+0 (unknown 4057322471 0 0) 0x7f4450012c00 con 0x7f44540a1b00 2024-09-16T14:47:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.586+0000 7f4449ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-16T14:47:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.586+0000 7f4449ffb640 10 monclient: _finish_command 2 = mon:22 must specify exact 1 weight values 2024-09-16T14:47:38.590 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: must specify exact 1 weight values 2024-09-16T14:47:38.595 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:47:38.595 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush weight-set reweight barset osd.0 .9 2024-09-16T14:47:38.596 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T14:47:38.596 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight barset osd.0 .9 2024-09-16T14:47:38.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 10 monclient: get_monmap_and_config 2024-09-16T14:47:38.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 10 monclient: build_initial_monmap 2024-09-16T14:47:38.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 10 monclient: monmap: 2024-09-16T14:47:38.711 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:38.711 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:38.711 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:38.710993+0000 2024-09-16T14:47:38.711 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:38.710993+0000 2024-09-16T14:47:38.711 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:38.711 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:38.711 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:38.711 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:38.711 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:38.711 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:38.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 1 Processor -- start 2024-09-16T14:47:38.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 1 -- start start 2024-09-16T14:47:38.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 10 monclient: init 2024-09-16T14:47:38.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:38.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:38.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 10 monclient(hunting): picked mon.noname-c con 0x7f88c4099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:38.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 10 monclient(hunting): picked mon.noname-a con 0x7f88c409c550 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:38.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 10 monclient(hunting): picked mon.noname-b con 0x7f88c409e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:38.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c4099e50 con 0x7f88c409c550 2024-09-16T14:47:38.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c409dbd0 con 0x7f88c4099a70 2024-09-16T14:47:38.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c409ee20 con 0x7f88c409e6d0 2024-09-16T14:47:38.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:38.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 10 monclient(hunting): authenticate will time out at 1240.826889s 2024-09-16T14:47:38.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88c8d35640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f88c409c550 legacy=0x7f88c41379d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:38.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88c9536640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f88c4099a70 0x7f88c409dfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54838/0 (socket says 172.21.15.189:54838) 2024-09-16T14:47:38.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88c9536640 1 -- 172.21.15.189:0/3494609660 learned_addr learned my addr 172.21.15.189:0/3494609660 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 1 -- 172.21.15.189:0/3494609660 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139890766 0 0) 0x7f88c409dbd0 con 0x7f88c4099a70 2024-09-16T14:47:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 10 monclient(hunting): my global_id is 95023 2024-09-16T14:47:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 1 -- 172.21.15.189:0/3494609660 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b0003530 con 0x7f88c4099a70 2024-09-16T14:47:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 1 -- 172.21.15.189:0/3494609660 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3930719270 0 0) 0x7f88b0003530 con 0x7f88c4099a70 2024-09-16T14:47:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 1 -- 172.21.15.189:0/3494609660 >> v1:172.21.15.26:6790/0 conn(0x7f88c409e6d0 legacy=0x7f88c413b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 1 -- 172.21.15.189:0/3494609660 >> v1:172.21.15.26:6789/0 conn(0x7f88c409c550 legacy=0x7f88c41379d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 10 monclient: _finish_hunting 0 2024-09-16T14:47:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 1 monclient: found mon.noname-c 2024-09-16T14:47:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:38.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 1 -- 172.21.15.189:0/3494609660 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c409f100 con 0x7f88c4099a70 2024-09-16T14:47:38.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 10 monclient: _finish_auth 0 2024-09-16T14:47:38.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 10 monclient: _check_auth_tickets 2024-09-16T14:47:38.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:38.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 1 -- 172.21.15.189:0/3494609660 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f88c0001620 con 0x7f88c4099a70 2024-09-16T14:47:38.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:38.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:38.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 10 monclient: dump: 2024-09-16T14:47:38.720 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:38.720 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:38.720 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:38.720 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:38.721 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:38.721 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:38.721 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:38.721 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:38.721 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:38.721 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:38.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 5 monclient: authenticate success, global_id 95023 2024-09-16T14:47:38.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:38.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 1 -- 172.21.15.189:0/3494609660 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88c00018d0 con 0x7f88c4099a70 2024-09-16T14:47:38.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:38.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:38.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 1 -- 172.21.15.189:0/3494609660 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f88c0001c60 con 0x7f88c4099a70 2024-09-16T14:47:38.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88cb79a640 4 set_mon_vals no callback set 2024-09-16T14:47:38.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:38.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:38.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.706+0000 7f88bbfff640 10 monclient: dump: 2024-09-16T14:47:38.722 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:38.722 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:38.722 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:38.722 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:38.722 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:38.722 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:38.722 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:38.722 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:38.723 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:38.723 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:38.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient: shutdown 2024-09-16T14:47:38.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:38.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 1 -- 172.21.15.189:0/3494609660 >> v1:172.21.15.189:6789/0 conn(0x7f88c4099a70 legacy=0x7f88c409dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:38.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 1 -- 172.21.15.189:0/3494609660 shutdown_connections 2024-09-16T14:47:38.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 1 -- 172.21.15.189:0/3494609660 >> 172.21.15.189:0/3494609660 conn(0x7f88c40a3380 msgr2=0x7f88c409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:38.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 1 -- 172.21.15.189:0/3494609660 shutdown_connections 2024-09-16T14:47:38.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 1 -- 172.21.15.189:0/3494609660 wait complete. 2024-09-16T14:47:38.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient: build_initial_monmap 2024-09-16T14:47:38.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient: monmap: 2024-09-16T14:47:38.723 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:38.723 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:38.724 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:38.724 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:38.724 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:38.724 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:38.724 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:38.724 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:38.724 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:38.724 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:38.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 1 Processor -- start 2024-09-16T14:47:38.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 1 -- start start 2024-09-16T14:47:38.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient: init 2024-09-16T14:47:38.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:38.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:38.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient(hunting): picked mon.noname-b con 0x7f88c4099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:38.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient(hunting): picked mon.noname-a con 0x7f88c409c550 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:38.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient(hunting): picked mon.noname-c con 0x7f88c409e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:38.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c409cf20 con 0x7f88c409c550 2024-09-16T14:47:38.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c4133a40 con 0x7f88c4099a70 2024-09-16T14:47:38.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c406b320 con 0x7f88c409e6d0 2024-09-16T14:47:38.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:38.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient(hunting): authenticate will time out at 1240.829836s 2024-09-16T14:47:38.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88c9536640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f88c4099a70 0x7f88c41c9930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54844/0 (socket says 172.21.15.189:54844) 2024-09-16T14:47:38.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88c9536640 1 -- 172.21.15.189:0/1243317613 learned_addr learned my addr 172.21.15.189:0/1243317613 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:38.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88c8d35640 1 -- 172.21.15.189:0/1243317613 >> v1:172.21.15.26:6789/0 conn(0x7f88c409c550 legacy=0x7f88c41ce910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:38.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042882628 0 0) 0x7f88c4133a40 con 0x7f88c4099a70 2024-09-16T14:47:38.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:38.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:38.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:38.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient(hunting): my global_id is 95026 2024-09-16T14:47:38.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a8003530 con 0x7f88c4099a70 2024-09-16T14:47:38.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2973667519 0 0) 0x7f88a8003530 con 0x7f88c4099a70 2024-09-16T14:47:38.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88c4133a40 con 0x7f88c4099a70 2024-09-16T14:47:38.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f88c00015e0 con 0x7f88c4099a70 2024-09-16T14:47:38.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:38.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:38.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient(hunting): dump: 2024-09-16T14:47:38.726 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:38.726 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:38.727 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:38.727 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:38.727 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:38.727 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:38.727 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:38.727 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:38.727 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:38.727 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:38.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:38.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033514529 0 0) 0x7f88c406b320 con 0x7f88c409e6d0 2024-09-16T14:47:38.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:38.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:38.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:38.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient(hunting): my global_id is 114395 2024-09-16T14:47:38.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a8003530 con 0x7f88c409e6d0 2024-09-16T14:47:38.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1600095781 0 0) 0x7f88c4133a40 con 0x7f88c4099a70 2024-09-16T14:47:38.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 >> v1:172.21.15.26:6790/0 conn(0x7f88c409e6d0 legacy=0x7f88c41d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:38.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 >> v1:172.21.15.26:6789/0 conn(0x7f88c409c550 legacy=0x7f88c41ce910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:38.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:47:38.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 1 monclient: found mon.b 2024-09-16T14:47:38.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:38.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:38.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c413aec0 con 0x7f88c4099a70 2024-09-16T14:47:38.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient: _finish_auth 0 2024-09-16T14:47:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 5 monclient: authenticate success, global_id 95026 2024-09-16T14:47:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient: _renew_subs 2024-09-16T14:47:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 1 -- 172.21.15.189:0/1243317613 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c409a780 con 0x7f88c4099a70 2024-09-16T14:47:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient: _renew_subs 2024-09-16T14:47:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 1 -- 172.21.15.189:0/1243317613 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c413ac40 con 0x7f88c4099a70 2024-09-16T14:47:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88c0004b20 con 0x7f88c4099a70 2024-09-16T14:47:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f88c0004eb0 con 0x7f88c4099a70 2024-09-16T14:47:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:38.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 10 monclient: dump: 2024-09-16T14:47:38.730 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:38.730 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:38.730 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:38.730 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:38.730 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:38.730 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:38.730 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:38.730 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:38.730 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:38.730 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:38.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88bb7fe640 4 set_mon_vals no callback set 2024-09-16T14:47:38.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:38.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f88c00050e0 con 0x7f88c4099a70 2024-09-16T14:47:38.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:38.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:38.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88cb79a640 1 -- 172.21.15.189:0/1243317613 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a4005050 con 0x7f88c4099a70 2024-09-16T14:47:38.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(129..129 src has 73..129) v4 ==== 7321+0+0 (unknown 195063541 0 0) 0x7f88c00442e0 con 0x7f88c4099a70 2024-09-16T14:47:38.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.710+0000 7f88c8d35640 10 monclient: get_auth_request con 0x7f88a80360e0 auth_method 0 2024-09-16T14:47:38.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.718+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f88c0014a40 con 0x7f88c4099a70 2024-09-16T14:47:38.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.718+0000 7f88b9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:38.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.718+0000 7f88b9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:38.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.882+0000 7f88cb79a640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T14:47:38.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.882+0000 7f88cb79a640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T14:47:38.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.882+0000 7f88cb79a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:38.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.882+0000 7f88cb79a640 1 -- 172.21.15.189:0/1243317613 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f88a4005880 con 0x7f88c4099a70 2024-09-16T14:47:38.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.886+0000 7f88b9ffb640 1 -- 172.21.15.189:0/1243317613 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}]=-22 must specify exact 3 weight values v129) v1 ==== 163+0+0 (unknown 479058460 0 0) 0x7f88c0014630 con 0x7f88c4099a70 2024-09-16T14:47:38.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.886+0000 7f88b9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T14:47:38.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:38.886+0000 7f88b9ffb640 10 monclient: _finish_command 2 = mon:22 must specify exact 3 weight values 2024-09-16T14:47:38.890 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: must specify exact 3 weight values 2024-09-16T14:47:38.895 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:47:38.895 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight barset osd.0 .9 .9 .9 2024-09-16T14:47:39.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd65c1e640 10 monclient: get_monmap_and_config 2024-09-16T14:47:39.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd65c1e640 10 monclient: build_initial_monmap 2024-09-16T14:47:39.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd65c1e640 10 monclient: monmap: 2024-09-16T14:47:39.019 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:39.019 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:39.019 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:39.019253+0000 2024-09-16T14:47:39.020 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:39.019253+0000 2024-09-16T14:47:39.020 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:39.020 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:39.020 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:39.020 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:39.020 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:39.020 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:39.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd65c1e640 1 Processor -- start 2024-09-16T14:47:39.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd65c1e640 1 -- start start 2024-09-16T14:47:39.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd65c1e640 10 monclient: init 2024-09-16T14:47:39.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd65c1e640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:39.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd65c1e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:39.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd65c1e640 10 monclient(hunting): picked mon.noname-b con 0x7fbd600a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:39.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd65c1e640 10 monclient(hunting): picked mon.noname-a con 0x7fbd600a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:39.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd65c1e640 10 monclient(hunting): picked mon.noname-c con 0x7fbd6013b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:39.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd65c1e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd600a30b0 con 0x7fbd600a25f0 2024-09-16T14:47:39.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd65c1e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd600a2e30 con 0x7fbd6013b430 2024-09-16T14:47:39.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd65c1e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd6009eb50 con 0x7fbd600a1b00 2024-09-16T14:47:39.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd65c1e640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:39.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd65c1e640 10 monclient(hunting): authenticate will time out at 1241.135359s 2024-09-16T14:47:39.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd5effd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fbd600a25f0 legacy=0x7fbd60139d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:39.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd5ffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbd6013b430 0x7fbd6013d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33506/0 (socket says 172.21.15.189:33506) 2024-09-16T14:47:39.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd5ffff640 1 -- 172.21.15.189:0/1585288456 learned_addr learned my addr 172.21.15.189:0/1585288456 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:39.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd5e7fc640 1 -- 172.21.15.189:0/1585288456 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236053214 0 0) 0x7fbd600a2e30 con 0x7fbd6013b430 2024-09-16T14:47:39.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd5e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:39.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:39.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:39.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd5e7fc640 10 monclient(hunting): my global_id is 95029 2024-09-16T14:47:39.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd5e7fc640 1 -- 172.21.15.189:0/1585288456 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd48003530 con 0x7fbd6013b430 2024-09-16T14:47:39.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.014+0000 7fbd5e7fc640 1 -- 172.21.15.189:0/1585288456 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 160020890 0 0) 0x7fbd48003530 con 0x7fbd6013b430 2024-09-16T14:47:39.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 1 -- 172.21.15.189:0/1585288456 >> v1:172.21.15.26:6790/0 conn(0x7fbd600a1b00 legacy=0x7fbd600a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:39.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 1 -- 172.21.15.189:0/1585288456 >> v1:172.21.15.26:6789/0 conn(0x7fbd600a25f0 legacy=0x7fbd60139d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:39.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:39.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 1 monclient: found mon.noname-c 2024-09-16T14:47:39.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:39.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:39.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 1 -- 172.21.15.189:0/1585288456 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6009ee30 con 0x7fbd6013b430 2024-09-16T14:47:39.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:39.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:39.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:39.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 5 monclient: authenticate success, global_id 95029 2024-09-16T14:47:39.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:39.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 1 -- 172.21.15.189:0/1585288456 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbd580019a0 con 0x7fbd6013b430 2024-09-16T14:47:39.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:39.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:39.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 10 monclient: dump: 2024-09-16T14:47:39.031 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:39.031 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:39.031 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.032 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.032 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:39.032 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:39.032 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:39.032 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:39.032 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:39.032 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:39.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:39.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 1 -- 172.21.15.189:0/1585288456 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd58001cc0 con 0x7fbd6013b430 2024-09-16T14:47:39.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:39.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:39.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 1 -- 172.21.15.189:0/1585288456 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbd58002090 con 0x7fbd6013b430 2024-09-16T14:47:39.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:39.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 4 set_mon_vals no callback set 2024-09-16T14:47:39.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:39.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5e7fc640 10 monclient: dump: 2024-09-16T14:47:39.033 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:39.033 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:39.033 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.033 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.033 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:39.034 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:39.034 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:39.034 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:39.034 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:39.034 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:39.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 10 monclient: shutdown 2024-09-16T14:47:39.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:39.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 1 -- 172.21.15.189:0/1585288456 >> v1:172.21.15.189:6789/0 conn(0x7fbd6013b430 legacy=0x7fbd6013d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:39.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 1 -- 172.21.15.189:0/1585288456 shutdown_connections 2024-09-16T14:47:39.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 1 -- 172.21.15.189:0/1585288456 >> 172.21.15.189:0/1585288456 conn(0x7fbd600a7dd0 msgr2=0x7fbd600a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 1 -- 172.21.15.189:0/1585288456 shutdown_connections 2024-09-16T14:47:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 1 -- 172.21.15.189:0/1585288456 wait complete. 2024-09-16T14:47:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 10 monclient: build_initial_monmap 2024-09-16T14:47:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 10 monclient: monmap: 2024-09-16T14:47:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:39.035 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:39.036 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:39.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 1 Processor -- start 2024-09-16T14:47:39.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 1 -- start start 2024-09-16T14:47:39.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 10 monclient: init 2024-09-16T14:47:39.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:39.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:39.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 10 monclient(hunting): picked mon.noname-c con 0x7fbd600a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:39.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 10 monclient(hunting): picked mon.noname-b con 0x7fbd600a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:39.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 10 monclient(hunting): picked mon.noname-a con 0x7fbd6013b430 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:39.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd6013ae60 con 0x7fbd6013b430 2024-09-16T14:47:39.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd6011db20 con 0x7fbd600a25f0 2024-09-16T14:47:39.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd600a5a60 con 0x7fbd600a1b00 2024-09-16T14:47:39.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 10 monclient(hunting): authenticate will time out at 1241.138456s 2024-09-16T14:47:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5effd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbd600a25f0 0x7fbd60139f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33512/0 (socket says 172.21.15.189:33512) 2024-09-16T14:47:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5effd640 1 -- 172.21.15.189:0/3449334345 learned_addr learned my addr 172.21.15.189:0/3449334345 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5ffff640 1 -- 172.21.15.189:0/3449334345 >> v1:172.21.15.26:6789/0 conn(0x7fbd6013b430 legacy=0x7fbd601d1600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:47:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64322622 0 0) 0x7fbd6011db20 con 0x7fbd600a25f0 2024-09-16T14:47:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient(hunting): my global_id is 95032 2024-09-16T14:47:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd4c003530 con 0x7fbd600a25f0 2024-09-16T14:47:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1810540688 0 0) 0x7fbd4c003530 con 0x7fbd600a25f0 2024-09-16T14:47:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd6011db20 con 0x7fbd600a25f0 2024-09-16T14:47:39.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbd54004190 con 0x7fbd600a25f0 2024-09-16T14:47:39.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:39.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:39.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient(hunting): dump: 2024-09-16T14:47:39.038 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:39.038 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:39.038 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.038 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.038 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:39.038 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:39.038 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:39.038 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:39.039 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434311707 0 0) 0x7fbd600a5a60 con 0x7fbd600a1b00 2024-09-16T14:47:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient(hunting): my global_id is 114401 2024-09-16T14:47:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd4c003530 con 0x7fbd600a1b00 2024-09-16T14:47:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 312107862 0 0) 0x7fbd6011db20 con 0x7fbd600a25f0 2024-09-16T14:47:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 >> v1:172.21.15.26:6790/0 conn(0x7fbd600a1b00 legacy=0x7fbd60139850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 >> v1:172.21.15.26:6789/0 conn(0x7fbd6013b430 legacy=0x7fbd601d1600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient: _finish_hunting 0 2024-09-16T14:47:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 1 monclient: found mon.b 2024-09-16T14:47:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6011fe60 con 0x7fbd600a25f0 2024-09-16T14:47:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient: _finish_auth 0 2024-09-16T14:47:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient: _check_auth_tickets 2024-09-16T14:47:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 5 monclient: authenticate success, global_id 95032 2024-09-16T14:47:39.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 10 monclient: _renew_subs 2024-09-16T14:47:39.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:39.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 1 -- 172.21.15.189:0/3449334345 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6011e450 con 0x7fbd600a25f0 2024-09-16T14:47:39.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 10 monclient: _renew_subs 2024-09-16T14:47:39.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:39.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd65c1e640 1 -- 172.21.15.189:0/3449334345 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6013a670 con 0x7fbd600a25f0 2024-09-16T14:47:39.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd540044b0 con 0x7fbd600a25f0 2024-09-16T14:47:39.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:39.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbd54004840 con 0x7fbd600a25f0 2024-09-16T14:47:39.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:39.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:39.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd47fff640 10 monclient: dump: 2024-09-16T14:47:39.043 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:39.043 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:39.043 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.043 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.043 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:39.043 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:39.043 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:39.043 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:39.043 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:39.044 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:39.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.018+0000 7fbd5dffb640 4 set_mon_vals no callback set 2024-09-16T14:47:39.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.022+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbd540049e0 con 0x7fbd600a25f0 2024-09-16T14:47:39.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.022+0000 7fbd65c1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:39.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.022+0000 7fbd65c1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:39.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.022+0000 7fbd65c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:39.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.022+0000 7fbd65c1e640 1 -- 172.21.15.189:0/3449334345 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd40005050 con 0x7fbd600a25f0 2024-09-16T14:47:39.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.022+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(129..129 src has 73..129) v4 ==== 7321+0+0 (unknown 195063541 0 0) 0x7fbd54044090 con 0x7fbd600a25f0 2024-09-16T14:47:39.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.022+0000 7fbd5f7fe640 10 monclient: get_auth_request con 0x7fbd4c0361d0 auth_method 0 2024-09-16T14:47:39.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.026+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbd540051b0 con 0x7fbd600a25f0 2024-09-16T14:47:39.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.026+0000 7fbd47fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:39.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.026+0000 7fbd47fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:39.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.190+0000 7fbd65c1e640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T14:47:39.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.190+0000 7fbd65c1e640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T14:47:39.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.190+0000 7fbd65c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:39.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.190+0000 7fbd65c1e640 1 -- 172.21.15.189:0/3449334345 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]} v 0) v1 -- 0x7fbd40005300 con 0x7fbd600a25f0 2024-09-16T14:47:39.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.250+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}]=0 v130) v1 ==== 139+0+0 (unknown 1393095541 0 0) 0x7fbd54012c00 con 0x7fbd600a25f0 2024-09-16T14:47:39.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.250+0000 7fbd47fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T14:47:39.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.250+0000 7fbd47fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:39.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.358+0000 7fbd65c1e640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T14:47:39.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.358+0000 7fbd65c1e640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T14:47:39.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.358+0000 7fbd65c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:39.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:39.358+0000 7fbd65c1e640 1 -- 172.21.15.189:0/3449334345 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]} v 0) v1 -- 0x7fbd400010a0 con 0x7fbd600a25f0 2024-09-16T14:47:39.469 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:47:39.469 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T14:47:39.469 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T14:47:39.470 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T14:47:39.473 INFO:tasks.ceph.mon.a:Started 2024-09-16T14:47:39.473 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:47:39.473 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:47:39.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf29bb5640 10 monclient: get_monmap_and_config 2024-09-16T14:47:39.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf29bb5640 10 monclient: build_initial_monmap 2024-09-16T14:47:39.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf29bb5640 10 monclient: monmap: 2024-09-16T14:47:39.595 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:39.595 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:39.595 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:39.595465+0000 2024-09-16T14:47:39.595 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:39.595465+0000 2024-09-16T14:47:39.595 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:39.595 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:39.595 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:39.596 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:39.596 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:39.596 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:39.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf29bb5640 1 Processor -- start 2024-09-16T14:47:39.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf29bb5640 1 -- start start 2024-09-16T14:47:39.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf29bb5640 10 monclient: init 2024-09-16T14:47:39.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf29bb5640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:39.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf29bb5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:39.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf29bb5640 10 monclient(hunting): picked mon.noname-a con 0x7faf240a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:39.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf29bb5640 10 monclient(hunting): picked mon.noname-c con 0x7faf24099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:39.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf29bb5640 10 monclient(hunting): picked mon.noname-b con 0x7faf240a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:39.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf29bb5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf24098ea0 con 0x7faf240a1c00 2024-09-16T14:47:39.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf29bb5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf240997b0 con 0x7faf24099b60 2024-09-16T14:47:39.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf29bb5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf24099460 con 0x7faf240a26f0 2024-09-16T14:47:39.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf29bb5640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:39.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf29bb5640 10 monclient(hunting): authenticate will time out at 1283.715271s 2024-09-16T14:47:39.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf23fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7faf240a26f0 0x7faf2413d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55088/0 (socket says 172.21.15.26:55088) 2024-09-16T14:47:39.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.591+0000 7faf23fff640 1 -- 172.21.15.26:0/2092582652 learned_addr learned my addr 172.21.15.26:0/2092582652 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:39.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 1 -- 172.21.15.26:0/2092582652 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555801886 0 0) 0x7faf240997b0 con 0x7faf24099b60 2024-09-16T14:47:39.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:39.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:39.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:39.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 10 monclient(hunting): my global_id is 95035 2024-09-16T14:47:39.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 1 -- 172.21.15.26:0/2092582652 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf10003530 con 0x7faf24099b60 2024-09-16T14:47:39.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 1 -- 172.21.15.26:0/2092582652 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1351914628 0 0) 0x7faf10003530 con 0x7faf24099b60 2024-09-16T14:47:39.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 1 -- 172.21.15.26:0/2092582652 >> v1:172.21.15.26:6790/0 conn(0x7faf240a26f0 legacy=0x7faf2413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:39.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 1 -- 172.21.15.26:0/2092582652 >> v1:172.21.15.26:6789/0 conn(0x7faf240a1c00 legacy=0x7faf240a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:39.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:39.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 1 monclient: found mon.noname-c 2024-09-16T14:47:39.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:39.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:39.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 1 -- 172.21.15.26:0/2092582652 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf240a34b0 con 0x7faf24099b60 2024-09-16T14:47:39.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 10 monclient: _finish_auth 0 2024-09-16T14:47:39.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:39.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:39.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 1 -- 172.21.15.26:0/2092582652 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faf0c001620 con 0x7faf24099b60 2024-09-16T14:47:39.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:39.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:39.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 10 monclient: dump: 2024-09-16T14:47:39.602 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:39.602 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:39.602 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.603 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.603 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:39.603 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:39.603 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:39.603 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:39.603 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:39.603 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:39.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf29bb5640 5 monclient: authenticate success, global_id 95035 2024-09-16T14:47:39.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf29bb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:39.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 1 -- 172.21.15.26:0/2092582652 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf0c001840 con 0x7faf24099b60 2024-09-16T14:47:39.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:39.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf29bb5640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:39.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf29bb5640 4 set_mon_vals no callback set 2024-09-16T14:47:39.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 1 -- 172.21.15.26:0/2092582652 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faf0c001c10 con 0x7faf24099b60 2024-09-16T14:47:39.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:39.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:39.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf22ffd640 10 monclient: dump: 2024-09-16T14:47:39.604 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:39.604 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:39.604 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.604 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.604 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:39.604 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:39.604 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:39.604 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:39.604 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:39.605 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:39.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf29bb5640 10 monclient: shutdown 2024-09-16T14:47:39.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf29bb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:39.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf29bb5640 1 -- 172.21.15.26:0/2092582652 >> v1:172.21.15.189:6789/0 conn(0x7faf24099b60 legacy=0x7faf24139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:39.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf29bb5640 1 -- 172.21.15.26:0/2092582652 shutdown_connections 2024-09-16T14:47:39.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf29bb5640 1 -- 172.21.15.26:0/2092582652 >> 172.21.15.26:0/2092582652 conn(0x7faf240a3d00 msgr2=0x7faf240a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:39.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf29bb5640 1 -- 172.21.15.26:0/2092582652 shutdown_connections 2024-09-16T14:47:39.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf29bb5640 1 -- 172.21.15.26:0/2092582652 wait complete. 2024-09-16T14:47:39.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf29bb5640 10 monclient: build_initial_monmap 2024-09-16T14:47:39.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf29bb5640 10 monclient: monmap: 2024-09-16T14:47:39.605 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:39.605 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:39.605 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:39.605 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:39.606 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:39.606 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:39.606 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:39.606 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:39.606 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:39.606 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:39.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.595+0000 7faf29bb5640 1 Processor -- start 2024-09-16T14:47:39.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 1 -- start start 2024-09-16T14:47:39.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 10 monclient: init 2024-09-16T14:47:39.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:39.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:39.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 10 monclient(hunting): picked mon.noname-c con 0x7faf24099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:39.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 10 monclient(hunting): picked mon.noname-a con 0x7faf240a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:39.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 10 monclient(hunting): picked mon.noname-b con 0x7faf240a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:39.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf2413c830 con 0x7faf240a1c00 2024-09-16T14:47:39.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf240a1520 con 0x7faf240a26f0 2024-09-16T14:47:39.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf240a0f60 con 0x7faf24099b60 2024-09-16T14:47:39.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:39.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 10 monclient(hunting): authenticate will time out at 1283.720864s 2024-09-16T14:47:39.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf1bfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7faf240a1c00 0x7faf2413a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47726/0 (socket says 172.21.15.26:47726) 2024-09-16T14:47:39.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf237fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7faf24099b60 0x7faf2413a220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55094/0 (socket says 172.21.15.26:55094) 2024-09-16T14:47:39.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf1bfff640 1 -- 172.21.15.26:0/3374044971 learned_addr learned my addr 172.21.15.26:0/3374044971 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:39.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 1 -- 172.21.15.26:0/3374044971 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188578789 0 0) 0x7faf240a0f60 con 0x7faf24099b60 2024-09-16T14:47:39.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:47:39.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:39.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:39.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 10 monclient(hunting): my global_id is 114404 2024-09-16T14:47:39.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 1 -- 172.21.15.26:0/3374044971 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf04003530 con 0x7faf24099b60 2024-09-16T14:47:39.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 1 -- 172.21.15.26:0/3374044971 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1946638153 0 0) 0x7faf04003530 con 0x7faf24099b60 2024-09-16T14:47:39.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 1 -- 172.21.15.26:0/3374044971 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf240a0f60 con 0x7faf24099b60 2024-09-16T14:47:39.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 1 -- 172.21.15.26:0/3374044971 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faf14001870 con 0x7faf24099b60 2024-09-16T14:47:39.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:39.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:39.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 10 monclient(hunting): dump: 2024-09-16T14:47:39.610 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:39.611 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:39.611 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.611 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.611 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:39.611 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:39.611 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:39.611 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:39.611 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:39.611 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:39.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:39.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 1 -- 172.21.15.26:0/3374044971 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 903318540 0 0) 0x7faf240a0f60 con 0x7faf24099b60 2024-09-16T14:47:39.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 1 -- 172.21.15.26:0/3374044971 >> v1:172.21.15.189:6789/0 conn(0x7faf240a26f0 legacy=0x7faf241d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:39.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 1 -- 172.21.15.26:0/3374044971 >> v1:172.21.15.26:6789/0 conn(0x7faf240a1c00 legacy=0x7faf2413a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:39.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:47:39.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 1 monclient: found mon.c 2024-09-16T14:47:39.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:39.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:39.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 1 -- 172.21.15.26:0/3374044971 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2413dfc0 con 0x7faf24099b60 2024-09-16T14:47:39.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 10 monclient: _finish_auth 0 2024-09-16T14:47:39.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:47:39.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:39.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 5 monclient: authenticate success, global_id 114404 2024-09-16T14:47:39.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 10 monclient: _renew_subs 2024-09-16T14:47:39.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:39.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 1 -- 172.21.15.26:0/3374044971 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf2409e450 con 0x7faf24099b60 2024-09-16T14:47:39.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 10 monclient: _renew_subs 2024-09-16T14:47:39.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:39.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 1 -- 172.21.15.26:0/3374044971 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf2413bc60 con 0x7faf24099b60 2024-09-16T14:47:39.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 1 -- 172.21.15.26:0/3374044971 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf14001c60 con 0x7faf24099b60 2024-09-16T14:47:39.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:39.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 1 -- 172.21.15.26:0/3374044971 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faf14003a70 con 0x7faf24099b60 2024-09-16T14:47:39.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:39.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:39.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf227fc640 4 set_mon_vals no callback set 2024-09-16T14:47:39.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 10 monclient: dump: 2024-09-16T14:47:39.613 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:39.613 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:39.613 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.613 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.614 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:39.614 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:39.614 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:39.614 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:39.614 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:39.614 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:39.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 1 -- 172.21.15.26:0/3374044971 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7faf14003c50 con 0x7faf24099b60 2024-09-16T14:47:39.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:39.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:39.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:39.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf29bb5640 1 -- 172.21.15.26:0/3374044971 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf08005050 con 0x7faf24099b60 2024-09-16T14:47:39.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.599+0000 7faf20ff9640 1 -- 172.21.15.26:0/3374044971 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(130..130 src has 73..130) v4 ==== 7321+0+0 (unknown 1629976124 0 0) 0x7faf14044060 con 0x7faf24099b60 2024-09-16T14:47:39.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.603+0000 7faf1bfff640 10 monclient: get_auth_request con 0x7faf04036060 auth_method 0 2024-09-16T14:47:39.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.607+0000 7faf20ff9640 1 -- 172.21.15.26:0/3374044971 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7faf14014700 con 0x7faf24099b60 2024-09-16T14:47:39.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.607+0000 7faf20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:39.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.607+0000 7faf20ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:39.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.823+0000 7faf29bb5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:47:39.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.823+0000 7faf29bb5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:39.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.823+0000 7faf29bb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:39.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.823+0000 7faf29bb5640 1 -- 172.21.15.26:0/3374044971 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7faf08005880 con 0x7faf24099b60 2024-09-16T14:47:39.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.823+0000 7faf20ff9640 1 -- 172.21.15.26:0/3374044971 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3875194232) 0x7faf14012b70 con 0x7faf24099b60 2024-09-16T14:47:39.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.823+0000 7faf20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:39.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.823+0000 7faf20ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:39.829 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:47:39.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.827+0000 7faf29bb5640 1 -- 172.21.15.26:0/3374044971 >> v1:172.21.15.189:6812/3196161582 conn(0x7faf04036060 legacy=0x7faf04038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:39.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.827+0000 7faf29bb5640 10 monclient: shutdown 2024-09-16T14:47:39.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.827+0000 7faf29bb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:39.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.827+0000 7faf29bb5640 1 -- 172.21.15.26:0/3374044971 >> v1:172.21.15.26:6790/0 conn(0x7faf24099b60 legacy=0x7faf2413a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:39.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.827+0000 7faf29bb5640 1 -- 172.21.15.26:0/3374044971 shutdown_connections 2024-09-16T14:47:39.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.827+0000 7faf29bb5640 1 -- 172.21.15.26:0/3374044971 >> 172.21.15.26:0/3374044971 conn(0x7faf240a3d00 msgr2=0x7faf2409f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:39.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.827+0000 7faf29bb5640 1 -- 172.21.15.26:0/3374044971 shutdown_connections 2024-09-16T14:47:39.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.827+0000 7faf29bb5640 1 -- 172.21.15.26:0/3374044971 wait complete. 2024-09-16T14:47:39.847 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:47:39.847 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:47:39.847 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:47:39.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 10 monclient: get_monmap_and_config 2024-09-16T14:47:39.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 10 monclient: build_initial_monmap 2024-09-16T14:47:39.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 10 monclient: monmap: 2024-09-16T14:47:39.974 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:39.974 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:39.974 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:39.974610+0000 2024-09-16T14:47:39.974 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:39.974610+0000 2024-09-16T14:47:39.975 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:39.975 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:39.975 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:39.975 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:39.975 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:39.975 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:39.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 1 Processor -- start 2024-09-16T14:47:39.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 1 -- start start 2024-09-16T14:47:39.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 10 monclient: init 2024-09-16T14:47:39.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:39.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:39.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 10 monclient(hunting): picked mon.noname-c con 0x7fc340134d00 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:39.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 10 monclient(hunting): picked mon.noname-a con 0x7fc34009a1c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:39.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 10 monclient(hunting): picked mon.noname-b con 0x7fc3401357f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:39.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc34006ca10 con 0x7fc34009a1c0 2024-09-16T14:47:39.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3400998e0 con 0x7fc340134d00 2024-09-16T14:47:39.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc340099040 con 0x7fc3401357f0 2024-09-16T14:47:39.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:39.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 10 monclient(hunting): authenticate will time out at 1284.094661s 2024-09-16T14:47:39.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc3456fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc3401357f0 0x7fc34013d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55106/0 (socket says 172.21.15.26:55106) 2024-09-16T14:47:39.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc34009a1c0 0x7fc34013a230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47734/0 (socket says 172.21.15.26:47734) 2024-09-16T14:47:39.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc3456fc640 1 -- 172.21.15.26:0/2573308477 learned_addr learned my addr 172.21.15.26:0/2573308477 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:39.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 1 -- 172.21.15.26:0/2573308477 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690653745 0 0) 0x7fc34006ca10 con 0x7fc34009a1c0 2024-09-16T14:47:39.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:39.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:39.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:39.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient(hunting): my global_id is 124098 2024-09-16T14:47:39.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 1 -- 172.21.15.26:0/2573308477 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc330003530 con 0x7fc34009a1c0 2024-09-16T14:47:39.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 1 -- 172.21.15.26:0/2573308477 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584629863 0 0) 0x7fc340099040 con 0x7fc3401357f0 2024-09-16T14:47:39.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:39.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:39.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:39.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient(hunting): my global_id is 114407 2024-09-16T14:47:39.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 1 -- 172.21.15.26:0/2573308477 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc34006ca10 con 0x7fc3401357f0 2024-09-16T14:47:39.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 1 -- 172.21.15.26:0/2573308477 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45718023 0 0) 0x7fc3400998e0 con 0x7fc340134d00 2024-09-16T14:47:39.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:39.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:39.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:39.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient(hunting): my global_id is 95038 2024-09-16T14:47:39.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 1 -- 172.21.15.26:0/2573308477 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc340099040 con 0x7fc340134d00 2024-09-16T14:47:39.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 1 -- 172.21.15.26:0/2573308477 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2544888969 0 0) 0x7fc330003530 con 0x7fc34009a1c0 2024-09-16T14:47:39.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 1 -- 172.21.15.26:0/2573308477 >> v1:172.21.15.26:6790/0 conn(0x7fc3401357f0 legacy=0x7fc34013d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:39.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 1 -- 172.21.15.26:0/2573308477 >> v1:172.21.15.189:6789/0 conn(0x7fc340134d00 legacy=0x7fc3401350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:39.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:39.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 1 monclient: found mon.noname-a 2024-09-16T14:47:39.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:39.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:47:39.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 1 -- 172.21.15.26:0/2573308477 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc340099320 con 0x7fc34009a1c0 2024-09-16T14:47:39.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:39.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:39.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:39.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 1 -- 172.21.15.26:0/2573308477 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc32c001620 con 0x7fc34009a1c0 2024-09-16T14:47:39.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:39.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:39.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient: dump: 2024-09-16T14:47:39.982 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:39.982 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:39.982 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.982 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.982 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:39.982 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:39.982 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:39.982 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:39.982 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:39.982 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:39.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 5 monclient: authenticate success, global_id 124098 2024-09-16T14:47:39.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:39.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 1 -- 172.21.15.26:0/2573308477 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc32c001900 con 0x7fc34009a1c0 2024-09-16T14:47:39.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:39.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 1 -- 172.21.15.26:0/2573308477 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc32c001c90 con 0x7fc34009a1c0 2024-09-16T14:47:39.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:39.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc34715f640 4 set_mon_vals no callback set 2024-09-16T14:47:39.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:39.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:39.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.971+0000 7fc33f7fe640 10 monclient: dump: 2024-09-16T14:47:39.983 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:39.983 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:39.983 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.983 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.983 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:39.984 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:39.984 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:39.984 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:39.984 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:39.984 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:39.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 10 monclient: shutdown 2024-09-16T14:47:39.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:39.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 1 -- 172.21.15.26:0/2573308477 >> v1:172.21.15.26:6789/0 conn(0x7fc34009a1c0 legacy=0x7fc34013a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:39.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 1 -- 172.21.15.26:0/2573308477 shutdown_connections 2024-09-16T14:47:39.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 1 -- 172.21.15.26:0/2573308477 >> 172.21.15.26:0/2573308477 conn(0x7fc3400a0da0 msgr2=0x7fc3400a11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:39.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 1 -- 172.21.15.26:0/2573308477 shutdown_connections 2024-09-16T14:47:39.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 1 -- 172.21.15.26:0/2573308477 wait complete. 2024-09-16T14:47:39.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 10 monclient: build_initial_monmap 2024-09-16T14:47:39.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 10 monclient: monmap: 2024-09-16T14:47:39.985 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:39.985 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:39.985 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:39.985 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:39.985 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:39.985 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:39.985 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:39.985 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:39.985 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:39.985 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:39.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 1 Processor -- start 2024-09-16T14:47:39.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 1 -- start start 2024-09-16T14:47:39.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 10 monclient: init 2024-09-16T14:47:39.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:39.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:39.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 10 monclient(hunting): picked mon.noname-a con 0x7fc34009a1c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:39.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 10 monclient(hunting): picked mon.noname-b con 0x7fc340134d00 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:39.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 10 monclient(hunting): picked mon.noname-c con 0x7fc3401357f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:39.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc340121160 con 0x7fc34009a1c0 2024-09-16T14:47:39.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc34009b2c0 con 0x7fc340134d00 2024-09-16T14:47:39.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc340120760 con 0x7fc3401357f0 2024-09-16T14:47:39.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:39.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 10 monclient(hunting): authenticate will time out at 1284.097894s 2024-09-16T14:47:39.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc3456fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc3401357f0 0x7fc3401d1ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55114/0 (socket says 172.21.15.26:55114) 2024-09-16T14:47:39.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc3456fc640 1 -- 172.21.15.26:0/2871536364 learned_addr learned my addr 172.21.15.26:0/2871536364 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:39.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc344efb640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc34009a1c0 0x7fc3401340f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47750/0 (socket says 172.21.15.26:47750) 2024-09-16T14:47:39.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447718166 0 0) 0x7fc340120760 con 0x7fc3401357f0 2024-09-16T14:47:39.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:39.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:39.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:39.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): my global_id is 114410 2024-09-16T14:47:39.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc324003530 con 0x7fc3401357f0 2024-09-16T14:47:39.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642926089 0 0) 0x7fc340121160 con 0x7fc34009a1c0 2024-09-16T14:47:39.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:39.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:39.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:39.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): my global_id is 124101 2024-09-16T14:47:39.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc340120760 con 0x7fc34009a1c0 2024-09-16T14:47:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3019094915 0 0) 0x7fc324003530 con 0x7fc3401357f0 2024-09-16T14:47:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc340121160 con 0x7fc3401357f0 2024-09-16T14:47:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc338001bc0 con 0x7fc3401357f0 2024-09-16T14:47:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): dump: 2024-09-16T14:47:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:39.993 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:39.994 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156763224 0 0) 0x7fc34009b2c0 con 0x7fc340134d00 2024-09-16T14:47:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): my global_id is 95041 2024-09-16T14:47:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc324003530 con 0x7fc340134d00 2024-09-16T14:47:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2957386453 0 0) 0x7fc340120760 con 0x7fc34009a1c0 2024-09-16T14:47:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc34009b2c0 con 0x7fc34009a1c0 2024-09-16T14:47:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc3340018c0 con 0x7fc34009a1c0 2024-09-16T14:47:39.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient(hunting): dump: 2024-09-16T14:47:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:39.995 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 558523140 0 0) 0x7fc340121160 con 0x7fc3401357f0 2024-09-16T14:47:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 >> v1:172.21.15.189:6789/0 conn(0x7fc340134d00 legacy=0x7fc3401ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 >> v1:172.21.15.26:6789/0 conn(0x7fc34009a1c0 legacy=0x7fc3401340f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:47:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 monclient: found mon.c 2024-09-16T14:47:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3400a33c0 con 0x7fc3401357f0 2024-09-16T14:47:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:47:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 5 monclient: authenticate success, global_id 114410 2024-09-16T14:47:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 10 monclient: _renew_subs 2024-09-16T14:47:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 1 -- 172.21.15.26:0/2871536364 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3400a2fa0 con 0x7fc3401357f0 2024-09-16T14:47:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 10 monclient: _renew_subs 2024-09-16T14:47:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 1 -- 172.21.15.26:0/2871536364 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc340122220 con 0x7fc3401357f0 2024-09-16T14:47:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc338004b70 con 0x7fc3401357f0 2024-09-16T14:47:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc338002110 con 0x7fc3401357f0 2024-09-16T14:47:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33e7fc640 4 set_mon_vals no callback set 2024-09-16T14:47:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:39.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 10 monclient: dump: 2024-09-16T14:47:39.998 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:39.998 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:39.998 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.998 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:39.998 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:39.998 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:39.998 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:39.998 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:39.998 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:39.998 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc34715f640 1 -- 172.21.15.26:0/2871536364 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc320006590 con 0x7fc320002830 2024-09-16T14:47:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.975+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc33800f790 con 0x7fc3401357f0 2024-09-16T14:47:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(130..130 src has 73..130) v4 ==== 7321+0+0 (unknown 1629976124 0 0) 0x7fc3380442c0 con 0x7fc3401357f0 2024-09-16T14:47:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51932226 0 0) 0x7fc320006590 con 0x7fc320002830 2024-09-16T14:47:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 10 monclient(hunting): my global_id is 124104 2024-09-16T14:47:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc340121160 con 0x7fc320002830 2024-09-16T14:47:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3985841750 0 0) 0x7fc340121160 con 0x7fc320002830 2024-09-16T14:47:40.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc320006590 con 0x7fc320002830 2024-09-16T14:47:40.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc3340037a0 con 0x7fc320002830 2024-09-16T14:47:40.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33ffff640 10 monclient: get_auth_request con 0x7fc324036440 auth_method 0 2024-09-16T14:47:40.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:40.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 10 monclient: dump: 2024-09-16T14:47:40.001 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.001 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.002 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.002 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.002 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.002 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.002 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.002 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.002 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.002 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3309983284 0 0) 0x7fc320006590 con 0x7fc320002830 2024-09-16T14:47:40.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc32000a0e0 con 0x7fc320002830 2024-09-16T14:47:40.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc3340057d0 con 0x7fc320002830 2024-09-16T14:47:40.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:40.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:40.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 >> v1:172.21.15.26:6789/0 conn(0x7fc320002830 legacy=0x7fc320004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.979+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 >> v1:172.21.15.26:6789/0 conn(0x7fc320002830 legacy=0x7fc320004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:40.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.983+0000 7fc34715f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:47:40.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.983+0000 7fc34715f640 1 -- 172.21.15.26:0/2871536364 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc320009d80 con 0x7fc3200090a0 2024-09-16T14:47:40.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.983+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782267243 0 0) 0x7fc320009d80 con 0x7fc3200090a0 2024-09-16T14:47:40.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.983+0000 7fc33d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.983+0000 7fc33d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.983+0000 7fc33d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.983+0000 7fc33d7fa640 10 monclient(hunting): my global_id is 124107 2024-09-16T14:47:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.983+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc320006590 con 0x7fc3200090a0 2024-09-16T14:47:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.983+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2389423892 0 0) 0x7fc320006590 con 0x7fc3200090a0 2024-09-16T14:47:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.983+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc320009d80 con 0x7fc3200090a0 2024-09-16T14:47:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.983+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc334003de0 con 0x7fc3200090a0 2024-09-16T14:47:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.983+0000 7fc33d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.983+0000 7fc33d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.983+0000 7fc33d7fa640 10 monclient: dump: 2024-09-16T14:47:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.006 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.983+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 635886250 0 0) 0x7fc320009d80 con 0x7fc3200090a0 2024-09-16T14:47:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.983+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc32000c290 con 0x7fc3200090a0 2024-09-16T14:47:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.987+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 3941364123) 0x7fc334004180 con 0x7fc3200090a0 2024-09-16T14:47:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.987+0000 7fc33d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:47:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.987+0000 7fc33d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.987+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 >> v1:172.21.15.26:6789/0 conn(0x7fc3200090a0 legacy=0x7fc320009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.987+0000 7fc33d7fa640 1 -- 172.21.15.26:0/2871536364 >> v1:172.21.15.26:6789/0 conn(0x7fc3200090a0 legacy=0x7fc320009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:40.007 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:47:40.007 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:40.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:40.008 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:47:40.008 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 48, 2024-09-16T14:47:40.008 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:47:40.008 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:47:40.008 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:47:40.008 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:47:40.008 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:40.008 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:47:40.008 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:40.008 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:47:40.008 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:47:40.009 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:40.009 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:40.009 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:40.009 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:40.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:40.009 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:40.009 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:40.010 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:40.010 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:40.010 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:40.010 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:40.010 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:47:40.010 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:47:40.010 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:40.010 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:40.010 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:40.010 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:40.010 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:40.010 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:40.011 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:40.011 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:40.011 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:40.011 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:40.011 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.011 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:47:40.011 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:47:40.011 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:47:40.011 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:47:40.011 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:47:40.011 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:47:40.011 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:40.011 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:40.012 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:47:40.012 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:47:40.012 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:47:40.012 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:47:40.012 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:47:40.012 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:47:40.012 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:47:40.012 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:40.012 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:47:40.012 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:40.012 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:40.012 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:40.012 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:40.012 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:40.013 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:40.013 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:40.013 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:40.013 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:40.013 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:40.013 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:40.013 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:47:40.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.013 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:47:40.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:40.014 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:40.014 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:40.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:40.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.014 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:40.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:47:40.014 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:40.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:47:40.015 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:47:40.015 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:40.015 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:40.015 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:40.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:40.015 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:40.015 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:40.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:40.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:40.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:47:40.016 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:40.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:47:40.016 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:47:40.016 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:40.016 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:40.016 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:40.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:40.017 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:40.017 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:40.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:40.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.017 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:40.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:47:40.018 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:40.018 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.018 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.018 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:47:40.019 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:47:40.019 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:40.019 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:40.019 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:40.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.019 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.019 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:47:40.019 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:47:40.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.019 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:40.019 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:40.020 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:40.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.020 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:40.020 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:47:40.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.020 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:40.020 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:40.020 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:40.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.020 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.020 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:47:40.021 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:47:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.987+0000 7fc34715f640 1 -- 172.21.15.26:0/2871536364 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc324036440 legacy=0x7fc324038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.987+0000 7fc34715f640 10 monclient: shutdown 2024-09-16T14:47:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.987+0000 7fc34715f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.987+0000 7fc34715f640 1 -- 172.21.15.26:0/2871536364 >> v1:172.21.15.26:6790/0 conn(0x7fc3401357f0 legacy=0x7fc3401d1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.987+0000 7fc3456fc640 1 -- 172.21.15.26:0/2871536364 reap_dead start 2024-09-16T14:47:40.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.987+0000 7fc3456fc640 1 -- 172.21.15.26:0/2871536364 reap_dead start 2024-09-16T14:47:40.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.987+0000 7fc34715f640 1 -- 172.21.15.26:0/2871536364 shutdown_connections 2024-09-16T14:47:40.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.987+0000 7fc34715f640 1 -- 172.21.15.26:0/2871536364 >> 172.21.15.26:0/2871536364 conn(0x7fc3400a0da0 msgr2=0x7fc3400a35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:40.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.987+0000 7fc34715f640 1 -- 172.21.15.26:0/2871536364 shutdown_connections 2024-09-16T14:47:40.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:39.987+0000 7fc34715f640 1 -- 172.21.15.26:0/2871536364 wait complete. 2024-09-16T14:47:40.023 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:47:40.023 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:47:40.024 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:47:40.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 10 monclient: get_monmap_and_config 2024-09-16T14:47:40.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 10 monclient: build_initial_monmap 2024-09-16T14:47:40.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 10 monclient: monmap: 2024-09-16T14:47:40.142 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:40.142 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.142 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:40.141954+0000 2024-09-16T14:47:40.142 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:40.141954+0000 2024-09-16T14:47:40.142 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:40.142 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.142 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:40.142 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:40.142 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:40.142 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 1 Processor -- start 2024-09-16T14:47:40.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 1 -- start start 2024-09-16T14:47:40.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 10 monclient: init 2024-09-16T14:47:40.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:40.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:40.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 10 monclient(hunting): picked mon.noname-b con 0x7ffa9c09a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:40.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 10 monclient(hunting): picked mon.noname-a con 0x7ffa9c09d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:40.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 10 monclient(hunting): picked mon.noname-c con 0x7ffa9c09b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:40.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c09dc60 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c099ae0 con 0x7ffa9c09b040 2024-09-16T14:47:40.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c09b8f0 con 0x7ffa9c09a550 2024-09-16T14:47:40.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:40.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 10 monclient(hunting): authenticate will time out at 1284.262148s 2024-09-16T14:47:40.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa9a7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ffa9c09d1b0 0x7ffa9c136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47782/0 (socket says 172.21.15.26:47782) 2024-09-16T14:47:40.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa9affd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ffa9c09a550 0x7ffa9c09a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55128/0 (socket says 172.21.15.26:55128) 2024-09-16T14:47:40.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa9a7fc640 1 -- 172.21.15.26:0/3737596898 learned_addr learned my addr 172.21.15.26:0/3737596898 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:40.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 1 -- 172.21.15.26:0/3737596898 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784476490 0 0) 0x7ffa9c09b8f0 con 0x7ffa9c09a550 2024-09-16T14:47:40.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient(hunting): my global_id is 114413 2024-09-16T14:47:40.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 1 -- 172.21.15.26:0/3737596898 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa88003530 con 0x7ffa9c09a550 2024-09-16T14:47:40.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 1 -- 172.21.15.26:0/3737596898 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204506505 0 0) 0x7ffa9c09dc60 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient(hunting): my global_id is 124110 2024-09-16T14:47:40.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 1 -- 172.21.15.26:0/3737596898 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa9c09b8f0 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 1 -- 172.21.15.26:0/3737596898 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 850412037 0 0) 0x7ffa88003530 con 0x7ffa9c09a550 2024-09-16T14:47:40.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 1 -- 172.21.15.26:0/3737596898 >> v1:172.21.15.189:6789/0 conn(0x7ffa9c09b040 legacy=0x7ffa9c13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 1 -- 172.21.15.26:0/3737596898 >> v1:172.21.15.26:6789/0 conn(0x7ffa9c09d1b0 legacy=0x7ffa9c136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:47:40.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 1 monclient: found mon.noname-b 2024-09-16T14:47:40.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:40.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:47:40.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 1 -- 172.21.15.26:0/3737596898 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9c09bbd0 con 0x7ffa9c09a550 2024-09-16T14:47:40.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient: _finish_auth 0 2024-09-16T14:47:40.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:40.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:40.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 1 -- 172.21.15.26:0/3737596898 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffa8c001620 con 0x7ffa9c09a550 2024-09-16T14:47:40.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:40.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient: dump: 2024-09-16T14:47:40.149 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.149 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.149 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.149 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.149 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.149 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.149 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.149 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.149 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.149 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 5 monclient: authenticate success, global_id 114413 2024-09-16T14:47:40.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:40.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 1 -- 172.21.15.26:0/3737596898 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa8c001840 con 0x7ffa9c09a550 2024-09-16T14:47:40.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:40.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 1 -- 172.21.15.26:0/3737596898 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffa8c001c10 con 0x7ffa9c09a550 2024-09-16T14:47:40.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:40.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffa99ffb640 10 monclient: dump: 2024-09-16T14:47:40.150 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.150 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.150 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.150 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.150 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.150 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.150 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.151 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.151 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.151 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:40.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 4 set_mon_vals no callback set 2024-09-16T14:47:40.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 10 monclient: shutdown 2024-09-16T14:47:40.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:40.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 1 -- 172.21.15.26:0/3737596898 >> v1:172.21.15.26:6790/0 conn(0x7ffa9c09a550 legacy=0x7ffa9c09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 1 -- 172.21.15.26:0/3737596898 shutdown_connections 2024-09-16T14:47:40.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 1 -- 172.21.15.26:0/3737596898 >> 172.21.15.26:0/3737596898 conn(0x7ffa9c09fb50 msgr2=0x7ffa9c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:40.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 1 -- 172.21.15.26:0/3737596898 shutdown_connections 2024-09-16T14:47:40.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.139+0000 7ffaa13d6640 1 -- 172.21.15.26:0/3737596898 wait complete. 2024-09-16T14:47:40.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 10 monclient: build_initial_monmap 2024-09-16T14:47:40.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 10 monclient: monmap: 2024-09-16T14:47:40.152 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:40.152 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:40.152 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:40.152 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:40.152 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:40.152 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:40.152 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:40.152 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:40.152 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:40.152 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 1 Processor -- start 2024-09-16T14:47:40.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 1 -- start start 2024-09-16T14:47:40.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 10 monclient: init 2024-09-16T14:47:40.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:40.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:40.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 10 monclient(hunting): picked mon.noname-b con 0x7ffa9c09a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:40.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 10 monclient(hunting): picked mon.noname-a con 0x7ffa9c09b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:40.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 10 monclient(hunting): picked mon.noname-c con 0x7ffa9c09d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:40.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c13af60 con 0x7ffa9c09b040 2024-09-16T14:47:40.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c06d4f0 con 0x7ffa9c09a550 2024-09-16T14:47:40.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c06e1a0 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:40.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 10 monclient(hunting): authenticate will time out at 1284.265341s 2024-09-16T14:47:40.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa9b7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ffa9c09d1b0 0x7ffa9c1d1730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55142/0 (socket says 172.21.15.26:55142) 2024-09-16T14:47:40.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa9b7fe640 1 -- 172.21.15.26:0/2703169935 learned_addr learned my addr 172.21.15.26:0/2703169935 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:40.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411655220 0 0) 0x7ffa9c06e1a0 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): my global_id is 114416 2024-09-16T14:47:40.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa7c003530 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033264496 0 0) 0x7ffa9c13af60 con 0x7ffa9c09b040 2024-09-16T14:47:40.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): my global_id is 124113 2024-09-16T14:47:40.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa9c06e1a0 con 0x7ffa9c09b040 2024-09-16T14:47:40.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 856310837 0 0) 0x7ffa7c003530 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa9c13af60 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffa94001cd0 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:40.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): dump: 2024-09-16T14:47:40.160 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.160 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.160 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.160 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.160 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.160 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.160 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.160 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.160 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.160 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:40.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306971674 0 0) 0x7ffa9c06d4f0 con 0x7ffa9c09a550 2024-09-16T14:47:40.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): my global_id is 95047 2024-09-16T14:47:40.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa7c003530 con 0x7ffa9c09a550 2024-09-16T14:47:40.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1950464148 0 0) 0x7ffa9c06e1a0 con 0x7ffa9c09b040 2024-09-16T14:47:40.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa9c06d4f0 con 0x7ffa9c09b040 2024-09-16T14:47:40.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffa840019e0 con 0x7ffa9c09b040 2024-09-16T14:47:40.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:40.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): dump: 2024-09-16T14:47:40.161 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.161 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.161 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.162 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.162 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.162 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.162 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.162 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.162 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.162 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 758010918 0 0) 0x7ffa9c13af60 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 >> v1:172.21.15.189:6789/0 conn(0x7ffa9c09a550 legacy=0x7ffa9c06c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 >> v1:172.21.15.26:6789/0 conn(0x7ffa9c09b040 legacy=0x7ffa9c06cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:40.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 monclient: found mon.c 2024-09-16T14:47:40.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:40.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:40.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9c0a24d0 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:40.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:40.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:40.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 5 monclient: authenticate success, global_id 114416 2024-09-16T14:47:40.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 10 monclient: _renew_subs 2024-09-16T14:47:40.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:40.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 1 -- 172.21.15.26:0/2703169935 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa9c09a240 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 10 monclient: _renew_subs 2024-09-16T14:47:40.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:40.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 1 -- 172.21.15.26:0/2703169935 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9c13a2c0 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa940039a0 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:40.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffa94003cb0 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:40.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa98ff9640 4 set_mon_vals no callback set 2024-09-16T14:47:40.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient: dump: 2024-09-16T14:47:40.164 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.164 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.164 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.165 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.165 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.165 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.165 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.165 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.165 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.165 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:40.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffaa13d6640 1 -- 172.21.15.26:0/2703169935 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ffa80006590 con 0x7ffa80002830 2024-09-16T14:47:40.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ffa94003e10 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(130..130 src has 73..130) v4 ==== 7321+0+0 (unknown 1629976124 0 0) 0x7ffa94044250 con 0x7ffa9c09d1b0 2024-09-16T14:47:40.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322021302 0 0) 0x7ffa80006590 con 0x7ffa80002830 2024-09-16T14:47:40.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient(hunting): my global_id is 114419 2024-09-16T14:47:40.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa9c13af60 con 0x7ffa80002830 2024-09-16T14:47:40.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3790779451 0 0) 0x7ffa9c13af60 con 0x7ffa80002830 2024-09-16T14:47:40.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa80006590 con 0x7ffa80002830 2024-09-16T14:47:40.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffa8c0011d0 con 0x7ffa80002830 2024-09-16T14:47:40.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa9a7fc640 10 monclient: get_auth_request con 0x7ffa7c0363f0 auth_method 0 2024-09-16T14:47:40.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:40.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.143+0000 7ffa937fe640 10 monclient: dump: 2024-09-16T14:47:40.166 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.166 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.167 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.167 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.167 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.167 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.167 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.167 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.167 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.167 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.147+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1460143757 0 0) 0x7ffa80006590 con 0x7ffa80002830 2024-09-16T14:47:40.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.147+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa8000a0e0 con 0x7ffa80002830 2024-09-16T14:47:40.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.147+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ffa8c001540 con 0x7ffa80002830 2024-09-16T14:47:40.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.147+0000 7ffa937fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:40.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.147+0000 7ffa937fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:40.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.147+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 >> v1:172.21.15.26:6790/0 conn(0x7ffa80002830 legacy=0x7ffa80004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.147+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 >> v1:172.21.15.26:6790/0 conn(0x7ffa80002830 legacy=0x7ffa80004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:40.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffaa13d6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:47:40.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffaa13d6640 1 -- 172.21.15.26:0/2703169935 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ffa80009da0 con 0x7ffa800090c0 2024-09-16T14:47:40.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636592454 0 0) 0x7ffa80009da0 con 0x7ffa800090c0 2024-09-16T14:47:40.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 10 monclient(hunting): my global_id is 114422 2024-09-16T14:47:40.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa80006590 con 0x7ffa800090c0 2024-09-16T14:47:40.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2540322517 0 0) 0x7ffa80006590 con 0x7ffa800090c0 2024-09-16T14:47:40.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa80009da0 con 0x7ffa800090c0 2024-09-16T14:47:40.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffa8c004c00 con 0x7ffa800090c0 2024-09-16T14:47:40.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:40.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 10 monclient: dump: 2024-09-16T14:47:40.169 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.169 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.169 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.169 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.169 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.169 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.169 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.169 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.169 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.169 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1490832289 0 0) 0x7ffa80009da0 con 0x7ffa800090c0 2024-09-16T14:47:40.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ffa8000c290 con 0x7ffa800090c0 2024-09-16T14:47:40.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 505057183) 0x7ffa8c006080 con 0x7ffa800090c0 2024-09-16T14:47:40.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:47:40.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:40.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 >> v1:172.21.15.26:6790/0 conn(0x7ffa800090c0 legacy=0x7ffa80009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.151+0000 7ffa937fe640 1 -- 172.21.15.26:0/2703169935 >> v1:172.21.15.26:6790/0 conn(0x7ffa800090c0 legacy=0x7ffa80009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:40.170 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:47:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:47:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 48, 2024-09-16T14:47:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:47:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:47:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:47:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:47:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:47:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:47:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:47:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:47:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:47:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:40.172 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:47:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:47:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:47:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:47:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:47:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:47:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:47:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:47:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:47:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:47:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:47:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:47:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:47:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:47:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:47:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:47:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:47:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:47:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:47:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:47:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:47:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:47:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:40.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:47:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:47:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:47:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:40.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:47:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:47:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:47:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.180 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:47:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:47:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:47:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:47:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.181 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.182 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:47:40.182 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:47:40.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.155+0000 7ffaa13d6640 1 -- 172.21.15.26:0/2703169935 >> v1:172.21.15.189:6812/3196161582 conn(0x7ffa7c0363f0 legacy=0x7ffa7c0388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.155+0000 7ffaa13d6640 10 monclient: shutdown 2024-09-16T14:47:40.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.155+0000 7ffaa13d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:40.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.155+0000 7ffaa13d6640 1 -- 172.21.15.26:0/2703169935 >> v1:172.21.15.26:6790/0 conn(0x7ffa9c09d1b0 legacy=0x7ffa9c1d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.155+0000 7ffa9b7fe640 1 -- 172.21.15.26:0/2703169935 reap_dead start 2024-09-16T14:47:40.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.155+0000 7ffa9b7fe640 1 -- 172.21.15.26:0/2703169935 reap_dead start 2024-09-16T14:47:40.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.155+0000 7ffaa13d6640 1 -- 172.21.15.26:0/2703169935 shutdown_connections 2024-09-16T14:47:40.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.155+0000 7ffaa13d6640 1 -- 172.21.15.26:0/2703169935 >> 172.21.15.26:0/2703169935 conn(0x7ffa9c09fb50 msgr2=0x7ffa9c09eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:40.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.155+0000 7ffaa13d6640 1 -- 172.21.15.26:0/2703169935 shutdown_connections 2024-09-16T14:47:40.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.155+0000 7ffaa13d6640 1 -- 172.21.15.26:0/2703169935 wait complete. 2024-09-16T14:47:40.184 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:47:40.184 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:47:40.185 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:47:40.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.266+0000 7fbd47fff640 1 -- 172.21.15.189:0/3449334345 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}]=0 v131) v1 ==== 139+0+0 (unknown 3077131513 0 0) 0x7fbd54014560 con 0x7fbd600a25f0 2024-09-16T14:47:40.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.266+0000 7fbd47fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T14:47:40.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.266+0000 7fbd47fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:40.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.270+0000 7fbd65c1e640 1 -- 172.21.15.189:0/3449334345 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbd4c0361d0 legacy=0x7fbd4c038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.270+0000 7fbd65c1e640 10 monclient: shutdown 2024-09-16T14:47:40.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.270+0000 7fbd65c1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:40.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.270+0000 7fbd65c1e640 1 -- 172.21.15.189:0/3449334345 >> v1:172.21.15.189:6789/0 conn(0x7fbd600a25f0 legacy=0x7fbd60139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.270+0000 7fbd65c1e640 1 -- 172.21.15.189:0/3449334345 shutdown_connections 2024-09-16T14:47:40.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.270+0000 7fbd65c1e640 1 -- 172.21.15.189:0/3449334345 >> 172.21.15.189:0/3449334345 conn(0x7fbd600a7dd0 msgr2=0x7fbd6009d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:40.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.270+0000 7fbd65c1e640 1 -- 172.21.15.189:0/3449334345 shutdown_connections 2024-09-16T14:47:40.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.270+0000 7fbd65c1e640 1 -- 172.21.15.189:0/3449334345 wait complete. 2024-09-16T14:47:40.295 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set ls 2024-09-16T14:47:40.295 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c fooset 2024-09-16T14:47:40.295 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 1 2024-09-16T14:47:40.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 10 monclient: get_monmap_and_config 2024-09-16T14:47:40.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 10 monclient: build_initial_monmap 2024-09-16T14:47:40.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 10 monclient: monmap: 2024-09-16T14:47:40.298 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:40.298 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.299 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:40.298994+0000 2024-09-16T14:47:40.299 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:40.298994+0000 2024-09-16T14:47:40.299 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:40.299 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.299 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:40.299 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:40.299 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:40.299 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 1 Processor -- start 2024-09-16T14:47:40.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 1 -- start start 2024-09-16T14:47:40.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 10 monclient: init 2024-09-16T14:47:40.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:40.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:40.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 10 monclient(hunting): picked mon.noname-a con 0x7fcd7c0a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:40.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 10 monclient(hunting): picked mon.noname-b con 0x7fcd7c099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:40.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 10 monclient(hunting): picked mon.noname-c con 0x7fcd7c0a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:40.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd7c098ea0 con 0x7fcd7c0a1c00 2024-09-16T14:47:40.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd7c0997b0 con 0x7fcd7c0a26f0 2024-09-16T14:47:40.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd7c099460 con 0x7fcd7c099b60 2024-09-16T14:47:40.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:40.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 10 monclient(hunting): authenticate will time out at 1284.418977s 2024-09-16T14:47:40.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd7affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fcd7c0a1c00 0x7fcd7c0a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47800/0 (socket says 172.21.15.26:47800) 2024-09-16T14:47:40.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd7a7fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fcd7c099b60 0x7fcd7c139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55160/0 (socket says 172.21.15.26:55160) 2024-09-16T14:47:40.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd7affd640 1 -- 172.21.15.26:0/1846587101 learned_addr learned my addr 172.21.15.26:0/1846587101 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:40.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 1 -- 172.21.15.26:0/1846587101 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511133052 0 0) 0x7fcd7c099460 con 0x7fcd7c099b60 2024-09-16T14:47:40.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient(hunting): my global_id is 114425 2024-09-16T14:47:40.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 1 -- 172.21.15.26:0/1846587101 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd64003530 con 0x7fcd7c099b60 2024-09-16T14:47:40.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 1 -- 172.21.15.26:0/1846587101 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033357842 0 0) 0x7fcd7c098ea0 con 0x7fcd7c0a1c00 2024-09-16T14:47:40.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient(hunting): my global_id is 124116 2024-09-16T14:47:40.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 1 -- 172.21.15.26:0/1846587101 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd7c099460 con 0x7fcd7c0a1c00 2024-09-16T14:47:40.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 1 -- 172.21.15.26:0/1846587101 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960783795 0 0) 0x7fcd7c0997b0 con 0x7fcd7c0a26f0 2024-09-16T14:47:40.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient(hunting): my global_id is 95050 2024-09-16T14:47:40.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 1 -- 172.21.15.26:0/1846587101 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd7c098ea0 con 0x7fcd7c0a26f0 2024-09-16T14:47:40.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 1 -- 172.21.15.26:0/1846587101 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2236762087 0 0) 0x7fcd64003530 con 0x7fcd7c099b60 2024-09-16T14:47:40.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 1 -- 172.21.15.26:0/1846587101 >> v1:172.21.15.189:6789/0 conn(0x7fcd7c0a26f0 legacy=0x7fcd7c13d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 1 -- 172.21.15.26:0/1846587101 >> v1:172.21.15.26:6789/0 conn(0x7fcd7c0a1c00 legacy=0x7fcd7c0a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:47:40.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 1 monclient: found mon.noname-b 2024-09-16T14:47:40.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:40.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:47:40.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 1 -- 172.21.15.26:0/1846587101 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c0a34b0 con 0x7fcd7c099b60 2024-09-16T14:47:40.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient: _finish_auth 0 2024-09-16T14:47:40.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:40.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:40.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 1 -- 172.21.15.26:0/1846587101 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcd70001620 con 0x7fcd7c099b60 2024-09-16T14:47:40.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:40.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient: dump: 2024-09-16T14:47:40.315 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.315 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.316 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.316 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.316 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.316 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.316 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.316 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.316 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.316 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 5 monclient: authenticate success, global_id 114425 2024-09-16T14:47:40.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:40.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 1 -- 172.21.15.26:0/1846587101 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd70001900 con 0x7fcd7c099b60 2024-09-16T14:47:40.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:40.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd79ffb640 1 -- 172.21.15.26:0/1846587101 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcd70001cd0 con 0x7fcd7c099b60 2024-09-16T14:47:40.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:40.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.295+0000 7fcd817bd640 4 set_mon_vals no callback set 2024-09-16T14:47:40.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:40.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd79ffb640 10 monclient: dump: 2024-09-16T14:47:40.317 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.317 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.317 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.317 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.317 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.317 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.317 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.317 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.317 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.318 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 10 monclient: shutdown 2024-09-16T14:47:40.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:40.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 1 -- 172.21.15.26:0/1846587101 >> v1:172.21.15.26:6790/0 conn(0x7fcd7c099b60 legacy=0x7fcd7c139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 1 -- 172.21.15.26:0/1846587101 shutdown_connections 2024-09-16T14:47:40.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 1 -- 172.21.15.26:0/1846587101 >> 172.21.15.26:0/1846587101 conn(0x7fcd7c0a3d00 msgr2=0x7fcd7c0a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:40.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 1 -- 172.21.15.26:0/1846587101 shutdown_connections 2024-09-16T14:47:40.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 1 -- 172.21.15.26:0/1846587101 wait complete. 2024-09-16T14:47:40.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 10 monclient: build_initial_monmap 2024-09-16T14:47:40.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 10 monclient: monmap: 2024-09-16T14:47:40.318 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:40.318 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:40.318 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:40.319 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:40.319 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:40.319 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:40.319 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:40.319 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:40.319 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:40.319 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 1 Processor -- start 2024-09-16T14:47:40.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 1 -- start start 2024-09-16T14:47:40.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 10 monclient: init 2024-09-16T14:47:40.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:40.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:40.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 10 monclient(hunting): picked mon.noname-c con 0x7fcd7c099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:40.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 10 monclient(hunting): picked mon.noname-b con 0x7fcd7c0a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:40.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 10 monclient(hunting): picked mon.noname-a con 0x7fcd7c0a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:40.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd7c09f0d0 con 0x7fcd7c0a26f0 2024-09-16T14:47:40.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd7c09eb80 con 0x7fcd7c0a1c00 2024-09-16T14:47:40.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd7c13db90 con 0x7fcd7c099b60 2024-09-16T14:47:40.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:40.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 10 monclient(hunting): authenticate will time out at 1284.422135s 2024-09-16T14:47:40.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd7affd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fcd7c099b60 0x7fcd7c1c9b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55176/0 (socket says 172.21.15.26:55176) 2024-09-16T14:47:40.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd7affd640 1 -- 172.21.15.26:0/3252884156 learned_addr learned my addr 172.21.15.26:0/3252884156 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:40.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244056509 0 0) 0x7fcd7c13db90 con 0x7fcd7c099b60 2024-09-16T14:47:40.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): my global_id is 114428 2024-09-16T14:47:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd68003530 con 0x7fcd7c099b60 2024-09-16T14:47:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554141767 0 0) 0x7fcd7c09f0d0 con 0x7fcd7c0a26f0 2024-09-16T14:47:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): my global_id is 124119 2024-09-16T14:47:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd7c13db90 con 0x7fcd7c0a26f0 2024-09-16T14:47:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695349478 0 0) 0x7fcd7c09eb80 con 0x7fcd7c0a1c00 2024-09-16T14:47:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): my global_id is 95053 2024-09-16T14:47:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd7c09f0d0 con 0x7fcd7c0a1c00 2024-09-16T14:47:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3080295530 0 0) 0x7fcd68003530 con 0x7fcd7c099b60 2024-09-16T14:47:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd7c09eb80 con 0x7fcd7c099b60 2024-09-16T14:47:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcd6c0019e0 con 0x7fcd7c099b60 2024-09-16T14:47:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): dump: 2024-09-16T14:47:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.323 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.323 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:40.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2260862060 0 0) 0x7fcd7c13db90 con 0x7fcd7c0a26f0 2024-09-16T14:47:40.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd68003530 con 0x7fcd7c0a26f0 2024-09-16T14:47:40.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcd74001c40 con 0x7fcd7c0a26f0 2024-09-16T14:47:40.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:40.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): dump: 2024-09-16T14:47:40.323 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.323 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.323 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.323 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.324 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1324801650 0 0) 0x7fcd7c09f0d0 con 0x7fcd7c0a1c00 2024-09-16T14:47:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd7c13db90 con 0x7fcd7c0a1c00 2024-09-16T14:47:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcd700015e0 con 0x7fcd7c0a1c00 2024-09-16T14:47:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient(hunting): dump: 2024-09-16T14:47:40.324 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.325 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.325 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.325 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.325 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.325 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.325 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.325 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.325 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.325 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2254589822 0 0) 0x7fcd7c09eb80 con 0x7fcd7c099b60 2024-09-16T14:47:40.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 >> v1:172.21.15.189:6789/0 conn(0x7fcd7c0a1c00 legacy=0x7fcd7c1ceb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 >> v1:172.21.15.26:6789/0 conn(0x7fcd7c0a26f0 legacy=0x7fcd7c1d2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:40.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 monclient: found mon.c 2024-09-16T14:47:40.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:40.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:40.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c11e4c0 con 0x7fcd7c099b60 2024-09-16T14:47:40.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:40.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:40.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:40.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 5 monclient: authenticate success, global_id 114428 2024-09-16T14:47:40.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 10 monclient: _renew_subs 2024-09-16T14:47:40.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:40.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7c11d850 con 0x7fcd7c099b60 2024-09-16T14:47:40.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 10 monclient: _renew_subs 2024-09-16T14:47:40.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:40.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7c13d040 con 0x7fcd7c099b60 2024-09-16T14:47:40.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd6c001cb0 con 0x7fcd7c099b60 2024-09-16T14:47:40.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:40.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcd6c004c40 con 0x7fcd7c099b60 2024-09-16T14:47:40.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:40.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd637fe640 10 monclient: dump: 2024-09-16T14:47:40.327 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.327 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.327 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.327 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.327 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.327 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.327 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.328 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.328 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.328 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd78ff9640 4 set_mon_vals no callback set 2024-09-16T14:47:40.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:40.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.299+0000 7fcd817bd640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcd7c11e8c0 con 0x7fcd7c1ca250 2024-09-16T14:47:40.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fcd6c00f6b0 con 0x7fcd7c099b60 2024-09-16T14:47:40.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(131..131 src has 73..131) v4 ==== 7321+0+0 (unknown 1197112955 0 0) 0x7fcd6c043e70 con 0x7fcd7c099b60 2024-09-16T14:47:40.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd7b7fe640 10 monclient: get_auth_request con 0x7fcd68036420 auth_method 0 2024-09-16T14:47:40.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161688894 0 0) 0x7fcd7c11e8c0 con 0x7fcd7c1ca250 2024-09-16T14:47:40.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 10 monclient(hunting): my global_id is 95056 2024-09-16T14:47:40.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd7c09eb80 con 0x7fcd7c1ca250 2024-09-16T14:47:40.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 745826343 0 0) 0x7fcd7c09eb80 con 0x7fcd7c1ca250 2024-09-16T14:47:40.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd7c11e8c0 con 0x7fcd7c1ca250 2024-09-16T14:47:40.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcd70004c20 con 0x7fcd7c1ca250 2024-09-16T14:47:40.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:40.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 10 monclient: dump: 2024-09-16T14:47:40.329 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.329 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.329 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.329 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.329 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.330 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.330 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.330 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.330 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.330 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3819756725 0 0) 0x7fcd7c11e8c0 con 0x7fcd7c1ca250 2024-09-16T14:47:40.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd7c09e9f0 con 0x7fcd7c1ca250 2024-09-16T14:47:40.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fcd70004ec0 con 0x7fcd7c1ca250 2024-09-16T14:47:40.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:40.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:40.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 >> v1:172.21.15.189:6789/0 conn(0x7fcd7c1ca250 legacy=0x7fcd7c138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.303+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 >> v1:172.21.15.189:6789/0 conn(0x7fcd7c1ca250 legacy=0x7fcd7c138e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:40.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.307+0000 7fcd817bd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:47:40.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.307+0000 7fcd817bd640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcd7c11e6b0 con 0x7fcd7c02c280 2024-09-16T14:47:40.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210468002 0 0) 0x7fcd7c11e6b0 con 0x7fcd7c02c280 2024-09-16T14:47:40.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 10 monclient(hunting): my global_id is 95059 2024-09-16T14:47:40.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd7c11e8c0 con 0x7fcd7c02c280 2024-09-16T14:47:40.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3731167167 0 0) 0x7fcd7c11e8c0 con 0x7fcd7c02c280 2024-09-16T14:47:40.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd7c11e6b0 con 0x7fcd7c02c280 2024-09-16T14:47:40.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcd70005440 con 0x7fcd7c02c280 2024-09-16T14:47:40.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:40.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 10 monclient: dump: 2024-09-16T14:47:40.333 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.333 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.333 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.333 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.333 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.333 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.333 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.333 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.333 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.333 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 991341595 0 0) 0x7fcd7c11e6b0 con 0x7fcd7c02c280 2024-09-16T14:47:40.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcd7c11edf0 con 0x7fcd7c02c280 2024-09-16T14:47:40.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 3104838162) 0x7fcd700068e0 con 0x7fcd7c02c280 2024-09-16T14:47:40.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:47:40.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:40.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 >> v1:172.21.15.189:6789/0 conn(0x7fcd7c02c280 legacy=0x7fcd7c139540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd637fe640 1 -- 172.21.15.26:0/3252884156 >> v1:172.21.15.189:6789/0 conn(0x7fcd7c02c280 legacy=0x7fcd7c139540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:40.334 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:47:40.334 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:40.334 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:40.335 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:47:40.335 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 48, 2024-09-16T14:47:40.335 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:47:40.335 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:47:40.335 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:47:40.335 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:47:40.335 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:40.335 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:47:40.335 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:40.335 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:47:40.335 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:47:40.335 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:40.335 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:40.336 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:40.336 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:40.336 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:40.336 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:40.336 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:40.336 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:40.336 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:40.336 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:40.336 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:40.336 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:47:40.336 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:47:40.336 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:40.336 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:40.337 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:40.337 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:40.337 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:40.337 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:40.337 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:40.337 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:40.337 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:40.337 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:40.337 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.337 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:47:40.337 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:47:40.337 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:47:40.337 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:47:40.338 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:47:40.338 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:47:40.338 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:40.338 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:40.338 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:47:40.338 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:47:40.338 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:47:40.338 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:47:40.338 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:47:40.338 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:47:40.338 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:47:40.338 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:40.338 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:47:40.339 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:40.339 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:40.339 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:40.339 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:40.339 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:40.339 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:40.339 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:40.339 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:40.339 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:40.339 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:40.339 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:40.339 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:47:40.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.339 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:47:40.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.340 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:40.340 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:40.340 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:40.340 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:40.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.340 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:40.340 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:47:40.340 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:40.341 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.341 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.341 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:47:40.341 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:47:40.341 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:40.341 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:40.341 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:40.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.341 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:40.341 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:40.342 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:40.342 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:40.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.342 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:40.342 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:47:40.342 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:40.342 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.342 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.342 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:47:40.342 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:47:40.342 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:40.342 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:40.342 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:40.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.343 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:40.343 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:40.343 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:40.343 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:40.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.343 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:40.343 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:47:40.343 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:40.343 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.343 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:47:40.344 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:47:40.344 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:40.344 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:40.344 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:40.344 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.344 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.344 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:47:40.344 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:47:40.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.344 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:40.344 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:40.345 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:40.345 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.345 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:40.345 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:47:40.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.345 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:40.345 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:40.345 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:40.345 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.345 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:40.345 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:47:40.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:40.345 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:40.345 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:40.346 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:47:40.346 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:40.346 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:40.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:40.346 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:47:40.346 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:47:40.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd817bd640 1 -- 172.21.15.26:0/3252884156 >> v1:172.21.15.189:6812/3196161582 conn(0x7fcd68036420 legacy=0x7fcd680388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd817bd640 10 monclient: shutdown 2024-09-16T14:47:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd817bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd817bd640 1 -- 172.21.15.26:0/3252884156 >> v1:172.21.15.26:6790/0 conn(0x7fcd7c099b60 legacy=0x7fcd7c1c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd7b7fe640 1 -- 172.21.15.26:0/3252884156 reap_dead start 2024-09-16T14:47:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd7b7fe640 1 -- 172.21.15.26:0/3252884156 reap_dead start 2024-09-16T14:47:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd817bd640 1 -- 172.21.15.26:0/3252884156 shutdown_connections 2024-09-16T14:47:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd817bd640 1 -- 172.21.15.26:0/3252884156 >> 172.21.15.26:0/3252884156 conn(0x7fcd7c0a3d00 msgr2=0x7fcd7c0a0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd817bd640 1 -- 172.21.15.26:0/3252884156 shutdown_connections 2024-09-16T14:47:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.311+0000 7fcd817bd640 1 -- 172.21.15.26:0/3252884156 wait complete. 2024-09-16T14:47:40.348 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:47:40.349 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:47:40.349 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:47:40.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 10 monclient: get_monmap_and_config 2024-09-16T14:47:40.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 10 monclient: build_initial_monmap 2024-09-16T14:47:40.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 10 monclient: monmap: 2024-09-16T14:47:40.415 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:40.415 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.415 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:40.415016+0000 2024-09-16T14:47:40.415 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:40.415016+0000 2024-09-16T14:47:40.415 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:40.415 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:40.415 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:40.415 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:40.415 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:40.415 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:40.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 1 Processor -- start 2024-09-16T14:47:40.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 1 -- start start 2024-09-16T14:47:40.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 10 monclient: init 2024-09-16T14:47:40.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:40.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:40.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 10 monclient(hunting): picked mon.noname-a con 0x7f4bc80a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:40.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 10 monclient(hunting): picked mon.noname-c con 0x7f4bc80a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:40.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 10 monclient(hunting): picked mon.noname-b con 0x7f4bc813b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:40.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc80a30b0 con 0x7f4bc80a1b00 2024-09-16T14:47:40.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc80a2e30 con 0x7f4bc80a25f0 2024-09-16T14:47:40.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc809eb50 con 0x7f4bc813b430 2024-09-16T14:47:40.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:40.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 10 monclient(hunting): authenticate will time out at 1242.530781s 2024-09-16T14:47:40.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc5d9b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4bc80a25f0 0x7f4bc8139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33524/0 (socket says 172.21.15.189:33524) 2024-09-16T14:47:40.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc5d9b640 1 -- 172.21.15.189:0/1035371649 learned_addr learned my addr 172.21.15.189:0/1035371649 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:40.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 1 -- 172.21.15.189:0/1035371649 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035689578 0 0) 0x7f4bc80a2e30 con 0x7f4bc80a25f0 2024-09-16T14:47:40.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 10 monclient(hunting): my global_id is 95062 2024-09-16T14:47:40.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 1 -- 172.21.15.189:0/1035371649 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb0003530 con 0x7f4bc80a25f0 2024-09-16T14:47:40.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 1 -- 172.21.15.189:0/1035371649 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3642469024 0 0) 0x7f4bb0003530 con 0x7f4bc80a25f0 2024-09-16T14:47:40.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 1 -- 172.21.15.189:0/1035371649 >> v1:172.21.15.26:6790/0 conn(0x7f4bc813b430 legacy=0x7f4bc813d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 1 -- 172.21.15.189:0/1035371649 >> v1:172.21.15.26:6789/0 conn(0x7f4bc80a1b00 legacy=0x7f4bc80a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 10 monclient: _finish_hunting 0 2024-09-16T14:47:40.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 1 monclient: found mon.noname-c 2024-09-16T14:47:40.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:40.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:40.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 1 -- 172.21.15.189:0/1035371649 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc809ee30 con 0x7f4bc80a25f0 2024-09-16T14:47:40.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 10 monclient: _finish_auth 0 2024-09-16T14:47:40.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 10 monclient: _check_auth_tickets 2024-09-16T14:47:40.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:40.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 1 -- 172.21.15.189:0/1035371649 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4bbc001620 con 0x7f4bc80a25f0 2024-09-16T14:47:40.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:40.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 10 monclient: dump: 2024-09-16T14:47:40.421 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:40.421 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.421 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.421 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.421 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.422 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:40.422 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.422 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.422 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.422 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:40.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 5 monclient: authenticate success, global_id 95062 2024-09-16T14:47:40.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:40.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 1 -- 172.21.15.189:0/1035371649 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bbc0018d0 con 0x7f4bc80a25f0 2024-09-16T14:47:40.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:40.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 1 -- 172.21.15.189:0/1035371649 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4bbc001ca0 con 0x7f4bc80a25f0 2024-09-16T14:47:40.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:40.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bccc15640 4 set_mon_vals no callback set 2024-09-16T14:47:40.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.410+0000 7f4bc559a640 10 monclient: dump: 2024-09-16T14:47:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.423 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient: shutdown 2024-09-16T14:47:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 1 -- 172.21.15.189:0/1035371649 >> v1:172.21.15.189:6789/0 conn(0x7f4bc80a25f0 legacy=0x7f4bc8139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 1 -- 172.21.15.189:0/1035371649 shutdown_connections 2024-09-16T14:47:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 1 -- 172.21.15.189:0/1035371649 >> 172.21.15.189:0/1035371649 conn(0x7f4bc80a7dd0 msgr2=0x7f4bc80a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 1 -- 172.21.15.189:0/1035371649 shutdown_connections 2024-09-16T14:47:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 1 -- 172.21.15.189:0/1035371649 wait complete. 2024-09-16T14:47:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient: build_initial_monmap 2024-09-16T14:47:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient: monmap: 2024-09-16T14:47:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:40.425 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 1 Processor -- start 2024-09-16T14:47:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 1 -- start start 2024-09-16T14:47:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient: init 2024-09-16T14:47:40.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient(hunting): picked mon.noname-c con 0x7f4bc80a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient(hunting): picked mon.noname-a con 0x7f4bc80a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient(hunting): picked mon.noname-b con 0x7f4bc813b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc813a2b0 con 0x7f4bc80a25f0 2024-09-16T14:47:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc811fda0 con 0x7f4bc813b430 2024-09-16T14:47:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc811e570 con 0x7f4bc80a1b00 2024-09-16T14:47:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient(hunting): authenticate will time out at 1242.534235s 2024-09-16T14:47:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bc6d9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4bc813b430 0x7f4bc81d1810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33536/0 (socket says 172.21.15.189:33536) 2024-09-16T14:47:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bc6d9d640 1 -- 172.21.15.189:0/2591235488 learned_addr learned my addr 172.21.15.189:0/2591235488 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567288685 0 0) 0x7f4bc811fda0 con 0x7f4bc813b430 2024-09-16T14:47:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient(hunting): my global_id is 95065 2024-09-16T14:47:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba8003530 con 0x7f4bc813b430 2024-09-16T14:47:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 990324355 0 0) 0x7f4ba8003530 con 0x7f4bc813b430 2024-09-16T14:47:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bc811fda0 con 0x7f4bc813b430 2024-09-16T14:47:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4bc0001c20 con 0x7f4bc813b430 2024-09-16T14:47:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient(hunting): dump: 2024-09-16T14:47:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.431 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.431 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.431 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:40.431 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.431 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.431 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.431 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:40.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:40.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663329055 0 0) 0x7f4bc813a2b0 con 0x7f4bc80a25f0 2024-09-16T14:47:40.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient(hunting): my global_id is 124125 2024-09-16T14:47:40.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba8003530 con 0x7f4bc80a25f0 2024-09-16T14:47:40.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674358403 0 0) 0x7f4bc811e570 con 0x7f4bc80a1b00 2024-09-16T14:47:40.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient(hunting): my global_id is 114434 2024-09-16T14:47:40.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc813a2b0 con 0x7f4bc80a1b00 2024-09-16T14:47:40.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3842449960 0 0) 0x7f4bc811fda0 con 0x7f4bc813b430 2024-09-16T14:47:40.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 >> v1:172.21.15.26:6790/0 conn(0x7f4bc80a1b00 legacy=0x7f4bc811d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 >> v1:172.21.15.26:6789/0 conn(0x7f4bc80a25f0 legacy=0x7f4bc811de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:40.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 1 monclient: found mon.b 2024-09-16T14:47:40.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc813a7c0 con 0x7f4bc813b430 2024-09-16T14:47:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:47:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 5 monclient: authenticate success, global_id 95065 2024-09-16T14:47:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient: _renew_subs 2024-09-16T14:47:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 1 -- 172.21.15.189:0/2591235488 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc8120080 con 0x7f4bc813b430 2024-09-16T14:47:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient: _renew_subs 2024-09-16T14:47:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 1 -- 172.21.15.189:0/2591235488 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc813b020 con 0x7f4bc813b430 2024-09-16T14:47:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bc0001ff0 con 0x7f4bc813b430 2024-09-16T14:47:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4bc0003e70 con 0x7f4bc813b430 2024-09-16T14:47:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bb6ffd640 10 monclient: dump: 2024-09-16T14:47:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.435 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bc4d99640 4 set_mon_vals no callback set 2024-09-16T14:47:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.414+0000 7f4bccc15640 1 -- 172.21.15.189:0/2591235488 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bac005050 con 0x7f4bc813b430 2024-09-16T14:47:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.418+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4bc0003fd0 con 0x7f4bc813b430 2024-09-16T14:47:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.418+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(131..131 src has 73..131) v4 ==== 7321+0+0 (unknown 1197112955 0 0) 0x7f4bc0044320 con 0x7f4bc813b430 2024-09-16T14:47:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.418+0000 7f4bc659c640 10 monclient: get_auth_request con 0x7f4ba8036360 auth_method 0 2024-09-16T14:47:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.422+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4bc00149c0 con 0x7f4bc813b430 2024-09-16T14:47:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.422+0000 7f4bb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.422+0000 7f4bb6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:40.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 10 monclient: get_monmap_and_config 2024-09-16T14:47:40.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 10 monclient: build_initial_monmap 2024-09-16T14:47:40.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 10 monclient: monmap: 2024-09-16T14:47:40.466 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:40.466 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.466 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:40.466219+0000 2024-09-16T14:47:40.467 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:40.466219+0000 2024-09-16T14:47:40.467 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:40.467 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.467 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:40.467 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:40.467 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:40.467 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 1 Processor -- start 2024-09-16T14:47:40.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 1 -- start start 2024-09-16T14:47:40.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 10 monclient: init 2024-09-16T14:47:40.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:40.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:40.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 10 monclient(hunting): picked mon.noname-c con 0x7f988c09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:40.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 10 monclient(hunting): picked mon.noname-a con 0x7f988c132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:40.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 10 monclient(hunting): picked mon.noname-b con 0x7f988c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:40.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c09d260 con 0x7f988c132650 2024-09-16T14:47:40.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c132a80 con 0x7f988c09cb80 2024-09-16T14:47:40.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c09cf60 con 0x7f988c138ff0 2024-09-16T14:47:40.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:40.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 10 monclient(hunting): authenticate will time out at 1284.586254s 2024-09-16T14:47:40.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98919a1640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f988c132650 0x7f988c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47818/0 (socket says 172.21.15.26:47818) 2024-09-16T14:47:40.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98919a1640 1 -- 172.21.15.26:0/2000464868 learned_addr learned my addr 172.21.15.26:0/2000464868 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:40.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98929a3640 1 --1- 172.21.15.26:0/2000464868 >> v1:172.21.15.26:6790/0 conn(0x7f988c138ff0 0x7f988c13b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55182/0 (socket says 172.21.15.26:55182) 2024-09-16T14:47:40.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 1 -- 172.21.15.26:0/2000464868 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433454378 0 0) 0x7f988c09d260 con 0x7f988c132650 2024-09-16T14:47:40.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient(hunting): my global_id is 124128 2024-09-16T14:47:40.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 1 -- 172.21.15.26:0/2000464868 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9878003530 con 0x7f988c132650 2024-09-16T14:47:40.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 1 -- 172.21.15.26:0/2000464868 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913916418 0 0) 0x7f988c09cf60 con 0x7f988c138ff0 2024-09-16T14:47:40.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient(hunting): my global_id is 114437 2024-09-16T14:47:40.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 1 -- 172.21.15.26:0/2000464868 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f988c09d260 con 0x7f988c138ff0 2024-09-16T14:47:40.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 1 -- 172.21.15.26:0/2000464868 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3618830811 0 0) 0x7f988c09d260 con 0x7f988c138ff0 2024-09-16T14:47:40.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 1 -- 172.21.15.26:0/2000464868 >> v1:172.21.15.189:6789/0 conn(0x7f988c09cb80 legacy=0x7f988c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 1 -- 172.21.15.26:0/2000464868 >> v1:172.21.15.26:6789/0 conn(0x7f988c132650 legacy=0x7f988c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient: _finish_hunting 0 2024-09-16T14:47:40.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 1 monclient: found mon.noname-b 2024-09-16T14:47:40.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:40.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:47:40.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 1 -- 172.21.15.26:0/2000464868 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c09f2d0 con 0x7f988c138ff0 2024-09-16T14:47:40.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient: _finish_auth 0 2024-09-16T14:47:40.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient: _check_auth_tickets 2024-09-16T14:47:40.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:40.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 1 -- 172.21.15.26:0/2000464868 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f98840019c0 con 0x7f988c138ff0 2024-09-16T14:47:40.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:40.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient: dump: 2024-09-16T14:47:40.479 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.479 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.480 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.480 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.480 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.480 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.480 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.480 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.480 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.480 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 5 monclient: authenticate success, global_id 114437 2024-09-16T14:47:40.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:40.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 1 -- 172.21.15.26:0/2000464868 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9884001b60 con 0x7f988c138ff0 2024-09-16T14:47:40.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:40.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 1 -- 172.21.15.26:0/2000464868 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9884001f30 con 0x7f988c138ff0 2024-09-16T14:47:40.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:40.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f98911a0640 10 monclient: dump: 2024-09-16T14:47:40.481 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.481 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.481 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.482 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.482 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.482 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.482 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.482 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.483 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.483 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:40.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 4 set_mon_vals no callback set 2024-09-16T14:47:40.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 10 monclient: shutdown 2024-09-16T14:47:40.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:40.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 1 -- 172.21.15.26:0/2000464868 >> v1:172.21.15.26:6790/0 conn(0x7f988c138ff0 legacy=0x7f988c13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 1 -- 172.21.15.26:0/2000464868 shutdown_connections 2024-09-16T14:47:40.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 1 -- 172.21.15.26:0/2000464868 >> 172.21.15.26:0/2000464868 conn(0x7f988c09fb50 msgr2=0x7f988c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:40.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 1 -- 172.21.15.26:0/2000464868 shutdown_connections 2024-09-16T14:47:40.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.463+0000 7f9894406640 1 -- 172.21.15.26:0/2000464868 wait complete. 2024-09-16T14:47:40.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 monclient: build_initial_monmap 2024-09-16T14:47:40.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 monclient: monmap: 2024-09-16T14:47:40.484 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:40.484 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:40.484 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:40.484 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:40.484 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:40.484 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:40.484 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:40.485 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:40.485 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:40.485 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 1 Processor -- start 2024-09-16T14:47:40.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 1 -- start start 2024-09-16T14:47:40.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 monclient: init 2024-09-16T14:47:40.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:40.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:40.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 monclient(hunting): picked mon.noname-c con 0x7f988c09cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:40.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 monclient(hunting): picked mon.noname-b con 0x7f988c132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:40.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 monclient(hunting): picked mon.noname-a con 0x7f988c138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:40.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c136440 con 0x7f988c138ff0 2024-09-16T14:47:40.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c09a240 con 0x7f988c132650 2024-09-16T14:47:40.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c13c770 con 0x7f988c09cb80 2024-09-16T14:47:40.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:40.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 monclient(hunting): authenticate will time out at 1284.589260s 2024-09-16T14:47:40.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f98921a2640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f988c09cb80 0x7f988c133e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55192/0 (socket says 172.21.15.26:55192) 2024-09-16T14:47:40.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f98921a2640 1 -- 172.21.15.26:0/3417403864 learned_addr learned my addr 172.21.15.26:0/3417403864 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:40.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691866818 0 0) 0x7f988c136440 con 0x7f988c138ff0 2024-09-16T14:47:40.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient(hunting): my global_id is 124131 2024-09-16T14:47:40.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9870003530 con 0x7f988c138ff0 2024-09-16T14:47:40.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076824447 0 0) 0x7f988c13c770 con 0x7f988c09cb80 2024-09-16T14:47:40.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient(hunting): my global_id is 114440 2024-09-16T14:47:40.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f988c136440 con 0x7f988c09cb80 2024-09-16T14:47:40.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 237075980 0 0) 0x7f9870003530 con 0x7f988c138ff0 2024-09-16T14:47:40.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f988c13c770 con 0x7f988c138ff0 2024-09-16T14:47:40.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f98840018c0 con 0x7f988c138ff0 2024-09-16T14:47:40.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:40.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient(hunting): dump: 2024-09-16T14:47:40.489 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.489 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.489 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.489 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.489 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.489 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.489 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.489 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.489 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.490 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:40.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 783943536 0 0) 0x7f988c136440 con 0x7f988c09cb80 2024-09-16T14:47:40.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9870003530 con 0x7f988c09cb80 2024-09-16T14:47:40.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9888001950 con 0x7f988c09cb80 2024-09-16T14:47:40.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:40.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient(hunting): dump: 2024-09-16T14:47:40.490 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.490 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.490 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.491 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.491 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.491 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.491 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.491 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.491 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.491 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 360465780 0 0) 0x7f988c13c770 con 0x7f988c138ff0 2024-09-16T14:47:40.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 >> v1:172.21.15.26:6790/0 conn(0x7f988c09cb80 legacy=0x7f988c133e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 >> v1:172.21.15.189:6789/0 conn(0x7f988c132650 legacy=0x7f988c134550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:40.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 monclient: found mon.a 2024-09-16T14:47:40.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:40.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:40.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c137da0 con 0x7f988c138ff0 2024-09-16T14:47:40.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient: _finish_auth 0 2024-09-16T14:47:40.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:40.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:40.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 5 monclient: authenticate success, global_id 124131 2024-09-16T14:47:40.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 monclient: _renew_subs 2024-09-16T14:47:40.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:40.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 1 -- 172.21.15.26:0/3417403864 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988c135aa0 con 0x7f988c138ff0 2024-09-16T14:47:40.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 monclient: _renew_subs 2024-09-16T14:47:40.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:40.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 1 -- 172.21.15.26:0/3417403864 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f988c137ac0 con 0x7f988c138ff0 2024-09-16T14:47:40.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9884004e20 con 0x7f988c138ff0 2024-09-16T14:47:40.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:40.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9884004f90 con 0x7f988c138ff0 2024-09-16T14:47:40.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:40.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 10 monclient: dump: 2024-09-16T14:47:40.494 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:40.494 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.494 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.495 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.495 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.495 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:40.495 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.495 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.495 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.495 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:40.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f989099f640 4 set_mon_vals no callback set 2024-09-16T14:47:40.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f988400fa00 con 0x7f988c138ff0 2024-09-16T14:47:40.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:40.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:40.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:40.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9894406640 1 -- 172.21.15.26:0/3417403864 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9874005050 con 0x7f988c138ff0 2024-09-16T14:47:40.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(131..131 src has 73..131) v4 ==== 7321+0+0 (unknown 1197112955 0 0) 0x7f98840441b0 con 0x7f988c138ff0 2024-09-16T14:47:40.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.467+0000 7f98921a2640 10 monclient: get_auth_request con 0x7f98700362c0 auth_method 0 2024-09-16T14:47:40.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.475+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9884014770 con 0x7f988c138ff0 2024-09-16T14:47:40.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.475+0000 7f9882ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:40.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.475+0000 7f9882ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:40.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.590+0000 7f4bccc15640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:40.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.590+0000 7f4bccc15640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:40.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.590+0000 7f4bccc15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:40.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.590+0000 7f4bccc15640 1 -- 172.21.15.189:0/2591235488 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f4bac005880 con 0x7f4bc813b430 2024-09-16T14:47:40.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.590+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v131) v1 ==== 71+0+14 (unknown 738765755 0 3003798303) 0x7f4bc0012e30 con 0x7f4bc813b430 2024-09-16T14:47:40.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.590+0000 7f4bb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:40.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.590+0000 7f4bb6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:40.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.686+0000 7f4bccc15640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:40.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.686+0000 7f4bccc15640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:40.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.686+0000 7f4bccc15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:40.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.686+0000 7f4bccc15640 1 -- 172.21.15.189:0/2591235488 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f4bac005370 con 0x7f4bc813b430 2024-09-16T14:47:40.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.686+0000 7f4bb6ffd640 1 -- 172.21.15.189:0/2591235488 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v131) v1 ==== 71+0+14 (unknown 738765755 0 3003798303) 0x7f4bc00145b0 con 0x7f4bc813b430 2024-09-16T14:47:40.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.686+0000 7f4bb6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:40.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.686+0000 7f4bb6ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:40.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.691+0000 7f9894406640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:47:40.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.691+0000 7f9894406640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:47:40.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.691+0000 7f9894406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:40.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.691+0000 7f9894406640 1 -- 172.21.15.26:0/3417403864 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f9874005b10 con 0x7f988c138ff0 2024-09-16T14:47:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.690+0000 7f4bccc15640 1 -- 172.21.15.189:0/2591235488 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4ba8036360 legacy=0x7f4ba8038820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.690+0000 7f4bccc15640 10 monclient: shutdown 2024-09-16T14:47:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.690+0000 7f4bccc15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.690+0000 7f4bccc15640 1 -- 172.21.15.189:0/2591235488 >> v1:172.21.15.189:6789/0 conn(0x7f4bc813b430 legacy=0x7f4bc81d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.690+0000 7f4bccc15640 1 -- 172.21.15.189:0/2591235488 shutdown_connections 2024-09-16T14:47:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.690+0000 7f4bccc15640 1 -- 172.21.15.189:0/2591235488 >> 172.21.15.189:0/2591235488 conn(0x7f4bc80a7dd0 msgr2=0x7f4bc806b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.690+0000 7f4bccc15640 1 -- 172.21.15.189:0/2591235488 shutdown_connections 2024-09-16T14:47:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.690+0000 7f4bccc15640 1 -- 172.21.15.189:0/2591235488 wait complete. 2024-09-16T14:47:40.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.695+0000 7f9882ffd640 1 -- 172.21.15.26:0/3417403864 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f9884012e10 con 0x7f988c138ff0 2024-09-16T14:47:40.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.695+0000 7f9882ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:47:40.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.695+0000 7f9882ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:40.699 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.695+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.699 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.695+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=4135365470,config=692619927,health=2815006004,kv=3455456366,logm=1524645833}) 2024-09-16T14:47:40.699 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.695+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=10,kv=10,logm=59} crc {auth=567137367,config=692619927,health=2311385902,kv=3604373077,logm=1762657694}) 2024-09-16T14:47:40.699 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.695+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.699 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.695+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=4135365470,config=692619927,health=2815006004,kv=3455456366,logm=1524645833}) 2024-09-16T14:47:40.700 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.695+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=12,kv=11,logm=55} crc {auth=2371384458,config=692619927,health=474105080,kv=1867599010,logm=821819865}) 2024-09-16T14:47:40.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.699+0000 7f9894406640 1 -- 172.21.15.26:0/3417403864 >> v1:172.21.15.189:6812/3196161582 conn(0x7f98700362c0 legacy=0x7f9870038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.699+0000 7f9894406640 10 monclient: shutdown 2024-09-16T14:47:40.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.699+0000 7f9894406640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:40.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.699+0000 7f9894406640 1 -- 172.21.15.26:0/3417403864 >> v1:172.21.15.26:6789/0 conn(0x7f988c138ff0 legacy=0x7f988c134c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.699+0000 7f9894406640 1 -- 172.21.15.26:0/3417403864 shutdown_connections 2024-09-16T14:47:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.699+0000 7f9894406640 1 -- 172.21.15.26:0/3417403864 >> 172.21.15.26:0/3417403864 conn(0x7f988c09fb50 msgr2=0x7f988c12eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.699+0000 7f9894406640 1 -- 172.21.15.26:0/3417403864 shutdown_connections 2024-09-16T14:47:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:40.699+0000 7f9894406640 1 -- 172.21.15.26:0/3417403864 wait complete. 2024-09-16T14:47:40.705 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.699+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.705 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.699+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2951796263}) 2024-09-16T14:47:40.705 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.699+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3521753248}) 2024-09-16T14:47:40.705 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.699+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.705 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.699+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2951796263}) 2024-09-16T14:47:40.705 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.699+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2332154030}) 2024-09-16T14:47:40.705 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.699+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.705 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.699+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=80800236}) 2024-09-16T14:47:40.706 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.699+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372087164}) 2024-09-16T14:47:40.706 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.699+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.706 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.699+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=80800236}) 2024-09-16T14:47:40.706 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.699+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1998016731}) 2024-09-16T14:47:40.707 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.703+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.707 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.703+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2251624225}) 2024-09-16T14:47:40.707 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.703+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307108839}) 2024-09-16T14:47:40.708 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.703+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.708 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.703+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2251624225}) 2024-09-16T14:47:40.708 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.703+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3308861470}) 2024-09-16T14:47:40.711 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.703+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.711 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.703+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4018969099}) 2024-09-16T14:47:40.711 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.703+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2049552341}) 2024-09-16T14:47:40.711 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.703+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.711 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.703+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4018969099}) 2024-09-16T14:47:40.711 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.703+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3347910378}) 2024-09-16T14:47:40.712 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.707+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.712 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.707+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=869504640}) 2024-09-16T14:47:40.712 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.707+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668101746}) 2024-09-16T14:47:40.712 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.707+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.712 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.707+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=869504640}) 2024-09-16T14:47:40.712 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.707+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=538953275}) 2024-09-16T14:47:40.713 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set rm fooset 2024-09-16T14:47:40.715 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.707+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.715 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.707+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197745928}) 2024-09-16T14:47:40.715 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.707+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=47648358}) 2024-09-16T14:47:40.716 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.707+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.716 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.707+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197745928}) 2024-09-16T14:47:40.716 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.707+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1743632438}) 2024-09-16T14:47:40.716 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.711+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.716 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.711+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=75912653}) 2024-09-16T14:47:40.716 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.711+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=933047617}) 2024-09-16T14:47:40.716 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.711+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.716 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.711+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=75912653}) 2024-09-16T14:47:40.717 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.711+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2238526992}) 2024-09-16T14:47:40.719 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.711+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.719 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.711+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3313918423}) 2024-09-16T14:47:40.719 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.711+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2213448564}) 2024-09-16T14:47:40.719 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.711+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.719 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.711+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3313918423}) 2024-09-16T14:47:40.719 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.711+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=656892229}) 2024-09-16T14:47:40.720 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.715+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.720 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.715+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=75280938}) 2024-09-16T14:47:40.720 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.715+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562463167}) 2024-09-16T14:47:40.720 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.715+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.720 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.715+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=75280938}) 2024-09-16T14:47:40.720 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.715+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1875249572}) 2024-09-16T14:47:40.722 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:47:40.724 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.715+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.725 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.715+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3147812364}) 2024-09-16T14:47:40.725 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.715+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2003704626}) 2024-09-16T14:47:40.725 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.715+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.725 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.715+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3147812364}) 2024-09-16T14:47:40.725 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.715+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=465413980}) 2024-09-16T14:47:40.725 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.719+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.725 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.719+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2138469007}) 2024-09-16T14:47:40.726 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.719+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505036416}) 2024-09-16T14:47:40.726 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.719+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.726 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.719+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2138469007}) 2024-09-16T14:47:40.726 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.719+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2956546616}) 2024-09-16T14:47:40.726 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.719+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.726 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.719+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4107973388}) 2024-09-16T14:47:40.726 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.719+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283373196}) 2024-09-16T14:47:40.726 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.719+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.726 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.719+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4107973388}) 2024-09-16T14:47:40.726 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.719+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4160545665}) 2024-09-16T14:47:40.728 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.723+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.728 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.723+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3623903435}) 2024-09-16T14:47:40.728 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.723+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130120298}) 2024-09-16T14:47:40.728 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.723+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.728 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.723+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3623903435}) 2024-09-16T14:47:40.728 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.723+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3245869406}) 2024-09-16T14:47:40.728 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.723+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.728 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.723+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734108546}) 2024-09-16T14:47:40.728 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.723+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2011412923}) 2024-09-16T14:47:40.728 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.723+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.728 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.723+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734108546}) 2024-09-16T14:47:40.729 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.723+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1780886773}) 2024-09-16T14:47:40.729 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.727+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.730 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.727+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2606237565}) 2024-09-16T14:47:40.730 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.727+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=82112267}) 2024-09-16T14:47:40.730 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.727+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.730 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.727+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2606237565}) 2024-09-16T14:47:40.730 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.727+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4006988305}) 2024-09-16T14:47:40.731 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.727+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.731 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.727+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1347007093}) 2024-09-16T14:47:40.731 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.727+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384469326}) 2024-09-16T14:47:40.731 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.727+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.732 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.727+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1347007093}) 2024-09-16T14:47:40.732 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.727+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3228123764}) 2024-09-16T14:47:40.733 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.727+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.733 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.727+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3989338978}) 2024-09-16T14:47:40.733 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.731+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1052710537}) 2024-09-16T14:47:40.733 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.731+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.733 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.731+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3989338978}) 2024-09-16T14:47:40.733 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.731+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2632580627}) 2024-09-16T14:47:40.735 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.731+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.735 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.731+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2230041632}) 2024-09-16T14:47:40.735 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.731+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1637536387}) 2024-09-16T14:47:40.736 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.731+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.736 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.731+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2230041632}) 2024-09-16T14:47:40.736 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.731+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=703313566}) 2024-09-16T14:47:40.737 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.731+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.737 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.731+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=936532390}) 2024-09-16T14:47:40.737 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.731+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215677971}) 2024-09-16T14:47:40.737 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.731+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.738 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.731+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=936532390}) 2024-09-16T14:47:40.738 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.731+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3503493273}) 2024-09-16T14:47:40.740 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.735+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.740 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.735+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=33,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=22} crc {logm=1777469776,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgrstat=102106200,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=2062150,osd_snap=3794734670,osdmap=2098256317}) 2024-09-16T14:47:40.740 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.735+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=33,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=22} crc {logm=696998248,mdsmap=1531151339,mgr=1843570494,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=1211393356,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=2062150,osd_snap=3794734670,osdmap=741480562}) 2024-09-16T14:47:40.741 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.735+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.741 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.735+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=33,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=22} crc {logm=1777469776,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgrstat=102106200,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=2062150,osd_snap=3794734670,osdmap=2098256317}) 2024-09-16T14:47:40.741 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.735+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=32,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=25} crc {logm=3125312784,mdsmap=1531151339,mgr=620920336,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=102106200,mon_config_key=1151016613,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=2062150,osd_snap=3794734670,osdmap=1586740681}) 2024-09-16T14:47:40.748 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.743+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.748 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.743+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=74} crc {osdmap=790965421}) 2024-09-16T14:47:40.748 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.743+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=80} crc {osdmap=215292484}) 2024-09-16T14:47:40.748 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.743+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:47:40.748 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.743+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=74} crc {osdmap=790965421}) 2024-09-16T14:47:40.748 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:47:40.743+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=77} crc {osdmap=1127397488}) 2024-09-16T14:47:40.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 10 monclient: get_monmap_and_config 2024-09-16T14:47:40.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 10 monclient: build_initial_monmap 2024-09-16T14:47:40.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 10 monclient: monmap: 2024-09-16T14:47:40.826 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:40.826 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.826 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:40.826688+0000 2024-09-16T14:47:40.827 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:40.826688+0000 2024-09-16T14:47:40.827 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:40.827 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:40.827 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:40.827 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:40.827 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:40.827 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:40.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 1 Processor -- start 2024-09-16T14:47:40.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 1 -- start start 2024-09-16T14:47:40.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 10 monclient: init 2024-09-16T14:47:40.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:40.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:40.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 10 monclient(hunting): picked mon.noname-b con 0x7faee41315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:40.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 10 monclient(hunting): picked mon.noname-a con 0x7faee409b320 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:40.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 10 monclient(hunting): picked mon.noname-c con 0x7faee4132090 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:40.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee409edd0 con 0x7faee409b320 2024-09-16T14:47:40.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee409bc00 con 0x7faee4132090 2024-09-16T14:47:40.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee409e900 con 0x7faee41315a0 2024-09-16T14:47:40.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:40.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 10 monclient(hunting): authenticate will time out at 1242.942340s 2024-09-16T14:47:40.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee37fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7faee4132090 0x7faee413a200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33540/0 (socket says 172.21.15.189:33540) 2024-09-16T14:47:40.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee37fe640 1 -- 172.21.15.189:0/278625256 learned_addr learned my addr 172.21.15.189:0/278625256 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:40.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 1 -- 172.21.15.189:0/278625256 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3940674172 0 0) 0x7faee409bc00 con 0x7faee4132090 2024-09-16T14:47:40.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 10 monclient(hunting): my global_id is 95074 2024-09-16T14:47:40.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 1 -- 172.21.15.189:0/278625256 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faecc003530 con 0x7faee4132090 2024-09-16T14:47:40.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 1 -- 172.21.15.189:0/278625256 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4013984455 0 0) 0x7faecc003530 con 0x7faee4132090 2024-09-16T14:47:40.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 1 -- 172.21.15.189:0/278625256 >> v1:172.21.15.26:6790/0 conn(0x7faee41315a0 legacy=0x7faee4131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 1 -- 172.21.15.189:0/278625256 >> v1:172.21.15.26:6789/0 conn(0x7faee409b320 legacy=0x7faee4136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:47:40.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 1 monclient: found mon.noname-c 2024-09-16T14:47:40.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:40.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:40.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 1 -- 172.21.15.189:0/278625256 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee409b920 con 0x7faee4132090 2024-09-16T14:47:40.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:47:40.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:40.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:40.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 1 -- 172.21.15.189:0/278625256 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faedc0019a0 con 0x7faee4132090 2024-09-16T14:47:40.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:40.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 10 monclient: dump: 2024-09-16T14:47:40.836 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:40.836 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.836 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.836 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.836 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.836 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:40.836 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.836 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.837 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.837 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:40.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 5 monclient: authenticate success, global_id 95074 2024-09-16T14:47:40.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:40.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 1 -- 172.21.15.189:0/278625256 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faedc001cc0 con 0x7faee4132090 2024-09-16T14:47:40.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:40.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 1 -- 172.21.15.189:0/278625256 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faedc002050 con 0x7faee4132090 2024-09-16T14:47:40.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:40.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee1ffb640 10 monclient: dump: 2024-09-16T14:47:40.838 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:40.838 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.838 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.838 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.838 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.838 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:40.838 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.838 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.838 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.838 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:40.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:40.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 4 set_mon_vals no callback set 2024-09-16T14:47:40.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 10 monclient: shutdown 2024-09-16T14:47:40.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:40.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 1 -- 172.21.15.189:0/278625256 >> v1:172.21.15.189:6789/0 conn(0x7faee4132090 legacy=0x7faee413a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 1 -- 172.21.15.189:0/278625256 shutdown_connections 2024-09-16T14:47:40.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 1 -- 172.21.15.189:0/278625256 >> 172.21.15.189:0/278625256 conn(0x7faee40a3320 msgr2=0x7faee409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:40.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 1 -- 172.21.15.189:0/278625256 shutdown_connections 2024-09-16T14:47:40.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.822+0000 7faee93d0640 1 -- 172.21.15.189:0/278625256 wait complete. 2024-09-16T14:47:40.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 monclient: build_initial_monmap 2024-09-16T14:47:40.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 monclient: monmap: 2024-09-16T14:47:40.840 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:40.840 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:40.840 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:40.840 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:40.840 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:40.840 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:40.840 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:40.840 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:40.840 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:40.840 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:40.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 1 Processor -- start 2024-09-16T14:47:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 1 -- start start 2024-09-16T14:47:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 monclient: init 2024-09-16T14:47:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 monclient(hunting): picked mon.noname-c con 0x7faee41315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 monclient(hunting): picked mon.noname-a con 0x7faee4132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 monclient(hunting): picked mon.noname-b con 0x7faee411ec80 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee409cd20 con 0x7faee4132090 2024-09-16T14:47:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee40a1170 con 0x7faee411ec80 2024-09-16T14:47:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee40a0c60 con 0x7faee41315a0 2024-09-16T14:47:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 monclient(hunting): authenticate will time out at 1242.945446s 2024-09-16T14:47:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee37fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7faee411ec80 0x7faee41d1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33552/0 (socket says 172.21.15.189:33552) 2024-09-16T14:47:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee37fe640 1 -- 172.21.15.189:0/3909156773 learned_addr learned my addr 172.21.15.189:0/3909156773 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743687302 0 0) 0x7faee40a1170 con 0x7faee411ec80 2024-09-16T14:47:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient(hunting): my global_id is 95077 2024-09-16T14:47:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faed0003530 con 0x7faee411ec80 2024-09-16T14:47:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2102833607 0 0) 0x7faed0003530 con 0x7faee411ec80 2024-09-16T14:47:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faee40a1170 con 0x7faee411ec80 2024-09-16T14:47:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faedc0017c0 con 0x7faee411ec80 2024-09-16T14:47:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient(hunting): dump: 2024-09-16T14:47:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.843 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673838291 0 0) 0x7faee40a0c60 con 0x7faee41315a0 2024-09-16T14:47:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient(hunting): my global_id is 114446 2024-09-16T14:47:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faed0003530 con 0x7faee41315a0 2024-09-16T14:47:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1778802640 0 0) 0x7faee40a1170 con 0x7faee411ec80 2024-09-16T14:47:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 >> v1:172.21.15.26:6790/0 conn(0x7faee41315a0 legacy=0x7faee4130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 >> v1:172.21.15.26:6789/0 conn(0x7faee4132090 legacy=0x7faee41ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 1 monclient: found mon.b 2024-09-16T14:47:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee406b380 con 0x7faee411ec80 2024-09-16T14:47:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 5 monclient: authenticate success, global_id 95077 2024-09-16T14:47:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 monclient: _renew_subs 2024-09-16T14:47:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 1 -- 172.21.15.189:0/3909156773 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee406dfb0 con 0x7faee411ec80 2024-09-16T14:47:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 monclient: _renew_subs 2024-09-16T14:47:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 1 -- 172.21.15.189:0/3909156773 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faee4131380 con 0x7faee411ec80 2024-09-16T14:47:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faedc004f80 con 0x7faee411ec80 2024-09-16T14:47:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faedc005290 con 0x7faee411ec80 2024-09-16T14:47:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 10 monclient: dump: 2024-09-16T14:47:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:40.846 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee17fa640 4 set_mon_vals no callback set 2024-09-16T14:47:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7faedc00fd00 con 0x7faee411ec80 2024-09-16T14:47:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(131..131 src has 73..131) v4 ==== 7321+0+0 (unknown 1197112955 0 0) 0x7faedc0445d0 con 0x7faee411ec80 2024-09-16T14:47:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee2ffd640 10 monclient: get_auth_request con 0x7faed0036290 auth_method 0 2024-09-16T14:47:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.826+0000 7faee93d0640 1 -- 172.21.15.189:0/3909156773 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee41b8ed0 con 0x7faee411ec80 2024-09-16T14:47:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.834+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7faedc014d90 con 0x7faee411ec80 2024-09-16T14:47:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.834+0000 7faecb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:40.834+0000 7faecb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:41.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:41.002+0000 7faee93d0640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T14:47:41.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:41.002+0000 7faee93d0640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T14:47:41.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:41.002+0000 7faee93d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:41.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:41.002+0000 7faee93d0640 1 -- 172.21.15.189:0/3909156773 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set rm", "pool": "fooset"} v 0) v1 -- 0x7faee406a0d0 con 0x7faee411ec80 2024-09-16T14:47:41.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:41.290+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "fooset"}]=0 v132) v1 ==== 89+0+0 (unknown 1276439885 0 0) 0x7faedc013200 con 0x7faee411ec80 2024-09-16T14:47:41.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:41.290+0000 7faecb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T14:47:41.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:41.290+0000 7faecb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:41.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:41.386+0000 7faee93d0640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T14:47:41.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:41.386+0000 7faee93d0640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T14:47:41.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:41.386+0000 7faee93d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:41.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:41.386+0000 7faee93d0640 1 -- 172.21.15.189:0/3909156773 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set rm", "pool": "fooset"} v 0) v1 -- 0x7faee409b320 con 0x7faee411ec80 2024-09-16T14:47:41.723 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:47:41.724 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:47:41.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501cc0d640 10 monclient: get_monmap_and_config 2024-09-16T14:47:41.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501cc0d640 10 monclient: build_initial_monmap 2024-09-16T14:47:41.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501cc0d640 10 monclient: monmap: 2024-09-16T14:47:41.855 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:41.855 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:41.855 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:41.855387+0000 2024-09-16T14:47:41.855 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:41.855387+0000 2024-09-16T14:47:41.855 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:41.856 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:41.856 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:41.856 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:41.856 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:41.856 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:41.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501cc0d640 1 Processor -- start 2024-09-16T14:47:41.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501cc0d640 1 -- start start 2024-09-16T14:47:41.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501cc0d640 10 monclient: init 2024-09-16T14:47:41.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501cc0d640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:41.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501cc0d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:41.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501cc0d640 10 monclient(hunting): picked mon.noname-a con 0x7f501809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:41.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501cc0d640 10 monclient(hunting): picked mon.noname-b con 0x7f5018132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:41.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501cc0d640 10 monclient(hunting): picked mon.noname-c con 0x7f5018138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:41.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501cc0d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501809d260 con 0x7f501809cb80 2024-09-16T14:47:41.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501cc0d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5018132a80 con 0x7f5018138ff0 2024-09-16T14:47:41.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501cc0d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501809cf60 con 0x7f5018132650 2024-09-16T14:47:41.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501cc0d640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:41.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501cc0d640 10 monclient(hunting): authenticate will time out at 1285.975426s 2024-09-16T14:47:41.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f5015d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5018132650 0x7f50181378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55206/0 (socket says 172.21.15.26:55206) 2024-09-16T14:47:41.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f5015d9b640 1 -- 172.21.15.26:0/549367348 learned_addr learned my addr 172.21.15.26:0/549367348 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:41.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501559a640 1 -- 172.21.15.26:0/549367348 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404459119 0 0) 0x7f501809d260 con 0x7f501809cb80 2024-09-16T14:47:41.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501559a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:41.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:41.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:41.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.851+0000 7f501559a640 10 monclient(hunting): my global_id is 124140 2024-09-16T14:47:41.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 1 -- 172.21.15.26:0/549367348 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5004003530 con 0x7f501809cb80 2024-09-16T14:47:41.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 1 -- 172.21.15.26:0/549367348 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850957915 0 0) 0x7f501809cf60 con 0x7f5018132650 2024-09-16T14:47:41.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:41.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:41.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:41.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 10 monclient(hunting): my global_id is 114449 2024-09-16T14:47:41.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 1 -- 172.21.15.26:0/549367348 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f501809d260 con 0x7f5018132650 2024-09-16T14:47:41.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 1 -- 172.21.15.26:0/549367348 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 165205843 0 0) 0x7f501809d260 con 0x7f5018132650 2024-09-16T14:47:41.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 1 -- 172.21.15.26:0/549367348 >> v1:172.21.15.189:6789/0 conn(0x7f5018138ff0 legacy=0x7f501813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:41.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 1 -- 172.21.15.26:0/549367348 >> v1:172.21.15.26:6789/0 conn(0x7f501809cb80 legacy=0x7f50180717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:41.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 10 monclient: _finish_hunting 0 2024-09-16T14:47:41.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 1 monclient: found mon.noname-b 2024-09-16T14:47:41.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:41.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:47:41.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 1 -- 172.21.15.26:0/549367348 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501809f2d0 con 0x7f5018132650 2024-09-16T14:47:41.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 10 monclient: _finish_auth 0 2024-09-16T14:47:41.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 10 monclient: _check_auth_tickets 2024-09-16T14:47:41.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:41.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 1 -- 172.21.15.26:0/549367348 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5000001660 con 0x7f5018132650 2024-09-16T14:47:41.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 5 monclient: authenticate success, global_id 114449 2024-09-16T14:47:41.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:41.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:41.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:41.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 10 monclient: dump: 2024-09-16T14:47:41.862 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:41.862 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:41.862 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:41.862 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:41.862 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:41.862 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:41.863 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:41.863 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:41.863 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:41.863 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:41.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:41.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 1 -- 172.21.15.26:0/549367348 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50000017c0 con 0x7f5018132650 2024-09-16T14:47:41.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:41.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:41.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 1 -- 172.21.15.26:0/549367348 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5000001b50 con 0x7f5018132650 2024-09-16T14:47:41.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 4 set_mon_vals no callback set 2024-09-16T14:47:41.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:41.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:41.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501559a640 10 monclient: dump: 2024-09-16T14:47:41.864 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:41.864 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:41.864 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:41.864 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:41.864 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:41.864 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:41.864 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:41.864 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:41.864 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:41.864 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:41.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 10 monclient: shutdown 2024-09-16T14:47:41.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:41.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 1 -- 172.21.15.26:0/549367348 >> v1:172.21.15.26:6790/0 conn(0x7f5018132650 legacy=0x7f50181378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:41.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 1 -- 172.21.15.26:0/549367348 shutdown_connections 2024-09-16T14:47:41.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 1 -- 172.21.15.26:0/549367348 >> 172.21.15.26:0/549367348 conn(0x7f501809fb50 msgr2=0x7f501809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:41.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 1 -- 172.21.15.26:0/549367348 shutdown_connections 2024-09-16T14:47:41.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 1 -- 172.21.15.26:0/549367348 wait complete. 2024-09-16T14:47:41.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 10 monclient: build_initial_monmap 2024-09-16T14:47:41.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 10 monclient: monmap: 2024-09-16T14:47:41.865 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:41.865 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:41.865 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:41.865 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:41.865 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:41.865 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:41.865 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:41.865 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:41.866 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:41.866 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:41.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 1 Processor -- start 2024-09-16T14:47:41.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 1 -- start start 2024-09-16T14:47:41.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 10 monclient: init 2024-09-16T14:47:41.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:41.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:41.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 10 monclient(hunting): picked mon.noname-b con 0x7f501809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:41.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 10 monclient(hunting): picked mon.noname-a con 0x7f5018132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:41.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 10 monclient(hunting): picked mon.noname-c con 0x7f5018138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:41.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501813c400 con 0x7f5018132650 2024-09-16T14:47:41.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5018134ff0 con 0x7f501809cb80 2024-09-16T14:47:41.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50180a2a30 con 0x7f5018138ff0 2024-09-16T14:47:41.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:41.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 10 monclient(hunting): authenticate will time out at 1285.978632s 2024-09-16T14:47:41.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f5016d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5018138ff0 0x7f50181d1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55216/0 (socket says 172.21.15.26:55216) 2024-09-16T14:47:41.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f5015d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5018132650 0x7f501812e380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47850/0 (socket says 172.21.15.26:47850) 2024-09-16T14:47:41.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f5016d9d640 1 -- 172.21.15.26:0/2849498697 learned_addr learned my addr 172.21.15.26:0/2849498697 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:41.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415385087 0 0) 0x7f501813c400 con 0x7f5018132650 2024-09-16T14:47:41.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:41.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:41.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:41.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): my global_id is 124143 2024-09-16T14:47:41.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff8003530 con 0x7f5018132650 2024-09-16T14:47:41.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691156416 0 0) 0x7f50180a2a30 con 0x7f5018138ff0 2024-09-16T14:47:41.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:41.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:41.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:41.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): my global_id is 114452 2024-09-16T14:47:41.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f501813c400 con 0x7f5018138ff0 2024-09-16T14:47:41.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1252378705 0 0) 0x7f4ff8003530 con 0x7f5018132650 2024-09-16T14:47:41.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50180a2a30 con 0x7f5018132650 2024-09-16T14:47:41.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5000001560 con 0x7f5018132650 2024-09-16T14:47:41.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:41.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:41.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): dump: 2024-09-16T14:47:41.872 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:41.872 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:41.872 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:41.872 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:41.872 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:41.872 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:41.872 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:41.872 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:41.872 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:41.872 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:41.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:41.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2569852131 0 0) 0x7f501813c400 con 0x7f5018138ff0 2024-09-16T14:47:41.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ff8003530 con 0x7f5018138ff0 2024-09-16T14:47:41.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222982415 0 0) 0x7f5018134ff0 con 0x7f501809cb80 2024-09-16T14:47:41.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:41.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:41.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:41.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): my global_id is 95083 2024-09-16T14:47:41.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f501813c400 con 0x7f501809cb80 2024-09-16T14:47:41.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f50100018c0 con 0x7f5018138ff0 2024-09-16T14:47:41.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:41.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:41.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient(hunting): dump: 2024-09-16T14:47:41.874 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:41.874 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:41.874 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:41.874 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:41.874 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:41.874 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:41.874 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:41.874 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:41.874 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:41.874 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:41.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3555022223 0 0) 0x7f50180a2a30 con 0x7f5018132650 2024-09-16T14:47:41.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 >> v1:172.21.15.26:6790/0 conn(0x7f5018138ff0 legacy=0x7f50181d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:41.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 >> v1:172.21.15.189:6789/0 conn(0x7f501809cb80 legacy=0x7f501812dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:41.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:41.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 monclient: found mon.a 2024-09-16T14:47:41.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:41.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:41.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501809a240 con 0x7f5018132650 2024-09-16T14:47:41.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient: _finish_auth 0 2024-09-16T14:47:41.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:41.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:41.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 5 monclient: authenticate success, global_id 124143 2024-09-16T14:47:41.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 10 monclient: _renew_subs 2024-09-16T14:47:41.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:41.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 1 -- 172.21.15.26:0/2849498697 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5018134490 con 0x7f5018132650 2024-09-16T14:47:41.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 10 monclient: _renew_subs 2024-09-16T14:47:41.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:41.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f501cc0d640 1 -- 172.21.15.26:0/2849498697 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f501812f570 con 0x7f5018132650 2024-09-16T14:47:41.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5000004aa0 con 0x7f5018132650 2024-09-16T14:47:41.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:41.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5000004db0 con 0x7f5018132650 2024-09-16T14:47:41.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:41.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:41.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f5014d99640 4 set_mon_vals no callback set 2024-09-16T14:47:41.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.855+0000 7f500effd640 10 monclient: dump: 2024-09-16T14:47:41.877 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:41.877 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:41.877 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:41.877 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:41.877 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:41.877 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:41.877 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:41.877 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:41.877 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:41.877 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:41.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.859+0000 7f501cc0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:41.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.859+0000 7f501cc0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:41.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.859+0000 7f501cc0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:41.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.859+0000 7f501cc0d640 1 -- 172.21.15.26:0/2849498697 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ffc005050 con 0x7f5018132650 2024-09-16T14:47:41.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.859+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5000004f90 con 0x7f5018132650 2024-09-16T14:47:41.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.859+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(132..132 src has 73..132) v4 ==== 6989+0+0 (unknown 1472322158 0 0) 0x7f50000440c0 con 0x7f5018132650 2024-09-16T14:47:41.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.859+0000 7f501659c640 10 monclient: get_auth_request con 0x7f4ff8036300 auth_method 0 2024-09-16T14:47:41.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.863+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5000014910 con 0x7f5018132650 2024-09-16T14:47:41.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.863+0000 7f500effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:41.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:41.863+0000 7f500effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:42.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.079+0000 7f501cc0d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:47:42.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.079+0000 7f501cc0d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:42.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.079+0000 7f501cc0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.079+0000 7f501cc0d640 1 -- 172.21.15.26:0/2849498697 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4ffc005880 con 0x7f5018132650 2024-09-16T14:47:42.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.083+0000 7f500effd640 1 -- 172.21.15.26:0/2849498697 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2465467895) 0x7f5000012d80 con 0x7f5018132650 2024-09-16T14:47:42.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.083+0000 7f500effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:42.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.083+0000 7f500effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:42.086 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:47:42.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.083+0000 7f501cc0d640 1 -- 172.21.15.26:0/2849498697 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4ff8036300 legacy=0x7f4ff80387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.083+0000 7f501cc0d640 10 monclient: shutdown 2024-09-16T14:47:42.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.083+0000 7f501cc0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:42.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.083+0000 7f501cc0d640 1 -- 172.21.15.26:0/2849498697 >> v1:172.21.15.26:6789/0 conn(0x7f5018132650 legacy=0x7f501812e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.083+0000 7f501cc0d640 1 -- 172.21.15.26:0/2849498697 shutdown_connections 2024-09-16T14:47:42.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.083+0000 7f501cc0d640 1 -- 172.21.15.26:0/2849498697 >> 172.21.15.26:0/2849498697 conn(0x7f501809fb50 msgr2=0x7f50181388c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:42.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.083+0000 7f501cc0d640 1 -- 172.21.15.26:0/2849498697 shutdown_connections 2024-09-16T14:47:42.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.083+0000 7f501cc0d640 1 -- 172.21.15.26:0/2849498697 wait complete. 2024-09-16T14:47:42.104 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:47:42.104 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:47:42.104 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:47:42.105 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:47:42.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ffe132640 10 monclient: get_monmap_and_config 2024-09-16T14:47:42.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ffe132640 10 monclient: build_initial_monmap 2024-09-16T14:47:42.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ffe132640 10 monclient: monmap: 2024-09-16T14:47:42.223 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:42.223 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.223 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:42.223168+0000 2024-09-16T14:47:42.223 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:42.223168+0000 2024-09-16T14:47:42.223 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:42.223 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.223 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:42.223 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:42.224 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:42.224 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ffe132640 1 Processor -- start 2024-09-16T14:47:42.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ffe132640 1 -- start start 2024-09-16T14:47:42.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ffe132640 10 monclient: init 2024-09-16T14:47:42.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ffe132640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:42.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ffe132640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:42.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ffe132640 10 monclient(hunting): picked mon.noname-c con 0x7f8ff80a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:42.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ffe132640 10 monclient(hunting): picked mon.noname-b con 0x7f8ff80a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:42.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ffe132640 10 monclient(hunting): picked mon.noname-a con 0x7f8ff80a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:42.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ffe132640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff80a3250 con 0x7f8ff80a2800 2024-09-16T14:47:42.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ffe132640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff809f600 con 0x7f8ff80a1d10 2024-09-16T14:47:42.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ffe132640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff80a3830 con 0x7f8ff80a0000 2024-09-16T14:47:42.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ffe132640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:42.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ffe132640 10 monclient(hunting): authenticate will time out at 1286.343288s 2024-09-16T14:47:42.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ff7fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8ff80a2800 0x7f8ff813d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47852/0 (socket says 172.21.15.26:47852) 2024-09-16T14:47:42.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ff7fff640 1 -- 172.21.15.26:0/3520691345 learned_addr learned my addr 172.21.15.26:0/3520691345 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:42.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ff67fc640 1 -- 172.21.15.26:0/3520691345 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254372436 0 0) 0x7f8ff80a3250 con 0x7f8ff80a2800 2024-09-16T14:47:42.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ff67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ff67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ff67fc640 10 monclient(hunting): my global_id is 124146 2024-09-16T14:47:42.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ff67fc640 1 -- 172.21.15.26:0/3520691345 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe4003530 con 0x7f8ff80a2800 2024-09-16T14:47:42.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ff67fc640 1 -- 172.21.15.26:0/3520691345 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799985106 0 0) 0x7f8ff80a3830 con 0x7f8ff80a0000 2024-09-16T14:47:42.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ff67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ff67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ff67fc640 10 monclient(hunting): my global_id is 114455 2024-09-16T14:47:42.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.219+0000 7f8ff67fc640 1 -- 172.21.15.26:0/3520691345 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff80a3250 con 0x7f8ff80a0000 2024-09-16T14:47:42.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 1 -- 172.21.15.26:0/3520691345 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 775379666 0 0) 0x7f8fe4003530 con 0x7f8ff80a2800 2024-09-16T14:47:42.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 1 -- 172.21.15.26:0/3520691345 >> v1:172.21.15.26:6790/0 conn(0x7f8ff80a0000 legacy=0x7f8ff8139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 1 -- 172.21.15.26:0/3520691345 >> v1:172.21.15.189:6789/0 conn(0x7f8ff80a1d10 legacy=0x7f8ff80a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:42.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 1 monclient: found mon.noname-a 2024-09-16T14:47:42.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:42.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 1 -- 172.21.15.26:0/3520691345 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff809ed60 con 0x7f8ff80a2800 2024-09-16T14:47:42.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:42.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:42.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:42.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 1 -- 172.21.15.26:0/3520691345 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ff0001620 con 0x7f8ff80a2800 2024-09-16T14:47:42.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 5 monclient: authenticate success, global_id 124146 2024-09-16T14:47:42.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:42.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:42.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 10 monclient: dump: 2024-09-16T14:47:42.230 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.230 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.230 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.230 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.230 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.230 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.230 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.230 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.230 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.231 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:42.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 1 -- 172.21.15.26:0/3520691345 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ff0001840 con 0x7f8ff80a2800 2024-09-16T14:47:42.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:42.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:42.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 1 -- 172.21.15.26:0/3520691345 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ff0001c10 con 0x7f8ff80a2800 2024-09-16T14:47:42.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 4 set_mon_vals no callback set 2024-09-16T14:47:42.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:42.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff67fc640 10 monclient: dump: 2024-09-16T14:47:42.231 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.231 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.232 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.232 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.232 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.232 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.232 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.232 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.232 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.232 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 10 monclient: shutdown 2024-09-16T14:47:42.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:42.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 1 -- 172.21.15.26:0/3520691345 >> v1:172.21.15.26:6789/0 conn(0x7f8ff80a2800 legacy=0x7f8ff813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 1 -- 172.21.15.26:0/3520691345 shutdown_connections 2024-09-16T14:47:42.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 1 -- 172.21.15.26:0/3520691345 >> 172.21.15.26:0/3520691345 conn(0x7f8ff80a7f30 msgr2=0x7f8ff80a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:42.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 1 -- 172.21.15.26:0/3520691345 shutdown_connections 2024-09-16T14:47:42.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 1 -- 172.21.15.26:0/3520691345 wait complete. 2024-09-16T14:47:42.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 10 monclient: build_initial_monmap 2024-09-16T14:47:42.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 10 monclient: monmap: 2024-09-16T14:47:42.233 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:42.233 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:42.233 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:42.233 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:42.233 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:42.233 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:42.233 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:42.233 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:42.233 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:42.233 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 1 Processor -- start 2024-09-16T14:47:42.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 1 -- start start 2024-09-16T14:47:42.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 10 monclient: init 2024-09-16T14:47:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 10 monclient(hunting): picked mon.noname-b con 0x7f8ff80a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 10 monclient(hunting): picked mon.noname-a con 0x7f8ff80a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 10 monclient(hunting): picked mon.noname-c con 0x7f8ff80a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff8137060 con 0x7f8ff80a1d10 2024-09-16T14:47:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff8136aa0 con 0x7f8ff80a0000 2024-09-16T14:47:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff8133cb0 con 0x7f8ff80a2800 2024-09-16T14:47:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:42.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 10 monclient(hunting): authenticate will time out at 1286.346526s 2024-09-16T14:47:42.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff6ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8ff80a1d10 0x7f8ff81335a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47866/0 (socket says 172.21.15.26:47866) 2024-09-16T14:47:42.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff6ffd640 1 -- 172.21.15.26:0/1667158873 learned_addr learned my addr 172.21.15.26:0/1667158873 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:42.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364330277 0 0) 0x7f8ff8133cb0 con 0x7f8ff80a2800 2024-09-16T14:47:42.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): my global_id is 114458 2024-09-16T14:47:42.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd8003530 con 0x7f8ff80a2800 2024-09-16T14:47:42.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527773245 0 0) 0x7f8ff8137060 con 0x7f8ff80a1d10 2024-09-16T14:47:42.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): my global_id is 124149 2024-09-16T14:47:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff8133cb0 con 0x7f8ff80a1d10 2024-09-16T14:47:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 338802341 0 0) 0x7f8ff8133cb0 con 0x7f8ff80a1d10 2024-09-16T14:47:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ff8137060 con 0x7f8ff80a1d10 2024-09-16T14:47:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8fe00019e0 con 0x7f8ff80a1d10 2024-09-16T14:47:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): dump: 2024-09-16T14:47:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.242 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3971580279 0 0) 0x7f8fd8003530 con 0x7f8ff80a2800 2024-09-16T14:47:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ff8133cb0 con 0x7f8ff80a2800 2024-09-16T14:47:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ff00019b0 con 0x7f8ff80a2800 2024-09-16T14:47:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): dump: 2024-09-16T14:47:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.244 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.244 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041308930 0 0) 0x7f8ff8136aa0 con 0x7f8ff80a0000 2024-09-16T14:47:42.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient(hunting): my global_id is 95089 2024-09-16T14:47:42.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd8003530 con 0x7f8ff80a0000 2024-09-16T14:47:42.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1088940941 0 0) 0x7f8ff8137060 con 0x7f8ff80a1d10 2024-09-16T14:47:42.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 >> v1:172.21.15.26:6790/0 conn(0x7f8ff80a2800 legacy=0x7f8ff81d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 >> v1:172.21.15.189:6789/0 conn(0x7f8ff80a0000 legacy=0x7f8ff8132e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient: _finish_hunting 0 2024-09-16T14:47:42.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 monclient: found mon.a 2024-09-16T14:47:42.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:42.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff8135000 con 0x7f8ff80a1d10 2024-09-16T14:47:42.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient: _finish_auth 0 2024-09-16T14:47:42.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient: _check_auth_tickets 2024-09-16T14:47:42.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:42.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 5 monclient: authenticate success, global_id 124149 2024-09-16T14:47:42.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 10 monclient: _renew_subs 2024-09-16T14:47:42.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 1 -- 172.21.15.26:0/1667158873 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff8136d80 con 0x7f8ff80a1d10 2024-09-16T14:47:42.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 10 monclient: _renew_subs 2024-09-16T14:47:42.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ffe132640 1 -- 172.21.15.26:0/1667158873 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff81367c0 con 0x7f8ff80a1d10 2024-09-16T14:47:42.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fe0001d10 con 0x7f8ff80a1d10 2024-09-16T14:47:42.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:42.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8fe0003ad0 con 0x7f8ff80a1d10 2024-09-16T14:47:42.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:42.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8ff5ffb640 4 set_mon_vals no callback set 2024-09-16T14:47:42.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.223+0000 7f8feffff640 10 monclient: dump: 2024-09-16T14:47:42.246 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.246 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.246 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.246 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.246 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.246 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.247 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.247 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.247 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.247 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8ffe132640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:42.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8ffe132640 1 -- 172.21.15.26:0/1667158873 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8fdc006590 con 0x7f8fdc002830 2024-09-16T14:47:42.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8fe0003c30 con 0x7f8ff80a1d10 2024-09-16T14:47:42.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(132..132 src has 73..132) v4 ==== 6989+0+0 (unknown 1472322158 0 0) 0x7f8fe0043f70 con 0x7f8ff80a1d10 2024-09-16T14:47:42.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725790465 0 0) 0x7f8fdc006590 con 0x7f8fdc002830 2024-09-16T14:47:42.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 10 monclient(hunting): my global_id is 124152 2024-09-16T14:47:42.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff8137060 con 0x7f8fdc002830 2024-09-16T14:47:42.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1400575207 0 0) 0x7f8ff8137060 con 0x7f8fdc002830 2024-09-16T14:47:42.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fdc006590 con 0x7f8fdc002830 2024-09-16T14:47:42.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8fe8001b20 con 0x7f8fdc002830 2024-09-16T14:47:42.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:42.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 10 monclient: dump: 2024-09-16T14:47:42.248 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.248 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.248 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.248 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.248 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.248 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.248 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.249 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.249 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.249 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8ff7fff640 10 monclient: get_auth_request con 0x7f8fd80363b0 auth_method 0 2024-09-16T14:47:42.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3509233130 0 0) 0x7f8fdc006590 con 0x7f8fdc002830 2024-09-16T14:47:42.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8fdc00a0e0 con 0x7f8fdc002830 2024-09-16T14:47:42.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8fe8001e50 con 0x7f8fdc002830 2024-09-16T14:47:42.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:42.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:42.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 >> v1:172.21.15.26:6789/0 conn(0x7f8fdc002830 legacy=0x7f8fdc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.227+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 >> v1:172.21.15.26:6789/0 conn(0x7f8fdc002830 legacy=0x7f8fdc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:42.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.231+0000 7f8ffe132640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:47:42.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.231+0000 7f8ffe132640 1 -- 172.21.15.26:0/1667158873 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8fdc009da0 con 0x7f8fdc0090c0 2024-09-16T14:47:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.231+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418095346 0 0) 0x7f8fdc009da0 con 0x7f8fdc0090c0 2024-09-16T14:47:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.231+0000 7f8feffff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.231+0000 7f8feffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.231+0000 7f8feffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.231+0000 7f8feffff640 10 monclient(hunting): my global_id is 124155 2024-09-16T14:47:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.231+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fdc006590 con 0x7f8fdc0090c0 2024-09-16T14:47:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1229136567 0 0) 0x7f8fdc006590 con 0x7f8fdc0090c0 2024-09-16T14:47:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fdc009da0 con 0x7f8fdc0090c0 2024-09-16T14:47:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8fe8003b40 con 0x7f8fdc0090c0 2024-09-16T14:47:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8feffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8feffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8feffff640 10 monclient: dump: 2024-09-16T14:47:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.250 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.251 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3079989434 0 0) 0x7f8fdc009da0 con 0x7f8fdc0090c0 2024-09-16T14:47:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8fdc00c290 con 0x7f8fdc0090c0 2024-09-16T14:47:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 763269639) 0x7f8fe8004f30 con 0x7f8fdc0090c0 2024-09-16T14:47:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8feffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:47:42.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8feffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:42.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 >> v1:172.21.15.26:6789/0 conn(0x7f8fdc0090c0 legacy=0x7f8fdc009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8feffff640 1 -- 172.21.15.26:0/1667158873 >> v1:172.21.15.26:6789/0 conn(0x7f8fdc0090c0 legacy=0x7f8fdc009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:42.252 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:47:42.252 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:42.252 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:42.252 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:47:42.252 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 48, 2024-09-16T14:47:42.252 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:47:42.252 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:47:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:47:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:47:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:47:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:47:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:47:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:47:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:47:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:47:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:47:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:47:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:47:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:47:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:47:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:47:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:47:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:47:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:47:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:47:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:47:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:47:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:47:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:47:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:47:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:47:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:47:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:47:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:47:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:47:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:47:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:47:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:47:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:47:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:47:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:47:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:47:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:47:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.263 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.263 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:47:42.263 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:47:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8ffe132640 1 -- 172.21.15.26:0/1667158873 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8fd80363b0 legacy=0x7f8fd8038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8ffe132640 10 monclient: shutdown 2024-09-16T14:47:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8ffe132640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8ffe132640 1 -- 172.21.15.26:0/1667158873 >> v1:172.21.15.26:6789/0 conn(0x7f8ff80a1d10 legacy=0x7f8ff81335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8ff7fff640 1 -- 172.21.15.26:0/1667158873 reap_dead start 2024-09-16T14:47:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8ff7fff640 1 -- 172.21.15.26:0/1667158873 reap_dead start 2024-09-16T14:47:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8ffe132640 1 -- 172.21.15.26:0/1667158873 shutdown_connections 2024-09-16T14:47:42.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8ffe132640 1 -- 172.21.15.26:0/1667158873 >> 172.21.15.26:0/1667158873 conn(0x7f8ff80a7f30 msgr2=0x7f8ff80a6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8ffe132640 1 -- 172.21.15.26:0/1667158873 shutdown_connections 2024-09-16T14:47:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.235+0000 7f8ffe132640 1 -- 172.21.15.26:0/1667158873 wait complete. 2024-09-16T14:47:42.265 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:47:42.265 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:47:42.265 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:47:42.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.282+0000 7faecb7fe640 1 -- 172.21.15.189:0/3909156773 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "fooset"}]=0 v133) v1 ==== 89+0+0 (unknown 241619364 0 0) 0x7faedc014980 con 0x7faee411ec80 2024-09-16T14:47:42.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.282+0000 7faecb7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T14:47:42.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.282+0000 7faecb7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:42.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.286+0000 7faee93d0640 1 -- 172.21.15.189:0/3909156773 >> v1:172.21.15.189:6812/3196161582 conn(0x7faed0036290 legacy=0x7faed0038750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.286+0000 7faee93d0640 10 monclient: shutdown 2024-09-16T14:47:42.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.286+0000 7faee93d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:42.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.286+0000 7faee93d0640 1 -- 172.21.15.189:0/3909156773 >> v1:172.21.15.189:6789/0 conn(0x7faee411ec80 legacy=0x7faee41d1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.286+0000 7faee93d0640 1 -- 172.21.15.189:0/3909156773 shutdown_connections 2024-09-16T14:47:42.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.286+0000 7faee93d0640 1 -- 172.21.15.189:0/3909156773 >> 172.21.15.189:0/3909156773 conn(0x7faee40a3320 msgr2=0x7faee4070930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:42.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.286+0000 7faee93d0640 1 -- 172.21.15.189:0/3909156773 shutdown_connections 2024-09-16T14:47:42.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.286+0000 7faee93d0640 1 -- 172.21.15.189:0/3909156773 wait complete. 2024-09-16T14:47:42.309 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set ls 2024-09-16T14:47:42.309 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 0 2024-09-16T14:47:42.309 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c fooset 2024-09-16T14:47:42.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.379+0000 7fe0a48af640 10 monclient: get_monmap_and_config 2024-09-16T14:47:42.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.379+0000 7fe0a48af640 10 monclient: build_initial_monmap 2024-09-16T14:47:42.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.379+0000 7fe0a48af640 10 monclient: monmap: 2024-09-16T14:47:42.385 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:42.385 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.385 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:42.384869+0000 2024-09-16T14:47:42.385 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:42.384869+0000 2024-09-16T14:47:42.386 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:42.386 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.386 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:42.386 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:42.386 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:42.386 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 1 Processor -- start 2024-09-16T14:47:42.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 1 -- start start 2024-09-16T14:47:42.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 10 monclient: init 2024-09-16T14:47:42.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:42.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:42.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 10 monclient(hunting): picked mon.noname-a con 0x7fe0a009a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:42.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 10 monclient(hunting): picked mon.noname-c con 0x7fe0a009d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:42.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 10 monclient(hunting): picked mon.noname-b con 0x7fe0a009b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:42.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a009dc60 con 0x7fe0a009a550 2024-09-16T14:47:42.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a0099ae0 con 0x7fe0a009d1b0 2024-09-16T14:47:42.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a009b8f0 con 0x7fe0a009b040 2024-09-16T14:47:42.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:42.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 10 monclient(hunting): authenticate will time out at 1286.504928s 2024-09-16T14:47:42.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09e59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe0a009a550 0x7fe0a009a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47910/0 (socket says 172.21.15.26:47910) 2024-09-16T14:47:42.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09ed9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe0a009b040 0x7fe0a013a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55246/0 (socket says 172.21.15.26:55246) 2024-09-16T14:47:42.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09e59c640 1 -- 172.21.15.26:0/806006640 learned_addr learned my addr 172.21.15.26:0/806006640 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:42.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 1 -- 172.21.15.26:0/806006640 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885451793 0 0) 0x7fe0a009b8f0 con 0x7fe0a009b040 2024-09-16T14:47:42.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient(hunting): my global_id is 114461 2024-09-16T14:47:42.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 1 -- 172.21.15.26:0/806006640 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe08c003530 con 0x7fe0a009b040 2024-09-16T14:47:42.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 1 -- 172.21.15.26:0/806006640 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397549634 0 0) 0x7fe0a009dc60 con 0x7fe0a009a550 2024-09-16T14:47:42.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient(hunting): my global_id is 124158 2024-09-16T14:47:42.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 1 -- 172.21.15.26:0/806006640 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a009b8f0 con 0x7fe0a009a550 2024-09-16T14:47:42.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 1 -- 172.21.15.26:0/806006640 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2073769528 0 0) 0x7fe08c003530 con 0x7fe0a009b040 2024-09-16T14:47:42.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 1 -- 172.21.15.26:0/806006640 >> v1:172.21.15.189:6789/0 conn(0x7fe0a009d1b0 legacy=0x7fe0a0136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 1 -- 172.21.15.26:0/806006640 >> v1:172.21.15.26:6789/0 conn(0x7fe0a009a550 legacy=0x7fe0a009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient: _finish_hunting 0 2024-09-16T14:47:42.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 1 monclient: found mon.noname-b 2024-09-16T14:47:42.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:42.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:47:42.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 1 -- 172.21.15.26:0/806006640 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a009bbd0 con 0x7fe0a009b040 2024-09-16T14:47:42.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient: _finish_auth 0 2024-09-16T14:47:42.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient: _check_auth_tickets 2024-09-16T14:47:42.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:42.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 1 -- 172.21.15.26:0/806006640 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe090001550 con 0x7fe0a009b040 2024-09-16T14:47:42.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 5 monclient: authenticate success, global_id 114461 2024-09-16T14:47:42.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:42.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:42.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient: dump: 2024-09-16T14:47:42.396 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.396 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.396 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.396 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.396 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.396 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.396 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.396 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.396 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.396 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:42.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 1 -- 172.21.15.26:0/806006640 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0900017f0 con 0x7fe0a009b040 2024-09-16T14:47:42.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:42.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 1 -- 172.21.15.26:0/806006640 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe090001bc0 con 0x7fe0a009b040 2024-09-16T14:47:42.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:42.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 4 set_mon_vals no callback set 2024-09-16T14:47:42.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:42.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe09d59a640 10 monclient: dump: 2024-09-16T14:47:42.397 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.397 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.397 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.397 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.398 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.398 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.398 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.398 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.398 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.398 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 10 monclient: shutdown 2024-09-16T14:47:42.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:42.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 1 -- 172.21.15.26:0/806006640 >> v1:172.21.15.26:6790/0 conn(0x7fe0a009b040 legacy=0x7fe0a013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 1 -- 172.21.15.26:0/806006640 shutdown_connections 2024-09-16T14:47:42.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 1 -- 172.21.15.26:0/806006640 >> 172.21.15.26:0/806006640 conn(0x7fe0a009fb50 msgr2=0x7fe0a009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:42.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 1 -- 172.21.15.26:0/806006640 shutdown_connections 2024-09-16T14:47:42.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 1 -- 172.21.15.26:0/806006640 wait complete. 2024-09-16T14:47:42.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 10 monclient: build_initial_monmap 2024-09-16T14:47:42.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 10 monclient: monmap: 2024-09-16T14:47:42.399 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:42.399 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:42.399 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:42.399 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:42.399 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:42.399 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:42.399 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:42.399 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:42.399 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:42.399 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 1 Processor -- start 2024-09-16T14:47:42.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 1 -- start start 2024-09-16T14:47:42.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 10 monclient: init 2024-09-16T14:47:42.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:42.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:42.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.383+0000 7fe0a48af640 10 monclient(hunting): picked mon.noname-c con 0x7fe0a009a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:42.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe0a48af640 10 monclient(hunting): picked mon.noname-a con 0x7fe0a009b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:42.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe0a48af640 10 monclient(hunting): picked mon.noname-b con 0x7fe0a009d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:42.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe0a48af640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a0139560 con 0x7fe0a009b040 2024-09-16T14:47:42.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe0a48af640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a0131560 con 0x7fe0a009d1b0 2024-09-16T14:47:42.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe0a48af640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a0137aa0 con 0x7fe0a009a550 2024-09-16T14:47:42.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe0a48af640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:42.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe0a48af640 10 monclient(hunting): authenticate will time out at 1286.508124s 2024-09-16T14:47:42.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe09e59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe0a009a550 0x7fe0a0136c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55254/0 (socket says 172.21.15.26:55254) 2024-09-16T14:47:42.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe09e59c640 1 -- 172.21.15.26:0/1329934273 learned_addr learned my addr 172.21.15.26:0/1329934273 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:42.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525083988 0 0) 0x7fe0a0137aa0 con 0x7fe0a009a550 2024-09-16T14:47:42.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): my global_id is 114464 2024-09-16T14:47:42.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe080003530 con 0x7fe0a009a550 2024-09-16T14:47:42.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897901689 0 0) 0x7fe0a0139560 con 0x7fe0a009b040 2024-09-16T14:47:42.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): my global_id is 124161 2024-09-16T14:47:42.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a0137aa0 con 0x7fe0a009b040 2024-09-16T14:47:42.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2436347452 0 0) 0x7fe080003530 con 0x7fe0a009a550 2024-09-16T14:47:42.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0a0139560 con 0x7fe0a009a550 2024-09-16T14:47:42.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe088001a00 con 0x7fe0a009a550 2024-09-16T14:47:42.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:42.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): dump: 2024-09-16T14:47:42.402 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.402 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.402 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.403 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.403 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.403 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.403 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.403 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.403 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.403 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:42.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1870596373 0 0) 0x7fe0a0137aa0 con 0x7fe0a009b040 2024-09-16T14:47:42.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe080003530 con 0x7fe0a009b040 2024-09-16T14:47:42.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe098001b80 con 0x7fe0a009b040 2024-09-16T14:47:42.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:42.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): dump: 2024-09-16T14:47:42.404 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.404 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.404 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.404 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.404 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.404 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.404 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.404 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.404 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.404 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640178866 0 0) 0x7fe0a0131560 con 0x7fe0a009d1b0 2024-09-16T14:47:42.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): my global_id is 95095 2024-09-16T14:47:42.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a0137aa0 con 0x7fe0a009d1b0 2024-09-16T14:47:42.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2763944839 0 0) 0x7fe0a0139560 con 0x7fe0a009a550 2024-09-16T14:47:42.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 >> v1:172.21.15.189:6789/0 conn(0x7fe0a009d1b0 legacy=0x7fe0a01d1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 >> v1:172.21.15.26:6789/0 conn(0x7fe0a009b040 legacy=0x7fe0a0137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:42.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 monclient: found mon.c 2024-09-16T14:47:42.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:42.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:42.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a0138b90 con 0x7fe0a009a550 2024-09-16T14:47:42.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient: _finish_auth 0 2024-09-16T14:47:42.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:42.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:42.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe0a48af640 5 monclient: authenticate success, global_id 114464 2024-09-16T14:47:42.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe0a48af640 10 monclient: _renew_subs 2024-09-16T14:47:42.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe0a48af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:42.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe0a48af640 1 -- 172.21.15.26:0/1329934273 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a009ede0 con 0x7fe0a009a550 2024-09-16T14:47:42.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe0a48af640 10 monclient: _renew_subs 2024-09-16T14:47:42.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0880047f0 con 0x7fe0a009a550 2024-09-16T14:47:42.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe0a48af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:42.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe0a48af640 1 -- 172.21.15.26:0/1329934273 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a0139840 con 0x7fe0a009a550 2024-09-16T14:47:42.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:42.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe088004b00 con 0x7fe0a009a550 2024-09-16T14:47:42.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:42.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient: dump: 2024-09-16T14:47:42.407 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.407 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.407 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.407 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.407 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.407 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.407 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.407 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.407 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.407 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe09cd99640 4 set_mon_vals no callback set 2024-09-16T14:47:42.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe0a48af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:42.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe0a48af640 1 -- 172.21.15.26:0/1329934273 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe0a00008e0 con 0x7fe0a0138390 2024-09-16T14:47:42.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe08800f570 con 0x7fe0a009a550 2024-09-16T14:47:42.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(133..133 src has 73..133) v4 ==== 6989+0+0 (unknown 2876426089 0 0) 0x7fe088043df0 con 0x7fe0a009a550 2024-09-16T14:47:42.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93117477 0 0) 0x7fe0a00008e0 con 0x7fe0a0138390 2024-09-16T14:47:42.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient(hunting): my global_id is 114467 2024-09-16T14:47:42.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a0139560 con 0x7fe0a0138390 2024-09-16T14:47:42.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe09ed9d640 10 monclient: get_auth_request con 0x7fe0800363d0 auth_method 0 2024-09-16T14:47:42.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 111620889 0 0) 0x7fe0a0139560 con 0x7fe0a0138390 2024-09-16T14:47:42.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0a00008e0 con 0x7fe0a0138390 2024-09-16T14:47:42.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe098003d90 con 0x7fe0a0138390 2024-09-16T14:47:42.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:42.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient: dump: 2024-09-16T14:47:42.409 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.409 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.409 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.409 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.409 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.409 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.409 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.409 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.409 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.409 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4289679263 0 0) 0x7fe0a00008e0 con 0x7fe0a0138390 2024-09-16T14:47:42.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe0a009df10 con 0x7fe0a0138390 2024-09-16T14:47:42.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe0980049b0 con 0x7fe0a0138390 2024-09-16T14:47:42.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:42.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:42.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 >> v1:172.21.15.26:6790/0 conn(0x7fe0a0138390 legacy=0x7fe0a006def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.387+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 >> v1:172.21.15.26:6790/0 conn(0x7fe0a0138390 legacy=0x7fe0a006def0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:42.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe0a48af640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:47:42.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe0a48af640 1 -- 172.21.15.26:0/1329934273 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe0a009f650 con 0x7fe0a002c280 2024-09-16T14:47:42.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326481540 0 0) 0x7fe0a009f650 con 0x7fe0a002c280 2024-09-16T14:47:42.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 10 monclient(hunting): my global_id is 114470 2024-09-16T14:47:42.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a00008e0 con 0x7fe0a002c280 2024-09-16T14:47:42.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1073548141 0 0) 0x7fe0a00008e0 con 0x7fe0a002c280 2024-09-16T14:47:42.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0a009f650 con 0x7fe0a002c280 2024-09-16T14:47:42.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe0980043d0 con 0x7fe0a002c280 2024-09-16T14:47:42.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:42.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 10 monclient: dump: 2024-09-16T14:47:42.413 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.413 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.413 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.413 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.413 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.413 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.413 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.413 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.413 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.413 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1116947325 0 0) 0x7fe0a009f650 con 0x7fe0a002c280 2024-09-16T14:47:42.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe0a009f7e0 con 0x7fe0a002c280 2024-09-16T14:47:42.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 1269018669) 0x7fe098004740 con 0x7fe0a002c280 2024-09-16T14:47:42.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:47:42.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:42.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 >> v1:172.21.15.26:6790/0 conn(0x7fe0a002c280 legacy=0x7fe0a006e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe096ffd640 1 -- 172.21.15.26:0/1329934273 >> v1:172.21.15.26:6790/0 conn(0x7fe0a002c280 legacy=0x7fe0a006e600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:42.414 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:47:42.414 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:42.414 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:42.414 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:47:42.414 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 48, 2024-09-16T14:47:42.414 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:47:42.415 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:47:42.415 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:47:42.415 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:47:42.415 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:42.415 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:47:42.415 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:42.415 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:47:42.415 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:47:42.415 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:42.415 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:42.415 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:42.415 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:42.415 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:42.415 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:42.416 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:42.416 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:42.416 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:42.416 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:42.416 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:42.416 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:47:42.416 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:47:42.416 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:42.416 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:42.416 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:42.416 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:42.416 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:42.416 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:42.417 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:42.417 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:42.417 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:42.417 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:42.417 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.417 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:47:42.417 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:47:42.417 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:47:42.417 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:47:42.417 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:47:42.417 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:47:42.417 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:42.417 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:42.418 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:47:42.418 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:47:42.418 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:47:42.418 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:47:42.418 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:47:42.418 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:47:42.418 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:47:42.418 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:42.418 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:47:42.418 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:42.418 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:42.418 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:42.418 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:42.419 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:42.419 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:42.419 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:42.419 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:42.419 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:42.419 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:42.419 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:42.419 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:47:42.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.419 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:47:42.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.419 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:42.419 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:42.419 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:42.420 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:42.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.420 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:42.420 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:47:42.420 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:42.420 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.420 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.420 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:47:42.420 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:47:42.420 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:42.420 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:42.420 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:42.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.421 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:42.421 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:42.421 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:42.421 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:42.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.421 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:42.421 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:47:42.421 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:42.421 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.421 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:47:42.422 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:47:42.422 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:42.422 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:42.422 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:42.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.422 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:42.422 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:42.422 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:42.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:42.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.422 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:42.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:47:42.423 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:42.423 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.423 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.423 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:47:42.423 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:47:42.423 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:42.423 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:42.423 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:42.423 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.423 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.423 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:47:42.423 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:47:42.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.424 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:42.424 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:42.424 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:42.424 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.424 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:42.424 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:47:42.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.424 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:42.424 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:42.424 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:47:42.424 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.424 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:42.425 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:47:42.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.425 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:42.425 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:42.425 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:47:42.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.425 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:42.425 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:47:42.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.425 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:42.425 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:42.425 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:42.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.425 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.426 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:47:42.426 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:47:42.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe0a48af640 1 -- 172.21.15.26:0/1329934273 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe0800363d0 legacy=0x7fe080038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe0a48af640 10 monclient: shutdown 2024-09-16T14:47:42.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe0a48af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:42.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe0a48af640 1 -- 172.21.15.26:0/1329934273 >> v1:172.21.15.26:6790/0 conn(0x7fe0a009a550 legacy=0x7fe0a0136c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe09ed9d640 1 -- 172.21.15.26:0/1329934273 reap_dead start 2024-09-16T14:47:42.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe09ed9d640 1 -- 172.21.15.26:0/1329934273 reap_dead start 2024-09-16T14:47:42.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe0a48af640 1 -- 172.21.15.26:0/1329934273 shutdown_connections 2024-09-16T14:47:42.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe0a48af640 1 -- 172.21.15.26:0/1329934273 >> 172.21.15.26:0/1329934273 conn(0x7fe0a009fb50 msgr2=0x7fe0a006d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:42.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe0a48af640 1 -- 172.21.15.26:0/1329934273 shutdown_connections 2024-09-16T14:47:42.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.395+0000 7fe0a48af640 1 -- 172.21.15.26:0/1329934273 wait complete. 2024-09-16T14:47:42.428 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:47:42.428 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:47:42.428 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:47:42.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 10 monclient: get_monmap_and_config 2024-09-16T14:47:42.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 10 monclient: build_initial_monmap 2024-09-16T14:47:42.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 10 monclient: monmap: 2024-09-16T14:47:42.435 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:42.435 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.435 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:42.426682+0000 2024-09-16T14:47:42.435 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:42.426682+0000 2024-09-16T14:47:42.435 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:42.436 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:42.436 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:42.436 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:42.436 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:42.436 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:42.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 1 Processor -- start 2024-09-16T14:47:42.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 1 -- start start 2024-09-16T14:47:42.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 10 monclient: init 2024-09-16T14:47:42.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:42.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:42.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 10 monclient(hunting): picked mon.noname-b con 0x7f1e881315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:42.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 10 monclient(hunting): picked mon.noname-c con 0x7f1e8809b320 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:42.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 10 monclient(hunting): picked mon.noname-a con 0x7f1e88132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:42.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8809edd0 con 0x7f1e88132090 2024-09-16T14:47:42.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8809bc00 con 0x7f1e8809b320 2024-09-16T14:47:42.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8809e900 con 0x7f1e881315a0 2024-09-16T14:47:42.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:42.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 10 monclient(hunting): authenticate will time out at 1244.542688s 2024-09-16T14:47:42.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e867fc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1e8809b320 0x7f1e88136ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33560/0 (socket says 172.21.15.189:33560) 2024-09-16T14:47:42.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e867fc640 1 -- 172.21.15.189:0/3095095527 learned_addr learned my addr 172.21.15.189:0/3095095527 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:42.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 1 -- 172.21.15.189:0/3095095527 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261809013 0 0) 0x7f1e8809bc00 con 0x7f1e8809b320 2024-09-16T14:47:42.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient(hunting): my global_id is 95098 2024-09-16T14:47:42.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 1 -- 172.21.15.189:0/3095095527 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e70003530 con 0x7f1e8809b320 2024-09-16T14:47:42.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 1 -- 172.21.15.189:0/3095095527 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578572015 0 0) 0x7f1e8809edd0 con 0x7f1e88132090 2024-09-16T14:47:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient(hunting): my global_id is 124164 2024-09-16T14:47:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 1 -- 172.21.15.189:0/3095095527 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e8809bc00 con 0x7f1e88132090 2024-09-16T14:47:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 1 -- 172.21.15.189:0/3095095527 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 546125340 0 0) 0x7f1e70003530 con 0x7f1e8809b320 2024-09-16T14:47:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 1 -- 172.21.15.189:0/3095095527 >> v1:172.21.15.26:6790/0 conn(0x7f1e881315a0 legacy=0x7f1e88131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 1 -- 172.21.15.189:0/3095095527 >> v1:172.21.15.26:6789/0 conn(0x7f1e88132090 legacy=0x7f1e8813a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:47:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 1 monclient: found mon.noname-c 2024-09-16T14:47:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:47:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 1 -- 172.21.15.189:0/3095095527 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8809b920 con 0x7f1e8809b320 2024-09-16T14:47:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient: _finish_auth 0 2024-09-16T14:47:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 1 -- 172.21.15.189:0/3095095527 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1e7c0015a0 con 0x7f1e8809b320 2024-09-16T14:47:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 5 monclient: authenticate success, global_id 95098 2024-09-16T14:47:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient: dump: 2024-09-16T14:47:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.440 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:42.440 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.440 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.440 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.440 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:42.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:42.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 1 -- 172.21.15.189:0/3095095527 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e7c0017c0 con 0x7f1e8809b320 2024-09-16T14:47:42.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:42.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 1 -- 172.21.15.189:0/3095095527 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1e7c001b50 con 0x7f1e8809b320 2024-09-16T14:47:42.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:42.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e8d66d640 4 set_mon_vals no callback set 2024-09-16T14:47:42.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:42.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.422+0000 7f1e85ffb640 10 monclient: dump: 2024-09-16T14:47:42.441 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:42.441 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.441 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.441 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.441 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.441 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:42.441 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.441 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.441 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.441 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:42.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient: shutdown 2024-09-16T14:47:42.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:42.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 1 -- 172.21.15.189:0/3095095527 >> v1:172.21.15.189:6789/0 conn(0x7f1e8809b320 legacy=0x7f1e88136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 1 -- 172.21.15.189:0/3095095527 shutdown_connections 2024-09-16T14:47:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 1 -- 172.21.15.189:0/3095095527 >> 172.21.15.189:0/3095095527 conn(0x7f1e880a3320 msgr2=0x7f1e8809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 1 -- 172.21.15.189:0/3095095527 shutdown_connections 2024-09-16T14:47:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 1 -- 172.21.15.189:0/3095095527 wait complete. 2024-09-16T14:47:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient: build_initial_monmap 2024-09-16T14:47:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient: monmap: 2024-09-16T14:47:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:42.443 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 1 Processor -- start 2024-09-16T14:47:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 1 -- start start 2024-09-16T14:47:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient: init 2024-09-16T14:47:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient(hunting): picked mon.noname-a con 0x7f1e881315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient(hunting): picked mon.noname-c con 0x7f1e88132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient(hunting): picked mon.noname-b con 0x7f1e881208b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8809cd20 con 0x7f1e881315a0 2024-09-16T14:47:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8809a830 con 0x7f1e881208b0 2024-09-16T14:47:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8806b380 con 0x7f1e88132090 2024-09-16T14:47:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient(hunting): authenticate will time out at 1244.545660s 2024-09-16T14:47:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e877fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1e881208b0 0x7f1e881d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33562/0 (socket says 172.21.15.189:33562) 2024-09-16T14:47:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e877fe640 1 -- 172.21.15.189:0/3417897662 learned_addr learned my addr 172.21.15.189:0/3417897662 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290198065 0 0) 0x7f1e8809a830 con 0x7f1e881208b0 2024-09-16T14:47:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): my global_id is 95101 2024-09-16T14:47:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e74003530 con 0x7f1e881208b0 2024-09-16T14:47:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277459285 0 0) 0x7f1e8809cd20 con 0x7f1e881315a0 2024-09-16T14:47:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): my global_id is 124167 2024-09-16T14:47:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e8809a830 con 0x7f1e881315a0 2024-09-16T14:47:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227719376 0 0) 0x7f1e8806b380 con 0x7f1e88132090 2024-09-16T14:47:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): my global_id is 114476 2024-09-16T14:47:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e8809cd20 con 0x7f1e88132090 2024-09-16T14:47:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1688806541 0 0) 0x7f1e74003530 con 0x7f1e881208b0 2024-09-16T14:47:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e8806b380 con 0x7f1e881208b0 2024-09-16T14:47:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1e80001b10 con 0x7f1e881208b0 2024-09-16T14:47:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): dump: 2024-09-16T14:47:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.446 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 846180494 0 0) 0x7f1e8809a830 con 0x7f1e881315a0 2024-09-16T14:47:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e74003530 con 0x7f1e881315a0 2024-09-16T14:47:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4069356028 0 0) 0x7f1e8809cd20 con 0x7f1e88132090 2024-09-16T14:47:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e8809a830 con 0x7f1e88132090 2024-09-16T14:47:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1e78001840 con 0x7f1e881315a0 2024-09-16T14:47:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): dump: 2024-09-16T14:47:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.448 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1e7c001ad0 con 0x7f1e88132090 2024-09-16T14:47:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient(hunting): dump: 2024-09-16T14:47:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.449 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3225359697 0 0) 0x7f1e8806b380 con 0x7f1e881208b0 2024-09-16T14:47:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 >> v1:172.21.15.26:6790/0 conn(0x7f1e88132090 legacy=0x7f1e881201a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 >> v1:172.21.15.26:6789/0 conn(0x7f1e881315a0 legacy=0x7f1e8811fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 monclient: found mon.b 2024-09-16T14:47:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e880a0ed0 con 0x7f1e881208b0 2024-09-16T14:47:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 5 monclient: authenticate success, global_id 95101 2024-09-16T14:47:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient: _renew_subs 2024-09-16T14:47:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e88123480 con 0x7f1e881208b0 2024-09-16T14:47:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient: _renew_subs 2024-09-16T14:47:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e88121d50 con 0x7f1e881208b0 2024-09-16T14:47:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e800020a0 con 0x7f1e881208b0 2024-09-16T14:47:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1e80003b40 con 0x7f1e881208b0 2024-09-16T14:47:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e857fa640 4 set_mon_vals no callback set 2024-09-16T14:47:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 10 monclient: dump: 2024-09-16T14:47:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.451 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:47:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1e80003d60 con 0x7f1e881208b0 2024-09-16T14:47:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e8d66d640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e68005050 con 0x7f1e881208b0 2024-09-16T14:47:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(133..133 src has 73..133) v4 ==== 6989+0+0 (unknown 2876426089 0 0) 0x7f1e80044090 con 0x7f1e881208b0 2024-09-16T14:47:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.426+0000 7f1e86ffd640 10 monclient: get_auth_request con 0x7f1e74036600 auth_method 0 2024-09-16T14:47:42.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.434+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1e80004530 con 0x7f1e881208b0 2024-09-16T14:47:42.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.434+0000 7f1e6f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:42.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.434+0000 7f1e6f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:42.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.543+0000 7fec1b246640 10 monclient: get_monmap_and_config 2024-09-16T14:47:42.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.543+0000 7fec1b246640 10 monclient: build_initial_monmap 2024-09-16T14:47:42.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.543+0000 7fec1b246640 10 monclient: monmap: 2024-09-16T14:47:42.549 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:42.549 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.549 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:42.548842+0000 2024-09-16T14:47:42.549 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:42.548842+0000 2024-09-16T14:47:42.549 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:42.549 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.549 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:42.549 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:42.549 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:42.549 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 1 Processor -- start 2024-09-16T14:47:42.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 1 -- start start 2024-09-16T14:47:42.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 10 monclient: init 2024-09-16T14:47:42.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:42.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:42.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 10 monclient(hunting): picked mon.noname-a con 0x7fec1409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:42.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 10 monclient(hunting): picked mon.noname-b con 0x7fec14132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:42.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 10 monclient(hunting): picked mon.noname-c con 0x7fec14138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:42.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec1409d260 con 0x7fec1409cb80 2024-09-16T14:47:42.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec14132a80 con 0x7fec14138ff0 2024-09-16T14:47:42.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec1409cf60 con 0x7fec14132650 2024-09-16T14:47:42.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:42.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 10 monclient(hunting): authenticate will time out at 1286.669098s 2024-09-16T14:47:42.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec18fe2640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fec1409cb80 0x7fec140717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47936/0 (socket says 172.21.15.26:47936) 2024-09-16T14:47:42.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec18fe2640 1 -- 172.21.15.26:0/3164842544 learned_addr learned my addr 172.21.15.26:0/3164842544 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:42.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec13fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fec14132650 0x7fec141378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55288/0 (socket says 172.21.15.26:55288) 2024-09-16T14:47:42.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 1 -- 172.21.15.26:0/3164842544 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4139588960 0 0) 0x7fec1409d260 con 0x7fec1409cb80 2024-09-16T14:47:42.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient(hunting): my global_id is 124170 2024-09-16T14:47:42.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 1 -- 172.21.15.26:0/3164842544 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febfc003530 con 0x7fec1409cb80 2024-09-16T14:47:42.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 1 -- 172.21.15.26:0/3164842544 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459654367 0 0) 0x7fec1409cf60 con 0x7fec14132650 2024-09-16T14:47:42.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient(hunting): my global_id is 114479 2024-09-16T14:47:42.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 1 -- 172.21.15.26:0/3164842544 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec1409d260 con 0x7fec14132650 2024-09-16T14:47:42.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 1 -- 172.21.15.26:0/3164842544 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 264312365 0 0) 0x7fec1409d260 con 0x7fec14132650 2024-09-16T14:47:42.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 1 -- 172.21.15.26:0/3164842544 >> v1:172.21.15.189:6789/0 conn(0x7fec14138ff0 legacy=0x7fec1413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 1 -- 172.21.15.26:0/3164842544 >> v1:172.21.15.26:6789/0 conn(0x7fec1409cb80 legacy=0x7fec140717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:42.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 1 monclient: found mon.noname-b 2024-09-16T14:47:42.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:42.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:47:42.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 1 -- 172.21.15.26:0/3164842544 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1409f2d0 con 0x7fec14132650 2024-09-16T14:47:42.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:42.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:42.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:42.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 1 -- 172.21.15.26:0/3164842544 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec08001660 con 0x7fec14132650 2024-09-16T14:47:42.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:42.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient: dump: 2024-09-16T14:47:42.556 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.556 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.556 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.556 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.556 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.556 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.556 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.556 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.556 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.556 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 5 monclient: authenticate success, global_id 114479 2024-09-16T14:47:42.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:42.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 1 -- 172.21.15.26:0/3164842544 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec080017c0 con 0x7fec14132650 2024-09-16T14:47:42.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:42.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 1 -- 172.21.15.26:0/3164842544 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec08001b10 con 0x7fec14132650 2024-09-16T14:47:42.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:42.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 4 set_mon_vals no callback set 2024-09-16T14:47:42.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:42.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec137fe640 10 monclient: dump: 2024-09-16T14:47:42.557 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.557 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.557 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.557 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.557 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.558 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.558 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.558 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.558 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.558 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 10 monclient: shutdown 2024-09-16T14:47:42.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:42.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 1 -- 172.21.15.26:0/3164842544 >> v1:172.21.15.26:6790/0 conn(0x7fec14132650 legacy=0x7fec141378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 1 -- 172.21.15.26:0/3164842544 shutdown_connections 2024-09-16T14:47:42.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 1 -- 172.21.15.26:0/3164842544 >> 172.21.15.26:0/3164842544 conn(0x7fec1409fb50 msgr2=0x7fec1409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:42.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 1 -- 172.21.15.26:0/3164842544 shutdown_connections 2024-09-16T14:47:42.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 1 -- 172.21.15.26:0/3164842544 wait complete. 2024-09-16T14:47:42.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 10 monclient: build_initial_monmap 2024-09-16T14:47:42.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 10 monclient: monmap: 2024-09-16T14:47:42.559 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:42.559 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:42.559 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:42.559 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:42.559 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:42.559 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:42.559 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:42.559 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:42.559 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:42.559 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 1 Processor -- start 2024-09-16T14:47:42.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 1 -- start start 2024-09-16T14:47:42.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.547+0000 7fec1b246640 10 monclient: init 2024-09-16T14:47:42.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:42.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:42.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 10 monclient(hunting): picked mon.noname-b con 0x7fec1409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:42.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 10 monclient(hunting): picked mon.noname-a con 0x7fec14132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:42.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 10 monclient(hunting): picked mon.noname-c con 0x7fec14138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:42.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec1409a240 con 0x7fec14132650 2024-09-16T14:47:42.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec14130120 con 0x7fec1409cb80 2024-09-16T14:47:42.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec1406fe20 con 0x7fec14138ff0 2024-09-16T14:47:42.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:42.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 10 monclient(hunting): authenticate will time out at 1286.672211s 2024-09-16T14:47:42.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec13fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fec14132650 0x7fec1406e220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47944/0 (socket says 172.21.15.26:47944) 2024-09-16T14:47:42.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec197e3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fec14138ff0 0x7fec141cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55294/0 (socket says 172.21.15.26:55294) 2024-09-16T14:47:42.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec13fff640 1 -- 172.21.15.26:0/286146481 learned_addr learned my addr 172.21.15.26:0/286146481 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:42.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368260573 0 0) 0x7fec1406fe20 con 0x7fec14138ff0 2024-09-16T14:47:42.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): my global_id is 114482 2024-09-16T14:47:42.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec00003530 con 0x7fec14138ff0 2024-09-16T14:47:42.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912312173 0 0) 0x7fec1409a240 con 0x7fec14132650 2024-09-16T14:47:42.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): my global_id is 124173 2024-09-16T14:47:42.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec1406fe20 con 0x7fec14132650 2024-09-16T14:47:42.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3898433150 0 0) 0x7fec1406fe20 con 0x7fec14132650 2024-09-16T14:47:42.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec1409a240 con 0x7fec14132650 2024-09-16T14:47:42.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec08001560 con 0x7fec14132650 2024-09-16T14:47:42.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:42.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): dump: 2024-09-16T14:47:42.567 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.567 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.567 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.567 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.568 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.568 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.568 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.568 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.568 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.568 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:42.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820950640 0 0) 0x7fec14130120 con 0x7fec1409cb80 2024-09-16T14:47:42.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): my global_id is 95107 2024-09-16T14:47:42.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec1406fe20 con 0x7fec1409cb80 2024-09-16T14:47:42.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2587169787 0 0) 0x7fec00003530 con 0x7fec14138ff0 2024-09-16T14:47:42.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec14130120 con 0x7fec14138ff0 2024-09-16T14:47:42.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec040017c0 con 0x7fec14138ff0 2024-09-16T14:47:42.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:42.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): dump: 2024-09-16T14:47:42.569 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.569 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.569 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.569 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.569 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.569 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.569 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.569 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.570 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.570 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4146502914 0 0) 0x7fec1409a240 con 0x7fec14132650 2024-09-16T14:47:42.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 >> v1:172.21.15.26:6790/0 conn(0x7fec14138ff0 legacy=0x7fec141cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 >> v1:172.21.15.189:6789/0 conn(0x7fec1409cb80 legacy=0x7fec1406db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient: _finish_hunting 0 2024-09-16T14:47:42.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 monclient: found mon.a 2024-09-16T14:47:42.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:42.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec140a28e0 con 0x7fec14132650 2024-09-16T14:47:42.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient: _finish_auth 0 2024-09-16T14:47:42.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient: _check_auth_tickets 2024-09-16T14:47:42.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:42.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 5 monclient: authenticate success, global_id 124173 2024-09-16T14:47:42.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 10 monclient: _renew_subs 2024-09-16T14:47:42.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 1 -- 172.21.15.26:0/286146481 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1406f4d0 con 0x7fec14132650 2024-09-16T14:47:42.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 10 monclient: _renew_subs 2024-09-16T14:47:42.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 1 -- 172.21.15.26:0/286146481 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1406e930 con 0x7fec14132650 2024-09-16T14:47:42.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec08004b10 con 0x7fec14132650 2024-09-16T14:47:42.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:42.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec08004ea0 con 0x7fec14132650 2024-09-16T14:47:42.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:42.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec12ffd640 4 set_mon_vals no callback set 2024-09-16T14:47:42.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient: dump: 2024-09-16T14:47:42.572 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.572 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.572 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.572 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.572 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.572 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.572 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.572 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.572 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.572 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:42.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec1b246640 1 -- 172.21.15.26:0/286146481 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7febf4006590 con 0x7febf4002830 2024-09-16T14:47:42.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fec08005080 con 0x7fec14132650 2024-09-16T14:47:42.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(133..133 src has 73..133) v4 ==== 6989+0+0 (unknown 2876426089 0 0) 0x7fec080441b0 con 0x7fec14132650 2024-09-16T14:47:42.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738973433 0 0) 0x7febf4006590 con 0x7febf4002830 2024-09-16T14:47:42.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient(hunting): my global_id is 95110 2024-09-16T14:47:42.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec1409a240 con 0x7febf4002830 2024-09-16T14:47:42.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec197e3640 10 monclient: get_auth_request con 0x7fec000363f0 auth_method 0 2024-09-16T14:47:42.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 133696648 0 0) 0x7fec1409a240 con 0x7febf4002830 2024-09-16T14:47:42.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febf4006590 con 0x7febf4002830 2024-09-16T14:47:42.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec0c001aa0 con 0x7febf4002830 2024-09-16T14:47:42.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:42.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 10 monclient: dump: 2024-09-16T14:47:42.574 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.574 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.574 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.574 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.574 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.574 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.574 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.574 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.574 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.574 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4184561073 0 0) 0x7febf4006590 con 0x7febf4002830 2024-09-16T14:47:42.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.551+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7febf400a0e0 con 0x7febf4002830 2024-09-16T14:47:42.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.555+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fec0c001e90 con 0x7febf4002830 2024-09-16T14:47:42.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.555+0000 7fec117fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:42.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.555+0000 7fec117fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:42.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.555+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 >> v1:172.21.15.189:6789/0 conn(0x7febf4002830 legacy=0x7febf4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.555+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 >> v1:172.21.15.189:6789/0 conn(0x7febf4002830 legacy=0x7febf4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:42.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.559+0000 7fec1b246640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:47:42.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.559+0000 7fec1b246640 1 -- 172.21.15.26:0/286146481 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7febf4009da0 con 0x7febf40090c0 2024-09-16T14:47:42.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.559+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117882763 0 0) 0x7febf4009da0 con 0x7febf40090c0 2024-09-16T14:47:42.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.559+0000 7fec117fa640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.559+0000 7fec117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.559+0000 7fec117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.559+0000 7fec117fa640 10 monclient(hunting): my global_id is 95113 2024-09-16T14:47:42.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.559+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febf4006590 con 0x7febf40090c0 2024-09-16T14:47:42.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.559+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 815861746 0 0) 0x7febf4006590 con 0x7febf40090c0 2024-09-16T14:47:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.559+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febf4009da0 con 0x7febf40090c0 2024-09-16T14:47:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.559+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec0c004d10 con 0x7febf40090c0 2024-09-16T14:47:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.559+0000 7fec117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.559+0000 7fec117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:47:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.559+0000 7fec117fa640 10 monclient: dump: 2024-09-16T14:47:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.577 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.559+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2296904431 0 0) 0x7febf4009da0 con 0x7febf40090c0 2024-09-16T14:47:42.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.559+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7febf400c290 con 0x7febf40090c0 2024-09-16T14:47:42.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.563+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 1576967483) 0x7fec0c007040 con 0x7febf40090c0 2024-09-16T14:47:42.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.563+0000 7fec117fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:47:42.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.563+0000 7fec117fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:42.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.563+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 >> v1:172.21.15.189:6789/0 conn(0x7febf40090c0 legacy=0x7febf4009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.563+0000 7fec117fa640 1 -- 172.21.15.26:0/286146481 >> v1:172.21.15.189:6789/0 conn(0x7febf40090c0 legacy=0x7febf4009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:42.577 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:47:42.578 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:42.578 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:42.578 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:47:42.578 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 48, 2024-09-16T14:47:42.578 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:47:42.578 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:47:42.578 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:47:42.578 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:47:42.578 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:42.578 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:47:42.578 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:42.578 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:47:42.578 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:47:42.579 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:42.579 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:42.579 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:42.579 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:42.579 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:42.579 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:42.579 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:42.579 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:42.579 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:42.579 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:42.579 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:42.579 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:47:42.579 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:47:42.579 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:42.580 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:42.580 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:42.580 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:42.580 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:42.580 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:42.580 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:42.580 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:42.580 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:42.580 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:42.580 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.580 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:47:42.580 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:47:42.581 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:47:42.581 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:47:42.581 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:47:42.581 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:47:42.581 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:42.581 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:42.581 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:47:42.581 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:47:42.581 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:47:42.581 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:47:42.581 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:47:42.581 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:47:42.581 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:47:42.582 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:42.582 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:47:42.582 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:42.582 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:42.582 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:42.582 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:42.582 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:42.582 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:42.582 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:42.582 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:42.582 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:42.583 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:42.583 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:42.583 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:47:42.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.583 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:47:42.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.583 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:42.583 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:42.583 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:42.583 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:42.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.583 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:42.583 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:47:42.584 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:42.584 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.584 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.584 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:47:42.584 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:47:42.584 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:42.584 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:42.584 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:42.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.584 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:42.584 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:42.585 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:42.585 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:42.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.585 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:42.585 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:47:42.585 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:42.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.585 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.585 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:47:42.585 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:47:42.585 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:42.585 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:42.585 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:42.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.586 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:42.586 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:42.586 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:42.586 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:42.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.586 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:42.586 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:47:42.586 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:42.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.586 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.587 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:47:42.587 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:47:42.587 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:42.587 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:42.587 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:42.587 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.587 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.587 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:47:42.587 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:47:42.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.587 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:42.587 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:42.587 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:42.588 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.588 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:42.588 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:47:42.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.588 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:42.588 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:42.588 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:42.588 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.588 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:42.588 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:47:42.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:42.588 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:42.588 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:42.588 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:47:42.589 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:42.589 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:42.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:42.589 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:47:42.589 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:47:42.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.563+0000 7fec1b246640 1 -- 172.21.15.26:0/286146481 >> v1:172.21.15.189:6812/3196161582 conn(0x7fec000363f0 legacy=0x7fec000388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.563+0000 7fec1b246640 10 monclient: shutdown 2024-09-16T14:47:42.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.563+0000 7fec1b246640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:42.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.563+0000 7fec1b246640 1 -- 172.21.15.26:0/286146481 >> v1:172.21.15.26:6789/0 conn(0x7fec14132650 legacy=0x7fec1406e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.563+0000 7fec197e3640 1 -- 172.21.15.26:0/286146481 reap_dead start 2024-09-16T14:47:42.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.563+0000 7fec197e3640 1 -- 172.21.15.26:0/286146481 reap_dead start 2024-09-16T14:47:42.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.563+0000 7fec1b246640 1 -- 172.21.15.26:0/286146481 shutdown_connections 2024-09-16T14:47:42.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.563+0000 7fec1b246640 1 -- 172.21.15.26:0/286146481 >> 172.21.15.26:0/286146481 conn(0x7fec1409fb50 msgr2=0x7fec14131e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:42.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.563+0000 7fec1b246640 1 -- 172.21.15.26:0/286146481 shutdown_connections 2024-09-16T14:47:42.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.563+0000 7fec1b246640 1 -- 172.21.15.26:0/286146481 wait complete. 2024-09-16T14:47:42.591 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:47:42.591 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T14:47:42.591 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:47:42.591 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T14:47:42.592 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:47:42.592 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T14:47:42.592 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:47:42.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e877fe640 1 -- 172.21.15.189:0/3417897662 >> v1:172.21.15.189:6789/0 conn(0x7f1e881208b0 legacy=0x7f1e881d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T14:47:42.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e877fe640 1 -- 172.21.15.189:0/3417897662 >> v1:172.21.15.189:6789/0 conn(0x7f1e881208b0 legacy=0x7f1e881d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:47:42.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e877fe640 1 --1- 172.21.15.189:0/3417897662 >> v1:172.21.15.189:6789/0 conn(0x7f1e881208b0 0x7f1e881d1680 :-1 s=OPENED pgs=690 cs=1 l=1).handle_message read tag failed 2024-09-16T14:47:42.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e877fe640 1 --1- 172.21.15.189:0/3417897662 >> v1:172.21.15.189:6789/0 conn(0x7f1e881208b0 0x7f1e881d1680 :-1 s=OPENED pgs=690 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:47:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 10 monclient: ms_handle_reset current mon v1:172.21.15.189:6789/0 2024-09-16T14:47:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 >> v1:172.21.15.189:6789/0 conn(0x7f1e881208b0 legacy=0x7f1e881d1680 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 10 monclient(hunting): picked mon.b con 0x7f1e74040aa0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 10 monclient(hunting): picked mon.a con 0x7f1e881315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 10 monclient(hunting): picked mon.c con 0x7f1e88132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e867fc640 1 -- 172.21.15.189:0/3417897662 >> v1:172.21.15.189:6789/0 conn(0x7f1e74040aa0 legacy=0x7f1e74042eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1e8806b380 con 0x7f1e881315a0 2024-09-16T14:47:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1e74003530 con 0x7f1e74040aa0 2024-09-16T14:47:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1e8809cd20 con 0x7f1e88132090 2024-09-16T14:47:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:42.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920003109 0 0) 0x7f1e8806b380 con 0x7f1e881315a0 2024-09-16T14:47:42.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:47:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.26:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f1e7404bf70 con 0x7f1e881315a0 2024-09-16T14:47:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442436524 0 0) 0x7f1e8809cd20 con 0x7f1e88132090 2024-09-16T14:47:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:47:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.594+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.26:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f1e8806b380 con 0x7f1e88132090 2024-09-16T14:47:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1380799578 0 0) 0x7f1e7404bf70 con 0x7f1e881315a0 2024-09-16T14:47:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 >> v1:172.21.15.26:6790/0 conn(0x7f1e88132090 legacy=0x7f1e7404a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 >> v1:172.21.15.189:6789/0 conn(0x7f1e74040aa0 legacy=0x7f1e74042eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 1 monclient: found mon.a 2024-09-16T14:47:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f1e7404f0c0 con 0x7f1e881315a0 2024-09-16T14:47:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1e80014b10 con 0x7f1e881315a0 2024-09-16T14:47:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 10 monclient: dump: 2024-09-16T14:47:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.605 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.605 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.605 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:42.605 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.605 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.605 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.605 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:42.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e8001ced0 con 0x7f1e881315a0 2024-09-16T14:47:42.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:42.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1e800414b0 con 0x7f1e881315a0 2024-09-16T14:47:42.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.598+0000 7f1e857fa640 4 set_mon_vals no callback set 2024-09-16T14:47:42.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.602+0000 7f1e8d66d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:42.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.602+0000 7f1e8d66d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:42.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.602+0000 7f1e8d66d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.602+0000 7f1e8d66d640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f1e68005880 con 0x7f1e881315a0 2024-09-16T14:47:42.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.602+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) v1 ==== 71+0+7 (unknown 1899167245 0 1352219128) 0x7f1e800146b0 con 0x7f1e881315a0 2024-09-16T14:47:42.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.602+0000 7f1e6f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:42.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.602+0000 7f1e6f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:42.613 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T14:47:42.613 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:47:42.613 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:47:42.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.706+0000 7f1e8d66d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:42.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.706+0000 7f1e8d66d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:42.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.706+0000 7f1e8d66d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.706+0000 7f1e8d66d640 1 -- 172.21.15.189:0/3417897662 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f1e68002290 con 0x7f1e881315a0 2024-09-16T14:47:42.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.706+0000 7f1e6f7fe640 1 -- 172.21.15.189:0/3417897662 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) v1 ==== 71+0+7 (unknown 1899167245 0 1352219128) 0x7f1e80018ae0 con 0x7f1e881315a0 2024-09-16T14:47:42.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.706+0000 7f1e6f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:42.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.706+0000 7f1e6f7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:42.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.710+0000 7f1e8d66d640 1 -- 172.21.15.189:0/3417897662 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1e74036600 legacy=0x7f1e74038ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.710+0000 7f1e8d66d640 10 monclient: shutdown 2024-09-16T14:47:42.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.710+0000 7f1e8d66d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:42.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.710+0000 7f1e8d66d640 1 -- 172.21.15.189:0/3417897662 >> v1:172.21.15.26:6789/0 conn(0x7f1e881315a0 legacy=0x7f1e74046900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.710+0000 7f1e8d66d640 1 -- 172.21.15.189:0/3417897662 shutdown_connections 2024-09-16T14:47:42.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.710+0000 7f1e8d66d640 1 -- 172.21.15.189:0/3417897662 >> 172.21.15.189:0/3417897662 conn(0x7f1e880a3320 msgr2=0x7f1e8809d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:42.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.710+0000 7f1e8d66d640 1 -- 172.21.15.189:0/3417897662 shutdown_connections 2024-09-16T14:47:42.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.710+0000 7f1e8d66d640 1 -- 172.21.15.189:0/3417897662 wait complete. 2024-09-16T14:47:42.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.723+0000 7f47fcd07640 10 monclient: get_monmap_and_config 2024-09-16T14:47:42.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.723+0000 7f47fcd07640 10 monclient: build_initial_monmap 2024-09-16T14:47:42.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.723+0000 7f47fcd07640 10 monclient: monmap: 2024-09-16T14:47:42.727 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:42.727 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.727 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:42.727315+0000 2024-09-16T14:47:42.727 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:42.727315+0000 2024-09-16T14:47:42.727 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:42.727 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.727 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:42.727 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:42.727 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:42.728 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.723+0000 7f47fcd07640 1 Processor -- start 2024-09-16T14:47:42.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.723+0000 7f47fcd07640 1 -- start start 2024-09-16T14:47:42.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.723+0000 7f47fcd07640 10 monclient: init 2024-09-16T14:47:42.735 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set ls 2024-09-16T14:47:42.735 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep barset 2024-09-16T14:47:42.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.723+0000 7f47fcd07640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:42.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.723+0000 7f47fcd07640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:42.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.723+0000 7f47fcd07640 10 monclient(hunting): picked mon.noname-b con 0x7f47f80a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:42.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.723+0000 7f47fcd07640 10 monclient(hunting): picked mon.noname-c con 0x7f47f8099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:42.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.723+0000 7f47fcd07640 10 monclient(hunting): picked mon.noname-a con 0x7f47f80a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:42.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.723+0000 7f47fcd07640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f8098ea0 con 0x7f47f80a26f0 2024-09-16T14:47:42.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.723+0000 7f47fcd07640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f80997b0 con 0x7f47f8099b60 2024-09-16T14:47:42.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.723+0000 7f47fcd07640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f8099460 con 0x7f47f80a1c00 2024-09-16T14:47:42.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.723+0000 7f47fcd07640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:42.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.723+0000 7f47fcd07640 10 monclient(hunting): authenticate will time out at 1286.847988s 2024-09-16T14:47:42.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f6d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f47f80a26f0 0x7f47f813d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47974/0 (socket says 172.21.15.26:47974) 2024-09-16T14:47:42.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f6d9d640 1 -- 172.21.15.26:0/3448298600 learned_addr learned my addr 172.21.15.26:0/3448298600 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:42.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47effff640 1 -- 172.21.15.26:0/3448298600 >> v1:172.21.15.189:6789/0 conn(0x7f47f8099b60 legacy=0x7f47f8139c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:42.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 1 -- 172.21.15.26:0/3448298600 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473346583 0 0) 0x7f47f8098ea0 con 0x7f47f80a26f0 2024-09-16T14:47:42.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient(hunting): my global_id is 124176 2024-09-16T14:47:42.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 1 -- 172.21.15.26:0/3448298600 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e4003530 con 0x7f47f80a26f0 2024-09-16T14:47:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 1 -- 172.21.15.26:0/3448298600 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038281403 0 0) 0x7f47f8099460 con 0x7f47f80a1c00 2024-09-16T14:47:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient(hunting): my global_id is 114485 2024-09-16T14:47:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 1 -- 172.21.15.26:0/3448298600 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f8098ea0 con 0x7f47f80a1c00 2024-09-16T14:47:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 1 -- 172.21.15.26:0/3448298600 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3238923314 0 0) 0x7f47e4003530 con 0x7f47f80a26f0 2024-09-16T14:47:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 1 -- 172.21.15.26:0/3448298600 >> v1:172.21.15.26:6790/0 conn(0x7f47f80a1c00 legacy=0x7f47f80a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 1 -- 172.21.15.26:0/3448298600 >> v1:172.21.15.189:6789/0 conn(0x7f47f8099b60 legacy=0x7f47f8139c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:47:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 1 monclient: found mon.noname-a 2024-09-16T14:47:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 1 -- 172.21.15.26:0/3448298600 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f80a34b0 con 0x7f47f80a26f0 2024-09-16T14:47:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient: _finish_auth 0 2024-09-16T14:47:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:47:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 1 -- 172.21.15.26:0/3448298600 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f47f00019a0 con 0x7f47f80a26f0 2024-09-16T14:47:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient: dump: 2024-09-16T14:47:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.742 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 5 monclient: authenticate success, global_id 124176 2024-09-16T14:47:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 1 -- 172.21.15.26:0/3448298600 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47f0001c00 con 0x7f47f80a26f0 2024-09-16T14:47:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 1 -- 172.21.15.26:0/3448298600 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f47f0001fd0 con 0x7f47f80a26f0 2024-09-16T14:47:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 4 set_mon_vals no callback set 2024-09-16T14:47:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f5d9b640 10 monclient: dump: 2024-09-16T14:47:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.743 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 10 monclient: shutdown 2024-09-16T14:47:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 1 -- 172.21.15.26:0/3448298600 >> v1:172.21.15.26:6789/0 conn(0x7f47f80a26f0 legacy=0x7f47f813d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 1 -- 172.21.15.26:0/3448298600 shutdown_connections 2024-09-16T14:47:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 1 -- 172.21.15.26:0/3448298600 >> 172.21.15.26:0/3448298600 conn(0x7f47f80a3d00 msgr2=0x7f47f80a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 1 -- 172.21.15.26:0/3448298600 shutdown_connections 2024-09-16T14:47:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 1 -- 172.21.15.26:0/3448298600 wait complete. 2024-09-16T14:47:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 10 monclient: build_initial_monmap 2024-09-16T14:47:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 10 monclient: monmap: 2024-09-16T14:47:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:42.745 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 1 Processor -- start 2024-09-16T14:47:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 1 -- start start 2024-09-16T14:47:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 10 monclient: init 2024-09-16T14:47:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 10 monclient(hunting): picked mon.noname-b con 0x7f47f8099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 10 monclient(hunting): picked mon.noname-a con 0x7f47f80a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 10 monclient(hunting): picked mon.noname-c con 0x7f47f80a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f811f700 con 0x7f47f80a1c00 2024-09-16T14:47:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f811f4b0 con 0x7f47f8099b60 2024-09-16T14:47:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f809efa0 con 0x7f47f80a26f0 2024-09-16T14:47:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 10 monclient(hunting): authenticate will time out at 1286.851006s 2024-09-16T14:47:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47f659c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f47f8099b60 legacy=0x7f47f8138080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47effff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f47f80a1c00 0x7f47f81ce210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47982/0 (socket says 172.21.15.26:47982) 2024-09-16T14:47:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47effff640 1 -- 172.21.15.26:0/786029397 learned_addr learned my addr 172.21.15.26:0/786029397 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687849293 0 0) 0x7f47f809efa0 con 0x7f47f80a26f0 2024-09-16T14:47:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient(hunting): my global_id is 114488 2024-09-16T14:47:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47d8003530 con 0x7f47f80a26f0 2024-09-16T14:47:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691668653 0 0) 0x7f47f811f700 con 0x7f47f80a1c00 2024-09-16T14:47:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient(hunting): my global_id is 124179 2024-09-16T14:47:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f809efa0 con 0x7f47f80a1c00 2024-09-16T14:47:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 113839949 0 0) 0x7f47d8003530 con 0x7f47f80a26f0 2024-09-16T14:47:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47f811f700 con 0x7f47f80a26f0 2024-09-16T14:47:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f47f00018a0 con 0x7f47f80a26f0 2024-09-16T14:47:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient(hunting): dump: 2024-09-16T14:47:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.748 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 383470223 0 0) 0x7f47f809efa0 con 0x7f47f80a1c00 2024-09-16T14:47:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47d8003530 con 0x7f47f80a1c00 2024-09-16T14:47:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f47e0004060 con 0x7f47f80a1c00 2024-09-16T14:47:42.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:42.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient(hunting): dump: 2024-09-16T14:47:42.750 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.750 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.750 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.750 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.750 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.750 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.750 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.750 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.750 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.750 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1280919473 0 0) 0x7f47f811f700 con 0x7f47f80a26f0 2024-09-16T14:47:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 >> v1:172.21.15.189:6789/0 conn(0x7f47f8099b60 legacy=0x7f47f8138080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 >> v1:172.21.15.26:6789/0 conn(0x7f47f80a1c00 legacy=0x7f47f81ce210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 1 monclient: found mon.c 2024-09-16T14:47:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f809ea50 con 0x7f47f80a26f0 2024-09-16T14:47:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient: _finish_auth 0 2024-09-16T14:47:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 5 monclient: authenticate success, global_id 114488 2024-09-16T14:47:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 10 monclient: _renew_subs 2024-09-16T14:47:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.727+0000 7f47fcd07640 1 -- 172.21.15.26:0/786029397 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f809e3f0 con 0x7f47f80a26f0 2024-09-16T14:47:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47fcd07640 10 monclient: _renew_subs 2024-09-16T14:47:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47fcd07640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47fcd07640 1 -- 172.21.15.26:0/786029397 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f811e470 con 0x7f47f80a26f0 2024-09-16T14:47:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47f0001370 con 0x7f47f80a26f0 2024-09-16T14:47:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47eeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f47f0005120 con 0x7f47f80a26f0 2024-09-16T14:47:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47f559a640 4 set_mon_vals no callback set 2024-09-16T14:47:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47eeffd640 10 monclient: dump: 2024-09-16T14:47:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.754 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47fcd07640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47fcd07640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47fcd07640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47fcd07640 1 -- 172.21.15.26:0/786029397 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47dc005050 con 0x7f47f80a26f0 2024-09-16T14:47:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f47f000fb90 con 0x7f47f80a26f0 2024-09-16T14:47:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(133..133 src has 73..133) v4 ==== 6989+0+0 (unknown 2876426089 0 0) 0x7f47f0044300 con 0x7f47f80a26f0 2024-09-16T14:47:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.731+0000 7f47f659c640 10 monclient: get_auth_request con 0x7f47d8036270 auth_method 0 2024-09-16T14:47:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.735+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f47f0014a40 con 0x7f47f80a26f0 2024-09-16T14:47:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.735+0000 7f47eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.735+0000 7f47eeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:42.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 10 monclient: get_monmap_and_config 2024-09-16T14:47:42.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 10 monclient: build_initial_monmap 2024-09-16T14:47:42.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 10 monclient: monmap: 2024-09-16T14:47:42.838 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:42.839 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.839 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:42.838707+0000 2024-09-16T14:47:42.839 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:42.838707+0000 2024-09-16T14:47:42.839 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:42.839 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:42.839 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:42.839 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:42.839 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:42.839 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:42.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 1 Processor -- start 2024-09-16T14:47:42.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 1 -- start start 2024-09-16T14:47:42.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 10 monclient: init 2024-09-16T14:47:42.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:42.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:42.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 10 monclient(hunting): picked mon.noname-c con 0x7f997809df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:42.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 10 monclient(hunting): picked mon.noname-a con 0x7f997809e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:42.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 10 monclient(hunting): picked mon.noname-b con 0x7f997809e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:42.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f997809ee20 con 0x7f997809e2f0 2024-09-16T14:47:42.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f997809f100 con 0x7f997809df10 2024-09-16T14:47:42.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f997809f3e0 con 0x7f997809e740 2024-09-16T14:47:42.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:42.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 10 monclient(hunting): authenticate will time out at 1244.954313s 2024-09-16T14:47:42.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997659c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f997809df10 legacy=0x7f997809d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:42.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f9976d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f997809e740 0x7f997813b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:36254/0 (socket says 172.21.15.189:36254) 2024-09-16T14:47:42.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f9976d9d640 1 -- 172.21.15.189:0/2451228248 learned_addr learned my addr 172.21.15.189:0/2451228248 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:42.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f9975d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f997809e2f0 0x7f99781378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:37014/0 (socket says 172.21.15.189:37014) 2024-09-16T14:47:42.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 1 -- 172.21.15.189:0/2451228248 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364849001 0 0) 0x7f997809ee20 con 0x7f997809e2f0 2024-09-16T14:47:42.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient(hunting): my global_id is 124182 2024-09-16T14:47:42.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 1 -- 172.21.15.189:0/2451228248 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9960003530 con 0x7f997809e2f0 2024-09-16T14:47:42.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 1 -- 172.21.15.189:0/2451228248 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878751805 0 0) 0x7f997809f3e0 con 0x7f997809e740 2024-09-16T14:47:42.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient(hunting): my global_id is 114491 2024-09-16T14:47:42.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 1 -- 172.21.15.189:0/2451228248 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f997809ee20 con 0x7f997809e740 2024-09-16T14:47:42.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 1 -- 172.21.15.189:0/2451228248 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3132189722 0 0) 0x7f9960003530 con 0x7f997809e2f0 2024-09-16T14:47:42.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 1 -- 172.21.15.189:0/2451228248 >> v1:172.21.15.26:6790/0 conn(0x7f997809e740 legacy=0x7f997813b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 1 -- 172.21.15.189:0/2451228248 >> v1:172.21.15.189:6789/0 conn(0x7f997809df10 legacy=0x7f997809d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:42.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient: _finish_hunting 0 2024-09-16T14:47:42.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 1 monclient: found mon.noname-a 2024-09-16T14:47:42.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:42.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 1 -- 172.21.15.189:0/2451228248 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997809c690 con 0x7f997809e2f0 2024-09-16T14:47:42.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient: _finish_auth 0 2024-09-16T14:47:42.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient: _check_auth_tickets 2024-09-16T14:47:42.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:42.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 1 -- 172.21.15.189:0/2451228248 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9968001620 con 0x7f997809e2f0 2024-09-16T14:47:42.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:42.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient: dump: 2024-09-16T14:47:42.850 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:42.850 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.850 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.851 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.851 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.851 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:42.851 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.851 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.851 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.851 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:42.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 5 monclient: authenticate success, global_id 124182 2024-09-16T14:47:42.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:42.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 1 -- 172.21.15.189:0/2451228248 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9968001840 con 0x7f997809e2f0 2024-09-16T14:47:42.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:42.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:42.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997c916640 4 set_mon_vals no callback set 2024-09-16T14:47:42.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 1 -- 172.21.15.189:0/2451228248 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9968001bd0 con 0x7f997809e2f0 2024-09-16T14:47:42.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:42.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.834+0000 7f997559a640 10 monclient: dump: 2024-09-16T14:47:42.852 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:42.852 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.852 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.852 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.852 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.853 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:42.853 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.853 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.853 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.853 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:42.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 10 monclient: shutdown 2024-09-16T14:47:42.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:42.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 1 -- 172.21.15.189:0/2451228248 >> v1:172.21.15.26:6789/0 conn(0x7f997809e2f0 legacy=0x7f99781378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 1 -- 172.21.15.189:0/2451228248 shutdown_connections 2024-09-16T14:47:42.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 1 -- 172.21.15.189:0/2451228248 >> 172.21.15.189:0/2451228248 conn(0x7f99780a3320 msgr2=0x7f997809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:42.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 1 -- 172.21.15.189:0/2451228248 shutdown_connections 2024-09-16T14:47:42.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 1 -- 172.21.15.189:0/2451228248 wait complete. 2024-09-16T14:47:42.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 10 monclient: build_initial_monmap 2024-09-16T14:47:42.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 10 monclient: monmap: 2024-09-16T14:47:42.854 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:42.854 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:42.854 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:42.854 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:42.854 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:42.854 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:42.854 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:42.854 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:42.854 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:42.855 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:42.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 1 Processor -- start 2024-09-16T14:47:42.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 1 -- start start 2024-09-16T14:47:42.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 10 monclient: init 2024-09-16T14:47:42.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:42.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:42.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 10 monclient(hunting): picked mon.noname-c con 0x7f997809df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:42.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 10 monclient(hunting): picked mon.noname-b con 0x7f997809e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:42.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 10 monclient(hunting): picked mon.noname-a con 0x7f997809e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:42.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f997813b690 con 0x7f997809e740 2024-09-16T14:47:42.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f997806b380 con 0x7f997809e2f0 2024-09-16T14:47:42.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9978132490 con 0x7f997809df10 2024-09-16T14:47:42.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:42.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 10 monclient(hunting): authenticate will time out at 1244.957938s 2024-09-16T14:47:42.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9975d9b640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f997809e2f0 legacy=0x7f997813a100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:42.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9976d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f997809e740 0x7f99781cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:37024/0 (socket says 172.21.15.189:37024) 2024-09-16T14:47:42.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9976d9d640 1 -- 172.21.15.189:0/3323346031 learned_addr learned my addr 172.21.15.189:0/3323346031 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:42.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686776276 0 0) 0x7f997813b690 con 0x7f997809e740 2024-09-16T14:47:42.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient(hunting): my global_id is 124185 2024-09-16T14:47:42.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9958003530 con 0x7f997809e740 2024-09-16T14:47:42.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516522920 0 0) 0x7f9978132490 con 0x7f997809df10 2024-09-16T14:47:42.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:42.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:42.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:42.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient(hunting): my global_id is 114494 2024-09-16T14:47:42.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f997813b690 con 0x7f997809df10 2024-09-16T14:47:42.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2889435356 0 0) 0x7f9958003530 con 0x7f997809e740 2024-09-16T14:47:42.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9978132490 con 0x7f997809e740 2024-09-16T14:47:42.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9970001c00 con 0x7f997809e740 2024-09-16T14:47:42.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:42.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient(hunting): dump: 2024-09-16T14:47:42.858 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:42.858 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.858 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.858 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.858 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.858 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:42.858 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.858 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.858 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.858 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4238991075 0 0) 0x7f997813b690 con 0x7f997809df10 2024-09-16T14:47:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9958003530 con 0x7f997809df10 2024-09-16T14:47:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f996c004090 con 0x7f997809df10 2024-09-16T14:47:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient(hunting): dump: 2024-09-16T14:47:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.860 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.860 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:42.860 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.860 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.860 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.860 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:42.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3034082522 0 0) 0x7f9978132490 con 0x7f997809e740 2024-09-16T14:47:42.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 >> v1:172.21.15.26:6790/0 conn(0x7f997809df10 legacy=0x7f99781399f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 >> v1:172.21.15.189:6789/0 conn(0x7f997809e2f0 legacy=0x7f997813a100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:42.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:42.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 monclient: found mon.a 2024-09-16T14:47:42.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:42.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997811a380 con 0x7f997809e740 2024-09-16T14:47:42.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient: _finish_auth 0 2024-09-16T14:47:42.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:42.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:42.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 5 monclient: authenticate success, global_id 124185 2024-09-16T14:47:42.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 10 monclient: _renew_subs 2024-09-16T14:47:42.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 1 -- 172.21.15.189:0/3323346031 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997813a810 con 0x7f997809e740 2024-09-16T14:47:42.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 10 monclient: _renew_subs 2024-09-16T14:47:42.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f997c916640 1 -- 172.21.15.189:0/3323346031 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f997811a6c0 con 0x7f997809e740 2024-09-16T14:47:42.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9970004b70 con 0x7f997809e740 2024-09-16T14:47:42.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:42.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9974d99640 4 set_mon_vals no callback set 2024-09-16T14:47:42.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9970004e80 con 0x7f997809e740 2024-09-16T14:47:42.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:42.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:42.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.838+0000 7f9966ffd640 10 monclient: dump: 2024-09-16T14:47:42.862 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:42.862 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:42.862 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.862 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:42.863 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:42.863 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:42.863 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:42.863 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:42.863 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:42.863 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:42.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.842+0000 7f997c916640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:42.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.842+0000 7f997c916640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:42.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.842+0000 7f997c916640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:42.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.842+0000 7f997c916640 1 -- 172.21.15.189:0/3323346031 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f995c005050 con 0x7f997809e740 2024-09-16T14:47:42.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.842+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f997000f8f0 con 0x7f997809e740 2024-09-16T14:47:42.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.842+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(133..133 src has 73..133) v4 ==== 6989+0+0 (unknown 2876426089 0 0) 0x7f9970044150 con 0x7f997809e740 2024-09-16T14:47:42.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.842+0000 7f997659c640 10 monclient: get_auth_request con 0x7f9958036200 auth_method 0 2024-09-16T14:47:42.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.854+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9970014770 con 0x7f997809e740 2024-09-16T14:47:42.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.854+0000 7f9966ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:42.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:42.854+0000 7f9966ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:42.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.963+0000 7f47fcd07640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:47:42.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.963+0000 7f47fcd07640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:42.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.963+0000 7f47fcd07640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:42.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.963+0000 7f47fcd07640 1 -- 172.21.15.26:0/786029397 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f47dc005880 con 0x7f47f80a26f0 2024-09-16T14:47:42.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.963+0000 7f47eeffd640 1 -- 172.21.15.26:0/786029397 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 704991944) 0x7f47f00130e0 con 0x7f47f80a26f0 2024-09-16T14:47:42.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.963+0000 7f47eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:42.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.963+0000 7f47eeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:42.967 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:47:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.967+0000 7f47fcd07640 1 -- 172.21.15.26:0/786029397 >> v1:172.21.15.189:6812/3196161582 conn(0x7f47d8036270 legacy=0x7f47d8038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.967+0000 7f47fcd07640 10 monclient: shutdown 2024-09-16T14:47:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.967+0000 7f47fcd07640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.967+0000 7f47fcd07640 1 -- 172.21.15.26:0/786029397 >> v1:172.21.15.26:6790/0 conn(0x7f47f80a26f0 legacy=0x7f47f81d1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.967+0000 7f47fcd07640 1 -- 172.21.15.26:0/786029397 shutdown_connections 2024-09-16T14:47:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.967+0000 7f47fcd07640 1 -- 172.21.15.26:0/786029397 >> 172.21.15.26:0/786029397 conn(0x7f47f80a3d00 msgr2=0x7f47f80a0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.967+0000 7f47fcd07640 1 -- 172.21.15.26:0/786029397 shutdown_connections 2024-09-16T14:47:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:42.967+0000 7f47fcd07640 1 -- 172.21.15.26:0/786029397 wait complete. 2024-09-16T14:47:43.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.006+0000 7f997c916640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:43.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.006+0000 7f997c916640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:43.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.006+0000 7f997c916640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:43.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.006+0000 7f997c916640 1 -- 172.21.15.189:0/3323346031 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f995c005300 con 0x7f997809e740 2024-09-16T14:47:43.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.010+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) v1 ==== 71+0+7 (unknown 1899167245 0 1352219128) 0x7f9970014590 con 0x7f997809e740 2024-09-16T14:47:43.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.010+0000 7f9966ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:43.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.010+0000 7f9966ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:43.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.102+0000 7f997c916640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:43.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.102+0000 7f997c916640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:43.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.102+0000 7f997c916640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:43.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.102+0000 7f997c916640 1 -- 172.21.15.189:0/3323346031 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f995c005540 con 0x7f997809e740 2024-09-16T14:47:43.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.102+0000 7f9966ffd640 1 -- 172.21.15.189:0/3323346031 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) v1 ==== 71+0+7 (unknown 1899167245 0 1352219128) 0x7f9970014590 con 0x7f997809e740 2024-09-16T14:47:43.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.102+0000 7f9966ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:43.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.102+0000 7f9966ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:43.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.106+0000 7f997c916640 1 -- 172.21.15.189:0/3323346031 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9958036200 legacy=0x7f99580386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:43.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.106+0000 7f997c916640 10 monclient: shutdown 2024-09-16T14:47:43.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.106+0000 7f997c916640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:43.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.106+0000 7f997c916640 1 -- 172.21.15.189:0/3323346031 >> v1:172.21.15.26:6789/0 conn(0x7f997809e740 legacy=0x7f99781cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:43.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.106+0000 7f997c916640 1 -- 172.21.15.189:0/3323346031 shutdown_connections 2024-09-16T14:47:43.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.106+0000 7f997c916640 1 -- 172.21.15.189:0/3323346031 >> 172.21.15.189:0/3323346031 conn(0x7f99780a3320 msgr2=0x7f99780714c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:43.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.106+0000 7f997c916640 1 -- 172.21.15.189:0/3323346031 shutdown_connections 2024-09-16T14:47:43.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.106+0000 7f997c916640 1 -- 172.21.15.189:0/3323346031 wait complete. 2024-09-16T14:47:43.124 INFO:tasks.workunit.client.0.smithi189.stdout:barset 2024-09-16T14:47:43.126 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set rm barset 2024-09-16T14:47:43.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.234+0000 7f7a28ff5640 10 monclient: get_monmap_and_config 2024-09-16T14:47:43.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.234+0000 7f7a28ff5640 10 monclient: build_initial_monmap 2024-09-16T14:47:43.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.234+0000 7f7a28ff5640 10 monclient: monmap: 2024-09-16T14:47:43.242 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:43.242 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:43.242 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:43.241513+0000 2024-09-16T14:47:43.242 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:43.241513+0000 2024-09-16T14:47:43.242 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:43.242 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:43.242 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:43.242 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:43.242 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:43.242 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:43.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 1 Processor -- start 2024-09-16T14:47:43.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 1 -- start start 2024-09-16T14:47:43.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 10 monclient: init 2024-09-16T14:47:43.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:43.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:43.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 10 monclient(hunting): picked mon.noname-a con 0x7f7a2409df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:43.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 10 monclient(hunting): picked mon.noname-c con 0x7f7a2409e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:43.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 10 monclient(hunting): picked mon.noname-b con 0x7f7a2409e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:43.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2409ee20 con 0x7f7a2409df10 2024-09-16T14:47:43.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2409f100 con 0x7f7a2409e2f0 2024-09-16T14:47:43.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2409f3e0 con 0x7f7a2409e740 2024-09-16T14:47:43.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:43.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 10 monclient(hunting): authenticate will time out at 1245.357139s 2024-09-16T14:47:43.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a21d9b640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f7a2409e2f0 legacy=0x7f7a241378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:43.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2259c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7a2409df10 0x7f7a2409d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:37034/0 (socket says 172.21.15.189:37034) 2024-09-16T14:47:43.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2259c640 1 -- 172.21.15.189:0/2483786172 learned_addr learned my addr 172.21.15.189:0/2483786172 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:43.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 1 -- 172.21.15.189:0/2483786172 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987519365 0 0) 0x7f7a2409f3e0 con 0x7f7a2409e740 2024-09-16T14:47:43.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:43.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:43.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:43.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient(hunting): my global_id is 114497 2024-09-16T14:47:43.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 1 -- 172.21.15.189:0/2483786172 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a0c003530 con 0x7f7a2409e740 2024-09-16T14:47:43.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 1 -- 172.21.15.189:0/2483786172 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779469401 0 0) 0x7f7a2409ee20 con 0x7f7a2409df10 2024-09-16T14:47:43.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient(hunting): _negotiate 2024-09-16T14:47:43.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:43.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:43.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient(hunting): my global_id is 124188 2024-09-16T14:47:43.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 1 -- 172.21.15.189:0/2483786172 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a2409f3e0 con 0x7f7a2409df10 2024-09-16T14:47:43.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 1 -- 172.21.15.189:0/2483786172 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2648366963 0 0) 0x7f7a2409f3e0 con 0x7f7a2409df10 2024-09-16T14:47:43.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 1 -- 172.21.15.189:0/2483786172 >> v1:172.21.15.26:6790/0 conn(0x7f7a2409e740 legacy=0x7f7a2413b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:43.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 1 -- 172.21.15.189:0/2483786172 >> v1:172.21.15.189:6789/0 conn(0x7f7a2409e2f0 legacy=0x7f7a241378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:43.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient: _finish_hunting 0 2024-09-16T14:47:43.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 1 monclient: found mon.noname-a 2024-09-16T14:47:43.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:43.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:47:43.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 1 -- 172.21.15.189:0/2483786172 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2409c690 con 0x7f7a2409df10 2024-09-16T14:47:43.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient: _finish_auth 0 2024-09-16T14:47:43.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient: _check_auth_tickets 2024-09-16T14:47:43.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:43.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 1 -- 172.21.15.189:0/2483786172 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a14001530 con 0x7f7a2409df10 2024-09-16T14:47:43.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:43.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:43.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient: dump: 2024-09-16T14:47:43.249 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:43.249 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:43.249 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:43.249 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:43.249 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:43.249 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:43.249 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:43.249 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:43.249 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:43.249 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:43.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 5 monclient: authenticate success, global_id 124188 2024-09-16T14:47:43.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:43.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 1 -- 172.21.15.189:0/2483786172 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a140017d0 con 0x7f7a2409df10 2024-09-16T14:47:43.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:43.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:43.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 4 set_mon_vals no callback set 2024-09-16T14:47:43.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 1 -- 172.21.15.189:0/2483786172 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a14001b60 con 0x7f7a2409df10 2024-09-16T14:47:43.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:43.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:43.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a2159a640 10 monclient: dump: 2024-09-16T14:47:43.250 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:43.250 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:43.250 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:43.251 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:43.251 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:43.251 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:43.251 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:43.251 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:43.251 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:43.251 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:43.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 10 monclient: shutdown 2024-09-16T14:47:43.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:43.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 1 -- 172.21.15.189:0/2483786172 >> v1:172.21.15.26:6789/0 conn(0x7f7a2409df10 legacy=0x7f7a2409d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:43.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 1 -- 172.21.15.189:0/2483786172 shutdown_connections 2024-09-16T14:47:43.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 1 -- 172.21.15.189:0/2483786172 >> 172.21.15.189:0/2483786172 conn(0x7f7a240a3320 msgr2=0x7f7a2409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:43.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 1 -- 172.21.15.189:0/2483786172 shutdown_connections 2024-09-16T14:47:43.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 1 -- 172.21.15.189:0/2483786172 wait complete. 2024-09-16T14:47:43.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.238+0000 7f7a28ff5640 10 monclient: build_initial_monmap 2024-09-16T14:47:43.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 10 monclient: monmap: 2024-09-16T14:47:43.252 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:43.252 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:43.252 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:43.252 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:43.252 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:43.252 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:43.252 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:43.252 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:43.252 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:43.252 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:43.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 1 Processor -- start 2024-09-16T14:47:43.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 1 -- start start 2024-09-16T14:47:43.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 10 monclient: init 2024-09-16T14:47:43.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:43.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:43.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 10 monclient(hunting): picked mon.noname-c con 0x7f7a2409df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:43.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 10 monclient(hunting): picked mon.noname-a con 0x7f7a2409e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:43.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 10 monclient(hunting): picked mon.noname-b con 0x7f7a2409e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:43.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2411d1a0 con 0x7f7a2409e2f0 2024-09-16T14:47:43.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2406b380 con 0x7f7a2409e740 2024-09-16T14:47:43.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2411e780 con 0x7f7a2409df10 2024-09-16T14:47:43.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:43.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 10 monclient(hunting): authenticate will time out at 1245.360994s 2024-09-16T14:47:43.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a22d9d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f7a2409e740 legacy=0x7f7a241cd380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:43.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a2259c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7a2409df10 0x7f7a2411a8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:36292/0 (socket says 172.21.15.189:36292) 2024-09-16T14:47:43.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a2259c640 1 -- 172.21.15.189:0/529177721 learned_addr learned my addr 172.21.15.189:0/529177721 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:43.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335323533 0 0) 0x7f7a2411e780 con 0x7f7a2409df10 2024-09-16T14:47:43.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:43.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:43.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:43.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient(hunting): my global_id is 114500 2024-09-16T14:47:43.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a04003530 con 0x7f7a2409df10 2024-09-16T14:47:43.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563055314 0 0) 0x7f7a2411d1a0 con 0x7f7a2409e2f0 2024-09-16T14:47:43.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:43.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:43.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:43.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient(hunting): my global_id is 124191 2024-09-16T14:47:43.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a2411e780 con 0x7f7a2409e2f0 2024-09-16T14:47:43.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3476010530 0 0) 0x7f7a04003530 con 0x7f7a2409df10 2024-09-16T14:47:43.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a2411d1a0 con 0x7f7a2409df10 2024-09-16T14:47:43.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a14001ba0 con 0x7f7a2409df10 2024-09-16T14:47:43.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:43.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:43.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient(hunting): dump: 2024-09-16T14:47:43.258 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:43.259 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:43.259 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:43.259 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:43.259 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:43.259 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:43.259 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:43.259 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:43.259 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:43.259 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:43.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:43.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1223038206 0 0) 0x7f7a2411e780 con 0x7f7a2409e2f0 2024-09-16T14:47:43.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a04003530 con 0x7f7a2409e2f0 2024-09-16T14:47:43.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a180040d0 con 0x7f7a2409e2f0 2024-09-16T14:47:43.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:43.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:43.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient(hunting): dump: 2024-09-16T14:47:43.260 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:43.260 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:43.260 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:43.260 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:43.260 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:43.260 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:43.260 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:43.260 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:43.260 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:43.260 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:43.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 447879965 0 0) 0x7f7a2411d1a0 con 0x7f7a2409df10 2024-09-16T14:47:43.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 >> v1:172.21.15.189:6789/0 conn(0x7f7a2409e740 legacy=0x7f7a241cd380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:43.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 >> v1:172.21.15.26:6789/0 conn(0x7f7a2409e2f0 legacy=0x7f7a2411afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:43.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:43.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 monclient: found mon.c 2024-09-16T14:47:43.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:43.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:43.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2411b6c0 con 0x7f7a2409df10 2024-09-16T14:47:43.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient: _finish_auth 0 2024-09-16T14:47:43.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:43.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:43.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 5 monclient: authenticate success, global_id 114500 2024-09-16T14:47:43.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 10 monclient: _renew_subs 2024-09-16T14:47:43.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:43.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 1 -- 172.21.15.189:0/529177721 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2411cb30 con 0x7f7a2409df10 2024-09-16T14:47:43.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 10 monclient: _renew_subs 2024-09-16T14:47:43.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:43.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 1 -- 172.21.15.189:0/529177721 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a240a1280 con 0x7f7a2409df10 2024-09-16T14:47:43.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a14004b30 con 0x7f7a2409df10 2024-09-16T14:47:43.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:43.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a14004ec0 con 0x7f7a2409df10 2024-09-16T14:47:43.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:43.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a20d99640 4 set_mon_vals no callback set 2024-09-16T14:47:43.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:43.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 10 monclient: dump: 2024-09-16T14:47:43.262 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:43.262 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:43.263 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:43.263 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:43.263 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:43.263 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:43.263 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:43.263 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:43.263 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:43.263 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:43.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:43.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:43.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:43.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a28ff5640 1 -- 172.21.15.189:0/529177721 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a08005050 con 0x7f7a2409df10 2024-09-16T14:47:43.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.242+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7a140050a0 con 0x7f7a2409df10 2024-09-16T14:47:43.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.246+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(133..133 src has 73..133) v4 ==== 6989+0+0 (unknown 2876426089 0 0) 0x7f7a14043ff0 con 0x7f7a2409df10 2024-09-16T14:47:43.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.246+0000 7f7a21d9b640 10 monclient: get_auth_request con 0x7f7a04036230 auth_method 0 2024-09-16T14:47:43.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.258+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7a14014900 con 0x7f7a2409df10 2024-09-16T14:47:43.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.258+0000 7f7a12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:43.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.258+0000 7f7a12ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:43.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.422+0000 7f7a28ff5640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T14:47:43.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.422+0000 7f7a28ff5640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T14:47:43.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.422+0000 7f7a28ff5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:43.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:43.422+0000 7f7a28ff5640 1 -- 172.21.15.189:0/529177721 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush weight-set rm", "pool": "barset"} v 0) v1 -- 0x7f7a08005300 con 0x7f7a2409df10 2024-09-16T14:47:45.985 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:47:46.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 10 monclient: get_monmap_and_config 2024-09-16T14:47:46.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 10 monclient: build_initial_monmap 2024-09-16T14:47:46.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 10 monclient: monmap: 2024-09-16T14:47:46.118 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:46.118 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:46.118 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:46.118575+0000 2024-09-16T14:47:46.118 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:46.118575+0000 2024-09-16T14:47:46.119 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:46.119 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:46.119 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:46.119 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:46.119 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:46.119 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:46.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 1 Processor -- start 2024-09-16T14:47:46.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 1 -- start start 2024-09-16T14:47:46.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 10 monclient: init 2024-09-16T14:47:46.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:46.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:46.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 10 monclient(hunting): picked mon.noname-a con 0x7fc0bc09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:46.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 10 monclient(hunting): picked mon.noname-b con 0x7fc0bc132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:46.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 10 monclient(hunting): picked mon.noname-c con 0x7fc0bc138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:46.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc09d260 con 0x7fc0bc09cb80 2024-09-16T14:47:46.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc132a80 con 0x7fc0bc138ff0 2024-09-16T14:47:46.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc09cf60 con 0x7fc0bc132650 2024-09-16T14:47:46.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:46.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 10 monclient(hunting): authenticate will time out at 1290.238627s 2024-09-16T14:47:46.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0bb7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc0bc132650 0x7fc0bc1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55328/0 (socket says 172.21.15.26:55328) 2024-09-16T14:47:46.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c0e3a640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fc0bc138ff0 legacy=0x7fc0bc13b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:46.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0bbfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc0bc09cb80 0x7fc0bc0717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47996/0 (socket says 172.21.15.26:47996) 2024-09-16T14:47:46.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0bb7fe640 1 -- 172.21.15.26:0/3914576593 learned_addr learned my addr 172.21.15.26:0/3914576593 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:46.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 1 -- 172.21.15.26:0/3914576593 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096729219 0 0) 0x7fc0bc09cf60 con 0x7fc0bc132650 2024-09-16T14:47:46.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:46.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:46.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:46.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 10 monclient(hunting): my global_id is 114503 2024-09-16T14:47:46.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 1 -- 172.21.15.26:0/3914576593 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a4003530 con 0x7fc0bc132650 2024-09-16T14:47:46.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 1 -- 172.21.15.26:0/3914576593 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3886596 0 0) 0x7fc0a4003530 con 0x7fc0bc132650 2024-09-16T14:47:46.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 1 -- 172.21.15.26:0/3914576593 >> v1:172.21.15.189:6789/0 conn(0x7fc0bc138ff0 legacy=0x7fc0bc13b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:46.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 1 -- 172.21.15.26:0/3914576593 >> v1:172.21.15.26:6789/0 conn(0x7fc0bc09cb80 legacy=0x7fc0bc0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:46.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:46.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 1 monclient: found mon.noname-b 2024-09-16T14:47:46.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:46.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:47:46.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 1 -- 172.21.15.26:0/3914576593 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0bc09f2d0 con 0x7fc0bc132650 2024-09-16T14:47:46.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 10 monclient: _finish_auth 0 2024-09-16T14:47:46.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:46.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:46.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 1 -- 172.21.15.26:0/3914576593 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc0b0001660 con 0x7fc0bc132650 2024-09-16T14:47:46.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:46.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:46.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 10 monclient: dump: 2024-09-16T14:47:46.135 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:46.135 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:46.135 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:46.135 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:46.136 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:46.136 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:46.136 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:46.136 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:46.136 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:46.137 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:46.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 5 monclient: authenticate success, global_id 114503 2024-09-16T14:47:46.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:46.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 1 -- 172.21.15.26:0/3914576593 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0b00018c0 con 0x7fc0bc132650 2024-09-16T14:47:46.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:46.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 1 -- 172.21.15.26:0/3914576593 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc0b0001c50 con 0x7fc0bc132650 2024-09-16T14:47:46.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:46.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0c289d640 4 set_mon_vals no callback set 2024-09-16T14:47:46.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:46.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:46.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.115+0000 7fc0baffd640 10 monclient: dump: 2024-09-16T14:47:46.139 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:46.139 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:46.139 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:46.139 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:46.139 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:46.139 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:46.140 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:46.140 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:46.140 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:46.140 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:46.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient: shutdown 2024-09-16T14:47:46.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:46.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 1 -- 172.21.15.26:0/3914576593 >> v1:172.21.15.26:6790/0 conn(0x7fc0bc132650 legacy=0x7fc0bc1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:46.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 1 -- 172.21.15.26:0/3914576593 shutdown_connections 2024-09-16T14:47:46.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 1 -- 172.21.15.26:0/3914576593 >> 172.21.15.26:0/3914576593 conn(0x7fc0bc09fb50 msgr2=0x7fc0bc09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:46.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 1 -- 172.21.15.26:0/3914576593 shutdown_connections 2024-09-16T14:47:46.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 1 -- 172.21.15.26:0/3914576593 wait complete. 2024-09-16T14:47:46.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient: build_initial_monmap 2024-09-16T14:47:46.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient: monmap: 2024-09-16T14:47:46.142 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:46.142 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:46.142 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:46.143 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:46.143 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:46.143 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:46.143 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:46.143 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:46.143 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:46.144 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:46.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 1 Processor -- start 2024-09-16T14:47:46.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 1 -- start start 2024-09-16T14:47:46.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient: init 2024-09-16T14:47:46.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:46.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient(hunting): picked mon.noname-b con 0x7fc0bc09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient(hunting): picked mon.noname-a con 0x7fc0bc132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient(hunting): picked mon.noname-c con 0x7fc0bc138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc13bf50 con 0x7fc0bc132650 2024-09-16T14:47:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc131e30 con 0x7fc0bc09cb80 2024-09-16T14:47:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc13c590 con 0x7fc0bc138ff0 2024-09-16T14:47:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient(hunting): authenticate will time out at 1290.241665s 2024-09-16T14:47:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0bbfff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fc0bc09cb80 legacy=0x7fc0bc139f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0bb7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc0bc132650 0x7fc0bc13a630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48006/0 (socket says 172.21.15.26:48006) 2024-09-16T14:47:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0bb7fe640 1 -- 172.21.15.26:0/2923972642 learned_addr learned my addr 172.21.15.26:0/2923972642 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934272216 0 0) 0x7fc0bc13c590 con 0x7fc0bc138ff0 2024-09-16T14:47:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:47:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient(hunting): my global_id is 114506 2024-09-16T14:47:46.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a8003530 con 0x7fc0bc138ff0 2024-09-16T14:47:46.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2175203809 0 0) 0x7fc0a8003530 con 0x7fc0bc138ff0 2024-09-16T14:47:46.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0bc13c590 con 0x7fc0bc138ff0 2024-09-16T14:47:46.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc0b4004040 con 0x7fc0bc138ff0 2024-09-16T14:47:46.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient(hunting): dump: 2024-09-16T14:47:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:46.151 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1791214401 0 0) 0x7fc0bc13c590 con 0x7fc0bc138ff0 2024-09-16T14:47:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 >> v1:172.21.15.189:6789/0 conn(0x7fc0bc09cb80 legacy=0x7fc0bc139f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 >> v1:172.21.15.26:6789/0 conn(0x7fc0bc132650 legacy=0x7fc0bc13a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:47:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 monclient: found mon.c 2024-09-16T14:47:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0bc09a240 con 0x7fc0bc138ff0 2024-09-16T14:47:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient: _finish_auth 0 2024-09-16T14:47:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:47:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 5 monclient: authenticate success, global_id 114506 2024-09-16T14:47:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient: _renew_subs 2024-09-16T14:47:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 1 -- 172.21.15.26:0/2923972642 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0bc1342d0 con 0x7fc0bc138ff0 2024-09-16T14:47:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient: _renew_subs 2024-09-16T14:47:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 1 -- 172.21.15.26:0/2923972642 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0bc134ea0 con 0x7fc0bc138ff0 2024-09-16T14:47:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0b4004260 con 0x7fc0bc138ff0 2024-09-16T14:47:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc0b4004570 con 0x7fc0bc138ff0 2024-09-16T14:47:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0ba7fc640 4 set_mon_vals no callback set 2024-09-16T14:47:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient: dump: 2024-09-16T14:47:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:46.157 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc0b4004790 con 0x7fc0bc138ff0 2024-09-16T14:47:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c289d640 1 -- 172.21.15.26:0/2923972642 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09c005050 con 0x7fc0bc138ff0 2024-09-16T14:47:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c0e3a640 1 -- 172.21.15.26:0/2923972642 >> v1:172.21.15.26:6790/0 conn(0x7fc0bc138ff0 legacy=0x7fc0bc13ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T14:47:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c0e3a640 1 -- 172.21.15.26:0/2923972642 >> v1:172.21.15.26:6790/0 conn(0x7fc0bc138ff0 legacy=0x7fc0bc13ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:47:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c0e3a640 1 --1- 172.21.15.26:0/2923972642 >> v1:172.21.15.26:6790/0 conn(0x7fc0bc138ff0 0x7fc0bc13ad40 :-1 s=OPENED pgs=715 cs=1 l=1).handle_message read tag failed 2024-09-16T14:47:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0c0e3a640 1 --1- 172.21.15.26:0/2923972642 >> v1:172.21.15.26:6790/0 conn(0x7fc0bc138ff0 0x7fc0bc13ad40 :-1 s=OPENED pgs=715 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:47:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient: ms_handle_reset current mon v1:172.21.15.26:6790/0 2024-09-16T14:47:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 >> v1:172.21.15.26:6790/0 conn(0x7fc0bc138ff0 legacy=0x7fc0bc13ad40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:46.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient(hunting): picked mon.b con 0x7fc0bc09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:46.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient(hunting): picked mon.a con 0x7fc0bc132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:46.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient(hunting): picked mon.c con 0x7fc0a8041350 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:46.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc0bc13c590 con 0x7fc0bc132650 2024-09-16T14:47:46.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc0bc131e30 con 0x7fc0bc09cb80 2024-09-16T14:47:46.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc0a8003530 con 0x7fc0a8041350 2024-09-16T14:47:46.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0b8ff9640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:46.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.119+0000 7fc0bb7fe640 1 -- 172.21.15.26:0/2923972642 >> v1:172.21.15.189:6789/0 conn(0x7fc0bc09cb80 legacy=0x7fc0a803bcc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:46.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.123+0000 7fc0bbfff640 10 monclient(hunting): get_auth_request con 0x7fc0a8035fe0 auth_method 0 2024-09-16T14:47:46.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319354723 0 0) 0x7fc0a8003530 con 0x7fc0a8041350 2024-09-16T14:47:46.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:47:46.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:46.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:47:46.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 --> v1:172.21.15.26:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fc0a80474a0 con 0x7fc0a8041350 2024-09-16T14:47:46.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 3137218980 0 0) 0x7fc0a80474a0 con 0x7fc0a8041350 2024-09-16T14:47:46.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 >> v1:172.21.15.189:6789/0 conn(0x7fc0bc09cb80 legacy=0x7fc0a803bcc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:46.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 >> v1:172.21.15.26:6789/0 conn(0x7fc0bc132650 legacy=0x7fc0a803f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:46.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:47:46.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 1 monclient: found mon.c 2024-09-16T14:47:46.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:46.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:46.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fc0a80484d0 con 0x7fc0a8041350 2024-09-16T14:47:46.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:46.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:46.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0bc131e30 con 0x7fc0a8041350 2024-09-16T14:47:46.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 10 monclient: _finish_auth 0 2024-09-16T14:47:46.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:47:46.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:46.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc0ac001dd0 con 0x7fc0a8041350 2024-09-16T14:47:46.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:46.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:46.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 10 monclient: dump: 2024-09-16T14:47:46.163 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:46.163 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:46.163 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:46.163 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:46.164 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:46.164 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:46.164 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:46.164 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:46.164 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:46.164 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:46.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0ac001c30 con 0x7fc0a8041350 2024-09-16T14:47:46.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:46.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b9ffb640 4 set_mon_vals no callback set 2024-09-16T14:47:46.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc0ac005ba0 con 0x7fc0a8041350 2024-09-16T14:47:46.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.127+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 <== mon.2 v1:172.21.15.26:6790/0 6 ==== osd_map(133..133 src has 73..133) v4 ==== 6989+0+0 (unknown 2876426089 0 0) 0x7fc0ac045b30 con 0x7fc0a8041350 2024-09-16T14:47:46.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.131+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc0ac002000 con 0x7fc0a8041350 2024-09-16T14:47:46.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.131+0000 7fc0b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:46.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.131+0000 7fc0b8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:46.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:46.246+0000 7f7a11ffb640 10 monclient: tick 2024-09-16T14:47:46.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:46.246+0000 7f7a11ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:46.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:46.246+0000 7f7a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:46.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:46.246+0000 7f7a11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:46.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.343+0000 7fc0c289d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:47:46.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.343+0000 7fc0c289d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:46.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.343+0000 7fc0c289d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:46.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.343+0000 7fc0c289d640 1 -- 172.21.15.26:0/2923972642 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc09c005300 con 0x7fc0a8041350 2024-09-16T14:47:46.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.343+0000 7fc0b8ff9640 1 -- 172.21.15.26:0/2923972642 <== mon.2 v1:172.21.15.26:6790/0 8 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2062130409) 0x7fc0ac002000 con 0x7fc0a8041350 2024-09-16T14:47:46.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.343+0000 7fc0b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:46.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.343+0000 7fc0b8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:46.349 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:47:46.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.347+0000 7fc0c289d640 1 -- 172.21.15.26:0/2923972642 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc0a8035fe0 legacy=0x7fc0a80384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:46.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.347+0000 7fc0c289d640 10 monclient: shutdown 2024-09-16T14:47:46.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.347+0000 7fc0c289d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:46.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.347+0000 7fc0c289d640 1 -- 172.21.15.26:0/2923972642 >> v1:172.21.15.26:6790/0 conn(0x7fc0a8041350 legacy=0x7fc0a80437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:46.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.347+0000 7fc0c289d640 1 -- 172.21.15.26:0/2923972642 shutdown_connections 2024-09-16T14:47:46.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.347+0000 7fc0c289d640 1 -- 172.21.15.26:0/2923972642 >> 172.21.15.26:0/2923972642 conn(0x7fc0bc09fb50 msgr2=0x7fc0bc1388c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:46.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.347+0000 7fc0c289d640 1 -- 172.21.15.26:0/2923972642 shutdown_connections 2024-09-16T14:47:46.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:46.347+0000 7fc0c289d640 1 -- 172.21.15.26:0/2923972642 wait complete. 2024-09-16T14:47:47.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:47.242+0000 7f7a11ffb640 10 monclient: tick 2024-09-16T14:47:47.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:47.242+0000 7f7a11ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:47.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:47.242+0000 7f7a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:48.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:48.242+0000 7f7a11ffb640 10 monclient: tick 2024-09-16T14:47:48.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:48.242+0000 7f7a11ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:48.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:48.242+0000 7f7a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:49.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:49.242+0000 7f7a11ffb640 10 monclient: tick 2024-09-16T14:47:49.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:49.242+0000 7f7a11ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:49.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:49.242+0000 7f7a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:49.367 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:47:49.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe00da64640 10 monclient: get_monmap_and_config 2024-09-16T14:47:49.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe00da64640 10 monclient: build_initial_monmap 2024-09-16T14:47:49.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe00da64640 10 monclient: monmap: 2024-09-16T14:47:49.502 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:49.502 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:49.502 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:49.502146+0000 2024-09-16T14:47:49.502 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:49.502146+0000 2024-09-16T14:47:49.503 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:49.503 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:49.503 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:49.503 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:49.503 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:49.503 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:49.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe00da64640 1 Processor -- start 2024-09-16T14:47:49.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe00da64640 1 -- start start 2024-09-16T14:47:49.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe00da64640 10 monclient: init 2024-09-16T14:47:49.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe00da64640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:49.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe00da64640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:49.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe00da64640 10 monclient(hunting): picked mon.noname-b con 0x7fe00809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:49.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe00da64640 10 monclient(hunting): picked mon.noname-c con 0x7fe008132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:49.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe00da64640 10 monclient(hunting): picked mon.noname-a con 0x7fe008138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:49.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe00da64640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00809d260 con 0x7fe008138ff0 2024-09-16T14:47:49.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe00da64640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe008132a80 con 0x7fe008132650 2024-09-16T14:47:49.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe00da64640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00809cf60 con 0x7fe00809cb80 2024-09-16T14:47:49.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe00da64640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:49.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe00da64640 10 monclient(hunting): authenticate will time out at 1293.622339s 2024-09-16T14:47:49.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe006ffd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fe008132650 legacy=0x7fe0081378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:49.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe0077fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe00809cb80 0x7fe0080717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:45538/0 (socket says 172.21.15.26:45538) 2024-09-16T14:47:49.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.499+0000 7fe0077fe640 1 -- 172.21.15.26:0/28432559 learned_addr learned my addr 172.21.15.26:0/28432559 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:49.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:49.703+0000 7fe006ffd640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.189:6789/0 conn(0x7fe008132650 legacy=0x7fe0081378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:50.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:50.107+0000 7fe006ffd640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.189:6789/0 conn(0x7fe008132650 legacy=0x7fe0081378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:50.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:50.246+0000 7f7a11ffb640 10 monclient: tick 2024-09-16T14:47:50.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:50.246+0000 7f7a11ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:50.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:50.246+0000 7f7a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:50.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:50.911+0000 7fe006ffd640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.189:6789/0 conn(0x7fe008132650 legacy=0x7fe0081378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:51.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:51.246+0000 7f7a11ffb640 10 monclient: tick 2024-09-16T14:47:51.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:51.246+0000 7f7a11ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:51.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:51.246+0000 7f7a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:52.242+0000 7f7a11ffb640 10 monclient: tick 2024-09-16T14:47:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:52.242+0000 7f7a11ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:52.242+0000 7f7a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:52.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.499+0000 7fe0057fa640 10 monclient(hunting): tick 2024-09-16T14:47:52.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.499+0000 7fe0057fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:47:52.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.499+0000 7fe0057fa640 1 monclient(hunting): continuing hunt 2024-09-16T14:47:52.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.499+0000 7fe0057fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:47:52.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.499+0000 7fe0057fa640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.26:6790/0 conn(0x7fe00809cb80 legacy=0x7fe0080717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:52.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.499+0000 7fe0057fa640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.189:6789/0 conn(0x7fe008132650 legacy=0x7fe0081378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:52.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.499+0000 7fe0057fa640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.26:6789/0 conn(0x7fe008138ff0 legacy=0x7fe00813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:52.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.499+0000 7fe0057fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:52.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.499+0000 7fe0057fa640 10 monclient(hunting): picked mon.noname-a con 0x7fdfe80036d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:52.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.499+0000 7fe0057fa640 10 monclient(hunting): picked mon.noname-b con 0x7fe00809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:52.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.499+0000 7fe0057fa640 10 monclient(hunting): picked mon.noname-c con 0x7fe008132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:52.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.499+0000 7fe0057fa640 1 -- 172.21.15.26:0/28432559 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00809f2d0 con 0x7fdfe80036d0 2024-09-16T14:47:52.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.499+0000 7fe0057fa640 1 -- 172.21.15.26:0/28432559 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe008132a80 con 0x7fe008132650 2024-09-16T14:47:52.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.499+0000 7fe0057fa640 1 -- 172.21.15.26:0/28432559 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe80103b0 con 0x7fe00809cb80 2024-09-16T14:47:52.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.499+0000 7fe0057fa640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:52.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.499+0000 7fe007fff640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.189:6789/0 conn(0x7fe008132650 legacy=0x7fdfe800cbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:52.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:52.703+0000 7fe007fff640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.189:6789/0 conn(0x7fe008132650 legacy=0x7fdfe800cbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:53.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:53.107+0000 7fe007fff640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.189:6789/0 conn(0x7fe008132650 legacy=0x7fdfe800cbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:53.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:53.242+0000 7f7a11ffb640 10 monclient: tick 2024-09-16T14:47:53.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:53.242+0000 7f7a11ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:53.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:53.242+0000 7f7a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:53.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:53.911+0000 7fe007fff640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.189:6789/0 conn(0x7fe008132650 legacy=0x7fdfe800cbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:54.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:54.242+0000 7f7a11ffb640 10 monclient: tick 2024-09-16T14:47:54.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:54.242+0000 7f7a11ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:54.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:54.242+0000 7f7a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:55.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:55.242+0000 7f7a11ffb640 10 monclient: tick 2024-09-16T14:47:55.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:55.242+0000 7f7a11ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:55.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:55.242+0000 7f7a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.499+0000 7fe0057fa640 10 monclient(hunting): tick 2024-09-16T14:47:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.499+0000 7fe0057fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:47:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.499+0000 7fe0057fa640 1 monclient(hunting): continuing hunt 2024-09-16T14:47:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.499+0000 7fe0057fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:47:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.499+0000 7fe0057fa640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.26:6790/0 conn(0x7fe00809cb80 legacy=0x7fdfe80093d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.499+0000 7fe0057fa640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.189:6789/0 conn(0x7fe008132650 legacy=0x7fdfe800cbd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.499+0000 7fe0057fa640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.26:6789/0 conn(0x7fdfe80036d0 legacy=0x7fdfe8005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.499+0000 7fe0057fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.499+0000 7fe0057fa640 10 monclient(hunting): picked mon.noname-c con 0x7fdfe80106f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.499+0000 7fe0057fa640 10 monclient(hunting): picked mon.noname-a con 0x7fdfe800d300 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.499+0000 7fe0057fa640 10 monclient(hunting): picked mon.noname-b con 0x7fe00809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.499+0000 7fe0057fa640 1 -- 172.21.15.26:0/28432559 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe8011620 con 0x7fdfe800d300 2024-09-16T14:47:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.499+0000 7fe0057fa640 1 -- 172.21.15.26:0/28432559 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe008132a80 con 0x7fdfe80106f0 2024-09-16T14:47:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.499+0000 7fe0077fe640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.189:6789/0 conn(0x7fdfe80106f0 legacy=0x7fdfe8010b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.499+0000 7fe0057fa640 1 -- 172.21.15.26:0/28432559 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe800d760 con 0x7fe00809cb80 2024-09-16T14:47:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.499+0000 7fe0057fa640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:55.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:55.703+0000 7fe0077fe640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.189:6789/0 conn(0x7fdfe80106f0 legacy=0x7fdfe8010b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:56.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:56.107+0000 7fe0077fe640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.189:6789/0 conn(0x7fdfe80106f0 legacy=0x7fdfe8010b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:56.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:56.242+0000 7f7a11ffb640 10 monclient: tick 2024-09-16T14:47:56.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:56.242+0000 7f7a11ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:56.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:56.242+0000 7f7a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:56.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:56.242+0000 7f7a11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:56.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:56.911+0000 7fe0077fe640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.189:6789/0 conn(0x7fdfe80106f0 legacy=0x7fdfe8010b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:57.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:57.242+0000 7f7a11ffb640 10 monclient: tick 2024-09-16T14:47:57.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:57.242+0000 7f7a11ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:57.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:57.242+0000 7f7a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:57.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 1 -- 172.21.15.26:0/28432559 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492423226 0 0) 0x7fdfe8011620 con 0x7fdfe800d300 2024-09-16T14:47:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 10 monclient(hunting): my global_id is 124194 2024-09-16T14:47:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 1 -- 172.21.15.26:0/28432559 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdff4003530 con 0x7fdfe800d300 2024-09-16T14:47:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 1 -- 172.21.15.26:0/28432559 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 490268981 0 0) 0x7fdff4003530 con 0x7fdfe800d300 2024-09-16T14:47:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.26:6790/0 conn(0x7fe00809cb80 legacy=0x7fdfe80093d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.189:6789/0 conn(0x7fdfe80106f0 legacy=0x7fdfe8010b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 1 monclient: found mon.noname-a 2024-09-16T14:47:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:47:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 1 -- 172.21.15.26:0/28432559 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe8005b30 con 0x7fdfe800d300 2024-09-16T14:47:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 1 -- 172.21.15.26:0/28432559 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdff8004160 con 0x7fdfe800d300 2024-09-16T14:47:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe00da64640 5 monclient: authenticate success, global_id 124194 2024-09-16T14:47:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe00da64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 10 monclient: dump: 2024-09-16T14:47:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.309 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe00da64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 1 -- 172.21.15.26:0/28432559 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdff8004400 con 0x7fdfe800d300 2024-09-16T14:47:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe00da64640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 1 -- 172.21.15.26:0/28432559 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdff80047d0 con 0x7fdfe800d300 2024-09-16T14:47:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe00da64640 4 set_mon_vals no callback set 2024-09-16T14:47:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe0067fc640 10 monclient: dump: 2024-09-16T14:47:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.311 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe00da64640 10 monclient: shutdown 2024-09-16T14:47:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe00da64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe00da64640 1 -- 172.21.15.26:0/28432559 >> v1:172.21.15.26:6789/0 conn(0x7fdfe800d300 legacy=0x7fdfe800cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe00da64640 1 -- 172.21.15.26:0/28432559 shutdown_connections 2024-09-16T14:47:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe00da64640 1 -- 172.21.15.26:0/28432559 >> 172.21.15.26:0/28432559 conn(0x7fe00809fb50 msgr2=0x7fe00809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe00da64640 1 -- 172.21.15.26:0/28432559 shutdown_connections 2024-09-16T14:47:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.303+0000 7fe00da64640 1 -- 172.21.15.26:0/28432559 wait complete. 2024-09-16T14:47:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe00da64640 10 monclient: build_initial_monmap 2024-09-16T14:47:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe00da64640 10 monclient: monmap: 2024-09-16T14:47:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:57.317 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe00da64640 1 Processor -- start 2024-09-16T14:47:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe00da64640 1 -- start start 2024-09-16T14:47:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe00da64640 10 monclient: init 2024-09-16T14:47:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe00da64640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe00da64640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe00da64640 10 monclient(hunting): picked mon.noname-b con 0x7fdfe800d300 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe00da64640 10 monclient(hunting): picked mon.noname-c con 0x7fdfe80106f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe00da64640 10 monclient(hunting): picked mon.noname-a con 0x7fe00809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe0077fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fdfe800d300 legacy=0x7fe0081c71e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:57.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe00da64640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe008135e20 con 0x7fe00809cb80 2024-09-16T14:47:57.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe00da64640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0081c8000 con 0x7fdfe800d300 2024-09-16T14:47:57.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe007fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe00809cb80 0x7fe0081cf520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42660/0 (socket says 172.21.15.26:42660) 2024-09-16T14:47:57.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe007fff640 1 -- 172.21.15.26:0/2466210963 learned_addr learned my addr 172.21.15.26:0/2466210963 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:57.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe00da64640 1 -- 172.21.15.26:0/2466210963 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00809a240 con 0x7fdfe80106f0 2024-09-16T14:47:57.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe00da64640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:57.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.307+0000 7fe00da64640 10 monclient(hunting): authenticate will time out at 1301.431720s 2024-09-16T14:47:57.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 1 -- 172.21.15.26:0/2466210963 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182883000 0 0) 0x7fe008135e20 con 0x7fe00809cb80 2024-09-16T14:47:57.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:57.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:57.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:57.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 10 monclient(hunting): my global_id is 124197 2024-09-16T14:47:57.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 1 -- 172.21.15.26:0/2466210963 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe80115c0 con 0x7fe00809cb80 2024-09-16T14:47:57.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 1 -- 172.21.15.26:0/2466210963 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3965533745 0 0) 0x7fdfe80115c0 con 0x7fe00809cb80 2024-09-16T14:47:57.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 1 -- 172.21.15.26:0/2466210963 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe008135e20 con 0x7fe00809cb80 2024-09-16T14:47:57.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 1 -- 172.21.15.26:0/2466210963 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe000001b20 con 0x7fe00809cb80 2024-09-16T14:47:57.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:57.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 10 monclient(hunting): dump: 2024-09-16T14:47:57.319 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.319 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.319 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.320 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.320 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.320 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.320 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.320 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.320 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.320 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:57.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 1 -- 172.21.15.26:0/2466210963 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 323659195 0 0) 0x7fe008135e20 con 0x7fe00809cb80 2024-09-16T14:47:57.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 1 -- 172.21.15.26:0/2466210963 >> v1:172.21.15.26:6790/0 conn(0x7fdfe80106f0 legacy=0x7fe0081c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 1 -- 172.21.15.26:0/2466210963 >> v1:172.21.15.189:6789/0 conn(0x7fdfe800d300 legacy=0x7fe0081c71e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:57.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 10 monclient: _finish_hunting 0 2024-09-16T14:47:57.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 1 monclient: found mon.a 2024-09-16T14:47:57.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:57.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:57.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 1 -- 172.21.15.26:0/2466210963 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0080a2470 con 0x7fe00809cb80 2024-09-16T14:47:57.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 10 monclient: _finish_auth 0 2024-09-16T14:47:57.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 10 monclient: _check_auth_tickets 2024-09-16T14:47:57.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:57.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe00da64640 5 monclient: authenticate success, global_id 124197 2024-09-16T14:47:57.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe00da64640 10 monclient: _renew_subs 2024-09-16T14:47:57.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe00da64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:57.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe00da64640 1 -- 172.21.15.26:0/2466210963 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe008138b40 con 0x7fe00809cb80 2024-09-16T14:47:57.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe00da64640 10 monclient: _renew_subs 2024-09-16T14:47:57.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe00da64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:57.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe00da64640 1 -- 172.21.15.26:0/2466210963 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe008071490 con 0x7fe00809cb80 2024-09-16T14:47:57.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 1 -- 172.21.15.26:0/2466210963 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe000003980 con 0x7fe00809cb80 2024-09-16T14:47:57.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:57.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 1 -- 172.21.15.26:0/2466210963 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe000003d10 con 0x7fe00809cb80 2024-09-16T14:47:57.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:57.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 10 monclient: dump: 2024-09-16T14:47:57.322 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.323 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.323 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.323 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.323 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.323 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.323 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.323 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.323 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.323 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 1 -- 172.21.15.26:0/2466210963 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe000003e70 con 0x7fe00809cb80 2024-09-16T14:47:57.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe0057fa640 4 set_mon_vals no callback set 2024-09-16T14:47:57.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe00da64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:57.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 1 -- 172.21.15.26:0/2466210963 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(133..133 src has 73..133) v4 ==== 6989+0+0 (unknown 2876426089 0 0) 0x7fe0000440e0 con 0x7fe00809cb80 2024-09-16T14:47:57.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe00da64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:57.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe00da64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:57.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe00da64640 1 -- 172.21.15.26:0/2466210963 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe008136060 con 0x7fe00809cb80 2024-09-16T14:47:57.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe0077fe640 10 monclient: get_auth_request con 0x7fdfe80376f0 auth_method 0 2024-09-16T14:47:57.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe0077fe640 1 -- 172.21.15.26:0/2466210963 >> v1:172.21.15.189:6812/3196161582 conn(0x7fdfe80376f0 legacy=0x7fdfe8039bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T14:47:57.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe0077fe640 1 -- 172.21.15.26:0/2466210963 >> v1:172.21.15.189:6812/3196161582 conn(0x7fdfe80376f0 legacy=0x7fdfe8039bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:47:57.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe0077fe640 1 --1- 172.21.15.26:0/2466210963 >> v1:172.21.15.189:6812/3196161582 conn(0x7fdfe80376f0 0x7fdfe8039bb0 :-1 s=OPENED pgs=331 cs=1 l=1).handle_message read tag failed 2024-09-16T14:47:57.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fe0077fe640 1 --1- 172.21.15.26:0/2466210963 >> v1:172.21.15.189:6812/3196161582 conn(0x7fdfe80376f0 0x7fdfe8039bb0 :-1 s=OPENED pgs=331 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:47:57.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.311+0000 7fdffffff640 1 -- 172.21.15.26:0/2466210963 >> v1:172.21.15.189:6812/3196161582 conn(0x7fdfe80376f0 legacy=0x7fdfe8039bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:57.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.315+0000 7fdffffff640 1 -- 172.21.15.26:0/2466210963 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe000014a60 con 0x7fe00809cb80 2024-09-16T14:47:57.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.315+0000 7fdffffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:57.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.315+0000 7fdffffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:57.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.543+0000 7fe00da64640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:47:57.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.543+0000 7fe00da64640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:57.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.543+0000 7fe00da64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:57.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.543+0000 7fe00da64640 1 -- 172.21.15.26:0/2466210963 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe00809c960 con 0x7fe00809cb80 2024-09-16T14:47:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.543+0000 7fdffffff640 1 -- 172.21.15.26:0/2466210963 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2335801557) 0x7fe000012ed0 con 0x7fe00809cb80 2024-09-16T14:47:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.543+0000 7fdffffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:47:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.543+0000 7fdffffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:57.549 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":52,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:47:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.547+0000 7fe00da64640 10 monclient: shutdown 2024-09-16T14:47:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.547+0000 7fe00da64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.547+0000 7fe00da64640 1 -- 172.21.15.26:0/2466210963 >> v1:172.21.15.26:6789/0 conn(0x7fe00809cb80 legacy=0x7fe0081cf520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.547+0000 7fe00da64640 1 -- 172.21.15.26:0/2466210963 shutdown_connections 2024-09-16T14:47:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.547+0000 7fe00da64640 1 -- 172.21.15.26:0/2466210963 >> 172.21.15.26:0/2466210963 conn(0x7fe00809fb50 msgr2=0x7fe0081315f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.547+0000 7fe00da64640 1 -- 172.21.15.26:0/2466210963 shutdown_connections 2024-09-16T14:47:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.547+0000 7fe00da64640 1 -- 172.21.15.26:0/2466210963 wait complete. 2024-09-16T14:47:57.567 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:47:57.567 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T14:47:57.567 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:47:57.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 10 monclient: get_monmap_and_config 2024-09-16T14:47:57.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 10 monclient: build_initial_monmap 2024-09-16T14:47:57.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 10 monclient: monmap: 2024-09-16T14:47:57.694 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:57.694 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.694 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:57.694171+0000 2024-09-16T14:47:57.694 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:57.694171+0000 2024-09-16T14:47:57.694 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:57.694 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.694 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:57.694 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:57.694 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:57.694 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 1 Processor -- start 2024-09-16T14:47:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 1 -- start start 2024-09-16T14:47:57.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 10 monclient: init 2024-09-16T14:47:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:47:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 10 monclient(hunting): picked mon.noname-c con 0x7f4cc80a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 10 monclient(hunting): picked mon.noname-b con 0x7f4cc80a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 10 monclient(hunting): picked mon.noname-a con 0x7f4cc80a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc80a3250 con 0x7f4cc80a2800 2024-09-16T14:47:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc809f600 con 0x7f4cc80a1d10 2024-09-16T14:47:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc80a3830 con 0x7f4cc80a0000 2024-09-16T14:47:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 10 monclient(hunting): authenticate will time out at 1301.814994s 2024-09-16T14:47:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4cce173640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4cc80a2800 0x7f4cc813d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42670/0 (socket says 172.21.15.26:42670) 2024-09-16T14:47:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccd171640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4cc80a0000 0x7f4cc8139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40890/0 (socket says 172.21.15.26:40890) 2024-09-16T14:47:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4cce173640 1 -- 172.21.15.26:0/848687297 learned_addr learned my addr 172.21.15.26:0/848687297 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccd972640 1 -- 172.21.15.26:0/848687297 >> v1:172.21.15.189:6789/0 conn(0x7f4cc80a1d10 legacy=0x7f4cc80a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 1 -- 172.21.15.26:0/848687297 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544771361 0 0) 0x7f4cc80a3250 con 0x7f4cc80a2800 2024-09-16T14:47:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient(hunting): _negotiate 2024-09-16T14:47:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient(hunting): my global_id is 124200 2024-09-16T14:47:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 1 -- 172.21.15.26:0/848687297 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb4003530 con 0x7f4cc80a2800 2024-09-16T14:47:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 1 -- 172.21.15.26:0/848687297 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029563654 0 0) 0x7f4cc80a3830 con 0x7f4cc80a0000 2024-09-16T14:47:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient(hunting): _negotiate 2024-09-16T14:47:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient(hunting): my global_id is 114509 2024-09-16T14:47:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 1 -- 172.21.15.26:0/848687297 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cc80a3250 con 0x7f4cc80a0000 2024-09-16T14:47:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 1 -- 172.21.15.26:0/848687297 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 52788550 0 0) 0x7f4cb4003530 con 0x7f4cc80a2800 2024-09-16T14:47:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 1 -- 172.21.15.26:0/848687297 >> v1:172.21.15.26:6790/0 conn(0x7f4cc80a0000 legacy=0x7f4cc8139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 1 -- 172.21.15.26:0/848687297 >> v1:172.21.15.189:6789/0 conn(0x7f4cc80a1d10 legacy=0x7f4cc80a20f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient: _finish_hunting 0 2024-09-16T14:47:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 1 monclient: found mon.noname-a 2024-09-16T14:47:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:47:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 1 -- 172.21.15.26:0/848687297 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc809ed60 con 0x7f4cc80a2800 2024-09-16T14:47:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient: _finish_auth 0 2024-09-16T14:47:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient: _check_auth_tickets 2024-09-16T14:47:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 5 monclient: authenticate success, global_id 124200 2024-09-16T14:47:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 1 -- 172.21.15.26:0/848687297 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4cc00018d0 con 0x7f4cc80a2800 2024-09-16T14:47:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient: dump: 2024-09-16T14:47:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.701 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 1 -- 172.21.15.26:0/848687297 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cc0001bb0 con 0x7f4cc80a2800 2024-09-16T14:47:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 1 -- 172.21.15.26:0/848687297 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4cc0001f40 con 0x7f4cc80a2800 2024-09-16T14:47:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccc970640 10 monclient: dump: 2024-09-16T14:47:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.702 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.691+0000 7f4ccfbd6640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 4 set_mon_vals no callback set 2024-09-16T14:47:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 10 monclient: shutdown 2024-09-16T14:47:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 1 -- 172.21.15.26:0/848687297 >> v1:172.21.15.26:6789/0 conn(0x7f4cc80a2800 legacy=0x7f4cc813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 1 -- 172.21.15.26:0/848687297 shutdown_connections 2024-09-16T14:47:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 1 -- 172.21.15.26:0/848687297 >> 172.21.15.26:0/848687297 conn(0x7f4cc80a7f30 msgr2=0x7f4cc80a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 1 -- 172.21.15.26:0/848687297 shutdown_connections 2024-09-16T14:47:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 1 -- 172.21.15.26:0/848687297 wait complete. 2024-09-16T14:47:57.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 10 monclient: build_initial_monmap 2024-09-16T14:47:57.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 10 monclient: monmap: 2024-09-16T14:47:57.707 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:57.707 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:57.708 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:57.708 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:57.708 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:57.708 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:57.708 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:57.708 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:57.708 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:57.708 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 1 Processor -- start 2024-09-16T14:47:57.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 1 -- start start 2024-09-16T14:47:57.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 10 monclient: init 2024-09-16T14:47:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:47:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 10 monclient(hunting): picked mon.noname-b con 0x7f4cc80a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 10 monclient(hunting): picked mon.noname-a con 0x7f4cc80a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 10 monclient(hunting): picked mon.noname-c con 0x7f4cc80a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc8133950 con 0x7f4cc80a1d10 2024-09-16T14:47:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc80a6740 con 0x7f4cc80a0000 2024-09-16T14:47:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc809e500 con 0x7f4cc80a2800 2024-09-16T14:47:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 10 monclient(hunting): authenticate will time out at 1301.818135s 2024-09-16T14:47:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccd171640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4cc80a1d10 0x7f4cc81cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42678/0 (socket says 172.21.15.26:42678) 2024-09-16T14:47:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccd171640 1 -- 172.21.15.26:0/3681839768 learned_addr learned my addr 172.21.15.26:0/3681839768 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccd972640 1 -- 172.21.15.26:0/3681839768 >> v1:172.21.15.189:6789/0 conn(0x7f4cc80a0000 legacy=0x7f4cc8134330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835034157 0 0) 0x7f4cc8133950 con 0x7f4cc80a1d10 2024-09-16T14:47:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient(hunting): my global_id is 124203 2024-09-16T14:47:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cac003530 con 0x7f4cc80a1d10 2024-09-16T14:47:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529174968 0 0) 0x7f4cc809e500 con 0x7f4cc80a2800 2024-09-16T14:47:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient(hunting): my global_id is 114512 2024-09-16T14:47:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cc8133950 con 0x7f4cc80a2800 2024-09-16T14:47:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1673492592 0 0) 0x7f4cc8133950 con 0x7f4cc80a2800 2024-09-16T14:47:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cc809e500 con 0x7f4cc80a2800 2024-09-16T14:47:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 606478017 0 0) 0x7f4cac003530 con 0x7f4cc80a1d10 2024-09-16T14:47:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cc8133950 con 0x7f4cc80a1d10 2024-09-16T14:47:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4cc0001ec0 con 0x7f4cc80a2800 2024-09-16T14:47:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient(hunting): dump: 2024-09-16T14:47:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.712 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4cb8001720 con 0x7f4cc80a1d10 2024-09-16T14:47:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient(hunting): dump: 2024-09-16T14:47:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.713 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 820855426 0 0) 0x7f4cc809e500 con 0x7f4cc80a2800 2024-09-16T14:47:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 >> v1:172.21.15.189:6789/0 conn(0x7f4cc80a0000 legacy=0x7f4cc8134330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 >> v1:172.21.15.26:6789/0 conn(0x7f4cc80a1d10 legacy=0x7f4cc81cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 monclient: found mon.c 2024-09-16T14:47:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc8133670 con 0x7f4cc80a2800 2024-09-16T14:47:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 5 monclient: authenticate success, global_id 114512 2024-09-16T14:47:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 10 monclient: _renew_subs 2024-09-16T14:47:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 1 -- 172.21.15.26:0/3681839768 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc8132930 con 0x7f4cc80a2800 2024-09-16T14:47:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 10 monclient: _renew_subs 2024-09-16T14:47:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4ccfbd6640 1 -- 172.21.15.26:0/3681839768 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc81d27d0 con 0x7f4cc80a2800 2024-09-16T14:47:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cc0004e90 con 0x7f4cc80a2800 2024-09-16T14:47:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4cc00051a0 con 0x7f4cc80a2800 2024-09-16T14:47:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbe7fc640 10 monclient: dump: 2024-09-16T14:47:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.716 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cbffff640 4 set_mon_vals no callback set 2024-09-16T14:47:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cabfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.695+0000 7f4cabfff640 1 -- 172.21.15.26:0/3681839768 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4ca4006590 con 0x7f4ca4002830 2024-09-16T14:47:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082002307 0 0) 0x7f4ca4006590 con 0x7f4ca4002830 2024-09-16T14:47:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 10 monclient(hunting): my global_id is 124206 2024-09-16T14:47:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cc809e500 con 0x7f4ca4002830 2024-09-16T14:47:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4cc000fc90 con 0x7f4cc80a2800 2024-09-16T14:47:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(133..133 src has 73..133) v4 ==== 6989+0+0 (unknown 2876426089 0 0) 0x7f4cc00442d0 con 0x7f4cc80a2800 2024-09-16T14:47:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2732773230 0 0) 0x7f4cc809e500 con 0x7f4ca4002830 2024-09-16T14:47:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ca4006590 con 0x7f4ca4002830 2024-09-16T14:47:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4cc4003e30 con 0x7f4ca4002830 2024-09-16T14:47:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 10 monclient: dump: 2024-09-16T14:47:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.718 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1052823753 0 0) 0x7f4ca4006590 con 0x7f4ca4002830 2024-09-16T14:47:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ca400a0e0 con 0x7f4ca4002830 2024-09-16T14:47:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4ccd171640 10 monclient: get_auth_request con 0x7f4cac036120 auth_method 0 2024-09-16T14:47:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f4cc4006190 con 0x7f4ca4002830 2024-09-16T14:47:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 >> v1:172.21.15.26:6789/0 conn(0x7f4ca4002830 legacy=0x7f4ca4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.699+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 >> v1:172.21.15.26:6789/0 conn(0x7f4ca4002830 legacy=0x7f4ca4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.703+0000 7f4cabfff640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:47:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.703+0000 7f4cabfff640 1 -- 172.21.15.26:0/3681839768 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4ca4009d80 con 0x7f4ca40090a0 2024-09-16T14:47:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.703+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341620223 0 0) 0x7f4ca4009d80 con 0x7f4ca40090a0 2024-09-16T14:47:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.703+0000 7f4cbe7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.703+0000 7f4cbe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.703+0000 7f4cbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.703+0000 7f4cbe7fc640 10 monclient(hunting): my global_id is 124209 2024-09-16T14:47:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.703+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ca4006590 con 0x7f4ca40090a0 2024-09-16T14:47:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.703+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 75010133 0 0) 0x7f4ca4006590 con 0x7f4ca40090a0 2024-09-16T14:47:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.703+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ca4009d80 con 0x7f4ca40090a0 2024-09-16T14:47:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.703+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4cc4004700 con 0x7f4ca40090a0 2024-09-16T14:47:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.703+0000 7f4cbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.703+0000 7f4cbe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.703+0000 7f4cbe7fc640 10 monclient: dump: 2024-09-16T14:47:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.721 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.703+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 638284378 0 0) 0x7f4ca4009d80 con 0x7f4ca40090a0 2024-09-16T14:47:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.703+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4ca400c290 con 0x7f4ca40090a0 2024-09-16T14:47:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.707+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4200 (unknown 0 0 2656818531) 0x7f4cc4005b10 con 0x7f4ca40090a0 2024-09-16T14:47:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.707+0000 7f4cbe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:47:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.707+0000 7f4cbe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.707+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 >> v1:172.21.15.26:6789/0 conn(0x7f4ca40090a0 legacy=0x7f4ca4009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.707+0000 7f4cbe7fc640 1 -- 172.21.15.26:0/3681839768 >> v1:172.21.15.26:6789/0 conn(0x7f4ca40090a0 legacy=0x7f4ca4009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:57.722 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:47:57.722 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:57.722 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:57.722 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:47:57.722 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-09-16T14:47:57.723 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:47:57.723 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:47:57.723 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:47:57.723 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:57.723 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:47:57.723 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:57.723 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:47:57.723 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:47:57.723 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:57.723 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:57.723 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:57.723 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:57.723 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:57.724 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:57.724 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:57.724 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:57.724 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:57.724 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:57.724 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:57.724 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:47:57.724 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:47:57.724 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:57.724 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:57.724 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:57.724 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:57.724 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:57.725 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:57.725 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:57.725 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:57.725 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:57.725 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:57.725 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:57.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.725 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:47:57.725 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:47:57.725 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:47:57.725 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:47:57.725 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:47:57.725 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:47:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:47:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:47:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:47:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:47:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:47:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:47:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:47:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:47:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:47:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:47:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:47:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:47:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:47:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:47:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:47:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:47:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:47:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:47:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:47:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:47:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:47:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:47:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:47:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:47:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:47:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:47:57.743 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:47:57.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.707+0000 7f4cabfff640 1 -- 172.21.15.26:0/3681839768 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4cac036120 legacy=0x7f4cac0385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.707+0000 7f4cabfff640 10 monclient: shutdown 2024-09-16T14:47:57.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.707+0000 7f4cabfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:57.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.707+0000 7f4cabfff640 1 -- 172.21.15.26:0/3681839768 >> v1:172.21.15.26:6790/0 conn(0x7f4cc80a2800 legacy=0x7f4cc81d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.707+0000 7f4cce173640 1 -- 172.21.15.26:0/3681839768 reap_dead start 2024-09-16T14:47:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.707+0000 7f4cce173640 1 -- 172.21.15.26:0/3681839768 reap_dead start 2024-09-16T14:47:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.707+0000 7f4cabfff640 1 -- 172.21.15.26:0/3681839768 shutdown_connections 2024-09-16T14:47:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.707+0000 7f4cabfff640 1 -- 172.21.15.26:0/3681839768 >> 172.21.15.26:0/3681839768 conn(0x7f4cc80a7f30 msgr2=0x7f4cc806b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.707+0000 7f4cabfff640 1 -- 172.21.15.26:0/3681839768 shutdown_connections 2024-09-16T14:47:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.707+0000 7f4cabfff640 1 -- 172.21.15.26:0/3681839768 wait complete. 2024-09-16T14:47:57.748 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:47:57.748 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:47:57.749 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:47:57.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.863+0000 7fb175d06640 10 monclient: get_monmap_and_config 2024-09-16T14:47:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.863+0000 7fb175d06640 10 monclient: build_initial_monmap 2024-09-16T14:47:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.863+0000 7fb175d06640 10 monclient: monmap: 2024-09-16T14:47:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:47:57.868370+0000 2024-09-16T14:47:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:47:57.868370+0000 2024-09-16T14:47:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:57.869 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.863+0000 7fb175d06640 1 Processor -- start 2024-09-16T14:47:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.863+0000 7fb175d06640 1 -- start start 2024-09-16T14:47:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.863+0000 7fb175d06640 10 monclient: init 2024-09-16T14:47:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient(hunting): picked mon.noname-b con 0x7fb1700a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient(hunting): picked mon.noname-a con 0x7fb170099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient(hunting): picked mon.noname-c con 0x7fb1700a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb170098ea0 con 0x7fb170099b60 2024-09-16T14:47:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1700997b0 con 0x7fb1700a26f0 2024-09-16T14:47:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb170099460 con 0x7fb1700a1c00 2024-09-16T14:47:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient(hunting): authenticate will time out at 1301.988718s 2024-09-16T14:47:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16effd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb170099b60 0x7fb170139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42706/0 (socket says 172.21.15.26:42706) 2024-09-16T14:47:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16ffff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fb1700a26f0 legacy=0x7fb17013d330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16effd640 1 -- 172.21.15.26:0/2577555591 learned_addr learned my addr 172.21.15.26:0/2577555591 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16f7fe640 1 --1- 172.21.15.26:0/2577555591 >> v1:172.21.15.26:6790/0 conn(0x7fb1700a1c00 0x7fb1700a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40906/0 (socket says 172.21.15.26:40906) 2024-09-16T14:47:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 1 -- 172.21.15.26:0/2577555591 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604894780 0 0) 0x7fb170098ea0 con 0x7fb170099b60 2024-09-16T14:47:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient(hunting): my global_id is 124212 2024-09-16T14:47:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 1 -- 172.21.15.26:0/2577555591 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb158003530 con 0x7fb170099b60 2024-09-16T14:47:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 1 -- 172.21.15.26:0/2577555591 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189760877 0 0) 0x7fb170099460 con 0x7fb1700a1c00 2024-09-16T14:47:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient(hunting): my global_id is 114515 2024-09-16T14:47:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 1 -- 172.21.15.26:0/2577555591 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb170098ea0 con 0x7fb1700a1c00 2024-09-16T14:47:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 1 -- 172.21.15.26:0/2577555591 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2961728840 0 0) 0x7fb170098ea0 con 0x7fb1700a1c00 2024-09-16T14:47:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 1 -- 172.21.15.26:0/2577555591 >> v1:172.21.15.189:6789/0 conn(0x7fb1700a26f0 legacy=0x7fb17013d330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 1 -- 172.21.15.26:0/2577555591 >> v1:172.21.15.26:6789/0 conn(0x7fb170099b60 legacy=0x7fb170139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 1 monclient: found mon.noname-b 2024-09-16T14:47:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:47:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 1 -- 172.21.15.26:0/2577555591 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1700a34b0 con 0x7fb1700a1c00 2024-09-16T14:47:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 1 -- 172.21.15.26:0/2577555591 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb160001580 con 0x7fb1700a1c00 2024-09-16T14:47:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient: dump: 2024-09-16T14:47:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.882 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 5 monclient: authenticate success, global_id 114515 2024-09-16T14:47:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 1 -- 172.21.15.26:0/2577555591 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1600017a0 con 0x7fb1700a1c00 2024-09-16T14:47:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 1 -- 172.21.15.26:0/2577555591 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb160001bb0 con 0x7fb1700a1c00 2024-09-16T14:47:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16e7fc640 10 monclient: dump: 2024-09-16T14:47:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.884 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 4 set_mon_vals no callback set 2024-09-16T14:47:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient: shutdown 2024-09-16T14:47:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 1 -- 172.21.15.26:0/2577555591 >> v1:172.21.15.26:6790/0 conn(0x7fb1700a1c00 legacy=0x7fb1700a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 1 -- 172.21.15.26:0/2577555591 shutdown_connections 2024-09-16T14:47:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 1 -- 172.21.15.26:0/2577555591 >> 172.21.15.26:0/2577555591 conn(0x7fb1700a3d00 msgr2=0x7fb1700a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 1 -- 172.21.15.26:0/2577555591 shutdown_connections 2024-09-16T14:47:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 1 -- 172.21.15.26:0/2577555591 wait complete. 2024-09-16T14:47:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient: build_initial_monmap 2024-09-16T14:47:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient: monmap: 2024-09-16T14:47:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:47:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:47:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:47:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:47:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:57.885 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 1 Processor -- start 2024-09-16T14:47:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 1 -- start start 2024-09-16T14:47:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient: init 2024-09-16T14:47:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:47:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient(hunting): picked mon.noname-a con 0x7fb170099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient(hunting): picked mon.noname-b con 0x7fb1700a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient(hunting): picked mon.noname-c con 0x7fb1700a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17009e3f0 con 0x7fb170099b60 2024-09-16T14:47:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17009eb00 con 0x7fb1700a1c00 2024-09-16T14:47:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17011fb10 con 0x7fb1700a26f0 2024-09-16T14:47:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb175d06640 10 monclient(hunting): authenticate will time out at 1301.991704s 2024-09-16T14:47:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16f7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb170099b60 0x7fb1701381b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42710/0 (socket says 172.21.15.26:42710) 2024-09-16T14:47:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16f7fe640 1 -- 172.21.15.26:0/1173274095 learned_addr learned my addr 172.21.15.26:0/1173274095 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:47:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.867+0000 7fb16effd640 1 -- 172.21.15.26:0/1173274095 >> v1:172.21.15.189:6789/0 conn(0x7fb1700a1c00 legacy=0x7fb1701ce1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181738160 0 0) 0x7fb17009e3f0 con 0x7fb170099b60 2024-09-16T14:47:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): my global_id is 124215 2024-09-16T14:47:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb15c003530 con 0x7fb170099b60 2024-09-16T14:47:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340952083 0 0) 0x7fb17011fb10 con 0x7fb1700a26f0 2024-09-16T14:47:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): my global_id is 114518 2024-09-16T14:47:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb17009e3f0 con 0x7fb1700a26f0 2024-09-16T14:47:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3341221302 0 0) 0x7fb15c003530 con 0x7fb170099b60 2024-09-16T14:47:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb17011fb10 con 0x7fb170099b60 2024-09-16T14:47:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb160001b70 con 0x7fb170099b60 2024-09-16T14:47:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): dump: 2024-09-16T14:47:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.889 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 302584008 0 0) 0x7fb17009e3f0 con 0x7fb1700a26f0 2024-09-16T14:47:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb15c003530 con 0x7fb1700a26f0 2024-09-16T14:47:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb168004340 con 0x7fb1700a26f0 2024-09-16T14:47:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): dump: 2024-09-16T14:47:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.890 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.890 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.890 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.890 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.890 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.890 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.890 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1760071991 0 0) 0x7fb17011fb10 con 0x7fb170099b60 2024-09-16T14:47:57.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 >> v1:172.21.15.26:6790/0 conn(0x7fb1700a26f0 legacy=0x7fb1701d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 >> v1:172.21.15.189:6789/0 conn(0x7fb1700a1c00 legacy=0x7fb1701ce1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:57.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient: _finish_hunting 0 2024-09-16T14:47:57.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 monclient: found mon.a 2024-09-16T14:47:57.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:57.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:57.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17011f270 con 0x7fb170099b60 2024-09-16T14:47:57.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient: _finish_auth 0 2024-09-16T14:47:57.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient: _check_auth_tickets 2024-09-16T14:47:57.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:57.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb175d06640 5 monclient: authenticate success, global_id 124215 2024-09-16T14:47:57.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb175d06640 10 monclient: _renew_subs 2024-09-16T14:47:57.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb175d06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:57.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb175d06640 1 -- 172.21.15.26:0/1173274095 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb17011f550 con 0x7fb170099b60 2024-09-16T14:47:57.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb175d06640 10 monclient: _renew_subs 2024-09-16T14:47:57.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb175d06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:47:57.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb175d06640 1 -- 172.21.15.26:0/1173274095 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb17011e4e0 con 0x7fb170099b60 2024-09-16T14:47:57.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1600011a0 con 0x7fb170099b60 2024-09-16T14:47:57.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:57.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb160004d10 con 0x7fb170099b60 2024-09-16T14:47:57.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:57.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb16dffb640 4 set_mon_vals no callback set 2024-09-16T14:47:57.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient: dump: 2024-09-16T14:47:57.893 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.893 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.893 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.893 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.893 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.893 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.894 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.894 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.894 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.894 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb175d06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:57.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb175d06640 1 -- 172.21.15.26:0/1173274095 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb150006590 con 0x7fb150002830 2024-09-16T14:47:57.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb160004ef0 con 0x7fb170099b60 2024-09-16T14:47:57.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(133..133 src has 73..133) v4 ==== 6989+0+0 (unknown 2876426089 0 0) 0x7fb160044050 con 0x7fb170099b60 2024-09-16T14:47:57.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770149555 0 0) 0x7fb150006590 con 0x7fb150002830 2024-09-16T14:47:57.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:57.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:57.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:57.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient(hunting): my global_id is 114521 2024-09-16T14:47:57.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb17011fb10 con 0x7fb150002830 2024-09-16T14:47:57.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb16ffff640 10 monclient: get_auth_request con 0x7fb15c0361e0 auth_method 0 2024-09-16T14:47:57.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3502604327 0 0) 0x7fb17011fb10 con 0x7fb150002830 2024-09-16T14:47:57.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb150006590 con 0x7fb150002830 2024-09-16T14:47:57.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb164001930 con 0x7fb150002830 2024-09-16T14:47:57.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:57.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient: dump: 2024-09-16T14:47:57.895 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.895 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.895 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.895 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.895 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.896 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.896 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.896 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.896 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.896 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3580858702 0 0) 0x7fb150006590 con 0x7fb150002830 2024-09-16T14:47:57.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb15000a0e0 con 0x7fb150002830 2024-09-16T14:47:57.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fb164001b60 con 0x7fb150002830 2024-09-16T14:47:57.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:57.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:57.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 >> v1:172.21.15.26:6790/0 conn(0x7fb150002830 legacy=0x7fb150004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.871+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 >> v1:172.21.15.26:6790/0 conn(0x7fb150002830 legacy=0x7fb150004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb175d06640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:47:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb175d06640 1 -- 172.21.15.26:0/1173274095 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb150009d80 con 0x7fb1500090a0 2024-09-16T14:47:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637638253 0 0) 0x7fb150009d80 con 0x7fb1500090a0 2024-09-16T14:47:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 10 monclient(hunting): _negotiate 2024-09-16T14:47:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 10 monclient(hunting): my global_id is 114524 2024-09-16T14:47:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb150006590 con 0x7fb1500090a0 2024-09-16T14:47:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2298329156 0 0) 0x7fb150006590 con 0x7fb1500090a0 2024-09-16T14:47:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb150009d80 con 0x7fb1500090a0 2024-09-16T14:47:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb1640049e0 con 0x7fb1500090a0 2024-09-16T14:47:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 10 monclient: dump: 2024-09-16T14:47:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:47:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:47:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:57.900 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:47:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1971045675 0 0) 0x7fb150009d80 con 0x7fb1500090a0 2024-09-16T14:47:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb15000c290 con 0x7fb1500090a0 2024-09-16T14:47:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4036 (unknown 0 0 2647945565) 0x7fb164004c80 con 0x7fb1500090a0 2024-09-16T14:47:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:47:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 >> v1:172.21.15.26:6790/0 conn(0x7fb1500090a0 legacy=0x7fb150009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb157fff640 1 -- 172.21.15.26:0/1173274095 >> v1:172.21.15.26:6790/0 conn(0x7fb1500090a0 legacy=0x7fb150009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:47:57.900 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:47:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:47:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-09-16T14:47:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:47:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:47:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:47:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:47:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:47:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:47:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:47:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:47:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:47:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:47:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:47:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:47:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:47:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:47:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:47:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:47:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:47:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:47:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:47:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:47:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:47:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:47:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:47:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:47:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:47:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:47:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:47:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:47:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:47:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:47:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:47:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:47:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:47:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:47:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:47:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:47:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:47:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:47:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:47:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:47:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:47:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:47:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:47:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:47:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:47:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:47:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:47:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:47:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:47:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:47:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:47:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:47:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:47:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:47:57.910 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:47:57.910 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:47:57.910 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:47:57.910 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:47:57.910 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:57.910 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:57.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.910 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:47:57.910 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:47:57.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.911 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:57.911 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:57.911 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:47:57.911 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:57.911 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:57.911 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:47:57.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.911 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:57.912 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:57.912 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:47:57.912 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:57.912 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:47:57.912 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:47:57.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:47:57.912 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:47:57.912 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:47:57.912 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:47:57.912 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:47:57.912 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:47:57.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:47:57.913 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:47:57.913 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:47:57.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb175d06640 1 -- 172.21.15.26:0/1173274095 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb15c0361e0 legacy=0x7fb15c0386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb175d06640 10 monclient: shutdown 2024-09-16T14:47:57.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb175d06640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:57.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb175d06640 1 -- 172.21.15.26:0/1173274095 >> v1:172.21.15.26:6789/0 conn(0x7fb170099b60 legacy=0x7fb1701381b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:57.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb16ffff640 1 -- 172.21.15.26:0/1173274095 reap_dead start 2024-09-16T14:47:57.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb16ffff640 1 -- 172.21.15.26:0/1173274095 reap_dead start 2024-09-16T14:47:57.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb175d06640 1 -- 172.21.15.26:0/1173274095 shutdown_connections 2024-09-16T14:47:57.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb175d06640 1 -- 172.21.15.26:0/1173274095 >> 172.21.15.26:0/1173274095 conn(0x7fb1700a3d00 msgr2=0x7fb1700a0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:57.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb175d06640 1 -- 172.21.15.26:0/1173274095 shutdown_connections 2024-09-16T14:47:57.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:47:57.879+0000 7fb175d06640 1 -- 172.21.15.26:0/1173274095 wait complete. 2024-09-16T14:47:57.915 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:47:57.915 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:47:58.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:58.242+0000 7f7a11ffb640 10 monclient: tick 2024-09-16T14:47:58.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:58.242+0000 7f7a11ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:58.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:58.242+0000 7f7a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:58.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:58.314+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "barset"}]=0 v134) v1 ==== 89+0+0 (unknown 803799983 0 0) 0x7f7a140144f0 con 0x7f7a2409df10 2024-09-16T14:47:58.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:58.314+0000 7f7a12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T14:47:58.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:58.314+0000 7f7a12ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:58.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:58.422+0000 7f7a28ff5640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T14:47:58.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:58.422+0000 7f7a28ff5640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T14:47:58.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:58.422+0000 7f7a28ff5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:58.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:58.422+0000 7f7a28ff5640 1 -- 172.21.15.189:0/529177721 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush weight-set rm", "pool": "barset"} v 0) v1 -- 0x7f7a08005a40 con 0x7f7a2409df10 2024-09-16T14:47:59.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.246+0000 7f7a11ffb640 10 monclient: tick 2024-09-16T14:47:59.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.246+0000 7f7a11ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:47:59.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.246+0000 7f7a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:59.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.322+0000 7f7a12ffd640 1 -- 172.21.15.189:0/529177721 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "barset"}]=0 v135) v1 ==== 89+0+0 (unknown 1838890822 0 0) 0x7f7a140146d0 con 0x7f7a2409df10 2024-09-16T14:47:59.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.322+0000 7f7a12ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T14:47:59.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.322+0000 7f7a12ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:59.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.330+0000 7f7a28ff5640 1 -- 172.21.15.189:0/529177721 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7a04036230 legacy=0x7f7a040386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:59.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.330+0000 7f7a28ff5640 10 monclient: shutdown 2024-09-16T14:47:59.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.330+0000 7f7a28ff5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:59.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.330+0000 7f7a28ff5640 1 -- 172.21.15.189:0/529177721 >> v1:172.21.15.26:6790/0 conn(0x7f7a2409df10 legacy=0x7f7a2411a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:59.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.330+0000 7f7a28ff5640 1 -- 172.21.15.189:0/529177721 shutdown_connections 2024-09-16T14:47:59.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.330+0000 7f7a28ff5640 1 -- 172.21.15.189:0/529177721 >> 172.21.15.189:0/529177721 conn(0x7f7a240a3320 msgr2=0x7f7a24131aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:59.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.330+0000 7f7a28ff5640 1 -- 172.21.15.189:0/529177721 shutdown_connections 2024-09-16T14:47:59.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.330+0000 7f7a28ff5640 1 -- 172.21.15.189:0/529177721 wait complete. 2024-09-16T14:47:59.356 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set ls 2024-09-16T14:47:59.356 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c barset 2024-09-16T14:47:59.356 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 0 2024-09-16T14:47:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 10 monclient: get_monmap_and_config 2024-09-16T14:47:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 10 monclient: build_initial_monmap 2024-09-16T14:47:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 10 monclient: monmap: 2024-09-16T14:47:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:59.470537+0000 2024-09-16T14:47:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:59.470537+0000 2024-09-16T14:47:59.471 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:59.471 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:59.471 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:59.471 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:59.471 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:59.471 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 1 Processor -- start 2024-09-16T14:47:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 1 -- start start 2024-09-16T14:47:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 10 monclient: init 2024-09-16T14:47:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 10 monclient(hunting): picked mon.noname-b con 0x7f459809df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 10 monclient(hunting): picked mon.noname-c con 0x7f459809e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 10 monclient(hunting): picked mon.noname-a con 0x7f459809e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459809ee20 con 0x7f459809e740 2024-09-16T14:47:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459809f100 con 0x7f459809e2f0 2024-09-16T14:47:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459809f3e0 con 0x7f459809df10 2024-09-16T14:47:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 10 monclient(hunting): authenticate will time out at 1261.586159s 2024-09-16T14:47:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d9c2640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f459809e2f0 legacy=0x7f45981378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459e1c3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f459809df10 0x7f459809d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:43746/0 (socket says 172.21.15.189:43746) 2024-09-16T14:47:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459e9c4640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f459809e740 0x7f459813b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33750/0 (socket says 172.21.15.189:33750) 2024-09-16T14:47:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459e1c3640 1 -- 172.21.15.189:0/1294664452 learned_addr learned my addr 172.21.15.189:0/1294664452 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 1 -- 172.21.15.189:0/1294664452 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712373184 0 0) 0x7f459809ee20 con 0x7f459809e740 2024-09-16T14:47:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient(hunting): _negotiate 2024-09-16T14:47:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient(hunting): my global_id is 124218 2024-09-16T14:47:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 1 -- 172.21.15.189:0/1294664452 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4584003530 con 0x7f459809e740 2024-09-16T14:47:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 1 -- 172.21.15.189:0/1294664452 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730621420 0 0) 0x7f459809f3e0 con 0x7f459809df10 2024-09-16T14:47:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient(hunting): _negotiate 2024-09-16T14:47:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient(hunting): my global_id is 114527 2024-09-16T14:47:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 1 -- 172.21.15.189:0/1294664452 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f459809ee20 con 0x7f459809df10 2024-09-16T14:47:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 1 -- 172.21.15.189:0/1294664452 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2053941001 0 0) 0x7f4584003530 con 0x7f459809e740 2024-09-16T14:47:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 1 -- 172.21.15.189:0/1294664452 >> v1:172.21.15.26:6790/0 conn(0x7f459809df10 legacy=0x7f459809d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 1 -- 172.21.15.189:0/1294664452 >> v1:172.21.15.189:6789/0 conn(0x7f459809e2f0 legacy=0x7f45981378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient: _finish_hunting 0 2024-09-16T14:47:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 1 monclient: found mon.noname-a 2024-09-16T14:47:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:47:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 1 -- 172.21.15.189:0/1294664452 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459809c690 con 0x7f459809e740 2024-09-16T14:47:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient: _finish_auth 0 2024-09-16T14:47:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient: _check_auth_tickets 2024-09-16T14:47:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 1 -- 172.21.15.189:0/1294664452 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f45900019a0 con 0x7f459809e740 2024-09-16T14:47:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:47:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient: dump: 2024-09-16T14:47:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:59.478 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 5 monclient: authenticate success, global_id 124218 2024-09-16T14:47:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 1 -- 172.21.15.189:0/1294664452 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4590001c00 con 0x7f459809e740 2024-09-16T14:47:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f45a0427640 4 set_mon_vals no callback set 2024-09-16T14:47:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 1 -- 172.21.15.189:0/1294664452 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4590001fd0 con 0x7f459809e740 2024-09-16T14:47:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.466+0000 7f459d1c1640 10 monclient: dump: 2024-09-16T14:47:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:59.479 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 10 monclient: shutdown 2024-09-16T14:47:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 1 -- 172.21.15.189:0/1294664452 >> v1:172.21.15.26:6789/0 conn(0x7f459809e740 legacy=0x7f459813b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 1 -- 172.21.15.189:0/1294664452 shutdown_connections 2024-09-16T14:47:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 1 -- 172.21.15.189:0/1294664452 >> 172.21.15.189:0/1294664452 conn(0x7f45980a3320 msgr2=0x7f459809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 1 -- 172.21.15.189:0/1294664452 shutdown_connections 2024-09-16T14:47:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 1 -- 172.21.15.189:0/1294664452 wait complete. 2024-09-16T14:47:59.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 10 monclient: build_initial_monmap 2024-09-16T14:47:59.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 10 monclient: monmap: 2024-09-16T14:47:59.484 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:59.484 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:59.484 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:59.484 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:59.484 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:59.484 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:59.484 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:59.484 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:59.484 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:59.484 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:59.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 1 Processor -- start 2024-09-16T14:47:59.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 1 -- start start 2024-09-16T14:47:59.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 10 monclient: init 2024-09-16T14:47:59.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:59.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:47:59.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 10 monclient(hunting): picked mon.noname-c con 0x7f459809df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:59.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 10 monclient(hunting): picked mon.noname-a con 0x7f459809e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:59.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 10 monclient(hunting): picked mon.noname-b con 0x7f459809e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:59.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459806b380 con 0x7f459809e2f0 2024-09-16T14:47:59.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459809be60 con 0x7f459809e740 2024-09-16T14:47:59.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4598120ea0 con 0x7f459809df10 2024-09-16T14:47:59.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:59.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 10 monclient(hunting): authenticate will time out at 1261.590057s 2024-09-16T14:47:59.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f459e9c4640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f459809e740 legacy=0x7f459809b090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:59.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f459e1c3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f459809df10 0x7f459809a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:43756/0 (socket says 172.21.15.189:43756) 2024-09-16T14:47:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f459e1c3640 1 -- 172.21.15.189:0/3885746302 learned_addr learned my addr 172.21.15.189:0/3885746302 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f459d9c2640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f459809e2f0 0x7f459809a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33760/0 (socket says 172.21.15.189:33760) 2024-09-16T14:47:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818664716 0 0) 0x7f4598120ea0 con 0x7f459809df10 2024-09-16T14:47:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient(hunting): my global_id is 114530 2024-09-16T14:47:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f457c003530 con 0x7f459809df10 2024-09-16T14:47:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213779379 0 0) 0x7f459806b380 con 0x7f459809e2f0 2024-09-16T14:47:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient(hunting): _negotiate 2024-09-16T14:47:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient(hunting): my global_id is 124221 2024-09-16T14:47:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4598120ea0 con 0x7f459809e2f0 2024-09-16T14:47:59.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2429492211 0 0) 0x7f457c003530 con 0x7f459809df10 2024-09-16T14:47:59.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f459806b380 con 0x7f459809df10 2024-09-16T14:47:59.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4594001380 con 0x7f459809df10 2024-09-16T14:47:59.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:59.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:59.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient(hunting): dump: 2024-09-16T14:47:59.487 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:59.487 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:59.487 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.487 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.487 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:59.487 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:59.487 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:59.488 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:59.488 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:59.488 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:59.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:59.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2887876608 0 0) 0x7f4598120ea0 con 0x7f459809e2f0 2024-09-16T14:47:59.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f457c003530 con 0x7f459809e2f0 2024-09-16T14:47:59.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4588004050 con 0x7f459809e2f0 2024-09-16T14:47:59.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:59.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:59.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient(hunting): dump: 2024-09-16T14:47:59.488 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:59.488 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:59.488 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.489 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.489 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:59.489 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:59.489 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:59.489 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:59.489 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:59.489 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:59.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 158661716 0 0) 0x7f459806b380 con 0x7f459809df10 2024-09-16T14:47:59.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 >> v1:172.21.15.189:6789/0 conn(0x7f459809e740 legacy=0x7f459809b090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:59.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 >> v1:172.21.15.26:6789/0 conn(0x7f459809e2f0 legacy=0x7f459809a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:59.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient: _finish_hunting 0 2024-09-16T14:47:59.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 1 monclient: found mon.c 2024-09-16T14:47:59.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:59.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:59.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4598121520 con 0x7f459809df10 2024-09-16T14:47:59.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient: _finish_auth 0 2024-09-16T14:47:59.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 10 monclient: _check_auth_tickets 2024-09-16T14:47:59.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f458effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:59.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 5 monclient: authenticate success, global_id 114530 2024-09-16T14:47:59.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 10 monclient: _renew_subs 2024-09-16T14:47:59.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:59.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.470+0000 7f45a0427640 1 -- 172.21.15.189:0/3885746302 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4598121c10 con 0x7f459809df10 2024-09-16T14:47:59.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f45a0427640 10 monclient: _renew_subs 2024-09-16T14:47:59.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f45a0427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:59.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f45a0427640 1 -- 172.21.15.189:0/3885746302 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f459813c3a0 con 0x7f459809df10 2024-09-16T14:47:59.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45940014e0 con 0x7f459809df10 2024-09-16T14:47:59.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f458effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:59.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4594004c40 con 0x7f459809df10 2024-09-16T14:47:59.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f458effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:59.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f459c9c0640 4 set_mon_vals no callback set 2024-09-16T14:47:59.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f458effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:59.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f458effd640 10 monclient: dump: 2024-09-16T14:47:59.491 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:59.491 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:59.491 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.491 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.491 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:59.491 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:59.491 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:59.491 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:59.491 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:59.492 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:59.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f45a0427640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:59.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f45a0427640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:59.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f45a0427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:59.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f45a0427640 1 -- 172.21.15.189:0/3885746302 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4580005050 con 0x7f459809df10 2024-09-16T14:47:59.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f459400f6b0 con 0x7f459809df10 2024-09-16T14:47:59.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(135..135 src has 73..135) v4 ==== 6377+0+0 (unknown 60273888 0 0) 0x7f4594043b80 con 0x7f459809df10 2024-09-16T14:47:59.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.474+0000 7f459d9c2640 10 monclient: get_auth_request con 0x7f457c0361e0 auth_method 0 2024-09-16T14:47:59.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.486+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4594014400 con 0x7f459809df10 2024-09-16T14:47:59.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.486+0000 7f458effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:59.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.486+0000 7f458effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:47:59.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.654+0000 7f45a0427640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:59.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.654+0000 7f45a0427640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:59.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.654+0000 7f45a0427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:59.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.654+0000 7f45a0427640 1 -- 172.21.15.189:0/3885746302 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f4580005300 con 0x7f459809df10 2024-09-16T14:47:59.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.654+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v135) v1 ==== 71+0+0 (unknown 1171993247 0 0) 0x7f4594012aa0 con 0x7f459809df10 2024-09-16T14:47:59.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.654+0000 7f458effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:59.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.654+0000 7f458effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:47:59.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.758+0000 7f45a0427640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:59.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.758+0000 7f45a0427640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:59.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.758+0000 7f45a0427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:59.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.758+0000 7f45a0427640 1 -- 172.21.15.189:0/3885746302 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f4580005540 con 0x7f459809df10 2024-09-16T14:47:59.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.758+0000 7f458effd640 1 -- 172.21.15.189:0/3885746302 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v135) v1 ==== 71+0+0 (unknown 1171993247 0 0) 0x7f4594012aa0 con 0x7f459809df10 2024-09-16T14:47:59.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.758+0000 7f458effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:47:59.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.758+0000 7f458effd640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:47:59.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.762+0000 7f45a0427640 1 -- 172.21.15.189:0/3885746302 >> v1:172.21.15.189:6812/3196161582 conn(0x7f457c0361e0 legacy=0x7f457c0386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:59.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.762+0000 7f45a0427640 10 monclient: shutdown 2024-09-16T14:47:59.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.762+0000 7f45a0427640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:59.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.762+0000 7f45a0427640 1 -- 172.21.15.189:0/3885746302 >> v1:172.21.15.26:6790/0 conn(0x7f459809df10 legacy=0x7f459809a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:59.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.762+0000 7f45a0427640 1 -- 172.21.15.189:0/3885746302 shutdown_connections 2024-09-16T14:47:59.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.762+0000 7f45a0427640 1 -- 172.21.15.189:0/3885746302 >> 172.21.15.189:0/3885746302 conn(0x7f45980a3320 msgr2=0x7f459809d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:59.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.762+0000 7f45a0427640 1 -- 172.21.15.189:0/3885746302 shutdown_connections 2024-09-16T14:47:59.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.762+0000 7f45a0427640 1 -- 172.21.15.189:0/3885746302 wait complete. 2024-09-16T14:47:59.780 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set create-compat 2024-09-16T14:47:59.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f269d6ad640 10 monclient: get_monmap_and_config 2024-09-16T14:47:59.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f269d6ad640 10 monclient: build_initial_monmap 2024-09-16T14:47:59.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f269d6ad640 10 monclient: monmap: 2024-09-16T14:47:59.896 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:59.896 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:59.896 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:47:59.896255+0000 2024-09-16T14:47:59.896 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:47:59.896255+0000 2024-09-16T14:47:59.896 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:59.896 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:59.897 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:59.897 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:47:59.897 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:47:59.897 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:59.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f269d6ad640 1 Processor -- start 2024-09-16T14:47:59.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f269d6ad640 1 -- start start 2024-09-16T14:47:59.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f269d6ad640 10 monclient: init 2024-09-16T14:47:59.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f269d6ad640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:59.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f269d6ad640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:47:59.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f269d6ad640 10 monclient(hunting): picked mon.noname-b con 0x7f2698099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:59.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f269d6ad640 10 monclient(hunting): picked mon.noname-c con 0x7f26981334d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:59.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f269d6ad640 10 monclient(hunting): picked mon.noname-a con 0x7f2698136fe0 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:59.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f269d6ad640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269809b350 con 0x7f2698136fe0 2024-09-16T14:47:59.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f269d6ad640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269809f2e0 con 0x7f26981334d0 2024-09-16T14:47:59.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f269d6ad640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269809b070 con 0x7f2698099a70 2024-09-16T14:47:59.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f268ffff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f26981334d0 legacy=0x7f26981358c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:59.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f269d6ad640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:59.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.890+0000 7f269d6ad640 10 monclient(hunting): authenticate will time out at 1262.011987s 2024-09-16T14:47:59.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f2696ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f2698099a70 0x7f269809bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:43762/0 (socket says 172.21.15.189:43762) 2024-09-16T14:47:59.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26977fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2698136fe0 0x7f26981393d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33770/0 (socket says 172.21.15.189:33770) 2024-09-16T14:47:59.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f2696ffd640 1 -- 172.21.15.189:0/232518940 learned_addr learned my addr 172.21.15.189:0/232518940 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:59.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 1 -- 172.21.15.189:0/232518940 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 448444200 0 0) 0x7f269809b350 con 0x7f2698136fe0 2024-09-16T14:47:59.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:59.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:59.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:59.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient(hunting): my global_id is 124224 2024-09-16T14:47:59.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 1 -- 172.21.15.189:0/232518940 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2684004640 con 0x7f2698136fe0 2024-09-16T14:47:59.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 1 -- 172.21.15.189:0/232518940 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552897057 0 0) 0x7f269809b070 con 0x7f2698099a70 2024-09-16T14:47:59.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient(hunting): _negotiate 2024-09-16T14:47:59.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:59.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:59.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient(hunting): my global_id is 114533 2024-09-16T14:47:59.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 1 -- 172.21.15.189:0/232518940 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f269809b350 con 0x7f2698099a70 2024-09-16T14:47:59.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 1 -- 172.21.15.189:0/232518940 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3534459750 0 0) 0x7f269809b350 con 0x7f2698099a70 2024-09-16T14:47:59.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 1 -- 172.21.15.189:0/232518940 >> v1:172.21.15.189:6789/0 conn(0x7f26981334d0 legacy=0x7f26981358c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:47:59.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 1 -- 172.21.15.189:0/232518940 >> v1:172.21.15.26:6789/0 conn(0x7f2698136fe0 legacy=0x7f26981393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:59.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient: _finish_hunting 0 2024-09-16T14:47:59.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 1 monclient: found mon.noname-b 2024-09-16T14:47:59.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:59.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:47:59.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 1 -- 172.21.15.189:0/232518940 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269809b630 con 0x7f2698099a70 2024-09-16T14:47:59.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient: _finish_auth 0 2024-09-16T14:47:59.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient: _check_auth_tickets 2024-09-16T14:47:59.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:59.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 1 -- 172.21.15.189:0/232518940 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2688001600 con 0x7f2698099a70 2024-09-16T14:47:59.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:59.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:47:59.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient: dump: 2024-09-16T14:47:59.904 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:59.904 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:59.905 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.905 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.905 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:59.905 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:59.905 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:59.905 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:59.905 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:59.905 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:59.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 5 monclient: authenticate success, global_id 114533 2024-09-16T14:47:59.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:47:59.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 1 -- 172.21.15.189:0/232518940 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26880018b0 con 0x7f2698099a70 2024-09-16T14:47:59.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 10 monclient: get_monmap_and_config success 2024-09-16T14:47:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 4 set_mon_vals no callback set 2024-09-16T14:47:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 1 -- 172.21.15.189:0/232518940 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2688001c80 con 0x7f2698099a70 2024-09-16T14:47:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f26967fc640 10 monclient: dump: 2024-09-16T14:47:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:59.906 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:59.907 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:59.907 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:59.907 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:59.907 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:59.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 10 monclient: shutdown 2024-09-16T14:47:59.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:47:59.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 1 -- 172.21.15.189:0/232518940 >> v1:172.21.15.26:6790/0 conn(0x7f2698099a70 legacy=0x7f269809bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:59.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 1 -- 172.21.15.189:0/232518940 shutdown_connections 2024-09-16T14:47:59.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 1 -- 172.21.15.189:0/232518940 >> 172.21.15.189:0/232518940 conn(0x7f26980a3320 msgr2=0x7f269809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:47:59.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 1 -- 172.21.15.189:0/232518940 shutdown_connections 2024-09-16T14:47:59.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 1 -- 172.21.15.189:0/232518940 wait complete. 2024-09-16T14:47:59.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 10 monclient: build_initial_monmap 2024-09-16T14:47:59.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 10 monclient: monmap: 2024-09-16T14:47:59.908 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:47:59.908 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:47:59.908 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:47:59.908 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:47:59.908 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:47:59.908 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:47:59.908 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:47:59.908 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:47:59.908 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:47:59.908 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:59.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 1 Processor -- start 2024-09-16T14:47:59.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 1 -- start start 2024-09-16T14:47:59.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 10 monclient: init 2024-09-16T14:47:59.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 10 monclient: _reopen_session rank -1 2024-09-16T14:47:59.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.894+0000 7f269d6ad640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:47:59.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 10 monclient(hunting): picked mon.noname-a con 0x7f2698099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:47:59.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 10 monclient(hunting): picked mon.noname-c con 0x7f26981334d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:47:59.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 10 monclient(hunting): picked mon.noname-b con 0x7f2698136fe0 addr v1:172.21.15.189:6789/0 2024-09-16T14:47:59.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269806b380 con 0x7f2698099a70 2024-09-16T14:47:59.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2698121c30 con 0x7f2698136fe0 2024-09-16T14:47:59.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2698136ca0 con 0x7f26981334d0 2024-09-16T14:47:59.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f26977fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2698136fe0 legacy=0x7f26981d1b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:47:59.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 10 monclient(hunting): _renew_subs 2024-09-16T14:47:59.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 10 monclient(hunting): authenticate will time out at 1262.016182s 2024-09-16T14:47:59.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f26981334d0 0x7f26981ce450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:43772/0 (socket says 172.21.15.189:43772) 2024-09-16T14:47:59.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268ffff640 1 -- 172.21.15.189:0/3811138590 learned_addr learned my addr 172.21.15.189:0/3811138590 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:47:59.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133776600 0 0) 0x7f2698136ca0 con 0x7f26981334d0 2024-09-16T14:47:59.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:59.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:59.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:59.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient(hunting): my global_id is 114536 2024-09-16T14:47:59.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2678003530 con 0x7f26981334d0 2024-09-16T14:47:59.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504413948 0 0) 0x7f269806b380 con 0x7f2698099a70 2024-09-16T14:47:59.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:47:59.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:47:59.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:47:59.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient(hunting): my global_id is 124227 2024-09-16T14:47:59.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2698136ca0 con 0x7f2698099a70 2024-09-16T14:47:59.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2863007190 0 0) 0x7f2678003530 con 0x7f26981334d0 2024-09-16T14:47:59.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f269806b380 con 0x7f26981334d0 2024-09-16T14:47:59.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2680004160 con 0x7f26981334d0 2024-09-16T14:47:59.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:59.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:47:59.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient(hunting): dump: 2024-09-16T14:47:59.916 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:59.916 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:59.916 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.916 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.916 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:59.916 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:59.916 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:59.916 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:59.917 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:59.917 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:59.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:47:59.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1281159158 0 0) 0x7f2698136ca0 con 0x7f2698099a70 2024-09-16T14:47:59.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2678003530 con 0x7f2698099a70 2024-09-16T14:47:59.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f26880015c0 con 0x7f2698099a70 2024-09-16T14:47:59.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:59.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:47:59.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient(hunting): dump: 2024-09-16T14:47:59.917 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:59.918 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:59.918 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.918 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.918 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:59.918 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:59.918 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:59.918 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:59.918 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:59.918 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:59.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2449702391 0 0) 0x7f269806b380 con 0x7f26981334d0 2024-09-16T14:47:59.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 >> v1:172.21.15.189:6789/0 conn(0x7f2698136fe0 legacy=0x7f26981d1b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:47:59.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 >> v1:172.21.15.26:6789/0 conn(0x7f2698099a70 legacy=0x7f26981c9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:47:59.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:47:59.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 monclient: found mon.c 2024-09-16T14:47:59.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:47:59.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:59.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26981368e0 con 0x7f26981334d0 2024-09-16T14:47:59.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:47:59.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:47:59.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:47:59.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 5 monclient: authenticate success, global_id 114536 2024-09-16T14:47:59.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 10 monclient: _renew_subs 2024-09-16T14:47:59.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:59.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 1 -- 172.21.15.189:0/3811138590 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2698121950 con 0x7f26981334d0 2024-09-16T14:47:59.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 10 monclient: _renew_subs 2024-09-16T14:47:59.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:59.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 1 -- 172.21.15.189:0/3811138590 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f269811ecc0 con 0x7f26981334d0 2024-09-16T14:47:59.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2680004380 con 0x7f26981334d0 2024-09-16T14:47:59.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:47:59.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2680004710 con 0x7f26981334d0 2024-09-16T14:47:59.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:47:59.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:47:59.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 10 monclient: dump: 2024-09-16T14:47:59.920 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:47:59.920 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:47:59.921 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.921 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:47:59.921 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:47:59.921 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:47:59.921 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:47:59.921 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:47:59.921 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:47:59.921 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:47:59.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f2695ffb640 4 set_mon_vals no callback set 2024-09-16T14:47:59.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:59.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:59.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:47:59.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f269d6ad640 1 -- 172.21.15.189:0/3811138590 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f267c005050 con 0x7f26981334d0 2024-09-16T14:47:59.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f26800048f0 con 0x7f26981334d0 2024-09-16T14:47:59.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(135..135 src has 73..135) v4 ==== 6377+0+0 (unknown 60273888 0 0) 0x7f2680043b40 con 0x7f26981334d0 2024-09-16T14:47:59.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.898+0000 7f2696ffd640 10 monclient: get_auth_request con 0x7f2678036270 auth_method 0 2024-09-16T14:47:59.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.910+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2680014420 con 0x7f26981334d0 2024-09-16T14:47:59.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.910+0000 7f268f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:47:59.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:47:59.910+0000 7f268f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.066+0000 7f269d6ad640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.066+0000 7f269d6ad640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.066+0000 7f269d6ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.066+0000 7f269d6ad640 1 -- 172.21.15.189:0/3811138590 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f267c005300 con 0x7f26981334d0 2024-09-16T14:48:00.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.321+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v136) v1 ==== 82+0+0 (unknown 2954770266 0 0) 0x7f2680014240 con 0x7f26981334d0 2024-09-16T14:48:00.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.321+0000 7f268f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:00.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.321+0000 7f268f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:00.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.425+0000 7f269d6ad640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:00.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.425+0000 7f269d6ad640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:00.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.425+0000 7f269d6ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:00.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.425+0000 7f269d6ad640 1 -- 172.21.15.189:0/3811138590 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f267c005a40 con 0x7f26981334d0 2024-09-16T14:48:00.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.425+0000 7f268f7fe640 1 -- 172.21.15.189:0/3811138590 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v136) v1 ==== 115+0+0 (unknown 1037366964 0 0) 0x7f2680014f80 con 0x7f26981334d0 2024-09-16T14:48:00.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.425+0000 7f268f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:00.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.425+0000 7f268f7fe640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-16T14:48:00.432 INFO:tasks.workunit.client.0.smithi189.stderr:compat weight-set already created 2024-09-16T14:48:00.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.429+0000 7f269d6ad640 1 -- 172.21.15.189:0/3811138590 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2678036270 legacy=0x7f2678038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:00.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.429+0000 7f269d6ad640 10 monclient: shutdown 2024-09-16T14:48:00.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.429+0000 7f269d6ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:00.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.429+0000 7f269d6ad640 1 -- 172.21.15.189:0/3811138590 >> v1:172.21.15.26:6790/0 conn(0x7f26981334d0 legacy=0x7f26981ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:00.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.429+0000 7f269d6ad640 1 -- 172.21.15.189:0/3811138590 shutdown_connections 2024-09-16T14:48:00.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.429+0000 7f269d6ad640 1 -- 172.21.15.189:0/3811138590 >> 172.21.15.189:0/3811138590 conn(0x7f26980a3320 msgr2=0x7f2698136180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:00.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.429+0000 7f269d6ad640 1 -- 172.21.15.189:0/3811138590 shutdown_connections 2024-09-16T14:48:00.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.429+0000 7f269d6ad640 1 -- 172.21.15.189:0/3811138590 wait complete. 2024-09-16T14:48:00.449 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set ls 2024-09-16T14:48:00.450 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '(compat)' 2024-09-16T14:48:00.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 10 monclient: get_monmap_and_config 2024-09-16T14:48:00.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 10 monclient: build_initial_monmap 2024-09-16T14:48:00.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 10 monclient: monmap: 2024-09-16T14:48:00.567 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:00.567 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:00.567 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:00.566825+0000 2024-09-16T14:48:00.567 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:00.566825+0000 2024-09-16T14:48:00.567 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:00.567 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:00.567 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:00.567 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:00.567 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:00.567 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:00.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 1 Processor -- start 2024-09-16T14:48:00.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 1 -- start start 2024-09-16T14:48:00.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 10 monclient: init 2024-09-16T14:48:00.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:00.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:00.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 10 monclient(hunting): picked mon.noname-c con 0x7fca9c09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:00.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 10 monclient(hunting): picked mon.noname-a con 0x7fca9c09e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:00.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 10 monclient(hunting): picked mon.noname-b con 0x7fca9c09e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:00.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9c09ee20 con 0x7fca9c09e2f0 2024-09-16T14:48:00.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9c09f100 con 0x7fca9c09df10 2024-09-16T14:48:00.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9c09f3e0 con 0x7fca9c09e740 2024-09-16T14:48:00.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:00.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 10 monclient(hunting): authenticate will time out at 1262.682474s 2024-09-16T14:48:00.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9bfff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fca9c09df10 legacy=0x7fca9c09d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:00.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa093e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fca9c09e740 0x7fca9c13b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:43784/0 (socket says 172.21.15.189:43784) 2024-09-16T14:48:00.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9b7fe640 1 --1- 172.21.15.189:0/4273811888 >> v1:172.21.15.26:6789/0 conn(0x7fca9c09e2f0 0x7fca9c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33788/0 (socket says 172.21.15.189:33788) 2024-09-16T14:48:00.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa093e640 1 -- 172.21.15.189:0/4273811888 learned_addr learned my addr 172.21.15.189:0/4273811888 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:00.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 1 -- 172.21.15.189:0/4273811888 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297803061 0 0) 0x7fca9c09f3e0 con 0x7fca9c09e740 2024-09-16T14:48:00.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:00.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:00.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:00.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 10 monclient(hunting): my global_id is 114539 2024-09-16T14:48:00.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 1 -- 172.21.15.189:0/4273811888 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca84003530 con 0x7fca9c09e740 2024-09-16T14:48:00.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 1 -- 172.21.15.189:0/4273811888 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371715578 0 0) 0x7fca9c09ee20 con 0x7fca9c09e2f0 2024-09-16T14:48:00.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:00.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:00.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:00.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 10 monclient(hunting): my global_id is 124230 2024-09-16T14:48:00.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 1 -- 172.21.15.189:0/4273811888 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca9c09f3e0 con 0x7fca9c09e2f0 2024-09-16T14:48:00.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 1 -- 172.21.15.189:0/4273811888 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1996443420 0 0) 0x7fca84003530 con 0x7fca9c09e740 2024-09-16T14:48:00.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 1 -- 172.21.15.189:0/4273811888 >> v1:172.21.15.189:6789/0 conn(0x7fca9c09df10 legacy=0x7fca9c09d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:00.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 1 -- 172.21.15.189:0/4273811888 >> v1:172.21.15.26:6789/0 conn(0x7fca9c09e2f0 legacy=0x7fca9c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:00.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:00.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 1 monclient: found mon.noname-b 2024-09-16T14:48:00.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:00.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:48:00.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 1 -- 172.21.15.189:0/4273811888 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9c09c690 con 0x7fca9c09e740 2024-09-16T14:48:00.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 10 monclient: _finish_auth 0 2024-09-16T14:48:00.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:00.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:00.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 1 -- 172.21.15.189:0/4273811888 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fca8c0019a0 con 0x7fca9c09e740 2024-09-16T14:48:00.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:00.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:00.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fca9affd640 10 monclient: dump: 2024-09-16T14:48:00.580 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:00.580 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:00.580 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.581 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.581 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:00.581 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:00.581 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:00.581 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:00.581 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:00.581 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:00.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 5 monclient: authenticate success, global_id 114539 2024-09-16T14:48:00.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.561+0000 7fcaa23a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:00.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca9affd640 1 -- 172.21.15.189:0/4273811888 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca8c001c80 con 0x7fca9c09e740 2024-09-16T14:48:00.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca9affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:00.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:00.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 4 set_mon_vals no callback set 2024-09-16T14:48:00.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca9affd640 1 -- 172.21.15.189:0/4273811888 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fca8c002010 con 0x7fca9c09e740 2024-09-16T14:48:00.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:00.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca9affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:00.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca9affd640 10 monclient: dump: 2024-09-16T14:48:00.582 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:00.582 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:00.582 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.582 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.582 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:00.582 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:00.582 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:00.582 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:00.582 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:00.583 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:00.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 10 monclient: shutdown 2024-09-16T14:48:00.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:00.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 1 -- 172.21.15.189:0/4273811888 >> v1:172.21.15.26:6790/0 conn(0x7fca9c09e740 legacy=0x7fca9c13b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:00.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 1 -- 172.21.15.189:0/4273811888 shutdown_connections 2024-09-16T14:48:00.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 1 -- 172.21.15.189:0/4273811888 >> 172.21.15.189:0/4273811888 conn(0x7fca9c0a3320 msgr2=0x7fca9c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:00.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 1 -- 172.21.15.189:0/4273811888 shutdown_connections 2024-09-16T14:48:00.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 1 -- 172.21.15.189:0/4273811888 wait complete. 2024-09-16T14:48:00.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 10 monclient: build_initial_monmap 2024-09-16T14:48:00.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 10 monclient: monmap: 2024-09-16T14:48:00.583 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:00.583 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:00.583 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:00.583 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:00.584 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:00.584 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:00.584 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:00.584 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:00.584 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:00.584 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:00.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 1 Processor -- start 2024-09-16T14:48:00.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 1 -- start start 2024-09-16T14:48:00.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 10 monclient: init 2024-09-16T14:48:00.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:00.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:00.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 10 monclient(hunting): picked mon.noname-b con 0x7fca9c09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:00.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 10 monclient(hunting): picked mon.noname-c con 0x7fca9c09e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:00.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 10 monclient(hunting): picked mon.noname-a con 0x7fca9c09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:00.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9c06ce70 con 0x7fca9c09e740 2024-09-16T14:48:00.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca9bfff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fca9c09df10 legacy=0x7fca9c09a220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:00.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9c06b380 con 0x7fca9c09df10 2024-09-16T14:48:00.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9c09b710 con 0x7fca9c09e2f0 2024-09-16T14:48:00.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:00.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 10 monclient(hunting): authenticate will time out at 1262.686044s 2024-09-16T14:48:00.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca9b7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fca9c09e2f0 0x7fca9c09a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:43794/0 (socket says 172.21.15.189:43794) 2024-09-16T14:48:00.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca9b7fe640 1 -- 172.21.15.189:0/2934294373 learned_addr learned my addr 172.21.15.189:0/2934294373 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:00.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808075059 0 0) 0x7fca9c09b710 con 0x7fca9c09e2f0 2024-09-16T14:48:00.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:00.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:00.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:00.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient(hunting): my global_id is 114542 2024-09-16T14:48:00.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca88003530 con 0x7fca9c09e2f0 2024-09-16T14:48:00.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131095255 0 0) 0x7fca9c06ce70 con 0x7fca9c09e740 2024-09-16T14:48:00.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:00.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:00.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:00.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient(hunting): my global_id is 124233 2024-09-16T14:48:00.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca9c09b710 con 0x7fca9c09e740 2024-09-16T14:48:00.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3200808364 0 0) 0x7fca88003530 con 0x7fca9c09e2f0 2024-09-16T14:48:00.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca9c06ce70 con 0x7fca9c09e2f0 2024-09-16T14:48:00.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fca900018c0 con 0x7fca9c09e2f0 2024-09-16T14:48:00.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:00.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:00.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient(hunting): dump: 2024-09-16T14:48:00.587 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:00.587 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:00.587 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.587 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.587 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:00.587 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:00.587 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:00.587 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:00.587 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:00.587 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:00.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:00.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 394291400 0 0) 0x7fca9c09b710 con 0x7fca9c09e740 2024-09-16T14:48:00.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca88003530 con 0x7fca9c09e740 2024-09-16T14:48:00.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fca8c0017c0 con 0x7fca9c09e740 2024-09-16T14:48:00.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:00.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:00.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient(hunting): dump: 2024-09-16T14:48:00.588 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:00.588 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:00.588 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.588 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.588 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:00.589 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:00.589 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:00.589 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:00.589 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:00.589 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:00.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3807332437 0 0) 0x7fca9c06ce70 con 0x7fca9c09e2f0 2024-09-16T14:48:00.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 >> v1:172.21.15.189:6789/0 conn(0x7fca9c09df10 legacy=0x7fca9c09a220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:00.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 >> v1:172.21.15.26:6789/0 conn(0x7fca9c09e740 legacy=0x7fca9c1d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:00.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:48:00.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 monclient: found mon.c 2024-09-16T14:48:00.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:00.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:00.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9c09b040 con 0x7fca9c09e2f0 2024-09-16T14:48:00.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient: _finish_auth 0 2024-09-16T14:48:00.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:48:00.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:00.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 5 monclient: authenticate success, global_id 114542 2024-09-16T14:48:00.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 10 monclient: _renew_subs 2024-09-16T14:48:00.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:00.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 1 -- 172.21.15.189:0/2934294373 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca9c13a150 con 0x7fca9c09e2f0 2024-09-16T14:48:00.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 10 monclient: _renew_subs 2024-09-16T14:48:00.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:00.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fcaa23a1640 1 -- 172.21.15.189:0/2934294373 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca9c13b470 con 0x7fca9c09e2f0 2024-09-16T14:48:00.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca900047f0 con 0x7fca9c09e2f0 2024-09-16T14:48:00.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:00.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fca90004b00 con 0x7fca9c09e2f0 2024-09-16T14:48:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca99ffb640 4 set_mon_vals no callback set 2024-09-16T14:48:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.565+0000 7fca98ff9640 10 monclient: dump: 2024-09-16T14:48:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:00.592 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:00.592 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:00.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.569+0000 7fcaa23a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:00.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.569+0000 7fcaa23a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:00.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.569+0000 7fcaa23a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:00.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.569+0000 7fcaa23a1640 1 -- 172.21.15.189:0/2934294373 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7c005050 con 0x7fca9c09e2f0 2024-09-16T14:48:00.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.569+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fca9000f5f0 con 0x7fca9c09e2f0 2024-09-16T14:48:00.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.569+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(136..136 src has 73..136) v4 ==== 6709+0+0 (unknown 2364296892 0 0) 0x7fca90043d60 con 0x7fca9c09e2f0 2024-09-16T14:48:00.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.569+0000 7fca9bfff640 10 monclient: get_auth_request con 0x7fca88036220 auth_method 0 2024-09-16T14:48:00.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.581+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fca90014490 con 0x7fca9c09e2f0 2024-09-16T14:48:00.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.581+0000 7fca98ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:00.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.581+0000 7fca98ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:00.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.745+0000 7fcaa23a1640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:48:00.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.745+0000 7fcaa23a1640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:48:00.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.745+0000 7fcaa23a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:00.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.745+0000 7fcaa23a1640 1 -- 172.21.15.189:0/2934294373 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fca7c005300 con 0x7fca9c09e2f0 2024-09-16T14:48:00.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.745+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v136) v1 ==== 71+0+9 (unknown 3509760808 0 1822970295) 0x7fca900142b0 con 0x7fca9c09e2f0 2024-09-16T14:48:00.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.745+0000 7fca98ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:48:00.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.745+0000 7fca98ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:00.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.841+0000 7fcaa23a1640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:48:00.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.841+0000 7fcaa23a1640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:48:00.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.841+0000 7fcaa23a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:00.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.841+0000 7fcaa23a1640 1 -- 172.21.15.189:0/2934294373 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fca7c005540 con 0x7fca9c09e2f0 2024-09-16T14:48:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.841+0000 7fca98ff9640 1 -- 172.21.15.189:0/2934294373 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v136) v1 ==== 71+0+9 (unknown 3509760808 0 1822970295) 0x7fca900142b0 con 0x7fca9c09e2f0 2024-09-16T14:48:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.841+0000 7fca98ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T14:48:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.841+0000 7fca98ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:00.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.845+0000 7fcaa23a1640 1 -- 172.21.15.189:0/2934294373 >> v1:172.21.15.189:6812/3196161582 conn(0x7fca88036220 legacy=0x7fca880386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:00.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.845+0000 7fcaa23a1640 10 monclient: shutdown 2024-09-16T14:48:00.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.845+0000 7fcaa23a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:00.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.845+0000 7fcaa23a1640 1 -- 172.21.15.189:0/2934294373 >> v1:172.21.15.26:6790/0 conn(0x7fca9c09e2f0 legacy=0x7fca9c09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:00.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.845+0000 7fcaa23a1640 1 -- 172.21.15.189:0/2934294373 shutdown_connections 2024-09-16T14:48:00.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.845+0000 7fcaa23a1640 1 -- 172.21.15.189:0/2934294373 >> 172.21.15.189:0/2934294373 conn(0x7fca9c0a3320 msgr2=0x7fca9c0a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:00.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.845+0000 7fcaa23a1640 1 -- 172.21.15.189:0/2934294373 shutdown_connections 2024-09-16T14:48:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.845+0000 7fcaa23a1640 1 -- 172.21.15.189:0/2934294373 wait complete. 2024-09-16T14:48:00.863 INFO:tasks.workunit.client.0.smithi189.stdout:(compat) 2024-09-16T14:48:00.863 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set rm-compat 2024-09-16T14:48:00.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fed1a640 10 monclient: get_monmap_and_config 2024-09-16T14:48:00.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fed1a640 10 monclient: build_initial_monmap 2024-09-16T14:48:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fed1a640 10 monclient: monmap: 2024-09-16T14:48:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:00.978860+0000 2024-09-16T14:48:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:00.978860+0000 2024-09-16T14:48:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:00.979 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:00.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fed1a640 1 Processor -- start 2024-09-16T14:48:00.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fed1a640 1 -- start start 2024-09-16T14:48:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fed1a640 10 monclient: init 2024-09-16T14:48:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fed1a640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fed1a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fed1a640 10 monclient(hunting): picked mon.noname-a con 0x7f61f809df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fed1a640 10 monclient(hunting): picked mon.noname-b con 0x7f61f809e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fed1a640 10 monclient(hunting): picked mon.noname-c con 0x7f61f809e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fed1a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f809ee20 con 0x7f61f809df10 2024-09-16T14:48:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fed1a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f809f100 con 0x7f61f809e740 2024-09-16T14:48:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fed1a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f809f3e0 con 0x7f61f809e2f0 2024-09-16T14:48:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fed1a640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fed1a640 10 monclient(hunting): authenticate will time out at 1263.094743s 2024-09-16T14:48:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fd2b7640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f61f809e740 legacy=0x7f61f813b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fcab6640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f61f809df10 0x7f61f809d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33816/0 (socket says 172.21.15.189:33816) 2024-09-16T14:48:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61fcab6640 1 -- 172.21.15.189:0/488503121 learned_addr learned my addr 172.21.15.189:0/488503121 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61f7fff640 1 --1- 172.21.15.189:0/488503121 >> v1:172.21.15.26:6790/0 conn(0x7f61f809e2f0 0x7f61f81378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:43804/0 (socket says 172.21.15.189:43804) 2024-09-16T14:48:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61f77fe640 1 -- 172.21.15.189:0/488503121 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013101972 0 0) 0x7f61f809ee20 con 0x7f61f809df10 2024-09-16T14:48:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61f77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61f77fe640 10 monclient(hunting): my global_id is 124236 2024-09-16T14:48:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61f77fe640 1 -- 172.21.15.189:0/488503121 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e0003530 con 0x7f61f809df10 2024-09-16T14:48:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61f77fe640 1 -- 172.21.15.189:0/488503121 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003798486 0 0) 0x7f61f809f3e0 con 0x7f61f809e2f0 2024-09-16T14:48:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61f77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61f77fe640 10 monclient(hunting): my global_id is 114545 2024-09-16T14:48:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.973+0000 7f61f77fe640 1 -- 172.21.15.189:0/488503121 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f809ee20 con 0x7f61f809e2f0 2024-09-16T14:48:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 1 -- 172.21.15.189:0/488503121 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3858013497 0 0) 0x7f61e0003530 con 0x7f61f809df10 2024-09-16T14:48:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 1 -- 172.21.15.189:0/488503121 >> v1:172.21.15.26:6790/0 conn(0x7f61f809e2f0 legacy=0x7f61f81378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 1 -- 172.21.15.189:0/488503121 >> v1:172.21.15.189:6789/0 conn(0x7f61f809e740 legacy=0x7f61f813b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:48:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 1 monclient: found mon.noname-a 2024-09-16T14:48:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 1 -- 172.21.15.189:0/488503121 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f809c690 con 0x7f61f809df10 2024-09-16T14:48:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 10 monclient: _finish_auth 0 2024-09-16T14:48:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:48:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 1 -- 172.21.15.189:0/488503121 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f61e8001620 con 0x7f61f809df10 2024-09-16T14:48:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 5 monclient: authenticate success, global_id 124236 2024-09-16T14:48:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 10 monclient: dump: 2024-09-16T14:48:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:00.987 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 1 -- 172.21.15.189:0/488503121 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61e8001900 con 0x7f61f809df10 2024-09-16T14:48:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 4 set_mon_vals no callback set 2024-09-16T14:48:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 1 -- 172.21.15.189:0/488503121 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f61e8001c90 con 0x7f61f809df10 2024-09-16T14:48:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f77fe640 10 monclient: dump: 2024-09-16T14:48:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:00.988 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 10 monclient: shutdown 2024-09-16T14:48:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 1 -- 172.21.15.189:0/488503121 >> v1:172.21.15.26:6789/0 conn(0x7f61f809df10 legacy=0x7f61f809d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 1 -- 172.21.15.189:0/488503121 shutdown_connections 2024-09-16T14:48:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 1 -- 172.21.15.189:0/488503121 >> 172.21.15.189:0/488503121 conn(0x7f61f80a3320 msgr2=0x7f61f809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 1 -- 172.21.15.189:0/488503121 shutdown_connections 2024-09-16T14:48:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 1 -- 172.21.15.189:0/488503121 wait complete. 2024-09-16T14:48:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 10 monclient: build_initial_monmap 2024-09-16T14:48:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 10 monclient: monmap: 2024-09-16T14:48:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:00.989 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:00.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 1 Processor -- start 2024-09-16T14:48:00.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 1 -- start start 2024-09-16T14:48:00.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 10 monclient: init 2024-09-16T14:48:00.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:00.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:00.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 10 monclient(hunting): picked mon.noname-b con 0x7f61f809df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:00.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 10 monclient(hunting): picked mon.noname-a con 0x7f61f809e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:00.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 10 monclient(hunting): picked mon.noname-c con 0x7f61f809e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:00.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f806f9e0 con 0x7f61f809e2f0 2024-09-16T14:48:00.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fcab6640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f61f809df10 legacy=0x7f61f811f830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:00.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f806cf00 con 0x7f61f809df10 2024-09-16T14:48:00.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f81231b0 con 0x7f61f809e740 2024-09-16T14:48:00.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:00.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61fed1a640 10 monclient(hunting): authenticate will time out at 1263.098499s 2024-09-16T14:48:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f7fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f61f809e2f0 0x7f61f811ff40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33824/0 (socket says 172.21.15.189:33824) 2024-09-16T14:48:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f7fff640 1 -- 172.21.15.189:0/2917265680 learned_addr learned my addr 172.21.15.189:0/2917265680 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561629190 0 0) 0x7f61f806f9e0 con 0x7f61f809e2f0 2024-09-16T14:48:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 10 monclient(hunting): my global_id is 124239 2024-09-16T14:48:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e4003530 con 0x7f61f809e2f0 2024-09-16T14:48:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183966059 0 0) 0x7f61f81231b0 con 0x7f61f809e740 2024-09-16T14:48:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 10 monclient(hunting): my global_id is 114548 2024-09-16T14:48:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f806f9e0 con 0x7f61f809e740 2024-09-16T14:48:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1157777414 0 0) 0x7f61e4003530 con 0x7f61f809e2f0 2024-09-16T14:48:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61f81231b0 con 0x7f61f809e2f0 2024-09-16T14:48:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f61ec001920 con 0x7f61f809e2f0 2024-09-16T14:48:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 10 monclient(hunting): dump: 2024-09-16T14:48:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:00.997 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3512954942 0 0) 0x7f61f806f9e0 con 0x7f61f809e740 2024-09-16T14:48:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61e4003530 con 0x7f61f809e740 2024-09-16T14:48:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f61f0004340 con 0x7f61f809e740 2024-09-16T14:48:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.977+0000 7f61f57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 10 monclient(hunting): dump: 2024-09-16T14:48:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:00.998 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3017568804 0 0) 0x7f61f81231b0 con 0x7f61f809e2f0 2024-09-16T14:48:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 >> v1:172.21.15.26:6790/0 conn(0x7f61f809e740 legacy=0x7f61f81cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 >> v1:172.21.15.189:6789/0 conn(0x7f61f809df10 legacy=0x7f61f811f830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 10 monclient: _finish_hunting 0 2024-09-16T14:48:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 1 monclient: found mon.a 2024-09-16T14:48:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f8120650 con 0x7f61f809e2f0 2024-09-16T14:48:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 10 monclient: _finish_auth 0 2024-09-16T14:48:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 10 monclient: _check_auth_tickets 2024-09-16T14:48:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61fed1a640 5 monclient: authenticate success, global_id 124239 2024-09-16T14:48:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61fed1a640 10 monclient: _renew_subs 2024-09-16T14:48:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61fed1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61fed1a640 1 -- 172.21.15.189:0/2917265680 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f8121fe0 con 0x7f61f809e2f0 2024-09-16T14:48:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61fed1a640 10 monclient: _renew_subs 2024-09-16T14:48:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61fed1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61fed1a640 1 -- 172.21.15.189:0/2917265680 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f81237d0 con 0x7f61f809e2f0 2024-09-16T14:48:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61ec0047f0 con 0x7f61f809e2f0 2024-09-16T14:48:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f67fc640 4 set_mon_vals no callback set 2024-09-16T14:48:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f61ec004b00 con 0x7f61f809e2f0 2024-09-16T14:48:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 10 monclient: dump: 2024-09-16T14:48:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:01.001 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:01.001 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:01.001 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:01.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61fed1a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:01.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61fed1a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:01.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61fed1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:01.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61fed1a640 1 -- 172.21.15.189:0/2917265680 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d8005050 con 0x7f61f809e2f0 2024-09-16T14:48:01.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f61ec00f5f0 con 0x7f61f809e2f0 2024-09-16T14:48:01.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(136..136 src has 73..136) v4 ==== 6709+0+0 (unknown 2364296892 0 0) 0x7f61ec043db0 con 0x7f61f809e2f0 2024-09-16T14:48:01.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.981+0000 7f61fcab6640 10 monclient: get_auth_request con 0x7f61e4036200 auth_method 0 2024-09-16T14:48:01.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.993+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f61ec0144e0 con 0x7f61f809e2f0 2024-09-16T14:48:01.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.993+0000 7f61f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:01.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:00.993+0000 7f61f57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:01.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:01.165+0000 7f61fed1a640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:48:01.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:01.165+0000 7f61fed1a640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:48:01.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:01.165+0000 7f61fed1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:01.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:01.165+0000 7f61fed1a640 1 -- 172.21.15.189:0/2917265680 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f61d8005300 con 0x7f61f809e2f0 2024-09-16T14:48:01.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:01.177+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v137) v1 ==== 78+0+0 (unknown 2298157204 0 0) 0x7f61ec014300 con 0x7f61f809e2f0 2024-09-16T14:48:01.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:01.177+0000 7f61f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:48:01.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:01.177+0000 7f61f57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:01.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:01.273+0000 7f61fed1a640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:48:01.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:01.273+0000 7f61fed1a640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:48:01.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:01.273+0000 7f61fed1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:01.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:01.273+0000 7f61fed1a640 1 -- 172.21.15.189:0/2917265680 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f61d8005a40 con 0x7f61f809e2f0 2024-09-16T14:48:02.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.189+0000 7f61f57fa640 1 -- 172.21.15.189:0/2917265680 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v138) v1 ==== 78+0+0 (unknown 1253623547 0 0) 0x7f61ec015040 con 0x7f61f809e2f0 2024-09-16T14:48:02.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.189+0000 7f61f57fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:48:02.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.189+0000 7f61f57fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:02.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.193+0000 7f61fed1a640 1 -- 172.21.15.189:0/2917265680 >> v1:172.21.15.189:6812/3196161582 conn(0x7f61e4036200 legacy=0x7f61e40386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:02.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.193+0000 7f61fed1a640 10 monclient: shutdown 2024-09-16T14:48:02.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.193+0000 7f61fed1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:02.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.193+0000 7f61fed1a640 1 -- 172.21.15.189:0/2917265680 >> v1:172.21.15.26:6789/0 conn(0x7f61f809e2f0 legacy=0x7f61f811ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:02.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.193+0000 7f61fed1a640 1 -- 172.21.15.189:0/2917265680 shutdown_connections 2024-09-16T14:48:02.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.193+0000 7f61fed1a640 1 -- 172.21.15.189:0/2917265680 >> 172.21.15.189:0/2917265680 conn(0x7f61f80a3320 msgr2=0x7f61f806ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:02.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.193+0000 7f61fed1a640 1 -- 172.21.15.189:0/2917265680 shutdown_connections 2024-09-16T14:48:02.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.193+0000 7f61fed1a640 1 -- 172.21.15.189:0/2917265680 wait complete. 2024-09-16T14:48:02.219 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create cool 2 2024-09-16T14:48:02.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe792475640 10 monclient: get_monmap_and_config 2024-09-16T14:48:02.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe792475640 10 monclient: build_initial_monmap 2024-09-16T14:48:02.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe792475640 10 monclient: monmap: 2024-09-16T14:48:02.342 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:02.342 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:02.342 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:02.341906+0000 2024-09-16T14:48:02.342 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:02.341906+0000 2024-09-16T14:48:02.342 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:02.342 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:02.342 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:02.342 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:02.342 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:02.342 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:02.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe792475640 1 Processor -- start 2024-09-16T14:48:02.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe792475640 1 -- start start 2024-09-16T14:48:02.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe792475640 10 monclient: init 2024-09-16T14:48:02.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe792475640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:02.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe792475640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:02.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe792475640 10 monclient(hunting): picked mon.noname-b con 0x7fe78c09df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:02.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe792475640 10 monclient(hunting): picked mon.noname-a con 0x7fe78c09e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:02.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe792475640 10 monclient(hunting): picked mon.noname-c con 0x7fe78c09e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:02.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe792475640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78c09ee20 con 0x7fe78c09e2f0 2024-09-16T14:48:02.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe792475640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78c09f100 con 0x7fe78c09e740 2024-09-16T14:48:02.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe792475640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78c09f3e0 con 0x7fe78c09df10 2024-09-16T14:48:02.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe792475640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:02.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe792475640 10 monclient(hunting): authenticate will time out at 1264.457573s 2024-09-16T14:48:02.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe790a12640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fe78c09e740 legacy=0x7fe78c13b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:02.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe78b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe78c09e2f0 0x7fe78c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33838/0 (socket says 172.21.15.189:33838) 2024-09-16T14:48:02.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.337+0000 7fe78b7fe640 1 -- 172.21.15.189:0/3573326255 learned_addr learned my addr 172.21.15.189:0/3573326255 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:02.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 1 -- 172.21.15.189:0/3573326255 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166759966 0 0) 0x7fe78c09f3e0 con 0x7fe78c09df10 2024-09-16T14:48:02.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:02.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:02.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:02.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 10 monclient(hunting): my global_id is 114551 2024-09-16T14:48:02.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 1 -- 172.21.15.189:0/3573326255 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe774003530 con 0x7fe78c09df10 2024-09-16T14:48:02.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 1 -- 172.21.15.189:0/3573326255 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1940996156 0 0) 0x7fe774003530 con 0x7fe78c09df10 2024-09-16T14:48:02.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 1 -- 172.21.15.189:0/3573326255 >> v1:172.21.15.189:6789/0 conn(0x7fe78c09e740 legacy=0x7fe78c13b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:02.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 1 -- 172.21.15.189:0/3573326255 >> v1:172.21.15.26:6789/0 conn(0x7fe78c09e2f0 legacy=0x7fe78c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:02.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:02.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 1 monclient: found mon.noname-b 2024-09-16T14:48:02.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:02.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:48:02.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 1 -- 172.21.15.189:0/3573326255 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78c09c690 con 0x7fe78c09df10 2024-09-16T14:48:02.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 10 monclient: _finish_auth 0 2024-09-16T14:48:02.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:02.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:02.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 1 -- 172.21.15.189:0/3573326255 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe77c001620 con 0x7fe78c09df10 2024-09-16T14:48:02.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:02.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:02.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 10 monclient: dump: 2024-09-16T14:48:02.354 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:02.354 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:02.354 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:02.355 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:02.355 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:02.355 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:02.355 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:02.355 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:02.355 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:02.355 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:02.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 5 monclient: authenticate success, global_id 114551 2024-09-16T14:48:02.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:02.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 1 -- 172.21.15.189:0/3573326255 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe77c001840 con 0x7fe78c09df10 2024-09-16T14:48:02.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:02.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 1 -- 172.21.15.189:0/3573326255 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe77c001c10 con 0x7fe78c09df10 2024-09-16T14:48:02.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:02.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:02.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78affd640 10 monclient: dump: 2024-09-16T14:48:02.356 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:02.356 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:02.356 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:02.356 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:02.356 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:02.356 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:02.356 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:02.356 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:02.356 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:02.356 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:02.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:02.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 4 set_mon_vals no callback set 2024-09-16T14:48:02.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 10 monclient: shutdown 2024-09-16T14:48:02.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:02.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 1 -- 172.21.15.189:0/3573326255 >> v1:172.21.15.26:6790/0 conn(0x7fe78c09df10 legacy=0x7fe78c09d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:02.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 1 -- 172.21.15.189:0/3573326255 shutdown_connections 2024-09-16T14:48:02.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 1 -- 172.21.15.189:0/3573326255 >> 172.21.15.189:0/3573326255 conn(0x7fe78c0a3320 msgr2=0x7fe78c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:02.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 1 -- 172.21.15.189:0/3573326255 shutdown_connections 2024-09-16T14:48:02.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 1 -- 172.21.15.189:0/3573326255 wait complete. 2024-09-16T14:48:02.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 10 monclient: build_initial_monmap 2024-09-16T14:48:02.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 10 monclient: monmap: 2024-09-16T14:48:02.357 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:02.357 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:02.357 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:02.357 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:02.358 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:02.358 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:02.358 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:02.358 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:02.358 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:02.358 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:02.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 1 Processor -- start 2024-09-16T14:48:02.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 1 -- start start 2024-09-16T14:48:02.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 10 monclient: init 2024-09-16T14:48:02.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:02.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:02.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 10 monclient(hunting): picked mon.noname-c con 0x7fe78c09df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:02.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 10 monclient(hunting): picked mon.noname-b con 0x7fe78c09e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:02.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 10 monclient(hunting): picked mon.noname-a con 0x7fe78c09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:02.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78c13bd50 con 0x7fe78c09e740 2024-09-16T14:48:02.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78c06fa80 con 0x7fe78c09e2f0 2024-09-16T14:48:02.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78c06cf70 con 0x7fe78c09df10 2024-09-16T14:48:02.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:02.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78b7fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fe78c09e2f0 legacy=0x7fe78c1cec10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:02.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe792475640 10 monclient(hunting): authenticate will time out at 1264.463554s 2024-09-16T14:48:02.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe790a12640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe78c09e740 0x7fe78c1d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33850/0 (socket says 172.21.15.189:33850) 2024-09-16T14:48:02.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe78bfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe78c09df10 0x7fe78c131fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:43832/0 (socket says 172.21.15.189:43832) 2024-09-16T14:48:02.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.341+0000 7fe790a12640 1 -- 172.21.15.189:0/2565867534 learned_addr learned my addr 172.21.15.189:0/2565867534 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:02.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128606261 0 0) 0x7fe78c13bd50 con 0x7fe78c09e740 2024-09-16T14:48:02.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:02.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:02.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:02.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient(hunting): my global_id is 124242 2024-09-16T14:48:02.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe778003530 con 0x7fe78c09e740 2024-09-16T14:48:02.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901801015 0 0) 0x7fe78c06cf70 con 0x7fe78c09df10 2024-09-16T14:48:02.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:02.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:02.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:02.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient(hunting): my global_id is 114554 2024-09-16T14:48:02.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe78c13bd50 con 0x7fe78c09df10 2024-09-16T14:48:02.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3042911115 0 0) 0x7fe778003530 con 0x7fe78c09e740 2024-09-16T14:48:02.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe78c06cf70 con 0x7fe78c09e740 2024-09-16T14:48:02.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe7840043c0 con 0x7fe78c09e740 2024-09-16T14:48:02.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:02.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:02.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient(hunting): dump: 2024-09-16T14:48:02.361 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:02.361 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:02.361 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:02.361 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:02.361 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:02.361 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:02.362 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:02.362 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:02.362 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:02.362 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:02.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:02.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3215656796 0 0) 0x7fe78c13bd50 con 0x7fe78c09df10 2024-09-16T14:48:02.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe778003530 con 0x7fe78c09df10 2024-09-16T14:48:02.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe77c001b50 con 0x7fe78c09df10 2024-09-16T14:48:02.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:02.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:02.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient(hunting): dump: 2024-09-16T14:48:02.363 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:02.363 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:02.363 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:02.363 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:02.363 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:02.363 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:02.363 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:02.363 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:02.363 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:02.363 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:02.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1863103847 0 0) 0x7fe78c06cf70 con 0x7fe78c09e740 2024-09-16T14:48:02.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 >> v1:172.21.15.26:6790/0 conn(0x7fe78c09df10 legacy=0x7fe78c131fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:02.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 >> v1:172.21.15.189:6789/0 conn(0x7fe78c09e2f0 legacy=0x7fe78c1cec10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:02.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:48:02.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 monclient: found mon.a 2024-09-16T14:48:02.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:02.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:02.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78c06b380 con 0x7fe78c09e740 2024-09-16T14:48:02.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient: _finish_auth 0 2024-09-16T14:48:02.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:48:02.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:02.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe792475640 5 monclient: authenticate success, global_id 124242 2024-09-16T14:48:02.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe792475640 10 monclient: _renew_subs 2024-09-16T14:48:02.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe792475640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:02.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe792475640 1 -- 172.21.15.189:0/2565867534 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe78c09be20 con 0x7fe78c09e740 2024-09-16T14:48:02.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe792475640 10 monclient: _renew_subs 2024-09-16T14:48:02.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe792475640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:02.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe792475640 1 -- 172.21.15.189:0/2565867534 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe78c09b200 con 0x7fe78c09e740 2024-09-16T14:48:02.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7840045e0 con 0x7fe78c09e740 2024-09-16T14:48:02.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:02.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe7840048f0 con 0x7fe78c09e740 2024-09-16T14:48:02.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:02.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe789ffb640 4 set_mon_vals no callback set 2024-09-16T14:48:02.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:02.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 10 monclient: dump: 2024-09-16T14:48:02.365 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:02.366 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:02.366 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:02.366 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:02.366 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:02.366 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:02.366 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:02.366 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:02.366 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:02.366 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:02.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe792475640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:02.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe792475640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:02.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe792475640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:02.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe792475640 1 -- 172.21.15.189:0/2565867534 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76c005050 con 0x7fe78c09e740 2024-09-16T14:48:02.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe784004ad0 con 0x7fe78c09e740 2024-09-16T14:48:02.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(138..138 src has 73..138) v4 ==== 6377+0+0 (unknown 1152112757 0 0) 0x7fe784043db0 con 0x7fe78c09e740 2024-09-16T14:48:02.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.345+0000 7fe78bfff640 10 monclient: get_auth_request con 0x7fe7780362c0 auth_method 0 2024-09-16T14:48:02.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.357+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe7840145d0 con 0x7fe78c09e740 2024-09-16T14:48:02.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.357+0000 7fe788ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:02.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.357+0000 7fe788ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:02.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.509+0000 7fe792475640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T14:48:02.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.509+0000 7fe792475640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T14:48:02.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.509+0000 7fe792475640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:02.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:02.509+0000 7fe792475640 1 -- 172.21.15.189:0/2565867534 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "cool", "pg_num": 2} v 0) v1 -- 0x7fe76c005b10 con 0x7fe78c09e740 2024-09-16T14:48:03.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.349+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}]=0 pool 'cool' created v139) v1 ==== 111+0+0 (unknown 4168928295 0 0) 0x7fe7840143f0 con 0x7fe78c09e740 2024-09-16T14:48:03.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.349+0000 7fe788ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T14:48:03.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.349+0000 7fe788ff9640 10 monclient: _finish_command 2 = system:0 pool 'cool' created 2024-09-16T14:48:03.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.441+0000 7fe792475640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T14:48:03.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.441+0000 7fe792475640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T14:48:03.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.441+0000 7fe792475640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:03.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.441+0000 7fe792475640 1 -- 172.21.15.189:0/2565867534 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "cool", "pg_num": 2} v 0) v1 -- 0x7fe76c0010a0 con 0x7fe78c09e740 2024-09-16T14:48:03.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.445+0000 7fe788ff9640 1 -- 172.21.15.189:0/2565867534 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}]=0 pool 'cool' already exists v139) v1 ==== 118+0+0 (unknown 1578187363 0 0) 0x7fe784015130 con 0x7fe78c09e740 2024-09-16T14:48:03.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.445+0000 7fe788ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T14:48:03.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.445+0000 7fe788ff9640 10 monclient: _finish_command 3 = system:0 pool 'cool' already exists 2024-09-16T14:48:03.451 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'cool' already exists 2024-09-16T14:48:03.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.445+0000 7fe792475640 1 -- 172.21.15.189:0/2565867534 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe7780362c0 legacy=0x7fe778038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:03.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.445+0000 7fe792475640 10 monclient: shutdown 2024-09-16T14:48:03.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.445+0000 7fe792475640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:03.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.445+0000 7fe792475640 1 -- 172.21.15.189:0/2565867534 >> v1:172.21.15.26:6789/0 conn(0x7fe78c09e740 legacy=0x7fe78c1d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:03.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.449+0000 7fe792475640 1 -- 172.21.15.189:0/2565867534 shutdown_connections 2024-09-16T14:48:03.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.449+0000 7fe792475640 1 -- 172.21.15.189:0/2565867534 >> 172.21.15.189:0/2565867534 conn(0x7fe78c0a3320 msgr2=0x7fe78c06e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:03.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.449+0000 7fe792475640 1 -- 172.21.15.189:0/2565867534 shutdown_connections 2024-09-16T14:48:03.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.449+0000 7fe792475640 1 -- 172.21.15.189:0/2565867534 wait complete. 2024-09-16T14:48:03.468 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create cold 2 2024-09-16T14:48:03.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd564f9a640 10 monclient: get_monmap_and_config 2024-09-16T14:48:03.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd564f9a640 10 monclient: build_initial_monmap 2024-09-16T14:48:03.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd564f9a640 10 monclient: monmap: 2024-09-16T14:48:03.584 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:03.584 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:03.584 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:03.583859+0000 2024-09-16T14:48:03.584 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:03.583859+0000 2024-09-16T14:48:03.584 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:03.584 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:03.584 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:03.584 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:03.584 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:03.584 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:03.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd564f9a640 1 Processor -- start 2024-09-16T14:48:03.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd564f9a640 1 -- start start 2024-09-16T14:48:03.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd564f9a640 10 monclient: init 2024-09-16T14:48:03.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd564f9a640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:03.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd564f9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:03.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd564f9a640 10 monclient(hunting): picked mon.noname-b con 0x7fd56009df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:03.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd564f9a640 10 monclient(hunting): picked mon.noname-a con 0x7fd56009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:03.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd564f9a640 10 monclient(hunting): picked mon.noname-c con 0x7fd56009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:03.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd564f9a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56009ee20 con 0x7fd56009e2f0 2024-09-16T14:48:03.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd564f9a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56009f100 con 0x7fd56009e740 2024-09-16T14:48:03.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd564f9a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56009f3e0 con 0x7fd56009df10 2024-09-16T14:48:03.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd564f9a640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:03.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd564f9a640 10 monclient(hunting): authenticate will time out at 1265.699397s 2024-09-16T14:48:03.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd55ed9d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fd56009e740 legacy=0x7fd56013b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:03.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd55dd9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd56009e2f0 0x7fd5601378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33862/0 (socket says 172.21.15.189:33862) 2024-09-16T14:48:03.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.577+0000 7fd55dd9b640 1 -- 172.21.15.189:0/3322611882 learned_addr learned my addr 172.21.15.189:0/3322611882 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:03.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 1 -- 172.21.15.189:0/3322611882 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2362789177 0 0) 0x7fd56009ee20 con 0x7fd56009e2f0 2024-09-16T14:48:03.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient(hunting): _negotiate 2024-09-16T14:48:03.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:03.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:03.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient(hunting): my global_id is 124245 2024-09-16T14:48:03.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 1 -- 172.21.15.189:0/3322611882 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd54c003530 con 0x7fd56009e2f0 2024-09-16T14:48:03.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 1 -- 172.21.15.189:0/3322611882 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926972198 0 0) 0x7fd56009f3e0 con 0x7fd56009df10 2024-09-16T14:48:03.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient(hunting): _negotiate 2024-09-16T14:48:03.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:03.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:03.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient(hunting): my global_id is 114557 2024-09-16T14:48:03.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 1 -- 172.21.15.189:0/3322611882 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd56009ee20 con 0x7fd56009df10 2024-09-16T14:48:03.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 1 -- 172.21.15.189:0/3322611882 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2350266034 0 0) 0x7fd54c003530 con 0x7fd56009e2f0 2024-09-16T14:48:03.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 1 -- 172.21.15.189:0/3322611882 >> v1:172.21.15.26:6790/0 conn(0x7fd56009df10 legacy=0x7fd56009d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:03.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 1 -- 172.21.15.189:0/3322611882 >> v1:172.21.15.189:6789/0 conn(0x7fd56009e740 legacy=0x7fd56013b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:03.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient: _finish_hunting 0 2024-09-16T14:48:03.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 1 monclient: found mon.noname-a 2024-09-16T14:48:03.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:03.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:03.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 1 -- 172.21.15.189:0/3322611882 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56009c690 con 0x7fd56009e2f0 2024-09-16T14:48:03.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient: _finish_auth 0 2024-09-16T14:48:03.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient: _check_auth_tickets 2024-09-16T14:48:03.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:03.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 1 -- 172.21.15.189:0/3322611882 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd548001620 con 0x7fd56009e2f0 2024-09-16T14:48:03.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:03.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:03.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient: dump: 2024-09-16T14:48:03.590 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:03.590 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:03.591 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:03.591 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:03.591 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:03.591 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:03.591 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:03.591 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:03.591 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:03.591 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:03.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 5 monclient: authenticate success, global_id 124245 2024-09-16T14:48:03.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:03.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 1 -- 172.21.15.189:0/3322611882 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd548001840 con 0x7fd56009e2f0 2024-09-16T14:48:03.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:03.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:03.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 4 set_mon_vals no callback set 2024-09-16T14:48:03.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 1 -- 172.21.15.189:0/3322611882 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd548001bd0 con 0x7fd56009e2f0 2024-09-16T14:48:03.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:03.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:03.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55d59a640 10 monclient: dump: 2024-09-16T14:48:03.592 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:03.592 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:03.592 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:03.592 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:03.592 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:03.592 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:03.592 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:03.592 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:03.592 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:03.593 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:03.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 10 monclient: shutdown 2024-09-16T14:48:03.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:03.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 1 -- 172.21.15.189:0/3322611882 >> v1:172.21.15.26:6789/0 conn(0x7fd56009e2f0 legacy=0x7fd5601378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:03.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 1 -- 172.21.15.189:0/3322611882 shutdown_connections 2024-09-16T14:48:03.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 1 -- 172.21.15.189:0/3322611882 >> 172.21.15.189:0/3322611882 conn(0x7fd5600a3320 msgr2=0x7fd56009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:03.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 1 -- 172.21.15.189:0/3322611882 shutdown_connections 2024-09-16T14:48:03.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 1 -- 172.21.15.189:0/3322611882 wait complete. 2024-09-16T14:48:03.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 10 monclient: build_initial_monmap 2024-09-16T14:48:03.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 10 monclient: monmap: 2024-09-16T14:48:03.598 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:03.598 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:03.598 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:03.598 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:03.598 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:03.598 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:03.598 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:03.598 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:03.598 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:03.599 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:03.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 1 Processor -- start 2024-09-16T14:48:03.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 1 -- start start 2024-09-16T14:48:03.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 10 monclient: init 2024-09-16T14:48:03.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:03.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:03.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 10 monclient(hunting): picked mon.noname-c con 0x7fd56009df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:03.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 10 monclient(hunting): picked mon.noname-b con 0x7fd56009e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:03.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 10 monclient(hunting): picked mon.noname-a con 0x7fd56009e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:03.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56006cf70 con 0x7fd56009e740 2024-09-16T14:48:03.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56013b9c0 con 0x7fd56009e2f0 2024-09-16T14:48:03.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55dd9b640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fd56009e2f0 legacy=0x7fd56009a930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:03.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56009b040 con 0x7fd56009df10 2024-09-16T14:48:03.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:03.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd564f9a640 10 monclient(hunting): authenticate will time out at 1265.703357s 2024-09-16T14:48:03.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55ed9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd56009e740 0x7fd5601d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33876/0 (socket says 172.21.15.189:33876) 2024-09-16T14:48:03.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.581+0000 7fd55ed9d640 1 -- 172.21.15.189:0/1155283005 learned_addr learned my addr 172.21.15.189:0/1155283005 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:03.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143063989 0 0) 0x7fd56006cf70 con 0x7fd56009e740 2024-09-16T14:48:03.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:03.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:03.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:03.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient(hunting): my global_id is 124248 2024-09-16T14:48:03.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd540003530 con 0x7fd56009e740 2024-09-16T14:48:03.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018200990 0 0) 0x7fd56009b040 con 0x7fd56009df10 2024-09-16T14:48:03.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:03.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:03.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:03.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient(hunting): my global_id is 114560 2024-09-16T14:48:03.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd56006cf70 con 0x7fd56009df10 2024-09-16T14:48:03.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3790796031 0 0) 0x7fd540003530 con 0x7fd56009e740 2024-09-16T14:48:03.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd56009b040 con 0x7fd56009e740 2024-09-16T14:48:03.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd558004040 con 0x7fd56009e740 2024-09-16T14:48:03.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:03.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:03.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient(hunting): dump: 2024-09-16T14:48:03.601 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:03.601 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:03.602 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:03.602 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:03.602 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:03.602 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:03.602 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:03.602 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:03.602 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:03.602 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:03.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:03.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1573385799 0 0) 0x7fd56006cf70 con 0x7fd56009df10 2024-09-16T14:48:03.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd540003530 con 0x7fd56009df10 2024-09-16T14:48:03.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd550001c40 con 0x7fd56009df10 2024-09-16T14:48:03.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:03.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:03.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient(hunting): dump: 2024-09-16T14:48:03.603 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:03.603 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:03.603 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:03.603 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:03.603 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:03.603 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:03.603 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:03.603 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:03.603 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:03.603 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:03.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3509143917 0 0) 0x7fd56009b040 con 0x7fd56009e740 2024-09-16T14:48:03.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 >> v1:172.21.15.26:6790/0 conn(0x7fd56009df10 legacy=0x7fd56009a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:03.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 >> v1:172.21.15.189:6789/0 conn(0x7fd56009e2f0 legacy=0x7fd56009a930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:03.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:03.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 monclient: found mon.a 2024-09-16T14:48:03.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:03.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:03.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56006fa60 con 0x7fd56009e740 2024-09-16T14:48:03.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient: _finish_auth 0 2024-09-16T14:48:03.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:03.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:03.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd564f9a640 5 monclient: authenticate success, global_id 124248 2024-09-16T14:48:03.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd564f9a640 10 monclient: _renew_subs 2024-09-16T14:48:03.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd564f9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:03.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd564f9a640 1 -- 172.21.15.189:0/1155283005 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd56006b380 con 0x7fd56009e740 2024-09-16T14:48:03.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd564f9a640 10 monclient: _renew_subs 2024-09-16T14:48:03.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd564f9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:03.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd564f9a640 1 -- 172.21.15.189:0/1155283005 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd56013a570 con 0x7fd56009e740 2024-09-16T14:48:03.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd558004260 con 0x7fd56009e740 2024-09-16T14:48:03.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:03.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd558004570 con 0x7fd56009e740 2024-09-16T14:48:03.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:03.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd557fff640 4 set_mon_vals no callback set 2024-09-16T14:48:03.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:03.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 10 monclient: dump: 2024-09-16T14:48:03.606 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:03.606 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:03.606 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:03.606 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:03.606 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:03.606 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:03.606 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:03.606 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:03.606 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:03.606 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:03.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd564f9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:03.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd564f9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:03.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd564f9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:03.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd564f9a640 1 -- 172.21.15.189:0/1155283005 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd544005050 con 0x7fd56009e740 2024-09-16T14:48:03.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd558004750 con 0x7fd56009e740 2024-09-16T14:48:03.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(139..139 src has 73..139) v4 ==== 6720+0+0 (unknown 2349613177 0 0) 0x7fd558043b90 con 0x7fd56009e740 2024-09-16T14:48:03.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.585+0000 7fd55e59c640 10 monclient: get_auth_request con 0x7fd540036200 auth_method 0 2024-09-16T14:48:03.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.597+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd558014260 con 0x7fd56009e740 2024-09-16T14:48:03.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.597+0000 7fd556ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:03.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.597+0000 7fd556ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:03.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.761+0000 7fd564f9a640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T14:48:03.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.761+0000 7fd564f9a640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T14:48:03.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.761+0000 7fd564f9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:03.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:03.761+0000 7fd564f9a640 1 -- 172.21.15.189:0/1155283005 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) v1 -- 0x7fd544005b10 con 0x7fd56009e740 2024-09-16T14:48:04.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.353+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}]=0 pool 'cold' created v140) v1 ==== 111+0+0 (unknown 4259662596 0 0) 0x7fd558012900 con 0x7fd56009e740 2024-09-16T14:48:04.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.353+0000 7fd556ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T14:48:04.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.353+0000 7fd556ffd640 10 monclient: _finish_command 2 = system:0 pool 'cold' created 2024-09-16T14:48:04.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.449+0000 7fd564f9a640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T14:48:04.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.449+0000 7fd564f9a640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T14:48:04.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.449+0000 7fd564f9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:04.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.449+0000 7fd564f9a640 1 -- 172.21.15.189:0/1155283005 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) v1 -- 0x7fd5440010a0 con 0x7fd56009e740 2024-09-16T14:48:04.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.453+0000 7fd556ffd640 1 -- 172.21.15.189:0/1155283005 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}]=0 pool 'cold' already exists v140) v1 ==== 118+0+0 (unknown 4190153842 0 0) 0x7fd558014080 con 0x7fd56009e740 2024-09-16T14:48:04.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.453+0000 7fd556ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T14:48:04.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.453+0000 7fd556ffd640 10 monclient: _finish_command 3 = system:0 pool 'cold' already exists 2024-09-16T14:48:04.459 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'cold' already exists 2024-09-16T14:48:04.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.457+0000 7fd564f9a640 1 -- 172.21.15.189:0/1155283005 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd540036200 legacy=0x7fd5400386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:04.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.457+0000 7fd564f9a640 10 monclient: shutdown 2024-09-16T14:48:04.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.457+0000 7fd564f9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:04.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.457+0000 7fd564f9a640 1 -- 172.21.15.189:0/1155283005 >> v1:172.21.15.26:6789/0 conn(0x7fd56009e740 legacy=0x7fd5601d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:04.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.457+0000 7fd564f9a640 1 -- 172.21.15.189:0/1155283005 shutdown_connections 2024-09-16T14:48:04.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.457+0000 7fd564f9a640 1 -- 172.21.15.189:0/1155283005 >> 172.21.15.189:0/1155283005 conn(0x7fd5600a3320 msgr2=0x7fd56006e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:04.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.457+0000 7fd564f9a640 1 -- 172.21.15.189:0/1155283005 shutdown_connections 2024-09-16T14:48:04.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.457+0000 7fd564f9a640 1 -- 172.21.15.189:0/1155283005 wait complete. 2024-09-16T14:48:04.476 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set cold size 2 2024-09-16T14:48:04.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5103924640 10 monclient: get_monmap_and_config 2024-09-16T14:48:04.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5103924640 10 monclient: build_initial_monmap 2024-09-16T14:48:04.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5103924640 10 monclient: monmap: 2024-09-16T14:48:04.595 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:04.595 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:04.595 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:04.595126+0000 2024-09-16T14:48:04.595 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:04.595126+0000 2024-09-16T14:48:04.595 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:04.595 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:04.595 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:04.595 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:04.596 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:04.596 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:04.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5103924640 1 Processor -- start 2024-09-16T14:48:04.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5103924640 1 -- start start 2024-09-16T14:48:04.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5103924640 10 monclient: init 2024-09-16T14:48:04.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5103924640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:04.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5103924640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:04.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5103924640 10 monclient(hunting): picked mon.noname-b con 0x7f50fc0a1a90 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:04.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5103924640 10 monclient(hunting): picked mon.noname-a con 0x7f50fc0a2580 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:04.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5103924640 10 monclient(hunting): picked mon.noname-c con 0x7f50fc09fd40 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:04.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5103924640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50fc09df00 con 0x7f50fc0a2580 2024-09-16T14:48:04.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5103924640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50fc09f0a0 con 0x7f50fc09fd40 2024-09-16T14:48:04.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5103924640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50fc0a3040 con 0x7f50fc0a1a90 2024-09-16T14:48:04.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5103924640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:04.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5103924640 10 monclient(hunting): authenticate will time out at 1266.710973s 2024-09-16T14:48:04.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5101ec1640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f50fc09fd40 legacy=0x7f50fc13d520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:04.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5100ebf640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f50fc0a2580 0x7f50fc139df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33890/0 (socket says 172.21.15.189:33890) 2024-09-16T14:48:04.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f5100ebf640 1 -- 172.21.15.189:0/2205051434 learned_addr learned my addr 172.21.15.189:0/2205051434 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:04.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f50f3fff640 1 -- 172.21.15.189:0/2205051434 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125645188 0 0) 0x7f50fc0a3040 con 0x7f50fc0a1a90 2024-09-16T14:48:04.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f50f3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:04.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f50f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:04.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f50f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:04.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f50f3fff640 10 monclient(hunting): my global_id is 114563 2024-09-16T14:48:04.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f50f3fff640 1 -- 172.21.15.189:0/2205051434 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50e8003530 con 0x7f50fc0a1a90 2024-09-16T14:48:04.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f50f3fff640 1 -- 172.21.15.189:0/2205051434 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816600733 0 0) 0x7f50fc09df00 con 0x7f50fc0a2580 2024-09-16T14:48:04.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f50f3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:04.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f50f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:04.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f50f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:04.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f50f3fff640 10 monclient(hunting): my global_id is 124251 2024-09-16T14:48:04.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.589+0000 7f50f3fff640 1 -- 172.21.15.189:0/2205051434 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50fc0a3040 con 0x7f50fc0a2580 2024-09-16T14:48:04.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 1 -- 172.21.15.189:0/2205051434 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2184057795 0 0) 0x7f50e8003530 con 0x7f50fc0a1a90 2024-09-16T14:48:04.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 1 -- 172.21.15.189:0/2205051434 >> v1:172.21.15.189:6789/0 conn(0x7f50fc09fd40 legacy=0x7f50fc13d520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:04.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 1 -- 172.21.15.189:0/2205051434 >> v1:172.21.15.26:6789/0 conn(0x7f50fc0a2580 legacy=0x7f50fc139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:04.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:04.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 1 monclient: found mon.noname-b 2024-09-16T14:48:04.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:04.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:48:04.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 1 -- 172.21.15.189:0/2205051434 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50fc09eae0 con 0x7f50fc0a1a90 2024-09-16T14:48:04.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 10 monclient: _finish_auth 0 2024-09-16T14:48:04.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:04.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:04.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 1 -- 172.21.15.189:0/2205051434 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f50f8001620 con 0x7f50fc0a1a90 2024-09-16T14:48:04.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 5 monclient: authenticate success, global_id 114563 2024-09-16T14:48:04.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:04.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:04.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 10 monclient: dump: 2024-09-16T14:48:04.602 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:04.602 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:04.602 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:04.602 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:04.602 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:04.602 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:04.603 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:04.603 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:04.603 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:04.603 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:04.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:04.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 1 -- 172.21.15.189:0/2205051434 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50f8001890 con 0x7f50fc0a1a90 2024-09-16T14:48:04.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:04.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 1 -- 172.21.15.189:0/2205051434 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f50f8001c20 con 0x7f50fc0a1a90 2024-09-16T14:48:04.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:04.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:04.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f3fff640 10 monclient: dump: 2024-09-16T14:48:04.603 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:04.603 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:04.603 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:04.604 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:04.604 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:04.604 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:04.604 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:04.604 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:04.604 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:04.604 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:04.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:04.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 4 set_mon_vals no callback set 2024-09-16T14:48:04.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 10 monclient: shutdown 2024-09-16T14:48:04.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:04.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 1 -- 172.21.15.189:0/2205051434 >> v1:172.21.15.26:6790/0 conn(0x7f50fc0a1a90 legacy=0x7f50fc0a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:04.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 1 -- 172.21.15.189:0/2205051434 shutdown_connections 2024-09-16T14:48:04.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 1 -- 172.21.15.189:0/2205051434 >> 172.21.15.189:0/2205051434 conn(0x7f50fc0a7a70 msgr2=0x7f50fc0a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:04.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 1 -- 172.21.15.189:0/2205051434 shutdown_connections 2024-09-16T14:48:04.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 1 -- 172.21.15.189:0/2205051434 wait complete. 2024-09-16T14:48:04.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 10 monclient: build_initial_monmap 2024-09-16T14:48:04.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 10 monclient: monmap: 2024-09-16T14:48:04.609 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:04.609 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:04.609 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:04.609 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:04.609 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:04.609 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:04.609 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:04.609 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:04.609 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:04.609 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:04.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 1 Processor -- start 2024-09-16T14:48:04.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 1 -- start start 2024-09-16T14:48:04.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 10 monclient: init 2024-09-16T14:48:04.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:04.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:04.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 10 monclient(hunting): picked mon.noname-a con 0x7f50fc09fd40 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:04.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 10 monclient(hunting): picked mon.noname-c con 0x7f50fc0a1a90 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:04.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 10 monclient(hunting): picked mon.noname-b con 0x7f50fc0a2580 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:04.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50fc0a65a0 con 0x7f50fc09fd40 2024-09-16T14:48:04.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50fc09e280 con 0x7f50fc0a2580 2024-09-16T14:48:04.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50fc11f340 con 0x7f50fc0a1a90 2024-09-16T14:48:04.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:04.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5103924640 10 monclient(hunting): authenticate will time out at 1266.714807s 2024-09-16T14:48:04.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5101ec1640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f50fc0a2580 legacy=0x7f50fc1d1bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:04.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5100ebf640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f50fc0a1a90 0x7f50fc1ce490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:43878/0 (socket says 172.21.15.189:43878) 2024-09-16T14:48:04.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f5100ebf640 1 -- 172.21.15.189:0/3577096875 learned_addr learned my addr 172.21.15.189:0/3577096875 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:04.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026949711 0 0) 0x7f50fc11f340 con 0x7f50fc0a1a90 2024-09-16T14:48:04.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:04.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:04.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:04.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f1ffb640 10 monclient(hunting): my global_id is 114566 2024-09-16T14:48:04.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50e0003530 con 0x7f50fc0a1a90 2024-09-16T14:48:04.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169546104 0 0) 0x7f50fc0a65a0 con 0x7f50fc09fd40 2024-09-16T14:48:04.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:04.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:04.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:04.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f1ffb640 10 monclient(hunting): my global_id is 124254 2024-09-16T14:48:04.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50fc11f340 con 0x7f50fc09fd40 2024-09-16T14:48:04.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1596697240 0 0) 0x7f50e0003530 con 0x7f50fc0a1a90 2024-09-16T14:48:04.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50fc0a65a0 con 0x7f50fc0a1a90 2024-09-16T14:48:04.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f50ec0018c0 con 0x7f50fc0a1a90 2024-09-16T14:48:04.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.593+0000 7f50f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:04.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:04.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 10 monclient(hunting): dump: 2024-09-16T14:48:04.612 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:04.612 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:04.612 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:04.612 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:04.612 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:04.613 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:04.613 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:04.613 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:04.613 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:04.613 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:04.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:04.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2324470039 0 0) 0x7f50fc0a65a0 con 0x7f50fc0a1a90 2024-09-16T14:48:04.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 >> v1:172.21.15.189:6789/0 conn(0x7f50fc0a2580 legacy=0x7f50fc1d1bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:04.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 >> v1:172.21.15.26:6789/0 conn(0x7f50fc09fd40 legacy=0x7f50fc133d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:04.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:48:04.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 1 monclient: found mon.c 2024-09-16T14:48:04.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:04.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:04.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50fc134970 con 0x7f50fc0a1a90 2024-09-16T14:48:04.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:48:04.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:04.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:04.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f5103924640 5 monclient: authenticate success, global_id 114566 2024-09-16T14:48:04.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f5103924640 10 monclient: _renew_subs 2024-09-16T14:48:04.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f5103924640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:04.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f5103924640 1 -- 172.21.15.189:0/3577096875 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50fc11fef0 con 0x7f50fc0a1a90 2024-09-16T14:48:04.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f5103924640 10 monclient: _renew_subs 2024-09-16T14:48:04.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f5103924640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:04.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f5103924640 1 -- 172.21.15.189:0/3577096875 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50fc121410 con 0x7f50fc0a1a90 2024-09-16T14:48:04.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50ec0047f0 con 0x7f50fc0a1a90 2024-09-16T14:48:04.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:04.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f50ec004b80 con 0x7f50fc0a1a90 2024-09-16T14:48:04.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:04.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f2ffd640 4 set_mon_vals no callback set 2024-09-16T14:48:04.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:04.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 10 monclient: dump: 2024-09-16T14:48:04.615 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:04.615 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:04.615 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:04.616 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:04.616 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:04.616 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:04.616 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:04.616 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:04.616 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:04.616 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:04.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f5103924640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:04.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f5103924640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:04.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f5103924640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:04.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f5103924640 1 -- 172.21.15.189:0/3577096875 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50dc005050 con 0x7f50fc0a1a90 2024-09-16T14:48:04.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f50ec004d60 con 0x7f50fc0a1a90 2024-09-16T14:48:04.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(140..140 src has 73..140) v4 ==== 7063+0+0 (unknown 2076921590 0 0) 0x7f50ec043ed0 con 0x7f50fc0a1a90 2024-09-16T14:48:04.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.597+0000 7f51016c0640 10 monclient: get_auth_request con 0x7f50e00361b0 auth_method 0 2024-09-16T14:48:04.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.609+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f50ec0144a0 con 0x7f50fc0a1a90 2024-09-16T14:48:04.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.609+0000 7f50f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:04.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.609+0000 7f50f1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:04.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.769+0000 7f5103924640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T14:48:04.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.769+0000 7f5103924640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T14:48:04.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.769+0000 7f5103924640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:04.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:04.769+0000 7f5103924640 1 -- 172.21.15.189:0/3577096875 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"} v 0) v1 -- 0x7f50dc005b10 con 0x7f50fc0a1a90 2024-09-16T14:48:05.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:05.369+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}]=0 set pool 12 size to 2 v141) v1 ==== 124+0+0 (unknown 2576833613 0 0) 0x7f50ec012b40 con 0x7f50fc0a1a90 2024-09-16T14:48:05.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:05.369+0000 7f50f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T14:48:05.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:05.369+0000 7f50f1ffb640 10 monclient: _finish_command 2 = system:0 set pool 12 size to 2 2024-09-16T14:48:05.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:05.469+0000 7f5103924640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T14:48:05.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:05.469+0000 7f5103924640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T14:48:05.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:05.469+0000 7f5103924640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:05.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:05.469+0000 7f5103924640 1 -- 172.21.15.189:0/3577096875 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"} v 0) v1 -- 0x7f50dc0010a0 con 0x7f50fc0a1a90 2024-09-16T14:48:06.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.369+0000 7f50f1ffb640 1 -- 172.21.15.189:0/3577096875 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}]=0 set pool 12 size to 2 v142) v1 ==== 124+0+0 (unknown 95314700 0 0) 0x7f50ec012b40 con 0x7f50fc0a1a90 2024-09-16T14:48:06.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.369+0000 7f50f1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T14:48:06.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.369+0000 7f50f1ffb640 10 monclient: _finish_command 3 = system:0 set pool 12 size to 2 2024-09-16T14:48:06.375 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 12 size to 2 2024-09-16T14:48:06.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.373+0000 7f5103924640 1 -- 172.21.15.189:0/3577096875 >> v1:172.21.15.189:6812/3196161582 conn(0x7f50e00361b0 legacy=0x7f50e0038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:06.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.373+0000 7f5103924640 10 monclient: shutdown 2024-09-16T14:48:06.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.373+0000 7f5103924640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:06.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.373+0000 7f5103924640 1 -- 172.21.15.189:0/3577096875 >> v1:172.21.15.26:6790/0 conn(0x7f50fc0a1a90 legacy=0x7f50fc1ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:06.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.373+0000 7f5103924640 1 -- 172.21.15.189:0/3577096875 shutdown_connections 2024-09-16T14:48:06.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.373+0000 7f5103924640 1 -- 172.21.15.189:0/3577096875 >> 172.21.15.189:0/3577096875 conn(0x7f50fc0a7a70 msgr2=0x7f50fc06b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:06.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.373+0000 7f5103924640 1 -- 172.21.15.189:0/3577096875 shutdown_connections 2024-09-16T14:48:06.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.373+0000 7f5103924640 1 -- 172.21.15.189:0/3577096875 wait complete. 2024-09-16T14:48:06.396 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set create-compat 2024-09-16T14:48:06.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f601931c640 10 monclient: get_monmap_and_config 2024-09-16T14:48:06.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f601931c640 10 monclient: build_initial_monmap 2024-09-16T14:48:06.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f601931c640 10 monclient: monmap: 2024-09-16T14:48:06.515 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:06.515 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:06.515 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:06.515232+0000 2024-09-16T14:48:06.515 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:06.515232+0000 2024-09-16T14:48:06.515 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:06.515 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:06.515 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:06.516 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:06.516 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:06.516 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:06.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f601931c640 1 Processor -- start 2024-09-16T14:48:06.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f601931c640 1 -- start start 2024-09-16T14:48:06.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f601931c640 10 monclient: init 2024-09-16T14:48:06.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f601931c640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f601931c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f601931c640 10 monclient(hunting): picked mon.noname-c con 0x7f60140a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f601931c640 10 monclient(hunting): picked mon.noname-a con 0x7f60140a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f601931c640 10 monclient(hunting): picked mon.noname-b con 0x7f601413b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f601931c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60140a30b0 con 0x7f60140a25f0 2024-09-16T14:48:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f601931c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60140a2e30 con 0x7f60140a1b00 2024-09-16T14:48:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f601931c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f601409eb50 con 0x7f601413b430 2024-09-16T14:48:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f601931c640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f601931c640 10 monclient(hunting): authenticate will time out at 1268.631644s 2024-09-16T14:48:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f6012ffd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f60140a1b00 legacy=0x7f60140a1ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f60127fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f60140a25f0 0x7f6014139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33912/0 (socket says 172.21.15.189:33912) 2024-09-16T14:48:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.509+0000 7f60127fc640 1 -- 172.21.15.189:0/814322878 learned_addr learned my addr 172.21.15.189:0/814322878 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 1 -- 172.21.15.189:0/814322878 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198301247 0 0) 0x7f601409eb50 con 0x7f601413b430 2024-09-16T14:48:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient(hunting): my global_id is 114569 2024-09-16T14:48:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 1 -- 172.21.15.189:0/814322878 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ffc003530 con 0x7f601413b430 2024-09-16T14:48:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 1 -- 172.21.15.189:0/814322878 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839854001 0 0) 0x7f60140a30b0 con 0x7f60140a25f0 2024-09-16T14:48:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient(hunting): my global_id is 124257 2024-09-16T14:48:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 1 -- 172.21.15.189:0/814322878 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f601409eb50 con 0x7f60140a25f0 2024-09-16T14:48:06.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 1 -- 172.21.15.189:0/814322878 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 843994002 0 0) 0x7f601409eb50 con 0x7f60140a25f0 2024-09-16T14:48:06.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 1 -- 172.21.15.189:0/814322878 >> v1:172.21.15.26:6790/0 conn(0x7f601413b430 legacy=0x7f601413d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:06.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 1 -- 172.21.15.189:0/814322878 >> v1:172.21.15.189:6789/0 conn(0x7f60140a1b00 legacy=0x7f60140a1ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:06.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:48:06.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 1 monclient: found mon.noname-a 2024-09-16T14:48:06.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:06.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 1 -- 172.21.15.189:0/814322878 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601409ee30 con 0x7f60140a25f0 2024-09-16T14:48:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient: _finish_auth 0 2024-09-16T14:48:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 1 -- 172.21.15.189:0/814322878 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6008001580 con 0x7f60140a25f0 2024-09-16T14:48:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient: dump: 2024-09-16T14:48:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:06.524 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 5 monclient: authenticate success, global_id 124257 2024-09-16T14:48:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 1 -- 172.21.15.189:0/814322878 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60080017a0 con 0x7f60140a25f0 2024-09-16T14:48:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 1 -- 172.21.15.189:0/814322878 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6008001bb0 con 0x7f60140a25f0 2024-09-16T14:48:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f6011ffb640 10 monclient: dump: 2024-09-16T14:48:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:06.526 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 4 set_mon_vals no callback set 2024-09-16T14:48:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 10 monclient: shutdown 2024-09-16T14:48:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 1 -- 172.21.15.189:0/814322878 >> v1:172.21.15.26:6789/0 conn(0x7f60140a25f0 legacy=0x7f6014139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 1 -- 172.21.15.189:0/814322878 shutdown_connections 2024-09-16T14:48:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 1 -- 172.21.15.189:0/814322878 >> 172.21.15.189:0/814322878 conn(0x7f60140a7dd0 msgr2=0x7f60140a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 1 -- 172.21.15.189:0/814322878 shutdown_connections 2024-09-16T14:48:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 1 -- 172.21.15.189:0/814322878 wait complete. 2024-09-16T14:48:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 10 monclient: build_initial_monmap 2024-09-16T14:48:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 10 monclient: monmap: 2024-09-16T14:48:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:06.527 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 1 Processor -- start 2024-09-16T14:48:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 1 -- start start 2024-09-16T14:48:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 10 monclient: init 2024-09-16T14:48:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 10 monclient(hunting): picked mon.noname-b con 0x7f60140a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 10 monclient(hunting): picked mon.noname-a con 0x7f60140a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 10 monclient(hunting): picked mon.noname-c con 0x7f601413b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.513+0000 7f601931c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f601409c250 con 0x7f60140a25f0 2024-09-16T14:48:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f601931c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60141ca3d0 con 0x7f60140a1b00 2024-09-16T14:48:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f6012ffd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f60140a1b00 legacy=0x7f60141c95b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f601931c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60140a59c0 con 0x7f601413b430 2024-09-16T14:48:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f601931c640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f601931c640 10 monclient(hunting): authenticate will time out at 1268.636049s 2024-09-16T14:48:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f60127fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f60140a25f0 0x7f60141c9cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33918/0 (socket says 172.21.15.189:33918) 2024-09-16T14:48:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f60127fc640 1 -- 172.21.15.189:0/2246103647 learned_addr learned my addr 172.21.15.189:0/2246103647 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743055235 0 0) 0x7f60140a59c0 con 0x7f601413b430 2024-09-16T14:48:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient(hunting): my global_id is 114572 2024-09-16T14:48:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6000003530 con 0x7f601413b430 2024-09-16T14:48:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198976710 0 0) 0x7f601409c250 con 0x7f60140a25f0 2024-09-16T14:48:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient(hunting): my global_id is 124260 2024-09-16T14:48:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60140a59c0 con 0x7f60140a25f0 2024-09-16T14:48:06.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 963041389 0 0) 0x7f6000003530 con 0x7f601413b430 2024-09-16T14:48:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f601409c250 con 0x7f601413b430 2024-09-16T14:48:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f600c001d80 con 0x7f601413b430 2024-09-16T14:48:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient(hunting): dump: 2024-09-16T14:48:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:06.533 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:06.533 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:06.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:06.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1628847190 0 0) 0x7f60140a59c0 con 0x7f60140a25f0 2024-09-16T14:48:06.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6000003530 con 0x7f60140a25f0 2024-09-16T14:48:06.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6008001ab0 con 0x7f60140a25f0 2024-09-16T14:48:06.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:06.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:06.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient(hunting): dump: 2024-09-16T14:48:06.533 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:06.533 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:06.533 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:06.533 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:06.534 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:06.534 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:06.534 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:06.534 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:06.534 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:06.534 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:06.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3922824077 0 0) 0x7f601409c250 con 0x7f601413b430 2024-09-16T14:48:06.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 >> v1:172.21.15.189:6789/0 conn(0x7f60140a1b00 legacy=0x7f60141c95b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:06.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 >> v1:172.21.15.26:6789/0 conn(0x7f60140a25f0 legacy=0x7f60141c9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:06.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:48:06.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 monclient: found mon.c 2024-09-16T14:48:06.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:06.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:06.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601409e2f0 con 0x7f601413b430 2024-09-16T14:48:06.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient: _finish_auth 0 2024-09-16T14:48:06.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:48:06.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:06.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f601931c640 5 monclient: authenticate success, global_id 114572 2024-09-16T14:48:06.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f601931c640 10 monclient: _renew_subs 2024-09-16T14:48:06.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f601931c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:06.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f601931c640 1 -- 172.21.15.189:0/2246103647 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6014136350 con 0x7f601413b430 2024-09-16T14:48:06.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f601931c640 10 monclient: _renew_subs 2024-09-16T14:48:06.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f601931c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:06.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f601931c640 1 -- 172.21.15.189:0/2246103647 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f601409c5f0 con 0x7f601413b430 2024-09-16T14:48:06.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f600c001ff0 con 0x7f601413b430 2024-09-16T14:48:06.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:06.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f600c003dd0 con 0x7f601413b430 2024-09-16T14:48:06.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:06.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f6010ff9640 4 set_mon_vals no callback set 2024-09-16T14:48:06.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:06.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 10 monclient: dump: 2024-09-16T14:48:06.536 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:06.536 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:06.536 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:06.536 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:06.536 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:06.536 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:06.536 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:06.536 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:06.536 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:06.537 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:06.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f601931c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:06.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f601931c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:06.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f601931c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:06.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f601931c640 1 -- 172.21.15.189:0/2246103647 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff4005050 con 0x7f601413b430 2024-09-16T14:48:06.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f600c00e840 con 0x7f601413b430 2024-09-16T14:48:06.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(142..142 src has 73..142) v4 ==== 7063+0+0 (unknown 3949000827 0 0) 0x7f600c043040 con 0x7f601413b430 2024-09-16T14:48:06.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.517+0000 7f6012ffd640 10 monclient: get_auth_request con 0x7f6000036220 auth_method 0 2024-09-16T14:48:06.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.533+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f600c00fd90 con 0x7f601413b430 2024-09-16T14:48:06.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.533+0000 7f5ffb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:06.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.533+0000 7f5ffb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.701+0000 7f601931c640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.701+0000 7f601931c640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:06.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.701+0000 7f601931c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:06.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:06.701+0000 7f601931c640 1 -- 172.21.15.189:0/2246103647 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f5ff4005300 con 0x7f601413b430 2024-09-16T14:48:07.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.385+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v143) v1 ==== 82+0+0 (unknown 2408500247 0 0) 0x7f600c011df0 con 0x7f601413b430 2024-09-16T14:48:07.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.385+0000 7f5ffb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:07.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.385+0000 7f5ffb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:07.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.489+0000 7f601931c640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:07.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.489+0000 7f601931c640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:07.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.489+0000 7f601931c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:07.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.489+0000 7f601931c640 1 -- 172.21.15.189:0/2246103647 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f5ff4005850 con 0x7f601413b430 2024-09-16T14:48:07.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.489+0000 7f5ffb7fe640 1 -- 172.21.15.189:0/2246103647 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v143) v1 ==== 115+0+0 (unknown 370841300 0 0) 0x7f600c013570 con 0x7f601413b430 2024-09-16T14:48:07.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.489+0000 7f5ffb7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:07.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.489+0000 7f5ffb7fe640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-16T14:48:07.497 INFO:tasks.workunit.client.0.smithi189.stderr:compat weight-set already created 2024-09-16T14:48:07.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.493+0000 7f601931c640 1 -- 172.21.15.189:0/2246103647 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6000036220 legacy=0x7f60000386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:07.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.493+0000 7f601931c640 10 monclient: shutdown 2024-09-16T14:48:07.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.493+0000 7f601931c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:07.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.493+0000 7f601931c640 1 -- 172.21.15.189:0/2246103647 >> v1:172.21.15.26:6790/0 conn(0x7f601413b430 legacy=0x7f60141d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:07.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.493+0000 7f601931c640 1 -- 172.21.15.189:0/2246103647 shutdown_connections 2024-09-16T14:48:07.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.493+0000 7f601931c640 1 -- 172.21.15.189:0/2246103647 >> 172.21.15.189:0/2246103647 conn(0x7f60140a7dd0 msgr2=0x7f601406b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:07.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.493+0000 7f601931c640 1 -- 172.21.15.189:0/2246103647 shutdown_connections 2024-09-16T14:48:07.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.493+0000 7f601931c640 1 -- 172.21.15.189:0/2246103647 wait complete. 2024-09-16T14:48:07.514 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set create cool flat 2024-09-16T14:48:07.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.625+0000 7f5a9935b640 10 monclient: get_monmap_and_config 2024-09-16T14:48:07.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.625+0000 7f5a9935b640 10 monclient: build_initial_monmap 2024-09-16T14:48:07.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.625+0000 7f5a9935b640 10 monclient: monmap: 2024-09-16T14:48:07.633 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:07.634 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:07.634 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:07.633554+0000 2024-09-16T14:48:07.634 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:07.633554+0000 2024-09-16T14:48:07.634 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:07.634 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:07.634 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:07.634 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:07.634 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:07.634 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:07.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 1 Processor -- start 2024-09-16T14:48:07.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 1 -- start start 2024-09-16T14:48:07.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 10 monclient: init 2024-09-16T14:48:07.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:07.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:07.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 10 monclient(hunting): picked mon.noname-c con 0x7f5a94099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:07.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 10 monclient(hunting): picked mon.noname-b con 0x7f5a9409bd30 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:07.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 10 monclient(hunting): picked mon.noname-a con 0x7f5a94137500 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:07.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a94099e50 con 0x7f5a94137500 2024-09-16T14:48:07.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9409e510 con 0x7f5a94099a70 2024-09-16T14:48:07.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9409c2c0 con 0x7f5a9409bd30 2024-09-16T14:48:07.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:07.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 10 monclient(hunting): authenticate will time out at 1269.749395s 2024-09-16T14:48:07.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a92ffd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f5a94099a70 legacy=0x7f5a9409d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:07.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a937fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5a94137500 0x7f5a941398f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33926/0 (socket says 172.21.15.189:33926) 2024-09-16T14:48:07.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a937fe640 1 -- 172.21.15.189:0/1846809596 learned_addr learned my addr 172.21.15.189:0/1846809596 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:07.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 1 -- 172.21.15.189:0/1846809596 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219747524 0 0) 0x7f5a94099e50 con 0x7f5a94137500 2024-09-16T14:48:07.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:07.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:07.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:07.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient(hunting): my global_id is 124263 2024-09-16T14:48:07.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 1 -- 172.21.15.189:0/1846809596 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a7c003530 con 0x7f5a94137500 2024-09-16T14:48:07.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 1 -- 172.21.15.189:0/1846809596 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565959007 0 0) 0x7f5a9409c2c0 con 0x7f5a9409bd30 2024-09-16T14:48:07.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:07.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:07.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:07.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient(hunting): my global_id is 114575 2024-09-16T14:48:07.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 1 -- 172.21.15.189:0/1846809596 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a94099e50 con 0x7f5a9409bd30 2024-09-16T14:48:07.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 1 -- 172.21.15.189:0/1846809596 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3136692377 0 0) 0x7f5a7c003530 con 0x7f5a94137500 2024-09-16T14:48:07.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 1 -- 172.21.15.189:0/1846809596 >> v1:172.21.15.26:6790/0 conn(0x7f5a9409bd30 legacy=0x7f5a9409de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:07.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 1 -- 172.21.15.189:0/1846809596 >> v1:172.21.15.189:6789/0 conn(0x7f5a94099a70 legacy=0x7f5a9409d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:07.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:48:07.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 1 monclient: found mon.noname-a 2024-09-16T14:48:07.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:07.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:07.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 1 -- 172.21.15.189:0/1846809596 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9409ee60 con 0x7f5a94137500 2024-09-16T14:48:07.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient: _finish_auth 0 2024-09-16T14:48:07.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:07.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:07.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 1 -- 172.21.15.189:0/1846809596 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5a8c0019a0 con 0x7f5a94137500 2024-09-16T14:48:07.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:07.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:07.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient: dump: 2024-09-16T14:48:07.642 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:07.642 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:07.642 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:07.642 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:07.642 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:07.642 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:07.642 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:07.642 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:07.642 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:07.642 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:07.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 5 monclient: authenticate success, global_id 124263 2024-09-16T14:48:07.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:07.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 1 -- 172.21.15.189:0/1846809596 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a8c001bc0 con 0x7f5a94137500 2024-09-16T14:48:07.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:07.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 1 -- 172.21.15.189:0/1846809596 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5a8c001f90 con 0x7f5a94137500 2024-09-16T14:48:07.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:07.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 4 set_mon_vals no callback set 2024-09-16T14:48:07.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:07.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:07.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a91ffb640 10 monclient: dump: 2024-09-16T14:48:07.643 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:07.643 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:07.643 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:07.644 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:07.644 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:07.644 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:07.644 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:07.644 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:07.644 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:07.644 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:07.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 10 monclient: shutdown 2024-09-16T14:48:07.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:07.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 1 -- 172.21.15.189:0/1846809596 >> v1:172.21.15.26:6789/0 conn(0x7f5a94137500 legacy=0x7f5a941398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:07.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 1 -- 172.21.15.189:0/1846809596 shutdown_connections 2024-09-16T14:48:07.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 1 -- 172.21.15.189:0/1846809596 >> 172.21.15.189:0/1846809596 conn(0x7f5a940a3380 msgr2=0x7f5a9409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:07.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 1 -- 172.21.15.189:0/1846809596 shutdown_connections 2024-09-16T14:48:07.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.629+0000 7f5a9935b640 1 -- 172.21.15.189:0/1846809596 wait complete. 2024-09-16T14:48:07.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 monclient: build_initial_monmap 2024-09-16T14:48:07.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 monclient: monmap: 2024-09-16T14:48:07.645 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:07.645 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:07.645 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:07.645 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:07.645 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:07.645 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:07.645 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:07.645 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:07.645 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:07.646 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:07.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 1 Processor -- start 2024-09-16T14:48:07.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 1 -- start start 2024-09-16T14:48:07.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 monclient: init 2024-09-16T14:48:07.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:07.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:07.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 monclient(hunting): picked mon.noname-b con 0x7f5a94099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:07.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 monclient(hunting): picked mon.noname-a con 0x7f5a9409bd30 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:07.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 monclient(hunting): picked mon.noname-c con 0x7f5a94137500 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:07.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a94134640 con 0x7f5a9409bd30 2024-09-16T14:48:07.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9406b320 con 0x7f5a94099a70 2024-09-16T14:48:07.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a92ffd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f5a94099a70 legacy=0x7f5a9411c840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:07.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a940a1160 con 0x7f5a94137500 2024-09-16T14:48:07.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:07.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 monclient(hunting): authenticate will time out at 1269.753298s 2024-09-16T14:48:07.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a927fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5a9409bd30 0x7f5a9411cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33930/0 (socket says 172.21.15.189:33930) 2024-09-16T14:48:07.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a927fc640 1 -- 172.21.15.189:0/1064498547 learned_addr learned my addr 172.21.15.189:0/1064498547 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:07.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168965195 0 0) 0x7f5a88001100 con 0x7f5a9409bd30 2024-09-16T14:48:07.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:07.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient(hunting): my global_id is 124266 2024-09-16T14:48:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a80003560 con 0x7f5a9409bd30 2024-09-16T14:48:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560009700 0 0) 0x7f5a940a1160 con 0x7f5a94137500 2024-09-16T14:48:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient(hunting): my global_id is 114578 2024-09-16T14:48:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a88001100 con 0x7f5a94137500 2024-09-16T14:48:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 509462855 0 0) 0x7f5a80003560 con 0x7f5a9409bd30 2024-09-16T14:48:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a940a1160 con 0x7f5a9409bd30 2024-09-16T14:48:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5a88001b50 con 0x7f5a9409bd30 2024-09-16T14:48:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient(hunting): dump: 2024-09-16T14:48:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:07.653 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3249827288 0 0) 0x7f5a88001100 con 0x7f5a94137500 2024-09-16T14:48:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a80003560 con 0x7f5a94137500 2024-09-16T14:48:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5a8c001ed0 con 0x7f5a94137500 2024-09-16T14:48:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient(hunting): dump: 2024-09-16T14:48:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:07.655 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:07.655 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:07.655 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:07.655 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:07.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2745039072 0 0) 0x7f5a940a1160 con 0x7f5a9409bd30 2024-09-16T14:48:07.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 >> v1:172.21.15.26:6790/0 conn(0x7f5a94137500 legacy=0x7f5a9411d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:07.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 >> v1:172.21.15.189:6789/0 conn(0x7f5a94099a70 legacy=0x7f5a9411c840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:07.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:48:07.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 monclient: found mon.a 2024-09-16T14:48:07.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:07.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:07.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a94132150 con 0x7f5a9409bd30 2024-09-16T14:48:07.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:48:07.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:48:07.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:07.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 5 monclient: authenticate success, global_id 124266 2024-09-16T14:48:07.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 monclient: _renew_subs 2024-09-16T14:48:07.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:07.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 1 -- 172.21.15.189:0/1064498547 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a941207c0 con 0x7f5a9409bd30 2024-09-16T14:48:07.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 monclient: _renew_subs 2024-09-16T14:48:07.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:07.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 1 -- 172.21.15.189:0/1064498547 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a9411e3f0 con 0x7f5a9409bd30 2024-09-16T14:48:07.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a88001390 con 0x7f5a9409bd30 2024-09-16T14:48:07.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:07.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5a88003a50 con 0x7f5a9409bd30 2024-09-16T14:48:07.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:07.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:07.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a90ff9640 4 set_mon_vals no callback set 2024-09-16T14:48:07.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a7b7fe640 10 monclient: dump: 2024-09-16T14:48:07.657 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:07.657 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:07.657 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:07.657 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:07.657 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:07.657 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:07.657 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:07.657 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:07.657 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:07.657 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.633+0000 7f5a9935b640 1 -- 172.21.15.189:0/1064498547 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a74005050 con 0x7f5a9409bd30 2024-09-16T14:48:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.637+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5a88003c30 con 0x7f5a9409bd30 2024-09-16T14:48:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.637+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(143..143 src has 73..143) v4 ==== 7395+0+0 (unknown 2543596744 0 0) 0x7f5a88043fd0 con 0x7f5a9409bd30 2024-09-16T14:48:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.637+0000 7f5a92ffd640 10 monclient: get_auth_request con 0x7f5a80036250 auth_method 0 2024-09-16T14:48:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.653+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5a88014680 con 0x7f5a9409bd30 2024-09-16T14:48:07.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.653+0000 7f5a7b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:07.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.653+0000 7f5a7b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:07.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.813+0000 7f5a9935b640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T14:48:07.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.813+0000 7f5a9935b640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T14:48:07.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.813+0000 7f5a9935b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:07.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:07.813+0000 7f5a9935b640 1 -- 172.21.15.189:0/1064498547 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"} v 0) v1 -- 0x7f5a74005300 con 0x7f5a9409bd30 2024-09-16T14:48:08.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.393+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}]=0 v144) v1 ==== 107+0+0 (unknown 4117805952 0 0) 0x7f5a88014290 con 0x7f5a9409bd30 2024-09-16T14:48:08.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.393+0000 7f5a7b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T14:48:08.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.393+0000 7f5a7b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:08.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.493+0000 7f5a9935b640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T14:48:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.493+0000 7f5a9935b640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T14:48:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.493+0000 7f5a9935b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.493+0000 7f5a9935b640 1 -- 172.21.15.189:0/1064498547 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"} v 0) v1 -- 0x7f5a74001130 con 0x7f5a9409bd30 2024-09-16T14:48:08.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.497+0000 7f5a7b7fe640 1 -- 172.21.15.189:0/1064498547 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}]=0 weight-set for pool 'cool' already created v144) v1 ==== 149+0+0 (unknown 2941027144 0 0) 0x7f5a88014470 con 0x7f5a9409bd30 2024-09-16T14:48:08.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.497+0000 7f5a7b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T14:48:08.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.497+0000 7f5a7b7fe640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'cool' already created 2024-09-16T14:48:08.499 INFO:tasks.workunit.client.0.smithi189.stderr:weight-set for pool 'cool' already created 2024-09-16T14:48:08.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.497+0000 7f5a9935b640 1 -- 172.21.15.189:0/1064498547 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5a80036250 legacy=0x7f5a80038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:08.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.497+0000 7f5a9935b640 10 monclient: shutdown 2024-09-16T14:48:08.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.497+0000 7f5a9935b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:08.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.497+0000 7f5a9935b640 1 -- 172.21.15.189:0/1064498547 >> v1:172.21.15.26:6789/0 conn(0x7f5a9409bd30 legacy=0x7f5a9411cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:08.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.497+0000 7f5a9935b640 1 -- 172.21.15.189:0/1064498547 shutdown_connections 2024-09-16T14:48:08.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.497+0000 7f5a9935b640 1 -- 172.21.15.189:0/1064498547 >> 172.21.15.189:0/1064498547 conn(0x7f5a940a3380 msgr2=0x7f5a94133ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:08.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.497+0000 7f5a9935b640 1 -- 172.21.15.189:0/1064498547 shutdown_connections 2024-09-16T14:48:08.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.497+0000 7f5a9935b640 1 -- 172.21.15.189:0/1064498547 wait complete. 2024-09-16T14:48:08.516 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set create cold positional 2024-09-16T14:48:08.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f2081d85640 10 monclient: get_monmap_and_config 2024-09-16T14:48:08.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f2081d85640 10 monclient: build_initial_monmap 2024-09-16T14:48:08.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f2081d85640 10 monclient: monmap: 2024-09-16T14:48:08.635 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:08.635 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:08.635 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:08.635125+0000 2024-09-16T14:48:08.636 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:08.635125+0000 2024-09-16T14:48:08.636 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:08.636 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:08.636 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:08.636 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:08.636 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:08.636 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:08.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f2081d85640 1 Processor -- start 2024-09-16T14:48:08.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f2081d85640 1 -- start start 2024-09-16T14:48:08.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f2081d85640 10 monclient: init 2024-09-16T14:48:08.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f2081d85640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:08.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f2081d85640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:08.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f2081d85640 10 monclient(hunting): picked mon.noname-a con 0x7f207c070830 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:08.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f2081d85640 10 monclient(hunting): picked mon.noname-b con 0x7f207c071320 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:08.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f2081d85640 10 monclient(hunting): picked mon.noname-c con 0x7f207c134cf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:08.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f2081d85640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f207c135740 con 0x7f207c070830 2024-09-16T14:48:08.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f2081d85640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f207c135160 con 0x7f207c134cf0 2024-09-16T14:48:08.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f2081d85640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f207c09d810 con 0x7f207c071320 2024-09-16T14:48:08.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f2081d85640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:08.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f2081d85640 10 monclient(hunting): authenticate will time out at 1270.750783s 2024-09-16T14:48:08.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f207bfff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f207c134cf0 legacy=0x7f207c13d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:08.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f207b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f207c070830 0x7f207c070c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33940/0 (socket says 172.21.15.189:33940) 2024-09-16T14:48:08.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f207b7fe640 1 -- 172.21.15.189:0/1125414093 learned_addr learned my addr 172.21.15.189:0/1125414093 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:08.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f207affd640 1 --1- 172.21.15.189:0/1125414093 >> v1:172.21.15.26:6790/0 conn(0x7f207c071320 0x7f207c139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:43922/0 (socket says 172.21.15.189:43922) 2024-09-16T14:48:08.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f207a7fc640 1 -- 172.21.15.189:0/1125414093 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063895259 0 0) 0x7f207c135740 con 0x7f207c070830 2024-09-16T14:48:08.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f207a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:08.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f207a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:08.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f207a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:08.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f207a7fc640 10 monclient(hunting): my global_id is 124269 2024-09-16T14:48:08.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f207a7fc640 1 -- 172.21.15.189:0/1125414093 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2064003530 con 0x7f207c070830 2024-09-16T14:48:08.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f207a7fc640 1 -- 172.21.15.189:0/1125414093 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927054526 0 0) 0x7f207c09d810 con 0x7f207c071320 2024-09-16T14:48:08.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f207a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:08.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f207a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:08.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f207a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:08.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f207a7fc640 10 monclient(hunting): my global_id is 114581 2024-09-16T14:48:08.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.633+0000 7f207a7fc640 1 -- 172.21.15.189:0/1125414093 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f207c135740 con 0x7f207c071320 2024-09-16T14:48:08.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 1 -- 172.21.15.189:0/1125414093 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2275232147 0 0) 0x7f2064003530 con 0x7f207c070830 2024-09-16T14:48:08.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 1 -- 172.21.15.189:0/1125414093 >> v1:172.21.15.26:6790/0 conn(0x7f207c071320 legacy=0x7f207c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:08.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 1 -- 172.21.15.189:0/1125414093 >> v1:172.21.15.189:6789/0 conn(0x7f207c134cf0 legacy=0x7f207c13d440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:08.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:08.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 1 monclient: found mon.noname-a 2024-09-16T14:48:08.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:08.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:08.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 1 -- 172.21.15.189:0/1125414093 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207c09f1c0 con 0x7f207c070830 2024-09-16T14:48:08.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:08.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:08.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:08.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 1 -- 172.21.15.189:0/1125414093 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f206c0019a0 con 0x7f207c070830 2024-09-16T14:48:08.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 5 monclient: authenticate success, global_id 124269 2024-09-16T14:48:08.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:08.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:08.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:08.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 10 monclient: dump: 2024-09-16T14:48:08.649 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:08.649 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:08.649 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:08.649 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:08.649 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:08.650 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:08.650 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:08.650 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:08.650 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:08.650 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:08.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:08.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 1 -- 172.21.15.189:0/1125414093 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f206c001c80 con 0x7f207c070830 2024-09-16T14:48:08.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:08.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:08.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 4 set_mon_vals no callback set 2024-09-16T14:48:08.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 1 -- 172.21.15.189:0/1125414093 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f206c002010 con 0x7f207c070830 2024-09-16T14:48:08.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:08.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:08.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207a7fc640 10 monclient: dump: 2024-09-16T14:48:08.651 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:08.651 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:08.651 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:08.651 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:08.651 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:08.651 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:08.651 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:08.651 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:08.651 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:08.651 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:08.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 10 monclient: shutdown 2024-09-16T14:48:08.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:08.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 1 -- 172.21.15.189:0/1125414093 >> v1:172.21.15.26:6789/0 conn(0x7f207c070830 legacy=0x7f207c070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:08.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 1 -- 172.21.15.189:0/1125414093 shutdown_connections 2024-09-16T14:48:08.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 1 -- 172.21.15.189:0/1125414093 >> 172.21.15.189:0/1125414093 conn(0x7f207c0a3380 msgr2=0x7f207c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:08.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 1 -- 172.21.15.189:0/1125414093 shutdown_connections 2024-09-16T14:48:08.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 1 -- 172.21.15.189:0/1125414093 wait complete. 2024-09-16T14:48:08.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 10 monclient: build_initial_monmap 2024-09-16T14:48:08.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 10 monclient: monmap: 2024-09-16T14:48:08.652 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:08.652 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:08.652 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:08.652 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:08.652 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:08.652 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:08.653 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:08.653 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:08.653 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:08.653 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:08.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 1 Processor -- start 2024-09-16T14:48:08.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 1 -- start start 2024-09-16T14:48:08.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 10 monclient: init 2024-09-16T14:48:08.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:08.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:08.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 10 monclient(hunting): picked mon.noname-a con 0x7f207c070830 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:08.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 10 monclient(hunting): picked mon.noname-c con 0x7f207c071320 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:08.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 10 monclient(hunting): picked mon.noname-b con 0x7f207c134cf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:08.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f207c132dc0 con 0x7f207c070830 2024-09-16T14:48:08.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f207c06b320 con 0x7f207c134cf0 2024-09-16T14:48:08.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f207c11d830 con 0x7f207c071320 2024-09-16T14:48:08.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:08.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 10 monclient(hunting): authenticate will time out at 1270.754401s 2024-09-16T14:48:08.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207bfff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f207c134cf0 legacy=0x7f207c1d1600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:08.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f207c070830 0x7f207c1c92a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:33946/0 (socket says 172.21.15.189:33946) 2024-09-16T14:48:08.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f207b7fe640 1 -- 172.21.15.189:0/3887942170 learned_addr learned my addr 172.21.15.189:0/3887942170 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:08.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 399972625 0 0) 0x7f207c132dc0 con 0x7f207c070830 2024-09-16T14:48:08.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:08.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:08.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:08.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient(hunting): my global_id is 124272 2024-09-16T14:48:08.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2068003560 con 0x7f207c070830 2024-09-16T14:48:08.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312403058 0 0) 0x7f207c11d830 con 0x7f207c071320 2024-09-16T14:48:08.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:08.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:08.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:08.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient(hunting): my global_id is 114584 2024-09-16T14:48:08.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f207c132dc0 con 0x7f207c071320 2024-09-16T14:48:08.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1127798808 0 0) 0x7f2068003560 con 0x7f207c070830 2024-09-16T14:48:08.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f207c11d830 con 0x7f207c070830 2024-09-16T14:48:08.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f206c001960 con 0x7f207c070830 2024-09-16T14:48:08.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:08.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:08.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient(hunting): dump: 2024-09-16T14:48:08.655 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:08.655 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:08.656 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2420880928 0 0) 0x7f207c132dc0 con 0x7f207c071320 2024-09-16T14:48:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2068003560 con 0x7f207c071320 2024-09-16T14:48:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f20700018c0 con 0x7f207c071320 2024-09-16T14:48:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:08.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:08.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient(hunting): dump: 2024-09-16T14:48:08.657 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:08.657 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:08.657 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:08.657 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:08.657 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:08.657 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:08.657 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:08.657 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:08.657 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:08.657 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:08.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 504048322 0 0) 0x7f207c11d830 con 0x7f207c070830 2024-09-16T14:48:08.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 >> v1:172.21.15.26:6790/0 conn(0x7f207c071320 legacy=0x7f207c130710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:08.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 >> v1:172.21.15.189:6789/0 conn(0x7f207c134cf0 legacy=0x7f207c1d1600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:08.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:08.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 1 monclient: found mon.a 2024-09-16T14:48:08.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:08.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:08.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207c11db10 con 0x7f207c070830 2024-09-16T14:48:08.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient: _finish_auth 0 2024-09-16T14:48:08.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:08.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2063fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:08.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.637+0000 7f2081d85640 5 monclient: authenticate success, global_id 124272 2024-09-16T14:48:08.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2081d85640 10 monclient: _renew_subs 2024-09-16T14:48:08.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2081d85640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:08.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2081d85640 1 -- 172.21.15.189:0/3887942170 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f207c11bd70 con 0x7f207c070830 2024-09-16T14:48:08.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2081d85640 10 monclient: _renew_subs 2024-09-16T14:48:08.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2081d85640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:08.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2081d85640 1 -- 172.21.15.189:0/3887942170 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f207c130e20 con 0x7f207c070830 2024-09-16T14:48:08.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f206c001540 con 0x7f207c070830 2024-09-16T14:48:08.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2063fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:08.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f206c005010 con 0x7f207c070830 2024-09-16T14:48:08.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2063fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:08.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2063fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:08.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2079ffb640 4 set_mon_vals no callback set 2024-09-16T14:48:08.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2063fff640 10 monclient: dump: 2024-09-16T14:48:08.659 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:08.659 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:08.659 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:08.660 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:08.660 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:08.660 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:08.660 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:08.660 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:08.660 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:08.660 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:08.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2081d85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:08.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2081d85640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:08.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2081d85640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:08.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2081d85640 1 -- 172.21.15.189:0/3887942170 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f205c005050 con 0x7f207c070830 2024-09-16T14:48:08.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f206c0051f0 con 0x7f207c070830 2024-09-16T14:48:08.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(144..144 src has 73..144) v4 ==== 7727+0+0 (unknown 3645070407 0 0) 0x7f206c044610 con 0x7f207c070830 2024-09-16T14:48:08.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.641+0000 7f207affd640 10 monclient: get_auth_request con 0x7f2068036250 auth_method 0 2024-09-16T14:48:08.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.653+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f206c0149b0 con 0x7f207c070830 2024-09-16T14:48:08.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.653+0000 7f2063fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:08.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.653+0000 7f2063fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:08.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.817+0000 7f2081d85640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T14:48:08.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.817+0000 7f2081d85640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T14:48:08.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.817+0000 7f2081d85640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:08.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:08.817+0000 7f2081d85640 1 -- 172.21.15.189:0/3887942170 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"} v 0) v1 -- 0x7f205c005300 con 0x7f207c070830 2024-09-16T14:48:09.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.409+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}]=0 v145) v1 ==== 113+0+0 (unknown 3163306448 0 0) 0x7f206c0147d0 con 0x7f207c070830 2024-09-16T14:48:09.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.409+0000 7f2063fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T14:48:09.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.409+0000 7f2063fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:09.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.513+0000 7f2081d85640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T14:48:09.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.513+0000 7f2081d85640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T14:48:09.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.513+0000 7f2081d85640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:09.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.513+0000 7f2081d85640 1 -- 172.21.15.189:0/3887942170 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"} v 0) v1 -- 0x7f205c001130 con 0x7f207c070830 2024-09-16T14:48:09.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.517+0000 7f2063fff640 1 -- 172.21.15.189:0/3887942170 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}]=0 weight-set for pool 'cold' already created v145) v1 ==== 155+0+0 (unknown 2343926855 0 0) 0x7f206c015510 con 0x7f207c070830 2024-09-16T14:48:09.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.517+0000 7f2063fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T14:48:09.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.517+0000 7f2063fff640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'cold' already created 2024-09-16T14:48:09.519 INFO:tasks.workunit.client.0.smithi189.stderr:weight-set for pool 'cold' already created 2024-09-16T14:48:09.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.517+0000 7f2081d85640 1 -- 172.21.15.189:0/3887942170 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2068036250 legacy=0x7f2068038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:09.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.517+0000 7f2081d85640 10 monclient: shutdown 2024-09-16T14:48:09.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.517+0000 7f2081d85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:09.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.517+0000 7f2081d85640 1 -- 172.21.15.189:0/3887942170 >> v1:172.21.15.26:6789/0 conn(0x7f207c070830 legacy=0x7f207c1c92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:09.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.517+0000 7f2081d85640 1 -- 172.21.15.189:0/3887942170 shutdown_connections 2024-09-16T14:48:09.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.517+0000 7f2081d85640 1 -- 172.21.15.189:0/3887942170 >> 172.21.15.189:0/3887942170 conn(0x7f207c0a3380 msgr2=0x7f207c1320d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:09.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.517+0000 7f2081d85640 1 -- 172.21.15.189:0/3887942170 shutdown_connections 2024-09-16T14:48:09.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.517+0000 7f2081d85640 1 -- 172.21.15.189:0/3887942170 wait complete. 2024-09-16T14:48:09.536 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-16T14:48:09.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.645+0000 7f784c161640 10 monclient: get_monmap_and_config 2024-09-16T14:48:09.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.645+0000 7f784c161640 10 monclient: build_initial_monmap 2024-09-16T14:48:09.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.645+0000 7f784c161640 10 monclient: monmap: 2024-09-16T14:48:09.648 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:09.648 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:09.648 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:09.648157+0000 2024-09-16T14:48:09.648 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:09.648157+0000 2024-09-16T14:48:09.648 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:09.649 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:09.649 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:09.649 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:09.649 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:09.649 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:09.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.645+0000 7f784c161640 1 Processor -- start 2024-09-16T14:48:09.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.645+0000 7f784c161640 1 -- start start 2024-09-16T14:48:09.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.645+0000 7f784c161640 10 monclient: init 2024-09-16T14:48:09.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.645+0000 7f784c161640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:09.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.645+0000 7f784c161640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:09.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.645+0000 7f784c161640 10 monclient(hunting): picked mon.noname-c con 0x7f784409df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:09.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.645+0000 7f784c161640 10 monclient(hunting): picked mon.noname-a con 0x7f784409e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:09.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 10 monclient(hunting): picked mon.noname-b con 0x7f784409e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:09.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784409ee20 con 0x7f784409e2f0 2024-09-16T14:48:09.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784409f100 con 0x7f784409df10 2024-09-16T14:48:09.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784409f3e0 con 0x7f784409e740 2024-09-16T14:48:09.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:09.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 10 monclient(hunting): authenticate will time out at 1271.764070s 2024-09-16T14:48:09.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7849efd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f784409df10 legacy=0x7f784409d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:09.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f78496fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f784409e2f0 0x7f78441378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:48592/0 (socket says 172.21.15.189:48592) 2024-09-16T14:48:09.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f78496fc640 1 -- 172.21.15.189:0/1319090418 learned_addr learned my addr 172.21.15.189:0/1319090418 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:09.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 1 -- 172.21.15.189:0/1319090418 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290071049 0 0) 0x7f784409f3e0 con 0x7f784409e740 2024-09-16T14:48:09.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:09.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:09.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:09.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient(hunting): my global_id is 114587 2024-09-16T14:48:09.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 1 -- 172.21.15.189:0/1319090418 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7830003530 con 0x7f784409e740 2024-09-16T14:48:09.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 1 -- 172.21.15.189:0/1319090418 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098851177 0 0) 0x7f784409ee20 con 0x7f784409e2f0 2024-09-16T14:48:09.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:09.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:09.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:09.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient(hunting): my global_id is 124275 2024-09-16T14:48:09.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 1 -- 172.21.15.189:0/1319090418 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f784409f3e0 con 0x7f784409e2f0 2024-09-16T14:48:09.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 1 -- 172.21.15.189:0/1319090418 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3523846000 0 0) 0x7f7830003530 con 0x7f784409e740 2024-09-16T14:48:09.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 1 -- 172.21.15.189:0/1319090418 >> v1:172.21.15.189:6789/0 conn(0x7f784409df10 legacy=0x7f784409d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:09.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 1 -- 172.21.15.189:0/1319090418 >> v1:172.21.15.26:6789/0 conn(0x7f784409e2f0 legacy=0x7f78441378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:09.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient: _finish_hunting 0 2024-09-16T14:48:09.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 1 monclient: found mon.noname-b 2024-09-16T14:48:09.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:09.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:48:09.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 1 -- 172.21.15.189:0/1319090418 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784409c690 con 0x7f784409e740 2024-09-16T14:48:09.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient: _finish_auth 0 2024-09-16T14:48:09.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:09.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:09.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 1 -- 172.21.15.189:0/1319090418 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f783c0019a0 con 0x7f784409e740 2024-09-16T14:48:09.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:09.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:09.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient: dump: 2024-09-16T14:48:09.655 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:09.655 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:09.655 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:09.656 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:09.656 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:09.656 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:09.656 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:09.656 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:09.656 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:09.656 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:09.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 5 monclient: authenticate success, global_id 114587 2024-09-16T14:48:09.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:09.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 1 -- 172.21.15.189:0/1319090418 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f783c001bc0 con 0x7f784409e740 2024-09-16T14:48:09.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:09.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 1 -- 172.21.15.189:0/1319090418 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f783c001f90 con 0x7f784409e740 2024-09-16T14:48:09.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:09.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:09.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7848efb640 10 monclient: dump: 2024-09-16T14:48:09.657 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:09.657 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:09.657 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:09.657 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:09.657 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:09.657 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:09.657 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:09.658 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:09.658 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:09.658 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:09.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:09.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 4 set_mon_vals no callback set 2024-09-16T14:48:09.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 10 monclient: shutdown 2024-09-16T14:48:09.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:09.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 1 -- 172.21.15.189:0/1319090418 >> v1:172.21.15.26:6790/0 conn(0x7f784409e740 legacy=0x7f784413b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:09.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 1 -- 172.21.15.189:0/1319090418 shutdown_connections 2024-09-16T14:48:09.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 1 -- 172.21.15.189:0/1319090418 >> 172.21.15.189:0/1319090418 conn(0x7f78440a3320 msgr2=0x7f784409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:09.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 1 -- 172.21.15.189:0/1319090418 shutdown_connections 2024-09-16T14:48:09.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 1 -- 172.21.15.189:0/1319090418 wait complete. 2024-09-16T14:48:09.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 10 monclient: build_initial_monmap 2024-09-16T14:48:09.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 10 monclient: monmap: 2024-09-16T14:48:09.659 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:09.659 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:09.659 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:09.659 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:09.659 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:09.659 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:09.659 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:09.659 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:09.659 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:09.659 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:09.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 1 Processor -- start 2024-09-16T14:48:09.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 1 -- start start 2024-09-16T14:48:09.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 10 monclient: init 2024-09-16T14:48:09.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:09.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:09.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 10 monclient(hunting): picked mon.noname-b con 0x7f784409df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:09.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 10 monclient(hunting): picked mon.noname-a con 0x7f784409e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:09.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 10 monclient(hunting): picked mon.noname-c con 0x7f784409e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:09.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784406b380 con 0x7f784409e2f0 2024-09-16T14:48:09.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784411fa70 con 0x7f784409df10 2024-09-16T14:48:09.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7844128480 con 0x7f784409e740 2024-09-16T14:48:09.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:09.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f784c161640 10 monclient(hunting): authenticate will time out at 1271.767749s 2024-09-16T14:48:09.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.649+0000 7f7849efd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f784409df10 legacy=0x7f784411c390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:09.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f78496fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f784409e2f0 0x7f784411caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:48594/0 (socket says 172.21.15.189:48594) 2024-09-16T14:48:09.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f78496fc640 1 -- 172.21.15.189:0/3787386181 learned_addr learned my addr 172.21.15.189:0/3787386181 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:09.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340860135 0 0) 0x7f7844128480 con 0x7f784409e740 2024-09-16T14:48:09.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:09.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:09.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:09.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient(hunting): my global_id is 114590 2024-09-16T14:48:09.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7828003530 con 0x7f784409e740 2024-09-16T14:48:09.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854661032 0 0) 0x7f784406b380 con 0x7f784409e2f0 2024-09-16T14:48:09.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:09.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:09.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:09.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient(hunting): my global_id is 124278 2024-09-16T14:48:09.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7844128480 con 0x7f784409e2f0 2024-09-16T14:48:09.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2333467270 0 0) 0x7f7828003530 con 0x7f784409e740 2024-09-16T14:48:09.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f784406b380 con 0x7f784409e740 2024-09-16T14:48:09.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f783c001ed0 con 0x7f784409e740 2024-09-16T14:48:09.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:09.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:09.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient(hunting): dump: 2024-09-16T14:48:09.667 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:09.667 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:09.667 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:09.667 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:09.667 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:09.667 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:09.667 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:09.667 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:09.667 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:09.667 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:09.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:09.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 749402467 0 0) 0x7f7844128480 con 0x7f784409e2f0 2024-09-16T14:48:09.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7828003530 con 0x7f784409e2f0 2024-09-16T14:48:09.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f78340019e0 con 0x7f784409e2f0 2024-09-16T14:48:09.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:09.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:09.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient(hunting): dump: 2024-09-16T14:48:09.668 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:09.668 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:09.668 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:09.669 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:09.669 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:09.669 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:09.669 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:09.669 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:09.669 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:09.669 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:09.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3951908897 0 0) 0x7f784406b380 con 0x7f784409e740 2024-09-16T14:48:09.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 >> v1:172.21.15.189:6789/0 conn(0x7f784409df10 legacy=0x7f784411c390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:09.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 >> v1:172.21.15.26:6789/0 conn(0x7f784409e2f0 legacy=0x7f784411caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:09.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:09.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 monclient: found mon.c 2024-09-16T14:48:09.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:09.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:09.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7844120030 con 0x7f784409e740 2024-09-16T14:48:09.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:09.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:09.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:09.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f784c161640 5 monclient: authenticate success, global_id 114590 2024-09-16T14:48:09.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f784c161640 10 monclient: _renew_subs 2024-09-16T14:48:09.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f784c161640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:09.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f784c161640 1 -- 172.21.15.189:0/3787386181 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f784411d1b0 con 0x7f784409e740 2024-09-16T14:48:09.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f784c161640 10 monclient: _renew_subs 2024-09-16T14:48:09.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f784c161640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:09.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f784c161640 1 -- 172.21.15.189:0/3787386181 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f784411eb10 con 0x7f784409e740 2024-09-16T14:48:09.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f783c003c30 con 0x7f784409e740 2024-09-16T14:48:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783bfff640 4 set_mon_vals no callback set 2024-09-16T14:48:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f783c003f40 con 0x7f784409e740 2024-09-16T14:48:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 10 monclient: dump: 2024-09-16T14:48:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:09.672 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:09.672 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:09.672 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:09.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f784c161640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:09.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f784c161640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:09.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f784c161640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:09.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f784c161640 1 -- 172.21.15.189:0/3787386181 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782c005050 con 0x7f784409e740 2024-09-16T14:48:09.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f783c00e9b0 con 0x7f784409e740 2024-09-16T14:48:09.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(145..145 src has 73..145) v4 ==== 8199+0+0 (unknown 1908104764 0 0) 0x7f783c0434e0 con 0x7f784409e740 2024-09-16T14:48:09.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.653+0000 7f7849efd640 10 monclient: get_auth_request con 0x7f78280361e0 auth_method 0 2024-09-16T14:48:09.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.665+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f783c013760 con 0x7f784409e740 2024-09-16T14:48:09.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.665+0000 7f783a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:09.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.665+0000 7f783a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:09.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.813+0000 7f784c161640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:09.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.813+0000 7f784c161640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:09.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.813+0000 7f784c161640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:09.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:09.813+0000 7f784c161640 1 -- 172.21.15.189:0/3787386181 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f782c005b10 con 0x7f784409e740 2024-09-16T14:48:10.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:10.409+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v146) v1 ==== 123+0+0 (unknown 3704696094 0 0) 0x7f783c013580 con 0x7f784409e740 2024-09-16T14:48:10.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:10.409+0000 7f783a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:10.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:10.409+0000 7f783a7fc640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-16T14:48:10.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:10.509+0000 7f784c161640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:10.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:10.509+0000 7f784c161640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:10.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:10.509+0000 7f784c161640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:10.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:10.509+0000 7f784c161640 1 -- 172.21.15.189:0/3787386181 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f782c0010a0 con 0x7f784409e740 2024-09-16T14:48:11.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.185+0000 7f783a7fc640 1 -- 172.21.15.189:0/3787386181 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v147) v1 ==== 149+0+0 (unknown 1452352023 0 0) 0x7f783c013580 con 0x7f784409e740 2024-09-16T14:48:11.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.185+0000 7f783a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:11.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.185+0000 7f783a7fc640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T14:48:11.189 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T14:48:11.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.189+0000 7f784c161640 1 -- 172.21.15.189:0/3787386181 >> v1:172.21.15.189:6812/3196161582 conn(0x7f78280361e0 legacy=0x7f78280386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:11.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.189+0000 7f784c161640 10 monclient: shutdown 2024-09-16T14:48:11.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.189+0000 7f784c161640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:11.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.189+0000 7f784c161640 1 -- 172.21.15.189:0/3787386181 >> v1:172.21.15.26:6790/0 conn(0x7f784409e740 legacy=0x7f78441cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:11.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.189+0000 7f784c161640 1 -- 172.21.15.189:0/3787386181 shutdown_connections 2024-09-16T14:48:11.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.189+0000 7f784c161640 1 -- 172.21.15.189:0/3787386181 >> 172.21.15.189:0/3787386181 conn(0x7f78440a3320 msgr2=0x7f78440a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:11.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.189+0000 7f784c161640 1 -- 172.21.15.189:0/3787386181 shutdown_connections 2024-09-16T14:48:11.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.189+0000 7f784c161640 1 -- 172.21.15.189:0/3787386181 wait complete. 2024-09-16T14:48:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight-compat osd.0 10.5 2024-09-16T14:48:11.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.329+0000 7f244a5ad640 10 monclient: get_monmap_and_config 2024-09-16T14:48:11.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.329+0000 7f244a5ad640 10 monclient: build_initial_monmap 2024-09-16T14:48:11.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.329+0000 7f244a5ad640 10 monclient: monmap: 2024-09-16T14:48:11.331 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:11.331 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:11.332 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:11.331729+0000 2024-09-16T14:48:11.332 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:11.331729+0000 2024-09-16T14:48:11.332 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:11.332 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:11.332 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:11.332 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:11.332 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:11.332 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:11.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.329+0000 7f244a5ad640 1 Processor -- start 2024-09-16T14:48:11.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.329+0000 7f244a5ad640 1 -- start start 2024-09-16T14:48:11.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.329+0000 7f244a5ad640 10 monclient: init 2024-09-16T14:48:11.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.329+0000 7f244a5ad640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:11.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.329+0000 7f244a5ad640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:11.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 10 monclient(hunting): picked mon.noname-c con 0x7f2444096590 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:11.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 10 monclient(hunting): picked mon.noname-a con 0x7f2444097440 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:11.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 10 monclient(hunting): picked mon.noname-b con 0x7f2444137fd0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:11.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2444073c00 con 0x7f2444097440 2024-09-16T14:48:11.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244409d050 con 0x7f2444096590 2024-09-16T14:48:11.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244409cd70 con 0x7f2444137fd0 2024-09-16T14:48:11.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:11.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 10 monclient(hunting): authenticate will time out at 1273.448118s 2024-09-16T14:48:11.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2443fff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2444096590 legacy=0x7f2444096970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:11.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f24437fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2444097440 0x7f24441368b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:48608/0 (socket says 172.21.15.189:48608) 2024-09-16T14:48:11.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f24437fe640 1 -- 172.21.15.189:0/2849033099 learned_addr learned my addr 172.21.15.189:0/2849033099 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:11.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 1 -- 172.21.15.189:0/2849033099 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238653465 0 0) 0x7f2444073c00 con 0x7f2444097440 2024-09-16T14:48:11.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:11.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:11.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:11.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient(hunting): my global_id is 124281 2024-09-16T14:48:11.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 1 -- 172.21.15.189:0/2849033099 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f242c005740 con 0x7f2444097440 2024-09-16T14:48:11.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 1 -- 172.21.15.189:0/2849033099 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087921922 0 0) 0x7f244409cd70 con 0x7f2444137fd0 2024-09-16T14:48:11.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:11.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:11.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient(hunting): my global_id is 114593 2024-09-16T14:48:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 1 -- 172.21.15.189:0/2849033099 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2444073c00 con 0x7f2444137fd0 2024-09-16T14:48:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 1 -- 172.21.15.189:0/2849033099 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 526656335 0 0) 0x7f2444073c00 con 0x7f2444137fd0 2024-09-16T14:48:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 1 -- 172.21.15.189:0/2849033099 >> v1:172.21.15.189:6789/0 conn(0x7f2444096590 legacy=0x7f2444096970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 1 -- 172.21.15.189:0/2849033099 >> v1:172.21.15.26:6789/0 conn(0x7f2444097440 legacy=0x7f24441368b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 1 monclient: found mon.noname-b 2024-09-16T14:48:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:48:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 1 -- 172.21.15.189:0/2849033099 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244409f690 con 0x7f2444137fd0 2024-09-16T14:48:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient: _finish_auth 0 2024-09-16T14:48:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 1 -- 172.21.15.189:0/2849033099 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f243c001830 con 0x7f2444137fd0 2024-09-16T14:48:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient: dump: 2024-09-16T14:48:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:11.341 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:11.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 5 monclient: authenticate success, global_id 114593 2024-09-16T14:48:11.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:11.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 1 -- 172.21.15.189:0/2849033099 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f243c001b50 con 0x7f2444137fd0 2024-09-16T14:48:11.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:11.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:11.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 1 -- 172.21.15.189:0/2849033099 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f243c001f20 con 0x7f2444137fd0 2024-09-16T14:48:11.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 4 set_mon_vals no callback set 2024-09-16T14:48:11.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:11.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:11.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f2442ffd640 10 monclient: dump: 2024-09-16T14:48:11.342 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:11.342 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:11.342 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:11.342 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:11.342 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:11.342 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:11.342 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:11.342 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:11.343 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:11.343 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:11.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 10 monclient: shutdown 2024-09-16T14:48:11.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:11.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 1 -- 172.21.15.189:0/2849033099 >> v1:172.21.15.26:6790/0 conn(0x7f2444137fd0 legacy=0x7f244413a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:11.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 1 -- 172.21.15.189:0/2849033099 shutdown_connections 2024-09-16T14:48:11.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 1 -- 172.21.15.189:0/2849033099 >> 172.21.15.189:0/2849033099 conn(0x7f24440a0340 msgr2=0x7f24440a0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:11.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 1 -- 172.21.15.189:0/2849033099 shutdown_connections 2024-09-16T14:48:11.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 1 -- 172.21.15.189:0/2849033099 wait complete. 2024-09-16T14:48:11.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 10 monclient: build_initial_monmap 2024-09-16T14:48:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 10 monclient: monmap: 2024-09-16T14:48:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:11.344 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 1 Processor -- start 2024-09-16T14:48:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 1 -- start start 2024-09-16T14:48:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.333+0000 7f244a5ad640 10 monclient: init 2024-09-16T14:48:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 10 monclient(hunting): picked mon.noname-b con 0x7f2444096590 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 10 monclient(hunting): picked mon.noname-a con 0x7f2444097440 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 10 monclient(hunting): picked mon.noname-c con 0x7f2444137fd0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24440a1800 con 0x7f2444097440 2024-09-16T14:48:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244408e230 con 0x7f2444096590 2024-09-16T14:48:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24440a1340 con 0x7f2444137fd0 2024-09-16T14:48:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2443fff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2444096590 legacy=0x7f244408a650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 10 monclient(hunting): authenticate will time out at 1273.452254s 2024-09-16T14:48:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f24437fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2444097440 0x7f244408ad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:48618/0 (socket says 172.21.15.189:48618) 2024-09-16T14:48:11.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f24437fe640 1 -- 172.21.15.189:0/816321174 learned_addr learned my addr 172.21.15.189:0/816321174 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:11.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787226416 0 0) 0x7f24440a1340 con 0x7f2444137fd0 2024-09-16T14:48:11.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:11.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:11.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:11.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient(hunting): my global_id is 114596 2024-09-16T14:48:11.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2430003530 con 0x7f2444137fd0 2024-09-16T14:48:11.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871371692 0 0) 0x7f2438001240 con 0x7f2444097440 2024-09-16T14:48:11.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:11.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient(hunting): my global_id is 124284 2024-09-16T14:48:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24440a1340 con 0x7f2444097440 2024-09-16T14:48:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1288608186 0 0) 0x7f2430003530 con 0x7f2444137fd0 2024-09-16T14:48:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2438001240 con 0x7f2444137fd0 2024-09-16T14:48:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f243c001e60 con 0x7f2444137fd0 2024-09-16T14:48:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient(hunting): dump: 2024-09-16T14:48:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:11.350 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2223582953 0 0) 0x7f24440a1340 con 0x7f2444097440 2024-09-16T14:48:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2430003530 con 0x7f2444097440 2024-09-16T14:48:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2438001840 con 0x7f2444097440 2024-09-16T14:48:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient(hunting): dump: 2024-09-16T14:48:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:11.351 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3781776452 0 0) 0x7f2438001240 con 0x7f2444137fd0 2024-09-16T14:48:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 >> v1:172.21.15.189:6789/0 conn(0x7f2444096590 legacy=0x7f244408a650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 >> v1:172.21.15.26:6789/0 conn(0x7f2444097440 legacy=0x7f244408ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:48:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 monclient: found mon.c 2024-09-16T14:48:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244409a630 con 0x7f2444137fd0 2024-09-16T14:48:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient: _finish_auth 0 2024-09-16T14:48:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:48:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 5 monclient: authenticate success, global_id 114596 2024-09-16T14:48:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 10 monclient: _renew_subs 2024-09-16T14:48:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 1 -- 172.21.15.189:0/816321174 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f244408df50 con 0x7f2444137fd0 2024-09-16T14:48:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 10 monclient: _renew_subs 2024-09-16T14:48:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 1 -- 172.21.15.189:0/816321174 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f244408bed0 con 0x7f2444137fd0 2024-09-16T14:48:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f243c0012a0 con 0x7f2444137fd0 2024-09-16T14:48:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f243c003dd0 con 0x7f2444137fd0 2024-09-16T14:48:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f24427fc640 4 set_mon_vals no callback set 2024-09-16T14:48:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 10 monclient: dump: 2024-09-16T14:48:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:11.354 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f244a5ad640 1 -- 172.21.15.189:0/816321174 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2424005050 con 0x7f2444137fd0 2024-09-16T14:48:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.337+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f243c00e840 con 0x7f2444137fd0 2024-09-16T14:48:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.341+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(147..147 src has 73..147) v4 ==== 8167+0+0 (unknown 2929757168 0 0) 0x7f243c043450 con 0x7f2444137fd0 2024-09-16T14:48:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.341+0000 7f2443fff640 10 monclient: get_auth_request con 0x7f2430036270 auth_method 0 2024-09-16T14:48:11.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.349+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f243c013690 con 0x7f2444137fd0 2024-09-16T14:48:11.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.349+0000 7f2440ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:11.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.349+0000 7f2440ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:11.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.521+0000 7f244a5ad640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T14:48:11.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.521+0000 7f244a5ad640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T14:48:11.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.521+0000 7f244a5ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:11.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:11.521+0000 7f244a5ad640 1 -- 172.21.15.189:0/816321174 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]} v 0) v1 -- 0x7f2424005300 con 0x7f2444137fd0 2024-09-16T14:48:12.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:12.429+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}]=0 v148) v1 ==== 119+0+0 (unknown 3924686237 0 0) 0x7f243c0134b0 con 0x7f2444137fd0 2024-09-16T14:48:12.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:12.429+0000 7f2440ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T14:48:12.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:12.429+0000 7f2440ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:12.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:12.541+0000 7f244a5ad640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T14:48:12.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:12.541+0000 7f244a5ad640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T14:48:12.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:12.541+0000 7f244a5ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:12.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:12.541+0000 7f244a5ad640 1 -- 172.21.15.189:0/816321174 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]} v 0) v1 -- 0x7f2424001130 con 0x7f2444137fd0 2024-09-16T14:48:13.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.429+0000 7f2440ff9640 1 -- 172.21.15.189:0/816321174 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}]=0 v149) v1 ==== 119+0+0 (unknown 4021517600 0 0) 0x7f243c0141f0 con 0x7f2444137fd0 2024-09-16T14:48:13.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.429+0000 7f2440ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T14:48:13.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.429+0000 7f2440ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:13.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.433+0000 7f244a5ad640 1 -- 172.21.15.189:0/816321174 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2430036270 legacy=0x7f2430038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:13.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.433+0000 7f244a5ad640 10 monclient: shutdown 2024-09-16T14:48:13.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.433+0000 7f244a5ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:13.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.433+0000 7f244a5ad640 1 -- 172.21.15.189:0/816321174 >> v1:172.21.15.26:6790/0 conn(0x7f2444137fd0 legacy=0x7f24441cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:13.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.433+0000 7f244a5ad640 1 -- 172.21.15.189:0/816321174 shutdown_connections 2024-09-16T14:48:13.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.433+0000 7f244a5ad640 1 -- 172.21.15.189:0/816321174 >> 172.21.15.189:0/816321174 conn(0x7f24440a0340 msgr2=0x7f24440979c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:13.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.433+0000 7f244a5ad640 1 -- 172.21.15.189:0/816321174 shutdown_connections 2024-09-16T14:48:13.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.433+0000 7f244a5ad640 1 -- 172.21.15.189:0/816321174 wait complete. 2024-09-16T14:48:13.456 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight cool osd.0 11.5 2024-09-16T14:48:13.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107ac90640 10 monclient: get_monmap_and_config 2024-09-16T14:48:13.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107ac90640 10 monclient: build_initial_monmap 2024-09-16T14:48:13.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107ac90640 10 monclient: monmap: 2024-09-16T14:48:13.583 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:13.583 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:13.583 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:13.583071+0000 2024-09-16T14:48:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:13.583071+0000 2024-09-16T14:48:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:13.584 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:13.584 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107ac90640 1 Processor -- start 2024-09-16T14:48:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107ac90640 1 -- start start 2024-09-16T14:48:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107ac90640 10 monclient: init 2024-09-16T14:48:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107ac90640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107ac90640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107ac90640 10 monclient(hunting): picked mon.noname-a con 0x7f1074070830 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107ac90640 10 monclient(hunting): picked mon.noname-b con 0x7f1074071320 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107ac90640 10 monclient(hunting): picked mon.noname-c con 0x7f1074134cf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107ac90640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1074135740 con 0x7f1074070830 2024-09-16T14:48:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107ac90640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1074135160 con 0x7f1074134cf0 2024-09-16T14:48:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107ac90640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107409d810 con 0x7f1074071320 2024-09-16T14:48:13.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107ac90640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107ac90640 10 monclient(hunting): authenticate will time out at 1275.698619s 2024-09-16T14:48:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f107922d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f1074134cf0 legacy=0x7f107413d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f1078a2c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1074070830 0x7f1074070c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:48622/0 (socket says 172.21.15.189:48622) 2024-09-16T14:48:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f1078a2c640 1 -- 172.21.15.189:0/2440722276 learned_addr learned my addr 172.21.15.189:0/2440722276 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 1 -- 172.21.15.189:0/2440722276 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227519887 0 0) 0x7f1074135740 con 0x7f1074070830 2024-09-16T14:48:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 10 monclient(hunting): my global_id is 124287 2024-09-16T14:48:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 1 -- 172.21.15.189:0/2440722276 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f105c003530 con 0x7f1074070830 2024-09-16T14:48:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 1 -- 172.21.15.189:0/2440722276 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802027202 0 0) 0x7f107409d810 con 0x7f1074071320 2024-09-16T14:48:13.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 10 monclient(hunting): my global_id is 114599 2024-09-16T14:48:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 1 -- 172.21.15.189:0/2440722276 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1074135740 con 0x7f1074071320 2024-09-16T14:48:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 1 -- 172.21.15.189:0/2440722276 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3896343071 0 0) 0x7f105c003530 con 0x7f1074070830 2024-09-16T14:48:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 1 -- 172.21.15.189:0/2440722276 >> v1:172.21.15.26:6790/0 conn(0x7f1074071320 legacy=0x7f1074139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 1 -- 172.21.15.189:0/2440722276 >> v1:172.21.15.189:6789/0 conn(0x7f1074134cf0 legacy=0x7f107413d440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 10 monclient: _finish_hunting 0 2024-09-16T14:48:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.581+0000 7f10737fe640 1 monclient: found mon.noname-a 2024-09-16T14:48:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10737fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:13.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10737fe640 1 -- 172.21.15.189:0/2440722276 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107409f1c0 con 0x7f1074070830 2024-09-16T14:48:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10737fe640 10 monclient: _finish_auth 0 2024-09-16T14:48:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10737fe640 10 monclient: _check_auth_tickets 2024-09-16T14:48:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10737fe640 1 -- 172.21.15.189:0/2440722276 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1064001620 con 0x7f1074070830 2024-09-16T14:48:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 5 monclient: authenticate success, global_id 124287 2024-09-16T14:48:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10737fe640 10 monclient: dump: 2024-09-16T14:48:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:13.595 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10737fe640 1 -- 172.21.15.189:0/2440722276 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1064001900 con 0x7f1074070830 2024-09-16T14:48:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10737fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 4 set_mon_vals no callback set 2024-09-16T14:48:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10737fe640 1 -- 172.21.15.189:0/2440722276 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1064001c90 con 0x7f1074070830 2024-09-16T14:48:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10737fe640 10 monclient: dump: 2024-09-16T14:48:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:13.596 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 10 monclient: shutdown 2024-09-16T14:48:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 1 -- 172.21.15.189:0/2440722276 >> v1:172.21.15.26:6789/0 conn(0x7f1074070830 legacy=0x7f1074070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 1 -- 172.21.15.189:0/2440722276 shutdown_connections 2024-09-16T14:48:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 1 -- 172.21.15.189:0/2440722276 >> 172.21.15.189:0/2440722276 conn(0x7f10740a3380 msgr2=0x7f107409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 1 -- 172.21.15.189:0/2440722276 shutdown_connections 2024-09-16T14:48:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 1 -- 172.21.15.189:0/2440722276 wait complete. 2024-09-16T14:48:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 10 monclient: build_initial_monmap 2024-09-16T14:48:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 10 monclient: monmap: 2024-09-16T14:48:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:13.598 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:13.598 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:13.598 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:13.598 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:13.598 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:13.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 1 Processor -- start 2024-09-16T14:48:13.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 1 -- start start 2024-09-16T14:48:13.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 10 monclient: init 2024-09-16T14:48:13.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:13.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:13.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 10 monclient(hunting): picked mon.noname-a con 0x7f1074070830 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:13.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 10 monclient(hunting): picked mon.noname-c con 0x7f1074071320 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:13.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 10 monclient(hunting): picked mon.noname-b con 0x7f1074134cf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:13.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107411e5e0 con 0x7f1074070830 2024-09-16T14:48:13.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107406b320 con 0x7f1074134cf0 2024-09-16T14:48:13.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1074132c30 con 0x7f1074071320 2024-09-16T14:48:13.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:13.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 10 monclient(hunting): authenticate will time out at 1275.702321s 2024-09-16T14:48:13.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107922d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f1074134cf0 legacy=0x7f10741d1450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:13.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f1078a2c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1074070830 0x7f1074130ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:48628/0 (socket says 172.21.15.189:48628) 2024-09-16T14:48:13.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f1078a2c640 1 -- 172.21.15.189:0/2506328828 learned_addr learned my addr 172.21.15.189:0/2506328828 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:13.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354493207 0 0) 0x7f107411e5e0 con 0x7f1074070830 2024-09-16T14:48:13.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:13.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:13.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:13.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient(hunting): my global_id is 124290 2024-09-16T14:48:13.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1060003530 con 0x7f1074070830 2024-09-16T14:48:13.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712835437 0 0) 0x7f1074132c30 con 0x7f1074071320 2024-09-16T14:48:13.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:13.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:13.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:13.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient(hunting): my global_id is 114602 2024-09-16T14:48:13.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f107411e5e0 con 0x7f1074071320 2024-09-16T14:48:13.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1152952612 0 0) 0x7f1060003530 con 0x7f1074070830 2024-09-16T14:48:13.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1074132c30 con 0x7f1074070830 2024-09-16T14:48:13.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1064001c10 con 0x7f1074070830 2024-09-16T14:48:13.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:13.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:13.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient(hunting): dump: 2024-09-16T14:48:13.601 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:13.601 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:13.601 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:13.601 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:13.601 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:13.601 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:13.601 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:13.601 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:13.601 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:13.601 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:13.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:13.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 274617537 0 0) 0x7f107411e5e0 con 0x7f1074071320 2024-09-16T14:48:13.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1060003530 con 0x7f1074071320 2024-09-16T14:48:13.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10680018c0 con 0x7f1074071320 2024-09-16T14:48:13.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:13.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:13.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient(hunting): dump: 2024-09-16T14:48:13.602 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:13.602 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:13.602 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:13.602 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:13.602 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:13.602 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:13.602 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:13.602 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:13.603 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:13.603 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:13.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3519007306 0 0) 0x7f1074132c30 con 0x7f1074070830 2024-09-16T14:48:13.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 >> v1:172.21.15.26:6790/0 conn(0x7f1074071320 legacy=0x7f10741cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:13.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 >> v1:172.21.15.189:6789/0 conn(0x7f1074134cf0 legacy=0x7f10741d1450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:13.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient: _finish_hunting 0 2024-09-16T14:48:13.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 1 monclient: found mon.a 2024-09-16T14:48:13.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:13.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:13.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107411cbd0 con 0x7f1074070830 2024-09-16T14:48:13.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient: _finish_auth 0 2024-09-16T14:48:13.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 10 monclient: _check_auth_tickets 2024-09-16T14:48:13.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f10717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:13.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 5 monclient: authenticate success, global_id 124290 2024-09-16T14:48:13.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 10 monclient: _renew_subs 2024-09-16T14:48:13.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:13.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 1 -- 172.21.15.189:0/2506328828 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107411dd40 con 0x7f1074070830 2024-09-16T14:48:13.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 10 monclient: _renew_subs 2024-09-16T14:48:13.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:13.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.585+0000 7f107ac90640 1 -- 172.21.15.189:0/2506328828 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10741d2b70 con 0x7f1074070830 2024-09-16T14:48:13.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.589+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1064004aa0 con 0x7f1074070830 2024-09-16T14:48:13.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.589+0000 7f10717fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:13.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.589+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1064004e30 con 0x7f1074070830 2024-09-16T14:48:13.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.589+0000 7f10717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:13.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.589+0000 7f10717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:13.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.589+0000 7f10717fa640 10 monclient: dump: 2024-09-16T14:48:13.605 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:13.605 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:13.605 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:13.605 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:13.605 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:13.605 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:13.605 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:13.605 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:13.605 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:13.605 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:13.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.589+0000 7f1072ffd640 4 set_mon_vals no callback set 2024-09-16T14:48:13.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.589+0000 7f107ac90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:13.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.589+0000 7f107ac90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:13.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.589+0000 7f107ac90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:13.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.589+0000 7f107ac90640 1 -- 172.21.15.189:0/2506328828 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1054005050 con 0x7f1074070830 2024-09-16T14:48:13.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.589+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1064005010 con 0x7f1074070830 2024-09-16T14:48:13.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.589+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(149..149 src has 73..149) v4 ==== 8349+0+0 (unknown 2163584723 0 0) 0x7f1064044600 con 0x7f1074070830 2024-09-16T14:48:13.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.589+0000 7f1073fff640 10 monclient: get_auth_request con 0x7f1060036220 auth_method 0 2024-09-16T14:48:13.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.601+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f10640149d0 con 0x7f1074070830 2024-09-16T14:48:13.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.601+0000 7f10717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:13.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.601+0000 7f10717fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:13.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.769+0000 7f107ac90640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T14:48:13.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.769+0000 7f107ac90640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T14:48:13.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.769+0000 7f107ac90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:13.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:13.769+0000 7f107ac90640 1 -- 172.21.15.189:0/2506328828 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) v1 -- 0x7f1054005300 con 0x7f1074070830 2024-09-16T14:48:14.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:14.433+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}]=0 v150) v1 ==== 128+0+0 (unknown 406991207 0 0) 0x7f1064012e40 con 0x7f1074070830 2024-09-16T14:48:14.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:14.433+0000 7f10717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T14:48:14.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:14.433+0000 7f10717fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:14.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:14.545+0000 7f107ac90640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T14:48:14.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:14.545+0000 7f107ac90640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T14:48:14.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:14.545+0000 7f107ac90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:14.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:14.545+0000 7f107ac90640 1 -- 172.21.15.189:0/2506328828 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) v1 -- 0x7f1054005a40 con 0x7f1074070830 2024-09-16T14:48:15.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.441+0000 7f10717fa640 1 -- 172.21.15.189:0/2506328828 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}]=0 v151) v1 ==== 128+0+0 (unknown 1909522373 0 0) 0x7f10640145c0 con 0x7f1074070830 2024-09-16T14:48:15.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.441+0000 7f10717fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T14:48:15.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.441+0000 7f10717fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:15.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.445+0000 7f107ac90640 1 -- 172.21.15.189:0/2506328828 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1060036220 legacy=0x7f10600386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:15.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.445+0000 7f107ac90640 10 monclient: shutdown 2024-09-16T14:48:15.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.445+0000 7f107ac90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:15.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.445+0000 7f107ac90640 1 -- 172.21.15.189:0/2506328828 >> v1:172.21.15.26:6789/0 conn(0x7f1074070830 legacy=0x7f1074130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:15.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.445+0000 7f107ac90640 1 -- 172.21.15.189:0/2506328828 shutdown_connections 2024-09-16T14:48:15.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.445+0000 7f107ac90640 1 -- 172.21.15.189:0/2506328828 >> 172.21.15.189:0/2506328828 conn(0x7f10740a3380 msgr2=0x7f1074131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:15.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.445+0000 7f107ac90640 1 -- 172.21.15.189:0/2506328828 shutdown_connections 2024-09-16T14:48:15.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.445+0000 7f107ac90640 1 -- 172.21.15.189:0/2506328828 wait complete. 2024-09-16T14:48:15.465 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight cold osd.0 12.5 12.4 2024-09-16T14:48:15.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.589+0000 7f68df4c5640 10 monclient: get_monmap_and_config 2024-09-16T14:48:15.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.589+0000 7f68df4c5640 10 monclient: build_initial_monmap 2024-09-16T14:48:15.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.589+0000 7f68df4c5640 10 monclient: monmap: 2024-09-16T14:48:15.592 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:15.592 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:15.592 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:15.592027+0000 2024-09-16T14:48:15.592 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:15.592027+0000 2024-09-16T14:48:15.592 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:15.593 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:15.593 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:15.593 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:15.593 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:15.593 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:15.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.589+0000 7f68df4c5640 1 Processor -- start 2024-09-16T14:48:15.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.589+0000 7f68df4c5640 1 -- start start 2024-09-16T14:48:15.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.589+0000 7f68df4c5640 10 monclient: init 2024-09-16T14:48:15.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.589+0000 7f68df4c5640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:15.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.589+0000 7f68df4c5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:15.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 10 monclient(hunting): picked mon.noname-a con 0x7f68d81315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:15.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 10 monclient(hunting): picked mon.noname-b con 0x7f68d809b320 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:15.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 10 monclient(hunting): picked mon.noname-c con 0x7f68d8132090 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:15.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d809edd0 con 0x7f68d81315a0 2024-09-16T14:48:15.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d809bc00 con 0x7f68d8132090 2024-09-16T14:48:15.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68dda62640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f68d8132090 legacy=0x7f68d813a200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:15.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d809e900 con 0x7f68d809b320 2024-09-16T14:48:15.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:15.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 10 monclient(hunting): authenticate will time out at 1277.708202s 2024-09-16T14:48:15.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68dd261640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f68d81315a0 0x7f68d8131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:48640/0 (socket says 172.21.15.189:48640) 2024-09-16T14:48:15.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68dd261640 1 -- 172.21.15.189:0/857756579 learned_addr learned my addr 172.21.15.189:0/857756579 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:15.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 1 -- 172.21.15.189:0/857756579 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145613957 0 0) 0x7f68d809edd0 con 0x7f68d81315a0 2024-09-16T14:48:15.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:15.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:15.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:15.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient(hunting): my global_id is 124293 2024-09-16T14:48:15.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 1 -- 172.21.15.189:0/857756579 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68c4003530 con 0x7f68d81315a0 2024-09-16T14:48:15.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 1 -- 172.21.15.189:0/857756579 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545712990 0 0) 0x7f68d809e900 con 0x7f68d809b320 2024-09-16T14:48:15.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:15.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:15.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:15.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient(hunting): my global_id is 114605 2024-09-16T14:48:15.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 1 -- 172.21.15.189:0/857756579 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d809edd0 con 0x7f68d809b320 2024-09-16T14:48:15.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 1 -- 172.21.15.189:0/857756579 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2366920514 0 0) 0x7f68c4003530 con 0x7f68d81315a0 2024-09-16T14:48:15.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 1 -- 172.21.15.189:0/857756579 >> v1:172.21.15.26:6790/0 conn(0x7f68d809b320 legacy=0x7f68d8136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:15.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 1 -- 172.21.15.189:0/857756579 >> v1:172.21.15.189:6789/0 conn(0x7f68d8132090 legacy=0x7f68d813a200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:15.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:15.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 1 monclient: found mon.noname-a 2024-09-16T14:48:15.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:15.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:15.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 1 -- 172.21.15.189:0/857756579 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d809b920 con 0x7f68d81315a0 2024-09-16T14:48:15.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient: _finish_auth 0 2024-09-16T14:48:15.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:15.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:15.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 1 -- 172.21.15.189:0/857756579 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f68d4001620 con 0x7f68d81315a0 2024-09-16T14:48:15.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 5 monclient: authenticate success, global_id 124293 2024-09-16T14:48:15.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:15.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:15.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:15.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient: dump: 2024-09-16T14:48:15.605 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:15.605 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:15.605 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:15.605 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:15.605 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:15.605 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:15.605 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:15.605 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:15.605 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:15.605 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:15.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:15.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 1 -- 172.21.15.189:0/857756579 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68d4001900 con 0x7f68d81315a0 2024-09-16T14:48:15.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:15.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:15.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 1 -- 172.21.15.189:0/857756579 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f68d4001c90 con 0x7f68d81315a0 2024-09-16T14:48:15.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:15.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 4 set_mon_vals no callback set 2024-09-16T14:48:15.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:15.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68cffff640 10 monclient: dump: 2024-09-16T14:48:15.606 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:15.606 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:15.606 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:15.606 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:15.606 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:15.606 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:15.607 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:15.607 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:15.607 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:15.607 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:15.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 10 monclient: shutdown 2024-09-16T14:48:15.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:15.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 1 -- 172.21.15.189:0/857756579 >> v1:172.21.15.26:6789/0 conn(0x7f68d81315a0 legacy=0x7f68d8131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:15.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 1 -- 172.21.15.189:0/857756579 shutdown_connections 2024-09-16T14:48:15.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 1 -- 172.21.15.189:0/857756579 >> 172.21.15.189:0/857756579 conn(0x7f68d80a3320 msgr2=0x7f68d809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:15.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 1 -- 172.21.15.189:0/857756579 shutdown_connections 2024-09-16T14:48:15.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 1 -- 172.21.15.189:0/857756579 wait complete. 2024-09-16T14:48:15.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 10 monclient: build_initial_monmap 2024-09-16T14:48:15.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 10 monclient: monmap: 2024-09-16T14:48:15.608 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:15.608 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:15.608 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:15.608 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:15.608 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:15.608 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:15.608 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:15.608 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:15.608 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:15.608 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:15.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 1 Processor -- start 2024-09-16T14:48:15.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 1 -- start start 2024-09-16T14:48:15.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 10 monclient: init 2024-09-16T14:48:15.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:15.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.593+0000 7f68df4c5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:15.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 10 monclient(hunting): picked mon.noname-c con 0x7f68d81315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:15.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 10 monclient(hunting): picked mon.noname-b con 0x7f68d8132090 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:15.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 10 monclient(hunting): picked mon.noname-a con 0x7f68d81c6fd0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:15.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d809cd20 con 0x7f68d81c6fd0 2024-09-16T14:48:15.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68dca60640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f68d8132090 legacy=0x7f68d81cc350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:15.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d811cc60 con 0x7f68d8132090 2024-09-16T14:48:15.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d806b380 con 0x7f68d81315a0 2024-09-16T14:48:15.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:15.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 10 monclient(hunting): authenticate will time out at 1277.712110s 2024-09-16T14:48:15.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68dda62640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f68d81c6fd0 0x7f68d81cfa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:48654/0 (socket says 172.21.15.189:48654) 2024-09-16T14:48:15.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68dda62640 1 -- 172.21.15.189:0/85832774 learned_addr learned my addr 172.21.15.189:0/85832774 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:15.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445700770 0 0) 0x7f68d809cd20 con 0x7f68d81c6fd0 2024-09-16T14:48:15.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:15.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:15.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:15.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient(hunting): my global_id is 124296 2024-09-16T14:48:15.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68bc003530 con 0x7f68d81c6fd0 2024-09-16T14:48:15.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101005937 0 0) 0x7f68d806b380 con 0x7f68d81315a0 2024-09-16T14:48:15.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:15.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:15.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:15.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient(hunting): my global_id is 114608 2024-09-16T14:48:15.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d809cd20 con 0x7f68d81315a0 2024-09-16T14:48:15.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4038816668 0 0) 0x7f68bc003530 con 0x7f68d81c6fd0 2024-09-16T14:48:15.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68d806b380 con 0x7f68d81c6fd0 2024-09-16T14:48:15.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f68d00043c0 con 0x7f68d81c6fd0 2024-09-16T14:48:15.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:15.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:15.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient(hunting): dump: 2024-09-16T14:48:15.611 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:15.611 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:15.611 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:15.611 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:15.611 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:15.611 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:15.612 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:15.612 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:15.612 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:15.612 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:15.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:15.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3378186888 0 0) 0x7f68d809cd20 con 0x7f68d81315a0 2024-09-16T14:48:15.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68bc003530 con 0x7f68d81315a0 2024-09-16T14:48:15.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f68d40015e0 con 0x7f68d81315a0 2024-09-16T14:48:15.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:15.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:15.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient(hunting): dump: 2024-09-16T14:48:15.613 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:15.613 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:15.613 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:15.613 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:15.613 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:15.613 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:15.613 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:15.613 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:15.613 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:15.613 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:15.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3004713178 0 0) 0x7f68d806b380 con 0x7f68d81c6fd0 2024-09-16T14:48:15.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 >> v1:172.21.15.26:6790/0 conn(0x7f68d81315a0 legacy=0x7f68d8130a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:15.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 >> v1:172.21.15.189:6789/0 conn(0x7f68d8132090 legacy=0x7f68d81cc350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:15.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient: _finish_hunting 0 2024-09-16T14:48:15.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 monclient: found mon.a 2024-09-16T14:48:15.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:15.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:15.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d811e110 con 0x7f68d81c6fd0 2024-09-16T14:48:15.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient: _finish_auth 0 2024-09-16T14:48:15.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:15.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:15.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 5 monclient: authenticate success, global_id 124296 2024-09-16T14:48:15.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 10 monclient: _renew_subs 2024-09-16T14:48:15.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:15.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 1 -- 172.21.15.189:0/85832774 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d811c020 con 0x7f68d81c6fd0 2024-09-16T14:48:15.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 10 monclient: _renew_subs 2024-09-16T14:48:15.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:15.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 1 -- 172.21.15.189:0/85832774 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d811de30 con 0x7f68d81c6fd0 2024-09-16T14:48:15.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68d00046a0 con 0x7f68d81c6fd0 2024-09-16T14:48:15.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:15.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f68d0004a30 con 0x7f68d81c6fd0 2024-09-16T14:48:15.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:15.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:15.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cf7fe640 4 set_mon_vals no callback set 2024-09-16T14:48:15.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 10 monclient: dump: 2024-09-16T14:48:15.615 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:15.615 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:15.615 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:15.615 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:15.615 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:15.616 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:15.616 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:15.616 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:15.616 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:15.616 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:15.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:15.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:15.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:15.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68df4c5640 1 -- 172.21.15.189:0/85832774 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b8005050 con 0x7f68d81c6fd0 2024-09-16T14:48:15.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f68d0004b90 con 0x7f68d81c6fd0 2024-09-16T14:48:15.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(151..151 src has 73..151) v4 ==== 8324+0+0 (unknown 385883599 0 0) 0x7f68d00445b0 con 0x7f68d81c6fd0 2024-09-16T14:48:15.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.597+0000 7f68dd261640 10 monclient: get_auth_request con 0x7f68bc036270 auth_method 0 2024-09-16T14:48:15.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.609+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f68d0014690 con 0x7f68d81c6fd0 2024-09-16T14:48:15.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.609+0000 7f68cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:15.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.609+0000 7f68cdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:15.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.781+0000 7f68df4c5640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T14:48:15.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.781+0000 7f68df4c5640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T14:48:15.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.781+0000 7f68df4c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:15.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:15.781+0000 7f68df4c5640 1 -- 172.21.15.189:0/85832774 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]} v 0) v1 -- 0x7f68b8005300 con 0x7f68d81c6fd0 2024-09-16T14:48:16.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:16.441+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}]=0 v152) v1 ==== 134+0+0 (unknown 2469453746 0 0) 0x7f68d0012d30 con 0x7f68d81c6fd0 2024-09-16T14:48:16.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:16.441+0000 7f68cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T14:48:16.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:16.441+0000 7f68cdffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:16.549+0000 7f68df4c5640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T14:48:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:16.549+0000 7f68df4c5640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T14:48:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:16.549+0000 7f68df4c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:16.549+0000 7f68df4c5640 1 -- 172.21.15.189:0/85832774 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]} v 0) v1 -- 0x7f68b8001150 con 0x7f68d81c6fd0 2024-09-16T14:48:17.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.449+0000 7f68cdffb640 1 -- 172.21.15.189:0/85832774 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}]=0 v153) v1 ==== 134+0+0 (unknown 1646364500 0 0) 0x7f68d00144b0 con 0x7f68d81c6fd0 2024-09-16T14:48:17.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.449+0000 7f68cdffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T14:48:17.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.449+0000 7f68cdffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:17.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.453+0000 7f68df4c5640 1 -- 172.21.15.189:0/85832774 >> v1:172.21.15.189:6812/3196161582 conn(0x7f68bc036270 legacy=0x7f68bc038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:17.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.453+0000 7f68df4c5640 10 monclient: shutdown 2024-09-16T14:48:17.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.453+0000 7f68df4c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:17.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.453+0000 7f68df4c5640 1 -- 172.21.15.189:0/85832774 >> v1:172.21.15.26:6789/0 conn(0x7f68d81c6fd0 legacy=0x7f68d81cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:17.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.453+0000 7f68df4c5640 1 -- 172.21.15.189:0/85832774 shutdown_connections 2024-09-16T14:48:17.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.453+0000 7f68df4c5640 1 -- 172.21.15.189:0/85832774 >> 172.21.15.189:0/85832774 conn(0x7f68d80a3320 msgr2=0x7f68d80a1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:17.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.453+0000 7f68df4c5640 1 -- 172.21.15.189:0/85832774 shutdown_connections 2024-09-16T14:48:17.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.453+0000 7f68df4c5640 1 -- 172.21.15.189:0/85832774 wait complete. 2024-09-16T14:48:17.475 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush set-device-class fish osd.0 2024-09-16T14:48:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.593+0000 7fa93c0e5640 10 monclient: get_monmap_and_config 2024-09-16T14:48:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.593+0000 7fa93c0e5640 10 monclient: build_initial_monmap 2024-09-16T14:48:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.593+0000 7fa93c0e5640 10 monclient: monmap: 2024-09-16T14:48:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:17.596230+0000 2024-09-16T14:48:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:17.596230+0000 2024-09-16T14:48:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:17.597 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.593+0000 7fa93c0e5640 1 Processor -- start 2024-09-16T14:48:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.593+0000 7fa93c0e5640 1 -- start start 2024-09-16T14:48:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.593+0000 7fa93c0e5640 10 monclient: init 2024-09-16T14:48:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 10 monclient(hunting): picked mon.noname-b con 0x7fa9340a0930 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 10 monclient(hunting): picked mon.noname-c con 0x7fa9340a0d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 10 monclient(hunting): picked mon.noname-a con 0x7fa9340a1150 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9340a2db0 con 0x7fa9340a1150 2024-09-16T14:48:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9340a2b40 con 0x7fa9340a0d10 2024-09-16T14:48:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9340a2f80 con 0x7fa9340a0930 2024-09-16T14:48:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 10 monclient(hunting): authenticate will time out at 1279.712231s 2024-09-16T14:48:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa939680640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa9340a0d10 legacy=0x7fa934139c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa939e81640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa9340a0930 0x7fa9340a18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:34650/0 (socket says 172.21.15.189:34650) 2024-09-16T14:48:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93a682640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa9340a1150 0x7fa93413d360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:48670/0 (socket says 172.21.15.189:48670) 2024-09-16T14:48:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa939e81640 1 -- 172.21.15.189:0/1260137220 learned_addr learned my addr 172.21.15.189:0/1260137220 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 1 -- 172.21.15.189:0/1260137220 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175285987 0 0) 0x7fa9340a2db0 con 0x7fa9340a1150 2024-09-16T14:48:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient(hunting): _negotiate 2024-09-16T14:48:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient(hunting): my global_id is 124299 2024-09-16T14:48:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 1 -- 172.21.15.189:0/1260137220 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa920003530 con 0x7fa9340a1150 2024-09-16T14:48:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 1 -- 172.21.15.189:0/1260137220 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801595234 0 0) 0x7fa9340a2f80 con 0x7fa9340a0930 2024-09-16T14:48:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient(hunting): _negotiate 2024-09-16T14:48:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient(hunting): my global_id is 114611 2024-09-16T14:48:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 1 -- 172.21.15.189:0/1260137220 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9340a2db0 con 0x7fa9340a0930 2024-09-16T14:48:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 1 -- 172.21.15.189:0/1260137220 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2240725947 0 0) 0x7fa920003530 con 0x7fa9340a1150 2024-09-16T14:48:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 1 -- 172.21.15.189:0/1260137220 >> v1:172.21.15.26:6790/0 conn(0x7fa9340a0930 legacy=0x7fa9340a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 1 -- 172.21.15.189:0/1260137220 >> v1:172.21.15.189:6789/0 conn(0x7fa9340a0d10 legacy=0x7fa934139c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient: _finish_hunting 0 2024-09-16T14:48:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 1 monclient: found mon.noname-a 2024-09-16T14:48:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 1 -- 172.21.15.189:0/1260137220 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93409eb60 con 0x7fa9340a1150 2024-09-16T14:48:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient: _finish_auth 0 2024-09-16T14:48:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient: _check_auth_tickets 2024-09-16T14:48:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 1 -- 172.21.15.189:0/1260137220 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa92c0018d0 con 0x7fa9340a1150 2024-09-16T14:48:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient: dump: 2024-09-16T14:48:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:17.603 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 5 monclient: authenticate success, global_id 124299 2024-09-16T14:48:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 1 -- 172.21.15.189:0/1260137220 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa92c001b40 con 0x7fa9340a1150 2024-09-16T14:48:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 1 -- 172.21.15.189:0/1260137220 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa92c001ed0 con 0x7fa9340a1150 2024-09-16T14:48:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa938e7f640 10 monclient: dump: 2024-09-16T14:48:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:17.605 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 4 set_mon_vals no callback set 2024-09-16T14:48:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 10 monclient: shutdown 2024-09-16T14:48:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 1 -- 172.21.15.189:0/1260137220 >> v1:172.21.15.26:6789/0 conn(0x7fa9340a1150 legacy=0x7fa93413d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 1 -- 172.21.15.189:0/1260137220 shutdown_connections 2024-09-16T14:48:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 1 -- 172.21.15.189:0/1260137220 >> 172.21.15.189:0/1260137220 conn(0x7fa9340a39d0 msgr2=0x7fa9340a3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 1 -- 172.21.15.189:0/1260137220 shutdown_connections 2024-09-16T14:48:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 1 -- 172.21.15.189:0/1260137220 wait complete. 2024-09-16T14:48:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 10 monclient: build_initial_monmap 2024-09-16T14:48:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 10 monclient: monmap: 2024-09-16T14:48:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:17.611 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 1 Processor -- start 2024-09-16T14:48:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 1 -- start start 2024-09-16T14:48:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.597+0000 7fa93c0e5640 10 monclient: init 2024-09-16T14:48:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 10 monclient(hunting): picked mon.noname-c con 0x7fa9340a0930 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 10 monclient(hunting): picked mon.noname-a con 0x7fa9340a0d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 10 monclient(hunting): picked mon.noname-b con 0x7fa9340a1150 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9340a5e80 con 0x7fa9340a0d10 2024-09-16T14:48:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93412d2c0 con 0x7fa9340a1150 2024-09-16T14:48:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93409c7f0 con 0x7fa9340a0930 2024-09-16T14:48:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 10 monclient(hunting): authenticate will time out at 1279.716504s 2024-09-16T14:48:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93a682640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa9340a1150 legacy=0x7fa93409ba30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:48:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa939e81640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa9340a0930 0x7fa93409ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:34654/0 (socket says 172.21.15.189:34654) 2024-09-16T14:48:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa939e81640 1 -- 172.21.15.189:0/2712373836 learned_addr learned my addr 172.21.15.189:0/2712373836 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490906124 0 0) 0x7fa93409c7f0 con 0x7fa9340a0930 2024-09-16T14:48:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient(hunting): my global_id is 114614 2024-09-16T14:48:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa918003530 con 0x7fa9340a0930 2024-09-16T14:48:17.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801541367 0 0) 0x7fa9340a5e80 con 0x7fa9340a0d10 2024-09-16T14:48:17.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:17.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:17.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:17.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient(hunting): my global_id is 124302 2024-09-16T14:48:17.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa93409c7f0 con 0x7fa9340a0d10 2024-09-16T14:48:17.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3137042772 0 0) 0x7fa93409c7f0 con 0x7fa9340a0d10 2024-09-16T14:48:17.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9340a5e80 con 0x7fa9340a0d10 2024-09-16T14:48:17.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa9300040d0 con 0x7fa9340a0d10 2024-09-16T14:48:17.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:17.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:17.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient(hunting): dump: 2024-09-16T14:48:17.614 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:17.614 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:17.614 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:17.614 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:17.614 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:17.614 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:17.614 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:17.614 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:17.614 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:17.614 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:17.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:17.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 738384310 0 0) 0x7fa918003530 con 0x7fa9340a0930 2024-09-16T14:48:17.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa93409c7f0 con 0x7fa9340a0930 2024-09-16T14:48:17.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa924001900 con 0x7fa9340a0930 2024-09-16T14:48:17.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:17.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:17.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient(hunting): dump: 2024-09-16T14:48:17.615 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:17.615 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:17.615 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:17.615 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:17.615 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:17.615 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:17.615 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:17.615 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:17.615 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:17.616 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:17.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 296011443 0 0) 0x7fa9340a5e80 con 0x7fa9340a0d10 2024-09-16T14:48:17.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 >> v1:172.21.15.26:6790/0 conn(0x7fa9340a0930 legacy=0x7fa93409ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:17.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 >> v1:172.21.15.189:6789/0 conn(0x7fa9340a1150 legacy=0x7fa93409ba30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:17.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:17.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 monclient: found mon.a 2024-09-16T14:48:17.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:17.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:17.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93409e130 con 0x7fa9340a0d10 2024-09-16T14:48:17.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:17.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:17.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:17.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 5 monclient: authenticate success, global_id 124302 2024-09-16T14:48:17.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 10 monclient: _renew_subs 2024-09-16T14:48:17.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:17.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 1 -- 172.21.15.189:0/2712373836 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa93412db60 con 0x7fa9340a0d10 2024-09-16T14:48:17.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 10 monclient: _renew_subs 2024-09-16T14:48:17.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:17.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 1 -- 172.21.15.189:0/2712373836 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa93409d3c0 con 0x7fa9340a0d10 2024-09-16T14:48:17.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9300046a0 con 0x7fa9340a0d10 2024-09-16T14:48:17.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:17.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa930004a30 con 0x7fa9340a0d10 2024-09-16T14:48:17.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:17.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:17.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92bfff640 4 set_mon_vals no callback set 2024-09-16T14:48:17.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 10 monclient: dump: 2024-09-16T14:48:17.618 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:17.618 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:17.618 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:17.618 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:17.618 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:17.618 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:17.618 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:17.618 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:17.618 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:17.618 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:17.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:17.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:17.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:17.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa93c0e5640 1 -- 172.21.15.189:0/2712373836 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa91c005050 con 0x7fa9340a0d10 2024-09-16T14:48:17.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa930004b90 con 0x7fa9340a0d10 2024-09-16T14:48:17.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(153..153 src has 73..153) v4 ==== 8324+0+0 (unknown 1409029903 0 0) 0x7fa930044270 con 0x7fa9340a0d10 2024-09-16T14:48:17.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.601+0000 7fa939e81640 10 monclient: get_auth_request con 0x7fa918036250 auth_method 0 2024-09-16T14:48:17.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.613+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa930014350 con 0x7fa9340a0d10 2024-09-16T14:48:17.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.613+0000 7fa92a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:17.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.613+0000 7fa92a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:17.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.781+0000 7fa93c0e5640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T14:48:17.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.781+0000 7fa93c0e5640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T14:48:17.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.781+0000 7fa93c0e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:17.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:17.781+0000 7fa93c0e5640 1 -- 172.21.15.189:0/2712373836 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) v1 -- 0x7fa91c005300 con 0x7fa9340a0d10 2024-09-16T14:48:17.917 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:48:17.917 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T14:48:17.917 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T14:48:17.917 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T14:48:17.920 INFO:tasks.ceph.mon.b:Started 2024-09-16T14:48:17.920 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:48:17.920 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:48:18.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78fb1c8640 10 monclient: get_monmap_and_config 2024-09-16T14:48:18.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78fb1c8640 10 monclient: build_initial_monmap 2024-09-16T14:48:18.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78fb1c8640 10 monclient: monmap: 2024-09-16T14:48:18.054 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:18.054 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.054 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:18.054432+0000 2024-09-16T14:48:18.054 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:18.054432+0000 2024-09-16T14:48:18.054 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:18.054 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.055 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:18.055 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:18.055 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:18.055 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78fb1c8640 1 Processor -- start 2024-09-16T14:48:18.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78fb1c8640 1 -- start start 2024-09-16T14:48:18.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78fb1c8640 10 monclient: init 2024-09-16T14:48:18.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78fb1c8640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:18.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78fb1c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:18.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78fb1c8640 10 monclient(hunting): picked mon.noname-a con 0x7f78f40a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:18.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78fb1c8640 10 monclient(hunting): picked mon.noname-b con 0x7f78f40a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:18.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78fb1c8640 10 monclient(hunting): picked mon.noname-c con 0x7f78f40a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:18.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78fb1c8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f40a3250 con 0x7f78f40a1d10 2024-09-16T14:48:18.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78fb1c8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f409f600 con 0x7f78f40a2800 2024-09-16T14:48:18.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78fb1c8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f40a3830 con 0x7f78f40a0000 2024-09-16T14:48:18.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78fb1c8640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:18.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78fb1c8640 10 monclient(hunting): authenticate will time out at 1322.174492s 2024-09-16T14:48:18.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78f8f64640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f78f40a1d10 0x7f78f40a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50078/0 (socket says 172.21.15.26:50078) 2024-09-16T14:48:18.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78f8f64640 1 -- 172.21.15.26:0/2741353102 learned_addr learned my addr 172.21.15.26:0/2741353102 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:18.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.051+0000 7f78f3fff640 1 --1- 172.21.15.26:0/2741353102 >> v1:172.21.15.26:6790/0 conn(0x7f78f40a0000 0x7f78f4139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:52980/0 (socket says 172.21.15.26:52980) 2024-09-16T14:48:18.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 1 -- 172.21.15.26:0/2741353102 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989446675 0 0) 0x7f78f40a3250 con 0x7f78f40a1d10 2024-09-16T14:48:18.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 10 monclient(hunting): my global_id is 124305 2024-09-16T14:48:18.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 1 -- 172.21.15.26:0/2741353102 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78dc003530 con 0x7f78f40a1d10 2024-09-16T14:48:18.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 1 -- 172.21.15.26:0/2741353102 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2489494383 0 0) 0x7f78dc003530 con 0x7f78f40a1d10 2024-09-16T14:48:18.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 1 -- 172.21.15.26:0/2741353102 >> v1:172.21.15.26:6790/0 conn(0x7f78f40a0000 legacy=0x7f78f4139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 1 -- 172.21.15.26:0/2741353102 >> v1:172.21.15.189:6789/0 conn(0x7f78f40a2800 legacy=0x7f78f413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:48:18.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 1 monclient: found mon.noname-a 2024-09-16T14:48:18.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:18.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:18.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 1 -- 172.21.15.26:0/2741353102 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f409ed60 con 0x7f78f40a1d10 2024-09-16T14:48:18.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 10 monclient: _finish_auth 0 2024-09-16T14:48:18.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:48:18.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:18.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 1 -- 172.21.15.26:0/2741353102 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f78e40019a0 con 0x7f78f40a1d10 2024-09-16T14:48:18.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:18.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 10 monclient: dump: 2024-09-16T14:48:18.069 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.069 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.069 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.069 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.069 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.069 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.069 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.069 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.069 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.069 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 5 monclient: authenticate success, global_id 124305 2024-09-16T14:48:18.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:18.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 1 -- 172.21.15.26:0/2741353102 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78e4001cc0 con 0x7f78f40a1d10 2024-09-16T14:48:18.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:18.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 1 -- 172.21.15.26:0/2741353102 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f78e4002050 con 0x7f78f40a1d10 2024-09-16T14:48:18.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:18.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f37fe640 10 monclient: dump: 2024-09-16T14:48:18.070 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.070 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.070 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.070 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.070 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.070 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.070 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.071 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.071 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.071 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:18.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 4 set_mon_vals no callback set 2024-09-16T14:48:18.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 10 monclient: shutdown 2024-09-16T14:48:18.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:18.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 1 -- 172.21.15.26:0/2741353102 >> v1:172.21.15.26:6789/0 conn(0x7f78f40a1d10 legacy=0x7f78f40a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 1 -- 172.21.15.26:0/2741353102 shutdown_connections 2024-09-16T14:48:18.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 1 -- 172.21.15.26:0/2741353102 >> 172.21.15.26:0/2741353102 conn(0x7f78f40a7f30 msgr2=0x7f78f40a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:18.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 1 -- 172.21.15.26:0/2741353102 shutdown_connections 2024-09-16T14:48:18.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 1 -- 172.21.15.26:0/2741353102 wait complete. 2024-09-16T14:48:18.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 10 monclient: build_initial_monmap 2024-09-16T14:48:18.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 10 monclient: monmap: 2024-09-16T14:48:18.075 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:18.075 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:18.075 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:18.076 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:18.076 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:18.076 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:18.076 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:18.076 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:18.076 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:18.076 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 1 Processor -- start 2024-09-16T14:48:18.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 1 -- start start 2024-09-16T14:48:18.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 10 monclient: init 2024-09-16T14:48:18.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:18.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:18.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 10 monclient(hunting): picked mon.noname-c con 0x7f78f40a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:18.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 10 monclient(hunting): picked mon.noname-b con 0x7f78f40a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:18.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 10 monclient(hunting): picked mon.noname-a con 0x7f78f40a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:18.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f413d250 con 0x7f78f40a2800 2024-09-16T14:48:18.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f413a070 con 0x7f78f40a1d10 2024-09-16T14:48:18.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f409e500 con 0x7f78f40a0000 2024-09-16T14:48:18.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:18.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78fb1c8640 10 monclient(hunting): authenticate will time out at 1322.187729s 2024-09-16T14:48:18.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f9765640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f78f40a2800 0x7f78f41d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50092/0 (socket says 172.21.15.26:50092) 2024-09-16T14:48:18.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f8f64640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f78f40a0000 0x7f78f4139250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:52982/0 (socket says 172.21.15.26:52982) 2024-09-16T14:48:18.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.063+0000 7f78f9765640 1 -- 172.21.15.26:0/3335007658 learned_addr learned my addr 172.21.15.26:0/3335007658 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:18.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 1 -- 172.21.15.26:0/3335007658 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105239023 0 0) 0x7f78f409e500 con 0x7f78f40a0000 2024-09-16T14:48:18.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 10 monclient(hunting): my global_id is 114617 2024-09-16T14:48:18.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 1 -- 172.21.15.26:0/3335007658 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78e0003530 con 0x7f78f40a0000 2024-09-16T14:48:18.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 1 -- 172.21.15.26:0/3335007658 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1712482841 0 0) 0x7f78e0003530 con 0x7f78f40a0000 2024-09-16T14:48:18.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 1 -- 172.21.15.26:0/3335007658 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78f409e500 con 0x7f78f40a0000 2024-09-16T14:48:18.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 1 -- 172.21.15.26:0/3335007658 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f78e4001960 con 0x7f78f40a0000 2024-09-16T14:48:18.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:18.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 10 monclient(hunting): dump: 2024-09-16T14:48:18.078 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.078 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.078 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.078 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.079 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.079 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.079 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.079 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.079 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.079 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:18.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 1 -- 172.21.15.26:0/3335007658 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1227023381 0 0) 0x7f78f409e500 con 0x7f78f40a0000 2024-09-16T14:48:18.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 1 -- 172.21.15.26:0/3335007658 >> v1:172.21.15.189:6789/0 conn(0x7f78f40a1d10 legacy=0x7f78f4139960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 1 -- 172.21.15.26:0/3335007658 >> v1:172.21.15.26:6789/0 conn(0x7f78f40a2800 legacy=0x7f78f41d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 10 monclient: _finish_hunting 0 2024-09-16T14:48:18.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 1 monclient: found mon.c 2024-09-16T14:48:18.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:18.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:18.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 1 -- 172.21.15.26:0/3335007658 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f40a6860 con 0x7f78f40a0000 2024-09-16T14:48:18.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 10 monclient: _finish_auth 0 2024-09-16T14:48:18.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 10 monclient: _check_auth_tickets 2024-09-16T14:48:18.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:18.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78fb1c8640 5 monclient: authenticate success, global_id 114617 2024-09-16T14:48:18.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78fb1c8640 10 monclient: _renew_subs 2024-09-16T14:48:18.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78fb1c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:18.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78fb1c8640 1 -- 172.21.15.26:0/3335007658 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f413b840 con 0x7f78f40a0000 2024-09-16T14:48:18.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78fb1c8640 10 monclient: _renew_subs 2024-09-16T14:48:18.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78fb1c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:18.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78fb1c8640 1 -- 172.21.15.26:0/3335007658 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f413aef0 con 0x7f78f40a0000 2024-09-16T14:48:18.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 1 -- 172.21.15.26:0/3335007658 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78e40015c0 con 0x7f78f40a0000 2024-09-16T14:48:18.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:18.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 1 -- 172.21.15.26:0/3335007658 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f78e40051f0 con 0x7f78f40a0000 2024-09-16T14:48:18.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:18.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f2ffd640 4 set_mon_vals no callback set 2024-09-16T14:48:18.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 10 monclient: dump: 2024-09-16T14:48:18.081 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.081 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.081 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.081 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.082 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.082 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.082 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.082 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.082 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.082 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78fb1c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:18.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78fb1c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:18.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78fb1c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:18.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 1 -- 172.21.15.26:0/3335007658 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f78e40053d0 con 0x7f78f40a0000 2024-09-16T14:48:18.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78fb1c8640 1 -- 172.21.15.26:0/3335007658 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d4005050 con 0x7f78f40a0000 2024-09-16T14:48:18.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f17fa640 1 -- 172.21.15.26:0/3335007658 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(153..153 src has 73..153) v4 ==== 8324+0+0 (unknown 1409029903 0 0) 0x7f78e4044a60 con 0x7f78f40a0000 2024-09-16T14:48:18.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.067+0000 7f78f3fff640 10 monclient: get_auth_request con 0x7f78e0036060 auth_method 0 2024-09-16T14:48:18.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.071+0000 7f78f17fa640 1 -- 172.21.15.26:0/3335007658 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f78e4014e90 con 0x7f78f40a0000 2024-09-16T14:48:18.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.071+0000 7f78f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:18.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.071+0000 7f78f17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:18.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.295+0000 7f78fb1c8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:48:18.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.295+0000 7f78fb1c8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:48:18.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.295+0000 7f78fb1c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:18.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.295+0000 7f78fb1c8640 1 -- 172.21.15.26:0/3335007658 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f78d4005300 con 0x7f78f40a0000 2024-09-16T14:48:18.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.295+0000 7f78f17fa640 1 -- 172.21.15.26:0/3335007658 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2952142209) 0x7f78e4013300 con 0x7f78f40a0000 2024-09-16T14:48:18.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.295+0000 7f78f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:48:18.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.295+0000 7f78f17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:18.299 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:48:18.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.295+0000 7f78fb1c8640 1 -- 172.21.15.26:0/3335007658 >> v1:172.21.15.189:6812/3196161582 conn(0x7f78e0036060 legacy=0x7f78e0038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.295+0000 7f78fb1c8640 10 monclient: shutdown 2024-09-16T14:48:18.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.295+0000 7f78fb1c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:18.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.295+0000 7f78fb1c8640 1 -- 172.21.15.26:0/3335007658 >> v1:172.21.15.26:6790/0 conn(0x7f78f40a0000 legacy=0x7f78f4139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.295+0000 7f78fb1c8640 1 -- 172.21.15.26:0/3335007658 shutdown_connections 2024-09-16T14:48:18.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.295+0000 7f78fb1c8640 1 -- 172.21.15.26:0/3335007658 >> 172.21.15.26:0/3335007658 conn(0x7f78f40a7f30 msgr2=0x7f78f409d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:18.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.295+0000 7f78fb1c8640 1 -- 172.21.15.26:0/3335007658 shutdown_connections 2024-09-16T14:48:18.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.295+0000 7f78fb1c8640 1 -- 172.21.15.26:0/3335007658 wait complete. 2024-09-16T14:48:18.317 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:48:18.318 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:48:18.318 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:48:18.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d659d0640 10 monclient: get_monmap_and_config 2024-09-16T14:48:18.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d659d0640 10 monclient: build_initial_monmap 2024-09-16T14:48:18.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d659d0640 10 monclient: monmap: 2024-09-16T14:48:18.434 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:18.434 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.434 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:18.434604+0000 2024-09-16T14:48:18.434 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:18.434604+0000 2024-09-16T14:48:18.434 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:18.434 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.434 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:18.435 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:18.435 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:18.435 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d659d0640 1 Processor -- start 2024-09-16T14:48:18.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d659d0640 1 -- start start 2024-09-16T14:48:18.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d659d0640 10 monclient: init 2024-09-16T14:48:18.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d659d0640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:18.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d659d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:18.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d659d0640 10 monclient(hunting): picked mon.noname-b con 0x7f5d600a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:18.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d659d0640 10 monclient(hunting): picked mon.noname-c con 0x7f5d600a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:18.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d659d0640 10 monclient(hunting): picked mon.noname-a con 0x7f5d600a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d659d0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d600a3250 con 0x7f5d600a2800 2024-09-16T14:48:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d659d0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6009f600 con 0x7f5d600a0000 2024-09-16T14:48:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d659d0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d600a3830 con 0x7f5d600a1d10 2024-09-16T14:48:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d659d0640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d659d0640 10 monclient(hunting): authenticate will time out at 1322.555157s 2024-09-16T14:48:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d5effd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5d600a1d10 0x7f5d600a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:52986/0 (socket says 172.21.15.26:52986) 2024-09-16T14:48:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d5effd640 1 -- 172.21.15.26:0/2429896594 learned_addr learned my addr 172.21.15.26:0/2429896594 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d5dffb640 1 -- 172.21.15.26:0/2429896594 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771195880 0 0) 0x7f5d600a3250 con 0x7f5d600a2800 2024-09-16T14:48:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d5dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d5dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d5dffb640 10 monclient(hunting): my global_id is 124308 2024-09-16T14:48:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d5dffb640 1 -- 172.21.15.26:0/2429896594 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d48003530 con 0x7f5d600a2800 2024-09-16T14:48:18.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d5dffb640 1 -- 172.21.15.26:0/2429896594 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799622280 0 0) 0x7f5d600a3830 con 0x7f5d600a1d10 2024-09-16T14:48:18.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d5dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d5dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d5dffb640 10 monclient(hunting): my global_id is 114620 2024-09-16T14:48:18.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.431+0000 7f5d5dffb640 1 -- 172.21.15.26:0/2429896594 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d600a3250 con 0x7f5d600a1d10 2024-09-16T14:48:18.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 1 -- 172.21.15.26:0/2429896594 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292498775 0 0) 0x7f5d6009f600 con 0x7f5d600a0000 2024-09-16T14:48:18.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 10 monclient(hunting): my global_id is 144100 2024-09-16T14:48:18.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 1 -- 172.21.15.26:0/2429896594 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d600a3830 con 0x7f5d600a0000 2024-09-16T14:48:18.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 1 -- 172.21.15.26:0/2429896594 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1129071227 0 0) 0x7f5d48003530 con 0x7f5d600a2800 2024-09-16T14:48:18.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 1 -- 172.21.15.26:0/2429896594 >> v1:172.21.15.26:6790/0 conn(0x7f5d600a1d10 legacy=0x7f5d600a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 1 -- 172.21.15.26:0/2429896594 >> v1:172.21.15.189:6789/0 conn(0x7f5d600a0000 legacy=0x7f5d60139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:48:18.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 1 monclient: found mon.noname-a 2024-09-16T14:48:18.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:18.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:18.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 1 -- 172.21.15.26:0/2429896594 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6009ed60 con 0x7f5d600a2800 2024-09-16T14:48:18.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 10 monclient: _finish_auth 0 2024-09-16T14:48:18.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:18.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:18.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 1 -- 172.21.15.26:0/2429896594 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5d580014d0 con 0x7f5d600a2800 2024-09-16T14:48:18.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 5 monclient: authenticate success, global_id 124308 2024-09-16T14:48:18.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:18.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:18.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 10 monclient: dump: 2024-09-16T14:48:18.442 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.442 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.442 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.442 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.442 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.442 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.442 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.442 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.442 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.442 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:18.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 1 -- 172.21.15.26:0/2429896594 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d580016f0 con 0x7f5d600a2800 2024-09-16T14:48:18.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:18.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:18.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 4 set_mon_vals no callback set 2024-09-16T14:48:18.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 1 -- 172.21.15.26:0/2429896594 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5d58001ac0 con 0x7f5d600a2800 2024-09-16T14:48:18.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:18.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5dffb640 10 monclient: dump: 2024-09-16T14:48:18.443 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.443 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.443 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.443 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.443 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.443 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.444 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.444 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.444 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.444 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 10 monclient: shutdown 2024-09-16T14:48:18.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:18.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 1 -- 172.21.15.26:0/2429896594 >> v1:172.21.15.26:6789/0 conn(0x7f5d600a2800 legacy=0x7f5d6013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 1 -- 172.21.15.26:0/2429896594 shutdown_connections 2024-09-16T14:48:18.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 1 -- 172.21.15.26:0/2429896594 >> 172.21.15.26:0/2429896594 conn(0x7f5d600a7f30 msgr2=0x7f5d600a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:18.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 1 -- 172.21.15.26:0/2429896594 shutdown_connections 2024-09-16T14:48:18.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 1 -- 172.21.15.26:0/2429896594 wait complete. 2024-09-16T14:48:18.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 10 monclient: build_initial_monmap 2024-09-16T14:48:18.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 10 monclient: monmap: 2024-09-16T14:48:18.445 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:18.445 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:18.445 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:18.445 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:18.445 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:18.445 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:18.445 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:18.445 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:18.445 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:18.445 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 1 Processor -- start 2024-09-16T14:48:18.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 1 -- start start 2024-09-16T14:48:18.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 10 monclient: init 2024-09-16T14:48:18.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:18.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:18.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 10 monclient(hunting): picked mon.noname-c con 0x7f5d600a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:18.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 10 monclient(hunting): picked mon.noname-b con 0x7f5d600a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:18.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 10 monclient(hunting): picked mon.noname-a con 0x7f5d600a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:18.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6013d0c0 con 0x7f5d600a2800 2024-09-16T14:48:18.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d600a6600 con 0x7f5d600a1d10 2024-09-16T14:48:18.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d60139f30 con 0x7f5d600a0000 2024-09-16T14:48:18.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:18.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 10 monclient(hunting): authenticate will time out at 1322.558485s 2024-09-16T14:48:18.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5effd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5d600a0000 0x7f5d60139110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53002/0 (socket says 172.21.15.26:53002) 2024-09-16T14:48:18.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5effd640 1 -- 172.21.15.26:0/1070145430 learned_addr learned my addr 172.21.15.26:0/1070145430 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:18.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212501998 0 0) 0x7f5d6013d0c0 con 0x7f5d600a2800 2024-09-16T14:48:18.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): my global_id is 124311 2024-09-16T14:48:18.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d4c003560 con 0x7f5d600a2800 2024-09-16T14:48:18.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647270792 0 0) 0x7f5d60139f30 con 0x7f5d600a0000 2024-09-16T14:48:18.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): my global_id is 114623 2024-09-16T14:48:18.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d6013d0c0 con 0x7f5d600a0000 2024-09-16T14:48:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4084711371 0 0) 0x7f5d4c003560 con 0x7f5d600a2800 2024-09-16T14:48:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d60139f30 con 0x7f5d600a2800 2024-09-16T14:48:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5d58001a00 con 0x7f5d600a2800 2024-09-16T14:48:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): dump: 2024-09-16T14:48:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.453 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.454 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 822082022 0 0) 0x7f5d6013d0c0 con 0x7f5d600a0000 2024-09-16T14:48:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d4c003560 con 0x7f5d600a0000 2024-09-16T14:48:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5d50001d60 con 0x7f5d600a0000 2024-09-16T14:48:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): dump: 2024-09-16T14:48:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.455 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807499416 0 0) 0x7f5d600a6600 con 0x7f5d600a1d10 2024-09-16T14:48:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient(hunting): my global_id is 144103 2024-09-16T14:48:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d6013d0c0 con 0x7f5d600a1d10 2024-09-16T14:48:18.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3820229483 0 0) 0x7f5d60139f30 con 0x7f5d600a2800 2024-09-16T14:48:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 >> v1:172.21.15.26:6790/0 conn(0x7f5d600a0000 legacy=0x7f5d60139110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 >> v1:172.21.15.189:6789/0 conn(0x7f5d600a1d10 legacy=0x7f5d60139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 monclient: found mon.a 2024-09-16T14:48:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6009e500 con 0x7f5d600a2800 2024-09-16T14:48:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient: _finish_auth 0 2024-09-16T14:48:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 5 monclient: authenticate success, global_id 124311 2024-09-16T14:48:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 10 monclient: _renew_subs 2024-09-16T14:48:18.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:18.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 1 -- 172.21.15.26:0/1070145430 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d6013b6b0 con 0x7f5d600a2800 2024-09-16T14:48:18.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 10 monclient: _renew_subs 2024-09-16T14:48:18.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:18.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d659d0640 1 -- 172.21.15.26:0/1070145430 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d6013ad60 con 0x7f5d600a2800 2024-09-16T14:48:18.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d580038b0 con 0x7f5d600a2800 2024-09-16T14:48:18.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:18.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5d58003bc0 con 0x7f5d600a2800 2024-09-16T14:48:18.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:18.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d5cff9640 4 set_mon_vals no callback set 2024-09-16T14:48:18.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.435+0000 7f5d47fff640 10 monclient: dump: 2024-09-16T14:48:18.457 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.457 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.458 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.458 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.458 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.458 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.458 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.458 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.458 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.458 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d659d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:18.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d659d0640 1 -- 172.21.15.26:0/1070145430 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5d40006590 con 0x7f5d40002830 2024-09-16T14:48:18.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5d5800e630 con 0x7f5d600a2800 2024-09-16T14:48:18.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(153..153 src has 73..153) v4 ==== 8324+0+0 (unknown 1409029903 0 0) 0x7f5d58043370 con 0x7f5d600a2800 2024-09-16T14:48:18.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775023332 0 0) 0x7f5d40006590 con 0x7f5d40002830 2024-09-16T14:48:18.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 10 monclient(hunting): my global_id is 124314 2024-09-16T14:48:18.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d60139f30 con 0x7f5d40002830 2024-09-16T14:48:18.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d5e7fc640 10 monclient: get_auth_request con 0x7f5d4c0363f0 auth_method 0 2024-09-16T14:48:18.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 390670713 0 0) 0x7f5d60139f30 con 0x7f5d40002830 2024-09-16T14:48:18.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d40006590 con 0x7f5d40002830 2024-09-16T14:48:18.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5d50004d10 con 0x7f5d40002830 2024-09-16T14:48:18.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:18.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 10 monclient: dump: 2024-09-16T14:48:18.460 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.460 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.460 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.460 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.460 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.460 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.460 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.460 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.460 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.460 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 764913038 0 0) 0x7f5d40006590 con 0x7f5d40002830 2024-09-16T14:48:18.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d4000a0e0 con 0x7f5d40002830 2024-09-16T14:48:18.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5d50007080 con 0x7f5d40002830 2024-09-16T14:48:18.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:18.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:18.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 >> v1:172.21.15.26:6789/0 conn(0x7f5d40002830 legacy=0x7f5d40004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.439+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 >> v1:172.21.15.26:6789/0 conn(0x7f5d40002830 legacy=0x7f5d40004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:18.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.443+0000 7f5d659d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:48:18.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.443+0000 7f5d659d0640 1 -- 172.21.15.26:0/1070145430 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5d40009dc0 con 0x7f5d40009080 2024-09-16T14:48:18.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331311327 0 0) 0x7f5d40009dc0 con 0x7f5d40009080 2024-09-16T14:48:18.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 10 monclient(hunting): my global_id is 124317 2024-09-16T14:48:18.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d40006590 con 0x7f5d40009080 2024-09-16T14:48:18.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3947847139 0 0) 0x7f5d40006590 con 0x7f5d40009080 2024-09-16T14:48:18.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d40009dc0 con 0x7f5d40009080 2024-09-16T14:48:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5d50005450 con 0x7f5d40009080 2024-09-16T14:48:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 10 monclient: dump: 2024-09-16T14:48:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.462 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.462 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4012668183 0 0) 0x7f5d40009dc0 con 0x7f5d40009080 2024-09-16T14:48:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5d4000c2d0 con 0x7f5d40009080 2024-09-16T14:48:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 2682155171) 0x7f5d50006920 con 0x7f5d40009080 2024-09-16T14:48:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:48:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 >> v1:172.21.15.26:6789/0 conn(0x7f5d40009080 legacy=0x7f5d40009510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d47fff640 1 -- 172.21.15.26:0/1070145430 >> v1:172.21.15.26:6789/0 conn(0x7f5d40009080 legacy=0x7f5d40009510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:18.463 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:48:18.463 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:18.464 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:18.464 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:48:18.464 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 54, 2024-09-16T14:48:18.464 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:48:18.464 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:48:18.464 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:48:18.464 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:48:18.464 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:18.464 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:48:18.464 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:18.464 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:48:18.464 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:48:18.465 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:18.465 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:18.465 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:18.465 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:18.465 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:18.465 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:18.465 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:18.465 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:18.465 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:18.465 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:18.465 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:18.465 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:48:18.465 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:48:18.466 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:18.466 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:18.466 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:18.466 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:18.466 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:18.466 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:18.466 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:18.466 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:18.466 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:18.466 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:18.466 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.467 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:48:18.467 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:48:18.467 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:48:18.467 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:48:18.467 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:48:18.467 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:48:18.467 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:18.467 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:18.467 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:48:18.467 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:48:18.467 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:48:18.467 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:48:18.467 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:48:18.468 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:48:18.468 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:48:18.468 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:18.468 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:48:18.468 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:18.468 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:18.468 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:18.468 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:18.468 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:18.468 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:18.468 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:18.468 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:18.468 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:18.469 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:18.469 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:18.469 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:48:18.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.469 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:48:18.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.469 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:18.469 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:18.469 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:18.469 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:18.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.469 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:18.469 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:48:18.470 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:18.470 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.470 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.470 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:48:18.470 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:48:18.470 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:18.470 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:18.470 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:18.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.470 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:18.470 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:18.470 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:18.471 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:18.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.471 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:18.471 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:48:18.471 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:18.471 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.471 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.471 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:48:18.471 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:48:18.471 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:18.471 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:18.472 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:18.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.472 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:18.472 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:18.472 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:18.472 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:18.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.472 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:18.472 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:48:18.472 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:18.472 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.472 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.473 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:48:18.473 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:48:18.473 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:18.473 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:18.473 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:18.473 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.473 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.473 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:48:18.473 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:48:18.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.473 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:18.474 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:18.474 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:18.474 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.474 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:18.474 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:48:18.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.474 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:18.474 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:18.474 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:18.474 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.474 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:18.474 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:48:18.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.474 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:18.475 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:18.475 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:48:18.475 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.475 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:18.475 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:48:18.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.475 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:18.475 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:18.475 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:18.475 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.475 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.475 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:48:18.476 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:48:18.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d659d0640 1 -- 172.21.15.26:0/1070145430 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5d4c0363f0 legacy=0x7f5d4c0388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d659d0640 10 monclient: shutdown 2024-09-16T14:48:18.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d659d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:18.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d659d0640 1 -- 172.21.15.26:0/1070145430 >> v1:172.21.15.26:6789/0 conn(0x7f5d600a2800 legacy=0x7f5d601d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d5f7fe640 1 -- 172.21.15.26:0/1070145430 reap_dead start 2024-09-16T14:48:18.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d5f7fe640 1 -- 172.21.15.26:0/1070145430 reap_dead start 2024-09-16T14:48:18.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d659d0640 1 -- 172.21.15.26:0/1070145430 shutdown_connections 2024-09-16T14:48:18.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d659d0640 1 -- 172.21.15.26:0/1070145430 >> 172.21.15.26:0/1070145430 conn(0x7f5d600a7f30 msgr2=0x7f5d6013e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:18.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d659d0640 1 -- 172.21.15.26:0/1070145430 shutdown_connections 2024-09-16T14:48:18.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.447+0000 7f5d659d0640 1 -- 172.21.15.26:0/1070145430 wait complete. 2024-09-16T14:48:18.478 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:48:18.478 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:48:18.478 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:48:18.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:18.477+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'fish' v154) v1 ==== 137+0+0 (unknown 3232080401 0 0) 0x7fa930014170 con 0x7fa9340a0d10 2024-09-16T14:48:18.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:18.477+0000 7fa92a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T14:48:18.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:18.477+0000 7fa92a7fc640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'fish' 2024-09-16T14:48:18.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:18.581+0000 7fa93c0e5640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T14:48:18.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:18.581+0000 7fa93c0e5640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T14:48:18.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:18.581+0000 7fa93c0e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:18.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:18.581+0000 7fa93c0e5640 1 -- 172.21.15.189:0/2712373836 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) v1 -- 0x7fa91c001170 con 0x7fa9340a0d10 2024-09-16T14:48:18.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 10 monclient: get_monmap_and_config 2024-09-16T14:48:18.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 10 monclient: build_initial_monmap 2024-09-16T14:48:18.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 10 monclient: monmap: 2024-09-16T14:48:18.593 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:18.594 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.594 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:18.594139+0000 2024-09-16T14:48:18.594 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:18.594139+0000 2024-09-16T14:48:18.594 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:18.594 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.594 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:18.594 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:18.594 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:18.594 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 1 Processor -- start 2024-09-16T14:48:18.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 1 -- start start 2024-09-16T14:48:18.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 10 monclient: init 2024-09-16T14:48:18.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:18.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:18.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 10 monclient(hunting): picked mon.noname-c con 0x7fa12009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:18.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 10 monclient(hunting): picked mon.noname-a con 0x7fa120132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:18.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 10 monclient(hunting): picked mon.noname-b con 0x7fa120138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:18.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12009d260 con 0x7fa120132650 2024-09-16T14:48:18.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa120132a80 con 0x7fa12009cb80 2024-09-16T14:48:18.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12009cf60 con 0x7fa120138ff0 2024-09-16T14:48:18.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:18.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 10 monclient(hunting): authenticate will time out at 1322.714129s 2024-09-16T14:48:18.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa120138ff0 0x7fa12013b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53008/0 (socket says 172.21.15.26:53008) 2024-09-16T14:48:18.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11ffff640 1 -- 172.21.15.26:0/2026500942 learned_addr learned my addr 172.21.15.26:0/2026500942 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:18.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 1 -- 172.21.15.26:0/2026500942 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189841580 0 0) 0x7fa12009cf60 con 0x7fa120138ff0 2024-09-16T14:48:18.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient(hunting): my global_id is 114626 2024-09-16T14:48:18.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 1 -- 172.21.15.26:0/2026500942 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa10c003530 con 0x7fa120138ff0 2024-09-16T14:48:18.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 1 -- 172.21.15.26:0/2026500942 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810441259 0 0) 0x7fa12009d260 con 0x7fa120132650 2024-09-16T14:48:18.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient(hunting): my global_id is 124320 2024-09-16T14:48:18.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 1 -- 172.21.15.26:0/2026500942 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa12009cf60 con 0x7fa120132650 2024-09-16T14:48:18.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 1 -- 172.21.15.26:0/2026500942 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2626271252 0 0) 0x7fa10c003530 con 0x7fa120138ff0 2024-09-16T14:48:18.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 1 -- 172.21.15.26:0/2026500942 >> v1:172.21.15.189:6789/0 conn(0x7fa12009cb80 legacy=0x7fa1200717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 1 -- 172.21.15.26:0/2026500942 >> v1:172.21.15.26:6789/0 conn(0x7fa120132650 legacy=0x7fa1201378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:18.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 1 monclient: found mon.noname-b 2024-09-16T14:48:18.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:18.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:48:18.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 1 -- 172.21.15.26:0/2026500942 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12009f2d0 con 0x7fa120138ff0 2024-09-16T14:48:18.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient: _finish_auth 0 2024-09-16T14:48:18.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 1 -- 172.21.15.26:0/2026500942 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa1180019e0 con 0x7fa120138ff0 2024-09-16T14:48:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 5 monclient: authenticate success, global_id 114626 2024-09-16T14:48:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient: dump: 2024-09-16T14:48:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.600 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.601 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.601 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.601 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.601 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:18.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 1 -- 172.21.15.26:0/2026500942 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa118001c00 con 0x7fa120138ff0 2024-09-16T14:48:18.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:18.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 1 -- 172.21.15.26:0/2026500942 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa118001f10 con 0x7fa120138ff0 2024-09-16T14:48:18.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:18.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa11effd640 10 monclient: dump: 2024-09-16T14:48:18.601 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.601 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.602 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.602 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.602 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.602 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.602 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.602 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.602 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.602 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:18.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 4 set_mon_vals no callback set 2024-09-16T14:48:18.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 10 monclient: shutdown 2024-09-16T14:48:18.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:18.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 1 -- 172.21.15.26:0/2026500942 >> v1:172.21.15.26:6790/0 conn(0x7fa120138ff0 legacy=0x7fa12013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 1 -- 172.21.15.26:0/2026500942 shutdown_connections 2024-09-16T14:48:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 1 -- 172.21.15.26:0/2026500942 >> 172.21.15.26:0/2026500942 conn(0x7fa12009fb50 msgr2=0x7fa12009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 1 -- 172.21.15.26:0/2026500942 shutdown_connections 2024-09-16T14:48:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.591+0000 7fa125c75640 1 -- 172.21.15.26:0/2026500942 wait complete. 2024-09-16T14:48:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 10 monclient: build_initial_monmap 2024-09-16T14:48:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 10 monclient: monmap: 2024-09-16T14:48:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:18.603 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:18.604 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:18.604 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:18.604 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 1 Processor -- start 2024-09-16T14:48:18.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 1 -- start start 2024-09-16T14:48:18.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 10 monclient: init 2024-09-16T14:48:18.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:18.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:18.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 10 monclient(hunting): picked mon.noname-c con 0x7fa12009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:18.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 10 monclient(hunting): picked mon.noname-b con 0x7fa120132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:18.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 10 monclient(hunting): picked mon.noname-a con 0x7fa120138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:18.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa120071310 con 0x7fa120138ff0 2024-09-16T14:48:18.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12006e4b0 con 0x7fa120132650 2024-09-16T14:48:18.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12006fe60 con 0x7fa12009cb80 2024-09-16T14:48:18.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:18.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 10 monclient(hunting): authenticate will time out at 1322.717154s 2024-09-16T14:48:18.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa120138ff0 0x7fa1201cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50136/0 (socket says 172.21.15.26:50136) 2024-09-16T14:48:18.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11ffff640 1 -- 172.21.15.26:0/4025175734 learned_addr learned my addr 172.21.15.26:0/4025175734 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:18.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808521806 0 0) 0x7fa120071310 con 0x7fa120138ff0 2024-09-16T14:48:18.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): my global_id is 124323 2024-09-16T14:48:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa100003530 con 0x7fa120138ff0 2024-09-16T14:48:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307183480 0 0) 0x7fa12006fe60 con 0x7fa12009cb80 2024-09-16T14:48:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): my global_id is 114629 2024-09-16T14:48:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa120071310 con 0x7fa12009cb80 2024-09-16T14:48:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1697805514 0 0) 0x7fa100003530 con 0x7fa120138ff0 2024-09-16T14:48:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa12006fe60 con 0x7fa120138ff0 2024-09-16T14:48:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa1180018e0 con 0x7fa120138ff0 2024-09-16T14:48:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): dump: 2024-09-16T14:48:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.612 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:18.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2639613998 0 0) 0x7fa120071310 con 0x7fa12009cb80 2024-09-16T14:48:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa100003530 con 0x7fa12009cb80 2024-09-16T14:48:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420524803 0 0) 0x7fa12006e4b0 con 0x7fa120132650 2024-09-16T14:48:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): my global_id is 144109 2024-09-16T14:48:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa120071310 con 0x7fa120132650 2024-09-16T14:48:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa1080017e0 con 0x7fa12009cb80 2024-09-16T14:48:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): dump: 2024-09-16T14:48:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.613 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.614 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 890618893 0 0) 0x7fa12006fe60 con 0x7fa120138ff0 2024-09-16T14:48:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 >> v1:172.21.15.26:6790/0 conn(0x7fa12009cb80 legacy=0x7fa12006d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 >> v1:172.21.15.189:6789/0 conn(0x7fa120132650 legacy=0x7fa12006dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:48:18.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 monclient: found mon.a 2024-09-16T14:48:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12013c400 con 0x7fa120138ff0 2024-09-16T14:48:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient: _finish_auth 0 2024-09-16T14:48:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:48:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 5 monclient: authenticate success, global_id 124323 2024-09-16T14:48:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 10 monclient: _renew_subs 2024-09-16T14:48:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 1 -- 172.21.15.26:0/4025175734 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa120071030 con 0x7fa120138ff0 2024-09-16T14:48:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 10 monclient: _renew_subs 2024-09-16T14:48:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:18.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 1 -- 172.21.15.26:0/4025175734 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa12006f290 con 0x7fa120138ff0 2024-09-16T14:48:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1180015b0 con 0x7fa120138ff0 2024-09-16T14:48:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa118004fc0 con 0x7fa120138ff0 2024-09-16T14:48:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11e7fc640 4 set_mon_vals no callback set 2024-09-16T14:48:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient: dump: 2024-09-16T14:48:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.616 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.617 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa11800fa30 con 0x7fa120138ff0 2024-09-16T14:48:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa125c75640 1 -- 172.21.15.26:0/4025175734 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa104006590 con 0x7fa104002830 2024-09-16T14:48:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(154..154 src has 73..154) v4 ==== 8989+0+0 (unknown 204141892 0 0) 0x7fa118044a40 con 0x7fa120138ff0 2024-09-16T14:48:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910671791 0 0) 0x7fa104006590 con 0x7fa104002830 2024-09-16T14:48:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient(hunting): my global_id is 114632 2024-09-16T14:48:18.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa12006fe60 con 0x7fa104002830 2024-09-16T14:48:18.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa117fff640 10 monclient: get_auth_request con 0x7fa1000363f0 auth_method 0 2024-09-16T14:48:18.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2192066729 0 0) 0x7fa12006fe60 con 0x7fa104002830 2024-09-16T14:48:18.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa104006590 con 0x7fa104002830 2024-09-16T14:48:18.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa108003620 con 0x7fa104002830 2024-09-16T14:48:18.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:18.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 10 monclient: dump: 2024-09-16T14:48:18.618 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.618 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.618 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.618 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.618 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.618 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.619 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.619 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.619 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.619 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 267092404 0 0) 0x7fa104006590 con 0x7fa104002830 2024-09-16T14:48:18.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.595+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa10400a0e0 con 0x7fa104002830 2024-09-16T14:48:18.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.599+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa108004650 con 0x7fa104002830 2024-09-16T14:48:18.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.599+0000 7fa11cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:18.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.599+0000 7fa11cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:18.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.599+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 >> v1:172.21.15.26:6790/0 conn(0x7fa104002830 legacy=0x7fa104004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.599+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 >> v1:172.21.15.26:6790/0 conn(0x7fa104002830 legacy=0x7fa104004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:18.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa125c75640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:48:18.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa125c75640 1 -- 172.21.15.26:0/4025175734 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa104009da0 con 0x7fa1040090c0 2024-09-16T14:48:18.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540715727 0 0) 0x7fa104009da0 con 0x7fa1040090c0 2024-09-16T14:48:18.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 10 monclient(hunting): my global_id is 114635 2024-09-16T14:48:18.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa104006590 con 0x7fa1040090c0 2024-09-16T14:48:18.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 252955867 0 0) 0x7fa104006590 con 0x7fa1040090c0 2024-09-16T14:48:18.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa104009da0 con 0x7fa1040090c0 2024-09-16T14:48:18.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa108003df0 con 0x7fa1040090c0 2024-09-16T14:48:18.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:18.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 10 monclient: dump: 2024-09-16T14:48:18.621 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.621 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.621 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.621 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.621 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.621 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.621 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.621 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.621 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.621 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1083759419 0 0) 0x7fa104009da0 con 0x7fa1040090c0 2024-09-16T14:48:18.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa10400c290 con 0x7fa1040090c0 2024-09-16T14:48:18.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 1452856956) 0x7fa108004190 con 0x7fa1040090c0 2024-09-16T14:48:18.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:48:18.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:18.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 >> v1:172.21.15.26:6790/0 conn(0x7fa1040090c0 legacy=0x7fa104009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11cff9640 1 -- 172.21.15.26:0/4025175734 >> v1:172.21.15.26:6790/0 conn(0x7fa1040090c0 legacy=0x7fa104009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:18.622 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:48:18.622 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:48:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 54, 2024-09-16T14:48:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:48:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:48:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:48:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:48:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:48:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:48:18.623 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:48:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:48:18.624 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:48:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.625 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:48:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:48:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:48:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:48:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:48:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:48:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:48:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:48:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:48:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:48:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:48:18.626 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:48:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:48:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:48:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:18.627 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:48:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:48:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:48:18.628 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:48:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:48:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:18.629 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:48:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:48:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:48:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:18.630 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:48:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.631 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:48:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:48:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:48:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:48:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.632 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:48:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:48:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:48:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.633 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:48:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.634 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:48:18.634 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:48:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa125c75640 1 -- 172.21.15.26:0/4025175734 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa1000363f0 legacy=0x7fa1000388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa125c75640 10 monclient: shutdown 2024-09-16T14:48:18.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa125c75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa125c75640 1 -- 172.21.15.26:0/4025175734 >> v1:172.21.15.26:6789/0 conn(0x7fa120138ff0 legacy=0x7fa1201cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11ffff640 1 -- 172.21.15.26:0/4025175734 reap_dead start 2024-09-16T14:48:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa11ffff640 1 -- 172.21.15.26:0/4025175734 reap_dead start 2024-09-16T14:48:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa125c75640 1 -- 172.21.15.26:0/4025175734 shutdown_connections 2024-09-16T14:48:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa125c75640 1 -- 172.21.15.26:0/4025175734 >> 172.21.15.26:0/4025175734 conn(0x7fa12009fb50 msgr2=0x7fa120131c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.603+0000 7fa125c75640 1 -- 172.21.15.26:0/4025175734 shutdown_connections 2024-09-16T14:48:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.607+0000 7fa125c75640 1 -- 172.21.15.26:0/4025175734 wait complete. 2024-09-16T14:48:18.636 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:48:18.636 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:48:18.637 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:48:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.747+0000 7f025632b640 10 monclient: get_monmap_and_config 2024-09-16T14:48:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.747+0000 7f025632b640 10 monclient: build_initial_monmap 2024-09-16T14:48:18.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.747+0000 7f025632b640 10 monclient: monmap: 2024-09-16T14:48:18.753 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:18.753 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.753 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:18.753122+0000 2024-09-16T14:48:18.753 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:18.753122+0000 2024-09-16T14:48:18.753 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:18.753 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.753 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:18.753 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:18.753 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:18.753 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 1 Processor -- start 2024-09-16T14:48:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 1 -- start start 2024-09-16T14:48:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 10 monclient: init 2024-09-16T14:48:18.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 10 monclient(hunting): picked mon.noname-c con 0x7f025009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 10 monclient(hunting): picked mon.noname-b con 0x7f0250132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 10 monclient(hunting): picked mon.noname-a con 0x7f0250138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025009d260 con 0x7f0250138ff0 2024-09-16T14:48:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0250132a80 con 0x7f025009cb80 2024-09-16T14:48:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025009cf60 con 0x7f0250132650 2024-09-16T14:48:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 10 monclient(hunting): authenticate will time out at 1322.873189s 2024-09-16T14:48:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f02477fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0250132650 0x7f02501378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53052/0 (socket says 172.21.15.26:53052) 2024-09-16T14:48:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f02548c8640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0250138ff0 0x7f025013b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50140/0 (socket says 172.21.15.26:50140) 2024-09-16T14:48:18.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f02477fe640 1 -- 172.21.15.26:0/1171613089 learned_addr learned my addr 172.21.15.26:0/1171613089 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 1 -- 172.21.15.26:0/1171613089 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220545360 0 0) 0x7f025009d260 con 0x7f0250138ff0 2024-09-16T14:48:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient(hunting): my global_id is 124326 2024-09-16T14:48:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 1 -- 172.21.15.26:0/1171613089 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f023c003530 con 0x7f0250138ff0 2024-09-16T14:48:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 1 -- 172.21.15.26:0/1171613089 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201262248 0 0) 0x7f025009cf60 con 0x7f0250132650 2024-09-16T14:48:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient(hunting): my global_id is 114638 2024-09-16T14:48:18.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 1 -- 172.21.15.26:0/1171613089 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f025009d260 con 0x7f0250132650 2024-09-16T14:48:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 1 -- 172.21.15.26:0/1171613089 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031958423 0 0) 0x7f0250132a80 con 0x7f025009cb80 2024-09-16T14:48:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient(hunting): my global_id is 144112 2024-09-16T14:48:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 1 -- 172.21.15.26:0/1171613089 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f025009cf60 con 0x7f025009cb80 2024-09-16T14:48:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 1 -- 172.21.15.26:0/1171613089 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3889925442 0 0) 0x7f023c003530 con 0x7f0250138ff0 2024-09-16T14:48:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 1 -- 172.21.15.26:0/1171613089 >> v1:172.21.15.26:6790/0 conn(0x7f0250132650 legacy=0x7f02501378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 1 -- 172.21.15.26:0/1171613089 >> v1:172.21.15.189:6789/0 conn(0x7f025009cb80 legacy=0x7f02500717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:48:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 1 monclient: found mon.noname-a 2024-09-16T14:48:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:18.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 1 -- 172.21.15.26:0/1171613089 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025009f2d0 con 0x7f0250138ff0 2024-09-16T14:48:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:48:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:48:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 1 -- 172.21.15.26:0/1171613089 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f02480019a0 con 0x7f0250138ff0 2024-09-16T14:48:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient: dump: 2024-09-16T14:48:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.764 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.765 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 5 monclient: authenticate success, global_id 124326 2024-09-16T14:48:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 1 -- 172.21.15.26:0/1171613089 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0248001c10 con 0x7f0250138ff0 2024-09-16T14:48:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 4 set_mon_vals no callback set 2024-09-16T14:48:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 1 -- 172.21.15.26:0/1171613089 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0248001fe0 con 0x7f0250138ff0 2024-09-16T14:48:18.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:18.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f024f7fe640 10 monclient: dump: 2024-09-16T14:48:18.766 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.766 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.766 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.766 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.766 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.766 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.766 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.766 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.766 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.766 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 10 monclient: shutdown 2024-09-16T14:48:18.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:18.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 1 -- 172.21.15.26:0/1171613089 >> v1:172.21.15.26:6789/0 conn(0x7f0250138ff0 legacy=0x7f025013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 1 -- 172.21.15.26:0/1171613089 shutdown_connections 2024-09-16T14:48:18.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 1 -- 172.21.15.26:0/1171613089 >> 172.21.15.26:0/1171613089 conn(0x7f025009fb50 msgr2=0x7f025009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:18.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 1 -- 172.21.15.26:0/1171613089 shutdown_connections 2024-09-16T14:48:18.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 1 -- 172.21.15.26:0/1171613089 wait complete. 2024-09-16T14:48:18.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 10 monclient: build_initial_monmap 2024-09-16T14:48:18.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 10 monclient: monmap: 2024-09-16T14:48:18.767 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:18.767 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:18.767 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:18.768 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:18.768 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:18.768 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:18.768 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:18.768 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:18.768 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:18.768 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 1 Processor -- start 2024-09-16T14:48:18.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 1 -- start start 2024-09-16T14:48:18.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.751+0000 7f025632b640 10 monclient: init 2024-09-16T14:48:18.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:18.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:18.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 10 monclient(hunting): picked mon.noname-c con 0x7f025009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:18.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 10 monclient(hunting): picked mon.noname-b con 0x7f0250132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:18.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 10 monclient(hunting): picked mon.noname-a con 0x7f0250138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:18.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0250133800 con 0x7f0250138ff0 2024-09-16T14:48:18.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025006e510 con 0x7f0250132650 2024-09-16T14:48:18.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025009a240 con 0x7f025009cb80 2024-09-16T14:48:18.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:18.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 10 monclient(hunting): authenticate will time out at 1322.876465s 2024-09-16T14:48:18.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f02548c8640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0250138ff0 0x7f02501cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50156/0 (socket says 172.21.15.26:50156) 2024-09-16T14:48:18.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f025009cb80 0x7f025006d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53060/0 (socket says 172.21.15.26:53060) 2024-09-16T14:48:18.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f02548c8640 1 -- 172.21.15.26:0/2211445041 learned_addr learned my addr 172.21.15.26:0/2211445041 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:18.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213465861 0 0) 0x7f0250133800 con 0x7f0250138ff0 2024-09-16T14:48:18.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): my global_id is 124329 2024-09-16T14:48:18.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0230003530 con 0x7f0250138ff0 2024-09-16T14:48:18.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173978972 0 0) 0x7f025009a240 con 0x7f025009cb80 2024-09-16T14:48:18.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): my global_id is 114641 2024-09-16T14:48:18.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0250133800 con 0x7f025009cb80 2024-09-16T14:48:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3876332251 0 0) 0x7f0230003530 con 0x7f0250138ff0 2024-09-16T14:48:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f025009a240 con 0x7f0250138ff0 2024-09-16T14:48:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0248001960 con 0x7f0250138ff0 2024-09-16T14:48:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): dump: 2024-09-16T14:48:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.772 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.772 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.772 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.772 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:18.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3291110984 0 0) 0x7f0250133800 con 0x7f025009cb80 2024-09-16T14:48:18.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0230003530 con 0x7f025009cb80 2024-09-16T14:48:18.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f02400017c0 con 0x7f025009cb80 2024-09-16T14:48:18.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:18.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): dump: 2024-09-16T14:48:18.772 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.773 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.773 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.773 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.773 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.773 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.773 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.773 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.773 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.773 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4005236605 0 0) 0x7f025009a240 con 0x7f0250138ff0 2024-09-16T14:48:18.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 >> v1:172.21.15.26:6790/0 conn(0x7f025009cb80 legacy=0x7f025006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 >> v1:172.21.15.189:6789/0 conn(0x7f0250132650 legacy=0x7f025006de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:48:18.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 monclient: found mon.a 2024-09-16T14:48:18.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:18.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:18.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025013c600 con 0x7f0250138ff0 2024-09-16T14:48:18.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:48:18.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:48:18.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:18.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 5 monclient: authenticate success, global_id 124329 2024-09-16T14:48:18.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 10 monclient: _renew_subs 2024-09-16T14:48:18.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:18.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 1 -- 172.21.15.26:0/2211445041 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0250071620 con 0x7f0250138ff0 2024-09-16T14:48:18.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 10 monclient: _renew_subs 2024-09-16T14:48:18.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:18.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 1 -- 172.21.15.26:0/2211445041 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f025006f250 con 0x7f0250138ff0 2024-09-16T14:48:18.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0248004e20 con 0x7f0250138ff0 2024-09-16T14:48:18.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:18.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f02480051b0 con 0x7f0250138ff0 2024-09-16T14:48:18.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:18.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient: dump: 2024-09-16T14:48:18.775 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.775 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.775 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.775 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.775 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.775 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.776 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.776 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.776 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.776 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024e7fc640 4 set_mon_vals no callback set 2024-09-16T14:48:18.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:18.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f025632b640 1 -- 172.21.15.26:0/2211445041 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0250138cf0 con 0x7f0250138910 2024-09-16T14:48:18.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f024800fc20 con 0x7f0250138ff0 2024-09-16T14:48:18.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(154..154 src has 73..154) v4 ==== 8989+0+0 (unknown 204141892 0 0) 0x7f0248044aa0 con 0x7f0250138ff0 2024-09-16T14:48:18.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054483390 0 0) 0x7f0250138cf0 con 0x7f0250138910 2024-09-16T14:48:18.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient(hunting): my global_id is 144118 2024-09-16T14:48:18.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f025009a240 con 0x7f0250138910 2024-09-16T14:48:18.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f02477fe640 10 monclient: get_auth_request con 0x7f0230036250 auth_method 0 2024-09-16T14:48:18.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2847527350 0 0) 0x7f025009a240 con 0x7f0250138910 2024-09-16T14:48:18.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0250138cf0 con 0x7f0250138910 2024-09-16T14:48:18.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0240003ab0 con 0x7f0250138910 2024-09-16T14:48:18.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:18.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 10 monclient: dump: 2024-09-16T14:48:18.777 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.777 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.777 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.777 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.777 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.778 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.778 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.778 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.778 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.778 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2787457749 0 0) 0x7f0250138cf0 con 0x7f0250138910 2024-09-16T14:48:18.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.755+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f025009f5b0 con 0x7f0250138910 2024-09-16T14:48:18.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.759+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f02400057d0 con 0x7f0250138910 2024-09-16T14:48:18.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.759+0000 7f024d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:18.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.759+0000 7f024d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:18.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.759+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 >> v1:172.21.15.189:6789/0 conn(0x7f0250138910 legacy=0x7f025012ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.759+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 >> v1:172.21.15.189:6789/0 conn(0x7f0250138910 legacy=0x7f025012ced0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:18.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.763+0000 7f025632b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:48:18.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.763+0000 7f025632b640 1 -- 172.21.15.26:0/2211445041 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f025009c960 con 0x7f025002c280 2024-09-16T14:48:18.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.763+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849255777 0 0) 0x7f025009c960 con 0x7f025002c280 2024-09-16T14:48:18.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.763+0000 7f024d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.763+0000 7f024d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.763+0000 7f024d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.763+0000 7f024d7fa640 10 monclient(hunting): my global_id is 144121 2024-09-16T14:48:18.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.763+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0250138cf0 con 0x7f025002c280 2024-09-16T14:48:18.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 859042407 0 0) 0x7f0250138cf0 con 0x7f025002c280 2024-09-16T14:48:18.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f025009c960 con 0x7f025002c280 2024-09-16T14:48:18.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f02400040f0 con 0x7f025002c280 2024-09-16T14:48:18.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f024d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f024d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:18.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f024d7fa640 10 monclient: dump: 2024-09-16T14:48:18.781 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.781 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.781 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.781 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.781 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.781 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.781 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.781 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.781 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.781 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1966520356 0 0) 0x7f025009c960 con 0x7f025002c280 2024-09-16T14:48:18.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0250071f00 con 0x7f025002c280 2024-09-16T14:48:18.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 1101715007) 0x7f0240004440 con 0x7f025002c280 2024-09-16T14:48:18.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f024d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:48:18.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f024d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:18.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 >> v1:172.21.15.189:6789/0 conn(0x7f025002c280 legacy=0x7f025012d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f024d7fa640 1 -- 172.21.15.26:0/2211445041 >> v1:172.21.15.189:6789/0 conn(0x7f025002c280 legacy=0x7f025012d5e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:18.782 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:48:18.782 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:18.782 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:18.782 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:48:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 54, 2024-09-16T14:48:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:48:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:48:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:48:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:48:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:48:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:48:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:48:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:18.783 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:18.784 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:18.784 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:18.784 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:18.784 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:18.784 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:18.784 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:18.784 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:18.784 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:18.784 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:48:18.784 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:48:18.784 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:18.784 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:18.785 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:18.785 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:18.785 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:18.785 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:18.785 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:18.785 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:18.785 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:18.785 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:18.785 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.785 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:48:18.785 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:48:18.785 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:48:18.786 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:48:18.786 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:48:18.786 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:48:18.786 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:18.786 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:18.786 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:48:18.786 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:48:18.786 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:48:18.786 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:48:18.786 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:48:18.786 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:48:18.786 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:48:18.786 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:18.787 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:48:18.787 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:18.787 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:18.787 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:18.787 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:18.787 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:18.787 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:18.787 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:18.787 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:18.787 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:18.787 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:18.787 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:18.787 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:48:18.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.788 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:48:18.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.788 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:18.788 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:18.788 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:18.788 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:18.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.788 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:18.788 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:48:18.788 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:18.788 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:48:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:48:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:48:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:48:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:48:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.790 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:18.791 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:18.791 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:18.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:18.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.791 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:18.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:48:18.791 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:18.791 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.791 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:48:18.791 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:48:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:48:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:48:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.792 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:18.793 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:48:18.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:18.793 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:18.793 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:18.793 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:18.793 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:18.793 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:18.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:18.793 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:48:18.793 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:48:18.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f025632b640 1 -- 172.21.15.26:0/2211445041 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0230036250 legacy=0x7f0230038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f025632b640 10 monclient: shutdown 2024-09-16T14:48:18.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f025632b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:18.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f025632b640 1 -- 172.21.15.26:0/2211445041 >> v1:172.21.15.26:6789/0 conn(0x7f0250138ff0 legacy=0x7f02501cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f02548c8640 1 -- 172.21.15.26:0/2211445041 reap_dead start 2024-09-16T14:48:18.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f02548c8640 1 -- 172.21.15.26:0/2211445041 reap_dead start 2024-09-16T14:48:18.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f025632b640 1 -- 172.21.15.26:0/2211445041 shutdown_connections 2024-09-16T14:48:18.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f025632b640 1 -- 172.21.15.26:0/2211445041 >> 172.21.15.26:0/2211445041 conn(0x7f025009fb50 msgr2=0x7f02500a13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:18.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f025632b640 1 -- 172.21.15.26:0/2211445041 shutdown_connections 2024-09-16T14:48:18.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.767+0000 7f025632b640 1 -- 172.21.15.26:0/2211445041 wait complete. 2024-09-16T14:48:18.796 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:48:18.796 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:48:18.796 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:48:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea7548640 10 monclient: get_monmap_and_config 2024-09-16T14:48:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea7548640 10 monclient: build_initial_monmap 2024-09-16T14:48:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea7548640 10 monclient: monmap: 2024-09-16T14:48:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:18.907660+0000 2024-09-16T14:48:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:18.907660+0000 2024-09-16T14:48:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.908 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:18.908 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:18.908 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:18.908 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea7548640 1 Processor -- start 2024-09-16T14:48:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea7548640 1 -- start start 2024-09-16T14:48:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea7548640 10 monclient: init 2024-09-16T14:48:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea7548640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea7548640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea7548640 10 monclient(hunting): picked mon.noname-b con 0x7f5ea009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea7548640 10 monclient(hunting): picked mon.noname-a con 0x7f5ea0132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea7548640 10 monclient(hunting): picked mon.noname-c con 0x7f5ea0138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea7548640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea009d260 con 0x7f5ea0132650 2024-09-16T14:48:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea7548640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea0132a80 con 0x7f5ea0138ff0 2024-09-16T14:48:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea7548640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea009cf60 con 0x7f5ea009cb80 2024-09-16T14:48:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea7548640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea7548640 10 monclient(hunting): authenticate will time out at 1323.027594s 2024-09-16T14:48:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea4ae3640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5ea0132650 0x7f5ea01378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50168/0 (socket says 172.21.15.26:50168) 2024-09-16T14:48:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.903+0000 7f5ea4ae3640 1 -- 172.21.15.26:0/101751290 learned_addr learned my addr 172.21.15.26:0/101751290 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 1 -- 172.21.15.26:0/101751290 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321241990 0 0) 0x7f5ea009d260 con 0x7f5ea0132650 2024-09-16T14:48:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient(hunting): my global_id is 124332 2024-09-16T14:48:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 1 -- 172.21.15.26:0/101751290 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e88003530 con 0x7f5ea0132650 2024-09-16T14:48:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 1 -- 172.21.15.26:0/101751290 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933308486 0 0) 0x7f5ea009cf60 con 0x7f5ea009cb80 2024-09-16T14:48:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient(hunting): my global_id is 114644 2024-09-16T14:48:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 1 -- 172.21.15.26:0/101751290 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea009d260 con 0x7f5ea009cb80 2024-09-16T14:48:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 1 -- 172.21.15.26:0/101751290 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3304339562 0 0) 0x7f5ea009d260 con 0x7f5ea009cb80 2024-09-16T14:48:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 1 -- 172.21.15.26:0/101751290 >> v1:172.21.15.189:6789/0 conn(0x7f5ea0138ff0 legacy=0x7f5ea013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 1 -- 172.21.15.26:0/101751290 >> v1:172.21.15.26:6789/0 conn(0x7f5ea0132650 legacy=0x7f5ea01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 1 monclient: found mon.noname-b 2024-09-16T14:48:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:48:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 1 -- 172.21.15.26:0/101751290 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea009f2d0 con 0x7f5ea009cb80 2024-09-16T14:48:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient: _finish_auth 0 2024-09-16T14:48:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 1 -- 172.21.15.26:0/101751290 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5e980019e0 con 0x7f5ea009cb80 2024-09-16T14:48:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 5 monclient: authenticate success, global_id 114644 2024-09-16T14:48:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient: dump: 2024-09-16T14:48:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.915 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.915 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.915 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:18.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 1 -- 172.21.15.26:0/101751290 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e98001c00 con 0x7f5ea009cb80 2024-09-16T14:48:18.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:18.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 1 -- 172.21.15.26:0/101751290 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5e98001f10 con 0x7f5ea009cb80 2024-09-16T14:48:18.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:18.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e97fff640 10 monclient: dump: 2024-09-16T14:48:18.915 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.915 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.915 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.916 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.916 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.916 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.916 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.916 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.916 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.916 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:18.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 4 set_mon_vals no callback set 2024-09-16T14:48:18.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 10 monclient: shutdown 2024-09-16T14:48:18.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:18.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 1 -- 172.21.15.26:0/101751290 >> v1:172.21.15.26:6790/0 conn(0x7f5ea009cb80 legacy=0x7f5ea00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 1 -- 172.21.15.26:0/101751290 shutdown_connections 2024-09-16T14:48:18.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 1 -- 172.21.15.26:0/101751290 >> 172.21.15.26:0/101751290 conn(0x7f5ea009fb50 msgr2=0x7f5ea009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:18.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 1 -- 172.21.15.26:0/101751290 shutdown_connections 2024-09-16T14:48:18.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 1 -- 172.21.15.26:0/101751290 wait complete. 2024-09-16T14:48:18.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 10 monclient: build_initial_monmap 2024-09-16T14:48:18.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 10 monclient: monmap: 2024-09-16T14:48:18.917 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:18.917 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:18.917 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:18.917 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:18.917 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:18.917 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:18.917 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:18.917 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:18.918 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:18.918 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 1 Processor -- start 2024-09-16T14:48:18.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 1 -- start start 2024-09-16T14:48:18.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 10 monclient: init 2024-09-16T14:48:18.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:18.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:18.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 10 monclient(hunting): picked mon.noname-a con 0x7f5ea009cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:18.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 10 monclient(hunting): picked mon.noname-c con 0x7f5ea0132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:18.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 10 monclient(hunting): picked mon.noname-b con 0x7f5ea0138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:18.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea00a2a70 con 0x7f5ea009cb80 2024-09-16T14:48:18.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea013c400 con 0x7f5ea0138ff0 2024-09-16T14:48:18.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea009a240 con 0x7f5ea0132650 2024-09-16T14:48:18.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:18.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 10 monclient(hunting): authenticate will time out at 1323.030736s 2024-09-16T14:48:18.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea52e4640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5ea009cb80 0x7f5ea012ba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50174/0 (socket says 172.21.15.26:50174) 2024-09-16T14:48:18.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea52e4640 1 -- 172.21.15.26:0/3095256393 learned_addr learned my addr 172.21.15.26:0/3095256393 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:18.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799346367 0 0) 0x7f5ea00a2a70 con 0x7f5ea009cb80 2024-09-16T14:48:18.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient(hunting): my global_id is 124335 2024-09-16T14:48:18.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e8c003530 con 0x7f5ea009cb80 2024-09-16T14:48:18.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331296134 0 0) 0x7f5ea009a240 con 0x7f5ea0132650 2024-09-16T14:48:18.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:18.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:18.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:18.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient(hunting): my global_id is 114647 2024-09-16T14:48:18.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea00a2a70 con 0x7f5ea0132650 2024-09-16T14:48:18.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 923186021 0 0) 0x7f5ea00a2a70 con 0x7f5ea0132650 2024-09-16T14:48:18.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ea009a240 con 0x7f5ea0132650 2024-09-16T14:48:18.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5e9c001860 con 0x7f5ea0132650 2024-09-16T14:48:18.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:18.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient(hunting): dump: 2024-09-16T14:48:18.924 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.924 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.924 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.924 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.924 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.924 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.924 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.924 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.924 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.924 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:18.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2981534982 0 0) 0x7f5e8c003530 con 0x7f5ea009cb80 2024-09-16T14:48:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ea00a2a70 con 0x7f5ea009cb80 2024-09-16T14:48:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5e98001f10 con 0x7f5ea009cb80 2024-09-16T14:48:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient(hunting): dump: 2024-09-16T14:48:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.926 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.926 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1489554162 0 0) 0x7f5ea009a240 con 0x7f5ea0132650 2024-09-16T14:48:18.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 >> v1:172.21.15.189:6789/0 conn(0x7f5ea0138ff0 legacy=0x7f5ea01cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 >> v1:172.21.15.26:6789/0 conn(0x7f5ea009cb80 legacy=0x7f5ea012ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:18.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:48:18.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 monclient: found mon.c 2024-09-16T14:48:18.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:18.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:18.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea0133b90 con 0x7f5ea0132650 2024-09-16T14:48:18.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient: _finish_auth 0 2024-09-16T14:48:18.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:18.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:18.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 5 monclient: authenticate success, global_id 114647 2024-09-16T14:48:18.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 10 monclient: _renew_subs 2024-09-16T14:48:18.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:18.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 1 -- 172.21.15.26:0/3095256393 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea006f840 con 0x7f5ea0132650 2024-09-16T14:48:18.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 10 monclient: _renew_subs 2024-09-16T14:48:18.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:18.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5ea7548640 1 -- 172.21.15.26:0/3095256393 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea0070150 con 0x7f5ea0132650 2024-09-16T14:48:18.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e9c0047f0 con 0x7f5ea0132650 2024-09-16T14:48:18.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:18.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5e9c004b00 con 0x7f5ea0132650 2024-09-16T14:48:18.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:18.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:18.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e96ffd640 4 set_mon_vals no callback set 2024-09-16T14:48:18.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.907+0000 7f5e95ffb640 10 monclient: dump: 2024-09-16T14:48:18.928 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:18.928 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:18.928 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.928 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:18.928 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:18.928 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:18.928 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:18.928 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:18.928 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:18.928 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:18.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.911+0000 7f5ea7548640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:18.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.911+0000 7f5ea7548640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:18.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.911+0000 7f5ea7548640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:18.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.911+0000 7f5ea7548640 1 -- 172.21.15.26:0/3095256393 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e80005050 con 0x7f5ea0132650 2024-09-16T14:48:18.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.911+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5e9c00f570 con 0x7f5ea0132650 2024-09-16T14:48:18.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.911+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(154..154 src has 73..154) v4 ==== 8989+0+0 (unknown 204141892 0 0) 0x7f5e9c044680 con 0x7f5ea0132650 2024-09-16T14:48:18.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.911+0000 7f5ea52e4640 10 monclient: get_auth_request con 0x7f5e8c036200 auth_method 0 2024-09-16T14:48:18.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.915+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5e9c0144e0 con 0x7f5ea0132650 2024-09-16T14:48:18.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.915+0000 7f5e95ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:18.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:18.915+0000 7f5e95ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:19.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:19.131+0000 7f5ea7548640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:48:19.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:19.131+0000 7f5ea7548640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:48:19.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:19.131+0000 7f5ea7548640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:19.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:19.131+0000 7f5ea7548640 1 -- 172.21.15.26:0/3095256393 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f5e80005b10 con 0x7f5ea0132650 2024-09-16T14:48:19.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:19.131+0000 7f5e95ffb640 1 -- 172.21.15.26:0/3095256393 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f5e9c012b80 con 0x7f5ea0132650 2024-09-16T14:48:19.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:19.131+0000 7f5e95ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:48:19.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:19.131+0000 7f5e95ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:19.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.131+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.131+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=2,health=11,kv=9,logm=59} crc {auth=218886442,config=692619927,health=3384616317,kv=2522593111,logm=3483441938}) 2024-09-16T14:48:19.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.131+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=11,kv=11,logm=56} crc {auth=3518407331,config=692619927,health=4058954246,kv=1867599010,logm=1812251884}) 2024-09-16T14:48:19.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.135+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.135+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=2,health=11,kv=9,logm=59} crc {auth=218886442,config=692619927,health=3384616317,kv=2522593111,logm=3483441938}) 2024-09-16T14:48:19.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.135+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=18,config=2,health=10,kv=11,logm=59} crc {auth=1819620842,config=692619927,health=705507107,kv=1867599010,logm=612359794}) 2024-09-16T14:48:19.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:19.135+0000 7f5ea7548640 1 -- 172.21.15.26:0/3095256393 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5e8c036200 legacy=0x7f5e8c0386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:19.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:19.135+0000 7f5ea7548640 10 monclient: shutdown 2024-09-16T14:48:19.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:19.135+0000 7f5ea7548640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:19.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:19.135+0000 7f5ea7548640 1 -- 172.21.15.26:0/3095256393 >> v1:172.21.15.26:6790/0 conn(0x7f5ea0132650 legacy=0x7f5ea012c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:19.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:19.135+0000 7f5ea7548640 1 -- 172.21.15.26:0/3095256393 shutdown_connections 2024-09-16T14:48:19.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:19.135+0000 7f5ea7548640 1 -- 172.21.15.26:0/3095256393 >> 172.21.15.26:0/3095256393 conn(0x7f5ea009fb50 msgr2=0x7f5ea0131e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:19.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:19.135+0000 7f5ea7548640 1 -- 172.21.15.26:0/3095256393 shutdown_connections 2024-09-16T14:48:19.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:19.135+0000 7f5ea7548640 1 -- 172.21.15.26:0/3095256393 wait complete. 2024-09-16T14:48:19.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.135+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.135+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3156170114}) 2024-09-16T14:48:19.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.135+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703675240}) 2024-09-16T14:48:19.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.135+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.135+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3156170114}) 2024-09-16T14:48:19.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.135+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=171660732}) 2024-09-16T14:48:19.141 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.139+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.139+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4080627649}) 2024-09-16T14:48:19.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.139+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4193142421}) 2024-09-16T14:48:19.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.139+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.139+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4080627649}) 2024-09-16T14:48:19.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.139+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1563023712}) 2024-09-16T14:48:19.143 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.139+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.143 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.139+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3139972711}) 2024-09-16T14:48:19.143 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.139+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1549059261}) 2024-09-16T14:48:19.143 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.139+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.139+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3139972711}) 2024-09-16T14:48:19.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.139+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1887802368}) 2024-09-16T14:48:19.145 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.143+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.145 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.143+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1519000977}) 2024-09-16T14:48:19.145 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.143+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2644245677}) 2024-09-16T14:48:19.145 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.143+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.146 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.143+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1519000977}) 2024-09-16T14:48:19.146 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.143+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4022320103}) 2024-09-16T14:48:19.147 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.143+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.147 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.143+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3875841544}) 2024-09-16T14:48:19.147 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.143+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2772003786}) 2024-09-16T14:48:19.147 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.143+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.147 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.143+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3875841544}) 2024-09-16T14:48:19.147 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.143+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3089137307}) 2024-09-16T14:48:19.149 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.149 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2830487975}) 2024-09-16T14:48:19.149 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714425639}) 2024-09-16T14:48:19.149 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.149 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2830487975}) 2024-09-16T14:48:19.150 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4272348626}) 2024-09-16T14:48:19.151 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.151 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2889638387}) 2024-09-16T14:48:19.151 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4091512046}) 2024-09-16T14:48:19.151 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.151 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2889638387}) 2024-09-16T14:48:19.151 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3719381514}) 2024-09-16T14:48:19.152 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.152 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1830834550}) 2024-09-16T14:48:19.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002601917}) 2024-09-16T14:48:19.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1830834550}) 2024-09-16T14:48:19.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.147+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3657074935}) 2024-09-16T14:48:19.154 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:48:19.154 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.151+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.151+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=778658271}) 2024-09-16T14:48:19.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.151+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3564765653}) 2024-09-16T14:48:19.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.151+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.151+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=778658271}) 2024-09-16T14:48:19.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.151+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1119885853}) 2024-09-16T14:48:19.156 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.151+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.156 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.151+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4211072712}) 2024-09-16T14:48:19.156 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.151+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=533142330}) 2024-09-16T14:48:19.156 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.151+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.156 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.151+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4211072712}) 2024-09-16T14:48:19.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.151+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2710146860}) 2024-09-16T14:48:19.158 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.155+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.158 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.155+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=398646335}) 2024-09-16T14:48:19.158 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.155+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=625124134}) 2024-09-16T14:48:19.158 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.155+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.158 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.155+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=398646335}) 2024-09-16T14:48:19.158 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.155+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3539255842}) 2024-09-16T14:48:19.160 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.155+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.160 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.155+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=258490163}) 2024-09-16T14:48:19.160 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.155+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2517077225}) 2024-09-16T14:48:19.160 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.155+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.160 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.155+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=258490163}) 2024-09-16T14:48:19.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.155+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=344554479}) 2024-09-16T14:48:19.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.159+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.159+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4205002802}) 2024-09-16T14:48:19.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.159+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3394505876}) 2024-09-16T14:48:19.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.159+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.159+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4205002802}) 2024-09-16T14:48:19.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.159+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686818997}) 2024-09-16T14:48:19.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.159+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.159+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=122738586}) 2024-09-16T14:48:19.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.159+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94115447}) 2024-09-16T14:48:19.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.159+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.159+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=122738586}) 2024-09-16T14:48:19.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.159+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1349510880}) 2024-09-16T14:48:19.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.163+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.163+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1683440472}) 2024-09-16T14:48:19.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.163+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268309963}) 2024-09-16T14:48:19.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.163+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.163+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1683440472}) 2024-09-16T14:48:19.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.163+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=188662578}) 2024-09-16T14:48:19.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.163+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.163+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4214841458}) 2024-09-16T14:48:19.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.163+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4180667172}) 2024-09-16T14:48:19.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.163+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.163+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4214841458}) 2024-09-16T14:48:19.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.163+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1227253872}) 2024-09-16T14:48:19.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3266745878}) 2024-09-16T14:48:19.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449491768}) 2024-09-16T14:48:19.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3266745878}) 2024-09-16T14:48:19.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4244515261}) 2024-09-16T14:48:19.171 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.171 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1549476633}) 2024-09-16T14:48:19.171 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911533520}) 2024-09-16T14:48:19.171 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.171 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1549476633}) 2024-09-16T14:48:19.171 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4251953325}) 2024-09-16T14:48:19.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3035230836}) 2024-09-16T14:48:19.173 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1504462803}) 2024-09-16T14:48:19.173 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.173 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3035230836}) 2024-09-16T14:48:19.173 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.167+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=407222108}) 2024-09-16T14:48:19.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.171+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.171+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3257675748}) 2024-09-16T14:48:19.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.171+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1909476273}) 2024-09-16T14:48:19.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.171+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.175 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.171+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3257675748}) 2024-09-16T14:48:19.175 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.171+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3835247310}) 2024-09-16T14:48:19.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.171+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.171+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2948445180}) 2024-09-16T14:48:19.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.171+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2910712828}) 2024-09-16T14:48:19.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.171+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.171+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2948445180}) 2024-09-16T14:48:19.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.171+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1675864992}) 2024-09-16T14:48:19.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.175+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.175+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=476044172}) 2024-09-16T14:48:19.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.175+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2888279384}) 2024-09-16T14:48:19.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.175+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.175+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=476044172}) 2024-09-16T14:48:19.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.175+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3830590781}) 2024-09-16T14:48:19.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.175+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.175+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3793643879}) 2024-09-16T14:48:19.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.175+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835169056}) 2024-09-16T14:48:19.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.179+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.179+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3793643879}) 2024-09-16T14:48:19.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.179+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2691630145}) 2024-09-16T14:48:19.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.179+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.179+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=77,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=11} crc {logm=672165349,mdsmap=1531151339,mgr=2120425907,mgr_command_descs=794974419,mgrstat=3442501673}) 2024-09-16T14:48:19.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.179+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=79,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=4243806349,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2643666243}) 2024-09-16T14:48:19.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.179+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.179+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=77,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=11} crc {logm=672165349,mdsmap=1531151339,mgr=2120425907,mgr_command_descs=794974419,mgrstat=3442501673}) 2024-09-16T14:48:19.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.179+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=75,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=1} crc {logm=3116581739,mdsmap=3585761405,mgr=1291789651,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=293914478,mon_config_key=2760860530}) 2024-09-16T14:48:19.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.183+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.183+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=1,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=77} crc {mgrstat=1864941183,mon_config_key=1518680965,monmap=970648238,osd_metadata=612206294,osd_pg_creating=2400246914,osd_snap=3794734670,osdmap=3466652017}) 2024-09-16T14:48:19.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.183+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=1,mon_config_key=8,monmap=3,osd_metadata=5,osd_snap=2,osdmap=81} crc {mgrstat=1864941183,mon_config_key=959146769,monmap=970648238,osd_metadata=1524398128,osd_snap=3794734670,osdmap=1153277274}) 2024-09-16T14:48:19.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.183+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.183+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=1,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=77} crc {mgrstat=1864941183,mon_config_key=1518680965,monmap=970648238,osd_metadata=612206294,osd_pg_creating=2400246914,osd_snap=3794734670,osdmap=3466652017}) 2024-09-16T14:48:19.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.183+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=1,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=78} crc {mgrstat=1864941183,mon_config_key=1508412840,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=2400246914,osd_snap=3718248,osdmap=3161238309}) 2024-09-16T14:48:19.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.191+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.191+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=59} crc {osdmap=1528810745}) 2024-09-16T14:48:19.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.191+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=59} crc {osdmap=1340639143}) 2024-09-16T14:48:19.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.191+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:19.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.191+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=59} crc {osdmap=1528810745}) 2024-09-16T14:48:19.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:19.191+0000 7fc29af71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=58} crc {osdmap=1065528337}) 2024-09-16T14:48:19.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.501+0000 7fa92a7fc640 1 -- 172.21.15.189:0/2712373836 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}]=0 osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' v155) v1 ==== 241+0+0 (unknown 4144828391 0 0) 0x7fa930014eb0 con 0x7fa9340a0d10 2024-09-16T14:48:19.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.501+0000 7fa92a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T14:48:19.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.501+0000 7fa92a7fc640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' 2024-09-16T14:48:19.503 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' 2024-09-16T14:48:19.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.505+0000 7fa93c0e5640 1 -- 172.21.15.189:0/2712373836 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa918036250 legacy=0x7fa918038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:19.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.505+0000 7fa93c0e5640 10 monclient: shutdown 2024-09-16T14:48:19.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.505+0000 7fa93c0e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:19.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.505+0000 7fa93c0e5640 1 -- 172.21.15.189:0/2712373836 >> v1:172.21.15.26:6789/0 conn(0x7fa9340a0d10 legacy=0x7fa93409b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:19.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.505+0000 7fa93c0e5640 1 -- 172.21.15.189:0/2712373836 shutdown_connections 2024-09-16T14:48:19.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.505+0000 7fa93c0e5640 1 -- 172.21.15.189:0/2712373836 >> 172.21.15.189:0/2712373836 conn(0x7fa9340a39d0 msgr2=0x7fa93406b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:19.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.505+0000 7fa93c0e5640 1 -- 172.21.15.189:0/2712373836 shutdown_connections 2024-09-16T14:48:19.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.505+0000 7fa93c0e5640 1 -- 172.21.15.189:0/2712373836 wait complete. 2024-09-16T14:48:19.525 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T14:48:19.525 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T14:48:19.525 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep fish 2024-09-16T14:48:19.525 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '10\.' 2024-09-16T14:48:19.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31e21b7640 10 monclient: get_monmap_and_config 2024-09-16T14:48:19.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31e21b7640 10 monclient: build_initial_monmap 2024-09-16T14:48:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31e21b7640 10 monclient: monmap: 2024-09-16T14:48:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:19.642685+0000 2024-09-16T14:48:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:19.642685+0000 2024-09-16T14:48:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:19.644 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:19.644 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31e21b7640 1 Processor -- start 2024-09-16T14:48:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31e21b7640 1 -- start start 2024-09-16T14:48:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31e21b7640 10 monclient: init 2024-09-16T14:48:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31e21b7640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31e21b7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31e21b7640 10 monclient(hunting): picked mon.noname-a con 0x7f31dc0a1a20 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31e21b7640 10 monclient(hunting): picked mon.noname-b con 0x7f31dc09fcd0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31e21b7640 10 monclient(hunting): picked mon.noname-c con 0x7f31dc0a2510 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31e21b7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc09e520 con 0x7f31dc0a1a20 2024-09-16T14:48:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31e21b7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc09ea90 con 0x7f31dc0a2510 2024-09-16T14:48:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31e21b7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc09efc0 con 0x7f31dc09fcd0 2024-09-16T14:48:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31e21b7640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31e21b7640 10 monclient(hunting): authenticate will time out at 1281.759168s 2024-09-16T14:48:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31dbfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f31dc0a2510 0x7f31dc13d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51222/0 (socket says 172.21.15.189:51222) 2024-09-16T14:48:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31dbfff640 1 -- 172.21.15.189:0/3673172639 learned_addr learned my addr 172.21.15.189:0/3673172639 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31da7fc640 1 -- 172.21.15.189:0/3673172639 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037011120 0 0) 0x7f31dc09ea90 con 0x7f31dc0a2510 2024-09-16T14:48:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31da7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31da7fc640 10 monclient(hunting): my global_id is 144130 2024-09-16T14:48:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31da7fc640 1 -- 172.21.15.189:0/3673172639 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31c4003530 con 0x7f31dc0a2510 2024-09-16T14:48:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31da7fc640 1 -- 172.21.15.189:0/3673172639 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2009915203 0 0) 0x7f31c4003530 con 0x7f31dc0a2510 2024-09-16T14:48:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31da7fc640 1 -- 172.21.15.189:0/3673172639 >> v1:172.21.15.26:6790/0 conn(0x7f31dc09fcd0 legacy=0x7f31dc139ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.641+0000 7f31da7fc640 1 -- 172.21.15.189:0/3673172639 >> v1:172.21.15.26:6789/0 conn(0x7f31dc0a1a20 legacy=0x7f31dc0a1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 1 monclient: found mon.noname-c 2024-09-16T14:48:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 1 -- 172.21.15.189:0/3673172639 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc09f2a0 con 0x7f31dc0a2510 2024-09-16T14:48:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:19.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:19.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:19.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 5 monclient: authenticate success, global_id 144130 2024-09-16T14:48:19.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 1 -- 172.21.15.189:0/3673172639 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f31d40019a0 con 0x7f31dc0a2510 2024-09-16T14:48:19.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:19.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:19.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:19.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 10 monclient: dump: 2024-09-16T14:48:19.650 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:19.650 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:19.650 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:19.650 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:19.650 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:19.651 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:19.651 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:19.651 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:19.651 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:19.651 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:19.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:19.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 1 -- 172.21.15.189:0/3673172639 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31d4001cc0 con 0x7f31dc0a2510 2024-09-16T14:48:19.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:19.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 1 -- 172.21.15.189:0/3673172639 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f31d4002050 con 0x7f31dc0a2510 2024-09-16T14:48:19.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:19.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 4 set_mon_vals no callback set 2024-09-16T14:48:19.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:19.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:19.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31da7fc640 10 monclient: dump: 2024-09-16T14:48:19.652 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:19.652 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:19.652 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:19.652 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:19.652 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:19.652 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:19.652 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:19.652 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:19.652 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:19.652 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:19.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 10 monclient: shutdown 2024-09-16T14:48:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 1 -- 172.21.15.189:0/3673172639 >> v1:172.21.15.189:6789/0 conn(0x7f31dc0a2510 legacy=0x7f31dc13d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 1 -- 172.21.15.189:0/3673172639 shutdown_connections 2024-09-16T14:48:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 1 -- 172.21.15.189:0/3673172639 >> 172.21.15.189:0/3673172639 conn(0x7f31dc0a3b20 msgr2=0x7f31dc0a3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 1 -- 172.21.15.189:0/3673172639 shutdown_connections 2024-09-16T14:48:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 1 -- 172.21.15.189:0/3673172639 wait complete. 2024-09-16T14:48:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 10 monclient: build_initial_monmap 2024-09-16T14:48:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 10 monclient: monmap: 2024-09-16T14:48:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:19.654 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 1 Processor -- start 2024-09-16T14:48:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 1 -- start start 2024-09-16T14:48:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 10 monclient: init 2024-09-16T14:48:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 10 monclient(hunting): picked mon.noname-b con 0x7f31dc09fcd0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 10 monclient(hunting): picked mon.noname-a con 0x7f31dc0a1a20 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 10 monclient(hunting): picked mon.noname-c con 0x7f31dc0a2510 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc0a2d40 con 0x7f31dc0a1a20 2024-09-16T14:48:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc09cce0 con 0x7f31dc09fcd0 2024-09-16T14:48:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc122590 con 0x7f31dc0a2510 2024-09-16T14:48:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 10 monclient(hunting): authenticate will time out at 1281.762313s 2024-09-16T14:48:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31db7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f31dc09fcd0 0x7f31dc134bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51236/0 (socket says 172.21.15.189:51236) 2024-09-16T14:48:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31db7fe640 1 -- 172.21.15.189:0/601059892 learned_addr learned my addr 172.21.15.189:0/601059892 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142482281 0 0) 0x7f31dc09e520 con 0x7f31dc09fcd0 2024-09-16T14:48:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient(hunting): my global_id is 144133 2024-09-16T14:48:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31c8003530 con 0x7f31dc09fcd0 2024-09-16T14:48:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2402863375 0 0) 0x7f31c8003530 con 0x7f31dc09fcd0 2024-09-16T14:48:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31dc09e520 con 0x7f31dc09fcd0 2024-09-16T14:48:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f31cc0017e0 con 0x7f31dc09fcd0 2024-09-16T14:48:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient(hunting): dump: 2024-09-16T14:48:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:19.661 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081806949 0 0) 0x7f31dc0a2d40 con 0x7f31dc0a1a20 2024-09-16T14:48:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient(hunting): my global_id is 124341 2024-09-16T14:48:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31c8003530 con 0x7f31dc0a1a20 2024-09-16T14:48:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 177263930 0 0) 0x7f31dc09e520 con 0x7f31dc09fcd0 2024-09-16T14:48:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 >> v1:172.21.15.26:6790/0 conn(0x7f31dc0a2510 legacy=0x7f31dc1d1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 >> v1:172.21.15.26:6789/0 conn(0x7f31dc0a1a20 legacy=0x7f31dc1ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 1 monclient: found mon.b 2024-09-16T14:48:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc0a6710 con 0x7f31dc09fcd0 2024-09-16T14:48:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient: _finish_auth 0 2024-09-16T14:48:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 5 monclient: authenticate success, global_id 144133 2024-09-16T14:48:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 10 monclient: _renew_subs 2024-09-16T14:48:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 1 -- 172.21.15.189:0/601059892 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31dc09e210 con 0x7f31dc09fcd0 2024-09-16T14:48:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 10 monclient: _renew_subs 2024-09-16T14:48:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31e21b7640 1 -- 172.21.15.189:0/601059892 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31dc1222b0 con 0x7f31dc09fcd0 2024-09-16T14:48:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31cc003620 con 0x7f31dc09fcd0 2024-09-16T14:48:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f31cc0039b0 con 0x7f31dc09fcd0 2024-09-16T14:48:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31d9ffb640 4 set_mon_vals no callback set 2024-09-16T14:48:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.645+0000 7f31c3fff640 10 monclient: dump: 2024-09-16T14:48:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:19.665 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.649+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f31cc003b10 con 0x7f31dc09fcd0 2024-09-16T14:48:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.649+0000 7f31e21b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.649+0000 7f31e21b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.649+0000 7f31e21b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.649+0000 7f31e21b7640 1 -- 172.21.15.189:0/601059892 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31bc005050 con 0x7f31dc09fcd0 2024-09-16T14:48:19.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.649+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(155..155 src has 73..155) v4 ==== 8989+0+0 (unknown 1378740788 0 0) 0x7f31cc0446a0 con 0x7f31dc09fcd0 2024-09-16T14:48:19.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.649+0000 7f31daffd640 10 monclient: get_auth_request con 0x7f31c80360c0 auth_method 0 2024-09-16T14:48:19.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.653+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f31cc014730 con 0x7f31dc09fcd0 2024-09-16T14:48:19.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.653+0000 7f31c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:19.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.653+0000 7f31c3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:19.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.817+0000 7f31e21b7640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:19.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.817+0000 7f31e21b7640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:19.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.817+0000 7f31e21b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:19.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.817+0000 7f31e21b7640 1 -- 172.21.15.189:0/601059892 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f31bc005300 con 0x7f31dc09fcd0 2024-09-16T14:48:19.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.817+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v155) v1 ==== 83+0+2175 (unknown 2279067650 0 1621702801) 0x7f31cc012ba0 con 0x7f31dc09fcd0 2024-09-16T14:48:19.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.817+0000 7f31c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:19.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.817+0000 7f31c3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:19.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.921+0000 7f31e21b7640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:19.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.921+0000 7f31e21b7640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:19.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.921+0000 7f31e21b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:19.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.921+0000 7f31e21b7640 1 -- 172.21.15.189:0/601059892 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f31bc0055a0 con 0x7f31dc09fcd0 2024-09-16T14:48:19.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.921+0000 7f31c3fff640 1 -- 172.21.15.189:0/601059892 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v155) v1 ==== 83+0+2175 (unknown 2279067650 0 1621702801) 0x7f31cc012ba0 con 0x7f31dc09fcd0 2024-09-16T14:48:19.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.921+0000 7f31c3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:19.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.921+0000 7f31c3fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:19.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.921+0000 7f31e21b7640 1 -- 172.21.15.189:0/601059892 >> v1:172.21.15.189:6812/3196161582 conn(0x7f31c80360c0 legacy=0x7f31c8038580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:19.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.921+0000 7f31e21b7640 10 monclient: shutdown 2024-09-16T14:48:19.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.921+0000 7f31e21b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:19.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.921+0000 7f31e21b7640 1 -- 172.21.15.189:0/601059892 >> v1:172.21.15.189:6789/0 conn(0x7f31dc09fcd0 legacy=0x7f31dc134bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:19.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.921+0000 7f31e21b7640 1 -- 172.21.15.189:0/601059892 shutdown_connections 2024-09-16T14:48:19.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.921+0000 7f31e21b7640 1 -- 172.21.15.189:0/601059892 >> 172.21.15.189:0/601059892 conn(0x7f31dc0a3b20 msgr2=0x7f31dc1bca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:19.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.921+0000 7f31e21b7640 1 -- 172.21.15.189:0/601059892 shutdown_connections 2024-09-16T14:48:19.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:19.921+0000 7f31e21b7640 1 -- 172.21.15.189:0/601059892 wait complete. 2024-09-16T14:48:19.954 INFO:tasks.workunit.client.0.smithi189.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T14:48:19.954 INFO:tasks.workunit.client.0.smithi189.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T14:48:19.954 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T14:48:19.954 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T14:48:19.954 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep fish 2024-09-16T14:48:19.955 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '11\.' 2024-09-16T14:48:20.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 10 monclient: get_monmap_and_config 2024-09-16T14:48:20.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 10 monclient: build_initial_monmap 2024-09-16T14:48:20.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 10 monclient: monmap: 2024-09-16T14:48:20.058 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:20.059 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.059 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:20.058549+0000 2024-09-16T14:48:20.059 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:20.058549+0000 2024-09-16T14:48:20.059 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:20.059 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:20.059 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:20.059 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:20.059 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:20.059 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 1 Processor -- start 2024-09-16T14:48:20.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 1 -- start start 2024-09-16T14:48:20.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 10 monclient: init 2024-09-16T14:48:20.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:20.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:20.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 10 monclient(hunting): picked mon.noname-c con 0x7f366c1315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:20.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 10 monclient(hunting): picked mon.noname-a con 0x7f366c09b320 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 10 monclient(hunting): picked mon.noname-b con 0x7f366c132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f366c09edd0 con 0x7f366c09b320 2024-09-16T14:48:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f366c09bc00 con 0x7f366c1315a0 2024-09-16T14:48:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f366c09e900 con 0x7f366c132090 2024-09-16T14:48:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 10 monclient(hunting): authenticate will time out at 1282.174747s 2024-09-16T14:48:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366b7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f366c1315a0 0x7f366c131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51248/0 (socket says 172.21.15.189:51248) 2024-09-16T14:48:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366b7fe640 1 -- 172.21.15.189:0/2016386877 learned_addr learned my addr 172.21.15.189:0/2016386877 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 1 -- 172.21.15.189:0/2016386877 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894311393 0 0) 0x7f366c09bc00 con 0x7f366c1315a0 2024-09-16T14:48:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 10 monclient(hunting): my global_id is 144136 2024-09-16T14:48:20.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 1 -- 172.21.15.189:0/2016386877 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3658003530 con 0x7f366c1315a0 2024-09-16T14:48:20.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 1 -- 172.21.15.189:0/2016386877 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3685770126 0 0) 0x7f3658003530 con 0x7f366c1315a0 2024-09-16T14:48:20.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 1 -- 172.21.15.189:0/2016386877 >> v1:172.21.15.26:6790/0 conn(0x7f366c132090 legacy=0x7f366c13a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 1 -- 172.21.15.189:0/2016386877 >> v1:172.21.15.26:6789/0 conn(0x7f366c09b320 legacy=0x7f366c136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:20.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 1 monclient: found mon.noname-c 2024-09-16T14:48:20.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:20.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 1 -- 172.21.15.189:0/2016386877 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c09b920 con 0x7f366c1315a0 2024-09-16T14:48:20.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 10 monclient: _finish_auth 0 2024-09-16T14:48:20.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:20.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:20.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 1 -- 172.21.15.189:0/2016386877 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f365c001620 con 0x7f366c1315a0 2024-09-16T14:48:20.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 5 monclient: authenticate success, global_id 144136 2024-09-16T14:48:20.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:20.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:20.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 10 monclient: dump: 2024-09-16T14:48:20.065 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:20.065 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.065 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.065 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.065 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.065 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:20.065 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.065 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.066 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.066 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:20.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 1 -- 172.21.15.189:0/2016386877 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f365c0018d0 con 0x7f366c1315a0 2024-09-16T14:48:20.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:20.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:20.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f3671a89640 4 set_mon_vals no callback set 2024-09-16T14:48:20.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 1 -- 172.21.15.189:0/2016386877 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f365c001ca0 con 0x7f366c1315a0 2024-09-16T14:48:20.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:20.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.057+0000 7f366affd640 10 monclient: dump: 2024-09-16T14:48:20.066 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:20.066 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.066 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.067 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.067 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.067 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:20.067 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.067 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.067 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.067 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient: shutdown 2024-09-16T14:48:20.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:20.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 1 -- 172.21.15.189:0/2016386877 >> v1:172.21.15.189:6789/0 conn(0x7f366c1315a0 legacy=0x7f366c131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 1 -- 172.21.15.189:0/2016386877 shutdown_connections 2024-09-16T14:48:20.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 1 -- 172.21.15.189:0/2016386877 >> 172.21.15.189:0/2016386877 conn(0x7f366c0a3320 msgr2=0x7f366c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:20.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 1 -- 172.21.15.189:0/2016386877 shutdown_connections 2024-09-16T14:48:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 1 -- 172.21.15.189:0/2016386877 wait complete. 2024-09-16T14:48:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient: build_initial_monmap 2024-09-16T14:48:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient: monmap: 2024-09-16T14:48:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:20.068 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 1 Processor -- start 2024-09-16T14:48:20.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 1 -- start start 2024-09-16T14:48:20.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient: init 2024-09-16T14:48:20.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:20.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:20.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient(hunting): picked mon.noname-a con 0x7f366c1315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:20.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient(hunting): picked mon.noname-c con 0x7f366c132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:20.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient(hunting): picked mon.noname-b con 0x7f366c1c7070 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:20.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f366c09cd20 con 0x7f366c1315a0 2024-09-16T14:48:20.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f366c06b380 con 0x7f366c1c7070 2024-09-16T14:48:20.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f366c0a1000 con 0x7f366c132090 2024-09-16T14:48:20.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:20.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient(hunting): authenticate will time out at 1282.177921s 2024-09-16T14:48:20.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f366bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f366c1c7070 0x7f366c1cfb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51260/0 (socket says 172.21.15.189:51260) 2024-09-16T14:48:20.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f366bfff640 1 -- 172.21.15.189:0/2859997035 learned_addr learned my addr 172.21.15.189:0/2859997035 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:20.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 1 -- 172.21.15.189:0/2859997035 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093401288 0 0) 0x7f366c06b380 con 0x7f366c1c7070 2024-09-16T14:48:20.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 10 monclient(hunting): my global_id is 144139 2024-09-16T14:48:20.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 1 -- 172.21.15.189:0/2859997035 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f364c003530 con 0x7f366c1c7070 2024-09-16T14:48:20.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 1 -- 172.21.15.189:0/2859997035 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1118006380 0 0) 0x7f364c003530 con 0x7f366c1c7070 2024-09-16T14:48:20.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 1 -- 172.21.15.189:0/2859997035 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f366c06b380 con 0x7f366c1c7070 2024-09-16T14:48:20.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 1 -- 172.21.15.189:0/2859997035 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3664001c20 con 0x7f366c1c7070 2024-09-16T14:48:20.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:20.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 10 monclient(hunting): dump: 2024-09-16T14:48:20.074 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:20.074 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.074 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.074 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.074 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.074 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:20.074 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.074 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.074 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.074 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:20.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 1 -- 172.21.15.189:0/2859997035 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3616483386 0 0) 0x7f366c06b380 con 0x7f366c1c7070 2024-09-16T14:48:20.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 1 -- 172.21.15.189:0/2859997035 >> v1:172.21.15.26:6790/0 conn(0x7f366c132090 legacy=0x7f366c1cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 1 -- 172.21.15.189:0/2859997035 >> v1:172.21.15.26:6789/0 conn(0x7f366c1315a0 legacy=0x7f366c130b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:48:20.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 1 monclient: found mon.b 2024-09-16T14:48:20.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:20.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 1 -- 172.21.15.189:0/2859997035 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c11bd00 con 0x7f366c1c7070 2024-09-16T14:48:20.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 10 monclient: _finish_auth 0 2024-09-16T14:48:20.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:48:20.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:20.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 5 monclient: authenticate success, global_id 144139 2024-09-16T14:48:20.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient: _renew_subs 2024-09-16T14:48:20.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 1 -- 172.21.15.189:0/2859997035 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366c0a0bf0 con 0x7f366c1c7070 2024-09-16T14:48:20.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient: _renew_subs 2024-09-16T14:48:20.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 1 -- 172.21.15.189:0/2859997035 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f366c11e540 con 0x7f366c1c7070 2024-09-16T14:48:20.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 1 -- 172.21.15.189:0/2859997035 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3664001ff0 con 0x7f366c1c7070 2024-09-16T14:48:20.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:20.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 1 -- 172.21.15.189:0/2859997035 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3664003df0 con 0x7f366c1c7070 2024-09-16T14:48:20.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:20.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f366a7fc640 4 set_mon_vals no callback set 2024-09-16T14:48:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 10 monclient: dump: 2024-09-16T14:48:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.077 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 1 -- 172.21.15.189:0/2859997035 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3664003fd0 con 0x7f366c1c7070 2024-09-16T14:48:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:20.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3671a89640 1 -- 172.21.15.189:0/2859997035 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3650005050 con 0x7f366c1c7070 2024-09-16T14:48:20.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f3668ff9640 1 -- 172.21.15.189:0/2859997035 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(155..155 src has 73..155) v4 ==== 8989+0+0 (unknown 1378740788 0 0) 0x7f3664044a80 con 0x7f366c1c7070 2024-09-16T14:48:20.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.061+0000 7f366b7fe640 10 monclient: get_auth_request con 0x7f364c036060 auth_method 0 2024-09-16T14:48:20.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.069+0000 7f3668ff9640 1 -- 172.21.15.189:0/2859997035 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3664014b10 con 0x7f366c1c7070 2024-09-16T14:48:20.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.069+0000 7f3668ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:20.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.069+0000 7f3668ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:20.155 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:48:20.155 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:48:20.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.233+0000 7f3671a89640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:20.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.233+0000 7f3671a89640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:20.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.233+0000 7f3671a89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.233+0000 7f3671a89640 1 -- 172.21.15.189:0/2859997035 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f3650005300 con 0x7f366c1c7070 2024-09-16T14:48:20.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.233+0000 7f3668ff9640 1 -- 172.21.15.189:0/2859997035 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v155) v1 ==== 83+0+2175 (unknown 2279067650 0 1621702801) 0x7f3664012f80 con 0x7f366c1c7070 2024-09-16T14:48:20.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.233+0000 7f3668ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:20.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.233+0000 7f3668ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:20.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb33f08640 10 monclient: get_monmap_and_config 2024-09-16T14:48:20.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb33f08640 10 monclient: build_initial_monmap 2024-09-16T14:48:20.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb33f08640 10 monclient: monmap: 2024-09-16T14:48:20.287 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:20.287 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.287 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:20.287203+0000 2024-09-16T14:48:20.287 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:20.287203+0000 2024-09-16T14:48:20.287 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:20.287 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.287 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:20.287 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:20.287 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:20.287 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb33f08640 1 Processor -- start 2024-09-16T14:48:20.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb33f08640 1 -- start start 2024-09-16T14:48:20.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb33f08640 10 monclient: init 2024-09-16T14:48:20.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb33f08640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:20.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb33f08640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:20.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb33f08640 10 monclient(hunting): picked mon.noname-b con 0x7fbb2c09cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:20.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb33f08640 10 monclient(hunting): picked mon.noname-a con 0x7fbb2c132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:20.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb33f08640 10 monclient(hunting): picked mon.noname-c con 0x7fbb2c138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:20.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb33f08640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2c09d260 con 0x7fbb2c132650 2024-09-16T14:48:20.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb33f08640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2c132a80 con 0x7fbb2c138ff0 2024-09-16T14:48:20.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb33f08640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2c09cf60 con 0x7fbb2c09cb80 2024-09-16T14:48:20.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb33f08640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:20.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb33f08640 10 monclient(hunting): authenticate will time out at 1324.407231s 2024-09-16T14:48:20.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb314a3640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbb2c132650 0x7fbb2c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50184/0 (socket says 172.21.15.26:50184) 2024-09-16T14:48:20.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb314a3640 1 -- 172.21.15.26:0/72062037 learned_addr learned my addr 172.21.15.26:0/72062037 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:20.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb30ca2640 1 -- 172.21.15.26:0/72062037 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140322189 0 0) 0x7fbb2c09d260 con 0x7fbb2c132650 2024-09-16T14:48:20.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb30ca2640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb30ca2640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb30ca2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb30ca2640 10 monclient(hunting): my global_id is 124350 2024-09-16T14:48:20.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb30ca2640 1 -- 172.21.15.26:0/72062037 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb18003530 con 0x7fbb2c132650 2024-09-16T14:48:20.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb30ca2640 1 -- 172.21.15.26:0/72062037 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623765279 0 0) 0x7fbb2c09cf60 con 0x7fbb2c09cb80 2024-09-16T14:48:20.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb30ca2640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb30ca2640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb30ca2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb30ca2640 10 monclient(hunting): my global_id is 114662 2024-09-16T14:48:20.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb30ca2640 1 -- 172.21.15.26:0/72062037 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb2c09d260 con 0x7fbb2c09cb80 2024-09-16T14:48:20.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.283+0000 7fbb30ca2640 1 -- 172.21.15.26:0/72062037 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014648196 0 0) 0x7fbb2c132a80 con 0x7fbb2c138ff0 2024-09-16T14:48:20.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 10 monclient(hunting): my global_id is 144142 2024-09-16T14:48:20.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 1 -- 172.21.15.26:0/72062037 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb2c09cf60 con 0x7fbb2c138ff0 2024-09-16T14:48:20.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 1 -- 172.21.15.26:0/72062037 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3284057814 0 0) 0x7fbb18003530 con 0x7fbb2c132650 2024-09-16T14:48:20.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 1 -- 172.21.15.26:0/72062037 >> v1:172.21.15.26:6790/0 conn(0x7fbb2c09cb80 legacy=0x7fbb2c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 1 -- 172.21.15.26:0/72062037 >> v1:172.21.15.189:6789/0 conn(0x7fbb2c138ff0 legacy=0x7fbb2c13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 10 monclient: _finish_hunting 0 2024-09-16T14:48:20.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 1 monclient: found mon.noname-a 2024-09-16T14:48:20.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:20.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:20.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 1 -- 172.21.15.26:0/72062037 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2c09f2d0 con 0x7fbb2c132650 2024-09-16T14:48:20.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 10 monclient: _finish_auth 0 2024-09-16T14:48:20.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 10 monclient: _check_auth_tickets 2024-09-16T14:48:20.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:20.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 1 -- 172.21.15.26:0/72062037 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbb28001620 con 0x7fbb2c132650 2024-09-16T14:48:20.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:20.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 10 monclient: dump: 2024-09-16T14:48:20.298 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.298 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.298 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.298 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.298 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.298 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.298 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.298 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.298 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.299 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 5 monclient: authenticate success, global_id 124350 2024-09-16T14:48:20.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:20.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 1 -- 172.21.15.26:0/72062037 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb28001840 con 0x7fbb2c132650 2024-09-16T14:48:20.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:20.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:20.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 4 set_mon_vals no callback set 2024-09-16T14:48:20.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 1 -- 172.21.15.26:0/72062037 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbb28001b50 con 0x7fbb2c132650 2024-09-16T14:48:20.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:20.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb30ca2640 10 monclient: dump: 2024-09-16T14:48:20.300 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.300 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.300 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.300 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.300 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.300 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.300 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.300 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.300 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.300 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 10 monclient: shutdown 2024-09-16T14:48:20.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:20.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 1 -- 172.21.15.26:0/72062037 >> v1:172.21.15.26:6789/0 conn(0x7fbb2c132650 legacy=0x7fbb2c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 1 -- 172.21.15.26:0/72062037 shutdown_connections 2024-09-16T14:48:20.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 1 -- 172.21.15.26:0/72062037 >> 172.21.15.26:0/72062037 conn(0x7fbb2c09fb50 msgr2=0x7fbb2c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:20.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 1 -- 172.21.15.26:0/72062037 shutdown_connections 2024-09-16T14:48:20.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 1 -- 172.21.15.26:0/72062037 wait complete. 2024-09-16T14:48:20.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 10 monclient: build_initial_monmap 2024-09-16T14:48:20.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 10 monclient: monmap: 2024-09-16T14:48:20.301 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:20.301 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:20.301 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:20.301 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:20.301 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:20.302 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:20.302 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:20.302 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:20.302 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:20.302 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 1 Processor -- start 2024-09-16T14:48:20.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 1 -- start start 2024-09-16T14:48:20.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 10 monclient: init 2024-09-16T14:48:20.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:20.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:48:20.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 10 monclient(hunting): picked mon.noname-a con 0x7fbb2c09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:20.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 10 monclient(hunting): picked mon.noname-b con 0x7fbb2c132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:20.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 10 monclient(hunting): picked mon.noname-c con 0x7fbb2c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:20.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2c13c400 con 0x7fbb2c09cb80 2024-09-16T14:48:20.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2c0a2190 con 0x7fbb2c132650 2024-09-16T14:48:20.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2c09a240 con 0x7fbb2c138ff0 2024-09-16T14:48:20.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:20.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 10 monclient(hunting): authenticate will time out at 1324.410611s 2024-09-16T14:48:20.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb31ca4640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbb2c09cb80 0x7fbb2c09a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50200/0 (socket says 172.21.15.26:50200) 2024-09-16T14:48:20.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb31ca4640 1 -- 172.21.15.26:0/229653829 learned_addr learned my addr 172.21.15.26:0/229653829 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:20.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949135638 0 0) 0x7fbb2c09a240 con 0x7fbb2c138ff0 2024-09-16T14:48:20.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): my global_id is 114665 2024-09-16T14:48:20.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb10003530 con 0x7fbb2c138ff0 2024-09-16T14:48:20.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446856315 0 0) 0x7fbb2c13c400 con 0x7fbb2c09cb80 2024-09-16T14:48:20.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): my global_id is 124353 2024-09-16T14:48:20.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb2c09a240 con 0x7fbb2c09cb80 2024-09-16T14:48:20.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3717102374 0 0) 0x7fbb2c09a240 con 0x7fbb2c09cb80 2024-09-16T14:48:20.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb2c13c400 con 0x7fbb2c09cb80 2024-09-16T14:48:20.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456989051 0 0) 0x7fbb2c0a2190 con 0x7fbb2c132650 2024-09-16T14:48:20.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): my global_id is 144145 2024-09-16T14:48:20.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb2c09a240 con 0x7fbb2c132650 2024-09-16T14:48:20.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3381081366 0 0) 0x7fbb10003530 con 0x7fbb2c138ff0 2024-09-16T14:48:20.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb2c0a2190 con 0x7fbb2c138ff0 2024-09-16T14:48:20.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbb1c0018c0 con 0x7fbb2c09cb80 2024-09-16T14:48:20.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:20.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): dump: 2024-09-16T14:48:20.305 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.305 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.305 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.305 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.305 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.305 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.305 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.306 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.306 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.306 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:20.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbb24001bc0 con 0x7fbb2c138ff0 2024-09-16T14:48:20.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:20.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient(hunting): dump: 2024-09-16T14:48:20.306 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.306 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.306 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.306 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.306 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.307 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.307 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.307 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.307 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.307 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3673241035 0 0) 0x7fbb2c13c400 con 0x7fbb2c09cb80 2024-09-16T14:48:20.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 >> v1:172.21.15.26:6790/0 conn(0x7fbb2c138ff0 legacy=0x7fbb2c1d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 >> v1:172.21.15.189:6789/0 conn(0x7fbb2c132650 legacy=0x7fbb2c09ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:20.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 monclient: found mon.a 2024-09-16T14:48:20.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:20.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:20.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2c06ded0 con 0x7fbb2c09cb80 2024-09-16T14:48:20.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:20.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:20.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:20.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 5 monclient: authenticate success, global_id 124353 2024-09-16T14:48:20.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 10 monclient: _renew_subs 2024-09-16T14:48:20.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:20.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 1 -- 172.21.15.26:0/229653829 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb2c138690 con 0x7fbb2c09cb80 2024-09-16T14:48:20.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 10 monclient: _renew_subs 2024-09-16T14:48:20.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:20.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb33f08640 1 -- 172.21.15.26:0/229653829 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb2c09b7e0 con 0x7fbb2c09cb80 2024-09-16T14:48:20.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb1c003600 con 0x7fbb2c09cb80 2024-09-16T14:48:20.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:20.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbb1c003910 con 0x7fbb2c09cb80 2024-09-16T14:48:20.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb23fff640 4 set_mon_vals no callback set 2024-09-16T14:48:20.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:20.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.287+0000 7fbb227fc640 10 monclient: dump: 2024-09-16T14:48:20.309 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.309 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.309 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.309 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.309 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.309 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.309 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.309 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.310 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.310 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.291+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbb1c003af0 con 0x7fbb2c09cb80 2024-09-16T14:48:20.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.291+0000 7fbb33f08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:20.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.291+0000 7fbb33f08640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:20.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.291+0000 7fbb33f08640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:20.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.291+0000 7fbb33f08640 1 -- 172.21.15.26:0/229653829 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb14005050 con 0x7fbb2c09cb80 2024-09-16T14:48:20.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.291+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(155..155 src has 73..155) v4 ==== 8989+0+0 (unknown 1378740788 0 0) 0x7fbb1c044630 con 0x7fbb2c09cb80 2024-09-16T14:48:20.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.291+0000 7fbb314a3640 10 monclient: get_auth_request con 0x7fbb100363a0 auth_method 0 2024-09-16T14:48:20.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.295+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbb1c0146c0 con 0x7fbb2c09cb80 2024-09-16T14:48:20.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.295+0000 7fbb227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:20.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.295+0000 7fbb227fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:20.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.333+0000 7f3671a89640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:20.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.333+0000 7f3671a89640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:20.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.333+0000 7f3671a89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.333+0000 7f3671a89640 1 -- 172.21.15.189:0/2859997035 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f36500055a0 con 0x7f366c1c7070 2024-09-16T14:48:20.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.333+0000 7f3668ff9640 1 -- 172.21.15.189:0/2859997035 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v155) v1 ==== 83+0+2175 (unknown 2279067650 0 1621702801) 0x7f3664014700 con 0x7f366c1c7070 2024-09-16T14:48:20.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.333+0000 7f3668ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:20.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.333+0000 7f3668ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.337+0000 7f3671a89640 1 -- 172.21.15.189:0/2859997035 >> v1:172.21.15.189:6812/3196161582 conn(0x7f364c036060 legacy=0x7f364c038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.337+0000 7f3671a89640 10 monclient: shutdown 2024-09-16T14:48:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.337+0000 7f3671a89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.337+0000 7f3671a89640 1 -- 172.21.15.189:0/2859997035 >> v1:172.21.15.189:6789/0 conn(0x7f366c1c7070 legacy=0x7f366c1cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.337+0000 7f3671a89640 1 -- 172.21.15.189:0/2859997035 shutdown_connections 2024-09-16T14:48:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.337+0000 7f3671a89640 1 -- 172.21.15.189:0/2859997035 >> 172.21.15.189:0/2859997035 conn(0x7f366c0a3320 msgr2=0x7f366c09d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:20.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.337+0000 7f3671a89640 1 -- 172.21.15.189:0/2859997035 shutdown_connections 2024-09-16T14:48:20.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.337+0000 7f3671a89640 1 -- 172.21.15.189:0/2859997035 wait complete. 2024-09-16T14:48:20.353 INFO:tasks.workunit.client.0.smithi189.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T14:48:20.354 INFO:tasks.workunit.client.0.smithi189.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T14:48:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T14:48:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep fish 2024-09-16T14:48:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T14:48:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '12\.' 2024-09-16T14:48:20.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.465+0000 7f4af21a4640 10 monclient: get_monmap_and_config 2024-09-16T14:48:20.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.465+0000 7f4af21a4640 10 monclient: build_initial_monmap 2024-09-16T14:48:20.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.465+0000 7f4af21a4640 10 monclient: monmap: 2024-09-16T14:48:20.468 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:20.469 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.469 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:20.468508+0000 2024-09-16T14:48:20.469 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:20.468508+0000 2024-09-16T14:48:20.469 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:20.469 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:20.469 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:20.469 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:20.469 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:20.469 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.465+0000 7f4af21a4640 1 Processor -- start 2024-09-16T14:48:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.465+0000 7f4af21a4640 1 -- start start 2024-09-16T14:48:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.465+0000 7f4af21a4640 10 monclient: init 2024-09-16T14:48:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient(hunting): picked mon.noname-a con 0x7f4aec06c120 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient(hunting): picked mon.noname-b con 0x7f4aec1353d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient(hunting): picked mon.noname-c con 0x7f4aec0a1040 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aec06c500 con 0x7f4aec06c120 2024-09-16T14:48:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aec0a2ef0 con 0x7f4aec0a1040 2024-09-16T14:48:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aec09e990 con 0x7f4aec1353d0 2024-09-16T14:48:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:20.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient(hunting): authenticate will time out at 1282.584291s 2024-09-16T14:48:20.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aebfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4aec0a1040 0x7f4aec13d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51266/0 (socket says 172.21.15.189:51266) 2024-09-16T14:48:20.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aebfff640 1 -- 172.21.15.189:0/2984061635 learned_addr learned my addr 172.21.15.189:0/2984061635 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:20.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aeb7fe640 1 -- 172.21.15.189:0/2984061635 >> v1:172.21.15.26:6789/0 conn(0x7f4aec06c120 legacy=0x7f4aec134cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T14:48:20.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aeb7fe640 1 -- 172.21.15.189:0/2984061635 >> v1:172.21.15.26:6789/0 conn(0x7f4aec06c120 legacy=0x7f4aec134cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T14:48:20.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aeb7fe640 1 --1- 172.21.15.189:0/2984061635 >> v1:172.21.15.26:6789/0 conn(0x7f4aec06c120 0x7f4aec134cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2024-09-16T14:48:20.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 1 -- 172.21.15.189:0/2984061635 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 626054721 0 0) 0x7f4aec0a2ef0 con 0x7f4aec0a1040 2024-09-16T14:48:20.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient(hunting): my global_id is 144148 2024-09-16T14:48:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 1 -- 172.21.15.189:0/2984061635 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad4003530 con 0x7f4aec0a1040 2024-09-16T14:48:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 1 -- 172.21.15.189:0/2984061635 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338907007 0 0) 0x7f4aec09e990 con 0x7f4aec1353d0 2024-09-16T14:48:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient(hunting): my global_id is 114668 2024-09-16T14:48:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 1 -- 172.21.15.189:0/2984061635 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aec0a2ef0 con 0x7f4aec1353d0 2024-09-16T14:48:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 1 -- 172.21.15.189:0/2984061635 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1277906239 0 0) 0x7f4ad4003530 con 0x7f4aec0a1040 2024-09-16T14:48:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 1 -- 172.21.15.189:0/2984061635 >> v1:172.21.15.26:6790/0 conn(0x7f4aec1353d0 legacy=0x7f4aec139b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 1 -- 172.21.15.189:0/2984061635 >> v1:172.21.15.26:6789/0 conn(0x7f4aec06c120 legacy=0x7f4aec134cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 1 monclient: found mon.noname-c 2024-09-16T14:48:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 1 -- 172.21.15.189:0/2984061635 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aec09ec70 con 0x7f4aec0a1040 2024-09-16T14:48:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 1 -- 172.21.15.189:0/2984061635 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4ae4001620 con 0x7f4aec0a1040 2024-09-16T14:48:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 5 monclient: authenticate success, global_id 144148 2024-09-16T14:48:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:20.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient: dump: 2024-09-16T14:48:20.481 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:20.481 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.481 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.481 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.481 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.481 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:20.481 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.481 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.481 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.481 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:20.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 1 -- 172.21.15.189:0/2984061635 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ae4001890 con 0x7f4aec0a1040 2024-09-16T14:48:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 1 -- 172.21.15.189:0/2984061635 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4ae4001c20 con 0x7f4aec0a1040 2024-09-16T14:48:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 4 set_mon_vals no callback set 2024-09-16T14:48:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aea7fc640 10 monclient: dump: 2024-09-16T14:48:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.483 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient: shutdown 2024-09-16T14:48:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 1 -- 172.21.15.189:0/2984061635 >> v1:172.21.15.189:6789/0 conn(0x7f4aec0a1040 legacy=0x7f4aec13d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 1 -- 172.21.15.189:0/2984061635 shutdown_connections 2024-09-16T14:48:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 1 -- 172.21.15.189:0/2984061635 >> 172.21.15.189:0/2984061635 conn(0x7f4aec0a3a40 msgr2=0x7f4aec0a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 1 -- 172.21.15.189:0/2984061635 shutdown_connections 2024-09-16T14:48:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 1 -- 172.21.15.189:0/2984061635 wait complete. 2024-09-16T14:48:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient: build_initial_monmap 2024-09-16T14:48:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient: monmap: 2024-09-16T14:48:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:20.484 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 1 Processor -- start 2024-09-16T14:48:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 1 -- start start 2024-09-16T14:48:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient: init 2024-09-16T14:48:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient(hunting): picked mon.noname-b con 0x7f4aec06c120 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient(hunting): picked mon.noname-c con 0x7f4aec0a1040 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient(hunting): picked mon.noname-a con 0x7f4aec1353d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aec09cc30 con 0x7f4aec1353d0 2024-09-16T14:48:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aec0a5450 con 0x7f4aec06c120 2024-09-16T14:48:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aec1328d0 con 0x7f4aec0a1040 2024-09-16T14:48:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4af21a4640 10 monclient(hunting): authenticate will time out at 1282.587959s 2024-09-16T14:48:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aeb7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4aec06c120 0x7f4aec12ed00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51278/0 (socket says 172.21.15.189:51278) 2024-09-16T14:48:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.469+0000 7f4aeb7fe640 1 -- 172.21.15.189:0/255915763 learned_addr learned my addr 172.21.15.189:0/255915763 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084013451 0 0) 0x7f4aec0a5450 con 0x7f4aec06c120 2024-09-16T14:48:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient(hunting): my global_id is 144151 2024-09-16T14:48:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad8003530 con 0x7f4aec06c120 2024-09-16T14:48:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2494528760 0 0) 0x7f4ad8003530 con 0x7f4aec06c120 2024-09-16T14:48:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4aec0a5450 con 0x7f4aec06c120 2024-09-16T14:48:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4adc001c70 con 0x7f4aec06c120 2024-09-16T14:48:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient(hunting): dump: 2024-09-16T14:48:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.487 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213995699 0 0) 0x7f4aec09cc30 con 0x7f4aec1353d0 2024-09-16T14:48:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient(hunting): my global_id is 124356 2024-09-16T14:48:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad8003530 con 0x7f4aec1353d0 2024-09-16T14:48:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189563603 0 0) 0x7f4aec1328d0 con 0x7f4aec0a1040 2024-09-16T14:48:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient(hunting): my global_id is 114671 2024-09-16T14:48:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aec09cc30 con 0x7f4aec0a1040 2024-09-16T14:48:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1052354616 0 0) 0x7f4aec0a5450 con 0x7f4aec06c120 2024-09-16T14:48:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 >> v1:172.21.15.26:6790/0 conn(0x7f4aec0a1040 legacy=0x7f4aec12f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 >> v1:172.21.15.26:6789/0 conn(0x7f4aec1353d0 legacy=0x7f4aec1d1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 monclient: found mon.b 2024-09-16T14:48:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aec12fb20 con 0x7f4aec06c120 2024-09-16T14:48:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient: _finish_auth 0 2024-09-16T14:48:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4af21a4640 5 monclient: authenticate success, global_id 144151 2024-09-16T14:48:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4af21a4640 10 monclient: _renew_subs 2024-09-16T14:48:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4af21a4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4af21a4640 1 -- 172.21.15.189:0/255915763 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aec09e130 con 0x7f4aec06c120 2024-09-16T14:48:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4af21a4640 10 monclient: _renew_subs 2024-09-16T14:48:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4af21a4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4af21a4640 1 -- 172.21.15.189:0/255915763 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aec132b90 con 0x7f4aec06c120 2024-09-16T14:48:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4adc002060 con 0x7f4aec06c120 2024-09-16T14:48:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4adc004f70 con 0x7f4aec06c120 2024-09-16T14:48:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ae9ffb640 4 set_mon_vals no callback set 2024-09-16T14:48:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 10 monclient: dump: 2024-09-16T14:48:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:20.491 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.491 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.491 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.491 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4adc00f9e0 con 0x7f4aec06c120 2024-09-16T14:48:20.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4af21a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:20.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4af21a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:20.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4af21a4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4af21a4640 1 -- 172.21.15.189:0/255915763 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4acc005050 con 0x7f4aec06c120 2024-09-16T14:48:20.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(155..155 src has 73..155) v4 ==== 8989+0+0 (unknown 1378740788 0 0) 0x7f4adc044a90 con 0x7f4aec06c120 2024-09-16T14:48:20.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.473+0000 7f4aeaffd640 10 monclient: get_auth_request con 0x7f4ad80362f0 auth_method 0 2024-09-16T14:48:20.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.477+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4adc0148d0 con 0x7f4aec06c120 2024-09-16T14:48:20.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.477+0000 7f4ad3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:20.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.477+0000 7f4ad3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:20.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.515+0000 7fbb33f08640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:48:20.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.515+0000 7fbb33f08640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:48:20.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.515+0000 7fbb33f08640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:20.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.515+0000 7fbb33f08640 1 -- 172.21.15.26:0/229653829 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbb14005300 con 0x7fbb2c09cb80 2024-09-16T14:48:20.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.515+0000 7fbb227fc640 1 -- 172.21.15.26:0/229653829 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3690517774) 0x7fbb1c0142b0 con 0x7fbb2c09cb80 2024-09-16T14:48:20.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.515+0000 7fbb227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:48:20.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.515+0000 7fbb227fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:20.520 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:48:20.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.519+0000 7fbb33f08640 1 -- 172.21.15.26:0/229653829 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbb100363a0 legacy=0x7fbb10038860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.519+0000 7fbb33f08640 10 monclient: shutdown 2024-09-16T14:48:20.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.519+0000 7fbb33f08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:20.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.519+0000 7fbb33f08640 1 -- 172.21.15.26:0/229653829 >> v1:172.21.15.26:6789/0 conn(0x7fbb2c09cb80 legacy=0x7fbb2c09a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.519+0000 7fbb33f08640 1 -- 172.21.15.26:0/229653829 shutdown_connections 2024-09-16T14:48:20.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.519+0000 7fbb33f08640 1 -- 172.21.15.26:0/229653829 >> 172.21.15.26:0/229653829 conn(0x7fbb2c09fb50 msgr2=0x7fbb2c130660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:20.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.519+0000 7fbb33f08640 1 -- 172.21.15.26:0/229653829 shutdown_connections 2024-09-16T14:48:20.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.519+0000 7fbb33f08640 1 -- 172.21.15.26:0/229653829 wait complete. 2024-09-16T14:48:20.538 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:48:20.538 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:48:20.538 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:48:20.538 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:48:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.645+0000 7f4af21a4640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.645+0000 7f4af21a4640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.645+0000 7f4af21a4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.645+0000 7f4af21a4640 1 -- 172.21.15.189:0/255915763 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f4acc005300 con 0x7f4aec06c120 2024-09-16T14:48:20.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.645+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v155) v1 ==== 83+0+2175 (unknown 2279067650 0 1621702801) 0x7f4adc012f70 con 0x7f4aec06c120 2024-09-16T14:48:20.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.645+0000 7f4ad3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:20.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.645+0000 7f4ad3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:20.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b30840640 10 monclient: get_monmap_and_config 2024-09-16T14:48:20.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b30840640 10 monclient: build_initial_monmap 2024-09-16T14:48:20.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b30840640 10 monclient: monmap: 2024-09-16T14:48:20.662 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:20.662 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.662 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:20.662360+0000 2024-09-16T14:48:20.662 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:20.662360+0000 2024-09-16T14:48:20.662 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:20.662 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.662 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:20.662 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:20.662 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:20.662 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b30840640 1 Processor -- start 2024-09-16T14:48:20.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b30840640 1 -- start start 2024-09-16T14:48:20.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b30840640 10 monclient: init 2024-09-16T14:48:20.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b30840640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:20.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b30840640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:48:20.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b30840640 10 monclient(hunting): picked mon.noname-a con 0x7f7b2c09a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:20.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b30840640 10 monclient(hunting): picked mon.noname-c con 0x7f7b2c09d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:20.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b30840640 10 monclient(hunting): picked mon.noname-b con 0x7f7b2c09b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:20.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b30840640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c09dc60 con 0x7f7b2c09a550 2024-09-16T14:48:20.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b30840640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c099ae0 con 0x7f7b2c09d1b0 2024-09-16T14:48:20.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b30840640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c09b8f0 con 0x7f7b2c09b040 2024-09-16T14:48:20.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b30840640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:20.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b30840640 10 monclient(hunting): authenticate will time out at 1324.783124s 2024-09-16T14:48:20.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2ad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7b2c09b040 0x7f7b2c13a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53098/0 (socket says 172.21.15.26:53098) 2024-09-16T14:48:20.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2a59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7b2c09a550 0x7f7b2c09a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50202/0 (socket says 172.21.15.26:50202) 2024-09-16T14:48:20.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2ad9d640 1 -- 172.21.15.26:0/989832165 learned_addr learned my addr 172.21.15.26:0/989832165 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:20.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 1 -- 172.21.15.26:0/989832165 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074383080 0 0) 0x7f7b2c09dc60 con 0x7f7b2c09a550 2024-09-16T14:48:20.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 10 monclient(hunting): my global_id is 124359 2024-09-16T14:48:20.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 1 -- 172.21.15.26:0/989832165 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b14003530 con 0x7f7b2c09a550 2024-09-16T14:48:20.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 1 -- 172.21.15.26:0/989832165 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128910894 0 0) 0x7f7b2c09b8f0 con 0x7f7b2c09b040 2024-09-16T14:48:20.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 10 monclient(hunting): my global_id is 114674 2024-09-16T14:48:20.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 1 -- 172.21.15.26:0/989832165 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b2c09dc60 con 0x7f7b2c09b040 2024-09-16T14:48:20.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 1 -- 172.21.15.26:0/989832165 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206170037 0 0) 0x7f7b2c099ae0 con 0x7f7b2c09d1b0 2024-09-16T14:48:20.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 10 monclient(hunting): my global_id is 144154 2024-09-16T14:48:20.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 1 -- 172.21.15.26:0/989832165 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b2c09b8f0 con 0x7f7b2c09d1b0 2024-09-16T14:48:20.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.659+0000 7f7b2959a640 1 -- 172.21.15.26:0/989832165 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1082303522 0 0) 0x7f7b14003530 con 0x7f7b2c09a550 2024-09-16T14:48:20.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 1 -- 172.21.15.26:0/989832165 >> v1:172.21.15.26:6790/0 conn(0x7f7b2c09b040 legacy=0x7f7b2c13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 1 -- 172.21.15.26:0/989832165 >> v1:172.21.15.189:6789/0 conn(0x7f7b2c09d1b0 legacy=0x7f7b2c136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 10 monclient: _finish_hunting 0 2024-09-16T14:48:20.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 1 monclient: found mon.noname-a 2024-09-16T14:48:20.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:20.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:20.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 1 -- 172.21.15.26:0/989832165 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c09bbd0 con 0x7f7b2c09a550 2024-09-16T14:48:20.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 10 monclient: _finish_auth 0 2024-09-16T14:48:20.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 10 monclient: _check_auth_tickets 2024-09-16T14:48:20.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:20.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 5 monclient: authenticate success, global_id 124359 2024-09-16T14:48:20.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 1 -- 172.21.15.26:0/989832165 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7b1c001620 con 0x7f7b2c09a550 2024-09-16T14:48:20.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:20.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:20.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 10 monclient: dump: 2024-09-16T14:48:20.672 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.672 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.672 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.673 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.673 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.673 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.673 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.673 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.673 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.673 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:20.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 1 -- 172.21.15.26:0/989832165 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b1c001900 con 0x7f7b2c09a550 2024-09-16T14:48:20.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:20.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 1 -- 172.21.15.26:0/989832165 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7b1c001cd0 con 0x7f7b2c09a550 2024-09-16T14:48:20.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:20.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b2959a640 10 monclient: dump: 2024-09-16T14:48:20.675 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.675 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.675 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.675 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.675 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.675 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.676 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.676 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.676 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.676 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:20.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 4 set_mon_vals no callback set 2024-09-16T14:48:20.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 10 monclient: shutdown 2024-09-16T14:48:20.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:20.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 1 -- 172.21.15.26:0/989832165 >> v1:172.21.15.26:6789/0 conn(0x7f7b2c09a550 legacy=0x7f7b2c09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 1 -- 172.21.15.26:0/989832165 shutdown_connections 2024-09-16T14:48:20.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 1 -- 172.21.15.26:0/989832165 >> 172.21.15.26:0/989832165 conn(0x7f7b2c09fb50 msgr2=0x7f7b2c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:20.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 1 -- 172.21.15.26:0/989832165 shutdown_connections 2024-09-16T14:48:20.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 1 -- 172.21.15.26:0/989832165 wait complete. 2024-09-16T14:48:20.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 10 monclient: build_initial_monmap 2024-09-16T14:48:20.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 10 monclient: monmap: 2024-09-16T14:48:20.678 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:20.678 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:20.678 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:20.678 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:20.679 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:20.679 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:20.679 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:20.679 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:20.679 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:20.679 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 1 Processor -- start 2024-09-16T14:48:20.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 1 -- start start 2024-09-16T14:48:20.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 10 monclient: init 2024-09-16T14:48:20.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:20.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:20.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 10 monclient(hunting): picked mon.noname-b con 0x7f7b2c09a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:20.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 10 monclient(hunting): picked mon.noname-a con 0x7f7b2c09b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:20.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 10 monclient(hunting): picked mon.noname-c con 0x7f7b2c09d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:20.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c1a3440 con 0x7f7b2c09b040 2024-09-16T14:48:20.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c0a2430 con 0x7f7b2c09a550 2024-09-16T14:48:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c1a31d0 con 0x7f7b2c09d1b0 2024-09-16T14:48:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 10 monclient(hunting): authenticate will time out at 1324.786516s 2024-09-16T14:48:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b29d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7b2c09b040 0x7f7b2c19ff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50204/0 (socket says 172.21.15.26:50204) 2024-09-16T14:48:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b29d9b640 1 -- 172.21.15.26:0/3666569889 learned_addr learned my addr 172.21.15.26:0/3666569889 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088849286 0 0) 0x7f7b2c1a3440 con 0x7f7b2c09b040 2024-09-16T14:48:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): my global_id is 124362 2024-09-16T14:48:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b0c003530 con 0x7f7b2c09b040 2024-09-16T14:48:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463110480 0 0) 0x7f7b240014c0 con 0x7f7b2c09d1b0 2024-09-16T14:48:20.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): my global_id is 114677 2024-09-16T14:48:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b2c1a3440 con 0x7f7b2c09d1b0 2024-09-16T14:48:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3706409518 0 0) 0x7f7b0c003530 con 0x7f7b2c09b040 2024-09-16T14:48:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b240014c0 con 0x7f7b2c09b040 2024-09-16T14:48:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47163153 0 0) 0x7f7b2c0a2430 con 0x7f7b2c09a550 2024-09-16T14:48:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): my global_id is 144157 2024-09-16T14:48:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b0c003530 con 0x7f7b2c09a550 2024-09-16T14:48:20.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7b20001940 con 0x7f7b2c09b040 2024-09-16T14:48:20.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:20.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): dump: 2024-09-16T14:48:20.690 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.690 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.690 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.690 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.690 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.690 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.690 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.691 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.691 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.691 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:20.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1007709404 0 0) 0x7f7b2c1a3440 con 0x7f7b2c09d1b0 2024-09-16T14:48:20.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b2c0a2430 con 0x7f7b2c09d1b0 2024-09-16T14:48:20.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7b24001bc0 con 0x7f7b2c09d1b0 2024-09-16T14:48:20.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:20.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient(hunting): dump: 2024-09-16T14:48:20.692 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.692 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.692 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.692 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.692 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.692 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.692 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.692 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.692 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.692 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1556144655 0 0) 0x7f7b240014c0 con 0x7f7b2c09b040 2024-09-16T14:48:20.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 >> v1:172.21.15.26:6790/0 conn(0x7f7b2c09d1b0 legacy=0x7f7b2c1cd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 >> v1:172.21.15.189:6789/0 conn(0x7f7b2c09a550 legacy=0x7f7b2c19f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:20.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 monclient: found mon.a 2024-09-16T14:48:20.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:20.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:20.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c1a0660 con 0x7f7b2c09b040 2024-09-16T14:48:20.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient: _finish_auth 0 2024-09-16T14:48:20.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:20.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:20.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 5 monclient: authenticate success, global_id 124362 2024-09-16T14:48:20.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 10 monclient: _renew_subs 2024-09-16T14:48:20.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:20.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b2c1a1f90 con 0x7f7b2c09b040 2024-09-16T14:48:20.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 10 monclient: _renew_subs 2024-09-16T14:48:20.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:20.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b30840640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2c1a3720 con 0x7f7b2c09b040 2024-09-16T14:48:20.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b20001b30 con 0x7f7b2c09b040 2024-09-16T14:48:20.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:20.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7b200038d0 con 0x7f7b2c09b040 2024-09-16T14:48:20.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:20.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b28d99640 4 set_mon_vals no callback set 2024-09-16T14:48:20.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.663+0000 7f7b1affd640 10 monclient: dump: 2024-09-16T14:48:20.695 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.695 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.695 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.695 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.695 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.695 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.695 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.695 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.695 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.695 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b30840640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:20.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b30840640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7b10006590 con 0x7f7b10002830 2024-09-16T14:48:20.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7b20003ab0 con 0x7f7b2c09b040 2024-09-16T14:48:20.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b2a59c640 1 -- 172.21.15.26:0/3666569889 >> v1:172.21.15.26:6789/0 conn(0x7f7b10002830 legacy=0x7f7b10004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T14:48:20.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b2a59c640 1 -- 172.21.15.26:0/3666569889 >> v1:172.21.15.26:6789/0 conn(0x7f7b10002830 legacy=0x7f7b10004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:48:20.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b2a59c640 1 --1- 172.21.15.26:0/3666569889 >> v1:172.21.15.26:6789/0 conn(0x7f7b10002830 0x7f7b10004cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify read banner and identify addresses failed 2024-09-16T14:48:20.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b2a59c640 1 --1- 172.21.15.26:0/3666569889 >> v1:172.21.15.26:6789/0 conn(0x7f7b10002830 0x7f7b10004cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-16T14:48:20.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 >> v1:172.21.15.26:6789/0 conn(0x7f7b10002830 legacy=0x7f7b10004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:20.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 >> v1:172.21.15.26:6789/0 conn(0x7f7b10002830 legacy=0x7f7b10004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:20.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7b240014c0 con 0x7f7b0c03a0a0 2024-09-16T14:48:20.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(155..155 src has 73..155) v4 ==== 8989+0+0 (unknown 1378740788 0 0) 0x7f7b20044760 con 0x7f7b2c09b040 2024-09-16T14:48:20.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b2a59c640 10 monclient: get_auth_request con 0x7f7b0c0363d0 auth_method 0 2024-09-16T14:48:20.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947170812 0 0) 0x7f7b240014c0 con 0x7f7b0c03a0a0 2024-09-16T14:48:20.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 10 monclient(hunting): my global_id is 124365 2024-09-16T14:48:20.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b2c1a3440 con 0x7f7b0c03a0a0 2024-09-16T14:48:20.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2587733768 0 0) 0x7f7b2c1a3440 con 0x7f7b0c03a0a0 2024-09-16T14:48:20.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b240014c0 con 0x7f7b0c03a0a0 2024-09-16T14:48:20.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7b24004d10 con 0x7f7b0c03a0a0 2024-09-16T14:48:20.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:20.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 10 monclient: dump: 2024-09-16T14:48:20.697 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.697 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.698 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.698 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.698 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.698 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.698 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.698 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.699 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.699 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2638406125 0 0) 0x7f7b240014c0 con 0x7f7b0c03a0a0 2024-09-16T14:48:20.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7b0c03ec80 con 0x7f7b0c03a0a0 2024-09-16T14:48:20.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7b24004fb0 con 0x7f7b0c03a0a0 2024-09-16T14:48:20.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:20.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:20.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 >> v1:172.21.15.26:6789/0 conn(0x7f7b0c03a0a0 legacy=0x7f7b0c03c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.667+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 >> v1:172.21.15.26:6789/0 conn(0x7f7b0c03a0a0 legacy=0x7f7b0c03c490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:20.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.671+0000 7f7b30840640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:48:20.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.671+0000 7f7b30840640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7b1000c3a0 con 0x7f7b100090c0 2024-09-16T14:48:20.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310933104 0 0) 0x7f7b1000c3a0 con 0x7f7b100090c0 2024-09-16T14:48:20.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 10 monclient(hunting): my global_id is 124368 2024-09-16T14:48:20.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b240014c0 con 0x7f7b100090c0 2024-09-16T14:48:20.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3327694730 0 0) 0x7f7b240014c0 con 0x7f7b100090c0 2024-09-16T14:48:20.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b1000c3a0 con 0x7f7b100090c0 2024-09-16T14:48:20.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7b24005500 con 0x7f7b100090c0 2024-09-16T14:48:20.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:20.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 10 monclient: dump: 2024-09-16T14:48:20.703 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.703 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.703 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.703 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.703 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.703 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.703 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.704 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.704 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.704 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 783249259 0 0) 0x7f7b1000c3a0 con 0x7f7b100090c0 2024-09-16T14:48:20.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7b10009c60 con 0x7f7b100090c0 2024-09-16T14:48:20.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 930182563) 0x7f7b24007780 con 0x7f7b100090c0 2024-09-16T14:48:20.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:48:20.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:20.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 >> v1:172.21.15.26:6789/0 conn(0x7f7b100090c0 legacy=0x7f7b10009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b1affd640 1 -- 172.21.15.26:0/3666569889 >> v1:172.21.15.26:6789/0 conn(0x7f7b100090c0 legacy=0x7f7b10009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:20.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b2ad9d640 1 -- 172.21.15.26:0/3666569889 reap_dead start 2024-09-16T14:48:20.706 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:48:20.706 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:20.706 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:20.707 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:48:20.707 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 54, 2024-09-16T14:48:20.707 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:48:20.707 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:48:20.707 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:48:20.708 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:48:20.708 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:20.708 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:48:20.708 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:20.708 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:48:20.709 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:48:20.709 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:20.709 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:20.709 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:20.709 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:20.709 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:20.710 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:20.710 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:20.710 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:20.710 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:20.710 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:20.710 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:20.711 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:48:20.711 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:48:20.711 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:20.711 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:20.711 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:20.711 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:20.712 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:20.712 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:20.712 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:20.712 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:20.712 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:20.712 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:20.713 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:20.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.713 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:48:20.713 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:48:20.713 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:48:20.713 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:48:20.714 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:48:20.714 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:48:20.714 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:20.714 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:20.714 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:48:20.714 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:48:20.715 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:48:20.715 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:48:20.715 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:48:20.715 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:48:20.715 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:48:20.716 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:20.716 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:48:20.716 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:20.716 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:20.716 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:20.716 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:20.717 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:20.717 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:20.717 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:20.717 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:20.717 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:20.718 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:20.718 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:20.718 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:48:20.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.718 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:48:20.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.719 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:20.719 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:20.719 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:20.719 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:20.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.720 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:20.720 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:48:20.720 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:20.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:20.720 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:20.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.721 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:48:20.721 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:48:20.721 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:20.721 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:20.721 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:20.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.722 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:20.722 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:20.722 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:20.722 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:20.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.723 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:20.723 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:48:20.723 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:20.723 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:20.724 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:20.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:48:20.724 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:48:20.724 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:20.724 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:20.725 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:20.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.725 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:20.725 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:20.726 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:20.726 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:20.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.726 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:20.726 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:48:20.726 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:20.727 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:20.727 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:20.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.727 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:48:20.727 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:48:20.727 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:20.728 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:20.728 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:20.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:20.728 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:20.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.729 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:48:20.729 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:48:20.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.729 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:20.729 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:20.729 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:20.729 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:20.730 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:20.730 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:48:20.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.730 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:20.730 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:20.730 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:20.731 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:20.731 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:20.731 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:48:20.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.731 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:20.732 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:20.732 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:48:20.732 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:20.732 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:20.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.732 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:48:20.733 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:48:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b30840640 1 -- 172.21.15.26:0/3666569889 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7b0c0363d0 legacy=0x7f7b0c038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b30840640 10 monclient: shutdown 2024-09-16T14:48:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b30840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b30840640 1 -- 172.21.15.26:0/3666569889 >> v1:172.21.15.26:6789/0 conn(0x7f7b2c09b040 legacy=0x7f7b2c19ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b30840640 1 -- 172.21.15.26:0/3666569889 shutdown_connections 2024-09-16T14:48:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b30840640 1 -- 172.21.15.26:0/3666569889 >> 172.21.15.26:0/3666569889 conn(0x7f7b2c09fb50 msgr2=0x7f7b2c0a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b30840640 1 -- 172.21.15.26:0/3666569889 shutdown_connections 2024-09-16T14:48:20.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.675+0000 7f7b30840640 1 -- 172.21.15.26:0/3666569889 wait complete. 2024-09-16T14:48:20.737 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:48:20.738 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:48:20.738 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:48:20.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.741+0000 7f4af21a4640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:20.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.741+0000 7f4af21a4640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:20.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.741+0000 7f4af21a4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.741+0000 7f4af21a4640 1 -- 172.21.15.189:0/255915763 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f4acc0055a0 con 0x7f4aec06c120 2024-09-16T14:48:20.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.745+0000 7f4ad3fff640 1 -- 172.21.15.189:0/255915763 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v155) v1 ==== 83+0+2175 (unknown 2279067650 0 1621702801) 0x7f4adc0146f0 con 0x7f4aec06c120 2024-09-16T14:48:20.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.745+0000 7f4ad3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:20.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.745+0000 7f4ad3fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:20.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.745+0000 7f4af21a4640 1 -- 172.21.15.189:0/255915763 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4ad80362f0 legacy=0x7f4ad80387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.745+0000 7f4af21a4640 10 monclient: shutdown 2024-09-16T14:48:20.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.745+0000 7f4af21a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:20.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.745+0000 7f4af21a4640 1 -- 172.21.15.189:0/255915763 >> v1:172.21.15.189:6789/0 conn(0x7f4aec06c120 legacy=0x7f4aec12ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.745+0000 7f4af21a4640 1 -- 172.21.15.189:0/255915763 shutdown_connections 2024-09-16T14:48:20.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.745+0000 7f4af21a4640 1 -- 172.21.15.189:0/255915763 >> 172.21.15.189:0/255915763 conn(0x7f4aec0a3a40 msgr2=0x7f4aec06b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:20.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.745+0000 7f4af21a4640 1 -- 172.21.15.189:0/255915763 shutdown_connections 2024-09-16T14:48:20.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.745+0000 7f4af21a4640 1 -- 172.21.15.189:0/255915763 wait complete. 2024-09-16T14:48:20.762 INFO:tasks.workunit.client.0.smithi189.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T14:48:20.762 INFO:tasks.workunit.client.0.smithi189.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T14:48:20.763 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-16T14:48:20.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 10 monclient: get_monmap_and_config 2024-09-16T14:48:20.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 10 monclient: build_initial_monmap 2024-09-16T14:48:20.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 10 monclient: monmap: 2024-09-16T14:48:20.857 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:20.857 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:20.857768+0000 2024-09-16T14:48:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:20.857768+0000 2024-09-16T14:48:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:20.858 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 1 Processor -- start 2024-09-16T14:48:20.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 1 -- start start 2024-09-16T14:48:20.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 10 monclient: init 2024-09-16T14:48:20.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:20.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:20.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 10 monclient(hunting): picked mon.noname-c con 0x7fac5009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:20.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 10 monclient(hunting): picked mon.noname-a con 0x7fac50132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:20.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 10 monclient(hunting): picked mon.noname-b con 0x7fac50138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:20.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5009d260 con 0x7fac50132650 2024-09-16T14:48:20.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac50132a80 con 0x7fac5009cb80 2024-09-16T14:48:20.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5009cf60 con 0x7fac50138ff0 2024-09-16T14:48:20.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 10 monclient(hunting): authenticate will time out at 1324.977749s 2024-09-16T14:48:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fac50138ff0 0x7fac5013b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53118/0 (socket says 172.21.15.26:53118) 2024-09-16T14:48:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4effd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fac50132650 0x7fac501378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50236/0 (socket says 172.21.15.26:50236) 2024-09-16T14:48:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4ffff640 1 -- 172.21.15.26:0/1700940277 learned_addr learned my addr 172.21.15.26:0/1700940277 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4effd640 1 -- 172.21.15.26:0/1700940277 >> v1:172.21.15.26:6789/0 conn(0x7fac50132650 legacy=0x7fac501378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T14:48:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4effd640 1 -- 172.21.15.26:0/1700940277 >> v1:172.21.15.26:6789/0 conn(0x7fac50132650 legacy=0x7fac501378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:48:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4effd640 1 --1- 172.21.15.26:0/1700940277 >> v1:172.21.15.26:6789/0 conn(0x7fac50132650 0x7fac501378d0 :-1 s=OPENED pgs=391 cs=1 l=1).handle_message read tag failed 2024-09-16T14:48:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4effd640 1 --1- 172.21.15.26:0/1700940277 >> v1:172.21.15.26:6789/0 conn(0x7fac50132650 0x7fac501378d0 :-1 s=OPENED pgs=391 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:48:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6789/0 2024-09-16T14:48:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 1 -- 172.21.15.26:0/1700940277 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831177739 0 0) 0x7fac5009cf60 con 0x7fac50138ff0 2024-09-16T14:48:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient(hunting): my global_id is 114680 2024-09-16T14:48:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 1 -- 172.21.15.26:0/1700940277 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac38003530 con 0x7fac50138ff0 2024-09-16T14:48:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 1 -- 172.21.15.26:0/1700940277 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910532269 0 0) 0x7fac50132a80 con 0x7fac5009cb80 2024-09-16T14:48:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient(hunting): my global_id is 144160 2024-09-16T14:48:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 1 -- 172.21.15.26:0/1700940277 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac5009cf60 con 0x7fac5009cb80 2024-09-16T14:48:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 1 -- 172.21.15.26:0/1700940277 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2540696766 0 0) 0x7fac38003530 con 0x7fac50138ff0 2024-09-16T14:48:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 1 -- 172.21.15.26:0/1700940277 >> v1:172.21.15.189:6789/0 conn(0x7fac5009cb80 legacy=0x7fac500717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 1 -- 172.21.15.26:0/1700940277 >> v1:172.21.15.26:6789/0 conn(0x7fac50132650 legacy=0x7fac501378d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 1 monclient: found mon.noname-b 2024-09-16T14:48:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:48:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 1 -- 172.21.15.26:0/1700940277 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5009f2d0 con 0x7fac50138ff0 2024-09-16T14:48:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 1 -- 172.21.15.26:0/1700940277 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fac48001660 con 0x7fac50138ff0 2024-09-16T14:48:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient: dump: 2024-09-16T14:48:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.871 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 5 monclient: authenticate success, global_id 114680 2024-09-16T14:48:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 1 -- 172.21.15.26:0/1700940277 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac48001880 con 0x7fac50138ff0 2024-09-16T14:48:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 1 -- 172.21.15.26:0/1700940277 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fac48001c10 con 0x7fac50138ff0 2024-09-16T14:48:20.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:20.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac4e7fc640 10 monclient: dump: 2024-09-16T14:48:20.872 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.872 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.872 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.872 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.872 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.872 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.872 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.872 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.872 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.872 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 4 set_mon_vals no callback set 2024-09-16T14:48:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 10 monclient: shutdown 2024-09-16T14:48:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 1 -- 172.21.15.26:0/1700940277 >> v1:172.21.15.26:6790/0 conn(0x7fac50138ff0 legacy=0x7fac5013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 1 -- 172.21.15.26:0/1700940277 shutdown_connections 2024-09-16T14:48:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 1 -- 172.21.15.26:0/1700940277 >> 172.21.15.26:0/1700940277 conn(0x7fac5009fb50 msgr2=0x7fac5009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 1 -- 172.21.15.26:0/1700940277 shutdown_connections 2024-09-16T14:48:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.855+0000 7fac55c2e640 1 -- 172.21.15.26:0/1700940277 wait complete. 2024-09-16T14:48:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 10 monclient: build_initial_monmap 2024-09-16T14:48:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 10 monclient: monmap: 2024-09-16T14:48:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:20.873 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:20.874 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:20.874 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:20.874 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:20.874 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:20.874 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:20.874 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:20.874 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 1 Processor -- start 2024-09-16T14:48:20.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 1 -- start start 2024-09-16T14:48:20.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 10 monclient: init 2024-09-16T14:48:20.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:20.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:20.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 10 monclient(hunting): picked mon.noname-b con 0x7fac5009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 10 monclient(hunting): picked mon.noname-c con 0x7fac50132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 10 monclient(hunting): picked mon.noname-a con 0x7fac50138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac50070d50 con 0x7fac50138ff0 2024-09-16T14:48:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5009a240 con 0x7fac5009cb80 2024-09-16T14:48:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5013c520 con 0x7fac50132650 2024-09-16T14:48:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 10 monclient(hunting): authenticate will time out at 1324.981036s 2024-09-16T14:48:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac4effd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fac50132650 0x7fac5009aec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53124/0 (socket says 172.21.15.26:53124) 2024-09-16T14:48:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac4effd640 1 -- 172.21.15.26:0/1654159624 learned_addr learned my addr 172.21.15.26:0/1654159624 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927579573 0 0) 0x7fac5013c520 con 0x7fac50132650 2024-09-16T14:48:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): my global_id is 114683 2024-09-16T14:48:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac3c003530 con 0x7fac50132650 2024-09-16T14:48:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212018137 0 0) 0x7fac50070d50 con 0x7fac50138ff0 2024-09-16T14:48:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): my global_id is 124371 2024-09-16T14:48:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac5013c520 con 0x7fac50138ff0 2024-09-16T14:48:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641526074 0 0) 0x7fac5009a240 con 0x7fac5009cb80 2024-09-16T14:48:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): my global_id is 144163 2024-09-16T14:48:20.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac50070d50 con 0x7fac5009cb80 2024-09-16T14:48:20.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2516140457 0 0) 0x7fac3c003530 con 0x7fac50132650 2024-09-16T14:48:20.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac5009a240 con 0x7fac50132650 2024-09-16T14:48:20.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fac44001970 con 0x7fac50132650 2024-09-16T14:48:20.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:20.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): dump: 2024-09-16T14:48:20.877 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.877 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.877 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.877 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.877 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.878 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.878 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.878 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.878 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.878 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:20.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2797273027 0 0) 0x7fac5013c520 con 0x7fac50138ff0 2024-09-16T14:48:20.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac3c003530 con 0x7fac50138ff0 2024-09-16T14:48:20.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fac48001620 con 0x7fac50138ff0 2024-09-16T14:48:20.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:20.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): dump: 2024-09-16T14:48:20.878 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.879 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.879 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.879 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.879 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.879 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.879 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.879 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.879 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.879 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 438561355 0 0) 0x7fac5009a240 con 0x7fac50132650 2024-09-16T14:48:20.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 >> v1:172.21.15.189:6789/0 conn(0x7fac5009cb80 legacy=0x7fac5009a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 >> v1:172.21.15.26:6789/0 conn(0x7fac50138ff0 legacy=0x7fac501d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:20.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 monclient: found mon.c 2024-09-16T14:48:20.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:20.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:20.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5006dee0 con 0x7fac50132650 2024-09-16T14:48:20.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient: _finish_auth 0 2024-09-16T14:48:20.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:20.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:20.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 5 monclient: authenticate success, global_id 114683 2024-09-16T14:48:20.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 10 monclient: _renew_subs 2024-09-16T14:48:20.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:20.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 1 -- 172.21.15.26:0/1654159624 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5009c4e0 con 0x7fac50132650 2024-09-16T14:48:20.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 10 monclient: _renew_subs 2024-09-16T14:48:20.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:20.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 1 -- 172.21.15.26:0/1654159624 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5009bb90 con 0x7fac50132650 2024-09-16T14:48:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac44001c50 con 0x7fac50132650 2024-09-16T14:48:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fac44001ad0 con 0x7fac50132650 2024-09-16T14:48:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient: dump: 2024-09-16T14:48:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.882 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.882 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac4dffb640 4 set_mon_vals no callback set 2024-09-16T14:48:20.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:20.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac55c2e640 1 -- 172.21.15.26:0/1654159624 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fac30006590 con 0x7fac30002830 2024-09-16T14:48:20.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fac44004c40 con 0x7fac50132650 2024-09-16T14:48:20.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(155..155 src has 73..155) v4 ==== 8989+0+0 (unknown 1378740788 0 0) 0x7fac440446f0 con 0x7fac50132650 2024-09-16T14:48:20.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618370355 0 0) 0x7fac30006590 con 0x7fac30002830 2024-09-16T14:48:20.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient(hunting): my global_id is 114686 2024-09-16T14:48:20.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac5009a240 con 0x7fac30002830 2024-09-16T14:48:20.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac4ffff640 10 monclient: get_auth_request con 0x7fac3c036300 auth_method 0 2024-09-16T14:48:20.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2630797947 0 0) 0x7fac5009a240 con 0x7fac30002830 2024-09-16T14:48:20.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac30006590 con 0x7fac30002830 2024-09-16T14:48:20.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fac40001a70 con 0x7fac30002830 2024-09-16T14:48:20.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:20.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 10 monclient: dump: 2024-09-16T14:48:20.883 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.883 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.883 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.883 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.883 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.883 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.884 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.884 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.884 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.884 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3121371161 0 0) 0x7fac30006590 con 0x7fac30002830 2024-09-16T14:48:20.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.859+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fac3000a0e0 con 0x7fac30002830 2024-09-16T14:48:20.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.863+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fac40001d20 con 0x7fac30002830 2024-09-16T14:48:20.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.863+0000 7fac37fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:20.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.863+0000 7fac37fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:20.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.863+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 >> v1:172.21.15.26:6790/0 conn(0x7fac30002830 legacy=0x7fac30004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.863+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 >> v1:172.21.15.26:6790/0 conn(0x7fac30002830 legacy=0x7fac30004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:20.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac55c2e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:48:20.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac55c2e640 1 -- 172.21.15.26:0/1654159624 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fac30009dc0 con 0x7fac30009080 2024-09-16T14:48:20.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517752898 0 0) 0x7fac30009dc0 con 0x7fac30009080 2024-09-16T14:48:20.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 10 monclient(hunting): my global_id is 114689 2024-09-16T14:48:20.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac30006590 con 0x7fac30009080 2024-09-16T14:48:20.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1978382528 0 0) 0x7fac30006590 con 0x7fac30009080 2024-09-16T14:48:20.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac30009dc0 con 0x7fac30009080 2024-09-16T14:48:20.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fac40003b20 con 0x7fac30009080 2024-09-16T14:48:20.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:20.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 10 monclient: dump: 2024-09-16T14:48:20.887 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:20.887 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.888 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.888 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.888 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.888 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:20.888 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.888 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.888 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.888 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:20.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2723145768 0 0) 0x7fac30009dc0 con 0x7fac30009080 2024-09-16T14:48:20.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fac3000c2d0 con 0x7fac30009080 2024-09-16T14:48:20.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 2994100565) 0x7fac40004fe0 con 0x7fac30009080 2024-09-16T14:48:20.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:48:20.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:20.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 >> v1:172.21.15.26:6790/0 conn(0x7fac30009080 legacy=0x7fac30009510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac37fff640 1 -- 172.21.15.26:0/1654159624 >> v1:172.21.15.26:6790/0 conn(0x7fac30009080 legacy=0x7fac30009510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:20.889 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:48:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:48:20.889 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 54, 2024-09-16T14:48:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:48:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:48:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:48:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:48:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:48:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:48:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:48:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:20.890 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:48:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:48:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:20.891 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.892 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:48:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:48:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:48:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:48:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:48:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:48:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:48:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:48:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:48:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:48:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:48:20.893 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:48:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:48:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:48:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:20.894 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:48:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:48:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:20.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:48:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:48:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:48:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:20.896 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:48:20.897 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:48:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:48:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:20.898 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:48:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:48:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:48:20.899 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:48:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:48:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:20.900 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:48:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:48:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:48:20.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:48:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:20.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:20.903 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:48:20.903 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:48:20.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac55c2e640 1 -- 172.21.15.26:0/1654159624 >> v1:172.21.15.189:6812/3196161582 conn(0x7fac3c036300 legacy=0x7fac3c0387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac55c2e640 10 monclient: shutdown 2024-09-16T14:48:20.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac55c2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:20.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac55c2e640 1 -- 172.21.15.26:0/1654159624 >> v1:172.21.15.26:6790/0 conn(0x7fac50132650 legacy=0x7fac5009aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac4ffff640 1 -- 172.21.15.26:0/1654159624 reap_dead start 2024-09-16T14:48:20.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac4ffff640 1 -- 172.21.15.26:0/1654159624 reap_dead start 2024-09-16T14:48:20.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac55c2e640 1 -- 172.21.15.26:0/1654159624 shutdown_connections 2024-09-16T14:48:20.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac55c2e640 1 -- 172.21.15.26:0/1654159624 >> 172.21.15.26:0/1654159624 conn(0x7fac5009fb50 msgr2=0x7fac501388c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:20.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac55c2e640 1 -- 172.21.15.26:0/1654159624 shutdown_connections 2024-09-16T14:48:20.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:20.867+0000 7fac55c2e640 1 -- 172.21.15.26:0/1654159624 wait complete. 2024-09-16T14:48:20.907 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:48:20.907 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:48:20.907 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:48:20.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 10 monclient: get_monmap_and_config 2024-09-16T14:48:20.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 10 monclient: build_initial_monmap 2024-09-16T14:48:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 10 monclient: monmap: 2024-09-16T14:48:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:20.878844+0000 2024-09-16T14:48:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:20.878844+0000 2024-09-16T14:48:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:20.913 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 1 Processor -- start 2024-09-16T14:48:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 1 -- start start 2024-09-16T14:48:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 10 monclient: init 2024-09-16T14:48:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 10 monclient(hunting): picked mon.noname-c con 0x7fe7f01315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 10 monclient(hunting): picked mon.noname-b con 0x7fe7f009b320 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 10 monclient(hunting): picked mon.noname-a con 0x7fe7f0132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f009edd0 con 0x7fe7f0132090 2024-09-16T14:48:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f009bc00 con 0x7fe7f01315a0 2024-09-16T14:48:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f009e900 con 0x7fe7f009b320 2024-09-16T14:48:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 10 monclient(hunting): authenticate will time out at 1282.995281s 2024-09-16T14:48:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ee59c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe7f01315a0 0x7fe7f0131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51280/0 (socket says 172.21.15.189:51280) 2024-09-16T14:48:20.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ee59c640 1 -- 172.21.15.189:0/1516532140 learned_addr learned my addr 172.21.15.189:0/1516532140 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:20.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 1 -- 172.21.15.189:0/1516532140 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729340532 0 0) 0x7fe7f009bc00 con 0x7fe7f01315a0 2024-09-16T14:48:20.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 10 monclient(hunting): my global_id is 144166 2024-09-16T14:48:20.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 1 -- 172.21.15.189:0/1516532140 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d8003530 con 0x7fe7f01315a0 2024-09-16T14:48:20.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 1 -- 172.21.15.189:0/1516532140 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1022557916 0 0) 0x7fe7d8003530 con 0x7fe7f01315a0 2024-09-16T14:48:20.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 1 -- 172.21.15.189:0/1516532140 >> v1:172.21.15.26:6790/0 conn(0x7fe7f009b320 legacy=0x7fe7f0136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 1 -- 172.21.15.189:0/1516532140 >> v1:172.21.15.26:6789/0 conn(0x7fe7f0132090 legacy=0x7fe7f013a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 10 monclient: _finish_hunting 0 2024-09-16T14:48:20.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 1 monclient: found mon.noname-c 2024-09-16T14:48:20.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 1 -- 172.21.15.189:0/1516532140 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f009b920 con 0x7fe7f01315a0 2024-09-16T14:48:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 10 monclient: _finish_auth 0 2024-09-16T14:48:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 10 monclient: _check_auth_tickets 2024-09-16T14:48:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7f487b640 5 monclient: authenticate success, global_id 144166 2024-09-16T14:48:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 1 -- 172.21.15.189:0/1516532140 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe7e0001620 con 0x7fe7f01315a0 2024-09-16T14:48:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.877+0000 7fe7ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7ed59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7ed59a640 10 monclient: dump: 2024-09-16T14:48:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.917 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7ed59a640 1 -- 172.21.15.189:0/1516532140 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7e0001940 con 0x7fe7f01315a0 2024-09-16T14:48:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7ed59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7ed59a640 1 -- 172.21.15.189:0/1516532140 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe7e0001cd0 con 0x7fe7f01315a0 2024-09-16T14:48:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 4 set_mon_vals no callback set 2024-09-16T14:48:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7ed59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7ed59a640 10 monclient: dump: 2024-09-16T14:48:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.918 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient: shutdown 2024-09-16T14:48:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 1 -- 172.21.15.189:0/1516532140 >> v1:172.21.15.189:6789/0 conn(0x7fe7f01315a0 legacy=0x7fe7f0131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 1 -- 172.21.15.189:0/1516532140 shutdown_connections 2024-09-16T14:48:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 1 -- 172.21.15.189:0/1516532140 >> 172.21.15.189:0/1516532140 conn(0x7fe7f00a3320 msgr2=0x7fe7f009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 1 -- 172.21.15.189:0/1516532140 shutdown_connections 2024-09-16T14:48:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 1 -- 172.21.15.189:0/1516532140 wait complete. 2024-09-16T14:48:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient: build_initial_monmap 2024-09-16T14:48:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient: monmap: 2024-09-16T14:48:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:20.920 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 1 Processor -- start 2024-09-16T14:48:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 1 -- start start 2024-09-16T14:48:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient: init 2024-09-16T14:48:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient(hunting): picked mon.noname-b con 0x7fe7f01315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient(hunting): picked mon.noname-c con 0x7fe7f0132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient(hunting): picked mon.noname-a con 0x7fe7f01c7010 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f009cd20 con 0x7fe7f01c7010 2024-09-16T14:48:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f011dce0 con 0x7fe7f01315a0 2024-09-16T14:48:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f01c7d40 con 0x7fe7f0132090 2024-09-16T14:48:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient(hunting): authenticate will time out at 1282.998087s 2024-09-16T14:48:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7ee59c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe7f01315a0 0x7fe7f0130b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51286/0 (socket says 172.21.15.189:51286) 2024-09-16T14:48:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7ee59c640 1 -- 172.21.15.189:0/3763706886 learned_addr learned my addr 172.21.15.189:0/3763706886 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992159079 0 0) 0x7fe7f011dce0 con 0x7fe7f01315a0 2024-09-16T14:48:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): my global_id is 144169 2024-09-16T14:48:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d0003530 con 0x7fe7f01315a0 2024-09-16T14:48:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379417479 0 0) 0x7fe7f009cd20 con 0x7fe7f01c7010 2024-09-16T14:48:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): my global_id is 124377 2024-09-16T14:48:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7f011dce0 con 0x7fe7f01c7010 2024-09-16T14:48:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635421915 0 0) 0x7fe7f01c7d40 con 0x7fe7f0132090 2024-09-16T14:48:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): my global_id is 114695 2024-09-16T14:48:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7f009cd20 con 0x7fe7f0132090 2024-09-16T14:48:20.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 440986656 0 0) 0x7fe7d0003530 con 0x7fe7f01315a0 2024-09-16T14:48:20.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7f01c7d40 con 0x7fe7f01315a0 2024-09-16T14:48:20.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe7e00015e0 con 0x7fe7f01315a0 2024-09-16T14:48:20.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:20.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): dump: 2024-09-16T14:48:20.923 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:20.923 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.923 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.923 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.923 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.923 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:20.923 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.924 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.924 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.924 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:20.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1937270384 0 0) 0x7fe7f011dce0 con 0x7fe7f01c7010 2024-09-16T14:48:20.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7d0003530 con 0x7fe7f01c7010 2024-09-16T14:48:20.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe7e8001b40 con 0x7fe7f01c7010 2024-09-16T14:48:20.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:20.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient(hunting): dump: 2024-09-16T14:48:20.924 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:20.924 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.924 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.925 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3045725155 0 0) 0x7fe7f01c7d40 con 0x7fe7f01315a0 2024-09-16T14:48:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 >> v1:172.21.15.26:6790/0 conn(0x7fe7f0132090 legacy=0x7fe7f01cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 >> v1:172.21.15.26:6789/0 conn(0x7fe7f01c7010 legacy=0x7fe7f01cf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 monclient: found mon.b 2024-09-16T14:48:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f011e2a0 con 0x7fe7f01315a0 2024-09-16T14:48:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient: _finish_auth 0 2024-09-16T14:48:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 5 monclient: authenticate success, global_id 144169 2024-09-16T14:48:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient: _renew_subs 2024-09-16T14:48:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 1 -- 172.21.15.189:0/3763706886 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f00a2810 con 0x7fe7f01315a0 2024-09-16T14:48:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient: _renew_subs 2024-09-16T14:48:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 1 -- 172.21.15.189:0/3763706886 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f011da00 con 0x7fe7f01315a0 2024-09-16T14:48:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7e0004c50 con 0x7fe7f01315a0 2024-09-16T14:48:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe7e00010b0 con 0x7fe7f01315a0 2024-09-16T14:48:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7ecd99640 4 set_mon_vals no callback set 2024-09-16T14:48:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 10 monclient: dump: 2024-09-16T14:48:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:20.928 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe7e000f870 con 0x7fe7f01315a0 2024-09-16T14:48:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7f487b640 1 -- 172.21.15.189:0/3763706886 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d4005050 con 0x7fe7f01315a0 2024-09-16T14:48:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.881+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(155..155 src has 73..155) v4 ==== 8989+0+0 (unknown 1378740788 0 0) 0x7fe7e00447c0 con 0x7fe7f01315a0 2024-09-16T14:48:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.885+0000 7fe7edd9b640 10 monclient: get_auth_request con 0x7fe7d0036220 auth_method 0 2024-09-16T14:48:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.889+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe7e0014690 con 0x7fe7f01315a0 2024-09-16T14:48:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.889+0000 7fe7deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:20.889+0000 7fe7deffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:21.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc40dc24640 10 monclient: get_monmap_and_config 2024-09-16T14:48:21.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc40dc24640 10 monclient: build_initial_monmap 2024-09-16T14:48:21.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc40dc24640 10 monclient: monmap: 2024-09-16T14:48:21.026 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:21.026 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:21.026 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:21.026355+0000 2024-09-16T14:48:21.026 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:21.026355+0000 2024-09-16T14:48:21.026 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:21.026 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:21.026 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:21.026 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:21.027 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:21.027 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:21.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc40dc24640 1 Processor -- start 2024-09-16T14:48:21.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc40dc24640 1 -- start start 2024-09-16T14:48:21.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc40dc24640 10 monclient: init 2024-09-16T14:48:21.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc40dc24640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:21.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc40dc24640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:21.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc40dc24640 10 monclient(hunting): picked mon.noname-b con 0x7fc40809cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:21.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc40dc24640 10 monclient(hunting): picked mon.noname-c con 0x7fc40809e7a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:21.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc40dc24640 10 monclient(hunting): picked mon.noname-a con 0x7fc40809d1a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:21.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc40dc24640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40809d5b0 con 0x7fc40809d1a0 2024-09-16T14:48:21.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc40dc24640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc408131a90 con 0x7fc40809e7a0 2024-09-16T14:48:21.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc40dc24640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40809aff0 con 0x7fc40809cdc0 2024-09-16T14:48:21.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc40dc24640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:21.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc40dc24640 10 monclient(hunting): authenticate will time out at 1325.147112s 2024-09-16T14:48:21.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4077fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc40809cdc0 0x7fc408130c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53146/0 (socket says 172.21.15.26:53146) 2024-09-16T14:48:21.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc407fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc40809d1a0 0x7fc408139fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50252/0 (socket says 172.21.15.26:50252) 2024-09-16T14:48:21.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4077fe640 1 -- 172.21.15.26:0/2513815803 learned_addr learned my addr 172.21.15.26:0/2513815803 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:21.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 1 -- 172.21.15.26:0/2513815803 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989651813 0 0) 0x7fc40809aff0 con 0x7fc40809cdc0 2024-09-16T14:48:21.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:21.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:21.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:21.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 10 monclient(hunting): my global_id is 114698 2024-09-16T14:48:21.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 1 -- 172.21.15.26:0/2513815803 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3f0003530 con 0x7fc40809cdc0 2024-09-16T14:48:21.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 1 -- 172.21.15.26:0/2513815803 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053775210 0 0) 0x7fc40809d5b0 con 0x7fc40809d1a0 2024-09-16T14:48:21.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:21.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:21.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:21.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 10 monclient(hunting): my global_id is 124380 2024-09-16T14:48:21.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 1 -- 172.21.15.26:0/2513815803 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc40809aff0 con 0x7fc40809d1a0 2024-09-16T14:48:21.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 1 -- 172.21.15.26:0/2513815803 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2419457939 0 0) 0x7fc40809aff0 con 0x7fc40809d1a0 2024-09-16T14:48:21.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 1 -- 172.21.15.26:0/2513815803 >> v1:172.21.15.26:6790/0 conn(0x7fc40809cdc0 legacy=0x7fc408130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:21.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 1 -- 172.21.15.26:0/2513815803 >> v1:172.21.15.189:6789/0 conn(0x7fc40809e7a0 legacy=0x7fc408131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:21.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:21.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 1 monclient: found mon.noname-a 2024-09-16T14:48:21.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.023+0000 7fc4067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:21.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc4067fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:21.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc4067fc640 1 -- 172.21.15.26:0/2513815803 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40809b2d0 con 0x7fc40809d1a0 2024-09-16T14:48:21.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc4067fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:21.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc4067fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:21.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc4067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:21.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc4067fc640 1 -- 172.21.15.26:0/2513815803 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc4000019a0 con 0x7fc40809d1a0 2024-09-16T14:48:21.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc4067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:21.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc4067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:21.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc4067fc640 10 monclient: dump: 2024-09-16T14:48:21.034 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:21.034 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:21.034 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.034 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.034 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:21.034 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:21.034 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:21.034 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:21.034 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:21.034 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:21.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 5 monclient: authenticate success, global_id 124380 2024-09-16T14:48:21.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:21.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc4067fc640 1 -- 172.21.15.26:0/2513815803 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc400001b50 con 0x7fc40809d1a0 2024-09-16T14:48:21.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc4067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:21.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc4067fc640 1 -- 172.21.15.26:0/2513815803 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc400001f20 con 0x7fc40809d1a0 2024-09-16T14:48:21.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:21.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 4 set_mon_vals no callback set 2024-09-16T14:48:21.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc4067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:21.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc4067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:21.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc4067fc640 10 monclient: dump: 2024-09-16T14:48:21.035 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:21.035 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:21.036 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 10 monclient: shutdown 2024-09-16T14:48:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 1 -- 172.21.15.26:0/2513815803 >> v1:172.21.15.26:6789/0 conn(0x7fc40809d1a0 legacy=0x7fc408139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 1 -- 172.21.15.26:0/2513815803 shutdown_connections 2024-09-16T14:48:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 1 -- 172.21.15.26:0/2513815803 >> 172.21.15.26:0/2513815803 conn(0x7fc40809fb50 msgr2=0x7fc40809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:21.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 1 -- 172.21.15.26:0/2513815803 shutdown_connections 2024-09-16T14:48:21.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 1 -- 172.21.15.26:0/2513815803 wait complete. 2024-09-16T14:48:21.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 10 monclient: build_initial_monmap 2024-09-16T14:48:21.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 10 monclient: monmap: 2024-09-16T14:48:21.037 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:21.037 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:21.037 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:21.037 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:21.037 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:21.037 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:21.037 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:21.037 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:21.037 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:21.037 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:21.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 1 Processor -- start 2024-09-16T14:48:21.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 1 -- start start 2024-09-16T14:48:21.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 10 monclient: init 2024-09-16T14:48:21.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:21.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:21.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 10 monclient(hunting): picked mon.noname-b con 0x7fc40809cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:21.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 10 monclient(hunting): picked mon.noname-c con 0x7fc40809d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:21.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 10 monclient(hunting): picked mon.noname-a con 0x7fc40809e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:21.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc408121010 con 0x7fc40809e7a0 2024-09-16T14:48:21.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40811fb60 con 0x7fc40809cdc0 2024-09-16T14:48:21.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4080a2360 con 0x7fc40809d1a0 2024-09-16T14:48:21.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:21.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 10 monclient(hunting): authenticate will time out at 1325.150253s 2024-09-16T14:48:21.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc407fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc40809e7a0 0x7fc4081d1e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50256/0 (socket says 172.21.15.26:50256) 2024-09-16T14:48:21.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc406ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc40809d1a0 0x7fc4081ce750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53150/0 (socket says 172.21.15.26:53150) 2024-09-16T14:48:21.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc407fff640 1 -- 172.21.15.26:0/2692022220 learned_addr learned my addr 172.21.15.26:0/2692022220 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:21.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834757148 0 0) 0x7fc4080a2360 con 0x7fc40809d1a0 2024-09-16T14:48:21.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:21.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:21.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:21.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): my global_id is 114701 2024-09-16T14:48:21.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3f4003530 con 0x7fc40809d1a0 2024-09-16T14:48:21.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795486542 0 0) 0x7fc408121010 con 0x7fc40809e7a0 2024-09-16T14:48:21.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:21.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:21.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:21.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): my global_id is 124383 2024-09-16T14:48:21.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4080a2360 con 0x7fc40809e7a0 2024-09-16T14:48:21.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4185348789 0 0) 0x7fc3f4003530 con 0x7fc40809d1a0 2024-09-16T14:48:21.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc408121010 con 0x7fc40809d1a0 2024-09-16T14:48:21.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc3fc0018c0 con 0x7fc40809d1a0 2024-09-16T14:48:21.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:21.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:21.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): dump: 2024-09-16T14:48:21.047 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:21.048 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:21.048 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.048 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.048 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:21.048 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:21.048 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:21.048 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:21.049 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:21.049 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:21.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:21.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 175054728 0 0) 0x7fc4080a2360 con 0x7fc40809e7a0 2024-09-16T14:48:21.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3f4003530 con 0x7fc40809e7a0 2024-09-16T14:48:21.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc400001e60 con 0x7fc40809e7a0 2024-09-16T14:48:21.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:21.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:21.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): dump: 2024-09-16T14:48:21.049 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:21.049 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:21.049 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.049 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.049 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:21.050 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:21.050 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:21.050 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:21.050 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:21.050 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:21.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265902944 0 0) 0x7fc40811fb60 con 0x7fc40809cdc0 2024-09-16T14:48:21.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:21.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:21.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:21.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient(hunting): my global_id is 144175 2024-09-16T14:48:21.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4080a2360 con 0x7fc40809cdc0 2024-09-16T14:48:21.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 397392133 0 0) 0x7fc408121010 con 0x7fc40809d1a0 2024-09-16T14:48:21.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 >> v1:172.21.15.189:6789/0 conn(0x7fc40809cdc0 legacy=0x7fc4081c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:21.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 >> v1:172.21.15.26:6789/0 conn(0x7fc40809e7a0 legacy=0x7fc4081d1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:21.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:21.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 monclient: found mon.c 2024-09-16T14:48:21.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:21.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:21.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc408120d30 con 0x7fc40809d1a0 2024-09-16T14:48:21.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient: _finish_auth 0 2024-09-16T14:48:21.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:21.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:21.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 5 monclient: authenticate success, global_id 114701 2024-09-16T14:48:21.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 10 monclient: _renew_subs 2024-09-16T14:48:21.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:21.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 1 -- 172.21.15.26:0/2692022220 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40811f8e0 con 0x7fc40809d1a0 2024-09-16T14:48:21.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 10 monclient: _renew_subs 2024-09-16T14:48:21.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:21.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc40dc24640 1 -- 172.21.15.26:0/2692022220 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc408071340 con 0x7fc40809d1a0 2024-09-16T14:48:21.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3fc0036a0 con 0x7fc40809d1a0 2024-09-16T14:48:21.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:21.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc3fc003a30 con 0x7fc40809d1a0 2024-09-16T14:48:21.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:21.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:21.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc3effff640 10 monclient: dump: 2024-09-16T14:48:21.053 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:21.053 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:21.053 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.053 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.053 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:21.053 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:21.053 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:21.053 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:21.054 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:21.054 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:21.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.027+0000 7fc405ffb640 4 set_mon_vals no callback set 2024-09-16T14:48:21.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc3fc003c10 con 0x7fc40809d1a0 2024-09-16T14:48:21.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc40dc24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:21.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc40dc24640 1 -- 172.21.15.26:0/2692022220 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc3e8006590 con 0x7fc3e8002830 2024-09-16T14:48:21.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(155..155 src has 73..155) v4 ==== 8989+0+0 (unknown 1378740788 0 0) 0x7fc3fc044750 con 0x7fc40809d1a0 2024-09-16T14:48:21.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc407fff640 10 monclient: get_auth_request con 0x7fc3f4036390 auth_method 0 2024-09-16T14:48:21.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10340166 0 0) 0x7fc3e8006590 con 0x7fc3e8002830 2024-09-16T14:48:21.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:21.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:21.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:21.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 10 monclient(hunting): my global_id is 144178 2024-09-16T14:48:21.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc408121010 con 0x7fc3e8002830 2024-09-16T14:48:21.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2622377733 0 0) 0x7fc408121010 con 0x7fc3e8002830 2024-09-16T14:48:21.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3e8006590 con 0x7fc3e8002830 2024-09-16T14:48:21.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc3f8001860 con 0x7fc3e8002830 2024-09-16T14:48:21.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:21.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:21.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 10 monclient: dump: 2024-09-16T14:48:21.056 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:21.056 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:21.057 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.057 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.057 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:21.057 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:21.057 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:21.057 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:21.057 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:21.057 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:21.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 931506784 0 0) 0x7fc3e8006590 con 0x7fc3e8002830 2024-09-16T14:48:21.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3e800a0e0 con 0x7fc3e8002830 2024-09-16T14:48:21.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc3f8001b10 con 0x7fc3e8002830 2024-09-16T14:48:21.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:21.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:21.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 >> v1:172.21.15.189:6789/0 conn(0x7fc3e8002830 legacy=0x7fc3e8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:21.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.031+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 >> v1:172.21.15.189:6789/0 conn(0x7fc3e8002830 legacy=0x7fc3e8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:21.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc40dc24640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:48:21.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc40dc24640 1 -- 172.21.15.26:0/2692022220 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc3e8009da0 con 0x7fc3e80090c0 2024-09-16T14:48:21.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:21.053+0000 7fe7f487b640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:21.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:21.053+0000 7fe7f487b640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:21.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:21.053+0000 7fe7f487b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:21.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:21.053+0000 7fe7f487b640 1 -- 172.21.15.189:0/3763706886 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7fe7d4005b10 con 0x7fe7f01315a0 2024-09-16T14:48:21.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264498144 0 0) 0x7fc3e8009da0 con 0x7fc3e80090c0 2024-09-16T14:48:21.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:21.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:21.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:21.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 10 monclient(hunting): my global_id is 144181 2024-09-16T14:48:21.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e8006590 con 0x7fc3e80090c0 2024-09-16T14:48:21.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4065313682 0 0) 0x7fc3e8006590 con 0x7fc3e80090c0 2024-09-16T14:48:21.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3e8009da0 con 0x7fc3e80090c0 2024-09-16T14:48:21.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc3f8004990 con 0x7fc3e80090c0 2024-09-16T14:48:21.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:21.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:21.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 10 monclient: dump: 2024-09-16T14:48:21.064 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:21.064 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:21.064 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.064 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.064 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:21.064 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:21.065 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:21.065 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:21.065 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:21.065 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:21.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1796238831 0 0) 0x7fc3e8009da0 con 0x7fc3e80090c0 2024-09-16T14:48:21.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc3e800c290 con 0x7fc3e80090c0 2024-09-16T14:48:21.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 1923863760) 0x7fc3f8004c80 con 0x7fc3e80090c0 2024-09-16T14:48:21.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:48:21.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:21.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 >> v1:172.21.15.189:6789/0 conn(0x7fc3e80090c0 legacy=0x7fc3e8009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:21.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc3effff640 1 -- 172.21.15.26:0/2692022220 >> v1:172.21.15.189:6789/0 conn(0x7fc3e80090c0 legacy=0x7fc3e8009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:48:21.067 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 54, 2024-09-16T14:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:48:21.068 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-09-16T14:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:21.069 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:21.070 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:48:21.071 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:21.072 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:21.073 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:48:21.074 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:48:21.075 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:48:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:48:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:48:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:48:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:48:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:21.076 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:21.077 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:21.078 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:21.079 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:21.080 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:21.081 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:21.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:21.083 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:48:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:21.084 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:21.085 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:21.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:21.087 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:21.088 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:48:21.089 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:48:21.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc40dc24640 1 -- 172.21.15.26:0/2692022220 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc3f4036390 legacy=0x7fc3f4038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:21.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc40dc24640 10 monclient: shutdown 2024-09-16T14:48:21.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc40dc24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:21.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc40dc24640 1 -- 172.21.15.26:0/2692022220 >> v1:172.21.15.26:6790/0 conn(0x7fc40809d1a0 legacy=0x7fc4081ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:21.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc407fff640 1 -- 172.21.15.26:0/2692022220 reap_dead start 2024-09-16T14:48:21.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc407fff640 1 -- 172.21.15.26:0/2692022220 reap_dead start 2024-09-16T14:48:21.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc40dc24640 1 -- 172.21.15.26:0/2692022220 shutdown_connections 2024-09-16T14:48:21.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc40dc24640 1 -- 172.21.15.26:0/2692022220 >> 172.21.15.26:0/2692022220 conn(0x7fc40809fb50 msgr2=0x7fc4081b77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:21.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc40dc24640 1 -- 172.21.15.26:0/2692022220 shutdown_connections 2024-09-16T14:48:21.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.039+0000 7fc40dc24640 1 -- 172.21.15.26:0/2692022220 wait complete. 2024-09-16T14:48:21.092 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:48:21.092 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T14:48:21.092 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:48:21.092 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T14:48:21.092 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T14:48:21.092 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T14:48:21.092 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:48:21.106 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T14:48:21.106 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:48:21.106 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:48:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.267+0000 7f41e5815640 10 monclient: get_monmap_and_config 2024-09-16T14:48:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.267+0000 7f41e5815640 10 monclient: build_initial_monmap 2024-09-16T14:48:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.267+0000 7f41e5815640 10 monclient: monmap: 2024-09-16T14:48:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:21.272540+0000 2024-09-16T14:48:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:21.272540+0000 2024-09-16T14:48:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:21.273 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:21.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.267+0000 7f41e5815640 1 Processor -- start 2024-09-16T14:48:21.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.267+0000 7f41e5815640 1 -- start start 2024-09-16T14:48:21.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.267+0000 7f41e5815640 10 monclient: init 2024-09-16T14:48:21.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:21.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:21.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient(hunting): picked mon.noname-b con 0x7f41e009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:21.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient(hunting): picked mon.noname-c con 0x7f41e0132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:21.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient(hunting): picked mon.noname-a con 0x7f41e0138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:21.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e009d260 con 0x7f41e0138ff0 2024-09-16T14:48:21.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e0132a80 con 0x7f41e0132650 2024-09-16T14:48:21.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e009cf60 con 0x7f41e009cb80 2024-09-16T14:48:21.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:21.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient(hunting): authenticate will time out at 1325.392714s 2024-09-16T14:48:21.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41deffd640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f41e009cb80 legacy=0x7f41e00717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:21.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41df7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f41e0138ff0 0x7f41e013b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50278/0 (socket says 172.21.15.26:50278) 2024-09-16T14:48:21.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41df7fe640 1 -- 172.21.15.26:0/1535430223 learned_addr learned my addr 172.21.15.26:0/1535430223 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:21.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 1 -- 172.21.15.26:0/1535430223 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155292439 0 0) 0x7f41e009d260 con 0x7f41e0138ff0 2024-09-16T14:48:21.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:21.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:21.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:21.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 10 monclient(hunting): my global_id is 124386 2024-09-16T14:48:21.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 1 -- 172.21.15.26:0/1535430223 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41c8003530 con 0x7f41e0138ff0 2024-09-16T14:48:21.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 1 -- 172.21.15.26:0/1535430223 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 289748220 0 0) 0x7f41c8003530 con 0x7f41e0138ff0 2024-09-16T14:48:21.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 1 -- 172.21.15.26:0/1535430223 >> v1:172.21.15.26:6790/0 conn(0x7f41e009cb80 legacy=0x7f41e00717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:21.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 1 -- 172.21.15.26:0/1535430223 >> v1:172.21.15.189:6789/0 conn(0x7f41e0132650 legacy=0x7f41e01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:21.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 10 monclient: _finish_hunting 0 2024-09-16T14:48:21.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 1 monclient: found mon.noname-a 2024-09-16T14:48:21.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:21.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:21.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 1 -- 172.21.15.26:0/1535430223 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e009f2d0 con 0x7f41e0138ff0 2024-09-16T14:48:21.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 10 monclient: _finish_auth 0 2024-09-16T14:48:21.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:21.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:21.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 1 -- 172.21.15.26:0/1535430223 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f41d80019a0 con 0x7f41e0138ff0 2024-09-16T14:48:21.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 5 monclient: authenticate success, global_id 124386 2024-09-16T14:48:21.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:21.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:21.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:21.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 10 monclient: dump: 2024-09-16T14:48:21.283 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:21.283 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:21.283 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.283 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.283 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:21.283 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:21.283 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:21.283 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:21.283 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:21.283 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:21.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:21.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 1 -- 172.21.15.26:0/1535430223 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41d8001c50 con 0x7f41e0138ff0 2024-09-16T14:48:21.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:21.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:21.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 1 -- 172.21.15.26:0/1535430223 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f41d8002020 con 0x7f41e0138ff0 2024-09-16T14:48:21.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:21.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 4 set_mon_vals no callback set 2024-09-16T14:48:21.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:21.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41ddffb640 10 monclient: dump: 2024-09-16T14:48:21.284 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:21.284 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:21.284 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.284 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.284 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:21.284 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:21.285 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:21.285 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:21.285 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:21.285 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:21.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient: shutdown 2024-09-16T14:48:21.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:21.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 1 -- 172.21.15.26:0/1535430223 >> v1:172.21.15.26:6789/0 conn(0x7f41e0138ff0 legacy=0x7f41e013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:21.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 1 -- 172.21.15.26:0/1535430223 shutdown_connections 2024-09-16T14:48:21.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 1 -- 172.21.15.26:0/1535430223 >> 172.21.15.26:0/1535430223 conn(0x7f41e009fb50 msgr2=0x7f41e009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:21.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 1 -- 172.21.15.26:0/1535430223 shutdown_connections 2024-09-16T14:48:21.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 1 -- 172.21.15.26:0/1535430223 wait complete. 2024-09-16T14:48:21.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient: build_initial_monmap 2024-09-16T14:48:21.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient: monmap: 2024-09-16T14:48:21.286 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:21.286 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:21.286 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:21.286 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:21.286 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:21.286 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:21.286 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:21.286 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:21.286 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:21.286 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:21.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 1 Processor -- start 2024-09-16T14:48:21.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 1 -- start start 2024-09-16T14:48:21.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient: init 2024-09-16T14:48:21.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:21.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:21.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient(hunting): picked mon.noname-c con 0x7f41e009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:21.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient(hunting): picked mon.noname-b con 0x7f41e0132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:21.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient(hunting): picked mon.noname-a con 0x7f41e0138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:21.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e009b9e0 con 0x7f41e0138ff0 2024-09-16T14:48:21.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e006dda0 con 0x7f41e0132650 2024-09-16T14:48:21.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41deffd640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f41e009cb80 legacy=0x7f41e009a550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:21.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e013c590 con 0x7f41e009cb80 2024-09-16T14:48:21.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:21.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41e5815640 10 monclient(hunting): authenticate will time out at 1325.395854s 2024-09-16T14:48:21.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41df7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f41e0138ff0 0x7f41e01d1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50294/0 (socket says 172.21.15.26:50294) 2024-09-16T14:48:21.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.271+0000 7f41df7fe640 1 -- 172.21.15.26:0/2370486685 learned_addr learned my addr 172.21.15.26:0/2370486685 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:21.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298855910 0 0) 0x7f41e009b9e0 con 0x7f41e0138ff0 2024-09-16T14:48:21.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:21.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:21.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:21.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient(hunting): my global_id is 124389 2024-09-16T14:48:21.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41cc003530 con 0x7f41e0138ff0 2024-09-16T14:48:21.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3268931203 0 0) 0x7f41cc003530 con 0x7f41e0138ff0 2024-09-16T14:48:21.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41e009b9e0 con 0x7f41e0138ff0 2024-09-16T14:48:21.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f41d8001f60 con 0x7f41e0138ff0 2024-09-16T14:48:21.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:21.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:21.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient(hunting): dump: 2024-09-16T14:48:21.288 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:21.288 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:21.289 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.289 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.289 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:21.289 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:21.289 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:21.289 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:21.289 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:21.289 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:21.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:21.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224668204 0 0) 0x7f41e006dda0 con 0x7f41e0132650 2024-09-16T14:48:21.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:21.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:21.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:21.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient(hunting): my global_id is 144187 2024-09-16T14:48:21.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41cc003530 con 0x7f41e0132650 2024-09-16T14:48:21.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4139749622 0 0) 0x7f41e009b9e0 con 0x7f41e0138ff0 2024-09-16T14:48:21.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 >> v1:172.21.15.26:6790/0 conn(0x7f41e009cb80 legacy=0x7f41e009a550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:21.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 >> v1:172.21.15.189:6789/0 conn(0x7f41e0132650 legacy=0x7f41e009ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:21.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:21.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 1 monclient: found mon.a 2024-09-16T14:48:21.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:21.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:21.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e009a240 con 0x7f41e0138ff0 2024-09-16T14:48:21.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient: _finish_auth 0 2024-09-16T14:48:21.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:21.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:21.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41e5815640 5 monclient: authenticate success, global_id 124389 2024-09-16T14:48:21.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41e5815640 10 monclient: _renew_subs 2024-09-16T14:48:21.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41e5815640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:21.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41e5815640 1 -- 172.21.15.26:0/2370486685 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e009b370 con 0x7f41e0138ff0 2024-09-16T14:48:21.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41e5815640 10 monclient: _renew_subs 2024-09-16T14:48:21.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41e5815640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:21.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41e5815640 1 -- 172.21.15.26:0/2370486685 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e0071740 con 0x7f41e0138ff0 2024-09-16T14:48:21.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41d8004f80 con 0x7f41e0138ff0 2024-09-16T14:48:21.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:21.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f41d8005290 con 0x7f41e0138ff0 2024-09-16T14:48:21.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:21.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:21.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41dd7fa640 4 set_mon_vals no callback set 2024-09-16T14:48:21.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 10 monclient: dump: 2024-09-16T14:48:21.292 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:21.292 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:21.292 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.292 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:21.292 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:21.292 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:21.292 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:21.292 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:21.292 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:21.292 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:21.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41e5815640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:21.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41e5815640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:21.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41e5815640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:21.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41e5815640 1 -- 172.21.15.26:0/2370486685 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c0005050 con 0x7f41e0138ff0 2024-09-16T14:48:21.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f41d800fd00 con 0x7f41e0138ff0 2024-09-16T14:48:21.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(155..155 src has 73..155) v4 ==== 8989+0+0 (unknown 1378740788 0 0) 0x7f41d8044b50 con 0x7f41e0138ff0 2024-09-16T14:48:21.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.275+0000 7f41deffd640 10 monclient: get_auth_request con 0x7f41cc036140 auth_method 0 2024-09-16T14:48:21.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.279+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f41d8014b60 con 0x7f41e0138ff0 2024-09-16T14:48:21.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.279+0000 7f41c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:21.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.279+0000 7f41c7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:21.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.503+0000 7f41e5815640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:48:21.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.503+0000 7f41e5815640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:48:21.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.503+0000 7f41e5815640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:21.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.507+0000 7f41e5815640 1 -- 172.21.15.26:0/2370486685 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f41c0005300 con 0x7f41e0138ff0 2024-09-16T14:48:21.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.507+0000 7f41c7fff640 1 -- 172.21.15.26:0/2370486685 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1661569585) 0x7f41d8001560 con 0x7f41e0138ff0 2024-09-16T14:48:21.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.507+0000 7f41c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:48:21.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.507+0000 7f41c7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:21.510 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:48:21.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.507+0000 7f41e5815640 1 -- 172.21.15.26:0/2370486685 >> v1:172.21.15.189:6812/3196161582 conn(0x7f41cc036140 legacy=0x7f41cc038600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:21.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.507+0000 7f41e5815640 10 monclient: shutdown 2024-09-16T14:48:21.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.507+0000 7f41e5815640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:21.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.507+0000 7f41e5815640 1 -- 172.21.15.26:0/2370486685 >> v1:172.21.15.26:6789/0 conn(0x7f41e0138ff0 legacy=0x7f41e01d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:21.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.507+0000 7f41e5815640 1 -- 172.21.15.26:0/2370486685 shutdown_connections 2024-09-16T14:48:21.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.507+0000 7f41e5815640 1 -- 172.21.15.26:0/2370486685 >> 172.21.15.26:0/2370486685 conn(0x7f41e009fb50 msgr2=0x7f41e01305e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:21.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.507+0000 7f41e5815640 1 -- 172.21.15.26:0/2370486685 shutdown_connections 2024-09-16T14:48:21.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:21.507+0000 7f41e5815640 1 -- 172.21.15.26:0/2370486685 wait complete. 2024-09-16T14:48:23.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:23.881+0000 7fe7ddffb640 10 monclient: tick 2024-09-16T14:48:23.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:23.881+0000 7fe7ddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:23.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:23.881+0000 7fe7ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:23.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:23.881+0000 7fe7ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:24.529 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:48:24.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 10 monclient: get_monmap_and_config 2024-09-16T14:48:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 10 monclient: build_initial_monmap 2024-09-16T14:48:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 10 monclient: monmap: 2024-09-16T14:48:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:24.662343+0000 2024-09-16T14:48:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:24.662343+0000 2024-09-16T14:48:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:24.663 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 1 Processor -- start 2024-09-16T14:48:24.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 1 -- start start 2024-09-16T14:48:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 10 monclient: init 2024-09-16T14:48:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 10 monclient(hunting): picked mon.noname-c con 0x7fd50809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 10 monclient(hunting): picked mon.noname-b con 0x7fd508132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 10 monclient(hunting): picked mon.noname-a con 0x7fd508138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50809d260 con 0x7fd508138ff0 2024-09-16T14:48:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd508132a80 con 0x7fd50809cb80 2024-09-16T14:48:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50809cf60 con 0x7fd508132650 2024-09-16T14:48:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 10 monclient(hunting): authenticate will time out at 1328.782338s 2024-09-16T14:48:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd506ffd640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fd508132650 legacy=0x7fd5081378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd507fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd508138ff0 0x7fd50813b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50308/0 (socket says 172.21.15.26:50308) 2024-09-16T14:48:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd507fff640 1 -- 172.21.15.26:0/3179473776 learned_addr learned my addr 172.21.15.26:0/3179473776 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 1 -- 172.21.15.26:0/3179473776 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255566433 0 0) 0x7fd50809d260 con 0x7fd508138ff0 2024-09-16T14:48:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 10 monclient(hunting): my global_id is 124392 2024-09-16T14:48:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 1 -- 172.21.15.26:0/3179473776 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f4003530 con 0x7fd508138ff0 2024-09-16T14:48:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 1 -- 172.21.15.26:0/3179473776 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3842177453 0 0) 0x7fd4f4003530 con 0x7fd508138ff0 2024-09-16T14:48:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 1 -- 172.21.15.26:0/3179473776 >> v1:172.21.15.26:6790/0 conn(0x7fd508132650 legacy=0x7fd5081378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 1 -- 172.21.15.26:0/3179473776 >> v1:172.21.15.189:6789/0 conn(0x7fd50809cb80 legacy=0x7fd5080717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 1 monclient: found mon.noname-a 2024-09-16T14:48:24.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 1 -- 172.21.15.26:0/3179473776 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50809f2d0 con 0x7fd508138ff0 2024-09-16T14:48:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 1 -- 172.21.15.26:0/3179473776 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd5000019a0 con 0x7fd508138ff0 2024-09-16T14:48:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 10 monclient: dump: 2024-09-16T14:48:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:24.671 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:24.672 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 5 monclient: authenticate success, global_id 124392 2024-09-16T14:48:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 1 -- 172.21.15.26:0/3179473776 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd500001c50 con 0x7fd508138ff0 2024-09-16T14:48:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 4 set_mon_vals no callback set 2024-09-16T14:48:24.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 1 -- 172.21.15.26:0/3179473776 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd500002020 con 0x7fd508138ff0 2024-09-16T14:48:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd5067fc640 10 monclient: dump: 2024-09-16T14:48:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:24.673 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:24.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 10 monclient: shutdown 2024-09-16T14:48:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 1 -- 172.21.15.26:0/3179473776 >> v1:172.21.15.26:6789/0 conn(0x7fd508138ff0 legacy=0x7fd50813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 1 -- 172.21.15.26:0/3179473776 shutdown_connections 2024-09-16T14:48:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.659+0000 7fd50de87640 1 -- 172.21.15.26:0/3179473776 >> 172.21.15.26:0/3179473776 conn(0x7fd50809fb50 msgr2=0x7fd50809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 1 -- 172.21.15.26:0/3179473776 shutdown_connections 2024-09-16T14:48:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 1 -- 172.21.15.26:0/3179473776 wait complete. 2024-09-16T14:48:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 monclient: build_initial_monmap 2024-09-16T14:48:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 monclient: monmap: 2024-09-16T14:48:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:24.674 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:24.675 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 1 Processor -- start 2024-09-16T14:48:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 1 -- start start 2024-09-16T14:48:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 monclient: init 2024-09-16T14:48:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 monclient(hunting): picked mon.noname-a con 0x7fd50809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 monclient(hunting): picked mon.noname-c con 0x7fd508132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:24.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 monclient(hunting): picked mon.noname-b con 0x7fd508138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5080719c0 con 0x7fd50809cb80 2024-09-16T14:48:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50813c720 con 0x7fd508138ff0 2024-09-16T14:48:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50809a240 con 0x7fd508132650 2024-09-16T14:48:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 monclient(hunting): authenticate will time out at 1328.785456s 2024-09-16T14:48:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd506ffd640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fd508132650 legacy=0x7fd50809ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd5077fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd50809cb80 0x7fd50809a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50324/0 (socket says 172.21.15.26:50324) 2024-09-16T14:48:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd5077fe640 1 -- 172.21.15.26:0/1842272628 learned_addr learned my addr 172.21.15.26:0/1842272628 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963379701 0 0) 0x7fd4f00011d0 con 0x7fd50809cb80 2024-09-16T14:48:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:24.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient(hunting): my global_id is 124395 2024-09-16T14:48:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e8003530 con 0x7fd50809cb80 2024-09-16T14:48:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3155912486 0 0) 0x7fd4e8003530 con 0x7fd50809cb80 2024-09-16T14:48:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4f00011d0 con 0x7fd50809cb80 2024-09-16T14:48:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd4f0001900 con 0x7fd50809cb80 2024-09-16T14:48:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient(hunting): dump: 2024-09-16T14:48:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:24.677 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:24.678 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563706837 0 0) 0x7fd50813c720 con 0x7fd508138ff0 2024-09-16T14:48:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient(hunting): my global_id is 144193 2024-09-16T14:48:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e8003530 con 0x7fd508138ff0 2024-09-16T14:48:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2130087726 0 0) 0x7fd4f00011d0 con 0x7fd50809cb80 2024-09-16T14:48:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 >> v1:172.21.15.26:6790/0 conn(0x7fd508132650 legacy=0x7fd50809ac60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 >> v1:172.21.15.189:6789/0 conn(0x7fd508138ff0 legacy=0x7fd5081d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 1 monclient: found mon.a 2024-09-16T14:48:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50809b370 con 0x7fd50809cb80 2024-09-16T14:48:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient: _finish_auth 0 2024-09-16T14:48:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 5 monclient: authenticate success, global_id 124395 2024-09-16T14:48:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 monclient: _renew_subs 2024-09-16T14:48:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 1 -- 172.21.15.26:0/1842272628 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5081349f0 con 0x7fd50809cb80 2024-09-16T14:48:24.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 monclient: _renew_subs 2024-09-16T14:48:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 1 -- 172.21.15.26:0/1842272628 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd50809ba10 con 0x7fd50809cb80 2024-09-16T14:48:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4f0003660 con 0x7fd50809cb80 2024-09-16T14:48:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd4f00039f0 con 0x7fd50809cb80 2024-09-16T14:48:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd505ffb640 4 set_mon_vals no callback set 2024-09-16T14:48:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 10 monclient: dump: 2024-09-16T14:48:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:24.681 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd4f0003b50 con 0x7fd50809cb80 2024-09-16T14:48:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd50de87640 1 -- 172.21.15.26:0/1842272628 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ec005050 con 0x7fd50809cb80 2024-09-16T14:48:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(155..155 src has 73..155) v4 ==== 8989+0+0 (unknown 1378740788 0 0) 0x7fd4f0044680 con 0x7fd50809cb80 2024-09-16T14:48:24.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.663+0000 7fd506ffd640 10 monclient: get_auth_request con 0x7fd4e8036240 auth_method 0 2024-09-16T14:48:24.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.671+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd4f00146f0 con 0x7fd50809cb80 2024-09-16T14:48:24.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.671+0000 7fd4fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:24.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.671+0000 7fd4fffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:24.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:24.881+0000 7fe7ddffb640 10 monclient: tick 2024-09-16T14:48:24.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:24.881+0000 7fe7ddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:24.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:24.881+0000 7fe7ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:24.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.891+0000 7fd50de87640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:48:24.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.891+0000 7fd50de87640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:48:24.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.891+0000 7fd50de87640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:24.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.891+0000 7fd50de87640 1 -- 172.21.15.26:0/1842272628 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd4ec005300 con 0x7fd50809cb80 2024-09-16T14:48:24.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.891+0000 7fd4fffff640 1 -- 172.21.15.26:0/1842272628 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 870663184) 0x7fd4f00142e0 con 0x7fd50809cb80 2024-09-16T14:48:24.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.891+0000 7fd4fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:48:24.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.891+0000 7fd4fffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:24.898 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:48:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.895+0000 7fd50de87640 1 -- 172.21.15.26:0/1842272628 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd4e8036240 legacy=0x7fd4e8038700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.895+0000 7fd50de87640 10 monclient: shutdown 2024-09-16T14:48:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.895+0000 7fd50de87640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.895+0000 7fd50de87640 1 -- 172.21.15.26:0/1842272628 >> v1:172.21.15.26:6789/0 conn(0x7fd50809cb80 legacy=0x7fd50809a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.895+0000 7fd50de87640 1 -- 172.21.15.26:0/1842272628 shutdown_connections 2024-09-16T14:48:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.895+0000 7fd50de87640 1 -- 172.21.15.26:0/1842272628 >> 172.21.15.26:0/1842272628 conn(0x7fd50809fb50 msgr2=0x7fd508131210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:24.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.895+0000 7fd50de87640 1 -- 172.21.15.26:0/1842272628 shutdown_connections 2024-09-16T14:48:24.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:24.895+0000 7fd50de87640 1 -- 172.21.15.26:0/1842272628 wait complete. 2024-09-16T14:48:25.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:25.881+0000 7fe7ddffb640 10 monclient: tick 2024-09-16T14:48:25.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:25.881+0000 7fe7ddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:25.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:25.881+0000 7fe7ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:26.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:26.881+0000 7fe7ddffb640 10 monclient: tick 2024-09-16T14:48:26.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:26.881+0000 7fe7ddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:26.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:26.881+0000 7fe7ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:27.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:27.881+0000 7fe7ddffb640 10 monclient: tick 2024-09-16T14:48:27.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:27.881+0000 7fe7ddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:27.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:27.881+0000 7fe7ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:27.916 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:48:28.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce2a37a640 10 monclient: get_monmap_and_config 2024-09-16T14:48:28.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce2a37a640 10 monclient: build_initial_monmap 2024-09-16T14:48:28.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce2a37a640 10 monclient: monmap: 2024-09-16T14:48:28.047 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:28.047 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:28.047 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:28.047188+0000 2024-09-16T14:48:28.047 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:28.047188+0000 2024-09-16T14:48:28.047 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:28.047 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:28.047 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:28.047 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:28.047 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:28.047 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:28.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce2a37a640 1 Processor -- start 2024-09-16T14:48:28.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce2a37a640 1 -- start start 2024-09-16T14:48:28.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce2a37a640 10 monclient: init 2024-09-16T14:48:28.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce2a37a640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:28.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce2a37a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:28.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce2a37a640 10 monclient(hunting): picked mon.noname-b con 0x7fce2406c5a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:28.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce2a37a640 10 monclient(hunting): picked mon.noname-a con 0x7fce240a23b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:28.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce2a37a640 10 monclient(hunting): picked mon.noname-c con 0x7fce240a27c0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:28.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce2a37a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2406cee0 con 0x7fce240a23b0 2024-09-16T14:48:28.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce2a37a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2406c980 con 0x7fce240a27c0 2024-09-16T14:48:28.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce2a37a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce240a15e0 con 0x7fce2406c5a0 2024-09-16T14:48:28.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce2a37a640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:28.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce2a37a640 10 monclient(hunting): authenticate will time out at 1332.167289s 2024-09-16T14:48:28.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce23fff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fce2406c5a0 legacy=0x7fce240a1ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:28.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce237fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fce240a23b0 0x7fce24139eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53898/0 (socket says 172.21.15.26:53898) 2024-09-16T14:48:28.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.043+0000 7fce237fe640 1 -- 172.21.15.26:0/3644725818 learned_addr learned my addr 172.21.15.26:0/3644725818 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:28.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.247+0000 7fce23fff640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6790/0 conn(0x7fce2406c5a0 legacy=0x7fce240a1ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:28.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:28.651+0000 7fce23fff640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6790/0 conn(0x7fce2406c5a0 legacy=0x7fce240a1ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:28.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:28.881+0000 7fe7ddffb640 10 monclient: tick 2024-09-16T14:48:28.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:28.881+0000 7fe7ddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:28.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:28.881+0000 7fe7ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:29.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:29.455+0000 7fce23fff640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6790/0 conn(0x7fce2406c5a0 legacy=0x7fce240a1ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:29.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:29.881+0000 7fe7ddffb640 10 monclient: tick 2024-09-16T14:48:29.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:29.881+0000 7fe7ddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:29.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:29.881+0000 7fe7ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:30.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:30.881+0000 7fe7ddffb640 10 monclient: tick 2024-09-16T14:48:30.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:30.881+0000 7fe7ddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:30.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:30.881+0000 7fe7ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.043+0000 7fce21ffb640 10 monclient(hunting): tick 2024-09-16T14:48:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.043+0000 7fce21ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:48:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.043+0000 7fce21ffb640 1 monclient(hunting): continuing hunt 2024-09-16T14:48:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.043+0000 7fce21ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:48:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.043+0000 7fce21ffb640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6790/0 conn(0x7fce2406c5a0 legacy=0x7fce240a1ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.043+0000 7fce21ffb640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.189:6789/0 conn(0x7fce240a27c0 legacy=0x7fce2413d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.043+0000 7fce21ffb640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6789/0 conn(0x7fce240a23b0 legacy=0x7fce24139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.043+0000 7fce21ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.043+0000 7fce21ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fce100036d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.043+0000 7fce21ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fce240a23b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.043+0000 7fce21ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fce2406c5a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.043+0000 7fce21ffb640 1 -- 172.21.15.26:0/3644725818 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2409edd0 con 0x7fce240a23b0 2024-09-16T14:48:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.043+0000 7fce21ffb640 1 -- 172.21.15.26:0/3644725818 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce240a15e0 con 0x7fce100036d0 2024-09-16T14:48:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.043+0000 7fce21ffb640 1 -- 172.21.15.26:0/3644725818 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce100103e0 con 0x7fce2406c5a0 2024-09-16T14:48:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.043+0000 7fce21ffb640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.043+0000 7fce28917640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6790/0 conn(0x7fce2406c5a0 legacy=0x7fce1000cc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:31.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.247+0000 7fce28917640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6790/0 conn(0x7fce2406c5a0 legacy=0x7fce1000cc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:31.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:31.651+0000 7fce28917640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6790/0 conn(0x7fce2406c5a0 legacy=0x7fce1000cc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:31.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:31.881+0000 7fe7ddffb640 10 monclient: tick 2024-09-16T14:48:31.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:31.881+0000 7fe7ddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:31.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:31.881+0000 7fe7ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:32.455+0000 7fce28917640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6790/0 conn(0x7fce2406c5a0 legacy=0x7fce1000cc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:32.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:32.881+0000 7fe7ddffb640 10 monclient: tick 2024-09-16T14:48:32.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:32.881+0000 7fe7ddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:32.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:32.881+0000 7fe7ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:33.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:33.885+0000 7fe7ddffb640 10 monclient: tick 2024-09-16T14:48:33.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:33.885+0000 7fe7ddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:33.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:33.885+0000 7fe7ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:33.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:33.885+0000 7fe7ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.042+0000 7fce21ffb640 10 monclient(hunting): tick 2024-09-16T14:48:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.042+0000 7fce21ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:48:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.042+0000 7fce21ffb640 1 monclient(hunting): continuing hunt 2024-09-16T14:48:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.042+0000 7fce21ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:48:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.042+0000 7fce21ffb640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6790/0 conn(0x7fce2406c5a0 legacy=0x7fce1000cc10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.042+0000 7fce21ffb640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.189:6789/0 conn(0x7fce100036d0 legacy=0x7fce10005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.042+0000 7fce21ffb640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6789/0 conn(0x7fce240a23b0 legacy=0x7fce100093e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.046+0000 7fce21ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.046+0000 7fce21ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fce100106f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.046+0000 7fce21ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fce10012870 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.046+0000 7fce21ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fce2406c5a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.046+0000 7fce21ffb640 1 -- 172.21.15.26:0/3644725818 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce100115f0 con 0x7fce10012870 2024-09-16T14:48:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.046+0000 7fce21ffb640 1 -- 172.21.15.26:0/3644725818 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce100103e0 con 0x7fce100106f0 2024-09-16T14:48:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.046+0000 7fce21ffb640 1 -- 172.21.15.26:0/3644725818 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce1000d3b0 con 0x7fce2406c5a0 2024-09-16T14:48:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.046+0000 7fce21ffb640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.046+0000 7fce28917640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6790/0 conn(0x7fce2406c5a0 legacy=0x7fce1000cc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:34.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.250+0000 7fce28917640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6790/0 conn(0x7fce2406c5a0 legacy=0x7fce1000cc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:34.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:34.654+0000 7fce28917640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6790/0 conn(0x7fce2406c5a0 legacy=0x7fce1000cc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:34.881+0000 7fe7ddffb640 10 monclient: tick 2024-09-16T14:48:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:34.881+0000 7fe7ddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:34.881+0000 7fe7ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:35.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.458+0000 7fce28917640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6790/0 conn(0x7fce2406c5a0 legacy=0x7fce1000cc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 1 -- 172.21.15.26:0/3644725818 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361308896 0 0) 0x7fce100115f0 con 0x7fce10012870 2024-09-16T14:48:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 10 monclient(hunting): my global_id is 124398 2024-09-16T14:48:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 1 -- 172.21.15.26:0/3644725818 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce0c003530 con 0x7fce10012870 2024-09-16T14:48:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 1 -- 172.21.15.26:0/3644725818 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2888776261 0 0) 0x7fce0c003530 con 0x7fce10012870 2024-09-16T14:48:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6790/0 conn(0x7fce2406c5a0 legacy=0x7fce1000cc10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.189:6789/0 conn(0x7fce100106f0 legacy=0x7fce10010b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 1 monclient: found mon.noname-a 2024-09-16T14:48:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 1 -- 172.21.15.26:0/3644725818 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1000d590 con 0x7fce10012870 2024-09-16T14:48:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 10 monclient: _finish_auth 0 2024-09-16T14:48:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 1 -- 172.21.15.26:0/3644725818 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fce18001520 con 0x7fce10012870 2024-09-16T14:48:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 10 monclient: dump: 2024-09-16T14:48:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:35.554 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:35.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce2a37a640 5 monclient: authenticate success, global_id 124398 2024-09-16T14:48:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce2a37a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 1 -- 172.21.15.26:0/3644725818 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce18001740 con 0x7fce10012870 2024-09-16T14:48:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 1 -- 172.21.15.26:0/3644725818 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fce18001b50 con 0x7fce10012870 2024-09-16T14:48:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce2a37a640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce2a37a640 4 set_mon_vals no callback set 2024-09-16T14:48:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.546+0000 7fce22ffd640 10 monclient: dump: 2024-09-16T14:48:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:35.556 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 10 monclient: shutdown 2024-09-16T14:48:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 1 -- 172.21.15.26:0/3644725818 >> v1:172.21.15.26:6789/0 conn(0x7fce10012870 legacy=0x7fce10014c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 1 -- 172.21.15.26:0/3644725818 shutdown_connections 2024-09-16T14:48:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 1 -- 172.21.15.26:0/3644725818 >> 172.21.15.26:0/3644725818 conn(0x7fce240a3da0 msgr2=0x7fce240a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 1 -- 172.21.15.26:0/3644725818 shutdown_connections 2024-09-16T14:48:35.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 1 -- 172.21.15.26:0/3644725818 wait complete. 2024-09-16T14:48:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 10 monclient: build_initial_monmap 2024-09-16T14:48:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 10 monclient: monmap: 2024-09-16T14:48:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:35.559 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 1 Processor -- start 2024-09-16T14:48:35.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 1 -- start start 2024-09-16T14:48:35.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 10 monclient: init 2024-09-16T14:48:35.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:35.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:35.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 10 monclient(hunting): picked mon.noname-b con 0x7fce100106f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:35.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 10 monclient(hunting): picked mon.noname-c con 0x7fce10012870 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:35.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 10 monclient(hunting): picked mon.noname-a con 0x7fce2406c5a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:35.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce24129930 con 0x7fce2406c5a0 2024-09-16T14:48:35.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2409cc70 con 0x7fce100106f0 2024-09-16T14:48:35.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2409e490 con 0x7fce10012870 2024-09-16T14:48:35.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:35.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce2a37a640 10 monclient(hunting): authenticate will time out at 1339.675150s 2024-09-16T14:48:35.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce237fe640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fce10012870 legacy=0x7fce241d2f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:35.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce28917640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fce2406c5a0 0x7fce241d6640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53932/0 (socket says 172.21.15.26:53932) 2024-09-16T14:48:35.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.550+0000 7fce28917640 1 -- 172.21.15.26:0/3673888171 learned_addr learned my addr 172.21.15.26:0/3673888171 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.558+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732113636 0 0) 0x7fce2409cc70 con 0x7fce100106f0 2024-09-16T14:48:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.558+0000 7fce20ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.558+0000 7fce20ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.558+0000 7fce20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.558+0000 7fce20ff9640 10 monclient(hunting): my global_id is 144196 2024-09-16T14:48:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.558+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce1000d3b0 con 0x7fce100106f0 2024-09-16T14:48:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3662277053 0 0) 0x7fce1000d3b0 con 0x7fce100106f0 2024-09-16T14:48:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce2409cc70 con 0x7fce100106f0 2024-09-16T14:48:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fce14004280 con 0x7fce100106f0 2024-09-16T14:48:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:35.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 10 monclient(hunting): dump: 2024-09-16T14:48:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:35.568 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993084045 0 0) 0x7fce24129930 con 0x7fce2406c5a0 2024-09-16T14:48:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 10 monclient(hunting): my global_id is 124401 2024-09-16T14:48:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce1000d3b0 con 0x7fce2406c5a0 2024-09-16T14:48:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2691731894 0 0) 0x7fce2409cc70 con 0x7fce100106f0 2024-09-16T14:48:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 >> v1:172.21.15.26:6790/0 conn(0x7fce10012870 legacy=0x7fce241d2f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 >> v1:172.21.15.26:6789/0 conn(0x7fce2406c5a0 legacy=0x7fce241d6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:48:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 1 monclient: found mon.b 2024-09-16T14:48:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:35.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce240a69c0 con 0x7fce100106f0 2024-09-16T14:48:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 10 monclient: _finish_auth 0 2024-09-16T14:48:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:48:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce2a37a640 5 monclient: authenticate success, global_id 144196 2024-09-16T14:48:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce2a37a640 10 monclient: _renew_subs 2024-09-16T14:48:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce2a37a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce2a37a640 1 -- 172.21.15.26:0/3673888171 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce24129c10 con 0x7fce100106f0 2024-09-16T14:48:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce2a37a640 10 monclient: _renew_subs 2024-09-16T14:48:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce2a37a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce2a37a640 1 -- 172.21.15.26:0/3673888171 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce24127840 con 0x7fce100106f0 2024-09-16T14:48:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce14004780 con 0x7fce100106f0 2024-09-16T14:48:35.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fce14004b10 con 0x7fce100106f0 2024-09-16T14:48:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 10 monclient: dump: 2024-09-16T14:48:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:35.571 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:35.571 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:35.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce21ffb640 4 set_mon_vals no callback set 2024-09-16T14:48:35.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce2a37a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:35.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce2a37a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:35.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce2a37a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:35.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce2a37a640 1 -- 172.21.15.26:0/3673888171 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce04005050 con 0x7fce100106f0 2024-09-16T14:48:35.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fce14004cf0 con 0x7fce100106f0 2024-09-16T14:48:35.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(155..155 src has 73..155) v4 ==== 8989+0+0 (unknown 1378740788 0 0) 0x7fce14044730 con 0x7fce100106f0 2024-09-16T14:48:35.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.562+0000 7fce237fe640 10 monclient: get_auth_request con 0x7fce10037510 auth_method 0 2024-09-16T14:48:35.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.578+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fce140147a0 con 0x7fce100106f0 2024-09-16T14:48:35.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.578+0000 7fce20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:35.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.578+0000 7fce20ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:35.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:35.617+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v156) v1 ==== 123+0+0 (unknown 1677448319 0 0) 0x7fe7e0012d30 con 0x7fe7f01315a0 2024-09-16T14:48:35.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:35.617+0000 7fe7deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:35.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:35.617+0000 7fe7deffd640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-16T14:48:35.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:35.721+0000 7fe7f487b640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:35.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:35.721+0000 7fe7f487b640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:35.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:35.721+0000 7fe7f487b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:35.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:35.721+0000 7fe7f487b640 1 -- 172.21.15.189:0/3763706886 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7fe7d4005300 con 0x7fe7f01315a0 2024-09-16T14:48:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.810+0000 7fce2a37a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:48:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.810+0000 7fce2a37a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:48:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.810+0000 7fce2a37a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.810+0000 7fce2a37a640 1 -- 172.21.15.26:0/3673888171 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fce04005300 con 0x7fce100106f0 2024-09-16T14:48:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.810+0000 7fce20ff9640 1 -- 172.21.15.26:0/3673888171 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2860859826) 0x7fce14012c10 con 0x7fce100106f0 2024-09-16T14:48:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.810+0000 7fce20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:48:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.810+0000 7fce20ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:35.815 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":56,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:48:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.810+0000 7fce2a37a640 1 -- 172.21.15.26:0/3673888171 >> v1:172.21.15.189:6812/3196161582 conn(0x7fce10037510 legacy=0x7fce100399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.810+0000 7fce2a37a640 10 monclient: shutdown 2024-09-16T14:48:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.810+0000 7fce2a37a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.810+0000 7fce2a37a640 1 -- 172.21.15.26:0/3673888171 >> v1:172.21.15.189:6789/0 conn(0x7fce100106f0 legacy=0x7fce241ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.810+0000 7fce2a37a640 1 -- 172.21.15.26:0/3673888171 shutdown_connections 2024-09-16T14:48:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.814+0000 7fce2a37a640 1 -- 172.21.15.26:0/3673888171 >> 172.21.15.26:0/3673888171 conn(0x7fce240a3da0 msgr2=0x7fce240a50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.814+0000 7fce2a37a640 1 -- 172.21.15.26:0/3673888171 shutdown_connections 2024-09-16T14:48:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.814+0000 7fce2a37a640 1 -- 172.21.15.26:0/3673888171 wait complete. 2024-09-16T14:48:35.834 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:48:35.834 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T14:48:35.835 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:48:35.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:35.881+0000 7fe7ddffb640 10 monclient: tick 2024-09-16T14:48:35.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:35.881+0000 7fe7ddffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:35.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:35.881+0000 7fe7ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:35.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 10 monclient: get_monmap_and_config 2024-09-16T14:48:35.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 10 monclient: build_initial_monmap 2024-09-16T14:48:35.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 10 monclient: monmap: 2024-09-16T14:48:35.953 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:35.953 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:35.953 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:35.953638+0000 2024-09-16T14:48:35.953 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:35.953638+0000 2024-09-16T14:48:35.953 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:35.954 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:35.954 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:35.954 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:35.954 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:35.954 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:35.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 1 Processor -- start 2024-09-16T14:48:35.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 1 -- start start 2024-09-16T14:48:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 10 monclient: init 2024-09-16T14:48:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 10 monclient(hunting): picked mon.noname-c con 0x7f51180a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 10 monclient(hunting): picked mon.noname-a con 0x7f51180a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 10 monclient(hunting): picked mon.noname-b con 0x7f51180a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51180a3250 con 0x7f51180a0000 2024-09-16T14:48:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511809f600 con 0x7f51180a1d10 2024-09-16T14:48:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51180a3830 con 0x7f51180a2800 2024-09-16T14:48:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 10 monclient(hunting): authenticate will time out at 1340.073703s 2024-09-16T14:48:35.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511d7fa640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f51180a2800 legacy=0x7f511813d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:35.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f5117fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f51180a0000 0x7f5118139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53946/0 (socket says 172.21.15.26:53946) 2024-09-16T14:48:35.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f5117fff640 1 -- 172.21.15.26:0/3944176279 learned_addr learned my addr 172.21.15.26:0/3944176279 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:35.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 1 -- 172.21.15.26:0/3944176279 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373445872 0 0) 0x7f51180a3250 con 0x7f51180a0000 2024-09-16T14:48:35.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:35.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:35.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:35.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 10 monclient(hunting): my global_id is 124404 2024-09-16T14:48:35.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 1 -- 172.21.15.26:0/3944176279 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5100003530 con 0x7f51180a0000 2024-09-16T14:48:35.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 1 -- 172.21.15.26:0/3944176279 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1957842727 0 0) 0x7f5100003530 con 0x7f51180a0000 2024-09-16T14:48:35.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 1 -- 172.21.15.26:0/3944176279 >> v1:172.21.15.26:6790/0 conn(0x7f51180a2800 legacy=0x7f511813d440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:35.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 1 -- 172.21.15.26:0/3944176279 >> v1:172.21.15.189:6789/0 conn(0x7f51180a1d10 legacy=0x7f51180a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:35.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 10 monclient: _finish_hunting 0 2024-09-16T14:48:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 1 monclient: found mon.noname-a 2024-09-16T14:48:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 1 -- 172.21.15.26:0/3944176279 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511809ed60 con 0x7f51180a0000 2024-09-16T14:48:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 10 monclient: _finish_auth 0 2024-09-16T14:48:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 10 monclient: _check_auth_tickets 2024-09-16T14:48:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 1 -- 172.21.15.26:0/3944176279 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f510c001620 con 0x7f51180a0000 2024-09-16T14:48:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 10 monclient: dump: 2024-09-16T14:48:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:35.960 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 5 monclient: authenticate success, global_id 124404 2024-09-16T14:48:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 1 -- 172.21.15.26:0/3944176279 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f510c001940 con 0x7f51180a0000 2024-09-16T14:48:35.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 1 -- 172.21.15.26:0/3944176279 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f510c001d10 con 0x7f51180a0000 2024-09-16T14:48:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f51177fe640 10 monclient: dump: 2024-09-16T14:48:35.961 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:35.962 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 4 set_mon_vals no callback set 2024-09-16T14:48:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 10 monclient: shutdown 2024-09-16T14:48:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:35.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 1 -- 172.21.15.26:0/3944176279 >> v1:172.21.15.26:6789/0 conn(0x7f51180a0000 legacy=0x7f5118139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:35.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 1 -- 172.21.15.26:0/3944176279 shutdown_connections 2024-09-16T14:48:35.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 1 -- 172.21.15.26:0/3944176279 >> 172.21.15.26:0/3944176279 conn(0x7f51180a7f30 msgr2=0x7f51180a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:35.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 1 -- 172.21.15.26:0/3944176279 shutdown_connections 2024-09-16T14:48:35.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 1 -- 172.21.15.26:0/3944176279 wait complete. 2024-09-16T14:48:35.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 10 monclient: build_initial_monmap 2024-09-16T14:48:35.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.950+0000 7f511f25d640 10 monclient: monmap: 2024-09-16T14:48:35.963 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:35.963 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:35.963 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:35.963 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:35.963 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:35.963 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:35.963 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:35.964 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:35.964 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:35.964 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:35.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 1 Processor -- start 2024-09-16T14:48:35.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 1 -- start start 2024-09-16T14:48:35.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 10 monclient: init 2024-09-16T14:48:35.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 10 monclient(hunting): picked mon.noname-c con 0x7f51180a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 10 monclient(hunting): picked mon.noname-a con 0x7f51180a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 10 monclient(hunting): picked mon.noname-b con 0x7f51180a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5118133d70 con 0x7f51180a1d10 2024-09-16T14:48:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5118134050 con 0x7f51180a2800 2024-09-16T14:48:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511cff9640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f51180a0000 legacy=0x7f5118130150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5118130f70 con 0x7f51180a0000 2024-09-16T14:48:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 10 monclient(hunting): authenticate will time out at 1340.076764s 2024-09-16T14:48:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f5117fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f51180a1d10 0x7f5118130860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53960/0 (socket says 172.21.15.26:53960) 2024-09-16T14:48:35.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f5117fff640 1 -- 172.21.15.26:0/1714905313 learned_addr learned my addr 172.21.15.26:0/1714905313 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951011651 0 0) 0x7f5118133d70 con 0x7f51180a1d10 2024-09-16T14:48:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient(hunting): my global_id is 124407 2024-09-16T14:48:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5104003530 con 0x7f51180a1d10 2024-09-16T14:48:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1327324338 0 0) 0x7f5104003530 con 0x7f51180a1d10 2024-09-16T14:48:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5118133d70 con 0x7f51180a1d10 2024-09-16T14:48:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f510c001620 con 0x7f51180a1d10 2024-09-16T14:48:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient(hunting): dump: 2024-09-16T14:48:35.970 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:35.971 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972349186 0 0) 0x7f5118134050 con 0x7f51180a2800 2024-09-16T14:48:35.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient(hunting): my global_id is 144202 2024-09-16T14:48:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5104003530 con 0x7f51180a2800 2024-09-16T14:48:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 859154731 0 0) 0x7f5118133d70 con 0x7f51180a1d10 2024-09-16T14:48:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 >> v1:172.21.15.26:6790/0 conn(0x7f51180a0000 legacy=0x7f5118130150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 >> v1:172.21.15.189:6789/0 conn(0x7f51180a2800 legacy=0x7f51181d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient: _finish_hunting 0 2024-09-16T14:48:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 monclient: found mon.a 2024-09-16T14:48:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:35.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5118132200 con 0x7f51180a1d10 2024-09-16T14:48:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient: _finish_auth 0 2024-09-16T14:48:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient: _check_auth_tickets 2024-09-16T14:48:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 5 monclient: authenticate success, global_id 124407 2024-09-16T14:48:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 10 monclient: _renew_subs 2024-09-16T14:48:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 1 -- 172.21.15.26:0/1714905313 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51180a65f0 con 0x7f51180a1d10 2024-09-16T14:48:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 10 monclient: _renew_subs 2024-09-16T14:48:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 1 -- 172.21.15.26:0/1714905313 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5118135840 con 0x7f51180a1d10 2024-09-16T14:48:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f510c004c30 con 0x7f51180a1d10 2024-09-16T14:48:35.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f510c004fc0 con 0x7f51180a1d10 2024-09-16T14:48:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient: dump: 2024-09-16T14:48:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:35.974 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:35.975 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51167fc640 4 set_mon_vals no callback set 2024-09-16T14:48:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f510c0051a0 con 0x7f51180a1d10 2024-09-16T14:48:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511f25d640 1 -- 172.21.15.26:0/1714905313 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f50f8006590 con 0x7f50f8002830 2024-09-16T14:48:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(156..156 src has 73..156) v4 ==== 8324+0+0 (unknown 3018032569 0 0) 0x7f510c044760 con 0x7f51180a1d10 2024-09-16T14:48:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936645037 0 0) 0x7f50f8006590 con 0x7f50f8002830 2024-09-16T14:48:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:35.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient(hunting): my global_id is 124410 2024-09-16T14:48:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5118133d70 con 0x7f50f8002830 2024-09-16T14:48:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f511d7fa640 10 monclient: get_auth_request con 0x7f5104036100 auth_method 0 2024-09-16T14:48:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 319802065 0 0) 0x7f5118133d70 con 0x7f50f8002830 2024-09-16T14:48:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50f8006590 con 0x7f50f8002830 2024-09-16T14:48:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5108001740 con 0x7f50f8002830 2024-09-16T14:48:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient: dump: 2024-09-16T14:48:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.976 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:35.977 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1228134341 0 0) 0x7f50f8006590 con 0x7f50f8002830 2024-09-16T14:48:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50f800a0e0 con 0x7f50f8002830 2024-09-16T14:48:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5108001ca0 con 0x7f50f8002830 2024-09-16T14:48:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 >> v1:172.21.15.26:6789/0 conn(0x7f50f8002830 legacy=0x7f50f8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:35.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.954+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 >> v1:172.21.15.26:6789/0 conn(0x7f50f8002830 legacy=0x7f50f8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f511f25d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:48:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f511f25d640 1 -- 172.21.15.26:0/1714905313 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f50f8009da0 con 0x7f50f8009060 2024-09-16T14:48:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2217514695 0 0) 0x7f50f8009da0 con 0x7f50f8009060 2024-09-16T14:48:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 10 monclient(hunting): my global_id is 124413 2024-09-16T14:48:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50f8006590 con 0x7f50f8009060 2024-09-16T14:48:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3991697428 0 0) 0x7f50f8006590 con 0x7f50f8009060 2024-09-16T14:48:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50f8009da0 con 0x7f50f8009060 2024-09-16T14:48:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5108003870 con 0x7f50f8009060 2024-09-16T14:48:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:35.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 10 monclient: dump: 2024-09-16T14:48:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:35.979 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 675381158 0 0) 0x7f50f8009da0 con 0x7f50f8009060 2024-09-16T14:48:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f50f800c290 con 0x7f50f8009060 2024-09-16T14:48:35.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4034 (unknown 0 0 1040332665) 0x7f51080057d0 con 0x7f50f8009060 2024-09-16T14:48:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:48:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 >> v1:172.21.15.26:6789/0 conn(0x7f50f8009060 legacy=0x7f50f80094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:35.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f51157fa640 1 -- 172.21.15.26:0/1714905313 >> v1:172.21.15.26:6789/0 conn(0x7f50f8009060 legacy=0x7f50f80094f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:35.980 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:48:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:48:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 56, 2024-09-16T14:48:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:48:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:48:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: 1 2024-09-16T14:48:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:48:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:48:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:48:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:48:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:48:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:48:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:48:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:48:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:48:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:48:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:48:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:48:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:48:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:48:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:48:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:48:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:48:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:48:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:48:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:48:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:48:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:48:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:48:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:48:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:48:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:48:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:48:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:48:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:48:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:48:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:48:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:48:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:48:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:48:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:48:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:48:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:48:35.992 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:48:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f511f25d640 1 -- 172.21.15.26:0/1714905313 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5104036100 legacy=0x7f51040385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f511f25d640 10 monclient: shutdown 2024-09-16T14:48:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f511f25d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f511f25d640 1 -- 172.21.15.26:0/1714905313 >> v1:172.21.15.26:6789/0 conn(0x7f51180a1d10 legacy=0x7f5118130860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f511d7fa640 1 -- 172.21.15.26:0/1714905313 reap_dead start 2024-09-16T14:48:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f511d7fa640 1 -- 172.21.15.26:0/1714905313 reap_dead start 2024-09-16T14:48:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f511f25d640 1 -- 172.21.15.26:0/1714905313 shutdown_connections 2024-09-16T14:48:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f511f25d640 1 -- 172.21.15.26:0/1714905313 >> 172.21.15.26:0/1714905313 conn(0x7f51180a7f30 msgr2=0x7f511806b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f511f25d640 1 -- 172.21.15.26:0/1714905313 shutdown_connections 2024-09-16T14:48:35.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:35.962+0000 7f511f25d640 1 -- 172.21.15.26:0/1714905313 wait complete. 2024-09-16T14:48:35.995 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:48:35.995 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T14:48:35.995 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:48:36.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.106+0000 7f8e5dbd7640 10 monclient: get_monmap_and_config 2024-09-16T14:48:36.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.106+0000 7f8e5dbd7640 10 monclient: build_initial_monmap 2024-09-16T14:48:36.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.106+0000 7f8e5dbd7640 10 monclient: monmap: 2024-09-16T14:48:36.112 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:36.112 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:36.112 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:36.112285+0000 2024-09-16T14:48:36.112 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:36.112285+0000 2024-09-16T14:48:36.112 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:36.112 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:36.112 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:36.112 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:36.112 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:36.112 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:36.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.106+0000 7f8e5dbd7640 1 Processor -- start 2024-09-16T14:48:36.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 1 -- start start 2024-09-16T14:48:36.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient: init 2024-09-16T14:48:36.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:36.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:36.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient(hunting): picked mon.noname-b con 0x7f8e580a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:36.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient(hunting): picked mon.noname-c con 0x7f8e580a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:36.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient(hunting): picked mon.noname-a con 0x7f8e580a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:36.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e580a3250 con 0x7f8e580a2800 2024-09-16T14:48:36.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5809f600 con 0x7f8e580a0000 2024-09-16T14:48:36.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e580a3830 con 0x7f8e580a1d10 2024-09-16T14:48:36.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:36.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient(hunting): authenticate will time out at 1340.232695s 2024-09-16T14:48:36.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e577fe640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f8e580a1d10 legacy=0x7f8e580a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:36.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e57fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8e580a2800 0x7f8e5813d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53974/0 (socket says 172.21.15.26:53974) 2024-09-16T14:48:36.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e57fff640 1 -- 172.21.15.26:0/4179267005 learned_addr learned my addr 172.21.15.26:0/4179267005 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:36.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 1 -- 172.21.15.26:0/4179267005 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178377396 0 0) 0x7f8e580a3250 con 0x7f8e580a2800 2024-09-16T14:48:36.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:36.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:36.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:36.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 10 monclient(hunting): my global_id is 124416 2024-09-16T14:48:36.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 1 -- 172.21.15.26:0/4179267005 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e40003530 con 0x7f8e580a2800 2024-09-16T14:48:36.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 1 -- 172.21.15.26:0/4179267005 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2584155054 0 0) 0x7f8e40003530 con 0x7f8e580a2800 2024-09-16T14:48:36.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 1 -- 172.21.15.26:0/4179267005 >> v1:172.21.15.26:6790/0 conn(0x7f8e580a1d10 legacy=0x7f8e580a20f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:36.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 1 -- 172.21.15.26:0/4179267005 >> v1:172.21.15.189:6789/0 conn(0x7f8e580a0000 legacy=0x7f8e58139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:36.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:36.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 1 monclient: found mon.noname-a 2024-09-16T14:48:36.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:36.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:36.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 1 -- 172.21.15.26:0/4179267005 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5809ed60 con 0x7f8e580a2800 2024-09-16T14:48:36.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:36.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:36.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:36.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 1 -- 172.21.15.26:0/4179267005 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8e500019a0 con 0x7f8e580a2800 2024-09-16T14:48:36.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 5 monclient: authenticate success, global_id 124416 2024-09-16T14:48:36.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:36.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:36.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:36.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 10 monclient: dump: 2024-09-16T14:48:36.118 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:36.118 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:36.119 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.119 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.119 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:36.119 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:36.119 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:36.119 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:36.119 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:36.119 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:36.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:36.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 1 -- 172.21.15.26:0/4179267005 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e50001cc0 con 0x7f8e580a2800 2024-09-16T14:48:36.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:36.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:36.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 4 set_mon_vals no callback set 2024-09-16T14:48:36.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 1 -- 172.21.15.26:0/4179267005 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8e50002050 con 0x7f8e580a2800 2024-09-16T14:48:36.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:36.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:36.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e567fc640 10 monclient: dump: 2024-09-16T14:48:36.120 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:36.120 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:36.120 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.120 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.120 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:36.120 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:36.120 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:36.120 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:36.120 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:36.120 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:36.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient: shutdown 2024-09-16T14:48:36.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:36.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 1 -- 172.21.15.26:0/4179267005 >> v1:172.21.15.26:6789/0 conn(0x7f8e580a2800 legacy=0x7f8e5813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:36.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 1 -- 172.21.15.26:0/4179267005 shutdown_connections 2024-09-16T14:48:36.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 1 -- 172.21.15.26:0/4179267005 >> 172.21.15.26:0/4179267005 conn(0x7f8e580a7f30 msgr2=0x7f8e580a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:36.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 1 -- 172.21.15.26:0/4179267005 shutdown_connections 2024-09-16T14:48:36.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 1 -- 172.21.15.26:0/4179267005 wait complete. 2024-09-16T14:48:36.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient: build_initial_monmap 2024-09-16T14:48:36.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient: monmap: 2024-09-16T14:48:36.121 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:36.121 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:36.121 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:36.121 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:36.122 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:36.122 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:36.122 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:36.122 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:36.122 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:36.122 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:36.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 1 Processor -- start 2024-09-16T14:48:36.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 1 -- start start 2024-09-16T14:48:36.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient: init 2024-09-16T14:48:36.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:36.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:36.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient(hunting): picked mon.noname-b con 0x7f8e580a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:36.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient(hunting): picked mon.noname-a con 0x7f8e580a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:36.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient(hunting): picked mon.noname-c con 0x7f8e580a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:36.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5809e500 con 0x7f8e580a1d10 2024-09-16T14:48:36.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e58135d90 con 0x7f8e580a0000 2024-09-16T14:48:36.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5813cc80 con 0x7f8e580a2800 2024-09-16T14:48:36.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:36.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e5dbd7640 10 monclient(hunting): authenticate will time out at 1340.235699s 2024-09-16T14:48:36.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e57fff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f8e580a2800 legacy=0x7f8e581d19b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:36.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e56ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8e580a1d10 0x7f8e58132720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53988/0 (socket says 172.21.15.26:53988) 2024-09-16T14:48:36.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e56ffd640 1 -- 172.21.15.26:0/3308065419 learned_addr learned my addr 172.21.15.26:0/3308065419 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:36.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.110+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822734120 0 0) 0x7f8e5809e500 con 0x7f8e580a1d10 2024-09-16T14:48:36.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:36.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:36.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:36.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient(hunting): my global_id is 124419 2024-09-16T14:48:36.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e44003530 con 0x7f8e580a1d10 2024-09-16T14:48:36.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2699822582 0 0) 0x7f8e44003530 con 0x7f8e580a1d10 2024-09-16T14:48:36.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e5809e500 con 0x7f8e580a1d10 2024-09-16T14:48:36.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8e4c001940 con 0x7f8e580a1d10 2024-09-16T14:48:36.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:36.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:36.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient(hunting): dump: 2024-09-16T14:48:36.128 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:36.128 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:36.128 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.128 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.128 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:36.128 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:36.129 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:36.129 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:36.129 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:36.129 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:36.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:36.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637944082 0 0) 0x7f8e58135d90 con 0x7f8e580a0000 2024-09-16T14:48:36.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:36.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:36.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:36.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient(hunting): my global_id is 144208 2024-09-16T14:48:36.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e44003530 con 0x7f8e580a0000 2024-09-16T14:48:36.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3574524476 0 0) 0x7f8e5809e500 con 0x7f8e580a1d10 2024-09-16T14:48:36.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 >> v1:172.21.15.26:6790/0 conn(0x7f8e580a2800 legacy=0x7f8e581d19b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:36.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 >> v1:172.21.15.189:6789/0 conn(0x7f8e580a0000 legacy=0x7f8e58132010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:36.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:36.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 monclient: found mon.a 2024-09-16T14:48:36.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:36.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:36.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e581355a0 con 0x7f8e580a1d10 2024-09-16T14:48:36.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient: _finish_auth 0 2024-09-16T14:48:36.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:36.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:36.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e5dbd7640 5 monclient: authenticate success, global_id 124419 2024-09-16T14:48:36.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e5dbd7640 10 monclient: _renew_subs 2024-09-16T14:48:36.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e5dbd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:36.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e5dbd7640 1 -- 172.21.15.26:0/3308065419 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e58136050 con 0x7f8e580a1d10 2024-09-16T14:48:36.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e5dbd7640 10 monclient: _renew_subs 2024-09-16T14:48:36.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e5dbd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:36.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e5dbd7640 1 -- 172.21.15.26:0/3308065419 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e58135940 con 0x7f8e580a1d10 2024-09-16T14:48:36.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e4c0036a0 con 0x7f8e580a1d10 2024-09-16T14:48:36.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:36.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8e4c003890 con 0x7f8e580a1d10 2024-09-16T14:48:36.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:36.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:36.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e55ffb640 4 set_mon_vals no callback set 2024-09-16T14:48:36.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient: dump: 2024-09-16T14:48:36.131 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:36.131 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:36.131 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.131 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.131 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:36.132 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e5dbd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e5dbd7640 1 -- 172.21.15.26:0/3308065419 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8e38006590 con 0x7f8e38002830 2024-09-16T14:48:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8e4c003a70 con 0x7f8e580a1d10 2024-09-16T14:48:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(156..156 src has 73..156) v4 ==== 8324+0+0 (unknown 3018032569 0 0) 0x7f8e4c044360 con 0x7f8e580a1d10 2024-09-16T14:48:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e57fff640 10 monclient: get_auth_request con 0x7f8e44036130 auth_method 0 2024-09-16T14:48:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058825983 0 0) 0x7f8e38006590 con 0x7f8e38002830 2024-09-16T14:48:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:36.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:36.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:36.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient(hunting): my global_id is 144211 2024-09-16T14:48:36.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e5809e500 con 0x7f8e38002830 2024-09-16T14:48:36.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 350003628 0 0) 0x7f8e5809e500 con 0x7f8e38002830 2024-09-16T14:48:36.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e38006590 con 0x7f8e38002830 2024-09-16T14:48:36.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8e480040b0 con 0x7f8e38002830 2024-09-16T14:48:36.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:36.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:36.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient: dump: 2024-09-16T14:48:36.137 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:36.137 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:36.137 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.137 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.138 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:36.138 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:36.138 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:36.138 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:36.138 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:36.138 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:36.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 92867346 0 0) 0x7f8e38006590 con 0x7f8e38002830 2024-09-16T14:48:36.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e3800a0e0 con 0x7f8e38002830 2024-09-16T14:48:36.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8e48006480 con 0x7f8e38002830 2024-09-16T14:48:36.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:36.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:36.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 >> v1:172.21.15.189:6789/0 conn(0x7f8e38002830 legacy=0x7f8e38004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:36.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.114+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 >> v1:172.21.15.189:6789/0 conn(0x7f8e38002830 legacy=0x7f8e38004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:36.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.122+0000 7f8e5dbd7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:48:36.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.122+0000 7f8e5dbd7640 1 -- 172.21.15.26:0/3308065419 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8e38009d80 con 0x7f8e380090a0 2024-09-16T14:48:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.122+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879619940 0 0) 0x7f8e38009d80 con 0x7f8e380090a0 2024-09-16T14:48:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.122+0000 7f8e3ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.122+0000 7f8e3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.122+0000 7f8e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.122+0000 7f8e3ffff640 10 monclient(hunting): my global_id is 144214 2024-09-16T14:48:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.122+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e38006590 con 0x7f8e380090a0 2024-09-16T14:48:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1380595950 0 0) 0x7f8e38006590 con 0x7f8e380090a0 2024-09-16T14:48:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e38009d80 con 0x7f8e380090a0 2024-09-16T14:48:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8e480048f0 con 0x7f8e380090a0 2024-09-16T14:48:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e3ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e3ffff640 10 monclient: dump: 2024-09-16T14:48:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:36.141 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 203872494 0 0) 0x7f8e38009d80 con 0x7f8e380090a0 2024-09-16T14:48:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8e3800c290 con 0x7f8e380090a0 2024-09-16T14:48:36.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4036 (unknown 0 0 1546395678) 0x7f8e480066b0 con 0x7f8e380090a0 2024-09-16T14:48:36.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e3ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:48:36.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e3ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:36.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 >> v1:172.21.15.189:6789/0 conn(0x7f8e380090a0 legacy=0x7f8e38009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:36.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e3ffff640 1 -- 172.21.15.26:0/3308065419 >> v1:172.21.15.189:6789/0 conn(0x7f8e380090a0 legacy=0x7f8e38009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:36.142 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:48:36.142 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:36.142 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:36.142 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:48:36.142 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 56, 2024-09-16T14:48:36.143 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:48:36.143 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:48:36.143 INFO:teuthology.orchestra.run.smithi026.stdout: 1 2024-09-16T14:48:36.143 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:36.143 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:48:36.143 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:36.143 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:48:36.143 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:48:36.143 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:36.143 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:36.143 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:36.143 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:36.143 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:36.144 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:36.144 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:36.144 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:36.144 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:36.144 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:36.144 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:36.144 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:48:36.144 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:48:36.144 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:36.144 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:36.144 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:36.144 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:36.144 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:36.145 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:36.145 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:36.145 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:36.145 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:36.145 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:36.145 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:36.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:36.145 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:48:36.145 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:48:36.145 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:48:36.145 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:48:36.145 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:48:36.145 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:48:36.145 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:48:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:48:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:48:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:48:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:48:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:48:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:48:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:48:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:36.146 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:48:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:48:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:36.147 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:48:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:48:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:48:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:36.148 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:48:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:36.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:48:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:48:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:48:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:36.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:48:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:48:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:48:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:48:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:36.151 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:48:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:48:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:48:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:36.152 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:36.153 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:48:36.153 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:48:36.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e5dbd7640 1 -- 172.21.15.26:0/3308065419 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8e44036130 legacy=0x7f8e440385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:36.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e5dbd7640 10 monclient: shutdown 2024-09-16T14:48:36.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e5dbd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:36.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e5dbd7640 1 -- 172.21.15.26:0/3308065419 >> v1:172.21.15.26:6789/0 conn(0x7f8e580a1d10 legacy=0x7f8e58132720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:36.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e57fff640 1 -- 172.21.15.26:0/3308065419 reap_dead start 2024-09-16T14:48:36.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e57fff640 1 -- 172.21.15.26:0/3308065419 reap_dead start 2024-09-16T14:48:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e5dbd7640 1 -- 172.21.15.26:0/3308065419 shutdown_connections 2024-09-16T14:48:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e5dbd7640 1 -- 172.21.15.26:0/3308065419 >> 172.21.15.26:0/3308065419 conn(0x7f8e580a7f30 msgr2=0x7f8e5806b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e5dbd7640 1 -- 172.21.15.26:0/3308065419 shutdown_connections 2024-09-16T14:48:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:36.126+0000 7f8e5dbd7640 1 -- 172.21.15.26:0/3308065419 wait complete. 2024-09-16T14:48:36.155 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:48:36.155 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:48:36.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.629+0000 7fe7deffd640 1 -- 172.21.15.189:0/3763706886 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v157) v1 ==== 149+0+0 (unknown 2977586863 0 0) 0x7fe7e00144b0 con 0x7fe7f01315a0 2024-09-16T14:48:36.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.629+0000 7fe7deffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:36.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.629+0000 7fe7deffd640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T14:48:36.630 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T14:48:36.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.629+0000 7fe7f487b640 1 -- 172.21.15.189:0/3763706886 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe7d0036220 legacy=0x7fe7d00386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:36.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.629+0000 7fe7f487b640 10 monclient: shutdown 2024-09-16T14:48:36.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.629+0000 7fe7f487b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:36.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.629+0000 7fe7f487b640 1 -- 172.21.15.189:0/3763706886 >> v1:172.21.15.189:6789/0 conn(0x7fe7f01315a0 legacy=0x7fe7f0130b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:36.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.629+0000 7fe7f487b640 1 -- 172.21.15.189:0/3763706886 shutdown_connections 2024-09-16T14:48:36.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.629+0000 7fe7f487b640 1 -- 172.21.15.189:0/3763706886 >> 172.21.15.189:0/3763706886 conn(0x7fe7f00a3320 msgr2=0x7fe7f0070640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:36.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.629+0000 7fe7f487b640 1 -- 172.21.15.189:0/3763706886 shutdown_connections 2024-09-16T14:48:36.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.629+0000 7fe7f487b640 1 -- 172.21.15.189:0/3763706886 wait complete. 2024-09-16T14:48:36.649 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush set-device-class globster osd.0 2024-09-16T14:48:36.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 10 monclient: get_monmap_and_config 2024-09-16T14:48:36.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 10 monclient: build_initial_monmap 2024-09-16T14:48:36.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 10 monclient: monmap: 2024-09-16T14:48:36.771 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:36.771 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:36.771 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:36.771167+0000 2024-09-16T14:48:36.771 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:36.771167+0000 2024-09-16T14:48:36.771 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:36.772 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:36.772 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:36.772 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:36.772 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:36.772 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:36.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 1 Processor -- start 2024-09-16T14:48:36.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 1 -- start start 2024-09-16T14:48:36.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 10 monclient: init 2024-09-16T14:48:36.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:36.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:36.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 10 monclient(hunting): picked mon.noname-b con 0x7fa35009df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:36.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 10 monclient(hunting): picked mon.noname-c con 0x7fa35009e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:36.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 10 monclient(hunting): picked mon.noname-a con 0x7fa35009e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:36.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35009ee20 con 0x7fa35009e740 2024-09-16T14:48:36.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35009f100 con 0x7fa35009e2f0 2024-09-16T14:48:36.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35009f3e0 con 0x7fa35009df10 2024-09-16T14:48:36.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:36.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 10 monclient(hunting): authenticate will time out at 1298.886780s 2024-09-16T14:48:36.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355c51640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa35009e2f0 0x7fa3501378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51622/0 (socket says 172.21.15.189:51622) 2024-09-16T14:48:36.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355c51640 1 -- 172.21.15.189:0/3710096886 learned_addr learned my addr 172.21.15.189:0/3710096886 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:36.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa356452640 1 -- 172.21.15.189:0/3710096886 >> v1:172.21.15.26:6790/0 conn(0x7fa35009df10 legacy=0x7fa35009d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:36.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 1 -- 172.21.15.189:0/3710096886 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305643176 0 0) 0x7fa35009f100 con 0x7fa35009e2f0 2024-09-16T14:48:36.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 10 monclient(hunting): _negotiate 2024-09-16T14:48:36.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:36.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:36.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 10 monclient(hunting): my global_id is 144217 2024-09-16T14:48:36.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 1 -- 172.21.15.189:0/3710096886 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa33c003530 con 0x7fa35009e2f0 2024-09-16T14:48:36.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 1 -- 172.21.15.189:0/3710096886 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3861220514 0 0) 0x7fa33c003530 con 0x7fa35009e2f0 2024-09-16T14:48:36.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 1 -- 172.21.15.189:0/3710096886 >> v1:172.21.15.26:6790/0 conn(0x7fa35009df10 legacy=0x7fa35009d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:36.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 1 -- 172.21.15.189:0/3710096886 >> v1:172.21.15.26:6789/0 conn(0x7fa35009e740 legacy=0x7fa35013b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:36.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 10 monclient: _finish_hunting 0 2024-09-16T14:48:36.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 1 monclient: found mon.noname-c 2024-09-16T14:48:36.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:36.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:36.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 1 -- 172.21.15.189:0/3710096886 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35009c690 con 0x7fa35009e2f0 2024-09-16T14:48:36.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 10 monclient: _finish_auth 0 2024-09-16T14:48:36.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 10 monclient: _check_auth_tickets 2024-09-16T14:48:36.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:36.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 5 monclient: authenticate success, global_id 144217 2024-09-16T14:48:36.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 1 -- 172.21.15.189:0/3710096886 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa340001620 con 0x7fa35009e2f0 2024-09-16T14:48:36.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:36.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:36.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:36.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 10 monclient: dump: 2024-09-16T14:48:36.782 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:36.782 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:36.783 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.783 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.783 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:36.783 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:36.783 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:36.783 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:36.783 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:36.783 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:36.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:36.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 1 -- 172.21.15.189:0/3710096886 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa340001940 con 0x7fa35009e2f0 2024-09-16T14:48:36.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:36.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 1 -- 172.21.15.189:0/3710096886 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa340001cd0 con 0x7fa35009e2f0 2024-09-16T14:48:36.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:36.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa3586b6640 4 set_mon_vals no callback set 2024-09-16T14:48:36.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:36.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:36.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.769+0000 7fa355450640 10 monclient: dump: 2024-09-16T14:48:36.784 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:36.784 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:36.784 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.784 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.784 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:36.784 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:36.784 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:36.784 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:36.784 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:36.784 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:36.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient: shutdown 2024-09-16T14:48:36.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:36.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 1 -- 172.21.15.189:0/3710096886 >> v1:172.21.15.189:6789/0 conn(0x7fa35009e2f0 legacy=0x7fa3501378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:36.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 1 -- 172.21.15.189:0/3710096886 shutdown_connections 2024-09-16T14:48:36.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 1 -- 172.21.15.189:0/3710096886 >> 172.21.15.189:0/3710096886 conn(0x7fa3500a3320 msgr2=0x7fa35009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:36.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 1 -- 172.21.15.189:0/3710096886 shutdown_connections 2024-09-16T14:48:36.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 1 -- 172.21.15.189:0/3710096886 wait complete. 2024-09-16T14:48:36.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient: build_initial_monmap 2024-09-16T14:48:36.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient: monmap: 2024-09-16T14:48:36.785 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:36.785 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:36.785 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:36.785 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:36.786 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:36.786 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:36.786 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:36.786 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:36.786 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:36.786 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:36.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 1 Processor -- start 2024-09-16T14:48:36.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 1 -- start start 2024-09-16T14:48:36.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient: init 2024-09-16T14:48:36.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:36.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:36.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient(hunting): picked mon.noname-b con 0x7fa35009df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:36.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient(hunting): picked mon.noname-a con 0x7fa35009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:36.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient(hunting): picked mon.noname-c con 0x7fa35009e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:36.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35006b380 con 0x7fa35009e2f0 2024-09-16T14:48:36.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa350122590 con 0x7fa35009df10 2024-09-16T14:48:36.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35006ce70 con 0x7fa35009e740 2024-09-16T14:48:36.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:36.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient(hunting): authenticate will time out at 1298.889671s 2024-09-16T14:48:36.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa356452640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa35009df10 0x7fa35009a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51626/0 (socket says 172.21.15.189:51626) 2024-09-16T14:48:36.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa356452640 1 -- 172.21.15.189:0/1905698502 learned_addr learned my addr 172.21.15.189:0/1905698502 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:36.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa356c53640 1 -- 172.21.15.189:0/1905698502 >> v1:172.21.15.26:6790/0 conn(0x7fa35009e740 legacy=0x7fa3501d17a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:36.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51757063 0 0) 0x7fa350122590 con 0x7fa35009df10 2024-09-16T14:48:36.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:36.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:36.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:36.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient(hunting): my global_id is 144220 2024-09-16T14:48:36.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa334003560 con 0x7fa35009df10 2024-09-16T14:48:36.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2912544553 0 0) 0x7fa334003560 con 0x7fa35009df10 2024-09-16T14:48:36.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa350122590 con 0x7fa35009df10 2024-09-16T14:48:36.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa34c004190 con 0x7fa35009df10 2024-09-16T14:48:36.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:36.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:36.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient(hunting): dump: 2024-09-16T14:48:36.788 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:36.788 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:36.789 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.789 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.789 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:36.789 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:36.789 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:36.789 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:36.789 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:36.789 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:36.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:36.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204002979 0 0) 0x7fa35006b380 con 0x7fa35009e2f0 2024-09-16T14:48:36.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:36.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:36.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:36.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient(hunting): my global_id is 124425 2024-09-16T14:48:36.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa334003560 con 0x7fa35009e2f0 2024-09-16T14:48:36.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3364545471 0 0) 0x7fa350122590 con 0x7fa35009df10 2024-09-16T14:48:36.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 >> v1:172.21.15.26:6790/0 conn(0x7fa35009e740 legacy=0x7fa3501d17a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:36.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 >> v1:172.21.15.26:6789/0 conn(0x7fa35009e2f0 legacy=0x7fa35009abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:36.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:36.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 1 monclient: found mon.b 2024-09-16T14:48:36.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:36.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:36.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa350120b80 con 0x7fa35009df10 2024-09-16T14:48:36.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient: _finish_auth 0 2024-09-16T14:48:36.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:36.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:36.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 5 monclient: authenticate success, global_id 144220 2024-09-16T14:48:36.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient: _renew_subs 2024-09-16T14:48:36.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:36.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 1 -- 172.21.15.189:0/1905698502 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa350121cf0 con 0x7fa35009df10 2024-09-16T14:48:36.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient: _renew_subs 2024-09-16T14:48:36.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:36.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 1 -- 172.21.15.189:0/1905698502 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa35013c480 con 0x7fa35009df10 2024-09-16T14:48:36.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa34c0043b0 con 0x7fa35009df10 2024-09-16T14:48:36.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:36.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa34c0046c0 con 0x7fa35009df10 2024-09-16T14:48:36.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:36.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa354c4f640 4 set_mon_vals no callback set 2024-09-16T14:48:36.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:36.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 10 monclient: dump: 2024-09-16T14:48:36.792 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:36.792 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:36.792 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.792 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:36.792 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:36.792 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:36.792 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:36.792 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:36.792 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:36.792 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:36.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:36.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:36.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:36.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa3586b6640 1 -- 172.21.15.189:0/1905698502 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa338005050 con 0x7fa35009df10 2024-09-16T14:48:36.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa34c004820 con 0x7fa35009df10 2024-09-16T14:48:36.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(157..157 src has 73..157) v4 ==== 8324+0+0 (unknown 3537063506 0 0) 0x7fa34c044390 con 0x7fa35009df10 2024-09-16T14:48:36.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.773+0000 7fa355c51640 10 monclient: get_auth_request con 0x7fa3340361b0 auth_method 0 2024-09-16T14:48:36.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.781+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa34c004ff0 con 0x7fa35009df10 2024-09-16T14:48:36.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.781+0000 7fa346ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:36.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.781+0000 7fa346ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:36.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.937+0000 7fa3586b6640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T14:48:36.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.937+0000 7fa3586b6640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T14:48:36.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.937+0000 7fa3586b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:36.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:36.937+0000 7fa3586b6640 1 -- 172.21.15.189:0/1905698502 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]} v 0) v1 -- 0x7fa338005300 con 0x7fa35009df10 2024-09-16T14:48:37.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:37.633+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'globster' v158) v1 ==== 145+0+0 (unknown 3771737401 0 0) 0x7fa34c012ad0 con 0x7fa35009df10 2024-09-16T14:48:37.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:37.633+0000 7fa346ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T14:48:37.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:37.633+0000 7fa346ffd640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'globster' 2024-09-16T14:48:37.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:37.737+0000 7fa3586b6640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T14:48:37.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:37.737+0000 7fa3586b6640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T14:48:37.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:37.737+0000 7fa3586b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:37.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:37.737+0000 7fa3586b6640 1 -- 172.21.15.189:0/1905698502 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]} v 0) v1 -- 0x7fa3380010a0 con 0x7fa35009df10 2024-09-16T14:48:38.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.013+0000 7fa346ffd640 1 -- 172.21.15.189:0/1905698502 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}]=0 osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' v159) v1 ==== 257+0+0 (unknown 4041001811 0 0) 0x7fa34c014250 con 0x7fa35009df10 2024-09-16T14:48:38.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.013+0000 7fa346ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T14:48:38.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.013+0000 7fa346ffd640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' 2024-09-16T14:48:38.015 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' 2024-09-16T14:48:38.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.017+0000 7fa3586b6640 1 -- 172.21.15.189:0/1905698502 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa3340361b0 legacy=0x7fa334038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:38.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.017+0000 7fa3586b6640 10 monclient: shutdown 2024-09-16T14:48:38.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.017+0000 7fa3586b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:38.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.017+0000 7fa3586b6640 1 -- 172.21.15.189:0/1905698502 >> v1:172.21.15.189:6789/0 conn(0x7fa35009df10 legacy=0x7fa35009a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:38.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.017+0000 7fa3586b6640 1 -- 172.21.15.189:0/1905698502 shutdown_connections 2024-09-16T14:48:38.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.017+0000 7fa3586b6640 1 -- 172.21.15.189:0/1905698502 >> 172.21.15.189:0/1905698502 conn(0x7fa3500a3320 msgr2=0x7fa35012a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:38.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.017+0000 7fa3586b6640 1 -- 172.21.15.189:0/1905698502 shutdown_connections 2024-09-16T14:48:38.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.017+0000 7fa3586b6640 1 -- 172.21.15.189:0/1905698502 wait complete. 2024-09-16T14:48:38.038 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T14:48:38.039 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T14:48:38.039 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep globster 2024-09-16T14:48:38.039 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '10\.' 2024-09-16T14:48:38.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.149+0000 7f0035a78640 10 monclient: get_monmap_and_config 2024-09-16T14:48:38.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.149+0000 7f0035a78640 10 monclient: build_initial_monmap 2024-09-16T14:48:38.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.149+0000 7f0035a78640 10 monclient: monmap: 2024-09-16T14:48:38.152 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:38.152 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:38.152 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:38.152350+0000 2024-09-16T14:48:38.152 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:38.152350+0000 2024-09-16T14:48:38.153 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:38.153 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:38.153 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:38.153 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:38.153 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:38.153 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.149+0000 7f0035a78640 1 Processor -- start 2024-09-16T14:48:38.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.149+0000 7f0035a78640 1 -- start start 2024-09-16T14:48:38.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.149+0000 7f0035a78640 10 monclient: init 2024-09-16T14:48:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient(hunting): picked mon.noname-c con 0x7f0030099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient(hunting): picked mon.noname-b con 0x7f00301334d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient(hunting): picked mon.noname-a con 0x7f0030136fe0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003009b350 con 0x7f0030136fe0 2024-09-16T14:48:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003009f2e0 con 0x7f0030099a70 2024-09-16T14:48:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003009b070 con 0x7f00301334d0 2024-09-16T14:48:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:38.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient(hunting): authenticate will time out at 1300.268546s 2024-09-16T14:48:38.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002f7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0030099a70 0x7f003009bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51636/0 (socket says 172.21.15.189:51636) 2024-09-16T14:48:38.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002f7fe640 1 -- 172.21.15.189:0/530613018 learned_addr learned my addr 172.21.15.189:0/530613018 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:38.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0027fff640 1 -- 172.21.15.189:0/530613018 >> v1:172.21.15.26:6790/0 conn(0x7f00301334d0 legacy=0x7f00301358c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:38.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 1 -- 172.21.15.189:0/530613018 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145709513 0 0) 0x7f003009f2e0 con 0x7f0030099a70 2024-09-16T14:48:38.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:38.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:38.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:38.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 10 monclient(hunting): my global_id is 144223 2024-09-16T14:48:38.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 1 -- 172.21.15.189:0/530613018 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f001c003530 con 0x7f0030099a70 2024-09-16T14:48:38.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 1 -- 172.21.15.189:0/530613018 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2501311034 0 0) 0x7f001c003530 con 0x7f0030099a70 2024-09-16T14:48:38.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 1 -- 172.21.15.189:0/530613018 >> v1:172.21.15.26:6790/0 conn(0x7f00301334d0 legacy=0x7f00301358c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:38.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 1 -- 172.21.15.189:0/530613018 >> v1:172.21.15.26:6789/0 conn(0x7f0030136fe0 legacy=0x7f00301393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:38.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:38.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 1 monclient: found mon.noname-c 2024-09-16T14:48:38.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:38.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 1 -- 172.21.15.189:0/530613018 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003009b630 con 0x7f0030099a70 2024-09-16T14:48:38.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 10 monclient: _finish_auth 0 2024-09-16T14:48:38.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:38.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:38.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 1 -- 172.21.15.189:0/530613018 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0020001620 con 0x7f0030099a70 2024-09-16T14:48:38.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:38.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:38.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 10 monclient: dump: 2024-09-16T14:48:38.164 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:38.164 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:38.164 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:38.165 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 5 monclient: authenticate success, global_id 144223 2024-09-16T14:48:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 1 -- 172.21.15.189:0/530613018 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00200018d0 con 0x7f0030099a70 2024-09-16T14:48:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 1 -- 172.21.15.189:0/530613018 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0020001ca0 con 0x7f0030099a70 2024-09-16T14:48:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002effd640 10 monclient: dump: 2024-09-16T14:48:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:38.167 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 4 set_mon_vals no callback set 2024-09-16T14:48:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient: shutdown 2024-09-16T14:48:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 1 -- 172.21.15.189:0/530613018 >> v1:172.21.15.189:6789/0 conn(0x7f0030099a70 legacy=0x7f003009bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 1 -- 172.21.15.189:0/530613018 shutdown_connections 2024-09-16T14:48:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 1 -- 172.21.15.189:0/530613018 >> 172.21.15.189:0/530613018 conn(0x7f00300a3320 msgr2=0x7f003009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 1 -- 172.21.15.189:0/530613018 shutdown_connections 2024-09-16T14:48:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 1 -- 172.21.15.189:0/530613018 wait complete. 2024-09-16T14:48:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient: build_initial_monmap 2024-09-16T14:48:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient: monmap: 2024-09-16T14:48:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:38.168 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 1 Processor -- start 2024-09-16T14:48:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 1 -- start start 2024-09-16T14:48:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient: init 2024-09-16T14:48:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient(hunting): picked mon.noname-a con 0x7f0030099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient(hunting): picked mon.noname-c con 0x7f00301334d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient(hunting): picked mon.noname-b con 0x7f0030136fe0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003011e670 con 0x7f0030099a70 2024-09-16T14:48:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003011fb90 con 0x7f0030136fe0 2024-09-16T14:48:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00300a21e0 con 0x7f00301334d0 2024-09-16T14:48:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0035a78640 10 monclient(hunting): authenticate will time out at 1300.271447s 2024-09-16T14:48:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f0027fff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f00301334d0 legacy=0x7f00301cc3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002ffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0030136fe0 0x7f00301cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51650/0 (socket says 172.21.15.189:51650) 2024-09-16T14:48:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002ffff640 1 -- 172.21.15.189:0/1869847830 learned_addr learned my addr 172.21.15.189:0/1869847830 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2017280949 0 0) 0x7f003011fb90 con 0x7f0030136fe0 2024-09-16T14:48:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002cff9640 10 monclient(hunting): my global_id is 144226 2024-09-16T14:48:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.153+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0010003530 con 0x7f0030136fe0 2024-09-16T14:48:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 888632350 0 0) 0x7f0010003530 con 0x7f0030136fe0 2024-09-16T14:48:38.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f003011fb90 con 0x7f0030136fe0 2024-09-16T14:48:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0028001c50 con 0x7f0030136fe0 2024-09-16T14:48:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 10 monclient(hunting): dump: 2024-09-16T14:48:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:38.171 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:38.171 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858176852 0 0) 0x7f003011e670 con 0x7f0030099a70 2024-09-16T14:48:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 10 monclient(hunting): my global_id is 124431 2024-09-16T14:48:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0010003530 con 0x7f0030099a70 2024-09-16T14:48:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1324754490 0 0) 0x7f003011fb90 con 0x7f0030136fe0 2024-09-16T14:48:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 >> v1:172.21.15.26:6790/0 conn(0x7f00301334d0 legacy=0x7f00301cc3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 >> v1:172.21.15.26:6789/0 conn(0x7f0030099a70 legacy=0x7f00301324b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:48:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 1 monclient: found mon.b 2024-09-16T14:48:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:38.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003011f8b0 con 0x7f0030136fe0 2024-09-16T14:48:38.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 10 monclient: _finish_auth 0 2024-09-16T14:48:38.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:48:38.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:38.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f0035a78640 5 monclient: authenticate success, global_id 144226 2024-09-16T14:48:38.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f0035a78640 10 monclient: _renew_subs 2024-09-16T14:48:38.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f0035a78640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f0035a78640 1 -- 172.21.15.189:0/1869847830 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003011f5d0 con 0x7f0030136fe0 2024-09-16T14:48:38.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f0035a78640 10 monclient: _renew_subs 2024-09-16T14:48:38.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f0035a78640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f0035a78640 1 -- 172.21.15.189:0/1869847830 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0030132bc0 con 0x7f0030136fe0 2024-09-16T14:48:38.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0028002020 con 0x7f0030136fe0 2024-09-16T14:48:38.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:38.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0028003e70 con 0x7f0030136fe0 2024-09-16T14:48:38.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:38.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:38.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 10 monclient: dump: 2024-09-16T14:48:38.174 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:38.174 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:38.174 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.174 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:38.175 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002e7fc640 4 set_mon_vals no callback set 2024-09-16T14:48:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f0035a78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f0035a78640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f0035a78640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f0035a78640 1 -- 172.21.15.189:0/1869847830 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0014005050 con 0x7f0030136fe0 2024-09-16T14:48:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0028003fd0 con 0x7f0030136fe0 2024-09-16T14:48:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(159..159 src has 73..159) v4 ==== 9013+0+0 (unknown 3655967767 0 0) 0x7f0028044a50 con 0x7f0030136fe0 2024-09-16T14:48:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.157+0000 7f002f7fe640 10 monclient: get_auth_request con 0x7f0010036130 auth_method 0 2024-09-16T14:48:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.161+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0028014b40 con 0x7f0030136fe0 2024-09-16T14:48:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.161+0000 7f002cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.161+0000 7f002cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:38.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.329+0000 7f0035a78640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:38.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.329+0000 7f0035a78640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:38.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.329+0000 7f0035a78640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.329+0000 7f0035a78640 1 -- 172.21.15.189:0/1869847830 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f0014005300 con 0x7f0030136fe0 2024-09-16T14:48:38.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.329+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v159) v1 ==== 83+0+2378 (unknown 350933362 0 619860170) 0x7f0028012fb0 con 0x7f0030136fe0 2024-09-16T14:48:38.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.329+0000 7f002cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:38.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.329+0000 7f002cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.433+0000 7f0035a78640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.433+0000 7f0035a78640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.433+0000 7f0035a78640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.433+0000 7f0035a78640 1 -- 172.21.15.189:0/1869847830 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f00140055a0 con 0x7f0030136fe0 2024-09-16T14:48:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.433+0000 7f002cff9640 1 -- 172.21.15.189:0/1869847830 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v159) v1 ==== 83+0+2378 (unknown 350933362 0 619860170) 0x7f0028014910 con 0x7f0030136fe0 2024-09-16T14:48:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.433+0000 7f002cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.433+0000 7f002cff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.437+0000 7f0035a78640 1 -- 172.21.15.189:0/1869847830 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0010036130 legacy=0x7f00100385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.437+0000 7f0035a78640 10 monclient: shutdown 2024-09-16T14:48:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.437+0000 7f0035a78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.437+0000 7f0035a78640 1 -- 172.21.15.189:0/1869847830 >> v1:172.21.15.189:6789/0 conn(0x7f0030136fe0 legacy=0x7f00301cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.437+0000 7f0035a78640 1 -- 172.21.15.189:0/1869847830 shutdown_connections 2024-09-16T14:48:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.437+0000 7f0035a78640 1 -- 172.21.15.189:0/1869847830 >> 172.21.15.189:0/1869847830 conn(0x7f00300a3320 msgr2=0x7f003009d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.437+0000 7f0035a78640 1 -- 172.21.15.189:0/1869847830 shutdown_connections 2024-09-16T14:48:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.437+0000 7f0035a78640 1 -- 172.21.15.189:0/1869847830 wait complete. 2024-09-16T14:48:38.453 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T14:48:38.453 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T14:48:38.454 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T14:48:38.454 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T14:48:38.454 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep globster 2024-09-16T14:48:38.454 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '11\.' 2024-09-16T14:48:38.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 10 monclient: get_monmap_and_config 2024-09-16T14:48:38.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 10 monclient: build_initial_monmap 2024-09-16T14:48:38.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 10 monclient: monmap: 2024-09-16T14:48:38.559 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:38.559 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:38.559 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:38.559288+0000 2024-09-16T14:48:38.559 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:38.559288+0000 2024-09-16T14:48:38.560 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:38.560 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:38.560 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:38.560 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:38.560 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:38.560 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 1 Processor -- start 2024-09-16T14:48:38.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 1 -- start start 2024-09-16T14:48:38.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 10 monclient: init 2024-09-16T14:48:38.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:38.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:38.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 10 monclient(hunting): picked mon.noname-b con 0x7f1eb41318a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:38.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 10 monclient(hunting): picked mon.noname-a con 0x7f1eb4134d50 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:38.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 10 monclient(hunting): picked mon.noname-c con 0x7f1eb413b2b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:38.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb409d8a0 con 0x7f1eb4134d50 2024-09-16T14:48:38.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb409ddf0 con 0x7f1eb413b2b0 2024-09-16T14:48:38.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb409e8b0 con 0x7f1eb41318a0 2024-09-16T14:48:38.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:38.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 10 monclient(hunting): authenticate will time out at 1300.675094s 2024-09-16T14:48:38.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb9aab640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f1eb41318a0 legacy=0x7f1eb4131c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:38.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eba2ac640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1eb413b2b0 0x7f1eb413d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51662/0 (socket says 172.21.15.189:51662) 2024-09-16T14:48:38.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eba2ac640 1 -- 172.21.15.189:0/2751128927 learned_addr learned my addr 172.21.15.189:0/2751128927 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:38.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 1 -- 172.21.15.189:0/2751128927 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396658124 0 0) 0x7f1eb409ddf0 con 0x7f1eb413b2b0 2024-09-16T14:48:38.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:38.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:38.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:38.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 10 monclient(hunting): my global_id is 144229 2024-09-16T14:48:38.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 1 -- 172.21.15.189:0/2751128927 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea0003530 con 0x7f1eb413b2b0 2024-09-16T14:48:38.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 1 -- 172.21.15.189:0/2751128927 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 314972616 0 0) 0x7f1ea0003530 con 0x7f1eb413b2b0 2024-09-16T14:48:38.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 1 -- 172.21.15.189:0/2751128927 >> v1:172.21.15.26:6790/0 conn(0x7f1eb41318a0 legacy=0x7f1eb4131c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:38.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 1 -- 172.21.15.189:0/2751128927 >> v1:172.21.15.26:6789/0 conn(0x7f1eb4134d50 legacy=0x7f1eb4139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:38.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 10 monclient: _finish_hunting 0 2024-09-16T14:48:38.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 1 monclient: found mon.noname-c 2024-09-16T14:48:38.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:38.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 1 -- 172.21.15.189:0/2751128927 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb409eb90 con 0x7f1eb413b2b0 2024-09-16T14:48:38.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 10 monclient: _finish_auth 0 2024-09-16T14:48:38.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 10 monclient: _check_auth_tickets 2024-09-16T14:48:38.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:38.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 1 -- 172.21.15.189:0/2751128927 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1eb0001620 con 0x7f1eb413b2b0 2024-09-16T14:48:38.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:38.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:38.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 10 monclient: dump: 2024-09-16T14:48:38.570 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:38.570 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:38.570 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.570 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.570 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:38.570 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:38.570 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:38.570 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:38.570 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:38.570 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 5 monclient: authenticate success, global_id 144229 2024-09-16T14:48:38.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1ebbd0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:38.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.557+0000 7f1eb8aa9640 1 -- 172.21.15.189:0/2751128927 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1eb00018d0 con 0x7f1eb413b2b0 2024-09-16T14:48:38.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eb8aa9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:38.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:38.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 4 set_mon_vals no callback set 2024-09-16T14:48:38.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eb8aa9640 1 -- 172.21.15.189:0/2751128927 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1eb0001c60 con 0x7f1eb413b2b0 2024-09-16T14:48:38.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eb8aa9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:38.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eb8aa9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:38.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eb8aa9640 10 monclient: dump: 2024-09-16T14:48:38.571 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:38.571 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:38.571 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.571 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:38.572 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient: shutdown 2024-09-16T14:48:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 1 -- 172.21.15.189:0/2751128927 >> v1:172.21.15.189:6789/0 conn(0x7f1eb413b2b0 legacy=0x7f1eb413d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 1 -- 172.21.15.189:0/2751128927 shutdown_connections 2024-09-16T14:48:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 1 -- 172.21.15.189:0/2751128927 >> 172.21.15.189:0/2751128927 conn(0x7f1eb40a3320 msgr2=0x7f1eb409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 1 -- 172.21.15.189:0/2751128927 shutdown_connections 2024-09-16T14:48:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 1 -- 172.21.15.189:0/2751128927 wait complete. 2024-09-16T14:48:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient: build_initial_monmap 2024-09-16T14:48:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient: monmap: 2024-09-16T14:48:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:38.573 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 1 Processor -- start 2024-09-16T14:48:38.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 1 -- start start 2024-09-16T14:48:38.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient: init 2024-09-16T14:48:38.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:38.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:38.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient(hunting): picked mon.noname-b con 0x7f1eb41318a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:38.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient(hunting): picked mon.noname-a con 0x7f1eb4134d50 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:38.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient(hunting): picked mon.noname-c con 0x7f1eb413b2b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:38.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb413abd0 con 0x7f1eb4134d50 2024-09-16T14:48:38.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb406b380 con 0x7f1eb41318a0 2024-09-16T14:48:38.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb411b070 con 0x7f1eb413b2b0 2024-09-16T14:48:38.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:38.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient(hunting): authenticate will time out at 1300.678070s 2024-09-16T14:48:38.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eb9aab640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1eb41318a0 0x7f1eb4139370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51670/0 (socket says 172.21.15.189:51670) 2024-09-16T14:48:38.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eba2ac640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f1eb413b2b0 legacy=0x7f1eb41d1480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:38.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eb9aab640 1 -- 172.21.15.189:0/2044041982 learned_addr learned my addr 172.21.15.189:0/2044041982 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:38.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795311518 0 0) 0x7f1eb406b380 con 0x7f1eb41318a0 2024-09-16T14:48:38.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:38.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:38.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:38.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient(hunting): my global_id is 144232 2024-09-16T14:48:38.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e98003530 con 0x7f1eb41318a0 2024-09-16T14:48:38.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1565939254 0 0) 0x7f1e98003530 con 0x7f1eb41318a0 2024-09-16T14:48:38.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1eb406b380 con 0x7f1eb41318a0 2024-09-16T14:48:38.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1eac004510 con 0x7f1eb41318a0 2024-09-16T14:48:38.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient(hunting): dump: 2024-09-16T14:48:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:38.577 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521464679 0 0) 0x7f1eb413abd0 con 0x7f1eb4134d50 2024-09-16T14:48:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient(hunting): my global_id is 124437 2024-09-16T14:48:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e98003530 con 0x7f1eb4134d50 2024-09-16T14:48:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3666262896 0 0) 0x7f1eb406b380 con 0x7f1eb41318a0 2024-09-16T14:48:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 >> v1:172.21.15.26:6790/0 conn(0x7f1eb413b2b0 legacy=0x7f1eb41d1480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 >> v1:172.21.15.26:6789/0 conn(0x7f1eb4134d50 legacy=0x7f1eb4139a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 1 monclient: found mon.b 2024-09-16T14:48:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb411b350 con 0x7f1eb41318a0 2024-09-16T14:48:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 5 monclient: authenticate success, global_id 144232 2024-09-16T14:48:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient: _renew_subs 2024-09-16T14:48:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 1 -- 172.21.15.189:0/2044041982 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb41311d0 con 0x7f1eb41318a0 2024-09-16T14:48:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient: _renew_subs 2024-09-16T14:48:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 1 -- 172.21.15.189:0/2044041982 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb4119790 con 0x7f1eb41318a0 2024-09-16T14:48:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1eac004830 con 0x7f1eb41318a0 2024-09-16T14:48:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1eac004bc0 con 0x7f1eb41318a0 2024-09-16T14:48:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eab7fe640 4 set_mon_vals no callback set 2024-09-16T14:48:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 10 monclient: dump: 2024-09-16T14:48:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:38.580 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1eac004d20 con 0x7f1eb41318a0 2024-09-16T14:48:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1ebbd0f640 1 -- 172.21.15.189:0/2044041982 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e9c005050 con 0x7f1eb41318a0 2024-09-16T14:48:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.561+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(159..159 src has 73..159) v4 ==== 9013+0+0 (unknown 3655967767 0 0) 0x7f1eac044b00 con 0x7f1eb41318a0 2024-09-16T14:48:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.565+0000 7f1eb92aa640 10 monclient: get_auth_request con 0x7f1e98036130 auth_method 0 2024-09-16T14:48:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.569+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1eac0054f0 con 0x7f1eb41318a0 2024-09-16T14:48:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.569+0000 7f1eaa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.569+0000 7f1eaa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:38.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.737+0000 7f1ebbd0f640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:38.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.737+0000 7f1ebbd0f640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:38.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.737+0000 7f1ebbd0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.737+0000 7f1ebbd0f640 1 -- 172.21.15.189:0/2044041982 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f1e9c005300 con 0x7f1eb41318a0 2024-09-16T14:48:38.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.737+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v159) v1 ==== 83+0+2378 (unknown 350933362 0 619860170) 0x7f1eac01d150 con 0x7f1eb41318a0 2024-09-16T14:48:38.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.737+0000 7f1eaa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:38.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.737+0000 7f1eaa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:38.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.833+0000 7f1ebbd0f640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:38.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.833+0000 7f1ebbd0f640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:38.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.833+0000 7f1ebbd0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.833+0000 7f1ebbd0f640 1 -- 172.21.15.189:0/2044041982 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f1e9c0055a0 con 0x7f1eb41318a0 2024-09-16T14:48:38.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.837+0000 7f1eaa7fc640 1 -- 172.21.15.189:0/2044041982 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v159) v1 ==== 83+0+2378 (unknown 350933362 0 619860170) 0x7f1eac012fd0 con 0x7f1eb41318a0 2024-09-16T14:48:38.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.837+0000 7f1eaa7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:38.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.837+0000 7f1eaa7fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:38.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.837+0000 7f1ebbd0f640 1 -- 172.21.15.189:0/2044041982 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1e98036130 legacy=0x7f1e980385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:38.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.837+0000 7f1ebbd0f640 10 monclient: shutdown 2024-09-16T14:48:38.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.837+0000 7f1ebbd0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:38.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.837+0000 7f1ebbd0f640 1 -- 172.21.15.189:0/2044041982 >> v1:172.21.15.189:6789/0 conn(0x7f1eb41318a0 legacy=0x7f1eb4139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:38.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.837+0000 7f1ebbd0f640 1 -- 172.21.15.189:0/2044041982 shutdown_connections 2024-09-16T14:48:38.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.837+0000 7f1ebbd0f640 1 -- 172.21.15.189:0/2044041982 >> 172.21.15.189:0/2044041982 conn(0x7f1eb40a3320 msgr2=0x7f1eb41333b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:38.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.837+0000 7f1ebbd0f640 1 -- 172.21.15.189:0/2044041982 shutdown_connections 2024-09-16T14:48:38.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.837+0000 7f1ebbd0f640 1 -- 172.21.15.189:0/2044041982 wait complete. 2024-09-16T14:48:38.854 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T14:48:38.854 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T14:48:38.856 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T14:48:38.856 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T14:48:38.856 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep globster 2024-09-16T14:48:38.856 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '12\.' 2024-09-16T14:48:38.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.965+0000 7fe016990640 10 monclient: get_monmap_and_config 2024-09-16T14:48:38.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.965+0000 7fe016990640 10 monclient: build_initial_monmap 2024-09-16T14:48:38.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.965+0000 7fe016990640 10 monclient: monmap: 2024-09-16T14:48:38.969 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:38.969 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:38.969 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:38.968412+0000 2024-09-16T14:48:38.969 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:38.968412+0000 2024-09-16T14:48:38.969 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:38.969 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:38.969 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:38.969 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:38.969 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:38.969 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.965+0000 7fe016990640 1 Processor -- start 2024-09-16T14:48:38.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.965+0000 7fe016990640 1 -- start start 2024-09-16T14:48:38.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.965+0000 7fe016990640 10 monclient: init 2024-09-16T14:48:38.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:38.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:38.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient(hunting): picked mon.noname-c con 0x7fe01009df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:38.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient(hunting): picked mon.noname-a con 0x7fe01009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:38.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient(hunting): picked mon.noname-b con 0x7fe01009e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:38.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01009ee20 con 0x7fe01009e2f0 2024-09-16T14:48:38.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01009f100 con 0x7fe01009df10 2024-09-16T14:48:38.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01009f3e0 con 0x7fe01009e740 2024-09-16T14:48:38.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:38.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient(hunting): authenticate will time out at 1301.084590s 2024-09-16T14:48:38.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00ffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe01009df10 0x7fe01009d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57490/0 (socket says 172.21.15.189:57490) 2024-09-16T14:48:38.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00ffff640 1 -- 172.21.15.189:0/2634451471 learned_addr learned my addr 172.21.15.189:0/2634451471 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:38.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe014f2d640 1 -- 172.21.15.189:0/2634451471 >> v1:172.21.15.26:6790/0 conn(0x7fe01009e740 legacy=0x7fe01013b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:38.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 1 -- 172.21.15.189:0/2634451471 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171993134 0 0) 0x7fe01009f100 con 0x7fe01009df10 2024-09-16T14:48:38.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:38.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:38.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:38.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 10 monclient(hunting): my global_id is 144235 2024-09-16T14:48:38.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 1 -- 172.21.15.189:0/2634451471 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdff8003530 con 0x7fe01009df10 2024-09-16T14:48:38.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00f7fe640 1 -- 172.21.15.189:0/2634451471 >> v1:172.21.15.26:6789/0 conn(0x7fe01009e2f0 legacy=0x7fe0101378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T14:48:38.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00f7fe640 1 -- 172.21.15.189:0/2634451471 >> v1:172.21.15.26:6789/0 conn(0x7fe01009e2f0 legacy=0x7fe0101378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:48:38.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00f7fe640 1 --1- 172.21.15.189:0/2634451471 >> v1:172.21.15.26:6789/0 conn(0x7fe01009e2f0 0x7fe0101378d0 :-1 s=OPENED pgs=432 cs=1 l=1).handle_message read tag failed 2024-09-16T14:48:38.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00f7fe640 1 --1- 172.21.15.189:0/2634451471 >> v1:172.21.15.26:6789/0 conn(0x7fe01009e2f0 0x7fe0101378d0 :-1 s=OPENED pgs=432 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:48:38.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6789/0 2024-09-16T14:48:38.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 1 -- 172.21.15.189:0/2634451471 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 455969312 0 0) 0x7fdff8003530 con 0x7fe01009df10 2024-09-16T14:48:38.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 1 -- 172.21.15.189:0/2634451471 >> v1:172.21.15.26:6790/0 conn(0x7fe01009e740 legacy=0x7fe01013b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:38.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 1 -- 172.21.15.189:0/2634451471 >> v1:172.21.15.26:6789/0 conn(0x7fe01009e2f0 legacy=0x7fe0101378d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:38.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:38.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 1 monclient: found mon.noname-c 2024-09-16T14:48:38.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:38.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 1 -- 172.21.15.189:0/2634451471 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01009c690 con 0x7fe01009df10 2024-09-16T14:48:38.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 10 monclient: _finish_auth 0 2024-09-16T14:48:38.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:38.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:38.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 1 -- 172.21.15.189:0/2634451471 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe0000019a0 con 0x7fe01009df10 2024-09-16T14:48:38.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:38.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:38.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 10 monclient: dump: 2024-09-16T14:48:38.980 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:38.980 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:38.980 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.980 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.980 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:38.980 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:38.981 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:38.981 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:38.981 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:38.981 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 5 monclient: authenticate success, global_id 144235 2024-09-16T14:48:38.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:38.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 1 -- 172.21.15.189:0/2634451471 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe000001c80 con 0x7fe01009df10 2024-09-16T14:48:38.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:38.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:38.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 4 set_mon_vals no callback set 2024-09-16T14:48:38.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 1 -- 172.21.15.189:0/2634451471 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe000002010 con 0x7fe01009df10 2024-09-16T14:48:38.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:38.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:38.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00effd640 10 monclient: dump: 2024-09-16T14:48:38.982 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:38.982 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:38.982 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.982 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.982 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:38.982 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:38.982 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:38.982 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:38.982 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:38.982 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient: shutdown 2024-09-16T14:48:38.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:38.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 1 -- 172.21.15.189:0/2634451471 >> v1:172.21.15.189:6789/0 conn(0x7fe01009df10 legacy=0x7fe01009d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:38.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 1 -- 172.21.15.189:0/2634451471 shutdown_connections 2024-09-16T14:48:38.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 1 -- 172.21.15.189:0/2634451471 >> 172.21.15.189:0/2634451471 conn(0x7fe0100a3320 msgr2=0x7fe01009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:38.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 1 -- 172.21.15.189:0/2634451471 shutdown_connections 2024-09-16T14:48:38.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 1 -- 172.21.15.189:0/2634451471 wait complete. 2024-09-16T14:48:38.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient: build_initial_monmap 2024-09-16T14:48:38.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient: monmap: 2024-09-16T14:48:38.983 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:38.983 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:38.983 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:38.983 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:38.983 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:38.983 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:38.983 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:38.984 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:38.984 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:38.984 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 1 Processor -- start 2024-09-16T14:48:38.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 1 -- start start 2024-09-16T14:48:38.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient: init 2024-09-16T14:48:38.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:38.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:38.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient(hunting): picked mon.noname-b con 0x7fe01009df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:38.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient(hunting): picked mon.noname-a con 0x7fe01009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:38.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient(hunting): picked mon.noname-c con 0x7fe01009e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:38.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01006ce70 con 0x7fe01009e2f0 2024-09-16T14:48:38.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01013b5e0 con 0x7fe01009df10 2024-09-16T14:48:38.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01006f920 con 0x7fe01009e740 2024-09-16T14:48:38.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:38.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe016990640 10 monclient(hunting): authenticate will time out at 1301.087565s 2024-09-16T14:48:38.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00ffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe01009df10 0x7fe01009a220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57496/0 (socket says 172.21.15.189:57496) 2024-09-16T14:48:38.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00ffff640 1 -- 172.21.15.189:0/2139067695 learned_addr learned my addr 172.21.15.189:0/2139067695 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:38.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe014f2d640 1 -- 172.21.15.189:0/2139067695 >> v1:172.21.15.26:6790/0 conn(0x7fe01009e740 legacy=0x7fe0101d17a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:38.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82271615 0 0) 0x7fe01013b5e0 con 0x7fe01009df10 2024-09-16T14:48:38.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:38.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:38.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:38.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00cff9640 10 monclient(hunting): my global_id is 144238 2024-09-16T14:48:38.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.969+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdffc003530 con 0x7fe01009df10 2024-09-16T14:48:38.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 227731785 0 0) 0x7fdffc003530 con 0x7fe01009df10 2024-09-16T14:48:38.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe01013b5e0 con 0x7fe01009df10 2024-09-16T14:48:38.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe000001960 con 0x7fe01009df10 2024-09-16T14:48:38.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:38.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:38.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 10 monclient(hunting): dump: 2024-09-16T14:48:38.986 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:38.986 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:38.986 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.986 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.986 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:38.986 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:38.986 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:38.986 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:38.987 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:38.987 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:38.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472317963 0 0) 0x7fe01006ce70 con 0x7fe01009e2f0 2024-09-16T14:48:38.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:38.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:38.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:38.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 10 monclient(hunting): my global_id is 124440 2024-09-16T14:48:38.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdffc003530 con 0x7fe01009e2f0 2024-09-16T14:48:38.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2730338584 0 0) 0x7fe01013b5e0 con 0x7fe01009df10 2024-09-16T14:48:38.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 >> v1:172.21.15.26:6790/0 conn(0x7fe01009e740 legacy=0x7fe0101d17a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:38.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 >> v1:172.21.15.26:6789/0 conn(0x7fe01009e2f0 legacy=0x7fe01009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:38.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:48:38.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 1 monclient: found mon.b 2024-09-16T14:48:38.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:38.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01009b040 con 0x7fe01009df10 2024-09-16T14:48:38.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 10 monclient: _finish_auth 0 2024-09-16T14:48:38.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:48:38.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:38.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe016990640 5 monclient: authenticate success, global_id 144238 2024-09-16T14:48:38.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe016990640 10 monclient: _renew_subs 2024-09-16T14:48:38.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe016990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe016990640 1 -- 172.21.15.189:0/2139067695 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01006b380 con 0x7fe01009df10 2024-09-16T14:48:38.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe016990640 10 monclient: _renew_subs 2024-09-16T14:48:38.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe016990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe016990640 1 -- 172.21.15.189:0/2139067695 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe010139e50 con 0x7fe01009df10 2024-09-16T14:48:38.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe000001590 con 0x7fe01009df10 2024-09-16T14:48:38.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:38.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe000005120 con 0x7fe01009df10 2024-09-16T14:48:38.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:38.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:38.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 10 monclient: dump: 2024-09-16T14:48:38.989 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:38.990 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:38.990 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.990 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:38.990 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:38.990 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:38.990 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:38.990 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:38.990 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:38.990 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:38.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00e7fc640 4 set_mon_vals no callback set 2024-09-16T14:48:38.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe00000fc10 con 0x7fe01009df10 2024-09-16T14:48:38.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe016990640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:38.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe016990640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:38.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe016990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:38.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe016990640 1 -- 172.21.15.189:0/2139067695 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff0005050 con 0x7fe01009df10 2024-09-16T14:48:38.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(159..159 src has 73..159) v4 ==== 9013+0+0 (unknown 3655967767 0 0) 0x7fe000044cc0 con 0x7fe01009df10 2024-09-16T14:48:38.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.973+0000 7fe00f7fe640 10 monclient: get_auth_request con 0x7fdffc0361f0 auth_method 0 2024-09-16T14:48:38.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.977+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe000014bb0 con 0x7fe01009df10 2024-09-16T14:48:38.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.977+0000 7fe00cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:38.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:38.977+0000 7fe00cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:39.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.129+0000 7fe016990640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:39.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.129+0000 7fe016990640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:39.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.129+0000 7fe016990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:39.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.129+0000 7fe016990640 1 -- 172.21.15.189:0/2139067695 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fdff0005300 con 0x7fe01009df10 2024-09-16T14:48:39.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.133+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v159) v1 ==== 83+0+2378 (unknown 350933362 0 619860170) 0x7fe000013250 con 0x7fe01009df10 2024-09-16T14:48:39.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.133+0000 7fe00cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:39.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.133+0000 7fe00cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:39.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.225+0000 7fe016990640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:39.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.225+0000 7fe016990640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:39.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.225+0000 7fe016990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:39.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.225+0000 7fe016990640 1 -- 172.21.15.189:0/2139067695 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fdff00055a0 con 0x7fe01009df10 2024-09-16T14:48:39.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.225+0000 7fe00cff9640 1 -- 172.21.15.189:0/2139067695 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v159) v1 ==== 83+0+2378 (unknown 350933362 0 619860170) 0x7fe0000149d0 con 0x7fe01009df10 2024-09-16T14:48:39.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.225+0000 7fe00cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:39.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.225+0000 7fe00cff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:39.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.229+0000 7fe016990640 1 -- 172.21.15.189:0/2139067695 >> v1:172.21.15.189:6812/3196161582 conn(0x7fdffc0361f0 legacy=0x7fdffc0386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:39.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.229+0000 7fe016990640 10 monclient: shutdown 2024-09-16T14:48:39.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.229+0000 7fe016990640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:39.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.229+0000 7fe016990640 1 -- 172.21.15.189:0/2139067695 >> v1:172.21.15.189:6789/0 conn(0x7fe01009df10 legacy=0x7fe01009a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:39.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.229+0000 7fe016990640 1 -- 172.21.15.189:0/2139067695 shutdown_connections 2024-09-16T14:48:39.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.229+0000 7fe016990640 1 -- 172.21.15.189:0/2139067695 >> 172.21.15.189:0/2139067695 conn(0x7fe0100a3320 msgr2=0x7fe0100a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:39.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.229+0000 7fe016990640 1 -- 172.21.15.189:0/2139067695 shutdown_connections 2024-09-16T14:48:39.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.229+0000 7fe016990640 1 -- 172.21.15.189:0/2139067695 wait complete. 2024-09-16T14:48:39.243 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T14:48:39.243 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T14:48:39.243 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight-compat osd.0 7.5 2024-09-16T14:48:39.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.357+0000 7f2ba2644640 10 monclient: get_monmap_and_config 2024-09-16T14:48:39.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.357+0000 7f2ba2644640 10 monclient: build_initial_monmap 2024-09-16T14:48:39.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.357+0000 7f2ba2644640 10 monclient: monmap: 2024-09-16T14:48:39.361 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:39.361 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:39.362 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:39.361661+0000 2024-09-16T14:48:39.362 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:39.361661+0000 2024-09-16T14:48:39.362 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:39.362 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:39.362 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:39.362 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:39.362 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:39.362 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:39.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 1 Processor -- start 2024-09-16T14:48:39.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 1 -- start start 2024-09-16T14:48:39.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 10 monclient: init 2024-09-16T14:48:39.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:39.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:48:39.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 10 monclient(hunting): picked mon.noname-a con 0x7f2b9c09f710 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:39.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 10 monclient(hunting): picked mon.noname-c con 0x7f2b9c098a60 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:39.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 10 monclient(hunting): picked mon.noname-b con 0x7f2b9c0a0200 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:39.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c06a730 con 0x7f2b9c09f710 2024-09-16T14:48:39.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c06aa00 con 0x7f2b9c098a60 2024-09-16T14:48:39.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c0991b0 con 0x7f2b9c0a0200 2024-09-16T14:48:39.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:39.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 10 monclient(hunting): authenticate will time out at 1301.477619s 2024-09-16T14:48:39.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9b7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2b9c098a60 0x7f2b9c139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57512/0 (socket says 172.21.15.189:57512) 2024-09-16T14:48:39.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba0be1640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2b9c0a0200 legacy=0x7f2b9c13d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:39.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9b7fe640 1 -- 172.21.15.189:0/3218948801 learned_addr learned my addr 172.21.15.189:0/3218948801 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:39.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 1 -- 172.21.15.189:0/3218948801 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286626513 0 0) 0x7f2b9c06aa00 con 0x7f2b9c098a60 2024-09-16T14:48:39.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:39.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:39.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:39.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 10 monclient(hunting): my global_id is 144241 2024-09-16T14:48:39.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 1 -- 172.21.15.189:0/3218948801 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b84003530 con 0x7f2b9c098a60 2024-09-16T14:48:39.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 1 -- 172.21.15.189:0/3218948801 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3127253312 0 0) 0x7f2b84003530 con 0x7f2b9c098a60 2024-09-16T14:48:39.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 1 -- 172.21.15.189:0/3218948801 >> v1:172.21.15.26:6790/0 conn(0x7f2b9c0a0200 legacy=0x7f2b9c13d440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:39.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 1 -- 172.21.15.189:0/3218948801 >> v1:172.21.15.26:6789/0 conn(0x7f2b9c09f710 legacy=0x7f2b9c09faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:39.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:39.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 1 monclient: found mon.noname-c 2024-09-16T14:48:39.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:39.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:39.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 1 -- 172.21.15.189:0/3218948801 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c099490 con 0x7f2b9c098a60 2024-09-16T14:48:39.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 10 monclient: _finish_auth 0 2024-09-16T14:48:39.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:39.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:39.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 1 -- 172.21.15.189:0/3218948801 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b90001620 con 0x7f2b9c098a60 2024-09-16T14:48:39.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:39.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:39.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 10 monclient: dump: 2024-09-16T14:48:39.371 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:39.371 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:39.371 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:39.371 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:39.371 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:39.371 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:39.371 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:39.371 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:39.371 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:39.371 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:39.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 5 monclient: authenticate success, global_id 144241 2024-09-16T14:48:39.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:39.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 1 -- 172.21.15.189:0/3218948801 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b90001940 con 0x7f2b9c098a60 2024-09-16T14:48:39.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:39.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:39.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 4 set_mon_vals no callback set 2024-09-16T14:48:39.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 1 -- 172.21.15.189:0/3218948801 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b90001d10 con 0x7f2b9c098a60 2024-09-16T14:48:39.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:39.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:39.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2b9affd640 10 monclient: dump: 2024-09-16T14:48:39.372 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:39.372 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:39.372 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:39.373 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:39.373 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:39.373 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:39.373 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:39.373 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:39.373 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:39.373 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:39.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 10 monclient: shutdown 2024-09-16T14:48:39.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:39.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 1 -- 172.21.15.189:0/3218948801 >> v1:172.21.15.189:6789/0 conn(0x7f2b9c098a60 legacy=0x7f2b9c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:39.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 1 -- 172.21.15.189:0/3218948801 shutdown_connections 2024-09-16T14:48:39.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 1 -- 172.21.15.189:0/3218948801 >> 172.21.15.189:0/3218948801 conn(0x7f2b9c0a4410 msgr2=0x7f2b9c0a0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:39.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 1 -- 172.21.15.189:0/3218948801 shutdown_connections 2024-09-16T14:48:39.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 1 -- 172.21.15.189:0/3218948801 wait complete. 2024-09-16T14:48:39.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 10 monclient: build_initial_monmap 2024-09-16T14:48:39.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.361+0000 7f2ba2644640 10 monclient: monmap: 2024-09-16T14:48:39.374 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:39.374 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:39.374 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:39.374 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:39.374 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:39.374 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:39.374 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:39.374 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:39.374 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:39.374 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:39.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 1 Processor -- start 2024-09-16T14:48:39.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 1 -- start start 2024-09-16T14:48:39.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 10 monclient: init 2024-09-16T14:48:39.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:39.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:39.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 10 monclient(hunting): picked mon.noname-c con 0x7f2b9c098a60 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:39.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 10 monclient(hunting): picked mon.noname-a con 0x7f2b9c09f710 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:39.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 10 monclient(hunting): picked mon.noname-b con 0x7f2b9c0a0200 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:39.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c09b790 con 0x7f2b9c09f710 2024-09-16T14:48:39.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c09bce0 con 0x7f2b9c0a0200 2024-09-16T14:48:39.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c06c3a0 con 0x7f2b9c098a60 2024-09-16T14:48:39.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:39.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 10 monclient(hunting): authenticate will time out at 1301.480609s 2024-09-16T14:48:39.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b9bfff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2b9c098a60 legacy=0x7f2b9c1348c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:39.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba0be1640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2b9c0a0200 0x7f2b9c1d1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57518/0 (socket says 172.21.15.189:57518) 2024-09-16T14:48:39.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba0be1640 1 -- 172.21.15.189:0/1378909844 learned_addr learned my addr 172.21.15.189:0/1378909844 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:39.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612937989 0 0) 0x7f2b9c09bce0 con 0x7f2b9c0a0200 2024-09-16T14:48:39.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:39.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:39.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:39.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient(hunting): my global_id is 144244 2024-09-16T14:48:39.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b88003530 con 0x7f2b9c0a0200 2024-09-16T14:48:39.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4233810140 0 0) 0x7f2b88003530 con 0x7f2b9c0a0200 2024-09-16T14:48:39.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b9c09bce0 con 0x7f2b9c0a0200 2024-09-16T14:48:39.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b94004450 con 0x7f2b9c0a0200 2024-09-16T14:48:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient(hunting): dump: 2024-09-16T14:48:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:39.377 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:39.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:39.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800904858 0 0) 0x7f2b9c09b790 con 0x7f2b9c09f710 2024-09-16T14:48:39.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:39.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:39.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:39.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient(hunting): my global_id is 124446 2024-09-16T14:48:39.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b88003530 con 0x7f2b9c09f710 2024-09-16T14:48:39.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2254454099 0 0) 0x7f2b9c09bce0 con 0x7f2b9c0a0200 2024-09-16T14:48:39.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 >> v1:172.21.15.26:6790/0 conn(0x7f2b9c098a60 legacy=0x7f2b9c1348c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:39.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 >> v1:172.21.15.26:6789/0 conn(0x7f2b9c09f710 legacy=0x7f2b9c1ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:39.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:48:39.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 1 monclient: found mon.b 2024-09-16T14:48:39.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:39.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c121fa0 con 0x7f2b9c0a0200 2024-09-16T14:48:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient: _finish_auth 0 2024-09-16T14:48:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:48:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 5 monclient: authenticate success, global_id 144244 2024-09-16T14:48:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 10 monclient: _renew_subs 2024-09-16T14:48:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 1 -- 172.21.15.189:0/1378909844 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9c122280 con 0x7f2b9c0a0200 2024-09-16T14:48:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 10 monclient: _renew_subs 2024-09-16T14:48:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 1 -- 172.21.15.189:0/1378909844 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9c134fd0 con 0x7f2b9c0a0200 2024-09-16T14:48:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b94004770 con 0x7f2b9c0a0200 2024-09-16T14:48:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b94004b00 con 0x7f2b9c0a0200 2024-09-16T14:48:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 10 monclient: dump: 2024-09-16T14:48:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:39.381 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:39.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b9a7fc640 4 set_mon_vals no callback set 2024-09-16T14:48:39.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2b94004c60 con 0x7f2b9c0a0200 2024-09-16T14:48:39.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:39.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:39.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:39.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(159..159 src has 73..159) v4 ==== 9013+0+0 (unknown 3655967767 0 0) 0x7f2b94044a30 con 0x7f2b9c0a0200 2024-09-16T14:48:39.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2ba2644640 1 -- 172.21.15.189:0/1378909844 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b7c005050 con 0x7f2b9c0a0200 2024-09-16T14:48:39.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.365+0000 7f2b9bfff640 10 monclient: get_auth_request con 0x7f2b88036130 auth_method 0 2024-09-16T14:48:39.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.369+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2b94005460 con 0x7f2b9c0a0200 2024-09-16T14:48:39.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.369+0000 7f2b98ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:39.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.369+0000 7f2b98ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:39.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.541+0000 7f2ba2644640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T14:48:39.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.541+0000 7f2ba2644640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T14:48:39.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.541+0000 7f2ba2644640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:39.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:39.541+0000 7f2ba2644640 1 -- 172.21.15.189:0/1378909844 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]} v 0) v1 -- 0x7f2b7c005300 con 0x7f2b9c0a0200 2024-09-16T14:48:40.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:40.025+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}]=0 v160) v1 ==== 118+0+0 (unknown 1854454076 0 0) 0x7f2b94012f20 con 0x7f2b9c0a0200 2024-09-16T14:48:40.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:40.025+0000 7f2b98ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T14:48:40.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:40.025+0000 7f2b98ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:40.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:40.133+0000 7f2ba2644640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T14:48:40.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:40.133+0000 7f2ba2644640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T14:48:40.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:40.133+0000 7f2ba2644640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:40.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:40.133+0000 7f2ba2644640 1 -- 172.21.15.189:0/1378909844 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]} v 0) v1 -- 0x7f2b7c005510 con 0x7f2b9c0a0200 2024-09-16T14:48:41.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.033+0000 7f2b98ff9640 1 -- 172.21.15.189:0/1378909844 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}]=0 v161) v1 ==== 118+0+0 (unknown 745091610 0 0) 0x7f2b940146a0 con 0x7f2b9c0a0200 2024-09-16T14:48:41.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.033+0000 7f2b98ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T14:48:41.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.033+0000 7f2b98ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:41.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.037+0000 7f2ba2644640 1 -- 172.21.15.189:0/1378909844 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2b88036130 legacy=0x7f2b880385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:41.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.037+0000 7f2ba2644640 10 monclient: shutdown 2024-09-16T14:48:41.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.037+0000 7f2ba2644640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:41.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.037+0000 7f2ba2644640 1 -- 172.21.15.189:0/1378909844 >> v1:172.21.15.189:6789/0 conn(0x7f2b9c0a0200 legacy=0x7f2b9c1d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:41.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.037+0000 7f2ba2644640 1 -- 172.21.15.189:0/1378909844 shutdown_connections 2024-09-16T14:48:41.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.037+0000 7f2ba2644640 1 -- 172.21.15.189:0/1378909844 >> 172.21.15.189:0/1378909844 conn(0x7f2b9c0a4410 msgr2=0x7f2b9c09d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:41.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.037+0000 7f2ba2644640 1 -- 172.21.15.189:0/1378909844 shutdown_connections 2024-09-16T14:48:41.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.037+0000 7f2ba2644640 1 -- 172.21.15.189:0/1378909844 wait complete. 2024-09-16T14:48:41.060 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight cool osd.0 8.5 2024-09-16T14:48:41.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 10 monclient: get_monmap_and_config 2024-09-16T14:48:41.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 10 monclient: build_initial_monmap 2024-09-16T14:48:41.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 10 monclient: monmap: 2024-09-16T14:48:41.182 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:41.183 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:41.183 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:41.182711+0000 2024-09-16T14:48:41.183 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:41.182711+0000 2024-09-16T14:48:41.183 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:41.183 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:41.183 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:41.183 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:41.183 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:41.184 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:41.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 1 Processor -- start 2024-09-16T14:48:41.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 1 -- start start 2024-09-16T14:48:41.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 10 monclient: init 2024-09-16T14:48:41.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:41.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:48:41.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 10 monclient(hunting): picked mon.noname-a con 0x7f311c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:41.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 10 monclient(hunting): picked mon.noname-c con 0x7f311c09c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:41.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 10 monclient(hunting): picked mon.noname-b con 0x7f311c09e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:41.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c099e50 con 0x7f311c099a70 2024-09-16T14:48:41.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c09dbd0 con 0x7f311c09c550 2024-09-16T14:48:41.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c09ee20 con 0x7f311c09e6d0 2024-09-16T14:48:41.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:41.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 10 monclient(hunting): authenticate will time out at 1303.298460s 2024-09-16T14:48:41.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311bfff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f311c09e6d0 legacy=0x7f311c13b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:41.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311affd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f311c09c550 0x7f311c1379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57522/0 (socket says 172.21.15.189:57522) 2024-09-16T14:48:41.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311affd640 1 -- 172.21.15.189:0/3409955857 learned_addr learned my addr 172.21.15.189:0/3409955857 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:41.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 1 -- 172.21.15.189:0/3409955857 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922471481 0 0) 0x7f311c09dbd0 con 0x7f311c09c550 2024-09-16T14:48:41.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:41.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:41.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:41.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 10 monclient(hunting): my global_id is 144247 2024-09-16T14:48:41.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 1 -- 172.21.15.189:0/3409955857 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3104003530 con 0x7f311c09c550 2024-09-16T14:48:41.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 1 -- 172.21.15.189:0/3409955857 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2122560966 0 0) 0x7f3104003530 con 0x7f311c09c550 2024-09-16T14:48:41.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 1 -- 172.21.15.189:0/3409955857 >> v1:172.21.15.26:6790/0 conn(0x7f311c09e6d0 legacy=0x7f311c13b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:41.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 1 -- 172.21.15.189:0/3409955857 >> v1:172.21.15.26:6789/0 conn(0x7f311c099a70 legacy=0x7f311c09dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:41.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:41.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 1 monclient: found mon.noname-c 2024-09-16T14:48:41.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:41.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:41.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 1 -- 172.21.15.189:0/3409955857 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c09f100 con 0x7f311c09c550 2024-09-16T14:48:41.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:41.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:41.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:41.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 1 -- 172.21.15.189:0/3409955857 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3110001620 con 0x7f311c09c550 2024-09-16T14:48:41.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 5 monclient: authenticate success, global_id 144247 2024-09-16T14:48:41.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:41.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:41.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:41.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 10 monclient: dump: 2024-09-16T14:48:41.194 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:41.194 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:41.194 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:41.194 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:41.194 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:41.194 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:41.194 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:41.194 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:41.194 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:41.194 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:41.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:41.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 1 -- 172.21.15.189:0/3409955857 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3110001840 con 0x7f311c09c550 2024-09-16T14:48:41.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:41.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 1 -- 172.21.15.189:0/3409955857 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3110001bd0 con 0x7f311c09c550 2024-09-16T14:48:41.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:41.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:41.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f311a7fc640 10 monclient: dump: 2024-09-16T14:48:41.195 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:41.195 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:41.195 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:41.195 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:41.195 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:41.195 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:41.195 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:41.195 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:41.195 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:41.195 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:41.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:41.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 4 set_mon_vals no callback set 2024-09-16T14:48:41.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 10 monclient: shutdown 2024-09-16T14:48:41.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:41.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 1 -- 172.21.15.189:0/3409955857 >> v1:172.21.15.189:6789/0 conn(0x7f311c09c550 legacy=0x7f311c1379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:41.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 1 -- 172.21.15.189:0/3409955857 shutdown_connections 2024-09-16T14:48:41.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 1 -- 172.21.15.189:0/3409955857 >> 172.21.15.189:0/3409955857 conn(0x7f311c0a3380 msgr2=0x7f311c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:41.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 1 -- 172.21.15.189:0/3409955857 shutdown_connections 2024-09-16T14:48:41.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.181+0000 7f3121ab2640 1 -- 172.21.15.189:0/3409955857 wait complete. 2024-09-16T14:48:41.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 monclient: build_initial_monmap 2024-09-16T14:48:41.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 monclient: monmap: 2024-09-16T14:48:41.196 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:41.196 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:41.197 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:41.197 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:41.197 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:41.197 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:41.197 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:41.197 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:41.197 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:41.197 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:41.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 1 Processor -- start 2024-09-16T14:48:41.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 1 -- start start 2024-09-16T14:48:41.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 monclient: init 2024-09-16T14:48:41.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:41.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:41.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 monclient(hunting): picked mon.noname-c con 0x7f311c099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:41.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 monclient(hunting): picked mon.noname-a con 0x7f311c09c550 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:41.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 monclient(hunting): picked mon.noname-b con 0x7f311c09e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:41.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c09cf20 con 0x7f311c09c550 2024-09-16T14:48:41.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c06ce40 con 0x7f311c09e6d0 2024-09-16T14:48:41.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c11e600 con 0x7f311c099a70 2024-09-16T14:48:41.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:41.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 monclient(hunting): authenticate will time out at 1303.301466s 2024-09-16T14:48:41.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f311b7fe640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f311c099a70 legacy=0x7f311c139da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:41.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f311bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f311c09e6d0 0x7f311c13abc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57534/0 (socket says 172.21.15.189:57534) 2024-09-16T14:48:41.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f311bfff640 1 -- 172.21.15.189:0/4197220438 learned_addr learned my addr 172.21.15.189:0/4197220438 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:41.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479312604 0 0) 0x7f311c06ce40 con 0x7f311c09e6d0 2024-09-16T14:48:41.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:41.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:41.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:41.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient(hunting): my global_id is 144250 2024-09-16T14:48:41.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3108003530 con 0x7f311c09e6d0 2024-09-16T14:48:41.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1375948254 0 0) 0x7f3108003530 con 0x7f311c09e6d0 2024-09-16T14:48:41.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f311c06ce40 con 0x7f311c09e6d0 2024-09-16T14:48:41.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3114004450 con 0x7f311c09e6d0 2024-09-16T14:48:41.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:41.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:41.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient(hunting): dump: 2024-09-16T14:48:41.199 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:41.199 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:41.200 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:41.200 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:41.200 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:41.200 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:41.200 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:41.200 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:41.200 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:41.200 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:41.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:41.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829699966 0 0) 0x7f311c09cf20 con 0x7f311c09c550 2024-09-16T14:48:41.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:41.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:41.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:41.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient(hunting): my global_id is 124452 2024-09-16T14:48:41.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3108003530 con 0x7f311c09c550 2024-09-16T14:48:41.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2350973769 0 0) 0x7f311c06ce40 con 0x7f311c09e6d0 2024-09-16T14:48:41.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 >> v1:172.21.15.26:6790/0 conn(0x7f311c099a70 legacy=0x7f311c139da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:41.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 >> v1:172.21.15.26:6789/0 conn(0x7f311c09c550 legacy=0x7f311c13a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:41.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:41.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 1 monclient: found mon.b 2024-09-16T14:48:41.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:41.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:41.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c11e8e0 con 0x7f311c09e6d0 2024-09-16T14:48:41.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient: _finish_auth 0 2024-09-16T14:48:41.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:41.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:41.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 5 monclient: authenticate success, global_id 144250 2024-09-16T14:48:41.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 monclient: _renew_subs 2024-09-16T14:48:41.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:41.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 1 -- 172.21.15.189:0/4197220438 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311c13b8c0 con 0x7f311c09e6d0 2024-09-16T14:48:41.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 monclient: _renew_subs 2024-09-16T14:48:41.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:41.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 1 -- 172.21.15.189:0/4197220438 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f311c11dca0 con 0x7f311c09e6d0 2024-09-16T14:48:41.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3114004670 con 0x7f311c09e6d0 2024-09-16T14:48:41.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:41.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f31140047e0 con 0x7f311c09e6d0 2024-09-16T14:48:41.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:41.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:41.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3119ffb640 4 set_mon_vals no callback set 2024-09-16T14:48:41.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 10 monclient: dump: 2024-09-16T14:48:41.203 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:41.203 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:41.203 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:41.203 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:41.203 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:41.203 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:41.203 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:41.203 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:41.203 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:41.203 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:41.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:41.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:41.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:41.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3121ab2640 1 -- 172.21.15.189:0/4197220438 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30fc005050 con 0x7f311c09e6d0 2024-09-16T14:48:41.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3114004940 con 0x7f311c09e6d0 2024-09-16T14:48:41.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(161..161 src has 73..161) v4 ==== 8951+0+0 (unknown 3563935496 0 0) 0x7f31140447b0 con 0x7f311c09e6d0 2024-09-16T14:48:41.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.185+0000 7f311b7fe640 10 monclient: get_auth_request con 0x7f3108036130 auth_method 0 2024-09-16T14:48:41.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.189+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3114005140 con 0x7f311c09e6d0 2024-09-16T14:48:41.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.193+0000 7f3103fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:41.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.193+0000 7f3103fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:41.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.357+0000 7f3121ab2640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T14:48:41.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.357+0000 7f3121ab2640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T14:48:41.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.357+0000 7f3121ab2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:41.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:41.357+0000 7f3121ab2640 1 -- 172.21.15.189:0/4197220438 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]} v 0) v1 -- 0x7f30fc005300 con 0x7f311c09e6d0 2024-09-16T14:48:42.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:42.041+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}]=0 v162) v1 ==== 127+0+0 (unknown 2663385207 0 0) 0x7f3114012c60 con 0x7f311c09e6d0 2024-09-16T14:48:42.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:42.041+0000 7f3103fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T14:48:42.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:42.041+0000 7f3103fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:42.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:42.149+0000 7f3121ab2640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T14:48:42.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:42.149+0000 7f3121ab2640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T14:48:42.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:42.149+0000 7f3121ab2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:42.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:42.149+0000 7f3121ab2640 1 -- 172.21.15.189:0/4197220438 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]} v 0) v1 -- 0x7f30fc005a40 con 0x7f311c09e6d0 2024-09-16T14:48:43.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.017+0000 7f3103fff640 1 -- 172.21.15.189:0/4197220438 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}]=0 v163) v1 ==== 127+0+0 (unknown 3858661279 0 0) 0x7f31140143e0 con 0x7f311c09e6d0 2024-09-16T14:48:43.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.017+0000 7f3103fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T14:48:43.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.017+0000 7f3103fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:43.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.021+0000 7f3121ab2640 1 -- 172.21.15.189:0/4197220438 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3108036130 legacy=0x7f31080385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:43.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.021+0000 7f3121ab2640 10 monclient: shutdown 2024-09-16T14:48:43.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.021+0000 7f3121ab2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:43.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.021+0000 7f3121ab2640 1 -- 172.21.15.189:0/4197220438 >> v1:172.21.15.189:6789/0 conn(0x7f311c09e6d0 legacy=0x7f311c13abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:43.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.021+0000 7f3121ab2640 1 -- 172.21.15.189:0/4197220438 shutdown_connections 2024-09-16T14:48:43.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.021+0000 7f3121ab2640 1 -- 172.21.15.189:0/4197220438 >> 172.21.15.189:0/4197220438 conn(0x7f311c0a3380 msgr2=0x7f311c0a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:43.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.021+0000 7f3121ab2640 1 -- 172.21.15.189:0/4197220438 shutdown_connections 2024-09-16T14:48:43.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.021+0000 7f3121ab2640 1 -- 172.21.15.189:0/4197220438 wait complete. 2024-09-16T14:48:43.041 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight cold osd.0 6.5 6.6 2024-09-16T14:48:43.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 10 monclient: get_monmap_and_config 2024-09-16T14:48:43.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 10 monclient: build_initial_monmap 2024-09-16T14:48:43.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 10 monclient: monmap: 2024-09-16T14:48:43.170 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:43.170 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:43.170 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:43.169943+0000 2024-09-16T14:48:43.170 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:43.169943+0000 2024-09-16T14:48:43.170 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:43.170 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:43.170 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:43.170 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:43.170 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:43.170 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:43.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 1 Processor -- start 2024-09-16T14:48:43.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 1 -- start start 2024-09-16T14:48:43.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 10 monclient: init 2024-09-16T14:48:43.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:43.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:43.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 10 monclient(hunting): picked mon.noname-c con 0x7f34f4099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:43.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 10 monclient(hunting): picked mon.noname-a con 0x7f34f409ff00 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:43.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 10 monclient(hunting): picked mon.noname-b con 0x7f34f40a0310 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:43.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f409b380 con 0x7f34f409ff00 2024-09-16T14:48:43.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f4099e50 con 0x7f34f4099a70 2024-09-16T14:48:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f409f2e0 con 0x7f34f40a0310 2024-09-16T14:48:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 10 monclient(hunting): authenticate will time out at 1305.285622s 2024-09-16T14:48:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f37fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f34f4099a70 0x7f34f409bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57550/0 (socket says 172.21.15.189:57550) 2024-09-16T14:48:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f37fe640 1 -- 172.21.15.189:0/3401251573 learned_addr learned my addr 172.21.15.189:0/3401251573 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f3fff640 1 -- 172.21.15.189:0/3401251573 >> v1:172.21.15.26:6790/0 conn(0x7f34f40a0310 legacy=0x7f34f4139d20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 1 -- 172.21.15.189:0/3401251573 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505141856 0 0) 0x7f34f4099e50 con 0x7f34f4099a70 2024-09-16T14:48:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 10 monclient(hunting): my global_id is 144253 2024-09-16T14:48:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 1 -- 172.21.15.189:0/3401251573 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34dc003530 con 0x7f34f4099a70 2024-09-16T14:48:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 1 -- 172.21.15.189:0/3401251573 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 411247511 0 0) 0x7f34dc003530 con 0x7f34f4099a70 2024-09-16T14:48:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 1 -- 172.21.15.189:0/3401251573 >> v1:172.21.15.26:6790/0 conn(0x7f34f40a0310 legacy=0x7f34f4139d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 1 -- 172.21.15.189:0/3401251573 >> v1:172.21.15.26:6789/0 conn(0x7f34f409ff00 legacy=0x7f34f41365f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 1 monclient: found mon.noname-c 2024-09-16T14:48:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 1 -- 172.21.15.189:0/3401251573 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f409b070 con 0x7f34f4099a70 2024-09-16T14:48:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 1 -- 172.21.15.189:0/3401251573 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34ec0019a0 con 0x7f34f4099a70 2024-09-16T14:48:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:43.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 10 monclient: dump: 2024-09-16T14:48:43.179 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:43.179 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:43.179 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:43.179 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:43.179 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:43.179 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:43.179 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:43.179 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:43.179 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:43.179 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:43.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 5 monclient: authenticate success, global_id 144253 2024-09-16T14:48:43.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:43.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 1 -- 172.21.15.189:0/3401251573 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34ec001cc0 con 0x7f34f4099a70 2024-09-16T14:48:43.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:43.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 1 -- 172.21.15.189:0/3401251573 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34ec002090 con 0x7f34f4099a70 2024-09-16T14:48:43.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:43.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 4 set_mon_vals no callback set 2024-09-16T14:48:43.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:43.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:43.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f27fc640 10 monclient: dump: 2024-09-16T14:48:43.180 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:43.180 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:43.180 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:43.180 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:43.180 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:43.181 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 10 monclient: shutdown 2024-09-16T14:48:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 1 -- 172.21.15.189:0/3401251573 >> v1:172.21.15.189:6789/0 conn(0x7f34f4099a70 legacy=0x7f34f409bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 1 -- 172.21.15.189:0/3401251573 shutdown_connections 2024-09-16T14:48:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 1 -- 172.21.15.189:0/3401251573 >> 172.21.15.189:0/3401251573 conn(0x7f34f40a3320 msgr2=0x7f34f409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 1 -- 172.21.15.189:0/3401251573 shutdown_connections 2024-09-16T14:48:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 1 -- 172.21.15.189:0/3401251573 wait complete. 2024-09-16T14:48:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 10 monclient: build_initial_monmap 2024-09-16T14:48:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.169+0000 7f34f9ed4640 10 monclient: monmap: 2024-09-16T14:48:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:43.182 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 1 Processor -- start 2024-09-16T14:48:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 1 -- start start 2024-09-16T14:48:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 10 monclient: init 2024-09-16T14:48:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 10 monclient(hunting): picked mon.noname-a con 0x7f34f4099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 10 monclient(hunting): picked mon.noname-c con 0x7f34f409ff00 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 10 monclient(hunting): picked mon.noname-b con 0x7f34f40a0310 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f411df50 con 0x7f34f4099a70 2024-09-16T14:48:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f411d750 con 0x7f34f40a0310 2024-09-16T14:48:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f411eb50 con 0x7f34f409ff00 2024-09-16T14:48:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 10 monclient(hunting): authenticate will time out at 1305.288666s 2024-09-16T14:48:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f3fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f34f40a0310 0x7f34f41d1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57566/0 (socket says 172.21.15.189:57566) 2024-09-16T14:48:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f3fff640 1 -- 172.21.15.189:0/3041293429 learned_addr learned my addr 172.21.15.189:0/3041293429 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f2ffd640 1 -- 172.21.15.189:0/3041293429 >> v1:172.21.15.26:6790/0 conn(0x7f34f409ff00 legacy=0x7f34f41ce7b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:43.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 1 -- 172.21.15.189:0/3041293429 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990399630 0 0) 0x7f34f411d750 con 0x7f34f40a0310 2024-09-16T14:48:43.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:43.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:43.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:43.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 10 monclient(hunting): my global_id is 144256 2024-09-16T14:48:43.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 1 -- 172.21.15.189:0/3041293429 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34e0003530 con 0x7f34f40a0310 2024-09-16T14:48:43.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 1 -- 172.21.15.189:0/3041293429 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3507813350 0 0) 0x7f34e0003530 con 0x7f34f40a0310 2024-09-16T14:48:43.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 1 -- 172.21.15.189:0/3041293429 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34f411d750 con 0x7f34f40a0310 2024-09-16T14:48:43.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 1 -- 172.21.15.189:0/3041293429 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34e40040d0 con 0x7f34f40a0310 2024-09-16T14:48:43.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:43.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:43.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 10 monclient(hunting): dump: 2024-09-16T14:48:43.184 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:43.185 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:43.185 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:43.185 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:43.185 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:43.185 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:43.185 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:43.185 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:43.185 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:43.185 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:43.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:43.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 1 -- 172.21.15.189:0/3041293429 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 303347901 0 0) 0x7f34f411d750 con 0x7f34f40a0310 2024-09-16T14:48:43.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 1 -- 172.21.15.189:0/3041293429 >> v1:172.21.15.26:6790/0 conn(0x7f34f409ff00 legacy=0x7f34f41ce7b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:43.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 1 -- 172.21.15.189:0/3041293429 >> v1:172.21.15.26:6789/0 conn(0x7f34f4099a70 legacy=0x7f34f41c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:43.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:43.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 1 monclient: found mon.b 2024-09-16T14:48:43.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:43.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:43.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 1 -- 172.21.15.189:0/3041293429 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f406fd30 con 0x7f34f40a0310 2024-09-16T14:48:43.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 10 monclient: _finish_auth 0 2024-09-16T14:48:43.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:43.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:43.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 5 monclient: authenticate success, global_id 144256 2024-09-16T14:48:43.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 10 monclient: _renew_subs 2024-09-16T14:48:43.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:43.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 1 -- 172.21.15.189:0/3041293429 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f411e590 con 0x7f34f40a0310 2024-09-16T14:48:43.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 10 monclient: _renew_subs 2024-09-16T14:48:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 1 -- 172.21.15.189:0/3041293429 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f41346d0 con 0x7f34f40a0310 2024-09-16T14:48:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 1 -- 172.21.15.189:0/3041293429 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34e40043f0 con 0x7f34f40a0310 2024-09-16T14:48:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 1 -- 172.21.15.189:0/3041293429 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34e4004780 con 0x7f34f40a0310 2024-09-16T14:48:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f1ffb640 4 set_mon_vals no callback set 2024-09-16T14:48:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 10 monclient: dump: 2024-09-16T14:48:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:43.188 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:43.188 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:43.188 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:43.188 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:43.188 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:43.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:43.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:43.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:43.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 1 -- 172.21.15.189:0/3041293429 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f34e40048e0 con 0x7f34f40a0310 2024-09-16T14:48:43.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f9ed4640 1 -- 172.21.15.189:0/3041293429 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d4005050 con 0x7f34f40a0310 2024-09-16T14:48:43.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34dbfff640 1 -- 172.21.15.189:0/3041293429 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(163..163 src has 73..163) v4 ==== 8951+0+0 (unknown 3407489443 0 0) 0x7f34e40445f0 con 0x7f34f40a0310 2024-09-16T14:48:43.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.173+0000 7f34f37fe640 10 monclient: get_auth_request con 0x7f34e00360a0 auth_method 0 2024-09-16T14:48:43.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.177+0000 7f34dbfff640 1 -- 172.21.15.189:0/3041293429 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f34e40050e0 con 0x7f34f40a0310 2024-09-16T14:48:43.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.177+0000 7f34dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:43.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.177+0000 7f34dbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:43.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.349+0000 7f34f9ed4640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T14:48:43.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.349+0000 7f34f9ed4640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T14:48:43.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.349+0000 7f34f9ed4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:43.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:43.349+0000 7f34f9ed4640 1 -- 172.21.15.189:0/3041293429 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]} v 0) v1 -- 0x7f34d4005300 con 0x7f34f40a0310 2024-09-16T14:48:44.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:44.057+0000 7f34dbfff640 1 -- 172.21.15.189:0/3041293429 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v164) v1 ==== 132+0+0 (unknown 4266501022 0 0) 0x7f34e4014280 con 0x7f34f40a0310 2024-09-16T14:48:44.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:44.057+0000 7f34dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T14:48:44.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:44.057+0000 7f34dbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:44.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:44.165+0000 7f34f9ed4640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T14:48:44.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:44.165+0000 7f34f9ed4640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T14:48:44.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:44.165+0000 7f34f9ed4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:44.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:44.165+0000 7f34f9ed4640 1 -- 172.21.15.189:0/3041293429 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]} v 0) v1 -- 0x7f34d4005a40 con 0x7f34f40a0310 2024-09-16T14:48:45.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.065+0000 7f34dbfff640 1 -- 172.21.15.189:0/3041293429 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v165) v1 ==== 132+0+0 (unknown 2604274139 0 0) 0x7f34e4014460 con 0x7f34f40a0310 2024-09-16T14:48:45.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.065+0000 7f34dbfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T14:48:45.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.065+0000 7f34dbfff640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:45.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.069+0000 7f34f9ed4640 1 -- 172.21.15.189:0/3041293429 >> v1:172.21.15.189:6812/3196161582 conn(0x7f34e00360a0 legacy=0x7f34e0038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:45.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.069+0000 7f34f9ed4640 10 monclient: shutdown 2024-09-16T14:48:45.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.069+0000 7f34f9ed4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:45.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.069+0000 7f34f9ed4640 1 -- 172.21.15.189:0/3041293429 >> v1:172.21.15.189:6789/0 conn(0x7f34f40a0310 legacy=0x7f34f41d1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:45.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.069+0000 7f34f9ed4640 1 -- 172.21.15.189:0/3041293429 shutdown_connections 2024-09-16T14:48:45.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.069+0000 7f34f9ed4640 1 -- 172.21.15.189:0/3041293429 >> 172.21.15.189:0/3041293429 conn(0x7f34f40a3320 msgr2=0x7f34f4070ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:45.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.069+0000 7f34f9ed4640 1 -- 172.21.15.189:0/3041293429 shutdown_connections 2024-09-16T14:48:45.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.069+0000 7f34f9ed4640 1 -- 172.21.15.189:0/3041293429 wait complete. 2024-09-16T14:48:45.092 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T14:48:45.092 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T14:48:45.092 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep globster 2024-09-16T14:48:45.092 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '7\.' 2024-09-16T14:48:45.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.205+0000 7fadac8cb640 10 monclient: get_monmap_and_config 2024-09-16T14:48:45.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.205+0000 7fadac8cb640 10 monclient: build_initial_monmap 2024-09-16T14:48:45.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.205+0000 7fadac8cb640 10 monclient: monmap: 2024-09-16T14:48:45.208 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:45.208 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:45.208 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:45.208372+0000 2024-09-16T14:48:45.208 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:45.208372+0000 2024-09-16T14:48:45.208 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:45.208 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:45.209 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:45.209 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:45.209 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:45.209 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:45.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.205+0000 7fadac8cb640 1 Processor -- start 2024-09-16T14:48:45.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.205+0000 7fadac8cb640 1 -- start start 2024-09-16T14:48:45.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.205+0000 7fadac8cb640 10 monclient: init 2024-09-16T14:48:45.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.205+0000 7fadac8cb640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:45.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.205+0000 7fadac8cb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:45.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.205+0000 7fadac8cb640 10 monclient(hunting): picked mon.noname-a con 0x7fada8132aa0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:45.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.205+0000 7fadac8cb640 10 monclient(hunting): picked mon.noname-b con 0x7fada8132440 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:45.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.205+0000 7fadac8cb640 10 monclient(hunting): picked mon.noname-c con 0x7fada80996c0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:45.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.205+0000 7fadac8cb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada8099000 con 0x7fada8132aa0 2024-09-16T14:48:45.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.205+0000 7fadac8cb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada809f420 con 0x7fada80996c0 2024-09-16T14:48:45.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.205+0000 7fadac8cb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada80711b0 con 0x7fada8132440 2024-09-16T14:48:45.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.205+0000 7fadac8cb640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:45.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.205+0000 7fadac8cb640 10 monclient(hunting): authenticate will time out at 1307.323917s 2024-09-16T14:48:45.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada5d9b640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fada8132440 legacy=0x7fada809ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:45.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada6d9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fada80996c0 0x7fada813b790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57572/0 (socket says 172.21.15.189:57572) 2024-09-16T14:48:45.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada6d9d640 1 -- 172.21.15.189:0/2127353916 learned_addr learned my addr 172.21.15.189:0/2127353916 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:45.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 1 -- 172.21.15.189:0/2127353916 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538452584 0 0) 0x7fada809f420 con 0x7fada80996c0 2024-09-16T14:48:45.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 10 monclient(hunting): _negotiate 2024-09-16T14:48:45.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:45.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:45.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 10 monclient(hunting): my global_id is 144259 2024-09-16T14:48:45.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 1 -- 172.21.15.189:0/2127353916 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad90003530 con 0x7fada80996c0 2024-09-16T14:48:45.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 1 -- 172.21.15.189:0/2127353916 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3274967827 0 0) 0x7fad90003530 con 0x7fada80996c0 2024-09-16T14:48:45.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 1 -- 172.21.15.189:0/2127353916 >> v1:172.21.15.26:6790/0 conn(0x7fada8132440 legacy=0x7fada809ed10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:45.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 1 -- 172.21.15.189:0/2127353916 >> v1:172.21.15.26:6789/0 conn(0x7fada8132aa0 legacy=0x7fada809e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:45.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 10 monclient: _finish_hunting 0 2024-09-16T14:48:45.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 1 monclient: found mon.noname-c 2024-09-16T14:48:45.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:45.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:45.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 1 -- 172.21.15.189:0/2127353916 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada809c900 con 0x7fada80996c0 2024-09-16T14:48:45.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 10 monclient: _finish_auth 0 2024-09-16T14:48:45.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 10 monclient: _check_auth_tickets 2024-09-16T14:48:45.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:45.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 1 -- 172.21.15.189:0/2127353916 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fada0001620 con 0x7fada80996c0 2024-09-16T14:48:45.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 5 monclient: authenticate success, global_id 144259 2024-09-16T14:48:45.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:45.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:45.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:45.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 10 monclient: dump: 2024-09-16T14:48:45.220 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:45.220 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:45.220 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:45.220 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:45.220 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:45.220 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:45.220 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:45.220 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:45.220 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:45.221 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:45.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:45.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 1 -- 172.21.15.189:0/2127353916 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fada0001940 con 0x7fada80996c0 2024-09-16T14:48:45.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:45.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 1 -- 172.21.15.189:0/2127353916 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fada0001cd0 con 0x7fada80996c0 2024-09-16T14:48:45.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:45.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:45.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada559a640 10 monclient: dump: 2024-09-16T14:48:45.221 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:45.221 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:45.221 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:45.221 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:45.221 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:45.222 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:45.222 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:45.222 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:45.222 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:45.222 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:45.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:45.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 4 set_mon_vals no callback set 2024-09-16T14:48:45.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 10 monclient: shutdown 2024-09-16T14:48:45.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:45.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 1 -- 172.21.15.189:0/2127353916 >> v1:172.21.15.189:6789/0 conn(0x7fada80996c0 legacy=0x7fada813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:45.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 1 -- 172.21.15.189:0/2127353916 shutdown_connections 2024-09-16T14:48:45.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 1 -- 172.21.15.189:0/2127353916 >> 172.21.15.189:0/2127353916 conn(0x7fada80a32d0 msgr2=0x7fada809f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:45.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 1 -- 172.21.15.189:0/2127353916 shutdown_connections 2024-09-16T14:48:45.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 1 -- 172.21.15.189:0/2127353916 wait complete. 2024-09-16T14:48:45.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 10 monclient: build_initial_monmap 2024-09-16T14:48:45.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 10 monclient: monmap: 2024-09-16T14:48:45.223 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:45.223 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:45.223 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:45.223 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:45.223 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:45.223 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:45.223 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:45.223 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:45.223 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:45.223 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:45.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 1 Processor -- start 2024-09-16T14:48:45.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 1 -- start start 2024-09-16T14:48:45.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 10 monclient: init 2024-09-16T14:48:45.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:45.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:48:45.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 10 monclient(hunting): picked mon.noname-a con 0x7fada80996c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:45.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 10 monclient(hunting): picked mon.noname-b con 0x7fada8132440 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:45.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 10 monclient(hunting): picked mon.noname-c con 0x7fada8132aa0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:45.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada81208b0 con 0x7fada80996c0 2024-09-16T14:48:45.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada81214f0 con 0x7fada8132440 2024-09-16T14:48:45.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada8122930 con 0x7fada8132aa0 2024-09-16T14:48:45.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:45.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fadac8cb640 10 monclient(hunting): authenticate will time out at 1307.327502s 2024-09-16T14:48:45.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada5d9b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fada8132440 0x7fada81cc1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57582/0 (socket says 172.21.15.189:57582) 2024-09-16T14:48:45.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada5d9b640 1 -- 172.21.15.189:0/3984075413 learned_addr learned my addr 172.21.15.189:0/3984075413 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:45.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fada6d9d640 1 -- 172.21.15.189:0/3984075413 >> v1:172.21.15.26:6790/0 conn(0x7fada8132aa0 legacy=0x7fada81cf920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:45.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172154891 0 0) 0x7fada81214f0 con 0x7fada8132440 2024-09-16T14:48:45.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fad96ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:45.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fad96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:45.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fad96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:45.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fad96ffd640 10 monclient(hunting): my global_id is 144262 2024-09-16T14:48:45.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.209+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad88003530 con 0x7fada8132440 2024-09-16T14:48:45.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2146273828 0 0) 0x7fad88003530 con 0x7fada8132440 2024-09-16T14:48:45.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fada81214f0 con 0x7fada8132440 2024-09-16T14:48:45.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fad9c004120 con 0x7fada8132440 2024-09-16T14:48:45.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:45.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:45.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 10 monclient(hunting): dump: 2024-09-16T14:48:45.226 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:45.226 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:45.226 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:45.226 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:45.226 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:45.226 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:45.226 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:45.226 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:45.226 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:45.226 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:45.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:45.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761306723 0 0) 0x7fada81208b0 con 0x7fada80996c0 2024-09-16T14:48:45.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:45.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:45.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:45.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 10 monclient(hunting): my global_id is 124464 2024-09-16T14:48:45.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad88003530 con 0x7fada80996c0 2024-09-16T14:48:45.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 579614652 0 0) 0x7fada81214f0 con 0x7fada8132440 2024-09-16T14:48:45.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 >> v1:172.21.15.26:6790/0 conn(0x7fada8132aa0 legacy=0x7fada81cf920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:45.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 >> v1:172.21.15.26:6789/0 conn(0x7fada80996c0 legacy=0x7fada81c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:45.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:45.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 1 monclient: found mon.b 2024-09-16T14:48:45.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:45.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:45.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada8122c10 con 0x7fada8132440 2024-09-16T14:48:45.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 10 monclient: _finish_auth 0 2024-09-16T14:48:45.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:45.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:45.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fadac8cb640 5 monclient: authenticate success, global_id 144262 2024-09-16T14:48:45.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fadac8cb640 10 monclient: _renew_subs 2024-09-16T14:48:45.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fadac8cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:45.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fadac8cb640 1 -- 172.21.15.189:0/3984075413 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada80a2230 con 0x7fada8132440 2024-09-16T14:48:45.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fadac8cb640 10 monclient: _renew_subs 2024-09-16T14:48:45.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fadac8cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:45.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fadac8cb640 1 -- 172.21.15.189:0/3984075413 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fada8122650 con 0x7fada8132440 2024-09-16T14:48:45.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad9c004440 con 0x7fada8132440 2024-09-16T14:48:45.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:45.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fad9c0047d0 con 0x7fada8132440 2024-09-16T14:48:45.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:45.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:45.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fada4d99640 4 set_mon_vals no callback set 2024-09-16T14:48:45.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 10 monclient: dump: 2024-09-16T14:48:45.229 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:45.229 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:45.229 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:45.229 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:45.229 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:45.230 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:45.230 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:45.230 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:45.230 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:45.230 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:45.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fadac8cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:45.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fadac8cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:45.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fadac8cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:45.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fadac8cb640 1 -- 172.21.15.189:0/3984075413 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad8c005050 con 0x7fada8132440 2024-09-16T14:48:45.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fad9c004930 con 0x7fada8132440 2024-09-16T14:48:45.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(165..165 src has 73..165) v4 ==== 8951+0+0 (unknown 3934497374 0 0) 0x7fad9c0446d0 con 0x7fada8132440 2024-09-16T14:48:45.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.213+0000 7fada659c640 10 monclient: get_auth_request con 0x7fad88036130 auth_method 0 2024-09-16T14:48:45.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.217+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fad9c005130 con 0x7fada8132440 2024-09-16T14:48:45.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.217+0000 7fad96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:45.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.217+0000 7fad96ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:45.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.381+0000 7fadac8cb640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:45.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.381+0000 7fadac8cb640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:45.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.381+0000 7fadac8cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:45.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.381+0000 7fadac8cb640 1 -- 172.21.15.189:0/3984075413 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fad8c005300 con 0x7fada8132440 2024-09-16T14:48:45.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.381+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v165) v1 ==== 83+0+2320 (unknown 3080134077 0 3745721447) 0x7fad9c012be0 con 0x7fada8132440 2024-09-16T14:48:45.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.381+0000 7fad96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:45.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.381+0000 7fad96ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.485+0000 7fadac8cb640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.485+0000 7fadac8cb640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.485+0000 7fadac8cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.485+0000 7fadac8cb640 1 -- 172.21.15.189:0/3984075413 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fad8c0055a0 con 0x7fada8132440 2024-09-16T14:48:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.489+0000 7fad96ffd640 1 -- 172.21.15.189:0/3984075413 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v165) v1 ==== 83+0+2320 (unknown 3080134077 0 3745721447) 0x7fad9c014540 con 0x7fada8132440 2024-09-16T14:48:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.489+0000 7fad96ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.489+0000 7fad96ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.489+0000 7fadac8cb640 1 -- 172.21.15.189:0/3984075413 >> v1:172.21.15.189:6812/3196161582 conn(0x7fad88036130 legacy=0x7fad880385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.489+0000 7fadac8cb640 10 monclient: shutdown 2024-09-16T14:48:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.489+0000 7fadac8cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.489+0000 7fadac8cb640 1 -- 172.21.15.189:0/3984075413 >> v1:172.21.15.189:6789/0 conn(0x7fada8132440 legacy=0x7fada81cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.489+0000 7fadac8cb640 1 -- 172.21.15.189:0/3984075413 shutdown_connections 2024-09-16T14:48:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.489+0000 7fadac8cb640 1 -- 172.21.15.189:0/3984075413 >> 172.21.15.189:0/3984075413 conn(0x7fada80a32d0 msgr2=0x7fada80a13a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.489+0000 7fadac8cb640 1 -- 172.21.15.189:0/3984075413 shutdown_connections 2024-09-16T14:48:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.489+0000 7fadac8cb640 1 -- 172.21.15.189:0/3984075413 wait complete. 2024-09-16T14:48:45.506 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T14:48:45.506 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T14:48:45.507 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T14:48:45.507 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T14:48:45.507 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep globster 2024-09-16T14:48:45.507 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '8\.' 2024-09-16T14:48:45.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.617+0000 7f9d55fc5640 10 monclient: get_monmap_and_config 2024-09-16T14:48:45.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.617+0000 7f9d55fc5640 10 monclient: build_initial_monmap 2024-09-16T14:48:45.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.617+0000 7f9d55fc5640 10 monclient: monmap: 2024-09-16T14:48:45.620 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:45.620 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:45.620 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:45.620212+0000 2024-09-16T14:48:45.620 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:45.620212+0000 2024-09-16T14:48:45.620 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:45.620 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:45.620 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:45.621 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:45.621 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:45.621 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:45.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.617+0000 7f9d55fc5640 1 Processor -- start 2024-09-16T14:48:45.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.617+0000 7f9d55fc5640 1 -- start start 2024-09-16T14:48:45.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.617+0000 7f9d55fc5640 10 monclient: init 2024-09-16T14:48:45.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.617+0000 7f9d55fc5640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:45.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.617+0000 7f9d55fc5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:45.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.617+0000 7f9d55fc5640 10 monclient(hunting): picked mon.noname-c con 0x7f9d500a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:45.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.617+0000 7f9d55fc5640 10 monclient(hunting): picked mon.noname-a con 0x7f9d500a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:45.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.617+0000 7f9d55fc5640 10 monclient(hunting): picked mon.noname-b con 0x7f9d5013b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:45.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.617+0000 7f9d55fc5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d500a30b0 con 0x7f9d500a25f0 2024-09-16T14:48:45.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.617+0000 7f9d55fc5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d500a2e30 con 0x7f9d500a1b00 2024-09-16T14:48:45.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.617+0000 7f9d55fc5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5009eb50 con 0x7f9d5013b430 2024-09-16T14:48:45.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.617+0000 7f9d55fc5640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:45.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.617+0000 7f9d55fc5640 10 monclient(hunting): authenticate will time out at 1307.735903s 2024-09-16T14:48:45.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4f7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9d500a1b00 0x7f9d500a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57592/0 (socket says 172.21.15.189:57592) 2024-09-16T14:48:45.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4f7fe640 1 -- 172.21.15.189:0/105438117 learned_addr learned my addr 172.21.15.189:0/105438117 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:45.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4ffff640 1 -- 172.21.15.189:0/105438117 >> v1:172.21.15.26:6790/0 conn(0x7f9d5013b430 legacy=0x7f9d5013d820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:45.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 1 -- 172.21.15.189:0/105438117 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121483452 0 0) 0x7f9d500a2e30 con 0x7f9d500a1b00 2024-09-16T14:48:45.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:45.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:45.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:45.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 10 monclient(hunting): my global_id is 144265 2024-09-16T14:48:45.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 1 -- 172.21.15.189:0/105438117 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d38003530 con 0x7f9d500a1b00 2024-09-16T14:48:45.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 1 -- 172.21.15.189:0/105438117 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3793967500 0 0) 0x7f9d38003530 con 0x7f9d500a1b00 2024-09-16T14:48:45.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 1 -- 172.21.15.189:0/105438117 >> v1:172.21.15.26:6790/0 conn(0x7f9d5013b430 legacy=0x7f9d5013d820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:45.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 1 -- 172.21.15.189:0/105438117 >> v1:172.21.15.26:6789/0 conn(0x7f9d500a25f0 legacy=0x7f9d50139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:45.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:45.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 1 monclient: found mon.noname-c 2024-09-16T14:48:45.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:45.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:45.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 1 -- 172.21.15.189:0/105438117 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5009ee30 con 0x7f9d500a1b00 2024-09-16T14:48:45.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:45.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:45.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:45.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 5 monclient: authenticate success, global_id 144265 2024-09-16T14:48:45.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:45.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 1 -- 172.21.15.189:0/105438117 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9d40001620 con 0x7f9d500a1b00 2024-09-16T14:48:45.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:45.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:45.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 10 monclient: dump: 2024-09-16T14:48:45.630 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:45.631 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:45.631 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:45.631 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:45.631 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:45.631 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:45.631 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:45.631 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:45.631 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:45.631 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:45.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:45.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 1 -- 172.21.15.189:0/105438117 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d40001940 con 0x7f9d500a1b00 2024-09-16T14:48:45.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:45.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:45.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 4 set_mon_vals no callback set 2024-09-16T14:48:45.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 1 -- 172.21.15.189:0/105438117 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9d40001cd0 con 0x7f9d500a1b00 2024-09-16T14:48:45.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:45.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:45.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4e7fc640 10 monclient: dump: 2024-09-16T14:48:45.632 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:45.632 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:45.632 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:45.632 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:45.632 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:45.633 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:45.633 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:45.633 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:45.633 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:45.633 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:45.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 10 monclient: shutdown 2024-09-16T14:48:45.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:45.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 1 -- 172.21.15.189:0/105438117 >> v1:172.21.15.189:6789/0 conn(0x7f9d500a1b00 legacy=0x7f9d500a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:45.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 1 -- 172.21.15.189:0/105438117 shutdown_connections 2024-09-16T14:48:45.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 1 -- 172.21.15.189:0/105438117 >> 172.21.15.189:0/105438117 conn(0x7f9d500a7dd0 msgr2=0x7f9d500a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:45.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 1 -- 172.21.15.189:0/105438117 shutdown_connections 2024-09-16T14:48:45.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 1 -- 172.21.15.189:0/105438117 wait complete. 2024-09-16T14:48:45.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 10 monclient: build_initial_monmap 2024-09-16T14:48:45.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 10 monclient: monmap: 2024-09-16T14:48:45.634 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:45.634 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:45.634 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:45.634 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:45.634 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:45.634 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:45.634 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:45.634 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:45.634 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:45.634 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:45.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 1 Processor -- start 2024-09-16T14:48:45.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 1 -- start start 2024-09-16T14:48:45.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 10 monclient: init 2024-09-16T14:48:45.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:45.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:45.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 10 monclient(hunting): picked mon.noname-c con 0x7f9d500a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:45.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 10 monclient(hunting): picked mon.noname-a con 0x7f9d500a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:45.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 10 monclient(hunting): picked mon.noname-b con 0x7f9d5013b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:45.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5011fee0 con 0x7f9d500a25f0 2024-09-16T14:48:45.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d500a5c90 con 0x7f9d5013b430 2024-09-16T14:48:45.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5013b060 con 0x7f9d500a1b00 2024-09-16T14:48:45.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:45.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 10 monclient(hunting): authenticate will time out at 1307.738728s 2024-09-16T14:48:45.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4ffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9d5013b430 0x7f9d501d1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57596/0 (socket says 172.21.15.189:57596) 2024-09-16T14:48:45.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4ffff640 1 -- 172.21.15.189:0/3986184858 learned_addr learned my addr 172.21.15.189:0/3986184858 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:45.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4f7fe640 1 -- 172.21.15.189:0/3986184858 >> v1:172.21.15.26:6790/0 conn(0x7f9d500a1b00 legacy=0x7f9d501c9810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:45.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025919063 0 0) 0x7f9d500a5c90 con 0x7f9d5013b430 2024-09-16T14:48:45.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:45.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:45.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:45.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient(hunting): my global_id is 144268 2024-09-16T14:48:45.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d3c003530 con 0x7f9d5013b430 2024-09-16T14:48:45.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1319525245 0 0) 0x7f9d3c003530 con 0x7f9d5013b430 2024-09-16T14:48:45.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d500a5c90 con 0x7f9d5013b430 2024-09-16T14:48:45.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9d48004450 con 0x7f9d5013b430 2024-09-16T14:48:45.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:45.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:45.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient(hunting): dump: 2024-09-16T14:48:45.637 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:45.637 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:45.637 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:45.637 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:45.637 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:45.637 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:45.637 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:45.637 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:45.637 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:45.637 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:45.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996031517 0 0) 0x7f9d5011fee0 con 0x7f9d500a25f0 2024-09-16T14:48:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient(hunting): my global_id is 124470 2024-09-16T14:48:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d3c003530 con 0x7f9d500a25f0 2024-09-16T14:48:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3590538164 0 0) 0x7f9d500a5c90 con 0x7f9d5013b430 2024-09-16T14:48:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 >> v1:172.21.15.26:6790/0 conn(0x7f9d500a1b00 legacy=0x7f9d501c9810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 >> v1:172.21.15.26:6789/0 conn(0x7f9d500a25f0 legacy=0x7f9d501ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 1 monclient: found mon.b 2024-09-16T14:48:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:45.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:45.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5009e2f0 con 0x7f9d5013b430 2024-09-16T14:48:45.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient: _finish_auth 0 2024-09-16T14:48:45.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:45.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:45.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 5 monclient: authenticate success, global_id 144268 2024-09-16T14:48:45.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 10 monclient: _renew_subs 2024-09-16T14:48:45.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:45.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 1 -- 172.21.15.189:0/3986184858 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d5011fc00 con 0x7f9d5013b430 2024-09-16T14:48:45.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 10 monclient: _renew_subs 2024-09-16T14:48:45.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:45.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d55fc5640 1 -- 172.21.15.189:0/3986184858 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d5011db80 con 0x7f9d5013b430 2024-09-16T14:48:45.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d48004770 con 0x7f9d5013b430 2024-09-16T14:48:45.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:45.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9d48004b00 con 0x7f9d5013b430 2024-09-16T14:48:45.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:45.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:45.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d37fff640 10 monclient: dump: 2024-09-16T14:48:45.640 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:45.640 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:45.640 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:45.640 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:45.640 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:45.640 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:45.640 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:45.640 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:45.641 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:45.641 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:45.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.621+0000 7f9d4dffb640 4 set_mon_vals no callback set 2024-09-16T14:48:45.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.625+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9d48004ce0 con 0x7f9d5013b430 2024-09-16T14:48:45.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.625+0000 7f9d55fc5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:45.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.625+0000 7f9d55fc5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:45.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.625+0000 7f9d55fc5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:45.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.625+0000 7f9d55fc5640 1 -- 172.21.15.189:0/3986184858 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d30005050 con 0x7f9d5013b430 2024-09-16T14:48:45.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.625+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(165..165 src has 73..165) v4 ==== 8951+0+0 (unknown 3934497374 0 0) 0x7f9d48044af0 con 0x7f9d5013b430 2024-09-16T14:48:45.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.625+0000 7f9d4f7fe640 10 monclient: get_auth_request con 0x7f9d3c0360e0 auth_method 0 2024-09-16T14:48:45.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.629+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9d48046e60 con 0x7f9d5013b430 2024-09-16T14:48:45.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.629+0000 7f9d37fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:45.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.629+0000 7f9d37fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:45.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.793+0000 7f9d55fc5640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:45.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.793+0000 7f9d55fc5640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:45.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.793+0000 7f9d55fc5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:45.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.793+0000 7f9d55fc5640 1 -- 172.21.15.189:0/3986184858 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f9d30005300 con 0x7f9d5013b430 2024-09-16T14:48:45.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.793+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v165) v1 ==== 83+0+2320 (unknown 3080134077 0 3745721447) 0x7f9d48014b90 con 0x7f9d5013b430 2024-09-16T14:48:45.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.793+0000 7f9d37fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:45.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.793+0000 7f9d37fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:45.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.885+0000 7f9d55fc5640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:45.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.885+0000 7f9d55fc5640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:45.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.885+0000 7f9d55fc5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:45.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.885+0000 7f9d55fc5640 1 -- 172.21.15.189:0/3986184858 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f9d300055a0 con 0x7f9d5013b430 2024-09-16T14:48:45.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.885+0000 7f9d37fff640 1 -- 172.21.15.189:0/3986184858 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v165) v1 ==== 83+0+2320 (unknown 3080134077 0 3745721447) 0x7f9d48013000 con 0x7f9d5013b430 2024-09-16T14:48:45.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.885+0000 7f9d37fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:45.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.885+0000 7f9d37fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:45.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.889+0000 7f9d55fc5640 1 -- 172.21.15.189:0/3986184858 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9d3c0360e0 legacy=0x7f9d3c0385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:45.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.889+0000 7f9d55fc5640 10 monclient: shutdown 2024-09-16T14:48:45.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.889+0000 7f9d55fc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:45.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.889+0000 7f9d55fc5640 1 -- 172.21.15.189:0/3986184858 >> v1:172.21.15.189:6789/0 conn(0x7f9d5013b430 legacy=0x7f9d501d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:45.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.889+0000 7f9d55fc5640 1 -- 172.21.15.189:0/3986184858 shutdown_connections 2024-09-16T14:48:45.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.889+0000 7f9d55fc5640 1 -- 172.21.15.189:0/3986184858 >> 172.21.15.189:0/3986184858 conn(0x7f9d500a7dd0 msgr2=0x7f9d5006b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:45.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.889+0000 7f9d55fc5640 1 -- 172.21.15.189:0/3986184858 shutdown_connections 2024-09-16T14:48:45.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:45.889+0000 7f9d55fc5640 1 -- 172.21.15.189:0/3986184858 wait complete. 2024-09-16T14:48:45.904 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T14:48:45.904 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T14:48:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T14:48:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T14:48:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep globster 2024-09-16T14:48:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '6\.' 2024-09-16T14:48:46.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 10 monclient: get_monmap_and_config 2024-09-16T14:48:46.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 10 monclient: build_initial_monmap 2024-09-16T14:48:46.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 10 monclient: monmap: 2024-09-16T14:48:46.018 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:46.018 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:46.018 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:46.018683+0000 2024-09-16T14:48:46.018 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:46.018683+0000 2024-09-16T14:48:46.019 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:46.019 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:46.019 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:46.019 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:46.019 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:46.019 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 1 Processor -- start 2024-09-16T14:48:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 1 -- start start 2024-09-16T14:48:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 10 monclient: init 2024-09-16T14:48:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:48:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 10 monclient(hunting): picked mon.noname-a con 0x7f7e0c06c1e0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 10 monclient(hunting): picked mon.noname-c con 0x7f7e0c0a1140 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 10 monclient(hunting): picked mon.noname-b con 0x7f7e0c135360 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0c06c5c0 con 0x7f7e0c06c1e0 2024-09-16T14:48:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0c09e4d0 con 0x7f7e0c0a1140 2024-09-16T14:48:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0c09de70 con 0x7f7e0c135360 2024-09-16T14:48:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 10 monclient(hunting): authenticate will time out at 1308.134399s 2024-09-16T14:48:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e11d0f640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7e0c0a1140 0x7f7e0c139a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57606/0 (socket says 172.21.15.189:57606) 2024-09-16T14:48:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e11d0f640 1 -- 172.21.15.189:0/2709106129 learned_addr learned my addr 172.21.15.189:0/2709106129 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e12d11640 1 -- 172.21.15.189:0/2709106129 >> v1:172.21.15.26:6790/0 conn(0x7f7e0c135360 legacy=0x7f7e0c13d1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 1 -- 172.21.15.189:0/2709106129 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807474214 0 0) 0x7f7e0c09e4d0 con 0x7f7e0c0a1140 2024-09-16T14:48:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 10 monclient(hunting): _negotiate 2024-09-16T14:48:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 10 monclient(hunting): my global_id is 144271 2024-09-16T14:48:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 1 -- 172.21.15.189:0/2709106129 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df8003530 con 0x7f7e0c0a1140 2024-09-16T14:48:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 1 -- 172.21.15.189:0/2709106129 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2652770730 0 0) 0x7f7df8003530 con 0x7f7e0c0a1140 2024-09-16T14:48:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 1 -- 172.21.15.189:0/2709106129 >> v1:172.21.15.26:6790/0 conn(0x7f7e0c135360 legacy=0x7f7e0c13d1c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 1 -- 172.21.15.189:0/2709106129 >> v1:172.21.15.26:6789/0 conn(0x7f7e0c06c1e0 legacy=0x7f7e0c134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 10 monclient: _finish_hunting 0 2024-09-16T14:48:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 1 monclient: found mon.noname-c 2024-09-16T14:48:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 1 -- 172.21.15.189:0/2709106129 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0c09e990 con 0x7f7e0c0a1140 2024-09-16T14:48:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 10 monclient: _finish_auth 0 2024-09-16T14:48:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 10 monclient: _check_auth_tickets 2024-09-16T14:48:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 1 -- 172.21.15.189:0/2709106129 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7e08001620 con 0x7f7e0c0a1140 2024-09-16T14:48:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 5 monclient: authenticate success, global_id 144271 2024-09-16T14:48:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 10 monclient: dump: 2024-09-16T14:48:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:46.029 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:46.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:46.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 1 -- 172.21.15.189:0/2709106129 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e080018d0 con 0x7f7e0c0a1140 2024-09-16T14:48:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 4 set_mon_vals no callback set 2024-09-16T14:48:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 1 -- 172.21.15.189:0/2709106129 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7e08001c60 con 0x7f7e0c0a1140 2024-09-16T14:48:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e1150e640 10 monclient: dump: 2024-09-16T14:48:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:46.031 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 10 monclient: shutdown 2024-09-16T14:48:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 1 -- 172.21.15.189:0/2709106129 >> v1:172.21.15.189:6789/0 conn(0x7f7e0c0a1140 legacy=0x7f7e0c139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 1 -- 172.21.15.189:0/2709106129 shutdown_connections 2024-09-16T14:48:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 1 -- 172.21.15.189:0/2709106129 >> 172.21.15.189:0/2709106129 conn(0x7f7e0c0a3a40 msgr2=0x7f7e0c0a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 1 -- 172.21.15.189:0/2709106129 shutdown_connections 2024-09-16T14:48:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.017+0000 7f7e14774640 1 -- 172.21.15.189:0/2709106129 wait complete. 2024-09-16T14:48:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 monclient: build_initial_monmap 2024-09-16T14:48:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 monclient: monmap: 2024-09-16T14:48:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:46.032 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 1 Processor -- start 2024-09-16T14:48:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 1 -- start start 2024-09-16T14:48:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 monclient: init 2024-09-16T14:48:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 monclient(hunting): picked mon.noname-a con 0x7f7e0c06c1e0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 monclient(hunting): picked mon.noname-c con 0x7f7e0c0a1140 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 monclient(hunting): picked mon.noname-b con 0x7f7e0c135360 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0c1311c0 con 0x7f7e0c06c1e0 2024-09-16T14:48:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0c09e130 con 0x7f7e0c135360 2024-09-16T14:48:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0c132dd0 con 0x7f7e0c0a1140 2024-09-16T14:48:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 monclient(hunting): authenticate will time out at 1308.137128s 2024-09-16T14:48:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e11d0f640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f7e0c0a1140 legacy=0x7f7e0c1cd7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e12d11640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7e0c135360 0x7f7e0c1d0f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57608/0 (socket says 172.21.15.189:57608) 2024-09-16T14:48:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e12d11640 1 -- 172.21.15.189:0/2521890975 learned_addr learned my addr 172.21.15.189:0/2521890975 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289403843 0 0) 0x7f7e0c09e130 con 0x7f7e0c135360 2024-09-16T14:48:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient(hunting): my global_id is 144274 2024-09-16T14:48:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df0003530 con 0x7f7e0c135360 2024-09-16T14:48:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1669165823 0 0) 0x7f7df0003530 con 0x7f7e0c135360 2024-09-16T14:48:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e0c09e130 con 0x7f7e0c135360 2024-09-16T14:48:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7e040043c0 con 0x7f7e0c135360 2024-09-16T14:48:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient(hunting): dump: 2024-09-16T14:48:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:46.036 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686844305 0 0) 0x7f7e0c1311c0 con 0x7f7e0c06c1e0 2024-09-16T14:48:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient(hunting): my global_id is 124476 2024-09-16T14:48:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df0003530 con 0x7f7e0c06c1e0 2024-09-16T14:48:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3772340300 0 0) 0x7f7e0c09e130 con 0x7f7e0c135360 2024-09-16T14:48:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 >> v1:172.21.15.26:6790/0 conn(0x7f7e0c0a1140 legacy=0x7f7e0c1cd7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 >> v1:172.21.15.26:6789/0 conn(0x7f7e0c06c1e0 legacy=0x7f7e0c1331f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 1 monclient: found mon.b 2024-09-16T14:48:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0c09cc10 con 0x7f7e0c135360 2024-09-16T14:48:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient: _finish_auth 0 2024-09-16T14:48:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 5 monclient: authenticate success, global_id 144274 2024-09-16T14:48:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 monclient: _renew_subs 2024-09-16T14:48:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 1 -- 172.21.15.189:0/2521890975 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0c0a5510 con 0x7f7e0c135360 2024-09-16T14:48:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 monclient: _renew_subs 2024-09-16T14:48:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 1 -- 172.21.15.189:0/2521890975 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0c1d2640 con 0x7f7e0c135360 2024-09-16T14:48:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e040046e0 con 0x7f7e0c135360 2024-09-16T14:48:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7e04004a70 con 0x7f7e0c135360 2024-09-16T14:48:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e10d0d640 4 set_mon_vals no callback set 2024-09-16T14:48:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 10 monclient: dump: 2024-09-16T14:48:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:46.040 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7e04004c50 con 0x7f7e0c135360 2024-09-16T14:48:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e14774640 1 -- 172.21.15.189:0/2521890975 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df4005050 con 0x7f7e0c135360 2024-09-16T14:48:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(165..165 src has 73..165) v4 ==== 8951+0+0 (unknown 3934497374 0 0) 0x7f7e04044a10 con 0x7f7e0c135360 2024-09-16T14:48:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.021+0000 7f7e12510640 10 monclient: get_auth_request con 0x7f7df0036100 auth_method 0 2024-09-16T14:48:46.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.025+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7e04014ab0 con 0x7f7e0c135360 2024-09-16T14:48:46.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.025+0000 7f7dfeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:46.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.025+0000 7f7dfeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:46.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.197+0000 7f7e14774640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:46.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.197+0000 7f7e14774640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:46.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.197+0000 7f7e14774640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:46.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.197+0000 7f7e14774640 1 -- 172.21.15.189:0/2521890975 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f7df4005300 con 0x7f7e0c135360 2024-09-16T14:48:46.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.197+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v165) v1 ==== 83+0+2320 (unknown 3080134077 0 3745721447) 0x7f7e04012f20 con 0x7f7e0c135360 2024-09-16T14:48:46.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.197+0000 7f7dfeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:46.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.197+0000 7f7dfeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:46.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.293+0000 7f7e14774640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:46.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.293+0000 7f7e14774640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:46.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.293+0000 7f7e14774640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:46.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.293+0000 7f7e14774640 1 -- 172.21.15.189:0/2521890975 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f7df40055a0 con 0x7f7e0c135360 2024-09-16T14:48:46.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.297+0000 7f7dfeffd640 1 -- 172.21.15.189:0/2521890975 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v165) v1 ==== 83+0+2320 (unknown 3080134077 0 3745721447) 0x7f7e040146a0 con 0x7f7e0c135360 2024-09-16T14:48:46.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.297+0000 7f7dfeffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T14:48:46.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.297+0000 7f7dfeffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:46.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.297+0000 7f7e14774640 1 -- 172.21.15.189:0/2521890975 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7df0036100 legacy=0x7f7df00385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:46.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.297+0000 7f7e14774640 10 monclient: shutdown 2024-09-16T14:48:46.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.297+0000 7f7e14774640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:46.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.297+0000 7f7e14774640 1 -- 172.21.15.189:0/2521890975 >> v1:172.21.15.189:6789/0 conn(0x7f7e0c135360 legacy=0x7f7e0c1d0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:46.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.297+0000 7f7e14774640 1 -- 172.21.15.189:0/2521890975 shutdown_connections 2024-09-16T14:48:46.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.297+0000 7f7e14774640 1 -- 172.21.15.189:0/2521890975 >> 172.21.15.189:0/2521890975 conn(0x7f7e0c0a3a40 msgr2=0x7f7e0c0a6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:46.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.297+0000 7f7e14774640 1 -- 172.21.15.189:0/2521890975 shutdown_connections 2024-09-16T14:48:46.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.297+0000 7f7e14774640 1 -- 172.21.15.189:0/2521890975 wait complete. 2024-09-16T14:48:46.313 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T14:48:46.313 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T14:48:46.313 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-16T14:48:46.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fca896640 10 monclient: get_monmap_and_config 2024-09-16T14:48:46.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fca896640 10 monclient: build_initial_monmap 2024-09-16T14:48:46.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fca896640 10 monclient: monmap: 2024-09-16T14:48:46.431 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:46.431 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:46.431 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:46.430886+0000 2024-09-16T14:48:46.431 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:46.430886+0000 2024-09-16T14:48:46.431 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:46.431 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:46.431 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:46.431 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:46.431 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:46.432 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:46.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fca896640 1 Processor -- start 2024-09-16T14:48:46.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fca896640 1 -- start start 2024-09-16T14:48:46.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fca896640 10 monclient: init 2024-09-16T14:48:46.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fca896640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:46.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fca896640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:46.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fca896640 10 monclient(hunting): picked mon.noname-b con 0x7f2fc40a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:46.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fca896640 10 monclient(hunting): picked mon.noname-c con 0x7f2fc40a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:46.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fca896640 10 monclient(hunting): picked mon.noname-a con 0x7f2fc413b430 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:46.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fca896640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc40a30b0 con 0x7f2fc413b430 2024-09-16T14:48:46.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fca896640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc40a2e30 con 0x7f2fc40a25f0 2024-09-16T14:48:46.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fca896640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc409eb50 con 0x7f2fc40a1b00 2024-09-16T14:48:46.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fca896640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:46.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fca896640 10 monclient(hunting): authenticate will time out at 1308.547305s 2024-09-16T14:48:46.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fbb7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2fc40a25f0 0x7f2fc4139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57614/0 (socket says 172.21.15.189:57614) 2024-09-16T14:48:46.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fbb7fe640 1 -- 172.21.15.189:0/2580407561 learned_addr learned my addr 172.21.15.189:0/2580407561 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:46.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fc3fff640 1 -- 172.21.15.189:0/2580407561 >> v1:172.21.15.26:6790/0 conn(0x7f2fc40a1b00 legacy=0x7f2fc40a1ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:46.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fc37fe640 1 -- 172.21.15.189:0/2580407561 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135089075 0 0) 0x7f2fc40a2e30 con 0x7f2fc40a25f0 2024-09-16T14:48:46.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fc37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:46.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:46.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:46.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fc37fe640 10 monclient(hunting): my global_id is 144277 2024-09-16T14:48:46.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.429+0000 7f2fc37fe640 1 -- 172.21.15.189:0/2580407561 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb0003530 con 0x7f2fc40a25f0 2024-09-16T14:48:46.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 1 -- 172.21.15.189:0/2580407561 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375136681 0 0) 0x7f2fc40a30b0 con 0x7f2fc413b430 2024-09-16T14:48:46.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:46.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:46.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:46.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 10 monclient(hunting): my global_id is 124479 2024-09-16T14:48:46.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 1 -- 172.21.15.189:0/2580407561 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc40a2e30 con 0x7f2fc413b430 2024-09-16T14:48:46.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 1 -- 172.21.15.189:0/2580407561 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 843346056 0 0) 0x7f2fb0003530 con 0x7f2fc40a25f0 2024-09-16T14:48:46.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 1 -- 172.21.15.189:0/2580407561 >> v1:172.21.15.26:6790/0 conn(0x7f2fc40a1b00 legacy=0x7f2fc40a1ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:46.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 1 -- 172.21.15.189:0/2580407561 >> v1:172.21.15.26:6789/0 conn(0x7f2fc413b430 legacy=0x7f2fc413d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:46.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:48:46.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 1 monclient: found mon.noname-c 2024-09-16T14:48:46.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:46.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:46.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 1 -- 172.21.15.189:0/2580407561 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc409ee30 con 0x7f2fc40a25f0 2024-09-16T14:48:46.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 10 monclient: _finish_auth 0 2024-09-16T14:48:46.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:48:46.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:46.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 5 monclient: authenticate success, global_id 144277 2024-09-16T14:48:46.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 1 -- 172.21.15.189:0/2580407561 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2fac001620 con 0x7f2fc40a25f0 2024-09-16T14:48:46.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:46.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:46.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:46.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 10 monclient: dump: 2024-09-16T14:48:46.445 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:46.445 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:46.445 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:46.445 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:46.446 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:46.446 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:46.446 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:46.446 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:46.446 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:46.446 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:46.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:46.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 1 -- 172.21.15.189:0/2580407561 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fac001840 con 0x7f2fc40a25f0 2024-09-16T14:48:46.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:46.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:46.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 4 set_mon_vals no callback set 2024-09-16T14:48:46.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 1 -- 172.21.15.189:0/2580407561 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2fac001bd0 con 0x7f2fc40a25f0 2024-09-16T14:48:46.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:46.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:46.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc37fe640 10 monclient: dump: 2024-09-16T14:48:46.447 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:46.448 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:46.448 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:46.448 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:46.448 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:46.448 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:46.448 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:46.448 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:46.448 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:46.448 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:46.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 10 monclient: shutdown 2024-09-16T14:48:46.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:46.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 1 -- 172.21.15.189:0/2580407561 >> v1:172.21.15.189:6789/0 conn(0x7f2fc40a25f0 legacy=0x7f2fc4139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:46.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 1 -- 172.21.15.189:0/2580407561 shutdown_connections 2024-09-16T14:48:46.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 1 -- 172.21.15.189:0/2580407561 >> 172.21.15.189:0/2580407561 conn(0x7f2fc40a7dd0 msgr2=0x7f2fc40a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:46.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 1 -- 172.21.15.189:0/2580407561 shutdown_connections 2024-09-16T14:48:46.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 1 -- 172.21.15.189:0/2580407561 wait complete. 2024-09-16T14:48:46.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 10 monclient: build_initial_monmap 2024-09-16T14:48:46.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 10 monclient: monmap: 2024-09-16T14:48:46.450 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:46.450 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:46.450 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:46.450 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:46.450 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:46.450 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:46.450 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:46.450 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:46.451 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 1 Processor -- start 2024-09-16T14:48:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 1 -- start start 2024-09-16T14:48:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 10 monclient: init 2024-09-16T14:48:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 10 monclient(hunting): picked mon.noname-b con 0x7f2fc40a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 10 monclient(hunting): picked mon.noname-a con 0x7f2fc40a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 10 monclient(hunting): picked mon.noname-c con 0x7f2fc413b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc4120080 con 0x7f2fc40a25f0 2024-09-16T14:48:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc411e960 con 0x7f2fc40a1b00 2024-09-16T14:48:46.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc40a54f0 con 0x7f2fc413b430 2024-09-16T14:48:46.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:46.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 10 monclient(hunting): authenticate will time out at 1308.550792s 2024-09-16T14:48:46.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc3fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2fc40a1b00 0x7f2fc41c9810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57624/0 (socket says 172.21.15.189:57624) 2024-09-16T14:48:46.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc3fff640 1 -- 172.21.15.189:0/2810321926 learned_addr learned my addr 172.21.15.189:0/2810321926 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:46.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc8e33640 1 -- 172.21.15.189:0/2810321926 >> v1:172.21.15.26:6790/0 conn(0x7f2fc413b430 legacy=0x7f2fc41d1f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:46.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945305998 0 0) 0x7f2fc411e960 con 0x7f2fc40a1b00 2024-09-16T14:48:46.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:46.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:46.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:46.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient(hunting): my global_id is 144280 2024-09-16T14:48:46.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa4003530 con 0x7f2fc40a1b00 2024-09-16T14:48:46.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3515180426 0 0) 0x7f2fa4003530 con 0x7f2fc40a1b00 2024-09-16T14:48:46.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fc411e960 con 0x7f2fc40a1b00 2024-09-16T14:48:46.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2fb4004190 con 0x7f2fc40a1b00 2024-09-16T14:48:46.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:46.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:46.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient(hunting): dump: 2024-09-16T14:48:46.453 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:46.453 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:46.453 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:46.453 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:46.453 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:46.453 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:46.453 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:46.453 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:46.454 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:46.454 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:46.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:46.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387451589 0 0) 0x7f2fc4120080 con 0x7f2fc40a25f0 2024-09-16T14:48:46.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:46.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:46.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:46.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient(hunting): my global_id is 124482 2024-09-16T14:48:46.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa4003530 con 0x7f2fc40a25f0 2024-09-16T14:48:46.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2497491199 0 0) 0x7f2fc411e960 con 0x7f2fc40a1b00 2024-09-16T14:48:46.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 >> v1:172.21.15.26:6790/0 conn(0x7f2fc413b430 legacy=0x7f2fc41d1f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:46.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 >> v1:172.21.15.26:6789/0 conn(0x7f2fc40a25f0 legacy=0x7f2fc41ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:46.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient: _finish_hunting 0 2024-09-16T14:48:46.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 1 monclient: found mon.b 2024-09-16T14:48:46.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:46.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:46.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc40a5af0 con 0x7f2fc40a1b00 2024-09-16T14:48:46.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient: _finish_auth 0 2024-09-16T14:48:46.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient: _check_auth_tickets 2024-09-16T14:48:46.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:46.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 5 monclient: authenticate success, global_id 144280 2024-09-16T14:48:46.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 10 monclient: _renew_subs 2024-09-16T14:48:46.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:46.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 1 -- 172.21.15.189:0/2810321926 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc411fda0 con 0x7f2fc40a1b00 2024-09-16T14:48:46.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 10 monclient: _renew_subs 2024-09-16T14:48:46.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:46.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fca896640 1 -- 172.21.15.189:0/2810321926 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc411fac0 con 0x7f2fc40a1b00 2024-09-16T14:48:46.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fb40043b0 con 0x7f2fc40a1b00 2024-09-16T14:48:46.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:46.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2fb4004740 con 0x7f2fc40a1b00 2024-09-16T14:48:46.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:46.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:46.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc2ffd640 4 set_mon_vals no callback set 2024-09-16T14:48:46.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.433+0000 7f2fc17fa640 10 monclient: dump: 2024-09-16T14:48:46.460 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:46.460 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:46.460 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:46.460 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:46.460 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:46.460 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:46.460 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:46.460 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:46.460 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:46.460 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:46.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.437+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2fb4004960 con 0x7f2fc40a1b00 2024-09-16T14:48:46.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.437+0000 7f2fca896640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:46.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.437+0000 7f2fca896640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:46.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.437+0000 7f2fca896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:46.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.437+0000 7f2fca896640 1 -- 172.21.15.189:0/2810321926 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa8005050 con 0x7f2fc40a1b00 2024-09-16T14:48:46.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.437+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(165..165 src has 73..165) v4 ==== 8951+0+0 (unknown 3934497374 0 0) 0x7f2fb40446e0 con 0x7f2fc40a1b00 2024-09-16T14:48:46.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.437+0000 7f2fbb7fe640 10 monclient: get_auth_request con 0x7f2fa40361d0 auth_method 0 2024-09-16T14:48:46.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.441+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2fb4014780 con 0x7f2fc40a1b00 2024-09-16T14:48:46.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.441+0000 7f2fc17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:46.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.441+0000 7f2fc17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:46.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.589+0000 7f2fca896640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:46.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.589+0000 7f2fca896640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:46.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.589+0000 7f2fca896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:46.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:46.589+0000 7f2fca896640 1 -- 172.21.15.189:0/2810321926 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f2fa8005b10 con 0x7f2fc40a1b00 2024-09-16T14:48:47.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:47.077+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v166) v1 ==== 123+0+0 (unknown 2875036770 0 0) 0x7f2fb4012bf0 con 0x7f2fc40a1b00 2024-09-16T14:48:47.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:47.081+0000 7f2fc17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:47.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:47.081+0000 7f2fc17fa640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-16T14:48:47.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:47.181+0000 7f2fca896640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:47.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:47.181+0000 7f2fca896640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:47.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:47.181+0000 7f2fca896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:47.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:47.181+0000 7f2fca896640 1 -- 172.21.15.189:0/2810321926 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f2fa8005300 con 0x7f2fc40a1b00 2024-09-16T14:48:48.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.021+0000 7f2fc17fa640 1 -- 172.21.15.189:0/2810321926 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v167) v1 ==== 149+0+0 (unknown 29425364 0 0) 0x7f2fb4014370 con 0x7f2fc40a1b00 2024-09-16T14:48:48.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.021+0000 7f2fc17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T14:48:48.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.021+0000 7f2fc17fa640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T14:48:48.023 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T14:48:48.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.025+0000 7f2fca896640 1 -- 172.21.15.189:0/2810321926 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2fa40361d0 legacy=0x7f2fa4038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:48.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.025+0000 7f2fca896640 10 monclient: shutdown 2024-09-16T14:48:48.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.025+0000 7f2fca896640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:48.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.025+0000 7f2fca896640 1 -- 172.21.15.189:0/2810321926 >> v1:172.21.15.189:6789/0 conn(0x7f2fc40a1b00 legacy=0x7f2fc41c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:48.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.025+0000 7f2fca896640 1 -- 172.21.15.189:0/2810321926 shutdown_connections 2024-09-16T14:48:48.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.025+0000 7f2fca896640 1 -- 172.21.15.189:0/2810321926 >> 172.21.15.189:0/2810321926 conn(0x7f2fc40a7dd0 msgr2=0x7f2fc409d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:48.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.025+0000 7f2fca896640 1 -- 172.21.15.189:0/2810321926 shutdown_connections 2024-09-16T14:48:48.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.025+0000 7f2fca896640 1 -- 172.21.15.189:0/2810321926 wait complete. 2024-09-16T14:48:48.048 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool rm cool cool --yes-i-really-really-mean-it 2024-09-16T14:48:48.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 10 monclient: get_monmap_and_config 2024-09-16T14:48:48.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 10 monclient: build_initial_monmap 2024-09-16T14:48:48.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 10 monclient: monmap: 2024-09-16T14:48:48.171 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:48.171 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:48.171 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:48.170972+0000 2024-09-16T14:48:48.171 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:48.170972+0000 2024-09-16T14:48:48.171 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:48.171 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:48.171 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:48.171 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:48.171 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:48.171 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:48.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 1 Processor -- start 2024-09-16T14:48:48.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 1 -- start start 2024-09-16T14:48:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 10 monclient: init 2024-09-16T14:48:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 10 monclient(hunting): picked mon.noname-b con 0x7f9f90099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 10 monclient(hunting): picked mon.noname-c con 0x7f9f9009c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 10 monclient(hunting): picked mon.noname-a con 0x7f9f9009e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f90099e50 con 0x7f9f9009e6d0 2024-09-16T14:48:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9009dbd0 con 0x7f9f9009c550 2024-09-16T14:48:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9009ee20 con 0x7f9f90099a70 2024-09-16T14:48:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 10 monclient(hunting): authenticate will time out at 1310.287052s 2024-09-16T14:48:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f9548d640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f9f90099a70 legacy=0x7f9f9009dfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:48.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f94c8c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9f9009c550 0x7f9f901379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57640/0 (socket says 172.21.15.189:57640) 2024-09-16T14:48:48.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f94c8c640 1 -- 172.21.15.189:0/2800095356 learned_addr learned my addr 172.21.15.189:0/2800095356 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:48.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 1 -- 172.21.15.189:0/2800095356 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024928368 0 0) 0x7f9f9009dbd0 con 0x7f9f9009c550 2024-09-16T14:48:48.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:48.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:48.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:48.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 10 monclient(hunting): my global_id is 144283 2024-09-16T14:48:48.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 1 -- 172.21.15.189:0/2800095356 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f7c003530 con 0x7f9f9009c550 2024-09-16T14:48:48.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 1 -- 172.21.15.189:0/2800095356 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 614868839 0 0) 0x7f9f7c003530 con 0x7f9f9009c550 2024-09-16T14:48:48.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 1 -- 172.21.15.189:0/2800095356 >> v1:172.21.15.26:6790/0 conn(0x7f9f90099a70 legacy=0x7f9f9009dfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:48.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 1 -- 172.21.15.189:0/2800095356 >> v1:172.21.15.26:6789/0 conn(0x7f9f9009e6d0 legacy=0x7f9f9013b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:48.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:48.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 1 monclient: found mon.noname-c 2024-09-16T14:48:48.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:48.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:48.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 1 -- 172.21.15.189:0/2800095356 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9009f100 con 0x7f9f9009c550 2024-09-16T14:48:48.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 10 monclient: _finish_auth 0 2024-09-16T14:48:48.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:48.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:48.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 1 -- 172.21.15.189:0/2800095356 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f80001620 con 0x7f9f9009c550 2024-09-16T14:48:48.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 5 monclient: authenticate success, global_id 144283 2024-09-16T14:48:48.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:48.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:48.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:48.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 10 monclient: dump: 2024-09-16T14:48:48.178 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:48.178 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:48.178 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:48.178 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:48.178 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:48.178 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:48.178 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:48.178 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:48.178 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:48.178 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:48.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:48.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 1 -- 172.21.15.189:0/2800095356 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f800018d0 con 0x7f9f9009c550 2024-09-16T14:48:48.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:48.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 1 -- 172.21.15.189:0/2800095356 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f80001ca0 con 0x7f9f9009c550 2024-09-16T14:48:48.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:48.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:48.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f87fff640 10 monclient: dump: 2024-09-16T14:48:48.179 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:48.179 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:48.179 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:48.179 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:48.179 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:48.179 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:48.179 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:48.179 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:48.179 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:48.180 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:48.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:48.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.169+0000 7f9f976f1640 4 set_mon_vals no callback set 2024-09-16T14:48:48.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient: shutdown 2024-09-16T14:48:48.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:48.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 1 -- 172.21.15.189:0/2800095356 >> v1:172.21.15.189:6789/0 conn(0x7f9f9009c550 legacy=0x7f9f901379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:48.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 1 -- 172.21.15.189:0/2800095356 shutdown_connections 2024-09-16T14:48:48.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 1 -- 172.21.15.189:0/2800095356 >> 172.21.15.189:0/2800095356 conn(0x7f9f900a3380 msgr2=0x7f9f9009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:48.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 1 -- 172.21.15.189:0/2800095356 shutdown_connections 2024-09-16T14:48:48.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 1 -- 172.21.15.189:0/2800095356 wait complete. 2024-09-16T14:48:48.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient: build_initial_monmap 2024-09-16T14:48:48.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient: monmap: 2024-09-16T14:48:48.180 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:48.181 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:48.181 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:48.181 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:48.181 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:48.181 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:48.181 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:48.181 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:48.181 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:48.181 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:48.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 1 Processor -- start 2024-09-16T14:48:48.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 1 -- start start 2024-09-16T14:48:48.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient: init 2024-09-16T14:48:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient(hunting): picked mon.noname-b con 0x7f9f90099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient(hunting): picked mon.noname-a con 0x7f9f9009c550 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient(hunting): picked mon.noname-c con 0x7f9f9009e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9009cf20 con 0x7f9f9009c550 2024-09-16T14:48:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9011b890 con 0x7f9f90099a70 2024-09-16T14:48:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f901325f0 con 0x7f9f9009e6d0 2024-09-16T14:48:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient(hunting): authenticate will time out at 1310.290181s 2024-09-16T14:48:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f9548d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9f90099a70 0x7f9f90134200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57642/0 (socket says 172.21.15.189:57642) 2024-09-16T14:48:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f9548d640 1 -- 172.21.15.189:0/720561777 learned_addr learned my addr 172.21.15.189:0/720561777 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f95c8e640 1 -- 172.21.15.189:0/720561777 >> v1:172.21.15.26:6790/0 conn(0x7f9f9009e6d0 legacy=0x7f9f901d1720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 1 -- 172.21.15.189:0/720561777 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667649090 0 0) 0x7f9f9011b890 con 0x7f9f90099a70 2024-09-16T14:48:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 10 monclient(hunting): my global_id is 144286 2024-09-16T14:48:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 1 -- 172.21.15.189:0/720561777 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f74003530 con 0x7f9f90099a70 2024-09-16T14:48:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 1 -- 172.21.15.189:0/720561777 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2616276564 0 0) 0x7f9f74003530 con 0x7f9f90099a70 2024-09-16T14:48:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 1 -- 172.21.15.189:0/720561777 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f9011b890 con 0x7f9f90099a70 2024-09-16T14:48:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 1 -- 172.21.15.189:0/720561777 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f8c0041e0 con 0x7f9f90099a70 2024-09-16T14:48:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 10 monclient(hunting): dump: 2024-09-16T14:48:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:48.188 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 1 -- 172.21.15.189:0/720561777 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2448102851 0 0) 0x7f9f9011b890 con 0x7f9f90099a70 2024-09-16T14:48:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 1 -- 172.21.15.189:0/720561777 >> v1:172.21.15.26:6790/0 conn(0x7f9f9009e6d0 legacy=0x7f9f901d1720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 1 -- 172.21.15.189:0/720561777 >> v1:172.21.15.26:6789/0 conn(0x7f9f9009c550 legacy=0x7f9f90134910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:48:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 1 monclient: found mon.b 2024-09-16T14:48:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 1 -- 172.21.15.189:0/720561777 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9006cd90 con 0x7f9f90099a70 2024-09-16T14:48:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 10 monclient: _finish_auth 0 2024-09-16T14:48:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 5 monclient: authenticate success, global_id 144286 2024-09-16T14:48:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient: _renew_subs 2024-09-16T14:48:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 1 -- 172.21.15.189:0/720561777 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9006b320 con 0x7f9f90099a70 2024-09-16T14:48:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient: _renew_subs 2024-09-16T14:48:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 1 -- 172.21.15.189:0/720561777 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f90119e80 con 0x7f9f90099a70 2024-09-16T14:48:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 1 -- 172.21.15.189:0/720561777 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f8c004500 con 0x7f9f90099a70 2024-09-16T14:48:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 1 -- 172.21.15.189:0/720561777 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f8c004890 con 0x7f9f90099a70 2024-09-16T14:48:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f86ffd640 4 set_mon_vals no callback set 2024-09-16T14:48:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 10 monclient: dump: 2024-09-16T14:48:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:48.191 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f85ffb640 1 -- 172.21.15.189:0/720561777 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9f8c0049f0 con 0x7f9f90099a70 2024-09-16T14:48:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.173+0000 7f9f976f1640 1 -- 172.21.15.189:0/720561777 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f70005050 con 0x7f9f90099a70 2024-09-16T14:48:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.177+0000 7f9f85ffb640 1 -- 172.21.15.189:0/720561777 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(167..167 src has 73..167) v4 ==== 8262+0+0 (unknown 1619474286 0 0) 0x7f9f8c044470 con 0x7f9f90099a70 2024-09-16T14:48:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.177+0000 7f9f94c8c640 10 monclient: get_auth_request con 0x7f9f74036010 auth_method 0 2024-09-16T14:48:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.181+0000 7f9f85ffb640 1 -- 172.21.15.189:0/720561777 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9f8c0051c0 con 0x7f9f90099a70 2024-09-16T14:48:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.181+0000 7f9f85ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.181+0000 7f9f85ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.333+0000 7f9f976f1640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T14:48:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.333+0000 7f9f976f1640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T14:48:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.333+0000 7f9f976f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:48.333+0000 7f9f976f1640 1 -- 172.21.15.189:0/720561777 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9f70005300 con 0x7f9f90099a70 2024-09-16T14:48:49.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.089+0000 7f9f85ffb640 1 -- 172.21.15.189:0/720561777 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}]=0 pool 'cool' removed v168) v1 ==== 148+0+0 (unknown 3885969402 0 0) 0x7f9f8c01cdb0 con 0x7f9f90099a70 2024-09-16T14:48:49.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.089+0000 7f9f85ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T14:48:49.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.089+0000 7f9f85ffb640 10 monclient: _finish_command 2 = system:0 pool 'cool' removed 2024-09-16T14:48:49.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.185+0000 7f9f976f1640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T14:48:49.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.185+0000 7f9f976f1640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T14:48:49.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.185+0000 7f9f976f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:49.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.185+0000 7f9f976f1640 1 -- 172.21.15.189:0/720561777 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9f70005a40 con 0x7f9f90099a70 2024-09-16T14:48:49.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.185+0000 7f9f85ffb640 1 -- 172.21.15.189:0/720561777 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}]=0 pool 'cool' does not exist v168) v1 ==== 155+0+0 (unknown 1175874072 0 0) 0x7f9f8c012c30 con 0x7f9f90099a70 2024-09-16T14:48:49.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.185+0000 7f9f85ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T14:48:49.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.185+0000 7f9f85ffb640 10 monclient: _finish_command 3 = system:0 pool 'cool' does not exist 2024-09-16T14:48:49.189 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'cool' does not exist 2024-09-16T14:48:49.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.189+0000 7f9f976f1640 1 -- 172.21.15.189:0/720561777 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9f74036010 legacy=0x7f9f740384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:49.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.189+0000 7f9f976f1640 10 monclient: shutdown 2024-09-16T14:48:49.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.189+0000 7f9f976f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:49.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.189+0000 7f9f976f1640 1 -- 172.21.15.189:0/720561777 >> v1:172.21.15.189:6789/0 conn(0x7f9f90099a70 legacy=0x7f9f90134200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:49.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.189+0000 7f9f976f1640 1 -- 172.21.15.189:0/720561777 shutdown_connections 2024-09-16T14:48:49.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.189+0000 7f9f976f1640 1 -- 172.21.15.189:0/720561777 >> 172.21.15.189:0/720561777 conn(0x7f9f900a3380 msgr2=0x7f9f900a1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:49.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.189+0000 7f9f976f1640 1 -- 172.21.15.189:0/720561777 shutdown_connections 2024-09-16T14:48:49.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.189+0000 7f9f976f1640 1 -- 172.21.15.189:0/720561777 wait complete. 2024-09-16T14:48:49.206 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool rm cold cold --yes-i-really-really-mean-it 2024-09-16T14:48:49.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.317+0000 7f6e68147640 10 monclient: get_monmap_and_config 2024-09-16T14:48:49.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.317+0000 7f6e68147640 10 monclient: build_initial_monmap 2024-09-16T14:48:49.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.317+0000 7f6e68147640 10 monclient: monmap: 2024-09-16T14:48:49.321 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:49.321 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:49.321 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:49.321720+0000 2024-09-16T14:48:49.322 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:49.321720+0000 2024-09-16T14:48:49.322 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:49.322 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:49.322 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:49.322 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:49.322 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:49.322 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:49.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 1 Processor -- start 2024-09-16T14:48:49.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 1 -- start start 2024-09-16T14:48:49.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 10 monclient: init 2024-09-16T14:48:49.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:49.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:49.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 10 monclient(hunting): picked mon.noname-a con 0x7f6e600a1a90 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:49.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 10 monclient(hunting): picked mon.noname-b con 0x7f6e600a2580 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:49.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 10 monclient(hunting): picked mon.noname-c con 0x7f6e6009fd40 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:49.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6009df00 con 0x7f6e600a1a90 2024-09-16T14:48:49.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6009f0a0 con 0x7f6e6009fd40 2024-09-16T14:48:49.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e600a3040 con 0x7f6e600a2580 2024-09-16T14:48:49.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:49.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 10 monclient(hunting): authenticate will time out at 1311.437318s 2024-09-16T14:48:49.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e666e4640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6e6009fd40 0x7f6e6013d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46876/0 (socket says 172.21.15.189:46876) 2024-09-16T14:48:49.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e666e4640 1 -- 172.21.15.189:0/3332608335 learned_addr learned my addr 172.21.15.189:0/3332608335 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:49.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e656e2640 1 -- 172.21.15.189:0/3332608335 >> v1:172.21.15.26:6790/0 conn(0x7f6e600a2580 legacy=0x7f6e60139df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:49.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 1 -- 172.21.15.189:0/3332608335 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942243108 0 0) 0x7f6e6009f0a0 con 0x7f6e6009fd40 2024-09-16T14:48:49.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 10 monclient(hunting): _negotiate 2024-09-16T14:48:49.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:49.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:49.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 10 monclient(hunting): my global_id is 144289 2024-09-16T14:48:49.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 1 -- 172.21.15.189:0/3332608335 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e4c003530 con 0x7f6e6009fd40 2024-09-16T14:48:49.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 1 -- 172.21.15.189:0/3332608335 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3073458328 0 0) 0x7f6e4c003530 con 0x7f6e6009fd40 2024-09-16T14:48:49.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 1 -- 172.21.15.189:0/3332608335 >> v1:172.21.15.26:6790/0 conn(0x7f6e600a2580 legacy=0x7f6e60139df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:49.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 1 -- 172.21.15.189:0/3332608335 >> v1:172.21.15.26:6789/0 conn(0x7f6e600a1a90 legacy=0x7f6e600a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:49.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 10 monclient: _finish_hunting 0 2024-09-16T14:48:49.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 1 monclient: found mon.noname-c 2024-09-16T14:48:49.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:49.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:49.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 1 -- 172.21.15.189:0/3332608335 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6009eae0 con 0x7f6e6009fd40 2024-09-16T14:48:49.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 10 monclient: _finish_auth 0 2024-09-16T14:48:49.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 10 monclient: _check_auth_tickets 2024-09-16T14:48:49.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:49.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 1 -- 172.21.15.189:0/3332608335 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e580019a0 con 0x7f6e6009fd40 2024-09-16T14:48:49.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:49.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:49.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 10 monclient: dump: 2024-09-16T14:48:49.337 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:49.337 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:49.337 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:49.337 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:49.337 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:49.337 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:49.337 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:49.337 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:49.337 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:49.337 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:49.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 5 monclient: authenticate success, global_id 144289 2024-09-16T14:48:49.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:49.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 1 -- 172.21.15.189:0/3332608335 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e58001c50 con 0x7f6e6009fd40 2024-09-16T14:48:49.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:49.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:49.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 1 -- 172.21.15.189:0/3332608335 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e58002020 con 0x7f6e6009fd40 2024-09-16T14:48:49.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 4 set_mon_vals no callback set 2024-09-16T14:48:49.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:49.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:49.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e64ee1640 10 monclient: dump: 2024-09-16T14:48:49.339 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:49.339 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:49.339 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:49.339 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:49.339 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:49.339 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:49.339 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:49.339 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:49.339 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:49.339 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:49.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 10 monclient: shutdown 2024-09-16T14:48:49.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:49.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 1 -- 172.21.15.189:0/3332608335 >> v1:172.21.15.189:6789/0 conn(0x7f6e6009fd40 legacy=0x7f6e6013d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:49.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 1 -- 172.21.15.189:0/3332608335 shutdown_connections 2024-09-16T14:48:49.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 1 -- 172.21.15.189:0/3332608335 >> 172.21.15.189:0/3332608335 conn(0x7f6e600a7a70 msgr2=0x7f6e600a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:49.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 1 -- 172.21.15.189:0/3332608335 shutdown_connections 2024-09-16T14:48:49.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 1 -- 172.21.15.189:0/3332608335 wait complete. 2024-09-16T14:48:49.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 10 monclient: build_initial_monmap 2024-09-16T14:48:49.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 10 monclient: monmap: 2024-09-16T14:48:49.340 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:49.340 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:49.341 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:49.341 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:49.341 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:49.341 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:49.341 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:49.341 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:49.341 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:49.341 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:49.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 1 Processor -- start 2024-09-16T14:48:49.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 1 -- start start 2024-09-16T14:48:49.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.321+0000 7f6e68147640 10 monclient: init 2024-09-16T14:48:49.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:49.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:49.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 10 monclient(hunting): picked mon.noname-c con 0x7f6e6009fd40 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:49.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 10 monclient(hunting): picked mon.noname-a con 0x7f6e600a1a90 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:49.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 10 monclient(hunting): picked mon.noname-b con 0x7f6e600a2580 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:49.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e60124c10 con 0x7f6e600a1a90 2024-09-16T14:48:49.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e600a5c90 con 0x7f6e600a2580 2024-09-16T14:48:49.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e600a61e0 con 0x7f6e6009fd40 2024-09-16T14:48:49.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:49.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 10 monclient(hunting): authenticate will time out at 1311.440117s 2024-09-16T14:48:49.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e65ee3640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f6e6009fd40 legacy=0x7f6e60120f40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:49.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e656e2640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6e600a1a90 0x7f6e60121650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:36566/0 (socket says 172.21.15.189:36566) 2024-09-16T14:48:49.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e656e2640 1 -- 172.21.15.189:0/298615587 learned_addr learned my addr 172.21.15.189:0/298615587 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:49.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886217148 0 0) 0x7f6e600a5c90 con 0x7f6e600a2580 2024-09-16T14:48:49.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:49.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:49.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:49.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient(hunting): my global_id is 144292 2024-09-16T14:48:49.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e44003530 con 0x7f6e600a2580 2024-09-16T14:48:49.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906078791 0 0) 0x7f6e60124c10 con 0x7f6e600a1a90 2024-09-16T14:48:49.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:49.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:49.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:49.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient(hunting): my global_id is 124494 2024-09-16T14:48:49.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e600a5c90 con 0x7f6e600a1a90 2024-09-16T14:48:49.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3869950697 0 0) 0x7f6e44003530 con 0x7f6e600a2580 2024-09-16T14:48:49.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e60124c10 con 0x7f6e600a2580 2024-09-16T14:48:49.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e58001f60 con 0x7f6e600a2580 2024-09-16T14:48:49.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:49.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:49.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient(hunting): dump: 2024-09-16T14:48:49.344 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:49.344 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:49.344 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:49.344 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:49.344 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:49.344 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:49.345 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:49.345 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:49.345 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:49.345 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:49.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:49.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4031002764 0 0) 0x7f6e60124c10 con 0x7f6e600a2580 2024-09-16T14:48:49.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 >> v1:172.21.15.26:6790/0 conn(0x7f6e6009fd40 legacy=0x7f6e60120f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:49.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 >> v1:172.21.15.26:6789/0 conn(0x7f6e600a1a90 legacy=0x7f6e60121650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:49.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:49.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 1 monclient: found mon.b 2024-09-16T14:48:49.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:49.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:49.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e60123080 con 0x7f6e600a2580 2024-09-16T14:48:49.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:49.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:49.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:49.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 5 monclient: authenticate success, global_id 144292 2024-09-16T14:48:49.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 10 monclient: _renew_subs 2024-09-16T14:48:49.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:49.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 1 -- 172.21.15.189:0/298615587 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6009e280 con 0x7f6e600a2580 2024-09-16T14:48:49.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 10 monclient: _renew_subs 2024-09-16T14:48:49.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:49.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 1 -- 172.21.15.189:0/298615587 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e601234c0 con 0x7f6e600a2580 2024-09-16T14:48:49.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e58001430 con 0x7f6e600a2580 2024-09-16T14:48:49.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:49.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e58005120 con 0x7f6e600a2580 2024-09-16T14:48:49.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:49.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:49.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e577fe640 4 set_mon_vals no callback set 2024-09-16T14:48:49.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 10 monclient: dump: 2024-09-16T14:48:49.348 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:49.348 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:49.348 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:49.348 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:49.349 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:49.349 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:49.350 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:49.350 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:49.350 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:49.350 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:49.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:49.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:49.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:49.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6e5800fb90 con 0x7f6e600a2580 2024-09-16T14:48:49.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e68147640 1 -- 172.21.15.189:0/298615587 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e48005050 con 0x7f6e600a2580 2024-09-16T14:48:49.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(168..168 src has 73..168) v4 ==== 7591+0+0 (unknown 2752859488 0 0) 0x7f6e580445e0 con 0x7f6e600a2580 2024-09-16T14:48:49.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.325+0000 7f6e65ee3640 10 monclient: get_auth_request con 0x7f6e44036160 auth_method 0 2024-09-16T14:48:49.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.329+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6e58014a60 con 0x7f6e600a2580 2024-09-16T14:48:49.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.329+0000 7f6e567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:49.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.329+0000 7f6e567fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:49.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.489+0000 7f6e68147640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T14:48:49.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.489+0000 7f6e68147640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T14:48:49.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.489+0000 7f6e68147640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:49.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:49.489+0000 7f6e68147640 1 -- 172.21.15.189:0/298615587 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6e48005300 con 0x7f6e600a2580 2024-09-16T14:48:50.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.093+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}]=0 pool 'cold' removed v169) v1 ==== 148+0+0 (unknown 2891678725 0 0) 0x7f6e58013100 con 0x7f6e600a2580 2024-09-16T14:48:50.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.093+0000 7f6e567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T14:48:50.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.093+0000 7f6e567fc640 10 monclient: _finish_command 2 = system:0 pool 'cold' removed 2024-09-16T14:48:50.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.193+0000 7f6e68147640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T14:48:50.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.193+0000 7f6e68147640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T14:48:50.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.193+0000 7f6e68147640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:50.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.193+0000 7f6e68147640 1 -- 172.21.15.189:0/298615587 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6e48005a40 con 0x7f6e600a2580 2024-09-16T14:48:50.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.193+0000 7f6e567fc640 1 -- 172.21.15.189:0/298615587 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}]=0 pool 'cold' does not exist v169) v1 ==== 155+0+0 (unknown 2400769086 0 0) 0x7f6e58014880 con 0x7f6e600a2580 2024-09-16T14:48:50.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.193+0000 7f6e567fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T14:48:50.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.193+0000 7f6e567fc640 10 monclient: _finish_command 3 = system:0 pool 'cold' does not exist 2024-09-16T14:48:50.195 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'cold' does not exist 2024-09-16T14:48:50.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.193+0000 7f6e68147640 1 -- 172.21.15.189:0/298615587 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6e44036160 legacy=0x7f6e44038620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:50.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.193+0000 7f6e68147640 10 monclient: shutdown 2024-09-16T14:48:50.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.193+0000 7f6e68147640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:50.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.193+0000 7f6e68147640 1 -- 172.21.15.189:0/298615587 >> v1:172.21.15.189:6789/0 conn(0x7f6e600a2580 legacy=0x7f6e601d1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:50.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.193+0000 7f6e68147640 1 -- 172.21.15.189:0/298615587 shutdown_connections 2024-09-16T14:48:50.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.193+0000 7f6e68147640 1 -- 172.21.15.189:0/298615587 >> 172.21.15.189:0/298615587 conn(0x7f6e600a7a70 msgr2=0x7f6e601baae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:50.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.197+0000 7f6e68147640 1 -- 172.21.15.189:0/298615587 shutdown_connections 2024-09-16T14:48:50.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.197+0000 7f6e68147640 1 -- 172.21.15.189:0/298615587 wait complete. 2024-09-16T14:48:50.212 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set rm-compat 2024-09-16T14:48:50.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.325+0000 7f24c6856640 10 monclient: get_monmap_and_config 2024-09-16T14:48:50.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.325+0000 7f24c6856640 10 monclient: build_initial_monmap 2024-09-16T14:48:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.325+0000 7f24c6856640 10 monclient: monmap: 2024-09-16T14:48:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:50.328278+0000 2024-09-16T14:48:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:50.328278+0000 2024-09-16T14:48:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:50.339 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.325+0000 7f24c6856640 1 Processor -- start 2024-09-16T14:48:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.325+0000 7f24c6856640 1 -- start start 2024-09-16T14:48:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.325+0000 7f24c6856640 10 monclient: init 2024-09-16T14:48:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.325+0000 7f24c6856640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.325+0000 7f24c6856640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:48:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.325+0000 7f24c6856640 10 monclient(hunting): picked mon.noname-a con 0x7f24c01315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.325+0000 7f24c6856640 10 monclient(hunting): picked mon.noname-c con 0x7f24c009b320 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.325+0000 7f24c6856640 10 monclient(hunting): picked mon.noname-b con 0x7f24c0132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c009edd0 con 0x7f24c01315a0 2024-09-16T14:48:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c009bc00 con 0x7f24c009b320 2024-09-16T14:48:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c009e900 con 0x7f24c0132090 2024-09-16T14:48:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient(hunting): authenticate will time out at 1312.444117s 2024-09-16T14:48:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bf7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f24c009b320 0x7f24c0136ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46882/0 (socket says 172.21.15.189:46882) 2024-09-16T14:48:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bf7fe640 1 -- 172.21.15.189:0/1963755055 learned_addr learned my addr 172.21.15.189:0/1963755055 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c4df3640 1 -- 172.21.15.189:0/1963755055 >> v1:172.21.15.26:6790/0 conn(0x7f24c0132090 legacy=0x7f24c013a200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 1 -- 172.21.15.189:0/1963755055 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913638229 0 0) 0x7f24c009bc00 con 0x7f24c009b320 2024-09-16T14:48:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 10 monclient(hunting): my global_id is 144295 2024-09-16T14:48:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 1 -- 172.21.15.189:0/1963755055 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24a8003530 con 0x7f24c009b320 2024-09-16T14:48:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 1 -- 172.21.15.189:0/1963755055 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1334385460 0 0) 0x7f24a8003530 con 0x7f24c009b320 2024-09-16T14:48:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 1 -- 172.21.15.189:0/1963755055 >> v1:172.21.15.26:6790/0 conn(0x7f24c0132090 legacy=0x7f24c013a200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 1 -- 172.21.15.189:0/1963755055 >> v1:172.21.15.26:6789/0 conn(0x7f24c01315a0 legacy=0x7f24c0131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:50.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:50.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 1 monclient: found mon.noname-c 2024-09-16T14:48:50.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:50.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:50.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 1 -- 172.21.15.189:0/1963755055 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c009b920 con 0x7f24c009b320 2024-09-16T14:48:50.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 10 monclient: _finish_auth 0 2024-09-16T14:48:50.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:50.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:50.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 5 monclient: authenticate success, global_id 144295 2024-09-16T14:48:50.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 1 -- 172.21.15.189:0/1963755055 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24b0001620 con 0x7f24c009b320 2024-09-16T14:48:50.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:50.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:50.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:50.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 10 monclient: dump: 2024-09-16T14:48:50.343 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:50.343 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:50.343 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:50.343 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:50.343 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:50.344 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:50.344 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:50.344 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:50.344 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:50.344 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:50.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:50.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 1 -- 172.21.15.189:0/1963755055 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24b0001940 con 0x7f24c009b320 2024-09-16T14:48:50.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:50.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 1 -- 172.21.15.189:0/1963755055 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24b0001d10 con 0x7f24c009b320 2024-09-16T14:48:50.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:50.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 4 set_mon_vals no callback set 2024-09-16T14:48:50.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:50.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:50.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24beffd640 10 monclient: dump: 2024-09-16T14:48:50.345 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:50.345 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:50.345 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:50.345 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:50.345 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:50.345 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:50.345 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:50.345 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:50.345 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:50.345 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:50.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient: shutdown 2024-09-16T14:48:50.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:50.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 1 -- 172.21.15.189:0/1963755055 >> v1:172.21.15.189:6789/0 conn(0x7f24c009b320 legacy=0x7f24c0136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:50.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 1 -- 172.21.15.189:0/1963755055 shutdown_connections 2024-09-16T14:48:50.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 1 -- 172.21.15.189:0/1963755055 >> 172.21.15.189:0/1963755055 conn(0x7f24c00a3320 msgr2=0x7f24c009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:50.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 1 -- 172.21.15.189:0/1963755055 shutdown_connections 2024-09-16T14:48:50.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 1 -- 172.21.15.189:0/1963755055 wait complete. 2024-09-16T14:48:50.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient: build_initial_monmap 2024-09-16T14:48:50.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient: monmap: 2024-09-16T14:48:50.346 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:50.346 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:50.346 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:50.346 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:50.346 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:50.346 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:50.346 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:50.347 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 1 Processor -- start 2024-09-16T14:48:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 1 -- start start 2024-09-16T14:48:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient: init 2024-09-16T14:48:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:48:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient(hunting): picked mon.noname-a con 0x7f24c01315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient(hunting): picked mon.noname-b con 0x7f24c0132090 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient(hunting): picked mon.noname-c con 0x7f24c01208f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c009cd20 con 0x7f24c01315a0 2024-09-16T14:48:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c009a960 con 0x7f24c0132090 2024-09-16T14:48:50.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c012f780 con 0x7f24c01208f0 2024-09-16T14:48:50.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:50.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient(hunting): authenticate will time out at 1312.446850s 2024-09-16T14:48:50.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bf7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f24c0132090 0x7f24c01201e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46886/0 (socket says 172.21.15.189:46886) 2024-09-16T14:48:50.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bf7fe640 1 -- 172.21.15.189:0/3675032505 learned_addr learned my addr 172.21.15.189:0/3675032505 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:50.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c4df3640 1 -- 172.21.15.189:0/3675032505 >> v1:172.21.15.26:6790/0 conn(0x7f24c01208f0 legacy=0x7f24c01d1930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:50.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 1 -- 172.21.15.189:0/3675032505 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839917459 0 0) 0x7f24c009b920 con 0x7f24c0132090 2024-09-16T14:48:50.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:50.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:50.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:50.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 10 monclient(hunting): my global_id is 144298 2024-09-16T14:48:50.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 1 -- 172.21.15.189:0/3675032505 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24ac003530 con 0x7f24c0132090 2024-09-16T14:48:50.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 1 -- 172.21.15.189:0/3675032505 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2598071795 0 0) 0x7f24ac003530 con 0x7f24c0132090 2024-09-16T14:48:50.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 1 -- 172.21.15.189:0/3675032505 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24c009b920 con 0x7f24c0132090 2024-09-16T14:48:50.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 1 -- 172.21.15.189:0/3675032505 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24b0001440 con 0x7f24c0132090 2024-09-16T14:48:50.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:50.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:50.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 10 monclient(hunting): dump: 2024-09-16T14:48:50.349 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:50.349 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:50.349 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:50.349 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:50.349 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:50.349 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:50.349 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:50.349 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:50.350 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:50.352 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:50.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:50.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 1 -- 172.21.15.189:0/3675032505 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2680969932 0 0) 0x7f24c009b920 con 0x7f24c0132090 2024-09-16T14:48:50.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 1 -- 172.21.15.189:0/3675032505 >> v1:172.21.15.26:6790/0 conn(0x7f24c01208f0 legacy=0x7f24c01d1930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:50.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 1 -- 172.21.15.189:0/3675032505 >> v1:172.21.15.26:6789/0 conn(0x7f24c01315a0 legacy=0x7f24c011fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:50.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 10 monclient: _finish_hunting 0 2024-09-16T14:48:50.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 1 monclient: found mon.b 2024-09-16T14:48:50.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:50.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:50.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 1 -- 172.21.15.189:0/3675032505 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c00a0f60 con 0x7f24c0132090 2024-09-16T14:48:50.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 10 monclient: _finish_auth 0 2024-09-16T14:48:50.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 10 monclient: _check_auth_tickets 2024-09-16T14:48:50.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:50.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 5 monclient: authenticate success, global_id 144298 2024-09-16T14:48:50.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient: _renew_subs 2024-09-16T14:48:50.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:50.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 1 -- 172.21.15.189:0/3675032505 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c006b380 con 0x7f24c0132090 2024-09-16T14:48:50.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient: _renew_subs 2024-09-16T14:48:50.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:50.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24c6856640 1 -- 172.21.15.189:0/3675032505 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c0123160 con 0x7f24c0132090 2024-09-16T14:48:50.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 1 -- 172.21.15.189:0/3675032505 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24b0004c00 con 0x7f24c0132090 2024-09-16T14:48:50.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:50.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 1 -- 172.21.15.189:0/3675032505 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24b0004f10 con 0x7f24c0132090 2024-09-16T14:48:50.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:50.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:50.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24bcff9640 10 monclient: dump: 2024-09-16T14:48:50.354 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:50.354 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:50.354 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:50.354 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:50.354 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:50.354 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:50.354 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:50.355 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:50.355 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:50.355 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:50.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.329+0000 7f24be7fc640 4 set_mon_vals no callback set 2024-09-16T14:48:50.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.333+0000 7f24c6856640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:50.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.333+0000 7f24c6856640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:50.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.333+0000 7f24c6856640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:50.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.333+0000 7f24c6856640 1 -- 172.21.15.189:0/3675032505 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a0005050 con 0x7f24c0132090 2024-09-16T14:48:50.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.333+0000 7f24bcff9640 1 -- 172.21.15.189:0/3675032505 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f24b000f980 con 0x7f24c0132090 2024-09-16T14:48:50.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.333+0000 7f24bcff9640 1 -- 172.21.15.189:0/3675032505 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(169..169 src has 73..169) v4 ==== 6784+0+0 (unknown 2393726062 0 0) 0x7f24b0043e50 con 0x7f24c0132090 2024-09-16T14:48:50.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.333+0000 7f24bffff640 10 monclient: get_auth_request con 0x7f24ac036060 auth_method 0 2024-09-16T14:48:50.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.337+0000 7f24bcff9640 1 -- 172.21.15.189:0/3675032505 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f24b00146a0 con 0x7f24c0132090 2024-09-16T14:48:50.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.337+0000 7f24bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:50.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.337+0000 7f24bcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:50.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.497+0000 7f24c6856640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:48:50.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.497+0000 7f24c6856640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:48:50.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.497+0000 7f24c6856640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:50.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:50.497+0000 7f24c6856640 1 -- 172.21.15.189:0/3675032505 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f24a0005300 con 0x7f24c0132090 2024-09-16T14:48:51.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:51.097+0000 7f24bcff9640 1 -- 172.21.15.189:0/3675032505 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v170) v1 ==== 78+0+0 (unknown 1576154579 0 0) 0x7f24b0012d40 con 0x7f24c0132090 2024-09-16T14:48:51.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:51.097+0000 7f24bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:48:51.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:51.097+0000 7f24bcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:51.213+0000 7f24c6856640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:48:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:51.213+0000 7f24c6856640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:48:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:51.213+0000 7f24c6856640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:51.213+0000 7f24c6856640 1 -- 172.21.15.189:0/3675032505 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f24a0005850 con 0x7f24c0132090 2024-09-16T14:48:52.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.113+0000 7f24bcff9640 1 -- 172.21.15.189:0/3675032505 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v171) v1 ==== 78+0+0 (unknown 473137142 0 0) 0x7f24b00144c0 con 0x7f24c0132090 2024-09-16T14:48:52.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.113+0000 7f24bcff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:48:52.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.113+0000 7f24bcff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:48:52.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.117+0000 7f24c6856640 1 -- 172.21.15.189:0/3675032505 >> v1:172.21.15.189:6812/3196161582 conn(0x7f24ac036060 legacy=0x7f24ac038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:52.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.117+0000 7f24c6856640 10 monclient: shutdown 2024-09-16T14:48:52.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.117+0000 7f24c6856640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:52.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.117+0000 7f24c6856640 1 -- 172.21.15.189:0/3675032505 >> v1:172.21.15.189:6789/0 conn(0x7f24c0132090 legacy=0x7f24c01201e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:52.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.121+0000 7f24c6856640 1 -- 172.21.15.189:0/3675032505 shutdown_connections 2024-09-16T14:48:52.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.121+0000 7f24c6856640 1 -- 172.21.15.189:0/3675032505 >> 172.21.15.189:0/3675032505 conn(0x7f24c00a3320 msgr2=0x7f24c00a1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:52.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.121+0000 7f24c6856640 1 -- 172.21.15.189:0/3675032505 shutdown_connections 2024-09-16T14:48:52.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.121+0000 7f24c6856640 1 -- 172.21.15.189:0/3675032505 wait complete. 2024-09-16T14:48:52.146 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set create-compat 2024-09-16T14:48:52.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 10 monclient: get_monmap_and_config 2024-09-16T14:48:52.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 10 monclient: build_initial_monmap 2024-09-16T14:48:52.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 10 monclient: monmap: 2024-09-16T14:48:52.266 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:52.266 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:52.266 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:52.266675+0000 2024-09-16T14:48:52.267 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:52.266675+0000 2024-09-16T14:48:52.267 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:52.267 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:52.267 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:52.267 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:52.267 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:52.267 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:52.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 1 Processor -- start 2024-09-16T14:48:52.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 1 -- start start 2024-09-16T14:48:52.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 10 monclient: init 2024-09-16T14:48:52.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:52.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:52.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 10 monclient(hunting): picked mon.noname-a con 0x7f692809df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:52.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 10 monclient(hunting): picked mon.noname-b con 0x7f692809e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:52.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 10 monclient(hunting): picked mon.noname-c con 0x7f692809e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:52.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692809ee20 con 0x7f692809df10 2024-09-16T14:48:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692809f100 con 0x7f692809e740 2024-09-16T14:48:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692809f3e0 con 0x7f692809e2f0 2024-09-16T14:48:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 10 monclient(hunting): authenticate will time out at 1314.382284s 2024-09-16T14:48:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692d917640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f692809e740 0x7f692813b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46896/0 (socket says 172.21.15.189:46896) 2024-09-16T14:48:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692d917640 1 -- 172.21.15.189:0/377827566 learned_addr learned my addr 172.21.15.189:0/377827566 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692c915640 1 -- 172.21.15.189:0/377827566 >> v1:172.21.15.26:6790/0 conn(0x7f692809e2f0 legacy=0x7f69281378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 1 -- 172.21.15.189:0/377827566 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841036328 0 0) 0x7f692809f100 con 0x7f692809e740 2024-09-16T14:48:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 10 monclient(hunting): my global_id is 144301 2024-09-16T14:48:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 1 -- 172.21.15.189:0/377827566 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6914003530 con 0x7f692809e740 2024-09-16T14:48:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 1 -- 172.21.15.189:0/377827566 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1615303698 0 0) 0x7f6914003530 con 0x7f692809e740 2024-09-16T14:48:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 1 -- 172.21.15.189:0/377827566 >> v1:172.21.15.26:6790/0 conn(0x7f692809e2f0 legacy=0x7f69281378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 1 -- 172.21.15.189:0/377827566 >> v1:172.21.15.26:6789/0 conn(0x7f692809df10 legacy=0x7f692809d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 1 monclient: found mon.noname-c 2024-09-16T14:48:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 1 -- 172.21.15.189:0/377827566 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692809c690 con 0x7f692809e740 2024-09-16T14:48:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 10 monclient: _finish_auth 0 2024-09-16T14:48:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 1 -- 172.21.15.189:0/377827566 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f69200019a0 con 0x7f692809e740 2024-09-16T14:48:52.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 5 monclient: authenticate success, global_id 144301 2024-09-16T14:48:52.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:52.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:52.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:52.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 10 monclient: dump: 2024-09-16T14:48:52.274 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:52.274 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:52.275 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 1 -- 172.21.15.189:0/377827566 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6920001cc0 con 0x7f692809e740 2024-09-16T14:48:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 1 -- 172.21.15.189:0/377827566 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6920002050 con 0x7f692809e740 2024-09-16T14:48:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 4 set_mon_vals no callback set 2024-09-16T14:48:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f691bfff640 10 monclient: dump: 2024-09-16T14:48:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:52.277 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 10 monclient: shutdown 2024-09-16T14:48:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 1 -- 172.21.15.189:0/377827566 >> v1:172.21.15.189:6789/0 conn(0x7f692809e740 legacy=0x7f692813b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 1 -- 172.21.15.189:0/377827566 shutdown_connections 2024-09-16T14:48:52.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.265+0000 7f692f37a640 1 -- 172.21.15.189:0/377827566 >> 172.21.15.189:0/377827566 conn(0x7f69280a3320 msgr2=0x7f692809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 1 -- 172.21.15.189:0/377827566 shutdown_connections 2024-09-16T14:48:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 1 -- 172.21.15.189:0/377827566 wait complete. 2024-09-16T14:48:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 monclient: build_initial_monmap 2024-09-16T14:48:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 monclient: monmap: 2024-09-16T14:48:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:52.278 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:52.279 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:52.279 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:52.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 1 Processor -- start 2024-09-16T14:48:52.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 1 -- start start 2024-09-16T14:48:52.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 monclient: init 2024-09-16T14:48:52.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:52.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:52.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 monclient(hunting): picked mon.noname-b con 0x7f692809df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:52.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 monclient(hunting): picked mon.noname-a con 0x7f692809e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:52.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 monclient(hunting): picked mon.noname-c con 0x7f692809e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:52.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692806f9a0 con 0x7f692809e2f0 2024-09-16T14:48:52.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692809b010 con 0x7f692809df10 2024-09-16T14:48:52.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692809bd00 con 0x7f692809e740 2024-09-16T14:48:52.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:52.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 monclient(hunting): authenticate will time out at 1314.385612s 2024-09-16T14:48:52.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692d116640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f692809df10 0x7f692809a1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46912/0 (socket says 172.21.15.189:46912) 2024-09-16T14:48:52.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692d116640 1 -- 172.21.15.189:0/398448325 learned_addr learned my addr 172.21.15.189:0/398448325 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:52.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692d917640 1 -- 172.21.15.189:0/398448325 >> v1:172.21.15.26:6790/0 conn(0x7f692809e740 legacy=0x7f69281d19b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:52.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 1 -- 172.21.15.189:0/398448325 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315807420 0 0) 0x7f692809b010 con 0x7f692809df10 2024-09-16T14:48:52.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:52.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:52.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:52.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 10 monclient(hunting): my global_id is 144304 2024-09-16T14:48:52.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 1 -- 172.21.15.189:0/398448325 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f690c003530 con 0x7f692809df10 2024-09-16T14:48:52.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 1 -- 172.21.15.189:0/398448325 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2258499903 0 0) 0x7f690c003530 con 0x7f692809df10 2024-09-16T14:48:52.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 1 -- 172.21.15.189:0/398448325 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f692809b010 con 0x7f692809df10 2024-09-16T14:48:52.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 1 -- 172.21.15.189:0/398448325 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f691c0018c0 con 0x7f692809df10 2024-09-16T14:48:52.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:52.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:52.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 10 monclient(hunting): dump: 2024-09-16T14:48:52.284 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:52.285 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:52.285 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:52.285 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:52.285 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:52.285 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:52.285 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:52.285 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:52.285 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:52.285 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:52.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:52.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 1 -- 172.21.15.189:0/398448325 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1150383646 0 0) 0x7f692809b010 con 0x7f692809df10 2024-09-16T14:48:52.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 1 -- 172.21.15.189:0/398448325 >> v1:172.21.15.26:6790/0 conn(0x7f692809e740 legacy=0x7f69281d19b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 1 -- 172.21.15.189:0/398448325 >> v1:172.21.15.26:6789/0 conn(0x7f692809e2f0 legacy=0x7f692809a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:48:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 1 monclient: found mon.b 2024-09-16T14:48:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 1 -- 172.21.15.189:0/398448325 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692809bf20 con 0x7f692809df10 2024-09-16T14:48:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 10 monclient: _finish_auth 0 2024-09-16T14:48:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 5 monclient: authenticate success, global_id 144304 2024-09-16T14:48:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 monclient: _renew_subs 2024-09-16T14:48:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 1 -- 172.21.15.189:0/398448325 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692811e220 con 0x7f692809df10 2024-09-16T14:48:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 monclient: _renew_subs 2024-09-16T14:48:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 1 -- 172.21.15.189:0/398448325 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f692809ba80 con 0x7f692809df10 2024-09-16T14:48:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 1 -- 172.21.15.189:0/398448325 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f691c003620 con 0x7f692809df10 2024-09-16T14:48:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 1 -- 172.21.15.189:0/398448325 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f691c0039b0 con 0x7f692809df10 2024-09-16T14:48:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 10 monclient: dump: 2024-09-16T14:48:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:52.288 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f691affd640 4 set_mon_vals no callback set 2024-09-16T14:48:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692f37a640 1 -- 172.21.15.189:0/398448325 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6928071bc0 con 0x7f692809df10 2024-09-16T14:48:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 1 -- 172.21.15.189:0/398448325 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f691c003b90 con 0x7f692809df10 2024-09-16T14:48:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f6919ffb640 1 -- 172.21.15.189:0/398448325 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(171..171 src has 73..171) v4 ==== 6386+0+0 (unknown 2228951249 0 0) 0x7f691c043c80 con 0x7f692809df10 2024-09-16T14:48:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.269+0000 7f692c915640 10 monclient: get_auth_request con 0x7f690c036030 auth_method 0 2024-09-16T14:48:52.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.273+0000 7f6919ffb640 1 -- 172.21.15.189:0/398448325 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f691c014720 con 0x7f692809df10 2024-09-16T14:48:52.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.273+0000 7f6919ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:52.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.273+0000 7f6919ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:52.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.445+0000 7f692f37a640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:52.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.445+0000 7f692f37a640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:52.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.445+0000 7f692f37a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:52.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:52.445+0000 7f692f37a640 1 -- 172.21.15.189:0/398448325 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f692811e610 con 0x7f692809df10 2024-09-16T14:48:53.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.129+0000 7f6919ffb640 1 -- 172.21.15.189:0/398448325 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v172) v1 ==== 82+0+0 (unknown 3073799927 0 0) 0x7f691c014310 con 0x7f692809df10 2024-09-16T14:48:53.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.129+0000 7f6919ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:53.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.129+0000 7f6919ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:53.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.237+0000 7f692f37a640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:53.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.237+0000 7f692f37a640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:53.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.237+0000 7f692f37a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:53.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.237+0000 7f692f37a640 1 -- 172.21.15.189:0/398448325 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f69280008e0 con 0x7f692809df10 2024-09-16T14:48:53.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.237+0000 7f6919ffb640 1 -- 172.21.15.189:0/398448325 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v172) v1 ==== 115+0+0 (unknown 3379692229 0 0) 0x7f691c0144f0 con 0x7f692809df10 2024-09-16T14:48:53.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.237+0000 7f6919ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:48:53.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.237+0000 7f6919ffb640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-16T14:48:53.240 INFO:tasks.workunit.client.0.smithi189.stderr:compat weight-set already created 2024-09-16T14:48:53.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.241+0000 7f692f37a640 1 -- 172.21.15.189:0/398448325 >> v1:172.21.15.189:6812/3196161582 conn(0x7f690c036030 legacy=0x7f690c0384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:53.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.241+0000 7f692f37a640 10 monclient: shutdown 2024-09-16T14:48:53.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.241+0000 7f692f37a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:53.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.241+0000 7f692f37a640 1 -- 172.21.15.189:0/398448325 >> v1:172.21.15.189:6789/0 conn(0x7f692809df10 legacy=0x7f692809a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:53.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.241+0000 7f692f37a640 1 -- 172.21.15.189:0/398448325 shutdown_connections 2024-09-16T14:48:53.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.241+0000 7f692f37a640 1 -- 172.21.15.189:0/398448325 >> 172.21.15.189:0/398448325 conn(0x7f69280a3320 msgr2=0x7f69280a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:53.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.241+0000 7f692f37a640 1 -- 172.21.15.189:0/398448325 shutdown_connections 2024-09-16T14:48:53.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.241+0000 7f692f37a640 1 -- 172.21.15.189:0/398448325 wait complete. 2024-09-16T14:48:53.257 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket fooo host 2024-09-16T14:48:53.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 10 monclient: get_monmap_and_config 2024-09-16T14:48:53.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 10 monclient: build_initial_monmap 2024-09-16T14:48:53.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 10 monclient: monmap: 2024-09-16T14:48:53.378 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:53.378 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:53.378 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:53.378410+0000 2024-09-16T14:48:53.378 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:53.378410+0000 2024-09-16T14:48:53.378 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:53.378 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:53.378 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:53.379 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:53.379 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:53.379 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:53.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 1 Processor -- start 2024-09-16T14:48:53.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 1 -- start start 2024-09-16T14:48:53.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 10 monclient: init 2024-09-16T14:48:53.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:53.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:53.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 10 monclient(hunting): picked mon.noname-c con 0x7fd48809df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:53.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 10 monclient(hunting): picked mon.noname-a con 0x7fd48809e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:53.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 10 monclient(hunting): picked mon.noname-b con 0x7fd48809e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:53.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48809ee20 con 0x7fd48809e2f0 2024-09-16T14:48:53.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48809f100 con 0x7fd48809df10 2024-09-16T14:48:53.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48809f3e0 con 0x7fd48809e740 2024-09-16T14:48:53.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:53.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 10 monclient(hunting): authenticate will time out at 1315.494235s 2024-09-16T14:48:53.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48d18d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd48809df10 0x7fd48809d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46920/0 (socket says 172.21.15.189:46920) 2024-09-16T14:48:53.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48d18d640 1 -- 172.21.15.189:0/3067577825 learned_addr learned my addr 172.21.15.189:0/3067577825 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:53.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48d98e640 1 -- 172.21.15.189:0/3067577825 >> v1:172.21.15.26:6790/0 conn(0x7fd48809e740 legacy=0x7fd48813b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:53.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 1 -- 172.21.15.189:0/3067577825 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261064688 0 0) 0x7fd48809f100 con 0x7fd48809df10 2024-09-16T14:48:53.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:53.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:53.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:53.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 10 monclient(hunting): my global_id is 144307 2024-09-16T14:48:53.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 1 -- 172.21.15.189:0/3067577825 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd474003530 con 0x7fd48809df10 2024-09-16T14:48:53.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 1 -- 172.21.15.189:0/3067577825 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 278014663 0 0) 0x7fd474003530 con 0x7fd48809df10 2024-09-16T14:48:53.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 1 -- 172.21.15.189:0/3067577825 >> v1:172.21.15.26:6790/0 conn(0x7fd48809e740 legacy=0x7fd48813b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:53.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 1 -- 172.21.15.189:0/3067577825 >> v1:172.21.15.26:6789/0 conn(0x7fd48809e2f0 legacy=0x7fd4881378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:53.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:53.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 1 monclient: found mon.noname-c 2024-09-16T14:48:53.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:53.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:53.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 1 -- 172.21.15.189:0/3067577825 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48809c690 con 0x7fd48809df10 2024-09-16T14:48:53.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 10 monclient: _finish_auth 0 2024-09-16T14:48:53.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:53.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:53.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 5 monclient: authenticate success, global_id 144307 2024-09-16T14:48:53.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 1 -- 172.21.15.189:0/3067577825 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd478001550 con 0x7fd48809df10 2024-09-16T14:48:53.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:53.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:53.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:53.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 10 monclient: dump: 2024-09-16T14:48:53.390 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:53.390 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:53.390 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:53.390 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:53.390 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:53.390 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:53.390 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:53.391 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:53.391 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:53.391 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:53.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:53.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 1 -- 172.21.15.189:0/3067577825 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd478001870 con 0x7fd48809df10 2024-09-16T14:48:53.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:53.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:53.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 4 set_mon_vals no callback set 2024-09-16T14:48:53.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 1 -- 172.21.15.189:0/3067577825 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd478001c00 con 0x7fd48809df10 2024-09-16T14:48:53.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:53.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:53.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd47ffff640 10 monclient: dump: 2024-09-16T14:48:53.392 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:53.392 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:53.392 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:53.392 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:53.392 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:53.392 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:53.393 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:53.393 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:53.393 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:53.393 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:53.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 10 monclient: shutdown 2024-09-16T14:48:53.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:53.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 1 -- 172.21.15.189:0/3067577825 >> v1:172.21.15.189:6789/0 conn(0x7fd48809df10 legacy=0x7fd48809d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:53.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 1 -- 172.21.15.189:0/3067577825 shutdown_connections 2024-09-16T14:48:53.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 1 -- 172.21.15.189:0/3067577825 >> 172.21.15.189:0/3067577825 conn(0x7fd4880a3320 msgr2=0x7fd48809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:53.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 1 -- 172.21.15.189:0/3067577825 shutdown_connections 2024-09-16T14:48:53.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.377+0000 7fd48f3f1640 1 -- 172.21.15.189:0/3067577825 wait complete. 2024-09-16T14:48:53.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 monclient: build_initial_monmap 2024-09-16T14:48:53.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 monclient: monmap: 2024-09-16T14:48:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:53.394 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 1 Processor -- start 2024-09-16T14:48:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 1 -- start start 2024-09-16T14:48:53.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 monclient: init 2024-09-16T14:48:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 monclient(hunting): picked mon.noname-b con 0x7fd48809df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 monclient(hunting): picked mon.noname-c con 0x7fd48809e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 monclient(hunting): picked mon.noname-a con 0x7fd48809e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48806b380 con 0x7fd48809e740 2024-09-16T14:48:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48806fa80 con 0x7fd48809df10 2024-09-16T14:48:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48811e570 con 0x7fd48809e2f0 2024-09-16T14:48:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 monclient(hunting): authenticate will time out at 1315.497171s 2024-09-16T14:48:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48d18d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd48809df10 0x7fd48811d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46924/0 (socket says 172.21.15.189:46924) 2024-09-16T14:48:53.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48d18d640 1 -- 172.21.15.189:0/2805786630 learned_addr learned my addr 172.21.15.189:0/2805786630 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48c98c640 1 -- 172.21.15.189:0/2805786630 >> v1:172.21.15.26:6790/0 conn(0x7fd48809e2f0 legacy=0x7fd48811de60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 1 -- 172.21.15.189:0/2805786630 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794628980 0 0) 0x7fd48809c690 con 0x7fd48809df10 2024-09-16T14:48:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 10 monclient(hunting): my global_id is 144310 2024-09-16T14:48:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 1 -- 172.21.15.189:0/2805786630 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd46c003530 con 0x7fd48809df10 2024-09-16T14:48:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 1 -- 172.21.15.189:0/2805786630 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2128541216 0 0) 0x7fd46c003530 con 0x7fd48809df10 2024-09-16T14:48:53.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 1 -- 172.21.15.189:0/2805786630 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd48809c690 con 0x7fd48809df10 2024-09-16T14:48:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 1 -- 172.21.15.189:0/2805786630 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd4780013f0 con 0x7fd48809df10 2024-09-16T14:48:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 10 monclient(hunting): dump: 2024-09-16T14:48:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:53.397 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:53.398 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 1 -- 172.21.15.189:0/2805786630 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2551928367 0 0) 0x7fd48809c690 con 0x7fd48809df10 2024-09-16T14:48:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 1 -- 172.21.15.189:0/2805786630 >> v1:172.21.15.26:6790/0 conn(0x7fd48809e2f0 legacy=0x7fd48811de60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 1 -- 172.21.15.189:0/2805786630 >> v1:172.21.15.26:6789/0 conn(0x7fd48809e740 legacy=0x7fd4881d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:48:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 1 monclient: found mon.b 2024-09-16T14:48:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 1 -- 172.21.15.189:0/2805786630 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48806cee0 con 0x7fd48809df10 2024-09-16T14:48:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 10 monclient: _finish_auth 0 2024-09-16T14:48:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:53.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 5 monclient: authenticate success, global_id 144310 2024-09-16T14:48:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 monclient: _renew_subs 2024-09-16T14:48:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 1 -- 172.21.15.189:0/2805786630 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd488120d40 con 0x7fd48809df10 2024-09-16T14:48:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 monclient: _renew_subs 2024-09-16T14:48:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 1 -- 172.21.15.189:0/2805786630 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd48811e970 con 0x7fd48809df10 2024-09-16T14:48:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 1 -- 172.21.15.189:0/2805786630 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd478004aa0 con 0x7fd48809df10 2024-09-16T14:48:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 1 -- 172.21.15.189:0/2805786630 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd478004db0 con 0x7fd48809df10 2024-09-16T14:48:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47f7fe640 4 set_mon_vals no callback set 2024-09-16T14:48:53.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 10 monclient: dump: 2024-09-16T14:48:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:53.400 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:53.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:53.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:53.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:53.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 1 -- 172.21.15.189:0/2805786630 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd47800f820 con 0x7fd48809df10 2024-09-16T14:48:53.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48f3f1640 1 -- 172.21.15.189:0/2805786630 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd468005050 con 0x7fd48809df10 2024-09-16T14:48:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd47dffb640 1 -- 172.21.15.189:0/2805786630 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(172..172 src has 73..172) v4 ==== 6714+0+0 (unknown 2237892603 0 0) 0x7fd478043dd0 con 0x7fd48809df10 2024-09-16T14:48:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.381+0000 7fd48c98c640 10 monclient: get_auth_request con 0x7fd46c036030 auth_method 0 2024-09-16T14:48:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.385+0000 7fd47dffb640 1 -- 172.21.15.189:0/2805786630 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd4780145c0 con 0x7fd48809df10 2024-09-16T14:48:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.385+0000 7fd47dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:53.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.385+0000 7fd47dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:53.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.549+0000 7fd48f3f1640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T14:48:53.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.549+0000 7fd48f3f1640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T14:48:53.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.549+0000 7fd48f3f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:53.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:53.549+0000 7fd48f3f1640 1 -- 172.21.15.189:0/2805786630 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"} v 0) v1 -- 0x7fd468005880 con 0x7fd48809df10 2024-09-16T14:48:54.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.129+0000 7fd47dffb640 1 -- 172.21.15.189:0/2805786630 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}]=0 added bucket fooo type host to crush map v173) v1 ==== 140+0+0 (unknown 3737524602 0 0) 0x7fd478012c60 con 0x7fd48809df10 2024-09-16T14:48:54.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.129+0000 7fd47dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T14:48:54.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.129+0000 7fd47dffb640 10 monclient: _finish_command 2 = system:0 added bucket fooo type host to crush map 2024-09-16T14:48:54.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.237+0000 7fd48f3f1640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T14:48:54.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.237+0000 7fd48f3f1640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T14:48:54.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.237+0000 7fd48f3f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:54.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.237+0000 7fd48f3f1640 1 -- 172.21.15.189:0/2805786630 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"} v 0) v1 -- 0x7fd468005300 con 0x7fd48809df10 2024-09-16T14:48:54.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.237+0000 7fd47dffb640 1 -- 172.21.15.189:0/2805786630 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}]=0 bucket 'fooo' already exists v173) v1 ==== 128+0+0 (unknown 351310710 0 0) 0x7fd4780143e0 con 0x7fd48809df10 2024-09-16T14:48:54.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.237+0000 7fd47dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T14:48:54.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.237+0000 7fd47dffb640 10 monclient: _finish_command 3 = system:0 bucket 'fooo' already exists 2024-09-16T14:48:54.240 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'fooo' already exists 2024-09-16T14:48:54.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.241+0000 7fd48f3f1640 1 -- 172.21.15.189:0/2805786630 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd46c036030 legacy=0x7fd46c0384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:54.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.241+0000 7fd48f3f1640 10 monclient: shutdown 2024-09-16T14:48:54.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.241+0000 7fd48f3f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:54.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.241+0000 7fd48f3f1640 1 -- 172.21.15.189:0/2805786630 >> v1:172.21.15.189:6789/0 conn(0x7fd48809df10 legacy=0x7fd48811d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:54.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.241+0000 7fd48f3f1640 1 -- 172.21.15.189:0/2805786630 shutdown_connections 2024-09-16T14:48:54.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.241+0000 7fd48f3f1640 1 -- 172.21.15.189:0/2805786630 >> 172.21.15.189:0/2805786630 conn(0x7fd4880a3320 msgr2=0x7fd48806d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:54.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.241+0000 7fd48f3f1640 1 -- 172.21.15.189:0/2805786630 shutdown_connections 2024-09-16T14:48:54.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.241+0000 7fd48f3f1640 1 -- 172.21.15.189:0/2805786630 wait complete. 2024-09-16T14:48:54.257 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush move fooo root=default 2024-09-16T14:48:54.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 10 monclient: get_monmap_and_config 2024-09-16T14:48:54.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 10 monclient: build_initial_monmap 2024-09-16T14:48:54.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 10 monclient: monmap: 2024-09-16T14:48:54.373 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:54.373 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:54.374 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:54.373833+0000 2024-09-16T14:48:54.374 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:54.373833+0000 2024-09-16T14:48:54.374 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:54.374 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:54.374 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:54.374 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:54.374 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:54.374 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:54.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 1 Processor -- start 2024-09-16T14:48:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 1 -- start start 2024-09-16T14:48:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 10 monclient: init 2024-09-16T14:48:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 10 monclient(hunting): picked mon.noname-b con 0x7fc4a81315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 10 monclient(hunting): picked mon.noname-c con 0x7fc4a809b320 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 10 monclient(hunting): picked mon.noname-a con 0x7fc4a8132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a809edd0 con 0x7fc4a8132090 2024-09-16T14:48:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a809bc00 con 0x7fc4a809b320 2024-09-16T14:48:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a809e900 con 0x7fc4a81315a0 2024-09-16T14:48:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 10 monclient(hunting): authenticate will time out at 1316.489523s 2024-09-16T14:48:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4ac9dd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc4a809b320 0x7fc4a8136ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46928/0 (socket says 172.21.15.189:46928) 2024-09-16T14:48:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4ac9dd640 1 -- 172.21.15.189:0/4022947010 learned_addr learned my addr 172.21.15.189:0/4022947010 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4ad1de640 1 -- 172.21.15.189:0/4022947010 >> v1:172.21.15.26:6790/0 conn(0x7fc4a81315a0 legacy=0x7fc4a8131980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 1 -- 172.21.15.189:0/4022947010 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140973410 0 0) 0x7fc4a809bc00 con 0x7fc4a809b320 2024-09-16T14:48:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 10 monclient(hunting): my global_id is 144313 2024-09-16T14:48:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 1 -- 172.21.15.189:0/4022947010 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc494003530 con 0x7fc4a809b320 2024-09-16T14:48:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 1 -- 172.21.15.189:0/4022947010 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3666998207 0 0) 0x7fc494003530 con 0x7fc4a809b320 2024-09-16T14:48:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 1 -- 172.21.15.189:0/4022947010 >> v1:172.21.15.26:6790/0 conn(0x7fc4a81315a0 legacy=0x7fc4a8131980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 1 -- 172.21.15.189:0/4022947010 >> v1:172.21.15.26:6789/0 conn(0x7fc4a8132090 legacy=0x7fc4a813a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 1 monclient: found mon.noname-c 2024-09-16T14:48:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 1 -- 172.21.15.189:0/4022947010 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a809b920 con 0x7fc4a809b320 2024-09-16T14:48:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 10 monclient: _finish_auth 0 2024-09-16T14:48:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 1 -- 172.21.15.189:0/4022947010 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc498001620 con 0x7fc4a809b320 2024-09-16T14:48:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 5 monclient: authenticate success, global_id 144313 2024-09-16T14:48:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 10 monclient: dump: 2024-09-16T14:48:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:54.384 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 1 -- 172.21.15.189:0/4022947010 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc498001940 con 0x7fc4a809b320 2024-09-16T14:48:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 1 -- 172.21.15.189:0/4022947010 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc498001cd0 con 0x7fc4a809b320 2024-09-16T14:48:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 4 set_mon_vals no callback set 2024-09-16T14:48:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc49ffff640 10 monclient: dump: 2024-09-16T14:48:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:54.386 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 10 monclient: shutdown 2024-09-16T14:48:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 1 -- 172.21.15.189:0/4022947010 >> v1:172.21.15.189:6789/0 conn(0x7fc4a809b320 legacy=0x7fc4a8136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 1 -- 172.21.15.189:0/4022947010 shutdown_connections 2024-09-16T14:48:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 1 -- 172.21.15.189:0/4022947010 >> 172.21.15.189:0/4022947010 conn(0x7fc4a80a3320 msgr2=0x7fc4a809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 1 -- 172.21.15.189:0/4022947010 shutdown_connections 2024-09-16T14:48:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 1 -- 172.21.15.189:0/4022947010 wait complete. 2024-09-16T14:48:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 10 monclient: build_initial_monmap 2024-09-16T14:48:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.373+0000 7fc4af442640 10 monclient: monmap: 2024-09-16T14:48:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:54.387 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 1 Processor -- start 2024-09-16T14:48:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 1 -- start start 2024-09-16T14:48:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 10 monclient: init 2024-09-16T14:48:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:48:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 10 monclient(hunting): picked mon.noname-a con 0x7fc4a81315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 10 monclient(hunting): picked mon.noname-b con 0x7fc4a8132090 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 10 monclient(hunting): picked mon.noname-c con 0x7fc4a8130ef0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a809cd20 con 0x7fc4a81315a0 2024-09-16T14:48:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a809a170 con 0x7fc4a8132090 2024-09-16T14:48:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a806b380 con 0x7fc4a8130ef0 2024-09-16T14:48:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 10 monclient(hunting): authenticate will time out at 1316.492687s 2024-09-16T14:48:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4ac9dd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc4a8132090 0x7fc4a81ce070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46934/0 (socket says 172.21.15.189:46934) 2024-09-16T14:48:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4ac9dd640 1 -- 172.21.15.189:0/2682143751 learned_addr learned my addr 172.21.15.189:0/2682143751 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4ad9df640 1 -- 172.21.15.189:0/2682143751 >> v1:172.21.15.26:6790/0 conn(0x7fc4a8130ef0 legacy=0x7fc4a81d17a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809931609 0 0) 0x7fc4a809a170 con 0x7fc4a8132090 2024-09-16T14:48:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient(hunting): my global_id is 144316 2024-09-16T14:48:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc48c003530 con 0x7fc4a8132090 2024-09-16T14:48:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 310810988 0 0) 0x7fc48c003530 con 0x7fc4a8132090 2024-09-16T14:48:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4a809a170 con 0x7fc4a8132090 2024-09-16T14:48:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc4980015e0 con 0x7fc4a8132090 2024-09-16T14:48:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient(hunting): dump: 2024-09-16T14:48:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:54.390 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652222194 0 0) 0x7fc4a809cd20 con 0x7fc4a81315a0 2024-09-16T14:48:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:54.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient(hunting): my global_id is 124518 2024-09-16T14:48:54.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc48c003530 con 0x7fc4a81315a0 2024-09-16T14:48:54.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 320247370 0 0) 0x7fc4a809a170 con 0x7fc4a8132090 2024-09-16T14:48:54.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 >> v1:172.21.15.26:6790/0 conn(0x7fc4a8130ef0 legacy=0x7fc4a81d17a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:54.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 >> v1:172.21.15.26:6789/0 conn(0x7fc4a81315a0 legacy=0x7fc4a81307e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:54.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:48:54.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 1 monclient: found mon.b 2024-09-16T14:48:54.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:54.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:54.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a811f1b0 con 0x7fc4a8132090 2024-09-16T14:48:54.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient: _finish_auth 0 2024-09-16T14:48:54.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:54.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 5 monclient: authenticate success, global_id 144316 2024-09-16T14:48:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 10 monclient: _renew_subs 2024-09-16T14:48:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 1 -- 172.21.15.189:0/2682143751 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a809a6c0 con 0x7fc4a8132090 2024-09-16T14:48:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 10 monclient: _renew_subs 2024-09-16T14:48:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 1 -- 172.21.15.189:0/2682143751 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a811d7a0 con 0x7fc4a8132090 2024-09-16T14:48:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc498004d40 con 0x7fc4a8132090 2024-09-16T14:48:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc4980050d0 con 0x7fc4a8132090 2024-09-16T14:48:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49f7fe640 4 set_mon_vals no callback set 2024-09-16T14:48:54.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 10 monclient: dump: 2024-09-16T14:48:54.393 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:54.393 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:54.393 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:54.393 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:54.393 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:54.393 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:54.393 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:54.393 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:54.393 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:54.393 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:54.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:54.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:54.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:54.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4af442640 1 -- 172.21.15.189:0/2682143751 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc488005050 con 0x7fc4a8132090 2024-09-16T14:48:54.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc498005230 con 0x7fc4a8132090 2024-09-16T14:48:54.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(173..173 src has 73..173) v4 ==== 6717+0+0 (unknown 378385951 0 0) 0x7fc498044060 con 0x7fc4a8132090 2024-09-16T14:48:54.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.377+0000 7fc4ad1de640 10 monclient: get_auth_request con 0x7fc48c0361d0 auth_method 0 2024-09-16T14:48:54.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.381+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc498014a80 con 0x7fc4a8132090 2024-09-16T14:48:54.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.381+0000 7fc49dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:54.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.381+0000 7fc49dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:54.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.541+0000 7fc4af442640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T14:48:54.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.541+0000 7fc4af442640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T14:48:54.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.541+0000 7fc4af442640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:54.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:54.541+0000 7fc4af442640 1 -- 172.21.15.189:0/2682143751 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]} v 0) v1 -- 0x7fc488005880 con 0x7fc4a8132090 2024-09-16T14:48:55.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.149+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}]=0 moved item id -16 name 'fooo' to location {root=default} in crush map v174) v1 ==== 173+0+0 (unknown 3722164873 0 0) 0x7fc498012ef0 con 0x7fc4a8132090 2024-09-16T14:48:55.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.149+0000 7fc49dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T14:48:55.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.149+0000 7fc49dffb640 10 monclient: _finish_command 2 = system:0 moved item id -16 name 'fooo' to location {root=default} in crush map 2024-09-16T14:48:55.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.253+0000 7fc4af442640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T14:48:55.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.253+0000 7fc4af442640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T14:48:55.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.253+0000 7fc4af442640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:55.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.253+0000 7fc4af442640 1 -- 172.21.15.189:0/2682143751 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]} v 0) v1 -- 0x7fc488005300 con 0x7fc4a8132090 2024-09-16T14:48:55.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.253+0000 7fc49dffb640 1 -- 172.21.15.189:0/2682143751 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}]=0 no need to move item id -16 name 'fooo' to location {root=default} in crush map v174) v1 ==== 183+0+0 (unknown 2005285716 0 0) 0x7fc498014850 con 0x7fc4a8132090 2024-09-16T14:48:55.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.253+0000 7fc49dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T14:48:55.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.253+0000 7fc49dffb640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'fooo' to location {root=default} in crush map 2024-09-16T14:48:55.256 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id -16 name 'fooo' to location {root=default} in crush map 2024-09-16T14:48:55.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.257+0000 7fc4af442640 1 -- 172.21.15.189:0/2682143751 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc48c0361d0 legacy=0x7fc48c038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:55.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.257+0000 7fc4af442640 10 monclient: shutdown 2024-09-16T14:48:55.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.257+0000 7fc4af442640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:55.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.257+0000 7fc4af442640 1 -- 172.21.15.189:0/2682143751 >> v1:172.21.15.189:6789/0 conn(0x7fc4a8132090 legacy=0x7fc4a81ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:55.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.257+0000 7fc4af442640 1 -- 172.21.15.189:0/2682143751 shutdown_connections 2024-09-16T14:48:55.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.257+0000 7fc4af442640 1 -- 172.21.15.189:0/2682143751 >> 172.21.15.189:0/2682143751 conn(0x7fc4a80a3320 msgr2=0x7fc4a809dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:55.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.257+0000 7fc4af442640 1 -- 172.21.15.189:0/2682143751 shutdown_connections 2024-09-16T14:48:55.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.257+0000 7fc4af442640 1 -- 172.21.15.189:0/2682143751 wait complete. 2024-09-16T14:48:55.271 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket barr rack 2024-09-16T14:48:55.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 10 monclient: get_monmap_and_config 2024-09-16T14:48:55.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 10 monclient: build_initial_monmap 2024-09-16T14:48:55.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 10 monclient: monmap: 2024-09-16T14:48:55.389 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:55.389 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:55.390 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:55.390007+0000 2024-09-16T14:48:55.390 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:55.390007+0000 2024-09-16T14:48:55.390 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:55.390 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:55.390 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:55.390 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:55.390 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:55.390 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:55.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 1 Processor -- start 2024-09-16T14:48:55.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 1 -- start start 2024-09-16T14:48:55.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 10 monclient: init 2024-09-16T14:48:55.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:55.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:55.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 10 monclient(hunting): picked mon.noname-c con 0x7ffb480a0ea0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:55.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 10 monclient(hunting): picked mon.noname-b con 0x7ffb481325d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:55.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 10 monclient(hunting): picked mon.noname-a con 0x7ffb4809df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:55.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4806a120 con 0x7ffb4809df10 2024-09-16T14:48:55.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb480a1280 con 0x7ffb480a0ea0 2024-09-16T14:48:55.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4806adf0 con 0x7ffb481325d0 2024-09-16T14:48:55.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:55.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 10 monclient(hunting): authenticate will time out at 1317.505806s 2024-09-16T14:48:55.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb45d9b640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7ffb481325d0 legacy=0x7ffb481378f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:55.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4659c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffb480a0ea0 0x7ffb4809d800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46948/0 (socket says 172.21.15.189:46948) 2024-09-16T14:48:55.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4659c640 1 -- 172.21.15.189:0/1150089191 learned_addr learned my addr 172.21.15.189:0/1150089191 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:55.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 1 -- 172.21.15.189:0/1150089191 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651450464 0 0) 0x7ffb480a1280 con 0x7ffb480a0ea0 2024-09-16T14:48:55.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 10 monclient(hunting): _negotiate 2024-09-16T14:48:55.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:55.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:55.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 10 monclient(hunting): my global_id is 144319 2024-09-16T14:48:55.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 1 -- 172.21.15.189:0/1150089191 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb30003530 con 0x7ffb480a0ea0 2024-09-16T14:48:55.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 1 -- 172.21.15.189:0/1150089191 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 591220906 0 0) 0x7ffb30003530 con 0x7ffb480a0ea0 2024-09-16T14:48:55.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 1 -- 172.21.15.189:0/1150089191 >> v1:172.21.15.26:6790/0 conn(0x7ffb481325d0 legacy=0x7ffb481378f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:55.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 1 -- 172.21.15.189:0/1150089191 >> v1:172.21.15.26:6789/0 conn(0x7ffb4809df10 legacy=0x7ffb4813b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:55.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 10 monclient: _finish_hunting 0 2024-09-16T14:48:55.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 1 monclient: found mon.noname-c 2024-09-16T14:48:55.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:55.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:55.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 1 -- 172.21.15.189:0/1150089191 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb48132a60 con 0x7ffb480a0ea0 2024-09-16T14:48:55.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 10 monclient: _finish_auth 0 2024-09-16T14:48:55.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 10 monclient: _check_auth_tickets 2024-09-16T14:48:55.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:55.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 1 -- 172.21.15.189:0/1150089191 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb40001620 con 0x7ffb480a0ea0 2024-09-16T14:48:55.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:55.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:55.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 10 monclient: dump: 2024-09-16T14:48:55.399 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:55.399 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:55.399 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:55.399 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:55.399 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:55.399 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:55.399 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:55.400 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:55.400 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:55.400 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:55.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 5 monclient: authenticate success, global_id 144319 2024-09-16T14:48:55.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:55.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 1 -- 172.21.15.189:0/1150089191 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb40001940 con 0x7ffb480a0ea0 2024-09-16T14:48:55.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:55.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:55.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 1 -- 172.21.15.189:0/1150089191 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb40001cd0 con 0x7ffb480a0ea0 2024-09-16T14:48:55.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 4 set_mon_vals no callback set 2024-09-16T14:48:55.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:55.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:55.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4559a640 10 monclient: dump: 2024-09-16T14:48:55.401 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:55.401 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:55.401 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:55.401 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:55.401 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:55.401 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:55.401 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:55.401 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:55.401 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:55.401 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:55.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 10 monclient: shutdown 2024-09-16T14:48:55.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:55.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 1 -- 172.21.15.189:0/1150089191 >> v1:172.21.15.189:6789/0 conn(0x7ffb480a0ea0 legacy=0x7ffb4809d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:55.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 1 -- 172.21.15.189:0/1150089191 shutdown_connections 2024-09-16T14:48:55.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 1 -- 172.21.15.189:0/1150089191 >> 172.21.15.189:0/1150089191 conn(0x7ffb480a4450 msgr2=0x7ffb480a0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:55.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 1 -- 172.21.15.189:0/1150089191 shutdown_connections 2024-09-16T14:48:55.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 1 -- 172.21.15.189:0/1150089191 wait complete. 2024-09-16T14:48:55.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 10 monclient: build_initial_monmap 2024-09-16T14:48:55.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.389+0000 7ffb4c94b640 10 monclient: monmap: 2024-09-16T14:48:55.402 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:55.402 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:55.402 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:55.402 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:55.402 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:55.402 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:55.403 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:55.403 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:55.403 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:55.403 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:55.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 1 Processor -- start 2024-09-16T14:48:55.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 1 -- start start 2024-09-16T14:48:55.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 10 monclient: init 2024-09-16T14:48:55.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:55.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:55.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 10 monclient(hunting): picked mon.noname-c con 0x7ffb4809df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:55.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 10 monclient(hunting): picked mon.noname-a con 0x7ffb480a0ea0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:55.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 10 monclient(hunting): picked mon.noname-b con 0x7ffb48099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:55.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4806a5e0 con 0x7ffb480a0ea0 2024-09-16T14:48:55.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4809f580 con 0x7ffb48099a70 2024-09-16T14:48:55.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb48136ff0 con 0x7ffb4809df10 2024-09-16T14:48:55.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:55.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 10 monclient(hunting): authenticate will time out at 1317.508921s 2024-09-16T14:48:55.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb46d9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffb48099a70 0x7ffb481d6300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46956/0 (socket says 172.21.15.189:46956) 2024-09-16T14:48:55.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4659c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7ffb4809df10 legacy=0x7ffb481cdb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:48:55.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb46d9d640 1 -- 172.21.15.189:0/3240412382 learned_addr learned my addr 172.21.15.189:0/3240412382 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:55.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685954245 0 0) 0x7ffb4809f580 con 0x7ffb48099a70 2024-09-16T14:48:55.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:55.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:55.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:55.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient(hunting): my global_id is 144322 2024-09-16T14:48:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb28003530 con 0x7ffb48099a70 2024-09-16T14:48:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1881850543 0 0) 0x7ffb28003530 con 0x7ffb48099a70 2024-09-16T14:48:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb4809f580 con 0x7ffb48099a70 2024-09-16T14:48:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb38001830 con 0x7ffb48099a70 2024-09-16T14:48:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient(hunting): dump: 2024-09-16T14:48:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:55.406 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:55.406 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:55.406 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:55.406 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:55.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:55.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025918148 0 0) 0x7ffb4806a5e0 con 0x7ffb480a0ea0 2024-09-16T14:48:55.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:55.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:55.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:55.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient(hunting): my global_id is 124524 2024-09-16T14:48:55.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb28003530 con 0x7ffb480a0ea0 2024-09-16T14:48:55.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 757410557 0 0) 0x7ffb4809f580 con 0x7ffb48099a70 2024-09-16T14:48:55.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 >> v1:172.21.15.26:6790/0 conn(0x7ffb4809df10 legacy=0x7ffb481cdb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:55.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 >> v1:172.21.15.26:6789/0 conn(0x7ffb480a0ea0 legacy=0x7ffb481d2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:55.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:55.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 1 monclient: found mon.b 2024-09-16T14:48:55.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:55.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:55.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb48138930 con 0x7ffb48099a70 2024-09-16T14:48:55.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient: _finish_auth 0 2024-09-16T14:48:55.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:55.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:55.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 5 monclient: authenticate success, global_id 144322 2024-09-16T14:48:55.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 10 monclient: _renew_subs 2024-09-16T14:48:55.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:55.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 1 -- 172.21.15.189:0/3240412382 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb48070660 con 0x7ffb48099a70 2024-09-16T14:48:55.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 10 monclient: _renew_subs 2024-09-16T14:48:55.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:55.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 1 -- 172.21.15.189:0/3240412382 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb48138370 con 0x7ffb48099a70 2024-09-16T14:48:55.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb38003660 con 0x7ffb48099a70 2024-09-16T14:48:55.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:55.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb380039f0 con 0x7ffb48099a70 2024-09-16T14:48:55.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:55.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:55.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb44d99640 4 set_mon_vals no callback set 2024-09-16T14:48:55.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 10 monclient: dump: 2024-09-16T14:48:55.408 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:55.408 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:55.408 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:55.408 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:55.409 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:55.409 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:55.409 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:55.409 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:55.409 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:55.409 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:55.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:55.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:55.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:55.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4c94b640 1 -- 172.21.15.189:0/3240412382 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb2c005050 con 0x7ffb48099a70 2024-09-16T14:48:55.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ffb38003bd0 con 0x7ffb48099a70 2024-09-16T14:48:55.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(174..174 src has 73..174) v4 ==== 6905+0+0 (unknown 3472450521 0 0) 0x7ffb38043e20 con 0x7ffb48099a70 2024-09-16T14:48:55.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.393+0000 7ffb4659c640 10 monclient: get_auth_request con 0x7ffb280360e0 auth_method 0 2024-09-16T14:48:55.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.397+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ffb38012c70 con 0x7ffb48099a70 2024-09-16T14:48:55.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.397+0000 7ffb36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:55.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.397+0000 7ffb36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.561+0000 7ffb4c94b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T14:48:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.561+0000 7ffb4c94b640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T14:48:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.561+0000 7ffb4c94b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:55.561+0000 7ffb4c94b640 1 -- 172.21.15.189:0/3240412382 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"} v 0) v1 -- 0x7ffb2c005880 con 0x7ffb48099a70 2024-09-16T14:48:56.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.149+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}]=0 added bucket barr type rack to crush map v175) v1 ==== 140+0+0 (unknown 1175284939 0 0) 0x7ffb38012c70 con 0x7ffb48099a70 2024-09-16T14:48:56.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.149+0000 7ffb36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T14:48:56.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.149+0000 7ffb36ffd640 10 monclient: _finish_command 2 = system:0 added bucket barr type rack to crush map 2024-09-16T14:48:56.156 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T14:48:56.156 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T14:48:56.156 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T14:48:56.157 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T14:48:56.159 INFO:tasks.ceph.mon.c:Started 2024-09-16T14:48:56.159 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:48:56.160 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:48:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.249+0000 7ffb4c94b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T14:48:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.249+0000 7ffb4c94b640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T14:48:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.249+0000 7ffb4c94b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.249+0000 7ffb4c94b640 1 -- 172.21.15.189:0/3240412382 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"} v 0) v1 -- 0x7ffb2c005300 con 0x7ffb48099a70 2024-09-16T14:48:56.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.249+0000 7ffb36ffd640 1 -- 172.21.15.189:0/3240412382 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}]=0 bucket 'barr' already exists v175) v1 ==== 128+0+0 (unknown 1342079467 0 0) 0x7ffb38012c70 con 0x7ffb48099a70 2024-09-16T14:48:56.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.249+0000 7ffb36ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T14:48:56.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.249+0000 7ffb36ffd640 10 monclient: _finish_command 3 = system:0 bucket 'barr' already exists 2024-09-16T14:48:56.251 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'barr' already exists 2024-09-16T14:48:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.249+0000 7ffb4c94b640 1 -- 172.21.15.189:0/3240412382 >> v1:172.21.15.189:6812/3196161582 conn(0x7ffb280360e0 legacy=0x7ffb280385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.249+0000 7ffb4c94b640 10 monclient: shutdown 2024-09-16T14:48:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.249+0000 7ffb4c94b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.249+0000 7ffb4c94b640 1 -- 172.21.15.189:0/3240412382 >> v1:172.21.15.189:6789/0 conn(0x7ffb48099a70 legacy=0x7ffb481d6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.249+0000 7ffb4c94b640 1 -- 172.21.15.189:0/3240412382 shutdown_connections 2024-09-16T14:48:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.249+0000 7ffb4c94b640 1 -- 172.21.15.189:0/3240412382 >> 172.21.15.189:0/3240412382 conn(0x7ffb480a4450 msgr2=0x7ffb4809f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.249+0000 7ffb4c94b640 1 -- 172.21.15.189:0/3240412382 shutdown_connections 2024-09-16T14:48:56.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.249+0000 7ffb4c94b640 1 -- 172.21.15.189:0/3240412382 wait complete. 2024-09-16T14:48:56.268 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush move barr root=default 2024-09-16T14:48:56.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8aebcdc640 10 monclient: get_monmap_and_config 2024-09-16T14:48:56.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8aebcdc640 10 monclient: build_initial_monmap 2024-09-16T14:48:56.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8aebcdc640 10 monclient: monmap: 2024-09-16T14:48:56.286 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:56.286 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.286 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:56.286608+0000 2024-09-16T14:48:56.286 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:56.286608+0000 2024-09-16T14:48:56.286 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:56.286 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.286 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:56.286 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:56.287 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:56.287 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8aebcdc640 1 Processor -- start 2024-09-16T14:48:56.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8aebcdc640 1 -- start start 2024-09-16T14:48:56.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8aebcdc640 10 monclient: init 2024-09-16T14:48:56.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8aebcdc640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:56.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8aebcdc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:56.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8aebcdc640 10 monclient(hunting): picked mon.noname-c con 0x7f8ae40a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:56.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8aebcdc640 10 monclient(hunting): picked mon.noname-a con 0x7f8ae4099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:56.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8aebcdc640 10 monclient(hunting): picked mon.noname-b con 0x7f8ae40a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:56.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8aebcdc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae4098ea0 con 0x7f8ae4099b60 2024-09-16T14:48:56.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8aebcdc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae40997b0 con 0x7f8ae40a1c00 2024-09-16T14:48:56.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8aebcdc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae4099460 con 0x7f8ae40a26f0 2024-09-16T14:48:56.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8aebcdc640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:56.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8aebcdc640 10 monclient(hunting): authenticate will time out at 1360.406279s 2024-09-16T14:48:56.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8ae9277640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8ae4099b60 0x7f8ae4139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53806/0 (socket says 172.21.15.26:53806) 2024-09-16T14:48:56.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.282+0000 7f8ae9277640 1 -- 172.21.15.26:0/2301662692 learned_addr learned my addr 172.21.15.26:0/2301662692 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:56.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 1 -- 172.21.15.26:0/2301662692 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348662892 0 0) 0x7f8ae4098ea0 con 0x7f8ae4099b60 2024-09-16T14:48:56.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 10 monclient(hunting): my global_id is 124527 2024-09-16T14:48:56.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 1 -- 172.21.15.26:0/2301662692 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad0003530 con 0x7f8ae4099b60 2024-09-16T14:48:56.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 1 -- 172.21.15.26:0/2301662692 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 242610262 0 0) 0x7f8ad0003530 con 0x7f8ae4099b60 2024-09-16T14:48:56.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 1 -- 172.21.15.26:0/2301662692 >> v1:172.21.15.26:6790/0 conn(0x7f8ae40a26f0 legacy=0x7f8ae413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 1 -- 172.21.15.26:0/2301662692 >> v1:172.21.15.189:6789/0 conn(0x7f8ae40a1c00 legacy=0x7f8ae40a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 10 monclient: _finish_hunting 0 2024-09-16T14:48:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 1 monclient: found mon.noname-a 2024-09-16T14:48:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 1 -- 172.21.15.26:0/2301662692 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae40a34b0 con 0x7f8ae4099b60 2024-09-16T14:48:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 10 monclient: _finish_auth 0 2024-09-16T14:48:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 10 monclient: _check_auth_tickets 2024-09-16T14:48:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 1 -- 172.21.15.26:0/2301662692 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ad4001620 con 0x7f8ae4099b60 2024-09-16T14:48:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 5 monclient: authenticate success, global_id 124527 2024-09-16T14:48:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 10 monclient: dump: 2024-09-16T14:48:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.296 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 1 -- 172.21.15.26:0/2301662692 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ad4001940 con 0x7f8ae4099b60 2024-09-16T14:48:56.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:56.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 1 -- 172.21.15.26:0/2301662692 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ad4001cd0 con 0x7f8ae4099b60 2024-09-16T14:48:56.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:56.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8ae8a76640 10 monclient: dump: 2024-09-16T14:48:56.297 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.297 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.297 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.297 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.297 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.297 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.297 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.297 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.298 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.298 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:56.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 4 set_mon_vals no callback set 2024-09-16T14:48:56.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 10 monclient: shutdown 2024-09-16T14:48:56.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:56.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 1 -- 172.21.15.26:0/2301662692 >> v1:172.21.15.26:6789/0 conn(0x7f8ae4099b60 legacy=0x7f8ae4139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 1 -- 172.21.15.26:0/2301662692 shutdown_connections 2024-09-16T14:48:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 1 -- 172.21.15.26:0/2301662692 >> 172.21.15.26:0/2301662692 conn(0x7f8ae40a3d00 msgr2=0x7f8ae40a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 1 -- 172.21.15.26:0/2301662692 shutdown_connections 2024-09-16T14:48:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 1 -- 172.21.15.26:0/2301662692 wait complete. 2024-09-16T14:48:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 10 monclient: build_initial_monmap 2024-09-16T14:48:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 10 monclient: monmap: 2024-09-16T14:48:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:56.303 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 1 Processor -- start 2024-09-16T14:48:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 1 -- start start 2024-09-16T14:48:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 10 monclient: init 2024-09-16T14:48:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.290+0000 7f8aebcdc640 10 monclient(hunting): picked mon.noname-c con 0x7f8ae4099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 10 monclient(hunting): picked mon.noname-a con 0x7f8ae40a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 10 monclient(hunting): picked mon.noname-b con 0x7f8ae40a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae413bae0 con 0x7f8ae40a1c00 2024-09-16T14:48:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae409e3f0 con 0x7f8ae40a26f0 2024-09-16T14:48:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae40a0f60 con 0x7f8ae4099b60 2024-09-16T14:48:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 10 monclient(hunting): authenticate will time out at 1360.416219s 2024-09-16T14:48:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ae9a78640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8ae4099b60 0x7f8ae4139ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49136/0 (socket says 172.21.15.26:49136) 2024-09-16T14:48:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ae9a78640 1 -- 172.21.15.26:0/1167853469 learned_addr learned my addr 172.21.15.26:0/1167853469 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 1 -- 172.21.15.26:0/1167853469 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019976237 0 0) 0x7f8ae40a0f60 con 0x7f8ae4099b60 2024-09-16T14:48:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 10 monclient(hunting): my global_id is 154100 2024-09-16T14:48:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 1 -- 172.21.15.26:0/1167853469 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac8003530 con 0x7f8ae4099b60 2024-09-16T14:48:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 1 -- 172.21.15.26:0/1167853469 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2050789430 0 0) 0x7f8ac8003530 con 0x7f8ae4099b60 2024-09-16T14:48:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 1 -- 172.21.15.26:0/1167853469 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ae40a0f60 con 0x7f8ae4099b60 2024-09-16T14:48:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 1 -- 172.21.15.26:0/1167853469 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ae0001920 con 0x7f8ae4099b60 2024-09-16T14:48:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 10 monclient(hunting): dump: 2024-09-16T14:48:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.305 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 1 -- 172.21.15.26:0/1167853469 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3398837800 0 0) 0x7f8ae40a0f60 con 0x7f8ae4099b60 2024-09-16T14:48:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 1 -- 172.21.15.26:0/1167853469 >> v1:172.21.15.189:6789/0 conn(0x7f8ae40a26f0 legacy=0x7f8ae41d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 1 -- 172.21.15.26:0/1167853469 >> v1:172.21.15.26:6789/0 conn(0x7f8ae40a1c00 legacy=0x7f8ae413a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 1 monclient: found mon.c 2024-09-16T14:48:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 1 -- 172.21.15.26:0/1167853469 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae413db60 con 0x7f8ae4099b60 2024-09-16T14:48:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 5 monclient: authenticate success, global_id 154100 2024-09-16T14:48:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 10 monclient: _renew_subs 2024-09-16T14:48:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 1 -- 172.21.15.26:0/1167853469 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae413ad00 con 0x7f8ae4099b60 2024-09-16T14:48:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 10 monclient: _renew_subs 2024-09-16T14:48:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 1 -- 172.21.15.26:0/1167853469 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae413d880 con 0x7f8ae4099b60 2024-09-16T14:48:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 1 -- 172.21.15.26:0/1167853469 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ae0003600 con 0x7f8ae4099b60 2024-09-16T14:48:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 1 -- 172.21.15.26:0/1167853469 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ae0003990 con 0x7f8ae4099b60 2024-09-16T14:48:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8adbfff640 4 set_mon_vals no callback set 2024-09-16T14:48:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 10 monclient: dump: 2024-09-16T14:48:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.308 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 1 -- 172.21.15.26:0/1167853469 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8ae0003af0 con 0x7f8ae4099b60 2024-09-16T14:48:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8aebcdc640 1 -- 172.21.15.26:0/1167853469 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8acc005050 con 0x7f8ae4099b60 2024-09-16T14:48:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.294+0000 7f8ada7fc640 1 -- 172.21.15.26:0/1167853469 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(175..175 src has 73..175) v4 ==== 6933+0+0 (unknown 3869440614 0 0) 0x7f8ae0043de0 con 0x7f8ae4099b60 2024-09-16T14:48:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.298+0000 7f8ae9277640 10 monclient: get_auth_request con 0x7f8ac8036100 auth_method 0 2024-09-16T14:48:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.302+0000 7f8ada7fc640 1 -- 172.21.15.26:0/1167853469 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8ae00042c0 con 0x7f8ae4099b60 2024-09-16T14:48:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.302+0000 7f8ada7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.302+0000 7f8ada7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:56.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f5173f20640 10 monclient: get_monmap_and_config 2024-09-16T14:48:56.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f5173f20640 10 monclient: build_initial_monmap 2024-09-16T14:48:56.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f5173f20640 10 monclient: monmap: 2024-09-16T14:48:56.388 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:56.388 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.388 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:56.387952+0000 2024-09-16T14:48:56.388 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:56.387952+0000 2024-09-16T14:48:56.388 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:56.388 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:56.388 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:56.388 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:56.388 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:56.388 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f5173f20640 1 Processor -- start 2024-09-16T14:48:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f5173f20640 1 -- start start 2024-09-16T14:48:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f5173f20640 10 monclient: init 2024-09-16T14:48:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f5173f20640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f5173f20640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f5173f20640 10 monclient(hunting): picked mon.noname-b con 0x7f516c1315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f5173f20640 10 monclient(hunting): picked mon.noname-a con 0x7f516c09b320 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f5173f20640 10 monclient(hunting): picked mon.noname-c con 0x7f516c132090 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f5173f20640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516c09edd0 con 0x7f516c09b320 2024-09-16T14:48:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f5173f20640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516c09bc00 con 0x7f516c132090 2024-09-16T14:48:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f5173f20640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516c09e900 con 0x7f516c1315a0 2024-09-16T14:48:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f5173f20640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f5173f20640 10 monclient(hunting): authenticate will time out at 1318.503564s 2024-09-16T14:48:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f51724bd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f516c132090 0x7f516c13a200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46958/0 (socket says 172.21.15.189:46958) 2024-09-16T14:48:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.385+0000 7f51724bd640 1 -- 172.21.15.189:0/1531862864 learned_addr learned my addr 172.21.15.189:0/1531862864 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 1 -- 172.21.15.189:0/1531862864 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938628875 0 0) 0x7f516c09bc00 con 0x7f516c132090 2024-09-16T14:48:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient(hunting): my global_id is 144325 2024-09-16T14:48:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 1 -- 172.21.15.189:0/1531862864 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5154003530 con 0x7f516c132090 2024-09-16T14:48:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 1 -- 172.21.15.189:0/1531862864 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119730737 0 0) 0x7f516c09e900 con 0x7f516c1315a0 2024-09-16T14:48:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient(hunting): my global_id is 154103 2024-09-16T14:48:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 1 -- 172.21.15.189:0/1531862864 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f516c09bc00 con 0x7f516c1315a0 2024-09-16T14:48:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 1 -- 172.21.15.189:0/1531862864 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 918322950 0 0) 0x7f5154003530 con 0x7f516c132090 2024-09-16T14:48:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 1 -- 172.21.15.189:0/1531862864 >> v1:172.21.15.26:6790/0 conn(0x7f516c1315a0 legacy=0x7f516c131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 1 -- 172.21.15.189:0/1531862864 >> v1:172.21.15.26:6789/0 conn(0x7f516c09b320 legacy=0x7f516c136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient: _finish_hunting 0 2024-09-16T14:48:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 1 monclient: found mon.noname-c 2024-09-16T14:48:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 1 -- 172.21.15.189:0/1531862864 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c09b920 con 0x7f516c132090 2024-09-16T14:48:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient: _finish_auth 0 2024-09-16T14:48:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient: _check_auth_tickets 2024-09-16T14:48:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 1 -- 172.21.15.189:0/1531862864 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f515c0019a0 con 0x7f516c132090 2024-09-16T14:48:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 5 monclient: authenticate success, global_id 144325 2024-09-16T14:48:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient: dump: 2024-09-16T14:48:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.399 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 1 -- 172.21.15.189:0/1531862864 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f515c001c80 con 0x7f516c132090 2024-09-16T14:48:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 1 -- 172.21.15.189:0/1531862864 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f515c002010 con 0x7f516c132090 2024-09-16T14:48:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 4 set_mon_vals no callback set 2024-09-16T14:48:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5170cba640 10 monclient: dump: 2024-09-16T14:48:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.400 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 10 monclient: shutdown 2024-09-16T14:48:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 1 -- 172.21.15.189:0/1531862864 >> v1:172.21.15.189:6789/0 conn(0x7f516c132090 legacy=0x7f516c13a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 1 -- 172.21.15.189:0/1531862864 shutdown_connections 2024-09-16T14:48:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 1 -- 172.21.15.189:0/1531862864 >> 172.21.15.189:0/1531862864 conn(0x7f516c0a3320 msgr2=0x7f516c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 1 -- 172.21.15.189:0/1531862864 shutdown_connections 2024-09-16T14:48:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 1 -- 172.21.15.189:0/1531862864 wait complete. 2024-09-16T14:48:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 10 monclient: build_initial_monmap 2024-09-16T14:48:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 10 monclient: monmap: 2024-09-16T14:48:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:56.402 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:56.402 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:56.402 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:56.402 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:56.402 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:56.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 1 Processor -- start 2024-09-16T14:48:56.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 1 -- start start 2024-09-16T14:48:56.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 10 monclient: init 2024-09-16T14:48:56.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:56.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:56.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 10 monclient(hunting): picked mon.noname-a con 0x7f516c1315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:56.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 10 monclient(hunting): picked mon.noname-c con 0x7f516c132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 10 monclient(hunting): picked mon.noname-b con 0x7f516c1c4ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516c09cd20 con 0x7f516c1315a0 2024-09-16T14:48:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516c0a0bf0 con 0x7f516c1c4ff0 2024-09-16T14:48:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516c06b380 con 0x7f516c132090 2024-09-16T14:48:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 10 monclient(hunting): authenticate will time out at 1318.506769s 2024-09-16T14:48:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f51724bd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f516c1c4ff0 0x7f516c1cd800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46974/0 (socket says 172.21.15.189:46974) 2024-09-16T14:48:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f51724bd640 1 -- 172.21.15.189:0/2433742701 learned_addr learned my addr 172.21.15.189:0/2433742701 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026689016 0 0) 0x7f516c0a0bf0 con 0x7f516c1c4ff0 2024-09-16T14:48:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient(hunting): my global_id is 144328 2024-09-16T14:48:56.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5158003530 con 0x7f516c1c4ff0 2024-09-16T14:48:56.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3068015485 0 0) 0x7f5158003530 con 0x7f516c1c4ff0 2024-09-16T14:48:56.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f516c0a0bf0 con 0x7f516c1c4ff0 2024-09-16T14:48:56.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239207089 0 0) 0x7f516c06b380 con 0x7f516c132090 2024-09-16T14:48:56.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient(hunting): my global_id is 154106 2024-09-16T14:48:56.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5158003530 con 0x7f516c132090 2024-09-16T14:48:56.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f515c001f90 con 0x7f516c1c4ff0 2024-09-16T14:48:56.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:56.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient(hunting): dump: 2024-09-16T14:48:56.406 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:56.406 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.406 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.406 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.406 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.406 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:56.406 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.406 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.406 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.406 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:56.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716724233 0 0) 0x7f516c09cd20 con 0x7f516c1315a0 2024-09-16T14:48:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient(hunting): my global_id is 124533 2024-09-16T14:48:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f516c06b380 con 0x7f516c1315a0 2024-09-16T14:48:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3664321981 0 0) 0x7f516c0a0bf0 con 0x7f516c1c4ff0 2024-09-16T14:48:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 >> v1:172.21.15.26:6790/0 conn(0x7f516c132090 legacy=0x7f516c1ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 >> v1:172.21.15.26:6789/0 conn(0x7f516c1315a0 legacy=0x7f516c12f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 1 monclient: found mon.b 2024-09-16T14:48:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c11b000 con 0x7f516c1c4ff0 2024-09-16T14:48:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 5 monclient: authenticate success, global_id 144328 2024-09-16T14:48:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 10 monclient: _renew_subs 2024-09-16T14:48:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 1 -- 172.21.15.189:0/2433742701 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f516c11d340 con 0x7f516c1c4ff0 2024-09-16T14:48:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 10 monclient: _renew_subs 2024-09-16T14:48:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f5173f20640 1 -- 172.21.15.189:0/2433742701 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f516c0a1020 con 0x7f516c1c4ff0 2024-09-16T14:48:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f515c004f80 con 0x7f516c1c4ff0 2024-09-16T14:48:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f515c005290 con 0x7f516c1c4ff0 2024-09-16T14:48:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516bfff640 4 set_mon_vals no callback set 2024-09-16T14:48:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.389+0000 7f516a7fc640 10 monclient: dump: 2024-09-16T14:48:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.410 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:56.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.393+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f515c00fd00 con 0x7f516c1c4ff0 2024-09-16T14:48:56.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.393+0000 7f5173f20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:56.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.393+0000 7f5173f20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:56.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.393+0000 7f5173f20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:56.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.393+0000 7f5173f20640 1 -- 172.21.15.189:0/2433742701 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514c005050 con 0x7f516c1c4ff0 2024-09-16T14:48:56.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.393+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(175..175 src has 73..175) v4 ==== 6933+0+0 (unknown 3869440614 0 0) 0x7f515c044360 con 0x7f516c1c4ff0 2024-09-16T14:48:56.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.393+0000 7f5171cbc640 10 monclient: get_auth_request con 0x7f51580363e0 auth_method 0 2024-09-16T14:48:56.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.397+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f515c014ca0 con 0x7f516c1c4ff0 2024-09-16T14:48:56.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.397+0000 7f516a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:56.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.397+0000 7f516a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:56.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.522+0000 7f8aebcdc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:48:56.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.522+0000 7f8aebcdc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:48:56.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.522+0000 7f8aebcdc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:56.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.522+0000 7f8aebcdc640 1 -- 172.21.15.26:0/1167853469 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8acc005880 con 0x7f8ae4099b60 2024-09-16T14:48:56.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.522+0000 7f8ada7fc640 1 -- 172.21.15.26:0/1167853469 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2178496452) 0x7f8ae0014250 con 0x7f8ae4099b60 2024-09-16T14:48:56.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.522+0000 7f8ada7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:48:56.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.522+0000 7f8ada7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:56.527 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:48:56.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.526+0000 7f8aebcdc640 1 -- 172.21.15.26:0/1167853469 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8ac8036100 legacy=0x7f8ac80385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.526+0000 7f8aebcdc640 10 monclient: shutdown 2024-09-16T14:48:56.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.526+0000 7f8aebcdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:56.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.526+0000 7f8aebcdc640 1 -- 172.21.15.26:0/1167853469 >> v1:172.21.15.26:6790/0 conn(0x7f8ae4099b60 legacy=0x7f8ae4139ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.526+0000 7f8aebcdc640 1 -- 172.21.15.26:0/1167853469 shutdown_connections 2024-09-16T14:48:56.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.526+0000 7f8aebcdc640 1 -- 172.21.15.26:0/1167853469 >> 172.21.15.26:0/1167853469 conn(0x7f8ae40a3d00 msgr2=0x7f8ae409f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:56.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.526+0000 7f8aebcdc640 1 -- 172.21.15.26:0/1167853469 shutdown_connections 2024-09-16T14:48:56.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.526+0000 7f8aebcdc640 1 -- 172.21.15.26:0/1167853469 wait complete. 2024-09-16T14:48:56.546 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:48:56.546 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:48:56.546 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:48:56.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.553+0000 7f5173f20640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T14:48:56.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.553+0000 7f5173f20640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T14:48:56.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.553+0000 7f5173f20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:56.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:56.553+0000 7f5173f20640 1 -- 172.21.15.189:0/2433742701 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush move", "name": "barr", "args": ["root=default"]} v 0) v1 -- 0x7f514c005880 con 0x7f516c1c4ff0 2024-09-16T14:48:56.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 10 monclient: get_monmap_and_config 2024-09-16T14:48:56.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 10 monclient: build_initial_monmap 2024-09-16T14:48:56.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 10 monclient: monmap: 2024-09-16T14:48:56.673 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:56.673 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.673 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:56.673842+0000 2024-09-16T14:48:56.674 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:56.673842+0000 2024-09-16T14:48:56.674 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:56.674 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.674 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:56.674 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:56.674 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:56.674 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 1 Processor -- start 2024-09-16T14:48:56.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 1 -- start start 2024-09-16T14:48:56.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 10 monclient: init 2024-09-16T14:48:56.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:56.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:56.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 10 monclient(hunting): picked mon.noname-c con 0x7f97dc09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:56.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 10 monclient(hunting): picked mon.noname-a con 0x7f97dc132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:56.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 10 monclient(hunting): picked mon.noname-b con 0x7f97dc138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:56.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc09d260 con 0x7f97dc132650 2024-09-16T14:48:56.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc132a80 con 0x7f97dc09cb80 2024-09-16T14:48:56.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc09cf60 con 0x7f97dc138ff0 2024-09-16T14:48:56.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:56.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 10 monclient(hunting): authenticate will time out at 1360.793837s 2024-09-16T14:48:56.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e1f70640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f97dc138ff0 0x7f97dc13b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49144/0 (socket says 172.21.15.26:49144) 2024-09-16T14:48:56.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e0f6e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f97dc132650 0x7f97dc1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53828/0 (socket says 172.21.15.26:53828) 2024-09-16T14:48:56.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e1f70640 1 -- 172.21.15.26:0/2044541514 learned_addr learned my addr 172.21.15.26:0/2044541514 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:56.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 1 -- 172.21.15.26:0/2044541514 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180822478 0 0) 0x7f97dc09d260 con 0x7f97dc132650 2024-09-16T14:48:56.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient(hunting): my global_id is 124536 2024-09-16T14:48:56.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 1 -- 172.21.15.26:0/2044541514 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97c8003530 con 0x7f97dc132650 2024-09-16T14:48:56.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 1 -- 172.21.15.26:0/2044541514 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831514682 0 0) 0x7f97dc09cf60 con 0x7f97dc138ff0 2024-09-16T14:48:56.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient(hunting): my global_id is 154109 2024-09-16T14:48:56.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 1 -- 172.21.15.26:0/2044541514 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97dc09d260 con 0x7f97dc138ff0 2024-09-16T14:48:56.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 1 -- 172.21.15.26:0/2044541514 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3645350557 0 0) 0x7f97c8003530 con 0x7f97dc132650 2024-09-16T14:48:56.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 1 -- 172.21.15.26:0/2044541514 >> v1:172.21.15.26:6790/0 conn(0x7f97dc138ff0 legacy=0x7f97dc13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 1 -- 172.21.15.26:0/2044541514 >> v1:172.21.15.189:6789/0 conn(0x7f97dc09cb80 legacy=0x7f97dc0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:56.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 1 monclient: found mon.noname-a 2024-09-16T14:48:56.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:56.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:56.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 1 -- 172.21.15.26:0/2044541514 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc09f2d0 con 0x7f97dc132650 2024-09-16T14:48:56.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient: _finish_auth 0 2024-09-16T14:48:56.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:56.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:56.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 1 -- 172.21.15.26:0/2044541514 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f97d8001620 con 0x7f97dc132650 2024-09-16T14:48:56.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 5 monclient: authenticate success, global_id 124536 2024-09-16T14:48:56.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:56.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:56.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient: dump: 2024-09-16T14:48:56.685 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.685 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.685 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.685 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.685 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.685 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.685 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.686 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.686 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.686 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:56.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 1 -- 172.21.15.26:0/2044541514 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97d8001850 con 0x7f97dc132650 2024-09-16T14:48:56.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:56.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 1 -- 172.21.15.26:0/2044541514 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f97d8001c20 con 0x7f97dc132650 2024-09-16T14:48:56.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:56.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97cffff640 10 monclient: dump: 2024-09-16T14:48:56.687 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.687 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.687 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.687 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.687 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.687 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.687 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.687 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.687 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.687 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:56.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 4 set_mon_vals no callback set 2024-09-16T14:48:56.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 10 monclient: shutdown 2024-09-16T14:48:56.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:56.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 1 -- 172.21.15.26:0/2044541514 >> v1:172.21.15.26:6789/0 conn(0x7f97dc132650 legacy=0x7f97dc1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 1 -- 172.21.15.26:0/2044541514 shutdown_connections 2024-09-16T14:48:56.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 1 -- 172.21.15.26:0/2044541514 >> 172.21.15.26:0/2044541514 conn(0x7f97dc09fb50 msgr2=0x7f97dc09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:56.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 1 -- 172.21.15.26:0/2044541514 shutdown_connections 2024-09-16T14:48:56.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 1 -- 172.21.15.26:0/2044541514 wait complete. 2024-09-16T14:48:56.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.670+0000 7f97e39d3640 10 monclient: build_initial_monmap 2024-09-16T14:48:56.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 10 monclient: monmap: 2024-09-16T14:48:56.688 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:56.688 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:56.688 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:56.688 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:56.688 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:56.689 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:56.689 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:56.689 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:56.689 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:56.689 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 1 Processor -- start 2024-09-16T14:48:56.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 1 -- start start 2024-09-16T14:48:56.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 10 monclient: init 2024-09-16T14:48:56.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:56.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:56.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 10 monclient(hunting): picked mon.noname-a con 0x7f97dc09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:56.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 10 monclient(hunting): picked mon.noname-c con 0x7f97dc132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:56.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 10 monclient(hunting): picked mon.noname-b con 0x7f97dc138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:56.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc06fdf0 con 0x7f97dc09cb80 2024-09-16T14:48:56.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc070f20 con 0x7f97dc138ff0 2024-09-16T14:48:56.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc138300 con 0x7f97dc132650 2024-09-16T14:48:56.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:56.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 10 monclient(hunting): authenticate will time out at 1360.796866s 2024-09-16T14:48:56.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e176f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f97dc09cb80 0x7f97dc06d840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53838/0 (socket says 172.21.15.26:53838) 2024-09-16T14:48:56.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e176f640 1 -- 172.21.15.26:0/2491671748 learned_addr learned my addr 172.21.15.26:0/2491671748 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:56.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389467730 0 0) 0x7f97dc06fdf0 con 0x7f97dc09cb80 2024-09-16T14:48:56.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): my global_id is 124539 2024-09-16T14:48:56.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97c0003530 con 0x7f97dc09cb80 2024-09-16T14:48:56.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776938901 0 0) 0x7f97dc09f2d0 con 0x7f97dc132650 2024-09-16T14:48:56.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): my global_id is 154112 2024-09-16T14:48:56.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97dc06fdf0 con 0x7f97dc132650 2024-09-16T14:48:56.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1747924061 0 0) 0x7f97c0003530 con 0x7f97dc09cb80 2024-09-16T14:48:56.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97dc09f2d0 con 0x7f97dc09cb80 2024-09-16T14:48:56.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f97d0001910 con 0x7f97dc09cb80 2024-09-16T14:48:56.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:56.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): dump: 2024-09-16T14:48:56.692 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.692 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.692 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.692 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.692 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.693 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.693 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.693 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.693 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.693 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:56.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4214897128 0 0) 0x7f97dc06fdf0 con 0x7f97dc132650 2024-09-16T14:48:56.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97c0003530 con 0x7f97dc132650 2024-09-16T14:48:56.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f97d80019c0 con 0x7f97dc132650 2024-09-16T14:48:56.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:56.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): dump: 2024-09-16T14:48:56.693 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.694 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.694 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.694 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.694 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.694 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.694 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.694 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.694 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.694 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146815658 0 0) 0x7f97dc070f20 con 0x7f97dc138ff0 2024-09-16T14:48:56.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): my global_id is 144334 2024-09-16T14:48:56.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97dc06fdf0 con 0x7f97dc138ff0 2024-09-16T14:48:56.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1752234299 0 0) 0x7f97dc09f2d0 con 0x7f97dc09cb80 2024-09-16T14:48:56.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 >> v1:172.21.15.26:6790/0 conn(0x7f97dc132650 legacy=0x7f97dc06df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 >> v1:172.21.15.189:6789/0 conn(0x7f97dc138ff0 legacy=0x7f97dc06e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient: _finish_hunting 0 2024-09-16T14:48:56.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 monclient: found mon.a 2024-09-16T14:48:56.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:56.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:56.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc09a240 con 0x7f97dc09cb80 2024-09-16T14:48:56.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient: _finish_auth 0 2024-09-16T14:48:56.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:56.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:56.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 5 monclient: authenticate success, global_id 124539 2024-09-16T14:48:56.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 10 monclient: _renew_subs 2024-09-16T14:48:56.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:56.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 1 -- 172.21.15.26:0/2491671748 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97dc133660 con 0x7f97dc09cb80 2024-09-16T14:48:56.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 10 monclient: _renew_subs 2024-09-16T14:48:56.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:56.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 1 -- 172.21.15.26:0/2491671748 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97dc070240 con 0x7f97dc09cb80 2024-09-16T14:48:56.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97d0001c60 con 0x7f97dc09cb80 2024-09-16T14:48:56.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:56.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f97d0003950 con 0x7f97dc09cb80 2024-09-16T14:48:56.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:56.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cf7fe640 4 set_mon_vals no callback set 2024-09-16T14:48:56.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient: dump: 2024-09-16T14:48:56.697 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.697 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.697 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.697 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.697 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.697 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.697 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.697 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.697 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.697 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:56.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e39d3640 1 -- 172.21.15.26:0/2491671748 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f97bc006590 con 0x7f97bc002830 2024-09-16T14:48:56.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f97d0003b30 con 0x7f97dc09cb80 2024-09-16T14:48:56.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(175..175 src has 73..175) v4 ==== 6933+0+0 (unknown 3869440614 0 0) 0x7f97d0043e30 con 0x7f97dc09cb80 2024-09-16T14:48:56.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870430505 0 0) 0x7f97bc006590 con 0x7f97bc002830 2024-09-16T14:48:56.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient(hunting): my global_id is 124542 2024-09-16T14:48:56.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97dc09f2d0 con 0x7f97bc002830 2024-09-16T14:48:56.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97e1f70640 10 monclient: get_auth_request con 0x7f97c00363f0 auth_method 0 2024-09-16T14:48:56.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3657191076 0 0) 0x7f97dc09f2d0 con 0x7f97bc002830 2024-09-16T14:48:56.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97bc006590 con 0x7f97bc002830 2024-09-16T14:48:56.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f97d80011f0 con 0x7f97bc002830 2024-09-16T14:48:56.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:56.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient: dump: 2024-09-16T14:48:56.699 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.699 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.699 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.699 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.699 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.699 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.699 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.699 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.699 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.699 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2951940973 0 0) 0x7f97bc006590 con 0x7f97bc002830 2024-09-16T14:48:56.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97bc00a0e0 con 0x7f97bc002830 2024-09-16T14:48:56.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f97d8006b50 con 0x7f97bc002830 2024-09-16T14:48:56.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:56.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:56.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 >> v1:172.21.15.26:6789/0 conn(0x7f97bc002830 legacy=0x7f97bc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.674+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 >> v1:172.21.15.26:6789/0 conn(0x7f97bc002830 legacy=0x7f97bc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:56.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97e39d3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:48:56.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97e39d3640 1 -- 172.21.15.26:0/2491671748 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f97bc009da0 con 0x7f97bc0090c0 2024-09-16T14:48:56.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553598864 0 0) 0x7f97bc009da0 con 0x7f97bc0090c0 2024-09-16T14:48:56.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 10 monclient(hunting): my global_id is 124545 2024-09-16T14:48:56.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97bc006590 con 0x7f97bc0090c0 2024-09-16T14:48:56.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 421496200 0 0) 0x7f97bc006590 con 0x7f97bc0090c0 2024-09-16T14:48:56.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97bc009da0 con 0x7f97bc0090c0 2024-09-16T14:48:56.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f97d8005040 con 0x7f97bc0090c0 2024-09-16T14:48:56.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:56.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 10 monclient: dump: 2024-09-16T14:48:56.702 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.702 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.702 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.702 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.703 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.703 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.703 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.703 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.703 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.703 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2446316999 0 0) 0x7f97bc009da0 con 0x7f97bc0090c0 2024-09-16T14:48:56.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f97bc00c290 con 0x7f97bc0090c0 2024-09-16T14:48:56.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 2500972490) 0x7f97d80063e0 con 0x7f97bc0090c0 2024-09-16T14:48:56.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:48:56.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:56.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 >> v1:172.21.15.26:6789/0 conn(0x7f97bc0090c0 legacy=0x7f97bc009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97cdffb640 1 -- 172.21.15.26:0/2491671748 >> v1:172.21.15.26:6789/0 conn(0x7f97bc0090c0 legacy=0x7f97bc009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:56.704 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:48:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:48:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 60, 2024-09-16T14:48:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:48:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:48:56.704 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:48:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:48:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:48:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:48:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:48:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:56.705 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:48:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:48:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:56.706 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:48:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:48:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:48:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:48:56.707 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:48:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:48:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:48:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:48:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:48:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:48:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:48:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:48:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:48:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:48:56.708 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:48:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.709 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:48:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:48:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.710 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:48:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:48:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:56.711 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:48:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:48:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:48:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:56.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:48:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:56.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:48:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:48:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:48:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:48:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:56.714 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:48:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:48:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:48:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.715 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:48:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.716 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:48:56.716 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:48:56.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97e39d3640 1 -- 172.21.15.26:0/2491671748 >> v1:172.21.15.189:6812/3196161582 conn(0x7f97c00363f0 legacy=0x7f97c00388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97e39d3640 10 monclient: shutdown 2024-09-16T14:48:56.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97e39d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:56.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97e39d3640 1 -- 172.21.15.26:0/2491671748 >> v1:172.21.15.26:6789/0 conn(0x7f97dc09cb80 legacy=0x7f97dc06d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97e1f70640 1 -- 172.21.15.26:0/2491671748 reap_dead start 2024-09-16T14:48:56.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97e1f70640 1 -- 172.21.15.26:0/2491671748 reap_dead start 2024-09-16T14:48:56.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97e39d3640 1 -- 172.21.15.26:0/2491671748 shutdown_connections 2024-09-16T14:48:56.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97e39d3640 1 -- 172.21.15.26:0/2491671748 >> 172.21.15.26:0/2491671748 conn(0x7f97dc09fb50 msgr2=0x7f97dc13b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:56.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97e39d3640 1 -- 172.21.15.26:0/2491671748 shutdown_connections 2024-09-16T14:48:56.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.682+0000 7f97e39d3640 1 -- 172.21.15.26:0/2491671748 wait complete. 2024-09-16T14:48:56.718 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:48:56.718 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:48:56.719 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:48:56.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 10 monclient: get_monmap_and_config 2024-09-16T14:48:56.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 10 monclient: build_initial_monmap 2024-09-16T14:48:56.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 10 monclient: monmap: 2024-09-16T14:48:56.837 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:56.838 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.838 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:56.838112+0000 2024-09-16T14:48:56.838 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:56.838112+0000 2024-09-16T14:48:56.838 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:56.838 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.838 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:56.838 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:56.838 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:56.838 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 1 Processor -- start 2024-09-16T14:48:56.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 1 -- start start 2024-09-16T14:48:56.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 10 monclient: init 2024-09-16T14:48:56.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:56.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:48:56.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 10 monclient(hunting): picked mon.noname-a con 0x7f8b100a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:56.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 10 monclient(hunting): picked mon.noname-c con 0x7f8b100a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:56.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 10 monclient(hunting): picked mon.noname-b con 0x7f8b100a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:56.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b100a3250 con 0x7f8b100a1d10 2024-09-16T14:48:56.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1009f600 con 0x7f8b100a0000 2024-09-16T14:48:56.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b100a3830 con 0x7f8b100a2800 2024-09-16T14:48:56.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:56.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 10 monclient(hunting): authenticate will time out at 1360.958154s 2024-09-16T14:48:56.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0effd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8b100a1d10 0x7f8b100a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53864/0 (socket says 172.21.15.26:53864) 2024-09-16T14:48:56.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0effd640 1 -- 172.21.15.26:0/3449823065 learned_addr learned my addr 172.21.15.26:0/3449823065 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:56.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 1 -- 172.21.15.26:0/3449823065 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383454618 0 0) 0x7f8b100a3250 con 0x7f8b100a1d10 2024-09-16T14:48:56.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient(hunting): my global_id is 124548 2024-09-16T14:48:56.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 1 -- 172.21.15.26:0/3449823065 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af8003530 con 0x7f8b100a1d10 2024-09-16T14:48:56.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 1 -- 172.21.15.26:0/3449823065 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012680052 0 0) 0x7f8b100a3830 con 0x7f8b100a2800 2024-09-16T14:48:56.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient(hunting): my global_id is 154115 2024-09-16T14:48:56.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 1 -- 172.21.15.26:0/3449823065 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b100a3250 con 0x7f8b100a2800 2024-09-16T14:48:56.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 1 -- 172.21.15.26:0/3449823065 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524375018 0 0) 0x7f8b1009f600 con 0x7f8b100a0000 2024-09-16T14:48:56.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient(hunting): my global_id is 144337 2024-09-16T14:48:56.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 1 -- 172.21.15.26:0/3449823065 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b100a3830 con 0x7f8b100a0000 2024-09-16T14:48:56.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 1 -- 172.21.15.26:0/3449823065 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3887100545 0 0) 0x7f8af8003530 con 0x7f8b100a1d10 2024-09-16T14:48:56.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 1 -- 172.21.15.26:0/3449823065 >> v1:172.21.15.26:6790/0 conn(0x7f8b100a2800 legacy=0x7f8b1013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 1 -- 172.21.15.26:0/3449823065 >> v1:172.21.15.189:6789/0 conn(0x7f8b100a0000 legacy=0x7f8b10139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:48:56.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 1 monclient: found mon.noname-a 2024-09-16T14:48:56.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:56.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:56.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 1 -- 172.21.15.26:0/3449823065 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1009ed60 con 0x7f8b100a1d10 2024-09-16T14:48:56.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient: _finish_auth 0 2024-09-16T14:48:56.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:56.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:56.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 5 monclient: authenticate success, global_id 124548 2024-09-16T14:48:56.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 1 -- 172.21.15.26:0/3449823065 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8b00001620 con 0x7f8b100a1d10 2024-09-16T14:48:56.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:56.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient: dump: 2024-09-16T14:48:56.849 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.849 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.849 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.849 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.849 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.849 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.850 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.850 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.850 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.850 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:56.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 1 -- 172.21.15.26:0/3449823065 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b00001900 con 0x7f8b100a1d10 2024-09-16T14:48:56.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:56.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 1 -- 172.21.15.26:0/3449823065 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8b00001cd0 con 0x7f8b100a1d10 2024-09-16T14:48:56.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:56.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 4 set_mon_vals no callback set 2024-09-16T14:48:56.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:56.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b0dffb640 10 monclient: dump: 2024-09-16T14:48:56.850 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.850 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.851 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.851 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.851 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.851 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.851 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.851 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.851 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.851 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 10 monclient: shutdown 2024-09-16T14:48:56.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:56.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 1 -- 172.21.15.26:0/3449823065 >> v1:172.21.15.26:6789/0 conn(0x7f8b100a1d10 legacy=0x7f8b100a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 1 -- 172.21.15.26:0/3449823065 shutdown_connections 2024-09-16T14:48:56.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 1 -- 172.21.15.26:0/3449823065 >> 172.21.15.26:0/3449823065 conn(0x7f8b100a7f30 msgr2=0x7f8b100a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:56.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 1 -- 172.21.15.26:0/3449823065 shutdown_connections 2024-09-16T14:48:56.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.834+0000 7f8b15935640 1 -- 172.21.15.26:0/3449823065 wait complete. 2024-09-16T14:48:56.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 10 monclient: build_initial_monmap 2024-09-16T14:48:56.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 10 monclient: monmap: 2024-09-16T14:48:56.852 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:56.852 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:56.852 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:56.852 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:56.852 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:56.852 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:56.852 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:56.852 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:56.852 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:56.853 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 1 Processor -- start 2024-09-16T14:48:56.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 1 -- start start 2024-09-16T14:48:56.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 10 monclient: init 2024-09-16T14:48:56.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:56.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:56.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 10 monclient(hunting): picked mon.noname-a con 0x7f8b100a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:56.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 10 monclient(hunting): picked mon.noname-c con 0x7f8b100a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:56.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 10 monclient(hunting): picked mon.noname-b con 0x7f8b100a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:56.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b101336b0 con 0x7f8b100a0000 2024-09-16T14:48:56.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b100a6970 con 0x7f8b100a2800 2024-09-16T14:48:56.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b100a6440 con 0x7f8b100a1d10 2024-09-16T14:48:56.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:56.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 10 monclient(hunting): authenticate will time out at 1360.961289s 2024-09-16T14:48:56.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b0effd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8b100a0000 0x7f8b10134370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53866/0 (socket says 172.21.15.26:53866) 2024-09-16T14:48:56.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b0effd640 1 -- 172.21.15.26:0/2124138002 learned_addr learned my addr 172.21.15.26:0/2124138002 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:56.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900540867 0 0) 0x7f8b101336b0 con 0x7f8b100a0000 2024-09-16T14:48:56.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): my global_id is 124551 2024-09-16T14:48:56.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8afc003560 con 0x7f8b100a0000 2024-09-16T14:48:56.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595517778 0 0) 0x7f8b100a6440 con 0x7f8b100a1d10 2024-09-16T14:48:56.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): my global_id is 154118 2024-09-16T14:48:56.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b101336b0 con 0x7f8b100a1d10 2024-09-16T14:48:56.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234088818 0 0) 0x7f8b100a6970 con 0x7f8b100a2800 2024-09-16T14:48:56.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): my global_id is 144340 2024-09-16T14:48:56.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b100a6440 con 0x7f8b100a2800 2024-09-16T14:48:56.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1234481371 0 0) 0x7f8afc003560 con 0x7f8b100a0000 2024-09-16T14:48:56.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b100a6970 con 0x7f8b100a0000 2024-09-16T14:48:56.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8b00001cd0 con 0x7f8b100a0000 2024-09-16T14:48:56.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:56.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): dump: 2024-09-16T14:48:56.856 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.856 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.856 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.856 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.856 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.856 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.856 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.856 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.856 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.856 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:56.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 685617405 0 0) 0x7f8b101336b0 con 0x7f8b100a1d10 2024-09-16T14:48:56.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8afc003560 con 0x7f8b100a1d10 2024-09-16T14:48:56.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8b04001820 con 0x7f8b100a1d10 2024-09-16T14:48:56.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:56.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): dump: 2024-09-16T14:48:56.857 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.857 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.857 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.857 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.857 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.857 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.857 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.857 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.858 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.858 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2232153689 0 0) 0x7f8b100a6970 con 0x7f8b100a0000 2024-09-16T14:48:56.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 >> v1:172.21.15.26:6790/0 conn(0x7f8b100a1d10 legacy=0x7f8b101cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 >> v1:172.21.15.189:6789/0 conn(0x7f8b100a2800 legacy=0x7f8b101d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:56.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 monclient: found mon.a 2024-09-16T14:48:56.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:56.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:56.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1009e500 con 0x7f8b100a0000 2024-09-16T14:48:56.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient: _finish_auth 0 2024-09-16T14:48:56.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:56.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:56.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 5 monclient: authenticate success, global_id 124551 2024-09-16T14:48:56.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 10 monclient: _renew_subs 2024-09-16T14:48:56.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:56.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 1 -- 172.21.15.26:0/2124138002 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b10133f50 con 0x7f8b100a0000 2024-09-16T14:48:56.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 10 monclient: _renew_subs 2024-09-16T14:48:56.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:56.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 1 -- 172.21.15.26:0/2124138002 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b10134a80 con 0x7f8b100a0000 2024-09-16T14:48:56.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b00004c00 con 0x7f8b100a0000 2024-09-16T14:48:56.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:56.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8b00004f10 con 0x7f8b100a0000 2024-09-16T14:48:56.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:56.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient: dump: 2024-09-16T14:48:56.860 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.860 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.860 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.860 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.860 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.860 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.860 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.860 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.860 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.860 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b0d7fa640 4 set_mon_vals no callback set 2024-09-16T14:48:56.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:56.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b15935640 1 -- 172.21.15.26:0/2124138002 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8af0006590 con 0x7f8af0002830 2024-09-16T14:48:56.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8b00005070 con 0x7f8b100a0000 2024-09-16T14:48:56.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(175..175 src has 73..175) v4 ==== 6933+0+0 (unknown 3869440614 0 0) 0x7f8b00043f80 con 0x7f8b100a0000 2024-09-16T14:48:56.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008896832 0 0) 0x7f8af0006590 con 0x7f8af0002830 2024-09-16T14:48:56.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 10 monclient(hunting): my global_id is 154121 2024-09-16T14:48:56.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b100a6970 con 0x7f8af0002830 2024-09-16T14:48:56.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8b0f7fe640 10 monclient: get_auth_request con 0x7f8afc036370 auth_method 0 2024-09-16T14:48:56.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1556461004 0 0) 0x7f8b100a6970 con 0x7f8af0002830 2024-09-16T14:48:56.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.838+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8af0006590 con 0x7f8af0002830 2024-09-16T14:48:56.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.842+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8b04003620 con 0x7f8af0002830 2024-09-16T14:48:56.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.842+0000 7f8af7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.842+0000 7f8af7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:56.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.842+0000 7f8af7fff640 10 monclient: dump: 2024-09-16T14:48:56.862 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.862 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.862 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.862 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.862 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.862 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.862 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.862 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.862 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.862 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.842+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1681705855 0 0) 0x7f8af0006590 con 0x7f8af0002830 2024-09-16T14:48:56.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.842+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8af000a0e0 con 0x7f8af0002830 2024-09-16T14:48:56.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.842+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8b04004650 con 0x7f8af0002830 2024-09-16T14:48:56.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.842+0000 7f8af7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:56.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.842+0000 7f8af7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:56.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.842+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 >> v1:172.21.15.26:6790/0 conn(0x7f8af0002830 legacy=0x7f8af0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.842+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 >> v1:172.21.15.26:6790/0 conn(0x7f8af0002830 legacy=0x7f8af0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:56.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8b15935640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:48:56.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8b15935640 1 -- 172.21.15.26:0/2124138002 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8af0009da0 con 0x7f8af00090c0 2024-09-16T14:48:56.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745660226 0 0) 0x7f8af0009da0 con 0x7f8af00090c0 2024-09-16T14:48:56.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:56.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:56.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:56.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 10 monclient(hunting): my global_id is 154124 2024-09-16T14:48:56.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af0006590 con 0x7f8af00090c0 2024-09-16T14:48:56.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 933388383 0 0) 0x7f8af0006590 con 0x7f8af00090c0 2024-09-16T14:48:56.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8af0009da0 con 0x7f8af00090c0 2024-09-16T14:48:56.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8b04003db0 con 0x7f8af00090c0 2024-09-16T14:48:56.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:56.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:56.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 10 monclient: dump: 2024-09-16T14:48:56.865 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:56.865 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:56.865 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.866 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:56.866 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:56.866 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:56.866 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:56.866 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:56.866 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:56.866 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:56.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1948679569 0 0) 0x7f8af0009da0 con 0x7f8af00090c0 2024-09-16T14:48:56.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8af000c290 con 0x7f8af00090c0 2024-09-16T14:48:56.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3881 (unknown 0 0 3951256240) 0x7f8b04004150 con 0x7f8af00090c0 2024-09-16T14:48:56.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:48:56.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:56.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 >> v1:172.21.15.26:6790/0 conn(0x7f8af00090c0 legacy=0x7f8af0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.846+0000 7f8af7fff640 1 -- 172.21.15.26:0/2124138002 >> v1:172.21.15.26:6790/0 conn(0x7f8af00090c0 legacy=0x7f8af0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:56.867 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:48:56.867 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:56.867 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:56.867 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 60, 2024-09-16T14:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:56.868 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:56.869 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:48:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:48:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:48:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:48:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:48:56.878 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:48:56.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.850+0000 7f8b15935640 1 -- 172.21.15.26:0/2124138002 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8afc036370 legacy=0x7f8afc038830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.850+0000 7f8b15935640 10 monclient: shutdown 2024-09-16T14:48:56.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.850+0000 7f8b15935640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:56.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.850+0000 7f8b15935640 1 -- 172.21.15.26:0/2124138002 >> v1:172.21.15.26:6789/0 conn(0x7f8b100a0000 legacy=0x7f8b10134370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:56.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.850+0000 7f8b0f7fe640 1 -- 172.21.15.26:0/2124138002 reap_dead start 2024-09-16T14:48:56.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.850+0000 7f8b0f7fe640 1 -- 172.21.15.26:0/2124138002 reap_dead start 2024-09-16T14:48:56.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.850+0000 7f8b15935640 1 -- 172.21.15.26:0/2124138002 shutdown_connections 2024-09-16T14:48:56.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.850+0000 7f8b15935640 1 -- 172.21.15.26:0/2124138002 >> 172.21.15.26:0/2124138002 conn(0x7f8b100a7f30 msgr2=0x7f8b1006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:56.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.850+0000 7f8b15935640 1 -- 172.21.15.26:0/2124138002 shutdown_connections 2024-09-16T14:48:56.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.850+0000 7f8b15935640 1 -- 172.21.15.26:0/2124138002 wait complete. 2024-09-16T14:48:56.880 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:48:56.880 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:48:56.881 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:48:57.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 10 monclient: get_monmap_and_config 2024-09-16T14:48:57.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 10 monclient: build_initial_monmap 2024-09-16T14:48:57.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 10 monclient: monmap: 2024-09-16T14:48:57.002 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:57.002 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.002 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:57.002727+0000 2024-09-16T14:48:57.002 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:57.002727+0000 2024-09-16T14:48:57.002 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:57.002 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:57.003 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:57.003 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:57.003 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:57.003 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:57.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 1 Processor -- start 2024-09-16T14:48:57.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 1 -- start start 2024-09-16T14:48:57.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 10 monclient: init 2024-09-16T14:48:57.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:57.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:57.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 10 monclient(hunting): picked mon.noname-b con 0x7f7bb00a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:57.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 10 monclient(hunting): picked mon.noname-a con 0x7f7bb00a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:57.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 10 monclient(hunting): picked mon.noname-c con 0x7f7bb00a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:57.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb00a3250 con 0x7f7bb00a0000 2024-09-16T14:48:57.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb009f600 con 0x7f7bb00a2800 2024-09-16T14:48:57.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb00a3830 con 0x7f7bb00a1d10 2024-09-16T14:48:57.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:57.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 10 monclient(hunting): authenticate will time out at 1361.122846s 2024-09-16T14:48:57.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7baeffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7bb00a1d10 0x7f7bb00a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47814/0 (socket says 172.21.15.26:47814) 2024-09-16T14:48:57.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bae7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7bb00a0000 0x7f7bb0139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33978/0 (socket says 172.21.15.26:33978) 2024-09-16T14:48:57.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7baeffd640 1 -- 172.21.15.26:0/3362686103 learned_addr learned my addr 172.21.15.26:0/3362686103 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:57.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 1 -- 172.21.15.26:0/3362686103 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220126358 0 0) 0x7f7bb00a3830 con 0x7f7bb00a1d10 2024-09-16T14:48:57.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:57.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:57.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:57.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient(hunting): my global_id is 154127 2024-09-16T14:48:57.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 1 -- 172.21.15.26:0/3362686103 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b98003530 con 0x7f7bb00a1d10 2024-09-16T14:48:57.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 1 -- 172.21.15.26:0/3362686103 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235558941 0 0) 0x7f7bb00a3250 con 0x7f7bb00a0000 2024-09-16T14:48:57.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:57.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:57.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:57.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient(hunting): my global_id is 124554 2024-09-16T14:48:57.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 1 -- 172.21.15.26:0/3362686103 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb00a3830 con 0x7f7bb00a0000 2024-09-16T14:48:57.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 1 -- 172.21.15.26:0/3362686103 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 733759236 0 0) 0x7f7b98003530 con 0x7f7bb00a1d10 2024-09-16T14:48:57.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 1 -- 172.21.15.26:0/3362686103 >> v1:172.21.15.189:6789/0 conn(0x7f7bb00a2800 legacy=0x7f7bb013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 1 -- 172.21.15.26:0/3362686103 >> v1:172.21.15.26:6789/0 conn(0x7f7bb00a0000 legacy=0x7f7bb0139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient: _finish_hunting 0 2024-09-16T14:48:57.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 1 monclient: found mon.noname-b 2024-09-16T14:48:57.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:57.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:48:57.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 1 -- 172.21.15.26:0/3362686103 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb009ed60 con 0x7f7bb00a1d10 2024-09-16T14:48:57.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient: _finish_auth 0 2024-09-16T14:48:57.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:57.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:57.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 1 -- 172.21.15.26:0/3362686103 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7ba0001620 con 0x7f7bb00a1d10 2024-09-16T14:48:57.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 5 monclient: authenticate success, global_id 154127 2024-09-16T14:48:57.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:57.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:57.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:57.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient: dump: 2024-09-16T14:48:57.016 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:57.016 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.016 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.016 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.016 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:57.016 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:57.016 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:57.016 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:57.016 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:57.017 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:57.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:57.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 1 -- 172.21.15.26:0/3362686103 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ba0001840 con 0x7f7bb00a1d10 2024-09-16T14:48:57.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:57.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 1 -- 172.21.15.26:0/3362686103 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7ba0001bd0 con 0x7f7bb00a1d10 2024-09-16T14:48:57.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:57.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7bb5908640 4 set_mon_vals no callback set 2024-09-16T14:48:57.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:57.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:57.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:56.998+0000 7f7badffb640 10 monclient: dump: 2024-09-16T14:48:57.017 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:57.017 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.017 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.018 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.018 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:57.018 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:57.018 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:57.018 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:57.018 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:57.018 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:57.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 10 monclient: shutdown 2024-09-16T14:48:57.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:57.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 1 -- 172.21.15.26:0/3362686103 >> v1:172.21.15.26:6790/0 conn(0x7f7bb00a1d10 legacy=0x7f7bb00a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 1 -- 172.21.15.26:0/3362686103 shutdown_connections 2024-09-16T14:48:57.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 1 -- 172.21.15.26:0/3362686103 >> 172.21.15.26:0/3362686103 conn(0x7f7bb00a7f30 msgr2=0x7f7bb00a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:57.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 1 -- 172.21.15.26:0/3362686103 shutdown_connections 2024-09-16T14:48:57.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 1 -- 172.21.15.26:0/3362686103 wait complete. 2024-09-16T14:48:57.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 10 monclient: build_initial_monmap 2024-09-16T14:48:57.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 10 monclient: monmap: 2024-09-16T14:48:57.019 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:57.019 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:57.019 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:57.019 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:57.019 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:57.019 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:57.019 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:57.019 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:57.019 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:57.019 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:57.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 1 Processor -- start 2024-09-16T14:48:57.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 1 -- start start 2024-09-16T14:48:57.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 10 monclient: init 2024-09-16T14:48:57.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:57.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:57.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 10 monclient(hunting): picked mon.noname-a con 0x7f7bb00a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:57.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 10 monclient(hunting): picked mon.noname-c con 0x7f7bb00a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:57.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 10 monclient(hunting): picked mon.noname-b con 0x7f7bb00a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:57.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb013ad90 con 0x7f7bb00a0000 2024-09-16T14:48:57.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb013ce10 con 0x7f7bb00a2800 2024-09-16T14:48:57.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb0139f50 con 0x7f7bb00a1d10 2024-09-16T14:48:57.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:57.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 10 monclient(hunting): authenticate will time out at 1361.125880s 2024-09-16T14:48:57.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7baeffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7bb00a0000 0x7f7bb0139130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33980/0 (socket says 172.21.15.26:33980) 2024-09-16T14:48:57.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7baeffd640 1 -- 172.21.15.26:0/2739446967 learned_addr learned my addr 172.21.15.26:0/2739446967 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:57.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088843912 0 0) 0x7f7bb013ad90 con 0x7f7bb00a0000 2024-09-16T14:48:57.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:57.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:57.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:57.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): my global_id is 124557 2024-09-16T14:48:57.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b9c003530 con 0x7f7bb00a0000 2024-09-16T14:48:57.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832342493 0 0) 0x7f7bb0139f50 con 0x7f7bb00a1d10 2024-09-16T14:48:57.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:57.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:57.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:57.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): my global_id is 154130 2024-09-16T14:48:57.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb013ad90 con 0x7f7bb00a1d10 2024-09-16T14:48:57.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3274328225 0 0) 0x7f7bb013ad90 con 0x7f7bb00a1d10 2024-09-16T14:48:57.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bb0139f50 con 0x7f7bb00a1d10 2024-09-16T14:48:57.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7ba40018e0 con 0x7f7bb00a1d10 2024-09-16T14:48:57.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:57.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:57.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): dump: 2024-09-16T14:48:57.023 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:57.023 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.023 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.023 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.023 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:57.023 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:57.023 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:57.023 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:57.024 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:57.024 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:57.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:57.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 643752293 0 0) 0x7f7b9c003530 con 0x7f7bb00a0000 2024-09-16T14:48:57.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bb013ad90 con 0x7f7bb00a0000 2024-09-16T14:48:57.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7ba0001520 con 0x7f7bb00a0000 2024-09-16T14:48:57.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:57.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:57.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): dump: 2024-09-16T14:48:57.024 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:57.024 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.024 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.024 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.025 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:57.025 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:57.025 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:57.025 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:57.025 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:57.025 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:57.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998937528 0 0) 0x7f7bb013ce10 con 0x7f7bb00a2800 2024-09-16T14:48:57.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:57.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:57.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:57.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient(hunting): my global_id is 144346 2024-09-16T14:48:57.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b9c003530 con 0x7f7bb00a2800 2024-09-16T14:48:57.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 717923436 0 0) 0x7f7bb0139f50 con 0x7f7bb00a1d10 2024-09-16T14:48:57.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 >> v1:172.21.15.189:6789/0 conn(0x7f7bb00a2800 legacy=0x7f7bb01d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 >> v1:172.21.15.26:6789/0 conn(0x7f7bb00a0000 legacy=0x7f7bb0139130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:57.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 monclient: found mon.c 2024-09-16T14:48:57.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:57.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:57.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb013b960 con 0x7f7bb00a1d10 2024-09-16T14:48:57.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient: _finish_auth 0 2024-09-16T14:48:57.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:57.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:57.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 5 monclient: authenticate success, global_id 154130 2024-09-16T14:48:57.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 10 monclient: _renew_subs 2024-09-16T14:48:57.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:57.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 1 -- 172.21.15.26:0/2739446967 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb013d150 con 0x7f7bb00a1d10 2024-09-16T14:48:57.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 10 monclient: _renew_subs 2024-09-16T14:48:57.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:57.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 1 -- 172.21.15.26:0/2739446967 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb013dd40 con 0x7f7bb00a1d10 2024-09-16T14:48:57.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ba40047f0 con 0x7f7bb00a1d10 2024-09-16T14:48:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7ba4004b80 con 0x7f7bb00a1d10 2024-09-16T14:48:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bad7fa640 4 set_mon_vals no callback set 2024-09-16T14:48:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 10 monclient: dump: 2024-09-16T14:48:57.028 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:57.029 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.029 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.029 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.029 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:57.029 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:57.029 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:57.030 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:57.030 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:57.030 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:57.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:57.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7bb5908640 1 -- 172.21.15.26:0/2739446967 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7b90006590 con 0x7f7b90002830 2024-09-16T14:48:57.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.002+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7ba400f5f0 con 0x7f7bb00a1d10 2024-09-16T14:48:57.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(175..175 src has 73..175) v4 ==== 6933+0+0 (unknown 3869440614 0 0) 0x7f7ba4043c90 con 0x7f7bb00a1d10 2024-09-16T14:48:57.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7baf7fe640 10 monclient: get_auth_request con 0x7f7b9c036370 auth_method 0 2024-09-16T14:48:57.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688992859 0 0) 0x7f7b90006590 con 0x7f7b90002830 2024-09-16T14:48:57.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:57.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:57.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:57.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 10 monclient(hunting): my global_id is 144349 2024-09-16T14:48:57.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb0139f50 con 0x7f7b90002830 2024-09-16T14:48:57.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1658036007 0 0) 0x7f7bb0139f50 con 0x7f7b90002830 2024-09-16T14:48:57.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b90006590 con 0x7f7b90002830 2024-09-16T14:48:57.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7ba0004fb0 con 0x7f7b90002830 2024-09-16T14:48:57.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:57.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:57.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 10 monclient: dump: 2024-09-16T14:48:57.033 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:57.033 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.034 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.034 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.034 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:57.034 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:57.034 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:57.034 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:57.035 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:57.035 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:57.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2438961518 0 0) 0x7f7b90006590 con 0x7f7b90002830 2024-09-16T14:48:57.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7b9000a0e0 con 0x7f7b90002830 2024-09-16T14:48:57.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7ba00072c0 con 0x7f7b90002830 2024-09-16T14:48:57.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:57.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:57.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 >> v1:172.21.15.189:6789/0 conn(0x7f7b90002830 legacy=0x7f7b90004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.006+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 >> v1:172.21.15.189:6789/0 conn(0x7f7b90002830 legacy=0x7f7b90004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:57.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7bb5908640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:48:57.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7bb5908640 1 -- 172.21.15.26:0/2739446967 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7b90009da0 con 0x7f7b900090c0 2024-09-16T14:48:57.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730755113 0 0) 0x7f7b90009da0 con 0x7f7b900090c0 2024-09-16T14:48:57.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:57.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:57.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:57.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 10 monclient(hunting): my global_id is 144352 2024-09-16T14:48:57.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b90006590 con 0x7f7b900090c0 2024-09-16T14:48:57.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2700371040 0 0) 0x7f7b90006590 con 0x7f7b900090c0 2024-09-16T14:48:57.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b90009da0 con 0x7f7b900090c0 2024-09-16T14:48:57.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7ba00057b0 con 0x7f7b900090c0 2024-09-16T14:48:57.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:57.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:57.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 10 monclient: dump: 2024-09-16T14:48:57.040 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:57.040 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.040 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.041 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.041 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:57.041 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:57.041 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:57.041 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:57.041 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:57.041 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:57.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1035181162 0 0) 0x7f7b90009da0 con 0x7f7b900090c0 2024-09-16T14:48:57.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7b9000c290 con 0x7f7b900090c0 2024-09-16T14:48:57.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 3669987787) 0x7f7ba0007430 con 0x7f7b900090c0 2024-09-16T14:48:57.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:48:57.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:57.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 >> v1:172.21.15.189:6789/0 conn(0x7f7b900090c0 legacy=0x7f7b90009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7b97fff640 1 -- 172.21.15.26:0/2739446967 >> v1:172.21.15.189:6789/0 conn(0x7f7b900090c0 legacy=0x7f7b90009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:57.043 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:48:57.043 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:57.043 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:57.043 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:48:57.043 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 60, 2024-09-16T14:48:57.044 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:48:57.044 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:48:57.044 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:48:57.044 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:48:57.044 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:57.044 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:48:57.044 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:57.044 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:48:57.044 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:48:57.045 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:57.045 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:57.045 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:57.045 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:57.045 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:57.045 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:57.045 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:57.045 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:57.045 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:57.045 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:57.045 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:57.045 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:48:57.045 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:48:57.046 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:57.046 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:57.046 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:57.046 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:57.046 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:57.046 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:57.046 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:57.046 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:57.046 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:57.046 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:57.046 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:57.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:57.046 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:48:57.047 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:48:57.047 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:48:57.047 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:48:57.047 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:48:57.047 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:48:57.047 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:57.047 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:57.047 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:48:57.047 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:48:57.047 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:48:57.047 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:48:57.047 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:48:57.047 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:48:57.047 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:48:57.048 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:57.048 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:48:57.048 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:57.048 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:57.048 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:57.048 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:57.048 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:57.048 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:57.048 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:57.048 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:57.048 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:57.048 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:57.048 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:57.049 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:48:57.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:57.049 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:48:57.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:57.049 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:57.049 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:57.049 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:57.049 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:57.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:57.049 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:57.049 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:48:57.049 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:57.049 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:57.049 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:57.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:57.050 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:48:57.050 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:48:57.050 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:57.050 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:57.050 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:57.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:57.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:57.050 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:57.050 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:57.050 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:57.050 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:57.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:57.051 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:57.051 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:48:57.051 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:57.051 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:57.051 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:57.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:57.051 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:48:57.051 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:48:57.051 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:57.051 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:57.051 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:57.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:57.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:57.051 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:57.052 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:57.052 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:57.052 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:57.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:57.052 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:57.052 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:48:57.052 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:57.052 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:57.052 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:57.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:57.052 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:48:57.052 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:48:57.052 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:57.053 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:57.053 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:57.053 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:57.053 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:57.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:57.053 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:48:57.053 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:48:57.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:57.053 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:57.053 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:57.053 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:57.053 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:57.053 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:48:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:48:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:48:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:57.054 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:57.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:57.055 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:48:57.055 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:48:57.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7bb5908640 1 -- 172.21.15.26:0/2739446967 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7b9c036370 legacy=0x7f7b9c038830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7bb5908640 10 monclient: shutdown 2024-09-16T14:48:57.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7bb5908640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:57.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7bb5908640 1 -- 172.21.15.26:0/2739446967 >> v1:172.21.15.26:6790/0 conn(0x7f7bb00a1d10 legacy=0x7f7bb0139840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7baf7fe640 1 -- 172.21.15.26:0/2739446967 reap_dead start 2024-09-16T14:48:57.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7baf7fe640 1 -- 172.21.15.26:0/2739446967 reap_dead start 2024-09-16T14:48:57.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7bb5908640 1 -- 172.21.15.26:0/2739446967 shutdown_connections 2024-09-16T14:48:57.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7bb5908640 1 -- 172.21.15.26:0/2739446967 >> 172.21.15.26:0/2739446967 conn(0x7f7bb00a7f30 msgr2=0x7f7bb013e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:57.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7bb5908640 1 -- 172.21.15.26:0/2739446967 shutdown_connections 2024-09-16T14:48:57.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.014+0000 7f7bb5908640 1 -- 172.21.15.26:0/2739446967 wait complete. 2024-09-16T14:48:57.057 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:48:57.057 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:48:57.057 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:48:57.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 10 monclient: get_monmap_and_config 2024-09-16T14:48:57.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 10 monclient: build_initial_monmap 2024-09-16T14:48:57.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 10 monclient: monmap: 2024-09-16T14:48:57.173 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:57.173 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.173 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:57.173704+0000 2024-09-16T14:48:57.173 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:57.173704+0000 2024-09-16T14:48:57.173 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:57.173 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:57.173 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:57.174 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:57.174 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:57.174 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:57.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 1 Processor -- start 2024-09-16T14:48:57.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 1 -- start start 2024-09-16T14:48:57.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 10 monclient: init 2024-09-16T14:48:57.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:57.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:57.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 10 monclient(hunting): picked mon.noname-c con 0x7f1ab80a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:57.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 10 monclient(hunting): picked mon.noname-a con 0x7f1ab8099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:57.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 10 monclient(hunting): picked mon.noname-b con 0x7f1ab80a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:57.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab8098ea0 con 0x7f1ab8099b60 2024-09-16T14:48:57.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab80997b0 con 0x7f1ab80a1c00 2024-09-16T14:48:57.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab8099460 con 0x7f1ab80a26f0 2024-09-16T14:48:57.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:57.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 10 monclient(hunting): authenticate will time out at 1361.293803s 2024-09-16T14:48:57.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abd731640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1ab8099b60 0x7f1ab8139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33986/0 (socket says 172.21.15.26:33986) 2024-09-16T14:48:57.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abd731640 1 -- 172.21.15.26:0/610687201 learned_addr learned my addr 172.21.15.26:0/610687201 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:57.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 1 -- 172.21.15.26:0/610687201 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297965927 0 0) 0x7f1ab8098ea0 con 0x7f1ab8099b60 2024-09-16T14:48:57.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient(hunting): _negotiate 2024-09-16T14:48:57.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:57.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:57.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient(hunting): my global_id is 124560 2024-09-16T14:48:57.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 1 -- 172.21.15.26:0/610687201 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa0003530 con 0x7f1ab8099b60 2024-09-16T14:48:57.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 1 -- 172.21.15.26:0/610687201 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120010562 0 0) 0x7f1ab8099460 con 0x7f1ab80a26f0 2024-09-16T14:48:57.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient(hunting): _negotiate 2024-09-16T14:48:57.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:57.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:57.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient(hunting): my global_id is 154133 2024-09-16T14:48:57.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 1 -- 172.21.15.26:0/610687201 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab8098ea0 con 0x7f1ab80a26f0 2024-09-16T14:48:57.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 1 -- 172.21.15.26:0/610687201 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 449875292 0 0) 0x7f1aa0003530 con 0x7f1ab8099b60 2024-09-16T14:48:57.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 1 -- 172.21.15.26:0/610687201 >> v1:172.21.15.26:6790/0 conn(0x7f1ab80a26f0 legacy=0x7f1ab813d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 1 -- 172.21.15.26:0/610687201 >> v1:172.21.15.189:6789/0 conn(0x7f1ab80a1c00 legacy=0x7f1ab80a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient: _finish_hunting 0 2024-09-16T14:48:57.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 1 monclient: found mon.noname-a 2024-09-16T14:48:57.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:57.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:57.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 1 -- 172.21.15.26:0/610687201 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab80a34b0 con 0x7f1ab8099b60 2024-09-16T14:48:57.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient: _finish_auth 0 2024-09-16T14:48:57.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient: _check_auth_tickets 2024-09-16T14:48:57.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:57.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 1 -- 172.21.15.26:0/610687201 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1aac001620 con 0x7f1ab8099b60 2024-09-16T14:48:57.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:57.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:57.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient: dump: 2024-09-16T14:48:57.183 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:57.183 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.183 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.183 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.183 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:57.183 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:57.183 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:57.184 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:57.184 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:57.184 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:57.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 5 monclient: authenticate success, global_id 124560 2024-09-16T14:48:57.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:57.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 1 -- 172.21.15.26:0/610687201 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1aac001900 con 0x7f1ab8099b60 2024-09-16T14:48:57.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:57.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:57.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 4 set_mon_vals no callback set 2024-09-16T14:48:57.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 1 -- 172.21.15.26:0/610687201 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1aac001c90 con 0x7f1ab8099b60 2024-09-16T14:48:57.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:57.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:57.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1abcf30640 10 monclient: dump: 2024-09-16T14:48:57.185 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:57.185 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.185 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.185 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.185 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:57.185 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:57.185 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:57.185 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:57.185 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:57.185 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:57.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 10 monclient: shutdown 2024-09-16T14:48:57.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:57.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 1 -- 172.21.15.26:0/610687201 >> v1:172.21.15.26:6789/0 conn(0x7f1ab8099b60 legacy=0x7f1ab8139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 1 -- 172.21.15.26:0/610687201 shutdown_connections 2024-09-16T14:48:57.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 1 -- 172.21.15.26:0/610687201 >> 172.21.15.26:0/610687201 conn(0x7f1ab80a3d00 msgr2=0x7f1ab80a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:57.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 1 -- 172.21.15.26:0/610687201 shutdown_connections 2024-09-16T14:48:57.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.170+0000 7f1ac0196640 1 -- 172.21.15.26:0/610687201 wait complete. 2024-09-16T14:48:57.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 monclient: build_initial_monmap 2024-09-16T14:48:57.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 monclient: monmap: 2024-09-16T14:48:57.186 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:57.186 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:57.186 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:57.187 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:57.187 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:57.187 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:57.187 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:57.187 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:57.187 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:57.187 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:57.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 1 Processor -- start 2024-09-16T14:48:57.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 1 -- start start 2024-09-16T14:48:57.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 monclient: init 2024-09-16T14:48:57.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:57.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:57.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 monclient(hunting): picked mon.noname-c con 0x7f1ab8099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:57.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 monclient(hunting): picked mon.noname-a con 0x7f1ab80a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:57.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 monclient(hunting): picked mon.noname-b con 0x7f1ab80a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:57.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab811fda0 con 0x7f1ab80a1c00 2024-09-16T14:48:57.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab809f290 con 0x7f1ab80a26f0 2024-09-16T14:48:57.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab809ed40 con 0x7f1ab8099b60 2024-09-16T14:48:57.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:57.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 monclient(hunting): authenticate will time out at 1361.296993s 2024-09-16T14:48:57.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1abd731640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1ab80a1c00 0x7f1ab813d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33998/0 (socket says 172.21.15.26:33998) 2024-09-16T14:48:57.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1abd731640 1 -- 172.21.15.26:0/3684941929 learned_addr learned my addr 172.21.15.26:0/3684941929 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:57.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1abdf32640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1ab8099b60 0x7f1ab813d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47848/0 (socket says 172.21.15.26:47848) 2024-09-16T14:48:57.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824182336 0 0) 0x7f1ab809ed40 con 0x7f1ab8099b60 2024-09-16T14:48:57.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:57.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:57.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:57.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): my global_id is 154136 2024-09-16T14:48:57.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa4003530 con 0x7f1ab8099b60 2024-09-16T14:48:57.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087342382 0 0) 0x7f1ab811fda0 con 0x7f1ab80a1c00 2024-09-16T14:48:57.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:57.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:57.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:57.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): my global_id is 124563 2024-09-16T14:48:57.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab809ed40 con 0x7f1ab80a1c00 2024-09-16T14:48:57.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3052838482 0 0) 0x7f1aa4003530 con 0x7f1ab8099b60 2024-09-16T14:48:57.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ab811fda0 con 0x7f1ab8099b60 2024-09-16T14:48:57.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 81768269 0 0) 0x7f1ab809ed40 con 0x7f1ab80a1c00 2024-09-16T14:48:57.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1aa4003530 con 0x7f1ab80a1c00 2024-09-16T14:48:57.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1ab0001b70 con 0x7f1ab8099b60 2024-09-16T14:48:57.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:57.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:57.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): dump: 2024-09-16T14:48:57.190 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:57.190 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.190 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.190 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.190 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:57.190 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:57.190 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:57.190 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:57.191 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:57.191 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:57.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:57.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1aac001440 con 0x7f1ab80a1c00 2024-09-16T14:48:57.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:57.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:57.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): dump: 2024-09-16T14:48:57.191 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:57.191 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.191 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.191 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.191 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:57.191 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:57.192 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:57.192 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:57.192 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:57.192 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:57.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553035852 0 0) 0x7f1ab809f290 con 0x7f1ab80a26f0 2024-09-16T14:48:57.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:57.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:57.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:57.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient(hunting): my global_id is 144358 2024-09-16T14:48:57.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab809ed40 con 0x7f1ab80a26f0 2024-09-16T14:48:57.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2161088414 0 0) 0x7f1ab811fda0 con 0x7f1ab8099b60 2024-09-16T14:48:57.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 >> v1:172.21.15.189:6789/0 conn(0x7f1ab80a26f0 legacy=0x7f1ab81d19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 >> v1:172.21.15.26:6789/0 conn(0x7f1ab80a1c00 legacy=0x7f1ab813d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:57.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 monclient: found mon.c 2024-09-16T14:48:57.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:57.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:57.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab809e3f0 con 0x7f1ab8099b60 2024-09-16T14:48:57.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:57.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:57.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:57.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 5 monclient: authenticate success, global_id 154136 2024-09-16T14:48:57.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 monclient: _renew_subs 2024-09-16T14:48:57.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:57.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 1 -- 172.21.15.26:0/3684941929 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab811fac0 con 0x7f1ab8099b60 2024-09-16T14:48:57.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 monclient: _renew_subs 2024-09-16T14:48:57.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:57.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 1 -- 172.21.15.26:0/3684941929 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab811da40 con 0x7f1ab8099b60 2024-09-16T14:48:57.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ab0001cd0 con 0x7f1ab8099b60 2024-09-16T14:48:57.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:57.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1ab0003b40 con 0x7f1ab8099b60 2024-09-16T14:48:57.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:57.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:57.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab7fff640 4 set_mon_vals no callback set 2024-09-16T14:48:57.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 10 monclient: dump: 2024-09-16T14:48:57.195 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:57.196 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.196 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.196 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.196 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:57.196 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:57.196 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:57.196 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:57.196 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:57.196 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:57.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:57.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:57.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:57.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ac0196640 1 -- 172.21.15.26:0/3684941929 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a98005050 con 0x7f1ab8099b60 2024-09-16T14:48:57.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1ab0003ca0 con 0x7f1ab8099b60 2024-09-16T14:48:57.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(175..175 src has 73..175) v4 ==== 6933+0+0 (unknown 3869440614 0 0) 0x7f1ab0043ff0 con 0x7f1ab8099b60 2024-09-16T14:48:57.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.174+0000 7f1abd731640 10 monclient: get_auth_request con 0x7f1aa4036560 auth_method 0 2024-09-16T14:48:57.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.178+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1ab0014680 con 0x7f1ab8099b60 2024-09-16T14:48:57.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.178+0000 7f1ab67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:57.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.178+0000 7f1ab67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:57.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.361+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}]=0 moved item id -19 name 'barr' to location {root=default} in crush map v176) v1 ==== 173+0+0 (unknown 522474870 0 0) 0x7f515c013110 con 0x7f516c1c4ff0 2024-09-16T14:48:57.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.361+0000 7f516a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T14:48:57.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.361+0000 7f516a7fc640 10 monclient: _finish_command 2 = system:0 moved item id -19 name 'barr' to location {root=default} in crush map 2024-09-16T14:48:57.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.406+0000 7f1ac0196640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:48:57.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.406+0000 7f1ac0196640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:48:57.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.406+0000 7f1ac0196640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:57.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.406+0000 7f1ac0196640 1 -- 172.21.15.26:0/3684941929 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f1a98005b10 con 0x7f1ab8099b60 2024-09-16T14:48:57.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.406+0000 7f1ab67fc640 1 -- 172.21.15.26:0/3684941929 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f1ab0012d20 con 0x7f1ab8099b60 2024-09-16T14:48:57.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.406+0000 7f1ab67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:48:57.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.406+0000 7f1ab67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:57.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.410+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.410+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=2951370839,config=692619927,health=4220061803,kv=2418557040,logm=717697860}) 2024-09-16T14:48:57.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.410+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=11,kv=11,logm=56} crc {auth=2392722814,config=692619927,health=1805846425,kv=1867599010,logm=2323307924}) 2024-09-16T14:48:57.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.410+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.410+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=2951370839,config=692619927,health=4220061803,kv=2418557040,logm=717697860}) 2024-09-16T14:48:57.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.410+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=22,config=2,health=11,kv=10,logm=55} crc {auth=4192238221,config=692619927,health=3973957579,kv=1367698,logm=3888971721}) 2024-09-16T14:48:57.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.410+0000 7f1ac0196640 1 -- 172.21.15.26:0/3684941929 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1aa4036560 legacy=0x7f1aa4038a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.410+0000 7f1ac0196640 10 monclient: shutdown 2024-09-16T14:48:57.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.410+0000 7f1ac0196640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:57.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.410+0000 7f1ac0196640 1 -- 172.21.15.26:0/3684941929 >> v1:172.21.15.26:6790/0 conn(0x7f1ab8099b60 legacy=0x7f1ab813d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.410+0000 7f1ac0196640 1 -- 172.21.15.26:0/3684941929 shutdown_connections 2024-09-16T14:48:57.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.410+0000 7f1ac0196640 1 -- 172.21.15.26:0/3684941929 >> 172.21.15.26:0/3684941929 conn(0x7f1ab80a3d00 msgr2=0x7f1ab80a0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:57.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.410+0000 7f1ac0196640 1 -- 172.21.15.26:0/3684941929 shutdown_connections 2024-09-16T14:48:57.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:57.410+0000 7f1ac0196640 1 -- 172.21.15.26:0/3684941929 wait complete. 2024-09-16T14:48:57.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.410+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.410+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2569499357}) 2024-09-16T14:48:57.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.410+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006901872}) 2024-09-16T14:48:57.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.410+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.410+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2569499357}) 2024-09-16T14:48:57.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.410+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3428921020}) 2024-09-16T14:48:57.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.414+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.414+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2301831944}) 2024-09-16T14:48:57.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.414+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2885667494}) 2024-09-16T14:48:57.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.414+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.414+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2301831944}) 2024-09-16T14:48:57.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.414+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2774443863}) 2024-09-16T14:48:57.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.414+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.414+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=282611227}) 2024-09-16T14:48:57.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.414+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2812700766}) 2024-09-16T14:48:57.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.414+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.414+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=282611227}) 2024-09-16T14:48:57.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.414+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572743175}) 2024-09-16T14:48:57.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.418+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.418+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=318403227}) 2024-09-16T14:48:57.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.418+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2757362297}) 2024-09-16T14:48:57.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.418+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.418+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=318403227}) 2024-09-16T14:48:57.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.418+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2521603288}) 2024-09-16T14:48:57.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.418+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.418+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3706032607}) 2024-09-16T14:48:57.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.418+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046362767}) 2024-09-16T14:48:57.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.418+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.418+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3706032607}) 2024-09-16T14:48:57.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.418+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1423059890}) 2024-09-16T14:48:57.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.422+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.422+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1343559890}) 2024-09-16T14:48:57.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.422+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=890323044}) 2024-09-16T14:48:57.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.422+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.422+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1343559890}) 2024-09-16T14:48:57.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.422+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1748789561}) 2024-09-16T14:48:57.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.422+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.422+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3806865253}) 2024-09-16T14:48:57.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.422+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1739337766}) 2024-09-16T14:48:57.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.422+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.422+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3806865253}) 2024-09-16T14:48:57.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.422+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=929588210}) 2024-09-16T14:48:57.430 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:48:57.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.426+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.426+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465099609}) 2024-09-16T14:48:57.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.426+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2914197680}) 2024-09-16T14:48:57.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.426+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.426+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465099609}) 2024-09-16T14:48:57.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.426+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2707761760}) 2024-09-16T14:48:57.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.426+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.426+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2896365082}) 2024-09-16T14:48:57.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.426+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2054809012}) 2024-09-16T14:48:57.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.426+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.426+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2896365082}) 2024-09-16T14:48:57.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.430+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2736074332}) 2024-09-16T14:48:57.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.430+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.430+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3741564856}) 2024-09-16T14:48:57.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.430+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3800917493}) 2024-09-16T14:48:57.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.430+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.430+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3741564856}) 2024-09-16T14:48:57.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.430+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3483375147}) 2024-09-16T14:48:57.436 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.430+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.430+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3881880406}) 2024-09-16T14:48:57.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.430+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385972827}) 2024-09-16T14:48:57.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.430+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.430+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3881880406}) 2024-09-16T14:48:57.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.430+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3573958180}) 2024-09-16T14:48:57.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.434+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.434+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1763195474}) 2024-09-16T14:48:57.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.434+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1497078750}) 2024-09-16T14:48:57.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.434+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.434+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1763195474}) 2024-09-16T14:48:57.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.434+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2480632955}) 2024-09-16T14:48:57.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.438+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.438+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3622961828}) 2024-09-16T14:48:57.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.438+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3586440048}) 2024-09-16T14:48:57.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.438+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.438+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3622961828}) 2024-09-16T14:48:57.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.438+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3226035385}) 2024-09-16T14:48:57.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.438+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.438+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=981727912}) 2024-09-16T14:48:57.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.438+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1470328777}) 2024-09-16T14:48:57.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.438+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.438+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=981727912}) 2024-09-16T14:48:57.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.438+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3428478544}) 2024-09-16T14:48:57.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.442+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.442+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4130912125}) 2024-09-16T14:48:57.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.442+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3602628439}) 2024-09-16T14:48:57.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.442+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.442+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4130912125}) 2024-09-16T14:48:57.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.442+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1558735397}) 2024-09-16T14:48:57.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.442+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.442+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3884292895}) 2024-09-16T14:48:57.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.442+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1442800109}) 2024-09-16T14:48:57.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.442+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.442+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3884292895}) 2024-09-16T14:48:57.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.442+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=347065572}) 2024-09-16T14:48:57.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.446+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.446+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2638890685}) 2024-09-16T14:48:57.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.446+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2405084882}) 2024-09-16T14:48:57.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.446+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.446+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2638890685}) 2024-09-16T14:48:57.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.446+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3476086892}) 2024-09-16T14:48:57.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.446+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.446+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=398851557}) 2024-09-16T14:48:57.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.446+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4033439829}) 2024-09-16T14:48:57.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.446+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.446+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=398851557}) 2024-09-16T14:48:57.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.446+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1271443699}) 2024-09-16T14:48:57.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.450+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.450+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3144514777}) 2024-09-16T14:48:57.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.450+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2486535694}) 2024-09-16T14:48:57.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.450+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.450+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3144514777}) 2024-09-16T14:48:57.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.450+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=851882884}) 2024-09-16T14:48:57.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.450+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.450+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2272704453}) 2024-09-16T14:48:57.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.450+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350041906}) 2024-09-16T14:48:57.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.450+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.450+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2272704453}) 2024-09-16T14:48:57.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.450+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3804598796}) 2024-09-16T14:48:57.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.454+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.454+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3022015402}) 2024-09-16T14:48:57.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.454+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676400685}) 2024-09-16T14:48:57.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.454+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.454+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3022015402}) 2024-09-16T14:48:57.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.454+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=880087964}) 2024-09-16T14:48:57.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.454+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.454+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1949157178}) 2024-09-16T14:48:57.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.454+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3856962301}) 2024-09-16T14:48:57.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.454+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.454+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1949157178}) 2024-09-16T14:48:57.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.454+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2700079522}) 2024-09-16T14:48:57.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.458+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.458+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=533963453}) 2024-09-16T14:48:57.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.458+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4262975114}) 2024-09-16T14:48:57.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.458+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.462 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.458+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=533963453}) 2024-09-16T14:48:57.462 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.458+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3671905035}) 2024-09-16T14:48:57.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.458+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.458+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1153916039}) 2024-09-16T14:48:57.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.458+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1970388452}) 2024-09-16T14:48:57.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.458+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.458+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1153916039}) 2024-09-16T14:48:57.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.458+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1047221444}) 2024-09-16T14:48:57.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.461+0000 7f5173f20640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T14:48:57.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.461+0000 7f5173f20640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T14:48:57.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.461+0000 7f5173f20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:57.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.461+0000 7f5173f20640 1 -- 172.21.15.189:0/2433742701 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush move", "name": "barr", "args": ["root=default"]} v 0) v1 -- 0x7f514c0010a0 con 0x7f516c1c4ff0 2024-09-16T14:48:57.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.462+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.462+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3264761867}) 2024-09-16T14:48:57.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.462+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=691102877}) 2024-09-16T14:48:57.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.462+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.462+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3264761867}) 2024-09-16T14:48:57.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.462+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=910779262}) 2024-09-16T14:48:57.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.462+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.462+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1673387479}) 2024-09-16T14:48:57.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.462+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3984621529}) 2024-09-16T14:48:57.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.462+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.462+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1673387479}) 2024-09-16T14:48:57.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.462+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=478538959}) 2024-09-16T14:48:57.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.466+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.466+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2354993891}) 2024-09-16T14:48:57.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.466+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1712816532}) 2024-09-16T14:48:57.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.466+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.466+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2354993891}) 2024-09-16T14:48:57.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.466+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=925626164}) 2024-09-16T14:48:57.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.466+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.466+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=554455492}) 2024-09-16T14:48:57.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.466+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2939441473}) 2024-09-16T14:48:57.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.466+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.466+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=554455492}) 2024-09-16T14:48:57.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.466+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=921847187}) 2024-09-16T14:48:57.475 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.470+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.475 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.470+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=52,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2} crc {logm=1541730428,mdsmap=1531151339,mgr=620920336,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3185416618,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=114150881,osd_snap=3794734670}) 2024-09-16T14:48:57.475 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.470+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=54,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2} crc {logm=1595397528,mdsmap=2394902515,mgr=3587032506,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3185416619,mon_config_key=321861987,monmap=970648239,osd_metadata=3424249799,osd_pg_creating=114150881,osd_snap=3794734670}) 2024-09-16T14:48:57.475 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.470+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.475 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.470+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=52,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2} crc {logm=1541730428,mdsmap=1531151339,mgr=620920336,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3185416618,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=114150881,osd_snap=3794734670}) 2024-09-16T14:48:57.475 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.470+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=49,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=6} crc {logm=3616934091,mdsmap=1531151339,mgr=2539502010,mgr_metadata=2109538692,mgrstat=4009131731,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=114150881,osd_snap=3794734670,osdmap=3828548121}) 2024-09-16T14:48:57.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.474+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.474+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3318618313}) 2024-09-16T14:48:57.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.474+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1266103307}) 2024-09-16T14:48:57.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.474+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.474+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3318618313}) 2024-09-16T14:48:57.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.474+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1677418853}) 2024-09-16T14:48:57.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.482+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.482+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=67} crc {osdmap=1580007588}) 2024-09-16T14:48:57.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.482+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=67} crc {osdmap=856255053}) 2024-09-16T14:48:57.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.482+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:48:57.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.482+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=67} crc {osdmap=1580007588}) 2024-09-16T14:48:57.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:48:57.482+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=64} crc {osdmap=2865212135}) 2024-09-16T14:48:57.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.493+0000 7f516a7fc640 1 -- 172.21.15.189:0/2433742701 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}]=0 no need to move item id -19 name 'barr' to location {root=default} in crush map v176) v1 ==== 183+0+0 (unknown 3026868651 0 0) 0x7f515c014890 con 0x7f516c1c4ff0 2024-09-16T14:48:57.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.493+0000 7f516a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T14:48:57.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.493+0000 7f516a7fc640 10 monclient: _finish_command 3 = system:0 no need to move item id -19 name 'barr' to location {root=default} in crush map 2024-09-16T14:48:57.494 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id -19 name 'barr' to location {root=default} in crush map 2024-09-16T14:48:57.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.493+0000 7f5173f20640 1 -- 172.21.15.189:0/2433742701 >> v1:172.21.15.189:6812/3196161582 conn(0x7f51580363e0 legacy=0x7f51580388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.493+0000 7f5173f20640 10 monclient: shutdown 2024-09-16T14:48:57.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.493+0000 7f5173f20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:57.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.493+0000 7f5173f20640 1 -- 172.21.15.189:0/2433742701 >> v1:172.21.15.189:6789/0 conn(0x7f516c1c4ff0 legacy=0x7f516c1cd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.497+0000 7f5173f20640 1 -- 172.21.15.189:0/2433742701 shutdown_connections 2024-09-16T14:48:57.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.497+0000 7f5173f20640 1 -- 172.21.15.189:0/2433742701 >> 172.21.15.189:0/2433742701 conn(0x7f516c0a3320 msgr2=0x7f516c0a1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:57.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.497+0000 7f5173f20640 1 -- 172.21.15.189:0/2433742701 shutdown_connections 2024-09-16T14:48:57.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.497+0000 7f5173f20640 1 -- 172.21.15.189:0/2433742701 wait complete. 2024-09-16T14:48:57.512 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush move fooo rack=barr 2024-09-16T14:48:57.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.629+0000 7f99ac21f640 10 monclient: get_monmap_and_config 2024-09-16T14:48:57.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.629+0000 7f99ac21f640 10 monclient: build_initial_monmap 2024-09-16T14:48:57.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.629+0000 7f99ac21f640 10 monclient: monmap: 2024-09-16T14:48:57.632 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:57.632 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.632 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:57.632156+0000 2024-09-16T14:48:57.632 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:57.632156+0000 2024-09-16T14:48:57.632 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:57.632 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:57.632 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:57.632 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:57.632 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:57.632 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:57.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.629+0000 7f99ac21f640 1 Processor -- start 2024-09-16T14:48:57.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.629+0000 7f99ac21f640 1 -- start start 2024-09-16T14:48:57.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.629+0000 7f99ac21f640 10 monclient: init 2024-09-16T14:48:57.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:57.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:57.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient(hunting): picked mon.noname-b con 0x7f99a41315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:57.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient(hunting): picked mon.noname-c con 0x7f99a409b320 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:57.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient(hunting): picked mon.noname-a con 0x7f99a4132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:57.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a409edd0 con 0x7f99a4132090 2024-09-16T14:48:57.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a409bc00 con 0x7f99a409b320 2024-09-16T14:48:57.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a409e900 con 0x7f99a41315a0 2024-09-16T14:48:57.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:57.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient(hunting): authenticate will time out at 1319.748301s 2024-09-16T14:48:57.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a97ba640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f99a409b320 0x7f99a4136ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46986/0 (socket says 172.21.15.189:46986) 2024-09-16T14:48:57.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a97ba640 1 -- 172.21.15.189:0/3330503123 learned_addr learned my addr 172.21.15.189:0/3330503123 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:57.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 1 -- 172.21.15.189:0/3330503123 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611615193 0 0) 0x7f99a409bc00 con 0x7f99a409b320 2024-09-16T14:48:57.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:57.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:57.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:57.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 10 monclient(hunting): my global_id is 144361 2024-09-16T14:48:57.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 1 -- 172.21.15.189:0/3330503123 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9990003530 con 0x7f99a409b320 2024-09-16T14:48:57.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 1 -- 172.21.15.189:0/3330503123 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 772071831 0 0) 0x7f9990003530 con 0x7f99a409b320 2024-09-16T14:48:57.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 1 -- 172.21.15.189:0/3330503123 >> v1:172.21.15.26:6790/0 conn(0x7f99a41315a0 legacy=0x7f99a4131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 1 -- 172.21.15.189:0/3330503123 >> v1:172.21.15.26:6789/0 conn(0x7f99a4132090 legacy=0x7f99a413a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 10 monclient: _finish_hunting 0 2024-09-16T14:48:57.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 1 monclient: found mon.noname-c 2024-09-16T14:48:57.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:57.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:57.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 1 -- 172.21.15.189:0/3330503123 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a409b920 con 0x7f99a409b320 2024-09-16T14:48:57.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 10 monclient: _finish_auth 0 2024-09-16T14:48:57.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 10 monclient: _check_auth_tickets 2024-09-16T14:48:57.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:57.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 5 monclient: authenticate success, global_id 144361 2024-09-16T14:48:57.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 1 -- 172.21.15.189:0/3330503123 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9994001620 con 0x7f99a409b320 2024-09-16T14:48:57.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:57.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:57.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:57.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 10 monclient: dump: 2024-09-16T14:48:57.638 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:57.638 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.638 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.639 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.639 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:57.639 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:57.639 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:57.639 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:57.639 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:57.639 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:57.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:57.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 1 -- 172.21.15.189:0/3330503123 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99940018d0 con 0x7f99a409b320 2024-09-16T14:48:57.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:57.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:57.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 4 set_mon_vals no callback set 2024-09-16T14:48:57.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 1 -- 172.21.15.189:0/3330503123 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9994001ca0 con 0x7f99a409b320 2024-09-16T14:48:57.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:57.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:57.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99a8fb9640 10 monclient: dump: 2024-09-16T14:48:57.640 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:57.640 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.640 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.640 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.640 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:57.640 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:57.640 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:57.640 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:57.640 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:57.640 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:57.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient: shutdown 2024-09-16T14:48:57.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:57.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 1 -- 172.21.15.189:0/3330503123 >> v1:172.21.15.189:6789/0 conn(0x7f99a409b320 legacy=0x7f99a4136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 1 -- 172.21.15.189:0/3330503123 shutdown_connections 2024-09-16T14:48:57.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 1 -- 172.21.15.189:0/3330503123 >> 172.21.15.189:0/3330503123 conn(0x7f99a40a3320 msgr2=0x7f99a409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:57.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 1 -- 172.21.15.189:0/3330503123 shutdown_connections 2024-09-16T14:48:57.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 1 -- 172.21.15.189:0/3330503123 wait complete. 2024-09-16T14:48:57.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient: build_initial_monmap 2024-09-16T14:48:57.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient: monmap: 2024-09-16T14:48:57.645 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:57.645 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:57.645 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:57.645 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:57.645 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:57.645 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:57.645 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:57.645 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:57.645 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:57.645 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:57.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 1 Processor -- start 2024-09-16T14:48:57.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 1 -- start start 2024-09-16T14:48:57.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient: init 2024-09-16T14:48:57.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:57.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:57.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient(hunting): picked mon.noname-c con 0x7f99a41315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:57.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient(hunting): picked mon.noname-a con 0x7f99a4132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:57.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient(hunting): picked mon.noname-b con 0x7f99a4070670 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:57.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a409cd20 con 0x7f99a4132090 2024-09-16T14:48:57.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a413d110 con 0x7f99a4070670 2024-09-16T14:48:57.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a413cac0 con 0x7f99a41315a0 2024-09-16T14:48:57.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:57.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99ac21f640 10 monclient(hunting): authenticate will time out at 1319.751754s 2024-09-16T14:48:57.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99aa7bc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f99a4070670 0x7f99a41d1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46990/0 (socket says 172.21.15.189:46990) 2024-09-16T14:48:57.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.633+0000 7f99aa7bc640 1 -- 172.21.15.189:0/2237255910 learned_addr learned my addr 172.21.15.189:0/2237255910 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:57.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710017027 0 0) 0x7f99a413d110 con 0x7f99a4070670 2024-09-16T14:48:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient(hunting): my global_id is 144364 2024-09-16T14:48:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9988003560 con 0x7f99a4070670 2024-09-16T14:48:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802078835 0 0) 0x7f99a409cd20 con 0x7f99a4132090 2024-09-16T14:48:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient(hunting): my global_id is 124569 2024-09-16T14:48:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99a413d110 con 0x7f99a4132090 2024-09-16T14:48:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3655245439 0 0) 0x7f9988003560 con 0x7f99a4070670 2024-09-16T14:48:57.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99a409cd20 con 0x7f99a4070670 2024-09-16T14:48:57.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f999c001c80 con 0x7f99a4070670 2024-09-16T14:48:57.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:57.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:57.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient(hunting): dump: 2024-09-16T14:48:57.648 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:57.648 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.648 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.648 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.648 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:57.648 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:57.648 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:57.648 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:57.649 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:57.649 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:57.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:57.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889861065 0 0) 0x7f99a413cac0 con 0x7f99a41315a0 2024-09-16T14:48:57.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:57.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:57.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:57.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient(hunting): my global_id is 154142 2024-09-16T14:48:57.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9988003560 con 0x7f99a41315a0 2024-09-16T14:48:57.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3658297360 0 0) 0x7f99a409cd20 con 0x7f99a4070670 2024-09-16T14:48:57.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 >> v1:172.21.15.26:6790/0 conn(0x7f99a41315a0 legacy=0x7f99a413b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 >> v1:172.21.15.26:6789/0 conn(0x7f99a4132090 legacy=0x7f99a413c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 monclient: found mon.b 2024-09-16T14:48:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a40a0ef0 con 0x7f99a4070670 2024-09-16T14:48:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f99ac21f640 5 monclient: authenticate success, global_id 144364 2024-09-16T14:48:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f99ac21f640 10 monclient: _renew_subs 2024-09-16T14:48:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f99ac21f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f99ac21f640 1 -- 172.21.15.189:0/2237255910 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a406b380 con 0x7f99a4070670 2024-09-16T14:48:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f99ac21f640 10 monclient: _renew_subs 2024-09-16T14:48:57.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f99ac21f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:57.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f99ac21f640 1 -- 172.21.15.189:0/2237255910 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a413c740 con 0x7f99a4070670 2024-09-16T14:48:57.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f999c0039a0 con 0x7f99a4070670 2024-09-16T14:48:57.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:57.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f999c003cb0 con 0x7f99a4070670 2024-09-16T14:48:57.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:57.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:57.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 10 monclient: dump: 2024-09-16T14:48:57.651 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:57.651 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:57.651 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.651 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:57.651 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:57.652 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:57.652 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:57.652 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:57.652 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:57.652 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:57.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999bfff640 4 set_mon_vals no callback set 2024-09-16T14:48:57.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f99ac21f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:57.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f99ac21f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:57.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f99ac21f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:57.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f99ac21f640 1 -- 172.21.15.189:0/2237255910 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f998c005050 con 0x7f99a4070670 2024-09-16T14:48:57.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f999c003e90 con 0x7f99a4070670 2024-09-16T14:48:57.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(176..176 src has 73..176) v4 ==== 7121+0+0 (unknown 2099983964 0 0) 0x7f999c044320 con 0x7f99a4070670 2024-09-16T14:48:57.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.637+0000 7f99a9fbb640 10 monclient: get_auth_request con 0x7f9988036390 auth_method 0 2024-09-16T14:48:57.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.641+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f999c014b40 con 0x7f99a4070670 2024-09-16T14:48:57.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.641+0000 7f999a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:57.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.641+0000 7f999a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:57.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.801+0000 7f99ac21f640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T14:48:57.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.801+0000 7f99ac21f640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T14:48:57.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.801+0000 7f99ac21f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:57.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:57.801+0000 7f99ac21f640 1 -- 172.21.15.189:0/2237255910 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]} v 0) v1 -- 0x7f998c005880 con 0x7f99a4070670 2024-09-16T14:48:58.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.365+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}]=0 moved item id -16 name 'fooo' to location {rack=barr} in crush map v177) v1 ==== 167+0+0 (unknown 374039905 0 0) 0x7f999c012fb0 con 0x7f99a4070670 2024-09-16T14:48:58.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.365+0000 7f999a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T14:48:58.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.365+0000 7f999a7fc640 10 monclient: _finish_command 2 = system:0 moved item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-16T14:48:58.432 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:48:58.432 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:48:58.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.465+0000 7f99ac21f640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T14:48:58.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.465+0000 7f99ac21f640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T14:48:58.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.465+0000 7f99ac21f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:58.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.465+0000 7f99ac21f640 1 -- 172.21.15.189:0/2237255910 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]} v 0) v1 -- 0x7f998c005300 con 0x7f99a4070670 2024-09-16T14:48:58.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.469+0000 7f999a7fc640 1 -- 172.21.15.189:0/2237255910 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}]=0 no need to move item id -16 name 'fooo' to location {rack=barr} in crush map v177) v1 ==== 177+0+0 (unknown 1323730099 0 0) 0x7f999c014910 con 0x7f99a4070670 2024-09-16T14:48:58.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.469+0000 7f999a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T14:48:58.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.469+0000 7f999a7fc640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-16T14:48:58.470 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-16T14:48:58.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.469+0000 7f99ac21f640 1 -- 172.21.15.189:0/2237255910 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9988036390 legacy=0x7f9988038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.469+0000 7f99ac21f640 10 monclient: shutdown 2024-09-16T14:48:58.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.469+0000 7f99ac21f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:58.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.469+0000 7f99ac21f640 1 -- 172.21.15.189:0/2237255910 >> v1:172.21.15.189:6789/0 conn(0x7f99a4070670 legacy=0x7f99a41d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.469+0000 7f99ac21f640 1 -- 172.21.15.189:0/2237255910 shutdown_connections 2024-09-16T14:48:58.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.469+0000 7f99ac21f640 1 -- 172.21.15.189:0/2237255910 >> 172.21.15.189:0/2237255910 conn(0x7f99a40a3320 msgr2=0x7f99a41296e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:58.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.469+0000 7f99ac21f640 1 -- 172.21.15.189:0/2237255910 shutdown_connections 2024-09-16T14:48:58.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.469+0000 7f99ac21f640 1 -- 172.21.15.189:0/2237255910 wait complete. 2024-09-16T14:48:58.487 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm fooo 2024-09-16T14:48:58.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.558+0000 7f2674f6f640 10 monclient: get_monmap_and_config 2024-09-16T14:48:58.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.558+0000 7f2674f6f640 10 monclient: build_initial_monmap 2024-09-16T14:48:58.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.558+0000 7f2674f6f640 10 monclient: monmap: 2024-09-16T14:48:58.563 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:58.563 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.563 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:58.563363+0000 2024-09-16T14:48:58.563 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:58.563363+0000 2024-09-16T14:48:58.563 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:58.563 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:58.563 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:58.563 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:58.563 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:58.564 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:58.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.558+0000 7f2674f6f640 1 Processor -- start 2024-09-16T14:48:58.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.558+0000 7f2674f6f640 1 -- start start 2024-09-16T14:48:58.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.558+0000 7f2674f6f640 10 monclient: init 2024-09-16T14:48:58.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.558+0000 7f2674f6f640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:58.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.558+0000 7f2674f6f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:48:58.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.558+0000 7f2674f6f640 10 monclient(hunting): picked mon.noname-c con 0x7f267009a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:58.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.558+0000 7f2674f6f640 10 monclient(hunting): picked mon.noname-a con 0x7f267009d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:58.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.558+0000 7f2674f6f640 10 monclient(hunting): picked mon.noname-b con 0x7f267009b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:58.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.558+0000 7f2674f6f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267009dc60 con 0x7f267009d1b0 2024-09-16T14:48:58.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.558+0000 7f2674f6f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2670099ae0 con 0x7f267009a550 2024-09-16T14:48:58.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.558+0000 7f2674f6f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267009b8f0 con 0x7f267009b040 2024-09-16T14:48:58.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.558+0000 7f2674f6f640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:58.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.558+0000 7f2674f6f640 10 monclient(hunting): authenticate will time out at 1362.683951s 2024-09-16T14:48:58.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2667fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f267009d1b0 0x7f2670136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34010/0 (socket says 172.21.15.26:34010) 2024-09-16T14:48:58.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2667fff640 1 -- 172.21.15.26:0/2723016940 learned_addr learned my addr 172.21.15.26:0/2723016940 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:58.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 1 -- 172.21.15.26:0/2723016940 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997731664 0 0) 0x7f267009dc60 con 0x7f267009d1b0 2024-09-16T14:48:58.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient(hunting): my global_id is 124572 2024-09-16T14:48:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 1 -- 172.21.15.26:0/2723016940 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f265c003530 con 0x7f267009d1b0 2024-09-16T14:48:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 1 -- 172.21.15.26:0/2723016940 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119414007 0 0) 0x7f267009b8f0 con 0x7f267009b040 2024-09-16T14:48:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient(hunting): my global_id is 154145 2024-09-16T14:48:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 1 -- 172.21.15.26:0/2723016940 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f267009dc60 con 0x7f267009b040 2024-09-16T14:48:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 1 -- 172.21.15.26:0/2723016940 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909136201 0 0) 0x7f2670099ae0 con 0x7f267009a550 2024-09-16T14:48:58.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient(hunting): my global_id is 144367 2024-09-16T14:48:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 1 -- 172.21.15.26:0/2723016940 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f267009b8f0 con 0x7f267009a550 2024-09-16T14:48:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 1 -- 172.21.15.26:0/2723016940 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1551898400 0 0) 0x7f265c003530 con 0x7f267009d1b0 2024-09-16T14:48:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 1 -- 172.21.15.26:0/2723016940 >> v1:172.21.15.26:6790/0 conn(0x7f267009b040 legacy=0x7f267013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 1 -- 172.21.15.26:0/2723016940 >> v1:172.21.15.189:6789/0 conn(0x7f267009a550 legacy=0x7f267009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient: _finish_hunting 0 2024-09-16T14:48:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 1 monclient: found mon.noname-a 2024-09-16T14:48:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 1 -- 172.21.15.26:0/2723016940 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267009bbd0 con 0x7f267009d1b0 2024-09-16T14:48:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient: _finish_auth 0 2024-09-16T14:48:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient: _check_auth_tickets 2024-09-16T14:48:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 1 -- 172.21.15.26:0/2723016940 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2658001620 con 0x7f267009d1b0 2024-09-16T14:48:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient: dump: 2024-09-16T14:48:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:58.571 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:58.572 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 5 monclient: authenticate success, global_id 124572 2024-09-16T14:48:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 1 -- 172.21.15.26:0/2723016940 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2658001900 con 0x7f267009d1b0 2024-09-16T14:48:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 1 -- 172.21.15.26:0/2723016940 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2658001c90 con 0x7f267009d1b0 2024-09-16T14:48:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:58.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f266dd9b640 10 monclient: dump: 2024-09-16T14:48:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:58.573 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 4 set_mon_vals no callback set 2024-09-16T14:48:58.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 10 monclient: shutdown 2024-09-16T14:48:58.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:58.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 1 -- 172.21.15.26:0/2723016940 >> v1:172.21.15.26:6789/0 conn(0x7f267009d1b0 legacy=0x7f2670136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 1 -- 172.21.15.26:0/2723016940 shutdown_connections 2024-09-16T14:48:58.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 1 -- 172.21.15.26:0/2723016940 >> 172.21.15.26:0/2723016940 conn(0x7f267009fb50 msgr2=0x7f267009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:58.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 1 -- 172.21.15.26:0/2723016940 shutdown_connections 2024-09-16T14:48:58.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 1 -- 172.21.15.26:0/2723016940 wait complete. 2024-09-16T14:48:58.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 10 monclient: build_initial_monmap 2024-09-16T14:48:58.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 10 monclient: monmap: 2024-09-16T14:48:58.578 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:58.578 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:58.578 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:58.578 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:58.578 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:58.578 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:58.578 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:58.578 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:58.579 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:58.579 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:58.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 1 Processor -- start 2024-09-16T14:48:58.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 1 -- start start 2024-09-16T14:48:58.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 10 monclient: init 2024-09-16T14:48:58.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:58.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:58.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 10 monclient(hunting): picked mon.noname-c con 0x7f267009a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:58.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 10 monclient(hunting): picked mon.noname-b con 0x7f267009b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:58.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 10 monclient(hunting): picked mon.noname-a con 0x7f267009d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:58.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.562+0000 7f2674f6f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26700a2450 con 0x7f267009d1b0 2024-09-16T14:48:58.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2674f6f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26701c5bd0 con 0x7f267009b040 2024-09-16T14:48:58.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2674f6f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26701a2fb0 con 0x7f267009a550 2024-09-16T14:48:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2674f6f640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2674f6f640 10 monclient(hunting): authenticate will time out at 1362.688081s 2024-09-16T14:48:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f266e59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f267009a550 0x7f26701b48a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47874/0 (socket says 172.21.15.26:47874) 2024-09-16T14:48:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f266e59c640 1 -- 172.21.15.26:0/3774420547 learned_addr learned my addr 172.21.15.26:0/3774420547 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723932177 0 0) 0x7f26701a2fb0 con 0x7f267009a550 2024-09-16T14:48:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): my global_id is 154148 2024-09-16T14:48:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2650003530 con 0x7f267009a550 2024-09-16T14:48:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755151602 0 0) 0x7f26700a2450 con 0x7f267009d1b0 2024-09-16T14:48:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): my global_id is 124575 2024-09-16T14:48:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26701a2fb0 con 0x7f267009d1b0 2024-09-16T14:48:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1479269362 0 0) 0x7f2650003530 con 0x7f267009a550 2024-09-16T14:48:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26700a2450 con 0x7f267009a550 2024-09-16T14:48:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2660001940 con 0x7f267009a550 2024-09-16T14:48:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): dump: 2024-09-16T14:48:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.582 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.582 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.582 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:58.582 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:58.582 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:58.582 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:58.582 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:58.582 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:58.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:58.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036196818 0 0) 0x7f26701c5bd0 con 0x7f267009b040 2024-09-16T14:48:58.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): my global_id is 144370 2024-09-16T14:48:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2650003530 con 0x7f267009b040 2024-09-16T14:48:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 20590278 0 0) 0x7f26701a2fb0 con 0x7f267009d1b0 2024-09-16T14:48:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26701c5bd0 con 0x7f267009d1b0 2024-09-16T14:48:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2668001bc0 con 0x7f267009d1b0 2024-09-16T14:48:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient(hunting): dump: 2024-09-16T14:48:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:58.584 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1542036590 0 0) 0x7f26700a2450 con 0x7f267009a550 2024-09-16T14:48:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 >> v1:172.21.15.189:6789/0 conn(0x7f267009b040 legacy=0x7f26701c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 >> v1:172.21.15.26:6789/0 conn(0x7f267009d1b0 legacy=0x7f26701cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 monclient: found mon.c 2024-09-16T14:48:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26701a3270 con 0x7f267009a550 2024-09-16T14:48:58.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient: _finish_auth 0 2024-09-16T14:48:58.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:58.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:58.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2674f6f640 5 monclient: authenticate success, global_id 154148 2024-09-16T14:48:58.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2674f6f640 10 monclient: _renew_subs 2024-09-16T14:48:58.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2674f6f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:58.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2674f6f640 1 -- 172.21.15.26:0/3774420547 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26701b5520 con 0x7f267009a550 2024-09-16T14:48:58.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2674f6f640 10 monclient: _renew_subs 2024-09-16T14:48:58.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2674f6f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:58.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2674f6f640 1 -- 172.21.15.26:0/3774420547 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26701a29d0 con 0x7f267009a550 2024-09-16T14:48:58.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2660003620 con 0x7f267009a550 2024-09-16T14:48:58.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:58.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2660003930 con 0x7f267009a550 2024-09-16T14:48:58.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:58.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:58.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f266d59a640 4 set_mon_vals no callback set 2024-09-16T14:48:58.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 10 monclient: dump: 2024-09-16T14:48:58.586 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:58.586 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.586 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.586 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.586 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:58.586 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:58.586 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:58.586 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:58.587 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:58.587 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:58.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2674f6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:58.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2674f6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:58.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2674f6f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:58.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2660003a90 con 0x7f267009a550 2024-09-16T14:48:58.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2674f6f640 1 -- 172.21.15.26:0/3774420547 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2654005050 con 0x7f267009a550 2024-09-16T14:48:58.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(177..177 src has 73..177) v4 ==== 7121+0+0 (unknown 2880899562 0 0) 0x7f2660043e80 con 0x7f267009a550 2024-09-16T14:48:58.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.566+0000 7f2667fff640 10 monclient: get_auth_request con 0x7f2650036490 auth_method 0 2024-09-16T14:48:58.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.570+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2660004290 con 0x7f267009a550 2024-09-16T14:48:58.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.570+0000 7f2666ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:58.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.570+0000 7f2666ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:58.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 10 monclient: get_monmap_and_config 2024-09-16T14:48:58.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 10 monclient: build_initial_monmap 2024-09-16T14:48:58.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 10 monclient: monmap: 2024-09-16T14:48:58.606 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:58.606 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.606 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:48:58.606426+0000 2024-09-16T14:48:58.606 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:48:58.606426+0000 2024-09-16T14:48:58.606 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:58.606 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:58.607 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:58.607 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:58.607 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:58.607 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:58.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 1 Processor -- start 2024-09-16T14:48:58.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 1 -- start start 2024-09-16T14:48:58.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 10 monclient: init 2024-09-16T14:48:58.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:58.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:58.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 10 monclient(hunting): picked mon.noname-b con 0x7f57a009df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:58.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 10 monclient(hunting): picked mon.noname-a con 0x7f57a009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:58.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 10 monclient(hunting): picked mon.noname-c con 0x7f57a009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:58.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a009ee20 con 0x7f57a009e2f0 2024-09-16T14:48:58.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a009f100 con 0x7f57a009e740 2024-09-16T14:48:58.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a009f3e0 con 0x7f57a009df10 2024-09-16T14:48:58.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:58.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 10 monclient(hunting): authenticate will time out at 1320.722204s 2024-09-16T14:48:58.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a4caf640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f57a009e740 0x7f57a013b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47006/0 (socket says 172.21.15.189:47006) 2024-09-16T14:48:58.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a4caf640 1 -- 172.21.15.189:0/1248521561 learned_addr learned my addr 172.21.15.189:0/1248521561 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:58.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 1 -- 172.21.15.189:0/1248521561 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3670513878 0 0) 0x7f57a009f100 con 0x7f57a009e740 2024-09-16T14:48:58.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient(hunting): my global_id is 144373 2024-09-16T14:48:58.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 1 -- 172.21.15.189:0/1248521561 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5788003530 con 0x7f57a009e740 2024-09-16T14:48:58.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 1 -- 172.21.15.189:0/1248521561 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344938146 0 0) 0x7f57a009f3e0 con 0x7f57a009df10 2024-09-16T14:48:58.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient(hunting): my global_id is 154151 2024-09-16T14:48:58.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 1 -- 172.21.15.189:0/1248521561 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a009f100 con 0x7f57a009df10 2024-09-16T14:48:58.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 1 -- 172.21.15.189:0/1248521561 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709253209 0 0) 0x7f57a009ee20 con 0x7f57a009e2f0 2024-09-16T14:48:58.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient(hunting): my global_id is 124578 2024-09-16T14:48:58.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 1 -- 172.21.15.189:0/1248521561 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a009f3e0 con 0x7f57a009e2f0 2024-09-16T14:48:58.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 1 -- 172.21.15.189:0/1248521561 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2030046886 0 0) 0x7f5788003530 con 0x7f57a009e740 2024-09-16T14:48:58.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 1 -- 172.21.15.189:0/1248521561 >> v1:172.21.15.26:6790/0 conn(0x7f57a009df10 legacy=0x7f57a009d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 1 -- 172.21.15.189:0/1248521561 >> v1:172.21.15.26:6789/0 conn(0x7f57a009e2f0 legacy=0x7f57a01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient: _finish_hunting 0 2024-09-16T14:48:58.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 1 monclient: found mon.noname-c 2024-09-16T14:48:58.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:58.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:48:58.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 1 -- 172.21.15.189:0/1248521561 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a009c690 con 0x7f57a009e740 2024-09-16T14:48:58.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient: _finish_auth 0 2024-09-16T14:48:58.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient: _check_auth_tickets 2024-09-16T14:48:58.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:58.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 5 monclient: authenticate success, global_id 144373 2024-09-16T14:48:58.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:58.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 1 -- 172.21.15.189:0/1248521561 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f57980019a0 con 0x7f57a009e740 2024-09-16T14:48:58.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:58.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:58.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient: dump: 2024-09-16T14:48:58.618 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:58.618 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.618 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.618 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.618 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:58.618 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:58.618 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:58.618 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:58.618 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:58.619 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:58.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:58.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 1 -- 172.21.15.189:0/1248521561 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5798001c80 con 0x7f57a009e740 2024-09-16T14:48:58.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:58.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 1 -- 172.21.15.189:0/1248521561 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5798002010 con 0x7f57a009e740 2024-09-16T14:48:58.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:58.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 4 set_mon_vals no callback set 2024-09-16T14:48:58.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:58.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:58.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f579effd640 10 monclient: dump: 2024-09-16T14:48:58.619 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:58.619 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.619 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.620 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.620 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:58.620 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:58.620 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:58.620 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:58.620 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:58.620 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:58.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 10 monclient: shutdown 2024-09-16T14:48:58.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:58.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 1 -- 172.21.15.189:0/1248521561 >> v1:172.21.15.189:6789/0 conn(0x7f57a009e740 legacy=0x7f57a013b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 1 -- 172.21.15.189:0/1248521561 shutdown_connections 2024-09-16T14:48:58.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 1 -- 172.21.15.189:0/1248521561 >> 172.21.15.189:0/1248521561 conn(0x7f57a00a3320 msgr2=0x7f57a009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:58.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 1 -- 172.21.15.189:0/1248521561 shutdown_connections 2024-09-16T14:48:58.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.605+0000 7f57a6712640 1 -- 172.21.15.189:0/1248521561 wait complete. 2024-09-16T14:48:58.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 monclient: build_initial_monmap 2024-09-16T14:48:58.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 monclient: monmap: 2024-09-16T14:48:58.621 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:48:58.621 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:58.621 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:48:58.621 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:48:58.621 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:58.621 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:48:58.621 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:58.622 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:58.622 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:58.622 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:58.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 1 Processor -- start 2024-09-16T14:48:58.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 1 -- start start 2024-09-16T14:48:58.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 monclient: init 2024-09-16T14:48:58.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:58.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:58.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 monclient(hunting): picked mon.noname-b con 0x7f57a009df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:58.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 monclient(hunting): picked mon.noname-c con 0x7f57a009e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:58.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 monclient(hunting): picked mon.noname-a con 0x7f57a009e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:58.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a006fae0 con 0x7f57a009e740 2024-09-16T14:48:58.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a011a2d0 con 0x7f57a009df10 2024-09-16T14:48:58.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a013a740 con 0x7f57a009e2f0 2024-09-16T14:48:58.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:58.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 monclient(hunting): authenticate will time out at 1320.725252s 2024-09-16T14:48:58.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579ffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f57a009df10 0x7f57a0139920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47008/0 (socket says 172.21.15.189:47008) 2024-09-16T14:48:58.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579ffff640 1 -- 172.21.15.189:0/2362306898 learned_addr learned my addr 172.21.15.189:0/2362306898 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:48:58.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1483989127 0 0) 0x7f57a011a2d0 con 0x7f57a009df10 2024-09-16T14:48:58.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient(hunting): my global_id is 144376 2024-09-16T14:48:58.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f578c003530 con 0x7f57a009df10 2024-09-16T14:48:58.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763235332 0 0) 0x7f57a006fae0 con 0x7f57a009e740 2024-09-16T14:48:58.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient(hunting): my global_id is 124581 2024-09-16T14:48:58.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a011a2d0 con 0x7f57a009e740 2024-09-16T14:48:58.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167723752 0 0) 0x7f57a013a740 con 0x7f57a009e2f0 2024-09-16T14:48:58.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient(hunting): my global_id is 154154 2024-09-16T14:48:58.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a006fae0 con 0x7f57a009e2f0 2024-09-16T14:48:58.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1777208526 0 0) 0x7f578c003530 con 0x7f57a009df10 2024-09-16T14:48:58.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57a013a740 con 0x7f57a009df10 2024-09-16T14:48:58.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5790001900 con 0x7f57a009df10 2024-09-16T14:48:58.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:58.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:58.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient(hunting): dump: 2024-09-16T14:48:58.625 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:58.625 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.625 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.625 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.625 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:58.625 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:58.625 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:58.626 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:58.626 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:58.626 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:58.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:58.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2595550939 0 0) 0x7f57a011a2d0 con 0x7f57a009e740 2024-09-16T14:48:58.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f578c003530 con 0x7f57a009e740 2024-09-16T14:48:58.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2140217417 0 0) 0x7f57a013a740 con 0x7f57a009df10 2024-09-16T14:48:58.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 >> v1:172.21.15.26:6790/0 conn(0x7f57a009e2f0 legacy=0x7f57a013a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 >> v1:172.21.15.26:6789/0 conn(0x7f57a009e740 legacy=0x7f57a01cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:48:58.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 monclient: found mon.b 2024-09-16T14:48:58.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:58.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:58.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a006cfb0 con 0x7f57a009df10 2024-09-16T14:48:58.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient: _finish_auth 0 2024-09-16T14:48:58.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:48:58.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:58.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 5 monclient: authenticate success, global_id 144376 2024-09-16T14:48:58.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 monclient: _renew_subs 2024-09-16T14:48:58.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:58.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 1 -- 172.21.15.189:0/2362306898 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a006b380 con 0x7f57a009df10 2024-09-16T14:48:58.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 monclient: _renew_subs 2024-09-16T14:48:58.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:58.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 1 -- 172.21.15.189:0/2362306898 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a013bf10 con 0x7f57a009df10 2024-09-16T14:48:58.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5790003660 con 0x7f57a009df10 2024-09-16T14:48:58.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:58.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5790003970 con 0x7f57a009df10 2024-09-16T14:48:58.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:58.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:58.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579e7fc640 4 set_mon_vals no callback set 2024-09-16T14:48:58.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 10 monclient: dump: 2024-09-16T14:48:58.630 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:48:58.630 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.630 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.631 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.631 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:58.631 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:48:58.631 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:58.631 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:58.631 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:58.631 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:48:58.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:58.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:58.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:58.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f57a6712640 1 -- 172.21.15.189:0/2362306898 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5780005050 con 0x7f57a009df10 2024-09-16T14:48:58.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5790003ad0 con 0x7f57a009df10 2024-09-16T14:48:58.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(177..177 src has 73..177) v4 ==== 7121+0+0 (unknown 2880899562 0 0) 0x7f5790043ec0 con 0x7f57a009df10 2024-09-16T14:48:58.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.609+0000 7f579f7fe640 10 monclient: get_auth_request con 0x7f578c036360 auth_method 0 2024-09-16T14:48:58.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.613+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f57900042a0 con 0x7f57a009df10 2024-09-16T14:48:58.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.613+0000 7f579cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:58.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.613+0000 7f579cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:58.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.777+0000 7f57a6712640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T14:48:58.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.777+0000 7f57a6712640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T14:48:58.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.777+0000 7f57a6712640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:48:58.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:48:58.777+0000 7f57a6712640 1 -- 172.21.15.189:0/2362306898 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "fooo"} v 0) v1 -- 0x7f5780005300 con 0x7f57a009df10 2024-09-16T14:48:58.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.790+0000 7f2674f6f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:48:58.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.790+0000 7f2674f6f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:48:58.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.790+0000 7f2674f6f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:58.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.790+0000 7f2674f6f640 1 -- 172.21.15.26:0/3774420547 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2654005880 con 0x7f267009a550 2024-09-16T14:48:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.790+0000 7f2666ffd640 1 -- 172.21.15.26:0/3774420547 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 4124573515) 0x7f2660012ad0 con 0x7f267009a550 2024-09-16T14:48:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.790+0000 7f2666ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:48:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.790+0000 7f2666ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:58.797 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:48:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.794+0000 7f2674f6f640 1 -- 172.21.15.26:0/3774420547 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2650036490 legacy=0x7f2650038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.794+0000 7f2674f6f640 10 monclient: shutdown 2024-09-16T14:48:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.794+0000 7f2674f6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.794+0000 7f2674f6f640 1 -- 172.21.15.26:0/3774420547 >> v1:172.21.15.26:6790/0 conn(0x7f267009a550 legacy=0x7f26701b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.794+0000 7f2674f6f640 1 -- 172.21.15.26:0/3774420547 shutdown_connections 2024-09-16T14:48:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.794+0000 7f2674f6f640 1 -- 172.21.15.26:0/3774420547 >> 172.21.15.26:0/3774420547 conn(0x7f267009fb50 msgr2=0x7f26701b73d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:58.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.794+0000 7f2674f6f640 1 -- 172.21.15.26:0/3774420547 shutdown_connections 2024-09-16T14:48:58.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.794+0000 7f2674f6f640 1 -- 172.21.15.26:0/3774420547 wait complete. 2024-09-16T14:48:58.814 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:48:58.815 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:48:58.815 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:48:58.815 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:48:58.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 10 monclient: get_monmap_and_config 2024-09-16T14:48:58.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 10 monclient: build_initial_monmap 2024-09-16T14:48:58.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 10 monclient: monmap: 2024-09-16T14:48:58.941 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:58.941 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.941 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:58.941781+0000 2024-09-16T14:48:58.941 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:58.941781+0000 2024-09-16T14:48:58.941 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:58.941 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:58.942 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:58.942 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:58.942 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:58.942 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:58.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 1 Processor -- start 2024-09-16T14:48:58.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 1 -- start start 2024-09-16T14:48:58.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 10 monclient: init 2024-09-16T14:48:58.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:58.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:48:58.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 10 monclient(hunting): picked mon.noname-b con 0x7f6c5c09cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:58.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 10 monclient(hunting): picked mon.noname-a con 0x7f6c5c132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:58.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 10 monclient(hunting): picked mon.noname-c con 0x7f6c5c138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:58.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5c09d260 con 0x7f6c5c132650 2024-09-16T14:48:58.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5c132a80 con 0x7f6c5c138ff0 2024-09-16T14:48:58.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5c09cf60 con 0x7f6c5c09cb80 2024-09-16T14:48:58.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:58.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 10 monclient(hunting): authenticate will time out at 1363.061816s 2024-09-16T14:48:58.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c60da7640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6c5c09cb80 0x7f6c5c0717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47886/0 (socket says 172.21.15.26:47886) 2024-09-16T14:48:58.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c60da7640 1 -- 172.21.15.26:0/3052886070 learned_addr learned my addr 172.21.15.26:0/3052886070 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:58.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5bfff640 1 --1- 172.21.15.26:0/3052886070 >> v1:172.21.15.26:6789/0 conn(0x7f6c5c132650 0x7f6c5c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34022/0 (socket says 172.21.15.26:34022) 2024-09-16T14:48:58.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 1 -- 172.21.15.26:0/3052886070 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464692978 0 0) 0x7f6c5c09cf60 con 0x7f6c5c09cb80 2024-09-16T14:48:58.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient(hunting): my global_id is 154157 2024-09-16T14:48:58.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 1 -- 172.21.15.26:0/3052886070 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c44003530 con 0x7f6c5c09cb80 2024-09-16T14:48:58.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 1 -- 172.21.15.26:0/3052886070 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652831683 0 0) 0x7f6c5c09d260 con 0x7f6c5c132650 2024-09-16T14:48:58.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient(hunting): my global_id is 124584 2024-09-16T14:48:58.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 1 -- 172.21.15.26:0/3052886070 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c5c09cf60 con 0x7f6c5c132650 2024-09-16T14:48:58.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 1 -- 172.21.15.26:0/3052886070 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 269290543 0 0) 0x7f6c44003530 con 0x7f6c5c09cb80 2024-09-16T14:48:58.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 1 -- 172.21.15.26:0/3052886070 >> v1:172.21.15.189:6789/0 conn(0x7f6c5c138ff0 legacy=0x7f6c5c13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 1 -- 172.21.15.26:0/3052886070 >> v1:172.21.15.26:6789/0 conn(0x7f6c5c132650 legacy=0x7f6c5c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:48:58.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 1 monclient: found mon.noname-b 2024-09-16T14:48:58.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:58.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:48:58.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 1 -- 172.21.15.26:0/3052886070 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5c09f2d0 con 0x7f6c5c09cb80 2024-09-16T14:48:58.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:48:58.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:48:58.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:58.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 1 -- 172.21.15.26:0/3052886070 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6c540019e0 con 0x7f6c5c09cb80 2024-09-16T14:48:58.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:58.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:58.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient: dump: 2024-09-16T14:48:58.948 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:58.948 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.948 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.948 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.948 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:58.948 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:58.949 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:58.949 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:58.949 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:58.949 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:58.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 5 monclient: authenticate success, global_id 154157 2024-09-16T14:48:58.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:58.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 1 -- 172.21.15.26:0/3052886070 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c54001bc0 con 0x7f6c5c09cb80 2024-09-16T14:48:58.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:58.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 1 -- 172.21.15.26:0/3052886070 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6c54001f50 con 0x7f6c5c09cb80 2024-09-16T14:48:58.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:58.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:58.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c5b7fe640 10 monclient: dump: 2024-09-16T14:48:58.949 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:58.950 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.950 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.950 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.950 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:58.950 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:58.950 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:58.950 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:58.950 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:58.950 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:58.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:58.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 4 set_mon_vals no callback set 2024-09-16T14:48:58.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 10 monclient: shutdown 2024-09-16T14:48:58.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:58.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 1 -- 172.21.15.26:0/3052886070 >> v1:172.21.15.26:6790/0 conn(0x7f6c5c09cb80 legacy=0x7f6c5c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 1 -- 172.21.15.26:0/3052886070 shutdown_connections 2024-09-16T14:48:58.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 1 -- 172.21.15.26:0/3052886070 >> 172.21.15.26:0/3052886070 conn(0x7f6c5c09fb50 msgr2=0x7f6c5c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:58.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 1 -- 172.21.15.26:0/3052886070 shutdown_connections 2024-09-16T14:48:58.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.938+0000 7f6c6300b640 1 -- 172.21.15.26:0/3052886070 wait complete. 2024-09-16T14:48:58.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 10 monclient: build_initial_monmap 2024-09-16T14:48:58.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 10 monclient: monmap: 2024-09-16T14:48:58.951 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:58.951 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:58.951 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:58.951 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:58.951 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:58.951 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:58.951 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:58.952 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:58.952 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:58.952 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:58.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 1 Processor -- start 2024-09-16T14:48:58.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 1 -- start start 2024-09-16T14:48:58.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 10 monclient: init 2024-09-16T14:48:58.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:58.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:48:58.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 10 monclient(hunting): picked mon.noname-a con 0x7f6c5c09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:58.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 10 monclient(hunting): picked mon.noname-c con 0x7f6c5c132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:58.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 10 monclient(hunting): picked mon.noname-b con 0x7f6c5c138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:58.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5c138b50 con 0x7f6c5c09cb80 2024-09-16T14:48:58.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5c09b370 con 0x7f6c5c138ff0 2024-09-16T14:48:58.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5c0712b0 con 0x7f6c5c132650 2024-09-16T14:48:58.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:58.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 10 monclient(hunting): authenticate will time out at 1363.065110s 2024-09-16T14:48:58.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c5bfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6c5c132650 0x7f6c5c09ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47888/0 (socket says 172.21.15.26:47888) 2024-09-16T14:48:58.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c5bfff640 1 -- 172.21.15.26:0/2653044720 learned_addr learned my addr 172.21.15.26:0/2653044720 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:58.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471211556 0 0) 0x7f6c5c0712b0 con 0x7f6c5c132650 2024-09-16T14:48:58.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): my global_id is 154160 2024-09-16T14:48:58.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c48003530 con 0x7f6c5c132650 2024-09-16T14:48:58.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606649013 0 0) 0x7f6c5c138b50 con 0x7f6c5c09cb80 2024-09-16T14:48:58.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): my global_id is 124587 2024-09-16T14:48:58.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c5c0712b0 con 0x7f6c5c09cb80 2024-09-16T14:48:58.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1870276829 0 0) 0x7f6c48003530 con 0x7f6c5c132650 2024-09-16T14:48:58.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c5c138b50 con 0x7f6c5c132650 2024-09-16T14:48:58.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6c500018e0 con 0x7f6c5c132650 2024-09-16T14:48:58.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:58.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:58.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): dump: 2024-09-16T14:48:58.959 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:58.959 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.959 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.959 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.959 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:58.959 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:58.959 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:58.959 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:58.960 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:58.960 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:58.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:58.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367201049 0 0) 0x7f6c5c09b370 con 0x7f6c5c138ff0 2024-09-16T14:48:58.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): my global_id is 144382 2024-09-16T14:48:58.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c48003530 con 0x7f6c5c138ff0 2024-09-16T14:48:58.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 125490234 0 0) 0x7f6c5c0712b0 con 0x7f6c5c09cb80 2024-09-16T14:48:58.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c5c09b370 con 0x7f6c5c09cb80 2024-09-16T14:48:58.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6c54001d30 con 0x7f6c5c09cb80 2024-09-16T14:48:58.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:58.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:58.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): dump: 2024-09-16T14:48:58.961 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:58.961 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.961 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.961 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.961 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:58.961 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:58.961 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:58.961 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:58.961 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:58.961 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:58.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3163888778 0 0) 0x7f6c5c138b50 con 0x7f6c5c132650 2024-09-16T14:48:58.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 >> v1:172.21.15.189:6789/0 conn(0x7f6c5c138ff0 legacy=0x7f6c5c1d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 >> v1:172.21.15.26:6789/0 conn(0x7f6c5c09cb80 legacy=0x7f6c5c09a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient: _finish_hunting 0 2024-09-16T14:48:58.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 monclient: found mon.c 2024-09-16T14:48:58.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:58.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:58.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5c138610 con 0x7f6c5c132650 2024-09-16T14:48:58.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient: _finish_auth 0 2024-09-16T14:48:58.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient: _check_auth_tickets 2024-09-16T14:48:58.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:58.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 5 monclient: authenticate success, global_id 154160 2024-09-16T14:48:58.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 10 monclient: _renew_subs 2024-09-16T14:48:58.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:58.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 1 -- 172.21.15.26:0/2653044720 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c5c06db20 con 0x7f6c5c132650 2024-09-16T14:48:58.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 10 monclient: _renew_subs 2024-09-16T14:48:58.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:58.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 1 -- 172.21.15.26:0/2653044720 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c5c06e0e0 con 0x7f6c5c132650 2024-09-16T14:48:58.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c500047f0 con 0x7f6c5c132650 2024-09-16T14:48:58.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:58.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6c50004b00 con 0x7f6c5c132650 2024-09-16T14:48:58.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:58.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:58.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c5affd640 4 set_mon_vals no callback set 2024-09-16T14:48:58.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient: dump: 2024-09-16T14:48:58.963 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:58.963 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.963 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.964 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.964 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:58.964 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:58.964 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:58.964 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:58.964 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:58.964 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:58.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:58.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c6300b640 1 -- 172.21.15.26:0/2653044720 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6c3c006590 con 0x7f6c3c002830 2024-09-16T14:48:58.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6c5000f570 con 0x7f6c5c132650 2024-09-16T14:48:58.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(177..177 src has 73..177) v4 ==== 7121+0+0 (unknown 2880899562 0 0) 0x7f6c50043f40 con 0x7f6c5c132650 2024-09-16T14:48:58.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463834201 0 0) 0x7f6c3c006590 con 0x7f6c3c002830 2024-09-16T14:48:58.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient(hunting): my global_id is 124590 2024-09-16T14:48:58.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c5c138b50 con 0x7f6c3c002830 2024-09-16T14:48:58.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 341800141 0 0) 0x7f6c5c138b50 con 0x7f6c3c002830 2024-09-16T14:48:58.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c3c006590 con 0x7f6c3c002830 2024-09-16T14:48:58.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6c54004fc0 con 0x7f6c3c002830 2024-09-16T14:48:58.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:58.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:58.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 10 monclient: dump: 2024-09-16T14:48:58.965 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:58.965 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.965 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.965 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.966 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:58.966 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:58.966 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:58.966 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:58.966 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:58.966 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:58.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c615a8640 10 monclient: get_auth_request con 0x7f6c48036470 auth_method 0 2024-09-16T14:48:58.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1411508494 0 0) 0x7f6c3c006590 con 0x7f6c3c002830 2024-09-16T14:48:58.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.942+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c3c00a0e0 con 0x7f6c3c002830 2024-09-16T14:48:58.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.946+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6c54005260 con 0x7f6c3c002830 2024-09-16T14:48:58.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.946+0000 7f6c597fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:58.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.946+0000 7f6c597fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:58.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.946+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 >> v1:172.21.15.26:6789/0 conn(0x7f6c3c002830 legacy=0x7f6c3c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.946+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 >> v1:172.21.15.26:6789/0 conn(0x7f6c3c002830 legacy=0x7f6c3c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:58.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c6300b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:48:58.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c6300b640 1 -- 172.21.15.26:0/2653044720 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6c3c009da0 con 0x7f6c3c0090c0 2024-09-16T14:48:58.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409008332 0 0) 0x7f6c3c009da0 con 0x7f6c3c0090c0 2024-09-16T14:48:58.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:58.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:58.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:58.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 10 monclient(hunting): my global_id is 124593 2024-09-16T14:48:58.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c3c006590 con 0x7f6c3c0090c0 2024-09-16T14:48:58.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1464917093 0 0) 0x7f6c3c006590 con 0x7f6c3c0090c0 2024-09-16T14:48:58.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c3c009da0 con 0x7f6c3c0090c0 2024-09-16T14:48:58.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6c540057e0 con 0x7f6c3c0090c0 2024-09-16T14:48:58.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:58.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:58.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 10 monclient: dump: 2024-09-16T14:48:58.968 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:58.968 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:58.968 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.968 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:58.968 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:58.968 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:58.968 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:58.968 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:58.968 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:58.968 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:58.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2725384162 0 0) 0x7f6c3c009da0 con 0x7f6c3c0090c0 2024-09-16T14:48:58.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6c3c00c290 con 0x7f6c3c0090c0 2024-09-16T14:48:58.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 3371372713) 0x7f6c54005a60 con 0x7f6c3c0090c0 2024-09-16T14:48:58.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:48:58.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:58.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 >> v1:172.21.15.26:6789/0 conn(0x7f6c3c0090c0 legacy=0x7f6c3c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c597fa640 1 -- 172.21.15.26:0/2653044720 >> v1:172.21.15.26:6789/0 conn(0x7f6c3c0090c0 legacy=0x7f6c3c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:58.969 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:48:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:48:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 60, 2024-09-16T14:48:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:48:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:48:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:48:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:48:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:48:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:48:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:48:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:48:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:48:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:48:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:48:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:48:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:48:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:48:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:48:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:48:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:48:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:48:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:48:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:48:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:48:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:48:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:48:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:48:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:48:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:48:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:48:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:48:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:48:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:48:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:48:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:48:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:48:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:48:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:48:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:48:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:48:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:48:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:48:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:48:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:48:58.980 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:48:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c6300b640 1 -- 172.21.15.26:0/2653044720 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6c48036470 legacy=0x7f6c48038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c6300b640 10 monclient: shutdown 2024-09-16T14:48:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c6300b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c6300b640 1 -- 172.21.15.26:0/2653044720 >> v1:172.21.15.26:6790/0 conn(0x7f6c5c132650 legacy=0x7f6c5c09ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.950+0000 7f6c615a8640 1 -- 172.21.15.26:0/2653044720 reap_dead start 2024-09-16T14:48:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.954+0000 7f6c615a8640 1 -- 172.21.15.26:0/2653044720 reap_dead start 2024-09-16T14:48:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.954+0000 7f6c6300b640 1 -- 172.21.15.26:0/2653044720 shutdown_connections 2024-09-16T14:48:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.954+0000 7f6c6300b640 1 -- 172.21.15.26:0/2653044720 >> 172.21.15.26:0/2653044720 conn(0x7f6c5c09fb50 msgr2=0x7f6c5c130910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.954+0000 7f6c6300b640 1 -- 172.21.15.26:0/2653044720 shutdown_connections 2024-09-16T14:48:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:58.954+0000 7f6c6300b640 1 -- 172.21.15.26:0/2653044720 wait complete. 2024-09-16T14:48:58.983 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:48:58.983 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:48:58.983 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:48:59.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.090+0000 7fb915d92640 10 monclient: get_monmap_and_config 2024-09-16T14:48:59.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.090+0000 7fb915d92640 10 monclient: build_initial_monmap 2024-09-16T14:48:59.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.090+0000 7fb915d92640 10 monclient: monmap: 2024-09-16T14:48:59.096 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:59.096 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.096 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:59.096424+0000 2024-09-16T14:48:59.096 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:59.096424+0000 2024-09-16T14:48:59.096 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:59.096 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.096 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:59.096 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:59.096 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:59.096 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.090+0000 7fb915d92640 1 Processor -- start 2024-09-16T14:48:59.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.090+0000 7fb915d92640 1 -- start start 2024-09-16T14:48:59.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.090+0000 7fb915d92640 10 monclient: init 2024-09-16T14:48:59.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:59.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:59.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient(hunting): picked mon.noname-c con 0x7fb9100a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:59.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient(hunting): picked mon.noname-b con 0x7fb9100a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:59.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient(hunting): picked mon.noname-a con 0x7fb9100a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:59.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9100a3250 con 0x7fb9100a2800 2024-09-16T14:48:59.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb91009f600 con 0x7fb9100a1d10 2024-09-16T14:48:59.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9100a3830 con 0x7fb9100a0000 2024-09-16T14:48:59.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:59.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient(hunting): authenticate will time out at 1363.216568s 2024-09-16T14:48:59.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb9100a2800 0x7fb91013d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34058/0 (socket says 172.21.15.26:34058) 2024-09-16T14:48:59.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90ffff640 1 -- 172.21.15.26:0/3183899781 learned_addr learned my addr 172.21.15.26:0/3183899781 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:59.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 1 -- 172.21.15.26:0/3183899781 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922887250 0 0) 0x7fb9100a3830 con 0x7fb9100a0000 2024-09-16T14:48:59.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient(hunting): my global_id is 154163 2024-09-16T14:48:59.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 1 -- 172.21.15.26:0/3183899781 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f8003530 con 0x7fb9100a0000 2024-09-16T14:48:59.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 1 -- 172.21.15.26:0/3183899781 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237818432 0 0) 0x7fb9100a3250 con 0x7fb9100a2800 2024-09-16T14:48:59.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient(hunting): my global_id is 124596 2024-09-16T14:48:59.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 1 -- 172.21.15.26:0/3183899781 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9100a3830 con 0x7fb9100a2800 2024-09-16T14:48:59.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 1 -- 172.21.15.26:0/3183899781 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 664936236 0 0) 0x7fb9100a3830 con 0x7fb9100a2800 2024-09-16T14:48:59.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 1 -- 172.21.15.26:0/3183899781 >> v1:172.21.15.26:6790/0 conn(0x7fb9100a0000 legacy=0x7fb910139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 1 -- 172.21.15.26:0/3183899781 >> v1:172.21.15.189:6789/0 conn(0x7fb9100a1d10 legacy=0x7fb9100a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:48:59.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 1 monclient: found mon.noname-a 2024-09-16T14:48:59.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:59.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:48:59.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 1 -- 172.21.15.26:0/3183899781 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91009ed60 con 0x7fb9100a2800 2024-09-16T14:48:59.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:48:59.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:48:59.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:59.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 1 -- 172.21.15.26:0/3183899781 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb908001600 con 0x7fb9100a2800 2024-09-16T14:48:59.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:59.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient: dump: 2024-09-16T14:48:59.102 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.102 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.102 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.102 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.103 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.103 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.103 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.103 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.103 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.103 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 5 monclient: authenticate success, global_id 124596 2024-09-16T14:48:59.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:59.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 1 -- 172.21.15.26:0/3183899781 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9080018a0 con 0x7fb9100a2800 2024-09-16T14:48:59.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:59.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 1 -- 172.21.15.26:0/3183899781 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb908001c70 con 0x7fb9100a2800 2024-09-16T14:48:59.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:59.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 4 set_mon_vals no callback set 2024-09-16T14:48:59.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:59.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90e7fc640 10 monclient: dump: 2024-09-16T14:48:59.104 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.104 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.104 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.104 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.104 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.104 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.104 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.104 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.104 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.104 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient: shutdown 2024-09-16T14:48:59.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:59.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 1 -- 172.21.15.26:0/3183899781 >> v1:172.21.15.26:6789/0 conn(0x7fb9100a2800 legacy=0x7fb91013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 1 -- 172.21.15.26:0/3183899781 shutdown_connections 2024-09-16T14:48:59.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 1 -- 172.21.15.26:0/3183899781 >> 172.21.15.26:0/3183899781 conn(0x7fb9100a7f30 msgr2=0x7fb9100a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:59.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 1 -- 172.21.15.26:0/3183899781 shutdown_connections 2024-09-16T14:48:59.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 1 -- 172.21.15.26:0/3183899781 wait complete. 2024-09-16T14:48:59.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient: build_initial_monmap 2024-09-16T14:48:59.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient: monmap: 2024-09-16T14:48:59.105 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:59.105 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:59.105 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:59.105 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:59.105 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:59.105 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:59.106 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:59.106 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:59.106 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:59.106 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 1 Processor -- start 2024-09-16T14:48:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 1 -- start start 2024-09-16T14:48:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient: init 2024-09-16T14:48:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:48:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient(hunting): picked mon.noname-c con 0x7fb9100a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient(hunting): picked mon.noname-b con 0x7fb9100a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient(hunting): picked mon.noname-a con 0x7fb9100a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9100a61a0 con 0x7fb9100a2800 2024-09-16T14:48:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb91009e500 con 0x7fb9100a1d10 2024-09-16T14:48:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb91013dc00 con 0x7fb9100a0000 2024-09-16T14:48:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:59.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb915d92640 10 monclient(hunting): authenticate will time out at 1363.219792s 2024-09-16T14:48:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb9100a2800 0x7fb9101d1990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34070/0 (socket says 172.21.15.26:34070) 2024-09-16T14:48:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90f7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb9100a0000 0x7fb91013d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47904/0 (socket says 172.21.15.26:47904) 2024-09-16T14:48:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.094+0000 7fb90ffff640 1 -- 172.21.15.26:0/3419620253 learned_addr learned my addr 172.21.15.26:0/3419620253 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050608838 0 0) 0x7fb9100a61a0 con 0x7fb9100a2800 2024-09-16T14:48:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): my global_id is 124599 2024-09-16T14:48:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8fc003530 con 0x7fb9100a2800 2024-09-16T14:48:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833490177 0 0) 0x7fb91013dc00 con 0x7fb9100a0000 2024-09-16T14:48:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): my global_id is 154166 2024-09-16T14:48:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9100a61a0 con 0x7fb9100a0000 2024-09-16T14:48:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 82039954 0 0) 0x7fb9100a61a0 con 0x7fb9100a0000 2024-09-16T14:48:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb91013dc00 con 0x7fb9100a0000 2024-09-16T14:48:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9000018a0 con 0x7fb9100a0000 2024-09-16T14:48:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): dump: 2024-09-16T14:48:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.113 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.114 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2378842445 0 0) 0x7fb8fc003530 con 0x7fb9100a2800 2024-09-16T14:48:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9100a61a0 con 0x7fb9100a2800 2024-09-16T14:48:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb908001bb0 con 0x7fb9100a2800 2024-09-16T14:48:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): dump: 2024-09-16T14:48:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.114 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.115 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423534535 0 0) 0x7fb91009e500 con 0x7fb9100a1d10 2024-09-16T14:48:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): my global_id is 144388 2024-09-16T14:48:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8fc003530 con 0x7fb9100a1d10 2024-09-16T14:48:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 324254330 0 0) 0x7fb91013dc00 con 0x7fb9100a0000 2024-09-16T14:48:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 >> v1:172.21.15.189:6789/0 conn(0x7fb9100a1d10 legacy=0x7fb9101ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 >> v1:172.21.15.26:6789/0 conn(0x7fb9100a2800 legacy=0x7fb9101d1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 monclient: found mon.c 2024-09-16T14:48:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9101349b0 con 0x7fb9100a0000 2024-09-16T14:48:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient: _finish_auth 0 2024-09-16T14:48:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb915d92640 5 monclient: authenticate success, global_id 154166 2024-09-16T14:48:59.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb915d92640 10 monclient: _renew_subs 2024-09-16T14:48:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb915d92640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb915d92640 1 -- 172.21.15.26:0/3419620253 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9101366f0 con 0x7fb9100a0000 2024-09-16T14:48:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb915d92640 10 monclient: _renew_subs 2024-09-16T14:48:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb915d92640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb915d92640 1 -- 172.21.15.26:0/3419620253 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9101c9960 con 0x7fb9100a0000 2024-09-16T14:48:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9000036a0 con 0x7fb9100a0000 2024-09-16T14:48:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb900003a30 con 0x7fb9100a0000 2024-09-16T14:48:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb90dffb640 4 set_mon_vals no callback set 2024-09-16T14:48:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient: dump: 2024-09-16T14:48:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.117 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.118 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb915d92640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb915d92640 1 -- 172.21.15.26:0/3419620253 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb8f0006590 con 0x7fb8f0002830 2024-09-16T14:48:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb900003b90 con 0x7fb9100a0000 2024-09-16T14:48:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(177..177 src has 73..177) v4 ==== 7121+0+0 (unknown 2880899562 0 0) 0x7fb900043e40 con 0x7fb9100a0000 2024-09-16T14:48:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274897260 0 0) 0x7fb8f0006590 con 0x7fb8f0002830 2024-09-16T14:48:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient(hunting): my global_id is 154169 2024-09-16T14:48:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb91013dc00 con 0x7fb8f0002830 2024-09-16T14:48:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb90ffff640 10 monclient: get_auth_request con 0x7fb8fc036390 auth_method 0 2024-09-16T14:48:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 341640199 0 0) 0x7fb91013dc00 con 0x7fb8f0002830 2024-09-16T14:48:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8f0006590 con 0x7fb8f0002830 2024-09-16T14:48:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb904001a50 con 0x7fb8f0002830 2024-09-16T14:48:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient: dump: 2024-09-16T14:48:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.119 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.120 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.120 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.120 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.120 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.120 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.120 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.120 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.120 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2112436618 0 0) 0x7fb8f0006590 con 0x7fb8f0002830 2024-09-16T14:48:59.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8f000a0e0 con 0x7fb8f0002830 2024-09-16T14:48:59.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fb904001df0 con 0x7fb8f0002830 2024-09-16T14:48:59.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:59.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:59.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 >> v1:172.21.15.26:6790/0 conn(0x7fb8f0002830 legacy=0x7fb8f0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.098+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 >> v1:172.21.15.26:6790/0 conn(0x7fb8f0002830 legacy=0x7fb8f0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:59.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb915d92640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:48:59.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb915d92640 1 -- 172.21.15.26:0/3419620253 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb8f0009da0 con 0x7fb8f00090c0 2024-09-16T14:48:59.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512791195 0 0) 0x7fb8f0009da0 con 0x7fb8f00090c0 2024-09-16T14:48:59.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 10 monclient(hunting): my global_id is 154172 2024-09-16T14:48:59.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f0006590 con 0x7fb8f00090c0 2024-09-16T14:48:59.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1099046181 0 0) 0x7fb8f0006590 con 0x7fb8f00090c0 2024-09-16T14:48:59.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8f0009da0 con 0x7fb8f00090c0 2024-09-16T14:48:59.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb904003b20 con 0x7fb8f00090c0 2024-09-16T14:48:59.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:59.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 10 monclient: dump: 2024-09-16T14:48:59.122 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.122 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.122 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.122 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.122 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.122 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.122 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.122 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.122 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.122 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3882423430 0 0) 0x7fb8f0009da0 con 0x7fb8f00090c0 2024-09-16T14:48:59.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb8f000c290 con 0x7fb8f00090c0 2024-09-16T14:48:59.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3881 (unknown 0 0 739344684) 0x7fb904003db0 con 0x7fb8f00090c0 2024-09-16T14:48:59.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:48:59.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:59.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 >> v1:172.21.15.26:6790/0 conn(0x7fb8f00090c0 legacy=0x7fb8f0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb8f7fff640 1 -- 172.21.15.26:0/3419620253 >> v1:172.21.15.26:6790/0 conn(0x7fb8f00090c0 legacy=0x7fb8f0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:59.123 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:48:59.123 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:59.123 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:59.123 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:48:59.124 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 60, 2024-09-16T14:48:59.124 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:48:59.124 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:48:59.124 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:48:59.124 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:48:59.124 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:59.124 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:48:59.124 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:59.124 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:48:59.124 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:48:59.124 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:59.124 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:59.124 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:59.124 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:59.125 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:59.125 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:59.125 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:59.125 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:59.125 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:59.125 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:59.125 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:59.125 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:48:59.125 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:48:59.125 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:59.125 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:59.125 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:59.125 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:59.126 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:59.126 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:59.126 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:59.126 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:59.126 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:59.126 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:59.126 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:59.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.126 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:48:59.126 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:48:59.126 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:48:59.126 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:48:59.126 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:48:59.126 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:48:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:48:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:48:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:48:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:48:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:48:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:48:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:48:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:48:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:59.127 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:48:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:48:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:59.128 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:48:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:48:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:48:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:59.129 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:59.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.130 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:59.130 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:59.130 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:59.130 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:59.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.130 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:59.130 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:48:59.130 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:59.130 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:59.130 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:59.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.130 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:48:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:48:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:48:59.131 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:48:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:48:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:48:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:48:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:48:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:48:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.133 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:48:59.134 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:48:59.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb915d92640 1 -- 172.21.15.26:0/3419620253 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb8fc036390 legacy=0x7fb8fc038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb915d92640 10 monclient: shutdown 2024-09-16T14:48:59.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb915d92640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:59.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb915d92640 1 -- 172.21.15.26:0/3419620253 >> v1:172.21.15.26:6790/0 conn(0x7fb9100a0000 legacy=0x7fb91013d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb90ffff640 1 -- 172.21.15.26:0/3419620253 reap_dead start 2024-09-16T14:48:59.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb90ffff640 1 -- 172.21.15.26:0/3419620253 reap_dead start 2024-09-16T14:48:59.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb915d92640 1 -- 172.21.15.26:0/3419620253 shutdown_connections 2024-09-16T14:48:59.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb915d92640 1 -- 172.21.15.26:0/3419620253 >> 172.21.15.26:0/3419620253 conn(0x7fb9100a7f30 msgr2=0x7fb91013df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:59.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb915d92640 1 -- 172.21.15.26:0/3419620253 shutdown_connections 2024-09-16T14:48:59.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.106+0000 7fb915d92640 1 -- 172.21.15.26:0/3419620253 wait complete. 2024-09-16T14:48:59.136 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:48:59.136 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:48:59.136 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:48:59.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.246+0000 7efd2b49c640 10 monclient: get_monmap_and_config 2024-09-16T14:48:59.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.246+0000 7efd2b49c640 10 monclient: build_initial_monmap 2024-09-16T14:48:59.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.246+0000 7efd2b49c640 10 monclient: monmap: 2024-09-16T14:48:59.252 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:59.252 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.252 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:59.253169+0000 2024-09-16T14:48:59.253 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:59.253169+0000 2024-09-16T14:48:59.253 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:59.253 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.253 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:59.253 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:59.253 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:59.253 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 1 Processor -- start 2024-09-16T14:48:59.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 1 -- start start 2024-09-16T14:48:59.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 10 monclient: init 2024-09-16T14:48:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 10 monclient(hunting): picked mon.noname-c con 0x7efd2406c5a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 10 monclient(hunting): picked mon.noname-b con 0x7efd240a23b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 10 monclient(hunting): picked mon.noname-a con 0x7efd240a2840 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd2409f390 con 0x7efd240a2840 2024-09-16T14:48:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd2406c980 con 0x7efd2406c5a0 2024-09-16T14:48:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd2409edd0 con 0x7efd240a23b0 2024-09-16T14:48:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 10 monclient(hunting): authenticate will time out at 1363.373232s 2024-09-16T14:48:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd28a37640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7efd240a23b0 0x7efd24139eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47942/0 (socket says 172.21.15.26:47942) 2024-09-16T14:48:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd28a37640 1 -- 172.21.15.26:0/889409625 learned_addr learned my addr 172.21.15.26:0/889409625 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:59.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 1 -- 172.21.15.26:0/889409625 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351381564 0 0) 0x7efd2409edd0 con 0x7efd240a23b0 2024-09-16T14:48:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient(hunting): my global_id is 154175 2024-09-16T14:48:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 1 -- 172.21.15.26:0/889409625 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd10003530 con 0x7efd240a23b0 2024-09-16T14:48:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 1 -- 172.21.15.26:0/889409625 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212386746 0 0) 0x7efd2409f390 con 0x7efd240a2840 2024-09-16T14:48:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient(hunting): my global_id is 124602 2024-09-16T14:48:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 1 -- 172.21.15.26:0/889409625 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd2409edd0 con 0x7efd240a2840 2024-09-16T14:48:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 1 -- 172.21.15.26:0/889409625 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624083168 0 0) 0x7efd2406c980 con 0x7efd2406c5a0 2024-09-16T14:48:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient(hunting): my global_id is 144391 2024-09-16T14:48:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 1 -- 172.21.15.26:0/889409625 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd2409f390 con 0x7efd2406c5a0 2024-09-16T14:48:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 1 -- 172.21.15.26:0/889409625 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 329482993 0 0) 0x7efd10003530 con 0x7efd240a23b0 2024-09-16T14:48:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 1 -- 172.21.15.26:0/889409625 >> v1:172.21.15.189:6789/0 conn(0x7efd2406c5a0 legacy=0x7efd240a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 1 -- 172.21.15.26:0/889409625 >> v1:172.21.15.26:6789/0 conn(0x7efd240a2840 legacy=0x7efd2413d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:48:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 1 monclient: found mon.noname-b 2024-09-16T14:48:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:48:59.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 1 -- 172.21.15.26:0/889409625 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd2409f0b0 con 0x7efd240a23b0 2024-09-16T14:48:59.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient: _finish_auth 0 2024-09-16T14:48:59.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:48:59.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 1 -- 172.21.15.26:0/889409625 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efd14001620 con 0x7efd240a23b0 2024-09-16T14:48:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 5 monclient: authenticate success, global_id 154175 2024-09-16T14:48:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient: dump: 2024-09-16T14:48:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.262 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.262 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.262 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:59.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 1 -- 172.21.15.26:0/889409625 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd14001840 con 0x7efd240a23b0 2024-09-16T14:48:59.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:59.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 1 -- 172.21.15.26:0/889409625 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efd14001bd0 con 0x7efd240a23b0 2024-09-16T14:48:59.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:59.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 4 set_mon_vals no callback set 2024-09-16T14:48:59.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:59.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd1bfff640 10 monclient: dump: 2024-09-16T14:48:59.262 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.262 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.263 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.263 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.263 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.263 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.263 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.263 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.263 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.263 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 10 monclient: shutdown 2024-09-16T14:48:59.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:59.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 1 -- 172.21.15.26:0/889409625 >> v1:172.21.15.26:6790/0 conn(0x7efd240a23b0 legacy=0x7efd24139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 1 -- 172.21.15.26:0/889409625 shutdown_connections 2024-09-16T14:48:59.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 1 -- 172.21.15.26:0/889409625 >> 172.21.15.26:0/889409625 conn(0x7efd240a3da0 msgr2=0x7efd240a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:59.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 1 -- 172.21.15.26:0/889409625 shutdown_connections 2024-09-16T14:48:59.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 1 -- 172.21.15.26:0/889409625 wait complete. 2024-09-16T14:48:59.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 10 monclient: build_initial_monmap 2024-09-16T14:48:59.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 10 monclient: monmap: 2024-09-16T14:48:59.264 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:59.264 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:59.264 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:59.264 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:59.264 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:59.264 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:59.264 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:59.264 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:59.264 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:59.264 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.250+0000 7efd2b49c640 1 Processor -- start 2024-09-16T14:48:59.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 1 -- start start 2024-09-16T14:48:59.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 10 monclient: init 2024-09-16T14:48:59.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:59.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:48:59.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 10 monclient(hunting): picked mon.noname-a con 0x7efd2406c5a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:59.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 10 monclient(hunting): picked mon.noname-b con 0x7efd240a23b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:59.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 10 monclient(hunting): picked mon.noname-c con 0x7efd240a2840 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:59.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd2409e490 con 0x7efd2406c5a0 2024-09-16T14:48:59.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd240a68c0 con 0x7efd240a23b0 2024-09-16T14:48:59.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd24122750 con 0x7efd240a2840 2024-09-16T14:48:59.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:59.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 10 monclient(hunting): authenticate will time out at 1363.376526s 2024-09-16T14:48:59.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd29238640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7efd2406c5a0 0x7efd24134d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34088/0 (socket says 172.21.15.26:34088) 2024-09-16T14:48:59.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd29238640 1 -- 172.21.15.26:0/1611209821 learned_addr learned my addr 172.21.15.26:0/1611209821 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:59.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755116417 0 0) 0x7efd2409e490 con 0x7efd2406c5a0 2024-09-16T14:48:59.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): my global_id is 124605 2024-09-16T14:48:59.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd08003530 con 0x7efd2406c5a0 2024-09-16T14:48:59.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306232843 0 0) 0x7efd24122750 con 0x7efd240a2840 2024-09-16T14:48:59.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): my global_id is 154178 2024-09-16T14:48:59.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd2409e490 con 0x7efd240a2840 2024-09-16T14:48:59.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2851790695 0 0) 0x7efd08003530 con 0x7efd2406c5a0 2024-09-16T14:48:59.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd24122750 con 0x7efd2406c5a0 2024-09-16T14:48:59.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efd20001900 con 0x7efd2406c5a0 2024-09-16T14:48:59.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:48:59.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): dump: 2024-09-16T14:48:59.271 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.272 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.272 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.272 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.272 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.272 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.272 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.272 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.272 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.272 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:59.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4217272965 0 0) 0x7efd2409e490 con 0x7efd240a2840 2024-09-16T14:48:59.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd08003530 con 0x7efd240a2840 2024-09-16T14:48:59.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efd1c001bc0 con 0x7efd240a2840 2024-09-16T14:48:59.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:59.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): dump: 2024-09-16T14:48:59.273 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.273 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.273 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.273 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.273 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.273 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.274 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.274 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.274 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.274 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007945336 0 0) 0x7efd240a68c0 con 0x7efd240a23b0 2024-09-16T14:48:59.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): my global_id is 144394 2024-09-16T14:48:59.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd2409e490 con 0x7efd240a23b0 2024-09-16T14:48:59.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2178577898 0 0) 0x7efd24122750 con 0x7efd2406c5a0 2024-09-16T14:48:59.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 >> v1:172.21.15.26:6790/0 conn(0x7efd240a2840 legacy=0x7efd241d1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 >> v1:172.21.15.189:6789/0 conn(0x7efd240a23b0 legacy=0x7efd241ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:48:59.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 monclient: found mon.a 2024-09-16T14:48:59.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:59.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:59.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd24120d40 con 0x7efd2406c5a0 2024-09-16T14:48:59.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient: _finish_auth 0 2024-09-16T14:48:59.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:48:59.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:59.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 5 monclient: authenticate success, global_id 124605 2024-09-16T14:48:59.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 10 monclient: _renew_subs 2024-09-16T14:48:59.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:59.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 1 -- 172.21.15.26:0/1611209821 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd24121eb0 con 0x7efd2406c5a0 2024-09-16T14:48:59.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 10 monclient: _renew_subs 2024-09-16T14:48:59.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:48:59.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 1 -- 172.21.15.26:0/1611209821 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd241354a0 con 0x7efd2406c5a0 2024-09-16T14:48:59.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd20003620 con 0x7efd2406c5a0 2024-09-16T14:48:59.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:59.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efd200039b0 con 0x7efd2406c5a0 2024-09-16T14:48:59.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:48:59.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd1b7fe640 4 set_mon_vals no callback set 2024-09-16T14:48:59.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient: dump: 2024-09-16T14:48:59.276 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.276 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.276 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.276 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.277 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.277 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.277 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.277 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.277 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.277 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:59.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd2b49c640 1 -- 172.21.15.26:0/1611209821 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efd04006590 con 0x7efd04002830 2024-09-16T14:48:59.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7efd20003b10 con 0x7efd2406c5a0 2024-09-16T14:48:59.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(177..177 src has 73..177) v4 ==== 7121+0+0 (unknown 2880899562 0 0) 0x7efd20043e80 con 0x7efd2406c5a0 2024-09-16T14:48:59.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd29a39640 10 monclient: get_auth_request con 0x7efd08036420 auth_method 0 2024-09-16T14:48:59.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415243960 0 0) 0x7efd04006590 con 0x7efd04002830 2024-09-16T14:48:59.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient(hunting): my global_id is 144397 2024-09-16T14:48:59.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd24122750 con 0x7efd04002830 2024-09-16T14:48:59.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1761535969 0 0) 0x7efd24122750 con 0x7efd04002830 2024-09-16T14:48:59.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd04006590 con 0x7efd04002830 2024-09-16T14:48:59.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efd14001130 con 0x7efd04002830 2024-09-16T14:48:59.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:59.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 10 monclient: dump: 2024-09-16T14:48:59.278 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.278 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.278 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.279 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.279 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.279 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.279 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.279 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.279 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.279 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3083363358 0 0) 0x7efd04006590 con 0x7efd04002830 2024-09-16T14:48:59.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.254+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd0400a0e0 con 0x7efd04002830 2024-09-16T14:48:59.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.258+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7efd140013c0 con 0x7efd04002830 2024-09-16T14:48:59.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.258+0000 7efd19ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:59.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.258+0000 7efd19ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:59.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.258+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 >> v1:172.21.15.189:6789/0 conn(0x7efd04002830 legacy=0x7efd04004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.258+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 >> v1:172.21.15.189:6789/0 conn(0x7efd04002830 legacy=0x7efd04004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:59.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.262+0000 7efd2b49c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:48:59.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.262+0000 7efd2b49c640 1 -- 172.21.15.26:0/1611209821 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efd04009da0 con 0x7efd040090c0 2024-09-16T14:48:59.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.262+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124635771 0 0) 0x7efd04009da0 con 0x7efd040090c0 2024-09-16T14:48:59.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.262+0000 7efd19ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.262+0000 7efd19ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.262+0000 7efd19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.262+0000 7efd19ffb640 10 monclient(hunting): my global_id is 144400 2024-09-16T14:48:59.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.262+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd04006590 con 0x7efd040090c0 2024-09-16T14:48:59.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1783420072 0 0) 0x7efd04006590 con 0x7efd040090c0 2024-09-16T14:48:59.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd04009da0 con 0x7efd040090c0 2024-09-16T14:48:59.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efd14004a10 con 0x7efd040090c0 2024-09-16T14:48:59.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd19ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:48:59.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd19ffb640 10 monclient: dump: 2024-09-16T14:48:59.281 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.281 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.281 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.281 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.282 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.282 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.282 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.282 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.282 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.282 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2550633652 0 0) 0x7efd04009da0 con 0x7efd040090c0 2024-09-16T14:48:59.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efd0400c290 con 0x7efd040090c0 2024-09-16T14:48:59.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 1334084836) 0x7efd14005fe0 con 0x7efd040090c0 2024-09-16T14:48:59.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd19ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:48:59.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd19ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:59.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 >> v1:172.21.15.189:6789/0 conn(0x7efd040090c0 legacy=0x7efd04009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd19ffb640 1 -- 172.21.15.26:0/1611209821 >> v1:172.21.15.189:6789/0 conn(0x7efd040090c0 legacy=0x7efd04009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:48:59.283 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:48:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:48:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 60, 2024-09-16T14:48:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:48:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:48:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:48:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:48:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:48:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:48:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:48:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:48:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:48:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.287 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:48:59.287 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:48:59.287 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:48:59.287 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:48:59.287 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:48:59.287 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:48:59.287 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:59.287 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:48:59.287 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:48:59.287 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:48:59.287 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:48:59.287 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:48:59.288 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:48:59.288 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:48:59.288 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:48:59.288 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:48:59.288 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:48:59.288 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:48:59.288 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:48:59.288 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:48:59.289 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:48:59.289 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:48:59.289 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:48:59.289 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:48:59.289 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:48:59.289 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:48:59.289 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:48:59.289 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:59.289 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:48:59.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.289 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:48:59.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.290 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:48:59.290 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:48:59.290 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:59.290 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:59.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.290 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:59.290 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:48:59.290 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:59.290 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:59.291 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:59.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.291 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:48:59.291 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:48:59.291 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:59.291 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:59.291 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:59.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.292 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:48:59.292 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:48:59.292 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:59.292 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:59.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.292 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:59.292 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:48:59.292 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:59.292 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:59.292 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:59.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.292 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:48:59.293 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:48:59.293 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:59.293 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:59.293 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:59.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.293 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:48:59.293 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:48:59.293 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:48:59.294 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:48:59.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.294 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:48:59.294 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:48:59.294 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:48:59.294 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:59.294 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:59.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.295 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:48:59.295 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:48:59.295 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:48:59.295 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:48:59.295 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:48:59.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:59.295 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:59.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.295 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:48:59.295 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:48:59.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.295 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:59.295 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:59.295 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:59.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:59.296 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:59.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:48:59.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.296 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:59.296 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:59.296 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:48:59.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:59.296 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:48:59.296 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:48:59.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:48:59.296 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:48:59.296 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:48:59.297 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:48:59.297 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:48:59.297 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:48:59.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:48:59.297 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:48:59.297 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:48:59.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd2b49c640 1 -- 172.21.15.26:0/1611209821 >> v1:172.21.15.189:6812/3196161582 conn(0x7efd08036420 legacy=0x7efd080388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd2b49c640 10 monclient: shutdown 2024-09-16T14:48:59.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd2b49c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:59.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd2b49c640 1 -- 172.21.15.26:0/1611209821 >> v1:172.21.15.26:6789/0 conn(0x7efd2406c5a0 legacy=0x7efd24134d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd29a39640 1 -- 172.21.15.26:0/1611209821 reap_dead start 2024-09-16T14:48:59.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd29a39640 1 -- 172.21.15.26:0/1611209821 reap_dead start 2024-09-16T14:48:59.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd2b49c640 1 -- 172.21.15.26:0/1611209821 shutdown_connections 2024-09-16T14:48:59.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd2b49c640 1 -- 172.21.15.26:0/1611209821 >> 172.21.15.26:0/1611209821 conn(0x7efd240a3da0 msgr2=0x7efd240a50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:59.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd2b49c640 1 -- 172.21.15.26:0/1611209821 shutdown_connections 2024-09-16T14:48:59.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.266+0000 7efd2b49c640 1 -- 172.21.15.26:0/1611209821 wait complete. 2024-09-16T14:48:59.299 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:48:59.299 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T14:48:59.299 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:48:59.299 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T14:48:59.299 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:48:59.300 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T14:48:59.300 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:48:59.312 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T14:48:59.313 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:48:59.313 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:48:59.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.474+0000 7f6e5f0de640 10 monclient: get_monmap_and_config 2024-09-16T14:48:59.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.474+0000 7f6e5f0de640 10 monclient: build_initial_monmap 2024-09-16T14:48:59.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.474+0000 7f6e5f0de640 10 monclient: monmap: 2024-09-16T14:48:59.479 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:59.480 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.480 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:48:59.479684+0000 2024-09-16T14:48:59.480 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:48:59.479684+0000 2024-09-16T14:48:59.480 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:59.480 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.481 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:59.481 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:48:59.481 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:48:59.481 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.474+0000 7f6e5f0de640 1 Processor -- start 2024-09-16T14:48:59.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.474+0000 7f6e5f0de640 1 -- start start 2024-09-16T14:48:59.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.474+0000 7f6e5f0de640 10 monclient: init 2024-09-16T14:48:59.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:59.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:59.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient(hunting): picked mon.noname-c con 0x7f6e5809cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:59.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient(hunting): picked mon.noname-b con 0x7f6e5809e7a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:59.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient(hunting): picked mon.noname-a con 0x7f6e5809d1a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:59.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5809d5b0 con 0x7f6e5809d1a0 2024-09-16T14:48:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e58131a90 con 0x7f6e5809cdc0 2024-09-16T14:48:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5809aff0 con 0x7f6e5809e7a0 2024-09-16T14:48:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient(hunting): authenticate will time out at 1363.600263s 2024-09-16T14:48:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5d67b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f6e5809d1a0 legacy=0x7f6e58139fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:48:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e57fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6e5809e7a0 0x7f6e58131380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47972/0 (socket says 172.21.15.26:47972) 2024-09-16T14:48:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e57fff640 1 -- 172.21.15.26:0/637749311 learned_addr learned my addr 172.21.15.26:0/637749311 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 1 -- 172.21.15.26:0/637749311 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994321541 0 0) 0x7f6e5809aff0 con 0x7f6e5809e7a0 2024-09-16T14:48:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 10 monclient(hunting): my global_id is 154181 2024-09-16T14:48:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 1 -- 172.21.15.26:0/637749311 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e40003450 con 0x7f6e5809e7a0 2024-09-16T14:48:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 1 -- 172.21.15.26:0/637749311 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1470414774 0 0) 0x7f6e40003450 con 0x7f6e5809e7a0 2024-09-16T14:48:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 1 -- 172.21.15.26:0/637749311 >> v1:172.21.15.189:6789/0 conn(0x7f6e5809cdc0 legacy=0x7f6e58130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 1 -- 172.21.15.26:0/637749311 >> v1:172.21.15.26:6789/0 conn(0x7f6e5809d1a0 legacy=0x7f6e58139fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:48:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 10 monclient: _finish_hunting 0 2024-09-16T14:48:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 1 monclient: found mon.noname-b 2024-09-16T14:48:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:48:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 1 -- 172.21.15.26:0/637749311 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5809b2d0 con 0x7f6e5809e7a0 2024-09-16T14:48:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 10 monclient: _finish_auth 0 2024-09-16T14:48:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 10 monclient: _check_auth_tickets 2024-09-16T14:48:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 5 monclient: authenticate success, global_id 154181 2024-09-16T14:48:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 1 -- 172.21.15.26:0/637749311 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e4c001620 con 0x7f6e5809e7a0 2024-09-16T14:48:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:48:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 10 monclient: dump: 2024-09-16T14:48:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.503 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:48:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 1 -- 172.21.15.26:0/637749311 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e4c0018d0 con 0x7f6e5809e7a0 2024-09-16T14:48:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 1 -- 172.21.15.26:0/637749311 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e4c001c60 con 0x7f6e5809e7a0 2024-09-16T14:48:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e577fe640 10 monclient: dump: 2024-09-16T14:48:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.507 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient: get_monmap_and_config success 2024-09-16T14:48:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 4 set_mon_vals no callback set 2024-09-16T14:48:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient: shutdown 2024-09-16T14:48:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 1 -- 172.21.15.26:0/637749311 >> v1:172.21.15.26:6790/0 conn(0x7f6e5809e7a0 legacy=0x7f6e58131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 1 -- 172.21.15.26:0/637749311 shutdown_connections 2024-09-16T14:48:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 1 -- 172.21.15.26:0/637749311 >> 172.21.15.26:0/637749311 conn(0x7f6e5809fb50 msgr2=0x7f6e5809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 1 -- 172.21.15.26:0/637749311 shutdown_connections 2024-09-16T14:48:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 1 -- 172.21.15.26:0/637749311 wait complete. 2024-09-16T14:48:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient: build_initial_monmap 2024-09-16T14:48:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient: monmap: 2024-09-16T14:48:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:48:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:48:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:48:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:48:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:48:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:48:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:48:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:48:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:48:59.511 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 1 Processor -- start 2024-09-16T14:48:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 1 -- start start 2024-09-16T14:48:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient: init 2024-09-16T14:48:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient: _reopen_session rank -1 2024-09-16T14:48:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:48:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient(hunting): picked mon.noname-b con 0x7f6e5809cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:48:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient(hunting): picked mon.noname-c con 0x7f6e5809d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:48:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient(hunting): picked mon.noname-a con 0x7f6e5809e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:48:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5811a940 con 0x7f6e5809e7a0 2024-09-16T14:48:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5811b320 con 0x7f6e5809cdc0 2024-09-16T14:48:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5811c3e0 con 0x7f6e5809d1a0 2024-09-16T14:48:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient(hunting): _renew_subs 2024-09-16T14:48:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5f0de640 10 monclient(hunting): authenticate will time out at 1363.603297s 2024-09-16T14:48:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e5d67b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f6e5809e7a0 legacy=0x7f6e581cd640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:48:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e57fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6e5809d1a0 0x7f6e58119db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47986/0 (socket says 172.21.15.26:47986) 2024-09-16T14:48:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e57fff640 1 -- 172.21.15.26:0/180782728 learned_addr learned my addr 172.21.15.26:0/180782728 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:48:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e557fa640 1 -- 172.21.15.26:0/180782728 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449362551 0 0) 0x7f6e5811c3e0 con 0x7f6e5809d1a0 2024-09-16T14:48:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e557fa640 10 monclient(hunting): _negotiate 2024-09-16T14:48:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e557fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:48:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e557fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:48:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e557fa640 10 monclient(hunting): my global_id is 154184 2024-09-16T14:48:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e557fa640 1 -- 172.21.15.26:0/180782728 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e44003530 con 0x7f6e5809d1a0 2024-09-16T14:48:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e557fa640 1 -- 172.21.15.26:0/180782728 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3101464073 0 0) 0x7f6e44003530 con 0x7f6e5809d1a0 2024-09-16T14:48:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e557fa640 1 -- 172.21.15.26:0/180782728 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e5811c3e0 con 0x7f6e5809d1a0 2024-09-16T14:48:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e557fa640 1 -- 172.21.15.26:0/180782728 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e4c001be0 con 0x7f6e5809d1a0 2024-09-16T14:48:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e557fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:48:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e557fa640 10 monclient(hunting): dump: 2024-09-16T14:48:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.518 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.478+0000 7f6e557fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:48:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 1 -- 172.21.15.26:0/180782728 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3382528664 0 0) 0x7f6e5811c3e0 con 0x7f6e5809d1a0 2024-09-16T14:48:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 1 -- 172.21.15.26:0/180782728 >> v1:172.21.15.189:6789/0 conn(0x7f6e5809cdc0 legacy=0x7f6e581196a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 1 -- 172.21.15.26:0/180782728 >> v1:172.21.15.26:6789/0 conn(0x7f6e5809e7a0 legacy=0x7f6e581cd640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:48:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 10 monclient: _finish_hunting 0 2024-09-16T14:48:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 1 monclient: found mon.c 2024-09-16T14:48:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:48:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 1 -- 172.21.15.26:0/180782728 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5811b710 con 0x7f6e5809d1a0 2024-09-16T14:48:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 10 monclient: _finish_auth 0 2024-09-16T14:48:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 10 monclient: _check_auth_tickets 2024-09-16T14:48:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:48:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e5f0de640 5 monclient: authenticate success, global_id 154184 2024-09-16T14:48:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e5f0de640 10 monclient: _renew_subs 2024-09-16T14:48:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e5f0de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e5f0de640 1 -- 172.21.15.26:0/180782728 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5811cc80 con 0x7f6e5809d1a0 2024-09-16T14:48:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e5f0de640 10 monclient: _renew_subs 2024-09-16T14:48:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e5f0de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e5f0de640 1 -- 172.21.15.26:0/180782728 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e580a1fe0 con 0x7f6e5809d1a0 2024-09-16T14:48:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 1 -- 172.21.15.26:0/180782728 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e4c0011d0 con 0x7f6e5809d1a0 2024-09-16T14:48:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:48:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 1 -- 172.21.15.26:0/180782728 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e4c004da0 con 0x7f6e5809d1a0 2024-09-16T14:48:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:48:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e56ffd640 4 set_mon_vals no callback set 2024-09-16T14:48:59.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:48:59.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 10 monclient: dump: 2024-09-16T14:48:59.523 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:48:59.523 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:48:59.523 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.523 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:48:59.524 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:48:59.524 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:48:59.524 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:48:59.524 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:48:59.524 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:48:59.525 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:48:59.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e5f0de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:59.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e5f0de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:59.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e5f0de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:59.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 1 -- 172.21.15.26:0/180782728 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6e4c00f810 con 0x7f6e5809d1a0 2024-09-16T14:48:59.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e5f0de640 1 -- 172.21.15.26:0/180782728 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e38005050 con 0x7f6e5809d1a0 2024-09-16T14:48:59.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e557fa640 1 -- 172.21.15.26:0/180782728 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(177..177 src has 73..177) v4 ==== 7121+0+0 (unknown 2880899562 0 0) 0x7f6e4c0440e0 con 0x7f6e5809d1a0 2024-09-16T14:48:59.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.482+0000 7f6e5ce7a640 10 monclient: get_auth_request con 0x7f6e440360a0 auth_method 0 2024-09-16T14:48:59.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.486+0000 7f6e557fa640 1 -- 172.21.15.26:0/180782728 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6e4c014730 con 0x7f6e5809d1a0 2024-09-16T14:48:59.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.486+0000 7f6e557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:48:59.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.486+0000 7f6e557fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:48:59.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.710+0000 7f6e5f0de640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:48:59.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.710+0000 7f6e5f0de640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:48:59.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.710+0000 7f6e5f0de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:48:59.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.710+0000 7f6e5f0de640 1 -- 172.21.15.26:0/180782728 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6e38005880 con 0x7f6e5809d1a0 2024-09-16T14:48:59.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.710+0000 7f6e557fa640 1 -- 172.21.15.26:0/180782728 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1294377076) 0x7f6e4c012dd0 con 0x7f6e5809d1a0 2024-09-16T14:48:59.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.710+0000 7f6e557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:48:59.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.710+0000 7f6e557fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:48:59.714 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:48:59.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.710+0000 7f6e5f0de640 1 -- 172.21.15.26:0/180782728 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6e440360a0 legacy=0x7f6e44038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.710+0000 7f6e5f0de640 10 monclient: shutdown 2024-09-16T14:48:59.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.710+0000 7f6e5f0de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:48:59.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.710+0000 7f6e5f0de640 1 -- 172.21.15.26:0/180782728 >> v1:172.21.15.26:6790/0 conn(0x7f6e5809d1a0 legacy=0x7f6e58119db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:48:59.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.710+0000 7f6e5f0de640 1 -- 172.21.15.26:0/180782728 shutdown_connections 2024-09-16T14:48:59.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.710+0000 7f6e5f0de640 1 -- 172.21.15.26:0/180782728 >> 172.21.15.26:0/180782728 conn(0x7f6e5809fb50 msgr2=0x7f6e58130fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:48:59.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.710+0000 7f6e5f0de640 1 -- 172.21.15.26:0/180782728 shutdown_connections 2024-09-16T14:48:59.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:48:59.714+0000 7f6e5f0de640 1 -- 172.21.15.26:0/180782728 wait complete. 2024-09-16T14:49:01.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:01.609+0000 7f57877fe640 10 monclient: tick 2024-09-16T14:49:01.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:01.609+0000 7f57877fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:01.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:01.609+0000 7f57877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:01.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:01.609+0000 7f57877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:02.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:02.609+0000 7f57877fe640 10 monclient: tick 2024-09-16T14:49:02.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:02.609+0000 7f57877fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:02.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:02.609+0000 7f57877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:02.735 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:49:02.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda70e5d640 10 monclient: get_monmap_and_config 2024-09-16T14:49:02.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda70e5d640 10 monclient: build_initial_monmap 2024-09-16T14:49:02.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda70e5d640 10 monclient: monmap: 2024-09-16T14:49:02.867 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:02.867 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:02.867 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:02.867739+0000 2024-09-16T14:49:02.867 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:02.867739+0000 2024-09-16T14:49:02.867 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:02.868 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:02.868 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:02.868 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:02.868 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:02.868 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:02.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda70e5d640 1 Processor -- start 2024-09-16T14:49:02.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda70e5d640 1 -- start start 2024-09-16T14:49:02.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda70e5d640 10 monclient: init 2024-09-16T14:49:02.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda70e5d640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:02.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda70e5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:02.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda70e5d640 10 monclient(hunting): picked mon.noname-b con 0x7fda6c0a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:02.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda70e5d640 10 monclient(hunting): picked mon.noname-a con 0x7fda6c0a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:02.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda70e5d640 10 monclient(hunting): picked mon.noname-c con 0x7fda6c0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:02.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda70e5d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6c0a3250 con 0x7fda6c0a0000 2024-09-16T14:49:02.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda70e5d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6c09f600 con 0x7fda6c0a2800 2024-09-16T14:49:02.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda70e5d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6c0a3830 con 0x7fda6c0a1d10 2024-09-16T14:49:02.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda70e5d640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:02.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda70e5d640 10 monclient(hunting): authenticate will time out at 1366.987876s 2024-09-16T14:49:02.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.862+0000 7fda69d9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fda6c0a0000 legacy=0x7fda6c139d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:02.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fda6c0a1d10 0x7fda6c0a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48002/0 (socket says 172.21.15.26:48002) 2024-09-16T14:49:02.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6a59c640 1 -- 172.21.15.26:0/1264654409 learned_addr learned my addr 172.21.15.26:0/1264654409 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:02.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 1 -- 172.21.15.26:0/1264654409 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848509571 0 0) 0x7fda6c0a3830 con 0x7fda6c0a1d10 2024-09-16T14:49:02.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 10 monclient(hunting): _negotiate 2024-09-16T14:49:02.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:02.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:02.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 10 monclient(hunting): my global_id is 154187 2024-09-16T14:49:02.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 1 -- 172.21.15.26:0/1264654409 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda54003530 con 0x7fda6c0a1d10 2024-09-16T14:49:02.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 1 -- 172.21.15.26:0/1264654409 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3253386458 0 0) 0x7fda54003530 con 0x7fda6c0a1d10 2024-09-16T14:49:02.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 1 -- 172.21.15.26:0/1264654409 >> v1:172.21.15.189:6789/0 conn(0x7fda6c0a2800 legacy=0x7fda6c13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:02.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 1 -- 172.21.15.26:0/1264654409 >> v1:172.21.15.26:6789/0 conn(0x7fda6c0a0000 legacy=0x7fda6c139d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:02.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 10 monclient: _finish_hunting 0 2024-09-16T14:49:02.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 1 monclient: found mon.noname-b 2024-09-16T14:49:02.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:02.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:49:02.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 1 -- 172.21.15.26:0/1264654409 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c09ed60 con 0x7fda6c0a1d10 2024-09-16T14:49:02.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 10 monclient: _finish_auth 0 2024-09-16T14:49:02.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 10 monclient: _check_auth_tickets 2024-09-16T14:49:02.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:02.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 1 -- 172.21.15.26:0/1264654409 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fda5c001620 con 0x7fda6c0a1d10 2024-09-16T14:49:02.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 5 monclient: authenticate success, global_id 154187 2024-09-16T14:49:02.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:02.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:02.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:02.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 10 monclient: dump: 2024-09-16T14:49:02.878 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:02.879 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:02.879 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:02.879 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:02.879 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:02.879 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:02.879 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:02.880 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:02.880 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:02.880 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:02.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:02.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 1 -- 172.21.15.26:0/1264654409 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda5c001940 con 0x7fda6c0a1d10 2024-09-16T14:49:02.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:02.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:02.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 4 set_mon_vals no callback set 2024-09-16T14:49:02.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 1 -- 172.21.15.26:0/1264654409 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fda5c001d10 con 0x7fda6c0a1d10 2024-09-16T14:49:02.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:02.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:02.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6959a640 10 monclient: dump: 2024-09-16T14:49:02.880 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:02.881 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:02.881 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:02.881 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:02.881 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:02.881 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:02.881 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:02.881 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:02.881 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:02.882 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:02.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 10 monclient: shutdown 2024-09-16T14:49:02.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:02.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 1 -- 172.21.15.26:0/1264654409 >> v1:172.21.15.26:6790/0 conn(0x7fda6c0a1d10 legacy=0x7fda6c0a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:02.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 1 -- 172.21.15.26:0/1264654409 shutdown_connections 2024-09-16T14:49:02.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 1 -- 172.21.15.26:0/1264654409 >> 172.21.15.26:0/1264654409 conn(0x7fda6c0a7f30 msgr2=0x7fda6c0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:02.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 1 -- 172.21.15.26:0/1264654409 shutdown_connections 2024-09-16T14:49:02.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 1 -- 172.21.15.26:0/1264654409 wait complete. 2024-09-16T14:49:02.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 10 monclient: build_initial_monmap 2024-09-16T14:49:02.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 10 monclient: monmap: 2024-09-16T14:49:02.883 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:02.883 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:02.883 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:02.883 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:02.883 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:02.883 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:02.883 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:02.883 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:02.884 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:02.884 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:02.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 1 Processor -- start 2024-09-16T14:49:02.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 1 -- start start 2024-09-16T14:49:02.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 10 monclient: init 2024-09-16T14:49:02.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:02.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:02.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 10 monclient(hunting): picked mon.noname-b con 0x7fda6c0a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:02.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 10 monclient(hunting): picked mon.noname-a con 0x7fda6c0a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:02.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 10 monclient(hunting): picked mon.noname-c con 0x7fda6c0a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:02.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6c136ed0 con 0x7fda6c0a1d10 2024-09-16T14:49:02.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6c0a68f0 con 0x7fda6c0a0000 2024-09-16T14:49:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6c133ee0 con 0x7fda6c0a2800 2024-09-16T14:49:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 10 monclient(hunting): authenticate will time out at 1366.991048s 2024-09-16T14:49:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda69d9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fda6c0a1d10 legacy=0x7fda6c1337d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6ad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fda6c0a2800 0x7fda6c1d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48012/0 (socket says 172.21.15.26:48012) 2024-09-16T14:49:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda6ad9d640 1 -- 172.21.15.26:0/3959389694 learned_addr learned my addr 172.21.15.26:0/3959389694 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 1 -- 172.21.15.26:0/3959389694 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675133873 0 0) 0x7fda6c133ee0 con 0x7fda6c0a2800 2024-09-16T14:49:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 10 monclient(hunting): my global_id is 154190 2024-09-16T14:49:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 1 -- 172.21.15.26:0/3959389694 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda4c003530 con 0x7fda6c0a2800 2024-09-16T14:49:02.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 1 -- 172.21.15.26:0/3959389694 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3805652666 0 0) 0x7fda4c003530 con 0x7fda6c0a2800 2024-09-16T14:49:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 1 -- 172.21.15.26:0/3959389694 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda6c133ee0 con 0x7fda6c0a2800 2024-09-16T14:49:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 1 -- 172.21.15.26:0/3959389694 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fda64001c80 con 0x7fda6c0a2800 2024-09-16T14:49:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 10 monclient(hunting): dump: 2024-09-16T14:49:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:02.886 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:02.887 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 1 -- 172.21.15.26:0/3959389694 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 645517808 0 0) 0x7fda6c133ee0 con 0x7fda6c0a2800 2024-09-16T14:49:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 1 -- 172.21.15.26:0/3959389694 >> v1:172.21.15.189:6789/0 conn(0x7fda6c0a0000 legacy=0x7fda6c1330c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 1 -- 172.21.15.26:0/3959389694 >> v1:172.21.15.26:6789/0 conn(0x7fda6c0a1d10 legacy=0x7fda6c1337d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 1 monclient: found mon.c 2024-09-16T14:49:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 1 -- 172.21.15.26:0/3959389694 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c0a63c0 con 0x7fda6c0a2800 2024-09-16T14:49:02.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 10 monclient: _finish_auth 0 2024-09-16T14:49:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 5 monclient: authenticate success, global_id 154190 2024-09-16T14:49:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 10 monclient: _renew_subs 2024-09-16T14:49:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.866+0000 7fda70e5d640 1 -- 172.21.15.26:0/3959389694 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6c09e500 con 0x7fda6c0a2800 2024-09-16T14:49:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda70e5d640 10 monclient: _renew_subs 2024-09-16T14:49:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda70e5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda70e5d640 1 -- 172.21.15.26:0/3959389694 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6c135cf0 con 0x7fda6c0a2800 2024-09-16T14:49:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda5affd640 1 -- 172.21.15.26:0/3959389694 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda640039a0 con 0x7fda6c0a2800 2024-09-16T14:49:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda5affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda5affd640 1 -- 172.21.15.26:0/3959389694 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fda64003d30 con 0x7fda6c0a2800 2024-09-16T14:49:02.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:02.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda5affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:02.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda5affd640 10 monclient: dump: 2024-09-16T14:49:02.889 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:02.889 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:02.889 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:02.889 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:02.889 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:02.889 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:02.889 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:02.889 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:02.889 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:02.889 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:02.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda68d99640 4 set_mon_vals no callback set 2024-09-16T14:49:02.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda5affd640 1 -- 172.21.15.26:0/3959389694 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fda64003e90 con 0x7fda6c0a2800 2024-09-16T14:49:02.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda70e5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:02.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda70e5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:02.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda70e5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:02.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda5affd640 1 -- 172.21.15.26:0/3959389694 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(177..177 src has 73..177) v4 ==== 7121+0+0 (unknown 2880899562 0 0) 0x7fda640442b0 con 0x7fda6c0a2800 2024-09-16T14:49:02.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda70e5d640 1 -- 172.21.15.26:0/3959389694 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda50005050 con 0x7fda6c0a2800 2024-09-16T14:49:02.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.870+0000 7fda6a59c640 10 monclient: get_auth_request con 0x7fda4c0360d0 auth_method 0 2024-09-16T14:49:02.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.874+0000 7fda5affd640 1 -- 172.21.15.26:0/3959389694 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fda64014a70 con 0x7fda6c0a2800 2024-09-16T14:49:02.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.874+0000 7fda5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:02.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:02.874+0000 7fda5affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:03.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:03.094+0000 7fda70e5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:49:03.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:03.094+0000 7fda70e5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:49:03.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:03.094+0000 7fda70e5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:03.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:03.094+0000 7fda70e5d640 1 -- 172.21.15.26:0/3959389694 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fda50005880 con 0x7fda6c0a2800 2024-09-16T14:49:03.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:03.094+0000 7fda5affd640 1 -- 172.21.15.26:0/3959389694 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 499805781) 0x7fda64012ee0 con 0x7fda6c0a2800 2024-09-16T14:49:03.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:03.094+0000 7fda5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:49:03.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:03.094+0000 7fda5affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:03.100 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:49:03.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:03.098+0000 7fda70e5d640 1 -- 172.21.15.26:0/3959389694 >> v1:172.21.15.189:6812/3196161582 conn(0x7fda4c0360d0 legacy=0x7fda4c038590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:03.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:03.098+0000 7fda70e5d640 10 monclient: shutdown 2024-09-16T14:49:03.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:03.098+0000 7fda70e5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:03.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:03.098+0000 7fda70e5d640 1 -- 172.21.15.26:0/3959389694 >> v1:172.21.15.26:6790/0 conn(0x7fda6c0a2800 legacy=0x7fda6c1d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:03.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:03.098+0000 7fda70e5d640 1 -- 172.21.15.26:0/3959389694 shutdown_connections 2024-09-16T14:49:03.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:03.098+0000 7fda70e5d640 1 -- 172.21.15.26:0/3959389694 >> 172.21.15.26:0/3959389694 conn(0x7fda6c0a7f30 msgr2=0x7fda6c06b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:03.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:03.098+0000 7fda70e5d640 1 -- 172.21.15.26:0/3959389694 shutdown_connections 2024-09-16T14:49:03.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:03.098+0000 7fda70e5d640 1 -- 172.21.15.26:0/3959389694 wait complete. 2024-09-16T14:49:03.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:03.609+0000 7f57877fe640 10 monclient: tick 2024-09-16T14:49:03.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:03.609+0000 7f57877fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:03.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:03.609+0000 7f57877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:04.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:04.609+0000 7f57877fe640 10 monclient: tick 2024-09-16T14:49:04.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:04.609+0000 7f57877fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:04.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:04.609+0000 7f57877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:05.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:05.609+0000 7f57877fe640 10 monclient: tick 2024-09-16T14:49:05.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:05.609+0000 7f57877fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:05.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:05.609+0000 7f57877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:06.119 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:49:06.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af27cc640 10 monclient: get_monmap_and_config 2024-09-16T14:49:06.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af27cc640 10 monclient: build_initial_monmap 2024-09-16T14:49:06.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af27cc640 10 monclient: monmap: 2024-09-16T14:49:06.254 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:06.254 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:06.254 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:06.254783+0000 2024-09-16T14:49:06.254 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:06.254783+0000 2024-09-16T14:49:06.254 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:06.254 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:06.254 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:06.254 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:06.255 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:06.255 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:06.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af27cc640 1 Processor -- start 2024-09-16T14:49:06.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af27cc640 1 -- start start 2024-09-16T14:49:06.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af27cc640 10 monclient: init 2024-09-16T14:49:06.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af27cc640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:06.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af27cc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:06.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af27cc640 10 monclient(hunting): picked mon.noname-a con 0x7f5aec0968c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:06.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af27cc640 10 monclient(hunting): picked mon.noname-c con 0x7f5aec098bc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:06.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af27cc640 10 monclient(hunting): picked mon.noname-b con 0x7f5aec096ca0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:06.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af27cc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec06b2d0 con 0x7f5aec0968c0 2024-09-16T14:49:06.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af27cc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec097940 con 0x7f5aec098bc0 2024-09-16T14:49:06.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af27cc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec09fec0 con 0x7f5aec096ca0 2024-09-16T14:49:06.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af27cc640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:06.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af27cc640 10 monclient(hunting): authenticate will time out at 1370.375016s 2024-09-16T14:49:06.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5aebfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f5aec0968c0 legacy=0x7f5aec09f0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:06.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af0d69640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5aec096ca0 0x7f5aec13da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48016/0 (socket says 172.21.15.26:48016) 2024-09-16T14:49:06.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.250+0000 7f5af0d69640 1 -- 172.21.15.26:0/1739093785 learned_addr learned my addr 172.21.15.26:0/1739093785 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:06.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.454+0000 7f5aebfff640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.26:6789/0 conn(0x7f5aec0968c0 legacy=0x7f5aec09f0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:06.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:06.609+0000 7f57877fe640 10 monclient: tick 2024-09-16T14:49:06.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:06.609+0000 7f57877fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:06.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:06.609+0000 7f57877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:06.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:06.858+0000 7f5aebfff640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.26:6789/0 conn(0x7f5aec0968c0 legacy=0x7f5aec09f0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:07.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:07.609+0000 7f57877fe640 10 monclient: tick 2024-09-16T14:49:07.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:07.609+0000 7f57877fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:07.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:07.609+0000 7f57877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:07.662+0000 7f5aebfff640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.26:6789/0 conn(0x7f5aec0968c0 legacy=0x7f5aec09f0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:08.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:08.608+0000 7f57877fe640 10 monclient: tick 2024-09-16T14:49:08.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:08.608+0000 7f57877fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:08.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:08.608+0000 7f57877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:09.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.254+0000 7f5ae9ffb640 10 monclient(hunting): tick 2024-09-16T14:49:09.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.254+0000 7f5ae9ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:49:09.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.254+0000 7f5ae9ffb640 1 monclient(hunting): continuing hunt 2024-09-16T14:49:09.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.254+0000 7f5ae9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:49:09.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.254+0000 7f5ae9ffb640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.26:6790/0 conn(0x7f5aec096ca0 legacy=0x7f5aec13da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:09.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.254+0000 7f5ae9ffb640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.189:6789/0 conn(0x7f5aec098bc0 legacy=0x7f5aec09f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:09.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.254+0000 7f5ae9ffb640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.26:6789/0 conn(0x7f5aec0968c0 legacy=0x7f5aec09f0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:09.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.254+0000 7f5ae9ffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:09.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.254+0000 7f5ae9ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f5aec0968c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:09.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.254+0000 7f5ae9ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f5ad8006e60 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:09.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.254+0000 7f5ae9ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f5ad800ab90 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:09.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.254+0000 7f5ae9ffb640 1 -- 172.21.15.26:0/1739093785 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec097c20 con 0x7f5aec0968c0 2024-09-16T14:49:09.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.254+0000 7f5ae9ffb640 1 -- 172.21.15.26:0/1739093785 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec06b2d0 con 0x7f5ad8006e60 2024-09-16T14:49:09.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.254+0000 7f5ae9ffb640 1 -- 172.21.15.26:0/1739093785 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad80108b0 con 0x7f5ad800ab90 2024-09-16T14:49:09.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.254+0000 7f5ae9ffb640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:09.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.254+0000 7f5aebfff640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.26:6789/0 conn(0x7f5aec0968c0 legacy=0x7f5ad8005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:09.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.458+0000 7f5aebfff640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.26:6789/0 conn(0x7f5aec0968c0 legacy=0x7f5ad8005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:09.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:09.608+0000 7f57877fe640 10 monclient: tick 2024-09-16T14:49:09.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:09.608+0000 7f57877fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:09.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:09.608+0000 7f57877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:09.862+0000 7f5aebfff640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.26:6789/0 conn(0x7f5aec0968c0 legacy=0x7f5ad8005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:10.608+0000 7f57877fe640 10 monclient: tick 2024-09-16T14:49:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:10.608+0000 7f57877fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:10.608+0000 7f57877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:10.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:10.666+0000 7f5aebfff640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.26:6789/0 conn(0x7f5aec0968c0 legacy=0x7f5ad8005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:11.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:11.608+0000 7f57877fe640 10 monclient: tick 2024-09-16T14:49:11.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:11.608+0000 7f57877fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:11.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:11.608+0000 7f57877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:11.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:11.608+0000 7f57877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:12.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.250+0000 7f5ae9ffb640 10 monclient(hunting): tick 2024-09-16T14:49:12.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.250+0000 7f5ae9ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:49:12.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.250+0000 7f5ae9ffb640 1 monclient(hunting): continuing hunt 2024-09-16T14:49:12.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.250+0000 7f5ae9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:49:12.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.250+0000 7f5ae9ffb640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.26:6790/0 conn(0x7f5ad800ab90 legacy=0x7f5ad800d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:12.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.250+0000 7f5ae9ffb640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.189:6789/0 conn(0x7f5ad8006e60 legacy=0x7f5ad80092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:12.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.250+0000 7f5ae9ffb640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.26:6789/0 conn(0x7f5aec0968c0 legacy=0x7f5ad8005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:12.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.250+0000 7f5ae9ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:49:12.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.250+0000 7f5ae9ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f5ad800f980 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:12.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.250+0000 7f5ae9ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f5ad8009ac0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:12.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.250+0000 7f5ae9ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f5aec0968c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:12.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.250+0000 7f5ae9ffb640 1 -- 172.21.15.26:0/1739093785 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad8011ac0 con 0x7f5aec0968c0 2024-09-16T14:49:12.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.250+0000 7f5ae9ffb640 1 -- 172.21.15.26:0/1739093785 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec097c20 con 0x7f5ad800f980 2024-09-16T14:49:12.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.250+0000 7f5ae9ffb640 1 -- 172.21.15.26:0/1739093785 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad8009ea0 con 0x7f5ad8009ac0 2024-09-16T14:49:12.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.250+0000 7f5ae9ffb640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:12.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.250+0000 7f5af0d69640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.26:6789/0 conn(0x7f5aec0968c0 legacy=0x7f5ad8005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:12.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.454+0000 7f5af0d69640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.26:6789/0 conn(0x7f5aec0968c0 legacy=0x7f5ad8005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:12.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:12.608+0000 7f57877fe640 10 monclient: tick 2024-09-16T14:49:12.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:12.608+0000 7f57877fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:12.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:12.608+0000 7f57877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:12.858+0000 7f5af0d69640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.26:6789/0 conn(0x7f5aec0968c0 legacy=0x7f5ad8005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:13.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.430+0000 7f5aeaffd640 1 -- 172.21.15.26:0/1739093785 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895944828 0 0) 0x7f5ad8009ea0 con 0x7f5ad8009ac0 2024-09-16T14:49:13.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.430+0000 7f5aeaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:13.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.430+0000 7f5aeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:13.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.430+0000 7f5aeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:13.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.430+0000 7f5aeaffd640 10 monclient(hunting): my global_id is 154193 2024-09-16T14:49:13.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.430+0000 7f5aeaffd640 1 -- 172.21.15.26:0/1739093785 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad4003530 con 0x7f5ad8009ac0 2024-09-16T14:49:13.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.430+0000 7f5aeaffd640 1 -- 172.21.15.26:0/1739093785 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746808344 0 0) 0x7f5aec097c20 con 0x7f5ad800f980 2024-09-16T14:49:13.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:13.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:13.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:13.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 10 monclient(hunting): my global_id is 144403 2024-09-16T14:49:13.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 1 -- 172.21.15.26:0/1739093785 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad8009ea0 con 0x7f5ad800f980 2024-09-16T14:49:13.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 1 -- 172.21.15.26:0/1739093785 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1255642660 0 0) 0x7f5ad4003530 con 0x7f5ad8009ac0 2024-09-16T14:49:13.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.189:6789/0 conn(0x7f5ad800f980 legacy=0x7f5ad800fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:13.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.26:6789/0 conn(0x7f5aec0968c0 legacy=0x7f5ad8005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:13.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:13.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 1 monclient: found mon.noname-b 2024-09-16T14:49:13.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:13.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:49:13.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 1 -- 172.21.15.26:0/1739093785 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad800d010 con 0x7f5ad8009ac0 2024-09-16T14:49:13.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 10 monclient: _finish_auth 0 2024-09-16T14:49:13.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:13.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:13.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 1 -- 172.21.15.26:0/1739093785 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5ae0001520 con 0x7f5ad8009ac0 2024-09-16T14:49:13.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5af27cc640 5 monclient: authenticate success, global_id 154193 2024-09-16T14:49:13.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5af27cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:13.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:13.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:13.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 10 monclient: dump: 2024-09-16T14:49:13.439 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:13.439 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.440 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.440 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.440 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:13.440 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.440 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:13.440 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:13.440 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:13.440 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5af27cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:13.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 1 -- 172.21.15.26:0/1739093785 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ae00017c0 con 0x7f5ad8009ac0 2024-09-16T14:49:13.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:13.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5af27cc640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:13.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5af27cc640 4 set_mon_vals no callback set 2024-09-16T14:49:13.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 1 -- 172.21.15.26:0/1739093785 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5ae0001b90 con 0x7f5ad8009ac0 2024-09-16T14:49:13.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:13.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:13.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5aeaffd640 10 monclient: dump: 2024-09-16T14:49:13.443 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:13.443 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.443 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.443 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.443 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:13.443 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.443 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:13.443 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:13.443 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:13.443 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5af27cc640 10 monclient: shutdown 2024-09-16T14:49:13.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5af27cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:13.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5af27cc640 1 -- 172.21.15.26:0/1739093785 >> v1:172.21.15.26:6790/0 conn(0x7f5ad8009ac0 legacy=0x7f5ad80092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:13.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5af27cc640 1 -- 172.21.15.26:0/1739093785 shutdown_connections 2024-09-16T14:49:13.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5af27cc640 1 -- 172.21.15.26:0/1739093785 >> 172.21.15.26:0/1739093785 conn(0x7f5aec0a0360 msgr2=0x7f5aec0a0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:13.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5af27cc640 1 -- 172.21.15.26:0/1739093785 shutdown_connections 2024-09-16T14:49:13.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.434+0000 7f5af27cc640 1 -- 172.21.15.26:0/1739093785 wait complete. 2024-09-16T14:49:13.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5af27cc640 10 monclient: build_initial_monmap 2024-09-16T14:49:13.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5af27cc640 10 monclient: monmap: 2024-09-16T14:49:13.444 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:13.444 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:13.444 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:13.444 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:13.444 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:13.445 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:13.445 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:13.445 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:13.445 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:13.445 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5af27cc640 1 Processor -- start 2024-09-16T14:49:13.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5af27cc640 1 -- start start 2024-09-16T14:49:13.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5af27cc640 10 monclient: init 2024-09-16T14:49:13.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5af27cc640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:13.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5af27cc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:49:13.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5af27cc640 10 monclient(hunting): picked mon.noname-b con 0x7f5ad8009ac0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:13.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5af27cc640 10 monclient(hunting): picked mon.noname-c con 0x7f5ad800f980 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:13.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5af27cc640 10 monclient(hunting): picked mon.noname-a con 0x7f5aec0968c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:13.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5af27cc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec09fbf0 con 0x7f5aec0968c0 2024-09-16T14:49:13.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5af27cc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec09a880 con 0x7f5ad8009ac0 2024-09-16T14:49:13.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5af27cc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec09f6c0 con 0x7f5ad800f980 2024-09-16T14:49:13.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5af27cc640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:13.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5af27cc640 10 monclient(hunting): authenticate will time out at 1377.562485s 2024-09-16T14:49:13.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5af0d69640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f5aec0968c0 legacy=0x7f5aec1d21a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:13.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5aeb7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5ad800f980 0x7f5aec1cea70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:52828/0 (socket says 172.21.15.26:52828) 2024-09-16T14:49:13.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.438+0000 7f5aeb7fe640 1 -- 172.21.15.26:0/2547455974 learned_addr learned my addr 172.21.15.26:0/2547455974 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:13.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 1 -- 172.21.15.26:0/2547455974 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529886711 0 0) 0x7f5aec09f6c0 con 0x7f5ad800f980 2024-09-16T14:49:13.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:49:13.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:13.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:13.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 10 monclient(hunting): my global_id is 154196 2024-09-16T14:49:13.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 1 -- 172.21.15.26:0/2547455974 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad800d1a0 con 0x7f5ad800f980 2024-09-16T14:49:13.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 1 -- 172.21.15.26:0/2547455974 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2564293643 0 0) 0x7f5ad800d1a0 con 0x7f5ad800f980 2024-09-16T14:49:13.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 1 -- 172.21.15.26:0/2547455974 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5aec09f6c0 con 0x7f5ad800f980 2024-09-16T14:49:13.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 1 -- 172.21.15.26:0/2547455974 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5ae00014a0 con 0x7f5ad800f980 2024-09-16T14:49:13.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:13.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:13.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 10 monclient(hunting): dump: 2024-09-16T14:49:13.452 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:13.452 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.452 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.452 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.452 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:13.452 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.452 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:13.452 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:13.452 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:13.452 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:13.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 1 -- 172.21.15.26:0/2547455974 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 880869797 0 0) 0x7f5aec09f6c0 con 0x7f5ad800f980 2024-09-16T14:49:13.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 1 -- 172.21.15.26:0/2547455974 >> v1:172.21.15.189:6789/0 conn(0x7f5ad8009ac0 legacy=0x7f5aec1c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:13.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 1 -- 172.21.15.26:0/2547455974 >> v1:172.21.15.26:6789/0 conn(0x7f5aec0968c0 legacy=0x7f5aec1d21a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:13.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:49:13.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 1 monclient: found mon.c 2024-09-16T14:49:13.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:13.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:13.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 1 -- 172.21.15.26:0/2547455974 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec0918a0 con 0x7f5ad800f980 2024-09-16T14:49:13.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 10 monclient: _finish_auth 0 2024-09-16T14:49:13.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:49:13.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:13.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5af27cc640 5 monclient: authenticate success, global_id 154196 2024-09-16T14:49:13.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5af27cc640 10 monclient: _renew_subs 2024-09-16T14:49:13.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5af27cc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:13.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5af27cc640 1 -- 172.21.15.26:0/2547455974 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec091b80 con 0x7f5ad800f980 2024-09-16T14:49:13.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5af27cc640 10 monclient: _renew_subs 2024-09-16T14:49:13.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5af27cc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:13.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5af27cc640 1 -- 172.21.15.26:0/2547455974 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec090fb0 con 0x7f5ad800f980 2024-09-16T14:49:13.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 1 -- 172.21.15.26:0/2547455974 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ae00038b0 con 0x7f5ad800f980 2024-09-16T14:49:13.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:13.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 1 -- 172.21.15.26:0/2547455974 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5ae0003c40 con 0x7f5ad800f980 2024-09-16T14:49:13.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:13.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:13.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5aea7fc640 4 set_mon_vals no callback set 2024-09-16T14:49:13.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 10 monclient: dump: 2024-09-16T14:49:13.454 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:13.454 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.454 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.454 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.455 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:13.455 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.455 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:13.455 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:13.455 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:13.455 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 1 -- 172.21.15.26:0/2547455974 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5ae0003da0 con 0x7f5ad800f980 2024-09-16T14:49:13.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5af27cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:13.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5af27cc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:13.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5af27cc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:13.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5af27cc640 1 -- 172.21.15.26:0/2547455974 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5acc005050 con 0x7f5ad800f980 2024-09-16T14:49:13.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5ae8ff9640 1 -- 172.21.15.26:0/2547455974 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(177..177 src has 73..177) v4 ==== 7121+0+0 (unknown 2880899562 0 0) 0x7f5ae0043ef0 con 0x7f5ad800f980 2024-09-16T14:49:13.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.442+0000 7f5aebfff640 10 monclient: get_auth_request con 0x7f5ad8037b50 auth_method 0 2024-09-16T14:49:13.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.450+0000 7f5ae8ff9640 1 -- 172.21.15.26:0/2547455974 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5ae0014540 con 0x7f5ad800f980 2024-09-16T14:49:13.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.450+0000 7f5ae8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:13.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.450+0000 7f5ae8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:13.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:13.608+0000 7f57877fe640 10 monclient: tick 2024-09-16T14:49:13.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:13.608+0000 7f57877fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:13.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:13.608+0000 7f57877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:13.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.666+0000 7f5af27cc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:49:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.666+0000 7f5af27cc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:49:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.666+0000 7f5af27cc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.666+0000 7f5af27cc640 1 -- 172.21.15.26:0/2547455974 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5acc005880 con 0x7f5ad800f980 2024-09-16T14:49:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.666+0000 7f5ae8ff9640 1 -- 172.21.15.26:0/2547455974 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1908546991) 0x7f5ae0012be0 con 0x7f5ad800f980 2024-09-16T14:49:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.670+0000 7f5ae8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:49:13.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.670+0000 7f5ae8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:13.673 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":64,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:49:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.670+0000 7f5af27cc640 1 -- 172.21.15.26:0/2547455974 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5ad8037b50 legacy=0x7f5ad803a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.670+0000 7f5af27cc640 10 monclient: shutdown 2024-09-16T14:49:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.670+0000 7f5af27cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.670+0000 7f5af27cc640 1 -- 172.21.15.26:0/2547455974 >> v1:172.21.15.26:6790/0 conn(0x7f5ad800f980 legacy=0x7f5aec1cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.670+0000 7f5af27cc640 1 -- 172.21.15.26:0/2547455974 shutdown_connections 2024-09-16T14:49:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.670+0000 7f5af27cc640 1 -- 172.21.15.26:0/2547455974 >> 172.21.15.26:0/2547455974 conn(0x7f5aec0a0360 msgr2=0x7f5aec0955e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.670+0000 7f5af27cc640 1 -- 172.21.15.26:0/2547455974 shutdown_connections 2024-09-16T14:49:13.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.670+0000 7f5af27cc640 1 -- 172.21.15.26:0/2547455974 wait complete. 2024-09-16T14:49:13.690 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:49:13.691 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:49:13.691 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:49:13.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 10 monclient: get_monmap_and_config 2024-09-16T14:49:13.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 10 monclient: build_initial_monmap 2024-09-16T14:49:13.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 10 monclient: monmap: 2024-09-16T14:49:13.813 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:13.813 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.813 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:13.813585+0000 2024-09-16T14:49:13.813 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:13.813585+0000 2024-09-16T14:49:13.813 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:13.813 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.813 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:13.813 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:13.814 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:13.814 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 1 Processor -- start 2024-09-16T14:49:13.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 1 -- start start 2024-09-16T14:49:13.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 10 monclient: init 2024-09-16T14:49:13.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:13.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:49:13.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 10 monclient(hunting): picked mon.noname-c con 0x7f2cc00a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:13.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 10 monclient(hunting): picked mon.noname-b con 0x7f2cc0099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:13.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 10 monclient(hunting): picked mon.noname-a con 0x7f2cc00a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:13.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc0098ea0 con 0x7f2cc00a26f0 2024-09-16T14:49:13.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc00997b0 con 0x7f2cc00a1c00 2024-09-16T14:49:13.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc0099460 con 0x7f2cc0099b60 2024-09-16T14:49:13.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:13.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 10 monclient(hunting): authenticate will time out at 1377.933676s 2024-09-16T14:49:13.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbffff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2cc00a26f0 legacy=0x7f2cc013d330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:13.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbeffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f2cc0099b60 0x7f2cc0139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:52832/0 (socket says 172.21.15.26:52832) 2024-09-16T14:49:13.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbeffd640 1 -- 172.21.15.26:0/940769178 learned_addr learned my addr 172.21.15.26:0/940769178 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:13.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 1 -- 172.21.15.26:0/940769178 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2072748548 0 0) 0x7f2cc0099460 con 0x7f2cc0099b60 2024-09-16T14:49:13.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:13.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:13.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:13.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 10 monclient(hunting): my global_id is 154199 2024-09-16T14:49:13.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 1 -- 172.21.15.26:0/940769178 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca8003530 con 0x7f2cc0099b60 2024-09-16T14:49:13.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 1 -- 172.21.15.26:0/940769178 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3978837475 0 0) 0x7f2ca8003530 con 0x7f2cc0099b60 2024-09-16T14:49:13.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 1 -- 172.21.15.26:0/940769178 >> v1:172.21.15.189:6789/0 conn(0x7f2cc00a1c00 legacy=0x7f2cc00a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:13.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 1 -- 172.21.15.26:0/940769178 >> v1:172.21.15.26:6789/0 conn(0x7f2cc00a26f0 legacy=0x7f2cc013d330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:13.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:49:13.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 1 monclient: found mon.noname-b 2024-09-16T14:49:13.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:13.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:49:13.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 1 -- 172.21.15.26:0/940769178 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc00a34b0 con 0x7f2cc0099b60 2024-09-16T14:49:13.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 10 monclient: _finish_auth 0 2024-09-16T14:49:13.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:13.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:13.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 5 monclient: authenticate success, global_id 154199 2024-09-16T14:49:13.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 1 -- 172.21.15.26:0/940769178 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2cb4001620 con 0x7f2cc0099b60 2024-09-16T14:49:13.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:13.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:13.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:13.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 10 monclient: dump: 2024-09-16T14:49:13.824 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:13.824 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.824 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.824 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.824 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:13.824 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.824 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:13.825 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 1 -- 172.21.15.26:0/940769178 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cb4001940 con 0x7f2cc0099b60 2024-09-16T14:49:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 1 -- 172.21.15.26:0/940769178 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2cb4001d10 con 0x7f2cc0099b60 2024-09-16T14:49:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cbe7fc640 10 monclient: dump: 2024-09-16T14:49:13.825 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:13.826 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.826 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.826 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.826 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:13.826 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.826 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:13.826 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:13.826 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:13.827 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:13.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 4 set_mon_vals no callback set 2024-09-16T14:49:13.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 10 monclient: shutdown 2024-09-16T14:49:13.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:13.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 1 -- 172.21.15.26:0/940769178 >> v1:172.21.15.26:6790/0 conn(0x7f2cc0099b60 legacy=0x7f2cc0139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:13.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 1 -- 172.21.15.26:0/940769178 shutdown_connections 2024-09-16T14:49:13.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 1 -- 172.21.15.26:0/940769178 >> 172.21.15.26:0/940769178 conn(0x7f2cc00a3d00 msgr2=0x7f2cc00a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:13.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 1 -- 172.21.15.26:0/940769178 shutdown_connections 2024-09-16T14:49:13.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 1 -- 172.21.15.26:0/940769178 wait complete. 2024-09-16T14:49:13.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 10 monclient: build_initial_monmap 2024-09-16T14:49:13.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.810+0000 7f2cc622e640 10 monclient: monmap: 2024-09-16T14:49:13.827 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:13.828 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:13.828 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:13.828 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:13.828 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:13.828 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:13.828 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:13.828 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:13.828 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:13.828 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 1 Processor -- start 2024-09-16T14:49:13.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 1 -- start start 2024-09-16T14:49:13.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 10 monclient: init 2024-09-16T14:49:13.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:13.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:49:13.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 10 monclient(hunting): picked mon.noname-c con 0x7f2cc0099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:13.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 10 monclient(hunting): picked mon.noname-b con 0x7f2cc00a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:13.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 10 monclient(hunting): picked mon.noname-a con 0x7f2cc00a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:13.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc011f890 con 0x7f2cc00a26f0 2024-09-16T14:49:13.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc009efe0 con 0x7f2cc00a1c00 2024-09-16T14:49:13.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc009ea90 con 0x7f2cc0099b60 2024-09-16T14:49:13.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:13.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 10 monclient(hunting): authenticate will time out at 1377.936727s 2024-09-16T14:49:13.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cbffff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2cc00a26f0 legacy=0x7f2cc013dc70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:13.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cbf7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f2cc0099b60 0x7f2cc013ce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:52834/0 (socket says 172.21.15.26:52834) 2024-09-16T14:49:13.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cbf7fe640 1 -- 172.21.15.26:0/2835139344 learned_addr learned my addr 172.21.15.26:0/2835139344 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:13.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895097808 0 0) 0x7f2cc009ea90 con 0x7f2cc0099b60 2024-09-16T14:49:13.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:13.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:13.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:13.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient(hunting): my global_id is 154202 2024-09-16T14:49:13.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cac003530 con 0x7f2cc0099b60 2024-09-16T14:49:13.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3686789945 0 0) 0x7f2cac003530 con 0x7f2cc0099b60 2024-09-16T14:49:13.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cc009ea90 con 0x7f2cc0099b60 2024-09-16T14:49:13.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2cb8001d50 con 0x7f2cc0099b60 2024-09-16T14:49:13.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:13.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:13.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient(hunting): dump: 2024-09-16T14:49:13.830 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:13.830 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.831 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.831 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.831 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:13.831 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.831 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:13.831 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:13.831 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:13.831 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:13.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19193977 0 0) 0x7f2cc009efe0 con 0x7f2cc00a1c00 2024-09-16T14:49:13.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:13.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:13.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:13.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient(hunting): my global_id is 144412 2024-09-16T14:49:13.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cac003530 con 0x7f2cc00a1c00 2024-09-16T14:49:13.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 76490255 0 0) 0x7f2cc009ea90 con 0x7f2cc0099b60 2024-09-16T14:49:13.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 >> v1:172.21.15.189:6789/0 conn(0x7f2cc00a1c00 legacy=0x7f2cc013d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:13.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 >> v1:172.21.15.26:6789/0 conn(0x7f2cc00a26f0 legacy=0x7f2cc013dc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:13.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:49:13.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 monclient: found mon.c 2024-09-16T14:49:13.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:13.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:13.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc009e3f0 con 0x7f2cc0099b60 2024-09-16T14:49:13.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient: _finish_auth 0 2024-09-16T14:49:13.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:49:13.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:13.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 5 monclient: authenticate success, global_id 154202 2024-09-16T14:49:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 10 monclient: _renew_subs 2024-09-16T14:49:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 1 -- 172.21.15.26:0/2835139344 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc011d850 con 0x7f2cc0099b60 2024-09-16T14:49:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 10 monclient: _renew_subs 2024-09-16T14:49:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 1 -- 172.21.15.26:0/2835139344 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc011e160 con 0x7f2cc0099b60 2024-09-16T14:49:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cb8002100 con 0x7f2cc0099b60 2024-09-16T14:49:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2cb8003e70 con 0x7f2cc0099b60 2024-09-16T14:49:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cbdffb640 4 set_mon_vals no callback set 2024-09-16T14:49:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient: dump: 2024-09-16T14:49:13.833 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:13.834 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cc622e640 1 -- 172.21.15.26:0/2835139344 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2ca0006590 con 0x7f2ca0002830 2024-09-16T14:49:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2cb8003fd0 con 0x7f2cc0099b60 2024-09-16T14:49:13.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(177..177 src has 73..177) v4 ==== 7121+0+0 (unknown 2880899562 0 0) 0x7f2cb80443e0 con 0x7f2cc0099b60 2024-09-16T14:49:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561621109 0 0) 0x7f2ca0006590 con 0x7f2ca0002830 2024-09-16T14:49:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient(hunting): my global_id is 154205 2024-09-16T14:49:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc009ea90 con 0x7f2ca0002830 2024-09-16T14:49:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2cbffff640 10 monclient: get_auth_request con 0x7f2cac036130 auth_method 0 2024-09-16T14:49:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2048139809 0 0) 0x7f2cc009ea90 con 0x7f2ca0002830 2024-09-16T14:49:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ca0006590 con 0x7f2ca0002830 2024-09-16T14:49:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2cb4001d50 con 0x7f2ca0002830 2024-09-16T14:49:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient: dump: 2024-09-16T14:49:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:13.836 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1344836088 0 0) 0x7f2ca0006590 con 0x7f2ca0002830 2024-09-16T14:49:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ca000a0e0 con 0x7f2ca0002830 2024-09-16T14:49:13.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2cb4001610 con 0x7f2ca0002830 2024-09-16T14:49:13.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:13.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:13.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 >> v1:172.21.15.26:6790/0 conn(0x7f2ca0002830 legacy=0x7f2ca0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:13.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.814+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 >> v1:172.21.15.26:6790/0 conn(0x7f2ca0002830 legacy=0x7f2ca0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:13.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2cc622e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:49:13.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2cc622e640 1 -- 172.21.15.26:0/2835139344 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2ca0009d80 con 0x7f2ca00090a0 2024-09-16T14:49:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254667863 0 0) 0x7f2ca0009d80 con 0x7f2ca00090a0 2024-09-16T14:49:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 10 monclient(hunting): my global_id is 154208 2024-09-16T14:49:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca0006590 con 0x7f2ca00090a0 2024-09-16T14:49:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4293724165 0 0) 0x7f2ca0006590 con 0x7f2ca00090a0 2024-09-16T14:49:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ca0009d80 con 0x7f2ca00090a0 2024-09-16T14:49:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2cb4004ce0 con 0x7f2ca00090a0 2024-09-16T14:49:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 10 monclient: dump: 2024-09-16T14:49:13.839 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:13.840 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3063549729 0 0) 0x7f2ca0009d80 con 0x7f2ca00090a0 2024-09-16T14:49:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2ca000c290 con 0x7f2ca00090a0 2024-09-16T14:49:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4036 (unknown 0 0 3888831520) 0x7f2cb4005fd0 con 0x7f2ca00090a0 2024-09-16T14:49:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:49:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 >> v1:172.21.15.26:6790/0 conn(0x7f2ca00090a0 legacy=0x7f2ca0009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2ca7fff640 1 -- 172.21.15.26:0/2835139344 >> v1:172.21.15.26:6790/0 conn(0x7f2ca00090a0 legacy=0x7f2ca0009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:13.841 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:49:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:49:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:49:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:49:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 64, 2024-09-16T14:49:13.841 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:49:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:49:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:49:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:49:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:49:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:49:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:13.842 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:49:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:49:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:13.843 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:49:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:49:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:49:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:49:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:49:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:49:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:13.844 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:49:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:49:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:49:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:49:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:49:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:49:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:49:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:49:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:13.845 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:49:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:49:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:49:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:49:13.846 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:49:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:49:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:49:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:13.847 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:49:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:49:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:49:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:13.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:49:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:49:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:49:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:49:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:13.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:49:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:49:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:49:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:13.850 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:49:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:49:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:49:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:49:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:13.851 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:49:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:49:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:49:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:13.852 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:49:13.852 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:49:13.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2cc622e640 1 -- 172.21.15.26:0/2835139344 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2cac036130 legacy=0x7f2cac0385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:13.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2cc622e640 10 monclient: shutdown 2024-09-16T14:49:13.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2cc622e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:13.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2cc622e640 1 -- 172.21.15.26:0/2835139344 >> v1:172.21.15.26:6790/0 conn(0x7f2cc0099b60 legacy=0x7f2cc013ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:13.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2cbffff640 1 -- 172.21.15.26:0/2835139344 reap_dead start 2024-09-16T14:49:13.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2cbffff640 1 -- 172.21.15.26:0/2835139344 reap_dead start 2024-09-16T14:49:13.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2cc622e640 1 -- 172.21.15.26:0/2835139344 shutdown_connections 2024-09-16T14:49:13.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2cc622e640 1 -- 172.21.15.26:0/2835139344 >> 172.21.15.26:0/2835139344 conn(0x7f2cc00a3d00 msgr2=0x7f2cc00a0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:13.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2cc622e640 1 -- 172.21.15.26:0/2835139344 shutdown_connections 2024-09-16T14:49:13.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.822+0000 7f2cc622e640 1 -- 172.21.15.26:0/2835139344 wait complete. 2024-09-16T14:49:13.855 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:49:13.855 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T14:49:13.855 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:49:13.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 10 monclient: get_monmap_and_config 2024-09-16T14:49:13.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 10 monclient: build_initial_monmap 2024-09-16T14:49:13.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 10 monclient: monmap: 2024-09-16T14:49:13.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:13.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:13.974005+0000 2024-09-16T14:49:13.973 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:13.974005+0000 2024-09-16T14:49:13.974 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:13.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.974 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:13.974 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:13.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:13.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 1 Processor -- start 2024-09-16T14:49:13.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 1 -- start start 2024-09-16T14:49:13.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 10 monclient: init 2024-09-16T14:49:13.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:13.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:49:13.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 10 monclient(hunting): picked mon.noname-a con 0x7fdaec09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:13.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 10 monclient(hunting): picked mon.noname-b con 0x7fdaec132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:13.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 10 monclient(hunting): picked mon.noname-c con 0x7fdaec138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:13.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec09d260 con 0x7fdaec09cb80 2024-09-16T14:49:13.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec132a80 con 0x7fdaec138ff0 2024-09-16T14:49:13.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec09cf60 con 0x7fdaec132650 2024-09-16T14:49:13.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:13.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 10 monclient(hunting): authenticate will time out at 1378.093938s 2024-09-16T14:49:13.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf0c0d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fdaec09cb80 legacy=0x7fdaec0717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:13.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaebfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fdaec132650 0x7fdaec1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:52860/0 (socket says 172.21.15.26:52860) 2024-09-16T14:49:13.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaebfff640 1 -- 172.21.15.26:0/4150340732 learned_addr learned my addr 172.21.15.26:0/4150340732 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:13.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 1 -- 172.21.15.26:0/4150340732 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996879486 0 0) 0x7fdaec09cf60 con 0x7fdaec132650 2024-09-16T14:49:13.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:13.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:13.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:13.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 10 monclient(hunting): my global_id is 154211 2024-09-16T14:49:13.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 1 -- 172.21.15.26:0/4150340732 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdacc003530 con 0x7fdaec132650 2024-09-16T14:49:13.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 1 -- 172.21.15.26:0/4150340732 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1764613906 0 0) 0x7fdacc003530 con 0x7fdaec132650 2024-09-16T14:49:13.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 1 -- 172.21.15.26:0/4150340732 >> v1:172.21.15.189:6789/0 conn(0x7fdaec138ff0 legacy=0x7fdaec13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:13.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 1 -- 172.21.15.26:0/4150340732 >> v1:172.21.15.26:6789/0 conn(0x7fdaec09cb80 legacy=0x7fdaec0717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:13.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:49:13.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 1 monclient: found mon.noname-b 2024-09-16T14:49:13.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:13.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:49:13.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 1 -- 172.21.15.26:0/4150340732 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec09f2d0 con 0x7fdaec132650 2024-09-16T14:49:13.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 10 monclient: _finish_auth 0 2024-09-16T14:49:13.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:13.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:13.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 1 -- 172.21.15.26:0/4150340732 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdae4001660 con 0x7fdaec132650 2024-09-16T14:49:13.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:13.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:13.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 10 monclient: dump: 2024-09-16T14:49:13.980 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:13.980 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.980 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.980 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.980 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:13.981 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.981 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:13.981 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:13.981 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:13.981 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 5 monclient: authenticate success, global_id 154211 2024-09-16T14:49:13.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:13.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 1 -- 172.21.15.26:0/4150340732 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdae40018c0 con 0x7fdaec132650 2024-09-16T14:49:13.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:13.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:13.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 4 set_mon_vals no callback set 2024-09-16T14:49:13.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 1 -- 172.21.15.26:0/4150340732 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdae4001c50 con 0x7fdaec132650 2024-09-16T14:49:13.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:13.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:13.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaeb7fe640 10 monclient: dump: 2024-09-16T14:49:13.982 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:13.982 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.982 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.982 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.982 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:13.982 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.982 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:13.982 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:13.983 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:13.983 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 10 monclient: shutdown 2024-09-16T14:49:13.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:13.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 1 -- 172.21.15.26:0/4150340732 >> v1:172.21.15.26:6790/0 conn(0x7fdaec132650 legacy=0x7fdaec1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:13.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 1 -- 172.21.15.26:0/4150340732 shutdown_connections 2024-09-16T14:49:13.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 1 -- 172.21.15.26:0/4150340732 >> 172.21.15.26:0/4150340732 conn(0x7fdaec09fb50 msgr2=0x7fdaec09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:13.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 1 -- 172.21.15.26:0/4150340732 shutdown_connections 2024-09-16T14:49:13.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.970+0000 7fdaf2e71640 1 -- 172.21.15.26:0/4150340732 wait complete. 2024-09-16T14:49:13.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 10 monclient: build_initial_monmap 2024-09-16T14:49:13.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 10 monclient: monmap: 2024-09-16T14:49:13.984 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:13.984 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:13.984 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:13.984 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:13.984 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:13.984 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:13.984 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:13.984 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:13.984 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:13.984 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 1 Processor -- start 2024-09-16T14:49:13.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 1 -- start start 2024-09-16T14:49:13.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 10 monclient: init 2024-09-16T14:49:13.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:13.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:13.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 10 monclient(hunting): picked mon.noname-b con 0x7fdaec09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:13.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 10 monclient(hunting): picked mon.noname-a con 0x7fdaec132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:13.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 10 monclient(hunting): picked mon.noname-c con 0x7fdaec138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:13.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec1355d0 con 0x7fdaec132650 2024-09-16T14:49:13.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec13c400 con 0x7fdaec09cb80 2024-09-16T14:49:13.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaebfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fdaec132650 legacy=0x7fdaec12e6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:13.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec0a28f0 con 0x7fdaec138ff0 2024-09-16T14:49:13.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf140e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fdaec138ff0 0x7fdaec12ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:52862/0 (socket says 172.21.15.26:52862) 2024-09-16T14:49:13.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:13.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf140e640 1 -- 172.21.15.26:0/712436339 learned_addr learned my addr 172.21.15.26:0/712436339 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:13.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 10 monclient(hunting): authenticate will time out at 1378.097767s 2024-09-16T14:49:13.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139391977 0 0) 0x7fdaec0a28f0 con 0x7fdaec138ff0 2024-09-16T14:49:13.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient(hunting): _negotiate 2024-09-16T14:49:13.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:13.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:13.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient(hunting): my global_id is 154214 2024-09-16T14:49:13.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad0003530 con 0x7fdaec138ff0 2024-09-16T14:49:13.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 10630030 0 0) 0x7fdad0003530 con 0x7fdaec138ff0 2024-09-16T14:49:13.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdaec0a28f0 con 0x7fdaec138ff0 2024-09-16T14:49:13.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149116160 0 0) 0x7fdaec13c400 con 0x7fdaec09cb80 2024-09-16T14:49:13.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient(hunting): _negotiate 2024-09-16T14:49:13.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:13.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:13.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient(hunting): my global_id is 144418 2024-09-16T14:49:13.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad0003530 con 0x7fdaec09cb80 2024-09-16T14:49:13.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdadc001c80 con 0x7fdaec138ff0 2024-09-16T14:49:13.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:13.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:13.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient(hunting): dump: 2024-09-16T14:49:13.991 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:13.992 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.993 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.993 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.993 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:13.993 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.993 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:13.993 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:13.994 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:13.994 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:13.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1209124894 0 0) 0x7fdaec0a28f0 con 0x7fdaec138ff0 2024-09-16T14:49:13.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 >> v1:172.21.15.189:6789/0 conn(0x7fdaec09cb80 legacy=0x7fdaec12dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:13.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 >> v1:172.21.15.26:6789/0 conn(0x7fdaec132650 legacy=0x7fdaec12e6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:13.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient: _finish_hunting 0 2024-09-16T14:49:13.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 1 monclient: found mon.c 2024-09-16T14:49:13.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:13.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:13.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec09a240 con 0x7fdaec138ff0 2024-09-16T14:49:13.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient: _finish_auth 0 2024-09-16T14:49:13.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient: _check_auth_tickets 2024-09-16T14:49:13.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:13.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 5 monclient: authenticate success, global_id 154214 2024-09-16T14:49:13.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 10 monclient: _renew_subs 2024-09-16T14:49:13.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:13.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 1 -- 172.21.15.26:0/712436339 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaec1352f0 con 0x7fdaec138ff0 2024-09-16T14:49:13.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 10 monclient: _renew_subs 2024-09-16T14:49:13.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:13.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 1 -- 172.21.15.26:0/712436339 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaec133aa0 con 0x7fdaec138ff0 2024-09-16T14:49:13.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdadc002050 con 0x7fdaec138ff0 2024-09-16T14:49:13.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:13.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdadc003df0 con 0x7fdaec138ff0 2024-09-16T14:49:13.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:13.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:13.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaeaffd640 4 set_mon_vals no callback set 2024-09-16T14:49:13.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 10 monclient: dump: 2024-09-16T14:49:13.996 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:13.996 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.997 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.997 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.997 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:13.997 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.997 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:13.997 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:13.997 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:13.997 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:13.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:13.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fdadc003f50 con 0x7fdaec138ff0 2024-09-16T14:49:13.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaf2e71640 1 -- 172.21.15.26:0/712436339 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdac4006590 con 0x7fdac4002830 2024-09-16T14:49:13.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(177..177 src has 73..177) v4 ==== 7121+0+0 (unknown 2880899562 0 0) 0x7fdadc0441d0 con 0x7fdaec138ff0 2024-09-16T14:49:13.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.974+0000 7fdaebfff640 10 monclient: get_auth_request con 0x7fdad0036130 auth_method 0 2024-09-16T14:49:13.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371443918 0 0) 0x7fdac4006590 con 0x7fdac4002830 2024-09-16T14:49:13.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 10 monclient(hunting): _negotiate 2024-09-16T14:49:13.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:13.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:13.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 10 monclient(hunting): my global_id is 144421 2024-09-16T14:49:13.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaec0a28f0 con 0x7fdac4002830 2024-09-16T14:49:13.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3240968021 0 0) 0x7fdaec0a28f0 con 0x7fdac4002830 2024-09-16T14:49:13.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdac4006590 con 0x7fdac4002830 2024-09-16T14:49:13.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdad40040b0 con 0x7fdac4002830 2024-09-16T14:49:13.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:13.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:13.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 10 monclient: dump: 2024-09-16T14:49:13.999 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:13.999 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:13.999 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.999 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:13.999 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:13.999 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:13.999 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:13.999 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:14.000 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:14.000 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:14.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 684394909 0 0) 0x7fdac4006590 con 0x7fdac4002830 2024-09-16T14:49:14.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdac400a0e0 con 0x7fdac4002830 2024-09-16T14:49:14.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fdad4006370 con 0x7fdac4002830 2024-09-16T14:49:14.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:14.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:14.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 >> v1:172.21.15.189:6789/0 conn(0x7fdac4002830 legacy=0x7fdac4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:14.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.978+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 >> v1:172.21.15.189:6789/0 conn(0x7fdac4002830 legacy=0x7fdac4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:14.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.982+0000 7fdaf2e71640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:49:14.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.982+0000 7fdaf2e71640 1 -- 172.21.15.26:0/712436339 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdac4009d80 con 0x7fdac40090a0 2024-09-16T14:49:14.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172257553 0 0) 0x7fdac4009d80 con 0x7fdac40090a0 2024-09-16T14:49:14.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 10 monclient(hunting): _negotiate 2024-09-16T14:49:14.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:14.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:14.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 10 monclient(hunting): my global_id is 144424 2024-09-16T14:49:14.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdac4006590 con 0x7fdac40090a0 2024-09-16T14:49:14.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1978528278 0 0) 0x7fdac4006590 con 0x7fdac40090a0 2024-09-16T14:49:14.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdac4009d80 con 0x7fdac40090a0 2024-09-16T14:49:14.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdad4004860 con 0x7fdac40090a0 2024-09-16T14:49:14.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:14.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:14.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 10 monclient: dump: 2024-09-16T14:49:14.003 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:14.003 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:14.003 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:14.003 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:14.003 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:14.003 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:14.003 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:14.004 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:14.004 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:14.004 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:14.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3392959833 0 0) 0x7fdac4009d80 con 0x7fdac40090a0 2024-09-16T14:49:14.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdac400c290 con 0x7fdac40090a0 2024-09-16T14:49:14.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4034 (unknown 0 0 1102012367) 0x7fdad4005c80 con 0x7fdac40090a0 2024-09-16T14:49:14.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:49:14.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:14.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 >> v1:172.21.15.189:6789/0 conn(0x7fdac40090a0 legacy=0x7fdac4009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:14.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdae97fa640 1 -- 172.21.15.26:0/712436339 >> v1:172.21.15.189:6789/0 conn(0x7fdac40090a0 legacy=0x7fdac4009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:14.005 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:49:14.005 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:49:14.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:49:14.005 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:49:14.005 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 64, 2024-09-16T14:49:14.005 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:49:14.005 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:49:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:49:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:49:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:49:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:49:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:14.006 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:49:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:49:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:14.007 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:49:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:49:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:49:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:49:14.008 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:49:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:49:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:49:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:49:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:49:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:49:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:49:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:49:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:49:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:14.009 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:49:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:14.010 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:49:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:49:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:49:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:49:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:14.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:49:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:49:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:49:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:14.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:49:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:49:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:49:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:49:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:49:14.013 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:49:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:49:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:49:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:14.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:49:14.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:14.016 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:14.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:14.018 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:49:14.018 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:49:14.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdaf2e71640 1 -- 172.21.15.26:0/712436339 >> v1:172.21.15.189:6812/3196161582 conn(0x7fdad0036130 legacy=0x7fdad00385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:14.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdaf2e71640 10 monclient: shutdown 2024-09-16T14:49:14.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdaf2e71640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:14.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdaf2e71640 1 -- 172.21.15.26:0/712436339 >> v1:172.21.15.26:6790/0 conn(0x7fdaec138ff0 legacy=0x7fdaec12ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:14.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdaf140e640 1 -- 172.21.15.26:0/712436339 reap_dead start 2024-09-16T14:49:14.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdaf140e640 1 -- 172.21.15.26:0/712436339 reap_dead start 2024-09-16T14:49:14.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdaf2e71640 1 -- 172.21.15.26:0/712436339 shutdown_connections 2024-09-16T14:49:14.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdaf2e71640 1 -- 172.21.15.26:0/712436339 >> 172.21.15.26:0/712436339 conn(0x7fdaec09fb50 msgr2=0x7fdaec131c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:14.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdaf2e71640 1 -- 172.21.15.26:0/712436339 shutdown_connections 2024-09-16T14:49:14.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:13.986+0000 7fdaf2e71640 1 -- 172.21.15.26:0/712436339 wait complete. 2024-09-16T14:49:14.020 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:49:14.020 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:49:14.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.424+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "fooo"}]=0 removed item id -16 name 'fooo' from crush map v178) v1 ==== 122+0+0 (unknown 1386201458 0 0) 0x7f5790012af0 con 0x7f57a009df10 2024-09-16T14:49:14.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.424+0000 7f579cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T14:49:14.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.424+0000 7f579cff9640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'fooo' from crush map 2024-09-16T14:49:14.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.528+0000 7f57a6712640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T14:49:14.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.528+0000 7f57a6712640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T14:49:14.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.528+0000 7f57a6712640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:14.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.528+0000 7f57a6712640 1 -- 172.21.15.189:0/2362306898 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "fooo"} v 0) v1 -- 0x7f57800055a0 con 0x7f57a009df10 2024-09-16T14:49:14.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.532+0000 7f579cff9640 1 -- 172.21.15.189:0/2362306898 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "fooo"}]=0 device 'fooo' does not appear in the crush map v178) v1 ==== 122+0+0 (unknown 2051004323 0 0) 0x7f5790014270 con 0x7f57a009df10 2024-09-16T14:49:14.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.532+0000 7f579cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T14:49:14.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.532+0000 7f579cff9640 10 monclient: _finish_command 3 = system:0 device 'fooo' does not appear in the crush map 2024-09-16T14:49:14.533 INFO:tasks.workunit.client.0.smithi189.stderr:device 'fooo' does not appear in the crush map 2024-09-16T14:49:14.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.532+0000 7f57a6712640 1 -- 172.21.15.189:0/2362306898 >> v1:172.21.15.189:6812/3196161582 conn(0x7f578c036360 legacy=0x7f578c038820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:14.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.532+0000 7f57a6712640 10 monclient: shutdown 2024-09-16T14:49:14.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.532+0000 7f57a6712640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:14.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.532+0000 7f57a6712640 1 -- 172.21.15.189:0/2362306898 >> v1:172.21.15.189:6789/0 conn(0x7f57a009df10 legacy=0x7f57a0139920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:14.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.532+0000 7f57a6712640 1 -- 172.21.15.189:0/2362306898 shutdown_connections 2024-09-16T14:49:14.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.532+0000 7f57a6712640 1 -- 172.21.15.189:0/2362306898 >> 172.21.15.189:0/2362306898 conn(0x7f57a00a3320 msgr2=0x7f57a0070010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:14.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.532+0000 7f57a6712640 1 -- 172.21.15.189:0/2362306898 shutdown_connections 2024-09-16T14:49:14.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.532+0000 7f57a6712640 1 -- 172.21.15.189:0/2362306898 wait complete. 2024-09-16T14:49:14.550 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm barr 2024-09-16T14:49:14.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.664+0000 7fed68e48640 10 monclient: get_monmap_and_config 2024-09-16T14:49:14.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.664+0000 7fed68e48640 10 monclient: build_initial_monmap 2024-09-16T14:49:14.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.664+0000 7fed68e48640 10 monclient: monmap: 2024-09-16T14:49:14.668 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:14.668 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:14.668 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:14.668353+0000 2024-09-16T14:49:14.668 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:14.668353+0000 2024-09-16T14:49:14.668 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:14.668 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:14.668 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:14.668 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:14.668 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:14.669 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:14.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.664+0000 7fed68e48640 1 Processor -- start 2024-09-16T14:49:14.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 1 -- start start 2024-09-16T14:49:14.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient: init 2024-09-16T14:49:14.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:14.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:49:14.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient(hunting): picked mon.noname-b con 0x7fed640a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:14.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient(hunting): picked mon.noname-c con 0x7fed640a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:14.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient(hunting): picked mon.noname-a con 0x7fed6413b430 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:14.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed640a30b0 con 0x7fed6413b430 2024-09-16T14:49:14.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed640a2e30 con 0x7fed640a25f0 2024-09-16T14:49:14.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6409eb50 con 0x7fed640a1b00 2024-09-16T14:49:14.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:14.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient(hunting): authenticate will time out at 1336.784741s 2024-09-16T14:49:14.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed61d9b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fed640a25f0 0x7fed64139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46228/0 (socket says 172.21.15.189:46228) 2024-09-16T14:49:14.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed61d9b640 1 -- 172.21.15.189:0/1760340582 learned_addr learned my addr 172.21.15.189:0/1760340582 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:14.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed62d9d640 1 -- 172.21.15.189:0/1760340582 >> v1:172.21.15.26:6789/0 conn(0x7fed6413b430 legacy=0x7fed6413d820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:14.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 1 -- 172.21.15.189:0/1760340582 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980309778 0 0) 0x7fed640a2e30 con 0x7fed640a25f0 2024-09-16T14:49:14.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 10 monclient(hunting): _negotiate 2024-09-16T14:49:14.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:14.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:14.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 10 monclient(hunting): my global_id is 144427 2024-09-16T14:49:14.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 1 -- 172.21.15.189:0/1760340582 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed4c003530 con 0x7fed640a25f0 2024-09-16T14:49:14.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 1 -- 172.21.15.189:0/1760340582 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1060265666 0 0) 0x7fed4c003530 con 0x7fed640a25f0 2024-09-16T14:49:14.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 1 -- 172.21.15.189:0/1760340582 >> v1:172.21.15.26:6790/0 conn(0x7fed640a1b00 legacy=0x7fed640a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:14.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 1 -- 172.21.15.189:0/1760340582 >> v1:172.21.15.26:6789/0 conn(0x7fed6413b430 legacy=0x7fed6413d820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:14.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 10 monclient: _finish_hunting 0 2024-09-16T14:49:14.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 1 monclient: found mon.noname-c 2024-09-16T14:49:14.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:14.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:14.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 1 -- 172.21.15.189:0/1760340582 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed6409ee30 con 0x7fed640a25f0 2024-09-16T14:49:14.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 10 monclient: _finish_auth 0 2024-09-16T14:49:14.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 10 monclient: _check_auth_tickets 2024-09-16T14:49:14.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:14.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 1 -- 172.21.15.189:0/1760340582 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fed58001620 con 0x7fed640a25f0 2024-09-16T14:49:14.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:14.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:14.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 10 monclient: dump: 2024-09-16T14:49:14.674 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:14.674 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:14.674 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:14.674 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:14.674 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:14.674 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:14.674 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:14.674 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:14.675 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:14.675 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:14.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 5 monclient: authenticate success, global_id 144427 2024-09-16T14:49:14.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:14.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 1 -- 172.21.15.189:0/1760340582 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed58001940 con 0x7fed640a25f0 2024-09-16T14:49:14.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:14.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 1 -- 172.21.15.189:0/1760340582 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fed58001cd0 con 0x7fed640a25f0 2024-09-16T14:49:14.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:14.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:14.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed6159a640 10 monclient: dump: 2024-09-16T14:49:14.675 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:14.675 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:14.675 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:14.676 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:14.676 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:14.676 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:14.676 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:14.676 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:14.676 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:14.676 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:14.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:14.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 4 set_mon_vals no callback set 2024-09-16T14:49:14.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient: shutdown 2024-09-16T14:49:14.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:14.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 1 -- 172.21.15.189:0/1760340582 >> v1:172.21.15.189:6789/0 conn(0x7fed640a25f0 legacy=0x7fed64139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:14.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 1 -- 172.21.15.189:0/1760340582 shutdown_connections 2024-09-16T14:49:14.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 1 -- 172.21.15.189:0/1760340582 >> 172.21.15.189:0/1760340582 conn(0x7fed640a7dd0 msgr2=0x7fed640a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:14.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 1 -- 172.21.15.189:0/1760340582 shutdown_connections 2024-09-16T14:49:14.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 1 -- 172.21.15.189:0/1760340582 wait complete. 2024-09-16T14:49:14.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient: build_initial_monmap 2024-09-16T14:49:14.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient: monmap: 2024-09-16T14:49:14.680 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:14.680 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:14.680 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:14.681 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:14.681 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:14.681 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:14.681 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:14.681 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:14.681 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:14.681 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:14.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 1 Processor -- start 2024-09-16T14:49:14.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 1 -- start start 2024-09-16T14:49:14.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient: init 2024-09-16T14:49:14.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient(hunting): picked mon.noname-c con 0x7fed640a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient(hunting): picked mon.noname-a con 0x7fed640a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient(hunting): picked mon.noname-b con 0x7fed6413b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6411ff30 con 0x7fed640a25f0 2024-09-16T14:49:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed640a5aa0 con 0x7fed6413b430 2024-09-16T14:49:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6413ae90 con 0x7fed640a1b00 2024-09-16T14:49:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed68e48640 10 monclient(hunting): authenticate will time out at 1336.787737s 2024-09-16T14:49:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed61d9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fed640a25f0 legacy=0x7fed6413a040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed62d9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fed6413b430 0x7fed641d1600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46240/0 (socket says 172.21.15.189:46240) 2024-09-16T14:49:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.668+0000 7fed62d9d640 1 -- 172.21.15.189:0/646440858 learned_addr learned my addr 172.21.15.189:0/646440858 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689441344 0 0) 0x7fed640a5aa0 con 0x7fed6413b430 2024-09-16T14:49:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient(hunting): my global_id is 144430 2024-09-16T14:49:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed44003530 con 0x7fed6413b430 2024-09-16T14:49:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3555269610 0 0) 0x7fed44003530 con 0x7fed6413b430 2024-09-16T14:49:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed640a5aa0 con 0x7fed6413b430 2024-09-16T14:49:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fed5c004450 con 0x7fed6413b430 2024-09-16T14:49:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient(hunting): dump: 2024-09-16T14:49:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:14.684 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:14.684 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:14.684 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:14.684 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:14.684 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:14.684 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:14.684 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:14.684 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:14.684 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:14.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:14.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121064657 0 0) 0x7fed6413ae90 con 0x7fed640a1b00 2024-09-16T14:49:14.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:14.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:14.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:14.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient(hunting): my global_id is 154220 2024-09-16T14:49:14.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed44003530 con 0x7fed640a1b00 2024-09-16T14:49:14.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2715671129 0 0) 0x7fed640a5aa0 con 0x7fed6413b430 2024-09-16T14:49:14.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 >> v1:172.21.15.26:6790/0 conn(0x7fed640a1b00 legacy=0x7fed64139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:14.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 >> v1:172.21.15.26:6789/0 conn(0x7fed640a25f0 legacy=0x7fed6413a040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:14.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:14.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 1 monclient: found mon.b 2024-09-16T14:49:14.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:14.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:14.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed6409e2f0 con 0x7fed6413b430 2024-09-16T14:49:14.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient: _finish_auth 0 2024-09-16T14:49:14.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:14.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:14.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed68e48640 5 monclient: authenticate success, global_id 144430 2024-09-16T14:49:14.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed68e48640 10 monclient: _renew_subs 2024-09-16T14:49:14.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed68e48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:14.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed68e48640 1 -- 172.21.15.189:0/646440858 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed6411fc50 con 0x7fed6413b430 2024-09-16T14:49:14.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed68e48640 10 monclient: _renew_subs 2024-09-16T14:49:14.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed68e48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:14.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed68e48640 1 -- 172.21.15.189:0/646440858 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed64135860 con 0x7fed6413b430 2024-09-16T14:49:14.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed5c004700 con 0x7fed6413b430 2024-09-16T14:49:14.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:14.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fed5c004a90 con 0x7fed6413b430 2024-09-16T14:49:14.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:14.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:14.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed60d99640 4 set_mon_vals no callback set 2024-09-16T14:49:14.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 10 monclient: dump: 2024-09-16T14:49:14.687 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:14.687 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:14.687 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:14.687 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:14.687 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:14.687 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:14.687 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:14.687 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:14.687 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:14.687 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:14.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fed5c004bf0 con 0x7fed6413b430 2024-09-16T14:49:14.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed68e48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:14.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed68e48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:14.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed68e48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:14.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(178..178 src has 73..178) v4 ==== 6905+0+0 (unknown 514350220 0 0) 0x7fed5c044210 con 0x7fed6413b430 2024-09-16T14:49:14.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed68e48640 1 -- 172.21.15.189:0/646440858 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed48005050 con 0x7fed6413b430 2024-09-16T14:49:14.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.672+0000 7fed6259c640 10 monclient: get_auth_request con 0x7fed440361d0 auth_method 0 2024-09-16T14:49:14.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.676+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fed5c0053f0 con 0x7fed6413b430 2024-09-16T14:49:14.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.676+0000 7fed52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:14.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.676+0000 7fed52ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:14.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.840+0000 7fed68e48640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T14:49:14.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.840+0000 7fed68e48640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T14:49:14.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.840+0000 7fed68e48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:14.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:14.840+0000 7fed68e48640 1 -- 172.21.15.189:0/646440858 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "barr"} v 0) v1 -- 0x7fed48005300 con 0x7fed6413b430 2024-09-16T14:49:15.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.432+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "barr"}]=0 removed item id -19 name 'barr' from crush map v179) v1 ==== 122+0+0 (unknown 1908779493 0 0) 0x7fed5c012ec0 con 0x7fed6413b430 2024-09-16T14:49:15.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.432+0000 7fed52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T14:49:15.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.432+0000 7fed52ffd640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'barr' from crush map 2024-09-16T14:49:15.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.536+0000 7fed68e48640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T14:49:15.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.536+0000 7fed68e48640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T14:49:15.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.536+0000 7fed68e48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:15.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.536+0000 7fed68e48640 1 -- 172.21.15.189:0/646440858 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "barr"} v 0) v1 -- 0x7fed480023c0 con 0x7fed6413b430 2024-09-16T14:49:15.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.536+0000 7fed52ffd640 1 -- 172.21.15.189:0/646440858 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "barr"}]=0 device 'barr' does not appear in the crush map v179) v1 ==== 122+0+0 (unknown 974834810 0 0) 0x7fed5c014820 con 0x7fed6413b430 2024-09-16T14:49:15.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.536+0000 7fed52ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T14:49:15.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.536+0000 7fed52ffd640 10 monclient: _finish_command 3 = system:0 device 'barr' does not appear in the crush map 2024-09-16T14:49:15.541 INFO:tasks.workunit.client.0.smithi189.stderr:device 'barr' does not appear in the crush map 2024-09-16T14:49:15.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.540+0000 7fed68e48640 1 -- 172.21.15.189:0/646440858 >> v1:172.21.15.189:6812/3196161582 conn(0x7fed440361d0 legacy=0x7fed44038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:15.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.540+0000 7fed68e48640 10 monclient: shutdown 2024-09-16T14:49:15.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.540+0000 7fed68e48640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:15.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.540+0000 7fed68e48640 1 -- 172.21.15.189:0/646440858 >> v1:172.21.15.189:6789/0 conn(0x7fed6413b430 legacy=0x7fed641d1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:15.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.540+0000 7fed68e48640 1 -- 172.21.15.189:0/646440858 shutdown_connections 2024-09-16T14:49:15.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.540+0000 7fed68e48640 1 -- 172.21.15.189:0/646440858 >> 172.21.15.189:0/646440858 conn(0x7fed640a7dd0 msgr2=0x7fed6406b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:15.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.540+0000 7fed68e48640 1 -- 172.21.15.189:0/646440858 shutdown_connections 2024-09-16T14:49:15.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.540+0000 7fed68e48640 1 -- 172.21.15.189:0/646440858 wait complete. 2024-09-16T14:49:15.558 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set rm-compat 2024-09-16T14:49:15.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.672+0000 7f7981b41640 10 monclient: get_monmap_and_config 2024-09-16T14:49:15.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.672+0000 7f7981b41640 10 monclient: build_initial_monmap 2024-09-16T14:49:15.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.672+0000 7f7981b41640 10 monclient: monmap: 2024-09-16T14:49:15.676 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:15.676 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:15.676 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:15.676472+0000 2024-09-16T14:49:15.676 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:15.676472+0000 2024-09-16T14:49:15.676 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:15.676 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:15.676 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:15.676 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:15.676 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:15.677 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:15.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 1 Processor -- start 2024-09-16T14:49:15.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 1 -- start start 2024-09-16T14:49:15.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 10 monclient: init 2024-09-16T14:49:15.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:15.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:15.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 10 monclient(hunting): picked mon.noname-b con 0x7f797c09c420 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:15.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 10 monclient(hunting): picked mon.noname-a con 0x7f797c09cf10 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:15.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 10 monclient(hunting): picked mon.noname-c con 0x7f797c1392c0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:15.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f797c09d8f0 con 0x7f797c09cf10 2024-09-16T14:49:15.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f797c0a1360 con 0x7f797c1392c0 2024-09-16T14:49:15.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f797c0a15a0 con 0x7f797c09c420 2024-09-16T14:49:15.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:15.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 10 monclient(hunting): authenticate will time out at 1337.793089s 2024-09-16T14:49:15.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f797c1392c0 0x7f797c13b6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46246/0 (socket says 172.21.15.189:46246) 2024-09-16T14:49:15.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797bfff640 1 -- 172.21.15.189:0/1012516203 learned_addr learned my addr 172.21.15.189:0/1012516203 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:15.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797affd640 1 -- 172.21.15.189:0/1012516203 >> v1:172.21.15.26:6789/0 conn(0x7f797c09cf10 legacy=0x7f797c137ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:15.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 1 -- 172.21.15.189:0/1012516203 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717998950 0 0) 0x7f797c0a1360 con 0x7f797c1392c0 2024-09-16T14:49:15.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:15.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:15.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:15.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 10 monclient(hunting): my global_id is 144433 2024-09-16T14:49:15.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 1 -- 172.21.15.189:0/1012516203 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7964003530 con 0x7f797c1392c0 2024-09-16T14:49:15.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 1 -- 172.21.15.189:0/1012516203 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1312147255 0 0) 0x7f7964003530 con 0x7f797c1392c0 2024-09-16T14:49:15.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 1 -- 172.21.15.189:0/1012516203 >> v1:172.21.15.26:6790/0 conn(0x7f797c09c420 legacy=0x7f797c09c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:15.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 1 -- 172.21.15.189:0/1012516203 >> v1:172.21.15.26:6789/0 conn(0x7f797c09cf10 legacy=0x7f797c137ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:15.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:49:15.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 1 monclient: found mon.noname-c 2024-09-16T14:49:15.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:15.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:15.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 1 -- 172.21.15.189:0/1012516203 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797c09e4d0 con 0x7f797c1392c0 2024-09-16T14:49:15.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:49:15.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:15.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:15.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 1 -- 172.21.15.189:0/1012516203 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f79740019a0 con 0x7f797c1392c0 2024-09-16T14:49:15.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 5 monclient: authenticate success, global_id 144433 2024-09-16T14:49:15.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:15.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:15.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:15.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 10 monclient: dump: 2024-09-16T14:49:15.683 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:15.683 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:15.683 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:15.683 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:15.683 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:15.683 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:15.683 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:15.683 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:15.683 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:15.684 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:15.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:15.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 1 -- 172.21.15.189:0/1012516203 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7974001c50 con 0x7f797c1392c0 2024-09-16T14:49:15.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:15.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:15.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 4 set_mon_vals no callback set 2024-09-16T14:49:15.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 1 -- 172.21.15.189:0/1012516203 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7974002020 con 0x7f797c1392c0 2024-09-16T14:49:15.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:15.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:15.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f797a7fc640 10 monclient: dump: 2024-09-16T14:49:15.684 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:15.684 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:15.684 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:15.685 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:15.685 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:15.685 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:15.685 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:15.685 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:15.685 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:15.685 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:15.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 10 monclient: shutdown 2024-09-16T14:49:15.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:15.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 1 -- 172.21.15.189:0/1012516203 >> v1:172.21.15.189:6789/0 conn(0x7f797c1392c0 legacy=0x7f797c13b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:15.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 1 -- 172.21.15.189:0/1012516203 shutdown_connections 2024-09-16T14:49:15.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 1 -- 172.21.15.189:0/1012516203 >> 172.21.15.189:0/1012516203 conn(0x7f797c0a1a20 msgr2=0x7f797c0a1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:15.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 1 -- 172.21.15.189:0/1012516203 shutdown_connections 2024-09-16T14:49:15.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 1 -- 172.21.15.189:0/1012516203 wait complete. 2024-09-16T14:49:15.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 10 monclient: build_initial_monmap 2024-09-16T14:49:15.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 10 monclient: monmap: 2024-09-16T14:49:15.686 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:15.686 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:15.686 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:15.686 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:15.686 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:15.686 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:15.686 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:15.686 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:15.686 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:15.686 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:15.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 1 Processor -- start 2024-09-16T14:49:15.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 1 -- start start 2024-09-16T14:49:15.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.676+0000 7f7981b41640 10 monclient: init 2024-09-16T14:49:15.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:15.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:15.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 10 monclient(hunting): picked mon.noname-c con 0x7f797c09c420 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:15.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 10 monclient(hunting): picked mon.noname-a con 0x7f797c09cf10 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:15.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 10 monclient(hunting): picked mon.noname-b con 0x7f797c1392c0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:15.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f797c11f9a0 con 0x7f797c09cf10 2024-09-16T14:49:15.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f797c11fc80 con 0x7f797c1392c0 2024-09-16T14:49:15.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f797c11e4f0 con 0x7f797c09c420 2024-09-16T14:49:15.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:15.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 10 monclient(hunting): authenticate will time out at 1337.796124s 2024-09-16T14:49:15.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f797bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f797c1392c0 0x7f797c1d1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46250/0 (socket says 172.21.15.189:46250) 2024-09-16T14:49:15.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f797bfff640 1 -- 172.21.15.189:0/2136872445 learned_addr learned my addr 172.21.15.189:0/2136872445 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:15.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f797affd640 1 -- 172.21.15.189:0/2136872445 >> v1:172.21.15.26:6789/0 conn(0x7f797c09cf10 legacy=0x7f797c1ce6a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:15.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 1 -- 172.21.15.189:0/2136872445 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779792964 0 0) 0x7f797c11fc80 con 0x7f797c1392c0 2024-09-16T14:49:15.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:15.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:15.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:15.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 10 monclient(hunting): my global_id is 144436 2024-09-16T14:49:15.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 1 -- 172.21.15.189:0/2136872445 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7968003530 con 0x7f797c1392c0 2024-09-16T14:49:15.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 1 -- 172.21.15.189:0/2136872445 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3037260731 0 0) 0x7f7968003530 con 0x7f797c1392c0 2024-09-16T14:49:15.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 1 -- 172.21.15.189:0/2136872445 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f797c11fc80 con 0x7f797c1392c0 2024-09-16T14:49:15.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 1 -- 172.21.15.189:0/2136872445 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7974001f60 con 0x7f797c1392c0 2024-09-16T14:49:15.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:15.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:15.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 10 monclient(hunting): dump: 2024-09-16T14:49:15.693 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:15.694 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:15.694 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:15.694 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:15.694 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:15.694 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:15.694 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:15.694 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:15.694 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:15.694 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:15.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:15.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 1 -- 172.21.15.189:0/2136872445 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4275173232 0 0) 0x7f797c11fc80 con 0x7f797c1392c0 2024-09-16T14:49:15.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 1 -- 172.21.15.189:0/2136872445 >> v1:172.21.15.26:6790/0 conn(0x7f797c09c420 legacy=0x7f797c1c96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:15.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 1 -- 172.21.15.189:0/2136872445 >> v1:172.21.15.26:6789/0 conn(0x7f797c09cf10 legacy=0x7f797c1ce6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:15.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 10 monclient: _finish_hunting 0 2024-09-16T14:49:15.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 1 monclient: found mon.b 2024-09-16T14:49:15.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:15.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:15.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 1 -- 172.21.15.189:0/2136872445 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797c06c860 con 0x7f797c1392c0 2024-09-16T14:49:15.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 10 monclient: _finish_auth 0 2024-09-16T14:49:15.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 10 monclient: _check_auth_tickets 2024-09-16T14:49:15.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:15.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 5 monclient: authenticate success, global_id 144436 2024-09-16T14:49:15.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 10 monclient: _renew_subs 2024-09-16T14:49:15.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:15.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 1 -- 172.21.15.189:0/2136872445 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f797c11d920 con 0x7f797c1392c0 2024-09-16T14:49:15.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 10 monclient: _renew_subs 2024-09-16T14:49:15.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:15.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 1 -- 172.21.15.189:0/2136872445 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f797c11e270 con 0x7f797c1392c0 2024-09-16T14:49:15.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 1 -- 172.21.15.189:0/2136872445 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7974001540 con 0x7f797c1392c0 2024-09-16T14:49:15.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:15.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 1 -- 172.21.15.189:0/2136872445 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7974005120 con 0x7f797c1392c0 2024-09-16T14:49:15.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:15.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:15.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7979ffb640 4 set_mon_vals no callback set 2024-09-16T14:49:15.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 10 monclient: dump: 2024-09-16T14:49:15.696 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:15.696 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:15.696 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:15.696 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:15.696 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:15.697 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:15.697 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:15.697 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:15.697 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:15.697 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:15.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:15.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:15.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 1 -- 172.21.15.189:0/2136872445 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f797400fb90 con 0x7f797c1392c0 2024-09-16T14:49:15.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:15.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7981b41640 1 -- 172.21.15.189:0/2136872445 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f795c005050 con 0x7f797c1392c0 2024-09-16T14:49:15.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f7963fff640 1 -- 172.21.15.189:0/2136872445 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(179..179 src has 73..179) v4 ==== 6689+0+0 (unknown 1308562733 0 0) 0x7f79740442a0 con 0x7f797c1392c0 2024-09-16T14:49:15.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.680+0000 7f797b7fe640 10 monclient: get_auth_request con 0x7f7968036030 auth_method 0 2024-09-16T14:49:15.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.684+0000 7f7963fff640 1 -- 172.21.15.189:0/2136872445 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7974014a50 con 0x7f797c1392c0 2024-09-16T14:49:15.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.684+0000 7f7963fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:15.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.684+0000 7f7963fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:15.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.848+0000 7f7981b41640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:49:15.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.848+0000 7f7981b41640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:49:15.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.848+0000 7f7981b41640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:15.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:15.848+0000 7f7981b41640 1 -- 172.21.15.189:0/2136872445 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f795c005300 con 0x7f797c1392c0 2024-09-16T14:49:16.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:16.436+0000 7f7963fff640 1 -- 172.21.15.189:0/2136872445 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v180) v1 ==== 78+0+0 (unknown 3538827938 0 0) 0x7f79740130f0 con 0x7f797c1392c0 2024-09-16T14:49:16.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:16.436+0000 7f7963fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:49:16.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:16.436+0000 7f7963fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:16.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:16.544+0000 7f7981b41640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:49:16.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:16.544+0000 7f7981b41640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:49:16.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:16.544+0000 7f7981b41640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:16.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:16.544+0000 7f7981b41640 1 -- 172.21.15.189:0/2136872445 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f795c005850 con 0x7f797c1392c0 2024-09-16T14:49:17.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.444+0000 7f7963fff640 1 -- 172.21.15.189:0/2136872445 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v181) v1 ==== 78+0+0 (unknown 2469359751 0 0) 0x7f7974014870 con 0x7f797c1392c0 2024-09-16T14:49:17.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.444+0000 7f7963fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:49:17.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.444+0000 7f7963fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:49:17.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.448+0000 7f7981b41640 1 -- 172.21.15.189:0/2136872445 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7968036030 legacy=0x7f79680384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:17.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.448+0000 7f7981b41640 10 monclient: shutdown 2024-09-16T14:49:17.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.448+0000 7f7981b41640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:17.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.448+0000 7f7981b41640 1 -- 172.21.15.189:0/2136872445 >> v1:172.21.15.189:6789/0 conn(0x7f797c1392c0 legacy=0x7f797c1d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:17.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.448+0000 7f7981b41640 1 -- 172.21.15.189:0/2136872445 shutdown_connections 2024-09-16T14:49:17.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.448+0000 7f7981b41640 1 -- 172.21.15.189:0/2136872445 >> 172.21.15.189:0/2136872445 conn(0x7f797c0a1a20 msgr2=0x7f797c06c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:17.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.448+0000 7f7981b41640 1 -- 172.21.15.189:0/2136872445 shutdown_connections 2024-09-16T14:49:17.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.448+0000 7f7981b41640 1 -- 172.21.15.189:0/2136872445 wait complete. 2024-09-16T14:49:17.468 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set create-compat 2024-09-16T14:49:17.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.580+0000 7f19fb5ef640 10 monclient: get_monmap_and_config 2024-09-16T14:49:17.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.580+0000 7f19fb5ef640 10 monclient: build_initial_monmap 2024-09-16T14:49:17.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.580+0000 7f19fb5ef640 10 monclient: monmap: 2024-09-16T14:49:17.584 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:17.584 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:17.584 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:17.584207+0000 2024-09-16T14:49:17.584 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:17.584207+0000 2024-09-16T14:49:17.584 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:17.584 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:17.584 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:17.584 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:17.584 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:17.584 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:17.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.580+0000 7f19fb5ef640 1 Processor -- start 2024-09-16T14:49:17.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.580+0000 7f19fb5ef640 1 -- start start 2024-09-16T14:49:17.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.580+0000 7f19fb5ef640 10 monclient: init 2024-09-16T14:49:17.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:17.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:49:17.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient(hunting): picked mon.noname-c con 0x7f19f409df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:17.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient(hunting): picked mon.noname-b con 0x7f19f409e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:17.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient(hunting): picked mon.noname-a con 0x7f19f409e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:17.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f409ee20 con 0x7f19f409e740 2024-09-16T14:49:17.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f409f100 con 0x7f19f409df10 2024-09-16T14:49:17.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f409f3e0 con 0x7f19f409e2f0 2024-09-16T14:49:17.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:17.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient(hunting): authenticate will time out at 1339.700267s 2024-09-16T14:49:17.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19f9b8c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f19f409e740 legacy=0x7f19f413b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:17.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19f938b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f19f409df10 0x7f19f409d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46254/0 (socket says 172.21.15.189:46254) 2024-09-16T14:49:17.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19f938b640 1 -- 172.21.15.189:0/4134082452 learned_addr learned my addr 172.21.15.189:0/4134082452 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:17.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 1 -- 172.21.15.189:0/4134082452 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403422785 0 0) 0x7f19f409f100 con 0x7f19f409df10 2024-09-16T14:49:17.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:17.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:17.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:17.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 10 monclient(hunting): my global_id is 144439 2024-09-16T14:49:17.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 1 -- 172.21.15.189:0/4134082452 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e0003530 con 0x7f19f409df10 2024-09-16T14:49:17.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 1 -- 172.21.15.189:0/4134082452 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2513339217 0 0) 0x7f19e0003530 con 0x7f19f409df10 2024-09-16T14:49:17.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 1 -- 172.21.15.189:0/4134082452 >> v1:172.21.15.26:6790/0 conn(0x7f19f409e2f0 legacy=0x7f19f41378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:17.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 1 -- 172.21.15.189:0/4134082452 >> v1:172.21.15.26:6789/0 conn(0x7f19f409e740 legacy=0x7f19f413b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:17.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 10 monclient: _finish_hunting 0 2024-09-16T14:49:17.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 1 monclient: found mon.noname-c 2024-09-16T14:49:17.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:17.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:17.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 1 -- 172.21.15.189:0/4134082452 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f409c690 con 0x7f19f409df10 2024-09-16T14:49:17.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 10 monclient: _finish_auth 0 2024-09-16T14:49:17.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 10 monclient: _check_auth_tickets 2024-09-16T14:49:17.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:17.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 1 -- 172.21.15.189:0/4134082452 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19ec0019a0 con 0x7f19f409df10 2024-09-16T14:49:17.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 5 monclient: authenticate success, global_id 144439 2024-09-16T14:49:17.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:17.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:17.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:17.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 10 monclient: dump: 2024-09-16T14:49:17.594 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:17.594 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:17.594 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:17.594 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:17.594 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:17.595 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 1 -- 172.21.15.189:0/4134082452 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19ec001cc0 con 0x7f19f409df10 2024-09-16T14:49:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 1 -- 172.21.15.189:0/4134082452 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19ec002050 con 0x7f19f409df10 2024-09-16T14:49:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 4 set_mon_vals no callback set 2024-09-16T14:49:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19ebfff640 10 monclient: dump: 2024-09-16T14:49:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:17.596 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient: shutdown 2024-09-16T14:49:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 1 -- 172.21.15.189:0/4134082452 >> v1:172.21.15.189:6789/0 conn(0x7f19f409df10 legacy=0x7f19f409d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 1 -- 172.21.15.189:0/4134082452 shutdown_connections 2024-09-16T14:49:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 1 -- 172.21.15.189:0/4134082452 >> 172.21.15.189:0/4134082452 conn(0x7f19f40a3320 msgr2=0x7f19f409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 1 -- 172.21.15.189:0/4134082452 shutdown_connections 2024-09-16T14:49:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 1 -- 172.21.15.189:0/4134082452 wait complete. 2024-09-16T14:49:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient: build_initial_monmap 2024-09-16T14:49:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient: monmap: 2024-09-16T14:49:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:17.598 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 1 Processor -- start 2024-09-16T14:49:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 1 -- start start 2024-09-16T14:49:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient: init 2024-09-16T14:49:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient(hunting): picked mon.noname-c con 0x7f19f409df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient(hunting): picked mon.noname-a con 0x7f19f409e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient(hunting): picked mon.noname-b con 0x7f19f409e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f406f970 con 0x7f19f409e2f0 2024-09-16T14:49:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f406b380 con 0x7f19f409e740 2024-09-16T14:49:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f406cea0 con 0x7f19f409df10 2024-09-16T14:49:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 10 monclient(hunting): authenticate will time out at 1339.703225s 2024-09-16T14:49:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19f8b8a640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f19f409e2f0 legacy=0x7f19f41cbfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19f9b8c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f19f409e740 0x7f19f41cf6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46266/0 (socket says 172.21.15.189:46266) 2024-09-16T14:49:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19f9b8c640 1 -- 172.21.15.189:0/122469111 learned_addr learned my addr 172.21.15.189:0/122469111 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 1 -- 172.21.15.189:0/122469111 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730505129 0 0) 0x7f19f406b380 con 0x7f19f409e740 2024-09-16T14:49:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 10 monclient(hunting): my global_id is 144442 2024-09-16T14:49:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 1 -- 172.21.15.189:0/122469111 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d8003530 con 0x7f19f409e740 2024-09-16T14:49:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 1 -- 172.21.15.189:0/122469111 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2046720585 0 0) 0x7f19d8003530 con 0x7f19f409e740 2024-09-16T14:49:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 1 -- 172.21.15.189:0/122469111 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19f406b380 con 0x7f19f409e740 2024-09-16T14:49:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 1 -- 172.21.15.189:0/122469111 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19f0004080 con 0x7f19f409e740 2024-09-16T14:49:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 10 monclient(hunting): dump: 2024-09-16T14:49:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:17.601 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 1 -- 172.21.15.189:0/122469111 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3177590278 0 0) 0x7f19f406b380 con 0x7f19f409e740 2024-09-16T14:49:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 1 -- 172.21.15.189:0/122469111 >> v1:172.21.15.26:6790/0 conn(0x7f19f409df10 legacy=0x7f19f4132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 1 -- 172.21.15.189:0/122469111 >> v1:172.21.15.26:6789/0 conn(0x7f19f409e2f0 legacy=0x7f19f41cbfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:49:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 1 monclient: found mon.b 2024-09-16T14:49:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 1 -- 172.21.15.189:0/122469111 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f411a490 con 0x7f19f409e740 2024-09-16T14:49:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 10 monclient: _finish_auth 0 2024-09-16T14:49:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:49:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.584+0000 7f19fb5ef640 5 monclient: authenticate success, global_id 144442 2024-09-16T14:49:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19fb5ef640 10 monclient: _renew_subs 2024-09-16T14:49:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19fb5ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19fb5ef640 1 -- 172.21.15.189:0/122469111 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f413b460 con 0x7f19f409e740 2024-09-16T14:49:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19fb5ef640 10 monclient: _renew_subs 2024-09-16T14:49:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19fb5ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19fb5ef640 1 -- 172.21.15.189:0/122469111 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f413c250 con 0x7f19f409e740 2024-09-16T14:49:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19e9ffb640 1 -- 172.21.15.189:0/122469111 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19f00043a0 con 0x7f19f409e740 2024-09-16T14:49:17.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19e9ffb640 1 -- 172.21.15.189:0/122469111 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19f0004730 con 0x7f19f409e740 2024-09-16T14:49:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19e9ffb640 10 monclient: dump: 2024-09-16T14:49:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:17.603 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19eb7fe640 4 set_mon_vals no callback set 2024-09-16T14:49:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19fb5ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19fb5ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19fb5ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19fb5ef640 1 -- 172.21.15.189:0/122469111 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d4005050 con 0x7f19f409e740 2024-09-16T14:49:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19e9ffb640 1 -- 172.21.15.189:0/122469111 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f19f0004890 con 0x7f19f409e740 2024-09-16T14:49:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19e9ffb640 1 -- 172.21.15.189:0/122469111 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(181..181 src has 154..181) v4 ==== 6361+0+0 (unknown 1583041975 0 0) 0x7f19f0043b60 con 0x7f19f409e740 2024-09-16T14:49:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.588+0000 7f19f938b640 10 monclient: get_auth_request con 0x7f19d8035fe0 auth_method 0 2024-09-16T14:49:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.592+0000 7f19e9ffb640 1 -- 172.21.15.189:0/122469111 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f19f0005090 con 0x7f19f409e740 2024-09-16T14:49:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.592+0000 7f19e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.592+0000 7f19e9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:17.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.744+0000 7f19fb5ef640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:49:17.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.744+0000 7f19fb5ef640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:49:17.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.744+0000 7f19fb5ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:17.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:17.744+0000 7f19fb5ef640 1 -- 172.21.15.189:0/122469111 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f19d4005300 con 0x7f19f409e740 2024-09-16T14:49:18.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.448+0000 7f19e9ffb640 1 -- 172.21.15.189:0/122469111 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v182) v1 ==== 82+0+0 (unknown 328874346 0 0) 0x7f19f0012af0 con 0x7f19f409e740 2024-09-16T14:49:18.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.448+0000 7f19e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:49:18.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.448+0000 7f19e9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:18.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.552+0000 7f19fb5ef640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:49:18.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.552+0000 7f19fb5ef640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:49:18.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.552+0000 7f19fb5ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:18.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.552+0000 7f19fb5ef640 1 -- 172.21.15.189:0/122469111 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f19d4005850 con 0x7f19f409e740 2024-09-16T14:49:18.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.552+0000 7f19e9ffb640 1 -- 172.21.15.189:0/122469111 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v182) v1 ==== 115+0+0 (unknown 2606304275 0 0) 0x7f19f0014270 con 0x7f19f409e740 2024-09-16T14:49:18.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.552+0000 7f19e9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T14:49:18.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.552+0000 7f19e9ffb640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-16T14:49:18.556 INFO:tasks.workunit.client.0.smithi189.stderr:compat weight-set already created 2024-09-16T14:49:18.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.556+0000 7f19fb5ef640 1 -- 172.21.15.189:0/122469111 >> v1:172.21.15.189:6812/3196161582 conn(0x7f19d8035fe0 legacy=0x7f19d80384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:18.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.556+0000 7f19fb5ef640 10 monclient: shutdown 2024-09-16T14:49:18.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.556+0000 7f19fb5ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:18.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.556+0000 7f19fb5ef640 1 -- 172.21.15.189:0/122469111 >> v1:172.21.15.189:6789/0 conn(0x7f19f409e740 legacy=0x7f19f41cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:18.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.556+0000 7f19fb5ef640 1 -- 172.21.15.189:0/122469111 shutdown_connections 2024-09-16T14:49:18.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.556+0000 7f19fb5ef640 1 -- 172.21.15.189:0/122469111 >> 172.21.15.189:0/122469111 conn(0x7f19f40a3320 msgr2=0x7f19f406e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:18.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.556+0000 7f19fb5ef640 1 -- 172.21.15.189:0/122469111 shutdown_connections 2024-09-16T14:49:18.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.556+0000 7f19fb5ef640 1 -- 172.21.15.189:0/122469111 wait complete. 2024-09-16T14:49:18.574 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket r1 rack root=default 2024-09-16T14:49:18.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.688+0000 7f58fb29a640 10 monclient: get_monmap_and_config 2024-09-16T14:49:18.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.688+0000 7f58fb29a640 10 monclient: build_initial_monmap 2024-09-16T14:49:18.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 10 monclient: monmap: 2024-09-16T14:49:18.693 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:18.693 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:18.693 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:18.693770+0000 2024-09-16T14:49:18.693 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:18.693770+0000 2024-09-16T14:49:18.693 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:18.694 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:18.694 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:18.694 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:18.694 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:18.694 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:18.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 1 Processor -- start 2024-09-16T14:49:18.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 1 -- start start 2024-09-16T14:49:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 10 monclient: init 2024-09-16T14:49:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:49:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 10 monclient(hunting): picked mon.noname-b con 0x7f58f4134c40 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 10 monclient(hunting): picked mon.noname-c con 0x7f58f40714d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 10 monclient(hunting): picked mon.noname-a con 0x7f58f413b0b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f4071220 con 0x7f58f413b0b0 2024-09-16T14:49:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f4099500 con 0x7f58f40714d0 2024-09-16T14:49:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f409c9e0 con 0x7f58f4134c40 2024-09-16T14:49:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 10 monclient(hunting): authenticate will time out at 1340.809381s 2024-09-16T14:49:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58f9837640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f58f413b0b0 legacy=0x7f58f413d4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58f8835640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f58f40714d0 0x7f58f4139990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46272/0 (socket says 172.21.15.189:46272) 2024-09-16T14:49:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58f8835640 1 -- 172.21.15.189:0/880366870 learned_addr learned my addr 172.21.15.189:0/880366870 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 1 -- 172.21.15.189:0/880366870 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912932035 0 0) 0x7f58f4099500 con 0x7f58f40714d0 2024-09-16T14:49:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 10 monclient(hunting): my global_id is 144445 2024-09-16T14:49:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 1 -- 172.21.15.189:0/880366870 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58e0003530 con 0x7f58f40714d0 2024-09-16T14:49:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 1 -- 172.21.15.189:0/880366870 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 394288535 0 0) 0x7f58e0003530 con 0x7f58f40714d0 2024-09-16T14:49:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 1 -- 172.21.15.189:0/880366870 >> v1:172.21.15.26:6790/0 conn(0x7f58f4134c40 legacy=0x7f58f4135020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 1 -- 172.21.15.189:0/880366870 >> v1:172.21.15.26:6789/0 conn(0x7f58f413b0b0 legacy=0x7f58f413d4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 10 monclient: _finish_hunting 0 2024-09-16T14:49:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 1 monclient: found mon.noname-c 2024-09-16T14:49:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 1 -- 172.21.15.189:0/880366870 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f4099220 con 0x7f58f40714d0 2024-09-16T14:49:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 10 monclient: _finish_auth 0 2024-09-16T14:49:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 10 monclient: _check_auth_tickets 2024-09-16T14:49:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 1 -- 172.21.15.189:0/880366870 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58e4001620 con 0x7f58f40714d0 2024-09-16T14:49:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 10 monclient: dump: 2024-09-16T14:49:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:18.704 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 5 monclient: authenticate success, global_id 144445 2024-09-16T14:49:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 1 -- 172.21.15.189:0/880366870 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58e4001940 con 0x7f58f40714d0 2024-09-16T14:49:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 4 set_mon_vals no callback set 2024-09-16T14:49:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 1 -- 172.21.15.189:0/880366870 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58e4001d10 con 0x7f58f40714d0 2024-09-16T14:49:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58ebfff640 10 monclient: dump: 2024-09-16T14:49:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:18.706 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:18.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 10 monclient: shutdown 2024-09-16T14:49:18.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:18.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 1 -- 172.21.15.189:0/880366870 >> v1:172.21.15.189:6789/0 conn(0x7f58f40714d0 legacy=0x7f58f4139990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:18.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 1 -- 172.21.15.189:0/880366870 shutdown_connections 2024-09-16T14:49:18.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 1 -- 172.21.15.189:0/880366870 >> 172.21.15.189:0/880366870 conn(0x7f58f40a3280 msgr2=0x7f58f409f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:18.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 1 -- 172.21.15.189:0/880366870 shutdown_connections 2024-09-16T14:49:18.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 1 -- 172.21.15.189:0/880366870 wait complete. 2024-09-16T14:49:18.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 10 monclient: build_initial_monmap 2024-09-16T14:49:18.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 10 monclient: monmap: 2024-09-16T14:49:18.706 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:18.706 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:18.706 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:18.707 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:18.707 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:18.707 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:18.707 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:18.707 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:18.707 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:18.707 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:18.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 1 Processor -- start 2024-09-16T14:49:18.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 1 -- start start 2024-09-16T14:49:18.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.692+0000 7f58fb29a640 10 monclient: init 2024-09-16T14:49:18.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:18.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:18.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 10 monclient(hunting): picked mon.noname-a con 0x7f58f40714d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:18.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 10 monclient(hunting): picked mon.noname-b con 0x7f58f4134c40 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:18.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 10 monclient(hunting): picked mon.noname-c con 0x7f58f413b0b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:18.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f411ca90 con 0x7f58f40714d0 2024-09-16T14:49:18.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f409baa0 con 0x7f58f4134c40 2024-09-16T14:49:18.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f409edc0 con 0x7f58f413b0b0 2024-09-16T14:49:18.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:18.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 10 monclient(hunting): authenticate will time out at 1340.812321s 2024-09-16T14:49:18.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58f8835640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f58f4134c40 0x7f58f41ce680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46278/0 (socket says 172.21.15.189:46278) 2024-09-16T14:49:18.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58f8835640 1 -- 172.21.15.189:0/1235366030 learned_addr learned my addr 172.21.15.189:0/1235366030 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:18.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58f9036640 1 -- 172.21.15.189:0/1235366030 >> v1:172.21.15.26:6789/0 conn(0x7f58f40714d0 legacy=0x7f58f4130ea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:18.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 1 -- 172.21.15.189:0/1235366030 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266911729 0 0) 0x7f58f409baa0 con 0x7f58f4134c40 2024-09-16T14:49:18.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:18.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:18.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 10 monclient(hunting): my global_id is 144448 2024-09-16T14:49:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 1 -- 172.21.15.189:0/1235366030 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d8003530 con 0x7f58f4134c40 2024-09-16T14:49:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 1 -- 172.21.15.189:0/1235366030 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1598154880 0 0) 0x7f58d8003530 con 0x7f58f4134c40 2024-09-16T14:49:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 1 -- 172.21.15.189:0/1235366030 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58f409baa0 con 0x7f58f4134c40 2024-09-16T14:49:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 1 -- 172.21.15.189:0/1235366030 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58e4001620 con 0x7f58f4134c40 2024-09-16T14:49:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 10 monclient(hunting): dump: 2024-09-16T14:49:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:18.710 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:18.710 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:18.710 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:18.710 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:18.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:18.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 1 -- 172.21.15.189:0/1235366030 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 290730165 0 0) 0x7f58f409baa0 con 0x7f58f4134c40 2024-09-16T14:49:18.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 1 -- 172.21.15.189:0/1235366030 >> v1:172.21.15.26:6790/0 conn(0x7f58f413b0b0 legacy=0x7f58f41d1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:18.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 1 -- 172.21.15.189:0/1235366030 >> v1:172.21.15.26:6789/0 conn(0x7f58f40714d0 legacy=0x7f58f4130ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:18.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:49:18.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 1 monclient: found mon.b 2024-09-16T14:49:18.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:18.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:18.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 1 -- 172.21.15.189:0/1235366030 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f411a9a0 con 0x7f58f4134c40 2024-09-16T14:49:18.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 10 monclient: _finish_auth 0 2024-09-16T14:49:18.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:49:18.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:18.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 5 monclient: authenticate success, global_id 144448 2024-09-16T14:49:18.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 10 monclient: _renew_subs 2024-09-16T14:49:18.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:18.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 1 -- 172.21.15.189:0/1235366030 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f406b1f0 con 0x7f58f4134c40 2024-09-16T14:49:18.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 10 monclient: _renew_subs 2024-09-16T14:49:18.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:18.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 1 -- 172.21.15.189:0/1235366030 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f411b2f0 con 0x7f58f4134c40 2024-09-16T14:49:18.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 1 -- 172.21.15.189:0/1235366030 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58e4001200 con 0x7f58f4134c40 2024-09-16T14:49:18.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:18.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 1 -- 172.21.15.189:0/1235366030 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58e4004da0 con 0x7f58f4134c40 2024-09-16T14:49:18.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:18.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:18.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58eb7fe640 4 set_mon_vals no callback set 2024-09-16T14:49:18.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 10 monclient: dump: 2024-09-16T14:49:18.712 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:18.712 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:18.712 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:18.712 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:18.712 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:18.712 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:18.712 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:18.712 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:18.713 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:18.713 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:18.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:18.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 1 -- 172.21.15.189:0/1235366030 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f58e4004f00 con 0x7f58f4134c40 2024-09-16T14:49:18.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:18.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:18.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58fb29a640 1 -- 172.21.15.189:0/1235366030 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d4005050 con 0x7f58f4134c40 2024-09-16T14:49:18.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58e9ffb640 1 -- 172.21.15.189:0/1235366030 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(182..182 src has 154..182) v4 ==== 6689+0+0 (unknown 502034059 0 0) 0x7f58e4043fd0 con 0x7f58f4134c40 2024-09-16T14:49:18.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.696+0000 7f58f9036640 10 monclient: get_auth_request con 0x7f58d8036050 auth_method 0 2024-09-16T14:49:18.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.700+0000 7f58e9ffb640 1 -- 172.21.15.189:0/1235366030 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f58e4006520 con 0x7f58f4134c40 2024-09-16T14:49:18.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.700+0000 7f58e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:18.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.700+0000 7f58e9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:18.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.856+0000 7f58fb29a640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T14:49:18.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.856+0000 7f58fb29a640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T14:49:18.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.856+0000 7f58fb29a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:18.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:18.856+0000 7f58fb29a640 1 -- 172.21.15.189:0/1235366030 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]} v 0) v1 -- 0x7f58d4005880 con 0x7f58f4134c40 2024-09-16T14:49:19.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.464+0000 7f58e9ffb640 1 -- 172.21.15.189:0/1235366030 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}]=0 added bucket r1 type rack to location {root=default} v183) v1 ==== 176+0+0 (unknown 2894638766 0 0) 0x7f58e4012ed0 con 0x7f58f4134c40 2024-09-16T14:49:19.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.464+0000 7f58e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T14:49:19.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.464+0000 7f58e9ffb640 10 monclient: _finish_command 2 = system:0 added bucket r1 type rack to location {root=default} 2024-09-16T14:49:19.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.556+0000 7f58fb29a640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T14:49:19.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.560+0000 7f58fb29a640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T14:49:19.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.560+0000 7f58fb29a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:19.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.560+0000 7f58fb29a640 1 -- 172.21.15.189:0/1235366030 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]} v 0) v1 -- 0x7f58d4005300 con 0x7f58f4134c40 2024-09-16T14:49:19.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.560+0000 7f58e9ffb640 1 -- 172.21.15.189:0/1235366030 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}]=0 bucket 'r1' already exists v183) v1 ==== 150+0+0 (unknown 3574563992 0 0) 0x7f58e4014650 con 0x7f58f4134c40 2024-09-16T14:49:19.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.560+0000 7f58e9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T14:49:19.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.560+0000 7f58e9ffb640 10 monclient: _finish_command 3 = system:0 bucket 'r1' already exists 2024-09-16T14:49:19.563 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'r1' already exists 2024-09-16T14:49:19.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.564+0000 7f58fb29a640 1 -- 172.21.15.189:0/1235366030 >> v1:172.21.15.189:6812/3196161582 conn(0x7f58d8036050 legacy=0x7f58d8038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:19.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.564+0000 7f58fb29a640 10 monclient: shutdown 2024-09-16T14:49:19.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.564+0000 7f58fb29a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:19.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.564+0000 7f58fb29a640 1 -- 172.21.15.189:0/1235366030 >> v1:172.21.15.189:6789/0 conn(0x7f58f4134c40 legacy=0x7f58f41ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:19.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.564+0000 7f58fb29a640 1 -- 172.21.15.189:0/1235366030 shutdown_connections 2024-09-16T14:49:19.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.564+0000 7f58fb29a640 1 -- 172.21.15.189:0/1235366030 >> 172.21.15.189:0/1235366030 conn(0x7f58f40a3280 msgr2=0x7f58f4099f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:19.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.564+0000 7f58fb29a640 1 -- 172.21.15.189:0/1235366030 shutdown_connections 2024-09-16T14:49:19.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.564+0000 7f58fb29a640 1 -- 172.21.15.189:0/1235366030 wait complete. 2024-09-16T14:49:19.580 INFO:tasks.workunit.client.0.smithi189.stderr:++ seq 1 32 2024-09-16T14:49:19.591 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:19.591 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h1 host rack=r1 2024-09-16T14:49:19.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 10 monclient: get_monmap_and_config 2024-09-16T14:49:19.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 10 monclient: build_initial_monmap 2024-09-16T14:49:19.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 10 monclient: monmap: 2024-09-16T14:49:19.709 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:19.709 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:19.709 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:19.709852+0000 2024-09-16T14:49:19.709 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:19.709852+0000 2024-09-16T14:49:19.710 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:19.710 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:19.710 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:19.710 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:19.710 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:19.710 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:19.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 1 Processor -- start 2024-09-16T14:49:19.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 1 -- start start 2024-09-16T14:49:19.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 10 monclient: init 2024-09-16T14:49:19.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:19.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:19.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 10 monclient(hunting): picked mon.noname-a con 0x7fd5b80a1a90 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:19.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 10 monclient(hunting): picked mon.noname-c con 0x7fd5b80a2580 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:19.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 10 monclient(hunting): picked mon.noname-b con 0x7fd5b809fd40 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:19.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b809df00 con 0x7fd5b80a1a90 2024-09-16T14:49:19.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b809f0a0 con 0x7fd5b80a2580 2024-09-16T14:49:19.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b80a3040 con 0x7fd5b809fd40 2024-09-16T14:49:19.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:19.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 10 monclient(hunting): authenticate will time out at 1341.825636s 2024-09-16T14:49:19.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b6ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd5b80a2580 0x7fd5b8139df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53320/0 (socket says 172.21.15.189:53320) 2024-09-16T14:49:19.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b6ffd640 1 -- 172.21.15.189:0/3177790681 learned_addr learned my addr 172.21.15.189:0/3177790681 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:19.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b77fe640 1 -- 172.21.15.189:0/3177790681 >> v1:172.21.15.26:6789/0 conn(0x7fd5b80a1a90 legacy=0x7fd5b80a1e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:19.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 1 -- 172.21.15.189:0/3177790681 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520574427 0 0) 0x7fd5b809f0a0 con 0x7fd5b80a2580 2024-09-16T14:49:19.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:19.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:19.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:19.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 10 monclient(hunting): my global_id is 144451 2024-09-16T14:49:19.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 1 -- 172.21.15.189:0/3177790681 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a4003530 con 0x7fd5b80a2580 2024-09-16T14:49:19.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 1 -- 172.21.15.189:0/3177790681 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2885248526 0 0) 0x7fd5a4003530 con 0x7fd5b80a2580 2024-09-16T14:49:19.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 1 -- 172.21.15.189:0/3177790681 >> v1:172.21.15.26:6790/0 conn(0x7fd5b809fd40 legacy=0x7fd5b813d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:19.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 1 -- 172.21.15.189:0/3177790681 >> v1:172.21.15.26:6789/0 conn(0x7fd5b80a1a90 legacy=0x7fd5b80a1e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:19.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:49:19.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 1 monclient: found mon.noname-c 2024-09-16T14:49:19.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:19.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:19.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 1 -- 172.21.15.189:0/3177790681 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b809eae0 con 0x7fd5b80a2580 2024-09-16T14:49:19.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 10 monclient: _finish_auth 0 2024-09-16T14:49:19.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:19.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:19.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 1 -- 172.21.15.189:0/3177790681 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd5a80019a0 con 0x7fd5b80a2580 2024-09-16T14:49:19.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 5 monclient: authenticate success, global_id 144451 2024-09-16T14:49:19.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:19.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:19.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:19.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 10 monclient: dump: 2024-09-16T14:49:19.721 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:19.721 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:19.721 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:19.721 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:19.721 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:19.721 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:19.721 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:19.721 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:19.721 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:19.722 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:19.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:19.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 1 -- 172.21.15.189:0/3177790681 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5a8001c50 con 0x7fd5b80a2580 2024-09-16T14:49:19.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:19.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:19.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 1 -- 172.21.15.189:0/3177790681 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd5a8001fe0 con 0x7fd5b80a2580 2024-09-16T14:49:19.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:19.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 4 set_mon_vals no callback set 2024-09-16T14:49:19.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:19.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5b67fc640 10 monclient: dump: 2024-09-16T14:49:19.722 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:19.722 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:19.722 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:19.723 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:19.723 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:19.723 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:19.723 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:19.723 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:19.724 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:19.724 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:19.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 10 monclient: shutdown 2024-09-16T14:49:19.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:19.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 1 -- 172.21.15.189:0/3177790681 >> v1:172.21.15.189:6789/0 conn(0x7fd5b80a2580 legacy=0x7fd5b8139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:19.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 1 -- 172.21.15.189:0/3177790681 shutdown_connections 2024-09-16T14:49:19.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 1 -- 172.21.15.189:0/3177790681 >> 172.21.15.189:0/3177790681 conn(0x7fd5b80a7a70 msgr2=0x7fd5b80a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:19.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 1 -- 172.21.15.189:0/3177790681 shutdown_connections 2024-09-16T14:49:19.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 1 -- 172.21.15.189:0/3177790681 wait complete. 2024-09-16T14:49:19.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 10 monclient: build_initial_monmap 2024-09-16T14:49:19.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 10 monclient: monmap: 2024-09-16T14:49:19.725 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:19.725 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:19.725 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:19.725 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:19.726 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:19.726 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:19.726 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:19.726 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:19.726 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:19.726 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:19.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.708+0000 7fd5bdffb640 1 Processor -- start 2024-09-16T14:49:19.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 1 -- start start 2024-09-16T14:49:19.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 10 monclient: init 2024-09-16T14:49:19.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:19.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:19.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 10 monclient(hunting): picked mon.noname-b con 0x7fd5b809fd40 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:19.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 10 monclient(hunting): picked mon.noname-a con 0x7fd5b80a1a90 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:19.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 10 monclient(hunting): picked mon.noname-c con 0x7fd5b80a2580 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:19.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b8120b00 con 0x7fd5b80a1a90 2024-09-16T14:49:19.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b81c9e30 con 0x7fd5b809fd40 2024-09-16T14:49:19.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b809e280 con 0x7fd5b80a2580 2024-09-16T14:49:19.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:19.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 10 monclient(hunting): authenticate will time out at 1341.828538s 2024-09-16T14:49:19.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5b77fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd5b809fd40 0x7fd5b8133b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53326/0 (socket says 172.21.15.189:53326) 2024-09-16T14:49:19.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5b77fe640 1 -- 172.21.15.189:0/3928447316 learned_addr learned my addr 172.21.15.189:0/3928447316 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:19.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5b6ffd640 1 -- 172.21.15.189:0/3928447316 >> v1:172.21.15.26:6789/0 conn(0x7fd5b80a1a90 legacy=0x7fd5b81ce200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:19.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 1 -- 172.21.15.189:0/3928447316 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786523072 0 0) 0x7fd5a0003a90 con 0x7fd5b809fd40 2024-09-16T14:49:19.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:19.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:19.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:19.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 10 monclient(hunting): my global_id is 144454 2024-09-16T14:49:19.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 1 -- 172.21.15.189:0/3928447316 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd598003530 con 0x7fd5b809fd40 2024-09-16T14:49:19.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 1 -- 172.21.15.189:0/3928447316 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 425883965 0 0) 0x7fd598003530 con 0x7fd5b809fd40 2024-09-16T14:49:19.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 1 -- 172.21.15.189:0/3928447316 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5a0003a90 con 0x7fd5b809fd40 2024-09-16T14:49:19.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 1 -- 172.21.15.189:0/3928447316 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd5a0004180 con 0x7fd5b809fd40 2024-09-16T14:49:19.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:19.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:19.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 10 monclient(hunting): dump: 2024-09-16T14:49:19.729 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:19.730 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:19.730 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:19.730 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:19.730 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:19.730 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:19.730 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:19.731 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:19.731 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:19.731 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:19.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:19.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 1 -- 172.21.15.189:0/3928447316 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 70422529 0 0) 0x7fd5a0003a90 con 0x7fd5b809fd40 2024-09-16T14:49:19.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 1 -- 172.21.15.189:0/3928447316 >> v1:172.21.15.26:6790/0 conn(0x7fd5b80a2580 legacy=0x7fd5b81d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:19.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 1 -- 172.21.15.189:0/3928447316 >> v1:172.21.15.26:6789/0 conn(0x7fd5b80a1a90 legacy=0x7fd5b81ce200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:19.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 10 monclient: _finish_hunting 0 2024-09-16T14:49:19.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 1 monclient: found mon.b 2024-09-16T14:49:19.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:19.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:19.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 1 -- 172.21.15.189:0/3928447316 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b80a6390 con 0x7fd5b809fd40 2024-09-16T14:49:19.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 10 monclient: _finish_auth 0 2024-09-16T14:49:19.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 10 monclient: _check_auth_tickets 2024-09-16T14:49:19.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:19.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 5 monclient: authenticate success, global_id 144454 2024-09-16T14:49:19.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 10 monclient: _renew_subs 2024-09-16T14:49:19.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:19.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 1 -- 172.21.15.189:0/3928447316 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b81224d0 con 0x7fd5b809fd40 2024-09-16T14:49:19.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 10 monclient: _renew_subs 2024-09-16T14:49:19.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:19.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 1 -- 172.21.15.189:0/3928447316 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b8120100 con 0x7fd5b809fd40 2024-09-16T14:49:19.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 1 -- 172.21.15.189:0/3928447316 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5a00043a0 con 0x7fd5b809fd40 2024-09-16T14:49:19.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:19.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 1 -- 172.21.15.189:0/3928447316 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd5a0004730 con 0x7fd5b809fd40 2024-09-16T14:49:19.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:19.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:19.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 10 monclient: dump: 2024-09-16T14:49:19.734 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:19.734 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:19.734 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:19.734 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:19.734 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:19.734 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:19.734 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:19.734 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:19.735 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:19.735 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:19.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5b5ffb640 4 set_mon_vals no callback set 2024-09-16T14:49:19.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 1 -- 172.21.15.189:0/3928447316 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd5a0004890 con 0x7fd5b809fd40 2024-09-16T14:49:19.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:19.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:19.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:19.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5affff640 1 -- 172.21.15.189:0/3928447316 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(183..183 src has 154..183) v4 ==== 6899+0+0 (unknown 2435975455 0 0) 0x7fd5a0043dc0 con 0x7fd5b809fd40 2024-09-16T14:49:19.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5bdffb640 1 -- 172.21.15.189:0/3928447316 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59c005050 con 0x7fd5b809fd40 2024-09-16T14:49:19.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.712+0000 7fd5b6ffd640 10 monclient: get_auth_request con 0x7fd5980360a0 auth_method 0 2024-09-16T14:49:19.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.716+0000 7fd5affff640 1 -- 172.21.15.189:0/3928447316 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd5a00146a0 con 0x7fd5b809fd40 2024-09-16T14:49:19.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.716+0000 7fd5affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:19.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.716+0000 7fd5affff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:19.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.876+0000 7fd5bdffb640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:19.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.876+0000 7fd5bdffb640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:19.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.876+0000 7fd5bdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:19.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:19.876+0000 7fd5bdffb640 1 -- 172.21.15.189:0/3928447316 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fd59c005880 con 0x7fd5b809fd40 2024-09-16T14:49:20.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.472+0000 7fd5affff640 1 -- 172.21.15.189:0/3928447316 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 added bucket h1 type host to location {rack=r1} v184) v1 ==== 166+0+0 (unknown 583486770 0 0) 0x7fd5a0012b10 con 0x7fd5b809fd40 2024-09-16T14:49:20.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.472+0000 7fd5affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:20.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.472+0000 7fd5affff640 10 monclient: _finish_command 2 = system:0 added bucket h1 type host to location {rack=r1} 2024-09-16T14:49:20.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.572+0000 7fd5bdffb640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:20.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.572+0000 7fd5bdffb640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:20.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.572+0000 7fd5bdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:20.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.572+0000 7fd5bdffb640 1 -- 172.21.15.189:0/3928447316 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fd59c005ac0 con 0x7fd5b809fd40 2024-09-16T14:49:20.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.572+0000 7fd5affff640 1 -- 172.21.15.189:0/3928447316 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h1' already exists v184) v1 ==== 145+0+0 (unknown 2346422746 0 0) 0x7fd5a0014290 con 0x7fd5b809fd40 2024-09-16T14:49:20.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.572+0000 7fd5affff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:20.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.572+0000 7fd5affff640 10 monclient: _finish_command 3 = system:0 bucket 'h1' already exists 2024-09-16T14:49:20.577 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h1' already exists 2024-09-16T14:49:20.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.576+0000 7fd5bdffb640 1 -- 172.21.15.189:0/3928447316 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd5980360a0 legacy=0x7fd598038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:20.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.576+0000 7fd5bdffb640 10 monclient: shutdown 2024-09-16T14:49:20.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.576+0000 7fd5bdffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:20.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.576+0000 7fd5bdffb640 1 -- 172.21.15.189:0/3928447316 >> v1:172.21.15.189:6789/0 conn(0x7fd5b809fd40 legacy=0x7fd5b8133b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:20.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.576+0000 7fd5bdffb640 1 -- 172.21.15.189:0/3928447316 shutdown_connections 2024-09-16T14:49:20.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.576+0000 7fd5bdffb640 1 -- 172.21.15.189:0/3928447316 >> 172.21.15.189:0/3928447316 conn(0x7fd5b80a7a70 msgr2=0x7fd5b81bb300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:20.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.576+0000 7fd5bdffb640 1 -- 172.21.15.189:0/3928447316 shutdown_connections 2024-09-16T14:49:20.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.576+0000 7fd5bdffb640 1 -- 172.21.15.189:0/3928447316 wait complete. 2024-09-16T14:49:20.592 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:20.593 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h2 host rack=r1 2024-09-16T14:49:20.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10afb3c640 10 monclient: get_monmap_and_config 2024-09-16T14:49:20.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10afb3c640 10 monclient: build_initial_monmap 2024-09-16T14:49:20.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10afb3c640 10 monclient: monmap: 2024-09-16T14:49:20.710 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:20.710 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:20.711 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:20.711121+0000 2024-09-16T14:49:20.711 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:20.711121+0000 2024-09-16T14:49:20.711 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:20.711 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:20.711 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:20.711 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:20.711 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:20.711 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10afb3c640 1 Processor -- start 2024-09-16T14:49:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10afb3c640 1 -- start start 2024-09-16T14:49:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10afb3c640 10 monclient: init 2024-09-16T14:49:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10afb3c640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10afb3c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10afb3c640 10 monclient(hunting): picked mon.noname-a con 0x7f10a80717e0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10afb3c640 10 monclient(hunting): picked mon.noname-c con 0x7f10a8071bc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10afb3c640 10 monclient(hunting): picked mon.noname-b con 0x7f10a8071fa0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10afb3c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a80a1370 con 0x7f10a80717e0 2024-09-16T14:49:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10afb3c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a809c6c0 con 0x7f10a8071bc0 2024-09-16T14:49:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10afb3c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a809cbf0 con 0x7f10a8071fa0 2024-09-16T14:49:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10afb3c640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10afb3c640 10 monclient(hunting): authenticate will time out at 1342.827554s 2024-09-16T14:49:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10ad8d8640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f10a80717e0 legacy=0x7f10a809d010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10ad0d7640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f10a8071bc0 0x7f10a8139a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53342/0 (socket says 172.21.15.189:53342) 2024-09-16T14:49:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10ad0d7640 1 -- 172.21.15.189:0/3101991895 learned_addr learned my addr 172.21.15.189:0/3101991895 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.708+0000 7f10ac8d6640 1 -- 172.21.15.189:0/3101991895 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445014994 0 0) 0x7f10a809c6c0 con 0x7f10a8071bc0 2024-09-16T14:49:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 10 monclient(hunting): _negotiate 2024-09-16T14:49:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:20.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 10 monclient(hunting): my global_id is 144457 2024-09-16T14:49:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 1 -- 172.21.15.189:0/3101991895 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1094003530 con 0x7f10a8071bc0 2024-09-16T14:49:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 1 -- 172.21.15.189:0/3101991895 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3212936386 0 0) 0x7f1094003530 con 0x7f10a8071bc0 2024-09-16T14:49:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 1 -- 172.21.15.189:0/3101991895 >> v1:172.21.15.26:6790/0 conn(0x7f10a8071fa0 legacy=0x7f10a813d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 1 -- 172.21.15.189:0/3101991895 >> v1:172.21.15.26:6789/0 conn(0x7f10a80717e0 legacy=0x7f10a809d010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 10 monclient: _finish_hunting 0 2024-09-16T14:49:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 1 monclient: found mon.noname-c 2024-09-16T14:49:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 1 -- 172.21.15.189:0/3101991895 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a806c9e0 con 0x7f10a8071bc0 2024-09-16T14:49:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 10 monclient: _finish_auth 0 2024-09-16T14:49:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 10 monclient: _check_auth_tickets 2024-09-16T14:49:20.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 1 -- 172.21.15.189:0/3101991895 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1098001620 con 0x7f10a8071bc0 2024-09-16T14:49:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 10 monclient: dump: 2024-09-16T14:49:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:20.718 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 5 monclient: authenticate success, global_id 144457 2024-09-16T14:49:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 1 -- 172.21.15.189:0/3101991895 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10980018d0 con 0x7f10a8071bc0 2024-09-16T14:49:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 1 -- 172.21.15.189:0/3101991895 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1098001ca0 con 0x7f10a8071bc0 2024-09-16T14:49:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 4 set_mon_vals no callback set 2024-09-16T14:49:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ac8d6640 10 monclient: dump: 2024-09-16T14:49:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:20.719 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 10 monclient: shutdown 2024-09-16T14:49:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 1 -- 172.21.15.189:0/3101991895 >> v1:172.21.15.189:6789/0 conn(0x7f10a8071bc0 legacy=0x7f10a8139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 1 -- 172.21.15.189:0/3101991895 shutdown_connections 2024-09-16T14:49:20.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 1 -- 172.21.15.189:0/3101991895 >> 172.21.15.189:0/3101991895 conn(0x7f10a80a19b0 msgr2=0x7f10a80a1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:20.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 1 -- 172.21.15.189:0/3101991895 shutdown_connections 2024-09-16T14:49:20.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 1 -- 172.21.15.189:0/3101991895 wait complete. 2024-09-16T14:49:20.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 10 monclient: build_initial_monmap 2024-09-16T14:49:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 10 monclient: monmap: 2024-09-16T14:49:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:20.724 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:20.725 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 1 Processor -- start 2024-09-16T14:49:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 1 -- start start 2024-09-16T14:49:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 10 monclient: init 2024-09-16T14:49:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 10 monclient(hunting): picked mon.noname-a con 0x7f10a80717e0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 10 monclient(hunting): picked mon.noname-b con 0x7f10a8071bc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 10 monclient(hunting): picked mon.noname-c con 0x7f10a8071fa0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a809c110 con 0x7f10a80717e0 2024-09-16T14:49:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a813a3b0 con 0x7f10a8071bc0 2024-09-16T14:49:20.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a8138050 con 0x7f10a8071fa0 2024-09-16T14:49:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 10 monclient(hunting): authenticate will time out at 1342.830496s 2024-09-16T14:49:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ad0d7640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f10a8071bc0 0x7f10a8136970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53344/0 (socket says 172.21.15.189:53344) 2024-09-16T14:49:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ad0d7640 1 -- 172.21.15.189:0/2101569217 learned_addr learned my addr 172.21.15.189:0/2101569217 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10ad8d8640 1 -- 172.21.15.189:0/2101569217 >> v1:172.21.15.26:6789/0 conn(0x7f10a80717e0 legacy=0x7f10a8136260 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 1 -- 172.21.15.189:0/2101569217 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062939261 0 0) 0x7f10a813a3b0 con 0x7f10a8071bc0 2024-09-16T14:49:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:20.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 10 monclient(hunting): my global_id is 144460 2024-09-16T14:49:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 1 -- 172.21.15.189:0/2101569217 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f108c003530 con 0x7f10a8071bc0 2024-09-16T14:49:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 1 -- 172.21.15.189:0/2101569217 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1776680793 0 0) 0x7f108c003530 con 0x7f10a8071bc0 2024-09-16T14:49:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 1 -- 172.21.15.189:0/2101569217 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10a813a3b0 con 0x7f10a8071bc0 2024-09-16T14:49:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 1 -- 172.21.15.189:0/2101569217 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1098001be0 con 0x7f10a8071bc0 2024-09-16T14:49:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 10 monclient(hunting): dump: 2024-09-16T14:49:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:20.728 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:20.728 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:20.728 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:20.728 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:20.728 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:20.728 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:20.728 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:20.728 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:20.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:20.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 1 -- 172.21.15.189:0/2101569217 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4093118641 0 0) 0x7f10a813a3b0 con 0x7f10a8071bc0 2024-09-16T14:49:20.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 1 -- 172.21.15.189:0/2101569217 >> v1:172.21.15.26:6790/0 conn(0x7f10a8071fa0 legacy=0x7f10a8137080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:20.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 1 -- 172.21.15.189:0/2101569217 >> v1:172.21.15.26:6789/0 conn(0x7f10a80717e0 legacy=0x7f10a8136260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:20.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:49:20.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 1 monclient: found mon.b 2024-09-16T14:49:20.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:20.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:20.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 1 -- 172.21.15.189:0/2101569217 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a80a2e40 con 0x7f10a8071bc0 2024-09-16T14:49:20.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:49:20.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:20.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:20.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 5 monclient: authenticate success, global_id 144460 2024-09-16T14:49:20.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 10 monclient: _renew_subs 2024-09-16T14:49:20.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:20.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 1 -- 172.21.15.189:0/2101569217 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a813a0d0 con 0x7f10a8071bc0 2024-09-16T14:49:20.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 10 monclient: _renew_subs 2024-09-16T14:49:20.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:20.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f10afb3c640 1 -- 172.21.15.189:0/2101569217 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a8138c20 con 0x7f10a8071bc0 2024-09-16T14:49:20.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 1 -- 172.21.15.189:0/2101569217 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1098001200 con 0x7f10a8071bc0 2024-09-16T14:49:20.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:20.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 1 -- 172.21.15.189:0/2101569217 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1098004da0 con 0x7f10a8071bc0 2024-09-16T14:49:20.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:20.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:20.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109ffff640 4 set_mon_vals no callback set 2024-09-16T14:49:20.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.712+0000 7f109e7fc640 10 monclient: dump: 2024-09-16T14:49:20.731 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:20.731 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:20.731 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:20.731 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:20.731 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:20.731 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:20.731 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:20.731 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:20.731 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:20.731 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:20.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.716+0000 7f10afb3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:20.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.716+0000 7f109e7fc640 1 -- 172.21.15.189:0/2101569217 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1098004fc0 con 0x7f10a8071bc0 2024-09-16T14:49:20.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.716+0000 7f10afb3c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:20.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.716+0000 7f10afb3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:20.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.716+0000 7f10afb3c640 1 -- 172.21.15.189:0/2101569217 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1090005050 con 0x7f10a8071bc0 2024-09-16T14:49:20.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.716+0000 7f109e7fc640 1 -- 172.21.15.189:0/2101569217 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(184..184 src has 154..184) v4 ==== 7109+0+0 (unknown 2578174566 0 0) 0x7f1098044320 con 0x7f10a8071bc0 2024-09-16T14:49:20.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.716+0000 7f10ad8d8640 10 monclient: get_auth_request con 0x7f108c036010 auth_method 0 2024-09-16T14:49:20.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.720+0000 7f109e7fc640 1 -- 172.21.15.189:0/2101569217 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1098006630 con 0x7f10a8071bc0 2024-09-16T14:49:20.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.720+0000 7f109e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:20.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.720+0000 7f109e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:20.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.872+0000 7f10afb3c640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:20.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.872+0000 7f10afb3c640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:20.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.872+0000 7f10afb3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:20.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:20.872+0000 7f10afb3c640 1 -- 172.21.15.189:0/2101569217 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f1090005880 con 0x7f10a8071bc0 2024-09-16T14:49:21.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.480+0000 7f109e7fc640 1 -- 172.21.15.189:0/2101569217 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}]=0 added bucket h2 type host to location {rack=r1} v185) v1 ==== 166+0+0 (unknown 115398724 0 0) 0x7f1098012fc0 con 0x7f10a8071bc0 2024-09-16T14:49:21.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.480+0000 7f109e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:21.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.480+0000 7f109e7fc640 10 monclient: _finish_command 2 = system:0 added bucket h2 type host to location {rack=r1} 2024-09-16T14:49:21.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.576+0000 7f10afb3c640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:21.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.576+0000 7f10afb3c640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:21.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.576+0000 7f10afb3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:21.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.576+0000 7f10afb3c640 1 -- 172.21.15.189:0/2101569217 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f1090005340 con 0x7f10a8071bc0 2024-09-16T14:49:21.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.576+0000 7f109e7fc640 1 -- 172.21.15.189:0/2101569217 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h2' already exists v185) v1 ==== 145+0+0 (unknown 3664088182 0 0) 0x7f1098014740 con 0x7f10a8071bc0 2024-09-16T14:49:21.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.576+0000 7f109e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:21.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.576+0000 7f109e7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h2' already exists 2024-09-16T14:49:21.580 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h2' already exists 2024-09-16T14:49:21.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.580+0000 7f10afb3c640 1 -- 172.21.15.189:0/2101569217 >> v1:172.21.15.189:6812/3196161582 conn(0x7f108c036010 legacy=0x7f108c0384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:21.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.580+0000 7f10afb3c640 10 monclient: shutdown 2024-09-16T14:49:21.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.580+0000 7f10afb3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:21.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.580+0000 7f10afb3c640 1 -- 172.21.15.189:0/2101569217 >> v1:172.21.15.189:6789/0 conn(0x7f10a8071bc0 legacy=0x7f10a8136970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:21.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.580+0000 7f10afb3c640 1 -- 172.21.15.189:0/2101569217 shutdown_connections 2024-09-16T14:49:21.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.580+0000 7f10afb3c640 1 -- 172.21.15.189:0/2101569217 >> 172.21.15.189:0/2101569217 conn(0x7f10a80a19b0 msgr2=0x7f10a809aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:21.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.580+0000 7f10afb3c640 1 -- 172.21.15.189:0/2101569217 shutdown_connections 2024-09-16T14:49:21.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.580+0000 7f10afb3c640 1 -- 172.21.15.189:0/2101569217 wait complete. 2024-09-16T14:49:21.597 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:21.597 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h3 host rack=r1 2024-09-16T14:49:21.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.700+0000 7f52eaa47640 10 monclient: get_monmap_and_config 2024-09-16T14:49:21.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.700+0000 7f52eaa47640 10 monclient: build_initial_monmap 2024-09-16T14:49:21.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.700+0000 7f52eaa47640 10 monclient: monmap: 2024-09-16T14:49:21.705 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:21.705 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:21.705 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:21.705201+0000 2024-09-16T14:49:21.705 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:21.705201+0000 2024-09-16T14:49:21.705 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:21.705 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:21.705 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:21.705 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:21.705 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:21.705 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:21.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 1 Processor -- start 2024-09-16T14:49:21.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 1 -- start start 2024-09-16T14:49:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 10 monclient: init 2024-09-16T14:49:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 10 monclient(hunting): picked mon.noname-a con 0x7f52e4070830 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 10 monclient(hunting): picked mon.noname-c con 0x7f52e4071320 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 10 monclient(hunting): picked mon.noname-b con 0x7f52e4134cf0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e4135740 con 0x7f52e4070830 2024-09-16T14:49:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e4135160 con 0x7f52e4071320 2024-09-16T14:49:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e409d810 con 0x7f52e4134cf0 2024-09-16T14:49:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 10 monclient(hunting): authenticate will time out at 1343.821120s 2024-09-16T14:49:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e37fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f52e4071320 0x7f52e4139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53354/0 (socket says 172.21.15.189:53354) 2024-09-16T14:49:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e37fe640 1 -- 172.21.15.189:0/1108362745 learned_addr learned my addr 172.21.15.189:0/1108362745 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e3fff640 1 -- 172.21.15.189:0/1108362745 >> v1:172.21.15.26:6789/0 conn(0x7f52e4070830 legacy=0x7f52e4070c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 1 -- 172.21.15.189:0/1108362745 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841457819 0 0) 0x7f52e4135160 con 0x7f52e4071320 2024-09-16T14:49:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 10 monclient(hunting): my global_id is 144463 2024-09-16T14:49:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 1 -- 172.21.15.189:0/1108362745 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52cc003530 con 0x7f52e4071320 2024-09-16T14:49:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 1 -- 172.21.15.189:0/1108362745 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1556213459 0 0) 0x7f52cc003530 con 0x7f52e4071320 2024-09-16T14:49:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 1 -- 172.21.15.189:0/1108362745 >> v1:172.21.15.26:6790/0 conn(0x7f52e4134cf0 legacy=0x7f52e413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:21.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 1 -- 172.21.15.189:0/1108362745 >> v1:172.21.15.26:6789/0 conn(0x7f52e4070830 legacy=0x7f52e4070c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:21.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:21.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 1 monclient: found mon.noname-c 2024-09-16T14:49:21.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:21.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:21.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 1 -- 172.21.15.189:0/1108362745 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e409f1c0 con 0x7f52e4071320 2024-09-16T14:49:21.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:49:21.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:21.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:21.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 1 -- 172.21.15.189:0/1108362745 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52d4001620 con 0x7f52e4071320 2024-09-16T14:49:21.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:21.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:21.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 10 monclient: dump: 2024-09-16T14:49:21.712 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:21.712 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:21.712 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:21.712 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:21.712 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:21.712 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:21.712 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:21.712 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:21.712 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:21.712 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:21.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 5 monclient: authenticate success, global_id 144463 2024-09-16T14:49:21.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:21.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 1 -- 172.21.15.189:0/1108362745 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52d4001940 con 0x7f52e4071320 2024-09-16T14:49:21.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:21.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 1 -- 172.21.15.189:0/1108362745 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52d4001d10 con 0x7f52e4071320 2024-09-16T14:49:21.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:21.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:21.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52e2ffd640 10 monclient: dump: 2024-09-16T14:49:21.713 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:21.714 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 4 set_mon_vals no callback set 2024-09-16T14:49:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 10 monclient: shutdown 2024-09-16T14:49:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:21.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 1 -- 172.21.15.189:0/1108362745 >> v1:172.21.15.189:6789/0 conn(0x7f52e4071320 legacy=0x7f52e4139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:21.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 1 -- 172.21.15.189:0/1108362745 shutdown_connections 2024-09-16T14:49:21.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 1 -- 172.21.15.189:0/1108362745 >> 172.21.15.189:0/1108362745 conn(0x7f52e40a3380 msgr2=0x7f52e409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:21.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 1 -- 172.21.15.189:0/1108362745 shutdown_connections 2024-09-16T14:49:21.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 1 -- 172.21.15.189:0/1108362745 wait complete. 2024-09-16T14:49:21.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 10 monclient: build_initial_monmap 2024-09-16T14:49:21.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 10 monclient: monmap: 2024-09-16T14:49:21.715 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:21.715 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:21.715 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:21.715 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:21.715 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:21.715 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:21.716 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:21.716 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:21.716 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:21.716 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:21.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 1 Processor -- start 2024-09-16T14:49:21.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 1 -- start start 2024-09-16T14:49:21.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.704+0000 7f52eaa47640 10 monclient: init 2024-09-16T14:49:21.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:21.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:21.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 10 monclient(hunting): picked mon.noname-c con 0x7f52e4070830 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:21.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 10 monclient(hunting): picked mon.noname-a con 0x7f52e4071320 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:21.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 10 monclient(hunting): picked mon.noname-b con 0x7f52e4134cf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:21.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e4132850 con 0x7f52e4071320 2024-09-16T14:49:21.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e406b320 con 0x7f52e4134cf0 2024-09-16T14:49:21.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e4132dc0 con 0x7f52e4070830 2024-09-16T14:49:21.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:21.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 10 monclient(hunting): authenticate will time out at 1343.824508s 2024-09-16T14:49:21.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e37fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f52e4071320 legacy=0x7f52e41ce400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:21.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e8fe4640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f52e4134cf0 0x7f52e41d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53358/0 (socket says 172.21.15.189:53358) 2024-09-16T14:49:21.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e8fe4640 1 -- 172.21.15.189:0/1145652055 learned_addr learned my addr 172.21.15.189:0/1145652055 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:21.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131883664 0 0) 0x7f52e406b320 con 0x7f52e4134cf0 2024-09-16T14:49:21.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:49:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient(hunting): my global_id is 144466 2024-09-16T14:49:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d0003530 con 0x7f52e4134cf0 2024-09-16T14:49:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 409891028 0 0) 0x7f52d0003530 con 0x7f52e4134cf0 2024-09-16T14:49:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52e406b320 con 0x7f52e4134cf0 2024-09-16T14:49:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52dc001c80 con 0x7f52e4134cf0 2024-09-16T14:49:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient(hunting): dump: 2024-09-16T14:49:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:21.723 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866246816 0 0) 0x7f52e4132dc0 con 0x7f52e4070830 2024-09-16T14:49:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:49:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:21.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient(hunting): my global_id is 154256 2024-09-16T14:49:21.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d0003530 con 0x7f52e4070830 2024-09-16T14:49:21.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 669665027 0 0) 0x7f52e406b320 con 0x7f52e4134cf0 2024-09-16T14:49:21.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 >> v1:172.21.15.26:6790/0 conn(0x7f52e4070830 legacy=0x7f52e4130a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:21.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 >> v1:172.21.15.26:6789/0 conn(0x7f52e4071320 legacy=0x7f52e41ce400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:21.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:49:21.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 1 monclient: found mon.b 2024-09-16T14:49:21.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:21.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:21.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e411e420 con 0x7f52e4134cf0 2024-09-16T14:49:21.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient: _finish_auth 0 2024-09-16T14:49:21.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:49:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 5 monclient: authenticate success, global_id 144466 2024-09-16T14:49:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 10 monclient: _renew_subs 2024-09-16T14:49:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 1 -- 172.21.15.189:0/1145652055 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e411ca10 con 0x7f52e4134cf0 2024-09-16T14:49:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 10 monclient: _renew_subs 2024-09-16T14:49:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 1 -- 172.21.15.189:0/1145652055 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e411c0c0 con 0x7f52e4134cf0 2024-09-16T14:49:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52dc002050 con 0x7f52e4134cf0 2024-09-16T14:49:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52dc003e70 con 0x7f52e4134cf0 2024-09-16T14:49:21.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:21.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:21.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 10 monclient: dump: 2024-09-16T14:49:21.726 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:21.726 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:21.726 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:21.726 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:21.726 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:21.726 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:21.726 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:21.726 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:21.726 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:21.727 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:21.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e27fc640 4 set_mon_vals no callback set 2024-09-16T14:49:21.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f52dc004050 con 0x7f52e4134cf0 2024-09-16T14:49:21.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:21.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(185..185 src has 159..185) v4 ==== 7319+0+0 (unknown 1348983324 0 0) 0x7f52dc0444e0 con 0x7f52e4134cf0 2024-09-16T14:49:21.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:21.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:21.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52eaa47640 1 -- 172.21.15.189:0/1145652055 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c4005050 con 0x7f52e4134cf0 2024-09-16T14:49:21.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.708+0000 7f52e3fff640 10 monclient: get_auth_request con 0x7f52d00360e0 auth_method 0 2024-09-16T14:49:21.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.712+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f52dc0149b0 con 0x7f52e4134cf0 2024-09-16T14:49:21.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.712+0000 7f52e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:21.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.712+0000 7f52e0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:21.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.860+0000 7f52eaa47640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:21.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.860+0000 7f52eaa47640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:21.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.860+0000 7f52eaa47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:21.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:21.860+0000 7f52eaa47640 1 -- 172.21.15.189:0/1145652055 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f52c4005880 con 0x7f52e4134cf0 2024-09-16T14:49:22.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.488+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}]=0 added bucket h3 type host to location {rack=r1} v186) v1 ==== 166+0+0 (unknown 302430699 0 0) 0x7f52dc013050 con 0x7f52e4134cf0 2024-09-16T14:49:22.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.488+0000 7f52e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:22.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.488+0000 7f52e0ff9640 10 monclient: _finish_command 2 = system:0 added bucket h3 type host to location {rack=r1} 2024-09-16T14:49:22.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.592+0000 7f52eaa47640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:22.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.592+0000 7f52eaa47640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:22.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.592+0000 7f52eaa47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:22.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.592+0000 7f52eaa47640 1 -- 172.21.15.189:0/1145652055 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f52c4005ac0 con 0x7f52e4134cf0 2024-09-16T14:49:22.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.592+0000 7f52e0ff9640 1 -- 172.21.15.189:0/1145652055 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h3' already exists v186) v1 ==== 145+0+0 (unknown 157880584 0 0) 0x7f52dc0147d0 con 0x7f52e4134cf0 2024-09-16T14:49:22.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.592+0000 7f52e0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:22.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.592+0000 7f52e0ff9640 10 monclient: _finish_command 3 = system:0 bucket 'h3' already exists 2024-09-16T14:49:22.596 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h3' already exists 2024-09-16T14:49:22.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.596+0000 7f52eaa47640 1 -- 172.21.15.189:0/1145652055 >> v1:172.21.15.189:6812/3196161582 conn(0x7f52d00360e0 legacy=0x7f52d00385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:22.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.596+0000 7f52eaa47640 10 monclient: shutdown 2024-09-16T14:49:22.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.596+0000 7f52eaa47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:22.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.596+0000 7f52eaa47640 1 -- 172.21.15.189:0/1145652055 >> v1:172.21.15.189:6789/0 conn(0x7f52e4134cf0 legacy=0x7f52e41d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:22.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.596+0000 7f52eaa47640 1 -- 172.21.15.189:0/1145652055 shutdown_connections 2024-09-16T14:49:22.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.596+0000 7f52eaa47640 1 -- 172.21.15.189:0/1145652055 >> 172.21.15.189:0/1145652055 conn(0x7f52e40a3380 msgr2=0x7f52e4131770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:22.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.596+0000 7f52eaa47640 1 -- 172.21.15.189:0/1145652055 shutdown_connections 2024-09-16T14:49:22.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.596+0000 7f52eaa47640 1 -- 172.21.15.189:0/1145652055 wait complete. 2024-09-16T14:49:22.612 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:22.612 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h4 host rack=r1 2024-09-16T14:49:22.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 10 monclient: get_monmap_and_config 2024-09-16T14:49:22.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 10 monclient: build_initial_monmap 2024-09-16T14:49:22.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 10 monclient: monmap: 2024-09-16T14:49:22.726 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:22.727 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:22.727 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:22.727170+0000 2024-09-16T14:49:22.727 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:22.727170+0000 2024-09-16T14:49:22.727 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:22.727 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:22.727 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:22.727 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:22.727 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:22.727 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:22.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 1 Processor -- start 2024-09-16T14:49:22.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 1 -- start start 2024-09-16T14:49:22.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 10 monclient: init 2024-09-16T14:49:22.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:22.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:22.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 10 monclient(hunting): picked mon.noname-a con 0x7fb8f8099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:22.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 10 monclient(hunting): picked mon.noname-c con 0x7fb8f809c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:22.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 10 monclient(hunting): picked mon.noname-b con 0x7fb8f809e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:22.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f8099e50 con 0x7fb8f8099a70 2024-09-16T14:49:22.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f809dbd0 con 0x7fb8f809c550 2024-09-16T14:49:22.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f809ee20 con 0x7fb8f809e6d0 2024-09-16T14:49:22.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:22.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 10 monclient(hunting): authenticate will time out at 1344.842784s 2024-09-16T14:49:22.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f77fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb8f809c550 0x7fb8f81379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53366/0 (socket says 172.21.15.189:53366) 2024-09-16T14:49:22.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f77fe640 1 -- 172.21.15.189:0/224236273 learned_addr learned my addr 172.21.15.189:0/224236273 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:22.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f7fff640 1 -- 172.21.15.189:0/224236273 >> v1:172.21.15.26:6789/0 conn(0x7fb8f8099a70 legacy=0x7fb8f809dfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:22.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 1 -- 172.21.15.189:0/224236273 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257456727 0 0) 0x7fb8f809dbd0 con 0x7fb8f809c550 2024-09-16T14:49:22.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:22.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:22.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:22.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 10 monclient(hunting): my global_id is 144469 2024-09-16T14:49:22.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 1 -- 172.21.15.189:0/224236273 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e0003530 con 0x7fb8f809c550 2024-09-16T14:49:22.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 1 -- 172.21.15.189:0/224236273 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1147836281 0 0) 0x7fb8e0003530 con 0x7fb8f809c550 2024-09-16T14:49:22.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 1 -- 172.21.15.189:0/224236273 >> v1:172.21.15.26:6790/0 conn(0x7fb8f809e6d0 legacy=0x7fb8f813b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:22.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 1 -- 172.21.15.189:0/224236273 >> v1:172.21.15.26:6789/0 conn(0x7fb8f8099a70 legacy=0x7fb8f809dfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:22.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:22.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 1 monclient: found mon.noname-c 2024-09-16T14:49:22.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:22.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:22.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 1 -- 172.21.15.189:0/224236273 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f809f100 con 0x7fb8f809c550 2024-09-16T14:49:22.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:49:22.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:22.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:22.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 1 -- 172.21.15.189:0/224236273 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb8e8001620 con 0x7fb8f809c550 2024-09-16T14:49:22.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:22.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:22.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 10 monclient: dump: 2024-09-16T14:49:22.737 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:22.737 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:22.737 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:22.737 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:22.738 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:22.738 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:22.738 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:22.738 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:22.738 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:22.738 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:22.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 5 monclient: authenticate success, global_id 144469 2024-09-16T14:49:22.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:22.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 1 -- 172.21.15.189:0/224236273 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8e80018d0 con 0x7fb8f809c550 2024-09-16T14:49:22.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:22.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 1 -- 172.21.15.189:0/224236273 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb8e8001ca0 con 0x7fb8f809c550 2024-09-16T14:49:22.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:22.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8fe5f3640 4 set_mon_vals no callback set 2024-09-16T14:49:22.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:22.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:22.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.724+0000 7fb8f6ffd640 10 monclient: dump: 2024-09-16T14:49:22.739 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:22.739 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:22.739 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:22.739 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:22.739 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:22.739 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:22.739 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:22.739 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:22.739 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:22.740 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:22.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient: shutdown 2024-09-16T14:49:22.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:22.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 1 -- 172.21.15.189:0/224236273 >> v1:172.21.15.189:6789/0 conn(0x7fb8f809c550 legacy=0x7fb8f81379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:22.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 1 -- 172.21.15.189:0/224236273 shutdown_connections 2024-09-16T14:49:22.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 1 -- 172.21.15.189:0/224236273 >> 172.21.15.189:0/224236273 conn(0x7fb8f80a3380 msgr2=0x7fb8f809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:22.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 1 -- 172.21.15.189:0/224236273 shutdown_connections 2024-09-16T14:49:22.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 1 -- 172.21.15.189:0/224236273 wait complete. 2024-09-16T14:49:22.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient: build_initial_monmap 2024-09-16T14:49:22.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient: monmap: 2024-09-16T14:49:22.740 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:22.740 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:22.740 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:22.741 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:22.741 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:22.741 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:22.741 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:22.741 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:22.741 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:22.741 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:22.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 1 Processor -- start 2024-09-16T14:49:22.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 1 -- start start 2024-09-16T14:49:22.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient: init 2024-09-16T14:49:22.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:22.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:49:22.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient(hunting): picked mon.noname-c con 0x7fb8f8099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:22.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient(hunting): picked mon.noname-b con 0x7fb8f809c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:22.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient(hunting): picked mon.noname-a con 0x7fb8f809e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:22.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f809cf20 con 0x7fb8f809e6d0 2024-09-16T14:49:22.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f806f880 con 0x7fb8f809c550 2024-09-16T14:49:22.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f806cd00 con 0x7fb8f8099a70 2024-09-16T14:49:22.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:22.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient(hunting): authenticate will time out at 1344.845724s 2024-09-16T14:49:22.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f77fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb8f809c550 0x7fb8f81cc040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53368/0 (socket says 172.21.15.189:53368) 2024-09-16T14:49:22.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f77fe640 1 -- 172.21.15.189:0/3762230742 learned_addr learned my addr 172.21.15.189:0/3762230742 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:22.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fcb90640 1 -- 172.21.15.189:0/3762230742 >> v1:172.21.15.26:6789/0 conn(0x7fb8f809e6d0 legacy=0x7fb8f81cf770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:22.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170195745 0 0) 0x7fb8f806f880 con 0x7fb8f809c550 2024-09-16T14:49:22.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:49:22.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:22.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:22.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient(hunting): my global_id is 144472 2024-09-16T14:49:22.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e4003530 con 0x7fb8f809c550 2024-09-16T14:49:22.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 293391607 0 0) 0x7fb8e4003530 con 0x7fb8f809c550 2024-09-16T14:49:22.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8f806f880 con 0x7fb8f809c550 2024-09-16T14:49:22.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb8e8001be0 con 0x7fb8f809c550 2024-09-16T14:49:22.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:22.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:22.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient(hunting): dump: 2024-09-16T14:49:22.743 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:22.743 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:22.743 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:22.743 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:22.744 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:22.744 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:22.744 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:22.744 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:22.744 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:22.744 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:22.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:22.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227204292 0 0) 0x7fb8f806cd00 con 0x7fb8f8099a70 2024-09-16T14:49:22.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:49:22.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:22.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:22.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient(hunting): my global_id is 154262 2024-09-16T14:49:22.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e4003530 con 0x7fb8f8099a70 2024-09-16T14:49:22.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2563570922 0 0) 0x7fb8f806f880 con 0x7fb8f809c550 2024-09-16T14:49:22.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 >> v1:172.21.15.26:6790/0 conn(0x7fb8f8099a70 legacy=0x7fb8f8132ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:22.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 >> v1:172.21.15.26:6789/0 conn(0x7fb8f809e6d0 legacy=0x7fb8f81cf770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:22.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:49:22.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 1 monclient: found mon.b 2024-09-16T14:49:22.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:22.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:22.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f806b320 con 0x7fb8f809c550 2024-09-16T14:49:22.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient: _finish_auth 0 2024-09-16T14:49:22.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:49:22.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:22.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 5 monclient: authenticate success, global_id 144472 2024-09-16T14:49:22.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient: _renew_subs 2024-09-16T14:49:22.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:22.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 1 -- 172.21.15.189:0/3762230742 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f81332b0 con 0x7fb8f809c550 2024-09-16T14:49:22.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient: _renew_subs 2024-09-16T14:49:22.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:22.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 1 -- 172.21.15.189:0/3762230742 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f8119c80 con 0x7fb8f809c550 2024-09-16T14:49:22.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8e8004c80 con 0x7fb8f809c550 2024-09-16T14:49:22.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:22.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb8e8005010 con 0x7fb8f809c550 2024-09-16T14:49:22.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:22.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:22.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 10 monclient: dump: 2024-09-16T14:49:22.747 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:22.747 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:22.747 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:22.747 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:22.747 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:22.747 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:22.747 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:22.747 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:22.747 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:22.747 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:22.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f67fc640 4 set_mon_vals no callback set 2024-09-16T14:49:22.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:22.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:22.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:22.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb8e8005170 con 0x7fb8f809c550 2024-09-16T14:49:22.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8fe5f3640 1 -- 172.21.15.189:0/3762230742 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d8005050 con 0x7fb8f809c550 2024-09-16T14:49:22.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(186..186 src has 159..186) v4 ==== 7529+0+0 (unknown 584025423 0 0) 0x7fb8e80443a0 con 0x7fb8f809c550 2024-09-16T14:49:22.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.728+0000 7fb8f7fff640 10 monclient: get_auth_request con 0x7fb8e4036160 auth_method 0 2024-09-16T14:49:22.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.736+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb8e8014af0 con 0x7fb8f809c550 2024-09-16T14:49:22.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.736+0000 7fb8f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:22.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.736+0000 7fb8f4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:22.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.884+0000 7fb8fe5f3640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:22.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.884+0000 7fb8fe5f3640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:22.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.884+0000 7fb8fe5f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:22.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:22.884+0000 7fb8fe5f3640 1 -- 172.21.15.189:0/3762230742 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb8d8005880 con 0x7fb8f809c550 2024-09-16T14:49:23.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.496+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}]=0 added bucket h4 type host to location {rack=r1} v187) v1 ==== 166+0+0 (unknown 1320153768 0 0) 0x7fb8e80148c0 con 0x7fb8f809c550 2024-09-16T14:49:23.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.496+0000 7fb8f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:23.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.496+0000 7fb8f4ff9640 10 monclient: _finish_command 2 = system:0 added bucket h4 type host to location {rack=r1} 2024-09-16T14:49:23.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.592+0000 7fb8fe5f3640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:23.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.592+0000 7fb8fe5f3640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:23.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.592+0000 7fb8fe5f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:23.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.592+0000 7fb8fe5f3640 1 -- 172.21.15.189:0/3762230742 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb8d8005340 con 0x7fb8f809c550 2024-09-16T14:49:23.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.592+0000 7fb8f4ff9640 1 -- 172.21.15.189:0/3762230742 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h4' already exists v187) v1 ==== 145+0+0 (unknown 2031728430 0 0) 0x7fb8e80148c0 con 0x7fb8f809c550 2024-09-16T14:49:23.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.592+0000 7fb8f4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:23.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.592+0000 7fb8f4ff9640 10 monclient: _finish_command 3 = system:0 bucket 'h4' already exists 2024-09-16T14:49:23.596 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h4' already exists 2024-09-16T14:49:23.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.596+0000 7fb8fe5f3640 1 -- 172.21.15.189:0/3762230742 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb8e4036160 legacy=0x7fb8e4038620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:23.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.596+0000 7fb8fe5f3640 10 monclient: shutdown 2024-09-16T14:49:23.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.596+0000 7fb8fe5f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:23.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.596+0000 7fb8fe5f3640 1 -- 172.21.15.189:0/3762230742 >> v1:172.21.15.189:6789/0 conn(0x7fb8f809c550 legacy=0x7fb8f81cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:23.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.596+0000 7fb8fe5f3640 1 -- 172.21.15.189:0/3762230742 shutdown_connections 2024-09-16T14:49:23.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.596+0000 7fb8fe5f3640 1 -- 172.21.15.189:0/3762230742 >> 172.21.15.189:0/3762230742 conn(0x7fb8f80a3380 msgr2=0x7fb8f80a0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:23.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.596+0000 7fb8fe5f3640 1 -- 172.21.15.189:0/3762230742 shutdown_connections 2024-09-16T14:49:23.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.596+0000 7fb8fe5f3640 1 -- 172.21.15.189:0/3762230742 wait complete. 2024-09-16T14:49:23.613 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:23.613 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h5 host rack=r1 2024-09-16T14:49:23.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 10 monclient: get_monmap_and_config 2024-09-16T14:49:23.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 10 monclient: build_initial_monmap 2024-09-16T14:49:23.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 10 monclient: monmap: 2024-09-16T14:49:23.734 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:23.734 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:23.734 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:23.734751+0000 2024-09-16T14:49:23.734 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:23.734751+0000 2024-09-16T14:49:23.734 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:23.734 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:23.735 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:23.735 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:23.735 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:23.735 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:23.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 1 Processor -- start 2024-09-16T14:49:23.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 1 -- start start 2024-09-16T14:49:23.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 10 monclient: init 2024-09-16T14:49:23.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:23.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:49:23.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 10 monclient(hunting): picked mon.noname-a con 0x7f4af0070830 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:23.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 10 monclient(hunting): picked mon.noname-b con 0x7f4af0071320 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:23.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 10 monclient(hunting): picked mon.noname-c con 0x7f4af0134cf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:23.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af0135740 con 0x7f4af0070830 2024-09-16T14:49:23.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af0135160 con 0x7f4af0134cf0 2024-09-16T14:49:23.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af009d810 con 0x7f4af0071320 2024-09-16T14:49:23.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:23.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 10 monclient(hunting): authenticate will time out at 1345.850425s 2024-09-16T14:49:23.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aee59c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f4af0070830 legacy=0x7f4af0070c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:23.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aeed9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4af0134cf0 0x7f4af013d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53372/0 (socket says 172.21.15.189:53372) 2024-09-16T14:49:23.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aeed9d640 1 -- 172.21.15.189:0/4122935476 learned_addr learned my addr 172.21.15.189:0/4122935476 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:23.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 1 -- 172.21.15.189:0/4122935476 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404361508 0 0) 0x7f4af0135160 con 0x7f4af0134cf0 2024-09-16T14:49:23.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 10 monclient(hunting): _negotiate 2024-09-16T14:49:23.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:23.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:23.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 10 monclient(hunting): my global_id is 144475 2024-09-16T14:49:23.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 1 -- 172.21.15.189:0/4122935476 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4adc003530 con 0x7f4af0134cf0 2024-09-16T14:49:23.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 1 -- 172.21.15.189:0/4122935476 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1599955027 0 0) 0x7f4adc003530 con 0x7f4af0134cf0 2024-09-16T14:49:23.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 1 -- 172.21.15.189:0/4122935476 >> v1:172.21.15.26:6790/0 conn(0x7f4af0071320 legacy=0x7f4af0139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:23.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 1 -- 172.21.15.189:0/4122935476 >> v1:172.21.15.26:6789/0 conn(0x7f4af0070830 legacy=0x7f4af0070c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:23.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 10 monclient: _finish_hunting 0 2024-09-16T14:49:23.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 1 monclient: found mon.noname-c 2024-09-16T14:49:23.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:23.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:23.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 1 -- 172.21.15.189:0/4122935476 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af009f1c0 con 0x7f4af0134cf0 2024-09-16T14:49:23.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 10 monclient: _finish_auth 0 2024-09-16T14:49:23.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 10 monclient: _check_auth_tickets 2024-09-16T14:49:23.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:23.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 1 -- 172.21.15.189:0/4122935476 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4ae80019a0 con 0x7f4af0134cf0 2024-09-16T14:49:23.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:23.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:23.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 10 monclient: dump: 2024-09-16T14:49:23.743 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:23.744 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:23.744 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:23.744 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:23.744 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:23.744 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:23.744 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:23.744 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:23.744 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:23.744 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:23.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 5 monclient: authenticate success, global_id 144475 2024-09-16T14:49:23.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:23.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 1 -- 172.21.15.189:0/4122935476 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ae8001cc0 con 0x7f4af0134cf0 2024-09-16T14:49:23.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:23.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:23.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 1 -- 172.21.15.189:0/4122935476 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4ae8002090 con 0x7f4af0134cf0 2024-09-16T14:49:23.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 4 set_mon_vals no callback set 2024-09-16T14:49:23.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:23.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:23.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4aed59a640 10 monclient: dump: 2024-09-16T14:49:23.745 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:23.745 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:23.745 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:23.745 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:23.745 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:23.745 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:23.745 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:23.746 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:23.746 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:23.746 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:23.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 10 monclient: shutdown 2024-09-16T14:49:23.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:23.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 1 -- 172.21.15.189:0/4122935476 >> v1:172.21.15.189:6789/0 conn(0x7f4af0134cf0 legacy=0x7f4af013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:23.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 1 -- 172.21.15.189:0/4122935476 shutdown_connections 2024-09-16T14:49:23.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 1 -- 172.21.15.189:0/4122935476 >> 172.21.15.189:0/4122935476 conn(0x7f4af00a3380 msgr2=0x7f4af009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:23.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 1 -- 172.21.15.189:0/4122935476 shutdown_connections 2024-09-16T14:49:23.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.732+0000 7f4af4b47640 1 -- 172.21.15.189:0/4122935476 wait complete. 2024-09-16T14:49:23.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 monclient: build_initial_monmap 2024-09-16T14:49:23.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 monclient: monmap: 2024-09-16T14:49:23.746 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:23.747 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:23.747 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:23.747 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:23.747 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:23.747 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:23.747 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:23.747 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:23.747 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:23.747 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:23.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 1 Processor -- start 2024-09-16T14:49:23.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 1 -- start start 2024-09-16T14:49:23.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 monclient: init 2024-09-16T14:49:23.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:23.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 monclient(hunting): picked mon.noname-c con 0x7f4af0070830 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 monclient(hunting): picked mon.noname-a con 0x7f4af0071320 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 monclient(hunting): picked mon.noname-b con 0x7f4af0134cf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af006b320 con 0x7f4af0071320 2024-09-16T14:49:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af011da90 con 0x7f4af0134cf0 2024-09-16T14:49:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af01329e0 con 0x7f4af0070830 2024-09-16T14:49:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 monclient(hunting): authenticate will time out at 1345.853344s 2024-09-16T14:49:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4aedd9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f4af0071320 legacy=0x7f4af01ce3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4aeed9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4af0134cf0 0x7f4af01d1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53374/0 (socket says 172.21.15.189:53374) 2024-09-16T14:49:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4aeed9d640 1 -- 172.21.15.189:0/3623814465 learned_addr learned my addr 172.21.15.189:0/3623814465 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318237654 0 0) 0x7f4af011da90 con 0x7f4af0134cf0 2024-09-16T14:49:23.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:23.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:23.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:23.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient(hunting): my global_id is 144478 2024-09-16T14:49:23.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad0003530 con 0x7f4af0134cf0 2024-09-16T14:49:23.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2450195047 0 0) 0x7f4ad0003530 con 0x7f4af0134cf0 2024-09-16T14:49:23.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4af011da90 con 0x7f4af0134cf0 2024-09-16T14:49:23.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4ae80017c0 con 0x7f4af0134cf0 2024-09-16T14:49:23.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:23.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:23.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient(hunting): dump: 2024-09-16T14:49:23.750 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:23.750 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:23.750 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:23.750 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:23.750 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:23.750 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:23.750 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:23.751 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:23.751 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:23.751 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:23.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:23.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465349544 0 0) 0x7f4af01329e0 con 0x7f4af0070830 2024-09-16T14:49:23.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:23.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:23.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:23.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient(hunting): my global_id is 154268 2024-09-16T14:49:23.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad0003530 con 0x7f4af0070830 2024-09-16T14:49:23.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1935969275 0 0) 0x7f4af011da90 con 0x7f4af0134cf0 2024-09-16T14:49:23.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 >> v1:172.21.15.26:6790/0 conn(0x7f4af0070830 legacy=0x7f4af0130690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:23.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 >> v1:172.21.15.26:6789/0 conn(0x7f4af0071320 legacy=0x7f4af01ce3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:23.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:23.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 1 monclient: found mon.b 2024-09-16T14:49:23.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:23.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:23.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af011e050 con 0x7f4af0134cf0 2024-09-16T14:49:23.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:49:23.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:23.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:23.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 5 monclient: authenticate success, global_id 144478 2024-09-16T14:49:23.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 monclient: _renew_subs 2024-09-16T14:49:23.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:23.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 1 -- 172.21.15.189:0/3623814465 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af011d7b0 con 0x7f4af0134cf0 2024-09-16T14:49:23.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 monclient: _renew_subs 2024-09-16T14:49:23.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:23.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 1 -- 172.21.15.189:0/3623814465 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af0131410 con 0x7f4af0134cf0 2024-09-16T14:49:23.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ae8004f80 con 0x7f4af0134cf0 2024-09-16T14:49:23.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:23.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4ae8005290 con 0x7f4af0134cf0 2024-09-16T14:49:23.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:23.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:23.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 10 monclient: dump: 2024-09-16T14:49:23.754 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:23.755 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:23.755 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:23.755 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:23.755 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:23.755 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:23.755 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:23.755 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:23.755 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:23.755 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:23.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4aecd99640 4 set_mon_vals no callback set 2024-09-16T14:49:23.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4ae800fd00 con 0x7f4af0134cf0 2024-09-16T14:49:23.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:23.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:23.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:23.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4af4b47640 1 -- 172.21.15.189:0/3623814465 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad4005050 con 0x7f4af0134cf0 2024-09-16T14:49:23.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(187..187 src has 159..187) v4 ==== 7739+0+0 (unknown 2554894911 0 0) 0x7f4ae8044620 con 0x7f4af0134cf0 2024-09-16T14:49:23.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.736+0000 7f4aee59c640 10 monclient: get_auth_request con 0x7f4ad0036320 auth_method 0 2024-09-16T14:49:23.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.740+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4ae80130b0 con 0x7f4af0134cf0 2024-09-16T14:49:23.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.740+0000 7f4ae6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:23.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.740+0000 7f4ae6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:23.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.896+0000 7f4af4b47640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:23.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.896+0000 7f4af4b47640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:23.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.896+0000 7f4af4b47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:23.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:23.896+0000 7f4af4b47640 1 -- 172.21.15.189:0/3623814465 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f4ad4005880 con 0x7f4af0134cf0 2024-09-16T14:49:24.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.504+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}]=0 added bucket h5 type host to location {rack=r1} v188) v1 ==== 166+0+0 (unknown 1128572032 0 0) 0x7f4ae8015570 con 0x7f4af0134cf0 2024-09-16T14:49:24.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.504+0000 7f4ae6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:24.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.504+0000 7f4ae6ffd640 10 monclient: _finish_command 2 = system:0 added bucket h5 type host to location {rack=r1} 2024-09-16T14:49:24.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.600+0000 7f4af4b47640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:24.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.600+0000 7f4af4b47640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:24.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.600+0000 7f4af4b47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:24.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.600+0000 7f4af4b47640 1 -- 172.21.15.189:0/3623814465 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f4ad4005340 con 0x7f4af0134cf0 2024-09-16T14:49:24.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.600+0000 7f4ae6ffd640 1 -- 172.21.15.189:0/3623814465 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h5' already exists v188) v1 ==== 145+0+0 (unknown 2337465999 0 0) 0x7f4ae8014e70 con 0x7f4af0134cf0 2024-09-16T14:49:24.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.600+0000 7f4ae6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:24.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.600+0000 7f4ae6ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h5' already exists 2024-09-16T14:49:24.603 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h5' already exists 2024-09-16T14:49:24.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.604+0000 7f4af4b47640 1 -- 172.21.15.189:0/3623814465 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4ad0036320 legacy=0x7f4ad00387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:24.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.604+0000 7f4af4b47640 10 monclient: shutdown 2024-09-16T14:49:24.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.604+0000 7f4af4b47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:24.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.604+0000 7f4af4b47640 1 -- 172.21.15.189:0/3623814465 >> v1:172.21.15.189:6789/0 conn(0x7f4af0134cf0 legacy=0x7f4af01d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:24.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.604+0000 7f4af4b47640 1 -- 172.21.15.189:0/3623814465 shutdown_connections 2024-09-16T14:49:24.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.604+0000 7f4af4b47640 1 -- 172.21.15.189:0/3623814465 >> 172.21.15.189:0/3623814465 conn(0x7f4af00a3380 msgr2=0x7f4af0131910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:24.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.604+0000 7f4af4b47640 1 -- 172.21.15.189:0/3623814465 shutdown_connections 2024-09-16T14:49:24.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.604+0000 7f4af4b47640 1 -- 172.21.15.189:0/3623814465 wait complete. 2024-09-16T14:49:24.620 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:24.621 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h6 host rack=r1 2024-09-16T14:49:24.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 10 monclient: get_monmap_and_config 2024-09-16T14:49:24.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 10 monclient: build_initial_monmap 2024-09-16T14:49:24.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 10 monclient: monmap: 2024-09-16T14:49:24.734 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:24.734 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:24.734 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:24.735058+0000 2024-09-16T14:49:24.734 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:24.735058+0000 2024-09-16T14:49:24.735 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:24.735 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:24.735 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:24.735 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:24.735 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:24.735 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 1 Processor -- start 2024-09-16T14:49:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 1 -- start start 2024-09-16T14:49:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 10 monclient: init 2024-09-16T14:49:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 10 monclient(hunting): picked mon.noname-a con 0x7f2f14131630 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 10 monclient(hunting): picked mon.noname-c con 0x7f2f1409edd0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 10 monclient(hunting): picked mon.noname-b con 0x7f2f1409e950 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f1409b920 con 0x7f2f14131630 2024-09-16T14:49:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f1409bc00 con 0x7f2f1409edd0 2024-09-16T14:49:24.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f1409cba0 con 0x7f2f1409e950 2024-09-16T14:49:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 10 monclient(hunting): authenticate will time out at 1346.851240s 2024-09-16T14:49:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19c37640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2f1409edd0 0x7f2f14136b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53378/0 (socket says 172.21.15.189:53378) 2024-09-16T14:49:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19c37640 1 -- 172.21.15.189:0/2290105805 learned_addr learned my addr 172.21.15.189:0/2290105805 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1a438640 1 -- 172.21.15.189:0/2290105805 >> v1:172.21.15.26:6789/0 conn(0x7f2f14131630 legacy=0x7f2f14131a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 1 -- 172.21.15.189:0/2290105805 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087671723 0 0) 0x7f2f1409bc00 con 0x7f2f1409edd0 2024-09-16T14:49:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 10 monclient(hunting): _negotiate 2024-09-16T14:49:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 10 monclient(hunting): my global_id is 144481 2024-09-16T14:49:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 1 -- 172.21.15.189:0/2290105805 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f00005740 con 0x7f2f1409edd0 2024-09-16T14:49:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 1 -- 172.21.15.189:0/2290105805 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 320702691 0 0) 0x7f2f00005740 con 0x7f2f1409edd0 2024-09-16T14:49:24.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 1 -- 172.21.15.189:0/2290105805 >> v1:172.21.15.26:6790/0 conn(0x7f2f1409e950 legacy=0x7f2f1413a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 1 -- 172.21.15.189:0/2290105805 >> v1:172.21.15.26:6789/0 conn(0x7f2f14131630 legacy=0x7f2f14131a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 10 monclient: _finish_hunting 0 2024-09-16T14:49:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 1 monclient: found mon.noname-c 2024-09-16T14:49:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 1 -- 172.21.15.189:0/2290105805 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1409b420 con 0x7f2f1409edd0 2024-09-16T14:49:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 10 monclient: _finish_auth 0 2024-09-16T14:49:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 10 monclient: _check_auth_tickets 2024-09-16T14:49:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 5 monclient: authenticate success, global_id 144481 2024-09-16T14:49:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f1c69c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:24.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.732+0000 7f2f19436640 1 -- 172.21.15.189:0/2290105805 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2f04001620 con 0x7f2f1409edd0 2024-09-16T14:49:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f19436640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f19436640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f19436640 10 monclient: dump: 2024-09-16T14:49:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:24.744 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f19436640 1 -- 172.21.15.189:0/2290105805 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f04001940 con 0x7f2f1409edd0 2024-09-16T14:49:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f19436640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f19436640 1 -- 172.21.15.189:0/2290105805 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2f04001cd0 con 0x7f2f1409edd0 2024-09-16T14:49:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 4 set_mon_vals no callback set 2024-09-16T14:49:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f19436640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f19436640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f19436640 10 monclient: dump: 2024-09-16T14:49:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:24.746 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient: shutdown 2024-09-16T14:49:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 1 -- 172.21.15.189:0/2290105805 >> v1:172.21.15.189:6789/0 conn(0x7f2f1409edd0 legacy=0x7f2f14136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 1 -- 172.21.15.189:0/2290105805 shutdown_connections 2024-09-16T14:49:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 1 -- 172.21.15.189:0/2290105805 >> 172.21.15.189:0/2290105805 conn(0x7f2f140a3380 msgr2=0x7f2f1409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 1 -- 172.21.15.189:0/2290105805 shutdown_connections 2024-09-16T14:49:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 1 -- 172.21.15.189:0/2290105805 wait complete. 2024-09-16T14:49:24.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient: build_initial_monmap 2024-09-16T14:49:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient: monmap: 2024-09-16T14:49:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:24.747 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 1 Processor -- start 2024-09-16T14:49:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 1 -- start start 2024-09-16T14:49:24.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient: init 2024-09-16T14:49:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:49:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient(hunting): picked mon.noname-a con 0x7f2f1409e950 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient(hunting): picked mon.noname-c con 0x7f2f1409edd0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient(hunting): picked mon.noname-b con 0x7f2f14131630 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f140a10b0 con 0x7f2f1409e950 2024-09-16T14:49:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f14121ed0 con 0x7f2f14131630 2024-09-16T14:49:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f1406b320 con 0x7f2f1409edd0 2024-09-16T14:49:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:24.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient(hunting): authenticate will time out at 1346.854032s 2024-09-16T14:49:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1ac39640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2f14131630 0x7f2f141d1880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53382/0 (socket says 172.21.15.189:53382) 2024-09-16T14:49:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1ac39640 1 -- 172.21.15.189:0/2296232169 learned_addr learned my addr 172.21.15.189:0/2296232169 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1a438640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2f1409e950 legacy=0x7f2f14130bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 1 -- 172.21.15.189:0/2296232169 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963324607 0 0) 0x7f2f14121ed0 con 0x7f2f14131630 2024-09-16T14:49:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 10 monclient(hunting): my global_id is 144484 2024-09-16T14:49:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 1 -- 172.21.15.189:0/2296232169 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef8003530 con 0x7f2f14131630 2024-09-16T14:49:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 1 -- 172.21.15.189:0/2296232169 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2317938645 0 0) 0x7f2ef8003530 con 0x7f2f14131630 2024-09-16T14:49:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 1 -- 172.21.15.189:0/2296232169 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f14121ed0 con 0x7f2f14131630 2024-09-16T14:49:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 1 -- 172.21.15.189:0/2296232169 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2f0c001c20 con 0x7f2f14131630 2024-09-16T14:49:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 10 monclient(hunting): dump: 2024-09-16T14:49:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:24.750 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 1 -- 172.21.15.189:0/2296232169 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 280890766 0 0) 0x7f2f14121ed0 con 0x7f2f14131630 2024-09-16T14:49:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 1 -- 172.21.15.189:0/2296232169 >> v1:172.21.15.26:6790/0 conn(0x7f2f1409edd0 legacy=0x7f2f141ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 1 -- 172.21.15.189:0/2296232169 >> v1:172.21.15.26:6789/0 conn(0x7f2f1409e950 legacy=0x7f2f14130bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 1 monclient: found mon.b 2024-09-16T14:49:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 1 -- 172.21.15.189:0/2296232169 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f141235c0 con 0x7f2f14131630 2024-09-16T14:49:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 10 monclient: _finish_auth 0 2024-09-16T14:49:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 5 monclient: authenticate success, global_id 144484 2024-09-16T14:49:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient: _renew_subs 2024-09-16T14:49:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 1 -- 172.21.15.189:0/2296232169 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f14122100 con 0x7f2f14131630 2024-09-16T14:49:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient: _renew_subs 2024-09-16T14:49:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 1 -- 172.21.15.189:0/2296232169 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f141211f0 con 0x7f2f14131630 2024-09-16T14:49:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f1c69c640 1 -- 172.21.15.189:0/2296232169 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f1406fb40 con 0x7f2f14131630 2024-09-16T14:49:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 1 -- 172.21.15.189:0/2296232169 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f0c001ff0 con 0x7f2f14131630 2024-09-16T14:49:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 1 -- 172.21.15.189:0/2296232169 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2f0c003df0 con 0x7f2f14131630 2024-09-16T14:49:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f0affd640 10 monclient: dump: 2024-09-16T14:49:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:24.753 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.736+0000 7f2f18c35640 4 set_mon_vals no callback set 2024-09-16T14:49:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.740+0000 7f2f0affd640 1 -- 172.21.15.189:0/2296232169 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2f0c003f50 con 0x7f2f14131630 2024-09-16T14:49:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.740+0000 7f2f0affd640 1 -- 172.21.15.189:0/2296232169 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(188..188 src has 159..188) v4 ==== 8077+0+0 (unknown 1268971511 0 0) 0x7f2f0c044560 con 0x7f2f14131630 2024-09-16T14:49:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.740+0000 7f2f1a438640 10 monclient: get_auth_request con 0x7f2ef8035e90 auth_method 0 2024-09-16T14:49:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.744+0000 7f2f0affd640 1 -- 172.21.15.189:0/2296232169 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2f0c0149d0 con 0x7f2f14131630 2024-09-16T14:49:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.744+0000 7f2f0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.744+0000 7f2f0affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:24.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.896+0000 7f2f1c69c640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:24.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.896+0000 7f2f1c69c640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:24.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.896+0000 7f2f1c69c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:24.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:24.896+0000 7f2f1c69c640 1 -- 172.21.15.189:0/2296232169 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f2f141320f0 con 0x7f2f14131630 2024-09-16T14:49:25.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.512+0000 7f2f0affd640 1 -- 172.21.15.189:0/2296232169 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}]=0 added bucket h6 type host to location {rack=r1} v189) v1 ==== 166+0+0 (unknown 1734554102 0 0) 0x7f2f0c012e40 con 0x7f2f14131630 2024-09-16T14:49:25.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.512+0000 7f2f0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:25.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.512+0000 7f2f0affd640 10 monclient: _finish_command 2 = system:0 added bucket h6 type host to location {rack=r1} 2024-09-16T14:49:25.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.608+0000 7f2f1c69c640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:25.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.608+0000 7f2f1c69c640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:25.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.608+0000 7f2f1c69c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:25.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.608+0000 7f2f1c69c640 1 -- 172.21.15.189:0/2296232169 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f2f141312c0 con 0x7f2f14131630 2024-09-16T14:49:25.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.612+0000 7f2f0affd640 1 -- 172.21.15.189:0/2296232169 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h6' already exists v189) v1 ==== 145+0+0 (unknown 3672963875 0 0) 0x7f2f0c015300 con 0x7f2f14131630 2024-09-16T14:49:25.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.612+0000 7f2f0affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:25.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.612+0000 7f2f0affd640 10 monclient: _finish_command 3 = system:0 bucket 'h6' already exists 2024-09-16T14:49:25.614 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h6' already exists 2024-09-16T14:49:25.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.612+0000 7f2f1c69c640 1 -- 172.21.15.189:0/2296232169 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2ef8035e90 legacy=0x7f2ef8038350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:25.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.612+0000 7f2f1c69c640 10 monclient: shutdown 2024-09-16T14:49:25.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.612+0000 7f2f1c69c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:25.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.612+0000 7f2f1c69c640 1 -- 172.21.15.189:0/2296232169 >> v1:172.21.15.189:6789/0 conn(0x7f2f14131630 legacy=0x7f2f141d1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:25.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.612+0000 7f2f1c69c640 1 -- 172.21.15.189:0/2296232169 shutdown_connections 2024-09-16T14:49:25.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.612+0000 7f2f1c69c640 1 -- 172.21.15.189:0/2296232169 >> 172.21.15.189:0/2296232169 conn(0x7f2f140a3380 msgr2=0x7f2f141b7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:25.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.612+0000 7f2f1c69c640 1 -- 172.21.15.189:0/2296232169 shutdown_connections 2024-09-16T14:49:25.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.612+0000 7f2f1c69c640 1 -- 172.21.15.189:0/2296232169 wait complete. 2024-09-16T14:49:25.631 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:25.631 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h7 host rack=r1 2024-09-16T14:49:25.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 10 monclient: get_monmap_and_config 2024-09-16T14:49:25.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 10 monclient: build_initial_monmap 2024-09-16T14:49:25.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 10 monclient: monmap: 2024-09-16T14:49:25.751 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:25.751 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:25.751 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:25.750819+0000 2024-09-16T14:49:25.751 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:25.750819+0000 2024-09-16T14:49:25.751 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:25.751 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:25.751 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:25.751 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:25.751 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:25.751 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:25.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 1 Processor -- start 2024-09-16T14:49:25.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 1 -- start start 2024-09-16T14:49:25.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 10 monclient: init 2024-09-16T14:49:25.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:25.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:49:25.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 10 monclient(hunting): picked mon.noname-a con 0x7f9d901393b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:25.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 10 monclient(hunting): picked mon.noname-b con 0x7f9d90139790 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:25.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 10 monclient(hunting): picked mon.noname-c con 0x7f9d9009e510 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:25.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9009aad0 con 0x7f9d901393b0 2024-09-16T14:49:25.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9009d030 con 0x7f9d9009e510 2024-09-16T14:49:25.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9009edd0 con 0x7f9d90139790 2024-09-16T14:49:25.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:25.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 10 monclient(hunting): authenticate will time out at 1347.866498s 2024-09-16T14:49:25.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8ffff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f9d901393b0 legacy=0x7f9d9009d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:25.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d94a9c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9d9009e510 0x7f9d90141fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53384/0 (socket says 172.21.15.189:53384) 2024-09-16T14:49:25.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d94a9c640 1 -- 172.21.15.189:0/2812996401 learned_addr learned my addr 172.21.15.189:0/2812996401 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:25.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 1 -- 172.21.15.189:0/2812996401 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6538017 0 0) 0x7f9d9009d030 con 0x7f9d9009e510 2024-09-16T14:49:25.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:25.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:25.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:25.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 10 monclient(hunting): my global_id is 144487 2024-09-16T14:49:25.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 1 -- 172.21.15.189:0/2812996401 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d7c003530 con 0x7f9d9009e510 2024-09-16T14:49:25.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 1 -- 172.21.15.189:0/2812996401 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2843645513 0 0) 0x7f9d7c003530 con 0x7f9d9009e510 2024-09-16T14:49:25.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 1 -- 172.21.15.189:0/2812996401 >> v1:172.21.15.26:6790/0 conn(0x7f9d90139790 legacy=0x7f9d9009de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:25.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 1 -- 172.21.15.189:0/2812996401 >> v1:172.21.15.26:6789/0 conn(0x7f9d901393b0 legacy=0x7f9d9009d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:25.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:25.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 1 monclient: found mon.noname-c 2024-09-16T14:49:25.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:25.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:25.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 1 -- 172.21.15.189:0/2812996401 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9009a7f0 con 0x7f9d9009e510 2024-09-16T14:49:25.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 10 monclient: _finish_auth 0 2024-09-16T14:49:25.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:25.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:25.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 1 -- 172.21.15.189:0/2812996401 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9d88001620 con 0x7f9d9009e510 2024-09-16T14:49:25.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:25.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:25.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 10 monclient: dump: 2024-09-16T14:49:25.759 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:25.759 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:25.759 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:25.760 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:25.760 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:25.760 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:25.760 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:25.760 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:25.760 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:25.760 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:25.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 5 monclient: authenticate success, global_id 144487 2024-09-16T14:49:25.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:25.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 1 -- 172.21.15.189:0/2812996401 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d88001940 con 0x7f9d9009e510 2024-09-16T14:49:25.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:25.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:25.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 1 -- 172.21.15.189:0/2812996401 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9d88001cd0 con 0x7f9d9009e510 2024-09-16T14:49:25.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:25.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d964ff640 4 set_mon_vals no callback set 2024-09-16T14:49:25.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:25.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.748+0000 7f9d8effd640 10 monclient: dump: 2024-09-16T14:49:25.761 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:25.761 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:25.761 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:25.761 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:25.761 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:25.762 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:25.762 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:25.762 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:25.762 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:25.762 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:25.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient: shutdown 2024-09-16T14:49:25.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:25.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 1 -- 172.21.15.189:0/2812996401 >> v1:172.21.15.189:6789/0 conn(0x7f9d9009e510 legacy=0x7f9d90141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:25.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 1 -- 172.21.15.189:0/2812996401 shutdown_connections 2024-09-16T14:49:25.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 1 -- 172.21.15.189:0/2812996401 >> 172.21.15.189:0/2812996401 conn(0x7f9d900a3380 msgr2=0x7f9d9009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:25.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 1 -- 172.21.15.189:0/2812996401 shutdown_connections 2024-09-16T14:49:25.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 1 -- 172.21.15.189:0/2812996401 wait complete. 2024-09-16T14:49:25.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient: build_initial_monmap 2024-09-16T14:49:25.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient: monmap: 2024-09-16T14:49:25.763 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:25.763 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:25.763 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:25.763 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:25.763 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:25.763 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:25.763 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:25.763 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:25.763 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:25.763 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:25.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 1 Processor -- start 2024-09-16T14:49:25.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 1 -- start start 2024-09-16T14:49:25.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient: init 2024-09-16T14:49:25.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:25.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:49:25.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient(hunting): picked mon.noname-c con 0x7f9d9009e510 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:25.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient(hunting): picked mon.noname-b con 0x7f9d901393b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:25.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient(hunting): picked mon.noname-a con 0x7f9d90139790 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:25.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d90122160 con 0x7f9d90139790 2024-09-16T14:49:25.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9006b320 con 0x7f9d901393b0 2024-09-16T14:49:25.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d90123ac0 con 0x7f9d9009e510 2024-09-16T14:49:25.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:25.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient(hunting): authenticate will time out at 1347.869692s 2024-09-16T14:49:25.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8f7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9d901393b0 0x7f9d90136100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53388/0 (socket says 172.21.15.189:53388) 2024-09-16T14:49:25.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8f7fe640 1 -- 172.21.15.189:0/2909884737 learned_addr learned my addr 172.21.15.189:0/2909884737 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:25.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d94a9c640 1 -- 172.21.15.189:0/2909884737 >> v1:172.21.15.26:6789/0 conn(0x7f9d90139790 legacy=0x7f9d901d5750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:25.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615933346 0 0) 0x7f9d9006b320 con 0x7f9d901393b0 2024-09-16T14:49:25.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:49:25.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:25.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:25.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient(hunting): my global_id is 144490 2024-09-16T14:49:25.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d70003530 con 0x7f9d901393b0 2024-09-16T14:49:25.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1651095365 0 0) 0x7f9d70003530 con 0x7f9d901393b0 2024-09-16T14:49:25.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d9006b320 con 0x7f9d901393b0 2024-09-16T14:49:25.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9d78001940 con 0x7f9d901393b0 2024-09-16T14:49:25.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:25.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:25.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient(hunting): dump: 2024-09-16T14:49:25.765 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:25.765 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:25.766 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:25.766 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:25.766 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:25.766 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:25.766 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:25.766 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:25.766 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:25.766 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:25.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:25.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452131512 0 0) 0x7f9d90123ac0 con 0x7f9d9009e510 2024-09-16T14:49:25.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:49:25.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:25.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:25.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient(hunting): my global_id is 154280 2024-09-16T14:49:25.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d70003530 con 0x7f9d9009e510 2024-09-16T14:49:25.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 760148772 0 0) 0x7f9d9006b320 con 0x7f9d901393b0 2024-09-16T14:49:25.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 >> v1:172.21.15.26:6790/0 conn(0x7f9d9009e510 legacy=0x7f9d901cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:25.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 >> v1:172.21.15.26:6789/0 conn(0x7f9d90139790 legacy=0x7f9d901d5750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:25.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:49:25.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 1 monclient: found mon.b 2024-09-16T14:49:25.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:25.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:25.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d900a1ec0 con 0x7f9d901393b0 2024-09-16T14:49:25.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient: _finish_auth 0 2024-09-16T14:49:25.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:49:25.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:25.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 5 monclient: authenticate success, global_id 144490 2024-09-16T14:49:25.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient: _renew_subs 2024-09-16T14:49:25.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:25.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 1 -- 172.21.15.189:0/2909884737 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d90137760 con 0x7f9d901393b0 2024-09-16T14:49:25.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient: _renew_subs 2024-09-16T14:49:25.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:25.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 1 -- 172.21.15.189:0/2909884737 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d90071280 con 0x7f9d901393b0 2024-09-16T14:49:25.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d780036a0 con 0x7f9d901393b0 2024-09-16T14:49:25.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:25.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9d780039b0 con 0x7f9d901393b0 2024-09-16T14:49:25.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:25.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:25.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8e7fc640 4 set_mon_vals no callback set 2024-09-16T14:49:25.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 10 monclient: dump: 2024-09-16T14:49:25.769 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:25.769 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:25.769 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:25.769 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:25.769 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:25.769 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:25.769 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:25.769 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:25.769 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:25.769 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:25.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9d78003b90 con 0x7f9d901393b0 2024-09-16T14:49:25.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:25.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:25.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:25.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d964ff640 1 -- 172.21.15.189:0/2909884737 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d74005050 con 0x7f9d901393b0 2024-09-16T14:49:25.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(189..189 src has 159..189) v4 ==== 8287+0+0 (unknown 3725545976 0 0) 0x7f9d780444e0 con 0x7f9d901393b0 2024-09-16T14:49:25.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.752+0000 7f9d8ffff640 10 monclient: get_auth_request con 0x7f9d700361d0 auth_method 0 2024-09-16T14:49:25.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.760+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9d780147b0 con 0x7f9d901393b0 2024-09-16T14:49:25.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.760+0000 7f9d8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:25.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.760+0000 7f9d8cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:25.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.920+0000 7f9d964ff640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:25.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.920+0000 7f9d964ff640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:25.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.920+0000 7f9d964ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:25.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:25.920+0000 7f9d964ff640 1 -- 172.21.15.189:0/2909884737 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f9d74005880 con 0x7f9d901393b0 2024-09-16T14:49:26.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.524+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}]=0 added bucket h7 type host to location {rack=r1} v190) v1 ==== 166+0+0 (unknown 1938116697 0 0) 0x7f9d78012c20 con 0x7f9d901393b0 2024-09-16T14:49:26.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.524+0000 7f9d8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:26.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.524+0000 7f9d8cff9640 10 monclient: _finish_command 2 = system:0 added bucket h7 type host to location {rack=r1} 2024-09-16T14:49:26.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.624+0000 7f9d964ff640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:26.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.624+0000 7f9d964ff640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:26.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.624+0000 7f9d964ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:26.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.624+0000 7f9d964ff640 1 -- 172.21.15.189:0/2909884737 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f9d74005340 con 0x7f9d901393b0 2024-09-16T14:49:26.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.628+0000 7f9d8cff9640 1 -- 172.21.15.189:0/2909884737 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h7' already exists v190) v1 ==== 145+0+0 (unknown 165701213 0 0) 0x7f9d78014580 con 0x7f9d901393b0 2024-09-16T14:49:26.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.628+0000 7f9d8cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:26.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.628+0000 7f9d8cff9640 10 monclient: _finish_command 3 = system:0 bucket 'h7' already exists 2024-09-16T14:49:26.630 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h7' already exists 2024-09-16T14:49:26.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.628+0000 7f9d964ff640 1 -- 172.21.15.189:0/2909884737 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9d700361d0 legacy=0x7f9d70038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:26.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.628+0000 7f9d964ff640 10 monclient: shutdown 2024-09-16T14:49:26.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.628+0000 7f9d964ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:26.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.628+0000 7f9d964ff640 1 -- 172.21.15.189:0/2909884737 >> v1:172.21.15.189:6789/0 conn(0x7f9d901393b0 legacy=0x7f9d90136100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:26.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.628+0000 7f9d964ff640 1 -- 172.21.15.189:0/2909884737 shutdown_connections 2024-09-16T14:49:26.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.628+0000 7f9d964ff640 1 -- 172.21.15.189:0/2909884737 >> 172.21.15.189:0/2909884737 conn(0x7f9d900a3380 msgr2=0x7f9d9009bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:26.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.628+0000 7f9d964ff640 1 -- 172.21.15.189:0/2909884737 shutdown_connections 2024-09-16T14:49:26.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.628+0000 7f9d964ff640 1 -- 172.21.15.189:0/2909884737 wait complete. 2024-09-16T14:49:26.647 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:26.647 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h8 host rack=r1 2024-09-16T14:49:26.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f691d53a640 10 monclient: get_monmap_and_config 2024-09-16T14:49:26.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f691d53a640 10 monclient: build_initial_monmap 2024-09-16T14:49:26.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f691d53a640 10 monclient: monmap: 2024-09-16T14:49:26.766 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:26.766 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:26.767 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:26.767272+0000 2024-09-16T14:49:26.767 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:26.767272+0000 2024-09-16T14:49:26.767 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:26.767 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:26.767 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:26.767 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:26.767 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:26.767 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:26.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f691d53a640 1 Processor -- start 2024-09-16T14:49:26.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f691d53a640 1 -- start start 2024-09-16T14:49:26.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f691d53a640 10 monclient: init 2024-09-16T14:49:26.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f691d53a640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:26.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f691d53a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:26.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f691d53a640 10 monclient(hunting): picked mon.noname-c con 0x7f691809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:26.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f691d53a640 10 monclient(hunting): picked mon.noname-a con 0x7f6918132bf0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:26.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f691d53a640 10 monclient(hunting): picked mon.noname-b con 0x7f6918139150 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:26.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f691d53a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691809abf0 con 0x7f6918132bf0 2024-09-16T14:49:26.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f691d53a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691809ae60 con 0x7f691809cb80 2024-09-16T14:49:26.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f691d53a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6918133060 con 0x7f6918139150 2024-09-16T14:49:26.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f691d53a640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:26.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f691d53a640 10 monclient(hunting): authenticate will time out at 1348.883665s 2024-09-16T14:49:26.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f690ffff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f6918132bf0 legacy=0x7f6918137a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:26.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f6916ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f691809cb80 0x7f691809cf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53398/0 (socket says 172.21.15.189:53398) 2024-09-16T14:49:26.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.764+0000 7f6916ffd640 1 -- 172.21.15.189:0/3304646411 learned_addr learned my addr 172.21.15.189:0/3304646411 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:26.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 1 -- 172.21.15.189:0/3304646411 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454472464 0 0) 0x7f691809ae60 con 0x7f691809cb80 2024-09-16T14:49:26.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:26.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:26.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 10 monclient(hunting): my global_id is 144493 2024-09-16T14:49:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 1 -- 172.21.15.189:0/3304646411 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6904003530 con 0x7f691809cb80 2024-09-16T14:49:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 1 -- 172.21.15.189:0/3304646411 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1907854512 0 0) 0x7f6904003530 con 0x7f691809cb80 2024-09-16T14:49:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 1 -- 172.21.15.189:0/3304646411 >> v1:172.21.15.26:6790/0 conn(0x7f6918139150 legacy=0x7f691813b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 1 -- 172.21.15.189:0/3304646411 >> v1:172.21.15.26:6789/0 conn(0x7f6918132bf0 legacy=0x7f6918137a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 10 monclient: _finish_hunting 0 2024-09-16T14:49:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 1 monclient: found mon.noname-c 2024-09-16T14:49:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 1 -- 172.21.15.189:0/3304646411 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691809b140 con 0x7f691809cb80 2024-09-16T14:49:26.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 10 monclient: _finish_auth 0 2024-09-16T14:49:26.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:26.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:26.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 1 -- 172.21.15.189:0/3304646411 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6908001620 con 0x7f691809cb80 2024-09-16T14:49:26.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:26.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:26.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 10 monclient: dump: 2024-09-16T14:49:26.777 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:26.777 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:26.777 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:26.777 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:26.777 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:26.778 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:26.778 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:26.778 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:26.778 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:26.778 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:26.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 5 monclient: authenticate success, global_id 144493 2024-09-16T14:49:26.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:26.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 1 -- 172.21.15.189:0/3304646411 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69080018d0 con 0x7f691809cb80 2024-09-16T14:49:26.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:26.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 1 -- 172.21.15.189:0/3304646411 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6908001ca0 con 0x7f691809cb80 2024-09-16T14:49:26.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:26.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:26.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f69167fc640 10 monclient: dump: 2024-09-16T14:49:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:26.779 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 4 set_mon_vals no callback set 2024-09-16T14:49:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 10 monclient: shutdown 2024-09-16T14:49:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:26.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 1 -- 172.21.15.189:0/3304646411 >> v1:172.21.15.189:6789/0 conn(0x7f691809cb80 legacy=0x7f691809cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:26.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 1 -- 172.21.15.189:0/3304646411 shutdown_connections 2024-09-16T14:49:26.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 1 -- 172.21.15.189:0/3304646411 >> 172.21.15.189:0/3304646411 conn(0x7f691809fed0 msgr2=0x7f69180a02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:26.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 1 -- 172.21.15.189:0/3304646411 shutdown_connections 2024-09-16T14:49:26.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 1 -- 172.21.15.189:0/3304646411 wait complete. 2024-09-16T14:49:26.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 10 monclient: build_initial_monmap 2024-09-16T14:49:26.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 10 monclient: monmap: 2024-09-16T14:49:26.780 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:26.780 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:26.780 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:26.780 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:26.780 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:26.780 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:26.781 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:26.781 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:26.781 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:26.781 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:26.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 1 Processor -- start 2024-09-16T14:49:26.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 1 -- start start 2024-09-16T14:49:26.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 10 monclient: init 2024-09-16T14:49:26.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:26.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:26.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 10 monclient(hunting): picked mon.noname-a con 0x7f691809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:26.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 10 monclient(hunting): picked mon.noname-b con 0x7f6918132bf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:26.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 10 monclient(hunting): picked mon.noname-c con 0x7f6918139150 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:26.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691812b4f0 con 0x7f691809cb80 2024-09-16T14:49:26.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69180a12d0 con 0x7f6918132bf0 2024-09-16T14:49:26.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691812ba20 con 0x7f6918139150 2024-09-16T14:49:26.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:26.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 10 monclient(hunting): authenticate will time out at 1348.886513s 2024-09-16T14:49:26.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f6916ffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f691809cb80 legacy=0x7f69181c9550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:26.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690ffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6918132bf0 0x7f69181294a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53410/0 (socket says 172.21.15.189:53410) 2024-09-16T14:49:26.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690ffff640 1 -- 172.21.15.189:0/3988949223 learned_addr learned my addr 172.21.15.189:0/3988949223 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:26.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272047946 0 0) 0x7f69180a12d0 con 0x7f6918132bf0 2024-09-16T14:49:26.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:26.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:26.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:26.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient(hunting): my global_id is 144496 2024-09-16T14:49:26.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68f8003530 con 0x7f6918132bf0 2024-09-16T14:49:26.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 959187490 0 0) 0x7f68f8003530 con 0x7f6918132bf0 2024-09-16T14:49:26.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69180a12d0 con 0x7f6918132bf0 2024-09-16T14:49:26.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6900004190 con 0x7f6918132bf0 2024-09-16T14:49:26.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:26.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:26.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient(hunting): dump: 2024-09-16T14:49:26.784 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:26.784 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:26.784 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:26.784 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:26.784 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:26.784 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:26.784 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:26.784 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:26.784 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:26.784 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:26.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:26.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252982963 0 0) 0x7f691812ba20 con 0x7f6918139150 2024-09-16T14:49:26.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:26.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:26.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:26.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient(hunting): my global_id is 154286 2024-09-16T14:49:26.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68f8003530 con 0x7f6918139150 2024-09-16T14:49:26.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1127235874 0 0) 0x7f69180a12d0 con 0x7f6918132bf0 2024-09-16T14:49:26.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 >> v1:172.21.15.26:6790/0 conn(0x7f6918139150 legacy=0x7f69181d15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:26.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 >> v1:172.21.15.26:6789/0 conn(0x7f691809cb80 legacy=0x7f69181c9550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:26.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:49:26.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 1 monclient: found mon.b 2024-09-16T14:49:26.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:26.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:26.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691812bd00 con 0x7f6918132bf0 2024-09-16T14:49:26.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:49:26.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:26.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:26.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 5 monclient: authenticate success, global_id 144496 2024-09-16T14:49:26.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 10 monclient: _renew_subs 2024-09-16T14:49:26.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:26.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 1 -- 172.21.15.189:0/3988949223 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691812a500 con 0x7f6918132bf0 2024-09-16T14:49:26.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 10 monclient: _renew_subs 2024-09-16T14:49:26.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:26.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f691d53a640 1 -- 172.21.15.189:0/3988949223 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6918131be0 con 0x7f6918132bf0 2024-09-16T14:49:26.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6900004440 con 0x7f6918132bf0 2024-09-16T14:49:26.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:26.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f69000047d0 con 0x7f6918132bf0 2024-09-16T14:49:26.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:26.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:26.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f690f7fe640 10 monclient: dump: 2024-09-16T14:49:26.787 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:26.787 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:26.787 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:26.787 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:26.787 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:26.787 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:26.787 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:26.787 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:26.788 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:26.788 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:26.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.768+0000 7f6915ffb640 4 set_mon_vals no callback set 2024-09-16T14:49:26.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.772+0000 7f691d53a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:26.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.772+0000 7f691d53a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:26.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.772+0000 7f691d53a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:26.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.772+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f69000049b0 con 0x7f6918132bf0 2024-09-16T14:49:26.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.772+0000 7f691d53a640 1 -- 172.21.15.189:0/3988949223 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68fc005050 con 0x7f6918132bf0 2024-09-16T14:49:26.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.772+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(190..190 src has 164..190) v4 ==== 8497+0+0 (unknown 489497426 0 0) 0x7f6900044620 con 0x7f6918132bf0 2024-09-16T14:49:26.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.772+0000 7f6916ffd640 10 monclient: get_auth_request con 0x7f68f80360e0 auth_method 0 2024-09-16T14:49:26.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.776+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f69000051d0 con 0x7f6918132bf0 2024-09-16T14:49:26.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.776+0000 7f690f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:26.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.776+0000 7f690f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:26.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.936+0000 7f691d53a640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:26.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.936+0000 7f691d53a640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:26.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.936+0000 7f691d53a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:26.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:26.936+0000 7f691d53a640 1 -- 172.21.15.189:0/3988949223 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f68fc005880 con 0x7f6918132bf0 2024-09-16T14:49:27.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.532+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}]=0 added bucket h8 type host to location {rack=r1} v191) v1 ==== 166+0+0 (unknown 3727796080 0 0) 0x7f6900012c90 con 0x7f6918132bf0 2024-09-16T14:49:27.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.532+0000 7f690f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:27.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.532+0000 7f690f7fe640 10 monclient: _finish_command 2 = system:0 added bucket h8 type host to location {rack=r1} 2024-09-16T14:49:27.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.636+0000 7f691d53a640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:27.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.636+0000 7f691d53a640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:27.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.636+0000 7f691d53a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:27.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.636+0000 7f691d53a640 1 -- 172.21.15.189:0/3988949223 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f68fc005340 con 0x7f6918132bf0 2024-09-16T14:49:27.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.636+0000 7f690f7fe640 1 -- 172.21.15.189:0/3988949223 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h8' already exists v191) v1 ==== 145+0+0 (unknown 973977455 0 0) 0x7f6900014410 con 0x7f6918132bf0 2024-09-16T14:49:27.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.636+0000 7f690f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:27.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.636+0000 7f690f7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h8' already exists 2024-09-16T14:49:27.640 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h8' already exists 2024-09-16T14:49:27.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.640+0000 7f691d53a640 1 -- 172.21.15.189:0/3988949223 >> v1:172.21.15.189:6812/3196161582 conn(0x7f68f80360e0 legacy=0x7f68f80385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:27.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.640+0000 7f691d53a640 10 monclient: shutdown 2024-09-16T14:49:27.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.640+0000 7f691d53a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:27.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.640+0000 7f691d53a640 1 -- 172.21.15.189:0/3988949223 >> v1:172.21.15.189:6789/0 conn(0x7f6918132bf0 legacy=0x7f69181294a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:27.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.640+0000 7f691d53a640 1 -- 172.21.15.189:0/3988949223 shutdown_connections 2024-09-16T14:49:27.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.640+0000 7f691d53a640 1 -- 172.21.15.189:0/3988949223 >> 172.21.15.189:0/3988949223 conn(0x7f691809fed0 msgr2=0x7f69180a19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:27.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.640+0000 7f691d53a640 1 -- 172.21.15.189:0/3988949223 shutdown_connections 2024-09-16T14:49:27.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.640+0000 7f691d53a640 1 -- 172.21.15.189:0/3988949223 wait complete. 2024-09-16T14:49:27.657 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:27.657 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h9 host rack=r1 2024-09-16T14:49:27.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7aaae640 10 monclient: get_monmap_and_config 2024-09-16T14:49:27.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7aaae640 10 monclient: build_initial_monmap 2024-09-16T14:49:27.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7aaae640 10 monclient: monmap: 2024-09-16T14:49:27.771 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:27.771 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:27.771 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:27.771268+0000 2024-09-16T14:49:27.771 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:27.771268+0000 2024-09-16T14:49:27.771 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:27.771 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:27.771 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:27.771 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:27.771 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:27.771 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:27.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7aaae640 1 Processor -- start 2024-09-16T14:49:27.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7aaae640 1 -- start start 2024-09-16T14:49:27.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7aaae640 10 monclient: init 2024-09-16T14:49:27.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7aaae640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:27.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7aaae640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:27.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7aaae640 10 monclient(hunting): picked mon.noname-a con 0x7fda740a1a90 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:27.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7aaae640 10 monclient(hunting): picked mon.noname-c con 0x7fda7409fdf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:27.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7aaae640 10 monclient(hunting): picked mon.noname-b con 0x7fda740a2580 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:27.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7aaae640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7409e670 con 0x7fda740a1a90 2024-09-16T14:49:27.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7aaae640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7409eb20 con 0x7fda7409fdf0 2024-09-16T14:49:27.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7aaae640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda740a3020 con 0x7fda740a2580 2024-09-16T14:49:27.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7aaae640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:27.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7aaae640 10 monclient(hunting): authenticate will time out at 1349.887388s 2024-09-16T14:49:27.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda7884a640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fda740a1a90 legacy=0x7fda740a1e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:27.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda73fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fda7409fdf0 0x7fda74139d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53412/0 (socket says 172.21.15.189:53412) 2024-09-16T14:49:27.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda73fff640 1 -- 172.21.15.189:0/2354820065 learned_addr learned my addr 172.21.15.189:0/2354820065 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:27.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda737fe640 1 -- 172.21.15.189:0/2354820065 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081384789 0 0) 0x7fda7409eb20 con 0x7fda7409fdf0 2024-09-16T14:49:27.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda737fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:27.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:27.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:27.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda737fe640 10 monclient(hunting): my global_id is 144499 2024-09-16T14:49:27.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.768+0000 7fda737fe640 1 -- 172.21.15.189:0/2354820065 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda5c003530 con 0x7fda7409fdf0 2024-09-16T14:49:27.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 1 -- 172.21.15.189:0/2354820065 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1429122779 0 0) 0x7fda5c003530 con 0x7fda7409fdf0 2024-09-16T14:49:27.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 1 -- 172.21.15.189:0/2354820065 >> v1:172.21.15.26:6790/0 conn(0x7fda740a2580 legacy=0x7fda7413d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:27.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 1 -- 172.21.15.189:0/2354820065 >> v1:172.21.15.26:6789/0 conn(0x7fda740a1a90 legacy=0x7fda740a1e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:27.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 10 monclient: _finish_hunting 0 2024-09-16T14:49:27.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 1 monclient: found mon.noname-c 2024-09-16T14:49:27.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:27.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:27.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 1 -- 172.21.15.189:0/2354820065 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7409ee00 con 0x7fda7409fdf0 2024-09-16T14:49:27.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 10 monclient: _finish_auth 0 2024-09-16T14:49:27.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:27.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:27.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 1 -- 172.21.15.189:0/2354820065 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fda68001640 con 0x7fda7409fdf0 2024-09-16T14:49:27.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 5 monclient: authenticate success, global_id 144499 2024-09-16T14:49:27.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:27.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:27.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:27.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 10 monclient: dump: 2024-09-16T14:49:27.781 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:27.781 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:27.781 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:27.781 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:27.781 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:27.781 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:27.781 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:27.781 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:27.781 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:27.781 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:27.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:27.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 1 -- 172.21.15.189:0/2354820065 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda68001960 con 0x7fda7409fdf0 2024-09-16T14:49:27.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:27.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:27.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 4 set_mon_vals no callback set 2024-09-16T14:49:27.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 1 -- 172.21.15.189:0/2354820065 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fda68001cf0 con 0x7fda7409fdf0 2024-09-16T14:49:27.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:27.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:27.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda737fe640 10 monclient: dump: 2024-09-16T14:49:27.782 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:27.782 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:27.782 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:27.783 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:27.783 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:27.783 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:27.783 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:27.783 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:27.783 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:27.783 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:27.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 10 monclient: shutdown 2024-09-16T14:49:27.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:27.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 1 -- 172.21.15.189:0/2354820065 >> v1:172.21.15.189:6789/0 conn(0x7fda7409fdf0 legacy=0x7fda74139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:27.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 1 -- 172.21.15.189:0/2354820065 shutdown_connections 2024-09-16T14:49:27.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 1 -- 172.21.15.189:0/2354820065 >> 172.21.15.189:0/2354820065 conn(0x7fda740a7a80 msgr2=0x7fda740a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:27.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 1 -- 172.21.15.189:0/2354820065 shutdown_connections 2024-09-16T14:49:27.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 1 -- 172.21.15.189:0/2354820065 wait complete. 2024-09-16T14:49:27.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 10 monclient: build_initial_monmap 2024-09-16T14:49:27.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 10 monclient: monmap: 2024-09-16T14:49:27.784 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:27.784 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:27.784 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:27.784 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:27.784 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:27.784 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:27.784 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:27.784 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:27.784 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:27.784 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:27.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 1 Processor -- start 2024-09-16T14:49:27.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 1 -- start start 2024-09-16T14:49:27.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 10 monclient: init 2024-09-16T14:49:27.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:27.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:27.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 10 monclient(hunting): picked mon.noname-b con 0x7fda7409fdf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:27.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 10 monclient(hunting): picked mon.noname-a con 0x7fda740a1a90 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:27.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 10 monclient(hunting): picked mon.noname-c con 0x7fda740a2580 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:27.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7409e280 con 0x7fda740a1a90 2024-09-16T14:49:27.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7413df00 con 0x7fda7409fdf0 2024-09-16T14:49:27.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda740a6690 con 0x7fda740a2580 2024-09-16T14:49:27.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:27.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 10 monclient(hunting): authenticate will time out at 1349.890337s 2024-09-16T14:49:27.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7884a640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fda7409fdf0 0x7fda74139520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53416/0 (socket says 172.21.15.189:53416) 2024-09-16T14:49:27.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7884a640 1 -- 172.21.15.189:0/2893046111 learned_addr learned my addr 172.21.15.189:0/2893046111 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:27.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda73fff640 1 -- 172.21.15.189:0/2893046111 >> v1:172.21.15.26:6789/0 conn(0x7fda740a1a90 legacy=0x7fda74139c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:27.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 1 -- 172.21.15.189:0/2893046111 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195321087 0 0) 0x7fda7413df00 con 0x7fda7409fdf0 2024-09-16T14:49:27.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 10 monclient(hunting): _negotiate 2024-09-16T14:49:27.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:27.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:27.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 10 monclient(hunting): my global_id is 144502 2024-09-16T14:49:27.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 1 -- 172.21.15.189:0/2893046111 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda60003530 con 0x7fda7409fdf0 2024-09-16T14:49:27.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 1 -- 172.21.15.189:0/2893046111 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 119611931 0 0) 0x7fda60003530 con 0x7fda7409fdf0 2024-09-16T14:49:27.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 1 -- 172.21.15.189:0/2893046111 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda7413df00 con 0x7fda7409fdf0 2024-09-16T14:49:27.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 1 -- 172.21.15.189:0/2893046111 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fda6c004490 con 0x7fda7409fdf0 2024-09-16T14:49:27.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:27.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:27.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 10 monclient(hunting): dump: 2024-09-16T14:49:27.787 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:27.787 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:27.787 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:27.787 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:27.787 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:27.787 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:27.787 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:27.787 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:27.787 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:27.788 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:27.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:27.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 1 -- 172.21.15.189:0/2893046111 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4203668315 0 0) 0x7fda7413df00 con 0x7fda7409fdf0 2024-09-16T14:49:27.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 1 -- 172.21.15.189:0/2893046111 >> v1:172.21.15.26:6790/0 conn(0x7fda740a2580 legacy=0x7fda741d1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:27.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 1 -- 172.21.15.189:0/2893046111 >> v1:172.21.15.26:6789/0 conn(0x7fda740a1a90 legacy=0x7fda74139c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:27.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 10 monclient: _finish_hunting 0 2024-09-16T14:49:27.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 1 monclient: found mon.b 2024-09-16T14:49:27.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:27.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:27.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 1 -- 172.21.15.189:0/2893046111 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7413d4a0 con 0x7fda7409fdf0 2024-09-16T14:49:27.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 10 monclient: _finish_auth 0 2024-09-16T14:49:27.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 10 monclient: _check_auth_tickets 2024-09-16T14:49:27.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:27.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 5 monclient: authenticate success, global_id 144502 2024-09-16T14:49:27.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 10 monclient: _renew_subs 2024-09-16T14:49:27.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:27.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 1 -- 172.21.15.189:0/2893046111 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda7413cc00 con 0x7fda7409fdf0 2024-09-16T14:49:27.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 10 monclient: _renew_subs 2024-09-16T14:49:27.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:27.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda7aaae640 1 -- 172.21.15.189:0/2893046111 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda7413a340 con 0x7fda7409fdf0 2024-09-16T14:49:27.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 1 -- 172.21.15.189:0/2893046111 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda6c0047b0 con 0x7fda7409fdf0 2024-09-16T14:49:27.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:27.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 1 -- 172.21.15.189:0/2893046111 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fda6c004b40 con 0x7fda7409fdf0 2024-09-16T14:49:27.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:27.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:27.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda72ffd640 4 set_mon_vals no callback set 2024-09-16T14:49:27.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 10 monclient: dump: 2024-09-16T14:49:27.790 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:27.790 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:27.790 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:27.790 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:27.790 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:27.790 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:27.790 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:27.791 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:27.791 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:27.791 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:27.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.772+0000 7fda717fa640 1 -- 172.21.15.189:0/2893046111 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fda6c004ca0 con 0x7fda7409fdf0 2024-09-16T14:49:27.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.776+0000 7fda717fa640 1 -- 172.21.15.189:0/2893046111 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(191..191 src has 164..191) v4 ==== 8707+0+0 (unknown 3802811376 0 0) 0x7fda6c0448e0 con 0x7fda7409fdf0 2024-09-16T14:49:27.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.776+0000 7fda7aaae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:27.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.776+0000 7fda7aaae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:27.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.776+0000 7fda7aaae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:27.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.776+0000 7fda7aaae640 1 -- 172.21.15.189:0/2893046111 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda54005050 con 0x7fda7409fdf0 2024-09-16T14:49:27.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.776+0000 7fda73fff640 10 monclient: get_auth_request con 0x7fda60036090 auth_method 0 2024-09-16T14:49:27.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.780+0000 7fda717fa640 1 -- 172.21.15.189:0/2893046111 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fda6c014a70 con 0x7fda7409fdf0 2024-09-16T14:49:27.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.780+0000 7fda717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:27.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.780+0000 7fda717fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:27.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.936+0000 7fda7aaae640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:27.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.940+0000 7fda7aaae640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:27.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.940+0000 7fda7aaae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:27.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:27.940+0000 7fda7aaae640 1 -- 172.21.15.189:0/2893046111 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fda54005880 con 0x7fda7409fdf0 2024-09-16T14:49:28.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.540+0000 7fda717fa640 1 -- 172.21.15.189:0/2893046111 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}]=0 added bucket h9 type host to location {rack=r1} v192) v1 ==== 166+0+0 (unknown 3634557683 0 0) 0x7fda6c01d060 con 0x7fda7409fdf0 2024-09-16T14:49:28.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.540+0000 7fda717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:28.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.540+0000 7fda717fa640 10 monclient: _finish_command 2 = system:0 added bucket h9 type host to location {rack=r1} 2024-09-16T14:49:28.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.640+0000 7fda7aaae640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:28.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.640+0000 7fda7aaae640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:28.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.640+0000 7fda7aaae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:28.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.640+0000 7fda7aaae640 1 -- 172.21.15.189:0/2893046111 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fda54005ac0 con 0x7fda7409fdf0 2024-09-16T14:49:28.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.644+0000 7fda717fa640 1 -- 172.21.15.189:0/2893046111 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h9' already exists v192) v1 ==== 145+0+0 (unknown 530611223 0 0) 0x7fda6c012ee0 con 0x7fda7409fdf0 2024-09-16T14:49:28.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.644+0000 7fda717fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:28.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.644+0000 7fda717fa640 10 monclient: _finish_command 3 = system:0 bucket 'h9' already exists 2024-09-16T14:49:28.645 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h9' already exists 2024-09-16T14:49:28.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.644+0000 7fda7aaae640 1 -- 172.21.15.189:0/2893046111 >> v1:172.21.15.189:6812/3196161582 conn(0x7fda60036090 legacy=0x7fda60038550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:28.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.644+0000 7fda7aaae640 10 monclient: shutdown 2024-09-16T14:49:28.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.644+0000 7fda7aaae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:28.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.644+0000 7fda7aaae640 1 -- 172.21.15.189:0/2893046111 >> v1:172.21.15.189:6789/0 conn(0x7fda7409fdf0 legacy=0x7fda74139520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:28.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.644+0000 7fda7aaae640 1 -- 172.21.15.189:0/2893046111 shutdown_connections 2024-09-16T14:49:28.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.644+0000 7fda7aaae640 1 -- 172.21.15.189:0/2893046111 >> 172.21.15.189:0/2893046111 conn(0x7fda740a7a80 msgr2=0x7fda7409b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:28.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.644+0000 7fda7aaae640 1 -- 172.21.15.189:0/2893046111 shutdown_connections 2024-09-16T14:49:28.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.644+0000 7fda7aaae640 1 -- 172.21.15.189:0/2893046111 wait complete. 2024-09-16T14:49:28.662 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:28.662 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h10 host rack=r1 2024-09-16T14:49:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 10 monclient: get_monmap_and_config 2024-09-16T14:49:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 10 monclient: build_initial_monmap 2024-09-16T14:49:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 10 monclient: monmap: 2024-09-16T14:49:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:28.785982+0000 2024-09-16T14:49:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:28.785982+0000 2024-09-16T14:49:28.786 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:28.786 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:28.786 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:28.786 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:28.786 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:28.786 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 1 Processor -- start 2024-09-16T14:49:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 1 -- start start 2024-09-16T14:49:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 10 monclient: init 2024-09-16T14:49:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 10 monclient(hunting): picked mon.noname-c con 0x7fe8340999a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 10 monclient(hunting): picked mon.noname-a con 0x7fe834134db0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 10 monclient(hunting): picked mon.noname-b con 0x7fe834135240 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe834071850 con 0x7fe834134db0 2024-09-16T14:49:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8340979d0 con 0x7fe8340999a0 2024-09-16T14:49:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe834097cb0 con 0x7fe834135240 2024-09-16T14:49:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 10 monclient(hunting): authenticate will time out at 1350.901872s 2024-09-16T14:49:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe839c48640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe8340999a0 0x7fe834099d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53430/0 (socket says 172.21.15.189:53430) 2024-09-16T14:49:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe839c48640 1 -- 172.21.15.189:0/3459027196 learned_addr learned my addr 172.21.15.189:0/3459027196 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe839447640 1 -- 172.21.15.189:0/3459027196 >> v1:172.21.15.26:6789/0 conn(0x7fe834134db0 legacy=0x7fe834139ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 1 -- 172.21.15.189:0/3459027196 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330210411 0 0) 0x7fe8340979d0 con 0x7fe8340999a0 2024-09-16T14:49:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 10 monclient(hunting): _negotiate 2024-09-16T14:49:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 10 monclient(hunting): my global_id is 144505 2024-09-16T14:49:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 1 -- 172.21.15.189:0/3459027196 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe820003530 con 0x7fe8340999a0 2024-09-16T14:49:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 1 -- 172.21.15.189:0/3459027196 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2174170777 0 0) 0x7fe820003530 con 0x7fe8340999a0 2024-09-16T14:49:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 1 -- 172.21.15.189:0/3459027196 >> v1:172.21.15.26:6790/0 conn(0x7fe834135240 legacy=0x7fe83413d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 1 -- 172.21.15.189:0/3459027196 >> v1:172.21.15.26:6789/0 conn(0x7fe834134db0 legacy=0x7fe834139ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 10 monclient: _finish_hunting 0 2024-09-16T14:49:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 1 monclient: found mon.noname-c 2024-09-16T14:49:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 1 -- 172.21.15.189:0/3459027196 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe834097f90 con 0x7fe8340999a0 2024-09-16T14:49:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 10 monclient: _finish_auth 0 2024-09-16T14:49:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 10 monclient: _check_auth_tickets 2024-09-16T14:49:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 1 -- 172.21.15.189:0/3459027196 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe830001620 con 0x7fe8340999a0 2024-09-16T14:49:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 5 monclient: authenticate success, global_id 144505 2024-09-16T14:49:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 10 monclient: dump: 2024-09-16T14:49:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:28.796 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 1 -- 172.21.15.189:0/3459027196 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe830001940 con 0x7fe8340999a0 2024-09-16T14:49:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 4 set_mon_vals no callback set 2024-09-16T14:49:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 1 -- 172.21.15.189:0/3459027196 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe830001cd0 con 0x7fe8340999a0 2024-09-16T14:49:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe838c46640 10 monclient: dump: 2024-09-16T14:49:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:28.798 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 10 monclient: shutdown 2024-09-16T14:49:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 1 -- 172.21.15.189:0/3459027196 >> v1:172.21.15.189:6789/0 conn(0x7fe8340999a0 legacy=0x7fe834099d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 1 -- 172.21.15.189:0/3459027196 shutdown_connections 2024-09-16T14:49:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 1 -- 172.21.15.189:0/3459027196 >> 172.21.15.189:0/3459027196 conn(0x7fe8340a0110 msgr2=0x7fe8340a0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 1 -- 172.21.15.189:0/3459027196 shutdown_connections 2024-09-16T14:49:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.784+0000 7fe83beac640 1 -- 172.21.15.189:0/3459027196 wait complete. 2024-09-16T14:49:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 monclient: build_initial_monmap 2024-09-16T14:49:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 monclient: monmap: 2024-09-16T14:49:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:28.799 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:28.799 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:28.799 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:28.799 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:28.799 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:28.799 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:28.799 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:28.799 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:28.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 1 Processor -- start 2024-09-16T14:49:28.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 1 -- start start 2024-09-16T14:49:28.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 monclient: init 2024-09-16T14:49:28.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:28.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 monclient(hunting): picked mon.noname-c con 0x7fe8340999a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 monclient(hunting): picked mon.noname-a con 0x7fe834134db0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 monclient(hunting): picked mon.noname-b con 0x7fe834135240 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83409a630 con 0x7fe834134db0 2024-09-16T14:49:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83408b2a0 con 0x7fe834135240 2024-09-16T14:49:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83408ae10 con 0x7fe8340999a0 2024-09-16T14:49:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 monclient(hunting): authenticate will time out at 1350.904934s 2024-09-16T14:49:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe839447640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fe834134db0 legacy=0x7fe83408a700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83a449640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe834135240 0x7fe8341d18e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53434/0 (socket says 172.21.15.189:53434) 2024-09-16T14:49:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83a449640 1 -- 172.21.15.189:0/3443068717 learned_addr learned my addr 172.21.15.189:0/3443068717 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:28.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578069509 0 0) 0x7fe83408b2a0 con 0x7fe834135240 2024-09-16T14:49:28.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:28.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:28.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:28.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient(hunting): my global_id is 144508 2024-09-16T14:49:28.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe818003530 con 0x7fe834135240 2024-09-16T14:49:28.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1571434501 0 0) 0x7fe818003530 con 0x7fe834135240 2024-09-16T14:49:28.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe83408b2a0 con 0x7fe834135240 2024-09-16T14:49:28.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe82c001c80 con 0x7fe834135240 2024-09-16T14:49:28.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:28.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:28.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient(hunting): dump: 2024-09-16T14:49:28.801 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:28.801 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:28.802 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:28.802 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:28.802 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:28.802 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:28.802 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:28.802 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:28.802 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:28.802 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:28.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:28.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962716801 0 0) 0x7fe83408ae10 con 0x7fe8340999a0 2024-09-16T14:49:28.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:28.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:28.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:28.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient(hunting): my global_id is 154298 2024-09-16T14:49:28.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe818003530 con 0x7fe8340999a0 2024-09-16T14:49:28.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 101783806 0 0) 0x7fe83408b2a0 con 0x7fe834135240 2024-09-16T14:49:28.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 >> v1:172.21.15.26:6790/0 conn(0x7fe8340999a0 legacy=0x7fe834089ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:28.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 >> v1:172.21.15.26:6789/0 conn(0x7fe834134db0 legacy=0x7fe83408a700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:28.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:49:28.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 1 monclient: found mon.b 2024-09-16T14:49:28.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:28.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:28.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83408c520 con 0x7fe834135240 2024-09-16T14:49:28.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:49:28.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:28.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 5 monclient: authenticate success, global_id 144508 2024-09-16T14:49:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 monclient: _renew_subs 2024-09-16T14:49:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 1 -- 172.21.15.189:0/3443068717 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83408cc10 con 0x7fe834135240 2024-09-16T14:49:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 monclient: _renew_subs 2024-09-16T14:49:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 1 -- 172.21.15.189:0/3443068717 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83408b670 con 0x7fe834135240 2024-09-16T14:49:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe82c0039e0 con 0x7fe834135240 2024-09-16T14:49:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe82c003d70 con 0x7fe834135240 2024-09-16T14:49:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 10 monclient: dump: 2024-09-16T14:49:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:28.805 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82bfff640 4 set_mon_vals no callback set 2024-09-16T14:49:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe83beac640 1 -- 172.21.15.189:0/3443068717 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe81c005050 con 0x7fe834135240 2024-09-16T14:49:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe82c003ed0 con 0x7fe834135240 2024-09-16T14:49:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(192..192 src has 164..192) v4 ==== 8917+0+0 (unknown 4237557639 0 0) 0x7fe82c0449e0 con 0x7fe834135240 2024-09-16T14:49:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.788+0000 7fe839c48640 10 monclient: get_auth_request con 0x7fe818036180 auth_method 0 2024-09-16T14:49:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.792+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe82c014aa0 con 0x7fe834135240 2024-09-16T14:49:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.792+0000 7fe82a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.792+0000 7fe82a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:28.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.956+0000 7fe83beac640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:28.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.956+0000 7fe83beac640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:28.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.956+0000 7fe83beac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:28.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:28.956+0000 7fe83beac640 1 -- 172.21.15.189:0/3443068717 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fe81c005880 con 0x7fe834135240 2024-09-16T14:49:29.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.540+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}]=0 added bucket h10 type host to location {rack=r1} v193) v1 ==== 168+0+0 (unknown 3245680003 0 0) 0x7fe82c014870 con 0x7fe834135240 2024-09-16T14:49:29.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.540+0000 7fe82a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:29.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.540+0000 7fe82a7fc640 10 monclient: _finish_command 2 = system:0 added bucket h10 type host to location {rack=r1} 2024-09-16T14:49:29.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.644+0000 7fe83beac640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:29.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.644+0000 7fe83beac640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:29.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.644+0000 7fe83beac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:29.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.644+0000 7fe83beac640 1 -- 172.21.15.189:0/3443068717 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fe81c005340 con 0x7fe834135240 2024-09-16T14:49:29.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.648+0000 7fe82a7fc640 1 -- 172.21.15.189:0/3443068717 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h10' already exists v193) v1 ==== 147+0+0 (unknown 1370988209 0 0) 0x7fe82c0153d0 con 0x7fe834135240 2024-09-16T14:49:29.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.648+0000 7fe82a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:29.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.648+0000 7fe82a7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h10' already exists 2024-09-16T14:49:29.650 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h10' already exists 2024-09-16T14:49:29.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.648+0000 7fe83beac640 1 -- 172.21.15.189:0/3443068717 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe818036180 legacy=0x7fe818038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:29.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.648+0000 7fe83beac640 10 monclient: shutdown 2024-09-16T14:49:29.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.648+0000 7fe83beac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:29.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.648+0000 7fe83beac640 1 -- 172.21.15.189:0/3443068717 >> v1:172.21.15.189:6789/0 conn(0x7fe834135240 legacy=0x7fe8341d18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:29.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.648+0000 7fe83beac640 1 -- 172.21.15.189:0/3443068717 shutdown_connections 2024-09-16T14:49:29.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.648+0000 7fe83beac640 1 -- 172.21.15.189:0/3443068717 >> 172.21.15.189:0/3443068717 conn(0x7fe8340a0110 msgr2=0x7fe83409ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:29.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.648+0000 7fe83beac640 1 -- 172.21.15.189:0/3443068717 shutdown_connections 2024-09-16T14:49:29.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.648+0000 7fe83beac640 1 -- 172.21.15.189:0/3443068717 wait complete. 2024-09-16T14:49:29.667 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:29.667 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h11 host rack=r1 2024-09-16T14:49:29.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 10 monclient: get_monmap_and_config 2024-09-16T14:49:29.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 10 monclient: build_initial_monmap 2024-09-16T14:49:29.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 10 monclient: monmap: 2024-09-16T14:49:29.781 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:29.782 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:29.782 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:29.782204+0000 2024-09-16T14:49:29.782 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:29.782204+0000 2024-09-16T14:49:29.782 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:29.782 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:29.782 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:29.782 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:29.782 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:29.782 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:29.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 1 Processor -- start 2024-09-16T14:49:29.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 1 -- start start 2024-09-16T14:49:29.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 10 monclient: init 2024-09-16T14:49:29.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:29.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:29.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 10 monclient(hunting): picked mon.noname-b con 0x7fd494070830 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:29.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 10 monclient(hunting): picked mon.noname-a con 0x7fd494071320 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:29.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 10 monclient(hunting): picked mon.noname-c con 0x7fd494134cf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:29.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd494135740 con 0x7fd494071320 2024-09-16T14:49:29.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd494135160 con 0x7fd494134cf0 2024-09-16T14:49:29.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49409d810 con 0x7fd494070830 2024-09-16T14:49:29.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:29.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 10 monclient(hunting): authenticate will time out at 1351.897867s 2024-09-16T14:49:29.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd498cc9640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd494134cf0 0x7fd49413d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60182/0 (socket says 172.21.15.189:60182) 2024-09-16T14:49:29.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd4937fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fd494071320 legacy=0x7fd494139d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:29.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd498cc9640 1 -- 172.21.15.189:0/1450068632 learned_addr learned my addr 172.21.15.189:0/1450068632 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:29.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 1 -- 172.21.15.189:0/1450068632 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974277246 0 0) 0x7fd494135160 con 0x7fd494134cf0 2024-09-16T14:49:29.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:29.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:29.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:29.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 10 monclient(hunting): my global_id is 144511 2024-09-16T14:49:29.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 1 -- 172.21.15.189:0/1450068632 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd47c003530 con 0x7fd494134cf0 2024-09-16T14:49:29.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 1 -- 172.21.15.189:0/1450068632 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 981660286 0 0) 0x7fd47c003530 con 0x7fd494134cf0 2024-09-16T14:49:29.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 1 -- 172.21.15.189:0/1450068632 >> v1:172.21.15.26:6790/0 conn(0x7fd494070830 legacy=0x7fd494070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:29.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 1 -- 172.21.15.189:0/1450068632 >> v1:172.21.15.26:6789/0 conn(0x7fd494071320 legacy=0x7fd494139d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:29.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:29.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 1 monclient: found mon.noname-c 2024-09-16T14:49:29.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:29.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:29.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 1 -- 172.21.15.189:0/1450068632 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49409f1c0 con 0x7fd494134cf0 2024-09-16T14:49:29.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 10 monclient: _finish_auth 0 2024-09-16T14:49:29.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:29.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:29.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 1 -- 172.21.15.189:0/1450068632 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd484001620 con 0x7fd494134cf0 2024-09-16T14:49:29.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 5 monclient: authenticate success, global_id 144511 2024-09-16T14:49:29.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:29.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:29.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:29.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 10 monclient: dump: 2024-09-16T14:49:29.791 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:29.791 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:29.791 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:29.791 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:29.791 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:29.791 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:29.791 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:29.791 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:29.791 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:29.792 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 1 -- 172.21.15.189:0/1450068632 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd484001940 con 0x7fd494134cf0 2024-09-16T14:49:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 1 -- 172.21.15.189:0/1450068632 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd484001d10 con 0x7fd494134cf0 2024-09-16T14:49:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd492ffd640 10 monclient: dump: 2024-09-16T14:49:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:29.793 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:29.793 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:29.793 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:29.793 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:29.793 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:29.793 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:29.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:29.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 4 set_mon_vals no callback set 2024-09-16T14:49:29.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 10 monclient: shutdown 2024-09-16T14:49:29.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:29.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 1 -- 172.21.15.189:0/1450068632 >> v1:172.21.15.189:6789/0 conn(0x7fd494134cf0 legacy=0x7fd49413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:29.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 1 -- 172.21.15.189:0/1450068632 shutdown_connections 2024-09-16T14:49:29.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 1 -- 172.21.15.189:0/1450068632 >> 172.21.15.189:0/1450068632 conn(0x7fd4940a3380 msgr2=0x7fd49409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:29.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 1 -- 172.21.15.189:0/1450068632 shutdown_connections 2024-09-16T14:49:29.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.780+0000 7fd49a72c640 1 -- 172.21.15.189:0/1450068632 wait complete. 2024-09-16T14:49:29.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 monclient: build_initial_monmap 2024-09-16T14:49:29.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 monclient: monmap: 2024-09-16T14:49:29.795 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:29.795 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:29.795 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:29.795 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:29.795 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:29.795 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:29.795 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:29.796 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:29.796 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:29.796 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:29.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 1 Processor -- start 2024-09-16T14:49:29.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 1 -- start start 2024-09-16T14:49:29.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 monclient: init 2024-09-16T14:49:29.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:29.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:49:29.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 monclient(hunting): picked mon.noname-b con 0x7fd494070830 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:29.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 monclient(hunting): picked mon.noname-c con 0x7fd494071320 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:29.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 monclient(hunting): picked mon.noname-a con 0x7fd494134cf0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:29.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd494132a70 con 0x7fd494134cf0 2024-09-16T14:49:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49411e110 con 0x7fd494070830 2024-09-16T14:49:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49406b320 con 0x7fd494071320 2024-09-16T14:49:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 monclient(hunting): authenticate will time out at 1351.900913s 2024-09-16T14:49:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd498cc9640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fd494134cf0 legacy=0x7fd4941d1b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd493fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd494070830 0x7fd494130730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60198/0 (socket says 172.21.15.189:60198) 2024-09-16T14:49:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd493fff640 1 -- 172.21.15.189:0/2810619210 learned_addr learned my addr 172.21.15.189:0/2810619210 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499154465 0 0) 0x7fd49411e110 con 0x7fd494070830 2024-09-16T14:49:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:49:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient(hunting): my global_id is 144514 2024-09-16T14:49:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd480003530 con 0x7fd494070830 2024-09-16T14:49:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2313723465 0 0) 0x7fd480003530 con 0x7fd494070830 2024-09-16T14:49:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd49411e110 con 0x7fd494070830 2024-09-16T14:49:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd48c001c20 con 0x7fd494070830 2024-09-16T14:49:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient(hunting): dump: 2024-09-16T14:49:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:29.799 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:29.799 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:29.799 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:29.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:29.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338941515 0 0) 0x7fd49406b320 con 0x7fd494071320 2024-09-16T14:49:29.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:49:29.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:29.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:29.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient(hunting): my global_id is 154304 2024-09-16T14:49:29.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd480003530 con 0x7fd494071320 2024-09-16T14:49:29.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1247500153 0 0) 0x7fd49411e110 con 0x7fd494070830 2024-09-16T14:49:29.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 >> v1:172.21.15.26:6790/0 conn(0x7fd494071320 legacy=0x7fd4941ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:29.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 >> v1:172.21.15.26:6789/0 conn(0x7fd494134cf0 legacy=0x7fd4941d1b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:29.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:49:29.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 1 monclient: found mon.b 2024-09-16T14:49:29.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:29.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:29.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49411d870 con 0x7fd494070830 2024-09-16T14:49:29.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient: _finish_auth 0 2024-09-16T14:49:29.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:49:29.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:29.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 5 monclient: authenticate success, global_id 144514 2024-09-16T14:49:29.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 monclient: _renew_subs 2024-09-16T14:49:29.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:29.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 1 -- 172.21.15.189:0/2810619210 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49411db50 con 0x7fd494070830 2024-09-16T14:49:29.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 monclient: _renew_subs 2024-09-16T14:49:29.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:29.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 1 -- 172.21.15.189:0/2810619210 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4941313c0 con 0x7fd494070830 2024-09-16T14:49:29.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd48c0039a0 con 0x7fd494070830 2024-09-16T14:49:29.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:29.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd48c003cb0 con 0x7fd494070830 2024-09-16T14:49:29.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:29.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:29.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd4927fc640 4 set_mon_vals no callback set 2024-09-16T14:49:29.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 10 monclient: dump: 2024-09-16T14:49:29.802 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:29.802 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:29.802 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:29.802 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:29.802 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:29.802 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:29.802 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:29.802 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:29.802 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:29.802 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:29.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd48c003e10 con 0x7fd494070830 2024-09-16T14:49:29.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:29.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:29.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:29.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(193..193 src has 164..193) v4 ==== 9130+0+0 (unknown 507613480 0 0) 0x7fd48c0449e0 con 0x7fd494070830 2024-09-16T14:49:29.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd49a72c640 1 -- 172.21.15.189:0/2810619210 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd474005050 con 0x7fd494070830 2024-09-16T14:49:29.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.784+0000 7fd4937fe640 10 monclient: get_auth_request con 0x7fd480036220 auth_method 0 2024-09-16T14:49:29.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.788+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd48c004610 con 0x7fd494070830 2024-09-16T14:49:29.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.788+0000 7fd490ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:29.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.788+0000 7fd490ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:29.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.952+0000 7fd49a72c640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:29.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.952+0000 7fd49a72c640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:29.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.952+0000 7fd49a72c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:29.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:29.952+0000 7fd49a72c640 1 -- 172.21.15.189:0/2810619210 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fd474005880 con 0x7fd494070830 2024-09-16T14:49:30.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.556+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}]=0 added bucket h11 type host to location {rack=r1} v194) v1 ==== 168+0+0 (unknown 223484147 0 0) 0x7fd48c012e30 con 0x7fd494070830 2024-09-16T14:49:30.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.556+0000 7fd490ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:30.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.556+0000 7fd490ff9640 10 monclient: _finish_command 2 = system:0 added bucket h11 type host to location {rack=r1} 2024-09-16T14:49:30.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.660+0000 7fd49a72c640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:30.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.660+0000 7fd49a72c640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:30.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.660+0000 7fd49a72c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:30.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.660+0000 7fd49a72c640 1 -- 172.21.15.189:0/2810619210 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fd474005ac0 con 0x7fd494070830 2024-09-16T14:49:30.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.660+0000 7fd490ff9640 1 -- 172.21.15.189:0/2810619210 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h11' already exists v194) v1 ==== 147+0+0 (unknown 1934786741 0 0) 0x7fd48c014790 con 0x7fd494070830 2024-09-16T14:49:30.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.660+0000 7fd490ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:30.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.660+0000 7fd490ff9640 10 monclient: _finish_command 3 = system:0 bucket 'h11' already exists 2024-09-16T14:49:30.664 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h11' already exists 2024-09-16T14:49:30.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.664+0000 7fd49a72c640 1 -- 172.21.15.189:0/2810619210 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd480036220 legacy=0x7fd4800386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:30.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.664+0000 7fd49a72c640 10 monclient: shutdown 2024-09-16T14:49:30.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.664+0000 7fd49a72c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:30.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.664+0000 7fd49a72c640 1 -- 172.21.15.189:0/2810619210 >> v1:172.21.15.189:6789/0 conn(0x7fd494070830 legacy=0x7fd494130730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:30.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.664+0000 7fd49a72c640 1 -- 172.21.15.189:0/2810619210 shutdown_connections 2024-09-16T14:49:30.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.664+0000 7fd49a72c640 1 -- 172.21.15.189:0/2810619210 >> 172.21.15.189:0/2810619210 conn(0x7fd4940a3380 msgr2=0x7fd494131910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:30.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.664+0000 7fd49a72c640 1 -- 172.21.15.189:0/2810619210 shutdown_connections 2024-09-16T14:49:30.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.664+0000 7fd49a72c640 1 -- 172.21.15.189:0/2810619210 wait complete. 2024-09-16T14:49:30.681 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:30.681 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h12 host rack=r1 2024-09-16T14:49:30.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 10 monclient: get_monmap_and_config 2024-09-16T14:49:30.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 10 monclient: build_initial_monmap 2024-09-16T14:49:30.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 10 monclient: monmap: 2024-09-16T14:49:30.802 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:30.802 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:30.802 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:30.802360+0000 2024-09-16T14:49:30.802 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:30.802360+0000 2024-09-16T14:49:30.802 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:30.802 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:30.802 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:30.802 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:30.802 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:30.803 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:30.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 1 Processor -- start 2024-09-16T14:49:30.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 1 -- start start 2024-09-16T14:49:30.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 10 monclient: init 2024-09-16T14:49:30.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:30.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:49:30.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 10 monclient(hunting): picked mon.noname-a con 0x7fa634099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:30.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 10 monclient(hunting): picked mon.noname-b con 0x7fa63409bd30 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:30.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 10 monclient(hunting): picked mon.noname-c con 0x7fa634137500 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:30.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa634099e50 con 0x7fa634099a70 2024-09-16T14:49:30.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63409e510 con 0x7fa634137500 2024-09-16T14:49:30.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63409c2c0 con 0x7fa63409bd30 2024-09-16T14:49:30.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:30.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 10 monclient(hunting): authenticate will time out at 1352.918581s 2024-09-16T14:49:30.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63a225640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fa634099a70 legacy=0x7fa63409d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:30.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63aa26640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa634137500 0x7fa6341398f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60214/0 (socket says 172.21.15.189:60214) 2024-09-16T14:49:30.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63aa26640 1 -- 172.21.15.189:0/398418495 learned_addr learned my addr 172.21.15.189:0/398418495 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:30.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 1 -- 172.21.15.189:0/398418495 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251747403 0 0) 0x7fa63409e510 con 0x7fa634137500 2024-09-16T14:49:30.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 10 monclient(hunting): _negotiate 2024-09-16T14:49:30.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:30.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:30.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 10 monclient(hunting): my global_id is 144517 2024-09-16T14:49:30.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 1 -- 172.21.15.189:0/398418495 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa620003530 con 0x7fa634137500 2024-09-16T14:49:30.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 1 -- 172.21.15.189:0/398418495 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1929660371 0 0) 0x7fa620003530 con 0x7fa634137500 2024-09-16T14:49:30.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 1 -- 172.21.15.189:0/398418495 >> v1:172.21.15.26:6790/0 conn(0x7fa63409bd30 legacy=0x7fa63409de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:30.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 1 -- 172.21.15.189:0/398418495 >> v1:172.21.15.26:6789/0 conn(0x7fa634099a70 legacy=0x7fa63409d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:30.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 10 monclient: _finish_hunting 0 2024-09-16T14:49:30.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 1 monclient: found mon.noname-c 2024-09-16T14:49:30.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:30.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:30.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 1 -- 172.21.15.189:0/398418495 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63409ee60 con 0x7fa634137500 2024-09-16T14:49:30.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 10 monclient: _finish_auth 0 2024-09-16T14:49:30.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 10 monclient: _check_auth_tickets 2024-09-16T14:49:30.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:30.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 5 monclient: authenticate success, global_id 144517 2024-09-16T14:49:30.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 1 -- 172.21.15.189:0/398418495 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa62c0019a0 con 0x7fa634137500 2024-09-16T14:49:30.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:30.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:30.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:30.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 10 monclient: dump: 2024-09-16T14:49:30.814 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:30.814 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:30.814 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:30.814 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:30.814 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:30.814 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:30.814 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:30.814 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:30.814 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:30.815 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:30.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:30.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 1 -- 172.21.15.189:0/398418495 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa62c001cc0 con 0x7fa634137500 2024-09-16T14:49:30.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:30.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 1 -- 172.21.15.189:0/398418495 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa62c002050 con 0x7fa634137500 2024-09-16T14:49:30.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:30.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:30.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa639223640 10 monclient: dump: 2024-09-16T14:49:30.815 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:30.815 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:30.815 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:30.815 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:30.815 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:30.816 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:30.816 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:30.816 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:30.816 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:30.816 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:30.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:30.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.800+0000 7fa63c489640 4 set_mon_vals no callback set 2024-09-16T14:49:30.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient: shutdown 2024-09-16T14:49:30.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:30.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 1 -- 172.21.15.189:0/398418495 >> v1:172.21.15.189:6789/0 conn(0x7fa634137500 legacy=0x7fa6341398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:30.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 1 -- 172.21.15.189:0/398418495 shutdown_connections 2024-09-16T14:49:30.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 1 -- 172.21.15.189:0/398418495 >> 172.21.15.189:0/398418495 conn(0x7fa6340a3380 msgr2=0x7fa63409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:30.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 1 -- 172.21.15.189:0/398418495 shutdown_connections 2024-09-16T14:49:30.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 1 -- 172.21.15.189:0/398418495 wait complete. 2024-09-16T14:49:30.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient: build_initial_monmap 2024-09-16T14:49:30.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient: monmap: 2024-09-16T14:49:30.817 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:30.817 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:30.817 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:30.817 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:30.817 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:30.817 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:30.817 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:30.817 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:30.817 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:30.817 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:30.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 1 Processor -- start 2024-09-16T14:49:30.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 1 -- start start 2024-09-16T14:49:30.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient: init 2024-09-16T14:49:30.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:30.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:30.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient(hunting): picked mon.noname-c con 0x7fa634099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:30.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient(hunting): picked mon.noname-a con 0x7fa63409bd30 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:30.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient(hunting): picked mon.noname-b con 0x7fa634137500 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:30.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6340a1f80 con 0x7fa63409bd30 2024-09-16T14:49:30.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63406b320 con 0x7fa634137500 2024-09-16T14:49:30.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6340a1a30 con 0x7fa634099a70 2024-09-16T14:49:30.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:30.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient(hunting): authenticate will time out at 1352.921674s 2024-09-16T14:49:30.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa639a24640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fa63409bd30 legacy=0x7fa6341d2940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:30.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63aa26640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa634137500 0x7fa6341d6070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60228/0 (socket says 172.21.15.189:60228) 2024-09-16T14:49:30.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63aa26640 1 -- 172.21.15.189:0/2405523178 learned_addr learned my addr 172.21.15.189:0/2405523178 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:30.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909637849 0 0) 0x7fa63406b320 con 0x7fa634137500 2024-09-16T14:49:30.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:30.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:30.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:30.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient(hunting): my global_id is 144520 2024-09-16T14:49:30.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa618003530 con 0x7fa634137500 2024-09-16T14:49:30.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2629897661 0 0) 0x7fa618003530 con 0x7fa634137500 2024-09-16T14:49:30.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa63406b320 con 0x7fa634137500 2024-09-16T14:49:30.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa62c0017c0 con 0x7fa634137500 2024-09-16T14:49:30.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:30.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:30.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient(hunting): dump: 2024-09-16T14:49:30.820 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:30.820 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:30.820 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:30.820 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:30.820 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:30.820 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:30.820 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:30.820 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:30.821 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:30.821 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:30.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:30.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002752590 0 0) 0x7fa6340a1a30 con 0x7fa634099a70 2024-09-16T14:49:30.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:30.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:30.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:30.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient(hunting): my global_id is 154310 2024-09-16T14:49:30.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa618003530 con 0x7fa634099a70 2024-09-16T14:49:30.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 224156469 0 0) 0x7fa63406b320 con 0x7fa634137500 2024-09-16T14:49:30.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 >> v1:172.21.15.26:6790/0 conn(0x7fa634099a70 legacy=0x7fa6341cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:30.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 >> v1:172.21.15.26:6789/0 conn(0x7fa63409bd30 legacy=0x7fa6341d2940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:30.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 1 monclient: found mon.b 2024-09-16T14:49:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa634070610 con 0x7fa634137500 2024-09-16T14:49:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient: _finish_auth 0 2024-09-16T14:49:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 5 monclient: authenticate success, global_id 144520 2024-09-16T14:49:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient: _renew_subs 2024-09-16T14:49:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 1 -- 172.21.15.189:0/2405523178 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa634121f60 con 0x7fa634137500 2024-09-16T14:49:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient: _renew_subs 2024-09-16T14:49:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:30.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 1 -- 172.21.15.189:0/2405523178 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa63406fa40 con 0x7fa634137500 2024-09-16T14:49:30.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:30.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:30.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:30.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa63c489640 1 -- 172.21.15.189:0/2405523178 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6340008e0 con 0x7fa634137500 2024-09-16T14:49:30.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa62c004fd0 con 0x7fa634137500 2024-09-16T14:49:30.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:30.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa62c0052e0 con 0x7fa634137500 2024-09-16T14:49:30.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:30.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:30.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 10 monclient: dump: 2024-09-16T14:49:30.823 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:30.823 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:30.823 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:30.824 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:30.824 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:30.824 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:30.824 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:30.824 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:30.824 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:30.824 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:30.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa638a22640 4 set_mon_vals no callback set 2024-09-16T14:49:30.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.804+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa62c00fd50 con 0x7fa634137500 2024-09-16T14:49:30.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.808+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(194..194 src has 164..194) v4 ==== 9343+0+0 (unknown 2447187741 0 0) 0x7fa62c044ca0 con 0x7fa634137500 2024-09-16T14:49:30.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.808+0000 7fa63a225640 10 monclient: get_auth_request con 0x7fa618036300 auth_method 0 2024-09-16T14:49:30.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.812+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa62c014a40 con 0x7fa634137500 2024-09-16T14:49:30.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.812+0000 7fa62affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:30.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.812+0000 7fa62affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:30.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.976+0000 7fa63c489640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:30.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.976+0000 7fa63c489640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:30.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.976+0000 7fa63c489640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:30.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:30.976+0000 7fa63c489640 1 -- 172.21.15.189:0/2405523178 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa6340a0bd0 con 0x7fa634137500 2024-09-16T14:49:31.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.560+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}]=0 added bucket h12 type host to location {rack=r1} v195) v1 ==== 168+0+0 (unknown 961556089 0 0) 0x7fa62c0130e0 con 0x7fa634137500 2024-09-16T14:49:31.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.560+0000 7fa62affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:31.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.560+0000 7fa62affd640 10 monclient: _finish_command 2 = system:0 added bucket h12 type host to location {rack=r1} 2024-09-16T14:49:31.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.664+0000 7fa63c489640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:31.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.664+0000 7fa63c489640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:31.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.664+0000 7fa63c489640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:31.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.664+0000 7fa63c489640 1 -- 172.21.15.189:0/2405523178 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa63409b020 con 0x7fa634137500 2024-09-16T14:49:31.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.668+0000 7fa62affd640 1 -- 172.21.15.189:0/2405523178 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h12' already exists v195) v1 ==== 147+0+0 (unknown 1188801899 0 0) 0x7fa62c014860 con 0x7fa634137500 2024-09-16T14:49:31.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.668+0000 7fa62affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:31.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.668+0000 7fa62affd640 10 monclient: _finish_command 3 = system:0 bucket 'h12' already exists 2024-09-16T14:49:31.670 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h12' already exists 2024-09-16T14:49:31.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.668+0000 7fa63c489640 1 -- 172.21.15.189:0/2405523178 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa618036300 legacy=0x7fa6180387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:31.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.668+0000 7fa63c489640 10 monclient: shutdown 2024-09-16T14:49:31.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.668+0000 7fa63c489640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:31.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.668+0000 7fa63c489640 1 -- 172.21.15.189:0/2405523178 >> v1:172.21.15.189:6789/0 conn(0x7fa634137500 legacy=0x7fa6341d6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:31.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.668+0000 7fa63c489640 1 -- 172.21.15.189:0/2405523178 shutdown_connections 2024-09-16T14:49:31.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.668+0000 7fa63c489640 1 -- 172.21.15.189:0/2405523178 >> 172.21.15.189:0/2405523178 conn(0x7fa6340a3380 msgr2=0x7fa634098a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:31.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.668+0000 7fa63c489640 1 -- 172.21.15.189:0/2405523178 shutdown_connections 2024-09-16T14:49:31.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.668+0000 7fa63c489640 1 -- 172.21.15.189:0/2405523178 wait complete. 2024-09-16T14:49:31.687 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:31.687 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h13 host rack=r1 2024-09-16T14:49:31.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.796+0000 7f3e71118640 10 monclient: get_monmap_and_config 2024-09-16T14:49:31.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.796+0000 7f3e71118640 10 monclient: build_initial_monmap 2024-09-16T14:49:31.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.796+0000 7f3e71118640 10 monclient: monmap: 2024-09-16T14:49:31.801 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:31.801 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:31.801 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:31.801675+0000 2024-09-16T14:49:31.802 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:31.801675+0000 2024-09-16T14:49:31.802 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:31.802 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:31.802 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:31.802 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:31.802 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:31.802 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:31.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 1 Processor -- start 2024-09-16T14:49:31.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 1 -- start start 2024-09-16T14:49:31.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 10 monclient: init 2024-09-16T14:49:31.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:31.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:49:31.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 10 monclient(hunting): picked mon.noname-a con 0x7f3e6c0a19b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:31.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 10 monclient(hunting): picked mon.noname-b con 0x7f3e6c0a24a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:31.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 10 monclient(hunting): picked mon.noname-c con 0x7f3e6c09fc80 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:31.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6c0a00d0 con 0x7f3e6c0a19b0 2024-09-16T14:49:31.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6c09ea00 con 0x7f3e6c09fc80 2024-09-16T14:49:31.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6c09dde0 con 0x7f3e6c0a24a0 2024-09-16T14:49:31.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:31.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 10 monclient(hunting): authenticate will time out at 1353.917626s 2024-09-16T14:49:31.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e6ad9d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f3e6c0a19b0 legacy=0x7f3e6c0a1d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:31.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e6b59e640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3e6c09fc80 0x7f3e6c13d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60234/0 (socket says 172.21.15.189:60234) 2024-09-16T14:49:31.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e6b59e640 1 -- 172.21.15.189:0/163533910 learned_addr learned my addr 172.21.15.189:0/163533910 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:31.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 1 -- 172.21.15.189:0/163533910 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814659817 0 0) 0x7f3e6c09ea00 con 0x7f3e6c09fc80 2024-09-16T14:49:31.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:49:31.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:31.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:31.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 10 monclient(hunting): my global_id is 144523 2024-09-16T14:49:31.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 1 -- 172.21.15.189:0/163533910 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e54003530 con 0x7f3e6c09fc80 2024-09-16T14:49:31.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 1 -- 172.21.15.189:0/163533910 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2584471715 0 0) 0x7f3e54003530 con 0x7f3e6c09fc80 2024-09-16T14:49:31.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 1 -- 172.21.15.189:0/163533910 >> v1:172.21.15.26:6790/0 conn(0x7f3e6c0a24a0 legacy=0x7f3e6c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:31.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 1 -- 172.21.15.189:0/163533910 >> v1:172.21.15.26:6789/0 conn(0x7f3e6c0a19b0 legacy=0x7f3e6c0a1d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:31.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:49:31.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 1 monclient: found mon.noname-c 2024-09-16T14:49:31.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:31.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:31.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 1 -- 172.21.15.189:0/163533910 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6c09ece0 con 0x7f3e6c09fc80 2024-09-16T14:49:31.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 10 monclient: _finish_auth 0 2024-09-16T14:49:31.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:49:31.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:31.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 1 -- 172.21.15.189:0/163533910 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e640019a0 con 0x7f3e6c09fc80 2024-09-16T14:49:31.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:31.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:31.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 10 monclient: dump: 2024-09-16T14:49:31.813 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:31.813 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:31.813 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:31.813 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:31.813 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:31.813 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:31.813 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:31.813 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:31.813 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:31.814 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:31.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 5 monclient: authenticate success, global_id 144523 2024-09-16T14:49:31.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:31.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 1 -- 172.21.15.189:0/163533910 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e64001cc0 con 0x7f3e6c09fc80 2024-09-16T14:49:31.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:31.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 1 -- 172.21.15.189:0/163533910 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e64002050 con 0x7f3e6c09fc80 2024-09-16T14:49:31.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:31.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 4 set_mon_vals no callback set 2024-09-16T14:49:31.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:31.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:31.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e69d9b640 10 monclient: dump: 2024-09-16T14:49:31.815 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:31.815 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:31.815 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:31.815 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:31.815 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:31.815 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:31.815 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:31.816 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:31.816 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:31.816 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:31.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 10 monclient: shutdown 2024-09-16T14:49:31.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:31.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 1 -- 172.21.15.189:0/163533910 >> v1:172.21.15.189:6789/0 conn(0x7f3e6c09fc80 legacy=0x7f3e6c13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:31.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 1 -- 172.21.15.189:0/163533910 shutdown_connections 2024-09-16T14:49:31.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 1 -- 172.21.15.189:0/163533910 >> 172.21.15.189:0/163533910 conn(0x7f3e6c0a3ab0 msgr2=0x7f3e6c0a3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:31.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 1 -- 172.21.15.189:0/163533910 shutdown_connections 2024-09-16T14:49:31.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 1 -- 172.21.15.189:0/163533910 wait complete. 2024-09-16T14:49:31.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 10 monclient: build_initial_monmap 2024-09-16T14:49:31.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.800+0000 7f3e71118640 10 monclient: monmap: 2024-09-16T14:49:31.817 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:31.817 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:31.817 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:31.817 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:31.817 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:31.817 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:31.817 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:31.817 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:31.817 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:31.817 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:31.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 1 Processor -- start 2024-09-16T14:49:31.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 1 -- start start 2024-09-16T14:49:31.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 10 monclient: init 2024-09-16T14:49:31.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:31.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:31.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 10 monclient(hunting): picked mon.noname-b con 0x7f3e6c09fc80 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:31.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 10 monclient(hunting): picked mon.noname-a con 0x7f3e6c0a19b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:31.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 10 monclient(hunting): picked mon.noname-c con 0x7f3e6c0a24a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:31.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6c121340 con 0x7f3e6c0a19b0 2024-09-16T14:49:31.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6c0a6150 con 0x7f3e6c09fc80 2024-09-16T14:49:31.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6c09e1a0 con 0x7f3e6c0a24a0 2024-09-16T14:49:31.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:31.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 10 monclient(hunting): authenticate will time out at 1353.920689s 2024-09-16T14:49:31.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e6ad9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3e6c09fc80 0x7f3e6c133980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60248/0 (socket says 172.21.15.189:60248) 2024-09-16T14:49:31.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e6ad9d640 1 -- 172.21.15.189:0/552562340 learned_addr learned my addr 172.21.15.189:0/552562340 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:31.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e6a59c640 1 -- 172.21.15.189:0/552562340 >> v1:172.21.15.26:6789/0 conn(0x7f3e6c0a19b0 legacy=0x7f3e6c1ce2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:31.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 1 -- 172.21.15.189:0/552562340 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684419731 0 0) 0x7f3e6c0a6150 con 0x7f3e6c09fc80 2024-09-16T14:49:31.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:31.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:31.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:31.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 10 monclient(hunting): my global_id is 144526 2024-09-16T14:49:31.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 1 -- 172.21.15.189:0/552562340 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e58003530 con 0x7f3e6c09fc80 2024-09-16T14:49:31.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 1 -- 172.21.15.189:0/552562340 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1677387430 0 0) 0x7f3e58003530 con 0x7f3e6c09fc80 2024-09-16T14:49:31.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 1 -- 172.21.15.189:0/552562340 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e6c0a6150 con 0x7f3e6c09fc80 2024-09-16T14:49:31.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 1 -- 172.21.15.189:0/552562340 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e5c0041e0 con 0x7f3e6c09fc80 2024-09-16T14:49:31.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:31.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:31.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 10 monclient(hunting): dump: 2024-09-16T14:49:31.820 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:31.820 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:31.820 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:31.820 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:31.820 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:31.820 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:31.820 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:31.820 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:31.820 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:31.820 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:31.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:31.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 1 -- 172.21.15.189:0/552562340 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1421551980 0 0) 0x7f3e6c0a6150 con 0x7f3e6c09fc80 2024-09-16T14:49:31.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 1 -- 172.21.15.189:0/552562340 >> v1:172.21.15.26:6790/0 conn(0x7f3e6c0a24a0 legacy=0x7f3e6c1d1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:31.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 1 -- 172.21.15.189:0/552562340 >> v1:172.21.15.26:6789/0 conn(0x7f3e6c0a19b0 legacy=0x7f3e6c1ce2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:31.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 10 monclient: _finish_hunting 0 2024-09-16T14:49:31.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 1 monclient: found mon.b 2024-09-16T14:49:31.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:31.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:31.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 1 -- 172.21.15.189:0/552562340 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6c11f8c0 con 0x7f3e6c09fc80 2024-09-16T14:49:31.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 10 monclient: _finish_auth 0 2024-09-16T14:49:31.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:31.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:31.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 5 monclient: authenticate success, global_id 144526 2024-09-16T14:49:31.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 10 monclient: _renew_subs 2024-09-16T14:49:31.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:31.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 1 -- 172.21.15.189:0/552562340 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6c120aa0 con 0x7f3e6c09fc80 2024-09-16T14:49:31.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 10 monclient: _renew_subs 2024-09-16T14:49:31.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:31.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 1 -- 172.21.15.189:0/552562340 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6c134090 con 0x7f3e6c09fc80 2024-09-16T14:49:31.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 1 -- 172.21.15.189:0/552562340 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e5c004500 con 0x7f3e6c09fc80 2024-09-16T14:49:31.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:31.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 1 -- 172.21.15.189:0/552562340 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e5c004890 con 0x7f3e6c09fc80 2024-09-16T14:49:31.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:31.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:31.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e6959a640 4 set_mon_vals no callback set 2024-09-16T14:49:31.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 10 monclient: dump: 2024-09-16T14:49:31.822 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:31.823 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:31.823 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:31.823 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:31.823 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:31.823 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:31.823 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:31.823 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:31.823 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:31.823 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:31.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 1 -- 172.21.15.189:0/552562340 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3e5c0049f0 con 0x7f3e6c09fc80 2024-09-16T14:49:31.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:31.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:31.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:31.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e71118640 1 -- 172.21.15.189:0/552562340 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e4c005050 con 0x7f3e6c09fc80 2024-09-16T14:49:31.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e537fe640 1 -- 172.21.15.189:0/552562340 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(195..195 src has 169..195) v4 ==== 9556+0+0 (unknown 210977109 0 0) 0x7f3e5c044950 con 0x7f3e6c09fc80 2024-09-16T14:49:31.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.804+0000 7f3e6a59c640 10 monclient: get_auth_request con 0x7f3e58036080 auth_method 0 2024-09-16T14:49:31.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.808+0000 7f3e537fe640 1 -- 172.21.15.189:0/552562340 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3e5c0051c0 con 0x7f3e6c09fc80 2024-09-16T14:49:31.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.808+0000 7f3e537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:31.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.808+0000 7f3e537fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:31.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.964+0000 7f3e71118640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:31.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.964+0000 7f3e71118640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:31.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.964+0000 7f3e71118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:31.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:31.964+0000 7f3e71118640 1 -- 172.21.15.189:0/552562340 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f3e4c005880 con 0x7f3e6c09fc80 2024-09-16T14:49:32.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.576+0000 7f3e537fe640 1 -- 172.21.15.189:0/552562340 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}]=0 added bucket h13 type host to location {rack=r1} v196) v1 ==== 168+0+0 (unknown 2431653090 0 0) 0x7f3e5c012c00 con 0x7f3e6c09fc80 2024-09-16T14:49:32.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.576+0000 7f3e537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:32.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.576+0000 7f3e537fe640 10 monclient: _finish_command 2 = system:0 added bucket h13 type host to location {rack=r1} 2024-09-16T14:49:32.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.676+0000 7f3e71118640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:32.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.676+0000 7f3e71118640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:32.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.676+0000 7f3e71118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:32.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.676+0000 7f3e71118640 1 -- 172.21.15.189:0/552562340 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f3e4c005340 con 0x7f3e6c09fc80 2024-09-16T14:49:32.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.680+0000 7f3e537fe640 1 -- 172.21.15.189:0/552562340 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h13' already exists v196) v1 ==== 147+0+0 (unknown 915979453 0 0) 0x7f3e5c014380 con 0x7f3e6c09fc80 2024-09-16T14:49:32.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.680+0000 7f3e537fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:32.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.680+0000 7f3e537fe640 10 monclient: _finish_command 3 = system:0 bucket 'h13' already exists 2024-09-16T14:49:32.681 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h13' already exists 2024-09-16T14:49:32.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.680+0000 7f3e71118640 1 -- 172.21.15.189:0/552562340 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3e58036080 legacy=0x7f3e58038540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:32.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.680+0000 7f3e71118640 10 monclient: shutdown 2024-09-16T14:49:32.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.680+0000 7f3e71118640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:32.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.680+0000 7f3e71118640 1 -- 172.21.15.189:0/552562340 >> v1:172.21.15.189:6789/0 conn(0x7f3e6c09fc80 legacy=0x7f3e6c133980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:32.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.680+0000 7f3e71118640 1 -- 172.21.15.189:0/552562340 shutdown_connections 2024-09-16T14:49:32.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.680+0000 7f3e71118640 1 -- 172.21.15.189:0/552562340 >> 172.21.15.189:0/552562340 conn(0x7f3e6c0a3ab0 msgr2=0x7f3e6c0a4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:32.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.680+0000 7f3e71118640 1 -- 172.21.15.189:0/552562340 shutdown_connections 2024-09-16T14:49:32.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.680+0000 7f3e71118640 1 -- 172.21.15.189:0/552562340 wait complete. 2024-09-16T14:49:32.698 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:32.698 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h14 host rack=r1 2024-09-16T14:49:32.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.812+0000 7f4f92658640 10 monclient: get_monmap_and_config 2024-09-16T14:49:32.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.812+0000 7f4f92658640 10 monclient: build_initial_monmap 2024-09-16T14:49:32.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.812+0000 7f4f92658640 10 monclient: monmap: 2024-09-16T14:49:32.817 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:32.817 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:32.817 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:32.817587+0000 2024-09-16T14:49:32.817 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:32.817587+0000 2024-09-16T14:49:32.817 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:32.817 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:32.817 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:32.817 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:32.818 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:32.818 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:32.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 1 Processor -- start 2024-09-16T14:49:32.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 1 -- start start 2024-09-16T14:49:32.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 10 monclient: init 2024-09-16T14:49:32.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:32.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:32.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 10 monclient(hunting): picked mon.noname-a con 0x7f4f8c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:32.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 10 monclient(hunting): picked mon.noname-c con 0x7f4f8c09c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:32.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 10 monclient(hunting): picked mon.noname-b con 0x7f4f8c09e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:32.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c099e50 con 0x7f4f8c099a70 2024-09-16T14:49:32.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c09dbd0 con 0x7f4f8c09c550 2024-09-16T14:49:32.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c09ee20 con 0x7f4f8c09e6d0 2024-09-16T14:49:32.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:32.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 10 monclient(hunting): authenticate will time out at 1354.933374s 2024-09-16T14:49:32.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8b7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4f8c09c550 0x7f4f8c1379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60254/0 (socket says 172.21.15.189:60254) 2024-09-16T14:49:32.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8b7fe640 1 -- 172.21.15.189:0/168907214 learned_addr learned my addr 172.21.15.189:0/168907214 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:32.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8bfff640 1 -- 172.21.15.189:0/168907214 >> v1:172.21.15.26:6789/0 conn(0x7f4f8c099a70 legacy=0x7f4f8c09dfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:32.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 1 -- 172.21.15.189:0/168907214 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623435634 0 0) 0x7f4f8c09dbd0 con 0x7f4f8c09c550 2024-09-16T14:49:32.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:32.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:32.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:32.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 10 monclient(hunting): my global_id is 144529 2024-09-16T14:49:32.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 1 -- 172.21.15.189:0/168907214 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f78003530 con 0x7f4f8c09c550 2024-09-16T14:49:32.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 1 -- 172.21.15.189:0/168907214 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1068991820 0 0) 0x7f4f78003530 con 0x7f4f8c09c550 2024-09-16T14:49:32.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 1 -- 172.21.15.189:0/168907214 >> v1:172.21.15.26:6790/0 conn(0x7f4f8c09e6d0 legacy=0x7f4f8c13b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:32.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 1 -- 172.21.15.189:0/168907214 >> v1:172.21.15.26:6789/0 conn(0x7f4f8c099a70 legacy=0x7f4f8c09dfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:32.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:32.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 1 monclient: found mon.noname-c 2024-09-16T14:49:32.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:32.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:32.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 1 -- 172.21.15.189:0/168907214 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c09f100 con 0x7f4f8c09c550 2024-09-16T14:49:32.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 10 monclient: _finish_auth 0 2024-09-16T14:49:32.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:32.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:32.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 1 -- 172.21.15.189:0/168907214 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4f7c0019a0 con 0x7f4f8c09c550 2024-09-16T14:49:32.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 5 monclient: authenticate success, global_id 144529 2024-09-16T14:49:32.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:32.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:32.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:32.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 10 monclient: dump: 2024-09-16T14:49:32.831 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:32.831 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:32.832 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:32.832 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:32.832 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:32.832 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:32.832 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:32.832 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:32.832 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:32.832 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:32.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:32.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 1 -- 172.21.15.189:0/168907214 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f7c001c50 con 0x7f4f8c09c550 2024-09-16T14:49:32.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:32.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:32.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 1 -- 172.21.15.189:0/168907214 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4f7c001fe0 con 0x7f4f8c09c550 2024-09-16T14:49:32.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:32.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 4 set_mon_vals no callback set 2024-09-16T14:49:32.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:32.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f8affd640 10 monclient: dump: 2024-09-16T14:49:32.834 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:32.834 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:32.834 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:32.834 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:32.834 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:32.835 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:32.835 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:32.835 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:32.835 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:32.835 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:32.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 10 monclient: shutdown 2024-09-16T14:49:32.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:32.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 1 -- 172.21.15.189:0/168907214 >> v1:172.21.15.189:6789/0 conn(0x7f4f8c09c550 legacy=0x7f4f8c1379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:32.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 1 -- 172.21.15.189:0/168907214 shutdown_connections 2024-09-16T14:49:32.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 1 -- 172.21.15.189:0/168907214 >> 172.21.15.189:0/168907214 conn(0x7f4f8c0a3380 msgr2=0x7f4f8c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:32.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 1 -- 172.21.15.189:0/168907214 shutdown_connections 2024-09-16T14:49:32.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 1 -- 172.21.15.189:0/168907214 wait complete. 2024-09-16T14:49:32.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 10 monclient: build_initial_monmap 2024-09-16T14:49:32.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 10 monclient: monmap: 2024-09-16T14:49:32.836 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:32.836 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:32.836 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:32.836 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:32.836 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:32.836 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:32.836 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:32.836 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:32.836 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:32.836 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:32.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 1 Processor -- start 2024-09-16T14:49:32.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 1 -- start start 2024-09-16T14:49:32.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.816+0000 7f4f92658640 10 monclient: init 2024-09-16T14:49:32.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:32.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:32.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 10 monclient(hunting): picked mon.noname-a con 0x7f4f8c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:32.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 10 monclient(hunting): picked mon.noname-b con 0x7f4f8c09c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:32.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 10 monclient(hunting): picked mon.noname-c con 0x7f4f8c09e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:32.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c09cf20 con 0x7f4f8c099a70 2024-09-16T14:49:32.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c06b320 con 0x7f4f8c09c550 2024-09-16T14:49:32.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c1323f0 con 0x7f4f8c09e6d0 2024-09-16T14:49:32.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:32.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 10 monclient(hunting): authenticate will time out at 1354.936398s 2024-09-16T14:49:32.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f8bfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f4f8c099a70 legacy=0x7f4f8c139ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:32.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f8b7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4f8c09c550 0x7f4f8c13a1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60256/0 (socket says 172.21.15.189:60256) 2024-09-16T14:49:32.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f8b7fe640 1 -- 172.21.15.189:0/691282494 learned_addr learned my addr 172.21.15.189:0/691282494 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:32.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 1 -- 172.21.15.189:0/691282494 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333998800 0 0) 0x7f4f8c06b320 con 0x7f4f8c09c550 2024-09-16T14:49:32.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:49:32.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:32.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:32.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 10 monclient(hunting): my global_id is 144532 2024-09-16T14:49:32.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 1 -- 172.21.15.189:0/691282494 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f6c003530 con 0x7f4f8c09c550 2024-09-16T14:49:32.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 1 -- 172.21.15.189:0/691282494 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1330403733 0 0) 0x7f4f6c003530 con 0x7f4f8c09c550 2024-09-16T14:49:32.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 1 -- 172.21.15.189:0/691282494 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f8c06b320 con 0x7f4f8c09c550 2024-09-16T14:49:32.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 1 -- 172.21.15.189:0/691282494 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4f7c001f60 con 0x7f4f8c09c550 2024-09-16T14:49:32.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:32.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:32.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 10 monclient(hunting): dump: 2024-09-16T14:49:32.839 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:32.839 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:32.839 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:32.839 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:32.839 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:32.840 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:32.840 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:32.840 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:32.840 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:32.840 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:32.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:32.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 1 -- 172.21.15.189:0/691282494 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3675760359 0 0) 0x7f4f8c06b320 con 0x7f4f8c09c550 2024-09-16T14:49:32.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 1 -- 172.21.15.189:0/691282494 >> v1:172.21.15.26:6790/0 conn(0x7f4f8c09e6d0 legacy=0x7f4f8c1cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:32.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 1 -- 172.21.15.189:0/691282494 >> v1:172.21.15.26:6789/0 conn(0x7f4f8c099a70 legacy=0x7f4f8c139ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:32.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:49:32.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 1 monclient: found mon.b 2024-09-16T14:49:32.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:32.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:32.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 1 -- 172.21.15.189:0/691282494 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c11a5b0 con 0x7f4f8c09c550 2024-09-16T14:49:32.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 10 monclient: _finish_auth 0 2024-09-16T14:49:32.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:49:32.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:32.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 5 monclient: authenticate success, global_id 144532 2024-09-16T14:49:32.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 10 monclient: _renew_subs 2024-09-16T14:49:32.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:32.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 1 -- 172.21.15.189:0/691282494 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f8c06f8f0 con 0x7f4f8c09c550 2024-09-16T14:49:32.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 10 monclient: _renew_subs 2024-09-16T14:49:32.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:32.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 1 -- 172.21.15.189:0/691282494 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f8c13a950 con 0x7f4f8c09c550 2024-09-16T14:49:32.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 1 -- 172.21.15.189:0/691282494 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f7c004f80 con 0x7f4f8c09c550 2024-09-16T14:49:32.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:32.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 1 -- 172.21.15.189:0/691282494 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4f7c005290 con 0x7f4f8c09c550 2024-09-16T14:49:32.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:32.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:32.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f8a7fc640 4 set_mon_vals no callback set 2024-09-16T14:49:32.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 10 monclient: dump: 2024-09-16T14:49:32.842 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:32.842 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:32.842 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:32.842 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:32.842 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:32.842 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:32.842 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:32.843 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:32.843 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:32.843 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:32.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:32.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:32.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:32.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f92658640 1 -- 172.21.15.189:0/691282494 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f70005050 con 0x7f4f8c09c550 2024-09-16T14:49:32.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 1 -- 172.21.15.189:0/691282494 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4f7c00fd00 con 0x7f4f8c09c550 2024-09-16T14:49:32.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f88ff9640 1 -- 172.21.15.189:0/691282494 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(196..196 src has 169..196) v4 ==== 9769+0+0 (unknown 2534499329 0 0) 0x7f4f7c045020 con 0x7f4f8c09c550 2024-09-16T14:49:32.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.820+0000 7f4f8bfff640 10 monclient: get_auth_request con 0x7f4f6c036010 auth_method 0 2024-09-16T14:49:32.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.824+0000 7f4f88ff9640 1 -- 172.21.15.189:0/691282494 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4f7c014bd0 con 0x7f4f8c09c550 2024-09-16T14:49:32.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.824+0000 7f4f88ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:32.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.824+0000 7f4f88ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:32.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.988+0000 7f4f92658640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:32.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.988+0000 7f4f92658640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:32.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.988+0000 7f4f92658640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:32.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:32.988+0000 7f4f92658640 1 -- 172.21.15.189:0/691282494 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f4f70005880 con 0x7f4f8c09c550 2024-09-16T14:49:33.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.584+0000 7f4f88ff9640 1 -- 172.21.15.189:0/691282494 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}]=0 added bucket h14 type host to location {rack=r1} v197) v1 ==== 168+0+0 (unknown 886260870 0 0) 0x7f4f7c0149f0 con 0x7f4f8c09c550 2024-09-16T14:49:33.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.584+0000 7f4f88ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:33.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.584+0000 7f4f88ff9640 10 monclient: _finish_command 2 = system:0 added bucket h14 type host to location {rack=r1} 2024-09-16T14:49:33.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.684+0000 7f4f92658640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:33.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.684+0000 7f4f92658640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:33.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.684+0000 7f4f92658640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:33.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.684+0000 7f4f92658640 1 -- 172.21.15.189:0/691282494 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f4f70005340 con 0x7f4f8c09c550 2024-09-16T14:49:33.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.688+0000 7f4f88ff9640 1 -- 172.21.15.189:0/691282494 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h14' already exists v197) v1 ==== 147+0+0 (unknown 2138044677 0 0) 0x7f4f7c0149f0 con 0x7f4f8c09c550 2024-09-16T14:49:33.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.688+0000 7f4f88ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:33.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.688+0000 7f4f88ff9640 10 monclient: _finish_command 3 = system:0 bucket 'h14' already exists 2024-09-16T14:49:33.689 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h14' already exists 2024-09-16T14:49:33.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.688+0000 7f4f92658640 1 -- 172.21.15.189:0/691282494 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4f6c036010 legacy=0x7f4f6c0384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:33.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.688+0000 7f4f92658640 10 monclient: shutdown 2024-09-16T14:49:33.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.688+0000 7f4f92658640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:33.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.688+0000 7f4f92658640 1 -- 172.21.15.189:0/691282494 >> v1:172.21.15.189:6789/0 conn(0x7f4f8c09c550 legacy=0x7f4f8c13a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:33.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.688+0000 7f4f92658640 1 -- 172.21.15.189:0/691282494 shutdown_connections 2024-09-16T14:49:33.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.688+0000 7f4f92658640 1 -- 172.21.15.189:0/691282494 >> 172.21.15.189:0/691282494 conn(0x7f4f8c0a3380 msgr2=0x7f4f8c0a0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:33.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.688+0000 7f4f92658640 1 -- 172.21.15.189:0/691282494 shutdown_connections 2024-09-16T14:49:33.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.688+0000 7f4f92658640 1 -- 172.21.15.189:0/691282494 wait complete. 2024-09-16T14:49:33.706 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:33.706 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h15 host rack=r1 2024-09-16T14:49:33.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.824+0000 7fbbad14a640 10 monclient: get_monmap_and_config 2024-09-16T14:49:33.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.824+0000 7fbbad14a640 10 monclient: build_initial_monmap 2024-09-16T14:49:33.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.824+0000 7fbbad14a640 10 monclient: monmap: 2024-09-16T14:49:33.829 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:33.829 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:33.829 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:33.829543+0000 2024-09-16T14:49:33.829 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:33.829543+0000 2024-09-16T14:49:33.829 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:33.829 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:33.829 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:33.830 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:33.830 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:33.830 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:33.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 1 Processor -- start 2024-09-16T14:49:33.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 1 -- start start 2024-09-16T14:49:33.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 10 monclient: init 2024-09-16T14:49:33.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:33.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:49:33.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 10 monclient(hunting): picked mon.noname-c con 0x7fbba81393b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:33.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 10 monclient(hunting): picked mon.noname-b con 0x7fbba8139790 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:33.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 10 monclient(hunting): picked mon.noname-a con 0x7fbba809e510 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:33.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba809aad0 con 0x7fbba809e510 2024-09-16T14:49:33.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba809d030 con 0x7fbba81393b0 2024-09-16T14:49:33.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba809edd0 con 0x7fbba8139790 2024-09-16T14:49:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 10 monclient(hunting): authenticate will time out at 1355.945487s 2024-09-16T14:49:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba6d9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbba81393b0 0x7fbba809d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60264/0 (socket says 172.21.15.189:60264) 2024-09-16T14:49:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba6d9d640 1 -- 172.21.15.189:0/1484159649 learned_addr learned my addr 172.21.15.189:0/1484159649 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba759e640 1 -- 172.21.15.189:0/1484159649 >> v1:172.21.15.26:6789/0 conn(0x7fbba809e510 legacy=0x7fbba8141fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 1 -- 172.21.15.189:0/1484159649 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877578449 0 0) 0x7fbba809d030 con 0x7fbba81393b0 2024-09-16T14:49:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:49:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 10 monclient(hunting): my global_id is 144535 2024-09-16T14:49:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 1 -- 172.21.15.189:0/1484159649 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb90003530 con 0x7fbba81393b0 2024-09-16T14:49:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 1 -- 172.21.15.189:0/1484159649 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 662130293 0 0) 0x7fbb90003530 con 0x7fbba81393b0 2024-09-16T14:49:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 1 -- 172.21.15.189:0/1484159649 >> v1:172.21.15.26:6790/0 conn(0x7fbba8139790 legacy=0x7fbba809de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 1 -- 172.21.15.189:0/1484159649 >> v1:172.21.15.26:6789/0 conn(0x7fbba809e510 legacy=0x7fbba8141fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:49:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 1 monclient: found mon.noname-c 2024-09-16T14:49:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 1 -- 172.21.15.189:0/1484159649 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba809a7f0 con 0x7fbba81393b0 2024-09-16T14:49:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 10 monclient: _finish_auth 0 2024-09-16T14:49:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:49:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 1 -- 172.21.15.189:0/1484159649 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbb9c001550 con 0x7fbba81393b0 2024-09-16T14:49:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 5 monclient: authenticate success, global_id 144535 2024-09-16T14:49:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 10 monclient: dump: 2024-09-16T14:49:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:33.842 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 1 -- 172.21.15.189:0/1484159649 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb9c001870 con 0x7fbba81393b0 2024-09-16T14:49:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 4 set_mon_vals no callback set 2024-09-16T14:49:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 1 -- 172.21.15.189:0/1484159649 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbb9c001c40 con 0x7fbba81393b0 2024-09-16T14:49:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbba5d9b640 10 monclient: dump: 2024-09-16T14:49:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:33.843 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 10 monclient: shutdown 2024-09-16T14:49:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 1 -- 172.21.15.189:0/1484159649 >> v1:172.21.15.189:6789/0 conn(0x7fbba81393b0 legacy=0x7fbba809d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 1 -- 172.21.15.189:0/1484159649 shutdown_connections 2024-09-16T14:49:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 1 -- 172.21.15.189:0/1484159649 >> 172.21.15.189:0/1484159649 conn(0x7fbba80a3380 msgr2=0x7fbba809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 1 -- 172.21.15.189:0/1484159649 shutdown_connections 2024-09-16T14:49:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 1 -- 172.21.15.189:0/1484159649 wait complete. 2024-09-16T14:49:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 10 monclient: build_initial_monmap 2024-09-16T14:49:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.828+0000 7fbbad14a640 10 monclient: monmap: 2024-09-16T14:49:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:33.845 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 1 Processor -- start 2024-09-16T14:49:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 1 -- start start 2024-09-16T14:49:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 10 monclient: init 2024-09-16T14:49:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:49:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 10 monclient(hunting): picked mon.noname-c con 0x7fbba809e510 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 10 monclient(hunting): picked mon.noname-b con 0x7fbba81393b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 10 monclient(hunting): picked mon.noname-a con 0x7fbba8139790 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba80a1ee0 con 0x7fbba8139790 2024-09-16T14:49:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba81ce410 con 0x7fbba81393b0 2024-09-16T14:49:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba806b320 con 0x7fbba809e510 2024-09-16T14:49:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 10 monclient(hunting): authenticate will time out at 1355.948673s 2024-09-16T14:49:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbba659c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbba81393b0 0x7fbba81d2840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60278/0 (socket says 172.21.15.189:60278) 2024-09-16T14:49:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbba759e640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fbba8139790 legacy=0x7fbba81d5f70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:49:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbba659c640 1 -- 172.21.15.189:0/2706477477 learned_addr learned my addr 172.21.15.189:0/2706477477 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498608177 0 0) 0x7fbba81ce410 con 0x7fbba81393b0 2024-09-16T14:49:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient(hunting): my global_id is 144538 2024-09-16T14:49:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb94003530 con 0x7fbba81393b0 2024-09-16T14:49:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1231408941 0 0) 0x7fbb94003530 con 0x7fbba81393b0 2024-09-16T14:49:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbba81ce410 con 0x7fbba81393b0 2024-09-16T14:49:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbb98001940 con 0x7fbba81393b0 2024-09-16T14:49:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient(hunting): dump: 2024-09-16T14:49:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:33.848 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:33.848 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:33.848 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:33.848 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:33.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:33.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779993712 0 0) 0x7fbba806b320 con 0x7fbba809e510 2024-09-16T14:49:33.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:33.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:33.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:33.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient(hunting): my global_id is 154328 2024-09-16T14:49:33.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb94003530 con 0x7fbba809e510 2024-09-16T14:49:33.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1511859274 0 0) 0x7fbba81ce410 con 0x7fbba81393b0 2024-09-16T14:49:33.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 >> v1:172.21.15.26:6790/0 conn(0x7fbba809e510 legacy=0x7fbba8136260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:33.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 >> v1:172.21.15.26:6789/0 conn(0x7fbba8139790 legacy=0x7fbba81d5f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:33.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:49:33.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 1 monclient: found mon.b 2024-09-16T14:49:33.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba8136bd0 con 0x7fbba81393b0 2024-09-16T14:49:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:49:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 5 monclient: authenticate success, global_id 144538 2024-09-16T14:49:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 10 monclient: _renew_subs 2024-09-16T14:49:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 1 -- 172.21.15.189:0/2706477477 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba8123940 con 0x7fbba81393b0 2024-09-16T14:49:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 10 monclient: _renew_subs 2024-09-16T14:49:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 1 -- 172.21.15.189:0/2706477477 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba81377a0 con 0x7fbba81393b0 2024-09-16T14:49:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb98001d10 con 0x7fbba81393b0 2024-09-16T14:49:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbb98003af0 con 0x7fbba81393b0 2024-09-16T14:49:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbba559a640 4 set_mon_vals no callback set 2024-09-16T14:49:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 10 monclient: dump: 2024-09-16T14:49:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:33.852 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbbad14a640 1 -- 172.21.15.189:0/2706477477 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb88005050 con 0x7fbba81393b0 2024-09-16T14:49:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbb98003c50 con 0x7fbba81393b0 2024-09-16T14:49:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(197..197 src has 169..197) v4 ==== 9982+0+0 (unknown 3559933277 0 0) 0x7fbb98044c10 con 0x7fbba81393b0 2024-09-16T14:49:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.832+0000 7fbba6d9d640 10 monclient: get_auth_request con 0x7fbb94036130 auth_method 0 2024-09-16T14:49:33.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.836+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbb98014630 con 0x7fbba81393b0 2024-09-16T14:49:33.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.836+0000 7fbb8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:33.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:33.836+0000 7fbb8f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:34.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.000+0000 7fbbad14a640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:34.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.000+0000 7fbbad14a640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:34.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.000+0000 7fbbad14a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:34.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.000+0000 7fbbad14a640 1 -- 172.21.15.189:0/2706477477 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fbb88005300 con 0x7fbba81393b0 2024-09-16T14:49:34.021 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:49:34.021 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T14:49:34.021 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T14:49:34.021 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T14:49:34.025 INFO:tasks.ceph.mon.a:Started 2024-09-16T14:49:34.025 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:49:34.025 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:49:34.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa16a008640 10 monclient: get_monmap_and_config 2024-09-16T14:49:34.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa16a008640 10 monclient: build_initial_monmap 2024-09-16T14:49:34.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa16a008640 10 monclient: monmap: 2024-09-16T14:49:34.150 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:34.150 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.150 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:34.151328+0000 2024-09-16T14:49:34.151 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:34.151328+0000 2024-09-16T14:49:34.151 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:34.151 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.151 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:34.151 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:34.151 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:34.151 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa16a008640 1 Processor -- start 2024-09-16T14:49:34.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa16a008640 1 -- start start 2024-09-16T14:49:34.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa16a008640 10 monclient: init 2024-09-16T14:49:34.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa16a008640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:34.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa16a008640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:34.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa16a008640 10 monclient(hunting): picked mon.noname-c con 0x7fa1640a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:34.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa16a008640 10 monclient(hunting): picked mon.noname-a con 0x7fa164099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:34.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa16a008640 10 monclient(hunting): picked mon.noname-b con 0x7fa1640a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:34.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa16a008640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa164098ea0 con 0x7fa164099b60 2024-09-16T14:49:34.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa16a008640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1640997b0 con 0x7fa1640a1c00 2024-09-16T14:49:34.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa16a008640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa164099460 con 0x7fa1640a26f0 2024-09-16T14:49:34.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa16a008640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:34.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa16a008640 10 monclient(hunting): authenticate will time out at 1398.270968s 2024-09-16T14:49:34.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa163fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa1640a26f0 0x7fa16413d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49574/0 (socket says 172.21.15.26:49574) 2024-09-16T14:49:34.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.146+0000 7fa163fff640 1 -- 172.21.15.26:0/3915731064 learned_addr learned my addr 172.21.15.26:0/3915731064 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 1 -- 172.21.15.26:0/3915731064 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699530240 0 0) 0x7fa1640997b0 con 0x7fa1640a1c00 2024-09-16T14:49:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 10 monclient(hunting): my global_id is 144541 2024-09-16T14:49:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 1 -- 172.21.15.26:0/3915731064 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa14c003530 con 0x7fa1640a1c00 2024-09-16T14:49:34.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 1 -- 172.21.15.26:0/3915731064 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 685708551 0 0) 0x7fa14c003530 con 0x7fa1640a1c00 2024-09-16T14:49:34.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 1 -- 172.21.15.26:0/3915731064 >> v1:172.21.15.26:6790/0 conn(0x7fa1640a26f0 legacy=0x7fa16413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 1 -- 172.21.15.26:0/3915731064 >> v1:172.21.15.26:6789/0 conn(0x7fa164099b60 legacy=0x7fa164139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 10 monclient: _finish_hunting 0 2024-09-16T14:49:34.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 1 monclient: found mon.noname-c 2024-09-16T14:49:34.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:34.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:34.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 1 -- 172.21.15.26:0/3915731064 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1640a34b0 con 0x7fa1640a1c00 2024-09-16T14:49:34.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 10 monclient: _finish_auth 0 2024-09-16T14:49:34.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:34.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:34.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 1 -- 172.21.15.26:0/3915731064 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa154001620 con 0x7fa1640a1c00 2024-09-16T14:49:34.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:34.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 10 monclient: dump: 2024-09-16T14:49:34.180 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.180 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.180 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.180 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.180 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.180 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.180 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.180 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.180 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.180 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 5 monclient: authenticate success, global_id 144541 2024-09-16T14:49:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 1 -- 172.21.15.26:0/3915731064 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa154001940 con 0x7fa1640a1c00 2024-09-16T14:49:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 1 -- 172.21.15.26:0/3915731064 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa154001cd0 con 0x7fa1640a1c00 2024-09-16T14:49:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 4 set_mon_vals no callback set 2024-09-16T14:49:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa1627fc640 10 monclient: dump: 2024-09-16T14:49:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.182 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 10 monclient: shutdown 2024-09-16T14:49:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 1 -- 172.21.15.26:0/3915731064 >> v1:172.21.15.189:6789/0 conn(0x7fa1640a1c00 legacy=0x7fa1640a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 1 -- 172.21.15.26:0/3915731064 shutdown_connections 2024-09-16T14:49:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 1 -- 172.21.15.26:0/3915731064 >> 172.21.15.26:0/3915731064 conn(0x7fa1640a3d00 msgr2=0x7fa1640a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 1 -- 172.21.15.26:0/3915731064 shutdown_connections 2024-09-16T14:49:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 1 -- 172.21.15.26:0/3915731064 wait complete. 2024-09-16T14:49:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 10 monclient: build_initial_monmap 2024-09-16T14:49:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 10 monclient: monmap: 2024-09-16T14:49:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:34.184 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 1 Processor -- start 2024-09-16T14:49:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 1 -- start start 2024-09-16T14:49:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 10 monclient: init 2024-09-16T14:49:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:49:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 10 monclient(hunting): picked mon.noname-b con 0x7fa164099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 10 monclient(hunting): picked mon.noname-c con 0x7fa1640a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 10 monclient(hunting): picked mon.noname-a con 0x7fa1640a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa164136c80 con 0x7fa1640a26f0 2024-09-16T14:49:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16411d970 con 0x7fa164099b60 2024-09-16T14:49:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16409ed40 con 0x7fa1640a1c00 2024-09-16T14:49:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa16a008640 10 monclient(hunting): authenticate will time out at 1398.295501s 2024-09-16T14:49:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa162ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa1640a1c00 0x7fa164136570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49578/0 (socket says 172.21.15.26:49578) 2024-09-16T14:49:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.170+0000 7fa162ffd640 1 -- 172.21.15.26:0/4098988899 learned_addr learned my addr 172.21.15.26:0/4098988899 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 1 -- 172.21.15.26:0/4098988899 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731479817 0 0) 0x7fa16411d970 con 0x7fa164099b60 2024-09-16T14:49:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 10 monclient(hunting): my global_id is 144544 2024-09-16T14:49:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 1 -- 172.21.15.26:0/4098988899 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa150003530 con 0x7fa164099b60 2024-09-16T14:49:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 1 -- 172.21.15.26:0/4098988899 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2009851081 0 0) 0x7fa150003530 con 0x7fa164099b60 2024-09-16T14:49:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 1 -- 172.21.15.26:0/4098988899 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa16411d970 con 0x7fa164099b60 2024-09-16T14:49:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 1 -- 172.21.15.26:0/4098988899 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa1540015e0 con 0x7fa164099b60 2024-09-16T14:49:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 10 monclient(hunting): dump: 2024-09-16T14:49:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.187 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 1 -- 172.21.15.26:0/4098988899 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2873307103 0 0) 0x7fa16411d970 con 0x7fa164099b60 2024-09-16T14:49:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 1 -- 172.21.15.26:0/4098988899 >> v1:172.21.15.26:6790/0 conn(0x7fa1640a1c00 legacy=0x7fa164136570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 1 -- 172.21.15.26:0/4098988899 >> v1:172.21.15.26:6789/0 conn(0x7fa1640a26f0 legacy=0x7fa1641d1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:49:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 1 monclient: found mon.b 2024-09-16T14:49:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 1 -- 172.21.15.26:0/4098988899 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16409f290 con 0x7fa164099b60 2024-09-16T14:49:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 10 monclient: _finish_auth 0 2024-09-16T14:49:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:49:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa16a008640 5 monclient: authenticate success, global_id 144544 2024-09-16T14:49:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa16a008640 10 monclient: _renew_subs 2024-09-16T14:49:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa16a008640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa16a008640 1 -- 172.21.15.26:0/4098988899 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa16409e3f0 con 0x7fa164099b60 2024-09-16T14:49:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa16a008640 10 monclient: _renew_subs 2024-09-16T14:49:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa16a008640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa16a008640 1 -- 172.21.15.26:0/4098988899 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa16411ebb0 con 0x7fa164099b60 2024-09-16T14:49:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 1 -- 172.21.15.26:0/4098988899 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1540039c0 con 0x7fa164099b60 2024-09-16T14:49:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa161ffb640 4 set_mon_vals no callback set 2024-09-16T14:49:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 1 -- 172.21.15.26:0/4098988899 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa154003cd0 con 0x7fa164099b60 2024-09-16T14:49:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 10 monclient: dump: 2024-09-16T14:49:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.189 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa16a008640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:34.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa16a008640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:34.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa16a008640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:34.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa16a008640 1 -- 172.21.15.26:0/4098988899 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa144005050 con 0x7fa164099b60 2024-09-16T14:49:34.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 1 -- 172.21.15.26:0/4098988899 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa154003eb0 con 0x7fa164099b60 2024-09-16T14:49:34.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa14bfff640 1 -- 172.21.15.26:0/4098988899 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(197..197 src has 169..197) v4 ==== 9982+0+0 (unknown 3559933277 0 0) 0x7fa154043b30 con 0x7fa164099b60 2024-09-16T14:49:34.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.174+0000 7fa162ffd640 10 monclient: get_auth_request con 0x7fa150036100 auth_method 0 2024-09-16T14:49:34.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.186+0000 7fa14bfff640 1 -- 172.21.15.26:0/4098988899 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa154013880 con 0x7fa164099b60 2024-09-16T14:49:34.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.186+0000 7fa14bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:34.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.186+0000 7fa14bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:34.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.414+0000 7fa16a008640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:49:34.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.414+0000 7fa16a008640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:49:34.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.414+0000 7fa16a008640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:34.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.414+0000 7fa16a008640 1 -- 172.21.15.26:0/4098988899 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa144005880 con 0x7fa164099b60 2024-09-16T14:49:34.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.414+0000 7fa14bfff640 1 -- 172.21.15.26:0/4098988899 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3813405378) 0x7fa154011cf0 con 0x7fa164099b60 2024-09-16T14:49:34.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.414+0000 7fa14bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:49:34.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.414+0000 7fa14bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:34.418 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:49:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.414+0000 7fa16a008640 1 -- 172.21.15.26:0/4098988899 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa150036100 legacy=0x7fa1500385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.414+0000 7fa16a008640 10 monclient: shutdown 2024-09-16T14:49:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.414+0000 7fa16a008640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.414+0000 7fa16a008640 1 -- 172.21.15.26:0/4098988899 >> v1:172.21.15.189:6789/0 conn(0x7fa164099b60 legacy=0x7fa164135e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.414+0000 7fa16a008640 1 -- 172.21.15.26:0/4098988899 shutdown_connections 2024-09-16T14:49:34.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.414+0000 7fa16a008640 1 -- 172.21.15.26:0/4098988899 >> 172.21.15.26:0/4098988899 conn(0x7fa1640a3d00 msgr2=0x7fa16406bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:34.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.414+0000 7fa16a008640 1 -- 172.21.15.26:0/4098988899 shutdown_connections 2024-09-16T14:49:34.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.414+0000 7fa16a008640 1 -- 172.21.15.26:0/4098988899 wait complete. 2024-09-16T14:49:34.437 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:49:34.437 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:49:34.437 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:49:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.550+0000 7fcce3225640 10 monclient: get_monmap_and_config 2024-09-16T14:49:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.550+0000 7fcce3225640 10 monclient: build_initial_monmap 2024-09-16T14:49:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.550+0000 7fcce3225640 10 monclient: monmap: 2024-09-16T14:49:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:34.557284+0000 2024-09-16T14:49:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:34.557284+0000 2024-09-16T14:49:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:34.557 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 1 Processor -- start 2024-09-16T14:49:34.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 1 -- start start 2024-09-16T14:49:34.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 10 monclient: init 2024-09-16T14:49:34.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:34.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:34.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 10 monclient(hunting): picked mon.noname-a con 0x7fccdc0a0050 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:34.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 10 monclient(hunting): picked mon.noname-c con 0x7fccdc0a0b40 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:34.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 10 monclient(hunting): picked mon.noname-b con 0x7fccdc0a0f50 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:34.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccdc06cb40 con 0x7fccdc0a0050 2024-09-16T14:49:34.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccdc06c210 con 0x7fccdc0a0b40 2024-09-16T14:49:34.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccdc06c4f0 con 0x7fccdc0a0f50 2024-09-16T14:49:34.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:34.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 10 monclient(hunting): authenticate will time out at 1398.677296s 2024-09-16T14:49:34.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce0fc1640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fccdc0a0050 0x7fccdc0a0430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:59878/0 (socket says 172.21.15.26:59878) 2024-09-16T14:49:34.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce0fc1640 1 -- 172.21.15.26:0/1742062898 learned_addr learned my addr 172.21.15.26:0/1742062898 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:34.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce17c2640 1 --1- 172.21.15.26:0/1742062898 >> v1:172.21.15.26:6790/0 conn(0x7fccdc0a0f50 0x7fccdc13d480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49584/0 (socket says 172.21.15.26:49584) 2024-09-16T14:49:34.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 1 -- 172.21.15.26:0/1742062898 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176616848 0 0) 0x7fccdc06c4f0 con 0x7fccdc0a0f50 2024-09-16T14:49:34.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient(hunting): my global_id is 154331 2024-09-16T14:49:34.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 1 -- 172.21.15.26:0/1742062898 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc4003530 con 0x7fccdc0a0f50 2024-09-16T14:49:34.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 1 -- 172.21.15.26:0/1742062898 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776344125 0 0) 0x7fccdc06c210 con 0x7fccdc0a0b40 2024-09-16T14:49:34.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient(hunting): my global_id is 144547 2024-09-16T14:49:34.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 1 -- 172.21.15.26:0/1742062898 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccdc06c4f0 con 0x7fccdc0a0b40 2024-09-16T14:49:34.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 1 -- 172.21.15.26:0/1742062898 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3838894748 0 0) 0x7fccdc06c4f0 con 0x7fccdc0a0b40 2024-09-16T14:49:34.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 1 -- 172.21.15.26:0/1742062898 >> v1:172.21.15.26:6790/0 conn(0x7fccdc0a0f50 legacy=0x7fccdc13d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 1 -- 172.21.15.26:0/1742062898 >> v1:172.21.15.26:6789/0 conn(0x7fccdc0a0050 legacy=0x7fccdc0a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:49:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 1 monclient: found mon.noname-c 2024-09-16T14:49:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 1 -- 172.21.15.26:0/1742062898 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccdc06c7d0 con 0x7fccdc0a0b40 2024-09-16T14:49:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient: _finish_auth 0 2024-09-16T14:49:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 1 -- 172.21.15.26:0/1742062898 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fccd0001600 con 0x7fccdc0a0b40 2024-09-16T14:49:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient: dump: 2024-09-16T14:49:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.563 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 5 monclient: authenticate success, global_id 144547 2024-09-16T14:49:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 1 -- 172.21.15.26:0/1742062898 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccd00018b0 con 0x7fccdc0a0b40 2024-09-16T14:49:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 1 -- 172.21.15.26:0/1742062898 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fccd0001bc0 con 0x7fccdc0a0b40 2024-09-16T14:49:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fccdb7fe640 10 monclient: dump: 2024-09-16T14:49:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.565 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 4 set_mon_vals no callback set 2024-09-16T14:49:34.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 10 monclient: shutdown 2024-09-16T14:49:34.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:34.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 1 -- 172.21.15.26:0/1742062898 >> v1:172.21.15.189:6789/0 conn(0x7fccdc0a0b40 legacy=0x7fccdc139d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 1 -- 172.21.15.26:0/1742062898 shutdown_connections 2024-09-16T14:49:34.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 1 -- 172.21.15.26:0/1742062898 >> 172.21.15.26:0/1742062898 conn(0x7fccdc0a5540 msgr2=0x7fccdc0a5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:34.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 1 -- 172.21.15.26:0/1742062898 shutdown_connections 2024-09-16T14:49:34.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.554+0000 7fcce3225640 1 -- 172.21.15.26:0/1742062898 wait complete. 2024-09-16T14:49:34.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 10 monclient: build_initial_monmap 2024-09-16T14:49:34.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 10 monclient: monmap: 2024-09-16T14:49:34.573 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:34.574 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:34.574 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:34.574 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:34.574 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:34.574 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:34.574 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:34.574 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:34.574 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:34.574 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 1 Processor -- start 2024-09-16T14:49:34.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 1 -- start start 2024-09-16T14:49:34.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 10 monclient: init 2024-09-16T14:49:34.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:34.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:34.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 10 monclient(hunting): picked mon.noname-c con 0x7fccdc0a0050 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:34.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 10 monclient(hunting): picked mon.noname-a con 0x7fccdc0a0b40 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:34.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 10 monclient(hunting): picked mon.noname-b con 0x7fccdc0a0f50 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:34.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccdc13a610 con 0x7fccdc0a0b40 2024-09-16T14:49:34.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccdc137440 con 0x7fccdc0a0f50 2024-09-16T14:49:34.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccdc09c360 con 0x7fccdc0a0050 2024-09-16T14:49:34.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:34.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 10 monclient(hunting): authenticate will time out at 1398.681493s 2024-09-16T14:49:34.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce0fc1640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fccdc0a0050 0x7fccdc136620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49594/0 (socket says 172.21.15.26:49594) 2024-09-16T14:49:34.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce0fc1640 1 -- 172.21.15.26:0/4282339901 learned_addr learned my addr 172.21.15.26:0/4282339901 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:34.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703622008 0 0) 0x7fccdc09c360 con 0x7fccdc0a0050 2024-09-16T14:49:34.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): my global_id is 154334 2024-09-16T14:49:34.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc8003530 con 0x7fccdc0a0050 2024-09-16T14:49:34.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036605371 0 0) 0x7fccdc13a610 con 0x7fccdc0a0b40 2024-09-16T14:49:34.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): my global_id is 164097 2024-09-16T14:49:34.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccdc09c360 con 0x7fccdc0a0b40 2024-09-16T14:49:34.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 466126601 0 0) 0x7fccc8003530 con 0x7fccdc0a0050 2024-09-16T14:49:34.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccdc13a610 con 0x7fccdc0a0050 2024-09-16T14:49:34.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fccd4001710 con 0x7fccdc0a0050 2024-09-16T14:49:34.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:34.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): dump: 2024-09-16T14:49:34.577 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.577 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.577 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.577 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.577 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.577 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.577 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.577 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.577 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.577 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2323110496 0 0) 0x7fccdc09c360 con 0x7fccdc0a0b40 2024-09-16T14:49:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccc8003530 con 0x7fccdc0a0b40 2024-09-16T14:49:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fccd0001600 con 0x7fccdc0a0b40 2024-09-16T14:49:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): dump: 2024-09-16T14:49:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.579 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 588001091 0 0) 0x7fccdc13a610 con 0x7fccdc0a0050 2024-09-16T14:49:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 >> v1:172.21.15.189:6789/0 conn(0x7fccdc0a0f50 legacy=0x7fccdc1d1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 >> v1:172.21.15.26:6789/0 conn(0x7fccdc0a0b40 legacy=0x7fccdc136d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient: _finish_hunting 0 2024-09-16T14:49:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 monclient: found mon.c 2024-09-16T14:49:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccdc099400 con 0x7fccdc0a0050 2024-09-16T14:49:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient: _finish_auth 0 2024-09-16T14:49:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient: _check_auth_tickets 2024-09-16T14:49:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 5 monclient: authenticate success, global_id 154334 2024-09-16T14:49:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 10 monclient: _renew_subs 2024-09-16T14:49:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 1 -- 172.21.15.26:0/4282339901 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccdc139d70 con 0x7fccdc0a0050 2024-09-16T14:49:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 10 monclient: _renew_subs 2024-09-16T14:49:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 1 -- 172.21.15.26:0/4282339901 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fccdc138b40 con 0x7fccdc0a0050 2024-09-16T14:49:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccd4001cc0 con 0x7fccdc0a0050 2024-09-16T14:49:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:34.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fccd4004990 con 0x7fccdc0a0050 2024-09-16T14:49:34.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:34.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccdaffd640 4 set_mon_vals no callback set 2024-09-16T14:49:34.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient: dump: 2024-09-16T14:49:34.581 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.581 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.581 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.581 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.581 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.582 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce3225640 1 -- 172.21.15.26:0/4282339901 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fccbc006590 con 0x7fccbc002830 2024-09-16T14:49:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fccd4004bf0 con 0x7fccdc0a0050 2024-09-16T14:49:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(197..197 src has 169..197) v4 ==== 9982+0+0 (unknown 3559933277 0 0) 0x7fccd40449d0 con 0x7fccdc0a0050 2024-09-16T14:49:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96774203 0 0) 0x7fccbc006590 con 0x7fccbc002830 2024-09-16T14:49:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 10 monclient(hunting): my global_id is 164100 2024-09-16T14:49:34.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccdc13a610 con 0x7fccbc002830 2024-09-16T14:49:34.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.558+0000 7fcce17c2640 10 monclient: get_auth_request con 0x7fccc8036220 auth_method 0 2024-09-16T14:49:34.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.562+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2139420170 0 0) 0x7fccdc13a610 con 0x7fccbc002830 2024-09-16T14:49:34.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.562+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccbc006590 con 0x7fccbc002830 2024-09-16T14:49:34.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.562+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fccd0003990 con 0x7fccbc002830 2024-09-16T14:49:34.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.562+0000 7fccd97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.562+0000 7fccd97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:34.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.562+0000 7fccd97fa640 10 monclient: dump: 2024-09-16T14:49:34.583 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.583 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.583 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.583 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.583 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.584 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.584 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.584 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.584 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.584 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.562+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 592603528 0 0) 0x7fccbc006590 con 0x7fccbc002830 2024-09-16T14:49:34.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.562+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fccbc00a0e0 con 0x7fccbc002830 2024-09-16T14:49:34.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.562+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fccd0003c80 con 0x7fccbc002830 2024-09-16T14:49:34.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.562+0000 7fccd97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:34.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.562+0000 7fccd97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:34.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.562+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 >> v1:172.21.15.26:6789/0 conn(0x7fccbc002830 legacy=0x7fccbc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.562+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 >> v1:172.21.15.26:6789/0 conn(0x7fccbc002830 legacy=0x7fccbc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:34.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.566+0000 7fcce3225640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:49:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.566+0000 7fcce3225640 1 -- 172.21.15.26:0/4282339901 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fccbc009da0 con 0x7fccbc0090c0 2024-09-16T14:49:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.566+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287007389 0 0) 0x7fccbc009da0 con 0x7fccbc0090c0 2024-09-16T14:49:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.566+0000 7fccd97fa640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.566+0000 7fccd97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.566+0000 7fccd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.566+0000 7fccd97fa640 10 monclient(hunting): my global_id is 164103 2024-09-16T14:49:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.566+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccbc006590 con 0x7fccbc0090c0 2024-09-16T14:49:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.566+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2126494082 0 0) 0x7fccbc006590 con 0x7fccbc0090c0 2024-09-16T14:49:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.566+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccbc009da0 con 0x7fccbc0090c0 2024-09-16T14:49:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.566+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fccd00042e0 con 0x7fccbc0090c0 2024-09-16T14:49:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.566+0000 7fccd97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.566+0000 7fccd97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:34.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.566+0000 7fccd97fa640 10 monclient: dump: 2024-09-16T14:49:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.586 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.566+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3610919370 0 0) 0x7fccbc009da0 con 0x7fccbc0090c0 2024-09-16T14:49:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.570+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fccbc00c290 con 0x7fccbc0090c0 2024-09-16T14:49:34.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.570+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 3150384701) 0x7fccd0005610 con 0x7fccbc0090c0 2024-09-16T14:49:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.570+0000 7fccd97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:49:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.570+0000 7fccd97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.570+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 >> v1:172.21.15.26:6789/0 conn(0x7fccbc0090c0 legacy=0x7fccbc009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.570+0000 7fccd97fa640 1 -- 172.21.15.26:0/4282339901 >> v1:172.21.15.26:6789/0 conn(0x7fccbc0090c0 legacy=0x7fccbc009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:34.587 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:49:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:49:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:49:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:49:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 66, 2024-09-16T14:49:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:49:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:49:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:49:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:49:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:49:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:49:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:49:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:49:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:49:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:49:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:49:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:49:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:49:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:49:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:49:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:49:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:49:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:49:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:49:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:49:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:49:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:49:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:49:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:49:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:49:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:49:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:49:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:49:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:49:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:49:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:49:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:49:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:49:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:49:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:49:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:49:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:49:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:49:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:49:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:49:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:49:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:49:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:49:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:49:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:49:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:49:34.598 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:49:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.570+0000 7fcce3225640 1 -- 172.21.15.26:0/4282339901 >> v1:172.21.15.189:6812/3196161582 conn(0x7fccc8036220 legacy=0x7fccc80386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.570+0000 7fcce3225640 10 monclient: shutdown 2024-09-16T14:49:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.570+0000 7fcce3225640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.570+0000 7fcce3225640 1 -- 172.21.15.26:0/4282339901 >> v1:172.21.15.26:6790/0 conn(0x7fccdc0a0050 legacy=0x7fccdc136620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.570+0000 7fcce17c2640 1 -- 172.21.15.26:0/4282339901 reap_dead start 2024-09-16T14:49:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.570+0000 7fcce17c2640 1 -- 172.21.15.26:0/4282339901 reap_dead start 2024-09-16T14:49:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.570+0000 7fcce3225640 1 -- 172.21.15.26:0/4282339901 shutdown_connections 2024-09-16T14:49:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.570+0000 7fcce3225640 1 -- 172.21.15.26:0/4282339901 >> 172.21.15.26:0/4282339901 conn(0x7fccdc0a5540 msgr2=0x7fccdc0a3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.570+0000 7fcce3225640 1 -- 172.21.15.26:0/4282339901 shutdown_connections 2024-09-16T14:49:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.570+0000 7fcce3225640 1 -- 172.21.15.26:0/4282339901 wait complete. 2024-09-16T14:49:34.601 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:49:34.601 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:49:34.601 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:49:34.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.584+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}]=0 added bucket h15 type host to location {rack=r1} v198) v1 ==== 168+0+0 (unknown 4176771574 0 0) 0x7fbb98012cd0 con 0x7fbba81393b0 2024-09-16T14:49:34.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.584+0000 7fbb8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:34.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.584+0000 7fbb8f7fe640 10 monclient: _finish_command 2 = system:0 added bucket h15 type host to location {rack=r1} 2024-09-16T14:49:34.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.688+0000 7fbbad14a640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:34.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.688+0000 7fbbad14a640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:34.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.688+0000 7fbbad14a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:34.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.688+0000 7fbbad14a640 1 -- 172.21.15.189:0/2706477477 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fbb88000bd0 con 0x7fbba81393b0 2024-09-16T14:49:34.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.688+0000 7fbb8f7fe640 1 -- 172.21.15.189:0/2706477477 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h15' already exists v198) v1 ==== 147+0+0 (unknown 1569384193 0 0) 0x7fbb98014450 con 0x7fbba81393b0 2024-09-16T14:49:34.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.688+0000 7fbb8f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:34.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.688+0000 7fbb8f7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h15' already exists 2024-09-16T14:49:34.693 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h15' already exists 2024-09-16T14:49:34.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.692+0000 7fbbad14a640 1 -- 172.21.15.189:0/2706477477 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbb94036130 legacy=0x7fbb940385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.692+0000 7fbbad14a640 10 monclient: shutdown 2024-09-16T14:49:34.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.692+0000 7fbbad14a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:34.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.692+0000 7fbbad14a640 1 -- 172.21.15.189:0/2706477477 >> v1:172.21.15.189:6789/0 conn(0x7fbba81393b0 legacy=0x7fbba81d2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.692+0000 7fbbad14a640 1 -- 172.21.15.189:0/2706477477 shutdown_connections 2024-09-16T14:49:34.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.692+0000 7fbbad14a640 1 -- 172.21.15.189:0/2706477477 >> 172.21.15.189:0/2706477477 conn(0x7fbba80a3380 msgr2=0x7fbba80a2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:34.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.692+0000 7fbbad14a640 1 -- 172.21.15.189:0/2706477477 shutdown_connections 2024-09-16T14:49:34.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.692+0000 7fbbad14a640 1 -- 172.21.15.189:0/2706477477 wait complete. 2024-09-16T14:49:34.709 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:34.710 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h16 host rack=r1 2024-09-16T14:49:34.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 10 monclient: get_monmap_and_config 2024-09-16T14:49:34.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 10 monclient: build_initial_monmap 2024-09-16T14:49:34.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 10 monclient: monmap: 2024-09-16T14:49:34.713 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:34.713 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:34.714508+0000 2024-09-16T14:49:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:34.714508+0000 2024-09-16T14:49:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:34.714 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 1 Processor -- start 2024-09-16T14:49:34.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 1 -- start start 2024-09-16T14:49:34.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 10 monclient: init 2024-09-16T14:49:34.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:34.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:49:34.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 10 monclient(hunting): picked mon.noname-c con 0x7f435c0a1ca0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:34.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 10 monclient(hunting): picked mon.noname-b con 0x7f435c0a2790 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:34.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 10 monclient(hunting): picked mon.noname-a con 0x7f435c09ff50 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:34.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c09f520 con 0x7f435c09ff50 2024-09-16T14:49:34.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c0a0360 con 0x7f435c0a1ca0 2024-09-16T14:49:34.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c09e7a0 con 0x7f435c0a2790 2024-09-16T14:49:34.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:34.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 10 monclient(hunting): authenticate will time out at 1398.834531s 2024-09-16T14:49:34.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f4360cb8640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f435c09ff50 0x7f435c13d590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:59918/0 (socket says 172.21.15.26:59918) 2024-09-16T14:49:34.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f4360cb8640 1 -- 172.21.15.26:0/3474199637 learned_addr learned my addr 172.21.15.26:0/3474199637 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:34.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f4353fff640 1 --1- 172.21.15.26:0/3474199637 >> v1:172.21.15.26:6790/0 conn(0x7f435c0a2790 0x7f435c139e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49604/0 (socket says 172.21.15.26:49604) 2024-09-16T14:49:34.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 1 -- 172.21.15.26:0/3474199637 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915467461 0 0) 0x7f435c09f520 con 0x7f435c09ff50 2024-09-16T14:49:34.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient(hunting): my global_id is 164106 2024-09-16T14:49:34.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 1 -- 172.21.15.26:0/3474199637 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4348003530 con 0x7f435c09ff50 2024-09-16T14:49:34.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 1 -- 172.21.15.26:0/3474199637 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924177619 0 0) 0x7f435c09e7a0 con 0x7f435c0a2790 2024-09-16T14:49:34.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient(hunting): my global_id is 154337 2024-09-16T14:49:34.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 1 -- 172.21.15.26:0/3474199637 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c09f520 con 0x7f435c0a2790 2024-09-16T14:49:34.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 1 -- 172.21.15.26:0/3474199637 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16183486 0 0) 0x7f435c0a0360 con 0x7f435c0a1ca0 2024-09-16T14:49:34.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient(hunting): my global_id is 144550 2024-09-16T14:49:34.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 1 -- 172.21.15.26:0/3474199637 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c09e7a0 con 0x7f435c0a1ca0 2024-09-16T14:49:34.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 1 -- 172.21.15.26:0/3474199637 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4200065738 0 0) 0x7f4348003530 con 0x7f435c09ff50 2024-09-16T14:49:34.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 1 -- 172.21.15.26:0/3474199637 >> v1:172.21.15.26:6790/0 conn(0x7f435c0a2790 legacy=0x7f435c139e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 1 -- 172.21.15.26:0/3474199637 >> v1:172.21.15.189:6789/0 conn(0x7f435c0a1ca0 legacy=0x7f435c0a2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:49:34.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 1 monclient: found mon.noname-a 2024-09-16T14:49:34.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:34.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:49:34.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 1 -- 172.21.15.26:0/3474199637 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c09ec80 con 0x7f435c09ff50 2024-09-16T14:49:34.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:49:34.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:34.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:34.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 1 -- 172.21.15.26:0/3474199637 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f43540018d0 con 0x7f435c09ff50 2024-09-16T14:49:34.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 5 monclient: authenticate success, global_id 164106 2024-09-16T14:49:34.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:34.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:34.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient: dump: 2024-09-16T14:49:34.728 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.728 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.728 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.728 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.728 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.728 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.728 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.728 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.728 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.728 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 1 -- 172.21.15.26:0/3474199637 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4354001bb0 con 0x7f435c09ff50 2024-09-16T14:49:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 1 -- 172.21.15.26:0/3474199637 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4354001f40 con 0x7f435c09ff50 2024-09-16T14:49:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f435b7fe640 10 monclient: dump: 2024-09-16T14:49:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.730 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.710+0000 7f436271b640 4 set_mon_vals no callback set 2024-09-16T14:49:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 10 monclient: shutdown 2024-09-16T14:49:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 1 -- 172.21.15.26:0/3474199637 >> v1:172.21.15.26:6789/0 conn(0x7f435c09ff50 legacy=0x7f435c13d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 1 -- 172.21.15.26:0/3474199637 shutdown_connections 2024-09-16T14:49:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 1 -- 172.21.15.26:0/3474199637 >> 172.21.15.26:0/3474199637 conn(0x7f435c0a3da0 msgr2=0x7f435c0a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 1 -- 172.21.15.26:0/3474199637 shutdown_connections 2024-09-16T14:49:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 1 -- 172.21.15.26:0/3474199637 wait complete. 2024-09-16T14:49:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 10 monclient: build_initial_monmap 2024-09-16T14:49:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 10 monclient: monmap: 2024-09-16T14:49:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:34.732 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 1 Processor -- start 2024-09-16T14:49:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 1 -- start start 2024-09-16T14:49:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 10 monclient: init 2024-09-16T14:49:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 10 monclient(hunting): picked mon.noname-b con 0x7f435c09ff50 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 10 monclient(hunting): picked mon.noname-a con 0x7f435c0a1ca0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 10 monclient(hunting): picked mon.noname-c con 0x7f435c0a2790 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c1336e0 con 0x7f435c0a1ca0 2024-09-16T14:49:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c131170 con 0x7f435c09ff50 2024-09-16T14:49:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c133950 con 0x7f435c0a2790 2024-09-16T14:49:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 10 monclient(hunting): authenticate will time out at 1398.837646s 2024-09-16T14:49:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f4360cb8640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f435c0a2790 0x7f435c1d1bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49612/0 (socket says 172.21.15.26:49612) 2024-09-16T14:49:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f4360cb8640 1 -- 172.21.15.26:0/2282412620 learned_addr learned my addr 172.21.15.26:0/2282412620 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760546144 0 0) 0x7f435c133950 con 0x7f435c0a2790 2024-09-16T14:49:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): my global_id is 154340 2024-09-16T14:49:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f433c003530 con 0x7f435c0a2790 2024-09-16T14:49:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465448107 0 0) 0x7f435c1336e0 con 0x7f435c0a1ca0 2024-09-16T14:49:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): my global_id is 164109 2024-09-16T14:49:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c133950 con 0x7f435c0a1ca0 2024-09-16T14:49:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507568170 0 0) 0x7f435c131170 con 0x7f435c09ff50 2024-09-16T14:49:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): my global_id is 144553 2024-09-16T14:49:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c1336e0 con 0x7f435c09ff50 2024-09-16T14:49:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1583134083 0 0) 0x7f433c003530 con 0x7f435c0a2790 2024-09-16T14:49:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f435c131170 con 0x7f435c0a2790 2024-09-16T14:49:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4354001890 con 0x7f435c0a2790 2024-09-16T14:49:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): dump: 2024-09-16T14:49:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.735 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3602930545 0 0) 0x7f435c133950 con 0x7f435c0a1ca0 2024-09-16T14:49:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f433c003530 con 0x7f435c0a1ca0 2024-09-16T14:49:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f43440018c0 con 0x7f435c0a1ca0 2024-09-16T14:49:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): dump: 2024-09-16T14:49:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.737 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1057658928 0 0) 0x7f435c131170 con 0x7f435c0a2790 2024-09-16T14:49:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 >> v1:172.21.15.189:6789/0 conn(0x7f435c09ff50 legacy=0x7f435c130350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 >> v1:172.21.15.26:6789/0 conn(0x7f435c0a1ca0 legacy=0x7f435c130a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient: _finish_hunting 0 2024-09-16T14:49:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 monclient: found mon.c 2024-09-16T14:49:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c133c30 con 0x7f435c0a2790 2024-09-16T14:49:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient: _finish_auth 0 2024-09-16T14:49:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient: _check_auth_tickets 2024-09-16T14:49:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 5 monclient: authenticate success, global_id 154340 2024-09-16T14:49:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 10 monclient: _renew_subs 2024-09-16T14:49:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 1 -- 172.21.15.26:0/2282412620 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435c133f10 con 0x7f435c0a2790 2024-09-16T14:49:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 10 monclient: _renew_subs 2024-09-16T14:49:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 1 -- 172.21.15.26:0/2282412620 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f435c131e70 con 0x7f435c0a2790 2024-09-16T14:49:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43540014b0 con 0x7f435c0a2790 2024-09-16T14:49:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4354004fc0 con 0x7f435c0a2790 2024-09-16T14:49:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f435affd640 4 set_mon_vals no callback set 2024-09-16T14:49:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient: dump: 2024-09-16T14:49:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.739 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f436271b640 1 -- 172.21.15.26:0/2282412620 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4340006590 con 0x7f4340002830 2024-09-16T14:49:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f435400fa30 con 0x7f435c0a2790 2024-09-16T14:49:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(198..198 src has 169..198) v4 ==== 10195+0+0 (unknown 798001432 0 0) 0x7f4354044f80 con 0x7f435c0a2790 2024-09-16T14:49:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745881339 0 0) 0x7f4340006590 con 0x7f4340002830 2024-09-16T14:49:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 10 monclient(hunting): my global_id is 154343 2024-09-16T14:49:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c131170 con 0x7f4340002830 2024-09-16T14:49:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f4353fff640 10 monclient: get_auth_request con 0x7f433c036390 auth_method 0 2024-09-16T14:49:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.714+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4081561661 0 0) 0x7f435c131170 con 0x7f4340002830 2024-09-16T14:49:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.718+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4340006590 con 0x7f4340002830 2024-09-16T14:49:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.718+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f434c0017d0 con 0x7f4340002830 2024-09-16T14:49:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.718+0000 7f43597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.718+0000 7f43597fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:34.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.718+0000 7f43597fa640 10 monclient: dump: 2024-09-16T14:49:34.741 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.741 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.741 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.741 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.741 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.741 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.741 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.741 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.741 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.741 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.718+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2868952936 0 0) 0x7f4340006590 con 0x7f4340002830 2024-09-16T14:49:34.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.718+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f434000a0e0 con 0x7f4340002830 2024-09-16T14:49:34.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.718+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f434c001b40 con 0x7f4340002830 2024-09-16T14:49:34.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.718+0000 7f43597fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:34.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.718+0000 7f43597fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:34.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.718+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 >> v1:172.21.15.26:6790/0 conn(0x7f4340002830 legacy=0x7f4340004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.718+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 >> v1:172.21.15.26:6790/0 conn(0x7f4340002830 legacy=0x7f4340004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:34.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f436271b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:49:34.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f436271b640 1 -- 172.21.15.26:0/2282412620 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4340009d80 con 0x7f43400090a0 2024-09-16T14:49:34.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454128362 0 0) 0x7f4340009d80 con 0x7f43400090a0 2024-09-16T14:49:34.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 10 monclient(hunting): my global_id is 154346 2024-09-16T14:49:34.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4340006590 con 0x7f43400090a0 2024-09-16T14:49:34.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3023183582 0 0) 0x7f4340006590 con 0x7f43400090a0 2024-09-16T14:49:34.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4340009d80 con 0x7f43400090a0 2024-09-16T14:49:34.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f434c0039b0 con 0x7f43400090a0 2024-09-16T14:49:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 10 monclient: dump: 2024-09-16T14:49:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.745 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.745 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4205672819 0 0) 0x7f4340009d80 con 0x7f43400090a0 2024-09-16T14:49:34.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f434000c290 con 0x7f43400090a0 2024-09-16T14:49:34.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 795275171) 0x7f434c005b90 con 0x7f43400090a0 2024-09-16T14:49:34.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:49:34.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:34.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 >> v1:172.21.15.26:6790/0 conn(0x7f43400090a0 legacy=0x7f4340009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f43597fa640 1 -- 172.21.15.26:0/2282412620 >> v1:172.21.15.26:6790/0 conn(0x7f43400090a0 legacy=0x7f4340009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:34.746 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:49:34.746 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:49:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:49:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:49:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 66, 2024-09-16T14:49:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:49:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:49:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:49:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:49:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:49:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:49:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:49:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:49:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:49:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:49:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:49:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:49:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:49:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:49:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:49:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:49:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:49:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:49:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:49:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:49:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:49:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:49:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:49:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:49:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:49:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:49:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:49:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:49:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:49:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:49:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:49:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:49:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:49:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:49:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:49:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:49:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:49:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:49:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:49:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:49:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:49:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:49:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:49:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:49:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:49:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:49:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:49:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:49:34.758 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:49:34.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f436271b640 1 -- 172.21.15.26:0/2282412620 >> v1:172.21.15.189:6812/3196161582 conn(0x7f433c036390 legacy=0x7f433c038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f436271b640 10 monclient: shutdown 2024-09-16T14:49:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f436271b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f436271b640 1 -- 172.21.15.26:0/2282412620 >> v1:172.21.15.26:6790/0 conn(0x7f435c0a2790 legacy=0x7f435c1d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f4360cb8640 1 -- 172.21.15.26:0/2282412620 reap_dead start 2024-09-16T14:49:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f4360cb8640 1 -- 172.21.15.26:0/2282412620 reap_dead start 2024-09-16T14:49:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f436271b640 1 -- 172.21.15.26:0/2282412620 shutdown_connections 2024-09-16T14:49:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f436271b640 1 -- 172.21.15.26:0/2282412620 >> 172.21.15.26:0/2282412620 conn(0x7f435c0a3da0 msgr2=0x7f435c12ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f436271b640 1 -- 172.21.15.26:0/2282412620 shutdown_connections 2024-09-16T14:49:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.722+0000 7f436271b640 1 -- 172.21.15.26:0/2282412620 wait complete. 2024-09-16T14:49:34.761 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:49:34.761 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:49:34.761 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:49:34.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 10 monclient: get_monmap_and_config 2024-09-16T14:49:34.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 10 monclient: build_initial_monmap 2024-09-16T14:49:34.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 10 monclient: monmap: 2024-09-16T14:49:34.830 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:34.830 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.830 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:34.830786+0000 2024-09-16T14:49:34.830 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:34.830786+0000 2024-09-16T14:49:34.830 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:34.830 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:34.830 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:34.831 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:34.831 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:34.831 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:34.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 1 Processor -- start 2024-09-16T14:49:34.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 1 -- start start 2024-09-16T14:49:34.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 10 monclient: init 2024-09-16T14:49:34.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:34.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:49:34.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 10 monclient(hunting): picked mon.noname-c con 0x7f1ae8099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:34.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 10 monclient(hunting): picked mon.noname-b con 0x7f1ae809c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:34.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 10 monclient(hunting): picked mon.noname-a con 0x7f1ae809e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:34.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae8099e50 con 0x7f1ae809e6d0 2024-09-16T14:49:34.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae809dbd0 con 0x7f1ae8099a70 2024-09-16T14:49:34.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae809ee20 con 0x7f1ae809c550 2024-09-16T14:49:34.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:34.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 10 monclient(hunting): authenticate will time out at 1356.946551s 2024-09-16T14:49:34.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aee380640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1ae8099a70 0x7f1ae809dfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60286/0 (socket says 172.21.15.189:60286) 2024-09-16T14:49:34.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aee380640 1 -- 172.21.15.189:0/4115637549 learned_addr learned my addr 172.21.15.189:0/4115637549 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:34.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 1 -- 172.21.15.189:0/4115637549 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101050311 0 0) 0x7f1ae809dbd0 con 0x7f1ae8099a70 2024-09-16T14:49:34.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient(hunting): my global_id is 144556 2024-09-16T14:49:34.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 1 -- 172.21.15.189:0/4115637549 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad4003530 con 0x7f1ae8099a70 2024-09-16T14:49:34.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 1 -- 172.21.15.189:0/4115637549 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909390325 0 0) 0x7f1ae8099e50 con 0x7f1ae809e6d0 2024-09-16T14:49:34.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient(hunting): my global_id is 164112 2024-09-16T14:49:34.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 1 -- 172.21.15.189:0/4115637549 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae809dbd0 con 0x7f1ae809e6d0 2024-09-16T14:49:34.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 1 -- 172.21.15.189:0/4115637549 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597077047 0 0) 0x7f1ae809ee20 con 0x7f1ae809c550 2024-09-16T14:49:34.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient(hunting): my global_id is 154349 2024-09-16T14:49:34.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 1 -- 172.21.15.189:0/4115637549 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae8099e50 con 0x7f1ae809c550 2024-09-16T14:49:34.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 1 -- 172.21.15.189:0/4115637549 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4116305444 0 0) 0x7f1ad4003530 con 0x7f1ae8099a70 2024-09-16T14:49:34.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 1 -- 172.21.15.189:0/4115637549 >> v1:172.21.15.26:6790/0 conn(0x7f1ae809c550 legacy=0x7f1ae81379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 1 -- 172.21.15.189:0/4115637549 >> v1:172.21.15.26:6789/0 conn(0x7f1ae809e6d0 legacy=0x7f1ae813b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient: _finish_hunting 0 2024-09-16T14:49:34.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 1 monclient: found mon.noname-c 2024-09-16T14:49:34.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:34.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:34.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 1 -- 172.21.15.189:0/4115637549 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae809f100 con 0x7f1ae8099a70 2024-09-16T14:49:34.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient: _finish_auth 0 2024-09-16T14:49:34.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient: _check_auth_tickets 2024-09-16T14:49:34.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:34.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 1 -- 172.21.15.189:0/4115637549 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1ae4001620 con 0x7f1ae8099a70 2024-09-16T14:49:34.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:34.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient: dump: 2024-09-16T14:49:34.842 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:34.842 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.842 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.843 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.843 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.843 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:34.843 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.843 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.843 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.843 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:34.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 5 monclient: authenticate success, global_id 144556 2024-09-16T14:49:34.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:34.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 1 -- 172.21.15.189:0/4115637549 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ae4001890 con 0x7f1ae8099a70 2024-09-16T14:49:34.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:34.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 1 -- 172.21.15.189:0/4115637549 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1ae4001c60 con 0x7f1ae8099a70 2024-09-16T14:49:34.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:34.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1aed37e640 10 monclient: dump: 2024-09-16T14:49:34.844 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:34.844 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.844 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.844 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.844 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.844 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:34.844 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.844 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.845 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.845 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:34.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:34.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 4 set_mon_vals no callback set 2024-09-16T14:49:34.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 10 monclient: shutdown 2024-09-16T14:49:34.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:34.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.828+0000 7f1af05e4640 1 -- 172.21.15.189:0/4115637549 >> v1:172.21.15.189:6789/0 conn(0x7f1ae8099a70 legacy=0x7f1ae809dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 1 -- 172.21.15.189:0/4115637549 shutdown_connections 2024-09-16T14:49:34.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 1 -- 172.21.15.189:0/4115637549 >> 172.21.15.189:0/4115637549 conn(0x7f1ae80a3380 msgr2=0x7f1ae809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:34.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 1 -- 172.21.15.189:0/4115637549 shutdown_connections 2024-09-16T14:49:34.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 1 -- 172.21.15.189:0/4115637549 wait complete. 2024-09-16T14:49:34.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 monclient: build_initial_monmap 2024-09-16T14:49:34.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 monclient: monmap: 2024-09-16T14:49:34.846 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:34.846 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:34.846 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:34.846 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:34.846 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:34.846 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:34.846 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:34.846 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:34.846 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:34.847 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:34.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 1 Processor -- start 2024-09-16T14:49:34.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 1 -- start start 2024-09-16T14:49:34.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 monclient: init 2024-09-16T14:49:34.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:34.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:49:34.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 monclient(hunting): picked mon.noname-c con 0x7f1ae8099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:34.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 monclient(hunting): picked mon.noname-b con 0x7f1ae809c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:34.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 monclient(hunting): picked mon.noname-a con 0x7f1ae809e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:34.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae809cf20 con 0x7f1ae809e6d0 2024-09-16T14:49:34.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae813b630 con 0x7f1ae809c550 2024-09-16T14:49:34.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae813b910 con 0x7f1ae8099a70 2024-09-16T14:49:34.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:34.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 monclient(hunting): authenticate will time out at 1356.949486s 2024-09-16T14:49:34.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1aedb7f640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1ae809c550 0x7f1ae809a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60290/0 (socket says 172.21.15.189:60290) 2024-09-16T14:49:34.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1aedb7f640 1 -- 172.21.15.189:0/1680309050 learned_addr learned my addr 172.21.15.189:0/1680309050 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:34.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256449628 0 0) 0x7f1ae813b630 con 0x7f1ae809c550 2024-09-16T14:49:34.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): my global_id is 144559 2024-09-16T14:49:34.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1acc003530 con 0x7f1ae809c550 2024-09-16T14:49:34.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487241337 0 0) 0x7f1ae809cf20 con 0x7f1ae809e6d0 2024-09-16T14:49:34.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): my global_id is 164115 2024-09-16T14:49:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae813b630 con 0x7f1ae809e6d0 2024-09-16T14:49:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593675979 0 0) 0x7f1ae813b910 con 0x7f1ae8099a70 2024-09-16T14:49:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): my global_id is 154352 2024-09-16T14:49:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae809cf20 con 0x7f1ae8099a70 2024-09-16T14:49:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1197899638 0 0) 0x7f1acc003530 con 0x7f1ae809c550 2024-09-16T14:49:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ae813b910 con 0x7f1ae809c550 2024-09-16T14:49:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1ad8001940 con 0x7f1ae809c550 2024-09-16T14:49:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:34.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): dump: 2024-09-16T14:49:34.850 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:34.850 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.850 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.850 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.850 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.850 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:34.850 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.850 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.850 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.850 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:34.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:34.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2115774359 0 0) 0x7f1ae813b630 con 0x7f1ae809e6d0 2024-09-16T14:49:34.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1acc003530 con 0x7f1ae809e6d0 2024-09-16T14:49:34.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1ae0001bc0 con 0x7f1ae809e6d0 2024-09-16T14:49:34.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:34.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient(hunting): dump: 2024-09-16T14:49:34.851 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:34.851 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.851 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.851 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.851 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.851 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:34.851 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.852 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.852 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.852 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:34.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1819379826 0 0) 0x7f1ae813b910 con 0x7f1ae809c550 2024-09-16T14:49:34.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 >> v1:172.21.15.26:6790/0 conn(0x7f1ae8099a70 legacy=0x7f1ae809a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 >> v1:172.21.15.26:6789/0 conn(0x7f1ae809e6d0 legacy=0x7f1ae81d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:34.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 monclient: found mon.b 2024-09-16T14:49:34.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:34.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:34.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae809b090 con 0x7f1ae809c550 2024-09-16T14:49:34.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient: _finish_auth 0 2024-09-16T14:49:34.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:34.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:34.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 5 monclient: authenticate success, global_id 144559 2024-09-16T14:49:34.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 monclient: _renew_subs 2024-09-16T14:49:34.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:34.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 1 -- 172.21.15.189:0/1680309050 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae806b380 con 0x7f1ae809c550 2024-09-16T14:49:34.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 monclient: _renew_subs 2024-09-16T14:49:34.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:34.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 1 -- 172.21.15.189:0/1680309050 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae813b070 con 0x7f1ae809c550 2024-09-16T14:49:34.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ad8003620 con 0x7f1ae809c550 2024-09-16T14:49:34.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:34.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1ad8003930 con 0x7f1ae809c550 2024-09-16T14:49:34.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:34.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1aecb7d640 4 set_mon_vals no callback set 2024-09-16T14:49:34.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 10 monclient: dump: 2024-09-16T14:49:34.854 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:34.854 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.854 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.854 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.854 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.854 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:34.854 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.854 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.854 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.855 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:34.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:34.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:34.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:34.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1af05e4640 1 -- 172.21.15.189:0/1680309050 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad0005050 con 0x7f1ae809c550 2024-09-16T14:49:34.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1ad8003a90 con 0x7f1ae809c550 2024-09-16T14:49:34.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(198..198 src has 169..198) v4 ==== 10195+0+0 (unknown 798001432 0 0) 0x7f1ad8044ab0 con 0x7f1ae809c550 2024-09-16T14:49:34.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.832+0000 7f1aee380640 10 monclient: get_auth_request con 0x7f1acc036440 auth_method 0 2024-09-16T14:49:34.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.840+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1ad8004260 con 0x7f1ae809c550 2024-09-16T14:49:34.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.840+0000 7f1adeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:34.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:34.840+0000 7f1adeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.874+0000 7fb1d7980640 10 monclient: get_monmap_and_config 2024-09-16T14:49:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.874+0000 7fb1d7980640 10 monclient: build_initial_monmap 2024-09-16T14:49:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.874+0000 7fb1d7980640 10 monclient: monmap: 2024-09-16T14:49:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:34.880135+0000 2024-09-16T14:49:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:34.880135+0000 2024-09-16T14:49:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:34.880 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.874+0000 7fb1d7980640 1 Processor -- start 2024-09-16T14:49:34.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.874+0000 7fb1d7980640 1 -- start start 2024-09-16T14:49:34.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.874+0000 7fb1d7980640 10 monclient: init 2024-09-16T14:49:34.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:34.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:49:34.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient(hunting): picked mon.noname-b con 0x7fb1d009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:34.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient(hunting): picked mon.noname-c con 0x7fb1d0132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:34.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient(hunting): picked mon.noname-a con 0x7fb1d0138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:34.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d009d260 con 0x7fb1d0138ff0 2024-09-16T14:49:34.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d0132a80 con 0x7fb1d0132650 2024-09-16T14:49:34.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d009cf60 con 0x7fb1d009cb80 2024-09-16T14:49:34.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:34.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient(hunting): authenticate will time out at 1399.000173s 2024-09-16T14:49:34.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d571c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb1d009cb80 0x7fb1d00717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49640/0 (socket says 172.21.15.26:49640) 2024-09-16T14:49:34.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d571c640 1 -- 172.21.15.26:0/16031870 learned_addr learned my addr 172.21.15.26:0/16031870 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:34.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 1 -- 172.21.15.26:0/16031870 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657031647 0 0) 0x7fb1d009cf60 con 0x7fb1d009cb80 2024-09-16T14:49:34.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient(hunting): my global_id is 154355 2024-09-16T14:49:34.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 1 -- 172.21.15.26:0/16031870 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1bc003530 con 0x7fb1d009cb80 2024-09-16T14:49:34.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 1 -- 172.21.15.26:0/16031870 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322336711 0 0) 0x7fb1d009d260 con 0x7fb1d0138ff0 2024-09-16T14:49:34.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient(hunting): my global_id is 164118 2024-09-16T14:49:34.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 1 -- 172.21.15.26:0/16031870 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d009cf60 con 0x7fb1d0138ff0 2024-09-16T14:49:34.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 1 -- 172.21.15.26:0/16031870 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785804132 0 0) 0x7fb1d0132a80 con 0x7fb1d0132650 2024-09-16T14:49:34.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient(hunting): my global_id is 144562 2024-09-16T14:49:34.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 1 -- 172.21.15.26:0/16031870 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d009d260 con 0x7fb1d0132650 2024-09-16T14:49:34.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 1 -- 172.21.15.26:0/16031870 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3247852396 0 0) 0x7fb1d009cf60 con 0x7fb1d0138ff0 2024-09-16T14:49:34.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 1 -- 172.21.15.26:0/16031870 >> v1:172.21.15.26:6790/0 conn(0x7fb1d009cb80 legacy=0x7fb1d00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 1 -- 172.21.15.26:0/16031870 >> v1:172.21.15.189:6789/0 conn(0x7fb1d0132650 legacy=0x7fb1d01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:49:34.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 1 monclient: found mon.noname-a 2024-09-16T14:49:34.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:34.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:49:34.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 1 -- 172.21.15.26:0/16031870 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d009f2d0 con 0x7fb1d0138ff0 2024-09-16T14:49:34.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient: _finish_auth 0 2024-09-16T14:49:34.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:49:34.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:34.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 1 -- 172.21.15.26:0/16031870 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb1cc001600 con 0x7fb1d0138ff0 2024-09-16T14:49:34.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 5 monclient: authenticate success, global_id 164118 2024-09-16T14:49:34.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:34.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:34.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient: dump: 2024-09-16T14:49:34.893 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.893 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.893 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.893 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.893 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.893 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.893 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.893 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.894 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.894 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:34.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 1 -- 172.21.15.26:0/16031870 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1cc001830 con 0x7fb1d0138ff0 2024-09-16T14:49:34.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:34.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:34.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 1 -- 172.21.15.26:0/16031870 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb1cc001c00 con 0x7fb1d0138ff0 2024-09-16T14:49:34.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 4 set_mon_vals no callback set 2024-09-16T14:49:34.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:34.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1c7fff640 10 monclient: dump: 2024-09-16T14:49:34.894 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.895 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient: shutdown 2024-09-16T14:49:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 1 -- 172.21.15.26:0/16031870 >> v1:172.21.15.26:6789/0 conn(0x7fb1d0138ff0 legacy=0x7fb1d013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 1 -- 172.21.15.26:0/16031870 shutdown_connections 2024-09-16T14:49:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 1 -- 172.21.15.26:0/16031870 >> 172.21.15.26:0/16031870 conn(0x7fb1d009fb50 msgr2=0x7fb1d009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 1 -- 172.21.15.26:0/16031870 shutdown_connections 2024-09-16T14:49:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 1 -- 172.21.15.26:0/16031870 wait complete. 2024-09-16T14:49:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient: build_initial_monmap 2024-09-16T14:49:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient: monmap: 2024-09-16T14:49:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:34.898 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:34.898 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:34.898 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:34.898 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:34.899 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 1 Processor -- start 2024-09-16T14:49:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 1 -- start start 2024-09-16T14:49:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient: init 2024-09-16T14:49:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient(hunting): picked mon.noname-a con 0x7fb1d009cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient(hunting): picked mon.noname-b con 0x7fb1d0132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient(hunting): picked mon.noname-c con 0x7fb1d0138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d009a240 con 0x7fb1d009cb80 2024-09-16T14:49:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d006e450 con 0x7fb1d0132650 2024-09-16T14:49:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d571c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb1d009cb80 0x7fb1d006d630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:59956/0 (socket says 172.21.15.26:59956) 2024-09-16T14:49:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d013c590 con 0x7fb1d0138ff0 2024-09-16T14:49:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d571c640 1 -- 172.21.15.26:0/1686909071 learned_addr learned my addr 172.21.15.26:0/1686909071 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.878+0000 7fb1d7980640 10 monclient(hunting): authenticate will time out at 1399.003931s 2024-09-16T14:49:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514589651 0 0) 0x7fb1d009a240 con 0x7fb1d009cb80 2024-09-16T14:49:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): my global_id is 164121 2024-09-16T14:49:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1b4003530 con 0x7fb1d009cb80 2024-09-16T14:49:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52578005 0 0) 0x7fb1d013c590 con 0x7fb1d0138ff0 2024-09-16T14:49:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): my global_id is 154358 2024-09-16T14:49:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d009a240 con 0x7fb1d0138ff0 2024-09-16T14:49:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186657803 0 0) 0x7fb1d006e450 con 0x7fb1d0132650 2024-09-16T14:49:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): my global_id is 144565 2024-09-16T14:49:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d013c590 con 0x7fb1d0132650 2024-09-16T14:49:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1558772393 0 0) 0x7fb1b4003530 con 0x7fb1d009cb80 2024-09-16T14:49:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1d006e450 con 0x7fb1d009cb80 2024-09-16T14:49:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb1c8001b80 con 0x7fb1d009cb80 2024-09-16T14:49:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): dump: 2024-09-16T14:49:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.904 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 689491920 0 0) 0x7fb1d009a240 con 0x7fb1d0138ff0 2024-09-16T14:49:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1b4003530 con 0x7fb1d0138ff0 2024-09-16T14:49:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb1cc001500 con 0x7fb1d0138ff0 2024-09-16T14:49:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): dump: 2024-09-16T14:49:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.906 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3681326705 0 0) 0x7fb1d013c590 con 0x7fb1d0132650 2024-09-16T14:49:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1d009a240 con 0x7fb1d0132650 2024-09-16T14:49:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb1c0001820 con 0x7fb1d0132650 2024-09-16T14:49:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): dump: 2024-09-16T14:49:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.907 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1132188049 0 0) 0x7fb1d006e450 con 0x7fb1d009cb80 2024-09-16T14:49:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 >> v1:172.21.15.26:6790/0 conn(0x7fb1d0138ff0 legacy=0x7fb1d01cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 >> v1:172.21.15.189:6789/0 conn(0x7fb1d0132650 legacy=0x7fb1d006dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:49:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 monclient: found mon.a 2024-09-16T14:49:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d00715e0 con 0x7fb1d009cb80 2024-09-16T14:49:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient: _finish_auth 0 2024-09-16T14:49:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:49:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1d7980640 5 monclient: authenticate success, global_id 164121 2024-09-16T14:49:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1d7980640 10 monclient: _renew_subs 2024-09-16T14:49:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1d7980640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1d7980640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d01380c0 con 0x7fb1d009cb80 2024-09-16T14:49:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1d7980640 10 monclient: _renew_subs 2024-09-16T14:49:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1d7980640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1d7980640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d006fbd0 con 0x7fb1d009cb80 2024-09-16T14:49:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1c8004b70 con 0x7fb1d009cb80 2024-09-16T14:49:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb1c8004e80 con 0x7fb1d009cb80 2024-09-16T14:49:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c77fe640 4 set_mon_vals no callback set 2024-09-16T14:49:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient: dump: 2024-09-16T14:49:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.910 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1d7980640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1d7980640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb1b0006590 con 0x7fb1b0002830 2024-09-16T14:49:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb1c800f8f0 con 0x7fb1d009cb80 2024-09-16T14:49:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(198..198 src has 169..198) v4 ==== 10195+0+0 (unknown 798001432 0 0) 0x7fb1c8044df0 con 0x7fb1d009cb80 2024-09-16T14:49:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790223645 0 0) 0x7fb1b0006590 con 0x7fb1b0002830 2024-09-16T14:49:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient(hunting): my global_id is 144568 2024-09-16T14:49:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d006e450 con 0x7fb1b0002830 2024-09-16T14:49:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1d5f1d640 10 monclient: get_auth_request con 0x7fb1b4036470 auth_method 0 2024-09-16T14:49:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2835714992 0 0) 0x7fb1d006e450 con 0x7fb1b0002830 2024-09-16T14:49:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1b0006590 con 0x7fb1b0002830 2024-09-16T14:49:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb1c0003600 con 0x7fb1b0002830 2024-09-16T14:49:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient: dump: 2024-09-16T14:49:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.912 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3936418202 0 0) 0x7fb1b0006590 con 0x7fb1b0002830 2024-09-16T14:49:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb1b000a0e0 con 0x7fb1b0002830 2024-09-16T14:49:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fb1c0005860 con 0x7fb1b0002830 2024-09-16T14:49:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 >> v1:172.21.15.189:6789/0 conn(0x7fb1b0002830 legacy=0x7fb1b0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.882+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 >> v1:172.21.15.189:6789/0 conn(0x7fb1b0002830 legacy=0x7fb1b0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:34.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1d7980640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:49:34.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1d7980640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb1b0009da0 con 0x7fb1b00090c0 2024-09-16T14:49:34.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761518370 0 0) 0x7fb1b0009da0 con 0x7fb1b00090c0 2024-09-16T14:49:34.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:34.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:34.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:34.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 10 monclient(hunting): my global_id is 144571 2024-09-16T14:49:34.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1b0006590 con 0x7fb1b00090c0 2024-09-16T14:49:34.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3260227221 0 0) 0x7fb1b0006590 con 0x7fb1b00090c0 2024-09-16T14:49:34.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1b0009da0 con 0x7fb1b00090c0 2024-09-16T14:49:34.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb1c0003d60 con 0x7fb1b00090c0 2024-09-16T14:49:34.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:34.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:34.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 10 monclient: dump: 2024-09-16T14:49:34.915 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:34.915 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:34.915 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.915 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:34.916 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:34.916 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:34.916 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:34.916 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:34.916 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:34.916 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:34.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1828871485 0 0) 0x7fb1b0009da0 con 0x7fb1b00090c0 2024-09-16T14:49:34.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb1b000c290 con 0x7fb1b00090c0 2024-09-16T14:49:34.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 546394841) 0x7fb1c0005a70 con 0x7fb1b00090c0 2024-09-16T14:49:34.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:49:34.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:34.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 >> v1:172.21.15.189:6789/0 conn(0x7fb1b00090c0 legacy=0x7fb1b0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1c5ffb640 1 -- 172.21.15.26:0/1686909071 >> v1:172.21.15.189:6789/0 conn(0x7fb1b00090c0 legacy=0x7fb1b0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:34.917 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:49:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:49:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:49:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:49:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 66, 2024-09-16T14:49:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:49:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:49:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:49:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:49:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:49:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:49:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:49:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:49:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:49:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:49:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:49:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:49:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:49:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:49:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:49:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:49:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:49:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:49:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:49:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:49:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:49:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:49:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:49:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:49:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:49:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:49:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:49:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:49:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:49:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:49:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:49:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:49:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:49:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:49:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:49:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:49:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:49:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:49:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:49:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:49:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:49:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:49:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:49:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:49:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:49:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:49:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:49:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:49:34.928 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:49:34.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.890+0000 7fb1d7980640 1 -- 172.21.15.26:0/1686909071 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb1b4036470 legacy=0x7fb1b4038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.894+0000 7fb1d7980640 10 monclient: shutdown 2024-09-16T14:49:34.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.894+0000 7fb1d7980640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:34.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.894+0000 7fb1d7980640 1 -- 172.21.15.26:0/1686909071 >> v1:172.21.15.26:6789/0 conn(0x7fb1d009cb80 legacy=0x7fb1d006d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:34.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.894+0000 7fb1d5f1d640 1 -- 172.21.15.26:0/1686909071 reap_dead start 2024-09-16T14:49:34.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.894+0000 7fb1d5f1d640 1 -- 172.21.15.26:0/1686909071 reap_dead start 2024-09-16T14:49:34.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.894+0000 7fb1d7980640 1 -- 172.21.15.26:0/1686909071 shutdown_connections 2024-09-16T14:49:34.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.894+0000 7fb1d7980640 1 -- 172.21.15.26:0/1686909071 >> 172.21.15.26:0/1686909071 conn(0x7fb1d009fb50 msgr2=0x7fb1d012eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:34.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.894+0000 7fb1d7980640 1 -- 172.21.15.26:0/1686909071 shutdown_connections 2024-09-16T14:49:34.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:34.894+0000 7fb1d7980640 1 -- 172.21.15.26:0/1686909071 wait complete. 2024-09-16T14:49:34.931 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:49:34.931 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:49:34.931 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:49:35.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.004+0000 7f1af05e4640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:35.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.004+0000 7f1af05e4640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:35.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.004+0000 7f1af05e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:35.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.004+0000 7f1af05e4640 1 -- 172.21.15.189:0/1680309050 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f1ad0005300 con 0x7f1ae809c550 2024-09-16T14:49:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 10 monclient: get_monmap_and_config 2024-09-16T14:49:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 10 monclient: build_initial_monmap 2024-09-16T14:49:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 10 monclient: monmap: 2024-09-16T14:49:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:35.045698+0000 2024-09-16T14:49:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:35.045698+0000 2024-09-16T14:49:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:35.045 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:35.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 1 Processor -- start 2024-09-16T14:49:35.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 1 -- start start 2024-09-16T14:49:35.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 10 monclient: init 2024-09-16T14:49:35.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:35.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:35.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 10 monclient(hunting): picked mon.noname-a con 0x7fe0ac0a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:35.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 10 monclient(hunting): picked mon.noname-c con 0x7fe0ac0a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:35.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 10 monclient(hunting): picked mon.noname-b con 0x7fe0ac0a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:35.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac0a3250 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac09f600 con 0x7fe0ac0a0000 2024-09-16T14:49:35.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac0a3830 con 0x7fe0ac0a2800 2024-09-16T14:49:35.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:35.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 10 monclient(hunting): authenticate will time out at 1399.165361s 2024-09-16T14:49:35.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b1fd1640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe0ac0a2800 0x7fe0ac13d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49658/0 (socket says 172.21.15.26:49658) 2024-09-16T14:49:35.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b1fd1640 1 -- 172.21.15.26:0/3443349833 learned_addr learned my addr 172.21.15.26:0/3443349833 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:35.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b17d0640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe0ac0a1d10 0x7fe0ac0a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:59962/0 (socket says 172.21.15.26:59962) 2024-09-16T14:49:35.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 1 -- 172.21.15.26:0/3443349833 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198593683 0 0) 0x7fe0ac0a3830 con 0x7fe0ac0a2800 2024-09-16T14:49:35.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient(hunting): my global_id is 154361 2024-09-16T14:49:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 1 -- 172.21.15.26:0/3443349833 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe098003530 con 0x7fe0ac0a2800 2024-09-16T14:49:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 1 -- 172.21.15.26:0/3443349833 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628589379 0 0) 0x7fe0ac0a3250 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient(hunting): my global_id is 164124 2024-09-16T14:49:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 1 -- 172.21.15.26:0/3443349833 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ac0a3830 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 1 -- 172.21.15.26:0/3443349833 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1405213816 0 0) 0x7fe0ac0a3830 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 1 -- 172.21.15.26:0/3443349833 >> v1:172.21.15.26:6790/0 conn(0x7fe0ac0a2800 legacy=0x7fe0ac13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 1 -- 172.21.15.26:0/3443349833 >> v1:172.21.15.189:6789/0 conn(0x7fe0ac0a0000 legacy=0x7fe0ac139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:49:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 1 monclient: found mon.noname-a 2024-09-16T14:49:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:49:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 1 -- 172.21.15.26:0/3443349833 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac09ed60 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient: _finish_auth 0 2024-09-16T14:49:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:49:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 1 -- 172.21.15.26:0/3443349833 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe0a8001620 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 5 monclient: authenticate success, global_id 164124 2024-09-16T14:49:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient: dump: 2024-09-16T14:49:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:35.055 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 1 -- 172.21.15.26:0/3443349833 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0a80018c0 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 1 -- 172.21.15.26:0/3443349833 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe0a8001c90 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0a3fff640 10 monclient: dump: 2024-09-16T14:49:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:35.057 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 4 set_mon_vals no callback set 2024-09-16T14:49:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 10 monclient: shutdown 2024-09-16T14:49:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 1 -- 172.21.15.26:0/3443349833 >> v1:172.21.15.26:6789/0 conn(0x7fe0ac0a1d10 legacy=0x7fe0ac0a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 1 -- 172.21.15.26:0/3443349833 shutdown_connections 2024-09-16T14:49:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 1 -- 172.21.15.26:0/3443349833 >> 172.21.15.26:0/3443349833 conn(0x7fe0ac0a7f30 msgr2=0x7fe0ac0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 1 -- 172.21.15.26:0/3443349833 shutdown_connections 2024-09-16T14:49:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 1 -- 172.21.15.26:0/3443349833 wait complete. 2024-09-16T14:49:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 10 monclient: build_initial_monmap 2024-09-16T14:49:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 10 monclient: monmap: 2024-09-16T14:49:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:35.058 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 1 Processor -- start 2024-09-16T14:49:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 1 -- start start 2024-09-16T14:49:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.042+0000 7fe0b3a34640 10 monclient: init 2024-09-16T14:49:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 10 monclient(hunting): picked mon.noname-c con 0x7fe0ac0a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 10 monclient(hunting): picked mon.noname-a con 0x7fe0ac0a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 10 monclient(hunting): picked mon.noname-b con 0x7fe0ac0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac0a63e0 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac09e500 con 0x7fe0ac0a2800 2024-09-16T14:49:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac133a30 con 0x7fe0ac0a0000 2024-09-16T14:49:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 10 monclient(hunting): authenticate will time out at 1399.168253s 2024-09-16T14:49:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b17d0640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe0ac0a0000 0x7fe0ac131f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49670/0 (socket says 172.21.15.26:49670) 2024-09-16T14:49:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b0fcf640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe0ac0a1d10 0x7fe0ac132610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:59974/0 (socket says 172.21.15.26:59974) 2024-09-16T14:49:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b17d0640 1 -- 172.21.15.26:0/237564582 learned_addr learned my addr 172.21.15.26:0/237564582 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433210503 0 0) 0x7fe0ac133a30 con 0x7fe0ac0a0000 2024-09-16T14:49:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): my global_id is 154364 2024-09-16T14:49:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe090003530 con 0x7fe0ac0a0000 2024-09-16T14:49:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918254604 0 0) 0x7fe0ac0a63e0 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): my global_id is 164127 2024-09-16T14:49:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ac133a30 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3390931805 0 0) 0x7fe0ac133a30 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0ac0a63e0 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe09c001840 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): dump: 2024-09-16T14:49:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:35.062 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 944718444 0 0) 0x7fe090003530 con 0x7fe0ac0a0000 2024-09-16T14:49:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0ac133a30 con 0x7fe0ac0a0000 2024-09-16T14:49:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe0a8001bd0 con 0x7fe0ac0a0000 2024-09-16T14:49:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): dump: 2024-09-16T14:49:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:35.063 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522606055 0 0) 0x7fe0ac09e500 con 0x7fe0ac0a2800 2024-09-16T14:49:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient(hunting): my global_id is 144577 2024-09-16T14:49:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe090003530 con 0x7fe0ac0a2800 2024-09-16T14:49:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2143165337 0 0) 0x7fe0ac0a63e0 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 >> v1:172.21.15.26:6790/0 conn(0x7fe0ac0a0000 legacy=0x7fe0ac131f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 >> v1:172.21.15.189:6789/0 conn(0x7fe0ac0a2800 legacy=0x7fe0ac1d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:49:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 monclient: found mon.a 2024-09-16T14:49:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac135560 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:49:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:49:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 5 monclient: authenticate success, global_id 164127 2024-09-16T14:49:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 10 monclient: _renew_subs 2024-09-16T14:49:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 1 -- 172.21.15.26:0/237564582 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ac135840 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 10 monclient: _renew_subs 2024-09-16T14:49:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 1 -- 172.21.15.26:0/237564582 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ac132d20 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe09c003620 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe09c0039b0 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a37fe640 4 set_mon_vals no callback set 2024-09-16T14:49:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 10 monclient: dump: 2024-09-16T14:49:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:35.066 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe09c003b10 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b3a34640 1 -- 172.21.15.26:0/237564582 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08c005050 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(198..198 src has 169..198) v4 ==== 10195+0+0 (unknown 798001432 0 0) 0x7fe09c044950 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.046+0000 7fe0b17d0640 10 monclient: get_auth_request con 0x7fe090036450 auth_method 0 2024-09-16T14:49:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.050+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe09c004240 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.050+0000 7fe0a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.050+0000 7fe0a1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:35.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.274+0000 7fe0b3a34640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:49:35.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.274+0000 7fe0b3a34640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:49:35.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.274+0000 7fe0b3a34640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:35.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.274+0000 7fe0b3a34640 1 -- 172.21.15.26:0/237564582 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fe08c005b10 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.274+0000 7fe0a1ffb640 1 -- 172.21.15.26:0/237564582 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fe09c012980 con 0x7fe0ac0a1d10 2024-09-16T14:49:35.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.274+0000 7fe0a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:49:35.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.274+0000 7fe0a1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:35.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.274+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.274+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=12,kv=9,logm=54} crc {auth=4016576241,config=692619927,health=2705715716,kv=1153156630,logm=1731233482}) 2024-09-16T14:49:35.281 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.274+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=10,logm=55} crc {auth=103676594,config=692619927,health=724060797,kv=3604373077,logm=2703649648}) 2024-09-16T14:49:35.281 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.274+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.281 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.274+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=12,kv=9,logm=54} crc {auth=4016576241,config=692619927,health=2705715716,kv=1153156630,logm=1731233482}) 2024-09-16T14:49:35.281 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.274+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=12,kv=10,logm=55} crc {auth=1659977820,config=692619927,health=2705715716,kv=3524478122,logm=4155589657}) 2024-09-16T14:49:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.278+0000 7fe0b3a34640 1 -- 172.21.15.26:0/237564582 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe090036450 legacy=0x7fe090038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.278+0000 7fe0b3a34640 10 monclient: shutdown 2024-09-16T14:49:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.278+0000 7fe0b3a34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.278+0000 7fe0b3a34640 1 -- 172.21.15.26:0/237564582 >> v1:172.21.15.26:6789/0 conn(0x7fe0ac0a1d10 legacy=0x7fe0ac132610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.278+0000 7fe0b3a34640 1 -- 172.21.15.26:0/237564582 shutdown_connections 2024-09-16T14:49:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.278+0000 7fe0b3a34640 1 -- 172.21.15.26:0/237564582 >> 172.21.15.26:0/237564582 conn(0x7fe0ac0a7f30 msgr2=0x7fe0ac06b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.278+0000 7fe0b3a34640 1 -- 172.21.15.26:0/237564582 shutdown_connections 2024-09-16T14:49:35.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:35.278+0000 7fe0b3a34640 1 -- 172.21.15.26:0/237564582 wait complete. 2024-09-16T14:49:35.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.278+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.278+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3536245481}) 2024-09-16T14:49:35.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.278+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2143449651}) 2024-09-16T14:49:35.284 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.278+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.284 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.278+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3536245481}) 2024-09-16T14:49:35.284 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.278+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3715233159}) 2024-09-16T14:49:35.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.282+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.282+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4094964928}) 2024-09-16T14:49:35.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.282+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2858785525}) 2024-09-16T14:49:35.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.282+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.282+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4094964928}) 2024-09-16T14:49:35.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.282+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4208678300}) 2024-09-16T14:49:35.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.282+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.282+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1907066631}) 2024-09-16T14:49:35.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.282+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1346819962}) 2024-09-16T14:49:35.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.282+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.282+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1907066631}) 2024-09-16T14:49:35.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.282+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1769580341}) 2024-09-16T14:49:35.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.286+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.286+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404318974}) 2024-09-16T14:49:35.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.286+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3929881720}) 2024-09-16T14:49:35.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.286+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.286+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404318974}) 2024-09-16T14:49:35.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.286+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2995584334}) 2024-09-16T14:49:35.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.290+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.290+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=816230249}) 2024-09-16T14:49:35.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.290+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4004219673}) 2024-09-16T14:49:35.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.290+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.290+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=816230249}) 2024-09-16T14:49:35.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.290+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2276058994}) 2024-09-16T14:49:35.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.290+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.290+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2106778760}) 2024-09-16T14:49:35.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.290+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3720665210}) 2024-09-16T14:49:35.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.290+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.296 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.290+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2106778760}) 2024-09-16T14:49:35.296 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.290+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=42663254}) 2024-09-16T14:49:35.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.294+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.294+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1030598121}) 2024-09-16T14:49:35.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.294+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4164626184}) 2024-09-16T14:49:35.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.294+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.294+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1030598121}) 2024-09-16T14:49:35.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.294+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2874228621}) 2024-09-16T14:49:35.299 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:49:35.299 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.294+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.294+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=991817831}) 2024-09-16T14:49:35.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.294+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1057426290}) 2024-09-16T14:49:35.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.294+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.294+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=991817831}) 2024-09-16T14:49:35.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.294+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978830003}) 2024-09-16T14:49:35.302 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.298+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.302 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.298+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1492636824}) 2024-09-16T14:49:35.302 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.298+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687946116}) 2024-09-16T14:49:35.302 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.298+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.302 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.298+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1492636824}) 2024-09-16T14:49:35.302 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.298+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=734382883}) 2024-09-16T14:49:35.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.298+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.298+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4243160310}) 2024-09-16T14:49:35.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.298+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=625232339}) 2024-09-16T14:49:35.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.298+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.298+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4243160310}) 2024-09-16T14:49:35.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.298+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=933785550}) 2024-09-16T14:49:35.306 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.302+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.306 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.302+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1747058206}) 2024-09-16T14:49:35.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.302+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3402179533}) 2024-09-16T14:49:35.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.302+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.302+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1747058206}) 2024-09-16T14:49:35.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.302+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2992482705}) 2024-09-16T14:49:35.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.306+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.306+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1617582405}) 2024-09-16T14:49:35.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.306+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230491398}) 2024-09-16T14:49:35.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.306+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.306+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1617582405}) 2024-09-16T14:49:35.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.306+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524191590}) 2024-09-16T14:49:35.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.306+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.306+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4291893145}) 2024-09-16T14:49:35.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.306+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=754309113}) 2024-09-16T14:49:35.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.306+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.306+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4291893145}) 2024-09-16T14:49:35.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.306+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=845380505}) 2024-09-16T14:49:35.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.310+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.310+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4142516634}) 2024-09-16T14:49:35.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.310+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215820052}) 2024-09-16T14:49:35.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.310+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.310+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4142516634}) 2024-09-16T14:49:35.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.310+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3269195846}) 2024-09-16T14:49:35.315 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.310+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.315 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.310+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2567283325}) 2024-09-16T14:49:35.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.310+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640182202}) 2024-09-16T14:49:35.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.310+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.310+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2567283325}) 2024-09-16T14:49:35.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.310+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655772458}) 2024-09-16T14:49:35.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.314+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.314+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=930094310}) 2024-09-16T14:49:35.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.314+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1424621952}) 2024-09-16T14:49:35.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.314+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.314+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=930094310}) 2024-09-16T14:49:35.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.314+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1375893811}) 2024-09-16T14:49:35.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.314+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.314+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=412642474}) 2024-09-16T14:49:35.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.314+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=126234807}) 2024-09-16T14:49:35.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.314+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.314+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=412642474}) 2024-09-16T14:49:35.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.314+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3313555099}) 2024-09-16T14:49:35.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.318+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.318+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2519732790}) 2024-09-16T14:49:35.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.318+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2768538269}) 2024-09-16T14:49:35.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.318+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.318+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2519732790}) 2024-09-16T14:49:35.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.318+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3266731660}) 2024-09-16T14:49:35.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.318+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.318+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2823465279}) 2024-09-16T14:49:35.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.318+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1126065929}) 2024-09-16T14:49:35.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.318+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.322+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2823465279}) 2024-09-16T14:49:35.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.322+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3679936007}) 2024-09-16T14:49:35.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.322+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.322+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=810065995}) 2024-09-16T14:49:35.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.322+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735695000}) 2024-09-16T14:49:35.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.322+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.322+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=810065995}) 2024-09-16T14:49:35.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.322+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=705634191}) 2024-09-16T14:49:35.328 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.326+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.328 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.326+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2978119400}) 2024-09-16T14:49:35.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.326+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312804031}) 2024-09-16T14:49:35.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.326+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.326+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2978119400}) 2024-09-16T14:49:35.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.326+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4152916593}) 2024-09-16T14:49:35.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.326+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.326+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2979751169}) 2024-09-16T14:49:35.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.326+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876472202}) 2024-09-16T14:49:35.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.326+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.326+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2979751169}) 2024-09-16T14:49:35.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.326+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1597926942}) 2024-09-16T14:49:35.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.330+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.330+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3493821711}) 2024-09-16T14:49:35.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.330+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2204625555}) 2024-09-16T14:49:35.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.330+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.330+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3493821711}) 2024-09-16T14:49:35.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.330+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3423399567}) 2024-09-16T14:49:35.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.330+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.330+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=483315604}) 2024-09-16T14:49:35.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.330+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3243332979}) 2024-09-16T14:49:35.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.330+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.330+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=483315604}) 2024-09-16T14:49:35.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.330+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=361506721}) 2024-09-16T14:49:35.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.334+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.334+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=74385298}) 2024-09-16T14:49:35.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.334+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678453705}) 2024-09-16T14:49:35.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.334+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.334+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=74385298}) 2024-09-16T14:49:35.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.334+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3717150449}) 2024-09-16T14:49:35.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.334+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.334+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1379004045}) 2024-09-16T14:49:35.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.334+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2611126673}) 2024-09-16T14:49:35.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.334+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.334+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1379004045}) 2024-09-16T14:49:35.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.334+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742372764}) 2024-09-16T14:49:35.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.338+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.338+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3699534452}) 2024-09-16T14:49:35.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.338+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251451485}) 2024-09-16T14:49:35.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.338+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.338+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3699534452}) 2024-09-16T14:49:35.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.338+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3525490162}) 2024-09-16T14:49:35.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.342+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.342+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=192779689}) 2024-09-16T14:49:35.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.342+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1741902324}) 2024-09-16T14:49:35.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.342+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.342+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=192779689}) 2024-09-16T14:49:35.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.342+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3131662758}) 2024-09-16T14:49:35.348 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.342+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.348 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.342+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3678294190}) 2024-09-16T14:49:35.348 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.342+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3445323152}) 2024-09-16T14:49:35.348 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.342+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.348 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.342+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3678294190}) 2024-09-16T14:49:35.348 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.342+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1121441295}) 2024-09-16T14:49:35.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.346+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.346+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3153402562}) 2024-09-16T14:49:35.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.346+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2081706982}) 2024-09-16T14:49:35.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.346+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.351 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.346+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3153402562}) 2024-09-16T14:49:35.351 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.346+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4021703777}) 2024-09-16T14:49:35.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.346+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.346+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=38159647}) 2024-09-16T14:49:35.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.346+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1505923957}) 2024-09-16T14:49:35.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.346+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.346+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=38159647}) 2024-09-16T14:49:35.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.346+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2594505292}) 2024-09-16T14:49:35.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.350+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.350+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2190125416}) 2024-09-16T14:49:35.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.350+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207542258}) 2024-09-16T14:49:35.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.350+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.350+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2190125416}) 2024-09-16T14:49:35.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.350+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1460273746}) 2024-09-16T14:49:35.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.350+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.350+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=998274088}) 2024-09-16T14:49:35.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.350+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1521614735}) 2024-09-16T14:49:35.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.350+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.350+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=998274088}) 2024-09-16T14:49:35.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.350+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2313810112}) 2024-09-16T14:49:35.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.354+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.354+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=31,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=22} crc {logm=1579734770,mdsmap=1531151339,mgr=2539502010,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2016625123,mon_config_key=1518680965,monmap=970648238,osd_metadata=2466911662,osd_pg_creating=1191323021,osd_snap=3794734670,osdmap=1121954591}) 2024-09-16T14:49:35.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.354+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=6,osd_snap=2,osdmap=20} crc {logm=3624508338,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=1208360744,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_snap=3794734670,osdmap=4044884288}) 2024-09-16T14:49:35.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.354+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.354+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=31,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=22} crc {logm=1579734770,mdsmap=1531151339,mgr=2539502010,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2016625123,mon_config_key=1518680965,monmap=970648238,osd_metadata=2466911662,osd_pg_creating=1191323021,osd_snap=3794734670,osdmap=1121954591}) 2024-09-16T14:49:35.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.354+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=30,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=21} crc {logm=968139744,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2139970781,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1191323021,osd_snap=3794734670,osdmap=1059400376}) 2024-09-16T14:49:35.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.366+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.366+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=37} crc {osdmap=748137707}) 2024-09-16T14:49:35.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.366+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=37} crc {osdmap=2208557144}) 2024-09-16T14:49:35.372 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.366+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:49:35.372 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.366+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=37} crc {osdmap=748137707}) 2024-09-16T14:49:35.372 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:49:35.366+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=38} crc {osdmap=992193930}) 2024-09-16T14:49:35.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.596+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}]=0 added bucket h16 type host to location {rack=r1} v199) v1 ==== 168+0+0 (unknown 3438694268 0 0) 0x7f1ad8012b00 con 0x7f1ae809c550 2024-09-16T14:49:35.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.596+0000 7f1adeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:35.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.596+0000 7f1adeffd640 10 monclient: _finish_command 2 = system:0 added bucket h16 type host to location {rack=r1} 2024-09-16T14:49:35.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.696+0000 7f1af05e4640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:35.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.696+0000 7f1af05e4640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:35.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.696+0000 7f1af05e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:35.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.696+0000 7f1af05e4640 1 -- 172.21.15.189:0/1680309050 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f1ad0000bd0 con 0x7f1ae809c550 2024-09-16T14:49:35.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.700+0000 7f1adeffd640 1 -- 172.21.15.189:0/1680309050 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h16' already exists v199) v1 ==== 147+0+0 (unknown 1745079007 0 0) 0x7f1ad8014460 con 0x7f1ae809c550 2024-09-16T14:49:35.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.700+0000 7f1adeffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:35.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.700+0000 7f1adeffd640 10 monclient: _finish_command 3 = system:0 bucket 'h16' already exists 2024-09-16T14:49:35.701 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h16' already exists 2024-09-16T14:49:35.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.700+0000 7f1af05e4640 1 -- 172.21.15.189:0/1680309050 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1acc036440 legacy=0x7f1acc038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:35.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.700+0000 7f1af05e4640 10 monclient: shutdown 2024-09-16T14:49:35.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.700+0000 7f1af05e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:35.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.700+0000 7f1af05e4640 1 -- 172.21.15.189:0/1680309050 >> v1:172.21.15.189:6789/0 conn(0x7f1ae809c550 legacy=0x7f1ae809a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:35.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.700+0000 7f1af05e4640 1 -- 172.21.15.189:0/1680309050 shutdown_connections 2024-09-16T14:49:35.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.700+0000 7f1af05e4640 1 -- 172.21.15.189:0/1680309050 >> 172.21.15.189:0/1680309050 conn(0x7f1ae80a3380 msgr2=0x7f1ae80a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:35.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.700+0000 7f1af05e4640 1 -- 172.21.15.189:0/1680309050 shutdown_connections 2024-09-16T14:49:35.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.700+0000 7f1af05e4640 1 -- 172.21.15.189:0/1680309050 wait complete. 2024-09-16T14:49:35.718 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:35.718 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h17 host rack=r1 2024-09-16T14:49:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 10 monclient: get_monmap_and_config 2024-09-16T14:49:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 10 monclient: build_initial_monmap 2024-09-16T14:49:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 10 monclient: monmap: 2024-09-16T14:49:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:35.837883+0000 2024-09-16T14:49:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:35.837883+0000 2024-09-16T14:49:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:35.838 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:35.838 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:35.838 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:35.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 1 Processor -- start 2024-09-16T14:49:35.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 1 -- start start 2024-09-16T14:49:35.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 10 monclient: init 2024-09-16T14:49:35.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:35.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:35.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 10 monclient(hunting): picked mon.noname-b con 0x7f4de40a1a90 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:35.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 10 monclient(hunting): picked mon.noname-a con 0x7f4de40a2580 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:35.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 10 monclient(hunting): picked mon.noname-c con 0x7f4de409fd40 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:35.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de409df00 con 0x7f4de40a2580 2024-09-16T14:49:35.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de409f0a0 con 0x7f4de409fd40 2024-09-16T14:49:35.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de40a3040 con 0x7f4de40a1a90 2024-09-16T14:49:35.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:35.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 10 monclient(hunting): authenticate will time out at 1357.953522s 2024-09-16T14:49:35.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4de9ea2640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4de409fd40 0x7f4de413d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60302/0 (socket says 172.21.15.189:60302) 2024-09-16T14:49:35.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4de9ea2640 1 -- 172.21.15.189:0/3699148583 learned_addr learned my addr 172.21.15.189:0/3699148583 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:35.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 1 -- 172.21.15.189:0/3699148583 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800588999 0 0) 0x7f4de409f0a0 con 0x7f4de409fd40 2024-09-16T14:49:35.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:35.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:35.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:35.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 10 monclient(hunting): my global_id is 144580 2024-09-16T14:49:35.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 1 -- 172.21.15.189:0/3699148583 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd0003530 con 0x7f4de409fd40 2024-09-16T14:49:35.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 1 -- 172.21.15.189:0/3699148583 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3091837237 0 0) 0x7f4dd0003530 con 0x7f4de409fd40 2024-09-16T14:49:35.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 1 -- 172.21.15.189:0/3699148583 >> v1:172.21.15.26:6790/0 conn(0x7f4de40a1a90 legacy=0x7f4de40a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 1 -- 172.21.15.189:0/3699148583 >> v1:172.21.15.26:6789/0 conn(0x7f4de40a2580 legacy=0x7f4de4139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 10 monclient: _finish_hunting 0 2024-09-16T14:49:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 1 monclient: found mon.noname-c 2024-09-16T14:49:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 1 -- 172.21.15.189:0/3699148583 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de409eae0 con 0x7f4de409fd40 2024-09-16T14:49:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 10 monclient: _finish_auth 0 2024-09-16T14:49:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 10 monclient: _check_auth_tickets 2024-09-16T14:49:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 1 -- 172.21.15.189:0/3699148583 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4de0001620 con 0x7f4de409fd40 2024-09-16T14:49:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 5 monclient: authenticate success, global_id 144580 2024-09-16T14:49:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 10 monclient: dump: 2024-09-16T14:49:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:35.844 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 1 -- 172.21.15.189:0/3699148583 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4de00018d0 con 0x7f4de409fd40 2024-09-16T14:49:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 1 -- 172.21.15.189:0/3699148583 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4de0001c20 con 0x7f4de409fd40 2024-09-16T14:49:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4ddbfff640 10 monclient: dump: 2024-09-16T14:49:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:35.846 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 4 set_mon_vals no callback set 2024-09-16T14:49:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 10 monclient: shutdown 2024-09-16T14:49:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 1 -- 172.21.15.189:0/3699148583 >> v1:172.21.15.189:6789/0 conn(0x7f4de409fd40 legacy=0x7f4de413d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 1 -- 172.21.15.189:0/3699148583 shutdown_connections 2024-09-16T14:49:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 1 -- 172.21.15.189:0/3699148583 >> 172.21.15.189:0/3699148583 conn(0x7f4de40a7a70 msgr2=0x7f4de40a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 1 -- 172.21.15.189:0/3699148583 shutdown_connections 2024-09-16T14:49:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 1 -- 172.21.15.189:0/3699148583 wait complete. 2024-09-16T14:49:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 10 monclient: build_initial_monmap 2024-09-16T14:49:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.836+0000 7f4deb905640 10 monclient: monmap: 2024-09-16T14:49:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:35.847 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 1 Processor -- start 2024-09-16T14:49:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 1 -- start start 2024-09-16T14:49:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 10 monclient: init 2024-09-16T14:49:35.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:35.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 10 monclient(hunting): picked mon.noname-a con 0x7f4de409fd40 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 10 monclient(hunting): picked mon.noname-b con 0x7f4de40a1a90 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 10 monclient(hunting): picked mon.noname-c con 0x7f4de40a2580 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de4120dd0 con 0x7f4de409fd40 2024-09-16T14:49:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de409e280 con 0x7f4de40a1a90 2024-09-16T14:49:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de40a6140 con 0x7f4de40a2580 2024-09-16T14:49:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 10 monclient(hunting): authenticate will time out at 1357.956666s 2024-09-16T14:49:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4de8ea0640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4de40a1a90 0x7f4de41ce570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60314/0 (socket says 172.21.15.189:60314) 2024-09-16T14:49:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4de8ea0640 1 -- 172.21.15.189:0/3463203289 learned_addr learned my addr 172.21.15.189:0/3463203289 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 1 -- 172.21.15.189:0/3463203289 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604943669 0 0) 0x7f4de409e280 con 0x7f4de40a1a90 2024-09-16T14:49:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 10 monclient(hunting): my global_id is 144583 2024-09-16T14:49:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 1 -- 172.21.15.189:0/3463203289 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dc8003530 con 0x7f4de40a1a90 2024-09-16T14:49:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 1 -- 172.21.15.189:0/3463203289 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1414495294 0 0) 0x7f4dc8003530 con 0x7f4de40a1a90 2024-09-16T14:49:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 1 -- 172.21.15.189:0/3463203289 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4de409e280 con 0x7f4de40a1a90 2024-09-16T14:49:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 1 -- 172.21.15.189:0/3463203289 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4dd40018b0 con 0x7f4de40a1a90 2024-09-16T14:49:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 10 monclient(hunting): dump: 2024-09-16T14:49:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:35.853 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 1 -- 172.21.15.189:0/3463203289 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2949147320 0 0) 0x7f4de409e280 con 0x7f4de40a1a90 2024-09-16T14:49:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 1 -- 172.21.15.189:0/3463203289 >> v1:172.21.15.26:6790/0 conn(0x7f4de40a2580 legacy=0x7f4de41d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 1 -- 172.21.15.189:0/3463203289 >> v1:172.21.15.26:6789/0 conn(0x7f4de409fd40 legacy=0x7f4de4133cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:49:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 1 monclient: found mon.b 2024-09-16T14:49:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 1 -- 172.21.15.189:0/3463203289 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de41c98e0 con 0x7f4de40a1a90 2024-09-16T14:49:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 10 monclient: _finish_auth 0 2024-09-16T14:49:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:49:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 5 monclient: authenticate success, global_id 144583 2024-09-16T14:49:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 10 monclient: _renew_subs 2024-09-16T14:49:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 1 -- 172.21.15.189:0/3463203289 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de41216d0 con 0x7f4de40a1a90 2024-09-16T14:49:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 10 monclient: _renew_subs 2024-09-16T14:49:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 1 -- 172.21.15.189:0/3463203289 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de41c9230 con 0x7f4de40a1a90 2024-09-16T14:49:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 1 -- 172.21.15.189:0/3463203289 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4dd4001c80 con 0x7f4de40a1a90 2024-09-16T14:49:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 1 -- 172.21.15.189:0/3463203289 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4dd4003a70 con 0x7f4de40a1a90 2024-09-16T14:49:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4ddb7fe640 4 set_mon_vals no callback set 2024-09-16T14:49:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 10 monclient: dump: 2024-09-16T14:49:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:35.856 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:35.856 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:35.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:35.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 1 -- 172.21.15.189:0/3463203289 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4dd4003bd0 con 0x7f4de40a1a90 2024-09-16T14:49:35.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:35.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:35.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4deb905640 1 -- 172.21.15.189:0/3463203289 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc4005050 con 0x7f4de40a1a90 2024-09-16T14:49:35.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4dd9ffb640 1 -- 172.21.15.189:0/3463203289 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(199..199 src has 169..199) v4 ==== 10664+0+0 (unknown 1988835959 0 0) 0x7f4dd4044e10 con 0x7f4de40a1a90 2024-09-16T14:49:35.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.840+0000 7f4de96a1640 10 monclient: get_auth_request con 0x7f4dc80360b0 auth_method 0 2024-09-16T14:49:35.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.844+0000 7f4dd9ffb640 1 -- 172.21.15.189:0/3463203289 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4dd4014820 con 0x7f4de40a1a90 2024-09-16T14:49:35.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.844+0000 7f4dd9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:35.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:35.844+0000 7f4dd9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:36.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.004+0000 7f4deb905640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:36.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.004+0000 7f4deb905640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:36.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.004+0000 7f4deb905640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:36.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.004+0000 7f4deb905640 1 -- 172.21.15.189:0/3463203289 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f4dc4005300 con 0x7f4de40a1a90 2024-09-16T14:49:36.300 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:49:36.301 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:49:36.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e877ee640 10 monclient: get_monmap_and_config 2024-09-16T14:49:36.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e877ee640 10 monclient: build_initial_monmap 2024-09-16T14:49:36.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e877ee640 10 monclient: monmap: 2024-09-16T14:49:36.431 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:36.431 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.431 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:36.431383+0000 2024-09-16T14:49:36.431 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:36.431383+0000 2024-09-16T14:49:36.431 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:36.431 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.432 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:36.432 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:36.432 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:36.432 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e877ee640 1 Processor -- start 2024-09-16T14:49:36.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e877ee640 1 -- start start 2024-09-16T14:49:36.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e877ee640 10 monclient: init 2024-09-16T14:49:36.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e877ee640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:36.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e877ee640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:49:36.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e877ee640 10 monclient(hunting): picked mon.noname-a con 0x7f3e800a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:36.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e877ee640 10 monclient(hunting): picked mon.noname-b con 0x7f3e800a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e877ee640 10 monclient(hunting): picked mon.noname-c con 0x7f3e800a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e877ee640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e800a3250 con 0x7f3e800a1d10 2024-09-16T14:49:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e877ee640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8009f600 con 0x7f3e800a2800 2024-09-16T14:49:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e877ee640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e800a3830 con 0x7f3e800a0000 2024-09-16T14:49:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e877ee640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e877ee640 10 monclient(hunting): authenticate will time out at 1400.551455s 2024-09-16T14:49:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e84d89640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3e800a0000 0x7f3e80139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49684/0 (socket says 172.21.15.26:49684) 2024-09-16T14:49:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e84d89640 1 -- 172.21.15.26:0/3511140194 learned_addr learned my addr 172.21.15.26:0/3511140194 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e8558a640 1 --1- 172.21.15.26:0/3511140194 >> v1:172.21.15.26:6789/0 conn(0x7f3e800a1d10 0x7f3e800a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:59986/0 (socket says 172.21.15.26:59986) 2024-09-16T14:49:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e77fff640 1 -- 172.21.15.26:0/3511140194 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903497280 0 0) 0x7f3e800a3830 con 0x7f3e800a0000 2024-09-16T14:49:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e77fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e77fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e77fff640 10 monclient(hunting): my global_id is 154373 2024-09-16T14:49:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e77fff640 1 -- 172.21.15.26:0/3511140194 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e68003530 con 0x7f3e800a0000 2024-09-16T14:49:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e77fff640 1 -- 172.21.15.26:0/3511140194 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264973013 0 0) 0x7f3e800a3250 con 0x7f3e800a1d10 2024-09-16T14:49:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e77fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e77fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.426+0000 7f3e77fff640 10 monclient(hunting): my global_id is 164136 2024-09-16T14:49:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 1 -- 172.21.15.26:0/3511140194 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e800a3830 con 0x7f3e800a1d10 2024-09-16T14:49:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 1 -- 172.21.15.26:0/3511140194 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4067321033 0 0) 0x7f3e68003530 con 0x7f3e800a0000 2024-09-16T14:49:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 1 -- 172.21.15.26:0/3511140194 >> v1:172.21.15.189:6789/0 conn(0x7f3e800a2800 legacy=0x7f3e8013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 1 -- 172.21.15.26:0/3511140194 >> v1:172.21.15.26:6789/0 conn(0x7f3e800a1d10 legacy=0x7f3e800a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 10 monclient: _finish_hunting 0 2024-09-16T14:49:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 1 monclient: found mon.noname-b 2024-09-16T14:49:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:49:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 1 -- 172.21.15.26:0/3511140194 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8009ed60 con 0x7f3e800a0000 2024-09-16T14:49:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 10 monclient: _finish_auth 0 2024-09-16T14:49:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 10 monclient: _check_auth_tickets 2024-09-16T14:49:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 1 -- 172.21.15.26:0/3511140194 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e7c001620 con 0x7f3e800a0000 2024-09-16T14:49:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 10 monclient: dump: 2024-09-16T14:49:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.442 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.443 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 5 monclient: authenticate success, global_id 154373 2024-09-16T14:49:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 1 -- 172.21.15.26:0/3511140194 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e7c0018c0 con 0x7f3e800a0000 2024-09-16T14:49:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 1 -- 172.21.15.26:0/3511140194 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e7c001c50 con 0x7f3e800a0000 2024-09-16T14:49:36.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:36.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 4 set_mon_vals no callback set 2024-09-16T14:49:36.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:36.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e77fff640 10 monclient: dump: 2024-09-16T14:49:36.444 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.444 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.444 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.445 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.445 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.445 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.445 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.445 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.445 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.445 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 10 monclient: shutdown 2024-09-16T14:49:36.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:36.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 1 -- 172.21.15.26:0/3511140194 >> v1:172.21.15.26:6790/0 conn(0x7f3e800a0000 legacy=0x7f3e80139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 1 -- 172.21.15.26:0/3511140194 shutdown_connections 2024-09-16T14:49:36.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 1 -- 172.21.15.26:0/3511140194 >> 172.21.15.26:0/3511140194 conn(0x7f3e800a7f30 msgr2=0x7f3e800a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:36.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 1 -- 172.21.15.26:0/3511140194 shutdown_connections 2024-09-16T14:49:36.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 1 -- 172.21.15.26:0/3511140194 wait complete. 2024-09-16T14:49:36.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 10 monclient: build_initial_monmap 2024-09-16T14:49:36.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 10 monclient: monmap: 2024-09-16T14:49:36.446 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:36.446 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:36.446 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:36.446 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:36.446 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:36.446 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:36.446 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:36.446 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:36.446 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:36.446 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 1 Processor -- start 2024-09-16T14:49:36.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 1 -- start start 2024-09-16T14:49:36.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 10 monclient: init 2024-09-16T14:49:36.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:36.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:49:36.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 10 monclient(hunting): picked mon.noname-c con 0x7f3e800a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:36.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 10 monclient(hunting): picked mon.noname-b con 0x7f3e800a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:36.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 10 monclient(hunting): picked mon.noname-a con 0x7f3e800a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:36.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e80132d40 con 0x7f3e800a2800 2024-09-16T14:49:36.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8009e500 con 0x7f3e800a1d10 2024-09-16T14:49:36.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e800a6920 con 0x7f3e800a0000 2024-09-16T14:49:36.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:36.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 10 monclient(hunting): authenticate will time out at 1400.554702s 2024-09-16T14:49:36.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e8558a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3e800a0000 0x7f3e80131f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49688/0 (socket says 172.21.15.26:49688) 2024-09-16T14:49:36.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e85d8b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3e800a2800 0x7f3e801d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60000/0 (socket says 172.21.15.26:60000) 2024-09-16T14:49:36.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e8558a640 1 -- 172.21.15.26:0/483659794 learned_addr learned my addr 172.21.15.26:0/483659794 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:36.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200710189 0 0) 0x7f3e80132d40 con 0x7f3e800a2800 2024-09-16T14:49:36.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): my global_id is 164139 2024-09-16T14:49:36.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e6c003530 con 0x7f3e800a2800 2024-09-16T14:49:36.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036072608 0 0) 0x7f3e800a6920 con 0x7f3e800a0000 2024-09-16T14:49:36.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): my global_id is 154376 2024-09-16T14:49:36.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e80132d40 con 0x7f3e800a0000 2024-09-16T14:49:36.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1065982567 0 0) 0x7f3e6c003530 con 0x7f3e800a2800 2024-09-16T14:49:36.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e800a6920 con 0x7f3e800a2800 2024-09-16T14:49:36.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e70001840 con 0x7f3e800a2800 2024-09-16T14:49:36.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:36.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): dump: 2024-09-16T14:49:36.450 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.450 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.450 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.450 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.450 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.450 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.450 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.450 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.451 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.451 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:36.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1547450292 0 0) 0x7f3e80132d40 con 0x7f3e800a0000 2024-09-16T14:49:36.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e6c003530 con 0x7f3e800a0000 2024-09-16T14:49:36.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21361760 0 0) 0x7f3e8009e500 con 0x7f3e800a1d10 2024-09-16T14:49:36.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): my global_id is 144589 2024-09-16T14:49:36.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e80132d40 con 0x7f3e800a1d10 2024-09-16T14:49:36.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e78001d80 con 0x7f3e800a0000 2024-09-16T14:49:36.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:36.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient(hunting): dump: 2024-09-16T14:49:36.452 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.452 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.452 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.452 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.452 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.452 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.452 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.452 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.452 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.452 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1133061673 0 0) 0x7f3e800a6920 con 0x7f3e800a2800 2024-09-16T14:49:36.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 >> v1:172.21.15.26:6790/0 conn(0x7f3e800a0000 legacy=0x7f3e80131f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 >> v1:172.21.15.189:6789/0 conn(0x7f3e800a1d10 legacy=0x7f3e80132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:49:36.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 monclient: found mon.a 2024-09-16T14:49:36.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:36.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:36.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e800a63f0 con 0x7f3e800a2800 2024-09-16T14:49:36.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient: _finish_auth 0 2024-09-16T14:49:36.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:49:36.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:36.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 5 monclient: authenticate success, global_id 164139 2024-09-16T14:49:36.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 10 monclient: _renew_subs 2024-09-16T14:49:36.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:36.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 1 -- 172.21.15.26:0/483659794 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e80136070 con 0x7f3e800a2800 2024-09-16T14:49:36.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 10 monclient: _renew_subs 2024-09-16T14:49:36.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:36.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e877ee640 1 -- 172.21.15.26:0/483659794 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e80135d90 con 0x7f3e800a2800 2024-09-16T14:49:36.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.430+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e70001d70 con 0x7f3e800a2800 2024-09-16T14:49:36.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.434+0000 7f3e75ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:36.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.434+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e700037c0 con 0x7f3e800a2800 2024-09-16T14:49:36.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.434+0000 7f3e75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.434+0000 7f3e75ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:36.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.434+0000 7f3e777fe640 4 set_mon_vals no callback set 2024-09-16T14:49:36.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.434+0000 7f3e75ffb640 10 monclient: dump: 2024-09-16T14:49:36.455 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.455 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.455 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.455 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.455 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.455 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.455 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.455 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.455 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.456 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.434+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3e700039a0 con 0x7f3e800a2800 2024-09-16T14:49:36.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.434+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(199..199 src has 169..199) v4 ==== 10664+0+0 (unknown 1988835959 0 0) 0x7f3e70044c70 con 0x7f3e800a2800 2024-09-16T14:49:36.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.434+0000 7f3e877ee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:36.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.434+0000 7f3e877ee640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:36.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.434+0000 7f3e877ee640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:36.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.434+0000 7f3e877ee640 1 -- 172.21.15.26:0/483659794 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e60005050 con 0x7f3e800a2800 2024-09-16T14:49:36.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.434+0000 7f3e8558a640 10 monclient: get_auth_request con 0x7f3e6c036390 auth_method 0 2024-09-16T14:49:36.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.438+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3e70014660 con 0x7f3e800a2800 2024-09-16T14:49:36.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.438+0000 7f3e75ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:36.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.438+0000 7f3e75ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:36.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.616+0000 7f4dd9ffb640 1 -- 172.21.15.189:0/3463203289 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}]=0 added bucket h17 type host to location {rack=r1} v200) v1 ==== 168+0+0 (unknown 2925034033 0 0) 0x7f4dd4012c90 con 0x7f4de40a1a90 2024-09-16T14:49:36.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.616+0000 7f4dd9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:36.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.616+0000 7f4dd9ffb640 10 monclient: _finish_command 2 = system:0 added bucket h17 type host to location {rack=r1} 2024-09-16T14:49:36.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.662+0000 7f3e877ee640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:49:36.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.662+0000 7f3e877ee640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:49:36.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.662+0000 7f3e877ee640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:36.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.662+0000 7f3e877ee640 1 -- 172.21.15.26:0/483659794 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3e60005880 con 0x7f3e800a2800 2024-09-16T14:49:36.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.662+0000 7f3e75ffb640 1 -- 172.21.15.26:0/483659794 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2537896525) 0x7f3e70012ad0 con 0x7f3e800a2800 2024-09-16T14:49:36.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.662+0000 7f3e75ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:49:36.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.662+0000 7f3e75ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:36.667 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:49:36.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.666+0000 7f3e877ee640 1 -- 172.21.15.26:0/483659794 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3e6c036390 legacy=0x7f3e6c038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.666+0000 7f3e877ee640 10 monclient: shutdown 2024-09-16T14:49:36.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.666+0000 7f3e877ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:36.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.666+0000 7f3e877ee640 1 -- 172.21.15.26:0/483659794 >> v1:172.21.15.26:6789/0 conn(0x7f3e800a2800 legacy=0x7f3e801d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.666+0000 7f3e877ee640 1 -- 172.21.15.26:0/483659794 shutdown_connections 2024-09-16T14:49:36.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.666+0000 7f3e877ee640 1 -- 172.21.15.26:0/483659794 >> 172.21.15.26:0/483659794 conn(0x7f3e800a7f30 msgr2=0x7f3e8006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:36.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.666+0000 7f3e877ee640 1 -- 172.21.15.26:0/483659794 shutdown_connections 2024-09-16T14:49:36.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.666+0000 7f3e877ee640 1 -- 172.21.15.26:0/483659794 wait complete. 2024-09-16T14:49:36.685 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:49:36.686 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:49:36.686 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:49:36.686 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:49:36.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.716+0000 7f4deb905640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:36.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.716+0000 7f4deb905640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:36.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.716+0000 7f4deb905640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:36.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.716+0000 7f4deb905640 1 -- 172.21.15.189:0/3463203289 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f4dc4000bd0 con 0x7f4de40a1a90 2024-09-16T14:49:36.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.716+0000 7f4dd9ffb640 1 -- 172.21.15.189:0/3463203289 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h17' already exists v200) v1 ==== 147+0+0 (unknown 3171730605 0 0) 0x7f4dd4015150 con 0x7f4de40a1a90 2024-09-16T14:49:36.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.716+0000 7f4dd9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:36.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.716+0000 7f4dd9ffb640 10 monclient: _finish_command 3 = system:0 bucket 'h17' already exists 2024-09-16T14:49:36.720 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h17' already exists 2024-09-16T14:49:36.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.720+0000 7f4deb905640 1 -- 172.21.15.189:0/3463203289 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4dc80360b0 legacy=0x7f4dc8038570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.720+0000 7f4deb905640 10 monclient: shutdown 2024-09-16T14:49:36.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.720+0000 7f4deb905640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:36.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.720+0000 7f4deb905640 1 -- 172.21.15.189:0/3463203289 >> v1:172.21.15.189:6789/0 conn(0x7f4de40a1a90 legacy=0x7f4de41ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.720+0000 7f4deb905640 1 -- 172.21.15.189:0/3463203289 shutdown_connections 2024-09-16T14:49:36.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.720+0000 7f4deb905640 1 -- 172.21.15.189:0/3463203289 >> 172.21.15.189:0/3463203289 conn(0x7f4de40a7a70 msgr2=0x7f4de406b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:36.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.720+0000 7f4deb905640 1 -- 172.21.15.189:0/3463203289 shutdown_connections 2024-09-16T14:49:36.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.720+0000 7f4deb905640 1 -- 172.21.15.189:0/3463203289 wait complete. 2024-09-16T14:49:36.737 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:36.737 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h18 host rack=r1 2024-09-16T14:49:36.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.806+0000 7f35abd2a640 10 monclient: get_monmap_and_config 2024-09-16T14:49:36.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.806+0000 7f35abd2a640 10 monclient: build_initial_monmap 2024-09-16T14:49:36.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.806+0000 7f35abd2a640 10 monclient: monmap: 2024-09-16T14:49:36.811 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:36.811 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.811 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:36.811786+0000 2024-09-16T14:49:36.811 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:36.811786+0000 2024-09-16T14:49:36.811 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:36.811 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.811 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:36.811 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:36.812 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:36.812 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.806+0000 7f35abd2a640 1 Processor -- start 2024-09-16T14:49:36.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.806+0000 7f35abd2a640 1 -- start start 2024-09-16T14:49:36.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.806+0000 7f35abd2a640 10 monclient: init 2024-09-16T14:49:36.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:36.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:36.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient(hunting): picked mon.noname-c con 0x7f35a40a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:36.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient(hunting): picked mon.noname-a con 0x7f35a40a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:36.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient(hunting): picked mon.noname-b con 0x7f35a40a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:36.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a40a3250 con 0x7f35a40a0000 2024-09-16T14:49:36.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a409f600 con 0x7f35a40a1d10 2024-09-16T14:49:36.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a40a3830 con 0x7f35a40a2800 2024-09-16T14:49:36.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:36.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient(hunting): authenticate will time out at 1400.932647s 2024-09-16T14:49:36.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a92c5640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f35a40a0000 0x7f35a4139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60008/0 (socket says 172.21.15.26:60008) 2024-09-16T14:49:36.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a92c5640 1 -- 172.21.15.26:0/3398523766 learned_addr learned my addr 172.21.15.26:0/3398523766 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:36.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 1 -- 172.21.15.26:0/3398523766 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055648479 0 0) 0x7f35a40a3250 con 0x7f35a40a0000 2024-09-16T14:49:36.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient(hunting): my global_id is 164142 2024-09-16T14:49:36.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 1 -- 172.21.15.26:0/3398523766 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3590003530 con 0x7f35a40a0000 2024-09-16T14:49:36.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 1 -- 172.21.15.26:0/3398523766 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770275268 0 0) 0x7f35a40a3830 con 0x7f35a40a2800 2024-09-16T14:49:36.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient(hunting): my global_id is 154379 2024-09-16T14:49:36.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 1 -- 172.21.15.26:0/3398523766 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a40a3250 con 0x7f35a40a2800 2024-09-16T14:49:36.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 1 -- 172.21.15.26:0/3398523766 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811751143 0 0) 0x7f35a409f600 con 0x7f35a40a1d10 2024-09-16T14:49:36.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient(hunting): my global_id is 144592 2024-09-16T14:49:36.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 1 -- 172.21.15.26:0/3398523766 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a40a3830 con 0x7f35a40a1d10 2024-09-16T14:49:36.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 1 -- 172.21.15.26:0/3398523766 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 350589566 0 0) 0x7f3590003530 con 0x7f35a40a0000 2024-09-16T14:49:36.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 1 -- 172.21.15.26:0/3398523766 >> v1:172.21.15.26:6790/0 conn(0x7f35a40a2800 legacy=0x7f35a413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 1 -- 172.21.15.26:0/3398523766 >> v1:172.21.15.189:6789/0 conn(0x7f35a40a1d10 legacy=0x7f35a40a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient: _finish_hunting 0 2024-09-16T14:49:36.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 1 monclient: found mon.noname-a 2024-09-16T14:49:36.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:36.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:49:36.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 1 -- 172.21.15.26:0/3398523766 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a409ed60 con 0x7f35a40a0000 2024-09-16T14:49:36.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient: _finish_auth 0 2024-09-16T14:49:36.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient: _check_auth_tickets 2024-09-16T14:49:36.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:36.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 1 -- 172.21.15.26:0/3398523766 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3594001620 con 0x7f35a40a0000 2024-09-16T14:49:36.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 5 monclient: authenticate success, global_id 164142 2024-09-16T14:49:36.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:36.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:36.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient: dump: 2024-09-16T14:49:36.819 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.819 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.819 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.819 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.819 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.820 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.820 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.820 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.820 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.820 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:36.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 1 -- 172.21.15.26:0/3398523766 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3594001900 con 0x7f35a40a0000 2024-09-16T14:49:36.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:36.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 1 -- 172.21.15.26:0/3398523766 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3594001cd0 con 0x7f35a40a0000 2024-09-16T14:49:36.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:36.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 4 set_mon_vals no callback set 2024-09-16T14:49:36.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:36.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a8ac4640 10 monclient: dump: 2024-09-16T14:49:36.821 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.821 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.821 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.821 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.821 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.821 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.821 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.821 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.821 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.821 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient: shutdown 2024-09-16T14:49:36.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:36.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 1 -- 172.21.15.26:0/3398523766 >> v1:172.21.15.26:6789/0 conn(0x7f35a40a0000 legacy=0x7f35a4139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 1 -- 172.21.15.26:0/3398523766 shutdown_connections 2024-09-16T14:49:36.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 1 -- 172.21.15.26:0/3398523766 >> 172.21.15.26:0/3398523766 conn(0x7f35a40a7f30 msgr2=0x7f35a40a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:36.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 1 -- 172.21.15.26:0/3398523766 shutdown_connections 2024-09-16T14:49:36.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 1 -- 172.21.15.26:0/3398523766 wait complete. 2024-09-16T14:49:36.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient: build_initial_monmap 2024-09-16T14:49:36.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient: monmap: 2024-09-16T14:49:36.822 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:36.822 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:36.822 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:36.822 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:36.822 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:36.822 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:36.823 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:36.823 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:36.823 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:36.823 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 1 Processor -- start 2024-09-16T14:49:36.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 1 -- start start 2024-09-16T14:49:36.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient: init 2024-09-16T14:49:36.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:36.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:36.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient(hunting): picked mon.noname-b con 0x7f35a40a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:36.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient(hunting): picked mon.noname-a con 0x7f35a40a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:36.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient(hunting): picked mon.noname-c con 0x7f35a40a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:36.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a4131c80 con 0x7f35a40a1d10 2024-09-16T14:49:36.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a4133fe0 con 0x7f35a40a0000 2024-09-16T14:49:36.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a4130f90 con 0x7f35a40a2800 2024-09-16T14:49:36.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:36.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35abd2a640 10 monclient(hunting): authenticate will time out at 1400.935752s 2024-09-16T14:49:36.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a92c5640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f35a40a1d10 0x7f35a4130880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60018/0 (socket says 172.21.15.26:60018) 2024-09-16T14:49:36.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.810+0000 7f35a92c5640 1 -- 172.21.15.26:0/3757643867 learned_addr learned my addr 172.21.15.26:0/3757643867 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:36.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737195990 0 0) 0x7f35a4131c80 con 0x7f35a40a1d10 2024-09-16T14:49:36.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): my global_id is 164145 2024-09-16T14:49:36.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3588003530 con 0x7f35a40a1d10 2024-09-16T14:49:36.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842018671 0 0) 0x7f35a4130f90 con 0x7f35a40a2800 2024-09-16T14:49:36.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): my global_id is 154382 2024-09-16T14:49:36.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a4131c80 con 0x7f35a40a2800 2024-09-16T14:49:36.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774239586 0 0) 0x7f35a4133fe0 con 0x7f35a40a0000 2024-09-16T14:49:36.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): my global_id is 144595 2024-09-16T14:49:36.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a4130f90 con 0x7f35a40a0000 2024-09-16T14:49:36.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1539447061 0 0) 0x7f3588003530 con 0x7f35a40a1d10 2024-09-16T14:49:36.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35a4133fe0 con 0x7f35a40a1d10 2024-09-16T14:49:36.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f35940015e0 con 0x7f35a40a1d10 2024-09-16T14:49:36.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:36.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): dump: 2024-09-16T14:49:36.831 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.831 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.831 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.831 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.831 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.831 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.831 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.831 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.831 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.832 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3546981372 0 0) 0x7f35a4131c80 con 0x7f35a40a2800 2024-09-16T14:49:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3588003530 con 0x7f35a40a2800 2024-09-16T14:49:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f359c001bc0 con 0x7f35a40a2800 2024-09-16T14:49:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): dump: 2024-09-16T14:49:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.833 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.833 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.833 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.833 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.833 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2837459734 0 0) 0x7f35a4133fe0 con 0x7f35a40a1d10 2024-09-16T14:49:36.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 >> v1:172.21.15.26:6790/0 conn(0x7f35a40a2800 legacy=0x7f35a41d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 >> v1:172.21.15.189:6789/0 conn(0x7f35a40a0000 legacy=0x7f35a4130170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:49:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 monclient: found mon.a 2024-09-16T14:49:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a40a63e0 con 0x7f35a40a1d10 2024-09-16T14:49:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:49:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f35abd2a640 5 monclient: authenticate success, global_id 164145 2024-09-16T14:49:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f35abd2a640 10 monclient: _renew_subs 2024-09-16T14:49:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f35abd2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f35abd2a640 1 -- 172.21.15.26:0/3757643867 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a40a6910 con 0x7f35a40a1d10 2024-09-16T14:49:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f35abd2a640 10 monclient: _renew_subs 2024-09-16T14:49:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f35abd2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f35abd2a640 1 -- 172.21.15.26:0/3757643867 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a4132850 con 0x7f35a40a1d10 2024-09-16T14:49:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3594004af0 con 0x7f35a40a1d10 2024-09-16T14:49:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3594004e00 con 0x7f35a40a1d10 2024-09-16T14:49:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359bfff640 4 set_mon_vals no callback set 2024-09-16T14:49:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient: dump: 2024-09-16T14:49:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.836 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f35abd2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f35abd2a640 1 -- 172.21.15.26:0/3757643867 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f358c006590 con 0x7f358c002830 2024-09-16T14:49:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3594004f60 con 0x7f35a40a1d10 2024-09-16T14:49:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(200..200 src has 169..200) v4 ==== 10877+0+0 (unknown 2031785554 0 0) 0x7f3594044f60 con 0x7f35a40a1d10 2024-09-16T14:49:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302086246 0 0) 0x7f358c006590 con 0x7f358c002830 2024-09-16T14:49:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient(hunting): my global_id is 164148 2024-09-16T14:49:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a4133fe0 con 0x7f358c002830 2024-09-16T14:49:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f35aa2c7640 10 monclient: get_auth_request con 0x7f35880361e0 auth_method 0 2024-09-16T14:49:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1930718410 0 0) 0x7f35a4133fe0 con 0x7f358c002830 2024-09-16T14:49:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f358c006590 con 0x7f358c002830 2024-09-16T14:49:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f35a0001d60 con 0x7f358c002830 2024-09-16T14:49:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient: dump: 2024-09-16T14:49:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.838 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3907120047 0 0) 0x7f358c006590 con 0x7f358c002830 2024-09-16T14:49:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f358c00a0e0 con 0x7f358c002830 2024-09-16T14:49:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f35a0005950 con 0x7f358c002830 2024-09-16T14:49:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 >> v1:172.21.15.26:6789/0 conn(0x7f358c002830 legacy=0x7f358c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.814+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 >> v1:172.21.15.26:6789/0 conn(0x7f358c002830 legacy=0x7f358c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f35abd2a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:49:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f35abd2a640 1 -- 172.21.15.26:0/3757643867 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f358c009da0 con 0x7f358c0090c0 2024-09-16T14:49:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912322288 0 0) 0x7f358c009da0 con 0x7f358c0090c0 2024-09-16T14:49:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 10 monclient(hunting): my global_id is 164151 2024-09-16T14:49:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f358c006590 con 0x7f358c0090c0 2024-09-16T14:49:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1188147201 0 0) 0x7f358c006590 con 0x7f358c0090c0 2024-09-16T14:49:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f358c009da0 con 0x7f358c0090c0 2024-09-16T14:49:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f35a0003dd0 con 0x7f358c0090c0 2024-09-16T14:49:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 10 monclient: dump: 2024-09-16T14:49:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.841 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2593623904 0 0) 0x7f358c009da0 con 0x7f358c0090c0 2024-09-16T14:49:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f358c00c290 con 0x7f358c0090c0 2024-09-16T14:49:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 2086006177) 0x7f35a0005ac0 con 0x7f358c0090c0 2024-09-16T14:49:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:49:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 >> v1:172.21.15.26:6789/0 conn(0x7f358c0090c0 legacy=0x7f358c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f359a7fc640 1 -- 172.21.15.26:0/3757643867 >> v1:172.21.15.26:6789/0 conn(0x7f358c0090c0 legacy=0x7f358c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:36.843 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:49:36.843 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:49:36.843 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:49:36.843 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:49:36.843 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 66, 2024-09-16T14:49:36.843 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:49:36.843 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:49:36.843 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:49:36.843 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:49:36.843 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:36.843 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:49:36.844 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:36.844 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:49:36.844 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:49:36.844 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:36.844 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:36.844 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:36.844 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:36.844 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:36.844 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:36.844 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:36.844 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:36.844 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:36.844 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:36.845 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:36.845 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:49:36.845 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:49:36.845 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:36.845 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:36.845 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:36.845 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:36.845 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:36.845 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:36.845 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:36.845 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:36.845 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:36.845 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:36.846 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:36.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:36.846 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:49:36.846 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:49:36.846 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:49:36.846 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:49:36.846 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:49:36.846 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:49:36.846 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:36.846 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:36.846 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:49:36.846 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:49:36.846 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:49:36.847 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:49:36.847 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:49:36.847 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:49:36.847 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:49:36.847 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:36.847 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:49:36.847 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:36.847 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:36.847 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:36.847 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:36.847 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:36.847 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:36.847 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:36.848 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:36.848 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:36.848 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:36.848 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:36.848 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:49:36.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:36.848 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:49:36.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:36.848 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:49:36.848 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:49:36.848 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:36.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:36.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:36.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:36.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:49:36.849 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:36.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:36.849 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:36.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:36.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:49:36.849 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:49:36.849 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:36.849 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:36.850 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:36.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:36.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:36.850 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:49:36.850 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:49:36.850 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:36.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:36.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:36.850 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:36.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:49:36.850 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:36.850 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:36.850 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:36.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:36.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:49:36.851 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:49:36.851 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:36.851 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:36.851 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:36.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:36.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:36.851 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:49:36.851 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:49:36.851 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:36.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:36.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:36.851 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:36.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:49:36.852 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:36.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:36.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:36.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:36.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:49:36.852 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:49:36.852 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:36.852 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:36.852 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:36.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:36.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:36.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:36.853 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:49:36.853 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:49:36.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:36.853 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:36.853 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:36.853 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:49:36.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:36.853 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:36.853 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:49:36.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:36.853 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:36.853 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:36.853 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:49:36.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:36.854 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:36.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:36.854 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:49:36.854 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:49:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f35abd2a640 1 -- 172.21.15.26:0/3757643867 >> v1:172.21.15.189:6812/3196161582 conn(0x7f35880361e0 legacy=0x7f35880386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f35abd2a640 10 monclient: shutdown 2024-09-16T14:49:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f35abd2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f35abd2a640 1 -- 172.21.15.26:0/3757643867 >> v1:172.21.15.26:6789/0 conn(0x7f35a40a1d10 legacy=0x7f35a4130880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f35aa2c7640 1 -- 172.21.15.26:0/3757643867 reap_dead start 2024-09-16T14:49:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f35aa2c7640 1 -- 172.21.15.26:0/3757643867 reap_dead start 2024-09-16T14:49:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f35abd2a640 1 -- 172.21.15.26:0/3757643867 shutdown_connections 2024-09-16T14:49:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f35abd2a640 1 -- 172.21.15.26:0/3757643867 >> 172.21.15.26:0/3757643867 conn(0x7f35a40a7f30 msgr2=0x7f35a406b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f35abd2a640 1 -- 172.21.15.26:0/3757643867 shutdown_connections 2024-09-16T14:49:36.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.822+0000 7f35abd2a640 1 -- 172.21.15.26:0/3757643867 wait complete. 2024-09-16T14:49:36.856 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:49:36.856 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:49:36.856 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:49:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc4a5645640 10 monclient: get_monmap_and_config 2024-09-16T14:49:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc4a5645640 10 monclient: build_initial_monmap 2024-09-16T14:49:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc4a5645640 10 monclient: monmap: 2024-09-16T14:49:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:36.858925+0000 2024-09-16T14:49:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:36.858925+0000 2024-09-16T14:49:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:36.859 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:36.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc4a5645640 1 Processor -- start 2024-09-16T14:49:36.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc4a5645640 1 -- start start 2024-09-16T14:49:36.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc4a5645640 10 monclient: init 2024-09-16T14:49:36.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc4a5645640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:36.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc4a5645640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:36.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc4a5645640 10 monclient(hunting): picked mon.noname-c con 0x7fc4a0099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:36.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc4a5645640 10 monclient(hunting): picked mon.noname-a con 0x7fc4a009c550 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:36.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc4a5645640 10 monclient(hunting): picked mon.noname-b con 0x7fc4a009e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:36.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc4a5645640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a0099e50 con 0x7fc4a009c550 2024-09-16T14:49:36.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc4a5645640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a009dbd0 con 0x7fc4a0099a70 2024-09-16T14:49:36.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc4a5645640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a009ee20 con 0x7fc4a009e6d0 2024-09-16T14:49:36.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc4a5645640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:36.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc4a5645640 10 monclient(hunting): authenticate will time out at 1358.975439s 2024-09-16T14:49:36.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc49effd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc4a0099a70 0x7fc4a009dfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60318/0 (socket says 172.21.15.189:60318) 2024-09-16T14:49:36.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc49effd640 1 -- 172.21.15.189:0/2586024366 learned_addr learned my addr 172.21.15.189:0/2586024366 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:36.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc49dffb640 1 -- 172.21.15.189:0/2586024366 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759404211 0 0) 0x7fc4a009dbd0 con 0x7fc4a0099a70 2024-09-16T14:49:36.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc49dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc49dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc49dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.856+0000 7fc49dffb640 10 monclient(hunting): my global_id is 144598 2024-09-16T14:49:36.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 1 -- 172.21.15.189:0/2586024366 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc488003530 con 0x7fc4a0099a70 2024-09-16T14:49:36.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 1 -- 172.21.15.189:0/2586024366 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270275377 0 0) 0x7fc4a009ee20 con 0x7fc4a009e6d0 2024-09-16T14:49:36.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 10 monclient(hunting): my global_id is 154385 2024-09-16T14:49:36.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 1 -- 172.21.15.189:0/2586024366 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a009dbd0 con 0x7fc4a009e6d0 2024-09-16T14:49:36.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 1 -- 172.21.15.189:0/2586024366 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 970337219 0 0) 0x7fc488003530 con 0x7fc4a0099a70 2024-09-16T14:49:36.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 1 -- 172.21.15.189:0/2586024366 >> v1:172.21.15.26:6790/0 conn(0x7fc4a009e6d0 legacy=0x7fc4a013b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 1 -- 172.21.15.189:0/2586024366 >> v1:172.21.15.26:6789/0 conn(0x7fc4a009c550 legacy=0x7fc4a01379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T14:49:36.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:49:36.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 1 monclient: found mon.noname-c 2024-09-16T14:49:36.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:36.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:49:36.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 1 -- 172.21.15.189:0/2586024366 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a009f100 con 0x7fc4a0099a70 2024-09-16T14:49:36.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 10 monclient: _finish_auth 0 2024-09-16T14:49:36.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:49:36.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:36.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 5 monclient: authenticate success, global_id 144598 2024-09-16T14:49:36.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 1 -- 172.21.15.189:0/2586024366 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc490001620 con 0x7fc4a0099a70 2024-09-16T14:49:36.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:36.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:36.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 10 monclient: dump: 2024-09-16T14:49:36.866 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:36.866 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.866 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.866 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.866 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.866 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:36.866 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.866 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.866 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.866 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:36.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:36.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 1 -- 172.21.15.189:0/2586024366 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc490001840 con 0x7fc4a0099a70 2024-09-16T14:49:36.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:36.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 1 -- 172.21.15.189:0/2586024366 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc490001b90 con 0x7fc4a0099a70 2024-09-16T14:49:36.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:36.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49dffb640 10 monclient: dump: 2024-09-16T14:49:36.867 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:36.867 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.867 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.867 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.867 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.867 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:36.867 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.868 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.868 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.868 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:36.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:36.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 4 set_mon_vals no callback set 2024-09-16T14:49:36.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 10 monclient: shutdown 2024-09-16T14:49:36.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:36.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 1 -- 172.21.15.189:0/2586024366 >> v1:172.21.15.189:6789/0 conn(0x7fc4a0099a70 legacy=0x7fc4a009dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 1 -- 172.21.15.189:0/2586024366 shutdown_connections 2024-09-16T14:49:36.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 1 -- 172.21.15.189:0/2586024366 >> 172.21.15.189:0/2586024366 conn(0x7fc4a00a3380 msgr2=0x7fc4a009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:36.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 1 -- 172.21.15.189:0/2586024366 shutdown_connections 2024-09-16T14:49:36.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 1 -- 172.21.15.189:0/2586024366 wait complete. 2024-09-16T14:49:36.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 10 monclient: build_initial_monmap 2024-09-16T14:49:36.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 10 monclient: monmap: 2024-09-16T14:49:36.869 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:36.869 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:36.869 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:36.869 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:36.869 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:36.869 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:36.869 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:36.869 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:36.869 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:36.869 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:36.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 1 Processor -- start 2024-09-16T14:49:36.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 1 -- start start 2024-09-16T14:49:36.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 10 monclient: init 2024-09-16T14:49:36.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:36.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:49:36.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 10 monclient(hunting): picked mon.noname-c con 0x7fc4a0099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:36.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 10 monclient(hunting): picked mon.noname-b con 0x7fc4a009c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:36.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 10 monclient(hunting): picked mon.noname-a con 0x7fc4a009e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:36.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a009cf20 con 0x7fc4a009e6d0 2024-09-16T14:49:36.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a006b320 con 0x7fc4a009c550 2024-09-16T14:49:36.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a013a1a0 con 0x7fc4a0099a70 2024-09-16T14:49:36.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:36.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 10 monclient(hunting): authenticate will time out at 1358.978559s 2024-09-16T14:49:36.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49e7fc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc4a009c550 0x7fc4a009ab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:60332/0 (socket says 172.21.15.189:60332) 2024-09-16T14:49:36.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49e7fc640 1 -- 172.21.15.189:0/3740926399 learned_addr learned my addr 172.21.15.189:0/3740926399 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:36.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49effd640 1 --1- 172.21.15.189:0/3740926399 >> v1:172.21.15.26:6790/0 conn(0x7fc4a0099a70 0x7fc4a009a480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:43958/0 (socket says 172.21.15.189:43958) 2024-09-16T14:49:36.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650194087 0 0) 0x7fc4a006b320 con 0x7fc4a009c550 2024-09-16T14:49:36.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): my global_id is 144601 2024-09-16T14:49:36.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc48c003530 con 0x7fc4a009c550 2024-09-16T14:49:36.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488259774 0 0) 0x7fc4a013a1a0 con 0x7fc4a0099a70 2024-09-16T14:49:36.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): my global_id is 154388 2024-09-16T14:49:36.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a006b320 con 0x7fc4a0099a70 2024-09-16T14:49:36.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106896732 0 0) 0x7fc4a009cf20 con 0x7fc4a009e6d0 2024-09-16T14:49:36.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): my global_id is 164154 2024-09-16T14:49:36.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a013a1a0 con 0x7fc4a009e6d0 2024-09-16T14:49:36.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2968958325 0 0) 0x7fc48c003530 con 0x7fc4a009c550 2024-09-16T14:49:36.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4a009cf20 con 0x7fc4a009c550 2024-09-16T14:49:36.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc4940018c0 con 0x7fc4a009c550 2024-09-16T14:49:36.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:36.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): dump: 2024-09-16T14:49:36.877 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:36.877 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.877 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.877 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.877 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.878 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:36.878 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.878 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.878 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.879 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:36.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:36.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2521188424 0 0) 0x7fc4a006b320 con 0x7fc4a0099a70 2024-09-16T14:49:36.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc48c003530 con 0x7fc4a0099a70 2024-09-16T14:49:36.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc490001b50 con 0x7fc4a0099a70 2024-09-16T14:49:36.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:36.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): dump: 2024-09-16T14:49:36.880 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:36.880 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.880 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.880 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.880 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.880 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:36.880 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.880 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.880 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.880 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:36.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1692123122 0 0) 0x7fc4a013a1a0 con 0x7fc4a009e6d0 2024-09-16T14:49:36.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4a006b320 con 0x7fc4a009e6d0 2024-09-16T14:49:36.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc498001d80 con 0x7fc4a009e6d0 2024-09-16T14:49:36.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:36.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient(hunting): dump: 2024-09-16T14:49:36.881 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:36.881 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.881 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.881 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.881 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.881 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:36.882 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.882 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.882 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.882 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:36.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 146798346 0 0) 0x7fc4a009cf20 con 0x7fc4a009c550 2024-09-16T14:49:36.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 >> v1:172.21.15.26:6790/0 conn(0x7fc4a0099a70 legacy=0x7fc4a009a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 >> v1:172.21.15.26:6789/0 conn(0x7fc4a009e6d0 legacy=0x7fc4a01d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient: _finish_hunting 0 2024-09-16T14:49:36.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 monclient: found mon.b 2024-09-16T14:49:36.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:36.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:36.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a013a480 con 0x7fc4a009c550 2024-09-16T14:49:36.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient: _finish_auth 0 2024-09-16T14:49:36.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:36.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:36.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 5 monclient: authenticate success, global_id 144601 2024-09-16T14:49:36.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 10 monclient: _renew_subs 2024-09-16T14:49:36.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:36.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a013a760 con 0x7fc4a009c550 2024-09-16T14:49:36.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 10 monclient: _renew_subs 2024-09-16T14:49:36.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:36.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4a5645640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a009bf30 con 0x7fc4a009c550 2024-09-16T14:49:36.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc494001c70 con 0x7fc4a009c550 2024-09-16T14:49:36.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:36.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc4940038b0 con 0x7fc4a009c550 2024-09-16T14:49:36.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:36.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc4877fe640 10 monclient: dump: 2024-09-16T14:49:36.884 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:36.884 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.884 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.884 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.884 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.884 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:36.884 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.884 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.884 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.884 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:36.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.860+0000 7fc49cff9640 4 set_mon_vals no callback set 2024-09-16T14:49:36.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.864+0000 7fc4a5645640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:36.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.864+0000 7fc4a5645640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:36.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.864+0000 7fc4a5645640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:36.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.864+0000 7fc4a5645640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc480005050 con 0x7fc4a009c550 2024-09-16T14:49:36.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.864+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc494003a10 con 0x7fc4a009c550 2024-09-16T14:49:36.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.864+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(200..200 src has 169..200) v4 ==== 10877+0+0 (unknown 2031785554 0 0) 0x7fc494044ba0 con 0x7fc4a009c550 2024-09-16T14:49:36.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.864+0000 7fc49effd640 10 monclient: get_auth_request con 0x7fc48c036570 auth_method 0 2024-09-16T14:49:36.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.868+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc494004210 con 0x7fc4a009c550 2024-09-16T14:49:36.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.868+0000 7fc4877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:36.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:36.868+0000 7fc4877fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 10 monclient: get_monmap_and_config 2024-09-16T14:49:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 10 monclient: build_initial_monmap 2024-09-16T14:49:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 10 monclient: monmap: 2024-09-16T14:49:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:36.974244+0000 2024-09-16T14:49:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:36.974244+0000 2024-09-16T14:49:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:36.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 1 Processor -- start 2024-09-16T14:49:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 1 -- start start 2024-09-16T14:49:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 10 monclient: init 2024-09-16T14:49:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:49:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 10 monclient(hunting): picked mon.noname-a con 0x7f62e40a1ca0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 10 monclient(hunting): picked mon.noname-b con 0x7f62e40a2790 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 10 monclient(hunting): picked mon.noname-c con 0x7f62e409ff50 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e409f520 con 0x7f62e40a1ca0 2024-09-16T14:49:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e40a0360 con 0x7f62e409ff50 2024-09-16T14:49:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e409e7a0 con 0x7f62e40a2790 2024-09-16T14:49:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 10 monclient(hunting): authenticate will time out at 1401.094131s 2024-09-16T14:49:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e9a0d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f62e40a2790 0x7f62e4139e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43678/0 (socket says 172.21.15.26:43678) 2024-09-16T14:49:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ea20e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f62e40a1ca0 0x7f62e40a2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:45478/0 (socket says 172.21.15.26:45478) 2024-09-16T14:49:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e9a0d640 1 -- 172.21.15.26:0/499903396 learned_addr learned my addr 172.21.15.26:0/499903396 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 1 -- 172.21.15.26:0/499903396 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550962435 0 0) 0x7f62e409e7a0 con 0x7f62e40a2790 2024-09-16T14:49:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient(hunting): my global_id is 154391 2024-09-16T14:49:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 1 -- 172.21.15.26:0/499903396 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62cc003530 con 0x7f62e40a2790 2024-09-16T14:49:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 1 -- 172.21.15.26:0/499903396 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071243582 0 0) 0x7f62e409f520 con 0x7f62e40a1ca0 2024-09-16T14:49:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient(hunting): my global_id is 164157 2024-09-16T14:49:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 1 -- 172.21.15.26:0/499903396 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e409e7a0 con 0x7f62e40a1ca0 2024-09-16T14:49:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 1 -- 172.21.15.26:0/499903396 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1252591029 0 0) 0x7f62cc003530 con 0x7f62e40a2790 2024-09-16T14:49:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 1 -- 172.21.15.26:0/499903396 >> v1:172.21.15.189:6789/0 conn(0x7f62e409ff50 legacy=0x7f62e413d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 1 -- 172.21.15.26:0/499903396 >> v1:172.21.15.26:6789/0 conn(0x7f62e40a1ca0 legacy=0x7f62e40a2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient: _finish_hunting 0 2024-09-16T14:49:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 1 monclient: found mon.noname-b 2024-09-16T14:49:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:36.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:49:36.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 1 -- 172.21.15.26:0/499903396 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e409ec80 con 0x7f62e40a2790 2024-09-16T14:49:36.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient: _finish_auth 0 2024-09-16T14:49:36.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient: _check_auth_tickets 2024-09-16T14:49:36.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:36.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 1 -- 172.21.15.26:0/499903396 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f62d80019a0 con 0x7f62e40a2790 2024-09-16T14:49:36.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:36.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient: dump: 2024-09-16T14:49:36.984 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.984 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.984 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.985 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.985 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.985 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.985 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.985 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.985 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.985 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 5 monclient: authenticate success, global_id 154391 2024-09-16T14:49:36.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:36.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 1 -- 172.21.15.26:0/499903396 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62d8001bc0 con 0x7f62e40a2790 2024-09-16T14:49:36.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:36.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 1 -- 172.21.15.26:0/499903396 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f62d8001f90 con 0x7f62e40a2790 2024-09-16T14:49:36.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:36.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62e920c640 10 monclient: dump: 2024-09-16T14:49:36.986 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.986 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.986 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.986 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.986 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.986 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.986 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.986 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.986 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.986 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:36.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 4 set_mon_vals no callback set 2024-09-16T14:49:36.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 10 monclient: shutdown 2024-09-16T14:49:36.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:36.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 1 -- 172.21.15.26:0/499903396 >> v1:172.21.15.26:6790/0 conn(0x7f62e40a2790 legacy=0x7f62e4139e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 1 -- 172.21.15.26:0/499903396 shutdown_connections 2024-09-16T14:49:36.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 1 -- 172.21.15.26:0/499903396 >> 172.21.15.26:0/499903396 conn(0x7f62e40a3da0 msgr2=0x7f62e40a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:36.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 1 -- 172.21.15.26:0/499903396 shutdown_connections 2024-09-16T14:49:36.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.970+0000 7f62ec472640 1 -- 172.21.15.26:0/499903396 wait complete. 2024-09-16T14:49:36.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 10 monclient: build_initial_monmap 2024-09-16T14:49:36.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 10 monclient: monmap: 2024-09-16T14:49:36.987 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:36.987 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:36.987 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:36.988 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:36.988 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:36.988 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:36.988 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:36.988 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:36.988 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:36.988 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 1 Processor -- start 2024-09-16T14:49:36.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 1 -- start start 2024-09-16T14:49:36.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 10 monclient: init 2024-09-16T14:49:36.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:36.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 10 monclient(hunting): picked mon.noname-b con 0x7f62e409ff50 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 10 monclient(hunting): picked mon.noname-a con 0x7f62e40a1ca0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 10 monclient(hunting): picked mon.noname-c con 0x7f62e40a2790 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e40a55f0 con 0x7f62e40a1ca0 2024-09-16T14:49:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e413cfb0 con 0x7f62e409ff50 2024-09-16T14:49:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e41c9d30 con 0x7f62e40a2790 2024-09-16T14:49:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 10 monclient(hunting): authenticate will time out at 1401.097430s 2024-09-16T14:49:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e9a0d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f62e40a1ca0 0x7f62e41c9620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:45492/0 (socket says 172.21.15.26:45492) 2024-09-16T14:49:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e9a0d640 1 -- 172.21.15.26:0/96460819 learned_addr learned my addr 172.21.15.26:0/96460819 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220006551 0 0) 0x7f62e40a55f0 con 0x7f62e40a1ca0 2024-09-16T14:49:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): my global_id is 164160 2024-09-16T14:49:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62d0003530 con 0x7f62e40a1ca0 2024-09-16T14:49:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078192502 0 0) 0x7f62e41c9d30 con 0x7f62e40a2790 2024-09-16T14:49:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): my global_id is 154394 2024-09-16T14:49:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e40a55f0 con 0x7f62e40a2790 2024-09-16T14:49:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018976664 0 0) 0x7f62e413cfb0 con 0x7f62e409ff50 2024-09-16T14:49:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): my global_id is 144607 2024-09-16T14:49:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e41c9d30 con 0x7f62e409ff50 2024-09-16T14:49:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1346361026 0 0) 0x7f62d0003530 con 0x7f62e40a1ca0 2024-09-16T14:49:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62e413cfb0 con 0x7f62e40a1ca0 2024-09-16T14:49:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f62d8001ed0 con 0x7f62e40a1ca0 2024-09-16T14:49:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): dump: 2024-09-16T14:49:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.992 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 490285705 0 0) 0x7f62e40a55f0 con 0x7f62e40a2790 2024-09-16T14:49:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62d0003530 con 0x7f62e40a2790 2024-09-16T14:49:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f62dc001840 con 0x7f62e40a2790 2024-09-16T14:49:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): dump: 2024-09-16T14:49:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.993 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3313388289 0 0) 0x7f62e413cfb0 con 0x7f62e40a1ca0 2024-09-16T14:49:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 >> v1:172.21.15.26:6790/0 conn(0x7f62e40a2790 legacy=0x7f62e41d1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 >> v1:172.21.15.189:6789/0 conn(0x7f62e409ff50 legacy=0x7f62e41c8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 monclient: found mon.a 2024-09-16T14:49:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e40a51a0 con 0x7f62e40a1ca0 2024-09-16T14:49:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:49:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 5 monclient: authenticate success, global_id 164160 2024-09-16T14:49:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 10 monclient: _renew_subs 2024-09-16T14:49:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 1 -- 172.21.15.26:0/96460819 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e409e490 con 0x7f62e40a1ca0 2024-09-16T14:49:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 10 monclient: _renew_subs 2024-09-16T14:49:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 1 -- 172.21.15.26:0/96460819 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e413ccd0 con 0x7f62e40a1ca0 2024-09-16T14:49:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62d8001580 con 0x7f62e40a1ca0 2024-09-16T14:49:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f62d8004fc0 con 0x7f62e40a1ca0 2024-09-16T14:49:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e8a0b640 4 set_mon_vals no callback set 2024-09-16T14:49:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient: dump: 2024-09-16T14:49:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.996 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62ec472640 1 -- 172.21.15.26:0/96460819 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f62c4006590 con 0x7f62c4002830 2024-09-16T14:49:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f62d800fa30 con 0x7f62e40a1ca0 2024-09-16T14:49:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(200..200 src has 169..200) v4 ==== 10877+0+0 (unknown 2031785554 0 0) 0x7f62d80451d0 con 0x7f62e40a1ca0 2024-09-16T14:49:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712601142 0 0) 0x7f62c4006590 con 0x7f62c4002830 2024-09-16T14:49:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 10 monclient(hunting): my global_id is 154397 2024-09-16T14:49:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.974+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e413cfb0 con 0x7f62c4002830 2024-09-16T14:49:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.978+0000 7f62eaa0f640 10 monclient: get_auth_request con 0x7f62d0036390 auth_method 0 2024-09-16T14:49:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.978+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1103185723 0 0) 0x7f62e413cfb0 con 0x7f62c4002830 2024-09-16T14:49:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.978+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62c4006590 con 0x7f62c4002830 2024-09-16T14:49:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.978+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f62d4001840 con 0x7f62c4002830 2024-09-16T14:49:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.978+0000 7f62e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.978+0000 7f62e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.978+0000 7f62e2ffd640 10 monclient: dump: 2024-09-16T14:49:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:36.998 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.978+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2144976797 0 0) 0x7f62c4006590 con 0x7f62c4002830 2024-09-16T14:49:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.978+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62c400a0e0 con 0x7f62c4002830 2024-09-16T14:49:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.978+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f62d4001c10 con 0x7f62c4002830 2024-09-16T14:49:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.978+0000 7f62e2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.978+0000 7f62e2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.978+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 >> v1:172.21.15.26:6790/0 conn(0x7f62c4002830 legacy=0x7f62c4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.978+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 >> v1:172.21.15.26:6790/0 conn(0x7f62c4002830 legacy=0x7f62c4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62ec472640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:49:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62ec472640 1 -- 172.21.15.26:0/96460819 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f62c4009d80 con 0x7f62c40090a0 2024-09-16T14:49:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989069190 0 0) 0x7f62c4009d80 con 0x7f62c40090a0 2024-09-16T14:49:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 10 monclient(hunting): my global_id is 154400 2024-09-16T14:49:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62c4006590 con 0x7f62c40090a0 2024-09-16T14:49:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1334298884 0 0) 0x7f62c4006590 con 0x7f62c40090a0 2024-09-16T14:49:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62c4009d80 con 0x7f62c40090a0 2024-09-16T14:49:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f62d4001d80 con 0x7f62c40090a0 2024-09-16T14:49:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 10 monclient: dump: 2024-09-16T14:49:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:37.002 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:37.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4000085968 0 0) 0x7f62c4009d80 con 0x7f62c40090a0 2024-09-16T14:49:37.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f62c400c290 con 0x7f62c40090a0 2024-09-16T14:49:37.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 1921265856) 0x7f62d4004dd0 con 0x7f62c40090a0 2024-09-16T14:49:37.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:49:37.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:37.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 >> v1:172.21.15.26:6790/0 conn(0x7f62c40090a0 legacy=0x7f62c4009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.982+0000 7f62e2ffd640 1 -- 172.21.15.26:0/96460819 >> v1:172.21.15.26:6790/0 conn(0x7f62c40090a0 legacy=0x7f62c4009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:37.002 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:49:37.002 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:49:37.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:49:37.003 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:49:37.003 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 66, 2024-09-16T14:49:37.003 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:49:37.003 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:49:37.003 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:49:37.003 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:49:37.003 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:37.003 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:49:37.003 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:37.003 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:49:37.003 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:49:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:49:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:49:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:49:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:49:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:49:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:49:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:49:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:49:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:49:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:49:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:49:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:49:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:49:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:49:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:49:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:49:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:49:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:49:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:49:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:49:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:49:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:49:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:49:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:49:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:49:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:49:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:49:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:49:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:49:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:49:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:49:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:49:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:49:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:49:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:49:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:49:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:49:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:49:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:49:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:49:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.014 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:49:37.014 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:49:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.986+0000 7f62ec472640 1 -- 172.21.15.26:0/96460819 >> v1:172.21.15.189:6812/3196161582 conn(0x7f62d0036390 legacy=0x7f62d0038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.986+0000 7f62ec472640 10 monclient: shutdown 2024-09-16T14:49:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.986+0000 7f62ec472640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.986+0000 7f62ec472640 1 -- 172.21.15.26:0/96460819 >> v1:172.21.15.26:6789/0 conn(0x7f62e40a1ca0 legacy=0x7f62e41c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.986+0000 7f62eaa0f640 1 -- 172.21.15.26:0/96460819 reap_dead start 2024-09-16T14:49:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.986+0000 7f62eaa0f640 1 -- 172.21.15.26:0/96460819 reap_dead start 2024-09-16T14:49:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.986+0000 7f62ec472640 1 -- 172.21.15.26:0/96460819 shutdown_connections 2024-09-16T14:49:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.986+0000 7f62ec472640 1 -- 172.21.15.26:0/96460819 >> 172.21.15.26:0/96460819 conn(0x7f62e40a3da0 msgr2=0x7f62e409aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.986+0000 7f62ec472640 1 -- 172.21.15.26:0/96460819 shutdown_connections 2024-09-16T14:49:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:36.986+0000 7f62ec472640 1 -- 172.21.15.26:0/96460819 wait complete. 2024-09-16T14:49:37.016 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:49:37.016 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:49:37.017 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:49:37.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.028+0000 7fc4a5645640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:37.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.028+0000 7fc4a5645640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:37.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.028+0000 7fc4a5645640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:49:37.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.028+0000 7fc4a5645640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fc480005300 con 0x7fc4a009c550 2024-09-16T14:49:37.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.134+0000 7f0a4784e640 10 monclient: get_monmap_and_config 2024-09-16T14:49:37.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.134+0000 7f0a4784e640 10 monclient: build_initial_monmap 2024-09-16T14:49:37.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.134+0000 7f0a4784e640 10 monclient: monmap: 2024-09-16T14:49:37.140 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:37.141 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:37.141 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:37.141371+0000 2024-09-16T14:49:37.141 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:37.141371+0000 2024-09-16T14:49:37.141 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:37.141 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:37.141 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:37.141 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:37.141 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:37.141 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:37.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 1 Processor -- start 2024-09-16T14:49:37.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 1 -- start start 2024-09-16T14:49:37.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 10 monclient: init 2024-09-16T14:49:37.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:37.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:37.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 10 monclient(hunting): picked mon.noname-b con 0x7f0a4009cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:37.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 10 monclient(hunting): picked mon.noname-a con 0x7f0a4009e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:37.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 10 monclient(hunting): picked mon.noname-c con 0x7f0a4009d1a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:37.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a4009d5b0 con 0x7f0a4009e7a0 2024-09-16T14:49:37.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a40131a90 con 0x7f0a4009d1a0 2024-09-16T14:49:37.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a4009aff0 con 0x7f0a4009cdc0 2024-09-16T14:49:37.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:37.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 10 monclient(hunting): authenticate will time out at 1401.261388s 2024-09-16T14:49:37.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a44de9640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0a4009e7a0 0x7f0a40131380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:45504/0 (socket says 172.21.15.26:45504) 2024-09-16T14:49:37.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a455ea640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0a4009cdc0 0x7f0a40130c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43710/0 (socket says 172.21.15.26:43710) 2024-09-16T14:49:37.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a44de9640 1 -- 172.21.15.26:0/276959952 learned_addr learned my addr 172.21.15.26:0/276959952 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:37.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 1 -- 172.21.15.26:0/276959952 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688597802 0 0) 0x7f0a4009aff0 con 0x7f0a4009cdc0 2024-09-16T14:49:37.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:37.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:37.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:37.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient(hunting): my global_id is 154403 2024-09-16T14:49:37.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 1 -- 172.21.15.26:0/276959952 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a2c004640 con 0x7f0a4009cdc0 2024-09-16T14:49:37.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 1 -- 172.21.15.26:0/276959952 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277162893 0 0) 0x7f0a4009d5b0 con 0x7f0a4009e7a0 2024-09-16T14:49:37.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:37.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:37.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:37.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient(hunting): my global_id is 164163 2024-09-16T14:49:37.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 1 -- 172.21.15.26:0/276959952 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a4009aff0 con 0x7f0a4009e7a0 2024-09-16T14:49:37.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 1 -- 172.21.15.26:0/276959952 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1077684691 0 0) 0x7f0a4009aff0 con 0x7f0a4009e7a0 2024-09-16T14:49:37.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 1 -- 172.21.15.26:0/276959952 >> v1:172.21.15.26:6790/0 conn(0x7f0a4009cdc0 legacy=0x7f0a40130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 1 -- 172.21.15.26:0/276959952 >> v1:172.21.15.189:6789/0 conn(0x7f0a4009d1a0 legacy=0x7f0a40139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient: _finish_hunting 0 2024-09-16T14:49:37.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 1 monclient: found mon.noname-a 2024-09-16T14:49:37.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:37.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:49:37.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 1 -- 172.21.15.26:0/276959952 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4009b2d0 con 0x7f0a4009e7a0 2024-09-16T14:49:37.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient: _finish_auth 0 2024-09-16T14:49:37.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient: _check_auth_tickets 2024-09-16T14:49:37.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:37.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 5 monclient: authenticate success, global_id 164163 2024-09-16T14:49:37.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:37.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 1 -- 172.21.15.26:0/276959952 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0a30001580 con 0x7f0a4009e7a0 2024-09-16T14:49:37.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:37.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:37.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient: dump: 2024-09-16T14:49:37.152 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:37.152 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:37.152 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.152 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.152 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:37.152 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:37.152 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:37.152 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:37.152 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:37.152 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:37.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:37.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 1 -- 172.21.15.26:0/276959952 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a30001830 con 0x7f0a4009e7a0 2024-09-16T14:49:37.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 1 -- 172.21.15.26:0/276959952 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0a30001c00 con 0x7f0a4009e7a0 2024-09-16T14:49:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 4 set_mon_vals no callback set 2024-09-16T14:49:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a37fff640 10 monclient: dump: 2024-09-16T14:49:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:37.154 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 10 monclient: shutdown 2024-09-16T14:49:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 1 -- 172.21.15.26:0/276959952 >> v1:172.21.15.26:6789/0 conn(0x7f0a4009e7a0 legacy=0x7f0a40131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 1 -- 172.21.15.26:0/276959952 shutdown_connections 2024-09-16T14:49:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 1 -- 172.21.15.26:0/276959952 >> 172.21.15.26:0/276959952 conn(0x7f0a4009fb50 msgr2=0x7f0a4009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 1 -- 172.21.15.26:0/276959952 shutdown_connections 2024-09-16T14:49:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 1 -- 172.21.15.26:0/276959952 wait complete. 2024-09-16T14:49:37.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 10 monclient: build_initial_monmap 2024-09-16T14:49:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.138+0000 7f0a4784e640 10 monclient: monmap: 2024-09-16T14:49:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:37.155 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 1 Processor -- start 2024-09-16T14:49:37.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 1 -- start start 2024-09-16T14:49:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 10 monclient: init 2024-09-16T14:49:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 10 monclient(hunting): picked mon.noname-c con 0x7f0a4009cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 10 monclient(hunting): picked mon.noname-a con 0x7f0a4009d1a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 10 monclient(hunting): picked mon.noname-b con 0x7f0a4009e7a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a400a1ff0 con 0x7f0a4009d1a0 2024-09-16T14:49:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a4009a240 con 0x7f0a4009e7a0 2024-09-16T14:49:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a4011c4a0 con 0x7f0a4009cdc0 2024-09-16T14:49:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 10 monclient(hunting): authenticate will time out at 1401.264620s 2024-09-16T14:49:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a455ea640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0a4009cdc0 0x7f0a4012dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43726/0 (socket says 172.21.15.26:43726) 2024-09-16T14:49:37.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a44de9640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0a4009d1a0 0x7f0a401c9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:45518/0 (socket says 172.21.15.26:45518) 2024-09-16T14:49:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a455ea640 1 -- 172.21.15.26:0/4269314163 learned_addr learned my addr 172.21.15.26:0/4269314163 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880086671 0 0) 0x7f0a4011c4a0 con 0x7f0a4009cdc0 2024-09-16T14:49:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): my global_id is 154406 2024-09-16T14:49:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a24003530 con 0x7f0a4009cdc0 2024-09-16T14:49:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857274383 0 0) 0x7f0a400a1ff0 con 0x7f0a4009d1a0 2024-09-16T14:49:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): my global_id is 164166 2024-09-16T14:49:37.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a4011c4a0 con 0x7f0a4009d1a0 2024-09-16T14:49:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 433164130 0 0) 0x7f0a24003530 con 0x7f0a4009cdc0 2024-09-16T14:49:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a400a1ff0 con 0x7f0a4009cdc0 2024-09-16T14:49:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0a3c001900 con 0x7f0a4009cdc0 2024-09-16T14:49:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): dump: 2024-09-16T14:49:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:37.159 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 157754886 0 0) 0x7f0a4011c4a0 con 0x7f0a4009d1a0 2024-09-16T14:49:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a24003530 con 0x7f0a4009d1a0 2024-09-16T14:49:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0a30001540 con 0x7f0a4009d1a0 2024-09-16T14:49:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): dump: 2024-09-16T14:49:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:37.160 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248769626 0 0) 0x7f0a4009a240 con 0x7f0a4009e7a0 2024-09-16T14:49:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:37.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): my global_id is 144613 2024-09-16T14:49:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a4011c4a0 con 0x7f0a4009e7a0 2024-09-16T14:49:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1446432657 0 0) 0x7f0a400a1ff0 con 0x7f0a4009cdc0 2024-09-16T14:49:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 >> v1:172.21.15.189:6789/0 conn(0x7f0a4009e7a0 legacy=0x7f0a401cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 >> v1:172.21.15.26:6789/0 conn(0x7f0a4009d1a0 legacy=0x7f0a401c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:49:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 monclient: found mon.c 2024-09-16T14:49:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4012e440 con 0x7f0a4009cdc0 2024-09-16T14:49:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient: _finish_auth 0 2024-09-16T14:49:37.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:49:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 5 monclient: authenticate success, global_id 154406 2024-09-16T14:49:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 10 monclient: _renew_subs 2024-09-16T14:49:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 1 -- 172.21.15.26:0/4269314163 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a4012e1e0 con 0x7f0a4009cdc0 2024-09-16T14:49:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 10 monclient: _renew_subs 2024-09-16T14:49:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 1 -- 172.21.15.26:0/4269314163 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4011ad10 con 0x7f0a4009cdc0 2024-09-16T14:49:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a3c0047f0 con 0x7f0a4009cdc0 2024-09-16T14:49:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0a3c004b80 con 0x7f0a4009cdc0 2024-09-16T14:49:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:37.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a377fe640 4 set_mon_vals no callback set 2024-09-16T14:49:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient: dump: 2024-09-16T14:49:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:37.163 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:37.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a4784e640 1 -- 172.21.15.26:0/4269314163 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0a20006590 con 0x7f0a20002830 2024-09-16T14:49:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0a3c00f5f0 con 0x7f0a4009cdc0 2024-09-16T14:49:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(200..200 src has 169..200) v4 ==== 10877+0+0 (unknown 2031785554 0 0) 0x7f0a3c044e00 con 0x7f0a4009cdc0 2024-09-16T14:49:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a45deb640 10 monclient: get_auth_request con 0x7f0a24036490 auth_method 0 2024-09-16T14:49:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672033090 0 0) 0x7f0a20006590 con 0x7f0a20002830 2024-09-16T14:49:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient(hunting): my global_id is 144616 2024-09-16T14:49:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a400a1ff0 con 0x7f0a20002830 2024-09-16T14:49:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3236053124 0 0) 0x7f0a400a1ff0 con 0x7f0a20002830 2024-09-16T14:49:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a20006590 con 0x7f0a20002830 2024-09-16T14:49:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0a30004b80 con 0x7f0a20002830 2024-09-16T14:49:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:37.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 10 monclient: dump: 2024-09-16T14:49:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:37.165 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1540076132 0 0) 0x7f0a20006590 con 0x7f0a20002830 2024-09-16T14:49:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.142+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a2000a0e0 con 0x7f0a20002830 2024-09-16T14:49:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.146+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0a30004e20 con 0x7f0a20002830 2024-09-16T14:49:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.146+0000 7f0a35ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.146+0000 7f0a35ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.146+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 >> v1:172.21.15.189:6789/0 conn(0x7f0a20002830 legacy=0x7f0a20004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.146+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 >> v1:172.21.15.189:6789/0 conn(0x7f0a20002830 legacy=0x7f0a20004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.150+0000 7f0a4784e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:49:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.150+0000 7f0a4784e640 1 -- 172.21.15.26:0/4269314163 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0a20009d80 con 0x7f0a200090a0 2024-09-16T14:49:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638392082 0 0) 0x7f0a20009d80 con 0x7f0a200090a0 2024-09-16T14:49:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 10 monclient(hunting): my global_id is 144619 2024-09-16T14:49:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a20006590 con 0x7f0a200090a0 2024-09-16T14:49:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 937875254 0 0) 0x7f0a20006590 con 0x7f0a200090a0 2024-09-16T14:49:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a20009d80 con 0x7f0a200090a0 2024-09-16T14:49:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0a300053e0 con 0x7f0a200090a0 2024-09-16T14:49:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:49:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 10 monclient: dump: 2024-09-16T14:49:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:37.169 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 495802135 0 0) 0x7f0a20009d80 con 0x7f0a200090a0 2024-09-16T14:49:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0a2000c290 con 0x7f0a200090a0 2024-09-16T14:49:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 3047835638) 0x7f0a300056f0 con 0x7f0a200090a0 2024-09-16T14:49:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:49:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 >> v1:172.21.15.189:6789/0 conn(0x7f0a200090a0 legacy=0x7f0a20009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a35ffb640 1 -- 172.21.15.26:0/4269314163 >> v1:172.21.15.189:6789/0 conn(0x7f0a200090a0 legacy=0x7f0a20009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:37.170 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:49:37.170 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:49:37.170 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:49:37.170 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:49:37.170 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 66, 2024-09-16T14:49:37.170 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:49:37.170 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:49:37.170 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:49:37.170 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:49:37.170 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:37.170 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:49:37.170 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:37.171 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:49:37.171 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:49:37.171 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:37.171 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:37.171 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:37.171 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:37.171 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:37.171 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:37.171 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:37.171 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:37.171 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:37.171 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:37.171 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:37.171 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:49:37.171 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:49:37.172 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:37.172 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:37.172 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:37.172 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:37.172 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:37.172 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:37.172 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:37.172 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:37.172 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:37.172 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:37.172 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:37.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.173 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:49:37.173 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:49:37.173 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:49:37.173 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:49:37.173 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:49:37.173 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:49:37.173 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:37.173 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:37.173 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:49:37.173 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:49:37.173 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:49:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:49:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:49:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:49:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:49:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:49:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:37.174 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:49:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:49:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:49:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:49:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:37.175 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:49:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:49:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:49:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:49:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:49:37.176 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:49:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:49:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:49:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:37.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:49:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:49:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:49:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:49:37.178 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:49:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:49:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:49:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:49:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:37.179 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:49:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:49:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:49:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:37.180 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:49:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:37.181 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:49:37.181 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:49:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a4784e640 1 -- 172.21.15.26:0/4269314163 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0a24036490 legacy=0x7f0a24038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a4784e640 10 monclient: shutdown 2024-09-16T14:49:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a4784e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a4784e640 1 -- 172.21.15.26:0/4269314163 >> v1:172.21.15.26:6790/0 conn(0x7f0a4009cdc0 legacy=0x7f0a4012dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a45deb640 1 -- 172.21.15.26:0/4269314163 reap_dead start 2024-09-16T14:49:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a45deb640 1 -- 172.21.15.26:0/4269314163 reap_dead start 2024-09-16T14:49:37.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a4784e640 1 -- 172.21.15.26:0/4269314163 shutdown_connections 2024-09-16T14:49:37.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a4784e640 1 -- 172.21.15.26:0/4269314163 >> 172.21.15.26:0/4269314163 conn(0x7f0a4009fb50 msgr2=0x7f0a4006ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:37.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a4784e640 1 -- 172.21.15.26:0/4269314163 shutdown_connections 2024-09-16T14:49:37.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.154+0000 7f0a4784e640 1 -- 172.21.15.26:0/4269314163 wait complete. 2024-09-16T14:49:37.183 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:49:37.183 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T14:49:37.183 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:49:37.183 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T14:49:37.183 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:49:37.184 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T14:49:37.184 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:49:37.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc49e7fc640 1 -- 172.21.15.189:0/3740926399 >> v1:172.21.15.189:6789/0 conn(0x7fc4a009c550 legacy=0x7fc4a009ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T14:49:37.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc49e7fc640 1 -- 172.21.15.189:0/3740926399 >> v1:172.21.15.189:6789/0 conn(0x7fc4a009c550 legacy=0x7fc4a009ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:49:37.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc49e7fc640 1 --1- 172.21.15.189:0/3740926399 >> v1:172.21.15.189:6789/0 conn(0x7fc4a009c550 0x7fc4a009ab90 :-1 s=OPENED pgs=895 cs=1 l=1).handle_message read tag failed 2024-09-16T14:49:37.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc49e7fc640 1 --1- 172.21.15.189:0/3740926399 >> v1:172.21.15.189:6789/0 conn(0x7fc4a009c550 0x7fc4a009ab90 :-1 s=OPENED pgs=895 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:49:37.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient: ms_handle_reset current mon v1:172.21.15.189:6789/0 2024-09-16T14:49:37.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:37.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 >> v1:172.21.15.189:6789/0 conn(0x7fc4a009c550 legacy=0x7fc4a009ab90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:37.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:37.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient(hunting): picked mon.a con 0x7fc4a009e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:37.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient(hunting): picked mon.b con 0x7fc48c04a940 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:37.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient(hunting): picked mon.c con 0x7fc4a0099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:37.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc4a009cf20 con 0x7fc4a009e6d0 2024-09-16T14:49:37.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc4a013a1a0 con 0x7fc48c04a940 2024-09-16T14:49:37.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc48c054530 con 0x7fc4a0099a70 2024-09-16T14:49:37.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:37.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc49f7fe640 1 -- 172.21.15.189:0/3740926399 >> v1:172.21.15.189:6789/0 conn(0x7fc48c04a940 legacy=0x7fc48c04cdd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:37.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633369726 0 0) 0x7fc4a009cf20 con 0x7fc4a009e6d0 2024-09-16T14:49:37.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:37.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:37.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:49:37.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.26:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fc48c0546c0 con 0x7fc4a009e6d0 2024-09-16T14:49:37.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912064214 0 0) 0x7fc48c054530 con 0x7fc4a0099a70 2024-09-16T14:49:37.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:37.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:37.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:49:37.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.26:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fc4a009cf20 con 0x7fc4a0099a70 2024-09-16T14:49:37.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2304605460 0 0) 0x7fc48c0546c0 con 0x7fc4a009e6d0 2024-09-16T14:49:37.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 >> v1:172.21.15.26:6790/0 conn(0x7fc4a0099a70 legacy=0x7fc48c050970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 >> v1:172.21.15.189:6789/0 conn(0x7fc48c04a940 legacy=0x7fc48c04cdd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:37.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient: _finish_hunting 0 2024-09-16T14:49:37.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 1 monclient: found mon.a 2024-09-16T14:49:37.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:37.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:37.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fc48c055750 con 0x7fc4a009e6d0 2024-09-16T14:49:37.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:37.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:37.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fc4a013a1a0 con 0x7fc4a009e6d0 2024-09-16T14:49:37.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient: _finish_auth 0 2024-09-16T14:49:37.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:37.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:37.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc494014e00 con 0x7fc4a009e6d0 2024-09-16T14:49:37.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:37.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:37.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.188+0000 7fc4877fe640 10 monclient: dump: 2024-09-16T14:49:37.195 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:37.195 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:37.195 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.196 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.196 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:37.196 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:37.196 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:37.196 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:37.196 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:37.196 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:37.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.192+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc494014700 con 0x7fc4a009e6d0 2024-09-16T14:49:37.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.192+0000 7fc4877fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:37.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.192+0000 7fc49cff9640 4 set_mon_vals no callback set 2024-09-16T14:49:37.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:37.192+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc4940410a0 con 0x7fc4a009e6d0 2024-09-16T14:49:37.204 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T14:49:37.204 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:49:37.204 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:49:37.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 10 monclient: get_monmap_and_config 2024-09-16T14:49:37.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 10 monclient: build_initial_monmap 2024-09-16T14:49:37.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 10 monclient: monmap: 2024-09-16T14:49:37.326 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:37.326 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:37.326 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:37.326548+0000 2024-09-16T14:49:37.326 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:37.326548+0000 2024-09-16T14:49:37.326 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:37.326 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:37.326 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:37.326 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:37.326 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:37.326 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:37.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 1 Processor -- start 2024-09-16T14:49:37.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 1 -- start start 2024-09-16T14:49:37.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 10 monclient: init 2024-09-16T14:49:37.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:37.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:37.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 10 monclient(hunting): picked mon.noname-c con 0x7fade00a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:37.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 10 monclient(hunting): picked mon.noname-a con 0x7fade00a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:37.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 10 monclient(hunting): picked mon.noname-b con 0x7fade00a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:37.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade00a3250 con 0x7fade00a0000 2024-09-16T14:49:37.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade009f600 con 0x7fade00a1d10 2024-09-16T14:49:37.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade00a3830 con 0x7fade00a2800 2024-09-16T14:49:37.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:37.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 10 monclient(hunting): authenticate will time out at 1401.446913s 2024-09-16T14:49:37.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddf7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fade00a0000 0x7fade0139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:45532/0 (socket says 172.21.15.26:45532) 2024-09-16T14:49:37.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddffff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fade00a1d10 legacy=0x7fade00a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:37.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddf7fe640 1 -- 172.21.15.26:0/2686259191 learned_addr learned my addr 172.21.15.26:0/2686259191 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:37.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade4ea1640 1 --1- 172.21.15.26:0/2686259191 >> v1:172.21.15.26:6790/0 conn(0x7fade00a2800 0x7fade013d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43736/0 (socket says 172.21.15.26:43736) 2024-09-16T14:49:37.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 1 -- 172.21.15.26:0/2686259191 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208853407 0 0) 0x7fade00a3830 con 0x7fade00a2800 2024-09-16T14:49:37.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:37.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:37.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:37.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 10 monclient(hunting): my global_id is 154409 2024-09-16T14:49:37.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 1 -- 172.21.15.26:0/2686259191 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadcc003530 con 0x7fade00a2800 2024-09-16T14:49:37.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 1 -- 172.21.15.26:0/2686259191 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1106872055 0 0) 0x7fadcc003530 con 0x7fade00a2800 2024-09-16T14:49:37.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 1 -- 172.21.15.26:0/2686259191 >> v1:172.21.15.189:6789/0 conn(0x7fade00a1d10 legacy=0x7fade00a20f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:37.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 1 -- 172.21.15.26:0/2686259191 >> v1:172.21.15.26:6789/0 conn(0x7fade00a0000 legacy=0x7fade0139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:37.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 1 monclient: found mon.noname-b 2024-09-16T14:49:37.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:37.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:49:37.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 1 -- 172.21.15.26:0/2686259191 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade009ed60 con 0x7fade00a2800 2024-09-16T14:49:37.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 10 monclient: _finish_auth 0 2024-09-16T14:49:37.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:37.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:37.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 5 monclient: authenticate success, global_id 154409 2024-09-16T14:49:37.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 1 -- 172.21.15.26:0/2686259191 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fadd0001550 con 0x7fade00a2800 2024-09-16T14:49:37.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:37.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:37.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:37.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 10 monclient: dump: 2024-09-16T14:49:37.332 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:37.332 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:37.332 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.332 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.332 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:37.332 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:37.333 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:37.333 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:37.333 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:37.333 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:37.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:37.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 1 -- 172.21.15.26:0/2686259191 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadd0001870 con 0x7fade00a2800 2024-09-16T14:49:37.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:37.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:37.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 1 -- 172.21.15.26:0/2686259191 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fadd0001c40 con 0x7fade00a2800 2024-09-16T14:49:37.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7fade6904640 4 set_mon_vals no callback set 2024-09-16T14:49:37.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:37.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:37.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.322+0000 7faddeffd640 10 monclient: dump: 2024-09-16T14:49:37.334 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:37.334 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:37.334 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.334 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.334 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:37.334 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:37.334 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:37.334 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:37.334 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:37.334 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:37.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 10 monclient: shutdown 2024-09-16T14:49:37.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:37.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 1 -- 172.21.15.26:0/2686259191 >> v1:172.21.15.26:6790/0 conn(0x7fade00a2800 legacy=0x7fade013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 1 -- 172.21.15.26:0/2686259191 shutdown_connections 2024-09-16T14:49:37.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 1 -- 172.21.15.26:0/2686259191 >> 172.21.15.26:0/2686259191 conn(0x7fade00a7f30 msgr2=0x7fade00a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:37.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 1 -- 172.21.15.26:0/2686259191 shutdown_connections 2024-09-16T14:49:37.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 1 -- 172.21.15.26:0/2686259191 wait complete. 2024-09-16T14:49:37.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 10 monclient: build_initial_monmap 2024-09-16T14:49:37.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 10 monclient: monmap: 2024-09-16T14:49:37.339 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:37.339 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:37.339 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:37.339 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:37.339 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:37.339 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:37.339 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:37.339 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:37.339 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:37.339 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:37.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 1 Processor -- start 2024-09-16T14:49:37.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 1 -- start start 2024-09-16T14:49:37.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 10 monclient: init 2024-09-16T14:49:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 10 monclient(hunting): picked mon.noname-a con 0x7fade00a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 10 monclient(hunting): picked mon.noname-b con 0x7fade00a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 10 monclient(hunting): picked mon.noname-c con 0x7fade00a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade0133460 con 0x7fade00a0000 2024-09-16T14:49:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade00a6560 con 0x7fade00a1d10 2024-09-16T14:49:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade01320f0 con 0x7fade00a2800 2024-09-16T14:49:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 10 monclient(hunting): authenticate will time out at 1401.450542s 2024-09-16T14:49:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fade00a0000 0x7fade012ff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:45542/0 (socket says 172.21.15.26:45542) 2024-09-16T14:49:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddffff640 1 -- 172.21.15.26:0/4254792964 learned_addr learned my addr 172.21.15.26:0/4254792964 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:37.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddf7fe640 1 -- 172.21.15.26:0/4254792964 >> v1:172.21.15.189:6789/0 conn(0x7fade00a1d10 legacy=0x7fade0130630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 1 -- 172.21.15.26:0/4254792964 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081788152 0 0) 0x7fade01320f0 con 0x7fade00a2800 2024-09-16T14:49:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 10 monclient(hunting): _negotiate 2024-09-16T14:49:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 10 monclient(hunting): my global_id is 154412 2024-09-16T14:49:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 1 -- 172.21.15.26:0/4254792964 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc0003560 con 0x7fade00a2800 2024-09-16T14:49:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 1 -- 172.21.15.26:0/4254792964 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2685186398 0 0) 0x7fadc0003560 con 0x7fade00a2800 2024-09-16T14:49:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 1 -- 172.21.15.26:0/4254792964 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fade01320f0 con 0x7fade00a2800 2024-09-16T14:49:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 1 -- 172.21.15.26:0/4254792964 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fadd0001510 con 0x7fade00a2800 2024-09-16T14:49:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:37.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 10 monclient(hunting): dump: 2024-09-16T14:49:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:37.342 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:37.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 1 -- 172.21.15.26:0/4254792964 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3616585525 0 0) 0x7fade01320f0 con 0x7fade00a2800 2024-09-16T14:49:37.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 1 -- 172.21.15.26:0/4254792964 >> v1:172.21.15.189:6789/0 conn(0x7fade00a1d10 legacy=0x7fade0130630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:37.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 1 -- 172.21.15.26:0/4254792964 >> v1:172.21.15.26:6789/0 conn(0x7fade00a0000 legacy=0x7fade012ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 10 monclient: _finish_hunting 0 2024-09-16T14:49:37.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 1 monclient: found mon.c 2024-09-16T14:49:37.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:37.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:37.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 1 -- 172.21.15.26:0/4254792964 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade0130d40 con 0x7fade00a2800 2024-09-16T14:49:37.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 10 monclient: _finish_auth 0 2024-09-16T14:49:37.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 10 monclient: _check_auth_tickets 2024-09-16T14:49:37.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:37.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 5 monclient: authenticate success, global_id 154412 2024-09-16T14:49:37.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 10 monclient: _renew_subs 2024-09-16T14:49:37.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:37.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 1 -- 172.21.15.26:0/4254792964 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade009e500 con 0x7fade00a2800 2024-09-16T14:49:37.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 10 monclient: _renew_subs 2024-09-16T14:49:37.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:37.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fade6904640 1 -- 172.21.15.26:0/4254792964 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fade0131950 con 0x7fade00a2800 2024-09-16T14:49:37.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 1 -- 172.21.15.26:0/4254792964 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadd0004b40 con 0x7fade00a2800 2024-09-16T14:49:37.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:37.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 1 -- 172.21.15.26:0/4254792964 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fadd0004ed0 con 0x7fade00a2800 2024-09-16T14:49:37.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:37.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:37.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7fadde7fc640 4 set_mon_vals no callback set 2024-09-16T14:49:37.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.326+0000 7faddcff9640 10 monclient: dump: 2024-09-16T14:49:37.345 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:37.345 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:37.345 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.345 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:37.345 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:37.345 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:37.345 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:37.345 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:37.345 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:37.345 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:37.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.330+0000 7fade6904640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:37.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.330+0000 7fade6904640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:37.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.330+0000 7fade6904640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:37.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.330+0000 7faddcff9640 1 -- 172.21.15.26:0/4254792964 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fadd00050b0 con 0x7fade00a2800 2024-09-16T14:49:37.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.330+0000 7fade6904640 1 -- 172.21.15.26:0/4254792964 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc4005050 con 0x7fade00a2800 2024-09-16T14:49:37.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.330+0000 7faddcff9640 1 -- 172.21.15.26:0/4254792964 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(200..200 src has 169..200) v4 ==== 10877+0+0 (unknown 2031785554 0 0) 0x7fadd0044ff0 con 0x7fade00a2800 2024-09-16T14:49:37.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.330+0000 7faddffff640 10 monclient: get_auth_request con 0x7fadc0036060 auth_method 0 2024-09-16T14:49:37.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.334+0000 7faddcff9640 1 -- 172.21.15.26:0/4254792964 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fadd0014a40 con 0x7fade00a2800 2024-09-16T14:49:37.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.334+0000 7faddcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:37.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.334+0000 7faddcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:37.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.554+0000 7fade6904640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:49:37.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.554+0000 7fade6904640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:49:37.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.554+0000 7fade6904640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:37.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.554+0000 7fade6904640 1 -- 172.21.15.26:0/4254792964 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fadc4005880 con 0x7fade00a2800 2024-09-16T14:49:37.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.554+0000 7faddcff9640 1 -- 172.21.15.26:0/4254792964 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 800351602) 0x7fadd0012eb0 con 0x7fade00a2800 2024-09-16T14:49:37.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.554+0000 7faddcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:49:37.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.554+0000 7faddcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:37.558 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:49:37.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.554+0000 7fade6904640 1 -- 172.21.15.26:0/4254792964 >> v1:172.21.15.189:6812/3196161582 conn(0x7fadc0036060 legacy=0x7fadc0038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.554+0000 7fade6904640 10 monclient: shutdown 2024-09-16T14:49:37.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.554+0000 7fade6904640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:37.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.554+0000 7fade6904640 1 -- 172.21.15.26:0/4254792964 >> v1:172.21.15.26:6790/0 conn(0x7fade00a2800 legacy=0x7fade01d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:37.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.554+0000 7fade6904640 1 -- 172.21.15.26:0/4254792964 shutdown_connections 2024-09-16T14:49:37.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.554+0000 7fade6904640 1 -- 172.21.15.26:0/4254792964 >> 172.21.15.26:0/4254792964 conn(0x7fade00a7f30 msgr2=0x7fade006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:37.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.554+0000 7fade6904640 1 -- 172.21.15.26:0/4254792964 shutdown_connections 2024-09-16T14:49:37.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:37.554+0000 7fade6904640 1 -- 172.21.15.26:0/4254792964 wait complete. 2024-09-16T14:49:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:39.860+0000 7fc4867fc640 10 monclient: tick 2024-09-16T14:49:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:39.860+0000 7fc4867fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:39.860+0000 7fc4867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:39.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:39.860+0000 7fc4867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:40.576 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:49:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 10 monclient: get_monmap_and_config 2024-09-16T14:49:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 10 monclient: build_initial_monmap 2024-09-16T14:49:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 10 monclient: monmap: 2024-09-16T14:49:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:40.710892+0000 2024-09-16T14:49:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:40.710892+0000 2024-09-16T14:49:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:40.711 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:40.711 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:40.711 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:40.711 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:40.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 1 Processor -- start 2024-09-16T14:49:40.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 1 -- start start 2024-09-16T14:49:40.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 10 monclient: init 2024-09-16T14:49:40.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:40.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:49:40.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 10 monclient(hunting): picked mon.noname-a con 0x7f67bc09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:40.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 10 monclient(hunting): picked mon.noname-b con 0x7f67bc132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:40.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 10 monclient(hunting): picked mon.noname-c con 0x7f67bc138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:40.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67bc09d260 con 0x7f67bc09cb80 2024-09-16T14:49:40.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67bc132a80 con 0x7f67bc138ff0 2024-09-16T14:49:40.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67bc09cf60 con 0x7f67bc132650 2024-09-16T14:49:40.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:40.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 10 monclient(hunting): authenticate will time out at 1404.830966s 2024-09-16T14:49:40.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67bb7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f67bc132650 0x7f67bc1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43756/0 (socket says 172.21.15.26:43756) 2024-09-16T14:49:40.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67bb7fe640 1 -- 172.21.15.26:0/2915794698 learned_addr learned my addr 172.21.15.26:0/2915794698 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:40.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c0fff640 1 -- 172.21.15.26:0/2915794698 >> v1:172.21.15.189:6789/0 conn(0x7f67bc138ff0 legacy=0x7f67bc13b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:40.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 1 -- 172.21.15.26:0/2915794698 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340522478 0 0) 0x7f67bc09cf60 con 0x7f67bc132650 2024-09-16T14:49:40.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:40.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:40.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:40.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 10 monclient(hunting): my global_id is 154415 2024-09-16T14:49:40.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 1 -- 172.21.15.26:0/2915794698 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67a4003530 con 0x7f67bc132650 2024-09-16T14:49:40.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 1 -- 172.21.15.26:0/2915794698 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1955330170 0 0) 0x7f67a4003530 con 0x7f67bc132650 2024-09-16T14:49:40.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 1 -- 172.21.15.26:0/2915794698 >> v1:172.21.15.189:6789/0 conn(0x7f67bc138ff0 legacy=0x7f67bc13b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:40.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 1 -- 172.21.15.26:0/2915794698 >> v1:172.21.15.26:6789/0 conn(0x7f67bc09cb80 legacy=0x7f67bc0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:40.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:40.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 1 monclient: found mon.noname-b 2024-09-16T14:49:40.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:40.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:49:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 1 -- 172.21.15.26:0/2915794698 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67bc09f2d0 con 0x7f67bc132650 2024-09-16T14:49:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 10 monclient: _finish_auth 0 2024-09-16T14:49:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 1 -- 172.21.15.26:0/2915794698 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67b0001660 con 0x7f67bc132650 2024-09-16T14:49:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 10 monclient: dump: 2024-09-16T14:49:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:40.719 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:40.720 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:40.720 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:40.720 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:40.720 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:40.720 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:40.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 5 monclient: authenticate success, global_id 154415 2024-09-16T14:49:40.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:40.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 1 -- 172.21.15.26:0/2915794698 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67b00018c0 con 0x7f67bc132650 2024-09-16T14:49:40.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:40.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:40.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67c2a62640 4 set_mon_vals no callback set 2024-09-16T14:49:40.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 1 -- 172.21.15.26:0/2915794698 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67b0001c50 con 0x7f67bc132650 2024-09-16T14:49:40.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:40.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.706+0000 7f67baffd640 10 monclient: dump: 2024-09-16T14:49:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:40.721 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient: shutdown 2024-09-16T14:49:40.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:40.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 1 -- 172.21.15.26:0/2915794698 >> v1:172.21.15.26:6790/0 conn(0x7f67bc132650 legacy=0x7f67bc1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:40.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 1 -- 172.21.15.26:0/2915794698 shutdown_connections 2024-09-16T14:49:40.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 1 -- 172.21.15.26:0/2915794698 >> 172.21.15.26:0/2915794698 conn(0x7f67bc09fb50 msgr2=0x7f67bc09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:40.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 1 -- 172.21.15.26:0/2915794698 shutdown_connections 2024-09-16T14:49:40.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 1 -- 172.21.15.26:0/2915794698 wait complete. 2024-09-16T14:49:40.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient: build_initial_monmap 2024-09-16T14:49:40.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient: monmap: 2024-09-16T14:49:40.722 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:40.722 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:40.722 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:40.722 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:40.722 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:40.722 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:40.723 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 1 Processor -- start 2024-09-16T14:49:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 1 -- start start 2024-09-16T14:49:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient: init 2024-09-16T14:49:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient(hunting): picked mon.noname-c con 0x7f67bc09cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient(hunting): picked mon.noname-a con 0x7f67bc132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient(hunting): picked mon.noname-b con 0x7f67bc138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:40.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67bc0a2250 con 0x7f67bc132650 2024-09-16T14:49:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67bc1381c0 con 0x7f67bc138ff0 2024-09-16T14:49:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67bc13c570 con 0x7f67bc09cb80 2024-09-16T14:49:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient(hunting): authenticate will time out at 1404.833954s 2024-09-16T14:49:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67bb7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f67bc132650 0x7f67bc06dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:45562/0 (socket says 172.21.15.26:45562) 2024-09-16T14:49:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67bb7fe640 1 -- 172.21.15.26:0/1510169932 learned_addr learned my addr 172.21.15.26:0/1510169932 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c0fff640 1 -- 172.21.15.26:0/1510169932 >> v1:172.21.15.189:6789/0 conn(0x7f67bc138ff0 legacy=0x7f67bc06e420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 1 -- 172.21.15.26:0/1510169932 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368837002 0 0) 0x7f67bc13c570 con 0x7f67bc09cb80 2024-09-16T14:49:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:49:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 10 monclient(hunting): my global_id is 154418 2024-09-16T14:49:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 1 -- 172.21.15.26:0/1510169932 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67a8003530 con 0x7f67bc09cb80 2024-09-16T14:49:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 1 -- 172.21.15.26:0/1510169932 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 612748455 0 0) 0x7f67a8003530 con 0x7f67bc09cb80 2024-09-16T14:49:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 1 -- 172.21.15.26:0/1510169932 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67bc13c570 con 0x7f67bc09cb80 2024-09-16T14:49:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 1 -- 172.21.15.26:0/1510169932 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67b4001a10 con 0x7f67bc09cb80 2024-09-16T14:49:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 10 monclient(hunting): dump: 2024-09-16T14:49:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:40.725 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:40.726 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 1 -- 172.21.15.26:0/1510169932 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1168530330 0 0) 0x7f67bc13c570 con 0x7f67bc09cb80 2024-09-16T14:49:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 1 -- 172.21.15.26:0/1510169932 >> v1:172.21.15.189:6789/0 conn(0x7f67bc138ff0 legacy=0x7f67bc06e420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 1 -- 172.21.15.26:0/1510169932 >> v1:172.21.15.26:6789/0 conn(0x7f67bc132650 legacy=0x7f67bc06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:49:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 1 monclient: found mon.c 2024-09-16T14:49:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:40.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 1 -- 172.21.15.26:0/1510169932 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67bc09a240 con 0x7f67bc09cb80 2024-09-16T14:49:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 10 monclient: _finish_auth 0 2024-09-16T14:49:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:49:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 5 monclient: authenticate success, global_id 154418 2024-09-16T14:49:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient: _renew_subs 2024-09-16T14:49:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 1 -- 172.21.15.26:0/1510169932 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67bc0712d0 con 0x7f67bc09cb80 2024-09-16T14:49:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient: _renew_subs 2024-09-16T14:49:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 1 -- 172.21.15.26:0/1510169932 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67bc06f250 con 0x7f67bc09cb80 2024-09-16T14:49:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 1 -- 172.21.15.26:0/1510169932 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67b4002080 con 0x7f67bc09cb80 2024-09-16T14:49:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:40.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 1 -- 172.21.15.26:0/1510169932 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67b4003b40 con 0x7f67bc09cb80 2024-09-16T14:49:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67ba7fc640 4 set_mon_vals no callback set 2024-09-16T14:49:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 10 monclient: dump: 2024-09-16T14:49:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:40.728 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:40.729 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:40.729 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:40.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67b8ff9640 1 -- 172.21.15.26:0/1510169932 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f67b4003d20 con 0x7f67bc09cb80 2024-09-16T14:49:40.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:40.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:40.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:40.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.710+0000 7f67c2a62640 1 -- 172.21.15.26:0/1510169932 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679c005050 con 0x7f67bc09cb80 2024-09-16T14:49:40.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.714+0000 7f67b8ff9640 1 -- 172.21.15.26:0/1510169932 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(200..200 src has 169..200) v4 ==== 10877+0+0 (unknown 2031785554 0 0) 0x7f67b4043e70 con 0x7f67bc09cb80 2024-09-16T14:49:40.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.714+0000 7f67bb7fe640 10 monclient: get_auth_request con 0x7f67a8036050 auth_method 0 2024-09-16T14:49:40.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.718+0000 7f67b8ff9640 1 -- 172.21.15.26:0/1510169932 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f67b40134f0 con 0x7f67bc09cb80 2024-09-16T14:49:40.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.718+0000 7f67b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:40.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.718+0000 7f67b8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:40.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:40.860+0000 7fc4867fc640 10 monclient: tick 2024-09-16T14:49:40.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:40.860+0000 7fc4867fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:40.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:40.860+0000 7fc4867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.934+0000 7f67c2a62640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.934+0000 7f67c2a62640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.934+0000 7f67c2a62640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.934+0000 7f67c2a62640 1 -- 172.21.15.26:0/1510169932 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f679c005880 con 0x7f67bc09cb80 2024-09-16T14:49:40.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.934+0000 7f67b8ff9640 1 -- 172.21.15.26:0/1510169932 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2136517459) 0x7f67b4013310 con 0x7f67bc09cb80 2024-09-16T14:49:40.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.934+0000 7f67b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:49:40.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.934+0000 7f67b8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:40.940 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:49:40.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.938+0000 7f67c2a62640 1 -- 172.21.15.26:0/1510169932 >> v1:172.21.15.189:6812/3196161582 conn(0x7f67a8036050 legacy=0x7f67a8038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:40.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.938+0000 7f67c2a62640 10 monclient: shutdown 2024-09-16T14:49:40.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.938+0000 7f67c2a62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:40.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.938+0000 7f67c2a62640 1 -- 172.21.15.26:0/1510169932 >> v1:172.21.15.26:6790/0 conn(0x7f67bc09cb80 legacy=0x7f67bc06d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:40.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.938+0000 7f67c2a62640 1 -- 172.21.15.26:0/1510169932 shutdown_connections 2024-09-16T14:49:40.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.938+0000 7f67c2a62640 1 -- 172.21.15.26:0/1510169932 >> 172.21.15.26:0/1510169932 conn(0x7f67bc09fb50 msgr2=0x7f67bc0a0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:40.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.938+0000 7f67c2a62640 1 -- 172.21.15.26:0/1510169932 shutdown_connections 2024-09-16T14:49:40.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:40.938+0000 7f67c2a62640 1 -- 172.21.15.26:0/1510169932 wait complete. 2024-09-16T14:49:41.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:41.860+0000 7fc4867fc640 10 monclient: tick 2024-09-16T14:49:41.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:41.860+0000 7fc4867fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:41.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:41.860+0000 7fc4867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:42.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:42.864+0000 7fc4867fc640 10 monclient: tick 2024-09-16T14:49:42.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:42.864+0000 7fc4867fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:42.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:42.864+0000 7fc4867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:43.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:43.864+0000 7fc4867fc640 10 monclient: tick 2024-09-16T14:49:43.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:43.864+0000 7fc4867fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:43.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:43.864+0000 7fc4867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:43.958 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:49:44.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38af681640 10 monclient: get_monmap_and_config 2024-09-16T14:49:44.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38af681640 10 monclient: build_initial_monmap 2024-09-16T14:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38af681640 10 monclient: monmap: 2024-09-16T14:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:44.090632+0000 2024-09-16T14:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:44.090632+0000 2024-09-16T14:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:44.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38af681640 1 Processor -- start 2024-09-16T14:49:44.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38af681640 1 -- start start 2024-09-16T14:49:44.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38af681640 10 monclient: init 2024-09-16T14:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38af681640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38af681640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38af681640 10 monclient(hunting): picked mon.noname-b con 0x7f38a809a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38af681640 10 monclient(hunting): picked mon.noname-c con 0x7f38a809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38af681640 10 monclient(hunting): picked mon.noname-a con 0x7f38a809b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38af681640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a809dc60 con 0x7f38a809b040 2024-09-16T14:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38af681640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a8099ae0 con 0x7f38a809d1b0 2024-09-16T14:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38af681640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a809b8f0 con 0x7f38a809a550 2024-09-16T14:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38af681640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38af681640 10 monclient(hunting): authenticate will time out at 1408.210755s 2024-09-16T14:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38ad41d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f38a809a550 0x7f38a809a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43774/0 (socket says 172.21.15.26:43774) 2024-09-16T14:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38ad41d640 1 -- 172.21.15.26:0/651573825 learned_addr learned my addr 172.21.15.26:0/651573825 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.086+0000 7f38acc1c640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.189:6789/0 conn(0x7f38a809d1b0 legacy=0x7f38a8136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:44.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.290+0000 7f38acc1c640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.189:6789/0 conn(0x7f38a809d1b0 legacy=0x7f38a8136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:44.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:44.694+0000 7f38acc1c640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.189:6789/0 conn(0x7f38a809d1b0 legacy=0x7f38a8136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:44.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:44.860+0000 7fc4867fc640 10 monclient: tick 2024-09-16T14:49:44.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:44.860+0000 7fc4867fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:44.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:44.860+0000 7fc4867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:45.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:45.498+0000 7f38acc1c640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.189:6789/0 conn(0x7f38a809d1b0 legacy=0x7f38a8136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:45.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:45.860+0000 7fc4867fc640 10 monclient: tick 2024-09-16T14:49:45.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:45.860+0000 7fc4867fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:45.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:45.860+0000 7fc4867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:46.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:46.860+0000 7fc4867fc640 10 monclient: tick 2024-09-16T14:49:46.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:46.860+0000 7fc4867fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:46.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:46.860+0000 7fc4867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:47.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.086+0000 7f389affd640 10 monclient(hunting): tick 2024-09-16T14:49:47.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.086+0000 7f389affd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:49:47.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.086+0000 7f389affd640 1 monclient(hunting): continuing hunt 2024-09-16T14:49:47.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.086+0000 7f389affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:49:47.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.086+0000 7f389affd640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.26:6790/0 conn(0x7f38a809a550 legacy=0x7f38a809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:47.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.086+0000 7f389affd640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.189:6789/0 conn(0x7f38a809d1b0 legacy=0x7f38a8136a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:47.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.086+0000 7f389affd640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.26:6789/0 conn(0x7f38a809b040 legacy=0x7f38a813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:47.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.086+0000 7f389affd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:47.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.086+0000 7f389affd640 10 monclient(hunting): picked mon.noname-b con 0x7f388c0036d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:47.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.086+0000 7f389affd640 10 monclient(hunting): picked mon.noname-a con 0x7f388c007360 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:47.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.086+0000 7f389affd640 10 monclient(hunting): picked mon.noname-c con 0x7f38a809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:47.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.086+0000 7f389affd640 1 -- 172.21.15.26:0/651573825 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a809bbd0 con 0x7f388c007360 2024-09-16T14:49:47.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.086+0000 7f389affd640 1 -- 172.21.15.26:0/651573825 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a8099ae0 con 0x7f38a809d1b0 2024-09-16T14:49:47.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.086+0000 7f389affd640 1 -- 172.21.15.26:0/651573825 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388c0108e0 con 0x7f388c0036d0 2024-09-16T14:49:47.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.086+0000 7f389affd640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:47.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.086+0000 7f38adc1e640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.189:6789/0 conn(0x7f38a809d1b0 legacy=0x7f388c00d110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:47.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.290+0000 7f38adc1e640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.189:6789/0 conn(0x7f38a809d1b0 legacy=0x7f388c00d110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:47.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:47.694+0000 7f38adc1e640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.189:6789/0 conn(0x7f38a809d1b0 legacy=0x7f388c00d110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:47.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:47.860+0000 7fc4867fc640 10 monclient: tick 2024-09-16T14:49:47.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:47.860+0000 7fc4867fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:47.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:47.860+0000 7fc4867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:48.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:48.498+0000 7f38adc1e640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.189:6789/0 conn(0x7f38a809d1b0 legacy=0x7f388c00d110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:48.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:48.864+0000 7fc4867fc640 10 monclient: tick 2024-09-16T14:49:48.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:48.864+0000 7fc4867fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:48.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:48.864+0000 7fc4867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:49.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:49.864+0000 7fc4867fc640 10 monclient: tick 2024-09-16T14:49:49.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:49.864+0000 7fc4867fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:49.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:49.864+0000 7fc4867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:49.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:49.864+0000 7fc4867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:50.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.086+0000 7f389affd640 10 monclient(hunting): tick 2024-09-16T14:49:50.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.086+0000 7f389affd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:49:50.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.086+0000 7f389affd640 1 monclient(hunting): continuing hunt 2024-09-16T14:49:50.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.086+0000 7f389affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:49:50.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.086+0000 7f389affd640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.26:6790/0 conn(0x7f388c0036d0 legacy=0x7f388c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:50.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.086+0000 7f389affd640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.189:6789/0 conn(0x7f38a809d1b0 legacy=0x7f388c00d110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.086+0000 7f389affd640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.26:6789/0 conn(0x7f388c007360 legacy=0x7f388c0097d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.086+0000 7f389affd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.086+0000 7f389affd640 10 monclient(hunting): picked mon.noname-c con 0x7f388c00e980 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.086+0000 7f389affd640 10 monclient(hunting): picked mon.noname-a con 0x7f388c0112a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.086+0000 7f389affd640 10 monclient(hunting): picked mon.noname-b con 0x7f388c00fdd0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.086+0000 7f389affd640 1 -- 172.21.15.26:0/651573825 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388c011af0 con 0x7f388c0112a0 2024-09-16T14:49:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.086+0000 7f389affd640 1 -- 172.21.15.26:0/651573825 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a8099ae0 con 0x7f388c00e980 2024-09-16T14:49:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.086+0000 7f389affd640 1 -- 172.21.15.26:0/651573825 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388c010250 con 0x7f388c00fdd0 2024-09-16T14:49:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.086+0000 7f389affd640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:50.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.086+0000 7f38ad41d640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.189:6789/0 conn(0x7f388c00e980 legacy=0x7f388c010a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:50.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.290+0000 7f38ad41d640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.189:6789/0 conn(0x7f388c00e980 legacy=0x7f388c010a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:50.694+0000 7f38ad41d640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.189:6789/0 conn(0x7f388c00e980 legacy=0x7f388c010a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:50.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:50.864+0000 7fc4867fc640 10 monclient: tick 2024-09-16T14:49:50.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:50.864+0000 7fc4867fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:50.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:50.864+0000 7fc4867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:51.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.498+0000 7f38ad41d640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.189:6789/0 conn(0x7f388c00e980 legacy=0x7f388c010a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:51.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 1 -- 172.21.15.26:0/651573825 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273193197 0 0) 0x7f388c011af0 con 0x7f388c0112a0 2024-09-16T14:49:51.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:51.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:51.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:51.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 10 monclient(hunting): my global_id is 164169 2024-09-16T14:49:51.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 1 -- 172.21.15.26:0/651573825 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3894003530 con 0x7f388c0112a0 2024-09-16T14:49:51.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 1 -- 172.21.15.26:0/651573825 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2103092691 0 0) 0x7f3894003530 con 0x7f388c0112a0 2024-09-16T14:49:51.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.26:6790/0 conn(0x7f388c00fdd0 legacy=0x7f388c017f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:51.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.189:6789/0 conn(0x7f388c00e980 legacy=0x7f388c010a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:51.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:49:51.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 1 monclient: found mon.noname-a 2024-09-16T14:49:51.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:51.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:49:51.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 1 -- 172.21.15.26:0/651573825 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c0103e0 con 0x7f388c0112a0 2024-09-16T14:49:51.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 10 monclient: _finish_auth 0 2024-09-16T14:49:51.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:49:51.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:51.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 1 -- 172.21.15.26:0/651573825 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f389c0041b0 con 0x7f388c0112a0 2024-09-16T14:49:51.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:51.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:51.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 10 monclient: dump: 2024-09-16T14:49:51.652 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:51.652 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:51.652 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:51.652 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:51.653 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:51.653 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:51.653 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:51.653 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:51.653 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:51.653 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:51.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f38af681640 5 monclient: authenticate success, global_id 164169 2024-09-16T14:49:51.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f38af681640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:51.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 1 -- 172.21.15.26:0/651573825 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f389c004450 con 0x7f388c0112a0 2024-09-16T14:49:51.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:51.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f38af681640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:51.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f38af681640 4 set_mon_vals no callback set 2024-09-16T14:49:51.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 1 -- 172.21.15.26:0/651573825 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f389c0047e0 con 0x7f388c0112a0 2024-09-16T14:49:51.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:51.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:51.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f389bfff640 10 monclient: dump: 2024-09-16T14:49:51.654 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:51.654 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:51.654 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:51.654 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:51.654 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:51.654 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:51.654 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:51.654 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:51.654 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:51.654 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:51.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f38af681640 10 monclient: shutdown 2024-09-16T14:49:51.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f38af681640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:51.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f38af681640 1 -- 172.21.15.26:0/651573825 >> v1:172.21.15.26:6789/0 conn(0x7f388c0112a0 legacy=0x7f388c00d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:51.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f38af681640 1 -- 172.21.15.26:0/651573825 shutdown_connections 2024-09-16T14:49:51.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f38af681640 1 -- 172.21.15.26:0/651573825 >> 172.21.15.26:0/651573825 conn(0x7f38a809fb50 msgr2=0x7f38a809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:51.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f38af681640 1 -- 172.21.15.26:0/651573825 shutdown_connections 2024-09-16T14:49:51.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f38af681640 1 -- 172.21.15.26:0/651573825 wait complete. 2024-09-16T14:49:51.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.646+0000 7f38af681640 10 monclient: build_initial_monmap 2024-09-16T14:49:51.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 10 monclient: monmap: 2024-09-16T14:49:51.660 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:51.660 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:51.660 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:51.660 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:51.660 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:51.661 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:51.661 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:51.661 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:51.661 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:51.661 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:51.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 1 Processor -- start 2024-09-16T14:49:51.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 1 -- start start 2024-09-16T14:49:51.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 10 monclient: init 2024-09-16T14:49:51.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:51.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:49:51.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 10 monclient(hunting): picked mon.noname-b con 0x7f388c00e980 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:51.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 10 monclient(hunting): picked mon.noname-c con 0x7f388c00fdd0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:51.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 10 monclient(hunting): picked mon.noname-a con 0x7f388c0112a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:51.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a80a2640 con 0x7f388c0112a0 2024-09-16T14:49:51.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a809a240 con 0x7f388c00e980 2024-09-16T14:49:51.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a81a5630 con 0x7f388c00fdd0 2024-09-16T14:49:51.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:51.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 10 monclient(hunting): authenticate will time out at 1415.773577s 2024-09-16T14:49:51.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38acc1c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f388c00fdd0 0x7f38a81ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55334/0 (socket says 172.21.15.26:55334) 2024-09-16T14:49:51.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38acc1c640 1 -- 172.21.15.26:0/2469833855 learned_addr learned my addr 172.21.15.26:0/2469833855 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:51.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38ad41d640 1 -- 172.21.15.26:0/2469833855 >> v1:172.21.15.189:6789/0 conn(0x7f388c00e980 legacy=0x7f38a81b4a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:51.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 1 -- 172.21.15.26:0/2469833855 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776381457 0 0) 0x7f38a80a2640 con 0x7f388c0112a0 2024-09-16T14:49:51.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:51.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:51.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:51.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 10 monclient(hunting): my global_id is 164172 2024-09-16T14:49:51.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 1 -- 172.21.15.26:0/2469833855 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f388c011a00 con 0x7f388c0112a0 2024-09-16T14:49:51.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 1 -- 172.21.15.26:0/2469833855 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1013778886 0 0) 0x7f388c011a00 con 0x7f388c0112a0 2024-09-16T14:49:51.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 1 -- 172.21.15.26:0/2469833855 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38a80a2640 con 0x7f388c0112a0 2024-09-16T14:49:51.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 1 -- 172.21.15.26:0/2469833855 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f38a0001bf0 con 0x7f388c0112a0 2024-09-16T14:49:51.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:51.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:51.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 10 monclient(hunting): dump: 2024-09-16T14:49:51.663 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:51.663 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:51.663 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:51.663 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:51.663 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:51.664 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:51.664 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:51.664 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:51.664 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:51.664 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:51.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:51.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 1 -- 172.21.15.26:0/2469833855 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1338609154 0 0) 0x7f38a80a2640 con 0x7f388c0112a0 2024-09-16T14:49:51.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 1 -- 172.21.15.26:0/2469833855 >> v1:172.21.15.26:6790/0 conn(0x7f388c00fdd0 legacy=0x7f38a81ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:51.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 1 -- 172.21.15.26:0/2469833855 >> v1:172.21.15.189:6789/0 conn(0x7f388c00e980 legacy=0x7f38a81b4a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:51.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:49:51.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 1 monclient: found mon.a 2024-09-16T14:49:51.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:51.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:51.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 1 -- 172.21.15.26:0/2469833855 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a81a5e80 con 0x7f388c0112a0 2024-09-16T14:49:51.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 10 monclient: _finish_auth 0 2024-09-16T14:49:51.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:49:51.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:51.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 5 monclient: authenticate success, global_id 164172 2024-09-16T14:49:51.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 10 monclient: _renew_subs 2024-09-16T14:49:51.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:51.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 1 -- 172.21.15.26:0/2469833855 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a81b5520 con 0x7f388c0112a0 2024-09-16T14:49:51.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 10 monclient: _renew_subs 2024-09-16T14:49:51.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:51.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f38af681640 1 -- 172.21.15.26:0/2469833855 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a81a5120 con 0x7f388c0112a0 2024-09-16T14:49:51.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 1 -- 172.21.15.26:0/2469833855 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38a0003980 con 0x7f388c0112a0 2024-09-16T14:49:51.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:51.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 1 -- 172.21.15.26:0/2469833855 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f38a0003d10 con 0x7f388c0112a0 2024-09-16T14:49:51.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:51.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:51.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f389b7fe640 4 set_mon_vals no callback set 2024-09-16T14:49:51.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 10 monclient: dump: 2024-09-16T14:49:51.666 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:51.666 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:51.666 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:51.666 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:51.666 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:51.666 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:51.666 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:51.666 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:51.666 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:51.667 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:51.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 1 -- 172.21.15.26:0/2469833855 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f38a0003ef0 con 0x7f388c0112a0 2024-09-16T14:49:51.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.650+0000 7f3899ffb640 1 -- 172.21.15.26:0/2469833855 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(200..200 src has 169..200) v4 ==== 10877+0+0 (unknown 2031785554 0 0) 0x7f38a0044f70 con 0x7f388c0112a0 2024-09-16T14:49:51.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.654+0000 7f38af681640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:51.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.654+0000 7f38af681640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:51.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.654+0000 7f38af681640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:51.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.654+0000 7f38af681640 1 -- 172.21.15.26:0/2469833855 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3888005050 con 0x7f388c0112a0 2024-09-16T14:49:51.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.654+0000 7f38ad41d640 10 monclient: get_auth_request con 0x7f388c037e40 auth_method 0 2024-09-16T14:49:51.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.658+0000 7f3899ffb640 1 -- 172.21.15.26:0/2469833855 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f38a0014930 con 0x7f388c0112a0 2024-09-16T14:49:51.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.658+0000 7f3899ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:51.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.658+0000 7f3899ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:51.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:51.864+0000 7fc4867fc640 10 monclient: tick 2024-09-16T14:49:51.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:51.864+0000 7fc4867fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:51.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:51.864+0000 7fc4867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.882+0000 7f38af681640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:49:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.882+0000 7f38af681640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:49:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.882+0000 7f38af681640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.882+0000 7f38af681640 1 -- 172.21.15.26:0/2469833855 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3888005880 con 0x7f388c0112a0 2024-09-16T14:49:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.882+0000 7f3899ffb640 1 -- 172.21.15.26:0/2469833855 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3748714124) 0x7f38a0012da0 con 0x7f388c0112a0 2024-09-16T14:49:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.882+0000 7f3899ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:49:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.882+0000 7f3899ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:51.886 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":70,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:49:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.882+0000 7f38af681640 1 -- 172.21.15.26:0/2469833855 >> v1:172.21.15.189:6812/3196161582 conn(0x7f388c037e40 legacy=0x7f388c03a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.882+0000 7f38af681640 10 monclient: shutdown 2024-09-16T14:49:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.882+0000 7f38af681640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.882+0000 7f38af681640 1 -- 172.21.15.26:0/2469833855 >> v1:172.21.15.26:6789/0 conn(0x7f388c0112a0 legacy=0x7f38a81d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.882+0000 7f38af681640 1 -- 172.21.15.26:0/2469833855 shutdown_connections 2024-09-16T14:49:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.882+0000 7f38af681640 1 -- 172.21.15.26:0/2469833855 >> 172.21.15.26:0/2469833855 conn(0x7f38a809fb50 msgr2=0x7f38a80a1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.882+0000 7f38af681640 1 -- 172.21.15.26:0/2469833855 shutdown_connections 2024-09-16T14:49:51.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:51.882+0000 7f38af681640 1 -- 172.21.15.26:0/2469833855 wait complete. 2024-09-16T14:49:51.904 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:49:51.905 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T14:49:51.905 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:49:52.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 10 monclient: get_monmap_and_config 2024-09-16T14:49:52.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 10 monclient: build_initial_monmap 2024-09-16T14:49:52.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 10 monclient: monmap: 2024-09-16T14:49:52.025 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:52.025 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.025 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:52.025815+0000 2024-09-16T14:49:52.025 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:52.025815+0000 2024-09-16T14:49:52.025 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:52.025 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:52.025 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:52.025 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:52.026 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:52.026 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 1 Processor -- start 2024-09-16T14:49:52.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 1 -- start start 2024-09-16T14:49:52.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 10 monclient: init 2024-09-16T14:49:52.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:52.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:52.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 10 monclient(hunting): picked mon.noname-a con 0x7f6d0809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:52.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 10 monclient(hunting): picked mon.noname-c con 0x7f6d08132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 10 monclient(hunting): picked mon.noname-b con 0x7f6d08138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d0809d260 con 0x7f6d0809cb80 2024-09-16T14:49:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d08132a80 con 0x7f6d08132650 2024-09-16T14:49:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d0809cf60 con 0x7f6d08138ff0 2024-09-16T14:49:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 10 monclient(hunting): authenticate will time out at 1416.145908s 2024-09-16T14:49:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d067fc640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f6d08132650 legacy=0x7f6d081378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d06ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6d0809cb80 0x7f6d080717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40080/0 (socket says 172.21.15.26:40080) 2024-09-16T14:49:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d06ffd640 1 -- 172.21.15.26:0/4041447915 learned_addr learned my addr 172.21.15.26:0/4041447915 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d077fe640 1 --1- 172.21.15.26:0/4041447915 >> v1:172.21.15.26:6790/0 conn(0x7f6d08138ff0 0x7f6d0813b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55350/0 (socket says 172.21.15.26:55350) 2024-09-16T14:49:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 1 -- 172.21.15.26:0/4041447915 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272371066 0 0) 0x7f6d0809cf60 con 0x7f6d08138ff0 2024-09-16T14:49:52.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient(hunting): my global_id is 154421 2024-09-16T14:49:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 1 -- 172.21.15.26:0/4041447915 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cf4003530 con 0x7f6d08138ff0 2024-09-16T14:49:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 1 -- 172.21.15.26:0/4041447915 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231094624 0 0) 0x7f6d0809d260 con 0x7f6d0809cb80 2024-09-16T14:49:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient(hunting): my global_id is 164175 2024-09-16T14:49:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 1 -- 172.21.15.26:0/4041447915 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d0809cf60 con 0x7f6d0809cb80 2024-09-16T14:49:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 1 -- 172.21.15.26:0/4041447915 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4157695436 0 0) 0x7f6cf4003530 con 0x7f6d08138ff0 2024-09-16T14:49:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 1 -- 172.21.15.26:0/4041447915 >> v1:172.21.15.189:6789/0 conn(0x7f6d08132650 legacy=0x7f6d081378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 1 -- 172.21.15.26:0/4041447915 >> v1:172.21.15.26:6789/0 conn(0x7f6d0809cb80 legacy=0x7f6d080717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:49:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 1 monclient: found mon.noname-b 2024-09-16T14:49:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:49:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 1 -- 172.21.15.26:0/4041447915 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d0809f2d0 con 0x7f6d08138ff0 2024-09-16T14:49:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient: _finish_auth 0 2024-09-16T14:49:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:49:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 1 -- 172.21.15.26:0/4041447915 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6d000019e0 con 0x7f6d08138ff0 2024-09-16T14:49:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient: dump: 2024-09-16T14:49:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:52.037 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.037 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.037 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.037 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.037 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:52.037 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.037 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.037 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.037 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 5 monclient: authenticate success, global_id 154421 2024-09-16T14:49:52.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:52.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 1 -- 172.21.15.26:0/4041447915 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d00001b80 con 0x7f6d08138ff0 2024-09-16T14:49:52.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 1 -- 172.21.15.26:0/4041447915 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6d00001f50 con 0x7f6d08138ff0 2024-09-16T14:49:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d05ffb640 10 monclient: dump: 2024-09-16T14:49:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.039 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.039 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:52.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 4 set_mon_vals no callback set 2024-09-16T14:49:52.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 10 monclient: shutdown 2024-09-16T14:49:52.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:52.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 1 -- 172.21.15.26:0/4041447915 >> v1:172.21.15.26:6790/0 conn(0x7f6d08138ff0 legacy=0x7f6d0813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 1 -- 172.21.15.26:0/4041447915 shutdown_connections 2024-09-16T14:49:52.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 1 -- 172.21.15.26:0/4041447915 >> 172.21.15.26:0/4041447915 conn(0x7f6d0809fb50 msgr2=0x7f6d0809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:52.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 1 -- 172.21.15.26:0/4041447915 shutdown_connections 2024-09-16T14:49:52.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.022+0000 7f6d0d3ac640 1 -- 172.21.15.26:0/4041447915 wait complete. 2024-09-16T14:49:52.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 10 monclient: build_initial_monmap 2024-09-16T14:49:52.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 10 monclient: monmap: 2024-09-16T14:49:52.040 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:52.040 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:52.040 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:52.040 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:52.040 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:52.040 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:52.040 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:52.040 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:52.040 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:52.040 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 1 Processor -- start 2024-09-16T14:49:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 1 -- start start 2024-09-16T14:49:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 10 monclient: init 2024-09-16T14:49:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:49:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 10 monclient(hunting): picked mon.noname-c con 0x7f6d0809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 10 monclient(hunting): picked mon.noname-b con 0x7f6d08132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 10 monclient(hunting): picked mon.noname-a con 0x7f6d08138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d0806f460 con 0x7f6d08138ff0 2024-09-16T14:49:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d0809a240 con 0x7f6d08132650 2024-09-16T14:49:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d08070ae0 con 0x7f6d0809cb80 2024-09-16T14:49:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 10 monclient(hunting): authenticate will time out at 1416.149104s 2024-09-16T14:49:52.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d077fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6d08138ff0 0x7f6d081cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40096/0 (socket says 172.21.15.26:40096) 2024-09-16T14:49:52.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d077fe640 1 -- 172.21.15.26:0/1366576386 learned_addr learned my addr 172.21.15.26:0/1366576386 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:52.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d06ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6d0809cb80 0x7f6d0806cdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55366/0 (socket says 172.21.15.26:55366) 2024-09-16T14:49:52.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d067fc640 1 -- 172.21.15.26:0/1366576386 >> v1:172.21.15.189:6789/0 conn(0x7f6d08132650 legacy=0x7f6d0806d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:52.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775803034 0 0) 0x7f6d08070ae0 con 0x7f6d0809cb80 2024-09-16T14:49:52.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:52.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:52.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:52.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): my global_id is 154424 2024-09-16T14:49:52.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ce8003530 con 0x7f6d0809cb80 2024-09-16T14:49:52.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537107369 0 0) 0x7f6d0806f460 con 0x7f6d08138ff0 2024-09-16T14:49:52.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:52.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:52.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): my global_id is 164178 2024-09-16T14:49:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d08070ae0 con 0x7f6d08138ff0 2024-09-16T14:49:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2657396587 0 0) 0x7f6ce8003530 con 0x7f6d0809cb80 2024-09-16T14:49:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d0806f460 con 0x7f6d0809cb80 2024-09-16T14:49:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1175079953 0 0) 0x7f6d08070ae0 con 0x7f6d08138ff0 2024-09-16T14:49:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ce8003530 con 0x7f6d08138ff0 2024-09-16T14:49:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6cf8001820 con 0x7f6d0809cb80 2024-09-16T14:49:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): dump: 2024-09-16T14:49:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.044 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6d000018e0 con 0x7f6d08138ff0 2024-09-16T14:49:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): dump: 2024-09-16T14:49:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.045 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3438993218 0 0) 0x7f6d0806f460 con 0x7f6d0809cb80 2024-09-16T14:49:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 >> v1:172.21.15.189:6789/0 conn(0x7f6d08132650 legacy=0x7f6d0806d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 >> v1:172.21.15.26:6789/0 conn(0x7f6d08138ff0 legacy=0x7f6d081cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:49:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 monclient: found mon.c 2024-09-16T14:49:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d08070dc0 con 0x7f6d0809cb80 2024-09-16T14:49:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient: _finish_auth 0 2024-09-16T14:49:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 5 monclient: authenticate success, global_id 154424 2024-09-16T14:49:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 10 monclient: _renew_subs 2024-09-16T14:49:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 1 -- 172.21.15.26:0/1366576386 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d0806dbe0 con 0x7f6d0809cb80 2024-09-16T14:49:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 10 monclient: _renew_subs 2024-09-16T14:49:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 1 -- 172.21.15.26:0/1366576386 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d080a20b0 con 0x7f6d0809cb80 2024-09-16T14:49:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6cf8001a10 con 0x7f6d0809cb80 2024-09-16T14:49:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6cf8004990 con 0x7f6d0809cb80 2024-09-16T14:49:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient: dump: 2024-09-16T14:49:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.048 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d057fa640 4 set_mon_vals no callback set 2024-09-16T14:49:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6cf800f400 con 0x7f6d0809cb80 2024-09-16T14:49:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d0d3ac640 1 -- 172.21.15.26:0/1366576386 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6cec006590 con 0x7f6cec002830 2024-09-16T14:49:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(200..200 src has 169..200) v4 ==== 10877+0+0 (unknown 2031785554 0 0) 0x7f6cf8044ca0 con 0x7f6d0809cb80 2024-09-16T14:49:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112607777 0 0) 0x7f6cec006590 con 0x7f6cec002830 2024-09-16T14:49:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient(hunting): my global_id is 164181 2024-09-16T14:49:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d0806f460 con 0x7f6cec002830 2024-09-16T14:49:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 812409090 0 0) 0x7f6d0806f460 con 0x7f6cec002830 2024-09-16T14:49:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cec006590 con 0x7f6cec002830 2024-09-16T14:49:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6cf0003fb0 con 0x7f6cec002830 2024-09-16T14:49:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6d077fe640 10 monclient: get_auth_request con 0x7f6ce8036210 auth_method 0 2024-09-16T14:49:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.026+0000 7f6cff7fe640 10 monclient: dump: 2024-09-16T14:49:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.050 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.030+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1750240423 0 0) 0x7f6cec006590 con 0x7f6cec002830 2024-09-16T14:49:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.030+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6cec00a0e0 con 0x7f6cec002830 2024-09-16T14:49:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.030+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6cf0006380 con 0x7f6cec002830 2024-09-16T14:49:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.030+0000 7f6cff7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.030+0000 7f6cff7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.030+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 >> v1:172.21.15.26:6789/0 conn(0x7f6cec002830 legacy=0x7f6cec004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.030+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 >> v1:172.21.15.26:6789/0 conn(0x7f6cec002830 legacy=0x7f6cec004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6d0d3ac640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:49:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6d0d3ac640 1 -- 172.21.15.26:0/1366576386 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6cec009da0 con 0x7f6cec0090c0 2024-09-16T14:49:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845877927 0 0) 0x7f6cec009da0 con 0x7f6cec0090c0 2024-09-16T14:49:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:52.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 10 monclient(hunting): my global_id is 164184 2024-09-16T14:49:52.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cec006590 con 0x7f6cec0090c0 2024-09-16T14:49:52.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2191570157 0 0) 0x7f6cec006590 con 0x7f6cec0090c0 2024-09-16T14:49:52.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cec009da0 con 0x7f6cec0090c0 2024-09-16T14:49:52.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6cf0004800 con 0x7f6cec0090c0 2024-09-16T14:49:52.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:52.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 10 monclient: dump: 2024-09-16T14:49:52.053 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:52.053 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.053 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.053 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.053 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.053 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:52.054 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.054 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.054 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.054 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3944823680 0 0) 0x7f6cec009da0 con 0x7f6cec0090c0 2024-09-16T14:49:52.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6cec00c290 con 0x7f6cec0090c0 2024-09-16T14:49:52.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4034 (unknown 0 0 752092636) 0x7f6cf0005c20 con 0x7f6cec0090c0 2024-09-16T14:49:52.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:49:52.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:52.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 >> v1:172.21.15.26:6789/0 conn(0x7f6cec0090c0 legacy=0x7f6cec009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6cff7fe640 1 -- 172.21.15.26:0/1366576386 >> v1:172.21.15.26:6789/0 conn(0x7f6cec0090c0 legacy=0x7f6cec009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 70, 2024-09-16T14:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:49:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:49:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:49:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:49:52.059 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:52.060 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:49:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:49:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:49:52.067 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:49:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6d0d3ac640 1 -- 172.21.15.26:0/1366576386 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6ce8036210 legacy=0x7f6ce80386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6d0d3ac640 10 monclient: shutdown 2024-09-16T14:49:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6d0d3ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6d0d3ac640 1 -- 172.21.15.26:0/1366576386 >> v1:172.21.15.26:6790/0 conn(0x7f6d0809cb80 legacy=0x7f6d0806cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6d077fe640 1 -- 172.21.15.26:0/1366576386 reap_dead start 2024-09-16T14:49:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6d077fe640 1 -- 172.21.15.26:0/1366576386 reap_dead start 2024-09-16T14:49:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6d0d3ac640 1 -- 172.21.15.26:0/1366576386 shutdown_connections 2024-09-16T14:49:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6d0d3ac640 1 -- 172.21.15.26:0/1366576386 >> 172.21.15.26:0/1366576386 conn(0x7f6d0809fb50 msgr2=0x7f6d08131a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6d0d3ac640 1 -- 172.21.15.26:0/1366576386 shutdown_connections 2024-09-16T14:49:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.034+0000 7f6d0d3ac640 1 -- 172.21.15.26:0/1366576386 wait complete. 2024-09-16T14:49:52.069 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:49:52.069 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:49:52.069 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:49:52.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.178+0000 7fe2c1da9640 10 monclient: get_monmap_and_config 2024-09-16T14:49:52.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.178+0000 7fe2c1da9640 10 monclient: build_initial_monmap 2024-09-16T14:49:52.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.178+0000 7fe2c1da9640 10 monclient: monmap: 2024-09-16T14:49:52.184 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:52.184 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.184 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:49:52.185353+0000 2024-09-16T14:49:52.185 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:49:52.185353+0000 2024-09-16T14:49:52.185 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:52.185 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:52.185 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:52.185 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:52.185 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:52.185 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 1 Processor -- start 2024-09-16T14:49:52.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 1 -- start start 2024-09-16T14:49:52.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 10 monclient: init 2024-09-16T14:49:52.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:52.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:52.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 10 monclient(hunting): picked mon.noname-c con 0x7fe2bc0a1a90 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:52.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 10 monclient(hunting): picked mon.noname-a con 0x7fe2bc0a2580 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:52.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 10 monclient(hunting): picked mon.noname-b con 0x7fe2bc09fba0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:52.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2bc06caa0 con 0x7fe2bc0a2580 2024-09-16T14:49:52.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2bc0a2e70 con 0x7fe2bc0a1a90 2024-09-16T14:49:52.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2bc09df00 con 0x7fe2bc09fba0 2024-09-16T14:49:52.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:52.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 10 monclient(hunting): authenticate will time out at 1416.305470s 2024-09-16T14:49:52.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2bb7fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fe2bc0a1a90 legacy=0x7fe2bc0a1e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:52.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2baffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe2bc0a2580 0x7fe2bc139ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40118/0 (socket says 172.21.15.26:40118) 2024-09-16T14:49:52.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2baffd640 1 -- 172.21.15.26:0/305034681 learned_addr learned my addr 172.21.15.26:0/305034681 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:52.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 1 -- 172.21.15.26:0/305034681 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970843033 0 0) 0x7fe2bc06caa0 con 0x7fe2bc0a2580 2024-09-16T14:49:52.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:52.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:52.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:52.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient(hunting): my global_id is 164187 2024-09-16T14:49:52.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 1 -- 172.21.15.26:0/305034681 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a4003530 con 0x7fe2bc0a2580 2024-09-16T14:49:52.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 1 -- 172.21.15.26:0/305034681 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762987707 0 0) 0x7fe2bc09df00 con 0x7fe2bc09fba0 2024-09-16T14:49:52.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:52.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:52.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:52.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient(hunting): my global_id is 154427 2024-09-16T14:49:52.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 1 -- 172.21.15.26:0/305034681 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2bc06caa0 con 0x7fe2bc09fba0 2024-09-16T14:49:52.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 1 -- 172.21.15.26:0/305034681 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1437301954 0 0) 0x7fe2a4003530 con 0x7fe2bc0a2580 2024-09-16T14:49:52.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 1 -- 172.21.15.26:0/305034681 >> v1:172.21.15.26:6790/0 conn(0x7fe2bc09fba0 legacy=0x7fe2bc13d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 1 -- 172.21.15.26:0/305034681 >> v1:172.21.15.189:6789/0 conn(0x7fe2bc0a1a90 legacy=0x7fe2bc0a1e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:52.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:49:52.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 1 monclient: found mon.noname-a 2024-09-16T14:49:52.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:52.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:49:52.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 1 -- 172.21.15.26:0/305034681 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc0a2bf0 con 0x7fe2bc0a2580 2024-09-16T14:49:52.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient: _finish_auth 0 2024-09-16T14:49:52.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:52.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:52.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 1 -- 172.21.15.26:0/305034681 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2ac001620 con 0x7fe2bc0a2580 2024-09-16T14:49:52.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:52.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient: dump: 2024-09-16T14:49:52.191 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:52.192 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.192 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.192 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.192 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.192 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:52.192 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.192 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.192 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.192 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 5 monclient: authenticate success, global_id 164187 2024-09-16T14:49:52.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:52.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 1 -- 172.21.15.26:0/305034681 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2ac001900 con 0x7fe2bc0a2580 2024-09-16T14:49:52.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:52.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:52.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 4 set_mon_vals no callback set 2024-09-16T14:49:52.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 1 -- 172.21.15.26:0/305034681 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2ac001cd0 con 0x7fe2bc0a2580 2024-09-16T14:49:52.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:52.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2ba7fc640 10 monclient: dump: 2024-09-16T14:49:52.193 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:52.193 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.193 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.193 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.193 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.193 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:52.193 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.194 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.194 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.194 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 10 monclient: shutdown 2024-09-16T14:49:52.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:52.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 1 -- 172.21.15.26:0/305034681 >> v1:172.21.15.26:6789/0 conn(0x7fe2bc0a2580 legacy=0x7fe2bc139ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 1 -- 172.21.15.26:0/305034681 shutdown_connections 2024-09-16T14:49:52.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 1 -- 172.21.15.26:0/305034681 >> 172.21.15.26:0/305034681 conn(0x7fe2bc0a7900 msgr2=0x7fe2bc0a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:52.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 1 -- 172.21.15.26:0/305034681 shutdown_connections 2024-09-16T14:49:52.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 1 -- 172.21.15.26:0/305034681 wait complete. 2024-09-16T14:49:52.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 10 monclient: build_initial_monmap 2024-09-16T14:49:52.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.182+0000 7fe2c1da9640 10 monclient: monmap: 2024-09-16T14:49:52.194 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:49:52.194 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:52.195 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:49:52.195 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:49:52.195 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:52.195 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:49:52.195 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:52.195 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:52.195 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:52.195 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 1 Processor -- start 2024-09-16T14:49:52.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 1 -- start start 2024-09-16T14:49:52.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 10 monclient: init 2024-09-16T14:49:52.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:52.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:49:52.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 10 monclient(hunting): picked mon.noname-b con 0x7fe2bc09fba0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:52.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 10 monclient(hunting): picked mon.noname-c con 0x7fe2bc0a1a90 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:52.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 10 monclient(hunting): picked mon.noname-a con 0x7fe2bc0a2580 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:52.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2bc0a6420 con 0x7fe2bc0a2580 2024-09-16T14:49:52.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2bc0a5ef0 con 0x7fe2bc09fba0 2024-09-16T14:49:52.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2bc09e280 con 0x7fe2bc0a1a90 2024-09-16T14:49:52.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:52.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 10 monclient(hunting): authenticate will time out at 1416.308762s 2024-09-16T14:49:52.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2baffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe2bc0a1a90 0x7fe2bc1337a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55378/0 (socket says 172.21.15.26:55378) 2024-09-16T14:49:52.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2baffd640 1 -- 172.21.15.26:0/1277312521 learned_addr learned my addr 172.21.15.26:0/1277312521 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:49:52.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2bb7fe640 1 -- 172.21.15.26:0/1277312521 >> v1:172.21.15.189:6789/0 conn(0x7fe2bc09fba0 legacy=0x7fe2bc133090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:52.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065753684 0 0) 0x7fe2bc0a6420 con 0x7fe2bc0a2580 2024-09-16T14:49:52.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:52.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:52.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:52.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): my global_id is 164190 2024-09-16T14:49:52.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a8003560 con 0x7fe2bc0a2580 2024-09-16T14:49:52.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304739088 0 0) 0x7fe2bc09e280 con 0x7fe2bc0a1a90 2024-09-16T14:49:52.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:52.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:52.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:52.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): my global_id is 154430 2024-09-16T14:49:52.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2bc0a6420 con 0x7fe2bc0a1a90 2024-09-16T14:49:52.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1871112405 0 0) 0x7fe2bc0a6420 con 0x7fe2bc0a1a90 2024-09-16T14:49:52.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2bc09e280 con 0x7fe2bc0a1a90 2024-09-16T14:49:52.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2963299680 0 0) 0x7fe2a8003560 con 0x7fe2bc0a2580 2024-09-16T14:49:52.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2bc0a6420 con 0x7fe2bc0a2580 2024-09-16T14:49:52.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2ac001c10 con 0x7fe2bc0a1a90 2024-09-16T14:49:52.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:52.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): dump: 2024-09-16T14:49:52.203 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:52.203 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.203 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.203 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.203 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.203 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:52.203 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.203 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.203 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.203 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:52.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2b4001b60 con 0x7fe2bc0a2580 2024-09-16T14:49:52.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:52.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): dump: 2024-09-16T14:49:52.204 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:52.204 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.204 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.204 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.204 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.204 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:52.204 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.204 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.204 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.204 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2479025750 0 0) 0x7fe2bc09e280 con 0x7fe2bc0a1a90 2024-09-16T14:49:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 >> v1:172.21.15.189:6789/0 conn(0x7fe2bc09fba0 legacy=0x7fe2bc133090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 >> v1:172.21.15.26:6789/0 conn(0x7fe2bc0a2580 legacy=0x7fe2bc133eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:49:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 monclient: found mon.c 2024-09-16T14:49:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc136e80 con 0x7fe2bc0a1a90 2024-09-16T14:49:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient: _finish_auth 0 2024-09-16T14:49:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:49:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 5 monclient: authenticate success, global_id 154430 2024-09-16T14:49:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 10 monclient: _renew_subs 2024-09-16T14:49:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 1 -- 172.21.15.26:0/1277312521 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2bc137160 con 0x7fe2bc0a1a90 2024-09-16T14:49:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 10 monclient: _renew_subs 2024-09-16T14:49:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 1 -- 172.21.15.26:0/1277312521 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2bc135c80 con 0x7fe2bc0a1a90 2024-09-16T14:49:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2ac004c70 con 0x7fe2bc0a1a90 2024-09-16T14:49:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2ac004f80 con 0x7fe2bc0a1a90 2024-09-16T14:49:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2b9ffb640 4 set_mon_vals no callback set 2024-09-16T14:49:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient: dump: 2024-09-16T14:49:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.207 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2c1da9640 1 -- 172.21.15.26:0/1277312521 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe29c006590 con 0x7fe29c002830 2024-09-16T14:49:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe2ac0050e0 con 0x7fe2bc0a1a90 2024-09-16T14:49:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(200..200 src has 169..200) v4 ==== 10877+0+0 (unknown 2031785554 0 0) 0x7fe2ac044f90 con 0x7fe2bc0a1a90 2024-09-16T14:49:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050789925 0 0) 0x7fe29c006590 con 0x7fe29c002830 2024-09-16T14:49:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient(hunting): my global_id is 154433 2024-09-16T14:49:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2bc09e280 con 0x7fe29c002830 2024-09-16T14:49:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 476061050 0 0) 0x7fe2bc09e280 con 0x7fe29c002830 2024-09-16T14:49:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe29c006590 con 0x7fe29c002830 2024-09-16T14:49:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2b0003f70 con 0x7fe29c002830 2024-09-16T14:49:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient: dump: 2024-09-16T14:49:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.209 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2bbfff640 10 monclient: get_auth_request con 0x7fe2a8036240 auth_method 0 2024-09-16T14:49:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4018981324 0 0) 0x7fe29c006590 con 0x7fe29c002830 2024-09-16T14:49:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe29c00a0e0 con 0x7fe29c002830 2024-09-16T14:49:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe2b0006380 con 0x7fe29c002830 2024-09-16T14:49:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 >> v1:172.21.15.26:6790/0 conn(0x7fe29c002830 legacy=0x7fe29c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.186+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 >> v1:172.21.15.26:6790/0 conn(0x7fe29c002830 legacy=0x7fe29c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2c1da9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:49:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2c1da9640 1 -- 172.21.15.26:0/1277312521 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe29c009da0 con 0x7fe29c0090c0 2024-09-16T14:49:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881077937 0 0) 0x7fe29c009da0 con 0x7fe29c0090c0 2024-09-16T14:49:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 10 monclient(hunting): my global_id is 154436 2024-09-16T14:49:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe29c006590 con 0x7fe29c0090c0 2024-09-16T14:49:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2715305710 0 0) 0x7fe29c006590 con 0x7fe29c0090c0 2024-09-16T14:49:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe29c009da0 con 0x7fe29c0090c0 2024-09-16T14:49:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2b0004800 con 0x7fe29c0090c0 2024-09-16T14:49:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 10 monclient: dump: 2024-09-16T14:49:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:49:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:49:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.211 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:49:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1623562705 0 0) 0x7fe29c009da0 con 0x7fe29c0090c0 2024-09-16T14:49:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe29c00c290 con 0x7fe29c0090c0 2024-09-16T14:49:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4036 (unknown 0 0 1037539776) 0x7fe2b0005ca0 con 0x7fe29c0090c0 2024-09-16T14:49:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:49:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:49:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 >> v1:172.21.15.26:6790/0 conn(0x7fe29c0090c0 legacy=0x7fe29c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2a3fff640 1 -- 172.21.15.26:0/1277312521 >> v1:172.21.15.26:6790/0 conn(0x7fe29c0090c0 legacy=0x7fe29c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:49:52.212 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:49:52.212 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:49:52.212 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:49:52.212 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:49:52.212 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 70, 2024-09-16T14:49:52.213 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:49:52.213 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:49:52.213 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:49:52.213 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:52.213 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:49:52.213 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:52.213 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:49:52.213 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:49:52.213 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:52.213 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:52.213 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:52.213 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:52.213 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:52.213 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:52.214 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:52.214 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:52.214 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:52.214 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:52.214 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:52.214 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:49:52.214 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:49:52.214 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:52.214 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:52.214 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:52.214 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:52.214 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:52.214 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:52.215 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:52.215 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:52.215 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:52.215 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:52.215 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:52.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.215 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:49:52.215 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:49:52.215 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:49:52.215 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:49:52.215 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:49:52.215 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:49:52.215 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:52.215 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:49:52.216 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:49:52.216 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:49:52.216 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:49:52.216 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:49:52.216 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:49:52.216 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:49:52.216 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:49:52.216 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:49:52.216 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:49:52.216 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:49:52.216 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:49:52.216 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:49:52.216 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:49:52.216 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:49:52.217 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:49:52.217 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:49:52.217 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:49:52.217 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:49:52.217 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:49:52.217 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:52.217 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:49:52.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.217 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:49:52.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.222 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:49:52.222 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:49:52.222 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:52.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:52.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.222 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:52.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:49:52.222 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:52.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:52.222 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:52.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.223 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:49:52.223 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:49:52.223 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:52.223 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:52.223 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:52.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.223 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:49:52.223 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:49:52.223 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:52.223 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:52.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.223 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:52.224 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:49:52.224 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:52.224 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:52.224 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:52.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.224 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:49:52.224 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:49:52.224 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:52.224 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:52.224 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:52.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.225 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:49:52.225 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:49:52.225 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:49:52.225 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:49:52.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.225 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:49:52.225 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:49:52.225 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:49:52.225 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:52.225 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:52.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.225 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:49:52.225 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:49:52.226 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:49:52.226 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:49:52.226 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:49:52.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:52.226 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:52.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.226 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:49:52.226 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:49:52.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.226 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:52.226 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:52.226 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:49:52.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:52.227 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:52.227 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:49:52.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.227 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:52.227 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:52.227 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:49:52.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:52.227 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:49:52.227 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:49:52.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:49:52.227 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:49:52.227 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:49:52.227 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:49:52.228 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:49:52.228 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:49:52.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:49:52.228 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:49:52.228 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:49:52.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2c1da9640 1 -- 172.21.15.26:0/1277312521 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe2a8036240 legacy=0x7fe2a8038700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2c1da9640 10 monclient: shutdown 2024-09-16T14:49:52.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2c1da9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:52.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2c1da9640 1 -- 172.21.15.26:0/1277312521 >> v1:172.21.15.26:6790/0 conn(0x7fe2bc0a1a90 legacy=0x7fe2bc1337a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2bbfff640 1 -- 172.21.15.26:0/1277312521 reap_dead start 2024-09-16T14:49:52.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2bbfff640 1 -- 172.21.15.26:0/1277312521 reap_dead start 2024-09-16T14:49:52.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2c1da9640 1 -- 172.21.15.26:0/1277312521 shutdown_connections 2024-09-16T14:49:52.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2c1da9640 1 -- 172.21.15.26:0/1277312521 >> 172.21.15.26:0/1277312521 conn(0x7fe2bc0a7900 msgr2=0x7fe2bc0a0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:52.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2c1da9640 1 -- 172.21.15.26:0/1277312521 shutdown_connections 2024-09-16T14:49:52.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:49:52.194+0000 7fe2c1da9640 1 -- 172.21.15.26:0/1277312521 wait complete. 2024-09-16T14:49:52.230 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:49:52.230 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:49:52.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.652+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}]=0 added bucket h18 type host to location {rack=r1} v201) v1 ==== 168+0+0 (unknown 802525560 0 0) 0x7fc4940144b0 con 0x7fc4a009e6d0 2024-09-16T14:49:52.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.652+0000 7fc4877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:52.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.652+0000 7fc4877fe640 10 monclient: _finish_command 2 = system:0 added bucket h18 type host to location {rack=r1} 2024-09-16T14:49:52.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.752+0000 7fc4a5645640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:52.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.752+0000 7fc4a5645640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:52.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.752+0000 7fc4a5645640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:52.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.752+0000 7fc4a5645640 1 -- 172.21.15.189:0/3740926399 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fc480005b10 con 0x7fc4a009e6d0 2024-09-16T14:49:52.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.756+0000 7fc4877fe640 1 -- 172.21.15.189:0/3740926399 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h18' already exists v201) v1 ==== 147+0+0 (unknown 201806297 0 0) 0x7fc494013a80 con 0x7fc4a009e6d0 2024-09-16T14:49:52.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.756+0000 7fc4877fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:52.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.756+0000 7fc4877fe640 10 monclient: _finish_command 3 = system:0 bucket 'h18' already exists 2024-09-16T14:49:52.758 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h18' already exists 2024-09-16T14:49:52.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.760+0000 7fc4a5645640 1 -- 172.21.15.189:0/3740926399 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc48c036570 legacy=0x7fc48c038a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.760+0000 7fc4a5645640 10 monclient: shutdown 2024-09-16T14:49:52.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.760+0000 7fc4a5645640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:52.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.760+0000 7fc4a5645640 1 -- 172.21.15.189:0/3740926399 >> v1:172.21.15.26:6789/0 conn(0x7fc4a009e6d0 legacy=0x7fc48c049020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.760+0000 7fc4a5645640 1 -- 172.21.15.189:0/3740926399 shutdown_connections 2024-09-16T14:49:52.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.760+0000 7fc4a5645640 1 -- 172.21.15.189:0/3740926399 >> 172.21.15.189:0/3740926399 conn(0x7fc4a00a3380 msgr2=0x7fc4a00a1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:52.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.760+0000 7fc4a5645640 1 -- 172.21.15.189:0/3740926399 shutdown_connections 2024-09-16T14:49:52.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.760+0000 7fc4a5645640 1 -- 172.21.15.189:0/3740926399 wait complete. 2024-09-16T14:49:52.776 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:52.776 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h19 host rack=r1 2024-09-16T14:49:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f677cfa5640 10 monclient: get_monmap_and_config 2024-09-16T14:49:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f677cfa5640 10 monclient: build_initial_monmap 2024-09-16T14:49:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f677cfa5640 10 monclient: monmap: 2024-09-16T14:49:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:52.895092+0000 2024-09-16T14:49:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:52.895092+0000 2024-09-16T14:49:52.894 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:52.895 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:52.895 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f677cfa5640 1 Processor -- start 2024-09-16T14:49:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f677cfa5640 1 -- start start 2024-09-16T14:49:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f677cfa5640 10 monclient: init 2024-09-16T14:49:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f677cfa5640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f677cfa5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:49:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f677cfa5640 10 monclient(hunting): picked mon.noname-a con 0x7f6778099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f677cfa5640 10 monclient(hunting): picked mon.noname-b con 0x7f677809c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f677cfa5640 10 monclient(hunting): picked mon.noname-c con 0x7f677809e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:52.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f677cfa5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6778099e50 con 0x7f6778099a70 2024-09-16T14:49:52.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f677cfa5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f677809dbd0 con 0x7f677809e6d0 2024-09-16T14:49:52.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f677cfa5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f677809ee20 con 0x7f677809c550 2024-09-16T14:49:52.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f677cfa5640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:52.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f677cfa5640 10 monclient(hunting): authenticate will time out at 1375.010828s 2024-09-16T14:49:52.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.892+0000 7f6776d9d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f677809e6d0 legacy=0x7f677813b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:52.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f6775d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f677809c550 0x7f67781379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56884/0 (socket says 172.21.15.189:56884) 2024-09-16T14:49:52.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f6775d9b640 1 -- 172.21.15.189:0/3478215920 learned_addr learned my addr 172.21.15.189:0/3478215920 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 1 -- 172.21.15.189:0/3478215920 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051799485 0 0) 0x7f6778099e50 con 0x7f6778099a70 2024-09-16T14:49:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 10 monclient(hunting): _negotiate 2024-09-16T14:49:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 10 monclient(hunting): my global_id is 164193 2024-09-16T14:49:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 1 -- 172.21.15.189:0/3478215920 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6760003530 con 0x7f6778099a70 2024-09-16T14:49:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 1 -- 172.21.15.189:0/3478215920 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380508754 0 0) 0x7f677809ee20 con 0x7f677809c550 2024-09-16T14:49:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 10 monclient(hunting): _negotiate 2024-09-16T14:49:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:52.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 10 monclient(hunting): my global_id is 154439 2024-09-16T14:49:52.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 1 -- 172.21.15.189:0/3478215920 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6778099e50 con 0x7f677809c550 2024-09-16T14:49:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 1 -- 172.21.15.189:0/3478215920 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2022966807 0 0) 0x7f6760003530 con 0x7f6778099a70 2024-09-16T14:49:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 1 -- 172.21.15.189:0/3478215920 >> v1:172.21.15.26:6790/0 conn(0x7f677809c550 legacy=0x7f67781379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 1 -- 172.21.15.189:0/3478215920 >> v1:172.21.15.189:6789/0 conn(0x7f677809e6d0 legacy=0x7f677813b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 10 monclient: _finish_hunting 0 2024-09-16T14:49:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 1 monclient: found mon.noname-a 2024-09-16T14:49:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:49:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 1 -- 172.21.15.189:0/3478215920 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677809f100 con 0x7f6778099a70 2024-09-16T14:49:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 10 monclient: _finish_auth 0 2024-09-16T14:49:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 10 monclient: _check_auth_tickets 2024-09-16T14:49:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 1 -- 172.21.15.189:0/3478215920 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f676c001620 con 0x7f6778099a70 2024-09-16T14:49:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677559a640 10 monclient: dump: 2024-09-16T14:49:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.902 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.903 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677cfa5640 5 monclient: authenticate success, global_id 164193 2024-09-16T14:49:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.896+0000 7f677cfa5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677559a640 1 -- 172.21.15.189:0/3478215920 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f676c001840 con 0x7f6778099a70 2024-09-16T14:49:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677559a640 1 -- 172.21.15.189:0/3478215920 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f676c001b90 con 0x7f6778099a70 2024-09-16T14:49:52.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677559a640 10 monclient: dump: 2024-09-16T14:49:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.904 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:52.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:52.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 4 set_mon_vals no callback set 2024-09-16T14:49:52.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 10 monclient: shutdown 2024-09-16T14:49:52.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:52.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 1 -- 172.21.15.189:0/3478215920 >> v1:172.21.15.26:6789/0 conn(0x7f6778099a70 legacy=0x7f677809dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 1 -- 172.21.15.189:0/3478215920 shutdown_connections 2024-09-16T14:49:52.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 1 -- 172.21.15.189:0/3478215920 >> 172.21.15.189:0/3478215920 conn(0x7f67780a3380 msgr2=0x7f677809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:52.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 1 -- 172.21.15.189:0/3478215920 shutdown_connections 2024-09-16T14:49:52.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 1 -- 172.21.15.189:0/3478215920 wait complete. 2024-09-16T14:49:52.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 10 monclient: build_initial_monmap 2024-09-16T14:49:52.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 10 monclient: monmap: 2024-09-16T14:49:52.910 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:52.910 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:52.910 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:52.910 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:52.910 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:52.910 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:52.910 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:52.910 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:52.910 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:52.910 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:52.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 1 Processor -- start 2024-09-16T14:49:52.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 1 -- start start 2024-09-16T14:49:52.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 10 monclient: init 2024-09-16T14:49:52.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:52.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:52.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 10 monclient(hunting): picked mon.noname-b con 0x7f6778099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:52.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 10 monclient(hunting): picked mon.noname-a con 0x7f677809c550 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:52.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 10 monclient(hunting): picked mon.noname-c con 0x7f677809e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:52.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f677809cf20 con 0x7f677809c550 2024-09-16T14:49:52.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f677809bd00 con 0x7f6778099a70 2024-09-16T14:49:52.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f677813b9b0 con 0x7f677809e6d0 2024-09-16T14:49:52.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:52.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677cfa5640 10 monclient(hunting): authenticate will time out at 1375.019018s 2024-09-16T14:49:52.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f677659c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f6778099a70 legacy=0x7f677809a1f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:52.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f6776d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f677809e6d0 0x7f67781d1720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56894/0 (socket says 172.21.15.189:56894) 2024-09-16T14:49:52.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f6776d9d640 1 -- 172.21.15.189:0/3348546957 learned_addr learned my addr 172.21.15.189:0/3348546957 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:52.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297655521 0 0) 0x7f677813b9b0 con 0x7f677809e6d0 2024-09-16T14:49:52.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f6766ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:52.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f6766ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:52.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f6766ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:52.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f6766ffd640 10 monclient(hunting): my global_id is 154442 2024-09-16T14:49:52.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.900+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6758003530 con 0x7f677809e6d0 2024-09-16T14:49:52.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171699087 0 0) 0x7f677809cf20 con 0x7f677809c550 2024-09-16T14:49:52.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:52.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:52.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:52.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient(hunting): my global_id is 164196 2024-09-16T14:49:52.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f677813b9b0 con 0x7f677809c550 2024-09-16T14:49:52.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2873690867 0 0) 0x7f6758003530 con 0x7f677809e6d0 2024-09-16T14:49:52.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f677809cf20 con 0x7f677809e6d0 2024-09-16T14:49:52.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67700043c0 con 0x7f677809e6d0 2024-09-16T14:49:52.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:52.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient(hunting): dump: 2024-09-16T14:49:52.913 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:52.913 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.913 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.913 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.913 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.913 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:52.914 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.914 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.914 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.914 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:52.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:52.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4136994333 0 0) 0x7f677813b9b0 con 0x7f677809c550 2024-09-16T14:49:52.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6758003530 con 0x7f677809c550 2024-09-16T14:49:52.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67680017c0 con 0x7f677809c550 2024-09-16T14:49:52.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:52.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient(hunting): dump: 2024-09-16T14:49:52.914 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:52.914 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.914 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.915 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2868234631 0 0) 0x7f677809cf20 con 0x7f677809e6d0 2024-09-16T14:49:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 >> v1:172.21.15.189:6789/0 conn(0x7f6778099a70 legacy=0x7f677809a1f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 >> v1:172.21.15.26:6789/0 conn(0x7f677809c550 legacy=0x7f677809a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 monclient: found mon.c 2024-09-16T14:49:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677809b010 con 0x7f677809e6d0 2024-09-16T14:49:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient: _finish_auth 0 2024-09-16T14:49:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f677cfa5640 5 monclient: authenticate success, global_id 154442 2024-09-16T14:49:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f677cfa5640 10 monclient: _renew_subs 2024-09-16T14:49:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f677cfa5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f677cfa5640 1 -- 172.21.15.189:0/3348546957 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677813a9e0 con 0x7f677809e6d0 2024-09-16T14:49:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f677cfa5640 10 monclient: _renew_subs 2024-09-16T14:49:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f677cfa5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:52.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f677cfa5640 1 -- 172.21.15.189:0/3348546957 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f677806b380 con 0x7f677809e6d0 2024-09-16T14:49:52.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67700045e0 con 0x7f677809e6d0 2024-09-16T14:49:52.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:52.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67700048f0 con 0x7f677809e6d0 2024-09-16T14:49:52.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:52.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6767fff640 4 set_mon_vals no callback set 2024-09-16T14:49:52.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:52.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 10 monclient: dump: 2024-09-16T14:49:52.917 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:52.917 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:52.917 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.918 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:52.918 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:52.918 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:52.918 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:52.918 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:52.918 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:52.918 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:52.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f677cfa5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:52.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f677cfa5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:52.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f677cfa5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:52.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f677cfa5640 1 -- 172.21.15.189:0/3348546957 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675c005050 con 0x7f677809e6d0 2024-09-16T14:49:52.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6770004ad0 con 0x7f677809e6d0 2024-09-16T14:49:52.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(201..201 src has 169..201) v4 ==== 11090+0+0 (unknown 486451468 0 0) 0x7f6770045040 con 0x7f677809e6d0 2024-09-16T14:49:52.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.904+0000 7f677659c640 10 monclient: get_auth_request con 0x7f6758036230 auth_method 0 2024-09-16T14:49:52.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.916+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6770014600 con 0x7f677809e6d0 2024-09-16T14:49:52.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.916+0000 7f6766ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:52.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:52.916+0000 7f6766ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:53.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.072+0000 7f677cfa5640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:53.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.072+0000 7f677cfa5640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:53.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.072+0000 7f677cfa5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:53.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.072+0000 7f677cfa5640 1 -- 172.21.15.189:0/3348546957 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f675c005300 con 0x7f677809e6d0 2024-09-16T14:49:53.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.660+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}]=0 added bucket h19 type host to location {rack=r1} v202) v1 ==== 168+0+0 (unknown 3824331784 0 0) 0x7f6770014420 con 0x7f677809e6d0 2024-09-16T14:49:53.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.660+0000 7f6766ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:53.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.660+0000 7f6766ffd640 10 monclient: _finish_command 2 = system:0 added bucket h19 type host to location {rack=r1} 2024-09-16T14:49:53.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.756+0000 7f677cfa5640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:53.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.756+0000 7f677cfa5640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:53.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.756+0000 7f677cfa5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:53.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.756+0000 7f677cfa5640 1 -- 172.21.15.189:0/3348546957 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f675c005a40 con 0x7f677809e6d0 2024-09-16T14:49:53.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.756+0000 7f6766ffd640 1 -- 172.21.15.189:0/3348546957 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h19' already exists v202) v1 ==== 147+0+0 (unknown 786583517 0 0) 0x7f6770014420 con 0x7f677809e6d0 2024-09-16T14:49:53.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.756+0000 7f6766ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:53.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.756+0000 7f6766ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h19' already exists 2024-09-16T14:49:53.758 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h19' already exists 2024-09-16T14:49:53.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.756+0000 7f677cfa5640 1 -- 172.21.15.189:0/3348546957 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6758036230 legacy=0x7f67580386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:53.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.756+0000 7f677cfa5640 10 monclient: shutdown 2024-09-16T14:49:53.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.756+0000 7f677cfa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:53.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.756+0000 7f677cfa5640 1 -- 172.21.15.189:0/3348546957 >> v1:172.21.15.26:6790/0 conn(0x7f677809e6d0 legacy=0x7f67781d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:53.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.756+0000 7f677cfa5640 1 -- 172.21.15.189:0/3348546957 shutdown_connections 2024-09-16T14:49:53.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.756+0000 7f677cfa5640 1 -- 172.21.15.189:0/3348546957 >> 172.21.15.189:0/3348546957 conn(0x7f67780a3380 msgr2=0x7f67780a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:53.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.756+0000 7f677cfa5640 1 -- 172.21.15.189:0/3348546957 shutdown_connections 2024-09-16T14:49:53.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.756+0000 7f677cfa5640 1 -- 172.21.15.189:0/3348546957 wait complete. 2024-09-16T14:49:53.775 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:53.775 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h20 host rack=r1 2024-09-16T14:49:53.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2efc7b640 10 monclient: get_monmap_and_config 2024-09-16T14:49:53.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2efc7b640 10 monclient: build_initial_monmap 2024-09-16T14:49:53.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2efc7b640 10 monclient: monmap: 2024-09-16T14:49:53.894 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:53.894 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:53.894 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:53.895114+0000 2024-09-16T14:49:53.894 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:53.895114+0000 2024-09-16T14:49:53.894 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:53.894 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:53.894 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:53.894 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:53.895 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:53.895 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:53.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2efc7b640 1 Processor -- start 2024-09-16T14:49:53.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2efc7b640 1 -- start start 2024-09-16T14:49:53.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2efc7b640 10 monclient: init 2024-09-16T14:49:53.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2efc7b640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:53.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2efc7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:53.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2efc7b640 10 monclient(hunting): picked mon.noname-c con 0x7fa2e80993f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:53.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2efc7b640 10 monclient(hunting): picked mon.noname-a con 0x7fa2e809a190 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:53.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2efc7b640 10 monclient(hunting): picked mon.noname-b con 0x7fa2e8131b70 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:53.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2efc7b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e809cca0 con 0x7fa2e809a190 2024-09-16T14:49:53.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2efc7b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e809cf80 con 0x7fa2e80993f0 2024-09-16T14:49:53.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2efc7b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e8073b70 con 0x7fa2e8131b70 2024-09-16T14:49:53.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2efc7b640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:53.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2efc7b640 10 monclient(hunting): authenticate will time out at 1376.011558s 2024-09-16T14:49:53.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2eda17640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa2e80993f0 legacy=0x7fa2e80997d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:53.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2ee218640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa2e8131b70 0x7fa2e813a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56900/0 (socket says 172.21.15.189:56900) 2024-09-16T14:49:53.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2ed216640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa2e809a190 0x7fa2e81369b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:41958/0 (socket says 172.21.15.189:41958) 2024-09-16T14:49:53.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.892+0000 7fa2ee218640 1 -- 172.21.15.189:0/43621987 learned_addr learned my addr 172.21.15.189:0/43621987 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:53.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 1 -- 172.21.15.189:0/43621987 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265479482 0 0) 0x7fa2e8073b70 con 0x7fa2e8131b70 2024-09-16T14:49:53.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient(hunting): _negotiate 2024-09-16T14:49:53.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:53.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:53.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient(hunting): my global_id is 154445 2024-09-16T14:49:53.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 1 -- 172.21.15.189:0/43621987 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2d4003530 con 0x7fa2e8131b70 2024-09-16T14:49:53.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 1 -- 172.21.15.189:0/43621987 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927507701 0 0) 0x7fa2e809cca0 con 0x7fa2e809a190 2024-09-16T14:49:53.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient(hunting): _negotiate 2024-09-16T14:49:53.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:53.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:53.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient(hunting): my global_id is 164199 2024-09-16T14:49:53.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 1 -- 172.21.15.189:0/43621987 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e8073b70 con 0x7fa2e809a190 2024-09-16T14:49:53.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 1 -- 172.21.15.189:0/43621987 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3835878107 0 0) 0x7fa2d4003530 con 0x7fa2e8131b70 2024-09-16T14:49:53.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 1 -- 172.21.15.189:0/43621987 >> v1:172.21.15.189:6789/0 conn(0x7fa2e80993f0 legacy=0x7fa2e80997d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:53.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 1 -- 172.21.15.189:0/43621987 >> v1:172.21.15.26:6789/0 conn(0x7fa2e809a190 legacy=0x7fa2e81369b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:53.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient: _finish_hunting 0 2024-09-16T14:49:53.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 1 monclient: found mon.noname-b 2024-09-16T14:49:53.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:53.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:49:53.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 1 -- 172.21.15.189:0/43621987 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e8099ee0 con 0x7fa2e8131b70 2024-09-16T14:49:53.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient: _finish_auth 0 2024-09-16T14:49:53.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient: _check_auth_tickets 2024-09-16T14:49:53.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:53.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 1 -- 172.21.15.189:0/43621987 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa2e00019a0 con 0x7fa2e8131b70 2024-09-16T14:49:53.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:53.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:53.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient: dump: 2024-09-16T14:49:53.902 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:53.902 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:53.902 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:53.902 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:53.902 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:53.902 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:53.902 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:53.903 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:53.903 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:53.903 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:53.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 5 monclient: authenticate success, global_id 154445 2024-09-16T14:49:53.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:53.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 1 -- 172.21.15.189:0/43621987 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2e0001c80 con 0x7fa2e8131b70 2024-09-16T14:49:53.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:53.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 1 -- 172.21.15.189:0/43621987 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa2e0002010 con 0x7fa2e8131b70 2024-09-16T14:49:53.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:53.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:53.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eca15640 10 monclient: dump: 2024-09-16T14:49:53.903 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:53.904 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:53.904 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:53.904 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:53.904 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:53.904 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:53.904 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:53.904 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:53.904 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:53.904 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:53.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:53.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 4 set_mon_vals no callback set 2024-09-16T14:49:53.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 10 monclient: shutdown 2024-09-16T14:49:53.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:53.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 1 -- 172.21.15.189:0/43621987 >> v1:172.21.15.26:6790/0 conn(0x7fa2e8131b70 legacy=0x7fa2e813a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:53.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 1 -- 172.21.15.189:0/43621987 shutdown_connections 2024-09-16T14:49:53.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 1 -- 172.21.15.189:0/43621987 >> 172.21.15.189:0/43621987 conn(0x7fa2e809fed0 msgr2=0x7fa2e80a02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:53.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 1 -- 172.21.15.189:0/43621987 shutdown_connections 2024-09-16T14:49:53.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 1 -- 172.21.15.189:0/43621987 wait complete. 2024-09-16T14:49:53.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 10 monclient: build_initial_monmap 2024-09-16T14:49:53.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 10 monclient: monmap: 2024-09-16T14:49:53.905 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:53.905 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:53.905 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:53.905 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:53.905 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:53.905 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:53.905 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:53.906 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:53.906 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:53.906 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:53.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 1 Processor -- start 2024-09-16T14:49:53.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 1 -- start start 2024-09-16T14:49:53.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 10 monclient: init 2024-09-16T14:49:53.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:53.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:49:53.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 10 monclient(hunting): picked mon.noname-c con 0x7fa2e80993f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:53.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 10 monclient(hunting): picked mon.noname-b con 0x7fa2e809a190 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:53.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 10 monclient(hunting): picked mon.noname-a con 0x7fa2e8131b70 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:53.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e806e570 con 0x7fa2e8131b70 2024-09-16T14:49:53.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e809a5c0 con 0x7fa2e809a190 2024-09-16T14:49:53.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e806e2d0 con 0x7fa2e80993f0 2024-09-16T14:49:53.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:53.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2efc7b640 10 monclient(hunting): authenticate will time out at 1376.015050s 2024-09-16T14:49:53.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2ed216640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa2e809a190 legacy=0x7fa2e806dbc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:53.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eda17640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa2e80993f0 0x7fa2e806d4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56906/0 (socket says 172.21.15.189:56906) 2024-09-16T14:49:53.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2eda17640 1 -- 172.21.15.189:0/2165287616 learned_addr learned my addr 172.21.15.189:0/2165287616 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:53.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672033609 0 0) 0x7fa2e806e2d0 con 0x7fa2e80993f0 2024-09-16T14:49:53.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2de7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:53.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2de7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:53.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:53.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2de7fc640 10 monclient(hunting): my global_id is 154448 2024-09-16T14:49:53.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2cc003530 con 0x7fa2e80993f0 2024-09-16T14:49:53.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285817459 0 0) 0x7fa2e806e570 con 0x7fa2e8131b70 2024-09-16T14:49:53.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2de7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:53.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2de7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:53.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:53.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2de7fc640 10 monclient(hunting): my global_id is 164202 2024-09-16T14:49:53.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.896+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e806e2d0 con 0x7fa2e8131b70 2024-09-16T14:49:53.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3381839462 0 0) 0x7fa2e806e2d0 con 0x7fa2e8131b70 2024-09-16T14:49:53.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2e806e570 con 0x7fa2e8131b70 2024-09-16T14:49:53.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa2e00017c0 con 0x7fa2e8131b70 2024-09-16T14:49:53.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:53.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:53.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 10 monclient(hunting): dump: 2024-09-16T14:49:53.912 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:53.912 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:53.912 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:53.912 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:53.912 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:53.912 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:53.913 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:53.913 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:53.913 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:53.913 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:53.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:53.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4064718221 0 0) 0x7fa2cc003530 con 0x7fa2e80993f0 2024-09-16T14:49:53.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2e806e2d0 con 0x7fa2e80993f0 2024-09-16T14:49:53.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa2e40041a0 con 0x7fa2e80993f0 2024-09-16T14:49:53.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:53.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:53.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 10 monclient(hunting): dump: 2024-09-16T14:49:53.913 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:53.913 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:53.914 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:53.914 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:53.914 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:53.914 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:53.914 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:53.914 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:53.914 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:53.914 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:53.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1102839379 0 0) 0x7fa2e806e570 con 0x7fa2e8131b70 2024-09-16T14:49:53.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 >> v1:172.21.15.26:6790/0 conn(0x7fa2e80993f0 legacy=0x7fa2e806d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:53.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 >> v1:172.21.15.189:6789/0 conn(0x7fa2e809a190 legacy=0x7fa2e806dbc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:53.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:49:53.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 1 monclient: found mon.a 2024-09-16T14:49:53.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:53.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:53.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e809f230 con 0x7fa2e8131b70 2024-09-16T14:49:53.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 10 monclient: _finish_auth 0 2024-09-16T14:49:53.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:53.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:53.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2efc7b640 5 monclient: authenticate success, global_id 164202 2024-09-16T14:49:53.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2efc7b640 10 monclient: _renew_subs 2024-09-16T14:49:53.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2efc7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:53.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2efc7b640 1 -- 172.21.15.189:0/2165287616 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e809f7a0 con 0x7fa2e8131b70 2024-09-16T14:49:53.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2efc7b640 10 monclient: _renew_subs 2024-09-16T14:49:53.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2efc7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:53.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2efc7b640 1 -- 172.21.15.189:0/2165287616 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e813b170 con 0x7fa2e8131b70 2024-09-16T14:49:53.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2e0001a40 con 0x7fa2e8131b70 2024-09-16T14:49:53.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:53.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2dffff640 4 set_mon_vals no callback set 2024-09-16T14:49:53.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa2e0005120 con 0x7fa2e8131b70 2024-09-16T14:49:53.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:53.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:53.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 10 monclient: dump: 2024-09-16T14:49:53.916 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:53.916 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:53.916 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:53.916 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:53.916 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:53.916 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:53.916 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:53.916 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:53.917 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:53.917 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:53.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2efc7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:53.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2efc7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:53.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2efc7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:53.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2efc7b640 1 -- 172.21.15.189:0/2165287616 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d0005050 con 0x7fa2e8131b70 2024-09-16T14:49:53.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa2e000fb90 con 0x7fa2e8131b70 2024-09-16T14:49:53.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(202..202 src has 169..202) v4 ==== 11303+0+0 (unknown 1970912734 0 0) 0x7fa2e00453b0 con 0x7fa2e8131b70 2024-09-16T14:49:53.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.900+0000 7fa2eda17640 10 monclient: get_auth_request con 0x7fa2cc036200 auth_method 0 2024-09-16T14:49:53.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.912+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa2e00149b0 con 0x7fa2e8131b70 2024-09-16T14:49:53.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.912+0000 7fa2de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:53.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:53.912+0000 7fa2de7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:54.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.076+0000 7fa2efc7b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:54.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.076+0000 7fa2efc7b640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:54.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.076+0000 7fa2efc7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:54.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.076+0000 7fa2efc7b640 1 -- 172.21.15.189:0/2165287616 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa2d0005300 con 0x7fa2e8131b70 2024-09-16T14:49:54.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.668+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}]=0 added bucket h20 type host to location {rack=r1} v203) v1 ==== 168+0+0 (unknown 3476003295 0 0) 0x7fa2e00147d0 con 0x7fa2e8131b70 2024-09-16T14:49:54.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.668+0000 7fa2de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:54.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.668+0000 7fa2de7fc640 10 monclient: _finish_command 2 = system:0 added bucket h20 type host to location {rack=r1} 2024-09-16T14:49:54.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.764+0000 7fa2efc7b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:54.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.764+0000 7fa2efc7b640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:54.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.764+0000 7fa2efc7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:54.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.764+0000 7fa2efc7b640 1 -- 172.21.15.189:0/2165287616 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa2d0005a40 con 0x7fa2e8131b70 2024-09-16T14:49:54.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.768+0000 7fa2de7fc640 1 -- 172.21.15.189:0/2165287616 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h20' already exists v203) v1 ==== 147+0+0 (unknown 1830156877 0 0) 0x7fa2e00147d0 con 0x7fa2e8131b70 2024-09-16T14:49:54.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.768+0000 7fa2de7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:54.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.768+0000 7fa2de7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h20' already exists 2024-09-16T14:49:54.769 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h20' already exists 2024-09-16T14:49:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.768+0000 7fa2efc7b640 1 -- 172.21.15.189:0/2165287616 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa2cc036200 legacy=0x7fa2cc0386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.768+0000 7fa2efc7b640 10 monclient: shutdown 2024-09-16T14:49:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.768+0000 7fa2efc7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.768+0000 7fa2efc7b640 1 -- 172.21.15.189:0/2165287616 >> v1:172.21.15.26:6789/0 conn(0x7fa2e8131b70 legacy=0x7fa2e81d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.768+0000 7fa2efc7b640 1 -- 172.21.15.189:0/2165287616 shutdown_connections 2024-09-16T14:49:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.768+0000 7fa2efc7b640 1 -- 172.21.15.189:0/2165287616 >> 172.21.15.189:0/2165287616 conn(0x7fa2e809fed0 msgr2=0x7fa2e809e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.768+0000 7fa2efc7b640 1 -- 172.21.15.189:0/2165287616 shutdown_connections 2024-09-16T14:49:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.768+0000 7fa2efc7b640 1 -- 172.21.15.189:0/2165287616 wait complete. 2024-09-16T14:49:54.785 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:54.785 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h21 host rack=r1 2024-09-16T14:49:54.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.904+0000 7fc809106640 10 monclient: get_monmap_and_config 2024-09-16T14:49:54.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.904+0000 7fc809106640 10 monclient: build_initial_monmap 2024-09-16T14:49:54.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.904+0000 7fc809106640 10 monclient: monmap: 2024-09-16T14:49:54.906 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:54.906 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:54.906 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:54.907296+0000 2024-09-16T14:49:54.906 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:54.907296+0000 2024-09-16T14:49:54.907 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:54.907 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:54.907 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:54.907 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:54.907 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:54.907 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:54.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.904+0000 7fc809106640 1 Processor -- start 2024-09-16T14:49:54.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.904+0000 7fc809106640 1 -- start start 2024-09-16T14:49:54.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.904+0000 7fc809106640 10 monclient: init 2024-09-16T14:49:54.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.904+0000 7fc809106640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:54.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.904+0000 7fc809106640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:49:54.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.904+0000 7fc809106640 10 monclient(hunting): picked mon.noname-c con 0x7fc804099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:54.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.904+0000 7fc809106640 10 monclient(hunting): picked mon.noname-b con 0x7fc80409bd30 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:54.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.904+0000 7fc809106640 10 monclient(hunting): picked mon.noname-a con 0x7fc804137500 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:54.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.904+0000 7fc809106640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc804099e50 con 0x7fc804137500 2024-09-16T14:49:54.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.904+0000 7fc802d9d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fc804099a70 legacy=0x7fc80409d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:54.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.904+0000 7fc809106640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80409e510 con 0x7fc804099a70 2024-09-16T14:49:54.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80409c2c0 con 0x7fc80409bd30 2024-09-16T14:49:54.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:54.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 10 monclient(hunting): authenticate will time out at 1377.024065s 2024-09-16T14:49:54.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc80259c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc80409bd30 0x7fc80409de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56916/0 (socket says 172.21.15.189:56916) 2024-09-16T14:49:54.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc80259c640 1 -- 172.21.15.189:0/2163056329 learned_addr learned my addr 172.21.15.189:0/2163056329 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:54.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 1 -- 172.21.15.189:0/2163056329 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136692658 0 0) 0x7fc804099e50 con 0x7fc804137500 2024-09-16T14:49:54.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:49:54.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:54.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:54.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient(hunting): my global_id is 164205 2024-09-16T14:49:54.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 1 -- 172.21.15.189:0/2163056329 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7ec003530 con 0x7fc804137500 2024-09-16T14:49:54.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 1 -- 172.21.15.189:0/2163056329 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351757989 0 0) 0x7fc80409c2c0 con 0x7fc80409bd30 2024-09-16T14:49:54.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:49:54.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:54.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:54.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient(hunting): my global_id is 154451 2024-09-16T14:49:54.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 1 -- 172.21.15.189:0/2163056329 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc804099e50 con 0x7fc80409bd30 2024-09-16T14:49:54.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 1 -- 172.21.15.189:0/2163056329 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4169713591 0 0) 0x7fc7ec003530 con 0x7fc804137500 2024-09-16T14:49:54.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 1 -- 172.21.15.189:0/2163056329 >> v1:172.21.15.26:6790/0 conn(0x7fc80409bd30 legacy=0x7fc80409de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:54.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 1 -- 172.21.15.189:0/2163056329 >> v1:172.21.15.189:6789/0 conn(0x7fc804099a70 legacy=0x7fc80409d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:54.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:49:54.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 1 monclient: found mon.noname-a 2024-09-16T14:49:54.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:54.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:49:54.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 1 -- 172.21.15.189:0/2163056329 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80409ee60 con 0x7fc804137500 2024-09-16T14:49:54.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient: _finish_auth 0 2024-09-16T14:49:54.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:49:54.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:54.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 1 -- 172.21.15.189:0/2163056329 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc7fc0019a0 con 0x7fc804137500 2024-09-16T14:49:54.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:54.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:54.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient: dump: 2024-09-16T14:49:54.913 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:54.913 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:54.913 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:54.913 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:54.913 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:54.914 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:54.914 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:54.914 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:54.914 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:54.914 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:54.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 5 monclient: authenticate success, global_id 164205 2024-09-16T14:49:54.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:54.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 1 -- 172.21.15.189:0/2163056329 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7fc001c80 con 0x7fc804137500 2024-09-16T14:49:54.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:54.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:54.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 4 set_mon_vals no callback set 2024-09-16T14:49:54.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 1 -- 172.21.15.189:0/2163056329 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc7fc002010 con 0x7fc804137500 2024-09-16T14:49:54.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:54.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:54.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc801d9b640 10 monclient: dump: 2024-09-16T14:49:54.915 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:54.915 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:54.915 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:54.915 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:54.915 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:54.915 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:54.915 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:54.915 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:54.915 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:54.915 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:54.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 10 monclient: shutdown 2024-09-16T14:49:54.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:54.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 1 -- 172.21.15.189:0/2163056329 >> v1:172.21.15.26:6789/0 conn(0x7fc804137500 legacy=0x7fc8041398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:54.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 1 -- 172.21.15.189:0/2163056329 shutdown_connections 2024-09-16T14:49:54.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 1 -- 172.21.15.189:0/2163056329 >> 172.21.15.189:0/2163056329 conn(0x7fc8040a3380 msgr2=0x7fc80409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:54.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 1 -- 172.21.15.189:0/2163056329 shutdown_connections 2024-09-16T14:49:54.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 1 -- 172.21.15.189:0/2163056329 wait complete. 2024-09-16T14:49:54.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 10 monclient: build_initial_monmap 2024-09-16T14:49:54.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 10 monclient: monmap: 2024-09-16T14:49:54.916 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:54.916 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:54.916 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:54.916 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:54.916 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:54.917 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:54.917 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:54.917 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:54.917 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:54.917 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:54.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 1 Processor -- start 2024-09-16T14:49:54.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 1 -- start start 2024-09-16T14:49:54.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.908+0000 7fc809106640 10 monclient: init 2024-09-16T14:49:54.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:54.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:54.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 10 monclient(hunting): picked mon.noname-a con 0x7fc804099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:54.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 10 monclient(hunting): picked mon.noname-b con 0x7fc80409bd30 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:54.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 10 monclient(hunting): picked mon.noname-c con 0x7fc804137500 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:54.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8041c4130 con 0x7fc804099a70 2024-09-16T14:49:54.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc80259c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fc80409bd30 legacy=0x7fc8041ce3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:54.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80411c270 con 0x7fc80409bd30 2024-09-16T14:49:54.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80411dc40 con 0x7fc804137500 2024-09-16T14:49:54.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:54.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 10 monclient(hunting): authenticate will time out at 1377.028424s 2024-09-16T14:49:54.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc802d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc804099a70 0x7fc8041c96d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:41986/0 (socket says 172.21.15.189:41986) 2024-09-16T14:49:54.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc80359e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc804137500 0x7fc8041d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56922/0 (socket says 172.21.15.189:56922) 2024-09-16T14:49:54.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc802d9d640 1 -- 172.21.15.189:0/3911805289 learned_addr learned my addr 172.21.15.189:0/3911805289 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:54.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412061620 0 0) 0x7fc8041c4130 con 0x7fc804099a70 2024-09-16T14:49:54.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:54.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:54.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:54.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient(hunting): my global_id is 164208 2024-09-16T14:49:54.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f0003530 con 0x7fc804099a70 2024-09-16T14:49:54.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059891462 0 0) 0x7fc80411dc40 con 0x7fc804137500 2024-09-16T14:49:54.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:54.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:54.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:54.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient(hunting): my global_id is 154454 2024-09-16T14:49:54.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8041c4130 con 0x7fc804137500 2024-09-16T14:49:54.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 966757513 0 0) 0x7fc7f0003530 con 0x7fc804099a70 2024-09-16T14:49:54.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc80411dc40 con 0x7fc804099a70 2024-09-16T14:49:54.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc7f4004160 con 0x7fc804099a70 2024-09-16T14:49:54.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:54.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:54.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient(hunting): dump: 2024-09-16T14:49:54.923 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:54.923 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:54.923 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:54.923 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:54.923 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:54.923 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:54.923 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:54.923 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:54.923 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:54.924 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:54.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:54.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1099745084 0 0) 0x7fc8041c4130 con 0x7fc804137500 2024-09-16T14:49:54.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7f0003530 con 0x7fc804137500 2024-09-16T14:49:54.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc7fc001960 con 0x7fc804137500 2024-09-16T14:49:54.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:54.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:54.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient(hunting): dump: 2024-09-16T14:49:54.924 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:54.924 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:54.924 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:54.924 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:54.924 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:54.925 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:54.925 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:54.925 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:54.925 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:54.925 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:54.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3210017222 0 0) 0x7fc80411dc40 con 0x7fc804099a70 2024-09-16T14:49:54.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 >> v1:172.21.15.26:6790/0 conn(0x7fc804137500 legacy=0x7fc8041d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:54.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 >> v1:172.21.15.189:6789/0 conn(0x7fc80409bd30 legacy=0x7fc8041ce3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:54.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:49:54.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 monclient: found mon.a 2024-09-16T14:49:54.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:54.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:54.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80413a6c0 con 0x7fc804099a70 2024-09-16T14:49:54.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient: _finish_auth 0 2024-09-16T14:49:54.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:54.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:54.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 5 monclient: authenticate success, global_id 164208 2024-09-16T14:49:54.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 10 monclient: _renew_subs 2024-09-16T14:49:54.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:54.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 1 -- 172.21.15.189:0/3911805289 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8040a1e70 con 0x7fc804099a70 2024-09-16T14:49:54.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 10 monclient: _renew_subs 2024-09-16T14:49:54.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:54.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 1 -- 172.21.15.189:0/3911805289 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc80411b870 con 0x7fc804099a70 2024-09-16T14:49:54.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7f4004380 con 0x7fc804099a70 2024-09-16T14:49:54.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:54.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc7f4004710 con 0x7fc804099a70 2024-09-16T14:49:54.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:54.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc80159a640 4 set_mon_vals no callback set 2024-09-16T14:49:54.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:54.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 10 monclient: dump: 2024-09-16T14:49:54.927 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:54.927 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:54.927 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:54.927 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:54.927 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:54.927 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:54.927 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:54.927 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:54.927 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:54.927 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:54.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:54.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:54.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:54.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc809106640 1 -- 172.21.15.189:0/3911805289 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e4005050 con 0x7fc804099a70 2024-09-16T14:49:54.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc7f40048f0 con 0x7fc804099a70 2024-09-16T14:49:54.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(203..203 src has 169..203) v4 ==== 11516+0+0 (unknown 3196149015 0 0) 0x7fc7f4044fc0 con 0x7fc804099a70 2024-09-16T14:49:54.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.912+0000 7fc80259c640 10 monclient: get_auth_request con 0x7fc7f0036220 auth_method 0 2024-09-16T14:49:54.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.928+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc7f4014420 con 0x7fc804099a70 2024-09-16T14:49:54.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.928+0000 7fc7eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:54.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:54.928+0000 7fc7eb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:55.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.084+0000 7fc809106640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:55.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.084+0000 7fc809106640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:55.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.084+0000 7fc809106640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:55.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.084+0000 7fc809106640 1 -- 172.21.15.189:0/3911805289 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fc7e4005300 con 0x7fc804099a70 2024-09-16T14:49:55.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.676+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}]=0 added bucket h21 type host to location {rack=r1} v204) v1 ==== 168+0+0 (unknown 1720689476 0 0) 0x7fc7f4014240 con 0x7fc804099a70 2024-09-16T14:49:55.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.676+0000 7fc7eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:55.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.676+0000 7fc7eb7fe640 10 monclient: _finish_command 2 = system:0 added bucket h21 type host to location {rack=r1} 2024-09-16T14:49:55.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.776+0000 7fc809106640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:55.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.776+0000 7fc809106640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:55.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.776+0000 7fc809106640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:55.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.776+0000 7fc809106640 1 -- 172.21.15.189:0/3911805289 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fc7e4005a40 con 0x7fc804099a70 2024-09-16T14:49:55.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.780+0000 7fc7eb7fe640 1 -- 172.21.15.189:0/3911805289 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h21' already exists v204) v1 ==== 147+0+0 (unknown 492235675 0 0) 0x7fc7f4014240 con 0x7fc804099a70 2024-09-16T14:49:55.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.780+0000 7fc7eb7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:55.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.780+0000 7fc7eb7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h21' already exists 2024-09-16T14:49:55.782 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h21' already exists 2024-09-16T14:49:55.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.780+0000 7fc809106640 1 -- 172.21.15.189:0/3911805289 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc7f0036220 legacy=0x7fc7f00386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:55.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.780+0000 7fc809106640 10 monclient: shutdown 2024-09-16T14:49:55.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.780+0000 7fc809106640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:55.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.780+0000 7fc809106640 1 -- 172.21.15.189:0/3911805289 >> v1:172.21.15.26:6789/0 conn(0x7fc804099a70 legacy=0x7fc8041c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:55.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.780+0000 7fc809106640 1 -- 172.21.15.189:0/3911805289 shutdown_connections 2024-09-16T14:49:55.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.780+0000 7fc809106640 1 -- 172.21.15.189:0/3911805289 >> 172.21.15.189:0/3911805289 conn(0x7fc8040a3380 msgr2=0x7fc804098a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:55.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.780+0000 7fc809106640 1 -- 172.21.15.189:0/3911805289 shutdown_connections 2024-09-16T14:49:55.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.780+0000 7fc809106640 1 -- 172.21.15.189:0/3911805289 wait complete. 2024-09-16T14:49:55.799 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:55.799 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h22 host rack=r1 2024-09-16T14:49:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e44b95640 10 monclient: get_monmap_and_config 2024-09-16T14:49:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e44b95640 10 monclient: build_initial_monmap 2024-09-16T14:49:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e44b95640 10 monclient: monmap: 2024-09-16T14:49:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:55.919531+0000 2024-09-16T14:49:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:55.919531+0000 2024-09-16T14:49:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:55.920 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:55.920 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:55.920 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:55.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e44b95640 1 Processor -- start 2024-09-16T14:49:55.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e44b95640 1 -- start start 2024-09-16T14:49:55.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e44b95640 10 monclient: init 2024-09-16T14:49:55.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e44b95640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:55.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e44b95640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:49:55.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e44b95640 10 monclient(hunting): picked mon.noname-a con 0x7f5e40131630 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:55.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e44b95640 10 monclient(hunting): picked mon.noname-b con 0x7f5e4009edd0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:55.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e44b95640 10 monclient(hunting): picked mon.noname-c con 0x7f5e4009e950 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:55.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e44b95640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e4009b920 con 0x7f5e40131630 2024-09-16T14:49:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e44b95640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e4009bc00 con 0x7f5e4009e950 2024-09-16T14:49:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e44b95640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e4009cba0 con 0x7f5e4009edd0 2024-09-16T14:49:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e44b95640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e44b95640 10 monclient(hunting): authenticate will time out at 1378.035121s 2024-09-16T14:49:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e3ed9d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f5e4009e950 legacy=0x7f5e4013a290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e3dd9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5e4009edd0 0x7f5e40136b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56924/0 (socket says 172.21.15.189:56924) 2024-09-16T14:49:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e3e59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5e40131630 0x7f5e40131a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:41996/0 (socket says 172.21.15.189:41996) 2024-09-16T14:49:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.916+0000 7f5e3dd9b640 1 -- 172.21.15.189:0/708387813 learned_addr learned my addr 172.21.15.189:0/708387813 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 1 -- 172.21.15.189:0/708387813 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982409703 0 0) 0x7f5e4009cba0 con 0x7f5e4009edd0 2024-09-16T14:49:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient(hunting): _negotiate 2024-09-16T14:49:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:55.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient(hunting): my global_id is 154457 2024-09-16T14:49:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 1 -- 172.21.15.189:0/708387813 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e2c003530 con 0x7f5e4009edd0 2024-09-16T14:49:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 1 -- 172.21.15.189:0/708387813 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765641910 0 0) 0x7f5e4009b920 con 0x7f5e40131630 2024-09-16T14:49:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient(hunting): _negotiate 2024-09-16T14:49:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient(hunting): my global_id is 164211 2024-09-16T14:49:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 1 -- 172.21.15.189:0/708387813 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e4009cba0 con 0x7f5e40131630 2024-09-16T14:49:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 1 -- 172.21.15.189:0/708387813 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3280090904 0 0) 0x7f5e2c003530 con 0x7f5e4009edd0 2024-09-16T14:49:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 1 -- 172.21.15.189:0/708387813 >> v1:172.21.15.189:6789/0 conn(0x7f5e4009e950 legacy=0x7f5e4013a290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 1 -- 172.21.15.189:0/708387813 >> v1:172.21.15.26:6789/0 conn(0x7f5e40131630 legacy=0x7f5e40131a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient: _finish_hunting 0 2024-09-16T14:49:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 1 monclient: found mon.noname-b 2024-09-16T14:49:55.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:49:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 1 -- 172.21.15.189:0/708387813 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4009b420 con 0x7f5e4009edd0 2024-09-16T14:49:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient: _finish_auth 0 2024-09-16T14:49:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient: _check_auth_tickets 2024-09-16T14:49:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 1 -- 172.21.15.189:0/708387813 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5e28001620 con 0x7f5e4009edd0 2024-09-16T14:49:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient: dump: 2024-09-16T14:49:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:55.927 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:55.928 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 5 monclient: authenticate success, global_id 154457 2024-09-16T14:49:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:55.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 1 -- 172.21.15.189:0/708387813 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e28001840 con 0x7f5e4009edd0 2024-09-16T14:49:55.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:55.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:55.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 1 -- 172.21.15.189:0/708387813 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5e28001bd0 con 0x7f5e4009edd0 2024-09-16T14:49:55.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 4 set_mon_vals no callback set 2024-09-16T14:49:55.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:55.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:55.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3d59a640 10 monclient: dump: 2024-09-16T14:49:55.929 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:55.929 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:55.929 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:55.929 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:55.929 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:55.930 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:55.930 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:55.930 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:55.930 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:55.930 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:55.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 10 monclient: shutdown 2024-09-16T14:49:55.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:55.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 1 -- 172.21.15.189:0/708387813 >> v1:172.21.15.26:6790/0 conn(0x7f5e4009edd0 legacy=0x7f5e40136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:55.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 1 -- 172.21.15.189:0/708387813 shutdown_connections 2024-09-16T14:49:55.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 1 -- 172.21.15.189:0/708387813 >> 172.21.15.189:0/708387813 conn(0x7f5e400a3380 msgr2=0x7f5e4009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:55.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 1 -- 172.21.15.189:0/708387813 shutdown_connections 2024-09-16T14:49:55.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 1 -- 172.21.15.189:0/708387813 wait complete. 2024-09-16T14:49:55.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 10 monclient: build_initial_monmap 2024-09-16T14:49:55.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 10 monclient: monmap: 2024-09-16T14:49:55.931 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:55.931 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:55.931 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:55.931 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:55.931 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:55.931 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:55.931 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:55.931 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:55.931 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:55.931 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:55.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 1 Processor -- start 2024-09-16T14:49:55.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 1 -- start start 2024-09-16T14:49:55.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 10 monclient: init 2024-09-16T14:49:55.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:55.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:55.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 10 monclient(hunting): picked mon.noname-c con 0x7f5e4009e950 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:55.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 10 monclient(hunting): picked mon.noname-a con 0x7f5e4009edd0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:55.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 10 monclient(hunting): picked mon.noname-b con 0x7f5e40131630 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:55.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e4011da80 con 0x7f5e4009edd0 2024-09-16T14:49:55.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e401c5a20 con 0x7f5e40131630 2024-09-16T14:49:55.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e4006b320 con 0x7f5e4009e950 2024-09-16T14:49:55.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:55.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e44b95640 10 monclient(hunting): authenticate will time out at 1378.039041s 2024-09-16T14:49:55.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3ed9d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f5e40131630 legacy=0x7f5e401cd530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:55.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3e59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5e4009e950 0x7f5e40130a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56932/0 (socket says 172.21.15.189:56932) 2024-09-16T14:49:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e3e59c640 1 -- 172.21.15.189:0/2367440607 learned_addr learned my addr 172.21.15.189:0/2367440607 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755058572 0 0) 0x7f5e4006b320 con 0x7f5e4009e950 2024-09-16T14:49:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e36ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e36ffd640 10 monclient(hunting): my global_id is 154460 2024-09-16T14:49:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e20003530 con 0x7f5e4009e950 2024-09-16T14:49:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223639505 0 0) 0x7f5e4011da80 con 0x7f5e4009edd0 2024-09-16T14:49:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e36ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:49:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e36ffd640 10 monclient(hunting): my global_id is 164214 2024-09-16T14:49:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.920+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e4006b320 con 0x7f5e4009edd0 2024-09-16T14:49:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 137809224 0 0) 0x7f5e4006b320 con 0x7f5e4009edd0 2024-09-16T14:49:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e4011da80 con 0x7f5e4009edd0 2024-09-16T14:49:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5e28001520 con 0x7f5e4009edd0 2024-09-16T14:49:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 10 monclient(hunting): dump: 2024-09-16T14:49:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:55.936 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:55.937 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4140165659 0 0) 0x7f5e20003530 con 0x7f5e4009e950 2024-09-16T14:49:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e4006b320 con 0x7f5e4009e950 2024-09-16T14:49:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5e30001840 con 0x7f5e4009e950 2024-09-16T14:49:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 10 monclient(hunting): dump: 2024-09-16T14:49:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:55.938 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 15734086 0 0) 0x7f5e4011da80 con 0x7f5e4009edd0 2024-09-16T14:49:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 >> v1:172.21.15.26:6790/0 conn(0x7f5e4009e950 legacy=0x7f5e40130a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 >> v1:172.21.15.189:6789/0 conn(0x7f5e40131630 legacy=0x7f5e401cd530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:49:55.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 1 monclient: found mon.a 2024-09-16T14:49:55.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:55.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:55.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4011f3e0 con 0x7f5e4009edd0 2024-09-16T14:49:55.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 10 monclient: _finish_auth 0 2024-09-16T14:49:55.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:49:55.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:55.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e44b95640 5 monclient: authenticate success, global_id 164214 2024-09-16T14:49:55.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e44b95640 10 monclient: _renew_subs 2024-09-16T14:49:55.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e44b95640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:55.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e44b95640 1 -- 172.21.15.189:0/2367440607 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e400a1120 con 0x7f5e4009edd0 2024-09-16T14:49:55.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e44b95640 10 monclient: _renew_subs 2024-09-16T14:49:55.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e44b95640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:55.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e44b95640 1 -- 172.21.15.189:0/2367440607 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e400a0c50 con 0x7f5e4009edd0 2024-09-16T14:49:55.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e280011e0 con 0x7f5e4009edd0 2024-09-16T14:49:55.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:55.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5e28004c40 con 0x7f5e4009edd0 2024-09-16T14:49:55.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:55.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:55.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e37fff640 4 set_mon_vals no callback set 2024-09-16T14:49:55.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 10 monclient: dump: 2024-09-16T14:49:55.940 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:55.940 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:55.940 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:55.940 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:55.940 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:55.941 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:55.941 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:55.941 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:55.941 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:55.941 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:55.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e44b95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:55.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e44b95640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:55.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e44b95640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:55.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e44b95640 1 -- 172.21.15.189:0/2367440607 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e24005050 con 0x7f5e4009edd0 2024-09-16T14:49:55.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5e2800f6b0 con 0x7f5e4009edd0 2024-09-16T14:49:55.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(204..204 src has 169..204) v4 ==== 11729+0+0 (unknown 1783105127 0 0) 0x7f5e280450e0 con 0x7f5e4009edd0 2024-09-16T14:49:55.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.924+0000 7f5e3e59c640 10 monclient: get_auth_request con 0x7f5e20036230 auth_method 0 2024-09-16T14:49:55.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.936+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5e28014530 con 0x7f5e4009edd0 2024-09-16T14:49:55.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.936+0000 7f5e36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:55.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:55.936+0000 7f5e36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:56.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.100+0000 7f5e44b95640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:56.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.100+0000 7f5e44b95640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:56.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.100+0000 7f5e44b95640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:56.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.100+0000 7f5e44b95640 1 -- 172.21.15.189:0/2367440607 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f5e24005300 con 0x7f5e4009edd0 2024-09-16T14:49:56.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.688+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}]=0 added bucket h22 type host to location {rack=r1} v205) v1 ==== 168+0+0 (unknown 1385010638 0 0) 0x7f5e28014350 con 0x7f5e4009edd0 2024-09-16T14:49:56.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.688+0000 7f5e36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:56.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.688+0000 7f5e36ffd640 10 monclient: _finish_command 2 = system:0 added bucket h22 type host to location {rack=r1} 2024-09-16T14:49:56.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.792+0000 7f5e44b95640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:56.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.792+0000 7f5e44b95640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:56.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.792+0000 7f5e44b95640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:56.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.792+0000 7f5e44b95640 1 -- 172.21.15.189:0/2367440607 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f5e24005a40 con 0x7f5e4009edd0 2024-09-16T14:49:56.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.796+0000 7f5e36ffd640 1 -- 172.21.15.189:0/2367440607 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h22' already exists v205) v1 ==== 147+0+0 (unknown 685753925 0 0) 0x7f5e28014990 con 0x7f5e4009edd0 2024-09-16T14:49:56.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.796+0000 7f5e36ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:56.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.796+0000 7f5e36ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h22' already exists 2024-09-16T14:49:56.798 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h22' already exists 2024-09-16T14:49:56.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.800+0000 7f5e44b95640 1 -- 172.21.15.189:0/2367440607 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5e20036230 legacy=0x7f5e200386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:56.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.800+0000 7f5e44b95640 10 monclient: shutdown 2024-09-16T14:49:56.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.800+0000 7f5e44b95640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:56.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.800+0000 7f5e44b95640 1 -- 172.21.15.189:0/2367440607 >> v1:172.21.15.26:6789/0 conn(0x7f5e4009edd0 legacy=0x7f5e401c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:56.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.800+0000 7f5e44b95640 1 -- 172.21.15.189:0/2367440607 shutdown_connections 2024-09-16T14:49:56.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.800+0000 7f5e44b95640 1 -- 172.21.15.189:0/2367440607 >> 172.21.15.189:0/2367440607 conn(0x7f5e400a3380 msgr2=0x7f5e4009d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:56.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.800+0000 7f5e44b95640 1 -- 172.21.15.189:0/2367440607 shutdown_connections 2024-09-16T14:49:56.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.800+0000 7f5e44b95640 1 -- 172.21.15.189:0/2367440607 wait complete. 2024-09-16T14:49:56.815 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:56.815 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h23 host rack=r1 2024-09-16T14:49:56.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb722b5d640 10 monclient: get_monmap_and_config 2024-09-16T14:49:56.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb722b5d640 10 monclient: build_initial_monmap 2024-09-16T14:49:56.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb722b5d640 10 monclient: monmap: 2024-09-16T14:49:56.926 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:56.926 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:56.926 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:56.926950+0000 2024-09-16T14:49:56.926 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:56.926950+0000 2024-09-16T14:49:56.926 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:56.926 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:56.926 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:56.926 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:56.927 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:56.927 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:56.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb722b5d640 1 Processor -- start 2024-09-16T14:49:56.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb722b5d640 1 -- start start 2024-09-16T14:49:56.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb722b5d640 10 monclient: init 2024-09-16T14:49:56.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb722b5d640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:56.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb722b5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:49:56.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb722b5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb71c073590 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:56.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb722b5d640 10 monclient(hunting): picked mon.noname-c con 0x7fb71c099eb0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:56.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb722b5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb71c13b380 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:56.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb722b5d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb71c098020 con 0x7fb71c073590 2024-09-16T14:49:56.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb722b5d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb71c073970 con 0x7fb71c099eb0 2024-09-16T14:49:56.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb722b5d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb71c0979d0 con 0x7fb71c13b380 2024-09-16T14:49:56.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb722b5d640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:56.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb722b5d640 10 monclient(hunting): authenticate will time out at 1379.043250s 2024-09-16T14:49:56.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb71bfff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fb71c099eb0 legacy=0x7fb71c139c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:56.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb7210fa640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb71c13b380 0x7fb71c13d770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56944/0 (socket says 172.21.15.189:56944) 2024-09-16T14:49:56.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.924+0000 7fb7210fa640 1 -- 172.21.15.189:0/3407253516 learned_addr learned my addr 172.21.15.189:0/3407253516 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:56.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 1 -- 172.21.15.189:0/3407253516 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646337056 0 0) 0x7fb71c0979d0 con 0x7fb71c13b380 2024-09-16T14:49:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient(hunting): my global_id is 154463 2024-09-16T14:49:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 1 -- 172.21.15.189:0/3407253516 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb70c003530 con 0x7fb71c13b380 2024-09-16T14:49:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 1 -- 172.21.15.189:0/3407253516 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132453559 0 0) 0x7fb71c098020 con 0x7fb71c073590 2024-09-16T14:49:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient(hunting): my global_id is 164217 2024-09-16T14:49:56.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 1 -- 172.21.15.189:0/3407253516 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb71c0979d0 con 0x7fb71c073590 2024-09-16T14:49:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 1 -- 172.21.15.189:0/3407253516 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3108800669 0 0) 0x7fb71c0979d0 con 0x7fb71c073590 2024-09-16T14:49:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 1 -- 172.21.15.189:0/3407253516 >> v1:172.21.15.26:6790/0 conn(0x7fb71c13b380 legacy=0x7fb71c13d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 1 -- 172.21.15.189:0/3407253516 >> v1:172.21.15.189:6789/0 conn(0x7fb71c099eb0 legacy=0x7fb71c139c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:49:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 1 monclient: found mon.noname-a 2024-09-16T14:49:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:49:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 1 -- 172.21.15.189:0/3407253516 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71c097cb0 con 0x7fb71c073590 2024-09-16T14:49:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:49:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:56.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:56.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 1 -- 172.21.15.189:0/3407253516 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb710001580 con 0x7fb71c073590 2024-09-16T14:49:56.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:56.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:56.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient: dump: 2024-09-16T14:49:56.940 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:56.940 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:56.940 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:56.941 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 5 monclient: authenticate success, global_id 164217 2024-09-16T14:49:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 1 -- 172.21.15.189:0/3407253516 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7100018a0 con 0x7fb71c073590 2024-09-16T14:49:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 4 set_mon_vals no callback set 2024-09-16T14:49:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 1 -- 172.21.15.189:0/3407253516 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb710001c30 con 0x7fb71c073590 2024-09-16T14:49:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71b7fe640 10 monclient: dump: 2024-09-16T14:49:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:56.942 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 10 monclient: shutdown 2024-09-16T14:49:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 1 -- 172.21.15.189:0/3407253516 >> v1:172.21.15.26:6789/0 conn(0x7fb71c073590 legacy=0x7fb71c0997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 1 -- 172.21.15.189:0/3407253516 shutdown_connections 2024-09-16T14:49:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 1 -- 172.21.15.189:0/3407253516 >> 172.21.15.189:0/3407253516 conn(0x7fb71c09fed0 msgr2=0x7fb71c0a02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 1 -- 172.21.15.189:0/3407253516 shutdown_connections 2024-09-16T14:49:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 1 -- 172.21.15.189:0/3407253516 wait complete. 2024-09-16T14:49:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 10 monclient: build_initial_monmap 2024-09-16T14:49:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 10 monclient: monmap: 2024-09-16T14:49:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:56.944 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 1 Processor -- start 2024-09-16T14:49:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 1 -- start start 2024-09-16T14:49:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 10 monclient: init 2024-09-16T14:49:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:49:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 10 monclient(hunting): picked mon.noname-c con 0x7fb71c073590 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb71c099eb0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb71c13b380 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb71c12ba10 con 0x7fb71c099eb0 2024-09-16T14:49:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb71c12b170 con 0x7fb71c13b380 2024-09-16T14:49:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb71c1292b0 con 0x7fb71c073590 2024-09-16T14:49:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb722b5d640 10 monclient(hunting): authenticate will time out at 1379.046929s 2024-09-16T14:49:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb7210fa640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fb71c13b380 legacy=0x7fb71c1d1550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71bfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb71c099eb0 0x7fb71c128ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:42016/0 (socket says 172.21.15.189:42016) 2024-09-16T14:49:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb71bfff640 1 -- 172.21.15.189:0/2676041993 learned_addr learned my addr 172.21.15.189:0/2676041993 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb7208f9640 1 --1- 172.21.15.189:0/2676041993 >> v1:172.21.15.26:6790/0 conn(0x7fb71c073590 0x7fb71c128490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56950/0 (socket says 172.21.15.189:56950) 2024-09-16T14:49:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533556074 0 0) 0x7fb71c1292b0 con 0x7fb71c073590 2024-09-16T14:49:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb7197fa640 10 monclient(hunting): _negotiate 2024-09-16T14:49:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb7197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb7197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb7197fa640 10 monclient(hunting): my global_id is 154466 2024-09-16T14:49:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb700003530 con 0x7fb71c073590 2024-09-16T14:49:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647128884 0 0) 0x7fb71c12ba10 con 0x7fb71c099eb0 2024-09-16T14:49:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb7197fa640 10 monclient(hunting): _negotiate 2024-09-16T14:49:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb7197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb7197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb7197fa640 10 monclient(hunting): my global_id is 164220 2024-09-16T14:49:56.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.928+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb71c1292b0 con 0x7fb71c099eb0 2024-09-16T14:49:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3753790964 0 0) 0x7fb71c1292b0 con 0x7fb71c099eb0 2024-09-16T14:49:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb71c12ba10 con 0x7fb71c099eb0 2024-09-16T14:49:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb708004160 con 0x7fb71c099eb0 2024-09-16T14:49:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 10 monclient(hunting): dump: 2024-09-16T14:49:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:56.948 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3135294311 0 0) 0x7fb700003530 con 0x7fb71c073590 2024-09-16T14:49:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb71c1292b0 con 0x7fb71c073590 2024-09-16T14:49:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb710001580 con 0x7fb71c073590 2024-09-16T14:49:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 10 monclient(hunting): dump: 2024-09-16T14:49:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:56.948 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:56.949 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3710436252 0 0) 0x7fb71c12ba10 con 0x7fb71c099eb0 2024-09-16T14:49:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 >> v1:172.21.15.26:6790/0 conn(0x7fb71c073590 legacy=0x7fb71c128490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 >> v1:172.21.15.189:6789/0 conn(0x7fb71c13b380 legacy=0x7fb71c1d1550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 10 monclient: _finish_hunting 0 2024-09-16T14:49:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 1 monclient: found mon.a 2024-09-16T14:49:56.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:56.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:56.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71c12b450 con 0x7fb71c099eb0 2024-09-16T14:49:56.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 10 monclient: _finish_auth 0 2024-09-16T14:49:56.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 10 monclient: _check_auth_tickets 2024-09-16T14:49:56.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:56.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb722b5d640 5 monclient: authenticate success, global_id 164220 2024-09-16T14:49:56.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb722b5d640 10 monclient: _renew_subs 2024-09-16T14:49:56.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb722b5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:56.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb722b5d640 1 -- 172.21.15.189:0/2676041993 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71c12b730 con 0x7fb71c099eb0 2024-09-16T14:49:56.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb722b5d640 10 monclient: _renew_subs 2024-09-16T14:49:56.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb722b5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb722b5d640 1 -- 172.21.15.189:0/2676041993 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb71c12a430 con 0x7fb71c099eb0 2024-09-16T14:49:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb708004380 con 0x7fb71c099eb0 2024-09-16T14:49:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb71affd640 4 set_mon_vals no callback set 2024-09-16T14:49:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb708004690 con 0x7fb71c099eb0 2024-09-16T14:49:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 10 monclient: dump: 2024-09-16T14:49:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:56.951 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:56.952 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb722b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb722b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb722b5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb722b5d640 1 -- 172.21.15.189:0/2676041993 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6fc005050 con 0x7fb71c099eb0 2024-09-16T14:49:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb708004870 con 0x7fb71c099eb0 2024-09-16T14:49:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(205..205 src has 169..205) v4 ==== 11942+0+0 (unknown 1999782166 0 0) 0x7fb708045130 con 0x7fb71c099eb0 2024-09-16T14:49:56.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.932+0000 7fb7208f9640 10 monclient: get_auth_request con 0x7fb700036200 auth_method 0 2024-09-16T14:49:56.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.944+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb7080143b0 con 0x7fb71c099eb0 2024-09-16T14:49:56.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.944+0000 7fb7197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:56.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:56.944+0000 7fb7197fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:57.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.096+0000 7fb722b5d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:57.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.096+0000 7fb722b5d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:57.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.096+0000 7fb722b5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:57.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.096+0000 7fb722b5d640 1 -- 172.21.15.189:0/2676041993 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb6fc005300 con 0x7fb71c099eb0 2024-09-16T14:49:57.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.692+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}]=0 added bucket h23 type host to location {rack=r1} v206) v1 ==== 168+0+0 (unknown 2661984446 0 0) 0x7fb708012a50 con 0x7fb71c099eb0 2024-09-16T14:49:57.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.692+0000 7fb7197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:57.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.692+0000 7fb7197fa640 10 monclient: _finish_command 2 = system:0 added bucket h23 type host to location {rack=r1} 2024-09-16T14:49:57.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.796+0000 7fb722b5d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:57.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.796+0000 7fb722b5d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:57.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.796+0000 7fb722b5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:49:57.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.796+0000 7fb722b5d640 1 -- 172.21.15.189:0/2676041993 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb6fc005a40 con 0x7fb71c099eb0 2024-09-16T14:49:57.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.796+0000 7fb7197fa640 1 -- 172.21.15.189:0/2676041993 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h23' already exists v206) v1 ==== 147+0+0 (unknown 171628609 0 0) 0x7fb7080141d0 con 0x7fb71c099eb0 2024-09-16T14:49:57.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.796+0000 7fb7197fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:57.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.796+0000 7fb7197fa640 10 monclient: _finish_command 3 = system:0 bucket 'h23' already exists 2024-09-16T14:49:57.800 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h23' already exists 2024-09-16T14:49:57.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.800+0000 7fb722b5d640 1 -- 172.21.15.189:0/2676041993 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb700036200 legacy=0x7fb7000386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:57.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.800+0000 7fb722b5d640 10 monclient: shutdown 2024-09-16T14:49:57.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.800+0000 7fb722b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:57.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.800+0000 7fb722b5d640 1 -- 172.21.15.189:0/2676041993 >> v1:172.21.15.26:6789/0 conn(0x7fb71c099eb0 legacy=0x7fb71c128ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:57.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.800+0000 7fb722b5d640 1 -- 172.21.15.189:0/2676041993 shutdown_connections 2024-09-16T14:49:57.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.800+0000 7fb722b5d640 1 -- 172.21.15.189:0/2676041993 >> 172.21.15.189:0/2676041993 conn(0x7fb71c09fed0 msgr2=0x7fb71c0a2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:57.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.800+0000 7fb722b5d640 1 -- 172.21.15.189:0/2676041993 shutdown_connections 2024-09-16T14:49:57.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.800+0000 7fb722b5d640 1 -- 172.21.15.189:0/2676041993 wait complete. 2024-09-16T14:49:57.817 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:57.817 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h24 host rack=r1 2024-09-16T14:49:57.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 10 monclient: get_monmap_and_config 2024-09-16T14:49:57.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 10 monclient: build_initial_monmap 2024-09-16T14:49:57.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 10 monclient: monmap: 2024-09-16T14:49:57.937 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:57.937 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:57.938 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:57.937960+0000 2024-09-16T14:49:57.938 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:57.937960+0000 2024-09-16T14:49:57.938 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:57.938 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:57.938 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:57.938 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:57.938 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:57.938 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:57.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 1 Processor -- start 2024-09-16T14:49:57.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 1 -- start start 2024-09-16T14:49:57.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 10 monclient: init 2024-09-16T14:49:57.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:57.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:57.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 10 monclient(hunting): picked mon.noname-c con 0x7efcb4131980 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:57.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 10 monclient(hunting): picked mon.noname-a con 0x7efcb4099300 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:57.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 10 monclient(hunting): picked mon.noname-b con 0x7efcb41377f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:57.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb4099780 con 0x7efcb4099300 2024-09-16T14:49:57.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb40a11c0 con 0x7efcb4131980 2024-09-16T14:49:57.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb40a1350 con 0x7efcb41377f0 2024-09-16T14:49:57.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:57.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 10 monclient(hunting): authenticate will time out at 1380.053695s 2024-09-16T14:49:57.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb2d9d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7efcb4131980 legacy=0x7efcb409bb30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:57.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb359e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7efcb41377f0 0x7efcb4139be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56960/0 (socket says 172.21.15.189:56960) 2024-09-16T14:49:57.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb359e640 1 -- 172.21.15.189:0/1666933290 learned_addr learned my addr 172.21.15.189:0/1666933290 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:57.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 1 -- 172.21.15.189:0/1666933290 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987195079 0 0) 0x7efcb4099780 con 0x7efcb4099300 2024-09-16T14:49:57.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient(hunting): _negotiate 2024-09-16T14:49:57.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:57.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:57.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient(hunting): my global_id is 164223 2024-09-16T14:49:57.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 1 -- 172.21.15.189:0/1666933290 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca0003530 con 0x7efcb4099300 2024-09-16T14:49:57.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 1 -- 172.21.15.189:0/1666933290 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739453628 0 0) 0x7efcb40a1350 con 0x7efcb41377f0 2024-09-16T14:49:57.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient(hunting): _negotiate 2024-09-16T14:49:57.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:57.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:57.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient(hunting): my global_id is 154469 2024-09-16T14:49:57.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 1 -- 172.21.15.189:0/1666933290 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcb4099780 con 0x7efcb41377f0 2024-09-16T14:49:57.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 1 -- 172.21.15.189:0/1666933290 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 399241380 0 0) 0x7efca0003530 con 0x7efcb4099300 2024-09-16T14:49:57.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 1 -- 172.21.15.189:0/1666933290 >> v1:172.21.15.26:6790/0 conn(0x7efcb41377f0 legacy=0x7efcb4139be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:57.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 1 -- 172.21.15.189:0/1666933290 >> v1:172.21.15.189:6789/0 conn(0x7efcb4131980 legacy=0x7efcb409bb30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:57.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient: _finish_hunting 0 2024-09-16T14:49:57.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 1 monclient: found mon.noname-a 2024-09-16T14:49:57.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:57.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:49:57.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 1 -- 172.21.15.189:0/1666933290 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb413e330 con 0x7efcb4099300 2024-09-16T14:49:57.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient: _finish_auth 0 2024-09-16T14:49:57.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient: _check_auth_tickets 2024-09-16T14:49:57.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:57.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 1 -- 172.21.15.189:0/1666933290 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efc9c001620 con 0x7efcb4099300 2024-09-16T14:49:57.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:57.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:57.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient: dump: 2024-09-16T14:49:57.945 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:57.945 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:57.945 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:57.945 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:57.945 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:57.945 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:57.945 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:57.946 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:57.946 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:57.946 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:57.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 5 monclient: authenticate success, global_id 164223 2024-09-16T14:49:57.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:57.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 1 -- 172.21.15.189:0/1666933290 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc9c001900 con 0x7efcb4099300 2024-09-16T14:49:57.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:57.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:57.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 4 set_mon_vals no callback set 2024-09-16T14:49:57.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 1 -- 172.21.15.189:0/1666933290 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efc9c001c90 con 0x7efcb4099300 2024-09-16T14:49:57.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:57.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:57.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb259c640 10 monclient: dump: 2024-09-16T14:49:57.947 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:57.947 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:57.947 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:57.947 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:57.947 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:57.947 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:57.947 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:57.947 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:57.947 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:57.947 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:57.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 10 monclient: shutdown 2024-09-16T14:49:57.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:57.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 1 -- 172.21.15.189:0/1666933290 >> v1:172.21.15.26:6789/0 conn(0x7efcb4099300 legacy=0x7efcb4131d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:57.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 1 -- 172.21.15.189:0/1666933290 shutdown_connections 2024-09-16T14:49:57.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 1 -- 172.21.15.189:0/1666933290 >> 172.21.15.189:0/1666933290 conn(0x7efcb40a4500 msgr2=0x7efcb40a0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:57.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 1 -- 172.21.15.189:0/1666933290 shutdown_connections 2024-09-16T14:49:57.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.936+0000 7efcb9043640 1 -- 172.21.15.189:0/1666933290 wait complete. 2024-09-16T14:49:57.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 10 monclient: build_initial_monmap 2024-09-16T14:49:57.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 10 monclient: monmap: 2024-09-16T14:49:57.948 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:57.948 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:57.948 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:57.949 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:57.949 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:57.949 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:57.949 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:57.949 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:57.949 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:57.949 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:57.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 1 Processor -- start 2024-09-16T14:49:57.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 1 -- start start 2024-09-16T14:49:57.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 10 monclient: init 2024-09-16T14:49:57.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:57.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:49:57.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 10 monclient(hunting): picked mon.noname-a con 0x7efcb4099300 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:57.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 10 monclient(hunting): picked mon.noname-c con 0x7efcb4131980 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:57.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 10 monclient(hunting): picked mon.noname-b con 0x7efcb41377f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:57.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb40a0720 con 0x7efcb4099300 2024-09-16T14:49:57.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb413dc30 con 0x7efcb41377f0 2024-09-16T14:49:57.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb413a200 con 0x7efcb4131980 2024-09-16T14:49:57.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:57.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb359e640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7efcb41377f0 legacy=0x7efcb41d1490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:57.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 10 monclient(hunting): authenticate will time out at 1380.057432s 2024-09-16T14:49:57.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb2d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7efcb4099300 0x7efcb4137bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:42022/0 (socket says 172.21.15.189:42022) 2024-09-16T14:49:57.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcabfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7efcb4131980 0x7efcb41382e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56968/0 (socket says 172.21.15.189:56968) 2024-09-16T14:49:57.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb2d9d640 1 -- 172.21.15.189:0/2864736591 learned_addr learned my addr 172.21.15.189:0/2864736591 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:57.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058683097 0 0) 0x7efcb40a0720 con 0x7efcb4099300 2024-09-16T14:49:57.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:57.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:57.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:57.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient(hunting): my global_id is 164226 2024-09-16T14:49:57.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc94003530 con 0x7efcb4099300 2024-09-16T14:49:57.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831689905 0 0) 0x7efcb413a200 con 0x7efcb4131980 2024-09-16T14:49:57.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:49:57.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:57.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:57.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient(hunting): my global_id is 154472 2024-09-16T14:49:57.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcb40a0720 con 0x7efcb4131980 2024-09-16T14:49:57.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1551440187 0 0) 0x7efc94003530 con 0x7efcb4099300 2024-09-16T14:49:57.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efcb413a200 con 0x7efcb4099300 2024-09-16T14:49:57.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4190067724 0 0) 0x7efcb40a0720 con 0x7efcb4131980 2024-09-16T14:49:57.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc94003530 con 0x7efcb4131980 2024-09-16T14:49:57.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efca4004090 con 0x7efcb4099300 2024-09-16T14:49:57.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:57.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:49:57.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient(hunting): dump: 2024-09-16T14:49:57.956 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:57.956 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:57.956 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:57.956 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:57.956 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:57.957 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:57.957 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:57.957 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:57.957 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:57.960 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:57.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:57.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efc9c001440 con 0x7efcb4131980 2024-09-16T14:49:57.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:57.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:57.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient(hunting): dump: 2024-09-16T14:49:57.961 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:57.962 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:57.962 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:57.962 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:57.962 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:57.962 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:57.962 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:57.963 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:57.963 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:57.963 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:57.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1793845392 0 0) 0x7efc94003530 con 0x7efcb4131980 2024-09-16T14:49:57.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 >> v1:172.21.15.189:6789/0 conn(0x7efcb41377f0 legacy=0x7efcb41d1490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:57.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 >> v1:172.21.15.26:6789/0 conn(0x7efcb4099300 legacy=0x7efcb4137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:57.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:49:57.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 monclient: found mon.c 2024-09-16T14:49:57.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:57.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:57.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb406c3c0 con 0x7efcb4131980 2024-09-16T14:49:57.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient: _finish_auth 0 2024-09-16T14:49:57.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:49:57.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:57.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 5 monclient: authenticate success, global_id 154472 2024-09-16T14:49:57.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 10 monclient: _renew_subs 2024-09-16T14:49:57.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:57.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 1 -- 172.21.15.189:0/2864736591 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb41389f0 con 0x7efcb4131980 2024-09-16T14:49:57.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 10 monclient: _renew_subs 2024-09-16T14:49:57.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:57.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb9043640 1 -- 172.21.15.189:0/2864736591 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb4136450 con 0x7efcb4131980 2024-09-16T14:49:57.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc9c003b00 con 0x7efcb4131980 2024-09-16T14:49:57.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:57.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efc9c003e90 con 0x7efcb4131980 2024-09-16T14:49:57.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:57.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:57.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcb1d9b640 4 set_mon_vals no callback set 2024-09-16T14:49:57.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.940+0000 7efcab7fe640 10 monclient: dump: 2024-09-16T14:49:57.968 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:57.968 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:57.968 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:57.968 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:57.969 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:57.969 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:57.969 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:57.969 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:57.969 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:57.970 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:57.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.944+0000 7efcb9043640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:57.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.944+0000 7efcb9043640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:57.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.944+0000 7efcb9043640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:57.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.944+0000 7efcb9043640 1 -- 172.21.15.189:0/2864736591 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc98005050 con 0x7efcb4131980 2024-09-16T14:49:57.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.944+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7efc9c004070 con 0x7efcb4131980 2024-09-16T14:49:57.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.944+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(206..206 src has 169..206) v4 ==== 12155+0+0 (unknown 1586987946 0 0) 0x7efc9c045320 con 0x7efcb4131980 2024-09-16T14:49:57.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.944+0000 7efcb2d9d640 10 monclient: get_auth_request con 0x7efc940361f0 auth_method 0 2024-09-16T14:49:57.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.956+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7efc9c0145c0 con 0x7efcb4131980 2024-09-16T14:49:57.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.956+0000 7efcab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:57.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:57.956+0000 7efcab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:58.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.120+0000 7efcb9043640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:58.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.120+0000 7efcb9043640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:58.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.120+0000 7efcb9043640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:58.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.120+0000 7efcb9043640 1 -- 172.21.15.189:0/2864736591 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7efc98005300 con 0x7efcb4131980 2024-09-16T14:49:58.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.708+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}]=0 added bucket h24 type host to location {rack=r1} v207) v1 ==== 168+0+0 (unknown 982120666 0 0) 0x7efc9c0143e0 con 0x7efcb4131980 2024-09-16T14:49:58.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.708+0000 7efcab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:58.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.708+0000 7efcab7fe640 10 monclient: _finish_command 2 = system:0 added bucket h24 type host to location {rack=r1} 2024-09-16T14:49:58.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.812+0000 7efcb9043640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:58.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.812+0000 7efcb9043640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:58.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.812+0000 7efcb9043640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:58.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.812+0000 7efcb9043640 1 -- 172.21.15.189:0/2864736591 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7efc98005a40 con 0x7efcb4131980 2024-09-16T14:49:58.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.812+0000 7efcab7fe640 1 -- 172.21.15.189:0/2864736591 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h24' already exists v207) v1 ==== 147+0+0 (unknown 1137548793 0 0) 0x7efc9c0143e0 con 0x7efcb4131980 2024-09-16T14:49:58.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.812+0000 7efcab7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:58.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.812+0000 7efcab7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h24' already exists 2024-09-16T14:49:58.815 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h24' already exists 2024-09-16T14:49:58.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.816+0000 7efcb9043640 1 -- 172.21.15.189:0/2864736591 >> v1:172.21.15.189:6812/3196161582 conn(0x7efc940361f0 legacy=0x7efc940386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:58.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.816+0000 7efcb9043640 10 monclient: shutdown 2024-09-16T14:49:58.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.816+0000 7efcb9043640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:58.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.816+0000 7efcb9043640 1 -- 172.21.15.189:0/2864736591 >> v1:172.21.15.26:6790/0 conn(0x7efcb4131980 legacy=0x7efcb41382e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:58.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.816+0000 7efcb9043640 1 -- 172.21.15.189:0/2864736591 shutdown_connections 2024-09-16T14:49:58.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.816+0000 7efcb9043640 1 -- 172.21.15.189:0/2864736591 >> 172.21.15.189:0/2864736591 conn(0x7efcb40a4500 msgr2=0x7efcb41370e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:58.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.816+0000 7efcb9043640 1 -- 172.21.15.189:0/2864736591 shutdown_connections 2024-09-16T14:49:58.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.816+0000 7efcb9043640 1 -- 172.21.15.189:0/2864736591 wait complete. 2024-09-16T14:49:58.833 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:58.833 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h25 host rack=r1 2024-09-16T14:49:58.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.936+0000 7f72039f0640 10 monclient: get_monmap_and_config 2024-09-16T14:49:58.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.936+0000 7f72039f0640 10 monclient: build_initial_monmap 2024-09-16T14:49:58.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.936+0000 7f72039f0640 10 monclient: monmap: 2024-09-16T14:49:58.941 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:58.941 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:58.941 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:58.941583+0000 2024-09-16T14:49:58.941 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:58.941583+0000 2024-09-16T14:49:58.941 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:58.941 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:58.941 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:58.941 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:58.941 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:58.941 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:58.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 1 Processor -- start 2024-09-16T14:49:58.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 1 -- start start 2024-09-16T14:49:58.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 10 monclient: init 2024-09-16T14:49:58.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:58.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:49:58.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 10 monclient(hunting): picked mon.noname-b con 0x7f71fc099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:58.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 10 monclient(hunting): picked mon.noname-c con 0x7f71fc09c550 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:58.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 10 monclient(hunting): picked mon.noname-a con 0x7f71fc09e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:58.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc099e50 con 0x7f71fc09e6d0 2024-09-16T14:49:58.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc09dbd0 con 0x7f71fc09c550 2024-09-16T14:49:58.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f7200f8b640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f71fc09c550 legacy=0x7f71fc1379d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:58.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc09ee20 con 0x7f71fc099a70 2024-09-16T14:49:58.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:58.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 10 monclient(hunting): authenticate will time out at 1381.057396s 2024-09-16T14:49:58.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f720178c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f71fc099a70 0x7f71fc09dfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52722/0 (socket says 172.21.15.189:52722) 2024-09-16T14:49:58.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f720178c640 1 -- 172.21.15.189:0/1440702722 learned_addr learned my addr 172.21.15.189:0/1440702722 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:58.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 1 -- 172.21.15.189:0/1440702722 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181754610 0 0) 0x7f71fc09ee20 con 0x7f71fc099a70 2024-09-16T14:49:58.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:58.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:58.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:58.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient(hunting): my global_id is 154475 2024-09-16T14:49:58.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 1 -- 172.21.15.189:0/1440702722 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71e8003530 con 0x7f71fc099a70 2024-09-16T14:49:58.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 1 -- 172.21.15.189:0/1440702722 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424570506 0 0) 0x7f71fc099e50 con 0x7f71fc09e6d0 2024-09-16T14:49:58.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:49:58.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:58.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:58.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient(hunting): my global_id is 164229 2024-09-16T14:49:58.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 1 -- 172.21.15.189:0/1440702722 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71fc09ee20 con 0x7f71fc09e6d0 2024-09-16T14:49:58.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 1 -- 172.21.15.189:0/1440702722 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1652264920 0 0) 0x7f71e8003530 con 0x7f71fc099a70 2024-09-16T14:49:58.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 1 -- 172.21.15.189:0/1440702722 >> v1:172.21.15.189:6789/0 conn(0x7f71fc09c550 legacy=0x7f71fc1379d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:58.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 1 -- 172.21.15.189:0/1440702722 >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e6d0 legacy=0x7f71fc13b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:58.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:49:58.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 1 monclient: found mon.noname-b 2024-09-16T14:49:58.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:58.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:49:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 1 -- 172.21.15.189:0/1440702722 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71fc09f100 con 0x7f71fc099a70 2024-09-16T14:49:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient: _finish_auth 0 2024-09-16T14:49:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:49:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 1 -- 172.21.15.189:0/1440702722 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f71f8001620 con 0x7f71fc099a70 2024-09-16T14:49:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 5 monclient: authenticate success, global_id 154475 2024-09-16T14:49:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient: dump: 2024-09-16T14:49:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:58.952 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 1 -- 172.21.15.189:0/1440702722 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71f8001890 con 0x7f71fc099a70 2024-09-16T14:49:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:58.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 4 set_mon_vals no callback set 2024-09-16T14:49:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 1 -- 172.21.15.189:0/1440702722 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f71f8001c20 con 0x7f71fc099a70 2024-09-16T14:49:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f71f3fff640 10 monclient: dump: 2024-09-16T14:49:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:58.953 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 10 monclient: shutdown 2024-09-16T14:49:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 1 -- 172.21.15.189:0/1440702722 >> v1:172.21.15.26:6790/0 conn(0x7f71fc099a70 legacy=0x7f71fc09dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 1 -- 172.21.15.189:0/1440702722 shutdown_connections 2024-09-16T14:49:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 1 -- 172.21.15.189:0/1440702722 >> 172.21.15.189:0/1440702722 conn(0x7f71fc0a3380 msgr2=0x7f71fc09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 1 -- 172.21.15.189:0/1440702722 shutdown_connections 2024-09-16T14:49:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.940+0000 7f72039f0640 1 -- 172.21.15.189:0/1440702722 wait complete. 2024-09-16T14:49:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 monclient: build_initial_monmap 2024-09-16T14:49:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 monclient: monmap: 2024-09-16T14:49:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:58.954 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:58.955 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 1 Processor -- start 2024-09-16T14:49:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 1 -- start start 2024-09-16T14:49:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 monclient: init 2024-09-16T14:49:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:49:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 monclient(hunting): picked mon.noname-a con 0x7f71fc099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 monclient(hunting): picked mon.noname-c con 0x7f71fc09c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 monclient(hunting): picked mon.noname-b con 0x7f71fc09e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc09cf20 con 0x7f71fc099a70 2024-09-16T14:49:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc11a340 con 0x7f71fc09e6d0 2024-09-16T14:49:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc06cd90 con 0x7f71fc09c550 2024-09-16T14:49:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 monclient(hunting): authenticate will time out at 1381.060826s 2024-09-16T14:49:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f7201f8d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f71fc09e6d0 legacy=0x7f71fc1cf700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f7200f8b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f71fc09c550 0x7f71fc13a320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52736/0 (socket says 172.21.15.189:52736) 2024-09-16T14:49:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f7200f8b640 1 -- 172.21.15.189:0/1955153846 learned_addr learned my addr 172.21.15.189:0/1955153846 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615372234 0 0) 0x7f71fc06cd90 con 0x7f71fc09c550 2024-09-16T14:49:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient(hunting): my global_id is 154478 2024-09-16T14:49:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71e0003530 con 0x7f71fc09c550 2024-09-16T14:49:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134979952 0 0) 0x7f71fc09cf20 con 0x7f71fc099a70 2024-09-16T14:49:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:49:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient(hunting): my global_id is 164232 2024-09-16T14:49:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71fc06cd90 con 0x7f71fc099a70 2024-09-16T14:49:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2047602530 0 0) 0x7f71e0003530 con 0x7f71fc09c550 2024-09-16T14:49:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71fc09cf20 con 0x7f71fc09c550 2024-09-16T14:49:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f71ec004150 con 0x7f71fc09c550 2024-09-16T14:49:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient(hunting): dump: 2024-09-16T14:49:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:58.958 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3399262414 0 0) 0x7f71fc06cd90 con 0x7f71fc099a70 2024-09-16T14:49:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71e0003530 con 0x7f71fc099a70 2024-09-16T14:49:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f71f80015e0 con 0x7f71fc099a70 2024-09-16T14:49:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient(hunting): dump: 2024-09-16T14:49:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:58.960 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1314400560 0 0) 0x7f71fc09cf20 con 0x7f71fc09c550 2024-09-16T14:49:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 >> v1:172.21.15.189:6789/0 conn(0x7f71fc09e6d0 legacy=0x7f71fc1cf700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 >> v1:172.21.15.26:6789/0 conn(0x7f71fc099a70 legacy=0x7f71fc139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:49:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 monclient: found mon.c 2024-09-16T14:49:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71fc06f920 con 0x7f71fc09c550 2024-09-16T14:49:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:49:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:49:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 5 monclient: authenticate success, global_id 154478 2024-09-16T14:49:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 monclient: _renew_subs 2024-09-16T14:49:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 1 -- 172.21.15.189:0/1955153846 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71fc13aa30 con 0x7f71fc09c550 2024-09-16T14:49:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 monclient: _renew_subs 2024-09-16T14:49:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 1 -- 172.21.15.189:0/1955153846 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71fc11a620 con 0x7f71fc09c550 2024-09-16T14:49:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71ec004370 con 0x7f71fc09c550 2024-09-16T14:49:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f71ec004700 con 0x7f71fc09c550 2024-09-16T14:49:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:58.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f37fe640 4 set_mon_vals no callback set 2024-09-16T14:49:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 10 monclient: dump: 2024-09-16T14:49:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:58.962 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f72039f0640 1 -- 172.21.15.189:0/1955153846 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71dc005050 con 0x7f71fc09c550 2024-09-16T14:49:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.944+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f71ec0048e0 con 0x7f71fc09c550 2024-09-16T14:49:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.948+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(207..207 src has 169..207) v4 ==== 12368+0+0 (unknown 1392005559 0 0) 0x7f71ec0452d0 con 0x7f71fc09c550 2024-09-16T14:49:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.948+0000 7f720178c640 10 monclient: get_auth_request con 0x7f71e00361d0 auth_method 0 2024-09-16T14:49:58.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.960+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f71ec014400 con 0x7f71fc09c550 2024-09-16T14:49:58.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.960+0000 7f71f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:58.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:58.960+0000 7f71f1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:49:59.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.112+0000 7f72039f0640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:59.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.112+0000 7f72039f0640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:59.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.112+0000 7f72039f0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:59.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.112+0000 7f72039f0640 1 -- 172.21.15.189:0/1955153846 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f71dc005300 con 0x7f71fc09c550 2024-09-16T14:49:59.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.712+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}]=0 added bucket h25 type host to location {rack=r1} v208) v1 ==== 168+0+0 (unknown 3419907274 0 0) 0x7f71ec014220 con 0x7f71fc09c550 2024-09-16T14:49:59.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.712+0000 7f71f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:59.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.712+0000 7f71f1ffb640 10 monclient: _finish_command 2 = system:0 added bucket h25 type host to location {rack=r1} 2024-09-16T14:49:59.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.812+0000 7f72039f0640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:59.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.812+0000 7f72039f0640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:59.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.812+0000 7f72039f0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:59.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.812+0000 7f72039f0640 1 -- 172.21.15.189:0/1955153846 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f71dc005a40 con 0x7f71fc09c550 2024-09-16T14:49:59.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.816+0000 7f71f1ffb640 1 -- 172.21.15.189:0/1955153846 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h25' already exists v208) v1 ==== 147+0+0 (unknown 3652639282 0 0) 0x7f71ec014f60 con 0x7f71fc09c550 2024-09-16T14:49:59.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.816+0000 7f71f1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:49:59.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.816+0000 7f71f1ffb640 10 monclient: _finish_command 3 = system:0 bucket 'h25' already exists 2024-09-16T14:49:59.817 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h25' already exists 2024-09-16T14:49:59.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.816+0000 7f72039f0640 1 -- 172.21.15.189:0/1955153846 >> v1:172.21.15.189:6812/3196161582 conn(0x7f71e00361d0 legacy=0x7f71e0038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:59.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.816+0000 7f72039f0640 10 monclient: shutdown 2024-09-16T14:49:59.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.816+0000 7f72039f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:59.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.816+0000 7f72039f0640 1 -- 172.21.15.189:0/1955153846 >> v1:172.21.15.26:6790/0 conn(0x7f71fc09c550 legacy=0x7f71fc13a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:59.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.816+0000 7f72039f0640 1 -- 172.21.15.189:0/1955153846 shutdown_connections 2024-09-16T14:49:59.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.816+0000 7f72039f0640 1 -- 172.21.15.189:0/1955153846 >> 172.21.15.189:0/1955153846 conn(0x7f71fc0a3380 msgr2=0x7f71fc0a0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:59.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.816+0000 7f72039f0640 1 -- 172.21.15.189:0/1955153846 shutdown_connections 2024-09-16T14:49:59.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.816+0000 7f72039f0640 1 -- 172.21.15.189:0/1955153846 wait complete. 2024-09-16T14:49:59.833 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:49:59.833 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h26 host rack=r1 2024-09-16T14:49:59.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.948+0000 7fba0fd74640 10 monclient: get_monmap_and_config 2024-09-16T14:49:59.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.948+0000 7fba0fd74640 10 monclient: build_initial_monmap 2024-09-16T14:49:59.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.948+0000 7fba0fd74640 10 monclient: monmap: 2024-09-16T14:49:59.951 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:59.952 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:59.952 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:49:59.952532+0000 2024-09-16T14:49:59.952 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:49:59.952532+0000 2024-09-16T14:49:59.952 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:59.952 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:59.952 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:59.952 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:49:59.952 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:49:59.952 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:59.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.948+0000 7fba0fd74640 1 Processor -- start 2024-09-16T14:49:59.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.948+0000 7fba0fd74640 1 -- start start 2024-09-16T14:49:59.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.948+0000 7fba0fd74640 10 monclient: init 2024-09-16T14:49:59.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:59.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:49:59.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 10 monclient(hunting): picked mon.noname-c con 0x7fba08070830 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:59.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 10 monclient(hunting): picked mon.noname-a con 0x7fba08071320 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:59.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 10 monclient(hunting): picked mon.noname-b con 0x7fba08134cf0 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:59.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba08135740 con 0x7fba08071320 2024-09-16T14:49:59.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba08135160 con 0x7fba08070830 2024-09-16T14:49:59.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0db10640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fba08070830 legacy=0x7fba08070c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:59.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0809d810 con 0x7fba08134cf0 2024-09-16T14:49:59.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:59.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 10 monclient(hunting): authenticate will time out at 1382.068594s 2024-09-16T14:49:59.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0e311640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fba08134cf0 0x7fba0813d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52744/0 (socket says 172.21.15.189:52744) 2024-09-16T14:49:59.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0e311640 1 -- 172.21.15.189:0/3551708237 learned_addr learned my addr 172.21.15.189:0/3551708237 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:59.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 1 -- 172.21.15.189:0/3551708237 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250456951 0 0) 0x7fba0809d810 con 0x7fba08134cf0 2024-09-16T14:49:59.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient(hunting): _negotiate 2024-09-16T14:49:59.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:59.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:59.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient(hunting): my global_id is 154481 2024-09-16T14:49:59.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 1 -- 172.21.15.189:0/3551708237 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f4003530 con 0x7fba08134cf0 2024-09-16T14:49:59.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 1 -- 172.21.15.189:0/3551708237 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221006840 0 0) 0x7fba08135740 con 0x7fba08071320 2024-09-16T14:49:59.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient(hunting): _negotiate 2024-09-16T14:49:59.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:59.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:59.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient(hunting): my global_id is 164235 2024-09-16T14:49:59.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 1 -- 172.21.15.189:0/3551708237 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba0809d810 con 0x7fba08071320 2024-09-16T14:49:59.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 1 -- 172.21.15.189:0/3551708237 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3106478480 0 0) 0x7fb9f4003530 con 0x7fba08134cf0 2024-09-16T14:49:59.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 1 -- 172.21.15.189:0/3551708237 >> v1:172.21.15.189:6789/0 conn(0x7fba08070830 legacy=0x7fba08070c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:49:59.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 1 -- 172.21.15.189:0/3551708237 >> v1:172.21.15.26:6789/0 conn(0x7fba08071320 legacy=0x7fba08139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:59.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient: _finish_hunting 0 2024-09-16T14:49:59.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 1 monclient: found mon.noname-b 2024-09-16T14:49:59.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:59.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:49:59.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 1 -- 172.21.15.189:0/3551708237 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0809f1c0 con 0x7fba08134cf0 2024-09-16T14:49:59.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient: _finish_auth 0 2024-09-16T14:49:59.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient: _check_auth_tickets 2024-09-16T14:49:59.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:59.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 1 -- 172.21.15.189:0/3551708237 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fba000019a0 con 0x7fba08134cf0 2024-09-16T14:49:59.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:59.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:49:59.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient: dump: 2024-09-16T14:49:59.959 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:59.959 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:59.959 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:59.959 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:59.959 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:59.960 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:59.960 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:59.960 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:59.960 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:59.960 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:59.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 5 monclient: authenticate success, global_id 154481 2024-09-16T14:49:59.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:49:59.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 1 -- 172.21.15.189:0/3551708237 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba00001c80 con 0x7fba08134cf0 2024-09-16T14:49:59.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:59.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 1 -- 172.21.15.189:0/3551708237 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fba00002010 con 0x7fba08134cf0 2024-09-16T14:49:59.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 10 monclient: get_monmap_and_config success 2024-09-16T14:49:59.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 4 set_mon_vals no callback set 2024-09-16T14:49:59.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:59.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:59.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0cb0e640 10 monclient: dump: 2024-09-16T14:49:59.961 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:59.961 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:59.961 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:59.961 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:59.961 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:59.961 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:59.961 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:59.961 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:59.961 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:59.961 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:59.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 10 monclient: shutdown 2024-09-16T14:49:59.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:49:59.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 1 -- 172.21.15.189:0/3551708237 >> v1:172.21.15.26:6790/0 conn(0x7fba08134cf0 legacy=0x7fba0813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:59.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 1 -- 172.21.15.189:0/3551708237 shutdown_connections 2024-09-16T14:49:59.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 1 -- 172.21.15.189:0/3551708237 >> 172.21.15.189:0/3551708237 conn(0x7fba080a3380 msgr2=0x7fba0809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:49:59.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 1 -- 172.21.15.189:0/3551708237 shutdown_connections 2024-09-16T14:49:59.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 1 -- 172.21.15.189:0/3551708237 wait complete. 2024-09-16T14:49:59.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 10 monclient: build_initial_monmap 2024-09-16T14:49:59.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 10 monclient: monmap: 2024-09-16T14:49:59.962 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:49:59.962 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:49:59.962 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:49:59.962 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:49:59.962 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:49:59.962 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:49:59.963 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:49:59.963 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:49:59.963 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:49:59.963 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:59.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.952+0000 7fba0fd74640 1 Processor -- start 2024-09-16T14:49:59.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 1 -- start start 2024-09-16T14:49:59.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 10 monclient: init 2024-09-16T14:49:59.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 10 monclient: _reopen_session rank -1 2024-09-16T14:49:59.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:49:59.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 10 monclient(hunting): picked mon.noname-c con 0x7fba08070830 addr v1:172.21.15.26:6790/0 2024-09-16T14:49:59.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 10 monclient(hunting): picked mon.noname-b con 0x7fba08071320 addr v1:172.21.15.189:6789/0 2024-09-16T14:49:59.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 10 monclient(hunting): picked mon.noname-a con 0x7fba08134cf0 addr v1:172.21.15.26:6789/0 2024-09-16T14:49:59.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0811ded0 con 0x7fba08134cf0 2024-09-16T14:49:59.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0d30f640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fba08071320 legacy=0x7fba081307f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:49:59.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0811ca20 con 0x7fba08071320 2024-09-16T14:49:59.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba08132ec0 con 0x7fba08070830 2024-09-16T14:49:59.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 10 monclient(hunting): _renew_subs 2024-09-16T14:49:59.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 10 monclient(hunting): authenticate will time out at 1382.072693s 2024-09-16T14:49:59.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0db10640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fba08070830 0x7fba081c9120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52746/0 (socket says 172.21.15.189:52746) 2024-09-16T14:49:59.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0db10640 1 -- 172.21.15.189:0/3685717290 learned_addr learned my addr 172.21.15.189:0/3685717290 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:49:59.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418034071 0 0) 0x7fba08132ec0 con 0x7fba08070830 2024-09-16T14:49:59.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:59.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:59.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:59.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient(hunting): my global_id is 154484 2024-09-16T14:49:59.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ec003530 con 0x7fba08070830 2024-09-16T14:49:59.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803492843 0 0) 0x7fba0811ded0 con 0x7fba08134cf0 2024-09-16T14:49:59.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:49:59.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:49:59.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:49:59.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient(hunting): my global_id is 164238 2024-09-16T14:49:59.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba08132ec0 con 0x7fba08134cf0 2024-09-16T14:49:59.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4198949468 0 0) 0x7fb9ec003530 con 0x7fba08070830 2024-09-16T14:49:59.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba0811ded0 con 0x7fba08070830 2024-09-16T14:49:59.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fba04004160 con 0x7fba08070830 2024-09-16T14:49:59.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:59.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:49:59.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient(hunting): dump: 2024-09-16T14:49:59.969 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:59.969 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:59.969 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:59.969 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:59.969 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:59.970 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:59.970 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:59.970 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:59.970 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:59.970 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:59.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:49:59.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 129458448 0 0) 0x7fba08132ec0 con 0x7fba08134cf0 2024-09-16T14:49:59.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9ec003530 con 0x7fba08134cf0 2024-09-16T14:49:59.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fba00001df0 con 0x7fba08134cf0 2024-09-16T14:49:59.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:59.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:49:59.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient(hunting): dump: 2024-09-16T14:49:59.970 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:59.971 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:59.971 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:59.971 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:59.971 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:59.971 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:59.971 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:59.971 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:59.971 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:59.971 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:59.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1790730915 0 0) 0x7fba0811ded0 con 0x7fba08070830 2024-09-16T14:49:59.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 >> v1:172.21.15.189:6789/0 conn(0x7fba08071320 legacy=0x7fba081307f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:49:59.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 >> v1:172.21.15.26:6789/0 conn(0x7fba08134cf0 legacy=0x7fba081d1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:49:59.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:49:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 monclient: found mon.c 2024-09-16T14:49:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:49:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0811dbf0 con 0x7fba08070830 2024-09-16T14:49:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient: _finish_auth 0 2024-09-16T14:49:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:49:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:49:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 5 monclient: authenticate success, global_id 154484 2024-09-16T14:49:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 10 monclient: _renew_subs 2024-09-16T14:49:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 1 -- 172.21.15.189:0/3685717290 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0811c7a0 con 0x7fba08070830 2024-09-16T14:49:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 10 monclient: _renew_subs 2024-09-16T14:49:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 1 -- 172.21.15.189:0/3685717290 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba081310d0 con 0x7fba08070830 2024-09-16T14:49:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba04004380 con 0x7fba08070830 2024-09-16T14:49:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:49:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fba04004690 con 0x7fba08070830 2024-09-16T14:49:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:49:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:49:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 10 monclient: dump: 2024-09-16T14:49:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:49:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:49:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:49:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:49:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:49:59.974 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:49:59.974 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:49:59.974 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:49:59.974 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:49:59.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fffff640 4 set_mon_vals no callback set 2024-09-16T14:49:59.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:59.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:59.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:49:59.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0fd74640 1 -- 172.21.15.189:0/3685717290 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f0005050 con 0x7fba08070830 2024-09-16T14:49:59.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fba04004870 con 0x7fba08070830 2024-09-16T14:49:59.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(208..208 src has 169..208) v4 ==== 12581+0+0 (unknown 2159324347 0 0) 0x7fba040453a0 con 0x7fba08070830 2024-09-16T14:49:59.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.956+0000 7fba0d30f640 10 monclient: get_auth_request con 0x7fb9ec036250 auth_method 0 2024-09-16T14:49:59.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.968+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fba040143a0 con 0x7fba08070830 2024-09-16T14:49:59.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.968+0000 7fb9fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:49:59.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:49:59.968+0000 7fb9fe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:00.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.132+0000 7fba0fd74640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:00.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.132+0000 7fba0fd74640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:00.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.132+0000 7fba0fd74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:00.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.132+0000 7fba0fd74640 1 -- 172.21.15.189:0/3685717290 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb9f0005300 con 0x7fba08070830 2024-09-16T14:50:00.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.728+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}]=0 added bucket h26 type host to location {rack=r1} v209) v1 ==== 168+0+0 (unknown 4292186688 0 0) 0x7fba040141c0 con 0x7fba08070830 2024-09-16T14:50:00.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.728+0000 7fb9fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:00.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.728+0000 7fb9fe7fc640 10 monclient: _finish_command 2 = system:0 added bucket h26 type host to location {rack=r1} 2024-09-16T14:50:00.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.824+0000 7fba0fd74640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:00.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.824+0000 7fba0fd74640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:00.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.824+0000 7fba0fd74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:00.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.824+0000 7fba0fd74640 1 -- 172.21.15.189:0/3685717290 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb9f0005a40 con 0x7fba08070830 2024-09-16T14:50:00.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.828+0000 7fb9fe7fc640 1 -- 172.21.15.189:0/3685717290 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h26' already exists v209) v1 ==== 147+0+0 (unknown 3963615212 0 0) 0x7fba040185d0 con 0x7fba08070830 2024-09-16T14:50:00.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.828+0000 7fb9fe7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:00.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.828+0000 7fb9fe7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h26' already exists 2024-09-16T14:50:00.829 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h26' already exists 2024-09-16T14:50:00.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.828+0000 7fba0fd74640 1 -- 172.21.15.189:0/3685717290 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb9ec036250 legacy=0x7fb9ec038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:00.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.828+0000 7fba0fd74640 10 monclient: shutdown 2024-09-16T14:50:00.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.828+0000 7fba0fd74640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:00.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.828+0000 7fba0fd74640 1 -- 172.21.15.189:0/3685717290 >> v1:172.21.15.26:6790/0 conn(0x7fba08070830 legacy=0x7fba081c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:00.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.828+0000 7fba0fd74640 1 -- 172.21.15.189:0/3685717290 shutdown_connections 2024-09-16T14:50:00.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.828+0000 7fba0fd74640 1 -- 172.21.15.189:0/3685717290 >> 172.21.15.189:0/3685717290 conn(0x7fba080a3380 msgr2=0x7fba080a1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:00.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.828+0000 7fba0fd74640 1 -- 172.21.15.189:0/3685717290 shutdown_connections 2024-09-16T14:50:00.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.828+0000 7fba0fd74640 1 -- 172.21.15.189:0/3685717290 wait complete. 2024-09-16T14:50:00.846 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:00.846 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h27 host rack=r1 2024-09-16T14:50:00.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f132295a640 10 monclient: get_monmap_and_config 2024-09-16T14:50:00.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f132295a640 10 monclient: build_initial_monmap 2024-09-16T14:50:00.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f132295a640 10 monclient: monmap: 2024-09-16T14:50:00.966 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:00.966 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:00.966 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:00.966981+0000 2024-09-16T14:50:00.966 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:00.966981+0000 2024-09-16T14:50:00.966 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:00.966 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:00.966 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:00.966 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:00.967 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:00.967 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:00.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f132295a640 1 Processor -- start 2024-09-16T14:50:00.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f132295a640 1 -- start start 2024-09-16T14:50:00.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f132295a640 10 monclient: init 2024-09-16T14:50:00.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f132295a640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:00.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f132295a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:00.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f132295a640 10 monclient(hunting): picked mon.noname-c con 0x7f131c0a1b90 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:00.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f132295a640 10 monclient(hunting): picked mon.noname-a con 0x7f131c0a2680 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:00.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f132295a640 10 monclient(hunting): picked mon.noname-b con 0x7f131c099b90 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:00.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f132295a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131c099770 con 0x7f131c0a2680 2024-09-16T14:50:00.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f132295a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131c0a2d60 con 0x7f131c0a1b90 2024-09-16T14:50:00.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f132295a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131c098e60 con 0x7f131c099b90 2024-09-16T14:50:00.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f132295a640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:00.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f132295a640 10 monclient(hunting): authenticate will time out at 1383.082721s 2024-09-16T14:50:00.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f131bfff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f131c0a1b90 legacy=0x7f131c0a1f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:00.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f1320ef7640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f131c099b90 0x7f131c13d5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52754/0 (socket says 172.21.15.189:52754) 2024-09-16T14:50:00.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f1320ef7640 1 -- 172.21.15.189:0/57556176 learned_addr learned my addr 172.21.15.189:0/57556176 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:00.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f131b7fe640 1 --1- 172.21.15.189:0/57556176 >> v1:172.21.15.26:6789/0 conn(0x7f131c0a2680 0x7f131c139e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:53972/0 (socket says 172.21.15.189:53972) 2024-09-16T14:50:00.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f131affd640 1 -- 172.21.15.189:0/57556176 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921296743 0 0) 0x7f131c098e60 con 0x7f131c099b90 2024-09-16T14:50:00.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f131affd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:00.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f131affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:00.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f131affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:00.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f131affd640 10 monclient(hunting): my global_id is 154487 2024-09-16T14:50:00.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f131affd640 1 -- 172.21.15.189:0/57556176 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1300003530 con 0x7f131c099b90 2024-09-16T14:50:00.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f131affd640 1 -- 172.21.15.189:0/57556176 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924431552 0 0) 0x7f131c099770 con 0x7f131c0a2680 2024-09-16T14:50:00.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f131affd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:00.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f131affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:00.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f131affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:00.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f131affd640 10 monclient(hunting): my global_id is 164241 2024-09-16T14:50:00.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.964+0000 7f131affd640 1 -- 172.21.15.189:0/57556176 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f131c098e60 con 0x7f131c0a2680 2024-09-16T14:50:00.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 1 -- 172.21.15.189:0/57556176 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3031232234 0 0) 0x7f1300003530 con 0x7f131c099b90 2024-09-16T14:50:00.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 1 -- 172.21.15.189:0/57556176 >> v1:172.21.15.189:6789/0 conn(0x7f131c0a1b90 legacy=0x7f131c0a1f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 1 -- 172.21.15.189:0/57556176 >> v1:172.21.15.26:6789/0 conn(0x7f131c0a2680 legacy=0x7f131c139e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 1 monclient: found mon.noname-b 2024-09-16T14:50:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 1 -- 172.21.15.189:0/57556176 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c099140 con 0x7f131c099b90 2024-09-16T14:50:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 10 monclient: _finish_auth 0 2024-09-16T14:50:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 5 monclient: authenticate success, global_id 154487 2024-09-16T14:50:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 1 -- 172.21.15.189:0/57556176 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f13140019a0 con 0x7f131c099b90 2024-09-16T14:50:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:00.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:00.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 10 monclient: dump: 2024-09-16T14:50:00.980 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:00.980 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:00.980 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:00.980 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:00.980 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:00.980 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:00.980 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:00.980 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:00.980 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:00.980 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:00.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:00.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 1 -- 172.21.15.189:0/57556176 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1314001c80 con 0x7f131c099b90 2024-09-16T14:50:00.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:00.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 1 -- 172.21.15.189:0/57556176 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1314002010 con 0x7f131c099b90 2024-09-16T14:50:00.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:00.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:00.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131affd640 10 monclient: dump: 2024-09-16T14:50:00.981 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:00.981 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:00.981 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:00.981 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:00.981 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:00.981 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:00.982 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:00.982 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:00.982 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:00.982 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:00.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:00.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 4 set_mon_vals no callback set 2024-09-16T14:50:00.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 10 monclient: shutdown 2024-09-16T14:50:00.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:00.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 1 -- 172.21.15.189:0/57556176 >> v1:172.21.15.26:6790/0 conn(0x7f131c099b90 legacy=0x7f131c13d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:00.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 1 -- 172.21.15.189:0/57556176 shutdown_connections 2024-09-16T14:50:00.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 1 -- 172.21.15.189:0/57556176 >> 172.21.15.189:0/57556176 conn(0x7f131c0a7a80 msgr2=0x7f131c0a3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:00.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 1 -- 172.21.15.189:0/57556176 shutdown_connections 2024-09-16T14:50:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 1 -- 172.21.15.189:0/57556176 wait complete. 2024-09-16T14:50:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 10 monclient: build_initial_monmap 2024-09-16T14:50:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 10 monclient: monmap: 2024-09-16T14:50:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:00.983 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 1 Processor -- start 2024-09-16T14:50:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 1 -- start start 2024-09-16T14:50:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 10 monclient: init 2024-09-16T14:50:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 10 monclient(hunting): picked mon.noname-c con 0x7f131c099b90 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 10 monclient(hunting): picked mon.noname-b con 0x7f131c0a1b90 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 10 monclient(hunting): picked mon.noname-a con 0x7f131c0a2680 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131c122b20 con 0x7f131c0a2680 2024-09-16T14:50:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131c124ba0 con 0x7f131c0a1b90 2024-09-16T14:50:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131c09ed30 con 0x7f131c099b90 2024-09-16T14:50:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:00.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f132295a640 10 monclient(hunting): authenticate will time out at 1383.086520s 2024-09-16T14:50:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131b7fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f131c0a1b90 legacy=0x7f131c1ce960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131bfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f131c099b90 0x7f131c1c9980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52768/0 (socket says 172.21.15.189:52768) 2024-09-16T14:50:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f131bfff640 1 -- 172.21.15.189:0/2861134414 learned_addr learned my addr 172.21.15.189:0/2861134414 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537237339 0 0) 0x7f131c09ed30 con 0x7f131c099b90 2024-09-16T14:50:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 10 monclient(hunting): my global_id is 154490 2024-09-16T14:50:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1304003530 con 0x7f131c099b90 2024-09-16T14:50:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985095639 0 0) 0x7f131c122b20 con 0x7f131c0a2680 2024-09-16T14:50:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 10 monclient(hunting): my global_id is 164244 2024-09-16T14:50:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f131c09ed30 con 0x7f131c0a2680 2024-09-16T14:50:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2526492604 0 0) 0x7f1304003530 con 0x7f131c099b90 2024-09-16T14:50:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f131c122b20 con 0x7f131c099b90 2024-09-16T14:50:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f130c004160 con 0x7f131c099b90 2024-09-16T14:50:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:00.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 10 monclient(hunting): dump: 2024-09-16T14:50:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:00.987 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:00.988 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3332431336 0 0) 0x7f131c09ed30 con 0x7f131c0a2680 2024-09-16T14:50:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1304003530 con 0x7f131c0a2680 2024-09-16T14:50:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1314001df0 con 0x7f131c0a2680 2024-09-16T14:50:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.968+0000 7f1318ff9640 10 monclient(hunting): dump: 2024-09-16T14:50:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:00.989 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2787794368 0 0) 0x7f131c122b20 con 0x7f131c099b90 2024-09-16T14:50:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 >> v1:172.21.15.189:6789/0 conn(0x7f131c0a1b90 legacy=0x7f131c1ce960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 >> v1:172.21.15.26:6789/0 conn(0x7f131c0a2680 legacy=0x7f131c1d2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:50:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 1 monclient: found mon.c 2024-09-16T14:50:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c09e7e0 con 0x7f131c099b90 2024-09-16T14:50:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 10 monclient: _finish_auth 0 2024-09-16T14:50:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:50:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f132295a640 5 monclient: authenticate success, global_id 154490 2024-09-16T14:50:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f132295a640 10 monclient: _renew_subs 2024-09-16T14:50:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f132295a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f132295a640 1 -- 172.21.15.189:0/2861134414 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131c09e380 con 0x7f131c099b90 2024-09-16T14:50:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f132295a640 10 monclient: _renew_subs 2024-09-16T14:50:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f132295a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f132295a640 1 -- 172.21.15.189:0/2861134414 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f131c124e80 con 0x7f131c099b90 2024-09-16T14:50:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f130c004380 con 0x7f131c099b90 2024-09-16T14:50:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1319ffb640 4 set_mon_vals no callback set 2024-09-16T14:50:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f130c004710 con 0x7f131c099b90 2024-09-16T14:50:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 10 monclient: dump: 2024-09-16T14:50:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:00.992 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:00.992 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:00.992 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:00.992 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:00.992 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:00.992 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:00.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f132295a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:00.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f132295a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:00.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f132295a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:00.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f132295a640 1 -- 172.21.15.189:0/2861134414 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f8005050 con 0x7f131c099b90 2024-09-16T14:50:00.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f130c0048f0 con 0x7f131c099b90 2024-09-16T14:50:00.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(209..209 src has 169..209) v4 ==== 12794+0+0 (unknown 675206049 0 0) 0x7f130c0454d0 con 0x7f131c099b90 2024-09-16T14:50:00.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.972+0000 7f131b7fe640 10 monclient: get_auth_request con 0x7f1304036250 auth_method 0 2024-09-16T14:50:00.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.984+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f130c014430 con 0x7f131c099b90 2024-09-16T14:50:00.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.984+0000 7f1318ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:00.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:00.984+0000 7f1318ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:01.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.132+0000 7f132295a640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:01.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.132+0000 7f132295a640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:01.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.132+0000 7f132295a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:01.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.132+0000 7f132295a640 1 -- 172.21.15.189:0/2861134414 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f12f8005300 con 0x7f131c099b90 2024-09-16T14:50:01.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.276+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}]=0 added bucket h27 type host to location {rack=r1} v210) v1 ==== 168+0+0 (unknown 871542576 0 0) 0x7f130c012ad0 con 0x7f131c099b90 2024-09-16T14:50:01.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.276+0000 7f1318ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:01.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.276+0000 7f1318ff9640 10 monclient: _finish_command 2 = system:0 added bucket h27 type host to location {rack=r1} 2024-09-16T14:50:01.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.368+0000 7f132295a640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:01.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.368+0000 7f132295a640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:01.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.368+0000 7f132295a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:01.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.368+0000 7f132295a640 1 -- 172.21.15.189:0/2861134414 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f12f8005a40 con 0x7f131c099b90 2024-09-16T14:50:01.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.368+0000 7f1318ff9640 1 -- 172.21.15.189:0/2861134414 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h27' already exists v210) v1 ==== 147+0+0 (unknown 3470453224 0 0) 0x7f130c014250 con 0x7f131c099b90 2024-09-16T14:50:01.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.368+0000 7f1318ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:01.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.368+0000 7f1318ff9640 10 monclient: _finish_command 3 = system:0 bucket 'h27' already exists 2024-09-16T14:50:01.370 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h27' already exists 2024-09-16T14:50:01.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.368+0000 7f132295a640 1 -- 172.21.15.189:0/2861134414 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1304036250 legacy=0x7f1304038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:01.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.368+0000 7f132295a640 10 monclient: shutdown 2024-09-16T14:50:01.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.368+0000 7f132295a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:01.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.368+0000 7f132295a640 1 -- 172.21.15.189:0/2861134414 >> v1:172.21.15.26:6790/0 conn(0x7f131c099b90 legacy=0x7f131c1c9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:01.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.368+0000 7f132295a640 1 -- 172.21.15.189:0/2861134414 shutdown_connections 2024-09-16T14:50:01.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.368+0000 7f132295a640 1 -- 172.21.15.189:0/2861134414 >> 172.21.15.189:0/2861134414 conn(0x7f131c0a7a80 msgr2=0x7f131c0a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:01.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.368+0000 7f132295a640 1 -- 172.21.15.189:0/2861134414 shutdown_connections 2024-09-16T14:50:01.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.368+0000 7f132295a640 1 -- 172.21.15.189:0/2861134414 wait complete. 2024-09-16T14:50:01.386 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:01.386 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h28 host rack=r1 2024-09-16T14:50:01.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e95a88640 10 monclient: get_monmap_and_config 2024-09-16T14:50:01.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e95a88640 10 monclient: build_initial_monmap 2024-09-16T14:50:01.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e95a88640 10 monclient: monmap: 2024-09-16T14:50:01.506 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:01.506 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:01.506 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:01.506898+0000 2024-09-16T14:50:01.506 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:01.506898+0000 2024-09-16T14:50:01.506 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:01.506 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:01.506 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:01.506 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:01.507 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:01.507 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:01.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e95a88640 1 Processor -- start 2024-09-16T14:50:01.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e95a88640 1 -- start start 2024-09-16T14:50:01.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e95a88640 10 monclient: init 2024-09-16T14:50:01.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e95a88640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:01.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e95a88640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:01.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e95a88640 10 monclient(hunting): picked mon.noname-a con 0x7f3e90099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:01.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e95a88640 10 monclient(hunting): picked mon.noname-b con 0x7f3e9009c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:01.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e95a88640 10 monclient(hunting): picked mon.noname-c con 0x7f3e9009e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:01.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e95a88640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e90099e50 con 0x7f3e90099a70 2024-09-16T14:50:01.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e95a88640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9009dbd0 con 0x7f3e9009e6d0 2024-09-16T14:50:01.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e95a88640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9009ee20 con 0x7f3e9009c550 2024-09-16T14:50:01.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e95a88640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:01.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e95a88640 10 monclient(hunting): authenticate will time out at 1383.622630s 2024-09-16T14:50:01.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e8ffff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f3e9009e6d0 legacy=0x7f3e9013b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:01.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e86ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3e9009c550 0x7f3e901379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52784/0 (socket says 172.21.15.189:52784) 2024-09-16T14:50:01.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e86ffd640 1 -- 172.21.15.189:0/3848605696 learned_addr learned my addr 172.21.15.189:0/3848605696 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:01.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e8effd640 1 -- 172.21.15.189:0/3848605696 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229202425 0 0) 0x7f3e9009ee20 con 0x7f3e9009c550 2024-09-16T14:50:01.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e8effd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:01.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e8effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:01.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:01.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e8effd640 10 monclient(hunting): my global_id is 154493 2024-09-16T14:50:01.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e8effd640 1 -- 172.21.15.189:0/3848605696 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e7c003530 con 0x7f3e9009c550 2024-09-16T14:50:01.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e8effd640 1 -- 172.21.15.189:0/3848605696 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28659789 0 0) 0x7f3e90099e50 con 0x7f3e90099a70 2024-09-16T14:50:01.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e8effd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:01.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e8effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:01.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:01.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e8effd640 10 monclient(hunting): my global_id is 164247 2024-09-16T14:50:01.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.504+0000 7f3e8effd640 1 -- 172.21.15.189:0/3848605696 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e9009ee20 con 0x7f3e90099a70 2024-09-16T14:50:01.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 1 -- 172.21.15.189:0/3848605696 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4165799615 0 0) 0x7f3e9009ee20 con 0x7f3e90099a70 2024-09-16T14:50:01.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 1 -- 172.21.15.189:0/3848605696 >> v1:172.21.15.26:6790/0 conn(0x7f3e9009c550 legacy=0x7f3e901379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:01.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 1 -- 172.21.15.189:0/3848605696 >> v1:172.21.15.189:6789/0 conn(0x7f3e9009e6d0 legacy=0x7f3e9013b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:01.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:01.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 1 monclient: found mon.noname-a 2024-09-16T14:50:01.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:01.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:01.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 1 -- 172.21.15.189:0/3848605696 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9009f100 con 0x7f3e90099a70 2024-09-16T14:50:01.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 10 monclient: _finish_auth 0 2024-09-16T14:50:01.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:01.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:01.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 1 -- 172.21.15.189:0/3848605696 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e80001600 con 0x7f3e90099a70 2024-09-16T14:50:01.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 5 monclient: authenticate success, global_id 164247 2024-09-16T14:50:01.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:01.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:01.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:01.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 10 monclient: dump: 2024-09-16T14:50:01.513 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:01.513 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:01.513 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:01.513 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:01.513 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:01.514 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:01.514 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:01.514 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:01.514 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:01.514 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:01.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:01.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 1 -- 172.21.15.189:0/3848605696 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e800018b0 con 0x7f3e90099a70 2024-09-16T14:50:01.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:01.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:01.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 4 set_mon_vals no callback set 2024-09-16T14:50:01.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 1 -- 172.21.15.189:0/3848605696 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e80001c40 con 0x7f3e90099a70 2024-09-16T14:50:01.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:01.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:01.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8effd640 10 monclient: dump: 2024-09-16T14:50:01.515 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:01.515 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:01.515 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:01.515 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:01.515 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:01.515 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:01.515 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:01.515 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:01.515 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:01.515 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:01.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 10 monclient: shutdown 2024-09-16T14:50:01.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:01.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 1 -- 172.21.15.189:0/3848605696 >> v1:172.21.15.26:6789/0 conn(0x7f3e90099a70 legacy=0x7f3e9009dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:01.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 1 -- 172.21.15.189:0/3848605696 shutdown_connections 2024-09-16T14:50:01.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 1 -- 172.21.15.189:0/3848605696 >> 172.21.15.189:0/3848605696 conn(0x7f3e900a3380 msgr2=0x7f3e9009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:01.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 1 -- 172.21.15.189:0/3848605696 shutdown_connections 2024-09-16T14:50:01.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 1 -- 172.21.15.189:0/3848605696 wait complete. 2024-09-16T14:50:01.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 10 monclient: build_initial_monmap 2024-09-16T14:50:01.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 10 monclient: monmap: 2024-09-16T14:50:01.516 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:01.516 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:01.516 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:01.516 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:01.516 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:01.516 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:01.517 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:01.517 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:01.517 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:01.517 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:01.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 1 Processor -- start 2024-09-16T14:50:01.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 1 -- start start 2024-09-16T14:50:01.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 10 monclient: init 2024-09-16T14:50:01.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:01.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:01.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 10 monclient(hunting): picked mon.noname-a con 0x7f3e90099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:01.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 10 monclient(hunting): picked mon.noname-c con 0x7f3e9009c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:01.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 10 monclient(hunting): picked mon.noname-b con 0x7f3e9009e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:01.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9009cf20 con 0x7f3e90099a70 2024-09-16T14:50:01.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9006b320 con 0x7f3e9009e6d0 2024-09-16T14:50:01.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9013bbd0 con 0x7f3e9009c550 2024-09-16T14:50:01.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:01.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e95a88640 10 monclient(hunting): authenticate will time out at 1383.626691s 2024-09-16T14:50:01.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8ffff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f3e9009e6d0 legacy=0x7f3e901d1720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:01.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e86ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3e9009c550 0x7f3e9009a900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52798/0 (socket says 172.21.15.189:52798) 2024-09-16T14:50:01.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e86ffd640 1 -- 172.21.15.189:0/1371131484 learned_addr learned my addr 172.21.15.189:0/1371131484 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:01.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028776028 0 0) 0x7f3e9013bbd0 con 0x7f3e9009c550 2024-09-16T14:50:01.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:01.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:01.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:01.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8cff9640 10 monclient(hunting): my global_id is 154496 2024-09-16T14:50:01.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e70003530 con 0x7f3e9009c550 2024-09-16T14:50:01.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310903414 0 0) 0x7f3e9009cf20 con 0x7f3e90099a70 2024-09-16T14:50:01.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:01.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:01.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:01.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8cff9640 10 monclient(hunting): my global_id is 164250 2024-09-16T14:50:01.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.508+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e9013bbd0 con 0x7f3e90099a70 2024-09-16T14:50:01.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1006160890 0 0) 0x7f3e70003530 con 0x7f3e9009c550 2024-09-16T14:50:01.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e9009cf20 con 0x7f3e9009c550 2024-09-16T14:50:01.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e78001820 con 0x7f3e9009c550 2024-09-16T14:50:01.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:01.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:01.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 10 monclient(hunting): dump: 2024-09-16T14:50:01.524 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:01.524 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:01.524 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:01.524 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:01.525 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:01.525 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:01.525 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:01.525 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:01.525 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:01.525 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:01.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:01.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3229031865 0 0) 0x7f3e9013bbd0 con 0x7f3e90099a70 2024-09-16T14:50:01.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e70003530 con 0x7f3e90099a70 2024-09-16T14:50:01.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e80001600 con 0x7f3e90099a70 2024-09-16T14:50:01.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:01.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:01.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 10 monclient(hunting): dump: 2024-09-16T14:50:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:01.526 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 687904215 0 0) 0x7f3e9009cf20 con 0x7f3e9009c550 2024-09-16T14:50:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 >> v1:172.21.15.189:6789/0 conn(0x7f3e9009e6d0 legacy=0x7f3e901d1720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 >> v1:172.21.15.26:6789/0 conn(0x7f3e90099a70 legacy=0x7f3e9009a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:01.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:50:01.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 1 monclient: found mon.c 2024-09-16T14:50:01.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:01.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:01.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9013b4d0 con 0x7f3e9009c550 2024-09-16T14:50:01.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 10 monclient: _finish_auth 0 2024-09-16T14:50:01.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:50:01.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:01.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e95a88640 5 monclient: authenticate success, global_id 154496 2024-09-16T14:50:01.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e95a88640 10 monclient: _renew_subs 2024-09-16T14:50:01.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e95a88640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:01.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e95a88640 1 -- 172.21.15.189:0/1371131484 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9009b010 con 0x7f3e9009c550 2024-09-16T14:50:01.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e95a88640 10 monclient: _renew_subs 2024-09-16T14:50:01.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e95a88640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:01.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e95a88640 1 -- 172.21.15.189:0/1371131484 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9013af10 con 0x7f3e9009c550 2024-09-16T14:50:01.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e78001d70 con 0x7f3e9009c550 2024-09-16T14:50:01.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:01.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e78004a10 con 0x7f3e9009c550 2024-09-16T14:50:01.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:01.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:01.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 10 monclient: dump: 2024-09-16T14:50:01.528 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:01.528 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:01.528 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:01.528 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:01.528 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:01.529 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:01.529 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:01.529 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:01.529 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:01.529 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:01.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8e7fc640 4 set_mon_vals no callback set 2024-09-16T14:50:01.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e95a88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:01.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e95a88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:01.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e95a88640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:01.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e95a88640 1 -- 172.21.15.189:0/1371131484 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e74005050 con 0x7f3e9009c550 2024-09-16T14:50:01.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3e7800f480 con 0x7f3e9009c550 2024-09-16T14:50:01.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(210..210 src has 169..210) v4 ==== 13007+0+0 (unknown 15948625 0 0) 0x7f3e780454d0 con 0x7f3e9009c550 2024-09-16T14:50:01.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.512+0000 7f3e8f7fe640 10 monclient: get_auth_request con 0x7f3e700361e0 auth_method 0 2024-09-16T14:50:01.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.524+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3e78014360 con 0x7f3e9009c550 2024-09-16T14:50:01.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.524+0000 7f3e8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:01.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.524+0000 7f3e8cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:01.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.684+0000 7f3e95a88640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:01.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.684+0000 7f3e95a88640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:01.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.684+0000 7f3e95a88640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:01.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:01.684+0000 7f3e95a88640 1 -- 172.21.15.189:0/1371131484 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f3e74005300 con 0x7f3e9009c550 2024-09-16T14:50:02.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.284+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}]=0 added bucket h28 type host to location {rack=r1} v211) v1 ==== 168+0+0 (unknown 2994685049 0 0) 0x7f3e78012a00 con 0x7f3e9009c550 2024-09-16T14:50:02.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.284+0000 7f3e8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:02.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.284+0000 7f3e8cff9640 10 monclient: _finish_command 2 = system:0 added bucket h28 type host to location {rack=r1} 2024-09-16T14:50:02.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.380+0000 7f3e95a88640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:02.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.380+0000 7f3e95a88640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:02.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.380+0000 7f3e95a88640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:02.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.380+0000 7f3e95a88640 1 -- 172.21.15.189:0/1371131484 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f3e74005a40 con 0x7f3e9009c550 2024-09-16T14:50:02.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.380+0000 7f3e8cff9640 1 -- 172.21.15.189:0/1371131484 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h28' already exists v211) v1 ==== 147+0+0 (unknown 2144435356 0 0) 0x7f3e78014180 con 0x7f3e9009c550 2024-09-16T14:50:02.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.380+0000 7f3e8cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:02.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.380+0000 7f3e8cff9640 10 monclient: _finish_command 3 = system:0 bucket 'h28' already exists 2024-09-16T14:50:02.383 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h28' already exists 2024-09-16T14:50:02.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.384+0000 7f3e95a88640 1 -- 172.21.15.189:0/1371131484 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3e700361e0 legacy=0x7f3e700386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:02.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.384+0000 7f3e95a88640 10 monclient: shutdown 2024-09-16T14:50:02.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.384+0000 7f3e95a88640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:02.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.384+0000 7f3e95a88640 1 -- 172.21.15.189:0/1371131484 >> v1:172.21.15.26:6790/0 conn(0x7f3e9009c550 legacy=0x7f3e9009a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:02.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.384+0000 7f3e95a88640 1 -- 172.21.15.189:0/1371131484 shutdown_connections 2024-09-16T14:50:02.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.384+0000 7f3e95a88640 1 -- 172.21.15.189:0/1371131484 >> 172.21.15.189:0/1371131484 conn(0x7f3e900a3380 msgr2=0x7f3e900a0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:02.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.384+0000 7f3e95a88640 1 -- 172.21.15.189:0/1371131484 shutdown_connections 2024-09-16T14:50:02.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.384+0000 7f3e95a88640 1 -- 172.21.15.189:0/1371131484 wait complete. 2024-09-16T14:50:02.400 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:02.400 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h29 host rack=r1 2024-09-16T14:50:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdddcb5640 10 monclient: get_monmap_and_config 2024-09-16T14:50:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdddcb5640 10 monclient: build_initial_monmap 2024-09-16T14:50:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdddcb5640 10 monclient: monmap: 2024-09-16T14:50:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:02.523145+0000 2024-09-16T14:50:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:02.523145+0000 2024-09-16T14:50:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:02.523 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:02.523 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:02.523 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:02.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdddcb5640 1 Processor -- start 2024-09-16T14:50:02.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdddcb5640 1 -- start start 2024-09-16T14:50:02.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdddcb5640 10 monclient: init 2024-09-16T14:50:02.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdddcb5640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdddcb5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdddcb5640 10 monclient(hunting): picked mon.noname-a con 0x7fbdd8099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdddcb5640 10 monclient(hunting): picked mon.noname-b con 0x7fbdd809c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdddcb5640 10 monclient(hunting): picked mon.noname-c con 0x7fbdd809e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdddcb5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd8099e50 con 0x7fbdd8099a70 2024-09-16T14:50:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdddcb5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd809dbd0 con 0x7fbdd809e6d0 2024-09-16T14:50:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdddcb5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd809ee20 con 0x7fbdd809c550 2024-09-16T14:50:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdddcb5640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdddcb5640 10 monclient(hunting): authenticate will time out at 1384.638783s 2024-09-16T14:50:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdd7fff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fbdd809e6d0 legacy=0x7fbdd813b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdd6ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbdd809c550 0x7fbdd81379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52804/0 (socket says 172.21.15.189:52804) 2024-09-16T14:50:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdd6ffd640 1 -- 172.21.15.189:0/462488520 learned_addr learned my addr 172.21.15.189:0/462488520 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdd67fc640 1 -- 172.21.15.189:0/462488520 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171600681 0 0) 0x7fbdd809ee20 con 0x7fbdd809c550 2024-09-16T14:50:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdd67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdd67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdd67fc640 10 monclient(hunting): my global_id is 154499 2024-09-16T14:50:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdd67fc640 1 -- 172.21.15.189:0/462488520 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdc4003530 con 0x7fbdd809c550 2024-09-16T14:50:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdd67fc640 1 -- 172.21.15.189:0/462488520 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398590934 0 0) 0x7fbdd8099e50 con 0x7fbdd8099a70 2024-09-16T14:50:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdd67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdd67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdd67fc640 10 monclient(hunting): my global_id is 164253 2024-09-16T14:50:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.520+0000 7fbdd67fc640 1 -- 172.21.15.189:0/462488520 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd809ee20 con 0x7fbdd8099a70 2024-09-16T14:50:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 1 -- 172.21.15.189:0/462488520 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 943239197 0 0) 0x7fbdc4003530 con 0x7fbdd809c550 2024-09-16T14:50:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 1 -- 172.21.15.189:0/462488520 >> v1:172.21.15.189:6789/0 conn(0x7fbdd809e6d0 legacy=0x7fbdd813b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 1 -- 172.21.15.189:0/462488520 >> v1:172.21.15.26:6789/0 conn(0x7fbdd8099a70 legacy=0x7fbdd809dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:50:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 1 monclient: found mon.noname-b 2024-09-16T14:50:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 1 -- 172.21.15.189:0/462488520 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd809f100 con 0x7fbdd809c550 2024-09-16T14:50:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 10 monclient: _finish_auth 0 2024-09-16T14:50:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 1 -- 172.21.15.189:0/462488520 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbdc0001620 con 0x7fbdd809c550 2024-09-16T14:50:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 10 monclient: dump: 2024-09-16T14:50:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:02.532 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 5 monclient: authenticate success, global_id 154499 2024-09-16T14:50:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 1 -- 172.21.15.189:0/462488520 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdc0001840 con 0x7fbdd809c550 2024-09-16T14:50:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 1 -- 172.21.15.189:0/462488520 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbdc0001b50 con 0x7fbdd809c550 2024-09-16T14:50:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd67fc640 10 monclient: dump: 2024-09-16T14:50:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:02.534 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 4 set_mon_vals no callback set 2024-09-16T14:50:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 10 monclient: shutdown 2024-09-16T14:50:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 1 -- 172.21.15.189:0/462488520 >> v1:172.21.15.26:6790/0 conn(0x7fbdd809c550 legacy=0x7fbdd81379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 1 -- 172.21.15.189:0/462488520 shutdown_connections 2024-09-16T14:50:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 1 -- 172.21.15.189:0/462488520 >> 172.21.15.189:0/462488520 conn(0x7fbdd80a3380 msgr2=0x7fbdd809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 1 -- 172.21.15.189:0/462488520 shutdown_connections 2024-09-16T14:50:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 1 -- 172.21.15.189:0/462488520 wait complete. 2024-09-16T14:50:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 10 monclient: build_initial_monmap 2024-09-16T14:50:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 10 monclient: monmap: 2024-09-16T14:50:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:02.535 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 1 Processor -- start 2024-09-16T14:50:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 1 -- start start 2024-09-16T14:50:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 10 monclient: init 2024-09-16T14:50:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 10 monclient(hunting): picked mon.noname-b con 0x7fbdd8099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 10 monclient(hunting): picked mon.noname-a con 0x7fbdd809c550 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 10 monclient(hunting): picked mon.noname-c con 0x7fbdd809e6d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd77fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fbdd8099a70 legacy=0x7fbdd809a270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd809cf20 con 0x7fbdd809c550 2024-09-16T14:50:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd806faf0 con 0x7fbdd8099a70 2024-09-16T14:50:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd813b580 con 0x7fbdd809e6d0 2024-09-16T14:50:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdddcb5640 10 monclient(hunting): authenticate will time out at 1384.643210s 2024-09-16T14:50:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd6ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbdd809c550 0x7fbdd809a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:54020/0 (socket says 172.21.15.189:54020) 2024-09-16T14:50:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd7fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbdd809e6d0 0x7fbdd81d1720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52820/0 (socket says 172.21.15.189:52820) 2024-09-16T14:50:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdd6ffd640 1 -- 172.21.15.189:0/3029022994 learned_addr learned my addr 172.21.15.189:0/3029022994 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951736049 0 0) 0x7fbdd809cf20 con 0x7fbdd809c550 2024-09-16T14:50:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdcffff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdcffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdcffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdcffff640 10 monclient(hunting): my global_id is 164256 2024-09-16T14:50:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.524+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb8003530 con 0x7fbdd809c550 2024-09-16T14:50:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261337887 0 0) 0x7fbdd813b580 con 0x7fbdd809e6d0 2024-09-16T14:50:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:02.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient(hunting): my global_id is 154502 2024-09-16T14:50:02.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd809cf20 con 0x7fbdd809e6d0 2024-09-16T14:50:02.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 383524403 0 0) 0x7fbdb8003530 con 0x7fbdd809c550 2024-09-16T14:50:02.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdd813b580 con 0x7fbdd809c550 2024-09-16T14:50:02.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbdc00015e0 con 0x7fbdd809c550 2024-09-16T14:50:02.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:02.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:02.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient(hunting): dump: 2024-09-16T14:50:02.541 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:02.541 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:02.541 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:02.541 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:02.541 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:02.541 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:02.541 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:02.541 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:02.541 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:02.541 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:02.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:02.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 832959763 0 0) 0x7fbdd809cf20 con 0x7fbdd809e6d0 2024-09-16T14:50:02.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdb8003530 con 0x7fbdd809e6d0 2024-09-16T14:50:02.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbdd00042c0 con 0x7fbdd809e6d0 2024-09-16T14:50:02.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:02.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:02.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient(hunting): dump: 2024-09-16T14:50:02.542 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:02.542 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:02.542 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:02.542 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:02.542 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:02.542 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:02.543 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:02.543 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:02.543 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:02.543 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:02.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2158150929 0 0) 0x7fbdd813b580 con 0x7fbdd809c550 2024-09-16T14:50:02.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 >> v1:172.21.15.26:6790/0 conn(0x7fbdd809e6d0 legacy=0x7fbdd81d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:02.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 >> v1:172.21.15.189:6789/0 conn(0x7fbdd8099a70 legacy=0x7fbdd809a270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:02.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient: _finish_hunting 0 2024-09-16T14:50:02.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 monclient: found mon.a 2024-09-16T14:50:02.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:02.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:02.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd806b320 con 0x7fbdd809c550 2024-09-16T14:50:02.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient: _finish_auth 0 2024-09-16T14:50:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient: _check_auth_tickets 2024-09-16T14:50:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdddcb5640 5 monclient: authenticate success, global_id 164256 2024-09-16T14:50:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdddcb5640 10 monclient: _renew_subs 2024-09-16T14:50:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdddcb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdddcb5640 1 -- 172.21.15.189:0/3029022994 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd809b090 con 0x7fbdd809c550 2024-09-16T14:50:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdddcb5640 10 monclient: _renew_subs 2024-09-16T14:50:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdddcb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdddcb5640 1 -- 172.21.15.189:0/3029022994 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd813a860 con 0x7fbdd809c550 2024-09-16T14:50:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdc0004af0 con 0x7fbdd809c550 2024-09-16T14:50:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbdc0004e00 con 0x7fbdd809c550 2024-09-16T14:50:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdd5ffb640 4 set_mon_vals no callback set 2024-09-16T14:50:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 10 monclient: dump: 2024-09-16T14:50:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:02.545 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdddcb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdddcb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdddcb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdddcb5640 1 -- 172.21.15.189:0/3029022994 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdbc005050 con 0x7fbdd809c550 2024-09-16T14:50:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbdc0004fe0 con 0x7fbdd809c550 2024-09-16T14:50:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(211..211 src has 169..211) v4 ==== 13220+0+0 (unknown 2347918610 0 0) 0x7fbdc0045820 con 0x7fbdd809c550 2024-09-16T14:50:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.528+0000 7fbdd77fe640 10 monclient: get_auth_request con 0x7fbdb8036060 auth_method 0 2024-09-16T14:50:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.540+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbdc0014870 con 0x7fbdd809c550 2024-09-16T14:50:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.540+0000 7fbdcffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:02.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.540+0000 7fbdcffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:02.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.704+0000 7fbdddcb5640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:02.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.704+0000 7fbdddcb5640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:02.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.704+0000 7fbdddcb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:02.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:02.704+0000 7fbdddcb5640 1 -- 172.21.15.189:0/3029022994 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fbdbc005300 con 0x7fbdd809c550 2024-09-16T14:50:03.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.300+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}]=0 added bucket h29 type host to location {rack=r1} v212) v1 ==== 168+0+0 (unknown 467631842 0 0) 0x7fbdc0014460 con 0x7fbdd809c550 2024-09-16T14:50:03.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.300+0000 7fbdcffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:03.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.300+0000 7fbdcffff640 10 monclient: _finish_command 2 = system:0 added bucket h29 type host to location {rack=r1} 2024-09-16T14:50:03.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.400+0000 7fbdddcb5640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:03.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.400+0000 7fbdddcb5640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:03.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.400+0000 7fbdddcb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:03.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.400+0000 7fbdddcb5640 1 -- 172.21.15.189:0/3029022994 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fbdbc005a40 con 0x7fbdd809c550 2024-09-16T14:50:03.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.400+0000 7fbdcffff640 1 -- 172.21.15.189:0/3029022994 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h29' already exists v212) v1 ==== 147+0+0 (unknown 261254474 0 0) 0x7fbdc0014640 con 0x7fbdd809c550 2024-09-16T14:50:03.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.400+0000 7fbdcffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:03.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.400+0000 7fbdcffff640 10 monclient: _finish_command 3 = system:0 bucket 'h29' already exists 2024-09-16T14:50:03.404 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h29' already exists 2024-09-16T14:50:03.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.404+0000 7fbdddcb5640 1 -- 172.21.15.189:0/3029022994 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbdb8036060 legacy=0x7fbdb8038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:03.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.404+0000 7fbdddcb5640 10 monclient: shutdown 2024-09-16T14:50:03.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.404+0000 7fbdddcb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:03.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.404+0000 7fbdddcb5640 1 -- 172.21.15.189:0/3029022994 >> v1:172.21.15.26:6789/0 conn(0x7fbdd809c550 legacy=0x7fbdd809a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:03.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.404+0000 7fbdddcb5640 1 -- 172.21.15.189:0/3029022994 shutdown_connections 2024-09-16T14:50:03.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.404+0000 7fbdddcb5640 1 -- 172.21.15.189:0/3029022994 >> 172.21.15.189:0/3029022994 conn(0x7fbdd80a3380 msgr2=0x7fbdd80a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:03.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.404+0000 7fbdddcb5640 1 -- 172.21.15.189:0/3029022994 shutdown_connections 2024-09-16T14:50:03.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.404+0000 7fbdddcb5640 1 -- 172.21.15.189:0/3029022994 wait complete. 2024-09-16T14:50:03.421 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:03.421 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h30 host rack=r1 2024-09-16T14:50:03.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.532+0000 7fecaae49640 10 monclient: get_monmap_and_config 2024-09-16T14:50:03.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.532+0000 7fecaae49640 10 monclient: build_initial_monmap 2024-09-16T14:50:03.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.532+0000 7fecaae49640 10 monclient: monmap: 2024-09-16T14:50:03.536 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:03.536 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:03.537496+0000 2024-09-16T14:50:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:03.537496+0000 2024-09-16T14:50:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:03.537 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:03.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 1 Processor -- start 2024-09-16T14:50:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 1 -- start start 2024-09-16T14:50:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 10 monclient: init 2024-09-16T14:50:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 10 monclient(hunting): picked mon.noname-b con 0x7feca40a1a90 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 10 monclient(hunting): picked mon.noname-c con 0x7feca40a2580 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 10 monclient(hunting): picked mon.noname-a con 0x7feca409fd40 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca409df00 con 0x7feca409fd40 2024-09-16T14:50:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca409f0a0 con 0x7feca40a2580 2024-09-16T14:50:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca40a3040 con 0x7feca40a1a90 2024-09-16T14:50:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 10 monclient(hunting): authenticate will time out at 1385.653088s 2024-09-16T14:50:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca3fff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7feca40a2580 legacy=0x7feca4139df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca8be5640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7feca40a1a90 0x7feca40a1e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52834/0 (socket says 172.21.15.189:52834) 2024-09-16T14:50:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca8be5640 1 -- 172.21.15.189:0/2439701849 learned_addr learned my addr 172.21.15.189:0/2439701849 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 1 -- 172.21.15.189:0/2439701849 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985231933 0 0) 0x7feca40a3040 con 0x7feca40a1a90 2024-09-16T14:50:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient(hunting): my global_id is 154505 2024-09-16T14:50:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 1 -- 172.21.15.189:0/2439701849 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec8c003530 con 0x7feca40a1a90 2024-09-16T14:50:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 1 -- 172.21.15.189:0/2439701849 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302321846 0 0) 0x7feca409df00 con 0x7feca409fd40 2024-09-16T14:50:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient(hunting): my global_id is 164259 2024-09-16T14:50:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 1 -- 172.21.15.189:0/2439701849 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca40a3040 con 0x7feca409fd40 2024-09-16T14:50:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 1 -- 172.21.15.189:0/2439701849 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1602494604 0 0) 0x7fec8c003530 con 0x7feca40a1a90 2024-09-16T14:50:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 1 -- 172.21.15.189:0/2439701849 >> v1:172.21.15.189:6789/0 conn(0x7feca40a2580 legacy=0x7feca4139df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 1 -- 172.21.15.189:0/2439701849 >> v1:172.21.15.26:6789/0 conn(0x7feca409fd40 legacy=0x7feca413d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 1 monclient: found mon.noname-b 2024-09-16T14:50:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 1 -- 172.21.15.189:0/2439701849 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca409eae0 con 0x7feca40a1a90 2024-09-16T14:50:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 1 -- 172.21.15.189:0/2439701849 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec94001620 con 0x7feca40a1a90 2024-09-16T14:50:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient: dump: 2024-09-16T14:50:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:03.544 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 5 monclient: authenticate success, global_id 154505 2024-09-16T14:50:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 1 -- 172.21.15.189:0/2439701849 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec94001890 con 0x7feca40a1a90 2024-09-16T14:50:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 1 -- 172.21.15.189:0/2439701849 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec94001c20 con 0x7feca40a1a90 2024-09-16T14:50:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 4 set_mon_vals no callback set 2024-09-16T14:50:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7feca37fe640 10 monclient: dump: 2024-09-16T14:50:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:03.545 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 10 monclient: shutdown 2024-09-16T14:50:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 1 -- 172.21.15.189:0/2439701849 >> v1:172.21.15.26:6790/0 conn(0x7feca40a1a90 legacy=0x7feca40a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 1 -- 172.21.15.189:0/2439701849 shutdown_connections 2024-09-16T14:50:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 1 -- 172.21.15.189:0/2439701849 >> 172.21.15.189:0/2439701849 conn(0x7feca40a7a70 msgr2=0x7feca40a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 1 -- 172.21.15.189:0/2439701849 shutdown_connections 2024-09-16T14:50:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 1 -- 172.21.15.189:0/2439701849 wait complete. 2024-09-16T14:50:03.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 10 monclient: build_initial_monmap 2024-09-16T14:50:03.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.536+0000 7fecaae49640 10 monclient: monmap: 2024-09-16T14:50:03.550 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:03.550 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:03.550 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:03.550 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:03.550 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:03.551 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:03.551 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:03.551 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:03.551 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:03.551 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:03.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 1 Processor -- start 2024-09-16T14:50:03.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 1 -- start start 2024-09-16T14:50:03.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 10 monclient: init 2024-09-16T14:50:03.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:03.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:50:03.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 10 monclient(hunting): picked mon.noname-a con 0x7feca409fd40 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:03.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 10 monclient(hunting): picked mon.noname-b con 0x7feca40a1a90 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:03.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 10 monclient(hunting): picked mon.noname-c con 0x7feca40a2580 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca4123700 con 0x7feca409fd40 2024-09-16T14:50:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca40a6390 con 0x7feca40a1a90 2024-09-16T14:50:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca409e280 con 0x7feca40a2580 2024-09-16T14:50:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 10 monclient(hunting): authenticate will time out at 1385.656772s 2024-09-16T14:50:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca3fff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7feca40a1a90 legacy=0x7feca41ce620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca8be5640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7feca409fd40 0x7feca41c9640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:54040/0 (socket says 172.21.15.189:54040) 2024-09-16T14:50:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca8be5640 1 -- 172.21.15.189:0/873268507 learned_addr learned my addr 172.21.15.189:0/873268507 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930881538 0 0) 0x7feca409e280 con 0x7feca40a2580 2024-09-16T14:50:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:50:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:03.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient(hunting): my global_id is 154508 2024-09-16T14:50:03.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec90003530 con 0x7feca40a2580 2024-09-16T14:50:03.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747560120 0 0) 0x7feca4123700 con 0x7feca409fd40 2024-09-16T14:50:03.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:50:03.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:03.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:03.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient(hunting): my global_id is 164262 2024-09-16T14:50:03.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca409e280 con 0x7feca409fd40 2024-09-16T14:50:03.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3336523256 0 0) 0x7fec90003530 con 0x7feca40a2580 2024-09-16T14:50:03.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feca4123700 con 0x7feca40a2580 2024-09-16T14:50:03.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec9c001c40 con 0x7feca40a2580 2024-09-16T14:50:03.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:03.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:03.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient(hunting): dump: 2024-09-16T14:50:03.554 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:03.554 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:03.554 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:03.554 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:03.554 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:03.554 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:03.554 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:03.555 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:03.555 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:03.555 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:03.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:03.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3980447882 0 0) 0x7feca409e280 con 0x7feca409fd40 2024-09-16T14:50:03.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec90003530 con 0x7feca409fd40 2024-09-16T14:50:03.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec940047f0 con 0x7feca409fd40 2024-09-16T14:50:03.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:03.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:03.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient(hunting): dump: 2024-09-16T14:50:03.555 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:03.555 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:03.555 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:03.556 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:03.556 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:03.556 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:03.556 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:03.556 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:03.556 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:03.556 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:03.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1740029690 0 0) 0x7feca4123700 con 0x7feca40a2580 2024-09-16T14:50:03.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 >> v1:172.21.15.189:6789/0 conn(0x7feca40a1a90 legacy=0x7feca41ce620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:03.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 >> v1:172.21.15.26:6789/0 conn(0x7feca409fd40 legacy=0x7feca41c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:03.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient: _finish_hunting 0 2024-09-16T14:50:03.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 monclient: found mon.c 2024-09-16T14:50:03.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:03.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:03.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca4122710 con 0x7feca40a2580 2024-09-16T14:50:03.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient: _finish_auth 0 2024-09-16T14:50:03.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient: _check_auth_tickets 2024-09-16T14:50:03.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:03.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 5 monclient: authenticate success, global_id 154508 2024-09-16T14:50:03.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 10 monclient: _renew_subs 2024-09-16T14:50:03.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:03.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 1 -- 172.21.15.189:0/873268507 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca4123c30 con 0x7feca40a2580 2024-09-16T14:50:03.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 10 monclient: _renew_subs 2024-09-16T14:50:03.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:03.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 1 -- 172.21.15.189:0/873268507 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feca4121b40 con 0x7feca40a2580 2024-09-16T14:50:03.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec9c0039a0 con 0x7feca40a2580 2024-09-16T14:50:03.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:03.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec9c002110 con 0x7feca40a2580 2024-09-16T14:50:03.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:03.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:03.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca27fc640 4 set_mon_vals no callback set 2024-09-16T14:50:03.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7feca17fa640 10 monclient: dump: 2024-09-16T14:50:03.559 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:03.559 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:03.559 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:03.559 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:03.559 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:03.559 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:03.559 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:03.559 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:03.559 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:03.559 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:03.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:03.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:03.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:03.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.540+0000 7fecaae49640 1 -- 172.21.15.189:0/873268507 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec84005050 con 0x7feca40a2580 2024-09-16T14:50:03.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.544+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fec9c00e5c0 con 0x7feca40a2580 2024-09-16T14:50:03.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.544+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(212..212 src has 169..212) v4 ==== 13433+0+0 (unknown 4292643935 0 0) 0x7fec9c044680 con 0x7feca40a2580 2024-09-16T14:50:03.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.544+0000 7feca8be5640 10 monclient: get_auth_request con 0x7fec90036230 auth_method 0 2024-09-16T14:50:03.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.556+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fec9c013380 con 0x7feca40a2580 2024-09-16T14:50:03.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.556+0000 7feca17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:03.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.556+0000 7feca17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:03.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.700+0000 7fecaae49640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:03.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.700+0000 7fecaae49640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:03.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.700+0000 7fecaae49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:03.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:03.700+0000 7fecaae49640 1 -- 172.21.15.189:0/873268507 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fec84005300 con 0x7feca40a2580 2024-09-16T14:50:04.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.308+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}]=0 added bucket h30 type host to location {rack=r1} v213) v1 ==== 168+0+0 (unknown 2451073888 0 0) 0x7fec9c011a20 con 0x7feca40a2580 2024-09-16T14:50:04.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.308+0000 7feca17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:04.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.308+0000 7feca17fa640 10 monclient: _finish_command 2 = system:0 added bucket h30 type host to location {rack=r1} 2024-09-16T14:50:04.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.408+0000 7fecaae49640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:04.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.408+0000 7fecaae49640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:04.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.408+0000 7fecaae49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:04.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.408+0000 7fecaae49640 1 -- 172.21.15.189:0/873268507 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fec84005a40 con 0x7feca40a2580 2024-09-16T14:50:04.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.408+0000 7feca17fa640 1 -- 172.21.15.189:0/873268507 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h30' already exists v213) v1 ==== 147+0+0 (unknown 1877499307 0 0) 0x7fec9c0131a0 con 0x7feca40a2580 2024-09-16T14:50:04.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.408+0000 7feca17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:04.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.408+0000 7feca17fa640 10 monclient: _finish_command 3 = system:0 bucket 'h30' already exists 2024-09-16T14:50:04.412 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h30' already exists 2024-09-16T14:50:04.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.412+0000 7fecaae49640 1 -- 172.21.15.189:0/873268507 >> v1:172.21.15.189:6812/3196161582 conn(0x7fec90036230 legacy=0x7fec900386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:04.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.412+0000 7fecaae49640 10 monclient: shutdown 2024-09-16T14:50:04.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.412+0000 7fecaae49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:04.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.412+0000 7fecaae49640 1 -- 172.21.15.189:0/873268507 >> v1:172.21.15.26:6790/0 conn(0x7feca40a2580 legacy=0x7feca41d1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:04.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.412+0000 7fecaae49640 1 -- 172.21.15.189:0/873268507 shutdown_connections 2024-09-16T14:50:04.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.412+0000 7fecaae49640 1 -- 172.21.15.189:0/873268507 >> 172.21.15.189:0/873268507 conn(0x7feca40a7a70 msgr2=0x7feca406b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:04.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.412+0000 7fecaae49640 1 -- 172.21.15.189:0/873268507 shutdown_connections 2024-09-16T14:50:04.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.412+0000 7fecaae49640 1 -- 172.21.15.189:0/873268507 wait complete. 2024-09-16T14:50:04.430 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:04.430 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h31 host rack=r1 2024-09-16T14:50:04.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2af589f640 10 monclient: get_monmap_and_config 2024-09-16T14:50:04.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2af589f640 10 monclient: build_initial_monmap 2024-09-16T14:50:04.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2af589f640 10 monclient: monmap: 2024-09-16T14:50:04.550 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:04.550 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:04.550 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:04.550834+0000 2024-09-16T14:50:04.550 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:04.550834+0000 2024-09-16T14:50:04.550 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:04.550 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:04.550 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:04.550 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:04.550 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:04.550 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:04.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2af589f640 1 Processor -- start 2024-09-16T14:50:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2af589f640 1 -- start start 2024-09-16T14:50:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2af589f640 10 monclient: init 2024-09-16T14:50:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2af589f640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2af589f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2af589f640 10 monclient(hunting): picked mon.noname-b con 0x7f2af0099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2af589f640 10 monclient(hunting): picked mon.noname-a con 0x7f2af009c550 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2af589f640 10 monclient(hunting): picked mon.noname-c con 0x7f2af009e6d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2af589f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af0099e50 con 0x7f2af009c550 2024-09-16T14:50:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2af589f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af009dbd0 con 0x7f2af009e6d0 2024-09-16T14:50:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2af589f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af009ee20 con 0x7f2af0099a70 2024-09-16T14:50:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2af589f640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2af589f640 10 monclient(hunting): authenticate will time out at 1386.667062s 2024-09-16T14:50:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2aef7fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2af009e6d0 legacy=0x7f2af013b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2aeeffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f2af0099a70 0x7f2af009dfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52844/0 (socket says 172.21.15.189:52844) 2024-09-16T14:50:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2aeeffd640 1 -- 172.21.15.189:0/1054171781 learned_addr learned my addr 172.21.15.189:0/1054171781 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2aee7fc640 1 -- 172.21.15.189:0/1054171781 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629309370 0 0) 0x7f2af009ee20 con 0x7f2af0099a70 2024-09-16T14:50:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2aee7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2aee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2aee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2aee7fc640 10 monclient(hunting): my global_id is 154511 2024-09-16T14:50:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2aee7fc640 1 -- 172.21.15.189:0/1054171781 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2adc003530 con 0x7f2af0099a70 2024-09-16T14:50:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2aee7fc640 1 -- 172.21.15.189:0/1054171781 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638677341 0 0) 0x7f2af0099e50 con 0x7f2af009c550 2024-09-16T14:50:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2aee7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:04.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2aee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:04.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2aee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:04.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.548+0000 7f2aee7fc640 10 monclient(hunting): my global_id is 164265 2024-09-16T14:50:04.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 1 -- 172.21.15.189:0/1054171781 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af009ee20 con 0x7f2af009c550 2024-09-16T14:50:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 1 -- 172.21.15.189:0/1054171781 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2312208185 0 0) 0x7f2adc003530 con 0x7f2af0099a70 2024-09-16T14:50:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 1 -- 172.21.15.189:0/1054171781 >> v1:172.21.15.189:6789/0 conn(0x7f2af009e6d0 legacy=0x7f2af013b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 1 -- 172.21.15.189:0/1054171781 >> v1:172.21.15.26:6789/0 conn(0x7f2af009c550 legacy=0x7f2af01379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:50:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 1 monclient: found mon.noname-b 2024-09-16T14:50:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 1 -- 172.21.15.189:0/1054171781 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af009f100 con 0x7f2af0099a70 2024-09-16T14:50:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 10 monclient: _finish_auth 0 2024-09-16T14:50:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 1 -- 172.21.15.189:0/1054171781 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2ae0001620 con 0x7f2af0099a70 2024-09-16T14:50:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 10 monclient: dump: 2024-09-16T14:50:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:04.558 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 5 monclient: authenticate success, global_id 154511 2024-09-16T14:50:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 1 -- 172.21.15.189:0/1054171781 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ae0001890 con 0x7f2af0099a70 2024-09-16T14:50:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 1 -- 172.21.15.189:0/1054171781 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2ae0001c20 con 0x7f2af0099a70 2024-09-16T14:50:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aee7fc640 10 monclient: dump: 2024-09-16T14:50:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:04.560 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 4 set_mon_vals no callback set 2024-09-16T14:50:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 10 monclient: shutdown 2024-09-16T14:50:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 1 -- 172.21.15.189:0/1054171781 >> v1:172.21.15.26:6790/0 conn(0x7f2af0099a70 legacy=0x7f2af009dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 1 -- 172.21.15.189:0/1054171781 shutdown_connections 2024-09-16T14:50:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 1 -- 172.21.15.189:0/1054171781 >> 172.21.15.189:0/1054171781 conn(0x7f2af00a3380 msgr2=0x7f2af009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:04.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 1 -- 172.21.15.189:0/1054171781 shutdown_connections 2024-09-16T14:50:04.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 1 -- 172.21.15.189:0/1054171781 wait complete. 2024-09-16T14:50:04.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 10 monclient: build_initial_monmap 2024-09-16T14:50:04.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 10 monclient: monmap: 2024-09-16T14:50:04.564 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:04.564 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:04.564 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:04.564 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:04.564 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:04.565 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:04.565 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:04.565 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:04.565 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:04.565 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:04.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 1 Processor -- start 2024-09-16T14:50:04.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 1 -- start start 2024-09-16T14:50:04.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 10 monclient: init 2024-09-16T14:50:04.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:04.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:04.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 10 monclient(hunting): picked mon.noname-b con 0x7f2af0099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:04.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 10 monclient(hunting): picked mon.noname-c con 0x7f2af009c550 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:04.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 10 monclient(hunting): picked mon.noname-a con 0x7f2af009e6d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:04.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af009cf20 con 0x7f2af009e6d0 2024-09-16T14:50:04.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af006fa80 con 0x7f2af0099a70 2024-09-16T14:50:04.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af006b320 con 0x7f2af009c550 2024-09-16T14:50:04.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:04.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aeeffd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2af0099a70 legacy=0x7f2af0139bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:04.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2af589f640 10 monclient(hunting): authenticate will time out at 1386.670742s 2024-09-16T14:50:04.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2ae7fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f2af009c550 0x7f2af013a2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52850/0 (socket says 172.21.15.189:52850) 2024-09-16T14:50:04.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2ae7fff640 1 -- 172.21.15.189:0/2023315618 learned_addr learned my addr 172.21.15.189:0/2023315618 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:04.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586479727 0 0) 0x7f2af006b320 con 0x7f2af009c550 2024-09-16T14:50:04.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:04.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:04.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:04.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 10 monclient(hunting): my global_id is 154514 2024-09-16T14:50:04.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad0003530 con 0x7f2af009c550 2024-09-16T14:50:04.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034496638 0 0) 0x7f2af009cf20 con 0x7f2af009e6d0 2024-09-16T14:50:04.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:04.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:04.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:04.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 10 monclient(hunting): my global_id is 164268 2024-09-16T14:50:04.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af006b320 con 0x7f2af009e6d0 2024-09-16T14:50:04.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3689841312 0 0) 0x7f2ad0003530 con 0x7f2af009c550 2024-09-16T14:50:04.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2af009cf20 con 0x7f2af009c550 2024-09-16T14:50:04.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2ad8001940 con 0x7f2af009c550 2024-09-16T14:50:04.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:04.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:04.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 10 monclient(hunting): dump: 2024-09-16T14:50:04.568 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:04.568 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:04.568 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:04.568 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:04.568 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:04.568 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:04.568 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:04.568 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:04.568 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:04.569 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:04.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:04.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1299772338 0 0) 0x7f2af006b320 con 0x7f2af009e6d0 2024-09-16T14:50:04.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ad0003530 con 0x7f2af009e6d0 2024-09-16T14:50:04.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2ae8004340 con 0x7f2af009e6d0 2024-09-16T14:50:04.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:04.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:04.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.552+0000 7f2aecff9640 10 monclient(hunting): dump: 2024-09-16T14:50:04.569 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:04.569 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:04.569 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:04.569 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:04.569 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:04.570 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:04.570 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:04.570 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:04.570 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:04.570 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:04.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 762795494 0 0) 0x7f2af009cf20 con 0x7f2af009c550 2024-09-16T14:50:04.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 >> v1:172.21.15.189:6789/0 conn(0x7f2af0099a70 legacy=0x7f2af0139bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:04.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 >> v1:172.21.15.26:6789/0 conn(0x7f2af009e6d0 legacy=0x7f2af01cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:04.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 10 monclient: _finish_hunting 0 2024-09-16T14:50:04.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 1 monclient: found mon.c 2024-09-16T14:50:04.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:04.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:04.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af013a9f0 con 0x7f2af009c550 2024-09-16T14:50:04.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 10 monclient: _finish_auth 0 2024-09-16T14:50:04.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 10 monclient: _check_auth_tickets 2024-09-16T14:50:04.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:04.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2af589f640 5 monclient: authenticate success, global_id 154514 2024-09-16T14:50:04.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2af589f640 10 monclient: _renew_subs 2024-09-16T14:50:04.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2af589f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:04.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2af589f640 1 -- 172.21.15.189:0/2023315618 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af013b690 con 0x7f2af009c550 2024-09-16T14:50:04.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2af589f640 10 monclient: _renew_subs 2024-09-16T14:50:04.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2af589f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:04.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2af589f640 1 -- 172.21.15.189:0/2023315618 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af011a2d0 con 0x7f2af009c550 2024-09-16T14:50:04.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ad8003620 con 0x7f2af009c550 2024-09-16T14:50:04.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:04.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aed7fa640 4 set_mon_vals no callback set 2024-09-16T14:50:04.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2ad80039b0 con 0x7f2af009c550 2024-09-16T14:50:04.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:04.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:04.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 10 monclient: dump: 2024-09-16T14:50:04.572 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:04.572 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:04.572 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:04.572 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:04.572 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:04.572 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:04.572 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:04.572 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:04.572 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:04.572 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:04.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2af589f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:04.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2af589f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:04.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2af589f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:04.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2af589f640 1 -- 172.21.15.189:0/2023315618 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad4005050 con 0x7f2af009c550 2024-09-16T14:50:04.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2ad8003bd0 con 0x7f2af009c550 2024-09-16T14:50:04.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(213..213 src has 169..213) v4 ==== 13646+0+0 (unknown 4175819988 0 0) 0x7f2ad8045850 con 0x7f2af009c550 2024-09-16T14:50:04.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.556+0000 7f2aeeffd640 10 monclient: get_auth_request con 0x7f2ad0036220 auth_method 0 2024-09-16T14:50:04.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.568+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2ad8014690 con 0x7f2af009c550 2024-09-16T14:50:04.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.568+0000 7f2aecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:04.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.568+0000 7f2aecff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:04.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.720+0000 7f2af589f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:04.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.720+0000 7f2af589f640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:04.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.720+0000 7f2af589f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:04.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:04.720+0000 7f2af589f640 1 -- 172.21.15.189:0/2023315618 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f2ad4005300 con 0x7f2af009c550 2024-09-16T14:50:05.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.308+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}]=0 added bucket h31 type host to location {rack=r1} v214) v1 ==== 168+0+0 (unknown 1581209104 0 0) 0x7f2ad8012b00 con 0x7f2af009c550 2024-09-16T14:50:05.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.312+0000 7f2aecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:05.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.312+0000 7f2aecff9640 10 monclient: _finish_command 2 = system:0 added bucket h31 type host to location {rack=r1} 2024-09-16T14:50:05.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.408+0000 7f2af589f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:05.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.408+0000 7f2af589f640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:05.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.408+0000 7f2af589f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:05.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.408+0000 7f2af589f640 1 -- 172.21.15.189:0/2023315618 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f2ad4005a40 con 0x7f2af009c550 2024-09-16T14:50:05.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.412+0000 7f2aecff9640 1 -- 172.21.15.189:0/2023315618 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h31' already exists v214) v1 ==== 147+0+0 (unknown 1292714927 0 0) 0x7f2ad8014460 con 0x7f2af009c550 2024-09-16T14:50:05.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.412+0000 7f2aecff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:05.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.412+0000 7f2aecff9640 10 monclient: _finish_command 3 = system:0 bucket 'h31' already exists 2024-09-16T14:50:05.414 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h31' already exists 2024-09-16T14:50:05.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.412+0000 7f2af589f640 1 -- 172.21.15.189:0/2023315618 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2ad0036220 legacy=0x7f2ad00386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:05.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.412+0000 7f2af589f640 10 monclient: shutdown 2024-09-16T14:50:05.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.412+0000 7f2af589f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:05.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.412+0000 7f2af589f640 1 -- 172.21.15.189:0/2023315618 >> v1:172.21.15.26:6790/0 conn(0x7f2af009c550 legacy=0x7f2af013a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:05.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.412+0000 7f2af589f640 1 -- 172.21.15.189:0/2023315618 shutdown_connections 2024-09-16T14:50:05.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.412+0000 7f2af589f640 1 -- 172.21.15.189:0/2023315618 >> 172.21.15.189:0/2023315618 conn(0x7f2af00a3380 msgr2=0x7f2af00a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:05.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.412+0000 7f2af589f640 1 -- 172.21.15.189:0/2023315618 shutdown_connections 2024-09-16T14:50:05.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.412+0000 7f2af589f640 1 -- 172.21.15.189:0/2023315618 wait complete. 2024-09-16T14:50:05.431 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:05.431 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h32 host rack=r1 2024-09-16T14:50:05.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 10 monclient: get_monmap_and_config 2024-09-16T14:50:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 10 monclient: build_initial_monmap 2024-09-16T14:50:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 10 monclient: monmap: 2024-09-16T14:50:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:05.553822+0000 2024-09-16T14:50:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:05.553822+0000 2024-09-16T14:50:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:05.553 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:05.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 1 Processor -- start 2024-09-16T14:50:05.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 1 -- start start 2024-09-16T14:50:05.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 10 monclient: init 2024-09-16T14:50:05.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:05.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:05.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 10 monclient(hunting): picked mon.noname-b con 0x7fbeac070830 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:05.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 10 monclient(hunting): picked mon.noname-c con 0x7fbeac071320 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:05.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 10 monclient(hunting): picked mon.noname-a con 0x7fbeac134cf0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:05.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeac135740 con 0x7fbeac134cf0 2024-09-16T14:50:05.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeac135160 con 0x7fbeac071320 2024-09-16T14:50:05.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeac09d810 con 0x7fbeac070830 2024-09-16T14:50:05.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:05.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 10 monclient(hunting): authenticate will time out at 1387.669429s 2024-09-16T14:50:05.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea9d9b640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fbeac071320 legacy=0x7fbeac139d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:05.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeaa59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbeac070830 0x7fbeac070c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52866/0 (socket says 172.21.15.189:52866) 2024-09-16T14:50:05.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeaa59c640 1 -- 172.21.15.189:0/1133974218 learned_addr learned my addr 172.21.15.189:0/1133974218 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:05.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeaad9d640 1 --1- 172.21.15.189:0/1133974218 >> v1:172.21.15.26:6789/0 conn(0x7fbeac134cf0 0x7fbeac13d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:54050/0 (socket says 172.21.15.189:54050) 2024-09-16T14:50:05.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 1 -- 172.21.15.189:0/1133974218 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331694548 0 0) 0x7fbeac135740 con 0x7fbeac134cf0 2024-09-16T14:50:05.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:05.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient(hunting): my global_id is 164271 2024-09-16T14:50:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 1 -- 172.21.15.189:0/1133974218 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe98003530 con 0x7fbeac134cf0 2024-09-16T14:50:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 1 -- 172.21.15.189:0/1133974218 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343587229 0 0) 0x7fbeac09d810 con 0x7fbeac070830 2024-09-16T14:50:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient(hunting): my global_id is 154517 2024-09-16T14:50:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 1 -- 172.21.15.189:0/1133974218 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeac135740 con 0x7fbeac070830 2024-09-16T14:50:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 1 -- 172.21.15.189:0/1133974218 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 426648644 0 0) 0x7fbe98003530 con 0x7fbeac134cf0 2024-09-16T14:50:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 1 -- 172.21.15.189:0/1133974218 >> v1:172.21.15.26:6790/0 conn(0x7fbeac070830 legacy=0x7fbeac070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 1 -- 172.21.15.189:0/1133974218 >> v1:172.21.15.189:6789/0 conn(0x7fbeac071320 legacy=0x7fbeac139d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient: _finish_hunting 0 2024-09-16T14:50:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 1 monclient: found mon.noname-a 2024-09-16T14:50:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 1 -- 172.21.15.189:0/1133974218 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac09f1c0 con 0x7fbeac134cf0 2024-09-16T14:50:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient: _finish_auth 0 2024-09-16T14:50:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient: _check_auth_tickets 2024-09-16T14:50:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 1 -- 172.21.15.189:0/1133974218 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe9c0019a0 con 0x7fbeac134cf0 2024-09-16T14:50:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient: dump: 2024-09-16T14:50:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:05.561 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 5 monclient: authenticate success, global_id 164271 2024-09-16T14:50:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 1 -- 172.21.15.189:0/1133974218 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe9c001bc0 con 0x7fbeac134cf0 2024-09-16T14:50:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 1 -- 172.21.15.189:0/1133974218 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe9c001f50 con 0x7fbeac134cf0 2024-09-16T14:50:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbea959a640 10 monclient: dump: 2024-09-16T14:50:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:05.563 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 4 set_mon_vals no callback set 2024-09-16T14:50:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 10 monclient: shutdown 2024-09-16T14:50:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 1 -- 172.21.15.189:0/1133974218 >> v1:172.21.15.26:6789/0 conn(0x7fbeac134cf0 legacy=0x7fbeac13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 1 -- 172.21.15.189:0/1133974218 shutdown_connections 2024-09-16T14:50:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 1 -- 172.21.15.189:0/1133974218 >> 172.21.15.189:0/1133974218 conn(0x7fbeac0a3380 msgr2=0x7fbeac09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 1 -- 172.21.15.189:0/1133974218 shutdown_connections 2024-09-16T14:50:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.552+0000 7fbeb0fb8640 1 -- 172.21.15.189:0/1133974218 wait complete. 2024-09-16T14:50:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 monclient: build_initial_monmap 2024-09-16T14:50:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 monclient: monmap: 2024-09-16T14:50:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:05.564 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:05.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 1 Processor -- start 2024-09-16T14:50:05.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 1 -- start start 2024-09-16T14:50:05.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 monclient: init 2024-09-16T14:50:05.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:05.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:05.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 monclient(hunting): picked mon.noname-c con 0x7fbeac070830 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:05.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 monclient(hunting): picked mon.noname-a con 0x7fbeac071320 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:05.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 monclient(hunting): picked mon.noname-b con 0x7fbeac134cf0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:05.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeac132c40 con 0x7fbeac071320 2024-09-16T14:50:05.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeac06b320 con 0x7fbeac134cf0 2024-09-16T14:50:05.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeac11e280 con 0x7fbeac070830 2024-09-16T14:50:05.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:05.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 monclient(hunting): authenticate will time out at 1387.673263s 2024-09-16T14:50:05.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeaad9d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fbeac134cf0 legacy=0x7fbeac1d1440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:05.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeaa59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbeac070830 0x7fbeac1c90c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52872/0 (socket says 172.21.15.189:52872) 2024-09-16T14:50:05.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeaa59c640 1 -- 172.21.15.189:0/2283878061 learned_addr learned my addr 172.21.15.189:0/2283878061 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:05.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050574390 0 0) 0x7fbeac11e280 con 0x7fbeac070830 2024-09-16T14:50:05.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:05.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:05.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:05.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient(hunting): my global_id is 154520 2024-09-16T14:50:05.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe8c003530 con 0x7fbeac070830 2024-09-16T14:50:05.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935852183 0 0) 0x7fbeac132c40 con 0x7fbeac071320 2024-09-16T14:50:05.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:05.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:05.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:05.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient(hunting): my global_id is 164274 2024-09-16T14:50:05.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeac11e280 con 0x7fbeac071320 2024-09-16T14:50:05.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2575827916 0 0) 0x7fbe8c003530 con 0x7fbeac070830 2024-09-16T14:50:05.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbeac132c40 con 0x7fbeac070830 2024-09-16T14:50:05.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1286158328 0 0) 0x7fbeac11e280 con 0x7fbeac071320 2024-09-16T14:50:05.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe8c003530 con 0x7fbeac071320 2024-09-16T14:50:05.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbea40016d0 con 0x7fbeac070830 2024-09-16T14:50:05.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:05.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:05.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:05.571 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:05.572 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:05.572 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:05.572 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:05.572 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:05.572 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:05.572 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:05.572 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:05.572 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:05.572 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:05.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:05.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe94004050 con 0x7fbeac071320 2024-09-16T14:50:05.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:05.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:05.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:05.573 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:05.573 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:05.573 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:05.573 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:05.573 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:05.573 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:05.573 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:05.573 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:05.573 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:05.573 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:05.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 432721011 0 0) 0x7fbe8c003530 con 0x7fbeac071320 2024-09-16T14:50:05.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 >> v1:172.21.15.26:6790/0 conn(0x7fbeac070830 legacy=0x7fbeac1c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:05.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 >> v1:172.21.15.189:6789/0 conn(0x7fbeac134cf0 legacy=0x7fbeac1d1440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:05.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:05.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 monclient: found mon.a 2024-09-16T14:50:05.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:05.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:05.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac131550 con 0x7fbeac071320 2024-09-16T14:50:05.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:05.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:05.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:05.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 5 monclient: authenticate success, global_id 164274 2024-09-16T14:50:05.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 monclient: _renew_subs 2024-09-16T14:50:05.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 1 -- 172.21.15.189:0/2283878061 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeac11dfa0 con 0x7fbeac071320 2024-09-16T14:50:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 monclient: _renew_subs 2024-09-16T14:50:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 1 -- 172.21.15.189:0/2283878061 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeac11bf20 con 0x7fbeac071320 2024-09-16T14:50:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe94004590 con 0x7fbeac071320 2024-09-16T14:50:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea3fff640 4 set_mon_vals no callback set 2024-09-16T14:50:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbe940048a0 con 0x7fbeac071320 2024-09-16T14:50:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbea2ffd640 10 monclient: dump: 2024-09-16T14:50:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:05.575 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:05.576 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:05.576 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:05.576 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:05.576 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:05.576 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:05.576 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:05.576 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:05.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:05.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:05.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:05.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.556+0000 7fbeb0fb8640 1 -- 172.21.15.189:0/2283878061 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe90005050 con 0x7fbeac071320 2024-09-16T14:50:05.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.560+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbe94004a00 con 0x7fbeac071320 2024-09-16T14:50:05.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.560+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(214..214 src has 169..214) v4 ==== 13859+0+0 (unknown 387178816 0 0) 0x7fbe94045730 con 0x7fbeac071320 2024-09-16T14:50:05.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.560+0000 7fbeaa59c640 10 monclient: get_auth_request con 0x7fbe8c0361f0 auth_method 0 2024-09-16T14:50:05.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.572+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbe94014290 con 0x7fbeac071320 2024-09-16T14:50:05.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.572+0000 7fbea2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:05.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.572+0000 7fbea2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:05.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.724+0000 7fbeb0fb8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:05.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.728+0000 7fbeb0fb8640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:05.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.728+0000 7fbeb0fb8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:05.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:05.728+0000 7fbeb0fb8640 1 -- 172.21.15.189:0/2283878061 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fbe90005300 con 0x7fbeac071320 2024-09-16T14:50:06.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.280+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}]=0 added bucket h32 type host to location {rack=r1} v215) v1 ==== 168+0+0 (unknown 1782407322 0 0) 0x7fbe940140b0 con 0x7fbeac071320 2024-09-16T14:50:06.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.280+0000 7fbea2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:06.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.280+0000 7fbea2ffd640 10 monclient: _finish_command 2 = system:0 added bucket h32 type host to location {rack=r1} 2024-09-16T14:50:06.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.376+0000 7fbeb0fb8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:06.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.376+0000 7fbeb0fb8640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:06.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.376+0000 7fbeb0fb8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:06.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.376+0000 7fbeb0fb8640 1 -- 172.21.15.189:0/2283878061 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fbe90005a40 con 0x7fbeac071320 2024-09-16T14:50:06.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.376+0000 7fbea2ffd640 1 -- 172.21.15.189:0/2283878061 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h32' already exists v215) v1 ==== 147+0+0 (unknown 2021944945 0 0) 0x7fbe9401cab0 con 0x7fbeac071320 2024-09-16T14:50:06.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.376+0000 7fbea2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T14:50:06.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.376+0000 7fbea2ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h32' already exists 2024-09-16T14:50:06.380 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h32' already exists 2024-09-16T14:50:06.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.380+0000 7fbeb0fb8640 1 -- 172.21.15.189:0/2283878061 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbe8c0361f0 legacy=0x7fbe8c0386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:06.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.380+0000 7fbeb0fb8640 10 monclient: shutdown 2024-09-16T14:50:06.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.380+0000 7fbeb0fb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:06.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.380+0000 7fbeb0fb8640 1 -- 172.21.15.189:0/2283878061 >> v1:172.21.15.26:6789/0 conn(0x7fbeac071320 legacy=0x7fbeac1308a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:06.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.380+0000 7fbeb0fb8640 1 -- 172.21.15.189:0/2283878061 shutdown_connections 2024-09-16T14:50:06.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.380+0000 7fbeb0fb8640 1 -- 172.21.15.189:0/2283878061 >> 172.21.15.189:0/2283878061 conn(0x7fbeac0a3380 msgr2=0x7fbeac131aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:06.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.380+0000 7fbeb0fb8640 1 -- 172.21.15.189:0/2283878061 shutdown_connections 2024-09-16T14:50:06.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.380+0000 7fbeb0fb8640 1 -- 172.21.15.189:0/2283878061 wait complete. 2024-09-16T14:50:06.397 INFO:tasks.workunit.client.0.smithi189.stderr:++ seq 1 32 2024-09-16T14:50:06.398 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:06.398 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h1 2024-09-16T14:50:06.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 10 monclient: get_monmap_and_config 2024-09-16T14:50:06.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 10 monclient: build_initial_monmap 2024-09-16T14:50:06.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 10 monclient: monmap: 2024-09-16T14:50:06.502 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:06.502 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:06.502 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:06.502760+0000 2024-09-16T14:50:06.502 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:06.502760+0000 2024-09-16T14:50:06.502 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:06.502 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:06.502 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:06.502 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:06.502 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:06.502 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:06.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 1 Processor -- start 2024-09-16T14:50:06.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 1 -- start start 2024-09-16T14:50:06.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 10 monclient: init 2024-09-16T14:50:06.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:06.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:06.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 10 monclient(hunting): picked mon.noname-c con 0x7fe73009df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:06.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 10 monclient(hunting): picked mon.noname-a con 0x7fe73009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:06.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 10 monclient(hunting): picked mon.noname-b con 0x7fe73009e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:06.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73009ee20 con 0x7fe73009e2f0 2024-09-16T14:50:06.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73009f100 con 0x7fe73009df10 2024-09-16T14:50:06.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73009f3e0 con 0x7fe73009e740 2024-09-16T14:50:06.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:06.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 10 monclient(hunting): authenticate will time out at 1388.618390s 2024-09-16T14:50:06.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7364dc640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fe73009df10 legacy=0x7fe73009d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:06.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe736cdd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe73009e740 0x7fe73013b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52882/0 (socket says 172.21.15.189:52882) 2024-09-16T14:50:06.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe735cdb640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe73009e2f0 0x7fe7301378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:54076/0 (socket says 172.21.15.189:54076) 2024-09-16T14:50:06.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe736cdd640 1 -- 172.21.15.189:0/2390917568 learned_addr learned my addr 172.21.15.189:0/2390917568 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:06.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 1 -- 172.21.15.189:0/2390917568 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147471604 0 0) 0x7fe73009ee20 con 0x7fe73009e2f0 2024-09-16T14:50:06.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 10 monclient(hunting): _negotiate 2024-09-16T14:50:06.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:06.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:06.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 10 monclient(hunting): my global_id is 164277 2024-09-16T14:50:06.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 1 -- 172.21.15.189:0/2390917568 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe71c003530 con 0x7fe73009e2f0 2024-09-16T14:50:06.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 1 -- 172.21.15.189:0/2390917568 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134431192 0 0) 0x7fe73009f3e0 con 0x7fe73009e740 2024-09-16T14:50:06.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 10 monclient(hunting): _negotiate 2024-09-16T14:50:06.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:06.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:06.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 10 monclient(hunting): my global_id is 154523 2024-09-16T14:50:06.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 1 -- 172.21.15.189:0/2390917568 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe73009ee20 con 0x7fe73009e740 2024-09-16T14:50:06.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 1 -- 172.21.15.189:0/2390917568 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1977172481 0 0) 0x7fe71c003530 con 0x7fe73009e2f0 2024-09-16T14:50:06.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 1 -- 172.21.15.189:0/2390917568 >> v1:172.21.15.26:6790/0 conn(0x7fe73009e740 legacy=0x7fe73013b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:06.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 1 -- 172.21.15.189:0/2390917568 >> v1:172.21.15.189:6789/0 conn(0x7fe73009df10 legacy=0x7fe73009d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:06.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 10 monclient: _finish_hunting 0 2024-09-16T14:50:06.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 1 monclient: found mon.noname-a 2024-09-16T14:50:06.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:06.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:06.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 1 -- 172.21.15.189:0/2390917568 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73009c690 con 0x7fe73009e2f0 2024-09-16T14:50:06.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 10 monclient: _finish_auth 0 2024-09-16T14:50:06.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 10 monclient: _check_auth_tickets 2024-09-16T14:50:06.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:06.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 1 -- 172.21.15.189:0/2390917568 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe72c0018d0 con 0x7fe73009e2f0 2024-09-16T14:50:06.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:06.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:06.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe7354da640 10 monclient: dump: 2024-09-16T14:50:06.517 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:06.517 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:06.517 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:06.517 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:06.517 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:06.517 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:06.518 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 5 monclient: authenticate success, global_id 164277 2024-09-16T14:50:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.500+0000 7fe738740640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe7354da640 1 -- 172.21.15.189:0/2390917568 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe72c001b30 con 0x7fe73009e2f0 2024-09-16T14:50:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe7354da640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:06.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 4 set_mon_vals no callback set 2024-09-16T14:50:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe7354da640 1 -- 172.21.15.189:0/2390917568 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe72c001f00 con 0x7fe73009e2f0 2024-09-16T14:50:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe7354da640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe7354da640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe7354da640 10 monclient: dump: 2024-09-16T14:50:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:06.519 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:06.520 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:06.520 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:06.520 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:06.520 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:06.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 10 monclient: shutdown 2024-09-16T14:50:06.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:06.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 1 -- 172.21.15.189:0/2390917568 >> v1:172.21.15.26:6789/0 conn(0x7fe73009e2f0 legacy=0x7fe7301378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:06.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 1 -- 172.21.15.189:0/2390917568 shutdown_connections 2024-09-16T14:50:06.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 1 -- 172.21.15.189:0/2390917568 >> 172.21.15.189:0/2390917568 conn(0x7fe7300a3320 msgr2=0x7fe73009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:06.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 1 -- 172.21.15.189:0/2390917568 shutdown_connections 2024-09-16T14:50:06.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 1 -- 172.21.15.189:0/2390917568 wait complete. 2024-09-16T14:50:06.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 10 monclient: build_initial_monmap 2024-09-16T14:50:06.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 10 monclient: monmap: 2024-09-16T14:50:06.521 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:06.521 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:06.521 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:06.521 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:06.521 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:06.521 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:06.521 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:06.521 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:06.521 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:06.522 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:06.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 1 Processor -- start 2024-09-16T14:50:06.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 1 -- start start 2024-09-16T14:50:06.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 10 monclient: init 2024-09-16T14:50:06.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:06.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:06.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 10 monclient(hunting): picked mon.noname-a con 0x7fe73009df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:06.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 10 monclient(hunting): picked mon.noname-c con 0x7fe73009e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 10 monclient(hunting): picked mon.noname-b con 0x7fe73009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73006cc00 con 0x7fe73009df10 2024-09-16T14:50:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73006b380 con 0x7fe73009e740 2024-09-16T14:50:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe730122660 con 0x7fe73009e2f0 2024-09-16T14:50:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 10 monclient(hunting): authenticate will time out at 1388.621888s 2024-09-16T14:50:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe736cdd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fe73009e740 legacy=0x7fe7301d20c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe735cdb640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe73009e2f0 0x7fe7301ce990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52886/0 (socket says 172.21.15.189:52886) 2024-09-16T14:50:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe735cdb640 1 -- 172.21.15.189:0/1834405420 learned_addr learned my addr 172.21.15.189:0/1834405420 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:06.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe7364dc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe73009df10 0x7fe7301c99b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:54078/0 (socket says 172.21.15.189:54078) 2024-09-16T14:50:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618199019 0 0) 0x7fe730122660 con 0x7fe73009e2f0 2024-09-16T14:50:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient(hunting): my global_id is 154526 2024-09-16T14:50:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe714003530 con 0x7fe73009e2f0 2024-09-16T14:50:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230767002 0 0) 0x7fe73006cc00 con 0x7fe73009df10 2024-09-16T14:50:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient(hunting): my global_id is 164280 2024-09-16T14:50:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe730122660 con 0x7fe73009df10 2024-09-16T14:50:06.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 767530876 0 0) 0x7fe730122660 con 0x7fe73009df10 2024-09-16T14:50:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe73006cc00 con 0x7fe73009df10 2024-09-16T14:50:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe720003f80 con 0x7fe73009df10 2024-09-16T14:50:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:06.525 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:06.526 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3769417346 0 0) 0x7fe714003530 con 0x7fe73009e2f0 2024-09-16T14:50:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe730122660 con 0x7fe73009e2f0 2024-09-16T14:50:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe72c001e80 con 0x7fe73009e2f0 2024-09-16T14:50:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:06.527 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:06.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3661356095 0 0) 0x7fe73006cc00 con 0x7fe73009df10 2024-09-16T14:50:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 >> v1:172.21.15.26:6790/0 conn(0x7fe73009e2f0 legacy=0x7fe7301ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 >> v1:172.21.15.189:6789/0 conn(0x7fe73009e740 legacy=0x7fe7301d20c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 monclient: found mon.a 2024-09-16T14:50:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe730121210 con 0x7fe73009df10 2024-09-16T14:50:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:06.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 5 monclient: authenticate success, global_id 164280 2024-09-16T14:50:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 10 monclient: _renew_subs 2024-09-16T14:50:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 1 -- 172.21.15.189:0/1834405420 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe730121dc0 con 0x7fe73009df10 2024-09-16T14:50:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 10 monclient: _renew_subs 2024-09-16T14:50:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe738740640 1 -- 172.21.15.189:0/1834405420 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe73013c430 con 0x7fe73009df10 2024-09-16T14:50:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7200044e0 con 0x7fe73009df10 2024-09-16T14:50:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe7200047f0 con 0x7fe73009df10 2024-09-16T14:50:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:06.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe734cd9640 4 set_mon_vals no callback set 2024-09-16T14:50:06.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.504+0000 7fe726ffd640 10 monclient: dump: 2024-09-16T14:50:06.530 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:06.530 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:06.530 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:06.530 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:06.530 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:06.530 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:06.530 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:06.530 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:06.530 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:06.530 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:06.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.508+0000 7fe738740640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:06.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.508+0000 7fe738740640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:06.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.508+0000 7fe738740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:06.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.508+0000 7fe738740640 1 -- 172.21.15.189:0/1834405420 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe718005050 con 0x7fe73009df10 2024-09-16T14:50:06.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.508+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe720004950 con 0x7fe73009df10 2024-09-16T14:50:06.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.508+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(215..215 src has 169..215) v4 ==== 14072+0+0 (unknown 3477893930 0 0) 0x7fe720045770 con 0x7fe73009df10 2024-09-16T14:50:06.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.508+0000 7fe735cdb640 10 monclient: get_auth_request con 0x7fe714036310 auth_method 0 2024-09-16T14:50:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.520+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe720014180 con 0x7fe73009df10 2024-09-16T14:50:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.520+0000 7fe726ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.520+0000 7fe726ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:06.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.668+0000 7fe738740640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T14:50:06.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.668+0000 7fe738740640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T14:50:06.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.668+0000 7fe738740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:06.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:06.668+0000 7fe738740640 1 -- 172.21.15.189:0/1834405420 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h1"} v 0) v1 -- 0x7fe718005300 con 0x7fe73009df10 2024-09-16T14:50:07.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.324+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h1"}]=0 removed item id -19 name 'h1' from crush map v216) v1 ==== 118+0+0 (unknown 2049919437 0 0) 0x7fe720013fa0 con 0x7fe73009df10 2024-09-16T14:50:07.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.324+0000 7fe726ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T14:50:07.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.324+0000 7fe726ffd640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'h1' from crush map 2024-09-16T14:50:07.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.424+0000 7fe738740640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T14:50:07.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.424+0000 7fe738740640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T14:50:07.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.424+0000 7fe738740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:07.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.424+0000 7fe738740640 1 -- 172.21.15.189:0/1834405420 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h1"} v 0) v1 -- 0x7fe718005a40 con 0x7fe73009df10 2024-09-16T14:50:07.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.424+0000 7fe726ffd640 1 -- 172.21.15.189:0/1834405420 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h1"}]=0 device 'h1' does not appear in the crush map v216) v1 ==== 118+0+0 (unknown 1184635400 0 0) 0x7fe720013fa0 con 0x7fe73009df10 2024-09-16T14:50:07.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.424+0000 7fe726ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T14:50:07.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.424+0000 7fe726ffd640 10 monclient: _finish_command 3 = system:0 device 'h1' does not appear in the crush map 2024-09-16T14:50:07.427 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h1' does not appear in the crush map 2024-09-16T14:50:07.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.428+0000 7fe738740640 1 -- 172.21.15.189:0/1834405420 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe714036310 legacy=0x7fe7140387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:07.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.428+0000 7fe738740640 10 monclient: shutdown 2024-09-16T14:50:07.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.428+0000 7fe738740640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:07.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.428+0000 7fe738740640 1 -- 172.21.15.189:0/1834405420 >> v1:172.21.15.26:6789/0 conn(0x7fe73009df10 legacy=0x7fe7301c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:07.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.428+0000 7fe738740640 1 -- 172.21.15.189:0/1834405420 shutdown_connections 2024-09-16T14:50:07.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.428+0000 7fe738740640 1 -- 172.21.15.189:0/1834405420 >> 172.21.15.189:0/1834405420 conn(0x7fe7300a3320 msgr2=0x7fe73009d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:07.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.428+0000 7fe738740640 1 -- 172.21.15.189:0/1834405420 shutdown_connections 2024-09-16T14:50:07.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.428+0000 7fe738740640 1 -- 172.21.15.189:0/1834405420 wait complete. 2024-09-16T14:50:07.443 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:07.443 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h2 2024-09-16T14:50:07.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e0ba5b640 10 monclient: get_monmap_and_config 2024-09-16T14:50:07.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e0ba5b640 10 monclient: build_initial_monmap 2024-09-16T14:50:07.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e0ba5b640 10 monclient: monmap: 2024-09-16T14:50:07.561 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:07.561 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:07.562 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:07.562718+0000 2024-09-16T14:50:07.562 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:07.562718+0000 2024-09-16T14:50:07.562 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:07.562 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:07.562 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:07.562 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:07.562 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:07.562 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:07.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e0ba5b640 1 Processor -- start 2024-09-16T14:50:07.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e0ba5b640 1 -- start start 2024-09-16T14:50:07.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e0ba5b640 10 monclient: init 2024-09-16T14:50:07.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e0ba5b640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:07.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e0ba5b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:07.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e0ba5b640 10 monclient(hunting): picked mon.noname-a con 0x7f6e040a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:07.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e0ba5b640 10 monclient(hunting): picked mon.noname-b con 0x7f6e040a25f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:07.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e0ba5b640 10 monclient(hunting): picked mon.noname-c con 0x7f6e0413b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:07.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e0ba5b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e040a30b0 con 0x7f6e040a1b00 2024-09-16T14:50:07.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e0ba5b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e040a2e30 con 0x7f6e0413b430 2024-09-16T14:50:07.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e0ba5b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0409eb50 con 0x7f6e040a25f0 2024-09-16T14:50:07.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e0ba5b640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:07.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e0ba5b640 10 monclient(hunting): authenticate will time out at 1389.678778s 2024-09-16T14:50:07.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e09ff8640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f6e0413b430 legacy=0x7f6e0413d820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:07.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e08ff6640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6e040a25f0 0x7f6e04139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52894/0 (socket says 172.21.15.189:52894) 2024-09-16T14:50:07.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6e08ff6640 1 -- 172.21.15.189:0/1466981898 learned_addr learned my addr 172.21.15.189:0/1466981898 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6dfbfff640 1 -- 172.21.15.189:0/1466981898 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841501100 0 0) 0x7f6e0409eb50 con 0x7f6e040a25f0 2024-09-16T14:50:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6dfbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6dfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6dfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6dfbfff640 10 monclient(hunting): my global_id is 154529 2024-09-16T14:50:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6dfbfff640 1 -- 172.21.15.189:0/1466981898 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df0003530 con 0x7f6e040a25f0 2024-09-16T14:50:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6dfbfff640 1 -- 172.21.15.189:0/1466981898 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972866336 0 0) 0x7f6e040a30b0 con 0x7f6e040a1b00 2024-09-16T14:50:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6dfbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6dfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6dfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6dfbfff640 10 monclient(hunting): my global_id is 164283 2024-09-16T14:50:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.560+0000 7f6dfbfff640 1 -- 172.21.15.189:0/1466981898 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e0409eb50 con 0x7f6e040a1b00 2024-09-16T14:50:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 1 -- 172.21.15.189:0/1466981898 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2371715696 0 0) 0x7f6df0003530 con 0x7f6e040a25f0 2024-09-16T14:50:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 1 -- 172.21.15.189:0/1466981898 >> v1:172.21.15.189:6789/0 conn(0x7f6e0413b430 legacy=0x7f6e0413d820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 1 -- 172.21.15.189:0/1466981898 >> v1:172.21.15.26:6789/0 conn(0x7f6e040a1b00 legacy=0x7f6e040a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 10 monclient: _finish_hunting 0 2024-09-16T14:50:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 1 monclient: found mon.noname-b 2024-09-16T14:50:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 1 -- 172.21.15.189:0/1466981898 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0409ee30 con 0x7f6e040a25f0 2024-09-16T14:50:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 10 monclient: _finish_auth 0 2024-09-16T14:50:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 10 monclient: _check_auth_tickets 2024-09-16T14:50:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 1 -- 172.21.15.189:0/1466981898 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6df4001620 con 0x7f6e040a25f0 2024-09-16T14:50:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 5 monclient: authenticate success, global_id 154529 2024-09-16T14:50:07.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:07.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:07.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:07.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 10 monclient: dump: 2024-09-16T14:50:07.569 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:07.569 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:07.569 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:07.569 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:07.569 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:07.569 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:07.569 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:07.569 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:07.569 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:07.569 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:07.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:07.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 1 -- 172.21.15.189:0/1466981898 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6df4001900 con 0x7f6e040a25f0 2024-09-16T14:50:07.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:07.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:07.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 1 -- 172.21.15.189:0/1466981898 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6df4001c90 con 0x7f6e040a25f0 2024-09-16T14:50:07.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 4 set_mon_vals no callback set 2024-09-16T14:50:07.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:07.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:07.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6dfbfff640 10 monclient: dump: 2024-09-16T14:50:07.570 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:07.570 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:07.570 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:07.570 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:07.570 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:07.571 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:07.571 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:07.571 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:07.572 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:07.572 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:07.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 10 monclient: shutdown 2024-09-16T14:50:07.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:07.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 1 -- 172.21.15.189:0/1466981898 >> v1:172.21.15.26:6790/0 conn(0x7f6e040a25f0 legacy=0x7f6e04139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:07.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 1 -- 172.21.15.189:0/1466981898 shutdown_connections 2024-09-16T14:50:07.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 1 -- 172.21.15.189:0/1466981898 >> 172.21.15.189:0/1466981898 conn(0x7f6e040a7dd0 msgr2=0x7f6e040a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:07.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 1 -- 172.21.15.189:0/1466981898 shutdown_connections 2024-09-16T14:50:07.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 1 -- 172.21.15.189:0/1466981898 wait complete. 2024-09-16T14:50:07.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 10 monclient: build_initial_monmap 2024-09-16T14:50:07.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 10 monclient: monmap: 2024-09-16T14:50:07.577 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:07.577 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:07.577 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:07.577 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:07.578 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:07.578 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:07.578 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:07.578 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:07.578 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:07.578 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:07.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 1 Processor -- start 2024-09-16T14:50:07.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 1 -- start start 2024-09-16T14:50:07.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 10 monclient: init 2024-09-16T14:50:07.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:07.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:07.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 10 monclient(hunting): picked mon.noname-c con 0x7f6e040a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:07.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 10 monclient(hunting): picked mon.noname-b con 0x7f6e040a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:07.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 10 monclient(hunting): picked mon.noname-a con 0x7f6e0413b430 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:07.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0409e2f0 con 0x7f6e0413b430 2024-09-16T14:50:07.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0411e570 con 0x7f6e040a25f0 2024-09-16T14:50:07.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0413aff0 con 0x7f6e040a1b00 2024-09-16T14:50:07.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:07.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e0ba5b640 10 monclient(hunting): authenticate will time out at 1389.683033s 2024-09-16T14:50:07.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e08ff6640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f6e040a25f0 legacy=0x7f6e0411de60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:07.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e097f7640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6e040a1b00 0x7f6e0411d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52902/0 (socket says 172.21.15.189:52902) 2024-09-16T14:50:07.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.564+0000 7f6e097f7640 1 -- 172.21.15.189:0/866153424 learned_addr learned my addr 172.21.15.189:0/866153424 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:07.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138918521 0 0) 0x7f6e0413aff0 con 0x7f6e040a1b00 2024-09-16T14:50:07.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:07.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:07.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:07.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient(hunting): my global_id is 154532 2024-09-16T14:50:07.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de8003530 con 0x7f6e040a1b00 2024-09-16T14:50:07.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398669680 0 0) 0x7f6e0409e2f0 con 0x7f6e0413b430 2024-09-16T14:50:07.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:07.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:07.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:07.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient(hunting): my global_id is 164286 2024-09-16T14:50:07.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e0413aff0 con 0x7f6e0413b430 2024-09-16T14:50:07.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2188187327 0 0) 0x7f6de8003530 con 0x7f6e040a1b00 2024-09-16T14:50:07.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e0409e2f0 con 0x7f6e040a1b00 2024-09-16T14:50:07.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e00001980 con 0x7f6e040a1b00 2024-09-16T14:50:07.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:07.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:07.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient(hunting): dump: 2024-09-16T14:50:07.581 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:07.581 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:07.581 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:07.581 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:07.581 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:07.581 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:07.581 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:07.581 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:07.582 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:07.582 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:07.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:07.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3203684385 0 0) 0x7f6e0409e2f0 con 0x7f6e040a1b00 2024-09-16T14:50:07.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 >> v1:172.21.15.189:6789/0 conn(0x7f6e040a25f0 legacy=0x7f6e0411de60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:07.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 >> v1:172.21.15.26:6789/0 conn(0x7f6e0413b430 legacy=0x7f6e041d19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:07.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:50:07.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 1 monclient: found mon.c 2024-09-16T14:50:07.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:07.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:07.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e041200f0 con 0x7f6e040a1b00 2024-09-16T14:50:07.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient: _finish_auth 0 2024-09-16T14:50:07.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:07.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:07.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6e0ba5b640 5 monclient: authenticate success, global_id 154532 2024-09-16T14:50:07.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6e0ba5b640 10 monclient: _renew_subs 2024-09-16T14:50:07.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6e0ba5b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:07.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6e0ba5b640 1 -- 172.21.15.189:0/866153424 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0411ed30 con 0x7f6e040a1b00 2024-09-16T14:50:07.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6e0ba5b640 10 monclient: _renew_subs 2024-09-16T14:50:07.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6e0ba5b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:07.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6e0ba5b640 1 -- 172.21.15.189:0/866153424 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0411e960 con 0x7f6e040a1b00 2024-09-16T14:50:07.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e00003620 con 0x7f6e040a1b00 2024-09-16T14:50:07.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:07.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6e00003930 con 0x7f6e040a1b00 2024-09-16T14:50:07.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:07.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:07.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 10 monclient: dump: 2024-09-16T14:50:07.584 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:07.584 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:07.584 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:07.584 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:07.584 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:07.584 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:07.584 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:07.584 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:07.584 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:07.584 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:07.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6dfb7fe640 4 set_mon_vals no callback set 2024-09-16T14:50:07.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6e0ba5b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:07.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6e0ba5b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:07.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6e0ba5b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:07.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6e0ba5b640 1 -- 172.21.15.189:0/866153424 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de4005050 con 0x7f6e040a1b00 2024-09-16T14:50:07.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6e00003b10 con 0x7f6e040a1b00 2024-09-16T14:50:07.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(216..216 src has 169..216) v4 ==== 13862+0+0 (unknown 3148986270 0 0) 0x7f6e00045930 con 0x7f6e040a1b00 2024-09-16T14:50:07.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.568+0000 7f6e08ff6640 10 monclient: get_auth_request con 0x7f6de8036110 auth_method 0 2024-09-16T14:50:07.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.580+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6e00014410 con 0x7f6e040a1b00 2024-09-16T14:50:07.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.580+0000 7f6df9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:07.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.580+0000 7f6df9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:07.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.744+0000 7f6e0ba5b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T14:50:07.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.744+0000 7f6e0ba5b640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T14:50:07.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.744+0000 7f6e0ba5b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:07.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:07.744+0000 7f6e0ba5b640 1 -- 172.21.15.189:0/866153424 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) v1 -- 0x7f6de4005300 con 0x7f6e040a1b00 2024-09-16T14:50:08.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.340+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h2"}]=0 removed item id -22 name 'h2' from crush map v217) v1 ==== 118+0+0 (unknown 2250716610 0 0) 0x7f6e00012ab0 con 0x7f6e040a1b00 2024-09-16T14:50:08.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.340+0000 7f6df9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T14:50:08.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.340+0000 7f6df9ffb640 10 monclient: _finish_command 2 = system:0 removed item id -22 name 'h2' from crush map 2024-09-16T14:50:08.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.440+0000 7f6e0ba5b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T14:50:08.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.440+0000 7f6e0ba5b640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T14:50:08.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.440+0000 7f6e0ba5b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:08.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.440+0000 7f6e0ba5b640 1 -- 172.21.15.189:0/866153424 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) v1 -- 0x7f6de4005a40 con 0x7f6e040a1b00 2024-09-16T14:50:08.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.440+0000 7f6df9ffb640 1 -- 172.21.15.189:0/866153424 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h2"}]=0 device 'h2' does not appear in the crush map v217) v1 ==== 118+0+0 (unknown 2197983673 0 0) 0x7f6e00014230 con 0x7f6e040a1b00 2024-09-16T14:50:08.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.440+0000 7f6df9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T14:50:08.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.440+0000 7f6df9ffb640 10 monclient: _finish_command 3 = system:0 device 'h2' does not appear in the crush map 2024-09-16T14:50:08.443 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h2' does not appear in the crush map 2024-09-16T14:50:08.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.444+0000 7f6e0ba5b640 1 -- 172.21.15.189:0/866153424 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6de8036110 legacy=0x7f6de80385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:08.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.444+0000 7f6e0ba5b640 10 monclient: shutdown 2024-09-16T14:50:08.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.444+0000 7f6e0ba5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:08.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.444+0000 7f6e0ba5b640 1 -- 172.21.15.189:0/866153424 >> v1:172.21.15.26:6790/0 conn(0x7f6e040a1b00 legacy=0x7f6e0411d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:08.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.444+0000 7f6e0ba5b640 1 -- 172.21.15.189:0/866153424 shutdown_connections 2024-09-16T14:50:08.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.444+0000 7f6e0ba5b640 1 -- 172.21.15.189:0/866153424 >> 172.21.15.189:0/866153424 conn(0x7f6e040a7dd0 msgr2=0x7f6e0406b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:08.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.444+0000 7f6e0ba5b640 1 -- 172.21.15.189:0/866153424 shutdown_connections 2024-09-16T14:50:08.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.444+0000 7f6e0ba5b640 1 -- 172.21.15.189:0/866153424 wait complete. 2024-09-16T14:50:08.459 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:08.460 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h3 2024-09-16T14:50:08.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.572+0000 7fa77ca28640 10 monclient: get_monmap_and_config 2024-09-16T14:50:08.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.572+0000 7fa77ca28640 10 monclient: build_initial_monmap 2024-09-16T14:50:08.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.572+0000 7fa77ca28640 10 monclient: monmap: 2024-09-16T14:50:08.575 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:08.575 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:08.575 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:08.576542+0000 2024-09-16T14:50:08.575 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:08.576542+0000 2024-09-16T14:50:08.575 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:08.576 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:08.576 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:08.576 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:08.576 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:08.576 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.572+0000 7fa77ca28640 1 Processor -- start 2024-09-16T14:50:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.572+0000 7fa77ca28640 1 -- start start 2024-09-16T14:50:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.572+0000 7fa77ca28640 10 monclient: init 2024-09-16T14:50:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 10 monclient(hunting): picked mon.noname-c con 0x7fa7780a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 10 monclient(hunting): picked mon.noname-b con 0x7fa7780a25f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 10 monclient(hunting): picked mon.noname-a con 0x7fa77813b430 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7780a30b0 con 0x7fa77813b430 2024-09-16T14:50:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7780a2e30 con 0x7fa7780a1b00 2024-09-16T14:50:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77659c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa7780a1b00 legacy=0x7fa7780a1ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77809eb50 con 0x7fa7780a25f0 2024-09-16T14:50:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 10 monclient(hunting): authenticate will time out at 1390.692399s 2024-09-16T14:50:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa775d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa7780a25f0 0x7fa778139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:52914/0 (socket says 172.21.15.189:52914) 2024-09-16T14:50:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa775d9b640 1 -- 172.21.15.189:0/3999046407 learned_addr learned my addr 172.21.15.189:0/3999046407 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 1 -- 172.21.15.189:0/3999046407 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638573676 0 0) 0x7fa7780a30b0 con 0x7fa77813b430 2024-09-16T14:50:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient(hunting): my global_id is 164289 2024-09-16T14:50:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 1 -- 172.21.15.189:0/3999046407 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa760003530 con 0x7fa77813b430 2024-09-16T14:50:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 1 -- 172.21.15.189:0/3999046407 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022406972 0 0) 0x7fa77809eb50 con 0x7fa7780a25f0 2024-09-16T14:50:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient(hunting): my global_id is 154535 2024-09-16T14:50:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 1 -- 172.21.15.189:0/3999046407 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7780a30b0 con 0x7fa7780a25f0 2024-09-16T14:50:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 1 -- 172.21.15.189:0/3999046407 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 563167696 0 0) 0x7fa760003530 con 0x7fa77813b430 2024-09-16T14:50:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 1 -- 172.21.15.189:0/3999046407 >> v1:172.21.15.26:6790/0 conn(0x7fa7780a25f0 legacy=0x7fa778139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 1 -- 172.21.15.189:0/3999046407 >> v1:172.21.15.189:6789/0 conn(0x7fa7780a1b00 legacy=0x7fa7780a1ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient: _finish_hunting 0 2024-09-16T14:50:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 1 monclient: found mon.noname-a 2024-09-16T14:50:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 1 -- 172.21.15.189:0/3999046407 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77809ee30 con 0x7fa77813b430 2024-09-16T14:50:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient: _finish_auth 0 2024-09-16T14:50:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient: _check_auth_tickets 2024-09-16T14:50:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 1 -- 172.21.15.189:0/3999046407 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa7700019a0 con 0x7fa77813b430 2024-09-16T14:50:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient: dump: 2024-09-16T14:50:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:08.583 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 5 monclient: authenticate success, global_id 164289 2024-09-16T14:50:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 1 -- 172.21.15.189:0/3999046407 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa770001bc0 con 0x7fa77813b430 2024-09-16T14:50:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 1 -- 172.21.15.189:0/3999046407 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa770001f90 con 0x7fa77813b430 2024-09-16T14:50:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 4 set_mon_vals no callback set 2024-09-16T14:50:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77559a640 10 monclient: dump: 2024-09-16T14:50:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:08.585 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 10 monclient: shutdown 2024-09-16T14:50:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 1 -- 172.21.15.189:0/3999046407 >> v1:172.21.15.26:6789/0 conn(0x7fa77813b430 legacy=0x7fa77813d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 1 -- 172.21.15.189:0/3999046407 shutdown_connections 2024-09-16T14:50:08.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 1 -- 172.21.15.189:0/3999046407 >> 172.21.15.189:0/3999046407 conn(0x7fa7780a7dd0 msgr2=0x7fa7780a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:08.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 1 -- 172.21.15.189:0/3999046407 shutdown_connections 2024-09-16T14:50:08.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 1 -- 172.21.15.189:0/3999046407 wait complete. 2024-09-16T14:50:08.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 10 monclient: build_initial_monmap 2024-09-16T14:50:08.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 10 monclient: monmap: 2024-09-16T14:50:08.593 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:08.593 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:08.593 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:08.593 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:08.593 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:08.593 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:08.593 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:08.593 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:08.593 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:08.593 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:08.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.576+0000 7fa77ca28640 1 Processor -- start 2024-09-16T14:50:08.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 1 -- start start 2024-09-16T14:50:08.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 10 monclient: init 2024-09-16T14:50:08.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:08.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:08.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 10 monclient(hunting): picked mon.noname-b con 0x7fa7780a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:08.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 10 monclient(hunting): picked mon.noname-a con 0x7fa7780a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:08.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 10 monclient(hunting): picked mon.noname-c con 0x7fa77813b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:08.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa778136170 con 0x7fa7780a25f0 2024-09-16T14:50:08.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77809a8c0 con 0x7fa7780a1b00 2024-09-16T14:50:08.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77659c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa7780a1b00 legacy=0x7fa778099aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:08.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77809c230 con 0x7fa77813b430 2024-09-16T14:50:08.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:08.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 10 monclient(hunting): authenticate will time out at 1390.696658s 2024-09-16T14:50:08.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa775d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa7780a25f0 0x7fa77809a1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:54102/0 (socket says 172.21.15.189:54102) 2024-09-16T14:50:08.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa775d9b640 1 -- 172.21.15.189:0/1073838791 learned_addr learned my addr 172.21.15.189:0/1073838791 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:08.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027369840 0 0) 0x7fa77809c230 con 0x7fa77813b430 2024-09-16T14:50:08.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:08.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:08.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:08.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient(hunting): my global_id is 154538 2024-09-16T14:50:08.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa758003530 con 0x7fa77813b430 2024-09-16T14:50:08.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597718343 0 0) 0x7fa778136170 con 0x7fa7780a25f0 2024-09-16T14:50:08.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:08.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:08.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:08.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient(hunting): my global_id is 164292 2024-09-16T14:50:08.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa77809c230 con 0x7fa7780a25f0 2024-09-16T14:50:08.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2606586630 0 0) 0x7fa758003530 con 0x7fa77813b430 2024-09-16T14:50:08.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa778136170 con 0x7fa77813b430 2024-09-16T14:50:08.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa770001ed0 con 0x7fa77813b430 2024-09-16T14:50:08.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:08.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:08.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:08.596 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:08.596 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:08.596 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:08.596 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:08.596 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:08.596 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:08.596 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:08.596 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:08.596 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:08.597 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:08.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:08.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 268770584 0 0) 0x7fa77809c230 con 0x7fa7780a25f0 2024-09-16T14:50:08.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa758003530 con 0x7fa7780a25f0 2024-09-16T14:50:08.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa768001960 con 0x7fa7780a25f0 2024-09-16T14:50:08.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:08.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:08.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:08.597 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:08.597 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:08.597 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:08.597 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:08.598 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:08.598 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:08.598 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:08.598 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:08.598 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:08.598 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:08.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2768057455 0 0) 0x7fa778136170 con 0x7fa77813b430 2024-09-16T14:50:08.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 >> v1:172.21.15.189:6789/0 conn(0x7fa7780a1b00 legacy=0x7fa778099aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:08.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 >> v1:172.21.15.26:6789/0 conn(0x7fa7780a25f0 legacy=0x7fa77809a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:08.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:08.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 monclient: found mon.c 2024-09-16T14:50:08.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:08.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:08.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7780a58e0 con 0x7fa77813b430 2024-09-16T14:50:08.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:08.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:08.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:08.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 5 monclient: authenticate success, global_id 154538 2024-09-16T14:50:08.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 10 monclient: _renew_subs 2024-09-16T14:50:08.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:08.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 1 -- 172.21.15.189:0/1073838791 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77809bfb0 con 0x7fa77813b430 2024-09-16T14:50:08.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 10 monclient: _renew_subs 2024-09-16T14:50:08.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:08.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 1 -- 172.21.15.189:0/1073838791 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77809c450 con 0x7fa77813b430 2024-09-16T14:50:08.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa770001560 con 0x7fa77813b430 2024-09-16T14:50:08.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:08.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa770003dd0 con 0x7fa77813b430 2024-09-16T14:50:08.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:08.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:08.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa774d99640 4 set_mon_vals no callback set 2024-09-16T14:50:08.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 10 monclient: dump: 2024-09-16T14:50:08.600 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:08.600 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:08.600 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:08.600 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:08.600 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:08.600 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:08.601 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:08.601 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:08.601 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:08.601 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:08.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:08.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:08.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:08.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa77ca28640 1 -- 172.21.15.189:0/1073838791 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa75c005050 con 0x7fa77813b430 2024-09-16T14:50:08.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.580+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa77000e840 con 0x7fa77813b430 2024-09-16T14:50:08.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.584+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(217..217 src has 169..217) v4 ==== 13652+0+0 (unknown 3640273540 0 0) 0x7fa770044990 con 0x7fa77813b430 2024-09-16T14:50:08.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.584+0000 7fa77659c640 10 monclient: get_auth_request con 0x7fa758036230 auth_method 0 2024-09-16T14:50:08.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.592+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa770013600 con 0x7fa77813b430 2024-09-16T14:50:08.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.592+0000 7fa766ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:08.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.592+0000 7fa766ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:08.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.760+0000 7fa77ca28640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T14:50:08.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.760+0000 7fa77ca28640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T14:50:08.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.760+0000 7fa77ca28640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:08.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:08.760+0000 7fa77ca28640 1 -- 172.21.15.189:0/1073838791 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h3"} v 0) v1 -- 0x7fa75c005300 con 0x7fa77813b430 2024-09-16T14:50:09.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.348+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h3"}]=0 removed item id -25 name 'h3' from crush map v218) v1 ==== 118+0+0 (unknown 4133981529 0 0) 0x7fa770013420 con 0x7fa77813b430 2024-09-16T14:50:09.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.348+0000 7fa766ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T14:50:09.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.348+0000 7fa766ffd640 10 monclient: _finish_command 2 = system:0 removed item id -25 name 'h3' from crush map 2024-09-16T14:50:09.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.452+0000 7fa77ca28640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T14:50:09.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.452+0000 7fa77ca28640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T14:50:09.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.452+0000 7fa77ca28640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:09.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.452+0000 7fa77ca28640 1 -- 172.21.15.189:0/1073838791 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h3"} v 0) v1 -- 0x7fa75c005a40 con 0x7fa77813b430 2024-09-16T14:50:09.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.452+0000 7fa766ffd640 1 -- 172.21.15.189:0/1073838791 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h3"}]=0 device 'h3' does not appear in the crush map v218) v1 ==== 118+0+0 (unknown 3316571889 0 0) 0x7fa770013420 con 0x7fa77813b430 2024-09-16T14:50:09.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.452+0000 7fa766ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T14:50:09.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.452+0000 7fa766ffd640 10 monclient: _finish_command 3 = system:0 device 'h3' does not appear in the crush map 2024-09-16T14:50:09.456 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h3' does not appear in the crush map 2024-09-16T14:50:09.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.456+0000 7fa77ca28640 1 -- 172.21.15.189:0/1073838791 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa758036230 legacy=0x7fa7580386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:09.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.456+0000 7fa77ca28640 10 monclient: shutdown 2024-09-16T14:50:09.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.456+0000 7fa77ca28640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:09.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.456+0000 7fa77ca28640 1 -- 172.21.15.189:0/1073838791 >> v1:172.21.15.26:6790/0 conn(0x7fa77813b430 legacy=0x7fa7781d1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:09.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.456+0000 7fa77ca28640 1 -- 172.21.15.189:0/1073838791 shutdown_connections 2024-09-16T14:50:09.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.456+0000 7fa77ca28640 1 -- 172.21.15.189:0/1073838791 >> 172.21.15.189:0/1073838791 conn(0x7fa7780a7dd0 msgr2=0x7fa77806b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:09.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.456+0000 7fa77ca28640 1 -- 172.21.15.189:0/1073838791 shutdown_connections 2024-09-16T14:50:09.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.456+0000 7fa77ca28640 1 -- 172.21.15.189:0/1073838791 wait complete. 2024-09-16T14:50:09.473 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:09.473 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h4 2024-09-16T14:50:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.584+0000 7f632bb4b640 10 monclient: get_monmap_and_config 2024-09-16T14:50:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.584+0000 7f632bb4b640 10 monclient: build_initial_monmap 2024-09-16T14:50:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.584+0000 7f632bb4b640 10 monclient: monmap: 2024-09-16T14:50:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:09.588560+0000 2024-09-16T14:50:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:09.588560+0000 2024-09-16T14:50:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:09.588 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 1 Processor -- start 2024-09-16T14:50:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 1 -- start start 2024-09-16T14:50:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 10 monclient: init 2024-09-16T14:50:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 10 monclient(hunting): picked mon.noname-a con 0x7f63240a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 10 monclient(hunting): picked mon.noname-b con 0x7f63240a25f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 10 monclient(hunting): picked mon.noname-c con 0x7f632413b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63240a30b0 con 0x7f63240a1b00 2024-09-16T14:50:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63240a2e30 con 0x7f632413b430 2024-09-16T14:50:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632409eb50 con 0x7f63240a25f0 2024-09-16T14:50:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 10 monclient(hunting): authenticate will time out at 1391.704971s 2024-09-16T14:50:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632a0e8640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f632413b430 legacy=0x7f632413d820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63290e6640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f63240a25f0 0x7f6324139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:53648/0 (socket says 172.21.15.189:53648) 2024-09-16T14:50:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63290e6640 1 -- 172.21.15.189:0/703502326 learned_addr learned my addr 172.21.15.189:0/703502326 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:09.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 1 -- 172.21.15.189:0/703502326 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528998147 0 0) 0x7f632409eb50 con 0x7f63240a25f0 2024-09-16T14:50:09.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient(hunting): _negotiate 2024-09-16T14:50:09.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:09.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient(hunting): my global_id is 154541 2024-09-16T14:50:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 1 -- 172.21.15.189:0/703502326 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6310003530 con 0x7f63240a25f0 2024-09-16T14:50:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 1 -- 172.21.15.189:0/703502326 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948511869 0 0) 0x7f63240a30b0 con 0x7f63240a1b00 2024-09-16T14:50:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient(hunting): _negotiate 2024-09-16T14:50:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient(hunting): my global_id is 164295 2024-09-16T14:50:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 1 -- 172.21.15.189:0/703502326 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f632409eb50 con 0x7f63240a1b00 2024-09-16T14:50:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 1 -- 172.21.15.189:0/703502326 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 967087183 0 0) 0x7f632409eb50 con 0x7f63240a1b00 2024-09-16T14:50:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 1 -- 172.21.15.189:0/703502326 >> v1:172.21.15.26:6790/0 conn(0x7f63240a25f0 legacy=0x7f6324139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 1 -- 172.21.15.189:0/703502326 >> v1:172.21.15.189:6789/0 conn(0x7f632413b430 legacy=0x7f632413d820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient: _finish_hunting 0 2024-09-16T14:50:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 1 monclient: found mon.noname-a 2024-09-16T14:50:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 1 -- 172.21.15.189:0/703502326 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632409ee30 con 0x7f63240a1b00 2024-09-16T14:50:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient: _finish_auth 0 2024-09-16T14:50:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient: _check_auth_tickets 2024-09-16T14:50:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 1 -- 172.21.15.189:0/703502326 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6320001600 con 0x7f63240a1b00 2024-09-16T14:50:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient: dump: 2024-09-16T14:50:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:09.596 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 5 monclient: authenticate success, global_id 164295 2024-09-16T14:50:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 1 -- 172.21.15.189:0/703502326 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63200018a0 con 0x7f63240a1b00 2024-09-16T14:50:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 1 -- 172.21.15.189:0/703502326 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6320001c70 con 0x7f63240a1b00 2024-09-16T14:50:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 4 set_mon_vals no callback set 2024-09-16T14:50:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f63288e5640 10 monclient: dump: 2024-09-16T14:50:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:09.598 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 10 monclient: shutdown 2024-09-16T14:50:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 1 -- 172.21.15.189:0/703502326 >> v1:172.21.15.26:6789/0 conn(0x7f63240a1b00 legacy=0x7f63240a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 1 -- 172.21.15.189:0/703502326 shutdown_connections 2024-09-16T14:50:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 1 -- 172.21.15.189:0/703502326 >> 172.21.15.189:0/703502326 conn(0x7f63240a7dd0 msgr2=0x7f63240a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 1 -- 172.21.15.189:0/703502326 shutdown_connections 2024-09-16T14:50:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 1 -- 172.21.15.189:0/703502326 wait complete. 2024-09-16T14:50:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 10 monclient: build_initial_monmap 2024-09-16T14:50:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.588+0000 7f632bb4b640 10 monclient: monmap: 2024-09-16T14:50:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:09.599 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 1 Processor -- start 2024-09-16T14:50:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 1 -- start start 2024-09-16T14:50:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 10 monclient: init 2024-09-16T14:50:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 10 monclient(hunting): picked mon.noname-b con 0x7f63240a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 10 monclient(hunting): picked mon.noname-a con 0x7f63240a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 10 monclient(hunting): picked mon.noname-c con 0x7f632413b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632409d090 con 0x7f63240a25f0 2024-09-16T14:50:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6324136e60 con 0x7f63240a1b00 2024-09-16T14:50:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632409b890 con 0x7f632413b430 2024-09-16T14:50:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 10 monclient(hunting): authenticate will time out at 1391.708700s 2024-09-16T14:50:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f63298e7640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f63240a1b00 legacy=0x7f632409aa70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632a0e8640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f632413b430 0x7f63241d1600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:53664/0 (socket says 172.21.15.189:53664) 2024-09-16T14:50:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632a0e8640 1 -- 172.21.15.189:0/876995587 learned_addr learned my addr 172.21.15.189:0/876995587 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:09.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939613923 0 0) 0x7f632409b890 con 0x7f632413b430 2024-09-16T14:50:09.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:09.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:09.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:09.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient(hunting): my global_id is 154544 2024-09-16T14:50:09.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6308003530 con 0x7f632413b430 2024-09-16T14:50:09.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889655338 0 0) 0x7f632409d090 con 0x7f63240a25f0 2024-09-16T14:50:09.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:09.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:09.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:09.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient(hunting): my global_id is 164298 2024-09-16T14:50:09.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f632409b890 con 0x7f63240a25f0 2024-09-16T14:50:09.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 471362554 0 0) 0x7f632409b890 con 0x7f63240a25f0 2024-09-16T14:50:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f632409d090 con 0x7f63240a25f0 2024-09-16T14:50:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6314001290 con 0x7f63240a25f0 2024-09-16T14:50:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient(hunting): dump: 2024-09-16T14:50:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:09.606 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4078651548 0 0) 0x7f6308003530 con 0x7f632413b430 2024-09-16T14:50:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f632409b890 con 0x7f632413b430 2024-09-16T14:50:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f631c0043c0 con 0x7f632413b430 2024-09-16T14:50:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient(hunting): dump: 2024-09-16T14:50:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:09.607 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:09.607 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:09.607 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:09.607 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:09.607 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:09.607 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:09.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 472500938 0 0) 0x7f632409d090 con 0x7f63240a25f0 2024-09-16T14:50:09.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 >> v1:172.21.15.26:6790/0 conn(0x7f632413b430 legacy=0x7f63241d1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:09.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 >> v1:172.21.15.189:6789/0 conn(0x7f63240a1b00 legacy=0x7f632409aa70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:09.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:50:09.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 monclient: found mon.a 2024-09-16T14:50:09.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:09.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:09.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632409c690 con 0x7f63240a25f0 2024-09-16T14:50:09.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:50:09.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:09.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:09.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 5 monclient: authenticate success, global_id 164298 2024-09-16T14:50:09.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 10 monclient: _renew_subs 2024-09-16T14:50:09.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:09.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 1 -- 172.21.15.189:0/876995587 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63240a59c0 con 0x7f63240a25f0 2024-09-16T14:50:09.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 10 monclient: _renew_subs 2024-09-16T14:50:09.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:09.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 1 -- 172.21.15.189:0/876995587 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6324135cf0 con 0x7f63240a25f0 2024-09-16T14:50:09.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63140047f0 con 0x7f63240a25f0 2024-09-16T14:50:09.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:09.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631bfff640 4 set_mon_vals no callback set 2024-09-16T14:50:09.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6314004b00 con 0x7f63240a25f0 2024-09-16T14:50:09.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:09.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:09.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 10 monclient: dump: 2024-09-16T14:50:09.609 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:09.609 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:09.609 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:09.609 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:09.612 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:09.612 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:09.613 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:09.613 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:09.613 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:09.613 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:09.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:09.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:09.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:09.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f632bb4b640 1 -- 172.21.15.189:0/876995587 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630c005050 con 0x7f63240a25f0 2024-09-16T14:50:09.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f631400f570 con 0x7f63240a25f0 2024-09-16T14:50:09.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.592+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(218..218 src has 169..218) v4 ==== 13442+0+0 (unknown 3295156939 0 0) 0x7f63140456a0 con 0x7f63240a25f0 2024-09-16T14:50:09.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.596+0000 7f63298e7640 10 monclient: get_auth_request con 0x7f6308036220 auth_method 0 2024-09-16T14:50:09.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.608+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6314014380 con 0x7f63240a25f0 2024-09-16T14:50:09.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.608+0000 7f631a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:09.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.608+0000 7f631a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:09.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.756+0000 7f632bb4b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T14:50:09.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.756+0000 7f632bb4b640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T14:50:09.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.756+0000 7f632bb4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:09.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:09.756+0000 7f632bb4b640 1 -- 172.21.15.189:0/876995587 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h4"} v 0) v1 -- 0x7f630c005300 con 0x7f63240a25f0 2024-09-16T14:50:10.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.352+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h4"}]=0 removed item id -28 name 'h4' from crush map v219) v1 ==== 118+0+0 (unknown 2293417083 0 0) 0x7f63140141a0 con 0x7f63240a25f0 2024-09-16T14:50:10.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.352+0000 7f631a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T14:50:10.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.352+0000 7f631a7fc640 10 monclient: _finish_command 2 = system:0 removed item id -28 name 'h4' from crush map 2024-09-16T14:50:10.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.452+0000 7f632bb4b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T14:50:10.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.452+0000 7f632bb4b640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T14:50:10.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.452+0000 7f632bb4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:10.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.452+0000 7f632bb4b640 1 -- 172.21.15.189:0/876995587 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h4"} v 0) v1 -- 0x7f630c005a40 con 0x7f63240a25f0 2024-09-16T14:50:10.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.456+0000 7f631a7fc640 1 -- 172.21.15.189:0/876995587 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h4"}]=0 device 'h4' does not appear in the crush map v219) v1 ==== 118+0+0 (unknown 231993386 0 0) 0x7f6314014380 con 0x7f63240a25f0 2024-09-16T14:50:10.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.456+0000 7f631a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T14:50:10.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.456+0000 7f631a7fc640 10 monclient: _finish_command 3 = system:0 device 'h4' does not appear in the crush map 2024-09-16T14:50:10.457 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h4' does not appear in the crush map 2024-09-16T14:50:10.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.456+0000 7f632bb4b640 1 -- 172.21.15.189:0/876995587 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6308036220 legacy=0x7f63080386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:10.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.456+0000 7f632bb4b640 10 monclient: shutdown 2024-09-16T14:50:10.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.456+0000 7f632bb4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:10.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.456+0000 7f632bb4b640 1 -- 172.21.15.189:0/876995587 >> v1:172.21.15.26:6789/0 conn(0x7f63240a25f0 legacy=0x7f632409b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:10.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.456+0000 7f632bb4b640 1 -- 172.21.15.189:0/876995587 shutdown_connections 2024-09-16T14:50:10.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.456+0000 7f632bb4b640 1 -- 172.21.15.189:0/876995587 >> 172.21.15.189:0/876995587 conn(0x7f63240a7dd0 msgr2=0x7f632406b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:10.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.456+0000 7f632bb4b640 1 -- 172.21.15.189:0/876995587 shutdown_connections 2024-09-16T14:50:10.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.456+0000 7f632bb4b640 1 -- 172.21.15.189:0/876995587 wait complete. 2024-09-16T14:50:10.474 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:10.474 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h5 2024-09-16T14:50:10.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83aae15640 10 monclient: get_monmap_and_config 2024-09-16T14:50:10.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83aae15640 10 monclient: build_initial_monmap 2024-09-16T14:50:10.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83aae15640 10 monclient: monmap: 2024-09-16T14:50:10.591 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:10.591 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:10.591 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:10.592225+0000 2024-09-16T14:50:10.591 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:10.592225+0000 2024-09-16T14:50:10.591 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:10.591 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:10.591 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:10.591 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:10.592 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:10.592 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:10.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83aae15640 1 Processor -- start 2024-09-16T14:50:10.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83aae15640 1 -- start start 2024-09-16T14:50:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83aae15640 10 monclient: init 2024-09-16T14:50:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83aae15640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83aae15640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83aae15640 10 monclient(hunting): picked mon.noname-c con 0x7f83a41315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83aae15640 10 monclient(hunting): picked mon.noname-b con 0x7f83a409b320 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83aae15640 10 monclient(hunting): picked mon.noname-a con 0x7f83a4132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83aae15640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a409edd0 con 0x7f83a4132090 2024-09-16T14:50:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83aae15640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a409bc00 con 0x7f83a41315a0 2024-09-16T14:50:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83aae15640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a409e900 con 0x7f83a409b320 2024-09-16T14:50:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83aae15640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83aae15640 10 monclient(hunting): authenticate will time out at 1392.707893s 2024-09-16T14:50:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.588+0000 7f83a8bb1640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f83a41315a0 legacy=0x7f83a4131980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a3fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f83a409b320 0x7f83a4136ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:53678/0 (socket says 172.21.15.189:53678) 2024-09-16T14:50:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a3fff640 1 -- 172.21.15.189:0/760422766 learned_addr learned my addr 172.21.15.189:0/760422766 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 1 -- 172.21.15.189:0/760422766 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372773202 0 0) 0x7f83a409e900 con 0x7f83a409b320 2024-09-16T14:50:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient(hunting): my global_id is 154547 2024-09-16T14:50:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 1 -- 172.21.15.189:0/760422766 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f838c003530 con 0x7f83a409b320 2024-09-16T14:50:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 1 -- 172.21.15.189:0/760422766 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005820874 0 0) 0x7f83a409edd0 con 0x7f83a4132090 2024-09-16T14:50:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient(hunting): my global_id is 164301 2024-09-16T14:50:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 1 -- 172.21.15.189:0/760422766 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83a409e900 con 0x7f83a4132090 2024-09-16T14:50:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 1 -- 172.21.15.189:0/760422766 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3626977740 0 0) 0x7f838c003530 con 0x7f83a409b320 2024-09-16T14:50:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 1 -- 172.21.15.189:0/760422766 >> v1:172.21.15.189:6789/0 conn(0x7f83a41315a0 legacy=0x7f83a4131980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 1 -- 172.21.15.189:0/760422766 >> v1:172.21.15.26:6789/0 conn(0x7f83a4132090 legacy=0x7f83a413a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 1 monclient: found mon.noname-b 2024-09-16T14:50:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 1 -- 172.21.15.189:0/760422766 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a409b920 con 0x7f83a409b320 2024-09-16T14:50:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 1 -- 172.21.15.189:0/760422766 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8398001620 con 0x7f83a409b320 2024-09-16T14:50:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient: dump: 2024-09-16T14:50:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:10.603 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 5 monclient: authenticate success, global_id 154547 2024-09-16T14:50:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 1 -- 172.21.15.189:0/760422766 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8398001880 con 0x7f83a409b320 2024-09-16T14:50:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 1 -- 172.21.15.189:0/760422766 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8398001c50 con 0x7f83a409b320 2024-09-16T14:50:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a37fe640 10 monclient: dump: 2024-09-16T14:50:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:10.605 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:10.605 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:10.605 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:10.605 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:10.605 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:10.605 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:10.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:10.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 4 set_mon_vals no callback set 2024-09-16T14:50:10.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 10 monclient: shutdown 2024-09-16T14:50:10.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:10.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 1 -- 172.21.15.189:0/760422766 >> v1:172.21.15.26:6790/0 conn(0x7f83a409b320 legacy=0x7f83a4136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:10.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 1 -- 172.21.15.189:0/760422766 shutdown_connections 2024-09-16T14:50:10.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 1 -- 172.21.15.189:0/760422766 >> 172.21.15.189:0/760422766 conn(0x7f83a40a3320 msgr2=0x7f83a409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:10.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 1 -- 172.21.15.189:0/760422766 shutdown_connections 2024-09-16T14:50:10.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 1 -- 172.21.15.189:0/760422766 wait complete. 2024-09-16T14:50:10.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 10 monclient: build_initial_monmap 2024-09-16T14:50:10.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 10 monclient: monmap: 2024-09-16T14:50:10.606 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:10.606 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:10.606 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:10.606 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:10.606 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:10.606 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:10.607 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 1 Processor -- start 2024-09-16T14:50:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 1 -- start start 2024-09-16T14:50:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 10 monclient: init 2024-09-16T14:50:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 10 monclient(hunting): picked mon.noname-b con 0x7f83a41315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 10 monclient(hunting): picked mon.noname-c con 0x7f83a4132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 10 monclient(hunting): picked mon.noname-a con 0x7f83a41209d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a409cd20 con 0x7f83a41209d0 2024-09-16T14:50:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a41223e0 con 0x7f83a41315a0 2024-09-16T14:50:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a409a740 con 0x7f83a4132090 2024-09-16T14:50:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83a8bb1640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f83a41315a0 legacy=0x7f83a411fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.592+0000 7f83aae15640 10 monclient(hunting): authenticate will time out at 1392.711826s 2024-09-16T14:50:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a3fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f83a4132090 0x7f83a41202c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:53682/0 (socket says 172.21.15.189:53682) 2024-09-16T14:50:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a3fff640 1 -- 172.21.15.189:0/2903962762 learned_addr learned my addr 172.21.15.189:0/2903962762 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413003975 0 0) 0x7f83a409cd20 con 0x7f83a41209d0 2024-09-16T14:50:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:50:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient(hunting): my global_id is 164304 2024-09-16T14:50:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8390003530 con 0x7f83a41209d0 2024-09-16T14:50:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754436743 0 0) 0x7f83a409a740 con 0x7f83a4132090 2024-09-16T14:50:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:50:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient(hunting): my global_id is 154550 2024-09-16T14:50:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83a409cd20 con 0x7f83a4132090 2024-09-16T14:50:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1827410588 0 0) 0x7f8390003530 con 0x7f83a41209d0 2024-09-16T14:50:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83a409a740 con 0x7f83a41209d0 2024-09-16T14:50:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f839c001bc0 con 0x7f83a41209d0 2024-09-16T14:50:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient(hunting): dump: 2024-09-16T14:50:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:10.610 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3725220738 0 0) 0x7f83a409cd20 con 0x7f83a4132090 2024-09-16T14:50:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8390003530 con 0x7f83a4132090 2024-09-16T14:50:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8398001b90 con 0x7f83a4132090 2024-09-16T14:50:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient(hunting): dump: 2024-09-16T14:50:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:10.612 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 784299093 0 0) 0x7f83a409a740 con 0x7f83a41209d0 2024-09-16T14:50:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 >> v1:172.21.15.26:6790/0 conn(0x7f83a4132090 legacy=0x7f83a41202c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 >> v1:172.21.15.189:6789/0 conn(0x7f83a41315a0 legacy=0x7f83a411fbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient: _finish_hunting 0 2024-09-16T14:50:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 monclient: found mon.a 2024-09-16T14:50:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a4121ff0 con 0x7f83a41209d0 2024-09-16T14:50:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient: _finish_auth 0 2024-09-16T14:50:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient: _check_auth_tickets 2024-09-16T14:50:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83aae15640 5 monclient: authenticate success, global_id 164304 2024-09-16T14:50:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83aae15640 10 monclient: _renew_subs 2024-09-16T14:50:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83aae15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83aae15640 1 -- 172.21.15.189:0/2903962762 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a40a2930 con 0x7f83a41209d0 2024-09-16T14:50:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83aae15640 10 monclient: _renew_subs 2024-09-16T14:50:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83aae15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83aae15640 1 -- 172.21.15.189:0/2903962762 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a41239b0 con 0x7f83a41209d0 2024-09-16T14:50:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f839c0020b0 con 0x7f83a41209d0 2024-09-16T14:50:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f839c003ba0 con 0x7f83a41209d0 2024-09-16T14:50:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a2ffd640 4 set_mon_vals no callback set 2024-09-16T14:50:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 10 monclient: dump: 2024-09-16T14:50:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:10.614 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83aae15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83aae15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83aae15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83aae15640 1 -- 172.21.15.189:0/2903962762 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8384005050 con 0x7f83a41209d0 2024-09-16T14:50:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f839c00e690 con 0x7f83a41209d0 2024-09-16T14:50:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(219..219 src has 169..219) v4 ==== 13232+0+0 (unknown 1532291848 0 0) 0x7f839c0446d0 con 0x7f83a41209d0 2024-09-16T14:50:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.596+0000 7f83a8bb1640 10 monclient: get_auth_request con 0x7f8390036270 auth_method 0 2024-09-16T14:50:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.608+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f839c013480 con 0x7f83a41209d0 2024-09-16T14:50:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.608+0000 7f83a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.608+0000 7f83a17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:10.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.776+0000 7f83aae15640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T14:50:10.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.776+0000 7f83aae15640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T14:50:10.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.776+0000 7f83aae15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:10.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:10.776+0000 7f83aae15640 1 -- 172.21.15.189:0/2903962762 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h5"} v 0) v1 -- 0x7f8384005300 con 0x7f83a41209d0 2024-09-16T14:50:11.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.368+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h5"}]=0 removed item id -31 name 'h5' from crush map v220) v1 ==== 118+0+0 (unknown 1674769786 0 0) 0x7f839c0132a0 con 0x7f83a41209d0 2024-09-16T14:50:11.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.368+0000 7f83a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T14:50:11.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.368+0000 7f83a17fa640 10 monclient: _finish_command 2 = system:0 removed item id -31 name 'h5' from crush map 2024-09-16T14:50:11.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.468+0000 7f83aae15640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T14:50:11.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.468+0000 7f83aae15640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T14:50:11.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.468+0000 7f83aae15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:11.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.468+0000 7f83aae15640 1 -- 172.21.15.189:0/2903962762 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h5"} v 0) v1 -- 0x7f8384005a40 con 0x7f83a41209d0 2024-09-16T14:50:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.472+0000 7f83a17fa640 1 -- 172.21.15.189:0/2903962762 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h5"}]=0 device 'h5' does not appear in the crush map v220) v1 ==== 118+0+0 (unknown 1159062795 0 0) 0x7f839c0132a0 con 0x7f83a41209d0 2024-09-16T14:50:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.472+0000 7f83a17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T14:50:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.472+0000 7f83a17fa640 10 monclient: _finish_command 3 = system:0 device 'h5' does not appear in the crush map 2024-09-16T14:50:11.474 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h5' does not appear in the crush map 2024-09-16T14:50:11.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.472+0000 7f83aae15640 1 -- 172.21.15.189:0/2903962762 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8390036270 legacy=0x7f8390038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:11.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.472+0000 7f83aae15640 10 monclient: shutdown 2024-09-16T14:50:11.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.472+0000 7f83aae15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:11.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.472+0000 7f83aae15640 1 -- 172.21.15.189:0/2903962762 >> v1:172.21.15.26:6789/0 conn(0x7f83a41209d0 legacy=0x7f83a41d1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:11.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.472+0000 7f83aae15640 1 -- 172.21.15.189:0/2903962762 shutdown_connections 2024-09-16T14:50:11.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.472+0000 7f83aae15640 1 -- 172.21.15.189:0/2903962762 >> 172.21.15.189:0/2903962762 conn(0x7f83a40a3320 msgr2=0x7f83a409d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:11.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.472+0000 7f83aae15640 1 -- 172.21.15.189:0/2903962762 shutdown_connections 2024-09-16T14:50:11.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.472+0000 7f83aae15640 1 -- 172.21.15.189:0/2903962762 wait complete. 2024-09-16T14:50:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h6 2024-09-16T14:50:11.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f042af18640 10 monclient: get_monmap_and_config 2024-09-16T14:50:11.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f042af18640 10 monclient: build_initial_monmap 2024-09-16T14:50:11.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f042af18640 10 monclient: monmap: 2024-09-16T14:50:11.609 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:11.609 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:11.609 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:11.610539+0000 2024-09-16T14:50:11.609 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:11.610539+0000 2024-09-16T14:50:11.610 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:11.610 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:11.610 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:11.610 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:11.610 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:11.610 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:11.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f042af18640 1 Processor -- start 2024-09-16T14:50:11.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f042af18640 1 -- start start 2024-09-16T14:50:11.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f042af18640 10 monclient: init 2024-09-16T14:50:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f042af18640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f042af18640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f042af18640 10 monclient(hunting): picked mon.noname-b con 0x7f042409df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f042af18640 10 monclient(hunting): picked mon.noname-c con 0x7f042409e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f042af18640 10 monclient(hunting): picked mon.noname-a con 0x7f042409e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f042af18640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f042409ee20 con 0x7f042409e740 2024-09-16T14:50:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f042af18640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f042409f100 con 0x7f042409e2f0 2024-09-16T14:50:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f042af18640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f042409f3e0 con 0x7f042409df10 2024-09-16T14:50:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f042af18640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f042af18640 10 monclient(hunting): authenticate will time out at 1393.726730s 2024-09-16T14:50:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f041bfff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f042409e2f0 legacy=0x7f04241378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f0428cb4640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f042409df10 0x7f042409d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:53690/0 (socket says 172.21.15.189:53690) 2024-09-16T14:50:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f0428cb4640 1 -- 172.21.15.189:0/2754491919 learned_addr learned my addr 172.21.15.189:0/2754491919 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f04294b5640 1 --1- 172.21.15.189:0/2754491919 >> v1:172.21.15.26:6789/0 conn(0x7f042409e740 0x7f042413b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:34582/0 (socket says 172.21.15.189:34582) 2024-09-16T14:50:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f041b7fe640 1 -- 172.21.15.189:0/2754491919 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222748753 0 0) 0x7f042409ee20 con 0x7f042409e740 2024-09-16T14:50:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f041b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f041b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f041b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f041b7fe640 10 monclient(hunting): my global_id is 164307 2024-09-16T14:50:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f041b7fe640 1 -- 172.21.15.189:0/2754491919 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f040c003530 con 0x7f042409e740 2024-09-16T14:50:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f041b7fe640 1 -- 172.21.15.189:0/2754491919 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283933727 0 0) 0x7f042409f3e0 con 0x7f042409df10 2024-09-16T14:50:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f041b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f041b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f041b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f041b7fe640 10 monclient(hunting): my global_id is 154553 2024-09-16T14:50:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.608+0000 7f041b7fe640 1 -- 172.21.15.189:0/2754491919 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f042409ee20 con 0x7f042409df10 2024-09-16T14:50:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 1 -- 172.21.15.189:0/2754491919 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1716646051 0 0) 0x7f040c003530 con 0x7f042409e740 2024-09-16T14:50:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 1 -- 172.21.15.189:0/2754491919 >> v1:172.21.15.26:6790/0 conn(0x7f042409df10 legacy=0x7f042409d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 1 -- 172.21.15.189:0/2754491919 >> v1:172.21.15.189:6789/0 conn(0x7f042409e2f0 legacy=0x7f04241378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 1 monclient: found mon.noname-a 2024-09-16T14:50:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 1 -- 172.21.15.189:0/2754491919 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042409c690 con 0x7f042409e740 2024-09-16T14:50:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 1 -- 172.21.15.189:0/2754491919 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f041c0018d0 con 0x7f042409e740 2024-09-16T14:50:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 10 monclient: dump: 2024-09-16T14:50:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:11.618 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 5 monclient: authenticate success, global_id 164307 2024-09-16T14:50:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 1 -- 172.21.15.189:0/2754491919 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f041c001bb0 con 0x7f042409e740 2024-09-16T14:50:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 4 set_mon_vals no callback set 2024-09-16T14:50:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 1 -- 172.21.15.189:0/2754491919 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f041c001f40 con 0x7f042409e740 2024-09-16T14:50:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041b7fe640 10 monclient: dump: 2024-09-16T14:50:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:11.619 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:11.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 10 monclient: shutdown 2024-09-16T14:50:11.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:11.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 1 -- 172.21.15.189:0/2754491919 >> v1:172.21.15.26:6789/0 conn(0x7f042409e740 legacy=0x7f042413b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:11.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 1 -- 172.21.15.189:0/2754491919 shutdown_connections 2024-09-16T14:50:11.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 1 -- 172.21.15.189:0/2754491919 >> 172.21.15.189:0/2754491919 conn(0x7f04240a3320 msgr2=0x7f042409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:11.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 1 -- 172.21.15.189:0/2754491919 shutdown_connections 2024-09-16T14:50:11.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 1 -- 172.21.15.189:0/2754491919 wait complete. 2024-09-16T14:50:11.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 10 monclient: build_initial_monmap 2024-09-16T14:50:11.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 10 monclient: monmap: 2024-09-16T14:50:11.624 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:11.624 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:11.624 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:11.624 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:11.624 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:11.624 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:11.624 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:11.625 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 1 Processor -- start 2024-09-16T14:50:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 1 -- start start 2024-09-16T14:50:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 10 monclient: init 2024-09-16T14:50:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 10 monclient(hunting): picked mon.noname-a con 0x7f042409df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 10 monclient(hunting): picked mon.noname-c con 0x7f042409e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 10 monclient(hunting): picked mon.noname-b con 0x7f042409e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f042406b380 con 0x7f042409df10 2024-09-16T14:50:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f042406ce10 con 0x7f042409e740 2024-09-16T14:50:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f042411a4d0 con 0x7f042409e2f0 2024-09-16T14:50:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f042af18640 10 monclient(hunting): authenticate will time out at 1393.730816s 2024-09-16T14:50:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f04294b5640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f042409e740 legacy=0x7f04241cf7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041bfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f042409e2f0 0x7f04241cc070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:53706/0 (socket says 172.21.15.189:53706) 2024-09-16T14:50:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f041bfff640 1 -- 172.21.15.189:0/642816973 learned_addr learned my addr 172.21.15.189:0/642816973 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829133087 0 0) 0x7f042406b380 con 0x7f042409df10 2024-09-16T14:50:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f04197fa640 10 monclient(hunting): _negotiate 2024-09-16T14:50:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f04197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f04197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f04197fa640 10 monclient(hunting): my global_id is 164310 2024-09-16T14:50:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0410003530 con 0x7f042409df10 2024-09-16T14:50:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910055442 0 0) 0x7f042411a4d0 con 0x7f042409e2f0 2024-09-16T14:50:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f04197fa640 10 monclient(hunting): _negotiate 2024-09-16T14:50:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f04197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f04197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f04197fa640 10 monclient(hunting): my global_id is 154556 2024-09-16T14:50:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.612+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f042406b380 con 0x7f042409e2f0 2024-09-16T14:50:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1963042678 0 0) 0x7f0410003530 con 0x7f042409df10 2024-09-16T14:50:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f042411a4d0 con 0x7f042409df10 2024-09-16T14:50:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f04200018c0 con 0x7f042409df10 2024-09-16T14:50:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 10 monclient(hunting): dump: 2024-09-16T14:50:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:11.628 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 746347497 0 0) 0x7f042406b380 con 0x7f042409e2f0 2024-09-16T14:50:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0410003530 con 0x7f042409e2f0 2024-09-16T14:50:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0414004160 con 0x7f042409e2f0 2024-09-16T14:50:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 10 monclient(hunting): dump: 2024-09-16T14:50:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:11.629 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1247794529 0 0) 0x7f042411a4d0 con 0x7f042409df10 2024-09-16T14:50:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 >> v1:172.21.15.26:6790/0 conn(0x7f042409e2f0 legacy=0x7f04241cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 >> v1:172.21.15.189:6789/0 conn(0x7f042409e740 legacy=0x7f04241cf7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 10 monclient: _finish_hunting 0 2024-09-16T14:50:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 1 monclient: found mon.a 2024-09-16T14:50:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0424119c30 con 0x7f042409df10 2024-09-16T14:50:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 10 monclient: _finish_auth 0 2024-09-16T14:50:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 10 monclient: _check_auth_tickets 2024-09-16T14:50:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f042af18640 5 monclient: authenticate success, global_id 164310 2024-09-16T14:50:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f042af18640 10 monclient: _renew_subs 2024-09-16T14:50:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f042af18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f042af18640 1 -- 172.21.15.189:0/642816973 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0424119f10 con 0x7f042409df10 2024-09-16T14:50:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f042af18640 10 monclient: _renew_subs 2024-09-16T14:50:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f042af18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f042af18640 1 -- 172.21.15.189:0/642816973 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f042413b490 con 0x7f042409df10 2024-09-16T14:50:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04200047f0 con 0x7f042409df10 2024-09-16T14:50:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0420001d90 con 0x7f042409df10 2024-09-16T14:50:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f041affd640 4 set_mon_vals no callback set 2024-09-16T14:50:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 10 monclient: dump: 2024-09-16T14:50:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:11.632 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f042af18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f042af18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f042af18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f042af18640 1 -- 172.21.15.189:0/642816973 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0404005050 con 0x7f042409df10 2024-09-16T14:50:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0420004c00 con 0x7f042409df10 2024-09-16T14:50:11.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(220..220 src has 169..220) v4 ==== 13022+0+0 (unknown 2556910883 0 0) 0x7f04200454e0 con 0x7f042409df10 2024-09-16T14:50:11.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.616+0000 7f041bfff640 10 monclient: get_auth_request con 0x7f0410036200 auth_method 0 2024-09-16T14:50:11.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.628+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f04200141a0 con 0x7f042409df10 2024-09-16T14:50:11.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.628+0000 7f04197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:11.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.628+0000 7f04197fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:11.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.776+0000 7f042af18640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T14:50:11.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.776+0000 7f042af18640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T14:50:11.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.776+0000 7f042af18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:11.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:11.776+0000 7f042af18640 1 -- 172.21.15.189:0/642816973 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h6"} v 0) v1 -- 0x7f0404005300 con 0x7f042409df10 2024-09-16T14:50:12.231 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:50:12.231 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T14:50:12.231 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T14:50:12.231 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T14:50:12.234 INFO:tasks.ceph.mon.b:Started 2024-09-16T14:50:12.234 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:50:12.235 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:50:12.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f861762d640 10 monclient: get_monmap_and_config 2024-09-16T14:50:12.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f861762d640 10 monclient: build_initial_monmap 2024-09-16T14:50:12.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f861762d640 10 monclient: monmap: 2024-09-16T14:50:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:12.366694+0000 2024-09-16T14:50:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:12.366694+0000 2024-09-16T14:50:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:12.366 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f861762d640 1 Processor -- start 2024-09-16T14:50:12.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f861762d640 1 -- start start 2024-09-16T14:50:12.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f861762d640 10 monclient: init 2024-09-16T14:50:12.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f861762d640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:12.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f861762d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:12.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f861762d640 10 monclient(hunting): picked mon.noname-c con 0x7f86100a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:12.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f861762d640 10 monclient(hunting): picked mon.noname-b con 0x7f86100a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:12.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f861762d640 10 monclient(hunting): picked mon.noname-a con 0x7f86100a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:12.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f861762d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86100a3250 con 0x7f86100a2800 2024-09-16T14:50:12.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f861762d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861009f600 con 0x7f86100a1d10 2024-09-16T14:50:12.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f861762d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86100a3830 con 0x7f86100a0000 2024-09-16T14:50:12.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f861762d640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:12.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f861762d640 10 monclient(hunting): authenticate will time out at 1436.486367s 2024-09-16T14:50:12.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f8615bca640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f86100a2800 0x7f861013d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48722/0 (socket says 172.21.15.26:48722) 2024-09-16T14:50:12.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f8614bc8640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f86100a0000 0x7f8610139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57522/0 (socket says 172.21.15.26:57522) 2024-09-16T14:50:12.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.361+0000 7f8615bca640 1 -- 172.21.15.26:0/2390233856 learned_addr learned my addr 172.21.15.26:0/2390233856 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 1 -- 172.21.15.26:0/2390233856 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189507058 0 0) 0x7f86100a3830 con 0x7f86100a0000 2024-09-16T14:50:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 10 monclient(hunting): my global_id is 154559 2024-09-16T14:50:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 1 -- 172.21.15.26:0/2390233856 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85fc003530 con 0x7f86100a0000 2024-09-16T14:50:12.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 1 -- 172.21.15.26:0/2390233856 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2223000763 0 0) 0x7f85fc003530 con 0x7f86100a0000 2024-09-16T14:50:12.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 1 -- 172.21.15.26:0/2390233856 >> v1:172.21.15.189:6789/0 conn(0x7f86100a1d10 legacy=0x7f86100a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 1 -- 172.21.15.26:0/2390233856 >> v1:172.21.15.26:6789/0 conn(0x7f86100a2800 legacy=0x7f861013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 10 monclient: _finish_hunting 0 2024-09-16T14:50:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 1 monclient: found mon.noname-b 2024-09-16T14:50:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 1 -- 172.21.15.26:0/2390233856 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861009ed60 con 0x7f86100a0000 2024-09-16T14:50:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 10 monclient: _finish_auth 0 2024-09-16T14:50:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 10 monclient: _check_auth_tickets 2024-09-16T14:50:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 1 -- 172.21.15.26:0/2390233856 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f860c001620 con 0x7f86100a0000 2024-09-16T14:50:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 10 monclient: dump: 2024-09-16T14:50:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.377 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f861762d640 5 monclient: authenticate success, global_id 154559 2024-09-16T14:50:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f861762d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 1 -- 172.21.15.26:0/2390233856 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f860c001840 con 0x7f86100a0000 2024-09-16T14:50:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 1 -- 172.21.15.26:0/2390233856 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f860c001c10 con 0x7f86100a0000 2024-09-16T14:50:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f8607fff640 10 monclient: dump: 2024-09-16T14:50:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.379 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f861762d640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f861762d640 4 set_mon_vals no callback set 2024-09-16T14:50:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f861762d640 10 monclient: shutdown 2024-09-16T14:50:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f861762d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f861762d640 1 -- 172.21.15.26:0/2390233856 >> v1:172.21.15.26:6790/0 conn(0x7f86100a0000 legacy=0x7f8610139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f861762d640 1 -- 172.21.15.26:0/2390233856 shutdown_connections 2024-09-16T14:50:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f861762d640 1 -- 172.21.15.26:0/2390233856 >> 172.21.15.26:0/2390233856 conn(0x7f86100a7f30 msgr2=0x7f86100a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f861762d640 1 -- 172.21.15.26:0/2390233856 shutdown_connections 2024-09-16T14:50:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.369+0000 7f861762d640 1 -- 172.21.15.26:0/2390233856 wait complete. 2024-09-16T14:50:12.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 monclient: build_initial_monmap 2024-09-16T14:50:12.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 monclient: monmap: 2024-09-16T14:50:12.383 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:12.384 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:12.384 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:12.384 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:12.384 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:12.384 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:12.384 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:12.384 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:12.384 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:12.384 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 1 Processor -- start 2024-09-16T14:50:12.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 1 -- start start 2024-09-16T14:50:12.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 monclient: init 2024-09-16T14:50:12.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:12.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:12.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 monclient(hunting): picked mon.noname-b con 0x7f86100a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:12.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 monclient(hunting): picked mon.noname-c con 0x7f86100a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:12.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 monclient(hunting): picked mon.noname-a con 0x7f86100a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:12.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8610134100 con 0x7f86100a2800 2024-09-16T14:50:12.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861009e500 con 0x7f86100a0000 2024-09-16T14:50:12.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8610131660 con 0x7f86100a1d10 2024-09-16T14:50:12.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:12.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 monclient(hunting): authenticate will time out at 1436.497720s 2024-09-16T14:50:12.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8614bc8640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f86100a1d10 0x7f8610130980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57536/0 (socket says 172.21.15.26:57536) 2024-09-16T14:50:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8614bc8640 1 -- 172.21.15.26:0/3663351349 learned_addr learned my addr 172.21.15.26:0/3663351349 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 1 -- 172.21.15.26:0/3663351349 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373554033 0 0) 0x7f8610131660 con 0x7f86100a1d10 2024-09-16T14:50:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 10 monclient(hunting): my global_id is 154562 2024-09-16T14:50:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 1 -- 172.21.15.26:0/3663351349 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f4003530 con 0x7f86100a1d10 2024-09-16T14:50:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 1 -- 172.21.15.26:0/3663351349 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1875215862 0 0) 0x7f85f4003530 con 0x7f86100a1d10 2024-09-16T14:50:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 1 -- 172.21.15.26:0/3663351349 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8610131660 con 0x7f86100a1d10 2024-09-16T14:50:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 1 -- 172.21.15.26:0/3663351349 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f860c001b90 con 0x7f86100a1d10 2024-09-16T14:50:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 10 monclient(hunting): dump: 2024-09-16T14:50:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.388 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 1 -- 172.21.15.26:0/3663351349 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2938030490 0 0) 0x7f8610131660 con 0x7f86100a1d10 2024-09-16T14:50:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 1 -- 172.21.15.26:0/3663351349 >> v1:172.21.15.189:6789/0 conn(0x7f86100a0000 legacy=0x7f8610130270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 1 -- 172.21.15.26:0/3663351349 >> v1:172.21.15.26:6789/0 conn(0x7f86100a2800 legacy=0x7f86101d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:50:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 1 monclient: found mon.c 2024-09-16T14:50:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 1 -- 172.21.15.26:0/3663351349 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8610131090 con 0x7f86100a1d10 2024-09-16T14:50:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 10 monclient: _finish_auth 0 2024-09-16T14:50:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 5 monclient: authenticate success, global_id 154562 2024-09-16T14:50:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 monclient: _renew_subs 2024-09-16T14:50:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 1 -- 172.21.15.26:0/3663351349 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8610132240 con 0x7f86100a1d10 2024-09-16T14:50:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 monclient: _renew_subs 2024-09-16T14:50:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 1 -- 172.21.15.26:0/3663351349 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86100a6980 con 0x7f86100a1d10 2024-09-16T14:50:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 1 -- 172.21.15.26:0/3663351349 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f860c004af0 con 0x7f86100a1d10 2024-09-16T14:50:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 1 -- 172.21.15.26:0/3663351349 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f860c004e80 con 0x7f86100a1d10 2024-09-16T14:50:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 10 monclient: dump: 2024-09-16T14:50:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.391 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f86077fe640 4 set_mon_vals no callback set 2024-09-16T14:50:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 1 -- 172.21.15.26:0/3663351349 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f860c004fe0 con 0x7f86100a1d10 2024-09-16T14:50:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f861762d640 1 -- 172.21.15.26:0/3663351349 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f0005050 con 0x7f86100a1d10 2024-09-16T14:50:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f8605ffb640 1 -- 172.21.15.26:0/3663351349 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(220..220 src has 169..220) v4 ==== 13022+0+0 (unknown 2556910883 0 0) 0x7f860c045870 con 0x7f86100a1d10 2024-09-16T14:50:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.373+0000 7f86153c9640 10 monclient: get_auth_request con 0x7f85f4036100 auth_method 0 2024-09-16T14:50:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.381+0000 7f8605ffb640 1 -- 172.21.15.26:0/3663351349 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f860c014980 con 0x7f86100a1d10 2024-09-16T14:50:12.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.381+0000 7f8605ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:12.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.381+0000 7f8605ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:12.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.424+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h6"}]=0 removed item id -34 name 'h6' from crush map v221) v1 ==== 118+0+0 (unknown 3865982686 0 0) 0x7f04200141a0 con 0x7f042409df10 2024-09-16T14:50:12.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.424+0000 7f04197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T14:50:12.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.424+0000 7f04197fa640 10 monclient: _finish_command 2 = system:0 removed item id -34 name 'h6' from crush map 2024-09-16T14:50:12.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.520+0000 7f042af18640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T14:50:12.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.520+0000 7f042af18640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T14:50:12.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.520+0000 7f042af18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:12.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.520+0000 7f042af18640 1 -- 172.21.15.189:0/642816973 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h6"} v 0) v1 -- 0x7f0404005a40 con 0x7f042409df10 2024-09-16T14:50:12.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.524+0000 7f04197fa640 1 -- 172.21.15.189:0/642816973 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h6"}]=0 device 'h6' does not appear in the crush map v221) v1 ==== 118+0+0 (unknown 2156621498 0 0) 0x7f0420014380 con 0x7f042409df10 2024-09-16T14:50:12.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.524+0000 7f04197fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T14:50:12.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.524+0000 7f04197fa640 10 monclient: _finish_command 3 = system:0 device 'h6' does not appear in the crush map 2024-09-16T14:50:12.526 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h6' does not appear in the crush map 2024-09-16T14:50:12.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.528+0000 7f042af18640 1 -- 172.21.15.189:0/642816973 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0410036200 legacy=0x7f04100386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.528+0000 7f042af18640 10 monclient: shutdown 2024-09-16T14:50:12.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.528+0000 7f042af18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:12.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.528+0000 7f042af18640 1 -- 172.21.15.189:0/642816973 >> v1:172.21.15.26:6789/0 conn(0x7f042409df10 legacy=0x7f0424132b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.528+0000 7f042af18640 1 -- 172.21.15.189:0/642816973 shutdown_connections 2024-09-16T14:50:12.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.528+0000 7f042af18640 1 -- 172.21.15.189:0/642816973 >> 172.21.15.189:0/642816973 conn(0x7f04240a3320 msgr2=0x7f042406d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:12.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.528+0000 7f042af18640 1 -- 172.21.15.189:0/642816973 shutdown_connections 2024-09-16T14:50:12.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.528+0000 7f042af18640 1 -- 172.21.15.189:0/642816973 wait complete. 2024-09-16T14:50:12.543 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:12.543 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h7 2024-09-16T14:50:12.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.605+0000 7f861762d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:50:12.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.605+0000 7f861762d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:12.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.605+0000 7f861762d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:12.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.605+0000 7f861762d640 1 -- 172.21.15.26:0/3663351349 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f85f0005880 con 0x7f86100a1d10 2024-09-16T14:50:12.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.605+0000 7f8605ffb640 1 -- 172.21.15.26:0/3663351349 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3503950748) 0x7f860c012df0 con 0x7f86100a1d10 2024-09-16T14:50:12.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.605+0000 7f8605ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:12.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.605+0000 7f8605ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:12.612 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:50:12.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.609+0000 7f861762d640 1 -- 172.21.15.26:0/3663351349 >> v1:172.21.15.189:6812/3196161582 conn(0x7f85f4036100 legacy=0x7f85f40385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.609+0000 7f861762d640 10 monclient: shutdown 2024-09-16T14:50:12.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.609+0000 7f861762d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:12.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.609+0000 7f861762d640 1 -- 172.21.15.26:0/3663351349 >> v1:172.21.15.26:6790/0 conn(0x7f86100a1d10 legacy=0x7f8610130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.609+0000 7f861762d640 1 -- 172.21.15.26:0/3663351349 shutdown_connections 2024-09-16T14:50:12.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.609+0000 7f861762d640 1 -- 172.21.15.26:0/3663351349 >> 172.21.15.26:0/3663351349 conn(0x7f86100a7f30 msgr2=0x7f861006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:12.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.609+0000 7f861762d640 1 -- 172.21.15.26:0/3663351349 shutdown_connections 2024-09-16T14:50:12.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.609+0000 7f861762d640 1 -- 172.21.15.26:0/3663351349 wait complete. 2024-09-16T14:50:12.628 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:50:12.629 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:50:12.629 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:50:12.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 10 monclient: get_monmap_and_config 2024-09-16T14:50:12.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 10 monclient: build_initial_monmap 2024-09-16T14:50:12.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 10 monclient: monmap: 2024-09-16T14:50:12.657 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:12.657 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:12.658842+0000 2024-09-16T14:50:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:12.658842+0000 2024-09-16T14:50:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:12.658 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:12.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 1 Processor -- start 2024-09-16T14:50:12.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 1 -- start start 2024-09-16T14:50:12.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 10 monclient: init 2024-09-16T14:50:12.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:12.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:12.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 10 monclient(hunting): picked mon.noname-c con 0x7f913809df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:12.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 10 monclient(hunting): picked mon.noname-a con 0x7f913809e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 10 monclient(hunting): picked mon.noname-b con 0x7f913809e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913809ee20 con 0x7f913809e2f0 2024-09-16T14:50:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913809f100 con 0x7f913809df10 2024-09-16T14:50:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913809f3e0 con 0x7f913809e740 2024-09-16T14:50:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 10 monclient(hunting): authenticate will time out at 1394.774461s 2024-09-16T14:50:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9136d9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f913809df10 0x7f913809d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51784/0 (socket says 172.21.15.189:51784) 2024-09-16T14:50:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9136d9d640 1 -- 172.21.15.189:0/1568810008 learned_addr learned my addr 172.21.15.189:0/1568810008 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 1 -- 172.21.15.189:0/1568810008 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778832370 0 0) 0x7f913809f100 con 0x7f913809df10 2024-09-16T14:50:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient(hunting): my global_id is 174100 2024-09-16T14:50:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 1 -- 172.21.15.189:0/1568810008 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9120003530 con 0x7f913809df10 2024-09-16T14:50:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 1 -- 172.21.15.189:0/1568810008 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848464754 0 0) 0x7f913809f3e0 con 0x7f913809e740 2024-09-16T14:50:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient(hunting): my global_id is 154565 2024-09-16T14:50:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 1 -- 172.21.15.189:0/1568810008 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f913809f100 con 0x7f913809e740 2024-09-16T14:50:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 1 -- 172.21.15.189:0/1568810008 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 816640910 0 0) 0x7f9120003530 con 0x7f913809df10 2024-09-16T14:50:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 1 -- 172.21.15.189:0/1568810008 >> v1:172.21.15.26:6790/0 conn(0x7f913809e740 legacy=0x7f913813b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 1 -- 172.21.15.189:0/1568810008 >> v1:172.21.15.26:6789/0 conn(0x7f913809e2f0 legacy=0x7f91381378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:50:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 1 monclient: found mon.noname-c 2024-09-16T14:50:12.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:12.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:50:12.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 1 -- 172.21.15.189:0/1568810008 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913809c690 con 0x7f913809df10 2024-09-16T14:50:12.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient: _finish_auth 0 2024-09-16T14:50:12.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:50:12.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:12.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 1 -- 172.21.15.189:0/1568810008 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9128001620 con 0x7f913809df10 2024-09-16T14:50:12.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 5 monclient: authenticate success, global_id 174100 2024-09-16T14:50:12.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:12.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:12.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient: dump: 2024-09-16T14:50:12.664 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:12.665 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.665 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.665 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.665 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.665 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:12.665 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.665 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.665 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.665 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:12.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:12.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 1 -- 172.21.15.189:0/1568810008 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9128001840 con 0x7f913809df10 2024-09-16T14:50:12.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:12.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 1 -- 172.21.15.189:0/1568810008 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9128001bd0 con 0x7f913809df10 2024-09-16T14:50:12.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:12.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f9135d9b640 10 monclient: dump: 2024-09-16T14:50:12.666 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:12.666 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.666 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.666 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.666 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.666 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:12.666 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.666 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.666 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.666 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:12.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:12.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.656+0000 7f913d045640 4 set_mon_vals no callback set 2024-09-16T14:50:12.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient: shutdown 2024-09-16T14:50:12.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:12.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 1 -- 172.21.15.189:0/1568810008 >> v1:172.21.15.189:6789/0 conn(0x7f913809df10 legacy=0x7f913809d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 1 -- 172.21.15.189:0/1568810008 shutdown_connections 2024-09-16T14:50:12.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 1 -- 172.21.15.189:0/1568810008 >> 172.21.15.189:0/1568810008 conn(0x7f91380a3320 msgr2=0x7f913809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:12.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 1 -- 172.21.15.189:0/1568810008 shutdown_connections 2024-09-16T14:50:12.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 1 -- 172.21.15.189:0/1568810008 wait complete. 2024-09-16T14:50:12.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient: build_initial_monmap 2024-09-16T14:50:12.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient: monmap: 2024-09-16T14:50:12.667 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:12.667 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:12.667 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:12.667 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:12.668 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:12.668 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:12.668 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:12.668 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:12.668 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:12.668 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:12.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 1 Processor -- start 2024-09-16T14:50:12.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 1 -- start start 2024-09-16T14:50:12.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient: init 2024-09-16T14:50:12.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:12.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:12.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient(hunting): picked mon.noname-c con 0x7f913809df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:12.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient(hunting): picked mon.noname-b con 0x7f913809e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:12.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient(hunting): picked mon.noname-a con 0x7f913809e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:12.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913811a420 con 0x7f913809e740 2024-09-16T14:50:12.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913806f9a0 con 0x7f913809e2f0 2024-09-16T14:50:12.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913813c480 con 0x7f913809df10 2024-09-16T14:50:12.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:12.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient(hunting): authenticate will time out at 1394.777726s 2024-09-16T14:50:12.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913659c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f913809e2f0 0x7f913813a200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51796/0 (socket says 172.21.15.189:51796) 2024-09-16T14:50:12.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913659c640 1 -- 172.21.15.189:0/624524096 learned_addr learned my addr 172.21.15.189:0/624524096 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:12.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878183075 0 0) 0x7f913806f9a0 con 0x7f913809e2f0 2024-09-16T14:50:12.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient(hunting): my global_id is 174103 2024-09-16T14:50:12.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9124003530 con 0x7f913809e2f0 2024-09-16T14:50:12.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802918631 0 0) 0x7f913813c480 con 0x7f913809df10 2024-09-16T14:50:12.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient(hunting): my global_id is 154568 2024-09-16T14:50:12.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f913806f9a0 con 0x7f913809df10 2024-09-16T14:50:12.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221737541 0 0) 0x7f913811a420 con 0x7f913809e740 2024-09-16T14:50:12.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient(hunting): my global_id is 164316 2024-09-16T14:50:12.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f913813c480 con 0x7f913809e740 2024-09-16T14:50:12.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 392740653 0 0) 0x7f9124003530 con 0x7f913809e2f0 2024-09-16T14:50:12.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f913811a420 con 0x7f913809e2f0 2024-09-16T14:50:12.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f912c001970 con 0x7f913809e2f0 2024-09-16T14:50:12.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:12.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient(hunting): dump: 2024-09-16T14:50:12.674 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:12.674 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.674 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.674 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.674 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.674 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:12.674 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.675 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.675 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.675 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:12.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:12.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 506816120 0 0) 0x7f913811a420 con 0x7f913809e2f0 2024-09-16T14:50:12.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 >> v1:172.21.15.26:6790/0 conn(0x7f913809df10 legacy=0x7f9138139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 >> v1:172.21.15.26:6789/0 conn(0x7f913809e740 legacy=0x7f913813a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:12.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 monclient: found mon.b 2024-09-16T14:50:12.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:12.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:12.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913806ce70 con 0x7f913809e2f0 2024-09-16T14:50:12.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:12.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:12.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:12.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 5 monclient: authenticate success, global_id 174103 2024-09-16T14:50:12.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient: _renew_subs 2024-09-16T14:50:12.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:12.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 1 -- 172.21.15.189:0/624524096 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913811a700 con 0x7f913809e2f0 2024-09-16T14:50:12.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient: _renew_subs 2024-09-16T14:50:12.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:12.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 1 -- 172.21.15.189:0/624524096 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f913813c140 con 0x7f913809e2f0 2024-09-16T14:50:12.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f912c001c50 con 0x7f913809e2f0 2024-09-16T14:50:12.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:12.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f912c003a70 con 0x7f913809e2f0 2024-09-16T14:50:12.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913559a640 4 set_mon_vals no callback set 2024-09-16T14:50:12.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:12.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 10 monclient: dump: 2024-09-16T14:50:12.677 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:12.677 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.677 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.677 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.677 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.677 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:12.677 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.677 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.677 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.677 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:12.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f912c003c50 con 0x7f913809e2f0 2024-09-16T14:50:12.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:12.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:12.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:12.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f913d045640 1 -- 172.21.15.189:0/624524096 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9118005050 con 0x7f913809e2f0 2024-09-16T14:50:12.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(221..221 src has 169..221) v4 ==== 12812+0+0 (unknown 3418031008 0 0) 0x7f912c0455e0 con 0x7f913809e2f0 2024-09-16T14:50:12.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.660+0000 7f9136d9d640 10 monclient: get_auth_request con 0x7f9124036340 auth_method 0 2024-09-16T14:50:12.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.668+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f912c014760 con 0x7f913809e2f0 2024-09-16T14:50:12.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.668+0000 7f911f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:12.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.668+0000 7f911f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:12.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 10 monclient: get_monmap_and_config 2024-09-16T14:50:12.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 10 monclient: build_initial_monmap 2024-09-16T14:50:12.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 10 monclient: monmap: 2024-09-16T14:50:12.741 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:12.741 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.741 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:12.741702+0000 2024-09-16T14:50:12.741 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:12.741702+0000 2024-09-16T14:50:12.741 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:12.741 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.741 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:12.741 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:12.741 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:12.741 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 1 Processor -- start 2024-09-16T14:50:12.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 1 -- start start 2024-09-16T14:50:12.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 10 monclient: init 2024-09-16T14:50:12.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:12.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:12.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 10 monclient(hunting): picked mon.noname-c con 0x7f848009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:12.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 10 monclient(hunting): picked mon.noname-b con 0x7f8480132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 10 monclient(hunting): picked mon.noname-a con 0x7f8480138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848009d260 con 0x7f8480138ff0 2024-09-16T14:50:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8480132a80 con 0x7f848009cb80 2024-09-16T14:50:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848009cf60 con 0x7f8480132650 2024-09-16T14:50:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 10 monclient(hunting): authenticate will time out at 1436.861740s 2024-09-16T14:50:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8480138ff0 0x7f848013b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48746/0 (socket says 172.21.15.26:48746) 2024-09-16T14:50:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847ffff640 1 -- 172.21.15.26:0/2631748057 learned_addr learned my addr 172.21.15.26:0/2631748057 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 1 -- 172.21.15.26:0/2631748057 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981364631 0 0) 0x7f848009cf60 con 0x7f8480132650 2024-09-16T14:50:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient(hunting): my global_id is 154571 2024-09-16T14:50:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 1 -- 172.21.15.26:0/2631748057 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f846c003530 con 0x7f8480132650 2024-09-16T14:50:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 1 -- 172.21.15.26:0/2631748057 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361860095 0 0) 0x7f848009d260 con 0x7f8480138ff0 2024-09-16T14:50:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient(hunting): my global_id is 164319 2024-09-16T14:50:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 1 -- 172.21.15.26:0/2631748057 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f848009cf60 con 0x7f8480138ff0 2024-09-16T14:50:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 1 -- 172.21.15.26:0/2631748057 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 576683885 0 0) 0x7f846c003530 con 0x7f8480132650 2024-09-16T14:50:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 1 -- 172.21.15.26:0/2631748057 >> v1:172.21.15.189:6789/0 conn(0x7f848009cb80 legacy=0x7f84800717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 1 -- 172.21.15.26:0/2631748057 >> v1:172.21.15.26:6789/0 conn(0x7f8480138ff0 legacy=0x7f848013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 1 monclient: found mon.noname-b 2024-09-16T14:50:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 1 -- 172.21.15.26:0/2631748057 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848009f2d0 con 0x7f8480132650 2024-09-16T14:50:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient: _finish_auth 0 2024-09-16T14:50:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 1 -- 172.21.15.26:0/2631748057 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8470001660 con 0x7f8480132650 2024-09-16T14:50:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient: dump: 2024-09-16T14:50:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.748 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.748 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.748 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.748 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.748 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.748 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.748 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.748 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 5 monclient: authenticate success, global_id 154571 2024-09-16T14:50:12.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:12.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 1 -- 172.21.15.26:0/2631748057 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8470001880 con 0x7f8480132650 2024-09-16T14:50:12.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 1 -- 172.21.15.26:0/2631748057 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8470001c10 con 0x7f8480132650 2024-09-16T14:50:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 4 set_mon_vals no callback set 2024-09-16T14:50:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f847effd640 10 monclient: dump: 2024-09-16T14:50:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.750 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 10 monclient: shutdown 2024-09-16T14:50:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 1 -- 172.21.15.26:0/2631748057 >> v1:172.21.15.26:6790/0 conn(0x7f8480132650 legacy=0x7f84801378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 1 -- 172.21.15.26:0/2631748057 shutdown_connections 2024-09-16T14:50:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 1 -- 172.21.15.26:0/2631748057 >> 172.21.15.26:0/2631748057 conn(0x7f848009fb50 msgr2=0x7f848009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 1 -- 172.21.15.26:0/2631748057 shutdown_connections 2024-09-16T14:50:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 1 -- 172.21.15.26:0/2631748057 wait complete. 2024-09-16T14:50:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 10 monclient: build_initial_monmap 2024-09-16T14:50:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.737+0000 7f8485e61640 10 monclient: monmap: 2024-09-16T14:50:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:12.751 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:12.751 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:12.751 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:12.751 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:12.751 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:12.751 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:12.751 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:12.751 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 1 Processor -- start 2024-09-16T14:50:12.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 1 -- start start 2024-09-16T14:50:12.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 10 monclient: init 2024-09-16T14:50:12.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:12.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:12.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 10 monclient(hunting): picked mon.noname-c con 0x7f848009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:12.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 10 monclient(hunting): picked mon.noname-b con 0x7f8480132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:12.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 10 monclient(hunting): picked mon.noname-a con 0x7f8480138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:12.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848006e030 con 0x7f8480138ff0 2024-09-16T14:50:12.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8480070ea0 con 0x7f8480132650 2024-09-16T14:50:12.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848006ddc0 con 0x7f848009cb80 2024-09-16T14:50:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 10 monclient(hunting): authenticate will time out at 1436.864598s 2024-09-16T14:50:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847f7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f848009cb80 0x7f84801c9d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57550/0 (socket says 172.21.15.26:57550) 2024-09-16T14:50:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847f7fe640 1 -- 172.21.15.26:0/3126634225 learned_addr learned my addr 172.21.15.26:0/3126634225 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740424825 0 0) 0x7f848006e030 con 0x7f8480138ff0 2024-09-16T14:50:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): my global_id is 164322 2024-09-16T14:50:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8460003530 con 0x7f8480138ff0 2024-09-16T14:50:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459995926 0 0) 0x7f848006ddc0 con 0x7f848009cb80 2024-09-16T14:50:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): my global_id is 154574 2024-09-16T14:50:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f848006e030 con 0x7f848009cb80 2024-09-16T14:50:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1003693013 0 0) 0x7f8460003530 con 0x7f8480138ff0 2024-09-16T14:50:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f848006ddc0 con 0x7f8480138ff0 2024-09-16T14:50:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8478001bc0 con 0x7f8480138ff0 2024-09-16T14:50:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): dump: 2024-09-16T14:50:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.760 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.760 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.760 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.760 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:12.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838267365 0 0) 0x7f8480070ea0 con 0x7f8480132650 2024-09-16T14:50:12.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): my global_id is 174109 2024-09-16T14:50:12.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8460003530 con 0x7f8480132650 2024-09-16T14:50:12.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2343895131 0 0) 0x7f848006e030 con 0x7f848009cb80 2024-09-16T14:50:12.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8480070ea0 con 0x7f848009cb80 2024-09-16T14:50:12.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f84680018a0 con 0x7f848009cb80 2024-09-16T14:50:12.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:12.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): dump: 2024-09-16T14:50:12.761 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.761 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.761 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.761 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.761 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.761 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.761 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.761 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.761 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.762 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3523172660 0 0) 0x7f848006ddc0 con 0x7f8480138ff0 2024-09-16T14:50:12.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 >> v1:172.21.15.26:6790/0 conn(0x7f848009cb80 legacy=0x7f84801c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 >> v1:172.21.15.189:6789/0 conn(0x7f8480132650 legacy=0x7f84801ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:50:12.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 monclient: found mon.a 2024-09-16T14:50:12.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:12.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:12.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8480130450 con 0x7f8480138ff0 2024-09-16T14:50:12.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient: _finish_auth 0 2024-09-16T14:50:12.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:50:12.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:12.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 5 monclient: authenticate success, global_id 164322 2024-09-16T14:50:12.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 10 monclient: _renew_subs 2024-09-16T14:50:12.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:12.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 1 -- 172.21.15.26:0/3126634225 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f848009a2a0 con 0x7f8480138ff0 2024-09-16T14:50:12.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 10 monclient: _renew_subs 2024-09-16T14:50:12.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:12.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 1 -- 172.21.15.26:0/3126634225 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f848009bd40 con 0x7f8480138ff0 2024-09-16T14:50:12.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8478002090 con 0x7f8480138ff0 2024-09-16T14:50:12.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:12.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8478004d10 con 0x7f8480138ff0 2024-09-16T14:50:12.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847dffb640 4 set_mon_vals no callback set 2024-09-16T14:50:12.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:12.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient: dump: 2024-09-16T14:50:12.764 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.764 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.764 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.764 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.764 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.764 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.764 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.764 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.764 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.764 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:12.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8485e61640 1 -- 172.21.15.26:0/3126634225 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8464006590 con 0x7f8464002830 2024-09-16T14:50:12.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f847800f780 con 0x7f8480138ff0 2024-09-16T14:50:12.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(221..221 src has 169..221) v4 ==== 12812+0+0 (unknown 3418031008 0 0) 0x7f84780456d0 con 0x7f8480138ff0 2024-09-16T14:50:12.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041858406 0 0) 0x7f8464006590 con 0x7f8464002830 2024-09-16T14:50:12.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient(hunting): my global_id is 164325 2024-09-16T14:50:12.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f848006ddc0 con 0x7f8464002830 2024-09-16T14:50:12.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f8476ffd640 10 monclient: get_auth_request con 0x7f84600363b0 auth_method 0 2024-09-16T14:50:12.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1155919135 0 0) 0x7f848006ddc0 con 0x7f8464002830 2024-09-16T14:50:12.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8464006590 con 0x7f8464002830 2024-09-16T14:50:12.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f84680037c0 con 0x7f8464002830 2024-09-16T14:50:12.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:12.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient: dump: 2024-09-16T14:50:12.766 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.766 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.766 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.766 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.766 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.766 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.766 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.766 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.766 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.766 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 829756947 0 0) 0x7f8464006590 con 0x7f8464002830 2024-09-16T14:50:12.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f846400a0e0 con 0x7f8464002830 2024-09-16T14:50:12.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f84680057d0 con 0x7f8464002830 2024-09-16T14:50:12.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:12.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:12.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 >> v1:172.21.15.26:6789/0 conn(0x7f8464002830 legacy=0x7f8464004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.741+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 >> v1:172.21.15.26:6789/0 conn(0x7f8464002830 legacy=0x7f8464004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:12.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f8485e61640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:50:12.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f8485e61640 1 -- 172.21.15.26:0/3126634225 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8464009d80 con 0x7f84640090a0 2024-09-16T14:50:12.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635167954 0 0) 0x7f8464009d80 con 0x7f84640090a0 2024-09-16T14:50:12.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 10 monclient(hunting): my global_id is 164328 2024-09-16T14:50:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8464006590 con 0x7f84640090a0 2024-09-16T14:50:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2049114958 0 0) 0x7f8464006590 con 0x7f84640090a0 2024-09-16T14:50:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8464009d80 con 0x7f84640090a0 2024-09-16T14:50:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8468003e30 con 0x7f84640090a0 2024-09-16T14:50:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 10 monclient: dump: 2024-09-16T14:50:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.768 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.769 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.769 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.769 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.769 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2970126189 0 0) 0x7f8464009d80 con 0x7f84640090a0 2024-09-16T14:50:12.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f846400c290 con 0x7f84640090a0 2024-09-16T14:50:12.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 3946833191) 0x7f84680041d0 con 0x7f84640090a0 2024-09-16T14:50:12.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:50:12.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:12.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 >> v1:172.21.15.26:6789/0 conn(0x7f84640090a0 legacy=0x7f8464009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847cff9640 1 -- 172.21.15.26:0/3126634225 >> v1:172.21.15.26:6789/0 conn(0x7f84640090a0 legacy=0x7f8464009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:12.770 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:50:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:50:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 72, 2024-09-16T14:50:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:50:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:50:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:50:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:50:12.770 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:50:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:50:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:50:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:12.771 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:50:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:50:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:12.772 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:50:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:50:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:50:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:50:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:50:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:50:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:50:12.773 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:50:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:50:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:50:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:50:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:50:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:50:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:50:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:12.774 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:50:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:50:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:12.775 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:50:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:50:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:50:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:12.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:50:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:12.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:50:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:50:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:12.778 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:50:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:50:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:50:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:12.779 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:50:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:50:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:12.780 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:50:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:50:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:50:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:12.781 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.782 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:50:12.782 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:50:12.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f8485e61640 1 -- 172.21.15.26:0/3126634225 >> v1:172.21.15.189:6812/3196161582 conn(0x7f84600363b0 legacy=0x7f8460038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f8485e61640 10 monclient: shutdown 2024-09-16T14:50:12.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f8485e61640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:12.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f8485e61640 1 -- 172.21.15.26:0/3126634225 >> v1:172.21.15.26:6789/0 conn(0x7f8480138ff0 legacy=0x7f84801d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847ffff640 1 -- 172.21.15.26:0/3126634225 reap_dead start 2024-09-16T14:50:12.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f847ffff640 1 -- 172.21.15.26:0/3126634225 reap_dead start 2024-09-16T14:50:12.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f8485e61640 1 -- 172.21.15.26:0/3126634225 shutdown_connections 2024-09-16T14:50:12.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f8485e61640 1 -- 172.21.15.26:0/3126634225 >> 172.21.15.26:0/3126634225 conn(0x7f848009fb50 msgr2=0x7f84801388c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:12.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f8485e61640 1 -- 172.21.15.26:0/3126634225 shutdown_connections 2024-09-16T14:50:12.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.749+0000 7f8485e61640 1 -- 172.21.15.26:0/3126634225 wait complete. 2024-09-16T14:50:12.784 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:50:12.784 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:50:12.784 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:50:12.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.816+0000 7f913d045640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T14:50:12.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.816+0000 7f913d045640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T14:50:12.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.816+0000 7f913d045640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:12.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:12.816+0000 7f913d045640 1 -- 172.21.15.189:0/624524096 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) v1 -- 0x7f9118005300 con 0x7f913809e2f0 2024-09-16T14:50:12.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.885+0000 7f4a08343640 10 monclient: get_monmap_and_config 2024-09-16T14:50:12.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.885+0000 7f4a08343640 10 monclient: build_initial_monmap 2024-09-16T14:50:12.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.885+0000 7f4a08343640 10 monclient: monmap: 2024-09-16T14:50:12.892 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:12.892 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.892 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:12.893194+0000 2024-09-16T14:50:12.892 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:12.893194+0000 2024-09-16T14:50:12.893 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:12.893 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.893 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:12.893 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:12.893 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:12.893 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 1 Processor -- start 2024-09-16T14:50:12.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 1 -- start start 2024-09-16T14:50:12.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 10 monclient: init 2024-09-16T14:50:12.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:12.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:12.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 10 monclient(hunting): picked mon.noname-c con 0x7f4a000968c0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:12.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 10 monclient(hunting): picked mon.noname-b con 0x7f4a00098bc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:12.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 10 monclient(hunting): picked mon.noname-a con 0x7f4a00096ca0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:12.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a0006b2d0 con 0x7f4a00096ca0 2024-09-16T14:50:12.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a00097940 con 0x7f4a000968c0 2024-09-16T14:50:12.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a0009fec0 con 0x7f4a00098bc0 2024-09-16T14:50:12.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:12.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 10 monclient(hunting): authenticate will time out at 1437.013407s 2024-09-16T14:50:12.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a068e0640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4a00096ca0 0x7f4a0013da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48784/0 (socket says 172.21.15.26:48784) 2024-09-16T14:50:12.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a058de640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4a00098bc0 0x7f4a0009f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57562/0 (socket says 172.21.15.26:57562) 2024-09-16T14:50:12.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a068e0640 1 -- 172.21.15.26:0/510483266 learned_addr learned my addr 172.21.15.26:0/510483266 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:12.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 1 -- 172.21.15.26:0/510483266 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596153214 0 0) 0x7f4a0006b2d0 con 0x7f4a00096ca0 2024-09-16T14:50:12.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient(hunting): my global_id is 164331 2024-09-16T14:50:12.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 1 -- 172.21.15.26:0/510483266 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ec003530 con 0x7f4a00096ca0 2024-09-16T14:50:12.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 1 -- 172.21.15.26:0/510483266 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222859738 0 0) 0x7f4a0009fec0 con 0x7f4a00098bc0 2024-09-16T14:50:12.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient(hunting): my global_id is 154577 2024-09-16T14:50:12.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 1 -- 172.21.15.26:0/510483266 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a0006b2d0 con 0x7f4a00098bc0 2024-09-16T14:50:12.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 1 -- 172.21.15.26:0/510483266 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840893380 0 0) 0x7f4a00097940 con 0x7f4a000968c0 2024-09-16T14:50:12.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient(hunting): my global_id is 174112 2024-09-16T14:50:12.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 1 -- 172.21.15.26:0/510483266 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a0009fec0 con 0x7f4a000968c0 2024-09-16T14:50:12.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 1 -- 172.21.15.26:0/510483266 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1174662901 0 0) 0x7f4a0006b2d0 con 0x7f4a00098bc0 2024-09-16T14:50:12.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 1 -- 172.21.15.26:0/510483266 >> v1:172.21.15.189:6789/0 conn(0x7f4a000968c0 legacy=0x7f4a0009f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 1 -- 172.21.15.26:0/510483266 >> v1:172.21.15.26:6789/0 conn(0x7f4a00096ca0 legacy=0x7f4a0013da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:12.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 1 monclient: found mon.noname-b 2024-09-16T14:50:12.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:12.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:12.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 1 -- 172.21.15.26:0/510483266 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a00097c20 con 0x7f4a00098bc0 2024-09-16T14:50:12.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient: _finish_auth 0 2024-09-16T14:50:12.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:12.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:12.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 1 -- 172.21.15.26:0/510483266 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f49f0001580 con 0x7f4a00098bc0 2024-09-16T14:50:12.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:12.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient: dump: 2024-09-16T14:50:12.906 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.907 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.907 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.907 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.907 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.907 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.907 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.907 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.907 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.907 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 5 monclient: authenticate success, global_id 154577 2024-09-16T14:50:12.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:12.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 1 -- 172.21.15.26:0/510483266 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49f00017a0 con 0x7f4a00098bc0 2024-09-16T14:50:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 1 -- 172.21.15.26:0/510483266 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f49f0001bb0 con 0x7f4a00098bc0 2024-09-16T14:50:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 4 set_mon_vals no callback set 2024-09-16T14:50:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a050dd640 10 monclient: dump: 2024-09-16T14:50:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.909 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 10 monclient: shutdown 2024-09-16T14:50:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 1 -- 172.21.15.26:0/510483266 >> v1:172.21.15.26:6790/0 conn(0x7f4a00098bc0 legacy=0x7f4a0009f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 1 -- 172.21.15.26:0/510483266 shutdown_connections 2024-09-16T14:50:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 1 -- 172.21.15.26:0/510483266 >> 172.21.15.26:0/510483266 conn(0x7f4a000a0360 msgr2=0x7f4a000a0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:12.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 1 -- 172.21.15.26:0/510483266 shutdown_connections 2024-09-16T14:50:12.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 1 -- 172.21.15.26:0/510483266 wait complete. 2024-09-16T14:50:12.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 10 monclient: build_initial_monmap 2024-09-16T14:50:12.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.889+0000 7f4a08343640 10 monclient: monmap: 2024-09-16T14:50:12.910 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:12.910 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:12.910 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:12.910 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:12.910 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:12.910 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:12.910 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:12.911 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:12.911 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:12.911 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 1 Processor -- start 2024-09-16T14:50:12.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 1 -- start start 2024-09-16T14:50:12.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 10 monclient: init 2024-09-16T14:50:12.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:12.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:12.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 10 monclient(hunting): picked mon.noname-b con 0x7f4a000968c0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:12.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 10 monclient(hunting): picked mon.noname-c con 0x7f4a00096ca0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:12.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 10 monclient(hunting): picked mon.noname-a con 0x7f4a00098bc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a000916b0 con 0x7f4a00098bc0 2024-09-16T14:50:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a00091990 con 0x7f4a000968c0 2024-09-16T14:50:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a00090200 con 0x7f4a00096ca0 2024-09-16T14:50:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 10 monclient(hunting): authenticate will time out at 1437.016698s 2024-09-16T14:50:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a058de640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4a00096ca0 0x7f4a001ce860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57574/0 (socket says 172.21.15.26:57574) 2024-09-16T14:50:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a058de640 1 -- 172.21.15.26:0/316691385 learned_addr learned my addr 172.21.15.26:0/316691385 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155448394 0 0) 0x7f4a00090200 con 0x7f4a00096ca0 2024-09-16T14:50:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): my global_id is 154580 2024-09-16T14:50:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e4003530 con 0x7f4a00096ca0 2024-09-16T14:50:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804144251 0 0) 0x7f4a000916b0 con 0x7f4a00098bc0 2024-09-16T14:50:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): my global_id is 164334 2024-09-16T14:50:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a00090200 con 0x7f4a00098bc0 2024-09-16T14:50:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3567716540 0 0) 0x7f4a00090200 con 0x7f4a00098bc0 2024-09-16T14:50:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a000916b0 con 0x7f4a00098bc0 2024-09-16T14:50:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f49f8001480 con 0x7f4a00098bc0 2024-09-16T14:50:12.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:12.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:12.914 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.914 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.914 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.914 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.914 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.914 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.914 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.914 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.914 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.914 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:12.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104222950 0 0) 0x7f4a00091990 con 0x7f4a000968c0 2024-09-16T14:50:12.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): my global_id is 174115 2024-09-16T14:50:12.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a00090200 con 0x7f4a000968c0 2024-09-16T14:50:12.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 415096024 0 0) 0x7f49e4003530 con 0x7f4a00096ca0 2024-09-16T14:50:12.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a00091990 con 0x7f4a00096ca0 2024-09-16T14:50:12.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f49f0001af0 con 0x7f4a00096ca0 2024-09-16T14:50:12.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:12.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:12.916 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.916 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.916 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.916 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.916 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.916 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.916 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.916 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.916 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.916 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 421636495 0 0) 0x7f4a000916b0 con 0x7f4a00098bc0 2024-09-16T14:50:12.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 >> v1:172.21.15.26:6790/0 conn(0x7f4a00096ca0 legacy=0x7f4a001ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 >> v1:172.21.15.189:6789/0 conn(0x7f4a000968c0 legacy=0x7f4a001c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:12.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 monclient: found mon.a 2024-09-16T14:50:12.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:12.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:12.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a000a2c30 con 0x7f4a00098bc0 2024-09-16T14:50:12.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:12.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:12.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:12.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 5 monclient: authenticate success, global_id 164334 2024-09-16T14:50:12.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 10 monclient: _renew_subs 2024-09-16T14:50:12.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:12.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 1 -- 172.21.15.26:0/316691385 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a0009f810 con 0x7f4a00098bc0 2024-09-16T14:50:12.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 10 monclient: _renew_subs 2024-09-16T14:50:12.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:12.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 1 -- 172.21.15.26:0/316691385 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a000913d0 con 0x7f4a00098bc0 2024-09-16T14:50:12.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49f80019b0 con 0x7f4a00098bc0 2024-09-16T14:50:12.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:12.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f49f8004d10 con 0x7f4a00098bc0 2024-09-16T14:50:12.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:12.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient: dump: 2024-09-16T14:50:12.919 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.919 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.919 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.919 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.919 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.919 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.919 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.919 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.919 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.919 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a048dc640 4 set_mon_vals no callback set 2024-09-16T14:50:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f49f800f800 con 0x7f4a00098bc0 2024-09-16T14:50:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a08343640 1 -- 172.21.15.26:0/316691385 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f49e8006590 con 0x7f49e8002830 2024-09-16T14:50:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(221..221 src has 169..221) v4 ==== 12812+0+0 (unknown 3418031008 0 0) 0x7f49f8045870 con 0x7f4a00098bc0 2024-09-16T14:50:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133719958 0 0) 0x7f49e8006590 con 0x7f49e8002830 2024-09-16T14:50:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient(hunting): my global_id is 154583 2024-09-16T14:50:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a000916b0 con 0x7f49e8002830 2024-09-16T14:50:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f4a058de640 10 monclient: get_auth_request con 0x7f49e4036440 auth_method 0 2024-09-16T14:50:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2325607993 0 0) 0x7f4a000916b0 con 0x7f49e8002830 2024-09-16T14:50:12.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49e8006590 con 0x7f49e8002830 2024-09-16T14:50:12.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f49fc001790 con 0x7f49e8002830 2024-09-16T14:50:12.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:12.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient: dump: 2024-09-16T14:50:12.921 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.921 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.921 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.921 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.921 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.921 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.922 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.922 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.922 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.922 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3012213296 0 0) 0x7f49e8006590 con 0x7f49e8002830 2024-09-16T14:50:12.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f49e800a0e0 con 0x7f49e8002830 2024-09-16T14:50:12.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f49fc001b00 con 0x7f49e8002830 2024-09-16T14:50:12.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:12.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:12.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 >> v1:172.21.15.26:6790/0 conn(0x7f49e8002830 legacy=0x7f49e8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.893+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 >> v1:172.21.15.26:6790/0 conn(0x7f49e8002830 legacy=0x7f49e8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:12.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f4a08343640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:50:12.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f4a08343640 1 -- 172.21.15.26:0/316691385 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f49e8009d80 con 0x7f49e80090a0 2024-09-16T14:50:12.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1128836098 0 0) 0x7f49e8009d80 con 0x7f49e80090a0 2024-09-16T14:50:12.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:12.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:12.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:12.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 10 monclient(hunting): my global_id is 154586 2024-09-16T14:50:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e8006590 con 0x7f49e80090a0 2024-09-16T14:50:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3622047485 0 0) 0x7f49e8006590 con 0x7f49e80090a0 2024-09-16T14:50:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49e8009d80 con 0x7f49e80090a0 2024-09-16T14:50:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f49fc0037a0 con 0x7f49e80090a0 2024-09-16T14:50:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 10 monclient: dump: 2024-09-16T14:50:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.925 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:12.926 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 769282591 0 0) 0x7f49e8009d80 con 0x7f49e80090a0 2024-09-16T14:50:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f49e800c290 con 0x7f49e80090a0 2024-09-16T14:50:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 496825664) 0x7f49fc0059a0 con 0x7f49e80090a0 2024-09-16T14:50:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:50:12.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:12.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 >> v1:172.21.15.26:6790/0 conn(0x7f49e80090a0 legacy=0x7f49e8009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f49f6ffd640 1 -- 172.21.15.26:0/316691385 >> v1:172.21.15.26:6790/0 conn(0x7f49e80090a0 legacy=0x7f49e8009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:12.927 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:50:12.927 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:12.927 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:12.927 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:50:12.927 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 72, 2024-09-16T14:50:12.927 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:50:12.928 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:50:12.928 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:50:12.928 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:50:12.928 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:12.928 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:50:12.928 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:12.928 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:50:12.928 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:50:12.928 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:12.928 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:12.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:12.929 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:12.929 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:12.929 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:12.929 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:12.929 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:12.929 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:12.929 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:12.929 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:12.929 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:50:12.929 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:50:12.929 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:12.930 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:12.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:12.930 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:12.930 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:12.930 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:12.930 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:12.930 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:12.930 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:12.930 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:12.930 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:12.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.931 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:50:12.931 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:50:12.931 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:50:12.931 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:50:12.931 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:50:12.931 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:50:12.931 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:12.931 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:12.931 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:50:12.931 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:50:12.931 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:50:12.931 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:50:12.932 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:50:12.932 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:50:12.932 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:50:12.932 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:12.932 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:50:12.932 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:12.932 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:12.932 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:12.932 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:12.932 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:12.932 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:12.933 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:12.933 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:12.933 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:12.933 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:12.933 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:12.933 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:50:12.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:50:12.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.933 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:12.933 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:12.933 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:12.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:12.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.934 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:12.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:50:12.934 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:12.934 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:12.934 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:12.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:50:12.934 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:50:12.934 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:12.934 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:12.935 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:12.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.935 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:12.935 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:12.935 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:12.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:12.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:12.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:50:12.936 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:12.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:12.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:12.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:50:12.936 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:50:12.936 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:12.936 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:12.936 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:12.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.937 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:12.937 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:12.937 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:12.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:12.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.937 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:12.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:50:12.937 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:12.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:12.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:12.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:50:12.937 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:50:12.938 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:12.938 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:12.938 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:12.938 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:12.938 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:12.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.938 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:50:12.938 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:50:12.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.938 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:12.938 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:12.939 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:12.939 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:12.939 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:12.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:50:12.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.939 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:12.939 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:12.939 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:50:12.939 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:12.939 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:12.939 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:50:12.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:12.940 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:12.940 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:12.940 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:50:12.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:12.940 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:12.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:12.940 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:50:12.940 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:50:12.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f4a08343640 1 -- 172.21.15.26:0/316691385 >> v1:172.21.15.189:6812/3196161582 conn(0x7f49e4036440 legacy=0x7f49e4038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f4a08343640 10 monclient: shutdown 2024-09-16T14:50:12.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f4a08343640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:12.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f4a08343640 1 -- 172.21.15.26:0/316691385 >> v1:172.21.15.26:6789/0 conn(0x7f4a00098bc0 legacy=0x7f4a001d1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:12.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f4a068e0640 1 -- 172.21.15.26:0/316691385 reap_dead start 2024-09-16T14:50:12.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f4a068e0640 1 -- 172.21.15.26:0/316691385 reap_dead start 2024-09-16T14:50:12.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f4a08343640 1 -- 172.21.15.26:0/316691385 shutdown_connections 2024-09-16T14:50:12.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f4a08343640 1 -- 172.21.15.26:0/316691385 >> 172.21.15.26:0/316691385 conn(0x7f4a000a0360 msgr2=0x7f4a000956a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:12.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f4a08343640 1 -- 172.21.15.26:0/316691385 shutdown_connections 2024-09-16T14:50:12.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:12.901+0000 7f4a08343640 1 -- 172.21.15.26:0/316691385 wait complete. 2024-09-16T14:50:12.943 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:50:12.943 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:50:12.943 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:50:13.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.061+0000 7f9e1a965640 10 monclient: get_monmap_and_config 2024-09-16T14:50:13.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.061+0000 7f9e1a965640 10 monclient: build_initial_monmap 2024-09-16T14:50:13.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.061+0000 7f9e1a965640 10 monclient: monmap: 2024-09-16T14:50:13.068 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:13.068 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.068 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:13.068909+0000 2024-09-16T14:50:13.068 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:13.068909+0000 2024-09-16T14:50:13.068 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:13.068 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:13.068 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:13.068 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:13.069 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:13.069 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:13.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 1 Processor -- start 2024-09-16T14:50:13.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 1 -- start start 2024-09-16T14:50:13.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 10 monclient: init 2024-09-16T14:50:13.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:13.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:13.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 10 monclient(hunting): picked mon.noname-b con 0x7f9e1406c5a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:13.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 10 monclient(hunting): picked mon.noname-c con 0x7f9e140a23b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:13.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 10 monclient(hunting): picked mon.noname-a con 0x7f9e140a27c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:13.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1406cee0 con 0x7f9e140a27c0 2024-09-16T14:50:13.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1406c980 con 0x7f9e140a23b0 2024-09-16T14:50:13.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e140a15e0 con 0x7f9e1406c5a0 2024-09-16T14:50:13.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:13.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 10 monclient(hunting): authenticate will time out at 1437.188903s 2024-09-16T14:50:13.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e18f02640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9e140a27c0 0x7f9e1413d5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48798/0 (socket says 172.21.15.26:48798) 2024-09-16T14:50:13.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e18f02640 1 -- 172.21.15.26:0/850735286 learned_addr learned my addr 172.21.15.26:0/850735286 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:13.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 1 -- 172.21.15.26:0/850735286 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164648660 0 0) 0x7f9e1406cee0 con 0x7f9e140a27c0 2024-09-16T14:50:13.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient(hunting): my global_id is 164337 2024-09-16T14:50:13.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 1 -- 172.21.15.26:0/850735286 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dfc003530 con 0x7f9e140a27c0 2024-09-16T14:50:13.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 1 -- 172.21.15.26:0/850735286 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156477262 0 0) 0x7f9e140a15e0 con 0x7f9e1406c5a0 2024-09-16T14:50:13.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient(hunting): my global_id is 154589 2024-09-16T14:50:13.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 1 -- 172.21.15.26:0/850735286 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e1406cee0 con 0x7f9e1406c5a0 2024-09-16T14:50:13.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 1 -- 172.21.15.26:0/850735286 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980579959 0 0) 0x7f9e1406c980 con 0x7f9e140a23b0 2024-09-16T14:50:13.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient(hunting): my global_id is 174118 2024-09-16T14:50:13.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 1 -- 172.21.15.26:0/850735286 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e140a15e0 con 0x7f9e140a23b0 2024-09-16T14:50:13.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 1 -- 172.21.15.26:0/850735286 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1588739400 0 0) 0x7f9dfc003530 con 0x7f9e140a27c0 2024-09-16T14:50:13.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 1 -- 172.21.15.26:0/850735286 >> v1:172.21.15.26:6790/0 conn(0x7f9e1406c5a0 legacy=0x7f9e140a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 1 -- 172.21.15.26:0/850735286 >> v1:172.21.15.189:6789/0 conn(0x7f9e140a23b0 legacy=0x7f9e14139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:13.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 1 monclient: found mon.noname-a 2024-09-16T14:50:13.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:13.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:13.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 1 -- 172.21.15.26:0/850735286 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1409edd0 con 0x7f9e140a27c0 2024-09-16T14:50:13.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 1 -- 172.21.15.26:0/850735286 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e0c0019a0 con 0x7f9e140a27c0 2024-09-16T14:50:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient: dump: 2024-09-16T14:50:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.082 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:13.083 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 5 monclient: authenticate success, global_id 164337 2024-09-16T14:50:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 1 -- 172.21.15.26:0/850735286 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e0c001c80 con 0x7f9e140a27c0 2024-09-16T14:50:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 1 -- 172.21.15.26:0/850735286 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e0c002010 con 0x7f9e140a27c0 2024-09-16T14:50:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e12ffd640 10 monclient: dump: 2024-09-16T14:50:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:13.084 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:13.085 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:13.085 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:13.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:13.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 4 set_mon_vals no callback set 2024-09-16T14:50:13.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 10 monclient: shutdown 2024-09-16T14:50:13.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:13.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 1 -- 172.21.15.26:0/850735286 >> v1:172.21.15.26:6789/0 conn(0x7f9e140a27c0 legacy=0x7f9e1413d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 1 -- 172.21.15.26:0/850735286 shutdown_connections 2024-09-16T14:50:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 1 -- 172.21.15.26:0/850735286 >> 172.21.15.26:0/850735286 conn(0x7f9e140a3da0 msgr2=0x7f9e140a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 1 -- 172.21.15.26:0/850735286 shutdown_connections 2024-09-16T14:50:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 1 -- 172.21.15.26:0/850735286 wait complete. 2024-09-16T14:50:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 10 monclient: build_initial_monmap 2024-09-16T14:50:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.065+0000 7f9e1a965640 10 monclient: monmap: 2024-09-16T14:50:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:13.086 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:13.087 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 1 Processor -- start 2024-09-16T14:50:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 1 -- start start 2024-09-16T14:50:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 10 monclient: init 2024-09-16T14:50:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:13.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 10 monclient(hunting): picked mon.noname-b con 0x7f9e1406c5a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 10 monclient(hunting): picked mon.noname-a con 0x7f9e140a23b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 10 monclient(hunting): picked mon.noname-c con 0x7f9e140a27c0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e141c6900 con 0x7f9e140a23b0 2024-09-16T14:50:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1409e490 con 0x7f9e1406c5a0 2024-09-16T14:50:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e14135a90 con 0x7f9e140a27c0 2024-09-16T14:50:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 10 monclient(hunting): authenticate will time out at 1437.192731s 2024-09-16T14:50:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e137fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9e140a23b0 0x7f9e141ce720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48804/0 (socket says 172.21.15.26:48804) 2024-09-16T14:50:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e137fe640 1 -- 172.21.15.26:0/2892836159 learned_addr learned my addr 172.21.15.26:0/2892836159 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451330097 0 0) 0x7f9e141c6900 con 0x7f9e140a23b0 2024-09-16T14:50:13.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): my global_id is 164340 2024-09-16T14:50:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e00003530 con 0x7f9e140a23b0 2024-09-16T14:50:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963945573 0 0) 0x7f9e14135a90 con 0x7f9e140a27c0 2024-09-16T14:50:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): my global_id is 154592 2024-09-16T14:50:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e141c6900 con 0x7f9e140a27c0 2024-09-16T14:50:13.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2760244173 0 0) 0x7f9e00003530 con 0x7f9e140a23b0 2024-09-16T14:50:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e14135a90 con 0x7f9e140a23b0 2024-09-16T14:50:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e080019a0 con 0x7f9e140a23b0 2024-09-16T14:50:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): dump: 2024-09-16T14:50:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:13.090 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:13.091 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2108411743 0 0) 0x7f9e141c6900 con 0x7f9e140a27c0 2024-09-16T14:50:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e00003530 con 0x7f9e140a27c0 2024-09-16T14:50:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e0c0017c0 con 0x7f9e140a27c0 2024-09-16T14:50:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:13.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): dump: 2024-09-16T14:50:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:13.092 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:13.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804824721 0 0) 0x7f9e1409e490 con 0x7f9e1406c5a0 2024-09-16T14:50:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): my global_id is 174121 2024-09-16T14:50:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e141c6900 con 0x7f9e1406c5a0 2024-09-16T14:50:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3045792796 0 0) 0x7f9e14135a90 con 0x7f9e140a23b0 2024-09-16T14:50:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 >> v1:172.21.15.26:6790/0 conn(0x7f9e140a27c0 legacy=0x7f9e141d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 >> v1:172.21.15.189:6789/0 conn(0x7f9e1406c5a0 legacy=0x7f9e14135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:50:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 monclient: found mon.a 2024-09-16T14:50:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:13.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:13.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e141335b0 con 0x7f9e140a23b0 2024-09-16T14:50:13.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient: _finish_auth 0 2024-09-16T14:50:13.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:50:13.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:13.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 5 monclient: authenticate success, global_id 164340 2024-09-16T14:50:13.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 10 monclient: _renew_subs 2024-09-16T14:50:13.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:13.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 1 -- 172.21.15.26:0/2892836159 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e14133890 con 0x7f9e140a23b0 2024-09-16T14:50:13.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 10 monclient: _renew_subs 2024-09-16T14:50:13.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:13.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 1 -- 172.21.15.26:0/2892836159 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e140a6490 con 0x7f9e140a23b0 2024-09-16T14:50:13.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e080036a0 con 0x7f9e140a23b0 2024-09-16T14:50:13.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:13.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e080039b0 con 0x7f9e140a23b0 2024-09-16T14:50:13.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:13.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:13.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e127fc640 4 set_mon_vals no callback set 2024-09-16T14:50:13.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient: dump: 2024-09-16T14:50:13.095 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:13.095 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.095 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.095 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.096 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:13.096 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:13.096 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:13.096 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:13.096 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:13.096 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:13.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:13.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e1a965640 1 -- 172.21.15.26:0/2892836159 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9df4006590 con 0x7f9df4002830 2024-09-16T14:50:13.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9e08003b10 con 0x7f9e140a23b0 2024-09-16T14:50:13.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(221..221 src has 169..221) v4 ==== 12812+0+0 (unknown 3418031008 0 0) 0x7f9e08045550 con 0x7f9e140a23b0 2024-09-16T14:50:13.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e18f02640 10 monclient: get_auth_request con 0x7f9e00036380 auth_method 0 2024-09-16T14:50:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586703423 0 0) 0x7f9df4006590 con 0x7f9df4002830 2024-09-16T14:50:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient(hunting): my global_id is 174124 2024-09-16T14:50:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e14135a90 con 0x7f9df4002830 2024-09-16T14:50:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3948772668 0 0) 0x7f9e14135a90 con 0x7f9df4002830 2024-09-16T14:50:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9df4006590 con 0x7f9df4002830 2024-09-16T14:50:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e04001690 con 0x7f9df4002830 2024-09-16T14:50:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:13.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.069+0000 7f9e10ff9640 10 monclient: dump: 2024-09-16T14:50:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:13.098 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.073+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 256442111 0 0) 0x7f9df4006590 con 0x7f9df4002830 2024-09-16T14:50:13.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.073+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9df400a0e0 con 0x7f9df4002830 2024-09-16T14:50:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.073+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f9e040019f0 con 0x7f9df4002830 2024-09-16T14:50:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.073+0000 7f9e10ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.073+0000 7f9e10ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.073+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 >> v1:172.21.15.189:6789/0 conn(0x7f9df4002830 legacy=0x7f9df4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.073+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 >> v1:172.21.15.189:6789/0 conn(0x7f9df4002830 legacy=0x7f9df4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.077+0000 7f9e1a965640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:50:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.077+0000 7f9e1a965640 1 -- 172.21.15.26:0/2892836159 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9df4009da0 con 0x7f9df40090c0 2024-09-16T14:50:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.077+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204182474 0 0) 0x7f9df4009da0 con 0x7f9df40090c0 2024-09-16T14:50:13.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.077+0000 7f9e10ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.077+0000 7f9e10ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.077+0000 7f9e10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.077+0000 7f9e10ff9640 10 monclient(hunting): my global_id is 174127 2024-09-16T14:50:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.077+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df4006590 con 0x7f9df40090c0 2024-09-16T14:50:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 48729051 0 0) 0x7f9df4006590 con 0x7f9df40090c0 2024-09-16T14:50:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9df4009da0 con 0x7f9df40090c0 2024-09-16T14:50:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e040037c0 con 0x7f9df40090c0 2024-09-16T14:50:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e10ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e10ff9640 10 monclient: dump: 2024-09-16T14:50:13.101 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:13.102 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2646143745 0 0) 0x7f9df4009da0 con 0x7f9df40090c0 2024-09-16T14:50:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9df400c290 con 0x7f9df40090c0 2024-09-16T14:50:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 3320709053) 0x7f9e040039d0 con 0x7f9df40090c0 2024-09-16T14:50:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e10ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:50:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e10ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 >> v1:172.21.15.189:6789/0 conn(0x7f9df40090c0 legacy=0x7f9df4009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e10ff9640 1 -- 172.21.15.26:0/2892836159 >> v1:172.21.15.189:6789/0 conn(0x7f9df40090c0 legacy=0x7f9df4009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:13.103 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:50:13.103 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:13.103 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:13.104 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:50:13.104 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 72, 2024-09-16T14:50:13.104 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:50:13.104 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:50:13.104 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:50:13.104 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:50:13.104 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:13.104 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:50:13.104 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:13.104 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:50:13.104 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:50:13.105 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:13.105 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:13.105 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:13.105 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:13.105 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:13.105 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:13.105 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:13.105 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:13.105 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:13.105 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:13.105 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:13.106 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:50:13.106 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:50:13.106 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:13.106 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:13.106 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:13.106 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:13.106 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:13.106 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:13.106 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:13.106 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:13.106 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:13.107 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:13.107 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:13.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:13.107 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:50:13.107 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:50:13.107 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:50:13.107 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:50:13.107 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:50:13.107 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:50:13.107 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:13.107 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:13.108 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:50:13.108 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:50:13.108 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:50:13.108 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:50:13.108 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:50:13.108 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:50:13.108 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:50:13.108 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:13.108 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:50:13.108 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:13.108 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:13.109 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:13.109 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:13.109 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:13.109 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:13.109 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:13.109 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:13.109 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:13.109 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:13.109 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:13.109 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:50:13.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:13.109 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:50:13.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:13.110 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:13.110 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:13.110 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:13.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:13.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:13.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:13.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:50:13.110 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:13.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:13.110 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:13.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:13.111 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:50:13.111 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:50:13.111 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:13.111 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:13.111 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:13.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:13.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:13.111 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:13.111 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:13.111 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:13.111 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:13.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:13.112 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:13.112 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:50:13.112 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:13.112 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:13.112 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:13.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:13.112 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:50:13.112 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:50:13.112 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:13.112 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:13.112 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:13.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:13.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:13.113 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:13.113 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:13.113 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:13.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:13.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:13.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:13.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:50:13.113 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:13.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:13.113 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:13.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:13.114 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:50:13.114 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:50:13.114 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:13.114 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:13.114 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:13.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:13.114 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:13.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:13.114 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:50:13.114 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:50:13.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:13.115 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:13.115 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:13.115 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:13.115 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:13.115 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:13.115 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:50:13.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:13.115 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:13.115 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:13.115 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:50:13.116 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:13.116 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:13.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:13.116 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:50:13.116 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:50:13.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e1a965640 1 -- 172.21.15.26:0/2892836159 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9e00036380 legacy=0x7f9e00038840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e1a965640 10 monclient: shutdown 2024-09-16T14:50:13.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e1a965640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:13.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e1a965640 1 -- 172.21.15.26:0/2892836159 >> v1:172.21.15.26:6789/0 conn(0x7f9e140a23b0 legacy=0x7f9e141ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e18f02640 1 -- 172.21.15.26:0/2892836159 reap_dead start 2024-09-16T14:50:13.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e18f02640 1 -- 172.21.15.26:0/2892836159 reap_dead start 2024-09-16T14:50:13.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e1a965640 1 -- 172.21.15.26:0/2892836159 shutdown_connections 2024-09-16T14:50:13.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e1a965640 1 -- 172.21.15.26:0/2892836159 >> 172.21.15.26:0/2892836159 conn(0x7f9e140a3da0 msgr2=0x7f9e140a51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:13.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e1a965640 1 -- 172.21.15.26:0/2892836159 shutdown_connections 2024-09-16T14:50:13.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.081+0000 7f9e1a965640 1 -- 172.21.15.26:0/2892836159 wait complete. 2024-09-16T14:50:13.118 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:50:13.119 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:50:13.119 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:50:13.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.225+0000 7f6fbd598640 10 monclient: get_monmap_and_config 2024-09-16T14:50:13.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.225+0000 7f6fbd598640 10 monclient: build_initial_monmap 2024-09-16T14:50:13.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.225+0000 7f6fbd598640 10 monclient: monmap: 2024-09-16T14:50:13.233 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:13.233 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.233 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:13.232146+0000 2024-09-16T14:50:13.233 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:13.232146+0000 2024-09-16T14:50:13.233 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:13.233 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:13.233 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:13.233 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:13.233 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:13.233 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:13.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.225+0000 7f6fbd598640 1 Processor -- start 2024-09-16T14:50:13.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.225+0000 7f6fbd598640 1 -- start start 2024-09-16T14:50:13.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.225+0000 7f6fbd598640 10 monclient: init 2024-09-16T14:50:13.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.225+0000 7f6fbd598640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:13.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.225+0000 7f6fbd598640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:13.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.225+0000 7f6fbd598640 10 monclient(hunting): picked mon.noname-b con 0x7f6fb80a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:13.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.225+0000 7f6fbd598640 10 monclient(hunting): picked mon.noname-a con 0x7f6fb80a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:13.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.225+0000 7f6fbd598640 10 monclient(hunting): picked mon.noname-c con 0x7f6fb80a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:13.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.225+0000 7f6fbd598640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb80a3250 con 0x7f6fb80a0000 2024-09-16T14:50:13.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.225+0000 7f6fbd598640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb809f600 con 0x7f6fb80a2800 2024-09-16T14:50:13.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb80a3830 con 0x7f6fb80a1d10 2024-09-16T14:50:13.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:13.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient(hunting): authenticate will time out at 1437.352022s 2024-09-16T14:50:13.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb6ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6fb80a1d10 0x7f6fb80a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57618/0 (socket says 172.21.15.26:57618) 2024-09-16T14:50:13.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb6ffd640 1 -- 172.21.15.26:0/1442021235 learned_addr learned my addr 172.21.15.26:0/1442021235 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 1 -- 172.21.15.26:0/1442021235 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878741744 0 0) 0x7f6fb80a3830 con 0x7f6fb80a1d10 2024-09-16T14:50:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient(hunting): my global_id is 154595 2024-09-16T14:50:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 1 -- 172.21.15.26:0/1442021235 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fa0003530 con 0x7f6fb80a1d10 2024-09-16T14:50:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 1 -- 172.21.15.26:0/1442021235 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055900416 0 0) 0x7f6fb80a3250 con 0x7f6fb80a0000 2024-09-16T14:50:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient(hunting): my global_id is 164343 2024-09-16T14:50:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 1 -- 172.21.15.26:0/1442021235 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb80a3830 con 0x7f6fb80a0000 2024-09-16T14:50:13.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 1 -- 172.21.15.26:0/1442021235 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3705591783 0 0) 0x7f6fa0003530 con 0x7f6fb80a1d10 2024-09-16T14:50:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 1 -- 172.21.15.26:0/1442021235 >> v1:172.21.15.189:6789/0 conn(0x7f6fb80a2800 legacy=0x7f6fb813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 1 -- 172.21.15.26:0/1442021235 >> v1:172.21.15.26:6789/0 conn(0x7f6fb80a0000 legacy=0x7f6fb8139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:50:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 1 monclient: found mon.noname-b 2024-09-16T14:50:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 1 -- 172.21.15.26:0/1442021235 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb809ed60 con 0x7f6fb80a1d10 2024-09-16T14:50:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient: _finish_auth 0 2024-09-16T14:50:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 1 -- 172.21.15.26:0/1442021235 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6fa80019a0 con 0x7f6fb80a1d10 2024-09-16T14:50:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:13.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient: dump: 2024-09-16T14:50:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:13.243 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 5 monclient: authenticate success, global_id 154595 2024-09-16T14:50:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 1 -- 172.21.15.26:0/1442021235 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fa8001c80 con 0x7f6fb80a1d10 2024-09-16T14:50:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 1 -- 172.21.15.26:0/1442021235 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6fa8002010 con 0x7f6fb80a1d10 2024-09-16T14:50:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 4 set_mon_vals no callback set 2024-09-16T14:50:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb5ffb640 10 monclient: dump: 2024-09-16T14:50:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:13.245 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:13.245 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:13.245 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:13.245 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:13.245 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:13.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient: shutdown 2024-09-16T14:50:13.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:13.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 1 -- 172.21.15.26:0/1442021235 >> v1:172.21.15.26:6790/0 conn(0x7f6fb80a1d10 legacy=0x7f6fb80a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 1 -- 172.21.15.26:0/1442021235 shutdown_connections 2024-09-16T14:50:13.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 1 -- 172.21.15.26:0/1442021235 >> 172.21.15.26:0/1442021235 conn(0x7f6fb80a7f30 msgr2=0x7f6fb80a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:13.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 1 -- 172.21.15.26:0/1442021235 shutdown_connections 2024-09-16T14:50:13.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 1 -- 172.21.15.26:0/1442021235 wait complete. 2024-09-16T14:50:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient: build_initial_monmap 2024-09-16T14:50:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient: monmap: 2024-09-16T14:50:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:13.246 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 1 Processor -- start 2024-09-16T14:50:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 1 -- start start 2024-09-16T14:50:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient: init 2024-09-16T14:50:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient(hunting): picked mon.noname-c con 0x7f6fb80a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient(hunting): picked mon.noname-b con 0x7f6fb80a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient(hunting): picked mon.noname-a con 0x7f6fb80a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb813a7c0 con 0x7f6fb80a2800 2024-09-16T14:50:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb813a180 con 0x7f6fb80a1d10 2024-09-16T14:50:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb809e500 con 0x7f6fb80a0000 2024-09-16T14:50:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb6ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6fb80a0000 0x7f6fb8139360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57628/0 (socket says 172.21.15.26:57628) 2024-09-16T14:50:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb6ffd640 1 -- 172.21.15.26:0/1436936944 learned_addr learned my addr 172.21.15.26:0/1436936944 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:13.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient(hunting): authenticate will time out at 1437.355070s 2024-09-16T14:50:13.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411968449 0 0) 0x7f6fb809e500 con 0x7f6fb80a0000 2024-09-16T14:50:13.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): my global_id is 154598 2024-09-16T14:50:13.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fa4003530 con 0x7f6fb80a0000 2024-09-16T14:50:13.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953093775 0 0) 0x7f6fb813a7c0 con 0x7f6fb80a2800 2024-09-16T14:50:13.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): my global_id is 164346 2024-09-16T14:50:13.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb809e500 con 0x7f6fb80a2800 2024-09-16T14:50:13.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4141363027 0 0) 0x7f6fa4003530 con 0x7f6fb80a0000 2024-09-16T14:50:13.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fb813a7c0 con 0x7f6fb80a0000 2024-09-16T14:50:13.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6fa8001960 con 0x7f6fb80a0000 2024-09-16T14:50:13.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:13.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:13.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): dump: 2024-09-16T14:50:13.249 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:13.249 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.249 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.249 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.249 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:13.249 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:13.250 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:13.250 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:13.250 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:13.250 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:13.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:13.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 972611780 0 0) 0x7f6fb809e500 con 0x7f6fb80a2800 2024-09-16T14:50:13.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fa4003530 con 0x7f6fb80a2800 2024-09-16T14:50:13.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6fb00016f0 con 0x7f6fb80a2800 2024-09-16T14:50:13.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:13.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:13.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): dump: 2024-09-16T14:50:13.251 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:13.251 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.251 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.251 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.251 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:13.251 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:13.251 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:13.251 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:13.251 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:13.251 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:13.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901262826 0 0) 0x7f6fb813a180 con 0x7f6fb80a1d10 2024-09-16T14:50:13.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient(hunting): my global_id is 174133 2024-09-16T14:50:13.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb809e500 con 0x7f6fb80a1d10 2024-09-16T14:50:13.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1513539115 0 0) 0x7f6fb813a7c0 con 0x7f6fb80a0000 2024-09-16T14:50:13.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 >> v1:172.21.15.189:6789/0 conn(0x7f6fb80a1d10 legacy=0x7f6fb8139a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 >> v1:172.21.15.26:6789/0 conn(0x7f6fb80a2800 legacy=0x7f6fb81d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:13.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 monclient: found mon.c 2024-09-16T14:50:13.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:13.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:13.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb813c920 con 0x7f6fb80a0000 2024-09-16T14:50:13.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:13.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:13.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:13.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 5 monclient: authenticate success, global_id 154598 2024-09-16T14:50:13.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient: _renew_subs 2024-09-16T14:50:13.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:13.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 1 -- 172.21.15.26:0/1436936944 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb813cc00 con 0x7f6fb80a0000 2024-09-16T14:50:13.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient: _renew_subs 2024-09-16T14:50:13.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:13.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fbd598640 1 -- 172.21.15.26:0/1436936944 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb81356a0 con 0x7f6fb80a0000 2024-09-16T14:50:13.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fa8004f80 con 0x7f6fb80a0000 2024-09-16T14:50:13.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:13.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6fa80015a0 con 0x7f6fb80a0000 2024-09-16T14:50:13.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:13.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:13.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6fb57fa640 4 set_mon_vals no callback set 2024-09-16T14:50:13.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.229+0000 7f6f9f7fe640 10 monclient: dump: 2024-09-16T14:50:13.254 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:13.254 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.254 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.254 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.254 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:13.254 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:13.254 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:13.254 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:13.254 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:13.254 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:13.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.233+0000 7f6fbd598640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:13.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.233+0000 7f6fbd598640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:13.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.233+0000 7f6fbd598640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:13.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.233+0000 7f6fbd598640 1 -- 172.21.15.26:0/1436936944 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f98005050 con 0x7f6fb80a0000 2024-09-16T14:50:13.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.233+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6fa800fc20 con 0x7f6fb80a0000 2024-09-16T14:50:13.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.233+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(221..221 src has 169..221) v4 ==== 12812+0+0 (unknown 3418031008 0 0) 0x7f6fa8045b70 con 0x7f6fb80a0000 2024-09-16T14:50:13.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.233+0000 7f6fb67fc640 10 monclient: get_auth_request con 0x7f6fa4036440 auth_method 0 2024-09-16T14:50:13.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.237+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6fa8014b80 con 0x7f6fb80a0000 2024-09-16T14:50:13.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.237+0000 7f6f9f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:13.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.237+0000 7f6f9f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:13.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.440+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h7"}]=0 removed item id -37 name 'h7' from crush map v222) v1 ==== 118+0+0 (unknown 1669829578 0 0) 0x7f912c014350 con 0x7f913809e2f0 2024-09-16T14:50:13.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.440+0000 7f911f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T14:50:13.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.440+0000 7f911f7fe640 10 monclient: _finish_command 2 = system:0 removed item id -37 name 'h7' from crush map 2024-09-16T14:50:13.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.453+0000 7f6fbd598640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:50:13.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.453+0000 7f6fbd598640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:50:13.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.453+0000 7f6fbd598640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:13.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.453+0000 7f6fbd598640 1 -- 172.21.15.26:0/1436936944 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f6f98005b10 con 0x7f6fb80a0000 2024-09-16T14:50:13.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.453+0000 7f6f9f7fe640 1 -- 172.21.15.26:0/1436936944 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f6fa8013220 con 0x7f6fb80a0000 2024-09-16T14:50:13.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.453+0000 7f6f9f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:50:13.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.453+0000 7f6f9f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:13.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.453+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.453+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=62496690,config=692619927,health=2960323591,kv=1738244453,logm=1647544626}) 2024-09-16T14:50:13.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.453+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=11,kv=10,logm=54} crc {auth=3463148586,config=692619927,health=1411272771,kv=3561600421,logm=1722426329}) 2024-09-16T14:50:13.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.453+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.453+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=62496690,config=692619927,health=2960323591,kv=1738244453,logm=1647544626}) 2024-09-16T14:50:13.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.453+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=11,kv=11,logm=56} crc {auth=1803004292,config=692619927,health=4187967776,kv=2091461713,logm=1772069274}) 2024-09-16T14:50:13.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.457+0000 7f6fbd598640 1 -- 172.21.15.26:0/1436936944 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6fa4036440 legacy=0x7f6fa4038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.457+0000 7f6fbd598640 10 monclient: shutdown 2024-09-16T14:50:13.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.457+0000 7f6fbd598640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:13.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.457+0000 7f6fbd598640 1 -- 172.21.15.26:0/1436936944 >> v1:172.21.15.26:6790/0 conn(0x7f6fb80a0000 legacy=0x7f6fb8139360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.457+0000 7f6fbd598640 1 -- 172.21.15.26:0/1436936944 shutdown_connections 2024-09-16T14:50:13.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.457+0000 7f6fbd598640 1 -- 172.21.15.26:0/1436936944 >> 172.21.15.26:0/1436936944 conn(0x7f6fb80a7f30 msgr2=0x7f6fb809d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:13.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.457+0000 7f6fbd598640 1 -- 172.21.15.26:0/1436936944 shutdown_connections 2024-09-16T14:50:13.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:13.457+0000 7f6fbd598640 1 -- 172.21.15.26:0/1436936944 wait complete. 2024-09-16T14:50:13.462 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.457+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.457+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3155896985}) 2024-09-16T14:50:13.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.457+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3685592235}) 2024-09-16T14:50:13.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.457+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.457+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3155896985}) 2024-09-16T14:50:13.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.457+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2288451848}) 2024-09-16T14:50:13.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=239437217}) 2024-09-16T14:50:13.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3717058880}) 2024-09-16T14:50:13.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=239437217}) 2024-09-16T14:50:13.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4040848377}) 2024-09-16T14:50:13.467 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.467 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3684063142}) 2024-09-16T14:50:13.467 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=838173325}) 2024-09-16T14:50:13.467 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.467 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3684063142}) 2024-09-16T14:50:13.467 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2137910637}) 2024-09-16T14:50:13.469 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.465+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.469 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.465+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276187221}) 2024-09-16T14:50:13.469 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.465+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1349865756}) 2024-09-16T14:50:13.469 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.465+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.469 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.465+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276187221}) 2024-09-16T14:50:13.469 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.465+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=73479553}) 2024-09-16T14:50:13.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.465+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.465+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2536184713}) 2024-09-16T14:50:13.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.465+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4289954481}) 2024-09-16T14:50:13.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.465+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.465+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2536184713}) 2024-09-16T14:50:13.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.465+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1638063845}) 2024-09-16T14:50:13.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1724598827}) 2024-09-16T14:50:13.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=280657775}) 2024-09-16T14:50:13.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1724598827}) 2024-09-16T14:50:13.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2709240557}) 2024-09-16T14:50:13.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3003835949}) 2024-09-16T14:50:13.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=351554469}) 2024-09-16T14:50:13.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.475 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3003835949}) 2024-09-16T14:50:13.475 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1117429412}) 2024-09-16T14:50:13.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2785431877}) 2024-09-16T14:50:13.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3120828586}) 2024-09-16T14:50:13.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.469+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2785431877}) 2024-09-16T14:50:13.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.473+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445350439}) 2024-09-16T14:50:13.478 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:50:13.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.473+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.473+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=779264475}) 2024-09-16T14:50:13.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.473+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2203897846}) 2024-09-16T14:50:13.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.473+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.473+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=779264475}) 2024-09-16T14:50:13.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.473+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1836075905}) 2024-09-16T14:50:13.480 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.473+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.480 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.473+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4230625541}) 2024-09-16T14:50:13.480 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.473+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2982394933}) 2024-09-16T14:50:13.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.473+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.473+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4230625541}) 2024-09-16T14:50:13.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.473+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2953071534}) 2024-09-16T14:50:13.482 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.477+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.482 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.477+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4037408633}) 2024-09-16T14:50:13.482 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.477+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2283456108}) 2024-09-16T14:50:13.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.477+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.477+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4037408633}) 2024-09-16T14:50:13.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.477+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4038266948}) 2024-09-16T14:50:13.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.477+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.477+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2271795395}) 2024-09-16T14:50:13.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.477+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1781402702}) 2024-09-16T14:50:13.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.477+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.477+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2271795395}) 2024-09-16T14:50:13.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.477+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=15819251}) 2024-09-16T14:50:13.485 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.481+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.485 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.481+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2879004254}) 2024-09-16T14:50:13.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.481+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2729933256}) 2024-09-16T14:50:13.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.481+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.481+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2879004254}) 2024-09-16T14:50:13.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.481+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3440370370}) 2024-09-16T14:50:13.487 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.481+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.487 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.481+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2902818072}) 2024-09-16T14:50:13.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.481+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=408123434}) 2024-09-16T14:50:13.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.481+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.481+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2902818072}) 2024-09-16T14:50:13.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.481+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3099523556}) 2024-09-16T14:50:13.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.485+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.485+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2074994966}) 2024-09-16T14:50:13.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.485+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2970331043}) 2024-09-16T14:50:13.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.485+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.485+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2074994966}) 2024-09-16T14:50:13.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.485+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=626965493}) 2024-09-16T14:50:13.491 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.485+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.491 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.485+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2094748789}) 2024-09-16T14:50:13.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.485+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2033551911}) 2024-09-16T14:50:13.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.485+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.485+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2094748789}) 2024-09-16T14:50:13.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.485+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=88296759}) 2024-09-16T14:50:13.497 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.489+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.497 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.489+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646538761}) 2024-09-16T14:50:13.497 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.489+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1553352908}) 2024-09-16T14:50:13.497 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.489+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.489+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646538761}) 2024-09-16T14:50:13.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.489+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=155226069}) 2024-09-16T14:50:13.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.489+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.489+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3971648217}) 2024-09-16T14:50:13.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.489+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656053723}) 2024-09-16T14:50:13.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.489+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.489+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3971648217}) 2024-09-16T14:50:13.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.489+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2250139312}) 2024-09-16T14:50:13.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.493+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.493+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1451416139}) 2024-09-16T14:50:13.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.493+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2547561461}) 2024-09-16T14:50:13.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.493+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.493+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1451416139}) 2024-09-16T14:50:13.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.493+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=552473020}) 2024-09-16T14:50:13.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.493+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.493+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=873509934}) 2024-09-16T14:50:13.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.493+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138388560}) 2024-09-16T14:50:13.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.493+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.493+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=873509934}) 2024-09-16T14:50:13.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.493+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2513404879}) 2024-09-16T14:50:13.502 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.497+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.502 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.497+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=322702230}) 2024-09-16T14:50:13.502 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.497+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2160870149}) 2024-09-16T14:50:13.502 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.497+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.502 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.497+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=322702230}) 2024-09-16T14:50:13.502 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.497+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=486288750}) 2024-09-16T14:50:13.504 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.497+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.504 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.501+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1833304446}) 2024-09-16T14:50:13.504 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.501+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1504103718}) 2024-09-16T14:50:13.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.501+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.501+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1833304446}) 2024-09-16T14:50:13.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.501+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3748873151}) 2024-09-16T14:50:13.506 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.501+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.506 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.501+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2284554697}) 2024-09-16T14:50:13.506 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.501+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=330119881}) 2024-09-16T14:50:13.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.501+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.501+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2284554697}) 2024-09-16T14:50:13.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.501+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3610156196}) 2024-09-16T14:50:13.508 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.505+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.508 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.505+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2410302740}) 2024-09-16T14:50:13.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.505+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3634772889}) 2024-09-16T14:50:13.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.505+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.505+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2410302740}) 2024-09-16T14:50:13.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.505+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=5754414}) 2024-09-16T14:50:13.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.505+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.505+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969553807}) 2024-09-16T14:50:13.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.505+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144371856}) 2024-09-16T14:50:13.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.505+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.505+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969553807}) 2024-09-16T14:50:13.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.505+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1673333059}) 2024-09-16T14:50:13.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.509+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.509+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1179557480}) 2024-09-16T14:50:13.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.509+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4201861909}) 2024-09-16T14:50:13.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.509+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.509+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1179557480}) 2024-09-16T14:50:13.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.509+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1042805870}) 2024-09-16T14:50:13.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.509+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.509+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3417458715}) 2024-09-16T14:50:13.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.509+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565988547}) 2024-09-16T14:50:13.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.509+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.509+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3417458715}) 2024-09-16T14:50:13.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.509+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1019168488}) 2024-09-16T14:50:13.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.513+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.513+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4032441563}) 2024-09-16T14:50:13.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.513+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=132348558}) 2024-09-16T14:50:13.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.513+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.513+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4032441563}) 2024-09-16T14:50:13.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.513+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2478610019}) 2024-09-16T14:50:13.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.513+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.513+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4195301402}) 2024-09-16T14:50:13.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.513+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1926971637}) 2024-09-16T14:50:13.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.513+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.513+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4195301402}) 2024-09-16T14:50:13.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.513+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3948958397}) 2024-09-16T14:50:13.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.517+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.517+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2915916734}) 2024-09-16T14:50:13.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.517+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=688486527}) 2024-09-16T14:50:13.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.517+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.517+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2915916734}) 2024-09-16T14:50:13.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.517+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1353955383}) 2024-09-16T14:50:13.523 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.517+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.523 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.517+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=541261349}) 2024-09-16T14:50:13.523 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.517+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124050608}) 2024-09-16T14:50:13.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.517+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.517+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=541261349}) 2024-09-16T14:50:13.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.517+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4132002153}) 2024-09-16T14:50:13.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.521+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.521+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=792234232}) 2024-09-16T14:50:13.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.521+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440440209}) 2024-09-16T14:50:13.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.521+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.521+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=792234232}) 2024-09-16T14:50:13.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.521+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=451730642}) 2024-09-16T14:50:13.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.521+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.521+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1782748982}) 2024-09-16T14:50:13.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.521+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1238329175}) 2024-09-16T14:50:13.528 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.521+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.528 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.521+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1782748982}) 2024-09-16T14:50:13.528 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.521+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3606245927}) 2024-09-16T14:50:13.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.525+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.525+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2915707939}) 2024-09-16T14:50:13.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.525+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3915238503}) 2024-09-16T14:50:13.530 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.525+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.530 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.525+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2915707939}) 2024-09-16T14:50:13.530 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.525+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2988129716}) 2024-09-16T14:50:13.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.525+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.525+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4046267148}) 2024-09-16T14:50:13.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.525+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1369792189}) 2024-09-16T14:50:13.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.525+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.532 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.525+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4046267148}) 2024-09-16T14:50:13.532 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.525+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524570790}) 2024-09-16T14:50:13.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.529+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.529+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207859788}) 2024-09-16T14:50:13.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.529+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443487020}) 2024-09-16T14:50:13.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.529+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.529+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207859788}) 2024-09-16T14:50:13.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.529+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2496890154}) 2024-09-16T14:50:13.535 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.529+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.535 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.529+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3893828931}) 2024-09-16T14:50:13.535 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.529+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105584130}) 2024-09-16T14:50:13.535 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.529+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.529+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3893828931}) 2024-09-16T14:50:13.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.529+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3843165251}) 2024-09-16T14:50:13.537 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.533+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.533+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4109189187}) 2024-09-16T14:50:13.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.533+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944865659}) 2024-09-16T14:50:13.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.533+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.533+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4109189187}) 2024-09-16T14:50:13.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.533+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2304943617}) 2024-09-16T14:50:13.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.536+0000 7f913d045640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T14:50:13.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.536+0000 7f913d045640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T14:50:13.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.536+0000 7f913d045640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:13.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.536+0000 7f913d045640 1 -- 172.21.15.189:0/624524096 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) v1 -- 0x7f91180055a0 con 0x7f913809e2f0 2024-09-16T14:50:13.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.537+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.537+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=82,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=3664887404,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3150500418}) 2024-09-16T14:50:13.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.537+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=77,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=12} crc {logm=1902419125,mdsmap=1531151339,mgr=2578290774,mgr_command_descs=794974419,mgrstat=1929937092}) 2024-09-16T14:50:13.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.537+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.537+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=82,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=3664887404,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3150500418}) 2024-09-16T14:50:13.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.537+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=3869179707,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=1494681157}) 2024-09-16T14:50:13.544 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.541+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.544 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.541+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=5,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=73} crc {mgrstat=2793910138,mon_config_key=2003939582,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1631021852,osd_snap=3794734670,osdmap=371156774}) 2024-09-16T14:50:13.544 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.541+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=5,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=72} crc {mgrstat=2967885738,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1631021852,osd_snap=3794734670,osdmap=2265710684}) 2024-09-16T14:50:13.544 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.541+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.541+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=5,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=73} crc {mgrstat=2793910138,mon_config_key=2003939582,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1631021852,osd_snap=3794734670,osdmap=371156774}) 2024-09-16T14:50:13.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.541+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=6,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=73} crc {mgrstat=1842358302,mon_config_key=3334742516,monmap=2765269217,osd_metadata=3424249799,osd_pg_creating=1631021852,osd_snap=1109659271,osdmap=1955803990}) 2024-09-16T14:50:13.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.544+0000 7f911f7fe640 1 -- 172.21.15.189:0/624524096 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h7"}]=0 device 'h7' does not appear in the crush map v222) v1 ==== 118+0+0 (unknown 3324453362 0 0) 0x7f912c014530 con 0x7f913809e2f0 2024-09-16T14:50:13.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.544+0000 7f911f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T14:50:13.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.544+0000 7f911f7fe640 10 monclient: _finish_command 3 = system:0 device 'h7' does not appear in the crush map 2024-09-16T14:50:13.545 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h7' does not appear in the crush map 2024-09-16T14:50:13.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.544+0000 7f913d045640 1 -- 172.21.15.189:0/624524096 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9124036340 legacy=0x7f9124038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.544+0000 7f913d045640 10 monclient: shutdown 2024-09-16T14:50:13.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.544+0000 7f913d045640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:13.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.544+0000 7f913d045640 1 -- 172.21.15.189:0/624524096 >> v1:172.21.15.189:6789/0 conn(0x7f913809e2f0 legacy=0x7f913813a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.544+0000 7f913d045640 1 -- 172.21.15.189:0/624524096 shutdown_connections 2024-09-16T14:50:13.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.544+0000 7f913d045640 1 -- 172.21.15.189:0/624524096 >> 172.21.15.189:0/624524096 conn(0x7f91380a3320 msgr2=0x7f91380a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:13.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.544+0000 7f913d045640 1 -- 172.21.15.189:0/624524096 shutdown_connections 2024-09-16T14:50:13.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.544+0000 7f913d045640 1 -- 172.21.15.189:0/624524096 wait complete. 2024-09-16T14:50:13.556 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.553+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.557 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.553+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=25} crc {osdmap=1443316657}) 2024-09-16T14:50:13.557 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.553+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=24} crc {osdmap=3595463421}) 2024-09-16T14:50:13.557 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.553+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:13.557 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.553+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=25} crc {osdmap=1443316657}) 2024-09-16T14:50:13.557 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:13.553+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=23} crc {osdmap=46532602}) 2024-09-16T14:50:13.561 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:13.562 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h8 2024-09-16T14:50:13.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74cc80f640 10 monclient: get_monmap_and_config 2024-09-16T14:50:13.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74cc80f640 10 monclient: build_initial_monmap 2024-09-16T14:50:13.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74cc80f640 10 monclient: monmap: 2024-09-16T14:50:13.667 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:13.667 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.667 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:13.667842+0000 2024-09-16T14:50:13.667 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:13.667842+0000 2024-09-16T14:50:13.667 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:13.667 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:13.667 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:13.667 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:13.667 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:13.667 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:13.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74cc80f640 1 Processor -- start 2024-09-16T14:50:13.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74cc80f640 1 -- start start 2024-09-16T14:50:13.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74cc80f640 10 monclient: init 2024-09-16T14:50:13.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74cc80f640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:13.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74cc80f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:13.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74cc80f640 10 monclient(hunting): picked mon.noname-c con 0x7f74c81315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:13.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74cc80f640 10 monclient(hunting): picked mon.noname-a con 0x7f74c809b320 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:13.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74cc80f640 10 monclient(hunting): picked mon.noname-b con 0x7f74c8132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:13.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74cc80f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c809edd0 con 0x7f74c809b320 2024-09-16T14:50:13.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74cc80f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c809bc00 con 0x7f74c81315a0 2024-09-16T14:50:13.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74cc80f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c809e900 con 0x7f74c8132090 2024-09-16T14:50:13.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74cc80f640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:13.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74cc80f640 10 monclient(hunting): authenticate will time out at 1395.783463s 2024-09-16T14:50:13.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74c659c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f74c81315a0 0x7f74c8131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51806/0 (socket says 172.21.15.189:51806) 2024-09-16T14:50:13.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.664+0000 7f74c659c640 1 -- 172.21.15.189:0/3550623152 learned_addr learned my addr 172.21.15.189:0/3550623152 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:13.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 1 -- 172.21.15.189:0/3550623152 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096611391 0 0) 0x7f74c809bc00 con 0x7f74c81315a0 2024-09-16T14:50:13.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient(hunting): my global_id is 174136 2024-09-16T14:50:13.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 1 -- 172.21.15.189:0/3550623152 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b0003530 con 0x7f74c81315a0 2024-09-16T14:50:13.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 1 -- 172.21.15.189:0/3550623152 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945687144 0 0) 0x7f74c809e900 con 0x7f74c8132090 2024-09-16T14:50:13.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient(hunting): my global_id is 154601 2024-09-16T14:50:13.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 1 -- 172.21.15.189:0/3550623152 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c809bc00 con 0x7f74c8132090 2024-09-16T14:50:13.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 1 -- 172.21.15.189:0/3550623152 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 25263639 0 0) 0x7f74b0003530 con 0x7f74c81315a0 2024-09-16T14:50:13.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 1 -- 172.21.15.189:0/3550623152 >> v1:172.21.15.26:6790/0 conn(0x7f74c8132090 legacy=0x7f74c813a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 1 -- 172.21.15.189:0/3550623152 >> v1:172.21.15.26:6789/0 conn(0x7f74c809b320 legacy=0x7f74c8136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient: _finish_hunting 0 2024-09-16T14:50:13.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 1 monclient: found mon.noname-c 2024-09-16T14:50:13.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:13.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:50:13.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 1 -- 172.21.15.189:0/3550623152 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c809b920 con 0x7f74c81315a0 2024-09-16T14:50:13.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient: _finish_auth 0 2024-09-16T14:50:13.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient: _check_auth_tickets 2024-09-16T14:50:13.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:13.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 1 -- 172.21.15.189:0/3550623152 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74b8001620 con 0x7f74c81315a0 2024-09-16T14:50:13.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:13.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:13.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient: dump: 2024-09-16T14:50:13.678 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:13.678 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.678 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.678 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.678 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:13.678 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:13.678 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:13.678 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:13.678 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:13.679 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:13.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 5 monclient: authenticate success, global_id 174136 2024-09-16T14:50:13.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:13.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 1 -- 172.21.15.189:0/3550623152 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74b8001880 con 0x7f74c81315a0 2024-09-16T14:50:13.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:13.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:13.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 4 set_mon_vals no callback set 2024-09-16T14:50:13.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 1 -- 172.21.15.189:0/3550623152 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74b8001c50 con 0x7f74c81315a0 2024-09-16T14:50:13.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:13.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:13.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c559a640 10 monclient: dump: 2024-09-16T14:50:13.680 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:13.680 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.680 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.680 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.680 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:13.680 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:13.681 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:13.681 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:13.681 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:13.681 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:13.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 10 monclient: shutdown 2024-09-16T14:50:13.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:13.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 1 -- 172.21.15.189:0/3550623152 >> v1:172.21.15.189:6789/0 conn(0x7f74c81315a0 legacy=0x7f74c8131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 1 -- 172.21.15.189:0/3550623152 shutdown_connections 2024-09-16T14:50:13.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 1 -- 172.21.15.189:0/3550623152 >> 172.21.15.189:0/3550623152 conn(0x7f74c80a3320 msgr2=0x7f74c809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:13.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 1 -- 172.21.15.189:0/3550623152 shutdown_connections 2024-09-16T14:50:13.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 1 -- 172.21.15.189:0/3550623152 wait complete. 2024-09-16T14:50:13.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 10 monclient: build_initial_monmap 2024-09-16T14:50:13.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 10 monclient: monmap: 2024-09-16T14:50:13.682 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:13.682 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:13.682 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:13.683 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:13.683 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:13.683 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:13.683 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:13.683 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:13.683 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:13.683 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:13.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 1 Processor -- start 2024-09-16T14:50:13.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 1 -- start start 2024-09-16T14:50:13.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 10 monclient: init 2024-09-16T14:50:13.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:13.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:13.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 10 monclient(hunting): picked mon.noname-b con 0x7f74c81315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:13.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 10 monclient(hunting): picked mon.noname-a con 0x7f74c8132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:13.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 10 monclient(hunting): picked mon.noname-c con 0x7f74c8126d90 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:13.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c809cd20 con 0x7f74c8132090 2024-09-16T14:50:13.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c809a7b0 con 0x7f74c81315a0 2024-09-16T14:50:13.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c80a2970 con 0x7f74c8126d90 2024-09-16T14:50:13.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:13.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74cc80f640 10 monclient(hunting): authenticate will time out at 1395.787115s 2024-09-16T14:50:13.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c659c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f74c81315a0 0x7f74c813e950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51814/0 (socket says 172.21.15.189:51814) 2024-09-16T14:50:13.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74c659c640 1 -- 172.21.15.189:0/963030229 learned_addr learned my addr 172.21.15.189:0/963030229 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:13.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527854583 0 0) 0x7f74c809a7b0 con 0x7f74c81315a0 2024-09-16T14:50:13.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 10 monclient(hunting): my global_id is 174139 2024-09-16T14:50:13.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74a8003530 con 0x7f74c81315a0 2024-09-16T14:50:13.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546861384 0 0) 0x7f74c80a2970 con 0x7f74c8126d90 2024-09-16T14:50:13.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 10 monclient(hunting): my global_id is 154604 2024-09-16T14:50:13.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c809a7b0 con 0x7f74c8126d90 2024-09-16T14:50:13.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 338517055 0 0) 0x7f74a8003530 con 0x7f74c81315a0 2024-09-16T14:50:13.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74c80a2970 con 0x7f74c81315a0 2024-09-16T14:50:13.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74b8001b90 con 0x7f74c81315a0 2024-09-16T14:50:13.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:13.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:13.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:13.686 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:13.687 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.687 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.687 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.687 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:13.687 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:13.687 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:13.687 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:13.687 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:13.687 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:13.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:13.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305900982 0 0) 0x7f74c809cd20 con 0x7f74c8132090 2024-09-16T14:50:13.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:13.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:13.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:13.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 10 monclient(hunting): my global_id is 164352 2024-09-16T14:50:13.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74a8003530 con 0x7f74c8132090 2024-09-16T14:50:13.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.668+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 539950092 0 0) 0x7f74c80a2970 con 0x7f74c81315a0 2024-09-16T14:50:13.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 >> v1:172.21.15.26:6790/0 conn(0x7f74c8126d90 legacy=0x7f74c81d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 >> v1:172.21.15.26:6789/0 conn(0x7f74c8132090 legacy=0x7f74c813f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:13.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:13.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 1 monclient: found mon.b 2024-09-16T14:50:13.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:13.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:13.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c812f0b0 con 0x7f74c81315a0 2024-09-16T14:50:13.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:13.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:13.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:13.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74cc80f640 5 monclient: authenticate success, global_id 174139 2024-09-16T14:50:13.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74cc80f640 10 monclient: _renew_subs 2024-09-16T14:50:13.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74cc80f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:13.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74cc80f640 1 -- 172.21.15.189:0/963030229 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c812f390 con 0x7f74c81315a0 2024-09-16T14:50:13.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74cc80f640 10 monclient: _renew_subs 2024-09-16T14:50:13.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74cc80f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:13.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74cc80f640 1 -- 172.21.15.189:0/963030229 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c812e6e0 con 0x7f74c81315a0 2024-09-16T14:50:13.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74b80011d0 con 0x7f74c81315a0 2024-09-16T14:50:13.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:13.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74b8004da0 con 0x7f74c81315a0 2024-09-16T14:50:13.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:13.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74c4d99640 4 set_mon_vals no callback set 2024-09-16T14:50:13.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:13.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 10 monclient: dump: 2024-09-16T14:50:13.690 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:13.690 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:13.690 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.690 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:13.690 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:13.691 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:13.691 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:13.691 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:13.691 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:13.691 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:13.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74cc80f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:13.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74cc80f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:13.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74cc80f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:13.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74cc80f640 1 -- 172.21.15.189:0/963030229 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74ac005050 con 0x7f74c81315a0 2024-09-16T14:50:13.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f74b800f890 con 0x7f74c81315a0 2024-09-16T14:50:13.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(222..222 src has 169..222) v4 ==== 12602+0+0 (unknown 3591673153 0 0) 0x7f74b8045740 con 0x7f74c81315a0 2024-09-16T14:50:13.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.672+0000 7f74c5d9b640 10 monclient: get_auth_request con 0x7f74a8036340 auth_method 0 2024-09-16T14:50:13.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.676+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f74b8010e80 con 0x7f74c81315a0 2024-09-16T14:50:13.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.676+0000 7f74b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:13.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.676+0000 7f74b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:13.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.824+0000 7f74cc80f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T14:50:13.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.824+0000 7f74cc80f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T14:50:13.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.824+0000 7f74cc80f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:13.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:13.824+0000 7f74cc80f640 1 -- 172.21.15.189:0/963030229 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h8"} v 0) v1 -- 0x7f74ac005300 con 0x7f74c81315a0 2024-09-16T14:50:14.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.444+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h8"}]=0 removed item id -40 name 'h8' from crush map v223) v1 ==== 118+0+0 (unknown 1727027295 0 0) 0x7f74b8014640 con 0x7f74c81315a0 2024-09-16T14:50:14.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.444+0000 7f74b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T14:50:14.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.444+0000 7f74b6ffd640 10 monclient: _finish_command 2 = system:0 removed item id -40 name 'h8' from crush map 2024-09-16T14:50:14.479 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:50:14.479 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:50:14.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.544+0000 7f74cc80f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T14:50:14.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.544+0000 7f74cc80f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T14:50:14.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.544+0000 7f74cc80f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:14.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.544+0000 7f74cc80f640 1 -- 172.21.15.189:0/963030229 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h8"} v 0) v1 -- 0x7f74ac0055a0 con 0x7f74c81315a0 2024-09-16T14:50:14.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.548+0000 7f74b6ffd640 1 -- 172.21.15.189:0/963030229 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h8"}]=0 device 'h8' does not appear in the crush map v223) v1 ==== 118+0+0 (unknown 362628605 0 0) 0x7f74b8014820 con 0x7f74c81315a0 2024-09-16T14:50:14.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.548+0000 7f74b6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T14:50:14.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.548+0000 7f74b6ffd640 10 monclient: _finish_command 3 = system:0 device 'h8' does not appear in the crush map 2024-09-16T14:50:14.550 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h8' does not appear in the crush map 2024-09-16T14:50:14.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.548+0000 7f74cc80f640 1 -- 172.21.15.189:0/963030229 >> v1:172.21.15.189:6812/3196161582 conn(0x7f74a8036340 legacy=0x7f74a8038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:14.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.548+0000 7f74cc80f640 10 monclient: shutdown 2024-09-16T14:50:14.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.548+0000 7f74cc80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:14.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.548+0000 7f74cc80f640 1 -- 172.21.15.189:0/963030229 >> v1:172.21.15.189:6789/0 conn(0x7f74c81315a0 legacy=0x7f74c813e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:14.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.548+0000 7f74cc80f640 1 -- 172.21.15.189:0/963030229 shutdown_connections 2024-09-16T14:50:14.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.548+0000 7f74cc80f640 1 -- 172.21.15.189:0/963030229 >> 172.21.15.189:0/963030229 conn(0x7f74c80a3320 msgr2=0x7f74c809d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:14.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.548+0000 7f74cc80f640 1 -- 172.21.15.189:0/963030229 shutdown_connections 2024-09-16T14:50:14.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.548+0000 7f74cc80f640 1 -- 172.21.15.189:0/963030229 wait complete. 2024-09-16T14:50:14.567 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:14.567 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h9 2024-09-16T14:50:14.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 10 monclient: get_monmap_and_config 2024-09-16T14:50:14.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 10 monclient: build_initial_monmap 2024-09-16T14:50:14.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 10 monclient: monmap: 2024-09-16T14:50:14.613 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:14.613 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:14.613 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:14.614485+0000 2024-09-16T14:50:14.613 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:14.614485+0000 2024-09-16T14:50:14.614 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:14.614 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:14.614 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:14.614 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:14.614 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:14.614 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:14.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 1 Processor -- start 2024-09-16T14:50:14.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 1 -- start start 2024-09-16T14:50:14.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 10 monclient: init 2024-09-16T14:50:14.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:14.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:50:14.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 10 monclient(hunting): picked mon.noname-a con 0x7fe9f806c5a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:14.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 10 monclient(hunting): picked mon.noname-c con 0x7fe9f80a23b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:14.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 10 monclient(hunting): picked mon.noname-b con 0x7fe9f80a27c0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:14.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f806cee0 con 0x7fe9f806c5a0 2024-09-16T14:50:14.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f806c980 con 0x7fe9f80a23b0 2024-09-16T14:50:14.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f80a15e0 con 0x7fe9f80a27c0 2024-09-16T14:50:14.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:14.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 10 monclient(hunting): authenticate will time out at 1438.734493s 2024-09-16T14:50:14.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f77fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe9f806c5a0 0x7fe9f80a1ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48828/0 (socket says 172.21.15.26:48828) 2024-09-16T14:50:14.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f77fe640 1 -- 172.21.15.26:0/2440844978 learned_addr learned my addr 172.21.15.26:0/2440844978 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:14.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 1 -- 172.21.15.26:0/2440844978 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089901517 0 0) 0x7fe9f806cee0 con 0x7fe9f806c5a0 2024-09-16T14:50:14.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:14.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient(hunting): my global_id is 164355 2024-09-16T14:50:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 1 -- 172.21.15.26:0/2440844978 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e0003530 con 0x7fe9f806c5a0 2024-09-16T14:50:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 1 -- 172.21.15.26:0/2440844978 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559850532 0 0) 0x7fe9f80a15e0 con 0x7fe9f80a27c0 2024-09-16T14:50:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient(hunting): my global_id is 154607 2024-09-16T14:50:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 1 -- 172.21.15.26:0/2440844978 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9f806cee0 con 0x7fe9f80a27c0 2024-09-16T14:50:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 1 -- 172.21.15.26:0/2440844978 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2442267328 0 0) 0x7fe9e0003530 con 0x7fe9f806c5a0 2024-09-16T14:50:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 1 -- 172.21.15.26:0/2440844978 >> v1:172.21.15.26:6790/0 conn(0x7fe9f80a27c0 legacy=0x7fe9f813d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 1 -- 172.21.15.26:0/2440844978 >> v1:172.21.15.189:6789/0 conn(0x7fe9f80a23b0 legacy=0x7fe9f8139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:50:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 1 monclient: found mon.noname-a 2024-09-16T14:50:14.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:14.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:14.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 1 -- 172.21.15.26:0/2440844978 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f809edd0 con 0x7fe9f806c5a0 2024-09-16T14:50:14.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient: _finish_auth 0 2024-09-16T14:50:14.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:14.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:14.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 1 -- 172.21.15.26:0/2440844978 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe9e8001620 con 0x7fe9f806c5a0 2024-09-16T14:50:14.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 5 monclient: authenticate success, global_id 164355 2024-09-16T14:50:14.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:14.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:14.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:14.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient: dump: 2024-09-16T14:50:14.620 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:14.620 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:14.621 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.621 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.621 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:14.621 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:14.621 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:14.621 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:14.621 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:14.621 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:14.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:14.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 1 -- 172.21.15.26:0/2440844978 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9e8001840 con 0x7fe9f806c5a0 2024-09-16T14:50:14.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:14.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 1 -- 172.21.15.26:0/2440844978 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe9e8001bd0 con 0x7fe9f806c5a0 2024-09-16T14:50:14.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:14.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9fe099640 4 set_mon_vals no callback set 2024-09-16T14:50:14.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:14.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:14.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.609+0000 7fe9f67fc640 10 monclient: dump: 2024-09-16T14:50:14.622 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:14.623 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:14.623 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.623 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.623 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:14.623 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:14.623 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:14.623 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:14.623 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:14.623 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:14.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient: shutdown 2024-09-16T14:50:14.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 1 -- 172.21.15.26:0/2440844978 >> v1:172.21.15.26:6789/0 conn(0x7fe9f806c5a0 legacy=0x7fe9f80a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 1 -- 172.21.15.26:0/2440844978 shutdown_connections 2024-09-16T14:50:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 1 -- 172.21.15.26:0/2440844978 >> 172.21.15.26:0/2440844978 conn(0x7fe9f80a3da0 msgr2=0x7fe9f80a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 1 -- 172.21.15.26:0/2440844978 shutdown_connections 2024-09-16T14:50:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 1 -- 172.21.15.26:0/2440844978 wait complete. 2024-09-16T14:50:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient: build_initial_monmap 2024-09-16T14:50:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient: monmap: 2024-09-16T14:50:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:14.624 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:14.625 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:14.625 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:14.625 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:14.625 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:14.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 1 Processor -- start 2024-09-16T14:50:14.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 1 -- start start 2024-09-16T14:50:14.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient: init 2024-09-16T14:50:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient(hunting): picked mon.noname-a con 0x7fe9f806c5a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient(hunting): picked mon.noname-c con 0x7fe9f80a23b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient(hunting): picked mon.noname-b con 0x7fe9f80a27c0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f8132ee0 con 0x7fe9f806c5a0 2024-09-16T14:50:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f8134270 con 0x7fe9f80a27c0 2024-09-16T14:50:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f8134930 con 0x7fe9f80a23b0 2024-09-16T14:50:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient(hunting): authenticate will time out at 1438.737742s 2024-09-16T14:50:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9f77fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe9f806c5a0 0x7fe9f81320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48842/0 (socket says 172.21.15.26:48842) 2024-09-16T14:50:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9f77fe640 1 -- 172.21.15.26:0/3261186160 learned_addr learned my addr 172.21.15.26:0/3261186160 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3057643667 0 0) 0x7fe9f8132ee0 con 0x7fe9f806c5a0 2024-09-16T14:50:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): my global_id is 164358 2024-09-16T14:50:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e4003530 con 0x7fe9f806c5a0 2024-09-16T14:50:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010866487 0 0) 0x7fe9f8134930 con 0x7fe9f80a23b0 2024-09-16T14:50:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): my global_id is 154610 2024-09-16T14:50:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9f8132ee0 con 0x7fe9f80a23b0 2024-09-16T14:50:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667371695 0 0) 0x7fe9f8134270 con 0x7fe9f80a27c0 2024-09-16T14:50:14.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:14.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:14.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:14.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): my global_id is 174145 2024-09-16T14:50:14.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9f8134930 con 0x7fe9f80a27c0 2024-09-16T14:50:14.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3285463702 0 0) 0x7fe9e4003530 con 0x7fe9f806c5a0 2024-09-16T14:50:14.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9f8134270 con 0x7fe9f806c5a0 2024-09-16T14:50:14.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe9e8001b50 con 0x7fe9f806c5a0 2024-09-16T14:50:14.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:14.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:14.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): dump: 2024-09-16T14:50:14.631 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:14.631 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:14.632 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.632 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.632 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:14.632 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:14.632 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:14.632 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:14.632 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:14.632 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:14.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:14.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1478173594 0 0) 0x7fe9f8132ee0 con 0x7fe9f80a23b0 2024-09-16T14:50:14.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9e4003530 con 0x7fe9f80a23b0 2024-09-16T14:50:14.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe9ec0017e0 con 0x7fe9f80a23b0 2024-09-16T14:50:14.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:14.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:14.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient(hunting): dump: 2024-09-16T14:50:14.633 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:14.633 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:14.633 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.633 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.633 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:14.633 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:14.633 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:14.634 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:14.634 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:14.634 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:14.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 553299375 0 0) 0x7fe9f8134270 con 0x7fe9f806c5a0 2024-09-16T14:50:14.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 >> v1:172.21.15.26:6790/0 conn(0x7fe9f80a23b0 legacy=0x7fe9f81327d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:14.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 >> v1:172.21.15.189:6789/0 conn(0x7fe9f80a27c0 legacy=0x7fe9f81d1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:14.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient: _finish_hunting 0 2024-09-16T14:50:14.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 monclient: found mon.a 2024-09-16T14:50:14.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:14.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:14.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f81362d0 con 0x7fe9f806c5a0 2024-09-16T14:50:14.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient: _finish_auth 0 2024-09-16T14:50:14.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient: _check_auth_tickets 2024-09-16T14:50:14.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:14.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 5 monclient: authenticate success, global_id 164358 2024-09-16T14:50:14.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient: _renew_subs 2024-09-16T14:50:14.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:14.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 1 -- 172.21.15.26:0/3261186160 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f81346b0 con 0x7fe9f806c5a0 2024-09-16T14:50:14.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient: _renew_subs 2024-09-16T14:50:14.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:14.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 1 -- 172.21.15.26:0/3261186160 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f8134b50 con 0x7fe9f806c5a0 2024-09-16T14:50:14.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9e8004aa0 con 0x7fe9f806c5a0 2024-09-16T14:50:14.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:14.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe9e8004db0 con 0x7fe9f806c5a0 2024-09-16T14:50:14.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:14.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:14.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9f5ffb640 4 set_mon_vals no callback set 2024-09-16T14:50:14.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 10 monclient: dump: 2024-09-16T14:50:14.636 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:14.636 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:14.636 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.636 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.636 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:14.636 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:14.636 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:14.636 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:14.636 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:14.636 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:14.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe9e800f820 con 0x7fe9f806c5a0 2024-09-16T14:50:14.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(223..223 src has 169..223) v4 ==== 12392+0+0 (unknown 467405984 0 0) 0x7fe9e80454a0 con 0x7fe9f806c5a0 2024-09-16T14:50:14.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:14.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:14.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:14.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.613+0000 7fe9fe099640 1 -- 172.21.15.26:0/3261186160 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d8005050 con 0x7fe9f806c5a0 2024-09-16T14:50:14.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.617+0000 7fe9f6ffd640 10 monclient: get_auth_request con 0x7fe9e4036470 auth_method 0 2024-09-16T14:50:14.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.621+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe9e80111f0 con 0x7fe9f806c5a0 2024-09-16T14:50:14.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.621+0000 7fe9dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:14.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.621+0000 7fe9dffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f542520f640 10 monclient: get_monmap_and_config 2024-09-16T14:50:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f542520f640 10 monclient: build_initial_monmap 2024-09-16T14:50:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f542520f640 10 monclient: monmap: 2024-09-16T14:50:14.682 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:14.683733+0000 2024-09-16T14:50:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:14.683733+0000 2024-09-16T14:50:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:14.683 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:14.683 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:14.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f542520f640 1 Processor -- start 2024-09-16T14:50:14.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f542520f640 1 -- start start 2024-09-16T14:50:14.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f542520f640 10 monclient: init 2024-09-16T14:50:14.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f542520f640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:14.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f542520f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:14.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f542520f640 10 monclient(hunting): picked mon.noname-c con 0x7f54201315a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:14.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f542520f640 10 monclient(hunting): picked mon.noname-b con 0x7f542009b320 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:14.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f542520f640 10 monclient(hunting): picked mon.noname-a con 0x7f5420132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:14.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f542520f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542009edd0 con 0x7f5420132090 2024-09-16T14:50:14.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f542520f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542009bc00 con 0x7f54201315a0 2024-09-16T14:50:14.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f542520f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542009e900 con 0x7f542009b320 2024-09-16T14:50:14.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f542520f640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:14.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f542520f640 10 monclient(hunting): authenticate will time out at 1396.799411s 2024-09-16T14:50:14.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f541ed9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f54201315a0 0x7f5420131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51820/0 (socket says 172.21.15.189:51820) 2024-09-16T14:50:14.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f541ed9d640 1 -- 172.21.15.189:0/2446239522 learned_addr learned my addr 172.21.15.189:0/2446239522 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:14.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f541dd9b640 1 -- 172.21.15.189:0/2446239522 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437488601 0 0) 0x7f542009bc00 con 0x7f54201315a0 2024-09-16T14:50:14.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f541dd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:50:14.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f541dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:14.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f541dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:14.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f541dd9b640 10 monclient(hunting): my global_id is 174148 2024-09-16T14:50:14.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.680+0000 7f541dd9b640 1 -- 172.21.15.189:0/2446239522 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f540c003530 con 0x7f54201315a0 2024-09-16T14:50:14.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 1 -- 172.21.15.189:0/2446239522 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3729862250 0 0) 0x7f540c003530 con 0x7f54201315a0 2024-09-16T14:50:14.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 1 -- 172.21.15.189:0/2446239522 >> v1:172.21.15.26:6790/0 conn(0x7f542009b320 legacy=0x7f5420136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:14.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 1 -- 172.21.15.189:0/2446239522 >> v1:172.21.15.26:6789/0 conn(0x7f5420132090 legacy=0x7f542013a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:14.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 10 monclient: _finish_hunting 0 2024-09-16T14:50:14.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 1 monclient: found mon.noname-c 2024-09-16T14:50:14.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:14.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:50:14.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 1 -- 172.21.15.189:0/2446239522 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542009b920 con 0x7f54201315a0 2024-09-16T14:50:14.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 10 monclient: _finish_auth 0 2024-09-16T14:50:14.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 10 monclient: _check_auth_tickets 2024-09-16T14:50:14.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:14.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 5 monclient: authenticate success, global_id 174148 2024-09-16T14:50:14.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:14.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 1 -- 172.21.15.189:0/2446239522 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f54100019a0 con 0x7f54201315a0 2024-09-16T14:50:14.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:14.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:14.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 10 monclient: dump: 2024-09-16T14:50:14.693 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:14.693 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:14.693 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.693 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.693 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:14.693 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:14.693 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:14.693 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:14.694 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:14.694 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:14.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:14.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 1 -- 172.21.15.189:0/2446239522 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5410001cc0 con 0x7f54201315a0 2024-09-16T14:50:14.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:14.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 1 -- 172.21.15.189:0/2446239522 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5410002050 con 0x7f54201315a0 2024-09-16T14:50:14.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:14.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:14.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541dd9b640 10 monclient: dump: 2024-09-16T14:50:14.694 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:14.694 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:14.694 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.695 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.695 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:14.695 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:14.695 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:14.695 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:14.695 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:14.695 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:14.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:14.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 4 set_mon_vals no callback set 2024-09-16T14:50:14.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 10 monclient: shutdown 2024-09-16T14:50:14.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:14.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 1 -- 172.21.15.189:0/2446239522 >> v1:172.21.15.189:6789/0 conn(0x7f54201315a0 legacy=0x7f5420131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:14.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 1 -- 172.21.15.189:0/2446239522 shutdown_connections 2024-09-16T14:50:14.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 1 -- 172.21.15.189:0/2446239522 >> 172.21.15.189:0/2446239522 conn(0x7f54200a3320 msgr2=0x7f542009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:14.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 1 -- 172.21.15.189:0/2446239522 shutdown_connections 2024-09-16T14:50:14.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 1 -- 172.21.15.189:0/2446239522 wait complete. 2024-09-16T14:50:14.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 10 monclient: build_initial_monmap 2024-09-16T14:50:14.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 10 monclient: monmap: 2024-09-16T14:50:14.696 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:14.696 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:14.696 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:14.697 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:14.697 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:14.697 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:14.697 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:14.697 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:14.697 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:14.697 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:14.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 1 Processor -- start 2024-09-16T14:50:14.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 1 -- start start 2024-09-16T14:50:14.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 10 monclient: init 2024-09-16T14:50:14.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:14.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:14.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 10 monclient(hunting): picked mon.noname-a con 0x7f54201315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:14.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 10 monclient(hunting): picked mon.noname-c con 0x7f5420132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:14.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 10 monclient(hunting): picked mon.noname-b con 0x7f542013c740 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:14.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542009cd20 con 0x7f54201315a0 2024-09-16T14:50:14.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542006b380 con 0x7f542013c740 2024-09-16T14:50:14.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542009a830 con 0x7f5420132090 2024-09-16T14:50:14.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:14.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 10 monclient(hunting): authenticate will time out at 1396.802573s 2024-09-16T14:50:14.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541f59e640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f542013c740 0x7f54201d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51826/0 (socket says 172.21.15.189:51826) 2024-09-16T14:50:14.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541f59e640 1 -- 172.21.15.189:0/1831705282 learned_addr learned my addr 172.21.15.189:0/1831705282 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:14.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998480545 0 0) 0x7f542006b380 con 0x7f542013c740 2024-09-16T14:50:14.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:14.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:14.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:14.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient(hunting): my global_id is 174151 2024-09-16T14:50:14.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5400003530 con 0x7f542013c740 2024-09-16T14:50:14.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1251521420 0 0) 0x7f5400003530 con 0x7f542013c740 2024-09-16T14:50:14.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f542006b380 con 0x7f542013c740 2024-09-16T14:50:14.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5418001900 con 0x7f542013c740 2024-09-16T14:50:14.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:14.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:14.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient(hunting): dump: 2024-09-16T14:50:14.700 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:14.700 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:14.700 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.700 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.700 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:14.700 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:14.700 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:14.700 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:14.700 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:14.700 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:14.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955908789 0 0) 0x7f5408001240 con 0x7f5420132090 2024-09-16T14:50:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient(hunting): my global_id is 154616 2024-09-16T14:50:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5400003530 con 0x7f5420132090 2024-09-16T14:50:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615654912 0 0) 0x7f542009cd20 con 0x7f54201315a0 2024-09-16T14:50:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient(hunting): my global_id is 164364 2024-09-16T14:50:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5408001240 con 0x7f54201315a0 2024-09-16T14:50:14.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1330432211 0 0) 0x7f542006b380 con 0x7f542013c740 2024-09-16T14:50:14.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 >> v1:172.21.15.26:6790/0 conn(0x7f5420132090 legacy=0x7f542013c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:14.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 >> v1:172.21.15.26:6789/0 conn(0x7f54201315a0 legacy=0x7f542013b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:14.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:14.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 1 monclient: found mon.b 2024-09-16T14:50:14.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:14.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:14.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54200a2900 con 0x7f542013c740 2024-09-16T14:50:14.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:14.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:14.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:14.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 5 monclient: authenticate success, global_id 174151 2024-09-16T14:50:14.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 10 monclient: _renew_subs 2024-09-16T14:50:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5420134570 con 0x7f542013c740 2024-09-16T14:50:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 10 monclient: _renew_subs 2024-09-16T14:50:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f542520f640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5420133390 con 0x7f542013c740 2024-09-16T14:50:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5418001cd0 con 0x7f542013c740 2024-09-16T14:50:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5418003af0 con 0x7f542013c740 2024-09-16T14:50:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f541d59a640 4 set_mon_vals no callback set 2024-09-16T14:50:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.684+0000 7f54177fe640 10 monclient: dump: 2024-09-16T14:50:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:14.704 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.704 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:14.704 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:14.704 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:14.704 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:14.704 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:14.704 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:14.704 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:14.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.688+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5418003c50 con 0x7f542013c740 2024-09-16T14:50:14.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.688+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(223..223 src has 169..223) v4 ==== 12392+0+0 (unknown 467405984 0 0) 0x7f5418045410 con 0x7f542013c740 2024-09-16T14:50:14.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.688+0000 7f542520f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:14.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.688+0000 7f542520f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:14.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.688+0000 7f542520f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:14.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.688+0000 7f542520f640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5404005050 con 0x7f542013c740 2024-09-16T14:50:14.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.688+0000 7f541ed9d640 10 monclient: get_auth_request con 0x7f5400036370 auth_method 0 2024-09-16T14:50:14.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.692+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f54180043d0 con 0x7f542013c740 2024-09-16T14:50:14.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.692+0000 7f54177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:14.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.692+0000 7f54177fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:14.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.841+0000 7fe9fe099640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:50:14.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.841+0000 7fe9fe099640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:14.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.841+0000 7fe9fe099640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:14.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.841+0000 7fe9fe099640 1 -- 172.21.15.26:0/3261186160 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe9d8005880 con 0x7fe9f806c5a0 2024-09-16T14:50:14.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.841+0000 7fe9dffff640 1 -- 172.21.15.26:0/3261186160 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2765406995) 0x7fe9e8012d00 con 0x7fe9f806c5a0 2024-09-16T14:50:14.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.841+0000 7fe9dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:14.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.841+0000 7fe9dffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:14.847 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:50:14.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.848+0000 7f542520f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T14:50:14.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.848+0000 7f542520f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T14:50:14.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.848+0000 7f542520f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:14.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:14.848+0000 7f542520f640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) v1 -- 0x7f5404005300 con 0x7f542013c740 2024-09-16T14:50:14.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.845+0000 7fe9fe099640 1 -- 172.21.15.26:0/3261186160 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe9e4036470 legacy=0x7fe9e4038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:14.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.845+0000 7fe9fe099640 10 monclient: shutdown 2024-09-16T14:50:14.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.845+0000 7fe9fe099640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:14.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.845+0000 7fe9fe099640 1 -- 172.21.15.26:0/3261186160 >> v1:172.21.15.26:6789/0 conn(0x7fe9f806c5a0 legacy=0x7fe9f81320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:14.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.845+0000 7fe9fe099640 1 -- 172.21.15.26:0/3261186160 shutdown_connections 2024-09-16T14:50:14.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.845+0000 7fe9fe099640 1 -- 172.21.15.26:0/3261186160 >> 172.21.15.26:0/3261186160 conn(0x7fe9f80a3da0 msgr2=0x7fe9f80a51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:14.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.845+0000 7fe9fe099640 1 -- 172.21.15.26:0/3261186160 shutdown_connections 2024-09-16T14:50:14.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.845+0000 7fe9fe099640 1 -- 172.21.15.26:0/3261186160 wait complete. 2024-09-16T14:50:14.865 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:50:14.866 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:50:14.866 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:50:14.866 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:50:14.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 10 monclient: get_monmap_and_config 2024-09-16T14:50:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 10 monclient: build_initial_monmap 2024-09-16T14:50:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 10 monclient: monmap: 2024-09-16T14:50:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:14.993826+0000 2024-09-16T14:50:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:14.993826+0000 2024-09-16T14:50:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:14.993 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 1 Processor -- start 2024-09-16T14:50:15.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 1 -- start start 2024-09-16T14:50:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 10 monclient: init 2024-09-16T14:50:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 10 monclient(hunting): picked mon.noname-a con 0x7f79400a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 10 monclient(hunting): picked mon.noname-b con 0x7f79400a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 10 monclient(hunting): picked mon.noname-c con 0x7f79400a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79400a3250 con 0x7f79400a1d10 2024-09-16T14:50:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794009f600 con 0x7f79400a2800 2024-09-16T14:50:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79400a3830 con 0x7f79400a0000 2024-09-16T14:50:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 10 monclient(hunting): authenticate will time out at 1439.113536s 2024-09-16T14:50:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f794548d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f79400a1d10 0x7f79400a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48850/0 (socket says 172.21.15.26:48850) 2024-09-16T14:50:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f794548d640 1 -- 172.21.15.26:0/1415302846 learned_addr learned my addr 172.21.15.26:0/1415302846 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7944c8c640 1 --1- 172.21.15.26:0/1415302846 >> v1:172.21.15.26:6790/0 conn(0x7f79400a0000 0x7f7940139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57660/0 (socket says 172.21.15.26:57660) 2024-09-16T14:50:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 1 -- 172.21.15.26:0/1415302846 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2406295559 0 0) 0x7f79400a3250 con 0x7f79400a1d10 2024-09-16T14:50:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient(hunting): my global_id is 164367 2024-09-16T14:50:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 1 -- 172.21.15.26:0/1415302846 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f792c003530 con 0x7f79400a1d10 2024-09-16T14:50:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 1 -- 172.21.15.26:0/1415302846 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522145967 0 0) 0x7f79400a3830 con 0x7f79400a0000 2024-09-16T14:50:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient(hunting): my global_id is 154619 2024-09-16T14:50:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 1 -- 172.21.15.26:0/1415302846 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79400a3250 con 0x7f79400a0000 2024-09-16T14:50:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 1 -- 172.21.15.26:0/1415302846 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1425690041 0 0) 0x7f792c003530 con 0x7f79400a1d10 2024-09-16T14:50:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 1 -- 172.21.15.26:0/1415302846 >> v1:172.21.15.26:6790/0 conn(0x7f79400a0000 legacy=0x7f7940139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 1 -- 172.21.15.26:0/1415302846 >> v1:172.21.15.189:6789/0 conn(0x7f79400a2800 legacy=0x7f794013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient: _finish_hunting 0 2024-09-16T14:50:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 1 monclient: found mon.noname-a 2024-09-16T14:50:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 1 -- 172.21.15.26:0/1415302846 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794009ed60 con 0x7f79400a1d10 2024-09-16T14:50:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient: _finish_auth 0 2024-09-16T14:50:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient: _check_auth_tickets 2024-09-16T14:50:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:15.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 1 -- 172.21.15.26:0/1415302846 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f793c001620 con 0x7f79400a1d10 2024-09-16T14:50:15.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:15.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient: dump: 2024-09-16T14:50:15.004 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.004 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.004 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.004 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.004 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.004 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.004 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.004 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.004 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.005 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 5 monclient: authenticate success, global_id 164367 2024-09-16T14:50:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 1 -- 172.21.15.26:0/1415302846 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f793c0018c0 con 0x7f79400a1d10 2024-09-16T14:50:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 1 -- 172.21.15.26:0/1415302846 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f793c001c50 con 0x7f79400a1d10 2024-09-16T14:50:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 4 set_mon_vals no callback set 2024-09-16T14:50:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f7937fff640 10 monclient: dump: 2024-09-16T14:50:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.005 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.006 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 10 monclient: shutdown 2024-09-16T14:50:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 1 -- 172.21.15.26:0/1415302846 >> v1:172.21.15.26:6789/0 conn(0x7f79400a1d10 legacy=0x7f79400a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 1 -- 172.21.15.26:0/1415302846 shutdown_connections 2024-09-16T14:50:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 1 -- 172.21.15.26:0/1415302846 >> 172.21.15.26:0/1415302846 conn(0x7f79400a7f30 msgr2=0x7f79400a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:15.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 1 -- 172.21.15.26:0/1415302846 shutdown_connections 2024-09-16T14:50:15.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 1 -- 172.21.15.26:0/1415302846 wait complete. 2024-09-16T14:50:15.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 10 monclient: build_initial_monmap 2024-09-16T14:50:15.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.989+0000 7f79476f1640 10 monclient: monmap: 2024-09-16T14:50:15.007 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:15.007 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:15.007 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:15.007 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:15.007 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:15.007 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:15.007 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:15.007 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:15.007 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:15.008 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 1 Processor -- start 2024-09-16T14:50:15.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 1 -- start start 2024-09-16T14:50:15.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 10 monclient: init 2024-09-16T14:50:15.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:15.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:15.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 10 monclient(hunting): picked mon.noname-b con 0x7f79400a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:15.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 10 monclient(hunting): picked mon.noname-a con 0x7f79400a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:15.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 10 monclient(hunting): picked mon.noname-c con 0x7f79400a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:15.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7940132380 con 0x7f79400a1d10 2024-09-16T14:50:15.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794009e500 con 0x7f79400a0000 2024-09-16T14:50:15.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7940130f70 con 0x7f79400a2800 2024-09-16T14:50:15.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:15.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 10 monclient(hunting): authenticate will time out at 1439.116723s 2024-09-16T14:50:15.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7944c8c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f79400a1d10 0x7f7940130860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48854/0 (socket says 172.21.15.26:48854) 2024-09-16T14:50:15.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7945c8e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f79400a2800 0x7f79401d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57672/0 (socket says 172.21.15.26:57672) 2024-09-16T14:50:15.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7944c8c640 1 -- 172.21.15.26:0/1382375923 learned_addr learned my addr 172.21.15.26:0/1382375923 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:15.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297991600 0 0) 0x7f7940130f70 con 0x7f79400a2800 2024-09-16T14:50:15.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): my global_id is 154622 2024-09-16T14:50:15.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7924003530 con 0x7f79400a2800 2024-09-16T14:50:15.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4132730699 0 0) 0x7f79400a3250 con 0x7f79400a1d10 2024-09-16T14:50:15.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): my global_id is 164370 2024-09-16T14:50:15.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7940130f70 con 0x7f79400a1d10 2024-09-16T14:50:15.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3689983687 0 0) 0x7f7924003530 con 0x7f79400a2800 2024-09-16T14:50:15.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79400a3250 con 0x7f79400a2800 2024-09-16T14:50:15.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7938001bc0 con 0x7f79400a2800 2024-09-16T14:50:15.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:15.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): dump: 2024-09-16T14:50:15.010 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.010 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.010 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.010 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.010 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.011 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.011 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.011 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.011 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.011 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:15.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1821706144 0 0) 0x7f7940130f70 con 0x7f79400a1d10 2024-09-16T14:50:15.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7924003530 con 0x7f79400a1d10 2024-09-16T14:50:15.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7930001660 con 0x7f79400a1d10 2024-09-16T14:50:15.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:15.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): dump: 2024-09-16T14:50:15.011 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.011 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.012 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.012 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.012 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.012 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.012 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.012 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.012 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.012 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695682389 0 0) 0x7f794009e500 con 0x7f79400a0000 2024-09-16T14:50:15.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): my global_id is 174157 2024-09-16T14:50:15.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7940130f70 con 0x7f79400a0000 2024-09-16T14:50:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3678042927 0 0) 0x7f79400a3250 con 0x7f79400a2800 2024-09-16T14:50:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 >> v1:172.21.15.189:6789/0 conn(0x7f79400a0000 legacy=0x7f7940130150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 >> v1:172.21.15.26:6789/0 conn(0x7f79400a1d10 legacy=0x7f7940130860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:50:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 monclient: found mon.c 2024-09-16T14:50:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7940131590 con 0x7f79400a2800 2024-09-16T14:50:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient: _finish_auth 0 2024-09-16T14:50:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 5 monclient: authenticate success, global_id 154622 2024-09-16T14:50:15.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 10 monclient: _renew_subs 2024-09-16T14:50:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 1 -- 172.21.15.26:0/1382375923 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79400a6730 con 0x7f79400a2800 2024-09-16T14:50:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 10 monclient: _renew_subs 2024-09-16T14:50:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 1 -- 172.21.15.26:0/1382375923 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7940131b90 con 0x7f79400a2800 2024-09-16T14:50:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79380039a0 con 0x7f79400a2800 2024-09-16T14:50:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7938003d30 con 0x7f79400a2800 2024-09-16T14:50:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:15.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79377fe640 4 set_mon_vals no callback set 2024-09-16T14:50:15.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient: dump: 2024-09-16T14:50:15.015 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.015 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.015 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.015 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.015 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.015 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.015 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.015 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.015 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.015 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:15.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f79476f1640 1 -- 172.21.15.26:0/1382375923 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7920006590 con 0x7f7920002830 2024-09-16T14:50:15.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7938003e90 con 0x7f79400a2800 2024-09-16T14:50:15.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(223..223 src has 169..223) v4 ==== 12392+0+0 (unknown 467405984 0 0) 0x7f79380456d0 con 0x7f79400a2800 2024-09-16T14:50:15.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97584425 0 0) 0x7f7920006590 con 0x7f7920002830 2024-09-16T14:50:15.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient(hunting): my global_id is 164373 2024-09-16T14:50:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79400a3250 con 0x7f7920002830 2024-09-16T14:50:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7944c8c640 10 monclient: get_auth_request con 0x7f79240363d0 auth_method 0 2024-09-16T14:50:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 944242202 0 0) 0x7f79400a3250 con 0x7f7920002830 2024-09-16T14:50:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7920006590 con 0x7f7920002830 2024-09-16T14:50:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f793c001190 con 0x7f7920002830 2024-09-16T14:50:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient: dump: 2024-09-16T14:50:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.017 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.018 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.018 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.018 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.018 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.018 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3296491708 0 0) 0x7f7920006590 con 0x7f7920002830 2024-09-16T14:50:15.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f792000a0e0 con 0x7f7920002830 2024-09-16T14:50:15.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f793c001b60 con 0x7f7920002830 2024-09-16T14:50:15.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:15.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:15.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 >> v1:172.21.15.26:6789/0 conn(0x7f7920002830 legacy=0x7f7920004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:14.993+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 >> v1:172.21.15.26:6789/0 conn(0x7f7920002830 legacy=0x7f7920004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:15.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f79476f1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:50:15.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f79476f1640 1 -- 172.21.15.26:0/1382375923 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7920009d80 con 0x7f79200090a0 2024-09-16T14:50:15.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076310470 0 0) 0x7f7920009d80 con 0x7f79200090a0 2024-09-16T14:50:15.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 10 monclient(hunting): my global_id is 164376 2024-09-16T14:50:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7920006590 con 0x7f79200090a0 2024-09-16T14:50:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 822303773 0 0) 0x7f7920006590 con 0x7f79200090a0 2024-09-16T14:50:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7920009d80 con 0x7f79200090a0 2024-09-16T14:50:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f793c004ce0 con 0x7f79200090a0 2024-09-16T14:50:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 10 monclient: dump: 2024-09-16T14:50:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.021 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.022 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.022 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.022 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.022 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3703624404 0 0) 0x7f7920009d80 con 0x7f79200090a0 2024-09-16T14:50:15.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f792000c290 con 0x7f79200090a0 2024-09-16T14:50:15.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 568936885) 0x7f793c005fe0 con 0x7f79200090a0 2024-09-16T14:50:15.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:50:15.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:15.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 >> v1:172.21.15.26:6789/0 conn(0x7f79200090a0 legacy=0x7f7920009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7935ffb640 1 -- 172.21.15.26:0/1382375923 >> v1:172.21.15.26:6789/0 conn(0x7f79200090a0 legacy=0x7f7920009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:15.023 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:50:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:50:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 72, 2024-09-16T14:50:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:50:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:50:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:50:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:50:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:50:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:50:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:50:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:50:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:50:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:15.025 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:50:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:50:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:50:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:50:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:50:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:50:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:50:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:50:15.026 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:50:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:50:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:50:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:50:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:50:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:50:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:15.027 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:50:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:50:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:15.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:50:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:50:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:50:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:50:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:50:15.030 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:50:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:50:15.031 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:50:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:50:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.032 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:50:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:50:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:50:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:50:15.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:50:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.034 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:50:15.034 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:50:15.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f79476f1640 1 -- 172.21.15.26:0/1382375923 >> v1:172.21.15.189:6812/3196161582 conn(0x7f79240363d0 legacy=0x7f7924038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f79476f1640 10 monclient: shutdown 2024-09-16T14:50:15.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f79476f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:15.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f79476f1640 1 -- 172.21.15.26:0/1382375923 >> v1:172.21.15.26:6790/0 conn(0x7f79400a2800 legacy=0x7f79401d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7945c8e640 1 -- 172.21.15.26:0/1382375923 reap_dead start 2024-09-16T14:50:15.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f7945c8e640 1 -- 172.21.15.26:0/1382375923 reap_dead start 2024-09-16T14:50:15.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f79476f1640 1 -- 172.21.15.26:0/1382375923 shutdown_connections 2024-09-16T14:50:15.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f79476f1640 1 -- 172.21.15.26:0/1382375923 >> 172.21.15.26:0/1382375923 conn(0x7f79400a7f30 msgr2=0x7f794006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:15.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f79476f1640 1 -- 172.21.15.26:0/1382375923 shutdown_connections 2024-09-16T14:50:15.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.001+0000 7f79476f1640 1 -- 172.21.15.26:0/1382375923 wait complete. 2024-09-16T14:50:15.037 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:50:15.037 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:50:15.037 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:50:15.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.157+0000 7f4c8a9f0640 10 monclient: get_monmap_and_config 2024-09-16T14:50:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.157+0000 7f4c8a9f0640 10 monclient: build_initial_monmap 2024-09-16T14:50:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.157+0000 7f4c8a9f0640 10 monclient: monmap: 2024-09-16T14:50:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:15.163758+0000 2024-09-16T14:50:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:15.163758+0000 2024-09-16T14:50:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:15.163 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.157+0000 7f4c8a9f0640 1 Processor -- start 2024-09-16T14:50:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.157+0000 7f4c8a9f0640 1 -- start start 2024-09-16T14:50:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.157+0000 7f4c8a9f0640 10 monclient: init 2024-09-16T14:50:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.157+0000 7f4c8a9f0640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.157+0000 7f4c8a9f0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.157+0000 7f4c8a9f0640 10 monclient(hunting): picked mon.noname-b con 0x7f4c8409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 10 monclient(hunting): picked mon.noname-c con 0x7f4c84132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 10 monclient(hunting): picked mon.noname-a con 0x7f4c84138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8409d260 con 0x7f4c84138ff0 2024-09-16T14:50:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c84132a80 con 0x7f4c84132650 2024-09-16T14:50:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8409cf60 con 0x7f4c8409cb80 2024-09-16T14:50:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 10 monclient(hunting): authenticate will time out at 1439.284097s 2024-09-16T14:50:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c88f8d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4c84138ff0 0x7f4c8413b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48876/0 (socket says 172.21.15.26:48876) 2024-09-16T14:50:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c88f8d640 1 -- 172.21.15.26:0/2656911850 learned_addr learned my addr 172.21.15.26:0/2656911850 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:15.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 1 -- 172.21.15.26:0/2656911850 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277102577 0 0) 0x7f4c8409d260 con 0x7f4c84138ff0 2024-09-16T14:50:15.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient(hunting): my global_id is 164379 2024-09-16T14:50:15.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 1 -- 172.21.15.26:0/2656911850 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c6c003530 con 0x7f4c84138ff0 2024-09-16T14:50:15.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 1 -- 172.21.15.26:0/2656911850 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369377868 0 0) 0x7f4c8409cf60 con 0x7f4c8409cb80 2024-09-16T14:50:15.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient(hunting): my global_id is 154625 2024-09-16T14:50:15.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 1 -- 172.21.15.26:0/2656911850 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c8409d260 con 0x7f4c8409cb80 2024-09-16T14:50:15.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 1 -- 172.21.15.26:0/2656911850 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2871176444 0 0) 0x7f4c8409d260 con 0x7f4c8409cb80 2024-09-16T14:50:15.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 1 -- 172.21.15.26:0/2656911850 >> v1:172.21.15.189:6789/0 conn(0x7f4c84132650 legacy=0x7f4c841378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 1 -- 172.21.15.26:0/2656911850 >> v1:172.21.15.26:6789/0 conn(0x7f4c84138ff0 legacy=0x7f4c8413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:15.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 1 monclient: found mon.noname-b 2024-09-16T14:50:15.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:15.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:15.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 1 -- 172.21.15.26:0/2656911850 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8409f2d0 con 0x7f4c8409cb80 2024-09-16T14:50:15.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:15.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:15.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:15.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 1 -- 172.21.15.26:0/2656911850 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4c74001640 con 0x7f4c8409cb80 2024-09-16T14:50:15.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:15.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient: dump: 2024-09-16T14:50:15.174 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.174 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.174 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.174 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.174 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.174 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.175 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.175 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.175 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.175 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 5 monclient: authenticate success, global_id 154625 2024-09-16T14:50:15.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:15.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 1 -- 172.21.15.26:0/2656911850 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c740017e0 con 0x7f4c8409cb80 2024-09-16T14:50:15.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:15.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:15.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 4 set_mon_vals no callback set 2024-09-16T14:50:15.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 1 -- 172.21.15.26:0/2656911850 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4c74001bb0 con 0x7f4c8409cb80 2024-09-16T14:50:15.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:15.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c82ffd640 10 monclient: dump: 2024-09-16T14:50:15.176 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.176 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.176 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.176 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.176 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.176 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.176 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.176 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.176 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.176 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 10 monclient: shutdown 2024-09-16T14:50:15.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:15.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 1 -- 172.21.15.26:0/2656911850 >> v1:172.21.15.26:6790/0 conn(0x7f4c8409cb80 legacy=0x7f4c840717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 1 -- 172.21.15.26:0/2656911850 shutdown_connections 2024-09-16T14:50:15.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 1 -- 172.21.15.26:0/2656911850 >> 172.21.15.26:0/2656911850 conn(0x7f4c8409fb50 msgr2=0x7f4c8409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:15.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 1 -- 172.21.15.26:0/2656911850 shutdown_connections 2024-09-16T14:50:15.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 1 -- 172.21.15.26:0/2656911850 wait complete. 2024-09-16T14:50:15.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 10 monclient: build_initial_monmap 2024-09-16T14:50:15.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 10 monclient: monmap: 2024-09-16T14:50:15.177 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:15.177 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:15.177 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:15.177 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:15.177 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:15.177 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:15.177 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:15.178 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:15.178 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:15.178 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 1 Processor -- start 2024-09-16T14:50:15.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 1 -- start start 2024-09-16T14:50:15.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 10 monclient: init 2024-09-16T14:50:15.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:15.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:15.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 10 monclient(hunting): picked mon.noname-b con 0x7f4c8409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:15.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 10 monclient(hunting): picked mon.noname-c con 0x7f4c84132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:15.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 10 monclient(hunting): picked mon.noname-a con 0x7f4c84138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:15.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c84138150 con 0x7f4c84138ff0 2024-09-16T14:50:15.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8406e450 con 0x7f4c8409cb80 2024-09-16T14:50:15.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c84071590 con 0x7f4c84132650 2024-09-16T14:50:15.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:15.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c8a9f0640 10 monclient(hunting): authenticate will time out at 1439.287479s 2024-09-16T14:50:15.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c88f8d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4c84138ff0 0x7f4c841cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48892/0 (socket says 172.21.15.26:48892) 2024-09-16T14:50:15.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c88f8d640 1 -- 172.21.15.26:0/4047098905 learned_addr learned my addr 172.21.15.26:0/4047098905 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:15.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c837fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4c84132650 0x7f4c8406dd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57690/0 (socket says 172.21.15.26:57690) 2024-09-16T14:50:15.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998059018 0 0) 0x7f4c7c0015c0 con 0x7f4c84138ff0 2024-09-16T14:50:15.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c80ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c80ff9640 10 monclient(hunting): my global_id is 164382 2024-09-16T14:50:15.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c70003530 con 0x7f4c84138ff0 2024-09-16T14:50:15.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902202651 0 0) 0x7f4c84071590 con 0x7f4c84132650 2024-09-16T14:50:15.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c80ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c80ff9640 10 monclient(hunting): my global_id is 154628 2024-09-16T14:50:15.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.161+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c7c0015c0 con 0x7f4c84132650 2024-09-16T14:50:15.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4009010154 0 0) 0x7f4c70003530 con 0x7f4c84138ff0 2024-09-16T14:50:15.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c84071590 con 0x7f4c84138ff0 2024-09-16T14:50:15.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4c7c001c60 con 0x7f4c84138ff0 2024-09-16T14:50:15.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:15.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient(hunting): dump: 2024-09-16T14:50:15.180 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.180 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.180 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.180 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.181 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.181 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.181 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.181 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.181 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.181 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:15.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4082070793 0 0) 0x7f4c7c0015c0 con 0x7f4c84132650 2024-09-16T14:50:15.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c70003530 con 0x7f4c84132650 2024-09-16T14:50:15.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4c78001830 con 0x7f4c84132650 2024-09-16T14:50:15.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:15.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient(hunting): dump: 2024-09-16T14:50:15.182 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.182 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.182 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.182 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.182 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.182 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.182 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.182 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.182 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.183 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551878910 0 0) 0x7f4c8406e450 con 0x7f4c8409cb80 2024-09-16T14:50:15.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient(hunting): my global_id is 174163 2024-09-16T14:50:15.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c7c0015c0 con 0x7f4c8409cb80 2024-09-16T14:50:15.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1361771208 0 0) 0x7f4c84071590 con 0x7f4c84138ff0 2024-09-16T14:50:15.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 >> v1:172.21.15.26:6790/0 conn(0x7f4c84132650 legacy=0x7f4c8406dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 >> v1:172.21.15.189:6789/0 conn(0x7f4c8409cb80 legacy=0x7f4c8406d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:50:15.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 monclient: found mon.a 2024-09-16T14:50:15.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:15.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:15.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8413c400 con 0x7f4c84138ff0 2024-09-16T14:50:15.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient: _finish_auth 0 2024-09-16T14:50:15.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:50:15.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:15.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c8a9f0640 5 monclient: authenticate success, global_id 164382 2024-09-16T14:50:15.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c8a9f0640 10 monclient: _renew_subs 2024-09-16T14:50:15.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c8a9f0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:15.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c8a9f0640 1 -- 172.21.15.26:0/4047098905 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c84070fd0 con 0x7f4c84138ff0 2024-09-16T14:50:15.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c8a9f0640 10 monclient: _renew_subs 2024-09-16T14:50:15.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c8a9f0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:15.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c8a9f0640 1 -- 172.21.15.26:0/4047098905 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8406fb80 con 0x7f4c84138ff0 2024-09-16T14:50:15.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c7c002000 con 0x7f4c84138ff0 2024-09-16T14:50:15.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:15.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4c7c004fc0 con 0x7f4c84138ff0 2024-09-16T14:50:15.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:15.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient: dump: 2024-09-16T14:50:15.185 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.185 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.185 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.185 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.185 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.186 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.186 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.186 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.186 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.186 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c827fc640 4 set_mon_vals no callback set 2024-09-16T14:50:15.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c8a9f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:15.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c8a9f0640 1 -- 172.21.15.26:0/4047098905 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4c64006590 con 0x7f4c64002830 2024-09-16T14:50:15.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4c7c00fa30 con 0x7f4c84138ff0 2024-09-16T14:50:15.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(223..223 src has 169..223) v4 ==== 12392+0+0 (unknown 467405984 0 0) 0x7f4c7c0456f0 con 0x7f4c84138ff0 2024-09-16T14:50:15.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460450871 0 0) 0x7f4c64006590 con 0x7f4c64002830 2024-09-16T14:50:15.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient(hunting): my global_id is 154631 2024-09-16T14:50:15.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c84071590 con 0x7f4c64002830 2024-09-16T14:50:15.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c837fe640 10 monclient: get_auth_request con 0x7f4c700363b0 auth_method 0 2024-09-16T14:50:15.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4004816034 0 0) 0x7f4c84071590 con 0x7f4c64002830 2024-09-16T14:50:15.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c64006590 con 0x7f4c64002830 2024-09-16T14:50:15.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4c74001950 con 0x7f4c64002830 2024-09-16T14:50:15.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:15.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient: dump: 2024-09-16T14:50:15.187 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.187 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.187 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.188 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 395573636 0 0) 0x7f4c64006590 con 0x7f4c64002830 2024-09-16T14:50:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c6400a0e0 con 0x7f4c64002830 2024-09-16T14:50:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f4c74003600 con 0x7f4c64002830 2024-09-16T14:50:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 >> v1:172.21.15.26:6790/0 conn(0x7f4c64002830 legacy=0x7f4c64004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.165+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 >> v1:172.21.15.26:6790/0 conn(0x7f4c64002830 legacy=0x7f4c64004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:15.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c8a9f0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:50:15.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c8a9f0640 1 -- 172.21.15.26:0/4047098905 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4c64009da0 con 0x7f4c640090c0 2024-09-16T14:50:15.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936102797 0 0) 0x7f4c64009da0 con 0x7f4c640090c0 2024-09-16T14:50:15.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 10 monclient(hunting): my global_id is 154634 2024-09-16T14:50:15.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c64006590 con 0x7f4c640090c0 2024-09-16T14:50:15.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 338591837 0 0) 0x7f4c64006590 con 0x7f4c640090c0 2024-09-16T14:50:15.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c64009da0 con 0x7f4c640090c0 2024-09-16T14:50:15.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4c74003bc0 con 0x7f4c640090c0 2024-09-16T14:50:15.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:15.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 10 monclient: dump: 2024-09-16T14:50:15.191 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.191 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.191 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.191 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.191 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.191 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.192 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.192 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.192 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.192 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2350995949 0 0) 0x7f4c64009da0 con 0x7f4c640090c0 2024-09-16T14:50:15.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4c6400c290 con 0x7f4c640090c0 2024-09-16T14:50:15.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 2292703343) 0x7f4c74004fe0 con 0x7f4c640090c0 2024-09-16T14:50:15.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:50:15.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:15.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 >> v1:172.21.15.26:6790/0 conn(0x7f4c640090c0 legacy=0x7f4c64009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c80ff9640 1 -- 172.21.15.26:0/4047098905 >> v1:172.21.15.26:6790/0 conn(0x7f4c640090c0 legacy=0x7f4c64009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 72, 2024-09-16T14:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:50:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:50:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:50:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.203 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:50:15.204 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:50:15.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c8a9f0640 1 -- 172.21.15.26:0/4047098905 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4c700363b0 legacy=0x7f4c70038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c8a9f0640 10 monclient: shutdown 2024-09-16T14:50:15.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c8a9f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:15.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c8a9f0640 1 -- 172.21.15.26:0/4047098905 >> v1:172.21.15.26:6789/0 conn(0x7f4c84138ff0 legacy=0x7f4c841cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c88f8d640 1 -- 172.21.15.26:0/4047098905 reap_dead start 2024-09-16T14:50:15.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c88f8d640 1 -- 172.21.15.26:0/4047098905 reap_dead start 2024-09-16T14:50:15.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c8a9f0640 1 -- 172.21.15.26:0/4047098905 shutdown_connections 2024-09-16T14:50:15.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c8a9f0640 1 -- 172.21.15.26:0/4047098905 >> 172.21.15.26:0/4047098905 conn(0x7f4c8409fb50 msgr2=0x7f4c84131ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:15.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c8a9f0640 1 -- 172.21.15.26:0/4047098905 shutdown_connections 2024-09-16T14:50:15.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.173+0000 7f4c8a9f0640 1 -- 172.21.15.26:0/4047098905 wait complete. 2024-09-16T14:50:15.207 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:50:15.207 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:50:15.207 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:50:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 10 monclient: get_monmap_and_config 2024-09-16T14:50:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 10 monclient: build_initial_monmap 2024-09-16T14:50:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 10 monclient: monmap: 2024-09-16T14:50:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:15.330345+0000 2024-09-16T14:50:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:15.330345+0000 2024-09-16T14:50:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:15.330 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 1 Processor -- start 2024-09-16T14:50:15.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 1 -- start start 2024-09-16T14:50:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 10 monclient: init 2024-09-16T14:50:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 10 monclient(hunting): picked mon.noname-c con 0x7f11140966a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 10 monclient(hunting): picked mon.noname-b con 0x7f1114071940 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 10 monclient(hunting): picked mon.noname-a con 0x7f111413b100 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111409b3c0 con 0x7f111413b100 2024-09-16T14:50:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111409fd90 con 0x7f11140966a0 2024-09-16T14:50:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111409b170 con 0x7f1114071940 2024-09-16T14:50:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 10 monclient(hunting): authenticate will time out at 1439.450723s 2024-09-16T14:50:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f1119eb8640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f111413b100 0x7f111413d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48902/0 (socket says 172.21.15.26:48902) 2024-09-16T14:50:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f1119eb8640 1 -- 172.21.15.26:0/2772171406 learned_addr learned my addr 172.21.15.26:0/2772171406 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 1 -- 172.21.15.26:0/2772171406 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549385673 0 0) 0x7f111409b3c0 con 0x7f111413b100 2024-09-16T14:50:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient(hunting): my global_id is 164385 2024-09-16T14:50:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 1 -- 172.21.15.26:0/2772171406 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10fc003530 con 0x7f111413b100 2024-09-16T14:50:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 1 -- 172.21.15.26:0/2772171406 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798128399 0 0) 0x7f111409b170 con 0x7f1114071940 2024-09-16T14:50:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient(hunting): my global_id is 154637 2024-09-16T14:50:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 1 -- 172.21.15.26:0/2772171406 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f111409b3c0 con 0x7f1114071940 2024-09-16T14:50:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 1 -- 172.21.15.26:0/2772171406 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2108337721 0 0) 0x7f10fc003530 con 0x7f111413b100 2024-09-16T14:50:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 1 -- 172.21.15.26:0/2772171406 >> v1:172.21.15.26:6790/0 conn(0x7f1114071940 legacy=0x7f11141399e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 1 -- 172.21.15.26:0/2772171406 >> v1:172.21.15.189:6789/0 conn(0x7f11140966a0 legacy=0x7f1114096a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:50:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 1 monclient: found mon.noname-a 2024-09-16T14:50:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 1 -- 172.21.15.26:0/2772171406 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1114072160 con 0x7f111413b100 2024-09-16T14:50:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient: _finish_auth 0 2024-09-16T14:50:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:50:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 1 -- 172.21.15.26:0/2772171406 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1110001620 con 0x7f111413b100 2024-09-16T14:50:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient: dump: 2024-09-16T14:50:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.336 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 5 monclient: authenticate success, global_id 164385 2024-09-16T14:50:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 1 -- 172.21.15.26:0/2772171406 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1110001850 con 0x7f111413b100 2024-09-16T14:50:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f111a6b9640 4 set_mon_vals no callback set 2024-09-16T14:50:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 1 -- 172.21.15.26:0/2772171406 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1110001c20 con 0x7f111413b100 2024-09-16T14:50:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:15.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.325+0000 7f110bfff640 10 monclient: dump: 2024-09-16T14:50:15.340 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.340 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.341 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 10 monclient: shutdown 2024-09-16T14:50:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 1 -- 172.21.15.26:0/2772171406 >> v1:172.21.15.26:6789/0 conn(0x7f111413b100 legacy=0x7f111413d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 1 -- 172.21.15.26:0/2772171406 shutdown_connections 2024-09-16T14:50:15.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 1 -- 172.21.15.26:0/2772171406 >> 172.21.15.26:0/2772171406 conn(0x7f11140a3bf0 msgr2=0x7f11140a4000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:15.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 1 -- 172.21.15.26:0/2772171406 shutdown_connections 2024-09-16T14:50:15.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 1 -- 172.21.15.26:0/2772171406 wait complete. 2024-09-16T14:50:15.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 10 monclient: build_initial_monmap 2024-09-16T14:50:15.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 10 monclient: monmap: 2024-09-16T14:50:15.348 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:15.348 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:15.349 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:15.349 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:15.349 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:15.349 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:15.349 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:15.349 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:15.349 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:15.349 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 1 Processor -- start 2024-09-16T14:50:15.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 1 -- start start 2024-09-16T14:50:15.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 10 monclient: init 2024-09-16T14:50:15.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:15.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:15.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 10 monclient(hunting): picked mon.noname-b con 0x7f1114071940 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:15.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 10 monclient(hunting): picked mon.noname-c con 0x7f11140966a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:15.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 10 monclient(hunting): picked mon.noname-a con 0x7f111413b100 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:15.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1114094610 con 0x7f111413b100 2024-09-16T14:50:15.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111408ae10 con 0x7f1114071940 2024-09-16T14:50:15.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111408c600 con 0x7f11140966a0 2024-09-16T14:50:15.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:15.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 10 monclient(hunting): authenticate will time out at 1439.453868s 2024-09-16T14:50:15.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1119eb8640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f111413b100 0x7f11141d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48912/0 (socket says 172.21.15.26:48912) 2024-09-16T14:50:15.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1118eb6640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f11140966a0 0x7f111408a700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57732/0 (socket says 172.21.15.26:57732) 2024-09-16T14:50:15.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1119eb8640 1 -- 172.21.15.26:0/4078667644 learned_addr learned my addr 172.21.15.26:0/4078667644 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:15.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298788846 0 0) 0x7f111408c600 con 0x7f11140966a0 2024-09-16T14:50:15.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): my global_id is 154640 2024-09-16T14:50:15.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1100003530 con 0x7f11140966a0 2024-09-16T14:50:15.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156985487 0 0) 0x7f1114072160 con 0x7f111413b100 2024-09-16T14:50:15.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): my global_id is 164388 2024-09-16T14:50:15.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f111408c600 con 0x7f111413b100 2024-09-16T14:50:15.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 954809838 0 0) 0x7f1100003530 con 0x7f11140966a0 2024-09-16T14:50:15.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1114072160 con 0x7f11140966a0 2024-09-16T14:50:15.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 370208610 0 0) 0x7f111408c600 con 0x7f111413b100 2024-09-16T14:50:15.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1100003530 con 0x7f111413b100 2024-09-16T14:50:15.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f110c001be0 con 0x7f11140966a0 2024-09-16T14:50:15.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:15.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): dump: 2024-09-16T14:50:15.353 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.353 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.354 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.354 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.354 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.354 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.354 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.354 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.354 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.354 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:15.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f11100019c0 con 0x7f111413b100 2024-09-16T14:50:15.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:15.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): dump: 2024-09-16T14:50:15.355 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.355 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.355 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.355 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.355 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.356 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.356 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.356 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.356 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.356 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692201729 0 0) 0x7f111408ae10 con 0x7f1114071940 2024-09-16T14:50:15.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient(hunting): my global_id is 174169 2024-09-16T14:50:15.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f111408c600 con 0x7f1114071940 2024-09-16T14:50:15.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1848458461 0 0) 0x7f1114072160 con 0x7f11140966a0 2024-09-16T14:50:15.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 >> v1:172.21.15.189:6789/0 conn(0x7f1114071940 legacy=0x7f1114089ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 >> v1:172.21.15.26:6789/0 conn(0x7f111413b100 legacy=0x7f11141d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:50:15.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 monclient: found mon.c 2024-09-16T14:50:15.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:15.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:15.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111408ba50 con 0x7f11140966a0 2024-09-16T14:50:15.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient: _finish_auth 0 2024-09-16T14:50:15.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:15.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:15.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 5 monclient: authenticate success, global_id 154640 2024-09-16T14:50:15.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 10 monclient: _renew_subs 2024-09-16T14:50:15.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:15.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 1 -- 172.21.15.26:0/4078667644 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f111409f230 con 0x7f11140966a0 2024-09-16T14:50:15.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 10 monclient: _renew_subs 2024-09-16T14:50:15.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:15.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 1 -- 172.21.15.26:0/4078667644 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f111408db20 con 0x7f11140966a0 2024-09-16T14:50:15.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f110c001d90 con 0x7f11140966a0 2024-09-16T14:50:15.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:15.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f110c003ba0 con 0x7f11140966a0 2024-09-16T14:50:15.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f110b7fe640 4 set_mon_vals no callback set 2024-09-16T14:50:15.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:15.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 10 monclient: dump: 2024-09-16T14:50:15.359 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.359 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.359 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.359 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.359 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.359 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.359 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.359 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.359 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.359 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:15.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f111a6b9640 1 -- 172.21.15.26:0/4078667644 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f10f4006590 con 0x7f10f4002830 2024-09-16T14:50:15.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.329+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f110c003dc0 con 0x7f11140966a0 2024-09-16T14:50:15.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(223..223 src has 169..223) v4 ==== 12392+0+0 (unknown 467405984 0 0) 0x7f110c045710 con 0x7f11140966a0 2024-09-16T14:50:15.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1119eb8640 10 monclient: get_auth_request con 0x7f1100036420 auth_method 0 2024-09-16T14:50:15.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082491492 0 0) 0x7f10f4006590 con 0x7f10f4002830 2024-09-16T14:50:15.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 10 monclient(hunting): my global_id is 174172 2024-09-16T14:50:15.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1114072160 con 0x7f10f4002830 2024-09-16T14:50:15.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3911776826 0 0) 0x7f1114072160 con 0x7f10f4002830 2024-09-16T14:50:15.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10f4006590 con 0x7f10f4002830 2024-09-16T14:50:15.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f11040017a0 con 0x7f10f4002830 2024-09-16T14:50:15.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:15.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 10 monclient: dump: 2024-09-16T14:50:15.361 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.361 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.361 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.361 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.361 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.361 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.361 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.361 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.361 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.362 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 458736587 0 0) 0x7f10f4006590 con 0x7f10f4002830 2024-09-16T14:50:15.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10f400a0e0 con 0x7f10f4002830 2024-09-16T14:50:15.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1104001b70 con 0x7f10f4002830 2024-09-16T14:50:15.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:15.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:15.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 >> v1:172.21.15.189:6789/0 conn(0x7f10f4002830 legacy=0x7f10f4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.333+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 >> v1:172.21.15.189:6789/0 conn(0x7f10f4002830 legacy=0x7f10f4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:15.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f111a6b9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:50:15.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f111a6b9640 1 -- 172.21.15.26:0/4078667644 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f10f4009d80 con 0x7f10f40090a0 2024-09-16T14:50:15.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143770011 0 0) 0x7f10f4009d80 con 0x7f10f40090a0 2024-09-16T14:50:15.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 10 monclient(hunting): my global_id is 174175 2024-09-16T14:50:15.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f4006590 con 0x7f10f40090a0 2024-09-16T14:50:15.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2843523671 0 0) 0x7f10f4006590 con 0x7f10f40090a0 2024-09-16T14:50:15.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10f4009d80 con 0x7f10f40090a0 2024-09-16T14:50:15.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f11040037a0 con 0x7f10f40090a0 2024-09-16T14:50:15.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:15.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 10 monclient: dump: 2024-09-16T14:50:15.363 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.363 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.363 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.363 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.363 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.364 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.364 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.364 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.364 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.364 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 413822086 0 0) 0x7f10f4009d80 con 0x7f10f40090a0 2024-09-16T14:50:15.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f10f400c290 con 0x7f10f40090a0 2024-09-16T14:50:15.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 1593390084) 0x7f1104003ad0 con 0x7f10f40090a0 2024-09-16T14:50:15.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:50:15.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:15.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 >> v1:172.21.15.189:6789/0 conn(0x7f10f40090a0 legacy=0x7f10f4009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1109ffb640 1 -- 172.21.15.26:0/4078667644 >> v1:172.21.15.189:6789/0 conn(0x7f10f40090a0 legacy=0x7f10f4009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:15.365 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:50:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:50:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 72, 2024-09-16T14:50:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:50:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:50:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:50:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:50:15.365 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:50:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:50:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:50:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:15.366 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:50:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:50:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:15.367 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:50:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:50:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:50:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:50:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:50:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:50:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:50:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:50:15.368 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:50:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:50:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:50:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:50:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:50:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:50:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:15.369 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:50:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:50:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.370 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:50:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:50:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:50:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:15.371 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:50:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:50:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:50:15.372 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:50:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:15.373 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:50:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:50:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:50:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:50:15.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:50:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:50:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:15.375 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:50:15.375 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:50:15.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f111a6b9640 1 -- 172.21.15.26:0/4078667644 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1100036420 legacy=0x7f11000388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f111a6b9640 10 monclient: shutdown 2024-09-16T14:50:15.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f111a6b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:15.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f111a6b9640 1 -- 172.21.15.26:0/4078667644 >> v1:172.21.15.26:6790/0 conn(0x7f11140966a0 legacy=0x7f111408a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1119eb8640 1 -- 172.21.15.26:0/4078667644 reap_dead start 2024-09-16T14:50:15.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f1119eb8640 1 -- 172.21.15.26:0/4078667644 reap_dead start 2024-09-16T14:50:15.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f111a6b9640 1 -- 172.21.15.26:0/4078667644 shutdown_connections 2024-09-16T14:50:15.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f111a6b9640 1 -- 172.21.15.26:0/4078667644 >> 172.21.15.26:0/4078667644 conn(0x7f11140a3bf0 msgr2=0x7f111409c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:15.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f111a6b9640 1 -- 172.21.15.26:0/4078667644 shutdown_connections 2024-09-16T14:50:15.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.341+0000 7f111a6b9640 1 -- 172.21.15.26:0/4078667644 wait complete. 2024-09-16T14:50:15.378 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:50:15.378 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T14:50:15.378 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:50:15.378 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T14:50:15.378 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:50:15.379 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T14:50:15.379 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:50:15.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f541f59e640 1 -- 172.21.15.189:0/1831705282 >> v1:172.21.15.189:6789/0 conn(0x7f542013c740 legacy=0x7f54201d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T14:50:15.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f541f59e640 1 -- 172.21.15.189:0/1831705282 >> v1:172.21.15.189:6789/0 conn(0x7f542013c740 legacy=0x7f54201d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:50:15.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f541f59e640 1 --1- 172.21.15.189:0/1831705282 >> v1:172.21.15.189:6789/0 conn(0x7f542013c740 0x7f54201d1900 :-1 s=OPENED pgs=911 cs=1 l=1).handle_message read tag failed 2024-09-16T14:50:15.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f541f59e640 1 --1- 172.21.15.189:0/1831705282 >> v1:172.21.15.189:6789/0 conn(0x7f542013c740 0x7f54201d1900 :-1 s=OPENED pgs=911 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:50:15.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f54177fe640 10 monclient: ms_handle_reset current mon v1:172.21.15.189:6789/0 2024-09-16T14:50:15.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f54177fe640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:15.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 >> v1:172.21.15.189:6789/0 conn(0x7f542013c740 legacy=0x7f54201d1900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:15.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f54177fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:15.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f54177fe640 10 monclient(hunting): picked mon.a con 0x7f54201315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:15.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f54177fe640 10 monclient(hunting): picked mon.c con 0x7f5420132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:15.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f54177fe640 10 monclient(hunting): picked mon.b con 0x7f5400050710 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:15.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f542006b380 con 0x7f54201315a0 2024-09-16T14:50:15.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f542009cd20 con 0x7f5400050710 2024-09-16T14:50:15.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5400056890 con 0x7f5420132090 2024-09-16T14:50:15.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f541ed9d640 1 -- 172.21.15.189:0/1831705282 >> v1:172.21.15.189:6789/0 conn(0x7f5400050710 legacy=0x7f5400052ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:15.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f54177fe640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:15.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669082536 0 0) 0x7f5400056890 con 0x7f5420132090 2024-09-16T14:50:15.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f54177fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f54177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.380+0000 7f54177fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:50:15.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.26:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f5400054830 con 0x7f5420132090 2024-09-16T14:50:15.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356720882 0 0) 0x7f542006b380 con 0x7f54201315a0 2024-09-16T14:50:15.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:50:15.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.26:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f5400056890 con 0x7f54201315a0 2024-09-16T14:50:15.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 3138707105 0 0) 0x7f5400054830 con 0x7f5420132090 2024-09-16T14:50:15.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 >> v1:172.21.15.189:6789/0 conn(0x7f5400050710 legacy=0x7f5400052ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:15.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 >> v1:172.21.15.26:6789/0 conn(0x7f54201315a0 legacy=0x7f540004b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:15.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 1 monclient: found mon.c 2024-09-16T14:50:15.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:15.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:15.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f5400057ab0 con 0x7f5420132090 2024-09-16T14:50:15.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T14:50:15.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:15.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) v1 -- 0x7f542009cd20 con 0x7f5420132090 2024-09-16T14:50:15.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:15.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:15.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:15.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5418014990 con 0x7f5420132090 2024-09-16T14:50:15.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:15.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 10 monclient: dump: 2024-09-16T14:50:15.389 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:15.389 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.389 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.389 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.389 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.389 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:15.389 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.389 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.389 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.389 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:15.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5418012bd0 con 0x7f5420132090 2024-09-16T14:50:15.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:15.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f541d59a640 4 set_mon_vals no callback set 2024-09-16T14:50:15.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:15.384+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f54180110f0 con 0x7f5420132090 2024-09-16T14:50:15.394 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T14:50:15.394 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:50:15.395 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:50:15.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb08fbf640 10 monclient: get_monmap_and_config 2024-09-16T14:50:15.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb08fbf640 10 monclient: build_initial_monmap 2024-09-16T14:50:15.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb08fbf640 10 monclient: monmap: 2024-09-16T14:50:15.510 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:15.510 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.510 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:15.510562+0000 2024-09-16T14:50:15.510 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:15.510562+0000 2024-09-16T14:50:15.510 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:15.510 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.510 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:15.510 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:15.510 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:15.510 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb08fbf640 1 Processor -- start 2024-09-16T14:50:15.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb08fbf640 1 -- start start 2024-09-16T14:50:15.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb08fbf640 10 monclient: init 2024-09-16T14:50:15.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb08fbf640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:15.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb08fbf640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:15.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb08fbf640 10 monclient(hunting): picked mon.noname-c con 0x7fbb0409e7a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:15.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb08fbf640 10 monclient(hunting): picked mon.noname-b con 0x7fbb0409cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:15.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb08fbf640 10 monclient(hunting): picked mon.noname-a con 0x7fbb0409d210 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:15.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb08fbf640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0409aff0 con 0x7fbb0409d210 2024-09-16T14:50:15.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb08fbf640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0409b2d0 con 0x7fbb0409e7a0 2024-09-16T14:50:15.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb08fbf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0409b5b0 con 0x7fbb0409cdc0 2024-09-16T14:50:15.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb08fbf640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:15.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb08fbf640 10 monclient(hunting): authenticate will time out at 1439.631433s 2024-09-16T14:50:15.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb0259c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fbb0409e7a0 legacy=0x7fbb0409da50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:15.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbafbfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbb0409cdc0 0x7fbb041370a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57744/0 (socket says 172.21.15.26:57744) 2024-09-16T14:50:15.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbafbfff640 1 -- 172.21.15.26:0/368454379 learned_addr learned my addr 172.21.15.26:0/368454379 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:15.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb01d9b640 1 -- 172.21.15.26:0/368454379 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522479146 0 0) 0x7fbb0409b5b0 con 0x7fbb0409cdc0 2024-09-16T14:50:15.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb01d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb01d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb01d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb01d9b640 10 monclient(hunting): my global_id is 154643 2024-09-16T14:50:15.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.505+0000 7fbb01d9b640 1 -- 172.21.15.26:0/368454379 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaf0003530 con 0x7fbb0409cdc0 2024-09-16T14:50:15.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 1 -- 172.21.15.26:0/368454379 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1774872209 0 0) 0x7fbaf0003530 con 0x7fbb0409cdc0 2024-09-16T14:50:15.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 1 -- 172.21.15.26:0/368454379 >> v1:172.21.15.189:6789/0 conn(0x7fbb0409e7a0 legacy=0x7fbb0409da50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:15.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 1 -- 172.21.15.26:0/368454379 >> v1:172.21.15.26:6789/0 conn(0x7fbb0409d210 legacy=0x7fbb0413a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:50:15.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 1 monclient: found mon.noname-b 2024-09-16T14:50:15.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:15.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:15.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 1 -- 172.21.15.26:0/368454379 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0409b890 con 0x7fbb0409cdc0 2024-09-16T14:50:15.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 10 monclient: _finish_auth 0 2024-09-16T14:50:15.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:50:15.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:15.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 1 -- 172.21.15.26:0/368454379 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbaec001620 con 0x7fbb0409cdc0 2024-09-16T14:50:15.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:15.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 10 monclient: dump: 2024-09-16T14:50:15.516 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.516 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.516 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.516 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.516 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.516 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.517 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 5 monclient: authenticate success, global_id 154643 2024-09-16T14:50:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 1 -- 172.21.15.26:0/368454379 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbaec001940 con 0x7fbb0409cdc0 2024-09-16T14:50:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 1 -- 172.21.15.26:0/368454379 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbaec001cd0 con 0x7fbb0409cdc0 2024-09-16T14:50:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb01d9b640 10 monclient: dump: 2024-09-16T14:50:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.518 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 4 set_mon_vals no callback set 2024-09-16T14:50:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 10 monclient: shutdown 2024-09-16T14:50:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 1 -- 172.21.15.26:0/368454379 >> v1:172.21.15.26:6790/0 conn(0x7fbb0409cdc0 legacy=0x7fbb041370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 1 -- 172.21.15.26:0/368454379 shutdown_connections 2024-09-16T14:50:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 1 -- 172.21.15.26:0/368454379 >> 172.21.15.26:0/368454379 conn(0x7fbb0409fb50 msgr2=0x7fbb0409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 1 -- 172.21.15.26:0/368454379 shutdown_connections 2024-09-16T14:50:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 1 -- 172.21.15.26:0/368454379 wait complete. 2024-09-16T14:50:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 10 monclient: build_initial_monmap 2024-09-16T14:50:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 10 monclient: monmap: 2024-09-16T14:50:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:15.520 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 1 Processor -- start 2024-09-16T14:50:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 1 -- start start 2024-09-16T14:50:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 10 monclient: init 2024-09-16T14:50:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 10 monclient(hunting): picked mon.noname-b con 0x7fbb0409cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 10 monclient(hunting): picked mon.noname-a con 0x7fbb0409d210 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 10 monclient(hunting): picked mon.noname-c con 0x7fbb0409e7a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb041c66c0 con 0x7fbb0409d210 2024-09-16T14:50:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb040a2210 con 0x7fbb0409cdc0 2024-09-16T14:50:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0412d690 con 0x7fbb0409e7a0 2024-09-16T14:50:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 10 monclient(hunting): authenticate will time out at 1439.635023s 2024-09-16T14:50:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb0259c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fbb0409cdc0 legacy=0x7fbb0412cf80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb02d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbb0409e7a0 0x7fbb041d1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57746/0 (socket says 172.21.15.26:57746) 2024-09-16T14:50:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb02d9d640 1 -- 172.21.15.26:0/2153626660 learned_addr learned my addr 172.21.15.26:0/2153626660 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 1 -- 172.21.15.26:0/2153626660 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648542886 0 0) 0x7fbb0412d690 con 0x7fbb0409e7a0 2024-09-16T14:50:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 10 monclient(hunting): my global_id is 154646 2024-09-16T14:50:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 1 -- 172.21.15.26:0/2153626660 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae4003530 con 0x7fbb0409e7a0 2024-09-16T14:50:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 1 -- 172.21.15.26:0/2153626660 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 56539966 0 0) 0x7fbae4003530 con 0x7fbb0409e7a0 2024-09-16T14:50:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 1 -- 172.21.15.26:0/2153626660 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb0412d690 con 0x7fbb0409e7a0 2024-09-16T14:50:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 1 -- 172.21.15.26:0/2153626660 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbafc001c20 con 0x7fbb0409e7a0 2024-09-16T14:50:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 10 monclient(hunting): dump: 2024-09-16T14:50:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.527 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 1 -- 172.21.15.26:0/2153626660 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4077168540 0 0) 0x7fbb0412d690 con 0x7fbb0409e7a0 2024-09-16T14:50:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 1 -- 172.21.15.26:0/2153626660 >> v1:172.21.15.189:6789/0 conn(0x7fbb0409cdc0 legacy=0x7fbb0412cf80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 1 -- 172.21.15.26:0/2153626660 >> v1:172.21.15.26:6789/0 conn(0x7fbb0409d210 legacy=0x7fbb041ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 1 monclient: found mon.c 2024-09-16T14:50:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 1 -- 172.21.15.26:0/2153626660 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb041219b0 con 0x7fbb0409e7a0 2024-09-16T14:50:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:15.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:15.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 5 monclient: authenticate success, global_id 154646 2024-09-16T14:50:15.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 10 monclient: _renew_subs 2024-09-16T14:50:15.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:15.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 1 -- 172.21.15.26:0/2153626660 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0409a240 con 0x7fbb0409e7a0 2024-09-16T14:50:15.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbafaffd640 1 -- 172.21.15.26:0/2153626660 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbafc001ff0 con 0x7fbb0409e7a0 2024-09-16T14:50:15.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 10 monclient: _renew_subs 2024-09-16T14:50:15.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:15.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.509+0000 7fbb08fbf640 1 -- 172.21.15.26:0/2153626660 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb041213f0 con 0x7fbb0409e7a0 2024-09-16T14:50:15.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.513+0000 7fbafaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:15.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.513+0000 7fbafaffd640 1 -- 172.21.15.26:0/2153626660 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbafc003df0 con 0x7fbb0409e7a0 2024-09-16T14:50:15.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.513+0000 7fbafaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:15.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.513+0000 7fbafaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:15.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.513+0000 7fbafaffd640 10 monclient: dump: 2024-09-16T14:50:15.529 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:15.529 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:15.529 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.529 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:15.529 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:15.529 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:15.529 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:15.529 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:15.529 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:15.529 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:15.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.513+0000 7fbb00d99640 4 set_mon_vals no callback set 2024-09-16T14:50:15.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.513+0000 7fbb08fbf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:15.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.513+0000 7fbb08fbf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:15.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.513+0000 7fbb08fbf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:15.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.513+0000 7fbb08fbf640 1 -- 172.21.15.26:0/2153626660 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae8005050 con 0x7fbb0409e7a0 2024-09-16T14:50:15.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.513+0000 7fbafaffd640 1 -- 172.21.15.26:0/2153626660 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbafc003fd0 con 0x7fbb0409e7a0 2024-09-16T14:50:15.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.513+0000 7fbafaffd640 1 -- 172.21.15.26:0/2153626660 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(223..223 src has 169..223) v4 ==== 12392+0+0 (unknown 467405984 0 0) 0x7fbafc045710 con 0x7fbb0409e7a0 2024-09-16T14:50:15.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.513+0000 7fbb0259c640 10 monclient: get_auth_request con 0x7fbae4036050 auth_method 0 2024-09-16T14:50:15.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.517+0000 7fbafaffd640 1 -- 172.21.15.26:0/2153626660 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbafc014a90 con 0x7fbb0409e7a0 2024-09-16T14:50:15.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.517+0000 7fbafaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:15.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.517+0000 7fbafaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:15.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.737+0000 7fbb08fbf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:50:15.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.737+0000 7fbb08fbf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:15.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.737+0000 7fbb08fbf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:15.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.737+0000 7fbb08fbf640 1 -- 172.21.15.26:0/2153626660 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbae8005880 con 0x7fbb0409e7a0 2024-09-16T14:50:15.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.737+0000 7fbafaffd640 1 -- 172.21.15.26:0/2153626660 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 472226860) 0x7fbafc012f00 con 0x7fbb0409e7a0 2024-09-16T14:50:15.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.737+0000 7fbafaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:15.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.737+0000 7fbafaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:15.744 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:50:15.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.741+0000 7fbb08fbf640 1 -- 172.21.15.26:0/2153626660 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbae4036050 legacy=0x7fbae4038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.741+0000 7fbb08fbf640 10 monclient: shutdown 2024-09-16T14:50:15.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.741+0000 7fbb08fbf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:15.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.741+0000 7fbb08fbf640 1 -- 172.21.15.26:0/2153626660 >> v1:172.21.15.26:6790/0 conn(0x7fbb0409e7a0 legacy=0x7fbb041d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:15.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.741+0000 7fbb08fbf640 1 -- 172.21.15.26:0/2153626660 shutdown_connections 2024-09-16T14:50:15.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.741+0000 7fbb08fbf640 1 -- 172.21.15.26:0/2153626660 >> 172.21.15.26:0/2153626660 conn(0x7fbb0409fb50 msgr2=0x7fbb040a0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:15.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.741+0000 7fbb08fbf640 1 -- 172.21.15.26:0/2153626660 shutdown_connections 2024-09-16T14:50:15.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:15.741+0000 7fbb08fbf640 1 -- 172.21.15.26:0/2153626660 wait complete. 2024-09-16T14:50:17.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:17.684+0000 7f54167fc640 10 monclient: tick 2024-09-16T14:50:17.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:17.684+0000 7f54167fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:17.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:17.684+0000 7f54167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:17.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:17.684+0000 7f54167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:18.684+0000 7f54167fc640 10 monclient: tick 2024-09-16T14:50:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:18.684+0000 7f54167fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:18.684+0000 7f54167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:18.762 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:50:18.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 10 monclient: get_monmap_and_config 2024-09-16T14:50:18.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 10 monclient: build_initial_monmap 2024-09-16T14:50:18.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 10 monclient: monmap: 2024-09-16T14:50:18.894 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:18.894 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:18.894 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:18.895030+0000 2024-09-16T14:50:18.894 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:18.895030+0000 2024-09-16T14:50:18.894 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:18.894 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:18.894 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:18.895 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:18.895 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:18.895 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:18.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 1 Processor -- start 2024-09-16T14:50:18.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 1 -- start start 2024-09-16T14:50:18.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 10 monclient: init 2024-09-16T14:50:18.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:18.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:18.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 10 monclient(hunting): picked mon.noname-c con 0x7f437409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:18.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 10 monclient(hunting): picked mon.noname-a con 0x7f4374132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:18.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 10 monclient(hunting): picked mon.noname-b con 0x7f4374138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:18.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437409d260 con 0x7f4374132650 2024-09-16T14:50:18.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4374132a80 con 0x7f437409cb80 2024-09-16T14:50:18.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437409cf60 con 0x7f4374138ff0 2024-09-16T14:50:18.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:18.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 10 monclient(hunting): authenticate will time out at 1443.015114s 2024-09-16T14:50:18.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437a3b2640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f437409cb80 legacy=0x7f43740717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:18.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437abb3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4374138ff0 0x7f437413b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51346/0 (socket says 172.21.15.26:51346) 2024-09-16T14:50:18.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437abb3640 1 -- 172.21.15.26:0/1917003765 learned_addr learned my addr 172.21.15.26:0/1917003765 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:18.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 1 -- 172.21.15.26:0/1917003765 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793094333 0 0) 0x7f437409cf60 con 0x7f4374138ff0 2024-09-16T14:50:18.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 10 monclient(hunting): _negotiate 2024-09-16T14:50:18.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:18.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:18.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 10 monclient(hunting): my global_id is 154649 2024-09-16T14:50:18.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 1 -- 172.21.15.26:0/1917003765 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c003530 con 0x7f4374138ff0 2024-09-16T14:50:18.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 1 -- 172.21.15.26:0/1917003765 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2534986656 0 0) 0x7f435c003530 con 0x7f4374138ff0 2024-09-16T14:50:18.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 1 -- 172.21.15.26:0/1917003765 >> v1:172.21.15.189:6789/0 conn(0x7f437409cb80 legacy=0x7f43740717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:18.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 1 -- 172.21.15.26:0/1917003765 >> v1:172.21.15.26:6789/0 conn(0x7f4374132650 legacy=0x7f43741378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:18.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 10 monclient: _finish_hunting 0 2024-09-16T14:50:18.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 1 monclient: found mon.noname-b 2024-09-16T14:50:18.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:18.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:18.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 1 -- 172.21.15.26:0/1917003765 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437409f2d0 con 0x7f4374138ff0 2024-09-16T14:50:18.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 10 monclient: _finish_auth 0 2024-09-16T14:50:18.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 10 monclient: _check_auth_tickets 2024-09-16T14:50:18.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:18.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f437c616640 5 monclient: authenticate success, global_id 154649 2024-09-16T14:50:18.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 1 -- 172.21.15.26:0/1917003765 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4364001660 con 0x7f4374138ff0 2024-09-16T14:50:18.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.889+0000 7f43793b0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:18.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f43793b0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:18.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f43793b0640 10 monclient: dump: 2024-09-16T14:50:18.904 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:18.904 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:18.904 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:18.904 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:18.904 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:18.904 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:18.905 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:18.905 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:18.905 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:18.905 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:18.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:18.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f43793b0640 1 -- 172.21.15.26:0/1917003765 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4364001880 con 0x7f4374138ff0 2024-09-16T14:50:18.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f43793b0640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:18.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f43793b0640 1 -- 172.21.15.26:0/1917003765 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4364001c10 con 0x7f4374138ff0 2024-09-16T14:50:18.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f43793b0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:18.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f43793b0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:18.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f43793b0640 10 monclient: dump: 2024-09-16T14:50:18.905 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:18.905 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:18.906 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:18.906 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:18.906 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:18.906 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:18.906 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:18.906 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:18.906 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:18.906 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:18.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:18.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 4 set_mon_vals no callback set 2024-09-16T14:50:18.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 10 monclient: shutdown 2024-09-16T14:50:18.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:18.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 1 -- 172.21.15.26:0/1917003765 >> v1:172.21.15.26:6790/0 conn(0x7f4374138ff0 legacy=0x7f437413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:18.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 1 -- 172.21.15.26:0/1917003765 shutdown_connections 2024-09-16T14:50:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 1 -- 172.21.15.26:0/1917003765 >> 172.21.15.26:0/1917003765 conn(0x7f437409fb50 msgr2=0x7f437409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 1 -- 172.21.15.26:0/1917003765 shutdown_connections 2024-09-16T14:50:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 1 -- 172.21.15.26:0/1917003765 wait complete. 2024-09-16T14:50:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 10 monclient: build_initial_monmap 2024-09-16T14:50:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 10 monclient: monmap: 2024-09-16T14:50:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:18.907 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:18.908 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:18.908 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:18.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 1 Processor -- start 2024-09-16T14:50:18.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 1 -- start start 2024-09-16T14:50:18.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 10 monclient: init 2024-09-16T14:50:18.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:18.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:18.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 10 monclient(hunting): picked mon.noname-c con 0x7f437409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:18.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 10 monclient(hunting): picked mon.noname-a con 0x7f4374132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:18.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 10 monclient(hunting): picked mon.noname-b con 0x7f4374138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:18.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437412ec90 con 0x7f4374132650 2024-09-16T14:50:18.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43740719a0 con 0x7f4374138ff0 2024-09-16T14:50:18.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43740a28b0 con 0x7f437409cb80 2024-09-16T14:50:18.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:18.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 10 monclient(hunting): authenticate will time out at 1443.018438s 2024-09-16T14:50:18.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437a3b2640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f437409cb80 0x7f437412de70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51348/0 (socket says 172.21.15.26:51348) 2024-09-16T14:50:18.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437a3b2640 1 -- 172.21.15.26:0/2079375843 learned_addr learned my addr 172.21.15.26:0/2079375843 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:18.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4379bb1640 1 --1- 172.21.15.26:0/2079375843 >> v1:172.21.15.26:6789/0 conn(0x7f4374132650 0x7f437412e580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:49286/0 (socket says 172.21.15.26:49286) 2024-09-16T14:50:18.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437abb3640 1 -- 172.21.15.26:0/2079375843 >> v1:172.21.15.189:6789/0 conn(0x7f4374138ff0 legacy=0x7f43741d1b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:18.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 1 -- 172.21.15.26:0/2079375843 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175256917 0 0) 0x7f43740a28b0 con 0x7f437409cb80 2024-09-16T14:50:18.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:18.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:18.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:18.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 10 monclient(hunting): my global_id is 154652 2024-09-16T14:50:18.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 1 -- 172.21.15.26:0/2079375843 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4360003530 con 0x7f437409cb80 2024-09-16T14:50:18.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 1 -- 172.21.15.26:0/2079375843 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1898609302 0 0) 0x7f4360003530 con 0x7f437409cb80 2024-09-16T14:50:18.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 1 -- 172.21.15.26:0/2079375843 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43740a28b0 con 0x7f437409cb80 2024-09-16T14:50:18.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 1 -- 172.21.15.26:0/2079375843 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f436c004560 con 0x7f437409cb80 2024-09-16T14:50:18.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:18.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:18.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:18.910 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:18.910 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:18.910 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:18.910 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:18.910 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:18.910 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:18.910 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:18.910 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:18.911 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 1 -- 172.21.15.26:0/2079375843 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4182160898 0 0) 0x7f43740a28b0 con 0x7f437409cb80 2024-09-16T14:50:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 1 -- 172.21.15.26:0/2079375843 >> v1:172.21.15.189:6789/0 conn(0x7f4374138ff0 legacy=0x7f43741d1b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 1 -- 172.21.15.26:0/2079375843 >> v1:172.21.15.26:6789/0 conn(0x7f4374132650 legacy=0x7f437412e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 1 monclient: found mon.c 2024-09-16T14:50:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 1 -- 172.21.15.26:0/2079375843 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43741354b0 con 0x7f437409cb80 2024-09-16T14:50:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:18.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 5 monclient: authenticate success, global_id 154652 2024-09-16T14:50:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 10 monclient: _renew_subs 2024-09-16T14:50:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 1 -- 172.21.15.26:0/2079375843 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4374133aa0 con 0x7f437409cb80 2024-09-16T14:50:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 10 monclient: _renew_subs 2024-09-16T14:50:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f437c616640 1 -- 172.21.15.26:0/2079375843 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4374133f40 con 0x7f437409cb80 2024-09-16T14:50:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 1 -- 172.21.15.26:0/2079375843 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f436c004780 con 0x7f437409cb80 2024-09-16T14:50:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 1 -- 172.21.15.26:0/2079375843 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f436c004b10 con 0x7f437409cb80 2024-09-16T14:50:18.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4372ffd640 10 monclient: dump: 2024-09-16T14:50:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:18.913 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:18.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.893+0000 7f4378baf640 4 set_mon_vals no callback set 2024-09-16T14:50:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.897+0000 7f437c616640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.897+0000 7f437c616640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.897+0000 7f437c616640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.897+0000 7f437c616640 1 -- 172.21.15.26:0/2079375843 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4354005050 con 0x7f437409cb80 2024-09-16T14:50:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.897+0000 7f4372ffd640 1 -- 172.21.15.26:0/2079375843 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f436c004c70 con 0x7f437409cb80 2024-09-16T14:50:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.897+0000 7f4372ffd640 1 -- 172.21.15.26:0/2079375843 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(223..223 src has 169..223) v4 ==== 12392+0+0 (unknown 467405984 0 0) 0x7f436c045750 con 0x7f437409cb80 2024-09-16T14:50:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.897+0000 7f4379bb1640 10 monclient: get_auth_request con 0x7f43600360f0 auth_method 0 2024-09-16T14:50:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.901+0000 7f4372ffd640 1 -- 172.21.15.26:0/2079375843 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f436c005440 con 0x7f437409cb80 2024-09-16T14:50:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.901+0000 7f4372ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:18.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:18.901+0000 7f4372ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:19.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:19.121+0000 7f437c616640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:50:19.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:19.121+0000 7f437c616640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:19.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:19.121+0000 7f437c616640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:19.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:19.121+0000 7f437c616640 1 -- 172.21.15.26:0/2079375843 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4354005880 con 0x7f437409cb80 2024-09-16T14:50:19.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:19.121+0000 7f4372ffd640 1 -- 172.21.15.26:0/2079375843 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1288262157) 0x7f436c012ef0 con 0x7f437409cb80 2024-09-16T14:50:19.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:19.121+0000 7f4372ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:19.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:19.121+0000 7f4372ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:19.127 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:50:19.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:19.125+0000 7f437c616640 1 -- 172.21.15.26:0/2079375843 >> v1:172.21.15.189:6812/3196161582 conn(0x7f43600360f0 legacy=0x7f43600385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:19.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:19.125+0000 7f437c616640 10 monclient: shutdown 2024-09-16T14:50:19.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:19.125+0000 7f437c616640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:19.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:19.125+0000 7f437c616640 1 -- 172.21.15.26:0/2079375843 >> v1:172.21.15.26:6790/0 conn(0x7f437409cb80 legacy=0x7f437412de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:19.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:19.125+0000 7f437c616640 1 -- 172.21.15.26:0/2079375843 shutdown_connections 2024-09-16T14:50:19.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:19.125+0000 7f437c616640 1 -- 172.21.15.26:0/2079375843 >> 172.21.15.26:0/2079375843 conn(0x7f437409fb50 msgr2=0x7f4374070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:19.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:19.125+0000 7f437c616640 1 -- 172.21.15.26:0/2079375843 shutdown_connections 2024-09-16T14:50:19.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:19.125+0000 7f437c616640 1 -- 172.21.15.26:0/2079375843 wait complete. 2024-09-16T14:50:19.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:19.684+0000 7f54167fc640 10 monclient: tick 2024-09-16T14:50:19.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:19.684+0000 7f54167fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:19.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:19.684+0000 7f54167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:20.688+0000 7f54167fc640 10 monclient: tick 2024-09-16T14:50:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:20.688+0000 7f54167fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:20.688+0000 7f54167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:21.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:21.688+0000 7f54167fc640 10 monclient: tick 2024-09-16T14:50:21.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:21.688+0000 7f54167fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:21.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:21.688+0000 7f54167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:22.146 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:50:22.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.273+0000 7febb52a1640 10 monclient: get_monmap_and_config 2024-09-16T14:50:22.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.273+0000 7febb52a1640 10 monclient: build_initial_monmap 2024-09-16T14:50:22.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.273+0000 7febb52a1640 10 monclient: monmap: 2024-09-16T14:50:22.279 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:22.279 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:22.279 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:22.279969+0000 2024-09-16T14:50:22.279 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:22.279969+0000 2024-09-16T14:50:22.279 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:22.279 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:22.279 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:22.280 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:22.280 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:22.280 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:22.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.273+0000 7febb52a1640 1 Processor -- start 2024-09-16T14:50:22.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.273+0000 7febb52a1640 1 -- start start 2024-09-16T14:50:22.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.273+0000 7febb52a1640 10 monclient: init 2024-09-16T14:50:22.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.277+0000 7febb52a1640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:22.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.277+0000 7febb52a1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:22.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.277+0000 7febb52a1640 10 monclient(hunting): picked mon.noname-b con 0x7febb009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:22.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.277+0000 7febb52a1640 10 monclient(hunting): picked mon.noname-a con 0x7febb0132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:22.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.277+0000 7febb52a1640 10 monclient(hunting): picked mon.noname-c con 0x7febb0138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:22.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.277+0000 7febb52a1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb009d260 con 0x7febb0132650 2024-09-16T14:50:22.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.277+0000 7febb52a1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb0132a80 con 0x7febb0138ff0 2024-09-16T14:50:22.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.277+0000 7febb52a1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb009cf60 con 0x7febb009cb80 2024-09-16T14:50:22.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.277+0000 7febb52a1640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:22.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.277+0000 7febb52a1640 10 monclient(hunting): authenticate will time out at 1446.400394s 2024-09-16T14:50:22.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.277+0000 7febae7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7febb0132650 0x7febb01378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:49296/0 (socket says 172.21.15.26:49296) 2024-09-16T14:50:22.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.277+0000 7febaeffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7febb009cb80 0x7febb00717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51354/0 (socket says 172.21.15.26:51354) 2024-09-16T14:50:22.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.277+0000 7febae7fc640 1 -- 172.21.15.26:0/3935997993 learned_addr learned my addr 172.21.15.26:0/3935997993 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:22.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.277+0000 7febaf7fe640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.189:6789/0 conn(0x7febb0138ff0 legacy=0x7febb013b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:22.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.481+0000 7febaf7fe640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.189:6789/0 conn(0x7febb0138ff0 legacy=0x7febb013b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:22.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:22.684+0000 7f54167fc640 10 monclient: tick 2024-09-16T14:50:22.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:22.684+0000 7f54167fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:22.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:22.684+0000 7f54167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:22.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:22.885+0000 7febaf7fe640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.189:6789/0 conn(0x7febb0138ff0 legacy=0x7febb013b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:23.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:23.684+0000 7f54167fc640 10 monclient: tick 2024-09-16T14:50:23.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:23.684+0000 7f54167fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:23.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:23.684+0000 7f54167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:23.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:23.689+0000 7febaf7fe640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.189:6789/0 conn(0x7febb0138ff0 legacy=0x7febb013b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:24.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:24.684+0000 7f54167fc640 10 monclient: tick 2024-09-16T14:50:24.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:24.684+0000 7f54167fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:24.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:24.684+0000 7f54167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:25.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.277+0000 7febacff9640 10 monclient(hunting): tick 2024-09-16T14:50:25.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.277+0000 7febacff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:50:25.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.277+0000 7febacff9640 1 monclient(hunting): continuing hunt 2024-09-16T14:50:25.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.277+0000 7febacff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:50:25.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.277+0000 7febacff9640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.26:6790/0 conn(0x7febb009cb80 legacy=0x7febb00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:25.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.277+0000 7febacff9640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.189:6789/0 conn(0x7febb0138ff0 legacy=0x7febb013b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:25.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.277+0000 7febacff9640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.26:6789/0 conn(0x7febb0132650 legacy=0x7febb01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:25.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.277+0000 7febacff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:25.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.277+0000 7febacff9640 10 monclient(hunting): picked mon.noname-c con 0x7feb900036d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:25.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.277+0000 7febacff9640 10 monclient(hunting): picked mon.noname-a con 0x7feb90007360 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:25.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.277+0000 7febacff9640 10 monclient(hunting): picked mon.noname-b con 0x7febb009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:25.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.277+0000 7febacff9640 1 -- 172.21.15.26:0/3935997993 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb009f2d0 con 0x7feb90007360 2024-09-16T14:50:25.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.277+0000 7febacff9640 1 -- 172.21.15.26:0/3935997993 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb0132a80 con 0x7feb900036d0 2024-09-16T14:50:25.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.277+0000 7febacff9640 1 -- 172.21.15.26:0/3935997993 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb900108b0 con 0x7febb009cb80 2024-09-16T14:50:25.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.277+0000 7febaeffd640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.189:6789/0 conn(0x7feb900036d0 legacy=0x7feb90005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:25.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.277+0000 7febacff9640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:25.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.481+0000 7febaeffd640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.189:6789/0 conn(0x7feb900036d0 legacy=0x7feb90005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:25.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:25.684+0000 7f54167fc640 10 monclient: tick 2024-09-16T14:50:25.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:25.684+0000 7f54167fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:25.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:25.684+0000 7f54167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:25.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:25.885+0000 7febaeffd640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.189:6789/0 conn(0x7feb900036d0 legacy=0x7feb90005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:26.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:26.688+0000 7f54167fc640 10 monclient: tick 2024-09-16T14:50:26.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:26.688+0000 7f54167fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:26.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:26.688+0000 7f54167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:26.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:26.689+0000 7febaeffd640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.189:6789/0 conn(0x7feb900036d0 legacy=0x7feb90005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:27.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:27.688+0000 7f54167fc640 10 monclient: tick 2024-09-16T14:50:27.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:27.688+0000 7f54167fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:27.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:27.688+0000 7f54167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:27.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:27.688+0000 7f54167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:28.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.281+0000 7febacff9640 10 monclient(hunting): tick 2024-09-16T14:50:28.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.281+0000 7febacff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:50:28.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.281+0000 7febacff9640 1 monclient(hunting): continuing hunt 2024-09-16T14:50:28.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.281+0000 7febacff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:50:28.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.281+0000 7febacff9640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.26:6790/0 conn(0x7febb009cb80 legacy=0x7feb9000d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:28.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.281+0000 7febacff9640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.189:6789/0 conn(0x7feb900036d0 legacy=0x7feb90005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:28.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.281+0000 7febacff9640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.26:6789/0 conn(0x7feb90007360 legacy=0x7feb900097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:28.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.281+0000 7febacff9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:28.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.281+0000 7febacff9640 10 monclient(hunting): picked mon.noname-a con 0x7feb90010b40 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:28.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.281+0000 7febacff9640 10 monclient(hunting): picked mon.noname-b con 0x7feb90012e20 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:28.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.281+0000 7febacff9640 10 monclient(hunting): picked mon.noname-c con 0x7feb900036d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:28.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.281+0000 7febacff9640 1 -- 172.21.15.26:0/3935997993 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb90011b20 con 0x7feb90010b40 2024-09-16T14:50:28.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.281+0000 7febacff9640 1 -- 172.21.15.26:0/3935997993 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb0132a80 con 0x7feb900036d0 2024-09-16T14:50:28.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.281+0000 7febacff9640 1 -- 172.21.15.26:0/3935997993 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb90003b10 con 0x7feb90012e20 2024-09-16T14:50:28.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.281+0000 7febacff9640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:28.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.281+0000 7febaf7fe640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.189:6789/0 conn(0x7feb900036d0 legacy=0x7feb900097f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:28.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.485+0000 7febaf7fe640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.189:6789/0 conn(0x7feb900036d0 legacy=0x7feb900097f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:28.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:28.688+0000 7f54167fc640 10 monclient: tick 2024-09-16T14:50:28.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:28.688+0000 7f54167fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:28.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:28.688+0000 7f54167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:28.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:28.889+0000 7febaf7fe640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.189:6789/0 conn(0x7feb900036d0 legacy=0x7feb900097f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:29.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 1 -- 172.21.15.26:0/3935997993 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648061961 0 0) 0x7feb90011b20 con 0x7feb90010b40 2024-09-16T14:50:29.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:29.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:29.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:29.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 10 monclient(hunting): my global_id is 164391 2024-09-16T14:50:29.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 1 -- 172.21.15.26:0/3935997993 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb9c003530 con 0x7feb90010b40 2024-09-16T14:50:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 1 -- 172.21.15.26:0/3935997993 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1766539580 0 0) 0x7feb9c003530 con 0x7feb90010b40 2024-09-16T14:50:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.26:6790/0 conn(0x7feb90012e20 legacy=0x7feb90015210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.189:6789/0 conn(0x7feb900036d0 legacy=0x7feb900097f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 10 monclient: _finish_hunting 0 2024-09-16T14:50:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 1 monclient: found mon.noname-a 2024-09-16T14:50:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 1 -- 172.21.15.26:0/3935997993 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb90004cb0 con 0x7feb90010b40 2024-09-16T14:50:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 10 monclient: _finish_auth 0 2024-09-16T14:50:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 1 -- 172.21.15.26:0/3935997993 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feba8001a10 con 0x7feb90010b40 2024-09-16T14:50:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 10 monclient: dump: 2024-09-16T14:50:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:29.500 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febb52a1640 5 monclient: authenticate success, global_id 164391 2024-09-16T14:50:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febb52a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 1 -- 172.21.15.26:0/3935997993 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feba8001c00 con 0x7feb90010b40 2024-09-16T14:50:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 1 -- 172.21.15.26:0/3935997993 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feba8001fd0 con 0x7feb90010b40 2024-09-16T14:50:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febb52a1640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febb52a1640 4 set_mon_vals no callback set 2024-09-16T14:50:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febadffb640 10 monclient: dump: 2024-09-16T14:50:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:29.501 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:29.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febb52a1640 10 monclient: shutdown 2024-09-16T14:50:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febb52a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febb52a1640 1 -- 172.21.15.26:0/3935997993 >> v1:172.21.15.26:6789/0 conn(0x7feb90010b40 legacy=0x7feb90010f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febb52a1640 1 -- 172.21.15.26:0/3935997993 shutdown_connections 2024-09-16T14:50:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febb52a1640 1 -- 172.21.15.26:0/3935997993 >> 172.21.15.26:0/3935997993 conn(0x7febb009fb50 msgr2=0x7febb009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febb52a1640 1 -- 172.21.15.26:0/3935997993 shutdown_connections 2024-09-16T14:50:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febb52a1640 1 -- 172.21.15.26:0/3935997993 wait complete. 2024-09-16T14:50:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febb52a1640 10 monclient: build_initial_monmap 2024-09-16T14:50:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.493+0000 7febb52a1640 10 monclient: monmap: 2024-09-16T14:50:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:29.506 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 1 Processor -- start 2024-09-16T14:50:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 1 -- start start 2024-09-16T14:50:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 10 monclient: init 2024-09-16T14:50:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 10 monclient(hunting): picked mon.noname-b con 0x7feb900036d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 10 monclient(hunting): picked mon.noname-a con 0x7feb90010b40 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 10 monclient(hunting): picked mon.noname-c con 0x7feb90012e20 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb01371e0 con 0x7feb90010b40 2024-09-16T14:50:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb00714b0 con 0x7feb900036d0 2024-09-16T14:50:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb00a2140 con 0x7feb90012e20 2024-09-16T14:50:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 10 monclient(hunting): authenticate will time out at 1453.621029s 2024-09-16T14:50:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febaeffd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7feb900036d0 legacy=0x7febb012bd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febaf7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7feb90012e20 0x7febb01d20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49792/0 (socket says 172.21.15.26:49792) 2024-09-16T14:50:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febaf7fe640 1 -- 172.21.15.26:0/1272395382 learned_addr learned my addr 172.21.15.26:0/1272395382 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 1 -- 172.21.15.26:0/1272395382 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428553964 0 0) 0x7febb01371e0 con 0x7feb90010b40 2024-09-16T14:50:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 10 monclient(hunting): my global_id is 164394 2024-09-16T14:50:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 1 -- 172.21.15.26:0/1272395382 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb90006040 con 0x7feb90010b40 2024-09-16T14:50:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 1 -- 172.21.15.26:0/1272395382 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2422601189 0 0) 0x7feb90006040 con 0x7feb90010b40 2024-09-16T14:50:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 1 -- 172.21.15.26:0/1272395382 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febb01371e0 con 0x7feb90010b40 2024-09-16T14:50:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 1 -- 172.21.15.26:0/1272395382 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb980016b0 con 0x7feb90010b40 2024-09-16T14:50:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 10 monclient(hunting): dump: 2024-09-16T14:50:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:29.509 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:29.509 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:29.509 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:29.509 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:29.509 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:29.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:29.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 1 -- 172.21.15.26:0/1272395382 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1589281412 0 0) 0x7febb01371e0 con 0x7feb90010b40 2024-09-16T14:50:29.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 1 -- 172.21.15.26:0/1272395382 >> v1:172.21.15.26:6790/0 conn(0x7feb90012e20 legacy=0x7febb01d20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:29.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 1 -- 172.21.15.26:0/1272395382 >> v1:172.21.15.189:6789/0 conn(0x7feb900036d0 legacy=0x7febb012bd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:29.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:29.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 1 monclient: found mon.a 2024-09-16T14:50:29.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:29.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:29.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 1 -- 172.21.15.26:0/1272395382 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb0138ae0 con 0x7feb90010b40 2024-09-16T14:50:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 5 monclient: authenticate success, global_id 164394 2024-09-16T14:50:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 10 monclient: _renew_subs 2024-09-16T14:50:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 1 -- 172.21.15.26:0/1272395382 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb012c490 con 0x7feb90010b40 2024-09-16T14:50:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 10 monclient: _renew_subs 2024-09-16T14:50:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 1 -- 172.21.15.26:0/1272395382 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febb01ca450 con 0x7feb90010b40 2024-09-16T14:50:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 1 -- 172.21.15.26:0/1272395382 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb98001c40 con 0x7feb90010b40 2024-09-16T14:50:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 1 -- 172.21.15.26:0/1272395382 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb98004990 con 0x7feb90010b40 2024-09-16T14:50:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 10 monclient: dump: 2024-09-16T14:50:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:29.511 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febad7fa640 4 set_mon_vals no callback set 2024-09-16T14:50:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 1 -- 172.21.15.26:0/1272395382 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7feb98004af0 con 0x7feb90010b40 2024-09-16T14:50:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7febb52a1640 1 -- 172.21.15.26:0/1272395382 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb94005050 con 0x7feb90010b40 2024-09-16T14:50:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.497+0000 7feba77fe640 1 -- 172.21.15.26:0/1272395382 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(223..223 src has 169..223) v4 ==== 12392+0+0 (unknown 467405984 0 0) 0x7feb98045140 con 0x7feb90010b40 2024-09-16T14:50:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.501+0000 7febaeffd640 10 monclient: get_auth_request con 0x7feb90037ae0 auth_method 0 2024-09-16T14:50:29.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.505+0000 7feba77fe640 1 -- 172.21.15.26:0/1272395382 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7feb980141e0 con 0x7feb90010b40 2024-09-16T14:50:29.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.505+0000 7feba77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:29.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.505+0000 7feba77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:29.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:29.688+0000 7f54167fc640 10 monclient: tick 2024-09-16T14:50:29.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:29.688+0000 7f54167fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:29.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:29.688+0000 7f54167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:29.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.741+0000 7febb52a1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:50:29.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.741+0000 7febb52a1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:29.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.741+0000 7febb52a1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:29.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.741+0000 7febb52a1640 1 -- 172.21.15.26:0/1272395382 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7feb94005880 con 0x7feb90010b40 2024-09-16T14:50:29.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.741+0000 7feba77fe640 1 -- 172.21.15.26:0/1272395382 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3193180739) 0x7feb98014000 con 0x7feb90010b40 2024-09-16T14:50:29.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.741+0000 7feba77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:29.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.741+0000 7feba77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:29.747 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":76,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:50:29.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.745+0000 7febb52a1640 1 -- 172.21.15.26:0/1272395382 >> v1:172.21.15.189:6812/3196161582 conn(0x7feb90037ae0 legacy=0x7feb90039fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:29.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.745+0000 7febb52a1640 10 monclient: shutdown 2024-09-16T14:50:29.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.745+0000 7febb52a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:29.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.745+0000 7febb52a1640 1 -- 172.21.15.26:0/1272395382 >> v1:172.21.15.26:6789/0 conn(0x7feb90010b40 legacy=0x7febb01ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:29.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.745+0000 7febb52a1640 1 -- 172.21.15.26:0/1272395382 shutdown_connections 2024-09-16T14:50:29.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.745+0000 7febb52a1640 1 -- 172.21.15.26:0/1272395382 >> 172.21.15.26:0/1272395382 conn(0x7febb009fb50 msgr2=0x7febb00a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:29.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.745+0000 7febb52a1640 1 -- 172.21.15.26:0/1272395382 shutdown_connections 2024-09-16T14:50:29.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.745+0000 7febb52a1640 1 -- 172.21.15.26:0/1272395382 wait complete. 2024-09-16T14:50:29.765 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:50:29.766 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T14:50:29.766 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:50:29.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 10 monclient: get_monmap_and_config 2024-09-16T14:50:29.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 10 monclient: build_initial_monmap 2024-09-16T14:50:29.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 10 monclient: monmap: 2024-09-16T14:50:29.893 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:29.893 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:29.893 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:29.894218+0000 2024-09-16T14:50:29.893 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:29.894218+0000 2024-09-16T14:50:29.893 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:29.893 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:29.894 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:29.894 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:29.894 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:29.894 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:29.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 1 Processor -- start 2024-09-16T14:50:29.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 1 -- start start 2024-09-16T14:50:29.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 10 monclient: init 2024-09-16T14:50:29.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:29.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:29.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 10 monclient(hunting): picked mon.noname-b con 0x7f139c134e40 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:29.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 10 monclient(hunting): picked mon.noname-a con 0x7f139c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:29.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 10 monclient(hunting): picked mon.noname-c con 0x7f139c135220 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:29.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c099e80 con 0x7f139c099a70 2024-09-16T14:50:29.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c0719d0 con 0x7f139c135220 2024-09-16T14:50:29.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c09e1e0 con 0x7f139c134e40 2024-09-16T14:50:29.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:29.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 10 monclient(hunting): authenticate will time out at 1454.014298s 2024-09-16T14:50:29.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f139c134e40 0x7f139c070bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49800/0 (socket says 172.21.15.26:49800) 2024-09-16T14:50:29.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139a59c640 1 -- 172.21.15.26:0/3832733616 learned_addr learned my addr 172.21.15.26:0/3832733616 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:29.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f1399d9b640 1 --1- 172.21.15.26:0/3832733616 >> v1:172.21.15.26:6789/0 conn(0x7f139c099a70 0x7f139c0712c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54344/0 (socket says 172.21.15.26:54344) 2024-09-16T14:50:29.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139ad9d640 1 -- 172.21.15.26:0/3832733616 >> v1:172.21.15.189:6789/0 conn(0x7f139c135220 legacy=0x7f139c13db30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:29.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 1 -- 172.21.15.26:0/3832733616 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601841582 0 0) 0x7f139c09e1e0 con 0x7f139c134e40 2024-09-16T14:50:29.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:29.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:29.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:29.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient(hunting): my global_id is 154655 2024-09-16T14:50:29.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 1 -- 172.21.15.26:0/3832733616 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1384003530 con 0x7f139c134e40 2024-09-16T14:50:29.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 1 -- 172.21.15.26:0/3832733616 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447074648 0 0) 0x7f13900014a0 con 0x7f139c099a70 2024-09-16T14:50:29.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:29.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:29.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:29.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient(hunting): my global_id is 164397 2024-09-16T14:50:29.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 1 -- 172.21.15.26:0/3832733616 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f139c09e1e0 con 0x7f139c099a70 2024-09-16T14:50:29.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 1 -- 172.21.15.26:0/3832733616 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1985616508 0 0) 0x7f1384003530 con 0x7f139c134e40 2024-09-16T14:50:29.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 1 -- 172.21.15.26:0/3832733616 >> v1:172.21.15.189:6789/0 conn(0x7f139c135220 legacy=0x7f139c13db30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:29.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 1 -- 172.21.15.26:0/3832733616 >> v1:172.21.15.26:6789/0 conn(0x7f139c099a70 legacy=0x7f139c0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:29.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient: _finish_hunting 0 2024-09-16T14:50:29.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 1 monclient: found mon.noname-b 2024-09-16T14:50:29.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:29.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:29.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 1 -- 172.21.15.26:0/3832733616 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c09e7d0 con 0x7f139c134e40 2024-09-16T14:50:29.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient: _finish_auth 0 2024-09-16T14:50:29.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient: _check_auth_tickets 2024-09-16T14:50:29.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:29.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 1 -- 172.21.15.26:0/3832733616 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f138c001620 con 0x7f139c134e40 2024-09-16T14:50:29.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 5 monclient: authenticate success, global_id 154655 2024-09-16T14:50:29.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:29.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:29.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:29.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient: dump: 2024-09-16T14:50:29.901 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:29.901 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:29.901 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.901 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.901 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:29.901 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:29.901 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:29.901 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:29.901 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:29.902 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:29.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:29.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 1 -- 172.21.15.26:0/3832733616 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f138c001890 con 0x7f139c134e40 2024-09-16T14:50:29.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:29.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:29.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 1 -- 172.21.15.26:0/3832733616 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f138c001c20 con 0x7f139c134e40 2024-09-16T14:50:29.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f13a087b640 4 set_mon_vals no callback set 2024-09-16T14:50:29.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:29.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:29.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.889+0000 7f139959a640 10 monclient: dump: 2024-09-16T14:50:29.902 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:29.902 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:29.902 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.903 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.903 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:29.903 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:29.903 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:29.903 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:29.903 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:29.903 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:29.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 10 monclient: shutdown 2024-09-16T14:50:29.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:29.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 1 -- 172.21.15.26:0/3832733616 >> v1:172.21.15.26:6790/0 conn(0x7f139c134e40 legacy=0x7f139c070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:29.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 1 -- 172.21.15.26:0/3832733616 shutdown_connections 2024-09-16T14:50:29.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 1 -- 172.21.15.26:0/3832733616 >> 172.21.15.26:0/3832733616 conn(0x7f139c09fb50 msgr2=0x7f139c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:29.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 1 -- 172.21.15.26:0/3832733616 shutdown_connections 2024-09-16T14:50:29.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 1 -- 172.21.15.26:0/3832733616 wait complete. 2024-09-16T14:50:29.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 10 monclient: build_initial_monmap 2024-09-16T14:50:29.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 10 monclient: monmap: 2024-09-16T14:50:29.904 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:29.904 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:29.904 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:29.904 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:29.904 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:29.904 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:29.904 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:29.904 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:29.904 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:29.904 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:29.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 1 Processor -- start 2024-09-16T14:50:29.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 1 -- start start 2024-09-16T14:50:29.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 10 monclient: init 2024-09-16T14:50:29.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 10 monclient(hunting): picked mon.noname-a con 0x7f139c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 10 monclient(hunting): picked mon.noname-c con 0x7f139c134e40 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 10 monclient(hunting): picked mon.noname-b con 0x7f139c135220 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c124380 con 0x7f139c099a70 2024-09-16T14:50:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c122bf0 con 0x7f139c135220 2024-09-16T14:50:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c134400 con 0x7f139c134e40 2024-09-16T14:50:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 10 monclient(hunting): authenticate will time out at 1454.017994s 2024-09-16T14:50:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f139a59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f139c099a70 0x7f139c1c9cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54354/0 (socket says 172.21.15.26:54354) 2024-09-16T14:50:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f139a59c640 1 -- 172.21.15.26:0/3207817057 learned_addr learned my addr 172.21.15.26:0/3207817057 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f139ad9d640 1 -- 172.21.15.26:0/3207817057 >> v1:172.21.15.189:6789/0 conn(0x7f139c135220 legacy=0x7f139c1d2040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:29.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722492478 0 0) 0x7f139c124380 con 0x7f139c099a70 2024-09-16T14:50:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient(hunting): my global_id is 164400 2024-09-16T14:50:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f137c003530 con 0x7f139c099a70 2024-09-16T14:50:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 241137992 0 0) 0x7f139c134400 con 0x7f139c134e40 2024-09-16T14:50:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient(hunting): my global_id is 154658 2024-09-16T14:50:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f139c124380 con 0x7f139c134e40 2024-09-16T14:50:29.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1525817564 0 0) 0x7f137c003530 con 0x7f139c099a70 2024-09-16T14:50:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f139c134400 con 0x7f139c099a70 2024-09-16T14:50:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f138c0016a0 con 0x7f139c099a70 2024-09-16T14:50:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient(hunting): dump: 2024-09-16T14:50:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:29.912 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:29.913 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2068305644 0 0) 0x7f139c124380 con 0x7f139c134e40 2024-09-16T14:50:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f137c003530 con 0x7f139c134e40 2024-09-16T14:50:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1390001ad0 con 0x7f139c134e40 2024-09-16T14:50:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient(hunting): dump: 2024-09-16T14:50:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.913 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:29.914 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1778724625 0 0) 0x7f137c003530 con 0x7f139c134e40 2024-09-16T14:50:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 >> v1:172.21.15.189:6789/0 conn(0x7f139c135220 legacy=0x7f139c1d2040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 >> v1:172.21.15.26:6789/0 conn(0x7f139c099a70 legacy=0x7f139c1c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 monclient: found mon.c 2024-09-16T14:50:29.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c09a240 con 0x7f139c134e40 2024-09-16T14:50:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient: _finish_auth 0 2024-09-16T14:50:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 5 monclient: authenticate success, global_id 154658 2024-09-16T14:50:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 10 monclient: _renew_subs 2024-09-16T14:50:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 1 -- 172.21.15.26:0/3207817057 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139c122020 con 0x7f139c134e40 2024-09-16T14:50:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 10 monclient: _renew_subs 2024-09-16T14:50:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 1 -- 172.21.15.26:0/3207817057 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f139c1ca400 con 0x7f139c134e40 2024-09-16T14:50:29.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1390001fa0 con 0x7f139c134e40 2024-09-16T14:50:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f13900014f0 con 0x7f139c134e40 2024-09-16T14:50:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f1398d99640 4 set_mon_vals no callback set 2024-09-16T14:50:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 10 monclient: dump: 2024-09-16T14:50:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:29.916 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:29.917 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f13a087b640 1 -- 172.21.15.26:0/3207817057 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1380006590 con 0x7f1380002830 2024-09-16T14:50:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.893+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1390004d10 con 0x7f139c134e40 2024-09-16T14:50:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(223..223 src has 169..223) v4 ==== 12392+0+0 (unknown 467405984 0 0) 0x7f1390045480 con 0x7f139c134e40 2024-09-16T14:50:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755662039 0 0) 0x7f1380006590 con 0x7f1380002830 2024-09-16T14:50:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:29.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 10 monclient(hunting): my global_id is 164403 2024-09-16T14:50:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f137c003530 con 0x7f1380002830 2024-09-16T14:50:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2190060982 0 0) 0x7f137c003530 con 0x7f1380002830 2024-09-16T14:50:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1380006590 con 0x7f1380002830 2024-09-16T14:50:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f139ad9d640 10 monclient: get_auth_request con 0x7f137c036220 auth_method 0 2024-09-16T14:50:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f138c004ec0 con 0x7f1380002830 2024-09-16T14:50:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 10 monclient: dump: 2024-09-16T14:50:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.918 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:29.919 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3030454915 0 0) 0x7f1380006590 con 0x7f1380002830 2024-09-16T14:50:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f138000a0e0 con 0x7f1380002830 2024-09-16T14:50:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f138c0071c0 con 0x7f1380002830 2024-09-16T14:50:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 >> v1:172.21.15.26:6789/0 conn(0x7f1380002830 legacy=0x7f1380004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.897+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 >> v1:172.21.15.26:6789/0 conn(0x7f1380002830 legacy=0x7f1380004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.901+0000 7f13a087b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:50:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.901+0000 7f13a087b640 1 -- 172.21.15.26:0/3207817057 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1380009da0 con 0x7f13800090c0 2024-09-16T14:50:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.901+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165898749 0 0) 0x7f1380009da0 con 0x7f13800090c0 2024-09-16T14:50:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.901+0000 7f138affd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.901+0000 7f138affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.901+0000 7f138affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.901+0000 7f138affd640 10 monclient(hunting): my global_id is 164406 2024-09-16T14:50:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.901+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1380006590 con 0x7f13800090c0 2024-09-16T14:50:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.901+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1372985240 0 0) 0x7f1380006590 con 0x7f13800090c0 2024-09-16T14:50:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.901+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1380009da0 con 0x7f13800090c0 2024-09-16T14:50:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.901+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f138c005400 con 0x7f13800090c0 2024-09-16T14:50:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.901+0000 7f138affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.901+0000 7f138affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.901+0000 7f138affd640 10 monclient: dump: 2024-09-16T14:50:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:29.921 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.901+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2752639159 0 0) 0x7f1380009da0 con 0x7f13800090c0 2024-09-16T14:50:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.901+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f138000c290 con 0x7f13800090c0 2024-09-16T14:50:29.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.905+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4034 (unknown 0 0 4250789345) 0x7f138c006850 con 0x7f13800090c0 2024-09-16T14:50:29.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.905+0000 7f138affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:50:29.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.905+0000 7f138affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:29.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.905+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 >> v1:172.21.15.26:6789/0 conn(0x7f13800090c0 legacy=0x7f1380009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:29.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.905+0000 7f138affd640 1 -- 172.21.15.26:0/3207817057 >> v1:172.21.15.26:6789/0 conn(0x7f13800090c0 legacy=0x7f1380009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:29.922 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:50:29.922 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:29.922 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:29.922 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:50:29.922 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 76, 2024-09-16T14:50:29.923 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:50:29.923 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:50:29.923 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:50:29.923 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:29.923 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:50:29.923 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:29.923 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:50:29.923 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:50:29.923 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:29.923 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:29.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:29.923 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:29.924 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:29.924 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:29.924 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:29.924 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:29.924 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:29.924 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:29.924 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:29.924 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:50:29.924 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:50:29.924 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:29.924 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:29.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:29.924 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:29.925 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:29.925 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:29.925 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:29.925 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:29.925 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:29.925 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:29.925 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:29.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:29.925 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:50:29.925 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:50:29.925 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:50:29.925 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:50:29.925 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:50:29.926 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:50:29.926 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:29.926 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:29.926 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:50:29.926 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:50:29.926 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:50:29.926 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:50:29.926 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:50:29.926 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:50:29.926 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:50:29.926 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:29.926 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:50:29.926 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:29.927 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:29.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:29.927 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:29.927 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:29.927 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:29.927 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:29.927 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:29.927 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:29.927 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:29.927 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:29.927 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:50:29.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:29.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:50:29.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:29.928 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:29.928 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:29.928 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:29.928 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:29.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:29.928 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:29.928 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:50:29.928 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:29.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:29.928 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:29.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:29.928 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:50:29.929 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:50:29.929 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:29.929 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:29.929 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:29.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:29.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:29.929 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:29.929 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:29.929 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:29.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:29.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:29.929 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:29.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:50:29.930 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:29.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:29.930 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:29.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:29.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:50:29.930 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:50:29.930 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:29.930 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:29.930 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:29.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:29.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:29.930 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:29.931 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:29.931 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:29.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:29.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:29.931 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:29.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:50:29.931 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:29.931 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:29.931 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:29.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:29.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:50:29.931 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:50:29.931 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:29.932 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:29.932 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:29.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:29.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:29.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:29.932 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:50:29.932 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:50:29.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:29.932 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:29.932 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:29.932 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:29.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:29.932 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:29.932 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:50:29.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:29.933 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:29.933 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:29.933 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:50:29.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:29.933 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:29.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:50:29.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:29.933 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:29.933 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:29.933 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:29.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:29.933 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:29.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:29.934 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:50:29.934 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:50:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.905+0000 7f13a087b640 1 -- 172.21.15.26:0/3207817057 >> v1:172.21.15.189:6812/3196161582 conn(0x7f137c036220 legacy=0x7f137c0386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.905+0000 7f13a087b640 10 monclient: shutdown 2024-09-16T14:50:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.905+0000 7f13a087b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.905+0000 7f13a087b640 1 -- 172.21.15.26:0/3207817057 >> v1:172.21.15.26:6790/0 conn(0x7f139c134e40 legacy=0x7f139c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.905+0000 7f139ad9d640 1 -- 172.21.15.26:0/3207817057 reap_dead start 2024-09-16T14:50:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.905+0000 7f139ad9d640 1 -- 172.21.15.26:0/3207817057 reap_dead start 2024-09-16T14:50:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.905+0000 7f13a087b640 1 -- 172.21.15.26:0/3207817057 shutdown_connections 2024-09-16T14:50:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.905+0000 7f13a087b640 1 -- 172.21.15.26:0/3207817057 >> 172.21.15.26:0/3207817057 conn(0x7f139c09fb50 msgr2=0x7f139c09a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.905+0000 7f13a087b640 1 -- 172.21.15.26:0/3207817057 shutdown_connections 2024-09-16T14:50:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:29.905+0000 7f13a087b640 1 -- 172.21.15.26:0/3207817057 wait complete. 2024-09-16T14:50:29.936 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:50:29.936 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:50:29.936 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:50:30.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 10 monclient: get_monmap_and_config 2024-09-16T14:50:30.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 10 monclient: build_initial_monmap 2024-09-16T14:50:30.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 10 monclient: monmap: 2024-09-16T14:50:30.053 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:30.053 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:30.053 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:30.053778+0000 2024-09-16T14:50:30.053 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:30.053778+0000 2024-09-16T14:50:30.053 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:30.053 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:30.053 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:30.053 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:30.053 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:30.053 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:30.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 1 Processor -- start 2024-09-16T14:50:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 1 -- start start 2024-09-16T14:50:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 10 monclient: init 2024-09-16T14:50:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 10 monclient(hunting): picked mon.noname-a con 0x7f371009cdc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 10 monclient(hunting): picked mon.noname-b con 0x7f371009e7a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 10 monclient(hunting): picked mon.noname-c con 0x7f371009d1a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371009d5b0 con 0x7f371009cdc0 2024-09-16T14:50:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3710131a90 con 0x7f371009d1a0 2024-09-16T14:50:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371009aff0 con 0x7f371009e7a0 2024-09-16T14:50:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:30.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 10 monclient(hunting): authenticate will time out at 1454.173955s 2024-09-16T14:50:30.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3705d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f371009e7a0 0x7f3710131380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49810/0 (socket says 172.21.15.26:49810) 2024-09-16T14:50:30.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370e59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f371009cdc0 0x7f3710130c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54388/0 (socket says 172.21.15.26:54388) 2024-09-16T14:50:30.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3705d9b640 1 -- 172.21.15.26:0/4228334536 learned_addr learned my addr 172.21.15.26:0/4228334536 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:30.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370ed9d640 1 -- 172.21.15.26:0/4228334536 >> v1:172.21.15.189:6789/0 conn(0x7f371009d1a0 legacy=0x7f3710139fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:30.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 1 -- 172.21.15.26:0/4228334536 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001246347 0 0) 0x7f371009aff0 con 0x7f371009e7a0 2024-09-16T14:50:30.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:50:30.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:30.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:30.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient(hunting): my global_id is 154661 2024-09-16T14:50:30.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 1 -- 172.21.15.26:0/4228334536 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36fc003450 con 0x7f371009e7a0 2024-09-16T14:50:30.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 1 -- 172.21.15.26:0/4228334536 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383216868 0 0) 0x7f371009d5b0 con 0x7f371009cdc0 2024-09-16T14:50:30.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:50:30.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:30.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:30.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient(hunting): my global_id is 164409 2024-09-16T14:50:30.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 1 -- 172.21.15.26:0/4228334536 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f371009aff0 con 0x7f371009cdc0 2024-09-16T14:50:30.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 1 -- 172.21.15.26:0/4228334536 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 862644370 0 0) 0x7f36fc003450 con 0x7f371009e7a0 2024-09-16T14:50:30.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 1 -- 172.21.15.26:0/4228334536 >> v1:172.21.15.189:6789/0 conn(0x7f371009d1a0 legacy=0x7f3710139fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:30.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 1 -- 172.21.15.26:0/4228334536 >> v1:172.21.15.26:6789/0 conn(0x7f371009cdc0 legacy=0x7f3710130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:30.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient: _finish_hunting 0 2024-09-16T14:50:30.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 1 monclient: found mon.noname-b 2024-09-16T14:50:30.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:30.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:30.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 1 -- 172.21.15.26:0/4228334536 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371009b2d0 con 0x7f371009e7a0 2024-09-16T14:50:30.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient: _finish_auth 0 2024-09-16T14:50:30.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient: _check_auth_tickets 2024-09-16T14:50:30.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 1 -- 172.21.15.26:0/4228334536 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f36f8001550 con 0x7f371009e7a0 2024-09-16T14:50:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 5 monclient: authenticate success, global_id 154661 2024-09-16T14:50:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient: dump: 2024-09-16T14:50:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:30.061 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 1 -- 172.21.15.26:0/4228334536 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36f80017c0 con 0x7f371009e7a0 2024-09-16T14:50:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 1 -- 172.21.15.26:0/4228334536 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f36f8001b90 con 0x7f371009e7a0 2024-09-16T14:50:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 4 set_mon_vals no callback set 2024-09-16T14:50:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f370dd9b640 10 monclient: dump: 2024-09-16T14:50:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:30.062 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.062 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.062 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:30.062 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:30.062 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:30.062 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:30.062 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:30.062 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:30.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 10 monclient: shutdown 2024-09-16T14:50:30.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:30.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 1 -- 172.21.15.26:0/4228334536 >> v1:172.21.15.26:6790/0 conn(0x7f371009e7a0 legacy=0x7f3710131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:30.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 1 -- 172.21.15.26:0/4228334536 shutdown_connections 2024-09-16T14:50:30.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 1 -- 172.21.15.26:0/4228334536 >> 172.21.15.26:0/4228334536 conn(0x7f371009fb50 msgr2=0x7f371009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:30.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 1 -- 172.21.15.26:0/4228334536 shutdown_connections 2024-09-16T14:50:30.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.049+0000 7f3714861640 1 -- 172.21.15.26:0/4228334536 wait complete. 2024-09-16T14:50:30.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 10 monclient: build_initial_monmap 2024-09-16T14:50:30.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 10 monclient: monmap: 2024-09-16T14:50:30.063 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:30.063 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:30.063 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:30.063 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:30.063 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:30.063 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:30.063 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:30.063 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:30.063 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:30.063 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:30.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 1 Processor -- start 2024-09-16T14:50:30.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 1 -- start start 2024-09-16T14:50:30.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 10 monclient: init 2024-09-16T14:50:30.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:30.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:30.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 10 monclient(hunting): picked mon.noname-a con 0x7f371009cdc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:30.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 10 monclient(hunting): picked mon.noname-c con 0x7f371009d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:30.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 10 monclient(hunting): picked mon.noname-b con 0x7f371009e7a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:30.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371011e740 con 0x7f371009cdc0 2024-09-16T14:50:30.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371011c390 con 0x7f371009e7a0 2024-09-16T14:50:30.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37100a2290 con 0x7f371009d1a0 2024-09-16T14:50:30.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:30.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 10 monclient(hunting): authenticate will time out at 1454.177191s 2024-09-16T14:50:30.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3705d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f371009d1a0 0x7f371012dd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49820/0 (socket says 172.21.15.26:49820) 2024-09-16T14:50:30.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3705d9b640 1 -- 172.21.15.26:0/2385871041 learned_addr learned my addr 172.21.15.26:0/2385871041 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:30.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f370ed9d640 1 -- 172.21.15.26:0/2385871041 >> v1:172.21.15.189:6789/0 conn(0x7f371009e7a0 legacy=0x7f37101cd230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:30.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602124404 0 0) 0x7f37100a2290 con 0x7f371009d1a0 2024-09-16T14:50:30.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:30.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:30.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:30.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): my global_id is 154664 2024-09-16T14:50:30.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f0003530 con 0x7f371009d1a0 2024-09-16T14:50:30.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236490808 0 0) 0x7f371011e740 con 0x7f371009cdc0 2024-09-16T14:50:30.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:30.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:30.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:30.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): my global_id is 164412 2024-09-16T14:50:30.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37100a2290 con 0x7f371009cdc0 2024-09-16T14:50:30.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 911779849 0 0) 0x7f36f0003530 con 0x7f371009d1a0 2024-09-16T14:50:30.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f371011e740 con 0x7f371009d1a0 2024-09-16T14:50:30.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f36f8001b90 con 0x7f371009d1a0 2024-09-16T14:50:30.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:30.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:30.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): dump: 2024-09-16T14:50:30.071 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:30.072 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1800933348 0 0) 0x7f37100a2290 con 0x7f371009cdc0 2024-09-16T14:50:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36f0003530 con 0x7f371009cdc0 2024-09-16T14:50:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f37000019e0 con 0x7f371009cdc0 2024-09-16T14:50:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): dump: 2024-09-16T14:50:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:30.073 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3443940449 0 0) 0x7f371011e740 con 0x7f371009d1a0 2024-09-16T14:50:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 >> v1:172.21.15.189:6789/0 conn(0x7f371009e7a0 legacy=0x7f37101cd230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 >> v1:172.21.15.26:6789/0 conn(0x7f371009cdc0 legacy=0x7f37101c4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 monclient: found mon.c 2024-09-16T14:50:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37100a1d40 con 0x7f371009d1a0 2024-09-16T14:50:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 5 monclient: authenticate success, global_id 154664 2024-09-16T14:50:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 10 monclient: _renew_subs 2024-09-16T14:50:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 1 -- 172.21.15.26:0/2385871041 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f371011e460 con 0x7f371009d1a0 2024-09-16T14:50:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 10 monclient: _renew_subs 2024-09-16T14:50:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 1 -- 172.21.15.26:0/2385871041 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f371012e460 con 0x7f371009d1a0 2024-09-16T14:50:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36f80010f0 con 0x7f371009d1a0 2024-09-16T14:50:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f36f8004cc0 con 0x7f371009d1a0 2024-09-16T14:50:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f370d59a640 4 set_mon_vals no callback set 2024-09-16T14:50:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient: dump: 2024-09-16T14:50:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:30.076 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f3714861640 1 -- 172.21.15.26:0/2385871041 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f36f4006590 con 0x7f36f4002830 2024-09-16T14:50:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f36f8004ea0 con 0x7f371009d1a0 2024-09-16T14:50:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(223..223 src has 169..223) v4 ==== 12392+0+0 (unknown 467405984 0 0) 0x7f36f8045640 con 0x7f371009d1a0 2024-09-16T14:50:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143650526 0 0) 0x7f36f4006590 con 0x7f36f4002830 2024-09-16T14:50:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient(hunting): my global_id is 154667 2024-09-16T14:50:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f371011e740 con 0x7f36f4002830 2024-09-16T14:50:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1643871410 0 0) 0x7f371011e740 con 0x7f36f4002830 2024-09-16T14:50:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36f4006590 con 0x7f36f4002830 2024-09-16T14:50:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3700004990 con 0x7f36f4002830 2024-09-16T14:50:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f37077fe640 10 monclient: dump: 2024-09-16T14:50:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:30.078 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.053+0000 7f370ed9d640 10 monclient: get_auth_request con 0x7f36f0036250 auth_method 0 2024-09-16T14:50:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.057+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 659986253 0 0) 0x7f36f4006590 con 0x7f36f4002830 2024-09-16T14:50:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.057+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36f400a0e0 con 0x7f36f4002830 2024-09-16T14:50:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.057+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f3700006b30 con 0x7f36f4002830 2024-09-16T14:50:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.057+0000 7f37077fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.057+0000 7f37077fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.057+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 >> v1:172.21.15.26:6790/0 conn(0x7f36f4002830 legacy=0x7f36f4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.057+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 >> v1:172.21.15.26:6790/0 conn(0x7f36f4002830 legacy=0x7f36f4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f3714861640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:50:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f3714861640 1 -- 172.21.15.26:0/2385871041 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f36f4009da0 con 0x7f36f40090c0 2024-09-16T14:50:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691218584 0 0) 0x7f36f4009da0 con 0x7f36f40090c0 2024-09-16T14:50:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 10 monclient(hunting): my global_id is 154670 2024-09-16T14:50:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f4006590 con 0x7f36f40090c0 2024-09-16T14:50:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2461466757 0 0) 0x7f36f4006590 con 0x7f36f40090c0 2024-09-16T14:50:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36f4009da0 con 0x7f36f40090c0 2024-09-16T14:50:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3700005040 con 0x7f36f40090c0 2024-09-16T14:50:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 10 monclient: dump: 2024-09-16T14:50:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:30.080 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3628798692 0 0) 0x7f36f4009da0 con 0x7f36f40090c0 2024-09-16T14:50:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f36f400c290 con 0x7f36f40090c0 2024-09-16T14:50:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4036 (unknown 0 0 4013287525) 0x7f37000064d0 con 0x7f36f40090c0 2024-09-16T14:50:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:50:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 >> v1:172.21.15.26:6790/0 conn(0x7f36f40090c0 legacy=0x7f36f4009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.061+0000 7f37077fe640 1 -- 172.21.15.26:0/2385871041 >> v1:172.21.15.26:6790/0 conn(0x7f36f40090c0 legacy=0x7f36f4009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:30.081 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:50:30.081 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:30.081 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:30.081 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:50:30.081 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 76, 2024-09-16T14:50:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:50:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:50:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:50:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:50:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:50:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:50:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:30.082 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:50:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:50:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:30.083 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:50:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:50:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:50:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:50:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:50:30.084 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:50:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:50:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:50:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:50:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:50:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:50:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:50:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:50:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:50:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:30.085 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:50:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:50:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:30.086 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:50:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:50:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:50:30.087 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:50:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:30.088 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:50:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:50:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:30.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:50:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:50:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:50:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:30.090 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:50:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:50:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:30.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:50:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:50:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:50:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:30.092 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:50:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:30.093 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:50:30.093 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:50:30.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.065+0000 7f3714861640 1 -- 172.21.15.26:0/2385871041 >> v1:172.21.15.189:6812/3196161582 conn(0x7f36f0036250 legacy=0x7f36f0038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:30.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.065+0000 7f3714861640 10 monclient: shutdown 2024-09-16T14:50:30.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.065+0000 7f3714861640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:30.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.065+0000 7f3714861640 1 -- 172.21.15.26:0/2385871041 >> v1:172.21.15.26:6790/0 conn(0x7f371009d1a0 legacy=0x7f371012dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:30.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.065+0000 7f370ed9d640 1 -- 172.21.15.26:0/2385871041 reap_dead start 2024-09-16T14:50:30.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.065+0000 7f370ed9d640 1 -- 172.21.15.26:0/2385871041 reap_dead start 2024-09-16T14:50:30.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.065+0000 7f3714861640 1 -- 172.21.15.26:0/2385871041 shutdown_connections 2024-09-16T14:50:30.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.065+0000 7f3714861640 1 -- 172.21.15.26:0/2385871041 >> 172.21.15.26:0/2385871041 conn(0x7f371009fb50 msgr2=0x7f371009c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:30.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.065+0000 7f3714861640 1 -- 172.21.15.26:0/2385871041 shutdown_connections 2024-09-16T14:50:30.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:30.065+0000 7f3714861640 1 -- 172.21.15.26:0/2385871041 wait complete. 2024-09-16T14:50:30.096 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:50:30.096 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:50:30.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.500+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h9"}]=0 removed item id -43 name 'h9' from crush map v224) v1 ==== 118+0+0 (unknown 3143411996 0 0) 0x7f5418014760 con 0x7f5420132090 2024-09-16T14:50:30.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.500+0000 7f54177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T14:50:30.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.500+0000 7f54177fe640 10 monclient: _finish_command 2 = system:0 removed item id -43 name 'h9' from crush map 2024-09-16T14:50:30.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.604+0000 7f542520f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T14:50:30.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.604+0000 7f542520f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T14:50:30.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.604+0000 7f542520f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:30.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.604+0000 7f542520f640 1 -- 172.21.15.189:0/1831705282 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) v1 -- 0x7f5404005bc0 con 0x7f5420132090 2024-09-16T14:50:30.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.604+0000 7f54177fe640 1 -- 172.21.15.189:0/1831705282 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h9"}]=0 device 'h9' does not appear in the crush map v224) v1 ==== 118+0+0 (unknown 192175842 0 0) 0x7f5418013530 con 0x7f5420132090 2024-09-16T14:50:30.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.604+0000 7f54177fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T14:50:30.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.604+0000 7f54177fe640 10 monclient: _finish_command 3 = system:0 device 'h9' does not appear in the crush map 2024-09-16T14:50:30.608 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h9' does not appear in the crush map 2024-09-16T14:50:30.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.608+0000 7f542520f640 1 -- 172.21.15.189:0/1831705282 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5400036370 legacy=0x7f5400038830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:30.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.608+0000 7f542520f640 10 monclient: shutdown 2024-09-16T14:50:30.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.608+0000 7f542520f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:30.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.608+0000 7f542520f640 1 -- 172.21.15.189:0/1831705282 >> v1:172.21.15.26:6790/0 conn(0x7f5420132090 legacy=0x7f540004ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:30.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.608+0000 7f542520f640 1 -- 172.21.15.189:0/1831705282 shutdown_connections 2024-09-16T14:50:30.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.608+0000 7f542520f640 1 -- 172.21.15.189:0/1831705282 >> 172.21.15.189:0/1831705282 conn(0x7f54200a3320 msgr2=0x7f542009d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:30.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.608+0000 7f542520f640 1 -- 172.21.15.189:0/1831705282 shutdown_connections 2024-09-16T14:50:30.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.608+0000 7f542520f640 1 -- 172.21.15.189:0/1831705282 wait complete. 2024-09-16T14:50:30.624 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:30.624 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h10 2024-09-16T14:50:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 10 monclient: get_monmap_and_config 2024-09-16T14:50:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 10 monclient: build_initial_monmap 2024-09-16T14:50:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 10 monclient: monmap: 2024-09-16T14:50:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:30.738442+0000 2024-09-16T14:50:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:30.738442+0000 2024-09-16T14:50:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:30.738 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:30.738 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:30.738 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:30.738 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:30.738 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 1 Processor -- start 2024-09-16T14:50:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 1 -- start start 2024-09-16T14:50:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 10 monclient: init 2024-09-16T14:50:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:50:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 10 monclient(hunting): picked mon.noname-a con 0x7fcdb41315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 10 monclient(hunting): picked mon.noname-c con 0x7fcdb409b320 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 10 monclient(hunting): picked mon.noname-b con 0x7fcdb4132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb409edd0 con 0x7fcdb41315a0 2024-09-16T14:50:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb409bc00 con 0x7fcdb409b320 2024-09-16T14:50:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb409e900 con 0x7fcdb4132090 2024-09-16T14:50:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 10 monclient(hunting): authenticate will time out at 1412.854065s 2024-09-16T14:50:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb967a640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fcdb409b320 legacy=0x7fcdb4136ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdba67c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fcdb4132090 0x7fcdb413a200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:49270/0 (socket says 172.21.15.189:49270) 2024-09-16T14:50:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdba67c640 1 -- 172.21.15.189:0/567285067 learned_addr learned my addr 172.21.15.189:0/567285067 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb9e7b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fcdb41315a0 0x7fcdb4131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47142/0 (socket says 172.21.15.189:47142) 2024-09-16T14:50:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 1 -- 172.21.15.189:0/567285067 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463338595 0 0) 0x7fcdb409edd0 con 0x7fcdb41315a0 2024-09-16T14:50:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient(hunting): _negotiate 2024-09-16T14:50:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient(hunting): my global_id is 164415 2024-09-16T14:50:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 1 -- 172.21.15.189:0/567285067 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcda0003530 con 0x7fcdb41315a0 2024-09-16T14:50:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 1 -- 172.21.15.189:0/567285067 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090841315 0 0) 0x7fcdb409e900 con 0x7fcdb4132090 2024-09-16T14:50:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient(hunting): _negotiate 2024-09-16T14:50:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient(hunting): my global_id is 154673 2024-09-16T14:50:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 1 -- 172.21.15.189:0/567285067 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdb409edd0 con 0x7fcdb4132090 2024-09-16T14:50:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 1 -- 172.21.15.189:0/567285067 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2724338297 0 0) 0x7fcda0003530 con 0x7fcdb41315a0 2024-09-16T14:50:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 1 -- 172.21.15.189:0/567285067 >> v1:172.21.15.26:6790/0 conn(0x7fcdb4132090 legacy=0x7fcdb413a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 1 -- 172.21.15.189:0/567285067 >> v1:172.21.15.189:6789/0 conn(0x7fcdb409b320 legacy=0x7fcdb4136ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient: _finish_hunting 0 2024-09-16T14:50:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 1 monclient: found mon.noname-a 2024-09-16T14:50:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 1 -- 172.21.15.189:0/567285067 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb409b920 con 0x7fcdb41315a0 2024-09-16T14:50:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient: _finish_auth 0 2024-09-16T14:50:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient: _check_auth_tickets 2024-09-16T14:50:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 5 monclient: authenticate success, global_id 164415 2024-09-16T14:50:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 1 -- 172.21.15.189:0/567285067 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcda4001620 con 0x7fcdb41315a0 2024-09-16T14:50:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient: dump: 2024-09-16T14:50:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:30.750 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 1 -- 172.21.15.189:0/567285067 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcda4001900 con 0x7fcdb41315a0 2024-09-16T14:50:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdbc0df640 4 set_mon_vals no callback set 2024-09-16T14:50:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 1 -- 172.21.15.189:0/567285067 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcda4001c90 con 0x7fcdb41315a0 2024-09-16T14:50:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.736+0000 7fcdb8e79640 10 monclient: dump: 2024-09-16T14:50:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:30.752 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:30.752 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:30.752 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:30.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 10 monclient: shutdown 2024-09-16T14:50:30.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:30.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 1 -- 172.21.15.189:0/567285067 >> v1:172.21.15.26:6789/0 conn(0x7fcdb41315a0 legacy=0x7fcdb4131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:30.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 1 -- 172.21.15.189:0/567285067 shutdown_connections 2024-09-16T14:50:30.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 1 -- 172.21.15.189:0/567285067 >> 172.21.15.189:0/567285067 conn(0x7fcdb40a3320 msgr2=0x7fcdb409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:30.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 1 -- 172.21.15.189:0/567285067 shutdown_connections 2024-09-16T14:50:30.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 1 -- 172.21.15.189:0/567285067 wait complete. 2024-09-16T14:50:30.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 10 monclient: build_initial_monmap 2024-09-16T14:50:30.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 10 monclient: monmap: 2024-09-16T14:50:30.752 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:30.753 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:30.753 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:30.753 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:30.753 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:30.753 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:30.753 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:30.753 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:30.753 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:30.753 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:30.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 1 Processor -- start 2024-09-16T14:50:30.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 1 -- start start 2024-09-16T14:50:30.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 10 monclient: init 2024-09-16T14:50:30.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:30.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:30.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 10 monclient(hunting): picked mon.noname-a con 0x7fcdb41315a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:30.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 10 monclient(hunting): picked mon.noname-c con 0x7fcdb4132090 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:30.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 10 monclient(hunting): picked mon.noname-b con 0x7fcdb4130f00 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:30.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb409cd20 con 0x7fcdb41315a0 2024-09-16T14:50:30.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb411f330 con 0x7fcdb4130f00 2024-09-16T14:50:30.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb411f0e0 con 0x7fcdb4132090 2024-09-16T14:50:30.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:30.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 10 monclient(hunting): authenticate will time out at 1412.857610s 2024-09-16T14:50:30.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdba67c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fcdb4130f00 legacy=0x7fcdb41cd710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:30.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdb967a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fcdb4132090 0x7fcdb411bdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:49276/0 (socket says 172.21.15.189:49276) 2024-09-16T14:50:30.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdb967a640 1 -- 172.21.15.189:0/3350396712 learned_addr learned my addr 172.21.15.189:0/3350396712 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:30.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdb9e7b640 1 --1- 172.21.15.189:0/3350396712 >> v1:172.21.15.26:6789/0 conn(0x7fcdb41315a0 0x7fcdb411b6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47152/0 (socket says 172.21.15.189:47152) 2024-09-16T14:50:30.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759215068 0 0) 0x7fcdb409cd20 con 0x7fcdb41315a0 2024-09-16T14:50:30.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:30.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:30.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:30.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient(hunting): my global_id is 164418 2024-09-16T14:50:30.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd98003530 con 0x7fcdb41315a0 2024-09-16T14:50:30.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565862674 0 0) 0x7fcdb411f0e0 con 0x7fcdb4132090 2024-09-16T14:50:30.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:30.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:30.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:30.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient(hunting): my global_id is 154676 2024-09-16T14:50:30.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdb409cd20 con 0x7fcdb4132090 2024-09-16T14:50:30.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 676349490 0 0) 0x7fcd98003530 con 0x7fcdb41315a0 2024-09-16T14:50:30.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdb411f0e0 con 0x7fcdb41315a0 2024-09-16T14:50:30.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcda40015e0 con 0x7fcdb41315a0 2024-09-16T14:50:30.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:30.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:30.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient(hunting): dump: 2024-09-16T14:50:30.756 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:30.757 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:30.757 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.757 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.757 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:30.757 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:30.757 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:30.757 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:30.757 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:30.757 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:30.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:30.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3035306185 0 0) 0x7fcdb409cd20 con 0x7fcdb4132090 2024-09-16T14:50:30.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd98003530 con 0x7fcdb4132090 2024-09-16T14:50:30.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcdb0004160 con 0x7fcdb4132090 2024-09-16T14:50:30.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:30.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:30.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient(hunting): dump: 2024-09-16T14:50:30.758 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:30.758 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:30.758 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.758 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.758 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:30.758 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:30.758 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:30.758 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:30.758 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:30.758 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:30.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1523361764 0 0) 0x7fcdb411f0e0 con 0x7fcdb41315a0 2024-09-16T14:50:30.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 >> v1:172.21.15.26:6790/0 conn(0x7fcdb4132090 legacy=0x7fcdb411bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:30.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 >> v1:172.21.15.189:6789/0 conn(0x7fcdb4130f00 legacy=0x7fcdb41cd710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:30.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:50:30.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 monclient: found mon.a 2024-09-16T14:50:30.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:30.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:30.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb411c4c0 con 0x7fcdb41315a0 2024-09-16T14:50:30.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient: _finish_auth 0 2024-09-16T14:50:30.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:30.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:30.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 5 monclient: authenticate success, global_id 164418 2024-09-16T14:50:30.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 10 monclient: _renew_subs 2024-09-16T14:50:30.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:30.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 1 -- 172.21.15.189:0/3350396712 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb40a0d80 con 0x7fcdb41315a0 2024-09-16T14:50:30.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 10 monclient: _renew_subs 2024-09-16T14:50:30.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:30.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdbc0df640 1 -- 172.21.15.189:0/3350396712 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb409a6d0 con 0x7fcdb41315a0 2024-09-16T14:50:30.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcda4004b40 con 0x7fcdb41315a0 2024-09-16T14:50:30.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:30.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcda4004ed0 con 0x7fcdb41315a0 2024-09-16T14:50:30.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:30.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:30.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdab7fe640 4 set_mon_vals no callback set 2024-09-16T14:50:30.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.740+0000 7fcdaa7fc640 10 monclient: dump: 2024-09-16T14:50:30.761 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:30.761 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:30.761 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.761 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:30.761 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:30.761 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:30.761 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:30.761 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:30.761 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:30.761 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:30.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.744+0000 7fcdbc0df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:30.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.744+0000 7fcdbc0df640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:30.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.744+0000 7fcdbc0df640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:30.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.744+0000 7fcdbc0df640 1 -- 172.21.15.189:0/3350396712 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9c005050 con 0x7fcdb41315a0 2024-09-16T14:50:30.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.744+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fcda40050b0 con 0x7fcdb41315a0 2024-09-16T14:50:30.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.744+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(224..224 src has 169..224) v4 ==== 12182+0+0 (unknown 3919601380 0 0) 0x7fcda4045450 con 0x7fcdb41315a0 2024-09-16T14:50:30.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.744+0000 7fcdb967a640 10 monclient: get_auth_request con 0x7fcd98036220 auth_method 0 2024-09-16T14:50:30.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.756+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fcda4014930 con 0x7fcdb41315a0 2024-09-16T14:50:30.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.756+0000 7fcdaa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:30.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.756+0000 7fcdaa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:30.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.916+0000 7fcdbc0df640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T14:50:30.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.916+0000 7fcdbc0df640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T14:50:30.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.916+0000 7fcdbc0df640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:30.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:30.916+0000 7fcdbc0df640 1 -- 172.21.15.189:0/3350396712 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h10"} v 0) v1 -- 0x7fcd9c005300 con 0x7fcdb41315a0 2024-09-16T14:50:31.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.284+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h10"}]=0 removed item id -46 name 'h10' from crush map v225) v1 ==== 120+0+0 (unknown 656416124 0 0) 0x7fcda4014520 con 0x7fcdb41315a0 2024-09-16T14:50:31.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.284+0000 7fcdaa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T14:50:31.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.284+0000 7fcdaa7fc640 10 monclient: _finish_command 2 = system:0 removed item id -46 name 'h10' from crush map 2024-09-16T14:50:31.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.384+0000 7fcdbc0df640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T14:50:31.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.384+0000 7fcdbc0df640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T14:50:31.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.384+0000 7fcdbc0df640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:31.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.384+0000 7fcdbc0df640 1 -- 172.21.15.189:0/3350396712 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h10"} v 0) v1 -- 0x7fcd9c005a40 con 0x7fcdb41315a0 2024-09-16T14:50:31.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.384+0000 7fcdaa7fc640 1 -- 172.21.15.189:0/3350396712 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h10"}]=0 device 'h10' does not appear in the crush map v225) v1 ==== 120+0+0 (unknown 3714582501 0 0) 0x7fcda4014700 con 0x7fcdb41315a0 2024-09-16T14:50:31.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.384+0000 7fcdaa7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T14:50:31.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.384+0000 7fcdaa7fc640 10 monclient: _finish_command 3 = system:0 device 'h10' does not appear in the crush map 2024-09-16T14:50:31.387 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h10' does not appear in the crush map 2024-09-16T14:50:31.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.388+0000 7fcdbc0df640 1 -- 172.21.15.189:0/3350396712 >> v1:172.21.15.189:6812/3196161582 conn(0x7fcd98036220 legacy=0x7fcd980386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:31.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.388+0000 7fcdbc0df640 10 monclient: shutdown 2024-09-16T14:50:31.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.388+0000 7fcdbc0df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:31.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.388+0000 7fcdbc0df640 1 -- 172.21.15.189:0/3350396712 >> v1:172.21.15.26:6789/0 conn(0x7fcdb41315a0 legacy=0x7fcdb411b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:31.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.388+0000 7fcdbc0df640 1 -- 172.21.15.189:0/3350396712 shutdown_connections 2024-09-16T14:50:31.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.388+0000 7fcdbc0df640 1 -- 172.21.15.189:0/3350396712 >> 172.21.15.189:0/3350396712 conn(0x7fcdb40a3320 msgr2=0x7fcdb41b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:31.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.388+0000 7fcdbc0df640 1 -- 172.21.15.189:0/3350396712 shutdown_connections 2024-09-16T14:50:31.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.388+0000 7fcdbc0df640 1 -- 172.21.15.189:0/3350396712 wait complete. 2024-09-16T14:50:31.404 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:31.404 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h11 2024-09-16T14:50:31.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f776d34f640 10 monclient: get_monmap_and_config 2024-09-16T14:50:31.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f776d34f640 10 monclient: build_initial_monmap 2024-09-16T14:50:31.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f776d34f640 10 monclient: monmap: 2024-09-16T14:50:31.522 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:31.522 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:31.522 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:31.523487+0000 2024-09-16T14:50:31.522 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:31.523487+0000 2024-09-16T14:50:31.522 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:31.522 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:31.522 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:31.523 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:31.523 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:31.523 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:31.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f776d34f640 1 Processor -- start 2024-09-16T14:50:31.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f776d34f640 1 -- start start 2024-09-16T14:50:31.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f776d34f640 10 monclient: init 2024-09-16T14:50:31.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f776d34f640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:31.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f776d34f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:31.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f776d34f640 10 monclient(hunting): picked mon.noname-b con 0x7f77681318a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:31.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f776d34f640 10 monclient(hunting): picked mon.noname-a con 0x7f7768134d50 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:31.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f776d34f640 10 monclient(hunting): picked mon.noname-c con 0x7f776813b2b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:31.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f776d34f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776809d8a0 con 0x7f7768134d50 2024-09-16T14:50:31.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f776d34f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776809ddf0 con 0x7f776813b2b0 2024-09-16T14:50:31.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f776d34f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776809e8b0 con 0x7f77681318a0 2024-09-16T14:50:31.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f776d34f640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:31.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f776d34f640 10 monclient(hunting): authenticate will time out at 1413.639282s 2024-09-16T14:50:31.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f77677fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f776813b2b0 legacy=0x7f776813d6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:31.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f77667fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7768134d50 0x7f7768139b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47162/0 (socket says 172.21.15.189:47162) 2024-09-16T14:50:31.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.520+0000 7f77667fc640 1 -- 172.21.15.189:0/1720568510 learned_addr learned my addr 172.21.15.189:0/1720568510 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:31.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 1 -- 172.21.15.189:0/1720568510 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638516024 0 0) 0x7f776809e8b0 con 0x7f77681318a0 2024-09-16T14:50:31.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:31.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:31.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:31.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient(hunting): my global_id is 154679 2024-09-16T14:50:31.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 1 -- 172.21.15.189:0/1720568510 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7750003530 con 0x7f77681318a0 2024-09-16T14:50:31.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 1 -- 172.21.15.189:0/1720568510 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918984664 0 0) 0x7f776809d8a0 con 0x7f7768134d50 2024-09-16T14:50:31.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:31.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:31.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:31.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient(hunting): my global_id is 164421 2024-09-16T14:50:31.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 1 -- 172.21.15.189:0/1720568510 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f776809e8b0 con 0x7f7768134d50 2024-09-16T14:50:31.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 1 -- 172.21.15.189:0/1720568510 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2805176165 0 0) 0x7f7750003530 con 0x7f77681318a0 2024-09-16T14:50:31.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 1 -- 172.21.15.189:0/1720568510 >> v1:172.21.15.189:6789/0 conn(0x7f776813b2b0 legacy=0x7f776813d6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:31.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 1 -- 172.21.15.189:0/1720568510 >> v1:172.21.15.26:6789/0 conn(0x7f7768134d50 legacy=0x7f7768139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:31.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:50:31.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 1 monclient: found mon.noname-b 2024-09-16T14:50:31.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:31.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 1 -- 172.21.15.189:0/1720568510 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776809eb90 con 0x7f77681318a0 2024-09-16T14:50:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient: _finish_auth 0 2024-09-16T14:50:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 1 -- 172.21.15.189:0/1720568510 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7758001620 con 0x7f77681318a0 2024-09-16T14:50:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient: dump: 2024-09-16T14:50:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:31.531 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:31.531 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:31.531 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:31.531 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:31.531 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:31.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 5 monclient: authenticate success, global_id 154679 2024-09-16T14:50:31.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:31.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 1 -- 172.21.15.189:0/1720568510 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7758001810 con 0x7f77681318a0 2024-09-16T14:50:31.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:31.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:31.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 4 set_mon_vals no callback set 2024-09-16T14:50:31.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 1 -- 172.21.15.189:0/1720568510 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7758001be0 con 0x7f77681318a0 2024-09-16T14:50:31.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:31.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:31.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7765ffb640 10 monclient: dump: 2024-09-16T14:50:31.532 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:31.532 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:31.532 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:31.532 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:31.532 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:31.532 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:31.533 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:31.533 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:31.533 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:31.533 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:31.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 10 monclient: shutdown 2024-09-16T14:50:31.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:31.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 1 -- 172.21.15.189:0/1720568510 >> v1:172.21.15.26:6790/0 conn(0x7f77681318a0 legacy=0x7f7768131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:31.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 1 -- 172.21.15.189:0/1720568510 shutdown_connections 2024-09-16T14:50:31.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 1 -- 172.21.15.189:0/1720568510 >> 172.21.15.189:0/1720568510 conn(0x7f77680a3320 msgr2=0x7f776809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:31.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 1 -- 172.21.15.189:0/1720568510 shutdown_connections 2024-09-16T14:50:31.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 1 -- 172.21.15.189:0/1720568510 wait complete. 2024-09-16T14:50:31.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 10 monclient: build_initial_monmap 2024-09-16T14:50:31.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 10 monclient: monmap: 2024-09-16T14:50:31.534 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:31.534 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:31.534 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:31.534 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:31.534 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:31.534 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:31.534 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:31.534 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:31.534 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:31.534 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:31.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 1 Processor -- start 2024-09-16T14:50:31.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 1 -- start start 2024-09-16T14:50:31.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 10 monclient: init 2024-09-16T14:50:31.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:31.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:31.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 10 monclient(hunting): picked mon.noname-c con 0x7f77681318a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:31.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 10 monclient(hunting): picked mon.noname-a con 0x7f7768134d50 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:31.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 10 monclient(hunting): picked mon.noname-b con 0x7f776813b2b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:31.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680a1ba0 con 0x7f7768134d50 2024-09-16T14:50:31.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776806b380 con 0x7f776813b2b0 2024-09-16T14:50:31.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776813abd0 con 0x7f77681318a0 2024-09-16T14:50:31.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:31.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f77677fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f776813b2b0 legacy=0x7f77681d1690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:31.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f776d34f640 10 monclient(hunting): authenticate will time out at 1413.643282s 2024-09-16T14:50:31.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7766ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f77681318a0 0x7f77681c9430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:49288/0 (socket says 172.21.15.189:49288) 2024-09-16T14:50:31.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7766ffd640 1 -- 172.21.15.189:0/3916035899 learned_addr learned my addr 172.21.15.189:0/3916035899 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:31.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7766ffd640 1 -- 172.21.15.189:0/3916035899 >> v1:172.21.15.26:6790/0 conn(0x7f77681318a0 legacy=0x7f77681c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T14:50:31.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7766ffd640 1 -- 172.21.15.189:0/3916035899 >> v1:172.21.15.26:6790/0 conn(0x7f77681318a0 legacy=0x7f77681c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:50:31.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7766ffd640 1 --1- 172.21.15.189:0/3916035899 >> v1:172.21.15.26:6790/0 conn(0x7f77681318a0 0x7f77681c9430 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-16T14:50:31.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f7766ffd640 1 --1- 172.21.15.189:0/3916035899 >> v1:172.21.15.26:6790/0 conn(0x7f77681318a0 0x7f77681c9430 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-16T14:50:31.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f774f7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6790/0 2024-09-16T14:50:31.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f774f7fe640 1 -- 172.21.15.189:0/3916035899 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531320940 0 0) 0x7f77680a1ba0 con 0x7f7768134d50 2024-09-16T14:50:31.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f774f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:31.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f774f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:31.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f774f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:31.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f774f7fe640 10 monclient(hunting): my global_id is 164424 2024-09-16T14:50:31.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.524+0000 7f774f7fe640 1 -- 172.21.15.189:0/3916035899 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7754003530 con 0x7f7768134d50 2024-09-16T14:50:31.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 1 -- 172.21.15.189:0/3916035899 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 832284951 0 0) 0x7f7754003530 con 0x7f7768134d50 2024-09-16T14:50:31.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 1 -- 172.21.15.189:0/3916035899 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77680a1ba0 con 0x7f7768134d50 2024-09-16T14:50:31.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 1 -- 172.21.15.189:0/3916035899 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f775c001380 con 0x7f7768134d50 2024-09-16T14:50:31.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:31.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:31.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 10 monclient(hunting): dump: 2024-09-16T14:50:31.542 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:31.542 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:31.542 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:31.542 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:31.542 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:31.542 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:31.542 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:31.542 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:31.543 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:31.543 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:31.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:31.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 1 -- 172.21.15.189:0/3916035899 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 927686227 0 0) 0x7f77680a1ba0 con 0x7f7768134d50 2024-09-16T14:50:31.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 1 -- 172.21.15.189:0/3916035899 >> v1:172.21.15.26:6790/0 conn(0x7f77681318a0 legacy=0x7f77681c9430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:31.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 1 -- 172.21.15.189:0/3916035899 >> v1:172.21.15.189:6789/0 conn(0x7f776813b2b0 legacy=0x7f77681d1690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:31.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:31.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 1 monclient: found mon.a 2024-09-16T14:50:31.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:31.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:31.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 1 -- 172.21.15.189:0/3916035899 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776811b7a0 con 0x7f7768134d50 2024-09-16T14:50:31.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:31.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:31.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f776d34f640 5 monclient: authenticate success, global_id 164424 2024-09-16T14:50:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f776d34f640 10 monclient: _renew_subs 2024-09-16T14:50:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f776d34f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f776d34f640 1 -- 172.21.15.189:0/3916035899 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776811b4c0 con 0x7f7768134d50 2024-09-16T14:50:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f776d34f640 10 monclient: _renew_subs 2024-09-16T14:50:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f776d34f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f776d34f640 1 -- 172.21.15.189:0/3916035899 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f776811a000 con 0x7f7768134d50 2024-09-16T14:50:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 1 -- 172.21.15.189:0/3916035899 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f775c0015a0 con 0x7f7768134d50 2024-09-16T14:50:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 1 -- 172.21.15.189:0/3916035899 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f775c004c40 con 0x7f7768134d50 2024-09-16T14:50:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 10 monclient: dump: 2024-09-16T14:50:31.545 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:31.545 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:31.545 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:31.545 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:31.545 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:31.545 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:31.545 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:31.545 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:31.545 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:31.545 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:31.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f7764ff9640 4 set_mon_vals no callback set 2024-09-16T14:50:31.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f776d34f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:31.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f776d34f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:31.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f776d34f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:31.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f776d34f640 1 -- 172.21.15.189:0/3916035899 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7748005050 con 0x7f7768134d50 2024-09-16T14:50:31.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 1 -- 172.21.15.189:0/3916035899 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f775c00f6b0 con 0x7f7768134d50 2024-09-16T14:50:31.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f774f7fe640 1 -- 172.21.15.189:0/3916035899 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(225..225 src has 169..225) v4 ==== 11969+0+0 (unknown 4251906508 0 0) 0x7f775c0451d0 con 0x7f7768134d50 2024-09-16T14:50:31.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.528+0000 7f7766ffd640 10 monclient: get_auth_request con 0x7f7754036060 auth_method 0 2024-09-16T14:50:31.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.540+0000 7f774f7fe640 1 -- 172.21.15.189:0/3916035899 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f775c014470 con 0x7f7768134d50 2024-09-16T14:50:31.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.540+0000 7f774f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:31.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.540+0000 7f774f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:31.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.696+0000 7f776d34f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T14:50:31.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.696+0000 7f776d34f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T14:50:31.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.696+0000 7f776d34f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:31.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:31.696+0000 7f776d34f640 1 -- 172.21.15.189:0/3916035899 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h11"} v 0) v1 -- 0x7f7748005300 con 0x7f7768134d50 2024-09-16T14:50:32.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.516+0000 7f774f7fe640 1 -- 172.21.15.189:0/3916035899 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h11"}]=0 removed item id -49 name 'h11' from crush map v226) v1 ==== 120+0+0 (unknown 1751865568 0 0) 0x7f775c014470 con 0x7f7768134d50 2024-09-16T14:50:32.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.516+0000 7f774f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T14:50:32.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.516+0000 7f774f7fe640 10 monclient: _finish_command 2 = system:0 removed item id -49 name 'h11' from crush map 2024-09-16T14:50:32.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.612+0000 7f776d34f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T14:50:32.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.612+0000 7f776d34f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T14:50:32.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.612+0000 7f776d34f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:32.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.612+0000 7f776d34f640 1 -- 172.21.15.189:0/3916035899 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h11"} v 0) v1 -- 0x7f7748005a40 con 0x7f7768134d50 2024-09-16T14:50:32.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.616+0000 7f774f7fe640 1 -- 172.21.15.189:0/3916035899 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h11"}]=0 device 'h11' does not appear in the crush map v226) v1 ==== 120+0+0 (unknown 1931931246 0 0) 0x7f775c014fd0 con 0x7f7768134d50 2024-09-16T14:50:32.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.616+0000 7f774f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T14:50:32.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.616+0000 7f774f7fe640 10 monclient: _finish_command 3 = system:0 device 'h11' does not appear in the crush map 2024-09-16T14:50:32.618 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h11' does not appear in the crush map 2024-09-16T14:50:32.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.616+0000 7f776d34f640 1 -- 172.21.15.189:0/3916035899 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7754036060 legacy=0x7f7754038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:32.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.616+0000 7f776d34f640 10 monclient: shutdown 2024-09-16T14:50:32.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.616+0000 7f776d34f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:32.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.616+0000 7f776d34f640 1 -- 172.21.15.189:0/3916035899 >> v1:172.21.15.26:6789/0 conn(0x7f7768134d50 legacy=0x7f77681c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:32.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.620+0000 7f776d34f640 1 -- 172.21.15.189:0/3916035899 shutdown_connections 2024-09-16T14:50:32.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.620+0000 7f776d34f640 1 -- 172.21.15.189:0/3916035899 >> 172.21.15.189:0/3916035899 conn(0x7f77680a3320 msgr2=0x7f7768133f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:32.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.620+0000 7f776d34f640 1 -- 172.21.15.189:0/3916035899 shutdown_connections 2024-09-16T14:50:32.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.620+0000 7f776d34f640 1 -- 172.21.15.189:0/3916035899 wait complete. 2024-09-16T14:50:32.635 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:32.635 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h12 2024-09-16T14:50:32.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.752+0000 7f06a26c9640 10 monclient: get_monmap_and_config 2024-09-16T14:50:32.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.752+0000 7f06a26c9640 10 monclient: build_initial_monmap 2024-09-16T14:50:32.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.752+0000 7f06a26c9640 10 monclient: monmap: 2024-09-16T14:50:32.755 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:32.755 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:32.755 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:32.755928+0000 2024-09-16T14:50:32.755 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:32.755928+0000 2024-09-16T14:50:32.755 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:32.756 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:32.756 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:32.756 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:32.756 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:32.757 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:32.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.752+0000 7f06a26c9640 1 Processor -- start 2024-09-16T14:50:32.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.752+0000 7f06a26c9640 1 -- start start 2024-09-16T14:50:32.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.752+0000 7f06a26c9640 10 monclient: init 2024-09-16T14:50:32.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:32.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:32.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 10 monclient(hunting): picked mon.noname-c con 0x7f069c0a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:32.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 10 monclient(hunting): picked mon.noname-a con 0x7f069c0a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:32.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 10 monclient(hunting): picked mon.noname-b con 0x7f069c13b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:32.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c0a30b0 con 0x7f069c0a25f0 2024-09-16T14:50:32.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c0a2e30 con 0x7f069c0a1b00 2024-09-16T14:50:32.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c09eb50 con 0x7f069c13b430 2024-09-16T14:50:32.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:32.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 10 monclient(hunting): authenticate will time out at 1414.872491s 2024-09-16T14:50:32.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069bfff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f069c0a1b00 legacy=0x7f069c0a1ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:32.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f069c0a25f0 0x7f069c139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47176/0 (socket says 172.21.15.189:47176) 2024-09-16T14:50:32.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069b7fe640 1 -- 172.21.15.189:0/255988770 learned_addr learned my addr 172.21.15.189:0/255988770 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:32.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a0c66640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f069c13b430 0x7f069c13d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:49292/0 (socket says 172.21.15.189:49292) 2024-09-16T14:50:32.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 1 -- 172.21.15.189:0/255988770 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237539507 0 0) 0x7f069c09eb50 con 0x7f069c13b430 2024-09-16T14:50:32.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:32.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:32.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:32.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient(hunting): my global_id is 154682 2024-09-16T14:50:32.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 1 -- 172.21.15.189:0/255988770 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0688003530 con 0x7f069c13b430 2024-09-16T14:50:32.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 1 -- 172.21.15.189:0/255988770 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420499083 0 0) 0x7f069c0a30b0 con 0x7f069c0a25f0 2024-09-16T14:50:32.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:32.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:32.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:32.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient(hunting): my global_id is 164427 2024-09-16T14:50:32.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 1 -- 172.21.15.189:0/255988770 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f069c09eb50 con 0x7f069c0a25f0 2024-09-16T14:50:32.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 1 -- 172.21.15.189:0/255988770 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 902538397 0 0) 0x7f0688003530 con 0x7f069c13b430 2024-09-16T14:50:32.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 1 -- 172.21.15.189:0/255988770 >> v1:172.21.15.189:6789/0 conn(0x7f069c0a1b00 legacy=0x7f069c0a1ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:32.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 1 -- 172.21.15.189:0/255988770 >> v1:172.21.15.26:6789/0 conn(0x7f069c0a25f0 legacy=0x7f069c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:32.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:32.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 1 monclient: found mon.noname-b 2024-09-16T14:50:32.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:32.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:32.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 1 -- 172.21.15.189:0/255988770 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c09ee30 con 0x7f069c13b430 2024-09-16T14:50:32.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient: _finish_auth 0 2024-09-16T14:50:32.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:32.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:32.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 1 -- 172.21.15.189:0/255988770 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0694001620 con 0x7f069c13b430 2024-09-16T14:50:32.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:32.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:32.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient: dump: 2024-09-16T14:50:32.768 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:32.769 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:32.769 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:32.769 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:32.769 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:32.769 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:32.769 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:32.769 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:32.769 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:32.769 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:32.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 5 monclient: authenticate success, global_id 154682 2024-09-16T14:50:32.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:32.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 1 -- 172.21.15.189:0/255988770 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0694001880 con 0x7f069c13b430 2024-09-16T14:50:32.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:32.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:32.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 4 set_mon_vals no callback set 2024-09-16T14:50:32.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 1 -- 172.21.15.189:0/255988770 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0694001c50 con 0x7f069c13b430 2024-09-16T14:50:32.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:32.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:32.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f069affd640 10 monclient: dump: 2024-09-16T14:50:32.770 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:32.770 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:32.770 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:32.770 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:32.771 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:32.771 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:32.771 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:32.771 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:32.771 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:32.771 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:32.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 10 monclient: shutdown 2024-09-16T14:50:32.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:32.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 1 -- 172.21.15.189:0/255988770 >> v1:172.21.15.26:6790/0 conn(0x7f069c13b430 legacy=0x7f069c13d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:32.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 1 -- 172.21.15.189:0/255988770 shutdown_connections 2024-09-16T14:50:32.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 1 -- 172.21.15.189:0/255988770 >> 172.21.15.189:0/255988770 conn(0x7f069c0a7dd0 msgr2=0x7f069c0a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:32.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 1 -- 172.21.15.189:0/255988770 shutdown_connections 2024-09-16T14:50:32.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 1 -- 172.21.15.189:0/255988770 wait complete. 2024-09-16T14:50:32.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 10 monclient: build_initial_monmap 2024-09-16T14:50:32.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 10 monclient: monmap: 2024-09-16T14:50:32.775 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:32.775 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:32.775 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:32.775 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:32.775 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:32.775 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:32.775 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:32.776 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:32.776 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:32.776 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:32.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 1 Processor -- start 2024-09-16T14:50:32.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 1 -- start start 2024-09-16T14:50:32.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.756+0000 7f06a26c9640 10 monclient: init 2024-09-16T14:50:32.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:32.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:50:32.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 10 monclient(hunting): picked mon.noname-a con 0x7f069c0a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:32.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 10 monclient(hunting): picked mon.noname-b con 0x7f069c0a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:32.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 10 monclient(hunting): picked mon.noname-c con 0x7f069c13b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:32.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c09d380 con 0x7f069c0a1b00 2024-09-16T14:50:32.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c11dbf0 con 0x7f069c0a25f0 2024-09-16T14:50:32.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c09d840 con 0x7f069c13b430 2024-09-16T14:50:32.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:32.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 10 monclient(hunting): authenticate will time out at 1414.876455s 2024-09-16T14:50:32.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f069b7fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f069c0a25f0 legacy=0x7f069c13a030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:32.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f069bfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f069c0a1b00 0x7f069c139920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47186/0 (socket says 172.21.15.189:47186) 2024-09-16T14:50:32.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f069bfff640 1 -- 172.21.15.189:0/501893478 learned_addr learned my addr 172.21.15.189:0/501893478 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:32.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575099546 0 0) 0x7f069c09d380 con 0x7f069c0a1b00 2024-09-16T14:50:32.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:32.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:32.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:32.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient(hunting): my global_id is 164430 2024-09-16T14:50:32.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f067c003530 con 0x7f069c0a1b00 2024-09-16T14:50:32.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053416823 0 0) 0x7f069c09d840 con 0x7f069c13b430 2024-09-16T14:50:32.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:32.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:32.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:32.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient(hunting): my global_id is 154685 2024-09-16T14:50:32.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f069c09d380 con 0x7f069c13b430 2024-09-16T14:50:32.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3951498422 0 0) 0x7f067c003530 con 0x7f069c0a1b00 2024-09-16T14:50:32.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f069c09d840 con 0x7f069c0a1b00 2024-09-16T14:50:32.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f068c0044e0 con 0x7f069c0a1b00 2024-09-16T14:50:32.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:32.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:32.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient(hunting): dump: 2024-09-16T14:50:32.778 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:32.779 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:32.779 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:32.779 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:32.779 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:32.779 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:32.779 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:32.779 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:32.779 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:32.779 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:32.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:32.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2682885242 0 0) 0x7f069c09d380 con 0x7f069c13b430 2024-09-16T14:50:32.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f067c003530 con 0x7f069c13b430 2024-09-16T14:50:32.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0694001520 con 0x7f069c13b430 2024-09-16T14:50:32.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:32.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:32.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient(hunting): dump: 2024-09-16T14:50:32.780 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:32.780 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:32.780 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:32.780 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:32.780 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:32.780 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:32.780 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:32.780 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:32.780 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:32.780 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:32.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1734084242 0 0) 0x7f069c09d840 con 0x7f069c0a1b00 2024-09-16T14:50:32.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 >> v1:172.21.15.26:6790/0 conn(0x7f069c13b430 legacy=0x7f069c1d1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:32.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 >> v1:172.21.15.189:6789/0 conn(0x7f069c0a25f0 legacy=0x7f069c13a030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:32.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:50:32.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 monclient: found mon.a 2024-09-16T14:50:32.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:32.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:32.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c11ff30 con 0x7f069c0a1b00 2024-09-16T14:50:32.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient: _finish_auth 0 2024-09-16T14:50:32.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:50:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 5 monclient: authenticate success, global_id 164430 2024-09-16T14:50:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 10 monclient: _renew_subs 2024-09-16T14:50:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 1 -- 172.21.15.189:0/501893478 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c11e520 con 0x7f069c0a1b00 2024-09-16T14:50:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 10 monclient: _renew_subs 2024-09-16T14:50:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 1 -- 172.21.15.189:0/501893478 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f069c13a740 con 0x7f069c0a1b00 2024-09-16T14:50:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f068c004700 con 0x7f069c0a1b00 2024-09-16T14:50:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:32.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f068c004a90 con 0x7f069c0a1b00 2024-09-16T14:50:32.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f069a7fc640 4 set_mon_vals no callback set 2024-09-16T14:50:32.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:32.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:32.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 10 monclient: dump: 2024-09-16T14:50:32.785 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:32.785 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:32.785 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:32.785 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:32.785 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:32.785 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:32.786 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:32.786 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:32.786 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:32.786 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:32.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:32.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:32.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:32.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f06a26c9640 1 -- 172.21.15.189:0/501893478 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0680005050 con 0x7f069c0a1b00 2024-09-16T14:50:32.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f068c004c70 con 0x7f069c0a1b00 2024-09-16T14:50:32.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(226..226 src has 169..226) v4 ==== 11756+0+0 (unknown 3133443365 0 0) 0x7f068c045410 con 0x7f069c0a1b00 2024-09-16T14:50:32.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.760+0000 7f069b7fe640 10 monclient: get_auth_request con 0x7f067c036220 auth_method 0 2024-09-16T14:50:32.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.772+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f068c0147a0 con 0x7f069c0a1b00 2024-09-16T14:50:32.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.772+0000 7f0698ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:32.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.772+0000 7f0698ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:32.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.940+0000 7f06a26c9640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T14:50:32.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.940+0000 7f06a26c9640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T14:50:32.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.940+0000 7f06a26c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:32.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:32.940+0000 7f06a26c9640 1 -- 172.21.15.189:0/501893478 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h12"} v 0) v1 -- 0x7f0680005300 con 0x7f069c0a1b00 2024-09-16T14:50:33.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.568+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h12"}]=0 removed item id -52 name 'h12' from crush map v227) v1 ==== 120+0+0 (unknown 1250119241 0 0) 0x7f068c0145c0 con 0x7f069c0a1b00 2024-09-16T14:50:33.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.568+0000 7f0698ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T14:50:33.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.568+0000 7f0698ff9640 10 monclient: _finish_command 2 = system:0 removed item id -52 name 'h12' from crush map 2024-09-16T14:50:33.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.668+0000 7f06a26c9640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T14:50:33.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.668+0000 7f06a26c9640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T14:50:33.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.668+0000 7f06a26c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:33.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.668+0000 7f06a26c9640 1 -- 172.21.15.189:0/501893478 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h12"} v 0) v1 -- 0x7f0680005a40 con 0x7f069c0a1b00 2024-09-16T14:50:33.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.672+0000 7f0698ff9640 1 -- 172.21.15.189:0/501893478 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h12"}]=0 device 'h12' does not appear in the crush map v227) v1 ==== 120+0+0 (unknown 2968789066 0 0) 0x7f068c015300 con 0x7f069c0a1b00 2024-09-16T14:50:33.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.672+0000 7f0698ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T14:50:33.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.672+0000 7f0698ff9640 10 monclient: _finish_command 3 = system:0 device 'h12' does not appear in the crush map 2024-09-16T14:50:33.672 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h12' does not appear in the crush map 2024-09-16T14:50:33.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.672+0000 7f06a26c9640 1 -- 172.21.15.189:0/501893478 >> v1:172.21.15.189:6812/3196161582 conn(0x7f067c036220 legacy=0x7f067c0386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:33.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.672+0000 7f06a26c9640 10 monclient: shutdown 2024-09-16T14:50:33.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.672+0000 7f06a26c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:33.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.672+0000 7f06a26c9640 1 -- 172.21.15.189:0/501893478 >> v1:172.21.15.26:6789/0 conn(0x7f069c0a1b00 legacy=0x7f069c139920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:33.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.672+0000 7f06a26c9640 1 -- 172.21.15.189:0/501893478 shutdown_connections 2024-09-16T14:50:33.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.672+0000 7f06a26c9640 1 -- 172.21.15.189:0/501893478 >> 172.21.15.189:0/501893478 conn(0x7f069c0a7dd0 msgr2=0x7f069c09ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:33.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.672+0000 7f06a26c9640 1 -- 172.21.15.189:0/501893478 shutdown_connections 2024-09-16T14:50:33.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.672+0000 7f06a26c9640 1 -- 172.21.15.189:0/501893478 wait complete. 2024-09-16T14:50:33.689 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:33.689 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h13 2024-09-16T14:50:33.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f23252640 10 monclient: get_monmap_and_config 2024-09-16T14:50:33.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f23252640 10 monclient: build_initial_monmap 2024-09-16T14:50:33.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f23252640 10 monclient: monmap: 2024-09-16T14:50:33.803 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:33.804 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:33.804 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:33.803991+0000 2024-09-16T14:50:33.804 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:33.803991+0000 2024-09-16T14:50:33.804 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:33.804 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:33.804 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:33.805 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:33.805 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:33.805 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:33.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f23252640 1 Processor -- start 2024-09-16T14:50:33.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f23252640 1 -- start start 2024-09-16T14:50:33.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f23252640 10 monclient: init 2024-09-16T14:50:33.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f23252640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:33.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f23252640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:33.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f23252640 10 monclient(hunting): picked mon.noname-a con 0x7f0f1c09df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:33.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f23252640 10 monclient(hunting): picked mon.noname-b con 0x7f0f1c09e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:33.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f23252640 10 monclient(hunting): picked mon.noname-c con 0x7f0f1c09e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:33.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f23252640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c09ee20 con 0x7f0f1c09df10 2024-09-16T14:50:33.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f23252640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c09f100 con 0x7f0f1c09e740 2024-09-16T14:50:33.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f23252640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c09f3e0 con 0x7f0f1c09e2f0 2024-09-16T14:50:33.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f23252640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:33.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f23252640 10 monclient(hunting): authenticate will time out at 1415.919558s 2024-09-16T14:50:33.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f217ef640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f0f1c09e740 legacy=0x7f0f1c13b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:33.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f1bfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0f1c09e2f0 0x7f0f1c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:49322/0 (socket says 172.21.15.189:49322) 2024-09-16T14:50:33.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.800+0000 7f0f1bfff640 1 -- 172.21.15.189:0/2368260099 learned_addr learned my addr 172.21.15.189:0/2368260099 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:33.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 1 -- 172.21.15.189:0/2368260099 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132100511 0 0) 0x7f0f1c09ee20 con 0x7f0f1c09df10 2024-09-16T14:50:33.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:33.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:33.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:33.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient(hunting): my global_id is 164433 2024-09-16T14:50:33.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 1 -- 172.21.15.189:0/2368260099 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f04003530 con 0x7f0f1c09df10 2024-09-16T14:50:33.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 1 -- 172.21.15.189:0/2368260099 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315594870 0 0) 0x7f0f1c09f3e0 con 0x7f0f1c09e2f0 2024-09-16T14:50:33.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:33.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:33.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:33.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient(hunting): my global_id is 154688 2024-09-16T14:50:33.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 1 -- 172.21.15.189:0/2368260099 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f1c09ee20 con 0x7f0f1c09e2f0 2024-09-16T14:50:33.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 1 -- 172.21.15.189:0/2368260099 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 702974093 0 0) 0x7f0f04003530 con 0x7f0f1c09df10 2024-09-16T14:50:33.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 1 -- 172.21.15.189:0/2368260099 >> v1:172.21.15.26:6790/0 conn(0x7f0f1c09e2f0 legacy=0x7f0f1c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:33.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 1 -- 172.21.15.189:0/2368260099 >> v1:172.21.15.189:6789/0 conn(0x7f0f1c09e740 legacy=0x7f0f1c13b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:33.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:33.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 1 monclient: found mon.noname-a 2024-09-16T14:50:33.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:33.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:33.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 1 -- 172.21.15.189:0/2368260099 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c09c690 con 0x7f0f1c09df10 2024-09-16T14:50:33.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:33.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:33.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:33.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 1 -- 172.21.15.189:0/2368260099 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0f0c0019a0 con 0x7f0f1c09df10 2024-09-16T14:50:33.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:33.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:33.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient: dump: 2024-09-16T14:50:33.826 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:33.826 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:33.826 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:33.826 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:33.826 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:33.827 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:33.827 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:33.827 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:33.827 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:33.827 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:33.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 5 monclient: authenticate success, global_id 164433 2024-09-16T14:50:33.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:33.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 1 -- 172.21.15.189:0/2368260099 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f0c001c80 con 0x7f0f1c09df10 2024-09-16T14:50:33.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:33.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:33.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 4 set_mon_vals no callback set 2024-09-16T14:50:33.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 1 -- 172.21.15.189:0/2368260099 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0f0c002010 con 0x7f0f1c09df10 2024-09-16T14:50:33.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:33.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:33.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1b7fe640 10 monclient: dump: 2024-09-16T14:50:33.829 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:33.830 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:33.830 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:33.830 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:33.830 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:33.830 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:33.830 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:33.831 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:33.831 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:33.831 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:33.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 10 monclient: shutdown 2024-09-16T14:50:33.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:33.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 1 -- 172.21.15.189:0/2368260099 >> v1:172.21.15.26:6789/0 conn(0x7f0f1c09df10 legacy=0x7f0f1c09d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:33.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 1 -- 172.21.15.189:0/2368260099 shutdown_connections 2024-09-16T14:50:33.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 1 -- 172.21.15.189:0/2368260099 >> 172.21.15.189:0/2368260099 conn(0x7f0f1c0a3320 msgr2=0x7f0f1c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:33.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 1 -- 172.21.15.189:0/2368260099 shutdown_connections 2024-09-16T14:50:33.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 1 -- 172.21.15.189:0/2368260099 wait complete. 2024-09-16T14:50:33.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 10 monclient: build_initial_monmap 2024-09-16T14:50:33.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 10 monclient: monmap: 2024-09-16T14:50:33.833 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:33.833 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:33.833 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:33.833 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:33.833 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:33.834 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:33.834 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:33.834 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:33.834 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:33.834 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:33.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 1 Processor -- start 2024-09-16T14:50:33.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 1 -- start start 2024-09-16T14:50:33.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 10 monclient: init 2024-09-16T14:50:33.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:33.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:33.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 10 monclient(hunting): picked mon.noname-c con 0x7f0f1c09df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:33.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 10 monclient(hunting): picked mon.noname-b con 0x7f0f1c09e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:33.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 10 monclient(hunting): picked mon.noname-a con 0x7f0f1c09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:33.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c06faa0 con 0x7f0f1c09e740 2024-09-16T14:50:33.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c06b380 con 0x7f0f1c09e2f0 2024-09-16T14:50:33.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c09b010 con 0x7f0f1c09df10 2024-09-16T14:50:33.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:33.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f23252640 10 monclient(hunting): authenticate will time out at 1415.923187s 2024-09-16T14:50:33.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f1bfff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f0f1c09e2f0 legacy=0x7f0f1c09a900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:33.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f20fee640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0f1c09df10 0x7f0f1c09a1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:49336/0 (socket says 172.21.15.189:49336) 2024-09-16T14:50:33.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f20fee640 1 -- 172.21.15.189:0/1364351433 learned_addr learned my addr 172.21.15.189:0/1364351433 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:33.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.804+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618994989 0 0) 0x7f0f140039c0 con 0x7f0f1c09e740 2024-09-16T14:50:33.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient(hunting): _negotiate 2024-09-16T14:50:33.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:33.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:33.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient(hunting): my global_id is 164436 2024-09-16T14:50:33.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f08003530 con 0x7f0f1c09e740 2024-09-16T14:50:33.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080519190 0 0) 0x7f0f1c09b010 con 0x7f0f1c09df10 2024-09-16T14:50:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient(hunting): _negotiate 2024-09-16T14:50:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient(hunting): my global_id is 154691 2024-09-16T14:50:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f140039c0 con 0x7f0f1c09df10 2024-09-16T14:50:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2895274204 0 0) 0x7f0f08003530 con 0x7f0f1c09e740 2024-09-16T14:50:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f1c09b010 con 0x7f0f1c09e740 2024-09-16T14:50:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0f14003fb0 con 0x7f0f1c09e740 2024-09-16T14:50:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient(hunting): dump: 2024-09-16T14:50:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:33.841 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:33.842 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:33.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 279663059 0 0) 0x7f0f140039c0 con 0x7f0f1c09df10 2024-09-16T14:50:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f08003530 con 0x7f0f1c09df10 2024-09-16T14:50:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0f0c001960 con 0x7f0f1c09df10 2024-09-16T14:50:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient(hunting): dump: 2024-09-16T14:50:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:33.844 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:33.845 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1687816219 0 0) 0x7f0f1c09b010 con 0x7f0f1c09e740 2024-09-16T14:50:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 >> v1:172.21.15.26:6790/0 conn(0x7f0f1c09df10 legacy=0x7f0f1c09a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 >> v1:172.21.15.189:6789/0 conn(0x7f0f1c09e2f0 legacy=0x7f0f1c09a900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient: _finish_hunting 0 2024-09-16T14:50:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 monclient: found mon.a 2024-09-16T14:50:33.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c06cea0 con 0x7f0f1c09e740 2024-09-16T14:50:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient: _finish_auth 0 2024-09-16T14:50:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient: _check_auth_tickets 2024-09-16T14:50:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f23252640 5 monclient: authenticate success, global_id 164436 2024-09-16T14:50:33.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f23252640 10 monclient: _renew_subs 2024-09-16T14:50:33.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f23252640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:33.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f23252640 1 -- 172.21.15.189:0/1364351433 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1c13b6f0 con 0x7f0f1c09e740 2024-09-16T14:50:33.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f23252640 10 monclient: _renew_subs 2024-09-16T14:50:33.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f23252640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:33.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f23252640 1 -- 172.21.15.189:0/1364351433 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1c13bb10 con 0x7f0f1c09e740 2024-09-16T14:50:33.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f140041d0 con 0x7f0f1c09e740 2024-09-16T14:50:33.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:33.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f1affd640 4 set_mon_vals no callback set 2024-09-16T14:50:33.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0f14004560 con 0x7f0f1c09e740 2024-09-16T14:50:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 10 monclient: dump: 2024-09-16T14:50:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:33.850 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:33.852 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f23252640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f23252640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f23252640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f23252640 1 -- 172.21.15.189:0/1364351433 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0efc005050 con 0x7f0f1c09e740 2024-09-16T14:50:33.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0f14004740 con 0x7f0f1c09e740 2024-09-16T14:50:33.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(227..227 src has 169..227) v4 ==== 11543+0+0 (unknown 3806696230 0 0) 0x7f0f14044dd0 con 0x7f0f1c09e740 2024-09-16T14:50:33.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.808+0000 7f0f20fee640 10 monclient: get_auth_request con 0x7f0f08036220 auth_method 0 2024-09-16T14:50:33.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.820+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0f14014220 con 0x7f0f1c09e740 2024-09-16T14:50:33.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.820+0000 7f0f197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:33.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.820+0000 7f0f197fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:33.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.976+0000 7f0f23252640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T14:50:33.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.976+0000 7f0f23252640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T14:50:33.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.976+0000 7f0f23252640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:33.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:33.976+0000 7f0f23252640 1 -- 172.21.15.189:0/1364351433 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h13"} v 0) v1 -- 0x7f0efc005300 con 0x7f0f1c09e740 2024-09-16T14:50:34.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.572+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h13"}]=0 removed item id -55 name 'h13' from crush map v228) v1 ==== 120+0+0 (unknown 1397809998 0 0) 0x7f0f140128c0 con 0x7f0f1c09e740 2024-09-16T14:50:34.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.572+0000 7f0f197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T14:50:34.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.572+0000 7f0f197fa640 10 monclient: _finish_command 2 = system:0 removed item id -55 name 'h13' from crush map 2024-09-16T14:50:34.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.668+0000 7f0f23252640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T14:50:34.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.668+0000 7f0f23252640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T14:50:34.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.668+0000 7f0f23252640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:34.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.668+0000 7f0f23252640 1 -- 172.21.15.189:0/1364351433 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h13"} v 0) v1 -- 0x7f0efc005a40 con 0x7f0f1c09e740 2024-09-16T14:50:34.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.672+0000 7f0f197fa640 1 -- 172.21.15.189:0/1364351433 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h13"}]=0 device 'h13' does not appear in the crush map v228) v1 ==== 120+0+0 (unknown 710364041 0 0) 0x7f0f14014040 con 0x7f0f1c09e740 2024-09-16T14:50:34.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.672+0000 7f0f197fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T14:50:34.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.672+0000 7f0f197fa640 10 monclient: _finish_command 3 = system:0 device 'h13' does not appear in the crush map 2024-09-16T14:50:34.673 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h13' does not appear in the crush map 2024-09-16T14:50:34.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.672+0000 7f0f23252640 1 -- 172.21.15.189:0/1364351433 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0f08036220 legacy=0x7f0f080386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:34.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.672+0000 7f0f23252640 10 monclient: shutdown 2024-09-16T14:50:34.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.672+0000 7f0f23252640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:34.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.672+0000 7f0f23252640 1 -- 172.21.15.189:0/1364351433 >> v1:172.21.15.26:6789/0 conn(0x7f0f1c09e740 legacy=0x7f0f1c1d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:34.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.672+0000 7f0f23252640 1 -- 172.21.15.189:0/1364351433 shutdown_connections 2024-09-16T14:50:34.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.672+0000 7f0f23252640 1 -- 172.21.15.189:0/1364351433 >> 172.21.15.189:0/1364351433 conn(0x7f0f1c0a3320 msgr2=0x7f0f1c06e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:34.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.672+0000 7f0f23252640 1 -- 172.21.15.189:0/1364351433 shutdown_connections 2024-09-16T14:50:34.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.672+0000 7f0f23252640 1 -- 172.21.15.189:0/1364351433 wait complete. 2024-09-16T14:50:34.689 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:34.689 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h14 2024-09-16T14:50:34.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.804+0000 7f159e8c0640 10 monclient: get_monmap_and_config 2024-09-16T14:50:34.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.804+0000 7f159e8c0640 10 monclient: build_initial_monmap 2024-09-16T14:50:34.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.804+0000 7f159e8c0640 10 monclient: monmap: 2024-09-16T14:50:34.807 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:34.807 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:34.807 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:34.808301+0000 2024-09-16T14:50:34.807 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:34.808301+0000 2024-09-16T14:50:34.808 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:34.808 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:34.808 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:34.808 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:34.808 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:34.808 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:34.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.804+0000 7f159e8c0640 1 Processor -- start 2024-09-16T14:50:34.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.804+0000 7f159e8c0640 1 -- start start 2024-09-16T14:50:34.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 10 monclient: init 2024-09-16T14:50:34.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:34.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:50:34.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 10 monclient(hunting): picked mon.noname-a con 0x7f1598071350 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:34.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 10 monclient(hunting): picked mon.noname-c con 0x7f159809dc40 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:34.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 10 monclient(hunting): picked mon.noname-b con 0x7f159806d7e0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:34.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1598071e40 con 0x7f1598071350 2024-09-16T14:50:34.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15980a0390 con 0x7f159809dc40 2024-09-16T14:50:34.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159806f580 con 0x7f159806d7e0 2024-09-16T14:50:34.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:34.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 10 monclient(hunting): authenticate will time out at 1416.924708s 2024-09-16T14:50:34.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f15977fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f159809dc40 legacy=0x7f15981368f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:34.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159ce5d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f159806d7e0 0x7f159813a020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:49346/0 (socket says 172.21.15.189:49346) 2024-09-16T14:50:34.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159ce5d640 1 -- 172.21.15.189:0/143906037 learned_addr learned my addr 172.21.15.189:0/143906037 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:34.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 1 -- 172.21.15.189:0/143906037 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520129208 0 0) 0x7f159806f580 con 0x7f159806d7e0 2024-09-16T14:50:34.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:34.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:34.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:34.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient(hunting): my global_id is 154694 2024-09-16T14:50:34.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 1 -- 172.21.15.189:0/143906037 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1580003530 con 0x7f159806d7e0 2024-09-16T14:50:34.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 1 -- 172.21.15.189:0/143906037 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189912245 0 0) 0x7f1598071e40 con 0x7f1598071350 2024-09-16T14:50:34.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:34.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:34.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:34.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient(hunting): my global_id is 164439 2024-09-16T14:50:34.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 1 -- 172.21.15.189:0/143906037 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f159806f580 con 0x7f1598071350 2024-09-16T14:50:34.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 1 -- 172.21.15.189:0/143906037 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 490894956 0 0) 0x7f1580003530 con 0x7f159806d7e0 2024-09-16T14:50:34.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 1 -- 172.21.15.189:0/143906037 >> v1:172.21.15.189:6789/0 conn(0x7f159809dc40 legacy=0x7f15981368f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:34.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 1 -- 172.21.15.189:0/143906037 >> v1:172.21.15.26:6789/0 conn(0x7f1598071350 legacy=0x7f1598071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:34.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:34.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 1 monclient: found mon.noname-b 2024-09-16T14:50:34.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:34.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:34.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 1 -- 172.21.15.189:0/143906037 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159806f860 con 0x7f159806d7e0 2024-09-16T14:50:34.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:34.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:34.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:34.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 1 -- 172.21.15.189:0/143906037 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f15900019a0 con 0x7f159806d7e0 2024-09-16T14:50:34.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:34.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:34.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient: dump: 2024-09-16T14:50:34.822 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:34.822 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:34.822 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:34.822 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:34.822 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:34.822 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:34.822 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:34.822 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:34.823 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:34.823 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:34.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 5 monclient: authenticate success, global_id 154694 2024-09-16T14:50:34.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:34.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 1 -- 172.21.15.189:0/143906037 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1590001bc0 con 0x7f159806d7e0 2024-09-16T14:50:34.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:34.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:34.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 4 set_mon_vals no callback set 2024-09-16T14:50:34.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 1 -- 172.21.15.189:0/143906037 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1590001ed0 con 0x7f159806d7e0 2024-09-16T14:50:34.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:34.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:34.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f1596ffd640 10 monclient: dump: 2024-09-16T14:50:34.824 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:34.824 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:34.824 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:34.824 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:34.824 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:34.825 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:34.825 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:34.825 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:34.825 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:34.825 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:34.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 10 monclient: shutdown 2024-09-16T14:50:34.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:34.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 1 -- 172.21.15.189:0/143906037 >> v1:172.21.15.26:6790/0 conn(0x7f159806d7e0 legacy=0x7f159813a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:34.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 1 -- 172.21.15.189:0/143906037 shutdown_connections 2024-09-16T14:50:34.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 1 -- 172.21.15.189:0/143906037 >> 172.21.15.189:0/143906037 conn(0x7f15980a0690 msgr2=0x7f15980a0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:34.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 1 -- 172.21.15.189:0/143906037 shutdown_connections 2024-09-16T14:50:34.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 1 -- 172.21.15.189:0/143906037 wait complete. 2024-09-16T14:50:34.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 10 monclient: build_initial_monmap 2024-09-16T14:50:34.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.808+0000 7f159e8c0640 10 monclient: monmap: 2024-09-16T14:50:34.826 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:34.826 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:34.826 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:34.826 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:34.826 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:34.827 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:34.827 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:34.827 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:34.827 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:34.827 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:34.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 1 Processor -- start 2024-09-16T14:50:34.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 1 -- start start 2024-09-16T14:50:34.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 10 monclient: init 2024-09-16T14:50:34.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:34.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:34.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 10 monclient(hunting): picked mon.noname-c con 0x7f159806d7e0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:34.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 10 monclient(hunting): picked mon.noname-b con 0x7f1598071350 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:34.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 10 monclient(hunting): picked mon.noname-a con 0x7f159809dc40 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159811f5d0 con 0x7f159809dc40 2024-09-16T14:50:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15981316f0 con 0x7f1598071350 2024-09-16T14:50:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15980a2140 con 0x7f159806d7e0 2024-09-16T14:50:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 10 monclient(hunting): authenticate will time out at 1416.928625s 2024-09-16T14:50:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f15977fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f1598071350 legacy=0x7f159811be80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1597fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f159806d7e0 0x7f159811b770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:49348/0 (socket says 172.21.15.189:49348) 2024-09-16T14:50:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1597fff640 1 -- 172.21.15.189:0/1498063229 learned_addr learned my addr 172.21.15.189:0/1498063229 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572517383 0 0) 0x7f15980a2140 con 0x7f159806d7e0 2024-09-16T14:50:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:34.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient(hunting): my global_id is 154697 2024-09-16T14:50:34.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1584003530 con 0x7f159806d7e0 2024-09-16T14:50:34.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1132122203 0 0) 0x7f159811f5d0 con 0x7f159809dc40 2024-09-16T14:50:34.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:34.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:34.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:34.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient(hunting): my global_id is 164442 2024-09-16T14:50:34.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15980a2140 con 0x7f159809dc40 2024-09-16T14:50:34.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 238152716 0 0) 0x7f1584003530 con 0x7f159806d7e0 2024-09-16T14:50:34.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f159811f5d0 con 0x7f159806d7e0 2024-09-16T14:50:34.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1588001980 con 0x7f159806d7e0 2024-09-16T14:50:34.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:34.830 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:34.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient(hunting): dump: 2024-09-16T14:50:34.833 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:34.833 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:34.833 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:34.833 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:34.833 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:34.833 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:34.833 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:34.833 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:34.833 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:34.833 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:34.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:34.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1220180057 0 0) 0x7f15980a2140 con 0x7f159809dc40 2024-09-16T14:50:34.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1584003530 con 0x7f159809dc40 2024-09-16T14:50:34.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f15900018a0 con 0x7f159809dc40 2024-09-16T14:50:34.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:34.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:34.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient(hunting): dump: 2024-09-16T14:50:34.834 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:34.834 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:34.834 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:34.834 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:34.834 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:34.834 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:34.834 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:34.834 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:34.835 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:34.835 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:34.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 215123523 0 0) 0x7f159811f5d0 con 0x7f159806d7e0 2024-09-16T14:50:34.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 >> v1:172.21.15.189:6789/0 conn(0x7f1598071350 legacy=0x7f159811be80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:34.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 >> v1:172.21.15.26:6789/0 conn(0x7f159809dc40 legacy=0x7f159811c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:34.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:50:34.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 monclient: found mon.c 2024-09-16T14:50:34.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:34.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:34.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159811f2f0 con 0x7f159806d7e0 2024-09-16T14:50:34.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient: _finish_auth 0 2024-09-16T14:50:34.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:50:34.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:34.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 5 monclient: authenticate success, global_id 154697 2024-09-16T14:50:34.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 10 monclient: _renew_subs 2024-09-16T14:50:34.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:34.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 1 -- 172.21.15.189:0/1498063229 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159811d720 con 0x7f159806d7e0 2024-09-16T14:50:34.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 10 monclient: _renew_subs 2024-09-16T14:50:34.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:34.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 1 -- 172.21.15.189:0/1498063229 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15981ce6e0 con 0x7f159806d7e0 2024-09-16T14:50:34.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1588003620 con 0x7f159806d7e0 2024-09-16T14:50:34.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:34.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f15880039b0 con 0x7f159806d7e0 2024-09-16T14:50:34.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:34.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f15967fc640 4 set_mon_vals no callback set 2024-09-16T14:50:34.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:34.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 10 monclient: dump: 2024-09-16T14:50:34.837 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:34.837 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:34.837 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:34.837 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:34.837 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:34.837 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:34.837 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:34.837 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:34.837 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:34.838 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:34.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:34.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:34.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:34.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f159e8c0640 1 -- 172.21.15.189:0/1498063229 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1578005050 con 0x7f159806d7e0 2024-09-16T14:50:34.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.812+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1588003b90 con 0x7f159806d7e0 2024-09-16T14:50:34.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.816+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(228..228 src has 169..228) v4 ==== 11330+0+0 (unknown 3307652809 0 0) 0x7f1588044f70 con 0x7f159806d7e0 2024-09-16T14:50:34.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.816+0000 7f15977fe640 10 monclient: get_auth_request con 0x7f1584036290 auth_method 0 2024-09-16T14:50:34.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.828+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f15880146c0 con 0x7f159806d7e0 2024-09-16T14:50:34.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.828+0000 7f1594ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:34.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.828+0000 7f1594ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:34.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.988+0000 7f159e8c0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T14:50:34.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.988+0000 7f159e8c0640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T14:50:34.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.988+0000 7f159e8c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:34.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:34.988+0000 7f159e8c0640 1 -- 172.21.15.189:0/1498063229 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h14"} v 0) v1 -- 0x7f1578005300 con 0x7f159806d7e0 2024-09-16T14:50:35.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.580+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h14"}]=0 removed item id -58 name 'h14' from crush map v229) v1 ==== 120+0+0 (unknown 1605276742 0 0) 0x7f1588012b30 con 0x7f159806d7e0 2024-09-16T14:50:35.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.580+0000 7f1594ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T14:50:35.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.580+0000 7f1594ff9640 10 monclient: _finish_command 2 = system:0 removed item id -58 name 'h14' from crush map 2024-09-16T14:50:35.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.684+0000 7f159e8c0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T14:50:35.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.684+0000 7f159e8c0640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T14:50:35.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.684+0000 7f159e8c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:35.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.684+0000 7f159e8c0640 1 -- 172.21.15.189:0/1498063229 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h14"} v 0) v1 -- 0x7f1578005a40 con 0x7f159806d7e0 2024-09-16T14:50:35.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.684+0000 7f1594ff9640 1 -- 172.21.15.189:0/1498063229 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h14"}]=0 device 'h14' does not appear in the crush map v229) v1 ==== 120+0+0 (unknown 105924795 0 0) 0x7f15880142b0 con 0x7f159806d7e0 2024-09-16T14:50:35.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.684+0000 7f1594ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T14:50:35.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.684+0000 7f1594ff9640 10 monclient: _finish_command 3 = system:0 device 'h14' does not appear in the crush map 2024-09-16T14:50:35.686 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h14' does not appear in the crush map 2024-09-16T14:50:35.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.688+0000 7f159e8c0640 1 -- 172.21.15.189:0/1498063229 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1584036290 legacy=0x7f1584038750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:35.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.688+0000 7f159e8c0640 10 monclient: shutdown 2024-09-16T14:50:35.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.688+0000 7f159e8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:35.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.688+0000 7f159e8c0640 1 -- 172.21.15.189:0/1498063229 >> v1:172.21.15.26:6790/0 conn(0x7f159806d7e0 legacy=0x7f159811b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:35.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.688+0000 7f159e8c0640 1 -- 172.21.15.189:0/1498063229 shutdown_connections 2024-09-16T14:50:35.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.688+0000 7f159e8c0640 1 -- 172.21.15.189:0/1498063229 >> 172.21.15.189:0/1498063229 conn(0x7f15980a0690 msgr2=0x7f1598131c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:35.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.688+0000 7f159e8c0640 1 -- 172.21.15.189:0/1498063229 shutdown_connections 2024-09-16T14:50:35.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.688+0000 7f159e8c0640 1 -- 172.21.15.189:0/1498063229 wait complete. 2024-09-16T14:50:35.702 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:35.702 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h15 2024-09-16T14:50:35.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.816+0000 7fd7836e9640 10 monclient: get_monmap_and_config 2024-09-16T14:50:35.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.816+0000 7fd7836e9640 10 monclient: build_initial_monmap 2024-09-16T14:50:35.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.816+0000 7fd7836e9640 10 monclient: monmap: 2024-09-16T14:50:35.832 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:35.832 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:35.832 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:35.820516+0000 2024-09-16T14:50:35.833 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:35.820516+0000 2024-09-16T14:50:35.833 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:35.833 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:35.833 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:35.833 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:35.833 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:35.834 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:35.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.816+0000 7fd7836e9640 1 Processor -- start 2024-09-16T14:50:35.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.816+0000 7fd7836e9640 1 -- start start 2024-09-16T14:50:35.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.816+0000 7fd7836e9640 10 monclient: init 2024-09-16T14:50:35.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:35.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:35.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient(hunting): picked mon.noname-b con 0x7fd77c09df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:35.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient(hunting): picked mon.noname-c con 0x7fd77c09e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:35.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient(hunting): picked mon.noname-a con 0x7fd77c09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:35.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77c09ee20 con 0x7fd77c09e740 2024-09-16T14:50:35.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77c09f100 con 0x7fd77c09e2f0 2024-09-16T14:50:35.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77c09f3e0 con 0x7fd77c09df10 2024-09-16T14:50:35.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:35.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient(hunting): authenticate will time out at 1417.936160s 2024-09-16T14:50:35.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd780c84640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fd77c09e2f0 legacy=0x7fd77c1378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:35.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd781485640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd77c09df10 0x7fd77c09d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:49358/0 (socket says 172.21.15.189:49358) 2024-09-16T14:50:35.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd781c86640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd77c09e740 0x7fd77c13b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47218/0 (socket says 172.21.15.189:47218) 2024-09-16T14:50:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd781485640 1 -- 172.21.15.189:0/1858144098 learned_addr learned my addr 172.21.15.189:0/1858144098 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 1 -- 172.21.15.189:0/1858144098 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204756337 0 0) 0x7fd77c09f3e0 con 0x7fd77c09df10 2024-09-16T14:50:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:35.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient(hunting): my global_id is 154700 2024-09-16T14:50:35.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 1 -- 172.21.15.189:0/1858144098 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd768003530 con 0x7fd77c09df10 2024-09-16T14:50:35.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 1 -- 172.21.15.189:0/1858144098 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057995712 0 0) 0x7fd77c09ee20 con 0x7fd77c09e740 2024-09-16T14:50:35.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:35.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:35.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:35.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient(hunting): my global_id is 164445 2024-09-16T14:50:35.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 1 -- 172.21.15.189:0/1858144098 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd77c09f3e0 con 0x7fd77c09e740 2024-09-16T14:50:35.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 1 -- 172.21.15.189:0/1858144098 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2384603910 0 0) 0x7fd768003530 con 0x7fd77c09df10 2024-09-16T14:50:35.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 1 -- 172.21.15.189:0/1858144098 >> v1:172.21.15.189:6789/0 conn(0x7fd77c09e2f0 legacy=0x7fd77c1378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:35.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 1 -- 172.21.15.189:0/1858144098 >> v1:172.21.15.26:6789/0 conn(0x7fd77c09e740 legacy=0x7fd77c13b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:35.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient: _finish_hunting 0 2024-09-16T14:50:35.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 1 monclient: found mon.noname-b 2024-09-16T14:50:35.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:35.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:35.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 1 -- 172.21.15.189:0/1858144098 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77c09c690 con 0x7fd77c09df10 2024-09-16T14:50:35.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient: _finish_auth 0 2024-09-16T14:50:35.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient: _check_auth_tickets 2024-09-16T14:50:35.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:35.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 1 -- 172.21.15.189:0/1858144098 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd778001620 con 0x7fd77c09df10 2024-09-16T14:50:35.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 5 monclient: authenticate success, global_id 154700 2024-09-16T14:50:35.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:35.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:35.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:35.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient: dump: 2024-09-16T14:50:35.842 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:35.842 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:35.842 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:35.844 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 1 -- 172.21.15.189:0/1858144098 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd778001880 con 0x7fd77c09df10 2024-09-16T14:50:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 4 set_mon_vals no callback set 2024-09-16T14:50:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 1 -- 172.21.15.189:0/1858144098 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd778001c50 con 0x7fd77c09df10 2024-09-16T14:50:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd773fff640 10 monclient: dump: 2024-09-16T14:50:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:35.847 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient: shutdown 2024-09-16T14:50:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 1 -- 172.21.15.189:0/1858144098 >> v1:172.21.15.26:6790/0 conn(0x7fd77c09df10 legacy=0x7fd77c09d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 1 -- 172.21.15.189:0/1858144098 shutdown_connections 2024-09-16T14:50:35.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 1 -- 172.21.15.189:0/1858144098 >> 172.21.15.189:0/1858144098 conn(0x7fd77c0a3320 msgr2=0x7fd77c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:35.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 1 -- 172.21.15.189:0/1858144098 shutdown_connections 2024-09-16T14:50:35.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 1 -- 172.21.15.189:0/1858144098 wait complete. 2024-09-16T14:50:35.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient: build_initial_monmap 2024-09-16T14:50:35.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient: monmap: 2024-09-16T14:50:35.848 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:35.849 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:35.849 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:35.849 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:35.849 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:35.849 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:35.849 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:35.849 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:35.850 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:35.850 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:35.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 1 Processor -- start 2024-09-16T14:50:35.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 1 -- start start 2024-09-16T14:50:35.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient: init 2024-09-16T14:50:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient(hunting): picked mon.noname-b con 0x7fd77c09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient(hunting): picked mon.noname-a con 0x7fd77c09e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient(hunting): picked mon.noname-c con 0x7fd77c09e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77c0a1030 con 0x7fd77c09e2f0 2024-09-16T14:50:35.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77c06b380 con 0x7fd77c09df10 2024-09-16T14:50:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77c11d1e0 con 0x7fd77c09e740 2024-09-16T14:50:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd7836e9640 10 monclient(hunting): authenticate will time out at 1417.939965s 2024-09-16T14:50:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.820+0000 7fd781485640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fd77c09df10 legacy=0x7fd77c12f820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd781c86640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd77c09e740 0x7fd77c1cd450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:49372/0 (socket says 172.21.15.189:49372) 2024-09-16T14:50:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd781c86640 1 -- 172.21.15.189:0/2835660810 learned_addr learned my addr 172.21.15.189:0/2835660810 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375961532 0 0) 0x7fd77c0a1030 con 0x7fd77c09e2f0 2024-09-16T14:50:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient(hunting): my global_id is 164448 2024-09-16T14:50:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd760003530 con 0x7fd77c09e2f0 2024-09-16T14:50:35.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174739056 0 0) 0x7fd77c11d1e0 con 0x7fd77c09e740 2024-09-16T14:50:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient(hunting): my global_id is 154703 2024-09-16T14:50:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd77c0a1030 con 0x7fd77c09e740 2024-09-16T14:50:35.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1245970457 0 0) 0x7fd760003530 con 0x7fd77c09e2f0 2024-09-16T14:50:35.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd77c11d1e0 con 0x7fd77c09e2f0 2024-09-16T14:50:35.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd76c0040d0 con 0x7fd77c09e2f0 2024-09-16T14:50:35.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:35.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:35.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient(hunting): dump: 2024-09-16T14:50:35.857 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:35.857 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:35.857 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:35.857 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:35.857 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:35.857 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:35.857 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:35.858 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:35.858 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:35.858 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:35.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:35.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1697029700 0 0) 0x7fd77c0a1030 con 0x7fd77c09e740 2024-09-16T14:50:35.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd760003530 con 0x7fd77c09e740 2024-09-16T14:50:35.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd774001d60 con 0x7fd77c09e740 2024-09-16T14:50:35.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:35.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:35.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient(hunting): dump: 2024-09-16T14:50:35.859 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:35.859 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:35.860 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:35.860 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:35.860 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:35.860 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:35.860 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:35.860 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:35.860 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:35.861 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:35.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 254611903 0 0) 0x7fd77c11d1e0 con 0x7fd77c09e2f0 2024-09-16T14:50:35.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 >> v1:172.21.15.26:6790/0 conn(0x7fd77c09e740 legacy=0x7fd77c1cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:35.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 >> v1:172.21.15.189:6789/0 conn(0x7fd77c09df10 legacy=0x7fd77c12f820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:35.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:50:35.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 monclient: found mon.a 2024-09-16T14:50:35.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:35.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:35.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77c11aea0 con 0x7fd77c09e2f0 2024-09-16T14:50:35.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient: _finish_auth 0 2024-09-16T14:50:35.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:35.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:35.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd7836e9640 5 monclient: authenticate success, global_id 164448 2024-09-16T14:50:35.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd7836e9640 10 monclient: _renew_subs 2024-09-16T14:50:35.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd7836e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:35.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd7836e9640 1 -- 172.21.15.189:0/2835660810 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd77c130490 con 0x7fd77c09e2f0 2024-09-16T14:50:35.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd7836e9640 10 monclient: _renew_subs 2024-09-16T14:50:35.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd7836e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:35.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd7836e9640 1 -- 172.21.15.189:0/2835660810 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd77c11b7d0 con 0x7fd77c09e2f0 2024-09-16T14:50:35.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd76c0042f0 con 0x7fd77c09e2f0 2024-09-16T14:50:35.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:35.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd76c004600 con 0x7fd77c09e2f0 2024-09-16T14:50:35.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:35.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:35.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd7737fe640 4 set_mon_vals no callback set 2024-09-16T14:50:35.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 10 monclient: dump: 2024-09-16T14:50:35.864 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:35.864 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:35.865 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:35.865 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:35.865 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:35.865 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:35.865 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:35.865 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:35.865 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:35.865 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:35.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd7836e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:35.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd7836e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:35.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd7836e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:35.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd7836e9640 1 -- 172.21.15.189:0/2835660810 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd75c005050 con 0x7fd77c09e2f0 2024-09-16T14:50:35.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd76c0047e0 con 0x7fd77c09e2f0 2024-09-16T14:50:35.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(229..229 src has 169..229) v4 ==== 11117+0+0 (unknown 569416299 0 0) 0x7fd76c044d60 con 0x7fd77c09e2f0 2024-09-16T14:50:35.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.824+0000 7fd781485640 10 monclient: get_auth_request con 0x7fd7600361d0 auth_method 0 2024-09-16T14:50:35.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.836+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd76c0142f0 con 0x7fd77c09e2f0 2024-09-16T14:50:35.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.840+0000 7fd771ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:35.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:35.840+0000 7fd771ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:36.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.000+0000 7fd7836e9640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T14:50:36.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.000+0000 7fd7836e9640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T14:50:36.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.000+0000 7fd7836e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:36.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.000+0000 7fd7836e9640 1 -- 172.21.15.189:0/2835660810 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h15"} v 0) v1 -- 0x7fd75c005300 con 0x7fd77c09e2f0 2024-09-16T14:50:36.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.592+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h15"}]=0 removed item id -61 name 'h15' from crush map v230) v1 ==== 120+0+0 (unknown 3006135946 0 0) 0x7fd76c012990 con 0x7fd77c09e2f0 2024-09-16T14:50:36.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.592+0000 7fd771ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T14:50:36.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.592+0000 7fd771ffb640 10 monclient: _finish_command 2 = system:0 removed item id -61 name 'h15' from crush map 2024-09-16T14:50:36.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.696+0000 7fd7836e9640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T14:50:36.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.696+0000 7fd7836e9640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T14:50:36.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.696+0000 7fd7836e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:36.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.696+0000 7fd7836e9640 1 -- 172.21.15.189:0/2835660810 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h15"} v 0) v1 -- 0x7fd75c005a40 con 0x7fd77c09e2f0 2024-09-16T14:50:36.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.696+0000 7fd771ffb640 1 -- 172.21.15.189:0/2835660810 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h15"}]=0 device 'h15' does not appear in the crush map v230) v1 ==== 120+0+0 (unknown 2820579632 0 0) 0x7fd76c014110 con 0x7fd77c09e2f0 2024-09-16T14:50:36.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.696+0000 7fd771ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T14:50:36.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.696+0000 7fd771ffb640 10 monclient: _finish_command 3 = system:0 device 'h15' does not appear in the crush map 2024-09-16T14:50:36.699 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h15' does not appear in the crush map 2024-09-16T14:50:36.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.700+0000 7fd7836e9640 1 -- 172.21.15.189:0/2835660810 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd7600361d0 legacy=0x7fd760038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:36.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.700+0000 7fd7836e9640 10 monclient: shutdown 2024-09-16T14:50:36.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.700+0000 7fd7836e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:36.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.700+0000 7fd7836e9640 1 -- 172.21.15.189:0/2835660810 >> v1:172.21.15.26:6789/0 conn(0x7fd77c09e2f0 legacy=0x7fd77c1c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:36.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.700+0000 7fd7836e9640 1 -- 172.21.15.189:0/2835660810 shutdown_connections 2024-09-16T14:50:36.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.700+0000 7fd7836e9640 1 -- 172.21.15.189:0/2835660810 >> 172.21.15.189:0/2835660810 conn(0x7fd77c0a3320 msgr2=0x7fd77c131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:36.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.700+0000 7fd7836e9640 1 -- 172.21.15.189:0/2835660810 shutdown_connections 2024-09-16T14:50:36.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.700+0000 7fd7836e9640 1 -- 172.21.15.189:0/2835660810 wait complete. 2024-09-16T14:50:36.717 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:36.717 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h16 2024-09-16T14:50:36.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 10 monclient: get_monmap_and_config 2024-09-16T14:50:36.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 10 monclient: build_initial_monmap 2024-09-16T14:50:36.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 10 monclient: monmap: 2024-09-16T14:50:36.833 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:36.833 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:36.833 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:36.834680+0000 2024-09-16T14:50:36.833 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:36.834680+0000 2024-09-16T14:50:36.833 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:36.834 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:36.834 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:36.834 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:36.834 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:36.834 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:36.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 1 Processor -- start 2024-09-16T14:50:36.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 1 -- start start 2024-09-16T14:50:36.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 10 monclient: init 2024-09-16T14:50:36.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:36.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:36.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 10 monclient(hunting): picked mon.noname-c con 0x7f471c09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:36.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 10 monclient(hunting): picked mon.noname-b con 0x7f471c09e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:36.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 10 monclient(hunting): picked mon.noname-a con 0x7f471c09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:36.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471c09ee20 con 0x7f471c09e740 2024-09-16T14:50:36.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471c09f100 con 0x7f471c09df10 2024-09-16T14:50:36.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471c09f3e0 con 0x7f471c09e2f0 2024-09-16T14:50:36.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:36.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 10 monclient(hunting): authenticate will time out at 1418.950433s 2024-09-16T14:50:36.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f471affd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f471c09df10 legacy=0x7f471c09d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:36.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f471b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f471c09e740 0x7f471c13b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47246/0 (socket says 172.21.15.189:47246) 2024-09-16T14:50:36.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f471b7fe640 1 -- 172.21.15.189:0/37860263 learned_addr learned my addr 172.21.15.189:0/37860263 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:36.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f471a7fc640 1 --1- 172.21.15.189:0/37860263 >> v1:172.21.15.26:6790/0 conn(0x7f471c09e2f0 0x7f471c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:49378/0 (socket says 172.21.15.189:49378) 2024-09-16T14:50:36.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 1 -- 172.21.15.189:0/37860263 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599350144 0 0) 0x7f471c09ee20 con 0x7f471c09e740 2024-09-16T14:50:36.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:36.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:36.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:36.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 10 monclient(hunting): my global_id is 164451 2024-09-16T14:50:36.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 1 -- 172.21.15.189:0/37860263 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4704003530 con 0x7f471c09e740 2024-09-16T14:50:36.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 1 -- 172.21.15.189:0/37860263 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808473252 0 0) 0x7f471c09f3e0 con 0x7f471c09e2f0 2024-09-16T14:50:36.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:36.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:36.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:36.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 10 monclient(hunting): my global_id is 154706 2024-09-16T14:50:36.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 1 -- 172.21.15.189:0/37860263 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f471c09ee20 con 0x7f471c09e2f0 2024-09-16T14:50:36.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 1 -- 172.21.15.189:0/37860263 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1750387533 0 0) 0x7f4704003530 con 0x7f471c09e740 2024-09-16T14:50:36.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 1 -- 172.21.15.189:0/37860263 >> v1:172.21.15.26:6790/0 conn(0x7f471c09e2f0 legacy=0x7f471c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:36.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 1 -- 172.21.15.189:0/37860263 >> v1:172.21.15.189:6789/0 conn(0x7f471c09df10 legacy=0x7f471c09d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:36.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:50:36.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 1 monclient: found mon.noname-a 2024-09-16T14:50:36.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:36.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:36.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 1 -- 172.21.15.189:0/37860263 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c09c690 con 0x7f471c09e740 2024-09-16T14:50:36.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 10 monclient: _finish_auth 0 2024-09-16T14:50:36.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:36.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:36.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 1 -- 172.21.15.189:0/37860263 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f47140019a0 con 0x7f471c09e740 2024-09-16T14:50:36.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 5 monclient: authenticate success, global_id 164451 2024-09-16T14:50:36.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f472199c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:36.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:36.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:36.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.832+0000 7f4719ffb640 10 monclient: dump: 2024-09-16T14:50:36.841 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:36.841 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:36.841 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:36.841 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:36.841 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:36.841 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:36.841 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:36.841 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:36.841 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:36.841 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:36.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:36.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4719ffb640 1 -- 172.21.15.189:0/37860263 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4714001c80 con 0x7f471c09e740 2024-09-16T14:50:36.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4719ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:36.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:36.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 4 set_mon_vals no callback set 2024-09-16T14:50:36.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4719ffb640 1 -- 172.21.15.189:0/37860263 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4714002010 con 0x7f471c09e740 2024-09-16T14:50:36.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4719ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:36.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4719ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:36.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4719ffb640 10 monclient: dump: 2024-09-16T14:50:36.842 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:36.842 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:36.842 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:36.842 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:36.843 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:36.843 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:36.843 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:36.843 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:36.843 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:36.843 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:36.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 10 monclient: shutdown 2024-09-16T14:50:36.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:36.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 1 -- 172.21.15.189:0/37860263 >> v1:172.21.15.26:6789/0 conn(0x7f471c09e740 legacy=0x7f471c13b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:36.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 1 -- 172.21.15.189:0/37860263 shutdown_connections 2024-09-16T14:50:36.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 1 -- 172.21.15.189:0/37860263 >> 172.21.15.189:0/37860263 conn(0x7f471c0a3320 msgr2=0x7f471c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:36.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 1 -- 172.21.15.189:0/37860263 shutdown_connections 2024-09-16T14:50:36.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 1 -- 172.21.15.189:0/37860263 wait complete. 2024-09-16T14:50:36.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 10 monclient: build_initial_monmap 2024-09-16T14:50:36.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 10 monclient: monmap: 2024-09-16T14:50:36.844 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:36.844 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:36.844 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:36.844 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:36.844 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:36.844 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:36.844 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:36.844 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:36.844 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:36.844 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:36.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 1 Processor -- start 2024-09-16T14:50:36.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 1 -- start start 2024-09-16T14:50:36.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 10 monclient: init 2024-09-16T14:50:36.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:36.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:36.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 10 monclient(hunting): picked mon.noname-a con 0x7f471c09df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:36.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 10 monclient(hunting): picked mon.noname-c con 0x7f471c09e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:36.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 10 monclient(hunting): picked mon.noname-b con 0x7f471c09e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:36.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471c128490 con 0x7f471c09df10 2024-09-16T14:50:36.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471c06cc70 con 0x7f471c09e740 2024-09-16T14:50:36.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471c06b380 con 0x7f471c09e2f0 2024-09-16T14:50:36.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:36.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f472199c640 10 monclient(hunting): authenticate will time out at 1418.954294s 2024-09-16T14:50:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f471b7fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f471c09e740 legacy=0x7f471c09b040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f471affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f471c09df10 0x7f471c09a220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47248/0 (socket says 172.21.15.189:47248) 2024-09-16T14:50:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f471affd640 1 -- 172.21.15.189:0/1383042144 learned_addr learned my addr 172.21.15.189:0/1383042144 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616583076 0 0) 0x7f471c128490 con 0x7f471c09df10 2024-09-16T14:50:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 10 monclient(hunting): my global_id is 164454 2024-09-16T14:50:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4708003530 con 0x7f471c09df10 2024-09-16T14:50:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851827986 0 0) 0x7f471c06b380 con 0x7f471c09e2f0 2024-09-16T14:50:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:36.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 10 monclient(hunting): my global_id is 154709 2024-09-16T14:50:36.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f471c128490 con 0x7f471c09e2f0 2024-09-16T14:50:36.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 338385889 0 0) 0x7f4708003530 con 0x7f471c09df10 2024-09-16T14:50:36.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f471c06b380 con 0x7f471c09df10 2024-09-16T14:50:36.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1110828313 0 0) 0x7f471c128490 con 0x7f471c09e2f0 2024-09-16T14:50:36.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4708003530 con 0x7f471c09e2f0 2024-09-16T14:50:36.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4710001640 con 0x7f471c09e2f0 2024-09-16T14:50:36.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:36.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:36.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 10 monclient(hunting): dump: 2024-09-16T14:50:36.850 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:36.850 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:36.850 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:36.851 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:36.851 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:36.851 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:36.851 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:36.851 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:36.851 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:36.851 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:36.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:36.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f470c005290 con 0x7f471c09df10 2024-09-16T14:50:36.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:36.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:36.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.836+0000 7f4703fff640 10 monclient(hunting): dump: 2024-09-16T14:50:36.851 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:36.851 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:36.852 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:36.852 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:36.852 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:36.852 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:36.852 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:36.852 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:36.852 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:36.852 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:36.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2693283768 0 0) 0x7f471c06b380 con 0x7f471c09df10 2024-09-16T14:50:36.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 >> v1:172.21.15.26:6790/0 conn(0x7f471c09e2f0 legacy=0x7f471c09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:36.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 >> v1:172.21.15.189:6789/0 conn(0x7f471c09e740 legacy=0x7f471c09b040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:36.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 10 monclient: _finish_hunting 0 2024-09-16T14:50:36.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 1 monclient: found mon.a 2024-09-16T14:50:36.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:36.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:36.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c120e20 con 0x7f471c09df10 2024-09-16T14:50:36.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 10 monclient: _finish_auth 0 2024-09-16T14:50:36.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 10 monclient: _check_auth_tickets 2024-09-16T14:50:36.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:36.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f472199c640 5 monclient: authenticate success, global_id 164454 2024-09-16T14:50:36.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f472199c640 10 monclient: _renew_subs 2024-09-16T14:50:36.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f472199c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:36.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f472199c640 1 -- 172.21.15.189:0/1383042144 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471c121b80 con 0x7f471c09df10 2024-09-16T14:50:36.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f472199c640 10 monclient: _renew_subs 2024-09-16T14:50:36.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f472199c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:36.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f472199c640 1 -- 172.21.15.189:0/1383042144 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f471c1207a0 con 0x7f471c09df10 2024-09-16T14:50:36.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f470c003f80 con 0x7f471c09df10 2024-09-16T14:50:36.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:36.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f470c004310 con 0x7f471c09df10 2024-09-16T14:50:36.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:36.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4718ff9640 4 set_mon_vals no callback set 2024-09-16T14:50:36.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:36.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 10 monclient: dump: 2024-09-16T14:50:36.854 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:36.854 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:36.854 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:36.854 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:36.854 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:36.854 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:36.854 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:36.854 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:36.855 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:36.855 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:36.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f472199c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:36.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f472199c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:36.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f472199c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:36.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f472199c640 1 -- 172.21.15.189:0/1383042144 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46fc005050 con 0x7f471c09df10 2024-09-16T14:50:36.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f470c004530 con 0x7f471c09df10 2024-09-16T14:50:36.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(230..230 src has 169..230) v4 ==== 10904+0+0 (unknown 3191321856 0 0) 0x7f470c044dc0 con 0x7f471c09df10 2024-09-16T14:50:36.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.840+0000 7f471a7fc640 10 monclient: get_auth_request con 0x7f4708036210 auth_method 0 2024-09-16T14:50:36.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.852+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f470c0144b0 con 0x7f471c09df10 2024-09-16T14:50:36.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.852+0000 7f4703fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:36.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:36.852+0000 7f4703fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:37.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.004+0000 7f472199c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T14:50:37.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.004+0000 7f472199c640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T14:50:37.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.004+0000 7f472199c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:37.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.004+0000 7f472199c640 1 -- 172.21.15.189:0/1383042144 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h16"} v 0) v1 -- 0x7f46fc005300 con 0x7f471c09df10 2024-09-16T14:50:37.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.600+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h16"}]=0 removed item id -64 name 'h16' from crush map v231) v1 ==== 120+0+0 (unknown 557574609 0 0) 0x7f470c0142d0 con 0x7f471c09df10 2024-09-16T14:50:37.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.600+0000 7f4703fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T14:50:37.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.600+0000 7f4703fff640 10 monclient: _finish_command 2 = system:0 removed item id -64 name 'h16' from crush map 2024-09-16T14:50:37.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.696+0000 7f472199c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T14:50:37.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.696+0000 7f472199c640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T14:50:37.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.696+0000 7f472199c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:37.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.696+0000 7f472199c640 1 -- 172.21.15.189:0/1383042144 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h16"} v 0) v1 -- 0x7f46fc005a40 con 0x7f471c09df10 2024-09-16T14:50:37.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.700+0000 7f4703fff640 1 -- 172.21.15.189:0/1383042144 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h16"}]=0 device 'h16' does not appear in the crush map v231) v1 ==== 120+0+0 (unknown 1808559892 0 0) 0x7f470c0142d0 con 0x7f471c09df10 2024-09-16T14:50:37.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.700+0000 7f4703fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T14:50:37.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.700+0000 7f4703fff640 10 monclient: _finish_command 3 = system:0 device 'h16' does not appear in the crush map 2024-09-16T14:50:37.702 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h16' does not appear in the crush map 2024-09-16T14:50:37.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.700+0000 7f472199c640 1 -- 172.21.15.189:0/1383042144 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4708036210 legacy=0x7f47080386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:37.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.700+0000 7f472199c640 10 monclient: shutdown 2024-09-16T14:50:37.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.700+0000 7f472199c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:37.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.700+0000 7f472199c640 1 -- 172.21.15.189:0/1383042144 >> v1:172.21.15.26:6789/0 conn(0x7f471c09df10 legacy=0x7f471c09a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:37.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.700+0000 7f472199c640 1 -- 172.21.15.189:0/1383042144 shutdown_connections 2024-09-16T14:50:37.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.700+0000 7f472199c640 1 -- 172.21.15.189:0/1383042144 >> 172.21.15.189:0/1383042144 conn(0x7f471c0a3320 msgr2=0x7f471c09d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:37.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.700+0000 7f472199c640 1 -- 172.21.15.189:0/1383042144 shutdown_connections 2024-09-16T14:50:37.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.700+0000 7f472199c640 1 -- 172.21.15.189:0/1383042144 wait complete. 2024-09-16T14:50:37.719 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:37.719 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h17 2024-09-16T14:50:37.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d9514c640 10 monclient: get_monmap_and_config 2024-09-16T14:50:37.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d9514c640 10 monclient: build_initial_monmap 2024-09-16T14:50:37.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d9514c640 10 monclient: monmap: 2024-09-16T14:50:37.838 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:37.838 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:37.838 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:37.838807+0000 2024-09-16T14:50:37.838 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:37.838807+0000 2024-09-16T14:50:37.838 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:37.838 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:37.838 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:37.838 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:37.838 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:37.839 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:37.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d9514c640 1 Processor -- start 2024-09-16T14:50:37.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d9514c640 1 -- start start 2024-09-16T14:50:37.839 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d9514c640 10 monclient: init 2024-09-16T14:50:37.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d9514c640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:37.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d9514c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:37.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d9514c640 10 monclient(hunting): picked mon.noname-b con 0x7f8d9009df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:37.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d9514c640 10 monclient(hunting): picked mon.noname-a con 0x7f8d9009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:37.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d9514c640 10 monclient(hunting): picked mon.noname-c con 0x7f8d9009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:37.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d9514c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9009ee20 con 0x7f8d9009e2f0 2024-09-16T14:50:37.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d9514c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9009f100 con 0x7f8d9009e740 2024-09-16T14:50:37.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d9514c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9009f3e0 con 0x7f8d9009df10 2024-09-16T14:50:37.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d9514c640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:37.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d9514c640 10 monclient(hunting): authenticate will time out at 1419.954607s 2024-09-16T14:50:37.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8f59e640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f8d9009e740 legacy=0x7f8d9013b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:37.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8e59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8d9009e2f0 0x7f8d901378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47258/0 (socket says 172.21.15.189:47258) 2024-09-16T14:50:37.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8ed9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8d9009df10 0x7f8d9009d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:49398/0 (socket says 172.21.15.189:49398) 2024-09-16T14:50:37.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8e59c640 1 -- 172.21.15.189:0/2192888051 learned_addr learned my addr 172.21.15.189:0/2192888051 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:37.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8dd9b640 1 -- 172.21.15.189:0/2192888051 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648725915 0 0) 0x7f8d9009f3e0 con 0x7f8d9009df10 2024-09-16T14:50:37.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8dd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:50:37.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:37.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:37.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8dd9b640 10 monclient(hunting): my global_id is 154712 2024-09-16T14:50:37.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8dd9b640 1 -- 172.21.15.189:0/2192888051 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d78003530 con 0x7f8d9009df10 2024-09-16T14:50:37.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8dd9b640 1 -- 172.21.15.189:0/2192888051 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691467128 0 0) 0x7f8d9009ee20 con 0x7f8d9009e2f0 2024-09-16T14:50:37.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8dd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:50:37.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:37.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:37.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8dd9b640 10 monclient(hunting): my global_id is 164457 2024-09-16T14:50:37.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8dd9b640 1 -- 172.21.15.189:0/2192888051 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d9009f3e0 con 0x7f8d9009e2f0 2024-09-16T14:50:37.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8dd9b640 1 -- 172.21.15.189:0/2192888051 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 460500606 0 0) 0x7f8d9009f3e0 con 0x7f8d9009e2f0 2024-09-16T14:50:37.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.836+0000 7f8d8dd9b640 1 -- 172.21.15.189:0/2192888051 >> v1:172.21.15.26:6790/0 conn(0x7f8d9009df10 legacy=0x7f8d9009d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:37.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 1 -- 172.21.15.189:0/2192888051 >> v1:172.21.15.189:6789/0 conn(0x7f8d9009e740 legacy=0x7f8d9013b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:37.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 10 monclient: _finish_hunting 0 2024-09-16T14:50:37.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 1 monclient: found mon.noname-a 2024-09-16T14:50:37.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:37.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:37.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 1 -- 172.21.15.189:0/2192888051 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9009c690 con 0x7f8d9009e2f0 2024-09-16T14:50:37.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 10 monclient: _finish_auth 0 2024-09-16T14:50:37.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 10 monclient: _check_auth_tickets 2024-09-16T14:50:37.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:37.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 5 monclient: authenticate success, global_id 164457 2024-09-16T14:50:37.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:37.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 1 -- 172.21.15.189:0/2192888051 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8d84001600 con 0x7f8d9009e2f0 2024-09-16T14:50:37.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:37.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:37.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 10 monclient: dump: 2024-09-16T14:50:37.846 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:37.846 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:37.846 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:37.846 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:37.846 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:37.846 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:37.847 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:37.847 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:37.847 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:37.847 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:37.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:37.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 1 -- 172.21.15.189:0/2192888051 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d840018a0 con 0x7f8d9009e2f0 2024-09-16T14:50:37.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:37.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 1 -- 172.21.15.189:0/2192888051 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8d84001c70 con 0x7f8d9009e2f0 2024-09-16T14:50:37.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:37.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 4 set_mon_vals no callback set 2024-09-16T14:50:37.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:37.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:37.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8dd9b640 10 monclient: dump: 2024-09-16T14:50:37.847 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:37.848 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:37.848 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:37.848 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:37.848 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:37.848 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:37.848 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:37.848 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:37.848 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:37.848 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:37.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 10 monclient: shutdown 2024-09-16T14:50:37.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:37.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 1 -- 172.21.15.189:0/2192888051 >> v1:172.21.15.26:6789/0 conn(0x7f8d9009e2f0 legacy=0x7f8d901378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:37.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 1 -- 172.21.15.189:0/2192888051 shutdown_connections 2024-09-16T14:50:37.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 1 -- 172.21.15.189:0/2192888051 >> 172.21.15.189:0/2192888051 conn(0x7f8d900a3320 msgr2=0x7f8d9009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:37.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 1 -- 172.21.15.189:0/2192888051 shutdown_connections 2024-09-16T14:50:37.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 1 -- 172.21.15.189:0/2192888051 wait complete. 2024-09-16T14:50:37.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 10 monclient: build_initial_monmap 2024-09-16T14:50:37.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 10 monclient: monmap: 2024-09-16T14:50:37.849 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:37.849 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:37.849 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:37.849 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:37.849 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:37.849 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:37.849 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:37.849 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:37.850 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:37.850 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:37.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 1 Processor -- start 2024-09-16T14:50:37.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 1 -- start start 2024-09-16T14:50:37.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 10 monclient: init 2024-09-16T14:50:37.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:37.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:37.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 10 monclient(hunting): picked mon.noname-a con 0x7f8d9009df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:37.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 10 monclient(hunting): picked mon.noname-c con 0x7f8d9009e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:37.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 10 monclient(hunting): picked mon.noname-b con 0x7f8d9009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:37.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9011fe80 con 0x7f8d9009df10 2024-09-16T14:50:37.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9006ce40 con 0x7f8d9009e740 2024-09-16T14:50:37.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9011d040 con 0x7f8d9009e2f0 2024-09-16T14:50:37.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:37.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8f59e640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f8d9009e740 legacy=0x7f8d901cf780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:37.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d9514c640 10 monclient(hunting): authenticate will time out at 1419.958589s 2024-09-16T14:50:37.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8ed9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8d9009df10 0x7f8d9011c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47270/0 (socket says 172.21.15.189:47270) 2024-09-16T14:50:37.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d8ed9d640 1 -- 172.21.15.189:0/1626590518 learned_addr learned my addr 172.21.15.189:0/1626590518 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:37.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319752448 0 0) 0x7f8d9011fe80 con 0x7f8d9009df10 2024-09-16T14:50:37.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:37.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:37.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:37.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 10 monclient(hunting): my global_id is 164460 2024-09-16T14:50:37.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d7c003530 con 0x7f8d9009df10 2024-09-16T14:50:37.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589967216 0 0) 0x7f8d9011d040 con 0x7f8d9009e2f0 2024-09-16T14:50:37.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:37.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:37.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:37.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 10 monclient(hunting): my global_id is 154715 2024-09-16T14:50:37.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d9011fe80 con 0x7f8d9009e2f0 2024-09-16T14:50:37.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 815688483 0 0) 0x7f8d7c003530 con 0x7f8d9009df10 2024-09-16T14:50:37.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d9011d040 con 0x7f8d9009df10 2024-09-16T14:50:37.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8d800013c0 con 0x7f8d9009df10 2024-09-16T14:50:37.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:37.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:37.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 10 monclient(hunting): dump: 2024-09-16T14:50:37.858 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:37.858 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:37.858 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:37.858 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:37.858 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:37.858 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:37.858 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:37.858 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:37.859 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:37.859 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:37.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:37.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2809611556 0 0) 0x7f8d9011fe80 con 0x7f8d9009e2f0 2024-09-16T14:50:37.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d7c003530 con 0x7f8d9009e2f0 2024-09-16T14:50:37.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.840+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8d84001bb0 con 0x7f8d9009e2f0 2024-09-16T14:50:37.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:37.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:37.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 10 monclient(hunting): dump: 2024-09-16T14:50:37.859 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:37.859 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:37.859 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:37.859 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:37.860 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:37.860 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:37.860 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:37.860 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:37.860 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:37.860 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:37.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3679522921 0 0) 0x7f8d9011d040 con 0x7f8d9009df10 2024-09-16T14:50:37.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 >> v1:172.21.15.26:6790/0 conn(0x7f8d9009e2f0 legacy=0x7f8d9011c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:37.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 >> v1:172.21.15.189:6789/0 conn(0x7f8d9009e740 legacy=0x7f8d901cf780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:37.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:37.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 1 monclient: found mon.a 2024-09-16T14:50:37.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:37.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:37.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9011e960 con 0x7f8d9009df10 2024-09-16T14:50:37.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:37.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:37.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:37.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d9514c640 5 monclient: authenticate success, global_id 164460 2024-09-16T14:50:37.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d9514c640 10 monclient: _renew_subs 2024-09-16T14:50:37.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d9514c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:37.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d9514c640 1 -- 172.21.15.189:0/1626590518 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9011dd90 con 0x7f8d9009df10 2024-09-16T14:50:37.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d9514c640 10 monclient: _renew_subs 2024-09-16T14:50:37.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d9514c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:37.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d9514c640 1 -- 172.21.15.189:0/1626590518 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d90128020 con 0x7f8d9009df10 2024-09-16T14:50:37.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d800015e0 con 0x7f8d9009df10 2024-09-16T14:50:37.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:37.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d8d59a640 4 set_mon_vals no callback set 2024-09-16T14:50:37.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8d80004c40 con 0x7f8d9009df10 2024-09-16T14:50:37.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:37.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:37.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 10 monclient: dump: 2024-09-16T14:50:37.863 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:37.863 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:37.863 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:37.863 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:37.863 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:37.863 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:37.863 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:37.863 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:37.863 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:37.863 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:37.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d9514c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:37.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d9514c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:37.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d9514c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:37.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d9514c640 1 -- 172.21.15.189:0/1626590518 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d70005050 con 0x7f8d9009df10 2024-09-16T14:50:37.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8d8000f6b0 con 0x7f8d9009df10 2024-09-16T14:50:37.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(231..231 src has 169..231) v4 ==== 10691+0+0 (unknown 1326449981 0 0) 0x7f8d80044cd0 con 0x7f8d9009df10 2024-09-16T14:50:37.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.844+0000 7f8d8e59c640 10 monclient: get_auth_request con 0x7f8d7c036220 auth_method 0 2024-09-16T14:50:37.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.856+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8d80014410 con 0x7f8d9009df10 2024-09-16T14:50:37.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.856+0000 7f8d777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:37.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:37.856+0000 7f8d777fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:38.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.004+0000 7f8d9514c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T14:50:38.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.004+0000 7f8d9514c640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T14:50:38.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.004+0000 7f8d9514c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:38.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.004+0000 7f8d9514c640 1 -- 172.21.15.189:0/1626590518 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h17"} v 0) v1 -- 0x7f8d70005300 con 0x7f8d9009df10 2024-09-16T14:50:38.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.616+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h17"}]=0 removed item id -67 name 'h17' from crush map v232) v1 ==== 120+0+0 (unknown 3808415319 0 0) 0x7f8d80014230 con 0x7f8d9009df10 2024-09-16T14:50:38.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.616+0000 7f8d777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T14:50:38.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.616+0000 7f8d777fe640 10 monclient: _finish_command 2 = system:0 removed item id -67 name 'h17' from crush map 2024-09-16T14:50:38.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.712+0000 7f8d9514c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T14:50:38.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.712+0000 7f8d9514c640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T14:50:38.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.712+0000 7f8d9514c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:38.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.712+0000 7f8d9514c640 1 -- 172.21.15.189:0/1626590518 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h17"} v 0) v1 -- 0x7f8d70005a40 con 0x7f8d9009df10 2024-09-16T14:50:38.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.716+0000 7f8d777fe640 1 -- 172.21.15.189:0/1626590518 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h17"}]=0 device 'h17' does not appear in the crush map v232) v1 ==== 120+0+0 (unknown 2561933383 0 0) 0x7f8d80014410 con 0x7f8d9009df10 2024-09-16T14:50:38.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.716+0000 7f8d777fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T14:50:38.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.716+0000 7f8d777fe640 10 monclient: _finish_command 3 = system:0 device 'h17' does not appear in the crush map 2024-09-16T14:50:38.717 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h17' does not appear in the crush map 2024-09-16T14:50:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.716+0000 7f8d9514c640 1 -- 172.21.15.189:0/1626590518 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8d7c036220 legacy=0x7f8d7c0386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.716+0000 7f8d9514c640 10 monclient: shutdown 2024-09-16T14:50:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.716+0000 7f8d9514c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.716+0000 7f8d9514c640 1 -- 172.21.15.189:0/1626590518 >> v1:172.21.15.26:6789/0 conn(0x7f8d9009df10 legacy=0x7f8d9011c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.716+0000 7f8d9514c640 1 -- 172.21.15.189:0/1626590518 shutdown_connections 2024-09-16T14:50:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.716+0000 7f8d9514c640 1 -- 172.21.15.189:0/1626590518 >> 172.21.15.189:0/1626590518 conn(0x7f8d900a3320 msgr2=0x7f8d900a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.716+0000 7f8d9514c640 1 -- 172.21.15.189:0/1626590518 shutdown_connections 2024-09-16T14:50:38.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.716+0000 7f8d9514c640 1 -- 172.21.15.189:0/1626590518 wait complete. 2024-09-16T14:50:38.734 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:38.735 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h18 2024-09-16T14:50:38.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65e1164640 10 monclient: get_monmap_and_config 2024-09-16T14:50:38.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65e1164640 10 monclient: build_initial_monmap 2024-09-16T14:50:38.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65e1164640 10 monclient: monmap: 2024-09-16T14:50:38.851 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:38.851 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:38.851 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:38.852233+0000 2024-09-16T14:50:38.851 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:38.852233+0000 2024-09-16T14:50:38.852 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:38.852 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:38.852 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:38.852 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:38.852 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:38.852 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:38.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65e1164640 1 Processor -- start 2024-09-16T14:50:38.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65e1164640 1 -- start start 2024-09-16T14:50:38.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65e1164640 10 monclient: init 2024-09-16T14:50:38.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65e1164640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:38.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65e1164640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:38.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65e1164640 10 monclient(hunting): picked mon.noname-b con 0x7f65dc09df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:38.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65e1164640 10 monclient(hunting): picked mon.noname-c con 0x7f65dc09e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:38.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65e1164640 10 monclient(hunting): picked mon.noname-a con 0x7f65dc09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:38.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65e1164640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65dc09ee20 con 0x7f65dc09e740 2024-09-16T14:50:38.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65e1164640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65dc09f100 con 0x7f65dc09e2f0 2024-09-16T14:50:38.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65e1164640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65dc09f3e0 con 0x7f65dc09df10 2024-09-16T14:50:38.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65e1164640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:38.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65e1164640 10 monclient(hunting): authenticate will time out at 1420.967835s 2024-09-16T14:50:38.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.848+0000 7f65da59c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f65dc09e2f0 legacy=0x7f65dc1378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:38.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65dad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f65dc09df10 0x7f65dc09d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:49418/0 (socket says 172.21.15.189:49418) 2024-09-16T14:50:38.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65dad9d640 1 -- 172.21.15.189:0/535289891 learned_addr learned my addr 172.21.15.189:0/535289891 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:38.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 1 -- 172.21.15.189:0/535289891 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253869280 0 0) 0x7f65dc09f3e0 con 0x7f65dc09df10 2024-09-16T14:50:38.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:50:38.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:38.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:38.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient(hunting): my global_id is 154718 2024-09-16T14:50:38.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 1 -- 172.21.15.189:0/535289891 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65c4003530 con 0x7f65dc09df10 2024-09-16T14:50:38.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 1 -- 172.21.15.189:0/535289891 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3670804936 0 0) 0x7f65dc09ee20 con 0x7f65dc09e740 2024-09-16T14:50:38.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:50:38.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:38.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:38.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient(hunting): my global_id is 164463 2024-09-16T14:50:38.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 1 -- 172.21.15.189:0/535289891 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65dc09f3e0 con 0x7f65dc09e740 2024-09-16T14:50:38.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 1 -- 172.21.15.189:0/535289891 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3809389659 0 0) 0x7f65c4003530 con 0x7f65dc09df10 2024-09-16T14:50:38.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 1 -- 172.21.15.189:0/535289891 >> v1:172.21.15.189:6789/0 conn(0x7f65dc09e2f0 legacy=0x7f65dc1378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:38.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 1 -- 172.21.15.189:0/535289891 >> v1:172.21.15.26:6789/0 conn(0x7f65dc09e740 legacy=0x7f65dc13b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:38.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:50:38.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 1 monclient: found mon.noname-b 2024-09-16T14:50:38.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:38.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:38.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 1 -- 172.21.15.189:0/535289891 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc09c690 con 0x7f65dc09df10 2024-09-16T14:50:38.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient: _finish_auth 0 2024-09-16T14:50:38.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:50:38.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:38.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 1 -- 172.21.15.189:0/535289891 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f65cc001620 con 0x7f65dc09df10 2024-09-16T14:50:38.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:38.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:38.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient: dump: 2024-09-16T14:50:38.863 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:38.863 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:38.863 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:38.863 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:38.863 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:38.863 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:38.863 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:38.863 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:38.863 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:38.864 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:38.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 5 monclient: authenticate success, global_id 154718 2024-09-16T14:50:38.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:38.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 1 -- 172.21.15.189:0/535289891 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65cc001900 con 0x7f65dc09df10 2024-09-16T14:50:38.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:38.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:38.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 4 set_mon_vals no callback set 2024-09-16T14:50:38.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 1 -- 172.21.15.189:0/535289891 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f65cc001c90 con 0x7f65dc09df10 2024-09-16T14:50:38.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:38.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:38.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65d9d9b640 10 monclient: dump: 2024-09-16T14:50:38.864 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:38.864 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:38.865 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:38.865 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:38.865 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:38.865 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:38.865 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:38.865 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:38.865 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:38.865 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:38.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 10 monclient: shutdown 2024-09-16T14:50:38.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:38.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 1 -- 172.21.15.189:0/535289891 >> v1:172.21.15.26:6790/0 conn(0x7f65dc09df10 legacy=0x7f65dc09d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:38.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 1 -- 172.21.15.189:0/535289891 shutdown_connections 2024-09-16T14:50:38.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 1 -- 172.21.15.189:0/535289891 >> 172.21.15.189:0/535289891 conn(0x7f65dc0a3320 msgr2=0x7f65dc09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:38.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 1 -- 172.21.15.189:0/535289891 shutdown_connections 2024-09-16T14:50:38.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 1 -- 172.21.15.189:0/535289891 wait complete. 2024-09-16T14:50:38.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 10 monclient: build_initial_monmap 2024-09-16T14:50:38.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 10 monclient: monmap: 2024-09-16T14:50:38.866 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:38.866 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:38.866 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:38.866 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:38.866 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:38.866 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:38.866 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:38.866 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:38.866 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:38.866 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:38.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 1 Processor -- start 2024-09-16T14:50:38.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 1 -- start start 2024-09-16T14:50:38.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 10 monclient: init 2024-09-16T14:50:38.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:38.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:38.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 10 monclient(hunting): picked mon.noname-c con 0x7f65dc09df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:38.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 10 monclient(hunting): picked mon.noname-a con 0x7f65dc09e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:38.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 10 monclient(hunting): picked mon.noname-b con 0x7f65dc09e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:38.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65dc06fa30 con 0x7f65dc09e2f0 2024-09-16T14:50:38.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65dc06b380 con 0x7f65dc09e740 2024-09-16T14:50:38.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65dc1303c0 con 0x7f65dc09df10 2024-09-16T14:50:38.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:38.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65e1164640 10 monclient(hunting): authenticate will time out at 1420.971628s 2024-09-16T14:50:38.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65db59e640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f65dc09e740 legacy=0x7f65dc1d18d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:38.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65dad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f65dc09df10 0x7f65dc12f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:49432/0 (socket says 172.21.15.189:49432) 2024-09-16T14:50:38.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.852+0000 7f65dad9d640 1 -- 172.21.15.189:0/2197235021 learned_addr learned my addr 172.21.15.189:0/2197235021 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:38.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807477269 0 0) 0x7f65dc1303c0 con 0x7f65dc09df10 2024-09-16T14:50:38.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:38.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:38.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:38.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient(hunting): my global_id is 154721 2024-09-16T14:50:38.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65c8003530 con 0x7f65dc09df10 2024-09-16T14:50:38.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917106525 0 0) 0x7f65dc06fa30 con 0x7f65dc09e2f0 2024-09-16T14:50:38.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:38.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:38.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:38.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient(hunting): my global_id is 164466 2024-09-16T14:50:38.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65dc1303c0 con 0x7f65dc09e2f0 2024-09-16T14:50:38.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2262643669 0 0) 0x7f65c8003530 con 0x7f65dc09df10 2024-09-16T14:50:38.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65dc06fa30 con 0x7f65dc09df10 2024-09-16T14:50:38.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f65cc0015e0 con 0x7f65dc09df10 2024-09-16T14:50:38.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:38.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:38.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient(hunting): dump: 2024-09-16T14:50:38.870 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:38.870 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:38.870 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:38.870 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:38.870 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:38.870 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:38.870 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:38.870 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:38.871 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:38.871 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:38.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:38.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2535972100 0 0) 0x7f65dc1303c0 con 0x7f65dc09e2f0 2024-09-16T14:50:38.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65c8003530 con 0x7f65dc09e2f0 2024-09-16T14:50:38.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f65d00040d0 con 0x7f65dc09e2f0 2024-09-16T14:50:38.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:38.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:38.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient(hunting): dump: 2024-09-16T14:50:38.871 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:38.871 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:38.871 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:38.871 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:38.872 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:38.872 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:38.872 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:38.872 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:38.872 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:38.872 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:38.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3434216988 0 0) 0x7f65dc06fa30 con 0x7f65dc09df10 2024-09-16T14:50:38.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 >> v1:172.21.15.189:6789/0 conn(0x7f65dc09e740 legacy=0x7f65dc1d18d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:38.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 >> v1:172.21.15.26:6789/0 conn(0x7f65dc09e2f0 legacy=0x7f65dc12fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:38.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:38.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 monclient: found mon.c 2024-09-16T14:50:38.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:38.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:38.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc0a11f0 con 0x7f65dc09df10 2024-09-16T14:50:38.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:38.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:38.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:38.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65e1164640 5 monclient: authenticate success, global_id 154721 2024-09-16T14:50:38.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65e1164640 10 monclient: _renew_subs 2024-09-16T14:50:38.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65e1164640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:38.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65e1164640 1 -- 172.21.15.189:0/2197235021 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65dc1395a0 con 0x7f65dc09df10 2024-09-16T14:50:38.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65e1164640 10 monclient: _renew_subs 2024-09-16T14:50:38.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65e1164640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:38.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65e1164640 1 -- 172.21.15.189:0/2197235021 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65dc137e70 con 0x7f65dc09df10 2024-09-16T14:50:38.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65cc004c00 con 0x7f65dc09df10 2024-09-16T14:50:38.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:38.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65d959a640 4 set_mon_vals no callback set 2024-09-16T14:50:38.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f65cc004f10 con 0x7f65dc09df10 2024-09-16T14:50:38.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:38.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:38.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 10 monclient: dump: 2024-09-16T14:50:38.874 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:38.874 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:38.875 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:38.875 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:38.875 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:38.875 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:38.875 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:38.875 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:38.875 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:38.875 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:38.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65e1164640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:38.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65e1164640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:38.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65e1164640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:38.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65e1164640 1 -- 172.21.15.189:0/2197235021 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65bc005050 con 0x7f65dc09df10 2024-09-16T14:50:38.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f65cc00f980 con 0x7f65dc09df10 2024-09-16T14:50:38.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(232..232 src has 169..232) v4 ==== 10478+0+0 (unknown 3350771983 0 0) 0x7f65cc044df0 con 0x7f65dc09df10 2024-09-16T14:50:38.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.856+0000 7f65da59c640 10 monclient: get_auth_request con 0x7f65c8036250 auth_method 0 2024-09-16T14:50:38.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.868+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f65cc014720 con 0x7f65dc09df10 2024-09-16T14:50:38.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.868+0000 7f65c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:38.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:38.868+0000 7f65c37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:39.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.032+0000 7f65e1164640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T14:50:39.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.032+0000 7f65e1164640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T14:50:39.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.032+0000 7f65e1164640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:39.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.032+0000 7f65e1164640 1 -- 172.21.15.189:0/2197235021 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h18"} v 0) v1 -- 0x7f65bc005300 con 0x7f65dc09df10 2024-09-16T14:50:39.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.616+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h18"}]=0 removed item id -70 name 'h18' from crush map v233) v1 ==== 120+0+0 (unknown 300242253 0 0) 0x7f65cc014540 con 0x7f65dc09df10 2024-09-16T14:50:39.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.616+0000 7f65c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T14:50:39.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.616+0000 7f65c37fe640 10 monclient: _finish_command 2 = system:0 removed item id -70 name 'h18' from crush map 2024-09-16T14:50:39.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.720+0000 7f65e1164640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T14:50:39.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.720+0000 7f65e1164640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T14:50:39.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.720+0000 7f65e1164640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:39.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.720+0000 7f65e1164640 1 -- 172.21.15.189:0/2197235021 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h18"} v 0) v1 -- 0x7f65bc005a40 con 0x7f65dc09df10 2024-09-16T14:50:39.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.724+0000 7f65c37fe640 1 -- 172.21.15.189:0/2197235021 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h18"}]=0 device 'h18' does not appear in the crush map v233) v1 ==== 120+0+0 (unknown 1861545896 0 0) 0x7f65cc015280 con 0x7f65dc09df10 2024-09-16T14:50:39.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.724+0000 7f65c37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T14:50:39.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.724+0000 7f65c37fe640 10 monclient: _finish_command 3 = system:0 device 'h18' does not appear in the crush map 2024-09-16T14:50:39.725 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h18' does not appear in the crush map 2024-09-16T14:50:39.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.724+0000 7f65e1164640 1 -- 172.21.15.189:0/2197235021 >> v1:172.21.15.189:6812/3196161582 conn(0x7f65c8036250 legacy=0x7f65c8038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:39.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.724+0000 7f65e1164640 10 monclient: shutdown 2024-09-16T14:50:39.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.724+0000 7f65e1164640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:39.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.724+0000 7f65e1164640 1 -- 172.21.15.189:0/2197235021 >> v1:172.21.15.26:6790/0 conn(0x7f65dc09df10 legacy=0x7f65dc12f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:39.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.724+0000 7f65e1164640 1 -- 172.21.15.189:0/2197235021 shutdown_connections 2024-09-16T14:50:39.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.724+0000 7f65e1164640 1 -- 172.21.15.189:0/2197235021 >> 172.21.15.189:0/2197235021 conn(0x7f65dc0a3320 msgr2=0x7f65dc09b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:39.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.724+0000 7f65e1164640 1 -- 172.21.15.189:0/2197235021 shutdown_connections 2024-09-16T14:50:39.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.724+0000 7f65e1164640 1 -- 172.21.15.189:0/2197235021 wait complete. 2024-09-16T14:50:39.741 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:39.741 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h19 2024-09-16T14:50:39.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.852+0000 7fd50943c640 10 monclient: get_monmap_and_config 2024-09-16T14:50:39.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.852+0000 7fd50943c640 10 monclient: build_initial_monmap 2024-09-16T14:50:39.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.852+0000 7fd50943c640 10 monclient: monmap: 2024-09-16T14:50:39.854 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:39.854 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:39.855 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:39.856057+0000 2024-09-16T14:50:39.855 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:39.856057+0000 2024-09-16T14:50:39.855 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:39.855 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:39.855 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:39.855 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:39.855 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:39.855 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:39.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.852+0000 7fd50943c640 1 Processor -- start 2024-09-16T14:50:39.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.852+0000 7fd50943c640 1 -- start start 2024-09-16T14:50:39.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.852+0000 7fd50943c640 10 monclient: init 2024-09-16T14:50:39.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.852+0000 7fd50943c640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:39.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.852+0000 7fd50943c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:39.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.852+0000 7fd50943c640 10 monclient(hunting): picked mon.noname-b con 0x7fd50409df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:39.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.852+0000 7fd50943c640 10 monclient(hunting): picked mon.noname-a con 0x7fd50409e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:39.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 10 monclient(hunting): picked mon.noname-c con 0x7fd50409e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:39.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50409ee20 con 0x7fd50409e2f0 2024-09-16T14:50:39.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50409f100 con 0x7fd50409e740 2024-09-16T14:50:39.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50409f3e0 con 0x7fd50409df10 2024-09-16T14:50:39.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:39.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 10 monclient(hunting): authenticate will time out at 1421.972078s 2024-09-16T14:50:39.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5037fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fd50409e740 legacy=0x7fd50413b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:39.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd502ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd50409df10 0x7fd50409d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56736/0 (socket says 172.21.15.189:56736) 2024-09-16T14:50:39.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd4fa7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd50409e2f0 0x7fd5041378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47050/0 (socket says 172.21.15.189:47050) 2024-09-16T14:50:39.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd502ffd640 1 -- 172.21.15.189:0/2042135570 learned_addr learned my addr 172.21.15.189:0/2042135570 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:39.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 1 -- 172.21.15.189:0/2042135570 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940745222 0 0) 0x7fd50409ee20 con 0x7fd50409e2f0 2024-09-16T14:50:39.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:39.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:39.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:39.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient(hunting): my global_id is 164469 2024-09-16T14:50:39.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 1 -- 172.21.15.189:0/2042135570 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f0003530 con 0x7fd50409e2f0 2024-09-16T14:50:39.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 1 -- 172.21.15.189:0/2042135570 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094930045 0 0) 0x7fd50409f3e0 con 0x7fd50409df10 2024-09-16T14:50:39.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:39.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:39.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:39.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient(hunting): my global_id is 154724 2024-09-16T14:50:39.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 1 -- 172.21.15.189:0/2042135570 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd50409ee20 con 0x7fd50409df10 2024-09-16T14:50:39.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 1 -- 172.21.15.189:0/2042135570 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3221863055 0 0) 0x7fd4f0003530 con 0x7fd50409e2f0 2024-09-16T14:50:39.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 1 -- 172.21.15.189:0/2042135570 >> v1:172.21.15.26:6790/0 conn(0x7fd50409df10 legacy=0x7fd50409d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:39.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 1 -- 172.21.15.189:0/2042135570 >> v1:172.21.15.189:6789/0 conn(0x7fd50409e740 legacy=0x7fd50413b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:39.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient: _finish_hunting 0 2024-09-16T14:50:39.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 1 monclient: found mon.noname-a 2024-09-16T14:50:39.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:39.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:39.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 1 -- 172.21.15.189:0/2042135570 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50409c690 con 0x7fd50409e2f0 2024-09-16T14:50:39.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient: _finish_auth 0 2024-09-16T14:50:39.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:39.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:39.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 1 -- 172.21.15.189:0/2042135570 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd4ec001550 con 0x7fd50409e2f0 2024-09-16T14:50:39.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:39.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:39.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient: dump: 2024-09-16T14:50:39.861 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:39.862 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:39.862 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:39.862 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:39.862 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:39.862 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:39.862 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:39.862 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:39.862 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:39.862 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:39.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 5 monclient: authenticate success, global_id 164469 2024-09-16T14:50:39.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:39.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 1 -- 172.21.15.189:0/2042135570 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4ec001770 con 0x7fd50409e2f0 2024-09-16T14:50:39.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:39.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 1 -- 172.21.15.189:0/2042135570 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd4ec001b40 con 0x7fd50409e2f0 2024-09-16T14:50:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 4 set_mon_vals no callback set 2024-09-16T14:50:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd5027fc640 10 monclient: dump: 2024-09-16T14:50:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:39.863 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:39.864 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:39.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 10 monclient: shutdown 2024-09-16T14:50:39.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:39.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 1 -- 172.21.15.189:0/2042135570 >> v1:172.21.15.26:6789/0 conn(0x7fd50409e2f0 legacy=0x7fd5041378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:39.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 1 -- 172.21.15.189:0/2042135570 shutdown_connections 2024-09-16T14:50:39.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 1 -- 172.21.15.189:0/2042135570 >> 172.21.15.189:0/2042135570 conn(0x7fd5040a3320 msgr2=0x7fd50409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:39.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 1 -- 172.21.15.189:0/2042135570 shutdown_connections 2024-09-16T14:50:39.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 1 -- 172.21.15.189:0/2042135570 wait complete. 2024-09-16T14:50:39.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 10 monclient: build_initial_monmap 2024-09-16T14:50:39.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 10 monclient: monmap: 2024-09-16T14:50:39.869 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:39.869 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:39.869 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:39.869 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:39.869 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:39.869 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:39.870 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:39.870 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:39.870 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:39.870 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:39.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 1 Processor -- start 2024-09-16T14:50:39.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 1 -- start start 2024-09-16T14:50:39.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 10 monclient: init 2024-09-16T14:50:39.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:39.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:39.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.856+0000 7fd50943c640 10 monclient(hunting): picked mon.noname-c con 0x7fd50409df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:39.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 10 monclient(hunting): picked mon.noname-a con 0x7fd50409e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:39.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 10 monclient(hunting): picked mon.noname-b con 0x7fd50409e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:39.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50413b780 con 0x7fd50409e2f0 2024-09-16T14:50:39.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50409b040 con 0x7fd50409e740 2024-09-16T14:50:39.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50406f970 con 0x7fd50409df10 2024-09-16T14:50:39.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:39.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 10 monclient(hunting): authenticate will time out at 1421.976107s 2024-09-16T14:50:39.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd5037fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fd50409e740 legacy=0x7fd5041d17a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:39.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd502ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd50409df10 0x7fd50409a220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56752/0 (socket says 172.21.15.189:56752) 2024-09-16T14:50:39.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd502ffd640 1 -- 172.21.15.189:0/99966608 learned_addr learned my addr 172.21.15.189:0/99966608 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:39.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817560436 0 0) 0x7fd50406f970 con 0x7fd50409df10 2024-09-16T14:50:39.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:39.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:39.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:39.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient(hunting): my global_id is 154727 2024-09-16T14:50:39.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e4003530 con 0x7fd50409df10 2024-09-16T14:50:39.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232450375 0 0) 0x7fd50413b780 con 0x7fd50409e2f0 2024-09-16T14:50:39.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:39.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:39.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:39.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient(hunting): my global_id is 164472 2024-09-16T14:50:39.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd50406f970 con 0x7fd50409e2f0 2024-09-16T14:50:39.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 287072216 0 0) 0x7fd4e4003530 con 0x7fd50409df10 2024-09-16T14:50:39.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd50413b780 con 0x7fd50409df10 2024-09-16T14:50:39.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd4f40019a0 con 0x7fd50409df10 2024-09-16T14:50:39.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:39.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:39.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient(hunting): dump: 2024-09-16T14:50:39.873 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:39.873 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:39.873 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:39.873 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:39.873 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:39.873 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:39.873 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:39.873 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:39.873 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:39.873 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:39.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:39.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3924544340 0 0) 0x7fd50406f970 con 0x7fd50409e2f0 2024-09-16T14:50:39.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4e4003530 con 0x7fd50409e2f0 2024-09-16T14:50:39.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd4ec0018e0 con 0x7fd50409e2f0 2024-09-16T14:50:39.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:39.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:39.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient(hunting): dump: 2024-09-16T14:50:39.874 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:39.874 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:39.874 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:39.875 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:39.875 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:39.875 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:39.875 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:39.875 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:39.875 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:39.875 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:39.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2101865299 0 0) 0x7fd50413b780 con 0x7fd50409df10 2024-09-16T14:50:39.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 >> v1:172.21.15.189:6789/0 conn(0x7fd50409e740 legacy=0x7fd5041d17a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:39.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 >> v1:172.21.15.26:6789/0 conn(0x7fd50409e2f0 legacy=0x7fd50409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:39.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient: _finish_hunting 0 2024-09-16T14:50:39.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 monclient: found mon.c 2024-09-16T14:50:39.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:39.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:39.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50406ce70 con 0x7fd50409df10 2024-09-16T14:50:39.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient: _finish_auth 0 2024-09-16T14:50:39.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient: _check_auth_tickets 2024-09-16T14:50:39.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:39.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 5 monclient: authenticate success, global_id 154727 2024-09-16T14:50:39.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 10 monclient: _renew_subs 2024-09-16T14:50:39.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:39.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 1 -- 172.21.15.189:0/99966608 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd50406b380 con 0x7fd50409df10 2024-09-16T14:50:39.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 10 monclient: _renew_subs 2024-09-16T14:50:39.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:39.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 1 -- 172.21.15.189:0/99966608 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd50413aee0 con 0x7fd50409df10 2024-09-16T14:50:39.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4f4001d10 con 0x7fd50409df10 2024-09-16T14:50:39.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:39.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd4f4004c40 con 0x7fd50409df10 2024-09-16T14:50:39.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd5017fa640 4 set_mon_vals no callback set 2024-09-16T14:50:39.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:39.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:39.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 10 monclient: dump: 2024-09-16T14:50:39.877 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:39.877 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:39.877 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:39.877 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:39.877 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:39.877 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:39.878 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:39.878 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:39.878 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:39.878 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:39.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:39.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:39.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:39.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd50943c640 1 -- 172.21.15.189:0/99966608 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e8005050 con 0x7fd50409df10 2024-09-16T14:50:39.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd4f400f730 con 0x7fd50409df10 2024-09-16T14:50:39.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(233..233 src has 169..233) v4 ==== 10265+0+0 (unknown 1368908577 0 0) 0x7fd4f4044c10 con 0x7fd50409df10 2024-09-16T14:50:39.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.860+0000 7fd4fa7fc640 10 monclient: get_auth_request con 0x7fd4e4036380 auth_method 0 2024-09-16T14:50:39.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.872+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd4f40144f0 con 0x7fd50409df10 2024-09-16T14:50:39.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.872+0000 7fd4fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:39.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:39.872+0000 7fd4fbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:40.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.040+0000 7fd50943c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T14:50:40.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.040+0000 7fd50943c640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T14:50:40.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.040+0000 7fd50943c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:40.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.040+0000 7fd50943c640 1 -- 172.21.15.189:0/99966608 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h19"} v 0) v1 -- 0x7fd4e8005300 con 0x7fd50409df10 2024-09-16T14:50:40.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.632+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h19"}]=0 removed item id -73 name 'h19' from crush map v234) v1 ==== 120+0+0 (unknown 2400711699 0 0) 0x7fd4f4014310 con 0x7fd50409df10 2024-09-16T14:50:40.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.632+0000 7fd4fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T14:50:40.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.632+0000 7fd4fbfff640 10 monclient: _finish_command 2 = system:0 removed item id -73 name 'h19' from crush map 2024-09-16T14:50:40.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.736+0000 7fd50943c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T14:50:40.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.736+0000 7fd50943c640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T14:50:40.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.736+0000 7fd50943c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:40.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.736+0000 7fd50943c640 1 -- 172.21.15.189:0/99966608 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h19"} v 0) v1 -- 0x7fd4e8005a40 con 0x7fd50409df10 2024-09-16T14:50:40.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.736+0000 7fd4fbfff640 1 -- 172.21.15.189:0/99966608 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h19"}]=0 device 'h19' does not appear in the crush map v234) v1 ==== 120+0+0 (unknown 3233416739 0 0) 0x7fd4f4014310 con 0x7fd50409df10 2024-09-16T14:50:40.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.736+0000 7fd4fbfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T14:50:40.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.736+0000 7fd4fbfff640 10 monclient: _finish_command 3 = system:0 device 'h19' does not appear in the crush map 2024-09-16T14:50:40.737 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h19' does not appear in the crush map 2024-09-16T14:50:40.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.736+0000 7fd50943c640 1 -- 172.21.15.189:0/99966608 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd4e4036380 legacy=0x7fd4e4038840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:40.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.736+0000 7fd50943c640 10 monclient: shutdown 2024-09-16T14:50:40.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.736+0000 7fd50943c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:40.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.736+0000 7fd50943c640 1 -- 172.21.15.189:0/99966608 >> v1:172.21.15.26:6790/0 conn(0x7fd50409df10 legacy=0x7fd50409a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:40.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.736+0000 7fd50943c640 1 -- 172.21.15.189:0/99966608 shutdown_connections 2024-09-16T14:50:40.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.736+0000 7fd50943c640 1 -- 172.21.15.189:0/99966608 >> 172.21.15.189:0/99966608 conn(0x7fd5040a3320 msgr2=0x7fd5040a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:40.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.736+0000 7fd50943c640 1 -- 172.21.15.189:0/99966608 shutdown_connections 2024-09-16T14:50:40.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.736+0000 7fd50943c640 1 -- 172.21.15.189:0/99966608 wait complete. 2024-09-16T14:50:40.754 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:40.754 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h20 2024-09-16T14:50:40.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.868+0000 7f4583c4d640 10 monclient: get_monmap_and_config 2024-09-16T14:50:40.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.868+0000 7f4583c4d640 10 monclient: build_initial_monmap 2024-09-16T14:50:40.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.868+0000 7f4583c4d640 10 monclient: monmap: 2024-09-16T14:50:40.871 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:40.871 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:40.871 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:40.872388+0000 2024-09-16T14:50:40.871 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:40.872388+0000 2024-09-16T14:50:40.871 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:40.871 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:40.871 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:40.871 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:40.872 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:40.872 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:40.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.868+0000 7f4583c4d640 1 Processor -- start 2024-09-16T14:50:40.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.868+0000 7f4583c4d640 1 -- start start 2024-09-16T14:50:40.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.868+0000 7f4583c4d640 10 monclient: init 2024-09-16T14:50:40.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:40.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:40.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 10 monclient(hunting): picked mon.noname-c con 0x7f457c09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:40.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 10 monclient(hunting): picked mon.noname-b con 0x7f457c09e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:40.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 10 monclient(hunting): picked mon.noname-a con 0x7f457c09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:40.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c09ee20 con 0x7f457c09e740 2024-09-16T14:50:40.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c09f100 con 0x7f457c09df10 2024-09-16T14:50:40.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45819e9640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f457c09df10 legacy=0x7f457c09d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:40.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c09f3e0 con 0x7f457c09e2f0 2024-09-16T14:50:40.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:40.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 10 monclient(hunting): authenticate will time out at 1422.988170s 2024-09-16T14:50:40.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45811e8640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f457c09e2f0 0x7f457c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56760/0 (socket says 172.21.15.189:56760) 2024-09-16T14:50:40.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45811e8640 1 -- 172.21.15.189:0/3053975868 learned_addr learned my addr 172.21.15.189:0/3053975868 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:40.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 1 -- 172.21.15.189:0/3053975868 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050993111 0 0) 0x7f457c09f3e0 con 0x7f457c09e2f0 2024-09-16T14:50:40.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient(hunting): _negotiate 2024-09-16T14:50:40.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:40.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:40.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient(hunting): my global_id is 154730 2024-09-16T14:50:40.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 1 -- 172.21.15.189:0/3053975868 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4568003530 con 0x7f457c09e2f0 2024-09-16T14:50:40.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 1 -- 172.21.15.189:0/3053975868 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567529002 0 0) 0x7f457c09ee20 con 0x7f457c09e740 2024-09-16T14:50:40.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient(hunting): _negotiate 2024-09-16T14:50:40.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:40.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:40.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient(hunting): my global_id is 164475 2024-09-16T14:50:40.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 1 -- 172.21.15.189:0/3053975868 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f457c09f3e0 con 0x7f457c09e740 2024-09-16T14:50:40.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 1 -- 172.21.15.189:0/3053975868 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2673508407 0 0) 0x7f4568003530 con 0x7f457c09e2f0 2024-09-16T14:50:40.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 1 -- 172.21.15.189:0/3053975868 >> v1:172.21.15.189:6789/0 conn(0x7f457c09df10 legacy=0x7f457c09d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:40.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 1 -- 172.21.15.189:0/3053975868 >> v1:172.21.15.26:6789/0 conn(0x7f457c09e740 legacy=0x7f457c13b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:40.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient: _finish_hunting 0 2024-09-16T14:50:40.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 1 monclient: found mon.noname-b 2024-09-16T14:50:40.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:40.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:40.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 1 -- 172.21.15.189:0/3053975868 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c09c690 con 0x7f457c09e2f0 2024-09-16T14:50:40.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient: _finish_auth 0 2024-09-16T14:50:40.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient: _check_auth_tickets 2024-09-16T14:50:40.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:40.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 1 -- 172.21.15.189:0/3053975868 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f456c001620 con 0x7f457c09e2f0 2024-09-16T14:50:40.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:40.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:40.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient: dump: 2024-09-16T14:50:40.878 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:40.879 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:40.879 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:40.879 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:40.879 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:40.879 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:40.879 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:40.879 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:40.879 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:40.879 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:40.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 5 monclient: authenticate success, global_id 154730 2024-09-16T14:50:40.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:40.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 1 -- 172.21.15.189:0/3053975868 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f456c001900 con 0x7f457c09e2f0 2024-09-16T14:50:40.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:40.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 1 -- 172.21.15.189:0/3053975868 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f456c001c90 con 0x7f457c09e2f0 2024-09-16T14:50:40.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:40.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 4 set_mon_vals no callback set 2024-09-16T14:50:40.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:40.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:40.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f45809e7640 10 monclient: dump: 2024-09-16T14:50:40.880 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:40.880 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:40.880 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:40.880 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:40.880 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:40.880 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:40.880 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:40.880 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:40.881 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:40.881 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:40.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 10 monclient: shutdown 2024-09-16T14:50:40.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:40.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 1 -- 172.21.15.189:0/3053975868 >> v1:172.21.15.26:6790/0 conn(0x7f457c09e2f0 legacy=0x7f457c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:40.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 1 -- 172.21.15.189:0/3053975868 shutdown_connections 2024-09-16T14:50:40.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 1 -- 172.21.15.189:0/3053975868 >> 172.21.15.189:0/3053975868 conn(0x7f457c0a3320 msgr2=0x7f457c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:40.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 1 -- 172.21.15.189:0/3053975868 shutdown_connections 2024-09-16T14:50:40.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 1 -- 172.21.15.189:0/3053975868 wait complete. 2024-09-16T14:50:40.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 10 monclient: build_initial_monmap 2024-09-16T14:50:40.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 10 monclient: monmap: 2024-09-16T14:50:40.881 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:40.881 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:40.881 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:40.882 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:40.882 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:40.882 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:40.882 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:40.882 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:40.882 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:40.882 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:40.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 1 Processor -- start 2024-09-16T14:50:40.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 1 -- start start 2024-09-16T14:50:40.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.872+0000 7f4583c4d640 10 monclient: init 2024-09-16T14:50:40.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:40.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:40.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 10 monclient(hunting): picked mon.noname-b con 0x7f457c09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:40.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 10 monclient(hunting): picked mon.noname-c con 0x7f457c09e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:40.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45819e9640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f457c09df10 legacy=0x7f457c132ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:40.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 10 monclient(hunting): picked mon.noname-a con 0x7f457c09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:40.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c06cdc0 con 0x7f457c09e740 2024-09-16T14:50:40.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c06b380 con 0x7f457c09df10 2024-09-16T14:50:40.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c11a440 con 0x7f457c09e2f0 2024-09-16T14:50:40.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:40.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 10 monclient(hunting): authenticate will time out at 1422.992218s 2024-09-16T14:50:40.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45811e8640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f457c09e2f0 0x7f457c1cbfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56764/0 (socket says 172.21.15.189:56764) 2024-09-16T14:50:40.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45811e8640 1 -- 172.21.15.189:0/2587257434 learned_addr learned my addr 172.21.15.189:0/2587257434 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:40.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517797436 0 0) 0x7f457c11a440 con 0x7f457c09e2f0 2024-09-16T14:50:40.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:40.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:40.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:40.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient(hunting): my global_id is 154733 2024-09-16T14:50:40.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4560003530 con 0x7f457c09e2f0 2024-09-16T14:50:40.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962400421 0 0) 0x7f457c06cdc0 con 0x7f457c09e740 2024-09-16T14:50:40.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:40.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:40.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:40.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient(hunting): my global_id is 164478 2024-09-16T14:50:40.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f457c11a440 con 0x7f457c09e740 2024-09-16T14:50:40.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3216896184 0 0) 0x7f4560003530 con 0x7f457c09e2f0 2024-09-16T14:50:40.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f457c06cdc0 con 0x7f457c09e2f0 2024-09-16T14:50:40.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f456c0015e0 con 0x7f457c09e2f0 2024-09-16T14:50:40.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:40.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:40.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient(hunting): dump: 2024-09-16T14:50:40.888 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:40.889 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:40.889 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:40.889 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:40.889 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:40.889 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:40.889 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:40.889 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:40.889 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:40.889 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:40.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:40.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1845060421 0 0) 0x7f457c11a440 con 0x7f457c09e740 2024-09-16T14:50:40.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4560003530 con 0x7f457c09e740 2024-09-16T14:50:40.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4574001c40 con 0x7f457c09e740 2024-09-16T14:50:40.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:40.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:40.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient(hunting): dump: 2024-09-16T14:50:40.890 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:40.890 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:40.890 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:40.890 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:40.890 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:40.890 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:40.890 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:40.890 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:40.890 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:40.890 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:40.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2303713297 0 0) 0x7f457c06cdc0 con 0x7f457c09e2f0 2024-09-16T14:50:40.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 >> v1:172.21.15.189:6789/0 conn(0x7f457c09df10 legacy=0x7f457c132ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:40.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 >> v1:172.21.15.26:6789/0 conn(0x7f457c09e740 legacy=0x7f457c1cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:40.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient: _finish_hunting 0 2024-09-16T14:50:40.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 monclient: found mon.c 2024-09-16T14:50:40.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:40.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:40.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c119ba0 con 0x7f457c09e2f0 2024-09-16T14:50:40.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient: _finish_auth 0 2024-09-16T14:50:40.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:40.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:40.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 5 monclient: authenticate success, global_id 154733 2024-09-16T14:50:40.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 10 monclient: _renew_subs 2024-09-16T14:50:40.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:40.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 1 -- 172.21.15.189:0/2587257434 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457c1331f0 con 0x7f457c09e2f0 2024-09-16T14:50:40.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 10 monclient: _renew_subs 2024-09-16T14:50:40.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:40.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 1 -- 172.21.15.189:0/2587257434 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f457c13c270 con 0x7f457c09e2f0 2024-09-16T14:50:40.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f456c004c00 con 0x7f457c09e2f0 2024-09-16T14:50:40.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:40.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f456c004f10 con 0x7f457c09e2f0 2024-09-16T14:50:40.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:40.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:40.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45737fe640 4 set_mon_vals no callback set 2024-09-16T14:50:40.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 10 monclient: dump: 2024-09-16T14:50:40.892 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:40.892 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:40.893 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:40.893 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:40.893 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:40.893 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:40.893 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:40.893 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:40.893 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:40.893 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:40.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:40.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:40.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:40.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f4583c4d640 1 -- 172.21.15.189:0/2587257434 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4564005050 con 0x7f457c09e2f0 2024-09-16T14:50:40.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f456c00f980 con 0x7f457c09e2f0 2024-09-16T14:50:40.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(234..234 src has 169..234) v4 ==== 10052+0+0 (unknown 3645877242 0 0) 0x7f456c044c00 con 0x7f457c09e2f0 2024-09-16T14:50:40.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.876+0000 7f45819e9640 10 monclient: get_auth_request con 0x7f4560036270 auth_method 0 2024-09-16T14:50:40.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.888+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f456c014680 con 0x7f457c09e2f0 2024-09-16T14:50:40.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.888+0000 7f45727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:40.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:40.888+0000 7f45727fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:41.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.052+0000 7f4583c4d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T14:50:41.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.052+0000 7f4583c4d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T14:50:41.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.052+0000 7f4583c4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:41.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.052+0000 7f4583c4d640 1 -- 172.21.15.189:0/2587257434 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h20"} v 0) v1 -- 0x7f4564005300 con 0x7f457c09e2f0 2024-09-16T14:50:41.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.288+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h20"}]=0 removed item id -76 name 'h20' from crush map v235) v1 ==== 120+0+0 (unknown 2095629348 0 0) 0x7f456c0144a0 con 0x7f457c09e2f0 2024-09-16T14:50:41.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.288+0000 7f45727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T14:50:41.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.288+0000 7f45727fc640 10 monclient: _finish_command 2 = system:0 removed item id -76 name 'h20' from crush map 2024-09-16T14:50:41.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.384+0000 7f4583c4d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T14:50:41.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.384+0000 7f4583c4d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T14:50:41.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.384+0000 7f4583c4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:41.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.384+0000 7f4583c4d640 1 -- 172.21.15.189:0/2587257434 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h20"} v 0) v1 -- 0x7f4564005a40 con 0x7f457c09e2f0 2024-09-16T14:50:41.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.384+0000 7f45727fc640 1 -- 172.21.15.189:0/2587257434 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h20"}]=0 device 'h20' does not appear in the crush map v235) v1 ==== 120+0+0 (unknown 985644766 0 0) 0x7f456c0144a0 con 0x7f457c09e2f0 2024-09-16T14:50:41.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.384+0000 7f45727fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T14:50:41.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.384+0000 7f45727fc640 10 monclient: _finish_command 3 = system:0 device 'h20' does not appear in the crush map 2024-09-16T14:50:41.386 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h20' does not appear in the crush map 2024-09-16T14:50:41.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.384+0000 7f4583c4d640 1 -- 172.21.15.189:0/2587257434 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4560036270 legacy=0x7f4560038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:41.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.384+0000 7f4583c4d640 10 monclient: shutdown 2024-09-16T14:50:41.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.384+0000 7f4583c4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:41.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.384+0000 7f4583c4d640 1 -- 172.21.15.189:0/2587257434 >> v1:172.21.15.26:6790/0 conn(0x7f457c09e2f0 legacy=0x7f457c1cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:41.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.384+0000 7f4583c4d640 1 -- 172.21.15.189:0/2587257434 shutdown_connections 2024-09-16T14:50:41.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.384+0000 7f4583c4d640 1 -- 172.21.15.189:0/2587257434 >> 172.21.15.189:0/2587257434 conn(0x7f457c0a3320 msgr2=0x7f457c071530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:41.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.384+0000 7f4583c4d640 1 -- 172.21.15.189:0/2587257434 shutdown_connections 2024-09-16T14:50:41.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.384+0000 7f4583c4d640 1 -- 172.21.15.189:0/2587257434 wait complete. 2024-09-16T14:50:41.402 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:41.402 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h21 2024-09-16T14:50:41.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2244490640 10 monclient: get_monmap_and_config 2024-09-16T14:50:41.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2244490640 10 monclient: build_initial_monmap 2024-09-16T14:50:41.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2244490640 10 monclient: monmap: 2024-09-16T14:50:41.521 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:41.521 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:41.521 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:41.522983+0000 2024-09-16T14:50:41.521 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:41.522983+0000 2024-09-16T14:50:41.522 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:41.522 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:41.522 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:41.522 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:41.522 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:41.522 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:41.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2244490640 1 Processor -- start 2024-09-16T14:50:41.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2244490640 1 -- start start 2024-09-16T14:50:41.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2244490640 10 monclient: init 2024-09-16T14:50:41.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2244490640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:41.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2244490640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:41.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2244490640 10 monclient(hunting): picked mon.noname-b con 0x7f223c073590 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:41.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2244490640 10 monclient(hunting): picked mon.noname-a con 0x7f223c132b40 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:41.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2244490640 10 monclient(hunting): picked mon.noname-c con 0x7f223c073970 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:41.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2244490640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223c09cc90 con 0x7f223c132b40 2024-09-16T14:50:41.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2244490640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223c133010 con 0x7f223c073970 2024-09-16T14:50:41.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2244490640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223c09cfb0 con 0x7f223c073590 2024-09-16T14:50:41.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2244490640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:41.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2244490640 10 monclient(hunting): authenticate will time out at 1423.639163s 2024-09-16T14:50:41.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2242a2d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f223c073970 legacy=0x7f223c13b080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:41.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f224222c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f223c073590 0x7f223c099680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56772/0 (socket says 172.21.15.189:56772) 2024-09-16T14:50:41.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f2241a2b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f223c132b40 0x7f223c137950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47088/0 (socket says 172.21.15.189:47088) 2024-09-16T14:50:41.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.520+0000 7f224222c640 1 -- 172.21.15.189:0/3912061273 learned_addr learned my addr 172.21.15.189:0/3912061273 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:41.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 1 -- 172.21.15.189:0/3912061273 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283528836 0 0) 0x7f223c09cc90 con 0x7f223c132b40 2024-09-16T14:50:41.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:41.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:41.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:41.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient(hunting): my global_id is 164481 2024-09-16T14:50:41.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 1 -- 172.21.15.189:0/3912061273 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2228003530 con 0x7f223c132b40 2024-09-16T14:50:41.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 1 -- 172.21.15.189:0/3912061273 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916189951 0 0) 0x7f223c09cfb0 con 0x7f223c073590 2024-09-16T14:50:41.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:41.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:41.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:41.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient(hunting): my global_id is 154736 2024-09-16T14:50:41.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 1 -- 172.21.15.189:0/3912061273 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f223c09cc90 con 0x7f223c073590 2024-09-16T14:50:41.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 1 -- 172.21.15.189:0/3912061273 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2996850146 0 0) 0x7f2228003530 con 0x7f223c132b40 2024-09-16T14:50:41.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 1 -- 172.21.15.189:0/3912061273 >> v1:172.21.15.26:6790/0 conn(0x7f223c073590 legacy=0x7f223c099680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:41.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 1 -- 172.21.15.189:0/3912061273 >> v1:172.21.15.189:6789/0 conn(0x7f223c073970 legacy=0x7f223c13b080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:41.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient: _finish_hunting 0 2024-09-16T14:50:41.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 1 monclient: found mon.noname-a 2024-09-16T14:50:41.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:41.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:41.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 1 -- 172.21.15.189:0/3912061273 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c097680 con 0x7f223c132b40 2024-09-16T14:50:41.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient: _finish_auth 0 2024-09-16T14:50:41.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient: _check_auth_tickets 2024-09-16T14:50:41.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:41.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 1 -- 172.21.15.189:0/3912061273 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f222c001550 con 0x7f223c132b40 2024-09-16T14:50:41.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 5 monclient: authenticate success, global_id 164481 2024-09-16T14:50:41.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:41.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:41.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:41.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient: dump: 2024-09-16T14:50:41.532 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:41.532 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:41.532 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:41.532 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:41.532 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:41.532 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:41.533 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:41.533 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:41.533 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:41.533 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:41.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:41.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 1 -- 172.21.15.189:0/3912061273 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f222c001830 con 0x7f223c132b40 2024-09-16T14:50:41.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:41.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 1 -- 172.21.15.189:0/3912061273 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f222c001c00 con 0x7f223c132b40 2024-09-16T14:50:41.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:41.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 4 set_mon_vals no callback set 2024-09-16T14:50:41.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:41.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:41.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224122a640 10 monclient: dump: 2024-09-16T14:50:41.534 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:41.534 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:41.534 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:41.534 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:41.534 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:41.534 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:41.534 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:41.534 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:41.534 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:41.534 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:41.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 10 monclient: shutdown 2024-09-16T14:50:41.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:41.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 1 -- 172.21.15.189:0/3912061273 >> v1:172.21.15.26:6789/0 conn(0x7f223c132b40 legacy=0x7f223c137950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:41.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 1 -- 172.21.15.189:0/3912061273 shutdown_connections 2024-09-16T14:50:41.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 1 -- 172.21.15.189:0/3912061273 >> 172.21.15.189:0/3912061273 conn(0x7f223c0a0030 msgr2=0x7f223c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:41.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 1 -- 172.21.15.189:0/3912061273 shutdown_connections 2024-09-16T14:50:41.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 1 -- 172.21.15.189:0/3912061273 wait complete. 2024-09-16T14:50:41.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 10 monclient: build_initial_monmap 2024-09-16T14:50:41.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 10 monclient: monmap: 2024-09-16T14:50:41.535 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:41.535 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:41.535 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:41.535 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:41.535 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:41.535 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:41.535 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:41.535 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:41.536 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:41.536 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:41.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 1 Processor -- start 2024-09-16T14:50:41.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 1 -- start start 2024-09-16T14:50:41.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 10 monclient: init 2024-09-16T14:50:41.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:41.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:41.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 10 monclient(hunting): picked mon.noname-c con 0x7f223c073590 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:41.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 10 monclient(hunting): picked mon.noname-a con 0x7f223c073970 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:41.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 10 monclient(hunting): picked mon.noname-b con 0x7f223c132b40 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:41.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223c128d60 con 0x7f223c073970 2024-09-16T14:50:41.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223c09e490 con 0x7f223c132b40 2024-09-16T14:50:41.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223c0716c0 con 0x7f223c073590 2024-09-16T14:50:41.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:41.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2244490640 10 monclient(hunting): authenticate will time out at 1423.643145s 2024-09-16T14:50:41.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2242a2d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f223c132b40 legacy=0x7f223c1d18b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:41.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224222c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f223c073590 0x7f223c09a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56784/0 (socket says 172.21.15.189:56784) 2024-09-16T14:50:41.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f224222c640 1 -- 172.21.15.189:0/2835147209 learned_addr learned my addr 172.21.15.189:0/2835147209 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:41.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141220968 0 0) 0x7f223c0716c0 con 0x7f223c073590 2024-09-16T14:50:41.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2232ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:41.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2232ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:41.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2232ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:41.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2232ffd640 10 monclient(hunting): my global_id is 154739 2024-09-16T14:50:41.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2220003530 con 0x7f223c073590 2024-09-16T14:50:41.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800816234 0 0) 0x7f223c128d60 con 0x7f223c073970 2024-09-16T14:50:41.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2232ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:41.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2232ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:41.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2232ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:41.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2232ffd640 10 monclient(hunting): my global_id is 164484 2024-09-16T14:50:41.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.524+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f223c0716c0 con 0x7f223c073970 2024-09-16T14:50:41.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2791959443 0 0) 0x7f2220003530 con 0x7f223c073590 2024-09-16T14:50:41.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f223c128d60 con 0x7f223c073590 2024-09-16T14:50:41.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22380018c0 con 0x7f223c073590 2024-09-16T14:50:41.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:41.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:41.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:41.538 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:41.538 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:41.539 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:41.539 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:41.539 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:41.539 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:41.539 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:41.539 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:41.539 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:41.539 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:41.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:41.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3418513422 0 0) 0x7f223c0716c0 con 0x7f223c073970 2024-09-16T14:50:41.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2220003530 con 0x7f223c073970 2024-09-16T14:50:41.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f222c001b40 con 0x7f223c073970 2024-09-16T14:50:41.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:41.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:41.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:41.540 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:41.540 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:41.540 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:41.540 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:41.540 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:41.540 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:41.540 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:41.540 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:41.540 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:41.540 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:41.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 146385216 0 0) 0x7f223c128d60 con 0x7f223c073590 2024-09-16T14:50:41.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 >> v1:172.21.15.189:6789/0 conn(0x7f223c132b40 legacy=0x7f223c1d18b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:41.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 >> v1:172.21.15.26:6789/0 conn(0x7f223c073970 legacy=0x7f223c09af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:41.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:41.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 1 monclient: found mon.c 2024-09-16T14:50:41.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:41.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:41.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c09b680 con 0x7f223c073590 2024-09-16T14:50:41.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:41.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:41.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:41.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2244490640 5 monclient: authenticate success, global_id 154739 2024-09-16T14:50:41.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2244490640 10 monclient: _renew_subs 2024-09-16T14:50:41.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2244490640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:41.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2244490640 1 -- 172.21.15.189:0/2835147209 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f223c132570 con 0x7f223c073590 2024-09-16T14:50:41.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2244490640 10 monclient: _renew_subs 2024-09-16T14:50:41.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2244490640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:41.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2244490640 1 -- 172.21.15.189:0/2835147209 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f223c1284c0 con 0x7f223c073590 2024-09-16T14:50:41.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22380047f0 con 0x7f223c073590 2024-09-16T14:50:41.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:41.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2233fff640 4 set_mon_vals no callback set 2024-09-16T14:50:41.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2238001a20 con 0x7f223c073590 2024-09-16T14:50:41.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:41.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:41.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 10 monclient: dump: 2024-09-16T14:50:41.542 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:41.542 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:41.542 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:41.543 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:41.543 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:41.543 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:41.543 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:41.543 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:41.543 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:41.543 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:41.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2244490640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:41.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2244490640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:41.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2244490640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:41.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2244490640 1 -- 172.21.15.189:0/2835147209 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2224005050 con 0x7f223c073590 2024-09-16T14:50:41.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f223800f410 con 0x7f223c073590 2024-09-16T14:50:41.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(235..235 src has 169..235) v4 ==== 9839+0+0 (unknown 1047287045 0 0) 0x7f2238044850 con 0x7f223c073590 2024-09-16T14:50:41.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.528+0000 7f2241a2b640 10 monclient: get_auth_request con 0x7f22200361e0 auth_method 0 2024-09-16T14:50:41.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.540+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2238014300 con 0x7f223c073590 2024-09-16T14:50:41.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.540+0000 7f2232ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:41.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.540+0000 7f2232ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:41.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.704+0000 7f2244490640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T14:50:41.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.704+0000 7f2244490640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T14:50:41.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.704+0000 7f2244490640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:41.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:41.704+0000 7f2244490640 1 -- 172.21.15.189:0/2835147209 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h21"} v 0) v1 -- 0x7f2224005300 con 0x7f223c073590 2024-09-16T14:50:42.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.644+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h21"}]=0 removed item id -79 name 'h21' from crush map v236) v1 ==== 120+0+0 (unknown 122553328 0 0) 0x7f22380129a0 con 0x7f223c073590 2024-09-16T14:50:42.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.644+0000 7f2232ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T14:50:42.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.644+0000 7f2232ffd640 10 monclient: _finish_command 2 = system:0 removed item id -79 name 'h21' from crush map 2024-09-16T14:50:42.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.744+0000 7f2244490640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T14:50:42.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.744+0000 7f2244490640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T14:50:42.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.744+0000 7f2244490640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:42.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.744+0000 7f2244490640 1 -- 172.21.15.189:0/2835147209 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h21"} v 0) v1 -- 0x7f2224005a40 con 0x7f223c073590 2024-09-16T14:50:42.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.744+0000 7f2232ffd640 1 -- 172.21.15.189:0/2835147209 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h21"}]=0 device 'h21' does not appear in the crush map v236) v1 ==== 120+0+0 (unknown 2686244125 0 0) 0x7f2238014120 con 0x7f223c073590 2024-09-16T14:50:42.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.744+0000 7f2232ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T14:50:42.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.744+0000 7f2232ffd640 10 monclient: _finish_command 3 = system:0 device 'h21' does not appear in the crush map 2024-09-16T14:50:42.747 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h21' does not appear in the crush map 2024-09-16T14:50:42.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.748+0000 7f2244490640 1 -- 172.21.15.189:0/2835147209 >> v1:172.21.15.189:6812/3196161582 conn(0x7f22200361e0 legacy=0x7f22200386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:42.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.748+0000 7f2244490640 10 monclient: shutdown 2024-09-16T14:50:42.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.748+0000 7f2244490640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:42.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.748+0000 7f2244490640 1 -- 172.21.15.189:0/2835147209 >> v1:172.21.15.26:6790/0 conn(0x7f223c073590 legacy=0x7f223c09a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:42.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.748+0000 7f2244490640 1 -- 172.21.15.189:0/2835147209 shutdown_connections 2024-09-16T14:50:42.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.748+0000 7f2244490640 1 -- 172.21.15.189:0/2835147209 >> 172.21.15.189:0/2835147209 conn(0x7f223c0a0030 msgr2=0x7f223c13bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:42.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.748+0000 7f2244490640 1 -- 172.21.15.189:0/2835147209 shutdown_connections 2024-09-16T14:50:42.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.748+0000 7f2244490640 1 -- 172.21.15.189:0/2835147209 wait complete. 2024-09-16T14:50:42.764 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:42.764 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h22 2024-09-16T14:50:42.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ecacf8640 10 monclient: get_monmap_and_config 2024-09-16T14:50:42.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ecacf8640 10 monclient: build_initial_monmap 2024-09-16T14:50:42.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ecacf8640 10 monclient: monmap: 2024-09-16T14:50:42.881 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:42.881 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:42.881 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:42.882744+0000 2024-09-16T14:50:42.882 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:42.882744+0000 2024-09-16T14:50:42.882 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:42.882 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:42.882 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:42.882 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:42.882 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:42.882 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:42.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ecacf8640 1 Processor -- start 2024-09-16T14:50:42.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ecacf8640 1 -- start start 2024-09-16T14:50:42.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ecacf8640 10 monclient: init 2024-09-16T14:50:42.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ecacf8640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:42.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ecacf8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:42.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ecacf8640 10 monclient(hunting): picked mon.noname-b con 0x7f3ec406c120 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:42.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ecacf8640 10 monclient(hunting): picked mon.noname-c con 0x7f3ec41353d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:42.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ecacf8640 10 monclient(hunting): picked mon.noname-a con 0x7f3ec40a1040 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:42.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ecacf8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec406c500 con 0x7f3ec40a1040 2024-09-16T14:50:42.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ecacf8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec40a2ef0 con 0x7f3ec41353d0 2024-09-16T14:50:42.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ecacf8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec409e990 con 0x7f3ec406c120 2024-09-16T14:50:42.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ecacf8640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:42.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ec3fff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f3ec41353d0 legacy=0x7f3ec4139b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:42.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ecacf8640 10 monclient(hunting): authenticate will time out at 1424.999436s 2024-09-16T14:50:42.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ec9295640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3ec40a1040 0x7f3ec413d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47112/0 (socket says 172.21.15.189:47112) 2024-09-16T14:50:42.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ec8a94640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3ec406c120 0x7f3ec4134cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56792/0 (socket says 172.21.15.189:56792) 2024-09-16T14:50:42.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.880+0000 7f3ec9295640 1 -- 172.21.15.189:0/1871143560 learned_addr learned my addr 172.21.15.189:0/1871143560 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:42.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 1 -- 172.21.15.189:0/1871143560 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375183310 0 0) 0x7f3ec406c500 con 0x7f3ec40a1040 2024-09-16T14:50:42.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:42.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:42.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:42.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient(hunting): my global_id is 164487 2024-09-16T14:50:42.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 1 -- 172.21.15.189:0/1871143560 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eb4003530 con 0x7f3ec40a1040 2024-09-16T14:50:42.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 1 -- 172.21.15.189:0/1871143560 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851808166 0 0) 0x7f3ec409e990 con 0x7f3ec406c120 2024-09-16T14:50:42.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:42.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:42.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:42.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient(hunting): my global_id is 154742 2024-09-16T14:50:42.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 1 -- 172.21.15.189:0/1871143560 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ec406c500 con 0x7f3ec406c120 2024-09-16T14:50:42.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 1 -- 172.21.15.189:0/1871143560 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2554818998 0 0) 0x7f3eb4003530 con 0x7f3ec40a1040 2024-09-16T14:50:42.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 1 -- 172.21.15.189:0/1871143560 >> v1:172.21.15.26:6790/0 conn(0x7f3ec406c120 legacy=0x7f3ec4134cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:42.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 1 -- 172.21.15.189:0/1871143560 >> v1:172.21.15.189:6789/0 conn(0x7f3ec41353d0 legacy=0x7f3ec4139b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:42.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:42.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 1 monclient: found mon.noname-a 2024-09-16T14:50:42.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:42.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:42.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 1 -- 172.21.15.189:0/1871143560 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec409ec70 con 0x7f3ec40a1040 2024-09-16T14:50:42.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:42.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:42.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:42.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 1 -- 172.21.15.189:0/1871143560 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ebc0019a0 con 0x7f3ec40a1040 2024-09-16T14:50:42.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:42.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:42.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient: dump: 2024-09-16T14:50:42.890 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:42.890 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:42.890 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:42.890 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:42.890 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:42.890 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:42.890 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:42.890 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:42.890 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:42.890 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:42.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 5 monclient: authenticate success, global_id 164487 2024-09-16T14:50:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 1 -- 172.21.15.189:0/1871143560 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ebc001c10 con 0x7f3ec40a1040 2024-09-16T14:50:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 1 -- 172.21.15.189:0/1871143560 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ebc001fa0 con 0x7f3ec40a1040 2024-09-16T14:50:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec37fe640 10 monclient: dump: 2024-09-16T14:50:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:42.892 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:42.892 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:42.892 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:42.892 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:42.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:42.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 4 set_mon_vals no callback set 2024-09-16T14:50:42.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 10 monclient: shutdown 2024-09-16T14:50:42.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:42.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 1 -- 172.21.15.189:0/1871143560 >> v1:172.21.15.26:6789/0 conn(0x7f3ec40a1040 legacy=0x7f3ec413d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:42.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 1 -- 172.21.15.189:0/1871143560 shutdown_connections 2024-09-16T14:50:42.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 1 -- 172.21.15.189:0/1871143560 >> 172.21.15.189:0/1871143560 conn(0x7f3ec40a3a40 msgr2=0x7f3ec40a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:42.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 1 -- 172.21.15.189:0/1871143560 shutdown_connections 2024-09-16T14:50:42.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 1 -- 172.21.15.189:0/1871143560 wait complete. 2024-09-16T14:50:42.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 10 monclient: build_initial_monmap 2024-09-16T14:50:42.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 10 monclient: monmap: 2024-09-16T14:50:42.893 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:42.893 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:42.893 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:42.893 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:42.893 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:42.893 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:42.893 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:42.894 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:42.894 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:42.894 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:42.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 1 Processor -- start 2024-09-16T14:50:42.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 1 -- start start 2024-09-16T14:50:42.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 10 monclient: init 2024-09-16T14:50:42.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:42.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:42.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 10 monclient(hunting): picked mon.noname-b con 0x7f3ec406c120 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:42.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 10 monclient(hunting): picked mon.noname-a con 0x7f3ec40a1040 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:42.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ec8a94640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f3ec406c120 legacy=0x7f3ec41381a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:42.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 10 monclient(hunting): picked mon.noname-c con 0x7f3ec41353d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:42.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec413c0d0 con 0x7f3ec40a1040 2024-09-16T14:50:42.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec409e130 con 0x7f3ec406c120 2024-09-16T14:50:42.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec4138fc0 con 0x7f3ec41353d0 2024-09-16T14:50:42.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:42.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.884+0000 7f3ecacf8640 10 monclient(hunting): authenticate will time out at 1425.003906s 2024-09-16T14:50:42.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec9295640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3ec41353d0 0x7f3ec41d1aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56798/0 (socket says 172.21.15.189:56798) 2024-09-16T14:50:42.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec9295640 1 -- 172.21.15.189:0/2609843203 learned_addr learned my addr 172.21.15.189:0/2609843203 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:42.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719217202 0 0) 0x7f3ec413c0d0 con 0x7f3ec40a1040 2024-09-16T14:50:42.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:50:42.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:42.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:42.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient(hunting): my global_id is 164490 2024-09-16T14:50:42.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ea8003560 con 0x7f3ec40a1040 2024-09-16T14:50:42.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180275918 0 0) 0x7f3ec4138fc0 con 0x7f3ec41353d0 2024-09-16T14:50:42.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:50:42.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:42.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:42.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient(hunting): my global_id is 154745 2024-09-16T14:50:42.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ec413c0d0 con 0x7f3ec41353d0 2024-09-16T14:50:42.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 797352039 0 0) 0x7f3ea8003560 con 0x7f3ec40a1040 2024-09-16T14:50:42.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ec4138fc0 con 0x7f3ec40a1040 2024-09-16T14:50:42.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3eb00040d0 con 0x7f3ec40a1040 2024-09-16T14:50:42.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:42.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:42.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient(hunting): dump: 2024-09-16T14:50:42.901 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:42.901 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:42.901 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:42.901 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:42.901 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:42.901 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:42.902 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:42.902 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:42.902 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:42.902 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:42.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:42.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 495215805 0 0) 0x7f3ec413c0d0 con 0x7f3ec41353d0 2024-09-16T14:50:42.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ea8003560 con 0x7f3ec41353d0 2024-09-16T14:50:42.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ebc001f20 con 0x7f3ec41353d0 2024-09-16T14:50:42.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:42.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:42.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient(hunting): dump: 2024-09-16T14:50:42.902 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:42.902 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:42.903 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:42.903 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:42.903 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:42.903 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:42.903 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:42.903 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:42.903 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:42.903 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:42.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3088918845 0 0) 0x7f3ec4138fc0 con 0x7f3ec40a1040 2024-09-16T14:50:42.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 >> v1:172.21.15.26:6790/0 conn(0x7f3ec41353d0 legacy=0x7f3ec41d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:42.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 >> v1:172.21.15.189:6789/0 conn(0x7f3ec406c120 legacy=0x7f3ec41381a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:42.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient: _finish_hunting 0 2024-09-16T14:50:42.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 monclient: found mon.a 2024-09-16T14:50:42.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:42.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:42.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec413b510 con 0x7f3ec40a1040 2024-09-16T14:50:42.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient: _finish_auth 0 2024-09-16T14:50:42.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient: _check_auth_tickets 2024-09-16T14:50:42.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:42.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ecacf8640 5 monclient: authenticate success, global_id 164490 2024-09-16T14:50:42.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ecacf8640 10 monclient: _renew_subs 2024-09-16T14:50:42.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ecacf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:42.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ecacf8640 1 -- 172.21.15.189:0/2609843203 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec409ce20 con 0x7f3ec40a1040 2024-09-16T14:50:42.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ecacf8640 10 monclient: _renew_subs 2024-09-16T14:50:42.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ecacf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:42.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ecacf8640 1 -- 172.21.15.189:0/2609843203 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec4139d00 con 0x7f3ec40a1040 2024-09-16T14:50:42.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3eb00042f0 con 0x7f3ec40a1040 2024-09-16T14:50:42.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:42.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3eb0004600 con 0x7f3ec40a1040 2024-09-16T14:50:42.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:42.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:42.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec27fc640 4 set_mon_vals no callback set 2024-09-16T14:50:42.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 10 monclient: dump: 2024-09-16T14:50:42.905 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:42.905 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:42.905 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:42.906 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:42.906 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:42.906 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:42.906 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:42.906 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:42.906 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:42.906 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:42.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ecacf8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:42.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ecacf8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:42.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ecacf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:42.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ecacf8640 1 -- 172.21.15.189:0/2609843203 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea4005050 con 0x7f3ec40a1040 2024-09-16T14:50:42.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3eb00047e0 con 0x7f3ec40a1040 2024-09-16T14:50:42.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(236..236 src has 169..236) v4 ==== 9626+0+0 (unknown 3927924674 0 0) 0x7f3eb0044840 con 0x7f3ec40a1040 2024-09-16T14:50:42.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.888+0000 7f3ec8a94640 10 monclient: get_auth_request con 0x7f3ea8036250 auth_method 0 2024-09-16T14:50:42.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.900+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3eb0005000 con 0x7f3ec40a1040 2024-09-16T14:50:42.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.900+0000 7f3ec17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:42.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:42.900+0000 7f3ec17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:43.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.064+0000 7f3ecacf8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T14:50:43.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.064+0000 7f3ecacf8640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T14:50:43.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.064+0000 7f3ecacf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:43.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.064+0000 7f3ecacf8640 1 -- 172.21.15.189:0/2609843203 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h22"} v 0) v1 -- 0x7f3ea4005300 con 0x7f3ec40a1040 2024-09-16T14:50:43.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.644+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h22"}]=0 removed item id -82 name 'h22' from crush map v237) v1 ==== 120+0+0 (unknown 1988280063 0 0) 0x7f3eb0012a60 con 0x7f3ec40a1040 2024-09-16T14:50:43.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.644+0000 7f3ec17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T14:50:43.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.644+0000 7f3ec17fa640 10 monclient: _finish_command 2 = system:0 removed item id -82 name 'h22' from crush map 2024-09-16T14:50:43.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.744+0000 7f3ecacf8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T14:50:43.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.744+0000 7f3ecacf8640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T14:50:43.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.744+0000 7f3ecacf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:43.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.744+0000 7f3ecacf8640 1 -- 172.21.15.189:0/2609843203 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h22"} v 0) v1 -- 0x7f3ea40055a0 con 0x7f3ec40a1040 2024-09-16T14:50:43.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.748+0000 7f3ec17fa640 1 -- 172.21.15.189:0/2609843203 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h22"}]=0 device 'h22' does not appear in the crush map v237) v1 ==== 120+0+0 (unknown 1674451769 0 0) 0x7f3eb00141e0 con 0x7f3ec40a1040 2024-09-16T14:50:43.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.748+0000 7f3ec17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T14:50:43.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.748+0000 7f3ec17fa640 10 monclient: _finish_command 3 = system:0 device 'h22' does not appear in the crush map 2024-09-16T14:50:43.749 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h22' does not appear in the crush map 2024-09-16T14:50:43.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.748+0000 7f3ecacf8640 1 -- 172.21.15.189:0/2609843203 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3ea8036250 legacy=0x7f3ea8038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:43.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.748+0000 7f3ecacf8640 10 monclient: shutdown 2024-09-16T14:50:43.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.748+0000 7f3ecacf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:43.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.748+0000 7f3ecacf8640 1 -- 172.21.15.189:0/2609843203 >> v1:172.21.15.26:6789/0 conn(0x7f3ec40a1040 legacy=0x7f3ec41388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:43.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.748+0000 7f3ecacf8640 1 -- 172.21.15.189:0/2609843203 shutdown_connections 2024-09-16T14:50:43.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.748+0000 7f3ecacf8640 1 -- 172.21.15.189:0/2609843203 >> 172.21.15.189:0/2609843203 conn(0x7f3ec40a3a40 msgr2=0x7f3ec413e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:43.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.748+0000 7f3ecacf8640 1 -- 172.21.15.189:0/2609843203 shutdown_connections 2024-09-16T14:50:43.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.748+0000 7f3ecacf8640 1 -- 172.21.15.189:0/2609843203 wait complete. 2024-09-16T14:50:43.765 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:43.766 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h23 2024-09-16T14:50:43.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd75860b640 10 monclient: get_monmap_and_config 2024-09-16T14:50:43.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd75860b640 10 monclient: build_initial_monmap 2024-09-16T14:50:43.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd75860b640 10 monclient: monmap: 2024-09-16T14:50:43.886 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:43.886 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:43.886 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:43.887262+0000 2024-09-16T14:50:43.886 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:43.887262+0000 2024-09-16T14:50:43.886 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:43.886 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:43.886 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:43.886 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:43.886 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:43.886 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:43.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd75860b640 1 Processor -- start 2024-09-16T14:50:43.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd75860b640 1 -- start start 2024-09-16T14:50:43.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd75860b640 10 monclient: init 2024-09-16T14:50:43.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd75860b640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:43.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd75860b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:43.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd75860b640 10 monclient(hunting): picked mon.noname-b con 0x7fd75009df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:43.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd75860b640 10 monclient(hunting): picked mon.noname-a con 0x7fd75009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:43.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd75860b640 10 monclient(hunting): picked mon.noname-c con 0x7fd75009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:43.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd75860b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75009ee20 con 0x7fd75009e2f0 2024-09-16T14:50:43.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd75860b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75009f100 con 0x7fd75009e740 2024-09-16T14:50:43.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd75860b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75009f3e0 con 0x7fd75009df10 2024-09-16T14:50:43.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd75860b640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:43.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd75860b640 10 monclient(hunting): authenticate will time out at 1426.003148s 2024-09-16T14:50:43.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd756ba8640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fd75009e740 legacy=0x7fd75013b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:43.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd7563a7640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd75009df10 0x7fd75009d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56802/0 (socket says 172.21.15.189:56802) 2024-09-16T14:50:43.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd755ba6640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd75009e2f0 0x7fd7501378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47144/0 (socket says 172.21.15.189:47144) 2024-09-16T14:50:43.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd7563a7640 1 -- 172.21.15.189:0/2107409534 learned_addr learned my addr 172.21.15.189:0/2107409534 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:43.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.884+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773496857 0 0) 0x7fd75009f3e0 con 0x7fd75009df10 2024-09-16T14:50:43.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): _negotiate 2024-09-16T14:50:43.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): my global_id is 154748 2024-09-16T14:50:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd73c003530 con 0x7fd75009df10 2024-09-16T14:50:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322377103 0 0) 0x7fd75009ee20 con 0x7fd75009e2f0 2024-09-16T14:50:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): _negotiate 2024-09-16T14:50:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): my global_id is 164493 2024-09-16T14:50:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd75009f3e0 con 0x7fd75009e2f0 2024-09-16T14:50:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 324140793 0 0) 0x7fd73c003530 con 0x7fd75009df10 2024-09-16T14:50:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 >> v1:172.21.15.189:6789/0 conn(0x7fd75009e740 legacy=0x7fd75013b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 >> v1:172.21.15.26:6789/0 conn(0x7fd75009e2f0 legacy=0x7fd7501378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: _finish_hunting 0 2024-09-16T14:50:43.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 monclient: found mon.noname-b 2024-09-16T14:50:43.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:43.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:43.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75009c690 con 0x7fd75009df10 2024-09-16T14:50:43.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: _finish_auth 0 2024-09-16T14:50:43.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: _check_auth_tickets 2024-09-16T14:50:43.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:43.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd74c001620 con 0x7fd75009df10 2024-09-16T14:50:43.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:43.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:43.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: dump: 2024-09-16T14:50:43.893 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:43.893 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:43.894 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:43.894 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:43.894 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:43.894 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:43.894 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:43.894 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:43.894 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:43.894 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:43.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 5 monclient: authenticate success, global_id 154748 2024-09-16T14:50:43.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:43.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7563a7640 1 -- 172.21.15.189:0/2107409534 >> v1:172.21.15.26:6790/0 conn(0x7fd75009df10 legacy=0x7fd75009d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T14:50:43.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7563a7640 1 -- 172.21.15.189:0/2107409534 >> v1:172.21.15.26:6790/0 conn(0x7fd75009df10 legacy=0x7fd75009d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:50:43.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7563a7640 1 --1- 172.21.15.189:0/2107409534 >> v1:172.21.15.26:6790/0 conn(0x7fd75009df10 0x7fd75009d6f0 :-1 s=OPENED pgs=1053 cs=1 l=1).handle_message read tag failed 2024-09-16T14:50:43.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7563a7640 1 --1- 172.21.15.189:0/2107409534 >> v1:172.21.15.26:6790/0 conn(0x7fd75009df10 0x7fd75009d6f0 :-1 s=OPENED pgs=1053 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:50:43.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: ms_handle_reset current mon v1:172.21.15.26:6790/0 2024-09-16T14:50:43.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:43.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 >> v1:172.21.15.26:6790/0 conn(0x7fd75009df10 legacy=0x7fd75009d6f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:43.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:43.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): picked mon.c con 0x7fd73c003c70 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:43.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): picked mon.a con 0x7fd75009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:43.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): picked mon.b con 0x7fd75009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:43.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd73c003530 con 0x7fd75009e2f0 2024-09-16T14:50:43.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd75009f100 con 0x7fd75009e740 2024-09-16T14:50:43.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd75009ee20 con 0x7fd73c003c70 2024-09-16T14:50:43.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:43.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6790/0 2024-09-16T14:50:43.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd756ba8640 1 -- 172.21.15.189:0/2107409534 >> v1:172.21.15.189:6789/0 conn(0x7fd75009e740 legacy=0x7fd73c00d9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:43.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469692309 0 0) 0x7fd75009ee20 con 0x7fd73c003c70 2024-09-16T14:50:43.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): _negotiate 2024-09-16T14:50:43.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:43.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:50:43.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 --> v1:172.21.15.26:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fd73c00f0d0 con 0x7fd73c003c70 2024-09-16T14:50:43.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268298898 0 0) 0x7fd73c003530 con 0x7fd75009e2f0 2024-09-16T14:50:43.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): _negotiate 2024-09-16T14:50:43.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:43.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:50:43.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 --> v1:172.21.15.26:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fd75009ee20 con 0x7fd75009e2f0 2024-09-16T14:50:43.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2110423736 0 0) 0x7fd73c00f0d0 con 0x7fd73c003c70 2024-09-16T14:50:43.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 >> v1:172.21.15.189:6789/0 conn(0x7fd75009e740 legacy=0x7fd73c00d9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:43.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 >> v1:172.21.15.26:6789/0 conn(0x7fd75009e2f0 legacy=0x7fd73c004950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:43.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: _finish_hunting 0 2024-09-16T14:50:43.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 monclient: found mon.c 2024-09-16T14:50:43.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:43.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:43.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7fd73c0055d0 con 0x7fd73c003c70 2024-09-16T14:50:43.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: _finish_auth 0 2024-09-16T14:50:43.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: _check_auth_tickets 2024-09-16T14:50:43.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:43.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd74c0048b0 con 0x7fd73c003c70 2024-09-16T14:50:43.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:43.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:43.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: dump: 2024-09-16T14:50:43.897 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:43.897 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:43.898 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:43.898 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:43.898 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:43.898 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:43.898 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:43.898 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:43.898 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:43.898 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:43.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:43.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 1 -- 172.21.15.189:0/2107409534 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd74c004a50 con 0x7fd73c003c70 2024-09-16T14:50:43.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd7553a5640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:43.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:43.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 4 set_mon_vals no callback set 2024-09-16T14:50:43.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 10 monclient: shutdown 2024-09-16T14:50:43.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:43.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 1 -- 172.21.15.189:0/2107409534 >> v1:172.21.15.26:6790/0 conn(0x7fd73c003c70 legacy=0x7fd73c004050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:43.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 1 -- 172.21.15.189:0/2107409534 shutdown_connections 2024-09-16T14:50:43.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 1 -- 172.21.15.189:0/2107409534 >> 172.21.15.189:0/2107409534 conn(0x7fd7500a3320 msgr2=0x7fd75009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:43.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 1 -- 172.21.15.189:0/2107409534 shutdown_connections 2024-09-16T14:50:43.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 1 -- 172.21.15.189:0/2107409534 wait complete. 2024-09-16T14:50:43.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 10 monclient: build_initial_monmap 2024-09-16T14:50:43.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 10 monclient: monmap: 2024-09-16T14:50:43.903 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:43.903 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:43.903 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:43.903 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:43.903 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:43.903 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:43.903 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:43.904 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:43.904 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:43.904 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:43.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 1 Processor -- start 2024-09-16T14:50:43.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 1 -- start start 2024-09-16T14:50:43.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.888+0000 7fd75860b640 10 monclient: init 2024-09-16T14:50:43.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:43.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:43.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 10 monclient(hunting): picked mon.noname-b con 0x7fd73c003c70 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:43.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 10 monclient(hunting): picked mon.noname-a con 0x7fd75009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:43.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 10 monclient(hunting): picked mon.noname-c con 0x7fd75009e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:43.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75006b380 con 0x7fd75009e2f0 2024-09-16T14:50:43.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd7563a7640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fd73c003c70 legacy=0x7fd750131890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:43.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75006fa00 con 0x7fd73c003c70 2024-09-16T14:50:43.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75006cef0 con 0x7fd75009e740 2024-09-16T14:50:43.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:43.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 10 monclient(hunting): authenticate will time out at 1426.008379s 2024-09-16T14:50:43.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd756ba8640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd75009e740 0x7fd7501cf610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56820/0 (socket says 172.21.15.189:56820) 2024-09-16T14:50:43.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd756ba8640 1 -- 172.21.15.189:0/3057467361 learned_addr learned my addr 172.21.15.189:0/3057467361 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:43.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472741307 0 0) 0x7fd75006cef0 con 0x7fd75009e740 2024-09-16T14:50:43.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:43.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:43.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:43.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient(hunting): my global_id is 154751 2024-09-16T14:50:43.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd734003530 con 0x7fd75009e740 2024-09-16T14:50:43.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134624499 0 0) 0x7fd75006b380 con 0x7fd75009e2f0 2024-09-16T14:50:43.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:43.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:43.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:43.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient(hunting): my global_id is 164496 2024-09-16T14:50:43.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd75006cef0 con 0x7fd75009e2f0 2024-09-16T14:50:43.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1832978845 0 0) 0x7fd734003530 con 0x7fd75009e740 2024-09-16T14:50:43.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd75006b380 con 0x7fd75009e740 2024-09-16T14:50:43.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd7480043c0 con 0x7fd75009e740 2024-09-16T14:50:43.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:43.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:43.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:43.906 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:43.906 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:43.907 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:43.907 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:43.907 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:43.907 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:43.907 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:43.907 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:43.907 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:43.907 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:43.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:43.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4105188567 0 0) 0x7fd75006cef0 con 0x7fd75009e2f0 2024-09-16T14:50:43.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd734003530 con 0x7fd75009e2f0 2024-09-16T14:50:43.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd740001900 con 0x7fd75009e2f0 2024-09-16T14:50:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:43.908 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2242662822 0 0) 0x7fd75006b380 con 0x7fd75009e740 2024-09-16T14:50:43.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 >> v1:172.21.15.189:6789/0 conn(0x7fd73c003c70 legacy=0x7fd750131890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:43.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 >> v1:172.21.15.26:6789/0 conn(0x7fd75009e2f0 legacy=0x7fd7501cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:43.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:43.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 monclient: found mon.c 2024-09-16T14:50:43.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:43.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:43.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75011f250 con 0x7fd75009e740 2024-09-16T14:50:43.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:43.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:43.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:43.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 5 monclient: authenticate success, global_id 154751 2024-09-16T14:50:43.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 10 monclient: _renew_subs 2024-09-16T14:50:43.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:43.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 1 -- 172.21.15.189:0/3057467361 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd75011d840 con 0x7fd75009e740 2024-09-16T14:50:43.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 10 monclient: _renew_subs 2024-09-16T14:50:43.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:43.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 1 -- 172.21.15.189:0/3057467361 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75011cef0 con 0x7fd75009e740 2024-09-16T14:50:43.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7480045e0 con 0x7fd75009e740 2024-09-16T14:50:43.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:43.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd748004970 con 0x7fd75009e740 2024-09-16T14:50:43.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:43.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd754ba4640 4 set_mon_vals no callback set 2024-09-16T14:50:43.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:43.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 10 monclient: dump: 2024-09-16T14:50:43.910 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:43.910 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:43.911 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd75860b640 1 -- 172.21.15.189:0/3057467361 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd738005050 con 0x7fd75009e740 2024-09-16T14:50:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd748004b50 con 0x7fd75009e740 2024-09-16T14:50:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(237..237 src has 169..237) v4 ==== 9413+0+0 (unknown 2907942727 0 0) 0x7fd7480449c0 con 0x7fd75009e740 2024-09-16T14:50:43.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.892+0000 7fd7563a7640 10 monclient: get_auth_request con 0x7fd734036200 auth_method 0 2024-09-16T14:50:43.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.904+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd748014660 con 0x7fd75009e740 2024-09-16T14:50:43.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.904+0000 7fd746ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:43.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:43.904+0000 7fd746ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:44.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.068+0000 7fd75860b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T14:50:44.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.068+0000 7fd75860b640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T14:50:44.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.068+0000 7fd75860b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:44.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.068+0000 7fd75860b640 1 -- 172.21.15.189:0/3057467361 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h23"} v 0) v1 -- 0x7fd738005300 con 0x7fd75009e740 2024-09-16T14:50:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.660+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h23"}]=0 removed item id -85 name 'h23' from crush map v238) v1 ==== 120+0+0 (unknown 1539084720 0 0) 0x7fd748012d00 con 0x7fd75009e740 2024-09-16T14:50:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.660+0000 7fd746ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T14:50:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.660+0000 7fd746ffd640 10 monclient: _finish_command 2 = system:0 removed item id -85 name 'h23' from crush map 2024-09-16T14:50:44.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.764+0000 7fd75860b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T14:50:44.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.764+0000 7fd75860b640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T14:50:44.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.764+0000 7fd75860b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:44.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.764+0000 7fd75860b640 1 -- 172.21.15.189:0/3057467361 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h23"} v 0) v1 -- 0x7fd738005a40 con 0x7fd75009e740 2024-09-16T14:50:44.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.764+0000 7fd746ffd640 1 -- 172.21.15.189:0/3057467361 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h23"}]=0 device 'h23' does not appear in the crush map v238) v1 ==== 120+0+0 (unknown 3447780018 0 0) 0x7fd748014480 con 0x7fd75009e740 2024-09-16T14:50:44.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.764+0000 7fd746ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T14:50:44.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.764+0000 7fd746ffd640 10 monclient: _finish_command 3 = system:0 device 'h23' does not appear in the crush map 2024-09-16T14:50:44.767 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h23' does not appear in the crush map 2024-09-16T14:50:44.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.768+0000 7fd75860b640 1 -- 172.21.15.189:0/3057467361 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd734036200 legacy=0x7fd7340386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:44.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.768+0000 7fd75860b640 10 monclient: shutdown 2024-09-16T14:50:44.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.768+0000 7fd75860b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:44.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.768+0000 7fd75860b640 1 -- 172.21.15.189:0/3057467361 >> v1:172.21.15.26:6790/0 conn(0x7fd75009e740 legacy=0x7fd7501cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:44.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.768+0000 7fd75860b640 1 -- 172.21.15.189:0/3057467361 shutdown_connections 2024-09-16T14:50:44.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.768+0000 7fd75860b640 1 -- 172.21.15.189:0/3057467361 >> 172.21.15.189:0/3057467361 conn(0x7fd7500a3320 msgr2=0x7fd7501b7250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:44.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.768+0000 7fd75860b640 1 -- 172.21.15.189:0/3057467361 shutdown_connections 2024-09-16T14:50:44.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.768+0000 7fd75860b640 1 -- 172.21.15.189:0/3057467361 wait complete. 2024-09-16T14:50:44.784 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:44.784 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h24 2024-09-16T14:50:44.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cf82c640 10 monclient: get_monmap_and_config 2024-09-16T14:50:44.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cf82c640 10 monclient: build_initial_monmap 2024-09-16T14:50:44.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cf82c640 10 monclient: monmap: 2024-09-16T14:50:44.885 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:44.885 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:44.885 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:44.886818+0000 2024-09-16T14:50:44.885 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:44.886818+0000 2024-09-16T14:50:44.886 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:44.886 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:44.886 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:44.886 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:44.886 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:44.886 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:44.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cf82c640 1 Processor -- start 2024-09-16T14:50:44.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cf82c640 1 -- start start 2024-09-16T14:50:44.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cf82c640 10 monclient: init 2024-09-16T14:50:44.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cf82c640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:44.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cf82c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:44.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cf82c640 10 monclient(hunting): picked mon.noname-b con 0x7f50c809df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:44.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cf82c640 10 monclient(hunting): picked mon.noname-c con 0x7f50c809e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:44.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cf82c640 10 monclient(hunting): picked mon.noname-a con 0x7f50c809e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:44.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cf82c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c809ee20 con 0x7f50c809e740 2024-09-16T14:50:44.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cf82c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c809f100 con 0x7f50c809e2f0 2024-09-16T14:50:44.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cf82c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c809f3e0 con 0x7f50c809df10 2024-09-16T14:50:44.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cf82c640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:44.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cf82c640 10 monclient(hunting): authenticate will time out at 1427.002709s 2024-09-16T14:50:44.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50ccdc7640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f50c809e2f0 legacy=0x7f50c81378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:44.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cddc9640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f50c809e740 0x7f50c813b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47166/0 (socket says 172.21.15.189:47166) 2024-09-16T14:50:44.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cddc9640 1 -- 172.21.15.189:0/535778822 learned_addr learned my addr 172.21.15.189:0/535778822 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:44.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50cd5c8640 1 --1- 172.21.15.189:0/535778822 >> v1:172.21.15.26:6790/0 conn(0x7f50c809df10 0x7f50c809d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56834/0 (socket says 172.21.15.189:56834) 2024-09-16T14:50:44.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50c7fff640 1 -- 172.21.15.189:0/535778822 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691581546 0 0) 0x7f50c809ee20 con 0x7f50c809e740 2024-09-16T14:50:44.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50c7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:44.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:44.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:44.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50c7fff640 10 monclient(hunting): my global_id is 164499 2024-09-16T14:50:44.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50c7fff640 1 -- 172.21.15.189:0/535778822 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b0003530 con 0x7f50c809e740 2024-09-16T14:50:44.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50c7fff640 1 -- 172.21.15.189:0/535778822 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245621123 0 0) 0x7f50c809f3e0 con 0x7f50c809df10 2024-09-16T14:50:44.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50c7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:44.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:44.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:44.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50c7fff640 10 monclient(hunting): my global_id is 154754 2024-09-16T14:50:44.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.884+0000 7f50c7fff640 1 -- 172.21.15.189:0/535778822 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c809ee20 con 0x7f50c809df10 2024-09-16T14:50:44.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 1 -- 172.21.15.189:0/535778822 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1463877161 0 0) 0x7f50b0003530 con 0x7f50c809e740 2024-09-16T14:50:44.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 1 -- 172.21.15.189:0/535778822 >> v1:172.21.15.26:6790/0 conn(0x7f50c809df10 legacy=0x7f50c809d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:44.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 1 -- 172.21.15.189:0/535778822 >> v1:172.21.15.189:6789/0 conn(0x7f50c809e2f0 legacy=0x7f50c81378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:44.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:50:44.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 1 monclient: found mon.noname-a 2024-09-16T14:50:44.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:44.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:44.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 1 -- 172.21.15.189:0/535778822 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c809c690 con 0x7f50c809e740 2024-09-16T14:50:44.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 10 monclient: _finish_auth 0 2024-09-16T14:50:44.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:50:44.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:44.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 1 -- 172.21.15.189:0/535778822 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f50b8001620 con 0x7f50c809e740 2024-09-16T14:50:44.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:44.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:44.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 10 monclient: dump: 2024-09-16T14:50:44.896 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:44.896 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:44.896 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:44.896 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:44.896 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:44.897 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:44.897 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:44.897 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:44.897 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:44.897 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:44.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 5 monclient: authenticate success, global_id 164499 2024-09-16T14:50:44.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:44.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 1 -- 172.21.15.189:0/535778822 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50b8001900 con 0x7f50c809e740 2024-09-16T14:50:44.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:44.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:44.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 4 set_mon_vals no callback set 2024-09-16T14:50:44.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 1 -- 172.21.15.189:0/535778822 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f50b8001c90 con 0x7f50c809e740 2024-09-16T14:50:44.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:44.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:44.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c7fff640 10 monclient: dump: 2024-09-16T14:50:44.898 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:44.898 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:44.898 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:44.898 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:44.898 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:44.898 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:44.898 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:44.898 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:44.898 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:44.898 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:44.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 10 monclient: shutdown 2024-09-16T14:50:44.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:44.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 1 -- 172.21.15.189:0/535778822 >> v1:172.21.15.26:6789/0 conn(0x7f50c809e740 legacy=0x7f50c813b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:44.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 1 -- 172.21.15.189:0/535778822 shutdown_connections 2024-09-16T14:50:44.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 1 -- 172.21.15.189:0/535778822 >> 172.21.15.189:0/535778822 conn(0x7f50c80a3320 msgr2=0x7f50c809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:44.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 1 -- 172.21.15.189:0/535778822 shutdown_connections 2024-09-16T14:50:44.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 1 -- 172.21.15.189:0/535778822 wait complete. 2024-09-16T14:50:44.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 10 monclient: build_initial_monmap 2024-09-16T14:50:44.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 10 monclient: monmap: 2024-09-16T14:50:44.899 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:44.899 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:44.899 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:44.899 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:44.899 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:44.900 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:44.900 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:44.900 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:44.900 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:44.900 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:44.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 1 Processor -- start 2024-09-16T14:50:44.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 1 -- start start 2024-09-16T14:50:44.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 10 monclient: init 2024-09-16T14:50:44.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:44.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:44.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 10 monclient(hunting): picked mon.noname-a con 0x7f50c809df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:44.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 10 monclient(hunting): picked mon.noname-c con 0x7f50c809e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:44.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 10 monclient(hunting): picked mon.noname-b con 0x7f50c809e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:44.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c8124880 con 0x7f50c809df10 2024-09-16T14:50:44.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c8125680 con 0x7f50c809e740 2024-09-16T14:50:44.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c8125960 con 0x7f50c809e2f0 2024-09-16T14:50:44.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:44.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cf82c640 10 monclient(hunting): authenticate will time out at 1427.006595s 2024-09-16T14:50:44.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cddc9640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f50c809e740 legacy=0x7f50c81d20c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:44.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cd5c8640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f50c809df10 0x7f50c81c99b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47182/0 (socket says 172.21.15.189:47182) 2024-09-16T14:50:44.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50cd5c8640 1 -- 172.21.15.189:0/3435364318 learned_addr learned my addr 172.21.15.189:0/3435364318 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:44.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2012153956 0 0) 0x7f50c8124880 con 0x7f50c809df10 2024-09-16T14:50:44.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:44.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:44.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:44.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 10 monclient(hunting): my global_id is 164502 2024-09-16T14:50:44.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b4003530 con 0x7f50c809df10 2024-09-16T14:50:44.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235192445 0 0) 0x7f50c8125960 con 0x7f50c809e2f0 2024-09-16T14:50:44.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:44.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:44.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:44.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 10 monclient(hunting): my global_id is 154757 2024-09-16T14:50:44.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c8124880 con 0x7f50c809e2f0 2024-09-16T14:50:44.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2039341926 0 0) 0x7f50b4003530 con 0x7f50c809df10 2024-09-16T14:50:44.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50c8125960 con 0x7f50c809df10 2024-09-16T14:50:44.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f50c0001bb0 con 0x7f50c809df10 2024-09-16T14:50:44.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:44.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:44.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 10 monclient(hunting): dump: 2024-09-16T14:50:44.902 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:44.903 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.888+0000 7f50c5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2396932733 0 0) 0x7f50c8124880 con 0x7f50c809e2f0 2024-09-16T14:50:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50b4003530 con 0x7f50c809e2f0 2024-09-16T14:50:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f50bc0040d0 con 0x7f50c809e2f0 2024-09-16T14:50:44.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:44.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:44.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 10 monclient(hunting): dump: 2024-09-16T14:50:44.904 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:44.904 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:44.904 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:44.904 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:44.904 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:44.904 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:44.904 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:44.904 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:44.904 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:44.904 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:44.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1861842583 0 0) 0x7f50c8125960 con 0x7f50c809df10 2024-09-16T14:50:44.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 >> v1:172.21.15.26:6790/0 conn(0x7f50c809e2f0 legacy=0x7f50c81ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:44.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 >> v1:172.21.15.189:6789/0 conn(0x7f50c809e740 legacy=0x7f50c81d20c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:44.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:50:44.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 1 monclient: found mon.a 2024-09-16T14:50:44.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:44.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:44.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c806fa30 con 0x7f50c809df10 2024-09-16T14:50:44.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 10 monclient: _finish_auth 0 2024-09-16T14:50:44.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:44.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:44.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50cf82c640 5 monclient: authenticate success, global_id 164502 2024-09-16T14:50:44.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50cf82c640 10 monclient: _renew_subs 2024-09-16T14:50:44.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50cf82c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:44.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50cf82c640 1 -- 172.21.15.189:0/3435364318 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c806b380 con 0x7f50c809df10 2024-09-16T14:50:44.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50cf82c640 10 monclient: _renew_subs 2024-09-16T14:50:44.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50cf82c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:44.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50cf82c640 1 -- 172.21.15.189:0/3435364318 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c8125c40 con 0x7f50c809df10 2024-09-16T14:50:44.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50c0004b70 con 0x7f50c809df10 2024-09-16T14:50:44.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:44.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c77fe640 4 set_mon_vals no callback set 2024-09-16T14:50:44.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f50c0004e80 con 0x7f50c809df10 2024-09-16T14:50:44.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:44.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:44.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 10 monclient: dump: 2024-09-16T14:50:44.906 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:44.906 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:44.907 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:44.907 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:44.907 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:44.907 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:44.907 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:44.907 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:44.907 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:44.907 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:44.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50cf82c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:44.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50cf82c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:44.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50cf82c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:44.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50cf82c640 1 -- 172.21.15.189:0/3435364318 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a8005050 con 0x7f50c809df10 2024-09-16T14:50:44.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f50c000f8f0 con 0x7f50c809df10 2024-09-16T14:50:44.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(238..238 src has 169..238) v4 ==== 9200+0+0 (unknown 1365634938 0 0) 0x7f50c00449b0 con 0x7f50c809df10 2024-09-16T14:50:44.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.892+0000 7f50ccdc7640 10 monclient: get_auth_request con 0x7f50b4036250 auth_method 0 2024-09-16T14:50:44.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.904+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f50c0012d60 con 0x7f50c809df10 2024-09-16T14:50:44.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.904+0000 7f50c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:44.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:44.904+0000 7f50c5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:45.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.056+0000 7f50cf82c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T14:50:45.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.056+0000 7f50cf82c640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T14:50:45.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.056+0000 7f50cf82c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:45.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.056+0000 7f50cf82c640 1 -- 172.21.15.189:0/3435364318 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h24"} v 0) v1 -- 0x7f50a8005300 con 0x7f50c809df10 2024-09-16T14:50:45.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.664+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h24"}]=0 removed item id -88 name 'h24' from crush map v239) v1 ==== 120+0+0 (unknown 1464002232 0 0) 0x7f50c0012d60 con 0x7f50c809df10 2024-09-16T14:50:45.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.664+0000 7f50c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T14:50:45.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.664+0000 7f50c5ffb640 10 monclient: _finish_command 2 = system:0 removed item id -88 name 'h24' from crush map 2024-09-16T14:50:45.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.760+0000 7f50cf82c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T14:50:45.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.760+0000 7f50cf82c640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T14:50:45.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.760+0000 7f50cf82c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:45.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.760+0000 7f50cf82c640 1 -- 172.21.15.189:0/3435364318 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h24"} v 0) v1 -- 0x7f50a8005a40 con 0x7f50c809df10 2024-09-16T14:50:45.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.760+0000 7f50c5ffb640 1 -- 172.21.15.189:0/3435364318 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h24"}]=0 device 'h24' does not appear in the crush map v239) v1 ==== 120+0+0 (unknown 3783782784 0 0) 0x7f50c0015220 con 0x7f50c809df10 2024-09-16T14:50:45.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.760+0000 7f50c5ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T14:50:45.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.760+0000 7f50c5ffb640 10 monclient: _finish_command 3 = system:0 device 'h24' does not appear in the crush map 2024-09-16T14:50:45.764 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h24' does not appear in the crush map 2024-09-16T14:50:45.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.764+0000 7f50cf82c640 1 -- 172.21.15.189:0/3435364318 >> v1:172.21.15.189:6812/3196161582 conn(0x7f50b4036250 legacy=0x7f50b4038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:45.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.764+0000 7f50cf82c640 10 monclient: shutdown 2024-09-16T14:50:45.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.764+0000 7f50cf82c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:45.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.764+0000 7f50cf82c640 1 -- 172.21.15.189:0/3435364318 >> v1:172.21.15.26:6789/0 conn(0x7f50c809df10 legacy=0x7f50c81c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:45.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.764+0000 7f50cf82c640 1 -- 172.21.15.189:0/3435364318 shutdown_connections 2024-09-16T14:50:45.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.764+0000 7f50cf82c640 1 -- 172.21.15.189:0/3435364318 >> 172.21.15.189:0/3435364318 conn(0x7f50c80a3320 msgr2=0x7f50c806ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:45.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.764+0000 7f50cf82c640 1 -- 172.21.15.189:0/3435364318 shutdown_connections 2024-09-16T14:50:45.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.764+0000 7f50cf82c640 1 -- 172.21.15.189:0/3435364318 wait complete. 2024-09-16T14:50:45.780 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:45.780 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h25 2024-09-16T14:50:45.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f0510579640 10 monclient: get_monmap_and_config 2024-09-16T14:50:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f0510579640 10 monclient: build_initial_monmap 2024-09-16T14:50:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f0510579640 10 monclient: monmap: 2024-09-16T14:50:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:45.899490+0000 2024-09-16T14:50:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:45.899490+0000 2024-09-16T14:50:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:45.898 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:45.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f0510579640 1 Processor -- start 2024-09-16T14:50:45.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f0510579640 1 -- start start 2024-09-16T14:50:45.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f0510579640 10 monclient: init 2024-09-16T14:50:45.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f0510579640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:45.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f0510579640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:45.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f0510579640 10 monclient(hunting): picked mon.noname-b con 0x7f05081318a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:45.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f0510579640 10 monclient(hunting): picked mon.noname-c con 0x7f0508134d50 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:45.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f0510579640 10 monclient(hunting): picked mon.noname-a con 0x7f050813b2b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:45.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f0510579640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050809d8a0 con 0x7f050813b2b0 2024-09-16T14:50:45.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f0510579640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050809ddf0 con 0x7f0508134d50 2024-09-16T14:50:45.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f0510579640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050809e8b0 con 0x7f05081318a0 2024-09-16T14:50:45.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f0510579640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:45.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f050db14640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f0508134d50 legacy=0x7f0508139b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:45.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f0510579640 10 monclient(hunting): authenticate will time out at 1428.015173s 2024-09-16T14:50:45.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f050eb16640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f050813b2b0 0x7f050813d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47188/0 (socket says 172.21.15.189:47188) 2024-09-16T14:50:45.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f050e315640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f05081318a0 0x7f0508131c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56848/0 (socket says 172.21.15.189:56848) 2024-09-16T14:50:45.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.896+0000 7f050eb16640 1 -- 172.21.15.189:0/426632178 learned_addr learned my addr 172.21.15.189:0/426632178 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:45.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 1 -- 172.21.15.189:0/426632178 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050202199 0 0) 0x7f050809e8b0 con 0x7f05081318a0 2024-09-16T14:50:45.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient(hunting): _negotiate 2024-09-16T14:50:45.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:45.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:45.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient(hunting): my global_id is 154760 2024-09-16T14:50:45.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 1 -- 172.21.15.189:0/426632178 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f4003530 con 0x7f05081318a0 2024-09-16T14:50:45.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 1 -- 172.21.15.189:0/426632178 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562815640 0 0) 0x7f050809d8a0 con 0x7f050813b2b0 2024-09-16T14:50:45.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient(hunting): _negotiate 2024-09-16T14:50:45.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:45.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:45.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient(hunting): my global_id is 164505 2024-09-16T14:50:45.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 1 -- 172.21.15.189:0/426632178 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f050809e8b0 con 0x7f050813b2b0 2024-09-16T14:50:45.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 1 -- 172.21.15.189:0/426632178 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 103042610 0 0) 0x7f04f4003530 con 0x7f05081318a0 2024-09-16T14:50:45.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 1 -- 172.21.15.189:0/426632178 >> v1:172.21.15.189:6789/0 conn(0x7f0508134d50 legacy=0x7f0508139b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:45.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 1 -- 172.21.15.189:0/426632178 >> v1:172.21.15.26:6789/0 conn(0x7f050813b2b0 legacy=0x7f050813d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:45.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient: _finish_hunting 0 2024-09-16T14:50:45.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 1 monclient: found mon.noname-b 2024-09-16T14:50:45.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:45.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:45.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 1 -- 172.21.15.189:0/426632178 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050809eb90 con 0x7f05081318a0 2024-09-16T14:50:45.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient: _finish_auth 0 2024-09-16T14:50:45.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient: _check_auth_tickets 2024-09-16T14:50:45.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:45.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 1 -- 172.21.15.189:0/426632178 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0504001620 con 0x7f05081318a0 2024-09-16T14:50:45.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 5 monclient: authenticate success, global_id 154760 2024-09-16T14:50:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient: dump: 2024-09-16T14:50:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:45.906 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:45.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:45.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 1 -- 172.21.15.189:0/426632178 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0504001840 con 0x7f05081318a0 2024-09-16T14:50:45.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:45.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 1 -- 172.21.15.189:0/426632178 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0504001b50 con 0x7f05081318a0 2024-09-16T14:50:45.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:45.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:45.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050d313640 10 monclient: dump: 2024-09-16T14:50:45.906 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:45.906 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:45.906 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:45.906 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:45.906 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:45.907 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:45.907 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:45.907 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:45.907 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:45.907 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:45.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:45.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 4 set_mon_vals no callback set 2024-09-16T14:50:45.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 10 monclient: shutdown 2024-09-16T14:50:45.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:45.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 1 -- 172.21.15.189:0/426632178 >> v1:172.21.15.26:6790/0 conn(0x7f05081318a0 legacy=0x7f0508131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:45.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 1 -- 172.21.15.189:0/426632178 shutdown_connections 2024-09-16T14:50:45.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 1 -- 172.21.15.189:0/426632178 >> 172.21.15.189:0/426632178 conn(0x7f05080a3320 msgr2=0x7f050809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:45.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 1 -- 172.21.15.189:0/426632178 shutdown_connections 2024-09-16T14:50:45.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 1 -- 172.21.15.189:0/426632178 wait complete. 2024-09-16T14:50:45.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 10 monclient: build_initial_monmap 2024-09-16T14:50:45.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 10 monclient: monmap: 2024-09-16T14:50:45.908 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:45.908 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:45.908 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:45.908 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:45.908 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:45.908 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:45.908 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:45.908 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:45.908 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:45.908 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:45.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 1 Processor -- start 2024-09-16T14:50:45.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 1 -- start start 2024-09-16T14:50:45.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 10 monclient: init 2024-09-16T14:50:45.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:45.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:45.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 10 monclient(hunting): picked mon.noname-b con 0x7f05081318a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:45.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 10 monclient(hunting): picked mon.noname-a con 0x7f0508134d50 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:45.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 10 monclient(hunting): picked mon.noname-c con 0x7f050813b2b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:45.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050813acd0 con 0x7f0508134d50 2024-09-16T14:50:45.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050806b380 con 0x7f05081318a0 2024-09-16T14:50:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05080a1db0 con 0x7f050813b2b0 2024-09-16T14:50:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f0510579640 10 monclient(hunting): authenticate will time out at 1428.018793s 2024-09-16T14:50:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050e315640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f05081318a0 legacy=0x7f050812ff20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050eb16640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f050813b2b0 0x7f05081d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56858/0 (socket says 172.21.15.189:56858) 2024-09-16T14:50:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050db14640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0508134d50 0x7f05081ce670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47198/0 (socket says 172.21.15.189:47198) 2024-09-16T14:50:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f050eb16640 1 -- 172.21.15.189:0/3207459205 learned_addr learned my addr 172.21.15.189:0/3207459205 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972867014 0 0) 0x7f05080a1db0 con 0x7f050813b2b0 2024-09-16T14:50:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f04feffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f04feffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f04feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f04feffd640 10 monclient(hunting): my global_id is 154763 2024-09-16T14:50:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ec003530 con 0x7f050813b2b0 2024-09-16T14:50:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862607269 0 0) 0x7f050813acd0 con 0x7f0508134d50 2024-09-16T14:50:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f04feffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f04feffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f04feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f04feffd640 10 monclient(hunting): my global_id is 164508 2024-09-16T14:50:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.900+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05080a1db0 con 0x7f0508134d50 2024-09-16T14:50:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1827434034 0 0) 0x7f04ec003530 con 0x7f050813b2b0 2024-09-16T14:50:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f050813acd0 con 0x7f050813b2b0 2024-09-16T14:50:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0500001bc0 con 0x7f050813b2b0 2024-09-16T14:50:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 10 monclient(hunting): dump: 2024-09-16T14:50:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:45.916 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1754050899 0 0) 0x7f05080a1db0 con 0x7f0508134d50 2024-09-16T14:50:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04ec003530 con 0x7f0508134d50 2024-09-16T14:50:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f04f80040d0 con 0x7f0508134d50 2024-09-16T14:50:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 10 monclient(hunting): dump: 2024-09-16T14:50:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:45.918 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 178435794 0 0) 0x7f050813acd0 con 0x7f050813b2b0 2024-09-16T14:50:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 >> v1:172.21.15.189:6789/0 conn(0x7f05081318a0 legacy=0x7f050812ff20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 >> v1:172.21.15.26:6789/0 conn(0x7f0508134d50 legacy=0x7f05081ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 1 monclient: found mon.c 2024-09-16T14:50:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050811b990 con 0x7f050813b2b0 2024-09-16T14:50:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f0510579640 5 monclient: authenticate success, global_id 154763 2024-09-16T14:50:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f0510579640 10 monclient: _renew_subs 2024-09-16T14:50:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f0510579640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f0510579640 1 -- 172.21.15.189:0/3207459205 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050811bc70 con 0x7f050813b2b0 2024-09-16T14:50:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f0510579640 10 monclient: _renew_subs 2024-09-16T14:50:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f0510579640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f0510579640 1 -- 172.21.15.189:0/3207459205 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0508139ef0 con 0x7f050813b2b0 2024-09-16T14:50:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0500004b70 con 0x7f050813b2b0 2024-09-16T14:50:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0500004e80 con 0x7f050813b2b0 2024-09-16T14:50:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 10 monclient: dump: 2024-09-16T14:50:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:45.920 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:45.920 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:45.920 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:45.920 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:45.920 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:45.920 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:45.920 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:45.920 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:45.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04fffff640 4 set_mon_vals no callback set 2024-09-16T14:50:45.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f0510579640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:45.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f0510579640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:45.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f0510579640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:45.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f0510579640 1 -- 172.21.15.189:0/3207459205 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f0005050 con 0x7f050813b2b0 2024-09-16T14:50:45.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f050000f8f0 con 0x7f050813b2b0 2024-09-16T14:50:45.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(239..239 src has 169..239) v4 ==== 8987+0+0 (unknown 2450892270 0 0) 0x7f0500044820 con 0x7f050813b2b0 2024-09-16T14:50:45.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.904+0000 7f050e315640 10 monclient: get_auth_request con 0x7f04ec0362a0 auth_method 0 2024-09-16T14:50:45.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.916+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0500014600 con 0x7f050813b2b0 2024-09-16T14:50:45.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.916+0000 7f04feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:45.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:45.916+0000 7f04feffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:46.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.064+0000 7f0510579640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T14:50:46.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.064+0000 7f0510579640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T14:50:46.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.064+0000 7f0510579640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:46.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.064+0000 7f0510579640 1 -- 172.21.15.189:0/3207459205 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h25"} v 0) v1 -- 0x7f04f0005300 con 0x7f050813b2b0 2024-09-16T14:50:46.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.676+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h25"}]=0 removed item id -91 name 'h25' from crush map v240) v1 ==== 120+0+0 (unknown 1141000189 0 0) 0x7f0500014420 con 0x7f050813b2b0 2024-09-16T14:50:46.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.676+0000 7f04feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T14:50:46.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.676+0000 7f04feffd640 10 monclient: _finish_command 2 = system:0 removed item id -91 name 'h25' from crush map 2024-09-16T14:50:46.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.776+0000 7f0510579640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T14:50:46.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.776+0000 7f0510579640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T14:50:46.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.776+0000 7f0510579640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:46.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.776+0000 7f0510579640 1 -- 172.21.15.189:0/3207459205 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h25"} v 0) v1 -- 0x7f04f0005a40 con 0x7f050813b2b0 2024-09-16T14:50:46.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.776+0000 7f04feffd640 1 -- 172.21.15.189:0/3207459205 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h25"}]=0 device 'h25' does not appear in the crush map v240) v1 ==== 120+0+0 (unknown 3242825715 0 0) 0x7f0500014420 con 0x7f050813b2b0 2024-09-16T14:50:46.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.776+0000 7f04feffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T14:50:46.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.776+0000 7f04feffd640 10 monclient: _finish_command 3 = system:0 device 'h25' does not appear in the crush map 2024-09-16T14:50:46.778 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h25' does not appear in the crush map 2024-09-16T14:50:46.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.780+0000 7f0510579640 1 -- 172.21.15.189:0/3207459205 >> v1:172.21.15.189:6812/3196161582 conn(0x7f04ec0362a0 legacy=0x7f04ec038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:46.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.780+0000 7f0510579640 10 monclient: shutdown 2024-09-16T14:50:46.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.780+0000 7f0510579640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:46.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.780+0000 7f0510579640 1 -- 172.21.15.189:0/3207459205 >> v1:172.21.15.26:6790/0 conn(0x7f050813b2b0 legacy=0x7f05081d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:46.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.780+0000 7f0510579640 1 -- 172.21.15.189:0/3207459205 shutdown_connections 2024-09-16T14:50:46.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.780+0000 7f0510579640 1 -- 172.21.15.189:0/3207459205 >> 172.21.15.189:0/3207459205 conn(0x7f05080a3320 msgr2=0x7f05080a1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:46.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.780+0000 7f0510579640 1 -- 172.21.15.189:0/3207459205 shutdown_connections 2024-09-16T14:50:46.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.780+0000 7f0510579640 1 -- 172.21.15.189:0/3207459205 wait complete. 2024-09-16T14:50:46.795 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:46.795 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h26 2024-09-16T14:50:46.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.908+0000 7f34755a5640 10 monclient: get_monmap_and_config 2024-09-16T14:50:46.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.908+0000 7f34755a5640 10 monclient: build_initial_monmap 2024-09-16T14:50:46.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.908+0000 7f34755a5640 10 monclient: monmap: 2024-09-16T14:50:46.911 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:46.911 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:46.911 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:46.912361+0000 2024-09-16T14:50:46.911 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:46.912361+0000 2024-09-16T14:50:46.911 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:46.911 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:46.911 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:46.911 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:46.911 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:46.911 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:46.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.908+0000 7f34755a5640 1 Processor -- start 2024-09-16T14:50:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.908+0000 7f34755a5640 1 -- start start 2024-09-16T14:50:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.908+0000 7f34755a5640 10 monclient: init 2024-09-16T14:50:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.908+0000 7f34755a5640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.908+0000 7f34755a5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.908+0000 7f34755a5640 10 monclient(hunting): picked mon.noname-b con 0x7f34701315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.908+0000 7f34755a5640 10 monclient(hunting): picked mon.noname-a con 0x7f347009b320 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.908+0000 7f34755a5640 10 monclient(hunting): picked mon.noname-c con 0x7f3470132090 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.908+0000 7f34755a5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347009edd0 con 0x7f347009b320 2024-09-16T14:50:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.908+0000 7f34755a5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347009bc00 con 0x7f3470132090 2024-09-16T14:50:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.908+0000 7f34755a5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347009e900 con 0x7f34701315a0 2024-09-16T14:50:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.908+0000 7f34755a5640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.908+0000 7f34755a5640 10 monclient(hunting): authenticate will time out at 1429.027938s 2024-09-16T14:50:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346f7fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f3470132090 legacy=0x7f347013a200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346effd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f34701315a0 0x7f3470131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56860/0 (socket says 172.21.15.189:56860) 2024-09-16T14:50:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346effd640 1 -- 172.21.15.189:0/1422992138 learned_addr learned my addr 172.21.15.189:0/1422992138 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346e7fc640 1 --1- 172.21.15.189:0/1422992138 >> v1:172.21.15.26:6789/0 conn(0x7f347009b320 0x7f3470136ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47212/0 (socket says 172.21.15.189:47212) 2024-09-16T14:50:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 1 -- 172.21.15.189:0/1422992138 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845129255 0 0) 0x7f347009e900 con 0x7f34701315a0 2024-09-16T14:50:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient(hunting): my global_id is 154766 2024-09-16T14:50:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 1 -- 172.21.15.189:0/1422992138 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f345c003530 con 0x7f34701315a0 2024-09-16T14:50:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 1 -- 172.21.15.189:0/1422992138 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691350227 0 0) 0x7f347009edd0 con 0x7f347009b320 2024-09-16T14:50:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient(hunting): my global_id is 164511 2024-09-16T14:50:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 1 -- 172.21.15.189:0/1422992138 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f347009e900 con 0x7f347009b320 2024-09-16T14:50:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 1 -- 172.21.15.189:0/1422992138 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3390057200 0 0) 0x7f345c003530 con 0x7f34701315a0 2024-09-16T14:50:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 1 -- 172.21.15.189:0/1422992138 >> v1:172.21.15.189:6789/0 conn(0x7f3470132090 legacy=0x7f347013a200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 1 -- 172.21.15.189:0/1422992138 >> v1:172.21.15.26:6789/0 conn(0x7f347009b320 legacy=0x7f3470136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:50:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 1 monclient: found mon.noname-b 2024-09-16T14:50:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 1 -- 172.21.15.189:0/1422992138 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347009b920 con 0x7f34701315a0 2024-09-16T14:50:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient: _finish_auth 0 2024-09-16T14:50:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 1 -- 172.21.15.189:0/1422992138 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3460001620 con 0x7f34701315a0 2024-09-16T14:50:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient: dump: 2024-09-16T14:50:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:46.919 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f34755a5640 5 monclient: authenticate success, global_id 154766 2024-09-16T14:50:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f34755a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 1 -- 172.21.15.189:0/1422992138 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3460001900 con 0x7f34701315a0 2024-09-16T14:50:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 1 -- 172.21.15.189:0/1422992138 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3460001c90 con 0x7f34701315a0 2024-09-16T14:50:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f346dffb640 10 monclient: dump: 2024-09-16T14:50:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:46.920 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f34755a5640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f34755a5640 4 set_mon_vals no callback set 2024-09-16T14:50:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f34755a5640 10 monclient: shutdown 2024-09-16T14:50:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f34755a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f34755a5640 1 -- 172.21.15.189:0/1422992138 >> v1:172.21.15.26:6790/0 conn(0x7f34701315a0 legacy=0x7f3470131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f34755a5640 1 -- 172.21.15.189:0/1422992138 shutdown_connections 2024-09-16T14:50:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f34755a5640 1 -- 172.21.15.189:0/1422992138 >> 172.21.15.189:0/1422992138 conn(0x7f34700a3320 msgr2=0x7f347009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f34755a5640 1 -- 172.21.15.189:0/1422992138 shutdown_connections 2024-09-16T14:50:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f34755a5640 1 -- 172.21.15.189:0/1422992138 wait complete. 2024-09-16T14:50:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f34755a5640 10 monclient: build_initial_monmap 2024-09-16T14:50:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f34755a5640 10 monclient: monmap: 2024-09-16T14:50:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:46.922 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:46.922 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:46.922 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:46.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f34755a5640 1 Processor -- start 2024-09-16T14:50:46.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f34755a5640 1 -- start start 2024-09-16T14:50:46.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.912+0000 7f34755a5640 10 monclient: init 2024-09-16T14:50:46.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:46.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:46.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 10 monclient(hunting): picked mon.noname-c con 0x7f34701315a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:46.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 10 monclient(hunting): picked mon.noname-a con 0x7f3470132090 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:46.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 10 monclient(hunting): picked mon.noname-b con 0x7f347009d6b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:46.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347009cd20 con 0x7f3470132090 2024-09-16T14:50:46.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347011a6a0 con 0x7f347009d6b0 2024-09-16T14:50:46.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347006b380 con 0x7f34701315a0 2024-09-16T14:50:46.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:46.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 10 monclient(hunting): authenticate will time out at 1429.032318s 2024-09-16T14:50:46.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f346f7fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f347009d6b0 legacy=0x7f34701d1a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:46.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f346effd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f34701315a0 0x7f347012fc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56870/0 (socket says 172.21.15.189:56870) 2024-09-16T14:50:46.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f346effd640 1 -- 172.21.15.189:0/817884733 learned_addr learned my addr 172.21.15.189:0/817884733 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:46.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415793262 0 0) 0x7f347006b380 con 0x7f34701315a0 2024-09-16T14:50:46.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:46.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:46.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:46.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient(hunting): my global_id is 154769 2024-09-16T14:50:46.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3450003530 con 0x7f34701315a0 2024-09-16T14:50:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235321546 0 0) 0x7f347009cd20 con 0x7f3470132090 2024-09-16T14:50:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient(hunting): my global_id is 164514 2024-09-16T14:50:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f347006b380 con 0x7f3470132090 2024-09-16T14:50:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3501560448 0 0) 0x7f3450003530 con 0x7f34701315a0 2024-09-16T14:50:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f347009cd20 con 0x7f34701315a0 2024-09-16T14:50:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34600015e0 con 0x7f34701315a0 2024-09-16T14:50:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient(hunting): dump: 2024-09-16T14:50:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:46.929 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:46.929 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:46.929 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:46.929 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:46.929 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:46.929 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:46.929 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:46.929 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:46.929 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:46.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:46.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3755788436 0 0) 0x7f347006b380 con 0x7f3470132090 2024-09-16T14:50:46.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3450003530 con 0x7f3470132090 2024-09-16T14:50:46.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3458001840 con 0x7f3470132090 2024-09-16T14:50:46.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient(hunting): dump: 2024-09-16T14:50:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:46.930 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 651572352 0 0) 0x7f347009cd20 con 0x7f34701315a0 2024-09-16T14:50:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 >> v1:172.21.15.189:6789/0 conn(0x7f347009d6b0 legacy=0x7f34701d1a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 >> v1:172.21.15.26:6789/0 conn(0x7f3470132090 legacy=0x7f34701ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 monclient: found mon.c 2024-09-16T14:50:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34700a2120 con 0x7f34701315a0 2024-09-16T14:50:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 5 monclient: authenticate success, global_id 154769 2024-09-16T14:50:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 10 monclient: _renew_subs 2024-09-16T14:50:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 1 -- 172.21.15.189:0/817884733 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34700a1bf0 con 0x7f34701315a0 2024-09-16T14:50:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 10 monclient: _renew_subs 2024-09-16T14:50:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 1 -- 172.21.15.189:0/817884733 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3470120650 con 0x7f34701315a0 2024-09-16T14:50:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3460004c00 con 0x7f34701315a0 2024-09-16T14:50:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3460004f90 con 0x7f34701315a0 2024-09-16T14:50:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f346d7fa640 4 set_mon_vals no callback set 2024-09-16T14:50:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 10 monclient: dump: 2024-09-16T14:50:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:46.933 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34755a5640 1 -- 172.21.15.189:0/817884733 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3454005050 con 0x7f34701315a0 2024-09-16T14:50:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3460005170 con 0x7f34701315a0 2024-09-16T14:50:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(240..240 src has 169..240) v4 ==== 8774+0+0 (unknown 3205113257 0 0) 0x7f34600448c0 con 0x7f34701315a0 2024-09-16T14:50:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.916+0000 7f346e7fc640 10 monclient: get_auth_request con 0x7f3450036220 auth_method 0 2024-09-16T14:50:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.928+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3460014b30 con 0x7f34701315a0 2024-09-16T14:50:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.932+0000 7f34677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:46.932+0000 7f34677fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:47.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.080+0000 7f34755a5640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T14:50:47.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.080+0000 7f34755a5640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T14:50:47.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.080+0000 7f34755a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:47.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.080+0000 7f34755a5640 1 -- 172.21.15.189:0/817884733 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h26"} v 0) v1 -- 0x7f3454005300 con 0x7f34701315a0 2024-09-16T14:50:47.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.684+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h26"}]=0 removed item id -94 name 'h26' from crush map v241) v1 ==== 120+0+0 (unknown 3591655590 0 0) 0x7f3460014720 con 0x7f34701315a0 2024-09-16T14:50:47.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.684+0000 7f34677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T14:50:47.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.684+0000 7f34677fe640 10 monclient: _finish_command 2 = system:0 removed item id -94 name 'h26' from crush map 2024-09-16T14:50:47.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.780+0000 7f34755a5640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T14:50:47.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.780+0000 7f34755a5640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T14:50:47.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.780+0000 7f34755a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:47.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.780+0000 7f34755a5640 1 -- 172.21.15.189:0/817884733 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h26"} v 0) v1 -- 0x7f3454005a40 con 0x7f34701315a0 2024-09-16T14:50:47.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.780+0000 7f34677fe640 1 -- 172.21.15.189:0/817884733 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h26"}]=0 device 'h26' does not appear in the crush map v241) v1 ==== 120+0+0 (unknown 43735511 0 0) 0x7f3460014900 con 0x7f34701315a0 2024-09-16T14:50:47.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.780+0000 7f34677fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T14:50:47.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.780+0000 7f34677fe640 10 monclient: _finish_command 3 = system:0 device 'h26' does not appear in the crush map 2024-09-16T14:50:47.784 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h26' does not appear in the crush map 2024-09-16T14:50:47.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.784+0000 7f34755a5640 1 -- 172.21.15.189:0/817884733 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3450036220 legacy=0x7f34500386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:47.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.784+0000 7f34755a5640 10 monclient: shutdown 2024-09-16T14:50:47.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.784+0000 7f34755a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:47.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.784+0000 7f34755a5640 1 -- 172.21.15.189:0/817884733 >> v1:172.21.15.26:6790/0 conn(0x7f34701315a0 legacy=0x7f347012fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:47.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.784+0000 7f34755a5640 1 -- 172.21.15.189:0/817884733 shutdown_connections 2024-09-16T14:50:47.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.784+0000 7f34755a5640 1 -- 172.21.15.189:0/817884733 >> 172.21.15.189:0/817884733 conn(0x7f34700a3320 msgr2=0x7f347009a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:47.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.784+0000 7f34755a5640 1 -- 172.21.15.189:0/817884733 shutdown_connections 2024-09-16T14:50:47.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.784+0000 7f34755a5640 1 -- 172.21.15.189:0/817884733 wait complete. 2024-09-16T14:50:47.801 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:47.801 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h27 2024-09-16T14:50:47.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3002709640 10 monclient: get_monmap_and_config 2024-09-16T14:50:47.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3002709640 10 monclient: build_initial_monmap 2024-09-16T14:50:47.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3002709640 10 monclient: monmap: 2024-09-16T14:50:47.917 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:47.917 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:47.917 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:47.918802+0000 2024-09-16T14:50:47.917 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:47.918802+0000 2024-09-16T14:50:47.917 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:47.918 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:47.918 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:47.918 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:47.918 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:47.918 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:47.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3002709640 1 Processor -- start 2024-09-16T14:50:47.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3002709640 1 -- start start 2024-09-16T14:50:47.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3002709640 10 monclient: init 2024-09-16T14:50:47.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3002709640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:47.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3002709640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:47.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3002709640 10 monclient(hunting): picked mon.noname-b con 0x7f2ffc09df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:47.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3002709640 10 monclient(hunting): picked mon.noname-a con 0x7f2ffc09e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:47.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3002709640 10 monclient(hunting): picked mon.noname-c con 0x7f2ffc09e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:47.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3002709640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ffc09ee20 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3002709640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ffc09f100 con 0x7f2ffc09e740 2024-09-16T14:50:47.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3002709640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ffc09f3e0 con 0x7f2ffc09df10 2024-09-16T14:50:47.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3002709640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:47.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3002709640 10 monclient(hunting): authenticate will time out at 1430.035282s 2024-09-16T14:50:47.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f3000ca6640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2ffc09e740 legacy=0x7f2ffc13b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:47.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f2ffbfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f2ffc09df10 0x7f2ffc09d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56880/0 (socket says 172.21.15.189:56880) 2024-09-16T14:50:47.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.916+0000 7f2ffbfff640 1 -- 172.21.15.189:0/3168487481 learned_addr learned my addr 172.21.15.189:0/3168487481 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:47.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 1 -- 172.21.15.189:0/3168487481 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069035435 0 0) 0x7f2ffc09f3e0 con 0x7f2ffc09df10 2024-09-16T14:50:47.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:47.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:47.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:47.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient(hunting): my global_id is 154772 2024-09-16T14:50:47.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 1 -- 172.21.15.189:0/3168487481 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fe4003530 con 0x7f2ffc09df10 2024-09-16T14:50:47.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 1 -- 172.21.15.189:0/3168487481 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081307774 0 0) 0x7f2ffc09ee20 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:47.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:47.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:47.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient(hunting): my global_id is 164517 2024-09-16T14:50:47.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 1 -- 172.21.15.189:0/3168487481 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ffc09f3e0 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 1 -- 172.21.15.189:0/3168487481 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2872848666 0 0) 0x7f2fe4003530 con 0x7f2ffc09df10 2024-09-16T14:50:47.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 1 -- 172.21.15.189:0/3168487481 >> v1:172.21.15.189:6789/0 conn(0x7f2ffc09e740 legacy=0x7f2ffc13b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:47.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 1 -- 172.21.15.189:0/3168487481 >> v1:172.21.15.26:6789/0 conn(0x7f2ffc09e2f0 legacy=0x7f2ffc1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:47.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:47.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 1 monclient: found mon.noname-b 2024-09-16T14:50:47.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:47.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:47.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 1 -- 172.21.15.189:0/3168487481 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc09c690 con 0x7f2ffc09df10 2024-09-16T14:50:47.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:47.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:47.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:47.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 1 -- 172.21.15.189:0/3168487481 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2fec001550 con 0x7f2ffc09df10 2024-09-16T14:50:47.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:47.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:47.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient: dump: 2024-09-16T14:50:47.926 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:47.926 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:47.926 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:47.926 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:47.927 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:47.927 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:47.927 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:47.927 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:47.927 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:47.927 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:47.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 5 monclient: authenticate success, global_id 154772 2024-09-16T14:50:47.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:47.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 1 -- 172.21.15.189:0/3168487481 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fec001770 con 0x7f2ffc09df10 2024-09-16T14:50:47.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:47.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 1 -- 172.21.15.189:0/3168487481 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2fec001b00 con 0x7f2ffc09df10 2024-09-16T14:50:47.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:47.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:47.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffaffd640 10 monclient: dump: 2024-09-16T14:50:47.928 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:47.928 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:47.928 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:47.928 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:47.928 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:47.928 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:47.928 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:47.928 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:47.928 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:47.928 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:47.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:47.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 4 set_mon_vals no callback set 2024-09-16T14:50:47.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 10 monclient: shutdown 2024-09-16T14:50:47.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:47.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 1 -- 172.21.15.189:0/3168487481 >> v1:172.21.15.26:6790/0 conn(0x7f2ffc09df10 legacy=0x7f2ffc09d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:47.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 1 -- 172.21.15.189:0/3168487481 shutdown_connections 2024-09-16T14:50:47.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 1 -- 172.21.15.189:0/3168487481 >> 172.21.15.189:0/3168487481 conn(0x7f2ffc0a3320 msgr2=0x7f2ffc09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:47.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 1 -- 172.21.15.189:0/3168487481 shutdown_connections 2024-09-16T14:50:47.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 1 -- 172.21.15.189:0/3168487481 wait complete. 2024-09-16T14:50:47.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 10 monclient: build_initial_monmap 2024-09-16T14:50:47.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 10 monclient: monmap: 2024-09-16T14:50:47.929 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:47.929 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:47.929 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:47.929 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:47.929 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:47.930 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:47.930 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:47.930 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:47.930 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:47.930 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:47.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 1 Processor -- start 2024-09-16T14:50:47.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 1 -- start start 2024-09-16T14:50:47.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 10 monclient: init 2024-09-16T14:50:47.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:47.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:47.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 10 monclient(hunting): picked mon.noname-b con 0x7f2ffc09df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:47.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 10 monclient(hunting): picked mon.noname-c con 0x7f2ffc09e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:47.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 10 monclient(hunting): picked mon.noname-a con 0x7f2ffc09e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:47.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ffc11a420 con 0x7f2ffc09e740 2024-09-16T14:50:47.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ffc06fa60 con 0x7f2ffc09df10 2024-09-16T14:50:47.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffbfff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2ffc09df10 legacy=0x7f2ffc139ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:47.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ffc13c4c0 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:47.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3002709640 10 monclient(hunting): authenticate will time out at 1430.039304s 2024-09-16T14:50:47.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3000ca6640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2ffc09e740 0x7f2ffc13a900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:47236/0 (socket says 172.21.15.189:47236) 2024-09-16T14:50:47.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f2ffb7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f2ffc09e2f0 0x7f2ffc13a1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:56892/0 (socket says 172.21.15.189:56892) 2024-09-16T14:50:47.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.920+0000 7f3000ca6640 1 -- 172.21.15.189:0/3509431014 learned_addr learned my addr 172.21.15.189:0/3509431014 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:47.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3830845643 0 0) 0x7f2ffc13c4c0 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:47.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:47.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:47.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient(hunting): my global_id is 154775 2024-09-16T14:50:47.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fe8003530 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772292421 0 0) 0x7f2ffc11a420 con 0x7f2ffc09e740 2024-09-16T14:50:47.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:47.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:47.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:47.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient(hunting): my global_id is 164520 2024-09-16T14:50:47.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ffc13c4c0 con 0x7f2ffc09e740 2024-09-16T14:50:47.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1776450396 0 0) 0x7f2fe8003530 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ffc11a420 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2ff00018c0 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:47.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:47.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient(hunting): dump: 2024-09-16T14:50:47.936 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:47.936 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:47.936 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:47.936 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:47.936 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:47.936 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:47.937 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:47.937 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:47.937 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:47.937 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:47.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:47.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3327018329 0 0) 0x7f2ffc13c4c0 con 0x7f2ffc09e740 2024-09-16T14:50:47.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fe8003530 con 0x7f2ffc09e740 2024-09-16T14:50:47.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2ff40043c0 con 0x7f2ffc09e740 2024-09-16T14:50:47.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:47.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:47.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient(hunting): dump: 2024-09-16T14:50:47.937 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:47.937 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:47.938 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:47.938 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:47.938 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:47.938 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:47.938 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:47.938 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:47.938 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:47.938 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:47.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2548060995 0 0) 0x7f2ffc11a420 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 >> v1:172.21.15.189:6789/0 conn(0x7f2ffc09df10 legacy=0x7f2ffc139ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:47.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 >> v1:172.21.15.26:6789/0 conn(0x7f2ffc09e740 legacy=0x7f2ffc13a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:47.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:50:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 monclient: found mon.c 2024-09-16T14:50:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc06cf30 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient: _finish_auth 0 2024-09-16T14:50:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:50:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f3002709640 5 monclient: authenticate success, global_id 154775 2024-09-16T14:50:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f3002709640 10 monclient: _renew_subs 2024-09-16T14:50:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f3002709640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f3002709640 1 -- 172.21.15.189:0/3509431014 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ffc11a700 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f3002709640 10 monclient: _renew_subs 2024-09-16T14:50:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f3002709640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:47.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f3002709640 1 -- 172.21.15.189:0/3509431014 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ffc13c180 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ff0003600 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:47.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2ff0003910 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:47.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:47.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 10 monclient: dump: 2024-09-16T14:50:47.940 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:47.940 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:47.940 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:47.940 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:47.940 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:47.941 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:47.941 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:47.941 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:47.941 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:47.941 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:47.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ffa7fc640 4 set_mon_vals no callback set 2024-09-16T14:50:47.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f3002709640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:47.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f3002709640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:47.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f3002709640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:47.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f3002709640 1 -- 172.21.15.189:0/3509431014 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fdc005050 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2ff0003af0 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(241..241 src has 169..241) v4 ==== 8561+0+0 (unknown 4265835762 0 0) 0x7f2ff0044480 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.924+0000 7f2ffbfff640 10 monclient: get_auth_request con 0x7f2fe8036270 auth_method 0 2024-09-16T14:50:47.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.936+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2ff0014490 con 0x7f2ffc09e2f0 2024-09-16T14:50:47.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.936+0000 7f2ff8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:47.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:47.936+0000 7f2ff8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:48.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.104+0000 7f3002709640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T14:50:48.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.104+0000 7f3002709640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T14:50:48.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.104+0000 7f3002709640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:48.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.104+0000 7f3002709640 1 -- 172.21.15.189:0/3509431014 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h27"} v 0) v1 -- 0x7f2fdc005300 con 0x7f2ffc09e2f0 2024-09-16T14:50:48.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.692+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h27"}]=0 removed item id -97 name 'h27' from crush map v242) v1 ==== 120+0+0 (unknown 1223094776 0 0) 0x7f2ff0012b30 con 0x7f2ffc09e2f0 2024-09-16T14:50:48.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.692+0000 7f2ff8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T14:50:48.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.692+0000 7f2ff8ff9640 10 monclient: _finish_command 2 = system:0 removed item id -97 name 'h27' from crush map 2024-09-16T14:50:48.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.796+0000 7f3002709640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T14:50:48.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.796+0000 7f3002709640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T14:50:48.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.796+0000 7f3002709640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:48.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.796+0000 7f3002709640 1 -- 172.21.15.189:0/3509431014 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h27"} v 0) v1 -- 0x7f2fdc0055a0 con 0x7f2ffc09e2f0 2024-09-16T14:50:48.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.796+0000 7f2ff8ff9640 1 -- 172.21.15.189:0/3509431014 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h27"}]=0 device 'h27' does not appear in the crush map v242) v1 ==== 120+0+0 (unknown 2899685468 0 0) 0x7f2ff00142b0 con 0x7f2ffc09e2f0 2024-09-16T14:50:48.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.796+0000 7f2ff8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T14:50:48.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.796+0000 7f2ff8ff9640 10 monclient: _finish_command 3 = system:0 device 'h27' does not appear in the crush map 2024-09-16T14:50:48.797 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h27' does not appear in the crush map 2024-09-16T14:50:48.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.796+0000 7f3002709640 1 -- 172.21.15.189:0/3509431014 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2fe8036270 legacy=0x7f2fe8038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:48.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.796+0000 7f3002709640 10 monclient: shutdown 2024-09-16T14:50:48.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.796+0000 7f3002709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:48.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.796+0000 7f3002709640 1 -- 172.21.15.189:0/3509431014 >> v1:172.21.15.26:6790/0 conn(0x7f2ffc09e2f0 legacy=0x7f2ffc13a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:48.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.796+0000 7f3002709640 1 -- 172.21.15.189:0/3509431014 shutdown_connections 2024-09-16T14:50:48.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.796+0000 7f3002709640 1 -- 172.21.15.189:0/3509431014 >> 172.21.15.189:0/3509431014 conn(0x7f2ffc0a3320 msgr2=0x7f2ffc06e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:48.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.800+0000 7f3002709640 1 -- 172.21.15.189:0/3509431014 shutdown_connections 2024-09-16T14:50:48.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.800+0000 7f3002709640 1 -- 172.21.15.189:0/3509431014 wait complete. 2024-09-16T14:50:48.814 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:48.815 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h28 2024-09-16T14:50:48.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.928+0000 7f1e18c65640 10 monclient: get_monmap_and_config 2024-09-16T14:50:48.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.928+0000 7f1e18c65640 10 monclient: build_initial_monmap 2024-09-16T14:50:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.928+0000 7f1e18c65640 10 monclient: monmap: 2024-09-16T14:50:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:48.932340+0000 2024-09-16T14:50:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:48.932340+0000 2024-09-16T14:50:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:48.931 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:48.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.928+0000 7f1e18c65640 1 Processor -- start 2024-09-16T14:50:48.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.928+0000 7f1e18c65640 1 -- start start 2024-09-16T14:50:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.928+0000 7f1e18c65640 10 monclient: init 2024-09-16T14:50:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.928+0000 7f1e18c65640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.928+0000 7f1e18c65640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:50:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.928+0000 7f1e18c65640 10 monclient(hunting): picked mon.noname-a con 0x7f1e141318a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.928+0000 7f1e18c65640 10 monclient(hunting): picked mon.noname-c con 0x7f1e14134d50 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.928+0000 7f1e18c65640 10 monclient(hunting): picked mon.noname-b con 0x7f1e1413b2b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.928+0000 7f1e18c65640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1409d8a0 con 0x7f1e141318a0 2024-09-16T14:50:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.928+0000 7f1e18c65640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1409ddf0 con 0x7f1e14134d50 2024-09-16T14:50:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.928+0000 7f1e18c65640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1409e8b0 con 0x7f1e1413b2b0 2024-09-16T14:50:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.928+0000 7f1e18c65640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.928+0000 7f1e18c65640 10 monclient(hunting): authenticate will time out at 1431.047986s 2024-09-16T14:50:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e11d9b640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f1e14134d50 legacy=0x7f1e14139b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1259c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1e141318a0 0x7f1e14131c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:44536/0 (socket says 172.21.15.189:44536) 2024-09-16T14:50:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1259c640 1 -- 172.21.15.189:0/3467730274 learned_addr learned my addr 172.21.15.189:0/3467730274 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 1 -- 172.21.15.189:0/3467730274 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713658924 0 0) 0x7f1e1409d8a0 con 0x7f1e141318a0 2024-09-16T14:50:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient(hunting): my global_id is 164523 2024-09-16T14:50:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 1 -- 172.21.15.189:0/3467730274 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dfc003530 con 0x7f1e141318a0 2024-09-16T14:50:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 1 -- 172.21.15.189:0/3467730274 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825610650 0 0) 0x7f1e1409e8b0 con 0x7f1e1413b2b0 2024-09-16T14:50:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient(hunting): my global_id is 154778 2024-09-16T14:50:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 1 -- 172.21.15.189:0/3467730274 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e1409d8a0 con 0x7f1e1413b2b0 2024-09-16T14:50:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 1 -- 172.21.15.189:0/3467730274 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1971653107 0 0) 0x7f1dfc003530 con 0x7f1e141318a0 2024-09-16T14:50:48.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 1 -- 172.21.15.189:0/3467730274 >> v1:172.21.15.26:6790/0 conn(0x7f1e1413b2b0 legacy=0x7f1e1413d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:48.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 1 -- 172.21.15.189:0/3467730274 >> v1:172.21.15.189:6789/0 conn(0x7f1e14134d50 legacy=0x7f1e14139b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:48.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient: _finish_hunting 0 2024-09-16T14:50:48.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 1 monclient: found mon.noname-a 2024-09-16T14:50:48.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:48.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:48.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 1 -- 172.21.15.189:0/3467730274 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1409eb90 con 0x7f1e141318a0 2024-09-16T14:50:48.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient: _finish_auth 0 2024-09-16T14:50:48.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient: _check_auth_tickets 2024-09-16T14:50:48.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:48.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 1 -- 172.21.15.189:0/3467730274 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1e08001620 con 0x7f1e141318a0 2024-09-16T14:50:48.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:48.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:48.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient: dump: 2024-09-16T14:50:48.937 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:48.937 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:48.937 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:48.937 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:48.937 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:48.937 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:48.937 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:48.937 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:48.937 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:48.937 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:48.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e18c65640 5 monclient: authenticate success, global_id 164523 2024-09-16T14:50:48.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e18c65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:48.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 1 -- 172.21.15.189:0/3467730274 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e08001890 con 0x7f1e141318a0 2024-09-16T14:50:48.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:48.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e18c65640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:48.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e18c65640 4 set_mon_vals no callback set 2024-09-16T14:50:48.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 1 -- 172.21.15.189:0/3467730274 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1e08001c20 con 0x7f1e141318a0 2024-09-16T14:50:48.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:48.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:48.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e1159a640 10 monclient: dump: 2024-09-16T14:50:48.938 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:48.938 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:48.938 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:48.938 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:48.938 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:48.938 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:48.939 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:48.939 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:48.939 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:48.939 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e18c65640 10 monclient: shutdown 2024-09-16T14:50:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e18c65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e18c65640 1 -- 172.21.15.189:0/3467730274 >> v1:172.21.15.26:6789/0 conn(0x7f1e141318a0 legacy=0x7f1e14131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e18c65640 1 -- 172.21.15.189:0/3467730274 shutdown_connections 2024-09-16T14:50:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e18c65640 1 -- 172.21.15.189:0/3467730274 >> 172.21.15.189:0/3467730274 conn(0x7f1e140a3320 msgr2=0x7f1e1409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e18c65640 1 -- 172.21.15.189:0/3467730274 shutdown_connections 2024-09-16T14:50:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e18c65640 1 -- 172.21.15.189:0/3467730274 wait complete. 2024-09-16T14:50:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e18c65640 10 monclient: build_initial_monmap 2024-09-16T14:50:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e18c65640 10 monclient: monmap: 2024-09-16T14:50:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:48.944 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e18c65640 1 Processor -- start 2024-09-16T14:50:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e18c65640 1 -- start start 2024-09-16T14:50:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.932+0000 7f1e18c65640 10 monclient: init 2024-09-16T14:50:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 10 monclient(hunting): picked mon.noname-c con 0x7f1e141318a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 10 monclient(hunting): picked mon.noname-a con 0x7f1e14134d50 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 10 monclient(hunting): picked mon.noname-b con 0x7f1e1413b2b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1406b380 con 0x7f1e14134d50 2024-09-16T14:50:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1413abd0 con 0x7f1e1413b2b0 2024-09-16T14:50:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e141194e0 con 0x7f1e141318a0 2024-09-16T14:50:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 10 monclient(hunting): authenticate will time out at 1431.052232s 2024-09-16T14:50:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e12d9d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f1e1413b2b0 legacy=0x7f1e141d1480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e1259c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1e141318a0 0x7f1e14139140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:55314/0 (socket says 172.21.15.189:55314) 2024-09-16T14:50:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e11d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1e14134d50 0x7f1e14139850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:44542/0 (socket says 172.21.15.189:44542) 2024-09-16T14:50:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e1259c640 1 -- 172.21.15.189:0/987907366 learned_addr learned my addr 172.21.15.189:0/987907366 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026168974 0 0) 0x7f1e141194e0 con 0x7f1e141318a0 2024-09-16T14:50:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient(hunting): my global_id is 154781 2024-09-16T14:50:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1df4003530 con 0x7f1e141318a0 2024-09-16T14:50:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40708421 0 0) 0x7f1e1406b380 con 0x7f1e14134d50 2024-09-16T14:50:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:48.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient(hunting): my global_id is 164526 2024-09-16T14:50:48.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e141194e0 con 0x7f1e14134d50 2024-09-16T14:50:48.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 519406731 0 0) 0x7f1df4003530 con 0x7f1e141318a0 2024-09-16T14:50:48.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e1406b380 con 0x7f1e141318a0 2024-09-16T14:50:48.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3426626229 0 0) 0x7f1e141194e0 con 0x7f1e14134d50 2024-09-16T14:50:48.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1df4003530 con 0x7f1e14134d50 2024-09-16T14:50:48.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1e080016a0 con 0x7f1e141318a0 2024-09-16T14:50:48.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:48.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:48.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:48.947 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:48.947 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:48.947 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:48.947 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:48.948 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:48.948 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:48.948 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:48.948 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:48.948 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:48.948 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:48.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:48.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1e04004050 con 0x7f1e14134d50 2024-09-16T14:50:48.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:48.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:48.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:48.948 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:48.948 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:48.949 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:48.949 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:48.949 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:48.949 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:48.949 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:48.949 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:48.949 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:48.949 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:48.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1497489599 0 0) 0x7f1e1406b380 con 0x7f1e141318a0 2024-09-16T14:50:48.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 >> v1:172.21.15.189:6789/0 conn(0x7f1e1413b2b0 legacy=0x7f1e141d1480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:48.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 >> v1:172.21.15.26:6789/0 conn(0x7f1e14134d50 legacy=0x7f1e14139850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:48.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:48.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 monclient: found mon.c 2024-09-16T14:50:48.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:48.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:48.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1411b840 con 0x7f1e141318a0 2024-09-16T14:50:48.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:48.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:48.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:48.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 5 monclient: authenticate success, global_id 154781 2024-09-16T14:50:48.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 10 monclient: _renew_subs 2024-09-16T14:50:48.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:48.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 1 -- 172.21.15.189:0/987907366 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e14119e30 con 0x7f1e141318a0 2024-09-16T14:50:48.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 10 monclient: _renew_subs 2024-09-16T14:50:48.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:48.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 1 -- 172.21.15.189:0/987907366 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e140a1bf0 con 0x7f1e141318a0 2024-09-16T14:50:48.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e080011c0 con 0x7f1e141318a0 2024-09-16T14:50:48.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:48.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1e08004d50 con 0x7f1e141318a0 2024-09-16T14:50:48.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:48.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:48.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 10 monclient: dump: 2024-09-16T14:50:48.951 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:48.951 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:48.951 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:48.951 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:48.951 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:48.951 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:48.951 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:48.951 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:48.951 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:48.952 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:48.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e03fff640 4 set_mon_vals no callback set 2024-09-16T14:50:48.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:48.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:48.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:48.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e18c65640 1 -- 172.21.15.189:0/987907366 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df8005050 con 0x7f1e141318a0 2024-09-16T14:50:48.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1e08004f30 con 0x7f1e141318a0 2024-09-16T14:50:48.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(242..242 src has 169..242) v4 ==== 8348+0+0 (unknown 4294020767 0 0) 0x7f1e080445e0 con 0x7f1e141318a0 2024-09-16T14:50:48.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.936+0000 7f1e11d9b640 10 monclient: get_auth_request con 0x7f1df40362a0 auth_method 0 2024-09-16T14:50:48.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.948+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1e08014760 con 0x7f1e141318a0 2024-09-16T14:50:48.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.948+0000 7f1e02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:48.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:48.948+0000 7f1e02ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:49.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.116+0000 7f1e18c65640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T14:50:49.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.116+0000 7f1e18c65640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T14:50:49.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.116+0000 7f1e18c65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:49.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.116+0000 7f1e18c65640 1 -- 172.21.15.189:0/987907366 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h28"} v 0) v1 -- 0x7f1df8005300 con 0x7f1e141318a0 2024-09-16T14:50:49.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.696+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h28"}]=0 removed item id -100 name 'h28' from crush map v243) v1 ==== 121+0+0 (unknown 2034464365 0 0) 0x7f1e08012e00 con 0x7f1e141318a0 2024-09-16T14:50:49.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.696+0000 7f1e02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T14:50:49.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.696+0000 7f1e02ffd640 10 monclient: _finish_command 2 = system:0 removed item id -100 name 'h28' from crush map 2024-09-16T14:50:49.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.792+0000 7f1e18c65640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T14:50:49.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.792+0000 7f1e18c65640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T14:50:49.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.792+0000 7f1e18c65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:49.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.792+0000 7f1e18c65640 1 -- 172.21.15.189:0/987907366 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h28"} v 0) v1 -- 0x7f1df8005a40 con 0x7f1e141318a0 2024-09-16T14:50:49.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.792+0000 7f1e02ffd640 1 -- 172.21.15.189:0/987907366 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h28"}]=0 device 'h28' does not appear in the crush map v243) v1 ==== 120+0+0 (unknown 1519738803 0 0) 0x7f1e08014580 con 0x7f1e141318a0 2024-09-16T14:50:49.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.792+0000 7f1e02ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T14:50:49.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.792+0000 7f1e02ffd640 10 monclient: _finish_command 3 = system:0 device 'h28' does not appear in the crush map 2024-09-16T14:50:49.793 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h28' does not appear in the crush map 2024-09-16T14:50:49.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.792+0000 7f1e18c65640 1 -- 172.21.15.189:0/987907366 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1df40362a0 legacy=0x7f1df4038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:49.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.792+0000 7f1e18c65640 10 monclient: shutdown 2024-09-16T14:50:49.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.792+0000 7f1e18c65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:49.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.792+0000 7f1e18c65640 1 -- 172.21.15.189:0/987907366 >> v1:172.21.15.26:6790/0 conn(0x7f1e141318a0 legacy=0x7f1e14139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:49.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.792+0000 7f1e18c65640 1 -- 172.21.15.189:0/987907366 shutdown_connections 2024-09-16T14:50:49.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.792+0000 7f1e18c65640 1 -- 172.21.15.189:0/987907366 >> 172.21.15.189:0/987907366 conn(0x7f1e140a3320 msgr2=0x7f1e141332b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:49.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.792+0000 7f1e18c65640 1 -- 172.21.15.189:0/987907366 shutdown_connections 2024-09-16T14:50:49.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.792+0000 7f1e18c65640 1 -- 172.21.15.189:0/987907366 wait complete. 2024-09-16T14:50:49.810 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:49.810 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h29 2024-09-16T14:50:49.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fde04b36640 10 monclient: get_monmap_and_config 2024-09-16T14:50:49.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fde04b36640 10 monclient: build_initial_monmap 2024-09-16T14:50:49.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fde04b36640 10 monclient: monmap: 2024-09-16T14:50:49.930 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:49.930 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:49.930 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:49.931288+0000 2024-09-16T14:50:49.930 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:49.931288+0000 2024-09-16T14:50:49.930 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:49.930 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:49.930 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:49.930 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:49.930 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:49.930 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:49.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fde04b36640 1 Processor -- start 2024-09-16T14:50:49.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fde04b36640 1 -- start start 2024-09-16T14:50:49.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fde04b36640 10 monclient: init 2024-09-16T14:50:49.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fde04b36640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:49.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fde04b36640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:49.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fde04b36640 10 monclient(hunting): picked mon.noname-b con 0x7fde0009df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:49.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fde04b36640 10 monclient(hunting): picked mon.noname-a con 0x7fde0009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:49.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fde04b36640 10 monclient(hunting): picked mon.noname-c con 0x7fde0009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:49.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fde04b36640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0009ee20 con 0x7fde0009e2f0 2024-09-16T14:50:49.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fde04b36640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0009f100 con 0x7fde0009e740 2024-09-16T14:50:49.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fde04b36640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0009f3e0 con 0x7fde0009df10 2024-09-16T14:50:49.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fde04b36640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:49.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fde04b36640 10 monclient(hunting): authenticate will time out at 1432.046937s 2024-09-16T14:50:49.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fddfed9d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fde0009e740 legacy=0x7fde0013b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:49.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fddfe59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fde0009df10 0x7fde0009d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:55320/0 (socket says 172.21.15.189:55320) 2024-09-16T14:50:49.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fddfe59c640 1 -- 172.21.15.189:0/2514023261 learned_addr learned my addr 172.21.15.189:0/2514023261 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:49.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fddfdd9b640 1 --1- 172.21.15.189:0/2514023261 >> v1:172.21.15.26:6789/0 conn(0x7fde0009e2f0 0x7fde001378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:44552/0 (socket says 172.21.15.189:44552) 2024-09-16T14:50:49.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fddfd59a640 1 -- 172.21.15.189:0/2514023261 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999713055 0 0) 0x7fde0009f3e0 con 0x7fde0009df10 2024-09-16T14:50:49.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fddfd59a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:49.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fddfd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:49.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fddfd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:49.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fddfd59a640 10 monclient(hunting): my global_id is 154784 2024-09-16T14:50:49.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.928+0000 7fddfd59a640 1 -- 172.21.15.189:0/2514023261 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddec003530 con 0x7fde0009df10 2024-09-16T14:50:49.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 1 -- 172.21.15.189:0/2514023261 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572574966 0 0) 0x7fde0009ee20 con 0x7fde0009e2f0 2024-09-16T14:50:49.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:49.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:49.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:49.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 10 monclient(hunting): my global_id is 164529 2024-09-16T14:50:49.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 1 -- 172.21.15.189:0/2514023261 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde0009f3e0 con 0x7fde0009e2f0 2024-09-16T14:50:49.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 1 -- 172.21.15.189:0/2514023261 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3616787736 0 0) 0x7fddec003530 con 0x7fde0009df10 2024-09-16T14:50:49.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 1 -- 172.21.15.189:0/2514023261 >> v1:172.21.15.189:6789/0 conn(0x7fde0009e740 legacy=0x7fde0013b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:49.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 1 -- 172.21.15.189:0/2514023261 >> v1:172.21.15.26:6789/0 conn(0x7fde0009e2f0 legacy=0x7fde001378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:49.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 10 monclient: _finish_hunting 0 2024-09-16T14:50:49.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 1 monclient: found mon.noname-b 2024-09-16T14:50:49.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:49.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:49.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 1 -- 172.21.15.189:0/2514023261 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0009c690 con 0x7fde0009df10 2024-09-16T14:50:49.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 10 monclient: _finish_auth 0 2024-09-16T14:50:49.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 10 monclient: _check_auth_tickets 2024-09-16T14:50:49.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:49.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 1 -- 172.21.15.189:0/2514023261 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdde8001620 con 0x7fde0009df10 2024-09-16T14:50:49.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:49.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:49.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 10 monclient: dump: 2024-09-16T14:50:49.937 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:49.937 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:49.937 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:49.937 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:49.937 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:49.937 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:49.937 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:49.937 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:49.937 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:49.937 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:49.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 5 monclient: authenticate success, global_id 154784 2024-09-16T14:50:49.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:49.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 1 -- 172.21.15.189:0/2514023261 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdde8001900 con 0x7fde0009df10 2024-09-16T14:50:49.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:49.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 1 -- 172.21.15.189:0/2514023261 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdde8001c90 con 0x7fde0009df10 2024-09-16T14:50:49.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:49.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:49.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfd59a640 10 monclient: dump: 2024-09-16T14:50:49.938 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:49.938 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:49.938 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:49.938 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:49.938 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:49.938 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:49.939 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:49.939 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:49.939 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:49.939 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:49.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:49.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 4 set_mon_vals no callback set 2024-09-16T14:50:49.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 10 monclient: shutdown 2024-09-16T14:50:49.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:49.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 1 -- 172.21.15.189:0/2514023261 >> v1:172.21.15.26:6790/0 conn(0x7fde0009df10 legacy=0x7fde0009d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:49.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 1 -- 172.21.15.189:0/2514023261 shutdown_connections 2024-09-16T14:50:49.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 1 -- 172.21.15.189:0/2514023261 >> 172.21.15.189:0/2514023261 conn(0x7fde000a3320 msgr2=0x7fde0009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:49.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 1 -- 172.21.15.189:0/2514023261 shutdown_connections 2024-09-16T14:50:49.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 1 -- 172.21.15.189:0/2514023261 wait complete. 2024-09-16T14:50:49.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 10 monclient: build_initial_monmap 2024-09-16T14:50:49.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 10 monclient: monmap: 2024-09-16T14:50:49.940 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:49.940 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:49.940 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:49.940 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:49.940 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:49.940 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:49.940 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:49.940 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:49.940 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:49.940 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:49.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 1 Processor -- start 2024-09-16T14:50:49.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 1 -- start start 2024-09-16T14:50:49.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 10 monclient: init 2024-09-16T14:50:49.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:49.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:49.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 10 monclient(hunting): picked mon.noname-b con 0x7fde0009df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:49.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 10 monclient(hunting): picked mon.noname-c con 0x7fde0009e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:49.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 10 monclient(hunting): picked mon.noname-a con 0x7fde0009e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:49.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0006b380 con 0x7fde0009e740 2024-09-16T14:50:49.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0006fa10 con 0x7fde0009df10 2024-09-16T14:50:49.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0006ce10 con 0x7fde0009e2f0 2024-09-16T14:50:49.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:49.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfe59c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fde0009df10 legacy=0x7fde00132ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:50:49.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fde04b36640 10 monclient(hunting): authenticate will time out at 1432.050705s 2024-09-16T14:50:49.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfdd9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fde0009e2f0 0x7fde001cc0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:55326/0 (socket says 172.21.15.189:55326) 2024-09-16T14:50:49.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddfdd9b640 1 -- 172.21.15.189:0/3960981754 learned_addr learned my addr 172.21.15.189:0/3960981754 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:49.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925273678 0 0) 0x7fde0006ce10 con 0x7fde0009e2f0 2024-09-16T14:50:49.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddf6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:49.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddf6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:49.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:49.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddf6ffd640 10 monclient(hunting): my global_id is 154787 2024-09-16T14:50:49.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdde0003530 con 0x7fde0009e2f0 2024-09-16T14:50:49.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633103352 0 0) 0x7fde0006b380 con 0x7fde0009e740 2024-09-16T14:50:49.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddf6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:49.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddf6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:49.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:49.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddf6ffd640 10 monclient(hunting): my global_id is 164532 2024-09-16T14:50:49.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.932+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde0006ce10 con 0x7fde0009e740 2024-09-16T14:50:49.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 183124619 0 0) 0x7fdde0003530 con 0x7fde0009e2f0 2024-09-16T14:50:49.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde0006b380 con 0x7fde0009e2f0 2024-09-16T14:50:49.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1961175308 0 0) 0x7fde0006ce10 con 0x7fde0009e740 2024-09-16T14:50:49.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdde0003530 con 0x7fde0009e740 2024-09-16T14:50:49.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fddf00018c0 con 0x7fde0009e2f0 2024-09-16T14:50:49.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:49.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:49.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:49.947 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:49.947 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:49.947 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:49.947 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:49.948 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:49.948 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:49.948 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:49.948 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:49.948 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:49.948 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:49.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:49.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fddf80042c0 con 0x7fde0009e740 2024-09-16T14:50:49.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:49.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:49.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:49.948 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:49.949 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:49.949 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:49.949 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:49.949 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:49.949 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:49.949 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:49.949 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:49.949 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:49.949 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:49.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1497077970 0 0) 0x7fdde0003530 con 0x7fde0009e740 2024-09-16T14:50:49.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 >> v1:172.21.15.26:6790/0 conn(0x7fde0009e2f0 legacy=0x7fde001cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:49.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 >> v1:172.21.15.189:6789/0 conn(0x7fde0009df10 legacy=0x7fde00132ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:49.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:49.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 1 monclient: found mon.a 2024-09-16T14:50:49.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:49.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:49.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde001332b0 con 0x7fde0009e740 2024-09-16T14:50:49.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:49.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:49.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:49.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fde04b36640 5 monclient: authenticate success, global_id 164532 2024-09-16T14:50:49.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fde04b36640 10 monclient: _renew_subs 2024-09-16T14:50:49.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fde04b36640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:49.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fde04b36640 1 -- 172.21.15.189:0/3960981754 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0011a520 con 0x7fde0009e740 2024-09-16T14:50:49.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fde04b36640 10 monclient: _renew_subs 2024-09-16T14:50:49.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fde04b36640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:49.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fde04b36640 1 -- 172.21.15.189:0/3960981754 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde0013c350 con 0x7fde0009e740 2024-09-16T14:50:49.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddf8004460 con 0x7fde0009e740 2024-09-16T14:50:49.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:49.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddfcd99640 4 set_mon_vals no callback set 2024-09-16T14:50:49.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fde04b36640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:49.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fde04b36640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:49.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fde04b36640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:49.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fddf80047f0 con 0x7fde0009e740 2024-09-16T14:50:49.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fde04b36640 1 -- 172.21.15.189:0/3960981754 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde4005050 con 0x7fde0009e740 2024-09-16T14:50:49.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:49.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:49.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 10 monclient: dump: 2024-09-16T14:50:49.952 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:49.952 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:49.952 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:49.952 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:49.952 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:49.952 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:49.952 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:49.952 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:49.952 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:49.952 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:49.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fddf80049d0 con 0x7fde0009e740 2024-09-16T14:50:49.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(243..243 src has 169..243) v4 ==== 8135+0+0 (unknown 2395465475 0 0) 0x7fddf80442f0 con 0x7fde0009e740 2024-09-16T14:50:49.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.936+0000 7fddfe59c640 10 monclient: get_auth_request con 0x7fdde00361f0 auth_method 0 2024-09-16T14:50:49.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.948+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fddf80144f0 con 0x7fde0009e740 2024-09-16T14:50:49.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.948+0000 7fddf6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:49.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:49.948+0000 7fddf6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:50.097 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:50:50.097 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T14:50:50.098 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T14:50:50.098 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T14:50:50.100 INFO:tasks.ceph.mon.b:Started 2024-09-16T14:50:50.100 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:50:50.100 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:50:50.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.108+0000 7fde04b36640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T14:50:50.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.108+0000 7fde04b36640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T14:50:50.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.108+0000 7fde04b36640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:50.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.108+0000 7fde04b36640 1 -- 172.21.15.189:0/3960981754 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h29"} v 0) v1 -- 0x7fdde4005300 con 0x7fde0009e740 2024-09-16T14:50:50.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fbd28640 10 monclient: get_monmap_and_config 2024-09-16T14:50:50.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fbd28640 10 monclient: build_initial_monmap 2024-09-16T14:50:50.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fbd28640 10 monclient: monmap: 2024-09-16T14:50:50.227 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:50.227 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.227 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:50.227587+0000 2024-09-16T14:50:50.228 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:50.227587+0000 2024-09-16T14:50:50.228 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:50.228 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.228 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:50.228 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:50.228 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:50.228 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fbd28640 1 Processor -- start 2024-09-16T14:50:50.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fbd28640 1 -- start start 2024-09-16T14:50:50.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fbd28640 10 monclient: init 2024-09-16T14:50:50.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fbd28640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:50.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fbd28640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:50.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fbd28640 10 monclient(hunting): picked mon.noname-c con 0x7fb1f409f240 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:50.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fbd28640 10 monclient(hunting): picked mon.noname-a con 0x7fb1f406d3a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:50.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fbd28640 10 monclient(hunting): picked mon.noname-b con 0x7fb1f409fd30 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:50.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fbd28640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f406d820 con 0x7fb1f406d3a0 2024-09-16T14:50:50.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fbd28640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f406f160 con 0x7fb1f409f240 2024-09-16T14:50:50.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fbd28640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f40a0d00 con 0x7fb1f409fd30 2024-09-16T14:50:50.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fbd28640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:50.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fbd28640 10 monclient(hunting): authenticate will time out at 1474.347842s 2024-09-16T14:50:50.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1f92c3640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb1f406d3a0 0x7fb1f4136230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35490/0 (socket says 172.21.15.26:35490) 2024-09-16T14:50:50.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1f92c3640 1 -- 172.21.15.26:0/2057999294 learned_addr learned my addr 172.21.15.26:0/2057999294 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:50.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.221+0000 7fb1fa2c5640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb1f409fd30 0x7fb1f4139960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33742/0 (socket says 172.21.15.26:33742) 2024-09-16T14:50:50.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 1 -- 172.21.15.26:0/2057999294 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231275798 0 0) 0x7fb1f40a0d00 con 0x7fb1f409fd30 2024-09-16T14:50:50.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 10 monclient(hunting): my global_id is 154790 2024-09-16T14:50:50.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 1 -- 172.21.15.26:0/2057999294 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1dc003530 con 0x7fb1f409fd30 2024-09-16T14:50:50.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 1 -- 172.21.15.26:0/2057999294 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 783098905 0 0) 0x7fb1dc003530 con 0x7fb1f409fd30 2024-09-16T14:50:50.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 1 -- 172.21.15.26:0/2057999294 >> v1:172.21.15.189:6789/0 conn(0x7fb1f409f240 legacy=0x7fb1f409f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 1 -- 172.21.15.26:0/2057999294 >> v1:172.21.15.26:6789/0 conn(0x7fb1f406d3a0 legacy=0x7fb1f4136230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 10 monclient: _finish_hunting 0 2024-09-16T14:50:50.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 1 monclient: found mon.noname-b 2024-09-16T14:50:50.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:50.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:50.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 1 -- 172.21.15.26:0/2057999294 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f40a0e90 con 0x7fb1f409fd30 2024-09-16T14:50:50.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 10 monclient: _finish_auth 0 2024-09-16T14:50:50.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 10 monclient: _check_auth_tickets 2024-09-16T14:50:50.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 1 -- 172.21.15.26:0/2057999294 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb1ec001620 con 0x7fb1f409fd30 2024-09-16T14:50:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 5 monclient: authenticate success, global_id 154790 2024-09-16T14:50:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 10 monclient: dump: 2024-09-16T14:50:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.243 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.244 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.244 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.244 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.244 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.244 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.244 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:50.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 1 -- 172.21.15.26:0/2057999294 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1ec0018d0 con 0x7fb1f409fd30 2024-09-16T14:50:50.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:50.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:50.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 4 set_mon_vals no callback set 2024-09-16T14:50:50.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 1 -- 172.21.15.26:0/2057999294 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb1ec001ac0 con 0x7fb1f409fd30 2024-09-16T14:50:50.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:50.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f8ac2640 10 monclient: dump: 2024-09-16T14:50:50.245 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.245 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.245 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.245 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.245 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.246 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.246 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.246 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.246 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.246 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 10 monclient: shutdown 2024-09-16T14:50:50.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:50.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 1 -- 172.21.15.26:0/2057999294 >> v1:172.21.15.26:6790/0 conn(0x7fb1f409fd30 legacy=0x7fb1f4139960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 1 -- 172.21.15.26:0/2057999294 shutdown_connections 2024-09-16T14:50:50.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 1 -- 172.21.15.26:0/2057999294 >> 172.21.15.26:0/2057999294 conn(0x7fb1f40a3e30 msgr2=0x7fb1f40a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:50.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 1 -- 172.21.15.26:0/2057999294 shutdown_connections 2024-09-16T14:50:50.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 1 -- 172.21.15.26:0/2057999294 wait complete. 2024-09-16T14:50:50.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 10 monclient: build_initial_monmap 2024-09-16T14:50:50.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 10 monclient: monmap: 2024-09-16T14:50:50.247 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:50.247 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:50.247 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:50.247 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:50.247 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:50.247 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:50.247 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:50.247 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:50.248 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:50.248 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 1 Processor -- start 2024-09-16T14:50:50.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 1 -- start start 2024-09-16T14:50:50.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 10 monclient: init 2024-09-16T14:50:50.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:50.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:50:50.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 10 monclient(hunting): picked mon.noname-a con 0x7fb1f406d3a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:50.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 10 monclient(hunting): picked mon.noname-b con 0x7fb1f409f240 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:50.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 10 monclient(hunting): picked mon.noname-c con 0x7fb1f409fd30 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f409d700 con 0x7fb1f406d3a0 2024-09-16T14:50:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f4121320 con 0x7fb1f409f240 2024-09-16T14:50:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f411f7c0 con 0x7fb1f409fd30 2024-09-16T14:50:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 10 monclient(hunting): authenticate will time out at 1474.350794s 2024-09-16T14:50:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f9ac4640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb1f406d3a0 0x7fb1f41c5260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35498/0 (socket says 172.21.15.26:35498) 2024-09-16T14:50:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f9ac4640 1 -- 172.21.15.26:0/3717732250 learned_addr learned my addr 172.21.15.26:0/3717732250 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fa2c5640 1 --1- 172.21.15.26:0/3717732250 >> v1:172.21.15.26:6790/0 conn(0x7fb1f409fd30 0x7fb1f41cd970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33756/0 (socket says 172.21.15.26:33756) 2024-09-16T14:50:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 1 -- 172.21.15.26:0/3717732250 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274920104 0 0) 0x7fb1f411f7c0 con 0x7fb1f409fd30 2024-09-16T14:50:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 10 monclient(hunting): my global_id is 154793 2024-09-16T14:50:50.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 1 -- 172.21.15.26:0/3717732250 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e0003560 con 0x7fb1f409fd30 2024-09-16T14:50:50.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 1 -- 172.21.15.26:0/3717732250 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1505088854 0 0) 0x7fb1e0003560 con 0x7fb1f409fd30 2024-09-16T14:50:50.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 1 -- 172.21.15.26:0/3717732250 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1f411f7c0 con 0x7fb1f409fd30 2024-09-16T14:50:50.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 1 -- 172.21.15.26:0/3717732250 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb1ec001a40 con 0x7fb1f409fd30 2024-09-16T14:50:50.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:50.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 10 monclient(hunting): dump: 2024-09-16T14:50:50.250 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.250 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.250 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.251 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.251 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.251 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.251 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.251 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.251 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.251 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:50.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 1 -- 172.21.15.26:0/3717732250 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2848968645 0 0) 0x7fb1f411f7c0 con 0x7fb1f409fd30 2024-09-16T14:50:50.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 1 -- 172.21.15.26:0/3717732250 >> v1:172.21.15.189:6789/0 conn(0x7fb1f409f240 legacy=0x7fb1f41ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 1 -- 172.21.15.26:0/3717732250 >> v1:172.21.15.26:6789/0 conn(0x7fb1f406d3a0 legacy=0x7fb1f41c5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 10 monclient: _finish_hunting 0 2024-09-16T14:50:50.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 1 monclient: found mon.c 2024-09-16T14:50:50.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:50.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:50.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 1 -- 172.21.15.26:0/3717732250 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f411fe70 con 0x7fb1f409fd30 2024-09-16T14:50:50.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 10 monclient: _finish_auth 0 2024-09-16T14:50:50.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:50.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:50.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 5 monclient: authenticate success, global_id 154793 2024-09-16T14:50:50.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 10 monclient: _renew_subs 2024-09-16T14:50:50.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:50.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 1 -- 172.21.15.26:0/3717732250 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f40a1d90 con 0x7fb1f409fd30 2024-09-16T14:50:50.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 10 monclient: _renew_subs 2024-09-16T14:50:50.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:50.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1fbd28640 1 -- 172.21.15.26:0/3717732250 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f4121600 con 0x7fb1f409fd30 2024-09-16T14:50:50.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 1 -- 172.21.15.26:0/3717732250 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1ec004a80 con 0x7fb1f409fd30 2024-09-16T14:50:50.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:50.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 1 -- 172.21.15.26:0/3717732250 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb1ec004d90 con 0x7fb1f409fd30 2024-09-16T14:50:50.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f3fff640 4 set_mon_vals no callback set 2024-09-16T14:50:50.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:50.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.225+0000 7fb1f27fc640 10 monclient: dump: 2024-09-16T14:50:50.253 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.253 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.253 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.253 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.254 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.254 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.254 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.254 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.254 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.254 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.229+0000 7fb1fbd28640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:50.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.229+0000 7fb1f27fc640 1 -- 172.21.15.26:0/3717732250 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb1ec004ef0 con 0x7fb1f409fd30 2024-09-16T14:50:50.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.229+0000 7fb1fbd28640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:50.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.229+0000 7fb1fbd28640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:50.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.229+0000 7fb1fbd28640 1 -- 172.21.15.26:0/3717732250 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d4005050 con 0x7fb1f409fd30 2024-09-16T14:50:50.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.229+0000 7fb1f27fc640 1 -- 172.21.15.26:0/3717732250 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(243..243 src has 169..243) v4 ==== 8135+0+0 (unknown 2395465475 0 0) 0x7fb1ec044410 con 0x7fb1f409fd30 2024-09-16T14:50:50.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.229+0000 7fb1f9ac4640 10 monclient: get_auth_request con 0x7fb1e00362d0 auth_method 0 2024-09-16T14:50:50.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.233+0000 7fb1f27fc640 1 -- 172.21.15.26:0/3717732250 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb1ec014610 con 0x7fb1f409fd30 2024-09-16T14:50:50.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.233+0000 7fb1f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:50.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.233+0000 7fb1f27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:50.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.453+0000 7fb1fbd28640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:50:50.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.453+0000 7fb1fbd28640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:50.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.453+0000 7fb1fbd28640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:50.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.453+0000 7fb1fbd28640 1 -- 172.21.15.26:0/3717732250 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb1d4005880 con 0x7fb1f409fd30 2024-09-16T14:50:50.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.453+0000 7fb1f27fc640 1 -- 172.21.15.26:0/3717732250 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2003649686) 0x7fb1ec012cb0 con 0x7fb1f409fd30 2024-09-16T14:50:50.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.453+0000 7fb1f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:50.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.453+0000 7fb1f27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:50.458 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:50:50.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.453+0000 7fb1fbd28640 1 -- 172.21.15.26:0/3717732250 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb1e00362d0 legacy=0x7fb1e0038790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.453+0000 7fb1fbd28640 10 monclient: shutdown 2024-09-16T14:50:50.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.453+0000 7fb1fbd28640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:50.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.453+0000 7fb1fbd28640 1 -- 172.21.15.26:0/3717732250 >> v1:172.21.15.26:6790/0 conn(0x7fb1f409fd30 legacy=0x7fb1f41cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.453+0000 7fb1fbd28640 1 -- 172.21.15.26:0/3717732250 shutdown_connections 2024-09-16T14:50:50.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.453+0000 7fb1fbd28640 1 -- 172.21.15.26:0/3717732250 >> 172.21.15.26:0/3717732250 conn(0x7fb1f40a3e30 msgr2=0x7fb1f40700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:50.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.453+0000 7fb1fbd28640 1 -- 172.21.15.26:0/3717732250 shutdown_connections 2024-09-16T14:50:50.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.453+0000 7fb1fbd28640 1 -- 172.21.15.26:0/3717732250 wait complete. 2024-09-16T14:50:50.474 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:50:50.474 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:50:50.474 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:50:50.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 10 monclient: get_monmap_and_config 2024-09-16T14:50:50.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 10 monclient: build_initial_monmap 2024-09-16T14:50:50.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 10 monclient: monmap: 2024-09-16T14:50:50.600 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:50.600 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.600 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:50.601488+0000 2024-09-16T14:50:50.600 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:50.601488+0000 2024-09-16T14:50:50.601 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:50.601 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.601 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:50.601 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:50.601 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:50.601 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 1 Processor -- start 2024-09-16T14:50:50.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 1 -- start start 2024-09-16T14:50:50.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 10 monclient: init 2024-09-16T14:50:50.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:50.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:50.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 10 monclient(hunting): picked mon.noname-b con 0x7f16540a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:50.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 10 monclient(hunting): picked mon.noname-c con 0x7f16540a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:50.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 10 monclient(hunting): picked mon.noname-a con 0x7f16540a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:50.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16540a3250 con 0x7f16540a2800 2024-09-16T14:50:50.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f165409f600 con 0x7f16540a0000 2024-09-16T14:50:50.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16540a3830 con 0x7f16540a1d10 2024-09-16T14:50:50.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:50.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 10 monclient(hunting): authenticate will time out at 1474.721537s 2024-09-16T14:50:50.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165259c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f16540a1d10 0x7f16540a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33772/0 (socket says 172.21.15.26:33772) 2024-09-16T14:50:50.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165259c640 1 -- 172.21.15.26:0/3111788663 learned_addr learned my addr 172.21.15.26:0/3111788663 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:50.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1652d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f16540a2800 0x7f165413d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35508/0 (socket says 172.21.15.26:35508) 2024-09-16T14:50:50.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 1 -- 172.21.15.26:0/3111788663 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182850491 0 0) 0x7f16540a3830 con 0x7f16540a1d10 2024-09-16T14:50:50.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient(hunting): my global_id is 154796 2024-09-16T14:50:50.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 1 -- 172.21.15.26:0/3111788663 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f163c003530 con 0x7f16540a1d10 2024-09-16T14:50:50.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 1 -- 172.21.15.26:0/3111788663 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218311753 0 0) 0x7f16540a3250 con 0x7f16540a2800 2024-09-16T14:50:50.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient(hunting): my global_id is 164535 2024-09-16T14:50:50.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 1 -- 172.21.15.26:0/3111788663 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16540a3830 con 0x7f16540a2800 2024-09-16T14:50:50.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 1 -- 172.21.15.26:0/3111788663 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 651877750 0 0) 0x7f163c003530 con 0x7f16540a1d10 2024-09-16T14:50:50.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 1 -- 172.21.15.26:0/3111788663 >> v1:172.21.15.189:6789/0 conn(0x7f16540a0000 legacy=0x7f1654139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 1 -- 172.21.15.26:0/3111788663 >> v1:172.21.15.26:6789/0 conn(0x7f16540a2800 legacy=0x7f165413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient: _finish_hunting 0 2024-09-16T14:50:50.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 1 monclient: found mon.noname-b 2024-09-16T14:50:50.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:50.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:50.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 1 -- 172.21.15.26:0/3111788663 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165409ed60 con 0x7f16540a1d10 2024-09-16T14:50:50.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient: _finish_auth 0 2024-09-16T14:50:50.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient: _check_auth_tickets 2024-09-16T14:50:50.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:50.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 1 -- 172.21.15.26:0/3111788663 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1644001620 con 0x7f16540a1d10 2024-09-16T14:50:50.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 5 monclient: authenticate success, global_id 154796 2024-09-16T14:50:50.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:50.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:50.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient: dump: 2024-09-16T14:50:50.609 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.609 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.609 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.609 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.609 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.609 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.609 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.610 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.610 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.610 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:50.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 1 -- 172.21.15.26:0/3111788663 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1644001840 con 0x7f16540a1d10 2024-09-16T14:50:50.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:50.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 1 -- 172.21.15.26:0/3111788663 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1644001bd0 con 0x7f16540a1d10 2024-09-16T14:50:50.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:50.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f165159a640 10 monclient: dump: 2024-09-16T14:50:50.610 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.610 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.610 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.611 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.611 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.611 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.611 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.611 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.611 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.611 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:50.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 4 set_mon_vals no callback set 2024-09-16T14:50:50.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 10 monclient: shutdown 2024-09-16T14:50:50.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:50.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 1 -- 172.21.15.26:0/3111788663 >> v1:172.21.15.26:6790/0 conn(0x7f16540a1d10 legacy=0x7f16540a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 1 -- 172.21.15.26:0/3111788663 shutdown_connections 2024-09-16T14:50:50.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 1 -- 172.21.15.26:0/3111788663 >> 172.21.15.26:0/3111788663 conn(0x7f16540a7f30 msgr2=0x7f16540a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:50.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 1 -- 172.21.15.26:0/3111788663 shutdown_connections 2024-09-16T14:50:50.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.597+0000 7f1658c56640 1 -- 172.21.15.26:0/3111788663 wait complete. 2024-09-16T14:50:50.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 10 monclient: build_initial_monmap 2024-09-16T14:50:50.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 10 monclient: monmap: 2024-09-16T14:50:50.612 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:50.612 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:50.613 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:50.613 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:50.613 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:50.613 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:50.613 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:50.613 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:50.613 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:50.613 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 1 Processor -- start 2024-09-16T14:50:50.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 1 -- start start 2024-09-16T14:50:50.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 10 monclient: init 2024-09-16T14:50:50.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:50.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:50.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 10 monclient(hunting): picked mon.noname-b con 0x7f16540a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:50.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 10 monclient(hunting): picked mon.noname-a con 0x7f16540a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:50.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 10 monclient(hunting): picked mon.noname-c con 0x7f16540a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:50.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f165413d210 con 0x7f16540a1d10 2024-09-16T14:50:50.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16540a6880 con 0x7f16540a0000 2024-09-16T14:50:50.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f165409e500 con 0x7f16540a2800 2024-09-16T14:50:50.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:50.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 10 monclient(hunting): authenticate will time out at 1474.725063s 2024-09-16T14:50:50.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1652d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f16540a2800 0x7f16541d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33788/0 (socket says 172.21.15.26:33788) 2024-09-16T14:50:50.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1652d9d640 1 -- 172.21.15.26:0/2618387124 learned_addr learned my addr 172.21.15.26:0/2618387124 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:50.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928328489 0 0) 0x7f165409e500 con 0x7f16540a2800 2024-09-16T14:50:50.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): my global_id is 154799 2024-09-16T14:50:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1634003530 con 0x7f16540a2800 2024-09-16T14:50:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726320427 0 0) 0x7f165413d210 con 0x7f16540a1d10 2024-09-16T14:50:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): my global_id is 164538 2024-09-16T14:50:50.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f165409e500 con 0x7f16540a1d10 2024-09-16T14:50:50.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1836886393 0 0) 0x7f1634003530 con 0x7f16540a2800 2024-09-16T14:50:50.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f165413d210 con 0x7f16540a2800 2024-09-16T14:50:50.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f164c001be0 con 0x7f16540a2800 2024-09-16T14:50:50.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:50.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:50.622 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.622 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.622 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.622 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.622 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.623 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.623 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.623 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.623 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.623 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:50.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 35340908 0 0) 0x7f165409e500 con 0x7f16540a1d10 2024-09-16T14:50:50.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1634003530 con 0x7f16540a1d10 2024-09-16T14:50:50.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1648001880 con 0x7f16540a1d10 2024-09-16T14:50:50.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:50.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:50.624 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.624 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.624 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.624 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.624 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.624 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.624 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.624 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.624 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.624 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 415671952 0 0) 0x7f165413d210 con 0x7f16540a2800 2024-09-16T14:50:50.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 >> v1:172.21.15.189:6789/0 conn(0x7f16540a0000 legacy=0x7f1654139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 >> v1:172.21.15.26:6789/0 conn(0x7f16540a1d10 legacy=0x7f1654139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:50.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 monclient: found mon.c 2024-09-16T14:50:50.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:50.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:50.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165413c970 con 0x7f16540a2800 2024-09-16T14:50:50.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:50.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:50.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:50.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 5 monclient: authenticate success, global_id 154799 2024-09-16T14:50:50.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 10 monclient: _renew_subs 2024-09-16T14:50:50.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:50.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 1 -- 172.21.15.26:0/2618387124 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165413ae40 con 0x7f16540a2800 2024-09-16T14:50:50.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 10 monclient: _renew_subs 2024-09-16T14:50:50.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:50.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 1 -- 172.21.15.26:0/2618387124 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f165413d900 con 0x7f16540a2800 2024-09-16T14:50:50.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f164c004b70 con 0x7f16540a2800 2024-09-16T14:50:50.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:50.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1650d99640 4 set_mon_vals no callback set 2024-09-16T14:50:50.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f164c004e80 con 0x7f16540a2800 2024-09-16T14:50:50.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:50.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient: dump: 2024-09-16T14:50:50.626 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.626 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.626 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.627 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.627 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.627 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.627 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.627 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.627 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.627 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:50.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f164c00f8f0 con 0x7f16540a2800 2024-09-16T14:50:50.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1658c56640 1 -- 172.21.15.26:0/2618387124 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1638006590 con 0x7f1638002830 2024-09-16T14:50:50.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(243..243 src has 169..243) v4 ==== 8135+0+0 (unknown 2395465475 0 0) 0x7f164c044580 con 0x7f16540a2800 2024-09-16T14:50:50.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291394416 0 0) 0x7f1638006590 con 0x7f1638002830 2024-09-16T14:50:50.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 10 monclient(hunting): my global_id is 164541 2024-09-16T14:50:50.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.601+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f165413d210 con 0x7f1638002830 2024-09-16T14:50:50.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.605+0000 7f1651d9b640 10 monclient: get_auth_request con 0x7f16340362a0 auth_method 0 2024-09-16T14:50:50.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.605+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 779836211 0 0) 0x7f165413d210 con 0x7f1638002830 2024-09-16T14:50:50.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.605+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1638006590 con 0x7f1638002830 2024-09-16T14:50:50.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.605+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f16440013e0 con 0x7f1638002830 2024-09-16T14:50:50.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.605+0000 7f1642ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.605+0000 7f1642ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:50.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.605+0000 7f1642ffd640 10 monclient: dump: 2024-09-16T14:50:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.629 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.605+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3755536112 0 0) 0x7f1638006590 con 0x7f1638002830 2024-09-16T14:50:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.605+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f163800a0e0 con 0x7f1638002830 2024-09-16T14:50:50.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.605+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1644000fa0 con 0x7f1638002830 2024-09-16T14:50:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.605+0000 7f1642ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.605+0000 7f1642ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.605+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 >> v1:172.21.15.26:6789/0 conn(0x7f1638002830 legacy=0x7f1638004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.605+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 >> v1:172.21.15.26:6789/0 conn(0x7f1638002830 legacy=0x7f1638004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1658c56640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:50:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1658c56640 1 -- 172.21.15.26:0/2618387124 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1638009da0 con 0x7f16380090c0 2024-09-16T14:50:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735776357 0 0) 0x7f1638009da0 con 0x7f16380090c0 2024-09-16T14:50:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 10 monclient(hunting): my global_id is 164544 2024-09-16T14:50:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1638006590 con 0x7f16380090c0 2024-09-16T14:50:50.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 519088454 0 0) 0x7f1638006590 con 0x7f16380090c0 2024-09-16T14:50:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1638009da0 con 0x7f16380090c0 2024-09-16T14:50:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1644004c00 con 0x7f16380090c0 2024-09-16T14:50:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 10 monclient: dump: 2024-09-16T14:50:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.631 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.632 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3391382140 0 0) 0x7f1638009da0 con 0x7f16380090c0 2024-09-16T14:50:50.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f163800c290 con 0x7f16380090c0 2024-09-16T14:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:50:50.632 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 78, 2024-09-16T14:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:50.633 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:50.634 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:50:50.635 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:50.636 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.637 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:50:50.638 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.639 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:50.640 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.641 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.642 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:50:50.643 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:50:50.644 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:50:50.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 3768637152) 0x7f1644005fe0 con 0x7f16380090c0 2024-09-16T14:50:50.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:50:50.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 >> v1:172.21.15.26:6789/0 conn(0x7f16380090c0 legacy=0x7f1638009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.609+0000 7f1642ffd640 1 -- 172.21.15.26:0/2618387124 >> v1:172.21.15.26:6789/0 conn(0x7f16380090c0 legacy=0x7f1638009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.613+0000 7f1658c56640 1 -- 172.21.15.26:0/2618387124 >> v1:172.21.15.189:6812/3196161582 conn(0x7f16340362a0 legacy=0x7f1634038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.613+0000 7f1658c56640 10 monclient: shutdown 2024-09-16T14:50:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.613+0000 7f1658c56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.613+0000 7f1658c56640 1 -- 172.21.15.26:0/2618387124 >> v1:172.21.15.26:6790/0 conn(0x7f16540a2800 legacy=0x7f16541d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.613+0000 7f1652d9d640 1 -- 172.21.15.26:0/2618387124 reap_dead start 2024-09-16T14:50:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.613+0000 7f1652d9d640 1 -- 172.21.15.26:0/2618387124 reap_dead start 2024-09-16T14:50:50.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.613+0000 7f1658c56640 1 -- 172.21.15.26:0/2618387124 shutdown_connections 2024-09-16T14:50:50.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.613+0000 7f1658c56640 1 -- 172.21.15.26:0/2618387124 >> 172.21.15.26:0/2618387124 conn(0x7f16540a7f30 msgr2=0x7f16540a54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:50.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.613+0000 7f1658c56640 1 -- 172.21.15.26:0/2618387124 shutdown_connections 2024-09-16T14:50:50.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.613+0000 7f1658c56640 1 -- 172.21.15.26:0/2618387124 wait complete. 2024-09-16T14:50:50.648 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:50:50.648 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:50:50.648 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:50:50.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.700+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h29"}]=0 removed item id -103 name 'h29' from crush map v244) v1 ==== 121+0+0 (unknown 2949187316 0 0) 0x7fddf8012b90 con 0x7fde0009e740 2024-09-16T14:50:50.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.700+0000 7fddf6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T14:50:50.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.700+0000 7fddf6ffd640 10 monclient: _finish_command 2 = system:0 removed item id -103 name 'h29' from crush map 2024-09-16T14:50:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: get_monmap_and_config 2024-09-16T14:50:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: build_initial_monmap 2024-09-16T14:50:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: monmap: 2024-09-16T14:50:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:50.767410+0000 2024-09-16T14:50:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:50.767410+0000 2024-09-16T14:50:50.766 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:50.767 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.767 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:50.767 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:50.767 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:50.767 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 Processor -- start 2024-09-16T14:50:50.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 -- start start 2024-09-16T14:50:50.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: init 2024-09-16T14:50:50.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:50.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:50:50.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient(hunting): picked mon.noname-a con 0x7f92e40a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:50.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient(hunting): picked mon.noname-c con 0x7f92e4099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:50.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient(hunting): picked mon.noname-b con 0x7f92e40a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:50.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e4098ea0 con 0x7f92e40a1c00 2024-09-16T14:50:50.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e40997b0 con 0x7f92e4099b60 2024-09-16T14:50:50.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e4099460 con 0x7f92e40a26f0 2024-09-16T14:50:50.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:50.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient(hunting): authenticate will time out at 1474.887496s 2024-09-16T14:50:50.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e2d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f92e40a1c00 0x7f92e40a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35530/0 (socket says 172.21.15.26:35530) 2024-09-16T14:50:50.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e2d9d640 1 -- 172.21.15.26:0/2075939800 learned_addr learned my addr 172.21.15.26:0/2075939800 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:50.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 1 -- 172.21.15.26:0/2075939800 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168972365 0 0) 0x7f92e4098ea0 con 0x7f92e40a1c00 2024-09-16T14:50:50.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient(hunting): my global_id is 164547 2024-09-16T14:50:50.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 1 -- 172.21.15.26:0/2075939800 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d0003530 con 0x7f92e40a1c00 2024-09-16T14:50:50.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 1 -- 172.21.15.26:0/2075939800 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732974393 0 0) 0x7f92e4099460 con 0x7f92e40a26f0 2024-09-16T14:50:50.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient(hunting): my global_id is 154802 2024-09-16T14:50:50.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 1 -- 172.21.15.26:0/2075939800 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e4098ea0 con 0x7f92e40a26f0 2024-09-16T14:50:50.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 1 -- 172.21.15.26:0/2075939800 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 108240157 0 0) 0x7f92d0003530 con 0x7f92e40a1c00 2024-09-16T14:50:50.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 1 -- 172.21.15.26:0/2075939800 >> v1:172.21.15.26:6790/0 conn(0x7f92e40a26f0 legacy=0x7f92e413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 1 -- 172.21.15.26:0/2075939800 >> v1:172.21.15.189:6789/0 conn(0x7f92e4099b60 legacy=0x7f92e4139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:50:50.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 1 monclient: found mon.noname-a 2024-09-16T14:50:50.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:50.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:50.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 1 -- 172.21.15.26:0/2075939800 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e40a34b0 con 0x7f92e40a1c00 2024-09-16T14:50:50.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient: _finish_auth 0 2024-09-16T14:50:50.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:50:50.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:50.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 1 -- 172.21.15.26:0/2075939800 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f92d4001620 con 0x7f92e40a1c00 2024-09-16T14:50:50.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:50.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient: dump: 2024-09-16T14:50:50.777 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.777 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.777 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.777 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.778 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.778 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.778 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.778 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.778 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.778 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 5 monclient: authenticate success, global_id 164547 2024-09-16T14:50:50.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:50.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 1 -- 172.21.15.26:0/2075939800 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92d4001840 con 0x7f92e40a1c00 2024-09-16T14:50:50.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:50.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:50.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 1 -- 172.21.15.26:0/2075939800 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f92d4001c10 con 0x7f92e40a1c00 2024-09-16T14:50:50.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 4 set_mon_vals no callback set 2024-09-16T14:50:50.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:50.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e1d9b640 10 monclient: dump: 2024-09-16T14:50:50.779 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.779 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.779 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.779 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.779 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.779 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.779 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.779 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.779 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.779 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: shutdown 2024-09-16T14:50:50.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:50.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 -- 172.21.15.26:0/2075939800 >> v1:172.21.15.26:6789/0 conn(0x7f92e40a1c00 legacy=0x7f92e40a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 -- 172.21.15.26:0/2075939800 shutdown_connections 2024-09-16T14:50:50.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 -- 172.21.15.26:0/2075939800 >> 172.21.15.26:0/2075939800 conn(0x7f92e40a3d00 msgr2=0x7f92e40a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:50.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 -- 172.21.15.26:0/2075939800 shutdown_connections 2024-09-16T14:50:50.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 -- 172.21.15.26:0/2075939800 wait complete. 2024-09-16T14:50:50.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: build_initial_monmap 2024-09-16T14:50:50.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: monmap: 2024-09-16T14:50:50.780 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:50.780 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:50.780 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:50.780 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:50.781 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:50.781 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:50.781 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:50.781 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:50.781 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:50.781 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 Processor -- start 2024-09-16T14:50:50.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 -- start start 2024-09-16T14:50:50.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: init 2024-09-16T14:50:50.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:50.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:50:50.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient(hunting): picked mon.noname-a con 0x7f92e4099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:50.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient(hunting): picked mon.noname-b con 0x7f92e40a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:50.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient(hunting): picked mon.noname-c con 0x7f92e40a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:50.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e413ad60 con 0x7f92e4099b60 2024-09-16T14:50:50.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e409eb40 con 0x7f92e40a1c00 2024-09-16T14:50:50.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e409e3f0 con 0x7f92e40a26f0 2024-09-16T14:50:50.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:50.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient(hunting): authenticate will time out at 1474.890651s 2024-09-16T14:50:50.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e2d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f92e4099b60 0x7f92e4139f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35544/0 (socket says 172.21.15.26:35544) 2024-09-16T14:50:50.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e2d9d640 1 -- 172.21.15.26:0/1938879041 learned_addr learned my addr 172.21.15.26:0/1938879041 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:50.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33174120 0 0) 0x7f92e413ad60 con 0x7f92e4099b60 2024-09-16T14:50:50.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): my global_id is 164550 2024-09-16T14:50:50.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c4003530 con 0x7f92e4099b60 2024-09-16T14:50:50.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956722958 0 0) 0x7f92e409e3f0 con 0x7f92e40a26f0 2024-09-16T14:50:50.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): my global_id is 154805 2024-09-16T14:50:50.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e413ad60 con 0x7f92e40a26f0 2024-09-16T14:50:50.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3888177975 0 0) 0x7f92c4003530 con 0x7f92e4099b60 2024-09-16T14:50:50.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92e409e3f0 con 0x7f92e4099b60 2024-09-16T14:50:50.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f92d4001440 con 0x7f92e4099b60 2024-09-16T14:50:50.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:50.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): dump: 2024-09-16T14:50:50.784 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.784 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.784 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.784 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.784 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.784 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.784 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.784 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.784 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.784 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:50.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3525122214 0 0) 0x7f92e413ad60 con 0x7f92e40a26f0 2024-09-16T14:50:50.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92c4003530 con 0x7f92e40a26f0 2024-09-16T14:50:50.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535732105 0 0) 0x7f92e409eb40 con 0x7f92e40a1c00 2024-09-16T14:50:50.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): my global_id is 184102 2024-09-16T14:50:50.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e413ad60 con 0x7f92e40a1c00 2024-09-16T14:50:50.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f92dc001100 con 0x7f92e40a26f0 2024-09-16T14:50:50.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:50.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient(hunting): dump: 2024-09-16T14:50:50.786 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.786 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.786 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.786 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.786 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.786 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.787 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.787 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.787 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.787 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2472238874 0 0) 0x7f92e409e3f0 con 0x7f92e4099b60 2024-09-16T14:50:50.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 >> v1:172.21.15.26:6790/0 conn(0x7f92e40a26f0 legacy=0x7f92e41d19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 >> v1:172.21.15.189:6789/0 conn(0x7f92e40a1c00 legacy=0x7f92e413a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:50.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 monclient: found mon.a 2024-09-16T14:50:50.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:50.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:50.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e409f090 con 0x7f92e4099b60 2024-09-16T14:50:50.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:50.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:50.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:50.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 5 monclient: authenticate success, global_id 164550 2024-09-16T14:50:50.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: _renew_subs 2024-09-16T14:50:50.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:50.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 -- 172.21.15.26:0/1938879041 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e413dd50 con 0x7f92e4099b60 2024-09-16T14:50:50.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: _renew_subs 2024-09-16T14:50:50.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:50.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e91a7640 1 -- 172.21.15.26:0/1938879041 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e413da70 con 0x7f92e4099b60 2024-09-16T14:50:50.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92d4004c00 con 0x7f92e4099b60 2024-09-16T14:50:50.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:50.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f92d4004f10 con 0x7f92e4099b60 2024-09-16T14:50:50.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:50.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92e159a640 4 set_mon_vals no callback set 2024-09-16T14:50:50.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.765+0000 7f92db7fe640 10 monclient: dump: 2024-09-16T14:50:50.790 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.790 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.790 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.790 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.790 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.790 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.790 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.790 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.790 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.790 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92e91a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:50.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92e91a7640 1 -- 172.21.15.26:0/1938879041 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f92c8006590 con 0x7f92c8002830 2024-09-16T14:50:50.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f92d400f980 con 0x7f92e4099b60 2024-09-16T14:50:50.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(244..244 src has 169..244) v4 ==== 7922+0+0 (unknown 1242468724 0 0) 0x7f92d4044480 con 0x7f92e4099b60 2024-09-16T14:50:50.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720356271 0 0) 0x7f92c8006590 con 0x7f92c8002830 2024-09-16T14:50:50.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 10 monclient(hunting): my global_id is 154808 2024-09-16T14:50:50.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e409e3f0 con 0x7f92c8002830 2024-09-16T14:50:50.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92e359e640 10 monclient: get_auth_request con 0x7f92c4036370 auth_method 0 2024-09-16T14:50:50.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 943780799 0 0) 0x7f92e409e3f0 con 0x7f92c8002830 2024-09-16T14:50:50.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92c8006590 con 0x7f92c8002830 2024-09-16T14:50:50.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f92cc001b20 con 0x7f92c8002830 2024-09-16T14:50:50.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:50.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 10 monclient: dump: 2024-09-16T14:50:50.792 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.792 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.792 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.793 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.793 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.793 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.793 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.793 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.793 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.793 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3584265016 0 0) 0x7f92c8006590 con 0x7f92c8002830 2024-09-16T14:50:50.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92c800a0e0 con 0x7f92c8002830 2024-09-16T14:50:50.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f92cc001ec0 con 0x7f92c8002830 2024-09-16T14:50:50.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:50.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:50.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 >> v1:172.21.15.26:6790/0 conn(0x7f92c8002830 legacy=0x7f92c8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.769+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 >> v1:172.21.15.26:6790/0 conn(0x7f92c8002830 legacy=0x7f92c8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:50.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92e91a7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:50:50.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92e91a7640 1 -- 172.21.15.26:0/1938879041 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f92c8009d80 con 0x7f92c80090a0 2024-09-16T14:50:50.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920223603 0 0) 0x7f92c8009d80 con 0x7f92c80090a0 2024-09-16T14:50:50.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 10 monclient(hunting): my global_id is 154811 2024-09-16T14:50:50.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c8006590 con 0x7f92c80090a0 2024-09-16T14:50:50.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1493072567 0 0) 0x7f92c8006590 con 0x7f92c80090a0 2024-09-16T14:50:50.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92c8009d80 con 0x7f92c80090a0 2024-09-16T14:50:50.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f92cc003d30 con 0x7f92c80090a0 2024-09-16T14:50:50.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:50.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 10 monclient: dump: 2024-09-16T14:50:50.797 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.798 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.798 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.798 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.798 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.798 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.798 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.798 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.798 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.798 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3737555501 0 0) 0x7f92c8009d80 con 0x7f92c80090a0 2024-09-16T14:50:50.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f92c800c290 con 0x7f92c80090a0 2024-09-16T14:50:50.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 369346183) 0x7f92cc0049d0 con 0x7f92c80090a0 2024-09-16T14:50:50.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:50:50.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:50.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 >> v1:172.21.15.26:6790/0 conn(0x7f92c80090a0 legacy=0x7f92c8009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92db7fe640 1 -- 172.21.15.26:0/1938879041 >> v1:172.21.15.26:6790/0 conn(0x7f92c80090a0 legacy=0x7f92c8009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:50.800 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:50:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:50:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 78, 2024-09-16T14:50:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:50:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:50:50.800 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:50:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:50:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:50:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:50:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:50:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:50.801 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:50:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:50:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:50.802 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:50:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:50:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:50:50.803 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:50:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:50:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:50:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:50:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:50:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:50:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:50:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:50:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:50:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:50:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:50:50.804 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:50:50.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:50:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:50:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:50:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:50:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.807 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:50:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:50:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:50:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.808 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:50:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:50:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:50:50.809 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:50:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:50:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.810 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:50:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:50:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:50:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-09-16T14:50:50.811 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.812 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:50:50.812 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:50:50.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92e91a7640 1 -- 172.21.15.26:0/1938879041 >> v1:172.21.15.189:6812/3196161582 conn(0x7f92c4036370 legacy=0x7f92c4038830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92e91a7640 10 monclient: shutdown 2024-09-16T14:50:50.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92e91a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:50.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92e91a7640 1 -- 172.21.15.26:0/1938879041 >> v1:172.21.15.26:6789/0 conn(0x7f92e4099b60 legacy=0x7f92e4139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92e359e640 1 -- 172.21.15.26:0/1938879041 reap_dead start 2024-09-16T14:50:50.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92e359e640 1 -- 172.21.15.26:0/1938879041 reap_dead start 2024-09-16T14:50:50.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92e91a7640 1 -- 172.21.15.26:0/1938879041 shutdown_connections 2024-09-16T14:50:50.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92e91a7640 1 -- 172.21.15.26:0/1938879041 >> 172.21.15.26:0/1938879041 conn(0x7f92e40a3d00 msgr2=0x7f92e40a0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:50.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92e91a7640 1 -- 172.21.15.26:0/1938879041 shutdown_connections 2024-09-16T14:50:50.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.777+0000 7f92e91a7640 1 -- 172.21.15.26:0/1938879041 wait complete. 2024-09-16T14:50:50.814 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:50:50.814 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:50:50.815 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:50:50.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.800+0000 7fde04b36640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T14:50:50.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.800+0000 7fde04b36640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T14:50:50.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.800+0000 7fde04b36640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:50:50.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.800+0000 7fde04b36640 1 -- 172.21.15.189:0/3960981754 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h29"} v 0) v1 -- 0x7fdde4005a40 con 0x7fde0009e740 2024-09-16T14:50:50.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.800+0000 7fddf6ffd640 1 -- 172.21.15.189:0/3960981754 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h29"}]=0 device 'h29' does not appear in the crush map v244) v1 ==== 120+0+0 (unknown 3224777840 0 0) 0x7fddf8014310 con 0x7fde0009e740 2024-09-16T14:50:50.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.800+0000 7fddf6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T14:50:50.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.800+0000 7fddf6ffd640 10 monclient: _finish_command 3 = system:0 device 'h29' does not appear in the crush map 2024-09-16T14:50:50.817 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h29' does not appear in the crush map 2024-09-16T14:50:50.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.800+0000 7fde04b36640 1 -- 172.21.15.189:0/3960981754 >> v1:172.21.15.189:6812/3196161582 conn(0x7fdde00361f0 legacy=0x7fdde00386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.800+0000 7fde04b36640 10 monclient: shutdown 2024-09-16T14:50:50.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.800+0000 7fde04b36640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:50.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.800+0000 7fde04b36640 1 -- 172.21.15.189:0/3960981754 >> v1:172.21.15.26:6789/0 conn(0x7fde0009e740 legacy=0x7fde001cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.800+0000 7fde04b36640 1 -- 172.21.15.189:0/3960981754 shutdown_connections 2024-09-16T14:50:50.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.800+0000 7fde04b36640 1 -- 172.21.15.189:0/3960981754 >> 172.21.15.189:0/3960981754 conn(0x7fde000a3320 msgr2=0x7fde0006d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:50.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.804+0000 7fde04b36640 1 -- 172.21.15.189:0/3960981754 shutdown_connections 2024-09-16T14:50:50.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.804+0000 7fde04b36640 1 -- 172.21.15.189:0/3960981754 wait complete. 2024-09-16T14:50:50.818 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:50.818 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h30 2024-09-16T14:50:50.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d3812640 10 monclient: get_monmap_and_config 2024-09-16T14:50:50.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d3812640 10 monclient: build_initial_monmap 2024-09-16T14:50:50.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d3812640 10 monclient: monmap: 2024-09-16T14:50:50.930 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:50.930 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.930 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:50.930741+0000 2024-09-16T14:50:50.930 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:50.930741+0000 2024-09-16T14:50:50.930 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:50.930 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.930 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:50.930 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:50.930 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:50.930 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d3812640 1 Processor -- start 2024-09-16T14:50:50.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d3812640 1 -- start start 2024-09-16T14:50:50.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d3812640 10 monclient: init 2024-09-16T14:50:50.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d3812640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:50.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d3812640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:50.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d3812640 10 monclient(hunting): picked mon.noname-b con 0x7f10cc09a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:50.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d3812640 10 monclient(hunting): picked mon.noname-c con 0x7f10cc09d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:50.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d3812640 10 monclient(hunting): picked mon.noname-a con 0x7f10cc09b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:50.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d3812640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc09dc60 con 0x7f10cc09b040 2024-09-16T14:50:50.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d3812640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc099ae0 con 0x7f10cc09d1b0 2024-09-16T14:50:50.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d3812640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc09b8f0 con 0x7f10cc09a550 2024-09-16T14:50:50.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d3812640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:50.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d3812640 10 monclient(hunting): authenticate will time out at 1475.051492s 2024-09-16T14:50:50.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d15ae640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f10cc09a550 0x7f10cc09a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33834/0 (socket says 172.21.15.26:33834) 2024-09-16T14:50:50.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d15ae640 1 -- 172.21.15.26:0/1339262950 learned_addr learned my addr 172.21.15.26:0/1339262950 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:50.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10d1daf640 1 --1- 172.21.15.26:0/1339262950 >> v1:172.21.15.26:6789/0 conn(0x7f10cc09b040 0x7f10cc13a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35546/0 (socket says 172.21.15.26:35546) 2024-09-16T14:50:50.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10c3fff640 1 -- 172.21.15.26:0/1339262950 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737142305 0 0) 0x7f10cc09dc60 con 0x7f10cc09b040 2024-09-16T14:50:50.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10c3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10c3fff640 10 monclient(hunting): my global_id is 164553 2024-09-16T14:50:50.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.925+0000 7f10c3fff640 1 -- 172.21.15.26:0/1339262950 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b8003530 con 0x7f10cc09b040 2024-09-16T14:50:50.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 1 -- 172.21.15.26:0/1339262950 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267672490 0 0) 0x7f10cc09b8f0 con 0x7f10cc09a550 2024-09-16T14:50:50.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient(hunting): my global_id is 154814 2024-09-16T14:50:50.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 1 -- 172.21.15.26:0/1339262950 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10cc09dc60 con 0x7f10cc09a550 2024-09-16T14:50:50.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 1 -- 172.21.15.26:0/1339262950 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535003252 0 0) 0x7f10cc099ae0 con 0x7f10cc09d1b0 2024-09-16T14:50:50.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient(hunting): my global_id is 184105 2024-09-16T14:50:50.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 1 -- 172.21.15.26:0/1339262950 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10cc09b8f0 con 0x7f10cc09d1b0 2024-09-16T14:50:50.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 1 -- 172.21.15.26:0/1339262950 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 718587243 0 0) 0x7f10b8003530 con 0x7f10cc09b040 2024-09-16T14:50:50.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 1 -- 172.21.15.26:0/1339262950 >> v1:172.21.15.26:6790/0 conn(0x7f10cc09a550 legacy=0x7f10cc09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 1 -- 172.21.15.26:0/1339262950 >> v1:172.21.15.189:6789/0 conn(0x7f10cc09d1b0 legacy=0x7f10cc136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:50:50.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 1 monclient: found mon.noname-a 2024-09-16T14:50:50.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:50.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:50.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 1 -- 172.21.15.26:0/1339262950 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc09bbd0 con 0x7f10cc09b040 2024-09-16T14:50:50.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient: _finish_auth 0 2024-09-16T14:50:50.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:50:50.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:50.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 1 -- 172.21.15.26:0/1339262950 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10c4001850 con 0x7f10cc09b040 2024-09-16T14:50:50.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:50.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient: dump: 2024-09-16T14:50:50.937 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.937 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.937 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.937 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.937 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.937 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.937 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.938 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.938 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.938 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 5 monclient: authenticate success, global_id 164553 2024-09-16T14:50:50.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:50.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 1 -- 172.21.15.26:0/1339262950 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10c4001a70 con 0x7f10cc09b040 2024-09-16T14:50:50.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:50.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 1 -- 172.21.15.26:0/1339262950 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10c4001e40 con 0x7f10cc09b040 2024-09-16T14:50:50.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:50.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 4 set_mon_vals no callback set 2024-09-16T14:50:50.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:50.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c3fff640 10 monclient: dump: 2024-09-16T14:50:50.939 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.939 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.939 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.939 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.939 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.940 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.940 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.940 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.940 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.940 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 10 monclient: shutdown 2024-09-16T14:50:50.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:50.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 1 -- 172.21.15.26:0/1339262950 >> v1:172.21.15.26:6789/0 conn(0x7f10cc09b040 legacy=0x7f10cc13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 1 -- 172.21.15.26:0/1339262950 shutdown_connections 2024-09-16T14:50:50.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 1 -- 172.21.15.26:0/1339262950 >> 172.21.15.26:0/1339262950 conn(0x7f10cc09fb50 msgr2=0x7f10cc09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:50.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 1 -- 172.21.15.26:0/1339262950 shutdown_connections 2024-09-16T14:50:50.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 1 -- 172.21.15.26:0/1339262950 wait complete. 2024-09-16T14:50:50.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 10 monclient: build_initial_monmap 2024-09-16T14:50:50.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 10 monclient: monmap: 2024-09-16T14:50:50.954 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:50.954 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:50.954 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:50.954 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:50.954 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:50.954 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:50.954 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:50.954 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:50.954 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:50.954 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 1 Processor -- start 2024-09-16T14:50:50.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 1 -- start start 2024-09-16T14:50:50.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 10 monclient: init 2024-09-16T14:50:50.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:50.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:50.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 10 monclient(hunting): picked mon.noname-c con 0x7f10cc09a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:50.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 10 monclient(hunting): picked mon.noname-a con 0x7f10cc09b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:50.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 10 monclient(hunting): picked mon.noname-b con 0x7f10cc09d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:50.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc0a25e0 con 0x7f10cc09b040 2024-09-16T14:50:50.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc09a240 con 0x7f10cc09d1b0 2024-09-16T14:50:50.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc06e320 con 0x7f10cc09a550 2024-09-16T14:50:50.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:50.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 10 monclient(hunting): authenticate will time out at 1475.054531s 2024-09-16T14:50:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d15ae640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f10cc09a550 0x7f10cc06c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33838/0 (socket says 172.21.15.26:33838) 2024-09-16T14:50:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d15ae640 1 -- 172.21.15.26:0/1131642572 learned_addr learned my addr 172.21.15.26:0/1131642572 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753898114 0 0) 0x7f10cc06e320 con 0x7f10cc09a550 2024-09-16T14:50:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): my global_id is 154817 2024-09-16T14:50:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b0003560 con 0x7f10cc09a550 2024-09-16T14:50:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433733662 0 0) 0x7f10cc09b8f0 con 0x7f10cc09b040 2024-09-16T14:50:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): my global_id is 164556 2024-09-16T14:50:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10cc06e320 con 0x7f10cc09b040 2024-09-16T14:50:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904505831 0 0) 0x7f10cc09a240 con 0x7f10cc09d1b0 2024-09-16T14:50:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): my global_id is 184108 2024-09-16T14:50:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10cc09b8f0 con 0x7f10cc09d1b0 2024-09-16T14:50:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2807211110 0 0) 0x7f10b0003560 con 0x7f10cc09a550 2024-09-16T14:50:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10cc09a240 con 0x7f10cc09a550 2024-09-16T14:50:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10c8001a00 con 0x7f10cc09a550 2024-09-16T14:50:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:50.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): dump: 2024-09-16T14:50:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.958 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3061811962 0 0) 0x7f10cc06e320 con 0x7f10cc09b040 2024-09-16T14:50:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10b0003560 con 0x7f10cc09b040 2024-09-16T14:50:50.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10bc001620 con 0x7f10cc09b040 2024-09-16T14:50:50.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:50.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): dump: 2024-09-16T14:50:50.959 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.959 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.959 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.959 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.959 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.959 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.959 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.959 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.959 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.960 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 54106611 0 0) 0x7f10cc09b8f0 con 0x7f10cc09d1b0 2024-09-16T14:50:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10cc06e320 con 0x7f10cc09d1b0 2024-09-16T14:50:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10c4001be0 con 0x7f10cc09d1b0 2024-09-16T14:50:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient(hunting): dump: 2024-09-16T14:50:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.960 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.961 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.961 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.961 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.961 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1896307361 0 0) 0x7f10cc09a240 con 0x7f10cc09a550 2024-09-16T14:50:50.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 >> v1:172.21.15.189:6789/0 conn(0x7f10cc09d1b0 legacy=0x7f10cc06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 >> v1:172.21.15.26:6789/0 conn(0x7f10cc09b040 legacy=0x7f10cc06cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:50:50.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 monclient: found mon.c 2024-09-16T14:50:50.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:50.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:50.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc11fca0 con 0x7f10cc09a550 2024-09-16T14:50:50.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:50:50.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:50.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:50.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 5 monclient: authenticate success, global_id 154817 2024-09-16T14:50:50.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 10 monclient: _renew_subs 2024-09-16T14:50:50.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:50.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10cc11f400 con 0x7f10cc09a550 2024-09-16T14:50:50.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 10 monclient: _renew_subs 2024-09-16T14:50:50.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:50.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10d3812640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10cc11df40 con 0x7f10cc09a550 2024-09-16T14:50:50.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10c8001c90 con 0x7f10cc09a550 2024-09-16T14:50:50.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:50.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10c8004c40 con 0x7f10cc09a550 2024-09-16T14:50:50.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:50.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c37fe640 4 set_mon_vals no callback set 2024-09-16T14:50:50.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.929+0000 7f10c1ffb640 10 monclient: dump: 2024-09-16T14:50:50.964 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.965 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.966 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10d3812640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10d3812640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f10ac006590 con 0x7f10ac002830 2024-09-16T14:50:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f10c800f6b0 con 0x7f10cc09a550 2024-09-16T14:50:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(244..244 src has 169..244) v4 ==== 7922+0+0 (unknown 1242468724 0 0) 0x7f10c8044240 con 0x7f10cc09a550 2024-09-16T14:50:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727966561 0 0) 0x7f10ac006590 con 0x7f10ac002830 2024-09-16T14:50:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 10 monclient(hunting): my global_id is 184111 2024-09-16T14:50:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10cc09a240 con 0x7f10ac002830 2024-09-16T14:50:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10d1daf640 10 monclient: get_auth_request con 0x7f10b0036790 auth_method 0 2024-09-16T14:50:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 946243139 0 0) 0x7f10cc09a240 con 0x7f10ac002830 2024-09-16T14:50:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10ac006590 con 0x7f10ac002830 2024-09-16T14:50:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10bc001cb0 con 0x7f10ac002830 2024-09-16T14:50:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:50.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 10 monclient: dump: 2024-09-16T14:50:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.969 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1926506320 0 0) 0x7f10ac006590 con 0x7f10ac002830 2024-09-16T14:50:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10ac00a0e0 con 0x7f10ac002830 2024-09-16T14:50:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f10bc0036b0 con 0x7f10ac002830 2024-09-16T14:50:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 >> v1:172.21.15.189:6789/0 conn(0x7f10ac002830 legacy=0x7f10ac004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.933+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 >> v1:172.21.15.189:6789/0 conn(0x7f10ac002830 legacy=0x7f10ac004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10d3812640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:50:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10d3812640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f10ac009da0 con 0x7f10ac0090c0 2024-09-16T14:50:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035241003 0 0) 0x7f10ac009da0 con 0x7f10ac0090c0 2024-09-16T14:50:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 10 monclient(hunting): my global_id is 184120 2024-09-16T14:50:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10ac006590 con 0x7f10ac0090c0 2024-09-16T14:50:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2905668053 0 0) 0x7f10ac006590 con 0x7f10ac0090c0 2024-09-16T14:50:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10ac009da0 con 0x7f10ac0090c0 2024-09-16T14:50:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10bc003d60 con 0x7f10ac0090c0 2024-09-16T14:50:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 10 monclient: dump: 2024-09-16T14:50:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.972 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2744168895 0 0) 0x7f10ac009da0 con 0x7f10ac0090c0 2024-09-16T14:50:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f10ac00c290 con 0x7f10ac0090c0 2024-09-16T14:50:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 788923264) 0x7f10bc004030 con 0x7f10ac0090c0 2024-09-16T14:50:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:50:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 >> v1:172.21.15.189:6789/0 conn(0x7f10ac0090c0 legacy=0x7f10ac009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10c1ffb640 1 -- 172.21.15.26:0/1131642572 >> v1:172.21.15.189:6789/0 conn(0x7f10ac0090c0 legacy=0x7f10ac009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:50.973 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:50:50.973 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:50.973 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:50.974 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:50:50.974 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 78, 2024-09-16T14:50:50.974 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:50:50.974 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:50:50.974 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:50:50.974 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:50:50.974 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:50.974 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:50:50.974 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:50.974 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:50:50.974 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:50:50.974 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:50.974 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:50.975 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:50.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:50.975 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:50.975 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:50.975 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:50.975 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:50.975 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:50.975 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:50.975 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:50.975 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:50:50.975 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:50:50.975 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:50.975 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:50.975 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:50.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:50.976 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:50.976 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:50.976 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:50.976 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:50.976 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:50.976 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:50.976 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.976 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:50:50.977 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:50:50.977 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:50:50.977 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:50:50.977 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:50:50.977 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:50:50.977 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:50.977 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:50.977 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:50:50.977 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:50:50.977 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:50:50.977 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:50:50.977 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:50:50.977 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:50:50.977 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:50:50.978 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:50.978 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:50:50.978 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:50.978 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:50.978 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:50.978 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:50.978 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:50.978 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:50.978 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:50.978 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:50.978 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:50.978 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:50.978 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:50.978 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:50:50.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.979 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:50:50.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.979 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:50.979 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:50.979 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:50.979 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:50.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.979 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:50.980 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:50:50.980 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:50.980 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.981 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.981 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:50:50.981 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:50:50.981 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:50.981 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:50.981 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:50.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.981 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:50.981 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:50.981 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:50.981 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:50.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.982 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:50.982 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:50:50.982 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:50.982 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.982 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.982 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:50:50.982 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:50:50.982 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:50.982 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:50.982 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:50.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:50.983 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:50.983 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:50.983 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:50.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.983 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:50.983 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:50:50.983 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:50.983 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.983 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.983 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:50:50.983 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:50:50.984 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:50.984 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:50.984 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:50.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.984 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.984 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:50:50.984 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:50:50.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.984 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:50.984 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:50.984 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:50.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.984 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:50.985 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:50:50.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:50.985 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:50.985 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:50.985 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:50:50.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:50.985 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:50.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:50.985 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:50:50.985 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:50:50.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10d3812640 1 -- 172.21.15.26:0/1131642572 >> v1:172.21.15.189:6812/3196161582 conn(0x7f10b0036790 legacy=0x7f10b0038c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10d3812640 10 monclient: shutdown 2024-09-16T14:50:50.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10d3812640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10d3812640 1 -- 172.21.15.26:0/1131642572 >> v1:172.21.15.26:6790/0 conn(0x7f10cc09a550 legacy=0x7f10cc06c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10d1daf640 1 -- 172.21.15.26:0/1131642572 reap_dead start 2024-09-16T14:50:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10d1daf640 1 -- 172.21.15.26:0/1131642572 reap_dead start 2024-09-16T14:50:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10d3812640 1 -- 172.21.15.26:0/1131642572 shutdown_connections 2024-09-16T14:50:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10d3812640 1 -- 172.21.15.26:0/1131642572 >> 172.21.15.26:0/1131642572 conn(0x7f10cc09fb50 msgr2=0x7f10cc09e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10d3812640 1 -- 172.21.15.26:0/1131642572 shutdown_connections 2024-09-16T14:50:50.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:50.945+0000 7f10d3812640 1 -- 172.21.15.26:0/1131642572 wait complete. 2024-09-16T14:50:50.989 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:50:50.989 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:50:50.990 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:50:50.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.932+0000 7fb8951cc640 10 monclient: get_monmap_and_config 2024-09-16T14:50:50.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.932+0000 7fb8951cc640 10 monclient: build_initial_monmap 2024-09-16T14:50:50.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.932+0000 7fb8951cc640 10 monclient: monmap: 2024-09-16T14:50:50.991 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:50.991 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.991 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:50.936527+0000 2024-09-16T14:50:50.992 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:50.936527+0000 2024-09-16T14:50:50.992 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:50.992 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:50.992 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:50.992 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:50.992 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:50.992 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:50.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 1 Processor -- start 2024-09-16T14:50:50.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 1 -- start start 2024-09-16T14:50:50.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 10 monclient: init 2024-09-16T14:50:50.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:50.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:50.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 10 monclient(hunting): picked mon.noname-a con 0x7fb890073590 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:50.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 10 monclient(hunting): picked mon.noname-b con 0x7fb8901348a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:50.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 10 monclient(hunting): picked mon.noname-c con 0x7fb8901383b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:50.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb890073970 con 0x7fb890073590 2024-09-16T14:50:50.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb890131ec0 con 0x7fb8901383b0 2024-09-16T14:50:50.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb890099710 con 0x7fb8901348a0 2024-09-16T14:50:50.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:50.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 10 monclient(hunting): authenticate will time out at 1433.053247s 2024-09-16T14:50:50.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88ed9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb890073590 0x7fb89009cab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.189:44584/0 (socket says 172.21.15.189:44584) 2024-09-16T14:50:50.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88f59e640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb8901383b0 0x7fb89013a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56812/0 (socket says 172.21.15.189:56812) 2024-09-16T14:50:50.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88ed9d640 1 -- 172.21.15.189:0/3437011259 learned_addr learned my addr 172.21.15.189:0/3437011259 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:50.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 1 -- 172.21.15.189:0/3437011259 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705369821 0 0) 0x7fb890131ec0 con 0x7fb8901383b0 2024-09-16T14:50:50.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient(hunting): my global_id is 184114 2024-09-16T14:50:50.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 1 -- 172.21.15.189:0/3437011259 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb878003530 con 0x7fb8901383b0 2024-09-16T14:50:50.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 1 -- 172.21.15.189:0/3437011259 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248666858 0 0) 0x7fb890099710 con 0x7fb8901348a0 2024-09-16T14:50:50.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient(hunting): my global_id is 154820 2024-09-16T14:50:50.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 1 -- 172.21.15.189:0/3437011259 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb890131ec0 con 0x7fb8901348a0 2024-09-16T14:50:50.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 1 -- 172.21.15.189:0/3437011259 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842940740 0 0) 0x7fb890073970 con 0x7fb890073590 2024-09-16T14:50:50.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:50:50.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:50.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:50.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient(hunting): my global_id is 164559 2024-09-16T14:50:50.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 1 -- 172.21.15.189:0/3437011259 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb890099710 con 0x7fb890073590 2024-09-16T14:50:50.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 1 -- 172.21.15.189:0/3437011259 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1045128255 0 0) 0x7fb878003530 con 0x7fb8901383b0 2024-09-16T14:50:50.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 1 -- 172.21.15.189:0/3437011259 >> v1:172.21.15.26:6790/0 conn(0x7fb8901348a0 legacy=0x7fb890136c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 1 -- 172.21.15.189:0/3437011259 >> v1:172.21.15.26:6789/0 conn(0x7fb890073590 legacy=0x7fb89009cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient: _finish_hunting 0 2024-09-16T14:50:50.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 1 monclient: found mon.noname-c 2024-09-16T14:50:50.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:50.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:50:50.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 1 -- 172.21.15.189:0/3437011259 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8900999f0 con 0x7fb8901383b0 2024-09-16T14:50:50.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient: _finish_auth 0 2024-09-16T14:50:50.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient: _check_auth_tickets 2024-09-16T14:50:50.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:50.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 1 -- 172.21.15.189:0/3437011259 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb8880019a0 con 0x7fb8901383b0 2024-09-16T14:50:50.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 5 monclient: authenticate success, global_id 184114 2024-09-16T14:50:50.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:50.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:50.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient: dump: 2024-09-16T14:50:50.996 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:50.996 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.996 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.996 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.996 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.996 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:50.997 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.997 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.997 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.997 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:50.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:50.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 1 -- 172.21.15.189:0/3437011259 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb888001c80 con 0x7fb8901383b0 2024-09-16T14:50:50.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:50.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:50.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 4 set_mon_vals no callback set 2024-09-16T14:50:50.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 1 -- 172.21.15.189:0/3437011259 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb888002010 con 0x7fb8901383b0 2024-09-16T14:50:50.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:50.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:50.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb88dd9b640 10 monclient: dump: 2024-09-16T14:50:50.998 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:50.998 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:50.998 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.998 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:50.998 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:50.998 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:50.998 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:50.998 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:50.998 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:50.998 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:50.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 10 monclient: shutdown 2024-09-16T14:50:50.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:50.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 1 -- 172.21.15.189:0/3437011259 >> v1:172.21.15.189:6789/0 conn(0x7fb8901383b0 legacy=0x7fb89013a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:50.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 1 -- 172.21.15.189:0/3437011259 shutdown_connections 2024-09-16T14:50:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 1 -- 172.21.15.189:0/3437011259 >> 172.21.15.189:0/3437011259 conn(0x7fb8900a0030 msgr2=0x7fb8900a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 1 -- 172.21.15.189:0/3437011259 shutdown_connections 2024-09-16T14:50:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 1 -- 172.21.15.189:0/3437011259 wait complete. 2024-09-16T14:50:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 10 monclient: build_initial_monmap 2024-09-16T14:50:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 10 monclient: monmap: 2024-09-16T14:50:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:51.000 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 1 Processor -- start 2024-09-16T14:50:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 1 -- start start 2024-09-16T14:50:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 10 monclient: init 2024-09-16T14:50:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:50:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.936+0000 7fb8951cc640 10 monclient(hunting): picked mon.noname-c con 0x7fb890073590 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 10 monclient(hunting): picked mon.noname-a con 0x7fb8901348a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 10 monclient(hunting): picked mon.noname-b con 0x7fb8901383b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8900717e0 con 0x7fb8901348a0 2024-09-16T14:50:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89009a550 con 0x7fb8901383b0 2024-09-16T14:50:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8900a13d0 con 0x7fb890073590 2024-09-16T14:50:51.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:51.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 10 monclient(hunting): authenticate will time out at 1433.056080s 2024-09-16T14:50:51.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb88f59e640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb8901383b0 0x7fb8901cde90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56826/0 (socket says 172.21.15.189:56826) 2024-09-16T14:50:51.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb88f59e640 1 -- 172.21.15.189:0/881696577 learned_addr learned my addr 172.21.15.189:0/881696577 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:51.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185254785 0 0) 0x7fb89009a550 con 0x7fb8901383b0 2024-09-16T14:50:51.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:51.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:51.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:51.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): my global_id is 184117 2024-09-16T14:50:51.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb87c003530 con 0x7fb8901383b0 2024-09-16T14:50:51.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158924450 0 0) 0x7fb8900a13d0 con 0x7fb890073590 2024-09-16T14:50:51.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:51.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:51.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:51.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): my global_id is 154823 2024-09-16T14:50:51.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb89009a550 con 0x7fb890073590 2024-09-16T14:50:51.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659730581 0 0) 0x7fb8900717e0 con 0x7fb8901348a0 2024-09-16T14:50:51.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:51.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:51.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:51.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): my global_id is 164562 2024-09-16T14:50:51.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8900a13d0 con 0x7fb8901348a0 2024-09-16T14:50:51.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1016382335 0 0) 0x7fb87c003530 con 0x7fb8901383b0 2024-09-16T14:50:51.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8900717e0 con 0x7fb8901383b0 2024-09-16T14:50:51.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb888001960 con 0x7fb8901383b0 2024-09-16T14:50:51.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:51.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:51.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): dump: 2024-09-16T14:50:51.003 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:51.003 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:51.003 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.003 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.003 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:51.003 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:51.003 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:51.003 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:51.003 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:51.003 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:51.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:51.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2605428765 0 0) 0x7fb8900a13d0 con 0x7fb8901348a0 2024-09-16T14:50:51.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb87c003530 con 0x7fb8901348a0 2024-09-16T14:50:51.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb8840018c0 con 0x7fb8901348a0 2024-09-16T14:50:51.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:51.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:51.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): dump: 2024-09-16T14:50:51.004 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:51.004 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:51.005 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.005 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.005 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:51.005 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:51.005 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:51.005 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:51.005 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:51.005 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:51.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1669866653 0 0) 0x7fb89009a550 con 0x7fb890073590 2024-09-16T14:50:51.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8900a13d0 con 0x7fb890073590 2024-09-16T14:50:51.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb8800017a0 con 0x7fb890073590 2024-09-16T14:50:51.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:51.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:51.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient(hunting): dump: 2024-09-16T14:50:51.006 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:51.006 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:51.006 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.006 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.006 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:51.006 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:51.006 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:51.006 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:51.006 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:51.006 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:51.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 203049904 0 0) 0x7fb8900717e0 con 0x7fb8901383b0 2024-09-16T14:50:51.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 >> v1:172.21.15.26:6790/0 conn(0x7fb890073590 legacy=0x7fb8901c5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:51.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 >> v1:172.21.15.26:6789/0 conn(0x7fb8901348a0 legacy=0x7fb8901ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:51.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:51.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 monclient: found mon.b 2024-09-16T14:50:51.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:51.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:51.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb890137fd0 con 0x7fb8901383b0 2024-09-16T14:50:51.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:51.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:51.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:51.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 5 monclient: authenticate success, global_id 184117 2024-09-16T14:50:51.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 10 monclient: _renew_subs 2024-09-16T14:50:51.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:51.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 1 -- 172.21.15.189:0/881696577 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb890071ac0 con 0x7fb8901383b0 2024-09-16T14:50:51.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 10 monclient: _renew_subs 2024-09-16T14:50:51.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:51.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 1 -- 172.21.15.189:0/881696577 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89006f760 con 0x7fb8901383b0 2024-09-16T14:50:51.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb888004ec0 con 0x7fb8901383b0 2024-09-16T14:50:51.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:51.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb8880051d0 con 0x7fb8901383b0 2024-09-16T14:50:51.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:51.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:51.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb88d59a640 4 set_mon_vals no callback set 2024-09-16T14:50:51.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 10 monclient: dump: 2024-09-16T14:50:51.008 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:51.009 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:51.009 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.009 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.009 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:51.009 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:51.009 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:51.009 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:51.009 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:51.009 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:51.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:51.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb88800fc40 con 0x7fb8901383b0 2024-09-16T14:50:51.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:51.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:51.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8951cc640 1 -- 172.21.15.189:0/881696577 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb870005050 con 0x7fb8901383b0 2024-09-16T14:50:51.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(244..244 src has 169..244) v4 ==== 7922+0+0 (unknown 1242468724 0 0) 0x7fb8880447d0 con 0x7fb8901383b0 2024-09-16T14:50:51.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.940+0000 7fb88ed9d640 10 monclient: get_auth_request con 0x7fb87c0364c0 auth_method 0 2024-09-16T14:50:51.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.944+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb888014b00 con 0x7fb8901383b0 2024-09-16T14:50:51.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.944+0000 7fb8777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:51.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:50.944+0000 7fb8777fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:51.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.108+0000 7fb8951cc640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T14:50:51.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.108+0000 7fb8951cc640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T14:50:51.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.112+0000 7fb8951cc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:51.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.112+0000 7fb8951cc640 1 -- 172.21.15.189:0/881696577 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h30"} v 0) v1 -- 0x7fb870005300 con 0x7fb8901383b0 2024-09-16T14:50:51.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient: get_monmap_and_config 2024-09-16T14:50:51.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient: build_initial_monmap 2024-09-16T14:50:51.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient: monmap: 2024-09-16T14:50:51.136 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:51.136 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:51.136 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:51.136671+0000 2024-09-16T14:50:51.136 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:51.136671+0000 2024-09-16T14:50:51.136 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:51.136 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:51.136 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:51.136 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:51.137 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:51.137 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:51.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 1 Processor -- start 2024-09-16T14:50:51.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 1 -- start start 2024-09-16T14:50:51.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient: init 2024-09-16T14:50:51.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:51.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:51.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient(hunting): picked mon.noname-b con 0x7fd42409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:51.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient(hunting): picked mon.noname-c con 0x7fd424132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:51.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient(hunting): picked mon.noname-a con 0x7fd424138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:51.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd42409d260 con 0x7fd424138ff0 2024-09-16T14:50:51.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd424132a80 con 0x7fd424132650 2024-09-16T14:50:51.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd42409cf60 con 0x7fd42409cb80 2024-09-16T14:50:51.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:51.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient(hunting): authenticate will time out at 1475.256568s 2024-09-16T14:50:51.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd422d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd42409cb80 0x7fd4240717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33846/0 (socket says 172.21.15.26:33846) 2024-09-16T14:50:51.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd422d9d640 1 -- 172.21.15.26:0/3424886566 learned_addr learned my addr 172.21.15.26:0/3424886566 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:51.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 1 -- 172.21.15.26:0/3424886566 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318638628 0 0) 0x7fd42409d260 con 0x7fd424138ff0 2024-09-16T14:50:51.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient(hunting): _negotiate 2024-09-16T14:50:51.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:51.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:51.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient(hunting): my global_id is 164565 2024-09-16T14:50:51.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 1 -- 172.21.15.26:0/3424886566 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd410003530 con 0x7fd424138ff0 2024-09-16T14:50:51.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 1 -- 172.21.15.26:0/3424886566 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514654442 0 0) 0x7fd42409cf60 con 0x7fd42409cb80 2024-09-16T14:50:51.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient(hunting): _negotiate 2024-09-16T14:50:51.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:51.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:51.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient(hunting): my global_id is 154826 2024-09-16T14:50:51.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 1 -- 172.21.15.26:0/3424886566 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd42409d260 con 0x7fd42409cb80 2024-09-16T14:50:51.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 1 -- 172.21.15.26:0/3424886566 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 899551652 0 0) 0x7fd410003530 con 0x7fd424138ff0 2024-09-16T14:50:51.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 1 -- 172.21.15.26:0/3424886566 >> v1:172.21.15.26:6790/0 conn(0x7fd42409cb80 legacy=0x7fd4240717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:51.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 1 -- 172.21.15.26:0/3424886566 >> v1:172.21.15.189:6789/0 conn(0x7fd424132650 legacy=0x7fd4241378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:51.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient: _finish_hunting 0 2024-09-16T14:50:51.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 1 monclient: found mon.noname-a 2024-09-16T14:50:51.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:51.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:51.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 1 -- 172.21.15.26:0/3424886566 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42409f2d0 con 0x7fd424138ff0 2024-09-16T14:50:51.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient: _finish_auth 0 2024-09-16T14:50:51.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient: _check_auth_tickets 2024-09-16T14:50:51.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:51.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 1 -- 172.21.15.26:0/3424886566 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd41c0019a0 con 0x7fd424138ff0 2024-09-16T14:50:51.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:51.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:51.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient: dump: 2024-09-16T14:50:51.146 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:51.146 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:51.146 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.146 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.146 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:51.146 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:51.146 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:51.146 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:51.146 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:51.146 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:51.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 5 monclient: authenticate success, global_id 164565 2024-09-16T14:50:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 1 -- 172.21.15.26:0/3424886566 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd41c001bd0 con 0x7fd424138ff0 2024-09-16T14:50:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 4 set_mon_vals no callback set 2024-09-16T14:50:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 1 -- 172.21.15.26:0/3424886566 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd41c001fa0 con 0x7fd424138ff0 2024-09-16T14:50:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd42259c640 10 monclient: dump: 2024-09-16T14:50:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:51.148 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:51.148 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:51.148 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:51.148 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:51.148 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:51.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient: shutdown 2024-09-16T14:50:51.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:51.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 1 -- 172.21.15.26:0/3424886566 >> v1:172.21.15.26:6789/0 conn(0x7fd424138ff0 legacy=0x7fd42413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:51.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 1 -- 172.21.15.26:0/3424886566 shutdown_connections 2024-09-16T14:50:51.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 1 -- 172.21.15.26:0/3424886566 >> 172.21.15.26:0/3424886566 conn(0x7fd42409fb50 msgr2=0x7fd42409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:51.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 1 -- 172.21.15.26:0/3424886566 shutdown_connections 2024-09-16T14:50:51.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 1 -- 172.21.15.26:0/3424886566 wait complete. 2024-09-16T14:50:51.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient: build_initial_monmap 2024-09-16T14:50:51.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient: monmap: 2024-09-16T14:50:51.149 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:51.149 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:51.149 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:51.149 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:51.149 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:51.149 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:51.149 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:51.149 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:51.149 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:51.149 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:51.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 1 Processor -- start 2024-09-16T14:50:51.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 1 -- start start 2024-09-16T14:50:51.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient: init 2024-09-16T14:50:51.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:51.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:51.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient(hunting): picked mon.noname-a con 0x7fd42409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:51.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient(hunting): picked mon.noname-c con 0x7fd424132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:51.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient(hunting): picked mon.noname-b con 0x7fd424138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:51.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd424071ac0 con 0x7fd42409cb80 2024-09-16T14:50:51.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd42409a240 con 0x7fd424138ff0 2024-09-16T14:50:51.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4240a29f0 con 0x7fd424132650 2024-09-16T14:50:51.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:51.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd429146640 10 monclient(hunting): authenticate will time out at 1475.259498s 2024-09-16T14:50:51.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd422d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd42409cb80 0x7fd42406dba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35576/0 (socket says 172.21.15.26:35576) 2024-09-16T14:50:51.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd422d9d640 1 -- 172.21.15.26:0/3543861270 learned_addr learned my addr 172.21.15.26:0/3543861270 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:51.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636481876 0 0) 0x7fd424071ac0 con 0x7fd42409cb80 2024-09-16T14:50:51.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd41b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:51.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd41b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:51.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd41b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:51.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd41b7fe640 10 monclient(hunting): my global_id is 164568 2024-09-16T14:50:51.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd404003530 con 0x7fd42409cb80 2024-09-16T14:50:51.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139639663 0 0) 0x7fd4240a29f0 con 0x7fd424132650 2024-09-16T14:50:51.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd41b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:51.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd41b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:51.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd41b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:51.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd41b7fe640 10 monclient(hunting): my global_id is 154829 2024-09-16T14:50:51.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.133+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd424071ac0 con 0x7fd424132650 2024-09-16T14:50:51.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4147546440 0 0) 0x7fd404003530 con 0x7fd42409cb80 2024-09-16T14:50:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4240a29f0 con 0x7fd42409cb80 2024-09-16T14:50:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd4140017e0 con 0x7fd42409cb80 2024-09-16T14:50:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient(hunting): dump: 2024-09-16T14:50:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:51.152 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:51.153 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:51.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:51.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 439741827 0 0) 0x7fd424071ac0 con 0x7fd424132650 2024-09-16T14:50:51.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd404003530 con 0x7fd424132650 2024-09-16T14:50:51.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd40c001840 con 0x7fd424132650 2024-09-16T14:50:51.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:51.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:51.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient(hunting): dump: 2024-09-16T14:50:51.153 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:51.153 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:51.153 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.153 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.153 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:51.154 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:51.154 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:51.154 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:51.154 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:51.154 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:51.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141071572 0 0) 0x7fd42409a240 con 0x7fd424138ff0 2024-09-16T14:50:51.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:50:51.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:51.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:51.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient(hunting): my global_id is 184126 2024-09-16T14:50:51.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd424071ac0 con 0x7fd424138ff0 2024-09-16T14:50:51.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 271167569 0 0) 0x7fd404003530 con 0x7fd424132650 2024-09-16T14:50:51.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 >> v1:172.21.15.189:6789/0 conn(0x7fd424138ff0 legacy=0x7fd4241cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:51.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 >> v1:172.21.15.26:6789/0 conn(0x7fd42409cb80 legacy=0x7fd42406dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:51.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:50:51.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 monclient: found mon.c 2024-09-16T14:50:51.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:51.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:51.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd424071050 con 0x7fd424132650 2024-09-16T14:50:51.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:50:51.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:50:51.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:51.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd429146640 5 monclient: authenticate success, global_id 154829 2024-09-16T14:50:51.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd429146640 10 monclient: _renew_subs 2024-09-16T14:50:51.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd429146640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:51.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd429146640 1 -- 172.21.15.26:0/3543861270 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd424071610 con 0x7fd424132650 2024-09-16T14:50:51.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd429146640 10 monclient: _renew_subs 2024-09-16T14:50:51.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd429146640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd429146640 1 -- 172.21.15.26:0/3543861270 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd424070be0 con 0x7fd424132650 2024-09-16T14:50:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd40c003620 con 0x7fd424132650 2024-09-16T14:50:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd40c0039b0 con 0x7fd424132650 2024-09-16T14:50:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 10 monclient: dump: 2024-09-16T14:50:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:51.156 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:51.157 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:51.157 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:51.157 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:51.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd421d9b640 4 set_mon_vals no callback set 2024-09-16T14:50:51.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd429146640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:51.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd429146640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:51.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd429146640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:51.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd429146640 1 -- 172.21.15.26:0/3543861270 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd408005050 con 0x7fd424132650 2024-09-16T14:50:51.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd40c003b10 con 0x7fd424132650 2024-09-16T14:50:51.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(244..244 src has 169..244) v4 ==== 7922+0+0 (unknown 1242468724 0 0) 0x7fd40c0430e0 con 0x7fd424132650 2024-09-16T14:50:51.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.137+0000 7fd422d9d640 10 monclient: get_auth_request con 0x7fd4040363a0 auth_method 0 2024-09-16T14:50:51.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.141+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd40c013580 con 0x7fd424132650 2024-09-16T14:50:51.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.141+0000 7fd41b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:51.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.141+0000 7fd41b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:51.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.292+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h30"}]=0 removed item id -106 name 'h30' from crush map v245) v1 ==== 121+0+0 (unknown 1670820584 0 0) 0x7fb8880131a0 con 0x7fb8901383b0 2024-09-16T14:50:51.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.292+0000 7fb8777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T14:50:51.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.292+0000 7fb8777fe640 10 monclient: _finish_command 2 = system:0 removed item id -106 name 'h30' from crush map 2024-09-16T14:50:51.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.365+0000 7fd429146640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:50:51.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.365+0000 7fd429146640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:50:51.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.365+0000 7fd429146640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:51.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.365+0000 7fd429146640 1 -- 172.21.15.26:0/3543861270 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fd408005b10 con 0x7fd424132650 2024-09-16T14:50:51.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.365+0000 7fd41b7fe640 1 -- 172.21.15.26:0/3543861270 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fd40c0119f0 con 0x7fd424132650 2024-09-16T14:50:51.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.365+0000 7fd41b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:50:51.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.365+0000 7fd41b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:51.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.369+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.369+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=11,logm=49} crc {auth=1965425179,config=692619927,health=2910473119,kv=1867599010,logm=2139276497}) 2024-09-16T14:50:51.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.369+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=423221643,config=692619927,health=2910473119,kv=3524478122,logm=3737553789}) 2024-09-16T14:50:51.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.369+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.369+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=11,logm=49} crc {auth=1965425179,config=692619927,health=2910473119,kv=1867599010,logm=2139276497}) 2024-09-16T14:50:51.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.369+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=12,kv=9,logm=52} crc {auth=3938642798,config=2435484489,health=2910473119,kv=2418557040,logm=3254701925}) 2024-09-16T14:50:51.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.369+0000 7fd429146640 1 -- 172.21.15.26:0/3543861270 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd4040363a0 legacy=0x7fd404038860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:51.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.369+0000 7fd429146640 10 monclient: shutdown 2024-09-16T14:50:51.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.369+0000 7fd429146640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:51.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.369+0000 7fd429146640 1 -- 172.21.15.26:0/3543861270 >> v1:172.21.15.26:6790/0 conn(0x7fd424132650 legacy=0x7fd42406e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:51.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.369+0000 7fd429146640 1 -- 172.21.15.26:0/3543861270 shutdown_connections 2024-09-16T14:50:51.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.369+0000 7fd429146640 1 -- 172.21.15.26:0/3543861270 >> 172.21.15.26:0/3543861270 conn(0x7fd42409fb50 msgr2=0x7fd424131bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:51.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.369+0000 7fd429146640 1 -- 172.21.15.26:0/3543861270 shutdown_connections 2024-09-16T14:50:51.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:51.369+0000 7fd429146640 1 -- 172.21.15.26:0/3543861270 wait complete. 2024-09-16T14:50:51.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.369+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.369+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2869605441}) 2024-09-16T14:50:51.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.369+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2185061364}) 2024-09-16T14:50:51.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.369+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.369+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2869605441}) 2024-09-16T14:50:51.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.369+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2402184316}) 2024-09-16T14:50:51.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.373+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.373+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1221690504}) 2024-09-16T14:50:51.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.373+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3482482830}) 2024-09-16T14:50:51.378 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.373+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.378 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.373+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1221690504}) 2024-09-16T14:50:51.378 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.373+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3330811361}) 2024-09-16T14:50:51.380 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.373+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.380 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.373+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=387101769}) 2024-09-16T14:50:51.380 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.373+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844886813}) 2024-09-16T14:50:51.380 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.373+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.380 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.373+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=387101769}) 2024-09-16T14:50:51.380 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.373+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2827999673}) 2024-09-16T14:50:51.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.377+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.377+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2462233626}) 2024-09-16T14:50:51.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.377+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085700119}) 2024-09-16T14:50:51.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.377+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.377+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2462233626}) 2024-09-16T14:50:51.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.377+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2385637810}) 2024-09-16T14:50:51.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.377+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.377+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=548427741}) 2024-09-16T14:50:51.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.377+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3414031367}) 2024-09-16T14:50:51.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.377+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.377+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=548427741}) 2024-09-16T14:50:51.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.377+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1348174385}) 2024-09-16T14:50:51.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2180198566}) 2024-09-16T14:50:51.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481960698}) 2024-09-16T14:50:51.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2180198566}) 2024-09-16T14:50:51.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=787342685}) 2024-09-16T14:50:51.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352670494}) 2024-09-16T14:50:51.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4147962064}) 2024-09-16T14:50:51.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352670494}) 2024-09-16T14:50:51.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1195771591}) 2024-09-16T14:50:51.388 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.388 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3890405884}) 2024-09-16T14:50:51.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3211519982}) 2024-09-16T14:50:51.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3890405884}) 2024-09-16T14:50:51.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.381+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4115778839}) 2024-09-16T14:50:51.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.385+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.385+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2116631348}) 2024-09-16T14:50:51.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.385+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1781607210}) 2024-09-16T14:50:51.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.385+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.385+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2116631348}) 2024-09-16T14:50:51.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.385+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2870223396}) 2024-09-16T14:50:51.391 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:50:51.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.385+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.385+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3079867130}) 2024-09-16T14:50:51.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.385+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1271884641}) 2024-09-16T14:50:51.393 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.385+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.393 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.385+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3079867130}) 2024-09-16T14:50:51.393 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.385+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=813699387}) 2024-09-16T14:50:51.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.389+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.389+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3693102138}) 2024-09-16T14:50:51.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.389+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4084304372}) 2024-09-16T14:50:51.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.389+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.389+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3693102138}) 2024-09-16T14:50:51.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.389+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1521047002}) 2024-09-16T14:50:51.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.389+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.389+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1873053163}) 2024-09-16T14:50:51.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.389+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1647509456}) 2024-09-16T14:50:51.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.389+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.389+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1873053163}) 2024-09-16T14:50:51.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.389+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1891804806}) 2024-09-16T14:50:51.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3188797517}) 2024-09-16T14:50:51.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=395585168}) 2024-09-16T14:50:51.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3188797517}) 2024-09-16T14:50:51.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2282563483}) 2024-09-16T14:50:51.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=661556351}) 2024-09-16T14:50:51.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=815336310}) 2024-09-16T14:50:51.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=661556351}) 2024-09-16T14:50:51.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3695212021}) 2024-09-16T14:50:51.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2465143045}) 2024-09-16T14:50:51.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2651270584}) 2024-09-16T14:50:51.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2465143045}) 2024-09-16T14:50:51.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.393+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1276521089}) 2024-09-16T14:50:51.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.397+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.397+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=871679158}) 2024-09-16T14:50:51.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.397+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494213009}) 2024-09-16T14:50:51.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.397+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.397+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=871679158}) 2024-09-16T14:50:51.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.397+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3482281674}) 2024-09-16T14:50:51.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.397+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.397+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794471647}) 2024-09-16T14:50:51.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.397+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334975331}) 2024-09-16T14:50:51.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.397+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.397+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794471647}) 2024-09-16T14:50:51.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.397+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3261058820}) 2024-09-16T14:50:51.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.401+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.401+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=164443070}) 2024-09-16T14:50:51.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.401+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053543242}) 2024-09-16T14:50:51.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.401+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.401+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=164443070}) 2024-09-16T14:50:51.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.401+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4160412632}) 2024-09-16T14:50:51.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.401+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.401+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2209137101}) 2024-09-16T14:50:51.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.401+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697030947}) 2024-09-16T14:50:51.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.401+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.401+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2209137101}) 2024-09-16T14:50:51.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.401+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1049084336}) 2024-09-16T14:50:51.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1678952935}) 2024-09-16T14:50:51.409 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1503776468}) 2024-09-16T14:50:51.409 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.409 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1678952935}) 2024-09-16T14:50:51.409 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2023351173}) 2024-09-16T14:50:51.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2423159662}) 2024-09-16T14:50:51.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2034346894}) 2024-09-16T14:50:51.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2423159662}) 2024-09-16T14:50:51.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1437714193}) 2024-09-16T14:50:51.411 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.411 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2077755511}) 2024-09-16T14:50:51.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3393224235}) 2024-09-16T14:50:51.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2077755511}) 2024-09-16T14:50:51.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3097977290}) 2024-09-16T14:50:51.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.412+0000 7fb8951cc640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T14:50:51.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.412+0000 7fb8951cc640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T14:50:51.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.412+0000 7fb8951cc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:51.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.412+0000 7fb8951cc640 1 -- 172.21.15.189:0/881696577 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h30"} v 0) v1 -- 0x7fb8700055a0 con 0x7fb8901383b0 2024-09-16T14:50:51.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1415625736}) 2024-09-16T14:50:51.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=796183343}) 2024-09-16T14:50:51.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1415625736}) 2024-09-16T14:50:51.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4061665912}) 2024-09-16T14:50:51.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1405397619}) 2024-09-16T14:50:51.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=606497757}) 2024-09-16T14:50:51.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1405397619}) 2024-09-16T14:50:51.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.409+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2919394596}) 2024-09-16T14:50:51.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.412+0000 7fb8777fe640 1 -- 172.21.15.189:0/881696577 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h30"}]=0 device 'h30' does not appear in the crush map v245) v1 ==== 120+0+0 (unknown 557577768 0 0) 0x7fb8880131a0 con 0x7fb8901383b0 2024-09-16T14:50:51.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.412+0000 7fb8777fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T14:50:51.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.412+0000 7fb8777fe640 10 monclient: _finish_command 3 = system:0 device 'h30' does not appear in the crush map 2024-09-16T14:50:51.419 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h30' does not appear in the crush map 2024-09-16T14:50:51.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.416+0000 7fb8951cc640 1 -- 172.21.15.189:0/881696577 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb87c0364c0 legacy=0x7fb87c038980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:51.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.416+0000 7fb8951cc640 10 monclient: shutdown 2024-09-16T14:50:51.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.416+0000 7fb8951cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:51.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.416+0000 7fb8951cc640 1 -- 172.21.15.189:0/881696577 >> v1:172.21.15.189:6789/0 conn(0x7fb8901383b0 legacy=0x7fb8901cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:51.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.416+0000 7fb8951cc640 1 -- 172.21.15.189:0/881696577 shutdown_connections 2024-09-16T14:50:51.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.416+0000 7fb8951cc640 1 -- 172.21.15.189:0/881696577 >> 172.21.15.189:0/881696577 conn(0x7fb8900a0030 msgr2=0x7fb89009bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:51.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.416+0000 7fb8951cc640 1 -- 172.21.15.189:0/881696577 shutdown_connections 2024-09-16T14:50:51.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.416+0000 7fb8951cc640 1 -- 172.21.15.189:0/881696577 wait complete. 2024-09-16T14:50:51.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3011822846}) 2024-09-16T14:50:51.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2060517078}) 2024-09-16T14:50:51.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3011822846}) 2024-09-16T14:50:51.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2300264866}) 2024-09-16T14:50:51.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=697276302}) 2024-09-16T14:50:51.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048351129}) 2024-09-16T14:50:51.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=697276302}) 2024-09-16T14:50:51.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1722966830}) 2024-09-16T14:50:51.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2893945001}) 2024-09-16T14:50:51.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3693690017}) 2024-09-16T14:50:51.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2893945001}) 2024-09-16T14:50:51.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.413+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2752441988}) 2024-09-16T14:50:51.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3294403891}) 2024-09-16T14:50:51.423 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921554506}) 2024-09-16T14:50:51.423 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.423 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3294403891}) 2024-09-16T14:50:51.423 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3117004952}) 2024-09-16T14:50:51.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2998359001}) 2024-09-16T14:50:51.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=66892353}) 2024-09-16T14:50:51.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2998359001}) 2024-09-16T14:50:51.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=697930123}) 2024-09-16T14:50:51.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2291290448}) 2024-09-16T14:50:51.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3406522042}) 2024-09-16T14:50:51.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2291290448}) 2024-09-16T14:50:51.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.417+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4218394388}) 2024-09-16T14:50:51.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=194289335}) 2024-09-16T14:50:51.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=171933939}) 2024-09-16T14:50:51.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=194289335}) 2024-09-16T14:50:51.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=367151739}) 2024-09-16T14:50:51.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3320351082}) 2024-09-16T14:50:51.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408136230}) 2024-09-16T14:50:51.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3320351082}) 2024-09-16T14:50:51.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2815804524}) 2024-09-16T14:50:51.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3380503087}) 2024-09-16T14:50:51.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1057984300}) 2024-09-16T14:50:51.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3380503087}) 2024-09-16T14:50:51.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.425+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2744647645}) 2024-09-16T14:50:51.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2101459838}) 2024-09-16T14:50:51.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590442610}) 2024-09-16T14:50:51.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2101459838}) 2024-09-16T14:50:51.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3095402508}) 2024-09-16T14:50:51.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2292606901}) 2024-09-16T14:50:51.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589647409}) 2024-09-16T14:50:51.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2292606901}) 2024-09-16T14:50:51.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2652016487}) 2024-09-16T14:50:51.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3149688234}) 2024-09-16T14:50:51.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93546551}) 2024-09-16T14:50:51.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3149688234}) 2024-09-16T14:50:51.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.429+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2251374559}) 2024-09-16T14:50:51.433 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:51.433 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h31 2024-09-16T14:50:51.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.433+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.433+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3360902128}) 2024-09-16T14:50:51.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.433+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917590627}) 2024-09-16T14:50:51.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.433+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.433+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3360902128}) 2024-09-16T14:50:51.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.433+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2427313451}) 2024-09-16T14:50:51.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.433+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.433+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=380135927}) 2024-09-16T14:50:51.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.433+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1111670313}) 2024-09-16T14:50:51.436 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.433+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.436 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.433+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=380135927}) 2024-09-16T14:50:51.436 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.433+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=325227269}) 2024-09-16T14:50:51.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=447002682}) 2024-09-16T14:50:51.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=363396509}) 2024-09-16T14:50:51.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=447002682}) 2024-09-16T14:50:51.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4214100201}) 2024-09-16T14:50:51.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126969753}) 2024-09-16T14:50:51.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1418136613}) 2024-09-16T14:50:51.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126969753}) 2024-09-16T14:50:51.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758432686}) 2024-09-16T14:50:51.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4017549008}) 2024-09-16T14:50:51.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4105453371}) 2024-09-16T14:50:51.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4017549008}) 2024-09-16T14:50:51.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.437+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2118842017}) 2024-09-16T14:50:51.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2149319148}) 2024-09-16T14:50:51.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3128680484}) 2024-09-16T14:50:51.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2149319148}) 2024-09-16T14:50:51.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1071844036}) 2024-09-16T14:50:51.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2641187153}) 2024-09-16T14:50:51.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=622798175}) 2024-09-16T14:50:51.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2641187153}) 2024-09-16T14:50:51.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=477064564}) 2024-09-16T14:50:51.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2883595608}) 2024-09-16T14:50:51.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672749861}) 2024-09-16T14:50:51.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2883595608}) 2024-09-16T14:50:51.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1994191132}) 2024-09-16T14:50:51.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=29} crc {logm=592172292,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=4185330790,mon_config_key=1508412840,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=390178219,osd_snap=3794734670,osdmap=2412873193}) 2024-09-16T14:50:51.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=28} crc {logm=4214175749,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=573805327,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=390178219,osd_snap=3794734670,osdmap=2134994193}) 2024-09-16T14:50:51.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=29} crc {logm=592172292,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=4185330790,mon_config_key=1508412840,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=390178219,osd_snap=3794734670,osdmap=2412873193}) 2024-09-16T14:50:51.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.441+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=29,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=31} crc {logm=823864091,mdsmap=1531151339,mgr=1291789651,mgr_metadata=2109538692,mgrstat=573805327,mon_config_key=953665057,monmap=3140514009,osd_metadata=3424249799,osd_pg_creating=390178219,osd_snap=3794734671,osdmap=3093877758}) 2024-09-16T14:50:51.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=99} crc {osdmap=2645884492}) 2024-09-16T14:50:51.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=773951418}) 2024-09-16T14:50:51.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:50:51.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=99} crc {osdmap=2645884492}) 2024-09-16T14:50:51.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:50:51.461+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=98} crc {osdmap=3282077846}) 2024-09-16T14:50:51.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.540+0000 7fad5e998640 10 monclient: get_monmap_and_config 2024-09-16T14:50:51.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.540+0000 7fad5e998640 10 monclient: build_initial_monmap 2024-09-16T14:50:51.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.540+0000 7fad5e998640 10 monclient: monmap: 2024-09-16T14:50:51.543 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:51.543 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:51.543 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:51.544482+0000 2024-09-16T14:50:51.543 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:51.544482+0000 2024-09-16T14:50:51.543 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:51.543 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:51.543 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:51.543 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:51.543 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:51.543 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:51.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.540+0000 7fad5e998640 1 Processor -- start 2024-09-16T14:50:51.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.540+0000 7fad5e998640 1 -- start start 2024-09-16T14:50:51.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.540+0000 7fad5e998640 10 monclient: init 2024-09-16T14:50:51.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:51.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:50:51.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient(hunting): picked mon.noname-a con 0x7fad580a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:51.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient(hunting): picked mon.noname-c con 0x7fad580a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:51.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient(hunting): picked mon.noname-b con 0x7fad5813b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:51.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad580a30b0 con 0x7fad580a1b00 2024-09-16T14:50:51.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad580a2e30 con 0x7fad580a25f0 2024-09-16T14:50:51.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5809eb50 con 0x7fad5813b430 2024-09-16T14:50:51.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:51.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient(hunting): authenticate will time out at 1433.660190s 2024-09-16T14:50:51.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad577fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fad580a25f0 0x7fad58139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56828/0 (socket says 172.21.15.189:56828) 2024-09-16T14:50:51.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad577fe640 1 -- 172.21.15.189:0/2741474582 learned_addr learned my addr 172.21.15.189:0/2741474582 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:51.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 1 -- 172.21.15.189:0/2741474582 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995306420 0 0) 0x7fad580a2e30 con 0x7fad580a25f0 2024-09-16T14:50:51.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:51.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:51.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:51.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 10 monclient(hunting): my global_id is 184129 2024-09-16T14:50:51.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 1 -- 172.21.15.189:0/2741474582 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad44003530 con 0x7fad580a25f0 2024-09-16T14:50:51.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 1 -- 172.21.15.189:0/2741474582 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3776466696 0 0) 0x7fad44003530 con 0x7fad580a25f0 2024-09-16T14:50:51.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 1 -- 172.21.15.189:0/2741474582 >> v1:172.21.15.26:6790/0 conn(0x7fad5813b430 legacy=0x7fad5813d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:51.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 1 -- 172.21.15.189:0/2741474582 >> v1:172.21.15.26:6789/0 conn(0x7fad580a1b00 legacy=0x7fad580a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:51.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:51.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 1 monclient: found mon.noname-c 2024-09-16T14:50:51.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:51.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:50:51.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 1 -- 172.21.15.189:0/2741474582 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5809ee30 con 0x7fad580a25f0 2024-09-16T14:50:51.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:51.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:51.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:51.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 1 -- 172.21.15.189:0/2741474582 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fad500019a0 con 0x7fad580a25f0 2024-09-16T14:50:51.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:51.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:51.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 10 monclient: dump: 2024-09-16T14:50:51.553 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:51.553 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:51.553 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.553 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.553 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:51.553 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:51.553 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:51.553 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:51.553 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:51.553 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:51.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 5 monclient: authenticate success, global_id 184129 2024-09-16T14:50:51.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:51.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 1 -- 172.21.15.189:0/2741474582 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad50001cc0 con 0x7fad580a25f0 2024-09-16T14:50:51.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:51.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 1 -- 172.21.15.189:0/2741474582 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fad50002050 con 0x7fad580a25f0 2024-09-16T14:50:51.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:51.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 4 set_mon_vals no callback set 2024-09-16T14:50:51.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:51.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:51.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad56ffd640 10 monclient: dump: 2024-09-16T14:50:51.554 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:51.554 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:51.554 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.555 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.555 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:51.555 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:51.556 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:51.556 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:51.556 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:51.556 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:51.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient: shutdown 2024-09-16T14:50:51.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:51.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 1 -- 172.21.15.189:0/2741474582 >> v1:172.21.15.189:6789/0 conn(0x7fad580a25f0 legacy=0x7fad58139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:51.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 1 -- 172.21.15.189:0/2741474582 shutdown_connections 2024-09-16T14:50:51.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 1 -- 172.21.15.189:0/2741474582 >> 172.21.15.189:0/2741474582 conn(0x7fad580a7dd0 msgr2=0x7fad580a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:51.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 1 -- 172.21.15.189:0/2741474582 shutdown_connections 2024-09-16T14:50:51.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 1 -- 172.21.15.189:0/2741474582 wait complete. 2024-09-16T14:50:51.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient: build_initial_monmap 2024-09-16T14:50:51.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient: monmap: 2024-09-16T14:50:51.557 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:51.557 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:51.557 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:51.557 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:51.557 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:51.557 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:51.557 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:51.558 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:51.558 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:51.558 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:51.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 1 Processor -- start 2024-09-16T14:50:51.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 1 -- start start 2024-09-16T14:50:51.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient: init 2024-09-16T14:50:51.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:51.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:51.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient(hunting): picked mon.noname-b con 0x7fad580a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:51.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient(hunting): picked mon.noname-c con 0x7fad580a25f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:51.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient(hunting): picked mon.noname-a con 0x7fad5813b430 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:51.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5811fd40 con 0x7fad5813b430 2024-09-16T14:50:51.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad58120020 con 0x7fad580a1b00 2024-09-16T14:50:51.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5811e890 con 0x7fad580a25f0 2024-09-16T14:50:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad5e998640 10 monclient(hunting): authenticate will time out at 1433.663182s 2024-09-16T14:50:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad57fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fad580a1b00 0x7fad581c9810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56838/0 (socket says 172.21.15.189:56838) 2024-09-16T14:50:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad57fff640 1 -- 172.21.15.189:0/1046950266 learned_addr learned my addr 172.21.15.189:0/1046950266 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479852660 0 0) 0x7fad58120020 con 0x7fad580a1b00 2024-09-16T14:50:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 10 monclient(hunting): my global_id is 184132 2024-09-16T14:50:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad38003530 con 0x7fad580a1b00 2024-09-16T14:50:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1059146143 0 0) 0x7fad38003530 con 0x7fad580a1b00 2024-09-16T14:50:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad58120020 con 0x7fad580a1b00 2024-09-16T14:50:51.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fad400018c0 con 0x7fad580a1b00 2024-09-16T14:50:51.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:51.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:51.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 10 monclient(hunting): dump: 2024-09-16T14:50:51.560 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:51.560 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:51.560 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.560 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.560 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:51.560 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:51.560 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:51.560 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:51.560 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:51.561 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:51.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:51.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438592848 0 0) 0x7fad5811e890 con 0x7fad580a25f0 2024-09-16T14:50:51.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:51.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:51.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:51.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 10 monclient(hunting): my global_id is 154835 2024-09-16T14:50:51.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad38003530 con 0x7fad580a25f0 2024-09-16T14:50:51.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484762098 0 0) 0x7fad5811fd40 con 0x7fad5813b430 2024-09-16T14:50:51.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:51.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:51.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:51.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 10 monclient(hunting): my global_id is 164574 2024-09-16T14:50:51.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.544+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad5811e890 con 0x7fad5813b430 2024-09-16T14:50:51.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3231726022 0 0) 0x7fad58120020 con 0x7fad580a1b00 2024-09-16T14:50:51.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 >> v1:172.21.15.26:6790/0 conn(0x7fad580a25f0 legacy=0x7fad581ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:51.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 >> v1:172.21.15.26:6789/0 conn(0x7fad5813b430 legacy=0x7fad581d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:51.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:50:51.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 1 monclient: found mon.b 2024-09-16T14:50:51.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:51.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:51.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5811e610 con 0x7fad580a1b00 2024-09-16T14:50:51.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 10 monclient: _finish_auth 0 2024-09-16T14:50:51.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:50:51.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:51.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad5e998640 5 monclient: authenticate success, global_id 184132 2024-09-16T14:50:51.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad5e998640 10 monclient: _renew_subs 2024-09-16T14:50:51.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad5e998640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:51.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad5e998640 1 -- 172.21.15.189:0/1046950266 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad580a5c80 con 0x7fad580a1b00 2024-09-16T14:50:51.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad5e998640 10 monclient: _renew_subs 2024-09-16T14:50:51.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad5e998640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:51.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad5e998640 1 -- 172.21.15.189:0/1046950266 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5811fa60 con 0x7fad580a1b00 2024-09-16T14:50:51.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad40003660 con 0x7fad580a1b00 2024-09-16T14:50:51.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:51.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fad40003970 con 0x7fad580a1b00 2024-09-16T14:50:51.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:51.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad567fc640 4 set_mon_vals no callback set 2024-09-16T14:50:51.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:51.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 10 monclient: dump: 2024-09-16T14:50:51.563 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:51.564 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:51.564 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.564 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:51.564 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:51.564 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:51.564 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:51.564 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:51.564 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:51.564 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:51.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad5e998640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:51.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad5e998640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:51.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad5e998640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:51.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad5e998640 1 -- 172.21.15.189:0/1046950266 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3c005050 con 0x7fad580a1b00 2024-09-16T14:50:51.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fad40003ad0 con 0x7fad580a1b00 2024-09-16T14:50:51.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(245..245 src has 169..245) v4 ==== 7709+0+0 (unknown 634301041 0 0) 0x7fad40044110 con 0x7fad580a1b00 2024-09-16T14:50:51.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.548+0000 7fad577fe640 10 monclient: get_auth_request con 0x7fad38036390 auth_method 0 2024-09-16T14:50:51.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.552+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fad400042a0 con 0x7fad580a1b00 2024-09-16T14:50:51.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.552+0000 7fad54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:51.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.552+0000 7fad54ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:51.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.704+0000 7fad5e998640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T14:50:51.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.704+0000 7fad5e998640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T14:50:51.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.704+0000 7fad5e998640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:51.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:51.704+0000 7fad5e998640 1 -- 172.21.15.189:0/1046950266 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h31"} v 0) v1 -- 0x7fad3c005300 con 0x7fad580a1b00 2024-09-16T14:50:52.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.300+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h31"}]=0 removed item id -109 name 'h31' from crush map v246) v1 ==== 121+0+0 (unknown 2939391554 0 0) 0x7fad40012aa0 con 0x7fad580a1b00 2024-09-16T14:50:52.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.304+0000 7fad54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T14:50:52.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.304+0000 7fad54ff9640 10 monclient: _finish_command 2 = system:0 removed item id -109 name 'h31' from crush map 2024-09-16T14:50:52.392 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:50:52.392 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:50:52.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.404+0000 7fad5e998640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T14:50:52.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.404+0000 7fad5e998640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T14:50:52.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.404+0000 7fad5e998640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:52.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.404+0000 7fad5e998640 1 -- 172.21.15.189:0/1046950266 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h31"} v 0) v1 -- 0x7fad3c0055a0 con 0x7fad580a1b00 2024-09-16T14:50:52.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.408+0000 7fad54ff9640 1 -- 172.21.15.189:0/1046950266 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h31"}]=0 device 'h31' does not appear in the crush map v246) v1 ==== 120+0+0 (unknown 2406817699 0 0) 0x7fad40012aa0 con 0x7fad580a1b00 2024-09-16T14:50:52.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.408+0000 7fad54ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T14:50:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.408+0000 7fad54ff9640 10 monclient: _finish_command 3 = system:0 device 'h31' does not appear in the crush map 2024-09-16T14:50:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h31' does not appear in the crush map 2024-09-16T14:50:52.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.408+0000 7fad5e998640 1 -- 172.21.15.189:0/1046950266 >> v1:172.21.15.189:6812/3196161582 conn(0x7fad38036390 legacy=0x7fad38038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.408+0000 7fad5e998640 10 monclient: shutdown 2024-09-16T14:50:52.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.408+0000 7fad5e998640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:52.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.408+0000 7fad5e998640 1 -- 172.21.15.189:0/1046950266 >> v1:172.21.15.189:6789/0 conn(0x7fad580a1b00 legacy=0x7fad581c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.408+0000 7fad5e998640 1 -- 172.21.15.189:0/1046950266 shutdown_connections 2024-09-16T14:50:52.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.408+0000 7fad5e998640 1 -- 172.21.15.189:0/1046950266 >> 172.21.15.189:0/1046950266 conn(0x7fad580a7dd0 msgr2=0x7fad5806b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:52.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.408+0000 7fad5e998640 1 -- 172.21.15.189:0/1046950266 shutdown_connections 2024-09-16T14:50:52.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.412+0000 7fad5e998640 1 -- 172.21.15.189:0/1046950266 wait complete. 2024-09-16T14:50:52.426 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T14:50:52.426 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h32 2024-09-16T14:50:52.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91bba23640 10 monclient: get_monmap_and_config 2024-09-16T14:50:52.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91bba23640 10 monclient: build_initial_monmap 2024-09-16T14:50:52.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91bba23640 10 monclient: monmap: 2024-09-16T14:50:52.522 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:52.522 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.522 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:52.523441+0000 2024-09-16T14:50:52.523 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:52.523441+0000 2024-09-16T14:50:52.523 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:52.523 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:52.523 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:52.523 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:52.523 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:52.523 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91bba23640 1 Processor -- start 2024-09-16T14:50:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91bba23640 1 -- start start 2024-09-16T14:50:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91bba23640 10 monclient: init 2024-09-16T14:50:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91bba23640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91bba23640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91bba23640 10 monclient(hunting): picked mon.noname-c con 0x7f91b409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91bba23640 10 monclient(hunting): picked mon.noname-b con 0x7f91b4132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91bba23640 10 monclient(hunting): picked mon.noname-a con 0x7f91b4138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91bba23640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b409d260 con 0x7f91b4138ff0 2024-09-16T14:50:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91bba23640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b4132a80 con 0x7f91b409cb80 2024-09-16T14:50:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91bba23640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b409cf60 con 0x7f91b4132650 2024-09-16T14:50:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91bba23640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91bba23640 10 monclient(hunting): authenticate will time out at 1476.643592s 2024-09-16T14:50:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91b9fc0640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f91b4138ff0 0x7f91b413b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35588/0 (socket says 172.21.15.26:35588) 2024-09-16T14:50:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91b8fbe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f91b4132650 0x7f91b41378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33864/0 (socket says 172.21.15.26:33864) 2024-09-16T14:50:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.521+0000 7f91b9fc0640 1 -- 172.21.15.26:0/3989463478 learned_addr learned my addr 172.21.15.26:0/3989463478 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 1 -- 172.21.15.26:0/3989463478 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483146759 0 0) 0x7f91b409cf60 con 0x7f91b4132650 2024-09-16T14:50:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient(hunting): my global_id is 154838 2024-09-16T14:50:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 1 -- 172.21.15.26:0/3989463478 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a0003530 con 0x7f91b4132650 2024-09-16T14:50:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 1 -- 172.21.15.26:0/3989463478 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945280386 0 0) 0x7f91b409d260 con 0x7f91b4138ff0 2024-09-16T14:50:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient(hunting): my global_id is 164577 2024-09-16T14:50:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 1 -- 172.21.15.26:0/3989463478 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b409cf60 con 0x7f91b4138ff0 2024-09-16T14:50:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 1 -- 172.21.15.26:0/3989463478 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215275112 0 0) 0x7f91b4132a80 con 0x7f91b409cb80 2024-09-16T14:50:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient(hunting): my global_id is 184135 2024-09-16T14:50:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 1 -- 172.21.15.26:0/3989463478 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b409d260 con 0x7f91b409cb80 2024-09-16T14:50:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 1 -- 172.21.15.26:0/3989463478 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2340093307 0 0) 0x7f91a0003530 con 0x7f91b4132650 2024-09-16T14:50:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 1 -- 172.21.15.26:0/3989463478 >> v1:172.21.15.189:6789/0 conn(0x7f91b409cb80 legacy=0x7f91b40717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 1 -- 172.21.15.26:0/3989463478 >> v1:172.21.15.26:6789/0 conn(0x7f91b4138ff0 legacy=0x7f91b413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:50:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 1 monclient: found mon.noname-b 2024-09-16T14:50:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 1 -- 172.21.15.26:0/3989463478 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b409f2d0 con 0x7f91b4132650 2024-09-16T14:50:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient: _finish_auth 0 2024-09-16T14:50:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:50:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 1 -- 172.21.15.26:0/3989463478 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f91b0001660 con 0x7f91b4132650 2024-09-16T14:50:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 5 monclient: authenticate success, global_id 154838 2024-09-16T14:50:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient: dump: 2024-09-16T14:50:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:52.534 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 1 -- 172.21.15.26:0/3989463478 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91b0001880 con 0x7f91b4132650 2024-09-16T14:50:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 1 -- 172.21.15.26:0/3989463478 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f91b0001bd0 con 0x7f91b4132650 2024-09-16T14:50:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 4 set_mon_vals no callback set 2024-09-16T14:50:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a7fff640 10 monclient: dump: 2024-09-16T14:50:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:52.536 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 10 monclient: shutdown 2024-09-16T14:50:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 1 -- 172.21.15.26:0/3989463478 >> v1:172.21.15.26:6790/0 conn(0x7f91b4132650 legacy=0x7f91b41378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 1 -- 172.21.15.26:0/3989463478 shutdown_connections 2024-09-16T14:50:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 1 -- 172.21.15.26:0/3989463478 >> 172.21.15.26:0/3989463478 conn(0x7f91b409fb50 msgr2=0x7f91b409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 1 -- 172.21.15.26:0/3989463478 shutdown_connections 2024-09-16T14:50:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 1 -- 172.21.15.26:0/3989463478 wait complete. 2024-09-16T14:50:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 10 monclient: build_initial_monmap 2024-09-16T14:50:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 10 monclient: monmap: 2024-09-16T14:50:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:52.537 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:52.537 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:52.537 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:52.537 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:52.537 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:52.537 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:52.537 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:52.537 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:52.537 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:52.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 1 Processor -- start 2024-09-16T14:50:52.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 1 -- start start 2024-09-16T14:50:52.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 10 monclient: init 2024-09-16T14:50:52.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:52.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:50:52.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 10 monclient(hunting): picked mon.noname-b con 0x7f91b409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:52.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 10 monclient(hunting): picked mon.noname-a con 0x7f91b4132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:52.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 10 monclient(hunting): picked mon.noname-c con 0x7f91b4138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:52.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b4133660 con 0x7f91b4132650 2024-09-16T14:50:52.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b4071a30 con 0x7f91b409cb80 2024-09-16T14:50:52.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b409a240 con 0x7f91b4138ff0 2024-09-16T14:50:52.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:52.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 10 monclient(hunting): authenticate will time out at 1476.646700s 2024-09-16T14:50:52.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91b8fbe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f91b4132650 0x7f91b412dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35602/0 (socket says 172.21.15.26:35602) 2024-09-16T14:50:52.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91b9fc0640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f91b4138ff0 0x7f91b412e440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33866/0 (socket says 172.21.15.26:33866) 2024-09-16T14:50:52.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91b8fbe640 1 -- 172.21.15.26:0/1517687330 learned_addr learned my addr 172.21.15.26:0/1517687330 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:52.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157556993 0 0) 0x7f91b409a240 con 0x7f91b4138ff0 2024-09-16T14:50:52.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient(hunting): my global_id is 154841 2024-09-16T14:50:52.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9198003530 con 0x7f91b4138ff0 2024-09-16T14:50:52.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15790499 0 0) 0x7f91b4133660 con 0x7f91b4132650 2024-09-16T14:50:52.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient(hunting): my global_id is 164580 2024-09-16T14:50:52.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b409a240 con 0x7f91b4132650 2024-09-16T14:50:52.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195898970 0 0) 0x7f91b4071a30 con 0x7f91b409cb80 2024-09-16T14:50:52.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient(hunting): my global_id is 184138 2024-09-16T14:50:52.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b4133660 con 0x7f91b409cb80 2024-09-16T14:50:52.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3148622668 0 0) 0x7f9198003530 con 0x7f91b4138ff0 2024-09-16T14:50:52.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91b4071a30 con 0x7f91b4138ff0 2024-09-16T14:50:52.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f91ac001d60 con 0x7f91b4138ff0 2024-09-16T14:50:52.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:52.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:52.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient(hunting): dump: 2024-09-16T14:50:52.540 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:52.540 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.540 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.540 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.540 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:52.541 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1106229887 0 0) 0x7f91b4071a30 con 0x7f91b4138ff0 2024-09-16T14:50:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 >> v1:172.21.15.189:6789/0 conn(0x7f91b409cb80 legacy=0x7f91b412d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 >> v1:172.21.15.26:6789/0 conn(0x7f91b4132650 legacy=0x7f91b412dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:50:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 1 monclient: found mon.c 2024-09-16T14:50:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b4133a80 con 0x7f91b4138ff0 2024-09-16T14:50:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient: _finish_auth 0 2024-09-16T14:50:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 5 monclient: authenticate success, global_id 154841 2024-09-16T14:50:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 10 monclient: _renew_subs 2024-09-16T14:50:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 1 -- 172.21.15.26:0/1517687330 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b41349f0 con 0x7f91b4138ff0 2024-09-16T14:50:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 10 monclient: _renew_subs 2024-09-16T14:50:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91bba23640 1 -- 172.21.15.26:0/1517687330 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b412f220 con 0x7f91b4138ff0 2024-09-16T14:50:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91ac004b70 con 0x7f91b4138ff0 2024-09-16T14:50:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f91ac004e80 con 0x7f91b4138ff0 2024-09-16T14:50:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a77fe640 4 set_mon_vals no callback set 2024-09-16T14:50:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.525+0000 7f91a5ffb640 10 monclient: dump: 2024-09-16T14:50:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:52.544 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.529+0000 7f91bba23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.529+0000 7f91bba23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.529+0000 7f91bba23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.529+0000 7f91bba23640 1 -- 172.21.15.26:0/1517687330 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9194005050 con 0x7f91b4138ff0 2024-09-16T14:50:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.529+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f91ac00f8f0 con 0x7f91b4138ff0 2024-09-16T14:50:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.529+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(246..246 src has 169..246) v4 ==== 7496+0+0 (unknown 771380485 0 0) 0x7f91ac044260 con 0x7f91b4138ff0 2024-09-16T14:50:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.529+0000 7f91b97bf640 10 monclient: get_auth_request con 0x7f9198036010 auth_method 0 2024-09-16T14:50:52.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.533+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f91ac010e90 con 0x7f91b4138ff0 2024-09-16T14:50:52.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.533+0000 7f91a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:52.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.533+0000 7f91a5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 10 monclient: get_monmap_and_config 2024-09-16T14:50:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 10 monclient: build_initial_monmap 2024-09-16T14:50:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 10 monclient: monmap: 2024-09-16T14:50:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:50:52.542665+0000 2024-09-16T14:50:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:50:52.542665+0000 2024-09-16T14:50:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:52.552 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 1 Processor -- start 2024-09-16T14:50:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 1 -- start start 2024-09-16T14:50:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 10 monclient: init 2024-09-16T14:50:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 10 monclient(hunting): picked mon.noname-c con 0x7ffb84099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 10 monclient(hunting): picked mon.noname-b con 0x7ffb841334d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 10 monclient(hunting): picked mon.noname-a con 0x7ffb84136fe0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb8409b350 con 0x7ffb84136fe0 2024-09-16T14:50:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb8409f2e0 con 0x7ffb84099a70 2024-09-16T14:50:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb8409b070 con 0x7ffb841334d0 2024-09-16T14:50:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8a40f640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffb84099a70 0x7ffb8409bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56852/0 (socket says 172.21.15.189:56852) 2024-09-16T14:50:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8a40f640 1 -- 172.21.15.189:0/1816667477 learned_addr learned my addr 172.21.15.189:0/1816667477 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 10 monclient(hunting): authenticate will time out at 1434.658431s 2024-09-16T14:50:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 1 -- 172.21.15.189:0/1816667477 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773071651 0 0) 0x7ffb8409f2e0 con 0x7ffb84099a70 2024-09-16T14:50:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 10 monclient(hunting): my global_id is 184141 2024-09-16T14:50:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb89c0e640 1 -- 172.21.15.189:0/1816667477 >> v1:172.21.15.26:6790/0 conn(0x7ffb841334d0 legacy=0x7ffb841358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T14:50:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb89c0e640 1 -- 172.21.15.189:0/1816667477 >> v1:172.21.15.26:6790/0 conn(0x7ffb841334d0 legacy=0x7ffb841358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T14:50:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 1 -- 172.21.15.189:0/1816667477 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb70003450 con 0x7ffb84099a70 2024-09-16T14:50:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb89c0e640 1 --1- 172.21.15.189:0/1816667477 >> v1:172.21.15.26:6790/0 conn(0x7ffb841334d0 0x7ffb841358c0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-09-16T14:50:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 1 -- 172.21.15.189:0/1816667477 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 238331006 0 0) 0x7ffb70003450 con 0x7ffb84099a70 2024-09-16T14:50:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 1 -- 172.21.15.189:0/1816667477 >> v1:172.21.15.26:6790/0 conn(0x7ffb841334d0 legacy=0x7ffb841358c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 1 -- 172.21.15.189:0/1816667477 >> v1:172.21.15.26:6789/0 conn(0x7ffb84136fe0 legacy=0x7ffb841393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 10 monclient: _finish_hunting 0 2024-09-16T14:50:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 1 monclient: found mon.noname-c 2024-09-16T14:50:52.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:52.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:50:52.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 1 -- 172.21.15.189:0/1816667477 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8409b630 con 0x7ffb84099a70 2024-09-16T14:50:52.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 10 monclient: _finish_auth 0 2024-09-16T14:50:52.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 10 monclient: _check_auth_tickets 2024-09-16T14:50:52.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:52.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 5 monclient: authenticate success, global_id 184141 2024-09-16T14:50:52.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 1 -- 172.21.15.189:0/1816667477 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb74001620 con 0x7ffb84099a70 2024-09-16T14:50:52.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:52.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:52.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 10 monclient: dump: 2024-09-16T14:50:52.555 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:52.555 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.556 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.556 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.556 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:52.556 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:52.556 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:52.556 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:52.556 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:52.556 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:52.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:52.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 1 -- 172.21.15.189:0/1816667477 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb74001910 con 0x7ffb84099a70 2024-09-16T14:50:52.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:52.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:52.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 1 -- 172.21.15.189:0/1816667477 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb74001ca0 con 0x7ffb84099a70 2024-09-16T14:50:52.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 4 set_mon_vals no callback set 2024-09-16T14:50:52.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:52.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:52.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8940d640 10 monclient: dump: 2024-09-16T14:50:52.557 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:52.557 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.557 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.557 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.557 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:52.557 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:52.557 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:52.557 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:52.557 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:52.558 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:52.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 10 monclient: shutdown 2024-09-16T14:50:52.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:52.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 1 -- 172.21.15.189:0/1816667477 >> v1:172.21.15.189:6789/0 conn(0x7ffb84099a70 legacy=0x7ffb8409bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 1 -- 172.21.15.189:0/1816667477 shutdown_connections 2024-09-16T14:50:52.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 1 -- 172.21.15.189:0/1816667477 >> 172.21.15.189:0/1816667477 conn(0x7ffb840a3320 msgr2=0x7ffb8409f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:52.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 1 -- 172.21.15.189:0/1816667477 shutdown_connections 2024-09-16T14:50:52.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.540+0000 7ffb8c673640 1 -- 172.21.15.189:0/1816667477 wait complete. 2024-09-16T14:50:52.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 monclient: build_initial_monmap 2024-09-16T14:50:52.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 monclient: monmap: 2024-09-16T14:50:52.558 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:50:52.558 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:52.559 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:50:52.559 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:50:52.559 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:52.559 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:50:52.559 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:52.559 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:52.559 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:52.559 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:52.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 1 Processor -- start 2024-09-16T14:50:52.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 1 -- start start 2024-09-16T14:50:52.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 monclient: init 2024-09-16T14:50:52.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:52.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:50:52.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 monclient(hunting): picked mon.noname-b con 0x7ffb84099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:52.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 monclient(hunting): picked mon.noname-c con 0x7ffb841334d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:52.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 monclient(hunting): picked mon.noname-a con 0x7ffb84136fe0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:52.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb84121b50 con 0x7ffb84136fe0 2024-09-16T14:50:52.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb841206a0 con 0x7ffb84099a70 2024-09-16T14:50:52.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb841363d0 con 0x7ffb841334d0 2024-09-16T14:50:52.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:52.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 monclient(hunting): authenticate will time out at 1434.660883s 2024-09-16T14:50:52.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8a40f640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffb84099a70 0x7ffb84132450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56866/0 (socket says 172.21.15.189:56866) 2024-09-16T14:50:52.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8a40f640 1 -- 172.21.15.189:0/3348107718 learned_addr learned my addr 172.21.15.189:0/3348107718 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:50:52.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336033820 0 0) 0x7ffb841206a0 con 0x7ffb84099a70 2024-09-16T14:50:52.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): my global_id is 184144 2024-09-16T14:50:52.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb68003530 con 0x7ffb84099a70 2024-09-16T14:50:52.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091103707 0 0) 0x7ffb84121b50 con 0x7ffb84136fe0 2024-09-16T14:50:52.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): my global_id is 164586 2024-09-16T14:50:52.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb841206a0 con 0x7ffb84136fe0 2024-09-16T14:50:52.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132029918 0 0) 0x7ffb841363d0 con 0x7ffb841334d0 2024-09-16T14:50:52.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): my global_id is 154844 2024-09-16T14:50:52.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb84121b50 con 0x7ffb841334d0 2024-09-16T14:50:52.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2548903977 0 0) 0x7ffb68003530 con 0x7ffb84099a70 2024-09-16T14:50:52.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb841363d0 con 0x7ffb84099a70 2024-09-16T14:50:52.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb740015e0 con 0x7ffb84099a70 2024-09-16T14:50:52.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:52.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:52.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): dump: 2024-09-16T14:50:52.562 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:52.562 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.562 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.562 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.562 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:52.563 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:52.563 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:52.563 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:52.563 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:52.563 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:52.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:52.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1466880218 0 0) 0x7ffb841206a0 con 0x7ffb84136fe0 2024-09-16T14:50:52.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb68003530 con 0x7ffb84136fe0 2024-09-16T14:50:52.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb7c001b80 con 0x7ffb84136fe0 2024-09-16T14:50:52.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:52.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:52.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient(hunting): dump: 2024-09-16T14:50:52.563 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:52.564 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.564 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.564 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.564 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:52.564 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:52.564 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:52.564 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:52.564 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:52.564 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:52.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 83586932 0 0) 0x7ffb841363d0 con 0x7ffb84099a70 2024-09-16T14:50:52.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 >> v1:172.21.15.26:6790/0 conn(0x7ffb841334d0 legacy=0x7ffb841ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 >> v1:172.21.15.26:6789/0 conn(0x7ffb84136fe0 legacy=0x7ffb841d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:52.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 monclient: found mon.b 2024-09-16T14:50:52.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:52.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:52.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb84121870 con 0x7ffb84099a70 2024-09-16T14:50:52.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient: _finish_auth 0 2024-09-16T14:50:52.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:52.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:52.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 5 monclient: authenticate success, global_id 184144 2024-09-16T14:50:52.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 monclient: _renew_subs 2024-09-16T14:50:52.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:52.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 1 -- 172.21.15.189:0/3348107718 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb84120420 con 0x7ffb84099a70 2024-09-16T14:50:52.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 monclient: _renew_subs 2024-09-16T14:50:52.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 1 -- 172.21.15.189:0/3348107718 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb84132d30 con 0x7ffb84099a70 2024-09-16T14:50:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb74004c00 con 0x7ffb84099a70 2024-09-16T14:50:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffb74004f10 con 0x7ffb84099a70 2024-09-16T14:50:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb88c0c640 4 set_mon_vals no callback set 2024-09-16T14:50:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 10 monclient: dump: 2024-09-16T14:50:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:50:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:52.567 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:50:52.567 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:52.567 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:52.567 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:52.567 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:50:52.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:52.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:52.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ffb7400f980 con 0x7ffb84099a70 2024-09-16T14:50:52.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:52.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb8c673640 1 -- 172.21.15.189:0/3348107718 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6c005050 con 0x7ffb84099a70 2024-09-16T14:50:52.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(246..246 src has 169..246) v4 ==== 7496+0+0 (unknown 771380485 0 0) 0x7ffb740442c0 con 0x7ffb84099a70 2024-09-16T14:50:52.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.544+0000 7ffb89c0e640 10 monclient: get_auth_request con 0x7ffb680361f0 auth_method 0 2024-09-16T14:50:52.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.548+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ffb74012de0 con 0x7ffb84099a70 2024-09-16T14:50:52.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.548+0000 7ffb7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:52.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.548+0000 7ffb7affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:52.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.712+0000 7ffb8c673640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T14:50:52.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.712+0000 7ffb8c673640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T14:50:52.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.712+0000 7ffb8c673640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:50:52.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:52.712+0000 7ffb8c673640 1 -- 172.21.15.189:0/3348107718 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h32"} v 0) v1 -- 0x7ffb6c005300 con 0x7ffb84099a70 2024-09-16T14:50:52.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.753+0000 7f91bba23640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:50:52.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.753+0000 7f91bba23640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:52.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.753+0000 7f91bba23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:52.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.753+0000 7f91bba23640 1 -- 172.21.15.26:0/1517687330 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9194005880 con 0x7f91b4138ff0 2024-09-16T14:50:52.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.753+0000 7f91a5ffb640 1 -- 172.21.15.26:0/1517687330 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 56855577) 0x7f91ac012f10 con 0x7f91b4138ff0 2024-09-16T14:50:52.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.753+0000 7f91a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:52.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.753+0000 7f91a5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:52.756 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:50:52.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.757+0000 7f91bba23640 1 -- 172.21.15.26:0/1517687330 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9198036010 legacy=0x7f91980384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.757+0000 7f91bba23640 10 monclient: shutdown 2024-09-16T14:50:52.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.757+0000 7f91bba23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:52.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.757+0000 7f91bba23640 1 -- 172.21.15.26:0/1517687330 >> v1:172.21.15.26:6790/0 conn(0x7f91b4138ff0 legacy=0x7f91b412e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.757+0000 7f91bba23640 1 -- 172.21.15.26:0/1517687330 shutdown_connections 2024-09-16T14:50:52.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.757+0000 7f91bba23640 1 -- 172.21.15.26:0/1517687330 >> 172.21.15.26:0/1517687330 conn(0x7f91b409fb50 msgr2=0x7f91b41388c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:52.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.757+0000 7f91bba23640 1 -- 172.21.15.26:0/1517687330 shutdown_connections 2024-09-16T14:50:52.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.757+0000 7f91bba23640 1 -- 172.21.15.26:0/1517687330 wait complete. 2024-09-16T14:50:52.774 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:50:52.774 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:50:52.774 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:50:52.774 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:50:52.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 10 monclient: get_monmap_and_config 2024-09-16T14:50:52.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 10 monclient: build_initial_monmap 2024-09-16T14:50:52.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 10 monclient: monmap: 2024-09-16T14:50:52.897 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:52.897 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.897 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:52.897975+0000 2024-09-16T14:50:52.897 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:52.897975+0000 2024-09-16T14:50:52.897 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:52.897 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:52.897 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:52.897 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:52.897 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:52.897 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:52.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 1 Processor -- start 2024-09-16T14:50:52.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 1 -- start start 2024-09-16T14:50:52.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 10 monclient: init 2024-09-16T14:50:52.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:52.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:50:52.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 10 monclient(hunting): picked mon.noname-a con 0x7fc66412e2a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:52.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 10 monclient(hunting): picked mon.noname-c con 0x7fc66406d070 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:52.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 10 monclient(hunting): picked mon.noname-b con 0x7fc664138f80 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:52.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6640a0e20 con 0x7fc66412e2a0 2024-09-16T14:50:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66409df00 con 0x7fc66406d070 2024-09-16T14:50:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6640702a0 con 0x7fc664138f80 2024-09-16T14:50:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 10 monclient(hunting): authenticate will time out at 1477.018061s 2024-09-16T14:50:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc663fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc66412e2a0 0x7fc66412e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35612/0 (socket says 172.21.15.26:35612) 2024-09-16T14:50:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc663fff640 1 -- 172.21.15.26:0/3755581071 learned_addr learned my addr 172.21.15.26:0/3755581071 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 1 -- 172.21.15.26:0/3755581071 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3241177902 0 0) 0x7fc6640a0e20 con 0x7fc66412e2a0 2024-09-16T14:50:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient(hunting): my global_id is 164589 2024-09-16T14:50:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 1 -- 172.21.15.26:0/3755581071 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc64c003530 con 0x7fc66412e2a0 2024-09-16T14:50:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 1 -- 172.21.15.26:0/3755581071 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914713263 0 0) 0x7fc6640702a0 con 0x7fc664138f80 2024-09-16T14:50:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient(hunting): my global_id is 154847 2024-09-16T14:50:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 1 -- 172.21.15.26:0/3755581071 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6640a0e20 con 0x7fc664138f80 2024-09-16T14:50:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 1 -- 172.21.15.26:0/3755581071 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 405193386 0 0) 0x7fc64c003530 con 0x7fc66412e2a0 2024-09-16T14:50:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 1 -- 172.21.15.26:0/3755581071 >> v1:172.21.15.26:6790/0 conn(0x7fc664138f80 legacy=0x7fc66413b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 1 -- 172.21.15.26:0/3755581071 >> v1:172.21.15.189:6789/0 conn(0x7fc66406d070 legacy=0x7fc664137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 1 monclient: found mon.noname-a 2024-09-16T14:50:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 1 -- 172.21.15.26:0/3755581071 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc664070580 con 0x7fc66412e2a0 2024-09-16T14:50:52.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:52.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:52.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:52.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 5 monclient: authenticate success, global_id 164589 2024-09-16T14:50:52.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:52.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 1 -- 172.21.15.26:0/3755581071 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc65c0019a0 con 0x7fc66412e2a0 2024-09-16T14:50:52.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:52.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:52.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient: dump: 2024-09-16T14:50:52.907 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:52.907 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.907 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.907 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.907 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:52.908 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:52.908 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:52.908 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:52.908 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:52.908 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:52.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:52.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 1 -- 172.21.15.26:0/3755581071 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc65c001c10 con 0x7fc66412e2a0 2024-09-16T14:50:52.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:52.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 1 -- 172.21.15.26:0/3755581071 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc65c001fe0 con 0x7fc66412e2a0 2024-09-16T14:50:52.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:52.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 4 set_mon_vals no callback set 2024-09-16T14:50:52.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:52.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc662ffd640 10 monclient: dump: 2024-09-16T14:50:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:52.909 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:52.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 10 monclient: shutdown 2024-09-16T14:50:52.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:52.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 1 -- 172.21.15.26:0/3755581071 >> v1:172.21.15.26:6789/0 conn(0x7fc66412e2a0 legacy=0x7fc66412e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 1 -- 172.21.15.26:0/3755581071 shutdown_connections 2024-09-16T14:50:52.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 1 -- 172.21.15.26:0/3755581071 >> 172.21.15.26:0/3755581071 conn(0x7fc6640a3dc0 msgr2=0x7fc6640a02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:52.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 1 -- 172.21.15.26:0/3755581071 shutdown_connections 2024-09-16T14:50:52.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.897+0000 7fc66a85e640 1 -- 172.21.15.26:0/3755581071 wait complete. 2024-09-16T14:50:52.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 10 monclient: build_initial_monmap 2024-09-16T14:50:52.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 10 monclient: monmap: 2024-09-16T14:50:52.910 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:52.910 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:52.910 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:52.910 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:52.910 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:52.911 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 1 Processor -- start 2024-09-16T14:50:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 1 -- start start 2024-09-16T14:50:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 10 monclient: init 2024-09-16T14:50:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:50:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 10 monclient(hunting): picked mon.noname-a con 0x7fc66406d070 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 10 monclient(hunting): picked mon.noname-b con 0x7fc66412e2a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 10 monclient(hunting): picked mon.noname-c con 0x7fc664138f80 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6640717f0 con 0x7fc66406d070 2024-09-16T14:50:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc664138c10 con 0x7fc66412e2a0 2024-09-16T14:50:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6641ca060 con 0x7fc664138f80 2024-09-16T14:50:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 10 monclient(hunting): authenticate will time out at 1477.021016s 2024-09-16T14:50:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc663fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc66406d070 0x7fc66412cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35622/0 (socket says 172.21.15.26:35622) 2024-09-16T14:50:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc663fff640 1 -- 172.21.15.26:0/1462219257 learned_addr learned my addr 172.21.15.26:0/1462219257 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978722457 0 0) 0x7fc6640717f0 con 0x7fc66406d070 2024-09-16T14:50:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): my global_id is 164592 2024-09-16T14:50:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc650003530 con 0x7fc66406d070 2024-09-16T14:50:52.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923026786 0 0) 0x7fc6641ca060 con 0x7fc664138f80 2024-09-16T14:50:52.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): my global_id is 154850 2024-09-16T14:50:52.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6640717f0 con 0x7fc664138f80 2024-09-16T14:50:52.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1884292454 0 0) 0x7fc6640717f0 con 0x7fc664138f80 2024-09-16T14:50:52.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6641ca060 con 0x7fc664138f80 2024-09-16T14:50:52.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1864110576 0 0) 0x7fc650003530 con 0x7fc66406d070 2024-09-16T14:50:52.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6640717f0 con 0x7fc66406d070 2024-09-16T14:50:52.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc6540018c0 con 0x7fc664138f80 2024-09-16T14:50:52.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:52.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:52.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): dump: 2024-09-16T14:50:52.914 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:52.914 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.914 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.914 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.914 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:52.914 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:52.914 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:52.914 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:52.914 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:52.914 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:52.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:52.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc65c001f20 con 0x7fc66406d070 2024-09-16T14:50:52.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:52.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:52.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): dump: 2024-09-16T14:50:52.915 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:52.915 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.915 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.915 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.915 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:52.915 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:52.915 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:52.915 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:52.915 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:52.915 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:52.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746249498 0 0) 0x7fc658001240 con 0x7fc66412e2a0 2024-09-16T14:50:52.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): my global_id is 184150 2024-09-16T14:50:52.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc650003530 con 0x7fc66412e2a0 2024-09-16T14:50:52.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1411656056 0 0) 0x7fc6641ca060 con 0x7fc664138f80 2024-09-16T14:50:52.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 >> v1:172.21.15.189:6789/0 conn(0x7fc66412e2a0 legacy=0x7fc6641ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 >> v1:172.21.15.26:6789/0 conn(0x7fc66406d070 legacy=0x7fc66412cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:50:52.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 monclient: found mon.c 2024-09-16T14:50:52.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:52.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:52.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc664071f70 con 0x7fc664138f80 2024-09-16T14:50:52.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient: _finish_auth 0 2024-09-16T14:50:52.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:50:52.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:52.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 5 monclient: authenticate success, global_id 154850 2024-09-16T14:50:52.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 10 monclient: _renew_subs 2024-09-16T14:50:52.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:52.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 1 -- 172.21.15.26:0/1462219257 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc664119e10 con 0x7fc664138f80 2024-09-16T14:50:52.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 10 monclient: _renew_subs 2024-09-16T14:50:52.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:52.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 1 -- 172.21.15.26:0/1462219257 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc66411a0d0 con 0x7fc664138f80 2024-09-16T14:50:52.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc654003620 con 0x7fc664138f80 2024-09-16T14:50:52.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:52.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc654003930 con 0x7fc664138f80 2024-09-16T14:50:52.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:52.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc6627fc640 4 set_mon_vals no callback set 2024-09-16T14:50:52.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:52.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient: dump: 2024-09-16T14:50:52.918 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:52.918 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.918 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.918 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.918 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:52.918 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:52.918 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:52.918 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:52.918 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:52.918 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:52.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:52.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc66a85e640 1 -- 172.21.15.26:0/1462219257 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc644006590 con 0x7fc644002830 2024-09-16T14:50:52.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc65400e3a0 con 0x7fc664138f80 2024-09-16T14:50:52.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(246..246 src has 169..246) v4 ==== 7496+0+0 (unknown 771380485 0 0) 0x7fc654042d60 con 0x7fc664138f80 2024-09-16T14:50:52.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918507156 0 0) 0x7fc644006590 con 0x7fc644002830 2024-09-16T14:50:52.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient(hunting): my global_id is 164595 2024-09-16T14:50:52.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6641ca060 con 0x7fc644002830 2024-09-16T14:50:52.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc6637fe640 10 monclient: get_auth_request con 0x7fc6500363d0 auth_method 0 2024-09-16T14:50:52.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 7537819 0 0) 0x7fc6641ca060 con 0x7fc644002830 2024-09-16T14:50:52.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc644006590 con 0x7fc644002830 2024-09-16T14:50:52.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc65c005120 con 0x7fc644002830 2024-09-16T14:50:52.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:52.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:52.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 10 monclient: dump: 2024-09-16T14:50:52.920 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:52.920 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.920 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.920 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.920 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:52.920 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:52.920 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:52.920 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:52.920 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:52.920 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:52.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3224047512 0 0) 0x7fc644006590 con 0x7fc644002830 2024-09-16T14:50:52.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.901+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc64400a0e0 con 0x7fc644002830 2024-09-16T14:50:52.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.905+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc65c007500 con 0x7fc644002830 2024-09-16T14:50:52.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.905+0000 7fc660ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:52.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.905+0000 7fc660ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:52.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.905+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 >> v1:172.21.15.26:6789/0 conn(0x7fc644002830 legacy=0x7fc644004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.905+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 >> v1:172.21.15.26:6789/0 conn(0x7fc644002830 legacy=0x7fc644004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:52.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc66a85e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:50:52.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc66a85e640 1 -- 172.21.15.26:0/1462219257 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc644009da0 con 0x7fc6440090c0 2024-09-16T14:50:52.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266661786 0 0) 0x7fc644009da0 con 0x7fc6440090c0 2024-09-16T14:50:52.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:50:52.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:52.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:52.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 10 monclient(hunting): my global_id is 164598 2024-09-16T14:50:52.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc644006590 con 0x7fc6440090c0 2024-09-16T14:50:52.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2621317012 0 0) 0x7fc644006590 con 0x7fc6440090c0 2024-09-16T14:50:52.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc644009da0 con 0x7fc6440090c0 2024-09-16T14:50:52.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc65c005970 con 0x7fc6440090c0 2024-09-16T14:50:52.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:52.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:52.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 10 monclient: dump: 2024-09-16T14:50:52.923 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:52.924 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1896837664 0 0) 0x7fc644009da0 con 0x7fc6440090c0 2024-09-16T14:50:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc64400c290 con 0x7fc6440090c0 2024-09-16T14:50:52.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 712407666) 0x7fc65c006de0 con 0x7fc6440090c0 2024-09-16T14:50:52.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:50:52.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:52.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 >> v1:172.21.15.26:6789/0 conn(0x7fc6440090c0 legacy=0x7fc644009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc660ff9640 1 -- 172.21.15.26:0/1462219257 >> v1:172.21.15.26:6789/0 conn(0x7fc6440090c0 legacy=0x7fc644009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:52.925 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:50:52.925 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:52.925 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:52.925 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:50:52.925 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 78, 2024-09-16T14:50:52.925 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:50:52.926 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:50:52.926 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:50:52.926 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:50:52.926 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:52.926 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:50:52.926 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:52.926 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:50:52.926 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:50:52.926 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:52.926 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:52.926 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:52.926 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:52.926 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:52.927 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:52.927 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:52.927 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:52.927 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:52.927 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:52.927 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:52.927 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:50:52.927 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:50:52.927 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:52.927 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:52.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:52.927 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:52.927 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:52.927 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:52.928 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:52.928 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:52.928 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:52.928 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:52.928 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:52.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:52.928 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:50:52.928 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:50:52.928 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:50:52.928 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:50:52.928 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:50:52.928 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:50:52.928 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:52.928 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:52.929 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:50:52.929 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:50:52.929 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:50:52.929 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:50:52.929 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:50:52.929 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:50:52.929 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:50:52.929 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:52.929 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:50:52.929 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:52.929 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:52.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:52.929 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:52.930 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:52.930 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:52.930 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:52.930 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:52.930 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:52.930 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:52.930 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:52.930 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:50:52.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:52.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:50:52.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:52.930 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:52.930 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:52.930 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:52.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:52.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:52.931 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:52.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:50:52.931 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:52.931 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:52.931 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:52.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:52.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:50:52.931 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:50:52.931 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:52.931 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:52.931 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:52.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:52.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:52.932 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:52.932 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:52.932 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:52.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:52.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:52.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:52.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:50:52.932 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:52.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:52.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:52.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:52.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:50:52.933 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:50:52.933 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:52.933 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:52.933 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:52.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:52.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:52.933 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:52.933 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:52.933 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:52.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:52.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:52.933 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:52.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:50:52.934 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:52.934 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:52.934 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:52.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:52.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:50:52.934 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:50:52.934 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:52.934 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:52.934 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:52.934 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:52.934 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:52.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:52.934 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:50:52.934 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:50:52.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:52.935 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:52.935 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:52.935 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:52.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:52.935 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:52.935 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:50:52.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:52.935 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:52.935 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:52.935 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:50:52.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:52.935 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:52.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:50:52.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:52.936 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:52.936 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:52.936 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:52.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:52.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:52.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:52.936 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:50:52.936 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:50:52.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc66a85e640 1 -- 172.21.15.26:0/1462219257 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc6500363d0 legacy=0x7fc650038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc66a85e640 10 monclient: shutdown 2024-09-16T14:50:52.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc66a85e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:52.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc66a85e640 1 -- 172.21.15.26:0/1462219257 >> v1:172.21.15.26:6790/0 conn(0x7fc664138f80 legacy=0x7fc6641d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:52.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc668dfb640 1 -- 172.21.15.26:0/1462219257 reap_dead start 2024-09-16T14:50:52.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc668dfb640 1 -- 172.21.15.26:0/1462219257 reap_dead start 2024-09-16T14:50:52.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc66a85e640 1 -- 172.21.15.26:0/1462219257 shutdown_connections 2024-09-16T14:50:52.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc66a85e640 1 -- 172.21.15.26:0/1462219257 >> 172.21.15.26:0/1462219257 conn(0x7fc6640a3dc0 msgr2=0x7fc66412f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:52.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc66a85e640 1 -- 172.21.15.26:0/1462219257 shutdown_connections 2024-09-16T14:50:52.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:52.909+0000 7fc66a85e640 1 -- 172.21.15.26:0/1462219257 wait complete. 2024-09-16T14:50:52.939 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:50:52.939 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:50:52.939 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:50:53.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 10 monclient: get_monmap_and_config 2024-09-16T14:50:53.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 10 monclient: build_initial_monmap 2024-09-16T14:50:53.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 10 monclient: monmap: 2024-09-16T14:50:53.061 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:53.061 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.061 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:53.062278+0000 2024-09-16T14:50:53.061 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:53.062278+0000 2024-09-16T14:50:53.061 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:53.062 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.062 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:53.062 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:53.062 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:53.062 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 1 Processor -- start 2024-09-16T14:50:53.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 1 -- start start 2024-09-16T14:50:53.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 10 monclient: init 2024-09-16T14:50:53.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:53.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:50:53.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 10 monclient(hunting): picked mon.noname-a con 0x7fe5300a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:53.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 10 monclient(hunting): picked mon.noname-b con 0x7fe5300a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:53.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 10 monclient(hunting): picked mon.noname-c con 0x7fe5300a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:53.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5300a3250 con 0x7fe5300a1d10 2024-09-16T14:50:53.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53009f600 con 0x7fe5300a2800 2024-09-16T14:50:53.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5300a3830 con 0x7fe5300a0000 2024-09-16T14:50:53.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:53.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 10 monclient(hunting): authenticate will time out at 1477.182405s 2024-09-16T14:50:53.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52e59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe5300a1d10 0x7fe5300a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35648/0 (socket says 172.21.15.26:35648) 2024-09-16T14:50:53.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52e59c640 1 -- 172.21.15.26:0/4060993130 learned_addr learned my addr 172.21.15.26:0/4060993130 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:53.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 1 -- 172.21.15.26:0/4060993130 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150567917 0 0) 0x7fe5300a3250 con 0x7fe5300a1d10 2024-09-16T14:50:53.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient(hunting): my global_id is 164601 2024-09-16T14:50:53.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 1 -- 172.21.15.26:0/4060993130 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe51c003530 con 0x7fe5300a1d10 2024-09-16T14:50:53.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 1 -- 172.21.15.26:0/4060993130 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806513679 0 0) 0x7fe5300a3830 con 0x7fe5300a0000 2024-09-16T14:50:53.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient(hunting): my global_id is 154853 2024-09-16T14:50:53.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 1 -- 172.21.15.26:0/4060993130 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5300a3250 con 0x7fe5300a0000 2024-09-16T14:50:53.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 1 -- 172.21.15.26:0/4060993130 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2291856766 0 0) 0x7fe51c003530 con 0x7fe5300a1d10 2024-09-16T14:50:53.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 1 -- 172.21.15.26:0/4060993130 >> v1:172.21.15.26:6790/0 conn(0x7fe5300a0000 legacy=0x7fe530139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 1 -- 172.21.15.26:0/4060993130 >> v1:172.21.15.189:6789/0 conn(0x7fe5300a2800 legacy=0x7fe53013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient: _finish_hunting 0 2024-09-16T14:50:53.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 1 monclient: found mon.noname-a 2024-09-16T14:50:53.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:53.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:53.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 1 -- 172.21.15.26:0/4060993130 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53009ed60 con 0x7fe5300a1d10 2024-09-16T14:50:53.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient: _finish_auth 0 2024-09-16T14:50:53.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient: _check_auth_tickets 2024-09-16T14:50:53.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:53.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 5 monclient: authenticate success, global_id 164601 2024-09-16T14:50:53.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 1 -- 172.21.15.26:0/4060993130 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe5200019a0 con 0x7fe5300a1d10 2024-09-16T14:50:53.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:53.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:53.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient: dump: 2024-09-16T14:50:53.073 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.073 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.073 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.073 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.073 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.073 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.073 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.073 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.073 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.074 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:53.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 1 -- 172.21.15.26:0/4060993130 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe520001c80 con 0x7fe5300a1d10 2024-09-16T14:50:53.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:53.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 1 -- 172.21.15.26:0/4060993130 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe520002050 con 0x7fe5300a1d10 2024-09-16T14:50:53.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:53.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe52d59a640 10 monclient: dump: 2024-09-16T14:50:53.074 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.074 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.074 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.074 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.074 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.075 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.075 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.075 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.075 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.075 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:53.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 4 set_mon_vals no callback set 2024-09-16T14:50:53.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 10 monclient: shutdown 2024-09-16T14:50:53.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:53.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.061+0000 7fe534f7d640 1 -- 172.21.15.26:0/4060993130 >> v1:172.21.15.26:6789/0 conn(0x7fe5300a1d10 legacy=0x7fe5300a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 1 -- 172.21.15.26:0/4060993130 shutdown_connections 2024-09-16T14:50:53.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 1 -- 172.21.15.26:0/4060993130 >> 172.21.15.26:0/4060993130 conn(0x7fe5300a7f30 msgr2=0x7fe5300a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:53.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 1 -- 172.21.15.26:0/4060993130 shutdown_connections 2024-09-16T14:50:53.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 1 -- 172.21.15.26:0/4060993130 wait complete. 2024-09-16T14:50:53.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 10 monclient: build_initial_monmap 2024-09-16T14:50:53.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 10 monclient: monmap: 2024-09-16T14:50:53.076 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:53.076 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:53.076 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:53.076 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:53.076 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:53.076 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:53.076 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:53.076 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:53.076 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:53.076 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 1 Processor -- start 2024-09-16T14:50:53.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 1 -- start start 2024-09-16T14:50:53.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 10 monclient: init 2024-09-16T14:50:53.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:53.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:53.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 10 monclient(hunting): picked mon.noname-c con 0x7fe5300a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:53.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 10 monclient(hunting): picked mon.noname-b con 0x7fe5300a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:53.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 10 monclient(hunting): picked mon.noname-a con 0x7fe5300a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:53.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe530133f00 con 0x7fe5300a2800 2024-09-16T14:50:53.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe530132920 con 0x7fe5300a1d10 2024-09-16T14:50:53.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5300a6a00 con 0x7fe5300a0000 2024-09-16T14:50:53.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:53.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 10 monclient(hunting): authenticate will time out at 1477.185617s 2024-09-16T14:50:53.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe52e59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe5300a0000 0x7fe530134320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33902/0 (socket says 172.21.15.26:33902) 2024-09-16T14:50:53.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe52e59c640 1 -- 172.21.15.26:0/2391884282 learned_addr learned my addr 172.21.15.26:0/2391884282 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:53.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736673974 0 0) 0x7fe5300a6a00 con 0x7fe5300a0000 2024-09-16T14:50:53.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): my global_id is 154856 2024-09-16T14:50:53.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe510003530 con 0x7fe5300a0000 2024-09-16T14:50:53.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639178010 0 0) 0x7fe530133f00 con 0x7fe5300a2800 2024-09-16T14:50:53.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): my global_id is 164604 2024-09-16T14:50:53.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5300a6a00 con 0x7fe5300a2800 2024-09-16T14:50:53.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1062370808 0 0) 0x7fe510003530 con 0x7fe5300a0000 2024-09-16T14:50:53.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe530133f00 con 0x7fe5300a0000 2024-09-16T14:50:53.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe520001f90 con 0x7fe5300a0000 2024-09-16T14:50:53.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:53.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): dump: 2024-09-16T14:50:53.079 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.079 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.079 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.079 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.080 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.080 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.080 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.080 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.080 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.080 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:53.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455575241 0 0) 0x7fe530132920 con 0x7fe5300a1d10 2024-09-16T14:50:53.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): my global_id is 184156 2024-09-16T14:50:53.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe510003530 con 0x7fe5300a1d10 2024-09-16T14:50:53.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 859713843 0 0) 0x7fe530133f00 con 0x7fe5300a0000 2024-09-16T14:50:53.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 >> v1:172.21.15.189:6789/0 conn(0x7fe5300a1d10 legacy=0x7fe5301cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 >> v1:172.21.15.26:6789/0 conn(0x7fe5300a2800 legacy=0x7fe5301d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:50:53.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 monclient: found mon.c 2024-09-16T14:50:53.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:53.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:53.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5300a64d0 con 0x7fe5300a0000 2024-09-16T14:50:53.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient: _finish_auth 0 2024-09-16T14:50:53.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:50:53.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:53.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 5 monclient: authenticate success, global_id 154856 2024-09-16T14:50:53.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 10 monclient: _renew_subs 2024-09-16T14:50:53.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:53.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 1 -- 172.21.15.26:0/2391884282 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53009e500 con 0x7fe5300a0000 2024-09-16T14:50:53.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 10 monclient: _renew_subs 2024-09-16T14:50:53.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:53.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 1 -- 172.21.15.26:0/2391884282 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe530133660 con 0x7fe5300a0000 2024-09-16T14:50:53.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe520005000 con 0x7fe5300a0000 2024-09-16T14:50:53.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:53.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe520005390 con 0x7fe5300a0000 2024-09-16T14:50:53.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:53.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe52cd99640 4 set_mon_vals no callback set 2024-09-16T14:50:53.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient: dump: 2024-09-16T14:50:53.083 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.083 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.083 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.083 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.083 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.083 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.083 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.083 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.083 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.083 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:53.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe534f7d640 1 -- 172.21.15.26:0/2391884282 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe514006590 con 0x7fe514002830 2024-09-16T14:50:53.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe5200054f0 con 0x7fe5300a0000 2024-09-16T14:50:53.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(246..246 src has 169..246) v4 ==== 7496+0+0 (unknown 771380485 0 0) 0x7fe5200444a0 con 0x7fe5300a0000 2024-09-16T14:50:53.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409905525 0 0) 0x7fe514006590 con 0x7fe514002830 2024-09-16T14:50:53.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 10 monclient(hunting): my global_id is 154859 2024-09-16T14:50:53.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.065+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe530133f00 con 0x7fe514002830 2024-09-16T14:50:53.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.069+0000 7fe52ed9d640 10 monclient: get_auth_request con 0x7fe5100360b0 auth_method 0 2024-09-16T14:50:53.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.069+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1726672468 0 0) 0x7fe530133f00 con 0x7fe514002830 2024-09-16T14:50:53.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.069+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe514006590 con 0x7fe514002830 2024-09-16T14:50:53.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.069+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe518001670 con 0x7fe514002830 2024-09-16T14:50:53.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.069+0000 7fe526ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.069+0000 7fe526ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:53.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.069+0000 7fe526ffd640 10 monclient: dump: 2024-09-16T14:50:53.085 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.085 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.085 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.085 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.086 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.086 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.086 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.086 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.086 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.086 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.069+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 238449639 0 0) 0x7fe514006590 con 0x7fe514002830 2024-09-16T14:50:53.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.069+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe51400a0e0 con 0x7fe514002830 2024-09-16T14:50:53.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.069+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe518001980 con 0x7fe514002830 2024-09-16T14:50:53.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.069+0000 7fe526ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:53.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.069+0000 7fe526ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:53.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.069+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 >> v1:172.21.15.26:6790/0 conn(0x7fe514002830 legacy=0x7fe514004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.069+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 >> v1:172.21.15.26:6790/0 conn(0x7fe514002830 legacy=0x7fe514004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe534f7d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:50:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe534f7d640 1 -- 172.21.15.26:0/2391884282 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe514009da0 con 0x7fe5140090c0 2024-09-16T14:50:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569312737 0 0) 0x7fe514009da0 con 0x7fe5140090c0 2024-09-16T14:50:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 10 monclient(hunting): my global_id is 154862 2024-09-16T14:50:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe514006590 con 0x7fe5140090c0 2024-09-16T14:50:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4243044330 0 0) 0x7fe514006590 con 0x7fe5140090c0 2024-09-16T14:50:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe514009da0 con 0x7fe5140090c0 2024-09-16T14:50:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe5180037a0 con 0x7fe5140090c0 2024-09-16T14:50:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 10 monclient: dump: 2024-09-16T14:50:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.090 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.090 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.090 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 333796425 0 0) 0x7fe514009da0 con 0x7fe5140090c0 2024-09-16T14:50:53.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe51400c290 con 0x7fe5140090c0 2024-09-16T14:50:53.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 1273043428) 0x7fe518005a00 con 0x7fe5140090c0 2024-09-16T14:50:53.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:50:53.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:53.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 >> v1:172.21.15.26:6790/0 conn(0x7fe5140090c0 legacy=0x7fe514009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe526ffd640 1 -- 172.21.15.26:0/2391884282 >> v1:172.21.15.26:6790/0 conn(0x7fe5140090c0 legacy=0x7fe514009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:53.091 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:50:53.091 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:53.091 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:53.091 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:50:53.091 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 78, 2024-09-16T14:50:53.091 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:50:53.091 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:50:53.091 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:50:53.091 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:50:53.091 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:53.092 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:50:53.092 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:53.092 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:50:53.092 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:50:53.092 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:53.092 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:53.092 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:53.092 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:53.092 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:53.092 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:53.092 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:53.092 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:53.092 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:53.092 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:53.093 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:53.093 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:50:53.093 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:50:53.093 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:53.093 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:53.093 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:53.093 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:53.093 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:53.093 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:53.093 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:53.093 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:53.093 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:53.094 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:53.094 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:53.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.094 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:50:53.094 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:50:53.094 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:50:53.094 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:50:53.094 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:50:53.094 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:50:53.094 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:53.094 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:53.094 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:50:53.094 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:50:53.095 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:50:53.095 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:50:53.095 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:50:53.095 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:50:53.095 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:50:53.095 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:53.095 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:50:53.095 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:53.095 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:53.095 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:53.095 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:53.095 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:53.095 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:53.096 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:53.096 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:53.096 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:53.096 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:53.096 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:53.096 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:50:53.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.096 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:50:53.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.096 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:53.096 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:53.096 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:53.096 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:53.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.097 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:53.097 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:50:53.097 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:53.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:53.097 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:53.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.097 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:50:53.097 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:50:53.097 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:53.097 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:53.097 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:53.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.098 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:53.098 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:53.098 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:53.098 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:53.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.098 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:53.098 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:50:53.098 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:53.098 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:53.098 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:53.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.098 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:50:53.098 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:50:53.099 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:53.099 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:53.099 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:53.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.099 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:53.099 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:53.099 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:53.099 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:53.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.099 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:53.100 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:50:53.100 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:53.100 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:53.100 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:53.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.100 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:50:53.100 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:50:53.101 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:53.101 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:53.101 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:53.101 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:53.101 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:53.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.101 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:50:53.101 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:50:53.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.101 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:53.101 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:53.101 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:53.101 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:53.102 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:53.102 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:50:53.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.102 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:53.102 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:53.102 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:50:53.102 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:53.102 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:53.102 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:50:53.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.102 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:53.102 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:53.103 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:50:53.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:53.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:53.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.103 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:50:53.103 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:50:53.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe534f7d640 1 -- 172.21.15.26:0/2391884282 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe5100360b0 legacy=0x7fe510038570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe534f7d640 10 monclient: shutdown 2024-09-16T14:50:53.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe534f7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:53.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe534f7d640 1 -- 172.21.15.26:0/2391884282 >> v1:172.21.15.26:6790/0 conn(0x7fe5300a0000 legacy=0x7fe530134320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.073+0000 7fe52ed9d640 1 -- 172.21.15.26:0/2391884282 reap_dead start 2024-09-16T14:50:53.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.077+0000 7fe52ed9d640 1 -- 172.21.15.26:0/2391884282 reap_dead start 2024-09-16T14:50:53.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.077+0000 7fe534f7d640 1 -- 172.21.15.26:0/2391884282 shutdown_connections 2024-09-16T14:50:53.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.077+0000 7fe534f7d640 1 -- 172.21.15.26:0/2391884282 >> 172.21.15.26:0/2391884282 conn(0x7fe5300a7f30 msgr2=0x7fe53006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:53.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.077+0000 7fe534f7d640 1 -- 172.21.15.26:0/2391884282 shutdown_connections 2024-09-16T14:50:53.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.077+0000 7fe534f7d640 1 -- 172.21.15.26:0/2391884282 wait complete. 2024-09-16T14:50:53.105 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:50:53.106 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:50:53.106 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:50:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.217+0000 7f4015a83640 10 monclient: get_monmap_and_config 2024-09-16T14:50:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.217+0000 7f4015a83640 10 monclient: build_initial_monmap 2024-09-16T14:50:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.217+0000 7f4015a83640 10 monclient: monmap: 2024-09-16T14:50:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:53.221160+0000 2024-09-16T14:50:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:53.221160+0000 2024-09-16T14:50:53.221 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:53.221 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.221 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:53.221 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:53.221 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:53.221 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 1 Processor -- start 2024-09-16T14:50:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 1 -- start start 2024-09-16T14:50:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 10 monclient: init 2024-09-16T14:50:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 10 monclient(hunting): picked mon.noname-b con 0x7f401009a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 10 monclient(hunting): picked mon.noname-c con 0x7f401009d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 10 monclient(hunting): picked mon.noname-a con 0x7f401009b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f401009dc60 con 0x7f401009b040 2024-09-16T14:50:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4010099ae0 con 0x7f401009d1b0 2024-09-16T14:50:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f401009b8f0 con 0x7f401009a550 2024-09-16T14:50:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 10 monclient(hunting): authenticate will time out at 1477.341184s 2024-09-16T14:50:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f401009b040 0x7f401013a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35656/0 (socket says 172.21.15.26:35656) 2024-09-16T14:50:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400ffff640 1 -- 172.21.15.26:0/455683818 learned_addr learned my addr 172.21.15.26:0/455683818 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 1 -- 172.21.15.26:0/455683818 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821249491 0 0) 0x7f401009dc60 con 0x7f401009b040 2024-09-16T14:50:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient(hunting): my global_id is 164607 2024-09-16T14:50:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 1 -- 172.21.15.26:0/455683818 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ff8003530 con 0x7f401009b040 2024-09-16T14:50:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 1 -- 172.21.15.26:0/455683818 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187243108 0 0) 0x7f401009b8f0 con 0x7f401009a550 2024-09-16T14:50:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient(hunting): my global_id is 154865 2024-09-16T14:50:53.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 1 -- 172.21.15.26:0/455683818 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f401009dc60 con 0x7f401009a550 2024-09-16T14:50:53.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 1 -- 172.21.15.26:0/455683818 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153682707 0 0) 0x7f4010099ae0 con 0x7f401009d1b0 2024-09-16T14:50:53.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient(hunting): my global_id is 184159 2024-09-16T14:50:53.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 1 -- 172.21.15.26:0/455683818 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f401009b8f0 con 0x7f401009d1b0 2024-09-16T14:50:53.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 1 -- 172.21.15.26:0/455683818 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1254530949 0 0) 0x7f3ff8003530 con 0x7f401009b040 2024-09-16T14:50:53.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 1 -- 172.21.15.26:0/455683818 >> v1:172.21.15.26:6790/0 conn(0x7f401009a550 legacy=0x7f401009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 1 -- 172.21.15.26:0/455683818 >> v1:172.21.15.189:6789/0 conn(0x7f401009d1b0 legacy=0x7f4010136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:50:53.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 1 monclient: found mon.noname-a 2024-09-16T14:50:53.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:53.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:50:53.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 1 -- 172.21.15.26:0/455683818 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401009bbd0 con 0x7f401009b040 2024-09-16T14:50:53.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:50:53.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:50:53.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:53.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 1 -- 172.21.15.26:0/455683818 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f40080019a0 con 0x7f401009b040 2024-09-16T14:50:53.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:50:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient: dump: 2024-09-16T14:50:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.228 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 5 monclient: authenticate success, global_id 164607 2024-09-16T14:50:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 1 -- 172.21.15.26:0/455683818 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4008001c00 con 0x7f401009b040 2024-09-16T14:50:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 1 -- 172.21.15.26:0/455683818 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4008001fd0 con 0x7f401009b040 2024-09-16T14:50:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 4 set_mon_vals no callback set 2024-09-16T14:50:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f400e7fc640 10 monclient: dump: 2024-09-16T14:50:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.230 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 10 monclient: shutdown 2024-09-16T14:50:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:53.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 1 -- 172.21.15.26:0/455683818 >> v1:172.21.15.26:6789/0 conn(0x7f401009b040 legacy=0x7f401013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 1 -- 172.21.15.26:0/455683818 shutdown_connections 2024-09-16T14:50:53.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 1 -- 172.21.15.26:0/455683818 >> 172.21.15.26:0/455683818 conn(0x7f401009fb50 msgr2=0x7f401009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:53.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 1 -- 172.21.15.26:0/455683818 shutdown_connections 2024-09-16T14:50:53.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 1 -- 172.21.15.26:0/455683818 wait complete. 2024-09-16T14:50:53.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 10 monclient: build_initial_monmap 2024-09-16T14:50:53.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 10 monclient: monmap: 2024-09-16T14:50:53.231 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:53.231 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:53.231 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:53.231 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:53.232 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:53.232 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:53.232 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:53.232 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:53.232 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:53.232 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 1 Processor -- start 2024-09-16T14:50:53.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 1 -- start start 2024-09-16T14:50:53.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.221+0000 7f4015a83640 10 monclient: init 2024-09-16T14:50:53.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:53.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:50:53.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 10 monclient(hunting): picked mon.noname-a con 0x7f401009a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:53.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 10 monclient(hunting): picked mon.noname-b con 0x7f401009b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:53.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 10 monclient(hunting): picked mon.noname-c con 0x7f401009d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4010139a10 con 0x7f401009a550 2024-09-16T14:50:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4010139cf0 con 0x7f401009b040 2024-09-16T14:50:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4010139fd0 con 0x7f401009d1b0 2024-09-16T14:50:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 10 monclient(hunting): authenticate will time out at 1477.344308s 2024-09-16T14:50:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f400f7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f401009a550 0x7f401013eb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35672/0 (socket says 172.21.15.26:35672) 2024-09-16T14:50:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f400f7fe640 1 -- 172.21.15.26:0/1160668469 learned_addr learned my addr 172.21.15.26:0/1160668469 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774459197 0 0) 0x7f4010139fd0 con 0x7f401009d1b0 2024-09-16T14:50:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): my global_id is 154868 2024-09-16T14:50:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ffc003530 con 0x7f401009d1b0 2024-09-16T14:50:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753223174 0 0) 0x7f4010139a10 con 0x7f401009a550 2024-09-16T14:50:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): my global_id is 164610 2024-09-16T14:50:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4010139fd0 con 0x7f401009a550 2024-09-16T14:50:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1717282642 0 0) 0x7f3ffc003530 con 0x7f401009d1b0 2024-09-16T14:50:53.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4010139a10 con 0x7f401009d1b0 2024-09-16T14:50:53.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4008001700 con 0x7f401009d1b0 2024-09-16T14:50:53.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:53.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): dump: 2024-09-16T14:50:53.242 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.242 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.242 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.242 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.242 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.242 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.242 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.242 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.242 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.242 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2024654158 0 0) 0x7f4010139fd0 con 0x7f401009a550 2024-09-16T14:50:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ffc003530 con 0x7f401009a550 2024-09-16T14:50:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4004001640 con 0x7f401009a550 2024-09-16T14:50:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:50:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): dump: 2024-09-16T14:50:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.244 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765824690 0 0) 0x7f4010139cf0 con 0x7f401009b040 2024-09-16T14:50:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): my global_id is 184162 2024-09-16T14:50:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4010139fd0 con 0x7f401009b040 2024-09-16T14:50:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4102619822 0 0) 0x7f4010139a10 con 0x7f401009d1b0 2024-09-16T14:50:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 >> v1:172.21.15.189:6789/0 conn(0x7f401009b040 legacy=0x7f401013f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 >> v1:172.21.15.26:6789/0 conn(0x7f401009a550 legacy=0x7f401013eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:50:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 monclient: found mon.c 2024-09-16T14:50:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4010140470 con 0x7f401009d1b0 2024-09-16T14:50:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient: _finish_auth 0 2024-09-16T14:50:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:50:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 5 monclient: authenticate success, global_id 154868 2024-09-16T14:50:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 10 monclient: _renew_subs 2024-09-16T14:50:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 1 -- 172.21.15.26:0/1160668469 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401013a6d0 con 0x7f401009d1b0 2024-09-16T14:50:53.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 10 monclient: _renew_subs 2024-09-16T14:50:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 1 -- 172.21.15.26:0/1160668469 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f401013dc20 con 0x7f401009d1b0 2024-09-16T14:50:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40080019d0 con 0x7f401009d1b0 2024-09-16T14:50:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4008005120 con 0x7f401009d1b0 2024-09-16T14:50:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f400dffb640 4 set_mon_vals no callback set 2024-09-16T14:50:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient: dump: 2024-09-16T14:50:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.246 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.247 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f4015a83640 1 -- 172.21.15.26:0/1160668469 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3ff0006590 con 0x7f3ff0002830 2024-09-16T14:50:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f400800fb90 con 0x7f401009d1b0 2024-09-16T14:50:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(246..246 src has 169..246) v4 ==== 7496+0+0 (unknown 771380485 0 0) 0x7f4008044370 con 0x7f401009d1b0 2024-09-16T14:50:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f400effd640 10 monclient: get_auth_request con 0x7f3ffc0363d0 auth_method 0 2024-09-16T14:50:53.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593206690 0 0) 0x7f3ff0006590 con 0x7f3ff0002830 2024-09-16T14:50:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient(hunting): my global_id is 184165 2024-09-16T14:50:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4010139a10 con 0x7f3ff0002830 2024-09-16T14:50:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3976523925 0 0) 0x7f4010139a10 con 0x7f3ff0002830 2024-09-16T14:50:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ff0006590 con 0x7f3ff0002830 2024-09-16T14:50:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4004003a10 con 0x7f3ff0002830 2024-09-16T14:50:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 10 monclient: dump: 2024-09-16T14:50:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.248 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.249 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.249 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.249 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.249 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.249 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.249 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.249 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.249 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.225+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2764703784 0 0) 0x7f3ff0006590 con 0x7f3ff0002830 2024-09-16T14:50:53.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.229+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ff000a0e0 con 0x7f3ff0002830 2024-09-16T14:50:53.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.229+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f4004005790 con 0x7f3ff0002830 2024-09-16T14:50:53.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.229+0000 7f3ff7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:53.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.229+0000 7f3ff7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:53.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.229+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 >> v1:172.21.15.189:6789/0 conn(0x7f3ff0002830 legacy=0x7f3ff0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.229+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 >> v1:172.21.15.189:6789/0 conn(0x7f3ff0002830 legacy=0x7f3ff0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:53.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.233+0000 7f4015a83640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:50:53.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.233+0000 7f4015a83640 1 -- 172.21.15.26:0/1160668469 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3ff0009da0 con 0x7f3ff00090c0 2024-09-16T14:50:53.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.233+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982923441 0 0) 0x7f3ff0009da0 con 0x7f3ff00090c0 2024-09-16T14:50:53.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.233+0000 7f3ff7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.233+0000 7f3ff7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.233+0000 7f3ff7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.233+0000 7f3ff7fff640 10 monclient(hunting): my global_id is 184168 2024-09-16T14:50:53.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.233+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ff0006590 con 0x7f3ff00090c0 2024-09-16T14:50:53.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2388611231 0 0) 0x7f3ff0006590 con 0x7f3ff00090c0 2024-09-16T14:50:53.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ff0009da0 con 0x7f3ff00090c0 2024-09-16T14:50:53.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f40040040d0 con 0x7f3ff00090c0 2024-09-16T14:50:53.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f3ff7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f3ff7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:50:53.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f3ff7fff640 10 monclient: dump: 2024-09-16T14:50:53.251 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.251 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.251 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.251 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.251 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.251 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.251 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.251 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.251 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.251 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3096939984 0 0) 0x7f3ff0009da0 con 0x7f3ff00090c0 2024-09-16T14:50:53.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3ff000c290 con 0x7f3ff00090c0 2024-09-16T14:50:53.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 2070845085) 0x7f4004004330 con 0x7f3ff00090c0 2024-09-16T14:50:53.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f3ff7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:50:53.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f3ff7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:53.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 >> v1:172.21.15.189:6789/0 conn(0x7f3ff00090c0 legacy=0x7f3ff0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f3ff7fff640 1 -- 172.21.15.26:0/1160668469 >> v1:172.21.15.189:6789/0 conn(0x7f3ff00090c0 legacy=0x7f3ff0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:50:53.252 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:50:53.252 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:53.252 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:53.253 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:50:53.253 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 78, 2024-09-16T14:50:53.253 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:50:53.253 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:50:53.253 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:50:53.253 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:50:53.253 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:53.253 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-09-16T14:50:53.253 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:53.253 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:50:53.253 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:50:53.253 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:53.253 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:53.254 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:53.254 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:53.254 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:53.254 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:53.254 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:53.254 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:53.254 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:53.254 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:53.254 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:53.254 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:50:53.254 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:50:53.254 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:53.254 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:53.254 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:53.255 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:53.255 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:53.255 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:53.255 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:53.255 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:53.255 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:53.255 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:53.255 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:53.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.255 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:50:53.255 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:50:53.255 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:50:53.255 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:50:53.255 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:50:53.256 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:50:53.256 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:53.256 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:50:53.256 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:50:53.256 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:50:53.256 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:50:53.256 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:50:53.256 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:50:53.256 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:50:53.256 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:50:53.256 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:50:53.256 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:50:53.256 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:50:53.256 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:50:53.257 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:50:53.257 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:50:53.257 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:50:53.257 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:50:53.257 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:50:53.257 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:50:53.257 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:50:53.257 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:50:53.257 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:53.257 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:50:53.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.257 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:50:53.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.258 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:50:53.258 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:50:53.258 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:53.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:53.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:53.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:50:53.258 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:53.258 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:53.258 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:53.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:50:53.258 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:50:53.258 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:53.259 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:53.259 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:53.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.259 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:50:53.259 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:50:53.259 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:53.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:53.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.259 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:53.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:50:53.259 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:53.260 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:53.260 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:53.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.260 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:50:53.260 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:50:53.260 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:53.260 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:53.260 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:53.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.261 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:50:53.261 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:50:53.261 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:50:53.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:50:53.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.261 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:50:53.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:50:53.261 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:50:53.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:53.261 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:53.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:50:53.262 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:50:53.262 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:50:53.262 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:50:53.262 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:50:53.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:53.262 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:53.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.262 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:50:53.262 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:50:53.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.263 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:53.263 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:53.263 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:50:53.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:53.263 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:50:53.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:50:53.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:50:53.263 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:50:53.263 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:50:53.264 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:50:53.264 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:50:53.264 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:50:53.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:50:53.264 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:50:53.264 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:50:53.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f4015a83640 1 -- 172.21.15.26:0/1160668469 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3ffc0363d0 legacy=0x7f3ffc038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f4015a83640 10 monclient: shutdown 2024-09-16T14:50:53.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f4015a83640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:53.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f4015a83640 1 -- 172.21.15.26:0/1160668469 >> v1:172.21.15.26:6790/0 conn(0x7f401009d1b0 legacy=0x7f40101d1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f400ffff640 1 -- 172.21.15.26:0/1160668469 reap_dead start 2024-09-16T14:50:53.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f400ffff640 1 -- 172.21.15.26:0/1160668469 reap_dead start 2024-09-16T14:50:53.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f4015a83640 1 -- 172.21.15.26:0/1160668469 shutdown_connections 2024-09-16T14:50:53.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f4015a83640 1 -- 172.21.15.26:0/1160668469 >> 172.21.15.26:0/1160668469 conn(0x7f401009fb50 msgr2=0x7f40100a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:53.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f4015a83640 1 -- 172.21.15.26:0/1160668469 shutdown_connections 2024-09-16T14:50:53.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.237+0000 7f4015a83640 1 -- 172.21.15.26:0/1160668469 wait complete. 2024-09-16T14:50:53.267 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:50:53.267 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T14:50:53.267 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:50:53.267 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T14:50:53.268 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:50:53.268 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T14:50:53.268 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:50:53.280 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T14:50:53.280 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:50:53.280 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:50:53.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 10 monclient: get_monmap_and_config 2024-09-16T14:50:53.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 10 monclient: build_initial_monmap 2024-09-16T14:50:53.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 10 monclient: monmap: 2024-09-16T14:50:53.434 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:53.434 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.434 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:53.435415+0000 2024-09-16T14:50:53.434 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:53.435415+0000 2024-09-16T14:50:53.434 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:53.435 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.435 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:53.435 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:53.435 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:53.435 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 1 Processor -- start 2024-09-16T14:50:53.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 1 -- start start 2024-09-16T14:50:53.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 10 monclient: init 2024-09-16T14:50:53.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:53.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:53.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 10 monclient(hunting): picked mon.noname-b con 0x7f900409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:53.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 10 monclient(hunting): picked mon.noname-c con 0x7f9004132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:53.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 10 monclient(hunting): picked mon.noname-a con 0x7f9004138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:53.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900409d260 con 0x7f9004138ff0 2024-09-16T14:50:53.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9004132a80 con 0x7f9004132650 2024-09-16T14:50:53.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900409cf60 con 0x7f900409cb80 2024-09-16T14:50:53.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:53.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 10 monclient(hunting): authenticate will time out at 1477.555083s 2024-09-16T14:50:53.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f90037fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f9004138ff0 legacy=0x7f900413b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:50:53.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9002ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f900409cb80 0x7f90040717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33950/0 (socket says 172.21.15.26:33950) 2024-09-16T14:50:53.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9002ffd640 1 -- 172.21.15.26:0/4011373746 learned_addr learned my addr 172.21.15.26:0/4011373746 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:53.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 1 -- 172.21.15.26:0/4011373746 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198317485 0 0) 0x7f900409cf60 con 0x7f900409cb80 2024-09-16T14:50:53.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 10 monclient(hunting): my global_id is 154871 2024-09-16T14:50:53.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 1 -- 172.21.15.26:0/4011373746 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fec003530 con 0x7f900409cb80 2024-09-16T14:50:53.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 1 -- 172.21.15.26:0/4011373746 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 167408204 0 0) 0x7f8fec003530 con 0x7f900409cb80 2024-09-16T14:50:53.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 1 -- 172.21.15.26:0/4011373746 >> v1:172.21.15.189:6789/0 conn(0x7f9004132650 legacy=0x7f90041378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 1 -- 172.21.15.26:0/4011373746 >> v1:172.21.15.26:6789/0 conn(0x7f9004138ff0 legacy=0x7f900413b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:53.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:50:53.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 1 monclient: found mon.noname-b 2024-09-16T14:50:53.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:53.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:53.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 1 -- 172.21.15.26:0/4011373746 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900409f2d0 con 0x7f900409cb80 2024-09-16T14:50:53.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 10 monclient: _finish_auth 0 2024-09-16T14:50:53.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:53.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:53.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 1 -- 172.21.15.26:0/4011373746 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ff8001660 con 0x7f900409cb80 2024-09-16T14:50:53.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 5 monclient: authenticate success, global_id 154871 2024-09-16T14:50:53.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9009937640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:53.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:53.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.433+0000 7f9001ffb640 10 monclient: dump: 2024-09-16T14:50:53.445 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.446 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:53.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9001ffb640 1 -- 172.21.15.26:0/4011373746 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ff80018c0 con 0x7f900409cb80 2024-09-16T14:50:53.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9001ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:53.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9001ffb640 1 -- 172.21.15.26:0/4011373746 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ff8001c90 con 0x7f900409cb80 2024-09-16T14:50:53.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:53.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 4 set_mon_vals no callback set 2024-09-16T14:50:53.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9001ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9001ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:53.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9001ffb640 10 monclient: dump: 2024-09-16T14:50:53.447 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.449 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient: shutdown 2024-09-16T14:50:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 1 -- 172.21.15.26:0/4011373746 >> v1:172.21.15.26:6790/0 conn(0x7f900409cb80 legacy=0x7f90040717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 1 -- 172.21.15.26:0/4011373746 shutdown_connections 2024-09-16T14:50:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 1 -- 172.21.15.26:0/4011373746 >> 172.21.15.26:0/4011373746 conn(0x7f900409fb50 msgr2=0x7f900409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 1 -- 172.21.15.26:0/4011373746 shutdown_connections 2024-09-16T14:50:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 1 -- 172.21.15.26:0/4011373746 wait complete. 2024-09-16T14:50:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient: build_initial_monmap 2024-09-16T14:50:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient: monmap: 2024-09-16T14:50:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:53.450 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 1 Processor -- start 2024-09-16T14:50:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 1 -- start start 2024-09-16T14:50:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient: init 2024-09-16T14:50:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:50:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient(hunting): picked mon.noname-a con 0x7f900409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient(hunting): picked mon.noname-b con 0x7f9004132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient(hunting): picked mon.noname-c con 0x7f9004138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900406de90 con 0x7f900409cb80 2024-09-16T14:50:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900409b370 con 0x7f9004132650 2024-09-16T14:50:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9002ffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f900409cb80 legacy=0x7f900409a550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:50:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9004131e30 con 0x7f9004138ff0 2024-09-16T14:50:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient(hunting): authenticate will time out at 1477.558228s 2024-09-16T14:50:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f90037fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9004138ff0 0x7f90041d1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33954/0 (socket says 172.21.15.26:33954) 2024-09-16T14:50:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f90037fe640 1 -- 172.21.15.26:0/1602419666 learned_addr learned my addr 172.21.15.26:0/1602419666 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 1 -- 172.21.15.26:0/1602419666 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888520761 0 0) 0x7f9004131e30 con 0x7f9004138ff0 2024-09-16T14:50:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 10 monclient(hunting): my global_id is 154874 2024-09-16T14:50:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 1 -- 172.21.15.26:0/1602419666 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff0003530 con 0x7f9004138ff0 2024-09-16T14:50:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 1 -- 172.21.15.26:0/1602419666 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1666866193 0 0) 0x7f8ff0003530 con 0x7f9004138ff0 2024-09-16T14:50:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 1 -- 172.21.15.26:0/1602419666 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9004131e30 con 0x7f9004138ff0 2024-09-16T14:50:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 1 -- 172.21.15.26:0/1602419666 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ffc0043c0 con 0x7f9004138ff0 2024-09-16T14:50:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 10 monclient(hunting): dump: 2024-09-16T14:50:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.453 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 1 -- 172.21.15.26:0/1602419666 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 669439425 0 0) 0x7f9004131e30 con 0x7f9004138ff0 2024-09-16T14:50:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 1 -- 172.21.15.26:0/1602419666 >> v1:172.21.15.189:6789/0 conn(0x7f9004132650 legacy=0x7f900409ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 1 -- 172.21.15.26:0/1602419666 >> v1:172.21.15.26:6789/0 conn(0x7f900409cb80 legacy=0x7f900409a550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 10 monclient: _finish_hunting 0 2024-09-16T14:50:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 1 monclient: found mon.c 2024-09-16T14:50:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 1 -- 172.21.15.26:0/1602419666 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900413c590 con 0x7f9004138ff0 2024-09-16T14:50:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 10 monclient: _finish_auth 0 2024-09-16T14:50:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 10 monclient: _check_auth_tickets 2024-09-16T14:50:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 5 monclient: authenticate success, global_id 154874 2024-09-16T14:50:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient: _renew_subs 2024-09-16T14:50:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 1 -- 172.21.15.26:0/1602419666 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900409b5d0 con 0x7f9004138ff0 2024-09-16T14:50:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient: _renew_subs 2024-09-16T14:50:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 1 -- 172.21.15.26:0/1602419666 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f900406d5d0 con 0x7f9004138ff0 2024-09-16T14:50:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 1 -- 172.21.15.26:0/1602419666 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ffc0046e0 con 0x7f9004138ff0 2024-09-16T14:50:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 1 -- 172.21.15.26:0/1602419666 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ffc004a70 con 0x7f9004138ff0 2024-09-16T14:50:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9000ff9640 4 set_mon_vals no callback set 2024-09-16T14:50:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 10 monclient: dump: 2024-09-16T14:50:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:53.456 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f8febfff640 1 -- 172.21.15.26:0/1602419666 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8ffc004bd0 con 0x7f9004138ff0 2024-09-16T14:50:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.437+0000 7f9009937640 1 -- 172.21.15.26:0/1602419666 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe4005050 con 0x7f9004138ff0 2024-09-16T14:50:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.441+0000 7f8febfff640 1 -- 172.21.15.26:0/1602419666 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(246..246 src has 169..246) v4 ==== 7496+0+0 (unknown 771380485 0 0) 0x7f8ffc0442c0 con 0x7f9004138ff0 2024-09-16T14:50:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.441+0000 7f9002ffd640 10 monclient: get_auth_request con 0x7f8ff0036080 auth_method 0 2024-09-16T14:50:53.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.445+0000 7f8febfff640 1 -- 172.21.15.26:0/1602419666 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8ffc0053d0 con 0x7f9004138ff0 2024-09-16T14:50:53.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.445+0000 7f8febfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:53.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.445+0000 7f8febfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:53.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.657+0000 7f9009937640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:50:53.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.657+0000 7f9009937640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:53.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.657+0000 7f9009937640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:53.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.657+0000 7f9009937640 1 -- 172.21.15.26:0/1602419666 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8fe4005880 con 0x7f9004138ff0 2024-09-16T14:50:53.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.661+0000 7f8febfff640 1 -- 172.21.15.26:0/1602419666 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3146961702) 0x7f8ffc012e30 con 0x7f9004138ff0 2024-09-16T14:50:53.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.661+0000 7f8febfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:53.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.661+0000 7f8febfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:53.661 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:50:53.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.661+0000 7f9009937640 1 -- 172.21.15.26:0/1602419666 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8ff0036080 legacy=0x7f8ff0038540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.661+0000 7f9009937640 10 monclient: shutdown 2024-09-16T14:50:53.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.661+0000 7f9009937640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:53.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.661+0000 7f9009937640 1 -- 172.21.15.26:0/1602419666 >> v1:172.21.15.26:6790/0 conn(0x7f9004138ff0 legacy=0x7f90041d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:53.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.661+0000 7f9009937640 1 -- 172.21.15.26:0/1602419666 shutdown_connections 2024-09-16T14:50:53.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.661+0000 7f9009937640 1 -- 172.21.15.26:0/1602419666 >> 172.21.15.26:0/1602419666 conn(0x7f900409fb50 msgr2=0x7f9004130820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:53.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.661+0000 7f9009937640 1 -- 172.21.15.26:0/1602419666 shutdown_connections 2024-09-16T14:50:53.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:53.661+0000 7f9009937640 1 -- 172.21.15.26:0/1602419666 wait complete. 2024-09-16T14:50:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:55.544+0000 7ffb79ffb640 10 monclient: tick 2024-09-16T14:50:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:55.544+0000 7ffb79ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:55.544+0000 7ffb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:55.544+0000 7ffb79ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:56.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:56.544+0000 7ffb79ffb640 10 monclient: tick 2024-09-16T14:50:56.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:56.544+0000 7ffb79ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:56.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:56.544+0000 7ffb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:56.679 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:50:56.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 10 monclient: get_monmap_and_config 2024-09-16T14:50:56.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 10 monclient: build_initial_monmap 2024-09-16T14:50:56.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 10 monclient: monmap: 2024-09-16T14:50:56.809 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:56.810 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:56.810 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:50:56.810669+0000 2024-09-16T14:50:56.810 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:50:56.810669+0000 2024-09-16T14:50:56.810 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:56.810 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:56.810 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:56.810 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:50:56.810 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:50:56.810 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:56.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 1 Processor -- start 2024-09-16T14:50:56.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 1 -- start start 2024-09-16T14:50:56.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 10 monclient: init 2024-09-16T14:50:56.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:56.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:50:56.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 10 monclient(hunting): picked mon.noname-a con 0x7f407809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:56.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 10 monclient(hunting): picked mon.noname-c con 0x7f4078132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:56.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 10 monclient(hunting): picked mon.noname-b con 0x7f4078138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:56.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407809d260 con 0x7f407809cb80 2024-09-16T14:50:56.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4078132a80 con 0x7f4078132650 2024-09-16T14:50:56.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407809cf60 con 0x7f4078138ff0 2024-09-16T14:50:56.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:56.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 10 monclient(hunting): authenticate will time out at 1480.930739s 2024-09-16T14:50:56.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407d58d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f407809cb80 legacy=0x7f40780717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:50:56.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407dd8e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4078138ff0 0x7f407813b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33958/0 (socket says 172.21.15.26:33958) 2024-09-16T14:50:56.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407dd8e640 1 -- 172.21.15.26:0/644807415 learned_addr learned my addr 172.21.15.26:0/644807415 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:56.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 1 -- 172.21.15.26:0/644807415 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454076331 0 0) 0x7f407809cf60 con 0x7f4078138ff0 2024-09-16T14:50:56.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:50:56.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:56.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:56.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 10 monclient(hunting): my global_id is 154877 2024-09-16T14:50:56.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 1 -- 172.21.15.26:0/644807415 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4064003530 con 0x7f4078138ff0 2024-09-16T14:50:56.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 1 -- 172.21.15.26:0/644807415 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3711647839 0 0) 0x7f4064003530 con 0x7f4078138ff0 2024-09-16T14:50:56.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 1 -- 172.21.15.26:0/644807415 >> v1:172.21.15.189:6789/0 conn(0x7f4078132650 legacy=0x7f40781378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:56.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 1 -- 172.21.15.26:0/644807415 >> v1:172.21.15.26:6789/0 conn(0x7f407809cb80 legacy=0x7f40780717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:50:56.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:50:56.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 1 monclient: found mon.noname-b 2024-09-16T14:50:56.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:56.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:50:56.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 1 -- 172.21.15.26:0/644807415 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407809f2d0 con 0x7f4078138ff0 2024-09-16T14:50:56.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 10 monclient: _finish_auth 0 2024-09-16T14:50:56.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:50:56.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:56.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 1 -- 172.21.15.26:0/644807415 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f40700019e0 con 0x7f4078138ff0 2024-09-16T14:50:56.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:56.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:50:56.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 10 monclient: dump: 2024-09-16T14:50:56.816 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:56.816 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:56.817 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:56.817 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:56.817 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:56.817 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:56.817 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:56.817 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:56.817 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:56.817 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:56.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 5 monclient: authenticate success, global_id 154877 2024-09-16T14:50:56.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:50:56.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 1 -- 172.21.15.26:0/644807415 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4070001c00 con 0x7f4078138ff0 2024-09-16T14:50:56.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:56.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 1 -- 172.21.15.26:0/644807415 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4070001f90 con 0x7f4078138ff0 2024-09-16T14:50:56.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:56.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:56.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f406ffff640 10 monclient: dump: 2024-09-16T14:50:56.818 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:56.818 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:56.818 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:56.818 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:56.818 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:56.818 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:56.818 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:56.818 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:56.818 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:56.818 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:56.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 10 monclient: get_monmap_and_config success 2024-09-16T14:50:56.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.809+0000 7f407f7f1640 4 set_mon_vals no callback set 2024-09-16T14:50:56.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 10 monclient: shutdown 2024-09-16T14:50:56.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:56.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 1 -- 172.21.15.26:0/644807415 >> v1:172.21.15.26:6790/0 conn(0x7f4078138ff0 legacy=0x7f407813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:56.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 1 -- 172.21.15.26:0/644807415 shutdown_connections 2024-09-16T14:50:56.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 1 -- 172.21.15.26:0/644807415 >> 172.21.15.26:0/644807415 conn(0x7f407809fb50 msgr2=0x7f407809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:56.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 1 -- 172.21.15.26:0/644807415 shutdown_connections 2024-09-16T14:50:56.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 1 -- 172.21.15.26:0/644807415 wait complete. 2024-09-16T14:50:56.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 10 monclient: build_initial_monmap 2024-09-16T14:50:56.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 10 monclient: monmap: 2024-09-16T14:50:56.823 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:50:56.823 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:50:56.823 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:50:56.823 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:50:56.823 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:50:56.823 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:50:56.823 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:50:56.823 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:50:56.823 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:50:56.823 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:56.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 1 Processor -- start 2024-09-16T14:50:56.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 1 -- start start 2024-09-16T14:50:56.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 10 monclient: init 2024-09-16T14:50:56.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 10 monclient: _reopen_session rank -1 2024-09-16T14:50:56.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:50:56.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 10 monclient(hunting): picked mon.noname-c con 0x7f407809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:50:56.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 10 monclient(hunting): picked mon.noname-b con 0x7f4078132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:50:56.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 10 monclient(hunting): picked mon.noname-a con 0x7f4078138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:50:56.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407813c590 con 0x7f4078138ff0 2024-09-16T14:50:56.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407806e450 con 0x7f4078132650 2024-09-16T14:50:56.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407dd8e640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f4078138ff0 legacy=0x7f40781cfb60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:50:56.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407809a240 con 0x7f407809cb80 2024-09-16T14:50:56.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 10 monclient(hunting): _renew_subs 2024-09-16T14:50:56.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 10 monclient(hunting): authenticate will time out at 1480.934590s 2024-09-16T14:50:56.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407d58d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f407809cb80 0x7f407806d630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33966/0 (socket says 172.21.15.26:33966) 2024-09-16T14:50:56.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407d58d640 1 -- 172.21.15.26:0/2166313393 learned_addr learned my addr 172.21.15.26:0/2166313393 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:50:56.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 1 -- 172.21.15.26:0/2166313393 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509821996 0 0) 0x7f407809a240 con 0x7f407809cb80 2024-09-16T14:50:56.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:50:56.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:50:56.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:50:56.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 10 monclient(hunting): my global_id is 154880 2024-09-16T14:50:56.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 1 -- 172.21.15.26:0/2166313393 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f405c003530 con 0x7f407809cb80 2024-09-16T14:50:56.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 1 -- 172.21.15.26:0/2166313393 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2246237331 0 0) 0x7f405c003530 con 0x7f407809cb80 2024-09-16T14:50:56.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 1 -- 172.21.15.26:0/2166313393 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f407809a240 con 0x7f407809cb80 2024-09-16T14:50:56.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 1 -- 172.21.15.26:0/2166313393 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4074004160 con 0x7f407809cb80 2024-09-16T14:50:56.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:56.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:50:56.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 10 monclient(hunting): dump: 2024-09-16T14:50:56.826 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:56.826 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:56.826 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:56.826 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:56.826 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:56.826 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:56.826 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:56.826 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:56.826 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:56.826 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:56.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:50:56.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 1 -- 172.21.15.26:0/2166313393 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4168526274 0 0) 0x7f407809a240 con 0x7f407809cb80 2024-09-16T14:50:56.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 1 -- 172.21.15.26:0/2166313393 >> v1:172.21.15.189:6789/0 conn(0x7f4078132650 legacy=0x7f407806dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:56.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 1 -- 172.21.15.26:0/2166313393 >> v1:172.21.15.26:6789/0 conn(0x7f4078138ff0 legacy=0x7f40781cfb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:50:56.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:50:56.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 1 monclient: found mon.c 2024-09-16T14:50:56.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:50:56.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:56.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 1 -- 172.21.15.26:0/2166313393 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4078071520 con 0x7f407809cb80 2024-09-16T14:50:56.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 10 monclient: _finish_auth 0 2024-09-16T14:50:56.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:56.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:56.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 5 monclient: authenticate success, global_id 154880 2024-09-16T14:50:56.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 10 monclient: _renew_subs 2024-09-16T14:50:56.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:56.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 1 -- 172.21.15.26:0/2166313393 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f407806fb10 con 0x7f407809cb80 2024-09-16T14:50:56.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 10 monclient: _renew_subs 2024-09-16T14:50:56.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:56.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f407f7f1640 1 -- 172.21.15.26:0/2166313393 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f407806f1c0 con 0x7f407809cb80 2024-09-16T14:50:56.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 1 -- 172.21.15.26:0/2166313393 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4074004480 con 0x7f407809cb80 2024-09-16T14:50:56.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:50:56.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 1 -- 172.21.15.26:0/2166313393 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4074004810 con 0x7f407809cb80 2024-09-16T14:50:56.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:50:56.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:50:56.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406f7fe640 4 set_mon_vals no callback set 2024-09-16T14:50:56.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.813+0000 7f406dffb640 10 monclient: dump: 2024-09-16T14:50:56.829 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:50:56.829 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:50:56.829 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:56.829 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:50:56.829 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:50:56.829 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:50:56.829 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:50:56.829 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:50:56.829 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:50:56.829 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:50:56.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.817+0000 7f407f7f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:56.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.817+0000 7f407f7f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:56.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.817+0000 7f407f7f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:56.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.817+0000 7f406dffb640 1 -- 172.21.15.26:0/2166313393 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4074004970 con 0x7f407809cb80 2024-09-16T14:50:56.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.817+0000 7f407f7f1640 1 -- 172.21.15.26:0/2166313393 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4058005050 con 0x7f407809cb80 2024-09-16T14:50:56.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.817+0000 7f406dffb640 1 -- 172.21.15.26:0/2166313393 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(246..246 src has 169..246) v4 ==== 7496+0+0 (unknown 771380485 0 0) 0x7f40740440d0 con 0x7f407809cb80 2024-09-16T14:50:56.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.817+0000 7f407cd8c640 10 monclient: get_auth_request con 0x7f405c036050 auth_method 0 2024-09-16T14:50:56.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.821+0000 7f406dffb640 1 -- 172.21.15.26:0/2166313393 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4074005170 con 0x7f407809cb80 2024-09-16T14:50:56.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.821+0000 7f406dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:50:56.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:56.821+0000 7f406dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:50:57.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:57.041+0000 7f407f7f1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:50:57.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:57.041+0000 7f407f7f1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:57.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:57.041+0000 7f407f7f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:50:57.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:57.041+0000 7f407f7f1640 1 -- 172.21.15.26:0/2166313393 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4058005880 con 0x7f407809cb80 2024-09-16T14:50:57.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:57.041+0000 7f406dffb640 1 -- 172.21.15.26:0/2166313393 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3950910727) 0x7f4074012bd0 con 0x7f407809cb80 2024-09-16T14:50:57.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:57.041+0000 7f406dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:50:57.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:57.041+0000 7f406dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:50:57.043 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:50:57.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:57.045+0000 7f407f7f1640 1 -- 172.21.15.26:0/2166313393 >> v1:172.21.15.189:6812/3196161582 conn(0x7f405c036050 legacy=0x7f405c038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:57.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:57.045+0000 7f407f7f1640 10 monclient: shutdown 2024-09-16T14:50:57.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:57.045+0000 7f407f7f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:50:57.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:57.045+0000 7f407f7f1640 1 -- 172.21.15.26:0/2166313393 >> v1:172.21.15.26:6790/0 conn(0x7f407809cb80 legacy=0x7f407806d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:50:57.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:57.045+0000 7f407f7f1640 1 -- 172.21.15.26:0/2166313393 shutdown_connections 2024-09-16T14:50:57.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:57.045+0000 7f407f7f1640 1 -- 172.21.15.26:0/2166313393 >> 172.21.15.26:0/2166313393 conn(0x7f407809fb50 msgr2=0x7f4078130680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:50:57.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:57.045+0000 7f407f7f1640 1 -- 172.21.15.26:0/2166313393 shutdown_connections 2024-09-16T14:50:57.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:50:57.045+0000 7f407f7f1640 1 -- 172.21.15.26:0/2166313393 wait complete. 2024-09-16T14:50:57.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:57.544+0000 7ffb79ffb640 10 monclient: tick 2024-09-16T14:50:57.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:57.544+0000 7ffb79ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:57.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:57.544+0000 7ffb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:58.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:58.544+0000 7ffb79ffb640 10 monclient: tick 2024-09-16T14:50:58.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:58.544+0000 7ffb79ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:58.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:58.544+0000 7ffb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:50:59.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:59.544+0000 7ffb79ffb640 10 monclient: tick 2024-09-16T14:50:59.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:59.544+0000 7ffb79ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:50:59.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:50:59.544+0000 7ffb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:00.062 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:51:00.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da6e45640 10 monclient: get_monmap_and_config 2024-09-16T14:51:00.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da6e45640 10 monclient: build_initial_monmap 2024-09-16T14:51:00.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da6e45640 10 monclient: monmap: 2024-09-16T14:51:00.198 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:00.198 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:00.198 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:00.198870+0000 2024-09-16T14:51:00.198 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:00.198870+0000 2024-09-16T14:51:00.198 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:00.198 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:00.198 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:00.198 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:00.198 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:00.198 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:00.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da6e45640 1 Processor -- start 2024-09-16T14:51:00.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da6e45640 1 -- start start 2024-09-16T14:51:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da6e45640 10 monclient: init 2024-09-16T14:51:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da6e45640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da6e45640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:51:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da6e45640 10 monclient(hunting): picked mon.noname-b con 0x7f0da009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da6e45640 10 monclient(hunting): picked mon.noname-c con 0x7f0da0132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da6e45640 10 monclient(hunting): picked mon.noname-a con 0x7f0da0138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da6e45640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da009d260 con 0x7f0da0138ff0 2024-09-16T14:51:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da6e45640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da0132a80 con 0x7f0da0132650 2024-09-16T14:51:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da6e45640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da009cf60 con 0x7f0da009cb80 2024-09-16T14:51:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da6e45640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da6e45640 10 monclient(hunting): authenticate will time out at 1484.318882s 2024-09-16T14:51:00.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da53e2640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f0da0138ff0 legacy=0x7f0da013b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:00.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da4be1640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0da009cb80 0x7f0da00717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35262/0 (socket says 172.21.15.26:35262) 2024-09-16T14:51:00.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.197+0000 7f0da4be1640 1 -- 172.21.15.26:0/2760978097 learned_addr learned my addr 172.21.15.26:0/2760978097 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:00.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.401+0000 7f0da53e2640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6789/0 conn(0x7f0da0138ff0 legacy=0x7f0da013b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:00.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:00.543+0000 7ffb79ffb640 10 monclient: tick 2024-09-16T14:51:00.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:00.543+0000 7ffb79ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:51:00.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:00.543+0000 7ffb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:00.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:00.805+0000 7f0da53e2640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6789/0 conn(0x7f0da0138ff0 legacy=0x7f0da013b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:01.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:01.543+0000 7ffb79ffb640 10 monclient: tick 2024-09-16T14:51:01.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:01.543+0000 7ffb79ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:51:01.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:01.543+0000 7ffb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:01.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:01.609+0000 7f0da53e2640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6789/0 conn(0x7f0da0138ff0 legacy=0x7f0da013b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:02.543+0000 7ffb79ffb640 10 monclient: tick 2024-09-16T14:51:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:02.543+0000 7ffb79ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:51:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:02.543+0000 7ffb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.197+0000 7f0d9e7fc640 10 monclient(hunting): tick 2024-09-16T14:51:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.197+0000 7f0d9e7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:51:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.197+0000 7f0d9e7fc640 1 monclient(hunting): continuing hunt 2024-09-16T14:51:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.197+0000 7f0d9e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:51:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.197+0000 7f0d9e7fc640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6790/0 conn(0x7f0da009cb80 legacy=0x7f0da00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.197+0000 7f0d9e7fc640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.189:6789/0 conn(0x7f0da0132650 legacy=0x7f0da01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.197+0000 7f0d9e7fc640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6789/0 conn(0x7f0da0138ff0 legacy=0x7f0da013b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.197+0000 7f0d9e7fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:51:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.197+0000 7f0d9e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f0d800036d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.197+0000 7f0d9e7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f0da0132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.197+0000 7f0d9e7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f0da0138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.197+0000 7f0d9e7fc640 1 -- 172.21.15.26:0/2760978097 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da009f2d0 con 0x7f0da0138ff0 2024-09-16T14:51:03.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.197+0000 7f0d9e7fc640 1 -- 172.21.15.26:0/2760978097 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da009d260 con 0x7f0da0132650 2024-09-16T14:51:03.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.197+0000 7f0d9e7fc640 1 -- 172.21.15.26:0/2760978097 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d800103e0 con 0x7f0d800036d0 2024-09-16T14:51:03.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.197+0000 7f0d9e7fc640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:03.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.197+0000 7f0da53e2640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6789/0 conn(0x7f0da0138ff0 legacy=0x7f0d8000cc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:03.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.401+0000 7f0da53e2640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6789/0 conn(0x7f0da0138ff0 legacy=0x7f0d8000cc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:03.543+0000 7ffb79ffb640 10 monclient: tick 2024-09-16T14:51:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:03.543+0000 7ffb79ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:51:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:03.543+0000 7ffb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:03.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:03.805+0000 7f0da53e2640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6789/0 conn(0x7f0da0138ff0 legacy=0x7f0d8000cc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:04.547+0000 7ffb79ffb640 10 monclient: tick 2024-09-16T14:51:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:04.547+0000 7ffb79ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:51:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:04.547+0000 7ffb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:04.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:04.609+0000 7f0da53e2640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6789/0 conn(0x7f0da0138ff0 legacy=0x7f0d8000cc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:05.547+0000 7ffb79ffb640 10 monclient: tick 2024-09-16T14:51:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:05.547+0000 7ffb79ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:51:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:05.547+0000 7ffb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:05.547+0000 7ffb79ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.197+0000 7f0d9e7fc640 10 monclient(hunting): tick 2024-09-16T14:51:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.197+0000 7f0d9e7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:51:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.197+0000 7f0d9e7fc640 1 monclient(hunting): continuing hunt 2024-09-16T14:51:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.197+0000 7f0d9e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:51:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.197+0000 7f0d9e7fc640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6790/0 conn(0x7f0d800036d0 legacy=0x7f0d80005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.197+0000 7f0d9e7fc640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.189:6789/0 conn(0x7f0da0132650 legacy=0x7f0d800093b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.197+0000 7f0d9e7fc640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6789/0 conn(0x7f0da0138ff0 legacy=0x7f0d8000cc10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:06.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.197+0000 7f0d9e7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:51:06.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.197+0000 7f0d9e7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f0da0138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:06.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.197+0000 7f0d9e7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f0d8000fb70 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:06.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.197+0000 7f0d9e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f0d800036d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:06.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.197+0000 7f0d9e7fc640 1 -- 172.21.15.26:0/2760978097 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d80011650 con 0x7f0da0138ff0 2024-09-16T14:51:06.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.197+0000 7f0d9e7fc640 1 -- 172.21.15.26:0/2760978097 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da009f2d0 con 0x7f0d8000fb70 2024-09-16T14:51:06.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.197+0000 7f0da4be1640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6789/0 conn(0x7f0da0138ff0 legacy=0x7f0d8000f460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:06.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.197+0000 7f0d9e7fc640 1 -- 172.21.15.26:0/2760978097 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d80010170 con 0x7f0d800036d0 2024-09-16T14:51:06.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.197+0000 7f0d9e7fc640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:06.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.401+0000 7f0da4be1640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6789/0 conn(0x7f0da0138ff0 legacy=0x7f0d8000f460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:06.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:06.547+0000 7ffb79ffb640 10 monclient: tick 2024-09-16T14:51:06.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:06.547+0000 7ffb79ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:51:06.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:06.547+0000 7ffb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:06.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:06.805+0000 7f0da4be1640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6789/0 conn(0x7f0da0138ff0 legacy=0x7f0d8000f460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:07.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:07.543+0000 7ffb79ffb640 10 monclient: tick 2024-09-16T14:51:07.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:07.543+0000 7ffb79ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:51:07.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:07.543+0000 7ffb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:07.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.609+0000 7f0da4be1640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6789/0 conn(0x7f0da0138ff0 legacy=0x7f0d8000f460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.757+0000 7f0d9f7fe640 1 -- 172.21.15.26:0/2760978097 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626794760 0 0) 0x7f0da009f2d0 con 0x7f0d8000fb70 2024-09-16T14:51:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.757+0000 7f0d9f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.757+0000 7f0d9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.757+0000 7f0d9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.757+0000 7f0d9f7fe640 10 monclient(hunting): my global_id is 184171 2024-09-16T14:51:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.757+0000 7f0d9f7fe640 1 -- 172.21.15.26:0/2760978097 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d8c003530 con 0x7f0d8000fb70 2024-09-16T14:51:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 1 -- 172.21.15.26:0/2760978097 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2029526398 0 0) 0x7f0d8c003530 con 0x7f0d8000fb70 2024-09-16T14:51:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6790/0 conn(0x7f0d800036d0 legacy=0x7f0d80005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.26:6789/0 conn(0x7f0da0138ff0 legacy=0x7f0d8000f460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:51:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 1 monclient: found mon.noname-c 2024-09-16T14:51:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:51:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 1 -- 172.21.15.26:0/2760978097 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d800093c0 con 0x7f0d8000fb70 2024-09-16T14:51:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:51:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:51:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 1 -- 172.21.15.26:0/2760978097 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0d88001520 con 0x7f0d8000fb70 2024-09-16T14:51:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 10 monclient: dump: 2024-09-16T14:51:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:07.763 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:07.763 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:07.763 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:07.763 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:07.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0da6e45640 5 monclient: authenticate success, global_id 184171 2024-09-16T14:51:07.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0da6e45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 1 -- 172.21.15.26:0/2760978097 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d88001740 con 0x7f0d8000fb70 2024-09-16T14:51:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 1 -- 172.21.15.26:0/2760978097 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0d88001b50 con 0x7f0d8000fb70 2024-09-16T14:51:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0d9f7fe640 10 monclient: dump: 2024-09-16T14:51:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:07.765 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0da6e45640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0da6e45640 4 set_mon_vals no callback set 2024-09-16T14:51:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0da6e45640 10 monclient: shutdown 2024-09-16T14:51:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0da6e45640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0da6e45640 1 -- 172.21.15.26:0/2760978097 >> v1:172.21.15.189:6789/0 conn(0x7f0d8000fb70 legacy=0x7f0d8000cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0da6e45640 1 -- 172.21.15.26:0/2760978097 shutdown_connections 2024-09-16T14:51:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0da6e45640 1 -- 172.21.15.26:0/2760978097 >> 172.21.15.26:0/2760978097 conn(0x7f0da009fb50 msgr2=0x7f0da009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0da6e45640 1 -- 172.21.15.26:0/2760978097 shutdown_connections 2024-09-16T14:51:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.761+0000 7f0da6e45640 1 -- 172.21.15.26:0/2760978097 wait complete. 2024-09-16T14:51:07.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da6e45640 10 monclient: build_initial_monmap 2024-09-16T14:51:07.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da6e45640 10 monclient: monmap: 2024-09-16T14:51:07.767 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:07.767 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:07.768 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da6e45640 1 Processor -- start 2024-09-16T14:51:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da6e45640 1 -- start start 2024-09-16T14:51:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da6e45640 10 monclient: init 2024-09-16T14:51:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da6e45640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da6e45640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:51:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da6e45640 10 monclient(hunting): picked mon.noname-c con 0x7f0d800036d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da6e45640 10 monclient(hunting): picked mon.noname-a con 0x7f0d8000fb70 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da6e45640 10 monclient(hunting): picked mon.noname-b con 0x7f0da0138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da6e45640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da00a2610 con 0x7f0d8000fb70 2024-09-16T14:51:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da6e45640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da006f2b0 con 0x7f0da0138ff0 2024-09-16T14:51:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0d9ffff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f0d8000fb70 legacy=0x7f0da006eba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da6e45640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da0138a40 con 0x7f0d800036d0 2024-09-16T14:51:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da6e45640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da6e45640 10 monclient(hunting): authenticate will time out at 1491.887213s 2024-09-16T14:51:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da4be1640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0d800036d0 0x7f0da006e490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:34562/0 (socket says 172.21.15.26:34562) 2024-09-16T14:51:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.765+0000 7f0da4be1640 1 -- 172.21.15.26:0/2149763165 learned_addr learned my addr 172.21.15.26:0/2149763165 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:07.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 1 -- 172.21.15.26:0/2149763165 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596055878 0 0) 0x7f0da006f2b0 con 0x7f0da0138ff0 2024-09-16T14:51:07.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:51:07.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:07.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:07.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 10 monclient(hunting): my global_id is 184174 2024-09-16T14:51:07.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 1 -- 172.21.15.26:0/2149763165 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d800051e0 con 0x7f0da0138ff0 2024-09-16T14:51:07.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 1 -- 172.21.15.26:0/2149763165 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 973314360 0 0) 0x7f0d800051e0 con 0x7f0da0138ff0 2024-09-16T14:51:07.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 1 -- 172.21.15.26:0/2149763165 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0da006f2b0 con 0x7f0da0138ff0 2024-09-16T14:51:07.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 1 -- 172.21.15.26:0/2149763165 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0d980044f0 con 0x7f0da0138ff0 2024-09-16T14:51:07.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:07.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:07.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 10 monclient(hunting): dump: 2024-09-16T14:51:07.772 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:07.772 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:07.772 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:07.772 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:07.772 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:07.773 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:07.773 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:07.773 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:07.773 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:07.773 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:07.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:07.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 1 -- 172.21.15.26:0/2149763165 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2139009689 0 0) 0x7f0da006f2b0 con 0x7f0da0138ff0 2024-09-16T14:51:07.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 1 -- 172.21.15.26:0/2149763165 >> v1:172.21.15.26:6790/0 conn(0x7f0d800036d0 legacy=0x7f0da006e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:07.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 1 -- 172.21.15.26:0/2149763165 >> v1:172.21.15.26:6789/0 conn(0x7f0d8000fb70 legacy=0x7f0da006eba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:07.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:51:07.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 1 monclient: found mon.b 2024-09-16T14:51:07.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:07.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:07.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 1 -- 172.21.15.26:0/2149763165 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da009a240 con 0x7f0da0138ff0 2024-09-16T14:51:07.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:51:07.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:51:07.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:07.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0da6e45640 5 monclient: authenticate success, global_id 184174 2024-09-16T14:51:07.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0da6e45640 10 monclient: _renew_subs 2024-09-16T14:51:07.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0da6e45640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:07.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0da6e45640 1 -- 172.21.15.26:0/2149763165 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da0070130 con 0x7f0da0138ff0 2024-09-16T14:51:07.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0da6e45640 10 monclient: _renew_subs 2024-09-16T14:51:07.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0da6e45640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:07.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0da6e45640 1 -- 172.21.15.26:0/2149763165 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da0134070 con 0x7f0da0138ff0 2024-09-16T14:51:07.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 1 -- 172.21.15.26:0/2149763165 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d98004940 con 0x7f0da0138ff0 2024-09-16T14:51:07.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:07.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 1 -- 172.21.15.26:0/2149763165 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0d98004c50 con 0x7f0da0138ff0 2024-09-16T14:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9effd640 4 set_mon_vals no callback set 2024-09-16T14:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 10 monclient: dump: 2024-09-16T14:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:07.775 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:07.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0da6e45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:07.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0da6e45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:07.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0da6e45640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:07.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0da6e45640 1 -- 172.21.15.26:0/2149763165 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d84005050 con 0x7f0da0138ff0 2024-09-16T14:51:07.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 1 -- 172.21.15.26:0/2149763165 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0d98004db0 con 0x7f0da0138ff0 2024-09-16T14:51:07.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.769+0000 7f0d9d7fa640 1 -- 172.21.15.26:0/2149763165 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(246..246 src has 169..246) v4 ==== 7496+0+0 (unknown 771380485 0 0) 0x7f0d980441d0 con 0x7f0da0138ff0 2024-09-16T14:51:07.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.773+0000 7f0da4be1640 10 monclient: get_auth_request con 0x7f0d80037190 auth_method 0 2024-09-16T14:51:07.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.785+0000 7f0d9d7fa640 1 -- 172.21.15.26:0/2149763165 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0d98014640 con 0x7f0da0138ff0 2024-09-16T14:51:07.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.785+0000 7f0d9d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:07.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:07.785+0000 7f0d9d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:08.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.025+0000 7f0da6e45640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:51:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.025+0000 7f0da6e45640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:51:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.025+0000 7f0da6e45640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.025+0000 7f0da6e45640 1 -- 172.21.15.26:0/2149763165 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0d84005880 con 0x7f0da0138ff0 2024-09-16T14:51:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.025+0000 7f0d9d7fa640 1 -- 172.21.15.26:0/2149763165 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2801407082) 0x7f0d98012ce0 con 0x7f0da0138ff0 2024-09-16T14:51:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.025+0000 7f0d9d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:51:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.025+0000 7f0d9d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:08.457 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":82,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:51:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.029+0000 7f0da6e45640 1 -- 172.21.15.26:0/2149763165 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0d80037190 legacy=0x7f0d80039650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.029+0000 7f0da6e45640 10 monclient: shutdown 2024-09-16T14:51:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.029+0000 7f0da6e45640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.029+0000 7f0da6e45640 1 -- 172.21.15.26:0/2149763165 >> v1:172.21.15.189:6789/0 conn(0x7f0da0138ff0 legacy=0x7f0da01cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.029+0000 7f0da6e45640 1 -- 172.21.15.26:0/2149763165 shutdown_connections 2024-09-16T14:51:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.029+0000 7f0da6e45640 1 -- 172.21.15.26:0/2149763165 >> 172.21.15.26:0/2149763165 conn(0x7f0da009fb50 msgr2=0x7f0da0131670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.029+0000 7f0da6e45640 1 -- 172.21.15.26:0/2149763165 shutdown_connections 2024-09-16T14:51:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.029+0000 7f0da6e45640 1 -- 172.21.15.26:0/2149763165 wait complete. 2024-09-16T14:51:08.459 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:51:08.459 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:51:08.459 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:51:08.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.547+0000 7ffb79ffb640 10 monclient: tick 2024-09-16T14:51:08.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.547+0000 7ffb79ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:51:08.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.547+0000 7ffb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:08.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.593+0000 7f22aa736640 10 monclient: get_monmap_and_config 2024-09-16T14:51:08.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.593+0000 7f22aa736640 10 monclient: build_initial_monmap 2024-09-16T14:51:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.593+0000 7f22aa736640 10 monclient: monmap: 2024-09-16T14:51:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:08.597315+0000 2024-09-16T14:51:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:08.597315+0000 2024-09-16T14:51:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:08.597 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 1 Processor -- start 2024-09-16T14:51:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 1 -- start start 2024-09-16T14:51:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 10 monclient: init 2024-09-16T14:51:08.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:51:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 10 monclient(hunting): picked mon.noname-b con 0x7f22a409cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 10 monclient(hunting): picked mon.noname-a con 0x7f22a409e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 10 monclient(hunting): picked mon.noname-c con 0x7f22a409d1a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a409d5b0 con 0x7f22a409e7a0 2024-09-16T14:51:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a4131a90 con 0x7f22a409d1a0 2024-09-16T14:51:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a409aff0 con 0x7f22a409cdc0 2024-09-16T14:51:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 10 monclient(hunting): authenticate will time out at 1492.717230s 2024-09-16T14:51:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a37fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f22a409e7a0 legacy=0x7f22a4131380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a3fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f22a409cdc0 0x7f22a4130c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:34566/0 (socket says 172.21.15.26:34566) 2024-09-16T14:51:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a3fff640 1 -- 172.21.15.26:0/3616588020 learned_addr learned my addr 172.21.15.26:0/3616588020 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 1 -- 172.21.15.26:0/3616588020 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640062004 0 0) 0x7f22a409aff0 con 0x7f22a409cdc0 2024-09-16T14:51:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:51:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 10 monclient(hunting): my global_id is 154883 2024-09-16T14:51:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 1 -- 172.21.15.26:0/3616588020 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f228c003530 con 0x7f22a409cdc0 2024-09-16T14:51:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 1 -- 172.21.15.26:0/3616588020 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 114507404 0 0) 0x7f228c003530 con 0x7f22a409cdc0 2024-09-16T14:51:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 1 -- 172.21.15.26:0/3616588020 >> v1:172.21.15.189:6789/0 conn(0x7f22a409d1a0 legacy=0x7f22a4139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 1 -- 172.21.15.26:0/3616588020 >> v1:172.21.15.26:6789/0 conn(0x7f22a409e7a0 legacy=0x7f22a4131380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:51:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 1 monclient: found mon.noname-b 2024-09-16T14:51:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:08.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:51:08.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 1 -- 172.21.15.26:0/3616588020 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a409b2d0 con 0x7f22a409cdc0 2024-09-16T14:51:08.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:51:08.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:51:08.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:08.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 1 -- 172.21.15.26:0/3616588020 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2298001620 con 0x7f22a409cdc0 2024-09-16T14:51:08.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:08.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:08.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 10 monclient: dump: 2024-09-16T14:51:08.606 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:08.606 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:08.606 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.606 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.607 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:08.607 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:08.607 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:08.607 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:08.607 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:08.607 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:08.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 5 monclient: authenticate success, global_id 154883 2024-09-16T14:51:08.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:08.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 1 -- 172.21.15.26:0/3616588020 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22980018d0 con 0x7f22a409cdc0 2024-09-16T14:51:08.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:08.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 1 -- 172.21.15.26:0/3616588020 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2298001ca0 con 0x7f22a409cdc0 2024-09-16T14:51:08.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:08.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:08.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22a2ffd640 10 monclient: dump: 2024-09-16T14:51:08.608 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:08.608 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:08.608 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.608 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.608 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:08.608 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:08.608 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:08.608 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:08.608 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:08.608 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:08.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:08.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 4 set_mon_vals no callback set 2024-09-16T14:51:08.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 10 monclient: shutdown 2024-09-16T14:51:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 1 -- 172.21.15.26:0/3616588020 >> v1:172.21.15.26:6790/0 conn(0x7f22a409cdc0 legacy=0x7f22a4130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 1 -- 172.21.15.26:0/3616588020 shutdown_connections 2024-09-16T14:51:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 1 -- 172.21.15.26:0/3616588020 >> 172.21.15.26:0/3616588020 conn(0x7f22a409fb50 msgr2=0x7f22a409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 1 -- 172.21.15.26:0/3616588020 shutdown_connections 2024-09-16T14:51:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 1 -- 172.21.15.26:0/3616588020 wait complete. 2024-09-16T14:51:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 10 monclient: build_initial_monmap 2024-09-16T14:51:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 10 monclient: monmap: 2024-09-16T14:51:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:08.610 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 1 Processor -- start 2024-09-16T14:51:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 1 -- start start 2024-09-16T14:51:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.597+0000 7f22aa736640 10 monclient: init 2024-09-16T14:51:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:51:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 10 monclient(hunting): picked mon.noname-b con 0x7f22a409cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 10 monclient(hunting): picked mon.noname-c con 0x7f22a409d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 10 monclient(hunting): picked mon.noname-a con 0x7f22a409e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a411d620 con 0x7f22a409e7a0 2024-09-16T14:51:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a40a15b0 con 0x7f22a409cdc0 2024-09-16T14:51:08.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a4130b20 con 0x7f22a409d1a0 2024-09-16T14:51:08.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a8cd3640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f22a409e7a0 legacy=0x7f22a41d1cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:08.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:08.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 10 monclient(hunting): authenticate will time out at 1492.720422s 2024-09-16T14:51:08.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a37fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f22a409d1a0 0x7f22a41ce590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:34568/0 (socket says 172.21.15.26:34568) 2024-09-16T14:51:08.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a37fe640 1 -- 172.21.15.26:0/3789701856 learned_addr learned my addr 172.21.15.26:0/3789701856 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:08.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282925094 0 0) 0x7f22a4130b20 con 0x7f22a409d1a0 2024-09-16T14:51:08.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:51:08.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:08.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:08.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient(hunting): my global_id is 154886 2024-09-16T14:51:08.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2290003560 con 0x7f22a409d1a0 2024-09-16T14:51:08.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1932202808 0 0) 0x7f2290003560 con 0x7f22a409d1a0 2024-09-16T14:51:08.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22a4130b20 con 0x7f22a409d1a0 2024-09-16T14:51:08.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2294004160 con 0x7f22a409d1a0 2024-09-16T14:51:08.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:08.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:08.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient(hunting): dump: 2024-09-16T14:51:08.612 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:08.612 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:08.612 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.612 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.612 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:08.612 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:08.612 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:08.612 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:08.613 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049173174 0 0) 0x7f22a40a15b0 con 0x7f22a409cdc0 2024-09-16T14:51:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:51:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient(hunting): my global_id is 184180 2024-09-16T14:51:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2290003560 con 0x7f22a409cdc0 2024-09-16T14:51:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 768846969 0 0) 0x7f22a4130b20 con 0x7f22a409d1a0 2024-09-16T14:51:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 >> v1:172.21.15.189:6789/0 conn(0x7f22a409cdc0 legacy=0x7f22a412fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 >> v1:172.21.15.26:6789/0 conn(0x7f22a409e7a0 legacy=0x7f22a41d1cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:51:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 monclient: found mon.c 2024-09-16T14:51:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a409a240 con 0x7f22a409d1a0 2024-09-16T14:51:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient: _finish_auth 0 2024-09-16T14:51:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:51:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 5 monclient: authenticate success, global_id 154886 2024-09-16T14:51:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 10 monclient: _renew_subs 2024-09-16T14:51:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 1 -- 172.21.15.26:0/3789701856 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a411d340 con 0x7f22a409d1a0 2024-09-16T14:51:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 10 monclient: _renew_subs 2024-09-16T14:51:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 1 -- 172.21.15.26:0/3789701856 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a4130350 con 0x7f22a409d1a0 2024-09-16T14:51:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2294004380 con 0x7f22a409d1a0 2024-09-16T14:51:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2294004710 con 0x7f22a409d1a0 2024-09-16T14:51:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a27fc640 4 set_mon_vals no callback set 2024-09-16T14:51:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient: dump: 2024-09-16T14:51:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:08.616 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22aa736640 1 -- 172.21.15.26:0/3789701856 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2284006590 con 0x7f2284002830 2024-09-16T14:51:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2294004870 con 0x7f22a409d1a0 2024-09-16T14:51:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(246..246 src has 169..246) v4 ==== 7496+0+0 (unknown 771380485 0 0) 0x7f2294044060 con 0x7f22a409d1a0 2024-09-16T14:51:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227777499 0 0) 0x7f2284006590 con 0x7f2284002830 2024-09-16T14:51:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:51:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient(hunting): my global_id is 154889 2024-09-16T14:51:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22a4130b20 con 0x7f2284002830 2024-09-16T14:51:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1151834658 0 0) 0x7f22a4130b20 con 0x7f2284002830 2024-09-16T14:51:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a8cd3640 10 monclient: get_auth_request con 0x7f2290036250 auth_method 0 2024-09-16T14:51:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2284006590 con 0x7f2284002830 2024-09-16T14:51:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22980016a0 con 0x7f2284002830 2024-09-16T14:51:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient: dump: 2024-09-16T14:51:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:08.618 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4278449708 0 0) 0x7f2284006590 con 0x7f2284002830 2024-09-16T14:51:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f228400a0e0 con 0x7f2284002830 2024-09-16T14:51:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2298001c10 con 0x7f2284002830 2024-09-16T14:51:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 >> v1:172.21.15.26:6790/0 conn(0x7f2284002830 legacy=0x7f2284004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.601+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 >> v1:172.21.15.26:6790/0 conn(0x7f2284002830 legacy=0x7f2284004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22aa736640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:51:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22aa736640 1 -- 172.21.15.26:0/3789701856 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2284009da0 con 0x7f22840090c0 2024-09-16T14:51:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988719076 0 0) 0x7f2284009da0 con 0x7f22840090c0 2024-09-16T14:51:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:51:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 10 monclient(hunting): my global_id is 154892 2024-09-16T14:51:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2284006590 con 0x7f22840090c0 2024-09-16T14:51:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3822822704 0 0) 0x7f2284006590 con 0x7f22840090c0 2024-09-16T14:51:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2284009da0 con 0x7f22840090c0 2024-09-16T14:51:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2298004ce0 con 0x7f22840090c0 2024-09-16T14:51:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 10 monclient: dump: 2024-09-16T14:51:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:08.621 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2975753984 0 0) 0x7f2284009da0 con 0x7f22840090c0 2024-09-16T14:51:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f228400c290 con 0x7f22840090c0 2024-09-16T14:51:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4202 (unknown 0 0 533233301) 0x7f2298004f50 con 0x7f22840090c0 2024-09-16T14:51:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:51:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 >> v1:172.21.15.26:6790/0 conn(0x7f22840090c0 legacy=0x7f2284009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a0ff9640 1 -- 172.21.15.26:0/3789701856 >> v1:172.21.15.26:6790/0 conn(0x7f22840090c0 legacy=0x7f2284009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:08.622 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:51:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:51:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:51:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:51:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 82, 2024-09-16T14:51:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:51:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:51:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:51:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:51:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:51:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:51:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:51:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:51:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:51:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:51:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:51:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:51:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:51:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:51:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:51:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:51:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:51:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:51:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:51:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:51:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:51:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:51:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:51:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:51:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:51:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:51:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:51:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:51:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:51:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:08.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:51:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:51:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:51:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:51:08.629 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:51:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:51:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:51:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:51:08.630 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:51:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:51:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:51:08.631 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:51:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:51:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:51:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:51:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:08.632 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:51:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:51:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:51:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:51:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:08.633 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:08.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.634 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:51:08.634 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:51:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22aa736640 1 -- 172.21.15.26:0/3789701856 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2290036250 legacy=0x7f2290038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22aa736640 10 monclient: shutdown 2024-09-16T14:51:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22aa736640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22aa736640 1 -- 172.21.15.26:0/3789701856 >> v1:172.21.15.26:6790/0 conn(0x7f22a409d1a0 legacy=0x7f22a41ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a8cd3640 1 -- 172.21.15.26:0/3789701856 reap_dead start 2024-09-16T14:51:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22a8cd3640 1 -- 172.21.15.26:0/3789701856 reap_dead start 2024-09-16T14:51:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22aa736640 1 -- 172.21.15.26:0/3789701856 shutdown_connections 2024-09-16T14:51:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22aa736640 1 -- 172.21.15.26:0/3789701856 >> 172.21.15.26:0/3789701856 conn(0x7f22a409fb50 msgr2=0x7f22a4131070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22aa736640 1 -- 172.21.15.26:0/3789701856 shutdown_connections 2024-09-16T14:51:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.609+0000 7f22aa736640 1 -- 172.21.15.26:0/3789701856 wait complete. 2024-09-16T14:51:08.636 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:51:08.636 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T14:51:08.636 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:51:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.761+0000 7fa38bbf6640 10 monclient: get_monmap_and_config 2024-09-16T14:51:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.761+0000 7fa38bbf6640 10 monclient: build_initial_monmap 2024-09-16T14:51:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.761+0000 7fa38bbf6640 10 monclient: monmap: 2024-09-16T14:51:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:08.763181+0000 2024-09-16T14:51:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:08.763181+0000 2024-09-16T14:51:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:08.763 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.761+0000 7fa38bbf6640 1 Processor -- start 2024-09-16T14:51:08.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.761+0000 7fa38bbf6640 1 -- start start 2024-09-16T14:51:08.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.761+0000 7fa38bbf6640 10 monclient: init 2024-09-16T14:51:08.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:08.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:51:08.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 10 monclient(hunting): picked mon.noname-c con 0x7fa38406f5d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 10 monclient(hunting): picked mon.noname-b con 0x7fa384131090 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 10 monclient(hunting): picked mon.noname-a con 0x7fa384131c10 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38406f1b0 con 0x7fa384131c10 2024-09-16T14:51:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa384071920 con 0x7fa38406f5d0 2024-09-16T14:51:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38406dc30 con 0x7fa384131090 2024-09-16T14:51:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 10 monclient(hunting): authenticate will time out at 1492.885156s 2024-09-16T14:51:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38a193640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fa384131c10 legacy=0x7fa38413a400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa389191640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa384131090 0x7fa384131500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:34608/0 (socket says 172.21.15.26:34608) 2024-09-16T14:51:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa389191640 1 -- 172.21.15.26:0/3649537600 learned_addr learned my addr 172.21.15.26:0/3649537600 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 1 -- 172.21.15.26:0/3649537600 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80027323 0 0) 0x7fa38406dc30 con 0x7fa384131090 2024-09-16T14:51:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 10 monclient(hunting): _negotiate 2024-09-16T14:51:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 10 monclient(hunting): my global_id is 154895 2024-09-16T14:51:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 1 -- 172.21.15.26:0/3649537600 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa370003530 con 0x7fa384131090 2024-09-16T14:51:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 1 -- 172.21.15.26:0/3649537600 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1750960005 0 0) 0x7fa370003530 con 0x7fa384131090 2024-09-16T14:51:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 1 -- 172.21.15.26:0/3649537600 >> v1:172.21.15.189:6789/0 conn(0x7fa38406f5d0 legacy=0x7fa384073530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 1 -- 172.21.15.26:0/3649537600 >> v1:172.21.15.26:6789/0 conn(0x7fa384131c10 legacy=0x7fa38413a400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 10 monclient: _finish_hunting 0 2024-09-16T14:51:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 1 monclient: found mon.noname-b 2024-09-16T14:51:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:51:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 1 -- 172.21.15.26:0/3649537600 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38406e110 con 0x7fa384131090 2024-09-16T14:51:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 10 monclient: _finish_auth 0 2024-09-16T14:51:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 10 monclient: _check_auth_tickets 2024-09-16T14:51:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 1 -- 172.21.15.26:0/3649537600 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa374001620 con 0x7fa384131090 2024-09-16T14:51:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 5 monclient: authenticate success, global_id 154895 2024-09-16T14:51:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 10 monclient: dump: 2024-09-16T14:51:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:08.771 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 1 -- 172.21.15.26:0/3649537600 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa374001940 con 0x7fa384131090 2024-09-16T14:51:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 1 -- 172.21.15.26:0/3649537600 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa374001d10 con 0x7fa384131090 2024-09-16T14:51:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa388990640 10 monclient: dump: 2024-09-16T14:51:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:08.772 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 4 set_mon_vals no callback set 2024-09-16T14:51:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 10 monclient: shutdown 2024-09-16T14:51:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 1 -- 172.21.15.26:0/3649537600 >> v1:172.21.15.26:6790/0 conn(0x7fa384131090 legacy=0x7fa384131500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 1 -- 172.21.15.26:0/3649537600 shutdown_connections 2024-09-16T14:51:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 1 -- 172.21.15.26:0/3649537600 >> 172.21.15.26:0/3649537600 conn(0x7fa3840a03d0 msgr2=0x7fa3840a07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.765+0000 7fa38bbf6640 1 -- 172.21.15.26:0/3649537600 shutdown_connections 2024-09-16T14:51:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 1 -- 172.21.15.26:0/3649537600 wait complete. 2024-09-16T14:51:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 10 monclient: build_initial_monmap 2024-09-16T14:51:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 10 monclient: monmap: 2024-09-16T14:51:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:08.773 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:08.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.763+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h32"}]=0 removed item id -112 name 'h32' from crush map v247) v1 ==== 121+0+0 (unknown 4085141497 0 0) 0x7ffb74012de0 con 0x7ffb84099a70 2024-09-16T14:51:08.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.763+0000 7ffb7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T14:51:08.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.763+0000 7ffb7affd640 10 monclient: _finish_command 2 = system:0 removed item id -112 name 'h32' from crush map 2024-09-16T14:51:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 1 Processor -- start 2024-09-16T14:51:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 1 -- start start 2024-09-16T14:51:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 10 monclient: init 2024-09-16T14:51:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:51:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 10 monclient(hunting): picked mon.noname-b con 0x7fa38406f5d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 10 monclient(hunting): picked mon.noname-a con 0x7fa384131090 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 10 monclient(hunting): picked mon.noname-c con 0x7fa384131c10 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa384123b20 con 0x7fa384131090 2024-09-16T14:51:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3840998b0 con 0x7fa38406f5d0 2024-09-16T14:51:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38409a6e0 con 0x7fa384131c10 2024-09-16T14:51:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa389191640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fa384131090 legacy=0x7fa3840991a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 10 monclient(hunting): authenticate will time out at 1492.889454s 2024-09-16T14:51:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38a193640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa384131c10 0x7fa3841d1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:34620/0 (socket says 172.21.15.26:34620) 2024-09-16T14:51:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38a193640 1 -- 172.21.15.26:0/1143349903 learned_addr learned my addr 172.21.15.26:0/1143349903 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184562470 0 0) 0x7fa38409a6e0 con 0x7fa384131c10 2024-09-16T14:51:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient(hunting): my global_id is 154898 2024-09-16T14:51:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa368003530 con 0x7fa384131c10 2024-09-16T14:51:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164694635 0 0) 0x7fa3840998b0 con 0x7fa38406f5d0 2024-09-16T14:51:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient(hunting): my global_id is 184183 2024-09-16T14:51:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa38409a6e0 con 0x7fa38406f5d0 2024-09-16T14:51:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1816323070 0 0) 0x7fa368003530 con 0x7fa384131c10 2024-09-16T14:51:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3840998b0 con 0x7fa384131c10 2024-09-16T14:51:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa37c0043c0 con 0x7fa384131c10 2024-09-16T14:51:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient(hunting): dump: 2024-09-16T14:51:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:08.782 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 701492433 0 0) 0x7fa3840998b0 con 0x7fa384131c10 2024-09-16T14:51:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 >> v1:172.21.15.189:6789/0 conn(0x7fa38406f5d0 legacy=0x7fa384098a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 >> v1:172.21.15.26:6789/0 conn(0x7fa384131090 legacy=0x7fa3840991a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:51:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 1 monclient: found mon.c 2024-09-16T14:51:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3840720a0 con 0x7fa384131c10 2024-09-16T14:51:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:51:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:51:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 5 monclient: authenticate success, global_id 154898 2024-09-16T14:51:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 10 monclient: _renew_subs 2024-09-16T14:51:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 1 -- 172.21.15.26:0/1143349903 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa384070990 con 0x7fa384131c10 2024-09-16T14:51:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 10 monclient: _renew_subs 2024-09-16T14:51:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 1 -- 172.21.15.26:0/1143349903 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa384130a70 con 0x7fa384131c10 2024-09-16T14:51:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa37c004630 con 0x7fa384131c10 2024-09-16T14:51:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa37c0049c0 con 0x7fa384131c10 2024-09-16T14:51:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37bfff640 4 set_mon_vals no callback set 2024-09-16T14:51:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 10 monclient: dump: 2024-09-16T14:51:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:08.784 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa38bbf6640 1 -- 172.21.15.26:0/1143349903 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa36c006590 con 0x7fa36c002830 2024-09-16T14:51:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa37c004be0 con 0x7fa384131c10 2024-09-16T14:51:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.769+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(247..247 src has 169..247) v4 ==== 7283+0+0 (unknown 4191992233 0 0) 0x7fa37c044140 con 0x7fa384131c10 2024-09-16T14:51:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562254675 0 0) 0x7fa36c006590 con 0x7fa36c002830 2024-09-16T14:51:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 10 monclient(hunting): my global_id is 184186 2024-09-16T14:51:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3840998b0 con 0x7fa36c002830 2024-09-16T14:51:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa389191640 10 monclient: get_auth_request con 0x7fa3680360e0 auth_method 0 2024-09-16T14:51:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1972997533 0 0) 0x7fa3840998b0 con 0x7fa36c002830 2024-09-16T14:51:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa36c006590 con 0x7fa36c002830 2024-09-16T14:51:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa380001750 con 0x7fa36c002830 2024-09-16T14:51:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 10 monclient: dump: 2024-09-16T14:51:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:08.786 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 426705438 0 0) 0x7fa36c006590 con 0x7fa36c002830 2024-09-16T14:51:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa36c00a0e0 con 0x7fa36c002830 2024-09-16T14:51:08.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa380001b00 con 0x7fa36c002830 2024-09-16T14:51:08.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:08.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:08.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 >> v1:172.21.15.189:6789/0 conn(0x7fa36c002830 legacy=0x7fa36c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.773+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 >> v1:172.21.15.189:6789/0 conn(0x7fa36c002830 legacy=0x7fa36c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:08.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.777+0000 7fa38bbf6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:51:08.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.777+0000 7fa38bbf6640 1 -- 172.21.15.26:0/1143349903 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa36c009d80 con 0x7fa36c0090a0 2024-09-16T14:51:08.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741406914 0 0) 0x7fa36c009d80 con 0x7fa36c0090a0 2024-09-16T14:51:08.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:08.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:08.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:08.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 10 monclient(hunting): my global_id is 184189 2024-09-16T14:51:08.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa36c006590 con 0x7fa36c0090a0 2024-09-16T14:51:08.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2039622945 0 0) 0x7fa36c006590 con 0x7fa36c0090a0 2024-09-16T14:51:08.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa36c009d80 con 0x7fa36c0090a0 2024-09-16T14:51:08.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa380003840 con 0x7fa36c0090a0 2024-09-16T14:51:08.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:08.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:08.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 10 monclient: dump: 2024-09-16T14:51:08.789 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:08.789 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:08.789 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.790 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:08.790 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:08.790 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:08.790 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:08.790 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:08.790 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:08.790 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:08.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3013591784 0 0) 0x7fa36c009d80 con 0x7fa36c0090a0 2024-09-16T14:51:08.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa36c00c290 con 0x7fa36c0090a0 2024-09-16T14:51:08.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3868 (unknown 0 0 105461660) 0x7fa380003b70 con 0x7fa36c0090a0 2024-09-16T14:51:08.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:51:08.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:08.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 >> v1:172.21.15.189:6789/0 conn(0x7fa36c0090a0 legacy=0x7fa36c009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa37a7fc640 1 -- 172.21.15.26:0/1143349903 >> v1:172.21.15.189:6789/0 conn(0x7fa36c0090a0 legacy=0x7fa36c009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:08.791 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:51:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:51:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:51:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:51:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 82, 2024-09-16T14:51:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:51:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:51:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:51:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-09-16T14:51:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:51:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:51:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:08.792 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:08.793 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:08.793 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:51:08.793 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:51:08.793 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:08.793 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:08.793 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:08.793 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:08.793 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:08.793 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:08.793 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:08.793 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:08.793 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:08.793 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:08.794 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:08.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.794 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:51:08.794 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:51:08.794 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:51:08.794 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:51:08.794 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:51:08.794 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:51:08.794 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:08.794 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:08.794 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:51:08.794 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:51:08.794 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:51:08.794 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:51:08.795 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:51:08.795 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:51:08.795 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:51:08.795 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:08.795 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:51:08.795 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:08.795 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:08.795 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:08.795 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:08.795 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:08.795 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:08.795 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:08.795 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:08.795 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:08.796 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:08.796 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:08.796 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:51:08.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.796 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:51:08.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.796 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:51:08.796 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:51:08.796 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:08.796 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:08.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.796 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:08.796 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:51:08.796 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:51:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:51:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:51:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:51:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:08.797 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:08.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.798 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:08.798 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:51:08.798 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:08.798 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:08.798 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:08.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.798 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:51:08.798 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:51:08.798 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:08.798 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:08.798 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:08.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:51:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:51:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:51:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:51:08.799 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:51:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:51:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:51:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:51:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:08.800 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:08.801 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:51:08.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:08.801 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:08.801 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:08.801 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:51:08.801 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:08.801 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:08.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:08.801 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:51:08.801 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:51:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa38bbf6640 1 -- 172.21.15.26:0/1143349903 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa3680360e0 legacy=0x7fa3680385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa38bbf6640 10 monclient: shutdown 2024-09-16T14:51:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa38bbf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa38bbf6640 1 -- 172.21.15.26:0/1143349903 >> v1:172.21.15.26:6790/0 conn(0x7fa384131c10 legacy=0x7fa3841d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa38a193640 1 -- 172.21.15.26:0/1143349903 reap_dead start 2024-09-16T14:51:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa38a193640 1 -- 172.21.15.26:0/1143349903 reap_dead start 2024-09-16T14:51:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa38bbf6640 1 -- 172.21.15.26:0/1143349903 shutdown_connections 2024-09-16T14:51:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa38bbf6640 1 -- 172.21.15.26:0/1143349903 >> 172.21.15.26:0/1143349903 conn(0x7fa3840a03d0 msgr2=0x7fa38409d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa38bbf6640 1 -- 172.21.15.26:0/1143349903 shutdown_connections 2024-09-16T14:51:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:08.781+0000 7fa38bbf6640 1 -- 172.21.15.26:0/1143349903 wait complete. 2024-09-16T14:51:08.804 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:51:08.804 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:51:08.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.871+0000 7ffb8c673640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T14:51:08.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.871+0000 7ffb8c673640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T14:51:08.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.871+0000 7ffb8c673640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:08.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.871+0000 7ffb8c673640 1 -- 172.21.15.189:0/3348107718 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h32"} v 0) v1 -- 0x7ffb6c0055a0 con 0x7ffb84099a70 2024-09-16T14:51:08.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.875+0000 7ffb7affd640 1 -- 172.21.15.189:0/3348107718 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h32"}]=0 device 'h32' does not appear in the crush map v247) v1 ==== 120+0+0 (unknown 1286066567 0 0) 0x7ffb74012de0 con 0x7ffb84099a70 2024-09-16T14:51:08.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.875+0000 7ffb7affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T14:51:08.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.875+0000 7ffb7affd640 10 monclient: _finish_command 3 = system:0 device 'h32' does not appear in the crush map 2024-09-16T14:51:08.876 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h32' does not appear in the crush map 2024-09-16T14:51:08.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.875+0000 7ffb8c673640 1 -- 172.21.15.189:0/3348107718 >> v1:172.21.15.189:6812/3196161582 conn(0x7ffb680361f0 legacy=0x7ffb680386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.875+0000 7ffb8c673640 10 monclient: shutdown 2024-09-16T14:51:08.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.875+0000 7ffb8c673640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:08.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.875+0000 7ffb8c673640 1 -- 172.21.15.189:0/3348107718 >> v1:172.21.15.189:6789/0 conn(0x7ffb84099a70 legacy=0x7ffb84132450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:08.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.875+0000 7ffb8c673640 1 -- 172.21.15.189:0/3348107718 shutdown_connections 2024-09-16T14:51:08.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.875+0000 7ffb8c673640 1 -- 172.21.15.189:0/3348107718 >> 172.21.15.189:0/3348107718 conn(0x7ffb840a3320 msgr2=0x7ffb840a1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:08.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.875+0000 7ffb8c673640 1 -- 172.21.15.189:0/3348107718 shutdown_connections 2024-09-16T14:51:08.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:08.875+0000 7ffb8c673640 1 -- 172.21.15.189:0/3348107718 wait complete. 2024-09-16T14:51:08.895 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm r1 2024-09-16T14:51:09.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.007+0000 7f98d5cd5640 10 monclient: get_monmap_and_config 2024-09-16T14:51:09.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.007+0000 7f98d5cd5640 10 monclient: build_initial_monmap 2024-09-16T14:51:09.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.007+0000 7f98d5cd5640 10 monclient: monmap: 2024-09-16T14:51:09.010 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:09.010 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:09.010 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:51:09.012227+0000 2024-09-16T14:51:09.010 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:51:09.012227+0000 2024-09-16T14:51:09.011 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:09.011 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:09.011 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:09.011 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:09.011 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:09.011 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:09.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.007+0000 7f98d5cd5640 1 Processor -- start 2024-09-16T14:51:09.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.007+0000 7f98d5cd5640 1 -- start start 2024-09-16T14:51:09.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.007+0000 7f98d5cd5640 10 monclient: init 2024-09-16T14:51:09.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.007+0000 7f98d5cd5640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:09.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.007+0000 7f98d5cd5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:51:09.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.007+0000 7f98d5cd5640 10 monclient(hunting): picked mon.noname-a con 0x7f98d009df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:09.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.007+0000 7f98d5cd5640 10 monclient(hunting): picked mon.noname-c con 0x7f98d009e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:09.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 10 monclient(hunting): picked mon.noname-b con 0x7f98d009e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:09.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d009ee20 con 0x7f98d009df10 2024-09-16T14:51:09.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d009f100 con 0x7f98d009e2f0 2024-09-16T14:51:09.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d009f3e0 con 0x7f98d009e740 2024-09-16T14:51:09.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:09.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 10 monclient(hunting): authenticate will time out at 1451.128120s 2024-09-16T14:51:09.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98cf7fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f98d009df10 legacy=0x7f98d009d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:09.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ceffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f98d009e2f0 0x7f98d01378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48176/0 (socket says 172.21.15.189:48176) 2024-09-16T14:51:09.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ceffd640 1 -- 172.21.15.189:0/1710595832 learned_addr learned my addr 172.21.15.189:0/1710595832 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:09.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 1 -- 172.21.15.189:0/1710595832 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476335110 0 0) 0x7f98d009f100 con 0x7f98d009e2f0 2024-09-16T14:51:09.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:09.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:09.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:09.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 10 monclient(hunting): my global_id is 184192 2024-09-16T14:51:09.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 1 -- 172.21.15.189:0/1710595832 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98bc003530 con 0x7f98d009e2f0 2024-09-16T14:51:09.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 1 -- 172.21.15.189:0/1710595832 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2551874624 0 0) 0x7f98bc003530 con 0x7f98d009e2f0 2024-09-16T14:51:09.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 1 -- 172.21.15.189:0/1710595832 >> v1:172.21.15.26:6790/0 conn(0x7f98d009e740 legacy=0x7f98d013b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:09.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 1 -- 172.21.15.189:0/1710595832 >> v1:172.21.15.26:6789/0 conn(0x7f98d009df10 legacy=0x7f98d009d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:09.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:51:09.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 1 monclient: found mon.noname-c 2024-09-16T14:51:09.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:09.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:51:09.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 1 -- 172.21.15.189:0/1710595832 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d009c690 con 0x7f98d009e2f0 2024-09-16T14:51:09.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 10 monclient: _finish_auth 0 2024-09-16T14:51:09.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:51:09.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:09.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 1 -- 172.21.15.189:0/1710595832 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f98b80019a0 con 0x7f98d009e2f0 2024-09-16T14:51:09.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:09.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:09.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 10 monclient: dump: 2024-09-16T14:51:09.022 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:09.022 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:09.022 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:09.023 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:09.023 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:09.023 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:09.023 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:09.023 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:09.023 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:09.023 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:09.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 5 monclient: authenticate success, global_id 184192 2024-09-16T14:51:09.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:09.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 1 -- 172.21.15.189:0/1710595832 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98b8001cc0 con 0x7f98d009e2f0 2024-09-16T14:51:09.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:09.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:09.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 1 -- 172.21.15.189:0/1710595832 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f98b8002050 con 0x7f98d009e2f0 2024-09-16T14:51:09.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:09.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 4 set_mon_vals no callback set 2024-09-16T14:51:09.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:09.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ce7fc640 10 monclient: dump: 2024-09-16T14:51:09.024 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:09.025 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:09.025 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:09.025 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:09.025 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:09.025 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:09.025 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:09.025 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:09.025 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:09.025 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:09.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 10 monclient: shutdown 2024-09-16T14:51:09.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:09.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 1 -- 172.21.15.189:0/1710595832 >> v1:172.21.15.189:6789/0 conn(0x7f98d009e2f0 legacy=0x7f98d01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:09.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 1 -- 172.21.15.189:0/1710595832 shutdown_connections 2024-09-16T14:51:09.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 1 -- 172.21.15.189:0/1710595832 >> 172.21.15.189:0/1710595832 conn(0x7f98d00a3320 msgr2=0x7f98d009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:09.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 1 -- 172.21.15.189:0/1710595832 shutdown_connections 2024-09-16T14:51:09.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 1 -- 172.21.15.189:0/1710595832 wait complete. 2024-09-16T14:51:09.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 10 monclient: build_initial_monmap 2024-09-16T14:51:09.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 10 monclient: monmap: 2024-09-16T14:51:09.027 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:09.027 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:09.027 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:51:09.027 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:51:09.027 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:09.027 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:51:09.027 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:09.027 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:09.027 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:09.028 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:09.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 1 Processor -- start 2024-09-16T14:51:09.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 1 -- start start 2024-09-16T14:51:09.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 10 monclient: init 2024-09-16T14:51:09.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:09.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:51:09.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 10 monclient(hunting): picked mon.noname-c con 0x7f98d009df10 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:09.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 10 monclient(hunting): picked mon.noname-a con 0x7f98d009e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:09.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 10 monclient(hunting): picked mon.noname-b con 0x7f98d009e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:09.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d006ce40 con 0x7f98d009e2f0 2024-09-16T14:51:09.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d0120fb0 con 0x7f98d009e740 2024-09-16T14:51:09.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d006b380 con 0x7f98d009df10 2024-09-16T14:51:09.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:09.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98d5cd5640 10 monclient(hunting): authenticate will time out at 1451.131202s 2024-09-16T14:51:09.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98cffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f98d009e740 0x7f98d01d17a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48182/0 (socket says 172.21.15.189:48182) 2024-09-16T14:51:09.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98ceffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f98d009e2f0 legacy=0x7f98d009ab90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:09.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98cffff640 1 -- 172.21.15.189:0/965854614 learned_addr learned my addr 172.21.15.189:0/965854614 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:09.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978887048 0 0) 0x7f98d0120fb0 con 0x7f98d009e740 2024-09-16T14:51:09.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:09.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:09.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:09.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 10 monclient(hunting): my global_id is 184195 2024-09-16T14:51:09.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b0003530 con 0x7f98d009e740 2024-09-16T14:51:09.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2518082559 0 0) 0x7f98b0003530 con 0x7f98d009e740 2024-09-16T14:51:09.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98d0120fb0 con 0x7f98d009e740 2024-09-16T14:51:09.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f98c8001880 con 0x7f98d009e740 2024-09-16T14:51:09.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:09.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:09.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 10 monclient(hunting): dump: 2024-09-16T14:51:09.031 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:09.031 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:09.031 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:09.031 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:09.031 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:09.031 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:09.031 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:09.031 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:09.031 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:09.032 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:09.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:09.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245692571 0 0) 0x7f98d006b380 con 0x7f98d009df10 2024-09-16T14:51:09.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:09.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:09.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:09.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 10 monclient(hunting): my global_id is 154904 2024-09-16T14:51:09.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b0003530 con 0x7f98d009df10 2024-09-16T14:51:09.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1650234229 0 0) 0x7f98d0120fb0 con 0x7f98d009e740 2024-09-16T14:51:09.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 >> v1:172.21.15.26:6790/0 conn(0x7f98d009df10 legacy=0x7f98d009a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:09.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.011+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 >> v1:172.21.15.26:6789/0 conn(0x7f98d009e2f0 legacy=0x7f98d009ab90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:09.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98c7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:51:09.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98c7fff640 1 monclient: found mon.b 2024-09-16T14:51:09.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:09.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98c7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:09.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d0120d70 con 0x7f98d009e740 2024-09-16T14:51:09.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98c7fff640 10 monclient: _finish_auth 0 2024-09-16T14:51:09.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98c7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:51:09.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:09.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98d5cd5640 5 monclient: authenticate success, global_id 184195 2024-09-16T14:51:09.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98d5cd5640 10 monclient: _renew_subs 2024-09-16T14:51:09.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98d5cd5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:09.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98d5cd5640 1 -- 172.21.15.189:0/965854614 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d0122500 con 0x7f98d009e740 2024-09-16T14:51:09.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98d5cd5640 10 monclient: _renew_subs 2024-09-16T14:51:09.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98d5cd5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:09.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98d5cd5640 1 -- 172.21.15.189:0/965854614 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d009be80 con 0x7f98d009e740 2024-09-16T14:51:09.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98c8001c50 con 0x7f98d009e740 2024-09-16T14:51:09.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98c7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:09.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f98c8003af0 con 0x7f98d009e740 2024-09-16T14:51:09.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:09.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:09.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98cdffb640 4 set_mon_vals no callback set 2024-09-16T14:51:09.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98c7fff640 10 monclient: dump: 2024-09-16T14:51:09.035 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:09.035 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:09.035 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:09.035 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:09.035 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:09.035 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:09.035 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:09.035 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:09.035 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:09.036 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:09.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f98c8003c50 con 0x7f98d009e740 2024-09-16T14:51:09.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98d5cd5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:09.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98d5cd5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:09.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98d5cd5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:09.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98d5cd5640 1 -- 172.21.15.189:0/965854614 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b4005050 con 0x7f98d009e740 2024-09-16T14:51:09.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(247..247 src has 169..247) v4 ==== 7283+0+0 (unknown 4191992233 0 0) 0x7f98c8044050 con 0x7f98d009e740 2024-09-16T14:51:09.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.015+0000 7f98cf7fe640 10 monclient: get_auth_request con 0x7f98b0036130 auth_method 0 2024-09-16T14:51:09.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.019+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f98c8014770 con 0x7f98d009e740 2024-09-16T14:51:09.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.019+0000 7f98c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:09.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.019+0000 7f98c7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:09.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.171+0000 7f98d5cd5640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T14:51:09.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.171+0000 7f98d5cd5640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T14:51:09.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.171+0000 7f98d5cd5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:09.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.171+0000 7f98d5cd5640 1 -- 172.21.15.189:0/965854614 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "r1"} v 0) v1 -- 0x7f98b4005300 con 0x7f98d009e740 2024-09-16T14:51:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.771+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "r1"}]=0 removed item id -16 name 'r1' from crush map v248) v1 ==== 118+0+0 (unknown 3450644807 0 0) 0x7f98c8012be0 con 0x7f98d009e740 2024-09-16T14:51:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.771+0000 7f98c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T14:51:09.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.771+0000 7f98c7fff640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'r1' from crush map 2024-09-16T14:51:09.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.867+0000 7f98d5cd5640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T14:51:09.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.867+0000 7f98d5cd5640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T14:51:09.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.867+0000 7f98d5cd5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:09.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.867+0000 7f98d5cd5640 1 -- 172.21.15.189:0/965854614 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "r1"} v 0) v1 -- 0x7f98b40055a0 con 0x7f98d009e740 2024-09-16T14:51:09.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.871+0000 7f98c7fff640 1 -- 172.21.15.189:0/965854614 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "r1"}]=0 device 'r1' does not appear in the crush map v248) v1 ==== 118+0+0 (unknown 2396752965 0 0) 0x7f98c8014540 con 0x7f98d009e740 2024-09-16T14:51:09.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.871+0000 7f98c7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T14:51:09.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.871+0000 7f98c7fff640 10 monclient: _finish_command 3 = system:0 device 'r1' does not appear in the crush map 2024-09-16T14:51:09.872 INFO:tasks.workunit.client.0.smithi189.stderr:device 'r1' does not appear in the crush map 2024-09-16T14:51:09.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.871+0000 7f98d5cd5640 1 -- 172.21.15.189:0/965854614 >> v1:172.21.15.189:6812/3196161582 conn(0x7f98b0036130 legacy=0x7f98b00385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:09.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.871+0000 7f98d5cd5640 10 monclient: shutdown 2024-09-16T14:51:09.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.871+0000 7f98d5cd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:09.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.871+0000 7f98d5cd5640 1 -- 172.21.15.189:0/965854614 >> v1:172.21.15.189:6789/0 conn(0x7f98d009e740 legacy=0x7f98d01d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:09.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.871+0000 7f98d5cd5640 1 -- 172.21.15.189:0/965854614 shutdown_connections 2024-09-16T14:51:09.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.871+0000 7f98d5cd5640 1 -- 172.21.15.189:0/965854614 >> 172.21.15.189:0/965854614 conn(0x7f98d00a3320 msgr2=0x7f98d00a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:09.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.871+0000 7f98d5cd5640 1 -- 172.21.15.189:0/965854614 shutdown_connections 2024-09-16T14:51:09.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:09.871+0000 7f98d5cd5640 1 -- 172.21.15.189:0/965854614 wait complete. 2024-09-16T14:51:09.889 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set rm-compat 2024-09-16T14:51:10.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.003+0000 7ff89d5cb640 10 monclient: get_monmap_and_config 2024-09-16T14:51:10.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.003+0000 7ff89d5cb640 10 monclient: build_initial_monmap 2024-09-16T14:51:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.003+0000 7ff89d5cb640 10 monclient: monmap: 2024-09-16T14:51:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:51:10.008377+0000 2024-09-16T14:51:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:51:10.008377+0000 2024-09-16T14:51:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:10.007 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:10.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.003+0000 7ff89d5cb640 1 Processor -- start 2024-09-16T14:51:10.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.003+0000 7ff89d5cb640 1 -- start start 2024-09-16T14:51:10.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.003+0000 7ff89d5cb640 10 monclient: init 2024-09-16T14:51:10.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.003+0000 7ff89d5cb640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:10.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.003+0000 7ff89d5cb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:51:10.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.003+0000 7ff89d5cb640 10 monclient(hunting): picked mon.noname-b con 0x7ff8980a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:10.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.003+0000 7ff89d5cb640 10 monclient(hunting): picked mon.noname-c con 0x7ff8980a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:10.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.003+0000 7ff89d5cb640 10 monclient(hunting): picked mon.noname-a con 0x7ff89813b430 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:10.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.003+0000 7ff89d5cb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8980a30b0 con 0x7ff89813b430 2024-09-16T14:51:10.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.003+0000 7ff89d5cb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8980a2e30 con 0x7ff8980a25f0 2024-09-16T14:51:10.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.003+0000 7ff89d5cb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89809eb50 con 0x7ff8980a1b00 2024-09-16T14:51:10.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.003+0000 7ff89d5cb640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:10.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.003+0000 7ff89d5cb640 10 monclient(hunting): authenticate will time out at 1452.123988s 2024-09-16T14:51:10.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff8967fc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff8980a25f0 0x7ff898139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48198/0 (socket says 172.21.15.189:48198) 2024-09-16T14:51:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff8967fc640 1 -- 172.21.15.189:0/2372794127 learned_addr learned my addr 172.21.15.189:0/2372794127 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff8977fe640 1 -- 172.21.15.189:0/2372794127 >> v1:172.21.15.26:6789/0 conn(0x7ff89813b430 legacy=0x7ff89813d820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 1 -- 172.21.15.189:0/2372794127 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272084532 0 0) 0x7ff8980a2e30 con 0x7ff8980a25f0 2024-09-16T14:51:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 10 monclient(hunting): my global_id is 184198 2024-09-16T14:51:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 1 -- 172.21.15.189:0/2372794127 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff880003530 con 0x7ff8980a25f0 2024-09-16T14:51:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 1 -- 172.21.15.189:0/2372794127 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3341653112 0 0) 0x7ff880003530 con 0x7ff8980a25f0 2024-09-16T14:51:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 1 -- 172.21.15.189:0/2372794127 >> v1:172.21.15.26:6790/0 conn(0x7ff8980a1b00 legacy=0x7ff8980a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 1 -- 172.21.15.189:0/2372794127 >> v1:172.21.15.26:6789/0 conn(0x7ff89813b430 legacy=0x7ff89813d820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:51:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 1 monclient: found mon.noname-c 2024-09-16T14:51:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:51:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 1 -- 172.21.15.189:0/2372794127 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89809ee30 con 0x7ff8980a25f0 2024-09-16T14:51:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 10 monclient: _finish_auth 0 2024-09-16T14:51:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:51:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 1 -- 172.21.15.189:0/2372794127 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff88c001620 con 0x7ff8980a25f0 2024-09-16T14:51:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 10 monclient: dump: 2024-09-16T14:51:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:10.020 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 5 monclient: authenticate success, global_id 184198 2024-09-16T14:51:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 1 -- 172.21.15.189:0/2372794127 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff88c001940 con 0x7ff8980a25f0 2024-09-16T14:51:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 1 -- 172.21.15.189:0/2372794127 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff88c001cd0 con 0x7ff8980a25f0 2024-09-16T14:51:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 4 set_mon_vals no callback set 2024-09-16T14:51:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff895ffb640 10 monclient: dump: 2024-09-16T14:51:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:10.022 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 10 monclient: shutdown 2024-09-16T14:51:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 1 -- 172.21.15.189:0/2372794127 >> v1:172.21.15.189:6789/0 conn(0x7ff8980a25f0 legacy=0x7ff898139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 1 -- 172.21.15.189:0/2372794127 shutdown_connections 2024-09-16T14:51:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 1 -- 172.21.15.189:0/2372794127 >> 172.21.15.189:0/2372794127 conn(0x7ff8980a7dd0 msgr2=0x7ff8980a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:10.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 1 -- 172.21.15.189:0/2372794127 shutdown_connections 2024-09-16T14:51:10.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 1 -- 172.21.15.189:0/2372794127 wait complete. 2024-09-16T14:51:10.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 10 monclient: build_initial_monmap 2024-09-16T14:51:10.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 10 monclient: monmap: 2024-09-16T14:51:10.023 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:10.023 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:10.023 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:51:10.023 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:51:10.024 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:10.024 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:51:10.024 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:10.024 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:10.024 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:10.024 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:10.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 1 Processor -- start 2024-09-16T14:51:10.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 1 -- start start 2024-09-16T14:51:10.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 10 monclient: init 2024-09-16T14:51:10.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:10.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:51:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 10 monclient(hunting): picked mon.noname-c con 0x7ff8980a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 10 monclient(hunting): picked mon.noname-a con 0x7ff8980a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 10 monclient(hunting): picked mon.noname-b con 0x7ff89813b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff898134d50 con 0x7ff8980a25f0 2024-09-16T14:51:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8980a5a60 con 0x7ff89813b430 2024-09-16T14:51:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89809e2f0 con 0x7ff8980a1b00 2024-09-16T14:51:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 10 monclient(hunting): authenticate will time out at 1452.126866s 2024-09-16T14:51:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff8977fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff89813b430 0x7ff8981d1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48206/0 (socket says 172.21.15.189:48206) 2024-09-16T14:51:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff8977fe640 1 -- 172.21.15.189:0/3269885407 learned_addr learned my addr 172.21.15.189:0/3269885407 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff8967fc640 1 -- 172.21.15.189:0/3269885407 >> v1:172.21.15.26:6789/0 conn(0x7ff8980a25f0 legacy=0x7ff8981ce7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 1 -- 172.21.15.189:0/3269885407 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608245804 0 0) 0x7ff890003dd0 con 0x7ff89813b430 2024-09-16T14:51:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 10 monclient(hunting): my global_id is 184201 2024-09-16T14:51:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 1 -- 172.21.15.189:0/3269885407 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff884003530 con 0x7ff89813b430 2024-09-16T14:51:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 1 -- 172.21.15.189:0/3269885407 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2242589975 0 0) 0x7ff884003530 con 0x7ff89813b430 2024-09-16T14:51:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 1 -- 172.21.15.189:0/3269885407 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff890003dd0 con 0x7ff89813b430 2024-09-16T14:51:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 1 -- 172.21.15.189:0/3269885407 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff890004440 con 0x7ff89813b430 2024-09-16T14:51:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 10 monclient(hunting): dump: 2024-09-16T14:51:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:10.028 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 1 -- 172.21.15.189:0/3269885407 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 30365252 0 0) 0x7ff890003dd0 con 0x7ff89813b430 2024-09-16T14:51:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 1 -- 172.21.15.189:0/3269885407 >> v1:172.21.15.26:6790/0 conn(0x7ff8980a1b00 legacy=0x7ff8981c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 1 -- 172.21.15.189:0/3269885407 >> v1:172.21.15.26:6789/0 conn(0x7ff8980a25f0 legacy=0x7ff8981ce7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:51:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 1 monclient: found mon.b 2024-09-16T14:51:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 1 -- 172.21.15.189:0/3269885407 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89813adb0 con 0x7ff89813b430 2024-09-16T14:51:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:51:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:51:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 5 monclient: authenticate success, global_id 184201 2024-09-16T14:51:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 10 monclient: _renew_subs 2024-09-16T14:51:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 1 -- 172.21.15.189:0/3269885407 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89811ff10 con 0x7ff89813b430 2024-09-16T14:51:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 10 monclient: _renew_subs 2024-09-16T14:51:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff89d5cb640 1 -- 172.21.15.189:0/3269885407 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff89811db40 con 0x7ff89813b430 2024-09-16T14:51:10.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 1 -- 172.21.15.189:0/3269885407 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff890004760 con 0x7ff89813b430 2024-09-16T14:51:10.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:10.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 1 -- 172.21.15.189:0/3269885407 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff890004af0 con 0x7ff89813b430 2024-09-16T14:51:10.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:10.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:10.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff8957fa640 4 set_mon_vals no callback set 2024-09-16T14:51:10.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.007+0000 7ff87f7fe640 10 monclient: dump: 2024-09-16T14:51:10.031 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:10.031 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:10.031 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:10.031 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:10.031 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:10.031 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:10.032 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:10.032 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:10.032 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:10.032 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:10.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.011+0000 7ff89d5cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:10.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.011+0000 7ff89d5cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:10.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.011+0000 7ff89d5cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:10.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.011+0000 7ff89d5cb640 1 -- 172.21.15.189:0/3269885407 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff878005050 con 0x7ff89813b430 2024-09-16T14:51:10.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.011+0000 7ff87f7fe640 1 -- 172.21.15.189:0/3269885407 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff890004cf0 con 0x7ff89813b430 2024-09-16T14:51:10.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.011+0000 7ff87f7fe640 1 -- 172.21.15.189:0/3269885407 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(248..248 src has 169..248) v4 ==== 7073+0+0 (unknown 2304808518 0 0) 0x7ff890044300 con 0x7ff89813b430 2024-09-16T14:51:10.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.011+0000 7ff896ffd640 10 monclient: get_auth_request con 0x7ff884036000 auth_method 0 2024-09-16T14:51:10.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.015+0000 7ff87f7fe640 1 -- 172.21.15.189:0/3269885407 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff890014af0 con 0x7ff89813b430 2024-09-16T14:51:10.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.015+0000 7ff87f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:10.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.015+0000 7ff87f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:10.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.179+0000 7ff89d5cb640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:51:10.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.179+0000 7ff89d5cb640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:51:10.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.179+0000 7ff89d5cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:10.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.179+0000 7ff89d5cb640 1 -- 172.21.15.189:0/3269885407 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7ff878005300 con 0x7ff89813b430 2024-09-16T14:51:10.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.783+0000 7ff87f7fe640 1 -- 172.21.15.189:0/3269885407 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v249) v1 ==== 78+0+0 (unknown 3129046136 0 0) 0x7ff890012f60 con 0x7ff89813b430 2024-09-16T14:51:10.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.783+0000 7ff87f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:51:10.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.783+0000 7ff87f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:10.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.883+0000 7ff89d5cb640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:51:10.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.883+0000 7ff89d5cb640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:51:10.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.883+0000 7ff89d5cb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:10.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:10.883+0000 7ff89d5cb640 1 -- 172.21.15.189:0/3269885407 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7ff878005850 con 0x7ff89813b430 2024-09-16T14:51:11.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:11.783+0000 7ff87f7fe640 1 -- 172.21.15.189:0/3269885407 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v250) v1 ==== 78+0+0 (unknown 2026023447 0 0) 0x7ff890012f60 con 0x7ff89813b430 2024-09-16T14:51:11.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:11.783+0000 7ff87f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T14:51:11.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:11.783+0000 7ff87f7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:51:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:11.787+0000 7ff89d5cb640 1 -- 172.21.15.189:0/3269885407 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff884036000 legacy=0x7ff8840384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:11.787+0000 7ff89d5cb640 10 monclient: shutdown 2024-09-16T14:51:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:11.787+0000 7ff89d5cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:11.787+0000 7ff89d5cb640 1 -- 172.21.15.189:0/3269885407 >> v1:172.21.15.189:6789/0 conn(0x7ff89813b430 legacy=0x7ff8981d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:11.787+0000 7ff89d5cb640 1 -- 172.21.15.189:0/3269885407 shutdown_connections 2024-09-16T14:51:11.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:11.787+0000 7ff89d5cb640 1 -- 172.21.15.189:0/3269885407 >> 172.21.15.189:0/3269885407 conn(0x7ff8980a7dd0 msgr2=0x7ff89806b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:11.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:11.787+0000 7ff89d5cb640 1 -- 172.21.15.189:0/3269885407 shutdown_connections 2024-09-16T14:51:11.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:11.787+0000 7ff89d5cb640 1 -- 172.21.15.189:0/3269885407 wait complete. 2024-09-16T14:51:11.810 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo OK 2024-09-16T14:51:11.811 INFO:tasks.workunit.client.0.smithi189.stdout:OK 2024-09-16T14:51:11.811 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T14:51:11.811 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T14:51:11.870 INFO:tasks.workunit:Running workunits matching mon/osd.sh on client.0... 2024-09-16T14:51:11.874 INFO:tasks.workunit:Running workunit mon/osd.sh... 2024-09-16T14:51:11.875 DEBUG:teuthology.orchestra.run.smithi189:workunit test mon/osd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/osd.sh 2024-09-16T14:51:11.920 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -e 2024-09-16T14:51:11.921 INFO:tasks.workunit.client.0.smithi189.stderr:+ uuidgen 2024-09-16T14:51:11.921 INFO:tasks.workunit.client.0.smithi189.stderr:+ ua=0689a78d-9a3c-48ff-a75a-01eedf6472fa 2024-09-16T14:51:11.921 INFO:tasks.workunit.client.0.smithi189.stderr:+ uuidgen 2024-09-16T14:51:11.922 INFO:tasks.workunit.client.0.smithi189.stderr:+ ub=ee517a82-a411-40e2-ac30-45aa35010440 2024-09-16T14:51:11.922 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd create 0689a78d-9a3c-48ff-a75a-01eedf6472fa 2024-09-16T14:51:12.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.043+0000 7fc895c9d640 10 monclient: get_monmap_and_config 2024-09-16T14:51:12.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.043+0000 7fc895c9d640 10 monclient: build_initial_monmap 2024-09-16T14:51:12.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.043+0000 7fc895c9d640 10 monclient: monmap: 2024-09-16T14:51:12.047 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:12.047 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:12.048 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:51:12.049434+0000 2024-09-16T14:51:12.048 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:51:12.049434+0000 2024-09-16T14:51:12.048 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:12.048 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:12.048 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:12.048 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:12.048 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:12.048 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:12.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 1 Processor -- start 2024-09-16T14:51:12.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 1 -- start start 2024-09-16T14:51:12.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient: init 2024-09-16T14:51:12.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:12.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:51:12.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc8900a1b00 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:12.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc8900a25f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:12.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient(hunting): picked mon.noname-a con 0x7fc89013b430 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:12.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8900a30b0 con 0x7fc89013b430 2024-09-16T14:51:12.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8900a2e30 con 0x7fc8900a1b00 2024-09-16T14:51:12.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89009eb50 con 0x7fc8900a25f0 2024-09-16T14:51:12.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:12.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient(hunting): authenticate will time out at 1454.164998s 2024-09-16T14:51:12.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88f7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc8900a1b00 0x7fc8900a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48208/0 (socket says 172.21.15.189:48208) 2024-09-16T14:51:12.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88f7fe640 1 -- 172.21.15.189:0/3837739185 learned_addr learned my addr 172.21.15.189:0/3837739185 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:12.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88ffff640 1 -- 172.21.15.189:0/3837739185 >> v1:172.21.15.26:6789/0 conn(0x7fc89013b430 legacy=0x7fc89013d820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:12.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 1 -- 172.21.15.189:0/3837739185 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782828205 0 0) 0x7fc8900a2e30 con 0x7fc8900a1b00 2024-09-16T14:51:12.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 10 monclient(hunting): _negotiate 2024-09-16T14:51:12.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:12.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:12.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 10 monclient(hunting): my global_id is 184204 2024-09-16T14:51:12.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 1 -- 172.21.15.189:0/3837739185 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc87c003530 con 0x7fc8900a1b00 2024-09-16T14:51:12.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 1 -- 172.21.15.189:0/3837739185 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2229719784 0 0) 0x7fc87c003530 con 0x7fc8900a1b00 2024-09-16T14:51:12.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 1 -- 172.21.15.189:0/3837739185 >> v1:172.21.15.26:6790/0 conn(0x7fc8900a25f0 legacy=0x7fc890139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:12.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 1 -- 172.21.15.189:0/3837739185 >> v1:172.21.15.26:6789/0 conn(0x7fc89013b430 legacy=0x7fc89013d820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:12.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 10 monclient: _finish_hunting 0 2024-09-16T14:51:12.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 1 monclient: found mon.noname-c 2024-09-16T14:51:12.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:12.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:51:12.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 1 -- 172.21.15.189:0/3837739185 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89009ee30 con 0x7fc8900a1b00 2024-09-16T14:51:12.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 10 monclient: _finish_auth 0 2024-09-16T14:51:12.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 10 monclient: _check_auth_tickets 2024-09-16T14:51:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 1 -- 172.21.15.189:0/3837739185 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc880001620 con 0x7fc8900a1b00 2024-09-16T14:51:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 5 monclient: authenticate success, global_id 184204 2024-09-16T14:51:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 10 monclient: dump: 2024-09-16T14:51:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:12.058 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:12.058 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:12.058 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:12.058 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:12.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:12.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 1 -- 172.21.15.189:0/3837739185 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc880001940 con 0x7fc8900a1b00 2024-09-16T14:51:12.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:12.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:12.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 1 -- 172.21.15.189:0/3837739185 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc880001d10 con 0x7fc8900a1b00 2024-09-16T14:51:12.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 4 set_mon_vals no callback set 2024-09-16T14:51:12.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:12.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:12.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc88effd640 10 monclient: dump: 2024-09-16T14:51:12.059 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:12.059 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:12.059 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:12.059 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:12.059 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:12.059 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:12.059 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:12.059 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:12.059 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:12.059 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:12.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient: shutdown 2024-09-16T14:51:12.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:12.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 1 -- 172.21.15.189:0/3837739185 >> v1:172.21.15.189:6789/0 conn(0x7fc8900a1b00 legacy=0x7fc8900a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:12.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 1 -- 172.21.15.189:0/3837739185 shutdown_connections 2024-09-16T14:51:12.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 1 -- 172.21.15.189:0/3837739185 >> 172.21.15.189:0/3837739185 conn(0x7fc8900a7dd0 msgr2=0x7fc8900a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:12.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 1 -- 172.21.15.189:0/3837739185 shutdown_connections 2024-09-16T14:51:12.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 1 -- 172.21.15.189:0/3837739185 wait complete. 2024-09-16T14:51:12.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient: build_initial_monmap 2024-09-16T14:51:12.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient: monmap: 2024-09-16T14:51:12.060 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:12.060 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:12.060 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:51:12.060 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:51:12.060 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:12.060 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:51:12.060 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:12.061 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:12.061 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:12.061 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:12.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 1 Processor -- start 2024-09-16T14:51:12.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 1 -- start start 2024-09-16T14:51:12.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient: init 2024-09-16T14:51:12.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:12.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:51:12.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient(hunting): picked mon.noname-a con 0x7fc8900a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:12.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc8900a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:12.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc89013b430 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:12.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc890134cd0 con 0x7fc8900a1b00 2024-09-16T14:51:12.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89011f8f0 con 0x7fc8900a25f0 2024-09-16T14:51:12.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89013afb0 con 0x7fc89013b430 2024-09-16T14:51:12.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:12.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc895c9d640 10 monclient(hunting): authenticate will time out at 1454.167889s 2024-09-16T14:51:12.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc886ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc8900a25f0 0x7fc8901ce7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48222/0 (socket says 172.21.15.189:48222) 2024-09-16T14:51:12.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.047+0000 7fc886ffd640 1 -- 172.21.15.189:0/676812475 learned_addr learned my addr 172.21.15.189:0/676812475 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:12.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88f7fe640 1 -- 172.21.15.189:0/676812475 >> v1:172.21.15.26:6789/0 conn(0x7fc8900a1b00 legacy=0x7fc8901c9810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:12.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 1 -- 172.21.15.189:0/676812475 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809602195 0 0) 0x7fc89011f8f0 con 0x7fc8900a25f0 2024-09-16T14:51:12.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:51:12.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:12.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:12.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 10 monclient(hunting): my global_id is 184207 2024-09-16T14:51:12.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 1 -- 172.21.15.189:0/676812475 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc870003560 con 0x7fc8900a25f0 2024-09-16T14:51:12.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 1 -- 172.21.15.189:0/676812475 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 594735424 0 0) 0x7fc870003560 con 0x7fc8900a25f0 2024-09-16T14:51:12.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 1 -- 172.21.15.189:0/676812475 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc89011f8f0 con 0x7fc8900a25f0 2024-09-16T14:51:12.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 1 -- 172.21.15.189:0/676812475 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc878001940 con 0x7fc8900a25f0 2024-09-16T14:51:12.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:12.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:12.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 10 monclient(hunting): dump: 2024-09-16T14:51:12.063 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:12.063 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:12.063 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:12.063 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:12.063 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:12.063 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:12.063 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:12.063 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:12.064 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:12.064 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:12.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:12.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 1 -- 172.21.15.189:0/676812475 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1546508882 0 0) 0x7fc89011f8f0 con 0x7fc8900a25f0 2024-09-16T14:51:12.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 1 -- 172.21.15.189:0/676812475 >> v1:172.21.15.26:6790/0 conn(0x7fc89013b430 legacy=0x7fc8901d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:12.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 1 -- 172.21.15.189:0/676812475 >> v1:172.21.15.26:6789/0 conn(0x7fc8900a1b00 legacy=0x7fc8901c9810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:12.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:51:12.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 1 monclient: found mon.b 2024-09-16T14:51:12.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:12.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:12.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 1 -- 172.21.15.189:0/676812475 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89009e2f0 con 0x7fc8900a25f0 2024-09-16T14:51:12.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 10 monclient: _finish_auth 0 2024-09-16T14:51:12.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:51:12.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:12.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc895c9d640 5 monclient: authenticate success, global_id 184207 2024-09-16T14:51:12.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc895c9d640 10 monclient: _renew_subs 2024-09-16T14:51:12.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc895c9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:12.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc895c9d640 1 -- 172.21.15.189:0/676812475 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc89011f610 con 0x7fc8900a25f0 2024-09-16T14:51:12.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc895c9d640 10 monclient: _renew_subs 2024-09-16T14:51:12.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc895c9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:12.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc895c9d640 1 -- 172.21.15.189:0/676812475 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc890135920 con 0x7fc8900a25f0 2024-09-16T14:51:12.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 1 -- 172.21.15.189:0/676812475 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc878001d50 con 0x7fc8900a25f0 2024-09-16T14:51:12.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:12.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88e7fc640 4 set_mon_vals no callback set 2024-09-16T14:51:12.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 1 -- 172.21.15.189:0/676812475 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc878003a70 con 0x7fc8900a25f0 2024-09-16T14:51:12.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:12.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:12.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 10 monclient: dump: 2024-09-16T14:51:12.066 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:12.066 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:12.066 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:12.066 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:12.066 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:12.066 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:12.066 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:12.066 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:12.066 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:12.066 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:12.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc895c9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:12.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc895c9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:12.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc895c9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:12.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc895c9d640 1 -- 172.21.15.189:0/676812475 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc874005050 con 0x7fc8900a25f0 2024-09-16T14:51:12.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 1 -- 172.21.15.189:0/676812475 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc878003bd0 con 0x7fc8900a25f0 2024-09-16T14:51:12.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88cff9640 1 -- 172.21.15.189:0/676812475 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(250..250 src has 169..250) v4 ==== 6745+0+0 (unknown 571902908 0 0) 0x7fc878043f70 con 0x7fc8900a25f0 2024-09-16T14:51:12.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.051+0000 7fc88f7fe640 10 monclient: get_auth_request con 0x7fc870036060 auth_method 0 2024-09-16T14:51:12.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.055+0000 7fc88cff9640 1 -- 172.21.15.189:0/676812475 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc8780148b0 con 0x7fc8900a25f0 2024-09-16T14:51:12.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.055+0000 7fc88cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:12.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.055+0000 7fc88cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:12.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.219+0000 7fc895c9d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:12.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.219+0000 7fc895c9d640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:12.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.219+0000 7fc895c9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:12.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.219+0000 7fc895c9d640 1 -- 172.21.15.189:0/676812475 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"} v 0) v1 -- 0x7fc874005880 con 0x7fc8900a25f0 2024-09-16T14:51:12.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.803+0000 7fc88cff9640 1 -- 172.21.15.189:0/676812475 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}]=0 v251) v1 ==== 106+0+3 (unknown 2093150984 0 944643765) 0x7fc878012d20 con 0x7fc8900a25f0 2024-09-16T14:51:12.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.803+0000 7fc88cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:12.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.803+0000 7fc88cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:12.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.903+0000 7fc895c9d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:12.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.903+0000 7fc895c9d640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:12.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.903+0000 7fc895c9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:12.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.903+0000 7fc895c9d640 1 -- 172.21.15.189:0/676812475 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"} v 0) v1 -- 0x7fc874005300 con 0x7fc8900a25f0 2024-09-16T14:51:12.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.907+0000 7fc88cff9640 1 -- 172.21.15.189:0/676812475 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}]=0 v251) v1 ==== 106+0+3 (unknown 2093150984 0 944643765) 0x7fc8780144a0 con 0x7fc8900a25f0 2024-09-16T14:51:12.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.907+0000 7fc88cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:12.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.907+0000 7fc88cff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:51:12.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.907+0000 7fc895c9d640 1 -- 172.21.15.189:0/676812475 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc870036060 legacy=0x7fc870038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:12.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.907+0000 7fc895c9d640 10 monclient: shutdown 2024-09-16T14:51:12.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.907+0000 7fc895c9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:12.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.907+0000 7fc895c9d640 1 -- 172.21.15.189:0/676812475 >> v1:172.21.15.189:6789/0 conn(0x7fc8900a25f0 legacy=0x7fc8901ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:12.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.907+0000 7fc895c9d640 1 -- 172.21.15.189:0/676812475 shutdown_connections 2024-09-16T14:51:12.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.907+0000 7fc895c9d640 1 -- 172.21.15.189:0/676812475 >> 172.21.15.189:0/676812475 conn(0x7fc8900a7dd0 msgr2=0x7fc89009d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:12.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.907+0000 7fc895c9d640 1 -- 172.21.15.189:0/676812475 shutdown_connections 2024-09-16T14:51:12.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:12.907+0000 7fc895c9d640 1 -- 172.21.15.189:0/676812475 wait complete. 2024-09-16T14:51:12.925 INFO:tasks.workunit.client.0.smithi189.stderr:+ na=12 2024-09-16T14:51:12.925 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd create 0689a78d-9a3c-48ff-a75a-01eedf6472fa 2024-09-16T14:51:13.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab9368640 10 monclient: get_monmap_and_config 2024-09-16T14:51:13.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab9368640 10 monclient: build_initial_monmap 2024-09-16T14:51:13.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab9368640 10 monclient: monmap: 2024-09-16T14:51:13.041 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:13.041 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:13.041 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:51:13.042988+0000 2024-09-16T14:51:13.041 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:51:13.042988+0000 2024-09-16T14:51:13.041 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:13.041 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:13.042 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:13.042 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:13.042 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:13.042 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:13.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab9368640 1 Processor -- start 2024-09-16T14:51:13.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab9368640 1 -- start start 2024-09-16T14:51:13.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab9368640 10 monclient: init 2024-09-16T14:51:13.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab9368640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:13.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab9368640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:51:13.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab9368640 10 monclient(hunting): picked mon.noname-a con 0x7fbab40a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:13.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab9368640 10 monclient(hunting): picked mon.noname-b con 0x7fbab40a25f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:13.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab9368640 10 monclient(hunting): picked mon.noname-c con 0x7fbab413b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:13.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab9368640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab40a30b0 con 0x7fbab40a1b00 2024-09-16T14:51:13.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab9368640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab40a2e30 con 0x7fbab413b430 2024-09-16T14:51:13.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab9368640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab409eb50 con 0x7fbab40a25f0 2024-09-16T14:51:13.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab9368640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:13.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab9368640 10 monclient(hunting): authenticate will time out at 1455.159451s 2024-09-16T14:51:13.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab2ffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fbab40a1b00 legacy=0x7fbab40a1ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:13.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab37fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbab413b430 0x7fbab413d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48226/0 (socket says 172.21.15.189:48226) 2024-09-16T14:51:13.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab37fe640 1 -- 172.21.15.189:0/1801280015 learned_addr learned my addr 172.21.15.189:0/1801280015 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:13.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab27fc640 1 -- 172.21.15.189:0/1801280015 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345584666 0 0) 0x7fbab40a2e30 con 0x7fbab413b430 2024-09-16T14:51:13.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab27fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:13.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:13.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:13.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab27fc640 10 monclient(hunting): my global_id is 184210 2024-09-16T14:51:13.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.039+0000 7fbab27fc640 1 -- 172.21.15.189:0/1801280015 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa0003530 con 0x7fbab413b430 2024-09-16T14:51:13.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 1 -- 172.21.15.189:0/1801280015 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3529805838 0 0) 0x7fbaa0003530 con 0x7fbab413b430 2024-09-16T14:51:13.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 1 -- 172.21.15.189:0/1801280015 >> v1:172.21.15.26:6790/0 conn(0x7fbab40a25f0 legacy=0x7fbab4139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:13.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 1 -- 172.21.15.189:0/1801280015 >> v1:172.21.15.26:6789/0 conn(0x7fbab40a1b00 legacy=0x7fbab40a1ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:13.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 10 monclient: _finish_hunting 0 2024-09-16T14:51:13.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 1 monclient: found mon.noname-c 2024-09-16T14:51:13.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:13.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:51:13.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 1 -- 172.21.15.189:0/1801280015 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab409ee30 con 0x7fbab413b430 2024-09-16T14:51:13.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 10 monclient: _finish_auth 0 2024-09-16T14:51:13.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 10 monclient: _check_auth_tickets 2024-09-16T14:51:13.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:13.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 1 -- 172.21.15.189:0/1801280015 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbaac0019a0 con 0x7fbab413b430 2024-09-16T14:51:13.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:13.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:13.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 10 monclient: dump: 2024-09-16T14:51:13.047 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:13.047 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:13.048 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:13.048 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:13.048 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:13.048 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:13.048 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:13.048 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:13.048 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:13.048 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:13.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 5 monclient: authenticate success, global_id 184210 2024-09-16T14:51:13.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:13.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 1 -- 172.21.15.189:0/1801280015 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbaac001cc0 con 0x7fbab413b430 2024-09-16T14:51:13.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:13.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 1 -- 172.21.15.189:0/1801280015 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbaac002050 con 0x7fbab413b430 2024-09-16T14:51:13.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:13.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 4 set_mon_vals no callback set 2024-09-16T14:51:13.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:13.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:13.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab27fc640 10 monclient: dump: 2024-09-16T14:51:13.049 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:13.049 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:13.049 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:13.049 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:13.049 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:13.049 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:13.050 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:13.050 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:13.050 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:13.050 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:13.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 10 monclient: shutdown 2024-09-16T14:51:13.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:13.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 1 -- 172.21.15.189:0/1801280015 >> v1:172.21.15.189:6789/0 conn(0x7fbab413b430 legacy=0x7fbab413d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:13.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 1 -- 172.21.15.189:0/1801280015 shutdown_connections 2024-09-16T14:51:13.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 1 -- 172.21.15.189:0/1801280015 >> 172.21.15.189:0/1801280015 conn(0x7fbab40a7dd0 msgr2=0x7fbab40a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:13.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 1 -- 172.21.15.189:0/1801280015 shutdown_connections 2024-09-16T14:51:13.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 1 -- 172.21.15.189:0/1801280015 wait complete. 2024-09-16T14:51:13.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 10 monclient: build_initial_monmap 2024-09-16T14:51:13.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 10 monclient: monmap: 2024-09-16T14:51:13.055 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:13.055 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:13.055 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:51:13.055 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:51:13.055 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:13.055 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:51:13.055 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:13.055 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:13.055 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:13.055 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:13.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 1 Processor -- start 2024-09-16T14:51:13.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 1 -- start start 2024-09-16T14:51:13.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 10 monclient: init 2024-09-16T14:51:13.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:13.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:51:13.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 10 monclient(hunting): picked mon.noname-c con 0x7fbab40a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:13.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 10 monclient(hunting): picked mon.noname-a con 0x7fbab40a25f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:13.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 10 monclient(hunting): picked mon.noname-b con 0x7fbab413b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:13.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab413a450 con 0x7fbab40a25f0 2024-09-16T14:51:13.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab411f830 con 0x7fbab413b430 2024-09-16T14:51:13.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab41ca3d0 con 0x7fbab40a1b00 2024-09-16T14:51:13.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:13.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 10 monclient(hunting): authenticate will time out at 1455.162504s 2024-09-16T14:51:13.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab37fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbab413b430 0x7fbab41c9cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48238/0 (socket says 172.21.15.189:48238) 2024-09-16T14:51:13.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab37fe640 1 -- 172.21.15.189:0/3787021816 learned_addr learned my addr 172.21.15.189:0/3787021816 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:13.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaabfff640 1 -- 172.21.15.189:0/3787021816 >> v1:172.21.15.26:6789/0 conn(0x7fbab40a25f0 legacy=0x7fbab41c95b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:13.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236726704 0 0) 0x7fbab411f830 con 0x7fbab413b430 2024-09-16T14:51:13.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:13.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:13.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:13.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient(hunting): my global_id is 184213 2024-09-16T14:51:13.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba94003530 con 0x7fbab413b430 2024-09-16T14:51:13.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3486023496 0 0) 0x7fba94003530 con 0x7fbab413b430 2024-09-16T14:51:13.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbab411f830 con 0x7fbab413b430 2024-09-16T14:51:13.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbaac0017c0 con 0x7fbab413b430 2024-09-16T14:51:13.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:13.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:13.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient(hunting): dump: 2024-09-16T14:51:13.058 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:13.058 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:13.058 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:13.058 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:13.058 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:13.058 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:13.058 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:13.058 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:13.058 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:13.059 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:13.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:13.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428468139 0 0) 0x7fbab41ca3d0 con 0x7fbab40a1b00 2024-09-16T14:51:13.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:13.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:13.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:13.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient(hunting): my global_id is 154922 2024-09-16T14:51:13.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba94003530 con 0x7fbab40a1b00 2024-09-16T14:51:13.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1597404229 0 0) 0x7fbab411f830 con 0x7fbab413b430 2024-09-16T14:51:13.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 >> v1:172.21.15.26:6790/0 conn(0x7fbab40a1b00 legacy=0x7fbab413a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:13.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 >> v1:172.21.15.26:6789/0 conn(0x7fbab40a25f0 legacy=0x7fbab41c95b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:13.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:51:13.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 1 monclient: found mon.b 2024-09-16T14:51:13.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:13.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:13.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab411fd80 con 0x7fbab413b430 2024-09-16T14:51:13.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient: _finish_auth 0 2024-09-16T14:51:13.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:51:13.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:13.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 5 monclient: authenticate success, global_id 184213 2024-09-16T14:51:13.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 10 monclient: _renew_subs 2024-09-16T14:51:13.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:13.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 1 -- 172.21.15.189:0/3787021816 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab411faa0 con 0x7fbab413b430 2024-09-16T14:51:13.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 10 monclient: _renew_subs 2024-09-16T14:51:13.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:13.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab9368640 1 -- 172.21.15.189:0/3787021816 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab41d2890 con 0x7fbab413b430 2024-09-16T14:51:13.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbaac004fd0 con 0x7fbab413b430 2024-09-16T14:51:13.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:13.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbaac0052e0 con 0x7fbab413b430 2024-09-16T14:51:13.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:13.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:13.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbab1ffb640 4 set_mon_vals no callback set 2024-09-16T14:51:13.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.043+0000 7fbaab7fe640 10 monclient: dump: 2024-09-16T14:51:13.061 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:13.061 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:13.061 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:13.062 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:13.062 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:13.062 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:13.062 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:13.062 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:13.062 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:13.062 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:13.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.047+0000 7fbab9368640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:13.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.047+0000 7fbab9368640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:13.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.047+0000 7fbab9368640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:13.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.047+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbaac00fd50 con 0x7fbab413b430 2024-09-16T14:51:13.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.047+0000 7fbab9368640 1 -- 172.21.15.189:0/3787021816 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba98005050 con 0x7fbab413b430 2024-09-16T14:51:13.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.047+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(251..251 src has 169..251) v4 ==== 6745+0+0 (unknown 1734246821 0 0) 0x7fbaac0443f0 con 0x7fbab413b430 2024-09-16T14:51:13.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.047+0000 7fbab2ffd640 10 monclient: get_auth_request con 0x7fba94036280 auth_method 0 2024-09-16T14:51:13.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.051+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbaac014c00 con 0x7fbab413b430 2024-09-16T14:51:13.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.051+0000 7fbaab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:13.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.051+0000 7fbaab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:13.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.211+0000 7fbab9368640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:13.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.211+0000 7fbab9368640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:13.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.211+0000 7fbab9368640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:13.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.211+0000 7fbab9368640 1 -- 172.21.15.189:0/3787021816 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"} v 0) v1 -- 0x7fba98005880 con 0x7fbab413b430 2024-09-16T14:51:13.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.211+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}]=0 v251) v1 ==== 106+0+3 (unknown 2093150984 0 944643765) 0x7fbaac0132a0 con 0x7fbab413b430 2024-09-16T14:51:13.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.211+0000 7fbaab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:13.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.211+0000 7fbaab7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:13.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.315+0000 7fbab9368640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:13.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.315+0000 7fbab9368640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:13.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.315+0000 7fbab9368640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:13.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.315+0000 7fbab9368640 1 -- 172.21.15.189:0/3787021816 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"} v 0) v1 -- 0x7fba98005300 con 0x7fbab413b430 2024-09-16T14:51:13.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.319+0000 7fbaab7fe640 1 -- 172.21.15.189:0/3787021816 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}]=0 v251) v1 ==== 106+0+3 (unknown 2093150984 0 944643765) 0x7fbaac014a20 con 0x7fbab413b430 2024-09-16T14:51:13.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.319+0000 7fbaab7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:13.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.319+0000 7fbaab7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:51:13.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.319+0000 7fbab9368640 1 -- 172.21.15.189:0/3787021816 >> v1:172.21.15.189:6812/3196161582 conn(0x7fba94036280 legacy=0x7fba94038740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:13.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.319+0000 7fbab9368640 10 monclient: shutdown 2024-09-16T14:51:13.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.319+0000 7fbab9368640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:13.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.319+0000 7fbab9368640 1 -- 172.21.15.189:0/3787021816 >> v1:172.21.15.189:6789/0 conn(0x7fbab413b430 legacy=0x7fbab41c9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:13.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.319+0000 7fbab9368640 1 -- 172.21.15.189:0/3787021816 shutdown_connections 2024-09-16T14:51:13.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.319+0000 7fbab9368640 1 -- 172.21.15.189:0/3787021816 >> 172.21.15.189:0/3787021816 conn(0x7fbab40a7dd0 msgr2=0x7fbab409d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:13.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.319+0000 7fbab9368640 1 -- 172.21.15.189:0/3787021816 shutdown_connections 2024-09-16T14:51:13.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.319+0000 7fbab9368640 1 -- 172.21.15.189:0/3787021816 wait complete. 2024-09-16T14:51:13.338 INFO:tasks.workunit.client.0.smithi189.stderr:+ test 12 -eq 12 2024-09-16T14:51:13.338 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd create ee517a82-a411-40e2-ac30-45aa35010440 2024-09-16T14:51:13.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.455+0000 7f8617118640 10 monclient: get_monmap_and_config 2024-09-16T14:51:13.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.455+0000 7f8617118640 10 monclient: build_initial_monmap 2024-09-16T14:51:13.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.455+0000 7f8617118640 10 monclient: monmap: 2024-09-16T14:51:13.459 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:13.459 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:13.459 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:51:13.460336+0000 2024-09-16T14:51:13.459 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:51:13.460336+0000 2024-09-16T14:51:13.459 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:13.459 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:13.459 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:13.459 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:13.459 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:13.459 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:13.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.455+0000 7f8617118640 1 Processor -- start 2024-09-16T14:51:13.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.455+0000 7f8617118640 1 -- start start 2024-09-16T14:51:13.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.455+0000 7f8617118640 10 monclient: init 2024-09-16T14:51:13.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.455+0000 7f8617118640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:13.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.455+0000 7f8617118640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:51:13.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.455+0000 7f8617118640 10 monclient(hunting): picked mon.noname-c con 0x7f861009df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:13.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.455+0000 7f8617118640 10 monclient(hunting): picked mon.noname-b con 0x7f861009e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:13.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.455+0000 7f8617118640 10 monclient(hunting): picked mon.noname-a con 0x7f861009e740 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:13.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.455+0000 7f8617118640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861009ee20 con 0x7f861009e740 2024-09-16T14:51:13.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.455+0000 7f8617118640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861009f100 con 0x7f861009df10 2024-09-16T14:51:13.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.455+0000 7f8617118640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861009f3e0 con 0x7f861009e2f0 2024-09-16T14:51:13.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.455+0000 7f8617118640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:13.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.455+0000 7f8617118640 10 monclient(hunting): authenticate will time out at 1455.575960s 2024-09-16T14:51:13.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8614eb4640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f861009df10 0x7f861009d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48244/0 (socket says 172.21.15.189:48244) 2024-09-16T14:51:13.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8614eb4640 1 -- 172.21.15.189:0/3873408722 learned_addr learned my addr 172.21.15.189:0/3873408722 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:13.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f86156b5640 1 -- 172.21.15.189:0/3873408722 >> v1:172.21.15.26:6789/0 conn(0x7f861009e740 legacy=0x7f861013b000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:13.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 1 -- 172.21.15.189:0/3873408722 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743421367 0 0) 0x7f861009f100 con 0x7f861009df10 2024-09-16T14:51:13.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:13.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:13.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:13.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 10 monclient(hunting): my global_id is 184216 2024-09-16T14:51:13.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 1 -- 172.21.15.189:0/3873408722 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f8003530 con 0x7f861009df10 2024-09-16T14:51:13.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 1 -- 172.21.15.189:0/3873408722 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3235360727 0 0) 0x7f85f8003530 con 0x7f861009df10 2024-09-16T14:51:13.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 1 -- 172.21.15.189:0/3873408722 >> v1:172.21.15.26:6790/0 conn(0x7f861009e2f0 legacy=0x7f86101378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:13.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 1 -- 172.21.15.189:0/3873408722 >> v1:172.21.15.26:6789/0 conn(0x7f861009e740 legacy=0x7f861013b000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:13.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:51:13.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 1 monclient: found mon.noname-c 2024-09-16T14:51:13.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:13.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:51:13.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 1 -- 172.21.15.189:0/3873408722 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861009c690 con 0x7f861009df10 2024-09-16T14:51:13.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:51:13.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:51:13.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:13.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 1 -- 172.21.15.189:0/3873408722 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8600001620 con 0x7f861009df10 2024-09-16T14:51:13.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:13.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:13.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 10 monclient: dump: 2024-09-16T14:51:13.468 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:13.468 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:13.468 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:13.468 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:13.469 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:13.469 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:13.469 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:13.469 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:13.469 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:13.469 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:13.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 5 monclient: authenticate success, global_id 184216 2024-09-16T14:51:13.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:13.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 1 -- 172.21.15.189:0/3873408722 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8600001940 con 0x7f861009df10 2024-09-16T14:51:13.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:13.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:13.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 1 -- 172.21.15.189:0/3873408722 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8600001d10 con 0x7f861009df10 2024-09-16T14:51:13.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:13.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 4 set_mon_vals no callback set 2024-09-16T14:51:13.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:13.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860f7fe640 10 monclient: dump: 2024-09-16T14:51:13.470 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:13.470 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:13.470 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:13.470 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:13.470 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:13.470 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:13.470 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:13.470 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:13.470 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:13.471 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:13.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 10 monclient: shutdown 2024-09-16T14:51:13.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:13.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 1 -- 172.21.15.189:0/3873408722 >> v1:172.21.15.189:6789/0 conn(0x7f861009df10 legacy=0x7f861009d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:13.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 1 -- 172.21.15.189:0/3873408722 shutdown_connections 2024-09-16T14:51:13.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 1 -- 172.21.15.189:0/3873408722 >> 172.21.15.189:0/3873408722 conn(0x7f86100a3320 msgr2=0x7f861009f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:13.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 1 -- 172.21.15.189:0/3873408722 shutdown_connections 2024-09-16T14:51:13.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 1 -- 172.21.15.189:0/3873408722 wait complete. 2024-09-16T14:51:13.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 10 monclient: build_initial_monmap 2024-09-16T14:51:13.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 10 monclient: monmap: 2024-09-16T14:51:13.471 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:13.471 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:13.471 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:51:13.471 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:51:13.472 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:13.472 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:51:13.472 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:13.472 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:13.472 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:13.472 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:13.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 1 Processor -- start 2024-09-16T14:51:13.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 1 -- start start 2024-09-16T14:51:13.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 10 monclient: init 2024-09-16T14:51:13.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:13.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:51:13.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 10 monclient(hunting): picked mon.noname-a con 0x7f861009df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:13.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 10 monclient(hunting): picked mon.noname-b con 0x7f861009e2f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:13.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 10 monclient(hunting): picked mon.noname-c con 0x7f861009e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:13.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861006f970 con 0x7f861009df10 2024-09-16T14:51:13.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861006b380 con 0x7f861009e2f0 2024-09-16T14:51:13.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8614eb4640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f861009df10 legacy=0x7f8610132b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:13.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861006cea0 con 0x7f861009e740 2024-09-16T14:51:13.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:13.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f8617118640 10 monclient(hunting): authenticate will time out at 1455.579298s 2024-09-16T14:51:13.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860ffff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f861009e2f0 0x7f86101cbfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48254/0 (socket says 172.21.15.189:48254) 2024-09-16T14:51:13.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860ffff640 1 -- 172.21.15.189:0/146465574 learned_addr learned my addr 172.21.15.189:0/146465574 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:13.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860d7fa640 1 -- 172.21.15.189:0/146465574 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373894863 0 0) 0x7f861006b380 con 0x7f861009e2f0 2024-09-16T14:51:13.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:51:13.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:13.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:13.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860d7fa640 10 monclient(hunting): my global_id is 184219 2024-09-16T14:51:13.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860d7fa640 1 -- 172.21.15.189:0/146465574 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85fc003530 con 0x7f861009e2f0 2024-09-16T14:51:13.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860d7fa640 1 -- 172.21.15.189:0/146465574 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1689948748 0 0) 0x7f85fc003530 con 0x7f861009e2f0 2024-09-16T14:51:13.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860d7fa640 1 -- 172.21.15.189:0/146465574 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f861006b380 con 0x7f861009e2f0 2024-09-16T14:51:13.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860d7fa640 1 -- 172.21.15.189:0/146465574 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8604001980 con 0x7f861009e2f0 2024-09-16T14:51:13.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:13.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:13.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860d7fa640 10 monclient(hunting): dump: 2024-09-16T14:51:13.474 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:13.475 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:13.475 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:13.475 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:13.475 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:13.475 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:13.475 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:13.475 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:13.475 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:13.475 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:13.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.459+0000 7f860d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:13.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 1 -- 172.21.15.189:0/146465574 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4172491277 0 0) 0x7f861006b380 con 0x7f861009e2f0 2024-09-16T14:51:13.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 1 -- 172.21.15.189:0/146465574 >> v1:172.21.15.26:6790/0 conn(0x7f861009e740 legacy=0x7f86101cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:13.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 1 -- 172.21.15.189:0/146465574 >> v1:172.21.15.26:6789/0 conn(0x7f861009df10 legacy=0x7f8610132b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:13.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:51:13.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 1 monclient: found mon.b 2024-09-16T14:51:13.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:13.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:13.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 1 -- 172.21.15.189:0/146465574 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861011a490 con 0x7f861009e2f0 2024-09-16T14:51:13.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:51:13.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:51:13.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:13.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f8617118640 5 monclient: authenticate success, global_id 184219 2024-09-16T14:51:13.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f8617118640 10 monclient: _renew_subs 2024-09-16T14:51:13.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f8617118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:13.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f8617118640 1 -- 172.21.15.189:0/146465574 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f861013b460 con 0x7f861009e2f0 2024-09-16T14:51:13.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f8617118640 10 monclient: _renew_subs 2024-09-16T14:51:13.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f8617118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:13.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f8617118640 1 -- 172.21.15.189:0/146465574 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f861013c250 con 0x7f861009e2f0 2024-09-16T14:51:13.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f8617118640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:13.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f8617118640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:13.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f8617118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:13.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f8617118640 1 -- 172.21.15.189:0/146465574 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86100008e0 con 0x7f861009e2f0 2024-09-16T14:51:13.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 1 -- 172.21.15.189:0/146465574 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86040038d0 con 0x7f861009e2f0 2024-09-16T14:51:13.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:13.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 1 -- 172.21.15.189:0/146465574 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8604001d90 con 0x7f861009e2f0 2024-09-16T14:51:13.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:13.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:13.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 10 monclient: dump: 2024-09-16T14:51:13.478 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:13.478 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:13.478 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:13.478 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:13.478 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:13.478 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:13.478 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:13.478 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:13.478 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:13.478 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:13.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860effd640 4 set_mon_vals no callback set 2024-09-16T14:51:13.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 1 -- 172.21.15.189:0/146465574 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8604003c60 con 0x7f861009e2f0 2024-09-16T14:51:13.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f860d7fa640 1 -- 172.21.15.189:0/146465574 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(251..251 src has 169..251) v4 ==== 6745+0+0 (unknown 1734246821 0 0) 0x7f8604043d90 con 0x7f861009e2f0 2024-09-16T14:51:13.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.463+0000 7f8614eb4640 10 monclient: get_auth_request con 0x7f85fc036000 auth_method 0 2024-09-16T14:51:13.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.467+0000 7f860d7fa640 1 -- 172.21.15.189:0/146465574 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8604014730 con 0x7f861009e2f0 2024-09-16T14:51:13.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.467+0000 7f860d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:13.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.467+0000 7f860d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:13.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.623+0000 7f8617118640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"}] 2024-09-16T14:51:13.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.623+0000 7f8617118640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"}] 2024-09-16T14:51:13.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.623+0000 7f8617118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:13.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.623+0000 7f8617118640 1 -- 172.21.15.189:0/146465574 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"} v 0) v1 -- 0x7f861013ae80 con 0x7f861009e2f0 2024-09-16T14:51:13.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.811+0000 7f860d7fa640 1 -- 172.21.15.189:0/146465574 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"}]=0 v252) v1 ==== 106+0+3 (unknown 2501489045 0 1367428999) 0x7f8604012ba0 con 0x7f861009e2f0 2024-09-16T14:51:13.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.811+0000 7f860d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"}] 2024-09-16T14:51:13.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.811+0000 7f860d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:13.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.907+0000 7f8617118640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"}] 2024-09-16T14:51:13.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.907+0000 7f8617118640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"}] 2024-09-16T14:51:13.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.907+0000 7f8617118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:13.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.907+0000 7f8617118640 1 -- 172.21.15.189:0/146465574 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"} v 0) v1 -- 0x7f8610099cd0 con 0x7f861009e2f0 2024-09-16T14:51:13.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.911+0000 7f860d7fa640 1 -- 172.21.15.189:0/146465574 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"}]=0 v252) v1 ==== 106+0+3 (unknown 2501489045 0 1367428999) 0x7f8604014320 con 0x7f861009e2f0 2024-09-16T14:51:13.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.911+0000 7f860d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"}] 2024-09-16T14:51:13.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.911+0000 7f860d7fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:51:13.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.911+0000 7f8617118640 1 -- 172.21.15.189:0/146465574 >> v1:172.21.15.189:6812/3196161582 conn(0x7f85fc036000 legacy=0x7f85fc0384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:13.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.911+0000 7f8617118640 10 monclient: shutdown 2024-09-16T14:51:13.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.911+0000 7f8617118640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:13.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.911+0000 7f8617118640 1 -- 172.21.15.189:0/146465574 >> v1:172.21.15.189:6789/0 conn(0x7f861009e2f0 legacy=0x7f86101cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:13.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.911+0000 7f8617118640 1 -- 172.21.15.189:0/146465574 shutdown_connections 2024-09-16T14:51:13.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.911+0000 7f8617118640 1 -- 172.21.15.189:0/146465574 >> 172.21.15.189:0/146465574 conn(0x7f86100a3320 msgr2=0x7f861006e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:13.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.911+0000 7f8617118640 1 -- 172.21.15.189:0/146465574 shutdown_connections 2024-09-16T14:51:13.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:13.911+0000 7f8617118640 1 -- 172.21.15.189:0/146465574 wait complete. 2024-09-16T14:51:13.928 INFO:tasks.workunit.client.0.smithi189.stderr:+ nb=14 2024-09-16T14:51:13.928 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd create ee517a82-a411-40e2-ac30-45aa35010440 2024-09-16T14:51:14.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8681f97640 10 monclient: get_monmap_and_config 2024-09-16T14:51:14.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8681f97640 10 monclient: build_initial_monmap 2024-09-16T14:51:14.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8681f97640 10 monclient: monmap: 2024-09-16T14:51:14.041 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:14.041 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:14.041 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:51:14.043169+0000 2024-09-16T14:51:14.042 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:51:14.043169+0000 2024-09-16T14:51:14.042 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:14.042 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:14.042 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:14.042 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:14.042 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:14.042 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:14.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8681f97640 1 Processor -- start 2024-09-16T14:51:14.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8681f97640 1 -- start start 2024-09-16T14:51:14.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8681f97640 10 monclient: init 2024-09-16T14:51:14.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8681f97640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:14.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8681f97640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:51:14.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8681f97640 10 monclient(hunting): picked mon.noname-c con 0x7f867c099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:14.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8681f97640 10 monclient(hunting): picked mon.noname-a con 0x7f867c1334d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:14.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8681f97640 10 monclient(hunting): picked mon.noname-b con 0x7f867c136fe0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:14.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8681f97640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867c09b350 con 0x7f867c1334d0 2024-09-16T14:51:14.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8681f97640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867c09f2e0 con 0x7f867c099a70 2024-09-16T14:51:14.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8681f97640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867c09b070 con 0x7f867c136fe0 2024-09-16T14:51:14.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8681f97640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8681f97640 10 monclient(hunting): authenticate will time out at 1456.159826s 2024-09-16T14:51:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f867b7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f867c099a70 0x7f867c09bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48262/0 (socket says 172.21.15.189:48262) 2024-09-16T14:51:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f867b7fe640 1 -- 172.21.15.189:0/2081186042 learned_addr learned my addr 172.21.15.189:0/2081186042 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.039+0000 7f8672ffd640 1 -- 172.21.15.189:0/2081186042 >> v1:172.21.15.26:6789/0 conn(0x7f867c1334d0 legacy=0x7f867c1358c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 1 -- 172.21.15.189:0/2081186042 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161791328 0 0) 0x7f867c09f2e0 con 0x7f867c099a70 2024-09-16T14:51:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 10 monclient(hunting): _negotiate 2024-09-16T14:51:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 10 monclient(hunting): my global_id is 184222 2024-09-16T14:51:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 1 -- 172.21.15.189:0/2081186042 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8668003530 con 0x7f867c099a70 2024-09-16T14:51:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 1 -- 172.21.15.189:0/2081186042 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 961841558 0 0) 0x7f8668003530 con 0x7f867c099a70 2024-09-16T14:51:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 1 -- 172.21.15.189:0/2081186042 >> v1:172.21.15.26:6790/0 conn(0x7f867c136fe0 legacy=0x7f867c1393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:14.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 1 -- 172.21.15.189:0/2081186042 >> v1:172.21.15.26:6789/0 conn(0x7f867c1334d0 legacy=0x7f867c1358c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:14.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 10 monclient: _finish_hunting 0 2024-09-16T14:51:14.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 1 monclient: found mon.noname-c 2024-09-16T14:51:14.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:14.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:51:14.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 1 -- 172.21.15.189:0/2081186042 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c09b630 con 0x7f867c099a70 2024-09-16T14:51:14.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 10 monclient: _finish_auth 0 2024-09-16T14:51:14.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 10 monclient: _check_auth_tickets 2024-09-16T14:51:14.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:14.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 1 -- 172.21.15.189:0/2081186042 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f866c001620 con 0x7f867c099a70 2024-09-16T14:51:14.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 5 monclient: authenticate success, global_id 184222 2024-09-16T14:51:14.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:14.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:14.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:14.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 10 monclient: dump: 2024-09-16T14:51:14.052 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:14.052 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:14.052 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:14.052 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:14.052 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:14.052 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:14.052 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:14.052 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:14.052 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:14.052 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:14.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:14.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 1 -- 172.21.15.189:0/2081186042 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f866c0018d0 con 0x7f867c099a70 2024-09-16T14:51:14.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:14.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 1 -- 172.21.15.189:0/2081186042 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f866c001c60 con 0x7f867c099a70 2024-09-16T14:51:14.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:14.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:14.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867affd640 10 monclient: dump: 2024-09-16T14:51:14.057 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:14.057 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:14.057 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:14.057 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:14.057 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:14.057 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:14.057 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:14.057 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:14.057 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:14.058 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:14.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:14.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 4 set_mon_vals no callback set 2024-09-16T14:51:14.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 10 monclient: shutdown 2024-09-16T14:51:14.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:14.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 1 -- 172.21.15.189:0/2081186042 >> v1:172.21.15.189:6789/0 conn(0x7f867c099a70 legacy=0x7f867c09bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:14.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 1 -- 172.21.15.189:0/2081186042 shutdown_connections 2024-09-16T14:51:14.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 1 -- 172.21.15.189:0/2081186042 >> 172.21.15.189:0/2081186042 conn(0x7f867c0a3320 msgr2=0x7f867c09f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:14.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 1 -- 172.21.15.189:0/2081186042 shutdown_connections 2024-09-16T14:51:14.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 1 -- 172.21.15.189:0/2081186042 wait complete. 2024-09-16T14:51:14.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 10 monclient: build_initial_monmap 2024-09-16T14:51:14.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 10 monclient: monmap: 2024-09-16T14:51:14.058 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:14.059 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:14.059 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:51:14.059 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:51:14.059 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:14.059 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:51:14.059 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:14.059 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:14.059 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:14.059 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:14.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 1 Processor -- start 2024-09-16T14:51:14.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 1 -- start start 2024-09-16T14:51:14.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 10 monclient: init 2024-09-16T14:51:14.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:14.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:51:14.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 10 monclient(hunting): picked mon.noname-c con 0x7f867c099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:14.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 10 monclient(hunting): picked mon.noname-a con 0x7f867c1334d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:14.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 10 monclient(hunting): picked mon.noname-b con 0x7f867c136fe0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:14.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867c06b380 con 0x7f867c1334d0 2024-09-16T14:51:14.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867c136b50 con 0x7f867c136fe0 2024-09-16T14:51:14.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867c135440 con 0x7f867c099a70 2024-09-16T14:51:14.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:14.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 10 monclient(hunting): authenticate will time out at 1456.162757s 2024-09-16T14:51:14.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8672ffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f867c1334d0 legacy=0x7f867c1ce4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:14.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f867c136fe0 0x7f867c1d1bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48272/0 (socket says 172.21.15.189:48272) 2024-09-16T14:51:14.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867bfff640 1 -- 172.21.15.189:0/312669813 learned_addr learned my addr 172.21.15.189:0/312669813 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:14.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098907912 0 0) 0x7f867c136b50 con 0x7f867c136fe0 2024-09-16T14:51:14.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:51:14.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:14.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:14.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient(hunting): my global_id is 184225 2024-09-16T14:51:14.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f865c003560 con 0x7f867c136fe0 2024-09-16T14:51:14.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2563406608 0 0) 0x7f865c003560 con 0x7f867c136fe0 2024-09-16T14:51:14.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f867c136b50 con 0x7f867c136fe0 2024-09-16T14:51:14.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8674001c00 con 0x7f867c136fe0 2024-09-16T14:51:14.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:14.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:14.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient(hunting): dump: 2024-09-16T14:51:14.062 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:14.062 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:14.062 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:14.062 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:14.062 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:14.062 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:14.063 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:14.063 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:14.063 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:14.063 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:14.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:14.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364027586 0 0) 0x7f867c135440 con 0x7f867c099a70 2024-09-16T14:51:14.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:51:14.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:14.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:14.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient(hunting): my global_id is 154934 2024-09-16T14:51:14.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f865c003560 con 0x7f867c099a70 2024-09-16T14:51:14.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4136107169 0 0) 0x7f867c136b50 con 0x7f867c136fe0 2024-09-16T14:51:14.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 >> v1:172.21.15.26:6790/0 conn(0x7f867c099a70 legacy=0x7f867c132540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 >> v1:172.21.15.26:6789/0 conn(0x7f867c1334d0 legacy=0x7f867c1ce4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:51:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 1 monclient: found mon.b 2024-09-16T14:51:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c1216a0 con 0x7f867c136fe0 2024-09-16T14:51:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient: _finish_auth 0 2024-09-16T14:51:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:51:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 5 monclient: authenticate success, global_id 184225 2024-09-16T14:51:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 10 monclient: _renew_subs 2024-09-16T14:51:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 1 -- 172.21.15.189:0/312669813 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f867c1bc590 con 0x7f867c136fe0 2024-09-16T14:51:14.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 10 monclient: _renew_subs 2024-09-16T14:51:14.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:14.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8681f97640 1 -- 172.21.15.189:0/312669813 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f867c120910 con 0x7f867c136fe0 2024-09-16T14:51:14.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8674001fd0 con 0x7f867c136fe0 2024-09-16T14:51:14.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:14.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8674003e70 con 0x7f867c136fe0 2024-09-16T14:51:14.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:14.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:14.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f867a7fc640 4 set_mon_vals no callback set 2024-09-16T14:51:14.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.043+0000 7f8678ff9640 10 monclient: dump: 2024-09-16T14:51:14.065 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:14.065 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:14.065 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:14.066 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.047+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8674004050 con 0x7f867c136fe0 2024-09-16T14:51:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.047+0000 7f8681f97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.047+0000 7f8681f97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.047+0000 7f8681f97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.047+0000 7f8681f97640 1 -- 172.21.15.189:0/312669813 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8660005050 con 0x7f867c136fe0 2024-09-16T14:51:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.047+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(252..252 src has 169..252) v4 ==== 6745+0+0 (unknown 3091906843 0 0) 0x7f8674044270 con 0x7f867c136fe0 2024-09-16T14:51:14.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.047+0000 7f867b7fe640 10 monclient: get_auth_request con 0x7f865c036110 auth_method 0 2024-09-16T14:51:14.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.051+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8674014980 con 0x7f867c136fe0 2024-09-16T14:51:14.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.051+0000 7f8678ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:14.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.051+0000 7f8678ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:14.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.207+0000 7f8681f97640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"}] 2024-09-16T14:51:14.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.207+0000 7f8681f97640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"}] 2024-09-16T14:51:14.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.207+0000 7f8681f97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:14.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.207+0000 7f8681f97640 1 -- 172.21.15.189:0/312669813 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"} v 0) v1 -- 0x7f8660005880 con 0x7f867c136fe0 2024-09-16T14:51:14.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.207+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"}]=0 v252) v1 ==== 106+0+3 (unknown 2501489045 0 1367428999) 0x7f8674013020 con 0x7f867c136fe0 2024-09-16T14:51:14.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.207+0000 7f8678ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"}] 2024-09-16T14:51:14.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.207+0000 7f8678ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:14.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.311+0000 7f8681f97640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"}] 2024-09-16T14:51:14.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.311+0000 7f8681f97640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"}] 2024-09-16T14:51:14.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.311+0000 7f8681f97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:14.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.311+0000 7f8681f97640 1 -- 172.21.15.189:0/312669813 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"} v 0) v1 -- 0x7f8660005300 con 0x7f867c136fe0 2024-09-16T14:51:14.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.315+0000 7f8678ff9640 1 -- 172.21.15.189:0/312669813 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"}]=0 v252) v1 ==== 106+0+3 (unknown 2501489045 0 1367428999) 0x7f86740147a0 con 0x7f867c136fe0 2024-09-16T14:51:14.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.315+0000 7f8678ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "ee517a82-a411-40e2-ac30-45aa35010440"}] 2024-09-16T14:51:14.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.315+0000 7f8678ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:51:14.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.315+0000 7f8681f97640 1 -- 172.21.15.189:0/312669813 >> v1:172.21.15.189:6812/3196161582 conn(0x7f865c036110 legacy=0x7f865c0385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:14.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.315+0000 7f8681f97640 10 monclient: shutdown 2024-09-16T14:51:14.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.315+0000 7f8681f97640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:14.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.315+0000 7f8681f97640 1 -- 172.21.15.189:0/312669813 >> v1:172.21.15.189:6789/0 conn(0x7f867c136fe0 legacy=0x7f867c1d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:14.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.315+0000 7f8681f97640 1 -- 172.21.15.189:0/312669813 shutdown_connections 2024-09-16T14:51:14.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.315+0000 7f8681f97640 1 -- 172.21.15.189:0/312669813 >> 172.21.15.189:0/312669813 conn(0x7f867c0a3320 msgr2=0x7f867c070f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:14.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.315+0000 7f8681f97640 1 -- 172.21.15.189:0/312669813 shutdown_connections 2024-09-16T14:51:14.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.315+0000 7f8681f97640 1 -- 172.21.15.189:0/312669813 wait complete. 2024-09-16T14:51:14.333 INFO:tasks.workunit.client.0.smithi189.stderr:+ test 14 -eq 14 2024-09-16T14:51:14.333 INFO:tasks.workunit.client.0.smithi189.stderr:+ test 14 -ne 12 2024-09-16T14:51:14.333 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd rm 12 2024-09-16T14:51:14.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 10 monclient: get_monmap_and_config 2024-09-16T14:51:14.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 10 monclient: build_initial_monmap 2024-09-16T14:51:14.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 10 monclient: monmap: 2024-09-16T14:51:14.449 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:14.449 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:14.449 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:51:14.450007+0000 2024-09-16T14:51:14.449 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:51:14.450007+0000 2024-09-16T14:51:14.449 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:14.449 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:14.449 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:14.449 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:14.449 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:14.449 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:14.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 1 Processor -- start 2024-09-16T14:51:14.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 1 -- start start 2024-09-16T14:51:14.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 10 monclient: init 2024-09-16T14:51:14.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:14.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:51:14.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 10 monclient(hunting): picked mon.noname-b con 0x7feb1c06c120 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:14.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 10 monclient(hunting): picked mon.noname-a con 0x7feb1c1353d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:14.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 10 monclient(hunting): picked mon.noname-c con 0x7feb1c0a1040 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:14.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1c06c500 con 0x7feb1c1353d0 2024-09-16T14:51:14.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1c0a2ef0 con 0x7feb1c0a1040 2024-09-16T14:51:14.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1c09e990 con 0x7feb1c06c120 2024-09-16T14:51:14.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:14.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 10 monclient(hunting): authenticate will time out at 1456.565721s 2024-09-16T14:51:14.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1bfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7feb1c1353d0 legacy=0x7feb1c139b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:14.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb2175e640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7feb1c0a1040 0x7feb1c13d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48282/0 (socket says 172.21.15.189:48282) 2024-09-16T14:51:14.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb2175e640 1 -- 172.21.15.189:0/3737413263 learned_addr learned my addr 172.21.15.189:0/3737413263 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:14.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 1 -- 172.21.15.189:0/3737413263 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098014234 0 0) 0x7feb1c0a2ef0 con 0x7feb1c0a1040 2024-09-16T14:51:14.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:14.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:14.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:14.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 10 monclient(hunting): my global_id is 184228 2024-09-16T14:51:14.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 1 -- 172.21.15.189:0/3737413263 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb04003530 con 0x7feb1c0a1040 2024-09-16T14:51:14.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 1 -- 172.21.15.189:0/3737413263 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 116325116 0 0) 0x7feb04003530 con 0x7feb1c0a1040 2024-09-16T14:51:14.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 1 -- 172.21.15.189:0/3737413263 >> v1:172.21.15.26:6790/0 conn(0x7feb1c06c120 legacy=0x7feb1c134cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:14.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 1 -- 172.21.15.189:0/3737413263 >> v1:172.21.15.26:6789/0 conn(0x7feb1c1353d0 legacy=0x7feb1c139b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:14.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:51:14.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 1 monclient: found mon.noname-c 2024-09-16T14:51:14.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:14.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:51:14.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 1 -- 172.21.15.189:0/3737413263 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1c09ec70 con 0x7feb1c0a1040 2024-09-16T14:51:14.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:51:14.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:51:14.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:14.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 1 -- 172.21.15.189:0/3737413263 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb140019a0 con 0x7feb1c0a1040 2024-09-16T14:51:14.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 5 monclient: authenticate success, global_id 184228 2024-09-16T14:51:14.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:14.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:14.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:14.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 10 monclient: dump: 2024-09-16T14:51:14.458 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:14.459 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:14.459 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:14.459 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:14.459 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:14.459 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:14.459 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:14.459 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:14.459 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:14.459 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:14.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:14.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 1 -- 172.21.15.189:0/3737413263 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb14001c50 con 0x7feb1c0a1040 2024-09-16T14:51:14.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:14.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 1 -- 172.21.15.189:0/3737413263 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb14002020 con 0x7feb1c0a1040 2024-09-16T14:51:14.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:14.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:14.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb1b7fe640 10 monclient: dump: 2024-09-16T14:51:14.460 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:14.460 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:14.460 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:14.460 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:14.460 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:14.460 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:14.460 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:14.460 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:14.460 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:14.461 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:14.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:14.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 4 set_mon_vals no callback set 2024-09-16T14:51:14.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 10 monclient: shutdown 2024-09-16T14:51:14.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:14.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 1 -- 172.21.15.189:0/3737413263 >> v1:172.21.15.189:6789/0 conn(0x7feb1c0a1040 legacy=0x7feb1c13d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:14.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 1 -- 172.21.15.189:0/3737413263 shutdown_connections 2024-09-16T14:51:14.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 1 -- 172.21.15.189:0/3737413263 >> 172.21.15.189:0/3737413263 conn(0x7feb1c0a3a40 msgr2=0x7feb1c0a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:14.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 1 -- 172.21.15.189:0/3737413263 shutdown_connections 2024-09-16T14:51:14.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 1 -- 172.21.15.189:0/3737413263 wait complete. 2024-09-16T14:51:14.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 10 monclient: build_initial_monmap 2024-09-16T14:51:14.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 10 monclient: monmap: 2024-09-16T14:51:14.466 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:14.466 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:14.466 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:51:14.466 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:51:14.466 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:14.466 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:51:14.467 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:14.467 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:14.467 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:14.467 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:14.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 1 Processor -- start 2024-09-16T14:51:14.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 1 -- start start 2024-09-16T14:51:14.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.447+0000 7feb231c1640 10 monclient: init 2024-09-16T14:51:14.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:14.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:51:14.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 10 monclient(hunting): picked mon.noname-a con 0x7feb1c06c120 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:14.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 10 monclient(hunting): picked mon.noname-b con 0x7feb1c0a1040 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:14.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 10 monclient(hunting): picked mon.noname-c con 0x7feb1c1353d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:14.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1c0a5460 con 0x7feb1c06c120 2024-09-16T14:51:14.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1c133620 con 0x7feb1c0a1040 2024-09-16T14:51:14.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1c132810 con 0x7feb1c1353d0 2024-09-16T14:51:14.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:14.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 10 monclient(hunting): authenticate will time out at 1456.568495s 2024-09-16T14:51:14.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb20f5d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7feb1c06c120 legacy=0x7feb1c132f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:14.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb1bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7feb1c0a1040 0x7feb1c1cd860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48286/0 (socket says 172.21.15.189:48286) 2024-09-16T14:51:14.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb1bfff640 1 -- 172.21.15.189:0/3431269422 learned_addr learned my addr 172.21.15.189:0/3431269422 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:14.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 1 -- 172.21.15.189:0/3431269422 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672787158 0 0) 0x7feb1c133620 con 0x7feb1c0a1040 2024-09-16T14:51:14.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 10 monclient(hunting): _negotiate 2024-09-16T14:51:14.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:14.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:14.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 10 monclient(hunting): my global_id is 184231 2024-09-16T14:51:14.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 1 -- 172.21.15.189:0/3431269422 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb08003530 con 0x7feb1c0a1040 2024-09-16T14:51:14.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 1 -- 172.21.15.189:0/3431269422 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2770403522 0 0) 0x7feb08003530 con 0x7feb1c0a1040 2024-09-16T14:51:14.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 1 -- 172.21.15.189:0/3431269422 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb1c133620 con 0x7feb1c0a1040 2024-09-16T14:51:14.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 1 -- 172.21.15.189:0/3431269422 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb0c004190 con 0x7feb1c0a1040 2024-09-16T14:51:14.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:14.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:14.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 10 monclient(hunting): dump: 2024-09-16T14:51:14.469 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:14.469 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:14.469 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:14.469 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:14.469 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:14.469 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:14.469 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:14.470 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:14.470 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:14.470 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:14.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:14.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 1 -- 172.21.15.189:0/3431269422 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2091571212 0 0) 0x7feb1c133620 con 0x7feb1c0a1040 2024-09-16T14:51:14.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 1 -- 172.21.15.189:0/3431269422 >> v1:172.21.15.26:6790/0 conn(0x7feb1c1353d0 legacy=0x7feb1c1d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:14.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 1 -- 172.21.15.189:0/3431269422 >> v1:172.21.15.26:6789/0 conn(0x7feb1c06c120 legacy=0x7feb1c132f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:14.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 10 monclient: _finish_hunting 0 2024-09-16T14:51:14.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 1 monclient: found mon.b 2024-09-16T14:51:14.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:14.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:14.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 1 -- 172.21.15.189:0/3431269422 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1c132af0 con 0x7feb1c0a1040 2024-09-16T14:51:14.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 10 monclient: _finish_auth 0 2024-09-16T14:51:14.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 10 monclient: _check_auth_tickets 2024-09-16T14:51:14.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:14.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 5 monclient: authenticate success, global_id 184231 2024-09-16T14:51:14.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 10 monclient: _renew_subs 2024-09-16T14:51:14.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:14.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 1 -- 172.21.15.189:0/3431269422 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb1c1315c0 con 0x7feb1c0a1040 2024-09-16T14:51:14.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 10 monclient: _renew_subs 2024-09-16T14:51:14.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:14.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 1 -- 172.21.15.189:0/3431269422 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb1c132250 con 0x7feb1c0a1040 2024-09-16T14:51:14.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 1 -- 172.21.15.189:0/3431269422 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb0c004360 con 0x7feb1c0a1040 2024-09-16T14:51:14.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:14.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 1 -- 172.21.15.189:0/3431269422 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb0c004550 con 0x7feb1c0a1040 2024-09-16T14:51:14.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:14.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb1affd640 4 set_mon_vals no callback set 2024-09-16T14:51:14.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:14.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 10 monclient: dump: 2024-09-16T14:51:14.472 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:14.472 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:14.472 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:14.472 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:14.472 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:14.472 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:14.472 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:14.472 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:14.472 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:14.472 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:14.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:14.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:14.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:14.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb231c1640 1 -- 172.21.15.189:0/3431269422 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feafc005050 con 0x7feb1c0a1040 2024-09-16T14:51:14.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 1 -- 172.21.15.189:0/3431269422 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7feb0c004770 con 0x7feb1c0a1040 2024-09-16T14:51:14.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb197fa640 1 -- 172.21.15.189:0/3431269422 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(252..252 src has 169..252) v4 ==== 6745+0+0 (unknown 3091906843 0 0) 0x7feb0c043c20 con 0x7feb1c0a1040 2024-09-16T14:51:14.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.451+0000 7feb20f5d640 10 monclient: get_auth_request con 0x7feb08036050 auth_method 0 2024-09-16T14:51:14.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.455+0000 7feb197fa640 1 -- 172.21.15.189:0/3431269422 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7feb0c004f90 con 0x7feb1c0a1040 2024-09-16T14:51:14.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.455+0000 7feb197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:14.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.455+0000 7feb197fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:14.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.611+0000 7feb231c1640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T14:51:14.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.611+0000 7feb231c1640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T14:51:14.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.611+0000 7feb231c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:14.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.611+0000 7feb231c1640 1 -- 172.21.15.189:0/3431269422 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7feafc005880 con 0x7feb1c0a1040 2024-09-16T14:51:14.614 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:14.613+0000 7faec309f640 -1 mon.c@2(leader).osd e252 definitely_dead 0 2024-09-16T14:51:14.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.811+0000 7feb197fa640 1 -- 172.21.15.189:0/3431269422 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 removed osd.12 v253) v1 ==== 83+0+0 (unknown 3002182018 0 0) 0x7feb0c012a30 con 0x7feb1c0a1040 2024-09-16T14:51:14.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.811+0000 7feb197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T14:51:14.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.811+0000 7feb197fa640 10 monclient: _finish_command 2 = system:0 removed osd.12 2024-09-16T14:51:14.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.915+0000 7feb231c1640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T14:51:14.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.915+0000 7feb231c1640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T14:51:14.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.915+0000 7feb231c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:14.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.915+0000 7feb231c1640 1 -- 172.21.15.189:0/3431269422 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7feafc005370 con 0x7feb1c0a1040 2024-09-16T14:51:14.917 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:14.917+0000 7faec309f640 -1 mon.c@2(leader).osd e253 definitely_dead 0 2024-09-16T14:51:14.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.915+0000 7feb197fa640 1 -- 172.21.15.189:0/3431269422 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v253) v1 ==== 92+0+0 (unknown 2474614984 0 0) 0x7feb0c0141b0 con 0x7feb1c0a1040 2024-09-16T14:51:14.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.915+0000 7feb197fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T14:51:14.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.915+0000 7feb197fa640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-16T14:51:14.918 INFO:tasks.workunit.client.0.smithi189.stderr:osd.12 does not exist. 2024-09-16T14:51:14.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.919+0000 7feb231c1640 1 -- 172.21.15.189:0/3431269422 >> v1:172.21.15.189:6812/3196161582 conn(0x7feb08036050 legacy=0x7feb08038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:14.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.919+0000 7feb231c1640 10 monclient: shutdown 2024-09-16T14:51:14.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.919+0000 7feb231c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:14.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.919+0000 7feb231c1640 1 -- 172.21.15.189:0/3431269422 >> v1:172.21.15.189:6789/0 conn(0x7feb1c0a1040 legacy=0x7feb1c1cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:14.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.919+0000 7feb231c1640 1 -- 172.21.15.189:0/3431269422 shutdown_connections 2024-09-16T14:51:14.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.919+0000 7feb231c1640 1 -- 172.21.15.189:0/3431269422 >> 172.21.15.189:0/3431269422 conn(0x7feb1c0a3a40 msgr2=0x7feb1c06b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:14.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.919+0000 7feb231c1640 1 -- 172.21.15.189:0/3431269422 shutdown_connections 2024-09-16T14:51:14.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:14.919+0000 7feb231c1640 1 -- 172.21.15.189:0/3431269422 wait complete. 2024-09-16T14:51:14.934 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd rm 12 2024-09-16T14:51:15.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 10 monclient: get_monmap_and_config 2024-09-16T14:51:15.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 10 monclient: build_initial_monmap 2024-09-16T14:51:15.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 10 monclient: monmap: 2024-09-16T14:51:15.041 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:15.041 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:15.041 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:51:15.042542+0000 2024-09-16T14:51:15.041 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:51:15.042542+0000 2024-09-16T14:51:15.041 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:15.041 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:15.041 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:15.041 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:15.041 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:15.041 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:15.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 1 Processor -- start 2024-09-16T14:51:15.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 1 -- start start 2024-09-16T14:51:15.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 10 monclient: init 2024-09-16T14:51:15.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:15.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:51:15.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 10 monclient(hunting): picked mon.noname-a con 0x7f2b281318a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:15.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 10 monclient(hunting): picked mon.noname-b con 0x7f2b28134d50 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:15.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 10 monclient(hunting): picked mon.noname-c con 0x7f2b2813b2b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:15.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2809d8a0 con 0x7f2b281318a0 2024-09-16T14:51:15.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2809ddf0 con 0x7f2b2813b2b0 2024-09-16T14:51:15.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2809e8b0 con 0x7f2b28134d50 2024-09-16T14:51:15.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:15.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 10 monclient(hunting): authenticate will time out at 1457.158506s 2024-09-16T14:51:15.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2d47b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2b2813b2b0 0x7f2b2813d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48290/0 (socket says 172.21.15.189:48290) 2024-09-16T14:51:15.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2cc7a640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2b281318a0 legacy=0x7f2b28131c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:15.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2d47b640 1 -- 172.21.15.189:0/3969592823 learned_addr learned my addr 172.21.15.189:0/3969592823 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:15.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 1 -- 172.21.15.189:0/3969592823 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811738069 0 0) 0x7f2b2809ddf0 con 0x7f2b2813b2b0 2024-09-16T14:51:15.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:15.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:15.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:15.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 10 monclient(hunting): my global_id is 184234 2024-09-16T14:51:15.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 1 -- 172.21.15.189:0/3969592823 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b14003530 con 0x7f2b2813b2b0 2024-09-16T14:51:15.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 1 -- 172.21.15.189:0/3969592823 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 115394937 0 0) 0x7f2b14003530 con 0x7f2b2813b2b0 2024-09-16T14:51:15.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 1 -- 172.21.15.189:0/3969592823 >> v1:172.21.15.26:6790/0 conn(0x7f2b28134d50 legacy=0x7f2b28139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:15.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 1 -- 172.21.15.189:0/3969592823 >> v1:172.21.15.26:6789/0 conn(0x7f2b281318a0 legacy=0x7f2b28131c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:15.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 10 monclient: _finish_hunting 0 2024-09-16T14:51:15.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 1 monclient: found mon.noname-c 2024-09-16T14:51:15.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:15.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:51:15.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 1 -- 172.21.15.189:0/3969592823 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2809eb90 con 0x7f2b2813b2b0 2024-09-16T14:51:15.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 10 monclient: _finish_auth 0 2024-09-16T14:51:15.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 10 monclient: _check_auth_tickets 2024-09-16T14:51:15.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:15.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 1 -- 172.21.15.189:0/3969592823 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b18001620 con 0x7f2b2813b2b0 2024-09-16T14:51:15.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 5 monclient: authenticate success, global_id 184234 2024-09-16T14:51:15.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:15.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:15.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:15.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 10 monclient: dump: 2024-09-16T14:51:15.047 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:15.047 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:15.047 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:15.047 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:15.047 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:15.047 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:15.047 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:15.047 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:15.047 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:15.047 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:15.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 1 -- 172.21.15.189:0/3969592823 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b180018d0 con 0x7f2b2813b2b0 2024-09-16T14:51:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 1 -- 172.21.15.189:0/3969592823 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b18001ca0 con 0x7f2b2813b2b0 2024-09-16T14:51:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b277fe640 10 monclient: dump: 2024-09-16T14:51:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:15.049 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:15.049 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:15.049 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:15.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:15.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 4 set_mon_vals no callback set 2024-09-16T14:51:15.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 10 monclient: shutdown 2024-09-16T14:51:15.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:15.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.039+0000 7f2b2eede640 1 -- 172.21.15.189:0/3969592823 >> v1:172.21.15.189:6789/0 conn(0x7f2b2813b2b0 legacy=0x7f2b2813d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:15.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 1 -- 172.21.15.189:0/3969592823 shutdown_connections 2024-09-16T14:51:15.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 1 -- 172.21.15.189:0/3969592823 >> 172.21.15.189:0/3969592823 conn(0x7f2b280a3320 msgr2=0x7f2b2809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:15.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 1 -- 172.21.15.189:0/3969592823 shutdown_connections 2024-09-16T14:51:15.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 1 -- 172.21.15.189:0/3969592823 wait complete. 2024-09-16T14:51:15.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 monclient: build_initial_monmap 2024-09-16T14:51:15.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 monclient: monmap: 2024-09-16T14:51:15.050 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:15.050 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:15.050 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:51:15.050 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:51:15.050 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:15.050 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:51:15.050 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:15.050 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:15.050 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:15.050 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:15.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 1 Processor -- start 2024-09-16T14:51:15.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 1 -- start start 2024-09-16T14:51:15.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 monclient: init 2024-09-16T14:51:15.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:15.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:51:15.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 monclient(hunting): picked mon.noname-a con 0x7f2b281318a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:15.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 monclient(hunting): picked mon.noname-c con 0x7f2b28134d50 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:15.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 monclient(hunting): picked mon.noname-b con 0x7f2b2813b2b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:15.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2813abd0 con 0x7f2b281318a0 2024-09-16T14:51:15.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b280a1d50 con 0x7f2b2813b2b0 2024-09-16T14:51:15.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2811a160 con 0x7f2b28134d50 2024-09-16T14:51:15.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:15.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 monclient(hunting): authenticate will time out at 1457.161530s 2024-09-16T14:51:15.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2cc7a640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2b281318a0 legacy=0x7f2b28119340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:15.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2d47b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2b2813b2b0 0x7f2b281d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48300/0 (socket says 172.21.15.189:48300) 2024-09-16T14:51:15.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2d47b640 1 -- 172.21.15.189:0/3113750970 learned_addr learned my addr 172.21.15.189:0/3113750970 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:15.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125378615 0 0) 0x7f2b2809eb90 con 0x7f2b2813b2b0 2024-09-16T14:51:15.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient(hunting): _negotiate 2024-09-16T14:51:15.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:15.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:15.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient(hunting): my global_id is 184237 2024-09-16T14:51:15.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b08003530 con 0x7f2b2813b2b0 2024-09-16T14:51:15.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2443809009 0 0) 0x7f2b08003530 con 0x7f2b2813b2b0 2024-09-16T14:51:15.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b2809eb90 con 0x7f2b2813b2b0 2024-09-16T14:51:15.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b180010d0 con 0x7f2b2813b2b0 2024-09-16T14:51:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient(hunting): dump: 2024-09-16T14:51:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:15.056 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685867491 0 0) 0x7f2b2811a160 con 0x7f2b28134d50 2024-09-16T14:51:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient(hunting): _negotiate 2024-09-16T14:51:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient(hunting): my global_id is 154946 2024-09-16T14:51:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b08003530 con 0x7f2b28134d50 2024-09-16T14:51:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1336402452 0 0) 0x7f2b2809eb90 con 0x7f2b2813b2b0 2024-09-16T14:51:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 >> v1:172.21.15.26:6790/0 conn(0x7f2b28134d50 legacy=0x7f2b28119a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 >> v1:172.21.15.26:6789/0 conn(0x7f2b281318a0 legacy=0x7f2b28119340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient: _finish_hunting 0 2024-09-16T14:51:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 1 monclient: found mon.b 2024-09-16T14:51:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b28139ed0 con 0x7f2b2813b2b0 2024-09-16T14:51:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient: _finish_auth 0 2024-09-16T14:51:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient: _check_auth_tickets 2024-09-16T14:51:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 5 monclient: authenticate success, global_id 184237 2024-09-16T14:51:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 monclient: _renew_subs 2024-09-16T14:51:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 1 -- 172.21.15.189:0/3113750970 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b2811b190 con 0x7f2b2813b2b0 2024-09-16T14:51:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 monclient: _renew_subs 2024-09-16T14:51:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 1 -- 172.21.15.189:0/3113750970 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2811bce0 con 0x7f2b2813b2b0 2024-09-16T14:51:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b18001cd0 con 0x7f2b2813b2b0 2024-09-16T14:51:15.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:15.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b18004da0 con 0x7f2b2813b2b0 2024-09-16T14:51:15.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:15.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:15.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b267fc640 4 set_mon_vals no callback set 2024-09-16T14:51:15.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 10 monclient: dump: 2024-09-16T14:51:15.059 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:15.059 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:15.059 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:15.059 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:15.059 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:15.059 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:15.059 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:15.060 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2b1800f810 con 0x7f2b2813b2b0 2024-09-16T14:51:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2eede640 1 -- 172.21.15.189:0/3113750970 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b0c005050 con 0x7f2b2813b2b0 2024-09-16T14:51:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(253..253 src has 169..253) v4 ==== 6745+0+0 (unknown 4244818690 0 0) 0x7f2b18043e30 con 0x7f2b2813b2b0 2024-09-16T14:51:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.043+0000 7f2b2cc7a640 10 monclient: get_auth_request con 0x7f2b08036180 auth_method 0 2024-09-16T14:51:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.047+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2b1806fdb0 con 0x7f2b2813b2b0 2024-09-16T14:51:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.047+0000 7f2b257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:15.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.047+0000 7f2b257fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:15.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.211+0000 7f2b2eede640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T14:51:15.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.211+0000 7f2b2eede640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T14:51:15.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.211+0000 7f2b2eede640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:15.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.211+0000 7f2b2eede640 1 -- 172.21.15.189:0/3113750970 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7f2b0c005880 con 0x7f2b2813b2b0 2024-09-16T14:51:15.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:15.213+0000 7faec309f640 -1 mon.c@2(leader).osd e253 definitely_dead 0 2024-09-16T14:51:15.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.211+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v253) v1 ==== 92+0+0 (unknown 2474614984 0 0) 0x7f2b180145a0 con 0x7f2b2813b2b0 2024-09-16T14:51:15.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.211+0000 7f2b257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T14:51:15.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.211+0000 7f2b257fa640 10 monclient: _finish_command 2 = system:0 osd.12 does not exist. 2024-09-16T14:51:15.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.307+0000 7f2b2eede640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T14:51:15.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.307+0000 7f2b2eede640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T14:51:15.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.307+0000 7f2b2eede640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:15.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.307+0000 7f2b2eede640 1 -- 172.21.15.189:0/3113750970 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7f2b0c005370 con 0x7f2b2813b2b0 2024-09-16T14:51:15.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:15.309+0000 7faec309f640 -1 mon.c@2(leader).osd e253 definitely_dead 0 2024-09-16T14:51:15.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.307+0000 7f2b257fa640 1 -- 172.21.15.189:0/3113750970 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v253) v1 ==== 92+0+0 (unknown 2474614984 0 0) 0x7f2b18012c40 con 0x7f2b2813b2b0 2024-09-16T14:51:15.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.307+0000 7f2b257fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T14:51:15.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.307+0000 7f2b257fa640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-16T14:51:15.311 INFO:tasks.workunit.client.0.smithi189.stderr:osd.12 does not exist. 2024-09-16T14:51:15.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.311+0000 7f2b2eede640 1 -- 172.21.15.189:0/3113750970 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2b08036180 legacy=0x7f2b08038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:15.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.311+0000 7f2b2eede640 10 monclient: shutdown 2024-09-16T14:51:15.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.311+0000 7f2b2eede640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:15.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.311+0000 7f2b2eede640 1 -- 172.21.15.189:0/3113750970 >> v1:172.21.15.189:6789/0 conn(0x7f2b2813b2b0 legacy=0x7f2b281d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:15.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.311+0000 7f2b2eede640 1 -- 172.21.15.189:0/3113750970 shutdown_connections 2024-09-16T14:51:15.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.311+0000 7f2b2eede640 1 -- 172.21.15.189:0/3113750970 >> 172.21.15.189:0/3113750970 conn(0x7f2b280a3320 msgr2=0x7f2b281341e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:15.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.311+0000 7f2b2eede640 1 -- 172.21.15.189:0/3113750970 shutdown_connections 2024-09-16T14:51:15.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.311+0000 7f2b2eede640 1 -- 172.21.15.189:0/3113750970 wait complete. 2024-09-16T14:51:15.326 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd rm 14 2024-09-16T14:51:15.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d81a77640 10 monclient: get_monmap_and_config 2024-09-16T14:51:15.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d81a77640 10 monclient: build_initial_monmap 2024-09-16T14:51:15.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d81a77640 10 monclient: monmap: 2024-09-16T14:51:15.441 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:15.441 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:15.442 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:51:15.443237+0000 2024-09-16T14:51:15.442 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:51:15.443237+0000 2024-09-16T14:51:15.442 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:15.442 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:15.442 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:15.442 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:15.442 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:15.442 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:15.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d81a77640 1 Processor -- start 2024-09-16T14:51:15.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d81a77640 1 -- start start 2024-09-16T14:51:15.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d81a77640 10 monclient: init 2024-09-16T14:51:15.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d81a77640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:15.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d81a77640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:51:15.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d81a77640 10 monclient(hunting): picked mon.noname-a con 0x7f5d7c0a1b00 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:15.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d81a77640 10 monclient(hunting): picked mon.noname-b con 0x7f5d7c0a25f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:15.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d81a77640 10 monclient(hunting): picked mon.noname-c con 0x7f5d7c13b430 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:15.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d81a77640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c0a30b0 con 0x7f5d7c0a1b00 2024-09-16T14:51:15.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d81a77640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c0a2e30 con 0x7f5d7c13b430 2024-09-16T14:51:15.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d81a77640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c09eb50 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d81a77640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:15.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d81a77640 10 monclient(hunting): authenticate will time out at 1457.559590s 2024-09-16T14:51:15.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d7b7fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f5d7c0a1b00 legacy=0x7f5d7c0a1ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:15.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d7bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5d7c13b430 0x7f5d7c13d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48306/0 (socket says 172.21.15.189:48306) 2024-09-16T14:51:15.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.439+0000 7f5d7bfff640 1 -- 172.21.15.189:0/3275883478 learned_addr learned my addr 172.21.15.189:0/3275883478 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:15.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 1 -- 172.21.15.189:0/3275883478 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181620659 0 0) 0x7f5d7c0a2e30 con 0x7f5d7c13b430 2024-09-16T14:51:15.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:15.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:15.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:15.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient(hunting): my global_id is 184240 2024-09-16T14:51:15.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 1 -- 172.21.15.189:0/3275883478 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d64003530 con 0x7f5d7c13b430 2024-09-16T14:51:15.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 1 -- 172.21.15.189:0/3275883478 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094643925 0 0) 0x7f5d7c09eb50 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:15.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:15.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:15.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient(hunting): my global_id is 154949 2024-09-16T14:51:15.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 1 -- 172.21.15.189:0/3275883478 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d7c0a2e30 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 1 -- 172.21.15.189:0/3275883478 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 657024318 0 0) 0x7f5d64003530 con 0x7f5d7c13b430 2024-09-16T14:51:15.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 1 -- 172.21.15.189:0/3275883478 >> v1:172.21.15.26:6790/0 conn(0x7f5d7c0a25f0 legacy=0x7f5d7c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:15.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 1 -- 172.21.15.189:0/3275883478 >> v1:172.21.15.26:6789/0 conn(0x7f5d7c0a1b00 legacy=0x7f5d7c0a1ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:15.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:51:15.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 1 monclient: found mon.noname-c 2024-09-16T14:51:15.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:15.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:51:15.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 1 -- 172.21.15.189:0/3275883478 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c09ee30 con 0x7f5d7c13b430 2024-09-16T14:51:15.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:51:15.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:51:15.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:15.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 1 -- 172.21.15.189:0/3275883478 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5d740019a0 con 0x7f5d7c13b430 2024-09-16T14:51:15.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:15.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:15.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient: dump: 2024-09-16T14:51:15.448 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:15.448 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:15.448 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:15.448 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:15.448 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:15.448 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:15.448 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:15.449 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:15.449 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:15.449 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:15.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 5 monclient: authenticate success, global_id 184240 2024-09-16T14:51:15.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:15.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 1 -- 172.21.15.189:0/3275883478 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d74001c80 con 0x7f5d7c13b430 2024-09-16T14:51:15.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:15.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 1 -- 172.21.15.189:0/3275883478 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5d74002010 con 0x7f5d7c13b430 2024-09-16T14:51:15.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:15.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 4 set_mon_vals no callback set 2024-09-16T14:51:15.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:15.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:15.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7a7fc640 10 monclient: dump: 2024-09-16T14:51:15.449 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:15.450 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:15.450 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:15.450 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:15.450 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:15.450 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:15.450 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:15.450 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:15.450 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:15.450 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:15.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 10 monclient: shutdown 2024-09-16T14:51:15.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:15.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 1 -- 172.21.15.189:0/3275883478 >> v1:172.21.15.189:6789/0 conn(0x7f5d7c13b430 legacy=0x7f5d7c13d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:15.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 1 -- 172.21.15.189:0/3275883478 shutdown_connections 2024-09-16T14:51:15.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 1 -- 172.21.15.189:0/3275883478 >> 172.21.15.189:0/3275883478 conn(0x7f5d7c0a7dd0 msgr2=0x7f5d7c0a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:15.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 1 -- 172.21.15.189:0/3275883478 shutdown_connections 2024-09-16T14:51:15.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 1 -- 172.21.15.189:0/3275883478 wait complete. 2024-09-16T14:51:15.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 10 monclient: build_initial_monmap 2024-09-16T14:51:15.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 10 monclient: monmap: 2024-09-16T14:51:15.454 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:15.454 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:15.455 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:51:15.455 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:51:15.455 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:15.455 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:51:15.455 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:15.455 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:15.455 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:15.455 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:15.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 1 Processor -- start 2024-09-16T14:51:15.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 1 -- start start 2024-09-16T14:51:15.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 10 monclient: init 2024-09-16T14:51:15.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:15.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:51:15.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 10 monclient(hunting): picked mon.noname-c con 0x7f5d7c0a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:15.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 10 monclient(hunting): picked mon.noname-b con 0x7f5d7c0a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:15.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 10 monclient(hunting): picked mon.noname-a con 0x7f5d7c13b430 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:15.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c0a5350 con 0x7f5d7c13b430 2024-09-16T14:51:15.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c11db20 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c0a5a50 con 0x7f5d7c0a1b00 2024-09-16T14:51:15.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:15.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 10 monclient(hunting): authenticate will time out at 1457.563213s 2024-09-16T14:51:15.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7affd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5d7c0a25f0 0x7f5d7c139f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48314/0 (socket says 172.21.15.189:48314) 2024-09-16T14:51:15.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7affd640 1 -- 172.21.15.189:0/2864623176 learned_addr learned my addr 172.21.15.189:0/2864623176 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:15.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d7bfff640 1 -- 172.21.15.189:0/2864623176 >> v1:172.21.15.26:6789/0 conn(0x7f5d7c13b430 legacy=0x7f5d7c1d1600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:15.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 1 -- 172.21.15.189:0/2864623176 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724448890 0 0) 0x7f5d7c11db20 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:15.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:15.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:15.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 10 monclient(hunting): my global_id is 184243 2024-09-16T14:51:15.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 1 -- 172.21.15.189:0/2864623176 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d68003530 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 1 -- 172.21.15.189:0/2864623176 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3652375810 0 0) 0x7f5d68003530 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 1 -- 172.21.15.189:0/2864623176 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d7c11db20 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 1 -- 172.21.15.189:0/2864623176 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5d70001920 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:15.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:15.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 10 monclient(hunting): dump: 2024-09-16T14:51:15.457 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:15.457 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:15.458 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:15.458 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:15.458 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:15.458 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:15.458 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:15.458 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:15.458 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:15.458 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:15.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:15.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 1 -- 172.21.15.189:0/2864623176 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1476442702 0 0) 0x7f5d7c11db20 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 1 -- 172.21.15.189:0/2864623176 >> v1:172.21.15.26:6790/0 conn(0x7f5d7c0a1b00 legacy=0x7f5d7c139850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:15.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 1 -- 172.21.15.189:0/2864623176 >> v1:172.21.15.26:6789/0 conn(0x7f5d7c13b430 legacy=0x7f5d7c1d1600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:15.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 10 monclient: _finish_hunting 0 2024-09-16T14:51:15.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 1 monclient: found mon.b 2024-09-16T14:51:15.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:15.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:15.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 1 -- 172.21.15.189:0/2864623176 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c11fe60 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 10 monclient: _finish_auth 0 2024-09-16T14:51:15.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 10 monclient: _check_auth_tickets 2024-09-16T14:51:15.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:15.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 5 monclient: authenticate success, global_id 184243 2024-09-16T14:51:15.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 10 monclient: _renew_subs 2024-09-16T14:51:15.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:15.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.443+0000 7f5d81a77640 1 -- 172.21.15.189:0/2864623176 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c11e450 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d81a77640 10 monclient: _renew_subs 2024-09-16T14:51:15.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d81a77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:15.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d81a77640 1 -- 172.21.15.189:0/2864623176 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c13a670 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d63fff640 1 -- 172.21.15.189:0/2864623176 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d70001cf0 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d63fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:15.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d63fff640 1 -- 172.21.15.189:0/2864623176 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5d70003a70 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:15.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:15.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d79ffb640 4 set_mon_vals no callback set 2024-09-16T14:51:15.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d63fff640 10 monclient: dump: 2024-09-16T14:51:15.461 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:15.461 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:15.461 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:15.461 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:15.461 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:15.461 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:15.461 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:15.461 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:15.461 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:15.461 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:15.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d81a77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:15.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d81a77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:15.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d81a77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:15.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d81a77640 1 -- 172.21.15.189:0/2864623176 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d5c005050 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d63fff640 1 -- 172.21.15.189:0/2864623176 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5d70003bd0 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d63fff640 1 -- 172.21.15.189:0/2864623176 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(253..253 src has 169..253) v4 ==== 6745+0+0 (unknown 4244818690 0 0) 0x7f5d70043e90 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.447+0000 7f5d7b7fe640 10 monclient: get_auth_request con 0x7f5d68036050 auth_method 0 2024-09-16T14:51:15.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.451+0000 7f5d63fff640 1 -- 172.21.15.189:0/2864623176 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5d700147d0 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.451+0000 7f5d63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:15.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.451+0000 7f5d63fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:15.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.607+0000 7f5d81a77640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T14:51:15.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.607+0000 7f5d81a77640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T14:51:15.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.607+0000 7f5d81a77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:15.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.611+0000 7f5d81a77640 1 -- 172.21.15.189:0/2864623176 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["14"]} v 0) v1 -- 0x7f5d5c005880 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.612 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:15.613+0000 7faec309f640 -1 mon.c@2(leader).osd e253 definitely_dead 0 2024-09-16T14:51:15.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.827+0000 7f5d63fff640 1 -- 172.21.15.189:0/2864623176 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["14"]}]=0 removed osd.14 v254) v1 ==== 83+0+0 (unknown 1609553255 0 0) 0x7f5d7001cdc0 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.827+0000 7f5d63fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T14:51:15.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.827+0000 7f5d63fff640 10 monclient: _finish_command 2 = system:0 removed osd.14 2024-09-16T14:51:15.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.923+0000 7f5d81a77640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T14:51:15.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.923+0000 7f5d81a77640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T14:51:15.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.923+0000 7f5d81a77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:15.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.923+0000 7f5d81a77640 1 -- 172.21.15.189:0/2864623176 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["14"]} v 0) v1 -- 0x7f5d5c005370 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.927 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:15.925+0000 7faec309f640 -1 mon.c@2(leader).osd e254 definitely_dead 0 2024-09-16T14:51:15.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.927+0000 7f5d63fff640 1 -- 172.21.15.189:0/2864623176 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["14"]}]=0 osd.14 does not exist. v254) v1 ==== 92+0+0 (unknown 3147415118 0 0) 0x7f5d70012c40 con 0x7f5d7c0a25f0 2024-09-16T14:51:15.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.927+0000 7f5d63fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T14:51:15.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.927+0000 7f5d63fff640 10 monclient: _finish_command 3 = system:0 osd.14 does not exist. 2024-09-16T14:51:15.928 INFO:tasks.workunit.client.0.smithi189.stderr:osd.14 does not exist. 2024-09-16T14:51:15.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.927+0000 7f5d81a77640 1 -- 172.21.15.189:0/2864623176 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5d68036050 legacy=0x7f5d68038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:15.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.927+0000 7f5d81a77640 10 monclient: shutdown 2024-09-16T14:51:15.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.927+0000 7f5d81a77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:15.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.927+0000 7f5d81a77640 1 -- 172.21.15.189:0/2864623176 >> v1:172.21.15.189:6789/0 conn(0x7f5d7c0a25f0 legacy=0x7f5d7c139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:15.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.927+0000 7f5d81a77640 1 -- 172.21.15.189:0/2864623176 shutdown_connections 2024-09-16T14:51:15.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.927+0000 7f5d81a77640 1 -- 172.21.15.189:0/2864623176 >> 172.21.15.189:0/2864623176 conn(0x7f5d7c0a7dd0 msgr2=0x7f5d7c09d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:15.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.927+0000 7f5d81a77640 1 -- 172.21.15.189:0/2864623176 shutdown_connections 2024-09-16T14:51:15.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:15.927+0000 7f5d81a77640 1 -- 172.21.15.189:0/2864623176 wait complete. 2024-09-16T14:51:15.944 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd rm 1000 2024-09-16T14:51:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 10 monclient: get_monmap_and_config 2024-09-16T14:51:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 10 monclient: build_initial_monmap 2024-09-16T14:51:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 10 monclient: monmap: 2024-09-16T14:51:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:51:16.051067+0000 2024-09-16T14:51:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:51:16.051067+0000 2024-09-16T14:51:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:16.050 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:16.050 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:16.050 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:16.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 1 Processor -- start 2024-09-16T14:51:16.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 1 -- start start 2024-09-16T14:51:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 10 monclient: init 2024-09-16T14:51:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:51:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 10 monclient(hunting): picked mon.noname-b con 0x7f13a40a2070 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 10 monclient(hunting): picked mon.noname-a con 0x7f13a4137b30 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 10 monclient(hunting): picked mon.noname-c con 0x7f13a413b640 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a409ec70 con 0x7f13a4137b30 2024-09-16T14:51:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a409ea00 con 0x7f13a413b640 2024-09-16T14:51:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a40a2ef0 con 0x7f13a40a2070 2024-09-16T14:51:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 10 monclient(hunting): authenticate will time out at 1458.166993s 2024-09-16T14:51:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a37fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f13a413b640 0x7f13a413da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48318/0 (socket says 172.21.15.189:48318) 2024-09-16T14:51:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a37fe640 1 -- 172.21.15.189:0/1987247058 learned_addr learned my addr 172.21.15.189:0/1987247058 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a27fc640 1 -- 172.21.15.189:0/1987247058 >> v1:172.21.15.26:6789/0 conn(0x7f13a4137b30 legacy=0x7f13a4139f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 1 -- 172.21.15.189:0/1987247058 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130901741 0 0) 0x7f13a409ea00 con 0x7f13a413b640 2024-09-16T14:51:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 10 monclient(hunting): my global_id is 184246 2024-09-16T14:51:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 1 -- 172.21.15.189:0/1987247058 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f138c003530 con 0x7f13a413b640 2024-09-16T14:51:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 1 -- 172.21.15.189:0/1987247058 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2345617537 0 0) 0x7f138c003530 con 0x7f13a413b640 2024-09-16T14:51:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 1 -- 172.21.15.189:0/1987247058 >> v1:172.21.15.26:6790/0 conn(0x7f13a40a2070 legacy=0x7f13a40a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 1 -- 172.21.15.189:0/1987247058 >> v1:172.21.15.26:6789/0 conn(0x7f13a4137b30 legacy=0x7f13a4139f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:51:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 1 monclient: found mon.noname-c 2024-09-16T14:51:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:51:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 1 -- 172.21.15.189:0/1987247058 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a409ef50 con 0x7f13a413b640 2024-09-16T14:51:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:51:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:51:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 5 monclient: authenticate success, global_id 184246 2024-09-16T14:51:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 1 -- 172.21.15.189:0/1987247058 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f139c0019a0 con 0x7f13a413b640 2024-09-16T14:51:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 10 monclient: dump: 2024-09-16T14:51:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:16.060 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 1 -- 172.21.15.189:0/1987247058 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f139c001cc0 con 0x7f13a413b640 2024-09-16T14:51:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 1 -- 172.21.15.189:0/1987247058 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f139c002050 con 0x7f13a413b640 2024-09-16T14:51:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a9696640 4 set_mon_vals no callback set 2024-09-16T14:51:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.047+0000 7f13a1ffb640 10 monclient: dump: 2024-09-16T14:51:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:16.062 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient: shutdown 2024-09-16T14:51:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 1 -- 172.21.15.189:0/1987247058 >> v1:172.21.15.189:6789/0 conn(0x7f13a413b640 legacy=0x7f13a413da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 1 -- 172.21.15.189:0/1987247058 shutdown_connections 2024-09-16T14:51:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 1 -- 172.21.15.189:0/1987247058 >> 172.21.15.189:0/1987247058 conn(0x7f13a40a3a40 msgr2=0x7f13a40a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 1 -- 172.21.15.189:0/1987247058 shutdown_connections 2024-09-16T14:51:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 1 -- 172.21.15.189:0/1987247058 wait complete. 2024-09-16T14:51:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient: build_initial_monmap 2024-09-16T14:51:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient: monmap: 2024-09-16T14:51:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:16.063 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:16.063 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:51:16.063 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:51:16.063 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:16.063 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:51:16.063 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:16.063 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:16.063 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:16.063 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:16.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 1 Processor -- start 2024-09-16T14:51:16.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 1 -- start start 2024-09-16T14:51:16.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient: init 2024-09-16T14:51:16.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:51:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient(hunting): picked mon.noname-b con 0x7f13a40a2070 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient(hunting): picked mon.noname-a con 0x7f13a4137b30 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient(hunting): picked mon.noname-c con 0x7f13a413b640 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a411ee40 con 0x7f13a4137b30 2024-09-16T14:51:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a409e130 con 0x7f13a40a2070 2024-09-16T14:51:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a413afd0 con 0x7f13a413b640 2024-09-16T14:51:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient(hunting): authenticate will time out at 1458.169924s 2024-09-16T14:51:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a2ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f13a40a2070 0x7f13a409b630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48320/0 (socket says 172.21.15.189:48320) 2024-09-16T14:51:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a2ffd640 1 -- 172.21.15.189:0/2592317818 learned_addr learned my addr 172.21.15.189:0/2592317818 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a27fc640 1 -- 172.21.15.189:0/2592317818 >> v1:172.21.15.26:6789/0 conn(0x7f13a4137b30 legacy=0x7f13a409bd40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 1 -- 172.21.15.189:0/2592317818 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185649114 0 0) 0x7f13a409e130 con 0x7f13a40a2070 2024-09-16T14:51:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 10 monclient(hunting): my global_id is 184249 2024-09-16T14:51:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 1 -- 172.21.15.189:0/2592317818 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1390003560 con 0x7f13a40a2070 2024-09-16T14:51:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 1 -- 172.21.15.189:0/2592317818 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3529585136 0 0) 0x7f1390003560 con 0x7f13a40a2070 2024-09-16T14:51:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 1 -- 172.21.15.189:0/2592317818 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13a409e130 con 0x7f13a40a2070 2024-09-16T14:51:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 1 -- 172.21.15.189:0/2592317818 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f13940019c0 con 0x7f13a40a2070 2024-09-16T14:51:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 10 monclient(hunting): dump: 2024-09-16T14:51:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:16.066 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 1 -- 172.21.15.189:0/2592317818 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3071980031 0 0) 0x7f13a409e130 con 0x7f13a40a2070 2024-09-16T14:51:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 1 -- 172.21.15.189:0/2592317818 >> v1:172.21.15.26:6790/0 conn(0x7f13a413b640 legacy=0x7f13a41d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 1 -- 172.21.15.189:0/2592317818 >> v1:172.21.15.26:6789/0 conn(0x7f13a4137b30 legacy=0x7f13a409bd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:51:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 1 monclient: found mon.b 2024-09-16T14:51:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 1 -- 172.21.15.189:0/2592317818 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a4135400 con 0x7f13a40a2070 2024-09-16T14:51:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:51:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:51:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 5 monclient: authenticate success, global_id 184249 2024-09-16T14:51:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient: _renew_subs 2024-09-16T14:51:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 1 -- 172.21.15.189:0/2592317818 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a411e5a0 con 0x7f13a40a2070 2024-09-16T14:51:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient: _renew_subs 2024-09-16T14:51:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 1 -- 172.21.15.189:0/2592317818 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a4134790 con 0x7f13a40a2070 2024-09-16T14:51:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 1 -- 172.21.15.189:0/2592317818 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1394001d40 con 0x7f13a40a2070 2024-09-16T14:51:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 1 -- 172.21.15.189:0/2592317818 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1394003a70 con 0x7f13a40a2070 2024-09-16T14:51:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 10 monclient: dump: 2024-09-16T14:51:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:16.069 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a17fa640 4 set_mon_vals no callback set 2024-09-16T14:51:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 1 -- 172.21.15.189:0/2592317818 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1394003bd0 con 0x7f13a40a2070 2024-09-16T14:51:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a9696640 1 -- 172.21.15.189:0/2592317818 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1384005050 con 0x7f13a40a2070 2024-09-16T14:51:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f138b7fe640 1 -- 172.21.15.189:0/2592317818 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(254..254 src has 169..254) v4 ==== 6745+0+0 (unknown 853202217 0 0) 0x7f1394043ef0 con 0x7f13a40a2070 2024-09-16T14:51:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.051+0000 7f13a27fc640 10 monclient: get_auth_request con 0x7f1390036080 auth_method 0 2024-09-16T14:51:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.055+0000 7f138b7fe640 1 -- 172.21.15.189:0/2592317818 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1394014830 con 0x7f13a40a2070 2024-09-16T14:51:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.055+0000 7f138b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.055+0000 7f138b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:16.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.227+0000 7f13a9696640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T14:51:16.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.227+0000 7f13a9696640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T14:51:16.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.227+0000 7f13a9696640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:16.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.227+0000 7f13a9696640 1 -- 172.21.15.189:0/2592317818 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["1000"]} v 0) v1 -- 0x7f1384005880 con 0x7f13a40a2070 2024-09-16T14:51:16.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:16.229+0000 7faec309f640 -1 mon.c@2(leader).osd e254 definitely_dead 0 2024-09-16T14:51:16.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.227+0000 7f138b7fe640 1 -- 172.21.15.189:0/2592317818 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["1000"]}]=0 osd.1000 does not exist. v254) v1 ==== 96+0+0 (unknown 3172144995 0 0) 0x7f1394012ca0 con 0x7f13a40a2070 2024-09-16T14:51:16.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.227+0000 7f138b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T14:51:16.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.227+0000 7f138b7fe640 10 monclient: _finish_command 2 = system:0 osd.1000 does not exist. 2024-09-16T14:51:16.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.327+0000 7f13a9696640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T14:51:16.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.327+0000 7f13a9696640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T14:51:16.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.327+0000 7f13a9696640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:16.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.327+0000 7f13a9696640 1 -- 172.21.15.189:0/2592317818 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["1000"]} v 0) v1 -- 0x7f1384005370 con 0x7f13a40a2070 2024-09-16T14:51:16.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:16.329+0000 7faec309f640 -1 mon.c@2(leader).osd e255 definitely_dead 0 2024-09-16T14:51:16.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.327+0000 7f138b7fe640 1 -- 172.21.15.189:0/2592317818 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["1000"]}]=0 osd.1000 does not exist. v255) v1 ==== 96+0+0 (unknown 2039240708 0 0) 0x7f1394012ca0 con 0x7f13a40a2070 2024-09-16T14:51:16.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.327+0000 7f138b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T14:51:16.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.327+0000 7f138b7fe640 10 monclient: _finish_command 3 = system:0 osd.1000 does not exist. 2024-09-16T14:51:16.331 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1000 does not exist. 2024-09-16T14:51:16.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.331+0000 7f13a9696640 1 -- 172.21.15.189:0/2592317818 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1390036080 legacy=0x7f1390038540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:16.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.331+0000 7f13a9696640 10 monclient: shutdown 2024-09-16T14:51:16.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.331+0000 7f13a9696640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:16.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.331+0000 7f13a9696640 1 -- 172.21.15.189:0/2592317818 >> v1:172.21.15.189:6789/0 conn(0x7f13a40a2070 legacy=0x7f13a409b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:16.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.331+0000 7f13a9696640 1 -- 172.21.15.189:0/2592317818 shutdown_connections 2024-09-16T14:51:16.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.331+0000 7f13a9696640 1 -- 172.21.15.189:0/2592317818 >> 172.21.15.189:0/2592317818 conn(0x7f13a40a3a40 msgr2=0x7f13a409f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:16.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.331+0000 7f13a9696640 1 -- 172.21.15.189:0/2592317818 shutdown_connections 2024-09-16T14:51:16.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.331+0000 7f13a9696640 1 -- 172.21.15.189:0/2592317818 wait complete. 2024-09-16T14:51:16.347 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd create 0689a78d-9a3c-48ff-a75a-01eedf6472fa 2024-09-16T14:51:16.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 10 monclient: get_monmap_and_config 2024-09-16T14:51:16.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 10 monclient: build_initial_monmap 2024-09-16T14:51:16.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 10 monclient: monmap: 2024-09-16T14:51:16.453 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:16.453 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:16.453 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:51:16.454630+0000 2024-09-16T14:51:16.453 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:51:16.454630+0000 2024-09-16T14:51:16.453 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:16.453 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:16.453 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:16.453 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:16.453 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:16.453 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:16.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 1 Processor -- start 2024-09-16T14:51:16.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 1 -- start start 2024-09-16T14:51:16.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 10 monclient: init 2024-09-16T14:51:16.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:16.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:51:16.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 10 monclient(hunting): picked mon.noname-b con 0x7f75740a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:16.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 10 monclient(hunting): picked mon.noname-c con 0x7f75740a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:16.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 10 monclient(hunting): picked mon.noname-a con 0x7f757413b430 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:16.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75740a30b0 con 0x7f757413b430 2024-09-16T14:51:16.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75740a2e30 con 0x7f75740a25f0 2024-09-16T14:51:16.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757409eb50 con 0x7f75740a1b00 2024-09-16T14:51:16.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:16.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 10 monclient(hunting): authenticate will time out at 1458.571119s 2024-09-16T14:51:16.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f756bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f75740a25f0 0x7f7574139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48328/0 (socket says 172.21.15.189:48328) 2024-09-16T14:51:16.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f756bfff640 1 -- 172.21.15.189:0/3696705831 learned_addr learned my addr 172.21.15.189:0/3696705831 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:16.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7573fff640 1 -- 172.21.15.189:0/3696705831 >> v1:172.21.15.26:6789/0 conn(0x7f757413b430 legacy=0x7f757413d820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:16.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 1 -- 172.21.15.189:0/3696705831 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 160233059 0 0) 0x7f75740a2e30 con 0x7f75740a25f0 2024-09-16T14:51:16.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:51:16.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:16.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:16.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 10 monclient(hunting): my global_id is 184252 2024-09-16T14:51:16.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 1 -- 172.21.15.189:0/3696705831 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7560003530 con 0x7f75740a25f0 2024-09-16T14:51:16.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 1 -- 172.21.15.189:0/3696705831 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4099676812 0 0) 0x7f7560003530 con 0x7f75740a25f0 2024-09-16T14:51:16.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 1 -- 172.21.15.189:0/3696705831 >> v1:172.21.15.26:6790/0 conn(0x7f75740a1b00 legacy=0x7f75740a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:16.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 1 -- 172.21.15.189:0/3696705831 >> v1:172.21.15.26:6789/0 conn(0x7f757413b430 legacy=0x7f757413d820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:16.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:51:16.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 1 monclient: found mon.noname-c 2024-09-16T14:51:16.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:16.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:51:16.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 1 -- 172.21.15.189:0/3696705831 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757409ee30 con 0x7f75740a25f0 2024-09-16T14:51:16.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 10 monclient: _finish_auth 0 2024-09-16T14:51:16.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:51:16.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:16.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 5 monclient: authenticate success, global_id 184252 2024-09-16T14:51:16.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 1 -- 172.21.15.189:0/3696705831 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f755c001620 con 0x7f75740a25f0 2024-09-16T14:51:16.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:16.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:16.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 10 monclient: dump: 2024-09-16T14:51:16.462 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:16.462 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:16.462 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:16.462 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:16.462 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:16.462 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:16.463 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:16.463 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:16.463 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:16.463 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:16.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:16.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 1 -- 172.21.15.189:0/3696705831 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f755c001940 con 0x7f75740a25f0 2024-09-16T14:51:16.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:16.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 1 -- 172.21.15.189:0/3696705831 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f755c001d10 con 0x7f75740a25f0 2024-09-16T14:51:16.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:16.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7579b53640 4 set_mon_vals no callback set 2024-09-16T14:51:16.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:16.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:16.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.451+0000 7f7572ffd640 10 monclient: dump: 2024-09-16T14:51:16.464 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:16.464 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:16.464 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:16.464 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:16.464 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:16.464 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:16.464 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:16.464 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:16.464 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:16.464 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:16.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient: shutdown 2024-09-16T14:51:16.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:16.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 1 -- 172.21.15.189:0/3696705831 >> v1:172.21.15.189:6789/0 conn(0x7f75740a25f0 legacy=0x7f7574139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:16.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 1 -- 172.21.15.189:0/3696705831 shutdown_connections 2024-09-16T14:51:16.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 1 -- 172.21.15.189:0/3696705831 >> 172.21.15.189:0/3696705831 conn(0x7f75740a7dd0 msgr2=0x7f75740a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:16.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 1 -- 172.21.15.189:0/3696705831 shutdown_connections 2024-09-16T14:51:16.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 1 -- 172.21.15.189:0/3696705831 wait complete. 2024-09-16T14:51:16.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient: build_initial_monmap 2024-09-16T14:51:16.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient: monmap: 2024-09-16T14:51:16.465 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:16.465 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:16.465 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:51:16.465 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:51:16.465 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:16.466 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:51:16.466 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:16.466 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:16.466 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:16.466 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:16.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 1 Processor -- start 2024-09-16T14:51:16.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 1 -- start start 2024-09-16T14:51:16.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient: init 2024-09-16T14:51:16.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:16.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:51:16.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient(hunting): picked mon.noname-c con 0x7f75740a1b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:16.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient(hunting): picked mon.noname-b con 0x7f75740a25f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:16.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient(hunting): picked mon.noname-a con 0x7f757413b430 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:16.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757413a490 con 0x7f757413b430 2024-09-16T14:51:16.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757411fda0 con 0x7f75740a25f0 2024-09-16T14:51:16.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757411e570 con 0x7f75740a1b00 2024-09-16T14:51:16.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:16.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient(hunting): authenticate will time out at 1458.573807s 2024-09-16T14:51:16.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f756bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f75740a25f0 0x7f757411de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:48344/0 (socket says 172.21.15.189:48344) 2024-09-16T14:51:16.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f756bfff640 1 -- 172.21.15.189:0/1533562018 learned_addr learned my addr 172.21.15.189:0/1533562018 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:16.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7573fff640 1 -- 172.21.15.189:0/1533562018 >> v1:172.21.15.26:6789/0 conn(0x7f757413b430 legacy=0x7f75741d1810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:51:16.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 1 -- 172.21.15.189:0/1533562018 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806237433 0 0) 0x7f757411fda0 con 0x7f75740a25f0 2024-09-16T14:51:16.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:51:16.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:16.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:16.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 10 monclient(hunting): my global_id is 184255 2024-09-16T14:51:16.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 1 -- 172.21.15.189:0/1533562018 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7554003530 con 0x7f75740a25f0 2024-09-16T14:51:16.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 1 -- 172.21.15.189:0/1533562018 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2211509900 0 0) 0x7f7554003530 con 0x7f75740a25f0 2024-09-16T14:51:16.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 1 -- 172.21.15.189:0/1533562018 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f757411fda0 con 0x7f75740a25f0 2024-09-16T14:51:16.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 1 -- 172.21.15.189:0/1533562018 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f755c0015e0 con 0x7f75740a25f0 2024-09-16T14:51:16.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:16.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:16.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 10 monclient(hunting): dump: 2024-09-16T14:51:16.468 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:16.468 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:16.468 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:16.468 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:16.468 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:16.469 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:16.469 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:16.469 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:16.469 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:16.469 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:16.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:16.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 1 -- 172.21.15.189:0/1533562018 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 697297638 0 0) 0x7f757411fda0 con 0x7f75740a25f0 2024-09-16T14:51:16.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 1 -- 172.21.15.189:0/1533562018 >> v1:172.21.15.26:6790/0 conn(0x7f75740a1b00 legacy=0x7f757411d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:16.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 1 -- 172.21.15.189:0/1533562018 >> v1:172.21.15.26:6789/0 conn(0x7f757413b430 legacy=0x7f75741d1810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:16.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:51:16.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 1 monclient: found mon.b 2024-09-16T14:51:16.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:16.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:16.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 1 -- 172.21.15.189:0/1533562018 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75740a5ac0 con 0x7f75740a25f0 2024-09-16T14:51:16.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 10 monclient: _finish_auth 0 2024-09-16T14:51:16.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:51:16.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:16.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 5 monclient: authenticate success, global_id 184255 2024-09-16T14:51:16.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient: _renew_subs 2024-09-16T14:51:16.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:16.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 1 -- 172.21.15.189:0/1533562018 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f757413af40 con 0x7f75740a25f0 2024-09-16T14:51:16.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient: _renew_subs 2024-09-16T14:51:16.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:16.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 1 -- 172.21.15.189:0/1533562018 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f757411fac0 con 0x7f75740a25f0 2024-09-16T14:51:16.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 1 -- 172.21.15.189:0/1533562018 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f755c0011d0 con 0x7f75740a25f0 2024-09-16T14:51:16.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:16.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 1 -- 172.21.15.189:0/1533562018 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f755c004e20 con 0x7f75740a25f0 2024-09-16T14:51:16.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:16.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7571ffb640 4 set_mon_vals no callback set 2024-09-16T14:51:16.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:16.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 10 monclient: dump: 2024-09-16T14:51:16.471 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:16.471 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:16.471 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:16.471 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:16.471 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:16.471 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:16.471 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:16.471 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:16.471 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:16.472 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:16.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:16.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:16.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:16.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 1 -- 172.21.15.189:0/1533562018 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f755c004f80 con 0x7f75740a25f0 2024-09-16T14:51:16.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7579b53640 1 -- 172.21.15.189:0/1533562018 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7558005050 con 0x7f75740a25f0 2024-09-16T14:51:16.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f7570ff9640 1 -- 172.21.15.189:0/1533562018 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(255..255 src has 169..255) v4 ==== 6745+0+0 (unknown 2006114096 0 0) 0x7f755c043fc0 con 0x7f75740a25f0 2024-09-16T14:51:16.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.455+0000 7f75737fe640 10 monclient: get_auth_request con 0x7f7554036050 auth_method 0 2024-09-16T14:51:16.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.459+0000 7f7570ff9640 1 -- 172.21.15.189:0/1533562018 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f755c012e70 con 0x7f75740a25f0 2024-09-16T14:51:16.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.459+0000 7f7570ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:16.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.459+0000 7f7570ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:16.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.631+0000 7f7579b53640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:16.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.631+0000 7f7579b53640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:16.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.631+0000 7f7579b53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:16.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:16.631+0000 7f7579b53640 1 -- 172.21.15.189:0/1533562018 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"} v 0) v1 -- 0x7f7558005880 con 0x7f75740a25f0 2024-09-16T14:51:17.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.307+0000 7f7570ff9640 1 -- 172.21.15.189:0/1533562018 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}]=0 v256) v1 ==== 106+0+3 (unknown 3920327900 0 944643765) 0x7f755c012e70 con 0x7f75740a25f0 2024-09-16T14:51:17.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.307+0000 7f7570ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:17.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.307+0000 7f7570ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:17.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.411+0000 7f7579b53640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:17.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.411+0000 7f7579b53640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:17.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.411+0000 7f7579b53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:17.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.411+0000 7f7579b53640 1 -- 172.21.15.189:0/1533562018 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"} v 0) v1 -- 0x7f7558005300 con 0x7f75740a25f0 2024-09-16T14:51:17.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.411+0000 7f7570ff9640 1 -- 172.21.15.189:0/1533562018 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}]=0 v256) v1 ==== 106+0+3 (unknown 3920327900 0 944643765) 0x7f755c015330 con 0x7f75740a25f0 2024-09-16T14:51:17.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.411+0000 7f7570ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "0689a78d-9a3c-48ff-a75a-01eedf6472fa"}] 2024-09-16T14:51:17.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.411+0000 7f7570ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:51:17.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.415+0000 7f7579b53640 1 -- 172.21.15.189:0/1533562018 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7554036050 legacy=0x7f7554038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:17.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.415+0000 7f7579b53640 10 monclient: shutdown 2024-09-16T14:51:17.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.415+0000 7f7579b53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:17.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.415+0000 7f7579b53640 1 -- 172.21.15.189:0/1533562018 >> v1:172.21.15.189:6789/0 conn(0x7f75740a25f0 legacy=0x7f757411de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:17.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.415+0000 7f7579b53640 1 -- 172.21.15.189:0/1533562018 shutdown_connections 2024-09-16T14:51:17.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.415+0000 7f7579b53640 1 -- 172.21.15.189:0/1533562018 >> 172.21.15.189:0/1533562018 conn(0x7f75740a7dd0 msgr2=0x7f757406b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:17.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.415+0000 7f7579b53640 1 -- 172.21.15.189:0/1533562018 shutdown_connections 2024-09-16T14:51:17.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:17.415+0000 7f7579b53640 1 -- 172.21.15.189:0/1533562018 wait complete. 2024-09-16T14:51:17.432 INFO:tasks.workunit.client.0.smithi189.stderr:+ na2=12 2024-09-16T14:51:17.432 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo OK 2024-09-16T14:51:17.433 INFO:tasks.workunit.client.0.smithi189.stdout:OK 2024-09-16T14:51:17.433 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T14:51:17.433 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T14:51:17.488 INFO:tasks.workunit:Running workunits matching mon/caps.sh on client.0... 2024-09-16T14:51:17.489 INFO:tasks.workunit:Running workunit mon/caps.sh... 2024-09-16T14:51:17.490 DEBUG:teuthology.orchestra.run.smithi189:workunit test mon/caps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh 2024-09-16T14:51:17.541 INFO:tasks.workunit.client.0.smithi189.stderr:+ tmp=/tmp/cephtest-mon-caps-madness 2024-09-16T14:51:17.541 INFO:tasks.workunit.client.0.smithi189.stderr:+ exit_on_error=1 2024-09-16T14:51:17.541 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ ! -z '' ]] 2024-09-16T14:51:17.541 INFO:tasks.workunit.client.0.smithi189.stderr:++ uname 2024-09-16T14:51:17.542 INFO:tasks.workunit.client.0.smithi189.stderr:+ '[' Linux = FreeBSD ']' 2024-09-16T14:51:17.543 INFO:tasks.workunit.client.0.smithi189.stderr:+ ETIMEDOUT=110 2024-09-16T14:51:17.543 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring' 0 2024-09-16T14:51:17.543 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring' 2024-09-16T14:51:17.543 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=0 2024-09-16T14:51:17.543 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph auth get-or-create client.bazar '>' /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-16T14:51:17.543 INFO:tasks.workunit.client.0.smithi189.stdout:ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-16T14:51:17.544 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph auth get-or-create client.bazar '>' /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-16T14:51:18.037 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=0 2024-09-16T14:51:18.037 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T14:51:18.038 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:51:18.038 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status' 13 2024-09-16T14:51:18.039 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status' 2024-09-16T14:51:18.039 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T14:51:18.039 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-16T14:51:18.039 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-16T14:51:18.039 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-16T14:51:28.805 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:51:28.805 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T14:51:28.805 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T14:51:28.805 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T14:51:28.808 INFO:tasks.ceph.mon.a:Started 2024-09-16T14:51:28.808 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:51:28.809 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:51:28.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.933+0000 7f267fe88640 10 monclient: get_monmap_and_config 2024-09-16T14:51:28.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.933+0000 7f267fe88640 10 monclient: build_initial_monmap 2024-09-16T14:51:28.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.933+0000 7f267fe88640 10 monclient: monmap: 2024-09-16T14:51:28.936 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:28.936 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:28.936 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:28.937424+0000 2024-09-16T14:51:28.936 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:28.937424+0000 2024-09-16T14:51:28.936 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:28.936 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:28.937 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:28.937 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:28.937 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:28.937 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:28.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 1 Processor -- start 2024-09-16T14:51:28.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 1 -- start start 2024-09-16T14:51:28.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 10 monclient: init 2024-09-16T14:51:28.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:28.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:51:28.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 10 monclient(hunting): picked mon.noname-b con 0x7f267806b090 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:28.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 10 monclient(hunting): picked mon.noname-c con 0x7f267806bb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:28.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 10 monclient(hunting): picked mon.noname-a con 0x7f267813b760 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:28.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267806d750 con 0x7f267813b760 2024-09-16T14:51:28.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267806ade0 con 0x7f267806bb80 2024-09-16T14:51:28.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26780736a0 con 0x7f267806b090 2024-09-16T14:51:28.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:28.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 10 monclient(hunting): authenticate will time out at 1513.057143s 2024-09-16T14:51:28.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267dc24640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f267806b090 0x7f267806b470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59698/0 (socket says 172.21.15.26:59698) 2024-09-16T14:51:28.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267e425640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f267813b760 0x7f267813db50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56226/0 (socket says 172.21.15.26:56226) 2024-09-16T14:51:28.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267dc24640 1 -- 172.21.15.26:0/2505289949 learned_addr learned my addr 172.21.15.26:0/2505289949 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:28.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 1 -- 172.21.15.26:0/2505289949 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114380983 0 0) 0x7f267806ade0 con 0x7f267806bb80 2024-09-16T14:51:28.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 10 monclient(hunting): _negotiate 2024-09-16T14:51:28.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:28.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:28.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 10 monclient(hunting): my global_id is 184276 2024-09-16T14:51:28.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 1 -- 172.21.15.26:0/2505289949 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2660003530 con 0x7f267806bb80 2024-09-16T14:51:28.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 1 -- 172.21.15.26:0/2505289949 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1572917746 0 0) 0x7f2660003530 con 0x7f267806bb80 2024-09-16T14:51:28.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 1 -- 172.21.15.26:0/2505289949 >> v1:172.21.15.26:6790/0 conn(0x7f267806b090 legacy=0x7f267806b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:28.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 1 -- 172.21.15.26:0/2505289949 >> v1:172.21.15.26:6789/0 conn(0x7f267813b760 legacy=0x7f267813db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:28.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 10 monclient: _finish_hunting 0 2024-09-16T14:51:28.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 1 monclient: found mon.noname-c 2024-09-16T14:51:28.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:28.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:51:28.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 1 -- 172.21.15.26:0/2505289949 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267806c490 con 0x7f267806bb80 2024-09-16T14:51:28.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 10 monclient: _finish_auth 0 2024-09-16T14:51:28.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 10 monclient: _check_auth_tickets 2024-09-16T14:51:28.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:28.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 1 -- 172.21.15.26:0/2505289949 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f266c001620 con 0x7f267806bb80 2024-09-16T14:51:28.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:28.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:28.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 10 monclient: dump: 2024-09-16T14:51:28.943 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:28.943 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:28.943 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:28.943 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:28.943 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:28.943 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:28.943 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:28.943 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:28.943 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:28.943 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:28.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 5 monclient: authenticate success, global_id 184276 2024-09-16T14:51:28.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:28.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 1 -- 172.21.15.26:0/2505289949 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f266c001940 con 0x7f267806bb80 2024-09-16T14:51:28.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:28.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 1 -- 172.21.15.26:0/2505289949 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f266c001cd0 con 0x7f267806bb80 2024-09-16T14:51:28.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:28.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:28.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267cc22640 10 monclient: dump: 2024-09-16T14:51:28.944 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:28.944 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:28.944 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:28.944 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:28.944 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:28.944 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:28.944 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:28.945 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:28.945 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:28.945 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:28.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:28.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 4 set_mon_vals no callback set 2024-09-16T14:51:28.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 10 monclient: shutdown 2024-09-16T14:51:28.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:28.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 1 -- 172.21.15.26:0/2505289949 >> v1:172.21.15.189:6789/0 conn(0x7f267806bb80 legacy=0x7f267813a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:28.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 1 -- 172.21.15.26:0/2505289949 shutdown_connections 2024-09-16T14:51:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 1 -- 172.21.15.26:0/2505289949 >> 172.21.15.26:0/2505289949 conn(0x7f26780a1c70 msgr2=0x7f26780a2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 1 -- 172.21.15.26:0/2505289949 shutdown_connections 2024-09-16T14:51:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 1 -- 172.21.15.26:0/2505289949 wait complete. 2024-09-16T14:51:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 10 monclient: build_initial_monmap 2024-09-16T14:51:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.937+0000 7f267fe88640 10 monclient: monmap: 2024-09-16T14:51:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:28.946 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:28.947 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:28.947 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:28.947 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:28.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 1 Processor -- start 2024-09-16T14:51:28.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 1 -- start start 2024-09-16T14:51:28.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 10 monclient: init 2024-09-16T14:51:28.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:28.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:51:28.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 10 monclient(hunting): picked mon.noname-c con 0x7f267806b090 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:28.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 10 monclient(hunting): picked mon.noname-a con 0x7f267806bb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:28.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 10 monclient(hunting): picked mon.noname-b con 0x7f267813b760 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:28.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267806d060 con 0x7f267806bb80 2024-09-16T14:51:28.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2678139b20 con 0x7f267813b760 2024-09-16T14:51:28.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267813b300 con 0x7f267806b090 2024-09-16T14:51:28.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:28.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 10 monclient(hunting): authenticate will time out at 1513.060839s 2024-09-16T14:51:28.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267d423640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f267806bb80 0x7f2678137d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56230/0 (socket says 172.21.15.26:56230) 2024-09-16T14:51:28.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267d423640 1 -- 172.21.15.26:0/1133117282 learned_addr learned my addr 172.21.15.26:0/1133117282 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:28.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 1 -- 172.21.15.26:0/1133117282 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567789275 0 0) 0x7f2678139b20 con 0x7f267813b760 2024-09-16T14:51:28.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:28.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:28.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:28.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 10 monclient(hunting): my global_id is 184279 2024-09-16T14:51:28.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 1 -- 172.21.15.26:0/1133117282 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2664003530 con 0x7f267813b760 2024-09-16T14:51:28.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 1 -- 172.21.15.26:0/1133117282 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 385542218 0 0) 0x7f2664003530 con 0x7f267813b760 2024-09-16T14:51:28.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 1 -- 172.21.15.26:0/1133117282 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2678139b20 con 0x7f267813b760 2024-09-16T14:51:28.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 1 -- 172.21.15.26:0/1133117282 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2668001af0 con 0x7f267813b760 2024-09-16T14:51:28.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:28.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:28.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 10 monclient(hunting): dump: 2024-09-16T14:51:28.952 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:28.952 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:28.952 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:28.952 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:28.952 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:28.952 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:28.952 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:28.952 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:28.952 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:28.953 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:28.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:28.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 1 -- 172.21.15.26:0/1133117282 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1395883093 0 0) 0x7f2678139b20 con 0x7f267813b760 2024-09-16T14:51:28.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 1 -- 172.21.15.26:0/1133117282 >> v1:172.21.15.26:6790/0 conn(0x7f267806b090 legacy=0x7f2678137670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:28.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 1 -- 172.21.15.26:0/1133117282 >> v1:172.21.15.26:6789/0 conn(0x7f267806bb80 legacy=0x7f2678137d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:28.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 10 monclient: _finish_hunting 0 2024-09-16T14:51:28.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 1 monclient: found mon.b 2024-09-16T14:51:28.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:28.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:28.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 1 -- 172.21.15.26:0/1133117282 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26780a04e0 con 0x7f267813b760 2024-09-16T14:51:28.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 10 monclient: _finish_auth 0 2024-09-16T14:51:28.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 10 monclient: _check_auth_tickets 2024-09-16T14:51:28.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:28.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 5 monclient: authenticate success, global_id 184279 2024-09-16T14:51:28.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 10 monclient: _renew_subs 2024-09-16T14:51:28.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:28.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 1 -- 172.21.15.26:0/1133117282 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f267809c360 con 0x7f267813b760 2024-09-16T14:51:28.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 10 monclient: _renew_subs 2024-09-16T14:51:28.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:28.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 1 -- 172.21.15.26:0/1133117282 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26781391d0 con 0x7f267813b760 2024-09-16T14:51:28.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 1 -- 172.21.15.26:0/1133117282 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2668004b70 con 0x7f267813b760 2024-09-16T14:51:28.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:28.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 1 -- 172.21.15.26:0/1133117282 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2668004f00 con 0x7f267813b760 2024-09-16T14:51:28.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:28.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:28.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f2677fff640 4 set_mon_vals no callback set 2024-09-16T14:51:28.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 10 monclient: dump: 2024-09-16T14:51:28.954 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:28.955 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:28.955 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:28.955 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:28.955 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:28.955 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:28.955 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:28.955 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:28.955 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:28.955 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:28.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:28.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:28.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:28.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267fe88640 1 -- 172.21.15.26:0/1133117282 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2658005050 con 0x7f267813b760 2024-09-16T14:51:28.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 1 -- 172.21.15.26:0/1133117282 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f26680050e0 con 0x7f267813b760 2024-09-16T14:51:28.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f26767fc640 1 -- 172.21.15.26:0/1133117282 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(257..257 src has 169..257) v4 ==== 6745+0+0 (unknown 920221734 0 0) 0x7f2668044120 con 0x7f267813b760 2024-09-16T14:51:28.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.941+0000 7f267dc24640 10 monclient: get_auth_request con 0x7f2664036060 auth_method 0 2024-09-16T14:51:28.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.957+0000 7f26767fc640 1 -- 172.21.15.26:0/1133117282 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2668014a60 con 0x7f267813b760 2024-09-16T14:51:28.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.957+0000 7f26767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:28.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:28.957+0000 7f26767fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:29.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.177+0000 7f267fe88640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:51:29.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.177+0000 7f267fe88640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:51:29.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.177+0000 7f267fe88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:29.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.177+0000 7f267fe88640 1 -- 172.21.15.26:0/1133117282 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2658005880 con 0x7f267813b760 2024-09-16T14:51:29.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.177+0000 7f26767fc640 1 -- 172.21.15.26:0/1133117282 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1851368750) 0x7f266801d050 con 0x7f267813b760 2024-09-16T14:51:29.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.177+0000 7f26767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:51:29.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.177+0000 7f26767fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:29.180 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":84,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:51:29.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.181+0000 7f267fe88640 1 -- 172.21.15.26:0/1133117282 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2664036060 legacy=0x7f2664038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.181+0000 7f267fe88640 10 monclient: shutdown 2024-09-16T14:51:29.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.181+0000 7f267fe88640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:29.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.181+0000 7f267fe88640 1 -- 172.21.15.26:0/1133117282 >> v1:172.21.15.189:6789/0 conn(0x7f267813b760 legacy=0x7f2678138490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.181+0000 7f267fe88640 1 -- 172.21.15.26:0/1133117282 shutdown_connections 2024-09-16T14:51:29.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.181+0000 7f267fe88640 1 -- 172.21.15.26:0/1133117282 >> 172.21.15.26:0/1133117282 conn(0x7f26780a1c70 msgr2=0x7f2678099890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:29.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.181+0000 7f267fe88640 1 -- 172.21.15.26:0/1133117282 shutdown_connections 2024-09-16T14:51:29.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.181+0000 7f267fe88640 1 -- 172.21.15.26:0/1133117282 wait complete. 2024-09-16T14:51:29.199 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:51:29.199 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:51:29.200 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:51:29.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.321+0000 7f051308b640 10 monclient: get_monmap_and_config 2024-09-16T14:51:29.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.321+0000 7f051308b640 10 monclient: build_initial_monmap 2024-09-16T14:51:29.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.321+0000 7f051308b640 10 monclient: monmap: 2024-09-16T14:51:29.324 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:29.324 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.324 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:29.325134+0000 2024-09-16T14:51:29.325 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:29.325134+0000 2024-09-16T14:51:29.325 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:29.325 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.325 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:29.325 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:29.325 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:29.325 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 1 Processor -- start 2024-09-16T14:51:29.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 1 -- start start 2024-09-16T14:51:29.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 10 monclient: init 2024-09-16T14:51:29.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:29.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:51:29.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 10 monclient(hunting): picked mon.noname-b con 0x7f050c09cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:29.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 10 monclient(hunting): picked mon.noname-a con 0x7f050c09e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:29.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 10 monclient(hunting): picked mon.noname-c con 0x7f050c09d1a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:29.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050c09d5b0 con 0x7f050c09e7a0 2024-09-16T14:51:29.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050c131a90 con 0x7f050c09d1a0 2024-09-16T14:51:29.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050c09aff0 con 0x7f050c09cdc0 2024-09-16T14:51:29.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:29.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 10 monclient(hunting): authenticate will time out at 1513.444823s 2024-09-16T14:51:29.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f0510e27640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f050c09cdc0 0x7f050c130c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59712/0 (socket says 172.21.15.26:59712) 2024-09-16T14:51:29.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f0510e27640 1 -- 172.21.15.26:0/2346290002 learned_addr learned my addr 172.21.15.26:0/2346290002 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:29.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 1 -- 172.21.15.26:0/2346290002 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792845411 0 0) 0x7f050c09aff0 con 0x7f050c09cdc0 2024-09-16T14:51:29.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient(hunting): my global_id is 154985 2024-09-16T14:51:29.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 1 -- 172.21.15.26:0/2346290002 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04fc003530 con 0x7f050c09cdc0 2024-09-16T14:51:29.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 1 -- 172.21.15.26:0/2346290002 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959082894 0 0) 0x7f050c09d5b0 con 0x7f050c09e7a0 2024-09-16T14:51:29.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient(hunting): my global_id is 194097 2024-09-16T14:51:29.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 1 -- 172.21.15.26:0/2346290002 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f050c09aff0 con 0x7f050c09e7a0 2024-09-16T14:51:29.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 1 -- 172.21.15.26:0/2346290002 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1102080893 0 0) 0x7f04fc003530 con 0x7f050c09cdc0 2024-09-16T14:51:29.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 1 -- 172.21.15.26:0/2346290002 >> v1:172.21.15.189:6789/0 conn(0x7f050c09d1a0 legacy=0x7f050c139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 1 -- 172.21.15.26:0/2346290002 >> v1:172.21.15.26:6789/0 conn(0x7f050c09e7a0 legacy=0x7f050c131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:51:29.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 1 monclient: found mon.noname-b 2024-09-16T14:51:29.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:29.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:51:29.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 1 -- 172.21.15.26:0/2346290002 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c09b2d0 con 0x7f050c09cdc0 2024-09-16T14:51:29.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:51:29.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:51:29.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:29.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 1 -- 172.21.15.26:0/2346290002 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0500001620 con 0x7f050c09cdc0 2024-09-16T14:51:29.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:29.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient: dump: 2024-09-16T14:51:29.334 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.334 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.335 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.335 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.335 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.335 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.335 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.335 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.335 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.335 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 5 monclient: authenticate success, global_id 154985 2024-09-16T14:51:29.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:29.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 1 -- 172.21.15.26:0/2346290002 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0500001850 con 0x7f050c09cdc0 2024-09-16T14:51:29.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:29.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 1 -- 172.21.15.26:0/2346290002 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0500001be0 con 0x7f050c09cdc0 2024-09-16T14:51:29.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:29.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 4 set_mon_vals no callback set 2024-09-16T14:51:29.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:29.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f050b7fe640 10 monclient: dump: 2024-09-16T14:51:29.336 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.336 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.336 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.336 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.336 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.336 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.336 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.336 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.336 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.337 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 10 monclient: shutdown 2024-09-16T14:51:29.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:29.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 1 -- 172.21.15.26:0/2346290002 >> v1:172.21.15.26:6790/0 conn(0x7f050c09cdc0 legacy=0x7f050c130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 1 -- 172.21.15.26:0/2346290002 shutdown_connections 2024-09-16T14:51:29.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 1 -- 172.21.15.26:0/2346290002 >> 172.21.15.26:0/2346290002 conn(0x7f050c09fb50 msgr2=0x7f050c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:29.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 1 -- 172.21.15.26:0/2346290002 shutdown_connections 2024-09-16T14:51:29.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 1 -- 172.21.15.26:0/2346290002 wait complete. 2024-09-16T14:51:29.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 10 monclient: build_initial_monmap 2024-09-16T14:51:29.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 10 monclient: monmap: 2024-09-16T14:51:29.337 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:29.337 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:29.337 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:29.338 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:29.338 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:29.338 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:29.338 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:29.338 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:29.338 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:29.338 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 1 Processor -- start 2024-09-16T14:51:29.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 1 -- start start 2024-09-16T14:51:29.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.325+0000 7f051308b640 10 monclient: init 2024-09-16T14:51:29.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:29.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:51:29.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 10 monclient(hunting): picked mon.noname-b con 0x7f050c09cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:29.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 10 monclient(hunting): picked mon.noname-c con 0x7f050c09d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:29.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 10 monclient(hunting): picked mon.noname-a con 0x7f050c09e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:29.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050c1c6140 con 0x7f050c09e7a0 2024-09-16T14:51:29.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050c12d7e0 con 0x7f050c09cdc0 2024-09-16T14:51:29.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050c12d5d0 con 0x7f050c09d1a0 2024-09-16T14:51:29.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:29.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 10 monclient(hunting): authenticate will time out at 1513.448522s 2024-09-16T14:51:29.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f050bfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f050c09d1a0 0x7f050c1ca5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59714/0 (socket says 172.21.15.26:59714) 2024-09-16T14:51:29.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f0511628640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f050c09e7a0 0x7f050c1cdcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56246/0 (socket says 172.21.15.26:56246) 2024-09-16T14:51:29.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f050bfff640 1 -- 172.21.15.26:0/742429437 learned_addr learned my addr 172.21.15.26:0/742429437 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:29.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754236617 0 0) 0x7f050c131a90 con 0x7f050c09e7a0 2024-09-16T14:51:29.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): my global_id is 194100 2024-09-16T14:51:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f0003530 con 0x7f050c09e7a0 2024-09-16T14:51:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097017141 0 0) 0x7f050c12d5d0 con 0x7f050c09d1a0 2024-09-16T14:51:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): my global_id is 154988 2024-09-16T14:51:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f050c131a90 con 0x7f050c09d1a0 2024-09-16T14:51:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902484545 0 0) 0x7f050c12d7e0 con 0x7f050c09cdc0 2024-09-16T14:51:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): my global_id is 184285 2024-09-16T14:51:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f050c12d5d0 con 0x7f050c09cdc0 2024-09-16T14:51:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4032026543 0 0) 0x7f050c131a90 con 0x7f050c09d1a0 2024-09-16T14:51:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f050c12d7e0 con 0x7f050c09d1a0 2024-09-16T14:51:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3027812972 0 0) 0x7f04f0003530 con 0x7f050c09e7a0 2024-09-16T14:51:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f050c131a90 con 0x7f050c09e7a0 2024-09-16T14:51:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f04f80019a0 con 0x7f050c09d1a0 2024-09-16T14:51:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): dump: 2024-09-16T14:51:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.341 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.342 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f05040019a0 con 0x7f050c09e7a0 2024-09-16T14:51:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): dump: 2024-09-16T14:51:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.343 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1811794308 0 0) 0x7f050c12d7e0 con 0x7f050c09d1a0 2024-09-16T14:51:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 >> v1:172.21.15.189:6789/0 conn(0x7f050c09cdc0 legacy=0x7f050c12cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 >> v1:172.21.15.26:6789/0 conn(0x7f050c09e7a0 legacy=0x7f050c1cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient: _finish_hunting 0 2024-09-16T14:51:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 monclient: found mon.c 2024-09-16T14:51:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c06fe20 con 0x7f050c09d1a0 2024-09-16T14:51:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient: _finish_auth 0 2024-09-16T14:51:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient: _check_auth_tickets 2024-09-16T14:51:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 5 monclient: authenticate success, global_id 154988 2024-09-16T14:51:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 10 monclient: _renew_subs 2024-09-16T14:51:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 1 -- 172.21.15.26:0/742429437 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050c0a2460 con 0x7f050c09d1a0 2024-09-16T14:51:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 10 monclient: _renew_subs 2024-09-16T14:51:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:29.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 1 -- 172.21.15.26:0/742429437 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f050c11a000 con 0x7f050c09d1a0 2024-09-16T14:51:29.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04f8001c60 con 0x7f050c09d1a0 2024-09-16T14:51:29.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:29.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f04f8003a70 con 0x7f050c09d1a0 2024-09-16T14:51:29.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f050affd640 4 set_mon_vals no callback set 2024-09-16T14:51:29.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:29.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient: dump: 2024-09-16T14:51:29.345 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.345 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.345 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.345 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.345 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.345 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.346 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.346 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.346 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.346 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:29.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f051308b640 1 -- 172.21.15.26:0/742429437 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f04ec006590 con 0x7f04ec002830 2024-09-16T14:51:29.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f04f8003bd0 con 0x7f050c09d1a0 2024-09-16T14:51:29.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 169..257) v4 ==== 6745+0+0 (unknown 920221734 0 0) 0x7f04f8043d20 con 0x7f050c09d1a0 2024-09-16T14:51:29.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002107370 0 0) 0x7f04ec006590 con 0x7f04ec002830 2024-09-16T14:51:29.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient(hunting): my global_id is 194103 2024-09-16T14:51:29.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f050c12d7e0 con 0x7f04ec002830 2024-09-16T14:51:29.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f0511628640 10 monclient: get_auth_request con 0x7f04f00385f0 auth_method 0 2024-09-16T14:51:29.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 278709767 0 0) 0x7f050c12d7e0 con 0x7f04ec002830 2024-09-16T14:51:29.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04ec006590 con 0x7f04ec002830 2024-09-16T14:51:29.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f05000019c0 con 0x7f04ec002830 2024-09-16T14:51:29.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:29.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient: dump: 2024-09-16T14:51:29.347 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.347 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.347 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.348 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.348 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.348 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.348 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.348 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.348 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.348 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3149036063 0 0) 0x7f04ec006590 con 0x7f04ec002830 2024-09-16T14:51:29.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04ec00a0e0 con 0x7f04ec002830 2024-09-16T14:51:29.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0500001be0 con 0x7f04ec002830 2024-09-16T14:51:29.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:29.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:29.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 >> v1:172.21.15.26:6789/0 conn(0x7f04ec002830 legacy=0x7f04ec004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.329+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 >> v1:172.21.15.26:6789/0 conn(0x7f04ec002830 legacy=0x7f04ec004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:29.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f051308b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:51:29.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f051308b640 1 -- 172.21.15.26:0/742429437 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f04ec009d80 con 0x7f04ec0090a0 2024-09-16T14:51:29.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896928909 0 0) 0x7f04ec009d80 con 0x7f04ec0090a0 2024-09-16T14:51:29.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 10 monclient(hunting): my global_id is 194106 2024-09-16T14:51:29.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ec006590 con 0x7f04ec0090a0 2024-09-16T14:51:29.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1478994722 0 0) 0x7f04ec006590 con 0x7f04ec0090a0 2024-09-16T14:51:29.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04ec009d80 con 0x7f04ec0090a0 2024-09-16T14:51:29.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0500004bd0 con 0x7f04ec0090a0 2024-09-16T14:51:29.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:29.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 10 monclient: dump: 2024-09-16T14:51:29.351 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.351 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.351 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.351 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.351 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.351 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.352 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.352 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.352 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.352 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2965653186 0 0) 0x7f04ec009d80 con 0x7f04ec0090a0 2024-09-16T14:51:29.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f04ec00c290 con 0x7f04ec0090a0 2024-09-16T14:51:29.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 2892000601) 0x7f0500005f30 con 0x7f04ec0090a0 2024-09-16T14:51:29.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:51:29.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:29.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 >> v1:172.21.15.26:6789/0 conn(0x7f04ec0090a0 legacy=0x7f04ec009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f05097fa640 1 -- 172.21.15.26:0/742429437 >> v1:172.21.15.26:6789/0 conn(0x7f04ec0090a0 legacy=0x7f04ec009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:29.353 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:51:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:51:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:51:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:51:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 84, 2024-09-16T14:51:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:51:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:51:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:51:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:51:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:51:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:29.353 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:51:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:51:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:51:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:51:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:51:29.355 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:51:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:51:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:51:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:51:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:51:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:51:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:51:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:51:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:51:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:51:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:51:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:51:29.356 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:51:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:29.357 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:51:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:51:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:51:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:51:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:51:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:51:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:51:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:51:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:51:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:51:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:29.359 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:51:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:51:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:51:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:51:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:29.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:51:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:51:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:51:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:29.361 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:51:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:51:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:51:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:51:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:29.362 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:29.363 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:51:29.363 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.363 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.363 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:51:29.363 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:51:29.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f051308b640 1 -- 172.21.15.26:0/742429437 >> v1:172.21.15.189:6812/3196161582 conn(0x7f04f00385f0 legacy=0x7f04f003aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f051308b640 10 monclient: shutdown 2024-09-16T14:51:29.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f051308b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:29.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f051308b640 1 -- 172.21.15.26:0/742429437 >> v1:172.21.15.26:6790/0 conn(0x7f050c09d1a0 legacy=0x7f050c1ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f0511628640 1 -- 172.21.15.26:0/742429437 reap_dead start 2024-09-16T14:51:29.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f0511628640 1 -- 172.21.15.26:0/742429437 reap_dead start 2024-09-16T14:51:29.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f051308b640 1 -- 172.21.15.26:0/742429437 shutdown_connections 2024-09-16T14:51:29.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f051308b640 1 -- 172.21.15.26:0/742429437 >> 172.21.15.26:0/742429437 conn(0x7f050c09fb50 msgr2=0x7f050c131140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:29.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f051308b640 1 -- 172.21.15.26:0/742429437 shutdown_connections 2024-09-16T14:51:29.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.337+0000 7f051308b640 1 -- 172.21.15.26:0/742429437 wait complete. 2024-09-16T14:51:29.365 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:51:29.365 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:51:29.365 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:51:29.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 10 monclient: get_monmap_and_config 2024-09-16T14:51:29.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 10 monclient: build_initial_monmap 2024-09-16T14:51:29.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 10 monclient: monmap: 2024-09-16T14:51:29.481 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:29.481 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.481 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:29.482500+0000 2024-09-16T14:51:29.481 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:29.482500+0000 2024-09-16T14:51:29.481 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:29.482 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.482 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:29.482 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:29.482 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:29.482 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 1 Processor -- start 2024-09-16T14:51:29.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 1 -- start start 2024-09-16T14:51:29.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 10 monclient: init 2024-09-16T14:51:29.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:29.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:51:29.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 10 monclient(hunting): picked mon.noname-c con 0x7fe1100a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:29.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 10 monclient(hunting): picked mon.noname-a con 0x7fe1100a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:29.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 10 monclient(hunting): picked mon.noname-b con 0x7fe1100a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:29.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1100a3250 con 0x7fe1100a0000 2024-09-16T14:51:29.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11009f600 con 0x7fe1100a1d10 2024-09-16T14:51:29.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1100a3830 con 0x7fe1100a2800 2024-09-16T14:51:29.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:29.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 10 monclient(hunting): authenticate will time out at 1513.602554s 2024-09-16T14:51:29.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe115f5b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe1100a2800 0x7fe11013d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59716/0 (socket says 172.21.15.26:59716) 2024-09-16T14:51:29.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe115f5b640 1 -- 172.21.15.26:0/1697598527 learned_addr learned my addr 172.21.15.26:0/1697598527 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:29.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 1 -- 172.21.15.26:0/1697598527 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009691086 0 0) 0x7fe1100a3250 con 0x7fe1100a0000 2024-09-16T14:51:29.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient(hunting): my global_id is 194109 2024-09-16T14:51:29.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 1 -- 172.21.15.26:0/1697598527 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0fc003530 con 0x7fe1100a0000 2024-09-16T14:51:29.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 1 -- 172.21.15.26:0/1697598527 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682001490 0 0) 0x7fe1100a3830 con 0x7fe1100a2800 2024-09-16T14:51:29.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient(hunting): my global_id is 154991 2024-09-16T14:51:29.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 1 -- 172.21.15.26:0/1697598527 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1100a3250 con 0x7fe1100a2800 2024-09-16T14:51:29.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 1 -- 172.21.15.26:0/1697598527 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351104098 0 0) 0x7fe11009f600 con 0x7fe1100a1d10 2024-09-16T14:51:29.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient(hunting): my global_id is 184288 2024-09-16T14:51:29.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 1 -- 172.21.15.26:0/1697598527 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1100a3830 con 0x7fe1100a1d10 2024-09-16T14:51:29.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 1 -- 172.21.15.26:0/1697598527 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 529860669 0 0) 0x7fe1100a3250 con 0x7fe1100a2800 2024-09-16T14:51:29.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 1 -- 172.21.15.26:0/1697598527 >> v1:172.21.15.189:6789/0 conn(0x7fe1100a1d10 legacy=0x7fe1100a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 1 -- 172.21.15.26:0/1697598527 >> v1:172.21.15.26:6789/0 conn(0x7fe1100a0000 legacy=0x7fe110139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient: _finish_hunting 0 2024-09-16T14:51:29.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 1 monclient: found mon.noname-b 2024-09-16T14:51:29.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:29.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:51:29.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 1 -- 172.21.15.26:0/1697598527 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11009ed60 con 0x7fe1100a2800 2024-09-16T14:51:29.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient: _finish_auth 0 2024-09-16T14:51:29.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient: _check_auth_tickets 2024-09-16T14:51:29.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:29.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 5 monclient: authenticate success, global_id 154991 2024-09-16T14:51:29.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:29.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 1 -- 172.21.15.26:0/1697598527 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe108001600 con 0x7fe1100a2800 2024-09-16T14:51:29.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:29.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient: dump: 2024-09-16T14:51:29.494 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.494 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.494 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.494 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.494 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.494 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.494 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.494 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.494 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.494 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:29.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 1 -- 172.21.15.26:0/1697598527 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe108001820 con 0x7fe1100a2800 2024-09-16T14:51:29.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:29.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:29.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe1179be640 4 set_mon_vals no callback set 2024-09-16T14:51:29.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 1 -- 172.21.15.26:0/1697598527 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe108001c30 con 0x7fe1100a2800 2024-09-16T14:51:29.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:29.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.481+0000 7fe103fff640 10 monclient: dump: 2024-09-16T14:51:29.496 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.496 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.496 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.496 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.496 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.496 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.496 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.496 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.496 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.496 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 10 monclient: shutdown 2024-09-16T14:51:29.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:29.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 1 -- 172.21.15.26:0/1697598527 >> v1:172.21.15.26:6790/0 conn(0x7fe1100a2800 legacy=0x7fe11013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 1 -- 172.21.15.26:0/1697598527 shutdown_connections 2024-09-16T14:51:29.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 1 -- 172.21.15.26:0/1697598527 >> 172.21.15.26:0/1697598527 conn(0x7fe1100a7f30 msgr2=0x7fe1100a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:29.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 1 -- 172.21.15.26:0/1697598527 shutdown_connections 2024-09-16T14:51:29.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 1 -- 172.21.15.26:0/1697598527 wait complete. 2024-09-16T14:51:29.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 10 monclient: build_initial_monmap 2024-09-16T14:51:29.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 10 monclient: monmap: 2024-09-16T14:51:29.497 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:29.497 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:29.497 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:29.497 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:29.497 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:29.498 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 1 Processor -- start 2024-09-16T14:51:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 1 -- start start 2024-09-16T14:51:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 10 monclient: init 2024-09-16T14:51:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:51:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 10 monclient(hunting): picked mon.noname-c con 0x7fe1100a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 10 monclient(hunting): picked mon.noname-b con 0x7fe1100a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:29.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 10 monclient(hunting): picked mon.noname-a con 0x7fe1100a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1101355e0 con 0x7fe1100a2800 2024-09-16T14:51:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1100a6900 con 0x7fe1100a1d10 2024-09-16T14:51:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe110133de0 con 0x7fe1100a0000 2024-09-16T14:51:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 10 monclient(hunting): authenticate will time out at 1513.605750s 2024-09-16T14:51:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe11575a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe1100a0000 0x7fe110132fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59730/0 (socket says 172.21.15.26:59730) 2024-09-16T14:51:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe11575a640 1 -- 172.21.15.26:0/2948869370 learned_addr learned my addr 172.21.15.26:0/2948869370 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296296095 0 0) 0x7fe110133de0 con 0x7fe1100a0000 2024-09-16T14:51:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): my global_id is 154994 2024-09-16T14:51:29.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f4003530 con 0x7fe1100a0000 2024-09-16T14:51:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524601583 0 0) 0x7fe1101355e0 con 0x7fe1100a2800 2024-09-16T14:51:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): my global_id is 194112 2024-09-16T14:51:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe110133de0 con 0x7fe1100a2800 2024-09-16T14:51:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4270060716 0 0) 0x7fe0f4003530 con 0x7fe1100a0000 2024-09-16T14:51:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1101355e0 con 0x7fe1100a0000 2024-09-16T14:51:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe1040019a0 con 0x7fe1100a0000 2024-09-16T14:51:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:29.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): dump: 2024-09-16T14:51:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.501 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2606620417 0 0) 0x7fe110133de0 con 0x7fe1100a2800 2024-09-16T14:51:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0f4003530 con 0x7fe1100a2800 2024-09-16T14:51:29.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe108001b30 con 0x7fe1100a2800 2024-09-16T14:51:29.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:29.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): dump: 2024-09-16T14:51:29.502 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.502 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.502 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.502 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.502 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.502 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.502 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.502 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.502 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.503 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622093571 0 0) 0x7fe1100a6900 con 0x7fe1100a1d10 2024-09-16T14:51:29.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient(hunting): my global_id is 184291 2024-09-16T14:51:29.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe110133de0 con 0x7fe1100a1d10 2024-09-16T14:51:29.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4110730186 0 0) 0x7fe1101355e0 con 0x7fe1100a0000 2024-09-16T14:51:29.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 >> v1:172.21.15.189:6789/0 conn(0x7fe1100a1d10 legacy=0x7fe1101336d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 >> v1:172.21.15.26:6789/0 conn(0x7fe1100a2800 legacy=0x7fe1101d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:51:29.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 monclient: found mon.c 2024-09-16T14:51:29.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:29.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:29.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1100a63d0 con 0x7fe1100a0000 2024-09-16T14:51:29.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient: _finish_auth 0 2024-09-16T14:51:29.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:51:29.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:29.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 5 monclient: authenticate success, global_id 154994 2024-09-16T14:51:29.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 10 monclient: _renew_subs 2024-09-16T14:51:29.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:29.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 1 -- 172.21.15.26:0/2948869370 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11009e500 con 0x7fe1100a0000 2024-09-16T14:51:29.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 10 monclient: _renew_subs 2024-09-16T14:51:29.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:29.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 1 -- 172.21.15.26:0/2948869370 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe110136e80 con 0x7fe1100a0000 2024-09-16T14:51:29.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1040036a0 con 0x7fe1100a0000 2024-09-16T14:51:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe104003a30 con 0x7fe1100a0000 2024-09-16T14:51:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1037fe640 4 set_mon_vals no callback set 2024-09-16T14:51:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 10 monclient: dump: 2024-09-16T14:51:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.505 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.506 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe1179be640 1 -- 172.21.15.26:0/2948869370 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe0f0006590 con 0x7fe0f0002830 2024-09-16T14:51:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe104003c10 con 0x7fe1100a0000 2024-09-16T14:51:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.485+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 169..257) v4 ==== 6745+0+0 (unknown 920221734 0 0) 0x7fe104043d40 con 0x7fe1100a0000 2024-09-16T14:51:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431180476 0 0) 0x7fe0f0006590 con 0x7fe0f0002830 2024-09-16T14:51:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 10 monclient(hunting): my global_id is 154997 2024-09-16T14:51:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1101355e0 con 0x7fe0f0002830 2024-09-16T14:51:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe115f5b640 10 monclient: get_auth_request con 0x7fe0f4036390 auth_method 0 2024-09-16T14:51:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2101830046 0 0) 0x7fe1101355e0 con 0x7fe0f0002830 2024-09-16T14:51:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0f0006590 con 0x7fe0f0002830 2024-09-16T14:51:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe10c001960 con 0x7fe0f0002830 2024-09-16T14:51:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 10 monclient: dump: 2024-09-16T14:51:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.507 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.508 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1927318576 0 0) 0x7fe0f0006590 con 0x7fe0f0002830 2024-09-16T14:51:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe0f000a0e0 con 0x7fe0f0002830 2024-09-16T14:51:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe10c001ce0 con 0x7fe0f0002830 2024-09-16T14:51:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:29.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 >> v1:172.21.15.26:6790/0 conn(0x7fe0f0002830 legacy=0x7fe0f0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.489+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 >> v1:172.21.15.26:6790/0 conn(0x7fe0f0002830 legacy=0x7fe0f0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.493+0000 7fe1179be640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:51:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.493+0000 7fe1179be640 1 -- 172.21.15.26:0/2948869370 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe0f0009da0 con 0x7fe0f00090c0 2024-09-16T14:51:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.493+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021818990 0 0) 0x7fe0f0009da0 con 0x7fe0f00090c0 2024-09-16T14:51:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.493+0000 7fe101ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.493+0000 7fe101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.493+0000 7fe101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.493+0000 7fe101ffb640 10 monclient(hunting): my global_id is 155000 2024-09-16T14:51:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.493+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f0006590 con 0x7fe0f00090c0 2024-09-16T14:51:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.493+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2579834088 0 0) 0x7fe0f0006590 con 0x7fe0f00090c0 2024-09-16T14:51:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.493+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0f0009da0 con 0x7fe0f00090c0 2024-09-16T14:51:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.493+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe10c0020e0 con 0x7fe0f00090c0 2024-09-16T14:51:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.493+0000 7fe101ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.493+0000 7fe101ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.493+0000 7fe101ffb640 10 monclient: dump: 2024-09-16T14:51:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.511 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.512 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.493+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3764008843 0 0) 0x7fe0f0009da0 con 0x7fe0f00090c0 2024-09-16T14:51:29.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.493+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe0f000c290 con 0x7fe0f00090c0 2024-09-16T14:51:29.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.497+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 3133735615) 0x7fe10c003d00 con 0x7fe0f00090c0 2024-09-16T14:51:29.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.497+0000 7fe101ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:51:29.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.497+0000 7fe101ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:29.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.497+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 >> v1:172.21.15.26:6790/0 conn(0x7fe0f00090c0 legacy=0x7fe0f0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.497+0000 7fe101ffb640 1 -- 172.21.15.26:0/2948869370 >> v1:172.21.15.26:6790/0 conn(0x7fe0f00090c0 legacy=0x7fe0f0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:29.513 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:51:29.513 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:51:29.513 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:51:29.513 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:51:29.514 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 84, 2024-09-16T14:51:29.514 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:51:29.514 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:51:29.514 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:51:29.514 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:51:29.514 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:29.514 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:51:29.514 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:29.514 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:51:29.514 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:51:29.514 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:29.514 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:29.514 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:29.515 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:29.515 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:29.515 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:29.515 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:29.515 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:29.515 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:29.515 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:29.515 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:29.515 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:51:29.515 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:51:29.515 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:29.515 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:29.515 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:29.516 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:29.516 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:29.516 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:29.516 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:29.516 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:29.516 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:29.516 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:29.516 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.516 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:51:29.516 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:51:29.516 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:51:29.517 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:51:29.517 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:51:29.517 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:51:29.517 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:29.517 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:29.517 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:51:29.517 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:51:29.517 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:51:29.517 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:51:29.517 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:51:29.517 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:51:29.517 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:51:29.518 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:29.518 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:51:29.518 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:29.518 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:29.518 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:29.518 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:29.518 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:29.518 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:29.518 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:29.518 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:29.518 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:29.518 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:29.518 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:29.519 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:51:29.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.519 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:51:29.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.519 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:51:29.519 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:51:29.519 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:29.519 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:29.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.519 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:29.519 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:51:29.519 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:29.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.520 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.520 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:51:29.520 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:51:29.520 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:29.520 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:29.520 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:29.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.520 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:51:29.520 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:51:29.520 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:29.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:29.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.521 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:29.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:51:29.521 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:29.521 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.521 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:51:29.521 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:51:29.521 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:29.521 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:29.521 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:29.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.522 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:51:29.522 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:51:29.522 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:29.522 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:29.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.522 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:29.522 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:51:29.522 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:29.522 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.522 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.522 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:51:29.522 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:51:29.523 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:29.523 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:29.523 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:29.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.523 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.523 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:51:29.523 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:51:29.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.523 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:29.523 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:29.523 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:51:29.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.523 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:29.524 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:51:29.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.524 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:29.524 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:29.524 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:51:29.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.524 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:29.524 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:51:29.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.524 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:29.524 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:29.524 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:51:29.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.525 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:29.525 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:51:29.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.525 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:29.525 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:29.525 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:51:29.525 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.525 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.525 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:51:29.525 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:51:29.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.497+0000 7fe1179be640 1 -- 172.21.15.26:0/2948869370 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe0f4036390 legacy=0x7fe0f4038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.497+0000 7fe1179be640 10 monclient: shutdown 2024-09-16T14:51:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.497+0000 7fe1179be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.497+0000 7fe1179be640 1 -- 172.21.15.26:0/2948869370 >> v1:172.21.15.26:6790/0 conn(0x7fe1100a0000 legacy=0x7fe110132fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.497+0000 7fe115f5b640 1 -- 172.21.15.26:0/2948869370 reap_dead start 2024-09-16T14:51:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.497+0000 7fe115f5b640 1 -- 172.21.15.26:0/2948869370 reap_dead start 2024-09-16T14:51:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.497+0000 7fe1179be640 1 -- 172.21.15.26:0/2948869370 shutdown_connections 2024-09-16T14:51:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.497+0000 7fe1179be640 1 -- 172.21.15.26:0/2948869370 >> 172.21.15.26:0/2948869370 conn(0x7fe1100a7f30 msgr2=0x7fe11006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.497+0000 7fe1179be640 1 -- 172.21.15.26:0/2948869370 shutdown_connections 2024-09-16T14:51:29.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.497+0000 7fe1179be640 1 -- 172.21.15.26:0/2948869370 wait complete. 2024-09-16T14:51:29.527 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:51:29.528 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:51:29.528 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:51:29.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf61a6d640 10 monclient: get_monmap_and_config 2024-09-16T14:51:29.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf61a6d640 10 monclient: build_initial_monmap 2024-09-16T14:51:29.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf61a6d640 10 monclient: monmap: 2024-09-16T14:51:29.642 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:29.643 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.643 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:29.643864+0000 2024-09-16T14:51:29.643 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:29.643864+0000 2024-09-16T14:51:29.643 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:29.643 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.643 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:29.643 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:29.643 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:29.643 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf61a6d640 1 Processor -- start 2024-09-16T14:51:29.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf61a6d640 1 -- start start 2024-09-16T14:51:29.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf61a6d640 10 monclient: init 2024-09-16T14:51:29.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf61a6d640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:29.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf61a6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:51:29.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf61a6d640 10 monclient(hunting): picked mon.noname-c con 0x7fbf5c09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:29.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf61a6d640 10 monclient(hunting): picked mon.noname-a con 0x7fbf5c132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:29.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf61a6d640 10 monclient(hunting): picked mon.noname-b con 0x7fbf5c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:29.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf61a6d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c09d260 con 0x7fbf5c132650 2024-09-16T14:51:29.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf61a6d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c132a80 con 0x7fbf5c09cb80 2024-09-16T14:51:29.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf61a6d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c09cf60 con 0x7fbf5c138ff0 2024-09-16T14:51:29.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf61a6d640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:29.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf61a6d640 10 monclient(hunting): authenticate will time out at 1513.763595s 2024-09-16T14:51:29.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf5bfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbf5c138ff0 0x7fbf5c13b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59772/0 (socket says 172.21.15.26:59772) 2024-09-16T14:51:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.641+0000 7fbf5bfff640 1 -- 172.21.15.26:0/1328309884 learned_addr learned my addr 172.21.15.26:0/1328309884 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 1 -- 172.21.15.26:0/1328309884 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540365652 0 0) 0x7fbf5c09d260 con 0x7fbf5c132650 2024-09-16T14:51:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient(hunting): my global_id is 194115 2024-09-16T14:51:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 1 -- 172.21.15.26:0/1328309884 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf44003530 con 0x7fbf5c132650 2024-09-16T14:51:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 1 -- 172.21.15.26:0/1328309884 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71038849 0 0) 0x7fbf5c09cf60 con 0x7fbf5c138ff0 2024-09-16T14:51:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient(hunting): my global_id is 155003 2024-09-16T14:51:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 1 -- 172.21.15.26:0/1328309884 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c09d260 con 0x7fbf5c138ff0 2024-09-16T14:51:29.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 1 -- 172.21.15.26:0/1328309884 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763062873 0 0) 0x7fbf5c132a80 con 0x7fbf5c09cb80 2024-09-16T14:51:29.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient(hunting): my global_id is 184294 2024-09-16T14:51:29.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 1 -- 172.21.15.26:0/1328309884 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c09cf60 con 0x7fbf5c09cb80 2024-09-16T14:51:29.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 1 -- 172.21.15.26:0/1328309884 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095655028 0 0) 0x7fbf44003530 con 0x7fbf5c132650 2024-09-16T14:51:29.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 1 -- 172.21.15.26:0/1328309884 >> v1:172.21.15.26:6790/0 conn(0x7fbf5c138ff0 legacy=0x7fbf5c13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 1 -- 172.21.15.26:0/1328309884 >> v1:172.21.15.189:6789/0 conn(0x7fbf5c09cb80 legacy=0x7fbf5c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:51:29.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 1 monclient: found mon.noname-a 2024-09-16T14:51:29.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:29.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:51:29.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 1 -- 172.21.15.26:0/1328309884 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c09f2d0 con 0x7fbf5c132650 2024-09-16T14:51:29.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:51:29.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:51:29.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:29.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 1 -- 172.21.15.26:0/1328309884 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf4c001620 con 0x7fbf5c132650 2024-09-16T14:51:29.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 5 monclient: authenticate success, global_id 194115 2024-09-16T14:51:29.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:29.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:29.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient: dump: 2024-09-16T14:51:29.655 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.656 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.656 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.656 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.656 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.656 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.656 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.656 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.656 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.656 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:29.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 1 -- 172.21.15.26:0/1328309884 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf4c001890 con 0x7fbf5c132650 2024-09-16T14:51:29.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:29.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 1 -- 172.21.15.26:0/1328309884 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf4c001c60 con 0x7fbf5c132650 2024-09-16T14:51:29.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:29.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 4 set_mon_vals no callback set 2024-09-16T14:51:29.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:29.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5a7fc640 10 monclient: dump: 2024-09-16T14:51:29.657 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.657 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.657 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.657 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.657 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.657 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.658 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.658 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.658 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.658 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 10 monclient: shutdown 2024-09-16T14:51:29.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:29.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 1 -- 172.21.15.26:0/1328309884 >> v1:172.21.15.26:6789/0 conn(0x7fbf5c132650 legacy=0x7fbf5c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 1 -- 172.21.15.26:0/1328309884 shutdown_connections 2024-09-16T14:51:29.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 1 -- 172.21.15.26:0/1328309884 >> 172.21.15.26:0/1328309884 conn(0x7fbf5c09fb50 msgr2=0x7fbf5c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:29.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 1 -- 172.21.15.26:0/1328309884 shutdown_connections 2024-09-16T14:51:29.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 1 -- 172.21.15.26:0/1328309884 wait complete. 2024-09-16T14:51:29.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 10 monclient: build_initial_monmap 2024-09-16T14:51:29.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 10 monclient: monmap: 2024-09-16T14:51:29.659 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:29.659 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:29.659 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:29.659 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:29.660 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:29.660 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:29.660 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:29.660 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:29.660 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:29.660 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 1 Processor -- start 2024-09-16T14:51:29.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 1 -- start start 2024-09-16T14:51:29.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 10 monclient: init 2024-09-16T14:51:29.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:29.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:51:29.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 10 monclient(hunting): picked mon.noname-b con 0x7fbf5c09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:29.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 10 monclient(hunting): picked mon.noname-a con 0x7fbf5c132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:29.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 10 monclient(hunting): picked mon.noname-c con 0x7fbf5c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:29.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c09b680 con 0x7fbf5c132650 2024-09-16T14:51:29.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c09a240 con 0x7fbf5c09cb80 2024-09-16T14:51:29.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c0719c0 con 0x7fbf5c138ff0 2024-09-16T14:51:29.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:29.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 10 monclient(hunting): authenticate will time out at 1513.766923s 2024-09-16T14:51:29.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbf5c132650 0x7fbf5c09ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56286/0 (socket says 172.21.15.26:56286) 2024-09-16T14:51:29.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5bfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbf5c138ff0 0x7fbf5c1d1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59774/0 (socket says 172.21.15.26:59774) 2024-09-16T14:51:29.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf5affd640 1 -- 172.21.15.26:0/159384771 learned_addr learned my addr 172.21.15.26:0/159384771 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:29.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158108067 0 0) 0x7fbf5c0719c0 con 0x7fbf5c138ff0 2024-09-16T14:51:29.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): my global_id is 155006 2024-09-16T14:51:29.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf48003530 con 0x7fbf5c138ff0 2024-09-16T14:51:29.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275670281 0 0) 0x7fbf5c09b680 con 0x7fbf5c132650 2024-09-16T14:51:29.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): my global_id is 194118 2024-09-16T14:51:29.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c0719c0 con 0x7fbf5c132650 2024-09-16T14:51:29.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1157981773 0 0) 0x7fbf48003530 con 0x7fbf5c138ff0 2024-09-16T14:51:29.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf5c09b680 con 0x7fbf5c138ff0 2024-09-16T14:51:29.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf540014c0 con 0x7fbf5c138ff0 2024-09-16T14:51:29.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:29.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): dump: 2024-09-16T14:51:29.663 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.663 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.663 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.663 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.663 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.663 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.663 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.663 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.663 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.663 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1091417596 0 0) 0x7fbf5c0719c0 con 0x7fbf5c132650 2024-09-16T14:51:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf48003530 con 0x7fbf5c132650 2024-09-16T14:51:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf4c001ba0 con 0x7fbf5c132650 2024-09-16T14:51:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): dump: 2024-09-16T14:51:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.664 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.665 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.665 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.665 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.665 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.665 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881756529 0 0) 0x7fbf5c09a240 con 0x7fbf5c09cb80 2024-09-16T14:51:29.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient(hunting): my global_id is 184297 2024-09-16T14:51:29.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c0719c0 con 0x7fbf5c09cb80 2024-09-16T14:51:29.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1630543979 0 0) 0x7fbf5c09b680 con 0x7fbf5c138ff0 2024-09-16T14:51:29.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 >> v1:172.21.15.189:6789/0 conn(0x7fbf5c09cb80 legacy=0x7fbf5c09a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 >> v1:172.21.15.26:6789/0 conn(0x7fbf5c132650 legacy=0x7fbf5c09ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient: _finish_hunting 0 2024-09-16T14:51:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 monclient: found mon.c 2024-09-16T14:51:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c09b370 con 0x7fbf5c138ff0 2024-09-16T14:51:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient: _finish_auth 0 2024-09-16T14:51:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 10 monclient: _check_auth_tickets 2024-09-16T14:51:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 5 monclient: authenticate success, global_id 155006 2024-09-16T14:51:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 10 monclient: _renew_subs 2024-09-16T14:51:29.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 1 -- 172.21.15.26:0/159384771 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5c0a2940 con 0x7fbf5c138ff0 2024-09-16T14:51:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 10 monclient: _renew_subs 2024-09-16T14:51:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.645+0000 7fbf61a6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf61a6d640 1 -- 172.21.15.26:0/159384771 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5c06d570 con 0x7fbf5c138ff0 2024-09-16T14:51:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf61a6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf61a6d640 1 -- 172.21.15.26:0/159384771 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbf5c137e80 con 0x7fbf5c137aa0 2024-09-16T14:51:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf540019d0 con 0x7fbf5c138ff0 2024-09-16T14:51:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf54004d90 con 0x7fbf5c138ff0 2024-09-16T14:51:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 10 monclient: dump: 2024-09-16T14:51:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.668 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf59ffb640 4 set_mon_vals no callback set 2024-09-16T14:51:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbf5400f800 con 0x7fbf5c138ff0 2024-09-16T14:51:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 169..257) v4 ==== 6745+0+0 (unknown 920221734 0 0) 0x7fbf54043ed0 con 0x7fbf5c138ff0 2024-09-16T14:51:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995524758 0 0) 0x7fbf5c137e80 con 0x7fbf5c137aa0 2024-09-16T14:51:29.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 10 monclient(hunting): my global_id is 184300 2024-09-16T14:51:29.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c09b680 con 0x7fbf5c137aa0 2024-09-16T14:51:29.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf5affd640 10 monclient: get_auth_request con 0x7fbf480363d0 auth_method 0 2024-09-16T14:51:29.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2600271813 0 0) 0x7fbf5c09b680 con 0x7fbf5c137aa0 2024-09-16T14:51:29.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf5c137e80 con 0x7fbf5c137aa0 2024-09-16T14:51:29.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf50001730 con 0x7fbf5c137aa0 2024-09-16T14:51:29.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:29.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 10 monclient: dump: 2024-09-16T14:51:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.670 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1433643554 0 0) 0x7fbf5c137e80 con 0x7fbf5c137aa0 2024-09-16T14:51:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf5c09f5b0 con 0x7fbf5c137aa0 2024-09-16T14:51:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fbf50001a80 con 0x7fbf5c137aa0 2024-09-16T14:51:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 >> v1:172.21.15.189:6789/0 conn(0x7fbf5c137aa0 legacy=0x7fbf5c1393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.649+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 >> v1:172.21.15.189:6789/0 conn(0x7fbf5c137aa0 legacy=0x7fbf5c1393d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:29.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf61a6d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:51:29.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf61a6d640 1 -- 172.21.15.26:0/159384771 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbf5c06d8d0 con 0x7fbf5c02c280 2024-09-16T14:51:29.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147806903 0 0) 0x7fbf5c0719c0 con 0x7fbf5c02c280 2024-09-16T14:51:29.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 10 monclient(hunting): my global_id is 184303 2024-09-16T14:51:29.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c137e80 con 0x7fbf5c02c280 2024-09-16T14:51:29.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 106058111 0 0) 0x7fbf5c137e80 con 0x7fbf5c02c280 2024-09-16T14:51:29.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf5c0719c0 con 0x7fbf5c02c280 2024-09-16T14:51:29.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf50003600 con 0x7fbf5c02c280 2024-09-16T14:51:29.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:29.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 10 monclient: dump: 2024-09-16T14:51:29.673 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.673 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.673 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.673 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.673 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.673 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.674 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.674 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.674 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.674 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3656058482 0 0) 0x7fbf5c0719c0 con 0x7fbf5c02c280 2024-09-16T14:51:29.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbf5c071f00 con 0x7fbf5c02c280 2024-09-16T14:51:29.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 4107814925) 0x7fbf500058c0 con 0x7fbf5c02c280 2024-09-16T14:51:29.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:51:29.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.657+0000 7fbf43fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:29.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.661+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 >> v1:172.21.15.189:6789/0 conn(0x7fbf5c02c280 legacy=0x7fbf5c139ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.661+0000 7fbf43fff640 1 -- 172.21.15.26:0/159384771 >> v1:172.21.15.189:6789/0 conn(0x7fbf5c02c280 legacy=0x7fbf5c139ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:29.675 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:51:29.675 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:51:29.675 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:51:29.675 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:51:29.675 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 84, 2024-09-16T14:51:29.675 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:51:29.675 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:51:29.675 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:51:29.675 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:51:29.675 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:29.676 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:51:29.676 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:29.676 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:51:29.676 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:51:29.676 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:29.676 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:29.676 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:29.676 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:29.676 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:29.676 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:29.676 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:29.676 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:29.677 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:29.677 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:29.677 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:29.677 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:51:29.677 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:51:29.677 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:29.677 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:29.677 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:29.677 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:29.677 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:29.677 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:29.677 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:29.677 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:29.678 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:29.678 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:29.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.678 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:51:29.678 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:51:29.678 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:51:29.678 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:51:29.678 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:51:29.678 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:51:29.678 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:29.678 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:29.678 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:51:29.679 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:51:29.679 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:51:29.679 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:51:29.679 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:51:29.679 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:51:29.679 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:51:29.679 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:29.679 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:51:29.679 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:29.679 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:29.679 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:29.679 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:29.679 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:29.680 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:29.680 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:29.680 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:29.680 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:29.680 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:29.680 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:29.680 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:51:29.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.680 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:51:29.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.680 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:51:29.680 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:51:29.680 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:29.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:29.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.681 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:29.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:51:29.681 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:29.681 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.681 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:51:29.681 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:51:29.681 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:29.681 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:29.681 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:29.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.682 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:51:29.682 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:51:29.682 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:29.682 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:29.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.682 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:29.682 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:51:29.682 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:29.682 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.682 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:51:29.683 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:51:29.683 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:29.683 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:29.683 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:29.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.683 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:51:29.683 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:51:29.683 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:29.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:29.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.684 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:29.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:51:29.684 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:29.684 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.684 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:51:29.684 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:51:29.684 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:29.684 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:29.684 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:29.684 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.684 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.685 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:51:29.685 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:51:29.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.685 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:29.685 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:29.685 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:51:29.685 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.685 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:29.685 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:51:29.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:29.685 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:29.685 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:29.686 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:51:29.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:29.686 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:29.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:29.686 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:51:29.686 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:51:29.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.661+0000 7fbf61a6d640 1 -- 172.21.15.26:0/159384771 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbf480363d0 legacy=0x7fbf48038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.661+0000 7fbf61a6d640 10 monclient: shutdown 2024-09-16T14:51:29.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.661+0000 7fbf61a6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:29.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.661+0000 7fbf61a6d640 1 -- 172.21.15.26:0/159384771 >> v1:172.21.15.26:6790/0 conn(0x7fbf5c138ff0 legacy=0x7fbf5c1d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.661+0000 7fbf5bfff640 1 -- 172.21.15.26:0/159384771 reap_dead start 2024-09-16T14:51:29.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.661+0000 7fbf5bfff640 1 -- 172.21.15.26:0/159384771 reap_dead start 2024-09-16T14:51:29.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.661+0000 7fbf61a6d640 1 -- 172.21.15.26:0/159384771 shutdown_connections 2024-09-16T14:51:29.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.661+0000 7fbf61a6d640 1 -- 172.21.15.26:0/159384771 >> 172.21.15.26:0/159384771 conn(0x7fbf5c09fb50 msgr2=0x7fbf5c0a1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:29.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.661+0000 7fbf61a6d640 1 -- 172.21.15.26:0/159384771 shutdown_connections 2024-09-16T14:51:29.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.661+0000 7fbf61a6d640 1 -- 172.21.15.26:0/159384771 wait complete. 2024-09-16T14:51:29.688 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:51:29.688 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:51:29.688 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:51:29.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.797+0000 7f09a0c98640 10 monclient: get_monmap_and_config 2024-09-16T14:51:29.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.797+0000 7f09a0c98640 10 monclient: build_initial_monmap 2024-09-16T14:51:29.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.797+0000 7f09a0c98640 10 monclient: monmap: 2024-09-16T14:51:29.799 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:29.799 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.799 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:29.800356+0000 2024-09-16T14:51:29.799 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:29.800356+0000 2024-09-16T14:51:29.799 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:29.800 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.800 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:29.800 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:29.800 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:29.800 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.797+0000 7f09a0c98640 1 Processor -- start 2024-09-16T14:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.797+0000 7f09a0c98640 1 -- start start 2024-09-16T14:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.797+0000 7f09a0c98640 10 monclient: init 2024-09-16T14:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 10 monclient(hunting): picked mon.noname-b con 0x7f099c09cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 10 monclient(hunting): picked mon.noname-a con 0x7f099c132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 10 monclient(hunting): picked mon.noname-c con 0x7f099c138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099c09d260 con 0x7f099c132650 2024-09-16T14:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099c132a80 con 0x7f099c138ff0 2024-09-16T14:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099c09cf60 con 0x7f099c09cb80 2024-09-16T14:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 10 monclient(hunting): authenticate will time out at 1513.920358s 2024-09-16T14:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f0999d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f099c132650 0x7f099c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56288/0 (socket says 172.21.15.26:56288) 2024-09-16T14:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f099c09cb80 0x7f099c0717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59786/0 (socket says 172.21.15.26:59786) 2024-09-16T14:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f0999d9b640 1 -- 172.21.15.26:0/2603973589 learned_addr learned my addr 172.21.15.26:0/2603973589 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 1 -- 172.21.15.26:0/2603973589 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142270839 0 0) 0x7f099c09d260 con 0x7f099c132650 2024-09-16T14:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient(hunting): my global_id is 194121 2024-09-16T14:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 1 -- 172.21.15.26:0/2603973589 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0988003530 con 0x7f099c132650 2024-09-16T14:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 1 -- 172.21.15.26:0/2603973589 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412288868 0 0) 0x7f099c09cf60 con 0x7f099c09cb80 2024-09-16T14:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient(hunting): my global_id is 155009 2024-09-16T14:51:29.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 1 -- 172.21.15.26:0/2603973589 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f099c09d260 con 0x7f099c09cb80 2024-09-16T14:51:29.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 1 -- 172.21.15.26:0/2603973589 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126588150 0 0) 0x7f099c132a80 con 0x7f099c138ff0 2024-09-16T14:51:29.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient(hunting): my global_id is 184306 2024-09-16T14:51:29.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 1 -- 172.21.15.26:0/2603973589 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f099c09cf60 con 0x7f099c138ff0 2024-09-16T14:51:29.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 1 -- 172.21.15.26:0/2603973589 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1513970414 0 0) 0x7f0988003530 con 0x7f099c132650 2024-09-16T14:51:29.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 1 -- 172.21.15.26:0/2603973589 >> v1:172.21.15.26:6790/0 conn(0x7f099c09cb80 legacy=0x7f099c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 1 -- 172.21.15.26:0/2603973589 >> v1:172.21.15.189:6789/0 conn(0x7f099c138ff0 legacy=0x7f099c13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient: _finish_hunting 0 2024-09-16T14:51:29.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 1 monclient: found mon.noname-a 2024-09-16T14:51:29.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:29.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:51:29.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 1 -- 172.21.15.26:0/2603973589 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099c09f2d0 con 0x7f099c132650 2024-09-16T14:51:29.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient: _finish_auth 0 2024-09-16T14:51:29.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient: _check_auth_tickets 2024-09-16T14:51:29.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:29.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 1 -- 172.21.15.26:0/2603973589 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0984001620 con 0x7f099c132650 2024-09-16T14:51:29.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 5 monclient: authenticate success, global_id 194121 2024-09-16T14:51:29.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:29.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:29.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient: dump: 2024-09-16T14:51:29.810 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.810 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.811 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 1 -- 172.21.15.26:0/2603973589 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0984001810 con 0x7f099c132650 2024-09-16T14:51:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 1 -- 172.21.15.26:0/2603973589 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0984001be0 con 0x7f099c132650 2024-09-16T14:51:29.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 4 set_mon_vals no callback set 2024-09-16T14:51:29.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:29.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f099959a640 10 monclient: dump: 2024-09-16T14:51:29.812 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.812 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.812 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.812 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.812 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.812 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.812 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.812 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.812 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.812 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 10 monclient: shutdown 2024-09-16T14:51:29.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:29.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 1 -- 172.21.15.26:0/2603973589 >> v1:172.21.15.26:6789/0 conn(0x7f099c132650 legacy=0x7f099c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 1 -- 172.21.15.26:0/2603973589 shutdown_connections 2024-09-16T14:51:29.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 1 -- 172.21.15.26:0/2603973589 >> 172.21.15.26:0/2603973589 conn(0x7f099c09fb50 msgr2=0x7f099c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:29.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 1 -- 172.21.15.26:0/2603973589 shutdown_connections 2024-09-16T14:51:29.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 1 -- 172.21.15.26:0/2603973589 wait complete. 2024-09-16T14:51:29.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 10 monclient: build_initial_monmap 2024-09-16T14:51:29.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 10 monclient: monmap: 2024-09-16T14:51:29.813 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:29.813 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:29.813 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:29.813 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:29.814 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 1 Processor -- start 2024-09-16T14:51:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 1 -- start start 2024-09-16T14:51:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 10 monclient: init 2024-09-16T14:51:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:51:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 10 monclient(hunting): picked mon.noname-c con 0x7f099c09cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.801+0000 7f09a0c98640 10 monclient(hunting): picked mon.noname-a con 0x7f099c132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:29.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 10 monclient(hunting): picked mon.noname-b con 0x7f099c138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099c0a2020 con 0x7f099c132650 2024-09-16T14:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099c138130 con 0x7f099c138ff0 2024-09-16T14:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099c06fdb0 con 0x7f099c09cb80 2024-09-16T14:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 10 monclient(hunting): authenticate will time out at 1513.924078s 2024-09-16T14:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0999d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f099c132650 0x7f099c06dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56302/0 (socket says 172.21.15.26:56302) 2024-09-16T14:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0999d9b640 1 -- 172.21.15.26:0/3085689337 learned_addr learned my addr 172.21.15.26:0/3085689337 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097393504 0 0) 0x7f099c0a2020 con 0x7f099c132650 2024-09-16T14:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): my global_id is 194124 2024-09-16T14:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f097c003530 con 0x7f099c132650 2024-09-16T14:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703286372 0 0) 0x7f099c06fdb0 con 0x7f099c09cb80 2024-09-16T14:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): my global_id is 155012 2024-09-16T14:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f099c0a2020 con 0x7f099c09cb80 2024-09-16T14:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1960065200 0 0) 0x7f097c003530 con 0x7f099c132650 2024-09-16T14:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f099c06fdb0 con 0x7f099c132650 2024-09-16T14:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0984001520 con 0x7f099c132650 2024-09-16T14:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): dump: 2024-09-16T14:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4073219967 0 0) 0x7f099c0a2020 con 0x7f099c09cb80 2024-09-16T14:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f097c003530 con 0x7f099c09cb80 2024-09-16T14:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0994001ba0 con 0x7f099c09cb80 2024-09-16T14:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:29.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): dump: 2024-09-16T14:51:29.818 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.818 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.818 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.818 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.818 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.818 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.818 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.818 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.818 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.818 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815815565 0 0) 0x7f099c138130 con 0x7f099c138ff0 2024-09-16T14:51:29.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:51:29.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient(hunting): my global_id is 184309 2024-09-16T14:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f099c0a2020 con 0x7f099c138ff0 2024-09-16T14:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2159916195 0 0) 0x7f099c06fdb0 con 0x7f099c132650 2024-09-16T14:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 >> v1:172.21.15.26:6790/0 conn(0x7f099c09cb80 legacy=0x7f099c06d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 >> v1:172.21.15.189:6789/0 conn(0x7f099c138ff0 legacy=0x7f099c1cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 monclient: found mon.a 2024-09-16T14:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099c13c400 con 0x7f099c132650 2024-09-16T14:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient: _finish_auth 0 2024-09-16T14:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 5 monclient: authenticate success, global_id 194124 2024-09-16T14:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 10 monclient: _renew_subs 2024-09-16T14:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 1 -- 172.21.15.26:0/3085689337 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f099c0712c0 con 0x7f099c132650 2024-09-16T14:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 10 monclient: _renew_subs 2024-09-16T14:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 1 -- 172.21.15.26:0/3085689337 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f099c06f1e0 con 0x7f099c132650 2024-09-16T14:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0984004b90 con 0x7f099c132650 2024-09-16T14:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0984004ea0 con 0x7f099c132650 2024-09-16T14:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0998d99640 4 set_mon_vals no callback set 2024-09-16T14:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 10 monclient: dump: 2024-09-16T14:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0984005000 con 0x7f099c132650 2024-09-16T14:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f09a0c98640 1 -- 172.21.15.26:0/3085689337 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0980005050 con 0x7f099c132650 2024-09-16T14:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 169..257) v4 ==== 6745+0+0 (unknown 920221734 0 0) 0x7f0984043f50 con 0x7f099c132650 2024-09-16T14:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.805+0000 7f099a59c640 10 monclient: get_auth_request con 0x7f097c036440 auth_method 0 2024-09-16T14:51:29.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.809+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0984006450 con 0x7f099c132650 2024-09-16T14:51:29.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.809+0000 7f0992ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:29.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:29.809+0000 7f0992ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:30.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:30.025+0000 7f09a0c98640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:51:30.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:30.025+0000 7f09a0c98640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:51:30.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:30.025+0000 7f09a0c98640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:30.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:30.025+0000 7f09a0c98640 1 -- 172.21.15.26:0/3085689337 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f0980005b10 con 0x7f099c132650 2024-09-16T14:51:30.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:30.029+0000 7f0992ffd640 1 -- 172.21.15.26:0/3085689337 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f0984012dc0 con 0x7f099c132650 2024-09-16T14:51:30.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:30.029+0000 7f0992ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:51:30.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:30.029+0000 7f0992ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:30.031 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.029+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.031 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.029+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=1,health=10,kv=11,logm=49} crc {auth=1351526299,config=3678701417,health=4162882277,kv=1867599010,logm=3635531510}) 2024-09-16T14:51:30.031 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.029+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=361214666,config=692619927,health=3005501367,kv=1367698,logm=111452205}) 2024-09-16T14:51:30.031 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.029+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.031 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.029+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=1,health=10,kv=11,logm=49} crc {auth=1351526299,config=3678701417,health=4162882277,kv=1867599010,logm=3635531510}) 2024-09-16T14:51:30.031 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.029+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=1456737775,config=692619927,health=336713944,kv=1867599010,logm=2780044935}) 2024-09-16T14:51:30.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:30.033+0000 7f09a0c98640 1 -- 172.21.15.26:0/3085689337 >> v1:172.21.15.189:6812/3196161582 conn(0x7f097c036440 legacy=0x7f097c038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:30.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:30.033+0000 7f09a0c98640 10 monclient: shutdown 2024-09-16T14:51:30.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:30.033+0000 7f09a0c98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:30.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:30.033+0000 7f09a0c98640 1 -- 172.21.15.26:0/3085689337 >> v1:172.21.15.26:6789/0 conn(0x7f099c132650 legacy=0x7f099c06dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:30.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:30.033+0000 7f09a0c98640 1 -- 172.21.15.26:0/3085689337 shutdown_connections 2024-09-16T14:51:30.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:30.033+0000 7f09a0c98640 1 -- 172.21.15.26:0/3085689337 >> 172.21.15.26:0/3085689337 conn(0x7f099c09fb50 msgr2=0x7f099c0a1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:30.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:30.033+0000 7f09a0c98640 1 -- 172.21.15.26:0/3085689337 shutdown_connections 2024-09-16T14:51:30.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:30.033+0000 7f09a0c98640 1 -- 172.21.15.26:0/3085689337 wait complete. 2024-09-16T14:51:30.033 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.033+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.033 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.033+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2484003157}) 2024-09-16T14:51:30.033 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.033+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234357124}) 2024-09-16T14:51:30.034 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.033+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.034 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.033+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2484003157}) 2024-09-16T14:51:30.034 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.033+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1650490682}) 2024-09-16T14:51:30.035 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.033+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.035 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.033+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1857174908}) 2024-09-16T14:51:30.036 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.033+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=324721142}) 2024-09-16T14:51:30.036 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.033+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.036 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.033+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1857174908}) 2024-09-16T14:51:30.036 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.033+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2969263258}) 2024-09-16T14:51:30.037 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.037+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.038 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.037+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=106022618}) 2024-09-16T14:51:30.038 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.037+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2004764387}) 2024-09-16T14:51:30.038 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.037+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.038 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.037+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=106022618}) 2024-09-16T14:51:30.038 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.037+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=714536674}) 2024-09-16T14:51:30.039 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.037+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.039 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.037+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=357007574}) 2024-09-16T14:51:30.039 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.037+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266458630}) 2024-09-16T14:51:30.040 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.037+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.040 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.037+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=357007574}) 2024-09-16T14:51:30.040 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.037+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3882819036}) 2024-09-16T14:51:30.041 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.041+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.041 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.041+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=733339973}) 2024-09-16T14:51:30.042 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.041+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1934868891}) 2024-09-16T14:51:30.042 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.041+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.042 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.041+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=733339973}) 2024-09-16T14:51:30.042 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.041+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3476913996}) 2024-09-16T14:51:30.044 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.041+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.044 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.041+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3185796438}) 2024-09-16T14:51:30.044 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.041+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3825496531}) 2024-09-16T14:51:30.044 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.041+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.044 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.041+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3185796438}) 2024-09-16T14:51:30.044 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.041+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3315614138}) 2024-09-16T14:51:30.046 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.045+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.046 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.045+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2717228626}) 2024-09-16T14:51:30.046 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.045+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846838000}) 2024-09-16T14:51:30.046 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.045+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.046 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.045+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2717228626}) 2024-09-16T14:51:30.046 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.045+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544108378}) 2024-09-16T14:51:30.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.049+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.049+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3673300476}) 2024-09-16T14:51:30.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.049+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3102554508}) 2024-09-16T14:51:30.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.049+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.049+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3673300476}) 2024-09-16T14:51:30.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.049+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3386203005}) 2024-09-16T14:51:30.050 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:51:30.050 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.049+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.050 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.049+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=454158812}) 2024-09-16T14:51:30.050 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.049+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464483815}) 2024-09-16T14:51:30.051 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.049+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.051 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.049+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=454158812}) 2024-09-16T14:51:30.051 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.049+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2626913608}) 2024-09-16T14:51:30.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.053+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.053+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=107704443}) 2024-09-16T14:51:30.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.053+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1602914224}) 2024-09-16T14:51:30.053 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.053+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.053 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.053+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=107704443}) 2024-09-16T14:51:30.053 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.053+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872740527}) 2024-09-16T14:51:30.054 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.053+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.054 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.053+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2797080016}) 2024-09-16T14:51:30.054 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.053+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3377353059}) 2024-09-16T14:51:30.054 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.053+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.053+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2797080016}) 2024-09-16T14:51:30.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.053+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3489434837}) 2024-09-16T14:51:30.056 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.057+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.056 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.057+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3530303319}) 2024-09-16T14:51:30.056 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.057+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1688401992}) 2024-09-16T14:51:30.056 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.057+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.056 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.057+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3530303319}) 2024-09-16T14:51:30.057 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.057+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3099414333}) 2024-09-16T14:51:30.058 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.057+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.058 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.057+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1218274484}) 2024-09-16T14:51:30.058 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.057+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1495472857}) 2024-09-16T14:51:30.058 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.057+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.058 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.057+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1218274484}) 2024-09-16T14:51:30.058 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.057+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1996283221}) 2024-09-16T14:51:30.060 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.057+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.060 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.057+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1589939716}) 2024-09-16T14:51:30.060 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.057+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461388671}) 2024-09-16T14:51:30.060 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.061+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.060 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.061+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1589939716}) 2024-09-16T14:51:30.060 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.061+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1661565022}) 2024-09-16T14:51:30.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.061+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.061+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1520175753}) 2024-09-16T14:51:30.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.061+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=134512504}) 2024-09-16T14:51:30.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.061+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.061+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1520175753}) 2024-09-16T14:51:30.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.061+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=450738795}) 2024-09-16T14:51:30.064 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.061+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.064 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.061+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=720378866}) 2024-09-16T14:51:30.064 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.061+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=475922057}) 2024-09-16T14:51:30.064 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.061+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.064 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.061+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=720378866}) 2024-09-16T14:51:30.064 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.061+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1304576581}) 2024-09-16T14:51:30.065 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.065+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.065+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3873413613}) 2024-09-16T14:51:30.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.065+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1382117315}) 2024-09-16T14:51:30.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.065+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.065+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3873413613}) 2024-09-16T14:51:30.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.065+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2052306305}) 2024-09-16T14:51:30.067 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.065+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.067 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.065+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4032053722}) 2024-09-16T14:51:30.067 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.065+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1960119740}) 2024-09-16T14:51:30.068 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.065+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.068 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.065+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4032053722}) 2024-09-16T14:51:30.068 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.065+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3806744974}) 2024-09-16T14:51:30.069 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.069+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.069 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.069+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=81206223}) 2024-09-16T14:51:30.069 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.069+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2599010476}) 2024-09-16T14:51:30.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.069+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.069+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=81206223}) 2024-09-16T14:51:30.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.069+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3061452789}) 2024-09-16T14:51:30.071 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.069+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.071 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.069+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1039065761}) 2024-09-16T14:51:30.071 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.069+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2126302481}) 2024-09-16T14:51:30.071 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.069+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.071 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.069+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1039065761}) 2024-09-16T14:51:30.071 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.069+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3141177239}) 2024-09-16T14:51:30.073 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.073+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.073 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.073+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669458977}) 2024-09-16T14:51:30.073 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.073+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=658370057}) 2024-09-16T14:51:30.073 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.073+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.073 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.073+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669458977}) 2024-09-16T14:51:30.073 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.073+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1624003873}) 2024-09-16T14:51:30.075 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.073+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.075 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.073+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2479336212}) 2024-09-16T14:51:30.075 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.073+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1094418927}) 2024-09-16T14:51:30.075 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.073+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.075 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.073+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2479336212}) 2024-09-16T14:51:30.075 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.073+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4054783452}) 2024-09-16T14:51:30.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.077+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.077+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3577799119}) 2024-09-16T14:51:30.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.077+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3690398851}) 2024-09-16T14:51:30.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.077+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.077+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3577799119}) 2024-09-16T14:51:30.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.077+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1002934897}) 2024-09-16T14:51:30.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.077+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.077+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2864334417}) 2024-09-16T14:51:30.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.077+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2463754563}) 2024-09-16T14:51:30.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.077+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.077+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2864334417}) 2024-09-16T14:51:30.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.077+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2722950645}) 2024-09-16T14:51:30.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.081+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.081+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2651320517}) 2024-09-16T14:51:30.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.081+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293627631}) 2024-09-16T14:51:30.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.081+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.081+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2651320517}) 2024-09-16T14:51:30.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.081+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4100665809}) 2024-09-16T14:51:30.083 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.081+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.083 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.081+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=562347196}) 2024-09-16T14:51:30.083 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.081+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2422991883}) 2024-09-16T14:51:30.083 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.081+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.083 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.081+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=562347196}) 2024-09-16T14:51:30.083 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.081+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3094284397}) 2024-09-16T14:51:30.084 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.085+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.085 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.085+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2025008745}) 2024-09-16T14:51:30.085 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.085+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779126673}) 2024-09-16T14:51:30.085 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.085+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.085 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.085+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2025008745}) 2024-09-16T14:51:30.085 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.085+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1488874184}) 2024-09-16T14:51:30.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.085+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.085+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=154564397}) 2024-09-16T14:51:30.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.085+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1722547366}) 2024-09-16T14:51:30.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.085+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.085+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=154564397}) 2024-09-16T14:51:30.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.085+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3724871339}) 2024-09-16T14:51:30.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1557328789}) 2024-09-16T14:51:30.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=506663992}) 2024-09-16T14:51:30.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1557328789}) 2024-09-16T14:51:30.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3363478778}) 2024-09-16T14:51:30.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2256342997}) 2024-09-16T14:51:30.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426006194}) 2024-09-16T14:51:30.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2256342997}) 2024-09-16T14:51:30.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3549185463}) 2024-09-16T14:51:30.091 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.091 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=970664479}) 2024-09-16T14:51:30.092 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4075997058}) 2024-09-16T14:51:30.092 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.092 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=970664479}) 2024-09-16T14:51:30.092 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.089+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4037130150}) 2024-09-16T14:51:30.093 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.093+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.093 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.093+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4188812899}) 2024-09-16T14:51:30.093 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.093+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980990578}) 2024-09-16T14:51:30.093 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.093+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.093 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.093+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4188812899}) 2024-09-16T14:51:30.094 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.093+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4161040662}) 2024-09-16T14:51:30.095 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.093+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.095 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.093+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2438329140}) 2024-09-16T14:51:30.095 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.093+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2901077022}) 2024-09-16T14:51:30.095 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.093+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.095 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.093+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2438329140}) 2024-09-16T14:51:30.095 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.093+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3523086745}) 2024-09-16T14:51:30.096 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.097+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.097 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.097+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2266929890}) 2024-09-16T14:51:30.097 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.097+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3412941415}) 2024-09-16T14:51:30.097 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.097+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.097 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.097+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2266929890}) 2024-09-16T14:51:30.097 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.097+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3105365328}) 2024-09-16T14:51:30.098 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.097+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.098 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.097+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3199143103}) 2024-09-16T14:51:30.098 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.097+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=72197564}) 2024-09-16T14:51:30.098 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.097+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.098 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.097+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3199143103}) 2024-09-16T14:51:30.099 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.097+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4077789378}) 2024-09-16T14:51:30.100 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.100 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4214448911}) 2024-09-16T14:51:30.100 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4191825022}) 2024-09-16T14:51:30.100 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.100 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4214448911}) 2024-09-16T14:51:30.100 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=843134236}) 2024-09-16T14:51:30.102 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.102 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2542415559}) 2024-09-16T14:51:30.102 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20253227}) 2024-09-16T14:51:30.102 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.102 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2542415559}) 2024-09-16T14:51:30.102 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2566724132}) 2024-09-16T14:51:30.104 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.104 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1125478917}) 2024-09-16T14:51:30.104 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=195551629}) 2024-09-16T14:51:30.104 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.104 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1125478917}) 2024-09-16T14:51:30.104 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.101+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1252354162}) 2024-09-16T14:51:30.105 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.105+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.105 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.105+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2135418155}) 2024-09-16T14:51:30.106 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.105+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=861308900}) 2024-09-16T14:51:30.106 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.105+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.106 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.105+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2135418155}) 2024-09-16T14:51:30.106 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.105+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3809697279}) 2024-09-16T14:51:30.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.105+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.105+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018322582}) 2024-09-16T14:51:30.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.105+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=246986557}) 2024-09-16T14:51:30.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.105+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.108 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.105+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018322582}) 2024-09-16T14:51:30.108 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.105+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4006606803}) 2024-09-16T14:51:30.109 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.109+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.109 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.109+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2147858020}) 2024-09-16T14:51:30.109 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.109+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2154223202}) 2024-09-16T14:51:30.109 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.109+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.109 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.109+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2147858020}) 2024-09-16T14:51:30.109 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.109+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=405996578}) 2024-09-16T14:51:30.110 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.109+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.110 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.109+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027200278}) 2024-09-16T14:51:30.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.109+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829721660}) 2024-09-16T14:51:30.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.109+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.109+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027200278}) 2024-09-16T14:51:30.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.109+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=149843772}) 2024-09-16T14:51:30.112 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.113+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.112 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.113+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3396617999}) 2024-09-16T14:51:30.113 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.113+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806007692}) 2024-09-16T14:51:30.113 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.113+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.113 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.113+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3396617999}) 2024-09-16T14:51:30.113 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.113+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1924988635}) 2024-09-16T14:51:30.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.113+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.113+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=269711226}) 2024-09-16T14:51:30.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.113+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2840887100}) 2024-09-16T14:51:30.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.113+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.113+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=269711226}) 2024-09-16T14:51:30.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.113+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3159606823}) 2024-09-16T14:51:30.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1609520172}) 2024-09-16T14:51:30.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=63403129}) 2024-09-16T14:51:30.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1609520172}) 2024-09-16T14:51:30.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2513760198}) 2024-09-16T14:51:30.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=796560895}) 2024-09-16T14:51:30.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=601650624}) 2024-09-16T14:51:30.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=796560895}) 2024-09-16T14:51:30.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2930030044}) 2024-09-16T14:51:30.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969656787}) 2024-09-16T14:51:30.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268060722}) 2024-09-16T14:51:30.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969656787}) 2024-09-16T14:51:30.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.117+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=546323876}) 2024-09-16T14:51:30.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.121+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.121+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2704140270}) 2024-09-16T14:51:30.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.121+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=353311429}) 2024-09-16T14:51:30.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.121+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.121+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2704140270}) 2024-09-16T14:51:30.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.121+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2310569601}) 2024-09-16T14:51:30.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.121+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.121+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3432654566}) 2024-09-16T14:51:30.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.121+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3087579053}) 2024-09-16T14:51:30.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.121+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.121+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3432654566}) 2024-09-16T14:51:30.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.121+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131443486}) 2024-09-16T14:51:30.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.125+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.125+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=54,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=1} crc {logm=1184243951,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2758631918,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=359867973}) 2024-09-16T14:51:30.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.125+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=58,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=9,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=1} crc {logm=4276138188,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2133452423,mon_config_key=1000370824,monmap=970648238,osd_metadata=2528375958,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3113166310}) 2024-09-16T14:51:30.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.125+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.125+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=54,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=1} crc {logm=1184243951,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2758631918,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=359867973}) 2024-09-16T14:51:30.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.125+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=56,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=1} crc {logm=1017303124,mdsmap=1531151339,mgr=2578290774,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2278275023,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3113166310}) 2024-09-16T14:51:30.129 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.129+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.129 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.129+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2229879720}) 2024-09-16T14:51:30.129 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.129+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2417737860}) 2024-09-16T14:51:30.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.129+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.129+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2229879720}) 2024-09-16T14:51:30.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.129+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1678816789}) 2024-09-16T14:51:30.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.145+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:51:30.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.145+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=44} crc {osdmap=652293568}) 2024-09-16T14:51:30.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:51:30.145+0000 7faec309f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=41} crc {osdmap=1903859534}) 2024-09-16T14:51:30.155 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T14:51:30.155 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T14:51:30.155 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:51:30.155 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph auth del client.bazar 2024-09-16T14:51:30.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25c89a640 10 monclient: get_monmap_and_config 2024-09-16T14:51:30.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25c89a640 10 monclient: build_initial_monmap 2024-09-16T14:51:30.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25c89a640 10 monclient: monmap: 2024-09-16T14:51:30.269 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:30.269 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:30.269 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:51:30.271220+0000 2024-09-16T14:51:30.269 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:51:30.271220+0000 2024-09-16T14:51:30.270 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:30.270 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:30.270 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:30.270 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:30.270 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:30.270 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:30.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25c89a640 1 Processor -- start 2024-09-16T14:51:30.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25c89a640 1 -- start start 2024-09-16T14:51:30.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25c89a640 10 monclient: init 2024-09-16T14:51:30.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25c89a640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:30.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25c89a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:51:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25c89a640 10 monclient(hunting): picked mon.noname-a con 0x7ff25809df10 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25c89a640 10 monclient(hunting): picked mon.noname-b con 0x7ff25809e2f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25c89a640 10 monclient(hunting): picked mon.noname-c con 0x7ff25809e740 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25c89a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25809ee20 con 0x7ff25809df10 2024-09-16T14:51:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25c89a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25809f100 con 0x7ff25809e740 2024-09-16T14:51:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25c89a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25809f3e0 con 0x7ff25809e2f0 2024-09-16T14:51:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25c89a640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25c89a640 10 monclient(hunting): authenticate will time out at 1472.387150s 2024-09-16T14:51:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff256d9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff25809e740 0x7ff25813b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:41598/0 (socket says 172.21.15.189:41598) 2024-09-16T14:51:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff256d9d640 1 -- 172.21.15.189:0/1595553020 learned_addr learned my addr 172.21.15.189:0/1595553020 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 1 -- 172.21.15.189:0/1595553020 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361980454 0 0) 0x7ff25809f100 con 0x7ff25809e740 2024-09-16T14:51:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 10 monclient(hunting): _negotiate 2024-09-16T14:51:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 10 monclient(hunting): my global_id is 184315 2024-09-16T14:51:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 1 -- 172.21.15.189:0/1595553020 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff240003530 con 0x7ff25809e740 2024-09-16T14:51:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 1 -- 172.21.15.189:0/1595553020 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485142991 0 0) 0x7ff25809f3e0 con 0x7ff25809e2f0 2024-09-16T14:51:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 10 monclient(hunting): _negotiate 2024-09-16T14:51:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 10 monclient(hunting): my global_id is 155018 2024-09-16T14:51:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 1 -- 172.21.15.189:0/1595553020 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff25809f100 con 0x7ff25809e2f0 2024-09-16T14:51:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 1 -- 172.21.15.189:0/1595553020 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3595275653 0 0) 0x7ff240003530 con 0x7ff25809e740 2024-09-16T14:51:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 1 -- 172.21.15.189:0/1595553020 >> v1:172.21.15.26:6790/0 conn(0x7ff25809e2f0 legacy=0x7ff2581378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 1 -- 172.21.15.189:0/1595553020 >> v1:172.21.15.26:6789/0 conn(0x7ff25809df10 legacy=0x7ff25809d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 10 monclient: _finish_hunting 0 2024-09-16T14:51:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 1 monclient: found mon.noname-c 2024-09-16T14:51:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:51:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 1 -- 172.21.15.189:0/1595553020 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25809c690 con 0x7ff25809e740 2024-09-16T14:51:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 10 monclient: _finish_auth 0 2024-09-16T14:51:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 10 monclient: _check_auth_tickets 2024-09-16T14:51:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.267+0000 7ff25559a640 1 -- 172.21.15.189:0/1595553020 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff250001920 con 0x7ff25809e740 2024-09-16T14:51:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:30.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25559a640 10 monclient: dump: 2024-09-16T14:51:30.280 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:30.280 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:30.280 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:30.280 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:30.280 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:30.280 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:30.280 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:30.280 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:30.280 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:30.280 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:30.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 5 monclient: authenticate success, global_id 184315 2024-09-16T14:51:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25559a640 1 -- 172.21.15.189:0/1595553020 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff250001b40 con 0x7ff25809e740 2024-09-16T14:51:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25559a640 1 -- 172.21.15.189:0/1595553020 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff250001ed0 con 0x7ff25809e740 2024-09-16T14:51:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25559a640 10 monclient: dump: 2024-09-16T14:51:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:30.282 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 4 set_mon_vals no callback set 2024-09-16T14:51:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient: shutdown 2024-09-16T14:51:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 1 -- 172.21.15.189:0/1595553020 >> v1:172.21.15.189:6789/0 conn(0x7ff25809e740 legacy=0x7ff25813b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 1 -- 172.21.15.189:0/1595553020 shutdown_connections 2024-09-16T14:51:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 1 -- 172.21.15.189:0/1595553020 >> 172.21.15.189:0/1595553020 conn(0x7ff2580a3320 msgr2=0x7ff25809f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 1 -- 172.21.15.189:0/1595553020 shutdown_connections 2024-09-16T14:51:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 1 -- 172.21.15.189:0/1595553020 wait complete. 2024-09-16T14:51:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient: build_initial_monmap 2024-09-16T14:51:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient: monmap: 2024-09-16T14:51:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:51:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:51:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:51:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:51:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:30.284 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 1 Processor -- start 2024-09-16T14:51:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 1 -- start start 2024-09-16T14:51:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient: init 2024-09-16T14:51:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:51:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient(hunting): picked mon.noname-b con 0x7ff25809df10 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient(hunting): picked mon.noname-a con 0x7ff25809e2f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient(hunting): picked mon.noname-c con 0x7ff25809e740 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25813afc0 con 0x7ff25809e2f0 2024-09-16T14:51:30.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25813b580 con 0x7ff25809df10 2024-09-16T14:51:30.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25813b860 con 0x7ff25809e740 2024-09-16T14:51:30.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:30.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient(hunting): authenticate will time out at 1472.390159s 2024-09-16T14:51:30.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25659c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff25809df10 0x7ff25809a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:41602/0 (socket says 172.21.15.189:41602) 2024-09-16T14:51:30.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25659c640 1 -- 172.21.15.189:0/2529486167 learned_addr learned my addr 172.21.15.189:0/2529486167 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:51:30.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709581231 0 0) 0x7ff25813b580 con 0x7ff25809df10 2024-09-16T14:51:30.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:51:30.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:30.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:30.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient(hunting): my global_id is 184318 2024-09-16T14:51:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff238003530 con 0x7ff25809df10 2024-09-16T14:51:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3197775863 0 0) 0x7ff238003530 con 0x7ff25809df10 2024-09-16T14:51:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff25813b580 con 0x7ff25809df10 2024-09-16T14:51:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff24c001980 con 0x7ff25809df10 2024-09-16T14:51:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient(hunting): dump: 2024-09-16T14:51:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:30.287 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609537866 0 0) 0x7ff25813b860 con 0x7ff25809e740 2024-09-16T14:51:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:51:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient(hunting): my global_id is 155021 2024-09-16T14:51:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff238003530 con 0x7ff25809e740 2024-09-16T14:51:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2508191333 0 0) 0x7ff25813b580 con 0x7ff25809df10 2024-09-16T14:51:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 >> v1:172.21.15.26:6790/0 conn(0x7ff25809e740 legacy=0x7ff2581d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 >> v1:172.21.15.26:6789/0 conn(0x7ff25809e2f0 legacy=0x7ff25809a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:51:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 1 monclient: found mon.b 2024-09-16T14:51:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25809b090 con 0x7ff25809df10 2024-09-16T14:51:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient: _finish_auth 0 2024-09-16T14:51:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:51:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 5 monclient: authenticate success, global_id 184318 2024-09-16T14:51:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient: _renew_subs 2024-09-16T14:51:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 1 -- 172.21.15.189:0/2529486167 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25806d020 con 0x7ff25809df10 2024-09-16T14:51:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient: _renew_subs 2024-09-16T14:51:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 1 -- 172.21.15.189:0/2529486167 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25809be20 con 0x7ff25809df10 2024-09-16T14:51:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff24c001d50 con 0x7ff25809df10 2024-09-16T14:51:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff24c003af0 con 0x7ff25809df10 2024-09-16T14:51:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 10 monclient: dump: 2024-09-16T14:51:30.290 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:51:30.290 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:30.290 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:30.290 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:30.290 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:30.290 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:51:30.290 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:30.290 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:30.290 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:30.290 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:51:30.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff254d99640 4 set_mon_vals no callback set 2024-09-16T14:51:30.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:30.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:30.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:30.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff24c003c50 con 0x7ff25809df10 2024-09-16T14:51:30.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.271+0000 7ff25c89a640 1 -- 172.21.15.189:0/2529486167 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff23c005050 con 0x7ff25809df10 2024-09-16T14:51:30.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.275+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(257..257 src has 169..257) v4 ==== 6745+0+0 (unknown 920221734 0 0) 0x7ff24c043f50 con 0x7ff25809df10 2024-09-16T14:51:30.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.275+0000 7ff255d9b640 10 monclient: get_auth_request con 0x7ff2380360c0 auth_method 0 2024-09-16T14:51:30.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.279+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff24c014830 con 0x7ff25809df10 2024-09-16T14:51:30.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.279+0000 7ff246ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:30.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.279+0000 7ff246ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:30.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.503+0000 7ff25c89a640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T14:51:30.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.503+0000 7ff25c89a640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T14:51:30.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.503+0000 7ff25c89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:30.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.503+0000 7ff25c89a640 1 -- 172.21.15.189:0/2529486167 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.bazar"} v 0) v1 -- 0x7ff23c005300 con 0x7ff25809df10 2024-09-16T14:51:30.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.511+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bazar"}]=0 v25) v1 ==== 82+0+0 (unknown 3744192161 0 0) 0x7ff24c012ca0 con 0x7ff25809df10 2024-09-16T14:51:30.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.511+0000 7ff246ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T14:51:30.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.511+0000 7ff246ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:30.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.655+0000 7ff25c89a640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T14:51:30.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.655+0000 7ff25c89a640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T14:51:30.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.655+0000 7ff25c89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:51:30.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.655+0000 7ff25c89a640 1 -- 172.21.15.189:0/2529486167 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.bazar"} v 0) v1 -- 0x7ff23c005850 con 0x7ff25809df10 2024-09-16T14:51:30.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.655+0000 7ff246ffd640 1 -- 172.21.15.189:0/2529486167 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bazar"}]=0 v25) v1 ==== 82+0+0 (unknown 3744192161 0 0) 0x7ff24c014420 con 0x7ff25809df10 2024-09-16T14:51:30.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.655+0000 7ff246ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T14:51:30.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.655+0000 7ff246ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:51:30.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.659+0000 7ff25c89a640 1 -- 172.21.15.189:0/2529486167 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff2380360c0 legacy=0x7ff238038580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:30.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.659+0000 7ff25c89a640 10 monclient: shutdown 2024-09-16T14:51:30.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.659+0000 7ff25c89a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:30.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.659+0000 7ff25c89a640 1 -- 172.21.15.189:0/2529486167 >> v1:172.21.15.189:6789/0 conn(0x7ff25809df10 legacy=0x7ff25809a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:30.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.659+0000 7ff25c89a640 1 -- 172.21.15.189:0/2529486167 shutdown_connections 2024-09-16T14:51:30.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.659+0000 7ff25c89a640 1 -- 172.21.15.189:0/2529486167 >> 172.21.15.189:0/2529486167 conn(0x7ff2580a3320 msgr2=0x7ff258070060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:30.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.659+0000 7ff25c89a640 1 -- 172.21.15.189:0/2529486167 shutdown_connections 2024-09-16T14:51:30.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:51:30.659+0000 7ff25c89a640 1 -- 172.21.15.189:0/2529486167 wait complete. 2024-09-16T14:51:30.674 INFO:tasks.workunit.client.0.smithi189.stderr:+ c=''\''allow command "auth ls", allow command quorum_status'\''' 2024-09-16T14:51:30.674 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph auth get-or-create client.foo mon '\''allow command "auth ls", allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.foo.keyring' 0 2024-09-16T14:51:30.675 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph auth get-or-create client.foo mon '\''allow command "auth ls", allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.foo.keyring' 2024-09-16T14:51:30.675 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=0 2024-09-16T14:51:30.675 INFO:tasks.workunit.client.0.smithi189.stdout:ceph auth get-or-create client.foo mon 'allow command "auth ls", allow command quorum_status' > /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-16T14:51:30.675 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph auth get-or-create client.foo mon ''\''allow' command '"auth' 'ls",' allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-16T14:51:30.675 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph auth get-or-create client.foo mon ''\''allow' command '"auth' 'ls",' allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-16T14:51:31.051 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:51:31.052 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 10 monclient: get_monmap_and_config 2024-09-16T14:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 10 monclient: build_initial_monmap 2024-09-16T14:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 10 monclient: monmap: 2024-09-16T14:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:31.186449+0000 2024-09-16T14:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:31.186449+0000 2024-09-16T14:51:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:31.186 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 1 Processor -- start 2024-09-16T14:51:31.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 1 -- start start 2024-09-16T14:51:31.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 10 monclient: init 2024-09-16T14:51:31.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:31.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:51:31.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 10 monclient(hunting): picked mon.noname-c con 0x7f3f1809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:31.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 10 monclient(hunting): picked mon.noname-b con 0x7f3f18132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:31.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 10 monclient(hunting): picked mon.noname-a con 0x7f3f18138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:31.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1809d260 con 0x7f3f18138ff0 2024-09-16T14:51:31.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f18132a80 con 0x7f3f1809cb80 2024-09-16T14:51:31.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1809cf60 con 0x7f3f18132650 2024-09-16T14:51:31.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:31.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 10 monclient(hunting): authenticate will time out at 1515.306426s 2024-09-16T14:51:31.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f17fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3f18132650 0x7f3f181378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59802/0 (socket says 172.21.15.26:59802) 2024-09-16T14:51:31.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f17fff640 1 -- 172.21.15.26:0/192861827 learned_addr learned my addr 172.21.15.26:0/192861827 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:31.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 1 -- 172.21.15.26:0/192861827 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491683570 0 0) 0x7f3f1809cf60 con 0x7f3f18132650 2024-09-16T14:51:31.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient(hunting): my global_id is 155030 2024-09-16T14:51:31.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 1 -- 172.21.15.26:0/192861827 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f00003530 con 0x7f3f18132650 2024-09-16T14:51:31.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 1 -- 172.21.15.26:0/192861827 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166653841 0 0) 0x7f3f1809d260 con 0x7f3f18138ff0 2024-09-16T14:51:31.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient(hunting): my global_id is 194139 2024-09-16T14:51:31.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 1 -- 172.21.15.26:0/192861827 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f1809cf60 con 0x7f3f18138ff0 2024-09-16T14:51:31.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 1 -- 172.21.15.26:0/192861827 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744360226 0 0) 0x7f3f18132a80 con 0x7f3f1809cb80 2024-09-16T14:51:31.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient(hunting): my global_id is 184327 2024-09-16T14:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 1 -- 172.21.15.26:0/192861827 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f1809d260 con 0x7f3f1809cb80 2024-09-16T14:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 1 -- 172.21.15.26:0/192861827 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2383314621 0 0) 0x7f3f00003530 con 0x7f3f18132650 2024-09-16T14:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 1 -- 172.21.15.26:0/192861827 >> v1:172.21.15.189:6789/0 conn(0x7f3f1809cb80 legacy=0x7f3f180717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 1 -- 172.21.15.26:0/192861827 >> v1:172.21.15.26:6789/0 conn(0x7f3f18138ff0 legacy=0x7f3f1813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient: _finish_hunting 0 2024-09-16T14:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 1 monclient: found mon.noname-b 2024-09-16T14:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 1 -- 172.21.15.26:0/192861827 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1809f2d0 con 0x7f3f18132650 2024-09-16T14:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient: _finish_auth 0 2024-09-16T14:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient: _check_auth_tickets 2024-09-16T14:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 1 -- 172.21.15.26:0/192861827 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f0c001660 con 0x7f3f18132650 2024-09-16T14:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 5 monclient: authenticate success, global_id 155030 2024-09-16T14:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient: dump: 2024-09-16T14:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 1 -- 172.21.15.26:0/192861827 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f0c001800 con 0x7f3f18132650 2024-09-16T14:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 1 -- 172.21.15.26:0/192861827 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f0c001bd0 con 0x7f3f18132650 2024-09-16T14:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 4 set_mon_vals no callback set 2024-09-16T14:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f177fe640 10 monclient: dump: 2024-09-16T14:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 10 monclient: shutdown 2024-09-16T14:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 1 -- 172.21.15.26:0/192861827 >> v1:172.21.15.26:6790/0 conn(0x7f3f18132650 legacy=0x7f3f181378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.185+0000 7f3f1f12c640 1 -- 172.21.15.26:0/192861827 shutdown_connections 2024-09-16T14:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 1 -- 172.21.15.26:0/192861827 >> 172.21.15.26:0/192861827 conn(0x7f3f1809fb50 msgr2=0x7f3f1809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 1 -- 172.21.15.26:0/192861827 shutdown_connections 2024-09-16T14:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 1 -- 172.21.15.26:0/192861827 wait complete. 2024-09-16T14:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 monclient: build_initial_monmap 2024-09-16T14:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 monclient: monmap: 2024-09-16T14:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 1 Processor -- start 2024-09-16T14:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 1 -- start start 2024-09-16T14:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 monclient: init 2024-09-16T14:51:31.200 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=0 2024-09-16T14:51:31.200 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T14:51:31.200 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:51:31.200 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 0 2024-09-16T14:51:31.200 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 2024-09-16T14:51:31.200 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=0 2024-09-16T14:51:31.200 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T14:51:31.201 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T14:51:31.202 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T14:51:31.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:31.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:51:31.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 monclient(hunting): picked mon.noname-a con 0x7f3f1809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:31.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 monclient(hunting): picked mon.noname-c con 0x7f3f18132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:31.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 monclient(hunting): picked mon.noname-b con 0x7f3f18138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:31.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f18138e50 con 0x7f3f1809cb80 2024-09-16T14:51:31.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f18071ac0 con 0x7f3f18138ff0 2024-09-16T14:51:31.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1809a240 con 0x7f3f18132650 2024-09-16T14:51:31.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:31.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 monclient(hunting): authenticate will time out at 1515.310142s 2024-09-16T14:51:31.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f17fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3f18132650 0x7f3f1809ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59810/0 (socket says 172.21.15.26:59810) 2024-09-16T14:51:31.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f17fff640 1 -- 172.21.15.26:0/3666869646 learned_addr learned my addr 172.21.15.26:0/3666869646 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:31.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360729914 0 0) 0x7f3f1809a240 con 0x7f3f18132650 2024-09-16T14:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): my global_id is 155033 2024-09-16T14:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f04003530 con 0x7f3f18132650 2024-09-16T14:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928555460 0 0) 0x7f3f1809d260 con 0x7f3f1809cb80 2024-09-16T14:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): my global_id is 194142 2024-09-16T14:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f1809a240 con 0x7f3f1809cb80 2024-09-16T14:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032696158 0 0) 0x7f3f18071ac0 con 0x7f3f18138ff0 2024-09-16T14:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): my global_id is 184330 2024-09-16T14:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f1809d260 con 0x7f3f18138ff0 2024-09-16T14:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3911231501 0 0) 0x7f3f04003530 con 0x7f3f18132650 2024-09-16T14:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f18071ac0 con 0x7f3f18132650 2024-09-16T14:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f0c001b10 con 0x7f3f18132650 2024-09-16T14:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): dump: 2024-09-16T14:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3481202962 0 0) 0x7f3f1809a240 con 0x7f3f1809cb80 2024-09-16T14:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f04003530 con 0x7f3f1809cb80 2024-09-16T14:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f08001640 con 0x7f3f1809cb80 2024-09-16T14:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): dump: 2024-09-16T14:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 377110942 0 0) 0x7f3f1809d260 con 0x7f3f18138ff0 2024-09-16T14:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f1809a240 con 0x7f3f18138ff0 2024-09-16T14:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f100019c0 con 0x7f3f18138ff0 2024-09-16T14:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient(hunting): dump: 2024-09-16T14:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1938304229 0 0) 0x7f3f18071ac0 con 0x7f3f18132650 2024-09-16T14:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 >> v1:172.21.15.189:6789/0 conn(0x7f3f18138ff0 legacy=0x7f3f181d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 >> v1:172.21.15.26:6789/0 conn(0x7f3f1809cb80 legacy=0x7f3f1809a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient: _finish_hunting 0 2024-09-16T14:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 monclient: found mon.c 2024-09-16T14:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1809b370 con 0x7f3f18132650 2024-09-16T14:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient: _finish_auth 0 2024-09-16T14:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient: _check_auth_tickets 2024-09-16T14:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 5 monclient: authenticate success, global_id 155033 2024-09-16T14:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 monclient: _renew_subs 2024-09-16T14:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 1 -- 172.21.15.26:0/3666869646 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1806d890 con 0x7f3f18132650 2024-09-16T14:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 monclient: _renew_subs 2024-09-16T14:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 1 -- 172.21.15.26:0/3666869646 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1809c380 con 0x7f3f18132650 2024-09-16T14:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f0c004aa0 con 0x7f3f18132650 2024-09-16T14:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f0c001260 con 0x7f3f18132650 2024-09-16T14:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f16ffd640 4 set_mon_vals no callback set 2024-09-16T14:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 10 monclient: dump: 2024-09-16T14:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3f0c00f6c0 con 0x7f3f18132650 2024-09-16T14:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.189+0000 7f3f1f12c640 1 -- 172.21.15.26:0/3666869646 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef8005050 con 0x7f3f18132650 2024-09-16T14:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.193+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 169..257) v4 ==== 6745+0+0 (unknown 920221734 0 0) 0x7f3f0c043df0 con 0x7f3f18132650 2024-09-16T14:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.193+0000 7f3f1cec8640 10 monclient: get_auth_request con 0x7f3f04036510 auth_method 0 2024-09-16T14:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.197+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3f0c014560 con 0x7f3f18132650 2024-09-16T14:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.197+0000 7f3f157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.197+0000 7f3f157fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:31.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.429+0000 7f3f1f12c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:51:31.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.429+0000 7f3f1f12c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:51:31.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.429+0000 7f3f1f12c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:51:31.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.429+0000 7f3f1f12c640 1 -- 172.21.15.26:0/3666869646 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3ef8005880 con 0x7f3f18132650 2024-09-16T14:51:31.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.433+0000 7f3f157fa640 1 -- 172.21.15.26:0/3666869646 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 441937313) 0x7f3f0c012c00 con 0x7f3f18132650 2024-09-16T14:51:31.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.433+0000 7f3f157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:51:31.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.433+0000 7f3f157fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:31.433 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":84,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:51:31.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.433+0000 7f3f1f12c640 1 -- 172.21.15.26:0/3666869646 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3f04036510 legacy=0x7f3f040389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.433+0000 7f3f1f12c640 10 monclient: shutdown 2024-09-16T14:51:31.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.433+0000 7f3f1f12c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:31.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.433+0000 7f3f1f12c640 1 -- 172.21.15.26:0/3666869646 >> v1:172.21.15.26:6790/0 conn(0x7f3f18132650 legacy=0x7f3f1809ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.433+0000 7f3f1f12c640 1 -- 172.21.15.26:0/3666869646 shutdown_connections 2024-09-16T14:51:31.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.433+0000 7f3f1f12c640 1 -- 172.21.15.26:0/3666869646 >> 172.21.15.26:0/3666869646 conn(0x7f3f1809fb50 msgr2=0x7f3f180a1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:31.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.433+0000 7f3f1f12c640 1 -- 172.21.15.26:0/3666869646 shutdown_connections 2024-09-16T14:51:31.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.433+0000 7f3f1f12c640 1 -- 172.21.15.26:0/3666869646 wait complete. 2024-09-16T14:51:31.450 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:51:31.451 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:51:31.451 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:51:31.451 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:51:31.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 10 monclient: get_monmap_and_config 2024-09-16T14:51:31.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 10 monclient: build_initial_monmap 2024-09-16T14:51:31.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 10 monclient: monmap: 2024-09-16T14:51:31.578 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:31.578 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.578 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:31.578439+0000 2024-09-16T14:51:31.578 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:31.578439+0000 2024-09-16T14:51:31.578 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:31.578 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.578 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:31.578 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:31.578 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:31.578 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 1 Processor -- start 2024-09-16T14:51:31.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 1 -- start start 2024-09-16T14:51:31.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 10 monclient: init 2024-09-16T14:51:31.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:31.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:51:31.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 10 monclient(hunting): picked mon.noname-b con 0x7f7a8809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:31.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 10 monclient(hunting): picked mon.noname-a con 0x7f7a88132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:31.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 10 monclient(hunting): picked mon.noname-c con 0x7f7a88138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:31.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8809d260 con 0x7f7a88132650 2024-09-16T14:51:31.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a88132a80 con 0x7f7a88138ff0 2024-09-16T14:51:31.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8809cf60 con 0x7f7a8809cb80 2024-09-16T14:51:31.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:31.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 10 monclient(hunting): authenticate will time out at 1515.698498s 2024-09-16T14:51:31.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8dea3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7a8809cb80 0x7f7a880717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59816/0 (socket says 172.21.15.26:59816) 2024-09-16T14:51:31.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8dea3640 1 -- 172.21.15.26:0/1712443536 learned_addr learned my addr 172.21.15.26:0/1712443536 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:31.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 1 -- 172.21.15.26:0/1712443536 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169327396 0 0) 0x7f7a8809cf60 con 0x7f7a8809cb80 2024-09-16T14:51:31.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient(hunting): my global_id is 155039 2024-09-16T14:51:31.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 1 -- 172.21.15.26:0/1712443536 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a74003530 con 0x7f7a8809cb80 2024-09-16T14:51:31.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 1 -- 172.21.15.26:0/1712443536 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882875043 0 0) 0x7f7a8809d260 con 0x7f7a88132650 2024-09-16T14:51:31.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient(hunting): my global_id is 194151 2024-09-16T14:51:31.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 1 -- 172.21.15.26:0/1712443536 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a8809cf60 con 0x7f7a88132650 2024-09-16T14:51:31.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 1 -- 172.21.15.26:0/1712443536 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322961638 0 0) 0x7f7a88132a80 con 0x7f7a88138ff0 2024-09-16T14:51:31.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient(hunting): my global_id is 184339 2024-09-16T14:51:31.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 1 -- 172.21.15.26:0/1712443536 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a8809d260 con 0x7f7a88138ff0 2024-09-16T14:51:31.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 1 -- 172.21.15.26:0/1712443536 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2168371978 0 0) 0x7f7a8809cf60 con 0x7f7a88132650 2024-09-16T14:51:31.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 1 -- 172.21.15.26:0/1712443536 >> v1:172.21.15.26:6790/0 conn(0x7f7a8809cb80 legacy=0x7f7a880717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 1 -- 172.21.15.26:0/1712443536 >> v1:172.21.15.189:6789/0 conn(0x7f7a88138ff0 legacy=0x7f7a8813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient: _finish_hunting 0 2024-09-16T14:51:31.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 1 monclient: found mon.noname-a 2024-09-16T14:51:31.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:31.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:51:31.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 1 -- 172.21.15.26:0/1712443536 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8809f2d0 con 0x7f7a88132650 2024-09-16T14:51:31.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient: _finish_auth 0 2024-09-16T14:51:31.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient: _check_auth_tickets 2024-09-16T14:51:31.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:31.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 1 -- 172.21.15.26:0/1712443536 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a78001600 con 0x7f7a88132650 2024-09-16T14:51:31.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 5 monclient: authenticate success, global_id 194151 2024-09-16T14:51:31.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:31.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:31.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient: dump: 2024-09-16T14:51:31.589 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.589 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.589 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.589 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.589 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.589 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.589 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.590 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.590 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.590 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:31.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 1 -- 172.21.15.26:0/1712443536 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a78001820 con 0x7f7a88132650 2024-09-16T14:51:31.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:31.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 1 -- 172.21.15.26:0/1712443536 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a78001b30 con 0x7f7a88132650 2024-09-16T14:51:31.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:31.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a8cea1640 10 monclient: dump: 2024-09-16T14:51:31.591 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.591 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.591 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.591 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.591 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.591 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.591 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.591 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.591 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.591 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:31.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.577+0000 7f7a90107640 4 set_mon_vals no callback set 2024-09-16T14:51:31.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 10 monclient: shutdown 2024-09-16T14:51:31.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:31.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 1 -- 172.21.15.26:0/1712443536 >> v1:172.21.15.26:6789/0 conn(0x7f7a88132650 legacy=0x7f7a881378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 1 -- 172.21.15.26:0/1712443536 shutdown_connections 2024-09-16T14:51:31.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 1 -- 172.21.15.26:0/1712443536 >> 172.21.15.26:0/1712443536 conn(0x7f7a8809fb50 msgr2=0x7f7a8809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:31.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 1 -- 172.21.15.26:0/1712443536 shutdown_connections 2024-09-16T14:51:31.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 1 -- 172.21.15.26:0/1712443536 wait complete. 2024-09-16T14:51:31.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 10 monclient: build_initial_monmap 2024-09-16T14:51:31.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 10 monclient: monmap: 2024-09-16T14:51:31.592 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:31.592 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:31.593 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:31.593 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:31.593 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:31.593 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:31.593 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:31.593 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:31.593 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:31.593 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 1 Processor -- start 2024-09-16T14:51:31.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 1 -- start start 2024-09-16T14:51:31.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 10 monclient: init 2024-09-16T14:51:31.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:31.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:51:31.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 10 monclient(hunting): picked mon.noname-a con 0x7f7a8809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:31.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 10 monclient(hunting): picked mon.noname-c con 0x7f7a88132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:31.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 10 monclient(hunting): picked mon.noname-b con 0x7f7a88138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:31.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8809b7e0 con 0x7f7a8809cb80 2024-09-16T14:51:31.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8813c400 con 0x7f7a88138ff0 2024-09-16T14:51:31.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a880a2a10 con 0x7f7a88132650 2024-09-16T14:51:31.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:31.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 10 monclient(hunting): authenticate will time out at 1515.701786s 2024-09-16T14:51:31.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a8dea3640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7a8809cb80 0x7f7a8809a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56348/0 (socket says 172.21.15.26:56348) 2024-09-16T14:51:31.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a8dea3640 1 -- 172.21.15.26:0/486297777 learned_addr learned my addr 172.21.15.26:0/486297777 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:31.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235899118 0 0) 0x7f7a8809b7e0 con 0x7f7a8809cb80 2024-09-16T14:51:31.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): my global_id is 194154 2024-09-16T14:51:31.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a6c003530 con 0x7f7a8809cb80 2024-09-16T14:51:31.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130724108 0 0) 0x7f7a880a2a10 con 0x7f7a88132650 2024-09-16T14:51:31.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): my global_id is 155042 2024-09-16T14:51:31.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a8809b7e0 con 0x7f7a88132650 2024-09-16T14:51:31.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 596787039 0 0) 0x7f7a6c003530 con 0x7f7a8809cb80 2024-09-16T14:51:31.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a880a2a10 con 0x7f7a8809cb80 2024-09-16T14:51:31.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3445120340 0 0) 0x7f7a8809b7e0 con 0x7f7a88132650 2024-09-16T14:51:31.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a6c003530 con 0x7f7a88132650 2024-09-16T14:51:31.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2266616652 0 0) 0x7f7a8813c400 con 0x7f7a88138ff0 2024-09-16T14:51:31.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): my global_id is 184342 2024-09-16T14:51:31.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a8809b7e0 con 0x7f7a88138ff0 2024-09-16T14:51:31.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a84001800 con 0x7f7a8809cb80 2024-09-16T14:51:31.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:31.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): dump: 2024-09-16T14:51:31.596 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.596 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.596 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.597 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.597 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.597 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.597 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.597 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.597 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.597 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:31.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a78001500 con 0x7f7a88132650 2024-09-16T14:51:31.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:31.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient(hunting): dump: 2024-09-16T14:51:31.597 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.597 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.598 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.598 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.598 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.598 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.598 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.598 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.598 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.598 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3254072230 0 0) 0x7f7a880a2a10 con 0x7f7a8809cb80 2024-09-16T14:51:31.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 >> v1:172.21.15.26:6790/0 conn(0x7f7a88132650 legacy=0x7f7a8809ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 >> v1:172.21.15.189:6789/0 conn(0x7f7a88138ff0 legacy=0x7f7a881d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:51:31.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 monclient: found mon.a 2024-09-16T14:51:31.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:31.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:31.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8806dc30 con 0x7f7a8809cb80 2024-09-16T14:51:31.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:51:31.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:51:31.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:31.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 5 monclient: authenticate success, global_id 194154 2024-09-16T14:51:31.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 10 monclient: _renew_subs 2024-09-16T14:51:31.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:31.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 1 -- 172.21.15.26:0/486297777 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8806df10 con 0x7f7a8809cb80 2024-09-16T14:51:31.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 10 monclient: _renew_subs 2024-09-16T14:51:31.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:31.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 1 -- 172.21.15.26:0/486297777 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8809b540 con 0x7f7a8809cb80 2024-09-16T14:51:31.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a840019f0 con 0x7f7a8809cb80 2024-09-16T14:51:31.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:31.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a84004a10 con 0x7f7a8809cb80 2024-09-16T14:51:31.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:31.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7ffff640 4 set_mon_vals no callback set 2024-09-16T14:51:31.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 10 monclient: dump: 2024-09-16T14:51:31.600 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.600 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.600 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.600 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.600 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.601 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.601 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.601 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.601 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.601 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:31.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a90107640 1 -- 172.21.15.26:0/486297777 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7a70006590 con 0x7f7a70002830 2024-09-16T14:51:31.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7a8400f500 con 0x7f7a8809cb80 2024-09-16T14:51:31.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.581+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 169..257) v4 ==== 6745+0+0 (unknown 920221734 0 0) 0x7f7a84043d40 con 0x7f7a8809cb80 2024-09-16T14:51:31.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914016471 0 0) 0x7f7a70006590 con 0x7f7a70002830 2024-09-16T14:51:31.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 10 monclient(hunting): my global_id is 194157 2024-09-16T14:51:31.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a880a2a10 con 0x7f7a70002830 2024-09-16T14:51:31.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a8e6a4640 10 monclient: get_auth_request con 0x7f7a6c0363d0 auth_method 0 2024-09-16T14:51:31.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4219949845 0 0) 0x7f7a880a2a10 con 0x7f7a70002830 2024-09-16T14:51:31.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a70006590 con 0x7f7a70002830 2024-09-16T14:51:31.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a780037f0 con 0x7f7a70002830 2024-09-16T14:51:31.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:31.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 10 monclient: dump: 2024-09-16T14:51:31.602 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.602 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.602 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.603 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.603 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.603 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.603 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.603 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.603 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.603 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1381062594 0 0) 0x7f7a70006590 con 0x7f7a70002830 2024-09-16T14:51:31.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a7000a0e0 con 0x7f7a70002830 2024-09-16T14:51:31.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7a780059b0 con 0x7f7a70002830 2024-09-16T14:51:31.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:31.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:31.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 >> v1:172.21.15.26:6789/0 conn(0x7f7a70002830 legacy=0x7f7a70004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.585+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 >> v1:172.21.15.26:6789/0 conn(0x7f7a70002830 legacy=0x7f7a70004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:31.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a90107640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:51:31.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a90107640 1 -- 172.21.15.26:0/486297777 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7a70009d80 con 0x7f7a700090a0 2024-09-16T14:51:31.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119939772 0 0) 0x7f7a70009d80 con 0x7f7a700090a0 2024-09-16T14:51:31.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 10 monclient(hunting): my global_id is 194160 2024-09-16T14:51:31.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a70006590 con 0x7f7a700090a0 2024-09-16T14:51:31.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3998517687 0 0) 0x7f7a70006590 con 0x7f7a700090a0 2024-09-16T14:51:31.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a70009d80 con 0x7f7a700090a0 2024-09-16T14:51:31.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a78003f20 con 0x7f7a700090a0 2024-09-16T14:51:31.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:31.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 10 monclient: dump: 2024-09-16T14:51:31.606 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.606 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.606 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.606 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.606 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.606 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.606 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.606 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.606 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.606 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3713836524 0 0) 0x7f7a70009d80 con 0x7f7a700090a0 2024-09-16T14:51:31.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7a7000c290 con 0x7f7a700090a0 2024-09-16T14:51:31.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 1811058373) 0x7f7a78005b20 con 0x7f7a700090a0 2024-09-16T14:51:31.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:51:31.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:31.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 >> v1:172.21.15.26:6789/0 conn(0x7f7a700090a0 legacy=0x7f7a70009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.589+0000 7f7a7e7fc640 1 -- 172.21.15.26:0/486297777 >> v1:172.21.15.26:6789/0 conn(0x7f7a700090a0 legacy=0x7f7a70009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:31.607 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:51:31.607 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:51:31.607 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:51:31.608 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:51:31.608 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 84, 2024-09-16T14:51:31.608 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:51:31.608 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:51:31.608 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:51:31.608 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:51:31.608 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:31.608 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:51:31.608 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:31.608 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:51:31.608 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:51:31.608 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:31.608 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:31.609 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:31.609 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:31.609 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:31.609 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:31.609 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:31.609 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:31.609 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:31.609 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:31.609 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:31.609 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:51:31.609 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:51:31.609 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:31.609 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:31.610 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:31.610 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:31.610 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:31.610 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:31.610 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:31.610 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:31.610 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:31.610 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:31.610 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.610 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:51:31.610 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:51:31.610 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:51:31.610 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:51:31.611 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:51:31.611 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:51:31.611 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:31.611 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:31.611 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:51:31.611 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:51:31.611 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:51:31.611 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:51:31.611 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:51:31.611 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:51:31.611 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:51:31.611 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:31.612 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:51:31.612 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:31.612 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:31.612 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:31.612 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:31.612 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:31.612 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:31.612 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:31.612 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:31.612 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:31.612 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:31.612 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:31.612 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:51:31.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.613 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:51:31.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.613 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:51:31.613 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:51:31.613 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:31.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:31.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:31.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:51:31.613 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:31.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.613 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:51:31.614 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:51:31.614 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:31.614 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:31.614 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:31.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.614 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:51:31.614 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:51:31.614 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:31.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:31.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.615 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:31.615 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:51:31.615 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:31.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.615 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.615 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:51:31.615 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:51:31.615 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:31.615 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:31.615 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:31.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.616 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:51:31.616 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:51:31.616 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:31.616 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:31.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.616 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:31.616 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:51:31.616 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:31.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.616 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.616 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:51:31.617 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:51:31.617 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:31.617 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:31.617 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:31.617 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.617 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.617 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:51:31.617 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:51:31.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.617 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:31.617 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:31.617 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:51:31.618 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.618 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:31.618 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:51:31.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.618 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:31.618 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:31.618 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:51:31.618 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.618 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.618 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:51:31.618 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:51:31.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.593+0000 7f7a90107640 1 -- 172.21.15.26:0/486297777 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7a6c0363d0 legacy=0x7f7a6c038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.593+0000 7f7a90107640 10 monclient: shutdown 2024-09-16T14:51:31.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.593+0000 7f7a90107640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:31.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.593+0000 7f7a90107640 1 -- 172.21.15.26:0/486297777 >> v1:172.21.15.26:6789/0 conn(0x7f7a8809cb80 legacy=0x7f7a8809a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.593+0000 7f7a8e6a4640 1 -- 172.21.15.26:0/486297777 reap_dead start 2024-09-16T14:51:31.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.593+0000 7f7a8e6a4640 1 -- 172.21.15.26:0/486297777 reap_dead start 2024-09-16T14:51:31.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.593+0000 7f7a90107640 1 -- 172.21.15.26:0/486297777 shutdown_connections 2024-09-16T14:51:31.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.593+0000 7f7a90107640 1 -- 172.21.15.26:0/486297777 >> 172.21.15.26:0/486297777 conn(0x7f7a8809fb50 msgr2=0x7f7a88131d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:31.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.593+0000 7f7a90107640 1 -- 172.21.15.26:0/486297777 shutdown_connections 2024-09-16T14:51:31.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.593+0000 7f7a90107640 1 -- 172.21.15.26:0/486297777 wait complete. 2024-09-16T14:51:31.621 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:51:31.621 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:51:31.621 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:51:31.659 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=0 2024-09-16T14:51:31.659 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T14:51:31.659 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:51:31.659 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls' 0 2024-09-16T14:51:31.659 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls' 2024-09-16T14:51:31.660 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=0 2024-09-16T14:51:31.660 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-16T14:51:31.660 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-16T14:51:31.660 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-16T14:51:31.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.733+0000 7f585f39b640 10 monclient: get_monmap_and_config 2024-09-16T14:51:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.733+0000 7f585f39b640 10 monclient: build_initial_monmap 2024-09-16T14:51:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.733+0000 7f585f39b640 10 monclient: monmap: 2024-09-16T14:51:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:31.737215+0000 2024-09-16T14:51:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:31.737215+0000 2024-09-16T14:51:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:31.736 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:31.736 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 1 Processor -- start 2024-09-16T14:51:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 1 -- start start 2024-09-16T14:51:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 10 monclient: init 2024-09-16T14:51:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:51:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 10 monclient(hunting): picked mon.noname-a con 0x7f58580966a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:31.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 10 monclient(hunting): picked mon.noname-c con 0x7f585809be00 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:31.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 10 monclient(hunting): picked mon.noname-b con 0x7f5858139150 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:31.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f585806c000 con 0x7f58580966a0 2024-09-16T14:51:31.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58580998a0 con 0x7f585809be00 2024-09-16T14:51:31.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f585809dbc0 con 0x7f5858139150 2024-09-16T14:51:31.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:31.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 10 monclient(hunting): authenticate will time out at 1515.857378s 2024-09-16T14:51:31.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585d137640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f58580966a0 0x7f5858096a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56372/0 (socket says 172.21.15.26:56372) 2024-09-16T14:51:31.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585d938640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5858139150 0x7f585813b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59824/0 (socket says 172.21.15.26:59824) 2024-09-16T14:51:31.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585d137640 1 -- 172.21.15.26:0/3352536772 learned_addr learned my addr 172.21.15.26:0/3352536772 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:31.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 1 -- 172.21.15.26:0/3352536772 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797974761 0 0) 0x7f585806c000 con 0x7f58580966a0 2024-09-16T14:51:31.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient(hunting): my global_id is 194163 2024-09-16T14:51:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 1 -- 172.21.15.26:0/3352536772 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5844003530 con 0x7f58580966a0 2024-09-16T14:51:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 1 -- 172.21.15.26:0/3352536772 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060520482 0 0) 0x7f585809dbc0 con 0x7f5858139150 2024-09-16T14:51:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient(hunting): my global_id is 155045 2024-09-16T14:51:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 1 -- 172.21.15.26:0/3352536772 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f585806c000 con 0x7f5858139150 2024-09-16T14:51:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 1 -- 172.21.15.26:0/3352536772 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3590605473 0 0) 0x7f5844003530 con 0x7f58580966a0 2024-09-16T14:51:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 1 -- 172.21.15.26:0/3352536772 >> v1:172.21.15.26:6790/0 conn(0x7f5858139150 legacy=0x7f585813b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 1 -- 172.21.15.26:0/3352536772 >> v1:172.21.15.189:6789/0 conn(0x7f585809be00 legacy=0x7f5858137a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:51:31.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 1 monclient: found mon.noname-a 2024-09-16T14:51:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:51:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 1 -- 172.21.15.26:0/3352536772 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585809c230 con 0x7f58580966a0 2024-09-16T14:51:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient: _finish_auth 0 2024-09-16T14:51:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:51:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 1 -- 172.21.15.26:0/3352536772 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5854001620 con 0x7f58580966a0 2024-09-16T14:51:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient: dump: 2024-09-16T14:51:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.743 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.744 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 5 monclient: authenticate success, global_id 194163 2024-09-16T14:51:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 1 -- 172.21.15.26:0/3352536772 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5854001880 con 0x7f58580966a0 2024-09-16T14:51:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 1 -- 172.21.15.26:0/3352536772 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5854001c50 con 0x7f58580966a0 2024-09-16T14:51:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 4 set_mon_vals no callback set 2024-09-16T14:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f584ffff640 10 monclient: dump: 2024-09-16T14:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 10 monclient: shutdown 2024-09-16T14:51:31.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:31.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 1 -- 172.21.15.26:0/3352536772 >> v1:172.21.15.26:6789/0 conn(0x7f58580966a0 legacy=0x7f5858096a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 1 -- 172.21.15.26:0/3352536772 shutdown_connections 2024-09-16T14:51:31.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 1 -- 172.21.15.26:0/3352536772 >> 172.21.15.26:0/3352536772 conn(0x7f58580a0360 msgr2=0x7f58580a0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:31.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 1 -- 172.21.15.26:0/3352536772 shutdown_connections 2024-09-16T14:51:31.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 1 -- 172.21.15.26:0/3352536772 wait complete. 2024-09-16T14:51:31.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 10 monclient: build_initial_monmap 2024-09-16T14:51:31.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 10 monclient: monmap: 2024-09-16T14:51:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:31.747 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 1 Processor -- start 2024-09-16T14:51:31.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 1 -- start start 2024-09-16T14:51:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.737+0000 7f585f39b640 10 monclient: init 2024-09-16T14:51:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:51:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 10 monclient(hunting): picked mon.noname-b con 0x7f58580966a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 10 monclient(hunting): picked mon.noname-a con 0x7f585809be00 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 10 monclient(hunting): picked mon.noname-c con 0x7f5858139150 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5858098e10 con 0x7f585809be00 2024-09-16T14:51:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5858128ff0 con 0x7f58580966a0 2024-09-16T14:51:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5858129520 con 0x7f5858139150 2024-09-16T14:51:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 10 monclient(hunting): authenticate will time out at 1515.860285s 2024-09-16T14:51:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585c936640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f585809be00 0x7f5858097da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56380/0 (socket says 172.21.15.26:56380) 2024-09-16T14:51:31.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585c936640 1 -- 172.21.15.26:0/3100272370 learned_addr learned my addr 172.21.15.26:0/3100272370 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610581918 0 0) 0x7f5858098e10 con 0x7f585809be00 2024-09-16T14:51:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): my global_id is 194166 2024-09-16T14:51:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f583c003560 con 0x7f585809be00 2024-09-16T14:51:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461467774 0 0) 0x7f5858129520 con 0x7f5858139150 2024-09-16T14:51:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): my global_id is 155048 2024-09-16T14:51:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5858098e10 con 0x7f5858139150 2024-09-16T14:51:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3025889263 0 0) 0x7f583c003560 con 0x7f585809be00 2024-09-16T14:51:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5858129520 con 0x7f585809be00 2024-09-16T14:51:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58480018c0 con 0x7f585809be00 2024-09-16T14:51:31.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): dump: 2024-09-16T14:51:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.754 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:31.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523798042 0 0) 0x7f5858128ff0 con 0x7f58580966a0 2024-09-16T14:51:31.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): my global_id is 184348 2024-09-16T14:51:31.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f583c003560 con 0x7f58580966a0 2024-09-16T14:51:31.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3496284582 0 0) 0x7f5858098e10 con 0x7f5858139150 2024-09-16T14:51:31.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5858128ff0 con 0x7f5858139150 2024-09-16T14:51:31.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5850001b70 con 0x7f5858139150 2024-09-16T14:51:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): dump: 2024-09-16T14:51:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.756 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.757 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3459121935 0 0) 0x7f5858129520 con 0x7f585809be00 2024-09-16T14:51:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 >> v1:172.21.15.26:6790/0 conn(0x7f5858139150 legacy=0x7f58581d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 >> v1:172.21.15.189:6789/0 conn(0x7f58580966a0 legacy=0x7f5858097690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:51:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 monclient: found mon.a 2024-09-16T14:51:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58580984b0 con 0x7f585809be00 2024-09-16T14:51:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient: _finish_auth 0 2024-09-16T14:51:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:51:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:31.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 5 monclient: authenticate success, global_id 194166 2024-09-16T14:51:31.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 10 monclient: _renew_subs 2024-09-16T14:51:31.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:31.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 1 -- 172.21.15.26:0/3100272370 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58580a1900 con 0x7f585809be00 2024-09-16T14:51:31.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 10 monclient: _renew_subs 2024-09-16T14:51:31.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:31.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 1 -- 172.21.15.26:0/3100272370 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5858128c80 con 0x7f585809be00 2024-09-16T14:51:31.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5848003620 con 0x7f585809be00 2024-09-16T14:51:31.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:31.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5848003930 con 0x7f585809be00 2024-09-16T14:51:31.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584f7fe640 4 set_mon_vals no callback set 2024-09-16T14:51:31.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:31.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient: dump: 2024-09-16T14:51:31.759 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.759 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.759 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.759 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.759 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.759 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.759 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.759 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.759 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.759 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:31.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585f39b640 1 -- 172.21.15.26:0/3100272370 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5838006590 con 0x7f5838002830 2024-09-16T14:51:31.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5848003a90 con 0x7f585809be00 2024-09-16T14:51:31.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 169..257) v4 ==== 6745+0+0 (unknown 920221734 0 0) 0x7f5848043c80 con 0x7f585809be00 2024-09-16T14:51:31.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704356767 0 0) 0x7f5838006590 con 0x7f5838002830 2024-09-16T14:51:31.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient(hunting): my global_id is 155051 2024-09-16T14:51:31.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5858129520 con 0x7f5838002830 2024-09-16T14:51:31.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f585d938640 10 monclient: get_auth_request con 0x7f583c036470 auth_method 0 2024-09-16T14:51:31.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2750313044 0 0) 0x7f5858129520 con 0x7f5838002830 2024-09-16T14:51:31.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5838006590 con 0x7f5838002830 2024-09-16T14:51:31.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5854001190 con 0x7f5838002830 2024-09-16T14:51:31.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:31.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient: dump: 2024-09-16T14:51:31.761 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.761 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.761 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.761 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.761 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.761 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.761 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.761 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.761 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.761 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2987843986 0 0) 0x7f5838006590 con 0x7f5838002830 2024-09-16T14:51:31.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f583800a0e0 con 0x7f5838002830 2024-09-16T14:51:31.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5854001c00 con 0x7f5838002830 2024-09-16T14:51:31.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:31.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:31.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 >> v1:172.21.15.26:6790/0 conn(0x7f5838002830 legacy=0x7f5838004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.741+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 >> v1:172.21.15.26:6790/0 conn(0x7f5838002830 legacy=0x7f5838004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:31.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f585f39b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:51:31.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f585f39b640 1 -- 172.21.15.26:0/3100272370 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5838009da0 con 0x7f58380090c0 2024-09-16T14:51:31.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632067038 0 0) 0x7f5838009da0 con 0x7f58380090c0 2024-09-16T14:51:31.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 10 monclient(hunting): my global_id is 155054 2024-09-16T14:51:31.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5838006590 con 0x7f58380090c0 2024-09-16T14:51:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1705220463 0 0) 0x7f5838006590 con 0x7f58380090c0 2024-09-16T14:51:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5838009da0 con 0x7f58380090c0 2024-09-16T14:51:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5854004c90 con 0x7f58380090c0 2024-09-16T14:51:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 10 monclient: dump: 2024-09-16T14:51:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.763 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.764 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1093002016 0 0) 0x7f5838009da0 con 0x7f58380090c0 2024-09-16T14:51:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f583800c290 con 0x7f58380090c0 2024-09-16T14:51:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 256309194) 0x7f5854004f60 con 0x7f58380090c0 2024-09-16T14:51:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:51:31.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 >> v1:172.21.15.26:6790/0 conn(0x7f58380090c0 legacy=0x7f5838009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f584dffb640 1 -- 172.21.15.26:0/3100272370 >> v1:172.21.15.26:6790/0 conn(0x7f58380090c0 legacy=0x7f5838009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 84, 2024-09-16T14:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:51:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:31.766 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:31.767 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:31.768 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:51:31.769 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:31.770 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:51:31.771 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:31.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:31.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:51:31.775 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.776 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.777 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:51:31.778 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:51:31.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f585f39b640 1 -- 172.21.15.26:0/3100272370 >> v1:172.21.15.189:6812/3196161582 conn(0x7f583c036470 legacy=0x7f583c038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f585f39b640 10 monclient: shutdown 2024-09-16T14:51:31.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f585f39b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:31.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f585f39b640 1 -- 172.21.15.26:0/3100272370 >> v1:172.21.15.26:6789/0 conn(0x7f585809be00 legacy=0x7f5858097da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f585d938640 1 -- 172.21.15.26:0/3100272370 reap_dead start 2024-09-16T14:51:31.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f585d938640 1 -- 172.21.15.26:0/3100272370 reap_dead start 2024-09-16T14:51:31.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f585f39b640 1 -- 172.21.15.26:0/3100272370 shutdown_connections 2024-09-16T14:51:31.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f585f39b640 1 -- 172.21.15.26:0/3100272370 >> 172.21.15.26:0/3100272370 conn(0x7f58580a0360 msgr2=0x7f5858132a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:31.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f585f39b640 1 -- 172.21.15.26:0/3100272370 shutdown_connections 2024-09-16T14:51:31.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.749+0000 7f585f39b640 1 -- 172.21.15.26:0/3100272370 wait complete. 2024-09-16T14:51:31.781 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:51:31.781 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:51:31.781 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:51:31.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 10 monclient: get_monmap_and_config 2024-09-16T14:51:31.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 10 monclient: build_initial_monmap 2024-09-16T14:51:31.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 10 monclient: monmap: 2024-09-16T14:51:31.893 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:31.893 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.893 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:31.894371+0000 2024-09-16T14:51:31.893 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:31.894371+0000 2024-09-16T14:51:31.893 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:31.894 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.894 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:31.894 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:31.894 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:31.894 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 1 Processor -- start 2024-09-16T14:51:31.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 1 -- start start 2024-09-16T14:51:31.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 10 monclient: init 2024-09-16T14:51:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:51:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 10 monclient(hunting): picked mon.noname-c con 0x7f122c09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 10 monclient(hunting): picked mon.noname-b con 0x7f122c132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 10 monclient(hunting): picked mon.noname-a con 0x7f122c138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c09d260 con 0x7f122c138ff0 2024-09-16T14:51:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c132a80 con 0x7f122c09cb80 2024-09-16T14:51:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c09cf60 con 0x7f122c132650 2024-09-16T14:51:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 10 monclient(hunting): authenticate will time out at 1516.014606s 2024-09-16T14:51:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1222ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f122c132650 0x7f122c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59842/0 (socket says 172.21.15.26:59842) 2024-09-16T14:51:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1222ffd640 1 -- 172.21.15.26:0/1897889301 learned_addr learned my addr 172.21.15.26:0/1897889301 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:31.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 1 -- 172.21.15.26:0/1897889301 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060178492 0 0) 0x7f122c09cf60 con 0x7f122c132650 2024-09-16T14:51:31.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient(hunting): my global_id is 155063 2024-09-16T14:51:31.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 1 -- 172.21.15.26:0/1897889301 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1218003530 con 0x7f122c132650 2024-09-16T14:51:31.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 1 -- 172.21.15.26:0/1897889301 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230923300 0 0) 0x7f122c09d260 con 0x7f122c138ff0 2024-09-16T14:51:31.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient(hunting): my global_id is 194175 2024-09-16T14:51:31.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 1 -- 172.21.15.26:0/1897889301 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f122c09cf60 con 0x7f122c138ff0 2024-09-16T14:51:31.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 1 -- 172.21.15.26:0/1897889301 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3685788977 0 0) 0x7f1218003530 con 0x7f122c132650 2024-09-16T14:51:31.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 1 -- 172.21.15.26:0/1897889301 >> v1:172.21.15.189:6789/0 conn(0x7f122c09cb80 legacy=0x7f122c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 1 -- 172.21.15.26:0/1897889301 >> v1:172.21.15.26:6789/0 conn(0x7f122c138ff0 legacy=0x7f122c13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient: _finish_hunting 0 2024-09-16T14:51:31.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 1 monclient: found mon.noname-b 2024-09-16T14:51:31.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:31.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:51:31.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 1 -- 172.21.15.26:0/1897889301 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122c09f2d0 con 0x7f122c132650 2024-09-16T14:51:31.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient: _finish_auth 0 2024-09-16T14:51:31.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient: _check_auth_tickets 2024-09-16T14:51:31.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:31.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 5 monclient: authenticate success, global_id 155063 2024-09-16T14:51:31.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 1 -- 172.21.15.26:0/1897889301 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1214001660 con 0x7f122c132650 2024-09-16T14:51:31.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:31.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:51:31.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient: dump: 2024-09-16T14:51:31.906 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.906 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.906 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.906 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.906 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.906 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.906 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.906 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.906 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.906 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:31.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 1 -- 172.21.15.26:0/1897889301 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1214001880 con 0x7f122c132650 2024-09-16T14:51:31.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:31.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:31.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 1 -- 172.21.15.26:0/1897889301 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1214001c50 con 0x7f122c132650 2024-09-16T14:51:31.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 4 set_mon_vals no callback set 2024-09-16T14:51:31.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:31.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f122affd640 10 monclient: dump: 2024-09-16T14:51:31.907 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.907 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.907 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.907 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.908 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.908 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.908 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.908 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.908 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.908 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 10 monclient: shutdown 2024-09-16T14:51:31.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:31.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 1 -- 172.21.15.26:0/1897889301 >> v1:172.21.15.26:6790/0 conn(0x7f122c132650 legacy=0x7f122c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 1 -- 172.21.15.26:0/1897889301 shutdown_connections 2024-09-16T14:51:31.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.893+0000 7f1231b1e640 1 -- 172.21.15.26:0/1897889301 >> 172.21.15.26:0/1897889301 conn(0x7f122c09fb50 msgr2=0x7f122c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:31.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 1 -- 172.21.15.26:0/1897889301 shutdown_connections 2024-09-16T14:51:31.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 1 -- 172.21.15.26:0/1897889301 wait complete. 2024-09-16T14:51:31.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 10 monclient: build_initial_monmap 2024-09-16T14:51:31.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 10 monclient: monmap: 2024-09-16T14:51:31.909 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:31.909 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:31.909 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:31.909 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:31.909 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:31.909 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:31.909 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:31.909 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:31.910 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:31.910 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 1 Processor -- start 2024-09-16T14:51:31.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 1 -- start start 2024-09-16T14:51:31.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 10 monclient: init 2024-09-16T14:51:31.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:31.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:51:31.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 10 monclient(hunting): picked mon.noname-c con 0x7f122c09cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:31.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 10 monclient(hunting): picked mon.noname-b con 0x7f122c132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:31.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 10 monclient(hunting): picked mon.noname-a con 0x7f122c138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:31.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c0717e0 con 0x7f122c138ff0 2024-09-16T14:51:31.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c0712e0 con 0x7f122c132650 2024-09-16T14:51:31.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c06e480 con 0x7f122c09cb80 2024-09-16T14:51:31.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:31.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 10 monclient(hunting): authenticate will time out at 1516.017497s 2024-09-16T14:51:31.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f122b7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f122c09cb80 0x7f122c06d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59850/0 (socket says 172.21.15.26:59850) 2024-09-16T14:51:31.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f122b7fe640 1 -- 172.21.15.26:0/2184003926 learned_addr learned my addr 172.21.15.26:0/2184003926 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:31.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968923845 0 0) 0x7f122c0717e0 con 0x7f122c138ff0 2024-09-16T14:51:31.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): my global_id is 194178 2024-09-16T14:51:31.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f120c003530 con 0x7f122c138ff0 2024-09-16T14:51:31.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522850580 0 0) 0x7f122c06e480 con 0x7f122c09cb80 2024-09-16T14:51:31.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): my global_id is 155066 2024-09-16T14:51:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f122c0717e0 con 0x7f122c09cb80 2024-09-16T14:51:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 534762970 0 0) 0x7f120c003530 con 0x7f122c138ff0 2024-09-16T14:51:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f122c06e480 con 0x7f122c138ff0 2024-09-16T14:51:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1490210697 0 0) 0x7f122c0717e0 con 0x7f122c09cb80 2024-09-16T14:51:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f120c003530 con 0x7f122c09cb80 2024-09-16T14:51:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730735165 0 0) 0x7f122c0712e0 con 0x7f122c132650 2024-09-16T14:51:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): my global_id is 184360 2024-09-16T14:51:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f122c0717e0 con 0x7f122c132650 2024-09-16T14:51:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1224001ce0 con 0x7f122c138ff0 2024-09-16T14:51:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): dump: 2024-09-16T14:51:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.913 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.914 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f121c001800 con 0x7f122c09cb80 2024-09-16T14:51:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:51:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): dump: 2024-09-16T14:51:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.915 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.915 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.915 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3496950617 0 0) 0x7f122c06e480 con 0x7f122c138ff0 2024-09-16T14:51:31.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 >> v1:172.21.15.26:6790/0 conn(0x7f122c09cb80 legacy=0x7f122c06d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 >> v1:172.21.15.189:6789/0 conn(0x7f122c132650 legacy=0x7f122c06dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:51:31.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 monclient: found mon.a 2024-09-16T14:51:31.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:31.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:31.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122c13c600 con 0x7f122c138ff0 2024-09-16T14:51:31.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient: _finish_auth 0 2024-09-16T14:51:31.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:51:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 5 monclient: authenticate success, global_id 194178 2024-09-16T14:51:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 10 monclient: _renew_subs 2024-09-16T14:51:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 1 -- 172.21.15.26:0/2184003926 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f122c09a240 con 0x7f122c138ff0 2024-09-16T14:51:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 10 monclient: _renew_subs 2024-09-16T14:51:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 1 -- 172.21.15.26:0/2184003926 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f122c06fbb0 con 0x7f122c138ff0 2024-09-16T14:51:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12240018e0 con 0x7f122c138ff0 2024-09-16T14:51:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1224003b20 con 0x7f122c138ff0 2024-09-16T14:51:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:31.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f122a7fc640 4 set_mon_vals no callback set 2024-09-16T14:51:31.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient: dump: 2024-09-16T14:51:31.917 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.917 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.917 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.917 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.917 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.917 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.917 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.917 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.917 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.917 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1231b1e640 1 -- 172.21.15.26:0/2184003926 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1210006590 con 0x7f1210002830 2024-09-16T14:51:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f122400e590 con 0x7f122c138ff0 2024-09-16T14:51:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 169..257) v4 ==== 6745+0+0 (unknown 920221734 0 0) 0x7f1224042d60 con 0x7f122c138ff0 2024-09-16T14:51:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1222ffd640 10 monclient: get_auth_request con 0x7f120c036450 auth_method 0 2024-09-16T14:51:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365996115 0 0) 0x7f1210006590 con 0x7f1210002830 2024-09-16T14:51:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 10 monclient(hunting): my global_id is 184363 2024-09-16T14:51:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.897+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f122c06e480 con 0x7f1210002830 2024-09-16T14:51:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.901+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 315638934 0 0) 0x7f122c06e480 con 0x7f1210002830 2024-09-16T14:51:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.901+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1210006590 con 0x7f1210002830 2024-09-16T14:51:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.901+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f121c003a90 con 0x7f1210002830 2024-09-16T14:51:31.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.901+0000 7f1228ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.901+0000 7f1228ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.901+0000 7f1228ff9640 10 monclient: dump: 2024-09-16T14:51:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.919 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.901+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1950489008 0 0) 0x7f1210006590 con 0x7f1210002830 2024-09-16T14:51:31.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.901+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f121000a0e0 con 0x7f1210002830 2024-09-16T14:51:31.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.901+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f121c0057d0 con 0x7f1210002830 2024-09-16T14:51:31.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.901+0000 7f1228ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:31.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.901+0000 7f1228ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:31.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.901+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 >> v1:172.21.15.189:6789/0 conn(0x7f1210002830 legacy=0x7f1210004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.901+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 >> v1:172.21.15.189:6789/0 conn(0x7f1210002830 legacy=0x7f1210004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:31.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.905+0000 7f1231b1e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:51:31.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.905+0000 7f1231b1e640 1 -- 172.21.15.26:0/2184003926 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1210009d60 con 0x7f12100090a0 2024-09-16T14:51:31.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281058789 0 0) 0x7f1210009d60 con 0x7f12100090a0 2024-09-16T14:51:31.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:51:31.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:31.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:31.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 10 monclient(hunting): my global_id is 184366 2024-09-16T14:51:31.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1210006590 con 0x7f12100090a0 2024-09-16T14:51:31.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2641121998 0 0) 0x7f1210006590 con 0x7f12100090a0 2024-09-16T14:51:31.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1210009d60 con 0x7f12100090a0 2024-09-16T14:51:31.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f121c004160 con 0x7f12100090a0 2024-09-16T14:51:31.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:31.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:31.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 10 monclient: dump: 2024-09-16T14:51:31.922 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:31.922 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:31.922 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:31.923 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4155599462 0 0) 0x7f1210009d60 con 0x7f12100090a0 2024-09-16T14:51:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f121000c290 con 0x7f12100090a0 2024-09-16T14:51:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 860531601) 0x7f121c0043d0 con 0x7f12100090a0 2024-09-16T14:51:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:51:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:31.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 >> v1:172.21.15.189:6789/0 conn(0x7f12100090a0 legacy=0x7f1210009510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1228ff9640 1 -- 172.21.15.26:0/2184003926 >> v1:172.21.15.189:6789/0 conn(0x7f12100090a0 legacy=0x7f1210009510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:31.924 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:51:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:51:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:51:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:51:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 84, 2024-09-16T14:51:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:51:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:51:31.924 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:51:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:51:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:51:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:51:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:51:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:31.925 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:51:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:51:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:31.926 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:51:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:51:31.927 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:51:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:51:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:51:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:51:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:51:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:51:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:51:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:51:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:51:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:51:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:51:31.928 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:51:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:31.929 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:51:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:51:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:51:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:51:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:51:31.930 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:51:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:51:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:51:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:51:31.931 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:31.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:31.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:31.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:51:31.932 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:31.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:51:31.933 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:51:31.933 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:31.934 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:31.934 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:31.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.934 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:51:31.935 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:51:31.935 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:31.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:31.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:31.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:51:31.935 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:31.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:51:31.936 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:51:31.936 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:31.936 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:31.936 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:31.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:51:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:51:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:51:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:51:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:31.937 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:31.938 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:31.938 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:51:31.938 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:31.938 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:31.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:31.938 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:51:31.938 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:51:31.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1231b1e640 1 -- 172.21.15.26:0/2184003926 >> v1:172.21.15.189:6812/3196161582 conn(0x7f120c036450 legacy=0x7f120c038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1231b1e640 10 monclient: shutdown 2024-09-16T14:51:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1231b1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1231b1e640 1 -- 172.21.15.26:0/2184003926 >> v1:172.21.15.26:6789/0 conn(0x7f122c138ff0 legacy=0x7f122c1cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f122bfff640 1 -- 172.21.15.26:0/2184003926 reap_dead start 2024-09-16T14:51:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f122bfff640 1 -- 172.21.15.26:0/2184003926 reap_dead start 2024-09-16T14:51:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1231b1e640 1 -- 172.21.15.26:0/2184003926 shutdown_connections 2024-09-16T14:51:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1231b1e640 1 -- 172.21.15.26:0/2184003926 >> 172.21.15.26:0/2184003926 conn(0x7f122c09fb50 msgr2=0x7f122c0a1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1231b1e640 1 -- 172.21.15.26:0/2184003926 shutdown_connections 2024-09-16T14:51:31.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:31.909+0000 7f1231b1e640 1 -- 172.21.15.26:0/2184003926 wait complete. 2024-09-16T14:51:31.940 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:51:31.941 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T14:51:31.941 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:51:31.941 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T14:51:31.941 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T14:51:31.941 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T14:51:31.941 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:51:31.955 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T14:51:31.956 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:51:31.956 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:51:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 10 monclient: get_monmap_and_config 2024-09-16T14:51:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 10 monclient: build_initial_monmap 2024-09-16T14:51:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 10 monclient: monmap: 2024-09-16T14:51:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:32.114281+0000 2024-09-16T14:51:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:32.114281+0000 2024-09-16T14:51:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:32.113 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:32.114 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:32.114 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:32.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 1 Processor -- start 2024-09-16T14:51:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 1 -- start start 2024-09-16T14:51:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 10 monclient: init 2024-09-16T14:51:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:51:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 10 monclient(hunting): picked mon.noname-a con 0x7fc66c09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 10 monclient(hunting): picked mon.noname-c con 0x7fc66c132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 10 monclient(hunting): picked mon.noname-b con 0x7fc66c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66c09d260 con 0x7fc66c09cb80 2024-09-16T14:51:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66c132a80 con 0x7fc66c132650 2024-09-16T14:51:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66c09cf60 con 0x7fc66c138ff0 2024-09-16T14:51:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 10 monclient(hunting): authenticate will time out at 1516.234365s 2024-09-16T14:51:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66bfff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fc66c138ff0 legacy=0x7fc66c13b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc66c09cb80 0x7fc66c0717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56444/0 (socket says 172.21.15.26:56444) 2024-09-16T14:51:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66b7fe640 1 -- 172.21.15.26:0/1162949212 learned_addr learned my addr 172.21.15.26:0/1162949212 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 1 -- 172.21.15.26:0/1162949212 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29167528 0 0) 0x7fc66c09d260 con 0x7fc66c09cb80 2024-09-16T14:51:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient(hunting): my global_id is 194181 2024-09-16T14:51:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 1 -- 172.21.15.26:0/1162949212 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc654003530 con 0x7fc66c09cb80 2024-09-16T14:51:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 1 -- 172.21.15.26:0/1162949212 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192526333 0 0) 0x7fc66c132a80 con 0x7fc66c132650 2024-09-16T14:51:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:51:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:32.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient(hunting): my global_id is 184369 2024-09-16T14:51:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 1 -- 172.21.15.26:0/1162949212 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc66c09d260 con 0x7fc66c132650 2024-09-16T14:51:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 1 -- 172.21.15.26:0/1162949212 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1287566599 0 0) 0x7fc654003530 con 0x7fc66c09cb80 2024-09-16T14:51:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 1 -- 172.21.15.26:0/1162949212 >> v1:172.21.15.26:6790/0 conn(0x7fc66c138ff0 legacy=0x7fc66c13b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 1 -- 172.21.15.26:0/1162949212 >> v1:172.21.15.189:6789/0 conn(0x7fc66c132650 legacy=0x7fc66c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:51:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 1 monclient: found mon.noname-a 2024-09-16T14:51:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:51:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 1 -- 172.21.15.26:0/1162949212 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66c09f2d0 con 0x7fc66c09cb80 2024-09-16T14:51:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:51:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:51:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 1 -- 172.21.15.26:0/1162949212 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc65c001620 con 0x7fc66c09cb80 2024-09-16T14:51:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 5 monclient: authenticate success, global_id 194181 2024-09-16T14:51:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient: dump: 2024-09-16T14:51:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 1 -- 172.21.15.26:0/1162949212 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc65c001890 con 0x7fc66c09cb80 2024-09-16T14:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 1 -- 172.21.15.26:0/1162949212 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc65c001c20 con 0x7fc66c09cb80 2024-09-16T14:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc66a7fc640 10 monclient: dump: 2024-09-16T14:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:32.125 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 4 set_mon_vals no callback set 2024-09-16T14:51:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 10 monclient: shutdown 2024-09-16T14:51:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 1 -- 172.21.15.26:0/1162949212 >> v1:172.21.15.26:6789/0 conn(0x7fc66c09cb80 legacy=0x7fc66c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 1 -- 172.21.15.26:0/1162949212 shutdown_connections 2024-09-16T14:51:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 1 -- 172.21.15.26:0/1162949212 >> 172.21.15.26:0/1162949212 conn(0x7fc66c09fb50 msgr2=0x7fc66c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 1 -- 172.21.15.26:0/1162949212 shutdown_connections 2024-09-16T14:51:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.113+0000 7fc671cf8640 1 -- 172.21.15.26:0/1162949212 wait complete. 2024-09-16T14:51:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 monclient: build_initial_monmap 2024-09-16T14:51:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 monclient: monmap: 2024-09-16T14:51:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:32.127 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 1 Processor -- start 2024-09-16T14:51:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 1 -- start start 2024-09-16T14:51:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 monclient: init 2024-09-16T14:51:32.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:32.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:51:32.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 monclient(hunting): picked mon.noname-a con 0x7fc66c09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:32.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 monclient(hunting): picked mon.noname-b con 0x7fc66c132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:32.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 monclient(hunting): picked mon.noname-c con 0x7fc66c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:32.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66c06fde0 con 0x7fc66c09cb80 2024-09-16T14:51:32.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66c13c5e0 con 0x7fc66c132650 2024-09-16T14:51:32.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66c0a2270 con 0x7fc66c138ff0 2024-09-16T14:51:32.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:32.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 monclient(hunting): authenticate will time out at 1516.237526s 2024-09-16T14:51:32.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc66bfff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fc66c138ff0 legacy=0x7fc66c1cfb60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:32.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc66b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc66c09cb80 0x7fc66c06d5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56446/0 (socket says 172.21.15.26:56446) 2024-09-16T14:51:32.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc66b7fe640 1 -- 172.21.15.26:0/3641359549 learned_addr learned my addr 172.21.15.26:0/3641359549 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:32.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110292318 0 0) 0x7fc66c06fde0 con 0x7fc66c09cb80 2024-09-16T14:51:32.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:32.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:32.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:32.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient(hunting): my global_id is 194184 2024-09-16T14:51:32.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc658003530 con 0x7fc66c09cb80 2024-09-16T14:51:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609627273 0 0) 0x7fc66c13c5e0 con 0x7fc66c132650 2024-09-16T14:51:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient(hunting): my global_id is 184372 2024-09-16T14:51:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc66c06fde0 con 0x7fc66c132650 2024-09-16T14:51:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4209512533 0 0) 0x7fc658003530 con 0x7fc66c09cb80 2024-09-16T14:51:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc66c13c5e0 con 0x7fc66c09cb80 2024-09-16T14:51:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc65c001be0 con 0x7fc66c09cb80 2024-09-16T14:51:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient(hunting): dump: 2024-09-16T14:51:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:32.132 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3435772089 0 0) 0x7fc66c13c5e0 con 0x7fc66c09cb80 2024-09-16T14:51:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 >> v1:172.21.15.26:6790/0 conn(0x7fc66c138ff0 legacy=0x7fc66c1cfb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 >> v1:172.21.15.189:6789/0 conn(0x7fc66c132650 legacy=0x7fc66c06dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient: _finish_hunting 0 2024-09-16T14:51:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 1 monclient: found mon.a 2024-09-16T14:51:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66c071220 con 0x7fc66c09cb80 2024-09-16T14:51:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient: _finish_auth 0 2024-09-16T14:51:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient: _check_auth_tickets 2024-09-16T14:51:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 5 monclient: authenticate success, global_id 194184 2024-09-16T14:51:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 monclient: _renew_subs 2024-09-16T14:51:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 1 -- 172.21.15.26:0/3641359549 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc66c06e3c0 con 0x7fc66c09cb80 2024-09-16T14:51:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 monclient: _renew_subs 2024-09-16T14:51:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 1 -- 172.21.15.26:0/3641359549 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc66c06f1a0 con 0x7fc66c09cb80 2024-09-16T14:51:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc65c0011c0 con 0x7fc66c09cb80 2024-09-16T14:51:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc65c004d10 con 0x7fc66c09cb80 2024-09-16T14:51:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc6697fa640 4 set_mon_vals no callback set 2024-09-16T14:51:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 10 monclient: dump: 2024-09-16T14:51:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:32.135 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc671cf8640 1 -- 172.21.15.26:0/3641359549 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc64c005050 con 0x7fc66c09cb80 2024-09-16T14:51:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc65c004e70 con 0x7fc66c09cb80 2024-09-16T14:51:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 169..257) v4 ==== 6745+0+0 (unknown 920221734 0 0) 0x7fc65c043f40 con 0x7fc66c09cb80 2024-09-16T14:51:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.117+0000 7fc66affd640 10 monclient: get_auth_request con 0x7fc6580361b0 auth_method 0 2024-09-16T14:51:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.121+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc65c006440 con 0x7fc66c09cb80 2024-09-16T14:51:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.121+0000 7fc653fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.121+0000 7fc653fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:32.141 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=0 2024-09-16T14:51:32.142 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T14:51:32.142 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:51:32.142 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export' 13 2024-09-16T14:51:32.142 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export' 2024-09-16T14:51:32.142 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T14:51:32.142 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-16T14:51:32.142 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-16T14:51:32.142 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-16T14:51:32.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.341+0000 7fc671cf8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:51:32.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.341+0000 7fc671cf8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:51:32.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.341+0000 7fc671cf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:32.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.341+0000 7fc671cf8640 1 -- 172.21.15.26:0/3641359549 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc64c005880 con 0x7fc66c09cb80 2024-09-16T14:51:32.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.345+0000 7fc653fff640 1 -- 172.21.15.26:0/3641359549 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2728800926) 0x7fc65c012db0 con 0x7fc66c09cb80 2024-09-16T14:51:32.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.345+0000 7fc653fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:51:32.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.345+0000 7fc653fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:32.345 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":84,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:51:32.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.345+0000 7fc671cf8640 1 -- 172.21.15.26:0/3641359549 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc6580361b0 legacy=0x7fc658038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:32.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.345+0000 7fc671cf8640 10 monclient: shutdown 2024-09-16T14:51:32.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.345+0000 7fc671cf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:32.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.345+0000 7fc671cf8640 1 -- 172.21.15.26:0/3641359549 >> v1:172.21.15.26:6789/0 conn(0x7fc66c09cb80 legacy=0x7fc66c06d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:32.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.345+0000 7fc671cf8640 1 -- 172.21.15.26:0/3641359549 shutdown_connections 2024-09-16T14:51:32.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.345+0000 7fc671cf8640 1 -- 172.21.15.26:0/3641359549 >> 172.21.15.26:0/3641359549 conn(0x7fc66c09fb50 msgr2=0x7fc66c1306a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:32.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.345+0000 7fc671cf8640 1 -- 172.21.15.26:0/3641359549 shutdown_connections 2024-09-16T14:51:32.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:32.345+0000 7fc671cf8640 1 -- 172.21.15.26:0/3641359549 wait complete. 2024-09-16T14:51:32.461 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T14:51:32.461 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T14:51:32.461 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:51:32.461 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar' 13 2024-09-16T14:51:32.461 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar' 2024-09-16T14:51:32.461 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T14:51:32.462 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-16T14:51:32.462 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-16T14:51:32.462 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-16T14:51:32.783 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T14:51:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T14:51:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:51:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump' 13 2024-09-16T14:51:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump' 2024-09-16T14:51:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T14:51:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-16T14:51:32.784 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-16T14:51:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-16T14:51:33.052 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T14:51:33.052 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T14:51:33.052 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:51:33.052 INFO:tasks.workunit.client.0.smithi189.stderr:+ export CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-16T14:51:33.052 INFO:tasks.workunit.client.0.smithi189.stderr:+ CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-16T14:51:33.052 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump' 110 2024-09-16T14:51:33.052 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump' 2024-09-16T14:51:33.052 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=110 2024-09-16T14:51:33.053 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-16T14:51:33.053 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-16T14:51:33.053 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-16T14:51:35.363 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:51:35.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.492+0000 7f8046765640 10 monclient: get_monmap_and_config 2024-09-16T14:51:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.492+0000 7f8046765640 10 monclient: build_initial_monmap 2024-09-16T14:51:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.492+0000 7f8046765640 10 monclient: monmap: 2024-09-16T14:51:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:35.496026+0000 2024-09-16T14:51:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:35.496026+0000 2024-09-16T14:51:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:35.496 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.492+0000 7f8046765640 1 Processor -- start 2024-09-16T14:51:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.492+0000 7f8046765640 1 -- start start 2024-09-16T14:51:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.492+0000 7f8046765640 10 monclient: init 2024-09-16T14:51:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:51:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 10 monclient(hunting): picked mon.noname-c con 0x7f804009cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 10 monclient(hunting): picked mon.noname-a con 0x7f804009e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 10 monclient(hunting): picked mon.noname-b con 0x7f804009d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804009d5b0 con 0x7f804009e7a0 2024-09-16T14:51:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8040131a90 con 0x7f804009cdc0 2024-09-16T14:51:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804009aff0 con 0x7f804009d1a0 2024-09-16T14:51:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 10 monclient(hunting): authenticate will time out at 1519.616830s 2024-09-16T14:51:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8044d02640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f804009d1a0 legacy=0x7f8040139fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8037fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f804009e7a0 0x7f8040131380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56454/0 (socket says 172.21.15.26:56454) 2024-09-16T14:51:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8037fff640 1 -- 172.21.15.26:0/3438787162 learned_addr learned my addr 172.21.15.26:0/3438787162 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:35.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 1 -- 172.21.15.26:0/3438787162 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193517549 0 0) 0x7f804009d5b0 con 0x7f804009e7a0 2024-09-16T14:51:35.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:35.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient(hunting): my global_id is 194211 2024-09-16T14:51:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 1 -- 172.21.15.26:0/3438787162 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f802c003450 con 0x7f804009e7a0 2024-09-16T14:51:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 1 -- 172.21.15.26:0/3438787162 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735025635 0 0) 0x7f8040131a90 con 0x7f804009cdc0 2024-09-16T14:51:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient(hunting): my global_id is 184399 2024-09-16T14:51:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 1 -- 172.21.15.26:0/3438787162 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f804009d5b0 con 0x7f804009cdc0 2024-09-16T14:51:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 1 -- 172.21.15.26:0/3438787162 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2239901347 0 0) 0x7f802c003450 con 0x7f804009e7a0 2024-09-16T14:51:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 1 -- 172.21.15.26:0/3438787162 >> v1:172.21.15.26:6790/0 conn(0x7f804009d1a0 legacy=0x7f8040139fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 1 -- 172.21.15.26:0/3438787162 >> v1:172.21.15.189:6789/0 conn(0x7f804009cdc0 legacy=0x7f8040130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:35.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:51:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 1 monclient: found mon.noname-a 2024-09-16T14:51:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:51:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 1 -- 172.21.15.26:0/3438787162 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804009b2d0 con 0x7f804009e7a0 2024-09-16T14:51:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:51:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:51:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 1 -- 172.21.15.26:0/3438787162 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8030001620 con 0x7f804009e7a0 2024-09-16T14:51:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient: dump: 2024-09-16T14:51:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:35.507 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 5 monclient: authenticate success, global_id 194211 2024-09-16T14:51:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 1 -- 172.21.15.26:0/3438787162 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8030001840 con 0x7f804009e7a0 2024-09-16T14:51:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 1 -- 172.21.15.26:0/3438787162 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8030001b50 con 0x7f804009e7a0 2024-09-16T14:51:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 4 set_mon_vals no callback set 2024-09-16T14:51:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f803f7fe640 10 monclient: dump: 2024-09-16T14:51:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:35.509 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:35.509 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:35.509 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:35.509 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:35.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 10 monclient: shutdown 2024-09-16T14:51:35.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:35.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 1 -- 172.21.15.26:0/3438787162 >> v1:172.21.15.26:6789/0 conn(0x7f804009e7a0 legacy=0x7f8040131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:35.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 1 -- 172.21.15.26:0/3438787162 shutdown_connections 2024-09-16T14:51:35.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 1 -- 172.21.15.26:0/3438787162 >> 172.21.15.26:0/3438787162 conn(0x7f804009fb50 msgr2=0x7f804009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:35.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 1 -- 172.21.15.26:0/3438787162 shutdown_connections 2024-09-16T14:51:35.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 1 -- 172.21.15.26:0/3438787162 wait complete. 2024-09-16T14:51:35.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 10 monclient: build_initial_monmap 2024-09-16T14:51:35.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 10 monclient: monmap: 2024-09-16T14:51:35.510 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:35.510 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:35.510 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:35.510 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:35.510 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:35.510 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:35.510 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:35.510 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:35.510 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:35.510 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:35.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 1 Processor -- start 2024-09-16T14:51:35.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 1 -- start start 2024-09-16T14:51:35.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.496+0000 7f8046765640 10 monclient: init 2024-09-16T14:51:35.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:35.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:51:35.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 10 monclient(hunting): picked mon.noname-b con 0x7f804009cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:35.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 10 monclient(hunting): picked mon.noname-c con 0x7f804009d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:35.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 10 monclient(hunting): picked mon.noname-a con 0x7f804009e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:35.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80400a2170 con 0x7f804009e7a0 2024-09-16T14:51:35.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80400a26a0 con 0x7f804009cdc0 2024-09-16T14:51:35.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8040121990 con 0x7f804009d1a0 2024-09-16T14:51:35.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:35.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 10 monclient(hunting): authenticate will time out at 1519.620206s 2024-09-16T14:51:35.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8037fff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f804009d1a0 legacy=0x7f804011e400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:35.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8044d02640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f804009e7a0 0x7f804011eb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56464/0 (socket says 172.21.15.26:56464) 2024-09-16T14:51:35.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8044d02640 1 -- 172.21.15.26:0/3083550419 learned_addr learned my addr 172.21.15.26:0/3083550419 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:35.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013263746 0 0) 0x7f80400a2170 con 0x7f804009e7a0 2024-09-16T14:51:35.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:51:35.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:35.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:35.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient(hunting): my global_id is 194214 2024-09-16T14:51:35.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8020003530 con 0x7f804009e7a0 2024-09-16T14:51:35.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4017407039 0 0) 0x7f8020003530 con 0x7f804009e7a0 2024-09-16T14:51:35.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80400a2170 con 0x7f804009e7a0 2024-09-16T14:51:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269562277 0 0) 0x7f80400a26a0 con 0x7f804009cdc0 2024-09-16T14:51:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:51:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient(hunting): my global_id is 184402 2024-09-16T14:51:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8020003530 con 0x7f804009cdc0 2024-09-16T14:51:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f80380043c0 con 0x7f804009e7a0 2024-09-16T14:51:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:35.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient(hunting): dump: 2024-09-16T14:51:35.517 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:35.517 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:35.517 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:35.517 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:35.517 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:35.517 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:35.517 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:35.518 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3385842053 0 0) 0x7f80400a2170 con 0x7f804009e7a0 2024-09-16T14:51:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 >> v1:172.21.15.26:6790/0 conn(0x7f804009d1a0 legacy=0x7f804011e400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 >> v1:172.21.15.189:6789/0 conn(0x7f804009cdc0 legacy=0x7f804011dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:51:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 1 monclient: found mon.a 2024-09-16T14:51:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8040121c70 con 0x7f804009e7a0 2024-09-16T14:51:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:51:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:51:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 5 monclient: authenticate success, global_id 194214 2024-09-16T14:51:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 10 monclient: _renew_subs 2024-09-16T14:51:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 1 -- 172.21.15.26:0/3083550419 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8040120260 con 0x7f804009e7a0 2024-09-16T14:51:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 10 monclient: _renew_subs 2024-09-16T14:51:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 1 -- 172.21.15.26:0/3083550419 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80401d2dd0 con 0x7f804009e7a0 2024-09-16T14:51:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8038004660 con 0x7f804009e7a0 2024-09-16T14:51:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f80380049f0 con 0x7f804009e7a0 2024-09-16T14:51:35.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:35.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:35.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 10 monclient: dump: 2024-09-16T14:51:35.520 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:35.520 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:35.520 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:35.520 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:35.520 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:35.520 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:35.520 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:35.520 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:35.520 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:35.520 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:35.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803effd640 4 set_mon_vals no callback set 2024-09-16T14:51:35.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8038004b50 con 0x7f804009e7a0 2024-09-16T14:51:35.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 169..257) v4 ==== 6745+0+0 (unknown 920221734 0 0) 0x7f8038043f60 con 0x7f804009e7a0 2024-09-16T14:51:35.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:35.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:35.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:35.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f8046765640 1 -- 172.21.15.26:0/3083550419 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8024005050 con 0x7f804009e7a0 2024-09-16T14:51:35.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.500+0000 7f803ffff640 10 monclient: get_auth_request con 0x7f8020036140 auth_method 0 2024-09-16T14:51:35.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.504+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8038014950 con 0x7f804009e7a0 2024-09-16T14:51:35.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.504+0000 7f803d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:35.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.504+0000 7f803d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.724+0000 7f8046765640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:51:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.724+0000 7f8046765640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:51:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.724+0000 7f8046765640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.724+0000 7f8046765640 1 -- 172.21.15.26:0/3083550419 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8024005880 con 0x7f804009e7a0 2024-09-16T14:51:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.724+0000 7f803d7fa640 1 -- 172.21.15.26:0/3083550419 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 4065000639) 0x7f803801ce90 con 0x7f804009e7a0 2024-09-16T14:51:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.724+0000 7f803d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:51:35.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.724+0000 7f803d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:35.728 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":84,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:51:35.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.728+0000 7f8046765640 1 -- 172.21.15.26:0/3083550419 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8020036140 legacy=0x7f8020038600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:35.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.728+0000 7f8046765640 10 monclient: shutdown 2024-09-16T14:51:35.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.728+0000 7f8046765640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:35.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.728+0000 7f8046765640 1 -- 172.21.15.26:0/3083550419 >> v1:172.21.15.26:6789/0 conn(0x7f804009e7a0 legacy=0x7f804011eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:35.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.728+0000 7f8046765640 1 -- 172.21.15.26:0/3083550419 shutdown_connections 2024-09-16T14:51:35.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.728+0000 7f8046765640 1 -- 172.21.15.26:0/3083550419 >> 172.21.15.26:0/3083550419 conn(0x7f804009fb50 msgr2=0x7f804009bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:35.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.728+0000 7f8046765640 1 -- 172.21.15.26:0/3083550419 shutdown_connections 2024-09-16T14:51:35.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:35.728+0000 7f8046765640 1 -- 172.21.15.26:0/3083550419 wait complete. 2024-09-16T14:51:38.747 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:51:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc0417ae640 10 monclient: get_monmap_and_config 2024-09-16T14:51:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc0417ae640 10 monclient: build_initial_monmap 2024-09-16T14:51:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc0417ae640 10 monclient: monmap: 2024-09-16T14:51:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:38.882442+0000 2024-09-16T14:51:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:38.882442+0000 2024-09-16T14:51:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:38.882 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc0417ae640 1 Processor -- start 2024-09-16T14:51:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc0417ae640 1 -- start start 2024-09-16T14:51:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc0417ae640 10 monclient: init 2024-09-16T14:51:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc0417ae640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc0417ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:51:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc0417ae640 10 monclient(hunting): picked mon.noname-a con 0x7fc03c09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:38.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc0417ae640 10 monclient(hunting): picked mon.noname-c con 0x7fc03c132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:38.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc0417ae640 10 monclient(hunting): picked mon.noname-b con 0x7fc03c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:38.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc0417ae640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c09d260 con 0x7fc03c09cb80 2024-09-16T14:51:38.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc0417ae640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c132a80 con 0x7fc03c132650 2024-09-16T14:51:38.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc0417ae640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c09cf60 con 0x7fc03c138ff0 2024-09-16T14:51:38.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc0417ae640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:38.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc0417ae640 10 monclient(hunting): authenticate will time out at 1523.002626s 2024-09-16T14:51:38.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc03b7fe640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fc03c138ff0 legacy=0x7fc03c13b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:38.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc03affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc03c09cb80 0x7fc03c0717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35146/0 (socket says 172.21.15.26:35146) 2024-09-16T14:51:38.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:38.880+0000 7fc03affd640 1 -- 172.21.15.26:0/498457143 learned_addr learned my addr 172.21.15.26:0/498457143 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:39.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:39.084+0000 7fc03b7fe640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6790/0 conn(0x7fc03c138ff0 legacy=0x7fc03c13b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:39.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:39.488+0000 7fc03b7fe640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6790/0 conn(0x7fc03c138ff0 legacy=0x7fc03c13b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:40.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:40.292+0000 7fc03b7fe640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6790/0 conn(0x7fc03c138ff0 legacy=0x7fc03c13b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:41.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:41.880+0000 7fc038ff9640 10 monclient(hunting): tick 2024-09-16T14:51:41.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:41.880+0000 7fc038ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:51:41.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:41.880+0000 7fc038ff9640 1 monclient(hunting): continuing hunt 2024-09-16T14:51:41.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:41.880+0000 7fc038ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:51:41.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:41.880+0000 7fc038ff9640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6790/0 conn(0x7fc03c138ff0 legacy=0x7fc03c13b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:41.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:41.880+0000 7fc038ff9640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.189:6789/0 conn(0x7fc03c132650 legacy=0x7fc03c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:41.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:41.880+0000 7fc038ff9640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6789/0 conn(0x7fc03c09cb80 legacy=0x7fc03c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:41.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:41.880+0000 7fc038ff9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:51:41.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:41.880+0000 7fc038ff9640 10 monclient(hunting): picked mon.noname-a con 0x7fc01c0036d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:41.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:41.880+0000 7fc038ff9640 10 monclient(hunting): picked mon.noname-c con 0x7fc01c007360 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:41.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:41.880+0000 7fc038ff9640 10 monclient(hunting): picked mon.noname-b con 0x7fc03c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:41.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:41.880+0000 7fc038ff9640 1 -- 172.21.15.26:0/498457143 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c09f2d0 con 0x7fc01c0036d0 2024-09-16T14:51:41.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:41.880+0000 7fc038ff9640 1 -- 172.21.15.26:0/498457143 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c09cf60 con 0x7fc01c007360 2024-09-16T14:51:41.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:41.880+0000 7fc038ff9640 1 -- 172.21.15.26:0/498457143 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01c010860 con 0x7fc03c138ff0 2024-09-16T14:51:41.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:41.880+0000 7fc038ff9640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:41.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:41.880+0000 7fc03b7fe640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6790/0 conn(0x7fc03c138ff0 legacy=0x7fc01c00d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:42.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:42.084+0000 7fc03b7fe640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6790/0 conn(0x7fc03c138ff0 legacy=0x7fc01c00d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:42.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:42.488+0000 7fc03b7fe640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6790/0 conn(0x7fc03c138ff0 legacy=0x7fc01c00d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:43.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:43.292+0000 7fc03b7fe640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6790/0 conn(0x7fc03c138ff0 legacy=0x7fc01c00d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:44.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:44.880+0000 7fc038ff9640 10 monclient(hunting): tick 2024-09-16T14:51:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:44.880+0000 7fc038ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:51:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:44.880+0000 7fc038ff9640 1 monclient(hunting): continuing hunt 2024-09-16T14:51:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:44.880+0000 7fc038ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:51:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:44.880+0000 7fc038ff9640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6790/0 conn(0x7fc03c138ff0 legacy=0x7fc01c00d0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:44.880+0000 7fc038ff9640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.189:6789/0 conn(0x7fc01c007360 legacy=0x7fc01c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:44.880+0000 7fc038ff9640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6789/0 conn(0x7fc01c0036d0 legacy=0x7fc01c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:44.880+0000 7fc038ff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:51:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:44.880+0000 7fc038ff9640 10 monclient(hunting): picked mon.noname-b con 0x7fc03c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:44.880+0000 7fc038ff9640 10 monclient(hunting): picked mon.noname-c con 0x7fc01c010af0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:44.880+0000 7fc038ff9640 10 monclient(hunting): picked mon.noname-a con 0x7fc01c009fc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:44.880+0000 7fc038ff9640 1 -- 172.21.15.26:0/498457143 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01c011b10 con 0x7fc01c009fc0 2024-09-16T14:51:44.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:44.880+0000 7fc038ff9640 1 -- 172.21.15.26:0/498457143 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01c010860 con 0x7fc01c010af0 2024-09-16T14:51:44.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:44.880+0000 7fc038ff9640 1 -- 172.21.15.26:0/498457143 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01c00a3c0 con 0x7fc03c138ff0 2024-09-16T14:51:44.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:44.880+0000 7fc038ff9640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:44.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:44.880+0000 7fc03affd640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6790/0 conn(0x7fc03c138ff0 legacy=0x7fc01c00d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:45.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:45.084+0000 7fc03affd640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6790/0 conn(0x7fc03c138ff0 legacy=0x7fc01c00d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:45.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:45.488+0000 7fc03affd640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6790/0 conn(0x7fc03c138ff0 legacy=0x7fc01c00d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:46.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.292+0000 7fc03affd640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6790/0 conn(0x7fc03c138ff0 legacy=0x7fc01c00d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:46.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 1 -- 172.21.15.26:0/498457143 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981650102 0 0) 0x7fc01c011b10 con 0x7fc01c009fc0 2024-09-16T14:51:46.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:46.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:46.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:46.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 10 monclient(hunting): my global_id is 194217 2024-09-16T14:51:46.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 1 -- 172.21.15.26:0/498457143 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc028003530 con 0x7fc01c009fc0 2024-09-16T14:51:46.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 1 -- 172.21.15.26:0/498457143 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2353075498 0 0) 0x7fc028003530 con 0x7fc01c009fc0 2024-09-16T14:51:46.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6790/0 conn(0x7fc03c138ff0 legacy=0x7fc01c00d0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:46.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.189:6789/0 conn(0x7fc01c010af0 legacy=0x7fc01c010f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:46.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:51:46.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 1 monclient: found mon.noname-a 2024-09-16T14:51:46.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:46.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:51:46.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 1 -- 172.21.15.26:0/498457143 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01c005ad0 con 0x7fc01c009fc0 2024-09-16T14:51:46.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 10 monclient: _finish_auth 0 2024-09-16T14:51:46.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:51:46.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:46.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc0417ae640 5 monclient: authenticate success, global_id 194217 2024-09-16T14:51:46.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc0417ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:46.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 1 -- 172.21.15.26:0/498457143 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc034004570 con 0x7fc01c009fc0 2024-09-16T14:51:46.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:46.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:46.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 10 monclient: dump: 2024-09-16T14:51:46.568 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:46.569 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:46.569 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.569 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.569 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:46.569 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:46.569 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:46.570 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:46.570 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:46.570 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:46.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc0417ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:46.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 1 -- 172.21.15.26:0/498457143 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc034004790 con 0x7fc01c009fc0 2024-09-16T14:51:46.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:46.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 1 -- 172.21.15.26:0/498457143 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc034004b60 con 0x7fc01c009fc0 2024-09-16T14:51:46.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc0417ae640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:46.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc0417ae640 4 set_mon_vals no callback set 2024-09-16T14:51:46.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:46.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:46.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.560+0000 7fc039ffb640 10 monclient: dump: 2024-09-16T14:51:46.572 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:46.572 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:46.572 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.572 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.572 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:46.573 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:46.573 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:46.573 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:46.573 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:46.573 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:46.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 10 monclient: shutdown 2024-09-16T14:51:46.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:46.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 1 -- 172.21.15.26:0/498457143 >> v1:172.21.15.26:6789/0 conn(0x7fc01c009fc0 legacy=0x7fc01c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:46.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 1 -- 172.21.15.26:0/498457143 shutdown_connections 2024-09-16T14:51:46.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 1 -- 172.21.15.26:0/498457143 >> 172.21.15.26:0/498457143 conn(0x7fc03c09fb50 msgr2=0x7fc03c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:46.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 1 -- 172.21.15.26:0/498457143 shutdown_connections 2024-09-16T14:51:46.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 1 -- 172.21.15.26:0/498457143 wait complete. 2024-09-16T14:51:46.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 10 monclient: build_initial_monmap 2024-09-16T14:51:46.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 10 monclient: monmap: 2024-09-16T14:51:46.580 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:46.580 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:46.580 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:46.580 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:46.580 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:46.580 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:46.581 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:46.581 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:46.581 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:46.581 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:46.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 1 Processor -- start 2024-09-16T14:51:46.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 1 -- start start 2024-09-16T14:51:46.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 10 monclient: init 2024-09-16T14:51:46.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:46.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:51:46.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 10 monclient(hunting): picked mon.noname-a con 0x7fc01c009fc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:46.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 10 monclient(hunting): picked mon.noname-c con 0x7fc01c010af0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:46.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 10 monclient(hunting): picked mon.noname-b con 0x7fc03c138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:46.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c1375e0 con 0x7fc01c009fc0 2024-09-16T14:51:46.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c138c60 con 0x7fc03c138ff0 2024-09-16T14:51:46.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c0a1e60 con 0x7fc01c010af0 2024-09-16T14:51:46.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:46.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc0417ae640 10 monclient(hunting): authenticate will time out at 1530.687371s 2024-09-16T14:51:46.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc03a7fc640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fc01c010af0 legacy=0x7fc03c12bc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:46.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc03affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc01c009fc0 0x7fc03c1c9580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35172/0 (socket says 172.21.15.26:35172) 2024-09-16T14:51:46.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.564+0000 7fc03affd640 1 -- 172.21.15.26:0/1820785159 learned_addr learned my addr 172.21.15.26:0/1820785159 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:46.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.568+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624878481 0 0) 0x7fc03c138c60 con 0x7fc03c138ff0 2024-09-16T14:51:46.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.568+0000 7fc0337fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:46.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.568+0000 7fc0337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:46.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.568+0000 7fc0337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:46.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.568+0000 7fc0337fe640 10 monclient(hunting): my global_id is 184405 2024-09-16T14:51:46.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.568+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc01c011a70 con 0x7fc03c138ff0 2024-09-16T14:51:46.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417523964 0 0) 0x7fc03c1375e0 con 0x7fc01c009fc0 2024-09-16T14:51:46.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:46.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:46.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:46.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 10 monclient(hunting): my global_id is 194220 2024-09-16T14:51:46.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc03c138c60 con 0x7fc01c009fc0 2024-09-16T14:51:46.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3497915673 0 0) 0x7fc03c138c60 con 0x7fc01c009fc0 2024-09-16T14:51:46.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc03c1375e0 con 0x7fc01c009fc0 2024-09-16T14:51:46.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc024001770 con 0x7fc01c009fc0 2024-09-16T14:51:46.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:46.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:46.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 10 monclient(hunting): dump: 2024-09-16T14:51:46.584 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:46.584 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:46.584 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.584 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.584 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:46.584 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:46.585 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:46.585 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:46.585 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:46.585 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:46.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:46.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1002176563 0 0) 0x7fc03c1375e0 con 0x7fc01c009fc0 2024-09-16T14:51:46.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 >> v1:172.21.15.26:6790/0 conn(0x7fc01c010af0 legacy=0x7fc03c12bc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:46.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 >> v1:172.21.15.189:6789/0 conn(0x7fc03c138ff0 legacy=0x7fc03c1d18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:46.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 10 monclient: _finish_hunting 0 2024-09-16T14:51:46.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 1 monclient: found mon.a 2024-09-16T14:51:46.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:46.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:46.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c070d40 con 0x7fc01c009fc0 2024-09-16T14:51:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 10 monclient: _finish_auth 0 2024-09-16T14:51:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 10 monclient: _check_auth_tickets 2024-09-16T14:51:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0417ae640 5 monclient: authenticate success, global_id 194220 2024-09-16T14:51:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0417ae640 10 monclient: _renew_subs 2024-09-16T14:51:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0417ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0417ae640 1 -- 172.21.15.26:0/1820785159 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03c138980 con 0x7fc01c009fc0 2024-09-16T14:51:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0417ae640 10 monclient: _renew_subs 2024-09-16T14:51:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0417ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0417ae640 1 -- 172.21.15.26:0/1820785159 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03c12c320 con 0x7fc01c009fc0 2024-09-16T14:51:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc024001c80 con 0x7fc01c009fc0 2024-09-16T14:51:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc024004990 con 0x7fc01c009fc0 2024-09-16T14:51:46.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:46.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:46.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc0337fe640 10 monclient: dump: 2024-09-16T14:51:46.587 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:46.587 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:46.587 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.587 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.587 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:46.587 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:46.587 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:46.587 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:46.587 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:46.587 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:46.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.572+0000 7fc038ff9640 4 set_mon_vals no callback set 2024-09-16T14:51:46.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.576+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc024004af0 con 0x7fc01c009fc0 2024-09-16T14:51:46.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.576+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 169..257) v4 ==== 6745+0+0 (unknown 920221734 0 0) 0x7fc024043bb0 con 0x7fc01c009fc0 2024-09-16T14:51:46.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.576+0000 7fc0417ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:46.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.576+0000 7fc0417ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:46.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.576+0000 7fc0417ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:46.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.576+0000 7fc0417ae640 1 -- 172.21.15.26:0/1820785159 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03c12c5d0 con 0x7fc01c009fc0 2024-09-16T14:51:46.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.576+0000 7fc03a7fc640 10 monclient: get_auth_request con 0x7fc01c0377a0 auth_method 0 2024-09-16T14:51:46.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.580+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc024014490 con 0x7fc01c009fc0 2024-09-16T14:51:46.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.580+0000 7fc0337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:46.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.580+0000 7fc0337fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:46.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.804+0000 7fc0417ae640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:51:46.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.804+0000 7fc0417ae640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:51:46.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.804+0000 7fc0417ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:46.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.804+0000 7fc0417ae640 1 -- 172.21.15.26:0/1820785159 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc03c1383c0 con 0x7fc01c009fc0 2024-09-16T14:51:46.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.804+0000 7fc0337fe640 1 -- 172.21.15.26:0/1820785159 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1382421418) 0x7fc024014260 con 0x7fc01c009fc0 2024-09-16T14:51:46.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.804+0000 7fc0337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:51:46.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.804+0000 7fc0337fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:46.808 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":86,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:51:46.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.808+0000 7fc0417ae640 1 -- 172.21.15.26:0/1820785159 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc01c0377a0 legacy=0x7fc01c039c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:46.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.808+0000 7fc0417ae640 10 monclient: shutdown 2024-09-16T14:51:46.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.808+0000 7fc0417ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:46.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.808+0000 7fc0417ae640 1 -- 172.21.15.26:0/1820785159 >> v1:172.21.15.26:6789/0 conn(0x7fc01c009fc0 legacy=0x7fc03c1c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:46.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.808+0000 7fc0417ae640 1 -- 172.21.15.26:0/1820785159 shutdown_connections 2024-09-16T14:51:46.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.808+0000 7fc0417ae640 1 -- 172.21.15.26:0/1820785159 >> 172.21.15.26:0/1820785159 conn(0x7fc03c09fb50 msgr2=0x7fc03c09b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:46.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.808+0000 7fc0417ae640 1 -- 172.21.15.26:0/1820785159 shutdown_connections 2024-09-16T14:51:46.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.808+0000 7fc0417ae640 1 -- 172.21.15.26:0/1820785159 wait complete. 2024-09-16T14:51:46.828 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:51:46.828 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T14:51:46.828 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:51:46.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 10 monclient: get_monmap_and_config 2024-09-16T14:51:46.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 10 monclient: build_initial_monmap 2024-09-16T14:51:46.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 10 monclient: monmap: 2024-09-16T14:51:46.957 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:46.957 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:46.957 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:46.957808+0000 2024-09-16T14:51:46.957 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:46.957808+0000 2024-09-16T14:51:46.957 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:46.957 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:46.957 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:46.957 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:46.957 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:46.957 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:46.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 1 Processor -- start 2024-09-16T14:51:46.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 1 -- start start 2024-09-16T14:51:46.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 10 monclient: init 2024-09-16T14:51:46.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:46.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:51:46.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 10 monclient(hunting): picked mon.noname-b con 0x7f2ec80a1ca0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:46.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 10 monclient(hunting): picked mon.noname-a con 0x7f2ec80a2790 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:46.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 10 monclient(hunting): picked mon.noname-c con 0x7f2ec809ff50 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:46.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec809f520 con 0x7f2ec80a2790 2024-09-16T14:51:46.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec80a0360 con 0x7f2ec809ff50 2024-09-16T14:51:46.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec809e7a0 con 0x7f2ec80a1ca0 2024-09-16T14:51:46.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:46.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 10 monclient(hunting): authenticate will time out at 1531.077817s 2024-09-16T14:51:46.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec659c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2ec80a1ca0 legacy=0x7f2ec80a2080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:46.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ebdd9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2ec80a2790 0x7f2ec8139e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39158/0 (socket says 172.21.15.26:39158) 2024-09-16T14:51:46.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ebdd9b640 1 -- 172.21.15.26:0/3591586657 learned_addr learned my addr 172.21.15.26:0/3591586657 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:46.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 1 -- 172.21.15.26:0/3591586657 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279186261 0 0) 0x7f2ec809f520 con 0x7f2ec80a2790 2024-09-16T14:51:46.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:51:46.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:46.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:46.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 10 monclient(hunting): my global_id is 194223 2024-09-16T14:51:46.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 1 -- 172.21.15.26:0/3591586657 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb4003530 con 0x7f2ec80a2790 2024-09-16T14:51:46.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 1 -- 172.21.15.26:0/3591586657 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3839481134 0 0) 0x7f2eb4003530 con 0x7f2ec80a2790 2024-09-16T14:51:46.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 1 -- 172.21.15.26:0/3591586657 >> v1:172.21.15.26:6790/0 conn(0x7f2ec80a1ca0 legacy=0x7f2ec80a2080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:46.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 1 -- 172.21.15.26:0/3591586657 >> v1:172.21.15.189:6789/0 conn(0x7f2ec809ff50 legacy=0x7f2ec813d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:46.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:51:46.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 1 monclient: found mon.noname-a 2024-09-16T14:51:46.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:46.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:51:46.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 1 -- 172.21.15.26:0/3591586657 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec809ec80 con 0x7f2ec80a2790 2024-09-16T14:51:46.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 10 monclient: _finish_auth 0 2024-09-16T14:51:46.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:51:46.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:46.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 1 -- 172.21.15.26:0/3591586657 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2eb0001620 con 0x7f2ec80a2790 2024-09-16T14:51:46.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:46.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:46.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 10 monclient: dump: 2024-09-16T14:51:46.969 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:46.969 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:46.969 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.970 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.970 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:46.970 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:46.970 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:46.970 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:46.970 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:46.970 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:46.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 5 monclient: authenticate success, global_id 194223 2024-09-16T14:51:46.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:46.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 1 -- 172.21.15.26:0/3591586657 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2eb0001940 con 0x7f2ec80a2790 2024-09-16T14:51:46.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:46.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 1 -- 172.21.15.26:0/3591586657 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2eb0001cd0 con 0x7f2ec80a2790 2024-09-16T14:51:46.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:46.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 4 set_mon_vals no callback set 2024-09-16T14:51:46.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:46.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:46.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ec5d9b640 10 monclient: dump: 2024-09-16T14:51:46.971 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:46.971 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:46.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.971 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:46.971 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:46.972 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:46.972 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:46.972 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:46.972 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:46.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 10 monclient: shutdown 2024-09-16T14:51:46.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:46.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 1 -- 172.21.15.26:0/3591586657 >> v1:172.21.15.26:6789/0 conn(0x7f2ec80a2790 legacy=0x7f2ec8139e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:46.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 1 -- 172.21.15.26:0/3591586657 shutdown_connections 2024-09-16T14:51:46.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 1 -- 172.21.15.26:0/3591586657 >> 172.21.15.26:0/3591586657 conn(0x7f2ec80a3da0 msgr2=0x7f2ec80a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:46.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 1 -- 172.21.15.26:0/3591586657 shutdown_connections 2024-09-16T14:51:46.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 1 -- 172.21.15.26:0/3591586657 wait complete. 2024-09-16T14:51:46.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 10 monclient: build_initial_monmap 2024-09-16T14:51:46.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.956+0000 7f2ecc90a640 10 monclient: monmap: 2024-09-16T14:51:46.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:46.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:46.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:46.973 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:46.973 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:46.973 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:46.973 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:46.973 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:46.973 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:46.973 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:46.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 1 Processor -- start 2024-09-16T14:51:46.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 1 -- start start 2024-09-16T14:51:46.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 10 monclient: init 2024-09-16T14:51:46.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:46.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:51:46.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 10 monclient(hunting): picked mon.noname-a con 0x7f2ec809ff50 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:46.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 10 monclient(hunting): picked mon.noname-c con 0x7f2ec80a1ca0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:46.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 10 monclient(hunting): picked mon.noname-b con 0x7f2ec80a2790 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:46.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec8133d20 con 0x7f2ec809ff50 2024-09-16T14:51:46.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec80a5230 con 0x7f2ec80a2790 2024-09-16T14:51:46.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec80a5780 con 0x7f2ec80a1ca0 2024-09-16T14:51:46.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebdd9b640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2ec80a1ca0 legacy=0x7f2ec8133610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:46.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:46.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 10 monclient(hunting): authenticate will time out at 1531.080949s 2024-09-16T14:51:46.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ec659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2ec809ff50 0x7f2ec8132f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39168/0 (socket says 172.21.15.26:39168) 2024-09-16T14:51:46.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ec659c640 1 -- 172.21.15.26:0/4004262520 learned_addr learned my addr 172.21.15.26:0/4004262520 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:46.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283339711 0 0) 0x7f2ec8133d20 con 0x7f2ec809ff50 2024-09-16T14:51:46.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:46.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:46.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:46.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient(hunting): my global_id is 194226 2024-09-16T14:51:46.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea8003530 con 0x7f2ec809ff50 2024-09-16T14:51:46.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3144754162 0 0) 0x7f2ea8003530 con 0x7f2ec809ff50 2024-09-16T14:51:46.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ec8133d20 con 0x7f2ec809ff50 2024-09-16T14:51:46.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2eb8004160 con 0x7f2ec809ff50 2024-09-16T14:51:46.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:46.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:46.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient(hunting): dump: 2024-09-16T14:51:46.975 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:46.975 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:46.976 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.976 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.976 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:46.976 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:46.976 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:46.976 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:46.976 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:46.976 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:46.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:46.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1980143052 0 0) 0x7f2ec8133d20 con 0x7f2ec809ff50 2024-09-16T14:51:46.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 >> v1:172.21.15.26:6790/0 conn(0x7f2ec80a1ca0 legacy=0x7f2ec8133610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:46.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 >> v1:172.21.15.189:6789/0 conn(0x7f2ec80a2790 legacy=0x7f2ec81d13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:46.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:51:46.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 monclient: found mon.a 2024-09-16T14:51:46.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:46.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:46.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec809e490 con 0x7f2ec809ff50 2024-09-16T14:51:46.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient: _finish_auth 0 2024-09-16T14:51:46.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:51:46.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:46.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 5 monclient: authenticate success, global_id 194226 2024-09-16T14:51:46.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 10 monclient: _renew_subs 2024-09-16T14:51:46.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:46.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 1 -- 172.21.15.26:0/4004262520 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec8134ad0 con 0x7f2ec809ff50 2024-09-16T14:51:46.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 10 monclient: _renew_subs 2024-09-16T14:51:46.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:46.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 1 -- 172.21.15.26:0/4004262520 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec81356a0 con 0x7f2ec809ff50 2024-09-16T14:51:46.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2eb8004410 con 0x7f2ec809ff50 2024-09-16T14:51:46.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:46.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2eb80047a0 con 0x7f2ec809ff50 2024-09-16T14:51:46.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:46.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:46.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ec559a640 4 set_mon_vals no callback set 2024-09-16T14:51:46.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient: dump: 2024-09-16T14:51:46.978 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:46.978 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:46.979 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.979 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.979 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:46.979 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:46.979 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:46.979 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:46.979 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:46.979 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:46.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:46.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ecc90a640 1 -- 172.21.15.26:0/4004262520 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2eac006590 con 0x7f2eac002830 2024-09-16T14:51:46.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2eb8004900 con 0x7f2ec809ff50 2024-09-16T14:51:46.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 169..257) v4 ==== 6745+0+0 (unknown 920221734 0 0) 0x7f2eb8043e10 con 0x7f2ec809ff50 2024-09-16T14:51:46.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180976776 0 0) 0x7f2eac006590 con 0x7f2eac002830 2024-09-16T14:51:46.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:46.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:46.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:46.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient(hunting): my global_id is 194229 2024-09-16T14:51:46.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec8133d20 con 0x7f2eac002830 2024-09-16T14:51:46.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4282564207 0 0) 0x7f2ec8133d20 con 0x7f2eac002830 2024-09-16T14:51:46.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2eac006590 con 0x7f2eac002830 2024-09-16T14:51:46.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2eb0001c50 con 0x7f2eac002830 2024-09-16T14:51:46.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:46.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:46.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient: dump: 2024-09-16T14:51:46.981 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:46.981 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:46.981 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.981 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.981 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:46.981 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:46.981 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:46.981 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:46.982 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:46.982 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:46.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ec6d9d640 10 monclient: get_auth_request con 0x7f2ea8036000 auth_method 0 2024-09-16T14:51:46.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2346610397 0 0) 0x7f2eac006590 con 0x7f2eac002830 2024-09-16T14:51:46.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2eac00a0e0 con 0x7f2eac002830 2024-09-16T14:51:46.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2eb00015a0 con 0x7f2eac002830 2024-09-16T14:51:46.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:46.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:46.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 >> v1:172.21.15.26:6789/0 conn(0x7f2eac002830 legacy=0x7f2eac004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:46.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.960+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 >> v1:172.21.15.26:6789/0 conn(0x7f2eac002830 legacy=0x7f2eac004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:46.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ecc90a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:51:46.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ecc90a640 1 -- 172.21.15.26:0/4004262520 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2eac009d80 con 0x7f2eac0090a0 2024-09-16T14:51:46.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048198390 0 0) 0x7f2eac009d80 con 0x7f2eac0090a0 2024-09-16T14:51:46.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:51:46.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:46.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:46.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 10 monclient(hunting): my global_id is 194232 2024-09-16T14:51:46.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eac006590 con 0x7f2eac0090a0 2024-09-16T14:51:46.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3478788624 0 0) 0x7f2eac006590 con 0x7f2eac0090a0 2024-09-16T14:51:46.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2eac009d80 con 0x7f2eac0090a0 2024-09-16T14:51:46.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2eb0004ce0 con 0x7f2eac0090a0 2024-09-16T14:51:46.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:46.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:46.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 10 monclient: dump: 2024-09-16T14:51:46.985 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:46.985 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:46.985 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.985 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:46.985 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:46.985 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:46.985 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:46.985 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:46.985 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:46.985 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2010858074 0 0) 0x7f2eac009d80 con 0x7f2eac0090a0 2024-09-16T14:51:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2eac00c290 con 0x7f2eac0090a0 2024-09-16T14:51:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4200 (unknown 0 0 1091327416) 0x7f2eb0007040 con 0x7f2eac0090a0 2024-09-16T14:51:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:51:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 >> v1:172.21.15.26:6789/0 conn(0x7f2eac0090a0 legacy=0x7f2eac009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ebf7fe640 1 -- 172.21.15.26:0/4004262520 >> v1:172.21.15.26:6789/0 conn(0x7f2eac0090a0 legacy=0x7f2eac009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:46.986 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:51:46.986 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:51:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:51:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:51:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 86, 2024-09-16T14:51:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:51:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:51:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: 1 2024-09-16T14:51:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:51:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:51:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:51:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:46.987 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:51:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:51:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:46.988 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:51:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:51:46.989 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:51:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:51:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:51:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:51:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:51:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:51:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:51:46.990 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:51:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:51:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:51:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:51:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:51:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:46.991 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:51:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:51:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:51:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:51:46.992 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:51:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:51:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:51:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:46.993 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:51:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:51:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:51:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:46.994 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:51:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:51:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:51:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:51:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:46.995 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:51:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:51:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:51:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:46.996 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:51:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:51:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:51:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:51:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:46.997 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:51:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:51:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:51:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:51:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:46.998 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:51:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:46.999 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:51:46.999 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:51:47.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ecc90a640 1 -- 172.21.15.26:0/4004262520 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2ea8036000 legacy=0x7f2ea80384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:47.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ecc90a640 10 monclient: shutdown 2024-09-16T14:51:47.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ecc90a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:47.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ecc90a640 1 -- 172.21.15.26:0/4004262520 >> v1:172.21.15.26:6789/0 conn(0x7f2ec809ff50 legacy=0x7f2ec8132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:47.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ec6d9d640 1 -- 172.21.15.26:0/4004262520 reap_dead start 2024-09-16T14:51:47.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ec6d9d640 1 -- 172.21.15.26:0/4004262520 reap_dead start 2024-09-16T14:51:47.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ecc90a640 1 -- 172.21.15.26:0/4004262520 shutdown_connections 2024-09-16T14:51:47.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ecc90a640 1 -- 172.21.15.26:0/4004262520 >> 172.21.15.26:0/4004262520 conn(0x7f2ec80a3da0 msgr2=0x7f2ec80a6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:47.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ecc90a640 1 -- 172.21.15.26:0/4004262520 shutdown_connections 2024-09-16T14:51:47.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:46.968+0000 7f2ecc90a640 1 -- 172.21.15.26:0/4004262520 wait complete. 2024-09-16T14:51:47.001 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:51:47.002 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T14:51:47.002 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:51:47.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 10 monclient: get_monmap_and_config 2024-09-16T14:51:47.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 10 monclient: build_initial_monmap 2024-09-16T14:51:47.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 10 monclient: monmap: 2024-09-16T14:51:47.129 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:47.129 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:47.129 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:51:47.130108+0000 2024-09-16T14:51:47.129 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:51:47.130108+0000 2024-09-16T14:51:47.129 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:47.129 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:47.129 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:47.130 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:51:47.130 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:51:47.130 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:47.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 1 Processor -- start 2024-09-16T14:51:47.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 1 -- start start 2024-09-16T14:51:47.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 10 monclient: init 2024-09-16T14:51:47.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:47.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:51:47.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 10 monclient(hunting): picked mon.noname-a con 0x7fdce409a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:47.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 10 monclient(hunting): picked mon.noname-b con 0x7fdce409d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:47.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 10 monclient(hunting): picked mon.noname-c con 0x7fdce409b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:47.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce409dc60 con 0x7fdce409a550 2024-09-16T14:51:47.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce4099ae0 con 0x7fdce409b040 2024-09-16T14:51:47.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce409b8f0 con 0x7fdce409d1b0 2024-09-16T14:51:47.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:47.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 10 monclient(hunting): authenticate will time out at 1531.250135s 2024-09-16T14:51:47.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdce8f36640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fdce409d1b0 legacy=0x7fdce4136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:47.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdce9737640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdce409a550 0x7fdce409a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39190/0 (socket says 172.21.15.26:39190) 2024-09-16T14:51:47.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdce9737640 1 -- 172.21.15.26:0/731433534 learned_addr learned my addr 172.21.15.26:0/731433534 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:47.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 1 -- 172.21.15.26:0/731433534 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968994680 0 0) 0x7fdce409dc60 con 0x7fdce409a550 2024-09-16T14:51:47.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:47.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:47.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:47.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient(hunting): my global_id is 194235 2024-09-16T14:51:47.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 1 -- 172.21.15.26:0/731433534 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd0005740 con 0x7fdce409a550 2024-09-16T14:51:47.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 1 -- 172.21.15.26:0/731433534 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766833338 0 0) 0x7fdce4099ae0 con 0x7fdce409b040 2024-09-16T14:51:47.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:51:47.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:47.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:47.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient(hunting): my global_id is 184414 2024-09-16T14:51:47.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 1 -- 172.21.15.26:0/731433534 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce409dc60 con 0x7fdce409b040 2024-09-16T14:51:47.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 1 -- 172.21.15.26:0/731433534 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 259552790 0 0) 0x7fdcd0005740 con 0x7fdce409a550 2024-09-16T14:51:47.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 1 -- 172.21.15.26:0/731433534 >> v1:172.21.15.26:6790/0 conn(0x7fdce409d1b0 legacy=0x7fdce4136a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:51:47.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 1 -- 172.21.15.26:0/731433534 >> v1:172.21.15.189:6789/0 conn(0x7fdce409b040 legacy=0x7fdce413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:47.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient: _finish_hunting 0 2024-09-16T14:51:47.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 1 monclient: found mon.noname-a 2024-09-16T14:51:47.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:47.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:51:47.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 1 -- 172.21.15.26:0/731433534 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce409bbd0 con 0x7fdce409a550 2024-09-16T14:51:47.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient: _finish_auth 0 2024-09-16T14:51:47.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient: _check_auth_tickets 2024-09-16T14:51:47.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:47.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 1 -- 172.21.15.26:0/731433534 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdcdc0019a0 con 0x7fdce409a550 2024-09-16T14:51:47.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 5 monclient: authenticate success, global_id 194235 2024-09-16T14:51:47.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:47.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:47.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:47.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient: dump: 2024-09-16T14:51:47.139 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:47.139 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:47.139 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:47.139 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:47.140 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:47.140 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:47.140 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:47.140 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:47.140 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:47.140 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:47.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:51:47.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 1 -- 172.21.15.26:0/731433534 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcdc001bc0 con 0x7fdce409a550 2024-09-16T14:51:47.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:47.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 1 -- 172.21.15.26:0/731433534 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdcdc001f50 con 0x7fdce409a550 2024-09-16T14:51:47.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:47.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:47.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdcdbfff640 10 monclient: dump: 2024-09-16T14:51:47.141 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:47.141 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:47.141 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:47.141 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:47.141 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:47.141 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:47.141 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:47.141 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:47.141 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:47.141 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:47.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 10 monclient: get_monmap_and_config success 2024-09-16T14:51:47.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 4 set_mon_vals no callback set 2024-09-16T14:51:47.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 10 monclient: shutdown 2024-09-16T14:51:47.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:47.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 1 -- 172.21.15.26:0/731433534 >> v1:172.21.15.26:6789/0 conn(0x7fdce409a550 legacy=0x7fdce409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:47.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 1 -- 172.21.15.26:0/731433534 shutdown_connections 2024-09-16T14:51:47.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 1 -- 172.21.15.26:0/731433534 >> 172.21.15.26:0/731433534 conn(0x7fdce409fb50 msgr2=0x7fdce409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:47.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 1 -- 172.21.15.26:0/731433534 shutdown_connections 2024-09-16T14:51:47.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.128+0000 7fdceb99b640 1 -- 172.21.15.26:0/731433534 wait complete. 2024-09-16T14:51:47.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 10 monclient: build_initial_monmap 2024-09-16T14:51:47.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 10 monclient: monmap: 2024-09-16T14:51:47.142 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:51:47.142 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:51:47.142 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:51:47.142 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:51:47.143 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:51:47.143 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:51:47.143 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:51:47.143 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:51:47.143 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:51:47.143 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:47.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 1 Processor -- start 2024-09-16T14:51:47.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 1 -- start start 2024-09-16T14:51:47.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 10 monclient: init 2024-09-16T14:51:47.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 10 monclient: _reopen_session rank -1 2024-09-16T14:51:47.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:51:47.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 10 monclient(hunting): picked mon.noname-b con 0x7fdce409a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:51:47.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 10 monclient(hunting): picked mon.noname-c con 0x7fdce409b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:51:47.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 10 monclient(hunting): picked mon.noname-a con 0x7fdce409d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:51:47.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce41a88a0 con 0x7fdce409d1b0 2024-09-16T14:51:47.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce40a2360 con 0x7fdce409a550 2024-09-16T14:51:47.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce409a240 con 0x7fdce409b040 2024-09-16T14:51:47.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 10 monclient(hunting): _renew_subs 2024-09-16T14:51:47.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 10 monclient(hunting): authenticate will time out at 1531.253254s 2024-09-16T14:51:47.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdce8f36640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fdce409b040 legacy=0x7fdce41ce410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:51:47.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdce9f38640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdce409d1b0 0x7fdce41d1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39198/0 (socket says 172.21.15.26:39198) 2024-09-16T14:51:47.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdce9f38640 1 -- 172.21.15.26:0/2886851921 learned_addr learned my addr 172.21.15.26:0/2886851921 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:51:47.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357120484 0 0) 0x7fdce41a88a0 con 0x7fdce409d1b0 2024-09-16T14:51:47.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:47.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:47.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient(hunting): my global_id is 194238 2024-09-16T14:51:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc8003530 con 0x7fdce409d1b0 2024-09-16T14:51:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3469948878 0 0) 0x7fdcc8003530 con 0x7fdce409d1b0 2024-09-16T14:51:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdce41a88a0 con 0x7fdce409d1b0 2024-09-16T14:51:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdce00018e0 con 0x7fdce409d1b0 2024-09-16T14:51:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:51:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient(hunting): dump: 2024-09-16T14:51:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:47.145 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:47.146 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:47.146 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:47.146 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:47.146 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:47.146 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:47.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:51:47.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017885817 0 0) 0x7fdce40a2360 con 0x7fdce409a550 2024-09-16T14:51:47.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:47.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:47.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:47.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient(hunting): my global_id is 184417 2024-09-16T14:51:47.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc8003530 con 0x7fdce409a550 2024-09-16T14:51:47.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 585775452 0 0) 0x7fdce41a88a0 con 0x7fdce409d1b0 2024-09-16T14:51:47.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 >> v1:172.21.15.26:6790/0 conn(0x7fdce409b040 legacy=0x7fdce41ce410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:51:47.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 >> v1:172.21.15.189:6789/0 conn(0x7fdce409a550 legacy=0x7fdce41c9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:47.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:51:47.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 monclient: found mon.a 2024-09-16T14:51:47.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:51:47.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:47.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce41a5b60 con 0x7fdce409d1b0 2024-09-16T14:51:47.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient: _finish_auth 0 2024-09-16T14:51:47.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:51:47.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:51:47.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 5 monclient: authenticate success, global_id 194238 2024-09-16T14:51:47.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 10 monclient: _renew_subs 2024-09-16T14:51:47.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:47.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 1 -- 172.21.15.26:0/2886851921 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce41a64b0 con 0x7fdce409d1b0 2024-09-16T14:51:47.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 10 monclient: _renew_subs 2024-09-16T14:51:47.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:51:47.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 1 -- 172.21.15.26:0/2886851921 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce41b5340 con 0x7fdce409d1b0 2024-09-16T14:51:47.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdce0001cb0 con 0x7fdce409d1b0 2024-09-16T14:51:47.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:51:47.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdce0003af0 con 0x7fdce409d1b0 2024-09-16T14:51:47.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:47.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:51:47.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcdb7fe640 4 set_mon_vals no callback set 2024-09-16T14:51:47.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient: dump: 2024-09-16T14:51:47.148 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:47.148 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:47.149 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:47.149 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:47.149 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:47.149 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:47.149 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:47.149 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:47.149 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:47.149 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:47.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fdce0003c50 con 0x7fdce409d1b0 2024-09-16T14:51:47.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:47.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdceb99b640 1 -- 172.21.15.26:0/2886851921 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdcc4006590 con 0x7fdcc4002830 2024-09-16T14:51:47.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 169..257) v4 ==== 6745+0+0 (unknown 920221734 0 0) 0x7fdce0043df0 con 0x7fdce409d1b0 2024-09-16T14:51:47.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdce8f36640 10 monclient: get_auth_request con 0x7fdcc8036140 auth_method 0 2024-09-16T14:51:47.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915167181 0 0) 0x7fdcc4006590 con 0x7fdcc4002830 2024-09-16T14:51:47.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:47.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:47.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:47.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 10 monclient(hunting): my global_id is 184420 2024-09-16T14:51:47.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.132+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce41a88a0 con 0x7fdcc4002830 2024-09-16T14:51:47.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.136+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1069315642 0 0) 0x7fdce41a88a0 con 0x7fdcc4002830 2024-09-16T14:51:47.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.136+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcc4006590 con 0x7fdcc4002830 2024-09-16T14:51:47.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.136+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdcdc001740 con 0x7fdcc4002830 2024-09-16T14:51:47.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.136+0000 7fdcd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:47.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.136+0000 7fdcd9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:47.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.136+0000 7fdcd9ffb640 10 monclient: dump: 2024-09-16T14:51:47.152 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:47.152 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:47.153 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:47.153 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:47.153 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:47.153 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:47.153 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:47.153 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:47.153 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:47.153 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:47.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.136+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 635959496 0 0) 0x7fdcc4006590 con 0x7fdcc4002830 2024-09-16T14:51:47.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.136+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdcc400a0e0 con 0x7fdcc4002830 2024-09-16T14:51:47.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.136+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fdcdc004b70 con 0x7fdcc4002830 2024-09-16T14:51:47.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.136+0000 7fdcd9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:51:47.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.136+0000 7fdcd9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:51:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.136+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 >> v1:172.21.15.189:6789/0 conn(0x7fdcc4002830 legacy=0x7fdcc4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.136+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 >> v1:172.21.15.189:6789/0 conn(0x7fdcc4002830 legacy=0x7fdcc4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.140+0000 7fdceb99b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:51:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.140+0000 7fdceb99b640 1 -- 172.21.15.26:0/2886851921 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdcc4009da0 con 0x7fdcc40090c0 2024-09-16T14:51:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422511471 0 0) 0x7fdcc4009da0 con 0x7fdcc40090c0 2024-09-16T14:51:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:51:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:51:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:51:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 10 monclient(hunting): my global_id is 184423 2024-09-16T14:51:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc4006590 con 0x7fdcc40090c0 2024-09-16T14:51:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3174074670 0 0) 0x7fdcc4006590 con 0x7fdcc40090c0 2024-09-16T14:51:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcc4009da0 con 0x7fdcc40090c0 2024-09-16T14:51:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdcdc005150 con 0x7fdcc40090c0 2024-09-16T14:51:47.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:51:47.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:51:47.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 10 monclient: dump: 2024-09-16T14:51:47.155 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:51:47.155 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:51:47.155 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:47.155 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:51:47.155 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:51:47.155 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:51:47.155 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:51:47.155 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:51:47.155 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:51:47.155 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:51:47.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 448034310 0 0) 0x7fdcc4009da0 con 0x7fdcc40090c0 2024-09-16T14:51:47.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdcc400c290 con 0x7fdcc40090c0 2024-09-16T14:51:47.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3870 (unknown 0 0 1799630673) 0x7fdcdc0052b0 con 0x7fdcc40090c0 2024-09-16T14:51:47.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:51:47.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:51:47.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 >> v1:172.21.15.189:6789/0 conn(0x7fdcc40090c0 legacy=0x7fdcc4009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:47.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdcd9ffb640 1 -- 172.21.15.26:0/2886851921 >> v1:172.21.15.189:6789/0 conn(0x7fdcc40090c0 legacy=0x7fdcc4009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:51:47.156 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:51:47.156 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:51:47.156 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:51:47.157 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:51:47.157 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 86, 2024-09-16T14:51:47.157 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:51:47.157 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:51:47.157 INFO:teuthology.orchestra.run.smithi026.stdout: 1 2024-09-16T14:51:47.157 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:47.157 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:51:47.157 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:47.157 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:51:47.157 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:51:47.157 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:47.157 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:47.157 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:47.158 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:47.158 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:47.158 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:47.158 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:47.158 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:47.158 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:47.158 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:47.158 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:47.158 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:51:47.158 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:51:47.158 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:47.158 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:47.158 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:47.159 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:47.159 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:47.159 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:47.159 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:47.159 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:47.159 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:47.159 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:47.159 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:47.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:47.159 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:51:47.159 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:51:47.159 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:51:47.159 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:51:47.160 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:51:47.160 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:51:47.160 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:47.160 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:51:47.160 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:51:47.160 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:51:47.160 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:51:47.160 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:51:47.160 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:51:47.160 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:51:47.160 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:51:47.161 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:51:47.161 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:51:47.161 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:51:47.161 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:51:47.161 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:51:47.161 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:51:47.161 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:51:47.161 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:51:47.161 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:51:47.161 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:51:47.161 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:51:47.162 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:51:47.162 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:47.162 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:51:47.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:47.162 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:51:47.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:47.162 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:51:47.162 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:51:47.162 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:47.162 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:47.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:47.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:47.162 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:51:47.163 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:47.164 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:47.164 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:47.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:47.164 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:51:47.164 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:51:47.165 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:47.165 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:47.165 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:47.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:47.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:47.165 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:51:47.165 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:51:47.165 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:47.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:47.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:47.165 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:47.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:51:47.165 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:47.166 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:47.166 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:47.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:47.166 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:51:47.166 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:51:47.166 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:47.166 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:47.166 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:47.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:47.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:47.166 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:51:47.166 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:51:47.166 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:51:47.167 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:51:47.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:47.167 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:51:47.167 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:51:47.167 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:51:47.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:47.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:47.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:47.167 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:51:47.167 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:51:47.167 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:51:47.167 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:51:47.167 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:51:47.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:47.168 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:47.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:47.168 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:51:47.168 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:51:47.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:47.168 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:47.168 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:47.168 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:51:47.168 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:47.168 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:51:47.168 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:51:47.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:51:47.168 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:51:47.168 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:51:47.169 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:51:47.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:51:47.169 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:51:47.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:51:47.169 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:51:47.169 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:51:47.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdceb99b640 1 -- 172.21.15.26:0/2886851921 >> v1:172.21.15.189:6812/3196161582 conn(0x7fdcc8036140 legacy=0x7fdcc8038600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:47.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdceb99b640 10 monclient: shutdown 2024-09-16T14:51:47.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdceb99b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:51:47.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdceb99b640 1 -- 172.21.15.26:0/2886851921 >> v1:172.21.15.26:6789/0 conn(0x7fdce409d1b0 legacy=0x7fdce41d1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:51:47.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdce9f38640 1 -- 172.21.15.26:0/2886851921 reap_dead start 2024-09-16T14:51:47.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdce9f38640 1 -- 172.21.15.26:0/2886851921 reap_dead start 2024-09-16T14:51:47.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdceb99b640 1 -- 172.21.15.26:0/2886851921 shutdown_connections 2024-09-16T14:51:47.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdceb99b640 1 -- 172.21.15.26:0/2886851921 >> 172.21.15.26:0/2886851921 conn(0x7fdce409fb50 msgr2=0x7fdce40a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:51:47.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdceb99b640 1 -- 172.21.15.26:0/2886851921 shutdown_connections 2024-09-16T14:51:47.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:51:47.144+0000 7fdceb99b640 1 -- 172.21.15.26:0/2886851921 wait complete. 2024-09-16T14:51:47.171 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:51:47.171 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:52:07.173 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T14:52:07.173 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T14:52:07.173 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T14:52:07.174 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T14:52:07.177 INFO:tasks.ceph.mon.c:Started 2024-09-16T14:52:07.177 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:52:07.177 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:52:07.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40bb94640 10 monclient: get_monmap_and_config 2024-09-16T14:52:07.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40bb94640 10 monclient: build_initial_monmap 2024-09-16T14:52:07.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40bb94640 10 monclient: monmap: 2024-09-16T14:52:07.305 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:07.305 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.305 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:07.306740+0000 2024-09-16T14:52:07.306 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:07.306740+0000 2024-09-16T14:52:07.306 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:07.306 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.306 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:07.306 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:07.306 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:07.306 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40bb94640 1 Processor -- start 2024-09-16T14:52:07.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40bb94640 1 -- start start 2024-09-16T14:52:07.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40bb94640 10 monclient: init 2024-09-16T14:52:07.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40bb94640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:07.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40bb94640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:52:07.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40bb94640 10 monclient(hunting): picked mon.noname-b con 0x7fe4040a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:07.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40bb94640 10 monclient(hunting): picked mon.noname-a con 0x7fe4040a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40bb94640 10 monclient(hunting): picked mon.noname-c con 0x7fe4040a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40bb94640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4040a3250 con 0x7fe4040a0000 2024-09-16T14:52:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40bb94640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40409f600 con 0x7fe4040a2800 2024-09-16T14:52:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40bb94640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4040a3830 con 0x7fe4040a1d10 2024-09-16T14:52:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40bb94640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40bb94640 10 monclient(hunting): authenticate will time out at 1551.426471s 2024-09-16T14:52:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40912f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe4040a0000 0x7fe404139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58286/0 (socket says 172.21.15.26:58286) 2024-09-16T14:52:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe409930640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe4040a1d10 0x7fe4040a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46804/0 (socket says 172.21.15.26:46804) 2024-09-16T14:52:07.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.304+0000 7fe40912f640 1 -- 172.21.15.26:0/2646891528 learned_addr learned my addr 172.21.15.26:0/2646891528 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.340+0000 7fe40892e640 1 -- 172.21.15.26:0/2646891528 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244081386 0 0) 0x7fe4040a3250 con 0x7fe4040a0000 2024-09-16T14:52:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.340+0000 7fe40892e640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.340+0000 7fe40892e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.340+0000 7fe40892e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.340+0000 7fe40892e640 10 monclient(hunting): my global_id is 194241 2024-09-16T14:52:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.340+0000 7fe40892e640 1 -- 172.21.15.26:0/2646891528 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3f0003530 con 0x7fe4040a0000 2024-09-16T14:52:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 1 -- 172.21.15.26:0/2646891528 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308657525 0 0) 0x7fe40409f600 con 0x7fe4040a2800 2024-09-16T14:52:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient(hunting): my global_id is 184426 2024-09-16T14:52:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 1 -- 172.21.15.26:0/2646891528 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4040a3250 con 0x7fe4040a2800 2024-09-16T14:52:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 1 -- 172.21.15.26:0/2646891528 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909289842 0 0) 0x7fe4040a3830 con 0x7fe4040a1d10 2024-09-16T14:52:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient(hunting): my global_id is 214100 2024-09-16T14:52:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 1 -- 172.21.15.26:0/2646891528 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe40409f600 con 0x7fe4040a1d10 2024-09-16T14:52:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 1 -- 172.21.15.26:0/2646891528 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3126801831 0 0) 0x7fe3f0003530 con 0x7fe4040a0000 2024-09-16T14:52:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 1 -- 172.21.15.26:0/2646891528 >> v1:172.21.15.26:6790/0 conn(0x7fe4040a1d10 legacy=0x7fe4040a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 1 -- 172.21.15.26:0/2646891528 >> v1:172.21.15.189:6789/0 conn(0x7fe4040a2800 legacy=0x7fe40413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient: _finish_hunting 0 2024-09-16T14:52:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 1 monclient: found mon.noname-a 2024-09-16T14:52:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:52:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 1 -- 172.21.15.26:0/2646891528 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40409ed60 con 0x7fe4040a0000 2024-09-16T14:52:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient: _finish_auth 0 2024-09-16T14:52:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient: _check_auth_tickets 2024-09-16T14:52:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 1 -- 172.21.15.26:0/2646891528 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe3f4001620 con 0x7fe4040a0000 2024-09-16T14:52:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient: dump: 2024-09-16T14:52:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.353 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.353 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.353 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.353 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40bb94640 5 monclient: authenticate success, global_id 194241 2024-09-16T14:52:07.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40bb94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:07.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 1 -- 172.21.15.26:0/2646891528 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3f4001900 con 0x7fe4040a0000 2024-09-16T14:52:07.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:07.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40bb94640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40bb94640 4 set_mon_vals no callback set 2024-09-16T14:52:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 1 -- 172.21.15.26:0/2646891528 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe3f4001c90 con 0x7fe4040a0000 2024-09-16T14:52:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.344+0000 7fe40892e640 10 monclient: dump: 2024-09-16T14:52:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.354 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.355 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 10 monclient: shutdown 2024-09-16T14:52:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 1 -- 172.21.15.26:0/2646891528 >> v1:172.21.15.26:6789/0 conn(0x7fe4040a0000 legacy=0x7fe404139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 1 -- 172.21.15.26:0/2646891528 shutdown_connections 2024-09-16T14:52:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 1 -- 172.21.15.26:0/2646891528 >> 172.21.15.26:0/2646891528 conn(0x7fe4040a7f30 msgr2=0x7fe4040a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 1 -- 172.21.15.26:0/2646891528 shutdown_connections 2024-09-16T14:52:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 1 -- 172.21.15.26:0/2646891528 wait complete. 2024-09-16T14:52:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 10 monclient: build_initial_monmap 2024-09-16T14:52:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 10 monclient: monmap: 2024-09-16T14:52:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:07.355 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:07.356 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 1 Processor -- start 2024-09-16T14:52:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 1 -- start start 2024-09-16T14:52:07.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 10 monclient: init 2024-09-16T14:52:07.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:07.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:52:07.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 10 monclient(hunting): picked mon.noname-c con 0x7fe4040a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:07.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 10 monclient(hunting): picked mon.noname-b con 0x7fe4040a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:07.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 10 monclient(hunting): picked mon.noname-a con 0x7fe4040a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:07.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe404134b90 con 0x7fe4040a2800 2024-09-16T14:52:07.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe404133980 con 0x7fe4040a1d10 2024-09-16T14:52:07.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe404131230 con 0x7fe4040a0000 2024-09-16T14:52:07.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:07.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe40bb94640 10 monclient(hunting): authenticate will time out at 1551.471465s 2024-09-16T14:52:07.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe409930640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe4040a0000 0x7fe404130410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46806/0 (socket says 172.21.15.26:46806) 2024-09-16T14:52:07.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.348+0000 7fe409930640 1 -- 172.21.15.26:0/2687080374 learned_addr learned my addr 172.21.15.26:0/2687080374 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:07.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656573471 0 0) 0x7fe404131230 con 0x7fe4040a0000 2024-09-16T14:52:07.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient(hunting): my global_id is 214103 2024-09-16T14:52:07.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3e8003530 con 0x7fe4040a0000 2024-09-16T14:52:07.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087254065 0 0) 0x7fe404134b90 con 0x7fe4040a2800 2024-09-16T14:52:07.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient(hunting): my global_id is 194244 2024-09-16T14:52:07.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe404131230 con 0x7fe4040a2800 2024-09-16T14:52:07.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1815837503 0 0) 0x7fe404131230 con 0x7fe4040a2800 2024-09-16T14:52:07.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe404134b90 con 0x7fe4040a2800 2024-09-16T14:52:07.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe3fc001c40 con 0x7fe4040a2800 2024-09-16T14:52:07.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:07.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient(hunting): dump: 2024-09-16T14:52:07.368 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.368 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.369 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.369 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.370 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.370 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.370 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.370 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.370 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.371 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:07.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 832494855 0 0) 0x7fe3e8003530 con 0x7fe4040a0000 2024-09-16T14:52:07.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe404131230 con 0x7fe4040a0000 2024-09-16T14:52:07.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe400001900 con 0x7fe4040a0000 2024-09-16T14:52:07.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:07.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient(hunting): dump: 2024-09-16T14:52:07.372 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.372 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.372 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.373 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.373 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.373 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.373 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.373 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.374 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.374 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3817330768 0 0) 0x7fe404134b90 con 0x7fe4040a2800 2024-09-16T14:52:07.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 >> v1:172.21.15.26:6790/0 conn(0x7fe4040a0000 legacy=0x7fe404130410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 >> v1:172.21.15.189:6789/0 conn(0x7fe4040a1d10 legacy=0x7fe404130b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:52:07.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 monclient: found mon.a 2024-09-16T14:52:07.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:07.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:07.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe404131db0 con 0x7fe4040a2800 2024-09-16T14:52:07.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient: _finish_auth 0 2024-09-16T14:52:07.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:52:07.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:07.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe40bb94640 5 monclient: authenticate success, global_id 194244 2024-09-16T14:52:07.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe40bb94640 10 monclient: _renew_subs 2024-09-16T14:52:07.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe40bb94640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:07.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe40bb94640 1 -- 172.21.15.26:0/2687080374 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe404133bd0 con 0x7fe4040a2800 2024-09-16T14:52:07.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe40bb94640 10 monclient: _renew_subs 2024-09-16T14:52:07.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe40bb94640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:07.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe40bb94640 1 -- 172.21.15.26:0/2687080374 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4041342d0 con 0x7fe4040a2800 2024-09-16T14:52:07.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3fc0039a0 con 0x7fe4040a2800 2024-09-16T14:52:07.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:07.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe3fc003cb0 con 0x7fe4040a2800 2024-09-16T14:52:07.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:07.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 10 monclient: dump: 2024-09-16T14:52:07.378 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.378 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.378 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.379 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.379 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.379 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.379 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.379 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.379 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.379 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fbfff640 4 set_mon_vals no callback set 2024-09-16T14:52:07.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe3fc00e720 con 0x7fe4040a2800 2024-09-16T14:52:07.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe40bb94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:07.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fe3fc042fc0 con 0x7fe4040a2800 2024-09-16T14:52:07.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe40bb94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:07.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe40bb94640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:07.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe40bb94640 1 -- 172.21.15.26:0/2687080374 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ec005050 con 0x7fe4040a2800 2024-09-16T14:52:07.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.352+0000 7fe409930640 10 monclient: get_auth_request con 0x7fe3e8036270 auth_method 0 2024-09-16T14:52:07.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.356+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe3fc0136d0 con 0x7fe4040a2800 2024-09-16T14:52:07.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.356+0000 7fe3fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:07.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.356+0000 7fe3fa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:07.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.580+0000 7fe40bb94640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:52:07.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.580+0000 7fe40bb94640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:07.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.580+0000 7fe40bb94640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:07.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.580+0000 7fe40bb94640 1 -- 172.21.15.26:0/2687080374 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe3ec005880 con 0x7fe4040a2800 2024-09-16T14:52:07.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.580+0000 7fe3fa7fc640 1 -- 172.21.15.26:0/2687080374 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1573411952) 0x7fe3fc011d70 con 0x7fe4040a2800 2024-09-16T14:52:07.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.580+0000 7fe3fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:07.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.580+0000 7fe3fa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:07.583 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":90,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:52:07.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.584+0000 7fe40bb94640 1 -- 172.21.15.26:0/2687080374 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe3e8036270 legacy=0x7fe3e8038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.584+0000 7fe40bb94640 10 monclient: shutdown 2024-09-16T14:52:07.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.584+0000 7fe40bb94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:07.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.584+0000 7fe40bb94640 1 -- 172.21.15.26:0/2687080374 >> v1:172.21.15.26:6789/0 conn(0x7fe4040a2800 legacy=0x7fe4041d1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.584+0000 7fe40bb94640 1 -- 172.21.15.26:0/2687080374 shutdown_connections 2024-09-16T14:52:07.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.584+0000 7fe40bb94640 1 -- 172.21.15.26:0/2687080374 >> 172.21.15.26:0/2687080374 conn(0x7fe4040a7f30 msgr2=0x7fe40406b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:07.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.584+0000 7fe40bb94640 1 -- 172.21.15.26:0/2687080374 shutdown_connections 2024-09-16T14:52:07.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.584+0000 7fe40bb94640 1 -- 172.21.15.26:0/2687080374 wait complete. 2024-09-16T14:52:07.601 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:52:07.602 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:52:07.602 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:52:07.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.716+0000 7fc111bf8640 10 monclient: get_monmap_and_config 2024-09-16T14:52:07.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.716+0000 7fc111bf8640 10 monclient: build_initial_monmap 2024-09-16T14:52:07.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.716+0000 7fc111bf8640 10 monclient: monmap: 2024-09-16T14:52:07.721 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:07.721 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.721 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:07.721480+0000 2024-09-16T14:52:07.721 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:07.721480+0000 2024-09-16T14:52:07.721 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:07.722 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.722 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:07.722 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:07.722 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:07.722 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 1 Processor -- start 2024-09-16T14:52:07.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 1 -- start start 2024-09-16T14:52:07.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 10 monclient: init 2024-09-16T14:52:07.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:07.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:52:07.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 10 monclient(hunting): picked mon.noname-b con 0x7fc10c09cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:07.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 10 monclient(hunting): picked mon.noname-a con 0x7fc10c132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:07.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 10 monclient(hunting): picked mon.noname-c con 0x7fc10c138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:07.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc10c09d260 con 0x7fc10c132650 2024-09-16T14:52:07.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc10c132a80 con 0x7fc10c138ff0 2024-09-16T14:52:07.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc10c09cf60 con 0x7fc10c09cb80 2024-09-16T14:52:07.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:07.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 10 monclient(hunting): authenticate will time out at 1551.841680s 2024-09-16T14:52:07.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10b7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc10c09cb80 0x7fc10c0717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46816/0 (socket says 172.21.15.26:46816) 2024-09-16T14:52:07.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10b7fe640 1 -- 172.21.15.26:0/240053898 learned_addr learned my addr 172.21.15.26:0/240053898 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:07.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 1 -- 172.21.15.26:0/240053898 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249118367 0 0) 0x7fc10c09cf60 con 0x7fc10c09cb80 2024-09-16T14:52:07.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient(hunting): my global_id is 214106 2024-09-16T14:52:07.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 1 -- 172.21.15.26:0/240053898 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0f4003530 con 0x7fc10c09cb80 2024-09-16T14:52:07.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 1 -- 172.21.15.26:0/240053898 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854931667 0 0) 0x7fc10c09d260 con 0x7fc10c132650 2024-09-16T14:52:07.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient(hunting): my global_id is 194247 2024-09-16T14:52:07.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 1 -- 172.21.15.26:0/240053898 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc10c09cf60 con 0x7fc10c132650 2024-09-16T14:52:07.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 1 -- 172.21.15.26:0/240053898 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234219420 0 0) 0x7fc10c132a80 con 0x7fc10c138ff0 2024-09-16T14:52:07.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient(hunting): my global_id is 184432 2024-09-16T14:52:07.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 1 -- 172.21.15.26:0/240053898 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc10c09d260 con 0x7fc10c138ff0 2024-09-16T14:52:07.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 1 -- 172.21.15.26:0/240053898 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4092759871 0 0) 0x7fc10c09cf60 con 0x7fc10c132650 2024-09-16T14:52:07.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 1 -- 172.21.15.26:0/240053898 >> v1:172.21.15.26:6790/0 conn(0x7fc10c09cb80 legacy=0x7fc10c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 1 -- 172.21.15.26:0/240053898 >> v1:172.21.15.189:6789/0 conn(0x7fc10c138ff0 legacy=0x7fc10c13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:52:07.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 1 monclient: found mon.noname-a 2024-09-16T14:52:07.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:07.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:52:07.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 1 -- 172.21.15.26:0/240053898 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10c09f2d0 con 0x7fc10c132650 2024-09-16T14:52:07.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:52:07.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:52:07.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:07.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 1 -- 172.21.15.26:0/240053898 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc100001600 con 0x7fc10c132650 2024-09-16T14:52:07.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 5 monclient: authenticate success, global_id 194247 2024-09-16T14:52:07.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:07.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:07.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient: dump: 2024-09-16T14:52:07.750 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.750 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.750 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.750 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.750 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.751 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.751 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.751 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.751 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.751 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:07.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 1 -- 172.21.15.26:0/240053898 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1000018b0 con 0x7fc10c132650 2024-09-16T14:52:07.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:07.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 1 -- 172.21.15.26:0/240053898 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc100001c80 con 0x7fc10c132650 2024-09-16T14:52:07.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:07.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc10a7fc640 10 monclient: dump: 2024-09-16T14:52:07.752 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.752 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.752 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.752 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.752 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.752 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.752 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.752 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.752 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.752 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:07.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 4 set_mon_vals no callback set 2024-09-16T14:52:07.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 10 monclient: shutdown 2024-09-16T14:52:07.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:07.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 1 -- 172.21.15.26:0/240053898 >> v1:172.21.15.26:6789/0 conn(0x7fc10c132650 legacy=0x7fc10c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 1 -- 172.21.15.26:0/240053898 shutdown_connections 2024-09-16T14:52:07.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 1 -- 172.21.15.26:0/240053898 >> 172.21.15.26:0/240053898 conn(0x7fc10c09fb50 msgr2=0x7fc10c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:07.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 1 -- 172.21.15.26:0/240053898 shutdown_connections 2024-09-16T14:52:07.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.720+0000 7fc111bf8640 1 -- 172.21.15.26:0/240053898 wait complete. 2024-09-16T14:52:07.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 10 monclient: build_initial_monmap 2024-09-16T14:52:07.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 10 monclient: monmap: 2024-09-16T14:52:07.754 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:07.754 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:07.754 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:07.754 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:07.754 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:07.754 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:07.754 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:07.754 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:07.754 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:07.754 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 1 Processor -- start 2024-09-16T14:52:07.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 1 -- start start 2024-09-16T14:52:07.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 10 monclient: init 2024-09-16T14:52:07.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:07.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:52:07.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 10 monclient(hunting): picked mon.noname-c con 0x7fc10c09cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:07.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 10 monclient(hunting): picked mon.noname-a con 0x7fc10c132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:07.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 10 monclient(hunting): picked mon.noname-b con 0x7fc10c138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:07.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc10c09a240 con 0x7fc10c132650 2024-09-16T14:52:07.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc10c0a2120 con 0x7fc10c138ff0 2024-09-16T14:52:07.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc10c137f60 con 0x7fc10c09cb80 2024-09-16T14:52:07.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:07.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 10 monclient(hunting): authenticate will time out at 1551.845130s 2024-09-16T14:52:07.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc10b7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc10c09cb80 0x7fc10c12d780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46830/0 (socket says 172.21.15.26:46830) 2024-09-16T14:52:07.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc10b7fe640 1 -- 172.21.15.26:0/2977293722 learned_addr learned my addr 172.21.15.26:0/2977293722 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:07.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999302977 0 0) 0x7fc10c137f60 con 0x7fc10c09cb80 2024-09-16T14:52:07.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): my global_id is 214109 2024-09-16T14:52:07.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0f8003530 con 0x7fc10c09cb80 2024-09-16T14:52:07.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310328614 0 0) 0x7fc10c09a240 con 0x7fc10c132650 2024-09-16T14:52:07.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): my global_id is 194250 2024-09-16T14:52:07.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc10c137f60 con 0x7fc10c132650 2024-09-16T14:52:07.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1019439712 0 0) 0x7fc10c137f60 con 0x7fc10c132650 2024-09-16T14:52:07.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc10c09a240 con 0x7fc10c132650 2024-09-16T14:52:07.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc100001600 con 0x7fc10c132650 2024-09-16T14:52:07.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:07.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): dump: 2024-09-16T14:52:07.758 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.758 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.758 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.758 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.759 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 311092254 0 0) 0x7fc0f8003530 con 0x7fc10c09cb80 2024-09-16T14:52:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc10c137f60 con 0x7fc10c09cb80 2024-09-16T14:52:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc0fc001780 con 0x7fc10c09cb80 2024-09-16T14:52:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): dump: 2024-09-16T14:52:07.759 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.760 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422058903 0 0) 0x7fc10c0a2120 con 0x7fc10c138ff0 2024-09-16T14:52:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): my global_id is 184435 2024-09-16T14:52:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0f8003530 con 0x7fc10c138ff0 2024-09-16T14:52:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 740761517 0 0) 0x7fc10c09a240 con 0x7fc10c132650 2024-09-16T14:52:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 >> v1:172.21.15.26:6790/0 conn(0x7fc10c09cb80 legacy=0x7fc10c12d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 >> v1:172.21.15.189:6789/0 conn(0x7fc10c138ff0 legacy=0x7fc10c12e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:52:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 monclient: found mon.a 2024-09-16T14:52:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10c134580 con 0x7fc10c132650 2024-09-16T14:52:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient: _finish_auth 0 2024-09-16T14:52:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:52:07.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 5 monclient: authenticate success, global_id 194250 2024-09-16T14:52:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 10 monclient: _renew_subs 2024-09-16T14:52:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 1 -- 172.21.15.26:0/2977293722 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc10c134e20 con 0x7fc10c132650 2024-09-16T14:52:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 10 monclient: _renew_subs 2024-09-16T14:52:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 1 -- 172.21.15.26:0/2977293722 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc10c1d2b90 con 0x7fc10c132650 2024-09-16T14:52:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1000039a0 con 0x7fc10c132650 2024-09-16T14:52:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc100003d30 con 0x7fc10c132650 2024-09-16T14:52:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc109ffb640 4 set_mon_vals no callback set 2024-09-16T14:52:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:07.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient: dump: 2024-09-16T14:52:07.763 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.763 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.763 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.763 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.763 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.763 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.763 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.763 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.764 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc111bf8640 1 -- 172.21.15.26:0/2977293722 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc0ec006590 con 0x7fc0ec002830 2024-09-16T14:52:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc100003e90 con 0x7fc10c132650 2024-09-16T14:52:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fc100042dd0 con 0x7fc10c132650 2024-09-16T14:52:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658537693 0 0) 0x7fc0ec006590 con 0x7fc0ec002830 2024-09-16T14:52:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient(hunting): my global_id is 194253 2024-09-16T14:52:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc10c09a240 con 0x7fc0ec002830 2024-09-16T14:52:07.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3320163502 0 0) 0x7fc10c09a240 con 0x7fc0ec002830 2024-09-16T14:52:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0ec006590 con 0x7fc0ec002830 2024-09-16T14:52:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc0fc004b80 con 0x7fc0ec002830 2024-09-16T14:52:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc10bfff640 10 monclient: get_auth_request con 0x7fc0f80363c0 auth_method 0 2024-09-16T14:52:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 10 monclient: dump: 2024-09-16T14:52:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.765 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.766 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3660871068 0 0) 0x7fc0ec006590 con 0x7fc0ec002830 2024-09-16T14:52:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.724+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0ec00a0e0 con 0x7fc0ec002830 2024-09-16T14:52:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.728+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc0fc006e90 con 0x7fc0ec002830 2024-09-16T14:52:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.728+0000 7fc0f3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.728+0000 7fc0f3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.728+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 >> v1:172.21.15.26:6789/0 conn(0x7fc0ec002830 legacy=0x7fc0ec004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.728+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 >> v1:172.21.15.26:6789/0 conn(0x7fc0ec002830 legacy=0x7fc0ec004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:07.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc111bf8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:52:07.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc111bf8640 1 -- 172.21.15.26:0/2977293722 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc0ec009da0 con 0x7fc0ec0090c0 2024-09-16T14:52:07.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843524673 0 0) 0x7fc0ec009da0 con 0x7fc0ec0090c0 2024-09-16T14:52:07.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 10 monclient(hunting): my global_id is 194256 2024-09-16T14:52:07.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0ec006590 con 0x7fc0ec0090c0 2024-09-16T14:52:07.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 190097676 0 0) 0x7fc0ec006590 con 0x7fc0ec0090c0 2024-09-16T14:52:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0ec009da0 con 0x7fc0ec0090c0 2024-09-16T14:52:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc0fc005240 con 0x7fc0ec0090c0 2024-09-16T14:52:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 10 monclient: dump: 2024-09-16T14:52:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.768 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.769 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 40638659 0 0) 0x7fc0ec009da0 con 0x7fc0ec0090c0 2024-09-16T14:52:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc0ec00c290 con 0x7fc0ec0090c0 2024-09-16T14:52:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 1747113841) 0x7fc0fc0066a0 con 0x7fc0ec0090c0 2024-09-16T14:52:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:52:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 >> v1:172.21.15.26:6789/0 conn(0x7fc0ec0090c0 legacy=0x7fc0ec009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc0f3fff640 1 -- 172.21.15.26:0/2977293722 >> v1:172.21.15.26:6789/0 conn(0x7fc0ec0090c0 legacy=0x7fc0ec009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:07.770 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:52:07.770 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:07.770 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:07.770 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:52:07.770 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 90, 2024-09-16T14:52:07.770 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:52:07.770 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:52:07.770 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:52:07.770 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:52:07.770 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:07.770 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:52:07.771 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:07.771 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:52:07.771 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:52:07.771 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:07.771 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:07.771 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:07.771 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:07.771 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:07.771 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:07.771 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:07.771 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:07.771 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:07.771 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:07.772 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:07.772 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:52:07.772 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:52:07.772 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:07.772 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:07.772 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:07.772 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:07.772 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:07.772 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:07.772 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:07.772 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:07.772 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:07.772 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:07.773 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:07.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.773 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:52:07.773 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:52:07.773 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:52:07.773 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:52:07.773 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:52:07.773 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:52:07.773 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:07.774 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:07.774 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:52:07.774 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:52:07.774 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:52:07.774 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:52:07.774 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:52:07.774 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:52:07.774 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:52:07.774 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:07.774 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:52:07.774 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:07.774 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:07.774 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:07.775 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:07.775 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:07.775 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:07.775 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:07.775 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:07.775 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:07.775 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:07.775 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:07.775 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:52:07.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.775 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:52:07.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.775 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:07.776 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:07.776 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:07.776 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:07.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.776 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:07.776 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:52:07.776 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:07.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:07.776 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:07.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.776 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:52:07.776 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:52:07.776 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:07.777 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:07.777 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:07.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.777 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:07.777 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:07.777 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:07.777 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:07.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.778 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:07.778 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:52:07.778 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:07.778 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:07.778 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:07.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.778 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:52:07.778 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:52:07.778 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:07.778 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:07.778 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:07.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.779 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:07.779 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:07.779 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:07.779 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:07.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.779 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:07.779 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:52:07.779 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:07.779 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:07.779 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:07.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.779 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:52:07.779 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:52:07.780 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:07.780 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:07.780 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:07.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:07.780 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:07.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.780 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:52:07.780 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:52:07.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.780 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:07.780 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:07.780 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:07.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:07.781 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:07.781 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:52:07.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.781 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:07.781 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:07.781 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:52:07.781 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:07.781 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:07.781 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:52:07.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.781 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:07.781 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:07.781 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:52:07.782 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:07.782 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:07.782 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:52:07.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.782 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:07.782 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:07.782 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:07.782 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:07.782 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:07.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.782 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:52:07.782 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:52:07.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc111bf8640 1 -- 172.21.15.26:0/2977293722 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc0f80363c0 legacy=0x7fc0f8038880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc111bf8640 10 monclient: shutdown 2024-09-16T14:52:07.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc111bf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:07.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc111bf8640 1 -- 172.21.15.26:0/2977293722 >> v1:172.21.15.26:6789/0 conn(0x7fc10c132650 legacy=0x7fc10c12de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc10bfff640 1 -- 172.21.15.26:0/2977293722 reap_dead start 2024-09-16T14:52:07.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc10bfff640 1 -- 172.21.15.26:0/2977293722 reap_dead start 2024-09-16T14:52:07.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc111bf8640 1 -- 172.21.15.26:0/2977293722 shutdown_connections 2024-09-16T14:52:07.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc111bf8640 1 -- 172.21.15.26:0/2977293722 >> 172.21.15.26:0/2977293722 conn(0x7fc10c09fb50 msgr2=0x7fc10c1388c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:07.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc111bf8640 1 -- 172.21.15.26:0/2977293722 shutdown_connections 2024-09-16T14:52:07.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.732+0000 7fc111bf8640 1 -- 172.21.15.26:0/2977293722 wait complete. 2024-09-16T14:52:07.785 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:52:07.785 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:52:07.786 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:52:07.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.900+0000 7f1f5989e640 10 monclient: get_monmap_and_config 2024-09-16T14:52:07.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.900+0000 7f1f5989e640 10 monclient: build_initial_monmap 2024-09-16T14:52:07.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.900+0000 7f1f5989e640 10 monclient: monmap: 2024-09-16T14:52:07.904 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:07.904 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.904 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:07.905252+0000 2024-09-16T14:52:07.904 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:07.905252+0000 2024-09-16T14:52:07.904 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:07.904 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.904 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:07.905 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:07.905 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:07.905 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 1 Processor -- start 2024-09-16T14:52:07.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 1 -- start start 2024-09-16T14:52:07.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 10 monclient: init 2024-09-16T14:52:07.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:07.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:52:07.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 10 monclient(hunting): picked mon.noname-b con 0x7f1f5409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:07.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 10 monclient(hunting): picked mon.noname-a con 0x7f1f54132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:07.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 10 monclient(hunting): picked mon.noname-c con 0x7f1f54138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f5409d260 con 0x7f1f54132650 2024-09-16T14:52:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f54132a80 con 0x7f1f54138ff0 2024-09-16T14:52:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f5409cf60 con 0x7f1f5409cb80 2024-09-16T14:52:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 10 monclient(hunting): authenticate will time out at 1552.025325s 2024-09-16T14:52:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f527fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1f54132650 0x7f1f541378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58346/0 (socket says 172.21.15.26:58346) 2024-09-16T14:52:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f527fc640 1 -- 172.21.15.26:0/3214423612 learned_addr learned my addr 172.21.15.26:0/3214423612 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 1 -- 172.21.15.26:0/3214423612 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792947191 0 0) 0x7f1f5409cf60 con 0x7f1f5409cb80 2024-09-16T14:52:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient(hunting): my global_id is 214112 2024-09-16T14:52:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 1 -- 172.21.15.26:0/3214423612 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f40003530 con 0x7f1f5409cb80 2024-09-16T14:52:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 1 -- 172.21.15.26:0/3214423612 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383652716 0 0) 0x7f1f5409d260 con 0x7f1f54132650 2024-09-16T14:52:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient(hunting): my global_id is 194259 2024-09-16T14:52:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 1 -- 172.21.15.26:0/3214423612 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f5409cf60 con 0x7f1f54132650 2024-09-16T14:52:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 1 -- 172.21.15.26:0/3214423612 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4119470164 0 0) 0x7f1f40003530 con 0x7f1f5409cb80 2024-09-16T14:52:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 1 -- 172.21.15.26:0/3214423612 >> v1:172.21.15.189:6789/0 conn(0x7f1f54138ff0 legacy=0x7f1f5413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 1 -- 172.21.15.26:0/3214423612 >> v1:172.21.15.26:6789/0 conn(0x7f1f54132650 legacy=0x7f1f541378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:52:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 1 monclient: found mon.noname-b 2024-09-16T14:52:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:52:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 1 -- 172.21.15.26:0/3214423612 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5409f2d0 con 0x7f1f5409cb80 2024-09-16T14:52:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient: _finish_auth 0 2024-09-16T14:52:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:52:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 5 monclient: authenticate success, global_id 214112 2024-09-16T14:52:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 1 -- 172.21.15.26:0/3214423612 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1f44001660 con 0x7f1f5409cb80 2024-09-16T14:52:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:52:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient: dump: 2024-09-16T14:52:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.917 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 1 -- 172.21.15.26:0/3214423612 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f44001840 con 0x7f1f5409cb80 2024-09-16T14:52:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 1 -- 172.21.15.26:0/3214423612 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1f44001bd0 con 0x7f1f5409cb80 2024-09-16T14:52:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f51ffb640 10 monclient: dump: 2024-09-16T14:52:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.918 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 4 set_mon_vals no callback set 2024-09-16T14:52:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 10 monclient: shutdown 2024-09-16T14:52:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 1 -- 172.21.15.26:0/3214423612 >> v1:172.21.15.26:6790/0 conn(0x7f1f5409cb80 legacy=0x7f1f540717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 1 -- 172.21.15.26:0/3214423612 shutdown_connections 2024-09-16T14:52:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 1 -- 172.21.15.26:0/3214423612 >> 172.21.15.26:0/3214423612 conn(0x7f1f5409fb50 msgr2=0x7f1f5409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 1 -- 172.21.15.26:0/3214423612 shutdown_connections 2024-09-16T14:52:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 1 -- 172.21.15.26:0/3214423612 wait complete. 2024-09-16T14:52:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 10 monclient: build_initial_monmap 2024-09-16T14:52:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.904+0000 7f1f5989e640 10 monclient: monmap: 2024-09-16T14:52:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:07.920 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 1 Processor -- start 2024-09-16T14:52:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 1 -- start start 2024-09-16T14:52:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 10 monclient: init 2024-09-16T14:52:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:52:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 10 monclient(hunting): picked mon.noname-a con 0x7f1f5409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 10 monclient(hunting): picked mon.noname-b con 0x7f1f54132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 10 monclient(hunting): picked mon.noname-c con 0x7f1f54138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f5406d450 con 0x7f1f5409cb80 2024-09-16T14:52:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f5406ddd0 con 0x7f1f54132650 2024-09-16T14:52:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f541ca950 con 0x7f1f54138ff0 2024-09-16T14:52:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 10 monclient(hunting): authenticate will time out at 1552.028571s 2024-09-16T14:52:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f52ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1f5409cb80 0x7f1f541c9b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58360/0 (socket says 172.21.15.26:58360) 2024-09-16T14:52:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f537fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1f54138ff0 0x7f1f541d1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46850/0 (socket says 172.21.15.26:46850) 2024-09-16T14:52:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f52ffd640 1 -- 172.21.15.26:0/3232761403 learned_addr learned my addr 172.21.15.26:0/3232761403 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035419958 0 0) 0x7f1f541ca950 con 0x7f1f54138ff0 2024-09-16T14:52:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): my global_id is 214115 2024-09-16T14:52:07.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f34003530 con 0x7f1f54138ff0 2024-09-16T14:52:07.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122863015 0 0) 0x7f1f5406d450 con 0x7f1f5409cb80 2024-09-16T14:52:07.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): my global_id is 194262 2024-09-16T14:52:07.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f541ca950 con 0x7f1f5409cb80 2024-09-16T14:52:07.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3539861603 0 0) 0x7f1f34003530 con 0x7f1f54138ff0 2024-09-16T14:52:07.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f5406d450 con 0x7f1f54138ff0 2024-09-16T14:52:07.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1f4c0017c0 con 0x7f1f54138ff0 2024-09-16T14:52:07.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:52:07.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): dump: 2024-09-16T14:52:07.923 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.923 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.923 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.923 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.923 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.923 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.923 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.923 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.923 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.923 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:07.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3991558726 0 0) 0x7f1f541ca950 con 0x7f1f5409cb80 2024-09-16T14:52:07.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f34003530 con 0x7f1f5409cb80 2024-09-16T14:52:07.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1f44001b90 con 0x7f1f5409cb80 2024-09-16T14:52:07.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:07.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): dump: 2024-09-16T14:52:07.924 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.924 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.924 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.924 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.924 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.924 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.924 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.924 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.925 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.925 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663890803 0 0) 0x7f1f5406ddd0 con 0x7f1f54132650 2024-09-16T14:52:07.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): my global_id is 184441 2024-09-16T14:52:07.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f541ca950 con 0x7f1f54132650 2024-09-16T14:52:07.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3485986594 0 0) 0x7f1f5406d450 con 0x7f1f54138ff0 2024-09-16T14:52:07.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 >> v1:172.21.15.189:6789/0 conn(0x7f1f54132650 legacy=0x7f1f541ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 >> v1:172.21.15.26:6789/0 conn(0x7f1f5409cb80 legacy=0x7f1f541c9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:52:07.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 monclient: found mon.c 2024-09-16T14:52:07.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:07.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:07.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5406c920 con 0x7f1f54138ff0 2024-09-16T14:52:07.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient: _finish_auth 0 2024-09-16T14:52:07.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:52:07.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:07.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 5 monclient: authenticate success, global_id 214115 2024-09-16T14:52:07.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 10 monclient: _renew_subs 2024-09-16T14:52:07.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:07.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 1 -- 172.21.15.26:0/3232761403 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f541303c0 con 0x7f1f54138ff0 2024-09-16T14:52:07.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 10 monclient: _renew_subs 2024-09-16T14:52:07.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:07.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 1 -- 172.21.15.26:0/3232761403 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f5406daf0 con 0x7f1f54138ff0 2024-09-16T14:52:07.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f4c001d70 con 0x7f1f54138ff0 2024-09-16T14:52:07.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:07.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1f4c0037c0 con 0x7f1f54138ff0 2024-09-16T14:52:07.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:07.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f517fa640 4 set_mon_vals no callback set 2024-09-16T14:52:07.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient: dump: 2024-09-16T14:52:07.927 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.927 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.927 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.927 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.927 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.927 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.927 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.927 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.928 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.928 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:07.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f5989e640 1 -- 172.21.15.26:0/3232761403 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1f38006590 con 0x7f1f38002830 2024-09-16T14:52:07.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1f4c003920 con 0x7f1f54138ff0 2024-09-16T14:52:07.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f1f4c0439f0 con 0x7f1f54138ff0 2024-09-16T14:52:07.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833903234 0 0) 0x7f1f38006590 con 0x7f1f38002830 2024-09-16T14:52:07.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient(hunting): my global_id is 214118 2024-09-16T14:52:07.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f5406d450 con 0x7f1f38002830 2024-09-16T14:52:07.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f527fc640 10 monclient: get_auth_request con 0x7f1f34036410 auth_method 0 2024-09-16T14:52:07.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1839539035 0 0) 0x7f1f5406d450 con 0x7f1f38002830 2024-09-16T14:52:07.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f38006590 con 0x7f1f38002830 2024-09-16T14:52:07.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1f44003d00 con 0x7f1f38002830 2024-09-16T14:52:07.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:07.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 10 monclient: dump: 2024-09-16T14:52:07.929 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.929 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.929 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.929 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.929 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.929 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.929 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.930 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.930 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.930 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1249485057 0 0) 0x7f1f38006590 con 0x7f1f38002830 2024-09-16T14:52:07.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.908+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f3800a0e0 con 0x7f1f38002830 2024-09-16T14:52:07.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.912+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1f44003fb0 con 0x7f1f38002830 2024-09-16T14:52:07.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.912+0000 7f1f4bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:07.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.912+0000 7f1f4bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:07.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.912+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 >> v1:172.21.15.26:6790/0 conn(0x7f1f38002830 legacy=0x7f1f38004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.912+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 >> v1:172.21.15.26:6790/0 conn(0x7f1f38002830 legacy=0x7f1f38004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:07.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f5989e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:52:07.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f5989e640 1 -- 172.21.15.26:0/3232761403 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1f38009da0 con 0x7f1f380090c0 2024-09-16T14:52:07.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506933895 0 0) 0x7f1f38009da0 con 0x7f1f380090c0 2024-09-16T14:52:07.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:07.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:07.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:07.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 10 monclient(hunting): my global_id is 214121 2024-09-16T14:52:07.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f38006590 con 0x7f1f380090c0 2024-09-16T14:52:07.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2947869459 0 0) 0x7f1f38006590 con 0x7f1f380090c0 2024-09-16T14:52:07.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f38009da0 con 0x7f1f380090c0 2024-09-16T14:52:07.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1f44005790 con 0x7f1f380090c0 2024-09-16T14:52:07.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:07.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:07.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 10 monclient: dump: 2024-09-16T14:52:07.933 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:07.933 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:07.933 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.933 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:07.933 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:07.933 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:07.933 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:07.933 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:07.933 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:07.933 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:07.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3401937321 0 0) 0x7f1f38009da0 con 0x7f1f380090c0 2024-09-16T14:52:07.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1f3800c290 con 0x7f1f380090c0 2024-09-16T14:52:07.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3881 (unknown 0 0 2142524228) 0x7f1f440058f0 con 0x7f1f380090c0 2024-09-16T14:52:07.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:52:07.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:07.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 >> v1:172.21.15.26:6790/0 conn(0x7f1f380090c0 legacy=0x7f1f38009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f4bfff640 1 -- 172.21.15.26:0/3232761403 >> v1:172.21.15.26:6790/0 conn(0x7f1f380090c0 legacy=0x7f1f38009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:07.934 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:52:07.934 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:07.934 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:07.934 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:52:07.935 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 90, 2024-09-16T14:52:07.935 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:52:07.935 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:52:07.935 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:52:07.935 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:52:07.935 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:07.935 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:52:07.935 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:07.935 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:52:07.935 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:52:07.935 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:07.935 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:07.935 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:07.936 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:07.936 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:07.936 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:07.936 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:07.936 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:07.936 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:07.936 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:07.936 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:07.936 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:52:07.936 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:52:07.936 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:07.936 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:07.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:52:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:52:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:52:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:52:07.937 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:52:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:52:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:52:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:52:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:52:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:52:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:52:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:52:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:52:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:07.938 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:52:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:52:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:52:07.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.940 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:07.940 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:07.940 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:07.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:07.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:07.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:52:07.940 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:07.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:07.940 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:07.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:52:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:52:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:52:07.941 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:07.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:07.942 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:07.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.942 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:52:07.942 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:52:07.942 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:07.942 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:07.942 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:07.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.942 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:07.942 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:07.942 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:07.943 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:07.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.943 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:07.943 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:52:07.943 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:07.943 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:07.943 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:07.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.943 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:52:07.943 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:52:07.943 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:07.943 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:07.943 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:07.944 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:07.944 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:07.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.944 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:52:07.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:52:07.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.944 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:07.944 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:07.944 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:07.945 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:07.945 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:07.945 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:52:07.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:07.945 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:07.945 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:07.945 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:52:07.945 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:07.945 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:07.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:07.945 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:52:07.945 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:52:07.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f5989e640 1 -- 172.21.15.26:0/3232761403 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1f34036410 legacy=0x7f1f340388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f5989e640 10 monclient: shutdown 2024-09-16T14:52:07.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f5989e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:07.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f5989e640 1 -- 172.21.15.26:0/3232761403 >> v1:172.21.15.26:6790/0 conn(0x7f1f54138ff0 legacy=0x7f1f541d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:07.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f537fe640 1 -- 172.21.15.26:0/3232761403 reap_dead start 2024-09-16T14:52:07.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f537fe640 1 -- 172.21.15.26:0/3232761403 reap_dead start 2024-09-16T14:52:07.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f5989e640 1 -- 172.21.15.26:0/3232761403 shutdown_connections 2024-09-16T14:52:07.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f5989e640 1 -- 172.21.15.26:0/3232761403 >> 172.21.15.26:0/3232761403 conn(0x7f1f5409fb50 msgr2=0x7f1f541388c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:07.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f5989e640 1 -- 172.21.15.26:0/3232761403 shutdown_connections 2024-09-16T14:52:07.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:07.916+0000 7f1f5989e640 1 -- 172.21.15.26:0/3232761403 wait complete. 2024-09-16T14:52:07.947 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:52:07.948 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:52:07.948 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:52:08.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.056+0000 7ff6f6497640 10 monclient: get_monmap_and_config 2024-09-16T14:52:08.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.056+0000 7ff6f6497640 10 monclient: build_initial_monmap 2024-09-16T14:52:08.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.056+0000 7ff6f6497640 10 monclient: monmap: 2024-09-16T14:52:08.060 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:08.060 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:08.060 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:08.060979+0000 2024-09-16T14:52:08.060 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:08.060979+0000 2024-09-16T14:52:08.060 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:08.060 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:08.060 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:08.060 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:08.061 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:08.065 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:08.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 1 Processor -- start 2024-09-16T14:52:08.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 1 -- start start 2024-09-16T14:52:08.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient: init 2024-09-16T14:52:08.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:08.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:52:08.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient(hunting): picked mon.noname-b con 0x7ff6f009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:08.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient(hunting): picked mon.noname-c con 0x7ff6f0132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:08.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient(hunting): picked mon.noname-a con 0x7ff6f0138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:08.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f009d260 con 0x7ff6f0138ff0 2024-09-16T14:52:08.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f0132a80 con 0x7ff6f0132650 2024-09-16T14:52:08.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f009cf60 con 0x7ff6f009cb80 2024-09-16T14:52:08.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:08.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient(hunting): authenticate will time out at 1552.180934s 2024-09-16T14:52:08.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6effff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff6f009cb80 0x7ff6f00717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46872/0 (socket says 172.21.15.26:46872) 2024-09-16T14:52:08.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6effff640 1 -- 172.21.15.26:0/1495841444 learned_addr learned my addr 172.21.15.26:0/1495841444 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:08.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 1 -- 172.21.15.26:0/1495841444 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414750565 0 0) 0x7ff6f009cf60 con 0x7ff6f009cb80 2024-09-16T14:52:08.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:08.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:08.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:08.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient(hunting): my global_id is 214124 2024-09-16T14:52:08.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 1 -- 172.21.15.26:0/1495841444 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d8003530 con 0x7ff6f009cb80 2024-09-16T14:52:08.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 1 -- 172.21.15.26:0/1495841444 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714477055 0 0) 0x7ff6f009d260 con 0x7ff6f0138ff0 2024-09-16T14:52:08.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:08.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:08.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:08.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient(hunting): my global_id is 194265 2024-09-16T14:52:08.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 1 -- 172.21.15.26:0/1495841444 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f009cf60 con 0x7ff6f0138ff0 2024-09-16T14:52:08.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 1 -- 172.21.15.26:0/1495841444 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827594132 0 0) 0x7ff6f0132a80 con 0x7ff6f0132650 2024-09-16T14:52:08.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:08.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:08.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:08.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient(hunting): my global_id is 184444 2024-09-16T14:52:08.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 1 -- 172.21.15.26:0/1495841444 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f009d260 con 0x7ff6f0132650 2024-09-16T14:52:08.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 1 -- 172.21.15.26:0/1495841444 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2444087777 0 0) 0x7ff6f009cf60 con 0x7ff6f0138ff0 2024-09-16T14:52:08.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 1 -- 172.21.15.26:0/1495841444 >> v1:172.21.15.26:6790/0 conn(0x7ff6f009cb80 legacy=0x7ff6f00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:08.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 1 -- 172.21.15.26:0/1495841444 >> v1:172.21.15.189:6789/0 conn(0x7ff6f0132650 legacy=0x7ff6f01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:08.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient: _finish_hunting 0 2024-09-16T14:52:08.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 1 monclient: found mon.noname-a 2024-09-16T14:52:08.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:08.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:52:08.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 1 -- 172.21.15.26:0/1495841444 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f009f2d0 con 0x7ff6f0138ff0 2024-09-16T14:52:08.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient: _finish_auth 0 2024-09-16T14:52:08.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient: _check_auth_tickets 2024-09-16T14:52:08.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:08.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 5 monclient: authenticate success, global_id 194265 2024-09-16T14:52:08.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 1 -- 172.21.15.26:0/1495841444 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6e8001900 con 0x7ff6f0138ff0 2024-09-16T14:52:08.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:08.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:08.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:08.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient: dump: 2024-09-16T14:52:08.076 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:08.076 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:08.076 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.076 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.076 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:08.076 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:08.076 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:08.076 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:08.076 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:08.076 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:08.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:08.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 1 -- 172.21.15.26:0/1495841444 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6e8001b30 con 0x7ff6f0138ff0 2024-09-16T14:52:08.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:08.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 1 -- 172.21.15.26:0/1495841444 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6e8001f00 con 0x7ff6f0138ff0 2024-09-16T14:52:08.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:08.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 4 set_mon_vals no callback set 2024-09-16T14:52:08.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:08.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:08.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6eeffd640 10 monclient: dump: 2024-09-16T14:52:08.077 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:08.077 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:08.078 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.078 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.078 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:08.078 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:08.078 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:08.078 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:08.078 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:08.078 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:08.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient: shutdown 2024-09-16T14:52:08.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:08.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 1 -- 172.21.15.26:0/1495841444 >> v1:172.21.15.26:6789/0 conn(0x7ff6f0138ff0 legacy=0x7ff6f013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:08.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 1 -- 172.21.15.26:0/1495841444 shutdown_connections 2024-09-16T14:52:08.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 1 -- 172.21.15.26:0/1495841444 >> 172.21.15.26:0/1495841444 conn(0x7ff6f009fb50 msgr2=0x7ff6f009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:08.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 1 -- 172.21.15.26:0/1495841444 shutdown_connections 2024-09-16T14:52:08.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 1 -- 172.21.15.26:0/1495841444 wait complete. 2024-09-16T14:52:08.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient: build_initial_monmap 2024-09-16T14:52:08.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient: monmap: 2024-09-16T14:52:08.079 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:08.079 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:08.079 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:08.079 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:08.079 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:08.079 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:08.079 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:08.079 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:08.079 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:08.079 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:08.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 1 Processor -- start 2024-09-16T14:52:08.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 1 -- start start 2024-09-16T14:52:08.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient: init 2024-09-16T14:52:08.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:08.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:52:08.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient(hunting): picked mon.noname-c con 0x7ff6f009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:08.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient(hunting): picked mon.noname-a con 0x7ff6f0132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:08.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient(hunting): picked mon.noname-b con 0x7ff6f0138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:08.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f0070d60 con 0x7ff6f0132650 2024-09-16T14:52:08.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f01381c0 con 0x7ff6f0138ff0 2024-09-16T14:52:08.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f009a240 con 0x7ff6f009cb80 2024-09-16T14:52:08.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:08.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.060+0000 7ff6f6497640 10 monclient(hunting): authenticate will time out at 1552.183990s 2024-09-16T14:52:08.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6effff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff6f009cb80 0x7ff6f006d740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46874/0 (socket says 172.21.15.26:46874) 2024-09-16T14:52:08.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6effff640 1 -- 172.21.15.26:0/2007135314 learned_addr learned my addr 172.21.15.26:0/2007135314 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:08.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206533417 0 0) 0x7ff6f009a240 con 0x7ff6f009cb80 2024-09-16T14:52:08.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): _negotiate 2024-09-16T14:52:08.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:08.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:08.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): my global_id is 214127 2024-09-16T14:52:08.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6dc003530 con 0x7ff6f009cb80 2024-09-16T14:52:08.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877012818 0 0) 0x7ff6f0070d60 con 0x7ff6f0132650 2024-09-16T14:52:08.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): _negotiate 2024-09-16T14:52:08.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:08.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:08.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): my global_id is 194268 2024-09-16T14:52:08.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f009a240 con 0x7ff6f0132650 2024-09-16T14:52:08.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1624735363 0 0) 0x7ff6dc003530 con 0x7ff6f009cb80 2024-09-16T14:52:08.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6f0070d60 con 0x7ff6f009cb80 2024-09-16T14:52:08.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6e0001860 con 0x7ff6f009cb80 2024-09-16T14:52:08.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:08.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:52:08.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): dump: 2024-09-16T14:52:08.083 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:08.083 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:08.083 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.083 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.083 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:08.083 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:08.083 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:08.083 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:08.083 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:08.083 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:08.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:08.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 591921956 0 0) 0x7ff6f009a240 con 0x7ff6f0132650 2024-09-16T14:52:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6dc003530 con 0x7ff6f0132650 2024-09-16T14:52:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6e40018c0 con 0x7ff6f0132650 2024-09-16T14:52:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): dump: 2024-09-16T14:52:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:08.085 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:08.085 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:08.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013521382 0 0) 0x7ff6f009f2d0 con 0x7ff6f0138ff0 2024-09-16T14:52:08.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): _negotiate 2024-09-16T14:52:08.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:08.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:08.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): my global_id is 184447 2024-09-16T14:52:08.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f009a240 con 0x7ff6f0138ff0 2024-09-16T14:52:08.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3276956296 0 0) 0x7ff6f0070d60 con 0x7ff6f009cb80 2024-09-16T14:52:08.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 >> v1:172.21.15.189:6789/0 conn(0x7ff6f0138ff0 legacy=0x7ff6f01cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:08.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 >> v1:172.21.15.26:6789/0 conn(0x7ff6f0132650 legacy=0x7ff6f006de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:08.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient: _finish_hunting 0 2024-09-16T14:52:08.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 monclient: found mon.c 2024-09-16T14:52:08.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:08.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:08.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f006e560 con 0x7ff6f009cb80 2024-09-16T14:52:08.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient: _finish_auth 0 2024-09-16T14:52:08.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient: _check_auth_tickets 2024-09-16T14:52:08.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:08.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6f6497640 5 monclient: authenticate success, global_id 214127 2024-09-16T14:52:08.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6f6497640 10 monclient: _renew_subs 2024-09-16T14:52:08.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6f6497640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:08.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6f6497640 1 -- 172.21.15.26:0/2007135314 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f013c400 con 0x7ff6f009cb80 2024-09-16T14:52:08.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6f6497640 10 monclient: _renew_subs 2024-09-16T14:52:08.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6f6497640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:08.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6f6497640 1 -- 172.21.15.26:0/2007135314 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f0071320 con 0x7ff6f009cb80 2024-09-16T14:52:08.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6e00047f0 con 0x7ff6f009cb80 2024-09-16T14:52:08.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:08.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6e0004b00 con 0x7ff6f009cb80 2024-09-16T14:52:08.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:08.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:08.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient: dump: 2024-09-16T14:52:08.087 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:08.088 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:08.088 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.088 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.089 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:08.089 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:08.089 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:08.089 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:08.089 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:08.089 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:08.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ee7fc640 4 set_mon_vals no callback set 2024-09-16T14:52:08.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6f6497640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:08.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6f6497640 1 -- 172.21.15.26:0/2007135314 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff6d0006590 con 0x7ff6d0002830 2024-09-16T14:52:08.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff6e000f570 con 0x7ff6f009cb80 2024-09-16T14:52:08.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7ff6e0043ce0 con 0x7ff6f009cb80 2024-09-16T14:52:08.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6f4a34640 10 monclient: get_auth_request con 0x7ff6dc036490 auth_method 0 2024-09-16T14:52:08.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211669649 0 0) 0x7ff6d0006590 con 0x7ff6d0002830 2024-09-16T14:52:08.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): _negotiate 2024-09-16T14:52:08.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:08.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:08.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient(hunting): my global_id is 184450 2024-09-16T14:52:08.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f0070d60 con 0x7ff6d0002830 2024-09-16T14:52:08.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3500984955 0 0) 0x7ff6f0070d60 con 0x7ff6d0002830 2024-09-16T14:52:08.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6d0006590 con 0x7ff6d0002830 2024-09-16T14:52:08.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6e40037c0 con 0x7ff6d0002830 2024-09-16T14:52:08.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:08.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:52:08.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 10 monclient: dump: 2024-09-16T14:52:08.091 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:08.091 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:08.091 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.091 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.091 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:08.091 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:08.091 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:08.091 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:08.091 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:08.091 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:08.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3276119359 0 0) 0x7ff6d0006590 con 0x7ff6d0002830 2024-09-16T14:52:08.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.064+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6d000a0e0 con 0x7ff6d0002830 2024-09-16T14:52:08.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.068+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ff6e4004650 con 0x7ff6d0002830 2024-09-16T14:52:08.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.068+0000 7ff6ecff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:08.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.068+0000 7ff6ecff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:08.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.068+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 >> v1:172.21.15.189:6789/0 conn(0x7ff6d0002830 legacy=0x7ff6d0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:08.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.068+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 >> v1:172.21.15.189:6789/0 conn(0x7ff6d0002830 legacy=0x7ff6d0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:08.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.072+0000 7ff6f6497640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:52:08.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.072+0000 7ff6f6497640 1 -- 172.21.15.26:0/2007135314 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff6d0009d80 con 0x7ff6d00090a0 2024-09-16T14:52:08.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.072+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761331614 0 0) 0x7ff6d0009d80 con 0x7ff6d00090a0 2024-09-16T14:52:08.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.072+0000 7ff6ecff9640 10 monclient(hunting): _negotiate 2024-09-16T14:52:08.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.072+0000 7ff6ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:08.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.072+0000 7ff6ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:08.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.072+0000 7ff6ecff9640 10 monclient(hunting): my global_id is 184453 2024-09-16T14:52:08.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.072+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d0006590 con 0x7ff6d00090a0 2024-09-16T14:52:08.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.072+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2499931925 0 0) 0x7ff6d0006590 con 0x7ff6d00090a0 2024-09-16T14:52:08.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.072+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6d0009d80 con 0x7ff6d00090a0 2024-09-16T14:52:08.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.072+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6e4003e40 con 0x7ff6d00090a0 2024-09-16T14:52:08.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.072+0000 7ff6ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:08.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.072+0000 7ff6ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:52:08.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.072+0000 7ff6ecff9640 10 monclient: dump: 2024-09-16T14:52:08.094 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:08.094 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:08.094 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.094 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.094 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:08.094 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:08.095 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:08.095 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:08.095 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:08.095 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:08.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.072+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2432454735 0 0) 0x7ff6d0009d80 con 0x7ff6d00090a0 2024-09-16T14:52:08.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.072+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff6d000c290 con 0x7ff6d00090a0 2024-09-16T14:52:08.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.076+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 3528105177) 0x7ff6e4004080 con 0x7ff6d00090a0 2024-09-16T14:52:08.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.076+0000 7ff6ecff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:52:08.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.076+0000 7ff6ecff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:08.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.076+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 >> v1:172.21.15.189:6789/0 conn(0x7ff6d00090a0 legacy=0x7ff6d0009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:08.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.076+0000 7ff6ecff9640 1 -- 172.21.15.26:0/2007135314 >> v1:172.21.15.189:6789/0 conn(0x7ff6d00090a0 legacy=0x7ff6d0009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:08.096 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:52:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:52:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 90, 2024-09-16T14:52:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:52:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:52:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:52:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:52:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:52:08.096 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:52:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:52:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:08.097 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:52:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:52:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:08.098 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:52:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:52:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:52:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:52:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:52:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:52:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:08.099 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:52:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:52:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:52:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:52:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:52:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:52:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:52:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:52:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:08.100 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:52:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:52:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:08.101 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:52:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:52:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:52:08.102 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:52:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:08.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:52:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:52:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:08.104 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:52:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:52:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:52:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:08.105 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:52:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:52:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:52:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:08.106 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:52:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:08.107 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:52:08.107 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:52:08.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.076+0000 7ff6f6497640 1 -- 172.21.15.26:0/2007135314 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff6dc036490 legacy=0x7ff6dc038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:08.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.076+0000 7ff6f6497640 10 monclient: shutdown 2024-09-16T14:52:08.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.076+0000 7ff6f6497640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:08.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.076+0000 7ff6f6497640 1 -- 172.21.15.26:0/2007135314 >> v1:172.21.15.26:6790/0 conn(0x7ff6f009cb80 legacy=0x7ff6f006d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:08.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.076+0000 7ff6f4a34640 1 -- 172.21.15.26:0/2007135314 reap_dead start 2024-09-16T14:52:08.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.076+0000 7ff6f4a34640 1 -- 172.21.15.26:0/2007135314 reap_dead start 2024-09-16T14:52:08.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.076+0000 7ff6f6497640 1 -- 172.21.15.26:0/2007135314 shutdown_connections 2024-09-16T14:52:08.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.076+0000 7ff6f6497640 1 -- 172.21.15.26:0/2007135314 >> 172.21.15.26:0/2007135314 conn(0x7ff6f009fb50 msgr2=0x7ff6f0131bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:08.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.076+0000 7ff6f6497640 1 -- 172.21.15.26:0/2007135314 shutdown_connections 2024-09-16T14:52:08.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.076+0000 7ff6f6497640 1 -- 172.21.15.26:0/2007135314 wait complete. 2024-09-16T14:52:08.111 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:52:08.111 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:52:08.111 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:52:08.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.216+0000 7f19fdd32640 10 monclient: get_monmap_and_config 2024-09-16T14:52:08.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.216+0000 7f19fdd32640 10 monclient: build_initial_monmap 2024-09-16T14:52:08.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.216+0000 7f19fdd32640 10 monclient: monmap: 2024-09-16T14:52:08.219 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:08.219 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:08.219 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:08.220618+0000 2024-09-16T14:52:08.219 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:08.220618+0000 2024-09-16T14:52:08.219 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:08.219 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:08.220 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:08.220 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:08.220 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:08.220 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:08.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.216+0000 7f19fdd32640 1 Processor -- start 2024-09-16T14:52:08.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.216+0000 7f19fdd32640 1 -- start start 2024-09-16T14:52:08.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.216+0000 7f19fdd32640 10 monclient: init 2024-09-16T14:52:08.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:08.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:52:08.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient(hunting): picked mon.noname-a con 0x7f19f809f240 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:08.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient(hunting): picked mon.noname-c con 0x7f19f8071520 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:08.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient(hunting): picked mon.noname-b con 0x7f19f809fd30 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:08.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f80a0f00 con 0x7f19f809f240 2024-09-16T14:52:08.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f809df00 con 0x7f19f8071520 2024-09-16T14:52:08.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f80702a0 con 0x7f19f809fd30 2024-09-16T14:52:08.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:08.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient(hunting): authenticate will time out at 1552.340675s 2024-09-16T14:52:08.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f77fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f19f809f240 0x7f19f809f620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58380/0 (socket says 172.21.15.26:58380) 2024-09-16T14:52:08.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f77fe640 1 -- 172.21.15.26:0/205764245 learned_addr learned my addr 172.21.15.26:0/205764245 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:08.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 1 -- 172.21.15.26:0/205764245 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136881349 0 0) 0x7f19f80a0f00 con 0x7f19f809f240 2024-09-16T14:52:08.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:08.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:08.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:08.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient(hunting): my global_id is 194271 2024-09-16T14:52:08.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 1 -- 172.21.15.26:0/205764245 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e0003530 con 0x7f19f809f240 2024-09-16T14:52:08.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 1 -- 172.21.15.26:0/205764245 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147322362 0 0) 0x7f19f80702a0 con 0x7f19f809fd30 2024-09-16T14:52:08.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:08.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:08.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:08.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient(hunting): my global_id is 214130 2024-09-16T14:52:08.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 1 -- 172.21.15.26:0/205764245 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f80a0f00 con 0x7f19f809fd30 2024-09-16T14:52:08.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 1 -- 172.21.15.26:0/205764245 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 503384096 0 0) 0x7f19e0003530 con 0x7f19f809f240 2024-09-16T14:52:08.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 1 -- 172.21.15.26:0/205764245 >> v1:172.21.15.26:6790/0 conn(0x7f19f809fd30 legacy=0x7f19f813b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:08.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 1 -- 172.21.15.26:0/205764245 >> v1:172.21.15.189:6789/0 conn(0x7f19f8071520 legacy=0x7f19f8137ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:08.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:52:08.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 1 monclient: found mon.noname-a 2024-09-16T14:52:08.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:08.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:52:08.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 1 -- 172.21.15.26:0/205764245 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f8073790 con 0x7f19f809f240 2024-09-16T14:52:08.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient: _finish_auth 0 2024-09-16T14:52:08.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:52:08.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:08.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 5 monclient: authenticate success, global_id 194271 2024-09-16T14:52:08.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:08.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 1 -- 172.21.15.26:0/205764245 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19e8001620 con 0x7f19f809f240 2024-09-16T14:52:08.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:08.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:08.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient: dump: 2024-09-16T14:52:08.233 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:08.233 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:08.233 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.233 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.233 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:08.233 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:08.233 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:08.233 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:08.233 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:08.233 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:08.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:08.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 1 -- 172.21.15.26:0/205764245 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19e80018c0 con 0x7f19f809f240 2024-09-16T14:52:08.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:08.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 1 -- 172.21.15.26:0/205764245 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19e8001c50 con 0x7f19f809f240 2024-09-16T14:52:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 4 set_mon_vals no callback set 2024-09-16T14:52:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f67fc640 10 monclient: dump: 2024-09-16T14:52:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:08.235 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient: shutdown 2024-09-16T14:52:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 1 -- 172.21.15.26:0/205764245 >> v1:172.21.15.26:6789/0 conn(0x7f19f809f240 legacy=0x7f19f809f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 1 -- 172.21.15.26:0/205764245 shutdown_connections 2024-09-16T14:52:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 1 -- 172.21.15.26:0/205764245 >> 172.21.15.26:0/205764245 conn(0x7f19f80a3e30 msgr2=0x7f19f80a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 1 -- 172.21.15.26:0/205764245 shutdown_connections 2024-09-16T14:52:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 1 -- 172.21.15.26:0/205764245 wait complete. 2024-09-16T14:52:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient: build_initial_monmap 2024-09-16T14:52:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient: monmap: 2024-09-16T14:52:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:08.236 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 1 Processor -- start 2024-09-16T14:52:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 1 -- start start 2024-09-16T14:52:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient: init 2024-09-16T14:52:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:52:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient(hunting): picked mon.noname-c con 0x7f19f8071520 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient(hunting): picked mon.noname-a con 0x7f19f809f240 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient(hunting): picked mon.noname-b con 0x7f19f809fd30 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f80a2a00 con 0x7f19f809f240 2024-09-16T14:52:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f8072000 con 0x7f19f809fd30 2024-09-16T14:52:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f81385f0 con 0x7f19f8071520 2024-09-16T14:52:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19fdd32640 10 monclient(hunting): authenticate will time out at 1552.343613s 2024-09-16T14:52:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f77fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f19f8071520 0x7f19f813b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46884/0 (socket says 172.21.15.26:46884) 2024-09-16T14:52:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19f77fe640 1 -- 172.21.15.26:0/1898026436 learned_addr learned my addr 172.21.15.26:0/1898026436 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453722796 0 0) 0x7f19f81385f0 con 0x7f19f8071520 2024-09-16T14:52:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19dffff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19dffff640 10 monclient(hunting): my global_id is 214133 2024-09-16T14:52:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e4003530 con 0x7f19f8071520 2024-09-16T14:52:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410644311 0 0) 0x7f19f80a2a00 con 0x7f19f809f240 2024-09-16T14:52:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19dffff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19dffff640 10 monclient(hunting): my global_id is 194274 2024-09-16T14:52:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.220+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f81385f0 con 0x7f19f809f240 2024-09-16T14:52:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1773644000 0 0) 0x7f19e4003530 con 0x7f19f8071520 2024-09-16T14:52:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19f80a2a00 con 0x7f19f8071520 2024-09-16T14:52:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19e8001bd0 con 0x7f19f8071520 2024-09-16T14:52:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:52:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient(hunting): dump: 2024-09-16T14:52:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:08.240 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1271998002 0 0) 0x7f19f81385f0 con 0x7f19f809f240 2024-09-16T14:52:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19e4003530 con 0x7f19f809f240 2024-09-16T14:52:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19f77fe640 1 -- 172.21.15.26:0/1898026436 >> v1:172.21.15.26:6790/0 conn(0x7f19f8071520 legacy=0x7f19f813b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T14:52:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19f77fe640 1 -- 172.21.15.26:0/1898026436 >> v1:172.21.15.26:6790/0 conn(0x7f19f8071520 legacy=0x7f19f813b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:52:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19ec001800 con 0x7f19f809f240 2024-09-16T14:52:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19f77fe640 1 --1- 172.21.15.26:0/1898026436 >> v1:172.21.15.26:6790/0 conn(0x7f19f8071520 0x7f19f813b520 :-1 s=OPENED pgs=1171 cs=1 l=1).handle_message read tag failed 2024-09-16T14:52:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19f77fe640 1 --1- 172.21.15.26:0/1898026436 >> v1:172.21.15.26:6790/0 conn(0x7f19f8071520 0x7f19f813b520 :-1 s=OPENED pgs=1171 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:52:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient(hunting): dump: 2024-09-16T14:52:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:08.241 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6790/0 2024-09-16T14:52:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940808784 0 0) 0x7f19f8072000 con 0x7f19f809fd30 2024-09-16T14:52:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient(hunting): my global_id is 184459 2024-09-16T14:52:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f81385f0 con 0x7f19f809fd30 2024-09-16T14:52:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 115230782 0 0) 0x7f19e4003530 con 0x7f19f809f240 2024-09-16T14:52:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 >> v1:172.21.15.26:6790/0 conn(0x7f19f8071520 legacy=0x7f19f813b520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 >> v1:172.21.15.189:6789/0 conn(0x7f19f809fd30 legacy=0x7f19f81cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient: _finish_hunting 0 2024-09-16T14:52:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 monclient: found mon.a 2024-09-16T14:52:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f81388d0 con 0x7f19f809f240 2024-09-16T14:52:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient: _finish_auth 0 2024-09-16T14:52:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient: _check_auth_tickets 2024-09-16T14:52:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19fdd32640 5 monclient: authenticate success, global_id 194274 2024-09-16T14:52:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19fdd32640 10 monclient: _renew_subs 2024-09-16T14:52:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19fdd32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19fdd32640 1 -- 172.21.15.26:0/1898026436 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f8136ac0 con 0x7f19f809f240 2024-09-16T14:52:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19fdd32640 10 monclient: _renew_subs 2024-09-16T14:52:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19fdd32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19fdd32640 1 -- 172.21.15.26:0/1898026436 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f81d0990 con 0x7f19f809f240 2024-09-16T14:52:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19ec003620 con 0x7f19f809f240 2024-09-16T14:52:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19ec003930 con 0x7f19f809f240 2024-09-16T14:52:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19f57fa640 4 set_mon_vals no callback set 2024-09-16T14:52:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 10 monclient: dump: 2024-09-16T14:52:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:08.245 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19fdd32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19fdd32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19fdd32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19fdd32640 1 -- 172.21.15.26:0/1898026436 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d8005050 con 0x7f19f809f240 2024-09-16T14:52:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f19ec003a90 con 0x7f19f809f240 2024-09-16T14:52:08.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f19ec042b50 con 0x7f19f809f240 2024-09-16T14:52:08.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.224+0000 7f19f77fe640 10 monclient: get_auth_request con 0x7f19e4036370 auth_method 0 2024-09-16T14:52:08.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.228+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f19ec013490 con 0x7f19f809f240 2024-09-16T14:52:08.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.228+0000 7f19dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:08.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.228+0000 7f19dffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.444+0000 7f19fdd32640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:52:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.444+0000 7f19fdd32640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:52:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.444+0000 7f19fdd32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.444+0000 7f19fdd32640 1 -- 172.21.15.26:0/1898026436 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f19d8005b10 con 0x7f19f809f240 2024-09-16T14:52:08.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.448+0000 7f19dffff640 1 -- 172.21.15.26:0/1898026436 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f19ec01ba80 con 0x7f19f809f240 2024-09-16T14:52:08.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.448+0000 7f19dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:52:08.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.448+0000 7f19dffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:08.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.448+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.448+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=11,logm=49} crc {auth=1561045913,config=692619927,health=2400470938,kv=1867599010,logm=2517362053}) 2024-09-16T14:52:08.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.448+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=11,kv=11,logm=54} crc {auth=2375110967,config=692619927,health=4185428187,kv=1867599010,logm=932067192}) 2024-09-16T14:52:08.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.448+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.448+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=11,logm=49} crc {auth=1561045913,config=692619927,health=2400470938,kv=1867599010,logm=2517362053}) 2024-09-16T14:52:08.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.448+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=12,kv=10,logm=52} crc {auth=2695423434,config=2435484489,health=2400470937,kv=2084167376,logm=1736219653}) 2024-09-16T14:52:08.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.452+0000 7f19fdd32640 1 -- 172.21.15.26:0/1898026436 >> v1:172.21.15.189:6812/3196161582 conn(0x7f19e4036370 legacy=0x7f19e4038830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:08.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.452+0000 7f19fdd32640 10 monclient: shutdown 2024-09-16T14:52:08.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.452+0000 7f19fdd32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:08.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.452+0000 7f19fdd32640 1 -- 172.21.15.26:0/1898026436 >> v1:172.21.15.26:6789/0 conn(0x7f19f809f240 legacy=0x7f19f81cbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:08.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.452+0000 7f19fdd32640 1 -- 172.21.15.26:0/1898026436 shutdown_connections 2024-09-16T14:52:08.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.452+0000 7f19fdd32640 1 -- 172.21.15.26:0/1898026436 >> 172.21.15.26:0/1898026436 conn(0x7f19f80a3e30 msgr2=0x7f19f8132dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:08.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.452+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.452+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2871165756}) 2024-09-16T14:52:08.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.452+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3595826959}) 2024-09-16T14:52:08.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.452+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.452+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2871165756}) 2024-09-16T14:52:08.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.452+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3646873285}) 2024-09-16T14:52:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.452+0000 7f19fdd32640 1 -- 172.21.15.26:0/1898026436 shutdown_connections 2024-09-16T14:52:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:08.452+0000 7f19fdd32640 1 -- 172.21.15.26:0/1898026436 wait complete. 2024-09-16T14:52:08.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.452+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.452+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=888322952}) 2024-09-16T14:52:08.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.452+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3104269096}) 2024-09-16T14:52:08.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.452+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.452+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=888322952}) 2024-09-16T14:52:08.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.452+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3386275894}) 2024-09-16T14:52:08.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.456+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.456+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=455095699}) 2024-09-16T14:52:08.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.456+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3537613203}) 2024-09-16T14:52:08.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.456+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.456+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=455095699}) 2024-09-16T14:52:08.458 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.456+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=649374166}) 2024-09-16T14:52:08.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.456+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.456+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1307217093}) 2024-09-16T14:52:08.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.456+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2112940644}) 2024-09-16T14:52:08.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.456+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.456+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1307217093}) 2024-09-16T14:52:08.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.456+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=197342481}) 2024-09-16T14:52:08.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.460+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.460+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2708095410}) 2024-09-16T14:52:08.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.460+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450330371}) 2024-09-16T14:52:08.462 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.460+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.462 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.460+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2708095410}) 2024-09-16T14:52:08.462 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.460+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=858397370}) 2024-09-16T14:52:08.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.460+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.460+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=728280373}) 2024-09-16T14:52:08.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.460+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445278548}) 2024-09-16T14:52:08.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.460+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.460+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=728280373}) 2024-09-16T14:52:08.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.460+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1321871257}) 2024-09-16T14:52:08.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.464+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.464+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3736479020}) 2024-09-16T14:52:08.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.464+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=859885790}) 2024-09-16T14:52:08.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.464+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.464+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3736479020}) 2024-09-16T14:52:08.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.464+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1166352126}) 2024-09-16T14:52:08.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.464+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.464+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3806785495}) 2024-09-16T14:52:08.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.464+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151914507}) 2024-09-16T14:52:08.467 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.464+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.467 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.464+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3806785495}) 2024-09-16T14:52:08.467 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.464+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1814068613}) 2024-09-16T14:52:08.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.468+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.468+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=736848595}) 2024-09-16T14:52:08.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.468+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182293222}) 2024-09-16T14:52:08.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.468+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.469 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.468+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=736848595}) 2024-09-16T14:52:08.469 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.468+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2952597727}) 2024-09-16T14:52:08.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.468+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.468+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=905511501}) 2024-09-16T14:52:08.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.468+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=785972709}) 2024-09-16T14:52:08.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.468+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.468+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=905511501}) 2024-09-16T14:52:08.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.468+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=469527498}) 2024-09-16T14:52:08.474 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:52:08.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.468+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.468+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3994842452}) 2024-09-16T14:52:08.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.468+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117157290}) 2024-09-16T14:52:08.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.468+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.472+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3994842452}) 2024-09-16T14:52:08.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.472+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4154199968}) 2024-09-16T14:52:08.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.472+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.472+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2035761157}) 2024-09-16T14:52:08.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.472+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=303896239}) 2024-09-16T14:52:08.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.472+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.472+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2035761157}) 2024-09-16T14:52:08.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.472+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1732997854}) 2024-09-16T14:52:08.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.472+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.472+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=673345005}) 2024-09-16T14:52:08.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.472+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1309313405}) 2024-09-16T14:52:08.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.472+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.472+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=673345005}) 2024-09-16T14:52:08.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.472+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3629010497}) 2024-09-16T14:52:08.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.476+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.476+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=818687106}) 2024-09-16T14:52:08.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.476+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3279716580}) 2024-09-16T14:52:08.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.476+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.476+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=818687106}) 2024-09-16T14:52:08.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.476+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1365897454}) 2024-09-16T14:52:08.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.476+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.476+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=33548529}) 2024-09-16T14:52:08.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.476+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=238241824}) 2024-09-16T14:52:08.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.476+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.476+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=33548529}) 2024-09-16T14:52:08.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.476+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1741866313}) 2024-09-16T14:52:08.480 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.480+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.480 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.480+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1298772431}) 2024-09-16T14:52:08.480 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.480+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2342903760}) 2024-09-16T14:52:08.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.480+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.480+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1298772431}) 2024-09-16T14:52:08.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.480+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1908484679}) 2024-09-16T14:52:08.482 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.480+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.482 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.480+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1892201324}) 2024-09-16T14:52:08.482 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.480+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2672225182}) 2024-09-16T14:52:08.482 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.480+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.480+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1892201324}) 2024-09-16T14:52:08.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.480+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3210249596}) 2024-09-16T14:52:08.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.484+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.484+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2461994567}) 2024-09-16T14:52:08.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.484+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463126845}) 2024-09-16T14:52:08.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.484+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.484+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2461994567}) 2024-09-16T14:52:08.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.484+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1078408421}) 2024-09-16T14:52:08.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.484+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.484+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371825679}) 2024-09-16T14:52:08.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.484+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3633103988}) 2024-09-16T14:52:08.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.484+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.484+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371825679}) 2024-09-16T14:52:08.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.484+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1305542580}) 2024-09-16T14:52:08.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=61528760}) 2024-09-16T14:52:08.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3739206032}) 2024-09-16T14:52:08.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=61528760}) 2024-09-16T14:52:08.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3745537591}) 2024-09-16T14:52:08.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1562359255}) 2024-09-16T14:52:08.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050498035}) 2024-09-16T14:52:08.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1562359255}) 2024-09-16T14:52:08.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1729340186}) 2024-09-16T14:52:08.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2340852840}) 2024-09-16T14:52:08.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3054211648}) 2024-09-16T14:52:08.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2340852840}) 2024-09-16T14:52:08.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.488+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2122856875}) 2024-09-16T14:52:08.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.492+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.492+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=194328927}) 2024-09-16T14:52:08.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.492+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3835276705}) 2024-09-16T14:52:08.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.492+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.492+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=194328927}) 2024-09-16T14:52:08.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.492+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781513015}) 2024-09-16T14:52:08.495 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.492+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.492+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756977664}) 2024-09-16T14:52:08.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.492+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3272848182}) 2024-09-16T14:52:08.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.492+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.492+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756977664}) 2024-09-16T14:52:08.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.492+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2460414717}) 2024-09-16T14:52:08.497 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.496+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.496+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3871534250}) 2024-09-16T14:52:08.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.496+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=500725428}) 2024-09-16T14:52:08.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.496+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.496+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3871534250}) 2024-09-16T14:52:08.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.496+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=388212057}) 2024-09-16T14:52:08.499 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.496+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.499 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.496+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3427569871}) 2024-09-16T14:52:08.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.496+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1607556898}) 2024-09-16T14:52:08.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.496+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.496+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3427569871}) 2024-09-16T14:52:08.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.496+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3981046497}) 2024-09-16T14:52:08.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.500+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.500+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1025418577}) 2024-09-16T14:52:08.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.500+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788835953}) 2024-09-16T14:52:08.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.500+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.500+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1025418577}) 2024-09-16T14:52:08.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.500+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=980101078}) 2024-09-16T14:52:08.503 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.500+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.503 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.500+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2613377800}) 2024-09-16T14:52:08.504 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.500+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=289538874}) 2024-09-16T14:52:08.504 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.500+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.504 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.500+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2613377800}) 2024-09-16T14:52:08.504 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.500+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1034534893}) 2024-09-16T14:52:08.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.504+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.504+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=769920166}) 2024-09-16T14:52:08.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.504+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=836300024}) 2024-09-16T14:52:08.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.504+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.504+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=769920166}) 2024-09-16T14:52:08.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.504+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3306304421}) 2024-09-16T14:52:08.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.504+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.504+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3840487313}) 2024-09-16T14:52:08.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.504+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1990433385}) 2024-09-16T14:52:08.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.504+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.504+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3840487313}) 2024-09-16T14:52:08.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.504+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=412728903}) 2024-09-16T14:52:08.508 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.508+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.508 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.508+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1006469680}) 2024-09-16T14:52:08.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.508+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48710558}) 2024-09-16T14:52:08.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.508+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.508+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1006469680}) 2024-09-16T14:52:08.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.508+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1712796623}) 2024-09-16T14:52:08.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.508+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.508+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2490728135}) 2024-09-16T14:52:08.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.508+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=892648026}) 2024-09-16T14:52:08.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.508+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.508+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2490728135}) 2024-09-16T14:52:08.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.508+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1904989089}) 2024-09-16T14:52:08.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.512+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.512+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2667718994}) 2024-09-16T14:52:08.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.512+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412873517}) 2024-09-16T14:52:08.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.512+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.512+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2667718994}) 2024-09-16T14:52:08.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.512+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4027270575}) 2024-09-16T14:52:08.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.512+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.512+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2671835255}) 2024-09-16T14:52:08.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.512+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635955174}) 2024-09-16T14:52:08.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.512+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.512+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2671835255}) 2024-09-16T14:52:08.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.512+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=981213580}) 2024-09-16T14:52:08.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.516+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.516+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4106953041}) 2024-09-16T14:52:08.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.516+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4026155958}) 2024-09-16T14:52:08.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.516+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.516+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4106953041}) 2024-09-16T14:52:08.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.516+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=122855401}) 2024-09-16T14:52:08.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.516+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.516+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3704293748}) 2024-09-16T14:52:08.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.516+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=219684810}) 2024-09-16T14:52:08.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.516+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.516+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3704293748}) 2024-09-16T14:52:08.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.516+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2860910365}) 2024-09-16T14:52:08.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.516+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4206117318}) 2024-09-16T14:52:08.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=270076153}) 2024-09-16T14:52:08.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4206117318}) 2024-09-16T14:52:08.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256997930}) 2024-09-16T14:52:08.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3978928428}) 2024-09-16T14:52:08.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3061602325}) 2024-09-16T14:52:08.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3978928428}) 2024-09-16T14:52:08.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2026578078}) 2024-09-16T14:52:08.523 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=252996931}) 2024-09-16T14:52:08.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1079337795}) 2024-09-16T14:52:08.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=252996931}) 2024-09-16T14:52:08.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.520+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3431604691}) 2024-09-16T14:52:08.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.524+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.524+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2533953005}) 2024-09-16T14:52:08.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.524+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2750767888}) 2024-09-16T14:52:08.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.524+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.524+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2533953005}) 2024-09-16T14:52:08.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.524+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3820923263}) 2024-09-16T14:52:08.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.524+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.524+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1300737491}) 2024-09-16T14:52:08.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.524+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1764925847}) 2024-09-16T14:52:08.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.524+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.524+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1300737491}) 2024-09-16T14:52:08.528 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.524+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2716368008}) 2024-09-16T14:52:08.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.528+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.528+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2546602448}) 2024-09-16T14:52:08.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.528+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1083811150}) 2024-09-16T14:52:08.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.528+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.528+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2546602448}) 2024-09-16T14:52:08.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.528+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=704932939}) 2024-09-16T14:52:08.530 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.528+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.530 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.528+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2535074812}) 2024-09-16T14:52:08.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.528+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1293471835}) 2024-09-16T14:52:08.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.528+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.528+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2535074812}) 2024-09-16T14:52:08.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.528+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=579885141}) 2024-09-16T14:52:08.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.532+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.532+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=54263668}) 2024-09-16T14:52:08.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.532+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2726810784}) 2024-09-16T14:52:08.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.532+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.532+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=54263668}) 2024-09-16T14:52:08.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.532+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1466215484}) 2024-09-16T14:52:08.535 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.532+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.535 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.532+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4024453578}) 2024-09-16T14:52:08.535 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.532+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1187756564}) 2024-09-16T14:52:08.535 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.532+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.532+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4024453578}) 2024-09-16T14:52:08.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.532+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3993920150}) 2024-09-16T14:52:08.537 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.536+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.537 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.536+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3052997700}) 2024-09-16T14:52:08.537 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.536+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203640547}) 2024-09-16T14:52:08.537 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.536+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.537 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.536+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3052997700}) 2024-09-16T14:52:08.537 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.536+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=672342634}) 2024-09-16T14:52:08.539 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.536+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.539 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.536+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3037675349}) 2024-09-16T14:52:08.539 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.536+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1005526544}) 2024-09-16T14:52:08.539 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.536+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.539 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.536+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3037675349}) 2024-09-16T14:52:08.539 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.536+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=672052501}) 2024-09-16T14:52:08.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.540+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.540+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2167386678}) 2024-09-16T14:52:08.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.540+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1569122002}) 2024-09-16T14:52:08.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.540+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.540+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2167386678}) 2024-09-16T14:52:08.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.540+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2398952173}) 2024-09-16T14:52:08.543 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.540+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.543 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.540+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=257730858}) 2024-09-16T14:52:08.543 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.540+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2726188740}) 2024-09-16T14:52:08.543 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.540+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.543 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.540+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=257730858}) 2024-09-16T14:52:08.543 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.540+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=53955453}) 2024-09-16T14:52:08.544 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.544+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.544+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1343384069}) 2024-09-16T14:52:08.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.544+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624266993}) 2024-09-16T14:52:08.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.544+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.544+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1343384069}) 2024-09-16T14:52:08.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.544+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2423271747}) 2024-09-16T14:52:08.547 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.544+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.547 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.544+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2341642557}) 2024-09-16T14:52:08.547 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.544+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172025811}) 2024-09-16T14:52:08.547 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.544+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.547 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.544+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2341642557}) 2024-09-16T14:52:08.547 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.544+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=321893633}) 2024-09-16T14:52:08.548 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.548+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.548 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.548+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1830711760}) 2024-09-16T14:52:08.548 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.548+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2368567918}) 2024-09-16T14:52:08.549 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.548+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.549 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.548+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1830711760}) 2024-09-16T14:52:08.549 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.548+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1308900135}) 2024-09-16T14:52:08.550 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.548+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.550 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.548+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4150894491}) 2024-09-16T14:52:08.550 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.548+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2601516210}) 2024-09-16T14:52:08.550 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.548+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.550 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.548+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4150894491}) 2024-09-16T14:52:08.550 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.548+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1310336237}) 2024-09-16T14:52:08.553 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.552+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.553 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.552+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=31,mdsmap=2,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=26} crc {logm=3912791789,mdsmap=3585761405,mgr=2006779800,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=4092277126,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3968680119}) 2024-09-16T14:52:08.553 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.552+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=31,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=23} crc {logm=437530170,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2882083426,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1473193302}) 2024-09-16T14:52:08.553 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.552+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.553 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.552+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=31,mdsmap=2,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=26} crc {logm=3912791789,mdsmap=3585761405,mgr=2006779800,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=4092277126,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3968680119}) 2024-09-16T14:52:08.553 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.552+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=32,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=22} crc {logm=905078065,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2882083426,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1427013293}) 2024-09-16T14:52:08.564 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.564+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.564 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.564+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=27} crc {osdmap=1202472651}) 2024-09-16T14:52:08.565 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.564+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=947665354}) 2024-09-16T14:52:08.565 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.564+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:08.565 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.564+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=27} crc {osdmap=1202472651}) 2024-09-16T14:52:08.565 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:08.564+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=28} crc {osdmap=3517865269}) 2024-09-16T14:52:09.475 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:52:09.476 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:52:09.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c79832640 10 monclient: get_monmap_and_config 2024-09-16T14:52:09.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c79832640 10 monclient: build_initial_monmap 2024-09-16T14:52:09.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c79832640 10 monclient: monmap: 2024-09-16T14:52:09.606 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:09.606 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:09.606 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:09.607091+0000 2024-09-16T14:52:09.606 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:09.607091+0000 2024-09-16T14:52:09.606 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:09.606 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:09.606 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:09.607 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:09.607 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:09.607 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:09.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c79832640 1 Processor -- start 2024-09-16T14:52:09.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c79832640 1 -- start start 2024-09-16T14:52:09.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c79832640 10 monclient: init 2024-09-16T14:52:09.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c79832640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:09.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c79832640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:52:09.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c79832640 10 monclient(hunting): picked mon.noname-c con 0x7f6c7409a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:09.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c79832640 10 monclient(hunting): picked mon.noname-b con 0x7f6c7409d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:09.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c79832640 10 monclient(hunting): picked mon.noname-a con 0x7f6c7409b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:09.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c79832640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c7409dc60 con 0x7f6c7409b040 2024-09-16T14:52:09.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c79832640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c74099ae0 con 0x7f6c7409a550 2024-09-16T14:52:09.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c79832640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c7409b8f0 con 0x7f6c7409d1b0 2024-09-16T14:52:09.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c79832640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:09.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c79832640 10 monclient(hunting): authenticate will time out at 1553.727407s 2024-09-16T14:52:09.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c6bfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6c7409d1b0 0x7f6c74136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46892/0 (socket says 172.21.15.26:46892) 2024-09-16T14:52:09.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c6bfff640 1 -- 172.21.15.26:0/2777992532 learned_addr learned my addr 172.21.15.26:0/2777992532 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:09.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c727fc640 1 -- 172.21.15.26:0/2777992532 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182119842 0 0) 0x7f6c7409b8f0 con 0x7f6c7409d1b0 2024-09-16T14:52:09.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c727fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:09.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:09.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:09.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c727fc640 10 monclient(hunting): my global_id is 214136 2024-09-16T14:52:09.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c727fc640 1 -- 172.21.15.26:0/2777992532 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c60005740 con 0x7f6c7409d1b0 2024-09-16T14:52:09.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c727fc640 1 -- 172.21.15.26:0/2777992532 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7648725 0 0) 0x7f6c7409dc60 con 0x7f6c7409b040 2024-09-16T14:52:09.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c727fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:09.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:09.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:09.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c727fc640 10 monclient(hunting): my global_id is 194277 2024-09-16T14:52:09.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.604+0000 7f6c727fc640 1 -- 172.21.15.26:0/2777992532 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c7409b8f0 con 0x7f6c7409b040 2024-09-16T14:52:09.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 1 -- 172.21.15.26:0/2777992532 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3771140191 0 0) 0x7f6c60005740 con 0x7f6c7409d1b0 2024-09-16T14:52:09.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 1 -- 172.21.15.26:0/2777992532 >> v1:172.21.15.189:6789/0 conn(0x7f6c7409a550 legacy=0x7f6c7409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:09.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 1 -- 172.21.15.26:0/2777992532 >> v1:172.21.15.26:6789/0 conn(0x7f6c7409b040 legacy=0x7f6c7413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:09.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 10 monclient: _finish_hunting 0 2024-09-16T14:52:09.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 1 monclient: found mon.noname-b 2024-09-16T14:52:09.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:09.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:52:09.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 1 -- 172.21.15.26:0/2777992532 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7409bbd0 con 0x7f6c7409d1b0 2024-09-16T14:52:09.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 10 monclient: _finish_auth 0 2024-09-16T14:52:09.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 10 monclient: _check_auth_tickets 2024-09-16T14:52:09.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:09.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 1 -- 172.21.15.26:0/2777992532 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6c5c001620 con 0x7f6c7409d1b0 2024-09-16T14:52:09.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 5 monclient: authenticate success, global_id 214136 2024-09-16T14:52:09.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:09.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:09.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:52:09.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 10 monclient: dump: 2024-09-16T14:52:09.618 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:09.618 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:09.618 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:09.618 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:09.618 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:09.618 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:09.618 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:09.619 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:09.619 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:09.619 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:09.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:09.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 1 -- 172.21.15.26:0/2777992532 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c5c001880 con 0x7f6c7409d1b0 2024-09-16T14:52:09.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:09.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 1 -- 172.21.15.26:0/2777992532 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6c5c001c50 con 0x7f6c7409d1b0 2024-09-16T14:52:09.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:09.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 4 set_mon_vals no callback set 2024-09-16T14:52:09.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:09.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:09.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c727fc640 10 monclient: dump: 2024-09-16T14:52:09.619 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:09.619 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:09.620 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 10 monclient: shutdown 2024-09-16T14:52:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 1 -- 172.21.15.26:0/2777992532 >> v1:172.21.15.26:6790/0 conn(0x7f6c7409d1b0 legacy=0x7f6c74136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 1 -- 172.21.15.26:0/2777992532 shutdown_connections 2024-09-16T14:52:09.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 1 -- 172.21.15.26:0/2777992532 >> 172.21.15.26:0/2777992532 conn(0x7f6c7409fb50 msgr2=0x7f6c7409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 1 -- 172.21.15.26:0/2777992532 shutdown_connections 2024-09-16T14:52:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 1 -- 172.21.15.26:0/2777992532 wait complete. 2024-09-16T14:52:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 10 monclient: build_initial_monmap 2024-09-16T14:52:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 10 monclient: monmap: 2024-09-16T14:52:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:09.621 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:09.622 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:09.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 1 Processor -- start 2024-09-16T14:52:09.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 1 -- start start 2024-09-16T14:52:09.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 10 monclient: init 2024-09-16T14:52:09.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:09.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:52:09.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 10 monclient(hunting): picked mon.noname-a con 0x7f6c7409a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:09.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 10 monclient(hunting): picked mon.noname-c con 0x7f6c7409b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:09.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 10 monclient(hunting): picked mon.noname-b con 0x7f6c7409d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:09.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c740a24a0 con 0x7f6c7409a550 2024-09-16T14:52:09.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c7409a240 con 0x7f6c7409d1b0 2024-09-16T14:52:09.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c741a3040 con 0x7f6c7409b040 2024-09-16T14:52:09.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:09.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 10 monclient(hunting): authenticate will time out at 1553.730618s 2024-09-16T14:52:09.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c72ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6c7409a550 0x7f6c741b4950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58400/0 (socket says 172.21.15.26:58400) 2024-09-16T14:52:09.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c72ffd640 1 -- 172.21.15.26:0/4097439383 learned_addr learned my addr 172.21.15.26:0/4097439383 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:09.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758675277 0 0) 0x7f6c740a24a0 con 0x7f6c7409a550 2024-09-16T14:52:09.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:52:09.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:09.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:09.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): my global_id is 194280 2024-09-16T14:52:09.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c54003530 con 0x7f6c7409a550 2024-09-16T14:52:09.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192904463 0 0) 0x7f6c741a3040 con 0x7f6c7409b040 2024-09-16T14:52:09.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:52:09.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:09.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:09.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): my global_id is 214139 2024-09-16T14:52:09.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c740a24a0 con 0x7f6c7409b040 2024-09-16T14:52:09.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2451224784 0 0) 0x7f6c54003530 con 0x7f6c7409a550 2024-09-16T14:52:09.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c741a3040 con 0x7f6c7409a550 2024-09-16T14:52:09.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6c64001970 con 0x7f6c7409a550 2024-09-16T14:52:09.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:09.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:09.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): dump: 2024-09-16T14:52:09.625 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:09.625 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:09.625 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:09.625 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:09.625 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:09.625 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:09.625 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:09.625 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:09.625 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:09.625 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:09.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:09.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473855520 0 0) 0x7f6c7409a240 con 0x7f6c7409d1b0 2024-09-16T14:52:09.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:52:09.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:09.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:09.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): my global_id is 184465 2024-09-16T14:52:09.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c54003530 con 0x7f6c7409d1b0 2024-09-16T14:52:09.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1568187138 0 0) 0x7f6c740a24a0 con 0x7f6c7409b040 2024-09-16T14:52:09.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c7409a240 con 0x7f6c7409b040 2024-09-16T14:52:09.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6c5c001b90 con 0x7f6c7409b040 2024-09-16T14:52:09.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:09.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:09.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient(hunting): dump: 2024-09-16T14:52:09.626 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:09.627 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:09.627 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:09.627 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:09.627 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:09.627 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:09.627 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:09.627 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:09.627 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:09.627 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:09.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1899204909 0 0) 0x7f6c741a3040 con 0x7f6c7409a550 2024-09-16T14:52:09.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 >> v1:172.21.15.26:6790/0 conn(0x7f6c7409b040 legacy=0x7f6c741ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:09.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 >> v1:172.21.15.189:6789/0 conn(0x7f6c7409d1b0 legacy=0x7f6c741cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:09.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:52:09.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 monclient: found mon.a 2024-09-16T14:52:09.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:09.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:09.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c741a3320 con 0x7f6c7409a550 2024-09-16T14:52:09.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient: _finish_auth 0 2024-09-16T14:52:09.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:52:09.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:09.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 5 monclient: authenticate success, global_id 194280 2024-09-16T14:52:09.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 10 monclient: _renew_subs 2024-09-16T14:52:09.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:09.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 1 -- 172.21.15.26:0/4097439383 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c741a1910 con 0x7f6c7409a550 2024-09-16T14:52:09.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 10 monclient: _renew_subs 2024-09-16T14:52:09.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:09.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c79832640 1 -- 172.21.15.26:0/4097439383 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c741b5060 con 0x7f6c7409a550 2024-09-16T14:52:09.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c64001c50 con 0x7f6c7409a550 2024-09-16T14:52:09.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:09.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c71ffb640 4 set_mon_vals no callback set 2024-09-16T14:52:09.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6c64003a70 con 0x7f6c7409a550 2024-09-16T14:52:09.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:09.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:09.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.608+0000 7f6c70ff9640 10 monclient: dump: 2024-09-16T14:52:09.629 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:09.629 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:09.629 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:09.629 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:09.629 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:09.629 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:09.630 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:09.630 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:09.630 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:09.630 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:09.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.612+0000 7f6c79832640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:09.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.612+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6c64003c50 con 0x7f6c7409a550 2024-09-16T14:52:09.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.612+0000 7f6c79832640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:09.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.612+0000 7f6c79832640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:09.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.612+0000 7f6c79832640 1 -- 172.21.15.26:0/4097439383 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c58005050 con 0x7f6c7409a550 2024-09-16T14:52:09.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.612+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f6c64043e20 con 0x7f6c7409a550 2024-09-16T14:52:09.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.612+0000 7f6c6bfff640 10 monclient: get_auth_request con 0x7f6c54036440 auth_method 0 2024-09-16T14:52:09.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.616+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6c64014700 con 0x7f6c7409a550 2024-09-16T14:52:09.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.616+0000 7f6c70ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:09.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.616+0000 7f6c70ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:09.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.832+0000 7f6c79832640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:52:09.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.832+0000 7f6c79832640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:09.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.832+0000 7f6c79832640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:09.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.832+0000 7f6c79832640 1 -- 172.21.15.26:0/4097439383 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6c58005880 con 0x7f6c7409a550 2024-09-16T14:52:09.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.832+0000 7f6c70ff9640 1 -- 172.21.15.26:0/4097439383 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 700880127) 0x7f6c640142f0 con 0x7f6c7409a550 2024-09-16T14:52:09.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.832+0000 7f6c70ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:09.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.832+0000 7f6c70ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:09.834 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":90,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:52:09.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.832+0000 7f6c79832640 1 -- 172.21.15.26:0/4097439383 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6c54036440 legacy=0x7f6c54038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:09.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.832+0000 7f6c79832640 10 monclient: shutdown 2024-09-16T14:52:09.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.832+0000 7f6c79832640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:09.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.832+0000 7f6c79832640 1 -- 172.21.15.26:0/4097439383 >> v1:172.21.15.26:6789/0 conn(0x7f6c7409a550 legacy=0x7f6c741b4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:09.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.832+0000 7f6c79832640 1 -- 172.21.15.26:0/4097439383 shutdown_connections 2024-09-16T14:52:09.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.832+0000 7f6c79832640 1 -- 172.21.15.26:0/4097439383 >> 172.21.15.26:0/4097439383 conn(0x7f6c7409fb50 msgr2=0x7f6c741b81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:09.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.832+0000 7f6c79832640 1 -- 172.21.15.26:0/4097439383 shutdown_connections 2024-09-16T14:52:09.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.832+0000 7f6c79832640 1 -- 172.21.15.26:0/4097439383 wait complete. 2024-09-16T14:52:09.851 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:52:09.851 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:52:09.851 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:52:09.851 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:52:09.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 10 monclient: get_monmap_and_config 2024-09-16T14:52:10.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 10 monclient: build_initial_monmap 2024-09-16T14:52:10.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 10 monclient: monmap: 2024-09-16T14:52:10.115 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:10.115 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.115 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:09.973710+0000 2024-09-16T14:52:10.115 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:09.973710+0000 2024-09-16T14:52:10.116 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:10.116 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.116 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:10.116 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:10.116 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:10.116 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 1 Processor -- start 2024-09-16T14:52:10.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 1 -- start start 2024-09-16T14:52:10.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 10 monclient: init 2024-09-16T14:52:10.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:10.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:52:10.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 10 monclient(hunting): picked mon.noname-a con 0x7f709c134e40 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:10.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 10 monclient(hunting): picked mon.noname-b con 0x7f709c099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:10.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 10 monclient(hunting): picked mon.noname-c con 0x7f709c135220 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:10.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f709c099e80 con 0x7f709c134e40 2024-09-16T14:52:10.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f709c0719d0 con 0x7f709c135220 2024-09-16T14:52:10.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f709c09e1e0 con 0x7f709c099a70 2024-09-16T14:52:10.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:10.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 10 monclient(hunting): authenticate will time out at 1554.093724s 2024-09-16T14:52:10.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709bfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f709c099a70 0x7f709c0712c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46910/0 (socket says 172.21.15.26:46910) 2024-09-16T14:52:10.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a0b8f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f709c134e40 0x7f709c070bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58410/0 (socket says 172.21.15.26:58410) 2024-09-16T14:52:10.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709bfff640 1 -- 172.21.15.26:0/4032457584 learned_addr learned my addr 172.21.15.26:0/4032457584 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:10.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 1 -- 172.21.15.26:0/4032457584 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161576245 0 0) 0x7f708c001120 con 0x7f709c134e40 2024-09-16T14:52:10.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient(hunting): my global_id is 194283 2024-09-16T14:52:10.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 1 -- 172.21.15.26:0/4032457584 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7084003530 con 0x7f709c134e40 2024-09-16T14:52:10.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 1 -- 172.21.15.26:0/4032457584 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640102503 0 0) 0x7f709c09e1e0 con 0x7f709c099a70 2024-09-16T14:52:10.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient(hunting): my global_id is 214142 2024-09-16T14:52:10.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 1 -- 172.21.15.26:0/4032457584 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f708c001120 con 0x7f709c099a70 2024-09-16T14:52:10.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 1 -- 172.21.15.26:0/4032457584 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 471642386 0 0) 0x7f7084003530 con 0x7f709c134e40 2024-09-16T14:52:10.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 1 -- 172.21.15.26:0/4032457584 >> v1:172.21.15.26:6790/0 conn(0x7f709c099a70 legacy=0x7f709c0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 1 -- 172.21.15.26:0/4032457584 >> v1:172.21.15.189:6789/0 conn(0x7f709c135220 legacy=0x7f709c13db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:52:10.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 1 monclient: found mon.noname-a 2024-09-16T14:52:10.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:10.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:52:10.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 1 -- 172.21.15.26:0/4032457584 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709c09e7d0 con 0x7f709c134e40 2024-09-16T14:52:10.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:52:10.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:52:10.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:10.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 5 monclient: authenticate success, global_id 194283 2024-09-16T14:52:10.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:10.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 1 -- 172.21.15.26:0/4032457584 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f708c001610 con 0x7f709c134e40 2024-09-16T14:52:10.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:10.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient: dump: 2024-09-16T14:52:10.128 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.128 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.128 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.128 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.128 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.128 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.128 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.128 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.128 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.128 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:10.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 1 -- 172.21.15.26:0/4032457584 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f708c001840 con 0x7f709c134e40 2024-09-16T14:52:10.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:10.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:10.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 4 set_mon_vals no callback set 2024-09-16T14:52:10.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 1 -- 172.21.15.26:0/4032457584 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f708c001bd0 con 0x7f709c134e40 2024-09-16T14:52:10.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:10.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f709b7fe640 10 monclient: dump: 2024-09-16T14:52:10.129 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.129 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.129 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.129 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.129 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.129 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.129 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.130 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.130 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.130 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 10 monclient: shutdown 2024-09-16T14:52:10.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:10.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 1 -- 172.21.15.26:0/4032457584 >> v1:172.21.15.26:6789/0 conn(0x7f709c134e40 legacy=0x7f709c070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 1 -- 172.21.15.26:0/4032457584 shutdown_connections 2024-09-16T14:52:10.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 1 -- 172.21.15.26:0/4032457584 >> 172.21.15.26:0/4032457584 conn(0x7f709c09fb50 msgr2=0x7f709c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:10.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 1 -- 172.21.15.26:0/4032457584 shutdown_connections 2024-09-16T14:52:10.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 1 -- 172.21.15.26:0/4032457584 wait complete. 2024-09-16T14:52:10.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 10 monclient: build_initial_monmap 2024-09-16T14:52:10.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.972+0000 7f70a2df3640 10 monclient: monmap: 2024-09-16T14:52:10.130 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:10.131 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:10.131 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:10.131 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:10.131 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:10.131 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:10.131 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:10.131 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:10.131 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:10.131 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 1 Processor -- start 2024-09-16T14:52:10.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 1 -- start start 2024-09-16T14:52:10.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 10 monclient: init 2024-09-16T14:52:10.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:10.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:52:10.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 10 monclient(hunting): picked mon.noname-a con 0x7f709c099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:10.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 10 monclient(hunting): picked mon.noname-b con 0x7f709c134e40 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:10.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 10 monclient(hunting): picked mon.noname-c con 0x7f709c135220 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:10.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f709c122de0 con 0x7f709c099a70 2024-09-16T14:52:10.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f709c09bdd0 con 0x7f709c134e40 2024-09-16T14:52:10.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f709c1345e0 con 0x7f709c135220 2024-09-16T14:52:10.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:10.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 10 monclient(hunting): authenticate will time out at 1554.096641s 2024-09-16T14:52:10.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a0b8f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f709c099a70 0x7f709c1c9930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58416/0 (socket says 172.21.15.26:58416) 2024-09-16T14:52:10.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a1390640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f709c135220 0x7f709c1d20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46920/0 (socket says 172.21.15.26:46920) 2024-09-16T14:52:10.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a0b8f640 1 -- 172.21.15.26:0/601423519 learned_addr learned my addr 172.21.15.26:0/601423519 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:10.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028383890 0 0) 0x7f709c122de0 con 0x7f709c099a70 2024-09-16T14:52:10.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): my global_id is 194286 2024-09-16T14:52:10.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7088003530 con 0x7f709c099a70 2024-09-16T14:52:10.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742940602 0 0) 0x7f709c1345e0 con 0x7f709c135220 2024-09-16T14:52:10.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): my global_id is 214145 2024-09-16T14:52:10.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f709c122de0 con 0x7f709c135220 2024-09-16T14:52:10.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2879502598 0 0) 0x7f7088003530 con 0x7f709c099a70 2024-09-16T14:52:10.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f709c1345e0 con 0x7f709c099a70 2024-09-16T14:52:10.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f708c001b50 con 0x7f709c099a70 2024-09-16T14:52:10.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:10.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): dump: 2024-09-16T14:52:10.134 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.134 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.134 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.134 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.134 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.134 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.134 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.134 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.134 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.134 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:10.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1133360178 0 0) 0x7f709c122de0 con 0x7f709c135220 2024-09-16T14:52:10.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7088003530 con 0x7f709c135220 2024-09-16T14:52:10.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7094001840 con 0x7f709c135220 2024-09-16T14:52:10.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:10.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): dump: 2024-09-16T14:52:10.135 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.135 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.135 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.135 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.135 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.135 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.135 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.135 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.136 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.136 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1945321417 0 0) 0x7f709c1345e0 con 0x7f709c099a70 2024-09-16T14:52:10.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 >> v1:172.21.15.26:6790/0 conn(0x7f709c135220 legacy=0x7f709c1d20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 >> v1:172.21.15.189:6789/0 conn(0x7f709c134e40 legacy=0x7f709c1ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient: _finish_hunting 0 2024-09-16T14:52:10.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 monclient: found mon.a 2024-09-16T14:52:10.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:10.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:10.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709c09a240 con 0x7f709c099a70 2024-09-16T14:52:10.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient: _finish_auth 0 2024-09-16T14:52:10.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient: _check_auth_tickets 2024-09-16T14:52:10.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:10.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 5 monclient: authenticate success, global_id 194286 2024-09-16T14:52:10.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 10 monclient: _renew_subs 2024-09-16T14:52:10.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:10.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 1 -- 172.21.15.26:0/601423519 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f709c123f90 con 0x7f709c099a70 2024-09-16T14:52:10.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 10 monclient: _renew_subs 2024-09-16T14:52:10.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:10.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 1 -- 172.21.15.26:0/601423519 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f709c124550 con 0x7f709c099a70 2024-09-16T14:52:10.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f708c001350 con 0x7f709c099a70 2024-09-16T14:52:10.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:10.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f708c004c40 con 0x7f709c099a70 2024-09-16T14:52:10.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:10.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient: dump: 2024-09-16T14:52:10.138 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.138 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.138 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.138 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.138 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.138 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.138 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.138 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.138 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.138 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f709affd640 4 set_mon_vals no callback set 2024-09-16T14:52:10.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:10.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a2df3640 1 -- 172.21.15.26:0/601423519 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f707c006590 con 0x7f707c002830 2024-09-16T14:52:10.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f708c004da0 con 0x7f709c099a70 2024-09-16T14:52:10.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f708c043ea0 con 0x7f709c099a70 2024-09-16T14:52:10.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842640387 0 0) 0x7f707c006590 con 0x7f707c002830 2024-09-16T14:52:10.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient(hunting): my global_id is 194289 2024-09-16T14:52:10.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f709c1345e0 con 0x7f707c002830 2024-09-16T14:52:10.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3275409437 0 0) 0x7f709c1345e0 con 0x7f707c002830 2024-09-16T14:52:10.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f707c006590 con 0x7f707c002830 2024-09-16T14:52:10.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f70900019a0 con 0x7f707c002830 2024-09-16T14:52:10.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:10.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient: dump: 2024-09-16T14:52:10.140 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.140 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.140 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.140 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.140 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.140 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.140 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.140 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.140 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.140 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70a1390640 10 monclient: get_auth_request con 0x7f7088036200 auth_method 0 2024-09-16T14:52:10.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2615391969 0 0) 0x7f707c006590 con 0x7f707c002830 2024-09-16T14:52:10.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f707c00a0e0 con 0x7f707c002830 2024-09-16T14:52:10.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7090004b70 con 0x7f707c002830 2024-09-16T14:52:10.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:10.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:10.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 >> v1:172.21.15.26:6789/0 conn(0x7f707c002830 legacy=0x7f707c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.976+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 >> v1:172.21.15.26:6789/0 conn(0x7f707c002830 legacy=0x7f707c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:10.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70a2df3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:52:10.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70a2df3640 1 -- 172.21.15.26:0/601423519 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f707c009da0 con 0x7f707c0090c0 2024-09-16T14:52:10.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681453366 0 0) 0x7f707c009da0 con 0x7f707c0090c0 2024-09-16T14:52:10.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 10 monclient(hunting): my global_id is 194292 2024-09-16T14:52:10.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f707c006590 con 0x7f707c0090c0 2024-09-16T14:52:10.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 191008251 0 0) 0x7f707c006590 con 0x7f707c0090c0 2024-09-16T14:52:10.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f707c009da0 con 0x7f707c0090c0 2024-09-16T14:52:10.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7090004d00 con 0x7f707c0090c0 2024-09-16T14:52:10.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:10.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 10 monclient: dump: 2024-09-16T14:52:10.142 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.142 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.142 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.142 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.142 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.142 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.142 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.142 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.143 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.143 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2292603601 0 0) 0x7f707c009da0 con 0x7f707c0090c0 2024-09-16T14:52:10.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f707c00c290 con 0x7f707c0090c0 2024-09-16T14:52:10.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 1033627843) 0x7f7090004ff0 con 0x7f707c0090c0 2024-09-16T14:52:10.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:52:10.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:10.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 >> v1:172.21.15.26:6789/0 conn(0x7f707c0090c0 legacy=0x7f707c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70997fa640 1 -- 172.21.15.26:0/601423519 >> v1:172.21.15.26:6789/0 conn(0x7f707c0090c0 legacy=0x7f707c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:10.143 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:52:10.144 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:10.144 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:10.144 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:52:10.144 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 90, 2024-09-16T14:52:10.144 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:52:10.144 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:52:10.144 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:52:10.144 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:52:10.144 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:10.144 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:52:10.144 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:10.144 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:52:10.145 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:52:10.145 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:10.145 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:10.145 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:10.145 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:10.145 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:10.145 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:10.145 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:10.145 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:10.145 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:10.145 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:10.145 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:10.145 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:52:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:52:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.146 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:52:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:52:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:52:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:52:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:52:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:52:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:52:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:52:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:52:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:52:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:52:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:52:10.147 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:52:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:52:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:10.148 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:52:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:52:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:52:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.149 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:52:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:52:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.150 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:10.151 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:52:10.151 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:10.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.151 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.151 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:52:10.151 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:52:10.151 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:10.151 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:10.151 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:10.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.151 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:10.152 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:10.152 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:10.152 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:10.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.152 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:10.152 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:52:10.152 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:10.152 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.152 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.152 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:52:10.152 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:52:10.152 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:10.153 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:10.153 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:10.153 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.153 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.153 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:52:10.153 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:52:10.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.153 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:10.153 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:10.153 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:10.153 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.153 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:10.153 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:52:10.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.154 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:10.154 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:10.154 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:52:10.154 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.154 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:10.154 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:52:10.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.155 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:10.155 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:10.155 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:52:10.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.155 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:10.155 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:52:10.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.155 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:10.155 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:10.155 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:10.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.155 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.155 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:52:10.156 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:52:10.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70a2df3640 1 -- 172.21.15.26:0/601423519 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7088036200 legacy=0x7f70880386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70a2df3640 10 monclient: shutdown 2024-09-16T14:52:10.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70a2df3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:10.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70a2df3640 1 -- 172.21.15.26:0/601423519 >> v1:172.21.15.26:6789/0 conn(0x7f709c099a70 legacy=0x7f709c1c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70a1390640 1 -- 172.21.15.26:0/601423519 reap_dead start 2024-09-16T14:52:10.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70a1390640 1 -- 172.21.15.26:0/601423519 reap_dead start 2024-09-16T14:52:10.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70a2df3640 1 -- 172.21.15.26:0/601423519 shutdown_connections 2024-09-16T14:52:10.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70a2df3640 1 -- 172.21.15.26:0/601423519 >> 172.21.15.26:0/601423519 conn(0x7f709c09fb50 msgr2=0x7f709c0a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:10.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70a2df3640 1 -- 172.21.15.26:0/601423519 shutdown_connections 2024-09-16T14:52:10.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:09.984+0000 7f70a2df3640 1 -- 172.21.15.26:0/601423519 wait complete. 2024-09-16T14:52:10.158 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:52:10.158 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:52:10.158 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:52:10.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.280+0000 7f0bf5b59640 10 monclient: get_monmap_and_config 2024-09-16T14:52:10.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.280+0000 7f0bf5b59640 10 monclient: build_initial_monmap 2024-09-16T14:52:10.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.280+0000 7f0bf5b59640 10 monclient: monmap: 2024-09-16T14:52:10.284 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:10.284 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.284 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:10.285336+0000 2024-09-16T14:52:10.284 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:10.285336+0000 2024-09-16T14:52:10.284 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:10.284 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.284 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:10.285 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:10.285 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:10.285 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 1 Processor -- start 2024-09-16T14:52:10.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 1 -- start start 2024-09-16T14:52:10.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 10 monclient: init 2024-09-16T14:52:10.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:10.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:52:10.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 10 monclient(hunting): picked mon.noname-b con 0x7f0bf009c480 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:10.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 10 monclient(hunting): picked mon.noname-a con 0x7f0bf009bbe0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:10.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 10 monclient(hunting): picked mon.noname-c con 0x7f0bf009c860 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:10.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf009d260 con 0x7f0bf009bbe0 2024-09-16T14:52:10.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf009e870 con 0x7f0bf009c860 2024-09-16T14:52:10.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf00a0a00 con 0x7f0bf009c480 2024-09-16T14:52:10.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:10.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 10 monclient(hunting): authenticate will time out at 1554.405688s 2024-09-16T14:52:10.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bef7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0bf009c480 0x7f0bf009da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46936/0 (socket says 172.21.15.26:46936) 2024-09-16T14:52:10.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0beeffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0bf009bbe0 0x7f0bf009e160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58442/0 (socket says 172.21.15.26:58442) 2024-09-16T14:52:10.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bef7fe640 1 -- 172.21.15.26:0/2147696879 learned_addr learned my addr 172.21.15.26:0/2147696879 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:10.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 1 -- 172.21.15.26:0/2147696879 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306516673 0 0) 0x7f0bf00a0a00 con 0x7f0bf009c480 2024-09-16T14:52:10.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient(hunting): my global_id is 214148 2024-09-16T14:52:10.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 1 -- 172.21.15.26:0/2147696879 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bdc003530 con 0x7f0bf009c480 2024-09-16T14:52:10.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 1 -- 172.21.15.26:0/2147696879 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876234961 0 0) 0x7f0bf009d260 con 0x7f0bf009bbe0 2024-09-16T14:52:10.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient(hunting): my global_id is 194295 2024-09-16T14:52:10.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 1 -- 172.21.15.26:0/2147696879 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf00a0a00 con 0x7f0bf009bbe0 2024-09-16T14:52:10.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 1 -- 172.21.15.26:0/2147696879 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 512278322 0 0) 0x7f0bdc003530 con 0x7f0bf009c480 2024-09-16T14:52:10.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 1 -- 172.21.15.26:0/2147696879 >> v1:172.21.15.189:6789/0 conn(0x7f0bf009c860 legacy=0x7f0bf0142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 1 -- 172.21.15.26:0/2147696879 >> v1:172.21.15.26:6789/0 conn(0x7f0bf009bbe0 legacy=0x7f0bf009e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:52:10.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 1 monclient: found mon.noname-b 2024-09-16T14:52:10.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:10.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:52:10.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 1 -- 172.21.15.26:0/2147696879 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf009a760 con 0x7f0bf009c480 2024-09-16T14:52:10.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient: _finish_auth 0 2024-09-16T14:52:10.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:52:10.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:10.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 1 -- 172.21.15.26:0/2147696879 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0be0001620 con 0x7f0bf009c480 2024-09-16T14:52:10.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 5 monclient: authenticate success, global_id 214148 2024-09-16T14:52:10.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:10.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:52:10.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient: dump: 2024-09-16T14:52:10.296 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.296 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.296 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.296 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.296 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.296 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.296 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.296 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.296 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.296 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:10.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 1 -- 172.21.15.26:0/2147696879 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0be0001880 con 0x7f0bf009c480 2024-09-16T14:52:10.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:10.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:10.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 1 -- 172.21.15.26:0/2147696879 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0be0001c50 con 0x7f0bf009c480 2024-09-16T14:52:10.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 4 set_mon_vals no callback set 2024-09-16T14:52:10.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:10.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bee7fc640 10 monclient: dump: 2024-09-16T14:52:10.297 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.297 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.297 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.297 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.297 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.298 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.298 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.298 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.298 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.298 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 10 monclient: shutdown 2024-09-16T14:52:10.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:10.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 1 -- 172.21.15.26:0/2147696879 >> v1:172.21.15.26:6790/0 conn(0x7f0bf009c480 legacy=0x7f0bf009da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 1 -- 172.21.15.26:0/2147696879 shutdown_connections 2024-09-16T14:52:10.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 1 -- 172.21.15.26:0/2147696879 >> 172.21.15.26:0/2147696879 conn(0x7f0bf009fb50 msgr2=0x7f0bf009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:10.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 1 -- 172.21.15.26:0/2147696879 shutdown_connections 2024-09-16T14:52:10.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 1 -- 172.21.15.26:0/2147696879 wait complete. 2024-09-16T14:52:10.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 10 monclient: build_initial_monmap 2024-09-16T14:52:10.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.284+0000 7f0bf5b59640 10 monclient: monmap: 2024-09-16T14:52:10.299 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:10.299 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:10.299 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:10.299 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:10.299 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:10.299 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:10.299 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:10.299 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:10.299 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:10.299 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 1 Processor -- start 2024-09-16T14:52:10.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 1 -- start start 2024-09-16T14:52:10.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 10 monclient: init 2024-09-16T14:52:10.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:10.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:52:10.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 10 monclient(hunting): picked mon.noname-a con 0x7f0bf009bbe0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:10.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 10 monclient(hunting): picked mon.noname-b con 0x7f0bf009c480 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:10.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 10 monclient(hunting): picked mon.noname-c con 0x7f0bf009c860 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:10.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf0071ec0 con 0x7f0bf009bbe0 2024-09-16T14:52:10.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf01bde10 con 0x7f0bf009c480 2024-09-16T14:52:10.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf00a27d0 con 0x7f0bf009c860 2024-09-16T14:52:10.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:10.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 10 monclient(hunting): authenticate will time out at 1554.408845s 2024-09-16T14:52:10.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bef7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0bf009bbe0 0x7f0bf01bd020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58448/0 (socket says 172.21.15.26:58448) 2024-09-16T14:52:10.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bef7fe640 1 -- 172.21.15.26:0/3060735786 learned_addr learned my addr 172.21.15.26:0/3060735786 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:10.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023194782 0 0) 0x7f0bf00a27d0 con 0x7f0bf009c860 2024-09-16T14:52:10.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): my global_id is 214151 2024-09-16T14:52:10.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bd0003530 con 0x7f0bf009c860 2024-09-16T14:52:10.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630531722 0 0) 0x7f0bf0071ec0 con 0x7f0bf009bbe0 2024-09-16T14:52:10.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): my global_id is 194298 2024-09-16T14:52:10.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf00a27d0 con 0x7f0bf009bbe0 2024-09-16T14:52:10.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3418031190 0 0) 0x7f0bd0003530 con 0x7f0bf009c860 2024-09-16T14:52:10.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bf0071ec0 con 0x7f0bf009c860 2024-09-16T14:52:10.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0be8001800 con 0x7f0bf009c860 2024-09-16T14:52:10.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:52:10.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): dump: 2024-09-16T14:52:10.302 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.302 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.302 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.302 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.302 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.302 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.302 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.302 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.302 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.303 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:10.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 950692723 0 0) 0x7f0bf00a27d0 con 0x7f0bf009bbe0 2024-09-16T14:52:10.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bd0003530 con 0x7f0bf009bbe0 2024-09-16T14:52:10.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0be0001b90 con 0x7f0bf009bbe0 2024-09-16T14:52:10.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:10.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): dump: 2024-09-16T14:52:10.303 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.303 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.303 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.303 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.303 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.304 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.304 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.304 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.304 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.304 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560381066 0 0) 0x7f0bf01bde10 con 0x7f0bf009c480 2024-09-16T14:52:10.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): my global_id is 184477 2024-09-16T14:52:10.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf00a27d0 con 0x7f0bf009c480 2024-09-16T14:52:10.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2174741188 0 0) 0x7f0bf0071ec0 con 0x7f0bf009c860 2024-09-16T14:52:10.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 >> v1:172.21.15.189:6789/0 conn(0x7f0bf009c480 legacy=0x7f0bf01d2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 >> v1:172.21.15.26:6789/0 conn(0x7f0bf009bbe0 legacy=0x7f0bf01bd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:52:10.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 monclient: found mon.c 2024-09-16T14:52:10.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:10.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:10.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf009a240 con 0x7f0bf009c860 2024-09-16T14:52:10.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient: _finish_auth 0 2024-09-16T14:52:10.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:52:10.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:10.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 5 monclient: authenticate success, global_id 214151 2024-09-16T14:52:10.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 10 monclient: _renew_subs 2024-09-16T14:52:10.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:10.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 1 -- 172.21.15.26:0/3060735786 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf01ab710 con 0x7f0bf009c860 2024-09-16T14:52:10.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 10 monclient: _renew_subs 2024-09-16T14:52:10.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:10.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 1 -- 172.21.15.26:0/3060735786 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf01a9fe0 con 0x7f0bf009c860 2024-09-16T14:52:10.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0be8003600 con 0x7f0bf009c860 2024-09-16T14:52:10.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:10.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0be8003910 con 0x7f0bf009c860 2024-09-16T14:52:10.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:10.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bedffb640 4 set_mon_vals no callback set 2024-09-16T14:52:10.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient: dump: 2024-09-16T14:52:10.310 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.310 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.310 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.310 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.310 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.310 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.310 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.310 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.310 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.310 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:10.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0bf5b59640 1 -- 172.21.15.26:0/3060735786 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0bd4006590 con 0x7f0bd4002830 2024-09-16T14:52:10.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0be800e380 con 0x7f0bf009c860 2024-09-16T14:52:10.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f0be8042a30 con 0x7f0bf009c860 2024-09-16T14:52:10.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012488690 0 0) 0x7f0bd4006590 con 0x7f0bd4002830 2024-09-16T14:52:10.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient(hunting): my global_id is 214154 2024-09-16T14:52:10.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf0071ec0 con 0x7f0bd4002830 2024-09-16T14:52:10.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3418603124 0 0) 0x7f0bf0071ec0 con 0x7f0bd4002830 2024-09-16T14:52:10.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bd4006590 con 0x7f0bd4002830 2024-09-16T14:52:10.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0be0004e20 con 0x7f0bd4002830 2024-09-16T14:52:10.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0beeffd640 10 monclient: get_auth_request con 0x7f0bd0036390 auth_method 0 2024-09-16T14:52:10.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:10.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient: dump: 2024-09-16T14:52:10.313 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.313 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.313 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.313 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.313 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.313 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.313 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.313 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.313 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.313 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1868802973 0 0) 0x7f0bd4006590 con 0x7f0bd4002830 2024-09-16T14:52:10.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0bd400a0e0 con 0x7f0bd4002830 2024-09-16T14:52:10.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0be0007240 con 0x7f0bd4002830 2024-09-16T14:52:10.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:10.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:10.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 >> v1:172.21.15.26:6790/0 conn(0x7f0bd4002830 legacy=0x7f0bd4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.288+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 >> v1:172.21.15.26:6790/0 conn(0x7f0bd4002830 legacy=0x7f0bd4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:10.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0bf5b59640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:52:10.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0bf5b59640 1 -- 172.21.15.26:0/3060735786 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0bd4009da0 con 0x7f0bd40090c0 2024-09-16T14:52:10.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973369967 0 0) 0x7f0bd4009da0 con 0x7f0bd40090c0 2024-09-16T14:52:10.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 10 monclient(hunting): my global_id is 214157 2024-09-16T14:52:10.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bd4006590 con 0x7f0bd40090c0 2024-09-16T14:52:10.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 168223951 0 0) 0x7f0bd4006590 con 0x7f0bd40090c0 2024-09-16T14:52:10.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bd4009da0 con 0x7f0bd40090c0 2024-09-16T14:52:10.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0be0005670 con 0x7f0bd40090c0 2024-09-16T14:52:10.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:10.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 10 monclient: dump: 2024-09-16T14:52:10.317 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.317 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.317 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.317 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.317 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.317 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.317 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.317 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.317 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.317 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 271762173 0 0) 0x7f0bd4009da0 con 0x7f0bd40090c0 2024-09-16T14:52:10.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0bd400c290 con 0x7f0bd40090c0 2024-09-16T14:52:10.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3881 (unknown 0 0 3835730173) 0x7f0be0006a90 con 0x7f0bd40090c0 2024-09-16T14:52:10.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:52:10.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:10.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 >> v1:172.21.15.26:6790/0 conn(0x7f0bd40090c0 legacy=0x7f0bd4009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0be7fff640 1 -- 172.21.15.26:0/3060735786 >> v1:172.21.15.26:6790/0 conn(0x7f0bd40090c0 legacy=0x7f0bd4009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:10.318 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:52:10.318 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:10.318 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:10.318 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:52:10.318 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 90, 2024-09-16T14:52:10.318 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:52:10.318 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:52:10.319 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:52:10.319 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:52:10.319 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:10.319 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:52:10.319 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:10.319 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:52:10.319 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:52:10.319 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:10.319 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:10.319 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:10.319 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:10.319 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:10.319 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:10.320 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:10.320 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:10.320 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:10.320 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:10.320 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:10.320 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:52:10.320 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:52:10.320 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:10.320 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:10.320 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:10.320 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:10.320 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:10.320 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:10.321 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:10.321 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:10.321 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:10.321 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:10.321 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.321 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:52:10.321 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:52:10.321 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:52:10.321 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:52:10.321 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:52:10.321 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:52:10.321 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:10.322 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:10.322 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:52:10.322 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:52:10.322 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:52:10.322 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:52:10.322 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:52:10.322 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:52:10.322 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:52:10.322 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:10.322 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:52:10.322 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:10.322 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:10.323 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:10.323 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:10.323 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:10.323 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:10.323 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:10.323 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:10.323 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:10.323 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:10.323 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:10.323 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:52:10.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.323 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:52:10.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.323 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:10.324 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:10.324 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:10.324 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:10.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.324 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:10.324 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:52:10.324 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:10.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.324 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.324 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:52:10.324 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:52:10.324 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:10.324 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:10.325 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:10.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.325 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:10.325 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:10.325 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:10.325 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:10.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.325 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:10.325 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:52:10.325 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:10.325 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.326 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.326 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:52:10.326 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:52:10.326 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:10.326 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:10.326 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:10.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.326 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:10.326 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:10.326 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:10.326 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:10.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.327 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:10.327 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:52:10.327 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:10.327 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.327 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.327 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:52:10.327 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:52:10.327 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:10.327 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:10.327 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:10.327 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.327 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.328 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:52:10.328 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:52:10.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.328 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:10.328 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:10.328 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:10.328 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.328 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:10.328 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:52:10.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.328 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:10.328 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:10.329 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:52:10.329 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.329 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.329 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:52:10.329 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:52:10.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0bf5b59640 1 -- 172.21.15.26:0/3060735786 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0bd0036390 legacy=0x7f0bd0038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0bf5b59640 10 monclient: shutdown 2024-09-16T14:52:10.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0bf5b59640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:10.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0bf5b59640 1 -- 172.21.15.26:0/3060735786 >> v1:172.21.15.26:6790/0 conn(0x7f0bf009c860 legacy=0x7f0bf01d60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0beffff640 1 -- 172.21.15.26:0/3060735786 reap_dead start 2024-09-16T14:52:10.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0beffff640 1 -- 172.21.15.26:0/3060735786 reap_dead start 2024-09-16T14:52:10.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0bf5b59640 1 -- 172.21.15.26:0/3060735786 shutdown_connections 2024-09-16T14:52:10.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0bf5b59640 1 -- 172.21.15.26:0/3060735786 >> 172.21.15.26:0/3060735786 conn(0x7f0bf009fb50 msgr2=0x7f0bf009f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:10.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0bf5b59640 1 -- 172.21.15.26:0/3060735786 shutdown_connections 2024-09-16T14:52:10.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.296+0000 7f0bf5b59640 1 -- 172.21.15.26:0/3060735786 wait complete. 2024-09-16T14:52:10.332 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:52:10.332 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:52:10.332 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:52:10.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 10 monclient: get_monmap_and_config 2024-09-16T14:52:10.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 10 monclient: build_initial_monmap 2024-09-16T14:52:10.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 10 monclient: monmap: 2024-09-16T14:52:10.446 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:10.446 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.446 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:10.447094+0000 2024-09-16T14:52:10.446 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:10.447094+0000 2024-09-16T14:52:10.446 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:10.446 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.446 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:10.446 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:10.446 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:10.446 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 1 Processor -- start 2024-09-16T14:52:10.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 1 -- start start 2024-09-16T14:52:10.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 10 monclient: init 2024-09-16T14:52:10.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:10.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:52:10.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 10 monclient(hunting): picked mon.noname-b con 0x7f26c00a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:10.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 10 monclient(hunting): picked mon.noname-c con 0x7f26c00a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:10.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 10 monclient(hunting): picked mon.noname-a con 0x7f26c00a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:10.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c00a3250 con 0x7f26c00a2800 2024-09-16T14:52:10.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c009f600 con 0x7f26c00a0000 2024-09-16T14:52:10.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c00a3830 con 0x7f26c00a1d10 2024-09-16T14:52:10.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:10.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 10 monclient(hunting): authenticate will time out at 1554.566882s 2024-09-16T14:52:10.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26bf7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f26c00a1d10 0x7f26c00a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46972/0 (socket says 172.21.15.26:46972) 2024-09-16T14:52:10.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26bffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f26c00a2800 0x7f26c013d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58450/0 (socket says 172.21.15.26:58450) 2024-09-16T14:52:10.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26bf7fe640 1 -- 172.21.15.26:0/3318401704 learned_addr learned my addr 172.21.15.26:0/3318401704 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:10.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 1 -- 172.21.15.26:0/3318401704 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521157823 0 0) 0x7f26c00a3830 con 0x7f26c00a1d10 2024-09-16T14:52:10.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient(hunting): my global_id is 214160 2024-09-16T14:52:10.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 1 -- 172.21.15.26:0/3318401704 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26ac003530 con 0x7f26c00a1d10 2024-09-16T14:52:10.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 1 -- 172.21.15.26:0/3318401704 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324882466 0 0) 0x7f26c00a3250 con 0x7f26c00a2800 2024-09-16T14:52:10.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient(hunting): my global_id is 194301 2024-09-16T14:52:10.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 1 -- 172.21.15.26:0/3318401704 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c00a3830 con 0x7f26c00a2800 2024-09-16T14:52:10.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 1 -- 172.21.15.26:0/3318401704 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1674377442 0 0) 0x7f26ac003530 con 0x7f26c00a1d10 2024-09-16T14:52:10.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 1 -- 172.21.15.26:0/3318401704 >> v1:172.21.15.189:6789/0 conn(0x7f26c00a0000 legacy=0x7f26c0139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 1 -- 172.21.15.26:0/3318401704 >> v1:172.21.15.26:6789/0 conn(0x7f26c00a2800 legacy=0x7f26c013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:52:10.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 1 monclient: found mon.noname-b 2024-09-16T14:52:10.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:10.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:52:10.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 1 -- 172.21.15.26:0/3318401704 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c009ed60 con 0x7f26c00a1d10 2024-09-16T14:52:10.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient: _finish_auth 0 2024-09-16T14:52:10.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:52:10.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:10.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 1 -- 172.21.15.26:0/3318401704 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f26b00019a0 con 0x7f26c00a1d10 2024-09-16T14:52:10.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 5 monclient: authenticate success, global_id 214160 2024-09-16T14:52:10.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:10.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:52:10.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient: dump: 2024-09-16T14:52:10.453 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.453 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.453 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.453 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.453 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.453 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.453 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.453 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.453 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.453 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:10.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 1 -- 172.21.15.26:0/3318401704 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26b0001c40 con 0x7f26c00a1d10 2024-09-16T14:52:10.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:10.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 1 -- 172.21.15.26:0/3318401704 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f26b0002010 con 0x7f26c00a1d10 2024-09-16T14:52:10.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:10.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26be7fc640 10 monclient: dump: 2024-09-16T14:52:10.454 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.454 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.454 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.454 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.454 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.455 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.455 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.455 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.455 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.455 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:10.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.444+0000 7f26c5f7d640 4 set_mon_vals no callback set 2024-09-16T14:52:10.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 10 monclient: shutdown 2024-09-16T14:52:10.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:10.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 1 -- 172.21.15.26:0/3318401704 >> v1:172.21.15.26:6790/0 conn(0x7f26c00a1d10 legacy=0x7f26c00a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 1 -- 172.21.15.26:0/3318401704 shutdown_connections 2024-09-16T14:52:10.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 1 -- 172.21.15.26:0/3318401704 >> 172.21.15.26:0/3318401704 conn(0x7f26c00a7f30 msgr2=0x7f26c00a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:10.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 1 -- 172.21.15.26:0/3318401704 shutdown_connections 2024-09-16T14:52:10.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 1 -- 172.21.15.26:0/3318401704 wait complete. 2024-09-16T14:52:10.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 10 monclient: build_initial_monmap 2024-09-16T14:52:10.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 10 monclient: monmap: 2024-09-16T14:52:10.456 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:10.456 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:10.456 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:10.456 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:10.456 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:10.457 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:10.457 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:10.457 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:10.457 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:10.457 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 1 Processor -- start 2024-09-16T14:52:10.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 1 -- start start 2024-09-16T14:52:10.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 10 monclient: init 2024-09-16T14:52:10.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:10.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:52:10.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 10 monclient(hunting): picked mon.noname-b con 0x7f26c00a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:10.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 10 monclient(hunting): picked mon.noname-a con 0x7f26c00a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:10.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 10 monclient(hunting): picked mon.noname-c con 0x7f26c00a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:10.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c009e500 con 0x7f26c00a1d10 2024-09-16T14:52:10.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c01364e0 con 0x7f26c00a0000 2024-09-16T14:52:10.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c0132dc0 con 0x7f26c00a2800 2024-09-16T14:52:10.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:10.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 10 monclient(hunting): authenticate will time out at 1554.570026s 2024-09-16T14:52:10.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26beffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f26c00a1d10 0x7f26c01326b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58454/0 (socket says 172.21.15.26:58454) 2024-09-16T14:52:10.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26beffd640 1 -- 172.21.15.26:0/407668319 learned_addr learned my addr 172.21.15.26:0/407668319 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:10.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26bffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f26c00a2800 0x7f26c01d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46976/0 (socket says 172.21.15.26:46976) 2024-09-16T14:52:10.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933821847 0 0) 0x7f26c009e500 con 0x7f26c00a1d10 2024-09-16T14:52:10.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): my global_id is 194304 2024-09-16T14:52:10.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26a0003530 con 0x7f26c00a1d10 2024-09-16T14:52:10.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228081893 0 0) 0x7f26c0132dc0 con 0x7f26c00a2800 2024-09-16T14:52:10.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): my global_id is 214163 2024-09-16T14:52:10.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c009e500 con 0x7f26c00a2800 2024-09-16T14:52:10.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2237272637 0 0) 0x7f26a0003530 con 0x7f26c00a1d10 2024-09-16T14:52:10.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26c0132dc0 con 0x7f26c00a1d10 2024-09-16T14:52:10.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 359100888 0 0) 0x7f26c009e500 con 0x7f26c00a2800 2024-09-16T14:52:10.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26a0003530 con 0x7f26c00a2800 2024-09-16T14:52:10.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532918860 0 0) 0x7f26c01364e0 con 0x7f26c00a0000 2024-09-16T14:52:10.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): my global_id is 184483 2024-09-16T14:52:10.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c009e500 con 0x7f26c00a0000 2024-09-16T14:52:10.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f26a8001940 con 0x7f26c00a1d10 2024-09-16T14:52:10.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:10.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): dump: 2024-09-16T14:52:10.465 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.465 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.465 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.465 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.465 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.465 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.465 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.465 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.465 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.465 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:10.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f26b8001100 con 0x7f26c00a2800 2024-09-16T14:52:10.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:10.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient(hunting): dump: 2024-09-16T14:52:10.466 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.466 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.466 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.466 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.466 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.466 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.466 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.466 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.466 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.466 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1322993550 0 0) 0x7f26a0003530 con 0x7f26c00a2800 2024-09-16T14:52:10.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 >> v1:172.21.15.189:6789/0 conn(0x7f26c00a0000 legacy=0x7f26c0131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 >> v1:172.21.15.26:6789/0 conn(0x7f26c00a1d10 legacy=0x7f26c01326b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:52:10.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 monclient: found mon.c 2024-09-16T14:52:10.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:10.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:10.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c00a6520 con 0x7f26c00a2800 2024-09-16T14:52:10.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient: _finish_auth 0 2024-09-16T14:52:10.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:52:10.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:10.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 5 monclient: authenticate success, global_id 214163 2024-09-16T14:52:10.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 10 monclient: _renew_subs 2024-09-16T14:52:10.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:10.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 1 -- 172.21.15.26:0/407668319 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c0134910 con 0x7f26c00a2800 2024-09-16T14:52:10.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 10 monclient: _renew_subs 2024-09-16T14:52:10.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:10.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 1 -- 172.21.15.26:0/407668319 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c0133d20 con 0x7f26c00a2800 2024-09-16T14:52:10.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26b80047f0 con 0x7f26c00a2800 2024-09-16T14:52:10.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:10.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f26b8004b00 con 0x7f26c00a2800 2024-09-16T14:52:10.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:10.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 10 monclient: dump: 2024-09-16T14:52:10.468 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.468 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.468 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.468 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.468 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.469 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.469 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.469 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.469 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.469 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26bdffb640 4 set_mon_vals no callback set 2024-09-16T14:52:10.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:10.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26c5f7d640 1 -- 172.21.15.26:0/407668319 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f26a4006590 con 0x7f26a4002830 2024-09-16T14:52:10.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.448+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f26b800f570 con 0x7f26c00a2800 2024-09-16T14:52:10.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f26b8043cb0 con 0x7f26c00a2800 2024-09-16T14:52:10.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26beffd640 10 monclient: get_auth_request con 0x7f26a0036420 auth_method 0 2024-09-16T14:52:10.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279569214 0 0) 0x7f26a4006590 con 0x7f26a4002830 2024-09-16T14:52:10.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 10 monclient(hunting): my global_id is 184486 2024-09-16T14:52:10.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26a0003530 con 0x7f26a4002830 2024-09-16T14:52:10.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 17770951 0 0) 0x7f26a0003530 con 0x7f26a4002830 2024-09-16T14:52:10.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26a4006590 con 0x7f26a4002830 2024-09-16T14:52:10.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f26b0001960 con 0x7f26a4002830 2024-09-16T14:52:10.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:52:10.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 10 monclient: dump: 2024-09-16T14:52:10.470 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.470 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.470 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.471 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.471 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.471 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.471 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.471 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.471 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.471 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1742212826 0 0) 0x7f26a4006590 con 0x7f26a4002830 2024-09-16T14:52:10.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26a400a0e0 con 0x7f26a4002830 2024-09-16T14:52:10.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f26b0001eb0 con 0x7f26a4002830 2024-09-16T14:52:10.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:10.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:10.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 >> v1:172.21.15.189:6789/0 conn(0x7f26a4002830 legacy=0x7f26a4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.452+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 >> v1:172.21.15.189:6789/0 conn(0x7f26a4002830 legacy=0x7f26a4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:10.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26c5f7d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:52:10.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26c5f7d640 1 -- 172.21.15.26:0/407668319 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f26a4009da0 con 0x7f26a40090c0 2024-09-16T14:52:10.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871686237 0 0) 0x7f26a4009da0 con 0x7f26a40090c0 2024-09-16T14:52:10.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 10 monclient(hunting): my global_id is 184489 2024-09-16T14:52:10.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26a4006590 con 0x7f26a40090c0 2024-09-16T14:52:10.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 527033090 0 0) 0x7f26a4006590 con 0x7f26a40090c0 2024-09-16T14:52:10.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26a4009da0 con 0x7f26a40090c0 2024-09-16T14:52:10.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f26b0005390 con 0x7f26a40090c0 2024-09-16T14:52:10.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:52:10.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 10 monclient: dump: 2024-09-16T14:52:10.473 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.473 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.473 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.473 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.473 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.473 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.473 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.473 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.473 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.473 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4211174700 0 0) 0x7f26a4009da0 con 0x7f26a40090c0 2024-09-16T14:52:10.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f26a400c290 con 0x7f26a40090c0 2024-09-16T14:52:10.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 2250377668) 0x7f26b0007550 con 0x7f26a40090c0 2024-09-16T14:52:10.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:52:10.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:10.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 >> v1:172.21.15.189:6789/0 conn(0x7f26a40090c0 legacy=0x7f26a4009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26b7fff640 1 -- 172.21.15.26:0/407668319 >> v1:172.21.15.189:6789/0 conn(0x7f26a40090c0 legacy=0x7f26a4009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:10.475 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:52:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:52:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 90, 2024-09-16T14:52:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:52:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:52:10.475 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:52:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:52:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-09-16T14:52:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:52:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:52:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:10.476 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:52:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:52:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:10.477 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:52:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:52:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:52:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:52:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:52:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:52:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:10.478 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:52:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:52:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:52:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:52:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:52:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:52:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:52:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:52:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:10.479 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:52:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:52:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.480 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:52:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:52:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:52:10.481 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:52:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.482 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:52:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:52:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:10.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:52:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:52:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:52:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.484 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:52:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:52:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:52:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:10.485 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:52:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:10.486 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:52:10.486 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:52:10.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26c5f7d640 1 -- 172.21.15.26:0/407668319 >> v1:172.21.15.189:6812/3196161582 conn(0x7f26a0036420 legacy=0x7f26a00388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26c5f7d640 10 monclient: shutdown 2024-09-16T14:52:10.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26c5f7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:10.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26c5f7d640 1 -- 172.21.15.26:0/407668319 >> v1:172.21.15.26:6790/0 conn(0x7f26c00a2800 legacy=0x7f26c01d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26bffff640 1 -- 172.21.15.26:0/407668319 reap_dead start 2024-09-16T14:52:10.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26bffff640 1 -- 172.21.15.26:0/407668319 reap_dead start 2024-09-16T14:52:10.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26c5f7d640 1 -- 172.21.15.26:0/407668319 shutdown_connections 2024-09-16T14:52:10.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26c5f7d640 1 -- 172.21.15.26:0/407668319 >> 172.21.15.26:0/407668319 conn(0x7f26c00a7f30 msgr2=0x7f26c006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:10.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26c5f7d640 1 -- 172.21.15.26:0/407668319 shutdown_connections 2024-09-16T14:52:10.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.460+0000 7f26c5f7d640 1 -- 172.21.15.26:0/407668319 wait complete. 2024-09-16T14:52:10.488 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:52:10.488 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T14:52:10.488 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:52:10.488 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T14:52:10.489 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:52:10.489 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T14:52:10.489 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:52:10.515 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T14:52:10.515 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:52:10.515 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:52:10.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 10 monclient: get_monmap_and_config 2024-09-16T14:52:10.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 10 monclient: build_initial_monmap 2024-09-16T14:52:10.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 10 monclient: monmap: 2024-09-16T14:52:10.634 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:10.634 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.634 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:10.634949+0000 2024-09-16T14:52:10.634 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:10.634949+0000 2024-09-16T14:52:10.634 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:10.634 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.634 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:10.634 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:10.635 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:10.635 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 1 Processor -- start 2024-09-16T14:52:10.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 1 -- start start 2024-09-16T14:52:10.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 10 monclient: init 2024-09-16T14:52:10.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:10.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:52:10.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 10 monclient(hunting): picked mon.noname-c con 0x7f24e809a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:10.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 10 monclient(hunting): picked mon.noname-b con 0x7f24e809d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:10.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 10 monclient(hunting): picked mon.noname-a con 0x7f24e809b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:10.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e809dc60 con 0x7f24e809b040 2024-09-16T14:52:10.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e8099ae0 con 0x7f24e809a550 2024-09-16T14:52:10.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e809b8f0 con 0x7f24e809d1b0 2024-09-16T14:52:10.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:10.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 10 monclient(hunting): authenticate will time out at 1554.754983s 2024-09-16T14:52:10.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e7fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f24e809d1b0 0x7f24e8136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46986/0 (socket says 172.21.15.26:46986) 2024-09-16T14:52:10.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e7fff640 1 -- 172.21.15.26:0/3054842580 learned_addr learned my addr 172.21.15.26:0/3054842580 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:10.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24ed169640 1 --1- 172.21.15.26:0/3054842580 >> v1:172.21.15.26:6789/0 conn(0x7f24e809b040 0x7f24e813a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58470/0 (socket says 172.21.15.26:58470) 2024-09-16T14:52:10.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24ec968640 1 -- 172.21.15.26:0/3054842580 >> v1:172.21.15.189:6789/0 conn(0x7f24e809a550 legacy=0x7f24e809a930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:10.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 1 -- 172.21.15.26:0/3054842580 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518704314 0 0) 0x7f24e809b8f0 con 0x7f24e809d1b0 2024-09-16T14:52:10.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 10 monclient(hunting): my global_id is 214166 2024-09-16T14:52:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 1 -- 172.21.15.26:0/3054842580 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d0003530 con 0x7f24e809d1b0 2024-09-16T14:52:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 1 -- 172.21.15.26:0/3054842580 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791525279 0 0) 0x7f24e809dc60 con 0x7f24e809b040 2024-09-16T14:52:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 10 monclient(hunting): my global_id is 194307 2024-09-16T14:52:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 1 -- 172.21.15.26:0/3054842580 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24e809b8f0 con 0x7f24e809b040 2024-09-16T14:52:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 1 -- 172.21.15.26:0/3054842580 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1762405613 0 0) 0x7f24d0003530 con 0x7f24e809d1b0 2024-09-16T14:52:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 1 -- 172.21.15.26:0/3054842580 >> v1:172.21.15.189:6789/0 conn(0x7f24e809a550 legacy=0x7f24e809a930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:52:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 1 -- 172.21.15.26:0/3054842580 >> v1:172.21.15.26:6789/0 conn(0x7f24e809b040 legacy=0x7f24e813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:52:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 1 monclient: found mon.noname-b 2024-09-16T14:52:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:52:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 1 -- 172.21.15.26:0/3054842580 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e809bbd0 con 0x7f24e809d1b0 2024-09-16T14:52:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 10 monclient: _finish_auth 0 2024-09-16T14:52:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:52:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 1 -- 172.21.15.26:0/3054842580 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24dc001620 con 0x7f24e809d1b0 2024-09-16T14:52:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 5 monclient: authenticate success, global_id 214166 2024-09-16T14:52:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:52:10.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 10 monclient: dump: 2024-09-16T14:52:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.647 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24eebcc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:10.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.632+0000 7f24e77fe640 1 -- 172.21.15.26:0/3054842580 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24dc001900 con 0x7f24e809d1b0 2024-09-16T14:52:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e77fe640 1 -- 172.21.15.26:0/3054842580 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24dc001cd0 con 0x7f24e809d1b0 2024-09-16T14:52:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e77fe640 10 monclient: dump: 2024-09-16T14:52:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.648 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.649 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.649 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.649 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:10.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 4 set_mon_vals no callback set 2024-09-16T14:52:10.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 10 monclient: shutdown 2024-09-16T14:52:10.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:10.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 1 -- 172.21.15.26:0/3054842580 >> v1:172.21.15.26:6790/0 conn(0x7f24e809d1b0 legacy=0x7f24e8136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 1 -- 172.21.15.26:0/3054842580 shutdown_connections 2024-09-16T14:52:10.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 1 -- 172.21.15.26:0/3054842580 >> 172.21.15.26:0/3054842580 conn(0x7f24e809fb50 msgr2=0x7f24e809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:10.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 1 -- 172.21.15.26:0/3054842580 shutdown_connections 2024-09-16T14:52:10.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 1 -- 172.21.15.26:0/3054842580 wait complete. 2024-09-16T14:52:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 10 monclient: build_initial_monmap 2024-09-16T14:52:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 10 monclient: monmap: 2024-09-16T14:52:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:10.650 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:10.650 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 1 Processor -- start 2024-09-16T14:52:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 1 -- start start 2024-09-16T14:52:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 10 monclient: init 2024-09-16T14:52:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:52:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 10 monclient(hunting): picked mon.noname-c con 0x7f24e809a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 10 monclient(hunting): picked mon.noname-a con 0x7f24e809b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 10 monclient(hunting): picked mon.noname-b con 0x7f24e809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e81a2e60 con 0x7f24e809b040 2024-09-16T14:52:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e81a47c0 con 0x7f24e809d1b0 2024-09-16T14:52:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e80a2620 con 0x7f24e809a550 2024-09-16T14:52:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 10 monclient(hunting): authenticate will time out at 1554.758439s 2024-09-16T14:52:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24ec968640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f24e809a550 0x7f24e81c4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46994/0 (socket says 172.21.15.26:46994) 2024-09-16T14:52:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24ec968640 1 -- 172.21.15.26:0/3894516679 learned_addr learned my addr 172.21.15.26:0/3894516679 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24ed169640 1 -- 172.21.15.26:0/3894516679 >> v1:172.21.15.189:6789/0 conn(0x7f24e809d1b0 legacy=0x7f24e81cce30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825803024 0 0) 0x7f24e80a2620 con 0x7f24e809a550 2024-09-16T14:52:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient(hunting): my global_id is 214169 2024-09-16T14:52:10.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d4003530 con 0x7f24e809a550 2024-09-16T14:52:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818285536 0 0) 0x7f24e81a2e60 con 0x7f24e809b040 2024-09-16T14:52:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient(hunting): _negotiate 2024-09-16T14:52:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient(hunting): my global_id is 194310 2024-09-16T14:52:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24e80a2620 con 0x7f24e809b040 2024-09-16T14:52:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 309666623 0 0) 0x7f24d4003530 con 0x7f24e809a550 2024-09-16T14:52:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24e81a2e60 con 0x7f24e809a550 2024-09-16T14:52:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24d8004190 con 0x7f24e809a550 2024-09-16T14:52:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:52:10.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient(hunting): dump: 2024-09-16T14:52:10.654 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.654 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.654 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.654 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.654 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.654 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.654 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.654 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.654 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.654 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:10.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2518577998 0 0) 0x7f24e80a2620 con 0x7f24e809b040 2024-09-16T14:52:10.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24d4003530 con 0x7f24e809b040 2024-09-16T14:52:10.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24dc001c10 con 0x7f24e809b040 2024-09-16T14:52:10.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:10.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient(hunting): dump: 2024-09-16T14:52:10.655 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.655 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.655 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.655 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.655 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.655 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.655 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.655 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.655 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.655 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 183925759 0 0) 0x7f24e81a2e60 con 0x7f24e809a550 2024-09-16T14:52:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 >> v1:172.21.15.189:6789/0 conn(0x7f24e809d1b0 legacy=0x7f24e81cce30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:52:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 >> v1:172.21.15.26:6789/0 conn(0x7f24e809b040 legacy=0x7f24e81b5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient: _finish_hunting 0 2024-09-16T14:52:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 monclient: found mon.c 2024-09-16T14:52:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e80a20f0 con 0x7f24e809a550 2024-09-16T14:52:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient: _finish_auth 0 2024-09-16T14:52:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient: _check_auth_tickets 2024-09-16T14:52:10.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 5 monclient: authenticate success, global_id 214169 2024-09-16T14:52:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 10 monclient: _renew_subs 2024-09-16T14:52:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 1 -- 172.21.15.26:0/3894516679 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e81a44e0 con 0x7f24e809a550 2024-09-16T14:52:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 10 monclient: _renew_subs 2024-09-16T14:52:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24eebcc640 1 -- 172.21.15.26:0/3894516679 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e81b6500 con 0x7f24e809a550 2024-09-16T14:52:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24d8004470 con 0x7f24e809a550 2024-09-16T14:52:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24d8004800 con 0x7f24e809a550 2024-09-16T14:52:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e57fa640 10 monclient: dump: 2024-09-16T14:52:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:10.658 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.636+0000 7f24e6ffd640 4 set_mon_vals no callback set 2024-09-16T14:52:10.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.640+0000 7f24eebcc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:10.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.640+0000 7f24eebcc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:10.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.640+0000 7f24eebcc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:10.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.640+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f24d80049e0 con 0x7f24e809a550 2024-09-16T14:52:10.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.640+0000 7f24eebcc640 1 -- 172.21.15.26:0/3894516679 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c8005050 con 0x7f24e809a550 2024-09-16T14:52:10.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.640+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f24d8043ec0 con 0x7f24e809a550 2024-09-16T14:52:10.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.640+0000 7f24e7fff640 10 monclient: get_auth_request con 0x7f24d4036200 auth_method 0 2024-09-16T14:52:10.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.644+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f24d8014800 con 0x7f24e809a550 2024-09-16T14:52:10.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.644+0000 7f24e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:10.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.644+0000 7f24e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:10.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.864+0000 7f24eebcc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:52:10.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.864+0000 7f24eebcc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:10.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.864+0000 7f24eebcc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:10.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.864+0000 7f24eebcc640 1 -- 172.21.15.26:0/3894516679 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f24c8005880 con 0x7f24e809a550 2024-09-16T14:52:10.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.864+0000 7f24e57fa640 1 -- 172.21.15.26:0/3894516679 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2436352960) 0x7f24d8012c70 con 0x7f24e809a550 2024-09-16T14:52:10.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.864+0000 7f24e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:10.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.864+0000 7f24e57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:10.867 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":90,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:52:10.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.868+0000 7f24eebcc640 1 -- 172.21.15.26:0/3894516679 >> v1:172.21.15.189:6812/3196161582 conn(0x7f24d4036200 legacy=0x7f24d40386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.868+0000 7f24eebcc640 10 monclient: shutdown 2024-09-16T14:52:10.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.868+0000 7f24eebcc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:10.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.868+0000 7f24eebcc640 1 -- 172.21.15.26:0/3894516679 >> v1:172.21.15.26:6790/0 conn(0x7f24e809a550 legacy=0x7f24e81c4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:10.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.868+0000 7f24eebcc640 1 -- 172.21.15.26:0/3894516679 shutdown_connections 2024-09-16T14:52:10.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.868+0000 7f24eebcc640 1 -- 172.21.15.26:0/3894516679 >> 172.21.15.26:0/3894516679 conn(0x7f24e809fb50 msgr2=0x7f24e80a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:10.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.868+0000 7f24eebcc640 1 -- 172.21.15.26:0/3894516679 shutdown_connections 2024-09-16T14:52:10.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:10.868+0000 7f24eebcc640 1 -- 172.21.15.26:0/3894516679 wait complete. 2024-09-16T14:52:13.886 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:52:14.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e9ba2a640 10 monclient: get_monmap_and_config 2024-09-16T14:52:14.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e9ba2a640 10 monclient: build_initial_monmap 2024-09-16T14:52:14.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e9ba2a640 10 monclient: monmap: 2024-09-16T14:52:14.018 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:14.018 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:14.018 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:14.019205+0000 2024-09-16T14:52:14.018 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:14.019205+0000 2024-09-16T14:52:14.018 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:14.018 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:14.018 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:14.018 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:14.018 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:14.019 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:14.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e9ba2a640 1 Processor -- start 2024-09-16T14:52:14.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e9ba2a640 1 -- start start 2024-09-16T14:52:14.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e9ba2a640 10 monclient: init 2024-09-16T14:52:14.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e9ba2a640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:14.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e9ba2a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:52:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e9ba2a640 10 monclient(hunting): picked mon.noname-c con 0x7f3e940a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e9ba2a640 10 monclient(hunting): picked mon.noname-a con 0x7f3e940a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e9ba2a640 10 monclient(hunting): picked mon.noname-b con 0x7f3e940a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e9ba2a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e940a3250 con 0x7f3e940a0000 2024-09-16T14:52:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e9ba2a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9409f600 con 0x7f3e940a1d10 2024-09-16T14:52:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e9ba2a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e940a3830 con 0x7f3e940a2800 2024-09-16T14:52:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e9ba2a640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e9ba2a640 10 monclient(hunting): authenticate will time out at 1558.139323s 2024-09-16T14:52:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e99fc7640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3e940a2800 0x7f3e9413d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47010/0 (socket says 172.21.15.26:47010) 2024-09-16T14:52:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e99fc7640 1 -- 172.21.15.26:0/4136062571 learned_addr learned my addr 172.21.15.26:0/4136062571 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e98fc5640 1 --1- 172.21.15.26:0/4136062571 >> v1:172.21.15.26:6789/0 conn(0x7f3e940a0000 0x7f3e94139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58482/0 (socket says 172.21.15.26:58482) 2024-09-16T14:52:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e997c6640 1 -- 172.21.15.26:0/4136062571 >> v1:172.21.15.189:6789/0 conn(0x7f3e940a1d10 legacy=0x7f3e940a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e93fff640 1 -- 172.21.15.26:0/4136062571 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258866922 0 0) 0x7f3e940a3830 con 0x7f3e940a2800 2024-09-16T14:52:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e93fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e93fff640 10 monclient(hunting): my global_id is 214172 2024-09-16T14:52:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.016+0000 7f3e93fff640 1 -- 172.21.15.26:0/4136062571 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e7c003530 con 0x7f3e940a2800 2024-09-16T14:52:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 1 -- 172.21.15.26:0/4136062571 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3227218309 0 0) 0x7f3e7c003530 con 0x7f3e940a2800 2024-09-16T14:52:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 1 -- 172.21.15.26:0/4136062571 >> v1:172.21.15.189:6789/0 conn(0x7f3e940a1d10 legacy=0x7f3e940a20f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:52:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 1 -- 172.21.15.26:0/4136062571 >> v1:172.21.15.26:6789/0 conn(0x7f3e940a0000 legacy=0x7f3e94139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 10 monclient: _finish_hunting 0 2024-09-16T14:52:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 1 monclient: found mon.noname-b 2024-09-16T14:52:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:14.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:52:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 1 -- 172.21.15.26:0/4136062571 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9409ed60 con 0x7f3e940a2800 2024-09-16T14:52:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 10 monclient: _finish_auth 0 2024-09-16T14:52:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 10 monclient: _check_auth_tickets 2024-09-16T14:52:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 1 -- 172.21.15.26:0/4136062571 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e84001620 con 0x7f3e940a2800 2024-09-16T14:52:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:52:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 10 monclient: dump: 2024-09-16T14:52:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:14.025 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:14.026 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:14.026 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:14.026 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:14.026 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:14.026 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:14.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 5 monclient: authenticate success, global_id 214172 2024-09-16T14:52:14.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:14.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 1 -- 172.21.15.26:0/4136062571 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e84001940 con 0x7f3e940a2800 2024-09-16T14:52:14.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:14.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 1 -- 172.21.15.26:0/4136062571 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e84001cd0 con 0x7f3e940a2800 2024-09-16T14:52:14.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:14.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:14.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e93fff640 10 monclient: dump: 2024-09-16T14:52:14.027 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:14.027 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:14.027 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:14.027 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:14.027 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:14.027 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:14.027 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:14.027 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:14.027 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:14.027 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:14.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:14.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 4 set_mon_vals no callback set 2024-09-16T14:52:14.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 10 monclient: shutdown 2024-09-16T14:52:14.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:14.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 1 -- 172.21.15.26:0/4136062571 >> v1:172.21.15.26:6790/0 conn(0x7f3e940a2800 legacy=0x7f3e9413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:14.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 1 -- 172.21.15.26:0/4136062571 shutdown_connections 2024-09-16T14:52:14.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 1 -- 172.21.15.26:0/4136062571 >> 172.21.15.26:0/4136062571 conn(0x7f3e940a7f30 msgr2=0x7f3e940a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:14.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 1 -- 172.21.15.26:0/4136062571 shutdown_connections 2024-09-16T14:52:14.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 1 -- 172.21.15.26:0/4136062571 wait complete. 2024-09-16T14:52:14.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 10 monclient: build_initial_monmap 2024-09-16T14:52:14.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 10 monclient: monmap: 2024-09-16T14:52:14.028 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:14.028 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:14.028 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:14.028 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:14.029 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:14.029 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:14.029 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:14.029 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:14.029 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:14.029 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:14.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 1 Processor -- start 2024-09-16T14:52:14.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 1 -- start start 2024-09-16T14:52:14.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 10 monclient: init 2024-09-16T14:52:14.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:14.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:52:14.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 10 monclient(hunting): picked mon.noname-a con 0x7f3e940a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:14.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 10 monclient(hunting): picked mon.noname-b con 0x7f3e940a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:14.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 10 monclient(hunting): picked mon.noname-c con 0x7f3e940a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:14.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e94131380 con 0x7f3e940a0000 2024-09-16T14:52:14.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e94130d40 con 0x7f3e940a1d10 2024-09-16T14:52:14.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9409e500 con 0x7f3e940a2800 2024-09-16T14:52:14.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:14.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 10 monclient(hunting): authenticate will time out at 1558.142556s 2024-09-16T14:52:14.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e99fc7640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3e940a2800 0x7f3e941d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47026/0 (socket says 172.21.15.26:47026) 2024-09-16T14:52:14.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e99fc7640 1 -- 172.21.15.26:0/3121165870 learned_addr learned my addr 172.21.15.26:0/3121165870 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:14.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e997c6640 1 --1- 172.21.15.26:0/3121165870 >> v1:172.21.15.26:6789/0 conn(0x7f3e940a0000 0x7f3e9412ff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58484/0 (socket says 172.21.15.26:58484) 2024-09-16T14:52:14.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e98fc5640 1 -- 172.21.15.26:0/3121165870 >> v1:172.21.15.189:6789/0 conn(0x7f3e940a1d10 legacy=0x7f3e94130630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:14.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 1 -- 172.21.15.26:0/3121165870 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061855357 0 0) 0x7f3e9409e500 con 0x7f3e940a2800 2024-09-16T14:52:14.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:52:14.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:14.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:14.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 10 monclient(hunting): my global_id is 214175 2024-09-16T14:52:14.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 1 -- 172.21.15.26:0/3121165870 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e80003530 con 0x7f3e940a2800 2024-09-16T14:52:14.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 1 -- 172.21.15.26:0/3121165870 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4207100057 0 0) 0x7f3e80003530 con 0x7f3e940a2800 2024-09-16T14:52:14.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 1 -- 172.21.15.26:0/3121165870 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e9409e500 con 0x7f3e940a2800 2024-09-16T14:52:14.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 1 -- 172.21.15.26:0/3121165870 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e840015e0 con 0x7f3e940a2800 2024-09-16T14:52:14.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:14.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:52:14.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 10 monclient(hunting): dump: 2024-09-16T14:52:14.033 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:14.034 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:14.034 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:14.034 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:14.034 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:14.034 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:14.034 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:14.034 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:14.034 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:14.034 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:14.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:14.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 1 -- 172.21.15.26:0/3121165870 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2930454003 0 0) 0x7f3e9409e500 con 0x7f3e940a2800 2024-09-16T14:52:14.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 1 -- 172.21.15.26:0/3121165870 >> v1:172.21.15.189:6789/0 conn(0x7f3e940a1d10 legacy=0x7f3e94130630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:52:14.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 1 -- 172.21.15.26:0/3121165870 >> v1:172.21.15.26:6789/0 conn(0x7f3e940a0000 legacy=0x7f3e9412ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:14.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:52:14.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 1 monclient: found mon.c 2024-09-16T14:52:14.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:14.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:14.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 1 -- 172.21.15.26:0/3121165870 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e940a6580 con 0x7f3e940a2800 2024-09-16T14:52:14.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 10 monclient: _finish_auth 0 2024-09-16T14:52:14.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:52:14.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:14.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 5 monclient: authenticate success, global_id 214175 2024-09-16T14:52:14.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 10 monclient: _renew_subs 2024-09-16T14:52:14.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:14.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 1 -- 172.21.15.26:0/3121165870 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e94131aa0 con 0x7f3e940a2800 2024-09-16T14:52:14.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 10 monclient: _renew_subs 2024-09-16T14:52:14.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:14.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e9ba2a640 1 -- 172.21.15.26:0/3121165870 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e94133e00 con 0x7f3e940a2800 2024-09-16T14:52:14.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 1 -- 172.21.15.26:0/3121165870 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e84004c00 con 0x7f3e940a2800 2024-09-16T14:52:14.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:14.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 1 -- 172.21.15.26:0/3121165870 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3e84004f90 con 0x7f3e940a2800 2024-09-16T14:52:14.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:14.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:14.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e937fe640 4 set_mon_vals no callback set 2024-09-16T14:52:14.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.020+0000 7f3e91ffb640 10 monclient: dump: 2024-09-16T14:52:14.036 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:14.036 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:14.036 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:14.036 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:14.037 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:14.037 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:14.037 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:14.037 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:14.037 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:14.037 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:14.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.024+0000 7f3e91ffb640 1 -- 172.21.15.26:0/3121165870 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3e84005170 con 0x7f3e940a2800 2024-09-16T14:52:14.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.024+0000 7f3e9ba2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:14.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.024+0000 7f3e9ba2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:14.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.024+0000 7f3e9ba2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:14.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.024+0000 7f3e9ba2a640 1 -- 172.21.15.26:0/3121165870 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e74005050 con 0x7f3e940a2800 2024-09-16T14:52:14.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.024+0000 7f3e91ffb640 1 -- 172.21.15.26:0/3121165870 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f3e840440b0 con 0x7f3e940a2800 2024-09-16T14:52:14.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.024+0000 7f3e997c6640 10 monclient: get_auth_request con 0x7f3e80036050 auth_method 0 2024-09-16T14:52:14.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.028+0000 7f3e91ffb640 1 -- 172.21.15.26:0/3121165870 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3e84014a50 con 0x7f3e940a2800 2024-09-16T14:52:14.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.028+0000 7f3e91ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:14.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.028+0000 7f3e91ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:14.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.256+0000 7f3e9ba2a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:52:14.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.256+0000 7f3e9ba2a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:14.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.256+0000 7f3e9ba2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:14.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.256+0000 7f3e9ba2a640 1 -- 172.21.15.26:0/3121165870 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3e74005880 con 0x7f3e940a2800 2024-09-16T14:52:14.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.256+0000 7f3e91ffb640 1 -- 172.21.15.26:0/3121165870 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3252356577) 0x7f3e84012ec0 con 0x7f3e940a2800 2024-09-16T14:52:14.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.256+0000 7f3e91ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:14.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.256+0000 7f3e91ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:14.258 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":90,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:52:14.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.256+0000 7f3e9ba2a640 1 -- 172.21.15.26:0/3121165870 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3e80036050 legacy=0x7f3e80038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:14.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.256+0000 7f3e9ba2a640 10 monclient: shutdown 2024-09-16T14:52:14.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.256+0000 7f3e9ba2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:14.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.256+0000 7f3e9ba2a640 1 -- 172.21.15.26:0/3121165870 >> v1:172.21.15.26:6790/0 conn(0x7f3e940a2800 legacy=0x7f3e941d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:14.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.256+0000 7f3e9ba2a640 1 -- 172.21.15.26:0/3121165870 shutdown_connections 2024-09-16T14:52:14.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.256+0000 7f3e9ba2a640 1 -- 172.21.15.26:0/3121165870 >> 172.21.15.26:0/3121165870 conn(0x7f3e940a7f30 msgr2=0x7f3e9406b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:14.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.256+0000 7f3e9ba2a640 1 -- 172.21.15.26:0/3121165870 shutdown_connections 2024-09-16T14:52:14.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:14.260+0000 7f3e9ba2a640 1 -- 172.21.15.26:0/3121165870 wait complete. 2024-09-16T14:52:17.277 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:52:17.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.408+0000 7f8eeef15640 10 monclient: get_monmap_and_config 2024-09-16T14:52:17.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.408+0000 7f8eeef15640 10 monclient: build_initial_monmap 2024-09-16T14:52:17.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.408+0000 7f8eeef15640 10 monclient: monmap: 2024-09-16T14:52:17.411 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:17.411 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:17.411 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:17.412309+0000 2024-09-16T14:52:17.411 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:17.412309+0000 2024-09-16T14:52:17.411 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:17.412 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:17.412 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:17.412 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:17.412 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:17.412 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:17.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.408+0000 7f8eeef15640 1 Processor -- start 2024-09-16T14:52:17.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.408+0000 7f8eeef15640 1 -- start start 2024-09-16T14:52:17.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.408+0000 7f8eeef15640 10 monclient: init 2024-09-16T14:52:17.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.412+0000 7f8eeef15640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:17.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.412+0000 7f8eeef15640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:52:17.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.412+0000 7f8eeef15640 10 monclient(hunting): picked mon.noname-c con 0x7f8ee809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:17.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.412+0000 7f8eeef15640 10 monclient(hunting): picked mon.noname-b con 0x7f8ee8132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:17.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.412+0000 7f8eeef15640 10 monclient(hunting): picked mon.noname-a con 0x7f8ee8138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:17.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.412+0000 7f8eeef15640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee809d260 con 0x7f8ee8138ff0 2024-09-16T14:52:17.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.412+0000 7f8eeef15640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee8132a80 con 0x7f8ee809cb80 2024-09-16T14:52:17.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.412+0000 7f8eeef15640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee809cf60 con 0x7f8ee8132650 2024-09-16T14:52:17.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.412+0000 7f8eeef15640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:17.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.412+0000 7f8eeef15640 10 monclient(hunting): authenticate will time out at 1561.532538s 2024-09-16T14:52:17.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.412+0000 7f8eeccb1640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f8ee809cb80 legacy=0x7f8ee80717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:17.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.412+0000 7f8eed4b2640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8ee8138ff0 0x7f8ee813b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:49800/0 (socket says 172.21.15.26:49800) 2024-09-16T14:52:17.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.412+0000 7f8ee7fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8ee8132650 0x7f8ee81378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:37732/0 (socket says 172.21.15.26:37732) 2024-09-16T14:52:17.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.412+0000 7f8eed4b2640 1 -- 172.21.15.26:0/2047227077 learned_addr learned my addr 172.21.15.26:0/2047227077 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:17.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:17.616+0000 7f8eeccb1640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.189:6789/0 conn(0x7f8ee809cb80 legacy=0x7f8ee80717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:18.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:18.020+0000 7f8eeccb1640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.189:6789/0 conn(0x7f8ee809cb80 legacy=0x7f8ee80717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:18.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:18.824+0000 7f8eeccb1640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.189:6789/0 conn(0x7f8ee809cb80 legacy=0x7f8ee80717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:20.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.412+0000 7f8ee67fc640 10 monclient(hunting): tick 2024-09-16T14:52:20.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.412+0000 7f8ee67fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:52:20.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.412+0000 7f8ee67fc640 1 monclient(hunting): continuing hunt 2024-09-16T14:52:20.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.412+0000 7f8ee67fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:52:20.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.412+0000 7f8ee67fc640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.26:6790/0 conn(0x7f8ee8132650 legacy=0x7f8ee81378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:20.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.412+0000 7f8ee67fc640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.189:6789/0 conn(0x7f8ee809cb80 legacy=0x7f8ee80717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:52:20.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.412+0000 7f8ee67fc640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.26:6789/0 conn(0x7f8ee8138ff0 legacy=0x7f8ee813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:20.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.412+0000 7f8ee67fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:52:20.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.412+0000 7f8ee67fc640 10 monclient(hunting): picked mon.noname-c con 0x7f8ec80036d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:20.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.412+0000 7f8ee67fc640 10 monclient(hunting): picked mon.noname-a con 0x7f8ec8007360 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:20.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.412+0000 7f8ee67fc640 10 monclient(hunting): picked mon.noname-b con 0x7f8ee8132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:20.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.412+0000 7f8ee67fc640 1 -- 172.21.15.26:0/2047227077 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee809f2d0 con 0x7f8ec8007360 2024-09-16T14:52:20.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.412+0000 7f8eeccb1640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.189:6789/0 conn(0x7f8ec80036d0 legacy=0x7f8ec8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:20.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.412+0000 7f8ee67fc640 1 -- 172.21.15.26:0/2047227077 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee8132a80 con 0x7f8ec80036d0 2024-09-16T14:52:20.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.412+0000 7f8ee67fc640 1 -- 172.21.15.26:0/2047227077 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec80108e0 con 0x7f8ee8132650 2024-09-16T14:52:20.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.412+0000 7f8ee67fc640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:20.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:20.616+0000 7f8eeccb1640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.189:6789/0 conn(0x7f8ec80036d0 legacy=0x7f8ec8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:21.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:21.020+0000 7f8eeccb1640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.189:6789/0 conn(0x7f8ec80036d0 legacy=0x7f8ec8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:21.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:21.824+0000 7f8eeccb1640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.189:6789/0 conn(0x7f8ec80036d0 legacy=0x7f8ec8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:23.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.412+0000 7f8ee67fc640 10 monclient(hunting): tick 2024-09-16T14:52:23.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.412+0000 7f8ee67fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:52:23.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.412+0000 7f8ee67fc640 1 monclient(hunting): continuing hunt 2024-09-16T14:52:23.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.412+0000 7f8ee67fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:52:23.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.412+0000 7f8ee67fc640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.26:6790/0 conn(0x7f8ee8132650 legacy=0x7f8ec800d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:23.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.412+0000 7f8ee67fc640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.189:6789/0 conn(0x7f8ec80036d0 legacy=0x7f8ec8005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:52:23.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.412+0000 7f8ee67fc640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.26:6789/0 conn(0x7f8ec8007360 legacy=0x7f8ec80097d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:23.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.412+0000 7f8ee67fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:52:23.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.412+0000 7f8ee67fc640 10 monclient(hunting): picked mon.noname-a con 0x7f8ec800e9a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:23.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.412+0000 7f8ee67fc640 10 monclient(hunting): picked mon.noname-c con 0x7f8ec80036d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:23.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.412+0000 7f8ee67fc640 10 monclient(hunting): picked mon.noname-b con 0x7f8ee8132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:23.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.412+0000 7f8ee67fc640 1 -- 172.21.15.26:0/2047227077 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec8011b50 con 0x7f8ec800e9a0 2024-09-16T14:52:23.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.412+0000 7f8ee67fc640 1 -- 172.21.15.26:0/2047227077 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee8132a80 con 0x7f8ec80036d0 2024-09-16T14:52:23.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.412+0000 7f8ee67fc640 1 -- 172.21.15.26:0/2047227077 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec80113a0 con 0x7f8ee8132650 2024-09-16T14:52:23.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.412+0000 7f8ee67fc640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:23.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.412+0000 7f8ee7fff640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.189:6789/0 conn(0x7f8ec80036d0 legacy=0x7f8ec8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:23.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:23.616+0000 7f8ee7fff640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.189:6789/0 conn(0x7f8ec80036d0 legacy=0x7f8ec8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:24.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:24.020+0000 7f8ee7fff640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.189:6789/0 conn(0x7f8ec80036d0 legacy=0x7f8ec8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:24.824+0000 7f8ee7fff640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.189:6789/0 conn(0x7f8ec80036d0 legacy=0x7f8ec8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:25.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 1 -- 172.21.15.26:0/2047227077 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212124618 0 0) 0x7f8ec8011b50 con 0x7f8ec800e9a0 2024-09-16T14:52:25.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:52:25.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:25.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:25.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 10 monclient(hunting): my global_id is 194313 2024-09-16T14:52:25.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 1 -- 172.21.15.26:0/2047227077 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ed4003530 con 0x7f8ec800e9a0 2024-09-16T14:52:25.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 1 -- 172.21.15.26:0/2047227077 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1340110523 0 0) 0x7f8ed4003530 con 0x7f8ec800e9a0 2024-09-16T14:52:25.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.26:6790/0 conn(0x7f8ee8132650 legacy=0x7f8ec800d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:25.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.189:6789/0 conn(0x7f8ec80036d0 legacy=0x7f8ec8005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:52:25.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:52:25.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 1 monclient: found mon.noname-a 2024-09-16T14:52:25.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:25.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:52:25.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 1 -- 172.21.15.26:0/2047227077 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec80115a0 con 0x7f8ec800e9a0 2024-09-16T14:52:25.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 10 monclient: _finish_auth 0 2024-09-16T14:52:25.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:52:25.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:25.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 1 -- 172.21.15.26:0/2047227077 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ed0004230 con 0x7f8ec800e9a0 2024-09-16T14:52:25.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:25.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 10 monclient: dump: 2024-09-16T14:52:25.432 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.432 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.432 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.432 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.433 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.433 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.433 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.433 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.433 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.433 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8eeef15640 5 monclient: authenticate success, global_id 194313 2024-09-16T14:52:25.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8eeef15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:25.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 1 -- 172.21.15.26:0/2047227077 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ed0004450 con 0x7f8ec800e9a0 2024-09-16T14:52:25.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:25.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 1 -- 172.21.15.26:0/2047227077 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ed0004820 con 0x7f8ec800e9a0 2024-09-16T14:52:25.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8eeef15640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:25.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8eeef15640 4 set_mon_vals no callback set 2024-09-16T14:52:25.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:25.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8ee77fe640 10 monclient: dump: 2024-09-16T14:52:25.434 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.434 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.434 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.434 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.434 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.434 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.434 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.434 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.434 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.434 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8eeef15640 10 monclient: shutdown 2024-09-16T14:52:25.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8eeef15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:25.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8eeef15640 1 -- 172.21.15.26:0/2047227077 >> v1:172.21.15.26:6789/0 conn(0x7f8ec800e9a0 legacy=0x7f8ec8010a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:25.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8eeef15640 1 -- 172.21.15.26:0/2047227077 shutdown_connections 2024-09-16T14:52:25.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.428+0000 7f8eeef15640 1 -- 172.21.15.26:0/2047227077 >> 172.21.15.26:0/2047227077 conn(0x7f8ee809fb50 msgr2=0x7f8ee809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:25.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 1 -- 172.21.15.26:0/2047227077 shutdown_connections 2024-09-16T14:52:25.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 1 -- 172.21.15.26:0/2047227077 wait complete. 2024-09-16T14:52:25.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 10 monclient: build_initial_monmap 2024-09-16T14:52:25.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 10 monclient: monmap: 2024-09-16T14:52:25.437 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:25.437 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:25.437 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:25.437 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:25.437 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:25.437 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:25.437 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:25.437 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:25.437 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:25.437 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 1 Processor -- start 2024-09-16T14:52:25.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 1 -- start start 2024-09-16T14:52:25.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 10 monclient: init 2024-09-16T14:52:25.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:25.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:52:25.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 10 monclient(hunting): picked mon.noname-a con 0x7f8ec80036d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:25.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 10 monclient(hunting): picked mon.noname-b con 0x7f8ec800e9a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:25.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 10 monclient(hunting): picked mon.noname-c con 0x7f8ee8132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:25.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee8071410 con 0x7f8ec80036d0 2024-09-16T14:52:25.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee80a20f0 con 0x7f8ec800e9a0 2024-09-16T14:52:25.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee8138190 con 0x7f8ee8132650 2024-09-16T14:52:25.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:25.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeef15640 10 monclient(hunting): authenticate will time out at 1569.555785s 2024-09-16T14:52:25.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8eeccb1640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8ec80036d0 0x7f8ee812b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:49824/0 (socket says 172.21.15.26:49824) 2024-09-16T14:52:25.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8eeccb1640 1 -- 172.21.15.26:0/943410143 learned_addr learned my addr 172.21.15.26:0/943410143 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:25.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.432+0000 7f8ee7fff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f8ec800e9a0 legacy=0x7f8ee81ce540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:25.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 1 -- 172.21.15.26:0/943410143 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771261129 0 0) 0x7f8ee8071410 con 0x7f8ec80036d0 2024-09-16T14:52:25.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 10 monclient(hunting): _negotiate 2024-09-16T14:52:25.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:25.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:25.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 10 monclient(hunting): my global_id is 194316 2024-09-16T14:52:25.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 1 -- 172.21.15.26:0/943410143 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec8011af0 con 0x7f8ec80036d0 2024-09-16T14:52:25.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 1 -- 172.21.15.26:0/943410143 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2993096676 0 0) 0x7f8ec8011af0 con 0x7f8ec80036d0 2024-09-16T14:52:25.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 1 -- 172.21.15.26:0/943410143 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ee8071410 con 0x7f8ec80036d0 2024-09-16T14:52:25.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 1 -- 172.21.15.26:0/943410143 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ed00045c0 con 0x7f8ec80036d0 2024-09-16T14:52:25.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:25.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 10 monclient(hunting): dump: 2024-09-16T14:52:25.441 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.442 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.442 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.442 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.442 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.442 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.442 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.442 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.442 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.442 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:25.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 1 -- 172.21.15.26:0/943410143 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 773723315 0 0) 0x7f8ee8071410 con 0x7f8ec80036d0 2024-09-16T14:52:25.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 1 -- 172.21.15.26:0/943410143 >> v1:172.21.15.26:6790/0 conn(0x7f8ee8132650 legacy=0x7f8ee81d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:25.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 1 -- 172.21.15.26:0/943410143 >> v1:172.21.15.189:6789/0 conn(0x7f8ec800e9a0 legacy=0x7f8ee81ce540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:52:25.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 10 monclient: _finish_hunting 0 2024-09-16T14:52:25.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 1 monclient: found mon.a 2024-09-16T14:52:25.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:25.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:25.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 1 -- 172.21.15.26:0/943410143 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee8138470 con 0x7f8ec80036d0 2024-09-16T14:52:25.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 10 monclient: _finish_auth 0 2024-09-16T14:52:25.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 10 monclient: _check_auth_tickets 2024-09-16T14:52:25.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:25.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8eeef15640 5 monclient: authenticate success, global_id 194316 2024-09-16T14:52:25.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8eeef15640 10 monclient: _renew_subs 2024-09-16T14:52:25.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8eeef15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:25.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8eeef15640 1 -- 172.21.15.26:0/943410143 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee8136a60 con 0x7f8ec80036d0 2024-09-16T14:52:25.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8eeef15640 10 monclient: _renew_subs 2024-09-16T14:52:25.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8eeef15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:25.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8eeef15640 1 -- 172.21.15.26:0/943410143 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee812c040 con 0x7f8ec80036d0 2024-09-16T14:52:25.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 1 -- 172.21.15.26:0/943410143 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ed0004aa0 con 0x7f8ec80036d0 2024-09-16T14:52:25.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:25.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 1 -- 172.21.15.26:0/943410143 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ed0004e30 con 0x7f8ec80036d0 2024-09-16T14:52:25.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:25.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 10 monclient: dump: 2024-09-16T14:52:25.444 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.444 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.444 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.445 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee6ffd640 4 set_mon_vals no callback set 2024-09-16T14:52:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 1 -- 172.21.15.26:0/943410143 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8ed0005010 con 0x7f8ec80036d0 2024-09-16T14:52:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee57fa640 1 -- 172.21.15.26:0/943410143 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f8ed0043fd0 con 0x7f8ec80036d0 2024-09-16T14:52:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8eeef15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8eeef15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8eeef15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:25.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8eeef15640 1 -- 172.21.15.26:0/943410143 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ecc005050 con 0x7f8ec80036d0 2024-09-16T14:52:25.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.436+0000 7f8ee7fff640 10 monclient: get_auth_request con 0x7f8ec80376d0 auth_method 0 2024-09-16T14:52:25.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.444+0000 7f8ee57fa640 1 -- 172.21.15.26:0/943410143 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8ed0014a30 con 0x7f8ec80036d0 2024-09-16T14:52:25.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.444+0000 7f8ee57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:25.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.444+0000 7f8ee57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:25.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.668+0000 7f8eeef15640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:52:25.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.668+0000 7f8eeef15640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:25.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.668+0000 7f8eeef15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:25.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.668+0000 7f8eeef15640 1 -- 172.21.15.26:0/943410143 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8ecc005880 con 0x7f8ec80036d0 2024-09-16T14:52:25.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.668+0000 7f8ee57fa640 1 -- 172.21.15.26:0/943410143 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3557772963) 0x7f8ed0012ea0 con 0x7f8ec80036d0 2024-09-16T14:52:25.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.668+0000 7f8ee57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:25.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.668+0000 7f8ee57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:25.671 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":94,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:52:25.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.672+0000 7f8eeef15640 1 -- 172.21.15.26:0/943410143 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8ec80376d0 legacy=0x7f8ec8039b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:25.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.672+0000 7f8eeef15640 10 monclient: shutdown 2024-09-16T14:52:25.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.672+0000 7f8eeef15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:25.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.672+0000 7f8eeef15640 1 -- 172.21.15.26:0/943410143 >> v1:172.21.15.26:6789/0 conn(0x7f8ec80036d0 legacy=0x7f8ee812b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:25.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.672+0000 7f8eeef15640 1 -- 172.21.15.26:0/943410143 shutdown_connections 2024-09-16T14:52:25.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.672+0000 7f8eeef15640 1 -- 172.21.15.26:0/943410143 >> 172.21.15.26:0/943410143 conn(0x7f8ee809fb50 msgr2=0x7f8ee80a1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:25.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.672+0000 7f8eeef15640 1 -- 172.21.15.26:0/943410143 shutdown_connections 2024-09-16T14:52:25.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.672+0000 7f8eeef15640 1 -- 172.21.15.26:0/943410143 wait complete. 2024-09-16T14:52:25.689 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:52:25.689 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T14:52:25.690 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:52:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 10 monclient: get_monmap_and_config 2024-09-16T14:52:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 10 monclient: build_initial_monmap 2024-09-16T14:52:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 10 monclient: monmap: 2024-09-16T14:52:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:25.805617+0000 2024-09-16T14:52:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:25.805617+0000 2024-09-16T14:52:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:25.805 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 1 Processor -- start 2024-09-16T14:52:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 1 -- start start 2024-09-16T14:52:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 10 monclient: init 2024-09-16T14:52:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:52:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 10 monclient(hunting): picked mon.noname-a con 0x7f922c0968c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 10 monclient(hunting): picked mon.noname-c con 0x7f922c098bc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 10 monclient(hunting): picked mon.noname-b con 0x7f922c096ca0 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922c06b2d0 con 0x7f922c0968c0 2024-09-16T14:52:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922c097940 con 0x7f922c098bc0 2024-09-16T14:52:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922c09fec0 con 0x7f922c096ca0 2024-09-16T14:52:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 10 monclient(hunting): authenticate will time out at 1569.925723s 2024-09-16T14:52:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f9231150640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f922c0968c0 0x7f922c09f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:49828/0 (socket says 172.21.15.26:49828) 2024-09-16T14:52:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f9231150640 1 -- 172.21.15.26:0/315067827 learned_addr learned my addr 172.21.15.26:0/315067827 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f923094f640 1 -- 172.21.15.26:0/315067827 >> v1:172.21.15.189:6789/0 conn(0x7f922c098bc0 legacy=0x7f922c09f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 1 -- 172.21.15.26:0/315067827 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174140431 0 0) 0x7f922c06b2d0 con 0x7f922c0968c0 2024-09-16T14:52:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient(hunting): my global_id is 194319 2024-09-16T14:52:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 1 -- 172.21.15.26:0/315067827 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9214003530 con 0x7f922c0968c0 2024-09-16T14:52:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 1 -- 172.21.15.26:0/315067827 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497701825 0 0) 0x7f922c09fec0 con 0x7f922c096ca0 2024-09-16T14:52:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient(hunting): my global_id is 214178 2024-09-16T14:52:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 1 -- 172.21.15.26:0/315067827 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f922c06b2d0 con 0x7f922c096ca0 2024-09-16T14:52:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 1 -- 172.21.15.26:0/315067827 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3196082292 0 0) 0x7f9214003530 con 0x7f922c0968c0 2024-09-16T14:52:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 1 -- 172.21.15.26:0/315067827 >> v1:172.21.15.26:6790/0 conn(0x7f922c096ca0 legacy=0x7f922c13da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 1 -- 172.21.15.26:0/315067827 >> v1:172.21.15.189:6789/0 conn(0x7f922c098bc0 legacy=0x7f922c09f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:52:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:52:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 1 monclient: found mon.noname-a 2024-09-16T14:52:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:52:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 1 -- 172.21.15.26:0/315067827 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922c097c20 con 0x7f922c0968c0 2024-09-16T14:52:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient: _finish_auth 0 2024-09-16T14:52:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:52:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 5 monclient: authenticate success, global_id 194319 2024-09-16T14:52:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 1 -- 172.21.15.26:0/315067827 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f921c001620 con 0x7f922c0968c0 2024-09-16T14:52:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient: dump: 2024-09-16T14:52:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.816 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 1 -- 172.21.15.26:0/315067827 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f921c001900 con 0x7f922c0968c0 2024-09-16T14:52:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 1 -- 172.21.15.26:0/315067827 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f921c001c90 con 0x7f922c0968c0 2024-09-16T14:52:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f922bfff640 10 monclient: dump: 2024-09-16T14:52:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.818 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 4 set_mon_vals no callback set 2024-09-16T14:52:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 10 monclient: shutdown 2024-09-16T14:52:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 1 -- 172.21.15.26:0/315067827 >> v1:172.21.15.26:6789/0 conn(0x7f922c0968c0 legacy=0x7f922c09f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 1 -- 172.21.15.26:0/315067827 shutdown_connections 2024-09-16T14:52:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 1 -- 172.21.15.26:0/315067827 >> 172.21.15.26:0/315067827 conn(0x7f922c0a0360 msgr2=0x7f922c0a0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 1 -- 172.21.15.26:0/315067827 shutdown_connections 2024-09-16T14:52:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.804+0000 7f92333b4640 1 -- 172.21.15.26:0/315067827 wait complete. 2024-09-16T14:52:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 10 monclient: build_initial_monmap 2024-09-16T14:52:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 10 monclient: monmap: 2024-09-16T14:52:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:25.819 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 1 Processor -- start 2024-09-16T14:52:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 1 -- start start 2024-09-16T14:52:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 10 monclient: init 2024-09-16T14:52:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:52:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 10 monclient(hunting): picked mon.noname-a con 0x7f922c0968c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 10 monclient(hunting): picked mon.noname-c con 0x7f922c096ca0 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 10 monclient(hunting): picked mon.noname-b con 0x7f922c098bc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922c09a880 con 0x7f922c0968c0 2024-09-16T14:52:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922c091620 con 0x7f922c098bc0 2024-09-16T14:52:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922c095170 con 0x7f922c096ca0 2024-09-16T14:52:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 10 monclient(hunting): authenticate will time out at 1569.929019s 2024-09-16T14:52:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9231150640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f922c0968c0 0x7f922c08d920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:49832/0 (socket says 172.21.15.26:49832) 2024-09-16T14:52:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9231150640 1 -- 172.21.15.26:0/3006460049 learned_addr learned my addr 172.21.15.26:0/3006460049 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9231951640 1 -- 172.21.15.26:0/3006460049 >> v1:172.21.15.189:6789/0 conn(0x7f922c098bc0 legacy=0x7f922c1d1670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946327414 0 0) 0x7f922c09a880 con 0x7f922c0968c0 2024-09-16T14:52:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:52:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): my global_id is 194322 2024-09-16T14:52:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9218003530 con 0x7f922c0968c0 2024-09-16T14:52:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403165351 0 0) 0x7f922c095170 con 0x7f922c096ca0 2024-09-16T14:52:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:52:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): my global_id is 214181 2024-09-16T14:52:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f922c09a880 con 0x7f922c096ca0 2024-09-16T14:52:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3828414567 0 0) 0x7f9218003530 con 0x7f922c0968c0 2024-09-16T14:52:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f922c095170 con 0x7f922c0968c0 2024-09-16T14:52:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f921c0015e0 con 0x7f922c0968c0 2024-09-16T14:52:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): dump: 2024-09-16T14:52:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.822 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3037547476 0 0) 0x7f922c09a880 con 0x7f922c096ca0 2024-09-16T14:52:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9218003530 con 0x7f922c096ca0 2024-09-16T14:52:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9220004400 con 0x7f922c096ca0 2024-09-16T14:52:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): dump: 2024-09-16T14:52:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.824 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 222888903 0 0) 0x7f9218003530 con 0x7f922c096ca0 2024-09-16T14:52:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 >> v1:172.21.15.189:6789/0 conn(0x7f922c098bc0 legacy=0x7f922c1d1670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:52:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 >> v1:172.21.15.26:6789/0 conn(0x7f922c0968c0 legacy=0x7f922c08d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:52:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 monclient: found mon.c 2024-09-16T14:52:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922c094c20 con 0x7f922c096ca0 2024-09-16T14:52:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient: _finish_auth 0 2024-09-16T14:52:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:52:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 5 monclient: authenticate success, global_id 214181 2024-09-16T14:52:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 10 monclient: _renew_subs 2024-09-16T14:52:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 1 -- 172.21.15.26:0/3006460049 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922c091340 con 0x7f922c096ca0 2024-09-16T14:52:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 10 monclient: _renew_subs 2024-09-16T14:52:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 1 -- 172.21.15.26:0/3006460049 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f922c08f2c0 con 0x7f922c096ca0 2024-09-16T14:52:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92200048d0 con 0x7f922c096ca0 2024-09-16T14:52:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9220004be0 con 0x7f922c096ca0 2024-09-16T14:52:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f922affd640 4 set_mon_vals no callback set 2024-09-16T14:52:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient: dump: 2024-09-16T14:52:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.826 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f92333b4640 1 -- 172.21.15.26:0/3006460049 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f920c006590 con 0x7f920c002830 2024-09-16T14:52:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9220004e00 con 0x7f922c096ca0 2024-09-16T14:52:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f9220043fe0 con 0x7f922c096ca0 2024-09-16T14:52:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892286340 0 0) 0x7f920c006590 con 0x7f920c002830 2024-09-16T14:52:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:52:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient(hunting): my global_id is 194325 2024-09-16T14:52:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9218003530 con 0x7f920c002830 2024-09-16T14:52:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3521012367 0 0) 0x7f9218003530 con 0x7f920c002830 2024-09-16T14:52:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f920c006590 con 0x7f920c002830 2024-09-16T14:52:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f921c004df0 con 0x7f920c002830 2024-09-16T14:52:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient: dump: 2024-09-16T14:52:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.830 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9231951640 10 monclient: get_auth_request con 0x7f92180361d0 auth_method 0 2024-09-16T14:52:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 654245219 0 0) 0x7f920c006590 con 0x7f920c002830 2024-09-16T14:52:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f920c00a0e0 con 0x7f920c002830 2024-09-16T14:52:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f921c00a480 con 0x7f920c002830 2024-09-16T14:52:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.808+0000 7f9229ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.812+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 >> v1:172.21.15.26:6789/0 conn(0x7f920c002830 legacy=0x7f920c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.812+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 >> v1:172.21.15.26:6789/0 conn(0x7f920c002830 legacy=0x7f920c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:25.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f92333b4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:52:25.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f92333b4640 1 -- 172.21.15.26:0/3006460049 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f920c009da0 con 0x7f920c0090c0 2024-09-16T14:52:25.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409585755 0 0) 0x7f920c009da0 con 0x7f920c0090c0 2024-09-16T14:52:25.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:52:25.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:25.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:25.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 10 monclient(hunting): my global_id is 194328 2024-09-16T14:52:25.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f920c006590 con 0x7f920c0090c0 2024-09-16T14:52:25.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2197057149 0 0) 0x7f920c006590 con 0x7f920c0090c0 2024-09-16T14:52:25.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f920c009da0 con 0x7f920c0090c0 2024-09-16T14:52:25.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f921c005830 con 0x7f920c0090c0 2024-09-16T14:52:25.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:25.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 10 monclient: dump: 2024-09-16T14:52:25.833 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.834 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.834 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.834 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.834 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.834 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.834 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.834 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.834 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.834 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2052472140 0 0) 0x7f920c009da0 con 0x7f920c0090c0 2024-09-16T14:52:25.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f920c00c290 con 0x7f920c0090c0 2024-09-16T14:52:25.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4200 (unknown 0 0 3686540818) 0x7f921c007ae0 con 0x7f920c0090c0 2024-09-16T14:52:25.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:52:25.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:25.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 >> v1:172.21.15.26:6789/0 conn(0x7f920c0090c0 legacy=0x7f920c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:25.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9229ffb640 1 -- 172.21.15.26:0/3006460049 >> v1:172.21.15.26:6789/0 conn(0x7f920c0090c0 legacy=0x7f920c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:25.835 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:52:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:52:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 94, 2024-09-16T14:52:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:52:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:52:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:52:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:52:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:52:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:52:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:52:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:52:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:52:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:52:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:52:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:52:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:52:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:52:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:52:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:52:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:52:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:52:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:52:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:52:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:52:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:52:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:52:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:52:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:52:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:52:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:52:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:52:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:52:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:52:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:52:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:52:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:52:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:52:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:52:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:52:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:52:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:52:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:52:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:52:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:52:25.851 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:52:25.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f92333b4640 1 -- 172.21.15.26:0/3006460049 >> v1:172.21.15.189:6812/3196161582 conn(0x7f92180361d0 legacy=0x7f9218038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:25.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f92333b4640 10 monclient: shutdown 2024-09-16T14:52:25.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f92333b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:25.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f92333b4640 1 -- 172.21.15.26:0/3006460049 >> v1:172.21.15.26:6790/0 conn(0x7f922c096ca0 legacy=0x7f922c08e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:25.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9231951640 1 -- 172.21.15.26:0/3006460049 reap_dead start 2024-09-16T14:52:25.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f9231951640 1 -- 172.21.15.26:0/3006460049 reap_dead start 2024-09-16T14:52:25.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f92333b4640 1 -- 172.21.15.26:0/3006460049 shutdown_connections 2024-09-16T14:52:25.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f92333b4640 1 -- 172.21.15.26:0/3006460049 >> 172.21.15.26:0/3006460049 conn(0x7f922c0a0360 msgr2=0x7f922c095380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:25.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f92333b4640 1 -- 172.21.15.26:0/3006460049 shutdown_connections 2024-09-16T14:52:25.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.816+0000 7f92333b4640 1 -- 172.21.15.26:0/3006460049 wait complete. 2024-09-16T14:52:25.854 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:52:25.854 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:52:25.854 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:52:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 10 monclient: get_monmap_and_config 2024-09-16T14:52:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 10 monclient: build_initial_monmap 2024-09-16T14:52:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 10 monclient: monmap: 2024-09-16T14:52:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:25.973737+0000 2024-09-16T14:52:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:25.973737+0000 2024-09-16T14:52:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:25.973 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 1 Processor -- start 2024-09-16T14:52:25.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 1 -- start start 2024-09-16T14:52:25.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 10 monclient: init 2024-09-16T14:52:25.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:25.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:52:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 10 monclient(hunting): picked mon.noname-c con 0x7fa7100a0b80 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 10 monclient(hunting): picked mon.noname-b con 0x7fa7100a1670 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 10 monclient(hunting): picked mon.noname-a con 0x7fa7101394a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7100a29d0 con 0x7fa7101394a0 2024-09-16T14:52:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7100992c0 con 0x7fa7100a0b80 2024-09-16T14:52:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa71009a5e0 con 0x7fa7100a1670 2024-09-16T14:52:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 10 monclient(hunting): authenticate will time out at 1570.093739s 2024-09-16T14:52:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa715060640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa7100a1670 0x7fa710137d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:37790/0 (socket says 172.21.15.26:37790) 2024-09-16T14:52:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa715060640 1 -- 172.21.15.26:0/2842969361 learned_addr learned my addr 172.21.15.26:0/2842969361 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa715861640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa7100a0b80 legacy=0x7fa7100a0f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 1 -- 172.21.15.26:0/2842969361 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651822539 0 0) 0x7fa71009a5e0 con 0x7fa7100a1670 2024-09-16T14:52:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient(hunting): _negotiate 2024-09-16T14:52:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient(hunting): my global_id is 214184 2024-09-16T14:52:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 1 -- 172.21.15.26:0/2842969361 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6fc003530 con 0x7fa7100a1670 2024-09-16T14:52:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 1 -- 172.21.15.26:0/2842969361 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064964653 0 0) 0x7fa7100a29d0 con 0x7fa7101394a0 2024-09-16T14:52:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient(hunting): _negotiate 2024-09-16T14:52:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient(hunting): my global_id is 194331 2024-09-16T14:52:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 1 -- 172.21.15.26:0/2842969361 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa71009a5e0 con 0x7fa7101394a0 2024-09-16T14:52:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 1 -- 172.21.15.26:0/2842969361 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2354264392 0 0) 0x7fa6fc003530 con 0x7fa7100a1670 2024-09-16T14:52:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 1 -- 172.21.15.26:0/2842969361 >> v1:172.21.15.189:6789/0 conn(0x7fa7100a0b80 legacy=0x7fa7100a0f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:52:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 1 -- 172.21.15.26:0/2842969361 >> v1:172.21.15.26:6789/0 conn(0x7fa7101394a0 legacy=0x7fa71013b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient: _finish_hunting 0 2024-09-16T14:52:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 1 monclient: found mon.noname-b 2024-09-16T14:52:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:52:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 1 -- 172.21.15.26:0/2842969361 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa710072400 con 0x7fa7100a1670 2024-09-16T14:52:25.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient: _finish_auth 0 2024-09-16T14:52:25.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient: _check_auth_tickets 2024-09-16T14:52:25.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:25.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 1 -- 172.21.15.26:0/2842969361 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa700001620 con 0x7fa7100a1670 2024-09-16T14:52:25.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:52:25.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient: dump: 2024-09-16T14:52:25.984 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.984 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.984 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.984 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.984 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.985 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.985 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.985 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.985 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.985 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 5 monclient: authenticate success, global_id 214184 2024-09-16T14:52:25.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:25.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 1 -- 172.21.15.26:0/2842969361 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa700001880 con 0x7fa7100a1670 2024-09-16T14:52:25.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:25.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 1 -- 172.21.15.26:0/2842969361 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa700001c10 con 0x7fa7100a1670 2024-09-16T14:52:25.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:25.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa71485f640 10 monclient: dump: 2024-09-16T14:52:25.986 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.986 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.986 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.986 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.986 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.986 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.986 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.986 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.986 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.987 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 4 set_mon_vals no callback set 2024-09-16T14:52:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 10 monclient: shutdown 2024-09-16T14:52:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 1 -- 172.21.15.26:0/2842969361 >> v1:172.21.15.26:6790/0 conn(0x7fa7100a1670 legacy=0x7fa710137d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 1 -- 172.21.15.26:0/2842969361 shutdown_connections 2024-09-16T14:52:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 1 -- 172.21.15.26:0/2842969361 >> 172.21.15.26:0/2842969361 conn(0x7fa7100a5770 msgr2=0x7fa7100a1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 1 -- 172.21.15.26:0/2842969361 shutdown_connections 2024-09-16T14:52:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.972+0000 7fa717ac5640 1 -- 172.21.15.26:0/2842969361 wait complete. 2024-09-16T14:52:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 10 monclient: build_initial_monmap 2024-09-16T14:52:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 10 monclient: monmap: 2024-09-16T14:52:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:25.988 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:25.988 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:25.988 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:25.988 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:25.988 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:25.988 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:25.988 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:25.988 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 1 Processor -- start 2024-09-16T14:52:25.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 1 -- start start 2024-09-16T14:52:25.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 10 monclient: init 2024-09-16T14:52:25.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:25.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:52:25.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 10 monclient(hunting): picked mon.noname-c con 0x7fa7100a0b80 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:25.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 10 monclient(hunting): picked mon.noname-a con 0x7fa7100a1670 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:25.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 10 monclient(hunting): picked mon.noname-b con 0x7fa7101394a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:25.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa71006b380 con 0x7fa7100a1670 2024-09-16T14:52:25.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa710133920 con 0x7fa7101394a0 2024-09-16T14:52:25.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa71006bf50 con 0x7fa7100a0b80 2024-09-16T14:52:25.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:25.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 10 monclient(hunting): authenticate will time out at 1570.096925s 2024-09-16T14:52:25.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa715861640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa7100a0b80 0x7fa7101c98a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:37804/0 (socket says 172.21.15.26:37804) 2024-09-16T14:52:25.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa715861640 1 -- 172.21.15.26:0/1593855416 learned_addr learned my addr 172.21.15.26:0/1593855416 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:25.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa716062640 1 -- 172.21.15.26:0/1593855416 >> v1:172.21.15.189:6789/0 conn(0x7fa7101394a0 legacy=0x7fa7101d2020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:52:25.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469034382 0 0) 0x7fa71006b380 con 0x7fa7100a1670 2024-09-16T14:52:25.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:25.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:25.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:25.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): my global_id is 194334 2024-09-16T14:52:25.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f4003530 con 0x7fa7100a1670 2024-09-16T14:52:25.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862863439 0 0) 0x7fa71006bf50 con 0x7fa7100a0b80 2024-09-16T14:52:25.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:25.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:25.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:25.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): my global_id is 214187 2024-09-16T14:52:25.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa71006b380 con 0x7fa7100a0b80 2024-09-16T14:52:25.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2078010381 0 0) 0x7fa6f4003530 con 0x7fa7100a1670 2024-09-16T14:52:25.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa71006bf50 con 0x7fa7100a1670 2024-09-16T14:52:25.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa700001b90 con 0x7fa7100a1670 2024-09-16T14:52:25.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:25.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): dump: 2024-09-16T14:52:25.991 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.991 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.991 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.991 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.991 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.991 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.991 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.991 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.992 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.992 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:25.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 922383979 0 0) 0x7fa71006b380 con 0x7fa7100a0b80 2024-09-16T14:52:25.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6f4003530 con 0x7fa7100a0b80 2024-09-16T14:52:25.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa70c004080 con 0x7fa7100a0b80 2024-09-16T14:52:25.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:25.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): dump: 2024-09-16T14:52:25.992 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.992 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.992 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.993 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.993 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.993 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.993 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.993 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.993 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.993 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 280977356 0 0) 0x7fa71006bf50 con 0x7fa7100a1670 2024-09-16T14:52:25.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 >> v1:172.21.15.26:6790/0 conn(0x7fa7100a0b80 legacy=0x7fa7101c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:25.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 >> v1:172.21.15.189:6789/0 conn(0x7fa7101394a0 legacy=0x7fa7101d2020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:52:25.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient: _finish_hunting 0 2024-09-16T14:52:25.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 monclient: found mon.a 2024-09-16T14:52:25.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:25.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:25.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71006ce40 con 0x7fa7100a1670 2024-09-16T14:52:25.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient: _finish_auth 0 2024-09-16T14:52:25.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient: _check_auth_tickets 2024-09-16T14:52:25.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:25.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 5 monclient: authenticate success, global_id 194334 2024-09-16T14:52:25.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 10 monclient: _renew_subs 2024-09-16T14:52:25.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:25.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 1 -- 172.21.15.26:0/1593855416 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa710120010 con 0x7fa7100a1670 2024-09-16T14:52:25.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 10 monclient: _renew_subs 2024-09-16T14:52:25.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:25.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 1 -- 172.21.15.26:0/1593855416 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa71006b830 con 0x7fa7100a1670 2024-09-16T14:52:25.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa700001190 con 0x7fa7100a1670 2024-09-16T14:52:25.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:25.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa700004c40 con 0x7fa7100a1670 2024-09-16T14:52:25.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:25.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa707fff640 4 set_mon_vals no callback set 2024-09-16T14:52:25.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient: dump: 2024-09-16T14:52:25.995 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.995 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.995 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.995 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.995 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.995 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.996 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.996 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.996 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.996 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:25.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa717ac5640 1 -- 172.21.15.26:0/1593855416 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa6f8006590 con 0x7fa6f8002830 2024-09-16T14:52:25.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa70000f6b0 con 0x7fa7100a1670 2024-09-16T14:52:25.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fa700043d50 con 0x7fa7100a1670 2024-09-16T14:52:25.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957225444 0 0) 0x7fa6f8006590 con 0x7fa6f8002830 2024-09-16T14:52:25.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:25.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:25.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:25.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient(hunting): my global_id is 214190 2024-09-16T14:52:25.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa71006bf50 con 0x7fa6f8002830 2024-09-16T14:52:25.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3213426594 0 0) 0x7fa71006bf50 con 0x7fa6f8002830 2024-09-16T14:52:25.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6f8006590 con 0x7fa6f8002830 2024-09-16T14:52:25.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa70c0046c0 con 0x7fa6f8002830 2024-09-16T14:52:25.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:25.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:25.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 10 monclient: dump: 2024-09-16T14:52:25.997 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:25.997 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:25.997 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.997 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:25.997 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:25.998 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:25.998 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:25.998 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:25.998 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:25.998 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:25.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa716062640 10 monclient: get_auth_request con 0x7fa6f4036360 auth_method 0 2024-09-16T14:52:25.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2908263410 0 0) 0x7fa6f8006590 con 0x7fa6f8002830 2024-09-16T14:52:25.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.976+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6f800a0e0 con 0x7fa6f8002830 2024-09-16T14:52:25.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.980+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa70c0069e0 con 0x7fa6f8002830 2024-09-16T14:52:25.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.980+0000 7fa7067fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:25.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.980+0000 7fa7067fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:25.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.980+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 >> v1:172.21.15.26:6790/0 conn(0x7fa6f8002830 legacy=0x7fa6f8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:25.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.980+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 >> v1:172.21.15.26:6790/0 conn(0x7fa6f8002830 legacy=0x7fa6f8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:26.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa717ac5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:52:26.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa717ac5640 1 -- 172.21.15.26:0/1593855416 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa6f8009d80 con 0x7fa6f80090a0 2024-09-16T14:52:26.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131162853 0 0) 0x7fa6f8009d80 con 0x7fa6f80090a0 2024-09-16T14:52:26.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:26.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:26.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:26.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 10 monclient(hunting): my global_id is 214193 2024-09-16T14:52:26.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f8006590 con 0x7fa6f80090a0 2024-09-16T14:52:26.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3747766006 0 0) 0x7fa6f8006590 con 0x7fa6f80090a0 2024-09-16T14:52:26.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6f8009d80 con 0x7fa6f80090a0 2024-09-16T14:52:26.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa70c004f80 con 0x7fa6f80090a0 2024-09-16T14:52:26.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:26.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:26.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 10 monclient: dump: 2024-09-16T14:52:26.002 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:26.002 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:26.002 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:26.002 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:26.002 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:26.002 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:26.002 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:26.002 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:26.002 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:26.002 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:26.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 978759754 0 0) 0x7fa6f8009d80 con 0x7fa6f80090a0 2024-09-16T14:52:26.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa6f800c290 con 0x7fa6f80090a0 2024-09-16T14:52:26.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3870 (unknown 0 0 318761524) 0x7fa70c006b50 con 0x7fa6f80090a0 2024-09-16T14:52:26.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:52:26.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:26.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 >> v1:172.21.15.26:6790/0 conn(0x7fa6f80090a0 legacy=0x7fa6f8009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:26.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa7067fc640 1 -- 172.21.15.26:0/1593855416 >> v1:172.21.15.26:6790/0 conn(0x7fa6f80090a0 legacy=0x7fa6f8009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:26.003 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:52:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:52:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 94, 2024-09-16T14:52:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:52:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:52:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:52:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:52:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:52:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:52:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:26.005 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:26.005 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:26.005 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:26.005 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:26.005 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:26.005 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:26.005 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:52:26.005 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:52:26.005 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:26.005 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:26.005 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:26.005 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:26.005 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:26.006 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:26.006 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:26.006 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:26.006 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:26.006 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:26.006 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:26.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:26.006 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:52:26.006 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:52:26.006 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:52:26.006 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:52:26.006 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:52:26.006 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:52:26.006 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:26.007 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:26.007 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:52:26.007 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:52:26.007 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:52:26.007 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:52:26.007 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:52:26.007 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:52:26.007 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:52:26.007 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:26.007 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:52:26.007 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:26.007 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:26.007 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:26.008 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:26.008 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:26.008 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:26.008 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:26.008 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:26.008 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:26.008 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:26.008 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:26.008 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:52:26.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:26.008 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:52:26.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:26.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:26.009 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:26.009 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:26.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:26.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:26.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:26.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:52:26.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:26.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:26.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:26.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:26.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:52:26.009 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:52:26.009 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:26.010 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:26.010 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:26.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:26.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:26.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:26.010 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:26.010 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:26.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:26.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:26.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:26.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:52:26.010 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:26.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:26.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:26.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:26.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:52:26.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:52:26.011 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:26.011 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:26.011 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:26.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:26.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:26.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:26.011 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:26.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:26.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:26.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:26.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:26.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:52:26.012 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:26.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:26.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:26.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:26.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:52:26.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:52:26.012 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:26.012 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:26.012 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:26.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:26.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:26.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:26.013 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:52:26.013 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:52:26.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:26.013 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:26.013 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:26.013 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:26.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:26.013 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:26.013 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:52:26.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:26.013 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:26.013 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:26.013 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:26.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:26.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:26.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:26.014 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:52:26.014 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:52:26.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa717ac5640 1 -- 172.21.15.26:0/1593855416 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa6f4036360 legacy=0x7fa6f4038820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:26.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa717ac5640 10 monclient: shutdown 2024-09-16T14:52:26.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa717ac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:26.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa717ac5640 1 -- 172.21.15.26:0/1593855416 >> v1:172.21.15.26:6789/0 conn(0x7fa7100a1670 legacy=0x7fa7101ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:26.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa716062640 1 -- 172.21.15.26:0/1593855416 reap_dead start 2024-09-16T14:52:26.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa716062640 1 -- 172.21.15.26:0/1593855416 reap_dead start 2024-09-16T14:52:26.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa717ac5640 1 -- 172.21.15.26:0/1593855416 shutdown_connections 2024-09-16T14:52:26.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa717ac5640 1 -- 172.21.15.26:0/1593855416 >> 172.21.15.26:0/1593855416 conn(0x7fa7100a5770 msgr2=0x7fa7100718a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:26.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa717ac5640 1 -- 172.21.15.26:0/1593855416 shutdown_connections 2024-09-16T14:52:26.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:25.984+0000 7fa717ac5640 1 -- 172.21.15.26:0/1593855416 wait complete. 2024-09-16T14:52:26.017 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:52:26.017 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:52:46.018 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:52:46.018 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T14:52:46.019 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T14:52:46.019 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T14:52:46.022 INFO:tasks.ceph.mon.b:Started 2024-09-16T14:52:46.022 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:52:46.022 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:52:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c4570c640 10 monclient: get_monmap_and_config 2024-09-16T14:52:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c4570c640 10 monclient: build_initial_monmap 2024-09-16T14:52:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c4570c640 10 monclient: monmap: 2024-09-16T14:52:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:46.155073+0000 2024-09-16T14:52:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:46.155073+0000 2024-09-16T14:52:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:46.155 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c4570c640 1 Processor -- start 2024-09-16T14:52:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c4570c640 1 -- start start 2024-09-16T14:52:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c4570c640 10 monclient: init 2024-09-16T14:52:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c4570c640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c4570c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:52:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c4570c640 10 monclient(hunting): picked mon.noname-c con 0x7f1c4009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c4570c640 10 monclient(hunting): picked mon.noname-a con 0x7f1c40132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c4570c640 10 monclient(hunting): picked mon.noname-b con 0x7f1c40138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c4570c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4009d260 con 0x7f1c40132650 2024-09-16T14:52:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c4570c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c40132a80 con 0x7f1c4009cb80 2024-09-16T14:52:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c4570c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4009cf60 con 0x7f1c40138ff0 2024-09-16T14:52:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c4570c640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c4570c640 10 monclient(hunting): authenticate will time out at 1590.275328s 2024-09-16T14:52:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c3f7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1c40138ff0 0x7f1c4013b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38466/0 (socket says 172.21.15.26:38466) 2024-09-16T14:52:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.152+0000 7f1c3f7fe640 1 -- 172.21.15.26:0/1335991817 learned_addr learned my addr 172.21.15.26:0/1335991817 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:46.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.164+0000 7f1c3dffb640 1 -- 172.21.15.26:0/1335991817 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463405502 0 0) 0x7f1c4009cf60 con 0x7f1c40138ff0 2024-09-16T14:52:46.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.164+0000 7f1c3dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.164+0000 7f1c3dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.164+0000 7f1c3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.164+0000 7f1c3dffb640 10 monclient(hunting): my global_id is 214196 2024-09-16T14:52:46.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.164+0000 7f1c3dffb640 1 -- 172.21.15.26:0/1335991817 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c2c003530 con 0x7f1c40138ff0 2024-09-16T14:52:46.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.164+0000 7f1c3dffb640 1 -- 172.21.15.26:0/1335991817 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1494777267 0 0) 0x7f1c2c003530 con 0x7f1c40138ff0 2024-09-16T14:52:46.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 1 -- 172.21.15.26:0/1335991817 >> v1:172.21.15.189:6789/0 conn(0x7f1c4009cb80 legacy=0x7f1c400717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 1 -- 172.21.15.26:0/1335991817 >> v1:172.21.15.26:6789/0 conn(0x7f1c40132650 legacy=0x7f1c401378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:52:46.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 1 monclient: found mon.noname-b 2024-09-16T14:52:46.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:46.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:52:46.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 1 -- 172.21.15.26:0/1335991817 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4009f2d0 con 0x7f1c40138ff0 2024-09-16T14:52:46.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 10 monclient: _finish_auth 0 2024-09-16T14:52:46.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:52:46.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:46.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 5 monclient: authenticate success, global_id 214196 2024-09-16T14:52:46.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:46.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 1 -- 172.21.15.26:0/1335991817 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1c380019e0 con 0x7f1c40138ff0 2024-09-16T14:52:46.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:52:46.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 10 monclient: dump: 2024-09-16T14:52:46.171 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.171 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.171 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.171 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.171 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.171 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.171 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.171 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.171 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.171 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:46.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 1 -- 172.21.15.26:0/1335991817 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c38001c00 con 0x7f1c40138ff0 2024-09-16T14:52:46.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:46.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:46.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 4 set_mon_vals no callback set 2024-09-16T14:52:46.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 1 -- 172.21.15.26:0/1335991817 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1c38001f90 con 0x7f1c40138ff0 2024-09-16T14:52:46.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:46.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3dffb640 10 monclient: dump: 2024-09-16T14:52:46.172 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.172 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.172 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.173 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.173 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.173 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.173 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.173 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.173 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.173 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 10 monclient: shutdown 2024-09-16T14:52:46.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:46.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 1 -- 172.21.15.26:0/1335991817 >> v1:172.21.15.26:6790/0 conn(0x7f1c40138ff0 legacy=0x7f1c4013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 1 -- 172.21.15.26:0/1335991817 shutdown_connections 2024-09-16T14:52:46.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 1 -- 172.21.15.26:0/1335991817 >> 172.21.15.26:0/1335991817 conn(0x7f1c4009fb50 msgr2=0x7f1c4009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:46.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 1 -- 172.21.15.26:0/1335991817 shutdown_connections 2024-09-16T14:52:46.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 1 -- 172.21.15.26:0/1335991817 wait complete. 2024-09-16T14:52:46.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 10 monclient: build_initial_monmap 2024-09-16T14:52:46.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 10 monclient: monmap: 2024-09-16T14:52:46.174 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:46.174 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:46.174 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:46.174 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:46.174 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:46.174 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:46.174 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:46.174 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:46.174 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:46.174 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 1 Processor -- start 2024-09-16T14:52:46.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 1 -- start start 2024-09-16T14:52:46.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 10 monclient: init 2024-09-16T14:52:46.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:46.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:52:46.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 10 monclient(hunting): picked mon.noname-c con 0x7f1c4009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:46.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 10 monclient(hunting): picked mon.noname-a con 0x7f1c40132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:46.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 10 monclient(hunting): picked mon.noname-b con 0x7f1c40138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:46.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4009b370 con 0x7f1c40132650 2024-09-16T14:52:46.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4009a240 con 0x7f1c40138ff0 2024-09-16T14:52:46.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c40130330 con 0x7f1c4009cb80 2024-09-16T14:52:46.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:46.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 10 monclient(hunting): authenticate will time out at 1590.290717s 2024-09-16T14:52:46.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3e7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1c40132650 0x7f1c4009ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52424/0 (socket says 172.21.15.26:52424) 2024-09-16T14:52:46.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3effd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1c4009cb80 0x7f1c4009a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38480/0 (socket says 172.21.15.26:38480) 2024-09-16T14:52:46.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3e7fc640 1 -- 172.21.15.26:0/473910904 learned_addr learned my addr 172.21.15.26:0/473910904 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:46.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 1 -- 172.21.15.26:0/473910904 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153971824 0 0) 0x7f1c40130330 con 0x7f1c4009cb80 2024-09-16T14:52:46.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 10 monclient(hunting): my global_id is 214199 2024-09-16T14:52:46.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 1 -- 172.21.15.26:0/473910904 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c20003530 con 0x7f1c4009cb80 2024-09-16T14:52:46.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 1 -- 172.21.15.26:0/473910904 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2523234030 0 0) 0x7f1c20003530 con 0x7f1c4009cb80 2024-09-16T14:52:46.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 1 -- 172.21.15.26:0/473910904 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c40130330 con 0x7f1c4009cb80 2024-09-16T14:52:46.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 1 -- 172.21.15.26:0/473910904 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1c300017a0 con 0x7f1c4009cb80 2024-09-16T14:52:46.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:52:46.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 10 monclient(hunting): dump: 2024-09-16T14:52:46.180 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.180 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.180 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.180 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.180 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.180 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.180 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.180 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.180 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.181 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:46.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 1 -- 172.21.15.26:0/473910904 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4254181818 0 0) 0x7f1c40130330 con 0x7f1c4009cb80 2024-09-16T14:52:46.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 1 -- 172.21.15.26:0/473910904 >> v1:172.21.15.189:6789/0 conn(0x7f1c40138ff0 legacy=0x7f1c401d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 1 -- 172.21.15.26:0/473910904 >> v1:172.21.15.26:6789/0 conn(0x7f1c40132650 legacy=0x7f1c4009ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 10 monclient: _finish_hunting 0 2024-09-16T14:52:46.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 1 monclient: found mon.c 2024-09-16T14:52:46.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:46.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:46.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 1 -- 172.21.15.26:0/473910904 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c40131f40 con 0x7f1c4009cb80 2024-09-16T14:52:46.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 10 monclient: _finish_auth 0 2024-09-16T14:52:46.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 10 monclient: _check_auth_tickets 2024-09-16T14:52:46.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:46.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 5 monclient: authenticate success, global_id 214199 2024-09-16T14:52:46.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 10 monclient: _renew_subs 2024-09-16T14:52:46.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:46.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 1 -- 172.21.15.26:0/473910904 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c4006dee0 con 0x7f1c4009cb80 2024-09-16T14:52:46.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 10 monclient: _renew_subs 2024-09-16T14:52:46.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:46.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c4570c640 1 -- 172.21.15.26:0/473910904 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c40138640 con 0x7f1c4009cb80 2024-09-16T14:52:46.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 1 -- 172.21.15.26:0/473910904 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c30001d50 con 0x7f1c4009cb80 2024-09-16T14:52:46.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:46.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 1 -- 172.21.15.26:0/473910904 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1c300037c0 con 0x7f1c4009cb80 2024-09-16T14:52:46.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:46.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c377fe640 10 monclient: dump: 2024-09-16T14:52:46.182 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.183 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.183 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.183 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.183 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.183 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.183 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.183 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.183 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.183 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.168+0000 7f1c3d7fa640 4 set_mon_vals no callback set 2024-09-16T14:52:46.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.172+0000 7f1c377fe640 1 -- 172.21.15.26:0/473910904 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1c30003920 con 0x7f1c4009cb80 2024-09-16T14:52:46.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.172+0000 7f1c4570c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:46.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.172+0000 7f1c4570c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:46.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.172+0000 7f1c4570c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:46.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.172+0000 7f1c4570c640 1 -- 172.21.15.26:0/473910904 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c24005050 con 0x7f1c4009cb80 2024-09-16T14:52:46.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.172+0000 7f1c377fe640 1 -- 172.21.15.26:0/473910904 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f1c30043b50 con 0x7f1c4009cb80 2024-09-16T14:52:46.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.172+0000 7f1c3e7fc640 10 monclient: get_auth_request con 0x7f1c20036150 auth_method 0 2024-09-16T14:52:46.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.176+0000 7f1c377fe640 1 -- 172.21.15.26:0/473910904 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1c300144f0 con 0x7f1c4009cb80 2024-09-16T14:52:46.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.176+0000 7f1c377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:46.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.176+0000 7f1c377fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:46.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.392+0000 7f1c4570c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:52:46.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.392+0000 7f1c4570c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:46.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.392+0000 7f1c4570c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:46.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.392+0000 7f1c4570c640 1 -- 172.21.15.26:0/473910904 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1c24005880 con 0x7f1c4009cb80 2024-09-16T14:52:46.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.396+0000 7f1c377fe640 1 -- 172.21.15.26:0/473910904 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1062902134) 0x7f1c3001cae0 con 0x7f1c4009cb80 2024-09-16T14:52:46.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.396+0000 7f1c377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:46.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.396+0000 7f1c377fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:46.396 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:52:46.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.396+0000 7f1c4570c640 1 -- 172.21.15.26:0/473910904 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1c20036150 legacy=0x7f1c20038610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.396+0000 7f1c4570c640 10 monclient: shutdown 2024-09-16T14:52:46.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.396+0000 7f1c4570c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:46.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.396+0000 7f1c4570c640 1 -- 172.21.15.26:0/473910904 >> v1:172.21.15.26:6790/0 conn(0x7f1c4009cb80 legacy=0x7f1c4009a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.396+0000 7f1c4570c640 1 -- 172.21.15.26:0/473910904 shutdown_connections 2024-09-16T14:52:46.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.396+0000 7f1c4570c640 1 -- 172.21.15.26:0/473910904 >> 172.21.15.26:0/473910904 conn(0x7f1c4009fb50 msgr2=0x7f1c4013b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:46.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.396+0000 7f1c4570c640 1 -- 172.21.15.26:0/473910904 shutdown_connections 2024-09-16T14:52:46.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.396+0000 7f1c4570c640 1 -- 172.21.15.26:0/473910904 wait complete. 2024-09-16T14:52:46.413 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:52:46.413 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:52:46.413 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:52:46.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 10 monclient: get_monmap_and_config 2024-09-16T14:52:46.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 10 monclient: build_initial_monmap 2024-09-16T14:52:46.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 10 monclient: monmap: 2024-09-16T14:52:46.537 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:46.538 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.538 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:46.538409+0000 2024-09-16T14:52:46.538 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:46.538409+0000 2024-09-16T14:52:46.538 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:46.538 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.538 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:46.538 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:46.538 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:46.538 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 1 Processor -- start 2024-09-16T14:52:46.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 1 -- start start 2024-09-16T14:52:46.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 10 monclient: init 2024-09-16T14:52:46.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:46.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:52:46.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 10 monclient(hunting): picked mon.noname-c con 0x7f1b440a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:46.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 10 monclient(hunting): picked mon.noname-a con 0x7f1b440a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:46.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 10 monclient(hunting): picked mon.noname-b con 0x7f1b440a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:46.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b440a3250 con 0x7f1b440a0000 2024-09-16T14:52:46.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4409f600 con 0x7f1b440a1d10 2024-09-16T14:52:46.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b440a3830 con 0x7f1b440a2800 2024-09-16T14:52:46.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:46.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 10 monclient(hunting): authenticate will time out at 1590.658556s 2024-09-16T14:52:46.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b41d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1b440a0000 0x7f1b44139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52440/0 (socket says 172.21.15.26:52440) 2024-09-16T14:52:46.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b41d9b640 1 -- 172.21.15.26:0/994384576 learned_addr learned my addr 172.21.15.26:0/994384576 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:46.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 1 -- 172.21.15.26:0/994384576 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669922623 0 0) 0x7f1b440a3250 con 0x7f1b440a0000 2024-09-16T14:52:46.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient(hunting): my global_id is 194337 2024-09-16T14:52:46.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 1 -- 172.21.15.26:0/994384576 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b30003530 con 0x7f1b440a0000 2024-09-16T14:52:46.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 1 -- 172.21.15.26:0/994384576 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040796174 0 0) 0x7f1b440a3830 con 0x7f1b440a2800 2024-09-16T14:52:46.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient(hunting): my global_id is 214202 2024-09-16T14:52:46.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 1 -- 172.21.15.26:0/994384576 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b440a3250 con 0x7f1b440a2800 2024-09-16T14:52:46.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 1 -- 172.21.15.26:0/994384576 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 917059677 0 0) 0x7f1b30003530 con 0x7f1b440a0000 2024-09-16T14:52:46.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 1 -- 172.21.15.26:0/994384576 >> v1:172.21.15.26:6790/0 conn(0x7f1b440a2800 legacy=0x7f1b4413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 1 -- 172.21.15.26:0/994384576 >> v1:172.21.15.189:6789/0 conn(0x7f1b440a1d10 legacy=0x7f1b440a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient: _finish_hunting 0 2024-09-16T14:52:46.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 1 monclient: found mon.noname-a 2024-09-16T14:52:46.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:46.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:52:46.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 1 -- 172.21.15.26:0/994384576 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4409ed60 con 0x7f1b440a0000 2024-09-16T14:52:46.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient: _finish_auth 0 2024-09-16T14:52:46.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient: _check_auth_tickets 2024-09-16T14:52:46.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:46.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 1 -- 172.21.15.26:0/994384576 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b2c001620 con 0x7f1b440a0000 2024-09-16T14:52:46.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:46.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient: dump: 2024-09-16T14:52:46.545 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.545 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.545 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.545 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.545 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.545 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.545 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.545 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.546 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.546 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 5 monclient: authenticate success, global_id 194337 2024-09-16T14:52:46.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:46.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 1 -- 172.21.15.26:0/994384576 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b2c001840 con 0x7f1b440a0000 2024-09-16T14:52:46.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:46.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 1 -- 172.21.15.26:0/994384576 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b2c001c10 con 0x7f1b440a0000 2024-09-16T14:52:46.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:46.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b4159a640 10 monclient: dump: 2024-09-16T14:52:46.547 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.547 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.547 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.547 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.547 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.547 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.547 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.547 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.547 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.547 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:46.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.536+0000 7f1b48f20640 4 set_mon_vals no callback set 2024-09-16T14:52:46.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 10 monclient: shutdown 2024-09-16T14:52:46.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:46.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 1 -- 172.21.15.26:0/994384576 >> v1:172.21.15.26:6789/0 conn(0x7f1b440a0000 legacy=0x7f1b44139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 1 -- 172.21.15.26:0/994384576 shutdown_connections 2024-09-16T14:52:46.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 1 -- 172.21.15.26:0/994384576 >> 172.21.15.26:0/994384576 conn(0x7f1b440a7f30 msgr2=0x7f1b440a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:46.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 1 -- 172.21.15.26:0/994384576 shutdown_connections 2024-09-16T14:52:46.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 1 -- 172.21.15.26:0/994384576 wait complete. 2024-09-16T14:52:46.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 10 monclient: build_initial_monmap 2024-09-16T14:52:46.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 10 monclient: monmap: 2024-09-16T14:52:46.548 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:46.548 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:46.548 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:46.549 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:46.549 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:46.549 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:46.549 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:46.549 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:46.549 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:46.549 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 1 Processor -- start 2024-09-16T14:52:46.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 1 -- start start 2024-09-16T14:52:46.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 10 monclient: init 2024-09-16T14:52:46.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:46.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:52:46.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 10 monclient(hunting): picked mon.noname-a con 0x7f1b440a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:46.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 10 monclient(hunting): picked mon.noname-c con 0x7f1b440a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:46.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 10 monclient(hunting): picked mon.noname-b con 0x7f1b440a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:46.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b440a6910 con 0x7f1b440a0000 2024-09-16T14:52:46.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4409e500 con 0x7f1b440a2800 2024-09-16T14:52:46.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b440a63e0 con 0x7f1b440a1d10 2024-09-16T14:52:46.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:46.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 10 monclient(hunting): authenticate will time out at 1590.661899s 2024-09-16T14:52:46.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b4259c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1b440a0000 0x7f1b44134410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52454/0 (socket says 172.21.15.26:52454) 2024-09-16T14:52:46.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b4259c640 1 -- 172.21.15.26:0/287341064 learned_addr learned my addr 172.21.15.26:0/287341064 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:46.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666759311 0 0) 0x7f1b440a63e0 con 0x7f1b440a1d10 2024-09-16T14:52:46.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient(hunting): my global_id is 214205 2024-09-16T14:52:46.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b24003530 con 0x7f1b440a1d10 2024-09-16T14:52:46.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 99492835 0 0) 0x7f1b24003530 con 0x7f1b440a1d10 2024-09-16T14:52:46.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b440a63e0 con 0x7f1b440a1d10 2024-09-16T14:52:46.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b2c0015e0 con 0x7f1b440a1d10 2024-09-16T14:52:46.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:52:46.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient(hunting): dump: 2024-09-16T14:52:46.555 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.555 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.555 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.555 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.555 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.555 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.555 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.555 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.556 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.556 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:46.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850719683 0 0) 0x7f1b440a6910 con 0x7f1b440a0000 2024-09-16T14:52:46.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient(hunting): my global_id is 194340 2024-09-16T14:52:46.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b24003530 con 0x7f1b440a0000 2024-09-16T14:52:46.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4128683570 0 0) 0x7f1b440a63e0 con 0x7f1b440a1d10 2024-09-16T14:52:46.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 >> v1:172.21.15.189:6789/0 conn(0x7f1b440a2800 legacy=0x7f1b441d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 >> v1:172.21.15.26:6789/0 conn(0x7f1b440a0000 legacy=0x7f1b44134410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient: _finish_hunting 0 2024-09-16T14:52:46.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 1 monclient: found mon.c 2024-09-16T14:52:46.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:46.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:46.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b44131c20 con 0x7f1b440a1d10 2024-09-16T14:52:46.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient: _finish_auth 0 2024-09-16T14:52:46.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient: _check_auth_tickets 2024-09-16T14:52:46.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:46.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 5 monclient: authenticate success, global_id 214205 2024-09-16T14:52:46.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 10 monclient: _renew_subs 2024-09-16T14:52:46.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:46.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 1 -- 172.21.15.26:0/287341064 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b44134bc0 con 0x7f1b440a1d10 2024-09-16T14:52:46.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 10 monclient: _renew_subs 2024-09-16T14:52:46.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:46.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b48f20640 1 -- 172.21.15.26:0/287341064 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b44133ff0 con 0x7f1b440a1d10 2024-09-16T14:52:46.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b2c004b90 con 0x7f1b440a1d10 2024-09-16T14:52:46.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:46.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b2c004d80 con 0x7f1b440a1d10 2024-09-16T14:52:46.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:46.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b40d99640 4 set_mon_vals no callback set 2024-09-16T14:52:46.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.540+0000 7f1b3affd640 10 monclient: dump: 2024-09-16T14:52:46.558 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.558 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.558 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.558 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.559 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.559 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.559 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.559 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.559 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.559 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b48f20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:46.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b48f20640 1 -- 172.21.15.26:0/287341064 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1b28006590 con 0x7f1b28002830 2024-09-16T14:52:46.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1b2c004f60 con 0x7f1b440a1d10 2024-09-16T14:52:46.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f1b2c043f80 con 0x7f1b440a1d10 2024-09-16T14:52:46.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977079536 0 0) 0x7f1b28006590 con 0x7f1b28002830 2024-09-16T14:52:46.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 10 monclient(hunting): my global_id is 194343 2024-09-16T14:52:46.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b440a63e0 con 0x7f1b28002830 2024-09-16T14:52:46.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 112733897 0 0) 0x7f1b440a63e0 con 0x7f1b28002830 2024-09-16T14:52:46.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b28006590 con 0x7f1b28002830 2024-09-16T14:52:46.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b34001780 con 0x7f1b28002830 2024-09-16T14:52:46.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b42d9d640 10 monclient: get_auth_request con 0x7f1b24036130 auth_method 0 2024-09-16T14:52:46.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:46.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 10 monclient: dump: 2024-09-16T14:52:46.561 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.561 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.561 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.561 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.561 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.561 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.561 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.561 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.561 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.562 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2157119414 0 0) 0x7f1b28006590 con 0x7f1b28002830 2024-09-16T14:52:46.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b2800a0e0 con 0x7f1b28002830 2024-09-16T14:52:46.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1b34001cb0 con 0x7f1b28002830 2024-09-16T14:52:46.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:46.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:46.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 >> v1:172.21.15.26:6789/0 conn(0x7f1b28002830 legacy=0x7f1b28004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.544+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 >> v1:172.21.15.26:6789/0 conn(0x7f1b28002830 legacy=0x7f1b28004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:46.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.548+0000 7f1b48f20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:52:46.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.548+0000 7f1b48f20640 1 -- 172.21.15.26:0/287341064 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1b28009d60 con 0x7f1b280090a0 2024-09-16T14:52:46.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.548+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620865409 0 0) 0x7f1b28009d60 con 0x7f1b280090a0 2024-09-16T14:52:46.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.548+0000 7f1b3affd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.548+0000 7f1b3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.548+0000 7f1b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.548+0000 7f1b3affd640 10 monclient(hunting): my global_id is 194346 2024-09-16T14:52:46.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.548+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b28006590 con 0x7f1b280090a0 2024-09-16T14:52:46.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 820311213 0 0) 0x7f1b28006590 con 0x7f1b280090a0 2024-09-16T14:52:46.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b28009d60 con 0x7f1b280090a0 2024-09-16T14:52:46.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b34003990 con 0x7f1b280090a0 2024-09-16T14:52:46.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:46.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b3affd640 10 monclient: dump: 2024-09-16T14:52:46.563 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.563 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.563 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.564 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.564 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.564 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.564 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.564 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.564 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.564 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 37348693 0 0) 0x7f1b28009d60 con 0x7f1b280090a0 2024-09-16T14:52:46.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1b2800c290 con 0x7f1b280090a0 2024-09-16T14:52:46.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 1906868779) 0x7f1b340057d0 con 0x7f1b280090a0 2024-09-16T14:52:46.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b3affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:52:46.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b3affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:46.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 >> v1:172.21.15.26:6789/0 conn(0x7f1b280090a0 legacy=0x7f1b28009510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b3affd640 1 -- 172.21.15.26:0/287341064 >> v1:172.21.15.26:6789/0 conn(0x7f1b280090a0 legacy=0x7f1b28009510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:46.565 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:52:46.565 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:46.565 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:46.565 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:52:46.565 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 96, 2024-09-16T14:52:46.565 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:52:46.565 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:52:46.566 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:52:46.566 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:52:46.566 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:46.566 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:52:46.566 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:46.566 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:52:46.566 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:52:46.566 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:46.566 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:46.566 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:46.566 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:46.566 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:46.566 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:46.567 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:46.567 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:46.567 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:46.567 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:46.567 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:46.567 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:52:46.567 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:52:46.567 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:46.567 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:46.567 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:46.567 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:46.567 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:46.568 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:46.568 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:46.568 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:46.568 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:46.568 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:46.568 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.568 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:52:46.568 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:52:46.568 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:52:46.568 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:52:46.568 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:52:46.568 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:52:46.569 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:46.569 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:46.569 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:52:46.569 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:52:46.569 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:52:46.569 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:52:46.569 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:52:46.569 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:52:46.569 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:52:46.569 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:46.569 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:52:46.569 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:46.569 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:46.570 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:46.570 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:46.570 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:46.570 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:46.570 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:46.570 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:46.570 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:46.570 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:46.570 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:46.570 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:52:46.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.570 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:52:46.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.571 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:46.571 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:46.571 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:46.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:46.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.571 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:46.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:52:46.571 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:46.571 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.571 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:52:46.571 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:52:46.571 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:46.572 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:46.572 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:46.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.572 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:46.572 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:46.572 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:46.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:46.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:46.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:52:46.572 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:46.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.573 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:52:46.573 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:52:46.573 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:46.573 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:46.573 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:46.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.573 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:46.573 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:46.573 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:46.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:46.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.574 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:46.574 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:52:46.574 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:46.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.574 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:52:46.574 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:52:46.574 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:46.574 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:46.574 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:46.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.575 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:52:46.575 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:52:46.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.575 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:46.575 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:46.575 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:46.575 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.575 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:46.575 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:52:46.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.575 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:46.575 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:46.575 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:52:46.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.576 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:46.576 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:52:46.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.576 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:46.576 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:46.576 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:52:46.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.576 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:46.576 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:52:46.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.576 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:46.576 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:46.577 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:46.577 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.577 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.577 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:52:46.577 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:52:46.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b48f20640 1 -- 172.21.15.26:0/287341064 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1b24036130 legacy=0x7f1b240385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b48f20640 10 monclient: shutdown 2024-09-16T14:52:46.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b48f20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:46.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b48f20640 1 -- 172.21.15.26:0/287341064 >> v1:172.21.15.26:6790/0 conn(0x7f1b440a1d10 legacy=0x7f1b441cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b42d9d640 1 -- 172.21.15.26:0/287341064 reap_dead start 2024-09-16T14:52:46.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b42d9d640 1 -- 172.21.15.26:0/287341064 reap_dead start 2024-09-16T14:52:46.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b48f20640 1 -- 172.21.15.26:0/287341064 shutdown_connections 2024-09-16T14:52:46.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b48f20640 1 -- 172.21.15.26:0/287341064 >> 172.21.15.26:0/287341064 conn(0x7f1b440a7f30 msgr2=0x7f1b4406b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:46.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b48f20640 1 -- 172.21.15.26:0/287341064 shutdown_connections 2024-09-16T14:52:46.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.552+0000 7f1b48f20640 1 -- 172.21.15.26:0/287341064 wait complete. 2024-09-16T14:52:46.579 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:52:46.579 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:52:46.579 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:52:46.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 10 monclient: get_monmap_and_config 2024-09-16T14:52:46.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 10 monclient: build_initial_monmap 2024-09-16T14:52:46.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 10 monclient: monmap: 2024-09-16T14:52:46.693 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:46.693 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.693 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:46.694787+0000 2024-09-16T14:52:46.694 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:46.694787+0000 2024-09-16T14:52:46.694 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:46.694 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.694 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:46.694 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:46.694 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:46.694 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 1 Processor -- start 2024-09-16T14:52:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 1 -- start start 2024-09-16T14:52:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 10 monclient: init 2024-09-16T14:52:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:52:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 10 monclient(hunting): picked mon.noname-c con 0x7fd64c06c5a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:46.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 10 monclient(hunting): picked mon.noname-a con 0x7fd64c0a23b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 10 monclient(hunting): picked mon.noname-b con 0x7fd64c0a27c0 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c06cee0 con 0x7fd64c0a23b0 2024-09-16T14:52:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c06c980 con 0x7fd64c06c5a0 2024-09-16T14:52:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c0a15e0 con 0x7fd64c0a27c0 2024-09-16T14:52:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 10 monclient(hunting): authenticate will time out at 1590.815016s 2024-09-16T14:52:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64bfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd64c0a27c0 0x7fd64c13d5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38516/0 (socket says 172.21.15.26:38516) 2024-09-16T14:52:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64bfff640 1 -- 172.21.15.26:0/187140516 learned_addr learned my addr 172.21.15.26:0/187140516 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd64c0a23b0 0x7fd64c139eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52490/0 (socket says 172.21.15.26:52490) 2024-09-16T14:52:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 1 -- 172.21.15.26:0/187140516 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987579433 0 0) 0x7fd64c06cee0 con 0x7fd64c0a23b0 2024-09-16T14:52:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 10 monclient(hunting): my global_id is 194349 2024-09-16T14:52:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 1 -- 172.21.15.26:0/187140516 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd634003530 con 0x7fd64c0a23b0 2024-09-16T14:52:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 1 -- 172.21.15.26:0/187140516 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042695379 0 0) 0x7fd64c0a15e0 con 0x7fd64c0a27c0 2024-09-16T14:52:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 10 monclient(hunting): my global_id is 214208 2024-09-16T14:52:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 1 -- 172.21.15.26:0/187140516 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd64c06cee0 con 0x7fd64c0a27c0 2024-09-16T14:52:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 1 -- 172.21.15.26:0/187140516 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200853879 0 0) 0x7fd64c06c980 con 0x7fd64c06c5a0 2024-09-16T14:52:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 10 monclient(hunting): my global_id is 224098 2024-09-16T14:52:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 1 -- 172.21.15.26:0/187140516 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd64c0a15e0 con 0x7fd64c06c5a0 2024-09-16T14:52:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 1 -- 172.21.15.26:0/187140516 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3940874816 0 0) 0x7fd64c06cee0 con 0x7fd64c0a27c0 2024-09-16T14:52:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 1 -- 172.21.15.26:0/187140516 >> v1:172.21.15.189:6789/0 conn(0x7fd64c06c5a0 legacy=0x7fd64c0a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 1 -- 172.21.15.26:0/187140516 >> v1:172.21.15.26:6789/0 conn(0x7fd64c0a23b0 legacy=0x7fd64c139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:52:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 1 monclient: found mon.noname-b 2024-09-16T14:52:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:52:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 1 -- 172.21.15.26:0/187140516 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c09edd0 con 0x7fd64c0a27c0 2024-09-16T14:52:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:52:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:52:46.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd64a7fc640 1 -- 172.21.15.26:0/187140516 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd644001980 con 0x7fd64c0a27c0 2024-09-16T14:52:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 5 monclient: authenticate success, global_id 214208 2024-09-16T14:52:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.692+0000 7fd651d24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd64a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd64a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:52:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd64a7fc640 10 monclient: dump: 2024-09-16T14:52:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.706 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.707 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd64a7fc640 1 -- 172.21.15.26:0/187140516 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd644001c20 con 0x7fd64c0a27c0 2024-09-16T14:52:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd64a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd64a7fc640 1 -- 172.21.15.26:0/187140516 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd644001ff0 con 0x7fd64c0a27c0 2024-09-16T14:52:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd64a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd64a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd64a7fc640 10 monclient: dump: 2024-09-16T14:52:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.708 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 4 set_mon_vals no callback set 2024-09-16T14:52:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 monclient: shutdown 2024-09-16T14:52:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 1 -- 172.21.15.26:0/187140516 >> v1:172.21.15.26:6790/0 conn(0x7fd64c0a27c0 legacy=0x7fd64c13d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 1 -- 172.21.15.26:0/187140516 shutdown_connections 2024-09-16T14:52:46.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 1 -- 172.21.15.26:0/187140516 >> 172.21.15.26:0/187140516 conn(0x7fd64c0a3da0 msgr2=0x7fd64c0a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:46.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 1 -- 172.21.15.26:0/187140516 shutdown_connections 2024-09-16T14:52:46.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 1 -- 172.21.15.26:0/187140516 wait complete. 2024-09-16T14:52:46.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 monclient: build_initial_monmap 2024-09-16T14:52:46.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 monclient: monmap: 2024-09-16T14:52:46.709 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:46.709 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:46.709 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:46.709 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:46.709 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:46.709 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:46.709 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:46.710 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:46.710 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:46.710 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 1 Processor -- start 2024-09-16T14:52:46.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 1 -- start start 2024-09-16T14:52:46.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 monclient: init 2024-09-16T14:52:46.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:46.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:52:46.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 monclient(hunting): picked mon.noname-c con 0x7fd64c06c5a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:46.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 monclient(hunting): picked mon.noname-a con 0x7fd64c0a23b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:46.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 monclient(hunting): picked mon.noname-b con 0x7fd64c0a27c0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:46.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c132460 con 0x7fd64c0a23b0 2024-09-16T14:52:46.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c133ab0 con 0x7fd64c0a27c0 2024-09-16T14:52:46.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c131430 con 0x7fd64c06c5a0 2024-09-16T14:52:46.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:46.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 monclient(hunting): authenticate will time out at 1590.818141s 2024-09-16T14:52:46.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd64affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd64c0a23b0 0x7fd64c130d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52504/0 (socket says 172.21.15.26:52504) 2024-09-16T14:52:46.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd64affd640 1 -- 172.21.15.26:0/712688726 learned_addr learned my addr 172.21.15.26:0/712688726 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:46.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd64b7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd64c06c5a0 0x7fd64c130610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38518/0 (socket says 172.21.15.26:38518) 2024-09-16T14:52:46.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421056761 0 0) 0x7fd64c131430 con 0x7fd64c06c5a0 2024-09-16T14:52:46.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): my global_id is 214211 2024-09-16T14:52:46.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd638003530 con 0x7fd64c06c5a0 2024-09-16T14:52:46.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506427213 0 0) 0x7fd64c132460 con 0x7fd64c0a23b0 2024-09-16T14:52:46.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): my global_id is 194352 2024-09-16T14:52:46.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd64c131430 con 0x7fd64c0a23b0 2024-09-16T14:52:46.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3933748284 0 0) 0x7fd638003530 con 0x7fd64c06c5a0 2024-09-16T14:52:46.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd64c132460 con 0x7fd64c06c5a0 2024-09-16T14:52:46.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1137643542 0 0) 0x7fd64c131430 con 0x7fd64c0a23b0 2024-09-16T14:52:46.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd638003530 con 0x7fd64c0a23b0 2024-09-16T14:52:46.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd640001840 con 0x7fd64c06c5a0 2024-09-16T14:52:46.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:52:46.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): dump: 2024-09-16T14:52:46.713 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.713 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.713 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.713 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.713 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.713 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.713 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.713 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.713 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.713 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:46.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd63c0012c0 con 0x7fd64c0a23b0 2024-09-16T14:52:46.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:46.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): dump: 2024-09-16T14:52:46.714 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.714 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.714 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.714 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.714 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.714 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.714 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.714 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.714 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.714 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 365883455 0 0) 0x7fd64c133ab0 con 0x7fd64c0a27c0 2024-09-16T14:52:46.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient(hunting): my global_id is 224101 2024-09-16T14:52:46.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd64c131430 con 0x7fd64c0a27c0 2024-09-16T14:52:46.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 684391351 0 0) 0x7fd64c132460 con 0x7fd64c06c5a0 2024-09-16T14:52:46.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 >> v1:172.21.15.189:6789/0 conn(0x7fd64c0a27c0 legacy=0x7fd64c1d1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 >> v1:172.21.15.26:6789/0 conn(0x7fd64c0a23b0 legacy=0x7fd64c130d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient: _finish_hunting 0 2024-09-16T14:52:46.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 monclient: found mon.c 2024-09-16T14:52:46.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:46.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:46.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c133d20 con 0x7fd64c06c5a0 2024-09-16T14:52:46.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient: _finish_auth 0 2024-09-16T14:52:46.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient: _check_auth_tickets 2024-09-16T14:52:46.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:46.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 5 monclient: authenticate success, global_id 214211 2024-09-16T14:52:46.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 monclient: _renew_subs 2024-09-16T14:52:46.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:46.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 1 -- 172.21.15.26:0/712688726 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd64c134000 con 0x7fd64c06c5a0 2024-09-16T14:52:46.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 monclient: _renew_subs 2024-09-16T14:52:46.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:46.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 1 -- 172.21.15.26:0/712688726 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd64c1358d0 con 0x7fd64c06c5a0 2024-09-16T14:52:46.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd640001a30 con 0x7fd64c06c5a0 2024-09-16T14:52:46.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:46.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd640003840 con 0x7fd64c06c5a0 2024-09-16T14:52:46.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:46.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd633fff640 10 monclient: dump: 2024-09-16T14:52:46.717 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.717 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.717 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.717 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.717 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.717 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.717 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.717 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.717 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.718 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd649ffb640 4 set_mon_vals no callback set 2024-09-16T14:52:46.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.696+0000 7fd651d24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:46.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd651d24640 1 -- 172.21.15.26:0/712688726 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd62c006590 con 0x7fd62c002830 2024-09-16T14:52:46.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd6400039a0 con 0x7fd64c06c5a0 2024-09-16T14:52:46.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fd640043ce0 con 0x7fd64c06c5a0 2024-09-16T14:52:46.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616792020 0 0) 0x7fd62c006590 con 0x7fd62c002830 2024-09-16T14:52:46.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 10 monclient(hunting): my global_id is 214214 2024-09-16T14:52:46.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd64c132460 con 0x7fd62c002830 2024-09-16T14:52:46.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd64bfff640 10 monclient: get_auth_request con 0x7fd638036440 auth_method 0 2024-09-16T14:52:46.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4228240320 0 0) 0x7fd64c132460 con 0x7fd62c002830 2024-09-16T14:52:46.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd62c006590 con 0x7fd62c002830 2024-09-16T14:52:46.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd63c004b80 con 0x7fd62c002830 2024-09-16T14:52:46.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:46.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 10 monclient: dump: 2024-09-16T14:52:46.719 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.719 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.719 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.719 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.719 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.719 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.719 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.720 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.720 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.720 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 365295680 0 0) 0x7fd62c006590 con 0x7fd62c002830 2024-09-16T14:52:46.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd62c00a0e0 con 0x7fd62c002830 2024-09-16T14:52:46.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fd63c004e20 con 0x7fd62c002830 2024-09-16T14:52:46.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:46.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:46.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 >> v1:172.21.15.26:6790/0 conn(0x7fd62c002830 legacy=0x7fd62c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.700+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 >> v1:172.21.15.26:6790/0 conn(0x7fd62c002830 legacy=0x7fd62c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:46.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.704+0000 7fd651d24640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:52:46.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.704+0000 7fd651d24640 1 -- 172.21.15.26:0/712688726 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd62c009da0 con 0x7fd62c0090c0 2024-09-16T14:52:46.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.704+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219947056 0 0) 0x7fd62c009da0 con 0x7fd62c0090c0 2024-09-16T14:52:46.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.704+0000 7fd633fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.704+0000 7fd633fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.704+0000 7fd633fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.704+0000 7fd633fff640 10 monclient(hunting): my global_id is 214217 2024-09-16T14:52:46.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.704+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd62c006590 con 0x7fd62c0090c0 2024-09-16T14:52:46.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2215371982 0 0) 0x7fd62c006590 con 0x7fd62c0090c0 2024-09-16T14:52:46.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd62c009da0 con 0x7fd62c0090c0 2024-09-16T14:52:46.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd63c001d60 con 0x7fd62c0090c0 2024-09-16T14:52:46.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd633fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd633fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:46.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd633fff640 10 monclient: dump: 2024-09-16T14:52:46.723 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.723 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.723 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.723 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.723 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.723 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.724 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.724 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.724 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.724 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3934208006 0 0) 0x7fd62c009da0 con 0x7fd62c0090c0 2024-09-16T14:52:46.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd62c00c290 con 0x7fd62c0090c0 2024-09-16T14:52:46.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3881 (unknown 0 0 3669415168) 0x7fd63c006760 con 0x7fd62c0090c0 2024-09-16T14:52:46.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd633fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:52:46.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd633fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:46.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 >> v1:172.21.15.26:6790/0 conn(0x7fd62c0090c0 legacy=0x7fd62c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd633fff640 1 -- 172.21.15.26:0/712688726 >> v1:172.21.15.26:6790/0 conn(0x7fd62c0090c0 legacy=0x7fd62c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:46.725 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:52:46.725 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:46.725 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:46.725 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:52:46.725 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 96, 2024-09-16T14:52:46.725 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:52:46.725 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:52:46.725 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:52:46.725 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:52:46.725 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:46.725 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:52:46.725 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:46.726 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:52:46.726 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:52:46.726 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:46.726 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:46.726 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:46.726 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:46.726 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:46.726 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:46.726 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:46.726 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:46.726 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:46.726 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:46.726 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:46.726 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:52:46.727 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:52:46.727 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:46.727 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:46.727 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:46.727 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:46.727 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:46.727 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:46.727 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:46.727 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:46.727 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:46.727 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:46.727 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.728 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:52:46.728 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:52:46.728 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:52:46.728 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:52:46.728 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:52:46.728 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:52:46.728 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:46.728 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:46.728 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:52:46.728 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:52:46.728 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:52:46.728 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:52:46.729 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:52:46.729 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:52:46.729 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:52:46.729 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:46.729 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:52:46.729 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:46.729 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:46.729 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:46.729 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:46.729 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:46.729 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:46.729 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:46.729 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:46.730 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:46.730 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:46.730 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:46.730 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:52:46.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.730 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:52:46.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.730 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:46.730 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:46.730 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:46.730 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:46.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.730 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:46.730 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:52:46.730 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:46.731 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.731 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:52:46.731 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:52:46.731 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:46.731 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:46.731 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:46.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.731 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:46.731 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:46.731 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:46.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:46.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.732 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:46.732 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:52:46.732 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:46.732 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.732 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.732 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:52:46.732 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:52:46.732 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:46.732 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:46.732 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:46.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.732 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:46.733 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:46.733 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:46.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:46.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.733 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:46.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:52:46.733 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:46.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.733 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:52:46.733 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:52:46.733 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:46.734 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:46.734 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:46.734 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.734 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.734 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:52:46.734 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:52:46.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.734 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:46.734 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:46.734 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:46.734 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.734 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:46.734 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:52:46.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.735 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:46.735 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:46.735 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:52:46.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.735 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.735 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:52:46.735 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:52:46.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd651d24640 1 -- 172.21.15.26:0/712688726 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd638036440 legacy=0x7fd638038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd651d24640 10 monclient: shutdown 2024-09-16T14:52:46.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd651d24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:46.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd651d24640 1 -- 172.21.15.26:0/712688726 >> v1:172.21.15.26:6790/0 conn(0x7fd64c06c5a0 legacy=0x7fd64c130610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd64bfff640 1 -- 172.21.15.26:0/712688726 reap_dead start 2024-09-16T14:52:46.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd64bfff640 1 -- 172.21.15.26:0/712688726 reap_dead start 2024-09-16T14:52:46.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd651d24640 1 -- 172.21.15.26:0/712688726 shutdown_connections 2024-09-16T14:52:46.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd651d24640 1 -- 172.21.15.26:0/712688726 >> 172.21.15.26:0/712688726 conn(0x7fd64c0a3da0 msgr2=0x7fd64c0a51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:46.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd651d24640 1 -- 172.21.15.26:0/712688726 shutdown_connections 2024-09-16T14:52:46.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.708+0000 7fd651d24640 1 -- 172.21.15.26:0/712688726 wait complete. 2024-09-16T14:52:46.737 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:52:46.737 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:52:46.738 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:52:46.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 10 monclient: get_monmap_and_config 2024-09-16T14:52:46.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 10 monclient: build_initial_monmap 2024-09-16T14:52:46.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 10 monclient: monmap: 2024-09-16T14:52:46.852 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:46.852 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.852 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:46.853660+0000 2024-09-16T14:52:46.853 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:46.853660+0000 2024-09-16T14:52:46.853 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:46.853 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.853 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:46.853 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:46.853 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:46.853 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 1 Processor -- start 2024-09-16T14:52:46.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 1 -- start start 2024-09-16T14:52:46.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 10 monclient: init 2024-09-16T14:52:46.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:46.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:52:46.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 10 monclient(hunting): picked mon.noname-b con 0x7f671c0a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:46.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 10 monclient(hunting): picked mon.noname-c con 0x7f671c099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:46.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 10 monclient(hunting): picked mon.noname-a con 0x7f671c0a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:46.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c098ea0 con 0x7f671c0a26f0 2024-09-16T14:52:46.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c0997b0 con 0x7f671c099b60 2024-09-16T14:52:46.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c099460 con 0x7f671c0a1c00 2024-09-16T14:52:46.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:46.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 10 monclient(hunting): authenticate will time out at 1590.973658s 2024-09-16T14:52:46.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67228a8640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f671c0a26f0 0x7f671c13d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52512/0 (socket says 172.21.15.26:52512) 2024-09-16T14:52:46.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67228a8640 1 -- 172.21.15.26:0/3130025099 learned_addr learned my addr 172.21.15.26:0/3130025099 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:46.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 1 -- 172.21.15.26:0/3130025099 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727510136 0 0) 0x7f671c098ea0 con 0x7f671c0a26f0 2024-09-16T14:52:46.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient(hunting): my global_id is 194355 2024-09-16T14:52:46.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 1 -- 172.21.15.26:0/3130025099 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6708003530 con 0x7f671c0a26f0 2024-09-16T14:52:46.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 1 -- 172.21.15.26:0/3130025099 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898855914 0 0) 0x7f671c099460 con 0x7f671c0a1c00 2024-09-16T14:52:46.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient(hunting): my global_id is 214220 2024-09-16T14:52:46.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 1 -- 172.21.15.26:0/3130025099 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f671c098ea0 con 0x7f671c0a1c00 2024-09-16T14:52:46.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 1 -- 172.21.15.26:0/3130025099 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 994498615 0 0) 0x7f6708003530 con 0x7f671c0a26f0 2024-09-16T14:52:46.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 1 -- 172.21.15.26:0/3130025099 >> v1:172.21.15.26:6790/0 conn(0x7f671c0a1c00 legacy=0x7f671c0a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 1 -- 172.21.15.26:0/3130025099 >> v1:172.21.15.189:6789/0 conn(0x7f671c099b60 legacy=0x7f671c139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient: _finish_hunting 0 2024-09-16T14:52:46.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 1 monclient: found mon.noname-a 2024-09-16T14:52:46.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:46.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:52:46.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 1 -- 172.21.15.26:0/3130025099 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c0a34b0 con 0x7f671c0a26f0 2024-09-16T14:52:46.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient: _finish_auth 0 2024-09-16T14:52:46.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient: _check_auth_tickets 2024-09-16T14:52:46.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:46.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 1 -- 172.21.15.26:0/3130025099 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6714001620 con 0x7f671c0a26f0 2024-09-16T14:52:46.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:46.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient: dump: 2024-09-16T14:52:46.863 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.864 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.864 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.864 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.864 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.864 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.864 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.864 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.864 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.864 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 5 monclient: authenticate success, global_id 194355 2024-09-16T14:52:46.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:46.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 1 -- 172.21.15.26:0/3130025099 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6714001880 con 0x7f671c0a26f0 2024-09-16T14:52:46.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:46.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 1 -- 172.21.15.26:0/3130025099 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6714001c50 con 0x7f671c0a26f0 2024-09-16T14:52:46.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:46.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f67210a5640 10 monclient: dump: 2024-09-16T14:52:46.865 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.865 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.865 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.865 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.865 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.865 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.865 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.866 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.866 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.866 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:46.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 4 set_mon_vals no callback set 2024-09-16T14:52:46.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 10 monclient: shutdown 2024-09-16T14:52:46.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:46.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 1 -- 172.21.15.26:0/3130025099 >> v1:172.21.15.26:6789/0 conn(0x7f671c0a26f0 legacy=0x7f671c13d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 1 -- 172.21.15.26:0/3130025099 shutdown_connections 2024-09-16T14:52:46.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 1 -- 172.21.15.26:0/3130025099 >> 172.21.15.26:0/3130025099 conn(0x7f671c0a3d00 msgr2=0x7f671c0a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:46.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 1 -- 172.21.15.26:0/3130025099 shutdown_connections 2024-09-16T14:52:46.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.852+0000 7f672430b640 1 -- 172.21.15.26:0/3130025099 wait complete. 2024-09-16T14:52:46.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 10 monclient: build_initial_monmap 2024-09-16T14:52:46.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 10 monclient: monmap: 2024-09-16T14:52:46.867 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:46.867 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:46.867 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:46.867 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:46.867 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:46.867 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:46.867 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:46.867 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:46.867 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:46.867 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 1 Processor -- start 2024-09-16T14:52:46.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 1 -- start start 2024-09-16T14:52:46.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 10 monclient: init 2024-09-16T14:52:46.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:46.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:52:46.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 10 monclient(hunting): picked mon.noname-a con 0x7f671c099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:46.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 10 monclient(hunting): picked mon.noname-b con 0x7f671c0a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:46.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 10 monclient(hunting): picked mon.noname-c con 0x7f671c0a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:46.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c13ca90 con 0x7f671c099b60 2024-09-16T14:52:46.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c09efc0 con 0x7f671c0a1c00 2024-09-16T14:52:46.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c09ea90 con 0x7f671c0a26f0 2024-09-16T14:52:46.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:46.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 10 monclient(hunting): authenticate will time out at 1590.976917s 2024-09-16T14:52:46.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f67220a7640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f671c099b60 0x7f671c1c9bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52522/0 (socket says 172.21.15.26:52522) 2024-09-16T14:52:46.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f67220a7640 1 -- 172.21.15.26:0/2831995395 learned_addr learned my addr 172.21.15.26:0/2831995395 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:46.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983215403 0 0) 0x7f671c13ca90 con 0x7f671c099b60 2024-09-16T14:52:46.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): my global_id is 194358 2024-09-16T14:52:46.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6700003530 con 0x7f671c099b60 2024-09-16T14:52:46.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678194660 0 0) 0x7f671c09ea90 con 0x7f671c0a26f0 2024-09-16T14:52:46.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): my global_id is 214223 2024-09-16T14:52:46.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f671c13ca90 con 0x7f671c0a26f0 2024-09-16T14:52:46.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2996759936 0 0) 0x7f6700003530 con 0x7f671c099b60 2024-09-16T14:52:46.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f671c09ea90 con 0x7f671c099b60 2024-09-16T14:52:46.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6718001d60 con 0x7f671c099b60 2024-09-16T14:52:46.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:46.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): dump: 2024-09-16T14:52:46.870 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.870 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.870 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.870 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.870 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.870 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.871 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.871 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.871 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.871 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:46.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569337052 0 0) 0x7f671c09efc0 con 0x7f671c0a1c00 2024-09-16T14:52:46.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): my global_id is 224107 2024-09-16T14:52:46.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6700003530 con 0x7f671c0a1c00 2024-09-16T14:52:46.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4135199740 0 0) 0x7f671c13ca90 con 0x7f671c0a26f0 2024-09-16T14:52:46.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f671c09efc0 con 0x7f671c0a26f0 2024-09-16T14:52:46.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67140019f0 con 0x7f671c0a26f0 2024-09-16T14:52:46.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:46.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): dump: 2024-09-16T14:52:46.872 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.872 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.872 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.872 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.872 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.872 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.872 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.872 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.872 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.873 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3159870867 0 0) 0x7f671c09ea90 con 0x7f671c099b60 2024-09-16T14:52:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 >> v1:172.21.15.26:6790/0 conn(0x7f671c0a26f0 legacy=0x7f671c1d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 >> v1:172.21.15.189:6789/0 conn(0x7f671c0a1c00 legacy=0x7f671c1cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:52:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 monclient: found mon.a 2024-09-16T14:52:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c13e1b0 con 0x7f671c099b60 2024-09-16T14:52:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient: _finish_auth 0 2024-09-16T14:52:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:52:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 5 monclient: authenticate success, global_id 194358 2024-09-16T14:52:46.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 10 monclient: _renew_subs 2024-09-16T14:52:46.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:46.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 1 -- 172.21.15.26:0/2831995395 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671c13c7a0 con 0x7f671c099b60 2024-09-16T14:52:46.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 10 monclient: _renew_subs 2024-09-16T14:52:46.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:46.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 1 -- 172.21.15.26:0/2831995395 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f671c13be50 con 0x7f671c099b60 2024-09-16T14:52:46.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6718002010 con 0x7f671c099b60 2024-09-16T14:52:46.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:46.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6718003dd0 con 0x7f671c099b60 2024-09-16T14:52:46.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:46.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f67208a4640 4 set_mon_vals no callback set 2024-09-16T14:52:46.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient: dump: 2024-09-16T14:52:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.875 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f672430b640 1 -- 172.21.15.26:0/2831995395 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6704006590 con 0x7f6704002830 2024-09-16T14:52:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6718003f30 con 0x7f671c099b60 2024-09-16T14:52:46.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f6718044190 con 0x7f671c099b60 2024-09-16T14:52:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026506229 0 0) 0x7f6704006590 con 0x7f6704002830 2024-09-16T14:52:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient(hunting): my global_id is 224110 2024-09-16T14:52:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f671c09ea90 con 0x7f6704002830 2024-09-16T14:52:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f67228a8640 10 monclient: get_auth_request con 0x7f6700036440 auth_method 0 2024-09-16T14:52:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2861751908 0 0) 0x7f671c09ea90 con 0x7f6704002830 2024-09-16T14:52:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6704006590 con 0x7f6704002830 2024-09-16T14:52:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f670c001750 con 0x7f6704002830 2024-09-16T14:52:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:52:46.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.856+0000 7f6712ffd640 10 monclient: dump: 2024-09-16T14:52:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.877 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.860+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1505775922 0 0) 0x7f6704006590 con 0x7f6704002830 2024-09-16T14:52:46.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.860+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f670400a0e0 con 0x7f6704002830 2024-09-16T14:52:46.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.860+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f670c001b10 con 0x7f6704002830 2024-09-16T14:52:46.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.860+0000 7f6712ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:46.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.860+0000 7f6712ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:46.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.860+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 >> v1:172.21.15.189:6789/0 conn(0x7f6704002830 legacy=0x7f6704004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.860+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 >> v1:172.21.15.189:6789/0 conn(0x7f6704002830 legacy=0x7f6704004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:46.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.864+0000 7f672430b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:52:46.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.864+0000 7f672430b640 1 -- 172.21.15.26:0/2831995395 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6704009da0 con 0x7f67040090c0 2024-09-16T14:52:46.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177933534 0 0) 0x7f6704009da0 con 0x7f67040090c0 2024-09-16T14:52:46.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:46.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:46.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:46.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 10 monclient(hunting): my global_id is 224113 2024-09-16T14:52:46.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6704006590 con 0x7f67040090c0 2024-09-16T14:52:46.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1495723492 0 0) 0x7f6704006590 con 0x7f67040090c0 2024-09-16T14:52:46.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6704009da0 con 0x7f67040090c0 2024-09-16T14:52:46.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f670c001d90 con 0x7f67040090c0 2024-09-16T14:52:46.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:46.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:52:46.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 10 monclient: dump: 2024-09-16T14:52:46.880 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:46.880 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:46.880 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.880 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:46.880 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:46.880 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:46.880 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:46.881 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:46.881 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:46.881 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:46.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4054371519 0 0) 0x7f6704009da0 con 0x7f67040090c0 2024-09-16T14:52:46.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f670400c290 con 0x7f67040090c0 2024-09-16T14:52:46.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 734933688) 0x7f670c003b20 con 0x7f67040090c0 2024-09-16T14:52:46.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:52:46.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:46.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 >> v1:172.21.15.189:6789/0 conn(0x7f67040090c0 legacy=0x7f6704009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f6712ffd640 1 -- 172.21.15.26:0/2831995395 >> v1:172.21.15.189:6789/0 conn(0x7f67040090c0 legacy=0x7f6704009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:46.881 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:52:46.882 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:46.882 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:46.882 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:52:46.882 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 96, 2024-09-16T14:52:46.882 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:52:46.882 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:52:46.882 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:52:46.882 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:52:46.882 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:46.882 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:52:46.882 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:46.882 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:52:46.883 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:52:46.883 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:46.883 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:46.883 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:46.883 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:46.883 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:46.883 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:46.883 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:46.883 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:46.883 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:46.883 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:46.883 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:46.884 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:52:46.884 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:52:46.884 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:46.884 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:46.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:46.884 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:46.884 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:46.884 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:46.884 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:46.884 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:46.884 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:46.884 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:46.884 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.885 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:52:46.885 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:52:46.885 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:52:46.885 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:52:46.885 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:52:46.885 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:52:46.885 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:46.885 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:46.885 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:52:46.885 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:52:46.885 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:52:46.885 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:52:46.885 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:52:46.886 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:52:46.886 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:52:46.886 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:46.886 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:52:46.886 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:46.886 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:46.886 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:46.886 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:46.886 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:46.886 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:46.886 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:46.886 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:46.886 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:46.887 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:46.887 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:46.887 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:52:46.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:52:46.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.887 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:46.887 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:46.887 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:46.887 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:46.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.887 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:46.887 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:52:46.888 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:46.888 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.888 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:52:46.888 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:52:46.888 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:46.888 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:46.888 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:46.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.888 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:46.888 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:46.889 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:46.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:46.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:46.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:52:46.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:46.889 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.889 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:52:46.889 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:52:46.889 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:46.889 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:52:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:52:46.891 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:52:46.891 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:46.891 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:46.891 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:46.891 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.891 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.891 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:52:46.891 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:52:46.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.891 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:46.891 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:46.892 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:46.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.892 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:46.892 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:52:46.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:46.892 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:46.892 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:46.892 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:46.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:46.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:46.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:46.892 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:52:46.892 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:52:46.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f672430b640 1 -- 172.21.15.26:0/2831995395 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6700036440 legacy=0x7f6700038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f672430b640 10 monclient: shutdown 2024-09-16T14:52:46.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f672430b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:46.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f672430b640 1 -- 172.21.15.26:0/2831995395 >> v1:172.21.15.26:6789/0 conn(0x7f671c099b60 legacy=0x7f671c1c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:46.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f67228a8640 1 -- 172.21.15.26:0/2831995395 reap_dead start 2024-09-16T14:52:46.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f67228a8640 1 -- 172.21.15.26:0/2831995395 reap_dead start 2024-09-16T14:52:46.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f672430b640 1 -- 172.21.15.26:0/2831995395 shutdown_connections 2024-09-16T14:52:46.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f672430b640 1 -- 172.21.15.26:0/2831995395 >> 172.21.15.26:0/2831995395 conn(0x7f671c0a3d00 msgr2=0x7f671c0a0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:46.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f672430b640 1 -- 172.21.15.26:0/2831995395 shutdown_connections 2024-09-16T14:52:46.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:46.868+0000 7f672430b640 1 -- 172.21.15.26:0/2831995395 wait complete. 2024-09-16T14:52:46.895 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:52:46.895 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:52:46.895 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:52:47.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.012+0000 7f4974088640 10 monclient: get_monmap_and_config 2024-09-16T14:52:47.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.012+0000 7f4974088640 10 monclient: build_initial_monmap 2024-09-16T14:52:47.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.012+0000 7f4974088640 10 monclient: monmap: 2024-09-16T14:52:47.015 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:47.015 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:47.015 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:47.016832+0000 2024-09-16T14:52:47.016 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:47.016832+0000 2024-09-16T14:52:47.016 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:47.016 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:47.016 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:47.016 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:47.016 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:47.016 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:47.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 1 Processor -- start 2024-09-16T14:52:47.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 1 -- start start 2024-09-16T14:52:47.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient: init 2024-09-16T14:52:47.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:47.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:52:47.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient(hunting): picked mon.noname-c con 0x7f496c06c5a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:47.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient(hunting): picked mon.noname-b con 0x7f496c0a23b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:47.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient(hunting): picked mon.noname-a con 0x7f496c0a27c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:47.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c06cee0 con 0x7f496c0a27c0 2024-09-16T14:52:47.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c06c980 con 0x7f496c06c5a0 2024-09-16T14:52:47.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c0a15e0 con 0x7f496c0a23b0 2024-09-16T14:52:47.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:47.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient(hunting): authenticate will time out at 1591.136882s 2024-09-16T14:52:47.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4971623640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f496c0a23b0 0x7f496c139eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54416/0 (socket says 172.21.15.26:54416) 2024-09-16T14:52:47.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4971623640 1 -- 172.21.15.26:0/2496367023 learned_addr learned my addr 172.21.15.26:0/2496367023 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:47.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 1 -- 172.21.15.26:0/2496367023 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827002417 0 0) 0x7f496c0a15e0 con 0x7f496c0a23b0 2024-09-16T14:52:47.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient(hunting): _negotiate 2024-09-16T14:52:47.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:47.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:47.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient(hunting): my global_id is 214226 2024-09-16T14:52:47.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 1 -- 172.21.15.26:0/2496367023 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4958003530 con 0x7f496c0a23b0 2024-09-16T14:52:47.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 1 -- 172.21.15.26:0/2496367023 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184715777 0 0) 0x7f496c06cee0 con 0x7f496c0a27c0 2024-09-16T14:52:47.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient(hunting): _negotiate 2024-09-16T14:52:47.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:47.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:47.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient(hunting): my global_id is 194361 2024-09-16T14:52:47.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 1 -- 172.21.15.26:0/2496367023 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f496c0a15e0 con 0x7f496c0a27c0 2024-09-16T14:52:47.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 1 -- 172.21.15.26:0/2496367023 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1947781899 0 0) 0x7f4958003530 con 0x7f496c0a23b0 2024-09-16T14:52:47.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 1 -- 172.21.15.26:0/2496367023 >> v1:172.21.15.189:6789/0 conn(0x7f496c06c5a0 legacy=0x7f496c0a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:47.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 1 -- 172.21.15.26:0/2496367023 >> v1:172.21.15.26:6789/0 conn(0x7f496c0a27c0 legacy=0x7f496c13d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:47.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient: _finish_hunting 0 2024-09-16T14:52:47.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 1 monclient: found mon.noname-b 2024-09-16T14:52:47.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:47.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:52:47.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 1 -- 172.21.15.26:0/2496367023 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496c09edd0 con 0x7f496c0a23b0 2024-09-16T14:52:47.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient: _finish_auth 0 2024-09-16T14:52:47.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient: _check_auth_tickets 2024-09-16T14:52:47.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:47.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 1 -- 172.21.15.26:0/2496367023 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4960001620 con 0x7f496c0a23b0 2024-09-16T14:52:47.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:47.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:52:47.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient: dump: 2024-09-16T14:52:47.022 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:47.022 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:47.022 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:47.022 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:47.022 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:47.023 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:47.023 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:47.023 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:47.023 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:47.023 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:47.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 5 monclient: authenticate success, global_id 214226 2024-09-16T14:52:47.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:47.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 1 -- 172.21.15.26:0/2496367023 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49600018c0 con 0x7f496c0a23b0 2024-09-16T14:52:47.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:47.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 1 -- 172.21.15.26:0/2496367023 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4960001c50 con 0x7f496c0a23b0 2024-09-16T14:52:47.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:47.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:47.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4970e22640 10 monclient: dump: 2024-09-16T14:52:47.024 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:47.024 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:47.024 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:47.024 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:47.024 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:47.024 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:47.024 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:47.024 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:47.024 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:47.024 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:47.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:47.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 4 set_mon_vals no callback set 2024-09-16T14:52:47.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient: shutdown 2024-09-16T14:52:47.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:47.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 1 -- 172.21.15.26:0/2496367023 >> v1:172.21.15.26:6790/0 conn(0x7f496c0a23b0 legacy=0x7f496c139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:47.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 1 -- 172.21.15.26:0/2496367023 shutdown_connections 2024-09-16T14:52:47.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 1 -- 172.21.15.26:0/2496367023 >> 172.21.15.26:0/2496367023 conn(0x7f496c0a3da0 msgr2=0x7f496c0a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:47.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 1 -- 172.21.15.26:0/2496367023 shutdown_connections 2024-09-16T14:52:47.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 1 -- 172.21.15.26:0/2496367023 wait complete. 2024-09-16T14:52:47.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient: build_initial_monmap 2024-09-16T14:52:47.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient: monmap: 2024-09-16T14:52:47.025 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:47.025 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:47.025 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:47.025 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:47.025 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:47.025 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:47.026 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:47.026 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:47.026 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:47.026 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:47.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 1 Processor -- start 2024-09-16T14:52:47.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 1 -- start start 2024-09-16T14:52:47.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient: init 2024-09-16T14:52:47.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:47.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:52:47.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient(hunting): picked mon.noname-b con 0x7f496c06c5a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:47.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient(hunting): picked mon.noname-c con 0x7f496c0a23b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:47.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient(hunting): picked mon.noname-a con 0x7f496c0a27c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:47.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c09e490 con 0x7f496c0a27c0 2024-09-16T14:52:47.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c1328d0 con 0x7f496c06c5a0 2024-09-16T14:52:47.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c133530 con 0x7f496c0a23b0 2024-09-16T14:52:47.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.016+0000 7f4974088640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:47.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f4974088640 10 monclient(hunting): authenticate will time out at 1591.140010s 2024-09-16T14:52:47.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f4971623640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f496c0a23b0 0x7f496c1ce660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54428/0 (socket says 172.21.15.26:54428) 2024-09-16T14:52:47.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f4971623640 1 -- 172.21.15.26:0/3610307517 learned_addr learned my addr 172.21.15.26:0/3610307517 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:47.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38569947 0 0) 0x7f496c133530 con 0x7f496c0a23b0 2024-09-16T14:52:47.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:47.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): my global_id is 214229 2024-09-16T14:52:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4950003530 con 0x7f496c0a23b0 2024-09-16T14:52:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198024081 0 0) 0x7f496c09e490 con 0x7f496c0a27c0 2024-09-16T14:52:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): my global_id is 194364 2024-09-16T14:52:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f496c133530 con 0x7f496c0a27c0 2024-09-16T14:52:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3295396930 0 0) 0x7f4950003530 con 0x7f496c0a23b0 2024-09-16T14:52:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f496c09e490 con 0x7f496c0a23b0 2024-09-16T14:52:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4960001bd0 con 0x7f496c0a23b0 2024-09-16T14:52:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:52:47.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): dump: 2024-09-16T14:52:47.032 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:47.032 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:47.032 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:47.032 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:47.032 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:47.032 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:47.032 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:47.032 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:47.032 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:47.032 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:47.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:47.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 957136467 0 0) 0x7f496c133530 con 0x7f496c0a27c0 2024-09-16T14:52:47.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4950003530 con 0x7f496c0a27c0 2024-09-16T14:52:47.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4968001860 con 0x7f496c0a27c0 2024-09-16T14:52:47.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:47.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:47.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): dump: 2024-09-16T14:52:47.033 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:47.033 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:47.033 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:47.033 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:47.033 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:47.033 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:47.033 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:47.033 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:47.033 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:47.033 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:47.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910564780 0 0) 0x7f496c1328d0 con 0x7f496c06c5a0 2024-09-16T14:52:47.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:47.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:47.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:47.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient(hunting): my global_id is 224119 2024-09-16T14:52:47.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f496c133530 con 0x7f496c06c5a0 2024-09-16T14:52:47.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 116552897 0 0) 0x7f496c09e490 con 0x7f496c0a23b0 2024-09-16T14:52:47.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 >> v1:172.21.15.189:6789/0 conn(0x7f496c06c5a0 legacy=0x7f496c135300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:47.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 >> v1:172.21.15.26:6789/0 conn(0x7f496c0a27c0 legacy=0x7f496c1d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:47.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:52:47.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 monclient: found mon.c 2024-09-16T14:52:47.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496c133810 con 0x7f496c0a23b0 2024-09-16T14:52:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:52:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:52:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f4974088640 5 monclient: authenticate success, global_id 214229 2024-09-16T14:52:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f4974088640 10 monclient: _renew_subs 2024-09-16T14:52:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f4974088640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f4974088640 1 -- 172.21.15.26:0/3610307517 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496c135a10 con 0x7f496c0a23b0 2024-09-16T14:52:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f4974088640 10 monclient: _renew_subs 2024-09-16T14:52:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f4974088640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f4974088640 1 -- 172.21.15.26:0/3610307517 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f496c131ab0 con 0x7f496c0a23b0 2024-09-16T14:52:47.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4960004cf0 con 0x7f496c0a23b0 2024-09-16T14:52:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4960005080 con 0x7f496c0a23b0 2024-09-16T14:52:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495ffff640 4 set_mon_vals no callback set 2024-09-16T14:52:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 10 monclient: dump: 2024-09-16T14:52:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:47.036 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:47.037 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4960005260 con 0x7f496c0a23b0 2024-09-16T14:52:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f4974088640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f4974088640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f4974088640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f4974088640 1 -- 172.21.15.26:0/3610307517 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4954005050 con 0x7f496c0a23b0 2024-09-16T14:52:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f4960044120 con 0x7f496c0a23b0 2024-09-16T14:52:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.020+0000 7f4971e24640 10 monclient: get_auth_request con 0x7f49500363b0 auth_method 0 2024-09-16T14:52:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.024+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f49600148f0 con 0x7f496c0a23b0 2024-09-16T14:52:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.024+0000 7f495e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:47.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.024+0000 7f495e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:47.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.240+0000 7f4974088640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:52:47.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.240+0000 7f4974088640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:52:47.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.240+0000 7f4974088640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:47.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.240+0000 7f4974088640 1 -- 172.21.15.26:0/3610307517 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f4954005b10 con 0x7f496c0a23b0 2024-09-16T14:52:47.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.244+0000 7f495e7fc640 1 -- 172.21.15.26:0/3610307517 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f4960012f90 con 0x7f496c0a23b0 2024-09-16T14:52:47.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.244+0000 7f495e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:52:47.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.244+0000 7f495e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:47.245 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.244+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.245 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.244+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=1,health=11,kv=10,logm=53} crc {auth=2301282090,config=1228700967,health=2897149690,kv=3455456366,logm=2212053753}) 2024-09-16T14:52:47.245 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.244+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=272167707,config=692619927,health=1157367994,kv=2418557040,logm=4193171647}) 2024-09-16T14:52:47.245 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.244+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.245 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.244+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=1,health=11,kv=10,logm=53} crc {auth=2301282090,config=1228700967,health=2897149690,kv=3455456366,logm=2212053753}) 2024-09-16T14:52:47.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.244+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=1,health=11,kv=8,logm=53} crc {auth=4237327148,config=1228700967,health=4222337298,kv=3563107213,logm=2675732175}) 2024-09-16T14:52:47.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.244+0000 7f4974088640 1 -- 172.21.15.26:0/3610307517 >> v1:172.21.15.189:6812/3196161582 conn(0x7f49500363b0 legacy=0x7f4950038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:47.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.244+0000 7f4974088640 10 monclient: shutdown 2024-09-16T14:52:47.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.244+0000 7f4974088640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:47.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.244+0000 7f4974088640 1 -- 172.21.15.26:0/3610307517 >> v1:172.21.15.26:6790/0 conn(0x7f496c0a23b0 legacy=0x7f496c1ce660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:47.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.244+0000 7f4974088640 1 -- 172.21.15.26:0/3610307517 shutdown_connections 2024-09-16T14:52:47.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.244+0000 7f4974088640 1 -- 172.21.15.26:0/3610307517 >> 172.21.15.26:0/3610307517 conn(0x7f496c0a3da0 msgr2=0x7f496c0a51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:47.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.244+0000 7f4974088640 1 -- 172.21.15.26:0/3610307517 shutdown_connections 2024-09-16T14:52:47.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:47.244+0000 7f4974088640 1 -- 172.21.15.26:0/3610307517 wait complete. 2024-09-16T14:52:47.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.248+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.248+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2845863698}) 2024-09-16T14:52:47.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.248+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2228694885}) 2024-09-16T14:52:47.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.248+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.248+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2845863698}) 2024-09-16T14:52:47.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.248+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1235973458}) 2024-09-16T14:52:47.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.248+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.248+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3813944653}) 2024-09-16T14:52:47.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.248+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3018985144}) 2024-09-16T14:52:47.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.248+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.248+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3813944653}) 2024-09-16T14:52:47.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.248+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1210033825}) 2024-09-16T14:52:47.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.252+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.252+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2703188914}) 2024-09-16T14:52:47.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.252+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=355709505}) 2024-09-16T14:52:47.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.252+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.252+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2703188914}) 2024-09-16T14:52:47.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.252+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2386222492}) 2024-09-16T14:52:47.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3363817486}) 2024-09-16T14:52:47.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612909993}) 2024-09-16T14:52:47.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3363817486}) 2024-09-16T14:52:47.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2371310739}) 2024-09-16T14:52:47.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3280430732}) 2024-09-16T14:52:47.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3008390678}) 2024-09-16T14:52:47.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3280430732}) 2024-09-16T14:52:47.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3995239029}) 2024-09-16T14:52:47.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=947647437}) 2024-09-16T14:52:47.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848389428}) 2024-09-16T14:52:47.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=947647437}) 2024-09-16T14:52:47.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.255+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3936321736}) 2024-09-16T14:52:47.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.259+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.259+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507290501}) 2024-09-16T14:52:47.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.259+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1762072744}) 2024-09-16T14:52:47.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.259+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.259+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507290501}) 2024-09-16T14:52:47.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.259+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2158948170}) 2024-09-16T14:52:47.263 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.259+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.263 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.259+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1476225680}) 2024-09-16T14:52:47.263 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.259+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1015037456}) 2024-09-16T14:52:47.263 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.259+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.259+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1476225680}) 2024-09-16T14:52:47.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.259+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2154294510}) 2024-09-16T14:52:47.265 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:52:47.265 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.263+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.265 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.263+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2531153290}) 2024-09-16T14:52:47.265 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.263+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=722690464}) 2024-09-16T14:52:47.265 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.263+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.265 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.263+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2531153290}) 2024-09-16T14:52:47.265 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.263+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2873966885}) 2024-09-16T14:52:47.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.263+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.263+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=720350223}) 2024-09-16T14:52:47.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.263+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860682895}) 2024-09-16T14:52:47.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.263+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.263+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=720350223}) 2024-09-16T14:52:47.267 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.263+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1132146308}) 2024-09-16T14:52:47.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.263+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.263+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1481898717}) 2024-09-16T14:52:47.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.263+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1950387636}) 2024-09-16T14:52:47.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.263+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.267+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1481898717}) 2024-09-16T14:52:47.269 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.267+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1513483205}) 2024-09-16T14:52:47.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.267+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.267+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1476641455}) 2024-09-16T14:52:47.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.267+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3124827489}) 2024-09-16T14:52:47.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.267+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.267+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1476641455}) 2024-09-16T14:52:47.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.267+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2387370946}) 2024-09-16T14:52:47.271 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.267+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.267+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4130897090}) 2024-09-16T14:52:47.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.267+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2726792075}) 2024-09-16T14:52:47.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.267+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.267+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4130897090}) 2024-09-16T14:52:47.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.267+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4080398443}) 2024-09-16T14:52:47.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.271+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.271+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2730236224}) 2024-09-16T14:52:47.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.271+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176910883}) 2024-09-16T14:52:47.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.271+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.271+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2730236224}) 2024-09-16T14:52:47.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.271+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2870539840}) 2024-09-16T14:52:47.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.271+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.271+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=383208009}) 2024-09-16T14:52:47.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.271+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687865592}) 2024-09-16T14:52:47.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.271+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.271+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=383208009}) 2024-09-16T14:52:47.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.271+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=866670492}) 2024-09-16T14:52:47.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.275+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.275+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2139451672}) 2024-09-16T14:52:47.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.275+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271116600}) 2024-09-16T14:52:47.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.275+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.275+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2139451672}) 2024-09-16T14:52:47.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.275+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=892141597}) 2024-09-16T14:52:47.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.279+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.279+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3676075291}) 2024-09-16T14:52:47.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.279+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665452010}) 2024-09-16T14:52:47.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.279+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.279+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3676075291}) 2024-09-16T14:52:47.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.279+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=926681347}) 2024-09-16T14:52:47.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.283+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.283+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2770271842}) 2024-09-16T14:52:47.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.283+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2397780924}) 2024-09-16T14:52:47.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.283+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.283+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2770271842}) 2024-09-16T14:52:47.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.283+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=31421057}) 2024-09-16T14:52:47.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.283+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.283+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1995288571}) 2024-09-16T14:52:47.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.283+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2719434463}) 2024-09-16T14:52:47.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.283+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.283+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1995288571}) 2024-09-16T14:52:47.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.283+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1657181433}) 2024-09-16T14:52:47.290 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.287+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.290 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.287+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=458186467}) 2024-09-16T14:52:47.290 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.287+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855411612}) 2024-09-16T14:52:47.290 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.287+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.290 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.287+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=458186467}) 2024-09-16T14:52:47.290 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.287+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4218600373}) 2024-09-16T14:52:47.292 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.291+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.292 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.291+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=704177208}) 2024-09-16T14:52:47.292 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.291+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3215463382}) 2024-09-16T14:52:47.292 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.291+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.292 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.291+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=704177208}) 2024-09-16T14:52:47.292 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.291+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2628015262}) 2024-09-16T14:52:47.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.291+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.291+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1117329374}) 2024-09-16T14:52:47.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.291+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=63254703}) 2024-09-16T14:52:47.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.291+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.291+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1117329374}) 2024-09-16T14:52:47.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.291+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1132841661}) 2024-09-16T14:52:47.296 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.295+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.296 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.295+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=972810492}) 2024-09-16T14:52:47.296 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.295+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109087200}) 2024-09-16T14:52:47.296 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.295+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.295+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=972810492}) 2024-09-16T14:52:47.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.295+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1648309152}) 2024-09-16T14:52:47.299 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.295+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.299 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.295+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1041385260}) 2024-09-16T14:52:47.299 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.295+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333185350}) 2024-09-16T14:52:47.299 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.295+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.299 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.295+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1041385260}) 2024-09-16T14:52:47.299 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.295+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2617114824}) 2024-09-16T14:52:47.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.299+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.299+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3802101005}) 2024-09-16T14:52:47.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.299+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=837508727}) 2024-09-16T14:52:47.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.299+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.299+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3802101005}) 2024-09-16T14:52:47.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.299+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1011586886}) 2024-09-16T14:52:47.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.299+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.299+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1471834169}) 2024-09-16T14:52:47.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.299+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=315838021}) 2024-09-16T14:52:47.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.299+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.299+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1471834169}) 2024-09-16T14:52:47.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.299+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1426953185}) 2024-09-16T14:52:47.306 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.303+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.306 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.303+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3411727848}) 2024-09-16T14:52:47.306 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.303+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1660285146}) 2024-09-16T14:52:47.306 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.303+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.306 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.303+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3411727848}) 2024-09-16T14:52:47.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.303+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2497698425}) 2024-09-16T14:52:47.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.307+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.307+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3548824945}) 2024-09-16T14:52:47.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.307+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310580037}) 2024-09-16T14:52:47.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.307+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.307+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3548824945}) 2024-09-16T14:52:47.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.307+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3345885602}) 2024-09-16T14:52:47.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.307+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.307+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=621430327}) 2024-09-16T14:52:47.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.307+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=539179778}) 2024-09-16T14:52:47.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.307+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.307+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=621430327}) 2024-09-16T14:52:47.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.307+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2490166742}) 2024-09-16T14:52:47.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.311+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.311+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=183011501}) 2024-09-16T14:52:47.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.311+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1013321545}) 2024-09-16T14:52:47.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.311+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.311+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=183011501}) 2024-09-16T14:52:47.314 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.311+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1272508490}) 2024-09-16T14:52:47.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.311+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.311+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3250728099}) 2024-09-16T14:52:47.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.311+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011013340}) 2024-09-16T14:52:47.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.315+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.315+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3250728099}) 2024-09-16T14:52:47.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.315+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3254161295}) 2024-09-16T14:52:47.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.315+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.315+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1987429173}) 2024-09-16T14:52:47.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.315+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367823168}) 2024-09-16T14:52:47.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.315+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.315+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1987429173}) 2024-09-16T14:52:47.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.315+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3311504276}) 2024-09-16T14:52:47.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.315+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.315+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3875944945}) 2024-09-16T14:52:47.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.315+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434465153}) 2024-09-16T14:52:47.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.315+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.315+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3875944945}) 2024-09-16T14:52:47.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.315+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3229276174}) 2024-09-16T14:52:47.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1294014553}) 2024-09-16T14:52:47.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019523296}) 2024-09-16T14:52:47.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1294014553}) 2024-09-16T14:52:47.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1655140763}) 2024-09-16T14:52:47.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2733158995}) 2024-09-16T14:52:47.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=283747090}) 2024-09-16T14:52:47.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2733158995}) 2024-09-16T14:52:47.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=724387118}) 2024-09-16T14:52:47.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2237975113}) 2024-09-16T14:52:47.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175465913}) 2024-09-16T14:52:47.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2237975113}) 2024-09-16T14:52:47.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.319+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2434371134}) 2024-09-16T14:52:47.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.323+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.323+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2383609578}) 2024-09-16T14:52:47.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.323+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131771216}) 2024-09-16T14:52:47.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.323+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.323+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2383609578}) 2024-09-16T14:52:47.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.323+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=538855526}) 2024-09-16T14:52:47.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1904290807}) 2024-09-16T14:52:47.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1961840802}) 2024-09-16T14:52:47.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1904290807}) 2024-09-16T14:52:47.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=462597252}) 2024-09-16T14:52:47.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3567446422}) 2024-09-16T14:52:47.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2491133555}) 2024-09-16T14:52:47.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3567446422}) 2024-09-16T14:52:47.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1226852702}) 2024-09-16T14:52:47.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2696787174}) 2024-09-16T14:52:47.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1654993191}) 2024-09-16T14:52:47.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2696787174}) 2024-09-16T14:52:47.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.327+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3289659849}) 2024-09-16T14:52:47.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.331+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.331+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2530568128}) 2024-09-16T14:52:47.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.331+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2219444420}) 2024-09-16T14:52:47.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.331+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.331+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2530568128}) 2024-09-16T14:52:47.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.331+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1544469900}) 2024-09-16T14:52:47.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.331+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.331+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3043346892}) 2024-09-16T14:52:47.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.331+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2882894}) 2024-09-16T14:52:47.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.331+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.331+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3043346892}) 2024-09-16T14:52:47.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.331+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3574967736}) 2024-09-16T14:52:47.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1208647537}) 2024-09-16T14:52:47.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=549653070}) 2024-09-16T14:52:47.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1208647537}) 2024-09-16T14:52:47.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1294249271}) 2024-09-16T14:52:47.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1080528677}) 2024-09-16T14:52:47.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2274660961}) 2024-09-16T14:52:47.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1080528677}) 2024-09-16T14:52:47.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=697033246}) 2024-09-16T14:52:47.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3649025021}) 2024-09-16T14:52:47.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2996488048}) 2024-09-16T14:52:47.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3649025021}) 2024-09-16T14:52:47.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.335+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2950723127}) 2024-09-16T14:52:47.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2951458488}) 2024-09-16T14:52:47.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876009891}) 2024-09-16T14:52:47.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2951458488}) 2024-09-16T14:52:47.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3100224166}) 2024-09-16T14:52:47.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3109167523}) 2024-09-16T14:52:47.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1176327988}) 2024-09-16T14:52:47.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3109167523}) 2024-09-16T14:52:47.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3690948450}) 2024-09-16T14:52:47.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3188713439}) 2024-09-16T14:52:47.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1787976723}) 2024-09-16T14:52:47.344 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.344 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3188713439}) 2024-09-16T14:52:47.344 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.339+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2650262914}) 2024-09-16T14:52:47.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.343+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.343+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3604338091}) 2024-09-16T14:52:47.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.343+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2627031812}) 2024-09-16T14:52:47.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.343+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.343+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3604338091}) 2024-09-16T14:52:47.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.343+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2974303627}) 2024-09-16T14:52:47.346 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.343+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.343+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1863036448}) 2024-09-16T14:52:47.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.343+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3531328564}) 2024-09-16T14:52:47.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.343+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.343+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1863036448}) 2024-09-16T14:52:47.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.343+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2290156136}) 2024-09-16T14:52:47.348 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.347+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.347+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4270984875}) 2024-09-16T14:52:47.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.347+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3355139509}) 2024-09-16T14:52:47.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.347+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.347+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4270984875}) 2024-09-16T14:52:47.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.347+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2754957859}) 2024-09-16T14:52:47.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.347+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.347+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=933449672}) 2024-09-16T14:52:47.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.347+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1555060193}) 2024-09-16T14:52:47.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.347+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.347+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=933449672}) 2024-09-16T14:52:47.351 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.347+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1722565665}) 2024-09-16T14:52:47.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2649557967}) 2024-09-16T14:52:47.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1324784015}) 2024-09-16T14:52:47.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2649557967}) 2024-09-16T14:52:47.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=917307050}) 2024-09-16T14:52:47.353 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=225624099}) 2024-09-16T14:52:47.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=638465084}) 2024-09-16T14:52:47.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=225624099}) 2024-09-16T14:52:47.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=166294760}) 2024-09-16T14:52:47.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181480013}) 2024-09-16T14:52:47.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1334656086}) 2024-09-16T14:52:47.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181480013}) 2024-09-16T14:52:47.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.351+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1709039442}) 2024-09-16T14:52:47.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.355+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.355+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2899544822}) 2024-09-16T14:52:47.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.355+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3505681087}) 2024-09-16T14:52:47.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.355+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.355+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2899544822}) 2024-09-16T14:52:47.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.355+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4267889148}) 2024-09-16T14:52:47.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.355+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.355+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=86,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=3666712966,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2576105420}) 2024-09-16T14:52:47.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.355+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=88,mdsmap=3,mgr=8,mgr_command_descs=1} crc {logm=2309541634,mdsmap=1531151339,mgr=3291276092,mgr_command_descs=794974419}) 2024-09-16T14:52:47.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.355+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.355+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=86,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=3666712966,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2576105420}) 2024-09-16T14:52:47.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.355+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=84,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=3074359412,mdsmap=3585761405,mgr=2539502010,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3198571663}) 2024-09-16T14:52:47.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.367+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.367+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=7,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=47} crc {mgrstat=1593415225,mon_config_key=1218644173,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1763434146}) 2024-09-16T14:52:47.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.367+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=6,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=55} crc {mgrstat=4020992801,mon_config_key=3840829532,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3819525376}) 2024-09-16T14:52:47.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.367+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:52:47.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.367+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=7,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=47} crc {mgrstat=1593415225,mon_config_key=1218644173,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1763434146}) 2024-09-16T14:52:47.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:52:47.367+0000 7efe86769640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=7,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgrstat=1593415225,mon_config_key=4052528784,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=128526695}) 2024-09-16T14:52:48.266 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:52:48.266 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:52:48.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f919d434640 10 monclient: get_monmap_and_config 2024-09-16T14:52:48.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f919d434640 10 monclient: build_initial_monmap 2024-09-16T14:52:48.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f919d434640 10 monclient: monmap: 2024-09-16T14:52:48.398 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:48.398 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.398 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:48.398987+0000 2024-09-16T14:52:48.398 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:48.398987+0000 2024-09-16T14:52:48.399 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:48.399 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.399 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:48.399 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:48.399 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:48.399 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f919d434640 1 Processor -- start 2024-09-16T14:52:48.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f919d434640 1 -- start start 2024-09-16T14:52:48.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f919d434640 10 monclient: init 2024-09-16T14:52:48.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f919d434640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:48.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f919d434640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:52:48.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f919d434640 10 monclient(hunting): picked mon.noname-c con 0x7f91980a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:48.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f919d434640 10 monclient(hunting): picked mon.noname-a con 0x7f91980a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:48.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f919d434640 10 monclient(hunting): picked mon.noname-b con 0x7f91980a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:48.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f919d434640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91980a3250 con 0x7f91980a0000 2024-09-16T14:52:48.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f919d434640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919809f600 con 0x7f91980a1d10 2024-09-16T14:52:48.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f919d434640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91980a3830 con 0x7f91980a2800 2024-09-16T14:52:48.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f919d434640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:48.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f919d434640 10 monclient(hunting): authenticate will time out at 1592.519764s 2024-09-16T14:52:48.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f918e7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f91980a0000 0x7f9198139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54386/0 (socket says 172.21.15.26:54386) 2024-09-16T14:52:48.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.395+0000 7f918e7fc640 1 -- 172.21.15.26:0/180869892 learned_addr learned my addr 172.21.15.26:0/180869892 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:48.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 1 -- 172.21.15.26:0/180869892 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793968735 0 0) 0x7f91980a3250 con 0x7f91980a0000 2024-09-16T14:52:48.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient(hunting): my global_id is 194367 2024-09-16T14:52:48.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 1 -- 172.21.15.26:0/180869892 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9184003530 con 0x7f91980a0000 2024-09-16T14:52:48.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 1 -- 172.21.15.26:0/180869892 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512809240 0 0) 0x7f91980a3830 con 0x7f91980a2800 2024-09-16T14:52:48.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient(hunting): my global_id is 214232 2024-09-16T14:52:48.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 1 -- 172.21.15.26:0/180869892 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91980a3250 con 0x7f91980a2800 2024-09-16T14:52:48.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 1 -- 172.21.15.26:0/180869892 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1600533846 0 0) 0x7f91980a3250 con 0x7f91980a2800 2024-09-16T14:52:48.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 1 -- 172.21.15.26:0/180869892 >> v1:172.21.15.189:6789/0 conn(0x7f91980a1d10 legacy=0x7f91980a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 1 -- 172.21.15.26:0/180869892 >> v1:172.21.15.26:6789/0 conn(0x7f91980a0000 legacy=0x7f9198139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient: _finish_hunting 0 2024-09-16T14:52:48.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 1 monclient: found mon.noname-b 2024-09-16T14:52:48.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:48.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:52:48.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 1 -- 172.21.15.26:0/180869892 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919809ed60 con 0x7f91980a2800 2024-09-16T14:52:48.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient: _finish_auth 0 2024-09-16T14:52:48.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient: _check_auth_tickets 2024-09-16T14:52:48.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:48.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 1 -- 172.21.15.26:0/180869892 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f91900019a0 con 0x7f91980a2800 2024-09-16T14:52:48.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 5 monclient: authenticate success, global_id 214232 2024-09-16T14:52:48.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:48.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:52:48.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient: dump: 2024-09-16T14:52:48.411 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.411 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.411 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.411 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.411 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.411 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.411 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.411 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.412 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.412 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:48.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 1 -- 172.21.15.26:0/180869892 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9190001b50 con 0x7f91980a2800 2024-09-16T14:52:48.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:48.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 1 -- 172.21.15.26:0/180869892 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9190001f20 con 0x7f91980a2800 2024-09-16T14:52:48.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:48.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 4 set_mon_vals no callback set 2024-09-16T14:52:48.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:48.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91967fc640 10 monclient: dump: 2024-09-16T14:52:48.412 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.412 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.413 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.413 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.413 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.413 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.413 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.413 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.413 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.413 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 10 monclient: shutdown 2024-09-16T14:52:48.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:48.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 1 -- 172.21.15.26:0/180869892 >> v1:172.21.15.26:6790/0 conn(0x7f91980a2800 legacy=0x7f919813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 1 -- 172.21.15.26:0/180869892 shutdown_connections 2024-09-16T14:52:48.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 1 -- 172.21.15.26:0/180869892 >> 172.21.15.26:0/180869892 conn(0x7f91980a7f30 msgr2=0x7f91980a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:48.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 1 -- 172.21.15.26:0/180869892 shutdown_connections 2024-09-16T14:52:48.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 1 -- 172.21.15.26:0/180869892 wait complete. 2024-09-16T14:52:48.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 10 monclient: build_initial_monmap 2024-09-16T14:52:48.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 10 monclient: monmap: 2024-09-16T14:52:48.414 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:48.414 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:48.414 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:48.414 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:48.414 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:48.414 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:48.414 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:48.414 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:48.414 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:48.414 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 1 Processor -- start 2024-09-16T14:52:48.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 1 -- start start 2024-09-16T14:52:48.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 10 monclient: init 2024-09-16T14:52:48.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:48.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:52:48.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 10 monclient(hunting): picked mon.noname-c con 0x7f91980a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:48.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 10 monclient(hunting): picked mon.noname-b con 0x7f91980a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:48.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 10 monclient(hunting): picked mon.noname-a con 0x7f91980a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:48.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919813d090 con 0x7f91980a2800 2024-09-16T14:52:48.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9198139f50 con 0x7f91980a1d10 2024-09-16T14:52:48.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91980a6590 con 0x7f91980a0000 2024-09-16T14:52:48.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:48.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 10 monclient(hunting): authenticate will time out at 1592.523012s 2024-09-16T14:52:48.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f9196ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f91980a0000 0x7f9198139130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54444/0 (socket says 172.21.15.26:54444) 2024-09-16T14:52:48.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f9196ffd640 1 -- 172.21.15.26:0/3240188236 learned_addr learned my addr 172.21.15.26:0/3240188236 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:48.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f91977fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f91980a2800 0x7f91981d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54388/0 (socket says 172.21.15.26:54388) 2024-09-16T14:52:48.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703107955 0 0) 0x7f91980a6590 con 0x7f91980a0000 2024-09-16T14:52:48.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): my global_id is 214235 2024-09-16T14:52:48.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9178003530 con 0x7f91980a0000 2024-09-16T14:52:48.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119570932 0 0) 0x7f919813d090 con 0x7f91980a2800 2024-09-16T14:52:48.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): my global_id is 194370 2024-09-16T14:52:48.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91980a6590 con 0x7f91980a2800 2024-09-16T14:52:48.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 968583216 0 0) 0x7f9178003530 con 0x7f91980a0000 2024-09-16T14:52:48.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f919813d090 con 0x7f91980a0000 2024-09-16T14:52:48.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9188001940 con 0x7f91980a0000 2024-09-16T14:52:48.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:52:48.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): dump: 2024-09-16T14:52:48.417 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.417 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.417 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.417 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.417 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.417 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.418 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.418 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.418 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.418 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:48.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 599331994 0 0) 0x7f91980a6590 con 0x7f91980a2800 2024-09-16T14:52:48.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9178003530 con 0x7f91980a2800 2024-09-16T14:52:48.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9190001e60 con 0x7f91980a2800 2024-09-16T14:52:48.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:48.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): dump: 2024-09-16T14:52:48.418 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.418 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.418 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.419 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.419 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.419 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.419 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.419 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.419 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.419 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242431596 0 0) 0x7f9198139f50 con 0x7f91980a1d10 2024-09-16T14:52:48.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient(hunting): my global_id is 224125 2024-09-16T14:52:48.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91980a6590 con 0x7f91980a1d10 2024-09-16T14:52:48.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1560141822 0 0) 0x7f919813d090 con 0x7f91980a0000 2024-09-16T14:52:48.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 >> v1:172.21.15.189:6789/0 conn(0x7f91980a1d10 legacy=0x7f9198139840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 >> v1:172.21.15.26:6789/0 conn(0x7f91980a2800 legacy=0x7f91981d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:52:48.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 monclient: found mon.c 2024-09-16T14:52:48.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:48.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:48.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919809e500 con 0x7f91980a0000 2024-09-16T14:52:48.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient: _finish_auth 0 2024-09-16T14:52:48.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:52:48.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f918ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:48.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 5 monclient: authenticate success, global_id 214235 2024-09-16T14:52:48.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.399+0000 7f919d434640 10 monclient: _renew_subs 2024-09-16T14:52:48.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f919d434640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:48.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f919d434640 1 -- 172.21.15.26:0/3240188236 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919813b680 con 0x7f91980a0000 2024-09-16T14:52:48.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f919d434640 10 monclient: _renew_subs 2024-09-16T14:52:48.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f919d434640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:48.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f919d434640 1 -- 172.21.15.26:0/3240188236 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f919813ad30 con 0x7f91980a0000 2024-09-16T14:52:48.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9188001d10 con 0x7f91980a0000 2024-09-16T14:52:48.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f918ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:48.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9188003a70 con 0x7f91980a0000 2024-09-16T14:52:48.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f918ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f918ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:48.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f9195ffb640 4 set_mon_vals no callback set 2024-09-16T14:52:48.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f918ffff640 10 monclient: dump: 2024-09-16T14:52:48.422 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.422 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.422 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.422 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.422 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.422 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.422 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.422 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.422 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.422 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f919d434640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:48.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f919d434640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:48.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f919d434640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:48.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f919d434640 1 -- 172.21.15.26:0/3240188236 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f917c005050 con 0x7f91980a0000 2024-09-16T14:52:48.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9188003c50 con 0x7f91980a0000 2024-09-16T14:52:48.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f9188043f40 con 0x7f91980a0000 2024-09-16T14:52:48.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.403+0000 7f918e7fc640 10 monclient: get_auth_request con 0x7f91780363f0 auth_method 0 2024-09-16T14:52:48.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.407+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9188014670 con 0x7f91980a0000 2024-09-16T14:52:48.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.407+0000 7f918ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:48.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.407+0000 7f918ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:48.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.631+0000 7f919d434640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:52:48.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.631+0000 7f919d434640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:48.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.631+0000 7f919d434640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:48.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.631+0000 7f919d434640 1 -- 172.21.15.26:0/3240188236 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f917c005880 con 0x7f91980a0000 2024-09-16T14:52:48.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.631+0000 7f918ffff640 1 -- 172.21.15.26:0/3240188236 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1263817209) 0x7f9188012d10 con 0x7f91980a0000 2024-09-16T14:52:48.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.631+0000 7f918ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:48.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.631+0000 7f918ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:48.634 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:52:48.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.631+0000 7f919d434640 1 -- 172.21.15.26:0/3240188236 >> v1:172.21.15.189:6812/3196161582 conn(0x7f91780363f0 legacy=0x7f91780388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.631+0000 7f919d434640 10 monclient: shutdown 2024-09-16T14:52:48.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.631+0000 7f919d434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:48.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.631+0000 7f919d434640 1 -- 172.21.15.26:0/3240188236 >> v1:172.21.15.26:6790/0 conn(0x7f91980a0000 legacy=0x7f9198139130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.631+0000 7f919d434640 1 -- 172.21.15.26:0/3240188236 shutdown_connections 2024-09-16T14:52:48.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.631+0000 7f919d434640 1 -- 172.21.15.26:0/3240188236 >> 172.21.15.26:0/3240188236 conn(0x7f91980a7f30 msgr2=0x7f919813e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:48.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.631+0000 7f919d434640 1 -- 172.21.15.26:0/3240188236 shutdown_connections 2024-09-16T14:52:48.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.631+0000 7f919d434640 1 -- 172.21.15.26:0/3240188236 wait complete. 2024-09-16T14:52:48.651 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:52:48.651 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:52:48.651 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:52:48.652 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:52:48.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 10 monclient: get_monmap_and_config 2024-09-16T14:52:48.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 10 monclient: build_initial_monmap 2024-09-16T14:52:48.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 10 monclient: monmap: 2024-09-16T14:52:48.777 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:48.777 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.777 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:48.777983+0000 2024-09-16T14:52:48.777 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:48.777983+0000 2024-09-16T14:52:48.777 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:48.777 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.777 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:48.777 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:48.777 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:48.777 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 1 Processor -- start 2024-09-16T14:52:48.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 1 -- start start 2024-09-16T14:52:48.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 10 monclient: init 2024-09-16T14:52:48.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:48.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:52:48.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 10 monclient(hunting): picked mon.noname-b con 0x7f934809a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:48.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 10 monclient(hunting): picked mon.noname-c con 0x7f934809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:48.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 10 monclient(hunting): picked mon.noname-a con 0x7f934809b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:48.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934809dc60 con 0x7f934809b040 2024-09-16T14:52:48.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9348099ae0 con 0x7f934809d1b0 2024-09-16T14:52:48.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934809b8f0 con 0x7f934809a550 2024-09-16T14:52:48.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:48.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 10 monclient(hunting): authenticate will time out at 1592.898076s 2024-09-16T14:52:48.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934759e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f934809b040 0x7f934813a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54398/0 (socket says 172.21.15.26:54398) 2024-09-16T14:52:48.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934759e640 1 -- 172.21.15.26:0/2779301355 learned_addr learned my addr 172.21.15.26:0/2779301355 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:48.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9346d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f934809a550 0x7f934809a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54454/0 (socket says 172.21.15.26:54454) 2024-09-16T14:52:48.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 1 -- 172.21.15.26:0/2779301355 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400464871 0 0) 0x7f934809b8f0 con 0x7f934809a550 2024-09-16T14:52:48.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient(hunting): my global_id is 214238 2024-09-16T14:52:48.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 1 -- 172.21.15.26:0/2779301355 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9330003530 con 0x7f934809a550 2024-09-16T14:52:48.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 1 -- 172.21.15.26:0/2779301355 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564966120 0 0) 0x7f934809dc60 con 0x7f934809b040 2024-09-16T14:52:48.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient(hunting): my global_id is 194373 2024-09-16T14:52:48.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 1 -- 172.21.15.26:0/2779301355 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f934809b8f0 con 0x7f934809b040 2024-09-16T14:52:48.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 1 -- 172.21.15.26:0/2779301355 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1095650329 0 0) 0x7f9330003530 con 0x7f934809a550 2024-09-16T14:52:48.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 1 -- 172.21.15.26:0/2779301355 >> v1:172.21.15.189:6789/0 conn(0x7f934809d1b0 legacy=0x7f9348136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 1 -- 172.21.15.26:0/2779301355 >> v1:172.21.15.26:6789/0 conn(0x7f934809b040 legacy=0x7f934813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:52:48.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 1 monclient: found mon.noname-b 2024-09-16T14:52:48.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:48.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:52:48.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 1 -- 172.21.15.26:0/2779301355 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934809bbd0 con 0x7f934809a550 2024-09-16T14:52:48.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient: _finish_auth 0 2024-09-16T14:52:48.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:52:48.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:48.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 1 -- 172.21.15.26:0/2779301355 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9338001620 con 0x7f934809a550 2024-09-16T14:52:48.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:52:48.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient: dump: 2024-09-16T14:52:48.793 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.793 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.793 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.793 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.793 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.793 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.793 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.793 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.793 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.794 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 5 monclient: authenticate success, global_id 214238 2024-09-16T14:52:48.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:48.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 1 -- 172.21.15.26:0/2779301355 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9338001840 con 0x7f934809a550 2024-09-16T14:52:48.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:48.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:48.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 4 set_mon_vals no callback set 2024-09-16T14:52:48.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 1 -- 172.21.15.26:0/2779301355 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9338001c10 con 0x7f934809a550 2024-09-16T14:52:48.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:48.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f9345d9b640 10 monclient: dump: 2024-09-16T14:52:48.794 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.794 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.794 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.795 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.795 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.795 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.795 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.795 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.795 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.795 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 10 monclient: shutdown 2024-09-16T14:52:48.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:48.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 1 -- 172.21.15.26:0/2779301355 >> v1:172.21.15.26:6790/0 conn(0x7f934809a550 legacy=0x7f934809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 1 -- 172.21.15.26:0/2779301355 shutdown_connections 2024-09-16T14:52:48.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 1 -- 172.21.15.26:0/2779301355 >> 172.21.15.26:0/2779301355 conn(0x7f934809fb50 msgr2=0x7f934809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:48.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 1 -- 172.21.15.26:0/2779301355 shutdown_connections 2024-09-16T14:52:48.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.775+0000 7f934d04f640 1 -- 172.21.15.26:0/2779301355 wait complete. 2024-09-16T14:52:48.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 10 monclient: build_initial_monmap 2024-09-16T14:52:48.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 10 monclient: monmap: 2024-09-16T14:52:48.796 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:48.796 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:48.796 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:48.796 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:48.796 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:48.796 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:48.797 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:48.797 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:48.797 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:48.797 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 1 Processor -- start 2024-09-16T14:52:48.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 1 -- start start 2024-09-16T14:52:48.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 10 monclient: init 2024-09-16T14:52:48.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:48.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:52:48.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 10 monclient(hunting): picked mon.noname-a con 0x7f934809a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:48.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 10 monclient(hunting): picked mon.noname-b con 0x7f934809b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:48.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 10 monclient(hunting): picked mon.noname-c con 0x7f934809d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:48.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93480a24c0 con 0x7f934809a550 2024-09-16T14:52:48.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93481a55e0 con 0x7f934809b040 2024-09-16T14:52:48.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934809a240 con 0x7f934809d1b0 2024-09-16T14:52:48.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:48.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 10 monclient(hunting): authenticate will time out at 1592.901215s 2024-09-16T14:52:48.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f9346d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f934809a550 0x7f93481b4a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54400/0 (socket says 172.21.15.26:54400) 2024-09-16T14:52:48.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f9346d9d640 1 -- 172.21.15.26:0/522682301 learned_addr learned my addr 172.21.15.26:0/522682301 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:48.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18190844 0 0) 0x7f934809a240 con 0x7f934809d1b0 2024-09-16T14:52:48.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): my global_id is 214241 2024-09-16T14:52:48.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9334003530 con 0x7f934809d1b0 2024-09-16T14:52:48.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502358848 0 0) 0x7f93480a24c0 con 0x7f934809a550 2024-09-16T14:52:48.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): my global_id is 194376 2024-09-16T14:52:48.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f934809a240 con 0x7f934809a550 2024-09-16T14:52:48.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 459475947 0 0) 0x7f9334003530 con 0x7f934809d1b0 2024-09-16T14:52:48.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93480a24c0 con 0x7f934809d1b0 2024-09-16T14:52:48.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3416710215 0 0) 0x7f934809a240 con 0x7f934809a550 2024-09-16T14:52:48.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9334003530 con 0x7f934809a550 2024-09-16T14:52:48.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9340001d80 con 0x7f934809d1b0 2024-09-16T14:52:48.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:52:48.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): dump: 2024-09-16T14:52:48.800 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.800 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.800 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.800 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.801 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.801 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.801 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.801 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.801 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.801 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:48.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9338001b50 con 0x7f934809a550 2024-09-16T14:52:48.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:48.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): dump: 2024-09-16T14:52:48.802 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.802 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.802 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.802 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.802 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.802 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.802 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.802 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.802 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.802 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968549820 0 0) 0x7f93481a55e0 con 0x7f934809b040 2024-09-16T14:52:48.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): my global_id is 224131 2024-09-16T14:52:48.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f934809a240 con 0x7f934809b040 2024-09-16T14:52:48.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1214816052 0 0) 0x7f93480a24c0 con 0x7f934809d1b0 2024-09-16T14:52:48.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 >> v1:172.21.15.189:6789/0 conn(0x7f934809b040 legacy=0x7f93481ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 >> v1:172.21.15.26:6789/0 conn(0x7f934809a550 legacy=0x7f93481b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:52:48.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 monclient: found mon.c 2024-09-16T14:52:48.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:48.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:48.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93481c9630 con 0x7f934809d1b0 2024-09-16T14:52:48.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:52:48.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:52:48.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:48.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 5 monclient: authenticate success, global_id 214241 2024-09-16T14:52:48.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 10 monclient: _renew_subs 2024-09-16T14:52:48.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:48.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 1 -- 172.21.15.26:0/522682301 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93481a6b90 con 0x7f934809d1b0 2024-09-16T14:52:48.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 10 monclient: _renew_subs 2024-09-16T14:52:48.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:48.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 1 -- 172.21.15.26:0/522682301 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93481b5130 con 0x7f934809d1b0 2024-09-16T14:52:48.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93400019c0 con 0x7f934809d1b0 2024-09-16T14:52:48.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:48.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9340004d90 con 0x7f934809d1b0 2024-09-16T14:52:48.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934559a640 4 set_mon_vals no callback set 2024-09-16T14:52:48.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:48.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient: dump: 2024-09-16T14:52:48.805 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.805 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.805 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.805 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.805 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.805 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.805 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.805 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.806 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.806 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:48.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934d04f640 1 -- 172.21.15.26:0/522682301 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9328006590 con 0x7f9328002830 2024-09-16T14:52:48.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f934000f800 con 0x7f934809d1b0 2024-09-16T14:52:48.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f9340043f40 con 0x7f934809d1b0 2024-09-16T14:52:48.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309255598 0 0) 0x7f9328006590 con 0x7f9328002830 2024-09-16T14:52:48.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient(hunting): my global_id is 194379 2024-09-16T14:52:48.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93480a24c0 con 0x7f9328002830 2024-09-16T14:52:48.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3800905383 0 0) 0x7f93480a24c0 con 0x7f9328002830 2024-09-16T14:52:48.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9328006590 con 0x7f9328002830 2024-09-16T14:52:48.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9338004d80 con 0x7f9328002830 2024-09-16T14:52:48.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:48.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 10 monclient: dump: 2024-09-16T14:52:48.807 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.807 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.807 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.807 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.807 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.807 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.807 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.807 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.808 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.808 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f934659c640 10 monclient: get_auth_request con 0x7f93340363d0 auth_method 0 2024-09-16T14:52:48.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3952038207 0 0) 0x7f9328006590 con 0x7f9328002830 2024-09-16T14:52:48.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.779+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f932800a0e0 con 0x7f9328002830 2024-09-16T14:52:48.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.783+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f93380070f0 con 0x7f9328002830 2024-09-16T14:52:48.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.783+0000 7f932f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:48.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.783+0000 7f932f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:48.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.783+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 >> v1:172.21.15.26:6789/0 conn(0x7f9328002830 legacy=0x7f9328004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.783+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 >> v1:172.21.15.26:6789/0 conn(0x7f9328002830 legacy=0x7f9328004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:48.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f934d04f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:52:48.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f934d04f640 1 -- 172.21.15.26:0/522682301 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9328009da0 con 0x7f93280090c0 2024-09-16T14:52:48.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470597655 0 0) 0x7f9328009da0 con 0x7f93280090c0 2024-09-16T14:52:48.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 10 monclient(hunting): my global_id is 194382 2024-09-16T14:52:48.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9328006590 con 0x7f93280090c0 2024-09-16T14:52:48.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 113500337 0 0) 0x7f9328006590 con 0x7f93280090c0 2024-09-16T14:52:48.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9328009da0 con 0x7f93280090c0 2024-09-16T14:52:48.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f93380055d0 con 0x7f93280090c0 2024-09-16T14:52:48.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:48.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 10 monclient: dump: 2024-09-16T14:52:48.811 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.811 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.811 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.811 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.811 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.811 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.811 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.811 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.811 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.811 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1345223513 0 0) 0x7f9328009da0 con 0x7f93280090c0 2024-09-16T14:52:48.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f932800c290 con 0x7f93280090c0 2024-09-16T14:52:48.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 605142425) 0x7f9338006a00 con 0x7f93280090c0 2024-09-16T14:52:48.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:52:48.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:48.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 >> v1:172.21.15.26:6789/0 conn(0x7f93280090c0 legacy=0x7f9328009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f932f7fe640 1 -- 172.21.15.26:0/522682301 >> v1:172.21.15.26:6789/0 conn(0x7f93280090c0 legacy=0x7f9328009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:48.812 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:52:48.812 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:48.812 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:48.812 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:52:48.813 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 96, 2024-09-16T14:52:48.813 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:52:48.813 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:52:48.813 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:52:48.813 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:52:48.813 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:48.813 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:52:48.813 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:48.813 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:52:48.813 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:52:48.813 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:48.813 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:48.813 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:48.814 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:48.814 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:48.814 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:48.814 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:48.814 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:48.814 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:48.814 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:48.814 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:48.814 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:52:48.814 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:52:48.814 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:48.814 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:48.814 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:48.815 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:48.815 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:48.815 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:48.815 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:48.815 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:48.815 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:48.815 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:48.815 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:48.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.815 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:52:48.815 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:52:48.815 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:52:48.815 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:52:48.816 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:52:48.816 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:52:48.816 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:48.816 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:48.816 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:52:48.816 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:52:48.816 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:52:48.816 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:52:48.816 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:52:48.816 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:52:48.816 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:52:48.816 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:48.816 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:52:48.817 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:48.817 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:48.817 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:48.817 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:48.817 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:48.817 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:48.817 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:48.817 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:48.817 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:48.817 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:48.817 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:48.817 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:52:48.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.818 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:52:48.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.818 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:48.818 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:48.818 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:48.818 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:48.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.818 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:48.818 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:52:48.818 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:48.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:48.818 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:48.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.819 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:52:48.819 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:52:48.819 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:48.819 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:48.819 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:48.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.819 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:48.819 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:48.819 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:48.819 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:48.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.820 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:48.820 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:52:48.820 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:48.820 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:48.820 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:48.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.820 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:52:48.820 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:52:48.820 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:48.820 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:48.820 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:48.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.821 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:48.821 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:48.821 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:48.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:48.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.821 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:48.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:52:48.821 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:48.821 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:48.821 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:48.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:52:48.821 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:52:48.822 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:48.822 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:48.822 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:48.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:48.822 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:48.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.822 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:52:48.822 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:52:48.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.822 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:48.822 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:48.822 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:48.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:48.823 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:48.823 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:52:48.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.823 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:48.823 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:48.823 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:52:48.823 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:48.823 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:48.823 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:52:48.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.823 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:48.823 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:48.823 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:52:48.823 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:48.824 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:48.824 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:52:48.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.824 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:48.824 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:48.824 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:48.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:48.824 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:48.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.825 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:52:48.825 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:52:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f934d04f640 1 -- 172.21.15.26:0/522682301 >> v1:172.21.15.189:6812/3196161582 conn(0x7f93340363d0 legacy=0x7f9334038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f934d04f640 10 monclient: shutdown 2024-09-16T14:52:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f934d04f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f934d04f640 1 -- 172.21.15.26:0/522682301 >> v1:172.21.15.26:6790/0 conn(0x7f934809d1b0 legacy=0x7f93481d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f934759e640 1 -- 172.21.15.26:0/522682301 reap_dead start 2024-09-16T14:52:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f934759e640 1 -- 172.21.15.26:0/522682301 reap_dead start 2024-09-16T14:52:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f934d04f640 1 -- 172.21.15.26:0/522682301 shutdown_connections 2024-09-16T14:52:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f934d04f640 1 -- 172.21.15.26:0/522682301 >> 172.21.15.26:0/522682301 conn(0x7f934809fb50 msgr2=0x7f93480a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.787+0000 7f934d04f640 1 -- 172.21.15.26:0/522682301 shutdown_connections 2024-09-16T14:52:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.791+0000 7f934d04f640 1 -- 172.21.15.26:0/522682301 wait complete. 2024-09-16T14:52:48.827 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:52:48.827 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:52:48.827 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:52:48.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 10 monclient: get_monmap_and_config 2024-09-16T14:52:48.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 10 monclient: build_initial_monmap 2024-09-16T14:52:48.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 10 monclient: monmap: 2024-09-16T14:52:48.945 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:48.945 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.945 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:48.945869+0000 2024-09-16T14:52:48.945 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:48.945869+0000 2024-09-16T14:52:48.945 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:48.945 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.945 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:48.945 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:48.945 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:48.945 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 1 Processor -- start 2024-09-16T14:52:48.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 1 -- start start 2024-09-16T14:52:48.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 10 monclient: init 2024-09-16T14:52:48.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:48.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:52:48.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 10 monclient(hunting): picked mon.noname-b con 0x7f178409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:48.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 10 monclient(hunting): picked mon.noname-c con 0x7f1784132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:48.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 10 monclient(hunting): picked mon.noname-a con 0x7f1784138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:48.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178409d260 con 0x7f1784138ff0 2024-09-16T14:52:48.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1784132a80 con 0x7f1784132650 2024-09-16T14:52:48.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178409cf60 con 0x7f178409cb80 2024-09-16T14:52:48.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:48.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 10 monclient(hunting): authenticate will time out at 1593.066004s 2024-09-16T14:52:48.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178a311640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1784138ff0 0x7f178413b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54434/0 (socket says 172.21.15.26:54434) 2024-09-16T14:52:48.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1789b10640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f178409cb80 0x7f17840717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54476/0 (socket says 172.21.15.26:54476) 2024-09-16T14:52:48.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178a311640 1 -- 172.21.15.26:0/4075295749 learned_addr learned my addr 172.21.15.26:0/4075295749 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:48.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 1 -- 172.21.15.26:0/4075295749 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464376482 0 0) 0x7f178409cf60 con 0x7f178409cb80 2024-09-16T14:52:48.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient(hunting): my global_id is 214244 2024-09-16T14:52:48.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 1 -- 172.21.15.26:0/4075295749 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1770003530 con 0x7f178409cb80 2024-09-16T14:52:48.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 1 -- 172.21.15.26:0/4075295749 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780744134 0 0) 0x7f178409d260 con 0x7f1784138ff0 2024-09-16T14:52:48.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient(hunting): my global_id is 194385 2024-09-16T14:52:48.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 1 -- 172.21.15.26:0/4075295749 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f178409cf60 con 0x7f1784138ff0 2024-09-16T14:52:48.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 1 -- 172.21.15.26:0/4075295749 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1464762656 0 0) 0x7f178409cf60 con 0x7f1784138ff0 2024-09-16T14:52:48.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 1 -- 172.21.15.26:0/4075295749 >> v1:172.21.15.26:6790/0 conn(0x7f178409cb80 legacy=0x7f17840717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 1 -- 172.21.15.26:0/4075295749 >> v1:172.21.15.189:6789/0 conn(0x7f1784132650 legacy=0x7f17841378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient: _finish_hunting 0 2024-09-16T14:52:48.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 1 monclient: found mon.noname-a 2024-09-16T14:52:48.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:48.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:52:48.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 1 -- 172.21.15.26:0/4075295749 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178409f2d0 con 0x7f1784138ff0 2024-09-16T14:52:48.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient: _finish_auth 0 2024-09-16T14:52:48.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient: _check_auth_tickets 2024-09-16T14:52:48.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:48.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 1 -- 172.21.15.26:0/4075295749 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f177c001550 con 0x7f1784138ff0 2024-09-16T14:52:48.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:48.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient: dump: 2024-09-16T14:52:48.958 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.958 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.959 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.959 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.959 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.959 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.959 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.959 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.959 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.959 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 5 monclient: authenticate success, global_id 194385 2024-09-16T14:52:48.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:48.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 1 -- 172.21.15.26:0/4075295749 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f177c001770 con 0x7f1784138ff0 2024-09-16T14:52:48.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:48.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 1 -- 172.21.15.26:0/4075295749 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f177c001a80 con 0x7f1784138ff0 2024-09-16T14:52:48.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:48.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f1788b0e640 10 monclient: dump: 2024-09-16T14:52:48.960 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.960 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.960 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.960 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.960 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.960 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.960 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.960 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.960 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.960 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:48.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 4 set_mon_vals no callback set 2024-09-16T14:52:48.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 10 monclient: shutdown 2024-09-16T14:52:48.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:48.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 1 -- 172.21.15.26:0/4075295749 >> v1:172.21.15.26:6789/0 conn(0x7f1784138ff0 legacy=0x7f178413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 1 -- 172.21.15.26:0/4075295749 shutdown_connections 2024-09-16T14:52:48.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 1 -- 172.21.15.26:0/4075295749 >> 172.21.15.26:0/4075295749 conn(0x7f178409fb50 msgr2=0x7f178409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:48.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 1 -- 172.21.15.26:0/4075295749 shutdown_connections 2024-09-16T14:52:48.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.943+0000 7f178bd74640 1 -- 172.21.15.26:0/4075295749 wait complete. 2024-09-16T14:52:48.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 10 monclient: build_initial_monmap 2024-09-16T14:52:48.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 10 monclient: monmap: 2024-09-16T14:52:48.962 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:48.962 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:48.962 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:48.962 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:48.962 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:48.962 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:48.963 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:48.963 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:48.963 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:48.963 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 1 Processor -- start 2024-09-16T14:52:48.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 1 -- start start 2024-09-16T14:52:48.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 10 monclient: init 2024-09-16T14:52:48.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:48.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:52:48.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 10 monclient(hunting): picked mon.noname-b con 0x7f178409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:48.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 10 monclient(hunting): picked mon.noname-c con 0x7f1784132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:48.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 10 monclient(hunting): picked mon.noname-a con 0x7f1784138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:48.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17841381a0 con 0x7f1784138ff0 2024-09-16T14:52:48.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17840a2960 con 0x7f178409cb80 2024-09-16T14:52:48.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178406e4b0 con 0x7f1784132650 2024-09-16T14:52:48.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:48.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 10 monclient(hunting): authenticate will time out at 1593.069511s 2024-09-16T14:52:48.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178a311640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1784138ff0 0x7f17841cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54442/0 (socket says 172.21.15.26:54442) 2024-09-16T14:52:48.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178930f640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1784132650 0x7f178406dda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54488/0 (socket says 172.21.15.26:54488) 2024-09-16T14:52:48.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178a311640 1 -- 172.21.15.26:0/850254054 learned_addr learned my addr 172.21.15.26:0/850254054 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:48.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754139310 0 0) 0x7f178406e4b0 con 0x7f1784132650 2024-09-16T14:52:48.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): my global_id is 214247 2024-09-16T14:52:48.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1768003530 con 0x7f1784132650 2024-09-16T14:52:48.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707012535 0 0) 0x7f178409f2d0 con 0x7f1784138ff0 2024-09-16T14:52:48.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): my global_id is 194388 2024-09-16T14:52:48.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f178406e4b0 con 0x7f1784138ff0 2024-09-16T14:52:48.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2363147289 0 0) 0x7f1768003530 con 0x7f1784132650 2024-09-16T14:52:48.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f178409f2d0 con 0x7f1784132650 2024-09-16T14:52:48.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f17740018c0 con 0x7f1784132650 2024-09-16T14:52:48.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:52:48.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): dump: 2024-09-16T14:52:48.966 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.966 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.966 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.966 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.966 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.966 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.966 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.966 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.966 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.966 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:48.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605351133 0 0) 0x7f17840a2960 con 0x7f178409cb80 2024-09-16T14:52:48.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): my global_id is 224137 2024-09-16T14:52:48.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1768003530 con 0x7f178409cb80 2024-09-16T14:52:48.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2858815321 0 0) 0x7f178406e4b0 con 0x7f1784138ff0 2024-09-16T14:52:48.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17840a2960 con 0x7f1784138ff0 2024-09-16T14:52:48.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f177c0012b0 con 0x7f1784138ff0 2024-09-16T14:52:48.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:48.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): dump: 2024-09-16T14:52:48.967 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.968 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.968 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.968 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.968 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.968 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.969 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.969 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.969 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.969 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1620993097 0 0) 0x7f178409f2d0 con 0x7f1784132650 2024-09-16T14:52:48.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 >> v1:172.21.15.189:6789/0 conn(0x7f178409cb80 legacy=0x7f178406d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 >> v1:172.21.15.26:6789/0 conn(0x7f1784138ff0 legacy=0x7f17841cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:52:48.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 monclient: found mon.c 2024-09-16T14:52:48.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:48.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:48.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17840700d0 con 0x7f1784132650 2024-09-16T14:52:48.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:52:48.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:52:48.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:48.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 5 monclient: authenticate success, global_id 214247 2024-09-16T14:52:48.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 10 monclient: _renew_subs 2024-09-16T14:52:48.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:48.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 1 -- 172.21.15.26:0/850254054 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178406ff10 con 0x7f1784132650 2024-09-16T14:52:48.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 10 monclient: _renew_subs 2024-09-16T14:52:48.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:48.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 1 -- 172.21.15.26:0/850254054 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f178406f2e0 con 0x7f1784132650 2024-09-16T14:52:48.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1774003620 con 0x7f1784132650 2024-09-16T14:52:48.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:48.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1774003790 con 0x7f1784132650 2024-09-16T14:52:48.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:48.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177bfff640 4 set_mon_vals no callback set 2024-09-16T14:52:48.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient: dump: 2024-09-16T14:52:48.971 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.971 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.971 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.971 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.971 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.971 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.972 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.972 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f17740038f0 con 0x7f1784132650 2024-09-16T14:52:48.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:48.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f178bd74640 1 -- 172.21.15.26:0/850254054 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f176c006590 con 0x7f176c002830 2024-09-16T14:52:48.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f1774043b40 con 0x7f1784132650 2024-09-16T14:52:48.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528503795 0 0) 0x7f176c006590 con 0x7f176c002830 2024-09-16T14:52:48.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 10 monclient(hunting): my global_id is 214250 2024-09-16T14:52:48.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.947+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f178409f2d0 con 0x7f176c002830 2024-09-16T14:52:48.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.951+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3820437403 0 0) 0x7f178409f2d0 con 0x7f176c002830 2024-09-16T14:52:48.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.951+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f176c006590 con 0x7f176c002830 2024-09-16T14:52:48.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.951+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f17800019d0 con 0x7f176c002830 2024-09-16T14:52:48.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.951+0000 7f178a311640 10 monclient: get_auth_request con 0x7f17680364e0 auth_method 0 2024-09-16T14:52:48.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.951+0000 7f177a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.951+0000 7f177a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:48.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.951+0000 7f177a7fc640 10 monclient: dump: 2024-09-16T14:52:48.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.973 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.973 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.973 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.973 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.973 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.951+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 351158012 0 0) 0x7f176c006590 con 0x7f176c002830 2024-09-16T14:52:48.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.951+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f176c00a0e0 con 0x7f176c002830 2024-09-16T14:52:48.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.951+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1780001e20 con 0x7f176c002830 2024-09-16T14:52:48.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.951+0000 7f177a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:48.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.951+0000 7f177a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:48.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.951+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 >> v1:172.21.15.26:6790/0 conn(0x7f176c002830 legacy=0x7f176c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.951+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 >> v1:172.21.15.26:6790/0 conn(0x7f176c002830 legacy=0x7f176c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:48.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f178bd74640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:52:48.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f178bd74640 1 -- 172.21.15.26:0/850254054 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f176c009da0 con 0x7f176c0090c0 2024-09-16T14:52:48.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594783115 0 0) 0x7f176c009da0 con 0x7f176c0090c0 2024-09-16T14:52:48.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:48.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:48.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:48.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 10 monclient(hunting): my global_id is 214253 2024-09-16T14:52:48.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f176c006590 con 0x7f176c0090c0 2024-09-16T14:52:48.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3803796247 0 0) 0x7f176c006590 con 0x7f176c0090c0 2024-09-16T14:52:48.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f176c009da0 con 0x7f176c0090c0 2024-09-16T14:52:48.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1780004d10 con 0x7f176c0090c0 2024-09-16T14:52:48.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:48.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:48.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 10 monclient: dump: 2024-09-16T14:52:48.977 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:48.977 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:48.977 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.977 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:48.977 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:48.977 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:48.978 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:48.978 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:48.978 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:48.978 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:48.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3820194940 0 0) 0x7f176c009da0 con 0x7f176c0090c0 2024-09-16T14:52:48.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f176c00c290 con 0x7f176c0090c0 2024-09-16T14:52:48.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3881 (unknown 0 0 1101143737) 0x7f1780005f40 con 0x7f176c0090c0 2024-09-16T14:52:48.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:52:48.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:48.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 >> v1:172.21.15.26:6790/0 conn(0x7f176c0090c0 legacy=0x7f176c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.955+0000 7f177a7fc640 1 -- 172.21.15.26:0/850254054 >> v1:172.21.15.26:6790/0 conn(0x7f176c0090c0 legacy=0x7f176c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:48.979 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:52:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:52:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 96, 2024-09-16T14:52:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:52:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:52:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:52:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:52:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:52:48.979 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:52:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:52:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:48.980 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:52:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:52:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:48.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:52:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:52:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:52:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:52:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:52:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:52:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:52:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:52:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:52:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:52:48.982 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:52:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:52:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:52:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:52:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:48.983 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:52:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:52:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:48.984 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:52:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:52:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:52:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:48.985 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:52:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:52:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:52:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:48.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:52:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:48.987 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:52:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:52:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:52:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:52:48.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:52:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:52:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:48.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:48.990 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:52:48.990 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:52:48.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.959+0000 7f178bd74640 1 -- 172.21.15.26:0/850254054 >> v1:172.21.15.189:6812/3196161582 conn(0x7f17680364e0 legacy=0x7f17680389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.959+0000 7f178bd74640 10 monclient: shutdown 2024-09-16T14:52:48.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.959+0000 7f178bd74640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:48.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.959+0000 7f178bd74640 1 -- 172.21.15.26:0/850254054 >> v1:172.21.15.26:6790/0 conn(0x7f1784132650 legacy=0x7f178406dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:48.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.959+0000 7f178a311640 1 -- 172.21.15.26:0/850254054 reap_dead start 2024-09-16T14:52:48.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.959+0000 7f178a311640 1 -- 172.21.15.26:0/850254054 reap_dead start 2024-09-16T14:52:48.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.959+0000 7f178bd74640 1 -- 172.21.15.26:0/850254054 shutdown_connections 2024-09-16T14:52:48.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.959+0000 7f178bd74640 1 -- 172.21.15.26:0/850254054 >> 172.21.15.26:0/850254054 conn(0x7f178409fb50 msgr2=0x7f1784131cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:48.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.959+0000 7f178bd74640 1 -- 172.21.15.26:0/850254054 shutdown_connections 2024-09-16T14:52:48.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:48.959+0000 7f178bd74640 1 -- 172.21.15.26:0/850254054 wait complete. 2024-09-16T14:52:48.992 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:52:48.993 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:52:48.993 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:52:49.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.107+0000 7f3464c6e640 10 monclient: get_monmap_and_config 2024-09-16T14:52:49.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 10 monclient: build_initial_monmap 2024-09-16T14:52:49.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 10 monclient: monmap: 2024-09-16T14:52:49.112 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:49.112 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:49.112 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:49.113535+0000 2024-09-16T14:52:49.112 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:49.113535+0000 2024-09-16T14:52:49.112 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:49.113 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:49.113 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:49.113 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:49.113 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:49.113 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:49.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 1 Processor -- start 2024-09-16T14:52:49.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 1 -- start start 2024-09-16T14:52:49.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 10 monclient: init 2024-09-16T14:52:49.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:49.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:52:49.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 10 monclient(hunting): picked mon.noname-b con 0x7f34600a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:49.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 10 monclient(hunting): picked mon.noname-c con 0x7f34600a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:49.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 10 monclient(hunting): picked mon.noname-a con 0x7f34600a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:49.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34600a3250 con 0x7f34600a2800 2024-09-16T14:52:49.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346009f600 con 0x7f34600a0000 2024-09-16T14:52:49.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34600a3830 con 0x7f34600a1d10 2024-09-16T14:52:49.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:49.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 10 monclient(hunting): authenticate will time out at 1593.233531s 2024-09-16T14:52:49.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345ed9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f34600a2800 0x7f346013d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54454/0 (socket says 172.21.15.26:54454) 2024-09-16T14:52:49.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345e59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f34600a1d10 0x7f34600a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54514/0 (socket says 172.21.15.26:54514) 2024-09-16T14:52:49.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345ed9d640 1 -- 172.21.15.26:0/3915874502 learned_addr learned my addr 172.21.15.26:0/3915874502 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:49.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 1 -- 172.21.15.26:0/3915874502 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280984454 0 0) 0x7f34600a3830 con 0x7f34600a1d10 2024-09-16T14:52:49.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient(hunting): _negotiate 2024-09-16T14:52:49.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:49.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:49.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient(hunting): my global_id is 214256 2024-09-16T14:52:49.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 1 -- 172.21.15.26:0/3915874502 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3448003530 con 0x7f34600a1d10 2024-09-16T14:52:49.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 1 -- 172.21.15.26:0/3915874502 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138334686 0 0) 0x7f34600a3250 con 0x7f34600a2800 2024-09-16T14:52:49.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient(hunting): _negotiate 2024-09-16T14:52:49.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:49.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:49.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient(hunting): my global_id is 194391 2024-09-16T14:52:49.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 1 -- 172.21.15.26:0/3915874502 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34600a3830 con 0x7f34600a2800 2024-09-16T14:52:49.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 1 -- 172.21.15.26:0/3915874502 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661173220 0 0) 0x7f346009f600 con 0x7f34600a0000 2024-09-16T14:52:49.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient(hunting): _negotiate 2024-09-16T14:52:49.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:49.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:49.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient(hunting): my global_id is 224140 2024-09-16T14:52:49.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 1 -- 172.21.15.26:0/3915874502 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34600a3250 con 0x7f34600a0000 2024-09-16T14:52:49.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 1 -- 172.21.15.26:0/3915874502 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1125749782 0 0) 0x7f34600a3830 con 0x7f34600a2800 2024-09-16T14:52:49.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 1 -- 172.21.15.26:0/3915874502 >> v1:172.21.15.26:6790/0 conn(0x7f34600a1d10 legacy=0x7f34600a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:49.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 1 -- 172.21.15.26:0/3915874502 >> v1:172.21.15.189:6789/0 conn(0x7f34600a0000 legacy=0x7f3460139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:49.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient: _finish_hunting 0 2024-09-16T14:52:49.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 1 monclient: found mon.noname-a 2024-09-16T14:52:49.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:49.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:52:49.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 1 -- 172.21.15.26:0/3915874502 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346009ed60 con 0x7f34600a2800 2024-09-16T14:52:49.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient: _finish_auth 0 2024-09-16T14:52:49.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient: _check_auth_tickets 2024-09-16T14:52:49.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:49.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 1 -- 172.21.15.26:0/3915874502 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3458001980 con 0x7f34600a2800 2024-09-16T14:52:49.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 5 monclient: authenticate success, global_id 194391 2024-09-16T14:52:49.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:49.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:49.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:49.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient: dump: 2024-09-16T14:52:49.120 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:49.120 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:49.120 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.120 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.120 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:49.120 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:49.120 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:49.120 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:49.120 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:49.120 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:49.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:49.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 1 -- 172.21.15.26:0/3915874502 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3458001ca0 con 0x7f34600a2800 2024-09-16T14:52:49.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 1 -- 172.21.15.26:0/3915874502 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3458002030 con 0x7f34600a2800 2024-09-16T14:52:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 4 set_mon_vals no callback set 2024-09-16T14:52:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f345d59a640 10 monclient: dump: 2024-09-16T14:52:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:49.121 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:49.122 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:49.122 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:49.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 10 monclient: shutdown 2024-09-16T14:52:49.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:49.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 1 -- 172.21.15.26:0/3915874502 >> v1:172.21.15.26:6789/0 conn(0x7f34600a2800 legacy=0x7f346013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:49.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 1 -- 172.21.15.26:0/3915874502 shutdown_connections 2024-09-16T14:52:49.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 1 -- 172.21.15.26:0/3915874502 >> 172.21.15.26:0/3915874502 conn(0x7f34600a7f30 msgr2=0x7f34600a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:49.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 1 -- 172.21.15.26:0/3915874502 shutdown_connections 2024-09-16T14:52:49.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.111+0000 7f3464c6e640 1 -- 172.21.15.26:0/3915874502 wait complete. 2024-09-16T14:52:49.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 10 monclient: build_initial_monmap 2024-09-16T14:52:49.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 10 monclient: monmap: 2024-09-16T14:52:49.122 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:49.122 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:49.122 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:49.123 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:49.123 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:49.123 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:49.123 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:49.123 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:49.123 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:49.123 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:49.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 1 Processor -- start 2024-09-16T14:52:49.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 1 -- start start 2024-09-16T14:52:49.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 10 monclient: init 2024-09-16T14:52:49.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:49.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:52:49.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 10 monclient(hunting): picked mon.noname-c con 0x7f34600a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:49.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 10 monclient(hunting): picked mon.noname-a con 0x7f34600a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:49.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 10 monclient(hunting): picked mon.noname-b con 0x7f34600a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:49.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346013d170 con 0x7f34600a1d10 2024-09-16T14:52:49.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3460139fd0 con 0x7f34600a2800 2024-09-16T14:52:49.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34600a65f0 con 0x7f34600a0000 2024-09-16T14:52:49.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:49.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 10 monclient(hunting): authenticate will time out at 1593.237422s 2024-09-16T14:52:49.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f345e59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f34600a0000 0x7f34601391b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54516/0 (socket says 172.21.15.26:54516) 2024-09-16T14:52:49.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f345e59c640 1 -- 172.21.15.26:0/1485546349 learned_addr learned my addr 172.21.15.26:0/1485546349 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f345dd9b640 1 --1- 172.21.15.26:0/1485546349 >> v1:172.21.15.26:6789/0 conn(0x7f34600a1d10 0x7f34601398c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54466/0 (socket says 172.21.15.26:54466) 2024-09-16T14:52:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966979935 0 0) 0x7f34600a65f0 con 0x7f34600a0000 2024-09-16T14:52:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): my global_id is 214259 2024-09-16T14:52:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3440003530 con 0x7f34600a0000 2024-09-16T14:52:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058360630 0 0) 0x7f346013d170 con 0x7f34600a1d10 2024-09-16T14:52:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): my global_id is 194394 2024-09-16T14:52:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34600a65f0 con 0x7f34600a1d10 2024-09-16T14:52:49.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3532552061 0 0) 0x7f3440003530 con 0x7f34600a0000 2024-09-16T14:52:49.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f346013d170 con 0x7f34600a0000 2024-09-16T14:52:49.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34500018e0 con 0x7f34600a0000 2024-09-16T14:52:49.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:49.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:52:49.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): dump: 2024-09-16T14:52:49.130 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:49.130 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:49.130 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.130 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.130 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:49.130 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:49.130 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:49.130 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:49.131 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:49.131 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:49.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:49.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2881745999 0 0) 0x7f34600a65f0 con 0x7f34600a1d10 2024-09-16T14:52:49.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3440003530 con 0x7f34600a1d10 2024-09-16T14:52:49.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3454001820 con 0x7f34600a1d10 2024-09-16T14:52:49.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:49.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:49.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): dump: 2024-09-16T14:52:49.131 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:49.131 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:49.131 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.131 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.131 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:49.132 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:49.132 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:49.132 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:49.132 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:49.132 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:49.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481583071 0 0) 0x7f3460139fd0 con 0x7f34600a2800 2024-09-16T14:52:49.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:49.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:49.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:49.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient(hunting): my global_id is 224143 2024-09-16T14:52:49.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34600a65f0 con 0x7f34600a2800 2024-09-16T14:52:49.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3635166691 0 0) 0x7f346013d170 con 0x7f34600a0000 2024-09-16T14:52:49.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 >> v1:172.21.15.189:6789/0 conn(0x7f34600a2800 legacy=0x7f34601d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:49.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 >> v1:172.21.15.26:6789/0 conn(0x7f34600a1d10 legacy=0x7f34601398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:49.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient: _finish_hunting 0 2024-09-16T14:52:49.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 monclient: found mon.c 2024-09-16T14:52:49.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:49.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:49.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346009e500 con 0x7f34600a0000 2024-09-16T14:52:49.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient: _finish_auth 0 2024-09-16T14:52:49.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient: _check_auth_tickets 2024-09-16T14:52:49.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:49.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 5 monclient: authenticate success, global_id 214259 2024-09-16T14:52:49.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 10 monclient: _renew_subs 2024-09-16T14:52:49.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:49.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 1 -- 172.21.15.26:0/1485546349 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f346013b760 con 0x7f34600a0000 2024-09-16T14:52:49.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 10 monclient: _renew_subs 2024-09-16T14:52:49.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:52:49.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 1 -- 172.21.15.26:0/1485546349 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f346013ae10 con 0x7f34600a0000 2024-09-16T14:52:49.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34500047f0 con 0x7f34600a0000 2024-09-16T14:52:49.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:49.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3450004b00 con 0x7f34600a0000 2024-09-16T14:52:49.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:49.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:52:49.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f345cd99640 4 set_mon_vals no callback set 2024-09-16T14:52:49.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 10 monclient: dump: 2024-09-16T14:52:49.135 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:49.135 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:49.135 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.135 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.135 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:49.135 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:49.135 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:49.135 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:49.135 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:49.135 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:49.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:49.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f3464c6e640 1 -- 172.21.15.26:0/1485546349 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3444006590 con 0x7f3444002830 2024-09-16T14:52:49.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f345000f5f0 con 0x7f34600a0000 2024-09-16T14:52:49.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f3450043e10 con 0x7f34600a0000 2024-09-16T14:52:49.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.115+0000 7f345ed9d640 10 monclient: get_auth_request con 0x7f3440036440 auth_method 0 2024-09-16T14:52:49.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065544964 0 0) 0x7f3444006590 con 0x7f3444002830 2024-09-16T14:52:49.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:49.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:49.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:49.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 10 monclient(hunting): my global_id is 224146 2024-09-16T14:52:49.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f346013d170 con 0x7f3444002830 2024-09-16T14:52:49.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 283635086 0 0) 0x7f346013d170 con 0x7f3444002830 2024-09-16T14:52:49.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3444006590 con 0x7f3444002830 2024-09-16T14:52:49.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3454003600 con 0x7f3444002830 2024-09-16T14:52:49.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:49.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:52:49.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 10 monclient: dump: 2024-09-16T14:52:49.137 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:49.137 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:49.137 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.137 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.137 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:49.137 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:49.137 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:49.137 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:49.137 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:49.137 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:49.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2267398212 0 0) 0x7f3444006590 con 0x7f3444002830 2024-09-16T14:52:49.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f344400a0e0 con 0x7f3444002830 2024-09-16T14:52:49.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f34540057d0 con 0x7f3444002830 2024-09-16T14:52:49.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:49.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:49.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 >> v1:172.21.15.189:6789/0 conn(0x7f3444002830 legacy=0x7f3444004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:49.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.119+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 >> v1:172.21.15.189:6789/0 conn(0x7f3444002830 legacy=0x7f3444004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:49.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.123+0000 7f3464c6e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:52:49.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.123+0000 7f3464c6e640 1 -- 172.21.15.26:0/1485546349 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3444009d80 con 0x7f34440090a0 2024-09-16T14:52:49.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604486714 0 0) 0x7f3444009d80 con 0x7f34440090a0 2024-09-16T14:52:49.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 10 monclient(hunting): _negotiate 2024-09-16T14:52:49.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:49.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:49.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 10 monclient(hunting): my global_id is 224149 2024-09-16T14:52:49.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3444006590 con 0x7f34440090a0 2024-09-16T14:52:49.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3699868796 0 0) 0x7f3444006590 con 0x7f34440090a0 2024-09-16T14:52:49.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3444009d80 con 0x7f34440090a0 2024-09-16T14:52:49.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3454003f70 con 0x7f34440090a0 2024-09-16T14:52:49.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:49.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:52:49.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 10 monclient: dump: 2024-09-16T14:52:49.140 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:49.140 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:49.140 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.140 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.141 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:49.141 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:49.141 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:49.141 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:49.141 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:49.141 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:49.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4052176866 0 0) 0x7f3444009d80 con 0x7f34440090a0 2024-09-16T14:52:49.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f344400c290 con 0x7f34440090a0 2024-09-16T14:52:49.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 2967140609) 0x7f34540041d0 con 0x7f34440090a0 2024-09-16T14:52:49.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:52:49.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:49.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 >> v1:172.21.15.189:6789/0 conn(0x7f34440090a0 legacy=0x7f3444009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:49.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f344effd640 1 -- 172.21.15.26:0/1485546349 >> v1:172.21.15.189:6789/0 conn(0x7f34440090a0 legacy=0x7f3444009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:49.142 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:52:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:52:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 96, 2024-09-16T14:52:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:52:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:52:49.142 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:52:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:52:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:52:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:52:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:52:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:49.143 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:52:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:52:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:49.144 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:52:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:52:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:52:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:52:49.145 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:52:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:52:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:52:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:52:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:52:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:52:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:52:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:52:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:52:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:52:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:52:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:52:49.146 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:52:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:52:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:52:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:52:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:52:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:52:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:52:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:52:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:52:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:52:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:52:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:49.147 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:52:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:52:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:52:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:52:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:49.148 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:52:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:52:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:52:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:52:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:52:49.149 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:52:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:52:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:52:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:52:49.150 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:52:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:52:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:52:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:52:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:52:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:52:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:52:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:52:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:52:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:52:49.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:52:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:52:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:52:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:52:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:52:49.152 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:52:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:52:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:52:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:52:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:52:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:52:49.153 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:52:49.153 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:52:49.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f3464c6e640 1 -- 172.21.15.26:0/1485546349 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3440036440 legacy=0x7f3440038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:49.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f3464c6e640 10 monclient: shutdown 2024-09-16T14:52:49.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f3464c6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:49.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f3464c6e640 1 -- 172.21.15.26:0/1485546349 >> v1:172.21.15.26:6790/0 conn(0x7f34600a0000 legacy=0x7f34601391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:49.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f345ed9d640 1 -- 172.21.15.26:0/1485546349 reap_dead start 2024-09-16T14:52:49.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f345ed9d640 1 -- 172.21.15.26:0/1485546349 reap_dead start 2024-09-16T14:52:49.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f3464c6e640 1 -- 172.21.15.26:0/1485546349 shutdown_connections 2024-09-16T14:52:49.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f3464c6e640 1 -- 172.21.15.26:0/1485546349 >> 172.21.15.26:0/1485546349 conn(0x7f34600a7f30 msgr2=0x7f346013e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:49.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f3464c6e640 1 -- 172.21.15.26:0/1485546349 shutdown_connections 2024-09-16T14:52:49.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.127+0000 7f3464c6e640 1 -- 172.21.15.26:0/1485546349 wait complete. 2024-09-16T14:52:49.156 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:52:49.156 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T14:52:49.156 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:52:49.156 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T14:52:49.156 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T14:52:49.157 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T14:52:49.157 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:52:49.169 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T14:52:49.169 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:52:49.169 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:52:49.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 10 monclient: get_monmap_and_config 2024-09-16T14:52:49.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 10 monclient: build_initial_monmap 2024-09-16T14:52:49.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 10 monclient: monmap: 2024-09-16T14:52:49.325 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:49.325 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:49.325 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:49.326363+0000 2024-09-16T14:52:49.325 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:49.326363+0000 2024-09-16T14:52:49.325 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:49.325 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:49.326 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:49.326 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:49.326 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:49.326 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:49.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 1 Processor -- start 2024-09-16T14:52:49.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 1 -- start start 2024-09-16T14:52:49.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 10 monclient: init 2024-09-16T14:52:49.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:49.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:52:49.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 10 monclient(hunting): picked mon.noname-b con 0x7f3fa809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:49.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 10 monclient(hunting): picked mon.noname-c con 0x7f3fa8132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:49.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 10 monclient(hunting): picked mon.noname-a con 0x7f3fa8138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:49.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa809d260 con 0x7f3fa8138ff0 2024-09-16T14:52:49.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa8132a80 con 0x7f3fa8132650 2024-09-16T14:52:49.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa809cf60 con 0x7f3fa809cb80 2024-09-16T14:52:49.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:49.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 10 monclient(hunting): authenticate will time out at 1593.446841s 2024-09-16T14:52:49.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa77fe640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f3fa809cb80 legacy=0x7f3fa80717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:52:49.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa7fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3fa8138ff0 0x7f3fa813b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54476/0 (socket says 172.21.15.26:54476) 2024-09-16T14:52:49.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa7fff640 1 -- 172.21.15.26:0/2290500129 learned_addr learned my addr 172.21.15.26:0/2290500129 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:49.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 1 -- 172.21.15.26:0/2290500129 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211720125 0 0) 0x7f3fa809d260 con 0x7f3fa8138ff0 2024-09-16T14:52:49.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:49.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:49.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:49.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 10 monclient(hunting): my global_id is 194397 2024-09-16T14:52:49.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 1 -- 172.21.15.26:0/2290500129 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f94003530 con 0x7f3fa8138ff0 2024-09-16T14:52:49.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 1 -- 172.21.15.26:0/2290500129 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197989636 0 0) 0x7f3fa8132a80 con 0x7f3fa8132650 2024-09-16T14:52:49.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:52:49.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:49.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:49.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 10 monclient(hunting): my global_id is 224152 2024-09-16T14:52:49.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 1 -- 172.21.15.26:0/2290500129 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa809d260 con 0x7f3fa8132650 2024-09-16T14:52:49.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 1 -- 172.21.15.26:0/2290500129 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 984519828 0 0) 0x7f3f94003530 con 0x7f3fa8138ff0 2024-09-16T14:52:49.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 1 -- 172.21.15.26:0/2290500129 >> v1:172.21.15.26:6790/0 conn(0x7f3fa809cb80 legacy=0x7f3fa80717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:52:49.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 1 -- 172.21.15.26:0/2290500129 >> v1:172.21.15.189:6789/0 conn(0x7f3fa8132650 legacy=0x7f3fa81378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:49.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:52:49.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 1 monclient: found mon.noname-a 2024-09-16T14:52:49.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:49.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:52:49.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 1 -- 172.21.15.26:0/2290500129 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa809f2d0 con 0x7f3fa8138ff0 2024-09-16T14:52:49.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 10 monclient: _finish_auth 0 2024-09-16T14:52:49.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:52:49.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:49.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 5 monclient: authenticate success, global_id 194397 2024-09-16T14:52:49.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:49.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 1 -- 172.21.15.26:0/2290500129 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3fa0001620 con 0x7f3fa8138ff0 2024-09-16T14:52:49.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:49.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:49.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fa67fc640 10 monclient: dump: 2024-09-16T14:52:49.333 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:49.333 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:49.333 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.333 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.333 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:49.333 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:49.333 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:49.333 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:49.333 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:49.333 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:49.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.323+0000 7f3fadacc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:49.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fa67fc640 1 -- 172.21.15.26:0/2290500129 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fa0001890 con 0x7f3fa8138ff0 2024-09-16T14:52:49.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fa67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:49.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:49.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fa67fc640 1 -- 172.21.15.26:0/2290500129 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3fa0001c20 con 0x7f3fa8138ff0 2024-09-16T14:52:49.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 4 set_mon_vals no callback set 2024-09-16T14:52:49.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:49.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fa67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:49.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fa67fc640 10 monclient: dump: 2024-09-16T14:52:49.334 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:49.335 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:49.335 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.335 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.335 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:49.335 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:49.335 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:49.335 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:49.335 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:49.335 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:49.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 10 monclient: shutdown 2024-09-16T14:52:49.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:49.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 1 -- 172.21.15.26:0/2290500129 >> v1:172.21.15.26:6789/0 conn(0x7f3fa8138ff0 legacy=0x7f3fa813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:49.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 1 -- 172.21.15.26:0/2290500129 shutdown_connections 2024-09-16T14:52:49.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 1 -- 172.21.15.26:0/2290500129 >> 172.21.15.26:0/2290500129 conn(0x7f3fa809fb50 msgr2=0x7f3fa809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:49.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 1 -- 172.21.15.26:0/2290500129 shutdown_connections 2024-09-16T14:52:49.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 1 -- 172.21.15.26:0/2290500129 wait complete. 2024-09-16T14:52:49.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 10 monclient: build_initial_monmap 2024-09-16T14:52:49.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 10 monclient: monmap: 2024-09-16T14:52:49.336 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:49.336 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:49.336 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:49.336 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:49.336 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:49.336 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:49.336 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:49.337 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:49.337 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:49.337 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:49.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 1 Processor -- start 2024-09-16T14:52:49.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 1 -- start start 2024-09-16T14:52:49.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 10 monclient: init 2024-09-16T14:52:49.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:49.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:52:49.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 10 monclient(hunting): picked mon.noname-a con 0x7f3fa809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:49.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 10 monclient(hunting): picked mon.noname-c con 0x7f3fa8132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:49.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 10 monclient(hunting): picked mon.noname-b con 0x7f3fa8138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:49.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa806f740 con 0x7f3fa809cb80 2024-09-16T14:52:49.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa806d880 con 0x7f3fa8138ff0 2024-09-16T14:52:49.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa80a29f0 con 0x7f3fa8132650 2024-09-16T14:52:49.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:49.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 10 monclient(hunting): authenticate will time out at 1593.450241s 2024-09-16T14:52:49.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fa6ffd640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f3fa8132650 legacy=0x7f3fa81d2ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:52:49.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fa77fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3fa809cb80 0x7f3fa812c180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54482/0 (socket says 172.21.15.26:54482) 2024-09-16T14:52:49.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fa77fe640 1 -- 172.21.15.26:0/1442191858 learned_addr learned my addr 172.21.15.26:0/1442191858 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:49.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fa77fe640 1 -- 172.21.15.26:0/1442191858 >> v1:172.21.15.26:6789/0 conn(0x7f3fa809cb80 legacy=0x7f3fa812c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T14:52:49.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fa77fe640 1 -- 172.21.15.26:0/1442191858 >> v1:172.21.15.26:6789/0 conn(0x7f3fa809cb80 legacy=0x7f3fa812c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:52:49.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fa77fe640 1 --1- 172.21.15.26:0/1442191858 >> v1:172.21.15.26:6789/0 conn(0x7f3fa809cb80 0x7f3fa812c180 :-1 s=OPENED pgs=804 cs=1 l=1).handle_message read tag failed 2024-09-16T14:52:49.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fa77fe640 1 --1- 172.21.15.26:0/1442191858 >> v1:172.21.15.26:6789/0 conn(0x7f3fa809cb80 0x7f3fa812c180 :-1 s=OPENED pgs=804 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:52:49.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6789/0 2024-09-16T14:52:49.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 1 -- 172.21.15.26:0/1442191858 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 108017626 0 0) 0x7f3fa806d880 con 0x7f3fa8138ff0 2024-09-16T14:52:49.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:49.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:49.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:49.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 10 monclient(hunting): my global_id is 224155 2024-09-16T14:52:49.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 1 -- 172.21.15.26:0/1442191858 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f88003530 con 0x7f3fa8138ff0 2024-09-16T14:52:49.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 1 -- 172.21.15.26:0/1442191858 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3157729794 0 0) 0x7f3f88003530 con 0x7f3fa8138ff0 2024-09-16T14:52:49.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 1 -- 172.21.15.26:0/1442191858 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fa806d880 con 0x7f3fa8138ff0 2024-09-16T14:52:49.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 1 -- 172.21.15.26:0/1442191858 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3fa0001ba0 con 0x7f3fa8138ff0 2024-09-16T14:52:49.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:49.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:52:49.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 10 monclient(hunting): dump: 2024-09-16T14:52:49.342 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:49.342 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:49.342 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.342 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.343 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:49.343 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:49.343 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:49.343 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:49.343 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:49.343 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:49.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:49.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 1 -- 172.21.15.26:0/1442191858 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2442313033 0 0) 0x7f3fa806d880 con 0x7f3fa8138ff0 2024-09-16T14:52:49.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 1 -- 172.21.15.26:0/1442191858 >> v1:172.21.15.26:6790/0 conn(0x7f3fa8132650 legacy=0x7f3fa81d2ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:52:49.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 1 -- 172.21.15.26:0/1442191858 >> v1:172.21.15.26:6789/0 conn(0x7f3fa809cb80 legacy=0x7f3fa812c180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:52:49.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:52:49.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 1 monclient: found mon.b 2024-09-16T14:52:49.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:49.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:52:49.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 1 -- 172.21.15.26:0/1442191858 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa8071ac0 con 0x7f3fa8138ff0 2024-09-16T14:52:49.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 10 monclient: _finish_auth 0 2024-09-16T14:52:49.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:52:49.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3f9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:49.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 5 monclient: authenticate success, global_id 224155 2024-09-16T14:52:49.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 10 monclient: _renew_subs 2024-09-16T14:52:49.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:52:49.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 1 -- 172.21.15.26:0/1442191858 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa809a240 con 0x7f3fa8138ff0 2024-09-16T14:52:49.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 10 monclient: _renew_subs 2024-09-16T14:52:49.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:52:49.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.327+0000 7f3fadacc640 1 -- 172.21.15.26:0/1442191858 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa812c890 con 0x7f3fa8138ff0 2024-09-16T14:52:49.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.331+0000 7f3f9ffff640 1 -- 172.21.15.26:0/1442191858 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fa0004aa0 con 0x7f3fa8138ff0 2024-09-16T14:52:49.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.331+0000 7f3f9ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:49.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.331+0000 7f3f9ffff640 1 -- 172.21.15.26:0/1442191858 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3fa0004db0 con 0x7f3fa8138ff0 2024-09-16T14:52:49.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.331+0000 7f3f9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:49.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.331+0000 7f3f9ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:52:49.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.331+0000 7f3f9ffff640 10 monclient: dump: 2024-09-16T14:52:49.345 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:49.346 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:49.346 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.346 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:49.346 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:49.346 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:49.346 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:49.346 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:49.346 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:49.346 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:49.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.331+0000 7f3fa5ffb640 4 set_mon_vals no callback set 2024-09-16T14:52:49.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.331+0000 7f3fadacc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:49.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.331+0000 7f3fadacc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:49.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.331+0000 7f3fadacc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:52:49.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.331+0000 7f3fadacc640 1 -- 172.21.15.26:0/1442191858 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f8c005050 con 0x7f3fa8138ff0 2024-09-16T14:52:49.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.331+0000 7f3f9ffff640 1 -- 172.21.15.26:0/1442191858 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3fa000f820 con 0x7f3fa8138ff0 2024-09-16T14:52:49.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.331+0000 7f3f9ffff640 1 -- 172.21.15.26:0/1442191858 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f3fa0043ca0 con 0x7f3fa8138ff0 2024-09-16T14:52:49.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.331+0000 7f3fa77fe640 10 monclient: get_auth_request con 0x7f3f88036030 auth_method 0 2024-09-16T14:52:49.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.343+0000 7f3f9ffff640 1 -- 172.21.15.26:0/1442191858 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3fa0014620 con 0x7f3fa8138ff0 2024-09-16T14:52:49.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.343+0000 7f3f9ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:49.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.343+0000 7f3f9ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:49.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.559+0000 7f3fadacc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:52:49.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.559+0000 7f3fadacc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:49.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.559+0000 7f3fadacc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:52:49.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.559+0000 7f3fadacc640 1 -- 172.21.15.26:0/1442191858 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3f8c005880 con 0x7f3fa8138ff0 2024-09-16T14:52:49.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.559+0000 7f3f9ffff640 1 -- 172.21.15.26:0/1442191858 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 4087680710) 0x7f3fa0012cc0 con 0x7f3fa8138ff0 2024-09-16T14:52:49.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.559+0000 7f3f9ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:49.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.559+0000 7f3f9ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:49.562 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:52:49.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.559+0000 7f3fadacc640 1 -- 172.21.15.26:0/1442191858 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3f88036030 legacy=0x7f3f880384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:49.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.559+0000 7f3fadacc640 10 monclient: shutdown 2024-09-16T14:52:49.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.559+0000 7f3fadacc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:49.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.559+0000 7f3fadacc640 1 -- 172.21.15.26:0/1442191858 >> v1:172.21.15.189:6789/0 conn(0x7f3fa8138ff0 legacy=0x7f3fa81d62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:49.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.559+0000 7f3fadacc640 1 -- 172.21.15.26:0/1442191858 shutdown_connections 2024-09-16T14:52:49.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.559+0000 7f3fadacc640 1 -- 172.21.15.26:0/1442191858 >> 172.21.15.26:0/1442191858 conn(0x7f3fa809fb50 msgr2=0x7f3fa80a0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:49.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.559+0000 7f3fadacc640 1 -- 172.21.15.26:0/1442191858 shutdown_connections 2024-09-16T14:52:49.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:49.559+0000 7f3fadacc640 1 -- 172.21.15.26:0/1442191858 wait complete. 2024-09-16T14:52:52.581 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:52:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 10 monclient: get_monmap_and_config 2024-09-16T14:52:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 10 monclient: build_initial_monmap 2024-09-16T14:52:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 10 monclient: monmap: 2024-09-16T14:52:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:52.718055+0000 2024-09-16T14:52:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:52.718055+0000 2024-09-16T14:52:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:52.717 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:52.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 1 Processor -- start 2024-09-16T14:52:52.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 1 -- start start 2024-09-16T14:52:52.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 10 monclient: init 2024-09-16T14:52:52.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:52.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:52:52.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 10 monclient(hunting): picked mon.noname-a con 0x7f60b409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:52.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 10 monclient(hunting): picked mon.noname-c con 0x7f60b4132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:52.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 10 monclient(hunting): picked mon.noname-b con 0x7f60b4138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:52.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b409d260 con 0x7f60b409cb80 2024-09-16T14:52:52.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b4132a80 con 0x7f60b4132650 2024-09-16T14:52:52.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b409cf60 con 0x7f60b4138ff0 2024-09-16T14:52:52.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:52.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 10 monclient(hunting): authenticate will time out at 1596.838274s 2024-09-16T14:52:52.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b99e7640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f60b4138ff0 legacy=0x7f60b413b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:52:52.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b91e6640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f60b409cb80 0x7f60b40717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54498/0 (socket says 172.21.15.26:54498) 2024-09-16T14:52:52.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b91e6640 1 -- 172.21.15.26:0/3178457514 learned_addr learned my addr 172.21.15.26:0/3178457514 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:52.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 1 -- 172.21.15.26:0/3178457514 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525995541 0 0) 0x7f60b409d260 con 0x7f60b409cb80 2024-09-16T14:52:52.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:52:52.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:52.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:52.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 10 monclient(hunting): my global_id is 194400 2024-09-16T14:52:52.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 1 -- 172.21.15.26:0/3178457514 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f609c003530 con 0x7f60b409cb80 2024-09-16T14:52:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 1 -- 172.21.15.26:0/3178457514 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 75041320 0 0) 0x7f609c003530 con 0x7f60b409cb80 2024-09-16T14:52:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 1 -- 172.21.15.26:0/3178457514 >> v1:172.21.15.26:6790/0 conn(0x7f60b4138ff0 legacy=0x7f60b413b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:52:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 1 -- 172.21.15.26:0/3178457514 >> v1:172.21.15.189:6789/0 conn(0x7f60b4132650 legacy=0x7f60b41378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:52:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 1 monclient: found mon.noname-a 2024-09-16T14:52:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:52:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 1 -- 172.21.15.26:0/3178457514 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b409f2d0 con 0x7f60b409cb80 2024-09-16T14:52:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 10 monclient: _finish_auth 0 2024-09-16T14:52:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:52:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 1 -- 172.21.15.26:0/3178457514 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f60ac0019a0 con 0x7f60b409cb80 2024-09-16T14:52:52.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:52.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:52.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 10 monclient: dump: 2024-09-16T14:52:52.724 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:52.724 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:52.724 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:52.724 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:52.724 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:52.724 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:52.724 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:52.724 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:52.724 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:52.724 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 5 monclient: authenticate success, global_id 194400 2024-09-16T14:52:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:52:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 1 -- 172.21.15.26:0/3178457514 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60ac001c50 con 0x7f60b409cb80 2024-09-16T14:52:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 1 -- 172.21.15.26:0/3178457514 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f60ac001fe0 con 0x7f60b409cb80 2024-09-16T14:52:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 10 monclient: get_monmap_and_config success 2024-09-16T14:52:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 4 set_mon_vals no callback set 2024-09-16T14:52:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60b3fff640 10 monclient: dump: 2024-09-16T14:52:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:52.725 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:52.726 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:52.726 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:52.726 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:52.726 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:52.726 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:52.726 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:52.726 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:52.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 10 monclient: shutdown 2024-09-16T14:52:52.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:52.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 1 -- 172.21.15.26:0/3178457514 >> v1:172.21.15.26:6789/0 conn(0x7f60b409cb80 legacy=0x7f60b40717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:52.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 1 -- 172.21.15.26:0/3178457514 shutdown_connections 2024-09-16T14:52:52.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 1 -- 172.21.15.26:0/3178457514 >> 172.21.15.26:0/3178457514 conn(0x7f60b409fb50 msgr2=0x7f60b409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:52.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 1 -- 172.21.15.26:0/3178457514 shutdown_connections 2024-09-16T14:52:52.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.715+0000 7f60bb44a640 1 -- 172.21.15.26:0/3178457514 wait complete. 2024-09-16T14:52:52.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 monclient: build_initial_monmap 2024-09-16T14:52:52.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 monclient: monmap: 2024-09-16T14:52:52.727 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:52.727 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:52:52.727 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:52:52.727 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:52:52.727 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:52.727 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:52:52.727 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:52.727 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:52:52.728 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:52:52.728 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:52.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 1 Processor -- start 2024-09-16T14:52:52.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 1 -- start start 2024-09-16T14:52:52.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 monclient: init 2024-09-16T14:52:52.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:52.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:52:52.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 monclient(hunting): picked mon.noname-c con 0x7f60b409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:52.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 monclient(hunting): picked mon.noname-b con 0x7f60b4132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:52.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 monclient(hunting): picked mon.noname-a con 0x7f60b4138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:52.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b40a2270 con 0x7f60b4138ff0 2024-09-16T14:52:52.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b4070170 con 0x7f60b4132650 2024-09-16T14:52:52.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b406ff40 con 0x7f60b409cb80 2024-09-16T14:52:52.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:52.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 monclient(hunting): authenticate will time out at 1596.841255s 2024-09-16T14:52:52.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b91e6640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f60b409cb80 legacy=0x7f60b406d660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:52:52.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b99e7640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f60b4138ff0 0x7f60b406e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54504/0 (socket says 172.21.15.26:54504) 2024-09-16T14:52:52.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b99e7640 1 -- 172.21.15.26:0/4137713611 learned_addr learned my addr 172.21.15.26:0/4137713611 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:52.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359073477 0 0) 0x7f60b40a2270 con 0x7f60b4138ff0 2024-09-16T14:52:52.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:52:52.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:52.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:52.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient(hunting): my global_id is 194403 2024-09-16T14:52:52.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60a0003530 con 0x7f60b4138ff0 2024-09-16T14:52:52.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 654093075 0 0) 0x7f60a0003530 con 0x7f60b4138ff0 2024-09-16T14:52:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60b40a2270 con 0x7f60b4138ff0 2024-09-16T14:52:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f60a4004040 con 0x7f60b4138ff0 2024-09-16T14:52:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:52:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient(hunting): dump: 2024-09-16T14:52:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:52.733 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:52.733 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:52.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:52:52.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61045153 0 0) 0x7f60b4070170 con 0x7f60b4132650 2024-09-16T14:52:52.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:52:52.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:52:52.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:52:52.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient(hunting): my global_id is 224161 2024-09-16T14:52:52.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60a0003530 con 0x7f60b4132650 2024-09-16T14:52:52.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1638416257 0 0) 0x7f60b40a2270 con 0x7f60b4138ff0 2024-09-16T14:52:52.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 >> v1:172.21.15.26:6790/0 conn(0x7f60b409cb80 legacy=0x7f60b406d660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:52:52.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 >> v1:172.21.15.189:6789/0 conn(0x7f60b4132650 legacy=0x7f60b406dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:52.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:52:52.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 1 monclient: found mon.a 2024-09-16T14:52:52.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:52:52.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:52.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b413c5e0 con 0x7f60b4138ff0 2024-09-16T14:52:52.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:52:52.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:52:52.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:52:52.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 5 monclient: authenticate success, global_id 194403 2024-09-16T14:52:52.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 monclient: _renew_subs 2024-09-16T14:52:52.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 1 -- 172.21.15.26:0/4137713611 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b409a240 con 0x7f60b4138ff0 2024-09-16T14:52:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 monclient: _renew_subs 2024-09-16T14:52:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 1 -- 172.21.15.26:0/4137713611 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b406f260 con 0x7f60b4138ff0 2024-09-16T14:52:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60a40042f0 con 0x7f60b4138ff0 2024-09-16T14:52:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:52:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f60a4004680 con 0x7f60b4138ff0 2024-09-16T14:52:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:52:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:52:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b2ffd640 4 set_mon_vals no callback set 2024-09-16T14:52:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 10 monclient: dump: 2024-09-16T14:52:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:52:52.735 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:52.736 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:52.736 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:52:52.736 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:52:52.736 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:52.736 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:52:52.736 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:52:52.736 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:52:52.736 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:52.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:52.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:52.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:52.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f60a40048a0 con 0x7f60b4138ff0 2024-09-16T14:52:52.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60bb44a640 1 -- 172.21.15.26:0/4137713611 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6094005050 con 0x7f60b4138ff0 2024-09-16T14:52:52.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f60a4043d40 con 0x7f60b4138ff0 2024-09-16T14:52:52.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.719+0000 7f60b91e6640 10 monclient: get_auth_request con 0x7f60a0036100 auth_method 0 2024-09-16T14:52:52.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.727+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f60a40146d0 con 0x7f60b4138ff0 2024-09-16T14:52:52.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.727+0000 7f60b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:52:52.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.727+0000 7f60b1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:52:52.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.947+0000 7f60bb44a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:52:52.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.947+0000 7f60bb44a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:52.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.947+0000 7f60bb44a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:52:52.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.947+0000 7f60bb44a640 1 -- 172.21.15.26:0/4137713611 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6094005880 con 0x7f60b4138ff0 2024-09-16T14:52:52.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.947+0000 7f60b1ffb640 1 -- 172.21.15.26:0/4137713611 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2739519719) 0x7f60a4012b40 con 0x7f60b4138ff0 2024-09-16T14:52:52.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.947+0000 7f60b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:52:52.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.947+0000 7f60b1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:52:52.951 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:52:52.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.951+0000 7f60bb44a640 1 -- 172.21.15.26:0/4137713611 >> v1:172.21.15.189:6812/3196161582 conn(0x7f60a0036100 legacy=0x7f60a00385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:52.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.951+0000 7f60bb44a640 10 monclient: shutdown 2024-09-16T14:52:52.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.951+0000 7f60bb44a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:52:52.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.951+0000 7f60bb44a640 1 -- 172.21.15.26:0/4137713611 >> v1:172.21.15.26:6789/0 conn(0x7f60b4138ff0 legacy=0x7f60b406e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:52.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.951+0000 7f60bb44a640 1 -- 172.21.15.26:0/4137713611 shutdown_connections 2024-09-16T14:52:52.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.951+0000 7f60bb44a640 1 -- 172.21.15.26:0/4137713611 >> 172.21.15.26:0/4137713611 conn(0x7f60b409fb50 msgr2=0x7f60b4130680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:52:52.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.951+0000 7f60bb44a640 1 -- 172.21.15.26:0/4137713611 shutdown_connections 2024-09-16T14:52:52.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:52.951+0000 7f60bb44a640 1 -- 172.21.15.26:0/4137713611 wait complete. 2024-09-16T14:52:55.971 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:52:56.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd7d7d640 10 monclient: get_monmap_and_config 2024-09-16T14:52:56.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd7d7d640 10 monclient: build_initial_monmap 2024-09-16T14:52:56.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd7d7d640 10 monclient: monmap: 2024-09-16T14:52:56.105 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:52:56.105 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:52:56.105 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:52:56.106221+0000 2024-09-16T14:52:56.105 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:52:56.106221+0000 2024-09-16T14:52:56.105 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:52:56.105 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:52:56.105 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:52:56.105 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:52:56.105 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:52:56.106 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:52:56.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd7d7d640 1 Processor -- start 2024-09-16T14:52:56.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd7d7d640 1 -- start start 2024-09-16T14:52:56.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd7d7d640 10 monclient: init 2024-09-16T14:52:56.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd7d7d640 10 monclient: _reopen_session rank -1 2024-09-16T14:52:56.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd7d7d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:52:56.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd7d7d640 10 monclient(hunting): picked mon.noname-b con 0x7f1fd00a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:56.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd7d7d640 10 monclient(hunting): picked mon.noname-a con 0x7f1fd00a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:56.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd7d7d640 10 monclient(hunting): picked mon.noname-c con 0x7f1fd00a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:56.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd7d7d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd00a3250 con 0x7f1fd00a0000 2024-09-16T14:52:56.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd7d7d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd009f600 con 0x7f1fd00a2800 2024-09-16T14:52:56.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd7d7d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd00a3830 con 0x7f1fd00a1d10 2024-09-16T14:52:56.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd7d7d640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:56.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd7d7d640 10 monclient(hunting): authenticate will time out at 1600.226968s 2024-09-16T14:52:56.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd5b19640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f1fd00a1d10 legacy=0x7f1fd00a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:52:56.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd5318640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1fd00a0000 0x7f1fd0139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54516/0 (socket says 172.21.15.26:54516) 2024-09-16T14:52:56.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.103+0000 7f1fd5318640 1 -- 172.21.15.26:0/592344187 learned_addr learned my addr 172.21.15.26:0/592344187 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:52:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.307+0000 7f1fd5b19640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6790/0 conn(0x7f1fd00a1d10 legacy=0x7f1fd00a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:52:56.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:56.711+0000 7f1fd5b19640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6790/0 conn(0x7f1fd00a1d10 legacy=0x7f1fd00a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:52:57.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:57.515+0000 7f1fd5b19640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6790/0 conn(0x7f1fd00a1d10 legacy=0x7f1fd00a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:52:59.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.107+0000 7f1fc77fe640 10 monclient(hunting): tick 2024-09-16T14:52:59.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.107+0000 7f1fc77fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:52:59.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.107+0000 7f1fc77fe640 1 monclient(hunting): continuing hunt 2024-09-16T14:52:59.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.107+0000 7f1fc77fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:52:59.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.107+0000 7f1fc77fe640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6790/0 conn(0x7f1fd00a1d10 legacy=0x7f1fd00a20f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:52:59.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.107+0000 7f1fc77fe640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.189:6789/0 conn(0x7f1fd00a2800 legacy=0x7f1fd013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:59.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.107+0000 7f1fc77fe640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6789/0 conn(0x7f1fd00a0000 legacy=0x7f1fd0139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:52:59.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.107+0000 7f1fc77fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:52:59.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.107+0000 7f1fc77fe640 10 monclient(hunting): picked mon.noname-b con 0x7f1fb40036d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:52:59.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.107+0000 7f1fc77fe640 10 monclient(hunting): picked mon.noname-a con 0x7f1fb4007360 addr v1:172.21.15.26:6789/0 2024-09-16T14:52:59.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.107+0000 7f1fc77fe640 10 monclient(hunting): picked mon.noname-c con 0x7f1fd00a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:52:59.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.107+0000 7f1fc77fe640 1 -- 172.21.15.26:0/592344187 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd009ed60 con 0x7f1fb4007360 2024-09-16T14:52:59.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.107+0000 7f1fc77fe640 1 -- 172.21.15.26:0/592344187 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd00a3830 con 0x7f1fd00a2800 2024-09-16T14:52:59.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.107+0000 7f1fc77fe640 1 -- 172.21.15.26:0/592344187 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb40108b0 con 0x7f1fb40036d0 2024-09-16T14:52:59.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.107+0000 7f1fc77fe640 10 monclient(hunting): _renew_subs 2024-09-16T14:52:59.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.107+0000 7f1fd5b19640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6790/0 conn(0x7f1fb40036d0 legacy=0x7f1fb4005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:52:59.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.311+0000 7f1fd5b19640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6790/0 conn(0x7f1fb40036d0 legacy=0x7f1fb4005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:52:59.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:52:59.715+0000 7f1fd5b19640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6790/0 conn(0x7f1fb40036d0 legacy=0x7f1fb4005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:53:00.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:00.519+0000 7f1fd5b19640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6790/0 conn(0x7f1fb40036d0 legacy=0x7f1fb4005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:53:02.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.107+0000 7f1fc77fe640 10 monclient(hunting): tick 2024-09-16T14:53:02.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.107+0000 7f1fc77fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:53:02.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.107+0000 7f1fc77fe640 1 monclient(hunting): continuing hunt 2024-09-16T14:53:02.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.107+0000 7f1fc77fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:53:02.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.107+0000 7f1fc77fe640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6790/0 conn(0x7f1fb40036d0 legacy=0x7f1fb4005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:53:02.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.107+0000 7f1fc77fe640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.189:6789/0 conn(0x7f1fd00a2800 legacy=0x7f1fb400d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:02.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.107+0000 7f1fc77fe640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6789/0 conn(0x7f1fb4007360 legacy=0x7f1fb40097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:02.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.107+0000 7f1fc77fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:53:02.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.107+0000 7f1fc77fe640 10 monclient(hunting): picked mon.noname-a con 0x7f1fb4010b40 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:02.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.107+0000 7f1fc77fe640 10 monclient(hunting): picked mon.noname-c con 0x7f1fb4012dc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:02.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.107+0000 7f1fc77fe640 10 monclient(hunting): picked mon.noname-b con 0x7f1fb40036d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:02.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.107+0000 7f1fc77fe640 1 -- 172.21.15.26:0/592344187 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb4011ac0 con 0x7f1fb4010b40 2024-09-16T14:53:02.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.107+0000 7f1fc77fe640 1 -- 172.21.15.26:0/592344187 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb40108b0 con 0x7f1fb4012dc0 2024-09-16T14:53:02.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.107+0000 7f1fc77fe640 1 -- 172.21.15.26:0/592344187 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb400d830 con 0x7f1fb40036d0 2024-09-16T14:53:02.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.107+0000 7f1fc77fe640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:02.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.107+0000 7f1fd631a640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6790/0 conn(0x7f1fb40036d0 legacy=0x7f1fb4005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:53:02.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.311+0000 7f1fd631a640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6790/0 conn(0x7f1fb40036d0 legacy=0x7f1fb4005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:53:02.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:02.715+0000 7f1fd631a640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6790/0 conn(0x7f1fb40036d0 legacy=0x7f1fb4005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:53:03.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.519+0000 7f1fd631a640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6790/0 conn(0x7f1fb40036d0 legacy=0x7f1fb4005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:53:03.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 1 -- 172.21.15.26:0/592344187 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220608482 0 0) 0x7f1fb4011ac0 con 0x7f1fb4010b40 2024-09-16T14:53:03.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 10 monclient(hunting): _negotiate 2024-09-16T14:53:03.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:03.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:03.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 10 monclient(hunting): my global_id is 194406 2024-09-16T14:53:03.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 1 -- 172.21.15.26:0/592344187 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fbc003530 con 0x7f1fb4010b40 2024-09-16T14:53:03.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 1 -- 172.21.15.26:0/592344187 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4112704962 0 0) 0x7f1fbc003530 con 0x7f1fb4010b40 2024-09-16T14:53:03.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6790/0 conn(0x7f1fb40036d0 legacy=0x7f1fb4005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:53:03.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.189:6789/0 conn(0x7f1fb4012dc0 legacy=0x7f1fb40151b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:03.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 10 monclient: _finish_hunting 0 2024-09-16T14:53:03.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 1 monclient: found mon.noname-a 2024-09-16T14:53:03.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:03.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:53:03.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 1 -- 172.21.15.26:0/592344187 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb400da10 con 0x7f1fb4010b40 2024-09-16T14:53:03.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 10 monclient: _finish_auth 0 2024-09-16T14:53:03.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 10 monclient: _check_auth_tickets 2024-09-16T14:53:03.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 1 -- 172.21.15.26:0/592344187 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1fcc004240 con 0x7f1fb4010b40 2024-09-16T14:53:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:53:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 10 monclient: dump: 2024-09-16T14:53:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:03.611 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd7d7d640 5 monclient: authenticate success, global_id 194406 2024-09-16T14:53:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd7d7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 1 -- 172.21.15.26:0/592344187 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fcc004460 con 0x7f1fb4010b40 2024-09-16T14:53:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 1 -- 172.21.15.26:0/592344187 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1fcc004870 con 0x7f1fb4010b40 2024-09-16T14:53:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd7d7d640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd7d7d640 4 set_mon_vals no callback set 2024-09-16T14:53:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.603+0000 7f1fd4b17640 10 monclient: dump: 2024-09-16T14:53:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:03.612 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:03.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.607+0000 7f1fd7d7d640 10 monclient: shutdown 2024-09-16T14:53:03.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.607+0000 7f1fd7d7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:03.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.607+0000 7f1fd7d7d640 1 -- 172.21.15.26:0/592344187 >> v1:172.21.15.26:6789/0 conn(0x7f1fb4010b40 legacy=0x7f1fb4010f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:03.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.607+0000 7f1fd7d7d640 1 -- 172.21.15.26:0/592344187 shutdown_connections 2024-09-16T14:53:03.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.607+0000 7f1fd7d7d640 1 -- 172.21.15.26:0/592344187 >> 172.21.15.26:0/592344187 conn(0x7f1fd00a7f30 msgr2=0x7f1fd00a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:03.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.607+0000 7f1fd7d7d640 1 -- 172.21.15.26:0/592344187 shutdown_connections 2024-09-16T14:53:03.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.607+0000 7f1fd7d7d640 1 -- 172.21.15.26:0/592344187 wait complete. 2024-09-16T14:53:03.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.607+0000 7f1fd7d7d640 10 monclient: build_initial_monmap 2024-09-16T14:53:03.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.607+0000 7f1fd7d7d640 10 monclient: monmap: 2024-09-16T14:53:03.615 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:03.616 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:03.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.607+0000 7f1fd7d7d640 1 Processor -- start 2024-09-16T14:53:03.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.607+0000 7f1fd7d7d640 1 -- start start 2024-09-16T14:53:03.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.607+0000 7f1fd7d7d640 10 monclient: init 2024-09-16T14:53:03.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.611+0000 7f1fd7d7d640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:03.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.611+0000 7f1fd7d7d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:53:03.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.611+0000 7f1fd7d7d640 10 monclient(hunting): picked mon.noname-c con 0x7f1fb40036d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:03.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.611+0000 7f1fd7d7d640 10 monclient(hunting): picked mon.noname-b con 0x7f1fb4010b40 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:03.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.611+0000 7f1fd5b19640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f1fb40036d0 legacy=0x7f1fd0135010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:53:03.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.611+0000 7f1fd7d7d640 10 monclient(hunting): picked mon.noname-a con 0x7f1fb4012dc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:03.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.611+0000 7f1fd7d7d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd0121240 con 0x7f1fb4012dc0 2024-09-16T14:53:03.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.611+0000 7f1fd7d7d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd009ce40 con 0x7f1fb4010b40 2024-09-16T14:53:03.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.611+0000 7f1fd7d7d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd00a6a00 con 0x7f1fb40036d0 2024-09-16T14:53:03.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.611+0000 7f1fd7d7d640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:03.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.611+0000 7f1fd7d7d640 10 monclient(hunting): authenticate will time out at 1607.733681s 2024-09-16T14:53:03.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.611+0000 7f1fd631a640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1fb4012dc0 0x7f1fd01d1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40918/0 (socket says 172.21.15.26:40918) 2024-09-16T14:53:03.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.611+0000 7f1fd631a640 1 -- 172.21.15.26:0/3725579363 learned_addr learned my addr 172.21.15.26:0/3725579363 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 1 -- 172.21.15.26:0/3725579363 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1128236705 0 0) 0x7f1fd0121240 con 0x7f1fb4012dc0 2024-09-16T14:53:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 10 monclient(hunting): _negotiate 2024-09-16T14:53:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 10 monclient(hunting): my global_id is 194409 2024-09-16T14:53:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 1 -- 172.21.15.26:0/3725579363 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb400d830 con 0x7f1fb4012dc0 2024-09-16T14:53:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 1 -- 172.21.15.26:0/3725579363 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 600637597 0 0) 0x7f1fb400d830 con 0x7f1fb4012dc0 2024-09-16T14:53:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 1 -- 172.21.15.26:0/3725579363 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fd0121240 con 0x7f1fb4012dc0 2024-09-16T14:53:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 1 -- 172.21.15.26:0/3725579363 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1fc8001b80 con 0x7f1fb4012dc0 2024-09-16T14:53:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:53:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 10 monclient(hunting): dump: 2024-09-16T14:53:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:03.622 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 1 -- 172.21.15.26:0/3725579363 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1373710780 0 0) 0x7f1fd0121240 con 0x7f1fb4012dc0 2024-09-16T14:53:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 1 -- 172.21.15.26:0/3725579363 >> v1:172.21.15.26:6790/0 conn(0x7f1fb40036d0 legacy=0x7f1fd0135010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:53:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 1 -- 172.21.15.26:0/3725579363 >> v1:172.21.15.189:6789/0 conn(0x7f1fb4010b40 legacy=0x7f1fd01ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 10 monclient: _finish_hunting 0 2024-09-16T14:53:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 1 monclient: found mon.a 2024-09-16T14:53:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 1 -- 172.21.15.26:0/3725579363 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd0122410 con 0x7f1fb4012dc0 2024-09-16T14:53:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 10 monclient: _finish_auth 0 2024-09-16T14:53:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 10 monclient: _check_auth_tickets 2024-09-16T14:53:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fd7d7d640 5 monclient: authenticate success, global_id 194409 2024-09-16T14:53:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fd7d7d640 10 monclient: _renew_subs 2024-09-16T14:53:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fd7d7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fd7d7d640 1 -- 172.21.15.26:0/3725579363 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd01226f0 con 0x7f1fb4012dc0 2024-09-16T14:53:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fd7d7d640 10 monclient: _renew_subs 2024-09-16T14:53:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 1 -- 172.21.15.26:0/3725579363 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fc8003980 con 0x7f1fb4012dc0 2024-09-16T14:53:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fd7d7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fd7d7d640 1 -- 172.21.15.26:0/3725579363 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd0120670 con 0x7f1fb4012dc0 2024-09-16T14:53:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 1 -- 172.21.15.26:0/3725579363 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1fc8003c90 con 0x7f1fb4012dc0 2024-09-16T14:53:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 10 monclient: dump: 2024-09-16T14:53:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:03.625 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc7fff640 4 set_mon_vals no callback set 2024-09-16T14:53:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fd7d7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fd7d7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fd7d7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fd7d7d640 1 -- 172.21.15.26:0/3725579363 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd0067960 con 0x7f1fb4012dc0 2024-09-16T14:53:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 1 -- 172.21.15.26:0/3725579363 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1fc8003df0 con 0x7f1fb4012dc0 2024-09-16T14:53:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fc67fc640 1 -- 172.21.15.26:0/3725579363 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f1fc8044090 con 0x7f1fb4012dc0 2024-09-16T14:53:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.615+0000 7f1fd5b19640 10 monclient: get_auth_request con 0x7f1fb40379b0 auth_method 0 2024-09-16T14:53:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.619+0000 7f1fc67fc640 1 -- 172.21.15.26:0/3725579363 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1fc8014970 con 0x7f1fb4012dc0 2024-09-16T14:53:03.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.619+0000 7f1fc67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:03.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.619+0000 7f1fc67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.847+0000 7f1fd7d7d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:53:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.847+0000 7f1fd7d7d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:53:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.847+0000 7f1fd7d7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.847+0000 7f1fd7d7d640 1 -- 172.21.15.26:0/3725579363 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1fd0073970 con 0x7f1fb4012dc0 2024-09-16T14:53:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.847+0000 7f1fc67fc640 1 -- 172.21.15.26:0/3725579363 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2325304191) 0x7f1fc8012de0 con 0x7f1fb4012dc0 2024-09-16T14:53:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.847+0000 7f1fc67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:53:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.847+0000 7f1fc67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:03.850 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":98,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:53:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.847+0000 7f1fd7d7d640 1 -- 172.21.15.26:0/3725579363 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1fb40379b0 legacy=0x7f1fb4039e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.847+0000 7f1fd7d7d640 10 monclient: shutdown 2024-09-16T14:53:03.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.847+0000 7f1fd7d7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:03.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.847+0000 7f1fd7d7d640 1 -- 172.21.15.26:0/3725579363 >> v1:172.21.15.26:6789/0 conn(0x7f1fb4012dc0 legacy=0x7f1fd01d1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:03.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.847+0000 7f1fd7d7d640 1 -- 172.21.15.26:0/3725579363 shutdown_connections 2024-09-16T14:53:03.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.847+0000 7f1fd7d7d640 1 -- 172.21.15.26:0/3725579363 >> 172.21.15.26:0/3725579363 conn(0x7f1fd00a7f30 msgr2=0x7f1fd00a5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:03.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.847+0000 7f1fd7d7d640 1 -- 172.21.15.26:0/3725579363 shutdown_connections 2024-09-16T14:53:03.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.847+0000 7f1fd7d7d640 1 -- 172.21.15.26:0/3725579363 wait complete. 2024-09-16T14:53:03.868 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:53:03.869 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T14:53:03.869 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:53:03.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.983+0000 7f9513975640 10 monclient: get_monmap_and_config 2024-09-16T14:53:03.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.983+0000 7f9513975640 10 monclient: build_initial_monmap 2024-09-16T14:53:03.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.983+0000 7f9513975640 10 monclient: monmap: 2024-09-16T14:53:03.986 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:03.986 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:03.987 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:53:03.987876+0000 2024-09-16T14:53:03.987 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:53:03.987876+0000 2024-09-16T14:53:03.987 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:03.987 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:03.987 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:03.987 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:53:03.987 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:53:03.987 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:03.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.983+0000 7f9513975640 1 Processor -- start 2024-09-16T14:53:03.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.983+0000 7f9513975640 1 -- start start 2024-09-16T14:53:03.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.983+0000 7f9513975640 10 monclient: init 2024-09-16T14:53:03.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.983+0000 7f9513975640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:03.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.983+0000 7f9513975640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:53:03.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.983+0000 7f9513975640 10 monclient(hunting): picked mon.noname-a con 0x7f950c09cdc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 10 monclient(hunting): picked mon.noname-b con 0x7f950c09e7a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 10 monclient(hunting): picked mon.noname-c con 0x7f950c09d1a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950c09d5b0 con 0x7f950c09cdc0 2024-09-16T14:53:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950c131a90 con 0x7f950c09d1a0 2024-09-16T14:53:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9510f10640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f950c09e7a0 legacy=0x7f950c131380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:53:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950c09aff0 con 0x7f950c09e7a0 2024-09-16T14:53:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 10 monclient(hunting): authenticate will time out at 1608.108126s 2024-09-16T14:53:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9511711640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f950c09cdc0 0x7f950c130c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40920/0 (socket says 172.21.15.26:40920) 2024-09-16T14:53:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9511711640 1 -- 172.21.15.26:0/1223849041 learned_addr learned my addr 172.21.15.26:0/1223849041 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 1 -- 172.21.15.26:0/1223849041 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659004458 0 0) 0x7f950c09d5b0 con 0x7f950c09cdc0 2024-09-16T14:53:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:53:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:03.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:03.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 10 monclient(hunting): my global_id is 194412 2024-09-16T14:53:03.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 1 -- 172.21.15.26:0/1223849041 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94f4004640 con 0x7f950c09cdc0 2024-09-16T14:53:03.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 1 -- 172.21.15.26:0/1223849041 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4155670470 0 0) 0x7f94f4004640 con 0x7f950c09cdc0 2024-09-16T14:53:03.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 1 -- 172.21.15.26:0/1223849041 >> v1:172.21.15.26:6790/0 conn(0x7f950c09e7a0 legacy=0x7f950c131380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:53:03.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 1 -- 172.21.15.26:0/1223849041 >> v1:172.21.15.189:6789/0 conn(0x7f950c09d1a0 legacy=0x7f950c139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:03.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:53:03.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 1 monclient: found mon.noname-a 2024-09-16T14:53:03.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:03.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:53:03.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 1 -- 172.21.15.26:0/1223849041 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c09b2d0 con 0x7f950c09cdc0 2024-09-16T14:53:03.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 10 monclient: _finish_auth 0 2024-09-16T14:53:03.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:53:03.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:03.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 5 monclient: authenticate success, global_id 194412 2024-09-16T14:53:03.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 1 -- 172.21.15.26:0/1223849041 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9504001620 con 0x7f950c09cdc0 2024-09-16T14:53:03.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:03.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:53:03.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 10 monclient: dump: 2024-09-16T14:53:03.993 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:03.993 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:03.993 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:03.993 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:03.993 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:03.993 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:03.994 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:03.994 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:03.994 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:03.994 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:03.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:03.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 1 -- 172.21.15.26:0/1223849041 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95040018d0 con 0x7f950c09cdc0 2024-09-16T14:53:03.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:03.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 1 -- 172.21.15.26:0/1223849041 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9504001c60 con 0x7f950c09cdc0 2024-09-16T14:53:03.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:03.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:03.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f950bfff640 10 monclient: dump: 2024-09-16T14:53:03.994 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:03.994 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:03.995 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:03.995 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:03.995 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:03.995 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:03.995 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:03.995 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:03.995 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:03.995 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:03.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:03.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 4 set_mon_vals no callback set 2024-09-16T14:53:03.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 10 monclient: shutdown 2024-09-16T14:53:03.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:03.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 1 -- 172.21.15.26:0/1223849041 >> v1:172.21.15.26:6789/0 conn(0x7f950c09cdc0 legacy=0x7f950c130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:03.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 1 -- 172.21.15.26:0/1223849041 shutdown_connections 2024-09-16T14:53:03.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 1 -- 172.21.15.26:0/1223849041 >> 172.21.15.26:0/1223849041 conn(0x7f950c09fb50 msgr2=0x7f950c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:03.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 1 -- 172.21.15.26:0/1223849041 shutdown_connections 2024-09-16T14:53:03.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 1 -- 172.21.15.26:0/1223849041 wait complete. 2024-09-16T14:53:03.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 10 monclient: build_initial_monmap 2024-09-16T14:53:03.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 10 monclient: monmap: 2024-09-16T14:53:03.996 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:03.996 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:03.996 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:03.996 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:03.996 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:03.996 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:03.997 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:03.997 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:03.997 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:03.997 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:03.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 1 Processor -- start 2024-09-16T14:53:03.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 1 -- start start 2024-09-16T14:53:03.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 10 monclient: init 2024-09-16T14:53:04.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:04.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:53:04.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 10 monclient(hunting): picked mon.noname-c con 0x7f950c09cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:04.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 10 monclient(hunting): picked mon.noname-a con 0x7f950c09d1a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:04.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 10 monclient(hunting): picked mon.noname-b con 0x7f950c09e7a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:04.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950c11a6b0 con 0x7f950c09d1a0 2024-09-16T14:53:04.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9511711640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f950c09cdc0 legacy=0x7f950c1c96b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:53:04.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950c0a1e90 con 0x7f950c09e7a0 2024-09-16T14:53:04.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950c120b00 con 0x7f950c09cdc0 2024-09-16T14:53:04.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:04.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9513975640 10 monclient(hunting): authenticate will time out at 1608.111329s 2024-09-16T14:53:04.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9510f10640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f950c09d1a0 0x7f950c1c9dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40928/0 (socket says 172.21.15.26:40928) 2024-09-16T14:53:04.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9510f10640 1 -- 172.21.15.26:0/2501519563 learned_addr learned my addr 172.21.15.26:0/2501519563 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:04.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225211836 0 0) 0x7f950c11a6b0 con 0x7f950c09d1a0 2024-09-16T14:53:04.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9509ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:04.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9509ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:04.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9509ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:04.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9509ffb640 10 monclient(hunting): my global_id is 194415 2024-09-16T14:53:04.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94f8003530 con 0x7f950c09d1a0 2024-09-16T14:53:04.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2872559278 0 0) 0x7f94f8003530 con 0x7f950c09d1a0 2024-09-16T14:53:04.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f950c11a6b0 con 0x7f950c09d1a0 2024-09-16T14:53:04.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f95000044e0 con 0x7f950c09d1a0 2024-09-16T14:53:04.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9509ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:04.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9509ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:53:04.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9509ffb640 10 monclient(hunting): dump: 2024-09-16T14:53:04.003 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:04.004 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:04.004 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.004 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.004 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:04.004 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:04.004 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:04.004 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:04.004 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:04.004 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:04.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.987+0000 7f9509ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:04.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1302546601 0 0) 0x7f950c11a6b0 con 0x7f950c09d1a0 2024-09-16T14:53:04.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 >> v1:172.21.15.26:6790/0 conn(0x7f950c09cdc0 legacy=0x7f950c1c96b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:53:04.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 >> v1:172.21.15.189:6789/0 conn(0x7f950c09e7a0 legacy=0x7f950c1d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:04.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:53:04.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 monclient: found mon.a 2024-09-16T14:53:04.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:04.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:04.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c1ca4d0 con 0x7f950c09d1a0 2024-09-16T14:53:04.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient: _finish_auth 0 2024-09-16T14:53:04.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:53:04.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:04.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9513975640 5 monclient: authenticate success, global_id 194415 2024-09-16T14:53:04.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9513975640 10 monclient: _renew_subs 2024-09-16T14:53:04.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9513975640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:04.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9513975640 1 -- 172.21.15.26:0/2501519563 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950c09a240 con 0x7f950c09d1a0 2024-09-16T14:53:04.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9513975640 10 monclient: _renew_subs 2024-09-16T14:53:04.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9513975640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:04.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9513975640 1 -- 172.21.15.26:0/2501519563 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c12e620 con 0x7f950c09d1a0 2024-09-16T14:53:04.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9500004700 con 0x7f950c09d1a0 2024-09-16T14:53:04.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:04.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9500004a90 con 0x7f950c09d1a0 2024-09-16T14:53:04.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:04.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:04.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f950b7fe640 4 set_mon_vals no callback set 2024-09-16T14:53:04.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient: dump: 2024-09-16T14:53:04.006 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:04.006 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:04.006 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.006 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.006 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:04.007 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:04.007 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:04.007 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:04.007 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:04.007 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:04.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9513975640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:04.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9513975640 1 -- 172.21.15.26:0/2501519563 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f94ec006590 con 0x7f94ec002830 2024-09-16T14:53:04.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9500004bf0 con 0x7f950c09d1a0 2024-09-16T14:53:04.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f95000440f0 con 0x7f950c09d1a0 2024-09-16T14:53:04.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112101718 0 0) 0x7f94ec006590 con 0x7f94ec002830 2024-09-16T14:53:04.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:04.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:04.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:04.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient(hunting): my global_id is 194418 2024-09-16T14:53:04.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f950c11a6b0 con 0x7f94ec002830 2024-09-16T14:53:04.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4125695104 0 0) 0x7f950c11a6b0 con 0x7f94ec002830 2024-09-16T14:53:04.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94ec006590 con 0x7f94ec002830 2024-09-16T14:53:04.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9504001c30 con 0x7f94ec002830 2024-09-16T14:53:04.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:04.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:04.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient: dump: 2024-09-16T14:53:04.008 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:04.008 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:04.008 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.008 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.008 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:04.008 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:04.009 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:04.009 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:04.009 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:04.009 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:04.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9511f12640 10 monclient: get_auth_request con 0x7f94f80360a0 auth_method 0 2024-09-16T14:53:04.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3464341235 0 0) 0x7f94ec006590 con 0x7f94ec002830 2024-09-16T14:53:04.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f94ec00a0e0 con 0x7f94ec002830 2024-09-16T14:53:04.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f9504001640 con 0x7f94ec002830 2024-09-16T14:53:04.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:04.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:04.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 >> v1:172.21.15.26:6789/0 conn(0x7f94ec002830 legacy=0x7f94ec004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:04.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.991+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 >> v1:172.21.15.26:6789/0 conn(0x7f94ec002830 legacy=0x7f94ec004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:04.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9513975640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:53:04.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9513975640 1 -- 172.21.15.26:0/2501519563 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f94ec009d80 con 0x7f94ec0090a0 2024-09-16T14:53:04.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555638832 0 0) 0x7f94ec009d80 con 0x7f94ec0090a0 2024-09-16T14:53:04.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:04.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:04.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:04.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 10 monclient(hunting): my global_id is 194421 2024-09-16T14:53:04.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ec006590 con 0x7f94ec0090a0 2024-09-16T14:53:04.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1880803508 0 0) 0x7f94ec006590 con 0x7f94ec0090a0 2024-09-16T14:53:04.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94ec009d80 con 0x7f94ec0090a0 2024-09-16T14:53:04.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9504004ce0 con 0x7f94ec0090a0 2024-09-16T14:53:04.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:04.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:04.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 10 monclient: dump: 2024-09-16T14:53:04.011 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:04.011 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:04.011 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.011 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.011 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:04.011 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:04.011 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:04.011 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:04.011 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:04.011 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:04.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2142819952 0 0) 0x7f94ec009d80 con 0x7f94ec0090a0 2024-09-16T14:53:04.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f94ec00c290 con 0x7f94ec0090a0 2024-09-16T14:53:04.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4200 (unknown 0 0 2846521466) 0x7f9504007080 con 0x7f94ec0090a0 2024-09-16T14:53:04.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:53:04.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:04.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 >> v1:172.21.15.26:6789/0 conn(0x7f94ec0090a0 legacy=0x7f94ec009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:04.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9509ffb640 1 -- 172.21.15.26:0/2501519563 >> v1:172.21.15.26:6789/0 conn(0x7f94ec0090a0 legacy=0x7f94ec009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:04.012 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:53:04.012 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:53:04.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:53:04.012 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:53:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 98, 2024-09-16T14:53:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:53:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:53:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: 1 2024-09-16T14:53:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:53:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:53:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:53:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:53:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:53:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:53:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:53:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:53:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:53:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:53:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:53:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:04.016 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:53:04.016 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:53:04.016 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:53:04.016 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:53:04.016 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:53:04.016 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:53:04.016 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:53:04.016 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:04.016 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:53:04.016 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:04.016 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:04.016 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:04.016 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:04.016 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:04.017 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:04.017 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:04.017 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:04.017 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:04.017 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:04.017 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:04.017 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:53:04.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.017 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:53:04.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:53:04.017 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:53:04.017 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:04.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:04.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.018 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:04.018 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:53:04.018 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:04.018 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:04.018 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:04.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.018 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:53:04.018 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:53:04.018 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:04.018 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:04.018 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:04.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:53:04.019 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:53:04.019 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:04.019 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:04.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.019 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:04.019 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:53:04.019 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:04.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:04.019 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:04.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.019 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:53:04.019 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:53:04.019 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:04.019 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:04.019 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:04.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:53:04.020 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:53:04.020 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:04.020 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:04.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.020 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:04.020 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:53:04.020 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:04.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:04.020 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:04.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.020 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:53:04.021 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:53:04.021 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:04.021 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:04.021 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:04.021 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:04.021 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:04.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.021 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:53:04.021 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:53:04.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.021 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:04.021 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:04.021 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:04.021 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:04.022 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:04.022 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:53:04.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.022 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:04.022 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:04.022 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:53:04.022 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:04.022 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:04.022 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:53:04.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.022 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:04.022 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:04.022 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 7 2024-09-16T14:53:04.022 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:04.022 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:04.023 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:53:04.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.023 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:04.023 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:04.023 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:04.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:04.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:04.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.023 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:53:04.023 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:53:04.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9513975640 1 -- 172.21.15.26:0/2501519563 >> v1:172.21.15.189:6812/3196161582 conn(0x7f94f80360a0 legacy=0x7f94f8038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:04.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9513975640 10 monclient: shutdown 2024-09-16T14:53:04.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9513975640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:04.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9513975640 1 -- 172.21.15.26:0/2501519563 >> v1:172.21.15.26:6789/0 conn(0x7f950c09d1a0 legacy=0x7f950c1c9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:04.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9511f12640 1 -- 172.21.15.26:0/2501519563 reap_dead start 2024-09-16T14:53:04.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9511f12640 1 -- 172.21.15.26:0/2501519563 reap_dead start 2024-09-16T14:53:04.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9513975640 1 -- 172.21.15.26:0/2501519563 shutdown_connections 2024-09-16T14:53:04.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9513975640 1 -- 172.21.15.26:0/2501519563 >> 172.21.15.26:0/2501519563 conn(0x7f950c09fb50 msgr2=0x7f950c1b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:04.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9513975640 1 -- 172.21.15.26:0/2501519563 shutdown_connections 2024-09-16T14:53:04.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:03.999+0000 7f9513975640 1 -- 172.21.15.26:0/2501519563 wait complete. 2024-09-16T14:53:04.026 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:53:04.026 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T14:53:04.026 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:53:04.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 10 monclient: get_monmap_and_config 2024-09-16T14:53:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 10 monclient: build_initial_monmap 2024-09-16T14:53:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 10 monclient: monmap: 2024-09-16T14:53:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:53:04.146302+0000 2024-09-16T14:53:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:53:04.146302+0000 2024-09-16T14:53:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:53:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:53:04.146 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 1 Processor -- start 2024-09-16T14:53:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 1 -- start start 2024-09-16T14:53:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 10 monclient: init 2024-09-16T14:53:04.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:04.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:53:04.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 10 monclient(hunting): picked mon.noname-b con 0x7f5a9409a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 10 monclient(hunting): picked mon.noname-c con 0x7f5a9409d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 10 monclient(hunting): picked mon.noname-a con 0x7f5a9409b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9409dc60 con 0x7f5a9409b040 2024-09-16T14:53:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a94099ae0 con 0x7f5a9409d1b0 2024-09-16T14:53:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9409b8f0 con 0x7f5a9409a550 2024-09-16T14:53:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a937fe640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f5a9409a550 legacy=0x7f5a9409a930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:53:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 10 monclient(hunting): authenticate will time out at 1608.266517s 2024-09-16T14:53:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a93fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5a9409b040 0x7f5a9413a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40952/0 (socket says 172.21.15.26:40952) 2024-09-16T14:53:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a93fff640 1 -- 172.21.15.26:0/296786582 learned_addr learned my addr 172.21.15.26:0/296786582 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 1 -- 172.21.15.26:0/296786582 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996940951 0 0) 0x7f5a9409dc60 con 0x7f5a9409b040 2024-09-16T14:53:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 10 monclient(hunting): _negotiate 2024-09-16T14:53:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 10 monclient(hunting): my global_id is 194424 2024-09-16T14:53:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 1 -- 172.21.15.26:0/296786582 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a80003530 con 0x7f5a9409b040 2024-09-16T14:53:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 1 -- 172.21.15.26:0/296786582 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2572499836 0 0) 0x7f5a80003530 con 0x7f5a9409b040 2024-09-16T14:53:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 1 -- 172.21.15.26:0/296786582 >> v1:172.21.15.26:6790/0 conn(0x7f5a9409a550 legacy=0x7f5a9409a930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:53:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 1 -- 172.21.15.26:0/296786582 >> v1:172.21.15.189:6789/0 conn(0x7f5a9409d1b0 legacy=0x7f5a94136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 10 monclient: _finish_hunting 0 2024-09-16T14:53:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 1 monclient: found mon.noname-a 2024-09-16T14:53:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:53:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 1 -- 172.21.15.26:0/296786582 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9409bbd0 con 0x7f5a9409b040 2024-09-16T14:53:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 10 monclient: _finish_auth 0 2024-09-16T14:53:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 10 monclient: _check_auth_tickets 2024-09-16T14:53:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 1 -- 172.21.15.26:0/296786582 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5a8c001620 con 0x7f5a9409b040 2024-09-16T14:53:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:53:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 10 monclient: dump: 2024-09-16T14:53:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:04.158 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 5 monclient: authenticate success, global_id 194424 2024-09-16T14:53:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 1 -- 172.21.15.26:0/296786582 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a8c001940 con 0x7f5a9409b040 2024-09-16T14:53:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 1 -- 172.21.15.26:0/296786582 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5a8c001cd0 con 0x7f5a9409b040 2024-09-16T14:53:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 4 set_mon_vals no callback set 2024-09-16T14:53:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a927fc640 10 monclient: dump: 2024-09-16T14:53:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:04.159 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 10 monclient: shutdown 2024-09-16T14:53:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.143+0000 7f5a99f2f640 1 -- 172.21.15.26:0/296786582 >> v1:172.21.15.26:6789/0 conn(0x7f5a9409b040 legacy=0x7f5a9413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 1 -- 172.21.15.26:0/296786582 shutdown_connections 2024-09-16T14:53:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 1 -- 172.21.15.26:0/296786582 >> 172.21.15.26:0/296786582 conn(0x7f5a9409fb50 msgr2=0x7f5a9409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 1 -- 172.21.15.26:0/296786582 shutdown_connections 2024-09-16T14:53:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 1 -- 172.21.15.26:0/296786582 wait complete. 2024-09-16T14:53:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 10 monclient: build_initial_monmap 2024-09-16T14:53:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 10 monclient: monmap: 2024-09-16T14:53:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:04.160 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 1 Processor -- start 2024-09-16T14:53:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 1 -- start start 2024-09-16T14:53:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 10 monclient: init 2024-09-16T14:53:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:53:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 10 monclient(hunting): picked mon.noname-a con 0x7f5a9409a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 10 monclient(hunting): picked mon.noname-c con 0x7f5a9409b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 10 monclient(hunting): picked mon.noname-b con 0x7f5a9409d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a940a25d0 con 0x7f5a9409a550 2024-09-16T14:53:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a941329c0 con 0x7f5a9409d1b0 2024-09-16T14:53:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a94119840 con 0x7f5a9409b040 2024-09-16T14:53:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 10 monclient(hunting): authenticate will time out at 1608.269991s 2024-09-16T14:53:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a92ffd640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f5a9409b040 legacy=0x7f5a941cc080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:53:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a937fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5a9409a550 0x7f5a941c0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40960/0 (socket says 172.21.15.26:40960) 2024-09-16T14:53:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a937fe640 1 -- 172.21.15.26:0/294733623 learned_addr learned my addr 172.21.15.26:0/294733623 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968286154 0 0) 0x7f5a940a25d0 con 0x7f5a9409a550 2024-09-16T14:53:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:53:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient(hunting): my global_id is 194427 2024-09-16T14:53:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a74003530 con 0x7f5a9409a550 2024-09-16T14:53:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659408356 0 0) 0x7f5a941329c0 con 0x7f5a9409d1b0 2024-09-16T14:53:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:53:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient(hunting): my global_id is 224173 2024-09-16T14:53:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a940a25d0 con 0x7f5a9409d1b0 2024-09-16T14:53:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1785333983 0 0) 0x7f5a74003530 con 0x7f5a9409a550 2024-09-16T14:53:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a941329c0 con 0x7f5a9409a550 2024-09-16T14:53:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5a84004160 con 0x7f5a9409a550 2024-09-16T14:53:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:53:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient(hunting): dump: 2024-09-16T14:53:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:04.164 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1676132753 0 0) 0x7f5a941329c0 con 0x7f5a9409a550 2024-09-16T14:53:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 >> v1:172.21.15.26:6790/0 conn(0x7f5a9409b040 legacy=0x7f5a941cc080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:53:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 >> v1:172.21.15.189:6789/0 conn(0x7f5a9409d1b0 legacy=0x7f5a941cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:53:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 1 monclient: found mon.a 2024-09-16T14:53:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a94139260 con 0x7f5a9409a550 2024-09-16T14:53:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient: _finish_auth 0 2024-09-16T14:53:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:53:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 5 monclient: authenticate success, global_id 194427 2024-09-16T14:53:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 10 monclient: _renew_subs 2024-09-16T14:53:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 1 -- 172.21.15.26:0/294733623 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a94139710 con 0x7f5a9409a550 2024-09-16T14:53:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 10 monclient: _renew_subs 2024-09-16T14:53:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 1 -- 172.21.15.26:0/294733623 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a941317d0 con 0x7f5a9409a550 2024-09-16T14:53:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a840043d0 con 0x7f5a9409a550 2024-09-16T14:53:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5a84004760 con 0x7f5a9409a550 2024-09-16T14:53:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a917fa640 4 set_mon_vals no callback set 2024-09-16T14:53:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a8bfff640 10 monclient: dump: 2024-09-16T14:53:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:04.166 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.147+0000 7f5a99f2f640 1 -- 172.21.15.26:0/294733623 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5a78006590 con 0x7f5a78002830 2024-09-16T14:53:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5a84004940 con 0x7f5a9409a550 2024-09-16T14:53:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f5a84043d60 con 0x7f5a9409a550 2024-09-16T14:53:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561648949 0 0) 0x7f5a78006590 con 0x7f5a78002830 2024-09-16T14:53:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:53:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 10 monclient(hunting): my global_id is 224176 2024-09-16T14:53:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a941329c0 con 0x7f5a78002830 2024-09-16T14:53:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a93fff640 10 monclient: get_auth_request con 0x7f5a740360e0 auth_method 0 2024-09-16T14:53:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4142240045 0 0) 0x7f5a941329c0 con 0x7f5a78002830 2024-09-16T14:53:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a78006590 con 0x7f5a78002830 2024-09-16T14:53:04.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5a7c001a80 con 0x7f5a78002830 2024-09-16T14:53:04.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:04.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:53:04.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 10 monclient: dump: 2024-09-16T14:53:04.168 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:04.168 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:04.168 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.168 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.168 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:04.168 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:04.168 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:04.168 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:04.168 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:04.169 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:04.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3270142059 0 0) 0x7f5a78006590 con 0x7f5a78002830 2024-09-16T14:53:04.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a7800a0e0 con 0x7f5a78002830 2024-09-16T14:53:04.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5a7c001fc0 con 0x7f5a78002830 2024-09-16T14:53:04.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:04.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:04.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 >> v1:172.21.15.189:6789/0 conn(0x7f5a78002830 legacy=0x7f5a78004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:04.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.151+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 >> v1:172.21.15.189:6789/0 conn(0x7f5a78002830 legacy=0x7f5a78004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:04.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a99f2f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:53:04.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a99f2f640 1 -- 172.21.15.26:0/294733623 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5a78009da0 con 0x7f5a780090c0 2024-09-16T14:53:04.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078438495 0 0) 0x7f5a78009da0 con 0x7f5a780090c0 2024-09-16T14:53:04.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:53:04.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:04.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:04.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 10 monclient(hunting): my global_id is 224179 2024-09-16T14:53:04.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a78006590 con 0x7f5a780090c0 2024-09-16T14:53:04.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2279499819 0 0) 0x7f5a78006590 con 0x7f5a780090c0 2024-09-16T14:53:04.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a78009da0 con 0x7f5a780090c0 2024-09-16T14:53:04.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5a7c003dd0 con 0x7f5a780090c0 2024-09-16T14:53:04.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:04.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:53:04.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 10 monclient: dump: 2024-09-16T14:53:04.171 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:04.171 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:04.171 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.172 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:04.172 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:04.172 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:04.172 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:04.172 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:04.172 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:04.172 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:04.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4272460944 0 0) 0x7f5a78009da0 con 0x7f5a780090c0 2024-09-16T14:53:04.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5a7800c290 con 0x7f5a780090c0 2024-09-16T14:53:04.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3870 (unknown 0 0 4204422173) 0x7f5a7c003bf0 con 0x7f5a780090c0 2024-09-16T14:53:04.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:53:04.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:04.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 >> v1:172.21.15.189:6789/0 conn(0x7f5a780090c0 legacy=0x7f5a78009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:04.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a8bfff640 1 -- 172.21.15.26:0/294733623 >> v1:172.21.15.189:6789/0 conn(0x7f5a780090c0 legacy=0x7f5a78009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:04.173 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:53:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:53:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:53:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:53:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 98, 2024-09-16T14:53:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:53:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:53:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: 1 2024-09-16T14:53:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:53:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:53:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:53:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:53:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:53:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:53:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:53:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:53:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:53:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:53:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:53:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:53:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:53:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:53:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:53:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:53:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:53:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:53:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:53:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:53:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:53:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:53:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:53:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:53:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:53:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:53:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:53:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:53:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:53:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:53:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:53:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:53:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:53:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:53:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:53:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:53:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:53:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:53:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:53:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:53:04.184 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:53:04.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a99f2f640 1 -- 172.21.15.26:0/294733623 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5a740360e0 legacy=0x7f5a740385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:04.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a99f2f640 10 monclient: shutdown 2024-09-16T14:53:04.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a99f2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:04.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a99f2f640 1 -- 172.21.15.26:0/294733623 >> v1:172.21.15.26:6789/0 conn(0x7f5a9409a550 legacy=0x7f5a941c0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:04.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a93fff640 1 -- 172.21.15.26:0/294733623 reap_dead start 2024-09-16T14:53:04.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a93fff640 1 -- 172.21.15.26:0/294733623 reap_dead start 2024-09-16T14:53:04.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a99f2f640 1 -- 172.21.15.26:0/294733623 shutdown_connections 2024-09-16T14:53:04.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a99f2f640 1 -- 172.21.15.26:0/294733623 >> 172.21.15.26:0/294733623 conn(0x7f5a9409fb50 msgr2=0x7f5a94134c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:04.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a99f2f640 1 -- 172.21.15.26:0/294733623 shutdown_connections 2024-09-16T14:53:04.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:04.159+0000 7f5a99f2f640 1 -- 172.21.15.26:0/294733623 wait complete. 2024-09-16T14:53:04.186 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:53:04.186 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:53:24.188 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T14:53:24.188 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T14:53:24.188 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T14:53:24.188 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T14:53:24.191 INFO:tasks.ceph.mon.c:Started 2024-09-16T14:53:24.191 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:53:24.191 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:53:24.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6348fce640 10 monclient: get_monmap_and_config 2024-09-16T14:53:24.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6348fce640 10 monclient: build_initial_monmap 2024-09-16T14:53:24.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6348fce640 10 monclient: monmap: 2024-09-16T14:53:24.323 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:24.323 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.323 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:53:24.324127+0000 2024-09-16T14:53:24.324 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:53:24.324127+0000 2024-09-16T14:53:24.324 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:24.324 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.324 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:24.324 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:53:24.324 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:53:24.325 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6348fce640 1 Processor -- start 2024-09-16T14:53:24.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6348fce640 1 -- start start 2024-09-16T14:53:24.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6348fce640 10 monclient: init 2024-09-16T14:53:24.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6348fce640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:24.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6348fce640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:53:24.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6348fce640 10 monclient(hunting): picked mon.noname-b con 0x7f6344134f80 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:24.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6348fce640 10 monclient(hunting): picked mon.noname-c con 0x7f63440a0db0 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:24.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6348fce640 10 monclient(hunting): picked mon.noname-a con 0x7f63440a1200 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:24.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6348fce640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6344099630 con 0x7f63440a1200 2024-09-16T14:53:24.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6348fce640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63440a1930 con 0x7f63440a0db0 2024-09-16T14:53:24.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6348fce640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634406c2c0 con 0x7f6344134f80 2024-09-16T14:53:24.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6348fce640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:24.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6348fce640 10 monclient(hunting): authenticate will time out at 1628.443792s 2024-09-16T14:53:24.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f634259c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6344134f80 0x7f6344135360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40930/0 (socket says 172.21.15.26:40930) 2024-09-16T14:53:24.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f634259c640 1 -- 172.21.15.26:0/3430236343 learned_addr learned my addr 172.21.15.26:0/3430236343 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:24.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.319+0000 7f6342d9d640 1 --1- 172.21.15.26:0/3430236343 >> v1:172.21.15.26:6789/0 conn(0x7f63440a1200 0x7f634413d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51178/0 (socket says 172.21.15.26:51178) 2024-09-16T14:53:24.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 1 -- 172.21.15.26:0/3430236343 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653369299 0 0) 0x7f63440a1930 con 0x7f63440a0db0 2024-09-16T14:53:24.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 10 monclient(hunting): my global_id is 224182 2024-09-16T14:53:24.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 1 -- 172.21.15.26:0/3430236343 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f632c003530 con 0x7f63440a0db0 2024-09-16T14:53:24.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 1 -- 172.21.15.26:0/3430236343 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3842933116 0 0) 0x7f632c003530 con 0x7f63440a0db0 2024-09-16T14:53:24.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 1 -- 172.21.15.26:0/3430236343 >> v1:172.21.15.26:6790/0 conn(0x7f6344134f80 legacy=0x7f6344135360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 1 -- 172.21.15.26:0/3430236343 >> v1:172.21.15.26:6789/0 conn(0x7f63440a1200 legacy=0x7f634413d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 10 monclient: _finish_hunting 0 2024-09-16T14:53:24.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 1 monclient: found mon.noname-c 2024-09-16T14:53:24.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:24.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:53:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 1 -- 172.21.15.26:0/3430236343 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634406c5a0 con 0x7f63440a0db0 2024-09-16T14:53:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 10 monclient: _finish_auth 0 2024-09-16T14:53:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 10 monclient: _check_auth_tickets 2024-09-16T14:53:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:24.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 5 monclient: authenticate success, global_id 224182 2024-09-16T14:53:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 1 -- 172.21.15.26:0/3430236343 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6334001620 con 0x7f63440a0db0 2024-09-16T14:53:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:53:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 10 monclient: dump: 2024-09-16T14:53:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.341 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.342 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.343 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 1 -- 172.21.15.26:0/3430236343 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63340018d0 con 0x7f63440a0db0 2024-09-16T14:53:24.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 1 -- 172.21.15.26:0/3430236343 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6334001ca0 con 0x7f63440a0db0 2024-09-16T14:53:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 4 set_mon_vals no callback set 2024-09-16T14:53:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:53:24.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634159a640 10 monclient: dump: 2024-09-16T14:53:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.345 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.346 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.346 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.346 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.346 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.346 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 10 monclient: shutdown 2024-09-16T14:53:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 1 -- 172.21.15.26:0/3430236343 >> v1:172.21.15.189:6789/0 conn(0x7f63440a0db0 legacy=0x7f6344139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 1 -- 172.21.15.26:0/3430236343 shutdown_connections 2024-09-16T14:53:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 1 -- 172.21.15.26:0/3430236343 >> 172.21.15.26:0/3430236343 conn(0x7f63440a5570 msgr2=0x7f63440a5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 1 -- 172.21.15.26:0/3430236343 shutdown_connections 2024-09-16T14:53:24.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 1 -- 172.21.15.26:0/3430236343 wait complete. 2024-09-16T14:53:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 10 monclient: build_initial_monmap 2024-09-16T14:53:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 10 monclient: monmap: 2024-09-16T14:53:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:24.348 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:24.349 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:24.350 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:24.350 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 1 Processor -- start 2024-09-16T14:53:24.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 1 -- start start 2024-09-16T14:53:24.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 10 monclient: init 2024-09-16T14:53:24.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:24.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:53:24.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 10 monclient(hunting): picked mon.noname-c con 0x7f63440a0db0 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:24.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 10 monclient(hunting): picked mon.noname-a con 0x7f63440a1200 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:24.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 10 monclient(hunting): picked mon.noname-b con 0x7f6344134f80 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:24.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634409adc0 con 0x7f63440a1200 2024-09-16T14:53:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634409c360 con 0x7f6344134f80 2024-09-16T14:53:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6344138150 con 0x7f63440a0db0 2024-09-16T14:53:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6348fce640 10 monclient(hunting): authenticate will time out at 1628.455628s 2024-09-16T14:53:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634259c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f63440a0db0 0x7f63441367b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40946/0 (socket says 172.21.15.26:40946) 2024-09-16T14:53:24.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f634259c640 1 -- 172.21.15.26:0/1816704522 learned_addr learned my addr 172.21.15.26:0/1816704522 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:24.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.331+0000 7f6341d9b640 1 --1- 172.21.15.26:0/1816704522 >> v1:172.21.15.26:6789/0 conn(0x7f63440a1200 0x7f6344136ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51192/0 (socket says 172.21.15.26:51192) 2024-09-16T14:53:24.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551855074 0 0) 0x7f6344138150 con 0x7f63440a0db0 2024-09-16T14:53:24.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient(hunting): my global_id is 234101 2024-09-16T14:53:24.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6324003560 con 0x7f63440a0db0 2024-09-16T14:53:24.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2299557026 0 0) 0x7f6324003560 con 0x7f63440a0db0 2024-09-16T14:53:24.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6344138150 con 0x7f63440a0db0 2024-09-16T14:53:24.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f63380017b0 con 0x7f63440a0db0 2024-09-16T14:53:24.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:53:24.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient(hunting): dump: 2024-09-16T14:53:24.360 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.361 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.362 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:24.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334905065 0 0) 0x7f634409adc0 con 0x7f63440a1200 2024-09-16T14:53:24.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient(hunting): my global_id is 194430 2024-09-16T14:53:24.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6324003560 con 0x7f63440a1200 2024-09-16T14:53:24.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4220147225 0 0) 0x7f6344138150 con 0x7f63440a0db0 2024-09-16T14:53:24.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 >> v1:172.21.15.189:6789/0 conn(0x7f6344134f80 legacy=0x7f63441d1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 >> v1:172.21.15.26:6789/0 conn(0x7f63440a1200 legacy=0x7f6344136ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:53:24.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 1 monclient: found mon.c 2024-09-16T14:53:24.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:24.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:24.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6344138aa0 con 0x7f63440a0db0 2024-09-16T14:53:24.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient: _finish_auth 0 2024-09-16T14:53:24.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:53:24.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:24.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6348fce640 5 monclient: authenticate success, global_id 234101 2024-09-16T14:53:24.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6348fce640 10 monclient: _renew_subs 2024-09-16T14:53:24.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6348fce640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:24.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6348fce640 1 -- 172.21.15.26:0/1816704522 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6344139c10 con 0x7f63440a0db0 2024-09-16T14:53:24.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6348fce640 10 monclient: _renew_subs 2024-09-16T14:53:24.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6348fce640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:24.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6348fce640 1 -- 172.21.15.26:0/1816704522 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6344138f40 con 0x7f63440a0db0 2024-09-16T14:53:24.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63380047f0 con 0x7f63440a0db0 2024-09-16T14:53:24.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:24.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6338004b80 con 0x7f63440a0db0 2024-09-16T14:53:24.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:24.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 10 monclient: dump: 2024-09-16T14:53:24.368 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.368 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.368 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.369 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.369 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.369 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.369 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.369 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.369 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.370 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6333fff640 4 set_mon_vals no callback set 2024-09-16T14:53:24.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.351+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6338004ce0 con 0x7f63440a0db0 2024-09-16T14:53:24.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.355+0000 7f6348fce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:24.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.355+0000 7f6348fce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:24.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.355+0000 7f6348fce640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:24.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.355+0000 7f6348fce640 1 -- 172.21.15.26:0/1816704522 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6328005050 con 0x7f63440a0db0 2024-09-16T14:53:24.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.355+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f6338043da0 con 0x7f63440a0db0 2024-09-16T14:53:24.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.355+0000 7f6341d9b640 10 monclient: get_auth_request con 0x7f63240361a0 auth_method 0 2024-09-16T14:53:24.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.359+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f63380146e0 con 0x7f63440a0db0 2024-09-16T14:53:24.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.359+0000 7f6332ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:24.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.359+0000 7f6332ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:24.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.579+0000 7f6348fce640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:53:24.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.579+0000 7f6348fce640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:53:24.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.579+0000 7f6348fce640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:24.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.579+0000 7f6348fce640 1 -- 172.21.15.26:0/1816704522 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6328005880 con 0x7f63440a0db0 2024-09-16T14:53:24.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.579+0000 7f6332ffd640 1 -- 172.21.15.26:0/1816704522 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1439015264) 0x7f6338012b50 con 0x7f63440a0db0 2024-09-16T14:53:24.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.579+0000 7f6332ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:53:24.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.579+0000 7f6332ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:24.583 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:53:24.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.583+0000 7f6348fce640 1 -- 172.21.15.26:0/1816704522 >> v1:172.21.15.189:6812/3196161582 conn(0x7f63240361a0 legacy=0x7f6324038660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.583+0000 7f6348fce640 10 monclient: shutdown 2024-09-16T14:53:24.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.583+0000 7f6348fce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:24.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.583+0000 7f6348fce640 1 -- 172.21.15.26:0/1816704522 >> v1:172.21.15.26:6790/0 conn(0x7f63440a0db0 legacy=0x7f63441367b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.583+0000 7f6348fce640 1 -- 172.21.15.26:0/1816704522 shutdown_connections 2024-09-16T14:53:24.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.583+0000 7f6348fce640 1 -- 172.21.15.26:0/1816704522 >> 172.21.15.26:0/1816704522 conn(0x7f63440a5570 msgr2=0x7f6344099ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:24.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.583+0000 7f6348fce640 1 -- 172.21.15.26:0/1816704522 shutdown_connections 2024-09-16T14:53:24.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.583+0000 7f6348fce640 1 -- 172.21.15.26:0/1816704522 wait complete. 2024-09-16T14:53:24.602 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:53:24.602 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:53:24.602 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:53:24.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.711+0000 7f70097c8640 10 monclient: get_monmap_and_config 2024-09-16T14:53:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.711+0000 7f70097c8640 10 monclient: build_initial_monmap 2024-09-16T14:53:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.711+0000 7f70097c8640 10 monclient: monmap: 2024-09-16T14:53:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:53:24.717300+0000 2024-09-16T14:53:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:53:24.717300+0000 2024-09-16T14:53:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:53:24.716 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:53:24.717 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 1 Processor -- start 2024-09-16T14:53:24.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 1 -- start start 2024-09-16T14:53:24.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 10 monclient: init 2024-09-16T14:53:24.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:24.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:53:24.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 10 monclient(hunting): picked mon.noname-b con 0x7f70040a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:24.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 10 monclient(hunting): picked mon.noname-c con 0x7f70040a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:24.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 10 monclient(hunting): picked mon.noname-a con 0x7f70040a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:24.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70040a3250 con 0x7f70040a2800 2024-09-16T14:53:24.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f700409f600 con 0x7f70040a0000 2024-09-16T14:53:24.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70040a3830 con 0x7f70040a1d10 2024-09-16T14:53:24.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:24.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 10 monclient(hunting): authenticate will time out at 1628.837407s 2024-09-16T14:53:24.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70037fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f70040a2800 0x7f700413d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51200/0 (socket says 172.21.15.26:51200) 2024-09-16T14:53:24.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7002ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f70040a1d10 0x7f70040a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40962/0 (socket says 172.21.15.26:40962) 2024-09-16T14:53:24.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70037fe640 1 -- 172.21.15.26:0/778532182 learned_addr learned my addr 172.21.15.26:0/778532182 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:24.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 1 -- 172.21.15.26:0/778532182 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893206613 0 0) 0x7f70040a3830 con 0x7f70040a1d10 2024-09-16T14:53:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient(hunting): my global_id is 234104 2024-09-16T14:53:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 1 -- 172.21.15.26:0/778532182 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fec003530 con 0x7f70040a1d10 2024-09-16T14:53:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 1 -- 172.21.15.26:0/778532182 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221216963 0 0) 0x7f70040a3250 con 0x7f70040a2800 2024-09-16T14:53:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient(hunting): my global_id is 194433 2024-09-16T14:53:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 1 -- 172.21.15.26:0/778532182 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70040a3830 con 0x7f70040a2800 2024-09-16T14:53:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 1 -- 172.21.15.26:0/778532182 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3100253207 0 0) 0x7f70040a3830 con 0x7f70040a2800 2024-09-16T14:53:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 1 -- 172.21.15.26:0/778532182 >> v1:172.21.15.26:6790/0 conn(0x7f70040a1d10 legacy=0x7f70040a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 1 -- 172.21.15.26:0/778532182 >> v1:172.21.15.189:6789/0 conn(0x7f70040a0000 legacy=0x7f7004139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:53:24.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 1 monclient: found mon.noname-a 2024-09-16T14:53:24.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:24.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:53:24.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 1 -- 172.21.15.26:0/778532182 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700409ed60 con 0x7f70040a2800 2024-09-16T14:53:24.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient: _finish_auth 0 2024-09-16T14:53:24.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:53:24.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:24.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 1 -- 172.21.15.26:0/778532182 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6ffc0018b0 con 0x7f70040a2800 2024-09-16T14:53:24.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 5 monclient: authenticate success, global_id 194433 2024-09-16T14:53:24.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:24.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:53:24.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient: dump: 2024-09-16T14:53:24.724 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.724 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.724 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.724 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.724 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.724 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.724 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.724 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.724 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.724 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:24.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 1 -- 172.21.15.26:0/778532182 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ffc001b50 con 0x7f70040a2800 2024-09-16T14:53:24.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:24.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:24.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 4 set_mon_vals no callback set 2024-09-16T14:53:24.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 1 -- 172.21.15.26:0/778532182 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6ffc001f20 con 0x7f70040a2800 2024-09-16T14:53:24.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:24.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f7001ffb640 10 monclient: dump: 2024-09-16T14:53:24.725 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.725 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.725 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.725 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.725 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.725 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.725 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.726 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.726 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.726 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 10 monclient: shutdown 2024-09-16T14:53:24.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:24.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 1 -- 172.21.15.26:0/778532182 >> v1:172.21.15.26:6789/0 conn(0x7f70040a2800 legacy=0x7f700413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 1 -- 172.21.15.26:0/778532182 shutdown_connections 2024-09-16T14:53:24.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 1 -- 172.21.15.26:0/778532182 >> 172.21.15.26:0/778532182 conn(0x7f70040a7f30 msgr2=0x7f70040a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:24.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 1 -- 172.21.15.26:0/778532182 shutdown_connections 2024-09-16T14:53:24.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 1 -- 172.21.15.26:0/778532182 wait complete. 2024-09-16T14:53:24.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 10 monclient: build_initial_monmap 2024-09-16T14:53:24.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.715+0000 7f70097c8640 10 monclient: monmap: 2024-09-16T14:53:24.726 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:24.726 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:24.727 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:24.727 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:24.727 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:24.727 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:24.727 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:24.727 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:24.727 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:24.727 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 1 Processor -- start 2024-09-16T14:53:24.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 1 -- start start 2024-09-16T14:53:24.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 10 monclient: init 2024-09-16T14:53:24.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:24.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:53:24.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 10 monclient(hunting): picked mon.noname-c con 0x7f70040a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:24.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 10 monclient(hunting): picked mon.noname-b con 0x7f70040a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:24.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 10 monclient(hunting): picked mon.noname-a con 0x7f70040a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:24.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f700413ca70 con 0x7f70040a2800 2024-09-16T14:53:24.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f700413cd50 con 0x7f70040a1d10 2024-09-16T14:53:24.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7004139f20 con 0x7f70040a0000 2024-09-16T14:53:24.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:24.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 10 monclient(hunting): authenticate will time out at 1628.840625s 2024-09-16T14:53:24.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70037fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f70040a2800 0x7f70041d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51216/0 (socket says 172.21.15.26:51216) 2024-09-16T14:53:24.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70037fe640 1 -- 172.21.15.26:0/4096655136 learned_addr learned my addr 172.21.15.26:0/4096655136 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:24.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707855506 0 0) 0x7f700413ca70 con 0x7f70040a2800 2024-09-16T14:53:24.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): my global_id is 194436 2024-09-16T14:53:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff0003530 con 0x7f70040a2800 2024-09-16T14:53:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295508072 0 0) 0x7f7004139f20 con 0x7f70040a0000 2024-09-16T14:53:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): my global_id is 234107 2024-09-16T14:53:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f700413ca70 con 0x7f70040a0000 2024-09-16T14:53:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1244698141 0 0) 0x7f6ff0003530 con 0x7f70040a2800 2024-09-16T14:53:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7004139f20 con 0x7f70040a2800 2024-09-16T14:53:24.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6ffc001ea0 con 0x7f70040a2800 2024-09-16T14:53:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:53:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): dump: 2024-09-16T14:53:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.735 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3400199428 0 0) 0x7f700413ca70 con 0x7f70040a0000 2024-09-16T14:53:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ff0003530 con 0x7f70040a0000 2024-09-16T14:53:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6ff8001210 con 0x7f70040a0000 2024-09-16T14:53:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): dump: 2024-09-16T14:53:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.737 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745036941 0 0) 0x7f700413cd50 con 0x7f70040a1d10 2024-09-16T14:53:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): my global_id is 224188 2024-09-16T14:53:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f700413ca70 con 0x7f70040a1d10 2024-09-16T14:53:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 129192088 0 0) 0x7f7004139f20 con 0x7f70040a2800 2024-09-16T14:53:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 >> v1:172.21.15.26:6790/0 conn(0x7f70040a0000 legacy=0x7f7004139100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 >> v1:172.21.15.189:6789/0 conn(0x7f70040a1d10 legacy=0x7f7004139810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:53:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 monclient: found mon.a 2024-09-16T14:53:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700409e500 con 0x7f70040a2800 2024-09-16T14:53:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient: _finish_auth 0 2024-09-16T14:53:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:53:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 5 monclient: authenticate success, global_id 194436 2024-09-16T14:53:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 10 monclient: _renew_subs 2024-09-16T14:53:24.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:24.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 1 -- 172.21.15.26:0/4096655136 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70040a65b0 con 0x7f70040a2800 2024-09-16T14:53:24.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 10 monclient: _renew_subs 2024-09-16T14:53:24.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:24.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 1 -- 172.21.15.26:0/4096655136 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f700413d030 con 0x7f70040a2800 2024-09-16T14:53:24.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ffc004ea0 con 0x7f70040a2800 2024-09-16T14:53:24.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:24.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6ffc005230 con 0x7f70040a2800 2024-09-16T14:53:24.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:24.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70017fa640 4 set_mon_vals no callback set 2024-09-16T14:53:24.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient: dump: 2024-09-16T14:53:24.740 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.740 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.740 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.740 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.740 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.740 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.740 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.740 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.740 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.740 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:24.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70097c8640 1 -- 172.21.15.26:0/4096655136 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6fe4006590 con 0x7f6fe4002830 2024-09-16T14:53:24.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6ffc005390 con 0x7f70040a2800 2024-09-16T14:53:24.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f6ffc044270 con 0x7f70040a2800 2024-09-16T14:53:24.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332469940 0 0) 0x7f6fe4006590 con 0x7f6fe4002830 2024-09-16T14:53:24.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient(hunting): my global_id is 194439 2024-09-16T14:53:24.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7004139f20 con 0x7f6fe4002830 2024-09-16T14:53:24.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2335189578 0 0) 0x7f7004139f20 con 0x7f6fe4002830 2024-09-16T14:53:24.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fe4006590 con 0x7f6fe4002830 2024-09-16T14:53:24.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6ff80047f0 con 0x7f6fe4002830 2024-09-16T14:53:24.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f70027fc640 10 monclient: get_auth_request con 0x7f6ff00364d0 auth_method 0 2024-09-16T14:53:24.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:24.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient: dump: 2024-09-16T14:53:24.742 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.742 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.742 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.742 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.742 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.742 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.742 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.742 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.742 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.742 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2016668568 0 0) 0x7f6fe4006590 con 0x7f6fe4002830 2024-09-16T14:53:24.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6fe400a0e0 con 0x7f6fe4002830 2024-09-16T14:53:24.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6ff8006a30 con 0x7f6fe4002830 2024-09-16T14:53:24.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:24.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:24.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 >> v1:172.21.15.26:6789/0 conn(0x7f6fe4002830 legacy=0x7f6fe4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.719+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 >> v1:172.21.15.26:6789/0 conn(0x7f6fe4002830 legacy=0x7f6fe4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:24.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f70097c8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:53:24.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f70097c8640 1 -- 172.21.15.26:0/4096655136 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6fe4009d80 con 0x7f6fe40090a0 2024-09-16T14:53:24.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1217196634 0 0) 0x7f6fe4009d80 con 0x7f6fe40090a0 2024-09-16T14:53:24.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 10 monclient(hunting): my global_id is 194442 2024-09-16T14:53:24.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe4006590 con 0x7f6fe40090a0 2024-09-16T14:53:24.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1969793427 0 0) 0x7f6fe4006590 con 0x7f6fe40090a0 2024-09-16T14:53:24.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fe4009d80 con 0x7f6fe40090a0 2024-09-16T14:53:24.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6ff8004e20 con 0x7f6fe40090a0 2024-09-16T14:53:24.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:24.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 10 monclient: dump: 2024-09-16T14:53:24.744 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.744 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.744 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.745 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.745 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.745 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.745 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.745 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.745 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.745 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1975359032 0 0) 0x7f6fe4009d80 con 0x7f6fe40090a0 2024-09-16T14:53:24.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6fe400c290 con 0x7f6fe40090a0 2024-09-16T14:53:24.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 213839128) 0x7f6ff8006ba0 con 0x7f6fe40090a0 2024-09-16T14:53:24.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:53:24.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:24.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 >> v1:172.21.15.26:6789/0 conn(0x7f6fe40090a0 legacy=0x7f6fe4009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f6feb7fe640 1 -- 172.21.15.26:0/4096655136 >> v1:172.21.15.26:6789/0 conn(0x7f6fe40090a0 legacy=0x7f6fe4009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:24.746 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:53:24.746 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:53:24.746 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:53:24.746 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:53:24.746 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 102, 2024-09-16T14:53:24.746 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:53:24.746 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:53:24.746 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:53:24.746 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:53:24.747 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:24.747 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:53:24.747 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:24.747 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:53:24.747 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:53:24.747 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:24.747 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:24.747 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:24.747 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:24.747 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:24.747 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:24.748 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:24.748 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:24.748 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:24.748 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:24.748 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:24.748 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:53:24.748 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:53:24.748 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:24.748 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:24.748 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:24.748 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:24.748 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:24.748 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:24.749 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:24.749 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:24.749 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:24.749 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:24.749 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:24.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.749 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:53:24.749 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:53:24.749 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:53:24.749 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:53:24.749 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:53:24.750 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:53:24.750 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:24.750 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:24.750 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:53:24.750 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:53:24.750 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:53:24.750 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:53:24.750 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:53:24.750 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:53:24.750 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:53:24.750 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:24.751 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:53:24.751 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:24.751 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:24.751 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:24.751 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:24.751 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:24.751 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:24.751 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:24.751 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:24.751 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:24.751 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:24.751 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:24.752 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:53:24.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.752 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:53:24.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.752 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:53:24.752 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:53:24.752 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:24.752 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:24.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.752 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:24.752 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:53:24.752 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:24.753 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:24.753 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:24.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.753 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:53:24.753 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:53:24.753 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:24.753 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:24.753 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:24.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.753 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:53:24.753 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:53:24.753 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:24.753 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:24.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.754 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:24.754 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:53:24.754 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:24.754 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:24.754 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:24.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.754 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:53:24.754 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:53:24.754 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:24.754 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:24.754 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:24.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.755 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:53:24.755 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:53:24.755 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:24.755 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:24.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.756 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:24.756 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:53:24.756 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:24.756 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:24.756 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:24.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.757 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:53:24.757 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:53:24.757 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:24.757 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:24.757 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:24.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:24.757 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:24.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.758 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:53:24.758 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:53:24.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.758 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:24.758 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:24.758 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:24.759 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:24.759 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:24.759 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:53:24.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.759 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:24.759 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:24.759 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:53:24.759 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:24.760 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:24.760 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:53:24.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.760 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:24.760 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:24.760 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 7 2024-09-16T14:53:24.761 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:24.761 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:24.761 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:53:24.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.761 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:24.761 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:24.761 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:24.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:24.762 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:24.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.762 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:53:24.762 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:53:24.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f70097c8640 1 -- 172.21.15.26:0/4096655136 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6ff00364d0 legacy=0x7f6ff0038990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f70097c8640 10 monclient: shutdown 2024-09-16T14:53:24.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f70097c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:24.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f70097c8640 1 -- 172.21.15.26:0/4096655136 >> v1:172.21.15.26:6789/0 conn(0x7f70040a2800 legacy=0x7f70041d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f70037fe640 1 -- 172.21.15.26:0/4096655136 reap_dead start 2024-09-16T14:53:24.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f70037fe640 1 -- 172.21.15.26:0/4096655136 reap_dead start 2024-09-16T14:53:24.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f70097c8640 1 -- 172.21.15.26:0/4096655136 shutdown_connections 2024-09-16T14:53:24.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f70097c8640 1 -- 172.21.15.26:0/4096655136 >> 172.21.15.26:0/4096655136 conn(0x7f70040a7f30 msgr2=0x7f700413e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:24.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f70097c8640 1 -- 172.21.15.26:0/4096655136 shutdown_connections 2024-09-16T14:53:24.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.727+0000 7f70097c8640 1 -- 172.21.15.26:0/4096655136 wait complete. 2024-09-16T14:53:24.765 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:53:24.765 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:53:24.765 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:53:24.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d52e0b640 10 monclient: get_monmap_and_config 2024-09-16T14:53:24.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d52e0b640 10 monclient: build_initial_monmap 2024-09-16T14:53:24.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d52e0b640 10 monclient: monmap: 2024-09-16T14:53:24.878 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:24.878 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.878 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:53:24.879048+0000 2024-09-16T14:53:24.878 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:53:24.879048+0000 2024-09-16T14:53:24.878 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:24.878 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.878 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:24.878 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:53:24.878 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:53:24.878 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d52e0b640 1 Processor -- start 2024-09-16T14:53:24.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d52e0b640 1 -- start start 2024-09-16T14:53:24.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d52e0b640 10 monclient: init 2024-09-16T14:53:24.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d52e0b640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:24.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d52e0b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:53:24.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d52e0b640 10 monclient(hunting): picked mon.noname-a con 0x7f0d4c09a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:24.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d52e0b640 10 monclient(hunting): picked mon.noname-c con 0x7f0d4c09d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:24.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d52e0b640 10 monclient(hunting): picked mon.noname-b con 0x7f0d4c09b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:24.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d52e0b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c09dc60 con 0x7f0d4c09a550 2024-09-16T14:53:24.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d52e0b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c099ae0 con 0x7f0d4c09d1b0 2024-09-16T14:53:24.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d52e0b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c09b8f0 con 0x7f0d4c09b040 2024-09-16T14:53:24.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d52e0b640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:24.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d52e0b640 10 monclient(hunting): authenticate will time out at 1628.999116s 2024-09-16T14:53:24.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d513a8640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0d4c09b040 0x7f0d4c13a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40982/0 (socket says 172.21.15.26:40982) 2024-09-16T14:53:24.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d513a8640 1 -- 172.21.15.26:0/2380439434 learned_addr learned my addr 172.21.15.26:0/2380439434 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:24.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 1 -- 172.21.15.26:0/2380439434 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897832711 0 0) 0x7f0d4c09b8f0 con 0x7f0d4c09b040 2024-09-16T14:53:24.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 10 monclient(hunting): my global_id is 234110 2024-09-16T14:53:24.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 1 -- 172.21.15.26:0/2380439434 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d34003530 con 0x7f0d4c09b040 2024-09-16T14:53:24.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 1 -- 172.21.15.26:0/2380439434 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715643988 0 0) 0x7f0d4c09dc60 con 0x7f0d4c09a550 2024-09-16T14:53:24.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 10 monclient(hunting): my global_id is 194445 2024-09-16T14:53:24.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 1 -- 172.21.15.26:0/2380439434 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d4c09b8f0 con 0x7f0d4c09a550 2024-09-16T14:53:24.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 1 -- 172.21.15.26:0/2380439434 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 972826734 0 0) 0x7f0d34003530 con 0x7f0d4c09b040 2024-09-16T14:53:24.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 1 -- 172.21.15.26:0/2380439434 >> v1:172.21.15.189:6789/0 conn(0x7f0d4c09d1b0 legacy=0x7f0d4c136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 1 -- 172.21.15.26:0/2380439434 >> v1:172.21.15.26:6789/0 conn(0x7f0d4c09a550 legacy=0x7f0d4c09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:53:24.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 1 monclient: found mon.noname-b 2024-09-16T14:53:24.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:24.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:53:24.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 1 -- 172.21.15.26:0/2380439434 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c09bbd0 con 0x7f0d4c09b040 2024-09-16T14:53:24.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:53:24.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.875+0000 7f0d4b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:53:24.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:24.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 5 monclient: authenticate success, global_id 234110 2024-09-16T14:53:24.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:24.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d4b7fe640 1 -- 172.21.15.26:0/2380439434 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0d3c001620 con 0x7f0d4c09b040 2024-09-16T14:53:24.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d4b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:53:24.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d4b7fe640 10 monclient: dump: 2024-09-16T14:53:24.889 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.889 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.889 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.889 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.890 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.890 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.890 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.890 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.890 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.890 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:24.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d4b7fe640 1 -- 172.21.15.26:0/2380439434 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d3c0018c0 con 0x7f0d4c09b040 2024-09-16T14:53:24.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:24.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d4b7fe640 1 -- 172.21.15.26:0/2380439434 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0d3c001c50 con 0x7f0d4c09b040 2024-09-16T14:53:24.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:24.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 4 set_mon_vals no callback set 2024-09-16T14:53:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d4b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d4b7fe640 10 monclient: dump: 2024-09-16T14:53:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.892 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.893 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 10 monclient: shutdown 2024-09-16T14:53:24.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:24.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 1 -- 172.21.15.26:0/2380439434 >> v1:172.21.15.26:6790/0 conn(0x7f0d4c09b040 legacy=0x7f0d4c13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 1 -- 172.21.15.26:0/2380439434 shutdown_connections 2024-09-16T14:53:24.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 1 -- 172.21.15.26:0/2380439434 >> 172.21.15.26:0/2380439434 conn(0x7f0d4c09fb50 msgr2=0x7f0d4c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:24.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 1 -- 172.21.15.26:0/2380439434 shutdown_connections 2024-09-16T14:53:24.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 1 -- 172.21.15.26:0/2380439434 wait complete. 2024-09-16T14:53:24.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 10 monclient: build_initial_monmap 2024-09-16T14:53:24.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 10 monclient: monmap: 2024-09-16T14:53:24.893 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:24.893 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:24.893 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:24.894 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:24.894 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:24.894 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:24.894 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:24.894 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:24.894 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:24.894 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 1 Processor -- start 2024-09-16T14:53:24.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 1 -- start start 2024-09-16T14:53:24.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 10 monclient: init 2024-09-16T14:53:24.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:24.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:53:24.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 10 monclient(hunting): picked mon.noname-a con 0x7f0d4c09a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:24.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 10 monclient(hunting): picked mon.noname-b con 0x7f0d4c09b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:24.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 10 monclient(hunting): picked mon.noname-c con 0x7f0d4c09d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:24.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c11aca0 con 0x7f0d4c09a550 2024-09-16T14:53:24.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c06ce40 con 0x7f0d4c09b040 2024-09-16T14:53:24.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c11a6e0 con 0x7f0d4c09d1b0 2024-09-16T14:53:24.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:24.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 10 monclient(hunting): authenticate will time out at 1629.002433s 2024-09-16T14:53:24.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d50ba7640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0d4c09a550 0x7f0d4c139fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51250/0 (socket says 172.21.15.26:51250) 2024-09-16T14:53:24.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d50ba7640 1 -- 172.21.15.26:0/4160275647 learned_addr learned my addr 172.21.15.26:0/4160275647 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:24.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544847393 0 0) 0x7f0d4c11a6e0 con 0x7f0d4c09d1b0 2024-09-16T14:53:24.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): my global_id is 234113 2024-09-16T14:53:24.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d38003530 con 0x7f0d4c09d1b0 2024-09-16T14:53:24.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519241658 0 0) 0x7f0d4c11aca0 con 0x7f0d4c09a550 2024-09-16T14:53:24.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): my global_id is 194448 2024-09-16T14:53:24.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d4c11a6e0 con 0x7f0d4c09a550 2024-09-16T14:53:24.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3122891051 0 0) 0x7f0d4c11a6e0 con 0x7f0d4c09a550 2024-09-16T14:53:24.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d4c11aca0 con 0x7f0d4c09a550 2024-09-16T14:53:24.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 72789479 0 0) 0x7f0d38003530 con 0x7f0d4c09d1b0 2024-09-16T14:53:24.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d4c11a6e0 con 0x7f0d4c09d1b0 2024-09-16T14:53:24.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0d44001c20 con 0x7f0d4c09a550 2024-09-16T14:53:24.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:53:24.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): dump: 2024-09-16T14:53:24.897 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.897 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.897 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.897 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.897 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.897 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.897 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.897 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.897 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.898 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:24.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0d3c001bd0 con 0x7f0d4c09d1b0 2024-09-16T14:53:24.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:24.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): dump: 2024-09-16T14:53:24.898 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.898 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.898 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.898 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.898 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.898 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.898 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.898 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.899 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597865001 0 0) 0x7f0d4c06ce40 con 0x7f0d4c09b040 2024-09-16T14:53:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient(hunting): my global_id is 224194 2024-09-16T14:53:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d38003530 con 0x7f0d4c09b040 2024-09-16T14:53:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2152694646 0 0) 0x7f0d4c11aca0 con 0x7f0d4c09a550 2024-09-16T14:53:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 >> v1:172.21.15.26:6790/0 conn(0x7f0d4c09d1b0 legacy=0x7f0d4c1cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 >> v1:172.21.15.189:6789/0 conn(0x7f0d4c09b040 legacy=0x7f0d4c13a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient: _finish_hunting 0 2024-09-16T14:53:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 monclient: found mon.a 2024-09-16T14:53:24.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:24.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:24.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c13ae00 con 0x7f0d4c09a550 2024-09-16T14:53:24.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient: _finish_auth 0 2024-09-16T14:53:24.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient: _check_auth_tickets 2024-09-16T14:53:24.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:24.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 5 monclient: authenticate success, global_id 194448 2024-09-16T14:53:24.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 10 monclient: _renew_subs 2024-09-16T14:53:24.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:24.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 1 -- 172.21.15.26:0/4160275647 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4c09f1f0 con 0x7f0d4c09a550 2024-09-16T14:53:24.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 10 monclient: _renew_subs 2024-09-16T14:53:24.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:24.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d52e0b640 1 -- 172.21.15.26:0/4160275647 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4c11a9c0 con 0x7f0d4c09a550 2024-09-16T14:53:24.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d44001dd0 con 0x7f0d4c09a550 2024-09-16T14:53:24.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:24.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0d44003bc0 con 0x7f0d4c09a550 2024-09-16T14:53:24.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:24.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d4affd640 4 set_mon_vals no callback set 2024-09-16T14:53:24.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.879+0000 7f0d497fa640 10 monclient: dump: 2024-09-16T14:53:24.901 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.901 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.901 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.901 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.901 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.901 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.902 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.902 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.902 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.902 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d52e0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:24.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d52e0b640 1 -- 172.21.15.26:0/4160275647 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0d2c006590 con 0x7f0d2c002830 2024-09-16T14:53:24.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0d44003da0 con 0x7f0d4c09a550 2024-09-16T14:53:24.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f0d44044150 con 0x7f0d4c09a550 2024-09-16T14:53:24.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317552360 0 0) 0x7f0d2c006590 con 0x7f0d2c002830 2024-09-16T14:53:24.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 10 monclient(hunting): my global_id is 234116 2024-09-16T14:53:24.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d4c11aca0 con 0x7f0d2c002830 2024-09-16T14:53:24.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2524736389 0 0) 0x7f0d4c11aca0 con 0x7f0d2c002830 2024-09-16T14:53:24.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d2c006590 con 0x7f0d2c002830 2024-09-16T14:53:24.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0d40001720 con 0x7f0d2c002830 2024-09-16T14:53:24.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d513a8640 10 monclient: get_auth_request con 0x7f0d380363b0 auth_method 0 2024-09-16T14:53:24.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:24.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 10 monclient: dump: 2024-09-16T14:53:24.904 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.904 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.904 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.905 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.905 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.905 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.905 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.905 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.905 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.905 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 959836355 0 0) 0x7f0d2c006590 con 0x7f0d2c002830 2024-09-16T14:53:24.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d2c00a0e0 con 0x7f0d2c002830 2024-09-16T14:53:24.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0d40001c30 con 0x7f0d2c002830 2024-09-16T14:53:24.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:24.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:24.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 >> v1:172.21.15.26:6790/0 conn(0x7f0d2c002830 legacy=0x7f0d2c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.883+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 >> v1:172.21.15.26:6790/0 conn(0x7f0d2c002830 legacy=0x7f0d2c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:24.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d52e0b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:53:24.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d52e0b640 1 -- 172.21.15.26:0/4160275647 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0d2c009d80 con 0x7f0d2c0090a0 2024-09-16T14:53:24.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301864361 0 0) 0x7f0d2c009d80 con 0x7f0d2c0090a0 2024-09-16T14:53:24.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 10 monclient(hunting): _negotiate 2024-09-16T14:53:24.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:24.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:24.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 10 monclient(hunting): my global_id is 234119 2024-09-16T14:53:24.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d2c006590 con 0x7f0d2c0090a0 2024-09-16T14:53:24.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1197461335 0 0) 0x7f0d2c006590 con 0x7f0d2c0090a0 2024-09-16T14:53:24.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d2c009d80 con 0x7f0d2c0090a0 2024-09-16T14:53:24.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0d400019d0 con 0x7f0d2c0090a0 2024-09-16T14:53:24.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:24.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:24.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 10 monclient: dump: 2024-09-16T14:53:24.910 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:24.910 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:24.911 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.911 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:24.911 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:24.911 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:24.911 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:24.911 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:24.911 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:24.912 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:24.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2459901822 0 0) 0x7f0d2c009d80 con 0x7f0d2c0090a0 2024-09-16T14:53:24.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0d2c00c290 con 0x7f0d2c0090a0 2024-09-16T14:53:24.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 735952477) 0x7f0d40004650 con 0x7f0d2c0090a0 2024-09-16T14:53:24.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:53:24.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:24.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 >> v1:172.21.15.26:6790/0 conn(0x7f0d2c0090a0 legacy=0x7f0d2c009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d497fa640 1 -- 172.21.15.26:0/4160275647 >> v1:172.21.15.26:6790/0 conn(0x7f0d2c0090a0 legacy=0x7f0d2c009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:24.913 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:53:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:53:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:53:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:53:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 102, 2024-09-16T14:53:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:53:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:53:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:53:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:53:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:53:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:53:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:53:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:24.915 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:24.916 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:53:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:53:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:24.917 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:24.918 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:53:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:53:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:53:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:53:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:53:24.919 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:53:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:53:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:53:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:53:24.920 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:53:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:53:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:53:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:53:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:53:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:24.921 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:24.922 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:53:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:53:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:53:24.923 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:53:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:53:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:53:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:53:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:24.924 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:53:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:53:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:53:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:24.925 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:53:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:53:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:53:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:53:24.926 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:53:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:53:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:53:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:24.927 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:53:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:53:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:53:24.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:24.929 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:53:24.929 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:53:24.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d52e0b640 1 -- 172.21.15.26:0/4160275647 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0d380363b0 legacy=0x7f0d38038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d52e0b640 10 monclient: shutdown 2024-09-16T14:53:24.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d52e0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:24.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d52e0b640 1 -- 172.21.15.26:0/4160275647 >> v1:172.21.15.26:6789/0 conn(0x7f0d4c09a550 legacy=0x7f0d4c139fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:24.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d513a8640 1 -- 172.21.15.26:0/4160275647 reap_dead start 2024-09-16T14:53:24.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d513a8640 1 -- 172.21.15.26:0/4160275647 reap_dead start 2024-09-16T14:53:24.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d52e0b640 1 -- 172.21.15.26:0/4160275647 shutdown_connections 2024-09-16T14:53:24.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d52e0b640 1 -- 172.21.15.26:0/4160275647 >> 172.21.15.26:0/4160275647 conn(0x7f0d4c09fb50 msgr2=0x7f0d4c0a1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:24.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d52e0b640 1 -- 172.21.15.26:0/4160275647 shutdown_connections 2024-09-16T14:53:24.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:24.891+0000 7f0d52e0b640 1 -- 172.21.15.26:0/4160275647 wait complete. 2024-09-16T14:53:24.932 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:53:24.932 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:53:24.932 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:53:25.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 10 monclient: get_monmap_and_config 2024-09-16T14:53:25.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 10 monclient: build_initial_monmap 2024-09-16T14:53:25.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 10 monclient: monmap: 2024-09-16T14:53:25.048 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:25.048 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:25.048 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:53:25.049557+0000 2024-09-16T14:53:25.048 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:53:25.049557+0000 2024-09-16T14:53:25.048 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:25.049 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:25.049 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:25.049 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:53:25.049 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:53:25.049 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:25.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 1 Processor -- start 2024-09-16T14:53:25.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 1 -- start start 2024-09-16T14:53:25.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 10 monclient: init 2024-09-16T14:53:25.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:25.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:53:25.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 10 monclient(hunting): picked mon.noname-a con 0x7f459c09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:25.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 10 monclient(hunting): picked mon.noname-c con 0x7f459c132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:25.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 10 monclient(hunting): picked mon.noname-b con 0x7f459c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:25.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c09d260 con 0x7f459c09cb80 2024-09-16T14:53:25.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c132a80 con 0x7f459c132650 2024-09-16T14:53:25.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c09cf60 con 0x7f459c138ff0 2024-09-16T14:53:25.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:25.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 10 monclient(hunting): authenticate will time out at 1629.169547s 2024-09-16T14:53:25.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a1793640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f459c09cb80 0x7f459c0717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51262/0 (socket says 172.21.15.26:51262) 2024-09-16T14:53:25.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a1793640 1 -- 172.21.15.26:0/3388189110 learned_addr learned my addr 172.21.15.26:0/3388189110 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:25.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 1 -- 172.21.15.26:0/3388189110 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74063327 0 0) 0x7f459c09d260 con 0x7f459c09cb80 2024-09-16T14:53:25.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient(hunting): _negotiate 2024-09-16T14:53:25.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:25.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:25.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient(hunting): my global_id is 194451 2024-09-16T14:53:25.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 1 -- 172.21.15.26:0/3388189110 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4588003530 con 0x7f459c09cb80 2024-09-16T14:53:25.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 1 -- 172.21.15.26:0/3388189110 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060909641 0 0) 0x7f459c09cf60 con 0x7f459c138ff0 2024-09-16T14:53:25.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient(hunting): _negotiate 2024-09-16T14:53:25.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:25.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:25.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient(hunting): my global_id is 234122 2024-09-16T14:53:25.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 1 -- 172.21.15.26:0/3388189110 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f459c09d260 con 0x7f459c138ff0 2024-09-16T14:53:25.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 1 -- 172.21.15.26:0/3388189110 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 798148076 0 0) 0x7f4588003530 con 0x7f459c09cb80 2024-09-16T14:53:25.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 1 -- 172.21.15.26:0/3388189110 >> v1:172.21.15.26:6790/0 conn(0x7f459c138ff0 legacy=0x7f459c13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:25.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 1 -- 172.21.15.26:0/3388189110 >> v1:172.21.15.189:6789/0 conn(0x7f459c132650 legacy=0x7f459c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:25.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient: _finish_hunting 0 2024-09-16T14:53:25.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 1 monclient: found mon.noname-a 2024-09-16T14:53:25.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:25.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:53:25.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 1 -- 172.21.15.26:0/3388189110 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c09f2d0 con 0x7f459c09cb80 2024-09-16T14:53:25.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient: _finish_auth 0 2024-09-16T14:53:25.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient: _check_auth_tickets 2024-09-16T14:53:25.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:25.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 1 -- 172.21.15.26:0/3388189110 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4594001620 con 0x7f459c09cb80 2024-09-16T14:53:25.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:25.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:53:25.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient: dump: 2024-09-16T14:53:25.063 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:25.063 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:25.063 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.063 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.063 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:25.063 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:25.063 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:25.063 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:25.064 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:25.064 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:25.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 5 monclient: authenticate success, global_id 194451 2024-09-16T14:53:25.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:25.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 1 -- 172.21.15.26:0/3388189110 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4594001890 con 0x7f459c09cb80 2024-09-16T14:53:25.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:25.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 1 -- 172.21.15.26:0/3388189110 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4594001c20 con 0x7f459c09cb80 2024-09-16T14:53:25.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:25.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 4 set_mon_vals no callback set 2024-09-16T14:53:25.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:25.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:25.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f4593fff640 10 monclient: dump: 2024-09-16T14:53:25.065 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:25.065 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:25.065 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.065 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.065 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:25.065 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:25.065 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:25.065 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:25.065 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:25.065 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:25.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 10 monclient: shutdown 2024-09-16T14:53:25.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:25.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 1 -- 172.21.15.26:0/3388189110 >> v1:172.21.15.26:6789/0 conn(0x7f459c09cb80 legacy=0x7f459c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:25.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 1 -- 172.21.15.26:0/3388189110 shutdown_connections 2024-09-16T14:53:25.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 1 -- 172.21.15.26:0/3388189110 >> 172.21.15.26:0/3388189110 conn(0x7f459c09fb50 msgr2=0x7f459c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:25.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 1 -- 172.21.15.26:0/3388189110 shutdown_connections 2024-09-16T14:53:25.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 1 -- 172.21.15.26:0/3388189110 wait complete. 2024-09-16T14:53:25.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 10 monclient: build_initial_monmap 2024-09-16T14:53:25.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.047+0000 7f45a39f7640 10 monclient: monmap: 2024-09-16T14:53:25.066 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:25.066 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:25.066 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:25.066 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:25.066 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:25.066 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:25.067 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:25.067 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:25.067 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:25.067 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:25.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 1 Processor -- start 2024-09-16T14:53:25.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 1 -- start start 2024-09-16T14:53:25.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 10 monclient: init 2024-09-16T14:53:25.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:25.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:53:25.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 10 monclient(hunting): picked mon.noname-b con 0x7f459c09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:25.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 10 monclient(hunting): picked mon.noname-a con 0x7f459c132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:25.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 10 monclient(hunting): picked mon.noname-c con 0x7f459c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:25.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c133950 con 0x7f459c132650 2024-09-16T14:53:25.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c134e00 con 0x7f459c09cb80 2024-09-16T14:53:25.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c13c700 con 0x7f459c138ff0 2024-09-16T14:53:25.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:25.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 10 monclient(hunting): authenticate will time out at 1629.172714s 2024-09-16T14:53:25.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a0f92640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f459c132650 0x7f459c1ca240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51270/0 (socket says 172.21.15.26:51270) 2024-09-16T14:53:25.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a0f92640 1 -- 172.21.15.26:0/3881779272 learned_addr learned my addr 172.21.15.26:0/3881779272 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:25.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a1f94640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f459c138ff0 0x7f459c1d1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:41024/0 (socket says 172.21.15.26:41024) 2024-09-16T14:53:25.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179432631 0 0) 0x7f459c133950 con 0x7f459c132650 2024-09-16T14:53:25.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:25.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:25.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:25.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): my global_id is 194454 2024-09-16T14:53:25.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4580003530 con 0x7f459c132650 2024-09-16T14:53:25.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322068446 0 0) 0x7f459c13c700 con 0x7f459c138ff0 2024-09-16T14:53:25.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:25.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:25.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:25.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): my global_id is 234125 2024-09-16T14:53:25.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f459c133950 con 0x7f459c138ff0 2024-09-16T14:53:25.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 825625896 0 0) 0x7f4580003530 con 0x7f459c132650 2024-09-16T14:53:25.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f459c13c700 con 0x7f459c132650 2024-09-16T14:53:25.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f458c001920 con 0x7f459c132650 2024-09-16T14:53:25.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:25.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:53:25.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): dump: 2024-09-16T14:53:25.070 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:25.070 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:25.070 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.070 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.070 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:25.070 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:25.070 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:25.070 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:25.070 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:25.070 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:25.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:25.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2416513238 0 0) 0x7f459c133950 con 0x7f459c138ff0 2024-09-16T14:53:25.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4580003530 con 0x7f459c138ff0 2024-09-16T14:53:25.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4598001c00 con 0x7f459c138ff0 2024-09-16T14:53:25.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:25.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:25.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): dump: 2024-09-16T14:53:25.071 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:25.071 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:25.071 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.072 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.072 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:25.072 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:25.072 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:25.072 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:25.072 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:25.072 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:25.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1852250578 0 0) 0x7f459c13c700 con 0x7f459c132650 2024-09-16T14:53:25.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 >> v1:172.21.15.26:6790/0 conn(0x7f459c138ff0 legacy=0x7f459c1d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:25.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 >> v1:172.21.15.189:6789/0 conn(0x7f459c09cb80 legacy=0x7f459c1c9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:25.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:53:25.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 monclient: found mon.a 2024-09-16T14:53:25.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:25.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:25.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c1ca950 con 0x7f459c132650 2024-09-16T14:53:25.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient: _finish_auth 0 2024-09-16T14:53:25.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:53:25.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:25.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 5 monclient: authenticate success, global_id 194454 2024-09-16T14:53:25.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 10 monclient: _renew_subs 2024-09-16T14:53:25.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:25.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 1 -- 172.21.15.26:0/3881779272 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459c138840 con 0x7f459c132650 2024-09-16T14:53:25.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 10 monclient: _renew_subs 2024-09-16T14:53:25.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:25.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 1 -- 172.21.15.26:0/3881779272 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f459c134b20 con 0x7f459c132650 2024-09-16T14:53:25.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f458c001c70 con 0x7f459c132650 2024-09-16T14:53:25.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:25.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f458c003a70 con 0x7f459c132650 2024-09-16T14:53:25.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:25.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:25.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45937fe640 4 set_mon_vals no callback set 2024-09-16T14:53:25.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient: dump: 2024-09-16T14:53:25.074 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:25.074 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:25.074 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.074 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.075 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:25.075 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:25.075 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:25.075 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:25.075 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:25.075 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:25.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:25.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a39f7640 1 -- 172.21.15.26:0/3881779272 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f457c006590 con 0x7f457c002830 2024-09-16T14:53:25.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f458c003bd0 con 0x7f459c132650 2024-09-16T14:53:25.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f458c043df0 con 0x7f459c132650 2024-09-16T14:53:25.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20737690 0 0) 0x7f457c006590 con 0x7f457c002830 2024-09-16T14:53:25.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:25.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:25.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:25.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient(hunting): my global_id is 224203 2024-09-16T14:53:25.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f459c13c700 con 0x7f457c002830 2024-09-16T14:53:25.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f45a1f94640 10 monclient: get_auth_request con 0x7f45800361c0 auth_method 0 2024-09-16T14:53:25.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4030795181 0 0) 0x7f459c13c700 con 0x7f457c002830 2024-09-16T14:53:25.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f457c006590 con 0x7f457c002830 2024-09-16T14:53:25.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4594001ba0 con 0x7f457c002830 2024-09-16T14:53:25.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:25.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:53:25.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.051+0000 7f4591ffb640 10 monclient: dump: 2024-09-16T14:53:25.076 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:25.076 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:25.076 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.077 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.077 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:25.077 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:25.077 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:25.077 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:25.077 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:25.077 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:25.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.055+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 125245872 0 0) 0x7f457c006590 con 0x7f457c002830 2024-09-16T14:53:25.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.055+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f457c00a0e0 con 0x7f457c002830 2024-09-16T14:53:25.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.055+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f4594001a00 con 0x7f457c002830 2024-09-16T14:53:25.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.055+0000 7f4591ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:25.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.055+0000 7f4591ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:25.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.055+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 >> v1:172.21.15.189:6789/0 conn(0x7f457c002830 legacy=0x7f457c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:25.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.055+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 >> v1:172.21.15.189:6789/0 conn(0x7f457c002830 legacy=0x7f457c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:25.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.059+0000 7f45a39f7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:53:25.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.059+0000 7f45a39f7640 1 -- 172.21.15.26:0/3881779272 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f457c009d80 con 0x7f457c0090a0 2024-09-16T14:53:25.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.059+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331083629 0 0) 0x7f457c009d80 con 0x7f457c0090a0 2024-09-16T14:53:25.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.059+0000 7f4591ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:25.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.059+0000 7f4591ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:25.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.059+0000 7f4591ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:25.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.059+0000 7f4591ffb640 10 monclient(hunting): my global_id is 224206 2024-09-16T14:53:25.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.059+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f457c006590 con 0x7f457c0090a0 2024-09-16T14:53:25.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1967790482 0 0) 0x7f457c006590 con 0x7f457c0090a0 2024-09-16T14:53:25.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f457c009d80 con 0x7f457c0090a0 2024-09-16T14:53:25.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4594004c40 con 0x7f457c0090a0 2024-09-16T14:53:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f4591ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f4591ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:53:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f4591ffb640 10 monclient: dump: 2024-09-16T14:53:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:25.080 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:25.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4043010784 0 0) 0x7f457c009d80 con 0x7f457c0090a0 2024-09-16T14:53:25.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f457c00c290 con 0x7f457c0090a0 2024-09-16T14:53:25.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 116453462) 0x7f4594005f30 con 0x7f457c0090a0 2024-09-16T14:53:25.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f4591ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:53:25.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f4591ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:25.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 >> v1:172.21.15.189:6789/0 conn(0x7f457c0090a0 legacy=0x7f457c009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:25.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f4591ffb640 1 -- 172.21.15.26:0/3881779272 >> v1:172.21.15.189:6789/0 conn(0x7f457c0090a0 legacy=0x7f457c009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:25.081 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:53:25.081 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:53:25.082 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:53:25.082 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:53:25.082 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 102, 2024-09-16T14:53:25.082 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:53:25.082 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:53:25.082 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:53:25.082 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:53:25.082 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:25.082 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:53:25.082 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:25.082 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:53:25.082 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:53:25.082 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:25.083 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:25.083 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:25.083 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:25.083 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:25.083 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:25.083 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:25.083 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:25.083 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:25.083 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:25.083 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:25.083 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:53:25.083 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:53:25.083 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:25.083 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:25.084 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:25.084 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:25.084 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:25.084 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:25.084 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:25.084 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:25.084 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:25.084 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:25.084 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:25.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:25.084 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:53:25.085 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:53:25.085 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:53:25.085 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:53:25.085 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:53:25.085 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:53:25.085 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:25.085 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:25.085 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:53:25.085 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:53:25.085 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:53:25.085 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:53:25.085 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:53:25.085 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:53:25.085 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:53:25.086 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:25.086 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:53:25.086 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:25.086 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:25.086 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:25.086 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:25.086 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:25.086 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:25.086 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:25.086 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:25.087 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:25.087 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:25.087 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:25.087 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:53:25.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:25.087 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:53:25.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:25.087 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:53:25.087 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:53:25.087 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:25.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:25.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:25.088 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:25.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:53:25.088 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:25.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:25.088 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:25.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:25.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:53:25.088 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:53:25.088 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:25.088 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:25.088 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:25.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:25.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:25.089 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:53:25.089 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:53:25.089 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:25.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:25.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:25.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:25.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:53:25.089 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:25.090 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:25.090 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:25.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:25.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:53:25.090 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:53:25.090 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:25.090 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:25.090 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:25.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:25.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:25.090 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:53:25.090 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:53:25.090 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:25.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:25.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:25.091 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:25.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:53:25.091 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:25.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:25.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:25.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:25.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:53:25.091 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:53:25.091 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:25.091 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:25.091 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:25.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:25.092 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:25.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:25.092 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:53:25.092 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:53:25.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:25.092 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:25.092 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:25.092 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:25.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:25.092 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:25.092 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:53:25.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:25.093 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:25.093 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:25.093 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:25.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:25.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:25.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:25.093 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:53:25.093 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:53:25.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f45a39f7640 1 -- 172.21.15.26:0/3881779272 >> v1:172.21.15.189:6812/3196161582 conn(0x7f45800361c0 legacy=0x7f4580038680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:25.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f45a39f7640 10 monclient: shutdown 2024-09-16T14:53:25.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f45a39f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:25.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f45a39f7640 1 -- 172.21.15.26:0/3881779272 >> v1:172.21.15.26:6789/0 conn(0x7f459c132650 legacy=0x7f459c1ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:25.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f45a1f94640 1 -- 172.21.15.26:0/3881779272 reap_dead start 2024-09-16T14:53:25.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f45a1f94640 1 -- 172.21.15.26:0/3881779272 reap_dead start 2024-09-16T14:53:25.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f45a39f7640 1 -- 172.21.15.26:0/3881779272 shutdown_connections 2024-09-16T14:53:25.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f45a39f7640 1 -- 172.21.15.26:0/3881779272 >> 172.21.15.26:0/3881779272 conn(0x7f459c09fb50 msgr2=0x7f459c0a1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:25.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f45a39f7640 1 -- 172.21.15.26:0/3881779272 shutdown_connections 2024-09-16T14:53:25.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.063+0000 7f45a39f7640 1 -- 172.21.15.26:0/3881779272 wait complete. 2024-09-16T14:53:25.095 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:53:25.095 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:53:25.096 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:53:25.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.203+0000 7fe057f7d640 10 monclient: get_monmap_and_config 2024-09-16T14:53:25.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.203+0000 7fe057f7d640 10 monclient: build_initial_monmap 2024-09-16T14:53:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.203+0000 7fe057f7d640 10 monclient: monmap: 2024-09-16T14:53:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:53:25.209159+0000 2024-09-16T14:53:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:53:25.209159+0000 2024-09-16T14:53:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:53:25.208 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:53:25.208 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:25.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 1 Processor -- start 2024-09-16T14:53:25.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 1 -- start start 2024-09-16T14:53:25.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 10 monclient: init 2024-09-16T14:53:25.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:25.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:53:25.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 10 monclient(hunting): picked mon.noname-a con 0x7fe0500a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:25.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 10 monclient(hunting): picked mon.noname-b con 0x7fe0500a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:25.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 10 monclient(hunting): picked mon.noname-c con 0x7fe0500a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:25.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0500a3250 con 0x7fe0500a1d10 2024-09-16T14:53:25.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05009f600 con 0x7fe0500a2800 2024-09-16T14:53:25.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0500a3830 con 0x7fe0500a0000 2024-09-16T14:53:25.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:25.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 10 monclient(hunting): authenticate will time out at 1629.329376s 2024-09-16T14:53:25.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe055518640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe0500a0000 0x7fe050139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:41032/0 (socket says 172.21.15.26:41032) 2024-09-16T14:53:25.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe055d19640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe0500a1d10 0x7fe0500a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51272/0 (socket says 172.21.15.26:51272) 2024-09-16T14:53:25.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe055518640 1 -- 172.21.15.26:0/843278793 learned_addr learned my addr 172.21.15.26:0/843278793 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:25.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 1 -- 172.21.15.26:0/843278793 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678856124 0 0) 0x7fe0500a3250 con 0x7fe0500a1d10 2024-09-16T14:53:25.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient(hunting): _negotiate 2024-09-16T14:53:25.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:25.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:25.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient(hunting): my global_id is 194457 2024-09-16T14:53:25.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 1 -- 172.21.15.26:0/843278793 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe03c003530 con 0x7fe0500a1d10 2024-09-16T14:53:25.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 1 -- 172.21.15.26:0/843278793 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520534 0 0) 0x7fe0500a3830 con 0x7fe0500a0000 2024-09-16T14:53:25.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient(hunting): _negotiate 2024-09-16T14:53:25.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:25.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:25.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient(hunting): my global_id is 234128 2024-09-16T14:53:25.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 1 -- 172.21.15.26:0/843278793 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0500a3250 con 0x7fe0500a0000 2024-09-16T14:53:25.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 1 -- 172.21.15.26:0/843278793 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3196625884 0 0) 0x7fe0500a3250 con 0x7fe0500a0000 2024-09-16T14:53:25.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 1 -- 172.21.15.26:0/843278793 >> v1:172.21.15.189:6789/0 conn(0x7fe0500a2800 legacy=0x7fe05013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:25.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 1 -- 172.21.15.26:0/843278793 >> v1:172.21.15.26:6789/0 conn(0x7fe0500a1d10 legacy=0x7fe0500a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:25.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient: _finish_hunting 0 2024-09-16T14:53:25.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 1 monclient: found mon.noname-b 2024-09-16T14:53:25.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:25.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:53:25.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 1 -- 172.21.15.26:0/843278793 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05009ed60 con 0x7fe0500a0000 2024-09-16T14:53:25.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient: _finish_auth 0 2024-09-16T14:53:25.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient: _check_auth_tickets 2024-09-16T14:53:25.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:25.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 1 -- 172.21.15.26:0/843278793 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe0400014b0 con 0x7fe0500a0000 2024-09-16T14:53:25.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:25.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:53:25.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient: dump: 2024-09-16T14:53:25.218 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:25.218 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:25.218 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.218 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.218 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:25.218 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:25.218 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:25.218 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:25.218 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:25.218 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:25.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 5 monclient: authenticate success, global_id 234128 2024-09-16T14:53:25.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:25.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 1 -- 172.21.15.26:0/843278793 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0400016e0 con 0x7fe0500a0000 2024-09-16T14:53:25.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:25.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 1 -- 172.21.15.26:0/843278793 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe040001ab0 con 0x7fe0500a0000 2024-09-16T14:53:25.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:25.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:25.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe054d17640 10 monclient: dump: 2024-09-16T14:53:25.219 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:25.219 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:25.219 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.219 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.219 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:25.220 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:25.220 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:25.220 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:25.220 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:25.220 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:25.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:25.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 4 set_mon_vals no callback set 2024-09-16T14:53:25.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 10 monclient: shutdown 2024-09-16T14:53:25.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:25.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 1 -- 172.21.15.26:0/843278793 >> v1:172.21.15.26:6790/0 conn(0x7fe0500a0000 legacy=0x7fe050139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:25.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 1 -- 172.21.15.26:0/843278793 shutdown_connections 2024-09-16T14:53:25.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 1 -- 172.21.15.26:0/843278793 >> 172.21.15.26:0/843278793 conn(0x7fe0500a7f30 msgr2=0x7fe0500a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:25.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 1 -- 172.21.15.26:0/843278793 shutdown_connections 2024-09-16T14:53:25.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 1 -- 172.21.15.26:0/843278793 wait complete. 2024-09-16T14:53:25.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 10 monclient: build_initial_monmap 2024-09-16T14:53:25.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 10 monclient: monmap: 2024-09-16T14:53:25.221 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:25.221 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:25.221 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:25.221 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:25.221 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:25.221 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:25.221 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:25.221 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:25.221 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:25.221 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:25.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.207+0000 7fe057f7d640 1 Processor -- start 2024-09-16T14:53:25.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 1 -- start start 2024-09-16T14:53:25.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 10 monclient: init 2024-09-16T14:53:25.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:25.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:53:25.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 10 monclient(hunting): picked mon.noname-a con 0x7fe0500a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:25.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 10 monclient(hunting): picked mon.noname-b con 0x7fe0500a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:25.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 10 monclient(hunting): picked mon.noname-c con 0x7fe0500a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:25.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe050136170 con 0x7fe0500a0000 2024-09-16T14:53:25.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0500a69e0 con 0x7fe0500a1d10 2024-09-16T14:53:25.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe050132df0 con 0x7fe0500a2800 2024-09-16T14:53:25.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:25.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 10 monclient(hunting): authenticate will time out at 1629.332502s 2024-09-16T14:53:25.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe055d19640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe0500a0000 0x7fe050131fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51276/0 (socket says 172.21.15.26:51276) 2024-09-16T14:53:25.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe055d19640 1 -- 172.21.15.26:0/3856379147 learned_addr learned my addr 172.21.15.26:0/3856379147 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:25.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085753313 0 0) 0x7fe050136170 con 0x7fe0500a0000 2024-09-16T14:53:25.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): _negotiate 2024-09-16T14:53:25.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:25.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:25.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): my global_id is 194460 2024-09-16T14:53:25.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe034003530 con 0x7fe0500a0000 2024-09-16T14:53:25.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018269976 0 0) 0x7fe050132df0 con 0x7fe0500a2800 2024-09-16T14:53:25.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): _negotiate 2024-09-16T14:53:25.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:25.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:25.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): my global_id is 234131 2024-09-16T14:53:25.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe050136170 con 0x7fe0500a2800 2024-09-16T14:53:25.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1656225924 0 0) 0x7fe034003530 con 0x7fe0500a0000 2024-09-16T14:53:25.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe050132df0 con 0x7fe0500a0000 2024-09-16T14:53:25.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe04c001e00 con 0x7fe0500a0000 2024-09-16T14:53:25.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:25.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:53:25.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): dump: 2024-09-16T14:53:25.224 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:25.224 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:25.224 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.224 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.224 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:25.225 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1366390740 0 0) 0x7fe050136170 con 0x7fe0500a2800 2024-09-16T14:53:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe034003530 con 0x7fe0500a2800 2024-09-16T14:53:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe0480016f0 con 0x7fe0500a2800 2024-09-16T14:53:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): dump: 2024-09-16T14:53:25.225 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:25.226 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:25.226 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.226 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.226 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:25.226 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:25.226 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:25.226 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:25.226 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:25.226 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:25.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe05651a640 1 -- 172.21.15.26:0/3856379147 >> v1:172.21.15.26:6790/0 conn(0x7fe0500a2800 legacy=0x7fe0501d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T14:53:25.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe05651a640 1 -- 172.21.15.26:0/3856379147 >> v1:172.21.15.26:6790/0 conn(0x7fe0500a2800 legacy=0x7fe0501d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:53:25.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe05651a640 1 --1- 172.21.15.26:0/3856379147 >> v1:172.21.15.26:6790/0 conn(0x7fe0500a2800 0x7fe0501d19b0 :-1 s=OPENED pgs=1222 cs=1 l=1).handle_message read tag failed 2024-09-16T14:53:25.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 244539426 0 0) 0x7fe0500a69e0 con 0x7fe0500a1d10 2024-09-16T14:53:25.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe05651a640 1 --1- 172.21.15.26:0/3856379147 >> v1:172.21.15.26:6790/0 conn(0x7fe0500a2800 0x7fe0501d19b0 :-1 s=OPENED pgs=1222 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:53:25.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): _negotiate 2024-09-16T14:53:25.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:25.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:25.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): my global_id is 224212 2024-09-16T14:53:25.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe050136170 con 0x7fe0500a1d10 2024-09-16T14:53:25.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6790/0 2024-09-16T14:53:25.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2998857297 0 0) 0x7fe050132df0 con 0x7fe0500a0000 2024-09-16T14:53:25.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 >> v1:172.21.15.26:6790/0 conn(0x7fe0500a2800 legacy=0x7fe0501d19b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:25.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 >> v1:172.21.15.189:6789/0 conn(0x7fe0500a1d10 legacy=0x7fe0501326e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:25.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient: _finish_hunting 0 2024-09-16T14:53:25.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 monclient: found mon.a 2024-09-16T14:53:25.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:25.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:25.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe050134750 con 0x7fe0500a0000 2024-09-16T14:53:25.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient: _finish_auth 0 2024-09-16T14:53:25.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient: _check_auth_tickets 2024-09-16T14:53:25.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:25.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 5 monclient: authenticate success, global_id 194460 2024-09-16T14:53:25.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 10 monclient: _renew_subs 2024-09-16T14:53:25.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:25.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 1 -- 172.21.15.26:0/3856379147 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0500a64b0 con 0x7fe0500a0000 2024-09-16T14:53:25.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 10 monclient: _renew_subs 2024-09-16T14:53:25.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:25.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 1 -- 172.21.15.26:0/3856379147 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0501344d0 con 0x7fe0500a0000 2024-09-16T14:53:25.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe04c0020f0 con 0x7fe0500a0000 2024-09-16T14:53:25.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:25.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe04c004e20 con 0x7fe0500a0000 2024-09-16T14:53:25.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:25.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:25.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe047fff640 4 set_mon_vals no callback set 2024-09-16T14:53:25.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 10 monclient: dump: 2024-09-16T14:53:25.229 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:25.229 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:25.229 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.229 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:25.229 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:25.229 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:25.229 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:25.230 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:25.230 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:25.230 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:25.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:25.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:25.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:25.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe057f7d640 1 -- 172.21.15.26:0/3856379147 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe038005050 con 0x7fe0500a0000 2024-09-16T14:53:25.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe04c00f890 con 0x7fe0500a0000 2024-09-16T14:53:25.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fe04c044090 con 0x7fe0500a0000 2024-09-16T14:53:25.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.211+0000 7fe055518640 10 monclient: get_auth_request con 0x7fe0340363d0 auth_method 0 2024-09-16T14:53:25.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.215+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe04c010f40 con 0x7fe0500a0000 2024-09-16T14:53:25.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.215+0000 7fe0467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:25.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.215+0000 7fe0467fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:25.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.435+0000 7fe057f7d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:53:25.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.435+0000 7fe057f7d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:53:25.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.435+0000 7fe057f7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:25.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.435+0000 7fe057f7d640 1 -- 172.21.15.26:0/3856379147 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fe038005b10 con 0x7fe0500a0000 2024-09-16T14:53:25.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.439+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.439+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=3931903047,config=692619927,health=2891791466,kv=263502946,logm=517422607}) 2024-09-16T14:53:25.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.439+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=1582742926,config=692619927,health=2496068776,kv=1867599010,logm=2171825210}) 2024-09-16T14:53:25.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.439+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.439+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=3931903047,config=692619927,health=2891791466,kv=263502946,logm=517422607}) 2024-09-16T14:53:25.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.439+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=1490897961,config=692619927,health=1552921670,kv=1867599010,logm=2255805976}) 2024-09-16T14:53:25.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.439+0000 7fe0467fc640 1 -- 172.21.15.26:0/3856379147 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fe04c012fe0 con 0x7fe0500a0000 2024-09-16T14:53:25.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.439+0000 7fe0467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:53:25.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.439+0000 7fe0467fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.439+0000 7fe057f7d640 1 -- 172.21.15.26:0/3856379147 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe0340363d0 legacy=0x7fe034038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.439+0000 7fe057f7d640 10 monclient: shutdown 2024-09-16T14:53:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.439+0000 7fe057f7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.439+0000 7fe057f7d640 1 -- 172.21.15.26:0/3856379147 >> v1:172.21.15.26:6789/0 conn(0x7fe0500a0000 legacy=0x7fe050131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.439+0000 7fe057f7d640 1 -- 172.21.15.26:0/3856379147 shutdown_connections 2024-09-16T14:53:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.439+0000 7fe057f7d640 1 -- 172.21.15.26:0/3856379147 >> 172.21.15.26:0/3856379147 conn(0x7fe0500a7f30 msgr2=0x7fe05006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.439+0000 7fe057f7d640 1 -- 172.21.15.26:0/3856379147 shutdown_connections 2024-09-16T14:53:25.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:25.439+0000 7fe057f7d640 1 -- 172.21.15.26:0/3856379147 wait complete. 2024-09-16T14:53:25.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.439+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.439+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1840370410}) 2024-09-16T14:53:25.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.439+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3611692672}) 2024-09-16T14:53:25.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.439+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.439+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1840370410}) 2024-09-16T14:53:25.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.439+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1775661340}) 2024-09-16T14:53:25.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.443+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.443+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=894976109}) 2024-09-16T14:53:25.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.443+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882951893}) 2024-09-16T14:53:25.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.443+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.443+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=894976109}) 2024-09-16T14:53:25.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.443+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2012582534}) 2024-09-16T14:53:25.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.447+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.447+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835125335}) 2024-09-16T14:53:25.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.447+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840615895}) 2024-09-16T14:53:25.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.447+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.447+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835125335}) 2024-09-16T14:53:25.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.447+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1388259843}) 2024-09-16T14:53:25.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.447+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.452 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.447+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3818949432}) 2024-09-16T14:53:25.452 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.447+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=263610002}) 2024-09-16T14:53:25.452 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.447+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.452 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.447+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3818949432}) 2024-09-16T14:53:25.452 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.447+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2935243107}) 2024-09-16T14:53:25.454 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.451+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.454 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.451+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2626589306}) 2024-09-16T14:53:25.454 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.451+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2092906369}) 2024-09-16T14:53:25.454 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.451+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.454 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.451+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2626589306}) 2024-09-16T14:53:25.454 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.451+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1158917603}) 2024-09-16T14:53:25.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.451+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.456 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.451+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1226991602}) 2024-09-16T14:53:25.456 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.451+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3668873988}) 2024-09-16T14:53:25.456 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.451+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.456 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.451+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1226991602}) 2024-09-16T14:53:25.456 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.451+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=66758132}) 2024-09-16T14:53:25.458 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.455+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.458 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.455+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3395794755}) 2024-09-16T14:53:25.458 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.455+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2161980662}) 2024-09-16T14:53:25.458 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.455+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.458 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.455+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3395794755}) 2024-09-16T14:53:25.458 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.455+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3480981518}) 2024-09-16T14:53:25.459 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:53:25.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.455+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.455+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=777602191}) 2024-09-16T14:53:25.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.455+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1553931449}) 2024-09-16T14:53:25.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.455+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.455+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=777602191}) 2024-09-16T14:53:25.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.455+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4238055630}) 2024-09-16T14:53:25.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.459+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.459+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057125354}) 2024-09-16T14:53:25.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.459+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2754586218}) 2024-09-16T14:53:25.462 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.459+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.462 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.459+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057125354}) 2024-09-16T14:53:25.462 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.459+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1490127208}) 2024-09-16T14:53:25.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.459+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.459+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=537689074}) 2024-09-16T14:53:25.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.459+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2598896008}) 2024-09-16T14:53:25.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.459+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.459+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=537689074}) 2024-09-16T14:53:25.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.459+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3128576120}) 2024-09-16T14:53:25.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.463+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.463+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3380037249}) 2024-09-16T14:53:25.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.463+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2791222895}) 2024-09-16T14:53:25.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.463+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.463+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3380037249}) 2024-09-16T14:53:25.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.463+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=457724384}) 2024-09-16T14:53:25.467 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.463+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.463+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3314278330}) 2024-09-16T14:53:25.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.463+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=594396659}) 2024-09-16T14:53:25.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.463+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.463+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3314278330}) 2024-09-16T14:53:25.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.463+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=798371577}) 2024-09-16T14:53:25.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.467+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.467+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2983828486}) 2024-09-16T14:53:25.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.467+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096239715}) 2024-09-16T14:53:25.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.467+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.467+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2983828486}) 2024-09-16T14:53:25.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.467+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1972143723}) 2024-09-16T14:53:25.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.471+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.471+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1377669035}) 2024-09-16T14:53:25.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.471+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3269853551}) 2024-09-16T14:53:25.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.471+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.471+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1377669035}) 2024-09-16T14:53:25.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.471+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3963058921}) 2024-09-16T14:53:25.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.471+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.471+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578109130}) 2024-09-16T14:53:25.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.471+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3001956632}) 2024-09-16T14:53:25.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.471+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.475 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.471+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578109130}) 2024-09-16T14:53:25.475 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.471+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1124767071}) 2024-09-16T14:53:25.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.475+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.475+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=57707880}) 2024-09-16T14:53:25.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.475+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3246283412}) 2024-09-16T14:53:25.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.475+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.475+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=57707880}) 2024-09-16T14:53:25.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.475+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3387386670}) 2024-09-16T14:53:25.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.475+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.475+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2296413488}) 2024-09-16T14:53:25.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.475+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=449953760}) 2024-09-16T14:53:25.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.475+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.475+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2296413488}) 2024-09-16T14:53:25.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.475+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=75473513}) 2024-09-16T14:53:25.480 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.479+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.479+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2036894267}) 2024-09-16T14:53:25.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.479+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3598064151}) 2024-09-16T14:53:25.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.479+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.479+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2036894267}) 2024-09-16T14:53:25.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.479+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2284492197}) 2024-09-16T14:53:25.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.479+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.479+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=838801126}) 2024-09-16T14:53:25.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.479+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649315093}) 2024-09-16T14:53:25.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.479+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.479+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=838801126}) 2024-09-16T14:53:25.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.479+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1780258352}) 2024-09-16T14:53:25.485 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.483+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.485 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.483+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3377405068}) 2024-09-16T14:53:25.485 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.483+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=343964368}) 2024-09-16T14:53:25.485 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.483+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.485 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.483+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3377405068}) 2024-09-16T14:53:25.485 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.483+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1310009742}) 2024-09-16T14:53:25.487 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.483+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.487 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.483+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2417572901}) 2024-09-16T14:53:25.487 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.483+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845695331}) 2024-09-16T14:53:25.487 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.483+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.487 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.483+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2417572901}) 2024-09-16T14:53:25.487 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.483+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2347227872}) 2024-09-16T14:53:25.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.487+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.487+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2642076986}) 2024-09-16T14:53:25.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.487+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2389716077}) 2024-09-16T14:53:25.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.487+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.487+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2642076986}) 2024-09-16T14:53:25.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.487+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1406538204}) 2024-09-16T14:53:25.491 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.487+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.491 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.487+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=966144771}) 2024-09-16T14:53:25.491 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.487+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=928160939}) 2024-09-16T14:53:25.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.487+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.487+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=966144771}) 2024-09-16T14:53:25.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.487+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=251212184}) 2024-09-16T14:53:25.493 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.491+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.491+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3735921121}) 2024-09-16T14:53:25.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.491+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1746095865}) 2024-09-16T14:53:25.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.491+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.491+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3735921121}) 2024-09-16T14:53:25.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.491+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=495715285}) 2024-09-16T14:53:25.495 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.491+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.491+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3424211302}) 2024-09-16T14:53:25.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.491+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=260205149}) 2024-09-16T14:53:25.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.491+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.491+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3424211302}) 2024-09-16T14:53:25.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.495+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3668730695}) 2024-09-16T14:53:25.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.495+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.495+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3291142754}) 2024-09-16T14:53:25.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.495+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635795752}) 2024-09-16T14:53:25.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.495+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.495+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3291142754}) 2024-09-16T14:53:25.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.495+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2457992626}) 2024-09-16T14:53:25.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.499+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.499+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=252911397}) 2024-09-16T14:53:25.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.499+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3918536711}) 2024-09-16T14:53:25.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.499+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.499+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=252911397}) 2024-09-16T14:53:25.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.499+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3872462037}) 2024-09-16T14:53:25.502 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.499+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.502 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.499+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3581808018}) 2024-09-16T14:53:25.502 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.499+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4277784763}) 2024-09-16T14:53:25.502 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.499+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.502 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.499+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3581808018}) 2024-09-16T14:53:25.502 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.499+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1045333238}) 2024-09-16T14:53:25.504 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.503+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.504 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.503+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3224613533}) 2024-09-16T14:53:25.504 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.503+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696718789}) 2024-09-16T14:53:25.504 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.503+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.504 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.503+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3224613533}) 2024-09-16T14:53:25.504 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.503+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1643776470}) 2024-09-16T14:53:25.506 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.503+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.506 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.503+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=925886898}) 2024-09-16T14:53:25.506 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.503+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2125160308}) 2024-09-16T14:53:25.506 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.503+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.506 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.503+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=925886898}) 2024-09-16T14:53:25.506 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.503+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1156623254}) 2024-09-16T14:53:25.508 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.507+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.508 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.507+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=340808794}) 2024-09-16T14:53:25.508 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.507+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622255362}) 2024-09-16T14:53:25.508 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.507+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.508 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.507+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=340808794}) 2024-09-16T14:53:25.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.507+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3119428567}) 2024-09-16T14:53:25.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.507+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.507+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3807598037}) 2024-09-16T14:53:25.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.507+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3924443758}) 2024-09-16T14:53:25.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.507+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.507+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3807598037}) 2024-09-16T14:53:25.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.507+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1826464186}) 2024-09-16T14:53:25.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.511+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.511+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3922735286}) 2024-09-16T14:53:25.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.511+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199088343}) 2024-09-16T14:53:25.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.511+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.511+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3922735286}) 2024-09-16T14:53:25.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.511+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3284146858}) 2024-09-16T14:53:25.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.511+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.511+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2020546031}) 2024-09-16T14:53:25.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.511+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3629105072}) 2024-09-16T14:53:25.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.511+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.511+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2020546031}) 2024-09-16T14:53:25.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.511+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3701604229}) 2024-09-16T14:53:25.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.515+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.515+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1783218121}) 2024-09-16T14:53:25.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.515+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3074263071}) 2024-09-16T14:53:25.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.515+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.515+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1783218121}) 2024-09-16T14:53:25.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.515+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4213411829}) 2024-09-16T14:53:25.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.515+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.515+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=894136668}) 2024-09-16T14:53:25.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.515+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2383358670}) 2024-09-16T14:53:25.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.515+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.515+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=894136668}) 2024-09-16T14:53:25.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.515+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1807779730}) 2024-09-16T14:53:25.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.519+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.519+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=196898006}) 2024-09-16T14:53:25.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.519+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000037443}) 2024-09-16T14:53:25.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.519+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.519+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=196898006}) 2024-09-16T14:53:25.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.519+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3781252656}) 2024-09-16T14:53:25.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.519+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.523 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.519+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=419910954}) 2024-09-16T14:53:25.523 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.519+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=400337065}) 2024-09-16T14:53:25.523 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.519+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.523 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.519+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=419910954}) 2024-09-16T14:53:25.523 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.519+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1278406227}) 2024-09-16T14:53:25.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.523+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.523+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3610167855}) 2024-09-16T14:53:25.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.523+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3627362090}) 2024-09-16T14:53:25.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.523+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.523+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3610167855}) 2024-09-16T14:53:25.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.523+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3331651367}) 2024-09-16T14:53:25.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.523+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.523+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3439065183}) 2024-09-16T14:53:25.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.523+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2003326527}) 2024-09-16T14:53:25.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.523+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.523+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3439065183}) 2024-09-16T14:53:25.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.523+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248613480}) 2024-09-16T14:53:25.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.527+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.527+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2824165069}) 2024-09-16T14:53:25.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.527+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1229465235}) 2024-09-16T14:53:25.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.527+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.527+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2824165069}) 2024-09-16T14:53:25.530 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.527+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=309091925}) 2024-09-16T14:53:25.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.527+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.527+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=316447735}) 2024-09-16T14:53:25.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.527+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3842640438}) 2024-09-16T14:53:25.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.527+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.527+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=316447735}) 2024-09-16T14:53:25.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.527+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1367321224}) 2024-09-16T14:53:25.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.531+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.531+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3868162983}) 2024-09-16T14:53:25.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.531+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613537427}) 2024-09-16T14:53:25.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.531+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.531+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3868162983}) 2024-09-16T14:53:25.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.531+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=426108389}) 2024-09-16T14:53:25.535 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.531+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.531+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=585725902}) 2024-09-16T14:53:25.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.531+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4105637266}) 2024-09-16T14:53:25.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.531+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.531+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=585725902}) 2024-09-16T14:53:25.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.531+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3175340371}) 2024-09-16T14:53:25.537 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.535+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.535+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2789773873}) 2024-09-16T14:53:25.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.535+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1896899472}) 2024-09-16T14:53:25.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.535+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.535+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2789773873}) 2024-09-16T14:53:25.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.535+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3497546771}) 2024-09-16T14:53:25.539 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.535+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.539 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.535+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=397927267}) 2024-09-16T14:53:25.540 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.535+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=87093915}) 2024-09-16T14:53:25.540 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.535+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.540 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.535+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=397927267}) 2024-09-16T14:53:25.540 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.535+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2419280199}) 2024-09-16T14:53:25.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.539+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.539+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022360652}) 2024-09-16T14:53:25.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.539+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=647400110}) 2024-09-16T14:53:25.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.539+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.542 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.539+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022360652}) 2024-09-16T14:53:25.542 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.539+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1327453722}) 2024-09-16T14:53:25.543 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.539+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.543 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.539+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2018751856}) 2024-09-16T14:53:25.543 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.539+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3549462162}) 2024-09-16T14:53:25.543 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.539+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.544 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.539+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2018751856}) 2024-09-16T14:53:25.544 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.539+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3447389303}) 2024-09-16T14:53:25.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.543+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.543+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=778184060}) 2024-09-16T14:53:25.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.543+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3052732381}) 2024-09-16T14:53:25.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.543+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.546 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.543+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=778184060}) 2024-09-16T14:53:25.546 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.543+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2136518081}) 2024-09-16T14:53:25.547 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.543+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.547 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.543+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4092848097}) 2024-09-16T14:53:25.547 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.543+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244406459}) 2024-09-16T14:53:25.548 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.543+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.548 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.543+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4092848097}) 2024-09-16T14:53:25.548 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.543+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3410816332}) 2024-09-16T14:53:25.549 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.547+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.549 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.547+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3296901592}) 2024-09-16T14:53:25.549 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.547+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1731467892}) 2024-09-16T14:53:25.550 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.547+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.550 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.547+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3296901592}) 2024-09-16T14:53:25.550 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.547+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3154609504}) 2024-09-16T14:53:25.551 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.547+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.551 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.547+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=473712958}) 2024-09-16T14:53:25.551 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.547+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=466067709}) 2024-09-16T14:53:25.551 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.547+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.551 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.547+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=473712958}) 2024-09-16T14:53:25.551 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.547+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3272221709}) 2024-09-16T14:53:25.553 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.551+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.553 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.551+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4289651044}) 2024-09-16T14:53:25.554 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.551+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528752630}) 2024-09-16T14:53:25.554 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.551+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.554 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.551+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4289651044}) 2024-09-16T14:53:25.554 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.551+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=20990607}) 2024-09-16T14:53:25.555 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.551+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.555 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.551+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3945077455}) 2024-09-16T14:53:25.555 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.551+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605632345}) 2024-09-16T14:53:25.556 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.551+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.556 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.551+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3945077455}) 2024-09-16T14:53:25.556 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.551+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2387715116}) 2024-09-16T14:53:25.557 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.555+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.557 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.555+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2845623129}) 2024-09-16T14:53:25.558 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.555+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=322926954}) 2024-09-16T14:53:25.558 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.555+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.558 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.555+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2845623129}) 2024-09-16T14:53:25.558 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.555+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=578962824}) 2024-09-16T14:53:25.560 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.555+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.560 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.555+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3626500395}) 2024-09-16T14:53:25.560 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.555+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1977858170}) 2024-09-16T14:53:25.560 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.555+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.560 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.555+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3626500395}) 2024-09-16T14:53:25.561 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.555+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2404879013}) 2024-09-16T14:53:25.562 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.559+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.562 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.559+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2058129261}) 2024-09-16T14:53:25.562 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.559+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1471822191}) 2024-09-16T14:53:25.563 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.559+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.563 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.559+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2058129261}) 2024-09-16T14:53:25.563 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.559+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=350091528}) 2024-09-16T14:53:25.564 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.559+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.564 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.559+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742245404}) 2024-09-16T14:53:25.564 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.559+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3400530331}) 2024-09-16T14:53:25.565 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.559+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.565 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.559+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742245404}) 2024-09-16T14:53:25.565 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.559+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3500019888}) 2024-09-16T14:53:25.566 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.563+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.566 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.563+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1206152755}) 2024-09-16T14:53:25.567 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.563+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139646639}) 2024-09-16T14:53:25.567 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.563+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.567 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.563+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1206152755}) 2024-09-16T14:53:25.567 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.563+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1127561260}) 2024-09-16T14:53:25.568 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.563+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.569 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.563+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3845286575}) 2024-09-16T14:53:25.569 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.563+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3818057103}) 2024-09-16T14:53:25.569 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.563+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.569 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.563+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3845286575}) 2024-09-16T14:53:25.569 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.563+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1206649179}) 2024-09-16T14:53:25.570 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.567+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.571 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.567+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2505579634}) 2024-09-16T14:53:25.571 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.567+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=861188645}) 2024-09-16T14:53:25.571 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.567+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.571 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.567+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2505579634}) 2024-09-16T14:53:25.571 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.567+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2661258220}) 2024-09-16T14:53:25.573 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.571+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.573 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.571+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=22} crc {logm=1526937156,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2108501759,mon_config_key=953665057,monmap=3961923254,osd_metadata=1524398128,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=149303113}) 2024-09-16T14:53:25.573 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.571+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=25,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=7,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=29} crc {logm=1853413095,mdsmap=1531151339,mgr=3091803317,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=1730664617,mon_config_key=4201966829,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=2711864677}) 2024-09-16T14:53:25.573 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.571+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.574 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.571+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=22} crc {logm=1526937156,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2108501759,mon_config_key=953665057,monmap=3961923254,osd_metadata=1524398128,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=149303113}) 2024-09-16T14:53:25.574 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.571+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=27,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=11,monmap=3,osd_metadata=6,osd_snap=2,osdmap=23} crc {logm=2439507549,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2108501759,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_snap=3794734670,osdmap=579929745}) 2024-09-16T14:53:25.584 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.583+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.585 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.583+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=27} crc {osdmap=929807613}) 2024-09-16T14:53:25.585 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.583+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=28} crc {osdmap=3687386703}) 2024-09-16T14:53:25.585 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.583+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:53:25.585 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.583+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=27} crc {osdmap=929807613}) 2024-09-16T14:53:25.585 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:53:25.583+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=28} crc {osdmap=3324778024}) 2024-09-16T14:53:26.460 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:53:26.461 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:53:26.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3fff9a1640 10 monclient: get_monmap_and_config 2024-09-16T14:53:26.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3fff9a1640 10 monclient: build_initial_monmap 2024-09-16T14:53:26.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3fff9a1640 10 monclient: monmap: 2024-09-16T14:53:26.593 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:53:26.594830+0000 2024-09-16T14:53:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:53:26.594830+0000 2024-09-16T14:53:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:53:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:53:26.594 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:26.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3fff9a1640 1 Processor -- start 2024-09-16T14:53:26.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3fff9a1640 1 -- start start 2024-09-16T14:53:26.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3fff9a1640 10 monclient: init 2024-09-16T14:53:26.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3fff9a1640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:26.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3fff9a1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:53:26.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3fff9a1640 10 monclient(hunting): picked mon.noname-c con 0x7f3ff809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:26.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3fff9a1640 10 monclient(hunting): picked mon.noname-a con 0x7f3ff8132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:26.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3fff9a1640 10 monclient(hunting): picked mon.noname-b con 0x7f3ff8138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:26.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3fff9a1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff809d260 con 0x7f3ff8132650 2024-09-16T14:53:26.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3fff9a1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff8132a80 con 0x7f3ff809cb80 2024-09-16T14:53:26.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3fff9a1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff809cf60 con 0x7f3ff8138ff0 2024-09-16T14:53:26.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3fff9a1640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:26.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3fff9a1640 10 monclient(hunting): authenticate will time out at 1630.715208s 2024-09-16T14:53:26.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3ffdf3e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3ff8138ff0 0x7f3ff813b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:41062/0 (socket says 172.21.15.26:41062) 2024-09-16T14:53:26.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3ffdf3e640 1 -- 172.21.15.26:0/2053284450 learned_addr learned my addr 172.21.15.26:0/2053284450 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:26.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3feffff640 1 -- 172.21.15.26:0/2053284450 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216574914 0 0) 0x7f3ff809cf60 con 0x7f3ff8138ff0 2024-09-16T14:53:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3feffff640 10 monclient(hunting): _negotiate 2024-09-16T14:53:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3feffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3feffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3feffff640 10 monclient(hunting): my global_id is 234134 2024-09-16T14:53:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3feffff640 1 -- 172.21.15.26:0/2053284450 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe4003530 con 0x7f3ff8138ff0 2024-09-16T14:53:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3feffff640 1 -- 172.21.15.26:0/2053284450 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386192989 0 0) 0x7f3ff809d260 con 0x7f3ff8132650 2024-09-16T14:53:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3feffff640 10 monclient(hunting): _negotiate 2024-09-16T14:53:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3feffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3feffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3feffff640 10 monclient(hunting): my global_id is 194463 2024-09-16T14:53:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3feffff640 1 -- 172.21.15.26:0/2053284450 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ff809cf60 con 0x7f3ff8132650 2024-09-16T14:53:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.591+0000 7f3feffff640 1 -- 172.21.15.26:0/2053284450 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1303524496 0 0) 0x7f3fe4003530 con 0x7f3ff8138ff0 2024-09-16T14:53:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 1 -- 172.21.15.26:0/2053284450 >> v1:172.21.15.189:6789/0 conn(0x7f3ff809cb80 legacy=0x7f3ff80717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 1 -- 172.21.15.26:0/2053284450 >> v1:172.21.15.26:6789/0 conn(0x7f3ff8132650 legacy=0x7f3ff81378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 10 monclient: _finish_hunting 0 2024-09-16T14:53:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 1 monclient: found mon.noname-b 2024-09-16T14:53:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:53:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 1 -- 172.21.15.26:0/2053284450 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff809f2d0 con 0x7f3ff8138ff0 2024-09-16T14:53:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 10 monclient: _finish_auth 0 2024-09-16T14:53:26.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 10 monclient: _check_auth_tickets 2024-09-16T14:53:26.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:26.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 1 -- 172.21.15.26:0/2053284450 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ff0001660 con 0x7f3ff8138ff0 2024-09-16T14:53:26.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 5 monclient: authenticate success, global_id 234134 2024-09-16T14:53:26.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:26.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:26.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:53:26.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 10 monclient: dump: 2024-09-16T14:53:26.605 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:26.605 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:26.605 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.606 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.606 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:26.606 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:26.606 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:26.606 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:26.606 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:26.606 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:26.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:26.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 1 -- 172.21.15.26:0/2053284450 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ff0001880 con 0x7f3ff8138ff0 2024-09-16T14:53:26.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:26.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 1 -- 172.21.15.26:0/2053284450 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ff0001b90 con 0x7f3ff8138ff0 2024-09-16T14:53:26.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:26.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:26.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3feffff640 10 monclient: dump: 2024-09-16T14:53:26.607 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:26.607 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:26.607 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.607 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.607 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:26.608 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:26.608 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:26.608 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:26.608 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:26.608 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:26.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:26.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 4 set_mon_vals no callback set 2024-09-16T14:53:26.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 10 monclient: shutdown 2024-09-16T14:53:26.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:26.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 1 -- 172.21.15.26:0/2053284450 >> v1:172.21.15.26:6790/0 conn(0x7f3ff8138ff0 legacy=0x7f3ff813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:26.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 1 -- 172.21.15.26:0/2053284450 shutdown_connections 2024-09-16T14:53:26.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 1 -- 172.21.15.26:0/2053284450 >> 172.21.15.26:0/2053284450 conn(0x7f3ff809fb50 msgr2=0x7f3ff809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:26.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 1 -- 172.21.15.26:0/2053284450 shutdown_connections 2024-09-16T14:53:26.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 1 -- 172.21.15.26:0/2053284450 wait complete. 2024-09-16T14:53:26.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 10 monclient: build_initial_monmap 2024-09-16T14:53:26.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 10 monclient: monmap: 2024-09-16T14:53:26.609 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:26.609 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:26.609 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:26.609 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:26.609 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:26.609 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:26.610 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:26.610 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:26.610 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:26.610 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:26.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 1 Processor -- start 2024-09-16T14:53:26.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 1 -- start start 2024-09-16T14:53:26.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 10 monclient: init 2024-09-16T14:53:26.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:26.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:53:26.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 10 monclient(hunting): picked mon.noname-a con 0x7f3ff809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:26.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 10 monclient(hunting): picked mon.noname-c con 0x7f3ff8132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:26.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 10 monclient(hunting): picked mon.noname-b con 0x7f3ff8138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:26.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff813c790 con 0x7f3ff809cb80 2024-09-16T14:53:26.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff8131fe0 con 0x7f3ff8138ff0 2024-09-16T14:53:26.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff813c230 con 0x7f3ff8132650 2024-09-16T14:53:26.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:26.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 10 monclient(hunting): authenticate will time out at 1630.718361s 2024-09-16T14:53:26.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3ffd73d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3ff809cb80 0x7f3ff81c9d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51294/0 (socket says 172.21.15.26:51294) 2024-09-16T14:53:26.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3ffd73d640 1 -- 172.21.15.26:0/238508849 learned_addr learned my addr 172.21.15.26:0/238508849 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:26.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912096605 0 0) 0x7f3ff813c230 con 0x7f3ff8132650 2024-09-16T14:53:26.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:26.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:26.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:26.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): my global_id is 234137 2024-09-16T14:53:26.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fdc003530 con 0x7f3ff8132650 2024-09-16T14:53:26.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644206157 0 0) 0x7f3ff813c790 con 0x7f3ff809cb80 2024-09-16T14:53:26.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:26.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:26.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:26.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): my global_id is 194466 2024-09-16T14:53:26.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ff813c230 con 0x7f3ff809cb80 2024-09-16T14:53:26.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3342634668 0 0) 0x7f3fdc003530 con 0x7f3ff8132650 2024-09-16T14:53:26.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ff813c790 con 0x7f3ff8132650 2024-09-16T14:53:26.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 540857302 0 0) 0x7f3ff813c230 con 0x7f3ff809cb80 2024-09-16T14:53:26.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fdc003530 con 0x7f3ff809cb80 2024-09-16T14:53:26.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3fe80018e0 con 0x7f3ff8132650 2024-09-16T14:53:26.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:26.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:53:26.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): dump: 2024-09-16T14:53:26.613 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:26.613 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:26.613 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.613 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.613 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:26.613 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:26.614 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:26.614 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:26.614 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:26.614 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:26.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:26.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459178591 0 0) 0x7f3ff8131fe0 con 0x7f3ff8138ff0 2024-09-16T14:53:26.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:26.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:26.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:26.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): my global_id is 224218 2024-09-16T14:53:26.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ff813c230 con 0x7f3ff8138ff0 2024-09-16T14:53:26.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ff4001c60 con 0x7f3ff809cb80 2024-09-16T14:53:26.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:26.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:26.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient(hunting): dump: 2024-09-16T14:53:26.615 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:26.615 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:26.615 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.615 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.615 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:26.615 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:26.615 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:26.616 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:26.616 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:26.616 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:26.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1552510395 0 0) 0x7f3ff813c790 con 0x7f3ff8132650 2024-09-16T14:53:26.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 >> v1:172.21.15.189:6789/0 conn(0x7f3ff8138ff0 legacy=0x7f3ff81d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:26.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 >> v1:172.21.15.26:6789/0 conn(0x7f3ff809cb80 legacy=0x7f3ff81c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:26.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient: _finish_hunting 0 2024-09-16T14:53:26.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 monclient: found mon.c 2024-09-16T14:53:26.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:26.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:26.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff8134c00 con 0x7f3ff8132650 2024-09-16T14:53:26.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient: _finish_auth 0 2024-09-16T14:53:26.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 10 monclient: _check_auth_tickets 2024-09-16T14:53:26.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:26.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 5 monclient: authenticate success, global_id 234137 2024-09-16T14:53:26.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 10 monclient: _renew_subs 2024-09-16T14:53:26.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:26.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 1 -- 172.21.15.26:0/238508849 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff8134ee0 con 0x7f3ff8132650 2024-09-16T14:53:26.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 10 monclient: _renew_subs 2024-09-16T14:53:26.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:26.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.595+0000 7f3fff9a1640 1 -- 172.21.15.26:0/238508849 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff8133710 con 0x7f3ff8132650 2024-09-16T14:53:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.599+0000 7f3fff9a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.599+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fe8003620 con 0x7f3ff8132650 2024-09-16T14:53:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.599+0000 7f3fff9a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.599+0000 7f3fff9a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.599+0000 7f3fff9a1640 1 -- 172.21.15.26:0/238508849 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff809c960 con 0x7f3ff8132650 2024-09-16T14:53:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.599+0000 7f3fedffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.599+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3fe8003930 con 0x7f3ff8132650 2024-09-16T14:53:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.599+0000 7f3fedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.599+0000 7f3fedffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.599+0000 7f3fedffb640 10 monclient: dump: 2024-09-16T14:53:26.618 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:26.619 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:26.619 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.619 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.619 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:26.619 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:26.619 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:26.619 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:26.619 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:26.619 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:26.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.599+0000 7f3fef7fe640 4 set_mon_vals no callback set 2024-09-16T14:53:26.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.599+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3fe800e3a0 con 0x7f3ff8132650 2024-09-16T14:53:26.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.599+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f3fe8042b80 con 0x7f3ff8132650 2024-09-16T14:53:26.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.599+0000 7f3ffd73d640 10 monclient: get_auth_request con 0x7f3fdc036420 auth_method 0 2024-09-16T14:53:26.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.603+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3fe8013350 con 0x7f3ff8132650 2024-09-16T14:53:26.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.603+0000 7f3fedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:26.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.603+0000 7f3fedffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:26.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.819+0000 7f3fff9a1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:53:26.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.819+0000 7f3fff9a1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:53:26.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.819+0000 7f3fff9a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:26.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.819+0000 7f3fff9a1640 1 -- 172.21.15.26:0/238508849 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3ff806a530 con 0x7f3ff8132650 2024-09-16T14:53:26.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.819+0000 7f3fedffb640 1 -- 172.21.15.26:0/238508849 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 566979055) 0x7f3fe80119f0 con 0x7f3ff8132650 2024-09-16T14:53:26.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.819+0000 7f3fedffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:53:26.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.819+0000 7f3fedffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:26.824 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:53:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.823+0000 7f3fff9a1640 1 -- 172.21.15.26:0/238508849 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3fdc036420 legacy=0x7f3fdc0388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.823+0000 7f3fff9a1640 10 monclient: shutdown 2024-09-16T14:53:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.823+0000 7f3fff9a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.823+0000 7f3fff9a1640 1 -- 172.21.15.26:0/238508849 >> v1:172.21.15.26:6790/0 conn(0x7f3ff8132650 legacy=0x7f3ff81ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.823+0000 7f3fff9a1640 1 -- 172.21.15.26:0/238508849 shutdown_connections 2024-09-16T14:53:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.823+0000 7f3fff9a1640 1 -- 172.21.15.26:0/238508849 >> 172.21.15.26:0/238508849 conn(0x7f3ff809fb50 msgr2=0x7f3ff8070b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.823+0000 7f3fff9a1640 1 -- 172.21.15.26:0/238508849 shutdown_connections 2024-09-16T14:53:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.823+0000 7f3fff9a1640 1 -- 172.21.15.26:0/238508849 wait complete. 2024-09-16T14:53:26.840 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:53:26.841 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:53:26.841 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:53:26.841 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:53:26.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.955+0000 7f7420739640 10 monclient: get_monmap_and_config 2024-09-16T14:53:26.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.955+0000 7f7420739640 10 monclient: build_initial_monmap 2024-09-16T14:53:26.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.955+0000 7f7420739640 10 monclient: monmap: 2024-09-16T14:53:26.959 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:26.959 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:26.959 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:53:26.960292+0000 2024-09-16T14:53:26.959 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:53:26.960292+0000 2024-09-16T14:53:26.959 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:26.959 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:26.959 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:26.959 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:53:26.959 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:53:26.960 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:26.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.955+0000 7f7420739640 1 Processor -- start 2024-09-16T14:53:26.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.955+0000 7f7420739640 1 -- start start 2024-09-16T14:53:26.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.955+0000 7f7420739640 10 monclient: init 2024-09-16T14:53:26.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:26.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:53:26.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient(hunting): picked mon.noname-c con 0x7f741809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:26.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient(hunting): picked mon.noname-b con 0x7f7418132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:26.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient(hunting): picked mon.noname-a con 0x7f7418138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:26.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741809d260 con 0x7f7418138ff0 2024-09-16T14:53:26.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7418132a80 con 0x7f741809cb80 2024-09-16T14:53:26.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741809cf60 con 0x7f7418132650 2024-09-16T14:53:26.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:26.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient(hunting): authenticate will time out at 1631.080329s 2024-09-16T14:53:26.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741dcd4640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7418132650 0x7f74181378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:34754/0 (socket says 172.21.15.26:34754) 2024-09-16T14:53:26.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741dcd4640 1 -- 172.21.15.26:0/2213283139 learned_addr learned my addr 172.21.15.26:0/2213283139 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:26.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 1 -- 172.21.15.26:0/2213283139 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989269389 0 0) 0x7f741809d260 con 0x7f7418138ff0 2024-09-16T14:53:26.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient(hunting): _negotiate 2024-09-16T14:53:26.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:26.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:26.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient(hunting): my global_id is 194469 2024-09-16T14:53:26.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 1 -- 172.21.15.26:0/2213283139 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7404003530 con 0x7f7418138ff0 2024-09-16T14:53:26.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 1 -- 172.21.15.26:0/2213283139 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134639265 0 0) 0x7f741809cf60 con 0x7f7418132650 2024-09-16T14:53:26.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient(hunting): _negotiate 2024-09-16T14:53:26.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:26.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:26.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient(hunting): my global_id is 234140 2024-09-16T14:53:26.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 1 -- 172.21.15.26:0/2213283139 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f741809d260 con 0x7f7418132650 2024-09-16T14:53:26.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 1 -- 172.21.15.26:0/2213283139 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3156418636 0 0) 0x7f7404003530 con 0x7f7418138ff0 2024-09-16T14:53:26.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 1 -- 172.21.15.26:0/2213283139 >> v1:172.21.15.26:6790/0 conn(0x7f7418132650 legacy=0x7f74181378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:26.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 1 -- 172.21.15.26:0/2213283139 >> v1:172.21.15.189:6789/0 conn(0x7f741809cb80 legacy=0x7f74180717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:26.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient: _finish_hunting 0 2024-09-16T14:53:26.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 1 monclient: found mon.noname-a 2024-09-16T14:53:26.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:26.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:53:26.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 1 -- 172.21.15.26:0/2213283139 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741809f2d0 con 0x7f7418138ff0 2024-09-16T14:53:26.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient: _finish_auth 0 2024-09-16T14:53:26.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient: _check_auth_tickets 2024-09-16T14:53:26.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:26.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 1 -- 172.21.15.26:0/2213283139 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74100019a0 con 0x7f7418138ff0 2024-09-16T14:53:26.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 5 monclient: authenticate success, global_id 194469 2024-09-16T14:53:26.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:26.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:26.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:53:26.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient: dump: 2024-09-16T14:53:26.970 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:26.970 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:26.970 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.971 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:26.971 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:26.971 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:26.971 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:26.971 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:26.971 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:26.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:26.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 1 -- 172.21.15.26:0/2213283139 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7410001c10 con 0x7f7418138ff0 2024-09-16T14:53:26.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:26.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 1 -- 172.21.15.26:0/2213283139 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7410001fe0 con 0x7f7418138ff0 2024-09-16T14:53:26.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:26.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:26.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741d4d3640 10 monclient: dump: 2024-09-16T14:53:26.972 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:26.972 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:26.972 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.972 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.972 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:26.972 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:26.972 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:26.972 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:26.972 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:26.972 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:26.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:26.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 4 set_mon_vals no callback set 2024-09-16T14:53:26.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient: shutdown 2024-09-16T14:53:26.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:26.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 1 -- 172.21.15.26:0/2213283139 >> v1:172.21.15.26:6789/0 conn(0x7f7418138ff0 legacy=0x7f741813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:26.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 1 -- 172.21.15.26:0/2213283139 shutdown_connections 2024-09-16T14:53:26.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 1 -- 172.21.15.26:0/2213283139 >> 172.21.15.26:0/2213283139 conn(0x7f741809fb50 msgr2=0x7f741809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:26.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 1 -- 172.21.15.26:0/2213283139 shutdown_connections 2024-09-16T14:53:26.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 1 -- 172.21.15.26:0/2213283139 wait complete. 2024-09-16T14:53:26.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient: build_initial_monmap 2024-09-16T14:53:26.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient: monmap: 2024-09-16T14:53:26.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:26.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:26.974 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:26.974 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:26.974 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:26.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:26.974 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:26.974 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:26.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:26.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:26.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 1 Processor -- start 2024-09-16T14:53:26.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 1 -- start start 2024-09-16T14:53:26.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient: init 2024-09-16T14:53:26.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:26.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:53:26.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient(hunting): picked mon.noname-a con 0x7f741809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:26.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient(hunting): picked mon.noname-b con 0x7f7418132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:26.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient(hunting): picked mon.noname-c con 0x7f7418138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:26.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741806de40 con 0x7f741809cb80 2024-09-16T14:53:26.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741809b370 con 0x7f7418132650 2024-09-16T14:53:26.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741809a240 con 0x7f7418138ff0 2024-09-16T14:53:26.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:26.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f7420739640 10 monclient(hunting): authenticate will time out at 1631.083579s 2024-09-16T14:53:26.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741e4d5640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f741809cb80 0x7f741809a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50022/0 (socket says 172.21.15.26:50022) 2024-09-16T14:53:26.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741ecd6640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7418138ff0 0x7f74181d1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:34770/0 (socket says 172.21.15.26:34770) 2024-09-16T14:53:26.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f741e4d5640 1 -- 172.21.15.26:0/3684060855 learned_addr learned my addr 172.21.15.26:0/3684060855 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:26.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838296501 0 0) 0x7f741809a240 con 0x7f7418138ff0 2024-09-16T14:53:26.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f740effd640 10 monclient(hunting): _negotiate 2024-09-16T14:53:26.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f740effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:26.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f740effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:26.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f740effd640 10 monclient(hunting): my global_id is 234143 2024-09-16T14:53:26.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73fc003530 con 0x7f7418138ff0 2024-09-16T14:53:26.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874525233 0 0) 0x7f741806de40 con 0x7f741809cb80 2024-09-16T14:53:26.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f740effd640 10 monclient(hunting): _negotiate 2024-09-16T14:53:26.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f740effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:26.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f740effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:26.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.959+0000 7f740effd640 10 monclient(hunting): my global_id is 194472 2024-09-16T14:53:26.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f741809a240 con 0x7f741809cb80 2024-09-16T14:53:26.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2671331054 0 0) 0x7f73fc003530 con 0x7f7418138ff0 2024-09-16T14:53:26.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f741806de40 con 0x7f7418138ff0 2024-09-16T14:53:26.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 760971565 0 0) 0x7f741809a240 con 0x7f741809cb80 2024-09-16T14:53:26.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73fc003530 con 0x7f741809cb80 2024-09-16T14:53:26.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7410001f20 con 0x7f7418138ff0 2024-09-16T14:53:26.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:26.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:53:26.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient(hunting): dump: 2024-09-16T14:53:26.977 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:26.977 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:26.977 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.977 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.977 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:26.977 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:26.978 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:26.978 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:26.978 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:26.978 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:26.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:26.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7408001780 con 0x7f741809cb80 2024-09-16T14:53:26.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:26.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:26.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient(hunting): dump: 2024-09-16T14:53:26.978 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:26.978 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:26.978 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.978 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.979 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:26.979 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:26.979 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:26.979 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:26.979 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:26.979 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:26.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208827502 0 0) 0x7f741809b370 con 0x7f7418132650 2024-09-16T14:53:26.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient(hunting): _negotiate 2024-09-16T14:53:26.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:26.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:26.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient(hunting): my global_id is 224224 2024-09-16T14:53:26.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f741809a240 con 0x7f7418132650 2024-09-16T14:53:26.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1337142642 0 0) 0x7f741806de40 con 0x7f7418138ff0 2024-09-16T14:53:26.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 >> v1:172.21.15.189:6789/0 conn(0x7f7418132650 legacy=0x7f741809ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:26.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 >> v1:172.21.15.26:6789/0 conn(0x7f741809cb80 legacy=0x7f741809a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:26.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient: _finish_hunting 0 2024-09-16T14:53:26.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 monclient: found mon.c 2024-09-16T14:53:26.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:26.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:26.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74180a20f0 con 0x7f7418138ff0 2024-09-16T14:53:26.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient: _finish_auth 0 2024-09-16T14:53:26.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient: _check_auth_tickets 2024-09-16T14:53:26.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:26.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f7420739640 5 monclient: authenticate success, global_id 234143 2024-09-16T14:53:26.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f7420739640 10 monclient: _renew_subs 2024-09-16T14:53:26.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f7420739640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:26.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f7420739640 1 -- 172.21.15.26:0/3684060855 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7418071b70 con 0x7f7418138ff0 2024-09-16T14:53:26.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f7420739640 10 monclient: _renew_subs 2024-09-16T14:53:26.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f7420739640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:26.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f7420739640 1 -- 172.21.15.26:0/3684060855 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f741806d7f0 con 0x7f7418138ff0 2024-09-16T14:53:26.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7410003d80 con 0x7f7418138ff0 2024-09-16T14:53:26.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:26.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7410004110 con 0x7f7418138ff0 2024-09-16T14:53:26.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:26.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:26.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f741ccd2640 4 set_mon_vals no callback set 2024-09-16T14:53:26.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient: dump: 2024-09-16T14:53:26.982 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:26.982 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:26.982 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.982 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.982 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:26.982 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:26.982 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:26.982 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:26.982 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:26.982 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:26.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f7420739640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:26.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f7420739640 1 -- 172.21.15.26:0/3684060855 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7400006590 con 0x7f7400002830 2024-09-16T14:53:26.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7410004270 con 0x7f7418138ff0 2024-09-16T14:53:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f7410044120 con 0x7f7418138ff0 2024-09-16T14:53:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323834830 0 0) 0x7f7400006590 con 0x7f7400002830 2024-09-16T14:53:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient(hunting): _negotiate 2024-09-16T14:53:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient(hunting): my global_id is 194475 2024-09-16T14:53:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f741806de40 con 0x7f7400002830 2024-09-16T14:53:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1037424097 0 0) 0x7f741806de40 con 0x7f7400002830 2024-09-16T14:53:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7400006590 con 0x7f7400002830 2024-09-16T14:53:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7408003a10 con 0x7f7400002830 2024-09-16T14:53:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f741dcd4640 10 monclient: get_auth_request con 0x7f73fc036420 auth_method 0 2024-09-16T14:53:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:26.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient: dump: 2024-09-16T14:53:26.984 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:26.984 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:26.984 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.984 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.984 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:26.984 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:26.984 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:26.984 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:26.984 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:26.984 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:26.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2373097217 0 0) 0x7f7400006590 con 0x7f7400002830 2024-09-16T14:53:26.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f740000a0e0 con 0x7f7400002830 2024-09-16T14:53:26.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f74080057b0 con 0x7f7400002830 2024-09-16T14:53:26.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:26.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:26.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 >> v1:172.21.15.26:6789/0 conn(0x7f7400002830 legacy=0x7f7400004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:26.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.963+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 >> v1:172.21.15.26:6789/0 conn(0x7f7400002830 legacy=0x7f7400004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:26.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f7420739640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:53:26.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f7420739640 1 -- 172.21.15.26:0/3684060855 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7400009da0 con 0x7f74000090c0 2024-09-16T14:53:26.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460376804 0 0) 0x7f7400009da0 con 0x7f74000090c0 2024-09-16T14:53:26.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 10 monclient(hunting): _negotiate 2024-09-16T14:53:26.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:26.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:26.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 10 monclient(hunting): my global_id is 194478 2024-09-16T14:53:26.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7400006590 con 0x7f74000090c0 2024-09-16T14:53:26.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2164849824 0 0) 0x7f7400006590 con 0x7f74000090c0 2024-09-16T14:53:26.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7400009da0 con 0x7f74000090c0 2024-09-16T14:53:26.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74080040d0 con 0x7f74000090c0 2024-09-16T14:53:26.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:26.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:26.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 10 monclient: dump: 2024-09-16T14:53:26.987 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:26.987 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:26.988 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.988 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:26.988 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:26.988 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:26.988 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:26.988 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:26.988 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:26.988 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:26.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2687905047 0 0) 0x7f7400009da0 con 0x7f74000090c0 2024-09-16T14:53:26.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f740000c290 con 0x7f74000090c0 2024-09-16T14:53:26.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 116026135) 0x7f74080043e0 con 0x7f74000090c0 2024-09-16T14:53:26.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:53:26.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:26.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 >> v1:172.21.15.26:6789/0 conn(0x7f74000090c0 legacy=0x7f7400009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:26.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f740effd640 1 -- 172.21.15.26:0/3684060855 >> v1:172.21.15.26:6789/0 conn(0x7f74000090c0 legacy=0x7f7400009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:26.989 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:53:26.989 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:53:26.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:53:26.989 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:53:26.989 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 102, 2024-09-16T14:53:26.989 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:53:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:53:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:53:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:53:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:53:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:53:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:53:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:26.990 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:53:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:53:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:26.991 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:26.992 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:53:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:53:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:53:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:53:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:53:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:53:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:53:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:53:26.993 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:53:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:53:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:53:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:53:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:53:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:26.994 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:53:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:53:26.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:53:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:53:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:53:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:53:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:26.996 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:53:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:53:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:53:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:53:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:26.997 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:53:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:53:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:53:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:26.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:53:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:53:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:53:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:53:26.999 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:53:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:53:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:53:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:27.000 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:53:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:53:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:27.001 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:53:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:53:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:53:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.002 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:27.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.003 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:53:27.003 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:53:27.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f7420739640 1 -- 172.21.15.26:0/3684060855 >> v1:172.21.15.189:6812/3196161582 conn(0x7f73fc036420 legacy=0x7f73fc0388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f7420739640 10 monclient: shutdown 2024-09-16T14:53:27.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f7420739640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:27.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f7420739640 1 -- 172.21.15.26:0/3684060855 >> v1:172.21.15.26:6790/0 conn(0x7f7418138ff0 legacy=0x7f74181d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f741ecd6640 1 -- 172.21.15.26:0/3684060855 reap_dead start 2024-09-16T14:53:27.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f741ecd6640 1 -- 172.21.15.26:0/3684060855 reap_dead start 2024-09-16T14:53:27.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f7420739640 1 -- 172.21.15.26:0/3684060855 shutdown_connections 2024-09-16T14:53:27.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f7420739640 1 -- 172.21.15.26:0/3684060855 >> 172.21.15.26:0/3684060855 conn(0x7f741809fb50 msgr2=0x7f7418131770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:27.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f7420739640 1 -- 172.21.15.26:0/3684060855 shutdown_connections 2024-09-16T14:53:27.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:26.971+0000 7f7420739640 1 -- 172.21.15.26:0/3684060855 wait complete. 2024-09-16T14:53:27.006 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:53:27.006 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:53:27.006 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:53:27.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.127+0000 7f30d88f7640 10 monclient: get_monmap_and_config 2024-09-16T14:53:27.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.127+0000 7f30d88f7640 10 monclient: build_initial_monmap 2024-09-16T14:53:27.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.127+0000 7f30d88f7640 10 monclient: monmap: 2024-09-16T14:53:27.131 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:27.131 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.131 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:53:27.132415+0000 2024-09-16T14:53:27.132 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:53:27.132415+0000 2024-09-16T14:53:27.132 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:27.132 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.132 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:27.132 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:53:27.132 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:53:27.132 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.127+0000 7f30d88f7640 1 Processor -- start 2024-09-16T14:53:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.127+0000 7f30d88f7640 1 -- start start 2024-09-16T14:53:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.127+0000 7f30d88f7640 10 monclient: init 2024-09-16T14:53:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:53:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient(hunting): picked mon.noname-b con 0x7f30d409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient(hunting): picked mon.noname-c con 0x7f30d4132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient(hunting): picked mon.noname-a con 0x7f30d4138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d409d260 con 0x7f30d4138ff0 2024-09-16T14:53:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d4132a80 con 0x7f30d4132650 2024-09-16T14:53:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d409cf60 con 0x7f30d409cb80 2024-09-16T14:53:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient(hunting): authenticate will time out at 1631.252510s 2024-09-16T14:53:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d259c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f30d409cb80 0x7f30d40717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:34778/0 (socket says 172.21.15.26:34778) 2024-09-16T14:53:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d259c640 1 -- 172.21.15.26:0/213425910 learned_addr learned my addr 172.21.15.26:0/213425910 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d2d9d640 1 --1- 172.21.15.26:0/213425910 >> v1:172.21.15.26:6789/0 conn(0x7f30d4138ff0 0x7f30d413b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50050/0 (socket says 172.21.15.26:50050) 2024-09-16T14:53:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 1 -- 172.21.15.26:0/213425910 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769108312 0 0) 0x7f30d409cf60 con 0x7f30d409cb80 2024-09-16T14:53:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient(hunting): _negotiate 2024-09-16T14:53:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient(hunting): my global_id is 234146 2024-09-16T14:53:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 1 -- 172.21.15.26:0/213425910 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30bc003530 con 0x7f30d409cb80 2024-09-16T14:53:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 1 -- 172.21.15.26:0/213425910 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998869204 0 0) 0x7f30d409d260 con 0x7f30d4138ff0 2024-09-16T14:53:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient(hunting): _negotiate 2024-09-16T14:53:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient(hunting): my global_id is 194481 2024-09-16T14:53:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 1 -- 172.21.15.26:0/213425910 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d409cf60 con 0x7f30d4138ff0 2024-09-16T14:53:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 1 -- 172.21.15.26:0/213425910 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3693379269 0 0) 0x7f30bc003530 con 0x7f30d409cb80 2024-09-16T14:53:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 1 -- 172.21.15.26:0/213425910 >> v1:172.21.15.189:6789/0 conn(0x7f30d4132650 legacy=0x7f30d41378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 1 -- 172.21.15.26:0/213425910 >> v1:172.21.15.26:6789/0 conn(0x7f30d4138ff0 legacy=0x7f30d413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient: _finish_hunting 0 2024-09-16T14:53:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 1 monclient: found mon.noname-b 2024-09-16T14:53:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:53:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 1 -- 172.21.15.26:0/213425910 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d409f2d0 con 0x7f30d409cb80 2024-09-16T14:53:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient: _finish_auth 0 2024-09-16T14:53:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient: _check_auth_tickets 2024-09-16T14:53:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 1 -- 172.21.15.26:0/213425910 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f30c4001660 con 0x7f30d409cb80 2024-09-16T14:53:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:53:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient: dump: 2024-09-16T14:53:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.143 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 5 monclient: authenticate success, global_id 234146 2024-09-16T14:53:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 1 -- 172.21.15.26:0/213425910 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30c4001840 con 0x7f30d409cb80 2024-09-16T14:53:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 4 set_mon_vals no callback set 2024-09-16T14:53:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 1 -- 172.21.15.26:0/213425910 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f30c4001c10 con 0x7f30d409cb80 2024-09-16T14:53:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d159a640 10 monclient: dump: 2024-09-16T14:53:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.145 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient: shutdown 2024-09-16T14:53:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 1 -- 172.21.15.26:0/213425910 >> v1:172.21.15.26:6790/0 conn(0x7f30d409cb80 legacy=0x7f30d40717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 1 -- 172.21.15.26:0/213425910 shutdown_connections 2024-09-16T14:53:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 1 -- 172.21.15.26:0/213425910 >> 172.21.15.26:0/213425910 conn(0x7f30d409fb50 msgr2=0x7f30d409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 1 -- 172.21.15.26:0/213425910 shutdown_connections 2024-09-16T14:53:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 1 -- 172.21.15.26:0/213425910 wait complete. 2024-09-16T14:53:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient: build_initial_monmap 2024-09-16T14:53:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient: monmap: 2024-09-16T14:53:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:27.146 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 1 Processor -- start 2024-09-16T14:53:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 1 -- start start 2024-09-16T14:53:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient: init 2024-09-16T14:53:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:53:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient(hunting): picked mon.noname-b con 0x7f30d409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient(hunting): picked mon.noname-a con 0x7f30d4132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient(hunting): picked mon.noname-c con 0x7f30d4138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d4138150 con 0x7f30d4132650 2024-09-16T14:53:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d406e450 con 0x7f30d409cb80 2024-09-16T14:53:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d4071590 con 0x7f30d4138ff0 2024-09-16T14:53:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d88f7640 10 monclient(hunting): authenticate will time out at 1631.255851s 2024-09-16T14:53:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d2d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f30d4138ff0 0x7f30d41cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:34788/0 (socket says 172.21.15.26:34788) 2024-09-16T14:53:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.131+0000 7f30d2d9d640 1 -- 172.21.15.26:0/4201951699 learned_addr learned my addr 172.21.15.26:0/4201951699 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581111742 0 0) 0x7f30d4071590 con 0x7f30d4138ff0 2024-09-16T14:53:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:53:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): my global_id is 234149 2024-09-16T14:53:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b4003530 con 0x7f30d4138ff0 2024-09-16T14:53:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192966276 0 0) 0x7f30d4132a80 con 0x7f30d4132650 2024-09-16T14:53:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:53:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): my global_id is 194484 2024-09-16T14:53:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d4071590 con 0x7f30d4132650 2024-09-16T14:53:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602316621 0 0) 0x7f30d406e450 con 0x7f30d409cb80 2024-09-16T14:53:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:53:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): my global_id is 224230 2024-09-16T14:53:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d4132a80 con 0x7f30d409cb80 2024-09-16T14:53:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4159764648 0 0) 0x7f30b4003530 con 0x7f30d4138ff0 2024-09-16T14:53:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30d406e450 con 0x7f30d4138ff0 2024-09-16T14:53:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3433010548 0 0) 0x7f30d4071590 con 0x7f30d4132650 2024-09-16T14:53:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30b4003530 con 0x7f30d4132650 2024-09-16T14:53:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f30cc001be0 con 0x7f30d4138ff0 2024-09-16T14:53:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:53:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): dump: 2024-09-16T14:53:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.151 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f30c80016a0 con 0x7f30d4132650 2024-09-16T14:53:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): dump: 2024-09-16T14:53:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.153 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.153 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1932213341 0 0) 0x7f30b4003530 con 0x7f30d4132650 2024-09-16T14:53:27.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 >> v1:172.21.15.26:6790/0 conn(0x7f30d4138ff0 legacy=0x7f30d41cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 >> v1:172.21.15.189:6789/0 conn(0x7f30d409cb80 legacy=0x7f30d406d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:53:27.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 monclient: found mon.a 2024-09-16T14:53:27.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:27.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:27.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d413c400 con 0x7f30d4132650 2024-09-16T14:53:27.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:53:27.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:53:27.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:27.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30d88f7640 5 monclient: authenticate success, global_id 194484 2024-09-16T14:53:27.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30d88f7640 10 monclient: _renew_subs 2024-09-16T14:53:27.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30d88f7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:27.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30d88f7640 1 -- 172.21.15.26:0/4201951699 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d4070fd0 con 0x7f30d4132650 2024-09-16T14:53:27.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30d88f7640 10 monclient: _renew_subs 2024-09-16T14:53:27.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30d88f7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:53:27.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30d88f7640 1 -- 172.21.15.26:0/4201951699 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d406fb80 con 0x7f30d4132650 2024-09-16T14:53:27.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30c8001c10 con 0x7f30d4132650 2024-09-16T14:53:27.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:27.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f30c80037c0 con 0x7f30d4132650 2024-09-16T14:53:27.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:27.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30d0d99640 4 set_mon_vals no callback set 2024-09-16T14:53:27.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient: dump: 2024-09-16T14:53:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.155 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30d88f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30d88f7640 1 -- 172.21.15.26:0/4201951699 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f30b8006590 con 0x7f30b8002830 2024-09-16T14:53:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f30c800e230 con 0x7f30d4132650 2024-09-16T14:53:27.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f30c80427d0 con 0x7f30d4132650 2024-09-16T14:53:27.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804455244 0 0) 0x7f30b8006590 con 0x7f30b8002830 2024-09-16T14:53:27.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:53:27.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:27.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:27.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient(hunting): my global_id is 234152 2024-09-16T14:53:27.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b4003530 con 0x7f30b8002830 2024-09-16T14:53:27.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1701236922 0 0) 0x7f30b4003530 con 0x7f30b8002830 2024-09-16T14:53:27.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30b8006590 con 0x7f30b8002830 2024-09-16T14:53:27.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f30c40016e0 con 0x7f30b8002830 2024-09-16T14:53:27.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:27.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient: dump: 2024-09-16T14:53:27.157 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.158 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30d2d9d640 10 monclient: get_auth_request con 0x7f30b40363a0 auth_method 0 2024-09-16T14:53:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3601031581 0 0) 0x7f30b8006590 con 0x7f30b8002830 2024-09-16T14:53:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30b800a0e0 con 0x7f30b8002830 2024-09-16T14:53:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f30c4001c00 con 0x7f30b8002830 2024-09-16T14:53:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:27.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:27.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 >> v1:172.21.15.26:6790/0 conn(0x7f30b8002830 legacy=0x7f30b8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.135+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 >> v1:172.21.15.26:6790/0 conn(0x7f30b8002830 legacy=0x7f30b8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:27.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30d88f7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:53:27.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30d88f7640 1 -- 172.21.15.26:0/4201951699 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f30b8009d80 con 0x7f30b80090a0 2024-09-16T14:53:27.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309720338 0 0) 0x7f30b8009d80 con 0x7f30b80090a0 2024-09-16T14:53:27.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:53:27.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:27.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:27.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 10 monclient(hunting): my global_id is 234155 2024-09-16T14:53:27.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b8006590 con 0x7f30b80090a0 2024-09-16T14:53:27.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2795478211 0 0) 0x7f30b8006590 con 0x7f30b80090a0 2024-09-16T14:53:27.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30b8009d80 con 0x7f30b80090a0 2024-09-16T14:53:27.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f30c4004b80 con 0x7f30b80090a0 2024-09-16T14:53:27.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:27.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 10 monclient: dump: 2024-09-16T14:53:27.162 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.162 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.162 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.162 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.163 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.163 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.163 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.163 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.163 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.163 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2986339914 0 0) 0x7f30b8009d80 con 0x7f30b80090a0 2024-09-16T14:53:27.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f30b800c290 con 0x7f30b80090a0 2024-09-16T14:53:27.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 2965992932) 0x7f30c4005f40 con 0x7f30b80090a0 2024-09-16T14:53:27.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:53:27.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:27.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 >> v1:172.21.15.26:6790/0 conn(0x7f30b80090a0 legacy=0x7f30b8009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30c2ffd640 1 -- 172.21.15.26:0/4201951699 >> v1:172.21.15.26:6790/0 conn(0x7f30b80090a0 legacy=0x7f30b8009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:27.164 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:53:27.164 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:53:27.164 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:53:27.164 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:53:27.164 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 102, 2024-09-16T14:53:27.164 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:53:27.164 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:53:27.164 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:53:27.164 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:53:27.165 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:27.165 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:53:27.165 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:27.165 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:53:27.165 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:53:27.165 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:27.165 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:27.165 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:27.165 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:27.165 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:27.165 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:27.165 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:27.165 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:27.166 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:27.166 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:27.166 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:27.166 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:53:27.166 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:53:27.166 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:27.166 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:27.166 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:27.166 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:27.166 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:27.166 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:27.166 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:27.166 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:27.166 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:27.167 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:27.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:27.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.167 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:53:27.167 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:53:27.167 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:53:27.167 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:53:27.167 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:53:27.167 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:53:27.167 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:27.167 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:27.167 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:53:27.168 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:53:27.168 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:53:27.168 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:53:27.168 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:53:27.168 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:53:27.168 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:53:27.168 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:27.168 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:53:27.168 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:27.168 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:27.168 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:27.168 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:27.168 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:27.169 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:27.169 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:27.169 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:27.169 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:27.169 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:27.169 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:27.169 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:53:27.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.169 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:53:27.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.169 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:53:27.169 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:53:27.169 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:27.169 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:27.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.170 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:27.170 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:53:27.170 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:27.170 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.170 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:27.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.170 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:53:27.170 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:53:27.171 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:27.171 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:27.171 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:27.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.171 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:53:27.172 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:53:27.172 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:27.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:27.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.172 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:27.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:53:27.172 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:27.172 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.172 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:27.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:53:27.172 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:53:27.172 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:27.173 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:27.173 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:27.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.173 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:53:27.173 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:53:27.173 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:27.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:27.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.173 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:27.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:53:27.173 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:27.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.173 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:27.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:53:27.174 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:53:27.174 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:27.174 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:27.174 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:27.174 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.174 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:27.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.174 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:53:27.174 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:53:27.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.174 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:27.174 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:27.175 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:27.175 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.175 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:27.175 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:53:27.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.175 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:27.175 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:27.175 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:27.175 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.175 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:27.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.175 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:53:27.175 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:53:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30d88f7640 1 -- 172.21.15.26:0/4201951699 >> v1:172.21.15.189:6812/3196161582 conn(0x7f30b40363a0 legacy=0x7f30b4038860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30d88f7640 10 monclient: shutdown 2024-09-16T14:53:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30d88f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30d88f7640 1 -- 172.21.15.26:0/4201951699 >> v1:172.21.15.26:6789/0 conn(0x7f30d4132650 legacy=0x7f30d406dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30d2d9d640 1 -- 172.21.15.26:0/4201951699 reap_dead start 2024-09-16T14:53:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30d2d9d640 1 -- 172.21.15.26:0/4201951699 reap_dead start 2024-09-16T14:53:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30d88f7640 1 -- 172.21.15.26:0/4201951699 shutdown_connections 2024-09-16T14:53:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30d88f7640 1 -- 172.21.15.26:0/4201951699 >> 172.21.15.26:0/4201951699 conn(0x7f30d409fb50 msgr2=0x7f30d4131ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30d88f7640 1 -- 172.21.15.26:0/4201951699 shutdown_connections 2024-09-16T14:53:27.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.143+0000 7f30d88f7640 1 -- 172.21.15.26:0/4201951699 wait complete. 2024-09-16T14:53:27.178 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:53:27.178 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:53:27.178 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:53:27.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 10 monclient: get_monmap_and_config 2024-09-16T14:53:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 10 monclient: build_initial_monmap 2024-09-16T14:53:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 10 monclient: monmap: 2024-09-16T14:53:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:53:27.290020+0000 2024-09-16T14:53:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:53:27.290020+0000 2024-09-16T14:53:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:27.290 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:53:27.290 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:53:27.290 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 1 Processor -- start 2024-09-16T14:53:27.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 1 -- start start 2024-09-16T14:53:27.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 10 monclient: init 2024-09-16T14:53:27.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:27.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:53:27.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 10 monclient(hunting): picked mon.noname-c con 0x7fa16809a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:27.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 10 monclient(hunting): picked mon.noname-a con 0x7fa16809d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:27.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 10 monclient(hunting): picked mon.noname-b con 0x7fa16809b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:27.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16809dc60 con 0x7fa16809d1b0 2024-09-16T14:53:27.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa168099ae0 con 0x7fa16809a550 2024-09-16T14:53:27.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16809b8f0 con 0x7fa16809b040 2024-09-16T14:53:27.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:27.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 10 monclient(hunting): authenticate will time out at 1631.409925s 2024-09-16T14:53:27.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa166d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa16809b040 0x7fa16813a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:34808/0 (socket says 172.21.15.26:34808) 2024-09-16T14:53:27.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa15dd9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa16809d1b0 0x7fa168136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50056/0 (socket says 172.21.15.26:50056) 2024-09-16T14:53:27.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa166d9d640 1 -- 172.21.15.26:0/2690877166 learned_addr learned my addr 172.21.15.26:0/2690877166 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:27.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 1 -- 172.21.15.26:0/2690877166 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960894294 0 0) 0x7fa16809dc60 con 0x7fa16809d1b0 2024-09-16T14:53:27.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:53:27.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:27.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:27.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient(hunting): my global_id is 194487 2024-09-16T14:53:27.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 1 -- 172.21.15.26:0/2690877166 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa154003530 con 0x7fa16809d1b0 2024-09-16T14:53:27.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 1 -- 172.21.15.26:0/2690877166 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819427388 0 0) 0x7fa16809b8f0 con 0x7fa16809b040 2024-09-16T14:53:27.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:53:27.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:27.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:27.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient(hunting): my global_id is 234158 2024-09-16T14:53:27.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 1 -- 172.21.15.26:0/2690877166 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa16809dc60 con 0x7fa16809b040 2024-09-16T14:53:27.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 1 -- 172.21.15.26:0/2690877166 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2061125653 0 0) 0x7fa154003530 con 0x7fa16809d1b0 2024-09-16T14:53:27.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 1 -- 172.21.15.26:0/2690877166 >> v1:172.21.15.26:6790/0 conn(0x7fa16809b040 legacy=0x7fa16813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 1 -- 172.21.15.26:0/2690877166 >> v1:172.21.15.189:6789/0 conn(0x7fa16809a550 legacy=0x7fa16809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:53:27.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 1 monclient: found mon.noname-a 2024-09-16T14:53:27.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:27.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:53:27.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 1 -- 172.21.15.26:0/2690877166 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16809bbd0 con 0x7fa16809d1b0 2024-09-16T14:53:27.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient: _finish_auth 0 2024-09-16T14:53:27.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:53:27.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:27.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 1 -- 172.21.15.26:0/2690877166 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa150001620 con 0x7fa16809d1b0 2024-09-16T14:53:27.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:53:27.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient: dump: 2024-09-16T14:53:27.300 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.300 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.300 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.300 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.300 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.300 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.300 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.301 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.301 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.301 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 5 monclient: authenticate success, global_id 194487 2024-09-16T14:53:27.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:27.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 1 -- 172.21.15.26:0/2690877166 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1500018c0 con 0x7fa16809d1b0 2024-09-16T14:53:27.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:27.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 1 -- 172.21.15.26:0/2690877166 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa150001c50 con 0x7fa16809d1b0 2024-09-16T14:53:27.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:27.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 4 set_mon_vals no callback set 2024-09-16T14:53:27.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:27.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa165d9b640 10 monclient: dump: 2024-09-16T14:53:27.301 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.302 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.302 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.302 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.302 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.302 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.302 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.302 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.302 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.302 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 10 monclient: shutdown 2024-09-16T14:53:27.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:27.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 1 -- 172.21.15.26:0/2690877166 >> v1:172.21.15.26:6789/0 conn(0x7fa16809d1b0 legacy=0x7fa168136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 1 -- 172.21.15.26:0/2690877166 shutdown_connections 2024-09-16T14:53:27.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 1 -- 172.21.15.26:0/2690877166 >> 172.21.15.26:0/2690877166 conn(0x7fa16809fb50 msgr2=0x7fa16809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:27.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 1 -- 172.21.15.26:0/2690877166 shutdown_connections 2024-09-16T14:53:27.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 1 -- 172.21.15.26:0/2690877166 wait complete. 2024-09-16T14:53:27.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 10 monclient: build_initial_monmap 2024-09-16T14:53:27.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.287+0000 7fa16cc00640 10 monclient: monmap: 2024-09-16T14:53:27.303 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:27.303 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:27.303 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:27.303 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:27.303 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:27.303 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:27.303 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:27.304 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:27.304 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:27.304 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 1 Processor -- start 2024-09-16T14:53:27.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 1 -- start start 2024-09-16T14:53:27.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 10 monclient: init 2024-09-16T14:53:27.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:27.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:53:27.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 10 monclient(hunting): picked mon.noname-a con 0x7fa16809a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:27.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 10 monclient(hunting): picked mon.noname-c con 0x7fa16809b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:27.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 10 monclient(hunting): picked mon.noname-b con 0x7fa16809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:27.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1681a93f0 con 0x7fa16809a550 2024-09-16T14:53:27.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1680a27f0 con 0x7fa16809d1b0 2024-09-16T14:53:27.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1681b4440 con 0x7fa16809b040 2024-09-16T14:53:27.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:27.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 10 monclient(hunting): authenticate will time out at 1631.412791s 2024-09-16T14:53:27.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa16809a550 0x7fa1681a5830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50068/0 (socket says 172.21.15.26:50068) 2024-09-16T14:53:27.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16659c640 1 -- 172.21.15.26:0/3035812233 learned_addr learned my addr 172.21.15.26:0/3035812233 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:27.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049210770 0 0) 0x7fa1681b4440 con 0x7fa16809b040 2024-09-16T14:53:27.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:53:27.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:27.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:27.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): my global_id is 234161 2024-09-16T14:53:27.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa148003530 con 0x7fa16809b040 2024-09-16T14:53:27.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539932710 0 0) 0x7fa1681a93f0 con 0x7fa16809a550 2024-09-16T14:53:27.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:53:27.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:27.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:27.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): my global_id is 194490 2024-09-16T14:53:27.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1681b4440 con 0x7fa16809a550 2024-09-16T14:53:27.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2683441635 0 0) 0x7fa148003530 con 0x7fa16809b040 2024-09-16T14:53:27.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1681a93f0 con 0x7fa16809b040 2024-09-16T14:53:27.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa150001bd0 con 0x7fa16809b040 2024-09-16T14:53:27.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:53:27.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): dump: 2024-09-16T14:53:27.306 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.306 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.307 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.307 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.307 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.307 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.307 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.307 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.307 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.307 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:27.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1229240554 0 0) 0x7fa1681b4440 con 0x7fa16809a550 2024-09-16T14:53:27.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa148003530 con 0x7fa16809a550 2024-09-16T14:53:27.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa158001980 con 0x7fa16809a550 2024-09-16T14:53:27.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:27.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): dump: 2024-09-16T14:53:27.308 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.308 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.308 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.308 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.308 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.308 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.308 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.308 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.308 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.308 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875390944 0 0) 0x7fa1680a27f0 con 0x7fa16809d1b0 2024-09-16T14:53:27.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:53:27.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:27.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:27.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): my global_id is 224236 2024-09-16T14:53:27.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1681b4440 con 0x7fa16809d1b0 2024-09-16T14:53:27.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 170253378 0 0) 0x7fa1681a93f0 con 0x7fa16809b040 2024-09-16T14:53:27.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 >> v1:172.21.15.189:6789/0 conn(0x7fa16809d1b0 legacy=0x7fa1681d1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 >> v1:172.21.15.26:6789/0 conn(0x7fa16809a550 legacy=0x7fa1681a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:53:27.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 monclient: found mon.c 2024-09-16T14:53:27.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:27.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:27.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1681b5600 con 0x7fa16809b040 2024-09-16T14:53:27.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:53:27.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:53:27.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:27.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 5 monclient: authenticate success, global_id 234161 2024-09-16T14:53:27.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 10 monclient: _renew_subs 2024-09-16T14:53:27.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:27.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 1 -- 172.21.15.26:0/3035812233 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1681a6650 con 0x7fa16809b040 2024-09-16T14:53:27.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 10 monclient: _renew_subs 2024-09-16T14:53:27.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:27.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 1 -- 172.21.15.26:0/3035812233 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1681a96d0 con 0x7fa16809b040 2024-09-16T14:53:27.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1500010f0 con 0x7fa16809b040 2024-09-16T14:53:27.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:27.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa150004ce0 con 0x7fa16809b040 2024-09-16T14:53:27.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:27.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16559a640 4 set_mon_vals no callback set 2024-09-16T14:53:27.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient: dump: 2024-09-16T14:53:27.311 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.311 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.311 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.311 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.311 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.311 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.311 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.311 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.311 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.311 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:27.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa16cc00640 1 -- 172.21.15.26:0/3035812233 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa14c006590 con 0x7fa14c002830 2024-09-16T14:53:27.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa15000f750 con 0x7fa16809b040 2024-09-16T14:53:27.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fa150043d10 con 0x7fa16809b040 2024-09-16T14:53:27.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa166d9d640 10 monclient: get_auth_request con 0x7fa148036420 auth_method 0 2024-09-16T14:53:27.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146758132 0 0) 0x7fa14c006590 con 0x7fa14c002830 2024-09-16T14:53:27.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:53:27.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:27.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:27.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient(hunting): my global_id is 224239 2024-09-16T14:53:27.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1681a93f0 con 0x7fa14c002830 2024-09-16T14:53:27.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2511868405 0 0) 0x7fa1681a93f0 con 0x7fa14c002830 2024-09-16T14:53:27.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa14c006590 con 0x7fa14c002830 2024-09-16T14:53:27.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa158003b40 con 0x7fa14c002830 2024-09-16T14:53:27.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:53:27.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.291+0000 7fa15f7fe640 10 monclient: dump: 2024-09-16T14:53:27.313 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.313 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.313 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.313 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.313 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.313 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.313 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.313 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.313 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.313 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.295+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2041449895 0 0) 0x7fa14c006590 con 0x7fa14c002830 2024-09-16T14:53:27.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.295+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa14c00a0e0 con 0x7fa14c002830 2024-09-16T14:53:27.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.295+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa158003e90 con 0x7fa14c002830 2024-09-16T14:53:27.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.295+0000 7fa15f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:27.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.295+0000 7fa15f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:27.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.295+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 >> v1:172.21.15.189:6789/0 conn(0x7fa14c002830 legacy=0x7fa14c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.295+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 >> v1:172.21.15.189:6789/0 conn(0x7fa14c002830 legacy=0x7fa14c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:27.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.299+0000 7fa16cc00640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:53:27.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.299+0000 7fa16cc00640 1 -- 172.21.15.26:0/3035812233 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa14c009d60 con 0x7fa14c0090a0 2024-09-16T14:53:27.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.299+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078527850 0 0) 0x7fa14c009d60 con 0x7fa14c0090a0 2024-09-16T14:53:27.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.299+0000 7fa15f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:53:27.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.299+0000 7fa15f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:27.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.299+0000 7fa15f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:27.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.299+0000 7fa15f7fe640 10 monclient(hunting): my global_id is 224242 2024-09-16T14:53:27.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.299+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa14c006590 con 0x7fa14c0090a0 2024-09-16T14:53:27.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 662534870 0 0) 0x7fa14c006590 con 0x7fa14c0090a0 2024-09-16T14:53:27.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa14c009d60 con 0x7fa14c0090a0 2024-09-16T14:53:27.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa158004450 con 0x7fa14c0090a0 2024-09-16T14:53:27.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa15f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa15f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:53:27.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa15f7fe640 10 monclient: dump: 2024-09-16T14:53:27.316 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.316 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.316 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.316 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.316 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.316 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.317 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.317 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.317 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.317 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 917943581 0 0) 0x7fa14c009d60 con 0x7fa14c0090a0 2024-09-16T14:53:27.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa14c00c290 con 0x7fa14c0090a0 2024-09-16T14:53:27.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 2648980463) 0x7fa1580058e0 con 0x7fa14c0090a0 2024-09-16T14:53:27.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa15f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:53:27.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa15f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:27.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 >> v1:172.21.15.189:6789/0 conn(0x7fa14c0090a0 legacy=0x7fa14c009510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa15f7fe640 1 -- 172.21.15.26:0/3035812233 >> v1:172.21.15.189:6789/0 conn(0x7fa14c0090a0 legacy=0x7fa14c009510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:27.318 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:53:27.318 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:53:27.318 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:53:27.318 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:53:27.318 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 102, 2024-09-16T14:53:27.318 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:53:27.318 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:53:27.318 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:53:27.318 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:53:27.318 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:27.318 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:53:27.318 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:27.319 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:53:27.319 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:53:27.319 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:27.319 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:27.319 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:27.319 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:27.319 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:27.319 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:27.319 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:27.319 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:27.319 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:27.319 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:27.319 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:27.320 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:53:27.320 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:53:27.320 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:27.320 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:27.320 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:27.320 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:27.320 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:27.320 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:27.320 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:27.320 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:27.320 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:27.320 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:27.320 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:27.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.321 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:53:27.321 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:53:27.321 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:53:27.321 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:53:27.321 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:53:27.321 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:53:27.321 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:27.321 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:27.321 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:53:27.321 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:53:27.321 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:53:27.321 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:53:27.321 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:53:27.322 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:53:27.322 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:53:27.322 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:27.322 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:53:27.322 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:27.322 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:27.322 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:27.322 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:27.322 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:27.322 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:27.322 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:27.322 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:27.322 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:27.323 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:27.323 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:27.323 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:53:27.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.323 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:53:27.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.323 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:53:27.323 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:53:27.323 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:27.323 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:27.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.323 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:27.323 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:53:27.323 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:27.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.324 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:27.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.324 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:53:27.324 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:53:27.324 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:27.324 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:27.324 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:27.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.324 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:53:27.324 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:53:27.324 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:27.325 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:27.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.325 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:27.325 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:53:27.325 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:27.325 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.325 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:27.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.325 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:53:27.325 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:53:27.325 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:27.325 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:27.325 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:27.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.326 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:53:27.326 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:53:27.326 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:27.326 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:27.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.326 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:27.326 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:53:27.326 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:27.326 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.326 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:27.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.326 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:53:27.327 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:53:27.327 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:27.327 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:27.327 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:27.327 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.327 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:27.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.327 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:53:27.327 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:53:27.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.327 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:27.327 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:27.327 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:27.327 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.328 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:27.328 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:53:27.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:27.328 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:27.328 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:27.328 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:27.328 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:27.328 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:27.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:27.328 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:53:27.328 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:53:27.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa16cc00640 1 -- 172.21.15.26:0/3035812233 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa148036420 legacy=0x7fa1480388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa16cc00640 10 monclient: shutdown 2024-09-16T14:53:27.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa16cc00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:27.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa16cc00640 1 -- 172.21.15.26:0/3035812233 >> v1:172.21.15.26:6790/0 conn(0x7fa16809b040 legacy=0x7fa1681a5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa166d9d640 1 -- 172.21.15.26:0/3035812233 reap_dead start 2024-09-16T14:53:27.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa166d9d640 1 -- 172.21.15.26:0/3035812233 reap_dead start 2024-09-16T14:53:27.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa16cc00640 1 -- 172.21.15.26:0/3035812233 shutdown_connections 2024-09-16T14:53:27.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa16cc00640 1 -- 172.21.15.26:0/3035812233 >> 172.21.15.26:0/3035812233 conn(0x7fa16809fb50 msgr2=0x7fa1681b6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:27.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa16cc00640 1 -- 172.21.15.26:0/3035812233 shutdown_connections 2024-09-16T14:53:27.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.303+0000 7fa16cc00640 1 -- 172.21.15.26:0/3035812233 wait complete. 2024-09-16T14:53:27.330 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:53:27.331 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T14:53:27.331 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:53:27.331 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T14:53:27.331 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:53:27.331 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T14:53:27.331 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:53:27.354 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T14:53:27.354 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:53:27.354 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:53:27.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f17608c1640 10 monclient: get_monmap_and_config 2024-09-16T14:53:27.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f17608c1640 10 monclient: build_initial_monmap 2024-09-16T14:53:27.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f17608c1640 10 monclient: monmap: 2024-09-16T14:53:27.473 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:27.473 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.473 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:53:27.474532+0000 2024-09-16T14:53:27.473 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:53:27.474532+0000 2024-09-16T14:53:27.473 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:27.473 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.474 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:27.474 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:53:27.474 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:53:27.474 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f17608c1640 1 Processor -- start 2024-09-16T14:53:27.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f17608c1640 1 -- start start 2024-09-16T14:53:27.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f17608c1640 10 monclient: init 2024-09-16T14:53:27.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f17608c1640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:27.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f17608c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:53:27.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f17608c1640 10 monclient(hunting): picked mon.noname-c con 0x7f175c0a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:27.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f17608c1640 10 monclient(hunting): picked mon.noname-a con 0x7f175c0a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:27.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f17608c1640 10 monclient(hunting): picked mon.noname-b con 0x7f175c0a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:27.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f17608c1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f175c0a3250 con 0x7f175c0a0000 2024-09-16T14:53:27.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f17608c1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f175c09f600 con 0x7f175c0a1d10 2024-09-16T14:53:27.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f17608c1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f175c0a3830 con 0x7f175c0a2800 2024-09-16T14:53:27.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f17608c1640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:27.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f17608c1640 10 monclient(hunting): authenticate will time out at 1631.595291s 2024-09-16T14:53:27.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f175a59c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f175c0a1d10 legacy=0x7f175c0a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:27.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f175ad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f175c0a2800 0x7f175c13d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:34822/0 (socket says 172.21.15.26:34822) 2024-09-16T14:53:27.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f175ad9d640 1 -- 172.21.15.26:0/2744637564 learned_addr learned my addr 172.21.15.26:0/2744637564 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:27.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.471+0000 7f1759d9b640 1 -- 172.21.15.26:0/2744637564 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723447489 0 0) 0x7f175c0a3830 con 0x7f175c0a2800 2024-09-16T14:53:27.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:53:27.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:27.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:27.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 10 monclient(hunting): my global_id is 234164 2024-09-16T14:53:27.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 1 -- 172.21.15.26:0/2744637564 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1748003530 con 0x7f175c0a2800 2024-09-16T14:53:27.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 1 -- 172.21.15.26:0/2744637564 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4190327921 0 0) 0x7f1748003530 con 0x7f175c0a2800 2024-09-16T14:53:27.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 1 -- 172.21.15.26:0/2744637564 >> v1:172.21.15.189:6789/0 conn(0x7f175c0a1d10 legacy=0x7f175c0a20f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:53:27.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 1 -- 172.21.15.26:0/2744637564 >> v1:172.21.15.26:6789/0 conn(0x7f175c0a0000 legacy=0x7f175c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:53:27.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 1 monclient: found mon.noname-b 2024-09-16T14:53:27.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:27.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:53:27.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 1 -- 172.21.15.26:0/2744637564 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175c09ed60 con 0x7f175c0a2800 2024-09-16T14:53:27.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 10 monclient: _finish_auth 0 2024-09-16T14:53:27.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:53:27.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:27.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 1 -- 172.21.15.26:0/2744637564 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f17540019a0 con 0x7f175c0a2800 2024-09-16T14:53:27.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:53:27.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 10 monclient: dump: 2024-09-16T14:53:27.482 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.482 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.482 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.482 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.482 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.482 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.482 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.482 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.482 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.482 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 5 monclient: authenticate success, global_id 234164 2024-09-16T14:53:27.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:27.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 1 -- 172.21.15.26:0/2744637564 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1754001bc0 con 0x7f175c0a2800 2024-09-16T14:53:27.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:27.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 1 -- 172.21.15.26:0/2744637564 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1754001f90 con 0x7f175c0a2800 2024-09-16T14:53:27.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:27.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 4 set_mon_vals no callback set 2024-09-16T14:53:27.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:27.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1759d9b640 10 monclient: dump: 2024-09-16T14:53:27.485 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.485 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.485 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.485 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.485 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.485 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.485 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.485 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.485 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.485 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 10 monclient: shutdown 2024-09-16T14:53:27.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:27.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 1 -- 172.21.15.26:0/2744637564 >> v1:172.21.15.26:6790/0 conn(0x7f175c0a2800 legacy=0x7f175c13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 1 -- 172.21.15.26:0/2744637564 shutdown_connections 2024-09-16T14:53:27.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 1 -- 172.21.15.26:0/2744637564 >> 172.21.15.26:0/2744637564 conn(0x7f175c0a7f30 msgr2=0x7f175c0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:27.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 1 -- 172.21.15.26:0/2744637564 shutdown_connections 2024-09-16T14:53:27.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 1 -- 172.21.15.26:0/2744637564 wait complete. 2024-09-16T14:53:27.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 10 monclient: build_initial_monmap 2024-09-16T14:53:27.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 10 monclient: monmap: 2024-09-16T14:53:27.486 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:27.486 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:27.487 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:27.487 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:27.487 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:27.487 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:27.487 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:27.487 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:27.487 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:27.487 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 1 Processor -- start 2024-09-16T14:53:27.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 1 -- start start 2024-09-16T14:53:27.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 10 monclient: init 2024-09-16T14:53:27.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:27.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:53:27.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 10 monclient(hunting): picked mon.noname-a con 0x7f175c0a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:27.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 10 monclient(hunting): picked mon.noname-c con 0x7f175c0a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:27.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 10 monclient(hunting): picked mon.noname-b con 0x7f175c0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:27.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f175c135590 con 0x7f175c0a0000 2024-09-16T14:53:27.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f175c132c50 con 0x7f175c0a2800 2024-09-16T14:53:27.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f175c09e500 con 0x7f175c0a1d10 2024-09-16T14:53:27.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:27.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 10 monclient(hunting): authenticate will time out at 1631.598573s 2024-09-16T14:53:27.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f175a59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f175c0a0000 0x7f175c131e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50086/0 (socket says 172.21.15.26:50086) 2024-09-16T14:53:27.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f175a59c640 1 -- 172.21.15.26:0/1157229211 learned_addr learned my addr 172.21.15.26:0/1157229211 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:27.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f175ad9d640 1 -- 172.21.15.26:0/1157229211 >> v1:172.21.15.189:6789/0 conn(0x7f175c0a2800 legacy=0x7f175c1d19b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:27.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 1 -- 172.21.15.26:0/1157229211 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721354759 0 0) 0x7f175c09e500 con 0x7f175c0a1d10 2024-09-16T14:53:27.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:53:27.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:27.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:27.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 10 monclient(hunting): my global_id is 234167 2024-09-16T14:53:27.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 1 -- 172.21.15.26:0/1157229211 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f173c003530 con 0x7f175c0a1d10 2024-09-16T14:53:27.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 1 -- 172.21.15.26:0/1157229211 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1628197073 0 0) 0x7f173c003530 con 0x7f175c0a1d10 2024-09-16T14:53:27.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 1 -- 172.21.15.26:0/1157229211 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f175c09e500 con 0x7f175c0a1d10 2024-09-16T14:53:27.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 1 -- 172.21.15.26:0/1157229211 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1744001880 con 0x7f175c0a1d10 2024-09-16T14:53:27.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:53:27.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 10 monclient(hunting): dump: 2024-09-16T14:53:27.489 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.489 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.489 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.490 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.490 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.490 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.490 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.490 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.490 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.490 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:27.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 1 -- 172.21.15.26:0/1157229211 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1505248231 0 0) 0x7f175c09e500 con 0x7f175c0a1d10 2024-09-16T14:53:27.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 1 -- 172.21.15.26:0/1157229211 >> v1:172.21.15.189:6789/0 conn(0x7f175c0a2800 legacy=0x7f175c1d19b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:53:27.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 1 -- 172.21.15.26:0/1157229211 >> v1:172.21.15.26:6789/0 conn(0x7f175c0a0000 legacy=0x7f175c131e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:53:27.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 1 monclient: found mon.c 2024-09-16T14:53:27.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:27.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:27.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 1 -- 172.21.15.26:0/1157229211 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175c0a6680 con 0x7f175c0a1d10 2024-09-16T14:53:27.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 10 monclient: _finish_auth 0 2024-09-16T14:53:27.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:53:27.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:27.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 5 monclient: authenticate success, global_id 234167 2024-09-16T14:53:27.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 10 monclient: _renew_subs 2024-09-16T14:53:27.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:27.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 1 -- 172.21.15.26:0/1157229211 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f175c136070 con 0x7f175c0a1d10 2024-09-16T14:53:27.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 10 monclient: _renew_subs 2024-09-16T14:53:27.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:27.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f17608c1640 1 -- 172.21.15.26:0/1157229211 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f175c133ad0 con 0x7f175c0a1d10 2024-09-16T14:53:27.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 1 -- 172.21.15.26:0/1157229211 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1744001c50 con 0x7f175c0a1d10 2024-09-16T14:53:27.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:27.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 1 -- 172.21.15.26:0/1157229211 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1744003a70 con 0x7f175c0a1d10 2024-09-16T14:53:27.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:27.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:27.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1758d99640 4 set_mon_vals no callback set 2024-09-16T14:53:27.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.475+0000 7f1752ffd640 10 monclient: dump: 2024-09-16T14:53:27.492 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:27.492 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:27.492 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.492 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:27.492 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:27.492 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:27.492 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:27.492 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:27.492 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:27.492 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:27.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.479+0000 7f17608c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:27.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.479+0000 7f17608c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:27.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.479+0000 7f17608c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:27.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.479+0000 7f17608c1640 1 -- 172.21.15.26:0/1157229211 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1740005050 con 0x7f175c0a1d10 2024-09-16T14:53:27.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.479+0000 7f1752ffd640 1 -- 172.21.15.26:0/1157229211 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1744003c50 con 0x7f175c0a1d10 2024-09-16T14:53:27.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.479+0000 7f1752ffd640 1 -- 172.21.15.26:0/1157229211 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f1744042d40 con 0x7f175c0a1d10 2024-09-16T14:53:27.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.479+0000 7f175a59c640 10 monclient: get_auth_request con 0x7f173c036030 auth_method 0 2024-09-16T14:53:27.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.483+0000 7f1752ffd640 1 -- 172.21.15.26:0/1157229211 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1744013450 con 0x7f175c0a1d10 2024-09-16T14:53:27.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.483+0000 7f1752ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:27.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.483+0000 7f1752ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:27.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.707+0000 7f17608c1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:53:27.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.707+0000 7f17608c1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:53:27.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.707+0000 7f17608c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:27.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.707+0000 7f17608c1640 1 -- 172.21.15.26:0/1157229211 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1740005880 con 0x7f175c0a1d10 2024-09-16T14:53:27.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.707+0000 7f1752ffd640 1 -- 172.21.15.26:0/1157229211 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2570729168) 0x7f174401bc70 con 0x7f175c0a1d10 2024-09-16T14:53:27.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.707+0000 7f1752ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:53:27.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.707+0000 7f1752ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:27.710 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:53:27.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.707+0000 7f17608c1640 1 -- 172.21.15.26:0/1157229211 >> v1:172.21.15.189:6812/3196161582 conn(0x7f173c036030 legacy=0x7f173c0384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.707+0000 7f17608c1640 10 monclient: shutdown 2024-09-16T14:53:27.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.707+0000 7f17608c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:27.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.707+0000 7f17608c1640 1 -- 172.21.15.26:0/1157229211 >> v1:172.21.15.26:6790/0 conn(0x7f175c0a1d10 legacy=0x7f175c132540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:27.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.707+0000 7f17608c1640 1 -- 172.21.15.26:0/1157229211 shutdown_connections 2024-09-16T14:53:27.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.707+0000 7f17608c1640 1 -- 172.21.15.26:0/1157229211 >> 172.21.15.26:0/1157229211 conn(0x7f175c0a7f30 msgr2=0x7f175c06b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:27.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.707+0000 7f17608c1640 1 -- 172.21.15.26:0/1157229211 shutdown_connections 2024-09-16T14:53:27.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:27.707+0000 7f17608c1640 1 -- 172.21.15.26:0/1157229211 wait complete. 2024-09-16T14:53:30.729 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:53:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfe25df640 10 monclient: get_monmap_and_config 2024-09-16T14:53:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfe25df640 10 monclient: build_initial_monmap 2024-09-16T14:53:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfe25df640 10 monclient: monmap: 2024-09-16T14:53:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:53:30.863028+0000 2024-09-16T14:53:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:53:30.863028+0000 2024-09-16T14:53:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:53:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:53:30.862 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfe25df640 1 Processor -- start 2024-09-16T14:53:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfe25df640 1 -- start start 2024-09-16T14:53:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfe25df640 10 monclient: init 2024-09-16T14:53:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfe25df640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfe25df640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:53:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfe25df640 10 monclient(hunting): picked mon.noname-c con 0x7fbfdc0a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfe25df640 10 monclient(hunting): picked mon.noname-b con 0x7fbfdc0a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfe25df640 10 monclient(hunting): picked mon.noname-a con 0x7fbfdc0a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfe25df640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc0a3250 con 0x7fbfdc0a2800 2024-09-16T14:53:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfe25df640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc09f600 con 0x7fbfdc0a1d10 2024-09-16T14:53:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfe25df640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc0a3830 con 0x7fbfdc0a0000 2024-09-16T14:53:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfe25df640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfe25df640 10 monclient(hunting): authenticate will time out at 1634.983119s 2024-09-16T14:53:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdb7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbfdc0a0000 0x7fbfdc139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:34834/0 (socket says 172.21.15.26:34834) 2024-09-16T14:53:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdb7fe640 1 -- 172.21.15.26:0/1832364434 learned_addr learned my addr 172.21.15.26:0/1832364434 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdbfff640 1 -- 172.21.15.26:0/1832364434 >> v1:172.21.15.189:6789/0 conn(0x7fbfdc0a1d10 legacy=0x7fbfdc0a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 1 -- 172.21.15.26:0/1832364434 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372800436 0 0) 0x7fbfdc0a3830 con 0x7fbfdc0a0000 2024-09-16T14:53:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:53:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 10 monclient(hunting): my global_id is 234170 2024-09-16T14:53:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 1 -- 172.21.15.26:0/1832364434 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc8003530 con 0x7fbfdc0a0000 2024-09-16T14:53:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 1 -- 172.21.15.26:0/1832364434 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2173466213 0 0) 0x7fbfc8003530 con 0x7fbfdc0a0000 2024-09-16T14:53:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 1 -- 172.21.15.26:0/1832364434 >> v1:172.21.15.189:6789/0 conn(0x7fbfdc0a1d10 legacy=0x7fbfdc0a20f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:53:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 1 -- 172.21.15.26:0/1832364434 >> v1:172.21.15.26:6789/0 conn(0x7fbfdc0a2800 legacy=0x7fbfdc13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 10 monclient: _finish_hunting 0 2024-09-16T14:53:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 1 monclient: found mon.noname-b 2024-09-16T14:53:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:53:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 1 -- 172.21.15.26:0/1832364434 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc09ed60 con 0x7fbfdc0a0000 2024-09-16T14:53:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 10 monclient: _finish_auth 0 2024-09-16T14:53:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 10 monclient: _check_auth_tickets 2024-09-16T14:53:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 1 -- 172.21.15.26:0/1832364434 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbfc4001620 con 0x7fbfdc0a0000 2024-09-16T14:53:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:53:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.859+0000 7fbfdaffd640 10 monclient: dump: 2024-09-16T14:53:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:30.869 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 5 monclient: authenticate success, global_id 234170 2024-09-16T14:53:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfdaffd640 1 -- 172.21.15.26:0/1832364434 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfc4001940 con 0x7fbfdc0a0000 2024-09-16T14:53:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfdaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfdaffd640 1 -- 172.21.15.26:0/1832364434 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbfc4001cd0 con 0x7fbfdc0a0000 2024-09-16T14:53:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfdaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfdaffd640 10 monclient: dump: 2024-09-16T14:53:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:30.870 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:30.870 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:30.870 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:30.870 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:30.870 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:30.870 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:30.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:30.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 4 set_mon_vals no callback set 2024-09-16T14:53:30.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient: shutdown 2024-09-16T14:53:30.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:30.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 1 -- 172.21.15.26:0/1832364434 >> v1:172.21.15.26:6790/0 conn(0x7fbfdc0a0000 legacy=0x7fbfdc139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:30.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 1 -- 172.21.15.26:0/1832364434 shutdown_connections 2024-09-16T14:53:30.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 1 -- 172.21.15.26:0/1832364434 >> 172.21.15.26:0/1832364434 conn(0x7fbfdc0a7f30 msgr2=0x7fbfdc0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:30.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 1 -- 172.21.15.26:0/1832364434 shutdown_connections 2024-09-16T14:53:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 1 -- 172.21.15.26:0/1832364434 wait complete. 2024-09-16T14:53:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient: build_initial_monmap 2024-09-16T14:53:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient: monmap: 2024-09-16T14:53:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:30.871 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:30.871 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:30.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 1 Processor -- start 2024-09-16T14:53:30.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 1 -- start start 2024-09-16T14:53:30.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient: init 2024-09-16T14:53:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:53:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient(hunting): picked mon.noname-a con 0x7fbfdc0a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient(hunting): picked mon.noname-b con 0x7fbfdc0a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient(hunting): picked mon.noname-c con 0x7fbfdc0a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc13a190 con 0x7fbfdc0a0000 2024-09-16T14:53:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc13bb40 con 0x7fbfdc0a1d10 2024-09-16T14:53:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc0a65a0 con 0x7fbfdc0a2800 2024-09-16T14:53:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient(hunting): authenticate will time out at 1634.986155s 2024-09-16T14:53:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfdbfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbfdc0a0000 0x7fbfdc139370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50098/0 (socket says 172.21.15.26:50098) 2024-09-16T14:53:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfdbfff640 1 -- 172.21.15.26:0/4241788045 learned_addr learned my addr 172.21.15.26:0/4241788045 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfdb7fe640 1 -- 172.21.15.26:0/4241788045 >> v1:172.21.15.189:6789/0 conn(0x7fbfdc0a1d10 legacy=0x7fbfdc139a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 1 -- 172.21.15.26:0/4241788045 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682200819 0 0) 0x7fbfdc0a65a0 con 0x7fbfdc0a2800 2024-09-16T14:53:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:53:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 10 monclient(hunting): my global_id is 234173 2024-09-16T14:53:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 1 -- 172.21.15.26:0/4241788045 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfbc003530 con 0x7fbfdc0a2800 2024-09-16T14:53:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 1 -- 172.21.15.26:0/4241788045 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3694250877 0 0) 0x7fbfbc003530 con 0x7fbfdc0a2800 2024-09-16T14:53:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 1 -- 172.21.15.26:0/4241788045 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfdc0a65a0 con 0x7fbfdc0a2800 2024-09-16T14:53:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 1 -- 172.21.15.26:0/4241788045 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbfcc001770 con 0x7fbfdc0a2800 2024-09-16T14:53:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:53:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 10 monclient(hunting): dump: 2024-09-16T14:53:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:30.878 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 1 -- 172.21.15.26:0/4241788045 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2971363571 0 0) 0x7fbfdc0a65a0 con 0x7fbfdc0a2800 2024-09-16T14:53:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 1 -- 172.21.15.26:0/4241788045 >> v1:172.21.15.189:6789/0 conn(0x7fbfdc0a1d10 legacy=0x7fbfdc139a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:53:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 1 -- 172.21.15.26:0/4241788045 >> v1:172.21.15.26:6789/0 conn(0x7fbfdc0a0000 legacy=0x7fbfdc139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:53:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 1 monclient: found mon.c 2024-09-16T14:53:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 1 -- 172.21.15.26:0/4241788045 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc09e500 con 0x7fbfdc0a2800 2024-09-16T14:53:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 10 monclient: _finish_auth 0 2024-09-16T14:53:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:53:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 5 monclient: authenticate success, global_id 234173 2024-09-16T14:53:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient: _renew_subs 2024-09-16T14:53:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 1 -- 172.21.15.26:0/4241788045 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfdc13d2d0 con 0x7fbfdc0a2800 2024-09-16T14:53:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient: _renew_subs 2024-09-16T14:53:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 1 -- 172.21.15.26:0/4241788045 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfdc13cff0 con 0x7fbfdc0a2800 2024-09-16T14:53:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 1 -- 172.21.15.26:0/4241788045 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfcc001d60 con 0x7fbfdc0a2800 2024-09-16T14:53:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 1 -- 172.21.15.26:0/4241788045 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbfcc0037c0 con 0x7fbfdc0a2800 2024-09-16T14:53:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfda7fc640 4 set_mon_vals no callback set 2024-09-16T14:53:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfd8ff9640 10 monclient: dump: 2024-09-16T14:53:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:30.881 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.863+0000 7fbfe25df640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.867+0000 7fbfe25df640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.867+0000 7fbfe25df640 1 -- 172.21.15.26:0/4241788045 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc0005050 con 0x7fbfdc0a2800 2024-09-16T14:53:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.867+0000 7fbfd8ff9640 1 -- 172.21.15.26:0/4241788045 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbfcc0039e0 con 0x7fbfdc0a2800 2024-09-16T14:53:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.867+0000 7fbfd8ff9640 1 -- 172.21.15.26:0/4241788045 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fbfcc042af0 con 0x7fbfdc0a2800 2024-09-16T14:53:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.867+0000 7fbfdbfff640 10 monclient: get_auth_request con 0x7fbfbc036050 auth_method 0 2024-09-16T14:53:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.871+0000 7fbfd8ff9640 1 -- 172.21.15.26:0/4241788045 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbfcc004200 con 0x7fbfdc0a2800 2024-09-16T14:53:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.871+0000 7fbfd8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:30.871+0000 7fbfd8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:31.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:31.091+0000 7fbfe25df640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:53:31.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:31.091+0000 7fbfe25df640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:53:31.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:31.091+0000 7fbfe25df640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:31.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:31.091+0000 7fbfe25df640 1 -- 172.21.15.26:0/4241788045 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbfc0005880 con 0x7fbfdc0a2800 2024-09-16T14:53:31.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:31.091+0000 7fbfd8ff9640 1 -- 172.21.15.26:0/4241788045 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 3386308849) 0x7fbfcc011900 con 0x7fbfdc0a2800 2024-09-16T14:53:31.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:31.091+0000 7fbfd8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:53:31.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:31.091+0000 7fbfd8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:31.093 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:53:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:31.091+0000 7fbfe25df640 1 -- 172.21.15.26:0/4241788045 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbfbc036050 legacy=0x7fbfbc038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:31.091+0000 7fbfe25df640 10 monclient: shutdown 2024-09-16T14:53:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:31.091+0000 7fbfe25df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:31.091+0000 7fbfe25df640 1 -- 172.21.15.26:0/4241788045 >> v1:172.21.15.26:6790/0 conn(0x7fbfdc0a2800 legacy=0x7fbfdc1d1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:31.091+0000 7fbfe25df640 1 -- 172.21.15.26:0/4241788045 shutdown_connections 2024-09-16T14:53:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:31.091+0000 7fbfe25df640 1 -- 172.21.15.26:0/4241788045 >> 172.21.15.26:0/4241788045 conn(0x7fbfdc0a7f30 msgr2=0x7fbfdc13e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:31.091+0000 7fbfe25df640 1 -- 172.21.15.26:0/4241788045 shutdown_connections 2024-09-16T14:53:31.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:31.091+0000 7fbfe25df640 1 -- 172.21.15.26:0/4241788045 wait complete. 2024-09-16T14:53:34.112 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:53:34.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bf500640 10 monclient: get_monmap_and_config 2024-09-16T14:53:34.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bf500640 10 monclient: build_initial_monmap 2024-09-16T14:53:34.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bf500640 10 monclient: monmap: 2024-09-16T14:53:34.246 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:34.246 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:34.246 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:53:34.247294+0000 2024-09-16T14:53:34.246 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:53:34.247294+0000 2024-09-16T14:53:34.246 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:34.246 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:34.246 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:34.246 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:53:34.247 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:53:34.247 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:34.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bf500640 1 Processor -- start 2024-09-16T14:53:34.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bf500640 1 -- start start 2024-09-16T14:53:34.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bf500640 10 monclient: init 2024-09-16T14:53:34.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bf500640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:34.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bf500640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:53:34.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bf500640 10 monclient(hunting): picked mon.noname-b con 0x7f29b80a4490 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:34.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bf500640 10 monclient(hunting): picked mon.noname-a con 0x7f29b80999d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:34.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bf500640 10 monclient(hunting): picked mon.noname-c con 0x7f29b813adb0 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:34.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bf500640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b80a4870 con 0x7f29b80999d0 2024-09-16T14:53:34.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bf500640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b80a29b0 con 0x7f29b813adb0 2024-09-16T14:53:34.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bf500640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b806ca00 con 0x7f29b80a4490 2024-09-16T14:53:34.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bf500640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:34.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bf500640 10 monclient(hunting): authenticate will time out at 1638.367408s 2024-09-16T14:53:34.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bda9d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f29b813adb0 legacy=0x7f29b813d1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:34.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bca9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f29b80999d0 0x7f29b80a22a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50106/0 (socket says 172.21.15.26:50106) 2024-09-16T14:53:34.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.243+0000 7f29bca9b640 1 -- 172.21.15.26:0/700689175 learned_addr learned my addr 172.21.15.26:0/700689175 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.447+0000 7f29bda9d640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.189:6789/0 conn(0x7f29b813adb0 legacy=0x7f29b813d1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:34.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:34.851+0000 7f29bda9d640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.189:6789/0 conn(0x7f29b813adb0 legacy=0x7f29b813d1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:35.655+0000 7f29bda9d640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.189:6789/0 conn(0x7f29b813adb0 legacy=0x7f29b813d1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:37.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.247+0000 7f29b6ffd640 10 monclient(hunting): tick 2024-09-16T14:53:37.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.247+0000 7f29b6ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:53:37.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.247+0000 7f29b6ffd640 1 monclient(hunting): continuing hunt 2024-09-16T14:53:37.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.247+0000 7f29b6ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:53:37.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.247+0000 7f29b6ffd640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.26:6790/0 conn(0x7f29b80a4490 legacy=0x7f29b80a1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:37.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.247+0000 7f29b6ffd640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.189:6789/0 conn(0x7f29b813adb0 legacy=0x7f29b813d1a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:53:37.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.247+0000 7f29b6ffd640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.26:6789/0 conn(0x7f29b80999d0 legacy=0x7f29b80a22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:37.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.247+0000 7f29b6ffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:53:37.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.247+0000 7f29b6ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f29b813adb0 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:37.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.247+0000 7f29b6ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f29a4006e60 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:37.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.247+0000 7f29b6ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f29b80a4490 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:37.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.247+0000 7f29b6ffd640 1 -- 172.21.15.26:0/700689175 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b8099e80 con 0x7f29a4006e60 2024-09-16T14:53:37.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.247+0000 7f29b6ffd640 1 -- 172.21.15.26:0/700689175 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b80a29b0 con 0x7f29b813adb0 2024-09-16T14:53:37.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.247+0000 7f29b6ffd640 1 -- 172.21.15.26:0/700689175 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a4010380 con 0x7f29b80a4490 2024-09-16T14:53:37.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.247+0000 7f29b6ffd640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:37.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.247+0000 7f29bd29c640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.189:6789/0 conn(0x7f29b813adb0 legacy=0x7f29a4005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:37.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.451+0000 7f29bd29c640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.189:6789/0 conn(0x7f29b813adb0 legacy=0x7f29a4005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:37.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:37.855+0000 7f29bd29c640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.189:6789/0 conn(0x7f29b813adb0 legacy=0x7f29a4005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:38.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:38.659+0000 7f29bd29c640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.189:6789/0 conn(0x7f29b813adb0 legacy=0x7f29a4005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:40.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.247+0000 7f29b6ffd640 10 monclient(hunting): tick 2024-09-16T14:53:40.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.247+0000 7f29b6ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:53:40.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.247+0000 7f29b6ffd640 1 monclient(hunting): continuing hunt 2024-09-16T14:53:40.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.247+0000 7f29b6ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:53:40.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.247+0000 7f29b6ffd640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.26:6790/0 conn(0x7f29b80a4490 legacy=0x7f29a400cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:40.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.247+0000 7f29b6ffd640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.189:6789/0 conn(0x7f29b813adb0 legacy=0x7f29a4005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:53:40.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.247+0000 7f29b6ffd640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.26:6789/0 conn(0x7f29a4006e60 legacy=0x7f29a40092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:40.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.247+0000 7f29b6ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:53:40.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.247+0000 7f29b6ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f29a4010610 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:40.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.247+0000 7f29b6ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f29a40128f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:40.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.247+0000 7f29b6ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f29b813adb0 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:40.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.247+0000 7f29b6ffd640 1 -- 172.21.15.26:0/700689175 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a40115f0 con 0x7f29a40128f0 2024-09-16T14:53:40.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.247+0000 7f29b6ffd640 1 -- 172.21.15.26:0/700689175 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b80a29b0 con 0x7f29b813adb0 2024-09-16T14:53:40.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.247+0000 7f29b6ffd640 1 -- 172.21.15.26:0/700689175 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a40082f0 con 0x7f29a4010610 2024-09-16T14:53:40.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.247+0000 7f29b6ffd640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:40.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.247+0000 7f29bda9d640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.189:6789/0 conn(0x7f29b813adb0 legacy=0x7f29a4005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:40.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.451+0000 7f29bda9d640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.189:6789/0 conn(0x7f29b813adb0 legacy=0x7f29a4005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:40.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:40.855+0000 7f29bda9d640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.189:6789/0 conn(0x7f29b813adb0 legacy=0x7f29a4005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.467+0000 7f29b7fff640 1 -- 172.21.15.26:0/700689175 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571892284 0 0) 0x7f29a40082f0 con 0x7f29a4010610 2024-09-16T14:53:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.467+0000 7f29b7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:53:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.467+0000 7f29b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.467+0000 7f29b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.467+0000 7f29b7fff640 10 monclient(hunting): my global_id is 234176 2024-09-16T14:53:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.467+0000 7f29b7fff640 1 -- 172.21.15.26:0/700689175 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a0003530 con 0x7f29a4010610 2024-09-16T14:53:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.467+0000 7f29b7fff640 1 -- 172.21.15.26:0/700689175 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774729286 0 0) 0x7f29a40115f0 con 0x7f29a40128f0 2024-09-16T14:53:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.467+0000 7f29b7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:53:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.467+0000 7f29b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.467+0000 7f29b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.467+0000 7f29b7fff640 10 monclient(hunting): my global_id is 194493 2024-09-16T14:53:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.467+0000 7f29b7fff640 1 -- 172.21.15.26:0/700689175 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a40082f0 con 0x7f29a40128f0 2024-09-16T14:53:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 1 -- 172.21.15.26:0/700689175 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2663079451 0 0) 0x7f29a0003530 con 0x7f29a4010610 2024-09-16T14:53:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.189:6789/0 conn(0x7f29b813adb0 legacy=0x7f29a4005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:53:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.26:6789/0 conn(0x7f29a40128f0 legacy=0x7f29a4014ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:53:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 1 monclient: found mon.noname-b 2024-09-16T14:53:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:53:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 1 -- 172.21.15.26:0/700689175 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a4009490 con 0x7f29a4010610 2024-09-16T14:53:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 10 monclient: _finish_auth 0 2024-09-16T14:53:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:53:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 1 -- 172.21.15.26:0/700689175 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f29a80019d0 con 0x7f29a4010610 2024-09-16T14:53:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29bf500640 5 monclient: authenticate success, global_id 234176 2024-09-16T14:53:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29bf500640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:53:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 10 monclient: dump: 2024-09-16T14:53:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:41.477 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29bf500640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 1 -- 172.21.15.26:0/700689175 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29a8001c70 con 0x7f29a4010610 2024-09-16T14:53:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 1 -- 172.21.15.26:0/700689175 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f29a8002000 con 0x7f29a4010610 2024-09-16T14:53:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29bf500640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29bf500640 4 set_mon_vals no callback set 2024-09-16T14:53:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29b7fff640 10 monclient: dump: 2024-09-16T14:53:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:41.479 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:41.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29bf500640 10 monclient: shutdown 2024-09-16T14:53:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29bf500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29bf500640 1 -- 172.21.15.26:0/700689175 >> v1:172.21.15.26:6790/0 conn(0x7f29a4010610 legacy=0x7f29a40109f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29bf500640 1 -- 172.21.15.26:0/700689175 shutdown_connections 2024-09-16T14:53:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29bf500640 1 -- 172.21.15.26:0/700689175 >> 172.21.15.26:0/700689175 conn(0x7f29b80a7a00 msgr2=0x7f29b80a3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29bf500640 1 -- 172.21.15.26:0/700689175 shutdown_connections 2024-09-16T14:53:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.471+0000 7f29bf500640 1 -- 172.21.15.26:0/700689175 wait complete. 2024-09-16T14:53:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bf500640 10 monclient: build_initial_monmap 2024-09-16T14:53:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bf500640 10 monclient: monmap: 2024-09-16T14:53:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:41.483 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bf500640 1 Processor -- start 2024-09-16T14:53:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bf500640 1 -- start start 2024-09-16T14:53:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bf500640 10 monclient: init 2024-09-16T14:53:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bf500640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bf500640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:53:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bf500640 10 monclient(hunting): picked mon.noname-a con 0x7f29a4010610 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bf500640 10 monclient(hunting): picked mon.noname-c con 0x7f29a40128f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bf500640 10 monclient(hunting): picked mon.noname-b con 0x7f29b813adb0 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bf500640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b811e7e0 con 0x7f29a4010610 2024-09-16T14:53:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bf500640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b809ece0 con 0x7f29b813adb0 2024-09-16T14:53:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bf500640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b809e790 con 0x7f29a40128f0 2024-09-16T14:53:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bf500640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bf500640 10 monclient(hunting): authenticate will time out at 1645.597937s 2024-09-16T14:53:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bd29c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f29a4010610 0x7f29b81c9390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35160/0 (socket says 172.21.15.26:35160) 2024-09-16T14:53:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bd29c640 1 -- 172.21.15.26:0/694144834 learned_addr learned my addr 172.21.15.26:0/694144834 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.475+0000 7f29bda9d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f29b813adb0 legacy=0x7f29b81d1710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 1 -- 172.21.15.26:0/694144834 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553553331 0 0) 0x7f29b809e790 con 0x7f29a40128f0 2024-09-16T14:53:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 10 monclient(hunting): my global_id is 234179 2024-09-16T14:53:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 1 -- 172.21.15.26:0/694144834 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a4011480 con 0x7f29a40128f0 2024-09-16T14:53:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 1 -- 172.21.15.26:0/694144834 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1513424338 0 0) 0x7f29a4011480 con 0x7f29a40128f0 2024-09-16T14:53:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 1 -- 172.21.15.26:0/694144834 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29b809e790 con 0x7f29a40128f0 2024-09-16T14:53:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 1 -- 172.21.15.26:0/694144834 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f29ac0017b0 con 0x7f29a40128f0 2024-09-16T14:53:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:53:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 10 monclient(hunting): dump: 2024-09-16T14:53:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:41.486 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:41.486 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:41.486 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:41.486 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:41.486 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:41.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:41.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 1 -- 172.21.15.26:0/694144834 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1588144953 0 0) 0x7f29b809e790 con 0x7f29a40128f0 2024-09-16T14:53:41.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 1 -- 172.21.15.26:0/694144834 >> v1:172.21.15.189:6789/0 conn(0x7f29b813adb0 legacy=0x7f29b81d1710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:53:41.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 1 -- 172.21.15.26:0/694144834 >> v1:172.21.15.26:6789/0 conn(0x7f29a4010610 legacy=0x7f29b81c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:41.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:53:41.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 1 monclient: found mon.c 2024-09-16T14:53:41.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:41.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:41.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 1 -- 172.21.15.26:0/694144834 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b809e380 con 0x7f29a40128f0 2024-09-16T14:53:41.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 10 monclient: _finish_auth 0 2024-09-16T14:53:41.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:53:41.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:41.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29bf500640 5 monclient: authenticate success, global_id 234179 2024-09-16T14:53:41.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29bf500640 10 monclient: _renew_subs 2024-09-16T14:53:41.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29bf500640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:41.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29bf500640 1 -- 172.21.15.26:0/694144834 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b811e500 con 0x7f29a40128f0 2024-09-16T14:53:41.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29bf500640 10 monclient: _renew_subs 2024-09-16T14:53:41.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29bf500640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29bf500640 1 -- 172.21.15.26:0/694144834 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b8138ad0 con 0x7f29a40128f0 2024-09-16T14:53:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 1 -- 172.21.15.26:0/694144834 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29ac0047f0 con 0x7f29a40128f0 2024-09-16T14:53:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 1 -- 172.21.15.26:0/694144834 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f29ac004b00 con 0x7f29a40128f0 2024-09-16T14:53:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b5ffb640 10 monclient: dump: 2024-09-16T14:53:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:41.490 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29b77fe640 4 set_mon_vals no callback set 2024-09-16T14:53:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29bf500640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29bf500640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29bf500640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.479+0000 7f29bf500640 1 -- 172.21.15.26:0/694144834 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2998005050 con 0x7f29a40128f0 2024-09-16T14:53:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.483+0000 7f29b5ffb640 1 -- 172.21.15.26:0/694144834 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f29ac00f570 con 0x7f29a40128f0 2024-09-16T14:53:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.483+0000 7f29b5ffb640 1 -- 172.21.15.26:0/694144834 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f29ac043c10 con 0x7f29a40128f0 2024-09-16T14:53:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.483+0000 7f29bd29c640 10 monclient: get_auth_request con 0x7f29a4037320 auth_method 0 2024-09-16T14:53:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.487+0000 7f29b5ffb640 1 -- 172.21.15.26:0/694144834 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f29ac014320 con 0x7f29a40128f0 2024-09-16T14:53:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.487+0000 7f29b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.487+0000 7f29b5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:41.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.739+0000 7f29bf500640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:53:41.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.739+0000 7f29bf500640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:53:41.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.739+0000 7f29bf500640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:41.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.739+0000 7f29bf500640 1 -- 172.21.15.26:0/694144834 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2998005880 con 0x7f29a40128f0 2024-09-16T14:53:41.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.739+0000 7f29b5ffb640 1 -- 172.21.15.26:0/694144834 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 1224498896) 0x7f29ac0129c0 con 0x7f29a40128f0 2024-09-16T14:53:41.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.739+0000 7f29b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:53:41.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.739+0000 7f29b5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:41.743 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":106,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:53:41.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.743+0000 7f29bf500640 1 -- 172.21.15.26:0/694144834 >> v1:172.21.15.189:6812/3196161582 conn(0x7f29a4037320 legacy=0x7f29a40397e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:41.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.743+0000 7f29bf500640 10 monclient: shutdown 2024-09-16T14:53:41.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.743+0000 7f29bf500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:41.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.743+0000 7f29bf500640 1 -- 172.21.15.26:0/694144834 >> v1:172.21.15.26:6790/0 conn(0x7f29a40128f0 legacy=0x7f29b8132e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:41.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.743+0000 7f29bf500640 1 -- 172.21.15.26:0/694144834 shutdown_connections 2024-09-16T14:53:41.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.743+0000 7f29bf500640 1 -- 172.21.15.26:0/694144834 >> 172.21.15.26:0/694144834 conn(0x7f29b80a7a00 msgr2=0x7f29b80a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:41.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.743+0000 7f29bf500640 1 -- 172.21.15.26:0/694144834 shutdown_connections 2024-09-16T14:53:41.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.743+0000 7f29bf500640 1 -- 172.21.15.26:0/694144834 wait complete. 2024-09-16T14:53:41.761 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:53:41.761 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T14:53:41.762 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:53:41.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.883+0000 7f8a0b5ac640 10 monclient: get_monmap_and_config 2024-09-16T14:53:41.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.883+0000 7f8a0b5ac640 10 monclient: build_initial_monmap 2024-09-16T14:53:41.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.883+0000 7f8a0b5ac640 10 monclient: monmap: 2024-09-16T14:53:41.888 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:41.888 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:41.888 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:53:41.889292+0000 2024-09-16T14:53:41.888 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:53:41.889292+0000 2024-09-16T14:53:41.888 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:41.888 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:41.888 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:41.888 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:53:41.888 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:53:41.888 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:41.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 1 Processor -- start 2024-09-16T14:53:41.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 1 -- start start 2024-09-16T14:53:41.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 10 monclient: init 2024-09-16T14:53:41.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:41.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:53:41.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 10 monclient(hunting): picked mon.noname-c con 0x7f8a0409cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:41.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 10 monclient(hunting): picked mon.noname-b con 0x7f8a0409e7a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:41.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 10 monclient(hunting): picked mon.noname-a con 0x7f8a0409d1a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:41.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0409d5b0 con 0x7f8a0409d1a0 2024-09-16T14:53:41.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a04131a90 con 0x7f8a0409cdc0 2024-09-16T14:53:41.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0409aff0 con 0x7f8a0409e7a0 2024-09-16T14:53:41.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:41.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 10 monclient(hunting): authenticate will time out at 1646.009293s 2024-09-16T14:53:41.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a08b47640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8a0409e7a0 0x7f8a04131380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60460/0 (socket says 172.21.15.26:60460) 2024-09-16T14:53:41.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a09348640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f8a0409cdc0 legacy=0x7f8a04130c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:41.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a09b49640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8a0409d1a0 0x7f8a04139fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35176/0 (socket says 172.21.15.26:35176) 2024-09-16T14:53:41.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a08b47640 1 -- 172.21.15.26:0/4067013859 learned_addr learned my addr 172.21.15.26:0/4067013859 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:41.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 1 -- 172.21.15.26:0/4067013859 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893629364 0 0) 0x7f8a0409aff0 con 0x7f8a0409e7a0 2024-09-16T14:53:41.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:53:41.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:41.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:41.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient(hunting): my global_id is 234182 2024-09-16T14:53:41.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 1 -- 172.21.15.26:0/4067013859 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89f0003530 con 0x7f8a0409e7a0 2024-09-16T14:53:41.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 1 -- 172.21.15.26:0/4067013859 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308551938 0 0) 0x7f8a0409d5b0 con 0x7f8a0409d1a0 2024-09-16T14:53:41.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient(hunting): _negotiate 2024-09-16T14:53:41.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:41.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:41.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient(hunting): my global_id is 194496 2024-09-16T14:53:41.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 1 -- 172.21.15.26:0/4067013859 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a0409aff0 con 0x7f8a0409d1a0 2024-09-16T14:53:41.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 1 -- 172.21.15.26:0/4067013859 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3891006568 0 0) 0x7f8a0409aff0 con 0x7f8a0409d1a0 2024-09-16T14:53:41.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 1 -- 172.21.15.26:0/4067013859 >> v1:172.21.15.26:6790/0 conn(0x7f8a0409e7a0 legacy=0x7f8a04131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:41.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 1 -- 172.21.15.26:0/4067013859 >> v1:172.21.15.189:6789/0 conn(0x7f8a0409cdc0 legacy=0x7f8a04130c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:53:41.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient: _finish_hunting 0 2024-09-16T14:53:41.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 1 monclient: found mon.noname-a 2024-09-16T14:53:41.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:41.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:53:41.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 1 -- 172.21.15.26:0/4067013859 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0409b2d0 con 0x7f8a0409d1a0 2024-09-16T14:53:41.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient: _finish_auth 0 2024-09-16T14:53:41.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient: _check_auth_tickets 2024-09-16T14:53:41.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:41.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 1 -- 172.21.15.26:0/4067013859 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f89fc001980 con 0x7f8a0409d1a0 2024-09-16T14:53:41.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:41.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:53:41.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient: dump: 2024-09-16T14:53:41.900 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:41.900 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:41.900 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.900 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.900 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:41.900 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:41.900 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:41.900 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:41.900 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:41.900 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:41.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 5 monclient: authenticate success, global_id 194496 2024-09-16T14:53:41.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:41.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 1 -- 172.21.15.26:0/4067013859 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89fc001c30 con 0x7f8a0409d1a0 2024-09-16T14:53:41.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:41.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 1 -- 172.21.15.26:0/4067013859 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f89fc002000 con 0x7f8a0409d1a0 2024-09-16T14:53:41.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:41.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 4 set_mon_vals no callback set 2024-09-16T14:53:41.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:41.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:41.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f89fbfff640 10 monclient: dump: 2024-09-16T14:53:41.901 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:41.902 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:41.902 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.902 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.902 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:41.902 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:41.902 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:41.902 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:41.902 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:41.902 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:41.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 10 monclient: shutdown 2024-09-16T14:53:41.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:41.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 1 -- 172.21.15.26:0/4067013859 >> v1:172.21.15.26:6789/0 conn(0x7f8a0409d1a0 legacy=0x7f8a04139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:41.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 1 -- 172.21.15.26:0/4067013859 shutdown_connections 2024-09-16T14:53:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 1 -- 172.21.15.26:0/4067013859 >> 172.21.15.26:0/4067013859 conn(0x7f8a0409fb50 msgr2=0x7f8a0409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 1 -- 172.21.15.26:0/4067013859 shutdown_connections 2024-09-16T14:53:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 1 -- 172.21.15.26:0/4067013859 wait complete. 2024-09-16T14:53:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 10 monclient: build_initial_monmap 2024-09-16T14:53:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 10 monclient: monmap: 2024-09-16T14:53:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:41.903 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.887+0000 7f8a0b5ac640 1 Processor -- start 2024-09-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 1 -- start start 2024-09-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 10 monclient: init 2024-09-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 10 monclient(hunting): picked mon.noname-c con 0x7f8a0409cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 10 monclient(hunting): picked mon.noname-b con 0x7f8a0409d1a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 10 monclient(hunting): picked mon.noname-a con 0x7f8a0409e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a040a1560 con 0x7f8a0409e7a0 2024-09-16T14:53:41.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a04122080 con 0x7f8a0409d1a0 2024-09-16T14:53:41.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a09348640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8a0409cdc0 0x7f8a0412def0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60466/0 (socket says 172.21.15.26:60466) 2024-09-16T14:53:41.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a09348640 1 -- 172.21.15.26:0/2077219283 learned_addr learned my addr 172.21.15.26:0/2077219283 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:41.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 1 -- 172.21.15.26:0/2077219283 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0409a240 con 0x7f8a0409cdc0 2024-09-16T14:53:41.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:41.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 10 monclient(hunting): authenticate will time out at 1646.012635s 2024-09-16T14:53:41.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a08b47640 1 -- 172.21.15.26:0/2077219283 >> v1:172.21.15.189:6789/0 conn(0x7f8a0409d1a0 legacy=0x7f8a041ce530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:41.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478328419 0 0) 0x7f8a0409b2d0 con 0x7f8a0409e7a0 2024-09-16T14:53:41.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:41.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:41.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:41.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): my global_id is 194499 2024-09-16T14:53:41.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e8003530 con 0x7f8a0409e7a0 2024-09-16T14:53:41.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322287586 0 0) 0x7f8a0409a240 con 0x7f8a0409cdc0 2024-09-16T14:53:41.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:41.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:41.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:41.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): my global_id is 234185 2024-09-16T14:53:41.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a0409b2d0 con 0x7f8a0409cdc0 2024-09-16T14:53:41.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 338767924 0 0) 0x7f8a0409b2d0 con 0x7f8a0409cdc0 2024-09-16T14:53:41.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a0409a240 con 0x7f8a0409cdc0 2024-09-16T14:53:41.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8a00004190 con 0x7f8a0409cdc0 2024-09-16T14:53:41.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:41.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:53:41.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): dump: 2024-09-16T14:53:41.906 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:41.906 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:41.907 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.907 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.907 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:41.907 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:41.907 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:41.907 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:41.907 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:41.907 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:41.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:41.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3163785250 0 0) 0x7f89e8003530 con 0x7f8a0409e7a0 2024-09-16T14:53:41.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a0409b2d0 con 0x7f8a0409e7a0 2024-09-16T14:53:41.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f89fc0017a0 con 0x7f8a0409e7a0 2024-09-16T14:53:41.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:41.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:41.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): dump: 2024-09-16T14:53:41.908 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:41.908 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:41.908 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.908 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.908 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:41.908 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:41.908 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:41.908 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:41.908 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:41.908 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:41.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1365322430 0 0) 0x7f8a0409a240 con 0x7f8a0409cdc0 2024-09-16T14:53:41.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 >> v1:172.21.15.189:6789/0 conn(0x7f8a0409d1a0 legacy=0x7f8a041ce530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:53:41.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 >> v1:172.21.15.26:6789/0 conn(0x7f8a0409e7a0 legacy=0x7f8a041d1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:41.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:53:41.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 monclient: found mon.c 2024-09-16T14:53:41.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:41.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:41.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0412ff40 con 0x7f8a0409cdc0 2024-09-16T14:53:41.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient: _finish_auth 0 2024-09-16T14:53:41.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:53:41.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:41.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 5 monclient: authenticate success, global_id 234185 2024-09-16T14:53:41.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 10 monclient: _renew_subs 2024-09-16T14:53:41.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:41.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 1 -- 172.21.15.26:0/2077219283 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a040a0fc0 con 0x7f8a0409cdc0 2024-09-16T14:53:41.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 10 monclient: _renew_subs 2024-09-16T14:53:41.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:41.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 1 -- 172.21.15.26:0/2077219283 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a04130740 con 0x7f8a0409cdc0 2024-09-16T14:53:41.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a00004660 con 0x7f8a0409cdc0 2024-09-16T14:53:41.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:41.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8a000049f0 con 0x7f8a0409cdc0 2024-09-16T14:53:41.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:41.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:41.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient: dump: 2024-09-16T14:53:41.911 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:41.911 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:41.911 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.911 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.911 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:41.911 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:41.911 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:41.911 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:41.911 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:41.911 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:41.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89fb7fe640 4 set_mon_vals no callback set 2024-09-16T14:53:41.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:41.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a0b5ac640 1 -- 172.21.15.26:0/2077219283 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8a0402c7d0 con 0x7f8a0402c3f0 2024-09-16T14:53:41.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8a0000f460 con 0x7f8a0409cdc0 2024-09-16T14:53:41.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f8a00043cb0 con 0x7f8a0409cdc0 2024-09-16T14:53:41.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103102175 0 0) 0x7f8a0402c7d0 con 0x7f8a0402c3f0 2024-09-16T14:53:41.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:41.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:41.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:41.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient(hunting): my global_id is 194502 2024-09-16T14:53:41.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a0409a240 con 0x7f8a0402c3f0 2024-09-16T14:53:41.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f8a09b49640 10 monclient: get_auth_request con 0x7f89e8036230 auth_method 0 2024-09-16T14:53:41.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 125390415 0 0) 0x7f8a0409a240 con 0x7f8a0402c3f0 2024-09-16T14:53:41.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a0402c7d0 con 0x7f8a0402c3f0 2024-09-16T14:53:41.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f89f4001080 con 0x7f8a0402c3f0 2024-09-16T14:53:41.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:41.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:41.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient: dump: 2024-09-16T14:53:41.913 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:41.913 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:41.913 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.913 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.913 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:41.913 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:41.914 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:41.914 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:41.914 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:41.914 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:41.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 920633069 0 0) 0x7f8a0402c7d0 con 0x7f8a0402c3f0 2024-09-16T14:53:41.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a0409b890 con 0x7f8a0402c3f0 2024-09-16T14:53:41.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f89f40019e0 con 0x7f8a0402c3f0 2024-09-16T14:53:41.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:41.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:41.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 >> v1:172.21.15.26:6789/0 conn(0x7f8a0402c3f0 legacy=0x7f8a04061160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:41.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.891+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 >> v1:172.21.15.26:6789/0 conn(0x7f8a0402c3f0 legacy=0x7f8a04061160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:41.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f8a0b5ac640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:53:41.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f8a0b5ac640 1 -- 172.21.15.26:0/2077219283 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8a041c60e0 con 0x7f8a041c5cd0 2024-09-16T14:53:41.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540526617 0 0) 0x7f8a041c60e0 con 0x7f8a041c5cd0 2024-09-16T14:53:41.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:41.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:41.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:41.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 10 monclient(hunting): my global_id is 194505 2024-09-16T14:53:41.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a0402c7d0 con 0x7f8a041c5cd0 2024-09-16T14:53:41.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2721829042 0 0) 0x7f8a0402c7d0 con 0x7f8a041c5cd0 2024-09-16T14:53:41.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a041c60e0 con 0x7f8a041c5cd0 2024-09-16T14:53:41.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f89f4004a50 con 0x7f8a041c5cd0 2024-09-16T14:53:41.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 10 monclient: dump: 2024-09-16T14:53:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:41.917 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:41.917 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:41.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2819993694 0 0) 0x7f8a041c60e0 con 0x7f8a041c5cd0 2024-09-16T14:53:41.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8a04122530 con 0x7f8a041c5cd0 2024-09-16T14:53:41.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4201 (unknown 0 0 3179968250) 0x7f89f4004d40 con 0x7f8a041c5cd0 2024-09-16T14:53:41.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:53:41.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:41.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 >> v1:172.21.15.26:6789/0 conn(0x7f8a041c5cd0 legacy=0x7f8a040706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:41.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f89f9ffb640 1 -- 172.21.15.26:0/2077219283 >> v1:172.21.15.26:6789/0 conn(0x7f8a041c5cd0 legacy=0x7f8a040706d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:41.918 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:53:41.918 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:53:41.919 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:53:41.919 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:53:41.919 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 106, 2024-09-16T14:53:41.919 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:53:41.919 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:53:41.919 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:53:41.919 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:41.919 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:53:41.919 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:41.919 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:53:41.919 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:53:41.919 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:41.919 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:41.919 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:41.920 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:41.920 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:41.920 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:41.920 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:41.920 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:41.920 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:41.920 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:41.920 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:41.920 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:53:41.920 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:53:41.920 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:41.920 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:41.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:41.921 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:41.921 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:41.921 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:41.921 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:41.921 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:41.921 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:41.921 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:41.921 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:41.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:41.921 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:53:41.921 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:53:41.921 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:53:41.922 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:53:41.922 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:53:41.922 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:53:41.922 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:41.922 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:41.922 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:53:41.922 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:53:41.922 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:53:41.922 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:53:41.922 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:53:41.922 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:53:41.922 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:53:41.923 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:41.923 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:53:41.923 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:41.923 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:41.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:41.923 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:41.923 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:41.923 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:41.923 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:41.923 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:41.923 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:41.923 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:41.923 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:41.923 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:53:41.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:41.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:53:41.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:41.924 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:53:41.924 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:53:41.924 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:41.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:41.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:41.924 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:41.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:53:41.924 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:41.924 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:41.925 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:41.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:41.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:53:41.925 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:53:41.925 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:41.925 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:41.925 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:41.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:41.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:41.925 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:53:41.925 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:53:41.926 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:41.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:41.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:41.926 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:41.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:53:41.926 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:41.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:41.926 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:41.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:41.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:53:41.926 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:53:41.926 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:41.926 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:41.927 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:41.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:41.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:41.927 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:53:41.927 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:53:41.927 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:41.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:41.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:41.927 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:41.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:53:41.927 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:41.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:41.928 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:41.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:41.928 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:53:41.928 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:53:41.928 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:41.928 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:41.928 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:41.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:41.928 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:41.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:41.929 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:53:41.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:53:41.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:41.929 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:41.929 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:41.929 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:41.929 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:41.929 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:41.929 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:53:41.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:41.929 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:41.930 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:41.930 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:53:41.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:41.930 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:41.930 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:53:41.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:41.930 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:41.930 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:41.930 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:53:41.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:41.930 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:41.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:53:41.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:41.931 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:41.931 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:41.931 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:41.931 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:41.931 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:41.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:41.931 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:53:41.931 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:53:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f8a0b5ac640 1 -- 172.21.15.26:0/2077219283 >> v1:172.21.15.189:6812/3196161582 conn(0x7f89e8036230 legacy=0x7f89e80386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f8a0b5ac640 10 monclient: shutdown 2024-09-16T14:53:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f8a0b5ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f8a0b5ac640 1 -- 172.21.15.26:0/2077219283 >> v1:172.21.15.26:6790/0 conn(0x7f8a0409cdc0 legacy=0x7f8a0412def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f8a09b49640 1 -- 172.21.15.26:0/2077219283 reap_dead start 2024-09-16T14:53:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f8a09b49640 1 -- 172.21.15.26:0/2077219283 reap_dead start 2024-09-16T14:53:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f8a0b5ac640 1 -- 172.21.15.26:0/2077219283 shutdown_connections 2024-09-16T14:53:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f8a0b5ac640 1 -- 172.21.15.26:0/2077219283 >> 172.21.15.26:0/2077219283 conn(0x7f8a0409fb50 msgr2=0x7f8a0409bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f8a0b5ac640 1 -- 172.21.15.26:0/2077219283 shutdown_connections 2024-09-16T14:53:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:41.899+0000 7f8a0b5ac640 1 -- 172.21.15.26:0/2077219283 wait complete. 2024-09-16T14:53:41.934 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:53:41.934 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:53:41.934 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:53:42.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.047+0000 7fd0d94bd640 10 monclient: get_monmap_and_config 2024-09-16T14:53:42.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.047+0000 7fd0d94bd640 10 monclient: build_initial_monmap 2024-09-16T14:53:42.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.047+0000 7fd0d94bd640 10 monclient: monmap: 2024-09-16T14:53:42.052 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:42.052 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:42.052 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:53:42.053061+0000 2024-09-16T14:53:42.052 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:53:42.053061+0000 2024-09-16T14:53:42.052 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:42.052 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:42.052 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:42.052 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:53:42.052 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:53:42.053 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:42.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 1 Processor -- start 2024-09-16T14:53:42.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 1 -- start start 2024-09-16T14:53:42.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 10 monclient: init 2024-09-16T14:53:42.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:42.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:53:42.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 10 monclient(hunting): picked mon.noname-c con 0x7fd0d409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:42.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 10 monclient(hunting): picked mon.noname-b con 0x7fd0d4132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:42.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 10 monclient(hunting): picked mon.noname-a con 0x7fd0d4138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:42.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d409d260 con 0x7fd0d4138ff0 2024-09-16T14:53:42.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d4132a80 con 0x7fd0d409cb80 2024-09-16T14:53:42.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d409cf60 con 0x7fd0d4132650 2024-09-16T14:53:42.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:42.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d27fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd0d4132650 0x7fd0d41378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60480/0 (socket says 172.21.15.26:60480) 2024-09-16T14:53:42.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 10 monclient(hunting): authenticate will time out at 1646.173385s 2024-09-16T14:53:42.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d27fc640 1 -- 172.21.15.26:0/1902762542 learned_addr learned my addr 172.21.15.26:0/1902762542 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:42.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d2ffd640 1 -- 172.21.15.26:0/1902762542 >> v1:172.21.15.189:6789/0 conn(0x7fd0d409cb80 legacy=0x7fd0d40717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:42.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 1 -- 172.21.15.26:0/1902762542 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100491335 0 0) 0x7fd0d409cf60 con 0x7fd0d4132650 2024-09-16T14:53:42.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:42.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:42.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:42.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient(hunting): my global_id is 234188 2024-09-16T14:53:42.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 1 -- 172.21.15.26:0/1902762542 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0bc003530 con 0x7fd0d4132650 2024-09-16T14:53:42.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 1 -- 172.21.15.26:0/1902762542 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590641400 0 0) 0x7fd0d409d260 con 0x7fd0d4138ff0 2024-09-16T14:53:42.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:53:42.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:42.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:42.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient(hunting): my global_id is 194508 2024-09-16T14:53:42.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 1 -- 172.21.15.26:0/1902762542 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d409cf60 con 0x7fd0d4138ff0 2024-09-16T14:53:42.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 1 -- 172.21.15.26:0/1902762542 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 202586175 0 0) 0x7fd0bc003530 con 0x7fd0d4132650 2024-09-16T14:53:42.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 1 -- 172.21.15.26:0/1902762542 >> v1:172.21.15.189:6789/0 conn(0x7fd0d409cb80 legacy=0x7fd0d40717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:53:42.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 1 -- 172.21.15.26:0/1902762542 >> v1:172.21.15.26:6789/0 conn(0x7fd0d4138ff0 legacy=0x7fd0d413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:42.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:53:42.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 1 monclient: found mon.noname-b 2024-09-16T14:53:42.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:42.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:53:42.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 1 -- 172.21.15.26:0/1902762542 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d409f2d0 con 0x7fd0d4132650 2024-09-16T14:53:42.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:53:42.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:53:42.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:42.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 1 -- 172.21.15.26:0/1902762542 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd0c8001660 con 0x7fd0d4132650 2024-09-16T14:53:42.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:42.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:53:42.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient: dump: 2024-09-16T14:53:42.063 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:42.063 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:42.063 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:42.064 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:42.064 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:42.064 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:42.064 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:42.064 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:42.064 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:42.064 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:42.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 5 monclient: authenticate success, global_id 234188 2024-09-16T14:53:42.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:53:42.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 1 -- 172.21.15.26:0/1902762542 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0c8001800 con 0x7fd0d4132650 2024-09-16T14:53:42.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:42.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 1 -- 172.21.15.26:0/1902762542 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd0c8001bd0 con 0x7fd0d4132650 2024-09-16T14:53:42.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 10 monclient: get_monmap_and_config success 2024-09-16T14:53:42.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 4 set_mon_vals no callback set 2024-09-16T14:53:42.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:42.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:42.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d1ffb640 10 monclient: dump: 2024-09-16T14:53:42.066 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:42.066 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:42.066 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:42.066 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:42.066 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:42.066 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:42.066 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:42.066 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:42.066 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:42.066 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:42.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 10 monclient: shutdown 2024-09-16T14:53:42.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:42.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 1 -- 172.21.15.26:0/1902762542 >> v1:172.21.15.26:6790/0 conn(0x7fd0d4132650 legacy=0x7fd0d41378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:42.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 1 -- 172.21.15.26:0/1902762542 shutdown_connections 2024-09-16T14:53:42.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 1 -- 172.21.15.26:0/1902762542 >> 172.21.15.26:0/1902762542 conn(0x7fd0d409fb50 msgr2=0x7fd0d409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:42.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 1 -- 172.21.15.26:0/1902762542 shutdown_connections 2024-09-16T14:53:42.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 1 -- 172.21.15.26:0/1902762542 wait complete. 2024-09-16T14:53:42.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 10 monclient: build_initial_monmap 2024-09-16T14:53:42.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 10 monclient: monmap: 2024-09-16T14:53:42.067 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:53:42.067 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:53:42.067 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:53:42.067 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:53:42.067 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:53:42.067 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:53:42.068 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:53:42.068 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:53:42.068 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:53:42.068 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:42.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 1 Processor -- start 2024-09-16T14:53:42.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 1 -- start start 2024-09-16T14:53:42.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.051+0000 7fd0d94bd640 10 monclient: init 2024-09-16T14:53:42.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 10 monclient: _reopen_session rank -1 2024-09-16T14:53:42.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:53:42.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 10 monclient(hunting): picked mon.noname-b con 0x7fd0d409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:53:42.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 10 monclient(hunting): picked mon.noname-c con 0x7fd0d4132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:53:42.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 10 monclient(hunting): picked mon.noname-a con 0x7fd0d4138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:53:42.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d4134ca0 con 0x7fd0d4138ff0 2024-09-16T14:53:42.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d40a2160 con 0x7fd0d409cb80 2024-09-16T14:53:42.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d409a240 con 0x7fd0d4132650 2024-09-16T14:53:42.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 10 monclient(hunting): _renew_subs 2024-09-16T14:53:42.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 10 monclient(hunting): authenticate will time out at 1646.176355s 2024-09-16T14:53:42.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d37fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd0d4138ff0 0x7fd0d41cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35212/0 (socket says 172.21.15.26:35212) 2024-09-16T14:53:42.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d2ffd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fd0d409cb80 legacy=0x7fd0d4133e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:53:42.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d27fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd0d4132650 0x7fd0d4134590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60490/0 (socket says 172.21.15.26:60490) 2024-09-16T14:53:42.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d37fe640 1 -- 172.21.15.26:0/3834721309 learned_addr learned my addr 172.21.15.26:0/3834721309 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:53:42.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819771664 0 0) 0x7fd0d4134ca0 con 0x7fd0d4138ff0 2024-09-16T14:53:42.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:53:42.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:42.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:42.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): my global_id is 194511 2024-09-16T14:53:42.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0c0003530 con 0x7fd0d4138ff0 2024-09-16T14:53:42.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288373311 0 0) 0x7fd0d409a240 con 0x7fd0d4132650 2024-09-16T14:53:42.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:53:42.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:42.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:42.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): my global_id is 234191 2024-09-16T14:53:42.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d4134ca0 con 0x7fd0d4132650 2024-09-16T14:53:42.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3571804907 0 0) 0x7fd0d4134ca0 con 0x7fd0d4132650 2024-09-16T14:53:42.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0d409a240 con 0x7fd0d4132650 2024-09-16T14:53:42.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd0c8001970 con 0x7fd0d4132650 2024-09-16T14:53:42.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:42.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:53:42.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): dump: 2024-09-16T14:53:42.071 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:42.071 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:42.071 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:42.071 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:42.071 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:42.072 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:42.072 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:42.072 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:42.072 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:42.072 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:42.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:53:42.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 518161813 0 0) 0x7fd0c0003530 con 0x7fd0d4138ff0 2024-09-16T14:53:42.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0d4134ca0 con 0x7fd0d4138ff0 2024-09-16T14:53:42.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd0cc001700 con 0x7fd0d4138ff0 2024-09-16T14:53:42.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:42.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:53:42.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): dump: 2024-09-16T14:53:42.073 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:42.073 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:42.073 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:42.073 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:42.073 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:42.073 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:42.073 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:42.073 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:42.074 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:42.074 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:42.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1823458295 0 0) 0x7fd0d409a240 con 0x7fd0d4132650 2024-09-16T14:53:42.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 >> v1:172.21.15.189:6789/0 conn(0x7fd0d409cb80 legacy=0x7fd0d4133e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:53:42.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 >> v1:172.21.15.26:6789/0 conn(0x7fd0d4138ff0 legacy=0x7fd0d41cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:42.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:53:42.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 monclient: found mon.c 2024-09-16T14:53:42.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:53:42.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:42.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d4130120 con 0x7fd0d4132650 2024-09-16T14:53:42.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient: _finish_auth 0 2024-09-16T14:53:42.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:53:42.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:53:42.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 5 monclient: authenticate success, global_id 234191 2024-09-16T14:53:42.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 10 monclient: _renew_subs 2024-09-16T14:53:42.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:42.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 1 -- 172.21.15.26:0/3834721309 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d4137d60 con 0x7fd0d4132650 2024-09-16T14:53:42.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 10 monclient: _renew_subs 2024-09-16T14:53:42.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:53:42.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 1 -- 172.21.15.26:0/3834721309 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d4135990 con 0x7fd0d4132650 2024-09-16T14:53:42.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0c8001c70 con 0x7fd0d4132650 2024-09-16T14:53:42.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:53:42.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd0c8004c40 con 0x7fd0d4132650 2024-09-16T14:53:42.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:42.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:42.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d0ff9640 4 set_mon_vals no callback set 2024-09-16T14:53:42.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient: dump: 2024-09-16T14:53:42.076 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:42.076 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:42.076 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:42.076 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:42.076 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:42.076 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:42.076 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:42.076 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:42.076 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:42.076 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:42.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:42.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd0c800f6b0 con 0x7fd0d4132650 2024-09-16T14:53:42.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d94bd640 1 -- 172.21.15.26:0/3834721309 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd0b4006590 con 0x7fd0b4002830 2024-09-16T14:53:42.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fd0c8043d40 con 0x7fd0d4132650 2024-09-16T14:53:42.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519424506 0 0) 0x7fd0b4006590 con 0x7fd0b4002830 2024-09-16T14:53:42.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:53:42.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:42.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:42.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient(hunting): my global_id is 234194 2024-09-16T14:53:42.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d409a240 con 0x7fd0b4002830 2024-09-16T14:53:42.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3758714510 0 0) 0x7fd0d409a240 con 0x7fd0b4002830 2024-09-16T14:53:42.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0b4006590 con 0x7fd0b4002830 2024-09-16T14:53:42.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd0c4003fc0 con 0x7fd0b4002830 2024-09-16T14:53:42.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:42.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:42.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient: dump: 2024-09-16T14:53:42.078 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:42.078 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:42.078 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:42.078 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:42.078 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:42.078 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:42.078 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:42.078 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:42.079 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:42.079 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:42.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0d37fe640 10 monclient: get_auth_request con 0x7fd0c0036310 auth_method 0 2024-09-16T14:53:42.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3088790871 0 0) 0x7fd0b4006590 con 0x7fd0b4002830 2024-09-16T14:53:42.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd0b400a0e0 con 0x7fd0b4002830 2024-09-16T14:53:42.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fd0c40063d0 con 0x7fd0b4002830 2024-09-16T14:53:42.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:53:42.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:53:42.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 >> v1:172.21.15.26:6790/0 conn(0x7fd0b4002830 legacy=0x7fd0b4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:42.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.055+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 >> v1:172.21.15.26:6790/0 conn(0x7fd0b4002830 legacy=0x7fd0b4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:42.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0d94bd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:53:42.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0d94bd640 1 -- 172.21.15.26:0/3834721309 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd0b4009da0 con 0x7fd0b40090c0 2024-09-16T14:53:42.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581719769 0 0) 0x7fd0b4009da0 con 0x7fd0b40090c0 2024-09-16T14:53:42.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:53:42.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:53:42.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:53:42.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 10 monclient(hunting): my global_id is 234197 2024-09-16T14:53:42.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0b4006590 con 0x7fd0b40090c0 2024-09-16T14:53:42.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 350505037 0 0) 0x7fd0b4006590 con 0x7fd0b40090c0 2024-09-16T14:53:42.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0b4009da0 con 0x7fd0b40090c0 2024-09-16T14:53:42.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd0c40048a0 con 0x7fd0b40090c0 2024-09-16T14:53:42.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:53:42.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:53:42.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 10 monclient: dump: 2024-09-16T14:53:42.082 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:53:42.082 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:53:42.082 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:42.082 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:53:42.082 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:53:42.082 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:53:42.082 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:53:42.082 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:53:42.082 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:53:42.082 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:53:42.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3684774693 0 0) 0x7fd0b4009da0 con 0x7fd0b40090c0 2024-09-16T14:53:42.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd0b400c290 con 0x7fd0b40090c0 2024-09-16T14:53:42.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3871 (unknown 0 0 1389791567) 0x7fd0c4005cc0 con 0x7fd0b40090c0 2024-09-16T14:53:42.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:53:42.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:53:42.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 >> v1:172.21.15.26:6790/0 conn(0x7fd0b40090c0 legacy=0x7fd0b4009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:42.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0bb7fe640 1 -- 172.21.15.26:0/3834721309 >> v1:172.21.15.26:6790/0 conn(0x7fd0b40090c0 legacy=0x7fd0b4009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:53:42.083 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:53:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:53:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:53:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:53:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 106, 2024-09-16T14:53:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:53:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:53:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:53:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:53:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:53:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:53:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:53:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:53:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:53:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:53:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:53:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:53:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:53:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:53:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:53:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:53:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:53:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:53:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:53:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:53:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:53:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:53:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:53:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:53:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:53:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:53:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:53:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:53:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:53:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:53:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:53:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:53:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:53:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:53:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:53:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:53:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:53:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:53:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:53:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:53:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:53:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:53:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:53:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:53:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:53:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:53:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:53:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:53:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:53:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:53:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:53:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:53:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:53:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:53:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:53:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:53:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:53:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:53:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:53:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:53:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:53:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:53:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:53:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:53:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:53:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:53:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:53:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:53:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:53:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:53:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:53:42.094 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:53:42.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0d94bd640 1 -- 172.21.15.26:0/3834721309 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd0c0036310 legacy=0x7fd0c00387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:42.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0d94bd640 10 monclient: shutdown 2024-09-16T14:53:42.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0d94bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:53:42.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0d94bd640 1 -- 172.21.15.26:0/3834721309 >> v1:172.21.15.26:6790/0 conn(0x7fd0d4132650 legacy=0x7fd0d4134590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:53:42.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0d37fe640 1 -- 172.21.15.26:0/3834721309 reap_dead start 2024-09-16T14:53:42.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0d37fe640 1 -- 172.21.15.26:0/3834721309 reap_dead start 2024-09-16T14:53:42.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0d94bd640 1 -- 172.21.15.26:0/3834721309 shutdown_connections 2024-09-16T14:53:42.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0d94bd640 1 -- 172.21.15.26:0/3834721309 >> 172.21.15.26:0/3834721309 conn(0x7fd0d409fb50 msgr2=0x7fd0d4130330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:53:42.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0d94bd640 1 -- 172.21.15.26:0/3834721309 shutdown_connections 2024-09-16T14:53:42.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:53:42.063+0000 7fd0d94bd640 1 -- 172.21.15.26:0/3834721309 wait complete. 2024-09-16T14:53:42.097 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:53:42.097 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:54:02.097 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:54:02.098 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T14:54:02.098 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T14:54:02.098 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T14:54:02.102 INFO:tasks.ceph.mon.b:Started 2024-09-16T14:54:02.102 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:54:02.102 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:54:02.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f30c1640 10 monclient: get_monmap_and_config 2024-09-16T14:54:02.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f30c1640 10 monclient: build_initial_monmap 2024-09-16T14:54:02.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f30c1640 10 monclient: monmap: 2024-09-16T14:54:02.234 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:02.234 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.234 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:02.235586+0000 2024-09-16T14:54:02.234 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:02.235586+0000 2024-09-16T14:54:02.234 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:02.235 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.235 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:02.235 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:02.235 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:02.235 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f30c1640 1 Processor -- start 2024-09-16T14:54:02.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f30c1640 1 -- start start 2024-09-16T14:54:02.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f30c1640 10 monclient: init 2024-09-16T14:54:02.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f30c1640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:02.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f30c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:54:02.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f30c1640 10 monclient(hunting): picked mon.noname-c con 0x7f24ec09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:02.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f30c1640 10 monclient(hunting): picked mon.noname-a con 0x7f24ec132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:02.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f30c1640 10 monclient(hunting): picked mon.noname-b con 0x7f24ec138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:02.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f30c1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ec09d260 con 0x7f24ec132650 2024-09-16T14:54:02.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f30c1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ec132a80 con 0x7f24ec09cb80 2024-09-16T14:54:02.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f30c1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ec09cf60 con 0x7f24ec138ff0 2024-09-16T14:54:02.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f30c1640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:02.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f30c1640 10 monclient(hunting): authenticate will time out at 1666.355615s 2024-09-16T14:54:02.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f165e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f24ec138ff0 0x7f24ec13b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47636/0 (socket says 172.21.15.26:47636) 2024-09-16T14:54:02.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.230+0000 7f24f165e640 1 -- 172.21.15.26:0/2655917044 learned_addr learned my addr 172.21.15.26:0/2655917044 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:02.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 1 -- 172.21.15.26:0/2655917044 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977990544 0 0) 0x7f24ec09cf60 con 0x7f24ec138ff0 2024-09-16T14:54:02.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 10 monclient(hunting): my global_id is 234200 2024-09-16T14:54:02.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 1 -- 172.21.15.26:0/2655917044 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d4003530 con 0x7f24ec138ff0 2024-09-16T14:54:02.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 1 -- 172.21.15.26:0/2655917044 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 999959478 0 0) 0x7f24d4003530 con 0x7f24ec138ff0 2024-09-16T14:54:02.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 1 -- 172.21.15.26:0/2655917044 >> v1:172.21.15.189:6789/0 conn(0x7f24ec09cb80 legacy=0x7f24ec0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 1 -- 172.21.15.26:0/2655917044 >> v1:172.21.15.26:6789/0 conn(0x7f24ec132650 legacy=0x7f24ec1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:54:02.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 1 monclient: found mon.noname-b 2024-09-16T14:54:02.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:02.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:54:02.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 1 -- 172.21.15.26:0/2655917044 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec09f2d0 con 0x7f24ec138ff0 2024-09-16T14:54:02.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 10 monclient: _finish_auth 0 2024-09-16T14:54:02.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:54:02.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:02.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 1 -- 172.21.15.26:0/2655917044 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24e40019e0 con 0x7f24ec138ff0 2024-09-16T14:54:02.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 5 monclient: authenticate success, global_id 234200 2024-09-16T14:54:02.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:02.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:54:02.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 10 monclient: dump: 2024-09-16T14:54:02.247 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.247 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.247 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.248 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.248 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.248 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.248 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.248 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.248 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.248 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:02.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 1 -- 172.21.15.26:0/2655917044 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24e4001c40 con 0x7f24ec138ff0 2024-09-16T14:54:02.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:02.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 1 -- 172.21.15.26:0/2655917044 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24e4001f90 con 0x7f24ec138ff0 2024-09-16T14:54:02.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:02.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eb7fe640 10 monclient: dump: 2024-09-16T14:54:02.249 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.249 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.249 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.249 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.249 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.249 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.249 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.249 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.249 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.249 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:02.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 4 set_mon_vals no callback set 2024-09-16T14:54:02.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 10 monclient: shutdown 2024-09-16T14:54:02.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:02.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 1 -- 172.21.15.26:0/2655917044 >> v1:172.21.15.26:6790/0 conn(0x7f24ec138ff0 legacy=0x7f24ec13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 1 -- 172.21.15.26:0/2655917044 shutdown_connections 2024-09-16T14:54:02.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 1 -- 172.21.15.26:0/2655917044 >> 172.21.15.26:0/2655917044 conn(0x7f24ec09fb50 msgr2=0x7f24ec09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:02.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 1 -- 172.21.15.26:0/2655917044 shutdown_connections 2024-09-16T14:54:02.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 1 -- 172.21.15.26:0/2655917044 wait complete. 2024-09-16T14:54:02.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 10 monclient: build_initial_monmap 2024-09-16T14:54:02.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 10 monclient: monmap: 2024-09-16T14:54:02.250 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:02.250 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:02.250 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:02.251 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 1 Processor -- start 2024-09-16T14:54:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 1 -- start start 2024-09-16T14:54:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 10 monclient: init 2024-09-16T14:54:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:54:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 10 monclient(hunting): picked mon.noname-c con 0x7f24ec09cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:02.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 10 monclient(hunting): picked mon.noname-a con 0x7f24ec132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:02.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 10 monclient(hunting): picked mon.noname-b con 0x7f24ec138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:02.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ec09a240 con 0x7f24ec132650 2024-09-16T14:54:02.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ec071680 con 0x7f24ec138ff0 2024-09-16T14:54:02.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ec130330 con 0x7f24ec09cb80 2024-09-16T14:54:02.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:02.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 10 monclient(hunting): authenticate will time out at 1666.358721s 2024-09-16T14:54:02.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f0e5d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f24ec09cb80 0x7f24ec06d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47652/0 (socket says 172.21.15.26:47652) 2024-09-16T14:54:02.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f0e5d640 1 -- 172.21.15.26:0/1566531292 learned_addr learned my addr 172.21.15.26:0/1566531292 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:02.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 1 -- 172.21.15.26:0/1566531292 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662416944 0 0) 0x7f24ec130330 con 0x7f24ec09cb80 2024-09-16T14:54:02.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 10 monclient(hunting): my global_id is 234203 2024-09-16T14:54:02.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 1 -- 172.21.15.26:0/1566531292 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d8003530 con 0x7f24ec09cb80 2024-09-16T14:54:02.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 1 -- 172.21.15.26:0/1566531292 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 50272613 0 0) 0x7f24d8003530 con 0x7f24ec09cb80 2024-09-16T14:54:02.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 1 -- 172.21.15.26:0/1566531292 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24ec130330 con 0x7f24ec09cb80 2024-09-16T14:54:02.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 1 -- 172.21.15.26:0/1566531292 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24dc0018e0 con 0x7f24ec09cb80 2024-09-16T14:54:02.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:02.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 10 monclient(hunting): dump: 2024-09-16T14:54:02.253 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.253 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.253 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.253 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.253 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.253 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.254 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.254 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.254 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.254 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:02.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 1 -- 172.21.15.26:0/1566531292 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4090198198 0 0) 0x7f24ec130330 con 0x7f24ec09cb80 2024-09-16T14:54:02.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 1 -- 172.21.15.26:0/1566531292 >> v1:172.21.15.189:6789/0 conn(0x7f24ec138ff0 legacy=0x7f24ec06e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 1 -- 172.21.15.26:0/1566531292 >> v1:172.21.15.26:6789/0 conn(0x7f24ec132650 legacy=0x7f24ec06e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 10 monclient: _finish_hunting 0 2024-09-16T14:54:02.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 1 monclient: found mon.c 2024-09-16T14:54:02.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:02.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:02.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 1 -- 172.21.15.26:0/1566531292 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec131f40 con 0x7f24ec09cb80 2024-09-16T14:54:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 10 monclient: _finish_auth 0 2024-09-16T14:54:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 10 monclient: _check_auth_tickets 2024-09-16T14:54:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 5 monclient: authenticate success, global_id 234203 2024-09-16T14:54:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 10 monclient: _renew_subs 2024-09-16T14:54:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 1 -- 172.21.15.26:0/1566531292 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ec133700 con 0x7f24ec09cb80 2024-09-16T14:54:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 10 monclient: _renew_subs 2024-09-16T14:54:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24f30c1640 1 -- 172.21.15.26:0/1566531292 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ec133b20 con 0x7f24ec09cb80 2024-09-16T14:54:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 1 -- 172.21.15.26:0/1566531292 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24dc001cb0 con 0x7f24ec09cb80 2024-09-16T14:54:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:02.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 1 -- 172.21.15.26:0/1566531292 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f24dc003a70 con 0x7f24ec09cb80 2024-09-16T14:54:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24eaffd640 4 set_mon_vals no callback set 2024-09-16T14:54:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.234+0000 7f24e97fa640 10 monclient: dump: 2024-09-16T14:54:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.256 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.257 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.238+0000 7f24e97fa640 1 -- 172.21.15.26:0/1566531292 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f24dc003c50 con 0x7f24ec09cb80 2024-09-16T14:54:02.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.238+0000 7f24f30c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:02.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.238+0000 7f24f30c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:02.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.238+0000 7f24f30c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:02.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.238+0000 7f24f30c1640 1 -- 172.21.15.26:0/1566531292 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24cc005050 con 0x7f24ec09cb80 2024-09-16T14:54:02.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.238+0000 7f24e97fa640 1 -- 172.21.15.26:0/1566531292 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f24dc043e80 con 0x7f24ec09cb80 2024-09-16T14:54:02.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.238+0000 7f24ebfff640 10 monclient: get_auth_request con 0x7f24d8036060 auth_method 0 2024-09-16T14:54:02.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.242+0000 7f24e97fa640 1 -- 172.21.15.26:0/1566531292 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f24dc0147c0 con 0x7f24ec09cb80 2024-09-16T14:54:02.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.242+0000 7f24e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:02.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.242+0000 7f24e97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:02.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.462+0000 7f24f30c1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:54:02.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.462+0000 7f24f30c1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:02.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.462+0000 7f24f30c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:02.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.462+0000 7f24f30c1640 1 -- 172.21.15.26:0/1566531292 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f24cc005880 con 0x7f24ec09cb80 2024-09-16T14:54:02.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.462+0000 7f24e97fa640 1 -- 172.21.15.26:0/1566531292 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 4067618410) 0x7f24dc012c30 con 0x7f24ec09cb80 2024-09-16T14:54:02.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.462+0000 7f24e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:02.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.462+0000 7f24e97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:02.465 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:54:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.462+0000 7f24f30c1640 1 -- 172.21.15.26:0/1566531292 >> v1:172.21.15.189:6812/3196161582 conn(0x7f24d8036060 legacy=0x7f24d8038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.462+0000 7f24f30c1640 10 monclient: shutdown 2024-09-16T14:54:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.462+0000 7f24f30c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.462+0000 7f24f30c1640 1 -- 172.21.15.26:0/1566531292 >> v1:172.21.15.26:6790/0 conn(0x7f24ec09cb80 legacy=0x7f24ec06d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.462+0000 7f24f30c1640 1 -- 172.21.15.26:0/1566531292 shutdown_connections 2024-09-16T14:54:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.462+0000 7f24f30c1640 1 -- 172.21.15.26:0/1566531292 >> 172.21.15.26:0/1566531292 conn(0x7f24ec09fb50 msgr2=0x7f24ec13b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.462+0000 7f24f30c1640 1 -- 172.21.15.26:0/1566531292 shutdown_connections 2024-09-16T14:54:02.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.462+0000 7f24f30c1640 1 -- 172.21.15.26:0/1566531292 wait complete. 2024-09-16T14:54:02.483 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:54:02.483 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:54:02.483 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:54:02.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.602+0000 7fddc53e6640 10 monclient: get_monmap_and_config 2024-09-16T14:54:02.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.602+0000 7fddc53e6640 10 monclient: build_initial_monmap 2024-09-16T14:54:02.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.602+0000 7fddc53e6640 10 monclient: monmap: 2024-09-16T14:54:02.607 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:02.608999+0000 2024-09-16T14:54:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:02.608999+0000 2024-09-16T14:54:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:02.608 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 1 Processor -- start 2024-09-16T14:54:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 1 -- start start 2024-09-16T14:54:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 10 monclient: init 2024-09-16T14:54:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:54:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 10 monclient(hunting): picked mon.noname-c con 0x7fddc009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 10 monclient(hunting): picked mon.noname-b con 0x7fddc0132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 10 monclient(hunting): picked mon.noname-a con 0x7fddc0138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc009d260 con 0x7fddc0138ff0 2024-09-16T14:54:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc0132a80 con 0x7fddc009cb80 2024-09-16T14:54:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc009cf60 con 0x7fddc0132650 2024-09-16T14:54:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 10 monclient(hunting): authenticate will time out at 1666.729111s 2024-09-16T14:54:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbe7fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fddc0132650 0x7fddc01378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47664/0 (socket says 172.21.15.26:47664) 2024-09-16T14:54:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbe7fc640 1 -- 172.21.15.26:0/2153563002 learned_addr learned my addr 172.21.15.26:0/2153563002 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbf7fe640 1 --1- 172.21.15.26:0/2153563002 >> v1:172.21.15.26:6789/0 conn(0x7fddc0138ff0 0x7fddc013b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53896/0 (socket says 172.21.15.26:53896) 2024-09-16T14:54:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 1 -- 172.21.15.26:0/2153563002 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522904466 0 0) 0x7fddc009d260 con 0x7fddc0138ff0 2024-09-16T14:54:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient(hunting): my global_id is 194514 2024-09-16T14:54:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 1 -- 172.21.15.26:0/2153563002 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda8003530 con 0x7fddc0138ff0 2024-09-16T14:54:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 1 -- 172.21.15.26:0/2153563002 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845980864 0 0) 0x7fddc009cf60 con 0x7fddc0132650 2024-09-16T14:54:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient(hunting): my global_id is 234206 2024-09-16T14:54:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 1 -- 172.21.15.26:0/2153563002 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc009d260 con 0x7fddc0132650 2024-09-16T14:54:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 1 -- 172.21.15.26:0/2153563002 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3817985489 0 0) 0x7fdda8003530 con 0x7fddc0138ff0 2024-09-16T14:54:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 1 -- 172.21.15.26:0/2153563002 >> v1:172.21.15.26:6790/0 conn(0x7fddc0132650 legacy=0x7fddc01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 1 -- 172.21.15.26:0/2153563002 >> v1:172.21.15.189:6789/0 conn(0x7fddc009cb80 legacy=0x7fddc00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient: _finish_hunting 0 2024-09-16T14:54:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 1 monclient: found mon.noname-a 2024-09-16T14:54:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:54:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 1 -- 172.21.15.26:0/2153563002 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc009f2d0 con 0x7fddc0138ff0 2024-09-16T14:54:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient: _finish_auth 0 2024-09-16T14:54:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient: _check_auth_tickets 2024-09-16T14:54:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 1 -- 172.21.15.26:0/2153563002 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fddb00019a0 con 0x7fddc0138ff0 2024-09-16T14:54:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 5 monclient: authenticate success, global_id 194514 2024-09-16T14:54:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient: dump: 2024-09-16T14:54:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.621 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 1 -- 172.21.15.26:0/2153563002 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddb0001bc0 con 0x7fddc0138ff0 2024-09-16T14:54:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 1 -- 172.21.15.26:0/2153563002 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fddb0001ed0 con 0x7fddc0138ff0 2024-09-16T14:54:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddbdffb640 10 monclient: dump: 2024-09-16T14:54:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.622 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 4 set_mon_vals no callback set 2024-09-16T14:54:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 10 monclient: shutdown 2024-09-16T14:54:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 1 -- 172.21.15.26:0/2153563002 >> v1:172.21.15.26:6789/0 conn(0x7fddc0138ff0 legacy=0x7fddc013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 1 -- 172.21.15.26:0/2153563002 shutdown_connections 2024-09-16T14:54:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 1 -- 172.21.15.26:0/2153563002 >> 172.21.15.26:0/2153563002 conn(0x7fddc009fb50 msgr2=0x7fddc009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 1 -- 172.21.15.26:0/2153563002 shutdown_connections 2024-09-16T14:54:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 1 -- 172.21.15.26:0/2153563002 wait complete. 2024-09-16T14:54:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 10 monclient: build_initial_monmap 2024-09-16T14:54:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 10 monclient: monmap: 2024-09-16T14:54:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:02.624 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 1 Processor -- start 2024-09-16T14:54:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 1 -- start start 2024-09-16T14:54:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.606+0000 7fddc53e6640 10 monclient: init 2024-09-16T14:54:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:54:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 10 monclient(hunting): picked mon.noname-c con 0x7fddc009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 10 monclient(hunting): picked mon.noname-a con 0x7fddc0132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 10 monclient(hunting): picked mon.noname-b con 0x7fddc0138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc01381a0 con 0x7fddc0132650 2024-09-16T14:54:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc00a2960 con 0x7fddc0138ff0 2024-09-16T14:54:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc006e4b0 con 0x7fddc009cb80 2024-09-16T14:54:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 10 monclient(hunting): authenticate will time out at 1666.732413s 2024-09-16T14:54:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddbeffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fddc009cb80 0x7fddc006d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47674/0 (socket says 172.21.15.26:47674) 2024-09-16T14:54:02.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddbeffd640 1 -- 172.21.15.26:0/3183220045 learned_addr learned my addr 172.21.15.26:0/3183220045 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:02.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379439954 0 0) 0x7fddc006e4b0 con 0x7fddc009cb80 2024-09-16T14:54:02.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient(hunting): my global_id is 234209 2024-09-16T14:54:02.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddac003530 con 0x7fddc009cb80 2024-09-16T14:54:02.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4062897347 0 0) 0x7fddac003530 con 0x7fddc009cb80 2024-09-16T14:54:02.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddc006e4b0 con 0x7fddc009cb80 2024-09-16T14:54:02.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fddb80018e0 con 0x7fddc009cb80 2024-09-16T14:54:02.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:02.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient(hunting): dump: 2024-09-16T14:54:02.627 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.627 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.627 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.627 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.627 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.627 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.627 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.627 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.627 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.628 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:02.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221721357 0 0) 0x7fddb4000ff0 con 0x7fddc0132650 2024-09-16T14:54:02.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient(hunting): my global_id is 194517 2024-09-16T14:54:02.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddac003530 con 0x7fddc0132650 2024-09-16T14:54:02.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2799737352 0 0) 0x7fddc006e4b0 con 0x7fddc009cb80 2024-09-16T14:54:02.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 >> v1:172.21.15.189:6789/0 conn(0x7fddc0138ff0 legacy=0x7fddc01cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 >> v1:172.21.15.26:6789/0 conn(0x7fddc0132650 legacy=0x7fddc006dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:54:02.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 monclient: found mon.c 2024-09-16T14:54:02.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:02.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:02.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc00700d0 con 0x7fddc009cb80 2024-09-16T14:54:02.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient: _finish_auth 0 2024-09-16T14:54:02.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:54:02.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:02.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 5 monclient: authenticate success, global_id 234209 2024-09-16T14:54:02.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 10 monclient: _renew_subs 2024-09-16T14:54:02.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:02.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 1 -- 172.21.15.26:0/3183220045 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc006ff10 con 0x7fddc009cb80 2024-09-16T14:54:02.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 10 monclient: _renew_subs 2024-09-16T14:54:02.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:02.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 1 -- 172.21.15.26:0/3183220045 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc006f2e0 con 0x7fddc009cb80 2024-09-16T14:54:02.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddb8001cb0 con 0x7fddc009cb80 2024-09-16T14:54:02.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:02.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fddb8003950 con 0x7fddc009cb80 2024-09-16T14:54:02.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:02.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient: dump: 2024-09-16T14:54:02.630 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.630 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.631 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.631 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.631 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.631 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.631 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.631 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.631 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.631 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddbd7fa640 4 set_mon_vals no callback set 2024-09-16T14:54:02.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fddb8003ab0 con 0x7fddc009cb80 2024-09-16T14:54:02.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddc53e6640 1 -- 172.21.15.26:0/3183220045 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdda0006590 con 0x7fdda0002830 2024-09-16T14:54:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fddb8043c90 con 0x7fddc009cb80 2024-09-16T14:54:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809876764 0 0) 0x7fdda0006590 con 0x7fdda0002830 2024-09-16T14:54:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient(hunting): my global_id is 194520 2024-09-16T14:54:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc006e4b0 con 0x7fdda0002830 2024-09-16T14:54:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fddbf7fe640 10 monclient: get_auth_request con 0x7fddac0361b0 auth_method 0 2024-09-16T14:54:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1550285350 0 0) 0x7fddc006e4b0 con 0x7fdda0002830 2024-09-16T14:54:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdda0006590 con 0x7fdda0002830 2024-09-16T14:54:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fddb4001600 con 0x7fdda0002830 2024-09-16T14:54:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient: dump: 2024-09-16T14:54:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.634 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3518374239 0 0) 0x7fdda0006590 con 0x7fdda0002830 2024-09-16T14:54:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdda000a0e0 con 0x7fdda0002830 2024-09-16T14:54:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fddb4001cc0 con 0x7fdda0002830 2024-09-16T14:54:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 >> v1:172.21.15.26:6789/0 conn(0x7fdda0002830 legacy=0x7fdda0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.610+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 >> v1:172.21.15.26:6789/0 conn(0x7fdda0002830 legacy=0x7fdda0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:02.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fddc53e6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:54:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fddc53e6640 1 -- 172.21.15.26:0/3183220045 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdda0009da0 con 0x7fdda00090c0 2024-09-16T14:54:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125371006 0 0) 0x7fdda0009da0 con 0x7fdda00090c0 2024-09-16T14:54:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 10 monclient(hunting): my global_id is 194523 2024-09-16T14:54:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda0006590 con 0x7fdda00090c0 2024-09-16T14:54:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2542762783 0 0) 0x7fdda0006590 con 0x7fdda00090c0 2024-09-16T14:54:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdda0009da0 con 0x7fdda00090c0 2024-09-16T14:54:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fddb4004b80 con 0x7fdda00090c0 2024-09-16T14:54:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 10 monclient: dump: 2024-09-16T14:54:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.638 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3626825495 0 0) 0x7fdda0009da0 con 0x7fdda00090c0 2024-09-16T14:54:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdda000c290 con 0x7fdda00090c0 2024-09-16T14:54:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 2043521867) 0x7fddb4004ea0 con 0x7fdda00090c0 2024-09-16T14:54:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:54:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 >> v1:172.21.15.26:6789/0 conn(0x7fdda00090c0 legacy=0x7fdda0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fdda77fe640 1 -- 172.21.15.26:0/3183220045 >> v1:172.21.15.26:6789/0 conn(0x7fdda00090c0 legacy=0x7fdda0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:02.638 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:54:02.639 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:02.639 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:02.639 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:54:02.639 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 108, 2024-09-16T14:54:02.639 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:54:02.639 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:54:02.639 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:54:02.639 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:54:02.639 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:02.639 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:54:02.639 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:02.640 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:54:02.640 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:54:02.640 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:02.640 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:02.640 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:02.640 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:02.640 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:02.640 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:02.640 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:02.640 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:02.640 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:02.641 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:02.641 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:02.641 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:54:02.641 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:54:02.641 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:02.641 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:02.641 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:02.641 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:02.641 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:02.641 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:02.641 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:02.641 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:02.642 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:02.642 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:02.642 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:02.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.642 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:54:02.642 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:54:02.642 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:54:02.642 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:54:02.643 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:54:02.643 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:54:02.643 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:02.643 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:02.643 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:54:02.643 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:54:02.643 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:54:02.643 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:54:02.643 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:54:02.643 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:54:02.644 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:54:02.644 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:02.644 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:54:02.644 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:02.644 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:02.644 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:02.644 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:02.644 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:02.644 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:02.644 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:02.644 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:02.644 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:02.645 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:02.645 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:02.645 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:54:02.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.645 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:54:02.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.645 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:02.645 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:02.645 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:02.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:02.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.646 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:02.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:54:02.646 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:02.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:02.646 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:02.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:54:02.646 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:54:02.646 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:02.646 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:02.647 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:02.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.647 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:02.647 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:02.647 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:02.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:02.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:02.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:54:02.647 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:02.647 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:02.648 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:02.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:54:02.648 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:54:02.648 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:02.648 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:02.648 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:02.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.648 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:02.648 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:02.649 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:02.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:02.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:02.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:54:02.649 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:02.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:02.649 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:02.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:54:02.649 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:54:02.649 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:54:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:54:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:54:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:54:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:02.652 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:02.652 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:54:02.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.652 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:02.652 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:02.652 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:54:02.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:54:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:54:02.653 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:54:02.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fddc53e6640 1 -- 172.21.15.26:0/3183220045 >> v1:172.21.15.189:6812/3196161582 conn(0x7fddac0361b0 legacy=0x7fddac038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fddc53e6640 10 monclient: shutdown 2024-09-16T14:54:02.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fddc53e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fddc53e6640 1 -- 172.21.15.26:0/3183220045 >> v1:172.21.15.26:6790/0 conn(0x7fddc009cb80 legacy=0x7fddc006d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fddbf7fe640 1 -- 172.21.15.26:0/3183220045 reap_dead start 2024-09-16T14:54:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fddbf7fe640 1 -- 172.21.15.26:0/3183220045 reap_dead start 2024-09-16T14:54:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fddc53e6640 1 -- 172.21.15.26:0/3183220045 shutdown_connections 2024-09-16T14:54:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fddc53e6640 1 -- 172.21.15.26:0/3183220045 >> 172.21.15.26:0/3183220045 conn(0x7fddc009fb50 msgr2=0x7fddc0131cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fddc53e6640 1 -- 172.21.15.26:0/3183220045 shutdown_connections 2024-09-16T14:54:02.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.618+0000 7fddc53e6640 1 -- 172.21.15.26:0/3183220045 wait complete. 2024-09-16T14:54:02.657 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:54:02.657 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:54:02.658 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:54:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.770+0000 7f90b5590640 10 monclient: get_monmap_and_config 2024-09-16T14:54:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 10 monclient: build_initial_monmap 2024-09-16T14:54:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 10 monclient: monmap: 2024-09-16T14:54:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.777 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:02.777535+0000 2024-09-16T14:54:02.777 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:02.777535+0000 2024-09-16T14:54:02.777 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:02.777 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.777 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:02.777 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:02.777 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:02.777 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 1 Processor -- start 2024-09-16T14:54:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 1 -- start start 2024-09-16T14:54:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 10 monclient: init 2024-09-16T14:54:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:54:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 10 monclient(hunting): picked mon.noname-a con 0x7f90b009a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 10 monclient(hunting): picked mon.noname-b con 0x7f90b009d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 10 monclient(hunting): picked mon.noname-c con 0x7f90b009b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b009dc60 con 0x7f90b009a550 2024-09-16T14:54:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b0099ae0 con 0x7f90b009b040 2024-09-16T14:54:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b009b8f0 con 0x7f90b009d1b0 2024-09-16T14:54:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 10 monclient(hunting): authenticate will time out at 1666.897583s 2024-09-16T14:54:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90aeffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f90b009a550 0x7f90b009a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53934/0 (socket says 172.21.15.26:53934) 2024-09-16T14:54:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90aeffd640 1 -- 172.21.15.26:0/4044284614 learned_addr learned my addr 172.21.15.26:0/4044284614 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90ae7fc640 1 --1- 172.21.15.26:0/4044284614 >> v1:172.21.15.26:6790/0 conn(0x7f90b009d1b0 0x7f90b0136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47686/0 (socket says 172.21.15.26:47686) 2024-09-16T14:54:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 1 -- 172.21.15.26:0/4044284614 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783211215 0 0) 0x7f90b009b8f0 con 0x7f90b009d1b0 2024-09-16T14:54:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient(hunting): my global_id is 234212 2024-09-16T14:54:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 1 -- 172.21.15.26:0/4044284614 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9098003530 con 0x7f90b009d1b0 2024-09-16T14:54:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 1 -- 172.21.15.26:0/4044284614 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385576242 0 0) 0x7f90b009dc60 con 0x7f90b009a550 2024-09-16T14:54:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient(hunting): my global_id is 194526 2024-09-16T14:54:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 1 -- 172.21.15.26:0/4044284614 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b009b8f0 con 0x7f90b009a550 2024-09-16T14:54:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 1 -- 172.21.15.26:0/4044284614 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 138571074 0 0) 0x7f9098003530 con 0x7f90b009d1b0 2024-09-16T14:54:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 1 -- 172.21.15.26:0/4044284614 >> v1:172.21.15.189:6789/0 conn(0x7f90b009b040 legacy=0x7f90b013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 1 -- 172.21.15.26:0/4044284614 >> v1:172.21.15.26:6789/0 conn(0x7f90b009a550 legacy=0x7f90b009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient: _finish_hunting 0 2024-09-16T14:54:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 1 monclient: found mon.noname-b 2024-09-16T14:54:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:54:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 1 -- 172.21.15.26:0/4044284614 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b009bbd0 con 0x7f90b009d1b0 2024-09-16T14:54:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient: _finish_auth 0 2024-09-16T14:54:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient: _check_auth_tickets 2024-09-16T14:54:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 1 -- 172.21.15.26:0/4044284614 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f90a0001620 con 0x7f90b009d1b0 2024-09-16T14:54:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 5 monclient: authenticate success, global_id 234212 2024-09-16T14:54:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:54:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient: dump: 2024-09-16T14:54:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.790 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 1 -- 172.21.15.26:0/4044284614 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90a0001840 con 0x7f90b009d1b0 2024-09-16T14:54:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 1 -- 172.21.15.26:0/4044284614 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f90a0001c10 con 0x7f90b009d1b0 2024-09-16T14:54:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 4 set_mon_vals no callback set 2024-09-16T14:54:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90adffb640 10 monclient: dump: 2024-09-16T14:54:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.792 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 10 monclient: shutdown 2024-09-16T14:54:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 1 -- 172.21.15.26:0/4044284614 >> v1:172.21.15.26:6790/0 conn(0x7f90b009d1b0 legacy=0x7f90b0136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 1 -- 172.21.15.26:0/4044284614 shutdown_connections 2024-09-16T14:54:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 1 -- 172.21.15.26:0/4044284614 >> 172.21.15.26:0/4044284614 conn(0x7f90b009fb50 msgr2=0x7f90b009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 1 -- 172.21.15.26:0/4044284614 shutdown_connections 2024-09-16T14:54:02.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 1 -- 172.21.15.26:0/4044284614 wait complete. 2024-09-16T14:54:02.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 10 monclient: build_initial_monmap 2024-09-16T14:54:02.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.774+0000 7f90b5590640 10 monclient: monmap: 2024-09-16T14:54:02.793 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:02.793 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:02.793 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:02.793 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:02.793 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:02.793 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:02.793 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:02.793 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:02.793 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:02.793 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 1 Processor -- start 2024-09-16T14:54:02.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 1 -- start start 2024-09-16T14:54:02.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 10 monclient: init 2024-09-16T14:54:02.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:02.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:54:02.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 10 monclient(hunting): picked mon.noname-b con 0x7f90b009a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:02.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 10 monclient(hunting): picked mon.noname-c con 0x7f90b009b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:02.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 10 monclient(hunting): picked mon.noname-a con 0x7f90b009d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:02.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b011a430 con 0x7f90b009d1b0 2024-09-16T14:54:02.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b011a040 con 0x7f90b009a550 2024-09-16T14:54:02.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b00a2570 con 0x7f90b009b040 2024-09-16T14:54:02.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:02.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 10 monclient(hunting): authenticate will time out at 1666.900719s 2024-09-16T14:54:02.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90ae7fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f90b009b040 0x7f90b013a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47692/0 (socket says 172.21.15.26:47692) 2024-09-16T14:54:02.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90af7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f90b009d1b0 0x7f90b013aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53942/0 (socket says 172.21.15.26:53942) 2024-09-16T14:54:02.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90ae7fc640 1 -- 172.21.15.26:0/1642917364 learned_addr learned my addr 172.21.15.26:0/1642917364 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:02.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154884699 0 0) 0x7f90b011a430 con 0x7f90b009d1b0 2024-09-16T14:54:02.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): my global_id is 194529 2024-09-16T14:54:02.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f909c003530 con 0x7f90b009d1b0 2024-09-16T14:54:02.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500813188 0 0) 0x7f90b00a2570 con 0x7f90b009b040 2024-09-16T14:54:02.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): my global_id is 234215 2024-09-16T14:54:02.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b011a430 con 0x7f90b009b040 2024-09-16T14:54:02.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2049084422 0 0) 0x7f909c003530 con 0x7f90b009d1b0 2024-09-16T14:54:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90b00a2570 con 0x7f90b009d1b0 2024-09-16T14:54:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f90a8001bc0 con 0x7f90b009d1b0 2024-09-16T14:54:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): dump: 2024-09-16T14:54:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.798 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 704977140 0 0) 0x7f90b011a040 con 0x7f90b009a550 2024-09-16T14:54:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): my global_id is 244102 2024-09-16T14:54:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f909c003530 con 0x7f90b009a550 2024-09-16T14:54:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1592568250 0 0) 0x7f90b011a430 con 0x7f90b009b040 2024-09-16T14:54:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90b011a040 con 0x7f90b009b040 2024-09-16T14:54:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f90a00015e0 con 0x7f90b009b040 2024-09-16T14:54:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): dump: 2024-09-16T14:54:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.800 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2428206356 0 0) 0x7f90b00a2570 con 0x7f90b009d1b0 2024-09-16T14:54:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 >> v1:172.21.15.26:6790/0 conn(0x7f90b009b040 legacy=0x7f90b013a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 >> v1:172.21.15.189:6789/0 conn(0x7f90b009a550 legacy=0x7f90b0139ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient: _finish_hunting 0 2024-09-16T14:54:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 monclient: found mon.a 2024-09-16T14:54:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b011ba10 con 0x7f90b009d1b0 2024-09-16T14:54:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient: _finish_auth 0 2024-09-16T14:54:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient: _check_auth_tickets 2024-09-16T14:54:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 5 monclient: authenticate success, global_id 194529 2024-09-16T14:54:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 10 monclient: _renew_subs 2024-09-16T14:54:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 1 -- 172.21.15.26:0/1642917364 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b00709a0 con 0x7f90b009d1b0 2024-09-16T14:54:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 10 monclient: _renew_subs 2024-09-16T14:54:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 1 -- 172.21.15.26:0/1642917364 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b011b450 con 0x7f90b009d1b0 2024-09-16T14:54:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90a80020f0 con 0x7f90b009d1b0 2024-09-16T14:54:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f90a8003bc0 con 0x7f90b009d1b0 2024-09-16T14:54:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90ad7fa640 4 set_mon_vals no callback set 2024-09-16T14:54:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient: dump: 2024-09-16T14:54:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.803 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90b5590640 1 -- 172.21.15.26:0/1642917364 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9090006590 con 0x7f9090002830 2024-09-16T14:54:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f90a8003d20 con 0x7f90b009d1b0 2024-09-16T14:54:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f90a8043ef0 con 0x7f90b009d1b0 2024-09-16T14:54:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831393006 0 0) 0x7f9090006590 con 0x7f9090002830 2024-09-16T14:54:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient(hunting): my global_id is 234218 2024-09-16T14:54:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b00a2570 con 0x7f9090002830 2024-09-16T14:54:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90ae7fc640 10 monclient: get_auth_request con 0x7f909c036490 auth_method 0 2024-09-16T14:54:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3195889341 0 0) 0x7f90b00a2570 con 0x7f9090002830 2024-09-16T14:54:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9090006590 con 0x7f9090002830 2024-09-16T14:54:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f90a4001080 con 0x7f9090002830 2024-09-16T14:54:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient: dump: 2024-09-16T14:54:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.806 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1159401262 0 0) 0x7f9090006590 con 0x7f9090002830 2024-09-16T14:54:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f909000a0e0 con 0x7f9090002830 2024-09-16T14:54:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f90a4001440 con 0x7f9090002830 2024-09-16T14:54:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 >> v1:172.21.15.26:6790/0 conn(0x7f9090002830 legacy=0x7f9090004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.778+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 >> v1:172.21.15.26:6790/0 conn(0x7f9090002830 legacy=0x7f9090004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90b5590640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:54:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90b5590640 1 -- 172.21.15.26:0/1642917364 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9090009da0 con 0x7f90900090c0 2024-09-16T14:54:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056902212 0 0) 0x7f9090009da0 con 0x7f90900090c0 2024-09-16T14:54:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 10 monclient(hunting): my global_id is 234221 2024-09-16T14:54:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9090006590 con 0x7f90900090c0 2024-09-16T14:54:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1191312363 0 0) 0x7f9090006590 con 0x7f90900090c0 2024-09-16T14:54:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9090009da0 con 0x7f90900090c0 2024-09-16T14:54:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f90a4004b80 con 0x7f90900090c0 2024-09-16T14:54:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 10 monclient: dump: 2024-09-16T14:54:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.810 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1210720970 0 0) 0x7f9090009da0 con 0x7f90900090c0 2024-09-16T14:54:02.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f909000c290 con 0x7f90900090c0 2024-09-16T14:54:02.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 3241558624) 0x7f90a4005f40 con 0x7f90900090c0 2024-09-16T14:54:02.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:54:02.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:02.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 >> v1:172.21.15.26:6790/0 conn(0x7f90900090c0 legacy=0x7f9090009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90977fe640 1 -- 172.21.15.26:0/1642917364 >> v1:172.21.15.26:6790/0 conn(0x7f90900090c0 legacy=0x7f9090009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:02.811 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:54:02.811 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:02.811 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:02.811 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:54:02.812 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 108, 2024-09-16T14:54:02.812 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:54:02.812 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:54:02.812 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:54:02.812 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:54:02.812 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:02.812 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:54:02.812 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:02.812 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:54:02.812 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:54:02.812 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:02.813 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:02.813 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:02.813 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:02.813 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:02.813 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:02.813 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:02.813 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:02.813 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:02.814 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:02.814 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:02.814 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:54:02.814 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:54:02.814 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:02.814 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:02.814 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:02.814 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:02.814 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:02.815 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:02.815 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:02.815 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:02.815 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:02.815 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:54:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:54:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:54:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:54:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:54:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:54:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:54:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:54:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:54:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:54:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:54:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:54:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:54:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:54:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:54:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:54:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:54:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:54:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:54:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:54:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:54:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:54:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:54:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:54:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:54:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:54:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:54:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:54:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:54:02.824 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:54:02.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90b5590640 1 -- 172.21.15.26:0/1642917364 >> v1:172.21.15.189:6812/3196161582 conn(0x7f909c036490 legacy=0x7f909c038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90b5590640 10 monclient: shutdown 2024-09-16T14:54:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90b5590640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90b5590640 1 -- 172.21.15.26:0/1642917364 >> v1:172.21.15.26:6789/0 conn(0x7f90b009d1b0 legacy=0x7f90b013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90af7fe640 1 -- 172.21.15.26:0/1642917364 reap_dead start 2024-09-16T14:54:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90af7fe640 1 -- 172.21.15.26:0/1642917364 reap_dead start 2024-09-16T14:54:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90b5590640 1 -- 172.21.15.26:0/1642917364 shutdown_connections 2024-09-16T14:54:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90b5590640 1 -- 172.21.15.26:0/1642917364 >> 172.21.15.26:0/1642917364 conn(0x7f90b009fb50 msgr2=0x7f90b009e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90b5590640 1 -- 172.21.15.26:0/1642917364 shutdown_connections 2024-09-16T14:54:02.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.786+0000 7f90b5590640 1 -- 172.21.15.26:0/1642917364 wait complete. 2024-09-16T14:54:02.826 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:54:02.827 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:54:02.827 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:54:02.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.946+0000 7f17c0e32640 10 monclient: get_monmap_and_config 2024-09-16T14:54:02.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.946+0000 7f17c0e32640 10 monclient: build_initial_monmap 2024-09-16T14:54:02.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.946+0000 7f17c0e32640 10 monclient: monmap: 2024-09-16T14:54:02.953 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:02.953 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.953 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:02.953342+0000 2024-09-16T14:54:02.953 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:02.953342+0000 2024-09-16T14:54:02.953 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:02.954 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 1 Processor -- start 2024-09-16T14:54:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 1 -- start start 2024-09-16T14:54:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 10 monclient: init 2024-09-16T14:54:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:54:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 10 monclient(hunting): picked mon.noname-a con 0x7f17bc09a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 10 monclient(hunting): picked mon.noname-b con 0x7f17bc09d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 10 monclient(hunting): picked mon.noname-c con 0x7f17bc09b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc09dc60 con 0x7f17bc09a550 2024-09-16T14:54:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc099ae0 con 0x7f17bc09b040 2024-09-16T14:54:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc09b8f0 con 0x7f17bc09d1b0 2024-09-16T14:54:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 10 monclient(hunting): authenticate will time out at 1667.073350s 2024-09-16T14:54:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17ba59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f17bc09a550 0x7f17bc09a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53946/0 (socket says 172.21.15.26:53946) 2024-09-16T14:54:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17ba59c640 1 -- 172.21.15.26:0/2878469102 learned_addr learned my addr 172.21.15.26:0/2878469102 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b9d9b640 1 --1- 172.21.15.26:0/2878469102 >> v1:172.21.15.26:6790/0 conn(0x7f17bc09d1b0 0x7f17bc136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47726/0 (socket says 172.21.15.26:47726) 2024-09-16T14:54:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 1 -- 172.21.15.26:0/2878469102 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223348322 0 0) 0x7f17bc09b8f0 con 0x7f17bc09d1b0 2024-09-16T14:54:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient(hunting): my global_id is 234224 2024-09-16T14:54:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 1 -- 172.21.15.26:0/2878469102 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a4005740 con 0x7f17bc09d1b0 2024-09-16T14:54:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 1 -- 172.21.15.26:0/2878469102 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894886059 0 0) 0x7f17bc09dc60 con 0x7f17bc09a550 2024-09-16T14:54:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient(hunting): my global_id is 194532 2024-09-16T14:54:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 1 -- 172.21.15.26:0/2878469102 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17bc09b8f0 con 0x7f17bc09a550 2024-09-16T14:54:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 1 -- 172.21.15.26:0/2878469102 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 903482501 0 0) 0x7f17a4005740 con 0x7f17bc09d1b0 2024-09-16T14:54:02.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 1 -- 172.21.15.26:0/2878469102 >> v1:172.21.15.189:6789/0 conn(0x7f17bc09b040 legacy=0x7f17bc13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 1 -- 172.21.15.26:0/2878469102 >> v1:172.21.15.26:6789/0 conn(0x7f17bc09a550 legacy=0x7f17bc09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient: _finish_hunting 0 2024-09-16T14:54:02.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 1 monclient: found mon.noname-b 2024-09-16T14:54:02.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:02.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:54:02.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 1 -- 172.21.15.26:0/2878469102 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17bc09bbd0 con 0x7f17bc09d1b0 2024-09-16T14:54:02.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient: _finish_auth 0 2024-09-16T14:54:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient: _check_auth_tickets 2024-09-16T14:54:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 1 -- 172.21.15.26:0/2878469102 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f17b0001550 con 0x7f17bc09d1b0 2024-09-16T14:54:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:54:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient: dump: 2024-09-16T14:54:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.982 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.982 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 5 monclient: authenticate success, global_id 234224 2024-09-16T14:54:02.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:02.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 1 -- 172.21.15.26:0/2878469102 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17b00017b0 con 0x7f17bc09d1b0 2024-09-16T14:54:02.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:02.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 1 -- 172.21.15.26:0/2878469102 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f17b0001b80 con 0x7f17bc09d1b0 2024-09-16T14:54:02.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:02.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17b959a640 10 monclient: dump: 2024-09-16T14:54:02.983 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.983 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.983 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.983 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.984 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.984 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.984 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.984 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.984 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.984 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:02.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 4 set_mon_vals no callback set 2024-09-16T14:54:02.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 10 monclient: shutdown 2024-09-16T14:54:02.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:02.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 1 -- 172.21.15.26:0/2878469102 >> v1:172.21.15.26:6790/0 conn(0x7f17bc09d1b0 legacy=0x7f17bc136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:02.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 1 -- 172.21.15.26:0/2878469102 shutdown_connections 2024-09-16T14:54:02.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 1 -- 172.21.15.26:0/2878469102 >> 172.21.15.26:0/2878469102 conn(0x7f17bc09fb50 msgr2=0x7f17bc09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:02.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 1 -- 172.21.15.26:0/2878469102 shutdown_connections 2024-09-16T14:54:02.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 1 -- 172.21.15.26:0/2878469102 wait complete. 2024-09-16T14:54:02.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 10 monclient: build_initial_monmap 2024-09-16T14:54:02.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 10 monclient: monmap: 2024-09-16T14:54:02.986 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:02.986 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:02.986 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:02.987 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:02.987 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:02.987 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:02.987 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:02.987 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:02.987 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:02.988 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 1 Processor -- start 2024-09-16T14:54:02.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 1 -- start start 2024-09-16T14:54:02.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.950+0000 7f17c0e32640 10 monclient: init 2024-09-16T14:54:02.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:02.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:54:02.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 10 monclient(hunting): picked mon.noname-b con 0x7f17bc09a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 10 monclient(hunting): picked mon.noname-a con 0x7f17bc09b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 10 monclient(hunting): picked mon.noname-c con 0x7f17bc09d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc1a1db0 con 0x7f17bc09b040 2024-09-16T14:54:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc1a11e0 con 0x7f17bc09a550 2024-09-16T14:54:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17bc09a240 con 0x7f17bc09d1b0 2024-09-16T14:54:02.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 10 monclient(hunting): authenticate will time out at 1667.076445s 2024-09-16T14:54:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17bad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f17bc09d1b0 0x7f17bc1cd9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47738/0 (socket says 172.21.15.26:47738) 2024-09-16T14:54:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17bad9d640 1 -- 172.21.15.26:0/4104009046 learned_addr learned my addr 172.21.15.26:0/4104009046 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736630879 0 0) 0x7f17bc09a240 con 0x7f17bc09d1b0 2024-09-16T14:54:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): my global_id is 234227 2024-09-16T14:54:02.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f179c003530 con 0x7f17bc09d1b0 2024-09-16T14:54:02.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133105581 0 0) 0x7f17bc1a1db0 con 0x7f17bc09b040 2024-09-16T14:54:02.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): my global_id is 194535 2024-09-16T14:54:02.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17bc09a240 con 0x7f17bc09b040 2024-09-16T14:54:02.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3457307108 0 0) 0x7f179c003530 con 0x7f17bc09d1b0 2024-09-16T14:54:02.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17bc1a1db0 con 0x7f17bc09d1b0 2024-09-16T14:54:02.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f17b4001c20 con 0x7f17bc09d1b0 2024-09-16T14:54:02.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:02.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): dump: 2024-09-16T14:54:02.993 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.993 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.994 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.994 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.994 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:02.994 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:02.995 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:02.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2651987138 0 0) 0x7f17bc09a240 con 0x7f17bc09b040 2024-09-16T14:54:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f179c003530 con 0x7f17bc09b040 2024-09-16T14:54:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769633888 0 0) 0x7f17bc1a11e0 con 0x7f17bc09a550 2024-09-16T14:54:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:02.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:02.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:02.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): my global_id is 244108 2024-09-16T14:54:02.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17bc09a240 con 0x7f17bc09a550 2024-09-16T14:54:02.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f17b0001ac0 con 0x7f17bc09b040 2024-09-16T14:54:02.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:02.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:02.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): dump: 2024-09-16T14:54:02.998 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:02.998 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:02.999 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.999 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:02.999 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:03.000 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:03.000 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:03.000 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:03.000 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:03.000 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:03.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3879626400 0 0) 0x7f17bc1a1db0 con 0x7f17bc09d1b0 2024-09-16T14:54:03.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 >> v1:172.21.15.189:6789/0 conn(0x7f17bc09a550 legacy=0x7f17bc1b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:03.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 >> v1:172.21.15.26:6789/0 conn(0x7f17bc09b040 legacy=0x7f17bc1ca280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:03.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient: _finish_hunting 0 2024-09-16T14:54:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 monclient: found mon.c 2024-09-16T14:54:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17bc1a2ca0 con 0x7f17bc09d1b0 2024-09-16T14:54:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient: _finish_auth 0 2024-09-16T14:54:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient: _check_auth_tickets 2024-09-16T14:54:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 5 monclient: authenticate success, global_id 234227 2024-09-16T14:54:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 10 monclient: _renew_subs 2024-09-16T14:54:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 1 -- 172.21.15.26:0/4104009046 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17bc1a2f80 con 0x7f17bc09d1b0 2024-09-16T14:54:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 10 monclient: _renew_subs 2024-09-16T14:54:03.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 1 -- 172.21.15.26:0/4104009046 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17bc1b5260 con 0x7f17bc09d1b0 2024-09-16T14:54:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17b40039a0 con 0x7f17bc09d1b0 2024-09-16T14:54:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f17b4003cb0 con 0x7f17bc09d1b0 2024-09-16T14:54:03.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:03.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:03.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient: dump: 2024-09-16T14:54:03.006 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:03.006 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:03.006 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:03.006 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:03.006 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:03.007 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:03.007 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:03.007 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:03.007 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:03.007 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:03.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17b8d99640 4 set_mon_vals no callback set 2024-09-16T14:54:03.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:03.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17c0e32640 1 -- 172.21.15.26:0/4104009046 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f17a0006590 con 0x7f17a0002830 2024-09-16T14:54:03.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f17b4003e90 con 0x7f17bc09d1b0 2024-09-16T14:54:03.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f17b4044040 con 0x7f17bc09d1b0 2024-09-16T14:54:03.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17b9d9b640 10 monclient: get_auth_request con 0x7f179c036440 auth_method 0 2024-09-16T14:54:03.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932835245 0 0) 0x7f17a0006590 con 0x7f17a0002830 2024-09-16T14:54:03.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:03.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:03.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:03.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient(hunting): my global_id is 244111 2024-09-16T14:54:03.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17bc1a1db0 con 0x7f17a0002830 2024-09-16T14:54:03.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1902348444 0 0) 0x7f17bc1a1db0 con 0x7f17a0002830 2024-09-16T14:54:03.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17a0006590 con 0x7f17a0002830 2024-09-16T14:54:03.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f17ac0011e0 con 0x7f17a0002830 2024-09-16T14:54:03.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:03.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:54:03.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.954+0000 7f17aaffd640 10 monclient: dump: 2024-09-16T14:54:03.010 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:03.010 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:03.010 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:03.010 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:03.011 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:03.011 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:03.011 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:03.011 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:03.011 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:03.011 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:03.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.958+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 748547591 0 0) 0x7f17a0006590 con 0x7f17a0002830 2024-09-16T14:54:03.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.958+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f17a000a0e0 con 0x7f17a0002830 2024-09-16T14:54:03.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.958+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f17ac001460 con 0x7f17a0002830 2024-09-16T14:54:03.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.958+0000 7f17aaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:03.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.958+0000 7f17aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:03.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.958+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 >> v1:172.21.15.189:6789/0 conn(0x7f17a0002830 legacy=0x7f17a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:03.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.958+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 >> v1:172.21.15.189:6789/0 conn(0x7f17a0002830 legacy=0x7f17a0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:03.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.962+0000 7f17c0e32640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:54:03.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.962+0000 7f17c0e32640 1 -- 172.21.15.26:0/4104009046 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f17a0009da0 con 0x7f17a00090c0 2024-09-16T14:54:03.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69326369 0 0) 0x7f17a0009da0 con 0x7f17a00090c0 2024-09-16T14:54:03.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:03.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:03.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:03.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 10 monclient(hunting): my global_id is 244114 2024-09-16T14:54:03.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a0006590 con 0x7f17a00090c0 2024-09-16T14:54:03.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3506187219 0 0) 0x7f17a0006590 con 0x7f17a00090c0 2024-09-16T14:54:03.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17a0009da0 con 0x7f17a00090c0 2024-09-16T14:54:03.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f17ac004a10 con 0x7f17a00090c0 2024-09-16T14:54:03.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:03.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:54:03.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 10 monclient: dump: 2024-09-16T14:54:03.015 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:03.015 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:03.015 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:03.015 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:03.015 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:03.015 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:03.015 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:03.015 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:03.016 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:03.016 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:03.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1408271240 0 0) 0x7f17a0009da0 con 0x7f17a00090c0 2024-09-16T14:54:03.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f17a000c290 con 0x7f17a00090c0 2024-09-16T14:54:03.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3961195627) 0x7f17ac005f30 con 0x7f17a00090c0 2024-09-16T14:54:03.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:54:03.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:03.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 >> v1:172.21.15.189:6789/0 conn(0x7f17a00090c0 legacy=0x7f17a0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:03.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17aaffd640 1 -- 172.21.15.26:0/4104009046 >> v1:172.21.15.189:6789/0 conn(0x7f17a00090c0 legacy=0x7f17a0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:03.017 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:54:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:54:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 108, 2024-09-16T14:54:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:54:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:54:03.018 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:54:03.019 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:54:03.019 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:03.019 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:54:03.019 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:03.019 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:54:03.019 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:54:03.019 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:03.020 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:03.020 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:03.020 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:03.020 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:03.020 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:03.020 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:03.020 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:03.021 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:03.021 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:03.021 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:03.021 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:54:03.021 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:54:03.021 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:03.021 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:03.022 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:03.022 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:03.022 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:03.022 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:03.022 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:03.022 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:03.023 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:03.023 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:03.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:03.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:03.023 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:54:03.023 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:54:03.023 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:54:03.023 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:54:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:54:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:54:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:54:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:54:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:54:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:54:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:54:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:54:03.024 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:54:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:54:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:03.025 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:54:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:54:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:03.026 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:54:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:54:03.027 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:54:03.028 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:03.028 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:03.028 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:03.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:03.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:03.028 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:03.028 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:54:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:54:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:54:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:03.029 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:54:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:03.030 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:54:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:54:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:03.031 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:54:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:54:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:03.032 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:54:03.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:03.033 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:03.033 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:03.033 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:03.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:03.033 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:03.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:03.033 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:54:03.033 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:54:03.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17c0e32640 1 -- 172.21.15.26:0/4104009046 >> v1:172.21.15.189:6812/3196161582 conn(0x7f179c036440 legacy=0x7f179c038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:03.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17c0e32640 10 monclient: shutdown 2024-09-16T14:54:03.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17c0e32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:03.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17c0e32640 1 -- 172.21.15.26:0/4104009046 >> v1:172.21.15.26:6790/0 conn(0x7f17bc09d1b0 legacy=0x7f17bc1cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:03.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17bad9d640 1 -- 172.21.15.26:0/4104009046 reap_dead start 2024-09-16T14:54:03.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17c0e32640 1 -- 172.21.15.26:0/4104009046 shutdown_connections 2024-09-16T14:54:03.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17c0e32640 1 -- 172.21.15.26:0/4104009046 >> 172.21.15.26:0/4104009046 conn(0x7f17bc09fb50 msgr2=0x7f17bc0a1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:03.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17c0e32640 1 -- 172.21.15.26:0/4104009046 shutdown_connections 2024-09-16T14:54:03.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:02.966+0000 7f17c0e32640 1 -- 172.21.15.26:0/4104009046 wait complete. 2024-09-16T14:54:03.036 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:54:03.036 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:54:03.036 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:54:03.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.174+0000 7f3f45614640 10 monclient: get_monmap_and_config 2024-09-16T14:54:03.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.174+0000 7f3f45614640 10 monclient: build_initial_monmap 2024-09-16T14:54:03.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.174+0000 7f3f45614640 10 monclient: monmap: 2024-09-16T14:54:03.179 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:03.179 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:03.179 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:03.180065+0000 2024-09-16T14:54:03.179 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:03.180065+0000 2024-09-16T14:54:03.180 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:03.180 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:03.180 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:03.180 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:03.180 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:03.180 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:03.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.174+0000 7f3f45614640 1 Processor -- start 2024-09-16T14:54:03.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 1 -- start start 2024-09-16T14:54:03.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient: init 2024-09-16T14:54:03.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:03.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:54:03.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient(hunting): picked mon.noname-c con 0x7f3f400a0b80 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:03.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient(hunting): picked mon.noname-a con 0x7f3f400a0f60 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:03.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient(hunting): picked mon.noname-b con 0x7f3f4013af70 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:03.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4006d0a0 con 0x7f3f400a0f60 2024-09-16T14:54:03.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4006ade0 con 0x7f3f400a0b80 2024-09-16T14:54:03.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4006cae0 con 0x7f3f4013af70 2024-09-16T14:54:03.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:03.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient(hunting): authenticate will time out at 1667.300937s 2024-09-16T14:54:03.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3e7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3f400a0f60 0x7f3f4006b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53970/0 (socket says 172.21.15.26:53970) 2024-09-16T14:54:03.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3f7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3f4013af70 0x7f3f4013d360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47752/0 (socket says 172.21.15.26:47752) 2024-09-16T14:54:03.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3e7fc640 1 -- 172.21.15.26:0/1768610010 learned_addr learned my addr 172.21.15.26:0/1768610010 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:03.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 1 -- 172.21.15.26:0/1768610010 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433686967 0 0) 0x7f3f4006d0a0 con 0x7f3f400a0f60 2024-09-16T14:54:03.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:03.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:03.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:03.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient(hunting): my global_id is 194538 2024-09-16T14:54:03.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 1 -- 172.21.15.26:0/1768610010 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f28003530 con 0x7f3f400a0f60 2024-09-16T14:54:03.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 1 -- 172.21.15.26:0/1768610010 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662507582 0 0) 0x7f3f4006cae0 con 0x7f3f4013af70 2024-09-16T14:54:03.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:03.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:03.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:03.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient(hunting): my global_id is 234230 2024-09-16T14:54:03.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 1 -- 172.21.15.26:0/1768610010 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f4006d0a0 con 0x7f3f4013af70 2024-09-16T14:54:03.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 1 -- 172.21.15.26:0/1768610010 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3011615856 0 0) 0x7f3f28003530 con 0x7f3f400a0f60 2024-09-16T14:54:03.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 1 -- 172.21.15.26:0/1768610010 >> v1:172.21.15.26:6790/0 conn(0x7f3f4013af70 legacy=0x7f3f4013d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:03.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 1 -- 172.21.15.26:0/1768610010 >> v1:172.21.15.189:6789/0 conn(0x7f3f400a0b80 legacy=0x7f3f4006b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:03.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:54:03.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 1 monclient: found mon.noname-a 2024-09-16T14:54:03.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:03.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:54:03.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 1 -- 172.21.15.26:0/1768610010 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4006beb0 con 0x7f3f400a0f60 2024-09-16T14:54:03.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient: _finish_auth 0 2024-09-16T14:54:03.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:54:03.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:03.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 1 -- 172.21.15.26:0/1768610010 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f34001620 con 0x7f3f400a0f60 2024-09-16T14:54:03.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:03.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:03.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient: dump: 2024-09-16T14:54:03.191 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:03.191 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:03.191 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:03.191 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:03.192 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:03.192 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:03.192 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:03.192 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:03.192 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:03.192 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:03.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 5 monclient: authenticate success, global_id 194538 2024-09-16T14:54:03.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:03.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 1 -- 172.21.15.26:0/1768610010 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f34001880 con 0x7f3f400a0f60 2024-09-16T14:54:03.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:03.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 1 -- 172.21.15.26:0/1768610010 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f34001c50 con 0x7f3f400a0f60 2024-09-16T14:54:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 4 set_mon_vals no callback set 2024-09-16T14:54:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f3dffb640 10 monclient: dump: 2024-09-16T14:54:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:03.194 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:03.194 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:03.194 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:03.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient: shutdown 2024-09-16T14:54:03.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:03.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 1 -- 172.21.15.26:0/1768610010 >> v1:172.21.15.26:6789/0 conn(0x7f3f400a0f60 legacy=0x7f3f4006b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:03.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 1 -- 172.21.15.26:0/1768610010 shutdown_connections 2024-09-16T14:54:03.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 1 -- 172.21.15.26:0/1768610010 >> 172.21.15.26:0/1768610010 conn(0x7f3f400a1c70 msgr2=0x7f3f400a2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:03.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 1 -- 172.21.15.26:0/1768610010 shutdown_connections 2024-09-16T14:54:03.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 1 -- 172.21.15.26:0/1768610010 wait complete. 2024-09-16T14:54:03.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient: build_initial_monmap 2024-09-16T14:54:03.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient: monmap: 2024-09-16T14:54:03.195 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:03.195 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:03.195 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:03.195 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:03.195 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:03.195 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:03.195 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:03.195 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:03.195 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:03.195 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:03.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 1 Processor -- start 2024-09-16T14:54:03.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 1 -- start start 2024-09-16T14:54:03.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient: init 2024-09-16T14:54:03.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:03.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:54:03.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient(hunting): picked mon.noname-b con 0x7f3f400a0b80 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:03.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.178+0000 7f3f45614640 10 monclient(hunting): picked mon.noname-a con 0x7f3f400a0f60 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:03.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 10 monclient(hunting): picked mon.noname-c con 0x7f3f4013af70 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:03.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f400703d0 con 0x7f3f400a0f60 2024-09-16T14:54:03.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4009b050 con 0x7f3f400a0b80 2024-09-16T14:54:03.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4013e710 con 0x7f3f4013af70 2024-09-16T14:54:03.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:03.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 10 monclient(hunting): authenticate will time out at 1667.304071s 2024-09-16T14:54:03.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f3f7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3f4013af70 0x7f3f401d1b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47758/0 (socket says 172.21.15.26:47758) 2024-09-16T14:54:03.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f3f7fe640 1 -- 172.21.15.26:0/768451945 learned_addr learned my addr 172.21.15.26:0/768451945 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:03.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f3e7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3f400a0f60 0x7f3f4006fcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53984/0 (socket says 172.21.15.26:53984) 2024-09-16T14:54:03.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528101070 0 0) 0x7f3f400703d0 con 0x7f3f400a0f60 2024-09-16T14:54:03.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:03.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:03.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:03.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): my global_id is 194541 2024-09-16T14:54:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f2c003530 con 0x7f3f400a0f60 2024-09-16T14:54:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496292492 0 0) 0x7f3f4013e710 con 0x7f3f4013af70 2024-09-16T14:54:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): my global_id is 234233 2024-09-16T14:54:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f400703d0 con 0x7f3f4013af70 2024-09-16T14:54:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4167346437 0 0) 0x7f3f2c003530 con 0x7f3f400a0f60 2024-09-16T14:54:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f4013e710 con 0x7f3f400a0f60 2024-09-16T14:54:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f34001b90 con 0x7f3f400a0f60 2024-09-16T14:54:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): dump: 2024-09-16T14:54:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:03.200 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2403305851 0 0) 0x7f3f400703d0 con 0x7f3f4013af70 2024-09-16T14:54:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f2c003530 con 0x7f3f4013af70 2024-09-16T14:54:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f38001bc0 con 0x7f3f4013af70 2024-09-16T14:54:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:03.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:03.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): dump: 2024-09-16T14:54:03.201 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:03.201 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:03.201 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:03.201 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:03.201 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:03.201 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:03.202 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:03.202 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:03.202 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:03.202 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:03.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542262829 0 0) 0x7f3f4009b050 con 0x7f3f400a0b80 2024-09-16T14:54:03.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:03.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:03.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:03.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient(hunting): my global_id is 244120 2024-09-16T14:54:03.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f400703d0 con 0x7f3f400a0b80 2024-09-16T14:54:03.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1314719807 0 0) 0x7f3f4013e710 con 0x7f3f400a0f60 2024-09-16T14:54:03.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 >> v1:172.21.15.26:6790/0 conn(0x7f3f4013af70 legacy=0x7f3f401d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:03.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 >> v1:172.21.15.189:6789/0 conn(0x7f3f400a0b80 legacy=0x7f3f4006f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:03.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient: _finish_hunting 0 2024-09-16T14:54:03.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 monclient: found mon.a 2024-09-16T14:54:03.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:03.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:03.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4009c360 con 0x7f3f400a0f60 2024-09-16T14:54:03.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient: _finish_auth 0 2024-09-16T14:54:03.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient: _check_auth_tickets 2024-09-16T14:54:03.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:03.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 5 monclient: authenticate success, global_id 194541 2024-09-16T14:54:03.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 10 monclient: _renew_subs 2024-09-16T14:54:03.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:03.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 1 -- 172.21.15.26:0/768451945 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f400711b0 con 0x7f3f400a0f60 2024-09-16T14:54:03.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 10 monclient: _renew_subs 2024-09-16T14:54:03.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:03.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 1 -- 172.21.15.26:0/768451945 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f40137240 con 0x7f3f400a0f60 2024-09-16T14:54:03.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f34004c00 con 0x7f3f400a0f60 2024-09-16T14:54:03.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:03.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f34004f10 con 0x7f3f400a0f60 2024-09-16T14:54:03.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:03.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:03.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f3d7fa640 4 set_mon_vals no callback set 2024-09-16T14:54:03.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 10 monclient: dump: 2024-09-16T14:54:03.208 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:03.208 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:03.208 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:03.208 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:03.208 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:03.208 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:03.208 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:03.208 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:03.208 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:03.208 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:03.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3f3400f980 con 0x7f3f400a0f60 2024-09-16T14:54:03.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:03.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:03.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:03.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f45614640 1 -- 172.21.15.26:0/768451945 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f20005050 con 0x7f3f400a0f60 2024-09-16T14:54:03.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f3f34043f30 con 0x7f3f400a0f60 2024-09-16T14:54:03.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.182+0000 7f3f3effd640 10 monclient: get_auth_request con 0x7f3f2c036390 auth_method 0 2024-09-16T14:54:03.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.186+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3f34014700 con 0x7f3f400a0f60 2024-09-16T14:54:03.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.186+0000 7f3f277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:03.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.186+0000 7f3f277fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:03.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.406+0000 7f3f45614640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:54:03.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.406+0000 7f3f45614640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:54:03.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.406+0000 7f3f45614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:03.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.406+0000 7f3f45614640 1 -- 172.21.15.26:0/768451945 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f3f20005b10 con 0x7f3f400a0f60 2024-09-16T14:54:03.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f3f277fe640 1 -- 172.21.15.26:0/768451945 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f3f34012da0 con 0x7f3f400a0f60 2024-09-16T14:54:03.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f3f277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:54:03.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f3f277fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:03.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=1302325841,config=692619927,health=3642136620,kv=263502946,logm=3886480483}) 2024-09-16T14:54:03.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=11,logm=46} crc {auth=3068715858,config=692619927,health=3177855593,kv=1867599010,logm=992625538}) 2024-09-16T14:54:03.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=1302325841,config=692619927,health=3642136620,kv=263502946,logm=3886480483}) 2024-09-16T14:54:03.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=10,kv=9,logm=50} crc {auth=1470963434,config=692619927,health=1130269926,kv=351829566,logm=3107005687}) 2024-09-16T14:54:03.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f3f45614640 1 -- 172.21.15.26:0/768451945 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3f2c036390 legacy=0x7f3f2c038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:03.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f3f45614640 10 monclient: shutdown 2024-09-16T14:54:03.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f3f45614640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:03.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f3f45614640 1 -- 172.21.15.26:0/768451945 >> v1:172.21.15.26:6789/0 conn(0x7f3f400a0f60 legacy=0x7f3f4006fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:03.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f3f45614640 1 -- 172.21.15.26:0/768451945 shutdown_connections 2024-09-16T14:54:03.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f3f45614640 1 -- 172.21.15.26:0/768451945 >> 172.21.15.26:0/768451945 conn(0x7f3f400a1c70 msgr2=0x7f3f400a4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:03.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f3f45614640 1 -- 172.21.15.26:0/768451945 shutdown_connections 2024-09-16T14:54:03.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:03.410+0000 7f3f45614640 1 -- 172.21.15.26:0/768451945 wait complete. 2024-09-16T14:54:03.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.414+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.414+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3012947373}) 2024-09-16T14:54:03.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.414+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3356772609}) 2024-09-16T14:54:03.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.414+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.414+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3012947373}) 2024-09-16T14:54:03.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.414+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=693753411}) 2024-09-16T14:54:03.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.418+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.418+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3363910723}) 2024-09-16T14:54:03.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.418+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121508331}) 2024-09-16T14:54:03.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.418+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.418+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3363910723}) 2024-09-16T14:54:03.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.418+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1882985242}) 2024-09-16T14:54:03.423 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.418+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.423 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.418+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1176566190}) 2024-09-16T14:54:03.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.418+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883991340}) 2024-09-16T14:54:03.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.418+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.418+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1176566190}) 2024-09-16T14:54:03.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.418+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2519864775}) 2024-09-16T14:54:03.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.422+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.422+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2643303777}) 2024-09-16T14:54:03.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.422+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3099757314}) 2024-09-16T14:54:03.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.422+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.422+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2643303777}) 2024-09-16T14:54:03.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.422+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2394885880}) 2024-09-16T14:54:03.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.426+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.426+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1735997366}) 2024-09-16T14:54:03.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.426+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3794667595}) 2024-09-16T14:54:03.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.426+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.426+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1735997366}) 2024-09-16T14:54:03.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.426+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1218060601}) 2024-09-16T14:54:03.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.426+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.426+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2897507038}) 2024-09-16T14:54:03.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.426+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3244532965}) 2024-09-16T14:54:03.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.426+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.426+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2897507038}) 2024-09-16T14:54:03.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.426+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3320142939}) 2024-09-16T14:54:03.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2606899013}) 2024-09-16T14:54:03.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1585306690}) 2024-09-16T14:54:03.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2606899013}) 2024-09-16T14:54:03.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=672742135}) 2024-09-16T14:54:03.433 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:54:03.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3557140610}) 2024-09-16T14:54:03.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1702231539}) 2024-09-16T14:54:03.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3557140610}) 2024-09-16T14:54:03.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2889280102}) 2024-09-16T14:54:03.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1508279525}) 2024-09-16T14:54:03.436 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3720496658}) 2024-09-16T14:54:03.436 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.436 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1508279525}) 2024-09-16T14:54:03.436 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.430+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3351893781}) 2024-09-16T14:54:03.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.434+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.434+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4248389304}) 2024-09-16T14:54:03.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.434+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=684809904}) 2024-09-16T14:54:03.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.434+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.434+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4248389304}) 2024-09-16T14:54:03.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.434+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2451863255}) 2024-09-16T14:54:03.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.434+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.434+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3814566082}) 2024-09-16T14:54:03.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.434+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2374669933}) 2024-09-16T14:54:03.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.434+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.434+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3814566082}) 2024-09-16T14:54:03.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.434+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=711717954}) 2024-09-16T14:54:03.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.438+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.438+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1088950315}) 2024-09-16T14:54:03.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.438+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3440498891}) 2024-09-16T14:54:03.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.438+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.438+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1088950315}) 2024-09-16T14:54:03.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.438+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1011490833}) 2024-09-16T14:54:03.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.438+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.438+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2579022408}) 2024-09-16T14:54:03.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.438+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3043612902}) 2024-09-16T14:54:03.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.438+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.438+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2579022408}) 2024-09-16T14:54:03.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.438+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=902573968}) 2024-09-16T14:54:03.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.442+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.442+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2494323730}) 2024-09-16T14:54:03.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.442+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1602848896}) 2024-09-16T14:54:03.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.442+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.442+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2494323730}) 2024-09-16T14:54:03.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.442+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3324126283}) 2024-09-16T14:54:03.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.442+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.442+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4049294796}) 2024-09-16T14:54:03.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.442+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3936920303}) 2024-09-16T14:54:03.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.442+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.442+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4049294796}) 2024-09-16T14:54:03.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.442+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3632593871}) 2024-09-16T14:54:03.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.446+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.446+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001265609}) 2024-09-16T14:54:03.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.446+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4140118057}) 2024-09-16T14:54:03.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.446+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.446+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001265609}) 2024-09-16T14:54:03.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.446+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3226087692}) 2024-09-16T14:54:03.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.446+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.446+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3008222438}) 2024-09-16T14:54:03.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.446+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3397398619}) 2024-09-16T14:54:03.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.446+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.446+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3008222438}) 2024-09-16T14:54:03.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.446+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=908733689}) 2024-09-16T14:54:03.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.450+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.450+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2801512136}) 2024-09-16T14:54:03.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.450+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869655546}) 2024-09-16T14:54:03.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.450+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.450+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2801512136}) 2024-09-16T14:54:03.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.450+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2540753997}) 2024-09-16T14:54:03.454 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.450+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.450+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3255048157}) 2024-09-16T14:54:03.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.450+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1458129177}) 2024-09-16T14:54:03.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.450+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.450+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3255048157}) 2024-09-16T14:54:03.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.450+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3619929778}) 2024-09-16T14:54:03.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.454+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.454+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4128235406}) 2024-09-16T14:54:03.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.454+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2442034829}) 2024-09-16T14:54:03.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.454+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.454+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4128235406}) 2024-09-16T14:54:03.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.454+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2559093485}) 2024-09-16T14:54:03.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.454+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.454+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2438368597}) 2024-09-16T14:54:03.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.454+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2406444898}) 2024-09-16T14:54:03.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.454+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.454+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2438368597}) 2024-09-16T14:54:03.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.454+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1493029943}) 2024-09-16T14:54:03.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.458+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.458+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1409971658}) 2024-09-16T14:54:03.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.458+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1962902257}) 2024-09-16T14:54:03.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.458+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.458+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1409971658}) 2024-09-16T14:54:03.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.458+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1163072314}) 2024-09-16T14:54:03.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.458+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.458+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2909391067}) 2024-09-16T14:54:03.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.458+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479249292}) 2024-09-16T14:54:03.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.458+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.458+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2909391067}) 2024-09-16T14:54:03.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.458+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3093551045}) 2024-09-16T14:54:03.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.462+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.462+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=855256381}) 2024-09-16T14:54:03.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.462+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060121915}) 2024-09-16T14:54:03.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.462+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.462+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=855256381}) 2024-09-16T14:54:03.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.462+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=139232348}) 2024-09-16T14:54:03.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.462+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.462+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1609593239}) 2024-09-16T14:54:03.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.462+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1648960296}) 2024-09-16T14:54:03.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.462+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.462+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1609593239}) 2024-09-16T14:54:03.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.462+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=482880362}) 2024-09-16T14:54:03.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.466+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.466+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3500098715}) 2024-09-16T14:54:03.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.466+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=45380157}) 2024-09-16T14:54:03.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.466+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.466+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3500098715}) 2024-09-16T14:54:03.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.466+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3888045797}) 2024-09-16T14:54:03.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.466+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.466+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3397680305}) 2024-09-16T14:54:03.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.466+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791391871}) 2024-09-16T14:54:03.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.466+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.466+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3397680305}) 2024-09-16T14:54:03.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.466+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=640886326}) 2024-09-16T14:54:03.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=244858464}) 2024-09-16T14:54:03.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3650671792}) 2024-09-16T14:54:03.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=244858464}) 2024-09-16T14:54:03.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1055073737}) 2024-09-16T14:54:03.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4114408524}) 2024-09-16T14:54:03.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1176327970}) 2024-09-16T14:54:03.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4114408524}) 2024-09-16T14:54:03.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3371231645}) 2024-09-16T14:54:03.475 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1984035938}) 2024-09-16T14:54:03.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528593923}) 2024-09-16T14:54:03.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1984035938}) 2024-09-16T14:54:03.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.470+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1726887597}) 2024-09-16T14:54:03.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.474+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.474+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2118706424}) 2024-09-16T14:54:03.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.474+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018360387}) 2024-09-16T14:54:03.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.474+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.474+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2118706424}) 2024-09-16T14:54:03.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.474+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3489025491}) 2024-09-16T14:54:03.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.474+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.474+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2652549190}) 2024-09-16T14:54:03.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.474+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=549873319}) 2024-09-16T14:54:03.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.474+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.474+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2652549190}) 2024-09-16T14:54:03.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.474+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2378691016}) 2024-09-16T14:54:03.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.478+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.478+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1690142205}) 2024-09-16T14:54:03.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.478+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2247986547}) 2024-09-16T14:54:03.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.478+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.478+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1690142205}) 2024-09-16T14:54:03.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.478+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=590863766}) 2024-09-16T14:54:03.482 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.478+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.482 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.478+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3172293574}) 2024-09-16T14:54:03.482 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.478+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1737099560}) 2024-09-16T14:54:03.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.478+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.478+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3172293574}) 2024-09-16T14:54:03.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.478+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1002529360}) 2024-09-16T14:54:03.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=480920893}) 2024-09-16T14:54:03.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3782021142}) 2024-09-16T14:54:03.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=480920893}) 2024-09-16T14:54:03.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2345223738}) 2024-09-16T14:54:03.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1072652667}) 2024-09-16T14:54:03.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1814000654}) 2024-09-16T14:54:03.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1072652667}) 2024-09-16T14:54:03.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2377624800}) 2024-09-16T14:54:03.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2696582480}) 2024-09-16T14:54:03.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170190616}) 2024-09-16T14:54:03.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2696582480}) 2024-09-16T14:54:03.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.482+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1574993857}) 2024-09-16T14:54:03.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.486+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.486+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=153244453}) 2024-09-16T14:54:03.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.486+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058028151}) 2024-09-16T14:54:03.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.486+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.486+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=153244453}) 2024-09-16T14:54:03.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.486+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2720306596}) 2024-09-16T14:54:03.491 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.486+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.491 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.486+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4266877297}) 2024-09-16T14:54:03.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.486+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1501326255}) 2024-09-16T14:54:03.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.486+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.486+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4266877297}) 2024-09-16T14:54:03.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.486+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2538062211}) 2024-09-16T14:54:03.495 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.490+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.495 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.490+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3045993724}) 2024-09-16T14:54:03.495 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.490+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=631101453}) 2024-09-16T14:54:03.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.490+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.490+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3045993724}) 2024-09-16T14:54:03.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.490+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=536174929}) 2024-09-16T14:54:03.499 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.494+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.499 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.494+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2798254426}) 2024-09-16T14:54:03.499 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.494+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077291919}) 2024-09-16T14:54:03.499 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.494+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.499 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.494+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2798254426}) 2024-09-16T14:54:03.499 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.494+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2231505266}) 2024-09-16T14:54:03.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.498+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.498+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3726591607}) 2024-09-16T14:54:03.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.498+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1984473219}) 2024-09-16T14:54:03.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.498+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.498+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3726591607}) 2024-09-16T14:54:03.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.498+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2727676972}) 2024-09-16T14:54:03.503 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.498+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.503 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.498+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=827683532}) 2024-09-16T14:54:03.503 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.498+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3033202487}) 2024-09-16T14:54:03.503 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.498+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.503 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.498+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=827683532}) 2024-09-16T14:54:03.503 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.498+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=977467591}) 2024-09-16T14:54:03.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.502+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.502+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2350860840}) 2024-09-16T14:54:03.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.502+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1790819060}) 2024-09-16T14:54:03.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.502+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.502+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2350860840}) 2024-09-16T14:54:03.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.502+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2438927029}) 2024-09-16T14:54:03.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.502+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.502+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2977720223}) 2024-09-16T14:54:03.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.502+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=137293678}) 2024-09-16T14:54:03.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.502+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.502+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2977720223}) 2024-09-16T14:54:03.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.502+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3074271276}) 2024-09-16T14:54:03.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.506+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.506+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=49588685}) 2024-09-16T14:54:03.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.506+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1084639425}) 2024-09-16T14:54:03.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.506+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.506+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=49588685}) 2024-09-16T14:54:03.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.506+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3720810512}) 2024-09-16T14:54:03.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.506+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.506+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3081240203}) 2024-09-16T14:54:03.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.506+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=785636629}) 2024-09-16T14:54:03.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.506+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.506+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3081240203}) 2024-09-16T14:54:03.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.506+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1011952996}) 2024-09-16T14:54:03.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.510+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.510+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=921921226}) 2024-09-16T14:54:03.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.510+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953932513}) 2024-09-16T14:54:03.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.510+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.510+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=921921226}) 2024-09-16T14:54:03.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.510+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=982639424}) 2024-09-16T14:54:03.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.510+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.510+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2604097417}) 2024-09-16T14:54:03.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.510+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=593009307}) 2024-09-16T14:54:03.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.510+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.510+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2604097417}) 2024-09-16T14:54:03.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.510+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=844344245}) 2024-09-16T14:54:03.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.514+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.514+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2376615857}) 2024-09-16T14:54:03.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.514+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3835090839}) 2024-09-16T14:54:03.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.514+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.514+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2376615857}) 2024-09-16T14:54:03.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.514+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=520773402}) 2024-09-16T14:54:03.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.514+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.514+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1372143441}) 2024-09-16T14:54:03.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.514+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818601070}) 2024-09-16T14:54:03.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.514+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.514+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1372143441}) 2024-09-16T14:54:03.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.514+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4192865570}) 2024-09-16T14:54:03.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.518+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.518+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3462998435}) 2024-09-16T14:54:03.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.518+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1512661100}) 2024-09-16T14:54:03.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.518+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.518+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3462998435}) 2024-09-16T14:54:03.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.518+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1883856729}) 2024-09-16T14:54:03.523 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.518+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.523 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.518+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2484221728}) 2024-09-16T14:54:03.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.518+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=195893149}) 2024-09-16T14:54:03.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.518+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.518+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2484221728}) 2024-09-16T14:54:03.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.518+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=752910911}) 2024-09-16T14:54:03.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.522+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.522+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1445685910}) 2024-09-16T14:54:03.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.522+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4164293691}) 2024-09-16T14:54:03.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.522+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.522+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1445685910}) 2024-09-16T14:54:03.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.522+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3503778152}) 2024-09-16T14:54:03.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.522+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.522+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1191792373}) 2024-09-16T14:54:03.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.522+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1966745521}) 2024-09-16T14:54:03.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.522+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.528 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.522+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1191792373}) 2024-09-16T14:54:03.528 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.522+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3636928409}) 2024-09-16T14:54:03.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.526+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.526+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4232214532}) 2024-09-16T14:54:03.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.526+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=691789283}) 2024-09-16T14:54:03.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.526+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.526+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4232214532}) 2024-09-16T14:54:03.530 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.526+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=881952576}) 2024-09-16T14:54:03.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.526+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.526+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4052042338}) 2024-09-16T14:54:03.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.526+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2193761862}) 2024-09-16T14:54:03.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.526+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.526+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4052042338}) 2024-09-16T14:54:03.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.526+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1003767895}) 2024-09-16T14:54:03.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.530+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.530+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3169056881}) 2024-09-16T14:54:03.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.530+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828450470}) 2024-09-16T14:54:03.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.530+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.530+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3169056881}) 2024-09-16T14:54:03.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.530+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2998139759}) 2024-09-16T14:54:03.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.530+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.530+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=644787601}) 2024-09-16T14:54:03.535 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.530+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2257232235}) 2024-09-16T14:54:03.535 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.530+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.535 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.530+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=644787601}) 2024-09-16T14:54:03.535 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.530+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=537733827}) 2024-09-16T14:54:03.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.534+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.534+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3268440846}) 2024-09-16T14:54:03.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.534+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2898160409}) 2024-09-16T14:54:03.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.534+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.534+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3268440846}) 2024-09-16T14:54:03.537 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.534+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2388082213}) 2024-09-16T14:54:03.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.534+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.534+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=457675306}) 2024-09-16T14:54:03.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.534+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4184188749}) 2024-09-16T14:54:03.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.534+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.534+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=457675306}) 2024-09-16T14:54:03.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.534+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=557198831}) 2024-09-16T14:54:03.540 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.540 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=48085694}) 2024-09-16T14:54:03.540 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=908400805}) 2024-09-16T14:54:03.540 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.540 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=48085694}) 2024-09-16T14:54:03.540 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4113672360}) 2024-09-16T14:54:03.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.541 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3259633186}) 2024-09-16T14:54:03.542 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4216290222}) 2024-09-16T14:54:03.542 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.542 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3259633186}) 2024-09-16T14:54:03.542 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=932350722}) 2024-09-16T14:54:03.543 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.543 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=725652009}) 2024-09-16T14:54:03.543 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=367674187}) 2024-09-16T14:54:03.543 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.543 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=725652009}) 2024-09-16T14:54:03.544 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.538+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1966102574}) 2024-09-16T14:54:03.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.542+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.542+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1976882235}) 2024-09-16T14:54:03.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.542+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3825894830}) 2024-09-16T14:54:03.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.542+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.542+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1976882235}) 2024-09-16T14:54:03.545 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.542+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1795072004}) 2024-09-16T14:54:03.548 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.546+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.548 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.546+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=45,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=7} crc {logm=2753544076,mdsmap=1531151339,mgr=1291789651,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=4258590676,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1320303861}) 2024-09-16T14:54:03.548 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.546+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=44,mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=9} crc {logm=1097082476,mdsmap=2826658659,mgr=4284542101,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=4258590676,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=398505062}) 2024-09-16T14:54:03.548 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.546+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.548 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.546+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=45,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=7} crc {logm=2753544076,mdsmap=1531151339,mgr=1291789651,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=4258590676,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1320303861}) 2024-09-16T14:54:03.548 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.546+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=44,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=7} crc {logm=3401338599,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=4258590676,mon_config_key=3334210756,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1320303861}) 2024-09-16T14:54:03.562 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.558+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.562 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.558+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=46} crc {osdmap=1064755661}) 2024-09-16T14:54:03.563 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.558+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=47} crc {osdmap=2044736475}) 2024-09-16T14:54:03.563 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.558+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:03.563 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.558+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=46} crc {osdmap=1064755661}) 2024-09-16T14:54:03.563 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:03.558+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=44} crc {osdmap=2411325077}) 2024-09-16T14:54:04.434 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:54:04.434 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:54:04.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 10 monclient: get_monmap_and_config 2024-09-16T14:54:04.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 10 monclient: build_initial_monmap 2024-09-16T14:54:04.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 10 monclient: monmap: 2024-09-16T14:54:04.569 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:04.569 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:04.569 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:04.570555+0000 2024-09-16T14:54:04.569 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:04.570555+0000 2024-09-16T14:54:04.569 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:04.569 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:04.569 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:04.569 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:04.569 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:04.570 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:04.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 1 Processor -- start 2024-09-16T14:54:04.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 1 -- start start 2024-09-16T14:54:04.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 10 monclient: init 2024-09-16T14:54:04.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:04.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:54:04.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 10 monclient(hunting): picked mon.noname-b con 0x7f0cf00a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:04.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 10 monclient(hunting): picked mon.noname-a con 0x7f0cf0099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:04.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 10 monclient(hunting): picked mon.noname-c con 0x7f0cf00a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:04.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf0098ea0 con 0x7f0cf0099b60 2024-09-16T14:54:04.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf00997b0 con 0x7f0cf00a26f0 2024-09-16T14:54:04.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf0099460 con 0x7f0cf00a1c00 2024-09-16T14:54:04.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:04.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 10 monclient(hunting): authenticate will time out at 1668.690762s 2024-09-16T14:54:04.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cef7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0cf00a1c00 0x7f0cf00a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47762/0 (socket says 172.21.15.26:47762) 2024-09-16T14:54:04.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cef7fe640 1 -- 172.21.15.26:0/1488746040 learned_addr learned my addr 172.21.15.26:0/1488746040 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:04.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 1 -- 172.21.15.26:0/1488746040 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828022189 0 0) 0x7f0cf0099460 con 0x7f0cf00a1c00 2024-09-16T14:54:04.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:54:04.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:04.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:04.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient(hunting): my global_id is 234236 2024-09-16T14:54:04.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 1 -- 172.21.15.26:0/1488746040 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cdc003530 con 0x7f0cf00a1c00 2024-09-16T14:54:04.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 1 -- 172.21.15.26:0/1488746040 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061976285 0 0) 0x7f0cf0098ea0 con 0x7f0cf0099b60 2024-09-16T14:54:04.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:54:04.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:04.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:04.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient(hunting): my global_id is 194544 2024-09-16T14:54:04.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 1 -- 172.21.15.26:0/1488746040 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf0099460 con 0x7f0cf0099b60 2024-09-16T14:54:04.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 1 -- 172.21.15.26:0/1488746040 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2064975024 0 0) 0x7f0cdc003530 con 0x7f0cf00a1c00 2024-09-16T14:54:04.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 1 -- 172.21.15.26:0/1488746040 >> v1:172.21.15.189:6789/0 conn(0x7f0cf00a26f0 legacy=0x7f0cf013d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:04.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 1 -- 172.21.15.26:0/1488746040 >> v1:172.21.15.26:6789/0 conn(0x7f0cf0099b60 legacy=0x7f0cf0139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:04.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:54:04.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 1 monclient: found mon.noname-b 2024-09-16T14:54:04.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:04.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:54:04.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 1 -- 172.21.15.26:0/1488746040 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf00a34b0 con 0x7f0cf00a1c00 2024-09-16T14:54:04.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient: _finish_auth 0 2024-09-16T14:54:04.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:54:04.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:04.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 1 -- 172.21.15.26:0/1488746040 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0ce00019a0 con 0x7f0cf00a1c00 2024-09-16T14:54:04.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:04.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:54:04.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient: dump: 2024-09-16T14:54:04.575 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:04.575 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:04.575 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.575 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.575 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:04.575 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:04.575 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:04.575 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:04.575 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:04.575 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:04.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 5 monclient: authenticate success, global_id 234236 2024-09-16T14:54:04.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:04.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 1 -- 172.21.15.26:0/1488746040 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ce0001c40 con 0x7f0cf00a1c00 2024-09-16T14:54:04.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:04.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:04.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cf5d44640 4 set_mon_vals no callback set 2024-09-16T14:54:04.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 1 -- 172.21.15.26:0/1488746040 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0ce0002010 con 0x7f0cf00a1c00 2024-09-16T14:54:04.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:04.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:04.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.566+0000 7f0cee7fc640 10 monclient: dump: 2024-09-16T14:54:04.576 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:04.576 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:04.576 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.577 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.577 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:04.577 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:04.577 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:04.577 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:04.577 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:04.577 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:04.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 monclient: shutdown 2024-09-16T14:54:04.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:04.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 1 -- 172.21.15.26:0/1488746040 >> v1:172.21.15.26:6790/0 conn(0x7f0cf00a1c00 legacy=0x7f0cf00a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:04.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 1 -- 172.21.15.26:0/1488746040 shutdown_connections 2024-09-16T14:54:04.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 1 -- 172.21.15.26:0/1488746040 >> 172.21.15.26:0/1488746040 conn(0x7f0cf00a3d00 msgr2=0x7f0cf00a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:04.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 1 -- 172.21.15.26:0/1488746040 shutdown_connections 2024-09-16T14:54:04.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 1 -- 172.21.15.26:0/1488746040 wait complete. 2024-09-16T14:54:04.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 monclient: build_initial_monmap 2024-09-16T14:54:04.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 monclient: monmap: 2024-09-16T14:54:04.581 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:04.581 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:04.581 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:04.583 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:04.583 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:04.584 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:04.584 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:04.584 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:04.584 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:04.584 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:04.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 1 Processor -- start 2024-09-16T14:54:04.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 1 -- start start 2024-09-16T14:54:04.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 monclient: init 2024-09-16T14:54:04.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:04.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:54:04.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 monclient(hunting): picked mon.noname-a con 0x7f0cf0099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:04.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 monclient(hunting): picked mon.noname-b con 0x7f0cf00a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:04.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 monclient(hunting): picked mon.noname-c con 0x7f0cf00a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:04.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf011eb30 con 0x7f0cf0099b60 2024-09-16T14:54:04.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf013cba0 con 0x7f0cf00a1c00 2024-09-16T14:54:04.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf009eb40 con 0x7f0cf00a26f0 2024-09-16T14:54:04.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:04.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 monclient(hunting): authenticate will time out at 1668.694059s 2024-09-16T14:54:04.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ceffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0cf00a26f0 0x7f0cf01d19c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47776/0 (socket says 172.21.15.26:47776) 2024-09-16T14:54:04.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ceffff640 1 -- 172.21.15.26:0/2738622736 learned_addr learned my addr 172.21.15.26:0/2738622736 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:04.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2406360422 0 0) 0x7f0cf009eb40 con 0x7f0cf00a26f0 2024-09-16T14:54:04.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:04.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:04.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:04.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): my global_id is 234239 2024-09-16T14:54:04.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd0003530 con 0x7f0cf00a26f0 2024-09-16T14:54:04.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622174092 0 0) 0x7f0cf011eb30 con 0x7f0cf0099b60 2024-09-16T14:54:04.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:04.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:04.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:04.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): my global_id is 194547 2024-09-16T14:54:04.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf009eb40 con 0x7f0cf0099b60 2024-09-16T14:54:04.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2419759653 0 0) 0x7f0cd0003530 con 0x7f0cf00a26f0 2024-09-16T14:54:04.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cf011eb30 con 0x7f0cf00a26f0 2024-09-16T14:54:04.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0ce80018c0 con 0x7f0cf00a26f0 2024-09-16T14:54:04.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:04.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:04.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): dump: 2024-09-16T14:54:04.587 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:04.587 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:04.587 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.588 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.588 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:04.588 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:04.588 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:04.588 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:04.588 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:04.588 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:04.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:04.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1300705324 0 0) 0x7f0cf009eb40 con 0x7f0cf0099b60 2024-09-16T14:54:04.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cd0003530 con 0x7f0cf0099b60 2024-09-16T14:54:04.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0ce0001f50 con 0x7f0cf0099b60 2024-09-16T14:54:04.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:04.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:04.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): dump: 2024-09-16T14:54:04.589 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:04.589 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:04.589 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.589 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.589 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:04.589 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:04.589 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:04.589 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:04.589 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:04.589 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:04.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301828668 0 0) 0x7f0cf013cba0 con 0x7f0cf00a1c00 2024-09-16T14:54:04.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:04.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:04.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:04.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient(hunting): my global_id is 244126 2024-09-16T14:54:04.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf009eb40 con 0x7f0cf00a1c00 2024-09-16T14:54:04.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 525944863 0 0) 0x7f0cf011eb30 con 0x7f0cf00a26f0 2024-09-16T14:54:04.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 >> v1:172.21.15.189:6789/0 conn(0x7f0cf00a1c00 legacy=0x7f0cf013c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:04.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 >> v1:172.21.15.26:6789/0 conn(0x7f0cf0099b60 legacy=0x7f0cf013bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:04.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:54:04.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 monclient: found mon.c 2024-09-16T14:54:04.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:04.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:04.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf009f070 con 0x7f0cf00a26f0 2024-09-16T14:54:04.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient: _finish_auth 0 2024-09-16T14:54:04.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:54:04.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:04.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 5 monclient: authenticate success, global_id 234239 2024-09-16T14:54:04.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 monclient: _renew_subs 2024-09-16T14:54:04.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:04.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 1 -- 172.21.15.26:0/2738622736 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf009e3f0 con 0x7f0cf00a26f0 2024-09-16T14:54:04.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 monclient: _renew_subs 2024-09-16T14:54:04.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:04.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 1 -- 172.21.15.26:0/2738622736 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf013e320 con 0x7f0cf00a26f0 2024-09-16T14:54:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ce8003620 con 0x7f0cf00a26f0 2024-09-16T14:54:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0ce8003930 con 0x7f0cf00a26f0 2024-09-16T14:54:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ced7fa640 4 set_mon_vals no callback set 2024-09-16T14:54:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0ce7fff640 10 monclient: dump: 2024-09-16T14:54:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:04.593 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.570+0000 7f0cf5d44640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.574+0000 7f0cf5d44640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.574+0000 7f0cf5d44640 1 -- 172.21.15.26:0/2738622736 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd4005050 con 0x7f0cf00a26f0 2024-09-16T14:54:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.574+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0ce8003b10 con 0x7f0cf00a26f0 2024-09-16T14:54:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.574+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f0ce8043e20 con 0x7f0cf00a26f0 2024-09-16T14:54:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.574+0000 7f0cef7fe640 10 monclient: get_auth_request con 0x7f0cd0036440 auth_method 0 2024-09-16T14:54:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.578+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0ce8014780 con 0x7f0cf00a26f0 2024-09-16T14:54:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.578+0000 7f0ce7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:04.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.578+0000 7f0ce7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:04.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.798+0000 7f0cf5d44640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:54:04.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.798+0000 7f0cf5d44640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:04.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.798+0000 7f0cf5d44640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:04.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.798+0000 7f0cf5d44640 1 -- 172.21.15.26:0/2738622736 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0cd4005880 con 0x7f0cf00a26f0 2024-09-16T14:54:04.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.798+0000 7f0ce7fff640 1 -- 172.21.15.26:0/2738622736 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2256283365) 0x7f0ce8012bf0 con 0x7f0cf00a26f0 2024-09-16T14:54:04.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.798+0000 7f0ce7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:04.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.798+0000 7f0ce7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:04.805 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:54:04.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.802+0000 7f0cf5d44640 1 -- 172.21.15.26:0/2738622736 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0cd0036440 legacy=0x7f0cd0038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:04.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.802+0000 7f0cf5d44640 10 monclient: shutdown 2024-09-16T14:54:04.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.802+0000 7f0cf5d44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:04.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.802+0000 7f0cf5d44640 1 -- 172.21.15.26:0/2738622736 >> v1:172.21.15.26:6790/0 conn(0x7f0cf00a26f0 legacy=0x7f0cf01d19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:04.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.802+0000 7f0cf5d44640 1 -- 172.21.15.26:0/2738622736 shutdown_connections 2024-09-16T14:54:04.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.802+0000 7f0cf5d44640 1 -- 172.21.15.26:0/2738622736 >> 172.21.15.26:0/2738622736 conn(0x7f0cf00a3d00 msgr2=0x7f0cf00a0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:04.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.802+0000 7f0cf5d44640 1 -- 172.21.15.26:0/2738622736 shutdown_connections 2024-09-16T14:54:04.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.802+0000 7f0cf5d44640 1 -- 172.21.15.26:0/2738622736 wait complete. 2024-09-16T14:54:04.821 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:54:04.821 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:54:04.821 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:54:04.821 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:54:04.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.930+0000 7f74c9d93640 10 monclient: get_monmap_and_config 2024-09-16T14:54:04.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.930+0000 7f74c9d93640 10 monclient: build_initial_monmap 2024-09-16T14:54:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.930+0000 7f74c9d93640 10 monclient: monmap: 2024-09-16T14:54:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:04.937114+0000 2024-09-16T14:54:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:04.937114+0000 2024-09-16T14:54:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:04.936 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 1 Processor -- start 2024-09-16T14:54:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 1 -- start start 2024-09-16T14:54:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 10 monclient: init 2024-09-16T14:54:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:54:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 10 monclient(hunting): picked mon.noname-a con 0x7f74c409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 10 monclient(hunting): picked mon.noname-c con 0x7f74c4132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 10 monclient(hunting): picked mon.noname-b con 0x7f74c4138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c409d260 con 0x7f74c409cb80 2024-09-16T14:54:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c4132a80 con 0x7f74c4132650 2024-09-16T14:54:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c409cf60 con 0x7f74c4138ff0 2024-09-16T14:54:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 10 monclient(hunting): authenticate will time out at 1669.057119s 2024-09-16T14:54:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c3fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f74c4138ff0 0x7f74c413b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47786/0 (socket says 172.21.15.26:47786) 2024-09-16T14:54:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c37fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f74c409cb80 0x7f74c40717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54006/0 (socket says 172.21.15.26:54006) 2024-09-16T14:54:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c3fff640 1 -- 172.21.15.26:0/3833131007 learned_addr learned my addr 172.21.15.26:0/3833131007 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 1 -- 172.21.15.26:0/3833131007 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609761178 0 0) 0x7f74c409d260 con 0x7f74c409cb80 2024-09-16T14:54:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient(hunting): my global_id is 194550 2024-09-16T14:54:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 1 -- 172.21.15.26:0/3833131007 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b0003530 con 0x7f74c409cb80 2024-09-16T14:54:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 1 -- 172.21.15.26:0/3833131007 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761772546 0 0) 0x7f74c409cf60 con 0x7f74c4138ff0 2024-09-16T14:54:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient(hunting): my global_id is 234242 2024-09-16T14:54:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 1 -- 172.21.15.26:0/3833131007 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c409d260 con 0x7f74c4138ff0 2024-09-16T14:54:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 1 -- 172.21.15.26:0/3833131007 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3506220087 0 0) 0x7f74b0003530 con 0x7f74c409cb80 2024-09-16T14:54:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 1 -- 172.21.15.26:0/3833131007 >> v1:172.21.15.26:6790/0 conn(0x7f74c4138ff0 legacy=0x7f74c413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 1 -- 172.21.15.26:0/3833131007 >> v1:172.21.15.189:6789/0 conn(0x7f74c4132650 legacy=0x7f74c41378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:54:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 1 monclient: found mon.noname-a 2024-09-16T14:54:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:54:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 1 -- 172.21.15.26:0/3833131007 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c409f2d0 con 0x7f74c409cb80 2024-09-16T14:54:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:54:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:54:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 1 -- 172.21.15.26:0/3833131007 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74b4001550 con 0x7f74c409cb80 2024-09-16T14:54:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 5 monclient: authenticate success, global_id 194550 2024-09-16T14:54:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient: dump: 2024-09-16T14:54:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:04.948 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 1 -- 172.21.15.26:0/3833131007 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74b4001770 con 0x7f74c409cb80 2024-09-16T14:54:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 1 -- 172.21.15.26:0/3833131007 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74b4001a80 con 0x7f74c409cb80 2024-09-16T14:54:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c2ffd640 10 monclient: dump: 2024-09-16T14:54:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:04.949 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 4 set_mon_vals no callback set 2024-09-16T14:54:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 10 monclient: shutdown 2024-09-16T14:54:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 1 -- 172.21.15.26:0/3833131007 >> v1:172.21.15.26:6789/0 conn(0x7f74c409cb80 legacy=0x7f74c40717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 1 -- 172.21.15.26:0/3833131007 shutdown_connections 2024-09-16T14:54:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 1 -- 172.21.15.26:0/3833131007 >> 172.21.15.26:0/3833131007 conn(0x7f74c409fb50 msgr2=0x7f74c409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 1 -- 172.21.15.26:0/3833131007 shutdown_connections 2024-09-16T14:54:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 1 -- 172.21.15.26:0/3833131007 wait complete. 2024-09-16T14:54:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 10 monclient: build_initial_monmap 2024-09-16T14:54:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 10 monclient: monmap: 2024-09-16T14:54:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:04.951 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:04.951 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:04.951 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:04.951 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:04.951 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:04.951 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:04.952 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:04.952 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:04.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 1 Processor -- start 2024-09-16T14:54:04.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 1 -- start start 2024-09-16T14:54:04.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.934+0000 7f74c9d93640 10 monclient: init 2024-09-16T14:54:04.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:04.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:54:04.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 10 monclient(hunting): picked mon.noname-b con 0x7f74c409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:04.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 10 monclient(hunting): picked mon.noname-a con 0x7f74c4132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:04.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 10 monclient(hunting): picked mon.noname-c con 0x7f74c4138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:04.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c409a240 con 0x7f74c4132650 2024-09-16T14:54:04.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c413c620 con 0x7f74c409cb80 2024-09-16T14:54:04.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c409b370 con 0x7f74c4138ff0 2024-09-16T14:54:04.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:04.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 10 monclient(hunting): authenticate will time out at 1669.060496s 2024-09-16T14:54:04.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c3fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f74c4138ff0 0x7f74c41d1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47796/0 (socket says 172.21.15.26:47796) 2024-09-16T14:54:04.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74bbfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f74c4132650 0x7f74c409ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54020/0 (socket says 172.21.15.26:54020) 2024-09-16T14:54:04.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c3fff640 1 -- 172.21.15.26:0/1010008838 learned_addr learned my addr 172.21.15.26:0/1010008838 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:04.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819818011 0 0) 0x7f74c409a240 con 0x7f74c4132650 2024-09-16T14:54:04.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:54:04.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:04.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:04.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): my global_id is 194553 2024-09-16T14:54:04.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74a4003530 con 0x7f74c4132650 2024-09-16T14:54:04.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620334858 0 0) 0x7f74c409b370 con 0x7f74c4138ff0 2024-09-16T14:54:04.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:54:04.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:04.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:04.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): my global_id is 234245 2024-09-16T14:54:04.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c409a240 con 0x7f74c4138ff0 2024-09-16T14:54:04.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1132464911 0 0) 0x7f74c409a240 con 0x7f74c4138ff0 2024-09-16T14:54:04.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74c409b370 con 0x7f74c4138ff0 2024-09-16T14:54:04.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2784882368 0 0) 0x7f74a4003530 con 0x7f74c4132650 2024-09-16T14:54:04.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74c409a240 con 0x7f74c4132650 2024-09-16T14:54:04.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74bc001dc0 con 0x7f74c4138ff0 2024-09-16T14:54:04.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:04.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:04.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): dump: 2024-09-16T14:54:04.954 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:04.955 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:04.955 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.955 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.955 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:04.955 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:04.955 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:04.955 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:04.955 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:04.955 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:04.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:04.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74ac001840 con 0x7f74c4132650 2024-09-16T14:54:04.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:04.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:04.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): dump: 2024-09-16T14:54:04.956 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:04.956 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:04.956 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.956 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.956 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:04.956 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:04.956 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:04.956 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:04.956 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:04.956 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:04.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2295504982 0 0) 0x7f74c409b370 con 0x7f74c4138ff0 2024-09-16T14:54:04.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 >> v1:172.21.15.189:6789/0 conn(0x7f74c409cb80 legacy=0x7f74c409a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:04.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 >> v1:172.21.15.26:6789/0 conn(0x7f74c4132650 legacy=0x7f74c409ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:04.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:54:04.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 monclient: found mon.c 2024-09-16T14:54:04.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:04.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:04.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c4138b60 con 0x7f74c4138ff0 2024-09-16T14:54:04.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient: _finish_auth 0 2024-09-16T14:54:04.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:54:04.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:04.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 5 monclient: authenticate success, global_id 234245 2024-09-16T14:54:04.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 10 monclient: _renew_subs 2024-09-16T14:54:04.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:04.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 1 -- 172.21.15.26:0/1010008838 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c406da20 con 0x7f74c4138ff0 2024-09-16T14:54:04.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 10 monclient: _renew_subs 2024-09-16T14:54:04.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:04.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 1 -- 172.21.15.26:0/1010008838 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c409c380 con 0x7f74c4138ff0 2024-09-16T14:54:04.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74bc0019c0 con 0x7f74c4138ff0 2024-09-16T14:54:04.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:04.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74bc004d90 con 0x7f74c4138ff0 2024-09-16T14:54:04.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:04.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:04.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c27fc640 4 set_mon_vals no callback set 2024-09-16T14:54:04.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient: dump: 2024-09-16T14:54:04.958 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:04.958 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:04.958 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.958 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.959 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:04.959 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:04.959 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:04.959 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:04.959 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:04.959 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:04.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:04.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c9d93640 1 -- 172.21.15.26:0/1010008838 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f74a8006590 con 0x7f74a8002830 2024-09-16T14:54:04.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f74bc00f800 con 0x7f74c4138ff0 2024-09-16T14:54:04.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f74bc0440b0 con 0x7f74c4138ff0 2024-09-16T14:54:04.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251592045 0 0) 0x7f74a8006590 con 0x7f74a8002830 2024-09-16T14:54:04.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:54:04.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:04.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient(hunting): my global_id is 194556 2024-09-16T14:54:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c409b370 con 0x7f74a8002830 2024-09-16T14:54:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 755135058 0 0) 0x7f74c409b370 con 0x7f74a8002830 2024-09-16T14:54:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74a8006590 con 0x7f74a8002830 2024-09-16T14:54:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74b4001080 con 0x7f74a8002830 2024-09-16T14:54:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient: dump: 2024-09-16T14:54:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.961 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:04.961 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:04.961 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:04.961 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:04.961 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:04.961 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:04.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74bbfff640 10 monclient: get_auth_request con 0x7f74a40361f0 auth_method 0 2024-09-16T14:54:04.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3846196503 0 0) 0x7f74a8006590 con 0x7f74a8002830 2024-09-16T14:54:04.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f74a800a0e0 con 0x7f74a8002830 2024-09-16T14:54:04.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f74b40013a0 con 0x7f74a8002830 2024-09-16T14:54:04.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:04.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:04.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 >> v1:172.21.15.26:6789/0 conn(0x7f74a8002830 legacy=0x7f74a8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:04.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.938+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 >> v1:172.21.15.26:6789/0 conn(0x7f74a8002830 legacy=0x7f74a8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c9d93640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:54:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c9d93640 1 -- 172.21.15.26:0/1010008838 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f74a8009da0 con 0x7f74a80090c0 2024-09-16T14:54:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148282299 0 0) 0x7f74a8009da0 con 0x7f74a80090c0 2024-09-16T14:54:04.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:54:04.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:04.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:04.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 10 monclient(hunting): my global_id is 194559 2024-09-16T14:54:04.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74a8006590 con 0x7f74a80090c0 2024-09-16T14:54:04.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2179966115 0 0) 0x7f74a8006590 con 0x7f74a80090c0 2024-09-16T14:54:04.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74a8009da0 con 0x7f74a80090c0 2024-09-16T14:54:04.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74b4004a10 con 0x7f74a80090c0 2024-09-16T14:54:04.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:04.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:04.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 10 monclient: dump: 2024-09-16T14:54:04.965 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:04.965 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:04.965 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.965 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:04.965 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:04.965 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:04.965 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:04.965 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:04.965 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:04.965 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:04.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1344543881 0 0) 0x7f74a8009da0 con 0x7f74a80090c0 2024-09-16T14:54:04.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f74a800c290 con 0x7f74a80090c0 2024-09-16T14:54:04.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 745837817) 0x7f74b4004d50 con 0x7f74a80090c0 2024-09-16T14:54:04.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:54:04.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:04.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 >> v1:172.21.15.26:6789/0 conn(0x7f74a80090c0 legacy=0x7f74a8009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:04.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c0ff9640 1 -- 172.21.15.26:0/1010008838 >> v1:172.21.15.26:6789/0 conn(0x7f74a80090c0 legacy=0x7f74a8009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:04.966 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:54:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:54:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 108, 2024-09-16T14:54:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:54:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:54:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:54:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:54:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:54:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:54:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:54:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:54:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:54:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:54:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:54:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:54:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:54:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:54:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:54:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:54:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:54:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:54:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:54:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:54:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:54:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:54:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:54:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:54:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:54:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:04.971 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:54:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:54:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:54:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:04.972 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:54:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:04.973 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:54:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:54:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:04.974 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:54:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:54:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:54:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:04.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:54:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:54:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:54:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:04.976 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:54:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:54:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:54:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:54:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:04.977 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:04.978 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:54:04.978 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:54:04.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c9d93640 1 -- 172.21.15.26:0/1010008838 >> v1:172.21.15.189:6812/3196161582 conn(0x7f74a40361f0 legacy=0x7f74a40386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:04.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c9d93640 10 monclient: shutdown 2024-09-16T14:54:04.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c9d93640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:04.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c9d93640 1 -- 172.21.15.26:0/1010008838 >> v1:172.21.15.26:6790/0 conn(0x7f74c4138ff0 legacy=0x7f74c41d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:04.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c3fff640 1 -- 172.21.15.26:0/1010008838 reap_dead start 2024-09-16T14:54:04.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c3fff640 1 -- 172.21.15.26:0/1010008838 reap_dead start 2024-09-16T14:54:04.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c9d93640 1 -- 172.21.15.26:0/1010008838 shutdown_connections 2024-09-16T14:54:04.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c9d93640 1 -- 172.21.15.26:0/1010008838 >> 172.21.15.26:0/1010008838 conn(0x7f74c409fb50 msgr2=0x7f74c40a1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:04.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c9d93640 1 -- 172.21.15.26:0/1010008838 shutdown_connections 2024-09-16T14:54:04.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:04.946+0000 7f74c9d93640 1 -- 172.21.15.26:0/1010008838 wait complete. 2024-09-16T14:54:04.980 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:54:04.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:54:04.981 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:54:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 10 monclient: get_monmap_and_config 2024-09-16T14:54:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 10 monclient: build_initial_monmap 2024-09-16T14:54:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 10 monclient: monmap: 2024-09-16T14:54:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:05.094950+0000 2024-09-16T14:54:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:05.094950+0000 2024-09-16T14:54:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:05.094 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 1 Processor -- start 2024-09-16T14:54:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 1 -- start start 2024-09-16T14:54:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 10 monclient: init 2024-09-16T14:54:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:54:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 10 monclient(hunting): picked mon.noname-b con 0x7f1b1809a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 10 monclient(hunting): picked mon.noname-a con 0x7f1b1809d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 10 monclient(hunting): picked mon.noname-c con 0x7f1b1809b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1809dc60 con 0x7f1b1809d1b0 2024-09-16T14:54:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b18099ae0 con 0x7f1b1809b040 2024-09-16T14:54:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1809b8f0 con 0x7f1b1809a550 2024-09-16T14:54:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 10 monclient(hunting): authenticate will time out at 1669.215071s 2024-09-16T14:54:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1cfdb640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1b1809a550 0x7f1b1809a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47808/0 (socket says 172.21.15.26:47808) 2024-09-16T14:54:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1cfdb640 1 -- 172.21.15.26:0/2431232479 learned_addr learned my addr 172.21.15.26:0/2431232479 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 1 -- 172.21.15.26:0/2431232479 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923572697 0 0) 0x7f1b1809dc60 con 0x7f1b1809d1b0 2024-09-16T14:54:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 10 monclient(hunting): my global_id is 194562 2024-09-16T14:54:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 1 -- 172.21.15.26:0/2431232479 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b00005740 con 0x7f1b1809d1b0 2024-09-16T14:54:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 1 -- 172.21.15.26:0/2431232479 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852469478 0 0) 0x7f1b1809b8f0 con 0x7f1b1809a550 2024-09-16T14:54:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 10 monclient(hunting): my global_id is 234248 2024-09-16T14:54:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 1 -- 172.21.15.26:0/2431232479 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b1809dc60 con 0x7f1b1809a550 2024-09-16T14:54:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 1 -- 172.21.15.26:0/2431232479 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 357874528 0 0) 0x7f1b00005740 con 0x7f1b1809d1b0 2024-09-16T14:54:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 1 -- 172.21.15.26:0/2431232479 >> v1:172.21.15.26:6790/0 conn(0x7f1b1809a550 legacy=0x7f1b1809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 1 -- 172.21.15.26:0/2431232479 >> v1:172.21.15.189:6789/0 conn(0x7f1b1809b040 legacy=0x7f1b1813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 10 monclient: _finish_hunting 0 2024-09-16T14:54:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 1 monclient: found mon.noname-a 2024-09-16T14:54:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:54:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 1 -- 172.21.15.26:0/2431232479 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1809bbd0 con 0x7f1b1809d1b0 2024-09-16T14:54:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 10 monclient: _finish_auth 0 2024-09-16T14:54:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 10 monclient: _check_auth_tickets 2024-09-16T14:54:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 1 -- 172.21.15.26:0/2431232479 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b0c001620 con 0x7f1b1809d1b0 2024-09-16T14:54:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b177fe640 10 monclient: dump: 2024-09-16T14:54:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.100 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.090+0000 7f1b1f23f640 5 monclient: authenticate success, global_id 194562 2024-09-16T14:54:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b177fe640 1 -- 172.21.15.26:0/2431232479 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b0c001900 con 0x7f1b1809d1b0 2024-09-16T14:54:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b177fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b177fe640 1 -- 172.21.15.26:0/2431232479 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b0c001cd0 con 0x7f1b1809d1b0 2024-09-16T14:54:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b177fe640 10 monclient: dump: 2024-09-16T14:54:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.102 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 4 set_mon_vals no callback set 2024-09-16T14:54:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 monclient: shutdown 2024-09-16T14:54:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 1 -- 172.21.15.26:0/2431232479 >> v1:172.21.15.26:6789/0 conn(0x7f1b1809d1b0 legacy=0x7f1b18136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 1 -- 172.21.15.26:0/2431232479 shutdown_connections 2024-09-16T14:54:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 1 -- 172.21.15.26:0/2431232479 >> 172.21.15.26:0/2431232479 conn(0x7f1b1809fb50 msgr2=0x7f1b1809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 1 -- 172.21.15.26:0/2431232479 shutdown_connections 2024-09-16T14:54:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 1 -- 172.21.15.26:0/2431232479 wait complete. 2024-09-16T14:54:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 monclient: build_initial_monmap 2024-09-16T14:54:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 monclient: monmap: 2024-09-16T14:54:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:05.108 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 1 Processor -- start 2024-09-16T14:54:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 1 -- start start 2024-09-16T14:54:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 monclient: init 2024-09-16T14:54:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:54:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 monclient(hunting): picked mon.noname-a con 0x7f1b1809a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 monclient(hunting): picked mon.noname-c con 0x7f1b1809b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 monclient(hunting): picked mon.noname-b con 0x7f1b1809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1809a240 con 0x7f1b1809a550 2024-09-16T14:54:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b180a2640 con 0x7f1b1809d1b0 2024-09-16T14:54:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b181b7600 con 0x7f1b1809b040 2024-09-16T14:54:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 monclient(hunting): authenticate will time out at 1669.218218s 2024-09-16T14:54:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b17fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1b1809b040 0x7f1b181a7000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47812/0 (socket says 172.21.15.26:47812) 2024-09-16T14:54:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b17fff640 1 -- 172.21.15.26:0/2345465371 learned_addr learned my addr 172.21.15.26:0/2345465371 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907322435 0 0) 0x7f1b181b7600 con 0x7f1b1809b040 2024-09-16T14:54:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): _negotiate 2024-09-16T14:54:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): my global_id is 234251 2024-09-16T14:54:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b04003530 con 0x7f1b1809b040 2024-09-16T14:54:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645498238 0 0) 0x7f1b1809a240 con 0x7f1b1809a550 2024-09-16T14:54:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): _negotiate 2024-09-16T14:54:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): my global_id is 194565 2024-09-16T14:54:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b181b7600 con 0x7f1b1809a550 2024-09-16T14:54:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4112942849 0 0) 0x7f1b04003530 con 0x7f1b1809b040 2024-09-16T14:54:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b1809a240 con 0x7f1b1809b040 2024-09-16T14:54:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b0c001cd0 con 0x7f1b1809b040 2024-09-16T14:54:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): dump: 2024-09-16T14:54:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.111 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 546359204 0 0) 0x7f1b181b7600 con 0x7f1b1809a550 2024-09-16T14:54:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b04003530 con 0x7f1b1809a550 2024-09-16T14:54:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b08001840 con 0x7f1b1809a550 2024-09-16T14:54:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): dump: 2024-09-16T14:54:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.113 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58265129 0 0) 0x7f1b100015c0 con 0x7f1b1809d1b0 2024-09-16T14:54:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): _negotiate 2024-09-16T14:54:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient(hunting): my global_id is 244138 2024-09-16T14:54:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b181b7600 con 0x7f1b1809d1b0 2024-09-16T14:54:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1851557438 0 0) 0x7f1b1809a240 con 0x7f1b1809b040 2024-09-16T14:54:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 >> v1:172.21.15.189:6789/0 conn(0x7f1b1809d1b0 legacy=0x7f1b181d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 >> v1:172.21.15.26:6789/0 conn(0x7f1b1809a550 legacy=0x7f1b181a68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient: _finish_hunting 0 2024-09-16T14:54:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 monclient: found mon.c 2024-09-16T14:54:05.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:05.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:05.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b180a2080 con 0x7f1b1809b040 2024-09-16T14:54:05.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient: _finish_auth 0 2024-09-16T14:54:05.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient: _check_auth_tickets 2024-09-16T14:54:05.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:05.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 5 monclient: authenticate success, global_id 234251 2024-09-16T14:54:05.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 monclient: _renew_subs 2024-09-16T14:54:05.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:05.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 1 -- 172.21.15.26:0/2345465371 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b181aa8d0 con 0x7f1b1809b040 2024-09-16T14:54:05.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 monclient: _renew_subs 2024-09-16T14:54:05.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:05.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 1 -- 172.21.15.26:0/2345465371 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b181a7710 con 0x7f1b1809b040 2024-09-16T14:54:05.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b0c0039f0 con 0x7f1b1809b040 2024-09-16T14:54:05.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:05.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b0c001220 con 0x7f1b1809b040 2024-09-16T14:54:05.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:05.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b16ffd640 4 set_mon_vals no callback set 2024-09-16T14:54:05.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b157fa640 10 monclient: dump: 2024-09-16T14:54:05.115 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.115 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.115 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.115 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.115 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.115 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.115 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.116 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.116 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.116 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:05.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.094+0000 7f1b1f23f640 1 -- 172.21.15.26:0/2345465371 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1af8006590 con 0x7f1af8002830 2024-09-16T14:54:05.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1b0c003e00 con 0x7f1b1809b040 2024-09-16T14:54:05.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f1b0c043f00 con 0x7f1b1809b040 2024-09-16T14:54:05.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547360747 0 0) 0x7f1af8006590 con 0x7f1af8002830 2024-09-16T14:54:05.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 10 monclient(hunting): _negotiate 2024-09-16T14:54:05.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:05.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:05.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 10 monclient(hunting): my global_id is 234254 2024-09-16T14:54:05.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b1809a240 con 0x7f1af8002830 2024-09-16T14:54:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 273968948 0 0) 0x7f1b1809a240 con 0x7f1af8002830 2024-09-16T14:54:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1af8006590 con 0x7f1af8002830 2024-09-16T14:54:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b08004b80 con 0x7f1af8002830 2024-09-16T14:54:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 10 monclient: dump: 2024-09-16T14:54:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.118 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b1d7dc640 10 monclient: get_auth_request con 0x7f1b040363d0 auth_method 0 2024-09-16T14:54:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2654060515 0 0) 0x7f1af8006590 con 0x7f1af8002830 2024-09-16T14:54:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1af800a0e0 con 0x7f1af8002830 2024-09-16T14:54:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1b08004e20 con 0x7f1af8002830 2024-09-16T14:54:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 >> v1:172.21.15.26:6790/0 conn(0x7f1af8002830 legacy=0x7f1af8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.098+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 >> v1:172.21.15.26:6790/0 conn(0x7f1af8002830 legacy=0x7f1af8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.102+0000 7f1b1f23f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:54:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.102+0000 7f1b1f23f640 1 -- 172.21.15.26:0/2345465371 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1af8009da0 con 0x7f1af80090c0 2024-09-16T14:54:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.102+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771861628 0 0) 0x7f1af8009da0 con 0x7f1af80090c0 2024-09-16T14:54:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.102+0000 7f1b157fa640 10 monclient(hunting): _negotiate 2024-09-16T14:54:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.102+0000 7f1b157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.102+0000 7f1b157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.102+0000 7f1b157fa640 10 monclient(hunting): my global_id is 234257 2024-09-16T14:54:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.102+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af8006590 con 0x7f1af80090c0 2024-09-16T14:54:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.102+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3967877328 0 0) 0x7f1af8006590 con 0x7f1af80090c0 2024-09-16T14:54:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.102+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1af8009da0 con 0x7f1af80090c0 2024-09-16T14:54:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.102+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1b08005360 con 0x7f1af80090c0 2024-09-16T14:54:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.102+0000 7f1b157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.102+0000 7f1b157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.102+0000 7f1b157fa640 10 monclient: dump: 2024-09-16T14:54:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.120 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.102+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3096261615 0 0) 0x7f1af8009da0 con 0x7f1af80090c0 2024-09-16T14:54:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.102+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1af800c290 con 0x7f1af80090c0 2024-09-16T14:54:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.106+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 111452668) 0x7f1b080067a0 con 0x7f1af80090c0 2024-09-16T14:54:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.106+0000 7f1b157fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:54:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.106+0000 7f1b157fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.106+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 >> v1:172.21.15.26:6790/0 conn(0x7f1af80090c0 legacy=0x7f1af8009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.106+0000 7f1b157fa640 1 -- 172.21.15.26:0/2345465371 >> v1:172.21.15.26:6790/0 conn(0x7f1af80090c0 legacy=0x7f1af8009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:05.121 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:54:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:54:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 108, 2024-09-16T14:54:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:54:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:54:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:54:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:54:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:54:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:54:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:54:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:54:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:54:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:54:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:54:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:54:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:54:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:54:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:54:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:54:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:54:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:54:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:54:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:54:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:54:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:54:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:54:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:05.127 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:05.127 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:05.127 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:05.127 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:05.127 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:05.127 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:54:05.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.127 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:54:05.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:05.127 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:05.127 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:05.127 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:05.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:05.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:05.128 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:05.128 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:54:05.128 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:05.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:05.128 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:05.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.128 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:54:05.128 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:54:05.128 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:05.128 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:05.128 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:05.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:05.129 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:05.129 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:05.129 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:05.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:05.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:05.129 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:05.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:54:05.129 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:05.129 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:05.129 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:05.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:54:05.129 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:54:05.130 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:05.130 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:05.130 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:05.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:05.130 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:05.130 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:05.130 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:05.130 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:05.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:05.130 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:05.130 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:54:05.130 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:05.130 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:05.131 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:05.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.131 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:54:05.131 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:54:05.131 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:05.131 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:05.131 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:05.131 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:05.131 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:05.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.131 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:54:05.131 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:54:05.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:05.132 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:05.132 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:05.132 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:05.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:05.132 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:05.132 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:54:05.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:05.132 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:05.132 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:05.132 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:54:05.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:05.132 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:05.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.132 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:54:05.133 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:54:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.106+0000 7f1b1f23f640 1 -- 172.21.15.26:0/2345465371 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1b040363d0 legacy=0x7f1b04038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.106+0000 7f1b1f23f640 10 monclient: shutdown 2024-09-16T14:54:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.106+0000 7f1b1f23f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.106+0000 7f1b1f23f640 1 -- 172.21.15.26:0/2345465371 >> v1:172.21.15.26:6790/0 conn(0x7f1b1809b040 legacy=0x7f1b181a7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.106+0000 7f1b1d7dc640 1 -- 172.21.15.26:0/2345465371 reap_dead start 2024-09-16T14:54:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.106+0000 7f1b1d7dc640 1 -- 172.21.15.26:0/2345465371 reap_dead start 2024-09-16T14:54:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.106+0000 7f1b1f23f640 1 -- 172.21.15.26:0/2345465371 shutdown_connections 2024-09-16T14:54:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.106+0000 7f1b1f23f640 1 -- 172.21.15.26:0/2345465371 >> 172.21.15.26:0/2345465371 conn(0x7f1b1809fb50 msgr2=0x7f1b180a14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.106+0000 7f1b1f23f640 1 -- 172.21.15.26:0/2345465371 shutdown_connections 2024-09-16T14:54:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.106+0000 7f1b1f23f640 1 -- 172.21.15.26:0/2345465371 wait complete. 2024-09-16T14:54:05.135 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:54:05.135 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:54:05.135 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:54:05.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 10 monclient: get_monmap_and_config 2024-09-16T14:54:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 10 monclient: build_initial_monmap 2024-09-16T14:54:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 10 monclient: monmap: 2024-09-16T14:54:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:05.253672+0000 2024-09-16T14:54:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:05.253672+0000 2024-09-16T14:54:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:05.254 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 1 Processor -- start 2024-09-16T14:54:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 1 -- start start 2024-09-16T14:54:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 10 monclient: init 2024-09-16T14:54:05.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:05.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:54:05.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 10 monclient(hunting): picked mon.noname-a con 0x7f5ba00a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:05.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 10 monclient(hunting): picked mon.noname-b con 0x7f5ba0099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:05.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 10 monclient(hunting): picked mon.noname-c con 0x7f5ba00a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:05.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba0098ea0 con 0x7f5ba00a1c00 2024-09-16T14:54:05.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba00997b0 con 0x7f5ba00a26f0 2024-09-16T14:54:05.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba0099460 con 0x7f5ba0099b60 2024-09-16T14:54:05.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:05.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 10 monclient(hunting): authenticate will time out at 1669.373890s 2024-09-16T14:54:05.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9dd9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5ba0099b60 0x7f5ba0139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47844/0 (socket says 172.21.15.26:47844) 2024-09-16T14:54:05.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9e59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5ba00a1c00 0x7f5ba00a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54060/0 (socket says 172.21.15.26:54060) 2024-09-16T14:54:05.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9dd9b640 1 -- 172.21.15.26:0/854225125 learned_addr learned my addr 172.21.15.26:0/854225125 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:05.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 1 -- 172.21.15.26:0/854225125 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569418829 0 0) 0x7f5ba0098ea0 con 0x7f5ba00a1c00 2024-09-16T14:54:05.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient(hunting): _negotiate 2024-09-16T14:54:05.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:05.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:05.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient(hunting): my global_id is 194568 2024-09-16T14:54:05.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 1 -- 172.21.15.26:0/854225125 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b8c003530 con 0x7f5ba00a1c00 2024-09-16T14:54:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 1 -- 172.21.15.26:0/854225125 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930636267 0 0) 0x7f5ba0099460 con 0x7f5ba0099b60 2024-09-16T14:54:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient(hunting): _negotiate 2024-09-16T14:54:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient(hunting): my global_id is 234260 2024-09-16T14:54:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 1 -- 172.21.15.26:0/854225125 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba0098ea0 con 0x7f5ba0099b60 2024-09-16T14:54:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 1 -- 172.21.15.26:0/854225125 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2103019772 0 0) 0x7f5b8c003530 con 0x7f5ba00a1c00 2024-09-16T14:54:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 1 -- 172.21.15.26:0/854225125 >> v1:172.21.15.26:6790/0 conn(0x7f5ba0099b60 legacy=0x7f5ba0139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 1 -- 172.21.15.26:0/854225125 >> v1:172.21.15.189:6789/0 conn(0x7f5ba00a26f0 legacy=0x7f5ba013d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient: _finish_hunting 0 2024-09-16T14:54:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 1 monclient: found mon.noname-a 2024-09-16T14:54:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:54:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 1 -- 172.21.15.26:0/854225125 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba00a34b0 con 0x7f5ba00a1c00 2024-09-16T14:54:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient: _finish_auth 0 2024-09-16T14:54:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient: _check_auth_tickets 2024-09-16T14:54:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 1 -- 172.21.15.26:0/854225125 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5b90001620 con 0x7f5ba00a1c00 2024-09-16T14:54:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient: dump: 2024-09-16T14:54:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.265 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.265 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.265 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.265 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.265 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.265 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 5 monclient: authenticate success, global_id 194568 2024-09-16T14:54:05.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:05.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 1 -- 172.21.15.26:0/854225125 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b900018c0 con 0x7f5ba00a1c00 2024-09-16T14:54:05.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:05.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 1 -- 172.21.15.26:0/854225125 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5b90001c50 con 0x7f5ba00a1c00 2024-09-16T14:54:05.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5b9d59a640 10 monclient: dump: 2024-09-16T14:54:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.266 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 4 set_mon_vals no callback set 2024-09-16T14:54:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 10 monclient: shutdown 2024-09-16T14:54:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 1 -- 172.21.15.26:0/854225125 >> v1:172.21.15.26:6789/0 conn(0x7f5ba00a1c00 legacy=0x7f5ba00a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 1 -- 172.21.15.26:0/854225125 shutdown_connections 2024-09-16T14:54:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 1 -- 172.21.15.26:0/854225125 >> 172.21.15.26:0/854225125 conn(0x7f5ba00a3d00 msgr2=0x7f5ba00a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 1 -- 172.21.15.26:0/854225125 shutdown_connections 2024-09-16T14:54:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.250+0000 7f5ba4fa8640 1 -- 172.21.15.26:0/854225125 wait complete. 2024-09-16T14:54:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 10 monclient: build_initial_monmap 2024-09-16T14:54:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 10 monclient: monmap: 2024-09-16T14:54:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:05.268 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 1 Processor -- start 2024-09-16T14:54:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 1 -- start start 2024-09-16T14:54:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 10 monclient: init 2024-09-16T14:54:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:54:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 10 monclient(hunting): picked mon.noname-c con 0x7f5ba0099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 10 monclient(hunting): picked mon.noname-a con 0x7f5ba00a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 10 monclient(hunting): picked mon.noname-b con 0x7f5ba00a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba009f340 con 0x7f5ba00a1c00 2024-09-16T14:54:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba009e3f0 con 0x7f5ba00a26f0 2024-09-16T14:54:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba011e9e0 con 0x7f5ba0099b60 2024-09-16T14:54:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 10 monclient(hunting): authenticate will time out at 1669.377402s 2024-09-16T14:54:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b9e59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5ba0099b60 0x7f5ba013bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47850/0 (socket says 172.21.15.26:47850) 2024-09-16T14:54:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b9e59c640 1 -- 172.21.15.26:0/1388654315 learned_addr learned my addr 172.21.15.26:0/1388654315 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854808060 0 0) 0x7f5ba011e9e0 con 0x7f5ba0099b60 2024-09-16T14:54:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): my global_id is 234263 2024-09-16T14:54:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b80003530 con 0x7f5ba0099b60 2024-09-16T14:54:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479893443 0 0) 0x7f5ba009f340 con 0x7f5ba00a1c00 2024-09-16T14:54:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): my global_id is 194571 2024-09-16T14:54:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba011e9e0 con 0x7f5ba00a1c00 2024-09-16T14:54:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1076695874 0 0) 0x7f5ba011e9e0 con 0x7f5ba00a1c00 2024-09-16T14:54:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ba009f340 con 0x7f5ba00a1c00 2024-09-16T14:54:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5b88001b90 con 0x7f5ba00a1c00 2024-09-16T14:54:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): dump: 2024-09-16T14:54:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.272 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3998223373 0 0) 0x7f5b80003530 con 0x7f5ba0099b60 2024-09-16T14:54:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ba011e9e0 con 0x7f5ba0099b60 2024-09-16T14:54:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5b90001c10 con 0x7f5ba0099b60 2024-09-16T14:54:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): dump: 2024-09-16T14:54:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.273 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284102124 0 0) 0x7f5ba009e3f0 con 0x7f5ba00a26f0 2024-09-16T14:54:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): my global_id is 244144 2024-09-16T14:54:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b80003530 con 0x7f5ba00a26f0 2024-09-16T14:54:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1926476901 0 0) 0x7f5ba009f340 con 0x7f5ba00a1c00 2024-09-16T14:54:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 >> v1:172.21.15.26:6790/0 conn(0x7f5ba0099b60 legacy=0x7f5ba013bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 >> v1:172.21.15.189:6789/0 conn(0x7f5ba00a26f0 legacy=0x7f5ba013cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:54:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 monclient: found mon.a 2024-09-16T14:54:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:05.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:05.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba009edf0 con 0x7f5ba00a1c00 2024-09-16T14:54:05.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient: _finish_auth 0 2024-09-16T14:54:05.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:54:05.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:05.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 5 monclient: authenticate success, global_id 194571 2024-09-16T14:54:05.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 10 monclient: _renew_subs 2024-09-16T14:54:05.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:05.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 1 -- 172.21.15.26:0/1388654315 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba013db10 con 0x7f5ba00a1c00 2024-09-16T14:54:05.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 10 monclient: _renew_subs 2024-09-16T14:54:05.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:05.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 1 -- 172.21.15.26:0/1388654315 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba011e700 con 0x7f5ba00a1c00 2024-09-16T14:54:05.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b88004b70 con 0x7f5ba00a1c00 2024-09-16T14:54:05.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:05.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5b88004f00 con 0x7f5ba00a1c00 2024-09-16T14:54:05.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:05.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b9cd99640 4 set_mon_vals no callback set 2024-09-16T14:54:05.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient: dump: 2024-09-16T14:54:05.277 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.277 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.277 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.277 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.277 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.277 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.277 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.277 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.277 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.277 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:05.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5ba4fa8640 1 -- 172.21.15.26:0/1388654315 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5b84006590 con 0x7f5b84002830 2024-09-16T14:54:05.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5b88005060 con 0x7f5ba00a1c00 2024-09-16T14:54:05.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f5b88043fe0 con 0x7f5ba00a1c00 2024-09-16T14:54:05.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013619906 0 0) 0x7f5b84006590 con 0x7f5b84002830 2024-09-16T14:54:05.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:05.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:05.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:05.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 10 monclient(hunting): my global_id is 244147 2024-09-16T14:54:05.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba009f340 con 0x7f5b84002830 2024-09-16T14:54:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.254+0000 7f5b9ed9d640 10 monclient: get_auth_request con 0x7f5b800363d0 auth_method 0 2024-09-16T14:54:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.258+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1624206779 0 0) 0x7f5ba009f340 con 0x7f5b84002830 2024-09-16T14:54:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.258+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b84006590 con 0x7f5b84002830 2024-09-16T14:54:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.258+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5b90004c60 con 0x7f5b84002830 2024-09-16T14:54:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.258+0000 7f5b96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.258+0000 7f5b96ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:54:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.258+0000 7f5b96ffd640 10 monclient: dump: 2024-09-16T14:54:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.280 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.258+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 711578279 0 0) 0x7f5b84006590 con 0x7f5b84002830 2024-09-16T14:54:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.258+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b8400a0e0 con 0x7f5b84002830 2024-09-16T14:54:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.258+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5b90004f80 con 0x7f5b84002830 2024-09-16T14:54:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.258+0000 7f5b96ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.258+0000 7f5b96ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.258+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 >> v1:172.21.15.189:6789/0 conn(0x7f5b84002830 legacy=0x7f5b84004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.258+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 >> v1:172.21.15.189:6789/0 conn(0x7f5b84002830 legacy=0x7f5b84004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.262+0000 7f5ba4fa8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:54:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.262+0000 7f5ba4fa8640 1 -- 172.21.15.26:0/1388654315 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5b84009da0 con 0x7f5b840090c0 2024-09-16T14:54:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920048071 0 0) 0x7f5b84009da0 con 0x7f5b840090c0 2024-09-16T14:54:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 10 monclient(hunting): my global_id is 244150 2024-09-16T14:54:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b84006590 con 0x7f5b840090c0 2024-09-16T14:54:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1483177488 0 0) 0x7f5b84006590 con 0x7f5b840090c0 2024-09-16T14:54:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b84009da0 con 0x7f5b840090c0 2024-09-16T14:54:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5b900055d0 con 0x7f5b840090c0 2024-09-16T14:54:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:54:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 10 monclient: dump: 2024-09-16T14:54:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.285 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 209106508 0 0) 0x7f5b84009da0 con 0x7f5b840090c0 2024-09-16T14:54:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5b8400c290 con 0x7f5b840090c0 2024-09-16T14:54:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3094583670) 0x7f5b90005860 con 0x7f5b840090c0 2024-09-16T14:54:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:54:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 >> v1:172.21.15.189:6789/0 conn(0x7f5b840090c0 legacy=0x7f5b84009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b96ffd640 1 -- 172.21.15.26:0/1388654315 >> v1:172.21.15.189:6789/0 conn(0x7f5b840090c0 legacy=0x7f5b84009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:05.286 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:54:05.286 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:05.287 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:05.287 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:54:05.287 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 108, 2024-09-16T14:54:05.287 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:54:05.287 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:54:05.287 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:54:05.287 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:54:05.287 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:05.287 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-09-16T14:54:05.288 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:05.288 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:54:05.288 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:54:05.288 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:05.288 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:05.288 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:05.288 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:05.288 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:05.288 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:05.288 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:05.288 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:05.288 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:05.289 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:05.289 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:05.289 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:54:05.289 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:54:05.289 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:05.289 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:05.289 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:05.289 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:05.289 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:05.289 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:05.289 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:05.289 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:05.290 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:05.290 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:05.290 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:05.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.290 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:54:05.290 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:54:05.290 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:54:05.290 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:54:05.290 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:54:05.290 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:54:05.290 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:05.290 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:05.291 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:54:05.291 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:54:05.291 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:54:05.291 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:54:05.291 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:54:05.291 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:54:05.291 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:54:05.291 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:05.291 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:54:05.291 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:05.291 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:05.291 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:05.291 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:05.292 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:05.292 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:05.292 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:05.292 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:05.292 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:05.292 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:05.292 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:05.292 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:54:05.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.292 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:54:05.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:05.292 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:05.292 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:54:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:54:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:54:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:54:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:54:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:54:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:54:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:54:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:54:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:54:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:54:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:54:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:54:05.298 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:54:05.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5ba4fa8640 1 -- 172.21.15.26:0/1388654315 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5b800363d0 legacy=0x7f5b80038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5ba4fa8640 10 monclient: shutdown 2024-09-16T14:54:05.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5ba4fa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5ba4fa8640 1 -- 172.21.15.26:0/1388654315 >> v1:172.21.15.26:6789/0 conn(0x7f5ba00a1c00 legacy=0x7f5ba013c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b9ed9d640 1 -- 172.21.15.26:0/1388654315 reap_dead start 2024-09-16T14:54:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5b9ed9d640 1 -- 172.21.15.26:0/1388654315 reap_dead start 2024-09-16T14:54:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5ba4fa8640 1 -- 172.21.15.26:0/1388654315 shutdown_connections 2024-09-16T14:54:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5ba4fa8640 1 -- 172.21.15.26:0/1388654315 >> 172.21.15.26:0/1388654315 conn(0x7f5ba00a3d00 msgr2=0x7f5ba00a0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5ba4fa8640 1 -- 172.21.15.26:0/1388654315 shutdown_connections 2024-09-16T14:54:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.266+0000 7f5ba4fa8640 1 -- 172.21.15.26:0/1388654315 wait complete. 2024-09-16T14:54:05.300 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:54:05.301 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T14:54:05.301 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:54:05.301 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T14:54:05.301 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:54:05.301 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T14:54:05.301 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:54:05.316 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T14:54:05.316 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:54:05.316 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:54:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed0aaf7640 10 monclient: get_monmap_and_config 2024-09-16T14:54:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed0aaf7640 10 monclient: build_initial_monmap 2024-09-16T14:54:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed0aaf7640 10 monclient: monmap: 2024-09-16T14:54:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:05.479677+0000 2024-09-16T14:54:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:05.479677+0000 2024-09-16T14:54:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:05.479 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed0aaf7640 1 Processor -- start 2024-09-16T14:54:05.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed0aaf7640 1 -- start start 2024-09-16T14:54:05.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed0aaf7640 10 monclient: init 2024-09-16T14:54:05.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed0aaf7640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:05.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed0aaf7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:54:05.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed0aaf7640 10 monclient(hunting): picked mon.noname-c con 0x7fed0409cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:05.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed0aaf7640 10 monclient(hunting): picked mon.noname-b con 0x7fed0409e7a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:05.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed0aaf7640 10 monclient(hunting): picked mon.noname-a con 0x7fed0409d1a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:05.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed0aaf7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0409d5b0 con 0x7fed0409d1a0 2024-09-16T14:54:05.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed0aaf7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed04131a90 con 0x7fed0409cdc0 2024-09-16T14:54:05.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed0aaf7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0409aff0 con 0x7fed0409e7a0 2024-09-16T14:54:05.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed0aaf7640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:05.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed0aaf7640 10 monclient(hunting): authenticate will time out at 1669.599893s 2024-09-16T14:54:05.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.474+0000 7fed09094640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fed0409d1a0 legacy=0x7fed04139fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:05.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed03fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fed0409e7a0 0x7fed04131380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47886/0 (socket says 172.21.15.26:47886) 2024-09-16T14:54:05.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed03fff640 1 -- 172.21.15.26:0/3634437936 learned_addr learned my addr 172.21.15.26:0/3634437936 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:05.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 1 -- 172.21.15.26:0/3634437936 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866755334 0 0) 0x7fed0409aff0 con 0x7fed0409e7a0 2024-09-16T14:54:05.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:05.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:05.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:05.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 10 monclient(hunting): my global_id is 234266 2024-09-16T14:54:05.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 1 -- 172.21.15.26:0/3634437936 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecec003530 con 0x7fed0409e7a0 2024-09-16T14:54:05.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 1 -- 172.21.15.26:0/3634437936 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3873381536 0 0) 0x7fecec003530 con 0x7fed0409e7a0 2024-09-16T14:54:05.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 1 -- 172.21.15.26:0/3634437936 >> v1:172.21.15.189:6789/0 conn(0x7fed0409cdc0 legacy=0x7fed04130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 1 -- 172.21.15.26:0/3634437936 >> v1:172.21.15.26:6789/0 conn(0x7fed0409d1a0 legacy=0x7fed04139fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:54:05.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 10 monclient: _finish_hunting 0 2024-09-16T14:54:05.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 1 monclient: found mon.noname-b 2024-09-16T14:54:05.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:05.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:54:05.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 1 -- 172.21.15.26:0/3634437936 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0409b2d0 con 0x7fed0409e7a0 2024-09-16T14:54:05.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 10 monclient: _finish_auth 0 2024-09-16T14:54:05.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 10 monclient: _check_auth_tickets 2024-09-16T14:54:05.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:05.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 5 monclient: authenticate success, global_id 234266 2024-09-16T14:54:05.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 1 -- 172.21.15.26:0/3634437936 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fecf8001620 con 0x7fed0409e7a0 2024-09-16T14:54:05.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:05.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:54:05.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 10 monclient: dump: 2024-09-16T14:54:05.489 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.489 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.489 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.489 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.489 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.490 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.490 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.490 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.490 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.490 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:05.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 1 -- 172.21.15.26:0/3634437936 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecf80018d0 con 0x7fed0409e7a0 2024-09-16T14:54:05.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:05.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:05.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 4 set_mon_vals no callback set 2024-09-16T14:54:05.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 1 -- 172.21.15.26:0/3634437936 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fecf8001c60 con 0x7fed0409e7a0 2024-09-16T14:54:05.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:05.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed037fe640 10 monclient: dump: 2024-09-16T14:54:05.491 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.491 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.491 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.491 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.492 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.492 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.492 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.492 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.492 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.492 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 10 monclient: shutdown 2024-09-16T14:54:05.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:05.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 1 -- 172.21.15.26:0/3634437936 >> v1:172.21.15.26:6790/0 conn(0x7fed0409e7a0 legacy=0x7fed04131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 1 -- 172.21.15.26:0/3634437936 shutdown_connections 2024-09-16T14:54:05.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 1 -- 172.21.15.26:0/3634437936 >> 172.21.15.26:0/3634437936 conn(0x7fed0409fb50 msgr2=0x7fed0409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:05.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 1 -- 172.21.15.26:0/3634437936 shutdown_connections 2024-09-16T14:54:05.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 1 -- 172.21.15.26:0/3634437936 wait complete. 2024-09-16T14:54:05.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 10 monclient: build_initial_monmap 2024-09-16T14:54:05.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 10 monclient: monmap: 2024-09-16T14:54:05.493 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:05.493 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:05.493 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:05.493 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:05.494 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:05.494 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:05.494 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:05.494 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:05.494 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:05.494 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 1 Processor -- start 2024-09-16T14:54:05.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 1 -- start start 2024-09-16T14:54:05.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 10 monclient: init 2024-09-16T14:54:05.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:05.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:54:05.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 10 monclient(hunting): picked mon.noname-c con 0x7fed0409cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:05.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 10 monclient(hunting): picked mon.noname-a con 0x7fed0409d1a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:05.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 10 monclient(hunting): picked mon.noname-b con 0x7fed0409e7a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:05.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0411d570 con 0x7fed0409d1a0 2024-09-16T14:54:05.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0411c050 con 0x7fed0409e7a0 2024-09-16T14:54:05.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed03fff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fed0409d1a0 legacy=0x7fed041ce850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:05.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed04130b00 con 0x7fed0409cdc0 2024-09-16T14:54:05.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed08893640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fed0409cdc0 0x7fed0412fe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47898/0 (socket says 172.21.15.26:47898) 2024-09-16T14:54:05.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:05.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed08893640 1 -- 172.21.15.26:0/633618747 learned_addr learned my addr 172.21.15.26:0/633618747 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:05.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 10 monclient(hunting): authenticate will time out at 1669.602939s 2024-09-16T14:54:05.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 1 -- 172.21.15.26:0/633618747 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842383113 0 0) 0x7fed04130b00 con 0x7fed0409cdc0 2024-09-16T14:54:05.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 10 monclient(hunting): _negotiate 2024-09-16T14:54:05.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:05.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:05.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 10 monclient(hunting): my global_id is 234269 2024-09-16T14:54:05.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 1 -- 172.21.15.26:0/633618747 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf0003560 con 0x7fed0409cdc0 2024-09-16T14:54:05.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 1 -- 172.21.15.26:0/633618747 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 840446784 0 0) 0x7fecf0003560 con 0x7fed0409cdc0 2024-09-16T14:54:05.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 1 -- 172.21.15.26:0/633618747 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed04130b00 con 0x7fed0409cdc0 2024-09-16T14:54:05.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 1 -- 172.21.15.26:0/633618747 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fecf4001980 con 0x7fed0409cdc0 2024-09-16T14:54:05.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:05.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 10 monclient(hunting): dump: 2024-09-16T14:54:05.497 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.497 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.497 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.497 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.497 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.497 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.497 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.498 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.498 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.498 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:05.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 1 -- 172.21.15.26:0/633618747 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3841480918 0 0) 0x7fed04130b00 con 0x7fed0409cdc0 2024-09-16T14:54:05.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 1 -- 172.21.15.26:0/633618747 >> v1:172.21.15.189:6789/0 conn(0x7fed0409e7a0 legacy=0x7fed041d1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 1 -- 172.21.15.26:0/633618747 >> v1:172.21.15.26:6789/0 conn(0x7fed0409d1a0 legacy=0x7fed041ce850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:54:05.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 10 monclient: _finish_hunting 0 2024-09-16T14:54:05.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 1 monclient: found mon.c 2024-09-16T14:54:05.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:05.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:05.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 1 -- 172.21.15.26:0/633618747 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0411d290 con 0x7fed0409cdc0 2024-09-16T14:54:05.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 10 monclient: _finish_auth 0 2024-09-16T14:54:05.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 10 monclient: _check_auth_tickets 2024-09-16T14:54:05.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:05.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 5 monclient: authenticate success, global_id 234269 2024-09-16T14:54:05.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 10 monclient: _renew_subs 2024-09-16T14:54:05.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:05.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 1 -- 172.21.15.26:0/633618747 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0411bd80 con 0x7fed0409cdc0 2024-09-16T14:54:05.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 10 monclient: _renew_subs 2024-09-16T14:54:05.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:05.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed0aaf7640 1 -- 172.21.15.26:0/633618747 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed041305a0 con 0x7fed0409cdc0 2024-09-16T14:54:05.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 1 -- 172.21.15.26:0/633618747 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecf4001d50 con 0x7fed0409cdc0 2024-09-16T14:54:05.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:05.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 1 -- 172.21.15.26:0/633618747 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fecf4003a70 con 0x7fed0409cdc0 2024-09-16T14:54:05.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:05.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:05.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed02ffd640 4 set_mon_vals no callback set 2024-09-16T14:54:05.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.478+0000 7fed017fa640 10 monclient: dump: 2024-09-16T14:54:05.500 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:05.500 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:05.500 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.500 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:05.500 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:05.500 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:05.500 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:05.500 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:05.500 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:05.500 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:05.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.482+0000 7fed0aaf7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:05.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.482+0000 7fed017fa640 1 -- 172.21.15.26:0/633618747 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fecf4003bd0 con 0x7fed0409cdc0 2024-09-16T14:54:05.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.482+0000 7fed0aaf7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:05.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.482+0000 7fed0aaf7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:05.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.482+0000 7fed0aaf7640 1 -- 172.21.15.26:0/633618747 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece4005050 con 0x7fed0409cdc0 2024-09-16T14:54:05.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.482+0000 7fed017fa640 1 -- 172.21.15.26:0/633618747 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fecf4043ed0 con 0x7fed0409cdc0 2024-09-16T14:54:05.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.482+0000 7fed03fff640 10 monclient: get_auth_request con 0x7fecf0036040 auth_method 0 2024-09-16T14:54:05.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.486+0000 7fed017fa640 1 -- 172.21.15.26:0/633618747 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fecf4014810 con 0x7fed0409cdc0 2024-09-16T14:54:05.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.486+0000 7fed017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:05.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.486+0000 7fed017fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:05.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.694+0000 7fed0aaf7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:54:05.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.694+0000 7fed0aaf7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:05.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.694+0000 7fed0aaf7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:05.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.694+0000 7fed0aaf7640 1 -- 172.21.15.26:0/633618747 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fece4005880 con 0x7fed0409cdc0 2024-09-16T14:54:05.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.694+0000 7fed017fa640 1 -- 172.21.15.26:0/633618747 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1049459162) 0x7fecf40145e0 con 0x7fed0409cdc0 2024-09-16T14:54:05.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.694+0000 7fed017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:05.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.694+0000 7fed017fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:05.699 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:54:05.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.698+0000 7fed0aaf7640 1 -- 172.21.15.26:0/633618747 >> v1:172.21.15.189:6812/3196161582 conn(0x7fecf0036040 legacy=0x7fecf0038500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.698+0000 7fed0aaf7640 10 monclient: shutdown 2024-09-16T14:54:05.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.698+0000 7fed0aaf7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:05.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.698+0000 7fed0aaf7640 1 -- 172.21.15.26:0/633618747 >> v1:172.21.15.26:6790/0 conn(0x7fed0409cdc0 legacy=0x7fed0412fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:05.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.698+0000 7fed0aaf7640 1 -- 172.21.15.26:0/633618747 shutdown_connections 2024-09-16T14:54:05.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.698+0000 7fed0aaf7640 1 -- 172.21.15.26:0/633618747 >> 172.21.15.26:0/633618747 conn(0x7fed0409fb50 msgr2=0x7fed04131010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:05.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.698+0000 7fed0aaf7640 1 -- 172.21.15.26:0/633618747 shutdown_connections 2024-09-16T14:54:05.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:05.698+0000 7fed0aaf7640 1 -- 172.21.15.26:0/633618747 wait complete. 2024-09-16T14:54:08.717 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:54:08.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcf04fce640 10 monclient: get_monmap_and_config 2024-09-16T14:54:08.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcf04fce640 10 monclient: build_initial_monmap 2024-09-16T14:54:08.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcf04fce640 10 monclient: monmap: 2024-09-16T14:54:08.850 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:08.850 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:08.850 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:08.851103+0000 2024-09-16T14:54:08.850 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:08.851103+0000 2024-09-16T14:54:08.850 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:08.850 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:08.850 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:08.850 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:08.850 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:08.850 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:08.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcf04fce640 1 Processor -- start 2024-09-16T14:54:08.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcf04fce640 1 -- start start 2024-09-16T14:54:08.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcf04fce640 10 monclient: init 2024-09-16T14:54:08.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcf04fce640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:08.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcf04fce640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:54:08.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcf04fce640 10 monclient(hunting): picked mon.noname-c con 0x7fcf000a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:08.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcf04fce640 10 monclient(hunting): picked mon.noname-b con 0x7fcf000a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:08.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcf04fce640 10 monclient(hunting): picked mon.noname-a con 0x7fcf000a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:08.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcf04fce640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf000a3250 con 0x7fcf000a2800 2024-09-16T14:54:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcf04fce640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0009f600 con 0x7fcf000a1d10 2024-09-16T14:54:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcf04fce640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf000a3830 con 0x7fcf000a0000 2024-09-16T14:54:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcf04fce640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcf04fce640 10 monclient(hunting): authenticate will time out at 1672.971306s 2024-09-16T14:54:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcefdd9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fcf000a0000 0x7fcf00139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55794/0 (socket says 172.21.15.26:55794) 2024-09-16T14:54:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcefdd9b640 1 -- 172.21.15.26:0/357524972 learned_addr learned my addr 172.21.15.26:0/357524972 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcefed9d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fcf000a2800 legacy=0x7fcf0013d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcefd59a640 1 -- 172.21.15.26:0/357524972 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311888605 0 0) 0x7fcf000a3830 con 0x7fcf000a0000 2024-09-16T14:54:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcefd59a640 10 monclient(hunting): _negotiate 2024-09-16T14:54:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcefd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcefd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcefd59a640 10 monclient(hunting): my global_id is 234272 2024-09-16T14:54:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.846+0000 7fcefd59a640 1 -- 172.21.15.26:0/357524972 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceec003530 con 0x7fcf000a0000 2024-09-16T14:54:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 1 -- 172.21.15.26:0/357524972 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 692717867 0 0) 0x7fceec003530 con 0x7fcf000a0000 2024-09-16T14:54:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 1 -- 172.21.15.26:0/357524972 >> v1:172.21.15.189:6789/0 conn(0x7fcf000a1d10 legacy=0x7fcf000a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 1 -- 172.21.15.26:0/357524972 >> v1:172.21.15.26:6789/0 conn(0x7fcf000a2800 legacy=0x7fcf0013d440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:54:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 10 monclient: _finish_hunting 0 2024-09-16T14:54:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 1 monclient: found mon.noname-b 2024-09-16T14:54:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:54:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 1 -- 172.21.15.26:0/357524972 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0009ed60 con 0x7fcf000a0000 2024-09-16T14:54:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 10 monclient: _finish_auth 0 2024-09-16T14:54:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 10 monclient: _check_auth_tickets 2024-09-16T14:54:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 5 monclient: authenticate success, global_id 234272 2024-09-16T14:54:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 1 -- 172.21.15.26:0/357524972 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcef0001620 con 0x7fcf000a0000 2024-09-16T14:54:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:54:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 10 monclient: dump: 2024-09-16T14:54:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:08.857 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 1 -- 172.21.15.26:0/357524972 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcef0001940 con 0x7fcf000a0000 2024-09-16T14:54:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 1 -- 172.21.15.26:0/357524972 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcef0001d10 con 0x7fcf000a0000 2024-09-16T14:54:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 4 set_mon_vals no callback set 2024-09-16T14:54:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefd59a640 10 monclient: dump: 2024-09-16T14:54:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:08.858 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 10 monclient: shutdown 2024-09-16T14:54:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 1 -- 172.21.15.26:0/357524972 >> v1:172.21.15.26:6790/0 conn(0x7fcf000a0000 legacy=0x7fcf00139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 1 -- 172.21.15.26:0/357524972 shutdown_connections 2024-09-16T14:54:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 1 -- 172.21.15.26:0/357524972 >> 172.21.15.26:0/357524972 conn(0x7fcf000a7f30 msgr2=0x7fcf000a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 1 -- 172.21.15.26:0/357524972 shutdown_connections 2024-09-16T14:54:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 1 -- 172.21.15.26:0/357524972 wait complete. 2024-09-16T14:54:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 10 monclient: build_initial_monmap 2024-09-16T14:54:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 10 monclient: monmap: 2024-09-16T14:54:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:08.860 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:08.860 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 1 Processor -- start 2024-09-16T14:54:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 1 -- start start 2024-09-16T14:54:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 10 monclient: init 2024-09-16T14:54:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:54:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 10 monclient(hunting): picked mon.noname-a con 0x7fcf000a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 10 monclient(hunting): picked mon.noname-b con 0x7fcf000a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 10 monclient(hunting): picked mon.noname-c con 0x7fcf000a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefe59c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fcf000a0000 legacy=0x7fcf00134550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf00131740 con 0x7fcf000a0000 2024-09-16T14:54:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf00131b50 con 0x7fcf000a1d10 2024-09-16T14:54:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf00133b10 con 0x7fcf000a2800 2024-09-16T14:54:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 10 monclient(hunting): authenticate will time out at 1672.974684s 2024-09-16T14:54:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefed9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fcf000a2800 0x7fcf001d10b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55798/0 (socket says 172.21.15.26:55798) 2024-09-16T14:54:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcefed9d640 1 -- 172.21.15.26:0/4091439343 learned_addr learned my addr 172.21.15.26:0/4091439343 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 1 -- 172.21.15.26:0/4091439343 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089278186 0 0) 0x7fcf00133b10 con 0x7fcf000a2800 2024-09-16T14:54:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 10 monclient(hunting): my global_id is 234275 2024-09-16T14:54:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 1 -- 172.21.15.26:0/4091439343 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee0003530 con 0x7fcf000a2800 2024-09-16T14:54:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 1 -- 172.21.15.26:0/4091439343 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1421371750 0 0) 0x7fcee0003530 con 0x7fcf000a2800 2024-09-16T14:54:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 1 -- 172.21.15.26:0/4091439343 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf00133b10 con 0x7fcf000a2800 2024-09-16T14:54:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 1 -- 172.21.15.26:0/4091439343 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcef80043c0 con 0x7fcf000a2800 2024-09-16T14:54:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 10 monclient(hunting): dump: 2024-09-16T14:54:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:08.866 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 1 -- 172.21.15.26:0/4091439343 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 893986628 0 0) 0x7fcf00133b10 con 0x7fcf000a2800 2024-09-16T14:54:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 1 -- 172.21.15.26:0/4091439343 >> v1:172.21.15.189:6789/0 conn(0x7fcf000a1d10 legacy=0x7fcf001cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 1 -- 172.21.15.26:0/4091439343 >> v1:172.21.15.26:6789/0 conn(0x7fcf000a0000 legacy=0x7fcf00134550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:54:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:54:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 1 monclient: found mon.c 2024-09-16T14:54:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 1 -- 172.21.15.26:0/4091439343 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf00133830 con 0x7fcf000a2800 2024-09-16T14:54:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:54:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:54:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 5 monclient: authenticate success, global_id 234275 2024-09-16T14:54:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 10 monclient: _renew_subs 2024-09-16T14:54:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 1 -- 172.21.15.26:0/4091439343 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf00132070 con 0x7fcf000a2800 2024-09-16T14:54:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 10 monclient: _renew_subs 2024-09-16T14:54:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcf04fce640 1 -- 172.21.15.26:0/4091439343 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf001d29a0 con 0x7fcf000a2800 2024-09-16T14:54:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 1 -- 172.21.15.26:0/4091439343 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcef80045e0 con 0x7fcf000a2800 2024-09-16T14:54:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 1 -- 172.21.15.26:0/4091439343 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcef80048f0 con 0x7fcf000a2800 2024-09-16T14:54:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef7fff640 4 set_mon_vals no callback set 2024-09-16T14:54:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.850+0000 7fcef6ffd640 10 monclient: dump: 2024-09-16T14:54:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:08.869 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.854+0000 7fcef6ffd640 1 -- 172.21.15.26:0/4091439343 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fcef8004ad0 con 0x7fcf000a2800 2024-09-16T14:54:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.854+0000 7fcf04fce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.854+0000 7fcf04fce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.854+0000 7fcf04fce640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.854+0000 7fcf04fce640 1 -- 172.21.15.26:0/4091439343 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee4005050 con 0x7fcf000a2800 2024-09-16T14:54:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.854+0000 7fcef6ffd640 1 -- 172.21.15.26:0/4091439343 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fcef8044080 con 0x7fcf000a2800 2024-09-16T14:54:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.854+0000 7fcefe59c640 10 monclient: get_auth_request con 0x7fcee0036050 auth_method 0 2024-09-16T14:54:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.858+0000 7fcef6ffd640 1 -- 172.21.15.26:0/4091439343 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fcef8005320 con 0x7fcf000a2800 2024-09-16T14:54:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.858+0000 7fcef6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:08.858+0000 7fcef6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:09.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:09.078+0000 7fcf04fce640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:54:09.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:09.078+0000 7fcf04fce640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:09.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:09.078+0000 7fcf04fce640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:09.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:09.078+0000 7fcf04fce640 1 -- 172.21.15.26:0/4091439343 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcee4005880 con 0x7fcf000a2800 2024-09-16T14:54:09.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:09.078+0000 7fcef6ffd640 1 -- 172.21.15.26:0/4091439343 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1851897851) 0x7fcef801cfb0 con 0x7fcf000a2800 2024-09-16T14:54:09.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:09.078+0000 7fcef6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:09.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:09.078+0000 7fcef6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:09.081 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:54:09.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:09.078+0000 7fcf04fce640 1 -- 172.21.15.26:0/4091439343 >> v1:172.21.15.189:6812/3196161582 conn(0x7fcee0036050 legacy=0x7fcee0038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:09.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:09.078+0000 7fcf04fce640 10 monclient: shutdown 2024-09-16T14:54:09.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:09.078+0000 7fcf04fce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:09.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:09.078+0000 7fcf04fce640 1 -- 172.21.15.26:0/4091439343 >> v1:172.21.15.26:6790/0 conn(0x7fcf000a2800 legacy=0x7fcf001d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:09.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:09.078+0000 7fcf04fce640 1 -- 172.21.15.26:0/4091439343 shutdown_connections 2024-09-16T14:54:09.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:09.078+0000 7fcf04fce640 1 -- 172.21.15.26:0/4091439343 >> 172.21.15.26:0/4091439343 conn(0x7fcf000a7f30 msgr2=0x7fcf0006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:09.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:09.078+0000 7fcf04fce640 1 -- 172.21.15.26:0/4091439343 shutdown_connections 2024-09-16T14:54:09.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:09.078+0000 7fcf04fce640 1 -- 172.21.15.26:0/4091439343 wait complete. 2024-09-16T14:54:12.101 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:54:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f543048b640 10 monclient: get_monmap_and_config 2024-09-16T14:54:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f543048b640 10 monclient: build_initial_monmap 2024-09-16T14:54:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f543048b640 10 monclient: monmap: 2024-09-16T14:54:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:12.235290+0000 2024-09-16T14:54:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:12.235290+0000 2024-09-16T14:54:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:12.234 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f543048b640 1 Processor -- start 2024-09-16T14:54:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f543048b640 1 -- start start 2024-09-16T14:54:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f543048b640 10 monclient: init 2024-09-16T14:54:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f543048b640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f543048b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:54:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f543048b640 10 monclient(hunting): picked mon.noname-a con 0x7f54280a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f543048b640 10 monclient(hunting): picked mon.noname-b con 0x7f54280a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f543048b640 10 monclient(hunting): picked mon.noname-c con 0x7f54280a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f543048b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54280a3250 con 0x7f54280a1d10 2024-09-16T14:54:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f543048b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542809f600 con 0x7f54280a2800 2024-09-16T14:54:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f543048b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54280a3830 con 0x7f54280a0000 2024-09-16T14:54:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f543048b640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f543048b640 10 monclient(hunting): authenticate will time out at 1676.355382s 2024-09-16T14:54:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f542e227640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f54280a1d10 legacy=0x7f54280a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f542da26640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f54280a0000 0x7f5428139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55804/0 (socket says 172.21.15.26:55804) 2024-09-16T14:54:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.230+0000 7f542da26640 1 -- 172.21.15.26:0/3817043154 learned_addr learned my addr 172.21.15.26:0/3817043154 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:12.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.434+0000 7f542e227640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6789/0 conn(0x7f54280a1d10 legacy=0x7f54280a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:12.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:12.838+0000 7f542e227640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6789/0 conn(0x7f54280a1d10 legacy=0x7f54280a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:13.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:13.642+0000 7f542e227640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6789/0 conn(0x7f54280a1d10 legacy=0x7f54280a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:15.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.230+0000 7f541bfff640 10 monclient(hunting): tick 2024-09-16T14:54:15.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.230+0000 7f541bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:54:15.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.230+0000 7f541bfff640 1 monclient(hunting): continuing hunt 2024-09-16T14:54:15.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.230+0000 7f541bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:54:15.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.230+0000 7f541bfff640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6790/0 conn(0x7f54280a0000 legacy=0x7f5428139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:15.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.230+0000 7f541bfff640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.189:6789/0 conn(0x7f54280a2800 legacy=0x7f542813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:15.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.230+0000 7f541bfff640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6789/0 conn(0x7f54280a1d10 legacy=0x7f54280a20f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:54:15.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.230+0000 7f541bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:54:15.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.230+0000 7f541bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f540c0036d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:15.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.230+0000 7f541bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f54280a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:15.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.230+0000 7f541bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f54280a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:15.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.230+0000 7f542da26640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6789/0 conn(0x7f54280a1d10 legacy=0x7f540c0093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:15.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.230+0000 7f541bfff640 1 -- 172.21.15.26:0/3817043154 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542809ed60 con 0x7f54280a1d10 2024-09-16T14:54:15.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.230+0000 7f541bfff640 1 -- 172.21.15.26:0/3817043154 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54280a3250 con 0x7f54280a2800 2024-09-16T14:54:15.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.230+0000 7f541bfff640 1 -- 172.21.15.26:0/3817043154 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540c010380 con 0x7f540c0036d0 2024-09-16T14:54:15.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.234+0000 7f541bfff640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:15.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.434+0000 7f542da26640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6789/0 conn(0x7f54280a1d10 legacy=0x7f540c0093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:15.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:15.838+0000 7f542da26640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6789/0 conn(0x7f54280a1d10 legacy=0x7f540c0093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:16.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:16.642+0000 7f542da26640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6789/0 conn(0x7f54280a1d10 legacy=0x7f540c0093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:18.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.234+0000 7f541bfff640 10 monclient(hunting): tick 2024-09-16T14:54:18.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.234+0000 7f541bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:54:18.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.234+0000 7f541bfff640 1 monclient(hunting): continuing hunt 2024-09-16T14:54:18.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.234+0000 7f541bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:54:18.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.234+0000 7f541bfff640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6790/0 conn(0x7f540c0036d0 legacy=0x7f540c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:18.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.234+0000 7f541bfff640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.189:6789/0 conn(0x7f54280a2800 legacy=0x7f540c00cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:18.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.234+0000 7f541bfff640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6789/0 conn(0x7f54280a1d10 legacy=0x7f540c0093b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:54:18.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.234+0000 7f541bfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:54:18.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.234+0000 7f541bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f540c00f400 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:18.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.234+0000 7f541bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f540c010510 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:18.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.234+0000 7f541bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f54280a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:18.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.234+0000 7f541bfff640 1 -- 172.21.15.26:0/3817043154 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540c011590 con 0x7f54280a1d10 2024-09-16T14:54:18.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.234+0000 7f541bfff640 1 -- 172.21.15.26:0/3817043154 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542809ed60 con 0x7f540c00f400 2024-09-16T14:54:18.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.234+0000 7f541bfff640 1 -- 172.21.15.26:0/3817043154 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540c011020 con 0x7f540c010510 2024-09-16T14:54:18.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.234+0000 7f541bfff640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:18.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.234+0000 7f542ea28640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6789/0 conn(0x7f54280a1d10 legacy=0x7f540c0093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:18.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.438+0000 7f542ea28640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6789/0 conn(0x7f54280a1d10 legacy=0x7f540c0093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:18.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:18.842+0000 7f542ea28640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6789/0 conn(0x7f54280a1d10 legacy=0x7f540c0093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:19.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:19.646+0000 7f542ea28640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6789/0 conn(0x7f54280a1d10 legacy=0x7f540c0093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:20.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.358+0000 7f542d225640 1 -- 172.21.15.26:0/3817043154 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124587094 0 0) 0x7f542809ed60 con 0x7f540c00f400 2024-09-16T14:54:20.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.358+0000 7f542d225640 10 monclient(hunting): _negotiate 2024-09-16T14:54:20.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.358+0000 7f542d225640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:20.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.358+0000 7f542d225640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:20.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.358+0000 7f542d225640 10 monclient(hunting): my global_id is 244156 2024-09-16T14:54:20.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.358+0000 7f542d225640 1 -- 172.21.15.26:0/3817043154 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5414003530 con 0x7f540c00f400 2024-09-16T14:54:20.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 1 -- 172.21.15.26:0/3817043154 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311236840 0 0) 0x7f540c011020 con 0x7f540c010510 2024-09-16T14:54:20.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 10 monclient(hunting): _negotiate 2024-09-16T14:54:20.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:20.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:20.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 10 monclient(hunting): my global_id is 234278 2024-09-16T14:54:20.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 1 -- 172.21.15.26:0/3817043154 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f542809ed60 con 0x7f540c010510 2024-09-16T14:54:20.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 1 -- 172.21.15.26:0/3817043154 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2012728754 0 0) 0x7f542809ed60 con 0x7f540c010510 2024-09-16T14:54:20.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.189:6789/0 conn(0x7f540c00f400 legacy=0x7f540c00f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6789/0 conn(0x7f54280a1d10 legacy=0x7f540c0093b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:54:20.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 10 monclient: _finish_hunting 0 2024-09-16T14:54:20.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 1 monclient: found mon.noname-b 2024-09-16T14:54:20.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:20.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:54:20.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 1 -- 172.21.15.26:0/3817043154 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c0111b0 con 0x7f540c010510 2024-09-16T14:54:20.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 10 monclient: _finish_auth 0 2024-09-16T14:54:20.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 10 monclient: _check_auth_tickets 2024-09-16T14:54:20.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:20.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 1 -- 172.21.15.26:0/3817043154 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5424001a70 con 0x7f540c010510 2024-09-16T14:54:20.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:20.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:54:20.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f542d225640 10 monclient: dump: 2024-09-16T14:54:20.371 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:20.371 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.371 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.372 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.372 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:20.372 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.372 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:20.372 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:20.372 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:20.372 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f543048b640 5 monclient: authenticate success, global_id 234278 2024-09-16T14:54:20.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.366+0000 7f543048b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:20.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.370+0000 7f542d225640 1 -- 172.21.15.26:0/3817043154 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5424001c90 con 0x7f540c010510 2024-09-16T14:54:20.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.370+0000 7f542d225640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:20.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.370+0000 7f542d225640 1 -- 172.21.15.26:0/3817043154 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5424002060 con 0x7f540c010510 2024-09-16T14:54:20.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.370+0000 7f543048b640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:20.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.370+0000 7f543048b640 4 set_mon_vals no callback set 2024-09-16T14:54:20.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.370+0000 7f542d225640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:20.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.370+0000 7f542d225640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:20.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.370+0000 7f542d225640 10 monclient: dump: 2024-09-16T14:54:20.374 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:20.374 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.374 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.374 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.374 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:20.374 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.375 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:20.375 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:20.375 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:20.375 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.370+0000 7f543048b640 10 monclient: shutdown 2024-09-16T14:54:20.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.370+0000 7f543048b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:20.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.370+0000 7f543048b640 1 -- 172.21.15.26:0/3817043154 >> v1:172.21.15.26:6790/0 conn(0x7f540c010510 legacy=0x7f540c010910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.370+0000 7f543048b640 1 -- 172.21.15.26:0/3817043154 shutdown_connections 2024-09-16T14:54:20.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.370+0000 7f543048b640 1 -- 172.21.15.26:0/3817043154 >> 172.21.15.26:0/3817043154 conn(0x7f54280a7f30 msgr2=0x7f54280a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:20.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.374+0000 7f543048b640 1 -- 172.21.15.26:0/3817043154 shutdown_connections 2024-09-16T14:54:20.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.374+0000 7f543048b640 1 -- 172.21.15.26:0/3817043154 wait complete. 2024-09-16T14:54:20.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.374+0000 7f543048b640 10 monclient: build_initial_monmap 2024-09-16T14:54:20.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.374+0000 7f543048b640 10 monclient: monmap: 2024-09-16T14:54:20.377 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:20.377 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:20.378 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:20.378 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:20.378 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:20.378 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:20.378 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:20.378 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:20.378 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:20.378 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.374+0000 7f543048b640 1 Processor -- start 2024-09-16T14:54:20.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.374+0000 7f543048b640 1 -- start start 2024-09-16T14:54:20.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.374+0000 7f543048b640 10 monclient: init 2024-09-16T14:54:20.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:20.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:54:20.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 10 monclient(hunting): picked mon.noname-a con 0x7f540c00f400 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:20.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 10 monclient(hunting): picked mon.noname-c con 0x7f540c010510 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:20.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 10 monclient(hunting): picked mon.noname-b con 0x7f54280a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:20.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542811fe80 con 0x7f540c00f400 2024-09-16T14:54:20.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f542e227640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f540c00f400 legacy=0x7f5428133e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:20.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54280a6890 con 0x7f54280a1d10 2024-09-16T14:54:20.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542809e500 con 0x7f540c010510 2024-09-16T14:54:20.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:20.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 10 monclient(hunting): authenticate will time out at 1684.500380s 2024-09-16T14:54:20.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f542da26640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f540c010510 0x7f54281ce450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40898/0 (socket says 172.21.15.26:40898) 2024-09-16T14:54:20.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f542da26640 1 -- 172.21.15.26:0/4077764991 learned_addr learned my addr 172.21.15.26:0/4077764991 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:20.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867214056 0 0) 0x7f542809e500 con 0x7f540c010510 2024-09-16T14:54:20.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:20.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:20.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:20.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient(hunting): my global_id is 234281 2024-09-16T14:54:20.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f540c011590 con 0x7f540c010510 2024-09-16T14:54:20.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689482673 0 0) 0x7f54280a6890 con 0x7f54280a1d10 2024-09-16T14:54:20.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:20.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:20.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:20.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient(hunting): my global_id is 244159 2024-09-16T14:54:20.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f542809e500 con 0x7f54280a1d10 2024-09-16T14:54:20.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1441129755 0 0) 0x7f540c011590 con 0x7f540c010510 2024-09-16T14:54:20.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54280a6890 con 0x7f540c010510 2024-09-16T14:54:20.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5424001fa0 con 0x7f540c010510 2024-09-16T14:54:20.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:20.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:20.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient(hunting): dump: 2024-09-16T14:54:20.385 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:20.385 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.385 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.385 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.385 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:20.385 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.385 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:20.386 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:20.386 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:20.386 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:20.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3939289796 0 0) 0x7f54280a6890 con 0x7f540c010510 2024-09-16T14:54:20.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 >> v1:172.21.15.189:6789/0 conn(0x7f54280a1d10 legacy=0x7f54281d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 >> v1:172.21.15.26:6789/0 conn(0x7f540c00f400 legacy=0x7f5428133e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:54:20.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient: _finish_hunting 0 2024-09-16T14:54:20.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 1 monclient: found mon.c 2024-09-16T14:54:20.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:20.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:20.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5428134a90 con 0x7f540c010510 2024-09-16T14:54:20.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient: _finish_auth 0 2024-09-16T14:54:20.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient: _check_auth_tickets 2024-09-16T14:54:20.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:20.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 5 monclient: authenticate success, global_id 234281 2024-09-16T14:54:20.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 10 monclient: _renew_subs 2024-09-16T14:54:20.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:20.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 1 -- 172.21.15.26:0/4077764991 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54281217e0 con 0x7f540c010510 2024-09-16T14:54:20.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 10 monclient: _renew_subs 2024-09-16T14:54:20.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:20.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 1 -- 172.21.15.26:0/4077764991 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f542811f480 con 0x7f540c010510 2024-09-16T14:54:20.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5424004e20 con 0x7f540c010510 2024-09-16T14:54:20.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:20.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5424005130 con 0x7f540c010510 2024-09-16T14:54:20.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:20.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f542ca24640 4 set_mon_vals no callback set 2024-09-16T14:54:20.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:20.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 10 monclient: dump: 2024-09-16T14:54:20.388 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:20.388 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.388 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.388 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.388 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:20.388 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.388 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:20.388 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:20.388 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:20.388 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f542400fba0 con 0x7f540c010510 2024-09-16T14:54:20.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:20.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:20.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:20.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f543048b640 1 -- 172.21.15.26:0/4077764991 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5410005050 con 0x7f540c010510 2024-09-16T14:54:20.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f5424044070 con 0x7f540c010510 2024-09-16T14:54:20.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.378+0000 7f542e227640 10 monclient: get_auth_request con 0x7f540c037680 auth_method 0 2024-09-16T14:54:20.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.386+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5424014a70 con 0x7f540c010510 2024-09-16T14:54:20.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.386+0000 7f541affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:20.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.386+0000 7f541affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:20.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.602+0000 7f543048b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:54:20.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.602+0000 7f543048b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:20.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.602+0000 7f543048b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:20.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.602+0000 7f543048b640 1 -- 172.21.15.26:0/4077764991 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5410005880 con 0x7f540c010510 2024-09-16T14:54:20.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.602+0000 7f541affd640 1 -- 172.21.15.26:0/4077764991 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 3863909875) 0x7f5424012ee0 con 0x7f540c010510 2024-09-16T14:54:20.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.602+0000 7f541affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:20.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.602+0000 7f541affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:20.606 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":112,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:54:20.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.602+0000 7f543048b640 1 -- 172.21.15.26:0/4077764991 >> v1:172.21.15.189:6812/3196161582 conn(0x7f540c037680 legacy=0x7f540c039b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.602+0000 7f543048b640 10 monclient: shutdown 2024-09-16T14:54:20.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.602+0000 7f543048b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:20.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.602+0000 7f543048b640 1 -- 172.21.15.26:0/4077764991 >> v1:172.21.15.26:6790/0 conn(0x7f540c010510 legacy=0x7f54281ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.602+0000 7f543048b640 1 -- 172.21.15.26:0/4077764991 shutdown_connections 2024-09-16T14:54:20.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.602+0000 7f543048b640 1 -- 172.21.15.26:0/4077764991 >> 172.21.15.26:0/4077764991 conn(0x7f54280a7f30 msgr2=0x7f542806b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:20.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.602+0000 7f543048b640 1 -- 172.21.15.26:0/4077764991 shutdown_connections 2024-09-16T14:54:20.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.602+0000 7f543048b640 1 -- 172.21.15.26:0/4077764991 wait complete. 2024-09-16T14:54:20.623 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:54:20.623 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:54:20.623 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:54:20.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 10 monclient: get_monmap_and_config 2024-09-16T14:54:20.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 10 monclient: build_initial_monmap 2024-09-16T14:54:20.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 10 monclient: monmap: 2024-09-16T14:54:20.732 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:20.734045+0000 2024-09-16T14:54:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:20.734045+0000 2024-09-16T14:54:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:20.733 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:20.733 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 1 Processor -- start 2024-09-16T14:54:20.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 1 -- start start 2024-09-16T14:54:20.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 10 monclient: init 2024-09-16T14:54:20.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:20.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:54:20.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 10 monclient(hunting): picked mon.noname-c con 0x7fadec0a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:20.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 10 monclient(hunting): picked mon.noname-b con 0x7fadec0a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:20.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 10 monclient(hunting): picked mon.noname-a con 0x7fadec099ab0 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:20.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadec06cf60 con 0x7fadec099ab0 2024-09-16T14:54:20.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadec098df0 con 0x7fadec0a1c00 2024-09-16T14:54:20.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadec099f00 con 0x7fadec0a26f0 2024-09-16T14:54:20.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:20.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 10 monclient(hunting): authenticate will time out at 1684.854313s 2024-09-16T14:54:20.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadeb7fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fadec099ab0 legacy=0x7fadec13d540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:20.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadea7fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fadec0a26f0 0x7fadec139e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40906/0 (socket says 172.21.15.26:40906) 2024-09-16T14:54:20.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadea7fc640 1 -- 172.21.15.26:0/3040479141 learned_addr learned my addr 172.21.15.26:0/3040479141 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:20.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 1 -- 172.21.15.26:0/3040479141 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063589334 0 0) 0x7fadec099f00 con 0x7fadec0a26f0 2024-09-16T14:54:20.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:20.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:20.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:20.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 10 monclient(hunting): my global_id is 234284 2024-09-16T14:54:20.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 1 -- 172.21.15.26:0/3040479141 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd8003530 con 0x7fadec0a26f0 2024-09-16T14:54:20.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 1 -- 172.21.15.26:0/3040479141 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2605919789 0 0) 0x7fadd8003530 con 0x7fadec0a26f0 2024-09-16T14:54:20.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 1 -- 172.21.15.26:0/3040479141 >> v1:172.21.15.189:6789/0 conn(0x7fadec0a1c00 legacy=0x7fadec0a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 1 -- 172.21.15.26:0/3040479141 >> v1:172.21.15.26:6789/0 conn(0x7fadec099ab0 legacy=0x7fadec13d540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:54:20.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:54:20.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 1 monclient: found mon.noname-b 2024-09-16T14:54:20.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:20.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:54:20.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 1 -- 172.21.15.26:0/3040479141 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadec0a2d10 con 0x7fadec0a26f0 2024-09-16T14:54:20.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 10 monclient: _finish_auth 0 2024-09-16T14:54:20.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:54:20.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:20.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 5 monclient: authenticate success, global_id 234284 2024-09-16T14:54:20.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 1 -- 172.21.15.26:0/3040479141 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fadd4001620 con 0x7fadec0a26f0 2024-09-16T14:54:20.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:20.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:20.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:54:20.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 10 monclient: dump: 2024-09-16T14:54:20.745 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:20.745 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.745 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.745 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.746 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:20.746 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.746 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:20.746 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:20.746 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:20.746 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:20.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 1 -- 172.21.15.26:0/3040479141 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadd4001940 con 0x7fadec0a26f0 2024-09-16T14:54:20.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:20.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:20.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 4 set_mon_vals no callback set 2024-09-16T14:54:20.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 1 -- 172.21.15.26:0/3040479141 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fadd4001d10 con 0x7fadec0a26f0 2024-09-16T14:54:20.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:20.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:20.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fade9ffb640 10 monclient: dump: 2024-09-16T14:54:20.747 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:20.747 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.747 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.747 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.747 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:20.747 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.747 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:20.747 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:20.747 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:20.748 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 10 monclient: shutdown 2024-09-16T14:54:20.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:20.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 1 -- 172.21.15.26:0/3040479141 >> v1:172.21.15.26:6790/0 conn(0x7fadec0a26f0 legacy=0x7fadec139e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 1 -- 172.21.15.26:0/3040479141 shutdown_connections 2024-09-16T14:54:20.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 1 -- 172.21.15.26:0/3040479141 >> 172.21.15.26:0/3040479141 conn(0x7fadec0a3ed0 msgr2=0x7fadec0a42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:20.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 1 -- 172.21.15.26:0/3040479141 shutdown_connections 2024-09-16T14:54:20.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.730+0000 7fadf1482640 1 -- 172.21.15.26:0/3040479141 wait complete. 2024-09-16T14:54:20.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 10 monclient: build_initial_monmap 2024-09-16T14:54:20.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 10 monclient: monmap: 2024-09-16T14:54:20.748 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:20.748 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:20.748 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:20.749 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:20.749 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:20.749 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:20.749 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:20.749 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:20.749 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:20.749 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 1 Processor -- start 2024-09-16T14:54:20.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 1 -- start start 2024-09-16T14:54:20.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 10 monclient: init 2024-09-16T14:54:20.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:20.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:54:20.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 10 monclient(hunting): picked mon.noname-b con 0x7fadec099ab0 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:20.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 10 monclient(hunting): picked mon.noname-a con 0x7fadec0a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:20.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 10 monclient(hunting): picked mon.noname-c con 0x7fadec0a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:20.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadec13be60 con 0x7fadec0a1c00 2024-09-16T14:54:20.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadec13cba0 con 0x7fadec099ab0 2024-09-16T14:54:20.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadea7fc640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fadec0a1c00 legacy=0x7fadec139c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:20.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadec13d440 con 0x7fadec0a26f0 2024-09-16T14:54:20.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:20.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 10 monclient(hunting): authenticate will time out at 1684.857096s 2024-09-16T14:54:20.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadeb7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fadec0a26f0 0x7fadec1d1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40912/0 (socket says 172.21.15.26:40912) 2024-09-16T14:54:20.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadeb7fe640 1 -- 172.21.15.26:0/3116798103 learned_addr learned my addr 172.21.15.26:0/3116798103 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:20.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72798402 0 0) 0x7fadec13d440 con 0x7fadec0a26f0 2024-09-16T14:54:20.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:20.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:20.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:20.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient(hunting): my global_id is 234287 2024-09-16T14:54:20.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadcc003530 con 0x7fadec0a26f0 2024-09-16T14:54:20.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3974593323 0 0) 0x7fadcc003530 con 0x7fadec0a26f0 2024-09-16T14:54:20.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadec13d440 con 0x7fadec0a26f0 2024-09-16T14:54:20.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faddc004040 con 0x7fadec0a26f0 2024-09-16T14:54:20.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:20.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:20.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient(hunting): dump: 2024-09-16T14:54:20.751 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:20.751 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.751 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.752 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.752 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:20.752 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.752 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:20.752 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:20.752 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:20.752 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:20.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 65383909 0 0) 0x7fadec13d440 con 0x7fadec0a26f0 2024-09-16T14:54:20.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 >> v1:172.21.15.189:6789/0 conn(0x7fadec099ab0 legacy=0x7fadec139500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 >> v1:172.21.15.26:6789/0 conn(0x7fadec0a1c00 legacy=0x7fadec139c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:54:20.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:54:20.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 monclient: found mon.c 2024-09-16T14:54:20.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:20.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:20.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadec13a320 con 0x7fadec0a26f0 2024-09-16T14:54:20.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient: _finish_auth 0 2024-09-16T14:54:20.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:54:20.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:20.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 5 monclient: authenticate success, global_id 234287 2024-09-16T14:54:20.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 10 monclient: _renew_subs 2024-09-16T14:54:20.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:20.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 1 -- 172.21.15.26:0/3116798103 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadec09ec90 con 0x7fadec0a26f0 2024-09-16T14:54:20.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 10 monclient: _renew_subs 2024-09-16T14:54:20.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:20.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 1 -- 172.21.15.26:0/3116798103 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fadec13d160 con 0x7fadec0a26f0 2024-09-16T14:54:20.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faddc004360 con 0x7fadec0a26f0 2024-09-16T14:54:20.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:20.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faddc0046f0 con 0x7fadec0a26f0 2024-09-16T14:54:20.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:20.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:20.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade97fa640 4 set_mon_vals no callback set 2024-09-16T14:54:20.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient: dump: 2024-09-16T14:54:20.754 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:20.754 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.754 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.754 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.754 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:20.755 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.755 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:20.755 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:20.755 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:20.755 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7faddc0048d0 con 0x7fadec0a26f0 2024-09-16T14:54:20.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:20.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadf1482640 1 -- 172.21.15.26:0/3116798103 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fadd0006590 con 0x7fadd0002830 2024-09-16T14:54:20.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7faddc043de0 con 0x7fadec0a26f0 2024-09-16T14:54:20.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528264854 0 0) 0x7fadd0006590 con 0x7fadd0002830 2024-09-16T14:54:20.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:20.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:20.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:20.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient(hunting): my global_id is 234290 2024-09-16T14:54:20.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadec13d440 con 0x7fadd0002830 2024-09-16T14:54:20.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 90061955 0 0) 0x7fadec13d440 con 0x7fadd0002830 2024-09-16T14:54:20.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadd0006590 con 0x7fadd0002830 2024-09-16T14:54:20.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fadea7fc640 10 monclient: get_auth_request con 0x7fadcc036000 auth_method 0 2024-09-16T14:54:20.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fade40019d0 con 0x7fadd0002830 2024-09-16T14:54:20.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:20.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:20.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient: dump: 2024-09-16T14:54:20.756 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:20.756 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.757 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.757 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.757 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:20.757 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.757 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:20.757 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:20.757 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:20.757 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3176315754 0 0) 0x7fadd0006590 con 0x7fadd0002830 2024-09-16T14:54:20.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadd000a0e0 con 0x7fadd0002830 2024-09-16T14:54:20.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fade4001ee0 con 0x7fadd0002830 2024-09-16T14:54:20.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:20.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:20.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 >> v1:172.21.15.26:6790/0 conn(0x7fadd0002830 legacy=0x7fadd0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.734+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 >> v1:172.21.15.26:6790/0 conn(0x7fadd0002830 legacy=0x7fadd0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:20.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fadf1482640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:54:20.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fadf1482640 1 -- 172.21.15.26:0/3116798103 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fadd0009da0 con 0x7fadd00090c0 2024-09-16T14:54:20.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522382706 0 0) 0x7fadd0009da0 con 0x7fadd00090c0 2024-09-16T14:54:20.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:20.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:20.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:20.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 10 monclient(hunting): my global_id is 234293 2024-09-16T14:54:20.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd0006590 con 0x7fadd00090c0 2024-09-16T14:54:20.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1483197680 0 0) 0x7fadd0006590 con 0x7fadd00090c0 2024-09-16T14:54:20.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadd0009da0 con 0x7fadd00090c0 2024-09-16T14:54:20.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fade4003b40 con 0x7fadd00090c0 2024-09-16T14:54:20.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:20.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:20.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 10 monclient: dump: 2024-09-16T14:54:20.760 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:20.761 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.761 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.761 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.761 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:20.761 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.761 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:20.761 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:20.761 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:20.761 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 351236452 0 0) 0x7fadd0009da0 con 0x7fadd00090c0 2024-09-16T14:54:20.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fadd000c290 con 0x7fadd00090c0 2024-09-16T14:54:20.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4203 (unknown 0 0 2218320042) 0x7fade40049d0 con 0x7fadd00090c0 2024-09-16T14:54:20.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:54:20.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:20.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 >> v1:172.21.15.26:6790/0 conn(0x7fadd00090c0 legacy=0x7fadd0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fade37fe640 1 -- 172.21.15.26:0/3116798103 >> v1:172.21.15.26:6790/0 conn(0x7fadd00090c0 legacy=0x7fadd0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:20.763 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:54:20.763 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:20.763 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:20.763 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:54:20.763 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 112, 2024-09-16T14:54:20.763 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:54:20.763 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:54:20.763 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:54:20.764 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:20.764 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:54:20.764 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:20.764 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:54:20.764 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:54:20.764 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:20.764 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:20.764 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:20.764 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:20.764 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:20.764 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:20.764 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:20.764 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:20.765 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:20.765 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:20.765 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:20.765 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:54:20.765 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:54:20.765 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:20.765 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:20.765 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:20.765 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:20.766 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:20.766 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:20.766 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:20.766 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:20.766 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:20.766 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:20.766 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:20.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.766 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:54:20.766 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:54:20.766 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:54:20.766 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:54:20.766 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:54:20.767 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:54:20.767 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:20.767 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:20.767 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:54:20.767 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:54:20.767 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:54:20.767 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:54:20.767 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:54:20.767 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:54:20.767 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:54:20.767 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:20.767 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:54:20.767 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:20.767 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:20.768 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:20.768 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:20.768 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:20.768 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:20.768 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:20.768 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:20.768 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:20.768 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:20.768 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:20.768 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:54:20.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.768 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:54:20.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.769 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:20.769 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:20.769 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:20.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:20.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.769 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:20.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:54:20.769 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:20.769 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:20.769 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:20.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:54:20.770 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:54:20.770 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:20.770 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:20.770 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:20.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.770 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:20.770 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:20.770 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:20.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:20.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.770 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:20.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:54:20.771 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:20.771 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:20.771 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:20.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:54:20.771 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:54:20.771 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:20.771 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:20.771 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:20.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.771 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:20.772 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:20.772 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:20.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:20.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.772 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:20.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:54:20.772 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:20.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:20.772 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:20.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:54:20.773 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:54:20.773 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:20.773 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:20.773 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:20.773 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:20.773 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:20.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.774 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:54:20.774 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:54:20.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.774 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:20.774 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:20.774 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:20.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:20.774 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:20.774 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:54:20.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.774 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:20.774 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:20.775 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:54:20.775 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:20.775 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:20.775 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:54:20.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.775 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:20.775 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:20.776 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 7 2024-09-16T14:54:20.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:20.776 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:20.776 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:54:20.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.776 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:20.776 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:20.776 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:20.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:20.776 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:20.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.776 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:54:20.776 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:54:20.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fadf1482640 1 -- 172.21.15.26:0/3116798103 >> v1:172.21.15.189:6812/3196161582 conn(0x7fadcc036000 legacy=0x7fadcc0384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fadf1482640 10 monclient: shutdown 2024-09-16T14:54:20.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fadf1482640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:20.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fadf1482640 1 -- 172.21.15.26:0/3116798103 >> v1:172.21.15.26:6790/0 conn(0x7fadec0a26f0 legacy=0x7fadec1d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fadeb7fe640 1 -- 172.21.15.26:0/3116798103 reap_dead start 2024-09-16T14:54:20.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fadeb7fe640 1 -- 172.21.15.26:0/3116798103 reap_dead start 2024-09-16T14:54:20.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fadf1482640 1 -- 172.21.15.26:0/3116798103 shutdown_connections 2024-09-16T14:54:20.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fadf1482640 1 -- 172.21.15.26:0/3116798103 >> 172.21.15.26:0/3116798103 conn(0x7fadec0a3ed0 msgr2=0x7fadec0a67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:20.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fadf1482640 1 -- 172.21.15.26:0/3116798103 shutdown_connections 2024-09-16T14:54:20.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.742+0000 7fadf1482640 1 -- 172.21.15.26:0/3116798103 wait complete. 2024-09-16T14:54:20.779 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:54:20.779 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T14:54:20.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:54:20.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 10 monclient: get_monmap_and_config 2024-09-16T14:54:20.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 10 monclient: build_initial_monmap 2024-09-16T14:54:20.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 10 monclient: monmap: 2024-09-16T14:54:20.892 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:20.892 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.892 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:20.893627+0000 2024-09-16T14:54:20.892 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:20.893627+0000 2024-09-16T14:54:20.892 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:20.892 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.893 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:20.893 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:20.893 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:20.893 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 1 Processor -- start 2024-09-16T14:54:20.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 1 -- start start 2024-09-16T14:54:20.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 10 monclient: init 2024-09-16T14:54:20.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:20.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:54:20.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 10 monclient(hunting): picked mon.noname-a con 0x7f9e5c0a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:20.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 10 monclient(hunting): picked mon.noname-b con 0x7f9e5c0a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:20.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 10 monclient(hunting): picked mon.noname-c con 0x7f9e5c0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:20.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c0a3250 con 0x7f9e5c0a1d10 2024-09-16T14:54:20.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c09f600 con 0x7f9e5c0a2800 2024-09-16T14:54:20.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c0a3830 con 0x7f9e5c0a0000 2024-09-16T14:54:20.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:20.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 10 monclient(hunting): authenticate will time out at 1685.013642s 2024-09-16T14:54:20.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e613c1640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f9e5c0a1d10 legacy=0x7f9e5c0a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:20.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e60bc0640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9e5c0a0000 0x7f9e5c139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40948/0 (socket says 172.21.15.26:40948) 2024-09-16T14:54:20.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e60bc0640 1 -- 172.21.15.26:0/361899741 learned_addr learned my addr 172.21.15.26:0/361899741 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:20.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 1 -- 172.21.15.26:0/361899741 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3631303839 0 0) 0x7f9e5c0a3830 con 0x7f9e5c0a0000 2024-09-16T14:54:20.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:20.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:20.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:20.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 10 monclient(hunting): my global_id is 234296 2024-09-16T14:54:20.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 1 -- 172.21.15.26:0/361899741 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e44003530 con 0x7f9e5c0a0000 2024-09-16T14:54:20.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 1 -- 172.21.15.26:0/361899741 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1517168060 0 0) 0x7f9e44003530 con 0x7f9e5c0a0000 2024-09-16T14:54:20.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 1 -- 172.21.15.26:0/361899741 >> v1:172.21.15.189:6789/0 conn(0x7f9e5c0a2800 legacy=0x7f9e5c13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 1 -- 172.21.15.26:0/361899741 >> v1:172.21.15.26:6789/0 conn(0x7f9e5c0a1d10 legacy=0x7f9e5c0a20f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:54:20.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:54:20.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 1 monclient: found mon.noname-b 2024-09-16T14:54:20.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:20.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:54:20.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 1 -- 172.21.15.26:0/361899741 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c09ed60 con 0x7f9e5c0a0000 2024-09-16T14:54:20.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 10 monclient: _finish_auth 0 2024-09-16T14:54:20.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:54:20.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:20.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 5 monclient: authenticate success, global_id 234296 2024-09-16T14:54:20.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:20.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 1 -- 172.21.15.26:0/361899741 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e50001620 con 0x7f9e5c0a0000 2024-09-16T14:54:20.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:20.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:54:20.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 10 monclient: dump: 2024-09-16T14:54:20.902 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:20.903 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.903 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.903 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.903 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:20.903 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.903 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:20.903 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:20.903 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:20.903 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:20.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 1 -- 172.21.15.26:0/361899741 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e50001940 con 0x7f9e5c0a0000 2024-09-16T14:54:20.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:20.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 1 -- 172.21.15.26:0/361899741 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e50001cd0 con 0x7f9e5c0a0000 2024-09-16T14:54:20.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:20.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:20.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e5bfff640 10 monclient: dump: 2024-09-16T14:54:20.904 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:20.904 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.904 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.904 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.904 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:20.904 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.904 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:20.904 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:20.904 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:20.904 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:20.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 4 set_mon_vals no callback set 2024-09-16T14:54:20.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 10 monclient: shutdown 2024-09-16T14:54:20.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:20.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 1 -- 172.21.15.26:0/361899741 >> v1:172.21.15.26:6790/0 conn(0x7f9e5c0a0000 legacy=0x7f9e5c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 1 -- 172.21.15.26:0/361899741 shutdown_connections 2024-09-16T14:54:20.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 1 -- 172.21.15.26:0/361899741 >> 172.21.15.26:0/361899741 conn(0x7f9e5c0a7f30 msgr2=0x7f9e5c0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:20.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 1 -- 172.21.15.26:0/361899741 shutdown_connections 2024-09-16T14:54:20.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 1 -- 172.21.15.26:0/361899741 wait complete. 2024-09-16T14:54:20.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 10 monclient: build_initial_monmap 2024-09-16T14:54:20.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 10 monclient: monmap: 2024-09-16T14:54:20.905 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:20.905 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:20.905 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:20.906 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:20.906 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:20.906 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:20.906 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:20.906 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:20.906 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:20.906 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.890+0000 7f9e63625640 1 Processor -- start 2024-09-16T14:54:20.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 1 -- start start 2024-09-16T14:54:20.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 10 monclient: init 2024-09-16T14:54:20.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:20.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:54:20.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 10 monclient(hunting): picked mon.noname-c con 0x7f9e5c0a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:20.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 10 monclient(hunting): picked mon.noname-b con 0x7f9e5c0a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:20.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 10 monclient(hunting): picked mon.noname-a con 0x7f9e5c0a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:20.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c13a800 con 0x7f9e5c0a2800 2024-09-16T14:54:20.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c13a130 con 0x7f9e5c0a1d10 2024-09-16T14:54:20.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c13bb80 con 0x7f9e5c0a0000 2024-09-16T14:54:20.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:20.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 10 monclient(hunting): authenticate will time out at 1685.016481s 2024-09-16T14:54:20.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e61bc2640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f9e5c0a2800 legacy=0x7f9e5c1d1bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:54:20.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e613c1640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9e5c0a0000 0x7f9e5c139310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40964/0 (socket says 172.21.15.26:40964) 2024-09-16T14:54:20.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e613c1640 1 -- 172.21.15.26:0/3369215799 learned_addr learned my addr 172.21.15.26:0/3369215799 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:20.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929875643 0 0) 0x7f9e5c13bb80 con 0x7f9e5c0a0000 2024-09-16T14:54:20.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:20.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:20.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:20.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient(hunting): my global_id is 234299 2024-09-16T14:54:20.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e48003530 con 0x7f9e5c0a0000 2024-09-16T14:54:20.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 323183245 0 0) 0x7f9e48003530 con 0x7f9e5c0a0000 2024-09-16T14:54:20.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e5c13bb80 con 0x7f9e5c0a0000 2024-09-16T14:54:20.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e540044e0 con 0x7f9e5c0a0000 2024-09-16T14:54:20.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:20.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:20.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient(hunting): dump: 2024-09-16T14:54:20.909 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:20.909 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.909 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.909 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.909 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:20.909 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.909 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:20.909 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:20.909 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:20.909 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:20.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015046497 0 0) 0x7f9e5c13a130 con 0x7f9e5c0a1d10 2024-09-16T14:54:20.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:20.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:20.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:20.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient(hunting): my global_id is 244171 2024-09-16T14:54:20.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e48003530 con 0x7f9e5c0a1d10 2024-09-16T14:54:20.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 170800583 0 0) 0x7f9e5c13bb80 con 0x7f9e5c0a0000 2024-09-16T14:54:20.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 >> v1:172.21.15.189:6789/0 conn(0x7f9e5c0a1d10 legacy=0x7f9e5c139a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 >> v1:172.21.15.26:6789/0 conn(0x7f9e5c0a2800 legacy=0x7f9e5c1d1bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:54:20.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:54:20.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 monclient: found mon.c 2024-09-16T14:54:20.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:20.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:20.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c0a65f0 con 0x7f9e5c0a0000 2024-09-16T14:54:20.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient: _finish_auth 0 2024-09-16T14:54:20.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:54:20.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:20.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 5 monclient: authenticate success, global_id 234299 2024-09-16T14:54:20.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 10 monclient: _renew_subs 2024-09-16T14:54:20.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:20.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 1 -- 172.21.15.26:0/3369215799 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5c09e500 con 0x7f9e5c0a0000 2024-09-16T14:54:20.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 10 monclient: _renew_subs 2024-09-16T14:54:20.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:20.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 1 -- 172.21.15.26:0/3369215799 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5c13b900 con 0x7f9e5c0a0000 2024-09-16T14:54:20.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e54004700 con 0x7f9e5c0a0000 2024-09-16T14:54:20.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:20.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e5b7fe640 4 set_mon_vals no callback set 2024-09-16T14:54:20.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e54004a10 con 0x7f9e5c0a0000 2024-09-16T14:54:20.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:20.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:20.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient: dump: 2024-09-16T14:54:20.912 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:20.912 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.912 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.912 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.912 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:20.912 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.912 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:20.912 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:20.912 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:20.912 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:20.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e63625640 1 -- 172.21.15.26:0/3369215799 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9e3c006590 con 0x7f9e3c002830 2024-09-16T14:54:20.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9e54004bf0 con 0x7f9e5c0a0000 2024-09-16T14:54:20.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f9e540441d0 con 0x7f9e5c0a0000 2024-09-16T14:54:20.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e61bc2640 10 monclient: get_auth_request con 0x7f9e480361d0 auth_method 0 2024-09-16T14:54:20.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010058743 0 0) 0x7f9e3c006590 con 0x7f9e3c002830 2024-09-16T14:54:20.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:20.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:20.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:20.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient(hunting): my global_id is 244174 2024-09-16T14:54:20.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e5c13bb80 con 0x7f9e3c002830 2024-09-16T14:54:20.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3210812668 0 0) 0x7f9e5c13bb80 con 0x7f9e3c002830 2024-09-16T14:54:20.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e3c006590 con 0x7f9e3c002830 2024-09-16T14:54:20.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e50001d10 con 0x7f9e3c002830 2024-09-16T14:54:20.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:20.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:54:20.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 10 monclient: dump: 2024-09-16T14:54:20.914 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:20.914 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.914 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.914 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.914 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:20.914 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.914 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:20.914 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:20.914 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:20.914 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1348645477 0 0) 0x7f9e3c006590 con 0x7f9e3c002830 2024-09-16T14:54:20.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.894+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e3c00a0e0 con 0x7f9e3c002830 2024-09-16T14:54:20.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.898+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f9e500015f0 con 0x7f9e3c002830 2024-09-16T14:54:20.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.898+0000 7f9e59ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:20.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.898+0000 7f9e59ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:20.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.898+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 >> v1:172.21.15.189:6789/0 conn(0x7f9e3c002830 legacy=0x7f9e3c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.898+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 >> v1:172.21.15.189:6789/0 conn(0x7f9e3c002830 legacy=0x7f9e3c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:20.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.902+0000 7f9e63625640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:54:20.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.902+0000 7f9e63625640 1 -- 172.21.15.26:0/3369215799 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9e3c009da0 con 0x7f9e3c0090c0 2024-09-16T14:54:20.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.902+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505154402 0 0) 0x7f9e3c009da0 con 0x7f9e3c0090c0 2024-09-16T14:54:20.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.902+0000 7f9e59ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:20.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.902+0000 7f9e59ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:20.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.902+0000 7f9e59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:20.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.902+0000 7f9e59ffb640 10 monclient(hunting): my global_id is 244177 2024-09-16T14:54:20.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.902+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e3c006590 con 0x7f9e3c0090c0 2024-09-16T14:54:20.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3524153890 0 0) 0x7f9e3c006590 con 0x7f9e3c0090c0 2024-09-16T14:54:20.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e3c009da0 con 0x7f9e3c0090c0 2024-09-16T14:54:20.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e50004c90 con 0x7f9e3c0090c0 2024-09-16T14:54:20.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:20.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e59ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:54:20.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e59ffb640 10 monclient: dump: 2024-09-16T14:54:20.918 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:20.918 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:20.918 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.918 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:20.918 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:20.918 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:20.918 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:20.918 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:20.918 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:20.918 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:20.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2059190701 0 0) 0x7f9e3c009da0 con 0x7f9e3c0090c0 2024-09-16T14:54:20.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9e3c00c290 con 0x7f9e3c0090c0 2024-09-16T14:54:20.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3869 (unknown 0 0 3398287265) 0x7f9e50005f30 con 0x7f9e3c0090c0 2024-09-16T14:54:20.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e59ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:54:20.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e59ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:20.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 >> v1:172.21.15.189:6789/0 conn(0x7f9e3c0090c0 legacy=0x7f9e3c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e59ffb640 1 -- 172.21.15.26:0/3369215799 >> v1:172.21.15.189:6789/0 conn(0x7f9e3c0090c0 legacy=0x7f9e3c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:20.919 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:54:20.919 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:20.919 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:20.919 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:54:20.919 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 112, 2024-09-16T14:54:20.919 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:54:20.919 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:54:20.920 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:54:20.920 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:20.920 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:54:20.920 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:20.920 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:54:20.920 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:54:20.920 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:20.920 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:20.920 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:20.920 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:20.920 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:20.920 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:20.920 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:20.921 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:20.921 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:20.921 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:20.921 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:20.921 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:54:20.921 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:54:20.921 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:20.921 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:20.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:20.921 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:20.921 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:20.921 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:20.922 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:20.922 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:20.922 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:20.922 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:20.922 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:20.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.922 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:54:20.922 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:54:20.922 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:54:20.922 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:54:20.922 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:54:20.922 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:54:20.922 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:20.923 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:20.923 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:54:20.923 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:54:20.923 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:54:20.923 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:54:20.923 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:54:20.923 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:54:20.923 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:54:20.923 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:20.923 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:54:20.923 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:20.923 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:20.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:20.924 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:20.924 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:20.924 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:20.924 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:20.924 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:20.924 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:20.924 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:20.924 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:20.924 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:54:20.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:54:20.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.924 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:20.924 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:20.925 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:20.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:20.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.925 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:20.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:54:20.925 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:20.925 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:20.925 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:20.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:54:20.925 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:54:20.925 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:20.926 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:20.926 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:20.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.926 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:20.926 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:20.926 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:20.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:20.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.926 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:20.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:54:20.926 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:20.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:20.927 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:20.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:54:20.927 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:54:20.927 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:20.927 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:20.927 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:20.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.927 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:20.927 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:20.927 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:20.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:20.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.928 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:20.928 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:54:20.928 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:20.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:20.928 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:20.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.928 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:54:20.928 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:54:20.928 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:20.928 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:20.928 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:20.929 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:20.929 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:20.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.929 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:54:20.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:54:20.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.929 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:20.929 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:20.929 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:20.929 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:20.929 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:20.929 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:54:20.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:20.929 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:20.930 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:20.930 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:20.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:20.930 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:20.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:20.930 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:54:20.930 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:54:20.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e63625640 1 -- 172.21.15.26:0/3369215799 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9e480361d0 legacy=0x7f9e48038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e63625640 10 monclient: shutdown 2024-09-16T14:54:20.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e63625640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:20.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e63625640 1 -- 172.21.15.26:0/3369215799 >> v1:172.21.15.26:6790/0 conn(0x7f9e5c0a0000 legacy=0x7f9e5c139310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:20.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e61bc2640 1 -- 172.21.15.26:0/3369215799 reap_dead start 2024-09-16T14:54:20.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e61bc2640 1 -- 172.21.15.26:0/3369215799 reap_dead start 2024-09-16T14:54:20.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e63625640 1 -- 172.21.15.26:0/3369215799 shutdown_connections 2024-09-16T14:54:20.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e63625640 1 -- 172.21.15.26:0/3369215799 >> 172.21.15.26:0/3369215799 conn(0x7f9e5c0a7f30 msgr2=0x7f9e5c09d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:20.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e63625640 1 -- 172.21.15.26:0/3369215799 shutdown_connections 2024-09-16T14:54:20.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:20.906+0000 7f9e63625640 1 -- 172.21.15.26:0/3369215799 wait complete. 2024-09-16T14:54:20.932 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:54:20.932 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:54:40.934 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:54:40.934 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T14:54:40.934 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T14:54:40.934 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T14:54:40.938 INFO:tasks.ceph.mon.a:Started 2024-09-16T14:54:40.938 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:54:40.938 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:54:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 10 monclient: get_monmap_and_config 2024-09-16T14:54:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 10 monclient: build_initial_monmap 2024-09-16T14:54:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 10 monclient: monmap: 2024-09-16T14:54:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:41.074590+0000 2024-09-16T14:54:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:41.074590+0000 2024-09-16T14:54:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.074 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:41.074 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:41.074 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:41.074 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 1 Processor -- start 2024-09-16T14:54:41.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 1 -- start start 2024-09-16T14:54:41.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 10 monclient: init 2024-09-16T14:54:41.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:41.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:54:41.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 10 monclient(hunting): picked mon.noname-a con 0x7f5ce40a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:41.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 10 monclient(hunting): picked mon.noname-b con 0x7f5ce4099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:41.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 10 monclient(hunting): picked mon.noname-c con 0x7f5ce40a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:41.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce4098ea0 con 0x7f5ce40a1c00 2024-09-16T14:54:41.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce40997b0 con 0x7f5ce40a26f0 2024-09-16T14:54:41.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce4099460 con 0x7f5ce4099b60 2024-09-16T14:54:41.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:41.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 10 monclient(hunting): authenticate will time out at 1705.194241s 2024-09-16T14:54:41.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce259c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5ce4099b60 0x7f5ce4139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60270/0 (socket says 172.21.15.26:60270) 2024-09-16T14:54:41.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce259c640 1 -- 172.21.15.26:0/3274018531 learned_addr learned my addr 172.21.15.26:0/3274018531 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:41.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce2d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5ce40a1c00 0x7f5ce40a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58404/0 (socket says 172.21.15.26:58404) 2024-09-16T14:54:41.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 1 -- 172.21.15.26:0/3274018531 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542577652 0 0) 0x7f5ce4099460 con 0x7f5ce4099b60 2024-09-16T14:54:41.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 10 monclient(hunting): my global_id is 234302 2024-09-16T14:54:41.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 1 -- 172.21.15.26:0/3274018531 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ccc003530 con 0x7f5ce4099b60 2024-09-16T14:54:41.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 1 -- 172.21.15.26:0/3274018531 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544113624 0 0) 0x7f5ce40997b0 con 0x7f5ce40a26f0 2024-09-16T14:54:41.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 10 monclient(hunting): my global_id is 244180 2024-09-16T14:54:41.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 1 -- 172.21.15.26:0/3274018531 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce4099460 con 0x7f5ce40a26f0 2024-09-16T14:54:41.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 1 -- 172.21.15.26:0/3274018531 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3552132707 0 0) 0x7f5ce4099460 con 0x7f5ce40a26f0 2024-09-16T14:54:41.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 1 -- 172.21.15.26:0/3274018531 >> v1:172.21.15.26:6790/0 conn(0x7f5ce4099b60 legacy=0x7f5ce4139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 1 -- 172.21.15.26:0/3274018531 >> v1:172.21.15.26:6789/0 conn(0x7f5ce40a1c00 legacy=0x7f5ce40a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:54:41.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 1 monclient: found mon.noname-c 2024-09-16T14:54:41.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:41.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:54:41.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 1 -- 172.21.15.26:0/3274018531 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce40a34b0 con 0x7f5ce40a26f0 2024-09-16T14:54:41.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 10 monclient: _finish_auth 0 2024-09-16T14:54:41.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:54:41.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:41.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 1 -- 172.21.15.26:0/3274018531 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5cdc001980 con 0x7f5ce40a26f0 2024-09-16T14:54:41.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 5 monclient: authenticate success, global_id 244180 2024-09-16T14:54:41.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:41.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:41.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce1d9b640 10 monclient: dump: 2024-09-16T14:54:41.085 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.085 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.085 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.085 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.085 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.085 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.085 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.085 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.085 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.085 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.070+0000 7f5ce9227640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:41.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce1d9b640 1 -- 172.21.15.26:0/3274018531 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cdc001ca0 con 0x7f5ce40a26f0 2024-09-16T14:54:41.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:41.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce1d9b640 1 -- 172.21.15.26:0/3274018531 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5cdc002070 con 0x7f5ce40a26f0 2024-09-16T14:54:41.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:41.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 4 set_mon_vals no callback set 2024-09-16T14:54:41.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:54:41.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce1d9b640 10 monclient: dump: 2024-09-16T14:54:41.086 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.086 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.086 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.086 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.086 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.086 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.087 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.087 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.087 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.087 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 10 monclient: shutdown 2024-09-16T14:54:41.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:41.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 1 -- 172.21.15.26:0/3274018531 >> v1:172.21.15.189:6789/0 conn(0x7f5ce40a26f0 legacy=0x7f5ce413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 1 -- 172.21.15.26:0/3274018531 shutdown_connections 2024-09-16T14:54:41.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 1 -- 172.21.15.26:0/3274018531 >> 172.21.15.26:0/3274018531 conn(0x7f5ce40a3d00 msgr2=0x7f5ce40a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:41.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 1 -- 172.21.15.26:0/3274018531 shutdown_connections 2024-09-16T14:54:41.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 1 -- 172.21.15.26:0/3274018531 wait complete. 2024-09-16T14:54:41.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 10 monclient: build_initial_monmap 2024-09-16T14:54:41.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 10 monclient: monmap: 2024-09-16T14:54:41.088 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:41.088 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:41.088 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:41.088 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:41.088 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:41.088 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:41.088 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:41.088 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:41.088 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:41.088 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 1 Processor -- start 2024-09-16T14:54:41.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 1 -- start start 2024-09-16T14:54:41.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 10 monclient: init 2024-09-16T14:54:41.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:41.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:54:41.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 10 monclient(hunting): picked mon.noname-c con 0x7f5ce4099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:41.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 10 monclient(hunting): picked mon.noname-b con 0x7f5ce40a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:41.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 10 monclient(hunting): picked mon.noname-a con 0x7f5ce40a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:41.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce409f140 con 0x7f5ce40a26f0 2024-09-16T14:54:41.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce409e3f0 con 0x7f5ce40a1c00 2024-09-16T14:54:41.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce411d850 con 0x7f5ce4099b60 2024-09-16T14:54:41.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:41.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 10 monclient(hunting): authenticate will time out at 1705.198177s 2024-09-16T14:54:41.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce2d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5ce4099b60 0x7f5ce413ce80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60278/0 (socket says 172.21.15.26:60278) 2024-09-16T14:54:41.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce2d9d640 1 -- 172.21.15.26:0/3173022500 learned_addr learned my addr 172.21.15.26:0/3173022500 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:41.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456359191 0 0) 0x7f5ce411d850 con 0x7f5ce4099b60 2024-09-16T14:54:41.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient(hunting): my global_id is 234305 2024-09-16T14:54:41.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cd0003530 con 0x7f5ce4099b60 2024-09-16T14:54:41.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825337148 0 0) 0x7f5ce40a34b0 con 0x7f5ce40a26f0 2024-09-16T14:54:41.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient(hunting): my global_id is 254097 2024-09-16T14:54:41.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce411d850 con 0x7f5ce40a26f0 2024-09-16T14:54:41.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2020686370 0 0) 0x7f5cd0003530 con 0x7f5ce4099b60 2024-09-16T14:54:41.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ce40a34b0 con 0x7f5ce4099b60 2024-09-16T14:54:41.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3747817516 0 0) 0x7f5ce411d850 con 0x7f5ce40a26f0 2024-09-16T14:54:41.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cd0003530 con 0x7f5ce40a26f0 2024-09-16T14:54:41.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5cd40018c0 con 0x7f5ce4099b60 2024-09-16T14:54:41.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:41.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient(hunting): dump: 2024-09-16T14:54:41.091 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.091 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.091 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.091 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.092 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.092 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.092 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.092 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.092 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.092 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:41.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5cdc001e10 con 0x7f5ce40a26f0 2024-09-16T14:54:41.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:41.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient(hunting): dump: 2024-09-16T14:54:41.092 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.092 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.092 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.093 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.093 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.093 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.093 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.093 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.093 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.093 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2710505731 0 0) 0x7f5ce40a34b0 con 0x7f5ce4099b60 2024-09-16T14:54:41.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 >> v1:172.21.15.189:6789/0 conn(0x7f5ce40a1c00 legacy=0x7f5ce413d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 >> v1:172.21.15.26:6789/0 conn(0x7f5ce40a26f0 legacy=0x7f5ce413dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:54:41.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 monclient: found mon.c 2024-09-16T14:54:41.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce409eb80 con 0x7f5ce4099b60 2024-09-16T14:54:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient: _finish_auth 0 2024-09-16T14:54:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:54:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 5 monclient: authenticate success, global_id 234305 2024-09-16T14:54:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 10 monclient: _renew_subs 2024-09-16T14:54:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 1 -- 172.21.15.26:0/3173022500 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce411fc00 con 0x7f5ce4099b60 2024-09-16T14:54:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 10 monclient: _renew_subs 2024-09-16T14:54:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce9227640 1 -- 172.21.15.26:0/3173022500 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce41384f0 con 0x7f5ce4099b60 2024-09-16T14:54:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cd40047f0 con 0x7f5ce4099b60 2024-09-16T14:54:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:41.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5cd4004b80 con 0x7f5ce4099b60 2024-09-16T14:54:41.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:41.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ce159a640 4 set_mon_vals no callback set 2024-09-16T14:54:41.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.074+0000 7f5ccb7fe640 10 monclient: dump: 2024-09-16T14:54:41.095 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.095 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.095 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.095 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.095 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.095 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.095 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.096 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.096 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.096 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.078+0000 7f5ce9227640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:41.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.078+0000 7f5ce9227640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:41.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.078+0000 7f5ce9227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:41.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.078+0000 7f5ce9227640 1 -- 172.21.15.26:0/3173022500 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc4005050 con 0x7f5ce4099b60 2024-09-16T14:54:41.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.078+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5cd4004d60 con 0x7f5ce4099b60 2024-09-16T14:54:41.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.078+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f5cd4043e00 con 0x7f5ce4099b60 2024-09-16T14:54:41.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.078+0000 7f5ce259c640 10 monclient: get_auth_request con 0x7f5cd0036300 auth_method 0 2024-09-16T14:54:41.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.082+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5cd4014760 con 0x7f5ce4099b60 2024-09-16T14:54:41.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.082+0000 7f5ccb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:41.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.082+0000 7f5ccb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:41.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.302+0000 7f5ce9227640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:54:41.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.302+0000 7f5ce9227640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:41.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.302+0000 7f5ce9227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:41.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.302+0000 7f5ce9227640 1 -- 172.21.15.26:0/3173022500 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5cc4005880 con 0x7f5ce4099b60 2024-09-16T14:54:41.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.302+0000 7f5ccb7fe640 1 -- 172.21.15.26:0/3173022500 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 3145610899) 0x7f5cd4012bd0 con 0x7f5ce4099b60 2024-09-16T14:54:41.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.302+0000 7f5ccb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:41.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.302+0000 7f5ccb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:41.305 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:54:41.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.302+0000 7f5ce9227640 1 -- 172.21.15.26:0/3173022500 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5cd0036300 legacy=0x7f5cd00387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.302+0000 7f5ce9227640 10 monclient: shutdown 2024-09-16T14:54:41.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.302+0000 7f5ce9227640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:41.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.302+0000 7f5ce9227640 1 -- 172.21.15.26:0/3173022500 >> v1:172.21.15.26:6790/0 conn(0x7f5ce4099b60 legacy=0x7f5ce413ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.302+0000 7f5ce9227640 1 -- 172.21.15.26:0/3173022500 shutdown_connections 2024-09-16T14:54:41.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.302+0000 7f5ce9227640 1 -- 172.21.15.26:0/3173022500 >> 172.21.15.26:0/3173022500 conn(0x7f5ce40a3d00 msgr2=0x7f5ce40a0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:41.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.302+0000 7f5ce9227640 1 -- 172.21.15.26:0/3173022500 shutdown_connections 2024-09-16T14:54:41.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.302+0000 7f5ce9227640 1 -- 172.21.15.26:0/3173022500 wait complete. 2024-09-16T14:54:41.324 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:54:41.324 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:54:41.324 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:54:41.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.446+0000 7f7a62aa5640 10 monclient: get_monmap_and_config 2024-09-16T14:54:41.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.446+0000 7f7a62aa5640 10 monclient: build_initial_monmap 2024-09-16T14:54:41.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.446+0000 7f7a62aa5640 10 monclient: monmap: 2024-09-16T14:54:41.452 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:41.452 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.452 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:41.453453+0000 2024-09-16T14:54:41.452 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:41.453453+0000 2024-09-16T14:54:41.452 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:41.452 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.452 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:41.452 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:41.452 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:41.453 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 1 Processor -- start 2024-09-16T14:54:41.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 1 -- start start 2024-09-16T14:54:41.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 10 monclient: init 2024-09-16T14:54:41.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:41.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:54:41.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 10 monclient(hunting): picked mon.noname-a con 0x7f7a5c09cdc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:41.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 10 monclient(hunting): picked mon.noname-c con 0x7f7a5c09e7a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:41.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 10 monclient(hunting): picked mon.noname-b con 0x7f7a5c09d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:41.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c09d5b0 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c131a90 con 0x7f7a5c09e7a0 2024-09-16T14:54:41.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c09aff0 con 0x7f7a5c09d1a0 2024-09-16T14:54:41.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:41.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 10 monclient(hunting): authenticate will time out at 1705.573446s 2024-09-16T14:54:41.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a60841640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7a5c09cdc0 0x7f7a5c130c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58420/0 (socket says 172.21.15.26:58420) 2024-09-16T14:54:41.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a60841640 1 -- 172.21.15.26:0/920241571 learned_addr learned my addr 172.21.15.26:0/920241571 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:41.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 1 -- 172.21.15.26:0/920241571 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605197780 0 0) 0x7f7a5c09d5b0 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient(hunting): my global_id is 254100 2024-09-16T14:54:41.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 1 -- 172.21.15.26:0/920241571 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a44003530 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 1 -- 172.21.15.26:0/920241571 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005696412 0 0) 0x7f7a5c09aff0 con 0x7f7a5c09d1a0 2024-09-16T14:54:41.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient(hunting): my global_id is 234308 2024-09-16T14:54:41.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 1 -- 172.21.15.26:0/920241571 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a5c09d5b0 con 0x7f7a5c09d1a0 2024-09-16T14:54:41.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 1 -- 172.21.15.26:0/920241571 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 986862502 0 0) 0x7f7a44003530 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 1 -- 172.21.15.26:0/920241571 >> v1:172.21.15.26:6790/0 conn(0x7f7a5c09d1a0 legacy=0x7f7a5c139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 1 -- 172.21.15.26:0/920241571 >> v1:172.21.15.189:6789/0 conn(0x7f7a5c09e7a0 legacy=0x7f7a5c131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:54:41.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 1 monclient: found mon.noname-a 2024-09-16T14:54:41.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:41.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:54:41.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 1 -- 172.21.15.26:0/920241571 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c09b2d0 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:54:41.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:54:41.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:41.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 1 -- 172.21.15.26:0/920241571 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a4c001620 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:41.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient: dump: 2024-09-16T14:54:41.459 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.459 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.459 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.459 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.459 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.459 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.459 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.459 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.459 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.459 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 5 monclient: authenticate success, global_id 254100 2024-09-16T14:54:41.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:41.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 1 -- 172.21.15.26:0/920241571 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a4c001840 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:41.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 1 -- 172.21.15.26:0/920241571 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a4c001b50 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:41.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 4 set_mon_vals no callback set 2024-09-16T14:54:41.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:41.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a5b7fe640 10 monclient: dump: 2024-09-16T14:54:41.460 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.460 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.460 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.460 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.460 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.460 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.460 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.461 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.461 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.461 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 10 monclient: shutdown 2024-09-16T14:54:41.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:41.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 1 -- 172.21.15.26:0/920241571 >> v1:172.21.15.26:6789/0 conn(0x7f7a5c09cdc0 legacy=0x7f7a5c130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 1 -- 172.21.15.26:0/920241571 shutdown_connections 2024-09-16T14:54:41.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 1 -- 172.21.15.26:0/920241571 >> 172.21.15.26:0/920241571 conn(0x7f7a5c09fb50 msgr2=0x7f7a5c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:41.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 1 -- 172.21.15.26:0/920241571 shutdown_connections 2024-09-16T14:54:41.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 1 -- 172.21.15.26:0/920241571 wait complete. 2024-09-16T14:54:41.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 10 monclient: build_initial_monmap 2024-09-16T14:54:41.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 10 monclient: monmap: 2024-09-16T14:54:41.461 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:41.462 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:41.462 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:41.462 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:41.462 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:41.462 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:41.462 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:41.462 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:41.462 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:41.462 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.450+0000 7f7a62aa5640 1 Processor -- start 2024-09-16T14:54:41.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 1 -- start start 2024-09-16T14:54:41.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 10 monclient: init 2024-09-16T14:54:41.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:41.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:54:41.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 10 monclient(hunting): picked mon.noname-a con 0x7f7a5c09cdc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:41.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 10 monclient(hunting): picked mon.noname-c con 0x7f7a5c09d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:41.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 10 monclient(hunting): picked mon.noname-b con 0x7f7a5c09e7a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:41.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c136620 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c136900 con 0x7f7a5c09e7a0 2024-09-16T14:54:41.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c135170 con 0x7f7a5c09d1a0 2024-09-16T14:54:41.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:41.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 10 monclient(hunting): authenticate will time out at 1705.576514s 2024-09-16T14:54:41.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a60841640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7a5c09cdc0 0x7f7a5c1c79b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58424/0 (socket says 172.21.15.26:58424) 2024-09-16T14:54:41.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a60841640 1 -- 172.21.15.26:0/3190951500 learned_addr learned my addr 172.21.15.26:0/3190951500 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:41.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291262452 0 0) 0x7f7a5c136620 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): my global_id is 254103 2024-09-16T14:54:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a48003530 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536795487 0 0) 0x7f7a5c135170 con 0x7f7a5c09d1a0 2024-09-16T14:54:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): my global_id is 234311 2024-09-16T14:54:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a5c136620 con 0x7f7a5c09d1a0 2024-09-16T14:54:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 414922063 0 0) 0x7f7a5c136620 con 0x7f7a5c09d1a0 2024-09-16T14:54:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a5c135170 con 0x7f7a5c09d1a0 2024-09-16T14:54:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 180652786 0 0) 0x7f7a48003530 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a5c136620 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a500018c0 con 0x7f7a5c09d1a0 2024-09-16T14:54:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): dump: 2024-09-16T14:54:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.471 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a4c001520 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): dump: 2024-09-16T14:54:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.472 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596009381 0 0) 0x7f7a5c136900 con 0x7f7a5c09e7a0 2024-09-16T14:54:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): my global_id is 244189 2024-09-16T14:54:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a48003530 con 0x7f7a5c09e7a0 2024-09-16T14:54:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1173344938 0 0) 0x7f7a5c136620 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 >> v1:172.21.15.26:6790/0 conn(0x7f7a5c09d1a0 legacy=0x7f7a5c1cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 >> v1:172.21.15.189:6789/0 conn(0x7f7a5c09e7a0 legacy=0x7f7a5c1d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient: _finish_hunting 0 2024-09-16T14:54:41.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 monclient: found mon.a 2024-09-16T14:54:41.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:41.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:41.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c13f540 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient: _finish_auth 0 2024-09-16T14:54:41.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient: _check_auth_tickets 2024-09-16T14:54:41.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:41.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 5 monclient: authenticate success, global_id 254103 2024-09-16T14:54:41.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 10 monclient: _renew_subs 2024-09-16T14:54:41.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:41.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 1 -- 172.21.15.26:0/3190951500 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a5c13f010 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 10 monclient: _renew_subs 2024-09-16T14:54:41.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 1 -- 172.21.15.26:0/3190951500 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5c136340 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a4c001160 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a4c004c40 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient: dump: 2024-09-16T14:54:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.474 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.475 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a5affd640 4 set_mon_vals no callback set 2024-09-16T14:54:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a62aa5640 1 -- 172.21.15.26:0/3190951500 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7a3c006590 con 0x7f7a3c002830 2024-09-16T14:54:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7a4c00f6b0 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f7a4c043e40 con 0x7f7a5c09cdc0 2024-09-16T14:54:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451919386 0 0) 0x7f7a3c006590 con 0x7f7a3c002830 2024-09-16T14:54:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient(hunting): my global_id is 254106 2024-09-16T14:54:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a5c136620 con 0x7f7a3c002830 2024-09-16T14:54:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a61042640 10 monclient: get_auth_request con 0x7f7a48036390 auth_method 0 2024-09-16T14:54:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3578579532 0 0) 0x7f7a5c136620 con 0x7f7a3c002830 2024-09-16T14:54:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a3c006590 con 0x7f7a3c002830 2024-09-16T14:54:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a50003a70 con 0x7f7a3c002830 2024-09-16T14:54:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient: dump: 2024-09-16T14:54:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.477 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 980618955 0 0) 0x7f7a3c006590 con 0x7f7a3c002830 2024-09-16T14:54:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a3c00a0e0 con 0x7f7a3c002830 2024-09-16T14:54:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7a50003e30 con 0x7f7a3c002830 2024-09-16T14:54:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 >> v1:172.21.15.26:6789/0 conn(0x7f7a3c002830 legacy=0x7f7a3c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.454+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 >> v1:172.21.15.26:6789/0 conn(0x7f7a3c002830 legacy=0x7f7a3c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a62aa5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:54:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a62aa5640 1 -- 172.21.15.26:0/3190951500 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7a3c009da0 con 0x7f7a3c0090c0 2024-09-16T14:54:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757268026 0 0) 0x7f7a3c009da0 con 0x7f7a3c0090c0 2024-09-16T14:54:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 10 monclient(hunting): my global_id is 254109 2024-09-16T14:54:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a3c006590 con 0x7f7a3c0090c0 2024-09-16T14:54:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4072618450 0 0) 0x7f7a3c006590 con 0x7f7a3c0090c0 2024-09-16T14:54:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a3c009da0 con 0x7f7a3c0090c0 2024-09-16T14:54:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7a500043b0 con 0x7f7a3c0090c0 2024-09-16T14:54:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 10 monclient: dump: 2024-09-16T14:54:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.479 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 751642383 0 0) 0x7f7a3c009da0 con 0x7f7a3c0090c0 2024-09-16T14:54:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7a3c00c290 con 0x7f7a3c0090c0 2024-09-16T14:54:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3582219095) 0x7f7a500065b0 con 0x7f7a3c0090c0 2024-09-16T14:54:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:54:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 >> v1:172.21.15.26:6789/0 conn(0x7f7a3c0090c0 legacy=0x7f7a3c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a597fa640 1 -- 172.21.15.26:0/3190951500 >> v1:172.21.15.26:6789/0 conn(0x7f7a3c0090c0 legacy=0x7f7a3c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:41.480 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:54:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:54:41.480 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 114, 2024-09-16T14:54:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:54:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:54:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:54:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:54:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:54:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:54:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:54:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:41.481 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:54:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:54:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:41.482 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:54:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:54:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:54:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:54:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:54:41.483 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:54:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:54:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:54:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:54:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:54:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:54:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:54:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:54:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:54:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:41.484 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:54:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:54:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.485 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:54:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:54:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:54:41.486 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:54:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.487 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:54:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:54:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:41.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:54:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:54:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:54:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:54:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:54:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:54:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:54:41.490 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.491 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:54:41.491 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:54:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a62aa5640 1 -- 172.21.15.26:0/3190951500 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7a48036390 legacy=0x7f7a48038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a62aa5640 10 monclient: shutdown 2024-09-16T14:54:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a62aa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a62aa5640 1 -- 172.21.15.26:0/3190951500 >> v1:172.21.15.26:6789/0 conn(0x7f7a5c09cdc0 legacy=0x7f7a5c1c79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a61042640 1 -- 172.21.15.26:0/3190951500 reap_dead start 2024-09-16T14:54:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a61042640 1 -- 172.21.15.26:0/3190951500 reap_dead start 2024-09-16T14:54:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a62aa5640 1 -- 172.21.15.26:0/3190951500 shutdown_connections 2024-09-16T14:54:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a62aa5640 1 -- 172.21.15.26:0/3190951500 >> 172.21.15.26:0/3190951500 conn(0x7f7a5c09fb50 msgr2=0x7f7a5c12eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a62aa5640 1 -- 172.21.15.26:0/3190951500 shutdown_connections 2024-09-16T14:54:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.462+0000 7f7a62aa5640 1 -- 172.21.15.26:0/3190951500 wait complete. 2024-09-16T14:54:41.493 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:54:41.493 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:54:41.493 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:54:41.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 10 monclient: get_monmap_and_config 2024-09-16T14:54:41.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 10 monclient: build_initial_monmap 2024-09-16T14:54:41.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 10 monclient: monmap: 2024-09-16T14:54:41.609 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:41.609 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.609 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:41.610403+0000 2024-09-16T14:54:41.609 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:41.610403+0000 2024-09-16T14:54:41.609 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:41.609 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.609 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:41.609 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:41.609 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:41.609 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 1 Processor -- start 2024-09-16T14:54:41.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 1 -- start start 2024-09-16T14:54:41.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 10 monclient: init 2024-09-16T14:54:41.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:41.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:54:41.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 10 monclient(hunting): picked mon.noname-a con 0x7f435009a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:41.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 10 monclient(hunting): picked mon.noname-c con 0x7f435009d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:41.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 10 monclient(hunting): picked mon.noname-b con 0x7f435009b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:41.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435009dc60 con 0x7f435009a550 2024-09-16T14:54:41.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4350099ae0 con 0x7f435009d1b0 2024-09-16T14:54:41.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435009b8f0 con 0x7f435009b040 2024-09-16T14:54:41.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:41.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 10 monclient(hunting): authenticate will time out at 1705.730426s 2024-09-16T14:54:41.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4356398640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f435009a550 0x7f435009a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58450/0 (socket says 172.21.15.26:58450) 2024-09-16T14:54:41.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4356398640 1 -- 172.21.15.26:0/1998631927 learned_addr learned my addr 172.21.15.26:0/1998631927 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:41.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 1 -- 172.21.15.26:0/1998631927 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641839956 0 0) 0x7f435009dc60 con 0x7f435009a550 2024-09-16T14:54:41.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient(hunting): my global_id is 254112 2024-09-16T14:54:41.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 1 -- 172.21.15.26:0/1998631927 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f433c003530 con 0x7f435009a550 2024-09-16T14:54:41.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 1 -- 172.21.15.26:0/1998631927 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023180836 0 0) 0x7f435009b8f0 con 0x7f435009b040 2024-09-16T14:54:41.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient(hunting): my global_id is 234314 2024-09-16T14:54:41.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 1 -- 172.21.15.26:0/1998631927 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435009dc60 con 0x7f435009b040 2024-09-16T14:54:41.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 1 -- 172.21.15.26:0/1998631927 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3441008279 0 0) 0x7f435009dc60 con 0x7f435009b040 2024-09-16T14:54:41.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 1 -- 172.21.15.26:0/1998631927 >> v1:172.21.15.189:6789/0 conn(0x7f435009d1b0 legacy=0x7f4350136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 1 -- 172.21.15.26:0/1998631927 >> v1:172.21.15.26:6789/0 conn(0x7f435009a550 legacy=0x7f435009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient: _finish_hunting 0 2024-09-16T14:54:41.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 1 monclient: found mon.noname-b 2024-09-16T14:54:41.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:41.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:54:41.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 1 -- 172.21.15.26:0/1998631927 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435009bbd0 con 0x7f435009b040 2024-09-16T14:54:41.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient: _finish_auth 0 2024-09-16T14:54:41.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient: _check_auth_tickets 2024-09-16T14:54:41.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:41.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 1 -- 172.21.15.26:0/1998631927 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4348001620 con 0x7f435009b040 2024-09-16T14:54:41.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:54:41.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient: dump: 2024-09-16T14:54:41.619 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.619 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.620 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.620 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.620 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.620 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.620 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.620 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.620 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.620 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 5 monclient: authenticate success, global_id 234314 2024-09-16T14:54:41.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:41.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 1 -- 172.21.15.26:0/1998631927 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43480017d0 con 0x7f435009b040 2024-09-16T14:54:41.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:41.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 1 -- 172.21.15.26:0/1998631927 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4348001ba0 con 0x7f435009b040 2024-09-16T14:54:41.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:41.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 4 set_mon_vals no callback set 2024-09-16T14:54:41.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:41.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f4355396640 10 monclient: dump: 2024-09-16T14:54:41.621 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.621 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.621 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.621 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.621 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.621 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.621 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.621 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.622 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.622 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 10 monclient: shutdown 2024-09-16T14:54:41.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:41.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.606+0000 7f43585fc640 1 -- 172.21.15.26:0/1998631927 >> v1:172.21.15.26:6790/0 conn(0x7f435009b040 legacy=0x7f435013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 1 -- 172.21.15.26:0/1998631927 shutdown_connections 2024-09-16T14:54:41.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 1 -- 172.21.15.26:0/1998631927 >> 172.21.15.26:0/1998631927 conn(0x7f435009fb50 msgr2=0x7f435009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:41.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 1 -- 172.21.15.26:0/1998631927 shutdown_connections 2024-09-16T14:54:41.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 1 -- 172.21.15.26:0/1998631927 wait complete. 2024-09-16T14:54:41.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 10 monclient: build_initial_monmap 2024-09-16T14:54:41.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 10 monclient: monmap: 2024-09-16T14:54:41.622 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:41.622 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:41.622 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:41.623 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:41.623 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:41.623 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:41.623 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:41.623 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:41.623 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:41.623 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 1 Processor -- start 2024-09-16T14:54:41.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 1 -- start start 2024-09-16T14:54:41.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 10 monclient: init 2024-09-16T14:54:41.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:41.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:54:41.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 10 monclient(hunting): picked mon.noname-b con 0x7f435009a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:41.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 10 monclient(hunting): picked mon.noname-a con 0x7f435009b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:41.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 10 monclient(hunting): picked mon.noname-c con 0x7f435009d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:41.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43501a62b0 con 0x7f435009b040 2024-09-16T14:54:41.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43501a6100 con 0x7f435009a550 2024-09-16T14:54:41.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43500a23a0 con 0x7f435009d1b0 2024-09-16T14:54:41.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:41.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 10 monclient(hunting): authenticate will time out at 1705.733580s 2024-09-16T14:54:41.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4355b97640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f435009b040 0x7f43501a3ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58460/0 (socket says 172.21.15.26:58460) 2024-09-16T14:54:41.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4356b99640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f435009d1b0 0x7f43501d1730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60320/0 (socket says 172.21.15.26:60320) 2024-09-16T14:54:41.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4355b97640 1 -- 172.21.15.26:0/3423221881 learned_addr learned my addr 172.21.15.26:0/3423221881 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:41.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225815935 0 0) 0x7f43500a23a0 con 0x7f435009d1b0 2024-09-16T14:54:41.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): my global_id is 234317 2024-09-16T14:54:41.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4334003530 con 0x7f435009d1b0 2024-09-16T14:54:41.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883878423 0 0) 0x7f43501a62b0 con 0x7f435009b040 2024-09-16T14:54:41.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): my global_id is 254115 2024-09-16T14:54:41.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43500a23a0 con 0x7f435009b040 2024-09-16T14:54:41.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3601595024 0 0) 0x7f43500a23a0 con 0x7f435009b040 2024-09-16T14:54:41.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43501a62b0 con 0x7f435009b040 2024-09-16T14:54:41.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f434c001d00 con 0x7f435009b040 2024-09-16T14:54:41.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:41.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): dump: 2024-09-16T14:54:41.626 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.626 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.626 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.626 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.626 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.626 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.626 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.626 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.626 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.626 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:41.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2423892673 0 0) 0x7f4334003530 con 0x7f435009d1b0 2024-09-16T14:54:41.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43500a23a0 con 0x7f435009d1b0 2024-09-16T14:54:41.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f43480015e0 con 0x7f435009d1b0 2024-09-16T14:54:41.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:41.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): dump: 2024-09-16T14:54:41.627 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.627 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.627 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.627 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.628 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.628 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.628 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.628 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.628 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.628 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621698061 0 0) 0x7f43501a6100 con 0x7f435009a550 2024-09-16T14:54:41.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): my global_id is 244195 2024-09-16T14:54:41.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4334003530 con 0x7f435009a550 2024-09-16T14:54:41.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1303380825 0 0) 0x7f43501a62b0 con 0x7f435009b040 2024-09-16T14:54:41.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 >> v1:172.21.15.26:6790/0 conn(0x7f435009d1b0 legacy=0x7f43501d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 >> v1:172.21.15.189:6789/0 conn(0x7f435009a550 legacy=0x7f43501a38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:54:41.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 monclient: found mon.a 2024-09-16T14:54:41.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:41.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:41.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43501a7820 con 0x7f435009b040 2024-09-16T14:54:41.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient: _finish_auth 0 2024-09-16T14:54:41.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:54:41.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:41.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 5 monclient: authenticate success, global_id 254115 2024-09-16T14:54:41.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 10 monclient: _renew_subs 2024-09-16T14:54:41.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:41.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 1 -- 172.21.15.26:0/3423221881 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43501a4700 con 0x7f435009b040 2024-09-16T14:54:41.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 10 monclient: _renew_subs 2024-09-16T14:54:41.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:41.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 1 -- 172.21.15.26:0/3423221881 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43501a7540 con 0x7f435009b040 2024-09-16T14:54:41.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f434c0039e0 con 0x7f435009b040 2024-09-16T14:54:41.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:41.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f434c003cf0 con 0x7f435009b040 2024-09-16T14:54:41.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:41.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4343fff640 4 set_mon_vals no callback set 2024-09-16T14:54:41.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient: dump: 2024-09-16T14:54:41.630 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.631 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.631 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.631 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.631 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.631 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.631 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.631 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.631 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.631 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:41.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f43585fc640 1 -- 172.21.15.26:0/3423221881 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4338006590 con 0x7f4338002830 2024-09-16T14:54:41.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f434c003e50 con 0x7f435009b040 2024-09-16T14:54:41.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f434c044180 con 0x7f435009b040 2024-09-16T14:54:41.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927397670 0 0) 0x7f4338006590 con 0x7f4338002830 2024-09-16T14:54:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 10 monclient(hunting): my global_id is 234320 2024-09-16T14:54:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.610+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43501a62b0 con 0x7f4338002830 2024-09-16T14:54:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.614+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1875996859 0 0) 0x7f43501a62b0 con 0x7f4338002830 2024-09-16T14:54:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.614+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4338006590 con 0x7f4338002830 2024-09-16T14:54:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.614+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4344001140 con 0x7f4338002830 2024-09-16T14:54:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.614+0000 7f4342ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.614+0000 7f4342ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.614+0000 7f4342ffd640 10 monclient: dump: 2024-09-16T14:54:41.632 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.633 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.633 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.633 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.633 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.633 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.633 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.633 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.633 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.633 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.614+0000 7f4356b99640 10 monclient: get_auth_request con 0x7f4334036470 auth_method 0 2024-09-16T14:54:41.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.614+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4195965499 0 0) 0x7f4338006590 con 0x7f4338002830 2024-09-16T14:54:41.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.614+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f433800a0e0 con 0x7f4338002830 2024-09-16T14:54:41.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.614+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f43440014c0 con 0x7f4338002830 2024-09-16T14:54:41.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.614+0000 7f4342ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:41.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.614+0000 7f4342ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:41.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.614+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 >> v1:172.21.15.26:6790/0 conn(0x7f4338002830 legacy=0x7f4338004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.614+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 >> v1:172.21.15.26:6790/0 conn(0x7f4338002830 legacy=0x7f4338004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:41.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f43585fc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:54:41.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f43585fc640 1 -- 172.21.15.26:0/3423221881 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4338009da0 con 0x7f43380090c0 2024-09-16T14:54:41.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526910506 0 0) 0x7f4338009da0 con 0x7f43380090c0 2024-09-16T14:54:41.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 10 monclient(hunting): my global_id is 234323 2024-09-16T14:54:41.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4338006590 con 0x7f43380090c0 2024-09-16T14:54:41.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1529680292 0 0) 0x7f4338006590 con 0x7f43380090c0 2024-09-16T14:54:41.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4338009da0 con 0x7f43380090c0 2024-09-16T14:54:41.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4344004a10 con 0x7f43380090c0 2024-09-16T14:54:41.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:41.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 10 monclient: dump: 2024-09-16T14:54:41.636 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.637 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.637 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.637 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.637 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.637 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.637 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.637 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.637 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.637 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 311330000 0 0) 0x7f4338009da0 con 0x7f43380090c0 2024-09-16T14:54:41.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f433800c290 con 0x7f43380090c0 2024-09-16T14:54:41.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 4226966078) 0x7f4344004d40 con 0x7f43380090c0 2024-09-16T14:54:41.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:54:41.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:41.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 >> v1:172.21.15.26:6790/0 conn(0x7f43380090c0 legacy=0x7f4338009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.618+0000 7f4342ffd640 1 -- 172.21.15.26:0/3423221881 >> v1:172.21.15.26:6790/0 conn(0x7f43380090c0 legacy=0x7f4338009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:41.638 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:54:41.638 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:41.638 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:41.638 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:54:41.638 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 114, 2024-09-16T14:54:41.638 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:54:41.638 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:54:41.639 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:54:41.639 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:54:41.639 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:41.639 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:54:41.639 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:41.639 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:54:41.639 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:54:41.639 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:41.639 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:41.639 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:41.639 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:41.639 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:41.639 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:41.640 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:41.640 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:41.640 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:41.640 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:41.640 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:41.640 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:54:41.640 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:54:41.640 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:41.640 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:41.640 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:41.640 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:41.640 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:41.640 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:41.640 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:41.641 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:41.641 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:41.641 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:41.641 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.641 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:54:41.641 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:54:41.641 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:54:41.641 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:54:41.641 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:54:41.641 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:54:41.641 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:41.641 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:41.642 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:54:41.642 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:54:41.642 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:54:41.642 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:54:41.642 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:54:41.642 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:54:41.642 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:54:41.642 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:41.642 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:54:41.642 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:41.642 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:41.642 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:41.642 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:41.642 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:41.643 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:41.643 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:41.643 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:41.643 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:41.643 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:41.643 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:41.643 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:54:41.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.643 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:54:41.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.643 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:41.643 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:41.643 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:41.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:41.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.644 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:41.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:54:41.644 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:41.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.644 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:54:41.644 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:54:41.644 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:41.644 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:41.644 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:41.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.645 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:41.645 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:41.645 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:41.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:41.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:41.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:54:41.645 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:41.645 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.645 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:54:41.646 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:54:41.646 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:41.646 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:41.646 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:41.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.646 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:41.646 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:41.646 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:41.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:41.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.646 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:41.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:54:41.647 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:41.647 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.647 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:54:41.647 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:54:41.647 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:41.647 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:41.647 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:41.647 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.647 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.647 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:54:41.647 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:54:41.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.648 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:41.648 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:41.648 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:41.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.648 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:41.648 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:54:41.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.648 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:41.648 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:41.648 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:54:41.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.648 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:41.648 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:54:41.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.649 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:41.649 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:41.649 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:54:41.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.649 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:41.649 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:54:41.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.649 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:41.649 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:41.649 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:41.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.649 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.650 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:54:41.650 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:54:41.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.622+0000 7f43585fc640 1 -- 172.21.15.26:0/3423221881 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4334036470 legacy=0x7f4334038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.622+0000 7f43585fc640 10 monclient: shutdown 2024-09-16T14:54:41.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.622+0000 7f43585fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:41.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.622+0000 7f43585fc640 1 -- 172.21.15.26:0/3423221881 >> v1:172.21.15.26:6789/0 conn(0x7f435009b040 legacy=0x7f43501a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.622+0000 7f4356b99640 1 -- 172.21.15.26:0/3423221881 reap_dead start 2024-09-16T14:54:41.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.622+0000 7f4356b99640 1 -- 172.21.15.26:0/3423221881 reap_dead start 2024-09-16T14:54:41.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.622+0000 7f43585fc640 1 -- 172.21.15.26:0/3423221881 shutdown_connections 2024-09-16T14:54:41.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.622+0000 7f43585fc640 1 -- 172.21.15.26:0/3423221881 >> 172.21.15.26:0/3423221881 conn(0x7f435009fb50 msgr2=0x7f43500a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:41.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.622+0000 7f43585fc640 1 -- 172.21.15.26:0/3423221881 shutdown_connections 2024-09-16T14:54:41.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.622+0000 7f43585fc640 1 -- 172.21.15.26:0/3423221881 wait complete. 2024-09-16T14:54:41.652 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:54:41.652 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:54:41.652 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:54:41.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 10 monclient: get_monmap_and_config 2024-09-16T14:54:41.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 10 monclient: build_initial_monmap 2024-09-16T14:54:41.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 10 monclient: monmap: 2024-09-16T14:54:41.768 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:41.768 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.768 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:41.769821+0000 2024-09-16T14:54:41.768 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:41.769821+0000 2024-09-16T14:54:41.768 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:41.769 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.769 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:41.769 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:41.769 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:41.769 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 1 Processor -- start 2024-09-16T14:54:41.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 1 -- start start 2024-09-16T14:54:41.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 10 monclient: init 2024-09-16T14:54:41.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:41.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:54:41.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 10 monclient(hunting): picked mon.noname-a con 0x7fe71c09a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:41.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 10 monclient(hunting): picked mon.noname-b con 0x7fe71c09d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:41.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 10 monclient(hunting): picked mon.noname-c con 0x7fe71c09b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:41.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe71c09dc60 con 0x7fe71c09a550 2024-09-16T14:54:41.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe71c099ae0 con 0x7fe71c09b040 2024-09-16T14:54:41.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe71c09b8f0 con 0x7fe71c09d1b0 2024-09-16T14:54:41.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:41.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 10 monclient(hunting): authenticate will time out at 1705.889931s 2024-09-16T14:54:41.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe71affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe71c09a550 0x7fe71c09a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58464/0 (socket says 172.21.15.26:58464) 2024-09-16T14:54:41.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe71affd640 1 -- 172.21.15.26:0/3592481272 learned_addr learned my addr 172.21.15.26:0/3592481272 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:41.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 1 -- 172.21.15.26:0/3592481272 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249410907 0 0) 0x7fe71c09b8f0 con 0x7fe71c09d1b0 2024-09-16T14:54:41.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient(hunting): my global_id is 234326 2024-09-16T14:54:41.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 1 -- 172.21.15.26:0/3592481272 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe708005740 con 0x7fe71c09d1b0 2024-09-16T14:54:41.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 1 -- 172.21.15.26:0/3592481272 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138441505 0 0) 0x7fe71c09dc60 con 0x7fe71c09a550 2024-09-16T14:54:41.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient(hunting): my global_id is 254118 2024-09-16T14:54:41.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 1 -- 172.21.15.26:0/3592481272 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe71c09b8f0 con 0x7fe71c09a550 2024-09-16T14:54:41.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 1 -- 172.21.15.26:0/3592481272 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2987656771 0 0) 0x7fe71c09b8f0 con 0x7fe71c09a550 2024-09-16T14:54:41.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 1 -- 172.21.15.26:0/3592481272 >> v1:172.21.15.26:6790/0 conn(0x7fe71c09d1b0 legacy=0x7fe71c136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 1 -- 172.21.15.26:0/3592481272 >> v1:172.21.15.189:6789/0 conn(0x7fe71c09b040 legacy=0x7fe71c13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:54:41.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 1 monclient: found mon.noname-a 2024-09-16T14:54:41.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:41.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:54:41.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 1 -- 172.21.15.26:0/3592481272 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71c09bbd0 con 0x7fe71c09a550 2024-09-16T14:54:41.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient: _finish_auth 0 2024-09-16T14:54:41.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:54:41.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:41.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 1 -- 172.21.15.26:0/3592481272 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe70c001980 con 0x7fe71c09a550 2024-09-16T14:54:41.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 5 monclient: authenticate success, global_id 254118 2024-09-16T14:54:41.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:41.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:41.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient: dump: 2024-09-16T14:54:41.779 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.779 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.779 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.779 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.779 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.780 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.780 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.780 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.780 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.780 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:41.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 1 -- 172.21.15.26:0/3592481272 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe70c001ca0 con 0x7fe71c09a550 2024-09-16T14:54:41.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:41.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 1 -- 172.21.15.26:0/3592481272 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe70c002070 con 0x7fe71c09a550 2024-09-16T14:54:41.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:41.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe719ffb640 10 monclient: dump: 2024-09-16T14:54:41.780 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.781 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.781 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.781 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.781 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.781 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.781 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.781 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.781 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.781 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:41.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 4 set_mon_vals no callback set 2024-09-16T14:54:41.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 10 monclient: shutdown 2024-09-16T14:54:41.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 1 -- 172.21.15.26:0/3592481272 >> v1:172.21.15.26:6789/0 conn(0x7fe71c09a550 legacy=0x7fe71c09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 1 -- 172.21.15.26:0/3592481272 shutdown_connections 2024-09-16T14:54:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 1 -- 172.21.15.26:0/3592481272 >> 172.21.15.26:0/3592481272 conn(0x7fe71c09fb50 msgr2=0x7fe71c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 1 -- 172.21.15.26:0/3592481272 shutdown_connections 2024-09-16T14:54:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.766+0000 7fe721900640 1 -- 172.21.15.26:0/3592481272 wait complete. 2024-09-16T14:54:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 10 monclient: build_initial_monmap 2024-09-16T14:54:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 10 monclient: monmap: 2024-09-16T14:54:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:41.783 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 1 Processor -- start 2024-09-16T14:54:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 1 -- start start 2024-09-16T14:54:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 10 monclient: init 2024-09-16T14:54:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:54:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 10 monclient(hunting): picked mon.noname-b con 0x7fe71c09a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 10 monclient(hunting): picked mon.noname-c con 0x7fe71c09b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 10 monclient(hunting): picked mon.noname-a con 0x7fe71c09d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe71c1a1a30 con 0x7fe71c09d1b0 2024-09-16T14:54:41.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe71c1a17b0 con 0x7fe71c09a550 2024-09-16T14:54:41.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe71c0a2080 con 0x7fe71c09b040 2024-09-16T14:54:41.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:41.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 10 monclient(hunting): authenticate will time out at 1705.893218s 2024-09-16T14:54:41.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe71a7fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe71c09b040 0x7fe71c1c9f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60354/0 (socket says 172.21.15.26:60354) 2024-09-16T14:54:41.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe71a7fc640 1 -- 172.21.15.26:0/2748036792 learned_addr learned my addr 172.21.15.26:0/2748036792 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:41.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490933784 0 0) 0x7fe71c1a1a30 con 0x7fe71c09d1b0 2024-09-16T14:54:41.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): my global_id is 254121 2024-09-16T14:54:41.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6fc003530 con 0x7fe71c09d1b0 2024-09-16T14:54:41.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1649653037 0 0) 0x7fe71c0a2080 con 0x7fe71c09b040 2024-09-16T14:54:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): my global_id is 234329 2024-09-16T14:54:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe71c1a1a30 con 0x7fe71c09b040 2024-09-16T14:54:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3490129905 0 0) 0x7fe6fc003530 con 0x7fe71c09d1b0 2024-09-16T14:54:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe71c0a2080 con 0x7fe71c09d1b0 2024-09-16T14:54:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe714001840 con 0x7fe71c09d1b0 2024-09-16T14:54:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): dump: 2024-09-16T14:54:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.786 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1988315420 0 0) 0x7fe71c1a1a30 con 0x7fe71c09b040 2024-09-16T14:54:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6fc003530 con 0x7fe71c09b040 2024-09-16T14:54:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe704001860 con 0x7fe71c09b040 2024-09-16T14:54:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): dump: 2024-09-16T14:54:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.787 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636571270 0 0) 0x7fe71c1a17b0 con 0x7fe71c09a550 2024-09-16T14:54:41.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): my global_id is 244201 2024-09-16T14:54:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe71c1a1a30 con 0x7fe71c09a550 2024-09-16T14:54:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1638312372 0 0) 0x7fe71c0a2080 con 0x7fe71c09d1b0 2024-09-16T14:54:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 >> v1:172.21.15.26:6790/0 conn(0x7fe71c09b040 legacy=0x7fe71c1c9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 >> v1:172.21.15.189:6789/0 conn(0x7fe71c09a550 legacy=0x7fe71c1b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient: _finish_hunting 0 2024-09-16T14:54:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 monclient: found mon.a 2024-09-16T14:54:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71c1a2ee0 con 0x7fe71c09d1b0 2024-09-16T14:54:41.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient: _finish_auth 0 2024-09-16T14:54:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient: _check_auth_tickets 2024-09-16T14:54:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 5 monclient: authenticate success, global_id 254121 2024-09-16T14:54:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 10 monclient: _renew_subs 2024-09-16T14:54:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 1 -- 172.21.15.26:0/2748036792 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe71c1a0e60 con 0x7fe71c09d1b0 2024-09-16T14:54:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 10 monclient: _renew_subs 2024-09-16T14:54:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 1 -- 172.21.15.26:0/2748036792 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71c1b4f00 con 0x7fe71c09d1b0 2024-09-16T14:54:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe714001d30 con 0x7fe71c09d1b0 2024-09-16T14:54:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe714003820 con 0x7fe71c09d1b0 2024-09-16T14:54:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe7197fa640 4 set_mon_vals no callback set 2024-09-16T14:54:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient: dump: 2024-09-16T14:54:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.790 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe721900640 1 -- 172.21.15.26:0/2748036792 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe700006590 con 0x7fe700002830 2024-09-16T14:54:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe71400e290 con 0x7fe71c09d1b0 2024-09-16T14:54:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fe7140429b0 con 0x7fe71c09d1b0 2024-09-16T14:54:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe71a7fc640 10 monclient: get_auth_request con 0x7fe6fc036420 auth_method 0 2024-09-16T14:54:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291350795 0 0) 0x7fe700006590 con 0x7fe700002830 2024-09-16T14:54:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 10 monclient(hunting): my global_id is 244204 2024-09-16T14:54:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.770+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe71c0a2080 con 0x7fe700002830 2024-09-16T14:54:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.774+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2765604306 0 0) 0x7fe71c0a2080 con 0x7fe700002830 2024-09-16T14:54:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.774+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe700006590 con 0x7fe700002830 2024-09-16T14:54:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.774+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe70c001fb0 con 0x7fe700002830 2024-09-16T14:54:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.774+0000 7fe713fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.774+0000 7fe713fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:54:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.774+0000 7fe713fff640 10 monclient: dump: 2024-09-16T14:54:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.792 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.774+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1446068284 0 0) 0x7fe700006590 con 0x7fe700002830 2024-09-16T14:54:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.774+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe70000a0e0 con 0x7fe700002830 2024-09-16T14:54:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.774+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe70c001990 con 0x7fe700002830 2024-09-16T14:54:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.774+0000 7fe713fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.774+0000 7fe713fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.774+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 >> v1:172.21.15.189:6789/0 conn(0x7fe700002830 legacy=0x7fe700004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.774+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 >> v1:172.21.15.189:6789/0 conn(0x7fe700002830 legacy=0x7fe700004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.778+0000 7fe721900640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:54:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.778+0000 7fe721900640 1 -- 172.21.15.26:0/2748036792 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe700009da0 con 0x7fe7000090c0 2024-09-16T14:54:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751443573 0 0) 0x7fe700009da0 con 0x7fe7000090c0 2024-09-16T14:54:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 10 monclient(hunting): my global_id is 244207 2024-09-16T14:54:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe700006590 con 0x7fe7000090c0 2024-09-16T14:54:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2643920280 0 0) 0x7fe700006590 con 0x7fe7000090c0 2024-09-16T14:54:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe700009da0 con 0x7fe7000090c0 2024-09-16T14:54:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe70c005390 con 0x7fe7000090c0 2024-09-16T14:54:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:54:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 10 monclient: dump: 2024-09-16T14:54:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.797 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1960806651 0 0) 0x7fe700009da0 con 0x7fe7000090c0 2024-09-16T14:54:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe70000c290 con 0x7fe7000090c0 2024-09-16T14:54:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 535137947) 0x7fe70c006f30 con 0x7fe7000090c0 2024-09-16T14:54:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:54:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 >> v1:172.21.15.189:6789/0 conn(0x7fe7000090c0 legacy=0x7fe700009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe713fff640 1 -- 172.21.15.26:0/2748036792 >> v1:172.21.15.189:6789/0 conn(0x7fe7000090c0 legacy=0x7fe700009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:41.798 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:54:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:54:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 114, 2024-09-16T14:54:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:54:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:54:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:54:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:54:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:54:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:54:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:54:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:54:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:54:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:54:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:54:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:54:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:54:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:54:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:54:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:54:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:54:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:54:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:54:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:54:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:54:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:54:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:54:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:54:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:54:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:54:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:54:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:54:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:54:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:54:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:54:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:54:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:54:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:54:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:54:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:54:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:54:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:41.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:41.809 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:54:41.809 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:54:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe721900640 1 -- 172.21.15.26:0/2748036792 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe6fc036420 legacy=0x7fe6fc0388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe721900640 10 monclient: shutdown 2024-09-16T14:54:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe721900640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe721900640 1 -- 172.21.15.26:0/2748036792 >> v1:172.21.15.26:6789/0 conn(0x7fe71c09d1b0 legacy=0x7fe71c1cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe71b7fe640 1 -- 172.21.15.26:0/2748036792 reap_dead start 2024-09-16T14:54:41.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe71b7fe640 1 -- 172.21.15.26:0/2748036792 reap_dead start 2024-09-16T14:54:41.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe721900640 1 -- 172.21.15.26:0/2748036792 shutdown_connections 2024-09-16T14:54:41.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe721900640 1 -- 172.21.15.26:0/2748036792 >> 172.21.15.26:0/2748036792 conn(0x7fe71c09fb50 msgr2=0x7fe71c0a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:41.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe721900640 1 -- 172.21.15.26:0/2748036792 shutdown_connections 2024-09-16T14:54:41.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.782+0000 7fe721900640 1 -- 172.21.15.26:0/2748036792 wait complete. 2024-09-16T14:54:41.811 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:54:41.811 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:54:41.811 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:54:41.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea4b7f6640 10 monclient: get_monmap_and_config 2024-09-16T14:54:41.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea4b7f6640 10 monclient: build_initial_monmap 2024-09-16T14:54:41.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea4b7f6640 10 monclient: monmap: 2024-09-16T14:54:41.926 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:41.926 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.926 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:41.927255+0000 2024-09-16T14:54:41.926 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:41.927255+0000 2024-09-16T14:54:41.926 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:41.926 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.926 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:41.926 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:41.926 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:41.926 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea4b7f6640 1 Processor -- start 2024-09-16T14:54:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea4b7f6640 1 -- start start 2024-09-16T14:54:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea4b7f6640 10 monclient: init 2024-09-16T14:54:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea4b7f6640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea4b7f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:54:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea4b7f6640 10 monclient(hunting): picked mon.noname-c con 0x7fea4409a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea4b7f6640 10 monclient(hunting): picked mon.noname-b con 0x7fea4409d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea4b7f6640 10 monclient(hunting): picked mon.noname-a con 0x7fea4409b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea4b7f6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4409dc60 con 0x7fea4409b040 2024-09-16T14:54:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea4b7f6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea44099ae0 con 0x7fea4409a550 2024-09-16T14:54:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea4b7f6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4409b8f0 con 0x7fea4409d1b0 2024-09-16T14:54:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea4b7f6640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea4b7f6640 10 monclient(hunting): authenticate will time out at 1706.047701s 2024-09-16T14:54:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea48d91640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fea4409d1b0 0x7fea44136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60366/0 (socket says 172.21.15.26:60366) 2024-09-16T14:54:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea48d91640 1 -- 172.21.15.26:0/3444756675 learned_addr learned my addr 172.21.15.26:0/3444756675 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.922+0000 7fea49d93640 1 --1- 172.21.15.26:0/3444756675 >> v1:172.21.15.26:6789/0 conn(0x7fea4409b040 0x7fea4413a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58480/0 (socket says 172.21.15.26:58480) 2024-09-16T14:54:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 1 -- 172.21.15.26:0/3444756675 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258108036 0 0) 0x7fea4409b8f0 con 0x7fea4409d1b0 2024-09-16T14:54:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient(hunting): my global_id is 234332 2024-09-16T14:54:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 1 -- 172.21.15.26:0/3444756675 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea2c003530 con 0x7fea4409d1b0 2024-09-16T14:54:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 1 -- 172.21.15.26:0/3444756675 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265610557 0 0) 0x7fea4409dc60 con 0x7fea4409b040 2024-09-16T14:54:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient(hunting): my global_id is 254124 2024-09-16T14:54:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 1 -- 172.21.15.26:0/3444756675 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea4409b8f0 con 0x7fea4409b040 2024-09-16T14:54:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 1 -- 172.21.15.26:0/3444756675 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 983316903 0 0) 0x7fea4409b8f0 con 0x7fea4409b040 2024-09-16T14:54:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 1 -- 172.21.15.26:0/3444756675 >> v1:172.21.15.26:6790/0 conn(0x7fea4409d1b0 legacy=0x7fea44136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 1 -- 172.21.15.26:0/3444756675 >> v1:172.21.15.189:6789/0 conn(0x7fea4409a550 legacy=0x7fea4409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient: _finish_hunting 0 2024-09-16T14:54:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 1 monclient: found mon.noname-a 2024-09-16T14:54:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:54:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 1 -- 172.21.15.26:0/3444756675 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4409bbd0 con 0x7fea4409b040 2024-09-16T14:54:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient: _finish_auth 0 2024-09-16T14:54:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient: _check_auth_tickets 2024-09-16T14:54:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 5 monclient: authenticate success, global_id 254124 2024-09-16T14:54:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 1 -- 172.21.15.26:0/3444756675 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fea34001620 con 0x7fea4409b040 2024-09-16T14:54:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient: dump: 2024-09-16T14:54:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.935 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 1 -- 172.21.15.26:0/3444756675 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea34001840 con 0x7fea4409b040 2024-09-16T14:54:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 4 set_mon_vals no callback set 2024-09-16T14:54:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 1 -- 172.21.15.26:0/3444756675 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fea34001bd0 con 0x7fea4409b040 2024-09-16T14:54:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea43fff640 10 monclient: dump: 2024-09-16T14:54:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.936 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 10 monclient: shutdown 2024-09-16T14:54:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 1 -- 172.21.15.26:0/3444756675 >> v1:172.21.15.26:6789/0 conn(0x7fea4409b040 legacy=0x7fea4413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 1 -- 172.21.15.26:0/3444756675 shutdown_connections 2024-09-16T14:54:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 1 -- 172.21.15.26:0/3444756675 >> 172.21.15.26:0/3444756675 conn(0x7fea4409fb50 msgr2=0x7fea4409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 1 -- 172.21.15.26:0/3444756675 shutdown_connections 2024-09-16T14:54:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 1 -- 172.21.15.26:0/3444756675 wait complete. 2024-09-16T14:54:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 10 monclient: build_initial_monmap 2024-09-16T14:54:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 10 monclient: monmap: 2024-09-16T14:54:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:41.938 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:41.938 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:41.938 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 1 Processor -- start 2024-09-16T14:54:41.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 1 -- start start 2024-09-16T14:54:41.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 10 monclient: init 2024-09-16T14:54:41.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:41.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:54:41.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 10 monclient(hunting): picked mon.noname-c con 0x7fea4409a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:41.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 10 monclient(hunting): picked mon.noname-b con 0x7fea4409b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:41.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 10 monclient(hunting): picked mon.noname-a con 0x7fea4409d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4411b370 con 0x7fea4409d1b0 2024-09-16T14:54:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea44119b50 con 0x7fea4409b040 2024-09-16T14:54:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea441326f0 con 0x7fea4409a550 2024-09-16T14:54:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 10 monclient(hunting): authenticate will time out at 1706.050943s 2024-09-16T14:54:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea49592640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fea4409a550 0x7fea44139be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60368/0 (socket says 172.21.15.26:60368) 2024-09-16T14:54:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea49592640 1 -- 172.21.15.26:0/2163867041 learned_addr learned my addr 172.21.15.26:0/2163867041 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea49d93640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fea4409d1b0 0x7fea441cf710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58494/0 (socket says 172.21.15.26:58494) 2024-09-16T14:54:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5815289 0 0) 0x7fea44099ae0 con 0x7fea4409a550 2024-09-16T14:54:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): my global_id is 234335 2024-09-16T14:54:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea30003530 con 0x7fea4409a550 2024-09-16T14:54:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063313524 0 0) 0x7fea4411b370 con 0x7fea4409d1b0 2024-09-16T14:54:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): my global_id is 254127 2024-09-16T14:54:41.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea44099ae0 con 0x7fea4409d1b0 2024-09-16T14:54:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 398750782 0 0) 0x7fea30003530 con 0x7fea4409a550 2024-09-16T14:54:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea4411b370 con 0x7fea4409a550 2024-09-16T14:54:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fea38001760 con 0x7fea4409a550 2024-09-16T14:54:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): dump: 2024-09-16T14:54:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.946 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3185511502 0 0) 0x7fea44099ae0 con 0x7fea4409d1b0 2024-09-16T14:54:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea30003530 con 0x7fea4409d1b0 2024-09-16T14:54:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fea34001520 con 0x7fea4409d1b0 2024-09-16T14:54:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): dump: 2024-09-16T14:54:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.947 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859843483 0 0) 0x7fea44119b50 con 0x7fea4409b040 2024-09-16T14:54:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient(hunting): my global_id is 244213 2024-09-16T14:54:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea44099ae0 con 0x7fea4409b040 2024-09-16T14:54:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1242243159 0 0) 0x7fea4411b370 con 0x7fea4409a550 2024-09-16T14:54:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 >> v1:172.21.15.189:6789/0 conn(0x7fea4409b040 legacy=0x7fea4413a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 >> v1:172.21.15.26:6789/0 conn(0x7fea4409d1b0 legacy=0x7fea441cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:54:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 monclient: found mon.c 2024-09-16T14:54:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea44070a30 con 0x7fea4409a550 2024-09-16T14:54:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient: _finish_auth 0 2024-09-16T14:54:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:54:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 5 monclient: authenticate success, global_id 234335 2024-09-16T14:54:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 10 monclient: _renew_subs 2024-09-16T14:54:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 1 -- 172.21.15.26:0/2163867041 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4413aa00 con 0x7fea4409a550 2024-09-16T14:54:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 10 monclient: _renew_subs 2024-09-16T14:54:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.926+0000 7fea4b7f6640 1 -- 172.21.15.26:0/2163867041 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4411b930 con 0x7fea4409a550 2024-09-16T14:54:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.930+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea38003620 con 0x7fea4409a550 2024-09-16T14:54:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.930+0000 7fea41ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.930+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fea38001d90 con 0x7fea4409a550 2024-09-16T14:54:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.930+0000 7fea41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.930+0000 7fea41ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.930+0000 7fea42ffd640 4 set_mon_vals no callback set 2024-09-16T14:54:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.930+0000 7fea41ffb640 10 monclient: dump: 2024-09-16T14:54:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:41.950 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.930+0000 7fea4b7f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.930+0000 7fea4b7f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.930+0000 7fea4b7f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.930+0000 7fea4b7f6640 1 -- 172.21.15.26:0/2163867041 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea24005050 con 0x7fea4409a550 2024-09-16T14:54:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.930+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fea380039b0 con 0x7fea4409a550 2024-09-16T14:54:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.930+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fea38043c70 con 0x7fea4409a550 2024-09-16T14:54:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.930+0000 7fea48d91640 10 monclient: get_auth_request con 0x7fea30036490 auth_method 0 2024-09-16T14:54:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.934+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fea380041b0 con 0x7fea4409a550 2024-09-16T14:54:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.934+0000 7fea41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:41.934+0000 7fea41ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:42.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:42.150+0000 7fea4b7f6640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:54:42.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:42.150+0000 7fea4b7f6640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:54:42.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:42.150+0000 7fea4b7f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:42.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:42.150+0000 7fea4b7f6640 1 -- 172.21.15.26:0/2163867041 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fea24005b10 con 0x7fea4409a550 2024-09-16T14:54:42.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:42.150+0000 7fea41ffb640 1 -- 172.21.15.26:0/2163867041 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fea38012a40 con 0x7fea4409a550 2024-09-16T14:54:42.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:42.150+0000 7fea41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:54:42.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:42.150+0000 7fea41ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:42.156 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.154+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.156 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.154+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=1924304224,config=692619928,health=2943641109,kv=1867599010,logm=4084520271}) 2024-09-16T14:54:42.156 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.154+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=12,kv=10,logm=51} crc {auth=2304493549,config=692619927,health=1490502967,kv=3604373077,logm=1403746015}) 2024-09-16T14:54:42.156 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.154+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.156 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.154+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=1924304224,config=692619928,health=2943641109,kv=1867599010,logm=4084520271}) 2024-09-16T14:54:42.156 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.154+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=10,logm=50} crc {auth=923689228,config=692619927,health=345414046,kv=2084167376,logm=1962290789}) 2024-09-16T14:54:42.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:42.154+0000 7fea4b7f6640 1 -- 172.21.15.26:0/2163867041 >> v1:172.21.15.189:6812/3196161582 conn(0x7fea30036490 legacy=0x7fea30038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:42.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:42.154+0000 7fea4b7f6640 10 monclient: shutdown 2024-09-16T14:54:42.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:42.154+0000 7fea4b7f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:42.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:42.154+0000 7fea4b7f6640 1 -- 172.21.15.26:0/2163867041 >> v1:172.21.15.26:6790/0 conn(0x7fea4409a550 legacy=0x7fea44139be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:42.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:42.154+0000 7fea4b7f6640 1 -- 172.21.15.26:0/2163867041 shutdown_connections 2024-09-16T14:54:42.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:42.154+0000 7fea4b7f6640 1 -- 172.21.15.26:0/2163867041 >> 172.21.15.26:0/2163867041 conn(0x7fea4409fb50 msgr2=0x7fea4409e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:42.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:42.154+0000 7fea4b7f6640 1 -- 172.21.15.26:0/2163867041 shutdown_connections 2024-09-16T14:54:42.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:42.154+0000 7fea4b7f6640 1 -- 172.21.15.26:0/2163867041 wait complete. 2024-09-16T14:54:42.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.154+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.154+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2537413914}) 2024-09-16T14:54:42.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.154+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1167300718}) 2024-09-16T14:54:42.160 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.154+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.160 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.154+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2537413914}) 2024-09-16T14:54:42.160 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.154+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1772601533}) 2024-09-16T14:54:42.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.158+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.158+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=536812748}) 2024-09-16T14:54:42.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.158+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4221481853}) 2024-09-16T14:54:42.162 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.158+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.162 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.158+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=536812748}) 2024-09-16T14:54:42.162 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.158+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3901248763}) 2024-09-16T14:54:42.164 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.162+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.164 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.162+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1047262715}) 2024-09-16T14:54:42.164 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.162+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3259635296}) 2024-09-16T14:54:42.164 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.162+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.164 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.162+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1047262715}) 2024-09-16T14:54:42.164 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.162+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2720781224}) 2024-09-16T14:54:42.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.162+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.162+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3912632360}) 2024-09-16T14:54:42.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.162+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2567901500}) 2024-09-16T14:54:42.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.162+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.162+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3912632360}) 2024-09-16T14:54:42.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.162+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1411823803}) 2024-09-16T14:54:42.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.166+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.166+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4092256957}) 2024-09-16T14:54:42.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.166+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4106411049}) 2024-09-16T14:54:42.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.166+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.166+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4092256957}) 2024-09-16T14:54:42.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.166+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3776548531}) 2024-09-16T14:54:42.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.166+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.166+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1726436152}) 2024-09-16T14:54:42.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.166+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2500504017}) 2024-09-16T14:54:42.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.166+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.166+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1726436152}) 2024-09-16T14:54:42.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.166+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=390114758}) 2024-09-16T14:54:42.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.170+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.170+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1401920416}) 2024-09-16T14:54:42.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.170+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2969974574}) 2024-09-16T14:54:42.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.170+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.170+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1401920416}) 2024-09-16T14:54:42.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.170+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1233329711}) 2024-09-16T14:54:42.173 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:54:42.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.170+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.170+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1879051375}) 2024-09-16T14:54:42.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.170+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1808701959}) 2024-09-16T14:54:42.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.170+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.170+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1879051375}) 2024-09-16T14:54:42.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.170+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4011653038}) 2024-09-16T14:54:42.175 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.175 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3977937428}) 2024-09-16T14:54:42.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3666755778}) 2024-09-16T14:54:42.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3977937428}) 2024-09-16T14:54:42.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2195856818}) 2024-09-16T14:54:42.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1331063195}) 2024-09-16T14:54:42.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=23249329}) 2024-09-16T14:54:42.178 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.178 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1331063195}) 2024-09-16T14:54:42.178 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=661760935}) 2024-09-16T14:54:42.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=951054166}) 2024-09-16T14:54:42.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=298667550}) 2024-09-16T14:54:42.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=951054166}) 2024-09-16T14:54:42.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.174+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4292021073}) 2024-09-16T14:54:42.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.178+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.178+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309186074}) 2024-09-16T14:54:42.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.178+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200082300}) 2024-09-16T14:54:42.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.178+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.178+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309186074}) 2024-09-16T14:54:42.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.178+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2230366101}) 2024-09-16T14:54:42.182 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.178+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.178+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3534983084}) 2024-09-16T14:54:42.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.178+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159698082}) 2024-09-16T14:54:42.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.178+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.178+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3534983084}) 2024-09-16T14:54:42.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.178+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=340846569}) 2024-09-16T14:54:42.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.182+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.182+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2206206955}) 2024-09-16T14:54:42.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.182+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2773675090}) 2024-09-16T14:54:42.185 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.182+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.185 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.182+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2206206955}) 2024-09-16T14:54:42.185 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.182+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=233105815}) 2024-09-16T14:54:42.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.182+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.182+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2979884925}) 2024-09-16T14:54:42.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.182+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096524828}) 2024-09-16T14:54:42.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.182+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.182+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2979884925}) 2024-09-16T14:54:42.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.182+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2633286174}) 2024-09-16T14:54:42.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.186+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.186+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3341311187}) 2024-09-16T14:54:42.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.186+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=142807525}) 2024-09-16T14:54:42.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.186+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.189 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.186+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3341311187}) 2024-09-16T14:54:42.189 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.186+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=304191346}) 2024-09-16T14:54:42.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.186+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.186+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3525989697}) 2024-09-16T14:54:42.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.186+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448539233}) 2024-09-16T14:54:42.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.186+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.186+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3525989697}) 2024-09-16T14:54:42.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.186+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=70760229}) 2024-09-16T14:54:42.192 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.190+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.192 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.190+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=599868188}) 2024-09-16T14:54:42.192 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.190+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=960767587}) 2024-09-16T14:54:42.192 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.190+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.192 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.190+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=599868188}) 2024-09-16T14:54:42.192 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.190+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1121355619}) 2024-09-16T14:54:42.194 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.190+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.194 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.190+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2708003187}) 2024-09-16T14:54:42.194 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.190+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=374038065}) 2024-09-16T14:54:42.194 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.190+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.194 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.190+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2708003187}) 2024-09-16T14:54:42.194 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.190+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2861825702}) 2024-09-16T14:54:42.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2097138746}) 2024-09-16T14:54:42.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4148261207}) 2024-09-16T14:54:42.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2097138746}) 2024-09-16T14:54:42.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1521563748}) 2024-09-16T14:54:42.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4275798111}) 2024-09-16T14:54:42.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1698437959}) 2024-09-16T14:54:42.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4275798111}) 2024-09-16T14:54:42.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1677626036}) 2024-09-16T14:54:42.199 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.199 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2284529498}) 2024-09-16T14:54:42.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2872847070}) 2024-09-16T14:54:42.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.194+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2284529498}) 2024-09-16T14:54:42.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.198+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1886316839}) 2024-09-16T14:54:42.201 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.198+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.201 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.198+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=504149906}) 2024-09-16T14:54:42.201 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.198+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1330036633}) 2024-09-16T14:54:42.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.198+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.198+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=504149906}) 2024-09-16T14:54:42.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.198+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3227714497}) 2024-09-16T14:54:42.203 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.198+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.198+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3663908907}) 2024-09-16T14:54:42.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.198+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1187097330}) 2024-09-16T14:54:42.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.202+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.202+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3663908907}) 2024-09-16T14:54:42.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.202+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3207330422}) 2024-09-16T14:54:42.205 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.202+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.205 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.202+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=845069957}) 2024-09-16T14:54:42.205 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.202+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1210598043}) 2024-09-16T14:54:42.205 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.202+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.202+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=845069957}) 2024-09-16T14:54:42.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.202+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=736673849}) 2024-09-16T14:54:42.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.202+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.202+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3570836181}) 2024-09-16T14:54:42.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.202+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3730699092}) 2024-09-16T14:54:42.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.202+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.202+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3570836181}) 2024-09-16T14:54:42.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.202+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4243257998}) 2024-09-16T14:54:42.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.206+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.206+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1202227714}) 2024-09-16T14:54:42.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.206+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4167875222}) 2024-09-16T14:54:42.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.206+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.206+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1202227714}) 2024-09-16T14:54:42.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.206+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2369768502}) 2024-09-16T14:54:42.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.206+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.206+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3543055981}) 2024-09-16T14:54:42.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.206+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2647440420}) 2024-09-16T14:54:42.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.206+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.206+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3543055981}) 2024-09-16T14:54:42.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.206+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1530230263}) 2024-09-16T14:54:42.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.210+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.210+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3990216110}) 2024-09-16T14:54:42.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.210+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1392377208}) 2024-09-16T14:54:42.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.210+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.210+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3990216110}) 2024-09-16T14:54:42.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.210+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1386669843}) 2024-09-16T14:54:42.215 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.210+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.215 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.210+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1012493273}) 2024-09-16T14:54:42.215 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.210+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=57081915}) 2024-09-16T14:54:42.215 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.210+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.215 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.210+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1012493273}) 2024-09-16T14:54:42.215 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.210+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=576935426}) 2024-09-16T14:54:42.217 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.214+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.217 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.214+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1185681030}) 2024-09-16T14:54:42.217 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.214+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1453964942}) 2024-09-16T14:54:42.217 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.214+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.217 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.214+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1185681030}) 2024-09-16T14:54:42.217 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.214+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1433357563}) 2024-09-16T14:54:42.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.214+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.214+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=802791834}) 2024-09-16T14:54:42.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.214+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1834770374}) 2024-09-16T14:54:42.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.214+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.214+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=802791834}) 2024-09-16T14:54:42.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.214+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1709449569}) 2024-09-16T14:54:42.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.218+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.218+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=744240054}) 2024-09-16T14:54:42.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.218+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1937262836}) 2024-09-16T14:54:42.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.218+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.218+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=744240054}) 2024-09-16T14:54:42.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.218+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3044410915}) 2024-09-16T14:54:42.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.218+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.218+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3336599971}) 2024-09-16T14:54:42.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.218+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1962536509}) 2024-09-16T14:54:42.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.218+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.218+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3336599971}) 2024-09-16T14:54:42.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.218+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2005456976}) 2024-09-16T14:54:42.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.222+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.222+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3062400217}) 2024-09-16T14:54:42.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.222+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3079033133}) 2024-09-16T14:54:42.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.222+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.222+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3062400217}) 2024-09-16T14:54:42.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.222+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=911718898}) 2024-09-16T14:54:42.226 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.222+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.222+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2917053802}) 2024-09-16T14:54:42.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.222+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1795906210}) 2024-09-16T14:54:42.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.222+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.222+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2917053802}) 2024-09-16T14:54:42.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.222+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3866075057}) 2024-09-16T14:54:42.228 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.226+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.226+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=428578878}) 2024-09-16T14:54:42.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.226+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1886136017}) 2024-09-16T14:54:42.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.226+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.226+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=428578878}) 2024-09-16T14:54:42.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.226+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766098894}) 2024-09-16T14:54:42.230 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.226+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.231 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.226+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276313196}) 2024-09-16T14:54:42.231 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.226+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485959836}) 2024-09-16T14:54:42.231 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.226+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.231 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.226+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276313196}) 2024-09-16T14:54:42.231 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.226+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=677193013}) 2024-09-16T14:54:42.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.230+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.230+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=783598055}) 2024-09-16T14:54:42.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.230+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225789954}) 2024-09-16T14:54:42.233 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.230+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.233 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.230+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=783598055}) 2024-09-16T14:54:42.233 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.230+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3035515470}) 2024-09-16T14:54:42.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.230+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.230+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=860119138}) 2024-09-16T14:54:42.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.230+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2796010709}) 2024-09-16T14:54:42.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.230+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.235 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.230+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=860119138}) 2024-09-16T14:54:42.235 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.230+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3694597073}) 2024-09-16T14:54:42.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.234+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.234+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=243595870}) 2024-09-16T14:54:42.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.234+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687403892}) 2024-09-16T14:54:42.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.234+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.237 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.234+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=243595870}) 2024-09-16T14:54:42.237 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.234+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3141378701}) 2024-09-16T14:54:42.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.234+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.234+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1785846906}) 2024-09-16T14:54:42.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.234+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006753489}) 2024-09-16T14:54:42.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.234+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.234+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1785846906}) 2024-09-16T14:54:42.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.234+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2352334356}) 2024-09-16T14:54:42.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.238+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.238+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2140132487}) 2024-09-16T14:54:42.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.238+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2781317653}) 2024-09-16T14:54:42.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.238+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.238+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2140132487}) 2024-09-16T14:54:42.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.238+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3588745093}) 2024-09-16T14:54:42.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.238+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.238+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3703300117}) 2024-09-16T14:54:42.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.238+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143953253}) 2024-09-16T14:54:42.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.238+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.238+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3703300117}) 2024-09-16T14:54:42.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.238+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1408009941}) 2024-09-16T14:54:42.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.242+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.242+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1227790634}) 2024-09-16T14:54:42.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.242+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=265156415}) 2024-09-16T14:54:42.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.242+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.242+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1227790634}) 2024-09-16T14:54:42.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.242+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1649728576}) 2024-09-16T14:54:42.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.242+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.242+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=218119612}) 2024-09-16T14:54:42.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.242+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2966669146}) 2024-09-16T14:54:42.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.242+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.242+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=218119612}) 2024-09-16T14:54:42.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.242+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=27351497}) 2024-09-16T14:54:42.247 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.242+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.247 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.242+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1840099095}) 2024-09-16T14:54:42.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.242+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485640040}) 2024-09-16T14:54:42.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.246+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.246+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1840099095}) 2024-09-16T14:54:42.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.246+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1752369451}) 2024-09-16T14:54:42.249 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.246+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.250 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.246+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1501890565}) 2024-09-16T14:54:42.250 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.246+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384503854}) 2024-09-16T14:54:42.250 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.246+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.250 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.246+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1501890565}) 2024-09-16T14:54:42.250 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.246+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3358198369}) 2024-09-16T14:54:42.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.250+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.250+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1623656545}) 2024-09-16T14:54:42.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.250+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2126413897}) 2024-09-16T14:54:42.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.250+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.250+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1623656545}) 2024-09-16T14:54:42.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.250+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=24087291}) 2024-09-16T14:54:42.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.250+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.250+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4043790000}) 2024-09-16T14:54:42.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.250+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1866650758}) 2024-09-16T14:54:42.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.250+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.250+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4043790000}) 2024-09-16T14:54:42.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.250+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2481478286}) 2024-09-16T14:54:42.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.254+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.254+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4112265258}) 2024-09-16T14:54:42.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.254+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1588707870}) 2024-09-16T14:54:42.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.254+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.254+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4112265258}) 2024-09-16T14:54:42.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.254+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2940127202}) 2024-09-16T14:54:42.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.254+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.254+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1159710083}) 2024-09-16T14:54:42.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.254+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=314902120}) 2024-09-16T14:54:42.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.254+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.254+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1159710083}) 2024-09-16T14:54:42.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.254+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570053637}) 2024-09-16T14:54:42.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.258+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.258+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=34585323}) 2024-09-16T14:54:42.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.258+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3388617501}) 2024-09-16T14:54:42.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.258+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.258+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=34585323}) 2024-09-16T14:54:42.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.258+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3834869253}) 2024-09-16T14:54:42.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.258+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.258+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2366713756}) 2024-09-16T14:54:42.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.258+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991250953}) 2024-09-16T14:54:42.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.258+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.258+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2366713756}) 2024-09-16T14:54:42.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.258+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1361358744}) 2024-09-16T14:54:42.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.262+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.262+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2880544460}) 2024-09-16T14:54:42.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.262+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2430861152}) 2024-09-16T14:54:42.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.262+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.262+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2880544460}) 2024-09-16T14:54:42.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.262+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=39380752}) 2024-09-16T14:54:42.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.262+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.262+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=76488785}) 2024-09-16T14:54:42.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.262+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668787727}) 2024-09-16T14:54:42.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.262+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.262+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=76488785}) 2024-09-16T14:54:42.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.262+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=604612067}) 2024-09-16T14:54:42.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.266+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.266+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4011683829}) 2024-09-16T14:54:42.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.266+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3713175649}) 2024-09-16T14:54:42.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.266+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.266+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4011683829}) 2024-09-16T14:54:42.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.266+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1046324820}) 2024-09-16T14:54:42.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.266+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.266+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=124954616}) 2024-09-16T14:54:42.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.266+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2242158318}) 2024-09-16T14:54:42.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.266+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.271 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.266+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=124954616}) 2024-09-16T14:54:42.271 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.266+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=844236074}) 2024-09-16T14:54:42.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.270+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.270+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4279657656}) 2024-09-16T14:54:42.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.270+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2442005242}) 2024-09-16T14:54:42.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.270+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.270+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4279657656}) 2024-09-16T14:54:42.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.270+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833724003}) 2024-09-16T14:54:42.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.270+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.270+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3721945328}) 2024-09-16T14:54:42.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.270+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2123703240}) 2024-09-16T14:54:42.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.270+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.270+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3721945328}) 2024-09-16T14:54:42.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.270+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=127246508}) 2024-09-16T14:54:42.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.274+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.274+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3140364899}) 2024-09-16T14:54:42.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.274+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360007458}) 2024-09-16T14:54:42.277 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.274+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.277 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.274+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3140364899}) 2024-09-16T14:54:42.277 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.274+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1514094508}) 2024-09-16T14:54:42.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.274+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.274+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4173768223}) 2024-09-16T14:54:42.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.274+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2732685263}) 2024-09-16T14:54:42.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.274+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.274+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4173768223}) 2024-09-16T14:54:42.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.274+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1842271847}) 2024-09-16T14:54:42.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.278+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.278+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=229881623}) 2024-09-16T14:54:42.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.278+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780106587}) 2024-09-16T14:54:42.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.278+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.278+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=229881623}) 2024-09-16T14:54:42.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.278+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1214240028}) 2024-09-16T14:54:42.282 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.278+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.282 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.278+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1765504164}) 2024-09-16T14:54:42.282 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.278+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191101614}) 2024-09-16T14:54:42.282 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.278+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.282 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.278+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1765504164}) 2024-09-16T14:54:42.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.278+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4086010390}) 2024-09-16T14:54:42.284 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.282+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.282+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3119940352}) 2024-09-16T14:54:42.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.282+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860399621}) 2024-09-16T14:54:42.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.282+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.282+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3119940352}) 2024-09-16T14:54:42.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.282+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2611301817}) 2024-09-16T14:54:42.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.282+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.282+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1607490106}) 2024-09-16T14:54:42.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.282+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643567808}) 2024-09-16T14:54:42.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.282+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.282+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1607490106}) 2024-09-16T14:54:42.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.282+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3190634253}) 2024-09-16T14:54:42.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.286+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.286+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1464364499}) 2024-09-16T14:54:42.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.286+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986077427}) 2024-09-16T14:54:42.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.286+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.286+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1464364499}) 2024-09-16T14:54:42.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.286+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2187992699}) 2024-09-16T14:54:42.290 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.286+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.290 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.286+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4119198813}) 2024-09-16T14:54:42.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.286+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619200189}) 2024-09-16T14:54:42.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.286+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.286+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4119198813}) 2024-09-16T14:54:42.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.286+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3326214147}) 2024-09-16T14:54:42.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.290+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.290+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2823478618}) 2024-09-16T14:54:42.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.290+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1434372725}) 2024-09-16T14:54:42.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.290+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.290+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2823478618}) 2024-09-16T14:54:42.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.290+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=648633572}) 2024-09-16T14:54:42.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.290+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.290+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=392406763,mdsmap=2826658659}) 2024-09-16T14:54:42.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.290+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=2905339159,mdsmap=2826658659}) 2024-09-16T14:54:42.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.290+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.290+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=392406763,mdsmap=2826658659}) 2024-09-16T14:54:42.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.290+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2288823313}) 2024-09-16T14:54:42.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.306+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.306+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {mdsmap=3140514009,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3599058074,mon_config_key=2029941355,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3667003649}) 2024-09-16T14:54:42.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.306+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=1,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=49} crc {mdsmap=1228700967,mgr=2539502010,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3599058074,mon_config_key=4058182103,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=381282379}) 2024-09-16T14:54:42.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.306+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:54:42.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.306+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {mdsmap=3140514009,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3599058074,mon_config_key=2029941355,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3667003649}) 2024-09-16T14:54:42.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:54:42.306+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6,osd_snap=2,osdmap=52} crc {mdsmap=3140514009,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=1146712122,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_snap=3794734670,osdmap=2686151376}) 2024-09-16T14:54:43.174 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:54:43.175 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:54:43.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cff03640 10 monclient: get_monmap_and_config 2024-09-16T14:54:43.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cff03640 10 monclient: build_initial_monmap 2024-09-16T14:54:43.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cff03640 10 monclient: monmap: 2024-09-16T14:54:43.306 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:43.306 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.306 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:43.307232+0000 2024-09-16T14:54:43.306 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:43.307232+0000 2024-09-16T14:54:43.306 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:43.306 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.306 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:43.306 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:43.306 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:43.306 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cff03640 1 Processor -- start 2024-09-16T14:54:43.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cff03640 1 -- start start 2024-09-16T14:54:43.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cff03640 10 monclient: init 2024-09-16T14:54:43.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cff03640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:43.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cff03640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:54:43.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cff03640 10 monclient(hunting): picked mon.noname-c con 0x7ff0c80a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:43.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cff03640 10 monclient(hunting): picked mon.noname-a con 0x7ff0c80a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:43.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cff03640 10 monclient(hunting): picked mon.noname-b con 0x7ff0c80a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:43.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cff03640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c80a3250 con 0x7ff0c80a0000 2024-09-16T14:54:43.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cff03640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c809f600 con 0x7ff0c80a1d10 2024-09-16T14:54:43.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cff03640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c80a3830 con 0x7ff0c80a2800 2024-09-16T14:54:43.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cff03640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:43.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cff03640 10 monclient(hunting): authenticate will time out at 1707.427718s 2024-09-16T14:54:43.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cd49e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff0c80a0000 0x7ff0c8139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58496/0 (socket says 172.21.15.26:58496) 2024-09-16T14:54:43.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0cd49e640 1 -- 172.21.15.26:0/3601622425 learned_addr learned my addr 172.21.15.26:0/3601622425 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:43.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.302+0000 7ff0ce4a0640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff0c80a2800 0x7ff0c813d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60370/0 (socket says 172.21.15.26:60370) 2024-09-16T14:54:43.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 1 -- 172.21.15.26:0/3601622425 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766422925 0 0) 0x7ff0c80a3830 con 0x7ff0c80a2800 2024-09-16T14:54:43.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient(hunting): my global_id is 234338 2024-09-16T14:54:43.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 1 -- 172.21.15.26:0/3601622425 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0b4003530 con 0x7ff0c80a2800 2024-09-16T14:54:43.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 1 -- 172.21.15.26:0/3601622425 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729878037 0 0) 0x7ff0c80a3250 con 0x7ff0c80a0000 2024-09-16T14:54:43.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient(hunting): my global_id is 254130 2024-09-16T14:54:43.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 1 -- 172.21.15.26:0/3601622425 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c80a3830 con 0x7ff0c80a0000 2024-09-16T14:54:43.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 1 -- 172.21.15.26:0/3601622425 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3969926665 0 0) 0x7ff0b4003530 con 0x7ff0c80a2800 2024-09-16T14:54:43.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 1 -- 172.21.15.26:0/3601622425 >> v1:172.21.15.189:6789/0 conn(0x7ff0c80a1d10 legacy=0x7ff0c80a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 1 -- 172.21.15.26:0/3601622425 >> v1:172.21.15.26:6789/0 conn(0x7ff0c80a0000 legacy=0x7ff0c8139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient: _finish_hunting 0 2024-09-16T14:54:43.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 1 monclient: found mon.noname-b 2024-09-16T14:54:43.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:43.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:54:43.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 1 -- 172.21.15.26:0/3601622425 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c809ed60 con 0x7ff0c80a2800 2024-09-16T14:54:43.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient: _finish_auth 0 2024-09-16T14:54:43.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient: _check_auth_tickets 2024-09-16T14:54:43.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:43.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 5 monclient: authenticate success, global_id 234338 2024-09-16T14:54:43.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 1 -- 172.21.15.26:0/3601622425 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff0c00019a0 con 0x7ff0c80a2800 2024-09-16T14:54:43.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:43.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:54:43.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient: dump: 2024-09-16T14:54:43.313 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.313 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.313 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.313 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.313 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.314 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.314 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.314 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.314 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.314 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:43.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 1 -- 172.21.15.26:0/3601622425 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0c0001bc0 con 0x7ff0c80a2800 2024-09-16T14:54:43.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:43.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:43.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 4 set_mon_vals no callback set 2024-09-16T14:54:43.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 1 -- 172.21.15.26:0/3601622425 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff0c0001f90 con 0x7ff0c80a2800 2024-09-16T14:54:43.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:43.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0ccc9d640 10 monclient: dump: 2024-09-16T14:54:43.315 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.315 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.315 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.315 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.315 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.315 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.315 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.315 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.315 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.315 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 10 monclient: shutdown 2024-09-16T14:54:43.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:43.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 1 -- 172.21.15.26:0/3601622425 >> v1:172.21.15.26:6790/0 conn(0x7ff0c80a2800 legacy=0x7ff0c813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 1 -- 172.21.15.26:0/3601622425 shutdown_connections 2024-09-16T14:54:43.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 1 -- 172.21.15.26:0/3601622425 >> 172.21.15.26:0/3601622425 conn(0x7ff0c80a7f30 msgr2=0x7ff0c80a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:43.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 1 -- 172.21.15.26:0/3601622425 shutdown_connections 2024-09-16T14:54:43.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 1 -- 172.21.15.26:0/3601622425 wait complete. 2024-09-16T14:54:43.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 10 monclient: build_initial_monmap 2024-09-16T14:54:43.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 10 monclient: monmap: 2024-09-16T14:54:43.316 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:43.316 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:43.316 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:43.316 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:43.316 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:43.316 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:43.316 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:43.316 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:43.317 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:43.317 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 1 Processor -- start 2024-09-16T14:54:43.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 1 -- start start 2024-09-16T14:54:43.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 10 monclient: init 2024-09-16T14:54:43.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:43.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:54:43.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 10 monclient(hunting): picked mon.noname-a con 0x7ff0c80a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:43.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 10 monclient(hunting): picked mon.noname-b con 0x7ff0c80a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:43.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 10 monclient(hunting): picked mon.noname-c con 0x7ff0c80a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:43.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c81347f0 con 0x7ff0c80a0000 2024-09-16T14:54:43.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c8135fd0 con 0x7ff0c80a1d10 2024-09-16T14:54:43.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c8132b80 con 0x7ff0c80a2800 2024-09-16T14:54:43.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:43.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 10 monclient(hunting): authenticate will time out at 1707.430923s 2024-09-16T14:54:43.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cdc9f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff0c80a0000 0x7ff0c8131d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58498/0 (socket says 172.21.15.26:58498) 2024-09-16T14:54:43.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cdc9f640 1 -- 172.21.15.26:0/2524927066 learned_addr learned my addr 172.21.15.26:0/2524927066 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:43.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205617427 0 0) 0x7ff0c81347f0 con 0x7ff0c80a0000 2024-09-16T14:54:43.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): my global_id is 254133 2024-09-16T14:54:43.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0ac003560 con 0x7ff0c80a0000 2024-09-16T14:54:43.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865133179 0 0) 0x7ff0c8132b80 con 0x7ff0c80a2800 2024-09-16T14:54:43.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): my global_id is 234341 2024-09-16T14:54:43.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c81347f0 con 0x7ff0c80a2800 2024-09-16T14:54:43.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3160359737 0 0) 0x7ff0ac003560 con 0x7ff0c80a0000 2024-09-16T14:54:43.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0c8132b80 con 0x7ff0c80a0000 2024-09-16T14:54:43.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff0c4001980 con 0x7ff0c80a0000 2024-09-16T14:54:43.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:43.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): dump: 2024-09-16T14:54:43.323 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.324 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.324 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.324 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.324 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.324 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.324 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.324 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.324 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.324 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:43.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3474629997 0 0) 0x7ff0c81347f0 con 0x7ff0c80a2800 2024-09-16T14:54:43.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0ac003560 con 0x7ff0c80a2800 2024-09-16T14:54:43.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff0c0001d30 con 0x7ff0c80a2800 2024-09-16T14:54:43.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:43.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): dump: 2024-09-16T14:54:43.325 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.325 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.325 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.325 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.325 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.325 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.326 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.326 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.326 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.326 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890044945 0 0) 0x7ff0c8135fd0 con 0x7ff0c80a1d10 2024-09-16T14:54:43.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient(hunting): my global_id is 244219 2024-09-16T14:54:43.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c81347f0 con 0x7ff0c80a1d10 2024-09-16T14:54:43.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 346704504 0 0) 0x7ff0c8132b80 con 0x7ff0c80a0000 2024-09-16T14:54:43.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 >> v1:172.21.15.26:6790/0 conn(0x7ff0c80a2800 legacy=0x7ff0c81d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 >> v1:172.21.15.189:6789/0 conn(0x7ff0c80a1d10 legacy=0x7ff0c8132470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:54:43.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 monclient: found mon.a 2024-09-16T14:54:43.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:43.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:43.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c81345d0 con 0x7ff0c80a0000 2024-09-16T14:54:43.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient: _finish_auth 0 2024-09-16T14:54:43.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:54:43.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:43.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 5 monclient: authenticate success, global_id 254133 2024-09-16T14:54:43.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 10 monclient: _renew_subs 2024-09-16T14:54:43.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:43.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.306+0000 7ff0cff03640 1 -- 172.21.15.26:0/2524927066 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c80a6580 con 0x7ff0c80a0000 2024-09-16T14:54:43.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0cff03640 10 monclient: _renew_subs 2024-09-16T14:54:43.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0cff03640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:43.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0cff03640 1 -- 172.21.15.26:0/2524927066 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c8135cf0 con 0x7ff0c80a0000 2024-09-16T14:54:43.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0c4003660 con 0x7ff0c80a0000 2024-09-16T14:54:43.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0be7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:43.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff0c40039f0 con 0x7ff0c80a0000 2024-09-16T14:54:43.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:43.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0bffff640 4 set_mon_vals no callback set 2024-09-16T14:54:43.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0be7fc640 10 monclient: dump: 2024-09-16T14:54:43.329 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.329 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.329 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.329 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.329 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.329 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.329 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.329 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.329 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.330 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0cff03640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:43.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0cff03640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:43.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0cff03640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:43.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0cff03640 1 -- 172.21.15.26:0/2524927066 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b0005050 con 0x7ff0c80a0000 2024-09-16T14:54:43.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff0c4003bd0 con 0x7ff0c80a0000 2024-09-16T14:54:43.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7ff0c4043ea0 con 0x7ff0c80a0000 2024-09-16T14:54:43.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.310+0000 7ff0cd49e640 10 monclient: get_auth_request con 0x7ff0ac036450 auth_method 0 2024-09-16T14:54:43.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.314+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff0c4012c70 con 0x7ff0c80a0000 2024-09-16T14:54:43.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.314+0000 7ff0be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:43.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.314+0000 7ff0be7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:43.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.538+0000 7ff0cff03640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:54:43.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.538+0000 7ff0cff03640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.538+0000 7ff0cff03640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.538+0000 7ff0cff03640 1 -- 172.21.15.26:0/2524927066 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff0b0005880 con 0x7ff0c80a0000 2024-09-16T14:54:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.542+0000 7ff0be7fc640 1 -- 172.21.15.26:0/2524927066 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 3480288796) 0x7ff0c4012c70 con 0x7ff0c80a0000 2024-09-16T14:54:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.542+0000 7ff0be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.542+0000 7ff0be7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:43.544 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:54:43.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.546+0000 7ff0cff03640 1 -- 172.21.15.26:0/2524927066 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff0ac036450 legacy=0x7ff0ac038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.546+0000 7ff0cff03640 10 monclient: shutdown 2024-09-16T14:54:43.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.546+0000 7ff0cff03640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:43.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.546+0000 7ff0cff03640 1 -- 172.21.15.26:0/2524927066 >> v1:172.21.15.26:6789/0 conn(0x7ff0c80a0000 legacy=0x7ff0c8131d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.546+0000 7ff0cff03640 1 -- 172.21.15.26:0/2524927066 shutdown_connections 2024-09-16T14:54:43.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.546+0000 7ff0cff03640 1 -- 172.21.15.26:0/2524927066 >> 172.21.15.26:0/2524927066 conn(0x7ff0c80a7f30 msgr2=0x7ff0c806b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:43.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.546+0000 7ff0cff03640 1 -- 172.21.15.26:0/2524927066 shutdown_connections 2024-09-16T14:54:43.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.546+0000 7ff0cff03640 1 -- 172.21.15.26:0/2524927066 wait complete. 2024-09-16T14:54:43.565 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:54:43.566 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:54:43.566 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:54:43.566 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:54:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.678+0000 7f67fa490640 10 monclient: get_monmap_and_config 2024-09-16T14:54:43.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.678+0000 7f67fa490640 10 monclient: build_initial_monmap 2024-09-16T14:54:43.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 10 monclient: monmap: 2024-09-16T14:54:43.684 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:43.684 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.684 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:43.685513+0000 2024-09-16T14:54:43.684 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:43.685513+0000 2024-09-16T14:54:43.684 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:43.684 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.684 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:43.684 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:43.685 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:43.685 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 1 Processor -- start 2024-09-16T14:54:43.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 1 -- start start 2024-09-16T14:54:43.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 10 monclient: init 2024-09-16T14:54:43.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:43.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:54:43.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 10 monclient(hunting): picked mon.noname-a con 0x7f67f40a1ca0 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:43.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 10 monclient(hunting): picked mon.noname-b con 0x7f67f40a2790 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:43.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 10 monclient(hunting): picked mon.noname-c con 0x7f67f409ff50 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:43.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f409f520 con 0x7f67f40a1ca0 2024-09-16T14:54:43.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f40a0360 con 0x7f67f409ff50 2024-09-16T14:54:43.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f409e7a0 con 0x7f67f40a2790 2024-09-16T14:54:43.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:43.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 10 monclient(hunting): authenticate will time out at 1707.805684s 2024-09-16T14:54:43.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f37fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f67f40a2790 0x7f67f4139e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60388/0 (socket says 172.21.15.26:60388) 2024-09-16T14:54:43.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f3fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f67f40a1ca0 0x7f67f40a2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58502/0 (socket says 172.21.15.26:58502) 2024-09-16T14:54:43.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f37fe640 1 -- 172.21.15.26:0/290940325 learned_addr learned my addr 172.21.15.26:0/290940325 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:43.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 1 -- 172.21.15.26:0/290940325 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710651876 0 0) 0x7f67f409f520 con 0x7f67f40a1ca0 2024-09-16T14:54:43.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient(hunting): my global_id is 254136 2024-09-16T14:54:43.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 1 -- 172.21.15.26:0/290940325 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e0003530 con 0x7f67f40a1ca0 2024-09-16T14:54:43.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 1 -- 172.21.15.26:0/290940325 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454858234 0 0) 0x7f67f409e7a0 con 0x7f67f40a2790 2024-09-16T14:54:43.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient(hunting): my global_id is 234344 2024-09-16T14:54:43.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 1 -- 172.21.15.26:0/290940325 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f409f520 con 0x7f67f40a2790 2024-09-16T14:54:43.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 1 -- 172.21.15.26:0/290940325 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2834181944 0 0) 0x7f67e0003530 con 0x7f67f40a1ca0 2024-09-16T14:54:43.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 1 -- 172.21.15.26:0/290940325 >> v1:172.21.15.26:6790/0 conn(0x7f67f40a2790 legacy=0x7f67f4139e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 1 -- 172.21.15.26:0/290940325 >> v1:172.21.15.189:6789/0 conn(0x7f67f409ff50 legacy=0x7f67f413d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:54:43.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 1 monclient: found mon.noname-a 2024-09-16T14:54:43.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:43.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:54:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 1 -- 172.21.15.26:0/290940325 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f409ec80 con 0x7f67f40a1ca0 2024-09-16T14:54:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:54:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:54:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 1 -- 172.21.15.26:0/290940325 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67e4001620 con 0x7f67f40a1ca0 2024-09-16T14:54:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 5 monclient: authenticate success, global_id 254136 2024-09-16T14:54:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient: dump: 2024-09-16T14:54:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.697 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.697 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.697 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.697 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.697 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.697 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.697 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:43.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 1 -- 172.21.15.26:0/290940325 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67e4001900 con 0x7f67f40a1ca0 2024-09-16T14:54:43.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:43.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:43.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 4 set_mon_vals no callback set 2024-09-16T14:54:43.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 1 -- 172.21.15.26:0/290940325 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67e4001c90 con 0x7f67f40a1ca0 2024-09-16T14:54:43.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:43.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67f2ffd640 10 monclient: dump: 2024-09-16T14:54:43.698 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.698 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.698 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.698 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.698 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.698 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.698 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.698 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.699 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 10 monclient: shutdown 2024-09-16T14:54:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 1 -- 172.21.15.26:0/290940325 >> v1:172.21.15.26:6789/0 conn(0x7f67f40a1ca0 legacy=0x7f67f40a2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 1 -- 172.21.15.26:0/290940325 shutdown_connections 2024-09-16T14:54:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 1 -- 172.21.15.26:0/290940325 >> 172.21.15.26:0/290940325 conn(0x7f67f40a3da0 msgr2=0x7f67f40a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 1 -- 172.21.15.26:0/290940325 shutdown_connections 2024-09-16T14:54:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 1 -- 172.21.15.26:0/290940325 wait complete. 2024-09-16T14:54:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 10 monclient: build_initial_monmap 2024-09-16T14:54:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.682+0000 7f67fa490640 10 monclient: monmap: 2024-09-16T14:54:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:43.700 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:43.700 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:43.700 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:43.700 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:43.700 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:43.700 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:43.700 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:43.700 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 1 Processor -- start 2024-09-16T14:54:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 1 -- start start 2024-09-16T14:54:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 10 monclient: init 2024-09-16T14:54:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:54:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 10 monclient(hunting): picked mon.noname-b con 0x7f67f409ff50 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 10 monclient(hunting): picked mon.noname-c con 0x7f67f40a1ca0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 10 monclient(hunting): picked mon.noname-a con 0x7f67f40a2790 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f409e490 con 0x7f67f40a2790 2024-09-16T14:54:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f413df50 con 0x7f67f409ff50 2024-09-16T14:54:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f40a57f0 con 0x7f67f40a1ca0 2024-09-16T14:54:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 10 monclient(hunting): authenticate will time out at 1707.808752s 2024-09-16T14:54:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f8a2d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f67f40a2790 0x7f67f41d12f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58506/0 (socket says 172.21.15.26:58506) 2024-09-16T14:54:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f8a2d640 1 -- 172.21.15.26:0/3649635869 learned_addr learned my addr 172.21.15.26:0/3649635869 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f37fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f67f40a1ca0 0x7f67f41cdbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60394/0 (socket says 172.21.15.26:60394) 2024-09-16T14:54:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120397495 0 0) 0x7f67f409e490 con 0x7f67f40a2790 2024-09-16T14:54:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): my global_id is 254139 2024-09-16T14:54:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d4003530 con 0x7f67f40a2790 2024-09-16T14:54:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238050375 0 0) 0x7f67f40a57f0 con 0x7f67f40a1ca0 2024-09-16T14:54:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): my global_id is 234347 2024-09-16T14:54:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f409e490 con 0x7f67f40a1ca0 2024-09-16T14:54:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50342998 0 0) 0x7f67f413df50 con 0x7f67f409ff50 2024-09-16T14:54:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): my global_id is 244225 2024-09-16T14:54:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f40a57f0 con 0x7f67f409ff50 2024-09-16T14:54:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3793883661 0 0) 0x7f67d4003530 con 0x7f67f40a2790 2024-09-16T14:54:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67f413df50 con 0x7f67f40a2790 2024-09-16T14:54:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67ec001b80 con 0x7f67f40a2790 2024-09-16T14:54:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): dump: 2024-09-16T14:54:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.705 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1838805098 0 0) 0x7f67f409e490 con 0x7f67f40a1ca0 2024-09-16T14:54:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67d4003530 con 0x7f67f40a1ca0 2024-09-16T14:54:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67dc001840 con 0x7f67f40a1ca0 2024-09-16T14:54:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): dump: 2024-09-16T14:54:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.706 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4287763335 0 0) 0x7f67f413df50 con 0x7f67f40a2790 2024-09-16T14:54:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 >> v1:172.21.15.26:6790/0 conn(0x7f67f40a1ca0 legacy=0x7f67f41cdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 >> v1:172.21.15.189:6789/0 conn(0x7f67f409ff50 legacy=0x7f67f4134320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:54:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 monclient: found mon.a 2024-09-16T14:54:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f4132f70 con 0x7f67f40a2790 2024-09-16T14:54:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient: _finish_auth 0 2024-09-16T14:54:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:54:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 5 monclient: authenticate success, global_id 254139 2024-09-16T14:54:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 10 monclient: _renew_subs 2024-09-16T14:54:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 1 -- 172.21.15.26:0/3649635869 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f4133c20 con 0x7f67f40a2790 2024-09-16T14:54:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 10 monclient: _renew_subs 2024-09-16T14:54:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 1 -- 172.21.15.26:0/3649635869 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f4134a30 con 0x7f67f40a2790 2024-09-16T14:54:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67ec0020f0 con 0x7f67f40a2790 2024-09-16T14:54:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67ec003b40 con 0x7f67f40a2790 2024-09-16T14:54:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f27fc640 4 set_mon_vals no callback set 2024-09-16T14:54:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient: dump: 2024-09-16T14:54:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.709 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67fa490640 1 -- 172.21.15.26:0/3649635869 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f67d8006590 con 0x7f67d8002830 2024-09-16T14:54:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f67ec003ca0 con 0x7f67f40a2790 2024-09-16T14:54:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f67ec043f90 con 0x7f67f40a2790 2024-09-16T14:54:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3565493015 0 0) 0x7f67d8006590 con 0x7f67d8002830 2024-09-16T14:54:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient(hunting): my global_id is 254142 2024-09-16T14:54:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f413df50 con 0x7f67d8002830 2024-09-16T14:54:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2072824842 0 0) 0x7f67f413df50 con 0x7f67d8002830 2024-09-16T14:54:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67d8006590 con 0x7f67d8002830 2024-09-16T14:54:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67e40015e0 con 0x7f67d8002830 2024-09-16T14:54:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient: dump: 2024-09-16T14:54:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.711 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f37fe640 10 monclient: get_auth_request con 0x7f67d4036490 auth_method 0 2024-09-16T14:54:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2279302970 0 0) 0x7f67d8006590 con 0x7f67d8002830 2024-09-16T14:54:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f67d800a0e0 con 0x7f67d8002830 2024-09-16T14:54:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f67e4004890 con 0x7f67d8002830 2024-09-16T14:54:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 >> v1:172.21.15.26:6789/0 conn(0x7f67d8002830 legacy=0x7f67d8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.686+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 >> v1:172.21.15.26:6789/0 conn(0x7f67d8002830 legacy=0x7f67d8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67fa490640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:54:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67fa490640 1 -- 172.21.15.26:0/3649635869 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f67d8009da0 con 0x7f67d80090c0 2024-09-16T14:54:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237253845 0 0) 0x7f67d8009da0 con 0x7f67d80090c0 2024-09-16T14:54:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 10 monclient(hunting): my global_id is 254145 2024-09-16T14:54:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d8006590 con 0x7f67d80090c0 2024-09-16T14:54:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 934522633 0 0) 0x7f67d8006590 con 0x7f67d80090c0 2024-09-16T14:54:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67d8009da0 con 0x7f67d80090c0 2024-09-16T14:54:43.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67e4004d90 con 0x7f67d80090c0 2024-09-16T14:54:43.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:43.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 10 monclient: dump: 2024-09-16T14:54:43.715 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.715 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.715 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.715 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.715 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.715 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.715 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.715 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.715 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.716 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3570537710 0 0) 0x7f67d8009da0 con 0x7f67d80090c0 2024-09-16T14:54:43.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f67d800c290 con 0x7f67d80090c0 2024-09-16T14:54:43.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 1318099182) 0x7f67e4005030 con 0x7f67d80090c0 2024-09-16T14:54:43.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:54:43.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:43.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 >> v1:172.21.15.26:6789/0 conn(0x7f67d80090c0 legacy=0x7f67d8009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f0ff9640 1 -- 172.21.15.26:0/3649635869 >> v1:172.21.15.26:6789/0 conn(0x7f67d80090c0 legacy=0x7f67d8009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:43.716 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:54:43.716 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:43.717 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:43.717 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:54:43.717 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 114, 2024-09-16T14:54:43.717 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:54:43.717 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:54:43.717 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:54:43.717 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:54:43.717 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:43.717 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:54:43.717 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:43.717 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:54:43.717 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:54:43.717 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:43.718 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:43.718 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:43.718 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:43.718 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:43.718 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:43.718 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:43.718 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:43.718 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:43.718 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:43.718 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:43.718 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:54:43.718 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:54:43.718 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:43.719 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:43.719 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:43.719 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:43.719 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:43.719 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:43.719 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:43.719 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:43.719 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:43.719 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:43.719 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:43.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.719 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:54:43.719 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:54:43.719 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:54:43.720 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:54:43.720 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:54:43.720 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:54:43.720 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:43.720 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:43.720 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:54:43.721 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:54:43.721 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:54:43.721 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:54:43.721 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:54:43.721 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:54:43.721 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:54:43.721 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:43.721 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:54:43.721 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:43.721 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:43.721 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:43.721 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:43.722 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:43.722 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:43.722 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:43.722 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:43.722 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:43.722 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:43.722 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:43.722 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:54:43.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.722 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:54:43.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.723 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:43.723 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:43.723 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:43.723 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:43.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.723 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:43.723 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:54:43.723 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:43.723 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:43.723 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:43.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.723 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:54:43.723 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:54:43.724 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:43.724 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:43.724 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:43.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.724 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:43.724 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:43.724 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:43.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:43.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.724 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:43.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:54:43.724 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:43.725 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:43.725 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:43.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.725 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:54:43.725 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:54:43.725 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:43.725 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:43.725 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:43.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.725 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:43.726 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:43.726 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:43.726 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:43.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.726 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:43.726 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:54:43.726 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:43.726 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:43.726 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:43.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.727 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:54:43.727 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:54:43.727 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:43.727 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:43.727 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:43.727 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:43.727 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:43.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.727 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:54:43.727 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:54:43.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.727 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:43.727 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:43.728 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:43.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:43.728 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:43.728 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:54:43.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.728 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:43.728 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:43.728 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:54:43.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:43.728 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:43.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.728 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:54:43.728 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:54:43.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67fa490640 1 -- 172.21.15.26:0/3649635869 >> v1:172.21.15.189:6812/3196161582 conn(0x7f67d4036490 legacy=0x7f67d4038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67fa490640 10 monclient: shutdown 2024-09-16T14:54:43.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67fa490640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:43.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67fa490640 1 -- 172.21.15.26:0/3649635869 >> v1:172.21.15.26:6789/0 conn(0x7f67f40a2790 legacy=0x7f67f41d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f8a2d640 1 -- 172.21.15.26:0/3649635869 reap_dead start 2024-09-16T14:54:43.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67f8a2d640 1 -- 172.21.15.26:0/3649635869 reap_dead start 2024-09-16T14:54:43.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67fa490640 1 -- 172.21.15.26:0/3649635869 shutdown_connections 2024-09-16T14:54:43.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67fa490640 1 -- 172.21.15.26:0/3649635869 >> 172.21.15.26:0/3649635869 conn(0x7f67f40a3da0 msgr2=0x7f67f412eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:43.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67fa490640 1 -- 172.21.15.26:0/3649635869 shutdown_connections 2024-09-16T14:54:43.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.694+0000 7f67fa490640 1 -- 172.21.15.26:0/3649635869 wait complete. 2024-09-16T14:54:43.731 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:54:43.731 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:54:43.731 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:54:43.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 10 monclient: get_monmap_and_config 2024-09-16T14:54:43.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 10 monclient: build_initial_monmap 2024-09-16T14:54:43.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 10 monclient: monmap: 2024-09-16T14:54:43.852 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:43.852 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.853 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:43.853930+0000 2024-09-16T14:54:43.853 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:43.853930+0000 2024-09-16T14:54:43.853 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:43.853 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.853 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:43.853 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:43.853 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:43.853 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 1 Processor -- start 2024-09-16T14:54:43.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 1 -- start start 2024-09-16T14:54:43.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 10 monclient: init 2024-09-16T14:54:43.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:43.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:54:43.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 10 monclient(hunting): picked mon.noname-b con 0x7f39c8134e40 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:43.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 10 monclient(hunting): picked mon.noname-c con 0x7f39c8099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:43.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 10 monclient(hunting): picked mon.noname-a con 0x7f39c8135220 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:43.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c8099e80 con 0x7f39c8135220 2024-09-16T14:54:43.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c80719d0 con 0x7f39c8099a70 2024-09-16T14:54:43.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c809e1e0 con 0x7f39c8134e40 2024-09-16T14:54:43.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:43.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 10 monclient(hunting): authenticate will time out at 1707.974090s 2024-09-16T14:54:43.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c77fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f39c8135220 0x7f39c813db30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58524/0 (socket says 172.21.15.26:58524) 2024-09-16T14:54:43.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c77fe640 1 -- 172.21.15.26:0/4035309321 learned_addr learned my addr 172.21.15.26:0/4035309321 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:43.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 1 -- 172.21.15.26:0/4035309321 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115992076 0 0) 0x7f39c0001120 con 0x7f39c8135220 2024-09-16T14:54:43.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient(hunting): my global_id is 254148 2024-09-16T14:54:43.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 1 -- 172.21.15.26:0/4035309321 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b0003530 con 0x7f39c8135220 2024-09-16T14:54:43.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 1 -- 172.21.15.26:0/4035309321 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469550426 0 0) 0x7f39c809e1e0 con 0x7f39c8134e40 2024-09-16T14:54:43.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient(hunting): my global_id is 234350 2024-09-16T14:54:43.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 1 -- 172.21.15.26:0/4035309321 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c0001120 con 0x7f39c8134e40 2024-09-16T14:54:43.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 1 -- 172.21.15.26:0/4035309321 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1375962199 0 0) 0x7f39b0003530 con 0x7f39c8135220 2024-09-16T14:54:43.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 1 -- 172.21.15.26:0/4035309321 >> v1:172.21.15.26:6790/0 conn(0x7f39c8134e40 legacy=0x7f39c8070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 1 -- 172.21.15.26:0/4035309321 >> v1:172.21.15.189:6789/0 conn(0x7f39c8099a70 legacy=0x7f39c80712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:54:43.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 1 monclient: found mon.noname-a 2024-09-16T14:54:43.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:43.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:54:43.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 1 -- 172.21.15.26:0/4035309321 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c809e7d0 con 0x7f39c8135220 2024-09-16T14:54:43.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient: _finish_auth 0 2024-09-16T14:54:43.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:54:43.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:43.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 1 -- 172.21.15.26:0/4035309321 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f39c0001610 con 0x7f39c8135220 2024-09-16T14:54:43.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:43.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient: dump: 2024-09-16T14:54:43.864 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.864 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.864 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.864 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.864 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.864 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.864 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.864 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.864 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.865 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 5 monclient: authenticate success, global_id 254148 2024-09-16T14:54:43.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:43.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 1 -- 172.21.15.26:0/4035309321 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39c0001880 con 0x7f39c8135220 2024-09-16T14:54:43.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:43.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 1 -- 172.21.15.26:0/4035309321 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f39c0001c50 con 0x7f39c8135220 2024-09-16T14:54:43.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:43.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39c5ffb640 10 monclient: dump: 2024-09-16T14:54:43.865 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.865 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.865 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.865 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.865 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.866 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.866 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.866 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.866 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.866 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:43.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 4 set_mon_vals no callback set 2024-09-16T14:54:43.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 10 monclient: shutdown 2024-09-16T14:54:43.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:43.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 1 -- 172.21.15.26:0/4035309321 >> v1:172.21.15.26:6789/0 conn(0x7f39c8135220 legacy=0x7f39c813db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 1 -- 172.21.15.26:0/4035309321 shutdown_connections 2024-09-16T14:54:43.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 1 -- 172.21.15.26:0/4035309321 >> 172.21.15.26:0/4035309321 conn(0x7f39c809fb50 msgr2=0x7f39c809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:43.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 1 -- 172.21.15.26:0/4035309321 shutdown_connections 2024-09-16T14:54:43.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.850+0000 7f39cd9f0640 1 -- 172.21.15.26:0/4035309321 wait complete. 2024-09-16T14:54:43.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 10 monclient: build_initial_monmap 2024-09-16T14:54:43.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 10 monclient: monmap: 2024-09-16T14:54:43.867 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:43.867 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:43.867 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:43.867 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:43.867 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:43.867 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:43.867 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:43.867 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:43.867 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:43.867 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 1 Processor -- start 2024-09-16T14:54:43.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 1 -- start start 2024-09-16T14:54:43.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 10 monclient: init 2024-09-16T14:54:43.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:43.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:54:43.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 10 monclient(hunting): picked mon.noname-a con 0x7f39c8099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:43.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 10 monclient(hunting): picked mon.noname-c con 0x7f39c8134e40 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:43.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 10 monclient(hunting): picked mon.noname-b con 0x7f39c8135220 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:43.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c8122570 con 0x7f39c8099a70 2024-09-16T14:54:43.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c8124070 con 0x7f39c8135220 2024-09-16T14:54:43.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c8122bc0 con 0x7f39c8134e40 2024-09-16T14:54:43.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:43.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 10 monclient(hunting): authenticate will time out at 1707.977152s 2024-09-16T14:54:43.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39c67fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f39c8134e40 0x7f39c81ceb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60400/0 (socket says 172.21.15.26:60400) 2024-09-16T14:54:43.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39c67fc640 1 -- 172.21.15.26:0/299821195 learned_addr learned my addr 172.21.15.26:0/299821195 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:43.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535124957 0 0) 0x7f39c8122bc0 con 0x7f39c8134e40 2024-09-16T14:54:43.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): my global_id is 234353 2024-09-16T14:54:43.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b4003530 con 0x7f39c8134e40 2024-09-16T14:54:43.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436864900 0 0) 0x7f39c8122570 con 0x7f39c8099a70 2024-09-16T14:54:43.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): my global_id is 254151 2024-09-16T14:54:43.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c8122bc0 con 0x7f39c8099a70 2024-09-16T14:54:43.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4039649629 0 0) 0x7f39b4003530 con 0x7f39c8134e40 2024-09-16T14:54:43.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39c8122570 con 0x7f39c8134e40 2024-09-16T14:54:43.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f39bc001820 con 0x7f39c8134e40 2024-09-16T14:54:43.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:43.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): dump: 2024-09-16T14:54:43.870 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.870 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.870 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.870 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.871 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.871 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.871 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.871 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.871 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.871 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:43.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281087736 0 0) 0x7f39c8124070 con 0x7f39c8135220 2024-09-16T14:54:43.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): my global_id is 244231 2024-09-16T14:54:43.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b4003530 con 0x7f39c8135220 2024-09-16T14:54:43.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4215152678 0 0) 0x7f39c8122bc0 con 0x7f39c8099a70 2024-09-16T14:54:43.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39c8124070 con 0x7f39c8099a70 2024-09-16T14:54:43.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f39b8001bc0 con 0x7f39c8099a70 2024-09-16T14:54:43.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:43.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): dump: 2024-09-16T14:54:43.872 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.872 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.872 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.872 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.872 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.872 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.872 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.873 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.873 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.873 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 299430393 0 0) 0x7f39c8122570 con 0x7f39c8134e40 2024-09-16T14:54:43.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 >> v1:172.21.15.189:6789/0 conn(0x7f39c8135220 legacy=0x7f39c81d2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 >> v1:172.21.15.26:6789/0 conn(0x7f39c8099a70 legacy=0x7f39c81c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient: _finish_hunting 0 2024-09-16T14:54:43.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 monclient: found mon.c 2024-09-16T14:54:43.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:43.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:43.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c81340f0 con 0x7f39c8134e40 2024-09-16T14:54:43.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient: _finish_auth 0 2024-09-16T14:54:43.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient: _check_auth_tickets 2024-09-16T14:54:43.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:43.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 5 monclient: authenticate success, global_id 234353 2024-09-16T14:54:43.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 10 monclient: _renew_subs 2024-09-16T14:54:43.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:43.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 1 -- 172.21.15.26:0/299821195 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c809c210 con 0x7f39c8134e40 2024-09-16T14:54:43.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 10 monclient: _renew_subs 2024-09-16T14:54:43.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:43.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 1 -- 172.21.15.26:0/299821195 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c8123d90 con 0x7f39c8134e40 2024-09-16T14:54:43.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39bc003620 con 0x7f39c8134e40 2024-09-16T14:54:43.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:43.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f39bc0039b0 con 0x7f39c8134e40 2024-09-16T14:54:43.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39c57fa640 4 set_mon_vals no callback set 2024-09-16T14:54:43.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:43.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient: dump: 2024-09-16T14:54:43.875 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.875 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.875 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.875 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.875 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.875 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.875 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.875 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.875 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.875 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:43.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39cd9f0640 1 -- 172.21.15.26:0/299821195 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f39a8006590 con 0x7f39a8002830 2024-09-16T14:54:43.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f39bc003b10 con 0x7f39c8134e40 2024-09-16T14:54:43.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f39bc043c20 con 0x7f39c8134e40 2024-09-16T14:54:43.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024927855 0 0) 0x7f39a8006590 con 0x7f39a8002830 2024-09-16T14:54:43.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient(hunting): my global_id is 234356 2024-09-16T14:54:43.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c8122570 con 0x7f39a8002830 2024-09-16T14:54:43.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3122492861 0 0) 0x7f39c8122570 con 0x7f39a8002830 2024-09-16T14:54:43.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39a8006590 con 0x7f39a8002830 2024-09-16T14:54:43.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f39b8003980 con 0x7f39a8002830 2024-09-16T14:54:43.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:43.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 10 monclient: dump: 2024-09-16T14:54:43.877 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.877 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.877 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.877 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.877 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.877 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.877 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.877 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.877 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.877 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39c77fe640 10 monclient: get_auth_request con 0x7f39b4036490 auth_method 0 2024-09-16T14:54:43.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2959635719 0 0) 0x7f39a8006590 con 0x7f39a8002830 2024-09-16T14:54:43.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.854+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39a800a0e0 con 0x7f39a8002830 2024-09-16T14:54:43.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.858+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f39b8003d10 con 0x7f39a8002830 2024-09-16T14:54:43.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.858+0000 7f39affff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:43.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.858+0000 7f39affff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:43.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.858+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 >> v1:172.21.15.26:6790/0 conn(0x7f39a8002830 legacy=0x7f39a8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.858+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 >> v1:172.21.15.26:6790/0 conn(0x7f39a8002830 legacy=0x7f39a8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:43.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39cd9f0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:54:43.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39cd9f0640 1 -- 172.21.15.26:0/299821195 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f39a8009da0 con 0x7f39a80090c0 2024-09-16T14:54:43.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967742941 0 0) 0x7f39a8009da0 con 0x7f39a80090c0 2024-09-16T14:54:43.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:43.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:43.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:43.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 10 monclient(hunting): my global_id is 234359 2024-09-16T14:54:43.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a8006590 con 0x7f39a80090c0 2024-09-16T14:54:43.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1049707332 0 0) 0x7f39a8006590 con 0x7f39a80090c0 2024-09-16T14:54:43.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39a8009da0 con 0x7f39a80090c0 2024-09-16T14:54:43.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f39b80043a0 con 0x7f39a80090c0 2024-09-16T14:54:43.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:43.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:43.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 10 monclient: dump: 2024-09-16T14:54:43.880 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:43.880 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:43.880 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.880 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:43.880 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:43.880 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:43.881 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:43.881 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:43.881 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:43.881 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:43.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2916041481 0 0) 0x7f39a8009da0 con 0x7f39a80090c0 2024-09-16T14:54:43.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f39a800c290 con 0x7f39a80090c0 2024-09-16T14:54:43.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 4054246449) 0x7f39b8005700 con 0x7f39a80090c0 2024-09-16T14:54:43.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:54:43.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:43.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 >> v1:172.21.15.26:6790/0 conn(0x7f39a80090c0 legacy=0x7f39a8009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39affff640 1 -- 172.21.15.26:0/299821195 >> v1:172.21.15.26:6790/0 conn(0x7f39a80090c0 legacy=0x7f39a8009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:43.882 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:54:43.882 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:43.882 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:43.882 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:54:43.882 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 114, 2024-09-16T14:54:43.882 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:54:43.882 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:54:43.882 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:54:43.882 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:54:43.882 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:43.882 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:54:43.882 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:43.882 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:54:43.883 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:54:43.883 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:43.883 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:43.883 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:43.883 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:43.883 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:43.883 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:43.883 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:43.883 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:43.883 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:43.883 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:43.883 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:43.883 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:54:43.883 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:54:43.884 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:43.884 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:43.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:43.884 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:43.884 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:43.884 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:43.884 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:43.884 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:43.884 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:43.884 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:43.884 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:43.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.884 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:54:43.885 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:54:43.885 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:54:43.885 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:54:43.885 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:54:43.885 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:54:43.885 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:43.885 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:43.885 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:54:43.885 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:54:43.885 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:54:43.885 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:54:43.885 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:54:43.885 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:54:43.886 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:54:43.886 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:43.886 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:54:43.886 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:43.886 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:43.886 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:43.886 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:43.886 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:43.886 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:43.886 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:43.886 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:43.886 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:43.886 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:43.887 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:43.887 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:54:43.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:54:43.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.887 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:43.887 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:43.887 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:43.887 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:43.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.887 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:43.887 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:54:43.887 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:43.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:43.888 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:43.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:54:43.888 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:54:43.888 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:43.888 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:43.888 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:43.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.888 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:43.888 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:43.888 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:43.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:43.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:43.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:54:43.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:43.889 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:43.889 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:43.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:54:43.889 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:54:43.889 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:43.889 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:43.889 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:43.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.890 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:43.890 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:43.890 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:43.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:43.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.890 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:43.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:54:43.890 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:43.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:43.890 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:43.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:54:43.890 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:54:43.891 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:43.891 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:43.891 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:43.891 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:43.891 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:43.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.891 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:54:43.891 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:54:43.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.891 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:43.891 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:43.891 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:43.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:43.892 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:43.892 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:54:43.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.892 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:43.892 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:43.892 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:54:43.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:43.892 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:43.892 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:54:43.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.892 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:43.892 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:43.892 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 7 2024-09-16T14:54:43.893 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:43.893 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:43.893 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:54:43.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:43.893 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:43.893 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:43.893 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:43.893 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:43.893 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:43.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:43.893 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:54:43.893 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:54:43.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39cd9f0640 1 -- 172.21.15.26:0/299821195 >> v1:172.21.15.189:6812/3196161582 conn(0x7f39b4036490 legacy=0x7f39b4038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39cd9f0640 10 monclient: shutdown 2024-09-16T14:54:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39cd9f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39cd9f0640 1 -- 172.21.15.26:0/299821195 >> v1:172.21.15.26:6790/0 conn(0x7f39c8134e40 legacy=0x7f39c81ceb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39c77fe640 1 -- 172.21.15.26:0/299821195 reap_dead start 2024-09-16T14:54:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39c77fe640 1 -- 172.21.15.26:0/299821195 reap_dead start 2024-09-16T14:54:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39cd9f0640 1 -- 172.21.15.26:0/299821195 shutdown_connections 2024-09-16T14:54:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39cd9f0640 1 -- 172.21.15.26:0/299821195 >> 172.21.15.26:0/299821195 conn(0x7f39c809fb50 msgr2=0x7f39c809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39cd9f0640 1 -- 172.21.15.26:0/299821195 shutdown_connections 2024-09-16T14:54:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:43.862+0000 7f39cd9f0640 1 -- 172.21.15.26:0/299821195 wait complete. 2024-09-16T14:54:43.895 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:54:43.896 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:54:43.896 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:54:44.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.006+0000 7f11088be640 10 monclient: get_monmap_and_config 2024-09-16T14:54:44.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.006+0000 7f11088be640 10 monclient: build_initial_monmap 2024-09-16T14:54:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.006+0000 7f11088be640 10 monclient: monmap: 2024-09-16T14:54:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:44.012624+0000 2024-09-16T14:54:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:44.012624+0000 2024-09-16T14:54:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:44.012 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.006+0000 7f11088be640 1 Processor -- start 2024-09-16T14:54:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.006+0000 7f11088be640 1 -- start start 2024-09-16T14:54:44.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.006+0000 7f11088be640 10 monclient: init 2024-09-16T14:54:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:54:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient(hunting): picked mon.noname-b con 0x7f110409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient(hunting): picked mon.noname-c con 0x7f1104132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient(hunting): picked mon.noname-a con 0x7f1104138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110409d260 con 0x7f1104138ff0 2024-09-16T14:54:44.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1104132a80 con 0x7f1104132650 2024-09-16T14:54:44.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110409cf60 con 0x7f110409cb80 2024-09-16T14:54:44.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:44.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient(hunting): authenticate will time out at 1708.132378s 2024-09-16T14:54:44.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f1102d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1104138ff0 0x7f110413b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58536/0 (socket says 172.21.15.26:58536) 2024-09-16T14:54:44.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110259c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f110409cb80 0x7f11040717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60418/0 (socket says 172.21.15.26:60418) 2024-09-16T14:54:44.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f1102d9d640 1 -- 172.21.15.26:0/4267074654 learned_addr learned my addr 172.21.15.26:0/4267074654 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:44.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 1 -- 172.21.15.26:0/4267074654 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816063344 0 0) 0x7f110409cf60 con 0x7f110409cb80 2024-09-16T14:54:44.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient(hunting): _negotiate 2024-09-16T14:54:44.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:44.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:44.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient(hunting): my global_id is 234362 2024-09-16T14:54:44.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 1 -- 172.21.15.26:0/4267074654 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f0003530 con 0x7f110409cb80 2024-09-16T14:54:44.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 1 -- 172.21.15.26:0/4267074654 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017494770 0 0) 0x7f110409d260 con 0x7f1104138ff0 2024-09-16T14:54:44.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient(hunting): _negotiate 2024-09-16T14:54:44.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:44.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:44.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient(hunting): my global_id is 254154 2024-09-16T14:54:44.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 1 -- 172.21.15.26:0/4267074654 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f110409cf60 con 0x7f1104138ff0 2024-09-16T14:54:44.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 1 -- 172.21.15.26:0/4267074654 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1134973429 0 0) 0x7f10f0003530 con 0x7f110409cb80 2024-09-16T14:54:44.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 1 -- 172.21.15.26:0/4267074654 >> v1:172.21.15.189:6789/0 conn(0x7f1104132650 legacy=0x7f11041378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:44.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 1 -- 172.21.15.26:0/4267074654 >> v1:172.21.15.26:6789/0 conn(0x7f1104138ff0 legacy=0x7f110413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:44.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient: _finish_hunting 0 2024-09-16T14:54:44.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 1 monclient: found mon.noname-b 2024-09-16T14:54:44.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:44.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:54:44.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 1 -- 172.21.15.26:0/4267074654 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110409f2d0 con 0x7f110409cb80 2024-09-16T14:54:44.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient: _finish_auth 0 2024-09-16T14:54:44.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient: _check_auth_tickets 2024-09-16T14:54:44.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:44.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 1 -- 172.21.15.26:0/4267074654 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10f4001910 con 0x7f110409cb80 2024-09-16T14:54:44.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:44.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:54:44.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient: dump: 2024-09-16T14:54:44.022 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:44.022 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:44.022 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.022 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.022 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:44.023 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:44.023 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:44.023 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:44.023 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:44.023 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:44.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 5 monclient: authenticate success, global_id 234362 2024-09-16T14:54:44.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:44.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 1 -- 172.21.15.26:0/4267074654 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10f4001a70 con 0x7f110409cb80 2024-09-16T14:54:44.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:44.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 1 -- 172.21.15.26:0/4267074654 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10f4001e40 con 0x7f110409cb80 2024-09-16T14:54:44.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:44.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:44.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f110159a640 10 monclient: dump: 2024-09-16T14:54:44.023 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:44.024 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:44.024 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.024 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.024 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:44.024 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:44.024 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:44.024 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:44.024 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:44.024 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:44.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:44.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 4 set_mon_vals no callback set 2024-09-16T14:54:44.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient: shutdown 2024-09-16T14:54:44.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:44.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 1 -- 172.21.15.26:0/4267074654 >> v1:172.21.15.26:6790/0 conn(0x7f110409cb80 legacy=0x7f11040717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:44.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 1 -- 172.21.15.26:0/4267074654 shutdown_connections 2024-09-16T14:54:44.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 1 -- 172.21.15.26:0/4267074654 >> 172.21.15.26:0/4267074654 conn(0x7f110409fb50 msgr2=0x7f110409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:44.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 1 -- 172.21.15.26:0/4267074654 shutdown_connections 2024-09-16T14:54:44.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 1 -- 172.21.15.26:0/4267074654 wait complete. 2024-09-16T14:54:44.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient: build_initial_monmap 2024-09-16T14:54:44.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient: monmap: 2024-09-16T14:54:44.025 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:44.025 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:44.025 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:44.025 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:44.025 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:44.025 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:44.026 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:44.026 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:44.026 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:44.026 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:44.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 1 Processor -- start 2024-09-16T14:54:44.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 1 -- start start 2024-09-16T14:54:44.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient: init 2024-09-16T14:54:44.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:44.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:54:44.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient(hunting): picked mon.noname-c con 0x7f110409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:44.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient(hunting): picked mon.noname-b con 0x7f1104132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:44.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient(hunting): picked mon.noname-a con 0x7f1104138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:44.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110409a240 con 0x7f1104138ff0 2024-09-16T14:54:44.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110406e450 con 0x7f1104132650 2024-09-16T14:54:44.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110413c520 con 0x7f110409cb80 2024-09-16T14:54:44.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:44.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f11088be640 10 monclient(hunting): authenticate will time out at 1708.135707s 2024-09-16T14:54:44.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f1102d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1104138ff0 0x7f11041cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58552/0 (socket says 172.21.15.26:58552) 2024-09-16T14:54:44.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.010+0000 7f1102d9d640 1 -- 172.21.15.26:0/1596672652 learned_addr learned my addr 172.21.15.26:0/1596672652 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:44.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397609641 0 0) 0x7f110409a240 con 0x7f1104138ff0 2024-09-16T14:54:44.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:44.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:44.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:44.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): my global_id is 254157 2024-09-16T14:54:44.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e4003530 con 0x7f1104138ff0 2024-09-16T14:54:44.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123122699 0 0) 0x7f110413c520 con 0x7f110409cb80 2024-09-16T14:54:44.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:44.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:44.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:44.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): my global_id is 234365 2024-09-16T14:54:44.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f110409a240 con 0x7f110409cb80 2024-09-16T14:54:44.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3906840462 0 0) 0x7f10e4003530 con 0x7f1104138ff0 2024-09-16T14:54:44.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f110413c520 con 0x7f1104138ff0 2024-09-16T14:54:44.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10ec001660 con 0x7f1104138ff0 2024-09-16T14:54:44.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:44.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:44.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): dump: 2024-09-16T14:54:44.028 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:44.028 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:44.029 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.029 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.029 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:44.029 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:44.029 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:44.029 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:44.029 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:44.029 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:44.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:44.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3121192717 0 0) 0x7f110409a240 con 0x7f110409cb80 2024-09-16T14:54:44.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10e4003530 con 0x7f110409cb80 2024-09-16T14:54:44.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10f4001730 con 0x7f110409cb80 2024-09-16T14:54:44.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): dump: 2024-09-16T14:54:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:44.030 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063776283 0 0) 0x7f110406e450 con 0x7f1104132650 2024-09-16T14:54:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): my global_id is 244237 2024-09-16T14:54:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f110409a240 con 0x7f1104132650 2024-09-16T14:54:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3713152424 0 0) 0x7f110413c520 con 0x7f1104138ff0 2024-09-16T14:54:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 >> v1:172.21.15.26:6790/0 conn(0x7f110409cb80 legacy=0x7f110406d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 >> v1:172.21.15.189:6789/0 conn(0x7f1104132650 legacy=0x7f110406dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient: _finish_hunting 0 2024-09-16T14:54:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 monclient: found mon.a 2024-09-16T14:54:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11040715e0 con 0x7f1104138ff0 2024-09-16T14:54:44.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient: _finish_auth 0 2024-09-16T14:54:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient: _check_auth_tickets 2024-09-16T14:54:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f11088be640 5 monclient: authenticate success, global_id 254157 2024-09-16T14:54:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f11088be640 10 monclient: _renew_subs 2024-09-16T14:54:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f11088be640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f11088be640 1 -- 172.21.15.26:0/1596672652 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11041380c0 con 0x7f1104138ff0 2024-09-16T14:54:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f11088be640 10 monclient: _renew_subs 2024-09-16T14:54:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f11088be640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f11088be640 1 -- 172.21.15.26:0/1596672652 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f110406fbd0 con 0x7f1104138ff0 2024-09-16T14:54:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10ec0047f0 con 0x7f1104138ff0 2024-09-16T14:54:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10ec004b00 con 0x7f1104138ff0 2024-09-16T14:54:44.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:44.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:44.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f1100d99640 4 set_mon_vals no callback set 2024-09-16T14:54:44.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient: dump: 2024-09-16T14:54:44.033 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:44.033 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:44.033 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.033 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.033 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:44.033 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:44.033 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:44.033 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:44.034 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:44.034 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:44.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f11088be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:44.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f11088be640 1 -- 172.21.15.26:0/1596672652 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f10e8006590 con 0x7f10e8002830 2024-09-16T14:54:44.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f10ec00f570 con 0x7f1104138ff0 2024-09-16T14:54:44.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f10ec043c40 con 0x7f1104138ff0 2024-09-16T14:54:44.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123011313 0 0) 0x7f10e8006590 con 0x7f10e8002830 2024-09-16T14:54:44.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:44.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:44.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:44.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient(hunting): my global_id is 244240 2024-09-16T14:54:44.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f110413c520 con 0x7f10e8002830 2024-09-16T14:54:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f1101d9b640 10 monclient: get_auth_request con 0x7f10e4036440 auth_method 0 2024-09-16T14:54:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2654130346 0 0) 0x7f110413c520 con 0x7f10e8002830 2024-09-16T14:54:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10e8006590 con 0x7f10e8002830 2024-09-16T14:54:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10f4004f00 con 0x7f10e8002830 2024-09-16T14:54:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:54:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 10 monclient: dump: 2024-09-16T14:54:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:44.036 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:44.036 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:44.036 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:44.036 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:44.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3984020971 0 0) 0x7f10e8006590 con 0x7f10e8002830 2024-09-16T14:54:44.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.014+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10e800a0e0 con 0x7f10e8002830 2024-09-16T14:54:44.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.018+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f10f4005060 con 0x7f10e8002830 2024-09-16T14:54:44.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.018+0000 7f10faffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:44.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.018+0000 7f10faffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:44.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.018+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 >> v1:172.21.15.189:6789/0 conn(0x7f10e8002830 legacy=0x7f10e8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:44.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.018+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 >> v1:172.21.15.189:6789/0 conn(0x7f10e8002830 legacy=0x7f10e8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:44.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.022+0000 7f11088be640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:54:44.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.022+0000 7f11088be640 1 -- 172.21.15.26:0/1596672652 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f10e8009da0 con 0x7f10e80090c0 2024-09-16T14:54:44.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.022+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146375324 0 0) 0x7f10e8009da0 con 0x7f10e80090c0 2024-09-16T14:54:44.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.022+0000 7f10faffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:44.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.022+0000 7f10faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:44.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.022+0000 7f10faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:44.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.022+0000 7f10faffd640 10 monclient(hunting): my global_id is 244243 2024-09-16T14:54:44.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.022+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e8006590 con 0x7f10e80090c0 2024-09-16T14:54:44.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.022+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3378222281 0 0) 0x7f10e8006590 con 0x7f10e80090c0 2024-09-16T14:54:44.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.022+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10e8009da0 con 0x7f10e80090c0 2024-09-16T14:54:44.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.022+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10f4005630 con 0x7f10e80090c0 2024-09-16T14:54:44.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.022+0000 7f10faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:44.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.022+0000 7f10faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:54:44.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.022+0000 7f10faffd640 10 monclient: dump: 2024-09-16T14:54:44.039 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:44.039 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:44.039 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.039 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.039 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:44.040 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:44.040 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:44.040 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:44.040 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:44.040 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:44.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1736518453 0 0) 0x7f10e8009da0 con 0x7f10e80090c0 2024-09-16T14:54:44.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f10e800c290 con 0x7f10e80090c0 2024-09-16T14:54:44.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 2230404386) 0x7f10f40058a0 con 0x7f10e80090c0 2024-09-16T14:54:44.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f10faffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:54:44.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f10faffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:44.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 >> v1:172.21.15.189:6789/0 conn(0x7f10e80090c0 legacy=0x7f10e8009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:44.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f10faffd640 1 -- 172.21.15.26:0/1596672652 >> v1:172.21.15.189:6789/0 conn(0x7f10e80090c0 legacy=0x7f10e8009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:44.041 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:54:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:54:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 114, 2024-09-16T14:54:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:54:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:54:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:54:44.041 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:54:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:54:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:54:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:54:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:44.042 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:54:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:54:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:44.043 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:54:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:54:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:54:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:54:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:54:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:54:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:44.044 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:54:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:54:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:54:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:54:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:54:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:54:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:54:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:54:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:44.045 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:54:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:54:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:44.046 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:54:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:54:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:54:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:44.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:54:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:54:44.048 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:54:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:54:44.049 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:54:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:54:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:44.050 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:54:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:54:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:54:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:44.051 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:44.052 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:54:44.052 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:54:44.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f11088be640 1 -- 172.21.15.26:0/1596672652 >> v1:172.21.15.189:6812/3196161582 conn(0x7f10e4036440 legacy=0x7f10e4038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:44.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f11088be640 10 monclient: shutdown 2024-09-16T14:54:44.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f11088be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:44.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f11088be640 1 -- 172.21.15.26:0/1596672652 >> v1:172.21.15.26:6789/0 conn(0x7f1104138ff0 legacy=0x7f11041cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:44.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f1102d9d640 1 -- 172.21.15.26:0/1596672652 reap_dead start 2024-09-16T14:54:44.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f1102d9d640 1 -- 172.21.15.26:0/1596672652 reap_dead start 2024-09-16T14:54:44.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f11088be640 1 -- 172.21.15.26:0/1596672652 shutdown_connections 2024-09-16T14:54:44.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f11088be640 1 -- 172.21.15.26:0/1596672652 >> 172.21.15.26:0/1596672652 conn(0x7f110409fb50 msgr2=0x7f11040a1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:44.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f11088be640 1 -- 172.21.15.26:0/1596672652 shutdown_connections 2024-09-16T14:54:44.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.026+0000 7f11088be640 1 -- 172.21.15.26:0/1596672652 wait complete. 2024-09-16T14:54:44.054 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:54:44.055 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T14:54:44.055 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:54:44.055 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T14:54:44.055 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:54:44.055 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T14:54:44.055 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:54:44.079 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T14:54:44.079 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:54:44.079 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:54:44.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 10 monclient: get_monmap_and_config 2024-09-16T14:54:44.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 10 monclient: build_initial_monmap 2024-09-16T14:54:44.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 10 monclient: monmap: 2024-09-16T14:54:44.209 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:44.210 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:44.210 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:44.206899+0000 2024-09-16T14:54:44.210 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:44.206899+0000 2024-09-16T14:54:44.210 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:44.210 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:44.210 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:44.210 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:44.210 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:44.210 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:44.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 1 Processor -- start 2024-09-16T14:54:44.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 1 -- start start 2024-09-16T14:54:44.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 10 monclient: init 2024-09-16T14:54:44.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:44.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:54:44.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 10 monclient(hunting): picked mon.noname-c con 0x7f40240a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:44.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 10 monclient(hunting): picked mon.noname-b con 0x7f4024099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:44.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 10 monclient(hunting): picked mon.noname-a con 0x7f40240a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:44.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4024098ea0 con 0x7f40240a26f0 2024-09-16T14:54:44.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40240997b0 con 0x7f40240a1c00 2024-09-16T14:54:44.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4024099460 con 0x7f4024099b60 2024-09-16T14:54:44.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:44.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 10 monclient(hunting): authenticate will time out at 1708.326866s 2024-09-16T14:54:44.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f40237fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4024099b60 0x7f4024139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60436/0 (socket says 172.21.15.26:60436) 2024-09-16T14:54:44.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f40237fe640 1 -- 172.21.15.26:0/957178829 learned_addr learned my addr 172.21.15.26:0/957178829 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:44.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4023fff640 1 -- 172.21.15.26:0/957178829 >> v1:172.21.15.189:6789/0 conn(0x7f40240a1c00 legacy=0x7f40240a1fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:44.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 1 -- 172.21.15.26:0/957178829 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818755216 0 0) 0x7f4024099460 con 0x7f4024099b60 2024-09-16T14:54:44.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:44.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:44.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:44.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient(hunting): my global_id is 234368 2024-09-16T14:54:44.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 1 -- 172.21.15.26:0/957178829 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f400c003530 con 0x7f4024099b60 2024-09-16T14:54:44.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 1 -- 172.21.15.26:0/957178829 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736540795 0 0) 0x7f4024098ea0 con 0x7f40240a26f0 2024-09-16T14:54:44.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:44.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:44.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:44.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient(hunting): my global_id is 254160 2024-09-16T14:54:44.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 1 -- 172.21.15.26:0/957178829 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4024099460 con 0x7f40240a26f0 2024-09-16T14:54:44.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 1 -- 172.21.15.26:0/957178829 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2890143361 0 0) 0x7f4024099460 con 0x7f40240a26f0 2024-09-16T14:54:44.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 1 -- 172.21.15.26:0/957178829 >> v1:172.21.15.26:6790/0 conn(0x7f4024099b60 legacy=0x7f4024139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 1 -- 172.21.15.26:0/957178829 >> v1:172.21.15.189:6789/0 conn(0x7f40240a1c00 legacy=0x7f40240a1fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:54:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:54:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 1 monclient: found mon.noname-a 2024-09-16T14:54:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:54:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 1 -- 172.21.15.26:0/957178829 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40240a34b0 con 0x7f40240a26f0 2024-09-16T14:54:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient: _finish_auth 0 2024-09-16T14:54:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:54:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 1 -- 172.21.15.26:0/957178829 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f401c001980 con 0x7f40240a26f0 2024-09-16T14:54:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient: dump: 2024-09-16T14:54:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:44.226 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 5 monclient: authenticate success, global_id 254160 2024-09-16T14:54:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:44.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 1 -- 172.21.15.26:0/957178829 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f401c001c20 con 0x7f40240a26f0 2024-09-16T14:54:44.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:44.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 1 -- 172.21.15.26:0/957178829 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f401c001ff0 con 0x7f40240a26f0 2024-09-16T14:54:44.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:44.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:44.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f4022ffd640 10 monclient: dump: 2024-09-16T14:54:44.227 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:44.227 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:44.227 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.227 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:44.228 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.202+0000 7f402a76d640 4 set_mon_vals no callback set 2024-09-16T14:54:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient: shutdown 2024-09-16T14:54:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 1 -- 172.21.15.26:0/957178829 >> v1:172.21.15.26:6789/0 conn(0x7f40240a26f0 legacy=0x7f402413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 1 -- 172.21.15.26:0/957178829 shutdown_connections 2024-09-16T14:54:44.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 1 -- 172.21.15.26:0/957178829 >> 172.21.15.26:0/957178829 conn(0x7f40240a3d00 msgr2=0x7f40240a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:44.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 1 -- 172.21.15.26:0/957178829 shutdown_connections 2024-09-16T14:54:44.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 1 -- 172.21.15.26:0/957178829 wait complete. 2024-09-16T14:54:44.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient: build_initial_monmap 2024-09-16T14:54:44.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient: monmap: 2024-09-16T14:54:44.229 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:44.229 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:44.229 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:44.230 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:44.230 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:44.230 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:44.230 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:44.230 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:44.230 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:44.230 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:44.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 1 Processor -- start 2024-09-16T14:54:44.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 1 -- start start 2024-09-16T14:54:44.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient: init 2024-09-16T14:54:44.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:44.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:54:44.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient(hunting): picked mon.noname-c con 0x7f4024099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:44.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient(hunting): picked mon.noname-b con 0x7f40240a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:44.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient(hunting): picked mon.noname-a con 0x7f40240a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:44.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f402409ea50 con 0x7f40240a26f0 2024-09-16T14:54:44.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f402411fa50 con 0x7f40240a1c00 2024-09-16T14:54:44.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f402409efa0 con 0x7f4024099b60 2024-09-16T14:54:44.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:44.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient(hunting): authenticate will time out at 1708.329950s 2024-09-16T14:54:44.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4023fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4024099b60 0x7f40241380f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60444/0 (socket says 172.21.15.26:60444) 2024-09-16T14:54:44.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4023fff640 1 -- 172.21.15.26:0/2090787881 learned_addr learned my addr 172.21.15.26:0/2090787881 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:44.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4028d0a640 1 --1- 172.21.15.26:0/2090787881 >> v1:172.21.15.26:6789/0 conn(0x7f40240a26f0 0x7f40241d18b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58556/0 (socket says 172.21.15.26:58556) 2024-09-16T14:54:44.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f40237fe640 1 -- 172.21.15.26:0/2090787881 >> v1:172.21.15.189:6789/0 conn(0x7f40240a1c00 legacy=0x7f40241ce180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:44.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252520789 0 0) 0x7f402409efa0 con 0x7f4024099b60 2024-09-16T14:54:44.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:54:44.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:44.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:44.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient(hunting): my global_id is 234371 2024-09-16T14:54:44.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4010003530 con 0x7f4024099b60 2024-09-16T14:54:44.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243720551 0 0) 0x7f402409ea50 con 0x7f40240a26f0 2024-09-16T14:54:44.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:54:44.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient(hunting): my global_id is 254163 2024-09-16T14:54:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f402409efa0 con 0x7f40240a26f0 2024-09-16T14:54:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4117861002 0 0) 0x7f4010003530 con 0x7f4024099b60 2024-09-16T14:54:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f402409ea50 con 0x7f4024099b60 2024-09-16T14:54:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4014004190 con 0x7f4024099b60 2024-09-16T14:54:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient(hunting): dump: 2024-09-16T14:54:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:44.234 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2877367527 0 0) 0x7f402409efa0 con 0x7f40240a26f0 2024-09-16T14:54:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4010003530 con 0x7f40240a26f0 2024-09-16T14:54:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f401c001f30 con 0x7f40240a26f0 2024-09-16T14:54:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient(hunting): dump: 2024-09-16T14:54:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:44.236 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 957313730 0 0) 0x7f402409ea50 con 0x7f4024099b60 2024-09-16T14:54:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 >> v1:172.21.15.189:6789/0 conn(0x7f40240a1c00 legacy=0x7f40241ce180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:54:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 >> v1:172.21.15.26:6789/0 conn(0x7f40240a26f0 legacy=0x7f40241d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:54:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 monclient: found mon.c 2024-09-16T14:54:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402411f770 con 0x7f4024099b60 2024-09-16T14:54:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient: _finish_auth 0 2024-09-16T14:54:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:54:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 5 monclient: authenticate success, global_id 234371 2024-09-16T14:54:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient: _renew_subs 2024-09-16T14:54:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 1 -- 172.21.15.26:0/2090787881 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f402411e040 con 0x7f4024099b60 2024-09-16T14:54:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient: _renew_subs 2024-09-16T14:54:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 1 -- 172.21.15.26:0/2090787881 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40241c9cd0 con 0x7f4024099b60 2024-09-16T14:54:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4014004400 con 0x7f4024099b60 2024-09-16T14:54:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4014004790 con 0x7f4024099b60 2024-09-16T14:54:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f40227fc640 4 set_mon_vals no callback set 2024-09-16T14:54:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 10 monclient: dump: 2024-09-16T14:54:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:44.239 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f402a76d640 1 -- 172.21.15.26:0/2090787881 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4004005050 con 0x7f4024099b60 2024-09-16T14:54:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.206+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f40140049b0 con 0x7f4024099b60 2024-09-16T14:54:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.210+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f4014043e50 con 0x7f4024099b60 2024-09-16T14:54:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.210+0000 7f40237fe640 10 monclient: get_auth_request con 0x7f4010036220 auth_method 0 2024-09-16T14:54:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.214+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4014014790 con 0x7f4024099b60 2024-09-16T14:54:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.214+0000 7f4020ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.214+0000 7f4020ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:44.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.434+0000 7f402a76d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:54:44.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.434+0000 7f402a76d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:44.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.434+0000 7f402a76d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:44.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.434+0000 7f402a76d640 1 -- 172.21.15.26:0/2090787881 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4004005880 con 0x7f4024099b60 2024-09-16T14:54:44.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.438+0000 7f4020ff9640 1 -- 172.21.15.26:0/2090787881 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2004984099) 0x7f4014012c00 con 0x7f4024099b60 2024-09-16T14:54:44.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.438+0000 7f4020ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:44.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.438+0000 7f4020ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:44.441 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:54:44.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.438+0000 7f402a76d640 1 -- 172.21.15.26:0/2090787881 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4010036220 legacy=0x7f40100386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:44.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.438+0000 7f402a76d640 10 monclient: shutdown 2024-09-16T14:54:44.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.438+0000 7f402a76d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:44.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.438+0000 7f402a76d640 1 -- 172.21.15.26:0/2090787881 >> v1:172.21.15.26:6790/0 conn(0x7f4024099b60 legacy=0x7f40241380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:44.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.438+0000 7f402a76d640 1 -- 172.21.15.26:0/2090787881 shutdown_connections 2024-09-16T14:54:44.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.438+0000 7f402a76d640 1 -- 172.21.15.26:0/2090787881 >> 172.21.15.26:0/2090787881 conn(0x7f40240a3d00 msgr2=0x7f40240a0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:44.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.438+0000 7f402a76d640 1 -- 172.21.15.26:0/2090787881 shutdown_connections 2024-09-16T14:54:44.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:44.438+0000 7f402a76d640 1 -- 172.21.15.26:0/2090787881 wait complete. 2024-09-16T14:54:47.460 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:54:47.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb72a132640 10 monclient: get_monmap_and_config 2024-09-16T14:54:47.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb72a132640 10 monclient: build_initial_monmap 2024-09-16T14:54:47.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb72a132640 10 monclient: monmap: 2024-09-16T14:54:47.594 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:47.594 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:47.594 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:47.595418+0000 2024-09-16T14:54:47.594 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:47.595418+0000 2024-09-16T14:54:47.594 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:47.594 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:47.595 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:47.595 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:47.595 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:47.595 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:47.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb72a132640 1 Processor -- start 2024-09-16T14:54:47.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb72a132640 1 -- start start 2024-09-16T14:54:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb72a132640 10 monclient: init 2024-09-16T14:54:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb72a132640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb72a132640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:54:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb72a132640 10 monclient(hunting): picked mon.noname-c con 0x7fb72409a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb72a132640 10 monclient(hunting): picked mon.noname-b con 0x7fb72409d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb72a132640 10 monclient(hunting): picked mon.noname-a con 0x7fb72409b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb72a132640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72409dc60 con 0x7fb72409b040 2024-09-16T14:54:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb72a132640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb724099ae0 con 0x7fb72409a550 2024-09-16T14:54:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb72a132640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72409b8f0 con 0x7fb72409d1b0 2024-09-16T14:54:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb72a132640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb72a132640 10 monclient(hunting): authenticate will time out at 1711.715430s 2024-09-16T14:54:47.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb7237fe640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fb72409a550 legacy=0x7fb72409a930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb723fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb72409b040 0x7fb72413a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57310/0 (socket says 172.21.15.26:57310) 2024-09-16T14:54:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb723fff640 1 -- 172.21.15.26:0/2896683240 learned_addr learned my addr 172.21.15.26:0/2896683240 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.590+0000 7fb722ffd640 1 --1- 172.21.15.26:0/2896683240 >> v1:172.21.15.26:6790/0 conn(0x7fb72409d1b0 0x7fb724136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55568/0 (socket says 172.21.15.26:55568) 2024-09-16T14:54:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 1 -- 172.21.15.26:0/2896683240 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246346932 0 0) 0x7fb72409b8f0 con 0x7fb72409d1b0 2024-09-16T14:54:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 10 monclient(hunting): _negotiate 2024-09-16T14:54:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 10 monclient(hunting): my global_id is 234374 2024-09-16T14:54:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 1 -- 172.21.15.26:0/2896683240 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb70c003530 con 0x7fb72409d1b0 2024-09-16T14:54:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 1 -- 172.21.15.26:0/2896683240 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 111753696 0 0) 0x7fb70c003530 con 0x7fb72409d1b0 2024-09-16T14:54:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 1 -- 172.21.15.26:0/2896683240 >> v1:172.21.15.189:6789/0 conn(0x7fb72409a550 legacy=0x7fb72409a930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:54:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 1 -- 172.21.15.26:0/2896683240 >> v1:172.21.15.26:6789/0 conn(0x7fb72409b040 legacy=0x7fb72413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:47.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 10 monclient: _finish_hunting 0 2024-09-16T14:54:47.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 1 monclient: found mon.noname-b 2024-09-16T14:54:47.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:47.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:54:47.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 1 -- 172.21.15.26:0/2896683240 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72409bbd0 con 0x7fb72409d1b0 2024-09-16T14:54:47.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 10 monclient: _finish_auth 0 2024-09-16T14:54:47.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 10 monclient: _check_auth_tickets 2024-09-16T14:54:47.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:47.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 1 -- 172.21.15.26:0/2896683240 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb718001620 con 0x7fb72409d1b0 2024-09-16T14:54:47.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:47.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:54:47.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 10 monclient: dump: 2024-09-16T14:54:47.605 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:47.605 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:47.605 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:47.605 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:47.605 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:47.605 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:47.605 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:47.605 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:47.605 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:47.605 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:47.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 5 monclient: authenticate success, global_id 234374 2024-09-16T14:54:47.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:47.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 1 -- 172.21.15.26:0/2896683240 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb718001840 con 0x7fb72409d1b0 2024-09-16T14:54:47.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:47.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 1 -- 172.21.15.26:0/2896683240 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb718001c10 con 0x7fb72409d1b0 2024-09-16T14:54:47.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:47.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:47.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7227fc640 10 monclient: dump: 2024-09-16T14:54:47.606 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:47.606 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:47.606 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:47.606 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:47.606 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:47.606 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:47.607 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 4 set_mon_vals no callback set 2024-09-16T14:54:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 10 monclient: shutdown 2024-09-16T14:54:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 1 -- 172.21.15.26:0/2896683240 >> v1:172.21.15.26:6790/0 conn(0x7fb72409d1b0 legacy=0x7fb724136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 1 -- 172.21.15.26:0/2896683240 shutdown_connections 2024-09-16T14:54:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 1 -- 172.21.15.26:0/2896683240 >> 172.21.15.26:0/2896683240 conn(0x7fb72409fb50 msgr2=0x7fb72409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 1 -- 172.21.15.26:0/2896683240 shutdown_connections 2024-09-16T14:54:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 1 -- 172.21.15.26:0/2896683240 wait complete. 2024-09-16T14:54:47.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 10 monclient: build_initial_monmap 2024-09-16T14:54:47.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 10 monclient: monmap: 2024-09-16T14:54:47.608 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:47.608 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:47.608 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:47.608 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:47.608 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:47.608 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:47.608 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:47.608 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:47.608 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:47.608 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:47.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 1 Processor -- start 2024-09-16T14:54:47.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 1 -- start start 2024-09-16T14:54:47.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 10 monclient: init 2024-09-16T14:54:47.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:47.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:54:47.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 10 monclient(hunting): picked mon.noname-a con 0x7fb72409a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:47.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 10 monclient(hunting): picked mon.noname-b con 0x7fb72409b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:47.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 10 monclient(hunting): picked mon.noname-c con 0x7fb72409d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:47.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72409a240 con 0x7fb72409a550 2024-09-16T14:54:47.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7240a24c0 con 0x7fb72409b040 2024-09-16T14:54:47.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb724135ea0 con 0x7fb72409d1b0 2024-09-16T14:54:47.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:47.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 10 monclient(hunting): authenticate will time out at 1711.718679s 2024-09-16T14:54:47.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7237fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb72409a550 0x7fb724134250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57326/0 (socket says 172.21.15.26:57326) 2024-09-16T14:54:47.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb7237fe640 1 -- 172.21.15.26:0/4276545961 learned_addr learned my addr 172.21.15.26:0/4276545961 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:47.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb722ffd640 1 -- 172.21.15.26:0/4276545961 >> v1:172.21.15.189:6789/0 conn(0x7fb72409b040 legacy=0x7fb724134960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:47.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 1 -- 172.21.15.26:0/4276545961 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923609968 0 0) 0x7fb724135ea0 con 0x7fb72409d1b0 2024-09-16T14:54:47.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:47.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:47.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:47.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 10 monclient(hunting): my global_id is 234377 2024-09-16T14:54:47.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 1 -- 172.21.15.26:0/4276545961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb710003560 con 0x7fb72409d1b0 2024-09-16T14:54:47.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 1 -- 172.21.15.26:0/4276545961 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 868837044 0 0) 0x7fb710003560 con 0x7fb72409d1b0 2024-09-16T14:54:47.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 1 -- 172.21.15.26:0/4276545961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb724135ea0 con 0x7fb72409d1b0 2024-09-16T14:54:47.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 1 -- 172.21.15.26:0/4276545961 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb71c001af0 con 0x7fb72409d1b0 2024-09-16T14:54:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 10 monclient(hunting): dump: 2024-09-16T14:54:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:47.611 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:47.611 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:47.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:47.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 1 -- 172.21.15.26:0/4276545961 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3081338627 0 0) 0x7fb724135ea0 con 0x7fb72409d1b0 2024-09-16T14:54:47.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 1 -- 172.21.15.26:0/4276545961 >> v1:172.21.15.189:6789/0 conn(0x7fb72409b040 legacy=0x7fb724134960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:54:47.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 1 -- 172.21.15.26:0/4276545961 >> v1:172.21.15.26:6789/0 conn(0x7fb72409a550 legacy=0x7fb724134250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:47.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:54:47.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 1 monclient: found mon.c 2024-09-16T14:54:47.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:47.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:47.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 1 -- 172.21.15.26:0/4276545961 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7241367f0 con 0x7fb72409d1b0 2024-09-16T14:54:47.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 10 monclient: _finish_auth 0 2024-09-16T14:54:47.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:54:47.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:47.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 5 monclient: authenticate success, global_id 234377 2024-09-16T14:54:47.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 10 monclient: _renew_subs 2024-09-16T14:54:47.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:47.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 1 -- 172.21.15.26:0/4276545961 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb724137960 con 0x7fb72409d1b0 2024-09-16T14:54:47.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 10 monclient: _renew_subs 2024-09-16T14:54:47.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:47.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb72a132640 1 -- 172.21.15.26:0/4276545961 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb724136c90 con 0x7fb72409d1b0 2024-09-16T14:54:47.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 1 -- 172.21.15.26:0/4276545961 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb71c002020 con 0x7fb72409d1b0 2024-09-16T14:54:47.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:47.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 1 -- 172.21.15.26:0/4276545961 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb71c004d10 con 0x7fb72409d1b0 2024-09-16T14:54:47.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:47.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb721ffb640 4 set_mon_vals no callback set 2024-09-16T14:54:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.594+0000 7fb70bfff640 10 monclient: dump: 2024-09-16T14:54:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:47.614 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:47.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.598+0000 7fb70bfff640 1 -- 172.21.15.26:0/4276545961 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb71c004e70 con 0x7fb72409d1b0 2024-09-16T14:54:47.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.598+0000 7fb72a132640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:47.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.598+0000 7fb72a132640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:47.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.598+0000 7fb72a132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:47.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.598+0000 7fb72a132640 1 -- 172.21.15.26:0/4276545961 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb704005050 con 0x7fb72409d1b0 2024-09-16T14:54:47.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.598+0000 7fb70bfff640 1 -- 172.21.15.26:0/4276545961 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fb71c043f20 con 0x7fb72409d1b0 2024-09-16T14:54:47.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.598+0000 7fb7237fe640 10 monclient: get_auth_request con 0x7fb710036120 auth_method 0 2024-09-16T14:54:47.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.602+0000 7fb70bfff640 1 -- 172.21.15.26:0/4276545961 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb71c005640 con 0x7fb72409d1b0 2024-09-16T14:54:47.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.602+0000 7fb70bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:47.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.602+0000 7fb70bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:47.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.818+0000 7fb72a132640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:54:47.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.818+0000 7fb72a132640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:47.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.818+0000 7fb72a132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:47.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.818+0000 7fb72a132640 1 -- 172.21.15.26:0/4276545961 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb704005880 con 0x7fb72409d1b0 2024-09-16T14:54:47.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.818+0000 7fb70bfff640 1 -- 172.21.15.26:0/4276545961 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 661483266) 0x7fb71c012d80 con 0x7fb72409d1b0 2024-09-16T14:54:47.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.818+0000 7fb70bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:47.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.818+0000 7fb70bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:47.824 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:54:47.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.822+0000 7fb72a132640 1 -- 172.21.15.26:0/4276545961 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb710036120 legacy=0x7fb7100385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:47.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.822+0000 7fb72a132640 10 monclient: shutdown 2024-09-16T14:54:47.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.822+0000 7fb72a132640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:47.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.822+0000 7fb72a132640 1 -- 172.21.15.26:0/4276545961 >> v1:172.21.15.26:6790/0 conn(0x7fb72409d1b0 legacy=0x7fb724135070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:47.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.822+0000 7fb72a132640 1 -- 172.21.15.26:0/4276545961 shutdown_connections 2024-09-16T14:54:47.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.822+0000 7fb72a132640 1 -- 172.21.15.26:0/4276545961 >> 172.21.15.26:0/4276545961 conn(0x7fb72409fb50 msgr2=0x7fb724132a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:47.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.822+0000 7fb72a132640 1 -- 172.21.15.26:0/4276545961 shutdown_connections 2024-09-16T14:54:47.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:47.822+0000 7fb72a132640 1 -- 172.21.15.26:0/4276545961 wait complete. 2024-09-16T14:54:50.841 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:54:50.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd376b640 10 monclient: get_monmap_and_config 2024-09-16T14:54:50.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd376b640 10 monclient: build_initial_monmap 2024-09-16T14:54:50.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd376b640 10 monclient: monmap: 2024-09-16T14:54:50.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:50.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:50.974 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:50.974977+0000 2024-09-16T14:54:50.974 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:50.974977+0000 2024-09-16T14:54:50.974 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:50.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:50.974 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:50.974 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:50.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:50.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:50.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd376b640 1 Processor -- start 2024-09-16T14:54:50.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd376b640 1 -- start start 2024-09-16T14:54:50.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd376b640 10 monclient: init 2024-09-16T14:54:50.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd376b640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:50.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd376b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:54:50.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd376b640 10 monclient(hunting): picked mon.noname-b con 0x7fafcc134e40 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:50.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd376b640 10 monclient(hunting): picked mon.noname-a con 0x7fafcc099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:50.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd376b640 10 monclient(hunting): picked mon.noname-c con 0x7fafcc135220 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:50.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd376b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafcc099e80 con 0x7fafcc099a70 2024-09-16T14:54:50.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd376b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafcc0719d0 con 0x7fafcc135220 2024-09-16T14:54:50.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd376b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafcc09e1e0 con 0x7fafcc134e40 2024-09-16T14:54:50.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd376b640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:50.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd376b640 10 monclient(hunting): authenticate will time out at 1715.095272s 2024-09-16T14:54:50.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd1507640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fafcc134e40 0x7fafcc070bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55584/0 (socket says 172.21.15.26:55584) 2024-09-16T14:54:50.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd1507640 1 -- 172.21.15.26:0/2555753539 learned_addr learned my addr 172.21.15.26:0/2555753539 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:50.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:50.970+0000 7fafd1d08640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafcc13db30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:51.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:51.174+0000 7fafd1d08640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafcc13db30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:51.578+0000 7fafd1d08640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafcc13db30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:52.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:52.382+0000 7fafd1d08640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafcc13db30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:53.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:53.974+0000 7fafc2ffd640 10 monclient(hunting): tick 2024-09-16T14:54:53.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:53.974+0000 7fafc2ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:54:53.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:53.974+0000 7fafc2ffd640 1 monclient(hunting): continuing hunt 2024-09-16T14:54:53.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:53.974+0000 7fafc2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:54:53.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:53.974+0000 7fafc2ffd640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.26:6790/0 conn(0x7fafcc134e40 legacy=0x7fafcc070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:53.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:53.974+0000 7fafc2ffd640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafcc13db30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:54:53.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:53.974+0000 7fafc2ffd640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.26:6789/0 conn(0x7fafcc099a70 legacy=0x7fafcc0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:53.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:53.974+0000 7fafc2ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:54:53.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:53.974+0000 7fafc2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7fafb00036d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:53.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:53.974+0000 7fafc2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7fafb0007360 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:53.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:53.974+0000 7fafc2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7fafcc135220 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:53.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:53.974+0000 7fafc2ffd640 1 -- 172.21.15.26:0/2555753539 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafcc09e7d0 con 0x7fafb0007360 2024-09-16T14:54:53.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:53.974+0000 7fafc2ffd640 1 -- 172.21.15.26:0/2555753539 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafcc0719d0 con 0x7fafcc135220 2024-09-16T14:54:53.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:53.974+0000 7fafc2ffd640 1 -- 172.21.15.26:0/2555753539 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb00108b0 con 0x7fafb00036d0 2024-09-16T14:54:53.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:53.974+0000 7fafc2ffd640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:53.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:53.974+0000 7fafd1d08640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafb000d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:54.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:54.178+0000 7fafd1d08640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafb000d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:54.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:54.582+0000 7fafd1d08640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafb000d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:55.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:55.386+0000 7fafd1d08640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafb000d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:56.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:56.974+0000 7fafc2ffd640 10 monclient(hunting): tick 2024-09-16T14:54:56.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:56.974+0000 7fafc2ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:54:56.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:56.974+0000 7fafc2ffd640 1 monclient(hunting): continuing hunt 2024-09-16T14:54:56.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:56.974+0000 7fafc2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:54:56.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:56.974+0000 7fafc2ffd640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.26:6790/0 conn(0x7fafb00036d0 legacy=0x7fafb0005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:56.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:56.974+0000 7fafc2ffd640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafb000d0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:54:56.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:56.974+0000 7fafc2ffd640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.26:6789/0 conn(0x7fafb0007360 legacy=0x7fafb00097d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:56.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:56.974+0000 7fafc2ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:54:56.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:56.974+0000 7fafc2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7fafcc135220 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:56.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:56.974+0000 7fafc2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7fafb0010a40 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:56.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:56.974+0000 7fafc2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7fafb0007360 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:56.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:56.974+0000 7fafc2ffd640 1 -- 172.21.15.26:0/2555753539 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb0011b20 con 0x7fafb0007360 2024-09-16T14:54:56.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:56.974+0000 7fafc2ffd640 1 -- 172.21.15.26:0/2555753539 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafcc0719d0 con 0x7fafcc135220 2024-09-16T14:54:56.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:56.974+0000 7fafc2ffd640 1 -- 172.21.15.26:0/2555753539 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb0011730 con 0x7fafb0010a40 2024-09-16T14:54:56.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:56.974+0000 7fafc2ffd640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:56.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:56.974+0000 7fafd1507640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafb000d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:57.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:57.178+0000 7fafd1507640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafb000d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:57.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:57.582+0000 7fafd1507640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafb000d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:58.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.386+0000 7fafd1507640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafb000d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:58.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 1 -- 172.21.15.26:0/2555753539 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124018097 0 0) 0x7fafb0011b20 con 0x7fafb0007360 2024-09-16T14:54:58.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:54:58.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:58.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:58.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 10 monclient(hunting): my global_id is 254166 2024-09-16T14:54:58.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 1 -- 172.21.15.26:0/2555753539 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb8003530 con 0x7fafb0007360 2024-09-16T14:54:58.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 1 -- 172.21.15.26:0/2555753539 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 302152582 0 0) 0x7fafb8003530 con 0x7fafb0007360 2024-09-16T14:54:58.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.26:6790/0 conn(0x7fafb0010a40 legacy=0x7fafb0010e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:58.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafb000d0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:54:58.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:54:58.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 1 monclient: found mon.noname-a 2024-09-16T14:54:58.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:58.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:54:58.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 1 -- 172.21.15.26:0/2555753539 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb0005b30 con 0x7fafb0007360 2024-09-16T14:54:58.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 10 monclient: _finish_auth 0 2024-09-16T14:54:58.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:54:58.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:58.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 1 -- 172.21.15.26:0/2555753539 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fafc4004570 con 0x7fafb0007360 2024-09-16T14:54:58.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:58.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafc3fff640 10 monclient: dump: 2024-09-16T14:54:58.433 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.433 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.433 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.433 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.433 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.433 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.434 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.434 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.434 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.434 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafd376b640 5 monclient: authenticate success, global_id 254166 2024-09-16T14:54:58.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.426+0000 7fafd376b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:58.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafc3fff640 1 -- 172.21.15.26:0/2555753539 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafc40047d0 con 0x7fafb0007360 2024-09-16T14:54:58.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafc3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:58.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafc3fff640 1 -- 172.21.15.26:0/2555753539 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fafc4004ba0 con 0x7fafb0007360 2024-09-16T14:54:58.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:58.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 4 set_mon_vals no callback set 2024-09-16T14:54:58.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafc3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:58.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafc3fff640 10 monclient: dump: 2024-09-16T14:54:58.435 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.435 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.435 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.435 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.435 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.435 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.435 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.435 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.435 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.435 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 10 monclient: shutdown 2024-09-16T14:54:58.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:58.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 1 -- 172.21.15.26:0/2555753539 >> v1:172.21.15.26:6789/0 conn(0x7fafb0007360 legacy=0x7fafb00097d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:58.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 1 -- 172.21.15.26:0/2555753539 shutdown_connections 2024-09-16T14:54:58.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 1 -- 172.21.15.26:0/2555753539 >> 172.21.15.26:0/2555753539 conn(0x7fafcc09fb50 msgr2=0x7fafcc09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:58.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 1 -- 172.21.15.26:0/2555753539 shutdown_connections 2024-09-16T14:54:58.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 1 -- 172.21.15.26:0/2555753539 wait complete. 2024-09-16T14:54:58.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 10 monclient: build_initial_monmap 2024-09-16T14:54:58.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 10 monclient: monmap: 2024-09-16T14:54:58.439 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:58.439 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:58.439 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:58.439 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:58.439 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:58.440 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:58.440 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:58.440 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:58.440 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:58.440 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 1 Processor -- start 2024-09-16T14:54:58.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 1 -- start start 2024-09-16T14:54:58.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 10 monclient: init 2024-09-16T14:54:58.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:58.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:54:58.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 10 monclient(hunting): picked mon.noname-c con 0x7fafb0007360 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:58.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 10 monclient(hunting): picked mon.noname-a con 0x7fafb0010a40 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:58.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 10 monclient(hunting): picked mon.noname-b con 0x7fafcc135220 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:58.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafcc133c80 con 0x7fafb0010a40 2024-09-16T14:54:58.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafcc124310 con 0x7fafcc135220 2024-09-16T14:54:58.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafcc122730 con 0x7fafb0007360 2024-09-16T14:54:58.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:58.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.430+0000 7fafd376b640 10 monclient(hunting): authenticate will time out at 1722.555954s 2024-09-16T14:54:58.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafd1d08640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafcc1d22c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:58.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafd1507640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fafb0007360 0x7fafcc1c9930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59652/0 (socket says 172.21.15.26:59652) 2024-09-16T14:54:58.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafd1507640 1 -- 172.21.15.26:0/568589533 learned_addr learned my addr 172.21.15.26:0/568589533 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:58.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 1 -- 172.21.15.26:0/568589533 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1702465787 0 0) 0x7fafcc133c80 con 0x7fafb0010a40 2024-09-16T14:54:58.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:54:58.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:58.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:58.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 10 monclient(hunting): my global_id is 254169 2024-09-16T14:54:58.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 1 -- 172.21.15.26:0/568589533 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb00119f0 con 0x7fafb0010a40 2024-09-16T14:54:58.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 1 -- 172.21.15.26:0/568589533 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3512835486 0 0) 0x7fafb00119f0 con 0x7fafb0010a40 2024-09-16T14:54:58.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 1 -- 172.21.15.26:0/568589533 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafcc133c80 con 0x7fafb0010a40 2024-09-16T14:54:58.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 1 -- 172.21.15.26:0/568589533 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fafbc0013f0 con 0x7fafb0010a40 2024-09-16T14:54:58.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:58.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 10 monclient(hunting): dump: 2024-09-16T14:54:58.442 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.442 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.442 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.442 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.442 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.442 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.442 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.443 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.443 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.443 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:58.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 1 -- 172.21.15.26:0/568589533 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3703151273 0 0) 0x7fafcc133c80 con 0x7fafb0010a40 2024-09-16T14:54:58.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 1 -- 172.21.15.26:0/568589533 >> v1:172.21.15.26:6790/0 conn(0x7fafb0007360 legacy=0x7fafcc1c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:58.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 1 -- 172.21.15.26:0/568589533 >> v1:172.21.15.189:6789/0 conn(0x7fafcc135220 legacy=0x7fafcc1d22c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:54:58.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:54:58.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 1 monclient: found mon.a 2024-09-16T14:54:58.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:58.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:58.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 1 -- 172.21.15.26:0/568589533 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc0a2a50 con 0x7fafb0010a40 2024-09-16T14:54:58.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:54:58.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:54:58.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:58.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafd376b640 5 monclient: authenticate success, global_id 254169 2024-09-16T14:54:58.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafd376b640 10 monclient: _renew_subs 2024-09-16T14:54:58.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafd376b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:58.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafd376b640 1 -- 172.21.15.26:0/568589533 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafcc124030 con 0x7fafb0010a40 2024-09-16T14:54:58.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafd376b640 10 monclient: _renew_subs 2024-09-16T14:54:58.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafd376b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:58.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafd376b640 1 -- 172.21.15.26:0/568589533 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafcc121700 con 0x7fafb0010a40 2024-09-16T14:54:58.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 1 -- 172.21.15.26:0/568589533 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafbc001960 con 0x7fafb0010a40 2024-09-16T14:54:58.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:58.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 1 -- 172.21.15.26:0/568589533 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fafbc001c70 con 0x7fafb0010a40 2024-09-16T14:54:58.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc2ffd640 4 set_mon_vals no callback set 2024-09-16T14:54:58.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:58.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 10 monclient: dump: 2024-09-16T14:54:58.445 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.445 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.445 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.445 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.445 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.445 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.445 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.445 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.445 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.445 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 1 -- 172.21.15.26:0/568589533 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fafbc003620 con 0x7fafb0010a40 2024-09-16T14:54:58.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafd376b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:58.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafc1ffb640 1 -- 172.21.15.26:0/568589533 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fafbc042670 con 0x7fafb0010a40 2024-09-16T14:54:58.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafd376b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:58.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafd376b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:58.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafd376b640 1 -- 172.21.15.26:0/568589533 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafac005050 con 0x7fafb0010a40 2024-09-16T14:54:58.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.434+0000 7fafd1507640 10 monclient: get_auth_request con 0x7fafb00377b0 auth_method 0 2024-09-16T14:54:58.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.438+0000 7fafc1ffb640 1 -- 172.21.15.26:0/568589533 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fafbc012d20 con 0x7fafb0010a40 2024-09-16T14:54:58.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.438+0000 7fafc1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:58.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.438+0000 7fafc1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:58.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.670+0000 7fafd376b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:54:58.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.670+0000 7fafd376b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:58.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.670+0000 7fafd376b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:58.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.670+0000 7fafd376b640 1 -- 172.21.15.26:0/568589533 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fafac005880 con 0x7fafb0010a40 2024-09-16T14:54:58.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.670+0000 7fafc1ffb640 1 -- 172.21.15.26:0/568589533 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 2416123397) 0x7fafbc0113c0 con 0x7fafb0010a40 2024-09-16T14:54:58.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.670+0000 7fafc1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:54:58.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.670+0000 7fafc1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:58.676 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":118,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:54:58.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.674+0000 7fafd376b640 1 -- 172.21.15.26:0/568589533 >> v1:172.21.15.189:6812/3196161582 conn(0x7fafb00377b0 legacy=0x7fafb0039c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:58.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.674+0000 7fafd376b640 10 monclient: shutdown 2024-09-16T14:54:58.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.674+0000 7fafd376b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:58.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.674+0000 7fafd376b640 1 -- 172.21.15.26:0/568589533 >> v1:172.21.15.26:6789/0 conn(0x7fafb0010a40 legacy=0x7fafcc1ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:58.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.674+0000 7fafd376b640 1 -- 172.21.15.26:0/568589533 shutdown_connections 2024-09-16T14:54:58.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.674+0000 7fafd376b640 1 -- 172.21.15.26:0/568589533 >> 172.21.15.26:0/568589533 conn(0x7fafcc09fb50 msgr2=0x7fafcc09c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:58.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.674+0000 7fafd376b640 1 -- 172.21.15.26:0/568589533 shutdown_connections 2024-09-16T14:54:58.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.674+0000 7fafd376b640 1 -- 172.21.15.26:0/568589533 wait complete. 2024-09-16T14:54:58.695 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:54:58.695 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T14:54:58.695 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:54:58.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 10 monclient: get_monmap_and_config 2024-09-16T14:54:58.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 10 monclient: build_initial_monmap 2024-09-16T14:54:58.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 10 monclient: monmap: 2024-09-16T14:54:58.813 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:58.813 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.813 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:58.814559+0000 2024-09-16T14:54:58.813 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:58.814559+0000 2024-09-16T14:54:58.813 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:58.813 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.814 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:58.814 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:58.814 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:58.814 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 1 Processor -- start 2024-09-16T14:54:58.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 1 -- start start 2024-09-16T14:54:58.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 10 monclient: init 2024-09-16T14:54:58.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:58.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:54:58.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 10 monclient(hunting): picked mon.noname-a con 0x7fb47009cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:58.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 10 monclient(hunting): picked mon.noname-c con 0x7fb470132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:58.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 10 monclient(hunting): picked mon.noname-b con 0x7fb470138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:58.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47009d260 con 0x7fb47009cb80 2024-09-16T14:54:58.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb470132a80 con 0x7fb470132650 2024-09-16T14:54:58.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47009cf60 con 0x7fb470138ff0 2024-09-16T14:54:58.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:58.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 10 monclient(hunting): authenticate will time out at 1722.934600s 2024-09-16T14:54:58.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb47642d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb470138ff0 0x7fb47013b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59664/0 (socket says 172.21.15.26:59664) 2024-09-16T14:54:58.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb47642d640 1 -- 172.21.15.26:0/513117521 learned_addr learned my addr 172.21.15.26:0/513117521 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:58.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb47542b640 1 -- 172.21.15.26:0/513117521 >> v1:172.21.15.189:6789/0 conn(0x7fb470132650 legacy=0x7fb4701378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:58.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 1 -- 172.21.15.26:0/513117521 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575085979 0 0) 0x7fb47009d260 con 0x7fb47009cb80 2024-09-16T14:54:58.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient(hunting): _negotiate 2024-09-16T14:54:58.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:58.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:58.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient(hunting): my global_id is 254172 2024-09-16T14:54:58.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 1 -- 172.21.15.26:0/513117521 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb45c003530 con 0x7fb47009cb80 2024-09-16T14:54:58.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 1 -- 172.21.15.26:0/513117521 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558722383 0 0) 0x7fb47009cf60 con 0x7fb470138ff0 2024-09-16T14:54:58.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient(hunting): _negotiate 2024-09-16T14:54:58.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:58.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:58.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient(hunting): my global_id is 234380 2024-09-16T14:54:58.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 1 -- 172.21.15.26:0/513117521 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb47009d260 con 0x7fb470138ff0 2024-09-16T14:54:58.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 1 -- 172.21.15.26:0/513117521 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3369393583 0 0) 0x7fb45c003530 con 0x7fb47009cb80 2024-09-16T14:54:58.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 1 -- 172.21.15.26:0/513117521 >> v1:172.21.15.26:6790/0 conn(0x7fb470138ff0 legacy=0x7fb47013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:58.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 1 -- 172.21.15.26:0/513117521 >> v1:172.21.15.189:6789/0 conn(0x7fb470132650 legacy=0x7fb4701378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:54:58.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient: _finish_hunting 0 2024-09-16T14:54:58.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 1 monclient: found mon.noname-a 2024-09-16T14:54:58.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:58.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:54:58.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 1 -- 172.21.15.26:0/513117521 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47009f2d0 con 0x7fb47009cb80 2024-09-16T14:54:58.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient: _finish_auth 0 2024-09-16T14:54:58.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient: _check_auth_tickets 2024-09-16T14:54:58.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:58.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 1 -- 172.21.15.26:0/513117521 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb460001620 con 0x7fb47009cb80 2024-09-16T14:54:58.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:58.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient: dump: 2024-09-16T14:54:58.824 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.824 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.824 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.824 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.824 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.824 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.824 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.824 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.824 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.825 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 5 monclient: authenticate success, global_id 254172 2024-09-16T14:54:58.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:58.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 1 -- 172.21.15.26:0/513117521 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb460001890 con 0x7fb47009cb80 2024-09-16T14:54:58.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:58.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 1 -- 172.21.15.26:0/513117521 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb460001c20 con 0x7fb47009cb80 2024-09-16T14:54:58.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:58.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb474c2a640 10 monclient: dump: 2024-09-16T14:54:58.825 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.825 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.825 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.825 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.826 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.826 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.826 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.826 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.826 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.826 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:58.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.810+0000 7fb477e90640 4 set_mon_vals no callback set 2024-09-16T14:54:58.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 10 monclient: shutdown 2024-09-16T14:54:58.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:58.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 1 -- 172.21.15.26:0/513117521 >> v1:172.21.15.26:6789/0 conn(0x7fb47009cb80 legacy=0x7fb4700717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:58.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 1 -- 172.21.15.26:0/513117521 shutdown_connections 2024-09-16T14:54:58.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 1 -- 172.21.15.26:0/513117521 >> 172.21.15.26:0/513117521 conn(0x7fb47009fb50 msgr2=0x7fb47009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:58.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 1 -- 172.21.15.26:0/513117521 shutdown_connections 2024-09-16T14:54:58.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 1 -- 172.21.15.26:0/513117521 wait complete. 2024-09-16T14:54:58.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 10 monclient: build_initial_monmap 2024-09-16T14:54:58.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 10 monclient: monmap: 2024-09-16T14:54:58.827 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:58.827 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:58.827 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:58.827 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:58.827 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:58.827 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:58.827 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:58.827 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:58.827 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:58.827 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 1 Processor -- start 2024-09-16T14:54:58.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 1 -- start start 2024-09-16T14:54:58.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 10 monclient: init 2024-09-16T14:54:58.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:58.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:54:58.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 10 monclient(hunting): picked mon.noname-a con 0x7fb47009cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:58.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 10 monclient(hunting): picked mon.noname-b con 0x7fb470132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:58.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 10 monclient(hunting): picked mon.noname-c con 0x7fb470138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:58.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47009b370 con 0x7fb47009cb80 2024-09-16T14:54:58.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47009a240 con 0x7fb470132650 2024-09-16T14:54:58.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47013c400 con 0x7fb470138ff0 2024-09-16T14:54:58.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:58.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 10 monclient(hunting): authenticate will time out at 1722.937750s 2024-09-16T14:54:58.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb475c2c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb47009cb80 0x7fb47009a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35508/0 (socket says 172.21.15.26:35508) 2024-09-16T14:54:58.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb47642d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb470138ff0 0x7fb4701d1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59678/0 (socket says 172.21.15.26:59678) 2024-09-16T14:54:58.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb475c2c640 1 -- 172.21.15.26:0/2587667642 learned_addr learned my addr 172.21.15.26:0/2587667642 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:58.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb47542b640 1 -- 172.21.15.26:0/2587667642 >> v1:172.21.15.189:6789/0 conn(0x7fb470132650 legacy=0x7fb47009ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:58.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803596410 0 0) 0x7fb47013c400 con 0x7fb470138ff0 2024-09-16T14:54:58.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient(hunting): _negotiate 2024-09-16T14:54:58.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:58.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:58.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient(hunting): my global_id is 234383 2024-09-16T14:54:58.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb454003530 con 0x7fb470138ff0 2024-09-16T14:54:58.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871723488 0 0) 0x7fb47009b370 con 0x7fb47009cb80 2024-09-16T14:54:58.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient(hunting): _negotiate 2024-09-16T14:54:58.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:58.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:58.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient(hunting): my global_id is 254175 2024-09-16T14:54:58.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb47013c400 con 0x7fb47009cb80 2024-09-16T14:54:58.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4120747432 0 0) 0x7fb454003530 con 0x7fb470138ff0 2024-09-16T14:54:58.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb47009b370 con 0x7fb470138ff0 2024-09-16T14:54:58.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 988747524 0 0) 0x7fb47013c400 con 0x7fb47009cb80 2024-09-16T14:54:58.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb454003530 con 0x7fb47009cb80 2024-09-16T14:54:58.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb468001880 con 0x7fb470138ff0 2024-09-16T14:54:58.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:54:58.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient(hunting): dump: 2024-09-16T14:54:58.830 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.830 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.831 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.831 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.831 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.831 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.831 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.831 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.831 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.831 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:58.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb4600015e0 con 0x7fb47009cb80 2024-09-16T14:54:58.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:58.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient(hunting): dump: 2024-09-16T14:54:58.831 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.832 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.832 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.832 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.832 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.832 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.832 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.833 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.833 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.833 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1672085522 0 0) 0x7fb47009b370 con 0x7fb470138ff0 2024-09-16T14:54:58.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 >> v1:172.21.15.189:6789/0 conn(0x7fb470132650 legacy=0x7fb47009ac60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:54:58.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 >> v1:172.21.15.26:6789/0 conn(0x7fb47009cb80 legacy=0x7fb47009a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:58.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient: _finish_hunting 0 2024-09-16T14:54:58.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 monclient: found mon.c 2024-09-16T14:54:58.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:58.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:58.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4700a28f0 con 0x7fb470138ff0 2024-09-16T14:54:58.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient: _finish_auth 0 2024-09-16T14:54:58.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient: _check_auth_tickets 2024-09-16T14:54:58.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:58.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 5 monclient: authenticate success, global_id 234383 2024-09-16T14:54:58.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 10 monclient: _renew_subs 2024-09-16T14:54:58.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:58.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 1 -- 172.21.15.26:0/2587667642 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47006d890 con 0x7fb470138ff0 2024-09-16T14:54:58.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 10 monclient: _renew_subs 2024-09-16T14:54:58.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:54:58.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 1 -- 172.21.15.26:0/2587667642 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4701386e0 con 0x7fb470138ff0 2024-09-16T14:54:58.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4680047f0 con 0x7fb470138ff0 2024-09-16T14:54:58.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:58.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb468004b00 con 0x7fb470138ff0 2024-09-16T14:54:58.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:58.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 10 monclient: dump: 2024-09-16T14:54:58.836 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.836 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.836 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.836 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.836 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.836 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.836 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.836 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.836 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.836 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb467fff640 4 set_mon_vals no callback set 2024-09-16T14:54:58.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:58.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb477e90640 1 -- 172.21.15.26:0/2587667642 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb458006590 con 0x7fb458002830 2024-09-16T14:54:58.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.814+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb46800f570 con 0x7fb470138ff0 2024-09-16T14:54:58.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fb468043a60 con 0x7fb470138ff0 2024-09-16T14:54:58.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 752834684 0 0) 0x7fb458006590 con 0x7fb458002830 2024-09-16T14:54:58.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 10 monclient(hunting): _negotiate 2024-09-16T14:54:58.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:58.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:58.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 10 monclient(hunting): my global_id is 254178 2024-09-16T14:54:58.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb47009b370 con 0x7fb458002830 2024-09-16T14:54:58.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 546232459 0 0) 0x7fb47009b370 con 0x7fb458002830 2024-09-16T14:54:58.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb458006590 con 0x7fb458002830 2024-09-16T14:54:58.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb460003d40 con 0x7fb458002830 2024-09-16T14:54:58.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:58.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 10 monclient: dump: 2024-09-16T14:54:58.838 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.838 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.838 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.838 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.838 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.838 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.838 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.838 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.838 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.838 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb47542b640 10 monclient: get_auth_request con 0x7fb454036210 auth_method 0 2024-09-16T14:54:58.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3640923492 0 0) 0x7fb458006590 con 0x7fb458002830 2024-09-16T14:54:58.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb45800a0e0 con 0x7fb458002830 2024-09-16T14:54:58.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fb460003ff0 con 0x7fb458002830 2024-09-16T14:54:58.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:58.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:58.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 >> v1:172.21.15.26:6789/0 conn(0x7fb458002830 legacy=0x7fb458004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:58.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.818+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 >> v1:172.21.15.26:6789/0 conn(0x7fb458002830 legacy=0x7fb458004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:58.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.822+0000 7fb477e90640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:54:58.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.822+0000 7fb477e90640 1 -- 172.21.15.26:0/2587667642 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb458009da0 con 0x7fb4580090c0 2024-09-16T14:54:58.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.822+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383086164 0 0) 0x7fb458009da0 con 0x7fb4580090c0 2024-09-16T14:54:58.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.822+0000 7fb4667fc640 10 monclient(hunting): _negotiate 2024-09-16T14:54:58.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.822+0000 7fb4667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:58.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.822+0000 7fb4667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:58.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.822+0000 7fb4667fc640 10 monclient(hunting): my global_id is 254181 2024-09-16T14:54:58.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.822+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb458006590 con 0x7fb4580090c0 2024-09-16T14:54:58.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.822+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4205536244 0 0) 0x7fb458006590 con 0x7fb4580090c0 2024-09-16T14:54:58.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.822+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb458009da0 con 0x7fb4580090c0 2024-09-16T14:54:58.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.822+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb460005790 con 0x7fb4580090c0 2024-09-16T14:54:58.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.822+0000 7fb4667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.822+0000 7fb4667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:58.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.822+0000 7fb4667fc640 10 monclient: dump: 2024-09-16T14:54:58.842 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.842 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.842 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.842 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.842 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.842 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.842 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.842 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.842 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.842 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.822+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1994395052 0 0) 0x7fb458009da0 con 0x7fb4580090c0 2024-09-16T14:54:58.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.822+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb45800c290 con 0x7fb4580090c0 2024-09-16T14:54:58.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.826+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3869 (unknown 0 0 2027952781) 0x7fb460007970 con 0x7fb4580090c0 2024-09-16T14:54:58.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.826+0000 7fb4667fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:54:58.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.826+0000 7fb4667fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:58.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.826+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 >> v1:172.21.15.26:6789/0 conn(0x7fb4580090c0 legacy=0x7fb458009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:58.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.826+0000 7fb4667fc640 1 -- 172.21.15.26:0/2587667642 >> v1:172.21.15.26:6789/0 conn(0x7fb4580090c0 legacy=0x7fb458009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:58.843 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:54:58.843 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:58.843 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:58.843 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:54:58.843 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 118, 2024-09-16T14:54:58.843 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:54:58.844 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:54:58.844 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:54:58.844 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:58.844 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:54:58.844 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:58.844 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:54:58.844 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:54:58.844 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:58.844 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:58.844 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:58.845 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:58.845 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:58.845 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:58.845 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:58.845 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:58.845 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:58.845 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:58.845 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:58.845 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:54:58.846 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:54:58.846 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:58.846 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:58.846 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:58.846 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:58.846 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:58.846 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:58.846 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:58.846 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:58.846 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:58.846 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:58.846 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:58.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:58.847 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:54:58.847 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:54:58.847 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:54:58.847 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:54:58.847 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:54:58.847 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:54:58.847 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:58.847 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:58.847 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:54:58.847 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:54:58.847 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:54:58.847 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:54:58.848 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:54:58.848 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:54:58.848 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:54:58.848 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:58.848 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:54:58.848 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:58.848 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:58.848 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:58.848 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:58.849 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:58.849 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:58.849 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:58.849 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:58.849 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:58.849 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:58.849 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:58.849 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:54:58.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:58.849 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:54:58.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:58.850 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:58.850 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:58.850 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:58.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:58.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:58.850 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:58.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:54:58.850 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:58.850 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:58.850 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:58.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:58.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:54:58.850 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:54:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:54:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:54:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:54:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:54:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:54:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:54:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:54:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:54:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:54:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:54:58.855 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:54:58.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.826+0000 7fb477e90640 1 -- 172.21.15.26:0/2587667642 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb454036210 legacy=0x7fb4540386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:58.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.826+0000 7fb477e90640 10 monclient: shutdown 2024-09-16T14:54:58.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.826+0000 7fb477e90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:58.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.826+0000 7fb477e90640 1 -- 172.21.15.26:0/2587667642 >> v1:172.21.15.26:6790/0 conn(0x7fb470138ff0 legacy=0x7fb4701d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:58.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.826+0000 7fb47642d640 1 -- 172.21.15.26:0/2587667642 reap_dead start 2024-09-16T14:54:58.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.826+0000 7fb47642d640 1 -- 172.21.15.26:0/2587667642 reap_dead start 2024-09-16T14:54:58.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.826+0000 7fb477e90640 1 -- 172.21.15.26:0/2587667642 shutdown_connections 2024-09-16T14:54:58.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.826+0000 7fb477e90640 1 -- 172.21.15.26:0/2587667642 >> 172.21.15.26:0/2587667642 conn(0x7fb47009fb50 msgr2=0x7fb470131d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:58.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.826+0000 7fb477e90640 1 -- 172.21.15.26:0/2587667642 shutdown_connections 2024-09-16T14:54:58.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.826+0000 7fb477e90640 1 -- 172.21.15.26:0/2587667642 wait complete. 2024-09-16T14:54:58.857 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:54:58.858 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:54:58.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:54:58.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 10 monclient: get_monmap_and_config 2024-09-16T14:54:58.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 10 monclient: build_initial_monmap 2024-09-16T14:54:58.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 10 monclient: monmap: 2024-09-16T14:54:58.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:58.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:54:58.974272+0000 2024-09-16T14:54:58.973 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:54:58.974272+0000 2024-09-16T14:54:58.973 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:58.973 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.973 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:58.973 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:54:58.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:54:58.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 1 Processor -- start 2024-09-16T14:54:58.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 1 -- start start 2024-09-16T14:54:58.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 10 monclient: init 2024-09-16T14:54:58.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:58.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:54:58.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 10 monclient(hunting): picked mon.noname-a con 0x7f5c9c09a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:58.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 10 monclient(hunting): picked mon.noname-c con 0x7f5c9c09d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:58.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 10 monclient(hunting): picked mon.noname-b con 0x7f5c9c09b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9c09dc60 con 0x7f5c9c09a550 2024-09-16T14:54:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9c099ae0 con 0x7f5c9c09d1b0 2024-09-16T14:54:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9c09b8f0 con 0x7f5c9c09b040 2024-09-16T14:54:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 10 monclient(hunting): authenticate will time out at 1723.094083s 2024-09-16T14:54:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1b24640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f5c9c09d1b0 legacy=0x7f5c9c136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca2b26640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5c9c09b040 0x7f5c9c13a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59688/0 (socket says 172.21.15.26:59688) 2024-09-16T14:54:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca2b26640 1 -- 172.21.15.26:0/19619009 learned_addr learned my addr 172.21.15.26:0/19619009 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 1 -- 172.21.15.26:0/19619009 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684099285 0 0) 0x7f5c9c09b8f0 con 0x7f5c9c09b040 2024-09-16T14:54:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient(hunting): _negotiate 2024-09-16T14:54:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:58.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient(hunting): my global_id is 234386 2024-09-16T14:54:58.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 1 -- 172.21.15.26:0/19619009 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c88003530 con 0x7f5c9c09b040 2024-09-16T14:54:58.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 1 -- 172.21.15.26:0/19619009 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505868352 0 0) 0x7f5c9c09dc60 con 0x7f5c9c09a550 2024-09-16T14:54:58.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient(hunting): _negotiate 2024-09-16T14:54:58.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:58.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:58.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient(hunting): my global_id is 254184 2024-09-16T14:54:58.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 1 -- 172.21.15.26:0/19619009 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c9c09b8f0 con 0x7f5c9c09a550 2024-09-16T14:54:58.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 1 -- 172.21.15.26:0/19619009 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3536760203 0 0) 0x7f5c88003530 con 0x7f5c9c09b040 2024-09-16T14:54:58.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 1 -- 172.21.15.26:0/19619009 >> v1:172.21.15.189:6789/0 conn(0x7f5c9c09d1b0 legacy=0x7f5c9c136a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:54:58.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 1 -- 172.21.15.26:0/19619009 >> v1:172.21.15.26:6789/0 conn(0x7f5c9c09a550 legacy=0x7f5c9c09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:58.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient: _finish_hunting 0 2024-09-16T14:54:58.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 1 monclient: found mon.noname-b 2024-09-16T14:54:58.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:58.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:54:58.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 1 -- 172.21.15.26:0/19619009 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c9c09bbd0 con 0x7f5c9c09b040 2024-09-16T14:54:58.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient: _finish_auth 0 2024-09-16T14:54:58.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient: _check_auth_tickets 2024-09-16T14:54:58.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:58.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 1 -- 172.21.15.26:0/19619009 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5c940019a0 con 0x7f5c9c09b040 2024-09-16T14:54:58.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:54:58.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient: dump: 2024-09-16T14:54:58.984 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.984 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.984 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.985 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.985 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.985 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.985 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.985 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.985 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.985 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 5 monclient: authenticate success, global_id 234386 2024-09-16T14:54:58.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:54:58.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 1 -- 172.21.15.26:0/19619009 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c94001c80 con 0x7f5c9c09b040 2024-09-16T14:54:58.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:58.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 1 -- 172.21.15.26:0/19619009 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5c94002010 con 0x7f5c9c09b040 2024-09-16T14:54:58.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:58.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca1323640 10 monclient: dump: 2024-09-16T14:54:58.986 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.986 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.986 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.986 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.986 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.986 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.986 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.986 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.986 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.986 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 10 monclient: get_monmap_and_config success 2024-09-16T14:54:58.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 4 set_mon_vals no callback set 2024-09-16T14:54:58.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 10 monclient: shutdown 2024-09-16T14:54:58.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:58.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 1 -- 172.21.15.26:0/19619009 >> v1:172.21.15.26:6790/0 conn(0x7f5c9c09b040 legacy=0x7f5c9c13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:58.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 1 -- 172.21.15.26:0/19619009 shutdown_connections 2024-09-16T14:54:58.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 1 -- 172.21.15.26:0/19619009 >> 172.21.15.26:0/19619009 conn(0x7f5c9c09fb50 msgr2=0x7f5c9c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:58.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 1 -- 172.21.15.26:0/19619009 shutdown_connections 2024-09-16T14:54:58.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.970+0000 7f5ca4589640 1 -- 172.21.15.26:0/19619009 wait complete. 2024-09-16T14:54:58.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 10 monclient: build_initial_monmap 2024-09-16T14:54:58.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 10 monclient: monmap: 2024-09-16T14:54:58.987 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:54:58.987 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:54:58.987 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:54:58.987 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:54:58.987 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:54:58.988 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:54:58.988 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:54:58.988 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:54:58.988 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:54:58.988 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 1 Processor -- start 2024-09-16T14:54:58.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 1 -- start start 2024-09-16T14:54:58.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 10 monclient: init 2024-09-16T14:54:58.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 10 monclient: _reopen_session rank -1 2024-09-16T14:54:58.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:54:58.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 10 monclient(hunting): picked mon.noname-b con 0x7f5c9c09a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:54:58.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 10 monclient(hunting): picked mon.noname-a con 0x7f5c9c09b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:54:58.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 10 monclient(hunting): picked mon.noname-c con 0x7f5c9c09d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:54:58.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9c1a3300 con 0x7f5c9c09b040 2024-09-16T14:54:58.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9c1a0d00 con 0x7f5c9c09a550 2024-09-16T14:54:58.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9c09a240 con 0x7f5c9c09d1b0 2024-09-16T14:54:58.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 10 monclient(hunting): _renew_subs 2024-09-16T14:54:58.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 10 monclient(hunting): authenticate will time out at 1723.097285s 2024-09-16T14:54:58.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca1b24640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5c9c09b040 0x7f5c9c1a0050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35560/0 (socket says 172.21.15.26:35560) 2024-09-16T14:54:58.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca1b24640 1 -- 172.21.15.26:0/2373788939 learned_addr learned my addr 172.21.15.26:0/2373788939 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:54:58.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca2b26640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5c9c09d1b0 0x7f5c9c1cd520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59702/0 (socket says 172.21.15.26:59702) 2024-09-16T14:54:58.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca2325640 1 -- 172.21.15.26:0/2373788939 >> v1:172.21.15.189:6789/0 conn(0x7f5c9c09a550 legacy=0x7f5c9c19f940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:54:58.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718774541 0 0) 0x7f5c9c09a240 con 0x7f5c9c09d1b0 2024-09-16T14:54:58.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:58.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:58.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:58.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): my global_id is 234389 2024-09-16T14:54:58.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c80003530 con 0x7f5c9c09d1b0 2024-09-16T14:54:58.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568119833 0 0) 0x7f5c9c1a3300 con 0x7f5c9c09b040 2024-09-16T14:54:58.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:58.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:58.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:58.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): my global_id is 254187 2024-09-16T14:54:58.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c9c09a240 con 0x7f5c9c09b040 2024-09-16T14:54:58.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 476492471 0 0) 0x7f5c9c09a240 con 0x7f5c9c09b040 2024-09-16T14:54:58.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c9c1a3300 con 0x7f5c9c09b040 2024-09-16T14:54:58.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5c8c004190 con 0x7f5c9c09b040 2024-09-16T14:54:58.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:54:58.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): dump: 2024-09-16T14:54:58.991 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.991 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.991 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.991 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.991 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.991 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.991 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.991 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.991 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.991 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:54:58.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4230291524 0 0) 0x7f5c80003530 con 0x7f5c9c09d1b0 2024-09-16T14:54:58.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c9c09a240 con 0x7f5c9c09d1b0 2024-09-16T14:54:58.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5c940017c0 con 0x7f5c9c09d1b0 2024-09-16T14:54:58.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:58.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): dump: 2024-09-16T14:54:58.992 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.992 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.992 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.992 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.992 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.992 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.992 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.992 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.992 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.993 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 604987047 0 0) 0x7f5c9c1a3300 con 0x7f5c9c09b040 2024-09-16T14:54:58.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 >> v1:172.21.15.26:6790/0 conn(0x7f5c9c09d1b0 legacy=0x7f5c9c1cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:58.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 >> v1:172.21.15.189:6789/0 conn(0x7f5c9c09a550 legacy=0x7f5c9c19f940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:54:58.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:54:58.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 monclient: found mon.a 2024-09-16T14:54:58.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:54:58.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:58.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c9c1a3020 con 0x7f5c9c09b040 2024-09-16T14:54:58.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient: _finish_auth 0 2024-09-16T14:54:58.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:54:58.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:54:58.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 5 monclient: authenticate success, global_id 254187 2024-09-16T14:54:58.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 10 monclient: _renew_subs 2024-09-16T14:54:58.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:58.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 1 -- 172.21.15.26:0/2373788939 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c9c1a14f0 con 0x7f5c9c09b040 2024-09-16T14:54:58.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 10 monclient: _renew_subs 2024-09-16T14:54:58.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:54:58.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 1 -- 172.21.15.26:0/2373788939 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c9c1a0930 con 0x7f5c9c09b040 2024-09-16T14:54:58.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c8c004660 con 0x7f5c9c09b040 2024-09-16T14:54:58.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:54:58.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5c8c004970 con 0x7f5c9c09b040 2024-09-16T14:54:58.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:54:58.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca0b22640 4 set_mon_vals no callback set 2024-09-16T14:54:58.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient: dump: 2024-09-16T14:54:58.994 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.995 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca4589640 1 -- 172.21.15.26:0/2373788939 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5c84006590 con 0x7f5c84002830 2024-09-16T14:54:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5c8c004ad0 con 0x7f5c9c09b040 2024-09-16T14:54:58.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f5c8c043d00 con 0x7f5c9c09b040 2024-09-16T14:54:58.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405058917 0 0) 0x7f5c84006590 con 0x7f5c84002830 2024-09-16T14:54:58.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:58.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:58.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:58.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient(hunting): my global_id is 234392 2024-09-16T14:54:58.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c9c1a3300 con 0x7f5c84002830 2024-09-16T14:54:58.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1347357052 0 0) 0x7f5c9c1a3300 con 0x7f5c84002830 2024-09-16T14:54:58.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c84006590 con 0x7f5c84002830 2024-09-16T14:54:58.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5c98001750 con 0x7f5c84002830 2024-09-16T14:54:58.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:58.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:58.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 10 monclient: dump: 2024-09-16T14:54:58.997 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:58.997 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:58.997 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.997 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:58.997 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:58.997 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:58.997 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:58.997 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:58.997 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:58.997 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:58.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5ca2b26640 10 monclient: get_auth_request con 0x7f5c80036270 auth_method 0 2024-09-16T14:54:58.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1579173438 0 0) 0x7f5c84006590 con 0x7f5c84002830 2024-09-16T14:54:58.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.974+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c8400a0e0 con 0x7f5c84002830 2024-09-16T14:54:58.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.978+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5c98001bb0 con 0x7f5c84002830 2024-09-16T14:54:58.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.978+0000 7f5c92ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:54:58.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.978+0000 7f5c92ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:54:58.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.978+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 >> v1:172.21.15.26:6790/0 conn(0x7f5c84002830 legacy=0x7f5c84004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:58.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.978+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 >> v1:172.21.15.26:6790/0 conn(0x7f5c84002830 legacy=0x7f5c84004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:58.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5ca4589640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:54:59.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5ca4589640 1 -- 172.21.15.26:0/2373788939 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5c84009da0 con 0x7f5c840090c0 2024-09-16T14:54:59.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407902360 0 0) 0x7f5c84009da0 con 0x7f5c840090c0 2024-09-16T14:54:59.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:54:59.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:54:59.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:54:59.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 10 monclient(hunting): my global_id is 234395 2024-09-16T14:54:59.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c84006590 con 0x7f5c840090c0 2024-09-16T14:54:59.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2532241483 0 0) 0x7f5c84006590 con 0x7f5c840090c0 2024-09-16T14:54:59.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c84009da0 con 0x7f5c840090c0 2024-09-16T14:54:59.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5c98003a30 con 0x7f5c840090c0 2024-09-16T14:54:59.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:54:59.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:54:59.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 10 monclient: dump: 2024-09-16T14:54:59.001 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:54:59.001 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:54:59.001 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:59.001 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:54:59.001 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:54:59.001 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:54:59.001 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:54:59.001 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:54:59.001 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:54:59.001 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:54:59.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3938771719 0 0) 0x7f5c84009da0 con 0x7f5c840090c0 2024-09-16T14:54:59.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5c8400c290 con 0x7f5c840090c0 2024-09-16T14:54:59.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4203 (unknown 0 0 2419387647) 0x7f5c980057d0 con 0x7f5c840090c0 2024-09-16T14:54:59.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:54:59.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:54:59.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 >> v1:172.21.15.26:6790/0 conn(0x7f5c840090c0 legacy=0x7f5c84009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:59.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5c92ffd640 1 -- 172.21.15.26:0/2373788939 >> v1:172.21.15.26:6790/0 conn(0x7f5c840090c0 legacy=0x7f5c84009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:54:59.002 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:54:59.002 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:59.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:59.002 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:54:59.002 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 118, 2024-09-16T14:54:59.003 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:54:59.003 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:54:59.003 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:54:59.003 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:59.003 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:54:59.003 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:59.003 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:54:59.003 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:54:59.003 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:59.003 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:59.003 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:59.003 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:59.003 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:59.004 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:59.004 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:59.004 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:59.004 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:59.004 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:59.004 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:59.004 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:54:59.004 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:54:59.004 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:59.004 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:59.004 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:59.004 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:59.004 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:59.004 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:59.005 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:59.005 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:59.005 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:59.005 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:59.005 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:59.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:59.005 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:54:59.005 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:54:59.005 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:54:59.005 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:54:59.005 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:54:59.005 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:54:59.005 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:59.005 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:54:59.006 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:54:59.006 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:54:59.006 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:54:59.006 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:54:59.006 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:54:59.006 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:54:59.006 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:54:59.006 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:54:59.006 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:54:59.006 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:54:59.006 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:54:59.006 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:54:59.006 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:54:59.006 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:54:59.007 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:54:59.007 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:54:59.007 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:54:59.007 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:54:59.007 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:54:59.007 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:59.007 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:54:59.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:59.007 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:54:59.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:59.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:54:59.007 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:54:59.008 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:59.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:59.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:59.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:59.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:54:59.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:59.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:59.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:59.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:59.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:54:59.009 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:54:59.009 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:59.009 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:59.009 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:59.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:59.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:59.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:54:59.009 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:54:59.010 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:59.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:59.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:59.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:59.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:54:59.010 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:59.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:59.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:59.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:59.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:54:59.010 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:54:59.010 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:59.011 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:59.011 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:59.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:59.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:59.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:54:59.011 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:54:59.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:54:59.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:54:59.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:59.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:54:59.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:54:59.011 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:54:59.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:59.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:59.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:59.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:54:59.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:54:59.012 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:54:59.012 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:54:59.012 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:54:59.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:59.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:59.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:59.012 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:54:59.012 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:54:59.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:59.013 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:59.013 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:59.013 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:59.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:59.013 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:59.013 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:54:59.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:59.013 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:59.013 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:59.013 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:54:59.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:59.013 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:59.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:54:59.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:59.014 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:59.014 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:59.014 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:54:59.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:59.014 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:54:59.014 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:54:59.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:54:59.014 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:54:59.014 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:54:59.014 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:54:59.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:54:59.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:54:59.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:54:59.015 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:54:59.015 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:54:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5ca4589640 1 -- 172.21.15.26:0/2373788939 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5c80036270 legacy=0x7f5c80038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5ca4589640 10 monclient: shutdown 2024-09-16T14:54:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5ca4589640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:54:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5ca4589640 1 -- 172.21.15.26:0/2373788939 >> v1:172.21.15.26:6789/0 conn(0x7f5c9c09b040 legacy=0x7f5c9c1a0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:54:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5ca2b26640 1 -- 172.21.15.26:0/2373788939 reap_dead start 2024-09-16T14:54:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5ca2b26640 1 -- 172.21.15.26:0/2373788939 reap_dead start 2024-09-16T14:54:59.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5ca4589640 1 -- 172.21.15.26:0/2373788939 shutdown_connections 2024-09-16T14:54:59.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5ca4589640 1 -- 172.21.15.26:0/2373788939 >> 172.21.15.26:0/2373788939 conn(0x7f5c9c09fb50 msgr2=0x7f5c9c0a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:54:59.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5ca4589640 1 -- 172.21.15.26:0/2373788939 shutdown_connections 2024-09-16T14:54:59.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:54:58.982+0000 7f5ca4589640 1 -- 172.21.15.26:0/2373788939 wait complete. 2024-09-16T14:54:59.017 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:54:59.017 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:55:19.018 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:55:19.018 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T14:55:19.018 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T14:55:19.019 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T14:55:19.021 INFO:tasks.ceph.mon.b:Started 2024-09-16T14:55:19.022 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:55:19.022 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:55:19.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5c762640 10 monclient: get_monmap_and_config 2024-09-16T14:55:19.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5c762640 10 monclient: build_initial_monmap 2024-09-16T14:55:19.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5c762640 10 monclient: monmap: 2024-09-16T14:55:19.153 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:19.153 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.153 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:19.154481+0000 2024-09-16T14:55:19.153 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:19.154481+0000 2024-09-16T14:55:19.153 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:19.153 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.153 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:19.153 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:19.153 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:19.154 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5c762640 1 Processor -- start 2024-09-16T14:55:19.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5c762640 1 -- start start 2024-09-16T14:55:19.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5c762640 10 monclient: init 2024-09-16T14:55:19.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5c762640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:19.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5c762640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:55:19.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5c762640 10 monclient(hunting): picked mon.noname-a con 0x7f3c54096600 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:19.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5c762640 10 monclient(hunting): picked mon.noname-b con 0x7f3c5409d580 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:19.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5c762640 10 monclient(hunting): picked mon.noname-c con 0x7f3c54137600 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:19.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5c762640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5409bed0 con 0x7f3c54096600 2024-09-16T14:55:19.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5c762640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5409cc00 con 0x7f3c54137600 2024-09-16T14:55:19.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5c762640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5409f7d0 con 0x7f3c5409d580 2024-09-16T14:55:19.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5c762640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:19.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5c762640 10 monclient(hunting): authenticate will time out at 1743.275659s 2024-09-16T14:55:19.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5a4fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3c54096600 0x7f3c540969e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38762/0 (socket says 172.21.15.26:38762) 2024-09-16T14:55:19.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.150+0000 7f3c5a4fe640 1 -- 172.21.15.26:0/4238970747 learned_addr learned my addr 172.21.15.26:0/4238970747 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:19.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 1 -- 172.21.15.26:0/4238970747 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288058772 0 0) 0x7f3c5409f7d0 con 0x7f3c5409d580 2024-09-16T14:55:19.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 10 monclient(hunting): my global_id is 234398 2024-09-16T14:55:19.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 1 -- 172.21.15.26:0/4238970747 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c40003450 con 0x7f3c5409d580 2024-09-16T14:55:19.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 1 -- 172.21.15.26:0/4238970747 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 245438208 0 0) 0x7f3c40003450 con 0x7f3c5409d580 2024-09-16T14:55:19.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 1 -- 172.21.15.26:0/4238970747 >> v1:172.21.15.189:6789/0 conn(0x7f3c54137600 legacy=0x7f3c541399f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 1 -- 172.21.15.26:0/4238970747 >> v1:172.21.15.26:6789/0 conn(0x7f3c54096600 legacy=0x7f3c540969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 10 monclient: _finish_hunting 0 2024-09-16T14:55:19.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 1 monclient: found mon.noname-b 2024-09-16T14:55:19.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:19.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:55:19.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 1 -- 172.21.15.26:0/4238970747 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5409c2f0 con 0x7f3c5409d580 2024-09-16T14:55:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 10 monclient: _finish_auth 0 2024-09-16T14:55:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 10 monclient: _check_auth_tickets 2024-09-16T14:55:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 1 -- 172.21.15.26:0/4238970747 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3c44001620 con 0x7f3c5409d580 2024-09-16T14:55:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:55:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 10 monclient: dump: 2024-09-16T14:55:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.159 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.160 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.160 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.160 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.160 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c5c762640 5 monclient: authenticate success, global_id 234398 2024-09-16T14:55:19.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c5c762640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:19.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 1 -- 172.21.15.26:0/4238970747 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c44001940 con 0x7f3c5409d580 2024-09-16T14:55:19.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:19.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 1 -- 172.21.15.26:0/4238970747 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3c44001cd0 con 0x7f3c5409d580 2024-09-16T14:55:19.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:19.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c594fc640 10 monclient: dump: 2024-09-16T14:55:19.160 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.161 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.161 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.161 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.161 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.161 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.161 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.161 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.161 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.161 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c5c762640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:19.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c5c762640 4 set_mon_vals no callback set 2024-09-16T14:55:19.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c5c762640 10 monclient: shutdown 2024-09-16T14:55:19.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c5c762640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:19.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c5c762640 1 -- 172.21.15.26:0/4238970747 >> v1:172.21.15.26:6790/0 conn(0x7f3c5409d580 legacy=0x7f3c54135ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c5c762640 1 -- 172.21.15.26:0/4238970747 shutdown_connections 2024-09-16T14:55:19.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c5c762640 1 -- 172.21.15.26:0/4238970747 >> 172.21.15.26:0/4238970747 conn(0x7f3c540a0360 msgr2=0x7f3c540a0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:19.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c5c762640 1 -- 172.21.15.26:0/4238970747 shutdown_connections 2024-09-16T14:55:19.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c5c762640 1 -- 172.21.15.26:0/4238970747 wait complete. 2024-09-16T14:55:19.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c5c762640 10 monclient: build_initial_monmap 2024-09-16T14:55:19.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c5c762640 10 monclient: monmap: 2024-09-16T14:55:19.166 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:19.166 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:19.166 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:19.166 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:19.166 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:19.166 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:19.166 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:19.166 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:19.166 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:19.167 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c5c762640 1 Processor -- start 2024-09-16T14:55:19.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c5c762640 1 -- start start 2024-09-16T14:55:19.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.154+0000 7f3c5c762640 10 monclient: init 2024-09-16T14:55:19.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:19.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:55:19.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 10 monclient(hunting): picked mon.noname-c con 0x7f3c54096600 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:19.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 10 monclient(hunting): picked mon.noname-b con 0x7f3c5409d580 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:19.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 10 monclient(hunting): picked mon.noname-a con 0x7f3c54137600 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:19.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c541361b0 con 0x7f3c54137600 2024-09-16T14:55:19.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5409a880 con 0x7f3c5409d580 2024-09-16T14:55:19.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c54137350 con 0x7f3c54096600 2024-09-16T14:55:19.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:19.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 10 monclient(hunting): authenticate will time out at 1743.280439s 2024-09-16T14:55:19.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5a4fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3c54096600 0x7f3c54134690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47278/0 (socket says 172.21.15.26:47278) 2024-09-16T14:55:19.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5a4fe640 1 -- 172.21.15.26:0/2184877999 learned_addr learned my addr 172.21.15.26:0/2184877999 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:19.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5acff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3c54137600 0x7f3c541354b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38774/0 (socket says 172.21.15.26:38774) 2024-09-16T14:55:19.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 1 -- 172.21.15.26:0/2184877999 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741161402 0 0) 0x7f3c5409bed0 con 0x7f3c54096600 2024-09-16T14:55:19.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 10 monclient(hunting): my global_id is 234401 2024-09-16T14:55:19.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 1 -- 172.21.15.26:0/2184877999 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c38003530 con 0x7f3c54096600 2024-09-16T14:55:19.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 1 -- 172.21.15.26:0/2184877999 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3943954716 0 0) 0x7f3c38003530 con 0x7f3c54096600 2024-09-16T14:55:19.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 1 -- 172.21.15.26:0/2184877999 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c5409bed0 con 0x7f3c54096600 2024-09-16T14:55:19.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 1 -- 172.21.15.26:0/2184877999 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3c50001610 con 0x7f3c54096600 2024-09-16T14:55:19.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:55:19.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 10 monclient(hunting): dump: 2024-09-16T14:55:19.169 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.169 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.169 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.169 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.169 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.169 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.169 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.169 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.169 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.170 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:19.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 1 -- 172.21.15.26:0/2184877999 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3547335106 0 0) 0x7f3c5409bed0 con 0x7f3c54096600 2024-09-16T14:55:19.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 1 -- 172.21.15.26:0/2184877999 >> v1:172.21.15.189:6789/0 conn(0x7f3c5409d580 legacy=0x7f3c54134da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 1 -- 172.21.15.26:0/2184877999 >> v1:172.21.15.26:6789/0 conn(0x7f3c54137600 legacy=0x7f3c541354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 10 monclient: _finish_hunting 0 2024-09-16T14:55:19.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 1 monclient: found mon.c 2024-09-16T14:55:19.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:19.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:19.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 1 -- 172.21.15.26:0/2184877999 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c54129f90 con 0x7f3c54096600 2024-09-16T14:55:19.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 10 monclient: _finish_auth 0 2024-09-16T14:55:19.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 10 monclient: _check_auth_tickets 2024-09-16T14:55:19.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:19.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 5 monclient: authenticate success, global_id 234401 2024-09-16T14:55:19.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 10 monclient: _renew_subs 2024-09-16T14:55:19.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:19.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 1 -- 172.21.15.26:0/2184877999 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c54129d10 con 0x7f3c54096600 2024-09-16T14:55:19.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 10 monclient: _renew_subs 2024-09-16T14:55:19.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:19.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 1 -- 172.21.15.26:0/2184877999 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c541d2820 con 0x7f3c54096600 2024-09-16T14:55:19.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 1 -- 172.21.15.26:0/2184877999 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c50001be0 con 0x7f3c54096600 2024-09-16T14:55:19.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:19.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 1 -- 172.21.15.26:0/2184877999 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3c50004990 con 0x7f3c54096600 2024-09-16T14:55:19.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:19.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c58cfb640 4 set_mon_vals no callback set 2024-09-16T14:55:19.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 10 monclient: dump: 2024-09-16T14:55:19.172 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.172 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.172 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.172 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.172 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.172 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.172 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.173 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.173 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.173 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 1 -- 172.21.15.26:0/2184877999 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3c50004af0 con 0x7f3c54096600 2024-09-16T14:55:19.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:19.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:19.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:19.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c5c762640 1 -- 172.21.15.26:0/2184877999 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c3c005050 con 0x7f3c54096600 2024-09-16T14:55:19.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c4affd640 1 -- 172.21.15.26:0/2184877999 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f3c50043ba0 con 0x7f3c54096600 2024-09-16T14:55:19.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.158+0000 7f3c59cfd640 10 monclient: get_auth_request con 0x7f3c38036100 auth_method 0 2024-09-16T14:55:19.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.162+0000 7f3c4affd640 1 -- 172.21.15.26:0/2184877999 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3c500052f0 con 0x7f3c54096600 2024-09-16T14:55:19.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.162+0000 7f3c4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:19.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.162+0000 7f3c4affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:19.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.377+0000 7f3c5c762640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:55:19.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.377+0000 7f3c5c762640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:55:19.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.377+0000 7f3c5c762640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:19.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.377+0000 7f3c5c762640 1 -- 172.21.15.26:0/2184877999 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3c3c005880 con 0x7f3c54096600 2024-09-16T14:55:19.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.377+0000 7f3c4affd640 1 -- 172.21.15.26:0/2184877999 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2505120982) 0x7f3c50012950 con 0x7f3c54096600 2024-09-16T14:55:19.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.377+0000 7f3c4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:55:19.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.377+0000 7f3c4affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:19.383 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":120,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:55:19.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.381+0000 7f3c5c762640 1 -- 172.21.15.26:0/2184877999 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3c38036100 legacy=0x7f3c380385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.381+0000 7f3c5c762640 10 monclient: shutdown 2024-09-16T14:55:19.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.381+0000 7f3c5c762640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:19.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.381+0000 7f3c5c762640 1 -- 172.21.15.26:0/2184877999 >> v1:172.21.15.26:6790/0 conn(0x7f3c54096600 legacy=0x7f3c54134690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.381+0000 7f3c5c762640 1 -- 172.21.15.26:0/2184877999 shutdown_connections 2024-09-16T14:55:19.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.381+0000 7f3c5c762640 1 -- 172.21.15.26:0/2184877999 >> 172.21.15.26:0/2184877999 conn(0x7f3c540a0360 msgr2=0x7f3c540a2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:19.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.381+0000 7f3c5c762640 1 -- 172.21.15.26:0/2184877999 shutdown_connections 2024-09-16T14:55:19.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.381+0000 7f3c5c762640 1 -- 172.21.15.26:0/2184877999 wait complete. 2024-09-16T14:55:19.400 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:55:19.400 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:55:19.400 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:55:19.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.505+0000 7f8e94404640 10 monclient: get_monmap_and_config 2024-09-16T14:55:19.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.505+0000 7f8e94404640 10 monclient: build_initial_monmap 2024-09-16T14:55:19.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.505+0000 7f8e94404640 10 monclient: monmap: 2024-09-16T14:55:19.511 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:19.511 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.511 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:19.512220+0000 2024-09-16T14:55:19.511 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:19.512220+0000 2024-09-16T14:55:19.511 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:19.512 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.512 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:19.512 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:19.512 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:19.512 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.505+0000 7f8e94404640 1 Processor -- start 2024-09-16T14:55:19.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.505+0000 7f8e94404640 1 -- start start 2024-09-16T14:55:19.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.505+0000 7f8e94404640 10 monclient: init 2024-09-16T14:55:19.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:19.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:55:19.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient(hunting): picked mon.noname-c con 0x7f8e8c09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:19.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient(hunting): picked mon.noname-b con 0x7f8e8c132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:19.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient(hunting): picked mon.noname-a con 0x7f8e8c138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:19.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8c09d260 con 0x7f8e8c138ff0 2024-09-16T14:55:19.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8c132a80 con 0x7f8e8c09cb80 2024-09-16T14:55:19.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8c09cf60 con 0x7f8e8c132650 2024-09-16T14:55:19.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:19.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient(hunting): authenticate will time out at 1743.632275s 2024-09-16T14:55:19.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e929a1640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8e8c138ff0 0x7f8e8c13b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38776/0 (socket says 172.21.15.26:38776) 2024-09-16T14:55:19.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9199f640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8e8c132650 0x7f8e8c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47282/0 (socket says 172.21.15.26:47282) 2024-09-16T14:55:19.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e929a1640 1 -- 172.21.15.26:0/2430028563 learned_addr learned my addr 172.21.15.26:0/2430028563 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:19.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 1 -- 172.21.15.26:0/2430028563 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996912032 0 0) 0x7f8e8c09d260 con 0x7f8e8c138ff0 2024-09-16T14:55:19.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient(hunting): my global_id is 254190 2024-09-16T14:55:19.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 1 -- 172.21.15.26:0/2430028563 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e74003530 con 0x7f8e8c138ff0 2024-09-16T14:55:19.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 1 -- 172.21.15.26:0/2430028563 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178508305 0 0) 0x7f8e8c09cf60 con 0x7f8e8c132650 2024-09-16T14:55:19.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient(hunting): my global_id is 234404 2024-09-16T14:55:19.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 1 -- 172.21.15.26:0/2430028563 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e8c09d260 con 0x7f8e8c132650 2024-09-16T14:55:19.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 1 -- 172.21.15.26:0/2430028563 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 169741397 0 0) 0x7f8e8c09d260 con 0x7f8e8c132650 2024-09-16T14:55:19.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 1 -- 172.21.15.26:0/2430028563 >> v1:172.21.15.189:6789/0 conn(0x7f8e8c09cb80 legacy=0x7f8e8c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 1 -- 172.21.15.26:0/2430028563 >> v1:172.21.15.26:6789/0 conn(0x7f8e8c138ff0 legacy=0x7f8e8c13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient: _finish_hunting 0 2024-09-16T14:55:19.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 1 monclient: found mon.noname-b 2024-09-16T14:55:19.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:19.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:55:19.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 1 -- 172.21.15.26:0/2430028563 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c09f2d0 con 0x7f8e8c132650 2024-09-16T14:55:19.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient: _finish_auth 0 2024-09-16T14:55:19.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient: _check_auth_tickets 2024-09-16T14:55:19.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:19.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 5 monclient: authenticate success, global_id 234404 2024-09-16T14:55:19.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 1 -- 172.21.15.26:0/2430028563 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8e800015e0 con 0x7f8e8c132650 2024-09-16T14:55:19.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:19.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:55:19.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient: dump: 2024-09-16T14:55:19.524 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.524 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.524 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.524 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.524 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.524 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.524 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.524 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.524 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.525 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 1 -- 172.21.15.26:0/2430028563 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e800017c0 con 0x7f8e8c132650 2024-09-16T14:55:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 1 -- 172.21.15.26:0/2430028563 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8e80001b10 con 0x7f8e8c132650 2024-09-16T14:55:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 4 set_mon_vals no callback set 2024-09-16T14:55:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9119e640 10 monclient: dump: 2024-09-16T14:55:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.526 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.526 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.526 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.526 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.526 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.526 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.526 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient: shutdown 2024-09-16T14:55:19.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:19.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 1 -- 172.21.15.26:0/2430028563 >> v1:172.21.15.26:6790/0 conn(0x7f8e8c132650 legacy=0x7f8e8c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 1 -- 172.21.15.26:0/2430028563 shutdown_connections 2024-09-16T14:55:19.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 1 -- 172.21.15.26:0/2430028563 >> 172.21.15.26:0/2430028563 conn(0x7f8e8c09fb50 msgr2=0x7f8e8c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:19.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 1 -- 172.21.15.26:0/2430028563 shutdown_connections 2024-09-16T14:55:19.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 1 -- 172.21.15.26:0/2430028563 wait complete. 2024-09-16T14:55:19.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient: build_initial_monmap 2024-09-16T14:55:19.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient: monmap: 2024-09-16T14:55:19.527 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:19.527 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:19.527 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:19.527 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:19.527 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:19.527 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:19.527 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:19.527 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:19.527 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:19.527 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 1 Processor -- start 2024-09-16T14:55:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 1 -- start start 2024-09-16T14:55:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient: init 2024-09-16T14:55:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:55:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient(hunting): picked mon.noname-b con 0x7f8e8c09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient(hunting): picked mon.noname-a con 0x7f8e8c132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient(hunting): picked mon.noname-c con 0x7f8e8c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8c09a240 con 0x7f8e8c132650 2024-09-16T14:55:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8c06e450 con 0x7f8e8c09cb80 2024-09-16T14:55:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8c13c520 con 0x7f8e8c138ff0 2024-09-16T14:55:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e94404640 10 monclient(hunting): authenticate will time out at 1743.635275s 2024-09-16T14:55:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9199f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8e8c132650 0x7f8e8c06dd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38780/0 (socket says 172.21.15.26:38780) 2024-09-16T14:55:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.509+0000 7f8e9199f640 1 -- 172.21.15.26:0/2050135574 learned_addr learned my addr 172.21.15.26:0/2050135574 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154002548 0 0) 0x7f8e8c13c520 con 0x7f8e8c138ff0 2024-09-16T14:55:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): my global_id is 234407 2024-09-16T14:55:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e78003530 con 0x7f8e8c138ff0 2024-09-16T14:55:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028960134 0 0) 0x7f8e8c09a240 con 0x7f8e8c132650 2024-09-16T14:55:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): my global_id is 254193 2024-09-16T14:55:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e8c13c520 con 0x7f8e8c132650 2024-09-16T14:55:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3078826357 0 0) 0x7f8e78003530 con 0x7f8e8c138ff0 2024-09-16T14:55:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e8c09a240 con 0x7f8e8c138ff0 2024-09-16T14:55:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8e7c001100 con 0x7f8e8c138ff0 2024-09-16T14:55:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:55:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): dump: 2024-09-16T14:55:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.531 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 566807856 0 0) 0x7f8e8c13c520 con 0x7f8e8c132650 2024-09-16T14:55:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e78003530 con 0x7f8e8c132650 2024-09-16T14:55:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8e80001ad0 con 0x7f8e8c132650 2024-09-16T14:55:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): dump: 2024-09-16T14:55:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.532 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 270277349 0 0) 0x7f8e8c09a240 con 0x7f8e8c138ff0 2024-09-16T14:55:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 >> v1:172.21.15.189:6789/0 conn(0x7f8e8c09cb80 legacy=0x7f8e8c06d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 >> v1:172.21.15.26:6789/0 conn(0x7f8e8c132650 legacy=0x7f8e8c06dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient: _finish_hunting 0 2024-09-16T14:55:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 monclient: found mon.c 2024-09-16T14:55:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c0715e0 con 0x7f8e8c138ff0 2024-09-16T14:55:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient: _finish_auth 0 2024-09-16T14:55:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient: _check_auth_tickets 2024-09-16T14:55:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e94404640 5 monclient: authenticate success, global_id 234407 2024-09-16T14:55:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e94404640 10 monclient: _renew_subs 2024-09-16T14:55:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e94404640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e94404640 1 -- 172.21.15.26:0/2050135574 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8c1380c0 con 0x7f8e8c138ff0 2024-09-16T14:55:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e94404640 10 monclient: _renew_subs 2024-09-16T14:55:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e94404640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e94404640 1 -- 172.21.15.26:0/2050135574 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8c06fbd0 con 0x7f8e8c138ff0 2024-09-16T14:55:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e7c0047f0 con 0x7f8e8c138ff0 2024-09-16T14:55:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8e7c004b00 con 0x7f8e8c138ff0 2024-09-16T14:55:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient: dump: 2024-09-16T14:55:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e9099d640 4 set_mon_vals no callback set 2024-09-16T14:55:19.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e94404640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:19.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e94404640 1 -- 172.21.15.26:0/2050135574 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8e6c006590 con 0x7f8e6c002830 2024-09-16T14:55:19.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8e7c00f570 con 0x7f8e8c138ff0 2024-09-16T14:55:19.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f8e7c043bf0 con 0x7f8e8c138ff0 2024-09-16T14:55:19.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580106760 0 0) 0x7f8e6c006590 con 0x7f8e6c002830 2024-09-16T14:55:19.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient(hunting): my global_id is 254196 2024-09-16T14:55:19.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e8c09a240 con 0x7f8e6c002830 2024-09-16T14:55:19.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e9199f640 10 monclient: get_auth_request con 0x7f8e78036250 auth_method 0 2024-09-16T14:55:19.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 655705294 0 0) 0x7f8e8c09a240 con 0x7f8e6c002830 2024-09-16T14:55:19.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e6c006590 con 0x7f8e6c002830 2024-09-16T14:55:19.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8e840018b0 con 0x7f8e6c002830 2024-09-16T14:55:19.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:19.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient: dump: 2024-09-16T14:55:19.537 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.537 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.537 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.537 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.537 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.537 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.537 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.537 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.537 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.538 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3286957342 0 0) 0x7f8e6c006590 con 0x7f8e6c002830 2024-09-16T14:55:19.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e6c00a0e0 con 0x7f8e6c002830 2024-09-16T14:55:19.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8e84001dc0 con 0x7f8e6c002830 2024-09-16T14:55:19.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:19.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:19.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 >> v1:172.21.15.26:6789/0 conn(0x7f8e6c002830 legacy=0x7f8e6c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.513+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 >> v1:172.21.15.26:6789/0 conn(0x7f8e6c002830 legacy=0x7f8e6c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e94404640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:55:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e94404640 1 -- 172.21.15.26:0/2050135574 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8e6c009da0 con 0x7f8e6c0090c0 2024-09-16T14:55:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681694139 0 0) 0x7f8e6c009da0 con 0x7f8e6c0090c0 2024-09-16T14:55:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 10 monclient(hunting): my global_id is 254199 2024-09-16T14:55:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e6c006590 con 0x7f8e6c0090c0 2024-09-16T14:55:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 670519203 0 0) 0x7f8e6c006590 con 0x7f8e6c0090c0 2024-09-16T14:55:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e6c009da0 con 0x7f8e6c0090c0 2024-09-16T14:55:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8e84003b40 con 0x7f8e6c0090c0 2024-09-16T14:55:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 10 monclient: dump: 2024-09-16T14:55:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.542 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3940627884 0 0) 0x7f8e6c009da0 con 0x7f8e6c0090c0 2024-09-16T14:55:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8e6c00c290 con 0x7f8e6c0090c0 2024-09-16T14:55:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 2396323936) 0x7f8e840049d0 con 0x7f8e6c0090c0 2024-09-16T14:55:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:55:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 >> v1:172.21.15.26:6789/0 conn(0x7f8e6c0090c0 legacy=0x7f8e6c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e8affd640 1 -- 172.21.15.26:0/2050135574 >> v1:172.21.15.26:6789/0 conn(0x7f8e6c0090c0 legacy=0x7f8e6c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:19.543 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:55:19.543 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:55:19.543 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:55:19.543 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:55:19.544 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 120, 2024-09-16T14:55:19.544 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:55:19.544 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:55:19.544 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:55:19.544 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:55:19.544 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:19.544 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:55:19.544 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:55:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:55:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:19.545 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:55:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:55:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:55:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:55:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:55:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:55:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:55:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:55:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:55:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:55:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:55:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:55:19.547 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:55:19.548 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:55:19.548 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:55:19.548 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:19.548 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:55:19.548 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:19.548 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:19.548 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:19.548 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:19.548 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:19.548 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:19.548 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:19.548 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:19.548 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:55:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:55:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:55:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:55:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:55:19.549 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:55:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:55:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:55:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:55:19.550 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:55:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:55:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:55:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:19.551 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:55:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:55:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:55:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.552 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:55:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:55:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:55:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:55:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:19.553 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:19.554 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:19.554 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.554 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:19.554 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:55:19.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.554 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:19.554 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:19.554 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:19.554 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.554 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.554 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:55:19.555 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:55:19.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e94404640 1 -- 172.21.15.26:0/2050135574 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8e78036250 legacy=0x7f8e78038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e94404640 10 monclient: shutdown 2024-09-16T14:55:19.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e94404640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:19.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e94404640 1 -- 172.21.15.26:0/2050135574 >> v1:172.21.15.26:6790/0 conn(0x7f8e8c138ff0 legacy=0x7f8e8c1cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e929a1640 1 -- 172.21.15.26:0/2050135574 reap_dead start 2024-09-16T14:55:19.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e929a1640 1 -- 172.21.15.26:0/2050135574 reap_dead start 2024-09-16T14:55:19.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e94404640 1 -- 172.21.15.26:0/2050135574 shutdown_connections 2024-09-16T14:55:19.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e94404640 1 -- 172.21.15.26:0/2050135574 >> 172.21.15.26:0/2050135574 conn(0x7f8e8c09fb50 msgr2=0x7f8e8c0a1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:19.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e94404640 1 -- 172.21.15.26:0/2050135574 shutdown_connections 2024-09-16T14:55:19.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.521+0000 7f8e94404640 1 -- 172.21.15.26:0/2050135574 wait complete. 2024-09-16T14:55:19.557 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:55:19.557 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:55:19.557 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:55:19.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 10 monclient: get_monmap_and_config 2024-09-16T14:55:19.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 10 monclient: build_initial_monmap 2024-09-16T14:55:19.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 10 monclient: monmap: 2024-09-16T14:55:19.686 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:19.686 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.686 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:19.666392+0000 2024-09-16T14:55:19.686 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:19.666392+0000 2024-09-16T14:55:19.686 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:19.686 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.686 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:19.686 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:19.686 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:19.687 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 1 Processor -- start 2024-09-16T14:55:19.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 1 -- start start 2024-09-16T14:55:19.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 10 monclient: init 2024-09-16T14:55:19.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:19.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:55:19.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 10 monclient(hunting): picked mon.noname-c con 0x7feb8809a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:19.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 10 monclient(hunting): picked mon.noname-a con 0x7feb8809d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:19.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 10 monclient(hunting): picked mon.noname-b con 0x7feb8809b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:19.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8809dc60 con 0x7feb8809d1b0 2024-09-16T14:55:19.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb88099ae0 con 0x7feb8809a550 2024-09-16T14:55:19.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8809b8f0 con 0x7feb8809b040 2024-09-16T14:55:19.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:19.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 10 monclient(hunting): authenticate will time out at 1743.786191s 2024-09-16T14:55:19.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb7ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7feb8809d1b0 0x7feb88136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38812/0 (socket says 172.21.15.26:38812) 2024-09-16T14:55:19.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb7ffff640 1 -- 172.21.15.26:0/100344661 learned_addr learned my addr 172.21.15.26:0/100344661 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:19.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 1 -- 172.21.15.26:0/100344661 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951242744 0 0) 0x7feb8809dc60 con 0x7feb8809d1b0 2024-09-16T14:55:19.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient(hunting): my global_id is 254202 2024-09-16T14:55:19.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 1 -- 172.21.15.26:0/100344661 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb74003530 con 0x7feb8809d1b0 2024-09-16T14:55:19.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 1 -- 172.21.15.26:0/100344661 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011431334 0 0) 0x7feb8809b8f0 con 0x7feb8809b040 2024-09-16T14:55:19.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient(hunting): my global_id is 234410 2024-09-16T14:55:19.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 1 -- 172.21.15.26:0/100344661 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb8809dc60 con 0x7feb8809b040 2024-09-16T14:55:19.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 1 -- 172.21.15.26:0/100344661 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2196320448 0 0) 0x7feb8809dc60 con 0x7feb8809b040 2024-09-16T14:55:19.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 1 -- 172.21.15.26:0/100344661 >> v1:172.21.15.189:6789/0 conn(0x7feb8809a550 legacy=0x7feb8809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 1 -- 172.21.15.26:0/100344661 >> v1:172.21.15.26:6789/0 conn(0x7feb8809d1b0 legacy=0x7feb88136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:55:19.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 1 monclient: found mon.noname-b 2024-09-16T14:55:19.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:19.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:55:19.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 1 -- 172.21.15.26:0/100344661 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8809bbd0 con 0x7feb8809b040 2024-09-16T14:55:19.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient: _finish_auth 0 2024-09-16T14:55:19.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:55:19.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:19.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 1 -- 172.21.15.26:0/100344661 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb800019a0 con 0x7feb8809b040 2024-09-16T14:55:19.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:55:19.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient: dump: 2024-09-16T14:55:19.690 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.691 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.691 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.691 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.691 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.691 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.692 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.692 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.692 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.692 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 5 monclient: authenticate success, global_id 234410 2024-09-16T14:55:19.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:19.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 1 -- 172.21.15.26:0/100344661 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb80001b50 con 0x7feb8809b040 2024-09-16T14:55:19.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:19.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 1 -- 172.21.15.26:0/100344661 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb80001f20 con 0x7feb8809b040 2024-09-16T14:55:19.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:19.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 4 set_mon_vals no callback set 2024-09-16T14:55:19.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:19.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb86ffd640 10 monclient: dump: 2024-09-16T14:55:19.693 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.693 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.693 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.693 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.693 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.693 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.694 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.694 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.694 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.694 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 10 monclient: shutdown 2024-09-16T14:55:19.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:19.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 1 -- 172.21.15.26:0/100344661 >> v1:172.21.15.26:6790/0 conn(0x7feb8809b040 legacy=0x7feb8813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 1 -- 172.21.15.26:0/100344661 shutdown_connections 2024-09-16T14:55:19.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 1 -- 172.21.15.26:0/100344661 >> 172.21.15.26:0/100344661 conn(0x7feb8809fb50 msgr2=0x7feb8809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:19.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 1 -- 172.21.15.26:0/100344661 shutdown_connections 2024-09-16T14:55:19.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.661+0000 7feb8df8e640 1 -- 172.21.15.26:0/100344661 wait complete. 2024-09-16T14:55:19.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 10 monclient: build_initial_monmap 2024-09-16T14:55:19.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 10 monclient: monmap: 2024-09-16T14:55:19.695 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:19.695 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:19.695 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:19.695 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:19.695 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:19.695 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:19.696 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:19.696 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:19.696 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:19.696 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 1 Processor -- start 2024-09-16T14:55:19.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 1 -- start start 2024-09-16T14:55:19.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 10 monclient: init 2024-09-16T14:55:19.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:19.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:55:19.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 10 monclient(hunting): picked mon.noname-a con 0x7feb8809a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:19.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 10 monclient(hunting): picked mon.noname-c con 0x7feb8809b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:19.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 10 monclient(hunting): picked mon.noname-b con 0x7feb8809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:19.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb881367e0 con 0x7feb8809a550 2024-09-16T14:55:19.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb880a2430 con 0x7feb8809d1b0 2024-09-16T14:55:19.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8809a240 con 0x7feb8809b040 2024-09-16T14:55:19.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:19.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 10 monclient(hunting): authenticate will time out at 1743.789107s 2024-09-16T14:55:19.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb7ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7feb8809b040 0x7feb881cb840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47306/0 (socket says 172.21.15.26:47306) 2024-09-16T14:55:19.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb877fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7feb8809a550 0x7feb881301f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38814/0 (socket says 172.21.15.26:38814) 2024-09-16T14:55:19.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb7ffff640 1 -- 172.21.15.26:0/2324838312 learned_addr learned my addr 172.21.15.26:0/2324838312 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:19.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658810749 0 0) 0x7feb8809a240 con 0x7feb8809b040 2024-09-16T14:55:19.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): my global_id is 234413 2024-09-16T14:55:19.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb68003530 con 0x7feb8809b040 2024-09-16T14:55:19.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461768015 0 0) 0x7feb881367e0 con 0x7feb8809a550 2024-09-16T14:55:19.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): my global_id is 254205 2024-09-16T14:55:19.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb8809a240 con 0x7feb8809a550 2024-09-16T14:55:19.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 626473209 0 0) 0x7feb68003530 con 0x7feb8809b040 2024-09-16T14:55:19.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb881367e0 con 0x7feb8809b040 2024-09-16T14:55:19.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb70001a00 con 0x7feb8809b040 2024-09-16T14:55:19.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:55:19.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): dump: 2024-09-16T14:55:19.701 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.701 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.701 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.701 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.701 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.701 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.701 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.701 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.701 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.701 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:19.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4013665848 0 0) 0x7feb8809a240 con 0x7feb8809a550 2024-09-16T14:55:19.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb68003530 con 0x7feb8809a550 2024-09-16T14:55:19.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb780017c0 con 0x7feb8809a550 2024-09-16T14:55:19.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:19.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): dump: 2024-09-16T14:55:19.703 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.703 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.703 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.703 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.703 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.703 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.703 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.703 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.704 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.704 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333777607 0 0) 0x7feb880a2430 con 0x7feb8809d1b0 2024-09-16T14:55:19.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): my global_id is 264103 2024-09-16T14:55:19.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb8809a240 con 0x7feb8809d1b0 2024-09-16T14:55:19.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3381110942 0 0) 0x7feb881367e0 con 0x7feb8809b040 2024-09-16T14:55:19.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 >> v1:172.21.15.189:6789/0 conn(0x7feb8809d1b0 legacy=0x7feb881cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 >> v1:172.21.15.26:6789/0 conn(0x7feb8809a550 legacy=0x7feb881301f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:55:19.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 monclient: found mon.c 2024-09-16T14:55:19.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:19.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:19.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb88138140 con 0x7feb8809b040 2024-09-16T14:55:19.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient: _finish_auth 0 2024-09-16T14:55:19.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:55:19.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:19.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 5 monclient: authenticate success, global_id 234413 2024-09-16T14:55:19.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 10 monclient: _renew_subs 2024-09-16T14:55:19.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:19.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 1 -- 172.21.15.26:0/2324838312 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb881378a0 con 0x7feb8809b040 2024-09-16T14:55:19.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 10 monclient: _renew_subs 2024-09-16T14:55:19.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:19.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 1 -- 172.21.15.26:0/2324838312 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8813dd60 con 0x7feb8809b040 2024-09-16T14:55:19.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb700047f0 con 0x7feb8809b040 2024-09-16T14:55:19.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:19.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb70004b00 con 0x7feb8809b040 2024-09-16T14:55:19.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:19.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb867fc640 4 set_mon_vals no callback set 2024-09-16T14:55:19.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient: dump: 2024-09-16T14:55:19.707 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.708 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.708 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.708 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.708 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.708 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.708 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.708 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.708 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.708 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:19.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb8df8e640 1 -- 172.21.15.26:0/2324838312 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7feb6c006590 con 0x7feb6c002830 2024-09-16T14:55:19.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7feb7000f570 con 0x7feb8809b040 2024-09-16T14:55:19.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7feb70043d60 con 0x7feb8809b040 2024-09-16T14:55:19.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460790778 0 0) 0x7feb6c006590 con 0x7feb6c002830 2024-09-16T14:55:19.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient(hunting): my global_id is 234416 2024-09-16T14:55:19.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb881367e0 con 0x7feb6c002830 2024-09-16T14:55:19.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb87fff640 10 monclient: get_auth_request con 0x7feb680363f0 auth_method 0 2024-09-16T14:55:19.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1753181882 0 0) 0x7feb881367e0 con 0x7feb6c002830 2024-09-16T14:55:19.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb6c006590 con 0x7feb6c002830 2024-09-16T14:55:19.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb78003620 con 0x7feb6c002830 2024-09-16T14:55:19.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:19.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient: dump: 2024-09-16T14:55:19.710 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.710 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.710 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.710 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.710 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.710 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.710 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.710 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.710 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.710 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2937047765 0 0) 0x7feb6c006590 con 0x7feb6c002830 2024-09-16T14:55:19.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb6c00a0e0 con 0x7feb6c002830 2024-09-16T14:55:19.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7feb780057d0 con 0x7feb6c002830 2024-09-16T14:55:19.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:19.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.665+0000 7feb84ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:19.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.669+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 >> v1:172.21.15.26:6790/0 conn(0x7feb6c002830 legacy=0x7feb6c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.669+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 >> v1:172.21.15.26:6790/0 conn(0x7feb6c002830 legacy=0x7feb6c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:19.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb8df8e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:55:19.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb8df8e640 1 -- 172.21.15.26:0/2324838312 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7feb6c009da0 con 0x7feb6c0090c0 2024-09-16T14:55:19.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286379804 0 0) 0x7feb6c009da0 con 0x7feb6c0090c0 2024-09-16T14:55:19.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 10 monclient(hunting): my global_id is 234419 2024-09-16T14:55:19.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb6c006590 con 0x7feb6c0090c0 2024-09-16T14:55:19.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1618090092 0 0) 0x7feb6c006590 con 0x7feb6c0090c0 2024-09-16T14:55:19.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb6c009da0 con 0x7feb6c0090c0 2024-09-16T14:55:19.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb78003d70 con 0x7feb6c0090c0 2024-09-16T14:55:19.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:19.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 10 monclient: dump: 2024-09-16T14:55:19.713 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.713 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.713 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.713 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.713 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.713 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.714 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.714 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.714 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.714 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3228749024 0 0) 0x7feb6c009da0 con 0x7feb6c0090c0 2024-09-16T14:55:19.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7feb6c00c290 con 0x7feb6c0090c0 2024-09-16T14:55:19.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 1423485983) 0x7feb78004090 con 0x7feb6c0090c0 2024-09-16T14:55:19.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:55:19.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:19.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 >> v1:172.21.15.26:6790/0 conn(0x7feb6c0090c0 legacy=0x7feb6c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb84ff9640 1 -- 172.21.15.26:0/2324838312 >> v1:172.21.15.26:6790/0 conn(0x7feb6c0090c0 legacy=0x7feb6c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:19.715 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:55:19.715 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:55:19.715 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:55:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:55:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 120, 2024-09-16T14:55:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:55:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:55:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:55:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:55:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:55:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:55:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:55:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:55:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:55:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:55:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:55:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:55:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:55:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:55:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:55:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:55:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:55:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:55:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:55:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:55:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:55:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:55:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:55:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:55:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:55:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:55:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:55:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:55:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:55:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:55:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:55:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:55:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:55:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:55:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:55:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:55:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:55:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:55:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:55:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:55:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:55:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:55:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:55:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:55:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:55:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 7 2024-09-16T14:55:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:55:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:55:19.729 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:55:19.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb8df8e640 1 -- 172.21.15.26:0/2324838312 >> v1:172.21.15.189:6812/3196161582 conn(0x7feb680363f0 legacy=0x7feb680388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb8df8e640 10 monclient: shutdown 2024-09-16T14:55:19.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb8df8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:19.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb8df8e640 1 -- 172.21.15.26:0/2324838312 >> v1:172.21.15.26:6790/0 conn(0x7feb8809b040 legacy=0x7feb881cb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb87fff640 1 -- 172.21.15.26:0/2324838312 reap_dead start 2024-09-16T14:55:19.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb87fff640 1 -- 172.21.15.26:0/2324838312 reap_dead start 2024-09-16T14:55:19.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb8df8e640 1 -- 172.21.15.26:0/2324838312 shutdown_connections 2024-09-16T14:55:19.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb8df8e640 1 -- 172.21.15.26:0/2324838312 >> 172.21.15.26:0/2324838312 conn(0x7feb8809fb50 msgr2=0x7feb880a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:19.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb8df8e640 1 -- 172.21.15.26:0/2324838312 shutdown_connections 2024-09-16T14:55:19.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.673+0000 7feb8df8e640 1 -- 172.21.15.26:0/2324838312 wait complete. 2024-09-16T14:55:19.731 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:55:19.732 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:55:19.732 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:55:19.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 10 monclient: get_monmap_and_config 2024-09-16T14:55:19.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 10 monclient: build_initial_monmap 2024-09-16T14:55:19.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 10 monclient: monmap: 2024-09-16T14:55:19.862 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:19.862 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.862 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:19.862369+0000 2024-09-16T14:55:19.862 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:19.862369+0000 2024-09-16T14:55:19.862 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:19.862 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.862 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:19.863 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:19.863 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:19.863 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 1 Processor -- start 2024-09-16T14:55:19.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 1 -- start start 2024-09-16T14:55:19.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 10 monclient: init 2024-09-16T14:55:19.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:19.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:55:19.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 10 monclient(hunting): picked mon.noname-c con 0x7f171009e7a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:19.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 10 monclient(hunting): picked mon.noname-a con 0x7f171009cdc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:19.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 10 monclient(hunting): picked mon.noname-b con 0x7f171009d210 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:19.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171009aff0 con 0x7f171009cdc0 2024-09-16T14:55:19.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171009b2d0 con 0x7f171009e7a0 2024-09-16T14:55:19.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171009b5b0 con 0x7f171009d210 2024-09-16T14:55:19.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:19.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 10 monclient(hunting): authenticate will time out at 1743.982894s 2024-09-16T14:55:19.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170effd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f171009cdc0 0x7f17101370a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38826/0 (socket says 172.21.15.26:38826) 2024-09-16T14:55:19.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170effd640 1 -- 172.21.15.26:0/2114280283 learned_addr learned my addr 172.21.15.26:0/2114280283 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:19.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 1 -- 172.21.15.26:0/2114280283 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139881153 0 0) 0x7f171009b5b0 con 0x7f171009d210 2024-09-16T14:55:19.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient(hunting): my global_id is 234422 2024-09-16T14:55:19.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 1 -- 172.21.15.26:0/2114280283 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f8003530 con 0x7f171009d210 2024-09-16T14:55:19.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 1 -- 172.21.15.26:0/2114280283 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699012220 0 0) 0x7f171009aff0 con 0x7f171009cdc0 2024-09-16T14:55:19.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient(hunting): my global_id is 254208 2024-09-16T14:55:19.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 1 -- 172.21.15.26:0/2114280283 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f171009b5b0 con 0x7f171009cdc0 2024-09-16T14:55:19.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 1 -- 172.21.15.26:0/2114280283 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448286144 0 0) 0x7f171009b2d0 con 0x7f171009e7a0 2024-09-16T14:55:19.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient(hunting): my global_id is 264106 2024-09-16T14:55:19.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 1 -- 172.21.15.26:0/2114280283 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f171009aff0 con 0x7f171009e7a0 2024-09-16T14:55:19.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 1 -- 172.21.15.26:0/2114280283 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3964118071 0 0) 0x7f16f8003530 con 0x7f171009d210 2024-09-16T14:55:19.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 1 -- 172.21.15.26:0/2114280283 >> v1:172.21.15.189:6789/0 conn(0x7f171009e7a0 legacy=0x7f171009da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 1 -- 172.21.15.26:0/2114280283 >> v1:172.21.15.26:6789/0 conn(0x7f171009cdc0 legacy=0x7f17101370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:55:19.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 1 monclient: found mon.noname-b 2024-09-16T14:55:19.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:19.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:55:19.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 1 -- 172.21.15.26:0/2114280283 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171009b890 con 0x7f171009d210 2024-09-16T14:55:19.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:55:19.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:55:19.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:19.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 1 -- 172.21.15.26:0/2114280283 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f17080019a0 con 0x7f171009d210 2024-09-16T14:55:19.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 5 monclient: authenticate success, global_id 234422 2024-09-16T14:55:19.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:55:19.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f170e7fc640 10 monclient: dump: 2024-09-16T14:55:19.875 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.875 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.875 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.875 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.875 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.875 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.875 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.875 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.875 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.876 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.857+0000 7f1715dfa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:19.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f170e7fc640 1 -- 172.21.15.26:0/2114280283 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1708001c80 con 0x7f171009d210 2024-09-16T14:55:19.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f170e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:19.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f170e7fc640 1 -- 172.21.15.26:0/2114280283 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1708002010 con 0x7f171009d210 2024-09-16T14:55:19.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f170e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f170e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:19.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f170e7fc640 10 monclient: dump: 2024-09-16T14:55:19.876 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.876 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.876 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.876 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.876 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.877 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.877 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.877 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.877 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.877 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:19.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 4 set_mon_vals no callback set 2024-09-16T14:55:19.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 monclient: shutdown 2024-09-16T14:55:19.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:19.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 1 -- 172.21.15.26:0/2114280283 >> v1:172.21.15.26:6790/0 conn(0x7f171009d210 legacy=0x7f171013a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 1 -- 172.21.15.26:0/2114280283 shutdown_connections 2024-09-16T14:55:19.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 1 -- 172.21.15.26:0/2114280283 >> 172.21.15.26:0/2114280283 conn(0x7f171009fb50 msgr2=0x7f171009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:19.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 1 -- 172.21.15.26:0/2114280283 shutdown_connections 2024-09-16T14:55:19.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 1 -- 172.21.15.26:0/2114280283 wait complete. 2024-09-16T14:55:19.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 monclient: build_initial_monmap 2024-09-16T14:55:19.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 monclient: monmap: 2024-09-16T14:55:19.878 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:19.878 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:19.878 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:19.878 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:19.878 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:19.878 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:19.878 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:19.878 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:19.879 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:19.879 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 1 Processor -- start 2024-09-16T14:55:19.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 1 -- start start 2024-09-16T14:55:19.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 monclient: init 2024-09-16T14:55:19.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:19.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:55:19.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 monclient(hunting): picked mon.noname-c con 0x7f171009cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:19.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 monclient(hunting): picked mon.noname-a con 0x7f171009d210 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:19.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 monclient(hunting): picked mon.noname-b con 0x7f171009e7a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:19.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17101c8340 con 0x7f171009d210 2024-09-16T14:55:19.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17100a2110 con 0x7f171009e7a0 2024-09-16T14:55:19.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171009a240 con 0x7f171009cdc0 2024-09-16T14:55:19.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:19.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 monclient(hunting): authenticate will time out at 1743.986191s 2024-09-16T14:55:19.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f170f7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f171009cdc0 0x7f171012dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47334/0 (socket says 172.21.15.26:47334) 2024-09-16T14:55:19.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f170f7fe640 1 -- 172.21.15.26:0/290259236 learned_addr learned my addr 172.21.15.26:0/290259236 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:19.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917009308 0 0) 0x7f17101c8340 con 0x7f171009d210 2024-09-16T14:55:19.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): my global_id is 254211 2024-09-16T14:55:19.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16fc003530 con 0x7f171009d210 2024-09-16T14:55:19.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579351418 0 0) 0x7f171009a240 con 0x7f171009cdc0 2024-09-16T14:55:19.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): my global_id is 234425 2024-09-16T14:55:19.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17101c8340 con 0x7f171009cdc0 2024-09-16T14:55:19.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3220847345 0 0) 0x7f16fc003530 con 0x7f171009d210 2024-09-16T14:55:19.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f171009a240 con 0x7f171009d210 2024-09-16T14:55:19.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f17040018c0 con 0x7f171009d210 2024-09-16T14:55:19.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:55:19.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): dump: 2024-09-16T14:55:19.881 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.881 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.882 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.882 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.882 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.882 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.882 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.882 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.882 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.882 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:19.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3542260738 0 0) 0x7f17101c8340 con 0x7f171009cdc0 2024-09-16T14:55:19.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16fc003530 con 0x7f171009cdc0 2024-09-16T14:55:19.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1700001920 con 0x7f171009cdc0 2024-09-16T14:55:19.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:19.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): dump: 2024-09-16T14:55:19.883 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.883 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.883 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.883 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.883 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.883 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.883 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.883 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.883 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.883 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754867119 0 0) 0x7f17100a2110 con 0x7f171009e7a0 2024-09-16T14:55:19.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient(hunting): my global_id is 264109 2024-09-16T14:55:19.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17101c8340 con 0x7f171009e7a0 2024-09-16T14:55:19.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 678592121 0 0) 0x7f171009a240 con 0x7f171009d210 2024-09-16T14:55:19.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 >> v1:172.21.15.26:6790/0 conn(0x7f171009cdc0 legacy=0x7f171012dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 >> v1:172.21.15.189:6789/0 conn(0x7f171009e7a0 legacy=0x7f17101cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:55:19.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 monclient: found mon.a 2024-09-16T14:55:19.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:19.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:19.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17100a2660 con 0x7f171009d210 2024-09-16T14:55:19.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient: _finish_auth 0 2024-09-16T14:55:19.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:55:19.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:19.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 5 monclient: authenticate success, global_id 254211 2024-09-16T14:55:19.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 monclient: _renew_subs 2024-09-16T14:55:19.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:19.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 1 -- 172.21.15.26:0/290259236 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171011ce20 con 0x7f171009d210 2024-09-16T14:55:19.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 monclient: _renew_subs 2024-09-16T14:55:19.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:19.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 1 -- 172.21.15.26:0/290259236 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1710132b00 con 0x7f171009d210 2024-09-16T14:55:19.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17040047f0 con 0x7f171009d210 2024-09-16T14:55:19.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:19.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1704004b80 con 0x7f171009d210 2024-09-16T14:55:19.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:19.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f170dffb640 4 set_mon_vals no callback set 2024-09-16T14:55:19.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f16f7fff640 10 monclient: dump: 2024-09-16T14:55:19.886 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.886 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.886 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.886 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.886 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.886 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.887 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.887 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.887 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.887 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:19.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.861+0000 7f1715dfa640 1 -- 172.21.15.26:0/290259236 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f16f0006590 con 0x7f16f0002830 2024-09-16T14:55:19.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1704004ce0 con 0x7f171009d210 2024-09-16T14:55:19.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f1704043c00 con 0x7f171009d210 2024-09-16T14:55:19.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688326799 0 0) 0x7f16f0006590 con 0x7f16f0002830 2024-09-16T14:55:19.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 10 monclient(hunting): my global_id is 264112 2024-09-16T14:55:19.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f171009a240 con 0x7f16f0002830 2024-09-16T14:55:19.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f170ffff640 10 monclient: get_auth_request con 0x7f16fc036470 auth_method 0 2024-09-16T14:55:19.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 263748583 0 0) 0x7f171009a240 con 0x7f16f0002830 2024-09-16T14:55:19.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16f0006590 con 0x7f16f0002830 2024-09-16T14:55:19.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1700003a80 con 0x7f16f0002830 2024-09-16T14:55:19.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:55:19.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 10 monclient: dump: 2024-09-16T14:55:19.888 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.888 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.889 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.889 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.889 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.889 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.889 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.889 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.889 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.889 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4061773761 0 0) 0x7f16f0006590 con 0x7f16f0002830 2024-09-16T14:55:19.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16f000a0e0 con 0x7f16f0002830 2024-09-16T14:55:19.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f17000057d0 con 0x7f16f0002830 2024-09-16T14:55:19.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:19.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:19.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 >> v1:172.21.15.189:6789/0 conn(0x7f16f0002830 legacy=0x7f16f0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.865+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 >> v1:172.21.15.189:6789/0 conn(0x7f16f0002830 legacy=0x7f16f0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:19.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f1715dfa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:55:19.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f1715dfa640 1 -- 172.21.15.26:0/290259236 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f16f0009d80 con 0x7f16f00090a0 2024-09-16T14:55:19.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635844091 0 0) 0x7f16f0009d80 con 0x7f16f00090a0 2024-09-16T14:55:19.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:55:19.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:19.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:19.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 10 monclient(hunting): my global_id is 264115 2024-09-16T14:55:19.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f0006590 con 0x7f16f00090a0 2024-09-16T14:55:19.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 379934159 0 0) 0x7f16f0006590 con 0x7f16f00090a0 2024-09-16T14:55:19.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16f0009d80 con 0x7f16f00090a0 2024-09-16T14:55:19.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f17000041a0 con 0x7f16f00090a0 2024-09-16T14:55:19.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:19.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:55:19.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 10 monclient: dump: 2024-09-16T14:55:19.892 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:19.892 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:19.892 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.893 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:19.893 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:19.893 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:19.893 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:19.893 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:19.893 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:19.893 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:19.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2905051435 0 0) 0x7f16f0009d80 con 0x7f16f00090a0 2024-09-16T14:55:19.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f16f000c290 con 0x7f16f00090a0 2024-09-16T14:55:19.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 405992841) 0x7f1700005940 con 0x7f16f00090a0 2024-09-16T14:55:19.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:55:19.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:19.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 >> v1:172.21.15.189:6789/0 conn(0x7f16f00090a0 legacy=0x7f16f0009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f16f7fff640 1 -- 172.21.15.26:0/290259236 >> v1:172.21.15.189:6789/0 conn(0x7f16f00090a0 legacy=0x7f16f0009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:19.894 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:55:19.894 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:55:19.894 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:55:19.894 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:55:19.894 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 120, 2024-09-16T14:55:19.894 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:55:19.894 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:55:19.894 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:55:19.895 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:55:19.895 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:19.895 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:55:19.895 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:19.895 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:55:19.895 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:55:19.895 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:19.895 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:19.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:19.895 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:19.895 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:19.896 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:19.896 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:19.896 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:19.896 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:19.896 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:19.896 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:19.896 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:55:19.896 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:55:19.896 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:19.896 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:19.896 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:19.897 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:19.897 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:19.897 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:19.897 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:19.897 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:19.897 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:19.897 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:19.897 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.897 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:55:19.897 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:55:19.897 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:55:19.897 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:55:19.897 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:55:19.898 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:55:19.898 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:19.898 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:19.898 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:55:19.898 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:55:19.898 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:55:19.898 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:55:19.898 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:55:19.898 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:55:19.899 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:55:19.899 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:19.899 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:55:19.899 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:19.899 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:19.899 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:19.899 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:19.899 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:19.899 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:19.899 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:19.899 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:19.899 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:19.900 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:19.900 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:19.900 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:55:19.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:55:19.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.900 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:55:19.900 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:55:19.900 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:19.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:19.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.900 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:19.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:55:19.901 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:19.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.901 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:55:19.901 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:55:19.901 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:19.901 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:19.901 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:19.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.902 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:55:19.902 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:55:19.902 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:19.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:19.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.902 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:19.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:55:19.902 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:19.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.902 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:55:19.903 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:55:19.903 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:19.903 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:19.903 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:19.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.903 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:55:19.903 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:55:19.903 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:19.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:19.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.903 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:19.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:55:19.903 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:19.904 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.904 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.904 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:55:19.904 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:55:19.904 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:19.904 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:19.904 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:19.904 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.904 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.905 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:55:19.905 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:55:19.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.905 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:19.905 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:19.905 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:19.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.906 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:19.906 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:55:19.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:19.906 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:19.906 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:19.906 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:19.906 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:19.906 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:19.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:19.906 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:55:19.906 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:55:19.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f1715dfa640 1 -- 172.21.15.26:0/290259236 >> v1:172.21.15.189:6812/3196161582 conn(0x7f16fc036470 legacy=0x7f16fc038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f1715dfa640 10 monclient: shutdown 2024-09-16T14:55:19.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f1715dfa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:19.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f1715dfa640 1 -- 172.21.15.26:0/290259236 >> v1:172.21.15.26:6789/0 conn(0x7f171009d210 legacy=0x7f17101cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:19.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f170ffff640 1 -- 172.21.15.26:0/290259236 reap_dead start 2024-09-16T14:55:19.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f170ffff640 1 -- 172.21.15.26:0/290259236 reap_dead start 2024-09-16T14:55:19.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f1715dfa640 1 -- 172.21.15.26:0/290259236 shutdown_connections 2024-09-16T14:55:19.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f1715dfa640 1 -- 172.21.15.26:0/290259236 >> 172.21.15.26:0/290259236 conn(0x7f171009fb50 msgr2=0x7f17100a1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:19.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f1715dfa640 1 -- 172.21.15.26:0/290259236 shutdown_connections 2024-09-16T14:55:19.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:19.873+0000 7f1715dfa640 1 -- 172.21.15.26:0/290259236 wait complete. 2024-09-16T14:55:19.909 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:55:19.909 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:55:19.909 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:55:20.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.017+0000 7fcb9d92a640 10 monclient: get_monmap_and_config 2024-09-16T14:55:20.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.017+0000 7fcb9d92a640 10 monclient: build_initial_monmap 2024-09-16T14:55:20.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.017+0000 7fcb9d92a640 10 monclient: monmap: 2024-09-16T14:55:20.023 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:20.023 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:20.023 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:20.024819+0000 2024-09-16T14:55:20.023 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:20.024819+0000 2024-09-16T14:55:20.023 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:20.023 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:20.023 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:20.024 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:20.024 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:20.024 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:20.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 1 Processor -- start 2024-09-16T14:55:20.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 1 -- start start 2024-09-16T14:55:20.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 10 monclient: init 2024-09-16T14:55:20.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:20.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:55:20.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 10 monclient(hunting): picked mon.noname-c con 0x7fcb980a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:20.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 10 monclient(hunting): picked mon.noname-b con 0x7fcb980a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:20.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 10 monclient(hunting): picked mon.noname-a con 0x7fcb980a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:20.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb980a3250 con 0x7fcb980a2800 2024-09-16T14:55:20.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9809f600 con 0x7fcb980a1d10 2024-09-16T14:55:20.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb980a3830 con 0x7fcb980a0000 2024-09-16T14:55:20.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:20.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 10 monclient(hunting): authenticate will time out at 1744.144986s 2024-09-16T14:55:20.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb967fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fcb980a0000 0x7fcb98139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47344/0 (socket says 172.21.15.26:47344) 2024-09-16T14:55:20.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb967fc640 1 -- 172.21.15.26:0/25420713 learned_addr learned my addr 172.21.15.26:0/25420713 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:20.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 1 -- 172.21.15.26:0/25420713 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516848209 0 0) 0x7fcb980a3250 con 0x7fcb980a2800 2024-09-16T14:55:20.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:55:20.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:20.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:20.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient(hunting): my global_id is 254214 2024-09-16T14:55:20.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 1 -- 172.21.15.26:0/25420713 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb80003530 con 0x7fcb980a2800 2024-09-16T14:55:20.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 1 -- 172.21.15.26:0/25420713 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257238468 0 0) 0x7fcb980a3830 con 0x7fcb980a0000 2024-09-16T14:55:20.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:55:20.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:20.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:20.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient(hunting): my global_id is 234428 2024-09-16T14:55:20.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 1 -- 172.21.15.26:0/25420713 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb980a3250 con 0x7fcb980a0000 2024-09-16T14:55:20.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 1 -- 172.21.15.26:0/25420713 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3377295402 0 0) 0x7fcb80003530 con 0x7fcb980a2800 2024-09-16T14:55:20.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 1 -- 172.21.15.26:0/25420713 >> v1:172.21.15.26:6790/0 conn(0x7fcb980a0000 legacy=0x7fcb98139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:20.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 1 -- 172.21.15.26:0/25420713 >> v1:172.21.15.189:6789/0 conn(0x7fcb980a1d10 legacy=0x7fcb980a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:20.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:55:20.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 1 monclient: found mon.noname-a 2024-09-16T14:55:20.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:20.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:55:20.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 1 -- 172.21.15.26:0/25420713 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9809ed60 con 0x7fcb980a2800 2024-09-16T14:55:20.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient: _finish_auth 0 2024-09-16T14:55:20.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:55:20.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:20.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 1 -- 172.21.15.26:0/25420713 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcb900019a0 con 0x7fcb980a2800 2024-09-16T14:55:20.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:20.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:55:20.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient: dump: 2024-09-16T14:55:20.030 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:20.030 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:20.030 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:20.031 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:20.031 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:20.031 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:20.031 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:20.031 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:20.031 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:20.031 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:20.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 5 monclient: authenticate success, global_id 254214 2024-09-16T14:55:20.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:20.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 1 -- 172.21.15.26:0/25420713 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb90001c80 con 0x7fcb980a2800 2024-09-16T14:55:20.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:20.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 1 -- 172.21.15.26:0/25420713 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcb90002010 con 0x7fcb980a2800 2024-09-16T14:55:20.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:20.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:20.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb95ffb640 10 monclient: dump: 2024-09-16T14:55:20.032 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:20.032 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:20.032 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:20.032 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:20.032 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:20.032 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:20.032 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:20.032 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:20.032 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:20.032 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:20.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:20.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 4 set_mon_vals no callback set 2024-09-16T14:55:20.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 10 monclient: shutdown 2024-09-16T14:55:20.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:20.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 1 -- 172.21.15.26:0/25420713 >> v1:172.21.15.26:6789/0 conn(0x7fcb980a2800 legacy=0x7fcb9813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:20.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 1 -- 172.21.15.26:0/25420713 shutdown_connections 2024-09-16T14:55:20.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 1 -- 172.21.15.26:0/25420713 >> 172.21.15.26:0/25420713 conn(0x7fcb980a7f30 msgr2=0x7fcb980a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:20.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 1 -- 172.21.15.26:0/25420713 shutdown_connections 2024-09-16T14:55:20.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 1 -- 172.21.15.26:0/25420713 wait complete. 2024-09-16T14:55:20.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 10 monclient: build_initial_monmap 2024-09-16T14:55:20.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.021+0000 7fcb9d92a640 10 monclient: monmap: 2024-09-16T14:55:20.033 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:20.033 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:20.033 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:20.033 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:20.033 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:20.034 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:20.034 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:20.034 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:20.034 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:20.034 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:20.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 1 Processor -- start 2024-09-16T14:55:20.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 1 -- start start 2024-09-16T14:55:20.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 10 monclient: init 2024-09-16T14:55:20.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:20.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:55:20.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 10 monclient(hunting): picked mon.noname-c con 0x7fcb980a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:20.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 10 monclient(hunting): picked mon.noname-a con 0x7fcb980a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:20.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 10 monclient(hunting): picked mon.noname-b con 0x7fcb980a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:20.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9809d770 con 0x7fcb980a1d10 2024-09-16T14:55:20.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9809e500 con 0x7fcb980a2800 2024-09-16T14:55:20.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb967fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fcb980a1d10 0x7fcb98130a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38838/0 (socket says 172.21.15.26:38838) 2024-09-16T14:55:20.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb96ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fcb980a0000 0x7fcb98130370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47358/0 (socket says 172.21.15.26:47358) 2024-09-16T14:55:20.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb967fc640 1 -- 172.21.15.26:0/60900440 learned_addr learned my addr 172.21.15.26:0/60900440 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:20.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 1 -- 172.21.15.26:0/60900440 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb98131190 con 0x7fcb980a0000 2024-09-16T14:55:20.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:20.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 10 monclient(hunting): authenticate will time out at 1744.148923s 2024-09-16T14:55:20.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190561870 0 0) 0x7fcb98131190 con 0x7fcb980a0000 2024-09-16T14:55:20.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:55:20.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:20.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:20.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): my global_id is 234431 2024-09-16T14:55:20.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb84003530 con 0x7fcb980a0000 2024-09-16T14:55:20.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249493105 0 0) 0x7fcb9809d770 con 0x7fcb980a1d10 2024-09-16T14:55:20.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:55:20.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:20.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:20.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): my global_id is 254217 2024-09-16T14:55:20.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb98131190 con 0x7fcb980a1d10 2024-09-16T14:55:20.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2841190706 0 0) 0x7fcb98131190 con 0x7fcb980a1d10 2024-09-16T14:55:20.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb9809d770 con 0x7fcb980a1d10 2024-09-16T14:55:20.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1811434429 0 0) 0x7fcb84003530 con 0x7fcb980a0000 2024-09-16T14:55:20.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb98131190 con 0x7fcb980a0000 2024-09-16T14:55:20.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcb8c0018c0 con 0x7fcb980a1d10 2024-09-16T14:55:20.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:20.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:55:20.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): dump: 2024-09-16T14:55:20.040 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:20.040 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:20.041 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:20.041 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:20.041 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:20.041 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:20.041 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:20.041 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:20.041 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:20.041 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:20.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:20.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcb88001770 con 0x7fcb980a0000 2024-09-16T14:55:20.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:20.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:20.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): dump: 2024-09-16T14:55:20.042 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:20.042 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:20.042 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:20.042 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:20.042 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:20.042 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:20.042 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:20.042 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:20.042 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:20.042 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:20.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406417342 0 0) 0x7fcb9809e500 con 0x7fcb980a2800 2024-09-16T14:55:20.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:55:20.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:20.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:20.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient(hunting): my global_id is 264121 2024-09-16T14:55:20.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb84003530 con 0x7fcb980a2800 2024-09-16T14:55:20.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3956197813 0 0) 0x7fcb9809d770 con 0x7fcb980a1d10 2024-09-16T14:55:20.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 >> v1:172.21.15.26:6790/0 conn(0x7fcb980a0000 legacy=0x7fcb98130370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:20.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 >> v1:172.21.15.189:6789/0 conn(0x7fcb980a2800 legacy=0x7fcb981d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:20.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:55:20.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 monclient: found mon.a 2024-09-16T14:55:20.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:20.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:20.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb980a67b0 con 0x7fcb980a1d10 2024-09-16T14:55:20.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient: _finish_auth 0 2024-09-16T14:55:20.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:55:20.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:20.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 5 monclient: authenticate success, global_id 254217 2024-09-16T14:55:20.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 10 monclient: _renew_subs 2024-09-16T14:55:20.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:20.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 1 -- 172.21.15.26:0/60900440 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb98132f50 con 0x7fcb980a1d10 2024-09-16T14:55:20.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 10 monclient: _renew_subs 2024-09-16T14:55:20.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:20.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 1 -- 172.21.15.26:0/60900440 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9809d1e0 con 0x7fcb980a1d10 2024-09-16T14:55:20.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb8c0047f0 con 0x7fcb980a1d10 2024-09-16T14:55:20.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:20.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb957fa640 4 set_mon_vals no callback set 2024-09-16T14:55:20.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcb8c004b80 con 0x7fcb980a1d10 2024-09-16T14:55:20.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:20.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:20.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 10 monclient: dump: 2024-09-16T14:55:20.046 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:20.046 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:20.046 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:20.046 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:20.046 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:20.046 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:20.046 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:20.046 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:20.046 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:20.046 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:20.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:20.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:20.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:20.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fcb8c004d60 con 0x7fcb980a1d10 2024-09-16T14:55:20.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb9d92a640 1 -- 172.21.15.26:0/60900440 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb78005050 con 0x7fcb980a1d10 2024-09-16T14:55:20.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fcb8c043f00 con 0x7fcb980a1d10 2024-09-16T14:55:20.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.025+0000 7fcb96ffd640 10 monclient: get_auth_request con 0x7fcb84036470 auth_method 0 2024-09-16T14:55:20.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.029+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fcb8c014840 con 0x7fcb980a1d10 2024-09-16T14:55:20.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.029+0000 7fcb7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:20.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.029+0000 7fcb7ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:20.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.249+0000 7fcb9d92a640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:55:20.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.249+0000 7fcb9d92a640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:55:20.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.249+0000 7fcb9d92a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:20.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.249+0000 7fcb9d92a640 1 -- 172.21.15.26:0/60900440 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fcb78005b10 con 0x7fcb980a1d10 2024-09-16T14:55:20.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.253+0000 7fcb7ffff640 1 -- 172.21.15.26:0/60900440 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fcb8c012cb0 con 0x7fcb980a1d10 2024-09-16T14:55:20.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.253+0000 7fcb7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:55:20.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.253+0000 7fcb7ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:20.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.253+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.253+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=11,kv=11,logm=51} crc {auth=1177784754,config=692619927,health=2512707069,kv=1867599010,logm=692760311}) 2024-09-16T14:55:20.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.253+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=1,health=12,kv=11,logm=48} crc {auth=1847166326,config=1228700967,health=4067136255,kv=1867599010,logm=2440762639}) 2024-09-16T14:55:20.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.253+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.253+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=11,kv=11,logm=51} crc {auth=1177784754,config=692619927,health=2512707069,kv=1867599010,logm=692760311}) 2024-09-16T14:55:20.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.253+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=10,kv=9,logm=53} crc {auth=757638009,config=692619927,health=2293501443,kv=655417215,logm=1140231559}) 2024-09-16T14:55:20.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.257+0000 7fcb9d92a640 1 -- 172.21.15.26:0/60900440 >> v1:172.21.15.189:6812/3196161582 conn(0x7fcb84036470 legacy=0x7fcb84038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:20.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.257+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.257+0000 7fcb9d92a640 10 monclient: shutdown 2024-09-16T14:55:20.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.257+0000 7fcb9d92a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:20.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.257+0000 7fcb9d92a640 1 -- 172.21.15.26:0/60900440 >> v1:172.21.15.26:6789/0 conn(0x7fcb980a1d10 legacy=0x7fcb98130a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:20.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.257+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3201208126}) 2024-09-16T14:55:20.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.257+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2572764793}) 2024-09-16T14:55:20.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.257+0000 7fcb9d92a640 1 -- 172.21.15.26:0/60900440 shutdown_connections 2024-09-16T14:55:20.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.257+0000 7fcb9d92a640 1 -- 172.21.15.26:0/60900440 >> 172.21.15.26:0/60900440 conn(0x7fcb980a7f30 msgr2=0x7fcb9813e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:20.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.257+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.257+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3201208126}) 2024-09-16T14:55:20.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.257+0000 7fcb9d92a640 1 -- 172.21.15.26:0/60900440 shutdown_connections 2024-09-16T14:55:20.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:20.257+0000 7fcb9d92a640 1 -- 172.21.15.26:0/60900440 wait complete. 2024-09-16T14:55:20.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.257+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1187216621}) 2024-09-16T14:55:20.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.261+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.261+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=139995299}) 2024-09-16T14:55:20.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.261+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3289412213}) 2024-09-16T14:55:20.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.261+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.261+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=139995299}) 2024-09-16T14:55:20.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.261+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=314964830}) 2024-09-16T14:55:20.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.261+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.261+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3493824022}) 2024-09-16T14:55:20.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.261+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1892567268}) 2024-09-16T14:55:20.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.261+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.261+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3493824022}) 2024-09-16T14:55:20.269 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.261+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=229647989}) 2024-09-16T14:55:20.269 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.265+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.269 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.265+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3924944338}) 2024-09-16T14:55:20.269 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.265+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1019992085}) 2024-09-16T14:55:20.269 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.265+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.269 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.265+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3924944338}) 2024-09-16T14:55:20.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.265+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4288937250}) 2024-09-16T14:55:20.271 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.265+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.271 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.265+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1910665265}) 2024-09-16T14:55:20.271 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.265+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=690326821}) 2024-09-16T14:55:20.271 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.265+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.265+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1910665265}) 2024-09-16T14:55:20.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.269+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2088805446}) 2024-09-16T14:55:20.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.269+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.269+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1814708034}) 2024-09-16T14:55:20.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.269+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1770422947}) 2024-09-16T14:55:20.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.269+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.269+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1814708034}) 2024-09-16T14:55:20.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.269+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=402352360}) 2024-09-16T14:55:20.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.273+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.273+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2022899235}) 2024-09-16T14:55:20.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.273+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408095480}) 2024-09-16T14:55:20.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.273+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.273+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2022899235}) 2024-09-16T14:55:20.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.273+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1242043864}) 2024-09-16T14:55:20.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.273+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.273+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3706854455}) 2024-09-16T14:55:20.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.273+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2316612920}) 2024-09-16T14:55:20.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.273+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.273+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3706854455}) 2024-09-16T14:55:20.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.273+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4075381504}) 2024-09-16T14:55:20.279 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:55:20.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.277+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.277+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3342370577}) 2024-09-16T14:55:20.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.277+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1267821283}) 2024-09-16T14:55:20.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.277+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.281 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.277+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3342370577}) 2024-09-16T14:55:20.281 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.277+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=377984074}) 2024-09-16T14:55:20.282 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.277+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.282 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.277+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2319040711}) 2024-09-16T14:55:20.282 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.277+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159788537}) 2024-09-16T14:55:20.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.277+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.277+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2319040711}) 2024-09-16T14:55:20.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.277+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3357417886}) 2024-09-16T14:55:20.284 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.281+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.284 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.281+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=278194813}) 2024-09-16T14:55:20.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.281+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=320772467}) 2024-09-16T14:55:20.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.281+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.281+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=278194813}) 2024-09-16T14:55:20.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.281+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2900617469}) 2024-09-16T14:55:20.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.281+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.281+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835647629}) 2024-09-16T14:55:20.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.281+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3195731884}) 2024-09-16T14:55:20.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.281+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.281+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835647629}) 2024-09-16T14:55:20.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.281+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=568306476}) 2024-09-16T14:55:20.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.285+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.285+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2178935910}) 2024-09-16T14:55:20.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.285+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1711579937}) 2024-09-16T14:55:20.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.285+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.285+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2178935910}) 2024-09-16T14:55:20.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.285+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3245310785}) 2024-09-16T14:55:20.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.285+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.285+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=867955721}) 2024-09-16T14:55:20.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.285+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456692973}) 2024-09-16T14:55:20.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.285+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.285+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=867955721}) 2024-09-16T14:55:20.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.285+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3818070794}) 2024-09-16T14:55:20.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.289+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.289+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=590625271}) 2024-09-16T14:55:20.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.289+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3081166108}) 2024-09-16T14:55:20.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.289+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.289+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=590625271}) 2024-09-16T14:55:20.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.289+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=684876481}) 2024-09-16T14:55:20.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.289+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.289+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975761678}) 2024-09-16T14:55:20.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.289+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1127313226}) 2024-09-16T14:55:20.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.289+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.289+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975761678}) 2024-09-16T14:55:20.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.289+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3312062234}) 2024-09-16T14:55:20.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.293+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.293+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=595794472}) 2024-09-16T14:55:20.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.293+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419067165}) 2024-09-16T14:55:20.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.293+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.293+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=595794472}) 2024-09-16T14:55:20.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.293+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667226375}) 2024-09-16T14:55:20.299 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.293+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.299 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.293+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3463706392}) 2024-09-16T14:55:20.299 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.293+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=663250822}) 2024-09-16T14:55:20.299 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.293+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.299 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.293+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3463706392}) 2024-09-16T14:55:20.299 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.293+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1305197491}) 2024-09-16T14:55:20.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.297+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.297+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1815732833}) 2024-09-16T14:55:20.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.297+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094386276}) 2024-09-16T14:55:20.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.297+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.297+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1815732833}) 2024-09-16T14:55:20.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.297+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3081337645}) 2024-09-16T14:55:20.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.297+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.297+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3479231417}) 2024-09-16T14:55:20.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.297+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248147931}) 2024-09-16T14:55:20.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.297+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.297+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3479231417}) 2024-09-16T14:55:20.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.297+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1025438850}) 2024-09-16T14:55:20.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.301+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.301+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3053475186}) 2024-09-16T14:55:20.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.301+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3387312387}) 2024-09-16T14:55:20.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.301+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.301+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3053475186}) 2024-09-16T14:55:20.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.301+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1947230380}) 2024-09-16T14:55:20.306 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.301+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.306 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.301+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1904518912}) 2024-09-16T14:55:20.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.301+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306317210}) 2024-09-16T14:55:20.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.301+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.301+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1904518912}) 2024-09-16T14:55:20.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.301+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3360635912}) 2024-09-16T14:55:20.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.305+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.305+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1251437931}) 2024-09-16T14:55:20.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.305+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846772487}) 2024-09-16T14:55:20.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.305+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.305+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1251437931}) 2024-09-16T14:55:20.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.305+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=989713121}) 2024-09-16T14:55:20.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.305+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.305+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2637300471}) 2024-09-16T14:55:20.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.305+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185961712}) 2024-09-16T14:55:20.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.305+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.305+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2637300471}) 2024-09-16T14:55:20.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.305+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=962721723}) 2024-09-16T14:55:20.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.309+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.309+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2892784459}) 2024-09-16T14:55:20.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.309+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=961821667}) 2024-09-16T14:55:20.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.309+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.314 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.309+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2892784459}) 2024-09-16T14:55:20.314 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.309+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4142301071}) 2024-09-16T14:55:20.315 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.309+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.309+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2214584512}) 2024-09-16T14:55:20.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.309+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657148212}) 2024-09-16T14:55:20.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.309+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.309+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2214584512}) 2024-09-16T14:55:20.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.309+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3898251885}) 2024-09-16T14:55:20.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.313+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.313+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3543045839}) 2024-09-16T14:55:20.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.313+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3289260701}) 2024-09-16T14:55:20.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.313+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.313+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3543045839}) 2024-09-16T14:55:20.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.313+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2581706152}) 2024-09-16T14:55:20.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.313+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.313+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4288205269}) 2024-09-16T14:55:20.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.313+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=443164461}) 2024-09-16T14:55:20.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.313+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.313+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4288205269}) 2024-09-16T14:55:20.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.313+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=145030659}) 2024-09-16T14:55:20.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.317+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.317+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=461163780}) 2024-09-16T14:55:20.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.317+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=829253767}) 2024-09-16T14:55:20.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.317+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.317+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=461163780}) 2024-09-16T14:55:20.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.317+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1239402064}) 2024-09-16T14:55:20.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.317+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.317+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2962579962}) 2024-09-16T14:55:20.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.317+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3295852127}) 2024-09-16T14:55:20.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.317+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.317+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2962579962}) 2024-09-16T14:55:20.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.317+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286656080}) 2024-09-16T14:55:20.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.321+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.321+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3044187920}) 2024-09-16T14:55:20.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.321+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3654157502}) 2024-09-16T14:55:20.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.321+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.321+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3044187920}) 2024-09-16T14:55:20.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.321+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2218603283}) 2024-09-16T14:55:20.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.321+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.321+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149722983}) 2024-09-16T14:55:20.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.321+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263547365}) 2024-09-16T14:55:20.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.321+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.321+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149722983}) 2024-09-16T14:55:20.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.321+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=293558538}) 2024-09-16T14:55:20.328 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.325+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.328 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.325+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2116019310}) 2024-09-16T14:55:20.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.325+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3265698682}) 2024-09-16T14:55:20.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.325+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.325+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2116019310}) 2024-09-16T14:55:20.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.325+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3182053931}) 2024-09-16T14:55:20.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.325+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.325+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1765108747}) 2024-09-16T14:55:20.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.325+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3054210705}) 2024-09-16T14:55:20.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.325+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.325+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1765108747}) 2024-09-16T14:55:20.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.325+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=728208067}) 2024-09-16T14:55:20.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.329+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.329+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=162870880}) 2024-09-16T14:55:20.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.329+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879847881}) 2024-09-16T14:55:20.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.329+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.329+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=162870880}) 2024-09-16T14:55:20.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.329+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2353500220}) 2024-09-16T14:55:20.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.329+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.329+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2548469565}) 2024-09-16T14:55:20.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.329+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=311062095}) 2024-09-16T14:55:20.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.329+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.329+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2548469565}) 2024-09-16T14:55:20.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.329+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2344071734}) 2024-09-16T14:55:20.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=453932022}) 2024-09-16T14:55:20.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1314909161}) 2024-09-16T14:55:20.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=453932022}) 2024-09-16T14:55:20.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1523865043}) 2024-09-16T14:55:20.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=592229221}) 2024-09-16T14:55:20.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1111905378}) 2024-09-16T14:55:20.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=592229221}) 2024-09-16T14:55:20.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588961352}) 2024-09-16T14:55:20.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4141853039}) 2024-09-16T14:55:20.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2321082422}) 2024-09-16T14:55:20.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4141853039}) 2024-09-16T14:55:20.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.333+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2188210292}) 2024-09-16T14:55:20.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.337+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.337+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3966017869}) 2024-09-16T14:55:20.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.337+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1687264706}) 2024-09-16T14:55:20.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.337+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.337+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3966017869}) 2024-09-16T14:55:20.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.337+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1175238370}) 2024-09-16T14:55:20.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.337+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.337+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3522654372}) 2024-09-16T14:55:20.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.337+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=684749684}) 2024-09-16T14:55:20.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.337+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.337+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3522654372}) 2024-09-16T14:55:20.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.337+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3345903256}) 2024-09-16T14:55:20.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.341+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.341+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=99043286}) 2024-09-16T14:55:20.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.341+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2788323028}) 2024-09-16T14:55:20.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.341+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.341+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=99043286}) 2024-09-16T14:55:20.346 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.341+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1820208996}) 2024-09-16T14:55:20.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.341+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.341+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3107737296}) 2024-09-16T14:55:20.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.341+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1730907346}) 2024-09-16T14:55:20.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.341+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.341+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3107737296}) 2024-09-16T14:55:20.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.341+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2725784963}) 2024-09-16T14:55:20.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.345+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.345+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3644887706}) 2024-09-16T14:55:20.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.345+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3354036387}) 2024-09-16T14:55:20.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.345+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.345+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3644887706}) 2024-09-16T14:55:20.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.345+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2581571346}) 2024-09-16T14:55:20.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.345+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.351 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.345+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=535064234}) 2024-09-16T14:55:20.351 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.345+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3734368546}) 2024-09-16T14:55:20.351 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.345+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.351 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.345+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=535064234}) 2024-09-16T14:55:20.351 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.345+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4168486704}) 2024-09-16T14:55:20.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.349+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.353 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.349+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3192074843}) 2024-09-16T14:55:20.353 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.349+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2677505164}) 2024-09-16T14:55:20.353 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.349+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.353 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.349+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3192074843}) 2024-09-16T14:55:20.353 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.349+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=45336763}) 2024-09-16T14:55:20.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.349+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.349+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3766205144}) 2024-09-16T14:55:20.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.349+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1470573081}) 2024-09-16T14:55:20.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.349+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.349+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3766205144}) 2024-09-16T14:55:20.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.349+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1466126689}) 2024-09-16T14:55:20.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.353+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.353+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2000665879}) 2024-09-16T14:55:20.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.353+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526526553}) 2024-09-16T14:55:20.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.353+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.353+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2000665879}) 2024-09-16T14:55:20.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.353+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1364818070}) 2024-09-16T14:55:20.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.353+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.353+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1554675195}) 2024-09-16T14:55:20.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.353+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=37443680}) 2024-09-16T14:55:20.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.353+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.353+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1554675195}) 2024-09-16T14:55:20.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.353+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=563002934}) 2024-09-16T14:55:20.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.357+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.357+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1387941640}) 2024-09-16T14:55:20.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.357+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168125135}) 2024-09-16T14:55:20.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.357+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.361 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.357+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1387941640}) 2024-09-16T14:55:20.361 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.357+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833041744}) 2024-09-16T14:55:20.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.357+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.357+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=971463692}) 2024-09-16T14:55:20.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.357+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752986948}) 2024-09-16T14:55:20.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.357+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.357+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=971463692}) 2024-09-16T14:55:20.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.357+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3446400636}) 2024-09-16T14:55:20.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.361+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.361+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466377831}) 2024-09-16T14:55:20.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.361+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3440129264}) 2024-09-16T14:55:20.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.361+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.361+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466377831}) 2024-09-16T14:55:20.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.361+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4225469255}) 2024-09-16T14:55:20.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.361+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.361+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1750267315}) 2024-09-16T14:55:20.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.361+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182604140}) 2024-09-16T14:55:20.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.361+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.361+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1750267315}) 2024-09-16T14:55:20.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.361+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=836282491}) 2024-09-16T14:55:20.367 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.365+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.365+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3206979386}) 2024-09-16T14:55:20.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.365+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=556356425}) 2024-09-16T14:55:20.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.365+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.365+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3206979386}) 2024-09-16T14:55:20.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.365+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873311348}) 2024-09-16T14:55:20.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.365+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.365+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1296518958}) 2024-09-16T14:55:20.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.365+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956921710}) 2024-09-16T14:55:20.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.365+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.365+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1296518958}) 2024-09-16T14:55:20.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.365+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=434515257}) 2024-09-16T14:55:20.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.365+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.365+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3376804886}) 2024-09-16T14:55:20.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.369+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1007927746}) 2024-09-16T14:55:20.372 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.369+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.372 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.369+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3376804886}) 2024-09-16T14:55:20.372 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.369+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4055543612}) 2024-09-16T14:55:20.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.369+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.369+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1833789071}) 2024-09-16T14:55:20.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.369+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2583321241}) 2024-09-16T14:55:20.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.369+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.369+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1833789071}) 2024-09-16T14:55:20.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.369+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3453903644}) 2024-09-16T14:55:20.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.369+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.369+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3494833457}) 2024-09-16T14:55:20.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.369+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2547389830}) 2024-09-16T14:55:20.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.369+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.369+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3494833457}) 2024-09-16T14:55:20.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.369+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2135844554}) 2024-09-16T14:55:20.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.373+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.373+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2850685159}) 2024-09-16T14:55:20.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.373+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1933883575}) 2024-09-16T14:55:20.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.373+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.373+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2850685159}) 2024-09-16T14:55:20.378 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.373+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=599145423}) 2024-09-16T14:55:20.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.373+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.373+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2161054420}) 2024-09-16T14:55:20.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.373+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=336736264}) 2024-09-16T14:55:20.380 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.373+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.380 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.373+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2161054420}) 2024-09-16T14:55:20.380 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.373+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=551460167}) 2024-09-16T14:55:20.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.377+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.377+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=706281931}) 2024-09-16T14:55:20.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.377+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237630335}) 2024-09-16T14:55:20.382 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.377+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.382 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.377+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=706281931}) 2024-09-16T14:55:20.382 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.377+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=829941249}) 2024-09-16T14:55:20.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.377+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.377+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2355610528}) 2024-09-16T14:55:20.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.377+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1849000465}) 2024-09-16T14:55:20.384 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.377+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.384 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.377+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2355610528}) 2024-09-16T14:55:20.384 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.377+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3094619215}) 2024-09-16T14:55:20.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.381+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.381+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=234855258}) 2024-09-16T14:55:20.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.381+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412221997}) 2024-09-16T14:55:20.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.381+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.386 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.381+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=234855258}) 2024-09-16T14:55:20.386 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.381+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3519856446}) 2024-09-16T14:55:20.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.381+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.381+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3815604166}) 2024-09-16T14:55:20.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.381+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=503162274}) 2024-09-16T14:55:20.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.381+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.381+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3815604166}) 2024-09-16T14:55:20.388 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.381+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2752823462}) 2024-09-16T14:55:20.388 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.385+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.385+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1492262235}) 2024-09-16T14:55:20.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.385+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2705030690}) 2024-09-16T14:55:20.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.385+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.385+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1492262235}) 2024-09-16T14:55:20.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.385+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2556453330}) 2024-09-16T14:55:20.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.385+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.385+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1987587259}) 2024-09-16T14:55:20.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.385+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3267392403}) 2024-09-16T14:55:20.391 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.385+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.391 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.385+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1987587259}) 2024-09-16T14:55:20.391 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.385+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1624284139}) 2024-09-16T14:55:20.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1744194643}) 2024-09-16T14:55:20.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2724360415}) 2024-09-16T14:55:20.393 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.393 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1744194643}) 2024-09-16T14:55:20.393 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=536435414}) 2024-09-16T14:55:20.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2088060894}) 2024-09-16T14:55:20.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=523046536}) 2024-09-16T14:55:20.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2088060894}) 2024-09-16T14:55:20.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1461114295}) 2024-09-16T14:55:20.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1184937817}) 2024-09-16T14:55:20.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2511594312}) 2024-09-16T14:55:20.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1184937817}) 2024-09-16T14:55:20.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=347772085}) 2024-09-16T14:55:20.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3950307344}) 2024-09-16T14:55:20.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562884922}) 2024-09-16T14:55:20.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3950307344}) 2024-09-16T14:55:20.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2273671601}) 2024-09-16T14:55:20.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2544162946}) 2024-09-16T14:55:20.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502989926}) 2024-09-16T14:55:20.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2544162946}) 2024-09-16T14:55:20.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4103710987}) 2024-09-16T14:55:20.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2633066932}) 2024-09-16T14:55:20.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=695955673}) 2024-09-16T14:55:20.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2633066932}) 2024-09-16T14:55:20.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1570813761}) 2024-09-16T14:55:20.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=548020905}) 2024-09-16T14:55:20.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3052362098}) 2024-09-16T14:55:20.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=548020905}) 2024-09-16T14:55:20.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=142034964}) 2024-09-16T14:55:20.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3964016187}) 2024-09-16T14:55:20.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3216628348}) 2024-09-16T14:55:20.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3964016187}) 2024-09-16T14:55:20.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1686300364}) 2024-09-16T14:55:20.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=88,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1} crc {logm=2940494089,mdsmap=1531151339,mgr=2683218624,mgr_command_descs=794974419,mgr_metadata=2109538692}) 2024-09-16T14:55:20.409 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=87,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=1734309855,mdsmap=1531151339,mgr=3253761504,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=55007578}) 2024-09-16T14:55:20.409 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.409 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=88,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1} crc {logm=2940494089,mdsmap=1531151339,mgr=2683218624,mgr_command_descs=794974419,mgr_metadata=2109538692}) 2024-09-16T14:55:20.409 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=1966040893,mdsmap=1531151339,mgr=1291789651}) 2024-09-16T14:55:20.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.417+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.417+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=49} crc {mgrstat=879179494,mon_config_key=2512756450,monmap=970648238,osd_metadata=2332200480,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=2918096420}) 2024-09-16T14:55:20.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.417+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgrstat=879179494,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=586171670}) 2024-09-16T14:55:20.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.421+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:20.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.421+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=49} crc {mgrstat=879179494,mon_config_key=2512756450,monmap=970648238,osd_metadata=2332200480,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=2918096420}) 2024-09-16T14:55:20.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:20.421+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgrstat=879179494,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1643806442}) 2024-09-16T14:55:21.281 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:55:21.281 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:55:21.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 10 monclient: get_monmap_and_config 2024-09-16T14:55:21.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 10 monclient: build_initial_monmap 2024-09-16T14:55:21.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 10 monclient: monmap: 2024-09-16T14:55:21.413 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:21.413 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.413 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:21.414577+0000 2024-09-16T14:55:21.413 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:21.414577+0000 2024-09-16T14:55:21.413 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:21.413 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.413 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:21.413 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:21.414 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:21.414 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 1 Processor -- start 2024-09-16T14:55:21.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 1 -- start start 2024-09-16T14:55:21.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 10 monclient: init 2024-09-16T14:55:21.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:21.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:55:21.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 10 monclient(hunting): picked mon.noname-c con 0x7fde1809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:21.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 10 monclient(hunting): picked mon.noname-a con 0x7fde18132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:21.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 10 monclient(hunting): picked mon.noname-b con 0x7fde18138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:21.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1809d260 con 0x7fde18132650 2024-09-16T14:55:21.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde18132a80 con 0x7fde1809cb80 2024-09-16T14:55:21.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1809cf60 con 0x7fde18138ff0 2024-09-16T14:55:21.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:21.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 10 monclient(hunting): authenticate will time out at 1745.534662s 2024-09-16T14:55:21.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1759e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fde18138ff0 0x7fde1813b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47368/0 (socket says 172.21.15.26:47368) 2024-09-16T14:55:21.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1759e640 1 -- 172.21.15.26:0/2480925929 learned_addr learned my addr 172.21.15.26:0/2480925929 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:21.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1659c640 1 --1- 172.21.15.26:0/2480925929 >> v1:172.21.15.26:6789/0 conn(0x7fde18132650 0x7fde181378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38854/0 (socket says 172.21.15.26:38854) 2024-09-16T14:55:21.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 1 -- 172.21.15.26:0/2480925929 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327970285 0 0) 0x7fde1809cf60 con 0x7fde18138ff0 2024-09-16T14:55:21.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient(hunting): my global_id is 234434 2024-09-16T14:55:21.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 1 -- 172.21.15.26:0/2480925929 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde04003530 con 0x7fde18138ff0 2024-09-16T14:55:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 1 -- 172.21.15.26:0/2480925929 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337295760 0 0) 0x7fde1809d260 con 0x7fde18132650 2024-09-16T14:55:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient(hunting): my global_id is 254220 2024-09-16T14:55:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 1 -- 172.21.15.26:0/2480925929 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde1809cf60 con 0x7fde18132650 2024-09-16T14:55:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 1 -- 172.21.15.26:0/2480925929 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2327228758 0 0) 0x7fde04003530 con 0x7fde18138ff0 2024-09-16T14:55:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 1 -- 172.21.15.26:0/2480925929 >> v1:172.21.15.189:6789/0 conn(0x7fde1809cb80 legacy=0x7fde180717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 1 -- 172.21.15.26:0/2480925929 >> v1:172.21.15.26:6789/0 conn(0x7fde18132650 legacy=0x7fde181378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:55:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 1 monclient: found mon.noname-b 2024-09-16T14:55:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:55:21.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 1 -- 172.21.15.26:0/2480925929 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1809f2d0 con 0x7fde18138ff0 2024-09-16T14:55:21.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient: _finish_auth 0 2024-09-16T14:55:21.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:55:21.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:21.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 1 -- 172.21.15.26:0/2480925929 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fde10001660 con 0x7fde18138ff0 2024-09-16T14:55:21.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 5 monclient: authenticate success, global_id 234434 2024-09-16T14:55:21.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:21.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:55:21.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient: dump: 2024-09-16T14:55:21.421 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.421 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.421 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.421 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.421 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.421 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.421 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.421 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.421 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.421 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:21.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 1 -- 172.21.15.26:0/2480925929 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde10001840 con 0x7fde18138ff0 2024-09-16T14:55:21.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:21.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 1 -- 172.21.15.26:0/2480925929 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fde10001c10 con 0x7fde18138ff0 2024-09-16T14:55:21.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:21.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde1d1c8640 4 set_mon_vals no callback set 2024-09-16T14:55:21.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:21.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.409+0000 7fde15d9b640 10 monclient: dump: 2024-09-16T14:55:21.422 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.422 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.422 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.422 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.422 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.423 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient: shutdown 2024-09-16T14:55:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 1 -- 172.21.15.26:0/2480925929 >> v1:172.21.15.26:6790/0 conn(0x7fde18138ff0 legacy=0x7fde1813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 1 -- 172.21.15.26:0/2480925929 shutdown_connections 2024-09-16T14:55:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 1 -- 172.21.15.26:0/2480925929 >> 172.21.15.26:0/2480925929 conn(0x7fde1809fb50 msgr2=0x7fde1809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 1 -- 172.21.15.26:0/2480925929 shutdown_connections 2024-09-16T14:55:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 1 -- 172.21.15.26:0/2480925929 wait complete. 2024-09-16T14:55:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient: build_initial_monmap 2024-09-16T14:55:21.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient: monmap: 2024-09-16T14:55:21.424 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:21.424 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:21.424 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:21.424 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:21.424 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:21.424 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:21.424 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:21.424 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:21.424 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:21.425 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 1 Processor -- start 2024-09-16T14:55:21.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 1 -- start start 2024-09-16T14:55:21.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient: init 2024-09-16T14:55:21.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:21.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:55:21.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient(hunting): picked mon.noname-c con 0x7fde1809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:21.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient(hunting): picked mon.noname-b con 0x7fde18132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:21.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient(hunting): picked mon.noname-a con 0x7fde18138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:21.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1809c290 con 0x7fde18138ff0 2024-09-16T14:55:21.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde18070f30 con 0x7fde18132650 2024-09-16T14:55:21.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1813c600 con 0x7fde1809cb80 2024-09-16T14:55:21.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:21.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient(hunting): authenticate will time out at 1745.537915s 2024-09-16T14:55:21.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde16d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fde1809cb80 0x7fde1809a7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47384/0 (socket says 172.21.15.26:47384) 2024-09-16T14:55:21.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde16d9d640 1 -- 172.21.15.26:0/790282372 learned_addr learned my addr 172.21.15.26:0/790282372 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:21.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255423351 0 0) 0x7fde1809c290 con 0x7fde18138ff0 2024-09-16T14:55:21.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): my global_id is 254223 2024-09-16T14:55:21.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddf8003530 con 0x7fde18138ff0 2024-09-16T14:55:21.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434830742 0 0) 0x7fde1813c600 con 0x7fde1809cb80 2024-09-16T14:55:21.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): my global_id is 234437 2024-09-16T14:55:21.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde1809c290 con 0x7fde1809cb80 2024-09-16T14:55:21.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2216416844 0 0) 0x7fde1809c290 con 0x7fde1809cb80 2024-09-16T14:55:21.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde1813c600 con 0x7fde1809cb80 2024-09-16T14:55:21.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fde080018a0 con 0x7fde1809cb80 2024-09-16T14:55:21.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:55:21.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): dump: 2024-09-16T14:55:21.431 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.431 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.431 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.431 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.432 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.432 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.432 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.432 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.432 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.432 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:21.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2254178549 0 0) 0x7fddf8003530 con 0x7fde18138ff0 2024-09-16T14:55:21.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde1809c290 con 0x7fde18138ff0 2024-09-16T14:55:21.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fde10001b50 con 0x7fde18138ff0 2024-09-16T14:55:21.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:21.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): dump: 2024-09-16T14:55:21.433 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.433 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.433 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.433 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.433 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.433 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.433 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.433 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.433 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.433 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656510258 0 0) 0x7fde18070f30 con 0x7fde18132650 2024-09-16T14:55:21.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient(hunting): my global_id is 264127 2024-09-16T14:55:21.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddf8003530 con 0x7fde18132650 2024-09-16T14:55:21.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3978977436 0 0) 0x7fde1813c600 con 0x7fde1809cb80 2024-09-16T14:55:21.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 >> v1:172.21.15.189:6789/0 conn(0x7fde18132650 legacy=0x7fde1809aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 >> v1:172.21.15.26:6789/0 conn(0x7fde18138ff0 legacy=0x7fde181d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:55:21.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 monclient: found mon.c 2024-09-16T14:55:21.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:21.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:21.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1809a240 con 0x7fde1809cb80 2024-09-16T14:55:21.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:55:21.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:55:21.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:21.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 5 monclient: authenticate success, global_id 234437 2024-09-16T14:55:21.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient: _renew_subs 2024-09-16T14:55:21.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:21.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 1 -- 172.21.15.26:0/790282372 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1806e0e0 con 0x7fde1809cb80 2024-09-16T14:55:21.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient: _renew_subs 2024-09-16T14:55:21.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:21.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 1 -- 172.21.15.26:0/790282372 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1806d510 con 0x7fde1809cb80 2024-09-16T14:55:21.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde08003660 con 0x7fde1809cb80 2024-09-16T14:55:21.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:21.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fde080039f0 con 0x7fde1809cb80 2024-09-16T14:55:21.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:21.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 10 monclient: dump: 2024-09-16T14:55:21.436 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.436 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.436 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.436 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.436 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.436 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.436 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.436 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.436 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.436 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1559a640 4 set_mon_vals no callback set 2024-09-16T14:55:21.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:21.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:21.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:21.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde1d1c8640 1 -- 172.21.15.26:0/790282372 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1809bde0 con 0x7fde1809cb80 2024-09-16T14:55:21.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.413+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fde08003bd0 con 0x7fde1809cb80 2024-09-16T14:55:21.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.417+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fde08043e60 con 0x7fde1809cb80 2024-09-16T14:55:21.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.417+0000 7fde1659c640 10 monclient: get_auth_request con 0x7fddf80363a0 auth_method 0 2024-09-16T14:55:21.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.421+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fde080147a0 con 0x7fde1809cb80 2024-09-16T14:55:21.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.421+0000 7fde0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:21.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.421+0000 7fde0f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:21.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.641+0000 7fde1d1c8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:55:21.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.641+0000 7fde1d1c8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:55:21.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.641+0000 7fde1d1c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:21.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.641+0000 7fde1d1c8640 1 -- 172.21.15.26:0/790282372 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fde1809c960 con 0x7fde1809cb80 2024-09-16T14:55:21.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.641+0000 7fde0f7fe640 1 -- 172.21.15.26:0/790282372 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 3781153881) 0x7fde08014390 con 0x7fde1809cb80 2024-09-16T14:55:21.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.641+0000 7fde0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:55:21.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.641+0000 7fde0f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:21.648 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":120,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:55:21.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.645+0000 7fde1d1c8640 1 -- 172.21.15.26:0/790282372 >> v1:172.21.15.189:6812/3196161582 conn(0x7fddf80363a0 legacy=0x7fddf8038860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.645+0000 7fde1d1c8640 10 monclient: shutdown 2024-09-16T14:55:21.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.645+0000 7fde1d1c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:21.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.645+0000 7fde1d1c8640 1 -- 172.21.15.26:0/790282372 >> v1:172.21.15.26:6790/0 conn(0x7fde1809cb80 legacy=0x7fde1809a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.645+0000 7fde1d1c8640 1 -- 172.21.15.26:0/790282372 shutdown_connections 2024-09-16T14:55:21.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.645+0000 7fde1d1c8640 1 -- 172.21.15.26:0/790282372 >> 172.21.15.26:0/790282372 conn(0x7fde1809fb50 msgr2=0x7fde181388c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:21.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.645+0000 7fde1d1c8640 1 -- 172.21.15.26:0/790282372 shutdown_connections 2024-09-16T14:55:21.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.645+0000 7fde1d1c8640 1 -- 172.21.15.26:0/790282372 wait complete. 2024-09-16T14:55:21.664 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:55:21.664 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:55:21.664 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:55:21.664 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:55:21.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.785+0000 7fbb0df43640 10 monclient: get_monmap_and_config 2024-09-16T14:55:21.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.785+0000 7fbb0df43640 10 monclient: build_initial_monmap 2024-09-16T14:55:21.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.785+0000 7fbb0df43640 10 monclient: monmap: 2024-09-16T14:55:21.790 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:21.791 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.791 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:21.792121+0000 2024-09-16T14:55:21.791 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:21.792121+0000 2024-09-16T14:55:21.791 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:21.791 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.791 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:21.791 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:21.791 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:21.791 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.785+0000 7fbb0df43640 1 Processor -- start 2024-09-16T14:55:21.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.785+0000 7fbb0df43640 1 -- start start 2024-09-16T14:55:21.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.785+0000 7fbb0df43640 10 monclient: init 2024-09-16T14:55:21.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:21.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:55:21.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient(hunting): picked mon.noname-a con 0x7fbb0809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:21.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient(hunting): picked mon.noname-c con 0x7fbb08132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:21.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient(hunting): picked mon.noname-b con 0x7fbb08138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:21.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0809d260 con 0x7fbb0809cb80 2024-09-16T14:55:21.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb08132a80 con 0x7fbb08132650 2024-09-16T14:55:21.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0809cf60 con 0x7fbb08138ff0 2024-09-16T14:55:21.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:21.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient(hunting): authenticate will time out at 1745.912189s 2024-09-16T14:55:21.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb07fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbb08138ff0 0x7fbb0813b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47394/0 (socket says 172.21.15.26:47394) 2024-09-16T14:55:21.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb07fff640 1 -- 172.21.15.26:0/3778799823 learned_addr learned my addr 172.21.15.26:0/3778799823 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:21.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 1 -- 172.21.15.26:0/3778799823 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943570028 0 0) 0x7fbb0809d260 con 0x7fbb0809cb80 2024-09-16T14:55:21.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient(hunting): my global_id is 254226 2024-09-16T14:55:21.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 1 -- 172.21.15.26:0/3778799823 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaf0003530 con 0x7fbb0809cb80 2024-09-16T14:55:21.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 1 -- 172.21.15.26:0/3778799823 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692895879 0 0) 0x7fbb0809cf60 con 0x7fbb08138ff0 2024-09-16T14:55:21.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient(hunting): my global_id is 234440 2024-09-16T14:55:21.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 1 -- 172.21.15.26:0/3778799823 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb0809d260 con 0x7fbb08138ff0 2024-09-16T14:55:21.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 1 -- 172.21.15.26:0/3778799823 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3764630926 0 0) 0x7fbb0809d260 con 0x7fbb08138ff0 2024-09-16T14:55:21.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 1 -- 172.21.15.26:0/3778799823 >> v1:172.21.15.189:6789/0 conn(0x7fbb08132650 legacy=0x7fbb081378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 1 -- 172.21.15.26:0/3778799823 >> v1:172.21.15.26:6789/0 conn(0x7fbb0809cb80 legacy=0x7fbb080717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient: _finish_hunting 0 2024-09-16T14:55:21.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 1 monclient: found mon.noname-b 2024-09-16T14:55:21.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:21.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:55:21.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 1 -- 172.21.15.26:0/3778799823 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0809f2d0 con 0x7fbb08138ff0 2024-09-16T14:55:21.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient: _finish_auth 0 2024-09-16T14:55:21.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient: _check_auth_tickets 2024-09-16T14:55:21.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:21.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 1 -- 172.21.15.26:0/3778799823 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbaf80015c0 con 0x7fbb08138ff0 2024-09-16T14:55:21.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:55:21.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient: dump: 2024-09-16T14:55:21.803 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.803 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.803 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.803 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.803 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.803 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.803 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.803 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.803 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.803 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 5 monclient: authenticate success, global_id 234440 2024-09-16T14:55:21.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:21.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 1 -- 172.21.15.26:0/3778799823 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbaf8001760 con 0x7fbb08138ff0 2024-09-16T14:55:21.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:21.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 1 -- 172.21.15.26:0/3778799823 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbaf8001b30 con 0x7fbb08138ff0 2024-09-16T14:55:21.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:21.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb067fc640 10 monclient: dump: 2024-09-16T14:55:21.804 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.804 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.804 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.804 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.804 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.804 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.804 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.805 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.805 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.805 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:21.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 4 set_mon_vals no callback set 2024-09-16T14:55:21.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient: shutdown 2024-09-16T14:55:21.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:21.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 1 -- 172.21.15.26:0/3778799823 >> v1:172.21.15.26:6790/0 conn(0x7fbb08138ff0 legacy=0x7fbb0813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 1 -- 172.21.15.26:0/3778799823 shutdown_connections 2024-09-16T14:55:21.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 1 -- 172.21.15.26:0/3778799823 >> 172.21.15.26:0/3778799823 conn(0x7fbb0809fb50 msgr2=0x7fbb0809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:21.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 1 -- 172.21.15.26:0/3778799823 shutdown_connections 2024-09-16T14:55:21.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 1 -- 172.21.15.26:0/3778799823 wait complete. 2024-09-16T14:55:21.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient: build_initial_monmap 2024-09-16T14:55:21.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient: monmap: 2024-09-16T14:55:21.806 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:21.806 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:21.806 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:21.806 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:21.806 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:21.806 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:21.806 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:21.806 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:21.807 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:21.807 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 1 Processor -- start 2024-09-16T14:55:21.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 1 -- start start 2024-09-16T14:55:21.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient: init 2024-09-16T14:55:21.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:21.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:55:21.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient(hunting): picked mon.noname-b con 0x7fbb0809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:21.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient(hunting): picked mon.noname-c con 0x7fbb08132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:21.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 10 monclient(hunting): picked mon.noname-a con 0x7fbb08138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:21.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0809a240 con 0x7fbb08138ff0 2024-09-16T14:55:21.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.789+0000 7fbb0df43640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb08071460 con 0x7fbb0809cb80 2024-09-16T14:55:21.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbb0df43640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb080a2060 con 0x7fbb08132650 2024-09-16T14:55:21.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbb0df43640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:21.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbb0df43640 10 monclient(hunting): authenticate will time out at 1745.916032s 2024-09-16T14:55:21.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbb06ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbb08132650 0x7fbb081cbb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47402/0 (socket says 172.21.15.26:47402) 2024-09-16T14:55:21.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbb06ffd640 1 -- 172.21.15.26:0/2655794338 learned_addr learned my addr 172.21.15.26:0/2655794338 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:21.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533939023 0 0) 0x7fbb080a2060 con 0x7fbb08132650 2024-09-16T14:55:21.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): my global_id is 234443 2024-09-16T14:55:21.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaf4003530 con 0x7fbb08132650 2024-09-16T14:55:21.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642122346 0 0) 0x7fbb0809a240 con 0x7fbb08138ff0 2024-09-16T14:55:21.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): my global_id is 254229 2024-09-16T14:55:21.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb080a2060 con 0x7fbb08138ff0 2024-09-16T14:55:21.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1121514799 0 0) 0x7fbb080a2060 con 0x7fbb08138ff0 2024-09-16T14:55:21.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb0809a240 con 0x7fbb08138ff0 2024-09-16T14:55:21.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbaf8001320 con 0x7fbb08138ff0 2024-09-16T14:55:21.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:55:21.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): dump: 2024-09-16T14:55:21.810 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.810 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.810 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.810 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.810 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.810 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.810 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.810 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.810 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.810 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:21.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1451137051 0 0) 0x7fbaf4003530 con 0x7fbb08132650 2024-09-16T14:55:21.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb080a2060 con 0x7fbb08132650 2024-09-16T14:55:21.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbafc001860 con 0x7fbb08132650 2024-09-16T14:55:21.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:21.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): dump: 2024-09-16T14:55:21.811 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.811 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.811 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.811 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.811 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.811 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.811 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.811 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.811 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.811 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471089179 0 0) 0x7fbb08071460 con 0x7fbb0809cb80 2024-09-16T14:55:21.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): my global_id is 264133 2024-09-16T14:55:21.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaf4003530 con 0x7fbb0809cb80 2024-09-16T14:55:21.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2504424611 0 0) 0x7fbb0809a240 con 0x7fbb08138ff0 2024-09-16T14:55:21.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 >> v1:172.21.15.26:6790/0 conn(0x7fbb08132650 legacy=0x7fbb081cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 >> v1:172.21.15.189:6789/0 conn(0x7fbb0809cb80 legacy=0x7fbb08138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient: _finish_hunting 0 2024-09-16T14:55:21.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 monclient: found mon.a 2024-09-16T14:55:21.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:21.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:21.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb08071b50 con 0x7fbb08138ff0 2024-09-16T14:55:21.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient: _finish_auth 0 2024-09-16T14:55:21.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient: _check_auth_tickets 2024-09-16T14:55:21.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:21.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbb0df43640 5 monclient: authenticate success, global_id 254229 2024-09-16T14:55:21.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbb0df43640 10 monclient: _renew_subs 2024-09-16T14:55:21.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbb0df43640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:21.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbb0df43640 1 -- 172.21.15.26:0/2655794338 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0806f100 con 0x7fbb08138ff0 2024-09-16T14:55:21.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbb0df43640 10 monclient: _renew_subs 2024-09-16T14:55:21.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbb0df43640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:21.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbb0df43640 1 -- 172.21.15.26:0/2655794338 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb08071180 con 0x7fbb08138ff0 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbaf8001540 con 0x7fbb08138ff0 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbaf8004c40 con 0x7fbb08138ff0 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbb05ffb640 4 set_mon_vals no callback set 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient: dump: 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.814 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.815 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.815 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.815 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.815 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.815 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbb0df43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:21.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbaf800f6b0 con 0x7fbb08138ff0 2024-09-16T14:55:21.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbb0df43640 1 -- 172.21.15.26:0/2655794338 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbae8006590 con 0x7fbae8002830 2024-09-16T14:55:21.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fbaf8043c30 con 0x7fbb08138ff0 2024-09-16T14:55:21.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274759268 0 0) 0x7fbae8006590 con 0x7fbae8002830 2024-09-16T14:55:21.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient(hunting): my global_id is 254232 2024-09-16T14:55:21.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb0809a240 con 0x7fbae8002830 2024-09-16T14:55:21.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbb06ffd640 10 monclient: get_auth_request con 0x7fbaf4036540 auth_method 0 2024-09-16T14:55:21.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2968704401 0 0) 0x7fbb0809a240 con 0x7fbae8002830 2024-09-16T14:55:21.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbae8006590 con 0x7fbae8002830 2024-09-16T14:55:21.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbb00001a70 con 0x7fbae8002830 2024-09-16T14:55:21.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:21.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient: dump: 2024-09-16T14:55:21.816 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.816 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.816 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.817 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.817 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.817 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.817 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.817 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.817 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.817 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1144215688 0 0) 0x7fbae8006590 con 0x7fbae8002830 2024-09-16T14:55:21.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbae800a0e0 con 0x7fbae8002830 2024-09-16T14:55:21.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fbb00001e60 con 0x7fbae8002830 2024-09-16T14:55:21.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:21.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:21.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 >> v1:172.21.15.26:6789/0 conn(0x7fbae8002830 legacy=0x7fbae8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.793+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 >> v1:172.21.15.26:6789/0 conn(0x7fbae8002830 legacy=0x7fbae8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:21.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbb0df43640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:55:21.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbb0df43640 1 -- 172.21.15.26:0/2655794338 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbae8009da0 con 0x7fbae80090c0 2024-09-16T14:55:21.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358974726 0 0) 0x7fbae8009da0 con 0x7fbae80090c0 2024-09-16T14:55:21.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 10 monclient(hunting): my global_id is 254235 2024-09-16T14:55:21.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae8006590 con 0x7fbae80090c0 2024-09-16T14:55:21.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 602438093 0 0) 0x7fbae8006590 con 0x7fbae80090c0 2024-09-16T14:55:21.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbae8009da0 con 0x7fbae80090c0 2024-09-16T14:55:21.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbb00003d10 con 0x7fbae80090c0 2024-09-16T14:55:21.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:21.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 10 monclient: dump: 2024-09-16T14:55:21.820 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.820 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.820 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.820 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.820 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.821 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.821 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.821 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.821 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.821 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1623323428 0 0) 0x7fbae8009da0 con 0x7fbae80090c0 2024-09-16T14:55:21.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbae800c290 con 0x7fbae80090c0 2024-09-16T14:55:21.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 1229448188) 0x7fbb00004f30 con 0x7fbae80090c0 2024-09-16T14:55:21.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:55:21.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:21.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 >> v1:172.21.15.26:6789/0 conn(0x7fbae80090c0 legacy=0x7fbae8009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbaeffff640 1 -- 172.21.15.26:0/2655794338 >> v1:172.21.15.26:6789/0 conn(0x7fbae80090c0 legacy=0x7fbae8009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:21.822 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:55:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:55:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:55:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:55:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 120, 2024-09-16T14:55:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:55:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:55:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:55:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:55:21.822 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:55:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:55:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:55:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:21.823 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:55:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:55:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:21.824 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:55:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:55:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:55:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:55:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:55:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:55:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:55:21.825 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:55:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:55:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:55:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:55:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:55:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:55:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:55:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:21.826 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:55:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:55:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:55:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:55:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:21.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:55:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:55:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:55:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:55:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:55:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:55:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:55:21.829 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:55:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:55:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:55:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:55:21.830 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:55:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:55:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:55:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:55:21.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:55:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:55:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:21.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.833 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:55:21.833 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:55:21.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbb0df43640 1 -- 172.21.15.26:0/2655794338 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbaf4036540 legacy=0x7fbaf4038a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbb0df43640 10 monclient: shutdown 2024-09-16T14:55:21.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbb0df43640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:21.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbb0df43640 1 -- 172.21.15.26:0/2655794338 >> v1:172.21.15.26:6789/0 conn(0x7fbb08138ff0 legacy=0x7fbb081cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbb07fff640 1 -- 172.21.15.26:0/2655794338 reap_dead start 2024-09-16T14:55:21.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbb07fff640 1 -- 172.21.15.26:0/2655794338 reap_dead start 2024-09-16T14:55:21.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbb0df43640 1 -- 172.21.15.26:0/2655794338 shutdown_connections 2024-09-16T14:55:21.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbb0df43640 1 -- 172.21.15.26:0/2655794338 >> 172.21.15.26:0/2655794338 conn(0x7fbb0809fb50 msgr2=0x7fbb08131900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:21.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbb0df43640 1 -- 172.21.15.26:0/2655794338 shutdown_connections 2024-09-16T14:55:21.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.801+0000 7fbb0df43640 1 -- 172.21.15.26:0/2655794338 wait complete. 2024-09-16T14:55:21.835 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:55:21.835 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:55:21.835 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:55:21.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 10 monclient: get_monmap_and_config 2024-09-16T14:55:21.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 10 monclient: build_initial_monmap 2024-09-16T14:55:21.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 10 monclient: monmap: 2024-09-16T14:55:21.948 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:21.948 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.948 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:21.949841+0000 2024-09-16T14:55:21.948 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:21.949841+0000 2024-09-16T14:55:21.948 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:21.948 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.949 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:21.949 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:21.949 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:21.949 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 1 Processor -- start 2024-09-16T14:55:21.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 1 -- start start 2024-09-16T14:55:21.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 10 monclient: init 2024-09-16T14:55:21.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:21.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:55:21.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 10 monclient(hunting): picked mon.noname-c con 0x7f9a1809a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:21.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 10 monclient(hunting): picked mon.noname-a con 0x7f9a1809d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:21.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 10 monclient(hunting): picked mon.noname-b con 0x7f9a1809b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:21.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a1809dc60 con 0x7f9a1809d1b0 2024-09-16T14:55:21.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a18099ae0 con 0x7f9a1809a550 2024-09-16T14:55:21.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a1809b8f0 con 0x7f9a1809b040 2024-09-16T14:55:21.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:21.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 10 monclient(hunting): authenticate will time out at 1746.069863s 2024-09-16T14:55:21.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9a1809d1b0 0x7f9a18136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38904/0 (socket says 172.21.15.26:38904) 2024-09-16T14:55:21.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1659c640 1 -- 172.21.15.26:0/1698706794 learned_addr learned my addr 172.21.15.26:0/1698706794 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:21.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 1 -- 172.21.15.26:0/1698706794 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292597097 0 0) 0x7f9a1809dc60 con 0x7f9a1809d1b0 2024-09-16T14:55:21.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient(hunting): my global_id is 254238 2024-09-16T14:55:21.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 1 -- 172.21.15.26:0/1698706794 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a04003530 con 0x7f9a1809d1b0 2024-09-16T14:55:21.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 1 -- 172.21.15.26:0/1698706794 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612207797 0 0) 0x7f9a1809b8f0 con 0x7f9a1809b040 2024-09-16T14:55:21.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient(hunting): my global_id is 234446 2024-09-16T14:55:21.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 1 -- 172.21.15.26:0/1698706794 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a1809dc60 con 0x7f9a1809b040 2024-09-16T14:55:21.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 1 -- 172.21.15.26:0/1698706794 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1400187816 0 0) 0x7f9a04003530 con 0x7f9a1809d1b0 2024-09-16T14:55:21.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 1 -- 172.21.15.26:0/1698706794 >> v1:172.21.15.26:6790/0 conn(0x7f9a1809b040 legacy=0x7f9a1813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 1 -- 172.21.15.26:0/1698706794 >> v1:172.21.15.189:6789/0 conn(0x7f9a1809a550 legacy=0x7f9a1809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:55:21.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 1 monclient: found mon.noname-a 2024-09-16T14:55:21.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:21.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:55:21.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 1 -- 172.21.15.26:0/1698706794 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1809bbd0 con 0x7f9a1809d1b0 2024-09-16T14:55:21.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient: _finish_auth 0 2024-09-16T14:55:21.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:55:21.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:21.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 1 -- 172.21.15.26:0/1698706794 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9a00001620 con 0x7f9a1809d1b0 2024-09-16T14:55:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:55:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient: dump: 2024-09-16T14:55:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.961 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 5 monclient: authenticate success, global_id 254238 2024-09-16T14:55:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 1 -- 172.21.15.26:0/1698706794 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a000018c0 con 0x7f9a1809d1b0 2024-09-16T14:55:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 4 set_mon_vals no callback set 2024-09-16T14:55:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 1 -- 172.21.15.26:0/1698706794 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9a00001c50 con 0x7f9a1809d1b0 2024-09-16T14:55:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a15d9b640 10 monclient: dump: 2024-09-16T14:55:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.962 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 10 monclient: shutdown 2024-09-16T14:55:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 1 -- 172.21.15.26:0/1698706794 >> v1:172.21.15.26:6789/0 conn(0x7f9a1809d1b0 legacy=0x7f9a18136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 1 -- 172.21.15.26:0/1698706794 shutdown_connections 2024-09-16T14:55:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 1 -- 172.21.15.26:0/1698706794 >> 172.21.15.26:0/1698706794 conn(0x7f9a1809fb50 msgr2=0x7f9a1809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 1 -- 172.21.15.26:0/1698706794 shutdown_connections 2024-09-16T14:55:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.945+0000 7f9a1d0e4640 1 -- 172.21.15.26:0/1698706794 wait complete. 2024-09-16T14:55:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 10 monclient: build_initial_monmap 2024-09-16T14:55:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 10 monclient: monmap: 2024-09-16T14:55:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:21.964 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 1 Processor -- start 2024-09-16T14:55:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 1 -- start start 2024-09-16T14:55:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 10 monclient: init 2024-09-16T14:55:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:55:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 10 monclient(hunting): picked mon.noname-a con 0x7f9a1809a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 10 monclient(hunting): picked mon.noname-b con 0x7f9a1809b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 10 monclient(hunting): picked mon.noname-c con 0x7f9a1809d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a181a93f0 con 0x7f9a1809a550 2024-09-16T14:55:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a180a27f0 con 0x7f9a1809b040 2024-09-16T14:55:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a181b4440 con 0x7f9a1809d1b0 2024-09-16T14:55:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 10 monclient(hunting): authenticate will time out at 1746.073313s 2024-09-16T14:55:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1759e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9a1809d1b0 0x7f9a181d1a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47422/0 (socket says 172.21.15.26:47422) 2024-09-16T14:55:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1759e640 1 -- 172.21.15.26:0/1802187004 learned_addr learned my addr 172.21.15.26:0/1802187004 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604683437 0 0) 0x7f9a181b4440 con 0x7f9a1809d1b0 2024-09-16T14:55:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): my global_id is 234449 2024-09-16T14:55:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f8003530 con 0x7f9a1809d1b0 2024-09-16T14:55:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334848129 0 0) 0x7f9a181a93f0 con 0x7f9a1809a550 2024-09-16T14:55:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): my global_id is 254241 2024-09-16T14:55:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a181b4440 con 0x7f9a1809a550 2024-09-16T14:55:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1220186290 0 0) 0x7f99f8003530 con 0x7f9a1809d1b0 2024-09-16T14:55:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a181a93f0 con 0x7f9a1809d1b0 2024-09-16T14:55:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9a100018e0 con 0x7f9a1809d1b0 2024-09-16T14:55:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:55:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): dump: 2024-09-16T14:55:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.967 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1633572636 0 0) 0x7f9a181b4440 con 0x7f9a1809a550 2024-09-16T14:55:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99f8003530 con 0x7f9a1809a550 2024-09-16T14:55:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773749327 0 0) 0x7f9a180a27f0 con 0x7f9a1809b040 2024-09-16T14:55:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): my global_id is 264139 2024-09-16T14:55:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a181b4440 con 0x7f9a1809b040 2024-09-16T14:55:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9a08001b80 con 0x7f9a1809a550 2024-09-16T14:55:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): dump: 2024-09-16T14:55:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.969 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 937715928 0 0) 0x7f9a181a93f0 con 0x7f9a1809d1b0 2024-09-16T14:55:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 >> v1:172.21.15.189:6789/0 conn(0x7f9a1809b040 legacy=0x7f9a181a5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 >> v1:172.21.15.26:6789/0 conn(0x7f9a1809a550 legacy=0x7f9a181a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:55:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 monclient: found mon.c 2024-09-16T14:55:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a181b5600 con 0x7f9a1809d1b0 2024-09-16T14:55:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:55:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:55:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 5 monclient: authenticate success, global_id 234449 2024-09-16T14:55:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 10 monclient: _renew_subs 2024-09-16T14:55:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 1 -- 172.21.15.26:0/1802187004 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a181a6650 con 0x7f9a1809d1b0 2024-09-16T14:55:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 10 monclient: _renew_subs 2024-09-16T14:55:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 1 -- 172.21.15.26:0/1802187004 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a181a96d0 con 0x7f9a1809d1b0 2024-09-16T14:55:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a10003620 con 0x7f9a1809d1b0 2024-09-16T14:55:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9a100039b0 con 0x7f9a1809d1b0 2024-09-16T14:55:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1559a640 4 set_mon_vals no callback set 2024-09-16T14:55:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient: dump: 2024-09-16T14:55:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.971 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1d0e4640 1 -- 172.21.15.26:0/1802187004 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f99fc006590 con 0x7f99fc002830 2024-09-16T14:55:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9a10003b10 con 0x7f9a1809d1b0 2024-09-16T14:55:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f9a10043c20 con 0x7f9a1809d1b0 2024-09-16T14:55:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620984281 0 0) 0x7f99fc006590 con 0x7f99fc002830 2024-09-16T14:55:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient(hunting): my global_id is 234452 2024-09-16T14:55:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a181a93f0 con 0x7f99fc002830 2024-09-16T14:55:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2363882795 0 0) 0x7f9a181a93f0 con 0x7f99fc002830 2024-09-16T14:55:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a1659c640 10 monclient: get_auth_request con 0x7f99f80363e0 auth_method 0 2024-09-16T14:55:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99fc006590 con 0x7f99fc002830 2024-09-16T14:55:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9a080039a0 con 0x7f99fc002830 2024-09-16T14:55:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.949+0000 7f9a0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.953+0000 7f9a0f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.953+0000 7f9a0f7fe640 10 monclient: dump: 2024-09-16T14:55:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.953+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3161475343 0 0) 0x7f99fc006590 con 0x7f99fc002830 2024-09-16T14:55:21.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.953+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99fc00a0e0 con 0x7f99fc002830 2024-09-16T14:55:21.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.953+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f9a080049d0 con 0x7f99fc002830 2024-09-16T14:55:21.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.953+0000 7f9a0f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:21.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.953+0000 7f9a0f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:21.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.953+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 >> v1:172.21.15.26:6790/0 conn(0x7f99fc002830 legacy=0x7f99fc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.953+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 >> v1:172.21.15.26:6790/0 conn(0x7f99fc002830 legacy=0x7f99fc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:21.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a1d0e4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:55:21.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a1d0e4640 1 -- 172.21.15.26:0/1802187004 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f99fc009d80 con 0x7f99fc0090a0 2024-09-16T14:55:21.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877277626 0 0) 0x7f99fc009d80 con 0x7f99fc0090a0 2024-09-16T14:55:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 10 monclient(hunting): my global_id is 234455 2024-09-16T14:55:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99fc006590 con 0x7f99fc0090a0 2024-09-16T14:55:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3138424746 0 0) 0x7f99fc006590 con 0x7f99fc0090a0 2024-09-16T14:55:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99fc009d80 con 0x7f99fc0090a0 2024-09-16T14:55:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9a08004070 con 0x7f99fc0090a0 2024-09-16T14:55:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 10 monclient: dump: 2024-09-16T14:55:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:21.977 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1457052326 0 0) 0x7f99fc009d80 con 0x7f99fc0090a0 2024-09-16T14:55:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f99fc00c290 con 0x7f99fc0090a0 2024-09-16T14:55:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 23172013) 0x7f9a08004b40 con 0x7f99fc0090a0 2024-09-16T14:55:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:55:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:21.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 >> v1:172.21.15.26:6790/0 conn(0x7f99fc0090a0 legacy=0x7f99fc009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.957+0000 7f9a0f7fe640 1 -- 172.21.15.26:0/1802187004 >> v1:172.21.15.26:6790/0 conn(0x7f99fc0090a0 legacy=0x7f99fc009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:21.978 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:55:21.978 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:55:21.978 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:55:21.978 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:55:21.978 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 120, 2024-09-16T14:55:21.978 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:55:21.978 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:55:21.979 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:55:21.979 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:55:21.979 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:21.979 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:55:21.979 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:21.979 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:55:21.979 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:55:21.979 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:21.979 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:21.979 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:21.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:21.979 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:21.979 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:21.980 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:21.980 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:21.980 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:21.980 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:21.980 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:21.980 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:55:21.980 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:55:21.980 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:21.980 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:21.980 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:21.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:21.980 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:21.981 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:21.981 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:21.981 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:21.981 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:21.981 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:21.981 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:21.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.981 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:55:21.981 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:55:21.981 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:55:21.981 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:55:21.981 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:55:21.981 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:55:21.982 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:21.982 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:21.982 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:55:21.982 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:55:21.982 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:55:21.982 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:55:21.982 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:55:21.982 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:55:21.982 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:55:21.982 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:21.982 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:55:21.982 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:21.982 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:21.982 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:21.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:21.983 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:21.983 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:21.983 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:21.983 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:21.983 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:21.983 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:21.983 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:21.983 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:55:21.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.983 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:55:21.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:55:21.983 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:55:21.984 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:21.984 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:21.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.984 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:21.984 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:55:21.984 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:21.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:21.984 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:21.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.984 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:55:21.984 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:55:21.984 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:21.984 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:21.985 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:21.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.985 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:55:21.985 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:55:21.985 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:21.985 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:21.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.985 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:21.985 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:55:21.985 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:21.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:21.985 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:21.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.986 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:55:21.986 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:55:21.986 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:21.986 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:21.986 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:21.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:55:21.986 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:55:21.986 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:21.986 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:21.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.987 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:21.987 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:55:21.987 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:21.987 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:21.987 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:21.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.987 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:55:21.987 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:55:21.987 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:21.987 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:21.987 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:21.987 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:21.987 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:21.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.988 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:55:21.988 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:55:21.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.988 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:21.988 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:21.988 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:21.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:21.988 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:21.988 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:55:21.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.988 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:21.988 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:21.989 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:55:21.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:21.989 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:21.989 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:55:21.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.989 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:21.989 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:21.989 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 7 2024-09-16T14:55:21.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:21.989 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:21.989 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:55:21.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:21.989 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:21.990 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:21.990 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:21.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:21.990 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:21.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:21.990 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:55:21.990 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:55:21.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.961+0000 7f9a1d0e4640 1 -- 172.21.15.26:0/1802187004 >> v1:172.21.15.189:6812/3196161582 conn(0x7f99f80363e0 legacy=0x7f99f80388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.961+0000 7f9a1d0e4640 10 monclient: shutdown 2024-09-16T14:55:21.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.961+0000 7f9a1d0e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:21.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.961+0000 7f9a1d0e4640 1 -- 172.21.15.26:0/1802187004 >> v1:172.21.15.26:6790/0 conn(0x7f9a1809d1b0 legacy=0x7f9a181d1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:21.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.961+0000 7f9a1759e640 1 -- 172.21.15.26:0/1802187004 reap_dead start 2024-09-16T14:55:21.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.961+0000 7f9a1759e640 1 -- 172.21.15.26:0/1802187004 reap_dead start 2024-09-16T14:55:21.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.961+0000 7f9a1d0e4640 1 -- 172.21.15.26:0/1802187004 shutdown_connections 2024-09-16T14:55:21.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.961+0000 7f9a1d0e4640 1 -- 172.21.15.26:0/1802187004 >> 172.21.15.26:0/1802187004 conn(0x7f9a1809fb50 msgr2=0x7f9a181b6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:21.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.961+0000 7f9a1d0e4640 1 -- 172.21.15.26:0/1802187004 shutdown_connections 2024-09-16T14:55:21.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:21.961+0000 7f9a1d0e4640 1 -- 172.21.15.26:0/1802187004 wait complete. 2024-09-16T14:55:21.992 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:55:21.992 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:55:21.993 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:55:22.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f53701c3640 10 monclient: get_monmap_and_config 2024-09-16T14:55:22.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f53701c3640 10 monclient: build_initial_monmap 2024-09-16T14:55:22.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f53701c3640 10 monclient: monmap: 2024-09-16T14:55:22.105 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:22.105 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:22.105 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:22.106746+0000 2024-09-16T14:55:22.105 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:22.106746+0000 2024-09-16T14:55:22.105 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:22.105 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:22.105 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:22.106 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:22.106 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:22.106 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:22.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f53701c3640 1 Processor -- start 2024-09-16T14:55:22.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f53701c3640 1 -- start start 2024-09-16T14:55:22.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f53701c3640 10 monclient: init 2024-09-16T14:55:22.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f53701c3640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:22.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f53701c3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:55:22.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f53701c3640 10 monclient(hunting): picked mon.noname-a con 0x7f53680a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:22.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f53701c3640 10 monclient(hunting): picked mon.noname-b con 0x7f53680a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:22.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f53701c3640 10 monclient(hunting): picked mon.noname-c con 0x7f53680a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:22.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f53701c3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53680a3250 con 0x7f53680a1d10 2024-09-16T14:55:22.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f53701c3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536809f600 con 0x7f53680a2800 2024-09-16T14:55:22.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f53701c3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53680a3830 con 0x7f53680a0000 2024-09-16T14:55:22.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f53701c3640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:22.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f53701c3640 10 monclient(hunting): authenticate will time out at 1746.227378s 2024-09-16T14:55:22.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f536df5f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f53680a1d10 0x7f53680a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38918/0 (socket says 172.21.15.26:38918) 2024-09-16T14:55:22.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f536df5f640 1 -- 172.21.15.26:0/4241040271 learned_addr learned my addr 172.21.15.26:0/4241040271 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:22.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f536d75e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f53680a0000 0x7f5368139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47450/0 (socket says 172.21.15.26:47450) 2024-09-16T14:55:22.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f536cf5d640 1 -- 172.21.15.26:0/4241040271 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909392996 0 0) 0x7f53680a3250 con 0x7f53680a1d10 2024-09-16T14:55:22.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f536cf5d640 10 monclient(hunting): _negotiate 2024-09-16T14:55:22.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f536cf5d640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:22.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f536cf5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:22.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f536cf5d640 10 monclient(hunting): my global_id is 254244 2024-09-16T14:55:22.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f536cf5d640 1 -- 172.21.15.26:0/4241040271 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5354003530 con 0x7f53680a1d10 2024-09-16T14:55:22.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f536cf5d640 1 -- 172.21.15.26:0/4241040271 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231217814 0 0) 0x7f53680a3830 con 0x7f53680a0000 2024-09-16T14:55:22.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f536cf5d640 10 monclient(hunting): _negotiate 2024-09-16T14:55:22.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f536cf5d640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:22.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f536cf5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:22.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f536cf5d640 10 monclient(hunting): my global_id is 234458 2024-09-16T14:55:22.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.101+0000 7f536cf5d640 1 -- 172.21.15.26:0/4241040271 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53680a3250 con 0x7f53680a0000 2024-09-16T14:55:22.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 1 -- 172.21.15.26:0/4241040271 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062836705 0 0) 0x7f536809f600 con 0x7f53680a2800 2024-09-16T14:55:22.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 10 monclient(hunting): _negotiate 2024-09-16T14:55:22.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:22.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:22.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 10 monclient(hunting): my global_id is 264142 2024-09-16T14:55:22.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 1 -- 172.21.15.26:0/4241040271 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53680a3830 con 0x7f53680a2800 2024-09-16T14:55:22.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 1 -- 172.21.15.26:0/4241040271 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2445618359 0 0) 0x7f5354003530 con 0x7f53680a1d10 2024-09-16T14:55:22.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 1 -- 172.21.15.26:0/4241040271 >> v1:172.21.15.26:6790/0 conn(0x7f53680a0000 legacy=0x7f5368139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:22.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 1 -- 172.21.15.26:0/4241040271 >> v1:172.21.15.189:6789/0 conn(0x7f53680a2800 legacy=0x7f536813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:22.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 10 monclient: _finish_hunting 0 2024-09-16T14:55:22.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 1 monclient: found mon.noname-a 2024-09-16T14:55:22.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:22.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:55:22.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 1 -- 172.21.15.26:0/4241040271 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536809ed60 con 0x7f53680a1d10 2024-09-16T14:55:22.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 10 monclient: _finish_auth 0 2024-09-16T14:55:22.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 10 monclient: _check_auth_tickets 2024-09-16T14:55:22.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:22.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 1 -- 172.21.15.26:0/4241040271 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f53640015a0 con 0x7f53680a1d10 2024-09-16T14:55:22.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:22.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:55:22.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 10 monclient: dump: 2024-09-16T14:55:22.113 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:22.113 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:22.113 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.113 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.113 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:22.113 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:22.113 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:22.113 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:22.113 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:22.114 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:22.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 5 monclient: authenticate success, global_id 254244 2024-09-16T14:55:22.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:22.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 1 -- 172.21.15.26:0/4241040271 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53640017c0 con 0x7f53680a1d10 2024-09-16T14:55:22.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:22.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 1 -- 172.21.15.26:0/4241040271 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5364001b50 con 0x7f53680a1d10 2024-09-16T14:55:22.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:22.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 4 set_mon_vals no callback set 2024-09-16T14:55:22.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:22.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:22.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536cf5d640 10 monclient: dump: 2024-09-16T14:55:22.114 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:22.114 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:22.115 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.115 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.115 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:22.115 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:22.115 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:22.115 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:22.115 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:22.115 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:22.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 10 monclient: shutdown 2024-09-16T14:55:22.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:22.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 1 -- 172.21.15.26:0/4241040271 >> v1:172.21.15.26:6789/0 conn(0x7f53680a1d10 legacy=0x7f53680a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:22.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 1 -- 172.21.15.26:0/4241040271 shutdown_connections 2024-09-16T14:55:22.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 1 -- 172.21.15.26:0/4241040271 >> 172.21.15.26:0/4241040271 conn(0x7f53680a7f30 msgr2=0x7f53680a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:22.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 1 -- 172.21.15.26:0/4241040271 shutdown_connections 2024-09-16T14:55:22.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 1 -- 172.21.15.26:0/4241040271 wait complete. 2024-09-16T14:55:22.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 10 monclient: build_initial_monmap 2024-09-16T14:55:22.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 10 monclient: monmap: 2024-09-16T14:55:22.116 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:22.116 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:22.116 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:22.116 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:22.116 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:22.116 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:22.116 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:22.116 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:22.116 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:22.116 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:22.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 1 Processor -- start 2024-09-16T14:55:22.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 1 -- start start 2024-09-16T14:55:22.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 10 monclient: init 2024-09-16T14:55:22.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:22.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:55:22.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 10 monclient(hunting): picked mon.noname-b con 0x7f53680a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:22.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 10 monclient(hunting): picked mon.noname-c con 0x7f53680a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:22.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 10 monclient(hunting): picked mon.noname-a con 0x7f53680a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:22.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536813a060 con 0x7f53680a2800 2024-09-16T14:55:22.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536813ba20 con 0x7f53680a0000 2024-09-16T14:55:22.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536813de00 con 0x7f53680a1d10 2024-09-16T14:55:22.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:22.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 10 monclient(hunting): authenticate will time out at 1746.230520s 2024-09-16T14:55:22.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536e760640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f53680a2800 0x7f53681d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38920/0 (socket says 172.21.15.26:38920) 2024-09-16T14:55:22.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f536e760640 1 -- 172.21.15.26:0/2360959610 learned_addr learned my addr 172.21.15.26:0/2360959610 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:22.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976896045 0 0) 0x7f536813a060 con 0x7f53680a2800 2024-09-16T14:55:22.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:22.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:22.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:22.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): my global_id is 254247 2024-09-16T14:55:22.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f534c003530 con 0x7f53680a2800 2024-09-16T14:55:22.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517914510 0 0) 0x7f536813de00 con 0x7f53680a1d10 2024-09-16T14:55:22.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:22.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:22.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:22.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): my global_id is 234461 2024-09-16T14:55:22.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f536813a060 con 0x7f53680a1d10 2024-09-16T14:55:22.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3260342419 0 0) 0x7f534c003530 con 0x7f53680a2800 2024-09-16T14:55:22.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f536813de00 con 0x7f53680a2800 2024-09-16T14:55:22.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5360001c40 con 0x7f53680a2800 2024-09-16T14:55:22.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:22.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:55:22.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): dump: 2024-09-16T14:55:22.123 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:22.123 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:22.123 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.123 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.123 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:22.123 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:22.123 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:22.123 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:22.123 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:22.124 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:22.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:22.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2637474405 0 0) 0x7f536813a060 con 0x7f53680a1d10 2024-09-16T14:55:22.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f534c003530 con 0x7f53680a1d10 2024-09-16T14:55:22.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5358001840 con 0x7f53680a1d10 2024-09-16T14:55:22.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:22.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:22.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): dump: 2024-09-16T14:55:22.124 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:22.124 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:22.124 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.124 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.124 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:22.125 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:22.125 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:22.125 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:22.125 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:22.125 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:22.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080279036 0 0) 0x7f536813ba20 con 0x7f53680a0000 2024-09-16T14:55:22.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:22.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:22.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:22.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient(hunting): my global_id is 264145 2024-09-16T14:55:22.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f536813a060 con 0x7f53680a0000 2024-09-16T14:55:22.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3775239133 0 0) 0x7f536813de00 con 0x7f53680a2800 2024-09-16T14:55:22.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 >> v1:172.21.15.26:6790/0 conn(0x7f53680a1d10 legacy=0x7f5368139950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:22.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 >> v1:172.21.15.189:6789/0 conn(0x7f53680a0000 legacy=0x7f5368139240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:22.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:55:22.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 monclient: found mon.a 2024-09-16T14:55:22.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:22.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:22.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536813ced0 con 0x7f53680a2800 2024-09-16T14:55:22.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:55:22.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:55:22.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:22.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 5 monclient: authenticate success, global_id 254247 2024-09-16T14:55:22.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 10 monclient: _renew_subs 2024-09-16T14:55:22.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:22.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 1 -- 172.21.15.26:0/2360959610 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536813b7a0 con 0x7f53680a2800 2024-09-16T14:55:22.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 10 monclient: _renew_subs 2024-09-16T14:55:22.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:22.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f53701c3640 1 -- 172.21.15.26:0/2360959610 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f536813d5d0 con 0x7f53680a2800 2024-09-16T14:55:22.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53600039a0 con 0x7f53680a2800 2024-09-16T14:55:22.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:22.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5360003cb0 con 0x7f53680a2800 2024-09-16T14:55:22.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:22.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:22.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535ffff640 4 set_mon_vals no callback set 2024-09-16T14:55:22.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.105+0000 7f535e7fc640 10 monclient: dump: 2024-09-16T14:55:22.127 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:22.127 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:22.127 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.127 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.128 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:22.128 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:22.128 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:22.128 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:22.128 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:22.128 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:22.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f53701c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:22.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f53701c3640 1 -- 172.21.15.26:0/2360959610 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5350006590 con 0x7f5350002830 2024-09-16T14:55:22.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5360003e10 con 0x7f53680a2800 2024-09-16T14:55:22.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f5360044070 con 0x7f53680a2800 2024-09-16T14:55:22.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f536d75e640 10 monclient: get_auth_request con 0x7f534c036410 auth_method 0 2024-09-16T14:55:22.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427022519 0 0) 0x7f5350006590 con 0x7f5350002830 2024-09-16T14:55:22.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:22.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:22.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:22.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 10 monclient(hunting): my global_id is 264148 2024-09-16T14:55:22.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f536813de00 con 0x7f5350002830 2024-09-16T14:55:22.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2465610485 0 0) 0x7f536813de00 con 0x7f5350002830 2024-09-16T14:55:22.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5350006590 con 0x7f5350002830 2024-09-16T14:55:22.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f53640013e0 con 0x7f5350002830 2024-09-16T14:55:22.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:22.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:55:22.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 10 monclient: dump: 2024-09-16T14:55:22.129 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:22.129 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:22.129 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.129 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.130 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:22.130 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:22.130 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:22.130 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:22.130 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:22.130 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:22.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3526918688 0 0) 0x7f5350006590 con 0x7f5350002830 2024-09-16T14:55:22.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f535000a0e0 con 0x7f5350002830 2024-09-16T14:55:22.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f53640047f0 con 0x7f5350002830 2024-09-16T14:55:22.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:22.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:22.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 >> v1:172.21.15.189:6789/0 conn(0x7f5350002830 legacy=0x7f5350004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:22.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.109+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 >> v1:172.21.15.189:6789/0 conn(0x7f5350002830 legacy=0x7f5350004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:22.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f53701c3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:55:22.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f53701c3640 1 -- 172.21.15.26:0/2360959610 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5350009da0 con 0x7f53500090c0 2024-09-16T14:55:22.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498796117 0 0) 0x7f5350009da0 con 0x7f53500090c0 2024-09-16T14:55:22.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:22.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:22.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:22.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 10 monclient(hunting): my global_id is 264151 2024-09-16T14:55:22.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5350006590 con 0x7f53500090c0 2024-09-16T14:55:22.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2181431214 0 0) 0x7f5350006590 con 0x7f53500090c0 2024-09-16T14:55:22.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5350009da0 con 0x7f53500090c0 2024-09-16T14:55:22.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5364004cf0 con 0x7f53500090c0 2024-09-16T14:55:22.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:22.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:55:22.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 10 monclient: dump: 2024-09-16T14:55:22.132 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:22.132 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:22.133 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.133 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.133 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:22.133 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:22.133 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:22.133 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:22.133 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:22.133 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:22.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1291327254 0 0) 0x7f5350009da0 con 0x7f53500090c0 2024-09-16T14:55:22.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f535000c290 con 0x7f53500090c0 2024-09-16T14:55:22.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 2200321584) 0x7f5364005f30 con 0x7f53500090c0 2024-09-16T14:55:22.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:55:22.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:22.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 >> v1:172.21.15.189:6789/0 conn(0x7f53500090c0 legacy=0x7f5350009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:22.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f535e7fc640 1 -- 172.21.15.26:0/2360959610 >> v1:172.21.15.189:6789/0 conn(0x7f53500090c0 legacy=0x7f5350009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:22.134 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:55:22.134 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:55:22.134 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:55:22.134 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:55:22.134 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 120, 2024-09-16T14:55:22.134 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:55:22.134 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:55:22.134 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:55:22.135 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:55:22.135 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:22.135 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:55:22.135 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:22.135 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:55:22.135 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:55:22.135 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:22.135 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:22.135 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:22.135 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:22.135 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:22.135 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:22.135 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:22.136 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:22.136 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:22.136 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:22.136 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:22.136 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:55:22.136 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:55:22.136 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:22.136 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:22.136 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:22.136 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:22.136 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:22.136 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:22.136 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:22.137 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:22.137 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:22.137 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:22.137 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:22.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:22.137 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:55:22.137 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:55:22.137 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:55:22.137 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:55:22.137 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:55:22.137 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:55:22.137 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:22.137 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:22.137 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:55:22.138 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:55:22.138 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:55:22.138 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:55:22.138 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:55:22.138 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:55:22.138 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:55:22.138 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:22.138 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:55:22.138 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:22.138 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:22.138 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:22.138 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:22.138 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:22.139 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:22.139 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:22.139 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:22.139 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:22.139 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:22.139 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:22.139 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:55:22.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:22.139 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:55:22.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:22.139 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:55:22.139 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:55:22.139 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:22.139 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:22.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:22.140 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:22.140 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:55:22.140 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:22.140 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:22.140 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:22.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:22.140 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:55:22.140 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:55:22.140 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:22.140 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:22.140 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:22.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:22.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:22.141 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:55:22.141 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:55:22.141 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:22.141 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:22.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:22.141 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:22.141 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:55:22.141 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:22.141 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:22.141 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:22.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:22.141 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:55:22.142 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:55:22.142 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:22.142 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:22.142 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:22.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:22.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:22.142 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:55:22.142 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:55:22.142 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:22.142 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:22.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:22.142 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:22.142 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:55:22.143 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:22.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:22.143 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:22.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:22.143 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:55:22.143 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:55:22.143 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:22.143 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:22.143 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:22.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:22.143 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:22.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:22.143 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:55:22.143 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:55:22.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:22.144 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:22.144 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:22.144 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:22.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:22.144 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:22.144 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:55:22.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:22.144 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:22.144 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:22.144 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:22.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:22.144 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:22.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:22.145 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:55:22.145 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:55:22.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f53701c3640 1 -- 172.21.15.26:0/2360959610 >> v1:172.21.15.189:6812/3196161582 conn(0x7f534c036410 legacy=0x7f534c0388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:22.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f53701c3640 10 monclient: shutdown 2024-09-16T14:55:22.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f53701c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:22.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f53701c3640 1 -- 172.21.15.26:0/2360959610 >> v1:172.21.15.26:6789/0 conn(0x7f53680a2800 legacy=0x7f53681d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:22.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f536e760640 1 -- 172.21.15.26:0/2360959610 reap_dead start 2024-09-16T14:55:22.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f536e760640 1 -- 172.21.15.26:0/2360959610 reap_dead start 2024-09-16T14:55:22.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f53701c3640 1 -- 172.21.15.26:0/2360959610 shutdown_connections 2024-09-16T14:55:22.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f53701c3640 1 -- 172.21.15.26:0/2360959610 >> 172.21.15.26:0/2360959610 conn(0x7f53680a7f30 msgr2=0x7f536813e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:22.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f53701c3640 1 -- 172.21.15.26:0/2360959610 shutdown_connections 2024-09-16T14:55:22.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.117+0000 7f53701c3640 1 -- 172.21.15.26:0/2360959610 wait complete. 2024-09-16T14:55:22.147 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:55:22.147 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T14:55:22.147 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:55:22.148 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T14:55:22.148 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:55:22.148 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T14:55:22.148 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:55:22.160 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T14:55:22.160 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:55:22.160 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:55:22.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21fa4a640 10 monclient: get_monmap_and_config 2024-09-16T14:55:22.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21fa4a640 10 monclient: build_initial_monmap 2024-09-16T14:55:22.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21fa4a640 10 monclient: monmap: 2024-09-16T14:55:22.322 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:22.322 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:22.322 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:22.323360+0000 2024-09-16T14:55:22.322 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:22.323360+0000 2024-09-16T14:55:22.322 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:22.322 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:22.322 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:22.322 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:22.322 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:22.322 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:22.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21fa4a640 1 Processor -- start 2024-09-16T14:55:22.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21fa4a640 1 -- start start 2024-09-16T14:55:22.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21fa4a640 10 monclient: init 2024-09-16T14:55:22.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21fa4a640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:22.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21fa4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:55:22.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21fa4a640 10 monclient(hunting): picked mon.noname-a con 0x7fa2180968c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:22.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21fa4a640 10 monclient(hunting): picked mon.noname-b con 0x7fa218098bc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:22.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21fa4a640 10 monclient(hunting): picked mon.noname-c con 0x7fa218096ca0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:22.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21fa4a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21806b2d0 con 0x7fa2180968c0 2024-09-16T14:55:22.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21fa4a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa218097940 con 0x7fa218096ca0 2024-09-16T14:55:22.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21fa4a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21809fec0 con 0x7fa218098bc0 2024-09-16T14:55:22.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21fa4a640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:22.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21d7e6640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fa2180968c0 legacy=0x7fa21809f0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:22.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21fa4a640 10 monclient(hunting): authenticate will time out at 1746.443462s 2024-09-16T14:55:22.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21cfe5640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa218098bc0 0x7fa21809f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47458/0 (socket says 172.21.15.26:47458) 2024-09-16T14:55:22.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa21cfe5640 1 -- 172.21.15.26:0/721772333 learned_addr learned my addr 172.21.15.26:0/721772333 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:22.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa20ffff640 1 -- 172.21.15.26:0/721772333 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082550277 0 0) 0x7fa21809fec0 con 0x7fa218098bc0 2024-09-16T14:55:22.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa20ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:55:22.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa20ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:22.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa20ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:22.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa20ffff640 10 monclient(hunting): my global_id is 234464 2024-09-16T14:55:22.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.317+0000 7fa20ffff640 1 -- 172.21.15.26:0/721772333 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa204003530 con 0x7fa218098bc0 2024-09-16T14:55:22.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 1 -- 172.21.15.26:0/721772333 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2910574533 0 0) 0x7fa204003530 con 0x7fa218098bc0 2024-09-16T14:55:22.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 1 -- 172.21.15.26:0/721772333 >> v1:172.21.15.189:6789/0 conn(0x7fa218096ca0 legacy=0x7fa21813da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:22.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 1 -- 172.21.15.26:0/721772333 >> v1:172.21.15.26:6789/0 conn(0x7fa2180968c0 legacy=0x7fa21809f0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:55:22.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:55:22.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 1 monclient: found mon.noname-b 2024-09-16T14:55:22.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:22.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:55:22.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 1 -- 172.21.15.26:0/721772333 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa218097c20 con 0x7fa218098bc0 2024-09-16T14:55:22.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 10 monclient: _finish_auth 0 2024-09-16T14:55:22.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:55:22.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:22.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 1 -- 172.21.15.26:0/721772333 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa208001620 con 0x7fa218098bc0 2024-09-16T14:55:22.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 5 monclient: authenticate success, global_id 234464 2024-09-16T14:55:22.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:22.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:22.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:55:22.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 10 monclient: dump: 2024-09-16T14:55:22.328 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:22.328 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:22.328 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.328 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.328 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:22.329 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:22.329 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:22.329 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:22.329 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:22.329 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:22.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:22.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 1 -- 172.21.15.26:0/721772333 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa208001940 con 0x7fa218098bc0 2024-09-16T14:55:22.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:22.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:22.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 1 -- 172.21.15.26:0/721772333 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa208001d10 con 0x7fa218098bc0 2024-09-16T14:55:22.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:22.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 4 set_mon_vals no callback set 2024-09-16T14:55:22.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:22.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20ffff640 10 monclient: dump: 2024-09-16T14:55:22.330 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:22.330 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:22.330 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.330 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.330 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:22.330 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:22.330 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:22.330 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:22.330 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:22.330 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:22.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 10 monclient: shutdown 2024-09-16T14:55:22.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:22.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 1 -- 172.21.15.26:0/721772333 >> v1:172.21.15.26:6790/0 conn(0x7fa218098bc0 legacy=0x7fa21809f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:22.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 1 -- 172.21.15.26:0/721772333 shutdown_connections 2024-09-16T14:55:22.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 1 -- 172.21.15.26:0/721772333 >> 172.21.15.26:0/721772333 conn(0x7fa2180a0360 msgr2=0x7fa2180a0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:22.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 1 -- 172.21.15.26:0/721772333 shutdown_connections 2024-09-16T14:55:22.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 1 -- 172.21.15.26:0/721772333 wait complete. 2024-09-16T14:55:22.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 10 monclient: build_initial_monmap 2024-09-16T14:55:22.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 10 monclient: monmap: 2024-09-16T14:55:22.331 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:22.331 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:22.331 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:22.331 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:22.331 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:22.332 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:22.332 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:22.332 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:22.332 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:22.332 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:22.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 1 Processor -- start 2024-09-16T14:55:22.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 1 -- start start 2024-09-16T14:55:22.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 10 monclient: init 2024-09-16T14:55:22.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:22.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:55:22.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 10 monclient(hunting): picked mon.noname-c con 0x7fa2180968c0 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:22.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 10 monclient(hunting): picked mon.noname-a con 0x7fa218096ca0 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:22.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 10 monclient(hunting): picked mon.noname-b con 0x7fa218098bc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:22.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa218091cf0 con 0x7fa218096ca0 2024-09-16T14:55:22.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21cfe5640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fa218096ca0 legacy=0x7fa2181ce860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:22.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa218094b70 con 0x7fa218098bc0 2024-09-16T14:55:22.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2180a2f70 con 0x7fa2180968c0 2024-09-16T14:55:22.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:22.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 10 monclient(hunting): authenticate will time out at 1746.447039s 2024-09-16T14:55:22.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21d7e6640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa2180968c0 0x7fa2181c9bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47468/0 (socket says 172.21.15.26:47468) 2024-09-16T14:55:22.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21d7e6640 1 -- 172.21.15.26:0/3590319065 learned_addr learned my addr 172.21.15.26:0/3590319065 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:22.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512381856 0 0) 0x7fa2180a2f70 con 0x7fa2180968c0 2024-09-16T14:55:22.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:55:22.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:22.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:22.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 10 monclient(hunting): my global_id is 234467 2024-09-16T14:55:22.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1fc003530 con 0x7fa2180968c0 2024-09-16T14:55:22.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1734304632 0 0) 0x7fa1fc003530 con 0x7fa2180968c0 2024-09-16T14:55:22.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2180a2f70 con 0x7fa2180968c0 2024-09-16T14:55:22.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa2140044e0 con 0x7fa2180968c0 2024-09-16T14:55:22.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:22.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:55:22.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 10 monclient(hunting): dump: 2024-09-16T14:55:22.338 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:22.338 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:22.338 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.338 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.338 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:22.338 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:22.338 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:22.338 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:22.338 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:22.338 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:22.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:22.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296376803 0 0) 0x7fa218094b70 con 0x7fa218098bc0 2024-09-16T14:55:22.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:55:22.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:22.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:22.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 10 monclient(hunting): my global_id is 264157 2024-09-16T14:55:22.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1fc003530 con 0x7fa218098bc0 2024-09-16T14:55:22.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3295144256 0 0) 0x7fa2180a2f70 con 0x7fa2180968c0 2024-09-16T14:55:22.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 >> v1:172.21.15.189:6789/0 conn(0x7fa218098bc0 legacy=0x7fa2181d1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:22.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 >> v1:172.21.15.26:6789/0 conn(0x7fa218096ca0 legacy=0x7fa2181ce860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:55:22.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:55:22.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 1 monclient: found mon.c 2024-09-16T14:55:22.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:22.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:22.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21809a880 con 0x7fa2180968c0 2024-09-16T14:55:22.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 10 monclient: _finish_auth 0 2024-09-16T14:55:22.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:55:22.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa20dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:22.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 5 monclient: authenticate success, global_id 234467 2024-09-16T14:55:22.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 10 monclient: _renew_subs 2024-09-16T14:55:22.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:22.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 1 -- 172.21.15.26:0/3590319065 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2180902e0 con 0x7fa2180968c0 2024-09-16T14:55:22.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 10 monclient: _renew_subs 2024-09-16T14:55:22.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:22.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.321+0000 7fa21fa4a640 1 -- 172.21.15.26:0/3590319065 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa21808f990 con 0x7fa2180968c0 2024-09-16T14:55:22.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.325+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa214004800 con 0x7fa2180968c0 2024-09-16T14:55:22.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.325+0000 7fa20dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:22.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.325+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa214004b90 con 0x7fa2180968c0 2024-09-16T14:55:22.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.325+0000 7fa20dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:22.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.325+0000 7fa20f7fe640 4 set_mon_vals no callback set 2024-09-16T14:55:22.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.325+0000 7fa20dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:22.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.325+0000 7fa20dffb640 10 monclient: dump: 2024-09-16T14:55:22.341 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:22.341 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:22.341 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.341 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:22.341 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:22.341 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:22.341 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:22.342 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:22.342 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:22.342 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:22.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.325+0000 7fa21fa4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:22.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.325+0000 7fa21fa4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:22.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.325+0000 7fa21fa4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:22.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.325+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa214004d70 con 0x7fa2180968c0 2024-09-16T14:55:22.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.325+0000 7fa21fa4a640 1 -- 172.21.15.26:0/3590319065 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f8005050 con 0x7fa2180968c0 2024-09-16T14:55:22.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.325+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fa2140442e0 con 0x7fa2180968c0 2024-09-16T14:55:22.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.325+0000 7fa21cfe5640 10 monclient: get_auth_request con 0x7fa1fc036130 auth_method 0 2024-09-16T14:55:22.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.329+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa214014c20 con 0x7fa2180968c0 2024-09-16T14:55:22.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.329+0000 7fa20dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:22.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.329+0000 7fa20dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:22.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.553+0000 7fa21fa4a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:55:22.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.553+0000 7fa21fa4a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:55:22.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.553+0000 7fa21fa4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:22.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.553+0000 7fa21fa4a640 1 -- 172.21.15.26:0/3590319065 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa1f8005880 con 0x7fa2180968c0 2024-09-16T14:55:22.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.553+0000 7fa20dffb640 1 -- 172.21.15.26:0/3590319065 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1504611174) 0x7fa214013090 con 0x7fa2180968c0 2024-09-16T14:55:22.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.553+0000 7fa20dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:55:22.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.553+0000 7fa20dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:22.557 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":120,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:55:22.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.553+0000 7fa21fa4a640 1 -- 172.21.15.26:0/3590319065 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa1fc036130 legacy=0x7fa1fc0385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:22.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.553+0000 7fa21fa4a640 10 monclient: shutdown 2024-09-16T14:55:22.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.553+0000 7fa21fa4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:22.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.553+0000 7fa21fa4a640 1 -- 172.21.15.26:0/3590319065 >> v1:172.21.15.26:6790/0 conn(0x7fa2180968c0 legacy=0x7fa2181c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:22.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.553+0000 7fa21fa4a640 1 -- 172.21.15.26:0/3590319065 shutdown_connections 2024-09-16T14:55:22.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.553+0000 7fa21fa4a640 1 -- 172.21.15.26:0/3590319065 >> 172.21.15.26:0/3590319065 conn(0x7fa2180a0360 msgr2=0x7fa2180a1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:22.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.553+0000 7fa21fa4a640 1 -- 172.21.15.26:0/3590319065 shutdown_connections 2024-09-16T14:55:22.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:22.553+0000 7fa21fa4a640 1 -- 172.21.15.26:0/3590319065 wait complete. 2024-09-16T14:55:25.576 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:55:25.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.709+0000 7f63596b1640 10 monclient: get_monmap_and_config 2024-09-16T14:55:25.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.709+0000 7f63596b1640 10 monclient: build_initial_monmap 2024-09-16T14:55:25.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.709+0000 7f63596b1640 10 monclient: monmap: 2024-09-16T14:55:25.714 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:25.714 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:25.714 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:25.715938+0000 2024-09-16T14:55:25.714 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:25.715938+0000 2024-09-16T14:55:25.715 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:25.715 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:25.715 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:25.715 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:25.715 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:25.715 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:25.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.709+0000 7f63596b1640 1 Processor -- start 2024-09-16T14:55:25.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.709+0000 7f63596b1640 1 -- start start 2024-09-16T14:55:25.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.709+0000 7f63596b1640 10 monclient: init 2024-09-16T14:55:25.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:25.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:55:25.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient(hunting): picked mon.noname-a con 0x7f635409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:25.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient(hunting): picked mon.noname-b con 0x7f6354132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:25.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient(hunting): picked mon.noname-c con 0x7f6354138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:25.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635409d260 con 0x7f635409cb80 2024-09-16T14:55:25.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6354132a80 con 0x7f6354138ff0 2024-09-16T14:55:25.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6352ffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f635409cb80 legacy=0x7f63540717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:25.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635409cf60 con 0x7f6354132650 2024-09-16T14:55:25.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:25.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient(hunting): authenticate will time out at 1749.836207s 2024-09-16T14:55:25.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63527fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6354132650 0x7f63541378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47478/0 (socket says 172.21.15.26:47478) 2024-09-16T14:55:25.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63527fc640 1 -- 172.21.15.26:0/4156218300 learned_addr learned my addr 172.21.15.26:0/4156218300 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:25.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 1 -- 172.21.15.26:0/4156218300 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784994363 0 0) 0x7f635409cf60 con 0x7f6354132650 2024-09-16T14:55:25.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:55:25.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:25.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:25.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 10 monclient(hunting): my global_id is 234470 2024-09-16T14:55:25.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 1 -- 172.21.15.26:0/4156218300 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6340003530 con 0x7f6354132650 2024-09-16T14:55:25.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 1 -- 172.21.15.26:0/4156218300 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1975879813 0 0) 0x7f6340003530 con 0x7f6354132650 2024-09-16T14:55:25.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 1 -- 172.21.15.26:0/4156218300 >> v1:172.21.15.189:6789/0 conn(0x7f6354138ff0 legacy=0x7f635413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:25.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 1 -- 172.21.15.26:0/4156218300 >> v1:172.21.15.26:6789/0 conn(0x7f635409cb80 legacy=0x7f63540717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:55:25.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:55:25.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 1 monclient: found mon.noname-b 2024-09-16T14:55:25.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:25.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:55:25.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 1 -- 172.21.15.26:0/4156218300 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635409f2d0 con 0x7f6354132650 2024-09-16T14:55:25.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 10 monclient: _finish_auth 0 2024-09-16T14:55:25.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:55:25.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:25.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 1 -- 172.21.15.26:0/4156218300 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f633c001660 con 0x7f6354132650 2024-09-16T14:55:25.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 5 monclient: authenticate success, global_id 234470 2024-09-16T14:55:25.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:25.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:25.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:55:25.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 10 monclient: dump: 2024-09-16T14:55:25.729 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:25.729 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:25.729 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:25.730 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:25.730 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:25.730 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:25.730 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:25.730 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:25.730 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:25.730 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:25.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:25.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 1 -- 172.21.15.26:0/4156218300 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f633c0018c0 con 0x7f6354132650 2024-09-16T14:55:25.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:25.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:25.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 1 -- 172.21.15.26:0/4156218300 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f633c001c90 con 0x7f6354132650 2024-09-16T14:55:25.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 4 set_mon_vals no callback set 2024-09-16T14:55:25.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:25.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:25.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f6351ffb640 10 monclient: dump: 2024-09-16T14:55:25.731 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:25.731 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:25.731 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:25.731 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:25.731 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:25.731 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:25.731 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:25.731 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:25.732 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:25.732 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:25.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient: shutdown 2024-09-16T14:55:25.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:25.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 1 -- 172.21.15.26:0/4156218300 >> v1:172.21.15.26:6790/0 conn(0x7f6354132650 legacy=0x7f63541378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:25.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 1 -- 172.21.15.26:0/4156218300 shutdown_connections 2024-09-16T14:55:25.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 1 -- 172.21.15.26:0/4156218300 >> 172.21.15.26:0/4156218300 conn(0x7f635409fb50 msgr2=0x7f635409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:25.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 1 -- 172.21.15.26:0/4156218300 shutdown_connections 2024-09-16T14:55:25.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 1 -- 172.21.15.26:0/4156218300 wait complete. 2024-09-16T14:55:25.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient: build_initial_monmap 2024-09-16T14:55:25.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient: monmap: 2024-09-16T14:55:25.733 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:25.733 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:25.733 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:25.733 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:25.733 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:25.733 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:25.733 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:25.733 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:25.733 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:25.733 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:25.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 1 Processor -- start 2024-09-16T14:55:25.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 1 -- start start 2024-09-16T14:55:25.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient: init 2024-09-16T14:55:25.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:25.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:55:25.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient(hunting): picked mon.noname-a con 0x7f635409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:25.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient(hunting): picked mon.noname-c con 0x7f6354132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:25.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.713+0000 7f63596b1640 10 monclient(hunting): picked mon.noname-b con 0x7f6354138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:25.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f6352ffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f635409cb80 legacy=0x7f635406d660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:25.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63596b1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635409a240 con 0x7f635409cb80 2024-09-16T14:55:25.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63596b1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635406e480 con 0x7f6354138ff0 2024-09-16T14:55:25.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63596b1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635406fb60 con 0x7f6354132650 2024-09-16T14:55:25.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63596b1640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:25.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63596b1640 10 monclient(hunting): authenticate will time out at 1749.840119s 2024-09-16T14:55:25.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63527fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6354132650 0x7f635406dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47492/0 (socket says 172.21.15.26:47492) 2024-09-16T14:55:25.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63527fc640 1 -- 172.21.15.26:0/752292380 learned_addr learned my addr 172.21.15.26:0/752292380 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:25.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 1 -- 172.21.15.26:0/752292380 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187928863 0 0) 0x7f635406fb60 con 0x7f6354132650 2024-09-16T14:55:25.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:25.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:25.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:25.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 10 monclient(hunting): my global_id is 234473 2024-09-16T14:55:25.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 1 -- 172.21.15.26:0/752292380 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6334003530 con 0x7f6354132650 2024-09-16T14:55:25.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 1 -- 172.21.15.26:0/752292380 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1875902188 0 0) 0x7f6334003530 con 0x7f6354132650 2024-09-16T14:55:25.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 1 -- 172.21.15.26:0/752292380 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f635406fb60 con 0x7f6354132650 2024-09-16T14:55:25.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 1 -- 172.21.15.26:0/752292380 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f633c001bd0 con 0x7f6354132650 2024-09-16T14:55:25.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:25.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:55:25.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 10 monclient(hunting): dump: 2024-09-16T14:55:25.736 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:25.736 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:25.736 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:25.736 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:25.736 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:25.736 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:25.737 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:25.737 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:25.737 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:25.737 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:25.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:25.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 1 -- 172.21.15.26:0/752292380 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1872573194 0 0) 0x7f635406fb60 con 0x7f6354132650 2024-09-16T14:55:25.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 1 -- 172.21.15.26:0/752292380 >> v1:172.21.15.189:6789/0 conn(0x7f6354138ff0 legacy=0x7f63541cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:25.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 1 -- 172.21.15.26:0/752292380 >> v1:172.21.15.26:6789/0 conn(0x7f635409cb80 legacy=0x7f635406d660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:55:25.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:55:25.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 1 monclient: found mon.c 2024-09-16T14:55:25.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:25.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:25.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 1 -- 172.21.15.26:0/752292380 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63540a2100 con 0x7f6354132650 2024-09-16T14:55:25.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:55:25.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:55:25.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:25.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63596b1640 5 monclient: authenticate success, global_id 234473 2024-09-16T14:55:25.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63596b1640 10 monclient: _renew_subs 2024-09-16T14:55:25.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63596b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:25.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63596b1640 1 -- 172.21.15.26:0/752292380 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635413c5e0 con 0x7f6354132650 2024-09-16T14:55:25.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63596b1640 10 monclient: _renew_subs 2024-09-16T14:55:25.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63596b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:25.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63596b1640 1 -- 172.21.15.26:0/752292380 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6354138110 con 0x7f6354132650 2024-09-16T14:55:25.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 1 -- 172.21.15.26:0/752292380 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f633c004c00 con 0x7f6354132650 2024-09-16T14:55:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 1 -- 172.21.15.26:0/752292380 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f633c004f90 con 0x7f6354132650 2024-09-16T14:55:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63517fa640 4 set_mon_vals no callback set 2024-09-16T14:55:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 10 monclient: dump: 2024-09-16T14:55:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:25.739 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:25.740 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:25.740 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:25.740 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:25.740 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:25.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63596b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:25.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63596b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:25.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63596b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:25.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f63596b1640 1 -- 172.21.15.26:0/752292380 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6338005050 con 0x7f6354132650 2024-09-16T14:55:25.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 1 -- 172.21.15.26:0/752292380 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f633c005170 con 0x7f6354132650 2024-09-16T14:55:25.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f634b7fe640 1 -- 172.21.15.26:0/752292380 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f633c044080 con 0x7f6354132650 2024-09-16T14:55:25.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.717+0000 7f6352ffd640 10 monclient: get_auth_request con 0x7f6334036030 auth_method 0 2024-09-16T14:55:25.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.721+0000 7f634b7fe640 1 -- 172.21.15.26:0/752292380 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f633c014a80 con 0x7f6354132650 2024-09-16T14:55:25.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.721+0000 7f634b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:25.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.721+0000 7f634b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:25.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.937+0000 7f63596b1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:55:25.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.937+0000 7f63596b1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:55:25.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.937+0000 7f63596b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:25.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.937+0000 7f63596b1640 1 -- 172.21.15.26:0/752292380 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6338005880 con 0x7f6354132650 2024-09-16T14:55:25.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.937+0000 7f634b7fe640 1 -- 172.21.15.26:0/752292380 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 155338055) 0x7f633c014a80 con 0x7f6354132650 2024-09-16T14:55:25.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.937+0000 7f634b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:55:25.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.937+0000 7f634b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:25.942 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":120,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:55:25.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.941+0000 7f63596b1640 1 -- 172.21.15.26:0/752292380 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6334036030 legacy=0x7f63340384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:25.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.941+0000 7f63596b1640 10 monclient: shutdown 2024-09-16T14:55:25.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.941+0000 7f63596b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:25.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.941+0000 7f63596b1640 1 -- 172.21.15.26:0/752292380 >> v1:172.21.15.26:6790/0 conn(0x7f6354132650 legacy=0x7f635406dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:25.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.941+0000 7f63596b1640 1 -- 172.21.15.26:0/752292380 shutdown_connections 2024-09-16T14:55:25.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.941+0000 7f63596b1640 1 -- 172.21.15.26:0/752292380 >> 172.21.15.26:0/752292380 conn(0x7f635409fb50 msgr2=0x7f6354131e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:25.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.941+0000 7f63596b1640 1 -- 172.21.15.26:0/752292380 shutdown_connections 2024-09-16T14:55:25.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:25.941+0000 7f63596b1640 1 -- 172.21.15.26:0/752292380 wait complete. 2024-09-16T14:55:28.961 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:55:29.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc9352640 10 monclient: get_monmap_and_config 2024-09-16T14:55:29.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc9352640 10 monclient: build_initial_monmap 2024-09-16T14:55:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc9352640 10 monclient: monmap: 2024-09-16T14:55:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:29.095280+0000 2024-09-16T14:55:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:29.095280+0000 2024-09-16T14:55:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:29.094 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:29.094 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:29.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc9352640 1 Processor -- start 2024-09-16T14:55:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc9352640 1 -- start start 2024-09-16T14:55:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc9352640 10 monclient: init 2024-09-16T14:55:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc9352640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc9352640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:55:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc9352640 10 monclient(hunting): picked mon.noname-b con 0x7f1cc409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc9352640 10 monclient(hunting): picked mon.noname-a con 0x7f1cc4132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc9352640 10 monclient(hunting): picked mon.noname-c con 0x7f1cc4138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc9352640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc409d260 con 0x7f1cc4132650 2024-09-16T14:55:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc9352640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc4132a80 con 0x7f1cc4138ff0 2024-09-16T14:55:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc27fc640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f1cc4132650 legacy=0x7f1cc41378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:29.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc9352640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc409cf60 con 0x7f1cc409cb80 2024-09-16T14:55:29.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc9352640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:29.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc9352640 10 monclient(hunting): authenticate will time out at 1753.215509s 2024-09-16T14:55:29.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc2ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1cc409cb80 0x7f1cc40717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57024/0 (socket says 172.21.15.26:57024) 2024-09-16T14:55:29.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.089+0000 7f1cc2ffd640 1 -- 172.21.15.26:0/1396080635 learned_addr learned my addr 172.21.15.26:0/1396080635 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:29.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.293+0000 7f1cc27fc640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6789/0 conn(0x7f1cc4132650 legacy=0x7f1cc41378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:29.697+0000 7f1cc27fc640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6789/0 conn(0x7f1cc4132650 legacy=0x7f1cc41378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:30.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:30.501+0000 7f1cc27fc640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6789/0 conn(0x7f1cc4132650 legacy=0x7f1cc41378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.093+0000 7f1cc0ff9640 10 monclient(hunting): tick 2024-09-16T14:55:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.093+0000 7f1cc0ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:55:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.093+0000 7f1cc0ff9640 1 monclient(hunting): continuing hunt 2024-09-16T14:55:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.093+0000 7f1cc0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:55:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.093+0000 7f1cc0ff9640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6790/0 conn(0x7f1cc409cb80 legacy=0x7f1cc40717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.093+0000 7f1cc0ff9640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.189:6789/0 conn(0x7f1cc4138ff0 legacy=0x7f1cc413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.093+0000 7f1cc0ff9640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6789/0 conn(0x7f1cc4132650 legacy=0x7f1cc41378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:55:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.093+0000 7f1cc0ff9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:55:32.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.093+0000 7f1cc0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f1ca40036d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:32.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.093+0000 7f1cc0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f1cc409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:32.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.093+0000 7f1cc0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f1cc4132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:32.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.093+0000 7f1cc0ff9640 1 -- 172.21.15.26:0/1396080635 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc409f2d0 con 0x7f1cc4132650 2024-09-16T14:55:32.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.093+0000 7f1cc0ff9640 1 -- 172.21.15.26:0/1396080635 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc409d260 con 0x7f1ca40036d0 2024-09-16T14:55:32.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.093+0000 7f1cc0ff9640 1 -- 172.21.15.26:0/1396080635 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca40103b0 con 0x7f1cc409cb80 2024-09-16T14:55:32.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.093+0000 7f1cc0ff9640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:32.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.093+0000 7f1cc37fe640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6789/0 conn(0x7f1cc4132650 legacy=0x7f1ca400cbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:32.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.297+0000 7f1cc37fe640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6789/0 conn(0x7f1cc4132650 legacy=0x7f1ca400cbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:32.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:32.701+0000 7f1cc37fe640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6789/0 conn(0x7f1cc4132650 legacy=0x7f1ca400cbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:33.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:33.505+0000 7f1cc37fe640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6789/0 conn(0x7f1cc4132650 legacy=0x7f1ca400cbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:35.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.093+0000 7f1cc0ff9640 10 monclient(hunting): tick 2024-09-16T14:55:35.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.093+0000 7f1cc0ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:55:35.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.093+0000 7f1cc0ff9640 1 monclient(hunting): continuing hunt 2024-09-16T14:55:35.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.093+0000 7f1cc0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:55:35.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.093+0000 7f1cc0ff9640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6790/0 conn(0x7f1cc409cb80 legacy=0x7f1ca40093d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:35.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.093+0000 7f1cc0ff9640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.189:6789/0 conn(0x7f1ca40036d0 legacy=0x7f1ca4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:35.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.093+0000 7f1cc0ff9640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6789/0 conn(0x7f1cc4132650 legacy=0x7f1ca400cbb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:55:35.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.093+0000 7f1cc0ff9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:55:35.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.093+0000 7f1cc0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f1cc4132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:35.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.093+0000 7f1cc0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f1ca40036d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:35.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.093+0000 7f1cc0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f1cc409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:35.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.093+0000 7f1cc0ff9640 1 -- 172.21.15.26:0/1396080635 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca4011620 con 0x7f1cc4132650 2024-09-16T14:55:35.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.093+0000 7f1cc0ff9640 1 -- 172.21.15.26:0/1396080635 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc409f2d0 con 0x7f1ca40036d0 2024-09-16T14:55:35.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.093+0000 7f1cc0ff9640 1 -- 172.21.15.26:0/1396080635 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca400cc30 con 0x7f1cc409cb80 2024-09-16T14:55:35.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.093+0000 7f1cc2ffd640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6789/0 conn(0x7f1cc4132650 legacy=0x7f1ca4010670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:35.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.093+0000 7f1cc0ff9640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.297+0000 7f1cc2ffd640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6789/0 conn(0x7f1cc4132650 legacy=0x7f1ca4010670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:35.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:35.701+0000 7f1cc2ffd640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6789/0 conn(0x7f1cc4132650 legacy=0x7f1ca4010670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:36.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.505+0000 7f1cc2ffd640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6789/0 conn(0x7f1cc4132650 legacy=0x7f1ca4010670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:36.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.577+0000 7f1cc1ffb640 1 -- 172.21.15.26:0/1396080635 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103897187 0 0) 0x7f1cc409f2d0 con 0x7f1ca40036d0 2024-09-16T14:55:36.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.577+0000 7f1cc1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:55:36.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.577+0000 7f1cc1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:36.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.577+0000 7f1cc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:36.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.577+0000 7f1cc1ffb640 10 monclient(hunting): my global_id is 264160 2024-09-16T14:55:36.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.577+0000 7f1cc1ffb640 1 -- 172.21.15.26:0/1396080635 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cb0003530 con 0x7f1ca40036d0 2024-09-16T14:55:36.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 1 -- 172.21.15.26:0/1396080635 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498248762 0 0) 0x7f1ca400cc30 con 0x7f1cc409cb80 2024-09-16T14:55:36.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:55:36.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:36.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:36.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 10 monclient(hunting): my global_id is 234476 2024-09-16T14:55:36.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 1 -- 172.21.15.26:0/1396080635 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc409f2d0 con 0x7f1cc409cb80 2024-09-16T14:55:36.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 1 -- 172.21.15.26:0/1396080635 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3610591966 0 0) 0x7f1cc409f2d0 con 0x7f1cc409cb80 2024-09-16T14:55:36.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.189:6789/0 conn(0x7f1ca40036d0 legacy=0x7f1ca4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:36.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6789/0 conn(0x7f1cc4132650 legacy=0x7f1ca4010670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:55:36.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:55:36.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 1 monclient: found mon.noname-b 2024-09-16T14:55:36.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:36.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:55:36.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 1 -- 172.21.15.26:0/1396080635 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca400ce30 con 0x7f1cc409cb80 2024-09-16T14:55:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 10 monclient: _finish_auth 0 2024-09-16T14:55:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:55:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc9352640 5 monclient: authenticate success, global_id 234476 2024-09-16T14:55:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc9352640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 1 -- 172.21.15.26:0/1396080635 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1cbc001620 con 0x7f1cc409cb80 2024-09-16T14:55:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:55:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc1ffb640 10 monclient: dump: 2024-09-16T14:55:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:36.590 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:36.591 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:36.591 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:36.591 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:36.591 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:36.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.585+0000 7f1cc9352640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:36.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc1ffb640 1 -- 172.21.15.26:0/1396080635 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cbc001880 con 0x7f1cc409cb80 2024-09-16T14:55:36.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:36.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc1ffb640 1 -- 172.21.15.26:0/1396080635 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1cbc001c90 con 0x7f1cc409cb80 2024-09-16T14:55:36.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc9352640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:36.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc9352640 4 set_mon_vals no callback set 2024-09-16T14:55:36.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:36.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:36.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc1ffb640 10 monclient: dump: 2024-09-16T14:55:36.592 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:36.592 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:36.592 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.593 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.593 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:36.593 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:36.593 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:36.593 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:36.593 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:36.593 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:36.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc9352640 10 monclient: shutdown 2024-09-16T14:55:36.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc9352640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:36.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc9352640 1 -- 172.21.15.26:0/1396080635 >> v1:172.21.15.26:6790/0 conn(0x7f1cc409cb80 legacy=0x7f1ca40093d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:36.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc9352640 1 -- 172.21.15.26:0/1396080635 shutdown_connections 2024-09-16T14:55:36.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc9352640 1 -- 172.21.15.26:0/1396080635 >> 172.21.15.26:0/1396080635 conn(0x7f1cc409fb50 msgr2=0x7f1cc409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:36.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc9352640 1 -- 172.21.15.26:0/1396080635 shutdown_connections 2024-09-16T14:55:36.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc9352640 1 -- 172.21.15.26:0/1396080635 wait complete. 2024-09-16T14:55:36.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc9352640 10 monclient: build_initial_monmap 2024-09-16T14:55:36.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc9352640 10 monclient: monmap: 2024-09-16T14:55:36.595 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:36.595 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:36.596 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:36.596 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:36.596 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:36.596 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:36.596 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:36.596 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:36.596 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:36.596 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:36.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.589+0000 7f1cc9352640 1 Processor -- start 2024-09-16T14:55:36.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 1 -- start start 2024-09-16T14:55:36.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 10 monclient: init 2024-09-16T14:55:36.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 10 monclient(hunting): picked mon.noname-a con 0x7f1ca40036d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 10 monclient(hunting): picked mon.noname-c con 0x7f1cc409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 10 monclient(hunting): picked mon.noname-b con 0x7f1cc4132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc4134cb0 con 0x7f1ca40036d0 2024-09-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc41365b0 con 0x7f1cc4132650 2024-09-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc2ffd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f1ca40036d0 legacy=0x7f1cc4133cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc4134e80 con 0x7f1cc409cb80 2024-09-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 10 monclient(hunting): authenticate will time out at 1760.717159s 2024-09-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc27fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1cc409cb80 0x7f1cc4134400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57048/0 (socket says 172.21.15.26:57048) 2024-09-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc27fc640 1 -- 172.21.15.26:0/3394763043 learned_addr learned my addr 172.21.15.26:0/3394763043 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:36.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076388849 0 0) 0x7f1cc4134e80 con 0x7f1cc409cb80 2024-09-16T14:55:36.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:36.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:36.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:36.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient(hunting): my global_id is 234479 2024-09-16T14:55:36.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ca40115c0 con 0x7f1cc409cb80 2024-09-16T14:55:36.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686097064 0 0) 0x7f1cc41365b0 con 0x7f1cc4132650 2024-09-16T14:55:36.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:36.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:36.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:36.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient(hunting): my global_id is 264163 2024-09-16T14:55:36.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc4134e80 con 0x7f1cc4132650 2024-09-16T14:55:36.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2559814722 0 0) 0x7f1ca40115c0 con 0x7f1cc409cb80 2024-09-16T14:55:36.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cc41365b0 con 0x7f1cc409cb80 2024-09-16T14:55:36.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1cac0041f0 con 0x7f1cc409cb80 2024-09-16T14:55:36.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:36.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:55:36.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient(hunting): dump: 2024-09-16T14:55:36.601 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:36.601 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:36.601 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.601 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.601 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:36.601 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:36.601 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:36.601 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:36.602 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:36.602 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:36.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:36.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4175258724 0 0) 0x7f1cc41365b0 con 0x7f1cc409cb80 2024-09-16T14:55:36.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 >> v1:172.21.15.189:6789/0 conn(0x7f1cc4132650 legacy=0x7f1cc41cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:36.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 >> v1:172.21.15.26:6789/0 conn(0x7f1ca40036d0 legacy=0x7f1cc4133cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:55:36.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:55:36.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 1 monclient: found mon.c 2024-09-16T14:55:36.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:36.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:36.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc40a2460 con 0x7f1cc409cb80 2024-09-16T14:55:36.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient: _finish_auth 0 2024-09-16T14:55:36.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:55:36.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:36.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 5 monclient: authenticate success, global_id 234479 2024-09-16T14:55:36.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 10 monclient: _renew_subs 2024-09-16T14:55:36.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:36.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 1 -- 172.21.15.26:0/3394763043 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc41362d0 con 0x7f1cc409cb80 2024-09-16T14:55:36.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 10 monclient: _renew_subs 2024-09-16T14:55:36.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:36.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc9352640 1 -- 172.21.15.26:0/3394763043 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc4070c40 con 0x7f1cc409cb80 2024-09-16T14:55:36.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cac0046d0 con 0x7f1cc409cb80 2024-09-16T14:55:36.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:36.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1cac004a60 con 0x7f1cc409cb80 2024-09-16T14:55:36.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:36.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:36.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cc17fa640 4 set_mon_vals no callback set 2024-09-16T14:55:36.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.593+0000 7f1cbb7fe640 10 monclient: dump: 2024-09-16T14:55:36.603 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:36.604 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:36.604 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.604 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.604 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:36.604 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:36.604 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:36.604 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:36.604 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:36.604 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:36.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.597+0000 7f1cc9352640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:36.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.597+0000 7f1cc9352640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:36.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.597+0000 7f1cc9352640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:36.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.597+0000 7f1cc9352640 1 -- 172.21.15.26:0/3394763043 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca8005050 con 0x7f1cc409cb80 2024-09-16T14:55:36.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.597+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1cac004bc0 con 0x7f1cc409cb80 2024-09-16T14:55:36.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.597+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f1cac043d50 con 0x7f1cc409cb80 2024-09-16T14:55:36.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.597+0000 7f1cc2ffd640 10 monclient: get_auth_request con 0x7f1ca4036ac0 auth_method 0 2024-09-16T14:55:36.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.601+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1cac0061e0 con 0x7f1cc409cb80 2024-09-16T14:55:36.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.601+0000 7f1cbb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:36.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.601+0000 7f1cbb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.829+0000 7f1cc9352640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:55:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.829+0000 7f1cc9352640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:55:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.829+0000 7f1cc9352640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:36.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.829+0000 7f1cc9352640 1 -- 172.21.15.26:0/3394763043 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1ca8005880 con 0x7f1cc409cb80 2024-09-16T14:55:36.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.829+0000 7f1cbb7fe640 1 -- 172.21.15.26:0/3394763043 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 3382038062) 0x7f1cac014220 con 0x7f1cc409cb80 2024-09-16T14:55:36.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.829+0000 7f1cbb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:55:36.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.829+0000 7f1cbb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:36.833 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":124,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:55:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.829+0000 7f1cc9352640 1 -- 172.21.15.26:0/3394763043 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1ca4036ac0 legacy=0x7f1ca4038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.829+0000 7f1cc9352640 10 monclient: shutdown 2024-09-16T14:55:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.829+0000 7f1cc9352640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.829+0000 7f1cc9352640 1 -- 172.21.15.26:0/3394763043 >> v1:172.21.15.26:6790/0 conn(0x7f1cc409cb80 legacy=0x7f1cc4134400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.829+0000 7f1cc9352640 1 -- 172.21.15.26:0/3394763043 shutdown_connections 2024-09-16T14:55:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.829+0000 7f1cc9352640 1 -- 172.21.15.26:0/3394763043 >> 172.21.15.26:0/3394763043 conn(0x7f1cc409fb50 msgr2=0x7f1cc4131430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.829+0000 7f1cc9352640 1 -- 172.21.15.26:0/3394763043 shutdown_connections 2024-09-16T14:55:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.829+0000 7f1cc9352640 1 -- 172.21.15.26:0/3394763043 wait complete. 2024-09-16T14:55:36.852 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:55:36.852 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:55:36.852 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:55:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 10 monclient: get_monmap_and_config 2024-09-16T14:55:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 10 monclient: build_initial_monmap 2024-09-16T14:55:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 10 monclient: monmap: 2024-09-16T14:55:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:36.974817+0000 2024-09-16T14:55:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:36.974817+0000 2024-09-16T14:55:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:36.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:36.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 1 Processor -- start 2024-09-16T14:55:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 1 -- start start 2024-09-16T14:55:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 10 monclient: init 2024-09-16T14:55:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:55:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 10 monclient(hunting): picked mon.noname-a con 0x7f993409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 10 monclient(hunting): picked mon.noname-b con 0x7f9934132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 10 monclient(hunting): picked mon.noname-c con 0x7f9934138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993409d260 con 0x7f993409cb80 2024-09-16T14:55:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9934132a80 con 0x7f9934138ff0 2024-09-16T14:55:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993409cf60 con 0x7f9934132650 2024-09-16T14:55:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993259c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f993409cb80 legacy=0x7f99340717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 10 monclient(hunting): authenticate will time out at 1761.094924s 2024-09-16T14:55:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9931d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9934132650 0x7f99341378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:37200/0 (socket says 172.21.15.26:37200) 2024-09-16T14:55:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9931d9b640 1 -- 172.21.15.26:0/900096290 learned_addr learned my addr 172.21.15.26:0/900096290 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 1 -- 172.21.15.26:0/900096290 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198789687 0 0) 0x7f993409cf60 con 0x7f9934132650 2024-09-16T14:55:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 10 monclient(hunting): _negotiate 2024-09-16T14:55:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 10 monclient(hunting): my global_id is 234482 2024-09-16T14:55:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 1 -- 172.21.15.26:0/900096290 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f991c003530 con 0x7f9934132650 2024-09-16T14:55:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 1 -- 172.21.15.26:0/900096290 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 616629932 0 0) 0x7f991c003530 con 0x7f9934132650 2024-09-16T14:55:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 1 -- 172.21.15.26:0/900096290 >> v1:172.21.15.189:6789/0 conn(0x7f9934138ff0 legacy=0x7f993413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 1 -- 172.21.15.26:0/900096290 >> v1:172.21.15.26:6789/0 conn(0x7f993409cb80 legacy=0x7f99340717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:55:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 10 monclient: _finish_hunting 0 2024-09-16T14:55:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 1 monclient: found mon.noname-b 2024-09-16T14:55:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:55:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 1 -- 172.21.15.26:0/900096290 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993409f2d0 con 0x7f9934132650 2024-09-16T14:55:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 10 monclient: _finish_auth 0 2024-09-16T14:55:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 10 monclient: _check_auth_tickets 2024-09-16T14:55:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 5 monclient: authenticate success, global_id 234482 2024-09-16T14:55:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 1 -- 172.21.15.26:0/900096290 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9928001660 con 0x7f9934132650 2024-09-16T14:55:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:55:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 10 monclient: dump: 2024-09-16T14:55:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:36.980 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 1 -- 172.21.15.26:0/900096290 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99280018c0 con 0x7f9934132650 2024-09-16T14:55:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 1 -- 172.21.15.26:0/900096290 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9928001c90 con 0x7f9934132650 2024-09-16T14:55:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f9938cfe640 4 set_mon_vals no callback set 2024-09-16T14:55:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.969+0000 7f993159a640 10 monclient: dump: 2024-09-16T14:55:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:36.982 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 10 monclient: shutdown 2024-09-16T14:55:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 1 -- 172.21.15.26:0/900096290 >> v1:172.21.15.26:6790/0 conn(0x7f9934132650 legacy=0x7f99341378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 1 -- 172.21.15.26:0/900096290 shutdown_connections 2024-09-16T14:55:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 1 -- 172.21.15.26:0/900096290 >> 172.21.15.26:0/900096290 conn(0x7f993409fb50 msgr2=0x7f993409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 1 -- 172.21.15.26:0/900096290 shutdown_connections 2024-09-16T14:55:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 1 -- 172.21.15.26:0/900096290 wait complete. 2024-09-16T14:55:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 10 monclient: build_initial_monmap 2024-09-16T14:55:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 10 monclient: monmap: 2024-09-16T14:55:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:36.983 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 1 Processor -- start 2024-09-16T14:55:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 1 -- start start 2024-09-16T14:55:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 10 monclient: init 2024-09-16T14:55:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:55:36.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 10 monclient(hunting): picked mon.noname-c con 0x7f993409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 10 monclient(hunting): picked mon.noname-a con 0x7f9934132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 10 monclient(hunting): picked mon.noname-b con 0x7f9934138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993406fcd0 con 0x7f9934132650 2024-09-16T14:55:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99340711f0 con 0x7f9934138ff0 2024-09-16T14:55:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993406e370 con 0x7f993409cb80 2024-09-16T14:55:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9931d9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f9934132650 legacy=0x7f993406dc60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 10 monclient(hunting): authenticate will time out at 1761.098655s 2024-09-16T14:55:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f993259c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f993409cb80 0x7f993406d550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:37210/0 (socket says 172.21.15.26:37210) 2024-09-16T14:55:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f993259c640 1 -- 172.21.15.26:0/3556127366 learned_addr learned my addr 172.21.15.26:0/3556127366 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487722487 0 0) 0x7f993406e370 con 0x7f993409cb80 2024-09-16T14:55:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:36.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient(hunting): my global_id is 234485 2024-09-16T14:55:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9914003530 con 0x7f993409cb80 2024-09-16T14:55:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2573869358 0 0) 0x7f9914003530 con 0x7f993409cb80 2024-09-16T14:55:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f993406e370 con 0x7f993409cb80 2024-09-16T14:55:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9924004160 con 0x7f993409cb80 2024-09-16T14:55:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:55:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient(hunting): dump: 2024-09-16T14:55:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:36.991 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:36.992 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645041221 0 0) 0x7f99340711f0 con 0x7f9934138ff0 2024-09-16T14:55:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient(hunting): my global_id is 264169 2024-09-16T14:55:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9914003530 con 0x7f9934138ff0 2024-09-16T14:55:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3241109934 0 0) 0x7f993406e370 con 0x7f993409cb80 2024-09-16T14:55:36.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 >> v1:172.21.15.189:6789/0 conn(0x7f9934138ff0 legacy=0x7f99341cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 >> v1:172.21.15.26:6789/0 conn(0x7f9934132650 legacy=0x7f993406dc60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:55:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:55:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 1 monclient: found mon.c 2024-09-16T14:55:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993413c400 con 0x7f993409cb80 2024-09-16T14:55:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient: _finish_auth 0 2024-09-16T14:55:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:55:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 5 monclient: authenticate success, global_id 234485 2024-09-16T14:55:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 10 monclient: _renew_subs 2024-09-16T14:55:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:36.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 1 -- 172.21.15.26:0/3556127366 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99340a28f0 con 0x7f993409cb80 2024-09-16T14:55:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 10 monclient: _renew_subs 2024-09-16T14:55:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9938cfe640 1 -- 172.21.15.26:0/3556127366 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9934070f10 con 0x7f993409cb80 2024-09-16T14:55:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9924004480 con 0x7f993409cb80 2024-09-16T14:55:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9924004810 con 0x7f993409cb80 2024-09-16T14:55:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9922ffd640 10 monclient: dump: 2024-09-16T14:55:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.994 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:36.995 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.973+0000 7f9930d99640 4 set_mon_vals no callback set 2024-09-16T14:55:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9938cfe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9938cfe640 1 -- 172.21.15.26:0/3556127366 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9918006590 con 0x7f9918002830 2024-09-16T14:55:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f99240049b0 con 0x7f993409cb80 2024-09-16T14:55:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f9924043df0 con 0x7f993409cb80 2024-09-16T14:55:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727474473 0 0) 0x7f9918006590 con 0x7f9918002830 2024-09-16T14:55:36.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 10 monclient(hunting): my global_id is 234488 2024-09-16T14:55:36.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f993406e370 con 0x7f9918002830 2024-09-16T14:55:36.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9932d9d640 10 monclient: get_auth_request con 0x7f9914036130 auth_method 0 2024-09-16T14:55:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3690116852 0 0) 0x7f993406e370 con 0x7f9918002830 2024-09-16T14:55:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9918006590 con 0x7f9918002830 2024-09-16T14:55:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9928001260 con 0x7f9918002830 2024-09-16T14:55:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 10 monclient: dump: 2024-09-16T14:55:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:36.998 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:36.999 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:36.999 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:36.999 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:36.999 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:36.999 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:36.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1254713375 0 0) 0x7f9918006590 con 0x7f9918002830 2024-09-16T14:55:36.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f991800a0e0 con 0x7f9918002830 2024-09-16T14:55:36.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f9928001b70 con 0x7f9918002830 2024-09-16T14:55:36.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:36.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:36.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 >> v1:172.21.15.26:6790/0 conn(0x7f9918002830 legacy=0x7f9918004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:36.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.977+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 >> v1:172.21.15.26:6790/0 conn(0x7f9918002830 legacy=0x7f9918004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.981+0000 7f9938cfe640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:55:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.981+0000 7f9938cfe640 1 -- 172.21.15.26:0/3556127366 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9918009d60 con 0x7f99180090a0 2024-09-16T14:55:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198680280 0 0) 0x7f9918009d60 con 0x7f99180090a0 2024-09-16T14:55:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:37.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 10 monclient(hunting): my global_id is 234491 2024-09-16T14:55:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9918006590 con 0x7f99180090a0 2024-09-16T14:55:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 290111723 0 0) 0x7f9918006590 con 0x7f99180090a0 2024-09-16T14:55:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9918009d60 con 0x7f99180090a0 2024-09-16T14:55:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9928004ce0 con 0x7f99180090a0 2024-09-16T14:55:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:37.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:37.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 10 monclient: dump: 2024-09-16T14:55:37.002 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:37.002 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:37.002 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:37.002 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:37.002 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:37.002 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:37.003 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:37.003 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:37.003 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:37.003 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:37.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 375518185 0 0) 0x7f9918009d60 con 0x7f99180090a0 2024-09-16T14:55:37.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f991800c290 con 0x7f99180090a0 2024-09-16T14:55:37.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4203 (unknown 0 0 2503121256) 0x7f9928004f70 con 0x7f99180090a0 2024-09-16T14:55:37.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:55:37.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:37.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 >> v1:172.21.15.26:6790/0 conn(0x7f99180090a0 legacy=0x7f9918009510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:37.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9922ffd640 1 -- 172.21.15.26:0/3556127366 >> v1:172.21.15.26:6790/0 conn(0x7f99180090a0 legacy=0x7f9918009510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:37.004 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:55:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:55:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:55:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:55:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-09-16T14:55:37.004 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:55:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:55:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:55:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:55:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:55:37.005 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:55:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:37.006 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:55:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:55:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:37.007 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:55:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:55:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:55:37.008 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:55:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:55:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:55:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:55:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:55:37.009 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:55:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:55:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:55:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:55:37.010 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:55:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:55:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:37.011 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:37.012 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:55:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:55:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:55:37.014 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:55:37.014 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:37.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:37.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.014 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:37.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:55:37.015 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:37.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:37.015 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:37.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:55:37.015 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:55:37.016 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:37.016 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:37.016 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:37.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:55:37.017 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:55:37.017 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:37.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:37.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.017 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:37.018 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:55:37.018 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:37.018 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:37.018 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:37.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.019 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:55:37.019 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:55:37.019 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:37.019 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:37.019 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:37.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:55:37.020 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:55:37.020 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:37.020 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:37.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.021 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:37.021 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:55:37.021 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:37.021 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:37.022 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:37.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.022 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:55:37.022 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:55:37.022 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:37.022 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:37.023 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:37.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:37.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:37.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.023 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:55:37.023 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:55:37.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.024 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:37.024 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:37.024 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:37.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:37.024 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:37.025 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:55:37.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.025 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:37.025 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:37.025 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:55:37.025 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:37.026 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:37.026 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:55:37.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.026 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:37.026 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:37.026 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 7 2024-09-16T14:55:37.026 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:55:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.027 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:55:37.028 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:55:37.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9938cfe640 1 -- 172.21.15.26:0/3556127366 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9914036130 legacy=0x7f99140385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:37.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9938cfe640 10 monclient: shutdown 2024-09-16T14:55:37.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9938cfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:37.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9938cfe640 1 -- 172.21.15.26:0/3556127366 >> v1:172.21.15.26:6790/0 conn(0x7f993409cb80 legacy=0x7f993406d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:37.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9932d9d640 1 -- 172.21.15.26:0/3556127366 reap_dead start 2024-09-16T14:55:37.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9932d9d640 1 -- 172.21.15.26:0/3556127366 reap_dead start 2024-09-16T14:55:37.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9938cfe640 1 -- 172.21.15.26:0/3556127366 shutdown_connections 2024-09-16T14:55:37.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9938cfe640 1 -- 172.21.15.26:0/3556127366 >> 172.21.15.26:0/3556127366 conn(0x7f993409fb50 msgr2=0x7f9934131bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:37.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9938cfe640 1 -- 172.21.15.26:0/3556127366 shutdown_connections 2024-09-16T14:55:37.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:36.985+0000 7f9938cfe640 1 -- 172.21.15.26:0/3556127366 wait complete. 2024-09-16T14:55:37.030 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:55:37.030 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T14:55:37.030 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:55:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.153+0000 7f74f6453640 10 monclient: get_monmap_and_config 2024-09-16T14:55:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.153+0000 7f74f6453640 10 monclient: build_initial_monmap 2024-09-16T14:55:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.153+0000 7f74f6453640 10 monclient: monmap: 2024-09-16T14:55:37.158 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:37.160155+0000 2024-09-16T14:55:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:37.160155+0000 2024-09-16T14:55:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:37.159 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:37.159 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.153+0000 7f74f6453640 1 Processor -- start 2024-09-16T14:55:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.153+0000 7f74f6453640 1 -- start start 2024-09-16T14:55:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.153+0000 7f74f6453640 10 monclient: init 2024-09-16T14:55:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:37.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:55:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient(hunting): picked mon.noname-b con 0x7f74f009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient(hunting): picked mon.noname-a con 0x7f74f0132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient(hunting): picked mon.noname-c con 0x7f74f0138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f009d260 con 0x7f74f0132650 2024-09-16T14:55:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f0132a80 con 0x7f74f0138ff0 2024-09-16T14:55:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f009cf60 con 0x7f74f009cb80 2024-09-16T14:55:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74ef7fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f74f0132650 legacy=0x7f74f01378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient(hunting): authenticate will time out at 1761.280471s 2024-09-16T14:55:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74effff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f74f009cb80 0x7f74f00717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:37236/0 (socket says 172.21.15.26:37236) 2024-09-16T14:55:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74effff640 1 -- 172.21.15.26:0/562084866 learned_addr learned my addr 172.21.15.26:0/562084866 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 1 -- 172.21.15.26:0/562084866 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3039180894 0 0) 0x7f74f009cf60 con 0x7f74f009cb80 2024-09-16T14:55:37.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 10 monclient(hunting): my global_id is 234494 2024-09-16T14:55:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 1 -- 172.21.15.26:0/562084866 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74dc003530 con 0x7f74f009cb80 2024-09-16T14:55:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 1 -- 172.21.15.26:0/562084866 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3397300589 0 0) 0x7f74dc003530 con 0x7f74f009cb80 2024-09-16T14:55:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 1 -- 172.21.15.26:0/562084866 >> v1:172.21.15.189:6789/0 conn(0x7f74f0138ff0 legacy=0x7f74f013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 1 -- 172.21.15.26:0/562084866 >> v1:172.21.15.26:6789/0 conn(0x7f74f0132650 legacy=0x7f74f01378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:55:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 10 monclient: _finish_hunting 0 2024-09-16T14:55:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 1 monclient: found mon.noname-b 2024-09-16T14:55:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:55:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 1 -- 172.21.15.26:0/562084866 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f009f2d0 con 0x7f74f009cb80 2024-09-16T14:55:37.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 10 monclient: _finish_auth 0 2024-09-16T14:55:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 10 monclient: _check_auth_tickets 2024-09-16T14:55:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 5 monclient: authenticate success, global_id 234494 2024-09-16T14:55:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 1 -- 172.21.15.26:0/562084866 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74e0001660 con 0x7f74f009cb80 2024-09-16T14:55:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:55:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 10 monclient: dump: 2024-09-16T14:55:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:37.169 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:37.170 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:37.170 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:37.170 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:37.170 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:37.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:37.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 1 -- 172.21.15.26:0/562084866 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74e00018c0 con 0x7f74f009cb80 2024-09-16T14:55:37.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:37.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:37.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 1 -- 172.21.15.26:0/562084866 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74e0001c50 con 0x7f74f009cb80 2024-09-16T14:55:37.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 4 set_mon_vals no callback set 2024-09-16T14:55:37.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:37.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:37.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74eeffd640 10 monclient: dump: 2024-09-16T14:55:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:37.171 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:37.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient: shutdown 2024-09-16T14:55:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 1 -- 172.21.15.26:0/562084866 >> v1:172.21.15.26:6790/0 conn(0x7f74f009cb80 legacy=0x7f74f00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 1 -- 172.21.15.26:0/562084866 shutdown_connections 2024-09-16T14:55:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 1 -- 172.21.15.26:0/562084866 >> 172.21.15.26:0/562084866 conn(0x7f74f009fb50 msgr2=0x7f74f009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 1 -- 172.21.15.26:0/562084866 shutdown_connections 2024-09-16T14:55:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 1 -- 172.21.15.26:0/562084866 wait complete. 2024-09-16T14:55:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient: build_initial_monmap 2024-09-16T14:55:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient: monmap: 2024-09-16T14:55:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:37.172 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:37.173 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 1 Processor -- start 2024-09-16T14:55:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 1 -- start start 2024-09-16T14:55:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient: init 2024-09-16T14:55:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:55:37.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient(hunting): picked mon.noname-b con 0x7f74f009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:37.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient(hunting): picked mon.noname-a con 0x7f74f0132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:37.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient(hunting): picked mon.noname-c con 0x7f74f0138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:37.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f01387d0 con 0x7f74f0132650 2024-09-16T14:55:37.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f013c690 con 0x7f74f009cb80 2024-09-16T14:55:37.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f009a240 con 0x7f74f0138ff0 2024-09-16T14:55:37.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:37.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f6453640 10 monclient(hunting): authenticate will time out at 1761.283560s 2024-09-16T14:55:37.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74ef7fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f74f0132650 legacy=0x7f74f012df00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:55:37.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f49f0640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f74f0138ff0 0x7f74f012e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:37246/0 (socket says 172.21.15.26:37246) 2024-09-16T14:55:37.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74f49f0640 1 -- 172.21.15.26:0/1591822573 learned_addr learned my addr 172.21.15.26:0/1591822573 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:37.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227008199 0 0) 0x7f74f009a240 con 0x7f74f0138ff0 2024-09-16T14:55:37.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74ecff9640 10 monclient(hunting): _negotiate 2024-09-16T14:55:37.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:37.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:37.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74ecff9640 10 monclient(hunting): my global_id is 234497 2024-09-16T14:55:37.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.157+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d0003530 con 0x7f74f0138ff0 2024-09-16T14:55:37.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3078686854 0 0) 0x7f74d0003530 con 0x7f74f0138ff0 2024-09-16T14:55:37.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74f009a240 con 0x7f74f0138ff0 2024-09-16T14:55:37.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74e8001900 con 0x7f74f0138ff0 2024-09-16T14:55:37.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:37.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:55:37.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient(hunting): dump: 2024-09-16T14:55:37.175 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:37.175 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:37.175 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:37.176 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:37.176 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:37.176 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:37.176 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:37.176 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:37.176 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:37.176 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:37.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:37.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243405221 0 0) 0x7f74f013c690 con 0x7f74f009cb80 2024-09-16T14:55:37.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient(hunting): _negotiate 2024-09-16T14:55:37.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:37.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:37.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient(hunting): my global_id is 264175 2024-09-16T14:55:37.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d0003530 con 0x7f74f009cb80 2024-09-16T14:55:37.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3658008648 0 0) 0x7f74f009a240 con 0x7f74f0138ff0 2024-09-16T14:55:37.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 >> v1:172.21.15.189:6789/0 conn(0x7f74f009cb80 legacy=0x7f74f012d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:37.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 >> v1:172.21.15.26:6789/0 conn(0x7f74f0132650 legacy=0x7f74f012df00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:55:37.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient: _finish_hunting 0 2024-09-16T14:55:37.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 monclient: found mon.c 2024-09-16T14:55:37.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:37.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:37.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f012f440 con 0x7f74f0138ff0 2024-09-16T14:55:37.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient: _finish_auth 0 2024-09-16T14:55:37.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient: _check_auth_tickets 2024-09-16T14:55:37.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:37.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74f6453640 5 monclient: authenticate success, global_id 234497 2024-09-16T14:55:37.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74f6453640 10 monclient: _renew_subs 2024-09-16T14:55:37.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74f6453640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:37.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74f6453640 1 -- 172.21.15.26:0/1591822573 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f0134d90 con 0x7f74f0138ff0 2024-09-16T14:55:37.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74f6453640 10 monclient: _renew_subs 2024-09-16T14:55:37.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74f6453640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:37.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74f6453640 1 -- 172.21.15.26:0/1591822573 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f01d2b90 con 0x7f74f0138ff0 2024-09-16T14:55:37.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74e8003660 con 0x7f74f0138ff0 2024-09-16T14:55:37.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:37.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74e80039f0 con 0x7f74f0138ff0 2024-09-16T14:55:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient: dump: 2024-09-16T14:55:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:37.179 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:37.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74edffb640 4 set_mon_vals no callback set 2024-09-16T14:55:37.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74f6453640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:37.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f74e8003b50 con 0x7f74f0138ff0 2024-09-16T14:55:37.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74f6453640 1 -- 172.21.15.26:0/1591822573 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f74d4006590 con 0x7f74d4002830 2024-09-16T14:55:37.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f74e8043d30 con 0x7f74f0138ff0 2024-09-16T14:55:37.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ef7fe640 10 monclient: get_auth_request con 0x7f74d0036180 auth_method 0 2024-09-16T14:55:37.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382733822 0 0) 0x7f74d4006590 con 0x7f74d4002830 2024-09-16T14:55:37.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient(hunting): _negotiate 2024-09-16T14:55:37.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:37.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:37.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient(hunting): my global_id is 264178 2024-09-16T14:55:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f009a240 con 0x7f74d4002830 2024-09-16T14:55:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2623323651 0 0) 0x7f74f009a240 con 0x7f74d4002830 2024-09-16T14:55:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74d4006590 con 0x7f74d4002830 2024-09-16T14:55:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74e0001c90 con 0x7f74d4002830 2024-09-16T14:55:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:55:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 10 monclient: dump: 2024-09-16T14:55:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:37.182 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1814435633 0 0) 0x7f74d4006590 con 0x7f74d4002830 2024-09-16T14:55:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.161+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f74d400a0e0 con 0x7f74d4002830 2024-09-16T14:55:37.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.165+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f74e0001670 con 0x7f74d4002830 2024-09-16T14:55:37.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.165+0000 7f74ecff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:37.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.165+0000 7f74ecff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:37.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.165+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 >> v1:172.21.15.189:6789/0 conn(0x7f74d4002830 legacy=0x7f74d4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:37.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.165+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 >> v1:172.21.15.189:6789/0 conn(0x7f74d4002830 legacy=0x7f74d4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:37.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.169+0000 7f74f6453640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:55:37.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.169+0000 7f74f6453640 1 -- 172.21.15.26:0/1591822573 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f74d4009da0 con 0x7f74d40090c0 2024-09-16T14:55:37.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.169+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16968247 0 0) 0x7f74d4009da0 con 0x7f74d40090c0 2024-09-16T14:55:37.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.169+0000 7f74ecff9640 10 monclient(hunting): _negotiate 2024-09-16T14:55:37.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.169+0000 7f74ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:37.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.169+0000 7f74ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:37.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.169+0000 7f74ecff9640 10 monclient(hunting): my global_id is 264181 2024-09-16T14:55:37.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.169+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d4006590 con 0x7f74d40090c0 2024-09-16T14:55:37.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.169+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2379292276 0 0) 0x7f74d4006590 con 0x7f74d40090c0 2024-09-16T14:55:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.169+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74d4009da0 con 0x7f74d40090c0 2024-09-16T14:55:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.169+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f74e0004b80 con 0x7f74d40090c0 2024-09-16T14:55:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.169+0000 7f74ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.169+0000 7f74ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:55:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.169+0000 7f74ecff9640 10 monclient: dump: 2024-09-16T14:55:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:37.185 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:37.186 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.169+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4134396891 0 0) 0x7f74d4009da0 con 0x7f74d40090c0 2024-09-16T14:55:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.169+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f74d400c290 con 0x7f74d40090c0 2024-09-16T14:55:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.173+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3869 (unknown 0 0 997773894) 0x7f74e0005f30 con 0x7f74d40090c0 2024-09-16T14:55:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.173+0000 7f74ecff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:55:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.173+0000 7f74ecff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.173+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 >> v1:172.21.15.189:6789/0 conn(0x7f74d40090c0 legacy=0x7f74d4009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.173+0000 7f74ecff9640 1 -- 172.21.15.26:0/1591822573 >> v1:172.21.15.189:6789/0 conn(0x7f74d40090c0 legacy=0x7f74d4009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:37.187 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:55:37.187 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:55:37.187 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:55:37.187 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:55:37.187 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-09-16T14:55:37.187 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:55:37.187 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:55:37.187 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:55:37.187 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:37.188 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:55:37.188 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:37.188 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:55:37.188 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:55:37.188 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:37.188 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:37.188 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:37.188 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:37.188 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:37.188 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:37.188 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:37.189 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:37.189 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:37.189 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:37.189 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:37.189 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:55:37.189 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:55:37.189 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:37.189 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:37.189 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:37.189 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:37.189 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:37.189 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:37.189 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:37.190 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:37.190 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:37.190 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:37.190 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:37.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.190 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:55:37.190 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:55:37.190 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:55:37.190 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:55:37.190 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:55:37.190 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:55:37.190 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:37.190 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:37.191 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:55:37.191 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:55:37.191 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:55:37.191 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:55:37.191 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:55:37.191 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:55:37.191 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:55:37.191 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:37.191 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:55:37.191 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:37.191 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:37.191 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:37.191 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:37.191 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:37.191 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:37.192 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:37.192 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:37.192 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:37.192 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:37.192 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:37.192 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:55:37.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.192 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:55:37.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.192 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:55:37.192 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:55:37.192 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:37.192 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:37.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.193 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:37.193 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:55:37.193 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:37.193 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:37.193 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:37.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.193 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:55:37.193 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:55:37.193 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:37.193 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:37.193 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:37.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.194 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:55:37.194 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:55:37.194 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:37.194 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:37.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.194 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:37.194 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:55:37.194 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:37.194 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:37.194 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:37.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.194 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:55:37.194 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:55:37.195 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:37.195 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:37.195 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:37.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.195 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:55:37.195 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:55:37.195 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:37.195 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:37.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.195 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:37.195 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:55:37.195 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:37.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:37.196 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:37.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.196 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:55:37.196 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:55:37.196 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:37.196 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:37.196 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:37.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:37.197 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:37.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.197 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:55:37.197 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:55:37.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.197 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:37.197 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:37.197 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:37.197 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:37.197 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:37.197 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:55:37.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:37.198 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:37.198 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:37.198 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:37.198 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:37.198 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:37.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:37.198 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:55:37.198 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:55:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.173+0000 7f74f6453640 1 -- 172.21.15.26:0/1591822573 >> v1:172.21.15.189:6812/3196161582 conn(0x7f74d0036180 legacy=0x7f74d0038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.173+0000 7f74f6453640 10 monclient: shutdown 2024-09-16T14:55:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.173+0000 7f74f6453640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.173+0000 7f74f6453640 1 -- 172.21.15.26:0/1591822573 >> v1:172.21.15.26:6790/0 conn(0x7f74f0138ff0 legacy=0x7f74f012e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.173+0000 7f74f49f0640 1 -- 172.21.15.26:0/1591822573 reap_dead start 2024-09-16T14:55:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.173+0000 7f74f49f0640 1 -- 172.21.15.26:0/1591822573 reap_dead start 2024-09-16T14:55:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.173+0000 7f74f6453640 1 -- 172.21.15.26:0/1591822573 shutdown_connections 2024-09-16T14:55:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.173+0000 7f74f6453640 1 -- 172.21.15.26:0/1591822573 >> 172.21.15.26:0/1591822573 conn(0x7f74f009fb50 msgr2=0x7f74f009bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.173+0000 7f74f6453640 1 -- 172.21.15.26:0/1591822573 shutdown_connections 2024-09-16T14:55:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:37.173+0000 7f74f6453640 1 -- 172.21.15.26:0/1591822573 wait complete. 2024-09-16T14:55:37.201 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:55:37.201 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:55:57.202 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:55:57.202 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T14:55:57.202 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T14:55:57.202 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T14:55:57.206 INFO:tasks.ceph.mon.a:Started 2024-09-16T14:55:57.206 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:55:57.206 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:55:57.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.329+0000 7fa136f27640 10 monclient: get_monmap_and_config 2024-09-16T14:55:57.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.329+0000 7fa136f27640 10 monclient: build_initial_monmap 2024-09-16T14:55:57.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.329+0000 7fa136f27640 10 monclient: monmap: 2024-09-16T14:55:57.334 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:57.334 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.334 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:57.336258+0000 2024-09-16T14:55:57.334 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:57.336258+0000 2024-09-16T14:55:57.335 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:57.335 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.335 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:57.335 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:57.335 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:57.335 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.329+0000 7fa136f27640 1 Processor -- start 2024-09-16T14:55:57.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.329+0000 7fa136f27640 1 -- start start 2024-09-16T14:55:57.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.329+0000 7fa136f27640 10 monclient: init 2024-09-16T14:55:57.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.329+0000 7fa136f27640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:57.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.329+0000 7fa136f27640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:55:57.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.329+0000 7fa136f27640 10 monclient(hunting): picked mon.noname-c con 0x7fa1300a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:57.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.329+0000 7fa136f27640 10 monclient(hunting): picked mon.noname-a con 0x7fa1300a1fe0 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:57.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.329+0000 7fa136f27640 10 monclient(hunting): picked mon.noname-b con 0x7fa1300a2b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:57.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.329+0000 7fa136f27640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa130098f10 con 0x7fa1300a1fe0 2024-09-16T14:55:57.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.329+0000 7fa136f27640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1300994d0 con 0x7fa1300a1c00 2024-09-16T14:55:57.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.329+0000 7fa136f27640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1300a34c0 con 0x7fa1300a2b00 2024-09-16T14:55:57.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.329+0000 7fa136f27640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:57.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.329+0000 7fa136f27640 10 monclient(hunting): authenticate will time out at 1781.455913s 2024-09-16T14:55:57.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa1354c4640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa1300a2b00 0x7fa13013cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35304/0 (socket says 172.21.15.26:35304) 2024-09-16T14:55:57.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa1354c4640 1 -- 172.21.15.26:0/3525212445 learned_addr learned my addr 172.21.15.26:0/3525212445 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:57.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 1 -- 172.21.15.26:0/3525212445 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32113 0 0) 0x7fa1300a34c0 con 0x7fa1300a2b00 2024-09-16T14:55:57.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient(hunting): my global_id is 234500 2024-09-16T14:55:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 1 -- 172.21.15.26:0/3525212445 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa120003530 con 0x7fa1300a2b00 2024-09-16T14:55:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 1 -- 172.21.15.26:0/3525212445 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171911716 0 0) 0x7fa1300994d0 con 0x7fa1300a1c00 2024-09-16T14:55:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient(hunting): my global_id is 264184 2024-09-16T14:55:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 1 -- 172.21.15.26:0/3525212445 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1300a34c0 con 0x7fa1300a1c00 2024-09-16T14:55:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 1 -- 172.21.15.26:0/3525212445 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2964093669 0 0) 0x7fa120003530 con 0x7fa1300a2b00 2024-09-16T14:55:57.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 1 -- 172.21.15.26:0/3525212445 >> v1:172.21.15.189:6789/0 conn(0x7fa1300a1c00 legacy=0x7fa130099bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 1 -- 172.21.15.26:0/3525212445 >> v1:172.21.15.26:6789/0 conn(0x7fa1300a1fe0 legacy=0x7fa1300a23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:55:57.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 1 monclient: found mon.noname-b 2024-09-16T14:55:57.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:57.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:55:57.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 1 -- 172.21.15.26:0/3525212445 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13006c600 con 0x7fa1300a2b00 2024-09-16T14:55:57.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:55:57.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:55:57.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:57.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 1 -- 172.21.15.26:0/3525212445 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa1280019a0 con 0x7fa1300a2b00 2024-09-16T14:55:57.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:55:57.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient: dump: 2024-09-16T14:55:57.342 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.342 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.342 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.342 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.342 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.342 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.342 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.342 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.342 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.342 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 5 monclient: authenticate success, global_id 234500 2024-09-16T14:55:57.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:57.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 1 -- 172.21.15.26:0/3525212445 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa128001c80 con 0x7fa1300a2b00 2024-09-16T14:55:57.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:57.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 1 -- 172.21.15.26:0/3525212445 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa128002050 con 0x7fa1300a2b00 2024-09-16T14:55:57.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:57.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12f7fe640 10 monclient: dump: 2024-09-16T14:55:57.343 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.343 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.343 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.343 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.343 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.343 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.343 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.343 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.344 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.344 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:57.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 4 set_mon_vals no callback set 2024-09-16T14:55:57.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 10 monclient: shutdown 2024-09-16T14:55:57.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:57.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 1 -- 172.21.15.26:0/3525212445 >> v1:172.21.15.26:6790/0 conn(0x7fa1300a2b00 legacy=0x7fa13013cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 1 -- 172.21.15.26:0/3525212445 shutdown_connections 2024-09-16T14:55:57.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 1 -- 172.21.15.26:0/3525212445 >> 172.21.15.26:0/3525212445 conn(0x7fa1300a3d00 msgr2=0x7fa1300a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:57.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 1 -- 172.21.15.26:0/3525212445 shutdown_connections 2024-09-16T14:55:57.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 1 -- 172.21.15.26:0/3525212445 wait complete. 2024-09-16T14:55:57.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 10 monclient: build_initial_monmap 2024-09-16T14:55:57.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 10 monclient: monmap: 2024-09-16T14:55:57.344 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:57.345 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:57.345 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:57.345 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:57.345 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:57.345 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:57.345 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:57.345 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:57.345 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:57.345 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 1 Processor -- start 2024-09-16T14:55:57.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 1 -- start start 2024-09-16T14:55:57.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 10 monclient: init 2024-09-16T14:55:57.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:57.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:55:57.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 10 monclient(hunting): picked mon.noname-b con 0x7fa1300a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:57.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 10 monclient(hunting): picked mon.noname-a con 0x7fa1300a1fe0 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:57.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 10 monclient(hunting): picked mon.noname-c con 0x7fa1300a2b00 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:57.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa130134120 con 0x7fa1300a1fe0 2024-09-16T14:55:57.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1301311f0 con 0x7fa1300a1c00 2024-09-16T14:55:57.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa130134cf0 con 0x7fa1300a2b00 2024-09-16T14:55:57.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:57.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa136f27640 10 monclient(hunting): authenticate will time out at 1781.459732s 2024-09-16T14:55:57.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa1354c4640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa1300a2b00 0x7fa1301d1aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35320/0 (socket says 172.21.15.26:35320) 2024-09-16T14:55:57.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa12ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa1300a1fe0 0x7fa130130ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60138/0 (socket says 172.21.15.26:60138) 2024-09-16T14:55:57.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.333+0000 7fa1354c4640 1 -- 172.21.15.26:0/4291761768 learned_addr learned my addr 172.21.15.26:0/4291761768 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:57.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167856889 0 0) 0x7fa130134cf0 con 0x7fa1300a2b00 2024-09-16T14:55:57.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient(hunting): my global_id is 234503 2024-09-16T14:55:57.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa114003530 con 0x7fa1300a2b00 2024-09-16T14:55:57.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273326821 0 0) 0x7fa130134120 con 0x7fa1300a1fe0 2024-09-16T14:55:57.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient(hunting): my global_id is 274098 2024-09-16T14:55:57.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa130134cf0 con 0x7fa1300a1fe0 2024-09-16T14:55:57.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1897135685 0 0) 0x7fa130134cf0 con 0x7fa1300a1fe0 2024-09-16T14:55:57.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa130134120 con 0x7fa1300a1fe0 2024-09-16T14:55:57.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa11c0017f0 con 0x7fa1300a1fe0 2024-09-16T14:55:57.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:55:57.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient(hunting): dump: 2024-09-16T14:55:57.351 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.351 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.351 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.351 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.351 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.351 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.352 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.352 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.352 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.352 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:57.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2252929420 0 0) 0x7fa114003530 con 0x7fa1300a2b00 2024-09-16T14:55:57.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa130134cf0 con 0x7fa1300a2b00 2024-09-16T14:55:57.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa128001f90 con 0x7fa1300a2b00 2024-09-16T14:55:57.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:57.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient(hunting): dump: 2024-09-16T14:55:57.352 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.352 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.353 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3255595194 0 0) 0x7fa130134120 con 0x7fa1300a1fe0 2024-09-16T14:55:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 >> v1:172.21.15.26:6790/0 conn(0x7fa1300a2b00 legacy=0x7fa1301d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 >> v1:172.21.15.189:6789/0 conn(0x7fa1300a1c00 legacy=0x7fa1301303d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:55:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 monclient: found mon.a 2024-09-16T14:55:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa130133880 con 0x7fa1300a1fe0 2024-09-16T14:55:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:55:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:55:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa136f27640 5 monclient: authenticate success, global_id 274098 2024-09-16T14:55:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa136f27640 10 monclient: _renew_subs 2024-09-16T14:55:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa136f27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa136f27640 1 -- 172.21.15.26:0/4291761768 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa130133b60 con 0x7fa1300a1fe0 2024-09-16T14:55:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa136f27640 10 monclient: _renew_subs 2024-09-16T14:55:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa136f27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa136f27640 1 -- 172.21.15.26:0/4291761768 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa13009e3f0 con 0x7fa1300a1fe0 2024-09-16T14:55:57.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa11c003620 con 0x7fa1300a1fe0 2024-09-16T14:55:57.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:57.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa11c0039b0 con 0x7fa1300a1fe0 2024-09-16T14:55:57.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:57.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 10 monclient: dump: 2024-09-16T14:55:57.355 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.355 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.355 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.355 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.355 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.355 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.355 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.355 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.355 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.356 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12effd640 4 set_mon_vals no callback set 2024-09-16T14:55:57.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa11c003b10 con 0x7fa1300a1fe0 2024-09-16T14:55:57.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fa11c043ca0 con 0x7fa1300a1fe0 2024-09-16T14:55:57.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa136f27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:57.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa136f27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:57.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa136f27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:57.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa136f27640 1 -- 172.21.15.26:0/4291761768 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa110005050 con 0x7fa1300a1fe0 2024-09-16T14:55:57.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.337+0000 7fa134cc3640 10 monclient: get_auth_request con 0x7fa1140362c0 auth_method 0 2024-09-16T14:55:57.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.341+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa11c014640 con 0x7fa1300a1fe0 2024-09-16T14:55:57.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.341+0000 7fa12d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:57.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.341+0000 7fa12d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:57.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.557+0000 7fa136f27640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:55:57.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.557+0000 7fa136f27640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:55:57.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.557+0000 7fa136f27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.557+0000 7fa136f27640 1 -- 172.21.15.26:0/4291761768 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa110005880 con 0x7fa1300a1fe0 2024-09-16T14:55:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.557+0000 7fa12d7fa640 1 -- 172.21.15.26:0/4291761768 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 4156807632) 0x7fa11c012ab0 con 0x7fa1300a1fe0 2024-09-16T14:55:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.557+0000 7fa12d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:55:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.557+0000 7fa12d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:57.563 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:55:57.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.561+0000 7fa136f27640 1 -- 172.21.15.26:0/4291761768 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa1140362c0 legacy=0x7fa114038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.561+0000 7fa136f27640 10 monclient: shutdown 2024-09-16T14:55:57.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.561+0000 7fa136f27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:57.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.561+0000 7fa136f27640 1 -- 172.21.15.26:0/4291761768 >> v1:172.21.15.26:6789/0 conn(0x7fa1300a1fe0 legacy=0x7fa130130ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.561+0000 7fa136f27640 1 -- 172.21.15.26:0/4291761768 shutdown_connections 2024-09-16T14:55:57.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.561+0000 7fa136f27640 1 -- 172.21.15.26:0/4291761768 >> 172.21.15.26:0/4291761768 conn(0x7fa1300a3d00 msgr2=0x7fa1300a5f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:57.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.561+0000 7fa136f27640 1 -- 172.21.15.26:0/4291761768 shutdown_connections 2024-09-16T14:55:57.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.561+0000 7fa136f27640 1 -- 172.21.15.26:0/4291761768 wait complete. 2024-09-16T14:55:57.580 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:55:57.580 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:55:57.580 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:55:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 10 monclient: get_monmap_and_config 2024-09-16T14:55:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 10 monclient: build_initial_monmap 2024-09-16T14:55:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 10 monclient: monmap: 2024-09-16T14:55:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:57.702129+0000 2024-09-16T14:55:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:57.702129+0000 2024-09-16T14:55:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:57.701 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 1 Processor -- start 2024-09-16T14:55:57.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 1 -- start start 2024-09-16T14:55:57.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 10 monclient: init 2024-09-16T14:55:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:55:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 10 monclient(hunting): picked mon.noname-a con 0x7f86f409cdc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 10 monclient(hunting): picked mon.noname-b con 0x7f86f409e7a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 10 monclient(hunting): picked mon.noname-c con 0x7f86f409d1a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f409d5b0 con 0x7f86f409cdc0 2024-09-16T14:55:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f4131a90 con 0x7f86f409d1a0 2024-09-16T14:55:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f409aff0 con 0x7f86f409e7a0 2024-09-16T14:55:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 10 monclient(hunting): authenticate will time out at 1781.822020s 2024-09-16T14:55:57.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9d11640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f86f409e7a0 0x7f86f4131380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35326/0 (socket says 172.21.15.26:35326) 2024-09-16T14:55:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fa512640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f86f409cdc0 0x7f86f4130c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60148/0 (socket says 172.21.15.26:60148) 2024-09-16T14:55:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9d11640 1 -- 172.21.15.26:0/1066435021 learned_addr learned my addr 172.21.15.26:0/1066435021 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 1 -- 172.21.15.26:0/1066435021 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681293806 0 0) 0x7f86f409aff0 con 0x7f86f409e7a0 2024-09-16T14:55:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient(hunting): my global_id is 234506 2024-09-16T14:55:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 1 -- 172.21.15.26:0/1066435021 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86dc003450 con 0x7f86f409e7a0 2024-09-16T14:55:57.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 1 -- 172.21.15.26:0/1066435021 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485537441 0 0) 0x7f86f409d5b0 con 0x7f86f409cdc0 2024-09-16T14:55:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient(hunting): my global_id is 274101 2024-09-16T14:55:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 1 -- 172.21.15.26:0/1066435021 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f409aff0 con 0x7f86f409cdc0 2024-09-16T14:55:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 1 -- 172.21.15.26:0/1066435021 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 34062282 0 0) 0x7f86f409aff0 con 0x7f86f409cdc0 2024-09-16T14:55:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 1 -- 172.21.15.26:0/1066435021 >> v1:172.21.15.26:6790/0 conn(0x7f86f409e7a0 legacy=0x7f86f4131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 1 -- 172.21.15.26:0/1066435021 >> v1:172.21.15.189:6789/0 conn(0x7f86f409d1a0 legacy=0x7f86f4139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient: _finish_hunting 0 2024-09-16T14:55:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 1 monclient: found mon.noname-a 2024-09-16T14:55:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:55:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 1 -- 172.21.15.26:0/1066435021 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f409b2d0 con 0x7f86f409cdc0 2024-09-16T14:55:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient: _finish_auth 0 2024-09-16T14:55:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient: _check_auth_tickets 2024-09-16T14:55:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 1 -- 172.21.15.26:0/1066435021 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f86ec001580 con 0x7f86f409cdc0 2024-09-16T14:55:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 5 monclient: authenticate success, global_id 274101 2024-09-16T14:55:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:55:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient: dump: 2024-09-16T14:55:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.714 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 1 -- 172.21.15.26:0/1066435021 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86ec0017b0 con 0x7f86f409cdc0 2024-09-16T14:55:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 1 -- 172.21.15.26:0/1066435021 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f86ec001b80 con 0x7f86f409cdc0 2024-09-16T14:55:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86f9510640 10 monclient: dump: 2024-09-16T14:55:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.716 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 4 set_mon_vals no callback set 2024-09-16T14:55:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 10 monclient: shutdown 2024-09-16T14:55:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 1 -- 172.21.15.26:0/1066435021 >> v1:172.21.15.26:6789/0 conn(0x7f86f409cdc0 legacy=0x7f86f4130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 1 -- 172.21.15.26:0/1066435021 shutdown_connections 2024-09-16T14:55:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 1 -- 172.21.15.26:0/1066435021 >> 172.21.15.26:0/1066435021 conn(0x7f86f409fb50 msgr2=0x7f86f409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 1 -- 172.21.15.26:0/1066435021 shutdown_connections 2024-09-16T14:55:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.697+0000 7f86fc776640 1 -- 172.21.15.26:0/1066435021 wait complete. 2024-09-16T14:55:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 10 monclient: build_initial_monmap 2024-09-16T14:55:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 10 monclient: monmap: 2024-09-16T14:55:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:57.717 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 1 Processor -- start 2024-09-16T14:55:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 1 -- start start 2024-09-16T14:55:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 10 monclient: init 2024-09-16T14:55:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:55:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 10 monclient(hunting): picked mon.noname-b con 0x7f86f409cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 10 monclient(hunting): picked mon.noname-c con 0x7f86f409d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 10 monclient(hunting): picked mon.noname-a con 0x7f86f409e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f411c700 con 0x7f86f409e7a0 2024-09-16T14:55:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f411ada0 con 0x7f86f409cdc0 2024-09-16T14:55:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f40a2600 con 0x7f86f409d1a0 2024-09-16T14:55:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 10 monclient(hunting): authenticate will time out at 1781.825250s 2024-09-16T14:55:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fad13640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f86f409e7a0 0x7f86f41cd9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60152/0 (socket says 172.21.15.26:60152) 2024-09-16T14:55:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fad13640 1 -- 172.21.15.26:0/282525920 learned_addr learned my addr 172.21.15.26:0/282525920 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975272398 0 0) 0x7f86f411c700 con 0x7f86f409e7a0 2024-09-16T14:55:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): my global_id is 274104 2024-09-16T14:55:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e0003530 con 0x7f86f409e7a0 2024-09-16T14:55:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548408040 0 0) 0x7f86f40a2600 con 0x7f86f409d1a0 2024-09-16T14:55:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): my global_id is 234509 2024-09-16T14:55:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f411c700 con 0x7f86f409d1a0 2024-09-16T14:55:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458399513 0 0) 0x7f86f411ada0 con 0x7f86f409cdc0 2024-09-16T14:55:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): my global_id is 264190 2024-09-16T14:55:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f40a2600 con 0x7f86f409cdc0 2024-09-16T14:55:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 801373907 0 0) 0x7f86e0003530 con 0x7f86f409e7a0 2024-09-16T14:55:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86f411ada0 con 0x7f86f409e7a0 2024-09-16T14:55:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1778821359 0 0) 0x7f86f411c700 con 0x7f86f409d1a0 2024-09-16T14:55:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86e0003530 con 0x7f86f409d1a0 2024-09-16T14:55:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f86e4001970 con 0x7f86f409e7a0 2024-09-16T14:55:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:55:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): dump: 2024-09-16T14:55:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.721 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f86e8001be0 con 0x7f86f409d1a0 2024-09-16T14:55:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): dump: 2024-09-16T14:55:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.722 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2644738917 0 0) 0x7f86f411ada0 con 0x7f86f409e7a0 2024-09-16T14:55:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 >> v1:172.21.15.26:6790/0 conn(0x7f86f409d1a0 legacy=0x7f86f41ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 >> v1:172.21.15.189:6789/0 conn(0x7f86f409cdc0 legacy=0x7f86f412dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:55:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 monclient: found mon.a 2024-09-16T14:55:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f40a20b0 con 0x7f86f409e7a0 2024-09-16T14:55:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:55:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:55:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 5 monclient: authenticate success, global_id 274104 2024-09-16T14:55:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 10 monclient: _renew_subs 2024-09-16T14:55:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 1 -- 172.21.15.26:0/282525920 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f411be60 con 0x7f86f409e7a0 2024-09-16T14:55:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 10 monclient: _renew_subs 2024-09-16T14:55:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 1 -- 172.21.15.26:0/282525920 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f4119b70 con 0x7f86f409e7a0 2024-09-16T14:55:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86e4001cd0 con 0x7f86f409e7a0 2024-09-16T14:55:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f86e4003a70 con 0x7f86f409e7a0 2024-09-16T14:55:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f8d0f640 4 set_mon_vals no callback set 2024-09-16T14:55:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient: dump: 2024-09-16T14:55:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.725 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86fc776640 1 -- 172.21.15.26:0/282525920 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f86d4006590 con 0x7f86d4002830 2024-09-16T14:55:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f86e4003bd0 con 0x7f86f409e7a0 2024-09-16T14:55:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f86e4043ca0 con 0x7f86f409e7a0 2024-09-16T14:55:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016138097 0 0) 0x7f86d4006590 con 0x7f86d4002830 2024-09-16T14:55:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient(hunting): my global_id is 274107 2024-09-16T14:55:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f411ada0 con 0x7f86d4002830 2024-09-16T14:55:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 194741153 0 0) 0x7f86f411ada0 con 0x7f86d4002830 2024-09-16T14:55:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86d4006590 con 0x7f86d4002830 2024-09-16T14:55:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f86ec001210 con 0x7f86d4002830 2024-09-16T14:55:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f2ffd640 10 monclient: dump: 2024-09-16T14:55:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.726 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.727 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.727 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.727 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.727 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.727 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.727 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.727 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.727 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.701+0000 7f86f9d11640 10 monclient: get_auth_request con 0x7f86e00363e0 auth_method 0 2024-09-16T14:55:57.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.705+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1605975875 0 0) 0x7f86d4006590 con 0x7f86d4002830 2024-09-16T14:55:57.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.705+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86d400a0e0 con 0x7f86d4002830 2024-09-16T14:55:57.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.705+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f86ec001540 con 0x7f86d4002830 2024-09-16T14:55:57.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.705+0000 7f86f2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:57.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.705+0000 7f86f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:57.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.705+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 >> v1:172.21.15.26:6789/0 conn(0x7f86d4002830 legacy=0x7f86d4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.705+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 >> v1:172.21.15.26:6789/0 conn(0x7f86d4002830 legacy=0x7f86d4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:57.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86fc776640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:55:57.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86fc776640 1 -- 172.21.15.26:0/282525920 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f86d4009da0 con 0x7f86d40090c0 2024-09-16T14:55:57.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075678094 0 0) 0x7f86d4009da0 con 0x7f86d40090c0 2024-09-16T14:55:57.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 10 monclient(hunting): my global_id is 274110 2024-09-16T14:55:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86d4006590 con 0x7f86d40090c0 2024-09-16T14:55:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3779041943 0 0) 0x7f86d4006590 con 0x7f86d40090c0 2024-09-16T14:55:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86d4009da0 con 0x7f86d40090c0 2024-09-16T14:55:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f86ec004bc0 con 0x7f86d40090c0 2024-09-16T14:55:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 10 monclient: dump: 2024-09-16T14:55:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.730 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.731 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3073094350 0 0) 0x7f86d4009da0 con 0x7f86d40090c0 2024-09-16T14:55:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f86d400c290 con 0x7f86d40090c0 2024-09-16T14:55:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 2001841665) 0x7f86ec005f30 con 0x7f86d40090c0 2024-09-16T14:55:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:55:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 >> v1:172.21.15.26:6789/0 conn(0x7f86d40090c0 legacy=0x7f86d4009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86f2ffd640 1 -- 172.21.15.26:0/282525920 >> v1:172.21.15.26:6789/0 conn(0x7f86d40090c0 legacy=0x7f86d4009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:57.732 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:55:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:55:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:55:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:55:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 126, 2024-09-16T14:55:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:55:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:55:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:55:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:55:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:55:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:55:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:55:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:55:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:55:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:55:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:55:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:55:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:55:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:55:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:55:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:55:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:55:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:55:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:55:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:55:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:55:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:55:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:55:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:55:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:55:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:55:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:55:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:55:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:55:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:55:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:55:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:55:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:55:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:55:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:55:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:55:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:55:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:55:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:55:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:55:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:55:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:55:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:55:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:55:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:55:57.743 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:55:57.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.709+0000 7f86fc776640 1 -- 172.21.15.26:0/282525920 >> v1:172.21.15.189:6812/3196161582 conn(0x7f86e00363e0 legacy=0x7f86e00388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.713+0000 7f86fc776640 10 monclient: shutdown 2024-09-16T14:55:57.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.713+0000 7f86fc776640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:57.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.713+0000 7f86fc776640 1 -- 172.21.15.26:0/282525920 >> v1:172.21.15.26:6789/0 conn(0x7f86f409e7a0 legacy=0x7f86f41cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.713+0000 7f86fad13640 1 -- 172.21.15.26:0/282525920 reap_dead start 2024-09-16T14:55:57.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.713+0000 7f86fad13640 1 -- 172.21.15.26:0/282525920 reap_dead start 2024-09-16T14:55:57.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.713+0000 7f86fc776640 1 -- 172.21.15.26:0/282525920 shutdown_connections 2024-09-16T14:55:57.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.713+0000 7f86fc776640 1 -- 172.21.15.26:0/282525920 >> 172.21.15.26:0/282525920 conn(0x7f86f409fb50 msgr2=0x7f86f406ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:57.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.713+0000 7f86fc776640 1 -- 172.21.15.26:0/282525920 shutdown_connections 2024-09-16T14:55:57.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.713+0000 7f86fc776640 1 -- 172.21.15.26:0/282525920 wait complete. 2024-09-16T14:55:57.745 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:55:57.745 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:55:57.745 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:55:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 10 monclient: get_monmap_and_config 2024-09-16T14:55:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 10 monclient: build_initial_monmap 2024-09-16T14:55:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 10 monclient: monmap: 2024-09-16T14:55:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:57.862291+0000 2024-09-16T14:55:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:57.862291+0000 2024-09-16T14:55:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:57.861 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 1 Processor -- start 2024-09-16T14:55:57.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 1 -- start start 2024-09-16T14:55:57.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 10 monclient: init 2024-09-16T14:55:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:55:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 10 monclient(hunting): picked mon.noname-b con 0x7f802409a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 10 monclient(hunting): picked mon.noname-c con 0x7f802409d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 10 monclient(hunting): picked mon.noname-a con 0x7f802409b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802409dc60 con 0x7f802409b040 2024-09-16T14:55:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8024099ae0 con 0x7f802409d1b0 2024-09-16T14:55:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802409b8f0 con 0x7f802409a550 2024-09-16T14:55:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 10 monclient(hunting): authenticate will time out at 1781.982341s 2024-09-16T14:55:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f80237fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f802409b040 0x7f802413a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60186/0 (socket says 172.21.15.26:60186) 2024-09-16T14:55:57.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f80237fe640 1 -- 172.21.15.26:0/337471593 learned_addr learned my addr 172.21.15.26:0/337471593 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 1 -- 172.21.15.26:0/337471593 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756538117 0 0) 0x7f802409dc60 con 0x7f802409b040 2024-09-16T14:55:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient(hunting): my global_id is 274113 2024-09-16T14:55:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 1 -- 172.21.15.26:0/337471593 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8010005740 con 0x7f802409b040 2024-09-16T14:55:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 1 -- 172.21.15.26:0/337471593 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717546350 0 0) 0x7f802409b8f0 con 0x7f802409a550 2024-09-16T14:55:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient(hunting): my global_id is 234512 2024-09-16T14:55:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 1 -- 172.21.15.26:0/337471593 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f802409dc60 con 0x7f802409a550 2024-09-16T14:55:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 1 -- 172.21.15.26:0/337471593 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3948048374 0 0) 0x7f8010005740 con 0x7f802409b040 2024-09-16T14:55:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 1 -- 172.21.15.26:0/337471593 >> v1:172.21.15.26:6790/0 conn(0x7f802409a550 legacy=0x7f802409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 1 -- 172.21.15.26:0/337471593 >> v1:172.21.15.189:6789/0 conn(0x7f802409d1b0 legacy=0x7f8024136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:55:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 1 monclient: found mon.noname-a 2024-09-16T14:55:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:55:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 1 -- 172.21.15.26:0/337471593 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802409bbd0 con 0x7f802409b040 2024-09-16T14:55:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient: _finish_auth 0 2024-09-16T14:55:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:55:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 1 -- 172.21.15.26:0/337471593 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f801c001620 con 0x7f802409b040 2024-09-16T14:55:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:55:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient: dump: 2024-09-16T14:55:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.872 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 5 monclient: authenticate success, global_id 274113 2024-09-16T14:55:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 1 -- 172.21.15.26:0/337471593 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f801c0018c0 con 0x7f802409b040 2024-09-16T14:55:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 1 -- 172.21.15.26:0/337471593 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f801c001c50 con 0x7f802409b040 2024-09-16T14:55:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 4 set_mon_vals no callback set 2024-09-16T14:55:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8021ffb640 10 monclient: dump: 2024-09-16T14:55:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.874 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 10 monclient: shutdown 2024-09-16T14:55:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 1 -- 172.21.15.26:0/337471593 >> v1:172.21.15.26:6789/0 conn(0x7f802409b040 legacy=0x7f802413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 1 -- 172.21.15.26:0/337471593 shutdown_connections 2024-09-16T14:55:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 1 -- 172.21.15.26:0/337471593 >> 172.21.15.26:0/337471593 conn(0x7f802409fb50 msgr2=0x7f802409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 1 -- 172.21.15.26:0/337471593 shutdown_connections 2024-09-16T14:55:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.857+0000 7f8029474640 1 -- 172.21.15.26:0/337471593 wait complete. 2024-09-16T14:55:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 10 monclient: build_initial_monmap 2024-09-16T14:55:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 10 monclient: monmap: 2024-09-16T14:55:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:57.875 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 1 Processor -- start 2024-09-16T14:55:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 1 -- start start 2024-09-16T14:55:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 10 monclient: init 2024-09-16T14:55:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:55:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 10 monclient(hunting): picked mon.noname-c con 0x7f802409a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 10 monclient(hunting): picked mon.noname-a con 0x7f802409b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 10 monclient(hunting): picked mon.noname-b con 0x7f802409d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8024071280 con 0x7f802409b040 2024-09-16T14:55:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80241a7380 con 0x7f802409d1b0 2024-09-16T14:55:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80241a5970 con 0x7f802409a550 2024-09-16T14:55:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 10 monclient(hunting): authenticate will time out at 1781.985319s 2024-09-16T14:55:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8022ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f802409a550 0x7f80241c92e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35360/0 (socket says 172.21.15.26:35360) 2024-09-16T14:55:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f80227fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f802409b040 0x7f80241b89b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60198/0 (socket says 172.21.15.26:60198) 2024-09-16T14:55:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8022ffd640 1 -- 172.21.15.26:0/390759586 learned_addr learned my addr 172.21.15.26:0/390759586 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824109557 0 0) 0x7f8024071280 con 0x7f802409b040 2024-09-16T14:55:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): my global_id is 274116 2024-09-16T14:55:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8004003530 con 0x7f802409b040 2024-09-16T14:55:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349384943 0 0) 0x7f80241a5970 con 0x7f802409a550 2024-09-16T14:55:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): my global_id is 234515 2024-09-16T14:55:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8024071280 con 0x7f802409a550 2024-09-16T14:55:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3850981167 0 0) 0x7f8024071280 con 0x7f802409a550 2024-09-16T14:55:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80241a5970 con 0x7f802409a550 2024-09-16T14:55:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2820777417 0 0) 0x7f8004003530 con 0x7f802409b040 2024-09-16T14:55:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8024071280 con 0x7f802409b040 2024-09-16T14:55:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8014001c50 con 0x7f802409a550 2024-09-16T14:55:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:55:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): dump: 2024-09-16T14:55:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.879 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f800c001840 con 0x7f802409b040 2024-09-16T14:55:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): dump: 2024-09-16T14:55:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.880 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355815018 0 0) 0x7f80241a7380 con 0x7f802409d1b0 2024-09-16T14:55:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): my global_id is 264196 2024-09-16T14:55:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8004003530 con 0x7f802409d1b0 2024-09-16T14:55:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1176921898 0 0) 0x7f8024071280 con 0x7f802409b040 2024-09-16T14:55:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 >> v1:172.21.15.26:6790/0 conn(0x7f802409a550 legacy=0x7f80241c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 >> v1:172.21.15.189:6789/0 conn(0x7f802409d1b0 legacy=0x7f80241d1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:55:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 monclient: found mon.a 2024-09-16T14:55:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80241b9910 con 0x7f802409b040 2024-09-16T14:55:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:55:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:55:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 5 monclient: authenticate success, global_id 274116 2024-09-16T14:55:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 10 monclient: _renew_subs 2024-09-16T14:55:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 1 -- 172.21.15.26:0/390759586 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802409a240 con 0x7f802409b040 2024-09-16T14:55:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 10 monclient: _renew_subs 2024-09-16T14:55:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 1 -- 172.21.15.26:0/390759586 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80241a5020 con 0x7f802409b040 2024-09-16T14:55:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f800c003620 con 0x7f802409b040 2024-09-16T14:55:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f800c0039b0 con 0x7f802409b040 2024-09-16T14:55:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f80217fa640 4 set_mon_vals no callback set 2024-09-16T14:55:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient: dump: 2024-09-16T14:55:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.883 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f8029474640 1 -- 172.21.15.26:0/390759586 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8008006590 con 0x7f8008002830 2024-09-16T14:55:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f800c003b90 con 0x7f802409b040 2024-09-16T14:55:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f800c043d20 con 0x7f802409b040 2024-09-16T14:55:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943293863 0 0) 0x7f8008006590 con 0x7f8008002830 2024-09-16T14:55:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient(hunting): my global_id is 234518 2024-09-16T14:55:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8024071280 con 0x7f8008002830 2024-09-16T14:55:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2615101992 0 0) 0x7f8024071280 con 0x7f8008002830 2024-09-16T14:55:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8008006590 con 0x7f8008002830 2024-09-16T14:55:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f80237fe640 10 monclient: get_auth_request con 0x7f8004036390 auth_method 0 2024-09-16T14:55:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8014003d50 con 0x7f8008002830 2024-09-16T14:55:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.861+0000 7f801b7fe640 10 monclient: dump: 2024-09-16T14:55:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.885 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.865+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2175832863 0 0) 0x7f8008006590 con 0x7f8008002830 2024-09-16T14:55:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.865+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f800800a0e0 con 0x7f8008002830 2024-09-16T14:55:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.865+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8014005b50 con 0x7f8008002830 2024-09-16T14:55:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.865+0000 7f801b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.865+0000 7f801b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.865+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 >> v1:172.21.15.26:6790/0 conn(0x7f8008002830 legacy=0x7f8008004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.865+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 >> v1:172.21.15.26:6790/0 conn(0x7f8008002830 legacy=0x7f8008004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f8029474640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:55:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f8029474640 1 -- 172.21.15.26:0/390759586 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8008009d80 con 0x7f80080090a0 2024-09-16T14:55:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568497590 0 0) 0x7f8008009d80 con 0x7f80080090a0 2024-09-16T14:55:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 10 monclient(hunting): my global_id is 234521 2024-09-16T14:55:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8008006590 con 0x7f80080090a0 2024-09-16T14:55:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2100569447 0 0) 0x7f8008006590 con 0x7f80080090a0 2024-09-16T14:55:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8008009d80 con 0x7f80080090a0 2024-09-16T14:55:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8014004490 con 0x7f80080090a0 2024-09-16T14:55:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 10 monclient: dump: 2024-09-16T14:55:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:57.888 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:57.889 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 530802816 0 0) 0x7f8008009d80 con 0x7f80080090a0 2024-09-16T14:55:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f800800c290 con 0x7f80080090a0 2024-09-16T14:55:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 4069100095) 0x7f80140042b0 con 0x7f80080090a0 2024-09-16T14:55:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:55:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 >> v1:172.21.15.26:6790/0 conn(0x7f80080090a0 legacy=0x7f8008009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f801b7fe640 1 -- 172.21.15.26:0/390759586 >> v1:172.21.15.26:6790/0 conn(0x7f80080090a0 legacy=0x7f8008009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:57.890 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:55:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:55:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:55:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:55:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 126, 2024-09-16T14:55:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:55:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:55:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:55:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:55:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:55:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:55:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:55:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:57.892 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:55:57.892 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:55:57.892 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:57.892 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:57.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:57.892 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:57.892 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:57.892 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:57.892 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:57.892 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:57.892 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:57.892 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:57.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:57.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.893 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:55:57.893 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:55:57.893 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:55:57.893 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:55:57.893 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:55:57.893 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:55:57.893 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:57.893 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:57.893 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:55:57.893 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:55:57.893 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:55:57.893 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:55:57.893 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:55:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:55:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:55:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:55:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:55:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:55:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:55:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:55:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:55:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:55:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:55:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:55:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:55:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:55:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:55:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:55:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:55:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:55:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:55:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:55:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:55:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:55:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:55:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:55:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:55:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:55:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:55:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:55:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:55:57.902 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:55:57.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f8029474640 1 -- 172.21.15.26:0/390759586 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8004036390 legacy=0x7f8004038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f8029474640 10 monclient: shutdown 2024-09-16T14:55:57.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f8029474640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:57.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f8029474640 1 -- 172.21.15.26:0/390759586 >> v1:172.21.15.26:6789/0 conn(0x7f802409b040 legacy=0x7f80241b89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:57.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f80237fe640 1 -- 172.21.15.26:0/390759586 reap_dead start 2024-09-16T14:55:57.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f80237fe640 1 -- 172.21.15.26:0/390759586 reap_dead start 2024-09-16T14:55:57.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f8029474640 1 -- 172.21.15.26:0/390759586 shutdown_connections 2024-09-16T14:55:57.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f8029474640 1 -- 172.21.15.26:0/390759586 >> 172.21.15.26:0/390759586 conn(0x7f802409fb50 msgr2=0x7f80240a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:57.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.869+0000 7f8029474640 1 -- 172.21.15.26:0/390759586 shutdown_connections 2024-09-16T14:55:57.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:57.873+0000 7f8029474640 1 -- 172.21.15.26:0/390759586 wait complete. 2024-09-16T14:55:57.904 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:55:57.904 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:55:57.904 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:55:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.009+0000 7f9f6674d640 10 monclient: get_monmap_and_config 2024-09-16T14:55:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.009+0000 7f9f6674d640 10 monclient: build_initial_monmap 2024-09-16T14:55:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.009+0000 7f9f6674d640 10 monclient: monmap: 2024-09-16T14:55:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:58.016890+0000 2024-09-16T14:55:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:58.016890+0000 2024-09-16T14:55:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:58.016 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:58.016 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:58.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 1 Processor -- start 2024-09-16T14:55:58.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 1 -- start start 2024-09-16T14:55:58.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient: init 2024-09-16T14:55:58.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:58.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:55:58.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient(hunting): picked mon.noname-b con 0x7f9f6009a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:58.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient(hunting): picked mon.noname-a con 0x7f9f6009d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:58.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient(hunting): picked mon.noname-c con 0x7f9f6009b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:58.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6009dc60 con 0x7f9f6009d1b0 2024-09-16T14:55:58.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f60099ae0 con 0x7f9f6009b040 2024-09-16T14:55:58.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6009b8f0 con 0x7f9f6009a550 2024-09-16T14:55:58.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:58.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient(hunting): authenticate will time out at 1782.136828s 2024-09-16T14:55:58.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9f6009a550 0x7f9f6009a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35382/0 (socket says 172.21.15.26:35382) 2024-09-16T14:55:58.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5ffff640 1 -- 172.21.15.26:0/810771413 learned_addr learned my addr 172.21.15.26:0/810771413 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:58.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f57fff640 1 --1- 172.21.15.26:0/810771413 >> v1:172.21.15.26:6789/0 conn(0x7f9f6009d1b0 0x7f9f60136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60208/0 (socket says 172.21.15.26:60208) 2024-09-16T14:55:58.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 1 -- 172.21.15.26:0/810771413 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154470503 0 0) 0x7f9f6009dc60 con 0x7f9f6009d1b0 2024-09-16T14:55:58.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:58.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:58.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:58.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient(hunting): my global_id is 274119 2024-09-16T14:55:58.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 1 -- 172.21.15.26:0/810771413 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f4c003530 con 0x7f9f6009d1b0 2024-09-16T14:55:58.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 1 -- 172.21.15.26:0/810771413 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976864327 0 0) 0x7f9f6009b8f0 con 0x7f9f6009a550 2024-09-16T14:55:58.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:55:58.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:58.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:58.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient(hunting): my global_id is 234524 2024-09-16T14:55:58.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 1 -- 172.21.15.26:0/810771413 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f6009dc60 con 0x7f9f6009a550 2024-09-16T14:55:58.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 1 -- 172.21.15.26:0/810771413 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1976914189 0 0) 0x7f9f4c003530 con 0x7f9f6009d1b0 2024-09-16T14:55:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 1 -- 172.21.15.26:0/810771413 >> v1:172.21.15.26:6790/0 conn(0x7f9f6009a550 legacy=0x7f9f6009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 1 -- 172.21.15.26:0/810771413 >> v1:172.21.15.189:6789/0 conn(0x7f9f6009b040 legacy=0x7f9f6013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:55:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 1 monclient: found mon.noname-a 2024-09-16T14:55:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:55:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 1 -- 172.21.15.26:0/810771413 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6009bbd0 con 0x7f9f6009d1b0 2024-09-16T14:55:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:55:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:55:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 1 -- 172.21.15.26:0/810771413 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f48001620 con 0x7f9f6009d1b0 2024-09-16T14:55:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:55:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient: dump: 2024-09-16T14:55:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:58.022 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 5 monclient: authenticate success, global_id 274119 2024-09-16T14:55:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 1 -- 172.21.15.26:0/810771413 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f480018c0 con 0x7f9f6009d1b0 2024-09-16T14:55:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 1 -- 172.21.15.26:0/810771413 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f48001c50 con 0x7f9f6009d1b0 2024-09-16T14:55:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 4 set_mon_vals no callback set 2024-09-16T14:55:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5f7fe640 10 monclient: dump: 2024-09-16T14:55:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:58.024 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient: shutdown 2024-09-16T14:55:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 1 -- 172.21.15.26:0/810771413 >> v1:172.21.15.26:6789/0 conn(0x7f9f6009d1b0 legacy=0x7f9f60136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 1 -- 172.21.15.26:0/810771413 shutdown_connections 2024-09-16T14:55:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 1 -- 172.21.15.26:0/810771413 >> 172.21.15.26:0/810771413 conn(0x7f9f6009fb50 msgr2=0x7f9f6009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 1 -- 172.21.15.26:0/810771413 shutdown_connections 2024-09-16T14:55:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 1 -- 172.21.15.26:0/810771413 wait complete. 2024-09-16T14:55:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient: build_initial_monmap 2024-09-16T14:55:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient: monmap: 2024-09-16T14:55:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:58.025 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 1 Processor -- start 2024-09-16T14:55:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 1 -- start start 2024-09-16T14:55:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient: init 2024-09-16T14:55:58.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:58.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:55:58.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient(hunting): picked mon.noname-c con 0x7f9f6009a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient(hunting): picked mon.noname-a con 0x7f9f6009b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient(hunting): picked mon.noname-b con 0x7f9f6009d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6009a240 con 0x7f9f6009b040 2024-09-16T14:55:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f601b5340 con 0x7f9f6009d1b0 2024-09-16T14:55:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f601b5590 con 0x7f9f6009a550 2024-09-16T14:55:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f6674d640 10 monclient(hunting): authenticate will time out at 1782.139781s 2024-09-16T14:55:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9f6009a550 0x7f9f601b4c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35392/0 (socket says 172.21.15.26:35392) 2024-09-16T14:55:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.013+0000 7f9f5ffff640 1 -- 172.21.15.26:0/408739439 learned_addr learned my addr 172.21.15.26:0/408739439 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167412893 0 0) 0x7f9f601b5590 con 0x7f9f6009a550 2024-09-16T14:55:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:55:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:58.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): my global_id is 234527 2024-09-16T14:55:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f40003530 con 0x7f9f6009a550 2024-09-16T14:55:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519878628 0 0) 0x7f9f6009a240 con 0x7f9f6009b040 2024-09-16T14:55:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:55:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): my global_id is 274122 2024-09-16T14:55:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f601b5590 con 0x7f9f6009b040 2024-09-16T14:55:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3484563103 0 0) 0x7f9f40003530 con 0x7f9f6009a550 2024-09-16T14:55:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f6009a240 con 0x7f9f6009a550 2024-09-16T14:55:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f50001850 con 0x7f9f6009a550 2024-09-16T14:55:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:55:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): dump: 2024-09-16T14:55:58.032 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:58.032 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:58.032 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.032 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.032 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:58.032 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:58.032 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:58.032 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:58.032 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:58.032 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:58.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:58.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 140939524 0 0) 0x7f9f601b5590 con 0x7f9f6009b040 2024-09-16T14:55:58.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f40003530 con 0x7f9f6009b040 2024-09-16T14:55:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f48001bd0 con 0x7f9f6009b040 2024-09-16T14:55:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): dump: 2024-09-16T14:55:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:58.033 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:58.033 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331573556 0 0) 0x7f9f601b5340 con 0x7f9f6009d1b0 2024-09-16T14:55:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:55:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): my global_id is 264202 2024-09-16T14:55:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f601b5590 con 0x7f9f6009d1b0 2024-09-16T14:55:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 905513336 0 0) 0x7f9f6009a240 con 0x7f9f6009a550 2024-09-16T14:55:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 >> v1:172.21.15.189:6789/0 conn(0x7f9f6009d1b0 legacy=0x7f9f601cd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 >> v1:172.21.15.26:6789/0 conn(0x7f9f6009b040 legacy=0x7f9f601ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:55:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 monclient: found mon.c 2024-09-16T14:55:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f600a26e0 con 0x7f9f6009a550 2024-09-16T14:55:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:55:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:55:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f6674d640 5 monclient: authenticate success, global_id 234527 2024-09-16T14:55:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f6674d640 10 monclient: _renew_subs 2024-09-16T14:55:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f6674d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f6674d640 1 -- 172.21.15.26:0/408739439 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f601a0b20 con 0x7f9f6009a550 2024-09-16T14:55:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f6674d640 10 monclient: _renew_subs 2024-09-16T14:55:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f6674d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f6674d640 1 -- 172.21.15.26:0/408739439 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f601a3320 con 0x7f9f6009a550 2024-09-16T14:55:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f500047f0 con 0x7f9f6009a550 2024-09-16T14:55:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f50004b00 con 0x7f9f6009a550 2024-09-16T14:55:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5effd640 4 set_mon_vals no callback set 2024-09-16T14:55:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient: dump: 2024-09-16T14:55:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:58.037 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f6674d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f6674d640 1 -- 172.21.15.26:0/408739439 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9f44006590 con 0x7f9f44002830 2024-09-16T14:55:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9f5000f5f0 con 0x7f9f6009a550 2024-09-16T14:55:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f9f50043ec0 con 0x7f9f6009a550 2024-09-16T14:55:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64911566 0 0) 0x7f9f44006590 con 0x7f9f44002830 2024-09-16T14:55:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:55:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient(hunting): my global_id is 264205 2024-09-16T14:55:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f6009a240 con 0x7f9f44002830 2024-09-16T14:55:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f64cea640 10 monclient: get_auth_request con 0x7f9f400363f0 auth_method 0 2024-09-16T14:55:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4269677726 0 0) 0x7f9f6009a240 con 0x7f9f44002830 2024-09-16T14:55:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f44006590 con 0x7f9f44002830 2024-09-16T14:55:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f48001200 con 0x7f9f44002830 2024-09-16T14:55:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:55:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient: dump: 2024-09-16T14:55:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:58.039 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3172614007 0 0) 0x7f9f44006590 con 0x7f9f44002830 2024-09-16T14:55:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f4400a0e0 con 0x7f9f44002830 2024-09-16T14:55:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f9f48004e60 con 0x7f9f44002830 2024-09-16T14:55:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 >> v1:172.21.15.189:6789/0 conn(0x7f9f44002830 legacy=0x7f9f44004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.017+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 >> v1:172.21.15.189:6789/0 conn(0x7f9f44002830 legacy=0x7f9f44004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f6674d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:55:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f6674d640 1 -- 172.21.15.26:0/408739439 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9f44009d80 con 0x7f9f440090a0 2024-09-16T14:55:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990123104 0 0) 0x7f9f44009d80 con 0x7f9f440090a0 2024-09-16T14:55:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:55:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 10 monclient(hunting): my global_id is 264208 2024-09-16T14:55:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f44006590 con 0x7f9f440090a0 2024-09-16T14:55:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4011685310 0 0) 0x7f9f44006590 con 0x7f9f440090a0 2024-09-16T14:55:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f44009d80 con 0x7f9f440090a0 2024-09-16T14:55:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f48005170 con 0x7f9f440090a0 2024-09-16T14:55:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:55:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 10 monclient: dump: 2024-09-16T14:55:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:58.042 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3556398795 0 0) 0x7f9f44009d80 con 0x7f9f440090a0 2024-09-16T14:55:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9f4400c290 con 0x7f9f440090a0 2024-09-16T14:55:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3786824680) 0x7f9f480065b0 con 0x7f9f440090a0 2024-09-16T14:55:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:55:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 >> v1:172.21.15.189:6789/0 conn(0x7f9f440090a0 legacy=0x7f9f44009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.025+0000 7f9f5d7fa640 1 -- 172.21.15.26:0/408739439 >> v1:172.21.15.189:6789/0 conn(0x7f9f440090a0 legacy=0x7f9f44009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:58.043 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:55:58.043 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:55:58.043 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:55:58.043 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:55:58.043 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 126, 2024-09-16T14:55:58.043 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:55:58.043 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:55:58.043 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:55:58.043 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:55:58.043 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:58.044 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:55:58.044 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:58.044 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:55:58.044 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:55:58.044 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:58.044 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:58.044 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:58.044 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:58.044 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:58.044 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:58.044 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:58.044 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:58.044 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:58.045 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:58.045 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:58.045 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:55:58.045 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:55:58.045 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:58.045 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:58.045 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:58.045 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:58.045 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:58.045 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:58.045 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:58.045 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:58.045 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:58.045 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:58.046 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:58.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:58.046 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:55:58.046 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:55:58.046 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:55:58.046 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:55:58.046 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:55:58.046 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:55:58.046 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:58.046 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:58.046 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:55:58.046 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:55:58.046 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:55:58.046 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:55:58.047 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:55:58.047 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:55:58.047 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:55:58.047 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:58.047 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:55:58.047 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:58.047 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:58.047 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:58.047 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:58.047 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:58.047 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:58.047 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:58.047 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:58.047 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:58.048 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:58.048 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:58.048 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:55:58.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:58.048 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:55:58.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:58.048 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:55:58.048 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:55:58.048 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:58.048 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:58.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:58.048 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:58.048 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:55:58.049 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:58.049 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:58.049 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:58.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:58.049 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:55:58.049 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:55:58.049 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:58.049 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:58.049 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:58.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:58.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:58.049 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:55:58.049 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:55:58.049 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:58.050 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:58.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:58.050 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:58.050 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:55:58.050 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:58.050 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:58.050 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:58.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:58.050 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:55:58.050 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:55:58.050 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:58.050 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:58.050 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:58.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:58.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:58.051 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:55:58.051 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:55:58.051 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:58.051 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:58.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:58.051 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:58.051 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:55:58.051 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:58.051 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:58.051 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:58.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:58.051 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:55:58.052 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:55:58.052 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:58.052 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:58.052 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:58.052 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:58.052 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:58.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:58.052 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:55:58.052 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:55:58.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:58.052 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:58.052 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:58.052 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:55:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:55:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:55:58.053 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:55:58.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.029+0000 7f9f6674d640 1 -- 172.21.15.26:0/408739439 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9f400363f0 legacy=0x7f9f400388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:58.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.029+0000 7f9f6674d640 10 monclient: shutdown 2024-09-16T14:55:58.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.029+0000 7f9f6674d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:58.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.029+0000 7f9f6674d640 1 -- 172.21.15.26:0/408739439 >> v1:172.21.15.26:6790/0 conn(0x7f9f6009a550 legacy=0x7f9f601b4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:58.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.029+0000 7f9f64cea640 1 -- 172.21.15.26:0/408739439 reap_dead start 2024-09-16T14:55:58.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.029+0000 7f9f64cea640 1 -- 172.21.15.26:0/408739439 reap_dead start 2024-09-16T14:55:58.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.029+0000 7f9f6674d640 1 -- 172.21.15.26:0/408739439 shutdown_connections 2024-09-16T14:55:58.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.029+0000 7f9f6674d640 1 -- 172.21.15.26:0/408739439 >> 172.21.15.26:0/408739439 conn(0x7f9f6009fb50 msgr2=0x7f9f600a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:58.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.029+0000 7f9f6674d640 1 -- 172.21.15.26:0/408739439 shutdown_connections 2024-09-16T14:55:58.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.029+0000 7f9f6674d640 1 -- 172.21.15.26:0/408739439 wait complete. 2024-09-16T14:55:58.055 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:55:58.056 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:55:58.056 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:55:58.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.153+0000 7f6bb40a0640 10 monclient: get_monmap_and_config 2024-09-16T14:55:58.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.153+0000 7f6bb40a0640 10 monclient: build_initial_monmap 2024-09-16T14:55:58.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.153+0000 7f6bb40a0640 10 monclient: monmap: 2024-09-16T14:55:58.159 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:58.159 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:58.159 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:58.161143+0000 2024-09-16T14:55:58.159 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:58.161143+0000 2024-09-16T14:55:58.159 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:58.159 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:58.160 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:58.160 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:58.160 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:58.160 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:58.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 1 Processor -- start 2024-09-16T14:55:58.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 1 -- start start 2024-09-16T14:55:58.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 10 monclient: init 2024-09-16T14:55:58.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:58.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:55:58.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 10 monclient(hunting): picked mon.noname-c con 0x7f6bac0a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:58.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 10 monclient(hunting): picked mon.noname-b con 0x7f6bac0a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:58.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 10 monclient(hunting): picked mon.noname-a con 0x7f6bac0a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:58.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bac0a3250 con 0x7f6bac0a2800 2024-09-16T14:55:58.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bac09f600 con 0x7f6bac0a1d10 2024-09-16T14:55:58.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bac0a3830 con 0x7f6bac0a0000 2024-09-16T14:55:58.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:58.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 10 monclient(hunting): authenticate will time out at 1782.281203s 2024-09-16T14:55:58.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb263d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6bac0a2800 0x7f6bac13d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60228/0 (socket says 172.21.15.26:60228) 2024-09-16T14:55:58.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb263d640 1 -- 172.21.15.26:0/1106592340 learned_addr learned my addr 172.21.15.26:0/1106592340 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:58.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 1 -- 172.21.15.26:0/1106592340 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239838387 0 0) 0x7f6bac0a3250 con 0x7f6bac0a2800 2024-09-16T14:55:58.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient(hunting): _negotiate 2024-09-16T14:55:58.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:58.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:58.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient(hunting): my global_id is 274125 2024-09-16T14:55:58.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 1 -- 172.21.15.26:0/1106592340 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b94003530 con 0x7f6bac0a2800 2024-09-16T14:55:58.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 1 -- 172.21.15.26:0/1106592340 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395945566 0 0) 0x7f6bac0a3830 con 0x7f6bac0a0000 2024-09-16T14:55:58.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient(hunting): _negotiate 2024-09-16T14:55:58.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:58.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:58.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient(hunting): my global_id is 234530 2024-09-16T14:55:58.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 1 -- 172.21.15.26:0/1106592340 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bac0a3250 con 0x7f6bac0a0000 2024-09-16T14:55:58.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 1 -- 172.21.15.26:0/1106592340 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1874203362 0 0) 0x7f6b94003530 con 0x7f6bac0a2800 2024-09-16T14:55:58.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 1 -- 172.21.15.26:0/1106592340 >> v1:172.21.15.26:6790/0 conn(0x7f6bac0a0000 legacy=0x7f6bac139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:58.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 1 -- 172.21.15.26:0/1106592340 >> v1:172.21.15.189:6789/0 conn(0x7f6bac0a1d10 legacy=0x7f6bac0a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:58.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient: _finish_hunting 0 2024-09-16T14:55:58.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 1 monclient: found mon.noname-a 2024-09-16T14:55:58.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:58.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:55:58.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 1 -- 172.21.15.26:0/1106592340 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bac09ed60 con 0x7f6bac0a2800 2024-09-16T14:55:58.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient: _finish_auth 0 2024-09-16T14:55:58.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient: _check_auth_tickets 2024-09-16T14:55:58.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:58.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 1 -- 172.21.15.26:0/1106592340 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6ba4001620 con 0x7f6bac0a2800 2024-09-16T14:55:58.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 5 monclient: authenticate success, global_id 274125 2024-09-16T14:55:58.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:58.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:58.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:55:58.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient: dump: 2024-09-16T14:55:58.169 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:58.169 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:58.169 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.169 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.169 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:58.170 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:58.170 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:58.170 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:58.170 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:58.170 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:58.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:58.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 1 -- 172.21.15.26:0/1106592340 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ba4001900 con 0x7f6bac0a2800 2024-09-16T14:55:58.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:58.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 1 -- 172.21.15.26:0/1106592340 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6ba4001cd0 con 0x7f6bac0a2800 2024-09-16T14:55:58.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:58.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 4 set_mon_vals no callback set 2024-09-16T14:55:58.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:58.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:58.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb0e3a640 10 monclient: dump: 2024-09-16T14:55:58.171 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:58.171 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:58.171 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.171 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.171 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:58.171 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:58.171 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:58.171 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:58.171 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:58.171 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:58.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 10 monclient: shutdown 2024-09-16T14:55:58.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:58.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 1 -- 172.21.15.26:0/1106592340 >> v1:172.21.15.26:6789/0 conn(0x7f6bac0a2800 legacy=0x7f6bac13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:58.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 1 -- 172.21.15.26:0/1106592340 shutdown_connections 2024-09-16T14:55:58.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 1 -- 172.21.15.26:0/1106592340 >> 172.21.15.26:0/1106592340 conn(0x7f6bac0a7f30 msgr2=0x7f6bac0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:58.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 1 -- 172.21.15.26:0/1106592340 shutdown_connections 2024-09-16T14:55:58.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 1 -- 172.21.15.26:0/1106592340 wait complete. 2024-09-16T14:55:58.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 10 monclient: build_initial_monmap 2024-09-16T14:55:58.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 10 monclient: monmap: 2024-09-16T14:55:58.172 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:58.172 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:58.172 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:58.172 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:58.172 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:58.172 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:58.172 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:58.172 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:58.173 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:58.173 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:58.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 1 Processor -- start 2024-09-16T14:55:58.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 1 -- start start 2024-09-16T14:55:58.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.157+0000 7f6bb40a0640 10 monclient: init 2024-09-16T14:55:58.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:58.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:55:58.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 10 monclient(hunting): picked mon.noname-c con 0x7f6bac0a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:58.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 10 monclient(hunting): picked mon.noname-a con 0x7f6bac0a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:58.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 10 monclient(hunting): picked mon.noname-b con 0x7f6bac0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:58.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bac0a6230 con 0x7f6bac0a1d10 2024-09-16T14:55:58.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bac09e500 con 0x7f6bac0a2800 2024-09-16T14:55:58.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bac13dc00 con 0x7f6bac0a0000 2024-09-16T14:55:58.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:58.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb1e3c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6bac0a0000 0x7f6bac13d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35412/0 (socket says 172.21.15.26:35412) 2024-09-16T14:55:58.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb1e3c640 1 -- 172.21.15.26:0/2202848462 learned_addr learned my addr 172.21.15.26:0/2202848462 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:58.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 10 monclient(hunting): authenticate will time out at 1782.284216s 2024-09-16T14:55:58.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237871165 0 0) 0x7f6bac13dc00 con 0x7f6bac0a0000 2024-09-16T14:55:58.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:58.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:58.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:58.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): my global_id is 234533 2024-09-16T14:55:58.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b98003530 con 0x7f6bac0a0000 2024-09-16T14:55:58.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747178882 0 0) 0x7f6bac0a6230 con 0x7f6bac0a1d10 2024-09-16T14:55:58.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:58.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:58.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): my global_id is 274128 2024-09-16T14:55:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bac13dc00 con 0x7f6bac0a1d10 2024-09-16T14:55:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1742485095 0 0) 0x7f6b98003530 con 0x7f6bac0a0000 2024-09-16T14:55:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bac0a6230 con 0x7f6bac0a0000 2024-09-16T14:55:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6b9c001cf0 con 0x7f6bac0a0000 2024-09-16T14:55:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:55:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): dump: 2024-09-16T14:55:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:58.176 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2174337547 0 0) 0x7f6bac13dc00 con 0x7f6bac0a1d10 2024-09-16T14:55:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b98003530 con 0x7f6bac0a1d10 2024-09-16T14:55:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6ba00017e0 con 0x7f6bac0a1d10 2024-09-16T14:55:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): dump: 2024-09-16T14:55:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:58.177 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:58.177 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.177 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.177 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:58.177 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:58.177 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:58.177 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:58.177 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:58.177 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:58.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159673927 0 0) 0x7f6bac09e500 con 0x7f6bac0a2800 2024-09-16T14:55:58.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:58.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:58.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient(hunting): my global_id is 264214 2024-09-16T14:55:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bac13dc00 con 0x7f6bac0a2800 2024-09-16T14:55:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2530310537 0 0) 0x7f6bac0a6230 con 0x7f6bac0a0000 2024-09-16T14:55:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 >> v1:172.21.15.189:6789/0 conn(0x7f6bac0a2800 legacy=0x7f6bac1d1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 >> v1:172.21.15.26:6789/0 conn(0x7f6bac0a1d10 legacy=0x7f6bac1ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:55:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 monclient: found mon.c 2024-09-16T14:55:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bac1349b0 con 0x7f6bac0a0000 2024-09-16T14:55:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient: _finish_auth 0 2024-09-16T14:55:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:55:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 5 monclient: authenticate success, global_id 234533 2024-09-16T14:55:58.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 10 monclient: _renew_subs 2024-09-16T14:55:58.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:58.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 1 -- 172.21.15.26:0/2202848462 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bac1366f0 con 0x7f6bac0a0000 2024-09-16T14:55:58.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 10 monclient: _renew_subs 2024-09-16T14:55:58.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:58.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 1 -- 172.21.15.26:0/2202848462 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bac1357a0 con 0x7f6bac0a0000 2024-09-16T14:55:58.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b9c002060 con 0x7f6bac0a0000 2024-09-16T14:55:58.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:58.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6b9c003e70 con 0x7f6bac0a0000 2024-09-16T14:55:58.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:58.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:58.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6babfff640 4 set_mon_vals no callback set 2024-09-16T14:55:58.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 10 monclient: dump: 2024-09-16T14:55:58.179 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:58.180 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:58.180 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.180 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:58.180 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:58.180 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:58.180 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:58.180 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:58.180 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:58.180 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:58.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6b9c003fd0 con 0x7f6bac0a0000 2024-09-16T14:55:58.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f6b9c044140 con 0x7f6bac0a0000 2024-09-16T14:55:58.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:58.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb40a0640 1 -- 172.21.15.26:0/2202848462 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b8c005050 con 0x7f6bac0a0000 2024-09-16T14:55:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.161+0000 7f6bb163b640 10 monclient: get_auth_request con 0x7f6b98036440 auth_method 0 2024-09-16T14:55:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.165+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6b9c014aa0 con 0x7f6bac0a0000 2024-09-16T14:55:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.165+0000 7f6baa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:58.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.165+0000 7f6baa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:58.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.385+0000 7f6bb40a0640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:55:58.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.385+0000 7f6bb40a0640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:55:58.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.385+0000 7f6bb40a0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:55:58.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.385+0000 7f6bb40a0640 1 -- 172.21.15.26:0/2202848462 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f6b8c005b10 con 0x7f6bac0a0000 2024-09-16T14:55:58.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f6baa7fc640 1 -- 172.21.15.26:0/2202848462 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f6b9c012f10 con 0x7f6bac0a0000 2024-09-16T14:55:58.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f6baa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:55:58.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f6baa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:58.393 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.393 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=1,health=11,kv=11,logm=50} crc {auth=1149496446,config=1228700967,health=2665500085,kv=1867599010,logm=1857559683}) 2024-09-16T14:55:58.393 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=619173869,config=692619927,health=13881894,kv=1867599010,logm=847863169}) 2024-09-16T14:55:58.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=1,health=11,kv=11,logm=50} crc {auth=1149496446,config=1228700967,health=2665500085,kv=1867599010,logm=1857559683}) 2024-09-16T14:55:58.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=12,kv=10,logm=48} crc {auth=343532102,config=692619927,health=1949255588,kv=2418557040,logm=585452018}) 2024-09-16T14:55:58.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f6bb40a0640 1 -- 172.21.15.26:0/2202848462 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6b98036440 legacy=0x7f6b98038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:58.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f6bb40a0640 10 monclient: shutdown 2024-09-16T14:55:58.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f6bb40a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:58.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f6bb40a0640 1 -- 172.21.15.26:0/2202848462 >> v1:172.21.15.26:6790/0 conn(0x7f6bac0a0000 legacy=0x7f6bac13d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:58.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f6bb40a0640 1 -- 172.21.15.26:0/2202848462 shutdown_connections 2024-09-16T14:55:58.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f6bb40a0640 1 -- 172.21.15.26:0/2202848462 >> 172.21.15.26:0/2202848462 conn(0x7f6bac0a7f30 msgr2=0x7f6bac06b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:58.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f6bb40a0640 1 -- 172.21.15.26:0/2202848462 shutdown_connections 2024-09-16T14:55:58.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:58.389+0000 7f6bb40a0640 1 -- 172.21.15.26:0/2202848462 wait complete. 2024-09-16T14:55:58.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174461864}) 2024-09-16T14:55:58.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=17553206}) 2024-09-16T14:55:58.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174461864}) 2024-09-16T14:55:58.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3369877724}) 2024-09-16T14:55:58.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2543707600}) 2024-09-16T14:55:58.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340050834}) 2024-09-16T14:55:58.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2543707600}) 2024-09-16T14:55:58.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.393+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=883744660}) 2024-09-16T14:55:58.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3418741294}) 2024-09-16T14:55:58.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=392229425}) 2024-09-16T14:55:58.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3418741294}) 2024-09-16T14:55:58.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2454222419}) 2024-09-16T14:55:58.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1434875648}) 2024-09-16T14:55:58.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1617438108}) 2024-09-16T14:55:58.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1434875648}) 2024-09-16T14:55:58.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.397+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=970914927}) 2024-09-16T14:55:58.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3246462803}) 2024-09-16T14:55:58.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=177205563}) 2024-09-16T14:55:58.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3246462803}) 2024-09-16T14:55:58.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3740282774}) 2024-09-16T14:55:58.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3167695170}) 2024-09-16T14:55:58.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4005372670}) 2024-09-16T14:55:58.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3167695170}) 2024-09-16T14:55:58.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.401+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=646029822}) 2024-09-16T14:55:58.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=62808276}) 2024-09-16T14:55:58.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598452270}) 2024-09-16T14:55:58.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=62808276}) 2024-09-16T14:55:58.409 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1540602754}) 2024-09-16T14:55:58.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=622840497}) 2024-09-16T14:55:58.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2171840782}) 2024-09-16T14:55:58.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=622840497}) 2024-09-16T14:55:58.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.405+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2171339864}) 2024-09-16T14:55:58.411 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:55:58.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1602885749}) 2024-09-16T14:55:58.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=688182230}) 2024-09-16T14:55:58.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1602885749}) 2024-09-16T14:55:58.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3655300655}) 2024-09-16T14:55:58.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4180314866}) 2024-09-16T14:55:58.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398891901}) 2024-09-16T14:55:58.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4180314866}) 2024-09-16T14:55:58.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1434624512}) 2024-09-16T14:55:58.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4130978075}) 2024-09-16T14:55:58.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3618727739}) 2024-09-16T14:55:58.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4130978075}) 2024-09-16T14:55:58.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.409+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=363248268}) 2024-09-16T14:55:58.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.413+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.413+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3814101230}) 2024-09-16T14:55:58.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.413+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030871137}) 2024-09-16T14:55:58.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.413+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.413+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3814101230}) 2024-09-16T14:55:58.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.413+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294510588}) 2024-09-16T14:55:58.419 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.413+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.419 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.413+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2267906197}) 2024-09-16T14:55:58.419 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.413+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2349314331}) 2024-09-16T14:55:58.419 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.413+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.419 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.413+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2267906197}) 2024-09-16T14:55:58.419 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.413+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3843697670}) 2024-09-16T14:55:58.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.417+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.417+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1727250607}) 2024-09-16T14:55:58.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.417+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3074601103}) 2024-09-16T14:55:58.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.417+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.417+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1727250607}) 2024-09-16T14:55:58.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.417+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1646304476}) 2024-09-16T14:55:58.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.417+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.417+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=373110043}) 2024-09-16T14:55:58.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.417+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2053304972}) 2024-09-16T14:55:58.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.417+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.417+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=373110043}) 2024-09-16T14:55:58.423 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.417+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4122769652}) 2024-09-16T14:55:58.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.421+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.421+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1287508939}) 2024-09-16T14:55:58.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.421+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2472230130}) 2024-09-16T14:55:58.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.421+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.421+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1287508939}) 2024-09-16T14:55:58.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.421+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2983095821}) 2024-09-16T14:55:58.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.421+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.421+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=900405319}) 2024-09-16T14:55:58.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.421+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414562961}) 2024-09-16T14:55:58.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.421+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.421+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=900405319}) 2024-09-16T14:55:58.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.421+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3528730283}) 2024-09-16T14:55:58.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1104047084}) 2024-09-16T14:55:58.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3459328309}) 2024-09-16T14:55:58.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1104047084}) 2024-09-16T14:55:58.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3690025734}) 2024-09-16T14:55:58.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1942509569}) 2024-09-16T14:55:58.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=8982252}) 2024-09-16T14:55:58.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1942509569}) 2024-09-16T14:55:58.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=967389112}) 2024-09-16T14:55:58.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735533685}) 2024-09-16T14:55:58.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1639488322}) 2024-09-16T14:55:58.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735533685}) 2024-09-16T14:55:58.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.425+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=367739936}) 2024-09-16T14:55:58.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.429+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.429+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3830059270}) 2024-09-16T14:55:58.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.429+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2912586101}) 2024-09-16T14:55:58.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.429+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.429+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3830059270}) 2024-09-16T14:55:58.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.429+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=880554341}) 2024-09-16T14:55:58.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.429+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.429+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1745303560}) 2024-09-16T14:55:58.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.429+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3899424805}) 2024-09-16T14:55:58.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.429+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.429+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1745303560}) 2024-09-16T14:55:58.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.429+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3969205939}) 2024-09-16T14:55:58.436 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.433+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.433+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=898331102}) 2024-09-16T14:55:58.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.433+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469815282}) 2024-09-16T14:55:58.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.433+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.433+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=898331102}) 2024-09-16T14:55:58.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.433+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2819532632}) 2024-09-16T14:55:58.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.433+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.433+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2588844713}) 2024-09-16T14:55:58.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.433+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=452707535}) 2024-09-16T14:55:58.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.433+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.433+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2588844713}) 2024-09-16T14:55:58.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.433+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4057327091}) 2024-09-16T14:55:58.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.437+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.437+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=773940763}) 2024-09-16T14:55:58.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.437+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=315171626}) 2024-09-16T14:55:58.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.437+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.437+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=773940763}) 2024-09-16T14:55:58.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.437+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390636031}) 2024-09-16T14:55:58.442 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.437+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.442 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.437+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3392929115}) 2024-09-16T14:55:58.442 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.437+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2842775385}) 2024-09-16T14:55:58.442 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.437+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.442 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.437+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3392929115}) 2024-09-16T14:55:58.442 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.437+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2254163851}) 2024-09-16T14:55:58.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.441+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.441+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2130073605}) 2024-09-16T14:55:58.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.441+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579326712}) 2024-09-16T14:55:58.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.441+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.441+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2130073605}) 2024-09-16T14:55:58.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.441+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=881464631}) 2024-09-16T14:55:58.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.441+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.441+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=983051347}) 2024-09-16T14:55:58.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.441+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3492129202}) 2024-09-16T14:55:58.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.441+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.441+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=983051347}) 2024-09-16T14:55:58.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.441+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=520913441}) 2024-09-16T14:55:58.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.448 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4278183649}) 2024-09-16T14:55:58.448 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3607709010}) 2024-09-16T14:55:58.448 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.448 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4278183649}) 2024-09-16T14:55:58.448 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=64220691}) 2024-09-16T14:55:58.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2182611523}) 2024-09-16T14:55:58.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020537217}) 2024-09-16T14:55:58.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2182611523}) 2024-09-16T14:55:58.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1478573641}) 2024-09-16T14:55:58.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3266048613}) 2024-09-16T14:55:58.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3548882399}) 2024-09-16T14:55:58.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3266048613}) 2024-09-16T14:55:58.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.445+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3513902398}) 2024-09-16T14:55:58.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.449+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.449+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3832392056}) 2024-09-16T14:55:58.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.449+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255839742}) 2024-09-16T14:55:58.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.449+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.449+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3832392056}) 2024-09-16T14:55:58.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.449+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1077824173}) 2024-09-16T14:55:58.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.449+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.449+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2946314909}) 2024-09-16T14:55:58.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.449+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2501187373}) 2024-09-16T14:55:58.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.449+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.449+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2946314909}) 2024-09-16T14:55:58.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.449+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1673602539}) 2024-09-16T14:55:58.456 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.453+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.456 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.453+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394822829}) 2024-09-16T14:55:58.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.453+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239967754}) 2024-09-16T14:55:58.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.453+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.453+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394822829}) 2024-09-16T14:55:58.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.453+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=758049883}) 2024-09-16T14:55:58.458 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.453+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.458 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.453+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=816545508}) 2024-09-16T14:55:58.458 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.453+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=232419213}) 2024-09-16T14:55:58.458 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.453+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.458 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.453+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=816545508}) 2024-09-16T14:55:58.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.453+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1499517306}) 2024-09-16T14:55:58.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.457+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.457+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2413748912}) 2024-09-16T14:55:58.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.457+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2672955704}) 2024-09-16T14:55:58.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.457+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.457+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2413748912}) 2024-09-16T14:55:58.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.457+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1692442705}) 2024-09-16T14:55:58.462 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.457+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.462 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.457+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516255087}) 2024-09-16T14:55:58.462 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.457+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1699982182}) 2024-09-16T14:55:58.462 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.457+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.462 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.457+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516255087}) 2024-09-16T14:55:58.462 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.457+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1544013899}) 2024-09-16T14:55:58.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.461+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.461+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=529514903}) 2024-09-16T14:55:58.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.461+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=760859576}) 2024-09-16T14:55:58.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.461+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.461+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=529514903}) 2024-09-16T14:55:58.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.461+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2081721587}) 2024-09-16T14:55:58.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.461+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.461+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1164749386}) 2024-09-16T14:55:58.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.461+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939249776}) 2024-09-16T14:55:58.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.461+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.461+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1164749386}) 2024-09-16T14:55:58.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.461+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3809107178}) 2024-09-16T14:55:58.467 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.467 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1866023606}) 2024-09-16T14:55:58.467 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1459743584}) 2024-09-16T14:55:58.467 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1866023606}) 2024-09-16T14:55:58.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1017527777}) 2024-09-16T14:55:58.469 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.469 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3595854862}) 2024-09-16T14:55:58.469 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4148363719}) 2024-09-16T14:55:58.469 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.469 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3595854862}) 2024-09-16T14:55:58.469 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3310837419}) 2024-09-16T14:55:58.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3309330404}) 2024-09-16T14:55:58.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2851857689}) 2024-09-16T14:55:58.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3309330404}) 2024-09-16T14:55:58.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.465+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3052619897}) 2024-09-16T14:55:58.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.469+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.469+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=396321486}) 2024-09-16T14:55:58.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.469+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1531682989}) 2024-09-16T14:55:58.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.469+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.469+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=396321486}) 2024-09-16T14:55:58.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.469+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2044123457}) 2024-09-16T14:55:58.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.469+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.469+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2700126643}) 2024-09-16T14:55:58.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.469+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485001065}) 2024-09-16T14:55:58.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.469+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.469+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2700126643}) 2024-09-16T14:55:58.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.469+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3661438913}) 2024-09-16T14:55:58.475 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.473+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.473+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=393081977}) 2024-09-16T14:55:58.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.473+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1712468094}) 2024-09-16T14:55:58.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.473+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.473+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=393081977}) 2024-09-16T14:55:58.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.473+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4287793511}) 2024-09-16T14:55:58.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.473+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.473+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1249618671}) 2024-09-16T14:55:58.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.473+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=750765473}) 2024-09-16T14:55:58.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.473+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.473+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1249618671}) 2024-09-16T14:55:58.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.473+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1858680621}) 2024-09-16T14:55:58.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.473+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.473+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1606966653}) 2024-09-16T14:55:58.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.477+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=797257673}) 2024-09-16T14:55:58.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.477+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.480 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.477+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1606966653}) 2024-09-16T14:55:58.480 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.477+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2937329393}) 2024-09-16T14:55:58.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.477+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.477+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2737673854}) 2024-09-16T14:55:58.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.477+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748660120}) 2024-09-16T14:55:58.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.477+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.477+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2737673854}) 2024-09-16T14:55:58.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.477+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1050295116}) 2024-09-16T14:55:58.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.477+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.477+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2035876350}) 2024-09-16T14:55:58.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.477+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1220774475}) 2024-09-16T14:55:58.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.477+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.477+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2035876350}) 2024-09-16T14:55:58.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.477+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1727418115}) 2024-09-16T14:55:58.485 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.481+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.485 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.481+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3993390868}) 2024-09-16T14:55:58.485 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.481+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2601447136}) 2024-09-16T14:55:58.485 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.481+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.485 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.481+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3993390868}) 2024-09-16T14:55:58.485 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.481+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2674375501}) 2024-09-16T14:55:58.487 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.481+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.487 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.481+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2277043688}) 2024-09-16T14:55:58.487 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.481+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1730159704}) 2024-09-16T14:55:58.487 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.481+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.487 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.481+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2277043688}) 2024-09-16T14:55:58.487 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.481+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1185907353}) 2024-09-16T14:55:58.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.485+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.485+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1709443948}) 2024-09-16T14:55:58.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.485+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4106460221}) 2024-09-16T14:55:58.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.485+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.485+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1709443948}) 2024-09-16T14:55:58.489 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.485+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=678279219}) 2024-09-16T14:55:58.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.485+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.485+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1545121045}) 2024-09-16T14:55:58.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.485+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2599145739}) 2024-09-16T14:55:58.491 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.485+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.491 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.485+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1545121045}) 2024-09-16T14:55:58.491 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.485+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3872151733}) 2024-09-16T14:55:58.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.489+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.489+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=301155663}) 2024-09-16T14:55:58.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.489+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=785597734}) 2024-09-16T14:55:58.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.489+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.489+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=301155663}) 2024-09-16T14:55:58.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.489+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2779094581}) 2024-09-16T14:55:58.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.489+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.489+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1581670160}) 2024-09-16T14:55:58.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.489+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3885384705}) 2024-09-16T14:55:58.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.489+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.489+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1581670160}) 2024-09-16T14:55:58.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.489+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=998283323}) 2024-09-16T14:55:58.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.493+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.493+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2484507474}) 2024-09-16T14:55:58.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.493+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515475276}) 2024-09-16T14:55:58.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.493+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.493+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2484507474}) 2024-09-16T14:55:58.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.493+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=967972506}) 2024-09-16T14:55:58.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.493+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.493+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=643541654}) 2024-09-16T14:55:58.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.493+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593174715}) 2024-09-16T14:55:58.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.493+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.493+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=643541654}) 2024-09-16T14:55:58.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.493+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3464380136}) 2024-09-16T14:55:58.499 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.499 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=472873675}) 2024-09-16T14:55:58.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1503934943}) 2024-09-16T14:55:58.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=472873675}) 2024-09-16T14:55:58.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=695219385}) 2024-09-16T14:55:58.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3686015207}) 2024-09-16T14:55:58.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783931696}) 2024-09-16T14:55:58.502 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.502 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3686015207}) 2024-09-16T14:55:58.502 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1470665958}) 2024-09-16T14:55:58.503 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.503 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1029657549}) 2024-09-16T14:55:58.503 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3968849471}) 2024-09-16T14:55:58.503 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.503 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1029657549}) 2024-09-16T14:55:58.503 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.497+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598046797}) 2024-09-16T14:55:58.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.501+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.501+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=502015258}) 2024-09-16T14:55:58.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.501+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589379750}) 2024-09-16T14:55:58.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.501+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.501+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=502015258}) 2024-09-16T14:55:58.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.501+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3006995061}) 2024-09-16T14:55:58.506 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.501+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.506 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.501+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3221174671}) 2024-09-16T14:55:58.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.501+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=210295534}) 2024-09-16T14:55:58.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.501+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.501+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3221174671}) 2024-09-16T14:55:58.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.501+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2917806419}) 2024-09-16T14:55:58.508 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.505+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.508 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.505+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=913555676}) 2024-09-16T14:55:58.508 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.505+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217812230}) 2024-09-16T14:55:58.508 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.505+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.508 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.505+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=913555676}) 2024-09-16T14:55:58.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.505+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=836452747}) 2024-09-16T14:55:58.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.505+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.505+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1789492680}) 2024-09-16T14:55:58.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.505+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217211559}) 2024-09-16T14:55:58.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.505+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.505+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1789492680}) 2024-09-16T14:55:58.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.505+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1358851836}) 2024-09-16T14:55:58.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.509+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.509+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3494117454}) 2024-09-16T14:55:58.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.509+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2798874311}) 2024-09-16T14:55:58.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.509+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.509+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3494117454}) 2024-09-16T14:55:58.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.509+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3808785666}) 2024-09-16T14:55:58.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.509+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.509+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=7676417}) 2024-09-16T14:55:58.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.509+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3746272566}) 2024-09-16T14:55:58.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.509+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.509+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=7676417}) 2024-09-16T14:55:58.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.509+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2713747644}) 2024-09-16T14:55:58.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1413149168}) 2024-09-16T14:55:58.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=474164699}) 2024-09-16T14:55:58.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1413149168}) 2024-09-16T14:55:58.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2699042378}) 2024-09-16T14:55:58.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2257608747}) 2024-09-16T14:55:58.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2291166619}) 2024-09-16T14:55:58.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2257608747}) 2024-09-16T14:55:58.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2076457857}) 2024-09-16T14:55:58.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1196375194}) 2024-09-16T14:55:58.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178701661}) 2024-09-16T14:55:58.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1196375194}) 2024-09-16T14:55:58.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.513+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2701443264}) 2024-09-16T14:55:58.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.517+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.517+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1177291594}) 2024-09-16T14:55:58.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.517+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028185597}) 2024-09-16T14:55:58.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.517+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.517+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1177291594}) 2024-09-16T14:55:58.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.517+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2876469529}) 2024-09-16T14:55:58.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.517+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.517+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2821767855}) 2024-09-16T14:55:58.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.517+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3433052231}) 2024-09-16T14:55:58.522 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.517+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.523 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.517+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2821767855}) 2024-09-16T14:55:58.523 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.517+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1545479922}) 2024-09-16T14:55:58.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.521+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.521+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=439417870}) 2024-09-16T14:55:58.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.521+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=238730546}) 2024-09-16T14:55:58.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.521+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.521+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=439417870}) 2024-09-16T14:55:58.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.521+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4126113858}) 2024-09-16T14:55:58.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.521+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.521+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3867027012}) 2024-09-16T14:55:58.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.521+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2330218963}) 2024-09-16T14:55:58.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.521+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.521+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3867027012}) 2024-09-16T14:55:58.526 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.521+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1405560515}) 2024-09-16T14:55:58.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2240526491}) 2024-09-16T14:55:58.527 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2041657155}) 2024-09-16T14:55:58.528 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.528 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2240526491}) 2024-09-16T14:55:58.528 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3061237298}) 2024-09-16T14:55:58.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3163050310}) 2024-09-16T14:55:58.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=168143776}) 2024-09-16T14:55:58.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3163050310}) 2024-09-16T14:55:58.529 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3113657524}) 2024-09-16T14:55:58.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4100844293}) 2024-09-16T14:55:58.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1527321879}) 2024-09-16T14:55:58.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4100844293}) 2024-09-16T14:55:58.531 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.525+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1032817756}) 2024-09-16T14:55:58.532 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.529+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.529+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2574454371}) 2024-09-16T14:55:58.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.529+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3261044290}) 2024-09-16T14:55:58.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.529+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.529+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2574454371}) 2024-09-16T14:55:58.533 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.529+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1252974567}) 2024-09-16T14:55:58.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.529+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.529+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4038224662}) 2024-09-16T14:55:58.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.529+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3923505584}) 2024-09-16T14:55:58.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.529+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.529+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4038224662}) 2024-09-16T14:55:58.534 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.529+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=907955585}) 2024-09-16T14:55:58.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.533+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.533+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3622022013}) 2024-09-16T14:55:58.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.533+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546603999}) 2024-09-16T14:55:58.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.533+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.533+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3622022013}) 2024-09-16T14:55:58.536 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.533+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1262359964}) 2024-09-16T14:55:58.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.533+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.533+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=1785992391,mdsmap=1531151339,mgr=2006779800}) 2024-09-16T14:55:58.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.533+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=88,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1} crc {logm=4211204749,mdsmap=1531151339,mgr=620920336,mgr_command_descs=794974419,mgr_metadata=2109538692}) 2024-09-16T14:55:58.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.533+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.533+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=1785992391,mdsmap=1531151339,mgr=2006779800}) 2024-09-16T14:55:58.538 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.533+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=3318179715,mdsmap=2394902515,mgr=1247582348}) 2024-09-16T14:55:58.554 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.549+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.554 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.549+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=51} crc {mgr=2586702516,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2665606852,mon_config_key=437482411,monmap=2765269217,osd_metadata=539330051,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=2707385348}) 2024-09-16T14:55:58.554 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.549+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {mgr=1747058506,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2272142827,mon_config_key=3576433448,monmap=970648238,osd_metadata=3830151232,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=406211766}) 2024-09-16T14:55:58.554 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.549+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:55:58.554 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.549+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=51} crc {mgr=2586702516,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2665606852,mon_config_key=437482411,monmap=2765269217,osd_metadata=539330051,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=2707385348}) 2024-09-16T14:55:58.554 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:55:58.549+0000 7f8eaee69640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=51} crc {mgr=1228700967,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2945729653,mon_config_key=951922474,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3059172578}) 2024-09-16T14:55:59.412 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:55:59.412 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:55:59.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 10 monclient: get_monmap_and_config 2024-09-16T14:55:59.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 10 monclient: build_initial_monmap 2024-09-16T14:55:59.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 10 monclient: monmap: 2024-09-16T14:55:59.545 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:59.545 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:59.546 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:59.546986+0000 2024-09-16T14:55:59.546 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:59.546986+0000 2024-09-16T14:55:59.546 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:59.546 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:59.546 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:59.546 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:59.546 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:59.546 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:59.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 1 Processor -- start 2024-09-16T14:55:59.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 1 -- start start 2024-09-16T14:55:59.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 10 monclient: init 2024-09-16T14:55:59.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:59.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:55:59.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 10 monclient(hunting): picked mon.noname-c con 0x7f35bc09a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:59.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 10 monclient(hunting): picked mon.noname-b con 0x7f35bc09d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:59.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 10 monclient(hunting): picked mon.noname-a con 0x7f35bc09b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:59.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc09dc60 con 0x7f35bc09b040 2024-09-16T14:55:59.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc099ae0 con 0x7f35bc09a550 2024-09-16T14:55:59.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc09b8f0 con 0x7f35bc09d1b0 2024-09-16T14:55:59.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:59.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 10 monclient(hunting): authenticate will time out at 1783.667192s 2024-09-16T14:55:59.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c15d1640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f35bc09d1b0 0x7f35bc136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35422/0 (socket says 172.21.15.26:35422) 2024-09-16T14:55:59.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c15d1640 1 -- 172.21.15.26:0/4108726749 learned_addr learned my addr 172.21.15.26:0/4108726749 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:59.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 1 -- 172.21.15.26:0/4108726749 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939947215 0 0) 0x7f35bc09b8f0 con 0x7f35bc09d1b0 2024-09-16T14:55:59.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient(hunting): _negotiate 2024-09-16T14:55:59.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:59.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:59.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient(hunting): my global_id is 234536 2024-09-16T14:55:59.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 1 -- 172.21.15.26:0/4108726749 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a8005740 con 0x7f35bc09d1b0 2024-09-16T14:55:59.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 1 -- 172.21.15.26:0/4108726749 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145070121 0 0) 0x7f35bc09dc60 con 0x7f35bc09b040 2024-09-16T14:55:59.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient(hunting): _negotiate 2024-09-16T14:55:59.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:59.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:59.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient(hunting): my global_id is 274131 2024-09-16T14:55:59.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 1 -- 172.21.15.26:0/4108726749 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35bc09b8f0 con 0x7f35bc09b040 2024-09-16T14:55:59.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 1 -- 172.21.15.26:0/4108726749 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998714223 0 0) 0x7f35bc099ae0 con 0x7f35bc09a550 2024-09-16T14:55:59.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient(hunting): _negotiate 2024-09-16T14:55:59.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:59.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:59.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient(hunting): my global_id is 264217 2024-09-16T14:55:59.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 1 -- 172.21.15.26:0/4108726749 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35bc09dc60 con 0x7f35bc09a550 2024-09-16T14:55:59.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 1 -- 172.21.15.26:0/4108726749 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1036030281 0 0) 0x7f35a8005740 con 0x7f35bc09d1b0 2024-09-16T14:55:59.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 1 -- 172.21.15.26:0/4108726749 >> v1:172.21.15.189:6789/0 conn(0x7f35bc09a550 legacy=0x7f35bc09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:59.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 1 -- 172.21.15.26:0/4108726749 >> v1:172.21.15.26:6789/0 conn(0x7f35bc09b040 legacy=0x7f35bc13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:59.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient: _finish_hunting 0 2024-09-16T14:55:59.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 1 monclient: found mon.noname-b 2024-09-16T14:55:59.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:59.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:55:59.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 1 -- 172.21.15.26:0/4108726749 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc09bbd0 con 0x7f35bc09d1b0 2024-09-16T14:55:59.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient: _finish_auth 0 2024-09-16T14:55:59.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient: _check_auth_tickets 2024-09-16T14:55:59.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:59.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 1 -- 172.21.15.26:0/4108726749 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f35ac001620 con 0x7f35bc09d1b0 2024-09-16T14:55:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:55:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c0dd0640 10 monclient: dump: 2024-09-16T14:55:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:59.559 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:59.560 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 5 monclient: authenticate success, global_id 234536 2024-09-16T14:55:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.541+0000 7f35c4036640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c0dd0640 1 -- 172.21.15.26:0/4108726749 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35ac001840 con 0x7f35bc09d1b0 2024-09-16T14:55:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c0dd0640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 4 set_mon_vals no callback set 2024-09-16T14:55:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c0dd0640 1 -- 172.21.15.26:0/4108726749 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f35ac001bd0 con 0x7f35bc09d1b0 2024-09-16T14:55:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c0dd0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c0dd0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c0dd0640 10 monclient: dump: 2024-09-16T14:55:59.561 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:59.561 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:59.561 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.561 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.561 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:59.561 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:59.561 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:59.561 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:59.561 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:59.561 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:59.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 10 monclient: shutdown 2024-09-16T14:55:59.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:59.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 1 -- 172.21.15.26:0/4108726749 >> v1:172.21.15.26:6790/0 conn(0x7f35bc09d1b0 legacy=0x7f35bc136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:59.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 1 -- 172.21.15.26:0/4108726749 shutdown_connections 2024-09-16T14:55:59.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 1 -- 172.21.15.26:0/4108726749 >> 172.21.15.26:0/4108726749 conn(0x7f35bc09fb50 msgr2=0x7f35bc09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:59.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 1 -- 172.21.15.26:0/4108726749 shutdown_connections 2024-09-16T14:55:59.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 1 -- 172.21.15.26:0/4108726749 wait complete. 2024-09-16T14:55:59.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 10 monclient: build_initial_monmap 2024-09-16T14:55:59.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 10 monclient: monmap: 2024-09-16T14:55:59.562 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:59.562 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:59.562 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:59.562 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:59.562 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:59.562 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:59.563 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:59.563 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:59.563 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:59.563 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:59.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 1 Processor -- start 2024-09-16T14:55:59.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 1 -- start start 2024-09-16T14:55:59.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 10 monclient: init 2024-09-16T14:55:59.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:59.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:55:59.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 10 monclient(hunting): picked mon.noname-a con 0x7f35bc09a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:59.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 10 monclient(hunting): picked mon.noname-c con 0x7f35bc09b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:59.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 10 monclient(hunting): picked mon.noname-b con 0x7f35bc09d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:59.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc1a7710 con 0x7f35bc09a550 2024-09-16T14:55:59.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc1a4580 con 0x7f35bc09d1b0 2024-09-16T14:55:59.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc1a58c0 con 0x7f35bc09b040 2024-09-16T14:55:59.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:59.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 10 monclient(hunting): authenticate will time out at 1783.670247s 2024-09-16T14:55:59.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c1dd2640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f35bc09a550 0x7f35bc1a3760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60266/0 (socket says 172.21.15.26:60266) 2024-09-16T14:55:59.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c1dd2640 1 -- 172.21.15.26:0/1027110879 learned_addr learned my addr 172.21.15.26:0/1027110879 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:59.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028494912 0 0) 0x7f35bc1a7710 con 0x7f35bc09a550 2024-09-16T14:55:59.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:59.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:59.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:59.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): my global_id is 274134 2024-09-16T14:55:59.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a0003530 con 0x7f35bc09a550 2024-09-16T14:55:59.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466475070 0 0) 0x7f35bc1a58c0 con 0x7f35bc09b040 2024-09-16T14:55:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): my global_id is 234539 2024-09-16T14:55:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35bc1a7710 con 0x7f35bc09b040 2024-09-16T14:55:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238945181 0 0) 0x7f35bc1a4580 con 0x7f35bc09d1b0 2024-09-16T14:55:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): _negotiate 2024-09-16T14:55:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): my global_id is 264220 2024-09-16T14:55:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35bc1a58c0 con 0x7f35bc09d1b0 2024-09-16T14:55:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1156591218 0 0) 0x7f35a0003530 con 0x7f35bc09a550 2024-09-16T14:55:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35bc1a4580 con 0x7f35bc09a550 2024-09-16T14:55:59.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 616972421 0 0) 0x7f35bc1a7710 con 0x7f35bc09b040 2024-09-16T14:55:59.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35a0003530 con 0x7f35bc09b040 2024-09-16T14:55:59.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f35b8001920 con 0x7f35bc09a550 2024-09-16T14:55:59.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:59.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:55:59.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): dump: 2024-09-16T14:55:59.566 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:59.566 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:59.566 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.566 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.567 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:59.567 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:59.567 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:59.567 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:59.567 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:59.567 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:59.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:59.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f35ac001380 con 0x7f35bc09b040 2024-09-16T14:55:59.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:59.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:59.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient(hunting): dump: 2024-09-16T14:55:59.567 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:59.567 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:59.567 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.568 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.568 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:59.568 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:59.568 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:59.568 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:59.568 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:59.568 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:59.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 591724974 0 0) 0x7f35bc1a4580 con 0x7f35bc09a550 2024-09-16T14:55:59.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 >> v1:172.21.15.26:6790/0 conn(0x7f35bc09b040 legacy=0x7f35bc1a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:59.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 >> v1:172.21.15.189:6789/0 conn(0x7f35bc09d1b0 legacy=0x7f35bc1d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:59.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient: _finish_hunting 0 2024-09-16T14:55:59.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 monclient: found mon.a 2024-09-16T14:55:59.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:59.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:59.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc070960 con 0x7f35bc09a550 2024-09-16T14:55:59.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient: _finish_auth 0 2024-09-16T14:55:59.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient: _check_auth_tickets 2024-09-16T14:55:59.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:59.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 5 monclient: authenticate success, global_id 274134 2024-09-16T14:55:59.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 10 monclient: _renew_subs 2024-09-16T14:55:59.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:59.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 1 -- 172.21.15.26:0/1027110879 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35bc0a2780 con 0x7f35bc09a550 2024-09-16T14:55:59.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 10 monclient: _renew_subs 2024-09-16T14:55:59.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:59.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35c4036640 1 -- 172.21.15.26:0/1027110879 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35bc1a53b0 con 0x7f35bc09a550 2024-09-16T14:55:59.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35b8001c80 con 0x7f35bc09a550 2024-09-16T14:55:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f35b8003a70 con 0x7f35bc09a550 2024-09-16T14:55:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b3fff640 4 set_mon_vals no callback set 2024-09-16T14:55:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.545+0000 7f35b27fc640 10 monclient: dump: 2024-09-16T14:55:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:59.571 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.549+0000 7f35c4036640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.549+0000 7f35c4036640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.549+0000 7f35c4036640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.549+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f35b8003bd0 con 0x7f35bc09a550 2024-09-16T14:55:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.549+0000 7f35c4036640 1 -- 172.21.15.26:0/1027110879 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a4005050 con 0x7f35bc09a550 2024-09-16T14:55:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.549+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f35b8043dd0 con 0x7f35bc09a550 2024-09-16T14:55:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.549+0000 7f35c15d1640 10 monclient: get_auth_request con 0x7f35a0036400 auth_method 0 2024-09-16T14:55:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.553+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f35b80146c0 con 0x7f35bc09a550 2024-09-16T14:55:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.553+0000 7f35b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.553+0000 7f35b27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:59.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.773+0000 7f35c4036640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:55:59.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.773+0000 7f35c4036640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:55:59.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.773+0000 7f35c4036640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:59.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.773+0000 7f35c4036640 1 -- 172.21.15.26:0/1027110879 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f35a4005880 con 0x7f35bc09a550 2024-09-16T14:55:59.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.777+0000 7f35b27fc640 1 -- 172.21.15.26:0/1027110879 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2211254623) 0x7f35b8012b30 con 0x7f35bc09a550 2024-09-16T14:55:59.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.777+0000 7f35b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:55:59.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.777+0000 7f35b27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:59.780 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:55:59.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.777+0000 7f35c4036640 1 -- 172.21.15.26:0/1027110879 >> v1:172.21.15.189:6812/3196161582 conn(0x7f35a0036400 legacy=0x7f35a00388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:59.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.777+0000 7f35c4036640 10 monclient: shutdown 2024-09-16T14:55:59.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.777+0000 7f35c4036640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:59.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.777+0000 7f35c4036640 1 -- 172.21.15.26:0/1027110879 >> v1:172.21.15.26:6789/0 conn(0x7f35bc09a550 legacy=0x7f35bc1a3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:59.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.777+0000 7f35c4036640 1 -- 172.21.15.26:0/1027110879 shutdown_connections 2024-09-16T14:55:59.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.777+0000 7f35c4036640 1 -- 172.21.15.26:0/1027110879 >> 172.21.15.26:0/1027110879 conn(0x7f35bc09fb50 msgr2=0x7f35bc1b86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:59.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.777+0000 7f35c4036640 1 -- 172.21.15.26:0/1027110879 shutdown_connections 2024-09-16T14:55:59.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.777+0000 7f35c4036640 1 -- 172.21.15.26:0/1027110879 wait complete. 2024-09-16T14:55:59.799 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:55:59.799 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:55:59.800 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:55:59.800 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:55:59.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 10 monclient: get_monmap_and_config 2024-09-16T14:55:59.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 10 monclient: build_initial_monmap 2024-09-16T14:55:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 10 monclient: monmap: 2024-09-16T14:55:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:55:59.925589+0000 2024-09-16T14:55:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:55:59.925589+0000 2024-09-16T14:55:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:55:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:55:59.924 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 1 Processor -- start 2024-09-16T14:55:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 1 -- start start 2024-09-16T14:55:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 10 monclient: init 2024-09-16T14:55:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:55:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 10 monclient(hunting): picked mon.noname-c con 0x7fbf9409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 10 monclient(hunting): picked mon.noname-a con 0x7fbf94132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 10 monclient(hunting): picked mon.noname-b con 0x7fbf94138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9409d260 con 0x7fbf94132650 2024-09-16T14:55:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf94132a80 con 0x7fbf9409cb80 2024-09-16T14:55:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9409cf60 con 0x7fbf94138ff0 2024-09-16T14:55:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 10 monclient(hunting): authenticate will time out at 1784.045589s 2024-09-16T14:55:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf98ca6640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbf94138ff0 0x7fbf9413b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35448/0 (socket says 172.21.15.26:35448) 2024-09-16T14:55:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf937fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbf94132650 0x7fbf941378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60282/0 (socket says 172.21.15.26:60282) 2024-09-16T14:55:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf98ca6640 1 -- 172.21.15.26:0/3342695976 learned_addr learned my addr 172.21.15.26:0/3342695976 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 1 -- 172.21.15.26:0/3342695976 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246703307 0 0) 0x7fbf9409cf60 con 0x7fbf94138ff0 2024-09-16T14:55:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient(hunting): my global_id is 234542 2024-09-16T14:55:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 1 -- 172.21.15.26:0/3342695976 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf7c003530 con 0x7fbf94138ff0 2024-09-16T14:55:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 1 -- 172.21.15.26:0/3342695976 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 218068793 0 0) 0x7fbf9409d260 con 0x7fbf94132650 2024-09-16T14:55:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient(hunting): my global_id is 274137 2024-09-16T14:55:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 1 -- 172.21.15.26:0/3342695976 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf9409cf60 con 0x7fbf94132650 2024-09-16T14:55:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 1 -- 172.21.15.26:0/3342695976 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847365672 0 0) 0x7fbf94132a80 con 0x7fbf9409cb80 2024-09-16T14:55:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:55:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient(hunting): my global_id is 264223 2024-09-16T14:55:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 1 -- 172.21.15.26:0/3342695976 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf9409d260 con 0x7fbf9409cb80 2024-09-16T14:55:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 1 -- 172.21.15.26:0/3342695976 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1307737215 0 0) 0x7fbf7c003530 con 0x7fbf94138ff0 2024-09-16T14:55:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 1 -- 172.21.15.26:0/3342695976 >> v1:172.21.15.189:6789/0 conn(0x7fbf9409cb80 legacy=0x7fbf940717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 1 -- 172.21.15.26:0/3342695976 >> v1:172.21.15.26:6789/0 conn(0x7fbf94132650 legacy=0x7fbf941378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:55:59.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 1 monclient: found mon.noname-b 2024-09-16T14:55:59.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:59.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:55:59.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 1 -- 172.21.15.26:0/3342695976 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9409f2d0 con 0x7fbf94138ff0 2024-09-16T14:55:59.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient: _finish_auth 0 2024-09-16T14:55:59.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:55:59.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:59.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 1 -- 172.21.15.26:0/3342695976 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf8c0019e0 con 0x7fbf94138ff0 2024-09-16T14:55:59.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 5 monclient: authenticate success, global_id 234542 2024-09-16T14:55:59.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:59.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:59.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:55:59.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient: dump: 2024-09-16T14:55:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:59.937 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:55:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 1 -- 172.21.15.26:0/3342695976 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf8c001c00 con 0x7fbf94138ff0 2024-09-16T14:55:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 10 monclient: get_monmap_and_config success 2024-09-16T14:55:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 4 set_mon_vals no callback set 2024-09-16T14:55:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 1 -- 172.21.15.26:0/3342695976 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf8c001f10 con 0x7fbf94138ff0 2024-09-16T14:55:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf92ffd640 10 monclient: dump: 2024-09-16T14:55:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:59.939 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 10 monclient: shutdown 2024-09-16T14:55:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 1 -- 172.21.15.26:0/3342695976 >> v1:172.21.15.26:6790/0 conn(0x7fbf94138ff0 legacy=0x7fbf9413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 1 -- 172.21.15.26:0/3342695976 shutdown_connections 2024-09-16T14:55:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 1 -- 172.21.15.26:0/3342695976 >> 172.21.15.26:0/3342695976 conn(0x7fbf9409fb50 msgr2=0x7fbf9409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 1 -- 172.21.15.26:0/3342695976 shutdown_connections 2024-09-16T14:55:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.921+0000 7fbf9a709640 1 -- 172.21.15.26:0/3342695976 wait complete. 2024-09-16T14:55:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 10 monclient: build_initial_monmap 2024-09-16T14:55:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 10 monclient: monmap: 2024-09-16T14:55:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:55:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:55:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:55:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:55:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:55:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:55:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:55:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:55:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:55:59.941 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 1 Processor -- start 2024-09-16T14:55:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 1 -- start start 2024-09-16T14:55:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 10 monclient: init 2024-09-16T14:55:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 10 monclient: _reopen_session rank -1 2024-09-16T14:55:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:55:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 10 monclient(hunting): picked mon.noname-a con 0x7fbf9409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:55:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 10 monclient(hunting): picked mon.noname-b con 0x7fbf94132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:55:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 10 monclient(hunting): picked mon.noname-c con 0x7fbf94138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:55:59.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf94133940 con 0x7fbf9409cb80 2024-09-16T14:55:59.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf941c8930 con 0x7fbf94132650 2024-09-16T14:55:59.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf940a2250 con 0x7fbf94138ff0 2024-09-16T14:55:59.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 10 monclient(hunting): _renew_subs 2024-09-16T14:55:59.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 10 monclient(hunting): authenticate will time out at 1784.049168s 2024-09-16T14:55:59.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf93fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbf9409cb80 0x7fbf941c7b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60286/0 (socket says 172.21.15.26:60286) 2024-09-16T14:55:59.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf93fff640 1 -- 172.21.15.26:0/1886042958 learned_addr learned my addr 172.21.15.26:0/1886042958 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:55:59.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819691443 0 0) 0x7fbf94133940 con 0x7fbf9409cb80 2024-09-16T14:55:59.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:55:59.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:59.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:59.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): my global_id is 274140 2024-09-16T14:55:59.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf80003530 con 0x7fbf9409cb80 2024-09-16T14:55:59.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172933200 0 0) 0x7fbf940a2250 con 0x7fbf94138ff0 2024-09-16T14:55:59.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:55:59.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:59.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:59.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): my global_id is 234545 2024-09-16T14:55:59.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf94133940 con 0x7fbf94138ff0 2024-09-16T14:55:59.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737235649 0 0) 0x7fbf941c8930 con 0x7fbf94132650 2024-09-16T14:55:59.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:55:59.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:59.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:59.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): my global_id is 264226 2024-09-16T14:55:59.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf940a2250 con 0x7fbf94132650 2024-09-16T14:55:59.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3259894587 0 0) 0x7fbf80003530 con 0x7fbf9409cb80 2024-09-16T14:55:59.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf941c8930 con 0x7fbf9409cb80 2024-09-16T14:55:59.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf88001900 con 0x7fbf9409cb80 2024-09-16T14:55:59.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:59.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:55:59.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): dump: 2024-09-16T14:55:59.944 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:59.944 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:59.944 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.944 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.944 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:59.944 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:59.944 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:59.944 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:59.945 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:59.945 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:59.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:55:59.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 205843669 0 0) 0x7fbf94133940 con 0x7fbf94138ff0 2024-09-16T14:55:59.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf80003530 con 0x7fbf94138ff0 2024-09-16T14:55:59.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf8c001740 con 0x7fbf94138ff0 2024-09-16T14:55:59.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:59.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:55:59.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): dump: 2024-09-16T14:55:59.945 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:59.945 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:59.945 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.945 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.945 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:59.946 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:59.946 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:59.946 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:59.946 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:59.946 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:59.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3317447189 0 0) 0x7fbf941c8930 con 0x7fbf9409cb80 2024-09-16T14:55:59.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 >> v1:172.21.15.26:6790/0 conn(0x7fbf94138ff0 legacy=0x7fbf941cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:59.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 >> v1:172.21.15.189:6789/0 conn(0x7fbf94132650 legacy=0x7fbf941c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:59.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:55:59.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 monclient: found mon.a 2024-09-16T14:55:59.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:55:59.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:59.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9409a240 con 0x7fbf9409cb80 2024-09-16T14:55:59.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient: _finish_auth 0 2024-09-16T14:55:59.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:55:59.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:55:59.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 5 monclient: authenticate success, global_id 274140 2024-09-16T14:55:59.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 10 monclient: _renew_subs 2024-09-16T14:55:59.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:59.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 1 -- 172.21.15.26:0/1886042958 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf94131e80 con 0x7fbf9409cb80 2024-09-16T14:55:59.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 10 monclient: _renew_subs 2024-09-16T14:55:59.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:55:59.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 1 -- 172.21.15.26:0/1886042958 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9406ffd0 con 0x7fbf9409cb80 2024-09-16T14:55:59.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf88001cb0 con 0x7fbf9409cb80 2024-09-16T14:55:59.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:55:59.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf88003a70 con 0x7fbf9409cb80 2024-09-16T14:55:59.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:59.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:59.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient: dump: 2024-09-16T14:55:59.948 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:59.948 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:59.948 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.948 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.948 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:59.948 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:59.948 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:59.948 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:59.948 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:59.948 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:59.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf927fc640 4 set_mon_vals no callback set 2024-09-16T14:55:59.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:59.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf9a709640 1 -- 172.21.15.26:0/1886042958 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbf74006590 con 0x7fbf74002830 2024-09-16T14:55:59.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbf88003bd0 con 0x7fbf9409cb80 2024-09-16T14:55:59.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fbf88043e30 con 0x7fbf9409cb80 2024-09-16T14:55:59.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071138767 0 0) 0x7fbf74006590 con 0x7fbf74002830 2024-09-16T14:55:59.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:55:59.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:59.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:59.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient(hunting): my global_id is 274143 2024-09-16T14:55:59.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf941c8930 con 0x7fbf74002830 2024-09-16T14:55:59.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1881957588 0 0) 0x7fbf941c8930 con 0x7fbf74002830 2024-09-16T14:55:59.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf74006590 con 0x7fbf74002830 2024-09-16T14:55:59.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf98ca6640 10 monclient: get_auth_request con 0x7fbf80036350 auth_method 0 2024-09-16T14:55:59.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf840016d0 con 0x7fbf74002830 2024-09-16T14:55:59.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:59.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:59.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 10 monclient: dump: 2024-09-16T14:55:59.950 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:59.950 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:59.950 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.950 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.950 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:59.950 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:59.950 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:59.950 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:59.950 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:59.951 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:59.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.925+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4289687633 0 0) 0x7fbf74006590 con 0x7fbf74002830 2024-09-16T14:55:59.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.929+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf7400a0e0 con 0x7fbf74002830 2024-09-16T14:55:59.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.929+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fbf840019b0 con 0x7fbf74002830 2024-09-16T14:55:59.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.929+0000 7fbf90ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:55:59.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.929+0000 7fbf90ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:55:59.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.929+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 >> v1:172.21.15.26:6789/0 conn(0x7fbf74002830 legacy=0x7fbf74004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:59.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.929+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 >> v1:172.21.15.26:6789/0 conn(0x7fbf74002830 legacy=0x7fbf74004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:59.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf9a709640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:55:59.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf9a709640 1 -- 172.21.15.26:0/1886042958 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbf74009da0 con 0x7fbf740090c0 2024-09-16T14:55:59.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525139143 0 0) 0x7fbf74009da0 con 0x7fbf740090c0 2024-09-16T14:55:59.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:55:59.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:55:59.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:55:59.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 10 monclient(hunting): my global_id is 274146 2024-09-16T14:55:59.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf74006590 con 0x7fbf740090c0 2024-09-16T14:55:59.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 232815999 0 0) 0x7fbf74006590 con 0x7fbf740090c0 2024-09-16T14:55:59.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf74009da0 con 0x7fbf740090c0 2024-09-16T14:55:59.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf84004990 con 0x7fbf740090c0 2024-09-16T14:55:59.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:55:59.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:55:59.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 10 monclient: dump: 2024-09-16T14:55:59.954 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:55:59.954 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:55:59.954 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.954 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:55:59.954 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:55:59.954 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:55:59.954 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:55:59.954 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:55:59.954 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:55:59.954 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:55:59.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1249285937 0 0) 0x7fbf74009da0 con 0x7fbf740090c0 2024-09-16T14:55:59.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbf7400c290 con 0x7fbf740090c0 2024-09-16T14:55:59.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3963971000) 0x7fbf84004ba0 con 0x7fbf740090c0 2024-09-16T14:55:59.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:55:59.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:55:59.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 >> v1:172.21.15.26:6789/0 conn(0x7fbf740090c0 legacy=0x7fbf74009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:59.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.933+0000 7fbf90ff9640 1 -- 172.21.15.26:0/1886042958 >> v1:172.21.15.26:6789/0 conn(0x7fbf740090c0 legacy=0x7fbf74009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:55:59.955 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:55:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:55:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:55:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:55:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 126, 2024-09-16T14:55:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:55:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:55:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:55:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:55:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:55:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:55:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:55:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:55:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:55:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:55:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:55:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:55:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:55:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:55:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:55:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:55:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:55:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:55:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:55:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:55:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:55:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:55:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:55:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:55:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:55:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:55:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:55:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:55:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:55:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:55:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:55:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:55:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:55:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:55:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:55:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:55:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:55:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:55:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:55:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:55:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:55:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:55:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:55:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:55:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:55:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:55:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:55:59.963 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:59.963 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:59.963 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:59.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:59.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:59.963 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:55:59.963 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:55:59.963 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:55:59.963 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:55:59.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:59.963 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:55:59.963 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:55:59.963 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:55:59.963 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:59.964 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:59.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:59.964 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:55:59.964 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:55:59.964 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:55:59.964 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:55:59.964 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:55:59.964 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:59.964 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:59.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:59.964 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:55:59.964 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:55:59.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:59.965 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:59.965 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:59.965 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:55:59.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:59.965 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:55:59.965 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:55:59.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:55:59.965 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:55:59.965 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:55:59.965 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:55:59.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:55:59.965 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:55:59.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:55:59.966 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:55:59.966 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:55:59.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.937+0000 7fbf9a709640 1 -- 172.21.15.26:0/1886042958 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbf80036350 legacy=0x7fbf80038810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:59.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.937+0000 7fbf9a709640 10 monclient: shutdown 2024-09-16T14:55:59.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.937+0000 7fbf9a709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:55:59.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.937+0000 7fbf9a709640 1 -- 172.21.15.26:0/1886042958 >> v1:172.21.15.26:6789/0 conn(0x7fbf9409cb80 legacy=0x7fbf941c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:55:59.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.937+0000 7fbf98ca6640 1 -- 172.21.15.26:0/1886042958 reap_dead start 2024-09-16T14:55:59.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.937+0000 7fbf98ca6640 1 -- 172.21.15.26:0/1886042958 reap_dead start 2024-09-16T14:55:59.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.937+0000 7fbf9a709640 1 -- 172.21.15.26:0/1886042958 shutdown_connections 2024-09-16T14:55:59.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.937+0000 7fbf9a709640 1 -- 172.21.15.26:0/1886042958 >> 172.21.15.26:0/1886042958 conn(0x7fbf9409fb50 msgr2=0x7fbf9413b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:55:59.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.937+0000 7fbf9a709640 1 -- 172.21.15.26:0/1886042958 shutdown_connections 2024-09-16T14:55:59.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:55:59.937+0000 7fbf9a709640 1 -- 172.21.15.26:0/1886042958 wait complete. 2024-09-16T14:55:59.968 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:55:59.968 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:55:59.968 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:56:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 10 monclient: get_monmap_and_config 2024-09-16T14:56:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 10 monclient: build_initial_monmap 2024-09-16T14:56:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 10 monclient: monmap: 2024-09-16T14:56:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:00.085840+0000 2024-09-16T14:56:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:00.085840+0000 2024-09-16T14:56:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.085 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:00.085 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:00.085 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:00.085 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 1 Processor -- start 2024-09-16T14:56:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 1 -- start start 2024-09-16T14:56:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 10 monclient: init 2024-09-16T14:56:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:56:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 10 monclient(hunting): picked mon.noname-b con 0x7f6748134e40 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 10 monclient(hunting): picked mon.noname-c con 0x7f6748099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 10 monclient(hunting): picked mon.noname-a con 0x7f6748135220 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6748099e80 con 0x7f6748135220 2024-09-16T14:56:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67480719d0 con 0x7f6748099a70 2024-09-16T14:56:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674809e1e0 con 0x7f6748134e40 2024-09-16T14:56:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 10 monclient(hunting): authenticate will time out at 1784.205952s 2024-09-16T14:56:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674ebec640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6748135220 0x7f674813db30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60322/0 (socket says 172.21.15.26:60322) 2024-09-16T14:56:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674ebec640 1 -- 172.21.15.26:0/3067069472 learned_addr learned my addr 172.21.15.26:0/3067069472 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674e3eb640 1 --1- 172.21.15.26:0/3067069472 >> v1:172.21.15.26:6790/0 conn(0x7f6748134e40 0x7f6748070bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35472/0 (socket says 172.21.15.26:35472) 2024-09-16T14:56:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 1 -- 172.21.15.26:0/3067069472 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982439873 0 0) 0x7f67400014a0 con 0x7f6748135220 2024-09-16T14:56:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient(hunting): my global_id is 274149 2024-09-16T14:56:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 1 -- 172.21.15.26:0/3067069472 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6734003530 con 0x7f6748135220 2024-09-16T14:56:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 1 -- 172.21.15.26:0/3067069472 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500958552 0 0) 0x7f674809e1e0 con 0x7f6748134e40 2024-09-16T14:56:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient(hunting): my global_id is 234548 2024-09-16T14:56:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 1 -- 172.21.15.26:0/3067069472 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67400014a0 con 0x7f6748134e40 2024-09-16T14:56:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 1 -- 172.21.15.26:0/3067069472 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121936111 0 0) 0x7f67480719d0 con 0x7f6748099a70 2024-09-16T14:56:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient(hunting): my global_id is 264229 2024-09-16T14:56:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 1 -- 172.21.15.26:0/3067069472 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f674809e1e0 con 0x7f6748099a70 2024-09-16T14:56:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 1 -- 172.21.15.26:0/3067069472 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2229146278 0 0) 0x7f6734003530 con 0x7f6748135220 2024-09-16T14:56:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 1 -- 172.21.15.26:0/3067069472 >> v1:172.21.15.26:6790/0 conn(0x7f6748134e40 legacy=0x7f6748070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 1 -- 172.21.15.26:0/3067069472 >> v1:172.21.15.189:6789/0 conn(0x7f6748099a70 legacy=0x7f67480712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient: _finish_hunting 0 2024-09-16T14:56:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 1 monclient: found mon.noname-a 2024-09-16T14:56:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:56:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 1 -- 172.21.15.26:0/3067069472 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674809e7d0 con 0x7f6748135220 2024-09-16T14:56:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient: _finish_auth 0 2024-09-16T14:56:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient: _check_auth_tickets 2024-09-16T14:56:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 1 -- 172.21.15.26:0/3067069472 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6740001990 con 0x7f6748135220 2024-09-16T14:56:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient: dump: 2024-09-16T14:56:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.096 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 5 monclient: authenticate success, global_id 274149 2024-09-16T14:56:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 1 -- 172.21.15.26:0/3067069472 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6740001c00 con 0x7f6748135220 2024-09-16T14:56:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 1 -- 172.21.15.26:0/3067069472 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6740001f90 con 0x7f6748135220 2024-09-16T14:56:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 4 set_mon_vals no callback set 2024-09-16T14:56:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f674d3e9640 10 monclient: dump: 2024-09-16T14:56:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.098 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 10 monclient: shutdown 2024-09-16T14:56:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 1 -- 172.21.15.26:0/3067069472 >> v1:172.21.15.26:6789/0 conn(0x7f6748135220 legacy=0x7f674813db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 1 -- 172.21.15.26:0/3067069472 shutdown_connections 2024-09-16T14:56:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 1 -- 172.21.15.26:0/3067069472 >> 172.21.15.26:0/3067069472 conn(0x7f674809fb50 msgr2=0x7f674809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 1 -- 172.21.15.26:0/3067069472 shutdown_connections 2024-09-16T14:56:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.081+0000 7f675064f640 1 -- 172.21.15.26:0/3067069472 wait complete. 2024-09-16T14:56:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 10 monclient: build_initial_monmap 2024-09-16T14:56:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 10 monclient: monmap: 2024-09-16T14:56:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:00.099 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 1 Processor -- start 2024-09-16T14:56:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 1 -- start start 2024-09-16T14:56:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 10 monclient: init 2024-09-16T14:56:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:56:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 10 monclient(hunting): picked mon.noname-a con 0x7f6748099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 10 monclient(hunting): picked mon.noname-b con 0x7f6748134e40 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 10 monclient(hunting): picked mon.noname-c con 0x7f6748135220 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67481220c0 con 0x7f6748099a70 2024-09-16T14:56:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6748124420 con 0x7f6748134e40 2024-09-16T14:56:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6748122c90 con 0x7f6748135220 2024-09-16T14:56:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 10 monclient(hunting): authenticate will time out at 1784.209031s 2024-09-16T14:56:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f674e3eb640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6748099a70 0x7f67481c9930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60336/0 (socket says 172.21.15.26:60336) 2024-09-16T14:56:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f674e3eb640 1 -- 172.21.15.26:0/635382843 learned_addr learned my addr 172.21.15.26:0/635382843 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197974249 0 0) 0x7f6748122c90 con 0x7f6748135220 2024-09-16T14:56:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): my global_id is 234551 2024-09-16T14:56:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f672c003530 con 0x7f6748135220 2024-09-16T14:56:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769440018 0 0) 0x7f67481220c0 con 0x7f6748099a70 2024-09-16T14:56:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): my global_id is 274152 2024-09-16T14:56:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6748122c90 con 0x7f6748099a70 2024-09-16T14:56:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 206001940 0 0) 0x7f672c003530 con 0x7f6748135220 2024-09-16T14:56:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67481220c0 con 0x7f6748135220 2024-09-16T14:56:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6740004b70 con 0x7f6748135220 2024-09-16T14:56:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:56:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): dump: 2024-09-16T14:56:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.102 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1229773161 0 0) 0x7f6748122c90 con 0x7f6748099a70 2024-09-16T14:56:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f672c003530 con 0x7f6748099a70 2024-09-16T14:56:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197663482 0 0) 0x7f6748124420 con 0x7f6748134e40 2024-09-16T14:56:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): my global_id is 264232 2024-09-16T14:56:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6748122c90 con 0x7f6748134e40 2024-09-16T14:56:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6744001840 con 0x7f6748099a70 2024-09-16T14:56:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): dump: 2024-09-16T14:56:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.104 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 962427380 0 0) 0x7f67481220c0 con 0x7f6748135220 2024-09-16T14:56:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 >> v1:172.21.15.189:6789/0 conn(0x7f6748134e40 legacy=0x7f67481ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 >> v1:172.21.15.26:6789/0 conn(0x7f6748099a70 legacy=0x7f67481c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient: _finish_hunting 0 2024-09-16T14:56:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 monclient: found mon.c 2024-09-16T14:56:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674809c110 con 0x7f6748135220 2024-09-16T14:56:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient: _finish_auth 0 2024-09-16T14:56:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient: _check_auth_tickets 2024-09-16T14:56:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 5 monclient: authenticate success, global_id 234551 2024-09-16T14:56:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 10 monclient: _renew_subs 2024-09-16T14:56:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 1 -- 172.21.15.26:0/635382843 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67481340e0 con 0x7f6748135220 2024-09-16T14:56:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 10 monclient: _renew_subs 2024-09-16T14:56:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 1 -- 172.21.15.26:0/635382843 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6748123870 con 0x7f6748135220 2024-09-16T14:56:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6740004f80 con 0x7f6748135220 2024-09-16T14:56:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6740005290 con 0x7f6748135220 2024-09-16T14:56:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f674cbe8640 4 set_mon_vals no callback set 2024-09-16T14:56:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient: dump: 2024-09-16T14:56:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.107 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f675064f640 1 -- 172.21.15.26:0/635382843 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6730006590 con 0x7f6730002830 2024-09-16T14:56:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f674000fd00 con 0x7f6748135220 2024-09-16T14:56:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f6740044370 con 0x7f6748135220 2024-09-16T14:56:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878633018 0 0) 0x7f6730006590 con 0x7f6730002830 2024-09-16T14:56:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient(hunting): my global_id is 234554 2024-09-16T14:56:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67481220c0 con 0x7f6730002830 2024-09-16T14:56:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f674dbea640 10 monclient: get_auth_request con 0x7f672c0363f0 auth_method 0 2024-09-16T14:56:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 269750747 0 0) 0x7f67481220c0 con 0x7f6730002830 2024-09-16T14:56:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6730006590 con 0x7f6730002830 2024-09-16T14:56:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f67440047f0 con 0x7f6730002830 2024-09-16T14:56:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 10 monclient: dump: 2024-09-16T14:56:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.109 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 585099187 0 0) 0x7f6730006590 con 0x7f6730002830 2024-09-16T14:56:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.085+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f673000a0e0 con 0x7f6730002830 2024-09-16T14:56:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.089+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6744005820 con 0x7f6730002830 2024-09-16T14:56:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.089+0000 7f673effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.089+0000 7f673effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.089+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 >> v1:172.21.15.26:6790/0 conn(0x7f6730002830 legacy=0x7f6730004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.089+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 >> v1:172.21.15.26:6790/0 conn(0x7f6730002830 legacy=0x7f6730004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f675064f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:56:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f675064f640 1 -- 172.21.15.26:0/635382843 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6730009d80 con 0x7f67300090a0 2024-09-16T14:56:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888552284 0 0) 0x7f6730009d80 con 0x7f67300090a0 2024-09-16T14:56:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 10 monclient(hunting): my global_id is 234557 2024-09-16T14:56:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6730006590 con 0x7f67300090a0 2024-09-16T14:56:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2739011462 0 0) 0x7f6730006590 con 0x7f67300090a0 2024-09-16T14:56:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6730009d80 con 0x7f67300090a0 2024-09-16T14:56:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6744004ef0 con 0x7f67300090a0 2024-09-16T14:56:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 10 monclient: dump: 2024-09-16T14:56:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.113 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 658175103 0 0) 0x7f6730009d80 con 0x7f67300090a0 2024-09-16T14:56:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f673000c290 con 0x7f67300090a0 2024-09-16T14:56:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 1191608138) 0x7f67440051b0 con 0x7f67300090a0 2024-09-16T14:56:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:56:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 >> v1:172.21.15.26:6790/0 conn(0x7f67300090a0 legacy=0x7f6730009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f673effd640 1 -- 172.21.15.26:0/635382843 >> v1:172.21.15.26:6790/0 conn(0x7f67300090a0 legacy=0x7f6730009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:00.113 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:56:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:56:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:56:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:56:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 126, 2024-09-16T14:56:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:56:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:56:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:56:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:56:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:56:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:56:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:56:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:56:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:56:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:56:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:56:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:56:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:56:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:56:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:56:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:56:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:56:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:56:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:56:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:56:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:56:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:56:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:56:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:56:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:56:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:56:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:56:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:56:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:56:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:56:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:56:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:56:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:56:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:56:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:56:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:56:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:56:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:56:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:56:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:56:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:56:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:56:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:56:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:56:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:56:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:56:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:56:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:56:00.125 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:56:00.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f675064f640 1 -- 172.21.15.26:0/635382843 >> v1:172.21.15.189:6812/3196161582 conn(0x7f672c0363f0 legacy=0x7f672c0388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f675064f640 10 monclient: shutdown 2024-09-16T14:56:00.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f675064f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:00.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f675064f640 1 -- 172.21.15.26:0/635382843 >> v1:172.21.15.26:6790/0 conn(0x7f6748135220 legacy=0x7f67481d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f674ebec640 1 -- 172.21.15.26:0/635382843 reap_dead start 2024-09-16T14:56:00.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f674ebec640 1 -- 172.21.15.26:0/635382843 reap_dead start 2024-09-16T14:56:00.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f675064f640 1 -- 172.21.15.26:0/635382843 shutdown_connections 2024-09-16T14:56:00.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f675064f640 1 -- 172.21.15.26:0/635382843 >> 172.21.15.26:0/635382843 conn(0x7f674809fb50 msgr2=0x7f6748071400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:00.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f675064f640 1 -- 172.21.15.26:0/635382843 shutdown_connections 2024-09-16T14:56:00.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.093+0000 7f675064f640 1 -- 172.21.15.26:0/635382843 wait complete. 2024-09-16T14:56:00.127 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:56:00.127 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:56:00.127 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:56:00.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 10 monclient: get_monmap_and_config 2024-09-16T14:56:00.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 10 monclient: build_initial_monmap 2024-09-16T14:56:00.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 10 monclient: monmap: 2024-09-16T14:56:00.243 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:00.243 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.244 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:00.245356+0000 2024-09-16T14:56:00.244 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:00.245356+0000 2024-09-16T14:56:00.244 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:00.244 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.244 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:00.244 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:00.244 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:00.244 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 1 Processor -- start 2024-09-16T14:56:00.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 1 -- start start 2024-09-16T14:56:00.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 10 monclient: init 2024-09-16T14:56:00.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:00.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:56:00.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 10 monclient(hunting): picked mon.noname-c con 0x7f900c0a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:00.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 10 monclient(hunting): picked mon.noname-a con 0x7f900c0a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:00.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 10 monclient(hunting): picked mon.noname-b con 0x7f900c0a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:00.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c0a3250 con 0x7f900c0a0000 2024-09-16T14:56:00.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c09f600 con 0x7f900c0a1d10 2024-09-16T14:56:00.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c0a3830 con 0x7f900c0a2800 2024-09-16T14:56:00.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:00.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 10 monclient(hunting): authenticate will time out at 1784.365362s 2024-09-16T14:56:00.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9009d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f900c0a0000 0x7f900c139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60342/0 (socket says 172.21.15.26:60342) 2024-09-16T14:56:00.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9009d9b640 1 -- 172.21.15.26:0/107312816 learned_addr learned my addr 172.21.15.26:0/107312816 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:00.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900ad9d640 1 --1- 172.21.15.26:0/107312816 >> v1:172.21.15.26:6790/0 conn(0x7f900c0a2800 0x7f900c13d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35498/0 (socket says 172.21.15.26:35498) 2024-09-16T14:56:00.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 1 -- 172.21.15.26:0/107312816 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76720925 0 0) 0x7f900c0a3250 con 0x7f900c0a0000 2024-09-16T14:56:00.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient(hunting): my global_id is 274155 2024-09-16T14:56:00.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 1 -- 172.21.15.26:0/107312816 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff4003530 con 0x7f900c0a0000 2024-09-16T14:56:00.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 1 -- 172.21.15.26:0/107312816 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91298226 0 0) 0x7f900c0a3830 con 0x7f900c0a2800 2024-09-16T14:56:00.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient(hunting): my global_id is 234560 2024-09-16T14:56:00.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 1 -- 172.21.15.26:0/107312816 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f900c0a3250 con 0x7f900c0a2800 2024-09-16T14:56:00.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 1 -- 172.21.15.26:0/107312816 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671334806 0 0) 0x7f900c09f600 con 0x7f900c0a1d10 2024-09-16T14:56:00.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient(hunting): my global_id is 264235 2024-09-16T14:56:00.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 1 -- 172.21.15.26:0/107312816 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f900c0a3830 con 0x7f900c0a1d10 2024-09-16T14:56:00.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 1 -- 172.21.15.26:0/107312816 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 102406734 0 0) 0x7f8ff4003530 con 0x7f900c0a0000 2024-09-16T14:56:00.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 1 -- 172.21.15.26:0/107312816 >> v1:172.21.15.26:6790/0 conn(0x7f900c0a2800 legacy=0x7f900c13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 1 -- 172.21.15.26:0/107312816 >> v1:172.21.15.189:6789/0 conn(0x7f900c0a1d10 legacy=0x7f900c0a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient: _finish_hunting 0 2024-09-16T14:56:00.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 1 monclient: found mon.noname-a 2024-09-16T14:56:00.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:00.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:56:00.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 1 -- 172.21.15.26:0/107312816 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c09ed60 con 0x7f900c0a0000 2024-09-16T14:56:00.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient: _finish_auth 0 2024-09-16T14:56:00.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient: _check_auth_tickets 2024-09-16T14:56:00.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:00.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 1 -- 172.21.15.26:0/107312816 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9000001620 con 0x7f900c0a0000 2024-09-16T14:56:00.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:00.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient: dump: 2024-09-16T14:56:00.254 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.254 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.254 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.255 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.255 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.255 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.255 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.255 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.255 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.255 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 5 monclient: authenticate success, global_id 274155 2024-09-16T14:56:00.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:00.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 1 -- 172.21.15.26:0/107312816 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9000001880 con 0x7f900c0a0000 2024-09-16T14:56:00.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:00.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 1 -- 172.21.15.26:0/107312816 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9000001c50 con 0x7f900c0a0000 2024-09-16T14:56:00.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:00.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f900959a640 10 monclient: dump: 2024-09-16T14:56:00.256 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.256 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.256 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.256 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.256 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.256 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.256 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.256 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.256 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.256 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:00.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 4 set_mon_vals no callback set 2024-09-16T14:56:00.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 10 monclient: shutdown 2024-09-16T14:56:00.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:00.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 1 -- 172.21.15.26:0/107312816 >> v1:172.21.15.26:6789/0 conn(0x7f900c0a0000 legacy=0x7f900c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 1 -- 172.21.15.26:0/107312816 shutdown_connections 2024-09-16T14:56:00.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 1 -- 172.21.15.26:0/107312816 >> 172.21.15.26:0/107312816 conn(0x7f900c0a7f30 msgr2=0x7f900c0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:00.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 1 -- 172.21.15.26:0/107312816 shutdown_connections 2024-09-16T14:56:00.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 1 -- 172.21.15.26:0/107312816 wait complete. 2024-09-16T14:56:00.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 10 monclient: build_initial_monmap 2024-09-16T14:56:00.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 10 monclient: monmap: 2024-09-16T14:56:00.257 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:00.257 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:00.257 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:00.257 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:00.258 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:00.258 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:00.258 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:00.258 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:00.258 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:00.258 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.241+0000 7f9010e68640 1 Processor -- start 2024-09-16T14:56:00.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 1 -- start start 2024-09-16T14:56:00.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 10 monclient: init 2024-09-16T14:56:00.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:00.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:56:00.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 10 monclient(hunting): picked mon.noname-b con 0x7f900c0a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:00.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 10 monclient(hunting): picked mon.noname-a con 0x7f900c0a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:00.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 10 monclient(hunting): picked mon.noname-c con 0x7f900c0a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:00.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c132000 con 0x7f900c0a1d10 2024-09-16T14:56:00.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c133820 con 0x7f900c0a0000 2024-09-16T14:56:00.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c09e500 con 0x7f900c0a2800 2024-09-16T14:56:00.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:00.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 10 monclient(hunting): authenticate will time out at 1784.368541s 2024-09-16T14:56:00.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9009d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f900c0a1d10 0x7f900c1cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60350/0 (socket says 172.21.15.26:60350) 2024-09-16T14:56:00.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9009d9b640 1 -- 172.21.15.26:0/233836101 learned_addr learned my addr 172.21.15.26:0/233836101 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:00.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104991782 0 0) 0x7f900c132000 con 0x7f900c0a1d10 2024-09-16T14:56:00.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): my global_id is 274158 2024-09-16T14:56:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fec003530 con 0x7f900c0a1d10 2024-09-16T14:56:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074045321 0 0) 0x7f900c09e500 con 0x7f900c0a2800 2024-09-16T14:56:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): my global_id is 234563 2024-09-16T14:56:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f900c132000 con 0x7f900c0a2800 2024-09-16T14:56:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061601619 0 0) 0x7f900c133820 con 0x7f900c0a0000 2024-09-16T14:56:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): my global_id is 264238 2024-09-16T14:56:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f900c09e500 con 0x7f900c0a0000 2024-09-16T14:56:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2719622803 0 0) 0x7f8fec003530 con 0x7f900c0a1d10 2024-09-16T14:56:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f900c133820 con 0x7f900c0a1d10 2024-09-16T14:56:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9000001520 con 0x7f900c0a1d10 2024-09-16T14:56:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): dump: 2024-09-16T14:56:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.262 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2313071472 0 0) 0x7f900c132000 con 0x7f900c0a2800 2024-09-16T14:56:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fec003530 con 0x7f900c0a2800 2024-09-16T14:56:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9004001c90 con 0x7f900c0a2800 2024-09-16T14:56:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): dump: 2024-09-16T14:56:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.263 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 401943813 0 0) 0x7f900c133820 con 0x7f900c0a1d10 2024-09-16T14:56:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 >> v1:172.21.15.26:6790/0 conn(0x7f900c0a2800 legacy=0x7f900c1d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 >> v1:172.21.15.189:6789/0 conn(0x7f900c0a0000 legacy=0x7f900c134200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient: _finish_hunting 0 2024-09-16T14:56:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 monclient: found mon.a 2024-09-16T14:56:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c132980 con 0x7f900c0a1d10 2024-09-16T14:56:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient: _finish_auth 0 2024-09-16T14:56:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient: _check_auth_tickets 2024-09-16T14:56:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 5 monclient: authenticate success, global_id 274158 2024-09-16T14:56:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 10 monclient: _renew_subs 2024-09-16T14:56:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 1 -- 172.21.15.26:0/233836101 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900c131a80 con 0x7f900c0a1d10 2024-09-16T14:56:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 10 monclient: _renew_subs 2024-09-16T14:56:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 1 -- 172.21.15.26:0/233836101 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f900c0a64f0 con 0x7f900c0a1d10 2024-09-16T14:56:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90000011a0 con 0x7f900c0a1d10 2024-09-16T14:56:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9000004cc0 con 0x7f900c0a1d10 2024-09-16T14:56:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:00.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9008d99640 4 set_mon_vals no callback set 2024-09-16T14:56:00.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient: dump: 2024-09-16T14:56:00.265 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.265 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.265 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.265 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.265 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.265 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.265 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.265 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.265 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.265 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f9010e68640 1 -- 172.21.15.26:0/233836101 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8ff0006590 con 0x7f8ff0002830 2024-09-16T14:56:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9000004e20 con 0x7f900c0a1d10 2024-09-16T14:56:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f9000043d60 con 0x7f900c0a1d10 2024-09-16T14:56:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416709722 0 0) 0x7f8ff0006590 con 0x7f8ff0002830 2024-09-16T14:56:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient(hunting): my global_id is 264241 2024-09-16T14:56:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f900c133820 con 0x7f8ff0002830 2024-09-16T14:56:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f900ad9d640 10 monclient: get_auth_request con 0x7f8fec0360f0 auth_method 0 2024-09-16T14:56:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2777317161 0 0) 0x7f900c133820 con 0x7f8ff0002830 2024-09-16T14:56:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ff0006590 con 0x7f8ff0002830 2024-09-16T14:56:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ffc0037a0 con 0x7f8ff0002830 2024-09-16T14:56:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:56:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 10 monclient: dump: 2024-09-16T14:56:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.267 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1600181425 0 0) 0x7f8ff0006590 con 0x7f8ff0002830 2024-09-16T14:56:00.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.245+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8ff000a0e0 con 0x7f8ff0002830 2024-09-16T14:56:00.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.249+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8ffc003a00 con 0x7f8ff0002830 2024-09-16T14:56:00.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.249+0000 7f8ffaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:00.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.249+0000 7f8ffaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:00.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.249+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 >> v1:172.21.15.189:6789/0 conn(0x7f8ff0002830 legacy=0x7f8ff0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.249+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 >> v1:172.21.15.189:6789/0 conn(0x7f8ff0002830 legacy=0x7f8ff0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:00.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.253+0000 7f9010e68640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:56:00.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.253+0000 7f9010e68640 1 -- 172.21.15.26:0/233836101 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8ff0009da0 con 0x7f8ff00090c0 2024-09-16T14:56:00.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.253+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294968331 0 0) 0x7f8ff0009da0 con 0x7f8ff00090c0 2024-09-16T14:56:00.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.253+0000 7f8ffaffd640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.253+0000 7f8ffaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.253+0000 7f8ffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.253+0000 7f8ffaffd640 10 monclient(hunting): my global_id is 264244 2024-09-16T14:56:00.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.253+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff0006590 con 0x7f8ff00090c0 2024-09-16T14:56:00.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.253+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3689915542 0 0) 0x7f8ff0006590 con 0x7f8ff00090c0 2024-09-16T14:56:00.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.253+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ff0009da0 con 0x7f8ff00090c0 2024-09-16T14:56:00.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.253+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8ffc003fc0 con 0x7f8ff00090c0 2024-09-16T14:56:00.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.253+0000 7f8ffaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.253+0000 7f8ffaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:56:00.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.253+0000 7f8ffaffd640 10 monclient: dump: 2024-09-16T14:56:00.271 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.271 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.271 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.271 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.271 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.271 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.271 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.271 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.271 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.271 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.253+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 62891120 0 0) 0x7f8ff0009da0 con 0x7f8ff00090c0 2024-09-16T14:56:00.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.253+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8ff000c290 con 0x7f8ff00090c0 2024-09-16T14:56:00.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.257+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 2057474129) 0x7f8ffc004360 con 0x7f8ff00090c0 2024-09-16T14:56:00.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.257+0000 7f8ffaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:56:00.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.257+0000 7f8ffaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:00.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.257+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 >> v1:172.21.15.189:6789/0 conn(0x7f8ff00090c0 legacy=0x7f8ff0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.257+0000 7f8ffaffd640 1 -- 172.21.15.26:0/233836101 >> v1:172.21.15.189:6789/0 conn(0x7f8ff00090c0 legacy=0x7f8ff0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:00.272 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:56:00.272 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:56:00.272 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:56:00.273 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:56:00.273 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 126, 2024-09-16T14:56:00.273 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:56:00.273 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:56:00.273 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:56:00.273 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:56:00.273 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:00.273 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:56:00.273 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:00.273 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:56:00.273 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:56:00.273 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:00.273 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:00.274 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:00.274 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:00.274 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:00.274 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:00.274 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:00.274 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:00.274 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:00.274 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:00.274 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:00.274 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:56:00.274 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:56:00.274 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:00.274 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:00.275 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:00.275 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:00.275 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:00.275 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:00.275 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:00.275 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:00.275 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:00.275 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:00.275 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:00.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.275 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:56:00.275 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:56:00.275 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:56:00.275 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:56:00.275 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:56:00.276 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:56:00.276 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:00.276 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:00.276 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:56:00.276 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:56:00.276 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:56:00.276 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:56:00.276 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:56:00.276 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:56:00.276 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:56:00.276 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:00.276 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:56:00.277 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:00.277 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:00.277 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:00.277 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:00.277 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:00.277 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:00.277 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:00.277 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:00.277 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:00.277 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:00.277 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:00.277 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:56:00.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.278 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:56:00.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.278 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:56:00.278 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:56:00.278 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:00.278 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:00.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.278 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:00.278 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:56:00.278 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:00.278 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:00.278 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:00.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.279 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:56:00.279 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:56:00.279 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:00.279 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:00.279 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:00.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.279 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:56:00.279 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:56:00.279 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:00.279 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:00.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.279 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:00.279 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:56:00.280 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:00.280 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:00.280 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:00.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.280 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:56:00.280 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:56:00.280 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:00.280 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:00.280 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:00.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.280 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:56:00.280 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:56:00.280 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:00.281 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:00.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.281 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:00.281 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:56:00.281 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:00.281 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:00.281 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:00.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.281 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:56:00.281 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:56:00.281 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:00.281 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:00.281 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:00.281 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:00.282 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:00.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.282 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:56:00.282 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:56:00.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.282 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:00.282 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:00.282 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:00.282 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:00.283 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:00.283 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:56:00.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:00.283 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:00.283 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:00.283 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:56:00.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:00.283 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:00.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:00.283 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:56:00.283 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:56:00.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.257+0000 7f9010e68640 1 -- 172.21.15.26:0/233836101 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8fec0360f0 legacy=0x7f8fec0385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.257+0000 7f9010e68640 10 monclient: shutdown 2024-09-16T14:56:00.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.257+0000 7f9010e68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:00.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.257+0000 7f9010e68640 1 -- 172.21.15.26:0/233836101 >> v1:172.21.15.26:6789/0 conn(0x7f900c0a1d10 legacy=0x7f900c1cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.257+0000 7f900ad9d640 1 -- 172.21.15.26:0/233836101 reap_dead start 2024-09-16T14:56:00.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.257+0000 7f900ad9d640 1 -- 172.21.15.26:0/233836101 reap_dead start 2024-09-16T14:56:00.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.257+0000 7f9010e68640 1 -- 172.21.15.26:0/233836101 shutdown_connections 2024-09-16T14:56:00.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.257+0000 7f9010e68640 1 -- 172.21.15.26:0/233836101 >> 172.21.15.26:0/233836101 conn(0x7f900c0a7f30 msgr2=0x7f900c06b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:00.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.257+0000 7f9010e68640 1 -- 172.21.15.26:0/233836101 shutdown_connections 2024-09-16T14:56:00.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.257+0000 7f9010e68640 1 -- 172.21.15.26:0/233836101 wait complete. 2024-09-16T14:56:00.286 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:56:00.286 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T14:56:00.286 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:56:00.286 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T14:56:00.286 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T14:56:00.286 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T14:56:00.286 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:56:00.303 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T14:56:00.303 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:56:00.303 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:56:00.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: get_monmap_and_config 2024-09-16T14:56:00.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: build_initial_monmap 2024-09-16T14:56:00.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: monmap: 2024-09-16T14:56:00.461 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:00.461 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.461 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:00.462955+0000 2024-09-16T14:56:00.461 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:00.462955+0000 2024-09-16T14:56:00.461 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:00.462 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.462 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:00.462 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:00.462 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:00.462 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 Processor -- start 2024-09-16T14:56:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 -- start start 2024-09-16T14:56:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: init 2024-09-16T14:56:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:56:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient(hunting): picked mon.noname-a con 0x7f6bd009cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient(hunting): picked mon.noname-c con 0x7f6bd0132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient(hunting): picked mon.noname-b con 0x7f6bd0138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd009d260 con 0x7f6bd009cb80 2024-09-16T14:56:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd0132a80 con 0x7f6bd0132650 2024-09-16T14:56:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd009cf60 con 0x7f6bd0138ff0 2024-09-16T14:56:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bced9d640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f6bd0138ff0 legacy=0x7f6bd013b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient(hunting): authenticate will time out at 1784.583053s 2024-09-16T14:56:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bce59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6bd009cb80 0x7f6bd00717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60402/0 (socket says 172.21.15.26:60402) 2024-09-16T14:56:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bce59c640 1 -- 172.21.15.26:0/566596930 learned_addr learned my addr 172.21.15.26:0/566596930 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 1 -- 172.21.15.26:0/566596930 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205939508 0 0) 0x7f6bd009d260 con 0x7f6bd009cb80 2024-09-16T14:56:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 10 monclient(hunting): my global_id is 274161 2024-09-16T14:56:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 1 -- 172.21.15.26:0/566596930 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bbc003530 con 0x7f6bd009cb80 2024-09-16T14:56:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 1 -- 172.21.15.26:0/566596930 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1801166580 0 0) 0x7f6bbc003530 con 0x7f6bd009cb80 2024-09-16T14:56:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 1 -- 172.21.15.26:0/566596930 >> v1:172.21.15.26:6790/0 conn(0x7f6bd0138ff0 legacy=0x7f6bd013b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:56:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 1 -- 172.21.15.26:0/566596930 >> v1:172.21.15.189:6789/0 conn(0x7f6bd0132650 legacy=0x7f6bd01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 10 monclient: _finish_hunting 0 2024-09-16T14:56:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 1 monclient: found mon.noname-a 2024-09-16T14:56:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:56:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 1 -- 172.21.15.26:0/566596930 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd009f2d0 con 0x7f6bd009cb80 2024-09-16T14:56:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 10 monclient: _finish_auth 0 2024-09-16T14:56:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 10 monclient: _check_auth_tickets 2024-09-16T14:56:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 1 -- 172.21.15.26:0/566596930 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6bc0001620 con 0x7f6bd009cb80 2024-09-16T14:56:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 10 monclient: dump: 2024-09-16T14:56:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.472 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 5 monclient: authenticate success, global_id 274161 2024-09-16T14:56:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 1 -- 172.21.15.26:0/566596930 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6bc00018d0 con 0x7f6bd009cb80 2024-09-16T14:56:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 4 set_mon_vals no callback set 2024-09-16T14:56:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 1 -- 172.21.15.26:0/566596930 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6bc0001c60 con 0x7f6bd009cb80 2024-09-16T14:56:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcdd9b640 10 monclient: dump: 2024-09-16T14:56:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.474 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.474 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: shutdown 2024-09-16T14:56:00.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:00.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 -- 172.21.15.26:0/566596930 >> v1:172.21.15.26:6789/0 conn(0x7f6bd009cb80 legacy=0x7f6bd00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 -- 172.21.15.26:0/566596930 shutdown_connections 2024-09-16T14:56:00.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 -- 172.21.15.26:0/566596930 >> 172.21.15.26:0/566596930 conn(0x7f6bd009fb50 msgr2=0x7f6bd009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:00.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 -- 172.21.15.26:0/566596930 shutdown_connections 2024-09-16T14:56:00.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 -- 172.21.15.26:0/566596930 wait complete. 2024-09-16T14:56:00.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: build_initial_monmap 2024-09-16T14:56:00.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: monmap: 2024-09-16T14:56:00.474 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:00.474 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:00.475 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 Processor -- start 2024-09-16T14:56:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 -- start start 2024-09-16T14:56:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: init 2024-09-16T14:56:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:56:00.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient(hunting): picked mon.noname-c con 0x7f6bd009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient(hunting): picked mon.noname-a con 0x7f6bd0132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient(hunting): picked mon.noname-b con 0x7f6bd0138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd0133710 con 0x7f6bd0132650 2024-09-16T14:56:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd013c5e0 con 0x7f6bd0138ff0 2024-09-16T14:56:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bce59c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f6bd009cb80 legacy=0x7f6bd0139da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd009a240 con 0x7f6bd009cb80 2024-09-16T14:56:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient(hunting): authenticate will time out at 1784.586301s 2024-09-16T14:56:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc5d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6bd0132650 0x7f6bd013a4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60412/0 (socket says 172.21.15.26:60412) 2024-09-16T14:56:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc5d9b640 1 -- 172.21.15.26:0/3413530468 learned_addr learned my addr 172.21.15.26:0/3413530468 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:00.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568004248 0 0) 0x7f6bd0133710 con 0x7f6bd0132650 2024-09-16T14:56:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient(hunting): my global_id is 274164 2024-09-16T14:56:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bb0003530 con 0x7f6bd0132650 2024-09-16T14:56:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814091515 0 0) 0x7f6bd013c5e0 con 0x7f6bd0138ff0 2024-09-16T14:56:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:56:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient(hunting): my global_id is 264250 2024-09-16T14:56:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bd0133710 con 0x7f6bd0138ff0 2024-09-16T14:56:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2770556221 0 0) 0x7f6bb0003530 con 0x7f6bd0132650 2024-09-16T14:56:00.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bd013c5e0 con 0x7f6bd0132650 2024-09-16T14:56:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6bb8001900 con 0x7f6bd0132650 2024-09-16T14:56:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient(hunting): dump: 2024-09-16T14:56:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.479 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 650995194 0 0) 0x7f6bd013c5e0 con 0x7f6bd0132650 2024-09-16T14:56:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 >> v1:172.21.15.26:6790/0 conn(0x7f6bd009cb80 legacy=0x7f6bd0139da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:56:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 >> v1:172.21.15.189:6789/0 conn(0x7f6bd0138ff0 legacy=0x7f6bd013abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:56:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 1 monclient: found mon.a 2024-09-16T14:56:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd0134ea0 con 0x7f6bd0132650 2024-09-16T14:56:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient: _finish_auth 0 2024-09-16T14:56:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:56:00.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 5 monclient: authenticate success, global_id 274164 2024-09-16T14:56:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: _renew_subs 2024-09-16T14:56:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 -- 172.21.15.26:0/3413530468 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd0134600 con 0x7f6bd0132650 2024-09-16T14:56:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: _renew_subs 2024-09-16T14:56:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bd4c18640 1 -- 172.21.15.26:0/3413530468 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd013ba40 con 0x7f6bd0132650 2024-09-16T14:56:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6bb8003620 con 0x7f6bd0132650 2024-09-16T14:56:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6bb8003930 con 0x7f6bd0132650 2024-09-16T14:56:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bcd59a640 4 set_mon_vals no callback set 2024-09-16T14:56:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.461+0000 7f6bc77fe640 10 monclient: dump: 2024-09-16T14:56:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:00.481 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.465+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6bb8003b30 con 0x7f6bd0132650 2024-09-16T14:56:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.465+0000 7f6bd4c18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.465+0000 7f6bd4c18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.465+0000 7f6bd4c18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.465+0000 7f6bd4c18640 1 -- 172.21.15.26:0/3413530468 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb4005050 con 0x7f6bd0132650 2024-09-16T14:56:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.465+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f6bb8043d00 con 0x7f6bd0132650 2024-09-16T14:56:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.465+0000 7f6bce59c640 10 monclient: get_auth_request con 0x7f6bb0036160 auth_method 0 2024-09-16T14:56:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.469+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6bb8014760 con 0x7f6bd0132650 2024-09-16T14:56:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.469+0000 7f6bc77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:00.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.469+0000 7f6bc77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:00.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.685+0000 7f6bd4c18640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:56:00.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.685+0000 7f6bd4c18640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:56:00.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.685+0000 7f6bd4c18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:00.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.685+0000 7f6bd4c18640 1 -- 172.21.15.26:0/3413530468 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6bb4005880 con 0x7f6bd0132650 2024-09-16T14:56:00.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.685+0000 7f6bc77fe640 1 -- 172.21.15.26:0/3413530468 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 993807968) 0x7f6bb8012bd0 con 0x7f6bd0132650 2024-09-16T14:56:00.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.685+0000 7f6bc77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:56:00.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.685+0000 7f6bc77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:00.691 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:56:00.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.689+0000 7f6bd4c18640 1 -- 172.21.15.26:0/3413530468 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6bb0036160 legacy=0x7f6bb0038620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.689+0000 7f6bd4c18640 10 monclient: shutdown 2024-09-16T14:56:00.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.689+0000 7f6bd4c18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:00.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.689+0000 7f6bd4c18640 1 -- 172.21.15.26:0/3413530468 >> v1:172.21.15.26:6789/0 conn(0x7f6bd0132650 legacy=0x7f6bd013a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:00.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.689+0000 7f6bd4c18640 1 -- 172.21.15.26:0/3413530468 shutdown_connections 2024-09-16T14:56:00.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.689+0000 7f6bd4c18640 1 -- 172.21.15.26:0/3413530468 >> 172.21.15.26:0/3413530468 conn(0x7f6bd009fb50 msgr2=0x7f6bd01388c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:00.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.689+0000 7f6bd4c18640 1 -- 172.21.15.26:0/3413530468 shutdown_connections 2024-09-16T14:56:00.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:00.689+0000 7f6bd4c18640 1 -- 172.21.15.26:0/3413530468 wait complete. 2024-09-16T14:56:03.709 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:56:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 10 monclient: get_monmap_and_config 2024-09-16T14:56:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 10 monclient: build_initial_monmap 2024-09-16T14:56:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 10 monclient: monmap: 2024-09-16T14:56:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:03.842971+0000 2024-09-16T14:56:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:03.842971+0000 2024-09-16T14:56:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:03.842 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:03.842 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:03.842 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:03.842 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:03.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 1 Processor -- start 2024-09-16T14:56:03.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 1 -- start start 2024-09-16T14:56:03.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 10 monclient: init 2024-09-16T14:56:03.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:03.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:56:03.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 10 monclient(hunting): picked mon.noname-b con 0x7fe89409c480 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:03.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 10 monclient(hunting): picked mon.noname-c con 0x7fe89409bbe0 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:03.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 10 monclient(hunting): picked mon.noname-a con 0x7fe89409c860 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89409d260 con 0x7fe89409c860 2024-09-16T14:56:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89409e870 con 0x7fe89409bbe0 2024-09-16T14:56:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8940a0a00 con 0x7fe89409c480 2024-09-16T14:56:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 10 monclient(hunting): authenticate will time out at 1787.963028s 2024-09-16T14:56:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8937fe640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fe89409c480 legacy=0x7fe89409da50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe893fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe89409c860 0x7fe894142080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60416/0 (socket says 172.21.15.26:60416) 2024-09-16T14:56:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe893fff640 1 -- 172.21.15.26:0/2322805493 learned_addr learned my addr 172.21.15.26:0/2322805493 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 1 -- 172.21.15.26:0/2322805493 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830399940 0 0) 0x7fe89409d260 con 0x7fe89409c860 2024-09-16T14:56:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 10 monclient(hunting): my global_id is 274167 2024-09-16T14:56:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 1 -- 172.21.15.26:0/2322805493 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe87c003530 con 0x7fe89409c860 2024-09-16T14:56:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 1 -- 172.21.15.26:0/2322805493 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1060117807 0 0) 0x7fe87c003530 con 0x7fe89409c860 2024-09-16T14:56:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 1 -- 172.21.15.26:0/2322805493 >> v1:172.21.15.26:6790/0 conn(0x7fe89409c480 legacy=0x7fe89409da50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:56:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 1 -- 172.21.15.26:0/2322805493 >> v1:172.21.15.189:6789/0 conn(0x7fe89409bbe0 legacy=0x7fe89409e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 10 monclient: _finish_hunting 0 2024-09-16T14:56:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 1 monclient: found mon.noname-a 2024-09-16T14:56:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:56:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 1 -- 172.21.15.26:0/2322805493 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89409a760 con 0x7fe89409c860 2024-09-16T14:56:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 10 monclient: _finish_auth 0 2024-09-16T14:56:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 10 monclient: _check_auth_tickets 2024-09-16T14:56:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 1 -- 172.21.15.26:0/2322805493 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe88c001620 con 0x7fe89409c860 2024-09-16T14:56:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 10 monclient: dump: 2024-09-16T14:56:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:03.848 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:03.848 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 5 monclient: authenticate success, global_id 274167 2024-09-16T14:56:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 1 -- 172.21.15.26:0/2322805493 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe88c001940 con 0x7fe89409c860 2024-09-16T14:56:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 1 -- 172.21.15.26:0/2322805493 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe88c001cd0 con 0x7fe89409c860 2024-09-16T14:56:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe8927fc640 10 monclient: dump: 2024-09-16T14:56:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:03.849 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:03.850 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.841+0000 7fe899f51640 4 set_mon_vals no callback set 2024-09-16T14:56:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 10 monclient: shutdown 2024-09-16T14:56:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 1 -- 172.21.15.26:0/2322805493 >> v1:172.21.15.26:6789/0 conn(0x7fe89409c860 legacy=0x7fe894142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 1 -- 172.21.15.26:0/2322805493 shutdown_connections 2024-09-16T14:56:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 1 -- 172.21.15.26:0/2322805493 >> 172.21.15.26:0/2322805493 conn(0x7fe89409fb50 msgr2=0x7fe89409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 1 -- 172.21.15.26:0/2322805493 shutdown_connections 2024-09-16T14:56:03.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 1 -- 172.21.15.26:0/2322805493 wait complete. 2024-09-16T14:56:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 10 monclient: build_initial_monmap 2024-09-16T14:56:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 10 monclient: monmap: 2024-09-16T14:56:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:03.851 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:03.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 1 Processor -- start 2024-09-16T14:56:03.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 1 -- start start 2024-09-16T14:56:03.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 10 monclient: init 2024-09-16T14:56:03.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:03.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:56:03.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 10 monclient(hunting): picked mon.noname-a con 0x7fe89409bbe0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:03.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 10 monclient(hunting): picked mon.noname-c con 0x7fe89409c480 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:03.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 10 monclient(hunting): picked mon.noname-b con 0x7fe89409c860 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:03.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe894071ec0 con 0x7fe89409bbe0 2024-09-16T14:56:03.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8940a29b0 con 0x7fe89409c860 2024-09-16T14:56:03.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8941c7ea0 con 0x7fe89409c480 2024-09-16T14:56:03.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:03.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe892ffd640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7fe89409c480 legacy=0x7fe8941d2ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:03.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 10 monclient(hunting): authenticate will time out at 1787.966807s 2024-09-16T14:56:03.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe8937fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe89409bbe0 0x7fe8941c7790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60428/0 (socket says 172.21.15.26:60428) 2024-09-16T14:56:03.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe8937fe640 1 -- 172.21.15.26:0/3095800542 learned_addr learned my addr 172.21.15.26:0/3095800542 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:03.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 1 -- 172.21.15.26:0/3095800542 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794136358 0 0) 0x7fe894071ec0 con 0x7fe89409bbe0 2024-09-16T14:56:03.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:03.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:03.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:03.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 10 monclient(hunting): my global_id is 274170 2024-09-16T14:56:03.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 1 -- 172.21.15.26:0/3095800542 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe880003530 con 0x7fe89409bbe0 2024-09-16T14:56:03.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 1 -- 172.21.15.26:0/3095800542 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 588917465 0 0) 0x7fe880003530 con 0x7fe89409bbe0 2024-09-16T14:56:03.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 1 -- 172.21.15.26:0/3095800542 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe894071ec0 con 0x7fe89409bbe0 2024-09-16T14:56:03.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 1 -- 172.21.15.26:0/3095800542 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe884004160 con 0x7fe89409bbe0 2024-09-16T14:56:03.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:03.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:03.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 10 monclient(hunting): dump: 2024-09-16T14:56:03.856 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:03.856 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:03.856 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:03.856 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:03.856 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:03.857 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:03.857 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:03.857 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:03.857 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:03.857 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:03.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:03.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 1 -- 172.21.15.26:0/3095800542 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1164202352 0 0) 0x7fe894071ec0 con 0x7fe89409bbe0 2024-09-16T14:56:03.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 1 -- 172.21.15.26:0/3095800542 >> v1:172.21.15.26:6790/0 conn(0x7fe89409c480 legacy=0x7fe8941d2ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:56:03.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 1 -- 172.21.15.26:0/3095800542 >> v1:172.21.15.189:6789/0 conn(0x7fe89409c860 legacy=0x7fe8941d61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:03.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:56:03.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 1 monclient: found mon.a 2024-09-16T14:56:03.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:03.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:03.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 1 -- 172.21.15.26:0/3095800542 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89409a240 con 0x7fe89409bbe0 2024-09-16T14:56:03.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 10 monclient: _finish_auth 0 2024-09-16T14:56:03.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:56:03.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe87bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:03.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 5 monclient: authenticate success, global_id 274170 2024-09-16T14:56:03.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 10 monclient: _renew_subs 2024-09-16T14:56:03.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:03.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 1 -- 172.21.15.26:0/3095800542 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8941ac5b0 con 0x7fe89409bbe0 2024-09-16T14:56:03.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 10 monclient: _renew_subs 2024-09-16T14:56:03.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:03.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.845+0000 7fe899f51640 1 -- 172.21.15.26:0/3095800542 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8941acdb0 con 0x7fe89409bbe0 2024-09-16T14:56:03.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.849+0000 7fe87bfff640 1 -- 172.21.15.26:0/3095800542 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe884004410 con 0x7fe89409bbe0 2024-09-16T14:56:03.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.849+0000 7fe87bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:03.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.849+0000 7fe87bfff640 1 -- 172.21.15.26:0/3095800542 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe8840047a0 con 0x7fe89409bbe0 2024-09-16T14:56:03.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.849+0000 7fe87bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:03.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.849+0000 7fe87bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:03.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.849+0000 7fe891ffb640 4 set_mon_vals no callback set 2024-09-16T14:56:03.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.849+0000 7fe87bfff640 10 monclient: dump: 2024-09-16T14:56:03.859 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:03.859 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:03.859 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:03.859 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:03.859 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:03.859 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:03.859 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:03.859 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:03.860 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:03.860 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:03.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.849+0000 7fe899f51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:03.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.849+0000 7fe87bfff640 1 -- 172.21.15.26:0/3095800542 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe884004900 con 0x7fe89409bbe0 2024-09-16T14:56:03.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.849+0000 7fe899f51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:03.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.849+0000 7fe899f51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:03.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.849+0000 7fe899f51640 1 -- 172.21.15.26:0/3095800542 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe874005050 con 0x7fe89409bbe0 2024-09-16T14:56:03.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.849+0000 7fe87bfff640 1 -- 172.21.15.26:0/3095800542 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fe884043e10 con 0x7fe89409bbe0 2024-09-16T14:56:03.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.849+0000 7fe892ffd640 10 monclient: get_auth_request con 0x7fe880035fe0 auth_method 0 2024-09-16T14:56:03.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.853+0000 7fe87bfff640 1 -- 172.21.15.26:0/3095800542 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe884005100 con 0x7fe89409bbe0 2024-09-16T14:56:03.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.853+0000 7fe87bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:03.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:03.853+0000 7fe87bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:04.069+0000 7fe899f51640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:56:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:04.069+0000 7fe899f51640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:56:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:04.069+0000 7fe899f51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:04.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:04.069+0000 7fe899f51640 1 -- 172.21.15.26:0/3095800542 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe874005880 con 0x7fe89409bbe0 2024-09-16T14:56:04.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:04.073+0000 7fe87bfff640 1 -- 172.21.15.26:0/3095800542 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1808826433) 0x7fe884012b60 con 0x7fe89409bbe0 2024-09-16T14:56:04.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:04.073+0000 7fe87bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:56:04.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:04.073+0000 7fe87bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:04.072 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:56:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:04.073+0000 7fe899f51640 1 -- 172.21.15.26:0/3095800542 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe880035fe0 legacy=0x7fe8800384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:04.073+0000 7fe899f51640 10 monclient: shutdown 2024-09-16T14:56:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:04.073+0000 7fe899f51640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:04.073+0000 7fe899f51640 1 -- 172.21.15.26:0/3095800542 >> v1:172.21.15.26:6789/0 conn(0x7fe89409bbe0 legacy=0x7fe8941c7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:04.073+0000 7fe899f51640 1 -- 172.21.15.26:0/3095800542 shutdown_connections 2024-09-16T14:56:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:04.073+0000 7fe899f51640 1 -- 172.21.15.26:0/3095800542 >> 172.21.15.26:0/3095800542 conn(0x7fe89409fb50 msgr2=0x7fe89409f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:04.073+0000 7fe899f51640 1 -- 172.21.15.26:0/3095800542 shutdown_connections 2024-09-16T14:56:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:04.073+0000 7fe899f51640 1 -- 172.21.15.26:0/3095800542 wait complete. 2024-09-16T14:56:07.091 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:56:07.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a93e98640 10 monclient: get_monmap_and_config 2024-09-16T14:56:07.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a93e98640 10 monclient: build_initial_monmap 2024-09-16T14:56:07.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a93e98640 10 monclient: monmap: 2024-09-16T14:56:07.225 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:07.225 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:07.225 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:07.226987+0000 2024-09-16T14:56:07.225 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:07.226987+0000 2024-09-16T14:56:07.226 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:07.226 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:07.226 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:07.226 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:07.226 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:07.226 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:07.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a93e98640 1 Processor -- start 2024-09-16T14:56:07.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a93e98640 1 -- start start 2024-09-16T14:56:07.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a93e98640 10 monclient: init 2024-09-16T14:56:07.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a93e98640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:07.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a93e98640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:56:07.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a93e98640 10 monclient(hunting): picked mon.noname-c con 0x7f4a8c134d00 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:07.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a93e98640 10 monclient(hunting): picked mon.noname-a con 0x7f4a8c09a1c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:07.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a93e98640 10 monclient(hunting): picked mon.noname-b con 0x7f4a8c1357f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:07.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a93e98640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8c06ca10 con 0x7f4a8c09a1c0 2024-09-16T14:56:07.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a93e98640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8c0998e0 con 0x7f4a8c134d00 2024-09-16T14:56:07.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a93e98640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8c099040 con 0x7f4a8c1357f0 2024-09-16T14:56:07.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a93e98640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:07.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a93e98640 10 monclient(hunting): authenticate will time out at 1791.347093s 2024-09-16T14:56:07.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a92435640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a8c13d960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:07.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a91433640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4a8c09a1c0 0x7f4a8c13a230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33948/0 (socket says 172.21.15.26:33948) 2024-09-16T14:56:07.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.225+0000 7f4a91433640 1 -- 172.21.15.26:0/1963293435 learned_addr learned my addr 172.21.15.26:0/1963293435 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:07.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.429+0000 7f4a92435640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a8c13d960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:07.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:07.833+0000 7f4a92435640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a8c13d960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:08.637+0000 7f4a92435640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a8c13d960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:10.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.225+0000 7f4a7f7fe640 10 monclient(hunting): tick 2024-09-16T14:56:10.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.225+0000 7f4a7f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:56:10.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.225+0000 7f4a7f7fe640 1 monclient(hunting): continuing hunt 2024-09-16T14:56:10.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.225+0000 7f4a7f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:56:10.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.225+0000 7f4a7f7fe640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a8c13d960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:56:10.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.225+0000 7f4a7f7fe640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.189:6789/0 conn(0x7f4a8c134d00 legacy=0x7f4a8c1350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:10.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.225+0000 7f4a7f7fe640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.26:6789/0 conn(0x7f4a8c09a1c0 legacy=0x7f4a8c13a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:10.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.225+0000 7f4a7f7fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:56:10.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.225+0000 7f4a7f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f4a700036d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:10.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.225+0000 7f4a7f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f4a8c1357f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:10.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.225+0000 7f4a7f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f4a8c09a1c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:10.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.225+0000 7f4a7f7fe640 1 -- 172.21.15.26:0/1963293435 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8c099320 con 0x7f4a8c09a1c0 2024-09-16T14:56:10.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.225+0000 7f4a7f7fe640 1 -- 172.21.15.26:0/1963293435 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8c099040 con 0x7f4a700036d0 2024-09-16T14:56:10.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.225+0000 7f4a7f7fe640 1 -- 172.21.15.26:0/1963293435 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a700103b0 con 0x7f4a8c1357f0 2024-09-16T14:56:10.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.225+0000 7f4a7f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:10.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.225+0000 7f4a91433640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a700093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:10.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.429+0000 7f4a91433640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a700093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:10.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:10.833+0000 7f4a91433640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a700093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:11.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:11.637+0000 7f4a91433640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a700093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:13.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.225+0000 7f4a7f7fe640 10 monclient(hunting): tick 2024-09-16T14:56:13.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.225+0000 7f4a7f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:56:13.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.225+0000 7f4a7f7fe640 1 monclient(hunting): continuing hunt 2024-09-16T14:56:13.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.225+0000 7f4a7f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:56:13.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.225+0000 7f4a7f7fe640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a700093f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:56:13.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.225+0000 7f4a7f7fe640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.189:6789/0 conn(0x7f4a700036d0 legacy=0x7f4a70005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:13.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.225+0000 7f4a7f7fe640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.26:6789/0 conn(0x7f4a8c09a1c0 legacy=0x7f4a7000cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:13.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.225+0000 7f4a7f7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:56:13.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.225+0000 7f4a7f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f4a700105c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:13.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.225+0000 7f4a7f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f4a70012880 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:13.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.225+0000 7f4a7f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f4a8c1357f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:13.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.225+0000 7f4a7f7fe640 1 -- 172.21.15.26:0/1963293435 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a700115c0 con 0x7f4a700105c0 2024-09-16T14:56:13.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.225+0000 7f4a7f7fe640 1 -- 172.21.15.26:0/1963293435 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a700103b0 con 0x7f4a70012880 2024-09-16T14:56:13.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.225+0000 7f4a7f7fe640 1 -- 172.21.15.26:0/1963293435 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7000bb80 con 0x7f4a8c1357f0 2024-09-16T14:56:13.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.225+0000 7f4a7f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:13.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.229+0000 7f4a92435640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a70005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:13.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.433+0000 7f4a92435640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a70005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:13.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:13.837+0000 7f4a92435640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a70005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:14.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 1 -- 172.21.15.26:0/1963293435 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306664757 0 0) 0x7f4a700115c0 con 0x7f4a700105c0 2024-09-16T14:56:14.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 10 monclient(hunting): _negotiate 2024-09-16T14:56:14.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:14.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:14.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 10 monclient(hunting): my global_id is 274173 2024-09-16T14:56:14.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 1 -- 172.21.15.26:0/1963293435 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a78003530 con 0x7f4a700105c0 2024-09-16T14:56:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 1 -- 172.21.15.26:0/1963293435 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2848763153 0 0) 0x7f4a78003530 con 0x7f4a700105c0 2024-09-16T14:56:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a70005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:56:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.189:6789/0 conn(0x7f4a70012880 legacy=0x7f4a70014c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 10 monclient: _finish_hunting 0 2024-09-16T14:56:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 1 monclient: found mon.noname-a 2024-09-16T14:56:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:56:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 1 -- 172.21.15.26:0/1963293435 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7000cd20 con 0x7f4a700105c0 2024-09-16T14:56:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 10 monclient: _finish_auth 0 2024-09-16T14:56:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 10 monclient: _check_auth_tickets 2024-09-16T14:56:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 1 -- 172.21.15.26:0/1963293435 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4a800018a0 con 0x7f4a700105c0 2024-09-16T14:56:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 10 monclient: dump: 2024-09-16T14:56:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.413 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a93e98640 5 monclient: authenticate success, global_id 274173 2024-09-16T14:56:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a93e98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 1 -- 172.21.15.26:0/1963293435 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a80001ac0 con 0x7f4a700105c0 2024-09-16T14:56:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a93e98640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a93e98640 4 set_mon_vals no callback set 2024-09-16T14:56:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 1 -- 172.21.15.26:0/1963293435 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4a80001ed0 con 0x7f4a700105c0 2024-09-16T14:56:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.409+0000 7f4a90c32640 10 monclient: dump: 2024-09-16T14:56:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.415 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.415 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.413+0000 7f4a93e98640 10 monclient: shutdown 2024-09-16T14:56:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.413+0000 7f4a93e98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.413+0000 7f4a93e98640 1 -- 172.21.15.26:0/1963293435 >> v1:172.21.15.26:6789/0 conn(0x7f4a700105c0 legacy=0x7f4a700109e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.413+0000 7f4a93e98640 1 -- 172.21.15.26:0/1963293435 shutdown_connections 2024-09-16T14:56:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.413+0000 7f4a93e98640 1 -- 172.21.15.26:0/1963293435 >> 172.21.15.26:0/1963293435 conn(0x7f4a8c0a0da0 msgr2=0x7f4a8c0a11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.413+0000 7f4a93e98640 1 -- 172.21.15.26:0/1963293435 shutdown_connections 2024-09-16T14:56:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.413+0000 7f4a93e98640 1 -- 172.21.15.26:0/1963293435 wait complete. 2024-09-16T14:56:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.413+0000 7f4a93e98640 10 monclient: build_initial_monmap 2024-09-16T14:56:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.413+0000 7f4a93e98640 10 monclient: monmap: 2024-09-16T14:56:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:14.418 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.413+0000 7f4a93e98640 1 Processor -- start 2024-09-16T14:56:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.413+0000 7f4a93e98640 1 -- start start 2024-09-16T14:56:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.413+0000 7f4a93e98640 10 monclient: init 2024-09-16T14:56:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a93e98640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a93e98640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:56:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a93e98640 10 monclient(hunting): picked mon.noname-a con 0x7f4a700105c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a93e98640 10 monclient(hunting): picked mon.noname-b con 0x7f4a70012880 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a93e98640 10 monclient(hunting): picked mon.noname-c con 0x7f4a8c1357f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a93e98640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8c0a3070 con 0x7f4a700105c0 2024-09-16T14:56:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a93e98640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8c123c50 con 0x7f4a70012880 2024-09-16T14:56:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a93e98640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8c121e00 con 0x7f4a8c1357f0 2024-09-16T14:56:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a93e98640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a92435640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a8c1d1750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a93e98640 10 monclient(hunting): authenticate will time out at 1798.536912s 2024-09-16T14:56:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a91c34640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4a700105c0 0x7f4a8c1c93f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33980/0 (socket says 172.21.15.26:33980) 2024-09-16T14:56:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a91c34640 1 -- 172.21.15.26:0/440040834 learned_addr learned my addr 172.21.15.26:0/440040834 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79049786 0 0) 0x7f4a8c123c50 con 0x7f4a70012880 2024-09-16T14:56:14.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:14.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:14.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:14.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a7e7fc640 10 monclient(hunting): my global_id is 264259 2024-09-16T14:56:14.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a700114f0 con 0x7f4a70012880 2024-09-16T14:56:14.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1010740562 0 0) 0x7f4a700114f0 con 0x7f4a70012880 2024-09-16T14:56:14.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a8c123c50 con 0x7f4a70012880 2024-09-16T14:56:14.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4a880017f0 con 0x7f4a70012880 2024-09-16T14:56:14.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a7e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a7e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:56:14.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a7e7fc640 10 monclient(hunting): dump: 2024-09-16T14:56:14.420 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.421 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.421 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.421 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.421 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.421 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.421 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.421 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.421 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.421 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.417+0000 7f4a7e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:14.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.421+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615627569 0 0) 0x7f4a8c0a3070 con 0x7f4a700105c0 2024-09-16T14:56:14.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.421+0000 7f4a7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:14.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.421+0000 7f4a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:14.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.421+0000 7f4a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:14.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.421+0000 7f4a7e7fc640 10 monclient(hunting): my global_id is 274176 2024-09-16T14:56:14.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.421+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a700114f0 con 0x7f4a700105c0 2024-09-16T14:56:14.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 463867485 0 0) 0x7f4a700114f0 con 0x7f4a700105c0 2024-09-16T14:56:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a8c0a3070 con 0x7f4a700105c0 2024-09-16T14:56:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4a80001dd0 con 0x7f4a700105c0 2024-09-16T14:56:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient(hunting): dump: 2024-09-16T14:56:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.425 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4026187438 0 0) 0x7f4a8c0a3070 con 0x7f4a700105c0 2024-09-16T14:56:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a8c1d1750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:56:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 >> v1:172.21.15.189:6789/0 conn(0x7f4a70012880 legacy=0x7f4a8c133240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:56:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 monclient: found mon.a 2024-09-16T14:56:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8c0a35a0 con 0x7f4a700105c0 2024-09-16T14:56:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:56:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:56:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a93e98640 5 monclient: authenticate success, global_id 274176 2024-09-16T14:56:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a93e98640 10 monclient: _renew_subs 2024-09-16T14:56:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a93e98640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a93e98640 1 -- 172.21.15.26:0/440040834 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a8c123970 con 0x7f4a700105c0 2024-09-16T14:56:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a80003c30 con 0x7f4a700105c0 2024-09-16T14:56:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4a80003f40 con 0x7f4a700105c0 2024-09-16T14:56:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: dump: 2024-09-16T14:56:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.430 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a93e98640 10 monclient: _renew_subs 2024-09-16T14:56:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a93e98640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7ffff640 4 set_mon_vals no callback set 2024-09-16T14:56:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a93e98640 1 -- 172.21.15.26:0/440040834 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8c133950 con 0x7f4a700105c0 2024-09-16T14:56:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4a800040a0 con 0x7f4a700105c0 2024-09-16T14:56:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a91c34640 1 -- 172.21.15.26:0/440040834 >> v1:172.21.15.26:6789/0 conn(0x7f4a700105c0 legacy=0x7f4a8c1c93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T14:56:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a91c34640 1 -- 172.21.15.26:0/440040834 >> v1:172.21.15.26:6789/0 conn(0x7f4a700105c0 legacy=0x7f4a8c1c93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:56:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a91c34640 1 --1- 172.21.15.26:0/440040834 >> v1:172.21.15.26:6789/0 conn(0x7f4a700105c0 0x7f4a8c1c93f0 :-1 s=OPENED pgs=963 cs=1 l=1).handle_message read tag failed 2024-09-16T14:56:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a91c34640 1 --1- 172.21.15.26:0/440040834 >> v1:172.21.15.26:6789/0 conn(0x7f4a700105c0 0x7f4a8c1c93f0 :-1 s=OPENED pgs=963 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:56:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a93e98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a93e98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a93e98640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a93e98640 1 -- 172.21.15.26:0/440040834 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a8c0a01d0 con 0x7f4a700105c0 2024-09-16T14:56:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.26:6789/0 2024-09-16T14:56:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:14.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 >> v1:172.21.15.26:6789/0 conn(0x7f4a700105c0 legacy=0x7f4a8c1c93f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:14.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:56:14.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient(hunting): picked mon.c con 0x7f4a8c1357f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:14.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient(hunting): picked mon.a con 0x7f4a7003ed30 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:14.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient(hunting): picked mon.b con 0x7f4a70012880 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:14.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4a8c0a3070 con 0x7f4a7003ed30 2024-09-16T14:56:14.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4a8c121e00 con 0x7f4a70012880 2024-09-16T14:56:14.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4a700114f0 con 0x7f4a8c1357f0 2024-09-16T14:56:14.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:14.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a91433640 1 -- 172.21.15.26:0/440040834 >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a7003d3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:14.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a91c34640 10 monclient(hunting): get_auth_request con 0x7f4a700376c0 auth_method 0 2024-09-16T14:56:14.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430324288 0 0) 0x7f4a8c0a3070 con 0x7f4a7003ed30 2024-09-16T14:56:14.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:14.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:14.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T14:56:14.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 --> v1:172.21.15.26:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f4a70046a60 con 0x7f4a7003ed30 2024-09-16T14:56:14.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2870561392 0 0) 0x7f4a70046a60 con 0x7f4a7003ed30 2024-09-16T14:56:14.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 >> v1:172.21.15.26:6790/0 conn(0x7f4a8c1357f0 legacy=0x7f4a7003d3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:56:14.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 >> v1:172.21.15.189:6789/0 conn(0x7f4a70012880 legacy=0x7f4a70044e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:56:14.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 monclient: found mon.a 2024-09-16T14:56:14.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:14.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:14.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f4a70049af0 con 0x7f4a7003ed30 2024-09-16T14:56:14.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:14.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:14.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a700114f0 con 0x7f4a7003ed30 2024-09-16T14:56:14.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:56:14.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:56:14.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:14.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4a84004100 con 0x7f4a7003ed30 2024-09-16T14:56:14.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:14.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: dump: 2024-09-16T14:56:14.434 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.434 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.435 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.435 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.435 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.435 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.435 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.435 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.435 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.435 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a84003f60 con 0x7f4a7003ed30 2024-09-16T14:56:14.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:14.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.425+0000 7f4a7f7fe640 4 set_mon_vals no callback set 2024-09-16T14:56:14.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.429+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4a8400eb70 con 0x7f4a7003ed30 2024-09-16T14:56:14.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.429+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.0 v1:172.21.15.26:6789/0 6 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f4a84043140 con 0x7f4a7003ed30 2024-09-16T14:56:14.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.433+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4a840139b0 con 0x7f4a7003ed30 2024-09-16T14:56:14.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.433+0000 7f4a7e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:14.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.433+0000 7f4a7e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:14.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.653+0000 7f4a93e98640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:56:14.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.653+0000 7f4a93e98640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:56:14.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.653+0000 7f4a93e98640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:14.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.653+0000 7f4a93e98640 1 -- 172.21.15.26:0/440040834 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4a8c0a01d0 con 0x7f4a7003ed30 2024-09-16T14:56:14.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.657+0000 7f4a7e7fc640 1 -- 172.21.15.26:0/440040834 <== mon.0 v1:172.21.15.26:6789/0 8 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 1109595477) 0x7f4a8c0a01d0 con 0x7f4a7003ed30 2024-09-16T14:56:14.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.657+0000 7f4a7e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:56:14.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.657+0000 7f4a7e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:14.656 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":128,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:56:14.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.657+0000 7f4a93e98640 1 -- 172.21.15.26:0/440040834 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4a700376c0 legacy=0x7f4a70039b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.657+0000 7f4a93e98640 10 monclient: shutdown 2024-09-16T14:56:14.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.657+0000 7f4a93e98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:14.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.657+0000 7f4a93e98640 1 -- 172.21.15.26:0/440040834 >> v1:172.21.15.26:6789/0 conn(0x7f4a7003ed30 legacy=0x7f4a700411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.657+0000 7f4a93e98640 1 -- 172.21.15.26:0/440040834 shutdown_connections 2024-09-16T14:56:14.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.657+0000 7f4a93e98640 1 -- 172.21.15.26:0/440040834 >> 172.21.15.26:0/440040834 conn(0x7f4a8c0a0da0 msgr2=0x7f4a8c0a1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:14.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.657+0000 7f4a93e98640 1 -- 172.21.15.26:0/440040834 shutdown_connections 2024-09-16T14:56:14.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.657+0000 7f4a93e98640 1 -- 172.21.15.26:0/440040834 wait complete. 2024-09-16T14:56:14.675 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:56:14.676 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T14:56:14.676 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:56:14.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 10 monclient: get_monmap_and_config 2024-09-16T14:56:14.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 10 monclient: build_initial_monmap 2024-09-16T14:56:14.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 10 monclient: monmap: 2024-09-16T14:56:14.796 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:14.796 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.796 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:14.797650+0000 2024-09-16T14:56:14.796 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:14.797650+0000 2024-09-16T14:56:14.796 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:14.796 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.796 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:14.796 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:14.796 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:14.796 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 1 Processor -- start 2024-09-16T14:56:14.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 1 -- start start 2024-09-16T14:56:14.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 10 monclient: init 2024-09-16T14:56:14.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:14.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:56:14.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 10 monclient(hunting): picked mon.noname-c con 0x7f01940a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:14.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 10 monclient(hunting): picked mon.noname-a con 0x7f0194099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:14.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 10 monclient(hunting): picked mon.noname-b con 0x7f01940a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:14.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0194098ea0 con 0x7f0194099b60 2024-09-16T14:56:14.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01940997b0 con 0x7f01940a1c00 2024-09-16T14:56:14.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0194099460 con 0x7f01940a26f0 2024-09-16T14:56:14.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:14.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 10 monclient(hunting): authenticate will time out at 1798.917841s 2024-09-16T14:56:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f01937fe640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f01940a26f0 legacy=0x7f019413d330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f01927fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0194099b60 0x7f0194139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33996/0 (socket says 172.21.15.26:33996) 2024-09-16T14:56:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f01927fc640 1 -- 172.21.15.26:0/1871712585 learned_addr learned my addr 172.21.15.26:0/1871712585 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 1 -- 172.21.15.26:0/1871712585 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291060346 0 0) 0x7f0194098ea0 con 0x7f0194099b60 2024-09-16T14:56:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 10 monclient(hunting): my global_id is 274179 2024-09-16T14:56:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 1 -- 172.21.15.26:0/1871712585 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f017c003530 con 0x7f0194099b60 2024-09-16T14:56:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 1 -- 172.21.15.26:0/1871712585 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1031186500 0 0) 0x7f017c003530 con 0x7f0194099b60 2024-09-16T14:56:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 1 -- 172.21.15.26:0/1871712585 >> v1:172.21.15.26:6790/0 conn(0x7f01940a26f0 legacy=0x7f019413d330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:56:14.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 1 -- 172.21.15.26:0/1871712585 >> v1:172.21.15.189:6789/0 conn(0x7f01940a1c00 legacy=0x7f01940a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:56:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 1 monclient: found mon.noname-a 2024-09-16T14:56:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:56:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 1 -- 172.21.15.26:0/1871712585 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01940a34b0 con 0x7f0194099b60 2024-09-16T14:56:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 10 monclient: _finish_auth 0 2024-09-16T14:56:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:56:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 1 -- 172.21.15.26:0/1871712585 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0188001620 con 0x7f0194099b60 2024-09-16T14:56:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 5 monclient: authenticate success, global_id 274179 2024-09-16T14:56:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 10 monclient: dump: 2024-09-16T14:56:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.807 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 1 -- 172.21.15.26:0/1871712585 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0188001940 con 0x7f0194099b60 2024-09-16T14:56:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 1 -- 172.21.15.26:0/1871712585 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0188001d10 con 0x7f0194099b60 2024-09-16T14:56:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0191ffb640 10 monclient: dump: 2024-09-16T14:56:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.809 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 4 set_mon_vals no callback set 2024-09-16T14:56:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 10 monclient: shutdown 2024-09-16T14:56:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 1 -- 172.21.15.26:0/1871712585 >> v1:172.21.15.26:6789/0 conn(0x7f0194099b60 legacy=0x7f0194139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 1 -- 172.21.15.26:0/1871712585 shutdown_connections 2024-09-16T14:56:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 1 -- 172.21.15.26:0/1871712585 >> 172.21.15.26:0/1871712585 conn(0x7f01940a3d00 msgr2=0x7f01940a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 1 -- 172.21.15.26:0/1871712585 shutdown_connections 2024-09-16T14:56:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 1 -- 172.21.15.26:0/1871712585 wait complete. 2024-09-16T14:56:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 10 monclient: build_initial_monmap 2024-09-16T14:56:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.797+0000 7f0199904640 10 monclient: monmap: 2024-09-16T14:56:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:14.811 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 1 Processor -- start 2024-09-16T14:56:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 1 -- start start 2024-09-16T14:56:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 10 monclient: init 2024-09-16T14:56:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:56:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 10 monclient(hunting): picked mon.noname-b con 0x7f0194099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 10 monclient(hunting): picked mon.noname-c con 0x7f01940a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 10 monclient(hunting): picked mon.noname-a con 0x7f01940a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019413db40 con 0x7f01940a26f0 2024-09-16T14:56:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019413bb60 con 0x7f0194099b60 2024-09-16T14:56:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019413ace0 con 0x7f01940a1c00 2024-09-16T14:56:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 10 monclient(hunting): authenticate will time out at 1798.920820s 2024-09-16T14:56:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f01927fc640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f01940a1c00 legacy=0x7f019413a5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f01937fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f01940a26f0 0x7f01941d1930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34006/0 (socket says 172.21.15.26:34006) 2024-09-16T14:56:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f01937fe640 1 -- 172.21.15.26:0/573268628 learned_addr learned my addr 172.21.15.26:0/573268628 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343198126 0 0) 0x7f019413db40 con 0x7f01940a26f0 2024-09-16T14:56:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient(hunting): my global_id is 274182 2024-09-16T14:56:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0180003530 con 0x7f01940a26f0 2024-09-16T14:56:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 342098590 0 0) 0x7f0180003530 con 0x7f01940a26f0 2024-09-16T14:56:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f019413db40 con 0x7f01940a26f0 2024-09-16T14:56:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f018c004040 con 0x7f01940a26f0 2024-09-16T14:56:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient(hunting): dump: 2024-09-16T14:56:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.814 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517635616 0 0) 0x7f019413bb60 con 0x7f0194099b60 2024-09-16T14:56:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient(hunting): my global_id is 264265 2024-09-16T14:56:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0180003530 con 0x7f0194099b60 2024-09-16T14:56:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1156188883 0 0) 0x7f019413db40 con 0x7f01940a26f0 2024-09-16T14:56:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 >> v1:172.21.15.26:6790/0 conn(0x7f01940a1c00 legacy=0x7f019413a5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:56:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 >> v1:172.21.15.189:6789/0 conn(0x7f0194099b60 legacy=0x7f0194139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:56:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 monclient: found mon.a 2024-09-16T14:56:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019409f270 con 0x7f01940a26f0 2024-09-16T14:56:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient: _finish_auth 0 2024-09-16T14:56:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:56:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 5 monclient: authenticate success, global_id 274182 2024-09-16T14:56:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 10 monclient: _renew_subs 2024-09-16T14:56:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:14.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 1 -- 172.21.15.26:0/573268628 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019409e3f0 con 0x7f01940a26f0 2024-09-16T14:56:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 10 monclient: _renew_subs 2024-09-16T14:56:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 1 -- 172.21.15.26:0/573268628 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f019413de20 con 0x7f01940a26f0 2024-09-16T14:56:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f018c004360 con 0x7f01940a26f0 2024-09-16T14:56:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f018c0046f0 con 0x7f01940a26f0 2024-09-16T14:56:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f01917fa640 4 set_mon_vals no callback set 2024-09-16T14:56:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient: dump: 2024-09-16T14:56:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.816 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.817 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.817 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.817 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.817 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.817 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.817 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.817 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:14.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f0199904640 1 -- 172.21.15.26:0/573268628 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0174006590 con 0x7f0174002830 2024-09-16T14:56:14.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f018c004850 con 0x7f01940a26f0 2024-09-16T14:56:14.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f018c043d60 con 0x7f01940a26f0 2024-09-16T14:56:14.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779268667 0 0) 0x7f0174006590 con 0x7f0174002830 2024-09-16T14:56:14.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:14.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:14.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:14.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient(hunting): my global_id is 274185 2024-09-16T14:56:14.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f019413db40 con 0x7f0174002830 2024-09-16T14:56:14.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1759257289 0 0) 0x7f019413db40 con 0x7f0174002830 2024-09-16T14:56:14.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0174006590 con 0x7f0174002830 2024-09-16T14:56:14.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0184001a20 con 0x7f0174002830 2024-09-16T14:56:14.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f01927fc640 10 monclient: get_auth_request con 0x7f0180036130 auth_method 0 2024-09-16T14:56:14.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:14.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient: dump: 2024-09-16T14:56:14.818 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.818 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.818 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.819 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1177107908 0 0) 0x7f0174006590 con 0x7f0174002830 2024-09-16T14:56:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f017400a0e0 con 0x7f0174002830 2024-09-16T14:56:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0184001d40 con 0x7f0174002830 2024-09-16T14:56:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 >> v1:172.21.15.26:6789/0 conn(0x7f0174002830 legacy=0x7f0174004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.801+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 >> v1:172.21.15.26:6789/0 conn(0x7f0174002830 legacy=0x7f0174004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f0199904640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:56:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f0199904640 1 -- 172.21.15.26:0/573268628 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0174009da0 con 0x7f01740090c0 2024-09-16T14:56:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233068527 0 0) 0x7f0174009da0 con 0x7f01740090c0 2024-09-16T14:56:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 10 monclient(hunting): my global_id is 274188 2024-09-16T14:56:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0174006590 con 0x7f01740090c0 2024-09-16T14:56:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3868062065 0 0) 0x7f0174006590 con 0x7f01740090c0 2024-09-16T14:56:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0174009da0 con 0x7f01740090c0 2024-09-16T14:56:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0184003b20 con 0x7f01740090c0 2024-09-16T14:56:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 10 monclient: dump: 2024-09-16T14:56:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.823 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1510888285 0 0) 0x7f0174009da0 con 0x7f01740090c0 2024-09-16T14:56:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f017400c290 con 0x7f01740090c0 2024-09-16T14:56:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4201 (unknown 0 0 1353524555) 0x7f0184003e20 con 0x7f01740090c0 2024-09-16T14:56:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:56:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 >> v1:172.21.15.26:6789/0 conn(0x7f01740090c0 legacy=0x7f0174009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f017bfff640 1 -- 172.21.15.26:0/573268628 >> v1:172.21.15.26:6789/0 conn(0x7f01740090c0 legacy=0x7f0174009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:14.823 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:56:14.823 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:56:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:56:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:56:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 128, 2024-09-16T14:56:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:56:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:56:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: 1 2024-09-16T14:56:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:56:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:56:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:56:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:14.824 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:56:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:56:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:14.825 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:56:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:56:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:56:14.826 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:56:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:56:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:56:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:56:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:56:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:56:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:56:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:56:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:56:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:56:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:14.827 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:56:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:56:14.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:56:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:56:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:56:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:56:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:56:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:56:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:56:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:56:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:56:14.830 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:56:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:56:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:56:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:56:14.831 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:56:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:56:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:56:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:14.832 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:56:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:56:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:56:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.833 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:56:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:56:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:56:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:56:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.834 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:56:14.835 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:56:14.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f0199904640 1 -- 172.21.15.26:0/573268628 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0180036130 legacy=0x7f01800385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f0199904640 10 monclient: shutdown 2024-09-16T14:56:14.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f0199904640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:14.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f01937fe640 1 -- 172.21.15.26:0/573268628 reap_dead start 2024-09-16T14:56:14.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f0199904640 1 -- 172.21.15.26:0/573268628 >> v1:172.21.15.26:6789/0 conn(0x7f01940a26f0 legacy=0x7f01941d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f0199904640 1 -- 172.21.15.26:0/573268628 shutdown_connections 2024-09-16T14:56:14.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f0199904640 1 -- 172.21.15.26:0/573268628 >> 172.21.15.26:0/573268628 conn(0x7f01940a3d00 msgr2=0x7f01940a0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:14.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f0199904640 1 -- 172.21.15.26:0/573268628 shutdown_connections 2024-09-16T14:56:14.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.809+0000 7f0199904640 1 -- 172.21.15.26:0/573268628 wait complete. 2024-09-16T14:56:14.837 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:56:14.837 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T14:56:14.837 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:56:14.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 10 monclient: get_monmap_and_config 2024-09-16T14:56:14.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 10 monclient: build_initial_monmap 2024-09-16T14:56:14.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 10 monclient: monmap: 2024-09-16T14:56:14.956 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:14.956 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.956 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:14.958251+0000 2024-09-16T14:56:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:14.958251+0000 2024-09-16T14:56:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:14.957 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 1 Processor -- start 2024-09-16T14:56:14.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 1 -- start start 2024-09-16T14:56:14.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 10 monclient: init 2024-09-16T14:56:14.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:14.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:56:14.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 10 monclient(hunting): picked mon.noname-a con 0x7ff7000a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:14.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 10 monclient(hunting): picked mon.noname-c con 0x7ff7000a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:14.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 10 monclient(hunting): picked mon.noname-b con 0x7ff7000a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:14.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7000a3250 con 0x7ff7000a1d10 2024-09-16T14:56:14.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70009f600 con 0x7ff7000a0000 2024-09-16T14:56:14.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7000a3830 con 0x7ff7000a2800 2024-09-16T14:56:14.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:14.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 10 monclient(hunting): authenticate will time out at 1799.078405s 2024-09-16T14:56:14.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fffff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7ff7000a2800 legacy=0x7ff70013d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:14.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6ff7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff7000a1d10 0x7ff7000a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34030/0 (socket says 172.21.15.26:34030) 2024-09-16T14:56:14.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6ff7fe640 1 -- 172.21.15.26:0/552862434 learned_addr learned my addr 172.21.15.26:0/552862434 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:14.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 1 -- 172.21.15.26:0/552862434 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433887622 0 0) 0x7ff7000a3250 con 0x7ff7000a1d10 2024-09-16T14:56:14.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:14.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:14.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:14.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 10 monclient(hunting): my global_id is 274191 2024-09-16T14:56:14.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 1 -- 172.21.15.26:0/552862434 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e8003530 con 0x7ff7000a1d10 2024-09-16T14:56:14.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 1 -- 172.21.15.26:0/552862434 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3595014201 0 0) 0x7ff6e8003530 con 0x7ff7000a1d10 2024-09-16T14:56:14.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 1 -- 172.21.15.26:0/552862434 >> v1:172.21.15.26:6790/0 conn(0x7ff7000a2800 legacy=0x7ff70013d440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:56:14.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 1 -- 172.21.15.26:0/552862434 >> v1:172.21.15.189:6789/0 conn(0x7ff7000a0000 legacy=0x7ff700139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:56:14.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 1 monclient: found mon.noname-a 2024-09-16T14:56:14.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:14.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:56:14.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 1 -- 172.21.15.26:0/552862434 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70009ed60 con 0x7ff7000a1d10 2024-09-16T14:56:14.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 10 monclient: _finish_auth 0 2024-09-16T14:56:14.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:56:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 1 -- 172.21.15.26:0/552862434 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6f0001620 con 0x7ff7000a1d10 2024-09-16T14:56:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 10 monclient: dump: 2024-09-16T14:56:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.967 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 5 monclient: authenticate success, global_id 274191 2024-09-16T14:56:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 1 -- 172.21.15.26:0/552862434 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6f0001940 con 0x7ff7000a1d10 2024-09-16T14:56:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 1 -- 172.21.15.26:0/552862434 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6f0001cd0 con 0x7ff7000a1d10 2024-09-16T14:56:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 4 set_mon_vals no callback set 2024-09-16T14:56:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff6fe7fc640 10 monclient: dump: 2024-09-16T14:56:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.968 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 10 monclient: shutdown 2024-09-16T14:56:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 1 -- 172.21.15.26:0/552862434 >> v1:172.21.15.26:6789/0 conn(0x7ff7000a1d10 legacy=0x7ff7000a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 1 -- 172.21.15.26:0/552862434 shutdown_connections 2024-09-16T14:56:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 1 -- 172.21.15.26:0/552862434 >> 172.21.15.26:0/552862434 conn(0x7ff7000a7f30 msgr2=0x7ff7000a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 1 -- 172.21.15.26:0/552862434 shutdown_connections 2024-09-16T14:56:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.957+0000 7ff705c3a640 1 -- 172.21.15.26:0/552862434 wait complete. 2024-09-16T14:56:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 10 monclient: build_initial_monmap 2024-09-16T14:56:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 10 monclient: monmap: 2024-09-16T14:56:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:14.970 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 1 Processor -- start 2024-09-16T14:56:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 1 -- start start 2024-09-16T14:56:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 10 monclient: init 2024-09-16T14:56:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:56:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 10 monclient(hunting): picked mon.noname-a con 0x7ff7000a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 10 monclient(hunting): picked mon.noname-c con 0x7ff7000a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 10 monclient(hunting): picked mon.noname-b con 0x7ff7000a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff700135780 con 0x7ff7000a0000 2024-09-16T14:56:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70013ad20 con 0x7ff7000a2800 2024-09-16T14:56:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70013c850 con 0x7ff7000a1d10 2024-09-16T14:56:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 10 monclient(hunting): authenticate will time out at 1799.081237s 2024-09-16T14:56:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6feffd640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7ff7000a1d10 legacy=0x7ff7001cd980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6ff7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff7000a0000 0x7ff70013d510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34042/0 (socket says 172.21.15.26:34042) 2024-09-16T14:56:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6ff7fe640 1 -- 172.21.15.26:0/130759525 learned_addr learned my addr 172.21.15.26:0/130759525 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464263880 0 0) 0x7ff700135780 con 0x7ff7000a0000 2024-09-16T14:56:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient(hunting): my global_id is 274194 2024-09-16T14:56:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6ec003530 con 0x7ff7000a0000 2024-09-16T14:56:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 190654687 0 0) 0x7ff6ec003530 con 0x7ff7000a0000 2024-09-16T14:56:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff700135780 con 0x7ff7000a0000 2024-09-16T14:56:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6f00015e0 con 0x7ff7000a0000 2024-09-16T14:56:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient(hunting): dump: 2024-09-16T14:56:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.973 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987032506 0 0) 0x7ff70013ad20 con 0x7ff7000a2800 2024-09-16T14:56:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient(hunting): my global_id is 264271 2024-09-16T14:56:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6ec003530 con 0x7ff7000a2800 2024-09-16T14:56:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 889434172 0 0) 0x7ff700135780 con 0x7ff7000a0000 2024-09-16T14:56:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 >> v1:172.21.15.26:6790/0 conn(0x7ff7000a1d10 legacy=0x7ff7001cd980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:56:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 >> v1:172.21.15.189:6789/0 conn(0x7ff7000a2800 legacy=0x7ff7001d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:56:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 monclient: found mon.a 2024-09-16T14:56:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70013cb30 con 0x7ff7000a0000 2024-09-16T14:56:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient: _finish_auth 0 2024-09-16T14:56:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:56:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 5 monclient: authenticate success, global_id 274194 2024-09-16T14:56:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 10 monclient: _renew_subs 2024-09-16T14:56:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 1 -- 172.21.15.26:0/130759525 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70013ce10 con 0x7ff7000a0000 2024-09-16T14:56:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 10 monclient: _renew_subs 2024-09-16T14:56:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 1 -- 172.21.15.26:0/130759525 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff70013ddf0 con 0x7ff7000a0000 2024-09-16T14:56:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6f0004cd0 con 0x7ff7000a0000 2024-09-16T14:56:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6f0005060 con 0x7ff7000a0000 2024-09-16T14:56:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6fd7fa640 4 set_mon_vals no callback set 2024-09-16T14:56:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient: dump: 2024-09-16T14:56:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.976 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff6f0005240 con 0x7ff7000a0000 2024-09-16T14:56:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff705c3a640 1 -- 172.21.15.26:0/130759525 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff6e0006590 con 0x7ff6e0002830 2024-09-16T14:56:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7ff6f0044140 con 0x7ff7000a0000 2024-09-16T14:56:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6fffff640 10 monclient: get_auth_request con 0x7ff6ec036110 auth_method 0 2024-09-16T14:56:14.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586256890 0 0) 0x7ff6e0006590 con 0x7ff6e0002830 2024-09-16T14:56:14.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:14.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:14.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:14.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 10 monclient(hunting): my global_id is 264274 2024-09-16T14:56:14.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.961+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff700135780 con 0x7ff6e0002830 2024-09-16T14:56:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.965+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2301608800 0 0) 0x7ff700135780 con 0x7ff6e0002830 2024-09-16T14:56:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.965+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6e0006590 con 0x7ff6e0002830 2024-09-16T14:56:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.965+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6f40016a0 con 0x7ff6e0002830 2024-09-16T14:56:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.965+0000 7ff6e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.965+0000 7ff6e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:56:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.965+0000 7ff6e7fff640 10 monclient: dump: 2024-09-16T14:56:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.980 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.980 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.980 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.980 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.965+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 758735196 0 0) 0x7ff6e0006590 con 0x7ff6e0002830 2024-09-16T14:56:14.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.965+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6e000a0e0 con 0x7ff6e0002830 2024-09-16T14:56:14.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.965+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ff6f4001a90 con 0x7ff6e0002830 2024-09-16T14:56:14.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.965+0000 7ff6e7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:14.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.965+0000 7ff6e7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:14.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.965+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 >> v1:172.21.15.189:6789/0 conn(0x7ff6e0002830 legacy=0x7ff6e0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.965+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 >> v1:172.21.15.189:6789/0 conn(0x7ff6e0002830 legacy=0x7ff6e0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:14.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.969+0000 7ff705c3a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:56:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.969+0000 7ff705c3a640 1 -- 172.21.15.26:0/130759525 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff6e0009d80 con 0x7ff6e00090a0 2024-09-16T14:56:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527404210 0 0) 0x7ff6e0009d80 con 0x7ff6e00090a0 2024-09-16T14:56:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 10 monclient(hunting): my global_id is 264277 2024-09-16T14:56:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e0006590 con 0x7ff6e00090a0 2024-09-16T14:56:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1842277141 0 0) 0x7ff6e0006590 con 0x7ff6e00090a0 2024-09-16T14:56:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6e0009d80 con 0x7ff6e00090a0 2024-09-16T14:56:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff6f40037a0 con 0x7ff6e00090a0 2024-09-16T14:56:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:56:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 10 monclient: dump: 2024-09-16T14:56:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:14.982 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 820164762 0 0) 0x7ff6e0009d80 con 0x7ff6e00090a0 2024-09-16T14:56:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff6e000c290 con 0x7ff6e00090a0 2024-09-16T14:56:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3871 (unknown 0 0 400307328) 0x7ff6f40057d0 con 0x7ff6e00090a0 2024-09-16T14:56:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:56:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 >> v1:172.21.15.189:6789/0 conn(0x7ff6e00090a0 legacy=0x7ff6e0009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6e7fff640 1 -- 172.21.15.26:0/130759525 >> v1:172.21.15.189:6789/0 conn(0x7ff6e00090a0 legacy=0x7ff6e0009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:14.983 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:56:14.983 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:56:14.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:56:14.983 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:56:14.983 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 128, 2024-09-16T14:56:14.983 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:56:14.983 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:56:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: 1 2024-09-16T14:56:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:56:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:56:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:56:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:14.984 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:56:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:56:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:14.985 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:56:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:56:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:56:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:56:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:56:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:56:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:14.986 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:56:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:56:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:56:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:56:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:56:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:56:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:56:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:56:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:14.987 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:56:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:56:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:56:14.988 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:56:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:56:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:56:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:56:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:14.989 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:56:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:56:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:56:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:14.990 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:56:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:56:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:56:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:56:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:14.991 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:56:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:56:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:56:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:14.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:56:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:56:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:56:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:14.993 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:14.994 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:14.994 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:56:14.994 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:14.994 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:14.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:14.994 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:56:14.994 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:56:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff705c3a640 1 -- 172.21.15.26:0/130759525 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff6ec036110 legacy=0x7ff6ec0385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff705c3a640 10 monclient: shutdown 2024-09-16T14:56:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff705c3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff705c3a640 1 -- 172.21.15.26:0/130759525 >> v1:172.21.15.26:6789/0 conn(0x7ff7000a0000 legacy=0x7ff70013d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6fffff640 1 -- 172.21.15.26:0/130759525 reap_dead start 2024-09-16T14:56:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff6fffff640 1 -- 172.21.15.26:0/130759525 reap_dead start 2024-09-16T14:56:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff705c3a640 1 -- 172.21.15.26:0/130759525 shutdown_connections 2024-09-16T14:56:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff705c3a640 1 -- 172.21.15.26:0/130759525 >> 172.21.15.26:0/130759525 conn(0x7ff7000a7f30 msgr2=0x7ff70009d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff705c3a640 1 -- 172.21.15.26:0/130759525 shutdown_connections 2024-09-16T14:56:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:14.973+0000 7ff705c3a640 1 -- 172.21.15.26:0/130759525 wait complete. 2024-09-16T14:56:14.997 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:56:14.997 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:56:33.324 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=110 2024-09-16T14:56:33.325 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 110 -ne 110 ]] 2024-09-16T14:56:33.325 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:56:33.325 INFO:tasks.workunit.client.0.smithi189.stderr:+ export CEPH_ARGS= 2024-09-16T14:56:33.326 INFO:tasks.workunit.client.0.smithi189.stderr:+ CEPH_ARGS= 2024-09-16T14:56:33.326 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph auth del client.foo 2024-09-16T14:56:33.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 10 monclient: get_monmap_and_config 2024-09-16T14:56:33.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 10 monclient: build_initial_monmap 2024-09-16T14:56:33.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 10 monclient: monmap: 2024-09-16T14:56:33.437 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:56:33.437 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:33.437 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:56:33.442056+0000 2024-09-16T14:56:33.438 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:56:33.442056+0000 2024-09-16T14:56:33.438 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:33.438 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:56:33.438 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:33.438 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:33.438 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:33.438 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:56:33.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 1 Processor -- start 2024-09-16T14:56:33.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 1 -- start start 2024-09-16T14:56:33.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 10 monclient: init 2024-09-16T14:56:33.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:33.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:56:33.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 10 monclient(hunting): picked mon.noname-c con 0x7f43f809c680 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:33.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 10 monclient(hunting): picked mon.noname-b con 0x7f43f809a960 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:33.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 10 monclient(hunting): picked mon.noname-a con 0x7f43f809d170 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:33.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f809b8c0 con 0x7f43f809d170 2024-09-16T14:56:33.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f809dcd0 con 0x7f43f809c680 2024-09-16T14:56:33.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f8099b30 con 0x7f43f809a960 2024-09-16T14:56:33.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:33.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 10 monclient(hunting): authenticate will time out at 1775.557680s 2024-09-16T14:56:33.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f6ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f43f809c680 0x7f43f809ca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47400/0 (socket says 172.21.15.189:47400) 2024-09-16T14:56:33.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f6ffd640 1 -- 172.21.15.189:0/2671456363 learned_addr learned my addr 172.21.15.189:0/2671456363 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:56:33.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f67fc640 1 -- 172.21.15.189:0/2671456363 >> v1:172.21.15.26:6790/0 conn(0x7f43f809a960 legacy=0x7f43f81369b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:33.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 1 -- 172.21.15.189:0/2671456363 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788975422 0 0) 0x7f43f809dcd0 con 0x7f43f809c680 2024-09-16T14:56:33.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:33.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:33.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:33.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 10 monclient(hunting): my global_id is 264280 2024-09-16T14:56:33.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 1 -- 172.21.15.189:0/2671456363 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e4003530 con 0x7f43f809c680 2024-09-16T14:56:33.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 1 -- 172.21.15.189:0/2671456363 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2998458011 0 0) 0x7f43e4003530 con 0x7f43f809c680 2024-09-16T14:56:33.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 1 -- 172.21.15.189:0/2671456363 >> v1:172.21.15.26:6790/0 conn(0x7f43f809a960 legacy=0x7f43f81369b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:56:33.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 1 -- 172.21.15.189:0/2671456363 >> v1:172.21.15.26:6789/0 conn(0x7f43f809d170 legacy=0x7f43f813a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:33.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:56:33.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 1 monclient: found mon.noname-c 2024-09-16T14:56:33.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:33.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:56:33.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 1 -- 172.21.15.189:0/2671456363 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f8099e00 con 0x7f43f809c680 2024-09-16T14:56:33.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 10 monclient: _finish_auth 0 2024-09-16T14:56:33.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:56:33.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 1 -- 172.21.15.189:0/2671456363 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f43f00019a0 con 0x7f43f809c680 2024-09-16T14:56:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:56:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 10 monclient: dump: 2024-09-16T14:56:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:56:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:56:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:33.448 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:56:33.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 5 monclient: authenticate success, global_id 264280 2024-09-16T14:56:33.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:33.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 1 -- 172.21.15.189:0/2671456363 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43f0001cc0 con 0x7f43f809c680 2024-09-16T14:56:33.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:33.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 1 -- 172.21.15.189:0/2671456363 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f43f0002050 con 0x7f43f809c680 2024-09-16T14:56:33.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:33.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:56:33.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43f5ffb640 10 monclient: dump: 2024-09-16T14:56:33.448 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:56:33.448 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:33.448 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:33.449 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:33.449 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:33.449 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:56:33.449 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:33.449 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:33.449 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:33.449 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:56:33.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:33.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 4 set_mon_vals no callback set 2024-09-16T14:56:33.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 10 monclient: shutdown 2024-09-16T14:56:33.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:33.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 1 -- 172.21.15.189:0/2671456363 >> v1:172.21.15.189:6789/0 conn(0x7f43f809c680 legacy=0x7f43f809ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:33.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 1 -- 172.21.15.189:0/2671456363 shutdown_connections 2024-09-16T14:56:33.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 1 -- 172.21.15.189:0/2671456363 >> 172.21.15.189:0/2671456363 conn(0x7f43f809faa0 msgr2=0x7f43f809feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:33.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 1 -- 172.21.15.189:0/2671456363 shutdown_connections 2024-09-16T14:56:33.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.440+0000 7f43fd729640 1 -- 172.21.15.189:0/2671456363 wait complete. 2024-09-16T14:56:33.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 monclient: build_initial_monmap 2024-09-16T14:56:33.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 monclient: monmap: 2024-09-16T14:56:33.451 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T14:56:33.451 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:33.451 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T14:56:33.451 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T14:56:33.451 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:33.451 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T14:56:33.451 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:33.451 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:33.451 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:33.451 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:56:33.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 1 Processor -- start 2024-09-16T14:56:33.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 1 -- start start 2024-09-16T14:56:33.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 monclient: init 2024-09-16T14:56:33.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:33.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:56:33.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 monclient(hunting): picked mon.noname-a con 0x7f43f809a960 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:33.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 monclient(hunting): picked mon.noname-c con 0x7f43f809c680 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:33.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 monclient(hunting): picked mon.noname-b con 0x7f43f809d170 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:33.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f81a52c0 con 0x7f43f809a960 2024-09-16T14:56:33.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f809a200 con 0x7f43f809d170 2024-09-16T14:56:33.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f81a6590 con 0x7f43f809c680 2024-09-16T14:56:33.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:33.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 monclient(hunting): authenticate will time out at 1775.561102s 2024-09-16T14:56:33.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43f77fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f43f809d170 0x7f43f81d13b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47416/0 (socket says 172.21.15.189:47416) 2024-09-16T14:56:33.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43f77fe640 1 -- 172.21.15.189:0/1203679220 learned_addr learned my addr 172.21.15.189:0/1203679220 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T14:56:33.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43f67fc640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f43f809c680 legacy=0x7f43f81b4b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T14:56:33.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443532331 0 0) 0x7f43f809b8c0 con 0x7f43f809d170 2024-09-16T14:56:33.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:56:33.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:33.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:33.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient(hunting): my global_id is 264283 2024-09-16T14:56:33.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d8003530 con 0x7f43f809d170 2024-09-16T14:56:33.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3616490355 0 0) 0x7f43d8003530 con 0x7f43f809d170 2024-09-16T14:56:33.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43f809b8c0 con 0x7f43f809d170 2024-09-16T14:56:33.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f43e8001720 con 0x7f43f809d170 2024-09-16T14:56:33.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:33.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:56:33.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient(hunting): dump: 2024-09-16T14:56:33.454 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:56:33.454 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:33.454 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:33.454 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:33.454 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:33.454 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:56:33.454 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:33.454 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:33.455 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:33.455 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:56:33.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:33.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2432711 0 0) 0x7f43f81a52c0 con 0x7f43f809a960 2024-09-16T14:56:33.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:56:33.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:33.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:33.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient(hunting): my global_id is 274200 2024-09-16T14:56:33.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d8003530 con 0x7f43f809a960 2024-09-16T14:56:33.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2322660689 0 0) 0x7f43f809b8c0 con 0x7f43f809d170 2024-09-16T14:56:33.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 >> v1:172.21.15.26:6790/0 conn(0x7f43f809c680 legacy=0x7f43f81b4b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:56:33.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 >> v1:172.21.15.26:6789/0 conn(0x7f43f809a960 legacy=0x7f43f81c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:33.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:56:33.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 1 monclient: found mon.b 2024-09-16T14:56:33.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:33.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:56:33.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f81a6c80 con 0x7f43f809d170 2024-09-16T14:56:33.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient: _finish_auth 0 2024-09-16T14:56:33.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:56:33.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:33.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 5 monclient: authenticate success, global_id 264283 2024-09-16T14:56:33.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 monclient: _renew_subs 2024-09-16T14:56:33.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:56:33.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 1 -- 172.21.15.189:0/1203679220 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f81a6f60 con 0x7f43f809d170 2024-09-16T14:56:33.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 monclient: _renew_subs 2024-09-16T14:56:33.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:56:33.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 1 -- 172.21.15.189:0/1203679220 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f81b5450 con 0x7f43f809d170 2024-09-16T14:56:33.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43e8003620 con 0x7f43f809d170 2024-09-16T14:56:33.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:33.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f43e80039b0 con 0x7f43f809d170 2024-09-16T14:56:33.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:33.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:56:33.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 10 monclient: dump: 2024-09-16T14:56:33.458 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T14:56:33.458 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:33.458 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:33.458 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:33.458 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:33.458 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T14:56:33.458 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:33.458 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:33.458 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:33.458 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T14:56:33.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f43e8003b90 con 0x7f43f809d170 2024-09-16T14:56:33.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43f4ff9640 4 set_mon_vals no callback set 2024-09-16T14:56:33.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:33.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f43e8043c90 con 0x7f43f809d170 2024-09-16T14:56:33.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:33.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:56:33.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43fd729640 1 -- 172.21.15.189:0/1203679220 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f806e3b0 con 0x7f43f809d170 2024-09-16T14:56:33.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.444+0000 7f43f6ffd640 10 monclient: get_auth_request con 0x7f43d8036120 auth_method 0 2024-09-16T14:56:33.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.448+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f43e8014620 con 0x7f43f809d170 2024-09-16T14:56:33.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.448+0000 7f43ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:33.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.448+0000 7f43ef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:33.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.668+0000 7f43fd729640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T14:56:33.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.668+0000 7f43fd729640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T14:56:33.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.668+0000 7f43fd729640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:56:33.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.668+0000 7f43fd729640 1 -- 172.21.15.189:0/1203679220 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.foo"} v 0) v1 -- 0x7f43f806a390 con 0x7f43f809d170 2024-09-16T14:56:33.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.672+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.foo"}]=0 v35) v1 ==== 80+0+0 (unknown 3476768894 0 0) 0x7f43e8012a90 con 0x7f43f809d170 2024-09-16T14:56:33.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.672+0000 7f43ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T14:56:33.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.672+0000 7f43ef7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:33.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.812+0000 7f43fd729640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T14:56:33.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.812+0000 7f43fd729640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T14:56:33.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.812+0000 7f43fd729640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T14:56:33.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.812+0000 7f43fd729640 1 -- 172.21.15.189:0/1203679220 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.foo"} v 0) v1 -- 0x7f43f809f5a0 con 0x7f43f809d170 2024-09-16T14:56:33.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.812+0000 7f43ef7fe640 1 -- 172.21.15.189:0/1203679220 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.foo"}]=0 v35) v1 ==== 80+0+0 (unknown 3476768894 0 0) 0x7f43e80143f0 con 0x7f43f809d170 2024-09-16T14:56:33.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.812+0000 7f43ef7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T14:56:33.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.812+0000 7f43ef7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T14:56:33.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.816+0000 7f43fd729640 1 -- 172.21.15.189:0/1203679220 >> v1:172.21.15.189:6812/3196161582 conn(0x7f43d8036120 legacy=0x7f43d80385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:33.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.816+0000 7f43fd729640 10 monclient: shutdown 2024-09-16T14:56:33.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.816+0000 7f43fd729640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:33.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.816+0000 7f43fd729640 1 -- 172.21.15.189:0/1203679220 >> v1:172.21.15.189:6789/0 conn(0x7f43f809d170 legacy=0x7f43f81d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:33.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.816+0000 7f43fd729640 1 -- 172.21.15.189:0/1203679220 shutdown_connections 2024-09-16T14:56:33.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.816+0000 7f43fd729640 1 -- 172.21.15.189:0/1203679220 >> 172.21.15.189:0/1203679220 conn(0x7f43f809faa0 msgr2=0x7f43f81b7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:33.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.816+0000 7f43fd729640 1 -- 172.21.15.189:0/1203679220 shutdown_connections 2024-09-16T14:56:33.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T14:56:33.816+0000 7f43fd729640 1 -- 172.21.15.189:0/1203679220 wait complete. 2024-09-16T14:56:33.826 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 13 2024-09-16T14:56:33.826 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 2024-09-16T14:56:33.826 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T14:56:33.827 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T14:56:33.827 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T14:56:33.827 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T14:56:34.998 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T14:56:34.998 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T14:56:34.998 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T14:56:34.998 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T14:56:35.001 INFO:tasks.ceph.mon.c:Started 2024-09-16T14:56:35.001 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:56:35.002 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:56:35.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.125+0000 7f5c5c121640 10 monclient: get_monmap_and_config 2024-09-16T14:56:35.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.125+0000 7f5c5c121640 10 monclient: build_initial_monmap 2024-09-16T14:56:35.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.125+0000 7f5c5c121640 10 monclient: monmap: 2024-09-16T14:56:35.127 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:35.127 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.127 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:35.129115+0000 2024-09-16T14:56:35.127 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:35.129115+0000 2024-09-16T14:56:35.127 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:35.127 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.128 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:35.128 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:35.128 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:35.128 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.129+0000 7f5c5c121640 1 Processor -- start 2024-09-16T14:56:35.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.129+0000 7f5c5c121640 1 -- start start 2024-09-16T14:56:35.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.129+0000 7f5c5c121640 10 monclient: init 2024-09-16T14:56:35.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.129+0000 7f5c5c121640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:35.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.129+0000 7f5c5c121640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:56:35.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.129+0000 7f5c5c121640 10 monclient(hunting): picked mon.noname-b con 0x7f5c5406c5a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:35.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.129+0000 7f5c5c121640 10 monclient(hunting): picked mon.noname-a con 0x7f5c540a23b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:35.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.129+0000 7f5c5c121640 10 monclient(hunting): picked mon.noname-c con 0x7f5c540a27c0 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:35.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.129+0000 7f5c5c121640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5406cee0 con 0x7f5c540a23b0 2024-09-16T14:56:35.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.129+0000 7f5c5c121640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5406c980 con 0x7f5c540a27c0 2024-09-16T14:56:35.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.129+0000 7f5c5c121640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c540a15e0 con 0x7f5c5406c5a0 2024-09-16T14:56:35.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.129+0000 7f5c5c121640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:35.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.129+0000 7f5c5c121640 10 monclient(hunting): authenticate will time out at 1819.249172s 2024-09-16T14:56:35.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.129+0000 7f5c59ebd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5c5406c5a0 0x7f5c540a1ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60848/0 (socket says 172.21.15.26:60848) 2024-09-16T14:56:35.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.129+0000 7f5c59ebd640 1 -- 172.21.15.26:0/1980460785 learned_addr learned my addr 172.21.15.26:0/1980460785 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:35.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 1 -- 172.21.15.26:0/1980460785 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747200367 0 0) 0x7f5c5406cee0 con 0x7f5c540a23b0 2024-09-16T14:56:35.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 10 monclient(hunting): my global_id is 274206 2024-09-16T14:56:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 1 -- 172.21.15.26:0/1980460785 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c40003530 con 0x7f5c540a23b0 2024-09-16T14:56:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 1 -- 172.21.15.26:0/1980460785 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1472426848 0 0) 0x7f5c40003530 con 0x7f5c540a23b0 2024-09-16T14:56:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 1 -- 172.21.15.26:0/1980460785 >> v1:172.21.15.26:6790/0 conn(0x7f5c5406c5a0 legacy=0x7f5c540a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 1 -- 172.21.15.26:0/1980460785 >> v1:172.21.15.189:6789/0 conn(0x7f5c540a27c0 legacy=0x7f5c5413d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 10 monclient: _finish_hunting 0 2024-09-16T14:56:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 1 monclient: found mon.noname-a 2024-09-16T14:56:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:56:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 1 -- 172.21.15.26:0/1980460785 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5409edd0 con 0x7f5c540a23b0 2024-09-16T14:56:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 10 monclient: _finish_auth 0 2024-09-16T14:56:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 10 monclient: _check_auth_tickets 2024-09-16T14:56:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 1 -- 172.21.15.26:0/1980460785 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5c44001620 con 0x7f5c540a23b0 2024-09-16T14:56:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c58ebb640 10 monclient: dump: 2024-09-16T14:56:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.159 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c5c121640 5 monclient: authenticate success, global_id 274206 2024-09-16T14:56:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.153+0000 7f5c5c121640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c58ebb640 1 -- 172.21.15.26:0/1980460785 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c44001840 con 0x7f5c540a23b0 2024-09-16T14:56:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c58ebb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c58ebb640 1 -- 172.21.15.26:0/1980460785 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5c44001bd0 con 0x7f5c540a23b0 2024-09-16T14:56:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c58ebb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c58ebb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c58ebb640 10 monclient: dump: 2024-09-16T14:56:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.160 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 4 set_mon_vals no callback set 2024-09-16T14:56:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 10 monclient: shutdown 2024-09-16T14:56:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 1 -- 172.21.15.26:0/1980460785 >> v1:172.21.15.26:6789/0 conn(0x7f5c540a23b0 legacy=0x7f5c54139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 1 -- 172.21.15.26:0/1980460785 shutdown_connections 2024-09-16T14:56:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 1 -- 172.21.15.26:0/1980460785 >> 172.21.15.26:0/1980460785 conn(0x7f5c540a3da0 msgr2=0x7f5c540a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 1 -- 172.21.15.26:0/1980460785 shutdown_connections 2024-09-16T14:56:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 1 -- 172.21.15.26:0/1980460785 wait complete. 2024-09-16T14:56:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 10 monclient: build_initial_monmap 2024-09-16T14:56:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 10 monclient: monmap: 2024-09-16T14:56:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:35.166 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 1 Processor -- start 2024-09-16T14:56:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 1 -- start start 2024-09-16T14:56:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 10 monclient: init 2024-09-16T14:56:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:56:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 10 monclient(hunting): picked mon.noname-b con 0x7f5c5406c5a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 10 monclient(hunting): picked mon.noname-c con 0x7f5c540a23b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 10 monclient(hunting): picked mon.noname-a con 0x7f5c540a27c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5413a1b0 con 0x7f5c540a27c0 2024-09-16T14:56:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c540a64d0 con 0x7f5c5406c5a0 2024-09-16T14:56:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5409e490 con 0x7f5c540a23b0 2024-09-16T14:56:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c5c121640 10 monclient(hunting): authenticate will time out at 1819.278684s 2024-09-16T14:56:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c596bc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5c540a23b0 0x7f5c54139aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60856/0 (socket says 172.21.15.26:60856) 2024-09-16T14:56:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.157+0000 7f5c596bc640 1 -- 172.21.15.26:0/2743920318 learned_addr learned my addr 172.21.15.26:0/2743920318 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 1 -- 172.21.15.26:0/2743920318 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606705337 0 0) 0x7f5c5409e490 con 0x7f5c540a23b0 2024-09-16T14:56:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 10 monclient(hunting): my global_id is 284099 2024-09-16T14:56:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 1 -- 172.21.15.26:0/2743920318 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c38003530 con 0x7f5c540a23b0 2024-09-16T14:56:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 1 -- 172.21.15.26:0/2743920318 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3955760830 0 0) 0x7f5c38003530 con 0x7f5c540a23b0 2024-09-16T14:56:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 1 -- 172.21.15.26:0/2743920318 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c5409e490 con 0x7f5c540a23b0 2024-09-16T14:56:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 1 -- 172.21.15.26:0/2743920318 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5c440015e0 con 0x7f5c540a23b0 2024-09-16T14:56:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:56:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 10 monclient(hunting): dump: 2024-09-16T14:56:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.169 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 1 -- 172.21.15.26:0/2743920318 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3060113236 0 0) 0x7f5c5409e490 con 0x7f5c540a23b0 2024-09-16T14:56:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 1 -- 172.21.15.26:0/2743920318 >> v1:172.21.15.189:6789/0 conn(0x7f5c5406c5a0 legacy=0x7f5c54139390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 1 -- 172.21.15.26:0/2743920318 >> v1:172.21.15.26:6789/0 conn(0x7f5c540a27c0 legacy=0x7f5c541d1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:56:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 1 monclient: found mon.c 2024-09-16T14:56:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 1 -- 172.21.15.26:0/2743920318 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5413c3b0 con 0x7f5c540a23b0 2024-09-16T14:56:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:56:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:56:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c5c121640 5 monclient: authenticate success, global_id 284099 2024-09-16T14:56:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c5c121640 10 monclient: _renew_subs 2024-09-16T14:56:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c5c121640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c5c121640 1 -- 172.21.15.26:0/2743920318 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5413def0 con 0x7f5c540a23b0 2024-09-16T14:56:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c5c121640 10 monclient: _renew_subs 2024-09-16T14:56:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c5c121640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c5c121640 1 -- 172.21.15.26:0/2743920318 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5413caa0 con 0x7f5c540a23b0 2024-09-16T14:56:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 1 -- 172.21.15.26:0/2743920318 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c44003a00 con 0x7f5c540a23b0 2024-09-16T14:56:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 1 -- 172.21.15.26:0/2743920318 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5c44003d90 con 0x7f5c540a23b0 2024-09-16T14:56:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 10 monclient: dump: 2024-09-16T14:56:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.172 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4bfff640 4 set_mon_vals no callback set 2024-09-16T14:56:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 1 -- 172.21.15.26:0/2743920318 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5c44003ef0 con 0x7f5c540a23b0 2024-09-16T14:56:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c5c121640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c4a7fc640 1 -- 172.21.15.26:0/2743920318 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f5c44043fa0 con 0x7f5c540a23b0 2024-09-16T14:56:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c5c121640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:35.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c5c121640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:35.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c5c121640 1 -- 172.21.15.26:0/2743920318 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3c005050 con 0x7f5c540a23b0 2024-09-16T14:56:35.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.161+0000 7f5c59ebd640 10 monclient: get_auth_request con 0x7f5c38036120 auth_method 0 2024-09-16T14:56:35.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.165+0000 7f5c4a7fc640 1 -- 172.21.15.26:0/2743920318 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5c44014770 con 0x7f5c540a23b0 2024-09-16T14:56:35.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.165+0000 7f5c4a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:35.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.165+0000 7f5c4a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:35.351 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T14:56:35.352 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T14:56:35.352 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:56:35.352 INFO:tasks.workunit.client.0.smithi189.stderr:+ c=''\''allow command service with prefix=list, allow command quorum_status'\''' 2024-09-16T14:56:35.352 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph auth get-or-create client.bar mon '\''allow command service with prefix=list, allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.bar.keyring' 0 2024-09-16T14:56:35.352 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph auth get-or-create client.bar mon '\''allow command service with prefix=list, allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.bar.keyring' 2024-09-16T14:56:35.352 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=0 2024-09-16T14:56:35.353 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph auth get-or-create client.bar mon ''\''allow' command service with prefix=list, allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-16T14:56:35.353 INFO:tasks.workunit.client.0.smithi189.stdout:ceph auth get-or-create client.bar mon 'allow command service with prefix=list, allow command quorum_status' > /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-16T14:56:35.354 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph auth get-or-create client.bar mon ''\''allow' command service with prefix=list, allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-16T14:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.389+0000 7f5c5c121640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.389+0000 7f5c5c121640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.389+0000 7f5c5c121640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.389+0000 7f5c5c121640 1 -- 172.21.15.26:0/2743920318 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5c3c005880 con 0x7f5c540a23b0 2024-09-16T14:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.389+0000 7f5c4a7fc640 1 -- 172.21.15.26:0/2743920318 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 3293716622) 0x7f5c44012e10 con 0x7f5c540a23b0 2024-09-16T14:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.389+0000 7f5c4a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.389+0000 7f5c4a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:35.390 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.389+0000 7f5c5c121640 1 -- 172.21.15.26:0/2743920318 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5c38036120 legacy=0x7f5c380385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.389+0000 7f5c5c121640 10 monclient: shutdown 2024-09-16T14:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.389+0000 7f5c5c121640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.389+0000 7f5c5c121640 1 -- 172.21.15.26:0/2743920318 >> v1:172.21.15.26:6790/0 conn(0x7f5c540a23b0 legacy=0x7f5c54139aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.389+0000 7f5c5c121640 1 -- 172.21.15.26:0/2743920318 shutdown_connections 2024-09-16T14:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.389+0000 7f5c5c121640 1 -- 172.21.15.26:0/2743920318 >> 172.21.15.26:0/2743920318 conn(0x7f5c540a3da0 msgr2=0x7f5c5413e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.389+0000 7f5c5c121640 1 -- 172.21.15.26:0/2743920318 shutdown_connections 2024-09-16T14:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.389+0000 7f5c5c121640 1 -- 172.21.15.26:0/2743920318 wait complete. 2024-09-16T14:56:35.408 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:56:35.408 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:56:35.409 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:56:35.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.525+0000 7fa346e76640 10 monclient: get_monmap_and_config 2024-09-16T14:56:35.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.525+0000 7fa346e76640 10 monclient: build_initial_monmap 2024-09-16T14:56:35.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.525+0000 7fa346e76640 10 monclient: monmap: 2024-09-16T14:56:35.527 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:35.528 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.528 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:35.529446+0000 2024-09-16T14:56:35.528 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:35.529446+0000 2024-09-16T14:56:35.528 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:35.528 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.528 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:35.528 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:35.528 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:35.528 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 1 Processor -- start 2024-09-16T14:56:35.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 1 -- start start 2024-09-16T14:56:35.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 10 monclient: init 2024-09-16T14:56:35.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:35.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:56:35.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 10 monclient(hunting): picked mon.noname-c con 0x7fa34009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:35.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 10 monclient(hunting): picked mon.noname-a con 0x7fa340132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:35.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 10 monclient(hunting): picked mon.noname-b con 0x7fa340138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:35.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34009d260 con 0x7fa340132650 2024-09-16T14:56:35.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa340132a80 con 0x7fa34009cb80 2024-09-16T14:56:35.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34009cf60 con 0x7fa340138ff0 2024-09-16T14:56:35.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:35.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 10 monclient(hunting): authenticate will time out at 1819.649466s 2024-09-16T14:56:35.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa345413640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa340138ff0 0x7fa34013b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60866/0 (socket says 172.21.15.26:60866) 2024-09-16T14:56:35.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa345413640 1 -- 172.21.15.26:0/2930770325 learned_addr learned my addr 172.21.15.26:0/2930770325 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:35.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 1 -- 172.21.15.26:0/2930770325 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460480884 0 0) 0x7fa34009cf60 con 0x7fa340138ff0 2024-09-16T14:56:35.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient(hunting): my global_id is 284111 2024-09-16T14:56:35.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 1 -- 172.21.15.26:0/2930770325 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa328003530 con 0x7fa340138ff0 2024-09-16T14:56:35.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 1 -- 172.21.15.26:0/2930770325 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2362088606 0 0) 0x7fa34009d260 con 0x7fa340132650 2024-09-16T14:56:35.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient(hunting): my global_id is 274215 2024-09-16T14:56:35.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 1 -- 172.21.15.26:0/2930770325 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa34009cf60 con 0x7fa340132650 2024-09-16T14:56:35.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 1 -- 172.21.15.26:0/2930770325 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523561355 0 0) 0x7fa340132a80 con 0x7fa34009cb80 2024-09-16T14:56:35.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient(hunting): my global_id is 264298 2024-09-16T14:56:35.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 1 -- 172.21.15.26:0/2930770325 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa34009d260 con 0x7fa34009cb80 2024-09-16T14:56:35.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 1 -- 172.21.15.26:0/2930770325 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3360563918 0 0) 0x7fa328003530 con 0x7fa340138ff0 2024-09-16T14:56:35.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 1 -- 172.21.15.26:0/2930770325 >> v1:172.21.15.189:6789/0 conn(0x7fa34009cb80 legacy=0x7fa3400717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 1 -- 172.21.15.26:0/2930770325 >> v1:172.21.15.26:6789/0 conn(0x7fa340132650 legacy=0x7fa3401378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:56:35.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 1 monclient: found mon.noname-b 2024-09-16T14:56:35.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:35.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:56:35.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 1 -- 172.21.15.26:0/2930770325 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34009f2d0 con 0x7fa340138ff0 2024-09-16T14:56:35.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:56:35.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:56:35.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:35.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 1 -- 172.21.15.26:0/2930770325 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa3380019e0 con 0x7fa340138ff0 2024-09-16T14:56:35.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 5 monclient: authenticate success, global_id 284111 2024-09-16T14:56:35.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:35.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:56:35.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient: dump: 2024-09-16T14:56:35.539 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.540 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.540 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.540 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.540 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.540 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.540 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.540 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.540 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.540 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:35.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 1 -- 172.21.15.26:0/2930770325 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa338001c00 con 0x7fa340138ff0 2024-09-16T14:56:35.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:35.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 1 -- 172.21.15.26:0/2930770325 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa338001f50 con 0x7fa340138ff0 2024-09-16T14:56:35.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:35.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 4 set_mon_vals no callback set 2024-09-16T14:56:35.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:35.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa33f7fe640 10 monclient: dump: 2024-09-16T14:56:35.541 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.541 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.541 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.541 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.541 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.541 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.541 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.541 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.542 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.542 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 10 monclient: shutdown 2024-09-16T14:56:35.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:35.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 1 -- 172.21.15.26:0/2930770325 >> v1:172.21.15.26:6790/0 conn(0x7fa340138ff0 legacy=0x7fa34013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 1 -- 172.21.15.26:0/2930770325 shutdown_connections 2024-09-16T14:56:35.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 1 -- 172.21.15.26:0/2930770325 >> 172.21.15.26:0/2930770325 conn(0x7fa34009fb50 msgr2=0x7fa34009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:35.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 1 -- 172.21.15.26:0/2930770325 shutdown_connections 2024-09-16T14:56:35.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 1 -- 172.21.15.26:0/2930770325 wait complete. 2024-09-16T14:56:35.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 10 monclient: build_initial_monmap 2024-09-16T14:56:35.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 10 monclient: monmap: 2024-09-16T14:56:35.542 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:35.542 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:35.543 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.529+0000 7fa346e76640 1 Processor -- start 2024-09-16T14:56:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 1 -- start start 2024-09-16T14:56:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 10 monclient: init 2024-09-16T14:56:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:35.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:56:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 10 monclient(hunting): picked mon.noname-c con 0x7fa34009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 10 monclient(hunting): picked mon.noname-a con 0x7fa340132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 10 monclient(hunting): picked mon.noname-b con 0x7fa340138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa340138b00 con 0x7fa340132650 2024-09-16T14:56:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34009b370 con 0x7fa340138ff0 2024-09-16T14:56:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34013c400 con 0x7fa34009cb80 2024-09-16T14:56:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 10 monclient(hunting): authenticate will time out at 1819.652575s 2024-09-16T14:56:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa344c12640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa34009cb80 0x7fa34009a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60870/0 (socket says 172.21.15.26:60870) 2024-09-16T14:56:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa344c12640 1 -- 172.21.15.26:0/3605294247 learned_addr learned my addr 172.21.15.26:0/3605294247 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033330618 0 0) 0x7fa34013c400 con 0x7fa34009cb80 2024-09-16T14:56:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): my global_id is 284114 2024-09-16T14:56:35.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa32c003530 con 0x7fa34009cb80 2024-09-16T14:56:35.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093329392 0 0) 0x7fa340138b00 con 0x7fa340132650 2024-09-16T14:56:35.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): my global_id is 274218 2024-09-16T14:56:35.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa34013c400 con 0x7fa340132650 2024-09-16T14:56:35.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1089834213 0 0) 0x7fa34013c400 con 0x7fa340132650 2024-09-16T14:56:35.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa340138b00 con 0x7fa340132650 2024-09-16T14:56:35.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339482776 0 0) 0x7fa34009b370 con 0x7fa340138ff0 2024-09-16T14:56:35.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): my global_id is 264301 2024-09-16T14:56:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa34013c400 con 0x7fa340138ff0 2024-09-16T14:56:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 589864527 0 0) 0x7fa32c003530 con 0x7fa34009cb80 2024-09-16T14:56:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa34009b370 con 0x7fa34009cb80 2024-09-16T14:56:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa334001890 con 0x7fa340132650 2024-09-16T14:56:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): dump: 2024-09-16T14:56:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.546 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.547 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa330001880 con 0x7fa34009cb80 2024-09-16T14:56:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): dump: 2024-09-16T14:56:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.547 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.548 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.548 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.548 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.548 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.548 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.548 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.548 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.548 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3153812803 0 0) 0x7fa340138b00 con 0x7fa340132650 2024-09-16T14:56:35.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 >> v1:172.21.15.26:6790/0 conn(0x7fa34009cb80 legacy=0x7fa34009a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 >> v1:172.21.15.189:6789/0 conn(0x7fa340138ff0 legacy=0x7fa3401d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:56:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 monclient: found mon.a 2024-09-16T14:56:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3400a28f0 con 0x7fa340132650 2024-09-16T14:56:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient: _finish_auth 0 2024-09-16T14:56:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:56:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 5 monclient: authenticate success, global_id 274218 2024-09-16T14:56:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 10 monclient: _renew_subs 2024-09-16T14:56:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 1 -- 172.21.15.26:0/3605294247 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34009a240 con 0x7fa340132650 2024-09-16T14:56:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 10 monclient: _renew_subs 2024-09-16T14:56:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:35.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 1 -- 172.21.15.26:0/3605294247 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3401385f0 con 0x7fa340132650 2024-09-16T14:56:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa334003600 con 0x7fa340132650 2024-09-16T14:56:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33e7fc640 4 set_mon_vals no callback set 2024-09-16T14:56:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa334003910 con 0x7fa340132650 2024-09-16T14:56:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient: dump: 2024-09-16T14:56:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.550 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.551 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa346e76640 1 -- 172.21.15.26:0/3605294247 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa320006590 con 0x7fa320002830 2024-09-16T14:56:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa33400e380 con 0x7fa340132650 2024-09-16T14:56:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fa334042bf0 con 0x7fa340132650 2024-09-16T14:56:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122768270 0 0) 0x7fa320006590 con 0x7fa320002830 2024-09-16T14:56:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient(hunting): my global_id is 274221 2024-09-16T14:56:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa340138b00 con 0x7fa320002830 2024-09-16T14:56:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa345413640 10 monclient: get_auth_request con 0x7fa32c0362e0 auth_method 0 2024-09-16T14:56:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2705753573 0 0) 0x7fa340138b00 con 0x7fa320002830 2024-09-16T14:56:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa320006590 con 0x7fa320002830 2024-09-16T14:56:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa330003a10 con 0x7fa320002830 2024-09-16T14:56:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient: dump: 2024-09-16T14:56:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.552 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.553 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2606027689 0 0) 0x7fa320006590 con 0x7fa320002830 2024-09-16T14:56:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa32000a0e0 con 0x7fa320002830 2024-09-16T14:56:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa3300057d0 con 0x7fa320002830 2024-09-16T14:56:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 >> v1:172.21.15.26:6789/0 conn(0x7fa320002830 legacy=0x7fa320004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.533+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 >> v1:172.21.15.26:6789/0 conn(0x7fa320002830 legacy=0x7fa320004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa346e76640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:56:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa346e76640 1 -- 172.21.15.26:0/3605294247 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa320009d80 con 0x7fa3200090a0 2024-09-16T14:56:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177238576 0 0) 0x7fa320009d80 con 0x7fa3200090a0 2024-09-16T14:56:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 10 monclient(hunting): my global_id is 274224 2024-09-16T14:56:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa320006590 con 0x7fa3200090a0 2024-09-16T14:56:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 519677602 0 0) 0x7fa320006590 con 0x7fa3200090a0 2024-09-16T14:56:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa320009d80 con 0x7fa3200090a0 2024-09-16T14:56:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa330004080 con 0x7fa3200090a0 2024-09-16T14:56:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 10 monclient: dump: 2024-09-16T14:56:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.556 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.557 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3226867042 0 0) 0x7fa320009d80 con 0x7fa3200090a0 2024-09-16T14:56:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa32000c290 con 0x7fa3200090a0 2024-09-16T14:56:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 2919932477) 0x7fa3300043f0 con 0x7fa3200090a0 2024-09-16T14:56:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:56:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 >> v1:172.21.15.26:6789/0 conn(0x7fa3200090a0 legacy=0x7fa320009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa33d7fa640 1 -- 172.21.15.26:0/3605294247 >> v1:172.21.15.26:6789/0 conn(0x7fa3200090a0 legacy=0x7fa320009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:35.558 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:56:35.558 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:56:35.558 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:56:35.558 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:56:35.558 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 132, 2024-09-16T14:56:35.558 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:56:35.558 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:56:35.558 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:56:35.558 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:56:35.558 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:35.559 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:56:35.559 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:35.559 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:56:35.559 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:56:35.559 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:35.559 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:35.559 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:35.559 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:35.559 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:35.559 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:35.559 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:35.559 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:35.559 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:35.560 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:35.560 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:35.560 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:56:35.560 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:56:35.560 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:35.560 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:35.560 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:35.560 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:35.560 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:35.560 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:35.560 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:35.561 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:35.561 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:35.561 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:35.561 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.561 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:56:35.561 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:56:35.561 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:56:35.561 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:56:35.561 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:56:35.561 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:56:35.561 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:35.561 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:35.561 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:56:35.562 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:56:35.562 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:56:35.562 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:56:35.562 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:56:35.562 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:56:35.562 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:56:35.562 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:35.562 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:56:35.563 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:35.563 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:35.563 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:35.563 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:35.563 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:35.563 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:35.563 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:35.563 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:35.563 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:35.563 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:35.563 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:35.563 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:56:35.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.563 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:56:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:56:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:56:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:56:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:56:35.565 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:56:35.565 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:35.565 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:35.565 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:35.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.565 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:56:35.565 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:56:35.565 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:35.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:35.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.565 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:35.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:56:35.566 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:35.566 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.566 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.566 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:56:35.566 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:56:35.566 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:35.566 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:35.566 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:35.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.566 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:56:35.566 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:56:35.566 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:35.567 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:35.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.567 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:35.567 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:56:35.567 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:35.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.567 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.567 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:56:35.567 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:56:35.567 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:35.567 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:35.567 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:35.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.568 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.568 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:56:35.568 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:56:35.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.568 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:35.568 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:35.568 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:35.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.569 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:35.569 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:56:35.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.569 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:35.569 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:35.569 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:56:35.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.569 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:35.569 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:56:35.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.569 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:35.569 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:35.570 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:56:35.570 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.570 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:35.570 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:56:35.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.570 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:35.570 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:35.570 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:35.570 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.570 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.570 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:56:35.570 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:56:35.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa346e76640 1 -- 172.21.15.26:0/3605294247 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa32c0362e0 legacy=0x7fa32c0387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa346e76640 10 monclient: shutdown 2024-09-16T14:56:35.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa346e76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:35.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa346e76640 1 -- 172.21.15.26:0/3605294247 >> v1:172.21.15.26:6789/0 conn(0x7fa340132650 legacy=0x7fa34009ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa345413640 1 -- 172.21.15.26:0/3605294247 reap_dead start 2024-09-16T14:56:35.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa345413640 1 -- 172.21.15.26:0/3605294247 reap_dead start 2024-09-16T14:56:35.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa346e76640 1 -- 172.21.15.26:0/3605294247 shutdown_connections 2024-09-16T14:56:35.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa346e76640 1 -- 172.21.15.26:0/3605294247 >> 172.21.15.26:0/3605294247 conn(0x7fa34009fb50 msgr2=0x7fa340131bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:35.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa346e76640 1 -- 172.21.15.26:0/3605294247 shutdown_connections 2024-09-16T14:56:35.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.541+0000 7fa346e76640 1 -- 172.21.15.26:0/3605294247 wait complete. 2024-09-16T14:56:35.573 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:56:35.573 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:56:35.573 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:56:35.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 10 monclient: get_monmap_and_config 2024-09-16T14:56:35.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 10 monclient: build_initial_monmap 2024-09-16T14:56:35.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 10 monclient: monmap: 2024-09-16T14:56:35.693 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:35.693 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.693 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:35.695084+0000 2024-09-16T14:56:35.694 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:35.695084+0000 2024-09-16T14:56:35.694 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:35.694 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.694 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:35.694 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:35.694 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:35.694 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 1 Processor -- start 2024-09-16T14:56:35.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 1 -- start start 2024-09-16T14:56:35.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 10 monclient: init 2024-09-16T14:56:35.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:35.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:56:35.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 10 monclient(hunting): picked mon.noname-b con 0x7f607809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:35.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 10 monclient(hunting): picked mon.noname-a con 0x7f6078132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:35.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 10 monclient(hunting): picked mon.noname-c con 0x7f6078138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:35.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607809d260 con 0x7f6078132650 2024-09-16T14:56:35.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6078132a80 con 0x7f6078138ff0 2024-09-16T14:56:35.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607809cf60 con 0x7f607809cb80 2024-09-16T14:56:35.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:35.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 10 monclient(hunting): authenticate will time out at 1819.815145s 2024-09-16T14:56:35.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607cba9640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6078132650 0x7f60781378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:55226/0 (socket says 172.21.15.26:55226) 2024-09-16T14:56:35.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607cba9640 1 -- 172.21.15.26:0/1255818115 learned_addr learned my addr 172.21.15.26:0/1255818115 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:35.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607d3aa640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f607809cb80 0x7f60780717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60884/0 (socket says 172.21.15.26:60884) 2024-09-16T14:56:35.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 1 -- 172.21.15.26:0/1255818115 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845758277 0 0) 0x7f607809d260 con 0x7f6078132650 2024-09-16T14:56:35.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 10 monclient(hunting): my global_id is 274227 2024-09-16T14:56:35.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 1 -- 172.21.15.26:0/1255818115 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6060003530 con 0x7f6078132650 2024-09-16T14:56:35.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 1 -- 172.21.15.26:0/1255818115 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819655940 0 0) 0x7f607809cf60 con 0x7f607809cb80 2024-09-16T14:56:35.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 10 monclient(hunting): my global_id is 284117 2024-09-16T14:56:35.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 1 -- 172.21.15.26:0/1255818115 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f607809d260 con 0x7f607809cb80 2024-09-16T14:56:35.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 1 -- 172.21.15.26:0/1255818115 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2335098667 0 0) 0x7f6060003530 con 0x7f6078132650 2024-09-16T14:56:35.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 1 -- 172.21.15.26:0/1255818115 >> v1:172.21.15.26:6790/0 conn(0x7f607809cb80 legacy=0x7f60780717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 1 -- 172.21.15.26:0/1255818115 >> v1:172.21.15.189:6789/0 conn(0x7f6078138ff0 legacy=0x7f607813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 10 monclient: _finish_hunting 0 2024-09-16T14:56:35.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 1 monclient: found mon.noname-a 2024-09-16T14:56:35.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:35.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:56:35.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 1 -- 172.21.15.26:0/1255818115 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607809f2d0 con 0x7f6078132650 2024-09-16T14:56:35.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 10 monclient: _finish_auth 0 2024-09-16T14:56:35.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 10 monclient: _check_auth_tickets 2024-09-16T14:56:35.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:35.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 5 monclient: authenticate success, global_id 274227 2024-09-16T14:56:35.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:35.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 1 -- 172.21.15.26:0/1255818115 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f606c001620 con 0x7f6078132650 2024-09-16T14:56:35.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:35.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f6077fff640 10 monclient: dump: 2024-09-16T14:56:35.712 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.712 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.712 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.713 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.713 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.713 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.713 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.713 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.713 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.714 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.693+0000 7f607f60e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:35.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6077fff640 1 -- 172.21.15.26:0/1255818115 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f606c001850 con 0x7f6078132650 2024-09-16T14:56:35.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6077fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:35.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:35.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 4 set_mon_vals no callback set 2024-09-16T14:56:35.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6077fff640 1 -- 172.21.15.26:0/1255818115 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f606c001be0 con 0x7f6078132650 2024-09-16T14:56:35.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6077fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6077fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:35.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6077fff640 10 monclient: dump: 2024-09-16T14:56:35.715 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.715 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.715 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.715 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.715 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.715 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.716 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.716 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.716 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.716 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 monclient: shutdown 2024-09-16T14:56:35.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:35.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 1 -- 172.21.15.26:0/1255818115 >> v1:172.21.15.26:6789/0 conn(0x7f6078132650 legacy=0x7f60781378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 1 -- 172.21.15.26:0/1255818115 shutdown_connections 2024-09-16T14:56:35.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 1 -- 172.21.15.26:0/1255818115 >> 172.21.15.26:0/1255818115 conn(0x7f607809fb50 msgr2=0x7f607809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:35.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 1 -- 172.21.15.26:0/1255818115 shutdown_connections 2024-09-16T14:56:35.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 1 -- 172.21.15.26:0/1255818115 wait complete. 2024-09-16T14:56:35.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 monclient: build_initial_monmap 2024-09-16T14:56:35.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 monclient: monmap: 2024-09-16T14:56:35.718 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:35.718 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:35.718 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:35.718 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:35.718 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:35.718 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:35.719 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:35.719 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:35.719 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:35.719 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 1 Processor -- start 2024-09-16T14:56:35.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 1 -- start start 2024-09-16T14:56:35.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 monclient: init 2024-09-16T14:56:35.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:35.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:56:35.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 monclient(hunting): picked mon.noname-a con 0x7f607809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:35.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 monclient(hunting): picked mon.noname-b con 0x7f6078132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:35.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 monclient(hunting): picked mon.noname-c con 0x7f6078138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:35.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6078138600 con 0x7f607809cb80 2024-09-16T14:56:35.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6078138d50 con 0x7f6078132650 2024-09-16T14:56:35.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607806dcb0 con 0x7f6078138ff0 2024-09-16T14:56:35.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:35.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 monclient(hunting): authenticate will time out at 1819.818148s 2024-09-16T14:56:35.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607d3aa640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f607809cb80 0x7f607809a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:55232/0 (socket says 172.21.15.26:55232) 2024-09-16T14:56:35.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607d3aa640 1 -- 172.21.15.26:0/4062445605 learned_addr learned my addr 172.21.15.26:0/4062445605 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:35.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797308184 0 0) 0x7f6078138600 con 0x7f607809cb80 2024-09-16T14:56:35.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): my global_id is 274230 2024-09-16T14:56:35.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6064003530 con 0x7f607809cb80 2024-09-16T14:56:35.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150777844 0 0) 0x7f607806dcb0 con 0x7f6078138ff0 2024-09-16T14:56:35.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): my global_id is 284120 2024-09-16T14:56:35.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6078138600 con 0x7f6078138ff0 2024-09-16T14:56:35.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3944618433 0 0) 0x7f6078138d50 con 0x7f6078132650 2024-09-16T14:56:35.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): my global_id is 264307 2024-09-16T14:56:35.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f607806dcb0 con 0x7f6078132650 2024-09-16T14:56:35.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3252087417 0 0) 0x7f6064003530 con 0x7f607809cb80 2024-09-16T14:56:35.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6078138d50 con 0x7f607809cb80 2024-09-16T14:56:35.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6068001bb0 con 0x7f607809cb80 2024-09-16T14:56:35.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:35.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): dump: 2024-09-16T14:56:35.723 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.723 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.723 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.723 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.724 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.724 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.724 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.724 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.724 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.724 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:35.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4261346704 0 0) 0x7f6078138600 con 0x7f6078138ff0 2024-09-16T14:56:35.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6064003530 con 0x7f6078138ff0 2024-09-16T14:56:35.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6070001780 con 0x7f6078138ff0 2024-09-16T14:56:35.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:35.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient(hunting): dump: 2024-09-16T14:56:35.725 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.725 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.725 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.725 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.725 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.725 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.725 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.725 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.725 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.725 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 536163653 0 0) 0x7f6078138d50 con 0x7f607809cb80 2024-09-16T14:56:35.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 >> v1:172.21.15.26:6790/0 conn(0x7f6078138ff0 legacy=0x7f60781d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 >> v1:172.21.15.189:6789/0 conn(0x7f6078132650 legacy=0x7f607809ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:56:35.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 monclient: found mon.a 2024-09-16T14:56:35.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:35.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:35.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607806df90 con 0x7f607809cb80 2024-09-16T14:56:35.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient: _finish_auth 0 2024-09-16T14:56:35.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:56:35.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:35.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 5 monclient: authenticate success, global_id 274230 2024-09-16T14:56:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 monclient: _renew_subs 2024-09-16T14:56:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 1 -- 172.21.15.26:0/4062445605 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607809b370 con 0x7f607809cb80 2024-09-16T14:56:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 monclient: _renew_subs 2024-09-16T14:56:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 1 -- 172.21.15.26:0/4062445605 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f607809c400 con 0x7f607809cb80 2024-09-16T14:56:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6068002100 con 0x7f607809cb80 2024-09-16T14:56:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6068004d10 con 0x7f607809cb80 2024-09-16T14:56:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f6075ffb640 10 monclient: dump: 2024-09-16T14:56:35.727 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.728 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.728 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.728 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.728 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.728 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.728 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.728 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.728 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.728 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f60777fe640 4 set_mon_vals no callback set 2024-09-16T14:56:35.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.697+0000 7f607f60e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:35.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f607f60e640 1 -- 172.21.15.26:0/4062445605 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f607813c620 con 0x7f60780a0e50 2024-09-16T14:56:35.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f606800f780 con 0x7f607809cb80 2024-09-16T14:56:35.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f6068043f40 con 0x7f607809cb80 2024-09-16T14:56:35.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256495236 0 0) 0x7f607813c620 con 0x7f60780a0e50 2024-09-16T14:56:35.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 10 monclient(hunting): my global_id is 284123 2024-09-16T14:56:35.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6078138d50 con 0x7f60780a0e50 2024-09-16T14:56:35.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f607dbab640 10 monclient: get_auth_request con 0x7f6064036470 auth_method 0 2024-09-16T14:56:35.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3616705085 0 0) 0x7f6078138d50 con 0x7f60780a0e50 2024-09-16T14:56:35.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f607813c620 con 0x7f60780a0e50 2024-09-16T14:56:35.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f606c0019c0 con 0x7f60780a0e50 2024-09-16T14:56:35.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:35.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 10 monclient: dump: 2024-09-16T14:56:35.731 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.731 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.731 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.731 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.731 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.731 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.731 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.731 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.731 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.731 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2134731140 0 0) 0x7f607813c620 con 0x7f60780a0e50 2024-09-16T14:56:35.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6078069f80 con 0x7f60780a0e50 2024-09-16T14:56:35.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f606c0047f0 con 0x7f60780a0e50 2024-09-16T14:56:35.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:35.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:35.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 >> v1:172.21.15.26:6790/0 conn(0x7f60780a0e50 legacy=0x7f60780a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.701+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 >> v1:172.21.15.26:6790/0 conn(0x7f60780a0e50 legacy=0x7f60780a1230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:35.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.705+0000 7f607f60e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:56:35.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.705+0000 7f607f60e640 1 -- 172.21.15.26:0/4062445605 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f607809c960 con 0x7f60780a1940 2024-09-16T14:56:35.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.705+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870443983 0 0) 0x7f607809c960 con 0x7f60780a1940 2024-09-16T14:56:35.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.705+0000 7f6075ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.705+0000 7f6075ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.705+0000 7f6075ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.705+0000 7f6075ffb640 10 monclient(hunting): my global_id is 284126 2024-09-16T14:56:35.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.705+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f607813c620 con 0x7f60780a1940 2024-09-16T14:56:35.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1830959201 0 0) 0x7f607813c620 con 0x7f60780a1940 2024-09-16T14:56:35.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f607809c960 con 0x7f60780a1940 2024-09-16T14:56:35.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f606c004cf0 con 0x7f60780a1940 2024-09-16T14:56:35.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f6075ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f6075ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:35.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f6075ffb640 10 monclient: dump: 2024-09-16T14:56:35.735 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.735 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.735 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.735 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.735 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.735 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.735 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.735 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.735 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.735 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3674009198 0 0) 0x7f607809c960 con 0x7f60780a1940 2024-09-16T14:56:35.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f607809f5b0 con 0x7f60780a1940 2024-09-16T14:56:35.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 3926507831) 0x7f606c005020 con 0x7f60780a1940 2024-09-16T14:56:35.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f6075ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:56:35.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f6075ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:35.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 >> v1:172.21.15.26:6790/0 conn(0x7f60780a1940 legacy=0x7f60781d72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f6075ffb640 1 -- 172.21.15.26:0/4062445605 >> v1:172.21.15.26:6790/0 conn(0x7f60780a1940 legacy=0x7f60781d72c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:35.736 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:56:35.736 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:56:35.736 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:56:35.736 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:56:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 132, 2024-09-16T14:56:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:56:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:56:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:56:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:56:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:56:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:56:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:56:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:35.737 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:56:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:56:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:35.738 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:56:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:56:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:56:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:56:35.739 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:56:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:56:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:56:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:56:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:56:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:56:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:56:35.740 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:56:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:56:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:56:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:35.741 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:56:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:56:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:56:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:56:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:35.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.743 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:35.743 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:56:35.743 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:35.743 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.743 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.743 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:56:35.743 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:56:35.743 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:35.743 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:35.743 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:35.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:56:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:56:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:56:35.744 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:56:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:56:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.745 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:56:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:56:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:35.746 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:56:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:56:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:56:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.747 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:56:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:56:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:56:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:35.748 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.749 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:56:35.749 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:56:35.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f607f60e640 1 -- 172.21.15.26:0/4062445605 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6064036470 legacy=0x7f6064038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f607f60e640 10 monclient: shutdown 2024-09-16T14:56:35.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f607f60e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:35.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f607f60e640 1 -- 172.21.15.26:0/4062445605 >> v1:172.21.15.26:6789/0 conn(0x7f607809cb80 legacy=0x7f607809a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f607dbab640 1 -- 172.21.15.26:0/4062445605 reap_dead start 2024-09-16T14:56:35.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f607dbab640 1 -- 172.21.15.26:0/4062445605 reap_dead start 2024-09-16T14:56:35.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f607f60e640 1 -- 172.21.15.26:0/4062445605 shutdown_connections 2024-09-16T14:56:35.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f607f60e640 1 -- 172.21.15.26:0/4062445605 >> 172.21.15.26:0/4062445605 conn(0x7f607809fb50 msgr2=0x7f6078131650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:35.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f607f60e640 1 -- 172.21.15.26:0/4062445605 shutdown_connections 2024-09-16T14:56:35.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.709+0000 7f607f60e640 1 -- 172.21.15.26:0/4062445605 wait complete. 2024-09-16T14:56:35.752 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:56:35.752 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:56:35.752 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:56:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=0 2024-09-16T14:56:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T14:56:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:56:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 0 2024-09-16T14:56:35.845 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T14:56:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 2024-09-16T14:56:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=0 2024-09-16T14:56:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T14:56:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T14:56:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 10 monclient: get_monmap_and_config 2024-09-16T14:56:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 10 monclient: build_initial_monmap 2024-09-16T14:56:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 10 monclient: monmap: 2024-09-16T14:56:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:35.873969+0000 2024-09-16T14:56:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:35.873969+0000 2024-09-16T14:56:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.873 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:35.874 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:35.874 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:35.874 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 1 Processor -- start 2024-09-16T14:56:35.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 1 -- start start 2024-09-16T14:56:35.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 10 monclient: init 2024-09-16T14:56:35.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:35.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:56:35.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 10 monclient(hunting): picked mon.noname-a con 0x7f7df809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:35.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 10 monclient(hunting): picked mon.noname-b con 0x7f7df8132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:35.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 10 monclient(hunting): picked mon.noname-c con 0x7f7df8138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:35.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df809d260 con 0x7f7df809cb80 2024-09-16T14:56:35.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df8132a80 con 0x7f7df8138ff0 2024-09-16T14:56:35.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df809cf60 con 0x7f7df8132650 2024-09-16T14:56:35.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:35.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 10 monclient(hunting): authenticate will time out at 1819.994328s 2024-09-16T14:56:35.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df67fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7df8132650 0x7f7df81378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60930/0 (socket says 172.21.15.26:60930) 2024-09-16T14:56:35.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df67fc640 1 -- 172.21.15.26:0/1455350390 learned_addr learned my addr 172.21.15.26:0/1455350390 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:35.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df6ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7df809cb80 0x7f7df80717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:55236/0 (socket says 172.21.15.26:55236) 2024-09-16T14:56:35.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 1 -- 172.21.15.26:0/1455350390 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407280523 0 0) 0x7f7df809cf60 con 0x7f7df8132650 2024-09-16T14:56:35.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient(hunting): my global_id is 284129 2024-09-16T14:56:35.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 1 -- 172.21.15.26:0/1455350390 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7de4003530 con 0x7f7df8132650 2024-09-16T14:56:35.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 1 -- 172.21.15.26:0/1455350390 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167604087 0 0) 0x7f7df809d260 con 0x7f7df809cb80 2024-09-16T14:56:35.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient(hunting): my global_id is 274233 2024-09-16T14:56:35.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 1 -- 172.21.15.26:0/1455350390 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df809cf60 con 0x7f7df809cb80 2024-09-16T14:56:35.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 1 -- 172.21.15.26:0/1455350390 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834803976 0 0) 0x7f7df8132a80 con 0x7f7df8138ff0 2024-09-16T14:56:35.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient(hunting): my global_id is 264310 2024-09-16T14:56:35.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 1 -- 172.21.15.26:0/1455350390 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df809d260 con 0x7f7df8138ff0 2024-09-16T14:56:35.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 1 -- 172.21.15.26:0/1455350390 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 553934144 0 0) 0x7f7de4003530 con 0x7f7df8132650 2024-09-16T14:56:35.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 1 -- 172.21.15.26:0/1455350390 >> v1:172.21.15.189:6789/0 conn(0x7f7df8138ff0 legacy=0x7f7df813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 1 -- 172.21.15.26:0/1455350390 >> v1:172.21.15.26:6789/0 conn(0x7f7df809cb80 legacy=0x7f7df80717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:56:35.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 1 monclient: found mon.noname-b 2024-09-16T14:56:35.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:35.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:56:35.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 1 -- 172.21.15.26:0/1455350390 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df809f2d0 con 0x7f7df8132650 2024-09-16T14:56:35.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient: _finish_auth 0 2024-09-16T14:56:35.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:56:35.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:35.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 1 -- 172.21.15.26:0/1455350390 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7de0001660 con 0x7f7df8132650 2024-09-16T14:56:35.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 5 monclient: authenticate success, global_id 284129 2024-09-16T14:56:35.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:35.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:56:35.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient: dump: 2024-09-16T14:56:35.885 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.885 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.885 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.886 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.886 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.886 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.886 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.886 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.886 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.886 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:35.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 1 -- 172.21.15.26:0/1455350390 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7de0001880 con 0x7f7df8132650 2024-09-16T14:56:35.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:35.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:35.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 1 -- 172.21.15.26:0/1455350390 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7de0001c10 con 0x7f7df8132650 2024-09-16T14:56:35.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 4 set_mon_vals no callback set 2024-09-16T14:56:35.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:35.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7df5ffb640 10 monclient: dump: 2024-09-16T14:56:35.887 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.887 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.887 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.887 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.887 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.887 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.887 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.887 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.887 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.887 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 10 monclient: shutdown 2024-09-16T14:56:35.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:35.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 1 -- 172.21.15.26:0/1455350390 >> v1:172.21.15.26:6790/0 conn(0x7f7df8132650 legacy=0x7f7df81378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 1 -- 172.21.15.26:0/1455350390 shutdown_connections 2024-09-16T14:56:35.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 1 -- 172.21.15.26:0/1455350390 >> 172.21.15.26:0/1455350390 conn(0x7f7df809fb50 msgr2=0x7f7df809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:35.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 1 -- 172.21.15.26:0/1455350390 shutdown_connections 2024-09-16T14:56:35.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.873+0000 7f7dfda5e640 1 -- 172.21.15.26:0/1455350390 wait complete. 2024-09-16T14:56:35.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 10 monclient: build_initial_monmap 2024-09-16T14:56:35.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 10 monclient: monmap: 2024-09-16T14:56:35.888 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:35.888 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:35.888 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:35.888 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:35.889 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:35.889 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:35.889 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:35.889 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:35.889 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:35.889 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 1 Processor -- start 2024-09-16T14:56:35.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 1 -- start start 2024-09-16T14:56:35.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 10 monclient: init 2024-09-16T14:56:35.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:35.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:56:35.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 10 monclient(hunting): picked mon.noname-c con 0x7f7df809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:35.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 10 monclient(hunting): picked mon.noname-a con 0x7f7df8132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:35.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 10 monclient(hunting): picked mon.noname-b con 0x7f7df8138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:35.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df8134dd0 con 0x7f7df8132650 2024-09-16T14:56:35.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df8130330 con 0x7f7df8138ff0 2024-09-16T14:56:35.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df809a240 con 0x7f7df809cb80 2024-09-16T14:56:35.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:35.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 10 monclient(hunting): authenticate will time out at 1819.997268s 2024-09-16T14:56:35.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7df6ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7df809cb80 0x7f7df813a080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60944/0 (socket says 172.21.15.26:60944) 2024-09-16T14:56:35.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7df6ffd640 1 -- 172.21.15.26:0/1116150493 learned_addr learned my addr 172.21.15.26:0/1116150493 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:35.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7df67fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7df8132650 0x7f7df813a790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:55240/0 (socket says 172.21.15.26:55240) 2024-09-16T14:56:35.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004424357 0 0) 0x7f7df8134dd0 con 0x7f7df8132650 2024-09-16T14:56:35.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): my global_id is 274236 2024-09-16T14:56:35.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd8003530 con 0x7f7df8132650 2024-09-16T14:56:35.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540431282 0 0) 0x7f7df809a240 con 0x7f7df809cb80 2024-09-16T14:56:35.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): my global_id is 284132 2024-09-16T14:56:35.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df8134dd0 con 0x7f7df809cb80 2024-09-16T14:56:35.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903429510 0 0) 0x7f7df8130330 con 0x7f7df8138ff0 2024-09-16T14:56:35.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): my global_id is 264313 2024-09-16T14:56:35.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df809a240 con 0x7f7df8138ff0 2024-09-16T14:56:35.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3898664760 0 0) 0x7f7dd8003530 con 0x7f7df8132650 2024-09-16T14:56:35.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7df8130330 con 0x7f7df8132650 2024-09-16T14:56:35.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7de0001620 con 0x7f7df8132650 2024-09-16T14:56:35.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:35.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): dump: 2024-09-16T14:56:35.895 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.895 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.895 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.895 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.895 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.895 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.895 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.896 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.896 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.896 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:35.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3234664084 0 0) 0x7f7df8134dd0 con 0x7f7df809cb80 2024-09-16T14:56:35.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dd8003530 con 0x7f7df809cb80 2024-09-16T14:56:35.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7de8001780 con 0x7f7df809cb80 2024-09-16T14:56:35.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:35.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): dump: 2024-09-16T14:56:35.897 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.897 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.897 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.897 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.897 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.897 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.898 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.898 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.898 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.898 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 612834014 0 0) 0x7f7df809a240 con 0x7f7df8138ff0 2024-09-16T14:56:35.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7df8134dd0 con 0x7f7df8138ff0 2024-09-16T14:56:35.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7df0001c40 con 0x7f7df8138ff0 2024-09-16T14:56:35.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T14:56:35.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): dump: 2024-09-16T14:56:35.899 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.899 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.899 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.899 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.899 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.899 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.899 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.900 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.900 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.900 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3685775177 0 0) 0x7f7df8130330 con 0x7f7df8132650 2024-09-16T14:56:35.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 >> v1:172.21.15.26:6790/0 conn(0x7f7df809cb80 legacy=0x7f7df813a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 >> v1:172.21.15.189:6789/0 conn(0x7f7df8138ff0 legacy=0x7f7df81d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient: _finish_hunting 0 2024-09-16T14:56:35.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 monclient: found mon.a 2024-09-16T14:56:35.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:35.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:35.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df8131ee0 con 0x7f7df8132650 2024-09-16T14:56:35.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient: _finish_auth 0 2024-09-16T14:56:35.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient: _check_auth_tickets 2024-09-16T14:56:35.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:35.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 5 monclient: authenticate success, global_id 274236 2024-09-16T14:56:35.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 10 monclient: _renew_subs 2024-09-16T14:56:35.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:35.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df8134530 con 0x7f7df8132650 2024-09-16T14:56:35.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 10 monclient: _renew_subs 2024-09-16T14:56:35.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:35.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df8133660 con 0x7f7df8132650 2024-09-16T14:56:35.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7de0004aa0 con 0x7f7df8132650 2024-09-16T14:56:35.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:35.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7de0004db0 con 0x7f7df8132650 2024-09-16T14:56:35.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:35.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7df57fa640 4 set_mon_vals no callback set 2024-09-16T14:56:35.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient: dump: 2024-09-16T14:56:35.903 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.903 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.903 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.903 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.903 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.904 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.904 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.904 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.904 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.904 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:35.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7dfda5e640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7ddc006590 con 0x7f7ddc002830 2024-09-16T14:56:35.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7de000f820 con 0x7f7df8132650 2024-09-16T14:56:35.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f7de0043df0 con 0x7f7df8132650 2024-09-16T14:56:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292793259 0 0) 0x7f7ddc006590 con 0x7f7ddc002830 2024-09-16T14:56:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient(hunting): my global_id is 264316 2024-09-16T14:56:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df8130330 con 0x7f7ddc002830 2024-09-16T14:56:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7df77fe640 10 monclient: get_auth_request con 0x7f7dd8036530 auth_method 0 2024-09-16T14:56:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3756562144 0 0) 0x7f7df8130330 con 0x7f7ddc002830 2024-09-16T14:56:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ddc006590 con 0x7f7ddc002830 2024-09-16T14:56:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7de80047f0 con 0x7f7ddc002830 2024-09-16T14:56:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.877+0000 7f7deffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.881+0000 7f7deffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:56:35.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.881+0000 7f7deffff640 10 monclient: dump: 2024-09-16T14:56:35.906 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.906 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.906 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.906 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.906 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.906 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.906 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.906 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.906 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.907 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.881+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 480739603 0 0) 0x7f7ddc006590 con 0x7f7ddc002830 2024-09-16T14:56:35.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.881+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ddc00a0e0 con 0x7f7ddc002830 2024-09-16T14:56:35.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.881+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7de8006ad0 con 0x7f7ddc002830 2024-09-16T14:56:35.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.881+0000 7f7deffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:35.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.881+0000 7f7deffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:35.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.881+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 >> v1:172.21.15.189:6789/0 conn(0x7f7ddc002830 legacy=0x7f7ddc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.881+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 >> v1:172.21.15.189:6789/0 conn(0x7f7ddc002830 legacy=0x7f7ddc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:35.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.885+0000 7f7dfda5e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:56:35.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.885+0000 7f7dfda5e640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7ddc009da0 con 0x7f7ddc0090c0 2024-09-16T14:56:35.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.885+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086714777 0 0) 0x7f7ddc009da0 con 0x7f7ddc0090c0 2024-09-16T14:56:35.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.885+0000 7f7deffff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:35.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.885+0000 7f7deffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:35.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.885+0000 7f7deffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:35.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.885+0000 7f7deffff640 10 monclient(hunting): my global_id is 264319 2024-09-16T14:56:35.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.885+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ddc006590 con 0x7f7ddc0090c0 2024-09-16T14:56:35.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.885+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1826571123 0 0) 0x7f7ddc006590 con 0x7f7ddc0090c0 2024-09-16T14:56:35.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.885+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ddc009da0 con 0x7f7ddc0090c0 2024-09-16T14:56:35.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.885+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7de8004f10 con 0x7f7ddc0090c0 2024-09-16T14:56:35.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.885+0000 7f7deffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:35.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.885+0000 7f7deffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:56:35.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.885+0000 7f7deffff640 10 monclient: dump: 2024-09-16T14:56:35.910 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:35.910 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:35.910 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.910 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:35.911 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.885+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 478103516 0 0) 0x7f7ddc009da0 con 0x7f7ddc0090c0 2024-09-16T14:56:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.889+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7ddc00c290 con 0x7f7ddc0090c0 2024-09-16T14:56:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.889+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 2611128089) 0x7f7de80062d0 con 0x7f7ddc0090c0 2024-09-16T14:56:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.889+0000 7f7deffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:56:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.889+0000 7f7deffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.889+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 >> v1:172.21.15.189:6789/0 conn(0x7f7ddc0090c0 legacy=0x7f7ddc009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.889+0000 7f7deffff640 1 -- 172.21.15.26:0/1116150493 >> v1:172.21.15.189:6789/0 conn(0x7f7ddc0090c0 legacy=0x7f7ddc009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:35.912 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:56:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:56:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:56:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:56:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 132, 2024-09-16T14:56:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:56:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:56:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:56:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:56:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:56:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:56:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:56:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:56:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:56:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:56:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:56:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:56:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:56:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:56:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:56:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:56:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:56:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:56:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:56:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:56:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:56:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:56:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:56:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:56:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:56:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:56:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:56:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:56:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:56:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:56:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:56:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:56:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:56:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:56:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:56:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:56:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:56:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:56:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:56:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:56:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:56:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:56:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:56:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:56:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:56:35.925 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:56:35.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.889+0000 7f7dfda5e640 1 -- 172.21.15.26:0/1116150493 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7dd8036530 legacy=0x7f7dd80389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.889+0000 7f7dfda5e640 10 monclient: shutdown 2024-09-16T14:56:35.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.889+0000 7f7dfda5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:35.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.889+0000 7f7dfda5e640 1 -- 172.21.15.26:0/1116150493 >> v1:172.21.15.26:6789/0 conn(0x7f7df8132650 legacy=0x7f7df813a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:35.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.889+0000 7f7df77fe640 1 -- 172.21.15.26:0/1116150493 reap_dead start 2024-09-16T14:56:35.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.889+0000 7f7df77fe640 1 -- 172.21.15.26:0/1116150493 reap_dead start 2024-09-16T14:56:35.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.889+0000 7f7dfda5e640 1 -- 172.21.15.26:0/1116150493 shutdown_connections 2024-09-16T14:56:35.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.889+0000 7f7dfda5e640 1 -- 172.21.15.26:0/1116150493 >> 172.21.15.26:0/1116150493 conn(0x7f7df809fb50 msgr2=0x7f7df81388c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:35.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.889+0000 7f7dfda5e640 1 -- 172.21.15.26:0/1116150493 shutdown_connections 2024-09-16T14:56:35.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:35.889+0000 7f7dfda5e640 1 -- 172.21.15.26:0/1116150493 wait complete. 2024-09-16T14:56:35.927 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:56:35.927 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:56:35.927 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:56:36.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 10 monclient: get_monmap_and_config 2024-09-16T14:56:36.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 10 monclient: build_initial_monmap 2024-09-16T14:56:36.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 10 monclient: monmap: 2024-09-16T14:56:36.052 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:36.053 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:36.053 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:36.054464+0000 2024-09-16T14:56:36.053 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:36.054464+0000 2024-09-16T14:56:36.053 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:36.053 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:36.053 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:36.053 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:36.053 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:36.053 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:36.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 1 Processor -- start 2024-09-16T14:56:36.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 1 -- start start 2024-09-16T14:56:36.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 10 monclient: init 2024-09-16T14:56:36.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:36.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:56:36.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 10 monclient(hunting): picked mon.noname-a con 0x7f431c134e40 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:36.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 10 monclient(hunting): picked mon.noname-c con 0x7f431c099a70 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:36.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 10 monclient(hunting): picked mon.noname-b con 0x7f431c135220 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:36.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431c099e80 con 0x7f431c134e40 2024-09-16T14:56:36.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431c0719d0 con 0x7f431c099a70 2024-09-16T14:56:36.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431c09e1e0 con 0x7f431c135220 2024-09-16T14:56:36.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:36.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 10 monclient(hunting): authenticate will time out at 1820.174494s 2024-09-16T14:56:36.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f431b59e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f431c135220 0x7f431c13db30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60958/0 (socket says 172.21.15.26:60958) 2024-09-16T14:56:36.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f431ad9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f431c134e40 0x7f431c070bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:55246/0 (socket says 172.21.15.26:55246) 2024-09-16T14:56:36.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f431b59e640 1 -- 172.21.15.26:0/45811949 learned_addr learned my addr 172.21.15.26:0/45811949 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:36.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 1 -- 172.21.15.26:0/45811949 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382676919 0 0) 0x7f431c09e1e0 con 0x7f431c135220 2024-09-16T14:56:36.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:56:36.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:36.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:36.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient(hunting): my global_id is 284141 2024-09-16T14:56:36.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 1 -- 172.21.15.26:0/45811949 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4304003530 con 0x7f431c135220 2024-09-16T14:56:36.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 1 -- 172.21.15.26:0/45811949 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527794598 0 0) 0x7f4310001120 con 0x7f431c134e40 2024-09-16T14:56:36.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:56:36.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:36.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:36.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient(hunting): my global_id is 274245 2024-09-16T14:56:36.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 1 -- 172.21.15.26:0/45811949 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f431c09e1e0 con 0x7f431c134e40 2024-09-16T14:56:36.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 1 -- 172.21.15.26:0/45811949 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3308509150 0 0) 0x7f431c09e1e0 con 0x7f431c134e40 2024-09-16T14:56:36.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 1 -- 172.21.15.26:0/45811949 >> v1:172.21.15.26:6790/0 conn(0x7f431c135220 legacy=0x7f431c13db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:36.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 1 -- 172.21.15.26:0/45811949 >> v1:172.21.15.189:6789/0 conn(0x7f431c099a70 legacy=0x7f431c0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:36.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:56:36.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 1 monclient: found mon.noname-a 2024-09-16T14:56:36.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:36.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:56:36.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 1 -- 172.21.15.26:0/45811949 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c09e7d0 con 0x7f431c134e40 2024-09-16T14:56:36.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient: _finish_auth 0 2024-09-16T14:56:36.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:56:36.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:36.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 1 -- 172.21.15.26:0/45811949 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f43100015f0 con 0x7f431c134e40 2024-09-16T14:56:36.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 5 monclient: authenticate success, global_id 274245 2024-09-16T14:56:36.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:36.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:36.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:36.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient: dump: 2024-09-16T14:56:36.060 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:36.060 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:36.060 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:36.060 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:36.060 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:36.060 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:36.060 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:36.060 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:36.060 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:36.060 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:36.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:36.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 1 -- 172.21.15.26:0/45811949 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43100018a0 con 0x7f431c134e40 2024-09-16T14:56:36.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:36.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 1 -- 172.21.15.26:0/45811949 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4310001c70 con 0x7f431c134e40 2024-09-16T14:56:36.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:36.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:36.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4319d9b640 10 monclient: dump: 2024-09-16T14:56:36.061 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:36.061 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:36.061 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:36.061 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:36.061 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:36.062 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:36.062 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:36.062 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:36.062 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:36.062 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:36.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:36.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 4 set_mon_vals no callback set 2024-09-16T14:56:36.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 10 monclient: shutdown 2024-09-16T14:56:36.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:36.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.053+0000 7f4321145640 1 -- 172.21.15.26:0/45811949 >> v1:172.21.15.26:6789/0 conn(0x7f431c134e40 legacy=0x7f431c070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:36.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 1 -- 172.21.15.26:0/45811949 shutdown_connections 2024-09-16T14:56:36.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 1 -- 172.21.15.26:0/45811949 >> 172.21.15.26:0/45811949 conn(0x7f431c09fb50 msgr2=0x7f431c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:36.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 1 -- 172.21.15.26:0/45811949 shutdown_connections 2024-09-16T14:56:36.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 1 -- 172.21.15.26:0/45811949 wait complete. 2024-09-16T14:56:36.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 monclient: build_initial_monmap 2024-09-16T14:56:36.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 monclient: monmap: 2024-09-16T14:56:36.063 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:36.063 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:36.063 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:36.063 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:36.063 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:36.063 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:36.063 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:36.063 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:36.063 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:36.064 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:36.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 1 Processor -- start 2024-09-16T14:56:36.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 1 -- start start 2024-09-16T14:56:36.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 monclient: init 2024-09-16T14:56:36.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:36.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:56:36.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 monclient(hunting): picked mon.noname-c con 0x7f431c099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:36.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 monclient(hunting): picked mon.noname-a con 0x7f431c134e40 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:36.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 monclient(hunting): picked mon.noname-b con 0x7f431c135220 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:36.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431c121f10 con 0x7f431c134e40 2024-09-16T14:56:36.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431c121950 con 0x7f431c135220 2024-09-16T14:56:36.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431c121c30 con 0x7f431c099a70 2024-09-16T14:56:36.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:36.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 monclient(hunting): authenticate will time out at 1820.177533s 2024-09-16T14:56:36.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f431a59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f431c134e40 0x7f431c1ce910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:55250/0 (socket says 172.21.15.26:55250) 2024-09-16T14:56:36.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f431a59c640 1 -- 172.21.15.26:0/4141366837 learned_addr learned my addr 172.21.15.26:0/4141366837 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:36.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781011399 0 0) 0x7f431c121f10 con 0x7f431c134e40 2024-09-16T14:56:36.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): _negotiate 2024-09-16T14:56:36.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:36.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:36.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): my global_id is 274248 2024-09-16T14:56:36.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4308003530 con 0x7f431c134e40 2024-09-16T14:56:36.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002880650 0 0) 0x7f431c121c30 con 0x7f431c099a70 2024-09-16T14:56:36.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): _negotiate 2024-09-16T14:56:36.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:36.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:36.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): my global_id is 284144 2024-09-16T14:56:36.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f431c121f10 con 0x7f431c099a70 2024-09-16T14:56:36.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1020276254 0 0) 0x7f4308003530 con 0x7f431c134e40 2024-09-16T14:56:36.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f431c121c30 con 0x7f431c134e40 2024-09-16T14:56:36.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f430c001880 con 0x7f431c134e40 2024-09-16T14:56:36.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:36.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:36.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): dump: 2024-09-16T14:56:36.072 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:36.072 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:36.072 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:36.072 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:36.073 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:36.073 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:36.073 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:36.073 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:36.073 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:36.074 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:36.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:36.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741984704 0 0) 0x7f431c121950 con 0x7f431c135220 2024-09-16T14:56:36.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): _negotiate 2024-09-16T14:56:36.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:36.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:36.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): my global_id is 264331 2024-09-16T14:56:36.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4308003530 con 0x7f431c135220 2024-09-16T14:56:36.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1341357849 0 0) 0x7f431c121f10 con 0x7f431c099a70 2024-09-16T14:56:36.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f431c121950 con 0x7f431c099a70 2024-09-16T14:56:36.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4310000ff0 con 0x7f431c099a70 2024-09-16T14:56:36.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:36.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:36.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient(hunting): dump: 2024-09-16T14:56:36.076 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:36.076 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:36.076 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:36.077 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:36.077 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:36.077 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:36.077 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:36.077 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:36.077 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:36.078 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:36.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1769713729 0 0) 0x7f431c121c30 con 0x7f431c134e40 2024-09-16T14:56:36.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 >> v1:172.21.15.26:6790/0 conn(0x7f431c099a70 legacy=0x7f431c1c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:36.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 >> v1:172.21.15.189:6789/0 conn(0x7f431c135220 legacy=0x7f431c1d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:36.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient: _finish_hunting 0 2024-09-16T14:56:36.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 monclient: found mon.a 2024-09-16T14:56:36.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:36.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:36.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c120060 con 0x7f431c134e40 2024-09-16T14:56:36.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient: _finish_auth 0 2024-09-16T14:56:36.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient: _check_auth_tickets 2024-09-16T14:56:36.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:36.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 5 monclient: authenticate success, global_id 274248 2024-09-16T14:56:36.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 monclient: _renew_subs 2024-09-16T14:56:36.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:36.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 1 -- 172.21.15.26:0/4141366837 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431c11fb40 con 0x7f431c134e40 2024-09-16T14:56:36.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 monclient: _renew_subs 2024-09-16T14:56:36.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:36.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 1 -- 172.21.15.26:0/4141366837 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f431c1343f0 con 0x7f431c134e40 2024-09-16T14:56:36.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f430c001c60 con 0x7f431c134e40 2024-09-16T14:56:36.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:36.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f430c003a70 con 0x7f431c134e40 2024-09-16T14:56:36.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:36.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:36.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f431959a640 4 set_mon_vals no callback set 2024-09-16T14:56:36.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 10 monclient: dump: 2024-09-16T14:56:36.082 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:36.083 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:36.083 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:36.083 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:36.083 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:36.083 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:36.083 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:36.084 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:36.084 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:36.084 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:36.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:36.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:36.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:36.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f4321145640 1 -- 172.21.15.26:0/4141366837 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42fc005050 con 0x7f431c134e40 2024-09-16T14:56:36.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f430c003bd0 con 0x7f431c134e40 2024-09-16T14:56:36.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.057+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f430c043cf0 con 0x7f431c134e40 2024-09-16T14:56:36.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.061+0000 7f431ad9d640 10 monclient: get_auth_request con 0x7f4308036390 auth_method 0 2024-09-16T14:56:36.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.065+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f430c014650 con 0x7f431c134e40 2024-09-16T14:56:36.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.065+0000 7f43037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:36.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.065+0000 7f43037fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:36.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.289+0000 7f4321145640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:56:36.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.289+0000 7f4321145640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:56:36.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.289+0000 7f4321145640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:36.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.289+0000 7f4321145640 1 -- 172.21.15.26:0/4141366837 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f42fc005b10 con 0x7f431c134e40 2024-09-16T14:56:36.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f43037fe640 1 -- 172.21.15.26:0/4141366837 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f430c012ac0 con 0x7f431c134e40 2024-09-16T14:56:36.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f43037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:56:36.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f43037fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:36.292 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=11,kv=11,logm=47} crc {auth=1902741231,config=692619927,health=390019710,kv=1867599010,logm=474770789}) 2024-09-16T14:56:36.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=12,kv=10,logm=45} crc {auth=1738523301,config=692619927,health=1696088487,kv=2084167376,logm=3110147554}) 2024-09-16T14:56:36.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=11,kv=11,logm=47} crc {auth=1902741231,config=692619927,health=390019710,kv=1867599010,logm=474770789}) 2024-09-16T14:56:36.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=1,health=12,kv=10,logm=47} crc {auth=818332936,config=1228700967,health=1696088487,kv=3524478122,logm=4138829886}) 2024-09-16T14:56:36.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4230377579}) 2024-09-16T14:56:36.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f4321145640 1 -- 172.21.15.26:0/4141366837 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4308036390 legacy=0x7f4308038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:36.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4207178001}) 2024-09-16T14:56:36.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f4321145640 10 monclient: shutdown 2024-09-16T14:56:36.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f4321145640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:36.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f4321145640 1 -- 172.21.15.26:0/4141366837 >> v1:172.21.15.26:6789/0 conn(0x7f431c134e40 legacy=0x7f431c1ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:36.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f4321145640 1 -- 172.21.15.26:0/4141366837 shutdown_connections 2024-09-16T14:56:36.296 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4230377579}) 2024-09-16T14:56:36.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f4321145640 1 -- 172.21.15.26:0/4141366837 >> 172.21.15.26:0/4141366837 conn(0x7f431c09fb50 msgr2=0x7f431c132130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:36.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2142884009}) 2024-09-16T14:56:36.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f4321145640 1 -- 172.21.15.26:0/4141366837 shutdown_connections 2024-09-16T14:56:36.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:36.293+0000 7f4321145640 1 -- 172.21.15.26:0/4141366837 wait complete. 2024-09-16T14:56:36.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.297+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.297+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3846251207}) 2024-09-16T14:56:36.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.297+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3547162666}) 2024-09-16T14:56:36.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.297+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.297+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3846251207}) 2024-09-16T14:56:36.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.297+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1610716958}) 2024-09-16T14:56:36.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.301+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.301+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1592066757}) 2024-09-16T14:56:36.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.301+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165507027}) 2024-09-16T14:56:36.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.301+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.301+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1592066757}) 2024-09-16T14:56:36.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.301+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3469470004}) 2024-09-16T14:56:36.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.301+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.302 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.301+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2045028323}) 2024-09-16T14:56:36.302 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.301+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2015249364}) 2024-09-16T14:56:36.302 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.301+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.302 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.301+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2045028323}) 2024-09-16T14:56:36.302 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.301+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3094650883}) 2024-09-16T14:56:36.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=42602684}) 2024-09-16T14:56:36.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=327025243}) 2024-09-16T14:56:36.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=42602684}) 2024-09-16T14:56:36.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2993600959}) 2024-09-16T14:56:36.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4020786031}) 2024-09-16T14:56:36.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3239032174}) 2024-09-16T14:56:36.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4020786031}) 2024-09-16T14:56:36.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2010086214}) 2024-09-16T14:56:36.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=774803499}) 2024-09-16T14:56:36.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=789783795}) 2024-09-16T14:56:36.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=774803499}) 2024-09-16T14:56:36.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.305+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3376318379}) 2024-09-16T14:56:36.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.309+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.309+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1979897361}) 2024-09-16T14:56:36.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.309+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2305153613}) 2024-09-16T14:56:36.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.309+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.309+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1979897361}) 2024-09-16T14:56:36.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.309+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=621815832}) 2024-09-16T14:56:36.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.309+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.309+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3239510311}) 2024-09-16T14:56:36.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.309+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=538830523}) 2024-09-16T14:56:36.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.309+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.309+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3239510311}) 2024-09-16T14:56:36.312 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.309+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=962058413}) 2024-09-16T14:56:36.312 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:56:36.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.313+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.313+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=925041920}) 2024-09-16T14:56:36.314 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.313+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3253538338}) 2024-09-16T14:56:36.314 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.313+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.314 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.313+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=925041920}) 2024-09-16T14:56:36.314 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.313+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=190386400}) 2024-09-16T14:56:36.315 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.317+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.315 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.317+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=208945007}) 2024-09-16T14:56:36.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.317+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=723341766}) 2024-09-16T14:56:36.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.317+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.317+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=208945007}) 2024-09-16T14:56:36.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.317+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=484133184}) 2024-09-16T14:56:36.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.317+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.317+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1999602531}) 2024-09-16T14:56:36.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.317+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736934781}) 2024-09-16T14:56:36.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.317+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.317+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1999602531}) 2024-09-16T14:56:36.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.317+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3071243207}) 2024-09-16T14:56:36.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.321+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.321+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1645627135}) 2024-09-16T14:56:36.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.321+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259242742}) 2024-09-16T14:56:36.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.321+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.321+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1645627135}) 2024-09-16T14:56:36.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.321+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4079361370}) 2024-09-16T14:56:36.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.321+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.321+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3491303540}) 2024-09-16T14:56:36.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.321+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2969852017}) 2024-09-16T14:56:36.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.321+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.321+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3491303540}) 2024-09-16T14:56:36.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.321+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2374840355}) 2024-09-16T14:56:36.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.325+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.325+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2953666695}) 2024-09-16T14:56:36.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.325+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=163044419}) 2024-09-16T14:56:36.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.325+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.325+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2953666695}) 2024-09-16T14:56:36.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.325+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3213455220}) 2024-09-16T14:56:36.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.325+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.325+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2374564781}) 2024-09-16T14:56:36.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.325+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3506049062}) 2024-09-16T14:56:36.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.325+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.325+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2374564781}) 2024-09-16T14:56:36.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.325+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1588295238}) 2024-09-16T14:56:36.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.329+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.329+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=963611741}) 2024-09-16T14:56:36.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.329+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3232984370}) 2024-09-16T14:56:36.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.329+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.329+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=963611741}) 2024-09-16T14:56:36.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.329+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4227516027}) 2024-09-16T14:56:36.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.329+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.329+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1673414797}) 2024-09-16T14:56:36.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.329+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3237151299}) 2024-09-16T14:56:36.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.329+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.329+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1673414797}) 2024-09-16T14:56:36.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.329+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2262488092}) 2024-09-16T14:56:36.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.333+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.333+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=160002892}) 2024-09-16T14:56:36.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.333+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2921781856}) 2024-09-16T14:56:36.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.333+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.333+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=160002892}) 2024-09-16T14:56:36.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.333+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1643440956}) 2024-09-16T14:56:36.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.333+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.333+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1758586990}) 2024-09-16T14:56:36.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.337+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2857246506}) 2024-09-16T14:56:36.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.337+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.337+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1758586990}) 2024-09-16T14:56:36.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.337+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4076254194}) 2024-09-16T14:56:36.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.337+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.337+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=179357414}) 2024-09-16T14:56:36.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.337+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3449927650}) 2024-09-16T14:56:36.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.337+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.337+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=179357414}) 2024-09-16T14:56:36.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.337+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=501410068}) 2024-09-16T14:56:36.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.341+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.341+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2201835703}) 2024-09-16T14:56:36.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.341+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52925551}) 2024-09-16T14:56:36.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.341+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.341+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2201835703}) 2024-09-16T14:56:36.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.341+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1357976825}) 2024-09-16T14:56:36.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.341+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.341+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1292182510}) 2024-09-16T14:56:36.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.341+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1176305166}) 2024-09-16T14:56:36.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.341+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.341+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1292182510}) 2024-09-16T14:56:36.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.341+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1387033534}) 2024-09-16T14:56:36.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=831521586}) 2024-09-16T14:56:36.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=226027855}) 2024-09-16T14:56:36.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=831521586}) 2024-09-16T14:56:36.344 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=757662626}) 2024-09-16T14:56:36.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3025780167}) 2024-09-16T14:56:36.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624446941}) 2024-09-16T14:56:36.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3025780167}) 2024-09-16T14:56:36.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=219731404}) 2024-09-16T14:56:36.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3685461356}) 2024-09-16T14:56:36.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945290700}) 2024-09-16T14:56:36.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3685461356}) 2024-09-16T14:56:36.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.345+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3604784707}) 2024-09-16T14:56:36.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.349+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.349+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=659073015}) 2024-09-16T14:56:36.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.349+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1434440718}) 2024-09-16T14:56:36.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.349+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.349+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=659073015}) 2024-09-16T14:56:36.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.349+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3256983216}) 2024-09-16T14:56:36.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.349+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.349+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1257714259}) 2024-09-16T14:56:36.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.349+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3452387857}) 2024-09-16T14:56:36.351 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.349+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.351 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.349+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1257714259}) 2024-09-16T14:56:36.351 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.349+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=976008827}) 2024-09-16T14:56:36.352 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=0 2024-09-16T14:56:36.352 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T14:56:36.352 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:56:36.352 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls' 13 2024-09-16T14:56:36.352 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls' 2024-09-16T14:56:36.352 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T14:56:36.352 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-16T14:56:36.353 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-16T14:56:36.353 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-16T14:56:36.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.353+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.353+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3284054519}) 2024-09-16T14:56:36.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.353+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=747667668}) 2024-09-16T14:56:36.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.353+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.353+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3284054519}) 2024-09-16T14:56:36.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.353+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=568915702}) 2024-09-16T14:56:36.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.353+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.353+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3996025500}) 2024-09-16T14:56:36.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.353+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585070593}) 2024-09-16T14:56:36.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.353+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.353+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3996025500}) 2024-09-16T14:56:36.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.353+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1139114997}) 2024-09-16T14:56:36.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3703764172}) 2024-09-16T14:56:36.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1934151420}) 2024-09-16T14:56:36.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3703764172}) 2024-09-16T14:56:36.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3782805768}) 2024-09-16T14:56:36.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=586123496}) 2024-09-16T14:56:36.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4021351496}) 2024-09-16T14:56:36.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=586123496}) 2024-09-16T14:56:36.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1765143412}) 2024-09-16T14:56:36.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2114551425}) 2024-09-16T14:56:36.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1074697053}) 2024-09-16T14:56:36.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2114551425}) 2024-09-16T14:56:36.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.357+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1273845619}) 2024-09-16T14:56:36.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.361+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.361+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3918438811}) 2024-09-16T14:56:36.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.361+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4042851543}) 2024-09-16T14:56:36.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.361+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.361+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3918438811}) 2024-09-16T14:56:36.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.361+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3124663055}) 2024-09-16T14:56:36.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.361+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.361+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=681325054}) 2024-09-16T14:56:36.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.361+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758824821}) 2024-09-16T14:56:36.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.361+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.361+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=681325054}) 2024-09-16T14:56:36.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.361+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1271028683}) 2024-09-16T14:56:36.363 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4291104396}) 2024-09-16T14:56:36.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630174292}) 2024-09-16T14:56:36.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4291104396}) 2024-09-16T14:56:36.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=468280928}) 2024-09-16T14:56:36.365 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.365 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181444345}) 2024-09-16T14:56:36.365 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1239338701}) 2024-09-16T14:56:36.365 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.365 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181444345}) 2024-09-16T14:56:36.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1505834450}) 2024-09-16T14:56:36.367 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.367 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3650047405}) 2024-09-16T14:56:36.367 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4043057726}) 2024-09-16T14:56:36.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3650047405}) 2024-09-16T14:56:36.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.365+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262162862}) 2024-09-16T14:56:36.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.369+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.369+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3122705097}) 2024-09-16T14:56:36.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.369+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586653801}) 2024-09-16T14:56:36.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.369+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.369+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3122705097}) 2024-09-16T14:56:36.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.369+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=851800617}) 2024-09-16T14:56:36.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.369+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.369+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=253837168}) 2024-09-16T14:56:36.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.369+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313454113}) 2024-09-16T14:56:36.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.369+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.369+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=253837168}) 2024-09-16T14:56:36.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.369+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=23380762}) 2024-09-16T14:56:36.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.373+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.373+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3121303187}) 2024-09-16T14:56:36.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.373+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2073910662}) 2024-09-16T14:56:36.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.373+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.373+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3121303187}) 2024-09-16T14:56:36.374 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.373+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2137437827}) 2024-09-16T14:56:36.374 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.373+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.374 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.373+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2403781713}) 2024-09-16T14:56:36.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.373+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188681914}) 2024-09-16T14:56:36.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.373+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.373+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2403781713}) 2024-09-16T14:56:36.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.373+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=288528964}) 2024-09-16T14:56:36.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1182246042}) 2024-09-16T14:56:36.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372075355}) 2024-09-16T14:56:36.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1182246042}) 2024-09-16T14:56:36.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1157897698}) 2024-09-16T14:56:36.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1397507518}) 2024-09-16T14:56:36.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1334800594}) 2024-09-16T14:56:36.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1397507518}) 2024-09-16T14:56:36.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1755381647}) 2024-09-16T14:56:36.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=164087779}) 2024-09-16T14:56:36.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2072921559}) 2024-09-16T14:56:36.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=164087779}) 2024-09-16T14:56:36.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.377+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3036671920}) 2024-09-16T14:56:36.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.381+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.381+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4205336662}) 2024-09-16T14:56:36.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.381+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1827236718}) 2024-09-16T14:56:36.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.381+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.381+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4205336662}) 2024-09-16T14:56:36.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.381+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2251008646}) 2024-09-16T14:56:36.382 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.381+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.382 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.381+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1763504541}) 2024-09-16T14:56:36.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.381+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464921003}) 2024-09-16T14:56:36.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.381+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.381+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1763504541}) 2024-09-16T14:56:36.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.381+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1377433201}) 2024-09-16T14:56:36.384 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.385+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.384 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.385+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1193210333}) 2024-09-16T14:56:36.384 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.385+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019279141}) 2024-09-16T14:56:36.384 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.385+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.384 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.385+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1193210333}) 2024-09-16T14:56:36.384 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.385+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2542628188}) 2024-09-16T14:56:36.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.385+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.386 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.385+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=762460632}) 2024-09-16T14:56:36.386 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.385+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3909729545}) 2024-09-16T14:56:36.386 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.385+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.386 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.385+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=762460632}) 2024-09-16T14:56:36.386 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.385+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2937726142}) 2024-09-16T14:56:36.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.385+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.385+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=52746003}) 2024-09-16T14:56:36.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.385+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2258145372}) 2024-09-16T14:56:36.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.389+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.389+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=52746003}) 2024-09-16T14:56:36.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.389+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2896870920}) 2024-09-16T14:56:36.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.389+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.389+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3646114220}) 2024-09-16T14:56:36.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.389+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=232568272}) 2024-09-16T14:56:36.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.389+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.389+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3646114220}) 2024-09-16T14:56:36.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.389+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3060366218}) 2024-09-16T14:56:36.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.389+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.389+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=627113876}) 2024-09-16T14:56:36.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.389+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205259380}) 2024-09-16T14:56:36.391 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.389+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.391 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.389+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=627113876}) 2024-09-16T14:56:36.391 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.389+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3770970443}) 2024-09-16T14:56:36.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.393+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.393+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2532664690}) 2024-09-16T14:56:36.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.393+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2100612250}) 2024-09-16T14:56:36.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.393+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.393+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2532664690}) 2024-09-16T14:56:36.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.393+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4220571882}) 2024-09-16T14:56:36.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.393+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.393+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2365260855}) 2024-09-16T14:56:36.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.393+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575718900}) 2024-09-16T14:56:36.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.393+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.393+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2365260855}) 2024-09-16T14:56:36.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.393+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3880186285}) 2024-09-16T14:56:36.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=122960705}) 2024-09-16T14:56:36.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2097824467}) 2024-09-16T14:56:36.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=122960705}) 2024-09-16T14:56:36.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2510930791}) 2024-09-16T14:56:36.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1171648640}) 2024-09-16T14:56:36.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1553669241}) 2024-09-16T14:56:36.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1171648640}) 2024-09-16T14:56:36.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3653573224}) 2024-09-16T14:56:36.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3967855494}) 2024-09-16T14:56:36.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2100842450}) 2024-09-16T14:56:36.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3967855494}) 2024-09-16T14:56:36.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.397+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2644187062}) 2024-09-16T14:56:36.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.401+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.401+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4004424871}) 2024-09-16T14:56:36.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.401+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=126458277}) 2024-09-16T14:56:36.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.401+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.401+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4004424871}) 2024-09-16T14:56:36.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.401+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833197522}) 2024-09-16T14:56:36.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.401+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.401+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3282584361}) 2024-09-16T14:56:36.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.401+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1083777858}) 2024-09-16T14:56:36.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.401+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.401+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3282584361}) 2024-09-16T14:56:36.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.401+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2612927921}) 2024-09-16T14:56:36.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3225472742}) 2024-09-16T14:56:36.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713736347}) 2024-09-16T14:56:36.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3225472742}) 2024-09-16T14:56:36.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=325863080}) 2024-09-16T14:56:36.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=40109204}) 2024-09-16T14:56:36.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=724345628}) 2024-09-16T14:56:36.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=40109204}) 2024-09-16T14:56:36.405 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1187360500}) 2024-09-16T14:56:36.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=712686385}) 2024-09-16T14:56:36.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010603797}) 2024-09-16T14:56:36.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=712686385}) 2024-09-16T14:56:36.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.405+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2452901064}) 2024-09-16T14:56:36.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.409+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.409+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=501336479}) 2024-09-16T14:56:36.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.409+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892194156}) 2024-09-16T14:56:36.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.409+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.409+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=501336479}) 2024-09-16T14:56:36.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.409+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3324605703}) 2024-09-16T14:56:36.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.409+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.409+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2281776476}) 2024-09-16T14:56:36.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.409+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=85403447}) 2024-09-16T14:56:36.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.409+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.409+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2281776476}) 2024-09-16T14:56:36.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.409+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1885217756}) 2024-09-16T14:56:36.411 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.411 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4150708311}) 2024-09-16T14:56:36.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3411554865}) 2024-09-16T14:56:36.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4150708311}) 2024-09-16T14:56:36.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3668277249}) 2024-09-16T14:56:36.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2598630946}) 2024-09-16T14:56:36.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=787768562}) 2024-09-16T14:56:36.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2598630946}) 2024-09-16T14:56:36.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=134585058}) 2024-09-16T14:56:36.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3942746551}) 2024-09-16T14:56:36.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2015583934}) 2024-09-16T14:56:36.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3942746551}) 2024-09-16T14:56:36.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.413+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3371144676}) 2024-09-16T14:56:36.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.417+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.417+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2884268288}) 2024-09-16T14:56:36.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.417+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1325814780}) 2024-09-16T14:56:36.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.417+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.417+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2884268288}) 2024-09-16T14:56:36.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.417+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1169011670}) 2024-09-16T14:56:36.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.417+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.417+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1209558496}) 2024-09-16T14:56:36.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.417+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3943693138}) 2024-09-16T14:56:36.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.417+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.417+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1209558496}) 2024-09-16T14:56:36.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.417+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2344153265}) 2024-09-16T14:56:36.419 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1653140881}) 2024-09-16T14:56:36.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227744703}) 2024-09-16T14:56:36.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1653140881}) 2024-09-16T14:56:36.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=855175739}) 2024-09-16T14:56:36.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3936440346}) 2024-09-16T14:56:36.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589889199}) 2024-09-16T14:56:36.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3936440346}) 2024-09-16T14:56:36.425 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3526168964}) 2024-09-16T14:56:36.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2596338914}) 2024-09-16T14:56:36.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3319392515}) 2024-09-16T14:56:36.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2596338914}) 2024-09-16T14:56:36.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.421+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=634600914}) 2024-09-16T14:56:36.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.425+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.425+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=624029468}) 2024-09-16T14:56:36.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.425+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1252598519}) 2024-09-16T14:56:36.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.425+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.425+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=624029468}) 2024-09-16T14:56:36.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.425+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1303494672}) 2024-09-16T14:56:36.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.425+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.425+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2786978834}) 2024-09-16T14:56:36.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.425+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1976783962}) 2024-09-16T14:56:36.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.425+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.425+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2786978834}) 2024-09-16T14:56:36.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.425+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2823915135}) 2024-09-16T14:56:36.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1636505794}) 2024-09-16T14:56:36.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1499042698}) 2024-09-16T14:56:36.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1636505794}) 2024-09-16T14:56:36.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3506602853}) 2024-09-16T14:56:36.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1333172701}) 2024-09-16T14:56:36.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=962814367}) 2024-09-16T14:56:36.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1333172701}) 2024-09-16T14:56:36.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1854705431}) 2024-09-16T14:56:36.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2048734438}) 2024-09-16T14:56:36.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4212260671}) 2024-09-16T14:56:36.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2048734438}) 2024-09-16T14:56:36.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.429+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=486844848}) 2024-09-16T14:56:36.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.433+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.433+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=512071141}) 2024-09-16T14:56:36.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.433+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2985606775}) 2024-09-16T14:56:36.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.433+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.433+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=512071141}) 2024-09-16T14:56:36.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.433+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2919083938}) 2024-09-16T14:56:36.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.433+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.433+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1423729731}) 2024-09-16T14:56:36.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.433+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546165356}) 2024-09-16T14:56:36.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.433+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.433+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1423729731}) 2024-09-16T14:56:36.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.433+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=723020925}) 2024-09-16T14:56:36.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.436 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=187606173}) 2024-09-16T14:56:36.436 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2518593429}) 2024-09-16T14:56:36.436 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.436 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=187606173}) 2024-09-16T14:56:36.436 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2105261220}) 2024-09-16T14:56:36.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2957290848}) 2024-09-16T14:56:36.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2092272953}) 2024-09-16T14:56:36.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2957290848}) 2024-09-16T14:56:36.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1094090386}) 2024-09-16T14:56:36.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1251842823}) 2024-09-16T14:56:36.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1406931977}) 2024-09-16T14:56:36.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1251842823}) 2024-09-16T14:56:36.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.437+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2138727617}) 2024-09-16T14:56:36.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.441+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.441+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=304974753}) 2024-09-16T14:56:36.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.441+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940066879}) 2024-09-16T14:56:36.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.441+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.441+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=304974753}) 2024-09-16T14:56:36.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.441+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2721516773}) 2024-09-16T14:56:36.442 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.441+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.442 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.441+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=772787803}) 2024-09-16T14:56:36.442 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.441+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498998961}) 2024-09-16T14:56:36.442 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.441+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.442 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.441+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=772787803}) 2024-09-16T14:56:36.442 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.441+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2961852701}) 2024-09-16T14:56:36.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.445+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.445+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2951644853}) 2024-09-16T14:56:36.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.445+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906844509}) 2024-09-16T14:56:36.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.445+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.445+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2951644853}) 2024-09-16T14:56:36.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.445+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2300005036}) 2024-09-16T14:56:36.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.449+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.449+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=10,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=15} crc {logm=3923670741,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgrstat=1173820649,mon_config_key=1000370824,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3524828109}) 2024-09-16T14:56:36.448 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.449+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=43,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=12} crc {logm=1854320870,mdsmap=1531151339,mgr=2218402998,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2012764740,mon_config_key=1518680965,monmap=970648238,osd_metadata=1033174197,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=230863296}) 2024-09-16T14:56:36.448 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.449+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.448 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.449+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=10,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=15} crc {logm=3923670741,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgrstat=1173820649,mon_config_key=1000370824,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3524828109}) 2024-09-16T14:56:36.448 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.449+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=44,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=11} crc {logm=2304847315,mdsmap=1531151339,mgr=1843570494,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2002379764,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1440335178}) 2024-09-16T14:56:36.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.461+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.461+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=37} crc {osdmap=2306377759}) 2024-09-16T14:56:36.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.461+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=36} crc {osdmap=1567199168}) 2024-09-16T14:56:36.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.461+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:56:36.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.461+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=37} crc {osdmap=2306377759}) 2024-09-16T14:56:36.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:56:36.461+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=38} crc {osdmap=737939505}) 2024-09-16T14:56:36.671 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T14:56:36.672 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T14:56:36.672 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:56:36.672 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export' 13 2024-09-16T14:56:36.672 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-16T14:56:36.673 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export' 2024-09-16T14:56:36.673 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T14:56:36.673 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-16T14:56:36.673 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-16T14:56:36.993 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T14:56:36.993 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T14:56:36.993 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:56:36.994 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-16T14:56:36.994 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo' 13 2024-09-16T14:56:36.994 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo' 2024-09-16T14:56:36.994 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T14:56:36.994 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-16T14:56:36.994 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-16T14:56:37.313 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:56:37.314 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:56:37.316 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T14:56:37.316 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T14:56:37.316 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:56:37.317 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump' 13 2024-09-16T14:56:37.317 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-16T14:56:37.317 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump' 2024-09-16T14:56:37.317 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T14:56:37.317 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-16T14:56:37.317 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-16T14:56:37.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.445+0000 7f7569e62640 10 monclient: get_monmap_and_config 2024-09-16T14:56:37.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.445+0000 7f7569e62640 10 monclient: build_initial_monmap 2024-09-16T14:56:37.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.445+0000 7f7569e62640 10 monclient: monmap: 2024-09-16T14:56:37.448 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:37.449 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:37.449 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:37.446996+0000 2024-09-16T14:56:37.449 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:37.446996+0000 2024-09-16T14:56:37.449 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:37.449 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:37.449 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:37.449 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:37.449 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:37.449 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.445+0000 7f7569e62640 1 Processor -- start 2024-09-16T14:56:37.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.445+0000 7f7569e62640 1 -- start start 2024-09-16T14:56:37.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.445+0000 7f7569e62640 10 monclient: init 2024-09-16T14:56:37.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.445+0000 7f7569e62640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:37.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.445+0000 7f7569e62640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:56:37.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.445+0000 7f7569e62640 10 monclient(hunting): picked mon.noname-b con 0x7f75640a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:37.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.445+0000 7f7569e62640 10 monclient(hunting): picked mon.noname-a con 0x7f75640a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:37.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.445+0000 7f7569e62640 10 monclient(hunting): picked mon.noname-c con 0x7f75640a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:37.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.445+0000 7f7569e62640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75640a3250 con 0x7f75640a0000 2024-09-16T14:56:37.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.445+0000 7f7569e62640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756409f600 con 0x7f75640a2800 2024-09-16T14:56:37.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.445+0000 7f7569e62640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75640a3830 con 0x7f75640a1d10 2024-09-16T14:56:37.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.445+0000 7f7569e62640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:37.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.445+0000 7f7569e62640 10 monclient(hunting): authenticate will time out at 1821.567802s 2024-09-16T14:56:37.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75637fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f75640a1d10 0x7f75640a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48078/0 (socket says 172.21.15.26:48078) 2024-09-16T14:56:37.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75637fe640 1 -- 172.21.15.26:0/2131664710 learned_addr learned my addr 172.21.15.26:0/2131664710 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:37.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7562ffd640 1 --1- 172.21.15.26:0/2131664710 >> v1:172.21.15.26:6789/0 conn(0x7f75640a0000 0x7f7564139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36176/0 (socket says 172.21.15.26:36176) 2024-09-16T14:56:37.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 1 -- 172.21.15.26:0/2131664710 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232645900 0 0) 0x7f75640a3250 con 0x7f75640a0000 2024-09-16T14:56:37.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:37.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:37.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:37.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient(hunting): my global_id is 274275 2024-09-16T14:56:37.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 1 -- 172.21.15.26:0/2131664710 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f754c003530 con 0x7f75640a0000 2024-09-16T14:56:37.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 1 -- 172.21.15.26:0/2131664710 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495705923 0 0) 0x7f75640a3830 con 0x7f75640a1d10 2024-09-16T14:56:37.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:37.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:37.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:37.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient(hunting): my global_id is 284171 2024-09-16T14:56:37.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 1 -- 172.21.15.26:0/2131664710 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75640a3250 con 0x7f75640a1d10 2024-09-16T14:56:37.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 1 -- 172.21.15.26:0/2131664710 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255276281 0 0) 0x7f756409f600 con 0x7f75640a2800 2024-09-16T14:56:37.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:37.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:37.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:37.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient(hunting): my global_id is 264355 2024-09-16T14:56:37.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 1 -- 172.21.15.26:0/2131664710 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75640a3830 con 0x7f75640a2800 2024-09-16T14:56:37.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 1 -- 172.21.15.26:0/2131664710 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 200222317 0 0) 0x7f754c003530 con 0x7f75640a0000 2024-09-16T14:56:37.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 1 -- 172.21.15.26:0/2131664710 >> v1:172.21.15.26:6790/0 conn(0x7f75640a1d10 legacy=0x7f75640a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 1 -- 172.21.15.26:0/2131664710 >> v1:172.21.15.189:6789/0 conn(0x7f75640a2800 legacy=0x7f756413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient: _finish_hunting 0 2024-09-16T14:56:37.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 1 monclient: found mon.noname-a 2024-09-16T14:56:37.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:37.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:56:37.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 1 -- 172.21.15.26:0/2131664710 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756409ed60 con 0x7f75640a0000 2024-09-16T14:56:37.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient: _finish_auth 0 2024-09-16T14:56:37.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient: _check_auth_tickets 2024-09-16T14:56:37.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:37.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 1 -- 172.21.15.26:0/2131664710 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7558001620 con 0x7f75640a0000 2024-09-16T14:56:37.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 5 monclient: authenticate success, global_id 274275 2024-09-16T14:56:37.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:37.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:37.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:37.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient: dump: 2024-09-16T14:56:37.465 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:37.466 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:37.466 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.466 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.466 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:37.466 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:37.467 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:37.467 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:37.467 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:37.467 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:37.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 1 -- 172.21.15.26:0/2131664710 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7558001900 con 0x7f75640a0000 2024-09-16T14:56:37.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:37.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 1 -- 172.21.15.26:0/2131664710 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7558001c90 con 0x7f75640a0000 2024-09-16T14:56:37.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:37.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 4 set_mon_vals no callback set 2024-09-16T14:56:37.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:37.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:37.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f75627fc640 10 monclient: dump: 2024-09-16T14:56:37.468 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:37.468 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:37.468 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.468 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.468 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:37.468 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:37.468 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:37.468 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:37.469 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:37.469 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 10 monclient: shutdown 2024-09-16T14:56:37.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:37.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 1 -- 172.21.15.26:0/2131664710 >> v1:172.21.15.26:6789/0 conn(0x7f75640a0000 legacy=0x7f7564139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 1 -- 172.21.15.26:0/2131664710 shutdown_connections 2024-09-16T14:56:37.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 1 -- 172.21.15.26:0/2131664710 >> 172.21.15.26:0/2131664710 conn(0x7f75640a7f30 msgr2=0x7f75640a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:37.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 1 -- 172.21.15.26:0/2131664710 shutdown_connections 2024-09-16T14:56:37.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 1 -- 172.21.15.26:0/2131664710 wait complete. 2024-09-16T14:56:37.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 10 monclient: build_initial_monmap 2024-09-16T14:56:37.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 10 monclient: monmap: 2024-09-16T14:56:37.470 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:37.470 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:37.470 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:37.470 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:37.470 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:37.470 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:37.470 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:37.470 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:37.470 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:37.471 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 1 Processor -- start 2024-09-16T14:56:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 1 -- start start 2024-09-16T14:56:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 10 monclient: init 2024-09-16T14:56:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:56:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 10 monclient(hunting): picked mon.noname-b con 0x7f75640a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 10 monclient(hunting): picked mon.noname-a con 0x7f75640a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 10 monclient(hunting): picked mon.noname-c con 0x7f75640a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7564134b90 con 0x7f75640a1d10 2024-09-16T14:56:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7564133980 con 0x7f75640a0000 2024-09-16T14:56:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7564131230 con 0x7f75640a2800 2024-09-16T14:56:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:37.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 10 monclient(hunting): authenticate will time out at 1821.570944s 2024-09-16T14:56:37.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7562ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f75640a1d10 0x7f7564130b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36178/0 (socket says 172.21.15.26:36178) 2024-09-16T14:56:37.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7562ffd640 1 -- 172.21.15.26:0/3812370259 learned_addr learned my addr 172.21.15.26:0/3812370259 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:37.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480402402 0 0) 0x7f7564134b90 con 0x7f75640a1d10 2024-09-16T14:56:37.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:37.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:37.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:37.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): my global_id is 274278 2024-09-16T14:56:37.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7550003530 con 0x7f75640a1d10 2024-09-16T14:56:37.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173519268 0 0) 0x7f7564131230 con 0x7f75640a2800 2024-09-16T14:56:37.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:37.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:37.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:37.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): my global_id is 284174 2024-09-16T14:56:37.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7564134b90 con 0x7f75640a2800 2024-09-16T14:56:37.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822565261 0 0) 0x7f7564133980 con 0x7f75640a0000 2024-09-16T14:56:37.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:37.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:37.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:37.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): my global_id is 264358 2024-09-16T14:56:37.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7564131230 con 0x7f75640a0000 2024-09-16T14:56:37.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2429699349 0 0) 0x7f7550003530 con 0x7f75640a1d10 2024-09-16T14:56:37.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7564133980 con 0x7f75640a1d10 2024-09-16T14:56:37.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7558001660 con 0x7f75640a1d10 2024-09-16T14:56:37.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:37.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:37.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): dump: 2024-09-16T14:56:37.474 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:37.474 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:37.474 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.474 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.475 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:37.475 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:37.475 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:37.475 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:37.475 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:37.475 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:37.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2542788372 0 0) 0x7f7564134b90 con 0x7f75640a2800 2024-09-16T14:56:37.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7550003530 con 0x7f75640a2800 2024-09-16T14:56:37.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f755c001c40 con 0x7f75640a2800 2024-09-16T14:56:37.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:37.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:37.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient(hunting): dump: 2024-09-16T14:56:37.476 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:37.476 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:37.476 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.476 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.476 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:37.476 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:37.477 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:37.477 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:37.477 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:37.477 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3204305363 0 0) 0x7f7564133980 con 0x7f75640a1d10 2024-09-16T14:56:37.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 >> v1:172.21.15.26:6790/0 conn(0x7f75640a2800 legacy=0x7f75641d1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 >> v1:172.21.15.189:6789/0 conn(0x7f75640a0000 legacy=0x7f7564130410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:56:37.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 monclient: found mon.a 2024-09-16T14:56:37.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:37.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:37.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7564131db0 con 0x7f75640a1d10 2024-09-16T14:56:37.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient: _finish_auth 0 2024-09-16T14:56:37.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:56:37.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f754bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:37.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 5 monclient: authenticate success, global_id 274278 2024-09-16T14:56:37.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 10 monclient: _renew_subs 2024-09-16T14:56:37.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:37.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 1 -- 172.21.15.26:0/3812370259 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7564133bd0 con 0x7f75640a1d10 2024-09-16T14:56:37.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 10 monclient: _renew_subs 2024-09-16T14:56:37.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:37.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.449+0000 7f7569e62640 1 -- 172.21.15.26:0/3812370259 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75641342d0 con 0x7f75640a1d10 2024-09-16T14:56:37.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.453+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75580011c0 con 0x7f75640a1d10 2024-09-16T14:56:37.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.453+0000 7f754bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:37.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.453+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7558004d00 con 0x7f75640a1d10 2024-09-16T14:56:37.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.453+0000 7f754bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:37.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.453+0000 7f754bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:37.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.453+0000 7f7561ffb640 4 set_mon_vals no callback set 2024-09-16T14:56:37.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.453+0000 7f754bfff640 10 monclient: dump: 2024-09-16T14:56:37.479 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:37.479 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:37.479 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.479 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.479 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:37.479 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:37.480 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:37.480 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:37.480 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:37.480 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.453+0000 7f7569e62640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:37.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.453+0000 7f7569e62640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:37.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.453+0000 7f7569e62640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:37.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.453+0000 7f7569e62640 1 -- 172.21.15.26:0/3812370259 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7544005050 con 0x7f75640a1d10 2024-09-16T14:56:37.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.453+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7558004e60 con 0x7f75640a1d10 2024-09-16T14:56:37.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.453+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f7558043f50 con 0x7f75640a1d10 2024-09-16T14:56:37.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.453+0000 7f75637fe640 10 monclient: get_auth_request con 0x7f7550036390 auth_method 0 2024-09-16T14:56:37.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.457+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7558006400 con 0x7f75640a1d10 2024-09-16T14:56:37.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.457+0000 7f754bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:37.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.457+0000 7f754bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:37.589 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T14:56:37.589 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T14:56:37.589 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T14:56:37.589 INFO:tasks.workunit.client.0.smithi189.stderr:+ export CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-16T14:56:37.589 INFO:tasks.workunit.client.0.smithi189.stderr:+ CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-16T14:56:37.589 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump' 110 2024-09-16T14:56:37.589 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump' 2024-09-16T14:56:37.589 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=110 2024-09-16T14:56:37.589 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-16T14:56:37.590 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-16T14:56:37.590 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-16T14:56:37.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.677+0000 7f7569e62640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:56:37.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.677+0000 7f7569e62640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:56:37.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.677+0000 7f7569e62640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:37.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.677+0000 7f7569e62640 1 -- 172.21.15.26:0/3812370259 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7544005880 con 0x7f75640a1d10 2024-09-16T14:56:37.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.677+0000 7f754bfff640 1 -- 172.21.15.26:0/3812370259 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2958878721) 0x7f7558014540 con 0x7f75640a1d10 2024-09-16T14:56:37.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.677+0000 7f754bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:56:37.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.677+0000 7f754bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:37.677 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:56:37.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.677+0000 7f7569e62640 1 -- 172.21.15.26:0/3812370259 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7550036390 legacy=0x7f7550038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.677+0000 7f7569e62640 10 monclient: shutdown 2024-09-16T14:56:37.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.677+0000 7f7569e62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:37.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.677+0000 7f7569e62640 1 -- 172.21.15.26:0/3812370259 >> v1:172.21.15.26:6789/0 conn(0x7f75640a1d10 legacy=0x7f7564130b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.677+0000 7f7569e62640 1 -- 172.21.15.26:0/3812370259 shutdown_connections 2024-09-16T14:56:37.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.677+0000 7f7569e62640 1 -- 172.21.15.26:0/3812370259 >> 172.21.15.26:0/3812370259 conn(0x7f75640a7f30 msgr2=0x7f756406b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:37.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.677+0000 7f7569e62640 1 -- 172.21.15.26:0/3812370259 shutdown_connections 2024-09-16T14:56:37.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.677+0000 7f7569e62640 1 -- 172.21.15.26:0/3812370259 wait complete. 2024-09-16T14:56:37.695 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:56:37.696 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:56:37.696 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:56:37.696 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:56:37.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.821+0000 7f32bb04b640 10 monclient: get_monmap_and_config 2024-09-16T14:56:37.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.821+0000 7f32bb04b640 10 monclient: build_initial_monmap 2024-09-16T14:56:37.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.821+0000 7f32bb04b640 10 monclient: monmap: 2024-09-16T14:56:37.822 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:37.822 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:37.822 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:37.824063+0000 2024-09-16T14:56:37.822 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:37.824063+0000 2024-09-16T14:56:37.822 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:37.822 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:37.822 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:37.823 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:37.823 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:37.823 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.821+0000 7f32bb04b640 1 Processor -- start 2024-09-16T14:56:37.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.821+0000 7f32bb04b640 1 -- start start 2024-09-16T14:56:37.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 10 monclient: init 2024-09-16T14:56:37.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:37.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:56:37.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 10 monclient(hunting): picked mon.noname-a con 0x7f32b4096600 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:37.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 10 monclient(hunting): picked mon.noname-c con 0x7f32b409d580 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:37.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 10 monclient(hunting): picked mon.noname-b con 0x7f32b4137600 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:37.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b409bed0 con 0x7f32b4096600 2024-09-16T14:56:37.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b409cc00 con 0x7f32b409d580 2024-09-16T14:56:37.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b409f7d0 con 0x7f32b4137600 2024-09-16T14:56:37.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:37.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 10 monclient(hunting): authenticate will time out at 1821.944689s 2024-09-16T14:56:37.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b8de7640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f32b4096600 0x7f32b40969e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36190/0 (socket says 172.21.15.26:36190) 2024-09-16T14:56:37.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b8de7640 1 -- 172.21.15.26:0/3920096886 learned_addr learned my addr 172.21.15.26:0/3920096886 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:37.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b95e8640 1 --1- 172.21.15.26:0/3920096886 >> v1:172.21.15.26:6790/0 conn(0x7f32b4137600 0x7f32b41399f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48088/0 (socket says 172.21.15.26:48088) 2024-09-16T14:56:37.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 1 -- 172.21.15.26:0/3920096886 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089847301 0 0) 0x7f32b409bed0 con 0x7f32b4096600 2024-09-16T14:56:37.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:56:37.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:37.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:37.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient(hunting): my global_id is 274287 2024-09-16T14:56:37.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 1 -- 172.21.15.26:0/3920096886 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32a4003530 con 0x7f32b4096600 2024-09-16T14:56:37.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 1 -- 172.21.15.26:0/3920096886 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650994211 0 0) 0x7f32b409f7d0 con 0x7f32b4137600 2024-09-16T14:56:37.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:56:37.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:37.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:37.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient(hunting): my global_id is 284183 2024-09-16T14:56:37.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 1 -- 172.21.15.26:0/3920096886 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b409bed0 con 0x7f32b4137600 2024-09-16T14:56:37.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 1 -- 172.21.15.26:0/3920096886 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741898714 0 0) 0x7f32b409cc00 con 0x7f32b409d580 2024-09-16T14:56:37.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:56:37.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:37.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:37.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient(hunting): my global_id is 264367 2024-09-16T14:56:37.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 1 -- 172.21.15.26:0/3920096886 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b409f7d0 con 0x7f32b409d580 2024-09-16T14:56:37.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 1 -- 172.21.15.26:0/3920096886 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3892690599 0 0) 0x7f32b409bed0 con 0x7f32b4137600 2024-09-16T14:56:37.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 1 -- 172.21.15.26:0/3920096886 >> v1:172.21.15.189:6789/0 conn(0x7f32b409d580 legacy=0x7f32b4135ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 1 -- 172.21.15.26:0/3920096886 >> v1:172.21.15.26:6789/0 conn(0x7f32b4096600 legacy=0x7f32b40969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:56:37.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 1 monclient: found mon.noname-b 2024-09-16T14:56:37.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:37.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:56:37.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 1 -- 172.21.15.26:0/3920096886 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b409c2f0 con 0x7f32b4137600 2024-09-16T14:56:37.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient: _finish_auth 0 2024-09-16T14:56:37.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:56:37.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:37.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 1 -- 172.21.15.26:0/3920096886 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f32ac001980 con 0x7f32b4137600 2024-09-16T14:56:37.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:37.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:56:37.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient: dump: 2024-09-16T14:56:37.830 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:37.830 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:37.830 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.830 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.830 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:37.831 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:37.831 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:37.831 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:37.831 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:37.831 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 5 monclient: authenticate success, global_id 284183 2024-09-16T14:56:37.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:37.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 1 -- 172.21.15.26:0/3920096886 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32ac001ca0 con 0x7f32b4137600 2024-09-16T14:56:37.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:37.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 1 -- 172.21.15.26:0/3920096886 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f32ac002070 con 0x7f32b4137600 2024-09-16T14:56:37.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:37.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 4 set_mon_vals no callback set 2024-09-16T14:56:37.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:37.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:37.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32b37fe640 10 monclient: dump: 2024-09-16T14:56:37.832 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:37.832 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:37.832 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.832 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.832 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:37.832 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:37.832 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:37.832 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:37.832 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:37.833 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 10 monclient: shutdown 2024-09-16T14:56:37.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:37.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 1 -- 172.21.15.26:0/3920096886 >> v1:172.21.15.26:6790/0 conn(0x7f32b4137600 legacy=0x7f32b41399f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 1 -- 172.21.15.26:0/3920096886 shutdown_connections 2024-09-16T14:56:37.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 1 -- 172.21.15.26:0/3920096886 >> 172.21.15.26:0/3920096886 conn(0x7f32b40a0360 msgr2=0x7f32b40a0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:37.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 1 -- 172.21.15.26:0/3920096886 shutdown_connections 2024-09-16T14:56:37.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 1 -- 172.21.15.26:0/3920096886 wait complete. 2024-09-16T14:56:37.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 10 monclient: build_initial_monmap 2024-09-16T14:56:37.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 10 monclient: monmap: 2024-09-16T14:56:37.833 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:37.833 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:37.833 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:37.834 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:37.834 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:37.834 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:37.834 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:37.834 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:37.834 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:37.834 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 1 Processor -- start 2024-09-16T14:56:37.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 1 -- start start 2024-09-16T14:56:37.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.825+0000 7f32bb04b640 10 monclient: init 2024-09-16T14:56:37.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:37.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:56:37.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 10 monclient(hunting): picked mon.noname-a con 0x7f32b4096600 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:37.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 10 monclient(hunting): picked mon.noname-b con 0x7f32b409d580 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:37.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 10 monclient(hunting): picked mon.noname-c con 0x7f32b4137600 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:37.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b409a880 con 0x7f32b4096600 2024-09-16T14:56:37.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b413bb40 con 0x7f32b409d580 2024-09-16T14:56:37.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b413c010 con 0x7f32b4137600 2024-09-16T14:56:37.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:37.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 10 monclient(hunting): authenticate will time out at 1821.948449s 2024-09-16T14:56:37.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b8de7640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f32b4096600 0x7f32b4136850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36194/0 (socket says 172.21.15.26:36194) 2024-09-16T14:56:37.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b8de7640 1 -- 172.21.15.26:0/2823856125 learned_addr learned my addr 172.21.15.26:0/2823856125 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:37.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983251968 0 0) 0x7f32b409a880 con 0x7f32b4096600 2024-09-16T14:56:37.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:56:37.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:37.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:37.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): my global_id is 274290 2024-09-16T14:56:37.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3298003530 con 0x7f32b4096600 2024-09-16T14:56:37.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216155439 0 0) 0x7f32b413c010 con 0x7f32b4137600 2024-09-16T14:56:37.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:56:37.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:37.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:37.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): my global_id is 284186 2024-09-16T14:56:37.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b409a880 con 0x7f32b4137600 2024-09-16T14:56:37.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745308003 0 0) 0x7f32b413bb40 con 0x7f32b409d580 2024-09-16T14:56:37.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:56:37.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:37.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:37.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): my global_id is 264370 2024-09-16T14:56:37.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b413c010 con 0x7f32b409d580 2024-09-16T14:56:37.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2583398215 0 0) 0x7f32b409a880 con 0x7f32b4137600 2024-09-16T14:56:37.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32b413bb40 con 0x7f32b4137600 2024-09-16T14:56:37.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f32ac001fb0 con 0x7f32b4137600 2024-09-16T14:56:37.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:37.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:56:37.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): dump: 2024-09-16T14:56:37.843 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:37.843 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:37.843 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.843 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.843 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:37.843 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:37.843 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:37.843 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:37.844 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:37.844 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:37.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 395632242 0 0) 0x7f3298003530 con 0x7f32b4096600 2024-09-16T14:56:37.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32b409a880 con 0x7f32b4096600 2024-09-16T14:56:37.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f32a8001a00 con 0x7f32b4096600 2024-09-16T14:56:37.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:37.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:37.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): dump: 2024-09-16T14:56:37.844 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:37.844 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:37.844 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.845 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.845 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:37.845 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:37.845 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:37.845 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:37.845 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:37.845 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3584517457 0 0) 0x7f32b409a880 con 0x7f32b4096600 2024-09-16T14:56:37.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 >> v1:172.21.15.26:6790/0 conn(0x7f32b4137600 legacy=0x7f32b41cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 >> v1:172.21.15.189:6789/0 conn(0x7f32b409d580 legacy=0x7f32b41cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient: _finish_hunting 0 2024-09-16T14:56:37.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 monclient: found mon.a 2024-09-16T14:56:37.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:37.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:37.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b4131d50 con 0x7f32b4096600 2024-09-16T14:56:37.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient: _finish_auth 0 2024-09-16T14:56:37.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient: _check_auth_tickets 2024-09-16T14:56:37.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:37.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 5 monclient: authenticate success, global_id 274290 2024-09-16T14:56:37.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 10 monclient: _renew_subs 2024-09-16T14:56:37.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:37.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 1 -- 172.21.15.26:0/2823856125 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b4132010 con 0x7f32b4096600 2024-09-16T14:56:37.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 10 monclient: _renew_subs 2024-09-16T14:56:37.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:37.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 1 -- 172.21.15.26:0/2823856125 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b41c74c0 con 0x7f32b4096600 2024-09-16T14:56:37.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32a80047f0 con 0x7f32b4096600 2024-09-16T14:56:37.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:37.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f32a8004b00 con 0x7f32b4096600 2024-09-16T14:56:37.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:37.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:37.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b27fc640 4 set_mon_vals no callback set 2024-09-16T14:56:37.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient: dump: 2024-09-16T14:56:37.847 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:37.847 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:37.847 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.847 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.847 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:37.847 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:37.847 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:37.848 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:37.848 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:37.848 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f32a800f570 con 0x7f32b4096600 2024-09-16T14:56:37.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:37.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32bb04b640 1 -- 172.21.15.26:0/2823856125 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3294006590 con 0x7f3294002830 2024-09-16T14:56:37.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f32a8043de0 con 0x7f32b4096600 2024-09-16T14:56:37.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268042827 0 0) 0x7f3294006590 con 0x7f3294002830 2024-09-16T14:56:37.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:56:37.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:37.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:37.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient(hunting): my global_id is 274293 2024-09-16T14:56:37.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b409a880 con 0x7f3294002830 2024-09-16T14:56:37.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b95e8640 10 monclient: get_auth_request con 0x7f32980363c0 auth_method 0 2024-09-16T14:56:37.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2972575024 0 0) 0x7f32b409a880 con 0x7f3294002830 2024-09-16T14:56:37.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3294006590 con 0x7f3294002830 2024-09-16T14:56:37.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f32a0001860 con 0x7f3294002830 2024-09-16T14:56:37.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:37.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:37.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient: dump: 2024-09-16T14:56:37.849 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:37.849 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:37.850 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.850 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.850 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:37.850 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:37.850 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:37.850 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:37.850 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:37.850 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 406861059 0 0) 0x7f3294006590 con 0x7f3294002830 2024-09-16T14:56:37.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f329400a0e0 con 0x7f3294002830 2024-09-16T14:56:37.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f32a0001bb0 con 0x7f3294002830 2024-09-16T14:56:37.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:37.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:37.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 >> v1:172.21.15.26:6789/0 conn(0x7f3294002830 legacy=0x7f3294004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.829+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 >> v1:172.21.15.26:6789/0 conn(0x7f3294002830 legacy=0x7f3294004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:37.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32bb04b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:56:37.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32bb04b640 1 -- 172.21.15.26:0/2823856125 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3294009da0 con 0x7f32940090c0 2024-09-16T14:56:37.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2406966489 0 0) 0x7f3294009da0 con 0x7f32940090c0 2024-09-16T14:56:37.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:56:37.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:37.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:37.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 10 monclient(hunting): my global_id is 274296 2024-09-16T14:56:37.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3294006590 con 0x7f32940090c0 2024-09-16T14:56:37.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2845749684 0 0) 0x7f3294006590 con 0x7f32940090c0 2024-09-16T14:56:37.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3294009da0 con 0x7f32940090c0 2024-09-16T14:56:37.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f32a0003990 con 0x7f32940090c0 2024-09-16T14:56:37.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:37.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:37.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 10 monclient: dump: 2024-09-16T14:56:37.852 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:37.852 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:37.852 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.852 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:37.853 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:37.853 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:37.853 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:37.853 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:37.853 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:37.853 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2666232039 0 0) 0x7f3294009da0 con 0x7f32940090c0 2024-09-16T14:56:37.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f329400c290 con 0x7f32940090c0 2024-09-16T14:56:37.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 4222852495) 0x7f32a0003c20 con 0x7f32940090c0 2024-09-16T14:56:37.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:56:37.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:37.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 >> v1:172.21.15.26:6789/0 conn(0x7f32940090c0 legacy=0x7f3294009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b17fa640 1 -- 172.21.15.26:0/2823856125 >> v1:172.21.15.26:6789/0 conn(0x7f32940090c0 legacy=0x7f3294009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:37.857 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:56:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:56:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:56:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:56:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 132, 2024-09-16T14:56:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:56:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:56:37.857 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:56:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:56:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:56:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:56:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:56:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:37.858 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:56:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:56:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:37.859 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:56:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:56:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:56:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:56:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:56:37.860 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:56:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:56:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:56:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:56:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:56:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:56:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:56:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:56:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:56:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:37.861 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:56:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:56:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:37.862 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:56:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:56:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:56:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:56:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:56:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:37.863 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:56:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:56:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:56:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:37.864 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:56:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:56:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:56:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:56:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:37.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:56:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:56:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:56:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:37.866 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:56:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:56:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:56:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:37.867 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:56:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:56:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:56:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:56:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:37.868 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:37.869 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:56:37.869 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:56:37.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32bb04b640 1 -- 172.21.15.26:0/2823856125 >> v1:172.21.15.189:6812/3196161582 conn(0x7f32980363c0 legacy=0x7f3298038880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32bb04b640 10 monclient: shutdown 2024-09-16T14:56:37.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32bb04b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:37.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32bb04b640 1 -- 172.21.15.26:0/2823856125 >> v1:172.21.15.26:6789/0 conn(0x7f32b4096600 legacy=0x7f32b4136850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b95e8640 1 -- 172.21.15.26:0/2823856125 reap_dead start 2024-09-16T14:56:37.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32b95e8640 1 -- 172.21.15.26:0/2823856125 reap_dead start 2024-09-16T14:56:37.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32bb04b640 1 -- 172.21.15.26:0/2823856125 shutdown_connections 2024-09-16T14:56:37.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32bb04b640 1 -- 172.21.15.26:0/2823856125 >> 172.21.15.26:0/2823856125 conn(0x7f32b40a0360 msgr2=0x7f32b40949a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:37.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32bb04b640 1 -- 172.21.15.26:0/2823856125 shutdown_connections 2024-09-16T14:56:37.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.837+0000 7f32bb04b640 1 -- 172.21.15.26:0/2823856125 wait complete. 2024-09-16T14:56:37.871 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:56:37.871 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:56:37.872 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:56:37.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 10 monclient: get_monmap_and_config 2024-09-16T14:56:37.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 10 monclient: build_initial_monmap 2024-09-16T14:56:37.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 10 monclient: monmap: 2024-09-16T14:56:37.988 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:37.988 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:37.988 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:37.990082+0000 2024-09-16T14:56:37.988 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:37.990082+0000 2024-09-16T14:56:37.988 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:37.988 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:37.988 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:37.989 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:37.989 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:37.989 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:37.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 1 Processor -- start 2024-09-16T14:56:37.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 1 -- start start 2024-09-16T14:56:37.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 10 monclient: init 2024-09-16T14:56:37.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:37.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:56:37.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 10 monclient(hunting): picked mon.noname-c con 0x7fe7700a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:37.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 10 monclient(hunting): picked mon.noname-b con 0x7fe7700a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:37.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 10 monclient(hunting): picked mon.noname-a con 0x7fe7700a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:37.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7700a3250 con 0x7fe7700a2800 2024-09-16T14:56:37.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe77009f600 con 0x7fe7700a1d10 2024-09-16T14:56:37.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7700a3830 con 0x7fe7700a0000 2024-09-16T14:56:37.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:37.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 10 monclient(hunting): authenticate will time out at 1822.110824s 2024-09-16T14:56:37.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76effd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe7700a0000 0x7fe770139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48108/0 (socket says 172.21.15.26:48108) 2024-09-16T14:56:37.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76effd640 1 -- 172.21.15.26:0/17999509 learned_addr learned my addr 172.21.15.26:0/17999509 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:37.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 1 -- 172.21.15.26:0/17999509 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098454813 0 0) 0x7fe7700a3830 con 0x7fe7700a0000 2024-09-16T14:56:37.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:37.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:37.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:37.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient(hunting): my global_id is 284189 2024-09-16T14:56:37.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 1 -- 172.21.15.26:0/17999509 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe758003530 con 0x7fe7700a0000 2024-09-16T14:56:37.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 1 -- 172.21.15.26:0/17999509 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297703654 0 0) 0x7fe7700a3250 con 0x7fe7700a2800 2024-09-16T14:56:37.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:37.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:37.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:37.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient(hunting): my global_id is 274299 2024-09-16T14:56:37.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 1 -- 172.21.15.26:0/17999509 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7700a3830 con 0x7fe7700a2800 2024-09-16T14:56:37.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 1 -- 172.21.15.26:0/17999509 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4235062101 0 0) 0x7fe758003530 con 0x7fe7700a0000 2024-09-16T14:56:37.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 1 -- 172.21.15.26:0/17999509 >> v1:172.21.15.189:6789/0 conn(0x7fe7700a1d10 legacy=0x7fe7700a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 1 -- 172.21.15.26:0/17999509 >> v1:172.21.15.26:6789/0 conn(0x7fe7700a2800 legacy=0x7fe77013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:37.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:56:37.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 1 monclient: found mon.noname-b 2024-09-16T14:56:37.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:37.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:56:37.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 1 -- 172.21.15.26:0/17999509 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77009ed60 con 0x7fe7700a0000 2024-09-16T14:56:37.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:56:38.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:56:38.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:38.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 1 -- 172.21.15.26:0/17999509 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe764001620 con 0x7fe7700a0000 2024-09-16T14:56:38.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:56:38.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient: dump: 2024-09-16T14:56:38.000 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.000 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.000 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.000 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.000 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.000 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.000 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.001 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.001 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.001 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 5 monclient: authenticate success, global_id 284189 2024-09-16T14:56:38.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:38.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 1 -- 172.21.15.26:0/17999509 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe764001900 con 0x7fe7700a0000 2024-09-16T14:56:38.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:38.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 1 -- 172.21.15.26:0/17999509 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe764001cd0 con 0x7fe7700a0000 2024-09-16T14:56:38.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:38.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe76e7fc640 10 monclient: dump: 2024-09-16T14:56:38.001 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.001 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.002 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.989+0000 7fe775c31640 4 set_mon_vals no callback set 2024-09-16T14:56:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 10 monclient: shutdown 2024-09-16T14:56:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 1 -- 172.21.15.26:0/17999509 >> v1:172.21.15.26:6790/0 conn(0x7fe7700a0000 legacy=0x7fe770139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 1 -- 172.21.15.26:0/17999509 shutdown_connections 2024-09-16T14:56:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 1 -- 172.21.15.26:0/17999509 >> 172.21.15.26:0/17999509 conn(0x7fe7700a7f30 msgr2=0x7fe7700a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 1 -- 172.21.15.26:0/17999509 shutdown_connections 2024-09-16T14:56:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 1 -- 172.21.15.26:0/17999509 wait complete. 2024-09-16T14:56:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 10 monclient: build_initial_monmap 2024-09-16T14:56:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 10 monclient: monmap: 2024-09-16T14:56:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:38.004 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:38.004 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:38.004 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:38.004 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:38.004 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:38.004 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 1 Processor -- start 2024-09-16T14:56:38.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 1 -- start start 2024-09-16T14:56:38.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 10 monclient: init 2024-09-16T14:56:38.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:38.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:56:38.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 10 monclient(hunting): picked mon.noname-a con 0x7fe7700a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:38.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 10 monclient(hunting): picked mon.noname-b con 0x7fe7700a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:38.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 10 monclient(hunting): picked mon.noname-c con 0x7fe7700a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:38.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe77009e500 con 0x7fe7700a0000 2024-09-16T14:56:38.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7700a6770 con 0x7fe7700a1d10 2024-09-16T14:56:38.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe770134250 con 0x7fe7700a2800 2024-09-16T14:56:38.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:38.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 10 monclient(hunting): authenticate will time out at 1822.114014s 2024-09-16T14:56:38.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe76ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe7700a2800 0x7fe7701d18e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48120/0 (socket says 172.21.15.26:48120) 2024-09-16T14:56:38.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe76ffff640 1 -- 172.21.15.26:0/2617402238 learned_addr learned my addr 172.21.15.26:0/2617402238 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:38.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260350521 0 0) 0x7fe770134250 con 0x7fe7700a2800 2024-09-16T14:56:38.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:38.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:38.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): my global_id is 284192 2024-09-16T14:56:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe75c003530 con 0x7fe7700a2800 2024-09-16T14:56:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636126954 0 0) 0x7fe77009e500 con 0x7fe7700a0000 2024-09-16T14:56:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): my global_id is 274302 2024-09-16T14:56:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe770134250 con 0x7fe7700a0000 2024-09-16T14:56:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 604070772 0 0) 0x7fe770134250 con 0x7fe7700a0000 2024-09-16T14:56:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe77009e500 con 0x7fe7700a0000 2024-09-16T14:56:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe760001860 con 0x7fe7700a0000 2024-09-16T14:56:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:38.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): dump: 2024-09-16T14:56:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.007 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 733396176 0 0) 0x7fe75c003530 con 0x7fe7700a2800 2024-09-16T14:56:38.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe770134250 con 0x7fe7700a2800 2024-09-16T14:56:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe768001d60 con 0x7fe7700a2800 2024-09-16T14:56:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): dump: 2024-09-16T14:56:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.009 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227336592 0 0) 0x7fe7700a6770 con 0x7fe7700a1d10 2024-09-16T14:56:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient(hunting): my global_id is 264376 2024-09-16T14:56:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe75c003530 con 0x7fe7700a1d10 2024-09-16T14:56:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3951330475 0 0) 0x7fe77009e500 con 0x7fe7700a0000 2024-09-16T14:56:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 >> v1:172.21.15.26:6790/0 conn(0x7fe7700a2800 legacy=0x7fe7701d18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 >> v1:172.21.15.189:6789/0 conn(0x7fe7700a1d10 legacy=0x7fe7701ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient: _finish_hunting 0 2024-09-16T14:56:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 monclient: found mon.a 2024-09-16T14:56:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7700a6220 con 0x7fe7700a0000 2024-09-16T14:56:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient: _finish_auth 0 2024-09-16T14:56:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient: _check_auth_tickets 2024-09-16T14:56:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 5 monclient: authenticate success, global_id 274302 2024-09-16T14:56:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 10 monclient: _renew_subs 2024-09-16T14:56:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 1 -- 172.21.15.26:0/2617402238 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe770135580 con 0x7fe7700a0000 2024-09-16T14:56:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 10 monclient: _renew_subs 2024-09-16T14:56:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 1 -- 172.21.15.26:0/2617402238 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7701357c0 con 0x7fe7700a0000 2024-09-16T14:56:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe760003620 con 0x7fe7700a0000 2024-09-16T14:56:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe760003930 con 0x7fe7700a0000 2024-09-16T14:56:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 10 monclient: dump: 2024-09-16T14:56:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.012 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe775c31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe76dffb640 4 set_mon_vals no callback set 2024-09-16T14:56:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.993+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe76000e3a0 con 0x7fe7700a0000 2024-09-16T14:56:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fe760042b40 con 0x7fe7700a0000 2024-09-16T14:56:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe775c31640 1 -- 172.21.15.26:0/2617402238 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe7701c9270 con 0x7fe7701c8e90 2024-09-16T14:56:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe76effd640 10 monclient: get_auth_request con 0x7fe75c036390 auth_method 0 2024-09-16T14:56:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285076792 0 0) 0x7fe7701c9270 con 0x7fe7701c8e90 2024-09-16T14:56:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 10 monclient(hunting): my global_id is 284195 2024-09-16T14:56:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe77009e500 con 0x7fe7701c8e90 2024-09-16T14:56:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 243016788 0 0) 0x7fe77009e500 con 0x7fe7701c8e90 2024-09-16T14:56:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7701c9270 con 0x7fe7701c8e90 2024-09-16T14:56:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe768003be0 con 0x7fe7701c8e90 2024-09-16T14:56:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 10 monclient: dump: 2024-09-16T14:56:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.014 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 199536817 0 0) 0x7fe7701c9270 con 0x7fe7701c8e90 2024-09-16T14:56:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe77009e810 con 0x7fe7701c8e90 2024-09-16T14:56:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe768005b10 con 0x7fe7701c8e90 2024-09-16T14:56:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 >> v1:172.21.15.26:6790/0 conn(0x7fe7701c8e90 legacy=0x7fe77009c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:37.997+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 >> v1:172.21.15.26:6790/0 conn(0x7fe7701c8e90 legacy=0x7fe77009c040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:38.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.001+0000 7fe775c31640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:56:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.001+0000 7fe775c31640 1 -- 172.21.15.26:0/2617402238 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe770134a10 con 0x7fe77002c280 2024-09-16T14:56:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.001+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604425690 0 0) 0x7fe770134a10 con 0x7fe77002c280 2024-09-16T14:56:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.001+0000 7fe757fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.001+0000 7fe757fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.001+0000 7fe757fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.001+0000 7fe757fff640 10 monclient(hunting): my global_id is 284198 2024-09-16T14:56:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.001+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7701c9270 con 0x7fe77002c280 2024-09-16T14:56:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.001+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2415515851 0 0) 0x7fe7701c9270 con 0x7fe77002c280 2024-09-16T14:56:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.001+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe770134a10 con 0x7fe77002c280 2024-09-16T14:56:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.001+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe7680043d0 con 0x7fe77002c280 2024-09-16T14:56:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.001+0000 7fe757fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.001+0000 7fe757fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.001+0000 7fe757fff640 10 monclient: dump: 2024-09-16T14:56:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.017 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 664238944 0 0) 0x7fe770134a10 con 0x7fe77002c280 2024-09-16T14:56:38.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe770073970 con 0x7fe77002c280 2024-09-16T14:56:38.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 1897728654) 0x7fe7680045c0 con 0x7fe77002c280 2024-09-16T14:56:38.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe757fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:56:38.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe757fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:38.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 >> v1:172.21.15.26:6790/0 conn(0x7fe77002c280 legacy=0x7fe77009c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe757fff640 1 -- 172.21.15.26:0/2617402238 >> v1:172.21.15.26:6790/0 conn(0x7fe77002c280 legacy=0x7fe77009c750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:38.018 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:56:38.018 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:56:38.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:56:38.019 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:56:38.019 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 132, 2024-09-16T14:56:38.019 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:56:38.019 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:56:38.019 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:56:38.019 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:56:38.019 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:38.019 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:56:38.019 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:38.019 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:56:38.019 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:56:38.019 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:38.019 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:38.019 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:38.020 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:38.020 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:38.020 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:38.020 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:38.020 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:38.020 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:38.020 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:38.020 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:38.020 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:56:38.020 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:56:38.020 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:38.020 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:38.020 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:38.021 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:38.021 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:38.021 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:38.021 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:38.021 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:38.021 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:38.021 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:38.021 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:38.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.021 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:56:38.021 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:56:38.021 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:56:38.021 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:56:38.022 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:56:38.022 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:56:38.022 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:38.022 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:38.022 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:56:38.022 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:56:38.022 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:56:38.022 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:56:38.022 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:56:38.022 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:56:38.022 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:56:38.022 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:38.022 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:56:38.022 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:38.023 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:38.023 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:38.023 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:38.023 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:38.023 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:38.023 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:38.023 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:38.023 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:38.023 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:38.023 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:38.023 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:56:38.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.023 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:56:38.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:38.024 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:56:38.024 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:56:38.024 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:38.024 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:38.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:38.024 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:38.024 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:56:38.024 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:38.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:38.024 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:38.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.024 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:56:38.025 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:56:38.025 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:38.025 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:38.025 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:38.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:38.025 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:56:38.025 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:56:38.025 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:38.025 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:38.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:38.026 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:38.026 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:56:38.026 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:38.026 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:38.026 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:38.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.026 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:56:38.026 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:56:38.026 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:38.026 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:38.026 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:38.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:38.027 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:56:38.027 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:56:38.028 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:38.028 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:38.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:38.028 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:38.028 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:56:38.028 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:38.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:38.028 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:38.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.028 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:56:38.028 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:56:38.028 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:38.028 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:38.029 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:38.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:38.029 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:38.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.029 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:56:38.029 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:56:38.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:38.029 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:38.029 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:38.029 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:56:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.030 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:56:38.030 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:56:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe775c31640 1 -- 172.21.15.26:0/2617402238 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe75c036390 legacy=0x7fe75c038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe775c31640 10 monclient: shutdown 2024-09-16T14:56:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe775c31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe775c31640 1 -- 172.21.15.26:0/2617402238 >> v1:172.21.15.26:6789/0 conn(0x7fe7700a0000 legacy=0x7fe77013d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe76ffff640 1 -- 172.21.15.26:0/2617402238 reap_dead start 2024-09-16T14:56:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe76ffff640 1 -- 172.21.15.26:0/2617402238 reap_dead start 2024-09-16T14:56:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe775c31640 1 -- 172.21.15.26:0/2617402238 shutdown_connections 2024-09-16T14:56:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe775c31640 1 -- 172.21.15.26:0/2617402238 >> 172.21.15.26:0/2617402238 conn(0x7fe7700a7f30 msgr2=0x7fe77006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe775c31640 1 -- 172.21.15.26:0/2617402238 shutdown_connections 2024-09-16T14:56:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.005+0000 7fe775c31640 1 -- 172.21.15.26:0/2617402238 wait complete. 2024-09-16T14:56:38.032 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:56:38.033 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:56:38.033 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:56:38.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 10 monclient: get_monmap_and_config 2024-09-16T14:56:38.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 10 monclient: build_initial_monmap 2024-09-16T14:56:38.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 10 monclient: monmap: 2024-09-16T14:56:38.148 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:38.148 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.149 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:38.150282+0000 2024-09-16T14:56:38.149 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:38.150282+0000 2024-09-16T14:56:38.149 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:38.149 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.149 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:38.149 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:38.149 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:38.149 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 1 Processor -- start 2024-09-16T14:56:38.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 1 -- start start 2024-09-16T14:56:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 10 monclient: init 2024-09-16T14:56:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:56:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 10 monclient(hunting): picked mon.noname-b con 0x7f281009a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 10 monclient(hunting): picked mon.noname-a con 0x7f281009d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 10 monclient(hunting): picked mon.noname-c con 0x7f281009b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281009dc60 con 0x7f281009d1b0 2024-09-16T14:56:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2810099ae0 con 0x7f281009b040 2024-09-16T14:56:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281009b8f0 con 0x7f281009a550 2024-09-16T14:56:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 10 monclient(hunting): authenticate will time out at 1822.270796s 2024-09-16T14:56:38.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280f7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f281009a550 0x7f281009a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48148/0 (socket says 172.21.15.26:48148) 2024-09-16T14:56:38.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280effd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f281009d1b0 0x7f2810136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36266/0 (socket says 172.21.15.26:36266) 2024-09-16T14:56:38.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280f7fe640 1 -- 172.21.15.26:0/3314559493 learned_addr learned my addr 172.21.15.26:0/3314559493 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:38.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 1 -- 172.21.15.26:0/3314559493 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1191575026 0 0) 0x7f281009dc60 con 0x7f281009d1b0 2024-09-16T14:56:38.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:38.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:38.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:38.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient(hunting): my global_id is 274305 2024-09-16T14:56:38.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 1 -- 172.21.15.26:0/3314559493 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27f8003530 con 0x7f281009d1b0 2024-09-16T14:56:38.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 1 -- 172.21.15.26:0/3314559493 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592339642 0 0) 0x7f281009b8f0 con 0x7f281009a550 2024-09-16T14:56:38.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:38.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:38.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient(hunting): my global_id is 284201 2024-09-16T14:56:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 1 -- 172.21.15.26:0/3314559493 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f281009dc60 con 0x7f281009a550 2024-09-16T14:56:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 1 -- 172.21.15.26:0/3314559493 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795557349 0 0) 0x7f2810099ae0 con 0x7f281009b040 2024-09-16T14:56:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient(hunting): my global_id is 264379 2024-09-16T14:56:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 1 -- 172.21.15.26:0/3314559493 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f281009b8f0 con 0x7f281009b040 2024-09-16T14:56:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 1 -- 172.21.15.26:0/3314559493 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1900428793 0 0) 0x7f27f8003530 con 0x7f281009d1b0 2024-09-16T14:56:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 1 -- 172.21.15.26:0/3314559493 >> v1:172.21.15.26:6790/0 conn(0x7f281009a550 legacy=0x7f281009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 1 -- 172.21.15.26:0/3314559493 >> v1:172.21.15.189:6789/0 conn(0x7f281009b040 legacy=0x7f281013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:56:38.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 1 monclient: found mon.noname-a 2024-09-16T14:56:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:56:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 1 -- 172.21.15.26:0/3314559493 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281009bbd0 con 0x7f281009d1b0 2024-09-16T14:56:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:56:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:56:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 5 monclient: authenticate success, global_id 274305 2024-09-16T14:56:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 1 -- 172.21.15.26:0/3314559493 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2804001620 con 0x7f281009d1b0 2024-09-16T14:56:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient: dump: 2024-09-16T14:56:38.156 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.157 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.157 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.157 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.157 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.157 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.157 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.157 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.157 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.157 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:38.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 1 -- 172.21.15.26:0/3314559493 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2804001840 con 0x7f281009d1b0 2024-09-16T14:56:38.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:38.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 1 -- 172.21.15.26:0/3314559493 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2804001c10 con 0x7f281009d1b0 2024-09-16T14:56:38.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:38.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f280e7fc640 10 monclient: dump: 2024-09-16T14:56:38.158 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.158 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.158 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.158 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.158 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.158 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.158 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.159 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.159 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.159 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:38.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.149+0000 7f2815a9d640 4 set_mon_vals no callback set 2024-09-16T14:56:38.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 10 monclient: shutdown 2024-09-16T14:56:38.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:38.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 1 -- 172.21.15.26:0/3314559493 >> v1:172.21.15.26:6789/0 conn(0x7f281009d1b0 legacy=0x7f2810136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 1 -- 172.21.15.26:0/3314559493 shutdown_connections 2024-09-16T14:56:38.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 1 -- 172.21.15.26:0/3314559493 >> 172.21.15.26:0/3314559493 conn(0x7f281009fb50 msgr2=0x7f281009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:38.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 1 -- 172.21.15.26:0/3314559493 shutdown_connections 2024-09-16T14:56:38.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 1 -- 172.21.15.26:0/3314559493 wait complete. 2024-09-16T14:56:38.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 10 monclient: build_initial_monmap 2024-09-16T14:56:38.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 10 monclient: monmap: 2024-09-16T14:56:38.160 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:38.160 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:38.160 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:38.160 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:38.160 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:38.160 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:38.160 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:38.160 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:38.160 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:38.160 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 1 Processor -- start 2024-09-16T14:56:38.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 1 -- start start 2024-09-16T14:56:38.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 10 monclient: init 2024-09-16T14:56:38.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:56:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 10 monclient(hunting): picked mon.noname-a con 0x7f281009a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 10 monclient(hunting): picked mon.noname-b con 0x7f281009b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 10 monclient(hunting): picked mon.noname-c con 0x7f281009d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281009a240 con 0x7f281009a550 2024-09-16T14:56:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28101a3700 con 0x7f281009b040 2024-09-16T14:56:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28101a2e60 con 0x7f281009d1b0 2024-09-16T14:56:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 10 monclient(hunting): authenticate will time out at 1822.274197s 2024-09-16T14:56:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f280f7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f281009a550 0x7f28101b4f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36268/0 (socket says 172.21.15.26:36268) 2024-09-16T14:56:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f280f7fe640 1 -- 172.21.15.26:0/1236106868 learned_addr learned my addr 172.21.15.26:0/1236106868 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412495489 0 0) 0x7f281009a240 con 0x7f281009a550 2024-09-16T14:56:38.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:38.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:38.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:38.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): my global_id is 274308 2024-09-16T14:56:38.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27fc003530 con 0x7f281009a550 2024-09-16T14:56:38.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747133748 0 0) 0x7f28101a2e60 con 0x7f281009d1b0 2024-09-16T14:56:38.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:38.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:38.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:38.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): my global_id is 284204 2024-09-16T14:56:38.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f281009a240 con 0x7f281009d1b0 2024-09-16T14:56:38.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3117021228 0 0) 0x7f27fc003530 con 0x7f281009a550 2024-09-16T14:56:38.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28101a2e60 con 0x7f281009a550 2024-09-16T14:56:38.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f28000019e0 con 0x7f281009a550 2024-09-16T14:56:38.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:38.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): dump: 2024-09-16T14:56:38.167 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.167 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.167 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.167 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.167 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.167 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.167 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.167 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.168 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.168 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:38.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 819774687 0 0) 0x7f281009a240 con 0x7f281009d1b0 2024-09-16T14:56:38.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27fc003530 con 0x7f281009d1b0 2024-09-16T14:56:38.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2808001bc0 con 0x7f281009d1b0 2024-09-16T14:56:38.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:38.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): dump: 2024-09-16T14:56:38.168 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.168 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.168 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.168 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.169 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175807648 0 0) 0x7f28101a3700 con 0x7f281009b040 2024-09-16T14:56:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient(hunting): my global_id is 264382 2024-09-16T14:56:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f281009a240 con 0x7f281009b040 2024-09-16T14:56:38.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1080202790 0 0) 0x7f28101a2e60 con 0x7f281009a550 2024-09-16T14:56:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 >> v1:172.21.15.26:6790/0 conn(0x7f281009d1b0 legacy=0x7f28101ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 >> v1:172.21.15.189:6789/0 conn(0x7f281009b040 legacy=0x7f28101c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:56:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 monclient: found mon.a 2024-09-16T14:56:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28101a1330 con 0x7f281009a550 2024-09-16T14:56:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient: _finish_auth 0 2024-09-16T14:56:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:56:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 5 monclient: authenticate success, global_id 274308 2024-09-16T14:56:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 10 monclient: _renew_subs 2024-09-16T14:56:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:38.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 1 -- 172.21.15.26:0/1236106868 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28101b43d0 con 0x7f281009a550 2024-09-16T14:56:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 10 monclient: _renew_subs 2024-09-16T14:56:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 1 -- 172.21.15.26:0/1236106868 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28101a0960 con 0x7f281009a550 2024-09-16T14:56:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28000047f0 con 0x7f281009a550 2024-09-16T14:56:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2800004b00 con 0x7f281009a550 2024-09-16T14:56:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f27f7fff640 10 monclient: dump: 2024-09-16T14:56:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.171 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.172 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.172 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.172 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.172 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.172 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f280dffb640 4 set_mon_vals no callback set 2024-09-16T14:56:38.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:38.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.153+0000 7f2815a9d640 1 -- 172.21.15.26:0/1236106868 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f27f0006590 con 0x7f27f0002830 2024-09-16T14:56:38.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f280000f570 con 0x7f281009a550 2024-09-16T14:56:38.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f2800043e40 con 0x7f281009a550 2024-09-16T14:56:38.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f280ffff640 10 monclient: get_auth_request con 0x7f27fc0363f0 auth_method 0 2024-09-16T14:56:38.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541304990 0 0) 0x7f27f0006590 con 0x7f27f0002830 2024-09-16T14:56:38.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:38.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:38.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:38.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 10 monclient(hunting): my global_id is 264385 2024-09-16T14:56:38.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28101a2e60 con 0x7f27f0002830 2024-09-16T14:56:38.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4131687242 0 0) 0x7f28101a2e60 con 0x7f27f0002830 2024-09-16T14:56:38.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27f0006590 con 0x7f27f0002830 2024-09-16T14:56:38.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f28040011b0 con 0x7f27f0002830 2024-09-16T14:56:38.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:56:38.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 10 monclient: dump: 2024-09-16T14:56:38.173 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.173 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.173 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.173 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.174 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.174 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.174 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.174 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.174 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.174 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2005385094 0 0) 0x7f27f0006590 con 0x7f27f0002830 2024-09-16T14:56:38.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27f000a0e0 con 0x7f27f0002830 2024-09-16T14:56:38.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f28040014a0 con 0x7f27f0002830 2024-09-16T14:56:38.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:38.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:38.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 >> v1:172.21.15.189:6789/0 conn(0x7f27f0002830 legacy=0x7f27f0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.157+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 >> v1:172.21.15.189:6789/0 conn(0x7f27f0002830 legacy=0x7f27f0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:38.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f2815a9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:56:38.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f2815a9d640 1 -- 172.21.15.26:0/1236106868 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f27f0009da0 con 0x7f27f00090c0 2024-09-16T14:56:38.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84550442 0 0) 0x7f27f0009da0 con 0x7f27f00090c0 2024-09-16T14:56:38.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:38.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:38.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:38.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 10 monclient(hunting): my global_id is 264388 2024-09-16T14:56:38.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27f0006590 con 0x7f27f00090c0 2024-09-16T14:56:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1953840954 0 0) 0x7f27f0006590 con 0x7f27f00090c0 2024-09-16T14:56:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27f0009da0 con 0x7f27f00090c0 2024-09-16T14:56:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2804004b80 con 0x7f27f00090c0 2024-09-16T14:56:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:56:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 10 monclient: dump: 2024-09-16T14:56:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.177 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 382740556 0 0) 0x7f27f0009da0 con 0x7f27f00090c0 2024-09-16T14:56:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f27f000c290 con 0x7f27f00090c0 2024-09-16T14:56:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 118749256) 0x7f2804005f30 con 0x7f27f00090c0 2024-09-16T14:56:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:56:38.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:38.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 >> v1:172.21.15.189:6789/0 conn(0x7f27f00090c0 legacy=0x7f27f0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.165+0000 7f27f7fff640 1 -- 172.21.15.26:0/1236106868 >> v1:172.21.15.189:6789/0 conn(0x7f27f00090c0 legacy=0x7f27f0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:38.178 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:56:38.178 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:56:38.178 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:56:38.178 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:56:38.178 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 132, 2024-09-16T14:56:38.178 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:56:38.178 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:56:38.178 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:56:38.179 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:56:38.179 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:38.179 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-09-16T14:56:38.179 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:38.179 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:56:38.179 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:56:38.179 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:38.179 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:38.179 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:38.179 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:38.179 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:38.179 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:38.179 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:38.180 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:38.180 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:38.180 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:38.180 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:38.180 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:56:38.180 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:56:38.180 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:38.180 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:38.180 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:38.180 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:38.180 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:38.180 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:38.180 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:38.181 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:38.181 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:38.181 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:38.181 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:38.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.181 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:56:38.181 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:56:38.181 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:56:38.181 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:56:38.181 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:56:38.181 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:56:38.181 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:38.181 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:38.181 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:56:38.182 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:56:38.182 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:56:38.182 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:56:38.182 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:56:38.182 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:56:38.182 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:56:38.182 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:38.182 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:56:38.182 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:38.182 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:38.182 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:38.182 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:38.182 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:38.183 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:38.183 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:38.183 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:38.183 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:38.183 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:38.183 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:38.183 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:56:38.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.183 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:56:38.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:38.183 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:56:38.183 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:56:38.183 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:38.184 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:38.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:38.184 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:38.184 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:56:38.184 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:38.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:38.184 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:38.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.184 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:56:38.184 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:56:38.184 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:38.184 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:38.184 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:38.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:38.185 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:56:38.185 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:56:38.185 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:38.185 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:38.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:38.185 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:38.185 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:56:38.185 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:38.185 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:38.185 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:38.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:56:38.186 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:56:38.186 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:38.186 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:38.186 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:38.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:38.186 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:56:38.186 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:56:38.186 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:38.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:38.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:38.186 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:38.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:56:38.187 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:38.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:38.187 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:38.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.187 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:56:38.187 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:56:38.187 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:38.187 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:38.187 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:38.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:38.187 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:38.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.187 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:56:38.188 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:56:38.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:38.188 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:38.188 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:38.188 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:38.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:38.188 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:38.188 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:56:38.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:38.188 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:38.188 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:38.188 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:56:38.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:38.189 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:38.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:38.189 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:56:38.189 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:56:38.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.169+0000 7f2815a9d640 1 -- 172.21.15.26:0/1236106868 >> v1:172.21.15.189:6812/3196161582 conn(0x7f27fc0363f0 legacy=0x7f27fc0388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.169+0000 7f2815a9d640 10 monclient: shutdown 2024-09-16T14:56:38.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.169+0000 7f2815a9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:38.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.169+0000 7f2815a9d640 1 -- 172.21.15.26:0/1236106868 >> v1:172.21.15.26:6789/0 conn(0x7f281009a550 legacy=0x7f28101b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.169+0000 7f280ffff640 1 -- 172.21.15.26:0/1236106868 reap_dead start 2024-09-16T14:56:38.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.169+0000 7f280ffff640 1 -- 172.21.15.26:0/1236106868 reap_dead start 2024-09-16T14:56:38.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.169+0000 7f2815a9d640 1 -- 172.21.15.26:0/1236106868 shutdown_connections 2024-09-16T14:56:38.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.169+0000 7f2815a9d640 1 -- 172.21.15.26:0/1236106868 >> 172.21.15.26:0/1236106868 conn(0x7f281009fb50 msgr2=0x7f281006cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:38.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.169+0000 7f2815a9d640 1 -- 172.21.15.26:0/1236106868 shutdown_connections 2024-09-16T14:56:38.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.169+0000 7f2815a9d640 1 -- 172.21.15.26:0/1236106868 wait complete. 2024-09-16T14:56:38.192 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:56:38.192 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T14:56:38.192 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:56:38.193 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T14:56:38.193 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:56:38.193 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T14:56:38.193 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:56:38.209 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T14:56:38.209 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:56:38.209 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:56:38.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f917530f640 10 monclient: get_monmap_and_config 2024-09-16T14:56:38.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f917530f640 10 monclient: build_initial_monmap 2024-09-16T14:56:38.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f917530f640 10 monclient: monmap: 2024-09-16T14:56:38.321 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:38.322 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.322 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:38.323442+0000 2024-09-16T14:56:38.322 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:38.323442+0000 2024-09-16T14:56:38.322 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:38.322 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.322 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:38.322 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:38.322 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:38.322 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f917530f640 1 Processor -- start 2024-09-16T14:56:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f917530f640 1 -- start start 2024-09-16T14:56:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f917530f640 10 monclient: init 2024-09-16T14:56:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f917530f640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f917530f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:56:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f917530f640 10 monclient(hunting): picked mon.noname-b con 0x7f917009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f917530f640 10 monclient(hunting): picked mon.noname-a con 0x7f9170132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f917530f640 10 monclient(hunting): picked mon.noname-c con 0x7f9170138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f917530f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f917009d260 con 0x7f9170132650 2024-09-16T14:56:38.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f917530f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9170132a80 con 0x7f9170138ff0 2024-09-16T14:56:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f917530f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f917009cf60 con 0x7f917009cb80 2024-09-16T14:56:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f917530f640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f917530f640 10 monclient(hunting): authenticate will time out at 1822.443717s 2024-09-16T14:56:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f916effd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f917009cb80 0x7f91700717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48162/0 (socket says 172.21.15.26:48162) 2024-09-16T14:56:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f916effd640 1 -- 172.21.15.26:0/3419479948 learned_addr learned my addr 172.21.15.26:0/3419479948 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.321+0000 7f916f7fe640 1 -- 172.21.15.26:0/3419479948 >> v1:172.21.15.189:6789/0 conn(0x7f9170138ff0 legacy=0x7f917013b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 1 -- 172.21.15.26:0/3419479948 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028352450 0 0) 0x7f917009cf60 con 0x7f917009cb80 2024-09-16T14:56:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 10 monclient(hunting): my global_id is 284207 2024-09-16T14:56:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 1 -- 172.21.15.26:0/3419479948 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9158003530 con 0x7f917009cb80 2024-09-16T14:56:38.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 1 -- 172.21.15.26:0/3419479948 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3492486273 0 0) 0x7f9158003530 con 0x7f917009cb80 2024-09-16T14:56:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 1 -- 172.21.15.26:0/3419479948 >> v1:172.21.15.189:6789/0 conn(0x7f9170138ff0 legacy=0x7f917013b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:56:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 1 -- 172.21.15.26:0/3419479948 >> v1:172.21.15.26:6789/0 conn(0x7f9170132650 legacy=0x7f91701378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:56:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 1 monclient: found mon.noname-b 2024-09-16T14:56:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:56:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 1 -- 172.21.15.26:0/3419479948 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917009f2d0 con 0x7f917009cb80 2024-09-16T14:56:38.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 10 monclient: _finish_auth 0 2024-09-16T14:56:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:56:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 1 -- 172.21.15.26:0/3419479948 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9160001660 con 0x7f917009cb80 2024-09-16T14:56:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 5 monclient: authenticate success, global_id 284207 2024-09-16T14:56:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:56:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 10 monclient: dump: 2024-09-16T14:56:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.332 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.333 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.333 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.333 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.333 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.333 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.333 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.333 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.333 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:38.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 1 -- 172.21.15.26:0/3419479948 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91600018c0 con 0x7f917009cb80 2024-09-16T14:56:38.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:38.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 1 -- 172.21.15.26:0/3419479948 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9160001c50 con 0x7f917009cb80 2024-09-16T14:56:38.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:38.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 4 set_mon_vals no callback set 2024-09-16T14:56:38.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:38.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916dffb640 10 monclient: dump: 2024-09-16T14:56:38.334 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.334 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.334 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.334 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.334 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.334 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.334 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.334 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.335 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.335 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 10 monclient: shutdown 2024-09-16T14:56:38.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:38.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 1 -- 172.21.15.26:0/3419479948 >> v1:172.21.15.26:6790/0 conn(0x7f917009cb80 legacy=0x7f91700717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 1 -- 172.21.15.26:0/3419479948 shutdown_connections 2024-09-16T14:56:38.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 1 -- 172.21.15.26:0/3419479948 >> 172.21.15.26:0/3419479948 conn(0x7f917009fb50 msgr2=0x7f917009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:38.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 1 -- 172.21.15.26:0/3419479948 shutdown_connections 2024-09-16T14:56:38.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 1 -- 172.21.15.26:0/3419479948 wait complete. 2024-09-16T14:56:38.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 10 monclient: build_initial_monmap 2024-09-16T14:56:38.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 10 monclient: monmap: 2024-09-16T14:56:38.336 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:38.336 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:38.336 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:38.336 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:38.336 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:38.337 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:38.337 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:38.337 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:38.337 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:38.337 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 1 Processor -- start 2024-09-16T14:56:38.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 1 -- start start 2024-09-16T14:56:38.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 10 monclient: init 2024-09-16T14:56:38.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:38.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:56:38.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 10 monclient(hunting): picked mon.noname-b con 0x7f917009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:38.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 10 monclient(hunting): picked mon.noname-a con 0x7f9170132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:38.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 10 monclient(hunting): picked mon.noname-c con 0x7f9170138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:38.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91701364a0 con 0x7f9170132650 2024-09-16T14:56:38.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9170137e70 con 0x7f917009cb80 2024-09-16T14:56:38.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9170134d10 con 0x7f9170138ff0 2024-09-16T14:56:38.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:38.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 10 monclient(hunting): authenticate will time out at 1822.446904s 2024-09-16T14:56:38.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916f7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9170138ff0 0x7f91701cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48172/0 (socket says 172.21.15.26:48172) 2024-09-16T14:56:38.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916f7fe640 1 -- 172.21.15.26:0/1724757699 learned_addr learned my addr 172.21.15.26:0/1724757699 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:38.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f916effd640 1 -- 172.21.15.26:0/1724757699 >> v1:172.21.15.189:6789/0 conn(0x7f917009cb80 legacy=0x7f9170133ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:38.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 1 -- 172.21.15.26:0/1724757699 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206620636 0 0) 0x7f9170134d10 con 0x7f9170138ff0 2024-09-16T14:56:38.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 10 monclient(hunting): _negotiate 2024-09-16T14:56:38.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:38.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:38.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 10 monclient(hunting): my global_id is 284210 2024-09-16T14:56:38.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 1 -- 172.21.15.26:0/1724757699 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f915c003530 con 0x7f9170138ff0 2024-09-16T14:56:38.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 1 -- 172.21.15.26:0/1724757699 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2384815573 0 0) 0x7f915c003530 con 0x7f9170138ff0 2024-09-16T14:56:38.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 1 -- 172.21.15.26:0/1724757699 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9170134d10 con 0x7f9170138ff0 2024-09-16T14:56:38.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 1 -- 172.21.15.26:0/1724757699 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9168004450 con 0x7f9170138ff0 2024-09-16T14:56:38.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:56:38.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 10 monclient(hunting): dump: 2024-09-16T14:56:38.339 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.339 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.339 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.339 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.340 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.340 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.340 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.340 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.340 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.340 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:38.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 1 -- 172.21.15.26:0/1724757699 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1733581710 0 0) 0x7f9170134d10 con 0x7f9170138ff0 2024-09-16T14:56:38.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 1 -- 172.21.15.26:0/1724757699 >> v1:172.21.15.189:6789/0 conn(0x7f917009cb80 legacy=0x7f9170133ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:56:38.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 1 -- 172.21.15.26:0/1724757699 >> v1:172.21.15.26:6789/0 conn(0x7f9170132650 legacy=0x7f9170134600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 10 monclient: _finish_hunting 0 2024-09-16T14:56:38.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 1 monclient: found mon.c 2024-09-16T14:56:38.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:38.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:38.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 1 -- 172.21.15.26:0/1724757699 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9170131df0 con 0x7f9170138ff0 2024-09-16T14:56:38.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 10 monclient: _finish_auth 0 2024-09-16T14:56:38.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 10 monclient: _check_auth_tickets 2024-09-16T14:56:38.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:38.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 5 monclient: authenticate success, global_id 284210 2024-09-16T14:56:38.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 10 monclient: _renew_subs 2024-09-16T14:56:38.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:38.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 1 -- 172.21.15.26:0/1724757699 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f917013c5e0 con 0x7f9170138ff0 2024-09-16T14:56:38.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 10 monclient: _renew_subs 2024-09-16T14:56:38.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:38.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f917530f640 1 -- 172.21.15.26:0/1724757699 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9170135aa0 con 0x7f9170138ff0 2024-09-16T14:56:38.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.325+0000 7f91577fe640 1 -- 172.21.15.26:0/1724757699 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9168004770 con 0x7f9170138ff0 2024-09-16T14:56:38.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.329+0000 7f91577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:38.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.329+0000 7f91577fe640 1 -- 172.21.15.26:0/1724757699 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9168004b00 con 0x7f9170138ff0 2024-09-16T14:56:38.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.329+0000 7f91577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:38.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.329+0000 7f91577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:38.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.329+0000 7f916d7fa640 4 set_mon_vals no callback set 2024-09-16T14:56:38.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.329+0000 7f91577fe640 10 monclient: dump: 2024-09-16T14:56:38.342 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:38.342 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:38.342 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.342 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:38.342 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:38.342 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:38.343 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:38.343 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:38.343 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:38.343 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:38.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.329+0000 7f917530f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:38.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.329+0000 7f917530f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:38.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.329+0000 7f917530f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:38.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.329+0000 7f917530f640 1 -- 172.21.15.26:0/1724757699 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9150005050 con 0x7f9170138ff0 2024-09-16T14:56:38.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.329+0000 7f91577fe640 1 -- 172.21.15.26:0/1724757699 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9168004ce0 con 0x7f9170138ff0 2024-09-16T14:56:38.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.329+0000 7f91577fe640 1 -- 172.21.15.26:0/1724757699 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f9168044110 con 0x7f9170138ff0 2024-09-16T14:56:38.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.329+0000 7f916effd640 10 monclient: get_auth_request con 0x7f915c036030 auth_method 0 2024-09-16T14:56:38.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.333+0000 7f91577fe640 1 -- 172.21.15.26:0/1724757699 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9168014b00 con 0x7f9170138ff0 2024-09-16T14:56:38.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.333+0000 7f91577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:38.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.333+0000 7f91577fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:38.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.545+0000 7f917530f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:56:38.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.545+0000 7f917530f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:56:38.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.545+0000 7f917530f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:38.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.545+0000 7f917530f640 1 -- 172.21.15.26:0/1724757699 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9150005880 con 0x7f9170138ff0 2024-09-16T14:56:38.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.549+0000 7f91577fe640 1 -- 172.21.15.26:0/1724757699 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 145602366) 0x7f9168012f70 con 0x7f9170138ff0 2024-09-16T14:56:38.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.549+0000 7f91577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:56:38.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.549+0000 7f91577fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:38.548 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:56:38.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.549+0000 7f917530f640 1 -- 172.21.15.26:0/1724757699 >> v1:172.21.15.189:6812/3196161582 conn(0x7f915c036030 legacy=0x7f915c0384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.549+0000 7f917530f640 10 monclient: shutdown 2024-09-16T14:56:38.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.549+0000 7f917530f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:38.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.549+0000 7f917530f640 1 -- 172.21.15.26:0/1724757699 >> v1:172.21.15.26:6790/0 conn(0x7f9170138ff0 legacy=0x7f91701cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:38.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.549+0000 7f917530f640 1 -- 172.21.15.26:0/1724757699 shutdown_connections 2024-09-16T14:56:38.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.549+0000 7f917530f640 1 -- 172.21.15.26:0/1724757699 >> 172.21.15.26:0/1724757699 conn(0x7f917009fb50 msgr2=0x7f9170130450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:38.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.549+0000 7f917530f640 1 -- 172.21.15.26:0/1724757699 shutdown_connections 2024-09-16T14:56:38.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:38.549+0000 7f917530f640 1 -- 172.21.15.26:0/1724757699 wait complete. 2024-09-16T14:56:41.566 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:56:41.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 10 monclient: get_monmap_and_config 2024-09-16T14:56:41.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 10 monclient: build_initial_monmap 2024-09-16T14:56:41.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 10 monclient: monmap: 2024-09-16T14:56:41.701 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:41.701 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:41.701 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:41.702626+0000 2024-09-16T14:56:41.701 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:41.702626+0000 2024-09-16T14:56:41.701 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:41.701 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:41.701 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:41.701 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:41.701 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:41.701 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:41.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 1 Processor -- start 2024-09-16T14:56:41.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 1 -- start start 2024-09-16T14:56:41.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 10 monclient: init 2024-09-16T14:56:41.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:41.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:56:41.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 10 monclient(hunting): picked mon.noname-c con 0x7f843809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:41.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 10 monclient(hunting): picked mon.noname-a con 0x7f8438132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:41.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 10 monclient(hunting): picked mon.noname-b con 0x7f8438138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:41.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f843809d260 con 0x7f8438132650 2024-09-16T14:56:41.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8438132a80 con 0x7f843809cb80 2024-09-16T14:56:41.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f843809cf60 con 0x7f8438138ff0 2024-09-16T14:56:41.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:41.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 10 monclient(hunting): authenticate will time out at 1825.822890s 2024-09-16T14:56:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843d0c2640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f843809cb80 legacy=0x7f84380717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843d8c3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8438138ff0 0x7f843813b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48186/0 (socket says 172.21.15.26:48186) 2024-09-16T14:56:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843c8c1640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8438132650 0x7f84381378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36292/0 (socket says 172.21.15.26:36292) 2024-09-16T14:56:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843d8c3640 1 -- 172.21.15.26:0/2303393174 learned_addr learned my addr 172.21.15.26:0/2303393174 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 1 -- 172.21.15.26:0/2303393174 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229976972 0 0) 0x7f843809cf60 con 0x7f8438138ff0 2024-09-16T14:56:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 10 monclient(hunting): my global_id is 284213 2024-09-16T14:56:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 1 -- 172.21.15.26:0/2303393174 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8424003530 con 0x7f8438138ff0 2024-09-16T14:56:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 1 -- 172.21.15.26:0/2303393174 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 707933467 0 0) 0x7f8424003530 con 0x7f8438138ff0 2024-09-16T14:56:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 1 -- 172.21.15.26:0/2303393174 >> v1:172.21.15.189:6789/0 conn(0x7f843809cb80 legacy=0x7f84380717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:56:41.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 1 -- 172.21.15.26:0/2303393174 >> v1:172.21.15.26:6789/0 conn(0x7f8438132650 legacy=0x7f84381378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:41.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:56:41.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 1 monclient: found mon.noname-b 2024-09-16T14:56:41.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:41.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:56:41.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 1 -- 172.21.15.26:0/2303393174 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843809f2d0 con 0x7f8438138ff0 2024-09-16T14:56:41.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 10 monclient: _finish_auth 0 2024-09-16T14:56:41.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:56:41.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:41.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 1 -- 172.21.15.26:0/2303393174 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f84300019e0 con 0x7f8438138ff0 2024-09-16T14:56:41.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 5 monclient: authenticate success, global_id 284213 2024-09-16T14:56:41.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:41.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:56:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 10 monclient: dump: 2024-09-16T14:56:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:41.709 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:41.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 1 -- 172.21.15.26:0/2303393174 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8430001c40 con 0x7f8438138ff0 2024-09-16T14:56:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 1 -- 172.21.15.26:0/2303393174 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8430001fd0 con 0x7f8438138ff0 2024-09-16T14:56:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f842ffff640 10 monclient: dump: 2024-09-16T14:56:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:41.710 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:41.711 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.701+0000 7f843f326640 4 set_mon_vals no callback set 2024-09-16T14:56:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 10 monclient: shutdown 2024-09-16T14:56:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 1 -- 172.21.15.26:0/2303393174 >> v1:172.21.15.26:6790/0 conn(0x7f8438138ff0 legacy=0x7f843813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 1 -- 172.21.15.26:0/2303393174 shutdown_connections 2024-09-16T14:56:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 1 -- 172.21.15.26:0/2303393174 >> 172.21.15.26:0/2303393174 conn(0x7f843809fb50 msgr2=0x7f843809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 1 -- 172.21.15.26:0/2303393174 shutdown_connections 2024-09-16T14:56:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 1 -- 172.21.15.26:0/2303393174 wait complete. 2024-09-16T14:56:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 10 monclient: build_initial_monmap 2024-09-16T14:56:41.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 10 monclient: monmap: 2024-09-16T14:56:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:41.712 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 1 Processor -- start 2024-09-16T14:56:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 1 -- start start 2024-09-16T14:56:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 10 monclient: init 2024-09-16T14:56:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:56:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 10 monclient(hunting): picked mon.noname-b con 0x7f843809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 10 monclient(hunting): picked mon.noname-c con 0x7f8438132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 10 monclient(hunting): picked mon.noname-a con 0x7f8438138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8438134c60 con 0x7f8438138ff0 2024-09-16T14:56:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8438133700 con 0x7f843809cb80 2024-09-16T14:56:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f843813c600 con 0x7f8438132650 2024-09-16T14:56:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 10 monclient(hunting): authenticate will time out at 1825.826061s 2024-09-16T14:56:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843d0c2640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f843809cb80 legacy=0x7f8438139c00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843d8c3640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8438138ff0 0x7f843813aa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36294/0 (socket says 172.21.15.26:36294) 2024-09-16T14:56:41.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843d8c3640 1 -- 172.21.15.26:0/2346926856 learned_addr learned my addr 172.21.15.26:0/2346926856 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 1 -- 172.21.15.26:0/2346926856 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145546496 0 0) 0x7f843813c600 con 0x7f8438132650 2024-09-16T14:56:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 10 monclient(hunting): my global_id is 284216 2024-09-16T14:56:41.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 1 -- 172.21.15.26:0/2346926856 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f841c003530 con 0x7f8438132650 2024-09-16T14:56:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 1 -- 172.21.15.26:0/2346926856 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3665004811 0 0) 0x7f841c003530 con 0x7f8438132650 2024-09-16T14:56:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 1 -- 172.21.15.26:0/2346926856 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f843813c600 con 0x7f8438132650 2024-09-16T14:56:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 1 -- 172.21.15.26:0/2346926856 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f84280017f0 con 0x7f8438132650 2024-09-16T14:56:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:56:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 10 monclient(hunting): dump: 2024-09-16T14:56:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:41.716 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:41.717 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 1 -- 172.21.15.26:0/2346926856 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1401529058 0 0) 0x7f843813c600 con 0x7f8438132650 2024-09-16T14:56:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 1 -- 172.21.15.26:0/2346926856 >> v1:172.21.15.189:6789/0 conn(0x7f843809cb80 legacy=0x7f8438139c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:56:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 1 -- 172.21.15.26:0/2346926856 >> v1:172.21.15.26:6789/0 conn(0x7f8438138ff0 legacy=0x7f843813aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:56:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 1 monclient: found mon.c 2024-09-16T14:56:41.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:41.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:41.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 1 -- 172.21.15.26:0/2346926856 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843809a240 con 0x7f8438132650 2024-09-16T14:56:41.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 10 monclient: _finish_auth 0 2024-09-16T14:56:41.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:56:41.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:41.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 5 monclient: authenticate success, global_id 284216 2024-09-16T14:56:41.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 10 monclient: _renew_subs 2024-09-16T14:56:41.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:41.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 1 -- 172.21.15.26:0/2346926856 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8438071810 con 0x7f8438132650 2024-09-16T14:56:41.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 10 monclient: _renew_subs 2024-09-16T14:56:41.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:41.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f843f326640 1 -- 172.21.15.26:0/2346926856 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84381d2b90 con 0x7f8438132650 2024-09-16T14:56:41.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 1 -- 172.21.15.26:0/2346926856 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84280047f0 con 0x7f8438132650 2024-09-16T14:56:41.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:41.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 1 -- 172.21.15.26:0/2346926856 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8428004b80 con 0x7f8438132650 2024-09-16T14:56:41.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:41.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:41.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842effd640 4 set_mon_vals no callback set 2024-09-16T14:56:41.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.705+0000 7f842dffb640 10 monclient: dump: 2024-09-16T14:56:41.719 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:41.719 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:41.719 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:41.719 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:41.719 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:41.719 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:41.719 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:41.720 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:41.720 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:41.720 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:41.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.709+0000 7f842dffb640 1 -- 172.21.15.26:0/2346926856 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8428004d60 con 0x7f8438132650 2024-09-16T14:56:41.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.709+0000 7f843f326640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:41.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.709+0000 7f843f326640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:41.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.709+0000 7f843f326640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:41.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.709+0000 7f843f326640 1 -- 172.21.15.26:0/2346926856 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8418005050 con 0x7f8438132650 2024-09-16T14:56:41.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.709+0000 7f842dffb640 1 -- 172.21.15.26:0/2346926856 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f8428043db0 con 0x7f8438132650 2024-09-16T14:56:41.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.709+0000 7f843d0c2640 10 monclient: get_auth_request con 0x7f841c036050 auth_method 0 2024-09-16T14:56:41.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.713+0000 7f842dffb640 1 -- 172.21.15.26:0/2346926856 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f84280146f0 con 0x7f8438132650 2024-09-16T14:56:41.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.713+0000 7f842dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:41.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.713+0000 7f842dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.933+0000 7f843f326640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:56:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.933+0000 7f843f326640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:56:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.933+0000 7f843f326640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.933+0000 7f843f326640 1 -- 172.21.15.26:0/2346926856 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8418005880 con 0x7f8438132650 2024-09-16T14:56:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.933+0000 7f842dffb640 1 -- 172.21.15.26:0/2346926856 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1480685855) 0x7f8428012b60 con 0x7f8438132650 2024-09-16T14:56:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.933+0000 7f842dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:56:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.933+0000 7f842dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:41.934 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:56:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.933+0000 7f843f326640 1 -- 172.21.15.26:0/2346926856 >> v1:172.21.15.189:6812/3196161582 conn(0x7f841c036050 legacy=0x7f841c038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.933+0000 7f843f326640 10 monclient: shutdown 2024-09-16T14:56:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.933+0000 7f843f326640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.933+0000 7f843f326640 1 -- 172.21.15.26:0/2346926856 >> v1:172.21.15.26:6790/0 conn(0x7f8438132650 legacy=0x7f843813a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.937+0000 7f843f326640 1 -- 172.21.15.26:0/2346926856 shutdown_connections 2024-09-16T14:56:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.937+0000 7f843f326640 1 -- 172.21.15.26:0/2346926856 >> 172.21.15.26:0/2346926856 conn(0x7f843809fb50 msgr2=0x7f84380710e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.937+0000 7f843f326640 1 -- 172.21.15.26:0/2346926856 shutdown_connections 2024-09-16T14:56:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:41.937+0000 7f843f326640 1 -- 172.21.15.26:0/2346926856 wait complete. 2024-09-16T14:56:44.953 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:56:45.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebe31c640 10 monclient: get_monmap_and_config 2024-09-16T14:56:45.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebe31c640 10 monclient: build_initial_monmap 2024-09-16T14:56:45.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebe31c640 10 monclient: monmap: 2024-09-16T14:56:45.088 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:45.088 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:45.089 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:45.090462+0000 2024-09-16T14:56:45.089 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:45.090462+0000 2024-09-16T14:56:45.089 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:45.089 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:45.089 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:45.089 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:45.089 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:45.089 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:45.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebe31c640 1 Processor -- start 2024-09-16T14:56:45.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebe31c640 1 -- start start 2024-09-16T14:56:45.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebe31c640 10 monclient: init 2024-09-16T14:56:45.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebe31c640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:45.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebe31c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:56:45.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebe31c640 10 monclient(hunting): picked mon.noname-c con 0x7fceb80a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:45.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebe31c640 10 monclient(hunting): picked mon.noname-b con 0x7fceb8099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:45.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebe31c640 10 monclient(hunting): picked mon.noname-a con 0x7fceb80a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:45.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebe31c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb8098ea0 con 0x7fceb80a26f0 2024-09-16T14:56:45.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebe31c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb80997b0 con 0x7fceb80a1c00 2024-09-16T14:56:45.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebe31c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb8099460 con 0x7fceb8099b60 2024-09-16T14:56:45.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebe31c640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:45.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebe31c640 10 monclient(hunting): authenticate will time out at 1829.210888s 2024-09-16T14:56:45.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fceb7fff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fceb80a1c00 legacy=0x7fceb80a1fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:45.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebc8b9640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fceb80a26f0 0x7fceb813d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36302/0 (socket says 172.21.15.26:36302) 2024-09-16T14:56:45.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.088+0000 7fcebc8b9640 1 -- 172.21.15.26:0/3721140279 learned_addr learned my addr 172.21.15.26:0/3721140279 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:45.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.292+0000 7fceb7fff640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.189:6789/0 conn(0x7fceb80a1c00 legacy=0x7fceb80a1fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:45.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:45.696+0000 7fceb7fff640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.189:6789/0 conn(0x7fceb80a1c00 legacy=0x7fceb80a1fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:46.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:46.500+0000 7fceb7fff640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.189:6789/0 conn(0x7fceb80a1c00 legacy=0x7fceb80a1fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:48.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.088+0000 7fceb5ffb640 10 monclient(hunting): tick 2024-09-16T14:56:48.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.088+0000 7fceb5ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:56:48.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.088+0000 7fceb5ffb640 1 monclient(hunting): continuing hunt 2024-09-16T14:56:48.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.088+0000 7fceb5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:56:48.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.088+0000 7fceb5ffb640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.26:6790/0 conn(0x7fceb8099b60 legacy=0x7fceb8139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:48.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.088+0000 7fceb5ffb640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.189:6789/0 conn(0x7fceb80a1c00 legacy=0x7fceb80a1fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:56:48.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.088+0000 7fceb5ffb640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.26:6789/0 conn(0x7fceb80a26f0 legacy=0x7fceb813d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:48.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.088+0000 7fceb5ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:56:48.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.088+0000 7fceb5ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fce980036d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:48.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.088+0000 7fceb5ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fce98007360 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:48.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.088+0000 7fceb5ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fceb80a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:48.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.088+0000 7fceb5ffb640 1 -- 172.21.15.26:0/3721140279 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb80a34b0 con 0x7fce980036d0 2024-09-16T14:56:48.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.088+0000 7fceb5ffb640 1 -- 172.21.15.26:0/3721140279 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb80997b0 con 0x7fceb80a1c00 2024-09-16T14:56:48.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.088+0000 7fcebc8b9640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.189:6789/0 conn(0x7fceb80a1c00 legacy=0x7fce9800d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:48.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.088+0000 7fceb5ffb640 1 -- 172.21.15.26:0/3721140279 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce980108b0 con 0x7fce98007360 2024-09-16T14:56:48.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.088+0000 7fceb5ffb640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:48.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.292+0000 7fcebc8b9640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.189:6789/0 conn(0x7fceb80a1c00 legacy=0x7fce9800d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:48.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:48.696+0000 7fcebc8b9640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.189:6789/0 conn(0x7fceb80a1c00 legacy=0x7fce9800d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:49.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:49.500+0000 7fcebc8b9640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.189:6789/0 conn(0x7fceb80a1c00 legacy=0x7fce9800d0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.092+0000 7fceb5ffb640 10 monclient(hunting): tick 2024-09-16T14:56:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.092+0000 7fceb5ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:56:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.092+0000 7fceb5ffb640 1 monclient(hunting): continuing hunt 2024-09-16T14:56:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.092+0000 7fceb5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:56:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.092+0000 7fceb5ffb640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.26:6790/0 conn(0x7fce98007360 legacy=0x7fce980097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.092+0000 7fceb5ffb640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.189:6789/0 conn(0x7fceb80a1c00 legacy=0x7fce9800d0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:56:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.092+0000 7fceb5ffb640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.26:6789/0 conn(0x7fce980036d0 legacy=0x7fce98005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.092+0000 7fceb5ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:56:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.092+0000 7fceb5ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fce98010b40 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.092+0000 7fceb5ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fce9800d810 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.092+0000 7fceb5ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fce9800dbf0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.092+0000 7fceb5ffb640 1 -- 172.21.15.26:0/3721140279 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce98011ac0 con 0x7fce9800dbf0 2024-09-16T14:56:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.092+0000 7fceb5ffb640 1 -- 172.21.15.26:0/3721140279 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb80997b0 con 0x7fce98010b40 2024-09-16T14:56:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.092+0000 7fceb5ffb640 1 -- 172.21.15.26:0/3721140279 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce98018e10 con 0x7fce9800d810 2024-09-16T14:56:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.092+0000 7fceb5ffb640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:51.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.092+0000 7fceb7fff640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.189:6789/0 conn(0x7fce98010b40 legacy=0x7fce98010f80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:51.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.296+0000 7fceb7fff640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.189:6789/0 conn(0x7fce98010b40 legacy=0x7fce98010f80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:51.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:51.700+0000 7fceb7fff640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.189:6789/0 conn(0x7fce98010b40 legacy=0x7fce98010f80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:52.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 1 -- 172.21.15.26:0/3721140279 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068779414 0 0) 0x7fce98011ac0 con 0x7fce9800dbf0 2024-09-16T14:56:52.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:56:52.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:52.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:52.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 10 monclient(hunting): my global_id is 274311 2024-09-16T14:56:52.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 1 -- 172.21.15.26:0/3721140279 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea4003530 con 0x7fce9800dbf0 2024-09-16T14:56:52.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 1 -- 172.21.15.26:0/3721140279 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2985425682 0 0) 0x7fcea4003530 con 0x7fce9800dbf0 2024-09-16T14:56:52.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.26:6790/0 conn(0x7fce9800d810 legacy=0x7fce9800d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.189:6789/0 conn(0x7fce98010b40 legacy=0x7fce98010f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:56:52.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:56:52.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 1 monclient: found mon.noname-a 2024-09-16T14:56:52.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:52.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:56:52.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 1 -- 172.21.15.26:0/3721140279 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce980062f0 con 0x7fce9800dbf0 2024-09-16T14:56:52.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 10 monclient: _finish_auth 0 2024-09-16T14:56:52.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:56:52.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:52.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 1 -- 172.21.15.26:0/3721140279 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fceb00015f0 con 0x7fce9800dbf0 2024-09-16T14:56:52.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:52.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 10 monclient: dump: 2024-09-16T14:56:52.295 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.295 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.295 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.295 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.295 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.295 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.295 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.295 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.295 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.295 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fcebe31c640 5 monclient: authenticate success, global_id 274311 2024-09-16T14:56:52.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fcebe31c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:52.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 1 -- 172.21.15.26:0/3721140279 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fceb0001850 con 0x7fce9800dbf0 2024-09-16T14:56:52.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:52.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fcebe31c640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:52.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fcebe31c640 4 set_mon_vals no callback set 2024-09-16T14:56:52.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 1 -- 172.21.15.26:0/3721140279 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fceb0001c20 con 0x7fce9800dbf0 2024-09-16T14:56:52.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:52.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fceb6ffd640 10 monclient: dump: 2024-09-16T14:56:52.296 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.296 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.296 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.297 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.297 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.297 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.297 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.297 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.297 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.297 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fcebe31c640 10 monclient: shutdown 2024-09-16T14:56:52.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fcebe31c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:52.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fcebe31c640 1 -- 172.21.15.26:0/3721140279 >> v1:172.21.15.26:6789/0 conn(0x7fce9800dbf0 legacy=0x7fce9800e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fcebe31c640 1 -- 172.21.15.26:0/3721140279 shutdown_connections 2024-09-16T14:56:52.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fcebe31c640 1 -- 172.21.15.26:0/3721140279 >> 172.21.15.26:0/3721140279 conn(0x7fceb80a3d00 msgr2=0x7fceb80a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:52.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fcebe31c640 1 -- 172.21.15.26:0/3721140279 shutdown_connections 2024-09-16T14:56:52.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.292+0000 7fcebe31c640 1 -- 172.21.15.26:0/3721140279 wait complete. 2024-09-16T14:56:52.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 10 monclient: build_initial_monmap 2024-09-16T14:56:52.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 10 monclient: monmap: 2024-09-16T14:56:52.302 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:52.302 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:52.302 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:52.302 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:52.302 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:52.302 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:52.302 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:52.302 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:52.302 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:52.303 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 1 Processor -- start 2024-09-16T14:56:52.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 1 -- start start 2024-09-16T14:56:52.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 10 monclient: init 2024-09-16T14:56:52.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:52.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:56:52.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 10 monclient(hunting): picked mon.noname-b con 0x7fce9800d810 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:52.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 10 monclient(hunting): picked mon.noname-a con 0x7fce9800dbf0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:52.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 10 monclient(hunting): picked mon.noname-c con 0x7fce98010b40 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:52.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb80a1520 con 0x7fce9800dbf0 2024-09-16T14:56:52.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb80a0fd0 con 0x7fce9800d810 2024-09-16T14:56:52.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb81a7300 con 0x7fce98010b40 2024-09-16T14:56:52.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:52.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 10 monclient(hunting): authenticate will time out at 1836.418123s 2024-09-16T14:56:52.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb7fff640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fce9800d810 legacy=0x7fceb81b9c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:52.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb77fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fce9800dbf0 0x7fceb81ce640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39852/0 (socket says 172.21.15.26:39852) 2024-09-16T14:56:52.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb77fe640 1 -- 172.21.15.26:0/4010805606 learned_addr learned my addr 172.21.15.26:0/4010805606 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:52.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 1 -- 172.21.15.26:0/4010805606 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149608778 0 0) 0x7fceb80a1520 con 0x7fce9800dbf0 2024-09-16T14:56:52.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:56:52.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:52.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:52.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 10 monclient(hunting): my global_id is 274314 2024-09-16T14:56:52.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 1 -- 172.21.15.26:0/4010805606 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce98010f20 con 0x7fce9800dbf0 2024-09-16T14:56:52.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 1 -- 172.21.15.26:0/4010805606 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2118565192 0 0) 0x7fce98010f20 con 0x7fce9800dbf0 2024-09-16T14:56:52.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 1 -- 172.21.15.26:0/4010805606 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fceb80a1520 con 0x7fce9800dbf0 2024-09-16T14:56:52.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 1 -- 172.21.15.26:0/4010805606 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcea0001ab0 con 0x7fce9800dbf0 2024-09-16T14:56:52.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:52.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 10 monclient(hunting): dump: 2024-09-16T14:56:52.305 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.305 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.305 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.305 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.305 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.305 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.305 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.305 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.305 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.306 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:52.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 1 -- 172.21.15.26:0/4010805606 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2271131102 0 0) 0x7fceb80a1520 con 0x7fce9800dbf0 2024-09-16T14:56:52.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 1 -- 172.21.15.26:0/4010805606 >> v1:172.21.15.26:6790/0 conn(0x7fce98010b40 legacy=0x7fceb81d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 1 -- 172.21.15.26:0/4010805606 >> v1:172.21.15.189:6789/0 conn(0x7fce9800d810 legacy=0x7fceb81b9c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:56:52.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:56:52.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 1 monclient: found mon.a 2024-09-16T14:56:52.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:52.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:52.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 1 -- 172.21.15.26:0/4010805606 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb809e3f0 con 0x7fce9800dbf0 2024-09-16T14:56:52.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 10 monclient: _finish_auth 0 2024-09-16T14:56:52.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:56:52.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:52.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 5 monclient: authenticate success, global_id 274314 2024-09-16T14:56:52.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 10 monclient: _renew_subs 2024-09-16T14:56:52.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:52.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 1 -- 172.21.15.26:0/4010805606 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb81a75e0 con 0x7fce9800dbf0 2024-09-16T14:56:52.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 10 monclient: _renew_subs 2024-09-16T14:56:52.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:52.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fcebe31c640 1 -- 172.21.15.26:0/4010805606 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb81ba330 con 0x7fce9800dbf0 2024-09-16T14:56:52.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 1 -- 172.21.15.26:0/4010805606 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcea00020f0 con 0x7fce9800dbf0 2024-09-16T14:56:52.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:52.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 1 -- 172.21.15.26:0/4010805606 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fcea0003b20 con 0x7fce9800dbf0 2024-09-16T14:56:52.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:52.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb67fc640 4 set_mon_vals no callback set 2024-09-16T14:56:52.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.296+0000 7fceb4ff9640 10 monclient: dump: 2024-09-16T14:56:52.308 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.308 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.308 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.308 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.308 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.308 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.308 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.308 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.308 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.308 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.300+0000 7fcebe31c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:52.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.300+0000 7fcebe31c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:52.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.300+0000 7fcebe31c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:52.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.300+0000 7fcebe31c640 1 -- 172.21.15.26:0/4010805606 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce9c005050 con 0x7fce9800dbf0 2024-09-16T14:56:52.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.300+0000 7fceb4ff9640 1 -- 172.21.15.26:0/4010805606 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fcea0003c80 con 0x7fce9800dbf0 2024-09-16T14:56:52.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.300+0000 7fceb4ff9640 1 -- 172.21.15.26:0/4010805606 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fcea0043f90 con 0x7fce9800dbf0 2024-09-16T14:56:52.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.300+0000 7fceb7fff640 10 monclient: get_auth_request con 0x7fce98037f50 auth_method 0 2024-09-16T14:56:52.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.304+0000 7fceb4ff9640 1 -- 172.21.15.26:0/4010805606 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fcea00146a0 con 0x7fce9800dbf0 2024-09-16T14:56:52.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.304+0000 7fceb4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:52.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.304+0000 7fceb4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:52.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.528+0000 7fcebe31c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:56:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.528+0000 7fcebe31c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:56:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.528+0000 7fcebe31c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:56:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.528+0000 7fcebe31c640 1 -- 172.21.15.26:0/4010805606 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fce9c005880 con 0x7fce9800dbf0 2024-09-16T14:56:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.528+0000 7fceb4ff9640 1 -- 172.21.15.26:0/4010805606 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 100909506) 0x7fcea0012d40 con 0x7fce9800dbf0 2024-09-16T14:56:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.528+0000 7fceb4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:56:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.528+0000 7fceb4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:52.529 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":136,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:56:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.528+0000 7fcebe31c640 1 -- 172.21.15.26:0/4010805606 >> v1:172.21.15.189:6812/3196161582 conn(0x7fce98037f50 legacy=0x7fce9803a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.528+0000 7fcebe31c640 10 monclient: shutdown 2024-09-16T14:56:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.528+0000 7fcebe31c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.528+0000 7fcebe31c640 1 -- 172.21.15.26:0/4010805606 >> v1:172.21.15.26:6789/0 conn(0x7fce9800dbf0 legacy=0x7fceb81ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.528+0000 7fcebe31c640 1 -- 172.21.15.26:0/4010805606 shutdown_connections 2024-09-16T14:56:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.528+0000 7fcebe31c640 1 -- 172.21.15.26:0/4010805606 >> 172.21.15.26:0/4010805606 conn(0x7fceb80a3d00 msgr2=0x7fceb809ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.528+0000 7fcebe31c640 1 -- 172.21.15.26:0/4010805606 shutdown_connections 2024-09-16T14:56:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.528+0000 7fcebe31c640 1 -- 172.21.15.26:0/4010805606 wait complete. 2024-09-16T14:56:52.546 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:56:52.546 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T14:56:52.546 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 10 monclient: get_monmap_and_config 2024-09-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 10 monclient: build_initial_monmap 2024-09-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 10 monclient: monmap: 2024-09-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:52.669938+0000 2024-09-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:52.669938+0000 2024-09-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:52.669 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:52.669 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 1 Processor -- start 2024-09-16T14:56:52.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 1 -- start start 2024-09-16T14:56:52.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 10 monclient: init 2024-09-16T14:56:52.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:52.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:56:52.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 10 monclient(hunting): picked mon.noname-c con 0x7f624009a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:52.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 10 monclient(hunting): picked mon.noname-a con 0x7f624009d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:52.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 10 monclient(hunting): picked mon.noname-b con 0x7f624009b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:52.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f624009dc60 con 0x7f624009d1b0 2024-09-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6240099ae0 con 0x7f624009a550 2024-09-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f624009b8f0 con 0x7f624009b040 2024-09-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 10 monclient(hunting): authenticate will time out at 1836.790076s 2024-09-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6245ff0640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f624009b040 0x7f624013a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57070/0 (socket says 172.21.15.26:57070) 2024-09-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6245ff0640 1 -- 172.21.15.26:0/3832613173 learned_addr learned my addr 172.21.15.26:0/3832613173 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6244fee640 1 --1- 172.21.15.26:0/3832613173 >> v1:172.21.15.26:6789/0 conn(0x7f624009d1b0 0x7f6240136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39876/0 (socket says 172.21.15.26:39876) 2024-09-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f62457ef640 1 -- 172.21.15.26:0/3832613173 >> v1:172.21.15.189:6789/0 conn(0x7f624009a550 legacy=0x7f624009a930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 1 -- 172.21.15.26:0/3832613173 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945688521 0 0) 0x7f624009dc60 con 0x7f624009d1b0 2024-09-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:52.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient(hunting): my global_id is 274317 2024-09-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 1 -- 172.21.15.26:0/3832613173 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f622c003530 con 0x7f624009d1b0 2024-09-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 1 -- 172.21.15.26:0/3832613173 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913283159 0 0) 0x7f624009b8f0 con 0x7f624009b040 2024-09-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient(hunting): _negotiate 2024-09-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient(hunting): my global_id is 284219 2024-09-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 1 -- 172.21.15.26:0/3832613173 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f624009dc60 con 0x7f624009b040 2024-09-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 1 -- 172.21.15.26:0/3832613173 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2487264496 0 0) 0x7f622c003530 con 0x7f624009d1b0 2024-09-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 1 -- 172.21.15.26:0/3832613173 >> v1:172.21.15.26:6790/0 conn(0x7f624009b040 legacy=0x7f624013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 1 -- 172.21.15.26:0/3832613173 >> v1:172.21.15.189:6789/0 conn(0x7f624009a550 legacy=0x7f624009a930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient: _finish_hunting 0 2024-09-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 1 monclient: found mon.noname-a 2024-09-16T14:56:52.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:52.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:56:52.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 1 -- 172.21.15.26:0/3832613173 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624009bbd0 con 0x7f624009d1b0 2024-09-16T14:56:52.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient: _finish_auth 0 2024-09-16T14:56:52.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient: _check_auth_tickets 2024-09-16T14:56:52.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:52.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 1 -- 172.21.15.26:0/3832613173 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6230001550 con 0x7f624009d1b0 2024-09-16T14:56:52.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:52.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient: dump: 2024-09-16T14:56:52.675 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.675 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.675 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 5 monclient: authenticate success, global_id 274317 2024-09-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 1 -- 172.21.15.26:0/3832613173 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62300017b0 con 0x7f624009d1b0 2024-09-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 1 -- 172.21.15.26:0/3832613173 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6230001b80 con 0x7f624009d1b0 2024-09-16T14:56:52.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6237fff640 10 monclient: dump: 2024-09-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 4 set_mon_vals no callback set 2024-09-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 10 monclient: shutdown 2024-09-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 1 -- 172.21.15.26:0/3832613173 >> v1:172.21.15.26:6789/0 conn(0x7f624009d1b0 legacy=0x7f6240136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 1 -- 172.21.15.26:0/3832613173 shutdown_connections 2024-09-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 1 -- 172.21.15.26:0/3832613173 >> 172.21.15.26:0/3832613173 conn(0x7f624009fb50 msgr2=0x7f624009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 1 -- 172.21.15.26:0/3832613173 shutdown_connections 2024-09-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.668+0000 7f6247a53640 1 -- 172.21.15.26:0/3832613173 wait complete. 2024-09-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 10 monclient: build_initial_monmap 2024-09-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 10 monclient: monmap: 2024-09-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:52.678 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:52.679 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 1 Processor -- start 2024-09-16T14:56:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 1 -- start start 2024-09-16T14:56:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 10 monclient: init 2024-09-16T14:56:52.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 10 monclient(hunting): picked mon.noname-a con 0x7f624009a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 10 monclient(hunting): picked mon.noname-c con 0x7f624009b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 10 monclient(hunting): picked mon.noname-b con 0x7f624009d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f624009a240 con 0x7f624009a550 2024-09-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62400a24c0 con 0x7f624009d1b0 2024-09-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62401c6830 con 0x7f624009b040 2024-09-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 10 monclient(hunting): authenticate will time out at 1836.793772s 2024-09-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f62457ef640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f624009a550 0x7f62401b7cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39884/0 (socket says 172.21.15.26:39884) 2024-09-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f62457ef640 1 -- 172.21.15.26:0/3262731019 learned_addr learned my addr 172.21.15.26:0/3262731019 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6245ff0640 1 -- 172.21.15.26:0/3262731019 >> v1:172.21.15.189:6789/0 conn(0x7f624009d1b0 legacy=0x7f62401d18f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883420080 0 0) 0x7f62401c6830 con 0x7f624009b040 2024-09-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient(hunting): my global_id is 284222 2024-09-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6224003530 con 0x7f624009b040 2024-09-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917844308 0 0) 0x7f624009a240 con 0x7f624009a550 2024-09-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient(hunting): my global_id is 274320 2024-09-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62401c6830 con 0x7f624009a550 2024-09-16T14:56:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2842955436 0 0) 0x7f6224003530 con 0x7f624009b040 2024-09-16T14:56:52.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f624009a240 con 0x7f624009b040 2024-09-16T14:56:52.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6230001ac0 con 0x7f624009b040 2024-09-16T14:56:52.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:56:52.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient(hunting): dump: 2024-09-16T14:56:52.686 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.686 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.686 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.686 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.686 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.686 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2186842809 0 0) 0x7f62401c6830 con 0x7f624009a550 2024-09-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6224003530 con 0x7f624009a550 2024-09-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6238004560 con 0x7f624009a550 2024-09-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient(hunting): dump: 2024-09-16T14:56:52.687 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.688 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.688 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.688 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.688 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.688 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.688 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.688 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.688 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.688 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2270622519 0 0) 0x7f624009a240 con 0x7f624009b040 2024-09-16T14:56:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 >> v1:172.21.15.189:6789/0 conn(0x7f624009d1b0 legacy=0x7f62401d18f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:56:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 >> v1:172.21.15.26:6789/0 conn(0x7f624009a550 legacy=0x7f62401b7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:56:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 monclient: found mon.c 2024-09-16T14:56:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62401a8360 con 0x7f624009b040 2024-09-16T14:56:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient: _finish_auth 0 2024-09-16T14:56:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 5 monclient: authenticate success, global_id 284222 2024-09-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 10 monclient: _renew_subs 2024-09-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 1 -- 172.21.15.26:0/3262731019 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62401b8420 con 0x7f624009b040 2024-09-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 10 monclient: _renew_subs 2024-09-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 1 -- 172.21.15.26:0/3262731019 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62401a9e20 con 0x7f624009b040 2024-09-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6230004aa0 con 0x7f624009b040 2024-09-16T14:56:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6230004c10 con 0x7f624009b040 2024-09-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 10 monclient: dump: 2024-09-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f62377fe640 4 set_mon_vals no callback set 2024-09-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6247a53640 1 -- 172.21.15.26:0/3262731019 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6220006590 con 0x7f6220002830 2024-09-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f623000f680 con 0x7f624009b040 2024-09-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.672+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f6230043d60 con 0x7f624009b040 2024-09-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337744932 0 0) 0x7f6220006590 con 0x7f6220002830 2024-09-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 10 monclient(hunting): my global_id is 274323 2024-09-16T14:56:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f624009a240 con 0x7f6220002830 2024-09-16T14:56:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2478710346 0 0) 0x7f624009a240 con 0x7f6220002830 2024-09-16T14:56:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6220006590 con 0x7f6220002830 2024-09-16T14:56:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6238004ba0 con 0x7f6220002830 2024-09-16T14:56:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6245ff0640 10 monclient: get_auth_request con 0x7f62240361d0 auth_method 0 2024-09-16T14:56:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 10 monclient: dump: 2024-09-16T14:56:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 188026768 0 0) 0x7f6220006590 con 0x7f6220002830 2024-09-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f622000a0e0 con 0x7f6220002830 2024-09-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6238006e90 con 0x7f6220002830 2024-09-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 >> v1:172.21.15.26:6789/0 conn(0x7f6220002830 legacy=0x7f6220004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.676+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 >> v1:172.21.15.26:6789/0 conn(0x7f6220002830 legacy=0x7f6220004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6247a53640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:56:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6247a53640 1 -- 172.21.15.26:0/3262731019 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6220009da0 con 0x7f62200090c0 2024-09-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856816570 0 0) 0x7f6220009da0 con 0x7f62200090c0 2024-09-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 10 monclient(hunting): my global_id is 274326 2024-09-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6220006590 con 0x7f62200090c0 2024-09-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 592921242 0 0) 0x7f6220006590 con 0x7f62200090c0 2024-09-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6220009da0 con 0x7f62200090c0 2024-09-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6238005330 con 0x7f62200090c0 2024-09-16T14:56:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 10 monclient: dump: 2024-09-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.697 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3191843190 0 0) 0x7f6220009da0 con 0x7f62200090c0 2024-09-16T14:56:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f622000c290 con 0x7f62200090c0 2024-09-16T14:56:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4201 (unknown 0 0 2456166631) 0x7f6238006820 con 0x7f62200090c0 2024-09-16T14:56:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:56:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 >> v1:172.21.15.26:6789/0 conn(0x7f62200090c0 legacy=0x7f6220009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.680+0000 7f6235ffb640 1 -- 172.21.15.26:0/3262731019 >> v1:172.21.15.26:6789/0 conn(0x7f62200090c0 legacy=0x7f6220009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:52.698 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:56:52.698 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:56:52.698 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:56:52.698 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:56:52.698 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 136, 2024-09-16T14:56:52.698 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:56:52.698 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:56:52.698 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:56:52.698 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:52.698 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:52.699 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:52.700 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:56:52.701 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:52.702 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.703 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.704 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:56:52.705 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:56:52.705 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:52.705 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:52.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.705 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:52.705 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:56:52.705 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:52.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:52.705 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:52.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.705 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:56:52.705 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:56:52.706 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:52.706 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:52.706 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:52.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.706 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:56:52.706 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:56:52.706 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:52.706 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:52.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.706 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:52.706 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:56:52.706 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:56:52.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:52.708 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:52.709 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:52.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.710 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:56:52.710 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:56:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.684+0000 7f6247a53640 1 -- 172.21.15.26:0/3262731019 >> v1:172.21.15.189:6812/3196161582 conn(0x7f62240361d0 legacy=0x7f6224038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.684+0000 7f6247a53640 10 monclient: shutdown 2024-09-16T14:56:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.684+0000 7f6247a53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.684+0000 7f6247a53640 1 -- 172.21.15.26:0/3262731019 >> v1:172.21.15.26:6790/0 conn(0x7f624009b040 legacy=0x7f62401ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.684+0000 7f6245ff0640 1 -- 172.21.15.26:0/3262731019 reap_dead start 2024-09-16T14:56:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.684+0000 7f6245ff0640 1 -- 172.21.15.26:0/3262731019 reap_dead start 2024-09-16T14:56:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.684+0000 7f6247a53640 1 -- 172.21.15.26:0/3262731019 shutdown_connections 2024-09-16T14:56:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.684+0000 7f6247a53640 1 -- 172.21.15.26:0/3262731019 >> 172.21.15.26:0/3262731019 conn(0x7f624009fb50 msgr2=0x7f62400a10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.684+0000 7f6247a53640 1 -- 172.21.15.26:0/3262731019 shutdown_connections 2024-09-16T14:56:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.684+0000 7f6247a53640 1 -- 172.21.15.26:0/3262731019 wait complete. 2024-09-16T14:56:52.712 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:56:52.712 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:56:52.712 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:56:52.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.824+0000 7f0e4be23640 10 monclient: get_monmap_and_config 2024-09-16T14:56:52.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.824+0000 7f0e4be23640 10 monclient: build_initial_monmap 2024-09-16T14:56:52.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.824+0000 7f0e4be23640 10 monclient: monmap: 2024-09-16T14:56:52.827 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:52.828 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.828 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:56:52.828926+0000 2024-09-16T14:56:52.828 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:56:52.828926+0000 2024-09-16T14:56:52.828 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:52.828 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.829 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:52.829 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:56:52.829 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:56:52.829 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 1 Processor -- start 2024-09-16T14:56:52.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 1 -- start start 2024-09-16T14:56:52.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient: init 2024-09-16T14:56:52.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:52.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:56:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient(hunting): picked mon.noname-b con 0x7f0e4409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient(hunting): picked mon.noname-c con 0x7f0e44132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient(hunting): picked mon.noname-a con 0x7f0e44138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e4409d260 con 0x7f0e44138ff0 2024-09-16T14:56:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e44132a80 con 0x7f0e44132650 2024-09-16T14:56:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e4409cf60 con 0x7f0e4409cb80 2024-09-16T14:56:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient(hunting): authenticate will time out at 1836.948782s 2024-09-16T14:56:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e49bbf640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0e4409cb80 0x7f0e440717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57090/0 (socket says 172.21.15.26:57090) 2024-09-16T14:56:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e49bbf640 1 -- 172.21.15.26:0/3932562796 learned_addr learned my addr 172.21.15.26:0/3932562796 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4a3c0640 1 --1- 172.21.15.26:0/3932562796 >> v1:172.21.15.26:6789/0 conn(0x7f0e44138ff0 0x7f0e4413b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39902/0 (socket says 172.21.15.26:39902) 2024-09-16T14:56:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e493be640 1 -- 172.21.15.26:0/3932562796 >> v1:172.21.15.189:6789/0 conn(0x7f0e44132650 legacy=0x7f0e441378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:52.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 1 -- 172.21.15.26:0/3932562796 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849857510 0 0) 0x7f0e4409d260 con 0x7f0e44138ff0 2024-09-16T14:56:52.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient(hunting): _negotiate 2024-09-16T14:56:52.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:52.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:52.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient(hunting): my global_id is 274329 2024-09-16T14:56:52.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 1 -- 172.21.15.26:0/3932562796 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e30003530 con 0x7f0e44138ff0 2024-09-16T14:56:52.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 1 -- 172.21.15.26:0/3932562796 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028382561 0 0) 0x7f0e4409cf60 con 0x7f0e4409cb80 2024-09-16T14:56:52.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient(hunting): _negotiate 2024-09-16T14:56:52.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:52.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:52.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient(hunting): my global_id is 284225 2024-09-16T14:56:52.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 1 -- 172.21.15.26:0/3932562796 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e4409d260 con 0x7f0e4409cb80 2024-09-16T14:56:52.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 1 -- 172.21.15.26:0/3932562796 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2310472221 0 0) 0x7f0e30003530 con 0x7f0e44138ff0 2024-09-16T14:56:52.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 1 -- 172.21.15.26:0/3932562796 >> v1:172.21.15.26:6790/0 conn(0x7f0e4409cb80 legacy=0x7f0e440717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 1 -- 172.21.15.26:0/3932562796 >> v1:172.21.15.189:6789/0 conn(0x7f0e44132650 legacy=0x7f0e441378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:56:52.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient: _finish_hunting 0 2024-09-16T14:56:52.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 1 monclient: found mon.noname-a 2024-09-16T14:56:52.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:52.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:56:52.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 1 -- 172.21.15.26:0/3932562796 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4409f2d0 con 0x7f0e44138ff0 2024-09-16T14:56:52.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient: _finish_auth 0 2024-09-16T14:56:52.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient: _check_auth_tickets 2024-09-16T14:56:52.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:52.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 1 -- 172.21.15.26:0/3932562796 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0e3c0019a0 con 0x7f0e44138ff0 2024-09-16T14:56:52.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 5 monclient: authenticate success, global_id 274329 2024-09-16T14:56:52.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:52.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:56:52.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient: dump: 2024-09-16T14:56:52.840 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.840 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.840 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.840 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.840 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.840 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.840 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.840 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.840 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.840 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:56:52.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 1 -- 172.21.15.26:0/3932562796 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e3c001c10 con 0x7f0e44138ff0 2024-09-16T14:56:52.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:52.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient: get_monmap_and_config success 2024-09-16T14:56:52.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 4 set_mon_vals no callback set 2024-09-16T14:56:52.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 1 -- 172.21.15.26:0/3932562796 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0e3c001fe0 con 0x7f0e44138ff0 2024-09-16T14:56:52.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:52.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e48bbd640 10 monclient: dump: 2024-09-16T14:56:52.841 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.841 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.841 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.841 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.842 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.842 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.842 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.842 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.842 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.842 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient: shutdown 2024-09-16T14:56:52.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:52.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 1 -- 172.21.15.26:0/3932562796 >> v1:172.21.15.26:6789/0 conn(0x7f0e44138ff0 legacy=0x7f0e4413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 1 -- 172.21.15.26:0/3932562796 shutdown_connections 2024-09-16T14:56:52.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 1 -- 172.21.15.26:0/3932562796 >> 172.21.15.26:0/3932562796 conn(0x7f0e4409fb50 msgr2=0x7f0e4409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:52.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 1 -- 172.21.15.26:0/3932562796 shutdown_connections 2024-09-16T14:56:52.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 1 -- 172.21.15.26:0/3932562796 wait complete. 2024-09-16T14:56:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient: build_initial_monmap 2024-09-16T14:56:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient: monmap: 2024-09-16T14:56:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:56:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:56:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:56:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:56:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:56:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:56:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:56:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:56:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:56:52.843 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 1 Processor -- start 2024-09-16T14:56:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 1 -- start start 2024-09-16T14:56:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient: init 2024-09-16T14:56:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient: _reopen_session rank -1 2024-09-16T14:56:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:56:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient(hunting): picked mon.noname-c con 0x7f0e4409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:56:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient(hunting): picked mon.noname-b con 0x7f0e44132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:56:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient(hunting): picked mon.noname-a con 0x7f0e44138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:56:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e4409c350 con 0x7f0e44138ff0 2024-09-16T14:56:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e4409a240 con 0x7f0e44132650 2024-09-16T14:56:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e4413c600 con 0x7f0e4409cb80 2024-09-16T14:56:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient(hunting): _renew_subs 2024-09-16T14:56:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e4be23640 10 monclient(hunting): authenticate will time out at 1836.951926s 2024-09-16T14:56:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e49bbf640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0e4409cb80 0x7f0e4409a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57106/0 (socket says 172.21.15.26:57106) 2024-09-16T14:56:52.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.828+0000 7f0e49bbf640 1 -- 172.21.15.26:0/819456463 learned_addr learned my addr 172.21.15.26:0/819456463 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:56:52.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e493be640 1 -- 172.21.15.26:0/819456463 >> v1:172.21.15.189:6789/0 conn(0x7f0e44132650 legacy=0x7f0e4409ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:56:52.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496154426 0 0) 0x7f0e4413c600 con 0x7f0e4409cb80 2024-09-16T14:56:52.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:52.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:52.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:52.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): my global_id is 284228 2024-09-16T14:56:52.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e28003560 con 0x7f0e4409cb80 2024-09-16T14:56:52.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229732594 0 0) 0x7f0e4409c350 con 0x7f0e44138ff0 2024-09-16T14:56:52.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:52.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:52.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): my global_id is 274332 2024-09-16T14:56:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e4413c600 con 0x7f0e44138ff0 2024-09-16T14:56:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3711540239 0 0) 0x7f0e28003560 con 0x7f0e4409cb80 2024-09-16T14:56:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e4409c350 con 0x7f0e4409cb80 2024-09-16T14:56:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0e40001820 con 0x7f0e4409cb80 2024-09-16T14:56:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:56:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): dump: 2024-09-16T14:56:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.847 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:56:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3357577208 0 0) 0x7f0e4413c600 con 0x7f0e44138ff0 2024-09-16T14:56:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e28003560 con 0x7f0e44138ff0 2024-09-16T14:56:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0e3c001f20 con 0x7f0e44138ff0 2024-09-16T14:56:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:56:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): dump: 2024-09-16T14:56:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.848 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.848 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.848 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.848 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.848 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.848 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.848 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.848 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1837381844 0 0) 0x7f0e4409c350 con 0x7f0e4409cb80 2024-09-16T14:56:52.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 >> v1:172.21.15.189:6789/0 conn(0x7f0e44132650 legacy=0x7f0e4409ac60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:56:52.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 >> v1:172.21.15.26:6789/0 conn(0x7f0e44138ff0 legacy=0x7f0e441d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:56:52.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 monclient: found mon.c 2024-09-16T14:56:52.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:56:52.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:52.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4406da30 con 0x7f0e4409cb80 2024-09-16T14:56:52.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:56:52.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:56:52.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:56:52.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e4be23640 5 monclient: authenticate success, global_id 284228 2024-09-16T14:56:52.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e4be23640 10 monclient: _renew_subs 2024-09-16T14:56:52.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e4be23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:52.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e4be23640 1 -- 172.21.15.26:0/819456463 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4406dd10 con 0x7f0e4409cb80 2024-09-16T14:56:52.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e4be23640 10 monclient: _renew_subs 2024-09-16T14:56:52.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e4be23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:56:52.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e4be23640 1 -- 172.21.15.26:0/819456463 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e4406d750 con 0x7f0e4409cb80 2024-09-16T14:56:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e400047f0 con 0x7f0e4409cb80 2024-09-16T14:56:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:56:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0e40004b00 con 0x7f0e4409cb80 2024-09-16T14:56:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3bfff640 4 set_mon_vals no callback set 2024-09-16T14:56:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient: dump: 2024-09-16T14:56:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.851 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e4be23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e4be23640 1 -- 172.21.15.26:0/819456463 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0e2c006590 con 0x7f0e2c002830 2024-09-16T14:56:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0e4000f570 con 0x7f0e4409cb80 2024-09-16T14:56:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f0e40043c30 con 0x7f0e4409cb80 2024-09-16T14:56:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775267216 0 0) 0x7f0e2c006590 con 0x7f0e2c002830 2024-09-16T14:56:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient(hunting): my global_id is 284231 2024-09-16T14:56:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e4409c350 con 0x7f0e2c002830 2024-09-16T14:56:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1684575723 0 0) 0x7f0e4409c350 con 0x7f0e2c002830 2024-09-16T14:56:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e2c006590 con 0x7f0e2c002830 2024-09-16T14:56:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0e34003f00 con 0x7f0e2c002830 2024-09-16T14:56:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e4a3c0640 10 monclient: get_auth_request con 0x7f0e280362f0 auth_method 0 2024-09-16T14:56:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient: dump: 2024-09-16T14:56:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.854 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3220240464 0 0) 0x7f0e2c006590 con 0x7f0e2c002830 2024-09-16T14:56:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e2c00a0e0 con 0x7f0e2c002830 2024-09-16T14:56:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0e34006260 con 0x7f0e2c002830 2024-09-16T14:56:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:56:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:56:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 >> v1:172.21.15.26:6790/0 conn(0x7f0e2c002830 legacy=0x7f0e2c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.832+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 >> v1:172.21.15.26:6790/0 conn(0x7f0e2c002830 legacy=0x7f0e2c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:52.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e4be23640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:56:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e4be23640 1 -- 172.21.15.26:0/819456463 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0e2c009d80 con 0x7f0e2c0090a0 2024-09-16T14:56:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997230432 0 0) 0x7f0e2c009d80 con 0x7f0e2c0090a0 2024-09-16T14:56:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:56:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:56:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:56:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 10 monclient(hunting): my global_id is 284234 2024-09-16T14:56:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e2c006590 con 0x7f0e2c0090a0 2024-09-16T14:56:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2001332855 0 0) 0x7f0e2c006590 con 0x7f0e2c0090a0 2024-09-16T14:56:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e2c009d80 con 0x7f0e2c0090a0 2024-09-16T14:56:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0e340047d0 con 0x7f0e2c0090a0 2024-09-16T14:56:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:56:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:56:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 10 monclient: dump: 2024-09-16T14:56:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:56:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:56:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:56:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:56:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:56:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:56:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:56:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:56:52.857 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:56:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2760594378 0 0) 0x7f0e2c009d80 con 0x7f0e2c0090a0 2024-09-16T14:56:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0e2c00c290 con 0x7f0e2c0090a0 2024-09-16T14:56:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3871 (unknown 0 0 3651428177) 0x7f0e34005b90 con 0x7f0e2c0090a0 2024-09-16T14:56:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:56:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:56:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 >> v1:172.21.15.26:6790/0 conn(0x7f0e2c0090a0 legacy=0x7f0e2c009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e3a7fc640 1 -- 172.21.15.26:0/819456463 >> v1:172.21.15.26:6790/0 conn(0x7f0e2c0090a0 legacy=0x7f0e2c009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:56:52.858 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:56:52.858 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:56:52.858 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:56:52.858 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:56:52.859 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 136, 2024-09-16T14:56:52.859 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:56:52.859 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:56:52.859 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:56:52.859 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:52.859 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:56:52.859 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:52.859 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:56:52.859 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:56:52.859 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:52.859 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:52.860 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:52.860 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:52.860 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:52.860 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:52.860 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:52.860 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:52.860 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:52.860 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:52.860 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:52.860 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:56:52.860 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:56:52.860 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:52.860 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:52.861 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:52.861 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:52.861 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:52.861 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:52.861 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:52.861 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:52.861 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:52.861 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:52.861 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:52.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.862 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:56:52.862 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:56:52.862 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:56:52.862 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:56:52.862 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:56:52.862 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:56:52.862 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:52.862 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:56:52.862 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:56:52.862 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:56:52.862 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:56:52.862 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:56:52.862 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:56:52.863 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:56:52.863 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:56:52.863 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:56:52.863 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:56:52.863 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:56:52.863 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:56:52.863 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:56:52.863 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:56:52.863 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:56:52.863 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:56:52.863 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:56:52.863 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:56:52.863 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:56:52.863 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:56:52.864 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:52.864 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:56:52.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.864 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:56:52.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.864 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:56:52.864 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:56:52.864 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:52.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:52.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:52.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:56:52.864 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:52.865 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:52.865 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:52.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:56:52.865 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:56:52.865 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:52.865 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:52.865 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:52.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.865 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:56:52.865 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:56:52.865 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:52.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:52.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:52.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:56:52.866 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:52.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:52.866 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:52.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:56:52.866 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:56:52.866 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:52.866 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:52.866 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:52.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.867 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:56:52.867 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:56:52.867 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:56:52.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:56:52.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:56:52.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:56:52.867 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:56:52.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:52.868 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:52.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.868 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:56:52.868 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:56:52.868 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:56:52.868 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:56:52.868 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:56:52.868 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:52.868 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:52.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.868 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:56:52.868 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:56:52.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.869 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:52.869 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:52.869 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:56:52.869 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:52.869 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:56:52.869 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:56:52.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:56:52.869 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:56:52.869 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:56:52.869 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:56:52.869 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:56:52.869 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:56:52.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:56:52.870 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:56:52.870 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:56:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e4be23640 1 -- 172.21.15.26:0/819456463 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0e280362f0 legacy=0x7f0e280387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e4be23640 10 monclient: shutdown 2024-09-16T14:56:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e4be23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:56:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e4be23640 1 -- 172.21.15.26:0/819456463 >> v1:172.21.15.26:6790/0 conn(0x7f0e4409cb80 legacy=0x7f0e4409a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:56:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e4a3c0640 1 -- 172.21.15.26:0/819456463 reap_dead start 2024-09-16T14:56:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e4a3c0640 1 -- 172.21.15.26:0/819456463 reap_dead start 2024-09-16T14:56:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e4be23640 1 -- 172.21.15.26:0/819456463 shutdown_connections 2024-09-16T14:56:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e4be23640 1 -- 172.21.15.26:0/819456463 >> 172.21.15.26:0/819456463 conn(0x7f0e4409fb50 msgr2=0x7f0e4413b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:56:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e4be23640 1 -- 172.21.15.26:0/819456463 shutdown_connections 2024-09-16T14:56:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:56:52.840+0000 7f0e4be23640 1 -- 172.21.15.26:0/819456463 wait complete. 2024-09-16T14:56:52.872 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:56:52.872 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:57:12.873 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:57:12.873 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T14:57:12.873 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T14:57:12.874 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T14:57:12.877 INFO:tasks.ceph.mon.b:Started 2024-09-16T14:57:12.877 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:57:12.877 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:57:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.008+0000 7fa7b9edb640 10 monclient: get_monmap_and_config 2024-09-16T14:57:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.008+0000 7fa7b9edb640 10 monclient: build_initial_monmap 2024-09-16T14:57:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.008+0000 7fa7b9edb640 10 monclient: monmap: 2024-09-16T14:57:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:13.012306+0000 2024-09-16T14:57:13.011 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:13.012306+0000 2024-09-16T14:57:13.011 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:13.011 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.011 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:13.011 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:13.011 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:13.011 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 1 Processor -- start 2024-09-16T14:57:13.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 1 -- start start 2024-09-16T14:57:13.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 10 monclient: init 2024-09-16T14:57:13.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:13.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:57:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 10 monclient(hunting): picked mon.noname-c con 0x7fa7b4096600 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 10 monclient(hunting): picked mon.noname-a con 0x7fa7b409d580 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 10 monclient(hunting): picked mon.noname-b con 0x7fa7b4137600 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b409bed0 con 0x7fa7b409d580 2024-09-16T14:57:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b409cc00 con 0x7fa7b4096600 2024-09-16T14:57:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b409f7d0 con 0x7fa7b4137600 2024-09-16T14:57:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 10 monclient(hunting): authenticate will time out at 1857.133053s 2024-09-16T14:57:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b2ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa7b409d580 0x7fa7b4135ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39408/0 (socket says 172.21.15.26:39408) 2024-09-16T14:57:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b2ffd640 1 -- 172.21.15.26:0/4169862445 learned_addr learned my addr 172.21.15.26:0/4169862445 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 1 -- 172.21.15.26:0/4169862445 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010290430 0 0) 0x7fa7b409f7d0 con 0x7fa7b4137600 2024-09-16T14:57:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 10 monclient(hunting): my global_id is 284237 2024-09-16T14:57:13.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 1 -- 172.21.15.26:0/4169862445 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa79c003530 con 0x7fa7b4137600 2024-09-16T14:57:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 1 -- 172.21.15.26:0/4169862445 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3585194680 0 0) 0x7fa79c003530 con 0x7fa7b4137600 2024-09-16T14:57:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 1 -- 172.21.15.26:0/4169862445 >> v1:172.21.15.189:6789/0 conn(0x7fa7b4096600 legacy=0x7fa7b40969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 1 -- 172.21.15.26:0/4169862445 >> v1:172.21.15.26:6789/0 conn(0x7fa7b409d580 legacy=0x7fa7b4135ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 10 monclient: _finish_hunting 0 2024-09-16T14:57:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 1 monclient: found mon.noname-b 2024-09-16T14:57:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:57:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 1 -- 172.21.15.26:0/4169862445 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b409c2f0 con 0x7fa7b4137600 2024-09-16T14:57:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 10 monclient: _finish_auth 0 2024-09-16T14:57:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 10 monclient: _check_auth_tickets 2024-09-16T14:57:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 1 -- 172.21.15.26:0/4169862445 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa7ac0019a0 con 0x7fa7b4137600 2024-09-16T14:57:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:57:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 10 monclient: dump: 2024-09-16T14:57:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.021 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 5 monclient: authenticate success, global_id 284237 2024-09-16T14:57:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 1 -- 172.21.15.26:0/4169862445 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7ac001c50 con 0x7fa7b4137600 2024-09-16T14:57:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 1 -- 172.21.15.26:0/4169862445 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa7ac002020 con 0x7fa7b4137600 2024-09-16T14:57:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b27fc640 10 monclient: dump: 2024-09-16T14:57:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.023 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 4 set_mon_vals no callback set 2024-09-16T14:57:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 10 monclient: shutdown 2024-09-16T14:57:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 1 -- 172.21.15.26:0/4169862445 >> v1:172.21.15.26:6790/0 conn(0x7fa7b4137600 legacy=0x7fa7b41399f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 1 -- 172.21.15.26:0/4169862445 shutdown_connections 2024-09-16T14:57:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 1 -- 172.21.15.26:0/4169862445 >> 172.21.15.26:0/4169862445 conn(0x7fa7b40a0360 msgr2=0x7fa7b40a0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 1 -- 172.21.15.26:0/4169862445 shutdown_connections 2024-09-16T14:57:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 1 -- 172.21.15.26:0/4169862445 wait complete. 2024-09-16T14:57:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 10 monclient: build_initial_monmap 2024-09-16T14:57:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 10 monclient: monmap: 2024-09-16T14:57:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:13.025 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 1 Processor -- start 2024-09-16T14:57:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 1 -- start start 2024-09-16T14:57:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.012+0000 7fa7b9edb640 10 monclient: init 2024-09-16T14:57:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:57:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 10 monclient(hunting): picked mon.noname-c con 0x7fa7b4096600 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 10 monclient(hunting): picked mon.noname-b con 0x7fa7b409d580 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 10 monclient(hunting): picked mon.noname-a con 0x7fa7b4137600 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b4136660 con 0x7fa7b4137600 2024-09-16T14:57:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b4132ee0 con 0x7fa7b409d580 2024-09-16T14:57:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b4134370 con 0x7fa7b4096600 2024-09-16T14:57:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 10 monclient(hunting): authenticate will time out at 1857.136169s 2024-09-16T14:57:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b37fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa7b4096600 0x7fa7b4131cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55454/0 (socket says 172.21.15.26:55454) 2024-09-16T14:57:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b37fe640 1 -- 172.21.15.26:0/420756805 learned_addr learned my addr 172.21.15.26:0/420756805 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b3fff640 1 --1- 172.21.15.26:0/420756805 >> v1:172.21.15.26:6789/0 conn(0x7fa7b4137600 0x7fa7b41cfa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39420/0 (socket says 172.21.15.26:39420) 2024-09-16T14:57:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 1 -- 172.21.15.26:0/420756805 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907045858 0 0) 0x7fa7b4134370 con 0x7fa7b4096600 2024-09-16T14:57:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 10 monclient(hunting): my global_id is 284240 2024-09-16T14:57:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 1 -- 172.21.15.26:0/420756805 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a0003530 con 0x7fa7b4096600 2024-09-16T14:57:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 1 -- 172.21.15.26:0/420756805 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4117086260 0 0) 0x7fa7a0003530 con 0x7fa7b4096600 2024-09-16T14:57:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 1 -- 172.21.15.26:0/420756805 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7b4134370 con 0x7fa7b4096600 2024-09-16T14:57:13.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 1 -- 172.21.15.26:0/420756805 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa7a4001920 con 0x7fa7b4096600 2024-09-16T14:57:13.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:13.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 10 monclient(hunting): dump: 2024-09-16T14:57:13.027 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.027 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.027 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.027 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.027 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.027 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.027 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.027 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.027 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.027 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:13.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 1 -- 172.21.15.26:0/420756805 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4032316334 0 0) 0x7fa7b4134370 con 0x7fa7b4096600 2024-09-16T14:57:13.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 1 -- 172.21.15.26:0/420756805 >> v1:172.21.15.189:6789/0 conn(0x7fa7b409d580 legacy=0x7fa7b41323c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 1 -- 172.21.15.26:0/420756805 >> v1:172.21.15.26:6789/0 conn(0x7fa7b4137600 legacy=0x7fa7b41cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:57:13.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 1 monclient: found mon.c 2024-09-16T14:57:13.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:13.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:13.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 1 -- 172.21.15.26:0/420756805 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b4132ad0 con 0x7fa7b4096600 2024-09-16T14:57:13.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 10 monclient: _finish_auth 0 2024-09-16T14:57:13.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:57:13.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:13.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 5 monclient: authenticate success, global_id 284240 2024-09-16T14:57:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 10 monclient: _renew_subs 2024-09-16T14:57:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 1 -- 172.21.15.26:0/420756805 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b41348c0 con 0x7fa7b4096600 2024-09-16T14:57:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 10 monclient: _renew_subs 2024-09-16T14:57:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 1 -- 172.21.15.26:0/420756805 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b40719e0 con 0x7fa7b4096600 2024-09-16T14:57:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 1 -- 172.21.15.26:0/420756805 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7a4003600 con 0x7fa7b4096600 2024-09-16T14:57:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 1 -- 172.21.15.26:0/420756805 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa7a4003990 con 0x7fa7b4096600 2024-09-16T14:57:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b1ffb640 4 set_mon_vals no callback set 2024-09-16T14:57:13.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 10 monclient: dump: 2024-09-16T14:57:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.030 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 1 -- 172.21.15.26:0/420756805 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa7a4003af0 con 0x7fa7b4096600 2024-09-16T14:57:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:13.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b9edb640 1 -- 172.21.15.26:0/420756805 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa794005050 con 0x7fa7b4096600 2024-09-16T14:57:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa79bfff640 1 -- 172.21.15.26:0/420756805 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fa7a4043cc0 con 0x7fa7b4096600 2024-09-16T14:57:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.016+0000 7fa7b2ffd640 10 monclient: get_auth_request con 0x7fa7a0036100 auth_method 0 2024-09-16T14:57:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.020+0000 7fa79bfff640 1 -- 172.21.15.26:0/420756805 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa7a4011040 con 0x7fa7b4096600 2024-09-16T14:57:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.020+0000 7fa79bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.020+0000 7fa79bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:13.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.244+0000 7fa7b9edb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:57:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.244+0000 7fa7b9edb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.244+0000 7fa7b9edb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.244+0000 7fa7b9edb640 1 -- 172.21.15.26:0/420756805 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa794005880 con 0x7fa7b4096600 2024-09-16T14:57:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.244+0000 7fa79bfff640 1 -- 172.21.15.26:0/420756805 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1675980676) 0x7fa7a4014250 con 0x7fa7b4096600 2024-09-16T14:57:13.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.244+0000 7fa79bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:13.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.244+0000 7fa79bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:13.245 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:57:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.244+0000 7fa7b9edb640 1 -- 172.21.15.26:0/420756805 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa7a0036100 legacy=0x7fa7a00385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.244+0000 7fa7b9edb640 10 monclient: shutdown 2024-09-16T14:57:13.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.244+0000 7fa7b9edb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.244+0000 7fa7b9edb640 1 -- 172.21.15.26:0/420756805 >> v1:172.21.15.26:6790/0 conn(0x7fa7b4096600 legacy=0x7fa7b4131cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.244+0000 7fa7b9edb640 1 -- 172.21.15.26:0/420756805 shutdown_connections 2024-09-16T14:57:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.244+0000 7fa7b9edb640 1 -- 172.21.15.26:0/420756805 >> 172.21.15.26:0/420756805 conn(0x7fa7b40a0360 msgr2=0x7fa7b41309d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.244+0000 7fa7b9edb640 1 -- 172.21.15.26:0/420756805 shutdown_connections 2024-09-16T14:57:13.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.244+0000 7fa7b9edb640 1 -- 172.21.15.26:0/420756805 wait complete. 2024-09-16T14:57:13.264 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:57:13.264 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:57:13.264 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:57:13.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6973e2a640 10 monclient: get_monmap_and_config 2024-09-16T14:57:13.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6973e2a640 10 monclient: build_initial_monmap 2024-09-16T14:57:13.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6973e2a640 10 monclient: monmap: 2024-09-16T14:57:13.389 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:13.389 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.389 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:13.390803+0000 2024-09-16T14:57:13.389 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:13.390803+0000 2024-09-16T14:57:13.389 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:13.389 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.389 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:13.389 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:13.389 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:13.389 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6973e2a640 1 Processor -- start 2024-09-16T14:57:13.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6973e2a640 1 -- start start 2024-09-16T14:57:13.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6973e2a640 10 monclient: init 2024-09-16T14:57:13.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6973e2a640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:13.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6973e2a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:57:13.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6973e2a640 10 monclient(hunting): picked mon.noname-a con 0x7f696c09a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:13.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6973e2a640 10 monclient(hunting): picked mon.noname-b con 0x7f696c09d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:13.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6973e2a640 10 monclient(hunting): picked mon.noname-c con 0x7f696c09b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:13.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6973e2a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c09dc60 con 0x7f696c09a550 2024-09-16T14:57:13.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6973e2a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c099ae0 con 0x7f696c09b040 2024-09-16T14:57:13.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6973e2a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c09b8f0 con 0x7f696c09d1b0 2024-09-16T14:57:13.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6973e2a640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:13.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6973e2a640 10 monclient(hunting): authenticate will time out at 1857.511586s 2024-09-16T14:57:13.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f69713c5640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f696c09d1b0 0x7f696c136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55462/0 (socket says 172.21.15.26:55462) 2024-09-16T14:57:13.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f6971bc6640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f696c09a550 0x7f696c09a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39430/0 (socket says 172.21.15.26:39430) 2024-09-16T14:57:13.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.388+0000 7f69713c5640 1 -- 172.21.15.26:0/4251093426 learned_addr learned my addr 172.21.15.26:0/4251093426 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:13.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 1 -- 172.21.15.26:0/4251093426 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257110853 0 0) 0x7f696c09dc60 con 0x7f696c09a550 2024-09-16T14:57:13.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient(hunting): my global_id is 274335 2024-09-16T14:57:13.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 1 -- 172.21.15.26:0/4251093426 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6958003530 con 0x7f696c09a550 2024-09-16T14:57:13.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 1 -- 172.21.15.26:0/4251093426 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440576853 0 0) 0x7f696c09b8f0 con 0x7f696c09d1b0 2024-09-16T14:57:13.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient(hunting): my global_id is 284243 2024-09-16T14:57:13.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 1 -- 172.21.15.26:0/4251093426 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f696c09dc60 con 0x7f696c09d1b0 2024-09-16T14:57:13.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 1 -- 172.21.15.26:0/4251093426 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4215753668 0 0) 0x7f6958003530 con 0x7f696c09a550 2024-09-16T14:57:13.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 1 -- 172.21.15.26:0/4251093426 >> v1:172.21.15.26:6790/0 conn(0x7f696c09d1b0 legacy=0x7f696c136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 1 -- 172.21.15.26:0/4251093426 >> v1:172.21.15.189:6789/0 conn(0x7f696c09b040 legacy=0x7f696c13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient: _finish_hunting 0 2024-09-16T14:57:13.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 1 monclient: found mon.noname-a 2024-09-16T14:57:13.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:13.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:57:13.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 1 -- 172.21.15.26:0/4251093426 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c09bbd0 con 0x7f696c09a550 2024-09-16T14:57:13.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient: _finish_auth 0 2024-09-16T14:57:13.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient: _check_auth_tickets 2024-09-16T14:57:13.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:13.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 1 -- 172.21.15.26:0/4251093426 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6968001620 con 0x7f696c09a550 2024-09-16T14:57:13.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 5 monclient: authenticate success, global_id 274335 2024-09-16T14:57:13.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:13.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:13.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient: dump: 2024-09-16T14:57:13.396 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.396 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.396 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.396 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.396 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.396 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.396 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.396 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.396 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.396 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:13.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 1 -- 172.21.15.26:0/4251093426 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69680018c0 con 0x7f696c09a550 2024-09-16T14:57:13.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:13.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:13.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 4 set_mon_vals no callback set 2024-09-16T14:57:13.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 1 -- 172.21.15.26:0/4251093426 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6968001c50 con 0x7f696c09a550 2024-09-16T14:57:13.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:13.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6970bc4640 10 monclient: dump: 2024-09-16T14:57:13.397 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.397 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.397 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.398 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.398 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.398 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.398 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.398 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.398 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.398 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 10 monclient: shutdown 2024-09-16T14:57:13.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:13.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 1 -- 172.21.15.26:0/4251093426 >> v1:172.21.15.26:6789/0 conn(0x7f696c09a550 legacy=0x7f696c09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 1 -- 172.21.15.26:0/4251093426 shutdown_connections 2024-09-16T14:57:13.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 1 -- 172.21.15.26:0/4251093426 >> 172.21.15.26:0/4251093426 conn(0x7f696c09fb50 msgr2=0x7f696c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:13.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 1 -- 172.21.15.26:0/4251093426 shutdown_connections 2024-09-16T14:57:13.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 1 -- 172.21.15.26:0/4251093426 wait complete. 2024-09-16T14:57:13.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 10 monclient: build_initial_monmap 2024-09-16T14:57:13.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 10 monclient: monmap: 2024-09-16T14:57:13.399 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:13.399 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:13.399 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:13.399 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:13.399 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:13.399 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:13.399 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:13.399 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:13.400 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:13.400 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 1 Processor -- start 2024-09-16T14:57:13.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 1 -- start start 2024-09-16T14:57:13.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 10 monclient: init 2024-09-16T14:57:13.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:13.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:57:13.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 10 monclient(hunting): picked mon.noname-a con 0x7f696c09a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:13.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 10 monclient(hunting): picked mon.noname-b con 0x7f696c09b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:13.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 10 monclient(hunting): picked mon.noname-c con 0x7f696c09d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:13.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c134a50 con 0x7f696c09a550 2024-09-16T14:57:13.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c134c80 con 0x7f696c09b040 2024-09-16T14:57:13.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c11ec60 con 0x7f696c09d1b0 2024-09-16T14:57:13.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:13.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6973e2a640 10 monclient(hunting): authenticate will time out at 1857.514915s 2024-09-16T14:57:13.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6971bc6640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f696c09a550 0x7f696c1c96e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39444/0 (socket says 172.21.15.26:39444) 2024-09-16T14:57:13.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.392+0000 7f6971bc6640 1 -- 172.21.15.26:0/1121542874 learned_addr learned my addr 172.21.15.26:0/1121542874 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:13.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623976954 0 0) 0x7f696c11ec60 con 0x7f696c09d1b0 2024-09-16T14:57:13.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): my global_id is 284246 2024-09-16T14:57:13.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6950003530 con 0x7f696c09d1b0 2024-09-16T14:57:13.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214999743 0 0) 0x7f696c134a50 con 0x7f696c09a550 2024-09-16T14:57:13.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): my global_id is 274338 2024-09-16T14:57:13.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f696c11ec60 con 0x7f696c09a550 2024-09-16T14:57:13.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3963796638 0 0) 0x7f6950003530 con 0x7f696c09d1b0 2024-09-16T14:57:13.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f696c134a50 con 0x7f696c09d1b0 2024-09-16T14:57:13.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6964001b80 con 0x7f696c09d1b0 2024-09-16T14:57:13.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:13.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): dump: 2024-09-16T14:57:13.408 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.408 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.408 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.408 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.408 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.408 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.408 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.408 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.408 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.408 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:13.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 525254824 0 0) 0x7f696c11ec60 con 0x7f696c09a550 2024-09-16T14:57:13.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6950003530 con 0x7f696c09a550 2024-09-16T14:57:13.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6968001bd0 con 0x7f696c09a550 2024-09-16T14:57:13.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:13.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): dump: 2024-09-16T14:57:13.409 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.409 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.409 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.410 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.410 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.410 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.410 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.410 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.410 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.410 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 594360664 0 0) 0x7f696c134a50 con 0x7f696c09d1b0 2024-09-16T14:57:13.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 >> v1:172.21.15.189:6789/0 conn(0x7f696c09b040 legacy=0x7f696c1c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 >> v1:172.21.15.26:6789/0 conn(0x7f696c09a550 legacy=0x7f696c1c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient: _finish_hunting 0 2024-09-16T14:57:13.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 monclient: found mon.c 2024-09-16T14:57:13.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:13.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:13.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c11f190 con 0x7f696c09d1b0 2024-09-16T14:57:13.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient: _finish_auth 0 2024-09-16T14:57:13.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient: _check_auth_tickets 2024-09-16T14:57:13.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:13.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f6973e2a640 5 monclient: authenticate success, global_id 284246 2024-09-16T14:57:13.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f6973e2a640 10 monclient: _renew_subs 2024-09-16T14:57:13.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f6973e2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:13.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f6973e2a640 1 -- 172.21.15.26:0/1121542874 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c1ca500 con 0x7f696c09d1b0 2024-09-16T14:57:13.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69640020f0 con 0x7f696c09d1b0 2024-09-16T14:57:13.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:13.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6964003b20 con 0x7f696c09d1b0 2024-09-16T14:57:13.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:13.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient: dump: 2024-09-16T14:57:13.412 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.412 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.412 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.412 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.412 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.412 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.412 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.412 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.412 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.413 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f6973e2a640 10 monclient: _renew_subs 2024-09-16T14:57:13.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f6963fff640 4 set_mon_vals no callback set 2024-09-16T14:57:13.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f6973e2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:13.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f6973e2a640 1 -- 172.21.15.26:0/1121542874 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c11e8f0 con 0x7f696c09d1b0 2024-09-16T14:57:13.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f6973e2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:13.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6964003d10 con 0x7f696c09d1b0 2024-09-16T14:57:13.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f6973e2a640 1 -- 172.21.15.26:0/1121542874 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f696c02c7d0 con 0x7f696c02c3f0 2024-09-16T14:57:13.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f6964042e00 con 0x7f696c09d1b0 2024-09-16T14:57:13.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126916866 0 0) 0x7f696c02c7d0 con 0x7f696c02c3f0 2024-09-16T14:57:13.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient(hunting): my global_id is 274341 2024-09-16T14:57:13.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f696c134a50 con 0x7f696c02c3f0 2024-09-16T14:57:13.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69713c5640 10 monclient: get_auth_request con 0x7f6950036250 auth_method 0 2024-09-16T14:57:13.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3540328788 0 0) 0x7f696c134a50 con 0x7f696c02c3f0 2024-09-16T14:57:13.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f696c02c7d0 con 0x7f696c02c3f0 2024-09-16T14:57:13.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6968004c20 con 0x7f696c02c3f0 2024-09-16T14:57:13.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:13.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient: dump: 2024-09-16T14:57:13.414 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.414 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.414 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.414 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.414 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.414 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.415 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.415 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.415 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.415 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2387917503 0 0) 0x7f696c02c7d0 con 0x7f696c02c3f0 2024-09-16T14:57:13.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f696c09c220 con 0x7f696c02c3f0 2024-09-16T14:57:13.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6968005010 con 0x7f696c02c3f0 2024-09-16T14:57:13.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:13.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:13.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 >> v1:172.21.15.26:6789/0 conn(0x7f696c02c3f0 legacy=0x7f696c061160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.396+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 >> v1:172.21.15.26:6789/0 conn(0x7f696c02c3f0 legacy=0x7f696c061160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:13.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f6973e2a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:57:13.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f6973e2a640 1 -- 172.21.15.26:0/1121542874 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f696c06fa30 con 0x7f696c06f620 2024-09-16T14:57:13.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853948136 0 0) 0x7f696c06fa30 con 0x7f696c06f620 2024-09-16T14:57:13.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 10 monclient(hunting): my global_id is 274344 2024-09-16T14:57:13.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f696c02c7d0 con 0x7f696c06f620 2024-09-16T14:57:13.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2583539472 0 0) 0x7f696c02c7d0 con 0x7f696c06f620 2024-09-16T14:57:13.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f696c06fa30 con 0x7f696c06f620 2024-09-16T14:57:13.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6968005620 con 0x7f696c06f620 2024-09-16T14:57:13.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:13.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 10 monclient: dump: 2024-09-16T14:57:13.416 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.416 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.417 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.417 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.417 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.417 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.417 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.417 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.417 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.417 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3937397129 0 0) 0x7f696c06fa30 con 0x7f696c06f620 2024-09-16T14:57:13.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f696c139830 con 0x7f696c06f620 2024-09-16T14:57:13.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 2224315859) 0x7f69680069f0 con 0x7f696c06f620 2024-09-16T14:57:13.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:57:13.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:13.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 >> v1:172.21.15.26:6789/0 conn(0x7f696c06f620 legacy=0x7f696c139120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69627fc640 1 -- 172.21.15.26:0/1121542874 >> v1:172.21.15.26:6789/0 conn(0x7f696c06f620 legacy=0x7f696c139120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:13.418 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:57:13.418 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:13.418 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:13.418 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:57:13.418 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 138, 2024-09-16T14:57:13.418 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:57:13.418 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:57:13.419 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:57:13.419 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:57:13.419 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:13.419 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:57:13.419 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:13.419 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:57:13.419 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:57:13.419 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:13.419 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:13.419 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:13.419 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:13.419 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:13.419 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:13.420 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:13.420 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:13.420 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:13.420 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:13.420 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:13.420 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:57:13.420 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:57:13.420 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:13.420 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:13.420 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:13.421 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:13.421 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:13.421 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:13.421 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:13.421 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:13.421 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:13.421 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:13.421 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.421 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:57:13.421 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:57:13.421 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:57:13.421 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:57:13.422 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:57:13.422 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:57:13.422 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:13.422 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:13.422 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:57:13.422 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:57:13.422 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:57:13.422 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:57:13.422 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:57:13.422 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:57:13.422 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:57:13.422 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:13.422 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:57:13.423 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:13.423 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:13.423 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:13.423 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:13.423 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:13.423 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:13.423 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:13.423 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:13.423 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:13.423 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:13.423 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:13.423 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:57:13.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.423 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:57:13.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.424 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:13.424 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:13.424 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:13.424 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:13.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.425 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:13.425 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:57:13.425 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:13.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.425 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.425 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:57:13.426 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:57:13.426 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:13.426 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:13.426 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:13.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.426 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:13.427 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:13.427 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:13.427 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:13.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.427 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:13.427 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:57:13.427 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:13.427 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.428 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.428 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:57:13.428 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:57:13.428 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:13.428 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:13.428 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:13.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.428 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:13.428 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:13.428 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:13.428 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:13.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.429 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:13.429 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:57:13.429 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:13.429 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.429 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.429 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:57:13.429 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:57:13.429 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:13.429 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:13.429 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:13.429 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.429 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.430 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:57:13.430 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:57:13.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.430 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:13.430 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:13.430 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:13.430 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.430 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:13.430 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:57:13.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.430 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:13.430 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:13.431 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:57:13.431 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.431 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:13.431 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:57:13.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.431 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:13.431 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:13.431 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:57:13.431 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.431 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:13.431 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:57:13.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.432 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:13.432 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:13.432 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:13.432 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.432 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.432 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:57:13.432 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:57:13.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f6973e2a640 1 -- 172.21.15.26:0/1121542874 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6950036250 legacy=0x7f6950038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f6973e2a640 10 monclient: shutdown 2024-09-16T14:57:13.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f6973e2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:13.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f6973e2a640 1 -- 172.21.15.26:0/1121542874 >> v1:172.21.15.26:6790/0 conn(0x7f696c09d1b0 legacy=0x7f696c1d1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69723c7640 1 -- 172.21.15.26:0/1121542874 reap_dead start 2024-09-16T14:57:13.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f69723c7640 1 -- 172.21.15.26:0/1121542874 reap_dead start 2024-09-16T14:57:13.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f6973e2a640 1 -- 172.21.15.26:0/1121542874 shutdown_connections 2024-09-16T14:57:13.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f6973e2a640 1 -- 172.21.15.26:0/1121542874 >> 172.21.15.26:0/1121542874 conn(0x7f696c09fb50 msgr2=0x7f696c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:13.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f6973e2a640 1 -- 172.21.15.26:0/1121542874 shutdown_connections 2024-09-16T14:57:13.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.404+0000 7f6973e2a640 1 -- 172.21.15.26:0/1121542874 wait complete. 2024-09-16T14:57:13.437 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:57:13.438 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:57:13.438 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:57:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.552+0000 7f4709f68640 10 monclient: get_monmap_and_config 2024-09-16T14:57:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.552+0000 7f4709f68640 10 monclient: build_initial_monmap 2024-09-16T14:57:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.552+0000 7f4709f68640 10 monclient: monmap: 2024-09-16T14:57:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.555 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:13.557391+0000 2024-09-16T14:57:13.556 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:13.557391+0000 2024-09-16T14:57:13.556 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:13.556 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.556 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:13.556 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:13.556 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:13.556 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 1 Processor -- start 2024-09-16T14:57:13.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 1 -- start start 2024-09-16T14:57:13.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 10 monclient: init 2024-09-16T14:57:13.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:13.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:57:13.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 10 monclient(hunting): picked mon.noname-c con 0x7f4704071930 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:13.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 10 monclient(hunting): picked mon.noname-b con 0x7f470409f340 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:13.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 10 monclient(hunting): picked mon.noname-a con 0x7f470413b3e0 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:13.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47040a1410 con 0x7f470413b3e0 2024-09-16T14:57:13.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47040a1850 con 0x7f4704071930 2024-09-16T14:57:13.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470409c780 con 0x7f470409f340 2024-09-16T14:57:13.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:13.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 10 monclient(hunting): authenticate will time out at 1857.677381s 2024-09-16T14:57:13.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4702ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f470409f340 0x7f4704139cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55476/0 (socket says 172.21.15.26:55476) 2024-09-16T14:57:13.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4703fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f470413b3e0 0x7f470413d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39466/0 (socket says 172.21.15.26:39466) 2024-09-16T14:57:13.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4702ffd640 1 -- 172.21.15.26:0/54436539 learned_addr learned my addr 172.21.15.26:0/54436539 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:13.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 1 -- 172.21.15.26:0/54436539 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047357962 0 0) 0x7f470409c780 con 0x7f470409f340 2024-09-16T14:57:13.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient(hunting): my global_id is 284249 2024-09-16T14:57:13.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 1 -- 172.21.15.26:0/54436539 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46ec003530 con 0x7f470409f340 2024-09-16T14:57:13.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 1 -- 172.21.15.26:0/54436539 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227473707 0 0) 0x7f47040a1410 con 0x7f470413b3e0 2024-09-16T14:57:13.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient(hunting): my global_id is 274347 2024-09-16T14:57:13.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 1 -- 172.21.15.26:0/54436539 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f470409c780 con 0x7f470413b3e0 2024-09-16T14:57:13.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 1 -- 172.21.15.26:0/54436539 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2884830568 0 0) 0x7f46ec003530 con 0x7f470409f340 2024-09-16T14:57:13.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 1 -- 172.21.15.26:0/54436539 >> v1:172.21.15.189:6789/0 conn(0x7f4704071930 legacy=0x7f4704071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 1 -- 172.21.15.26:0/54436539 >> v1:172.21.15.26:6789/0 conn(0x7f470413b3e0 legacy=0x7f470413d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient: _finish_hunting 0 2024-09-16T14:57:13.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 1 monclient: found mon.noname-b 2024-09-16T14:57:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:57:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 1 -- 172.21.15.26:0/54436539 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470409d740 con 0x7f470409f340 2024-09-16T14:57:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient: _finish_auth 0 2024-09-16T14:57:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient: _check_auth_tickets 2024-09-16T14:57:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 1 -- 172.21.15.26:0/54436539 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f46f8001620 con 0x7f470409f340 2024-09-16T14:57:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:57:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient: dump: 2024-09-16T14:57:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.562 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.563 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 5 monclient: authenticate success, global_id 284249 2024-09-16T14:57:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 1 -- 172.21.15.26:0/54436539 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46f80018c0 con 0x7f470409f340 2024-09-16T14:57:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 1 -- 172.21.15.26:0/54436539 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f46f8001c90 con 0x7f470409f340 2024-09-16T14:57:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:13.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f47027fc640 10 monclient: dump: 2024-09-16T14:57:13.564 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.564 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.564 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.564 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.564 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.564 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.564 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.564 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.564 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.564 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:13.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 4 set_mon_vals no callback set 2024-09-16T14:57:13.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 10 monclient: shutdown 2024-09-16T14:57:13.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:13.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 1 -- 172.21.15.26:0/54436539 >> v1:172.21.15.26:6790/0 conn(0x7f470409f340 legacy=0x7f4704139cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 1 -- 172.21.15.26:0/54436539 shutdown_connections 2024-09-16T14:57:13.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 1 -- 172.21.15.26:0/54436539 >> 172.21.15.26:0/54436539 conn(0x7f47040a1be0 msgr2=0x7f47040a1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:13.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 1 -- 172.21.15.26:0/54436539 shutdown_connections 2024-09-16T14:57:13.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 1 -- 172.21.15.26:0/54436539 wait complete. 2024-09-16T14:57:13.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 10 monclient: build_initial_monmap 2024-09-16T14:57:13.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.556+0000 7f4709f68640 10 monclient: monmap: 2024-09-16T14:57:13.565 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:13.565 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:13.565 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:13.565 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:13.565 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:13.565 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:13.566 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:13.566 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:13.566 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:13.566 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 1 Processor -- start 2024-09-16T14:57:13.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 1 -- start start 2024-09-16T14:57:13.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 10 monclient: init 2024-09-16T14:57:13.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:13.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:57:13.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 10 monclient(hunting): picked mon.noname-b con 0x7f4704071930 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:13.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 10 monclient(hunting): picked mon.noname-a con 0x7f470409f340 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:13.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 10 monclient(hunting): picked mon.noname-c con 0x7f470413b3e0 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:13.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4704123de0 con 0x7f470409f340 2024-09-16T14:57:13.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47040a2fe0 con 0x7f4704071930 2024-09-16T14:57:13.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47040a0610 con 0x7f470413b3e0 2024-09-16T14:57:13.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:13.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 10 monclient(hunting): authenticate will time out at 1857.680802s 2024-09-16T14:57:13.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4703fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f470413b3e0 0x7f47041d1ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55478/0 (socket says 172.21.15.26:55478) 2024-09-16T14:57:13.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4702ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f470409f340 0x7f47041ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39480/0 (socket says 172.21.15.26:39480) 2024-09-16T14:57:13.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4703fff640 1 -- 172.21.15.26:0/3795480437 learned_addr learned my addr 172.21.15.26:0/3795480437 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:13.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429646706 0 0) 0x7f47040a0610 con 0x7f470413b3e0 2024-09-16T14:57:13.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): my global_id is 284252 2024-09-16T14:57:13.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f0003560 con 0x7f470413b3e0 2024-09-16T14:57:13.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555227772 0 0) 0x7f4704123de0 con 0x7f470409f340 2024-09-16T14:57:13.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): my global_id is 274350 2024-09-16T14:57:13.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47040a0610 con 0x7f470409f340 2024-09-16T14:57:13.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2041996035 0 0) 0x7f46f0003560 con 0x7f470413b3e0 2024-09-16T14:57:13.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4704123de0 con 0x7f470413b3e0 2024-09-16T14:57:13.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 900706371 0 0) 0x7f47040a0610 con 0x7f470409f340 2024-09-16T14:57:13.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46f0003560 con 0x7f470409f340 2024-09-16T14:57:13.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f46f8001bd0 con 0x7f470409f340 2024-09-16T14:57:13.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:13.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): dump: 2024-09-16T14:57:13.574 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.574 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.574 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.574 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.574 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.574 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.574 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.574 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.574 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.575 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:13.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f46fc001be0 con 0x7f470413b3e0 2024-09-16T14:57:13.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:13.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): dump: 2024-09-16T14:57:13.575 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.575 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.575 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.575 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.575 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.575 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.575 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.576 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480407619 0 0) 0x7f47040a2fe0 con 0x7f4704071930 2024-09-16T14:57:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): my global_id is 294103 2024-09-16T14:57:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47040a0610 con 0x7f4704071930 2024-09-16T14:57:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1882267880 0 0) 0x7f46f0003560 con 0x7f470409f340 2024-09-16T14:57:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 >> v1:172.21.15.26:6790/0 conn(0x7f470413b3e0 legacy=0x7f47041d1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 >> v1:172.21.15.189:6789/0 conn(0x7f4704071930 legacy=0x7f47041c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient: _finish_hunting 0 2024-09-16T14:57:13.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 monclient: found mon.a 2024-09-16T14:57:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4704123210 con 0x7f470409f340 2024-09-16T14:57:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient: _finish_auth 0 2024-09-16T14:57:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient: _check_auth_tickets 2024-09-16T14:57:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 5 monclient: authenticate success, global_id 274350 2024-09-16T14:57:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 10 monclient: _renew_subs 2024-09-16T14:57:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:13.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 1 -- 172.21.15.26:0/3795480437 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4704124fb0 con 0x7f470409f340 2024-09-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 10 monclient: _renew_subs 2024-09-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 1 -- 172.21.15.26:0/3795480437 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4704125570 con 0x7f470409f340 2024-09-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46f8001240 con 0x7f470409f340 2024-09-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f46f8004da0 con 0x7f470409f340 2024-09-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient: dump: 2024-09-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.578 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4701ffb640 4 set_mon_vals no callback set 2024-09-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4709f68640 1 -- 172.21.15.26:0/3795480437 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f46e4006590 con 0x7f46e4002830 2024-09-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f46f800f810 con 0x7f470409f340 2024-09-16T14:57:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f46f8043ed0 con 0x7f470409f340 2024-09-16T14:57:13.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507315109 0 0) 0x7f46e4006590 con 0x7f46e4002830 2024-09-16T14:57:13.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient(hunting): my global_id is 284255 2024-09-16T14:57:13.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f0003560 con 0x7f46e4002830 2024-09-16T14:57:13.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f4703fff640 10 monclient: get_auth_request con 0x7f46f00365b0 auth_method 0 2024-09-16T14:57:13.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 548535580 0 0) 0x7f46f0003560 con 0x7f46e4002830 2024-09-16T14:57:13.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46e4006590 con 0x7f46e4002830 2024-09-16T14:57:13.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f46f40017a0 con 0x7f46e4002830 2024-09-16T14:57:13.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:13.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 10 monclient: dump: 2024-09-16T14:57:13.581 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.581 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.581 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.581 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.581 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.581 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.581 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.581 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.581 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.581 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 641788302 0 0) 0x7f46e4006590 con 0x7f46e4002830 2024-09-16T14:57:13.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.560+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f46e400a0e0 con 0x7f46e4002830 2024-09-16T14:57:13.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.564+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f46f4001c10 con 0x7f46e4002830 2024-09-16T14:57:13.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.564+0000 7f46ebfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:13.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.564+0000 7f46ebfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:13.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.564+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 >> v1:172.21.15.26:6790/0 conn(0x7f46e4002830 legacy=0x7f46e4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.564+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 >> v1:172.21.15.26:6790/0 conn(0x7f46e4002830 legacy=0x7f46e4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:13.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f4709f68640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:57:13.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f4709f68640 1 -- 172.21.15.26:0/3795480437 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f46e4009d80 con 0x7f46e40090a0 2024-09-16T14:57:13.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452025595 0 0) 0x7f46e4009d80 con 0x7f46e40090a0 2024-09-16T14:57:13.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 10 monclient(hunting): my global_id is 284258 2024-09-16T14:57:13.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e4006590 con 0x7f46e40090a0 2024-09-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 646064420 0 0) 0x7f46e4006590 con 0x7f46e40090a0 2024-09-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46e4009d80 con 0x7f46e40090a0 2024-09-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f46f4003a10 con 0x7f46e40090a0 2024-09-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 10 monclient: dump: 2024-09-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.583 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.584 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.584 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.584 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.584 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.584 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.584 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2376699597 0 0) 0x7f46e4009d80 con 0x7f46e40090a0 2024-09-16T14:57:13.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f46e400c290 con 0x7f46e40090a0 2024-09-16T14:57:13.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 1550062895) 0x7f46f4004650 con 0x7f46e40090a0 2024-09-16T14:57:13.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:57:13.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:13.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 >> v1:172.21.15.26:6790/0 conn(0x7f46e40090a0 legacy=0x7f46e4009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f46ebfff640 1 -- 172.21.15.26:0/3795480437 >> v1:172.21.15.26:6790/0 conn(0x7f46e40090a0 legacy=0x7f46e4009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:13.585 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:57:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:13.585 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:57:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 138, 2024-09-16T14:57:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:57:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:57:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:57:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:57:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:57:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:57:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:57:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:13.586 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:57:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:57:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:13.587 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:57:13.588 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:57:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:57:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:57:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:57:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:57:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:57:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:57:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:57:13.589 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:57:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:57:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:57:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:57:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:57:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:13.590 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:57:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:57:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:13.591 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:57:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:57:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:57:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:13.592 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:57:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.593 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:57:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:57:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:13.594 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:57:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:57:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:57:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.595 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:57:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:57:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:57:13.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:57:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.597 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:57:13.597 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:57:13.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f4709f68640 1 -- 172.21.15.26:0/3795480437 >> v1:172.21.15.189:6812/3196161582 conn(0x7f46f00365b0 legacy=0x7f46f0038a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f4709f68640 10 monclient: shutdown 2024-09-16T14:57:13.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f4709f68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:13.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f4709f68640 1 -- 172.21.15.26:0/3795480437 >> v1:172.21.15.26:6789/0 conn(0x7f470409f340 legacy=0x7f47041ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f4703fff640 1 -- 172.21.15.26:0/3795480437 reap_dead start 2024-09-16T14:57:13.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f4703fff640 1 -- 172.21.15.26:0/3795480437 reap_dead start 2024-09-16T14:57:13.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f4709f68640 1 -- 172.21.15.26:0/3795480437 shutdown_connections 2024-09-16T14:57:13.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f4709f68640 1 -- 172.21.15.26:0/3795480437 >> 172.21.15.26:0/3795480437 conn(0x7f47040a1be0 msgr2=0x7f470409b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:13.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f4709f68640 1 -- 172.21.15.26:0/3795480437 shutdown_connections 2024-09-16T14:57:13.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.568+0000 7f4709f68640 1 -- 172.21.15.26:0/3795480437 wait complete. 2024-09-16T14:57:13.599 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:57:13.600 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:57:13.600 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:57:13.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.712+0000 7f6033e73640 10 monclient: get_monmap_and_config 2024-09-16T14:57:13.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.712+0000 7f6033e73640 10 monclient: build_initial_monmap 2024-09-16T14:57:13.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.712+0000 7f6033e73640 10 monclient: monmap: 2024-09-16T14:57:13.716 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:13.716 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.716 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:13.717393+0000 2024-09-16T14:57:13.716 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:13.717393+0000 2024-09-16T14:57:13.716 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:13.716 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.716 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:13.716 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:13.716 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:13.716 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 1 Processor -- start 2024-09-16T14:57:13.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 1 -- start start 2024-09-16T14:57:13.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 10 monclient: init 2024-09-16T14:57:13.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:13.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:57:13.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 10 monclient(hunting): picked mon.noname-a con 0x7f602c09a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:13.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 10 monclient(hunting): picked mon.noname-b con 0x7f602c09d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:13.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 10 monclient(hunting): picked mon.noname-c con 0x7f602c09b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:13.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c09dc60 con 0x7f602c09a550 2024-09-16T14:57:13.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c099ae0 con 0x7f602c09b040 2024-09-16T14:57:13.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c09b8f0 con 0x7f602c09d1b0 2024-09-16T14:57:13.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:13.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 10 monclient(hunting): authenticate will time out at 1857.837379s 2024-09-16T14:57:13.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6031c0f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f602c09a550 0x7f602c09a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39496/0 (socket says 172.21.15.26:39496) 2024-09-16T14:57:13.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6031c0f640 1 -- 172.21.15.26:0/2549721060 learned_addr learned my addr 172.21.15.26:0/2549721060 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:13.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f603140e640 1 --1- 172.21.15.26:0/2549721060 >> v1:172.21.15.26:6790/0 conn(0x7f602c09d1b0 0x7f602c136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55502/0 (socket says 172.21.15.26:55502) 2024-09-16T14:57:13.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 1 -- 172.21.15.26:0/2549721060 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723129368 0 0) 0x7f602c09b8f0 con 0x7f602c09d1b0 2024-09-16T14:57:13.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient(hunting): my global_id is 284261 2024-09-16T14:57:13.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 1 -- 172.21.15.26:0/2549721060 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6018005740 con 0x7f602c09d1b0 2024-09-16T14:57:13.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 1 -- 172.21.15.26:0/2549721060 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296139713 0 0) 0x7f602c09dc60 con 0x7f602c09a550 2024-09-16T14:57:13.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient(hunting): my global_id is 274353 2024-09-16T14:57:13.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 1 -- 172.21.15.26:0/2549721060 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f602c09b8f0 con 0x7f602c09a550 2024-09-16T14:57:13.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 1 -- 172.21.15.26:0/2549721060 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3573654821 0 0) 0x7f6018005740 con 0x7f602c09d1b0 2024-09-16T14:57:13.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 1 -- 172.21.15.26:0/2549721060 >> v1:172.21.15.189:6789/0 conn(0x7f602c09b040 legacy=0x7f602c13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 1 -- 172.21.15.26:0/2549721060 >> v1:172.21.15.26:6789/0 conn(0x7f602c09a550 legacy=0x7f602c09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient: _finish_hunting 0 2024-09-16T14:57:13.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 1 monclient: found mon.noname-b 2024-09-16T14:57:13.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:13.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:57:13.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 1 -- 172.21.15.26:0/2549721060 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c09bbd0 con 0x7f602c09d1b0 2024-09-16T14:57:13.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient: _finish_auth 0 2024-09-16T14:57:13.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient: _check_auth_tickets 2024-09-16T14:57:13.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:13.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 1 -- 172.21.15.26:0/2549721060 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f601c001620 con 0x7f602c09d1b0 2024-09-16T14:57:13.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 5 monclient: authenticate success, global_id 284261 2024-09-16T14:57:13.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:13.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:57:13.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient: dump: 2024-09-16T14:57:13.731 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.731 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.731 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.731 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.731 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.731 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.731 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.731 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.731 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.731 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:13.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 1 -- 172.21.15.26:0/2549721060 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f601c0018c0 con 0x7f602c09d1b0 2024-09-16T14:57:13.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:13.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:13.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 4 set_mon_vals no callback set 2024-09-16T14:57:13.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 1 -- 172.21.15.26:0/2549721060 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f601c001c50 con 0x7f602c09d1b0 2024-09-16T14:57:13.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:13.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6030c0d640 10 monclient: dump: 2024-09-16T14:57:13.732 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.732 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.733 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.733 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.733 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.733 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.733 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.733 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.733 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.733 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 10 monclient: shutdown 2024-09-16T14:57:13.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:13.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 1 -- 172.21.15.26:0/2549721060 >> v1:172.21.15.26:6790/0 conn(0x7f602c09d1b0 legacy=0x7f602c136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 1 -- 172.21.15.26:0/2549721060 shutdown_connections 2024-09-16T14:57:13.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 1 -- 172.21.15.26:0/2549721060 >> 172.21.15.26:0/2549721060 conn(0x7f602c09fb50 msgr2=0x7f602c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:13.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 1 -- 172.21.15.26:0/2549721060 shutdown_connections 2024-09-16T14:57:13.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 1 -- 172.21.15.26:0/2549721060 wait complete. 2024-09-16T14:57:13.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 10 monclient: build_initial_monmap 2024-09-16T14:57:13.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 10 monclient: monmap: 2024-09-16T14:57:13.734 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:13.734 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:13.734 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:13.734 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:13.734 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:13.735 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:13.735 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:13.735 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:13.735 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:13.735 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.716+0000 7f6033e73640 1 Processor -- start 2024-09-16T14:57:13.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 1 -- start start 2024-09-16T14:57:13.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 10 monclient: init 2024-09-16T14:57:13.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:13.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:57:13.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 10 monclient(hunting): picked mon.noname-c con 0x7f602c09a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:13.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 10 monclient(hunting): picked mon.noname-b con 0x7f602c09b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:13.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 10 monclient(hunting): picked mon.noname-a con 0x7f602c09d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:13.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c09a240 con 0x7f602c09d1b0 2024-09-16T14:57:13.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c1a3180 con 0x7f602c09b040 2024-09-16T14:57:13.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c070a80 con 0x7f602c09a550 2024-09-16T14:57:13.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:13.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 10 monclient(hunting): authenticate will time out at 1857.840603s 2024-09-16T14:57:13.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6032410640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f602c09d1b0 0x7f602c1cd860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39510/0 (socket says 172.21.15.26:39510) 2024-09-16T14:57:13.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6032410640 1 -- 172.21.15.26:0/1106232649 learned_addr learned my addr 172.21.15.26:0/1106232649 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:13.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6032410640 1 -- 172.21.15.26:0/1106232649 >> v1:172.21.15.26:6789/0 conn(0x7f602c09d1b0 legacy=0x7f602c1cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T14:57:13.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6032410640 1 -- 172.21.15.26:0/1106232649 >> v1:172.21.15.26:6789/0 conn(0x7f602c09d1b0 legacy=0x7f602c1cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:57:13.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6032410640 1 --1- 172.21.15.26:0/1106232649 >> v1:172.21.15.26:6789/0 conn(0x7f602c09d1b0 0x7f602c1cd860 :-1 s=OPENED pgs=1053 cs=1 l=1).handle_message read tag failed 2024-09-16T14:57:13.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6032410640 1 --1- 172.21.15.26:0/1106232649 >> v1:172.21.15.26:6789/0 conn(0x7f602c09d1b0 0x7f602c1cd860 :-1 s=OPENED pgs=1053 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:57:13.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6789/0 2024-09-16T14:57:13.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028829003 0 0) 0x7f602c070a80 con 0x7f602c09a550 2024-09-16T14:57:13.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): my global_id is 284264 2024-09-16T14:57:13.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6010003530 con 0x7f602c09a550 2024-09-16T14:57:13.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3242972167 0 0) 0x7f6010003530 con 0x7f602c09a550 2024-09-16T14:57:13.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f602c070a80 con 0x7f602c09a550 2024-09-16T14:57:13.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6028001810 con 0x7f602c09a550 2024-09-16T14:57:13.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:13.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): dump: 2024-09-16T14:57:13.740 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.740 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.740 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.740 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.740 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.741 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.741 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.741 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.741 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.741 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:13.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453941763 0 0) 0x7f602c1a3180 con 0x7f602c09b040 2024-09-16T14:57:13.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): my global_id is 294109 2024-09-16T14:57:13.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6010003530 con 0x7f602c09b040 2024-09-16T14:57:13.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 569020788 0 0) 0x7f602c070a80 con 0x7f602c09a550 2024-09-16T14:57:13.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 >> v1:172.21.15.189:6789/0 conn(0x7f602c09b040 legacy=0x7f602c1ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 >> v1:172.21.15.26:6789/0 conn(0x7f602c09d1b0 legacy=0x7f602c1cd860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:13.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient: _finish_hunting 0 2024-09-16T14:57:13.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 monclient: found mon.c 2024-09-16T14:57:13.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:13.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:13.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c0a2850 con 0x7f602c09a550 2024-09-16T14:57:13.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient: _finish_auth 0 2024-09-16T14:57:13.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient: _check_auth_tickets 2024-09-16T14:57:13.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:13.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 5 monclient: authenticate success, global_id 284264 2024-09-16T14:57:13.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 10 monclient: _renew_subs 2024-09-16T14:57:13.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:13.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 1 -- 172.21.15.26:0/1106232649 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f602c1a2ea0 con 0x7f602c09a550 2024-09-16T14:57:13.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 10 monclient: _renew_subs 2024-09-16T14:57:13.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:13.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 1 -- 172.21.15.26:0/1106232649 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f602c1a0e20 con 0x7f602c09a550 2024-09-16T14:57:13.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60280047f0 con 0x7f602c09a550 2024-09-16T14:57:13.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:13.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6028004b80 con 0x7f602c09a550 2024-09-16T14:57:13.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:13.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6023fff640 4 set_mon_vals no callback set 2024-09-16T14:57:13.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient: dump: 2024-09-16T14:57:13.744 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.744 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.744 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.744 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.744 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.744 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.744 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.744 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.744 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.744 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:13.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6033e73640 1 -- 172.21.15.26:0/1106232649 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6014006590 con 0x7f6014002830 2024-09-16T14:57:13.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6028004ce0 con 0x7f602c09a550 2024-09-16T14:57:13.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f6028043ba0 con 0x7f602c09a550 2024-09-16T14:57:13.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f6032410640 10 monclient: get_auth_request con 0x7f6010036130 auth_method 0 2024-09-16T14:57:13.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609827583 0 0) 0x7f6014006590 con 0x7f6014002830 2024-09-16T14:57:13.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient(hunting): my global_id is 294112 2024-09-16T14:57:13.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f602c070a80 con 0x7f6014002830 2024-09-16T14:57:13.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 383481754 0 0) 0x7f602c070a80 con 0x7f6014002830 2024-09-16T14:57:13.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6014006590 con 0x7f6014002830 2024-09-16T14:57:13.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f601c001a30 con 0x7f6014002830 2024-09-16T14:57:13.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:57:13.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.720+0000 7f60227fc640 10 monclient: dump: 2024-09-16T14:57:13.746 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.746 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.746 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.746 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.746 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.746 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.746 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.746 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.746 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.746 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.724+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2709026433 0 0) 0x7f6014006590 con 0x7f6014002830 2024-09-16T14:57:13.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.724+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f601400a0e0 con 0x7f6014002830 2024-09-16T14:57:13.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.724+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f601c001d70 con 0x7f6014002830 2024-09-16T14:57:13.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.724+0000 7f60227fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:13.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.724+0000 7f60227fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:13.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.724+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 >> v1:172.21.15.189:6789/0 conn(0x7f6014002830 legacy=0x7f6014004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.724+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 >> v1:172.21.15.189:6789/0 conn(0x7f6014002830 legacy=0x7f6014004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:13.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.728+0000 7f6033e73640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:57:13.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.728+0000 7f6033e73640 1 -- 172.21.15.26:0/1106232649 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6014009da0 con 0x7f60140090c0 2024-09-16T14:57:13.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.728+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477735245 0 0) 0x7f6014009da0 con 0x7f60140090c0 2024-09-16T14:57:13.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.728+0000 7f60227fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.728+0000 7f60227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.728+0000 7f60227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.728+0000 7f60227fc640 10 monclient(hunting): my global_id is 294115 2024-09-16T14:57:13.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.728+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6014006590 con 0x7f60140090c0 2024-09-16T14:57:13.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 470320119 0 0) 0x7f6014006590 con 0x7f60140090c0 2024-09-16T14:57:13.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6014009da0 con 0x7f60140090c0 2024-09-16T14:57:13.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f601c004ca0 con 0x7f60140090c0 2024-09-16T14:57:13.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f60227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f60227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:57:13.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f60227fc640 10 monclient: dump: 2024-09-16T14:57:13.750 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.750 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.750 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.750 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.750 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.750 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.750 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.750 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.750 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.750 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1492302629 0 0) 0x7f6014009da0 con 0x7f60140090c0 2024-09-16T14:57:13.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f601400c290 con 0x7f60140090c0 2024-09-16T14:57:13.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 768570113) 0x7f601c005f30 con 0x7f60140090c0 2024-09-16T14:57:13.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f60227fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:57:13.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f60227fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:13.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 >> v1:172.21.15.189:6789/0 conn(0x7f60140090c0 legacy=0x7f6014009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f60227fc640 1 -- 172.21.15.26:0/1106232649 >> v1:172.21.15.189:6789/0 conn(0x7f60140090c0 legacy=0x7f6014009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:13.751 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:57:13.751 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:13.751 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:13.751 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:57:13.752 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 138, 2024-09-16T14:57:13.752 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:57:13.752 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:57:13.752 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:57:13.752 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:57:13.752 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:13.752 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:57:13.752 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:13.752 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:57:13.752 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:57:13.752 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:13.752 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:13.752 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:13.753 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:13.753 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:13.753 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:13.753 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:13.753 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:13.753 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:13.753 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:13.753 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:13.753 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:57:13.753 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:57:13.754 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:13.754 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:13.754 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:13.754 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:13.754 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:13.754 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:13.754 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:13.754 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:13.754 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:13.754 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:13.754 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.755 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:57:13.755 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:57:13.755 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:57:13.755 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:57:13.755 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:57:13.755 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:57:13.755 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:13.755 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:13.755 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:57:13.755 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:57:13.755 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:57:13.755 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:57:13.755 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:57:13.755 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:57:13.756 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:57:13.756 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:13.756 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:57:13.756 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:13.756 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:13.756 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:13.756 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:13.756 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:13.756 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:13.756 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:13.756 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:13.756 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:13.757 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:13.757 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:13.757 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:57:13.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.757 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:57:13.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.757 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:13.757 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:13.757 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:13.757 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:13.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.757 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:13.757 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:57:13.758 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:13.758 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.758 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.758 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:57:13.758 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:57:13.758 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:13.758 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:13.758 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:13.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.758 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:13.758 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:57:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:57:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:57:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:13.759 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:13.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.760 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:13.760 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:13.760 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:13.760 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:13.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.760 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:13.760 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:57:13.760 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:13.760 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.760 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.761 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:57:13.761 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:57:13.761 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:13.761 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:13.761 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:13.761 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.761 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.761 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:57:13.761 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:57:13.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.761 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:13.761 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:13.762 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:13.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.762 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:13.762 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:57:13.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:13.762 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:13.762 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:13.762 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:13.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:13.762 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:13.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:13.762 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:57:13.762 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:57:13.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f6033e73640 1 -- 172.21.15.26:0/1106232649 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6010036130 legacy=0x7f60100385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f6033e73640 10 monclient: shutdown 2024-09-16T14:57:13.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f6033e73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:13.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f6033e73640 1 -- 172.21.15.26:0/1106232649 >> v1:172.21.15.26:6790/0 conn(0x7f602c09a550 legacy=0x7f602c1b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f6032410640 1 -- 172.21.15.26:0/1106232649 reap_dead start 2024-09-16T14:57:13.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f6032410640 1 -- 172.21.15.26:0/1106232649 reap_dead start 2024-09-16T14:57:13.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f6033e73640 1 -- 172.21.15.26:0/1106232649 shutdown_connections 2024-09-16T14:57:13.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f6033e73640 1 -- 172.21.15.26:0/1106232649 >> 172.21.15.26:0/1106232649 conn(0x7f602c09fb50 msgr2=0x7f602c0a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:13.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f6033e73640 1 -- 172.21.15.26:0/1106232649 shutdown_connections 2024-09-16T14:57:13.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.732+0000 7f6033e73640 1 -- 172.21.15.26:0/1106232649 wait complete. 2024-09-16T14:57:13.765 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:57:13.765 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:57:13.765 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:57:13.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.880+0000 7f79332d3640 10 monclient: get_monmap_and_config 2024-09-16T14:57:13.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.880+0000 7f79332d3640 10 monclient: build_initial_monmap 2024-09-16T14:57:13.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.880+0000 7f79332d3640 10 monclient: monmap: 2024-09-16T14:57:13.882 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:13.882 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.882 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:13.883985+0000 2024-09-16T14:57:13.882 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:13.883985+0000 2024-09-16T14:57:13.882 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:13.882 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.882 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:13.882 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:13.883 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:13.883 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.880+0000 7f79332d3640 1 Processor -- start 2024-09-16T14:57:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.880+0000 7f79332d3640 1 -- start start 2024-09-16T14:57:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.880+0000 7f79332d3640 10 monclient: init 2024-09-16T14:57:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:57:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 10 monclient(hunting): picked mon.noname-c con 0x7f792c09cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 10 monclient(hunting): picked mon.noname-a con 0x7f792c09e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 10 monclient(hunting): picked mon.noname-b con 0x7f792c09d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792c09d5b0 con 0x7f792c09e7a0 2024-09-16T14:57:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792c131a90 con 0x7f792c09cdc0 2024-09-16T14:57:13.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792c09aff0 con 0x7f792c09d1a0 2024-09-16T14:57:13.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:13.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 10 monclient(hunting): authenticate will time out at 1858.004713s 2024-09-16T14:57:13.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f793086e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f792c09e7a0 0x7f792c131380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39524/0 (socket says 172.21.15.26:39524) 2024-09-16T14:57:13.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7931870640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f792c09d1a0 0x7f792c139fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55520/0 (socket says 172.21.15.26:55520) 2024-09-16T14:57:13.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f793086e640 1 -- 172.21.15.26:0/2783507746 learned_addr learned my addr 172.21.15.26:0/2783507746 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:13.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 1 -- 172.21.15.26:0/2783507746 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129449490 0 0) 0x7f792c09d5b0 con 0x7f792c09e7a0 2024-09-16T14:57:13.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient(hunting): my global_id is 274356 2024-09-16T14:57:13.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 1 -- 172.21.15.26:0/2783507746 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7918003530 con 0x7f792c09e7a0 2024-09-16T14:57:13.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 1 -- 172.21.15.26:0/2783507746 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950831839 0 0) 0x7f792c09aff0 con 0x7f792c09d1a0 2024-09-16T14:57:13.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient(hunting): my global_id is 284267 2024-09-16T14:57:13.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 1 -- 172.21.15.26:0/2783507746 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f792c09d5b0 con 0x7f792c09d1a0 2024-09-16T14:57:13.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 1 -- 172.21.15.26:0/2783507746 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 979950050 0 0) 0x7f7918003530 con 0x7f792c09e7a0 2024-09-16T14:57:13.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 1 -- 172.21.15.26:0/2783507746 >> v1:172.21.15.26:6790/0 conn(0x7f792c09d1a0 legacy=0x7f792c139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 1 -- 172.21.15.26:0/2783507746 >> v1:172.21.15.189:6789/0 conn(0x7f792c09cdc0 legacy=0x7f792c130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient: _finish_hunting 0 2024-09-16T14:57:13.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 1 monclient: found mon.noname-a 2024-09-16T14:57:13.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:13.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:57:13.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 1 -- 172.21.15.26:0/2783507746 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792c09b2d0 con 0x7f792c09e7a0 2024-09-16T14:57:13.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient: _finish_auth 0 2024-09-16T14:57:13.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient: _check_auth_tickets 2024-09-16T14:57:13.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:13.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 1 -- 172.21.15.26:0/2783507746 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f791c001620 con 0x7f792c09e7a0 2024-09-16T14:57:13.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:13.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient: dump: 2024-09-16T14:57:13.893 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.893 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.893 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.893 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.893 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.893 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.893 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.893 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.893 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.893 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 5 monclient: authenticate success, global_id 274356 2024-09-16T14:57:13.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:13.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 1 -- 172.21.15.26:0/2783507746 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f791c001850 con 0x7f792c09e7a0 2024-09-16T14:57:13.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:13.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 1 -- 172.21.15.26:0/2783507746 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f791c001be0 con 0x7f792c09e7a0 2024-09-16T14:57:13.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:13.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f7923fff640 10 monclient: dump: 2024-09-16T14:57:13.894 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.894 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.894 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.894 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.894 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.894 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.895 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.895 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.895 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.895 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:13.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 4 set_mon_vals no callback set 2024-09-16T14:57:13.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 10 monclient: shutdown 2024-09-16T14:57:13.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:13.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 1 -- 172.21.15.26:0/2783507746 >> v1:172.21.15.26:6789/0 conn(0x7f792c09e7a0 legacy=0x7f792c131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 1 -- 172.21.15.26:0/2783507746 shutdown_connections 2024-09-16T14:57:13.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 1 -- 172.21.15.26:0/2783507746 >> 172.21.15.26:0/2783507746 conn(0x7f792c09fb50 msgr2=0x7f792c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:13.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 1 -- 172.21.15.26:0/2783507746 shutdown_connections 2024-09-16T14:57:13.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 1 -- 172.21.15.26:0/2783507746 wait complete. 2024-09-16T14:57:13.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 10 monclient: build_initial_monmap 2024-09-16T14:57:13.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 10 monclient: monmap: 2024-09-16T14:57:13.896 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:13.896 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:13.896 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:13.896 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:13.896 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:13.896 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:13.897 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:13.897 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:13.897 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:13.897 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 1 Processor -- start 2024-09-16T14:57:13.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 1 -- start start 2024-09-16T14:57:13.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.884+0000 7f79332d3640 10 monclient: init 2024-09-16T14:57:13.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:13.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:57:13.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 10 monclient(hunting): picked mon.noname-b con 0x7f792c09cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:13.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 10 monclient(hunting): picked mon.noname-c con 0x7f792c09d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:13.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 10 monclient(hunting): picked mon.noname-a con 0x7f792c09e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:13.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792c09a240 con 0x7f792c09e7a0 2024-09-16T14:57:13.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792c11db40 con 0x7f792c09cdc0 2024-09-16T14:57:13.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792c070190 con 0x7f792c09d1a0 2024-09-16T14:57:13.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:13.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 10 monclient(hunting): authenticate will time out at 1858.008226s 2024-09-16T14:57:13.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f793086e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f792c09d1a0 0x7f792c1ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55536/0 (socket says 172.21.15.26:55536) 2024-09-16T14:57:13.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f793086e640 1 -- 172.21.15.26:0/2488966954 learned_addr learned my addr 172.21.15.26:0/2488966954 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:13.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227189139 0 0) 0x7f792c09a240 con 0x7f792c09e7a0 2024-09-16T14:57:13.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): my global_id is 274359 2024-09-16T14:57:13.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7910003530 con 0x7f792c09e7a0 2024-09-16T14:57:13.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256159302 0 0) 0x7f792c070190 con 0x7f792c09d1a0 2024-09-16T14:57:13.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): my global_id is 284270 2024-09-16T14:57:13.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f792c09a240 con 0x7f792c09d1a0 2024-09-16T14:57:13.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4049816079 0 0) 0x7f7910003530 con 0x7f792c09e7a0 2024-09-16T14:57:13.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f792c070190 con 0x7f792c09e7a0 2024-09-16T14:57:13.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7924001be0 con 0x7f792c09e7a0 2024-09-16T14:57:13.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:13.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): dump: 2024-09-16T14:57:13.899 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.899 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.900 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.900 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.900 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.900 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.900 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.900 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.900 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.900 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:13.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3859458202 0 0) 0x7f792c09a240 con 0x7f792c09d1a0 2024-09-16T14:57:13.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7910003530 con 0x7f792c09d1a0 2024-09-16T14:57:13.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562110166 0 0) 0x7f792c11db40 con 0x7f792c09cdc0 2024-09-16T14:57:13.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:13.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:13.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:13.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): my global_id is 294121 2024-09-16T14:57:13.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f792c09a240 con 0x7f792c09cdc0 2024-09-16T14:57:13.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f791c001b60 con 0x7f792c09d1a0 2024-09-16T14:57:13.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:13.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient(hunting): dump: 2024-09-16T14:57:13.901 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.901 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.901 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.901 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.901 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.902 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.902 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.902 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.902 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.902 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1869390171 0 0) 0x7f792c070190 con 0x7f792c09e7a0 2024-09-16T14:57:13.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 >> v1:172.21.15.26:6790/0 conn(0x7f792c09d1a0 legacy=0x7f792c1ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 >> v1:172.21.15.189:6789/0 conn(0x7f792c09cdc0 legacy=0x7f792c12cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:13.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:57:13.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 monclient: found mon.a 2024-09-16T14:57:13.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:13.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:13.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792c11e060 con 0x7f792c09e7a0 2024-09-16T14:57:13.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient: _finish_auth 0 2024-09-16T14:57:13.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:57:13.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:13.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 5 monclient: authenticate success, global_id 274359 2024-09-16T14:57:13.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 10 monclient: _renew_subs 2024-09-16T14:57:13.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:13.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 1 -- 172.21.15.26:0/2488966954 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f792c11e340 con 0x7f792c09e7a0 2024-09-16T14:57:13.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 10 monclient: _renew_subs 2024-09-16T14:57:13.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:13.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 1 -- 172.21.15.26:0/2488966954 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f792c070d60 con 0x7f792c09e7a0 2024-09-16T14:57:13.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79240020b0 con 0x7f792c09e7a0 2024-09-16T14:57:13.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:13.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7924004d10 con 0x7f792c09e7a0 2024-09-16T14:57:13.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:13.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:13.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79237fe640 4 set_mon_vals no callback set 2024-09-16T14:57:13.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 10 monclient: dump: 2024-09-16T14:57:13.904 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:13.904 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:13.904 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.904 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:13.904 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:13.904 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:13.904 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:13.904 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:13.904 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:13.904 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:13.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:13.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f792400f800 con 0x7f792c09e7a0 2024-09-16T14:57:13.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:13.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:13.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f79332d3640 1 -- 172.21.15.26:0/2488966954 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790c005050 con 0x7f792c09e7a0 2024-09-16T14:57:13.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f7924043fe0 con 0x7f792c09e7a0 2024-09-16T14:57:13.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.888+0000 7f793106f640 10 monclient: get_auth_request con 0x7f7910036440 auth_method 0 2024-09-16T14:57:13.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.892+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f79240147c0 con 0x7f792c09e7a0 2024-09-16T14:57:13.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.892+0000 7f7921ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:13.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:13.892+0000 7f7921ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:14.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:14.108+0000 7f79332d3640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:57:14.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:14.108+0000 7f79332d3640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:57:14.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:14.108+0000 7f79332d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:14.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:14.108+0000 7f79332d3640 1 -- 172.21.15.26:0/2488966954 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f790c005b10 con 0x7f792c09e7a0 2024-09-16T14:57:14.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f7921ffb640 1 -- 172.21.15.26:0/2488966954 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f7924012e60 con 0x7f792c09e7a0 2024-09-16T14:57:14.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f7921ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:57:14.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f7921ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:14.112 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.112 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=11,kv=10,logm=53} crc {auth=1041464455,config=692619927,health=1541897506,kv=2084167376,logm=1476099594}) 2024-09-16T14:57:14.112 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=11,logm=49} crc {auth=565526629,config=692619927,health=1447119778,kv=1867599010,logm=2468179378}) 2024-09-16T14:57:14.112 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.113 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=11,kv=10,logm=53} crc {auth=1041464455,config=692619927,health=1541897506,kv=2084167376,logm=1476099594}) 2024-09-16T14:57:14.113 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=1,health=11,kv=9,logm=52} crc {auth=1511708698,config=1228700967,health=3078988827,kv=2303705537,logm=3624540291}) 2024-09-16T14:57:14.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f79332d3640 1 -- 172.21.15.26:0/2488966954 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7910036440 legacy=0x7f7910038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:14.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f79332d3640 10 monclient: shutdown 2024-09-16T14:57:14.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f79332d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:14.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f79332d3640 1 -- 172.21.15.26:0/2488966954 >> v1:172.21.15.26:6789/0 conn(0x7f792c09e7a0 legacy=0x7f792c1d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:14.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f79332d3640 1 -- 172.21.15.26:0/2488966954 shutdown_connections 2024-09-16T14:57:14.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f79332d3640 1 -- 172.21.15.26:0/2488966954 >> 172.21.15.26:0/2488966954 conn(0x7f792c09fb50 msgr2=0x7f792c09bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:14.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f79332d3640 1 -- 172.21.15.26:0/2488966954 shutdown_connections 2024-09-16T14:57:14.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:14.112+0000 7f79332d3640 1 -- 172.21.15.26:0/2488966954 wait complete. 2024-09-16T14:57:14.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.116+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.116+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3289039089}) 2024-09-16T14:57:14.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.116+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4277797870}) 2024-09-16T14:57:14.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.116+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.116+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3289039089}) 2024-09-16T14:57:14.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.116+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565446892}) 2024-09-16T14:57:14.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.116+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.116+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2258344323}) 2024-09-16T14:57:14.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.116+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1089264193}) 2024-09-16T14:57:14.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.116+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.116+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2258344323}) 2024-09-16T14:57:14.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.116+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=533867939}) 2024-09-16T14:57:14.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.120+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.120+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=70956982}) 2024-09-16T14:57:14.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.120+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288994831}) 2024-09-16T14:57:14.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.120+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.120+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=70956982}) 2024-09-16T14:57:14.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.120+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2103062690}) 2024-09-16T14:57:14.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.124+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.124+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756333546}) 2024-09-16T14:57:14.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.124+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1160357896}) 2024-09-16T14:57:14.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.124+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.124+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756333546}) 2024-09-16T14:57:14.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.124+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1479806677}) 2024-09-16T14:57:14.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.124+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.127 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.124+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2421747111}) 2024-09-16T14:57:14.127 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.124+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2997906272}) 2024-09-16T14:57:14.127 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.124+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.127 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.124+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2421747111}) 2024-09-16T14:57:14.127 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.124+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4009735846}) 2024-09-16T14:57:14.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.128+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.128+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2080143234}) 2024-09-16T14:57:14.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.128+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1662565890}) 2024-09-16T14:57:14.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.128+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.129 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.128+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2080143234}) 2024-09-16T14:57:14.129 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.128+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=308429330}) 2024-09-16T14:57:14.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.128+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.128+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402951376}) 2024-09-16T14:57:14.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.128+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121084726}) 2024-09-16T14:57:14.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.128+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.128+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402951376}) 2024-09-16T14:57:14.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.128+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4009538262}) 2024-09-16T14:57:14.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.132+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.132+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2581956310}) 2024-09-16T14:57:14.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.132+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=344581716}) 2024-09-16T14:57:14.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.132+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.132+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2581956310}) 2024-09-16T14:57:14.133 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.132+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532586285}) 2024-09-16T14:57:14.133 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:57:14.134 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.132+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.134 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.132+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1069991856}) 2024-09-16T14:57:14.134 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.132+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1254645891}) 2024-09-16T14:57:14.134 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.132+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.134 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.132+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1069991856}) 2024-09-16T14:57:14.134 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.132+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2942638491}) 2024-09-16T14:57:14.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.136+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.136+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2667526142}) 2024-09-16T14:57:14.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.136+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3800490133}) 2024-09-16T14:57:14.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.136+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.136+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2667526142}) 2024-09-16T14:57:14.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.136+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1540824801}) 2024-09-16T14:57:14.138 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.136+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.138 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.136+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=522613919}) 2024-09-16T14:57:14.138 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.136+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408217939}) 2024-09-16T14:57:14.138 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.136+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.138 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.136+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=522613919}) 2024-09-16T14:57:14.138 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.136+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1199463638}) 2024-09-16T14:57:14.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.140+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.140+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2056129453}) 2024-09-16T14:57:14.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.140+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1012929577}) 2024-09-16T14:57:14.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.140+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.140+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2056129453}) 2024-09-16T14:57:14.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.140+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2507117621}) 2024-09-16T14:57:14.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.140+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.140+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=89384698}) 2024-09-16T14:57:14.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.140+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176608162}) 2024-09-16T14:57:14.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.140+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.140+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=89384698}) 2024-09-16T14:57:14.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.140+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=11325011}) 2024-09-16T14:57:14.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.144+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.144+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2392280368}) 2024-09-16T14:57:14.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.144+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1299445783}) 2024-09-16T14:57:14.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.144+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.144+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2392280368}) 2024-09-16T14:57:14.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.144+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=390464683}) 2024-09-16T14:57:14.146 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.144+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.146 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.144+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3574896821}) 2024-09-16T14:57:14.148 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.144+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869265005}) 2024-09-16T14:57:14.148 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.144+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.148 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.144+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3574896821}) 2024-09-16T14:57:14.148 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.144+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=953126696}) 2024-09-16T14:57:14.149 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.149 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4192889620}) 2024-09-16T14:57:14.150 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3683347810}) 2024-09-16T14:57:14.150 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.150 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4192889620}) 2024-09-16T14:57:14.150 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=56158449}) 2024-09-16T14:57:14.150 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.151 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2629274904}) 2024-09-16T14:57:14.151 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2564840114}) 2024-09-16T14:57:14.151 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.151 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2629274904}) 2024-09-16T14:57:14.151 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1237910681}) 2024-09-16T14:57:14.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255175115}) 2024-09-16T14:57:14.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2241429541}) 2024-09-16T14:57:14.154 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.154 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.148+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255175115}) 2024-09-16T14:57:14.154 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.152+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3800849721}) 2024-09-16T14:57:14.154 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.152+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.154 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.152+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1798467268}) 2024-09-16T14:57:14.154 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.152+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2150044890}) 2024-09-16T14:57:14.154 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.152+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.154 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.152+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1798467268}) 2024-09-16T14:57:14.154 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.152+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1390063528}) 2024-09-16T14:57:14.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.152+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.152+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=853541343}) 2024-09-16T14:57:14.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.152+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370788382}) 2024-09-16T14:57:14.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.152+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.152+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=853541343}) 2024-09-16T14:57:14.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.152+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3731230077}) 2024-09-16T14:57:14.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.156+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.156+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=18378414}) 2024-09-16T14:57:14.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.156+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067656044}) 2024-09-16T14:57:14.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.156+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.156+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=18378414}) 2024-09-16T14:57:14.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.156+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615329736}) 2024-09-16T14:57:14.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.156+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.156+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2622366270}) 2024-09-16T14:57:14.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.156+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2989869426}) 2024-09-16T14:57:14.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.156+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.156+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2622366270}) 2024-09-16T14:57:14.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.156+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229518631}) 2024-09-16T14:57:14.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.160+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.160+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389628253}) 2024-09-16T14:57:14.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.160+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3651895267}) 2024-09-16T14:57:14.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.160+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.160+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389628253}) 2024-09-16T14:57:14.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.160+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1813361577}) 2024-09-16T14:57:14.162 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.160+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.163 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.160+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4282153461}) 2024-09-16T14:57:14.163 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.160+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1500989599}) 2024-09-16T14:57:14.163 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.160+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.163 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.160+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4282153461}) 2024-09-16T14:57:14.163 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.160+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=831190733}) 2024-09-16T14:57:14.164 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.164+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.164 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.164+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3100682956}) 2024-09-16T14:57:14.164 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.164+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440843528}) 2024-09-16T14:57:14.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.164+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.164+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3100682956}) 2024-09-16T14:57:14.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.164+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=422106722}) 2024-09-16T14:57:14.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.164+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.164+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=161413589}) 2024-09-16T14:57:14.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.164+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3555155783}) 2024-09-16T14:57:14.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.164+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.164+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=161413589}) 2024-09-16T14:57:14.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.164+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2368826503}) 2024-09-16T14:57:14.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.168+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.168+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=660428897}) 2024-09-16T14:57:14.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.168+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486615904}) 2024-09-16T14:57:14.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.168+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.168+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=660428897}) 2024-09-16T14:57:14.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.168+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2673584826}) 2024-09-16T14:57:14.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.168+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.168+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3602095418}) 2024-09-16T14:57:14.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.168+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1163086112}) 2024-09-16T14:57:14.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.168+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.168+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3602095418}) 2024-09-16T14:57:14.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.168+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=459975620}) 2024-09-16T14:57:14.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.172+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.172+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4193951394}) 2024-09-16T14:57:14.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.172+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226332578}) 2024-09-16T14:57:14.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.172+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.172+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4193951394}) 2024-09-16T14:57:14.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.172+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2787992658}) 2024-09-16T14:57:14.173 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.172+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.173 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.172+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3656170768}) 2024-09-16T14:57:14.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.172+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=223815420}) 2024-09-16T14:57:14.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.172+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.172+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3656170768}) 2024-09-16T14:57:14.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.172+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4156230945}) 2024-09-16T14:57:14.175 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.176+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.176+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1264047216}) 2024-09-16T14:57:14.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.176+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3024996118}) 2024-09-16T14:57:14.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.176+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.176+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1264047216}) 2024-09-16T14:57:14.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.176+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=249635673}) 2024-09-16T14:57:14.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.176+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.176+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2046107662}) 2024-09-16T14:57:14.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.176+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448096242}) 2024-09-16T14:57:14.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.176+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.178 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.176+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2046107662}) 2024-09-16T14:57:14.178 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.176+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=200673904}) 2024-09-16T14:57:14.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.180+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.180+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=634225666}) 2024-09-16T14:57:14.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.180+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1285015779}) 2024-09-16T14:57:14.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.180+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.180+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=634225666}) 2024-09-16T14:57:14.180 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.180+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2767506580}) 2024-09-16T14:57:14.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.180+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.180+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3326015699}) 2024-09-16T14:57:14.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.180+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3932499712}) 2024-09-16T14:57:14.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.180+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.180+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3326015699}) 2024-09-16T14:57:14.182 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.180+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3574207300}) 2024-09-16T14:57:14.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.180+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458734503}) 2024-09-16T14:57:14.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1711891946}) 2024-09-16T14:57:14.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458734503}) 2024-09-16T14:57:14.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2130926200}) 2024-09-16T14:57:14.185 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.185 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2562685597}) 2024-09-16T14:57:14.185 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192382561}) 2024-09-16T14:57:14.185 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.185 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2562685597}) 2024-09-16T14:57:14.185 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=840427729}) 2024-09-16T14:57:14.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2773181686}) 2024-09-16T14:57:14.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283201480}) 2024-09-16T14:57:14.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2773181686}) 2024-09-16T14:57:14.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.184+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2802881239}) 2024-09-16T14:57:14.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.188+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.189 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.188+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2870886394}) 2024-09-16T14:57:14.189 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.188+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1522039243}) 2024-09-16T14:57:14.189 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.188+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.189 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.188+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2870886394}) 2024-09-16T14:57:14.189 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.188+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3443904973}) 2024-09-16T14:57:14.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.188+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.188+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=666195293}) 2024-09-16T14:57:14.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.188+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=842225764}) 2024-09-16T14:57:14.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.188+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.188+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=666195293}) 2024-09-16T14:57:14.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.188+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=20042783}) 2024-09-16T14:57:14.192 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.192+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.192 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.192+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3595689695}) 2024-09-16T14:57:14.192 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.192+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=567450191}) 2024-09-16T14:57:14.192 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.192+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.192+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3595689695}) 2024-09-16T14:57:14.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.192+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1715846206}) 2024-09-16T14:57:14.194 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.192+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.194 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.192+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247863469}) 2024-09-16T14:57:14.194 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.192+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2005996880}) 2024-09-16T14:57:14.194 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.192+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.194 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.192+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247863469}) 2024-09-16T14:57:14.194 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.192+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1195362745}) 2024-09-16T14:57:14.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.196+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.196+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3045970715}) 2024-09-16T14:57:14.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.196+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2180464993}) 2024-09-16T14:57:14.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.196+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.196+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3045970715}) 2024-09-16T14:57:14.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.196+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1867078845}) 2024-09-16T14:57:14.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.196+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.196+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1805653188}) 2024-09-16T14:57:14.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.196+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1276071765}) 2024-09-16T14:57:14.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.196+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.196+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1805653188}) 2024-09-16T14:57:14.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.196+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1979731416}) 2024-09-16T14:57:14.199 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.200+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.200+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1763617979}) 2024-09-16T14:57:14.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.200+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558129400}) 2024-09-16T14:57:14.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.200+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.200+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1763617979}) 2024-09-16T14:57:14.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.200+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=924212616}) 2024-09-16T14:57:14.201 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.200+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.201 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.200+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2234569238}) 2024-09-16T14:57:14.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.200+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3782525287}) 2024-09-16T14:57:14.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.200+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.200+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2234569238}) 2024-09-16T14:57:14.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.200+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1427005327}) 2024-09-16T14:57:14.203 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.204+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.203 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.204+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1549245101}) 2024-09-16T14:57:14.203 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.204+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988038420}) 2024-09-16T14:57:14.203 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.204+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.203 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.204+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1549245101}) 2024-09-16T14:57:14.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.204+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2068536320}) 2024-09-16T14:57:14.205 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.204+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.205 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.204+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=42554853}) 2024-09-16T14:57:14.205 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.204+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=828142526}) 2024-09-16T14:57:14.205 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.204+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.205 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.204+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=42554853}) 2024-09-16T14:57:14.205 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.204+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3359554079}) 2024-09-16T14:57:14.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.208+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.208+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2961865214}) 2024-09-16T14:57:14.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.208+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809639083}) 2024-09-16T14:57:14.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.208+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.208+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2961865214}) 2024-09-16T14:57:14.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.208+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449737925}) 2024-09-16T14:57:14.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.208+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.208+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018249259}) 2024-09-16T14:57:14.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.208+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2508978920}) 2024-09-16T14:57:14.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.208+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.208+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018249259}) 2024-09-16T14:57:14.210 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.208+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=633076294}) 2024-09-16T14:57:14.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.212+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.212+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3400609058}) 2024-09-16T14:57:14.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.212+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271885291}) 2024-09-16T14:57:14.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.212+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.212+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3400609058}) 2024-09-16T14:57:14.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.212+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3658556840}) 2024-09-16T14:57:14.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.212+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.212+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2853895731}) 2024-09-16T14:57:14.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.212+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1342385220}) 2024-09-16T14:57:14.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.212+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.212+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2853895731}) 2024-09-16T14:57:14.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.212+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2074681075}) 2024-09-16T14:57:14.215 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.216+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.215 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.216+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2264749332}) 2024-09-16T14:57:14.215 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.216+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733796280}) 2024-09-16T14:57:14.215 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.216+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.216+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2264749332}) 2024-09-16T14:57:14.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.216+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3813233156}) 2024-09-16T14:57:14.217 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.216+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.217 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.216+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2822320938}) 2024-09-16T14:57:14.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.216+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1976645292}) 2024-09-16T14:57:14.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.216+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.216+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2822320938}) 2024-09-16T14:57:14.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.216+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=218314402}) 2024-09-16T14:57:14.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.220+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.220 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.220+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2483228334}) 2024-09-16T14:57:14.220 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.220+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4069238392}) 2024-09-16T14:57:14.220 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.220+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.220 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.220+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2483228334}) 2024-09-16T14:57:14.220 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.220+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2258743717}) 2024-09-16T14:57:14.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.220+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.222 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.220+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1693927610}) 2024-09-16T14:57:14.222 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.220+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1804023528}) 2024-09-16T14:57:14.222 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.220+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.222 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.220+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1693927610}) 2024-09-16T14:57:14.222 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.220+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2279305029}) 2024-09-16T14:57:14.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.224+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.224 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.224+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1368306245}) 2024-09-16T14:57:14.224 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.224+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2619607288}) 2024-09-16T14:57:14.224 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.224+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.224 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.224+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1368306245}) 2024-09-16T14:57:14.224 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.224+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=145355657}) 2024-09-16T14:57:14.226 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.224+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.226 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.224+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1604276102}) 2024-09-16T14:57:14.226 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.224+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3300203210}) 2024-09-16T14:57:14.226 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.224+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.226 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.224+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1604276102}) 2024-09-16T14:57:14.226 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.224+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1928580431}) 2024-09-16T14:57:14.228 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.228+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.228 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.228+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=525829240}) 2024-09-16T14:57:14.228 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.228+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3551346950}) 2024-09-16T14:57:14.228 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.228+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.228 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.228+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=525829240}) 2024-09-16T14:57:14.228 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.228+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1338912616}) 2024-09-16T14:57:14.230 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.228+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.230 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.228+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=998100114}) 2024-09-16T14:57:14.230 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.228+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3598530281}) 2024-09-16T14:57:14.230 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.228+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.230 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.228+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=998100114}) 2024-09-16T14:57:14.230 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.228+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4117347061}) 2024-09-16T14:57:14.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.232+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.232+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3904951246}) 2024-09-16T14:57:14.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.232+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595657248}) 2024-09-16T14:57:14.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.232+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.232+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3904951246}) 2024-09-16T14:57:14.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.232+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2892411201}) 2024-09-16T14:57:14.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.232+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.232+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3100462945}) 2024-09-16T14:57:14.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.232+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2274611674}) 2024-09-16T14:57:14.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.232+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.232+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3100462945}) 2024-09-16T14:57:14.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.232+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=11917811}) 2024-09-16T14:57:14.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.236+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.236+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1748245750}) 2024-09-16T14:57:14.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.236+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795646258}) 2024-09-16T14:57:14.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.236+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.236+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1748245750}) 2024-09-16T14:57:14.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.236+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4156526932}) 2024-09-16T14:57:14.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.236+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.236+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1820362844}) 2024-09-16T14:57:14.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.236+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1662259013}) 2024-09-16T14:57:14.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.236+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.236+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1820362844}) 2024-09-16T14:57:14.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.236+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=862145302}) 2024-09-16T14:57:14.239 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.240+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.239 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.240+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=771940276}) 2024-09-16T14:57:14.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.240+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909384612}) 2024-09-16T14:57:14.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.240+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.240+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=771940276}) 2024-09-16T14:57:14.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.240+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1963721308}) 2024-09-16T14:57:14.241 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.240+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.241 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.240+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1496198445}) 2024-09-16T14:57:14.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.240+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1064488805}) 2024-09-16T14:57:14.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.240+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.240+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1496198445}) 2024-09-16T14:57:14.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.240+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588831625}) 2024-09-16T14:57:14.243 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.244+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.243 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.244+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2360322756}) 2024-09-16T14:57:14.243 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.244+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3614823318}) 2024-09-16T14:57:14.243 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.244+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.244+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2360322756}) 2024-09-16T14:57:14.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.244+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1391060380}) 2024-09-16T14:57:14.245 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.244+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.245 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.244+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2404422558}) 2024-09-16T14:57:14.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.244+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1911332229}) 2024-09-16T14:57:14.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.244+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.244+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2404422558}) 2024-09-16T14:57:14.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.244+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4124840254}) 2024-09-16T14:57:14.247 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.248+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.247 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.248+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2316671644}) 2024-09-16T14:57:14.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.248+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2546465297}) 2024-09-16T14:57:14.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.248+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.248+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2316671644}) 2024-09-16T14:57:14.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.248+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4112047930}) 2024-09-16T14:57:14.250 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.248+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.250 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.248+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2356486440}) 2024-09-16T14:57:14.250 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.248+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3113424911}) 2024-09-16T14:57:14.250 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.248+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.250 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.248+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2356486440}) 2024-09-16T14:57:14.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.248+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3146298023}) 2024-09-16T14:57:14.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.252+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.252+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3645562087}) 2024-09-16T14:57:14.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.252+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360784327}) 2024-09-16T14:57:14.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.252+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.252+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3645562087}) 2024-09-16T14:57:14.253 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.252+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2343099887}) 2024-09-16T14:57:14.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.252+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.252+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2642640307}) 2024-09-16T14:57:14.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.252+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736800578}) 2024-09-16T14:57:14.255 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.252+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.255 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.252+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2642640307}) 2024-09-16T14:57:14.255 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.252+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4100226076}) 2024-09-16T14:57:14.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.256+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.256+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2710761845}) 2024-09-16T14:57:14.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.256+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3541382786}) 2024-09-16T14:57:14.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.256+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.256+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2710761845}) 2024-09-16T14:57:14.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.256+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1491559972}) 2024-09-16T14:57:14.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.256+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.256+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3653012628}) 2024-09-16T14:57:14.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.256+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=197725434}) 2024-09-16T14:57:14.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.256+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.256+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3653012628}) 2024-09-16T14:57:14.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.256+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=740123596}) 2024-09-16T14:57:14.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.260+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.260+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2686194207}) 2024-09-16T14:57:14.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.260+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1674139004}) 2024-09-16T14:57:14.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.260+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.260+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2686194207}) 2024-09-16T14:57:14.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.260+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3338896483}) 2024-09-16T14:57:14.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.260+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.260+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1470550463}) 2024-09-16T14:57:14.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.260+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2268098320}) 2024-09-16T14:57:14.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.260+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.260+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1470550463}) 2024-09-16T14:57:14.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.260+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3418702920}) 2024-09-16T14:57:14.263 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.264+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.264+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=995422121}) 2024-09-16T14:57:14.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.264+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3872443262}) 2024-09-16T14:57:14.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.264+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.264+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=995422121}) 2024-09-16T14:57:14.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.264+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2118896405}) 2024-09-16T14:57:14.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.264+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.264+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2899645494}) 2024-09-16T14:57:14.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.264+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=733658398}) 2024-09-16T14:57:14.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.264+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.264+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2899645494}) 2024-09-16T14:57:14.267 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.264+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2652136536}) 2024-09-16T14:57:14.267 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.268+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.268+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2533229509}) 2024-09-16T14:57:14.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.268+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742378985}) 2024-09-16T14:57:14.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.268+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.268+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2533229509}) 2024-09-16T14:57:14.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.268+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4006307196}) 2024-09-16T14:57:14.269 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.268+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.268+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3480390269}) 2024-09-16T14:57:14.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.268+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4049179026}) 2024-09-16T14:57:14.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.268+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.268+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3480390269}) 2024-09-16T14:57:14.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.268+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2896261106}) 2024-09-16T14:57:14.271 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.272+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.272+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3997362291}) 2024-09-16T14:57:14.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.272+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4044788354}) 2024-09-16T14:57:14.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.272+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.272+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3997362291}) 2024-09-16T14:57:14.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.272+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=475582126}) 2024-09-16T14:57:14.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.272+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.272+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3896428292}) 2024-09-16T14:57:14.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.272+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2630078489}) 2024-09-16T14:57:14.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.272+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.272+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3896428292}) 2024-09-16T14:57:14.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.272+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1405442882}) 2024-09-16T14:57:14.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.276+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.276+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1310091530}) 2024-09-16T14:57:14.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.276+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3024474246}) 2024-09-16T14:57:14.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.276+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.276+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1310091530}) 2024-09-16T14:57:14.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.276+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2291413548}) 2024-09-16T14:57:14.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.276+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.276+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3715378955}) 2024-09-16T14:57:14.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.276+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809404086}) 2024-09-16T14:57:14.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.276+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.276+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3715378955}) 2024-09-16T14:57:14.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.276+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2701934054}) 2024-09-16T14:57:14.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.280+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.280+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2511272185}) 2024-09-16T14:57:14.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.280+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543386575}) 2024-09-16T14:57:14.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.280+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.280+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2511272185}) 2024-09-16T14:57:14.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.280+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=390580079}) 2024-09-16T14:57:14.281 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.280+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.281 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.280+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871903985}) 2024-09-16T14:57:14.282 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.280+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136381259}) 2024-09-16T14:57:14.282 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.280+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.282 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.280+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871903985}) 2024-09-16T14:57:14.282 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.280+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2777716809}) 2024-09-16T14:57:14.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.284+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.284 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.284+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2616941425}) 2024-09-16T14:57:14.284 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.284+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297143601}) 2024-09-16T14:57:14.284 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.284+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.284 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.284+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2616941425}) 2024-09-16T14:57:14.284 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.284+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3989422741}) 2024-09-16T14:57:14.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.284+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.284+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1380419336}) 2024-09-16T14:57:14.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.284+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3594084553}) 2024-09-16T14:57:14.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.284+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.284+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1380419336}) 2024-09-16T14:57:14.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.284+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4279764957}) 2024-09-16T14:57:14.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=467849007}) 2024-09-16T14:57:14.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293860906}) 2024-09-16T14:57:14.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=467849007}) 2024-09-16T14:57:14.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1023352005}) 2024-09-16T14:57:14.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2183133506}) 2024-09-16T14:57:14.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2312609918}) 2024-09-16T14:57:14.290 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.290 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2183133506}) 2024-09-16T14:57:14.290 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=388147068}) 2024-09-16T14:57:14.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3159897519}) 2024-09-16T14:57:14.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448729434}) 2024-09-16T14:57:14.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.292 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3159897519}) 2024-09-16T14:57:14.292 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3522506256}) 2024-09-16T14:57:14.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=1566431141,mdsmap=1531151339}) 2024-09-16T14:57:14.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=152037609,mdsmap=2147943554}) 2024-09-16T14:57:14.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=1566431141,mdsmap=1531151339}) 2024-09-16T14:57:14.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=2358039911,mdsmap=1531151339,mgr=2439950282}) 2024-09-16T14:57:14.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {mgr=866925232,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=307978004,mon_config_key=4052528785,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1510949214}) 2024-09-16T14:57:14.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=8,mgr_command_descs=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {mgr=2120425906,mgr_command_descs=794974419,mgrstat=307978004,mon_config_key=2780249362,monmap=970648238,osd_metadata=1033174197,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=812215803}) 2024-09-16T14:57:14.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:14.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {mgr=866925232,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=307978004,mon_config_key=4052528785,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1510949214}) 2024-09-16T14:57:14.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:14.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=4019727853,mon_config_key=4052528784,monmap=3140514009,osd_metadata=2172984177,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1834044785}) 2024-09-16T14:57:15.134 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:57:15.135 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:57:15.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 10 monclient: get_monmap_and_config 2024-09-16T14:57:15.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 10 monclient: build_initial_monmap 2024-09-16T14:57:15.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 10 monclient: monmap: 2024-09-16T14:57:15.265 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:15.265 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.265 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:15.266394+0000 2024-09-16T14:57:15.265 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:15.266394+0000 2024-09-16T14:57:15.265 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:15.265 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.265 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:15.265 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:15.265 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:15.266 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 1 Processor -- start 2024-09-16T14:57:15.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 1 -- start start 2024-09-16T14:57:15.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 10 monclient: init 2024-09-16T14:57:15.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:15.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:57:15.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 10 monclient(hunting): picked mon.noname-a con 0x7f641409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:15.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 10 monclient(hunting): picked mon.noname-b con 0x7f6414132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:15.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 10 monclient(hunting): picked mon.noname-c con 0x7f6414138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:15.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f641409d260 con 0x7f641409cb80 2024-09-16T14:57:15.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6414132a80 con 0x7f6414138ff0 2024-09-16T14:57:15.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f641409cf60 con 0x7f6414132650 2024-09-16T14:57:15.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:15.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 10 monclient(hunting): authenticate will time out at 1859.386539s 2024-09-16T14:57:15.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64137fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f641409cb80 0x7f64140717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39536/0 (socket says 172.21.15.26:39536) 2024-09-16T14:57:15.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64137fe640 1 -- 172.21.15.26:0/1556358034 learned_addr learned my addr 172.21.15.26:0/1556358034 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:15.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 1 -- 172.21.15.26:0/1556358034 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716265936 0 0) 0x7f641409d260 con 0x7f641409cb80 2024-09-16T14:57:15.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient(hunting): my global_id is 274362 2024-09-16T14:57:15.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 1 -- 172.21.15.26:0/1556358034 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6400003530 con 0x7f641409cb80 2024-09-16T14:57:15.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 1 -- 172.21.15.26:0/1556358034 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493494925 0 0) 0x7f641409cf60 con 0x7f6414132650 2024-09-16T14:57:15.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient(hunting): my global_id is 284273 2024-09-16T14:57:15.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 1 -- 172.21.15.26:0/1556358034 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f641409d260 con 0x7f6414132650 2024-09-16T14:57:15.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 1 -- 172.21.15.26:0/1556358034 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 291707167 0 0) 0x7f6400003530 con 0x7f641409cb80 2024-09-16T14:57:15.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 1 -- 172.21.15.26:0/1556358034 >> v1:172.21.15.26:6790/0 conn(0x7f6414132650 legacy=0x7f64141378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 1 -- 172.21.15.26:0/1556358034 >> v1:172.21.15.189:6789/0 conn(0x7f6414138ff0 legacy=0x7f641413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient: _finish_hunting 0 2024-09-16T14:57:15.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 1 monclient: found mon.noname-a 2024-09-16T14:57:15.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:15.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:57:15.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 1 -- 172.21.15.26:0/1556358034 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641409f2d0 con 0x7f641409cb80 2024-09-16T14:57:15.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient: _finish_auth 0 2024-09-16T14:57:15.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient: _check_auth_tickets 2024-09-16T14:57:15.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 1 -- 172.21.15.26:0/1556358034 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6404001620 con 0x7f641409cb80 2024-09-16T14:57:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient: dump: 2024-09-16T14:57:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.277 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.278 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 5 monclient: authenticate success, global_id 274362 2024-09-16T14:57:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 1 -- 172.21.15.26:0/1556358034 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6404001850 con 0x7f641409cb80 2024-09-16T14:57:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 1 -- 172.21.15.26:0/1556358034 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6404001be0 con 0x7f641409cb80 2024-09-16T14:57:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f64127fc640 10 monclient: dump: 2024-09-16T14:57:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.278 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.279 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.279 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.279 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.279 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.279 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.279 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.279 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:15.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.264+0000 7f6419fd5640 4 set_mon_vals no callback set 2024-09-16T14:57:15.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient: shutdown 2024-09-16T14:57:15.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:15.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 1 -- 172.21.15.26:0/1556358034 >> v1:172.21.15.26:6789/0 conn(0x7f641409cb80 legacy=0x7f64140717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 1 -- 172.21.15.26:0/1556358034 shutdown_connections 2024-09-16T14:57:15.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 1 -- 172.21.15.26:0/1556358034 >> 172.21.15.26:0/1556358034 conn(0x7f641409fb50 msgr2=0x7f641409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:15.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 1 -- 172.21.15.26:0/1556358034 shutdown_connections 2024-09-16T14:57:15.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 1 -- 172.21.15.26:0/1556358034 wait complete. 2024-09-16T14:57:15.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient: build_initial_monmap 2024-09-16T14:57:15.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient: monmap: 2024-09-16T14:57:15.280 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:15.280 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:15.280 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:15.280 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:15.280 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:15.281 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:15.281 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:15.281 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:15.281 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:15.281 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 1 Processor -- start 2024-09-16T14:57:15.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 1 -- start start 2024-09-16T14:57:15.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient: init 2024-09-16T14:57:15.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:15.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:57:15.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient(hunting): picked mon.noname-a con 0x7f641409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:15.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient(hunting): picked mon.noname-b con 0x7f6414132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:15.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient(hunting): picked mon.noname-c con 0x7f6414138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:15.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6414137670 con 0x7f641409cb80 2024-09-16T14:57:15.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6414132120 con 0x7f6414132650 2024-09-16T14:57:15.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6414130450 con 0x7f6414138ff0 2024-09-16T14:57:15.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:15.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient(hunting): authenticate will time out at 1859.389747s 2024-09-16T14:57:15.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f64137fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f641409cb80 0x7f6414134210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39544/0 (socket says 172.21.15.26:39544) 2024-09-16T14:57:15.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f64137fe640 1 -- 172.21.15.26:0/853235308 learned_addr learned my addr 172.21.15.26:0/853235308 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:15.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189372088 0 0) 0x7f6414137670 con 0x7f641409cb80 2024-09-16T14:57:15.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): my global_id is 274365 2024-09-16T14:57:15.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f4003530 con 0x7f641409cb80 2024-09-16T14:57:15.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184631610 0 0) 0x7f6414130450 con 0x7f6414138ff0 2024-09-16T14:57:15.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): my global_id is 284276 2024-09-16T14:57:15.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6414137670 con 0x7f6414138ff0 2024-09-16T14:57:15.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3550630641 0 0) 0x7f6414137670 con 0x7f6414138ff0 2024-09-16T14:57:15.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6414130450 con 0x7f6414138ff0 2024-09-16T14:57:15.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f640c001b80 con 0x7f6414138ff0 2024-09-16T14:57:15.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:15.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): dump: 2024-09-16T14:57:15.284 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.284 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.284 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.284 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.284 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.284 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.285 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.285 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.285 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.285 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:15.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1301349829 0 0) 0x7f63f4003530 con 0x7f641409cb80 2024-09-16T14:57:15.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6414137670 con 0x7f641409cb80 2024-09-16T14:57:15.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015188928 0 0) 0x7f6414132120 con 0x7f6414132650 2024-09-16T14:57:15.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): my global_id is 294127 2024-09-16T14:57:15.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f4003530 con 0x7f6414132650 2024-09-16T14:57:15.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6404001ba0 con 0x7f641409cb80 2024-09-16T14:57:15.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:15.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient(hunting): dump: 2024-09-16T14:57:15.286 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.286 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.286 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.286 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.287 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.287 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.287 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.287 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.287 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.287 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1384123135 0 0) 0x7f6414130450 con 0x7f6414138ff0 2024-09-16T14:57:15.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 >> v1:172.21.15.189:6789/0 conn(0x7f6414132650 legacy=0x7f6414134920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 >> v1:172.21.15.26:6789/0 conn(0x7f641409cb80 legacy=0x7f6414134210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:57:15.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 monclient: found mon.c 2024-09-16T14:57:15.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:15.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:15.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641409a240 con 0x7f6414138ff0 2024-09-16T14:57:15.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient: _finish_auth 0 2024-09-16T14:57:15.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:57:15.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:15.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 5 monclient: authenticate success, global_id 284276 2024-09-16T14:57:15.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient: _renew_subs 2024-09-16T14:57:15.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:15.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 1 -- 172.21.15.26:0/853235308 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6414135bb0 con 0x7f6414138ff0 2024-09-16T14:57:15.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient: _renew_subs 2024-09-16T14:57:15.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:15.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 1 -- 172.21.15.26:0/853235308 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6414138330 con 0x7f6414138ff0 2024-09-16T14:57:15.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f640c0039a0 con 0x7f6414138ff0 2024-09-16T14:57:15.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:15.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f640c003d30 con 0x7f6414138ff0 2024-09-16T14:57:15.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:15.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6411ffb640 4 set_mon_vals no callback set 2024-09-16T14:57:15.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 10 monclient: dump: 2024-09-16T14:57:15.290 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.290 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.290 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.290 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.290 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.290 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.290 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.290 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.291 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.291 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:15.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:15.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:15.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f6419fd5640 1 -- 172.21.15.26:0/853235308 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f8005050 con 0x7f6414138ff0 2024-09-16T14:57:15.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.268+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f640c003e90 con 0x7f6414138ff0 2024-09-16T14:57:15.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.272+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f640c042e60 con 0x7f6414138ff0 2024-09-16T14:57:15.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.272+0000 7f64137fe640 10 monclient: get_auth_request con 0x7f63f40363d0 auth_method 0 2024-09-16T14:57:15.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.276+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f640c013850 con 0x7f6414138ff0 2024-09-16T14:57:15.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.276+0000 7f640bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:15.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.276+0000 7f640bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:15.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.488+0000 7f6419fd5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:57:15.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.488+0000 7f6419fd5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:15.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.488+0000 7f6419fd5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:15.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.488+0000 7f6419fd5640 1 -- 172.21.15.26:0/853235308 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f63f8005880 con 0x7f6414138ff0 2024-09-16T14:57:15.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.488+0000 7f640bfff640 1 -- 172.21.15.26:0/853235308 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 401319691) 0x7f640c011cc0 con 0x7f6414138ff0 2024-09-16T14:57:15.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.488+0000 7f640bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:15.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.488+0000 7f640bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:15.490 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:57:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.492+0000 7f6419fd5640 1 -- 172.21.15.26:0/853235308 >> v1:172.21.15.189:6812/3196161582 conn(0x7f63f40363d0 legacy=0x7f63f4038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.492+0000 7f6419fd5640 10 monclient: shutdown 2024-09-16T14:57:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.492+0000 7f6419fd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.492+0000 7f6419fd5640 1 -- 172.21.15.26:0/853235308 >> v1:172.21.15.26:6790/0 conn(0x7f6414138ff0 legacy=0x7f64141cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.492+0000 7f6419fd5640 1 -- 172.21.15.26:0/853235308 shutdown_connections 2024-09-16T14:57:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.492+0000 7f6419fd5640 1 -- 172.21.15.26:0/853235308 >> 172.21.15.26:0/853235308 conn(0x7f641409fb50 msgr2=0x7f64141388c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.492+0000 7f6419fd5640 1 -- 172.21.15.26:0/853235308 shutdown_connections 2024-09-16T14:57:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.492+0000 7f6419fd5640 1 -- 172.21.15.26:0/853235308 wait complete. 2024-09-16T14:57:15.506 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:57:15.506 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:57:15.506 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:57:15.507 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:57:15.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 10 monclient: get_monmap_and_config 2024-09-16T14:57:15.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 10 monclient: build_initial_monmap 2024-09-16T14:57:15.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 10 monclient: monmap: 2024-09-16T14:57:15.628 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:15.628 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.628 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:15.629678+0000 2024-09-16T14:57:15.628 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:15.629678+0000 2024-09-16T14:57:15.628 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:15.628 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.628 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:15.628 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:15.628 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:15.628 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 1 Processor -- start 2024-09-16T14:57:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 1 -- start start 2024-09-16T14:57:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 10 monclient: init 2024-09-16T14:57:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:57:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 10 monclient(hunting): picked mon.noname-c con 0x7fe2d00a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 10 monclient(hunting): picked mon.noname-a con 0x7fe2d0099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 10 monclient(hunting): picked mon.noname-b con 0x7fe2d00a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d0098ea0 con 0x7fe2d0099b60 2024-09-16T14:57:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d00997b0 con 0x7fe2d00a1c00 2024-09-16T14:57:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d0099460 con 0x7fe2d00a26f0 2024-09-16T14:57:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 10 monclient(hunting): authenticate will time out at 1859.749687s 2024-09-16T14:57:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ceffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe2d0099b60 0x7fe2d0139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39548/0 (socket says 172.21.15.26:39548) 2024-09-16T14:57:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ceffd640 1 -- 172.21.15.26:0/1883728389 learned_addr learned my addr 172.21.15.26:0/1883728389 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 1 -- 172.21.15.26:0/1883728389 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560104531 0 0) 0x7fe2d0098ea0 con 0x7fe2d0099b60 2024-09-16T14:57:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient(hunting): my global_id is 274368 2024-09-16T14:57:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 1 -- 172.21.15.26:0/1883728389 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b8003530 con 0x7fe2d0099b60 2024-09-16T14:57:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 1 -- 172.21.15.26:0/1883728389 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595572410 0 0) 0x7fe2d0099460 con 0x7fe2d00a26f0 2024-09-16T14:57:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient(hunting): my global_id is 284279 2024-09-16T14:57:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 1 -- 172.21.15.26:0/1883728389 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d0098ea0 con 0x7fe2d00a26f0 2024-09-16T14:57:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 1 -- 172.21.15.26:0/1883728389 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2228497203 0 0) 0x7fe2b8003530 con 0x7fe2d0099b60 2024-09-16T14:57:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 1 -- 172.21.15.26:0/1883728389 >> v1:172.21.15.26:6790/0 conn(0x7fe2d00a26f0 legacy=0x7fe2d013d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 1 -- 172.21.15.26:0/1883728389 >> v1:172.21.15.189:6789/0 conn(0x7fe2d00a1c00 legacy=0x7fe2d00a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:57:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 1 monclient: found mon.noname-a 2024-09-16T14:57:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:57:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 1 -- 172.21.15.26:0/1883728389 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d00a34b0 con 0x7fe2d0099b60 2024-09-16T14:57:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient: _finish_auth 0 2024-09-16T14:57:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:57:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 1 -- 172.21.15.26:0/1883728389 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2c0001620 con 0x7fe2d0099b60 2024-09-16T14:57:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient: dump: 2024-09-16T14:57:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.640 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 5 monclient: authenticate success, global_id 274368 2024-09-16T14:57:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 1 -- 172.21.15.26:0/1883728389 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2c0001840 con 0x7fe2d0099b60 2024-09-16T14:57:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 1 -- 172.21.15.26:0/1883728389 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2c0001c10 con 0x7fe2d0099b60 2024-09-16T14:57:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2ce7fc640 10 monclient: dump: 2024-09-16T14:57:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.642 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 4 set_mon_vals no callback set 2024-09-16T14:57:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 10 monclient: shutdown 2024-09-16T14:57:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 1 -- 172.21.15.26:0/1883728389 >> v1:172.21.15.26:6789/0 conn(0x7fe2d0099b60 legacy=0x7fe2d0139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 1 -- 172.21.15.26:0/1883728389 shutdown_connections 2024-09-16T14:57:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 1 -- 172.21.15.26:0/1883728389 >> 172.21.15.26:0/1883728389 conn(0x7fe2d00a3d00 msgr2=0x7fe2d00a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 1 -- 172.21.15.26:0/1883728389 shutdown_connections 2024-09-16T14:57:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 1 -- 172.21.15.26:0/1883728389 wait complete. 2024-09-16T14:57:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 10 monclient: build_initial_monmap 2024-09-16T14:57:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.628+0000 7fe2d6255640 10 monclient: monmap: 2024-09-16T14:57:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:15.644 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 1 Processor -- start 2024-09-16T14:57:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 1 -- start start 2024-09-16T14:57:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 10 monclient: init 2024-09-16T14:57:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:57:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 10 monclient(hunting): picked mon.noname-b con 0x7fe2d0099b60 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 10 monclient(hunting): picked mon.noname-c con 0x7fe2d00a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 10 monclient(hunting): picked mon.noname-a con 0x7fe2d00a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d01c9c90 con 0x7fe2d00a26f0 2024-09-16T14:57:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d009eab0 con 0x7fe2d0099b60 2024-09-16T14:57:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d009e3f0 con 0x7fe2d00a1c00 2024-09-16T14:57:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 10 monclient(hunting): authenticate will time out at 1859.752892s 2024-09-16T14:57:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2ceffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe2d00a1c00 0x7fe2d01ce0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55586/0 (socket says 172.21.15.26:55586) 2024-09-16T14:57:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2ceffd640 1 -- 172.21.15.26:0/1642060384 learned_addr learned my addr 172.21.15.26:0/1642060384 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2cffff640 1 --1- 172.21.15.26:0/1642060384 >> v1:172.21.15.26:6789/0 conn(0x7fe2d00a26f0 0x7fe2d01d17f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39564/0 (socket says 172.21.15.26:39564) 2024-09-16T14:57:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963804049 0 0) 0x7fe2d009e3f0 con 0x7fe2d00a1c00 2024-09-16T14:57:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): my global_id is 284282 2024-09-16T14:57:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2bc003530 con 0x7fe2d00a1c00 2024-09-16T14:57:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320382526 0 0) 0x7fe2d01c9c90 con 0x7fe2d00a26f0 2024-09-16T14:57:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): my global_id is 274371 2024-09-16T14:57:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d009e3f0 con 0x7fe2d00a26f0 2024-09-16T14:57:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2994186429 0 0) 0x7fe2bc003530 con 0x7fe2d00a1c00 2024-09-16T14:57:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2d01c9c90 con 0x7fe2d00a1c00 2024-09-16T14:57:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2c0001c10 con 0x7fe2d00a1c00 2024-09-16T14:57:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): dump: 2024-09-16T14:57:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.647 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2699112191 0 0) 0x7fe2d009e3f0 con 0x7fe2d00a26f0 2024-09-16T14:57:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2bc003530 con 0x7fe2d00a26f0 2024-09-16T14:57:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006970742 0 0) 0x7fe2d009eab0 con 0x7fe2d0099b60 2024-09-16T14:57:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): my global_id is 294133 2024-09-16T14:57:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d009e3f0 con 0x7fe2d0099b60 2024-09-16T14:57:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2c8001d00 con 0x7fe2d00a26f0 2024-09-16T14:57:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): dump: 2024-09-16T14:57:15.649 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.649 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.649 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.649 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.649 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.649 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.649 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.649 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.649 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.649 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2116946222 0 0) 0x7fe2d01c9c90 con 0x7fe2d00a1c00 2024-09-16T14:57:15.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 >> v1:172.21.15.189:6789/0 conn(0x7fe2d0099b60 legacy=0x7fe2d0137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 >> v1:172.21.15.26:6789/0 conn(0x7fe2d00a26f0 legacy=0x7fe2d01d17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient: _finish_hunting 0 2024-09-16T14:57:15.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 monclient: found mon.c 2024-09-16T14:57:15.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:15.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:15.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d011e450 con 0x7fe2d00a1c00 2024-09-16T14:57:15.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient: _finish_auth 0 2024-09-16T14:57:15.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient: _check_auth_tickets 2024-09-16T14:57:15.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:15.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 5 monclient: authenticate success, global_id 284282 2024-09-16T14:57:15.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 10 monclient: _renew_subs 2024-09-16T14:57:15.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:15.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 1 -- 172.21.15.26:0/1642060384 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d011f5c0 con 0x7fe2d00a1c00 2024-09-16T14:57:15.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 10 monclient: _renew_subs 2024-09-16T14:57:15.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 1 -- 172.21.15.26:0/1642060384 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d013e710 con 0x7fe2d00a1c00 2024-09-16T14:57:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2c0004aa0 con 0x7fe2d00a1c00 2024-09-16T14:57:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2cdffb640 4 set_mon_vals no callback set 2024-09-16T14:57:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2c0004e30 con 0x7fe2d00a1c00 2024-09-16T14:57:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient: dump: 2024-09-16T14:57:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.652 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe2c0005010 con 0x7fe2d00a1c00 2024-09-16T14:57:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2d6255640 1 -- 172.21.15.26:0/1642060384 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe2b0006590 con 0x7fe2b0002830 2024-09-16T14:57:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fe2c0043f40 con 0x7fe2d00a1c00 2024-09-16T14:57:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702622223 0 0) 0x7fe2b0006590 con 0x7fe2b0002830 2024-09-16T14:57:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient(hunting): my global_id is 274374 2024-09-16T14:57:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d01c9c90 con 0x7fe2b0002830 2024-09-16T14:57:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1994287136 0 0) 0x7fe2d01c9c90 con 0x7fe2b0002830 2024-09-16T14:57:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2b0006590 con 0x7fe2b0002830 2024-09-16T14:57:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2c40017a0 con 0x7fe2b0002830 2024-09-16T14:57:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient: dump: 2024-09-16T14:57:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.654 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2cffff640 10 monclient: get_auth_request con 0x7fe2bc036440 auth_method 0 2024-09-16T14:57:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3907287496 0 0) 0x7fe2b0006590 con 0x7fe2b0002830 2024-09-16T14:57:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2b000a0e0 con 0x7fe2b0002830 2024-09-16T14:57:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe2c4001bb0 con 0x7fe2b0002830 2024-09-16T14:57:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 >> v1:172.21.15.26:6789/0 conn(0x7fe2b0002830 legacy=0x7fe2b0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.632+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 >> v1:172.21.15.26:6789/0 conn(0x7fe2b0002830 legacy=0x7fe2b0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2d6255640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:57:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2d6255640 1 -- 172.21.15.26:0/1642060384 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe2b0009da0 con 0x7fe2b00090c0 2024-09-16T14:57:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703289873 0 0) 0x7fe2b0009da0 con 0x7fe2b00090c0 2024-09-16T14:57:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 10 monclient(hunting): my global_id is 274377 2024-09-16T14:57:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b0006590 con 0x7fe2b00090c0 2024-09-16T14:57:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3604039359 0 0) 0x7fe2b0006590 con 0x7fe2b00090c0 2024-09-16T14:57:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2b0009da0 con 0x7fe2b00090c0 2024-09-16T14:57:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2c4003990 con 0x7fe2b00090c0 2024-09-16T14:57:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 10 monclient: dump: 2024-09-16T14:57:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.658 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4078759078 0 0) 0x7fe2b0009da0 con 0x7fe2b00090c0 2024-09-16T14:57:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe2b000c290 con 0x7fe2b00090c0 2024-09-16T14:57:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 3509412449) 0x7fe2c4005080 con 0x7fe2b00090c0 2024-09-16T14:57:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:57:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 >> v1:172.21.15.26:6789/0 conn(0x7fe2b00090c0 legacy=0x7fe2b0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2b7fff640 1 -- 172.21.15.26:0/1642060384 >> v1:172.21.15.26:6789/0 conn(0x7fe2b00090c0 legacy=0x7fe2b0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:15.659 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:57:15.659 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:15.659 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:15.659 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:57:15.659 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 138, 2024-09-16T14:57:15.659 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:57:15.659 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:57:15.659 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:57:15.659 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:57:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:57:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:57:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:57:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:15.660 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:57:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:57:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:15.661 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:57:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:57:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:57:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:57:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:57:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:57:15.662 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:57:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:57:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:57:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:57:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:57:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:57:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:57:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:15.663 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:57:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:15.664 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:57:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:57:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:57:15.665 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:57:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:57:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:15.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:57:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:57:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:57:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:15.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:57:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:57:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:57:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:57:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:57:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:15.669 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:57:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:57:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:57:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:15.670 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:57:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:57:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.671 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:57:15.671 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:57:15.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2d6255640 1 -- 172.21.15.26:0/1642060384 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe2bc036440 legacy=0x7fe2bc038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2d6255640 10 monclient: shutdown 2024-09-16T14:57:15.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2d6255640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:15.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2d6255640 1 -- 172.21.15.26:0/1642060384 >> v1:172.21.15.26:6790/0 conn(0x7fe2d00a1c00 legacy=0x7fe2d01ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2cffff640 1 -- 172.21.15.26:0/1642060384 reap_dead start 2024-09-16T14:57:15.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2cffff640 1 -- 172.21.15.26:0/1642060384 reap_dead start 2024-09-16T14:57:15.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2d6255640 1 -- 172.21.15.26:0/1642060384 shutdown_connections 2024-09-16T14:57:15.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2d6255640 1 -- 172.21.15.26:0/1642060384 >> 172.21.15.26:0/1642060384 conn(0x7fe2d00a3d00 msgr2=0x7fe2d00a0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:15.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2d6255640 1 -- 172.21.15.26:0/1642060384 shutdown_connections 2024-09-16T14:57:15.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.640+0000 7fe2d6255640 1 -- 172.21.15.26:0/1642060384 wait complete. 2024-09-16T14:57:15.674 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:57:15.674 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:57:15.674 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:57:15.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.796+0000 7fe185ef3640 10 monclient: get_monmap_and_config 2024-09-16T14:57:15.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.796+0000 7fe185ef3640 10 monclient: build_initial_monmap 2024-09-16T14:57:15.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.796+0000 7fe185ef3640 10 monclient: monmap: 2024-09-16T14:57:15.799 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:15.799 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.799 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:15.800770+0000 2024-09-16T14:57:15.799 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:15.800770+0000 2024-09-16T14:57:15.800 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:15.800 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.800 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:15.800 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:15.800 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:15.800 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.796+0000 7fe185ef3640 1 Processor -- start 2024-09-16T14:57:15.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 1 -- start start 2024-09-16T14:57:15.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient: init 2024-09-16T14:57:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:57:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient(hunting): picked mon.noname-b con 0x7fe18009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient(hunting): picked mon.noname-a con 0x7fe180132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient(hunting): picked mon.noname-c con 0x7fe180138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18009d260 con 0x7fe180132650 2024-09-16T14:57:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe180132a80 con 0x7fe180138ff0 2024-09-16T14:57:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18009cf60 con 0x7fe18009cb80 2024-09-16T14:57:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient(hunting): authenticate will time out at 1859.920673s 2024-09-16T14:57:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17effd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe180132650 0x7fe1801378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39608/0 (socket says 172.21.15.26:39608) 2024-09-16T14:57:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17effd640 1 -- 172.21.15.26:0/2560013852 learned_addr learned my addr 172.21.15.26:0/2560013852 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 1 -- 172.21.15.26:0/2560013852 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490459572 0 0) 0x7fe18009d260 con 0x7fe180132650 2024-09-16T14:57:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient(hunting): my global_id is 274380 2024-09-16T14:57:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 1 -- 172.21.15.26:0/2560013852 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe168003530 con 0x7fe180132650 2024-09-16T14:57:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 1 -- 172.21.15.26:0/2560013852 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205415760 0 0) 0x7fe18009cf60 con 0x7fe18009cb80 2024-09-16T14:57:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient(hunting): my global_id is 284285 2024-09-16T14:57:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 1 -- 172.21.15.26:0/2560013852 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe18009d260 con 0x7fe18009cb80 2024-09-16T14:57:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 1 -- 172.21.15.26:0/2560013852 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2895193508 0 0) 0x7fe18009d260 con 0x7fe18009cb80 2024-09-16T14:57:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 1 -- 172.21.15.26:0/2560013852 >> v1:172.21.15.189:6789/0 conn(0x7fe180138ff0 legacy=0x7fe18013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 1 -- 172.21.15.26:0/2560013852 >> v1:172.21.15.26:6789/0 conn(0x7fe180132650 legacy=0x7fe1801378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:57:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 1 monclient: found mon.noname-b 2024-09-16T14:57:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:57:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 1 -- 172.21.15.26:0/2560013852 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18009f2d0 con 0x7fe18009cb80 2024-09-16T14:57:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:57:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:57:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 1 -- 172.21.15.26:0/2560013852 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe170001660 con 0x7fe18009cb80 2024-09-16T14:57:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:57:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient: dump: 2024-09-16T14:57:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.810 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 5 monclient: authenticate success, global_id 284285 2024-09-16T14:57:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 1 -- 172.21.15.26:0/2560013852 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe170001840 con 0x7fe18009cb80 2024-09-16T14:57:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 1 -- 172.21.15.26:0/2560013852 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe170001bd0 con 0x7fe18009cb80 2024-09-16T14:57:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 4 set_mon_vals no callback set 2024-09-16T14:57:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17e7fc640 10 monclient: dump: 2024-09-16T14:57:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.812 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient: shutdown 2024-09-16T14:57:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 1 -- 172.21.15.26:0/2560013852 >> v1:172.21.15.26:6790/0 conn(0x7fe18009cb80 legacy=0x7fe1800717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 1 -- 172.21.15.26:0/2560013852 shutdown_connections 2024-09-16T14:57:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 1 -- 172.21.15.26:0/2560013852 >> 172.21.15.26:0/2560013852 conn(0x7fe18009fb50 msgr2=0x7fe18009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 1 -- 172.21.15.26:0/2560013852 shutdown_connections 2024-09-16T14:57:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 1 -- 172.21.15.26:0/2560013852 wait complete. 2024-09-16T14:57:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient: build_initial_monmap 2024-09-16T14:57:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient: monmap: 2024-09-16T14:57:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:15.814 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:15.814 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:15.814 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 1 Processor -- start 2024-09-16T14:57:15.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 1 -- start start 2024-09-16T14:57:15.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient: init 2024-09-16T14:57:15.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:15.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:57:15.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient(hunting): picked mon.noname-a con 0x7fe18009cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:15.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient(hunting): picked mon.noname-c con 0x7fe180132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:15.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient(hunting): picked mon.noname-b con 0x7fe180138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:15.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18009a240 con 0x7fe18009cb80 2024-09-16T14:57:15.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18009b370 con 0x7fe180138ff0 2024-09-16T14:57:15.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1800a2170 con 0x7fe180132650 2024-09-16T14:57:15.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:15.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe185ef3640 10 monclient(hunting): authenticate will time out at 1859.923898s 2024-09-16T14:57:15.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17effd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe180132650 0x7fe18009ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55610/0 (socket says 172.21.15.26:55610) 2024-09-16T14:57:15.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17effd640 1 -- 172.21.15.26:0/147369201 learned_addr learned my addr 172.21.15.26:0/147369201 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:15.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.800+0000 7fe17f7fe640 1 --1- 172.21.15.26:0/147369201 >> v1:172.21.15.26:6789/0 conn(0x7fe18009cb80 0x7fe18009a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39616/0 (socket says 172.21.15.26:39616) 2024-09-16T14:57:15.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375464721 0 0) 0x7fe1800a2170 con 0x7fe180132650 2024-09-16T14:57:15.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): my global_id is 284288 2024-09-16T14:57:15.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe16c003530 con 0x7fe180132650 2024-09-16T14:57:15.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287436369 0 0) 0x7fe18009a240 con 0x7fe18009cb80 2024-09-16T14:57:15.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): my global_id is 274383 2024-09-16T14:57:15.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1800a2170 con 0x7fe18009cb80 2024-09-16T14:57:15.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 490975911 0 0) 0x7fe16c003530 con 0x7fe180132650 2024-09-16T14:57:15.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe18009a240 con 0x7fe180132650 2024-09-16T14:57:15.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe1740018e0 con 0x7fe180132650 2024-09-16T14:57:15.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:15.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): dump: 2024-09-16T14:57:15.816 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.816 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.817 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.817 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.817 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.817 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.817 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.817 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.817 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.817 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:15.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3309125989 0 0) 0x7fe1800a2170 con 0x7fe18009cb80 2024-09-16T14:57:15.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe16c003530 con 0x7fe18009cb80 2024-09-16T14:57:15.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe170001b50 con 0x7fe18009cb80 2024-09-16T14:57:15.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:15.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): dump: 2024-09-16T14:57:15.818 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.818 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.818 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.818 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.818 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.818 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.818 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.818 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.818 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.819 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054003523 0 0) 0x7fe18009b370 con 0x7fe180138ff0 2024-09-16T14:57:15.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): my global_id is 294139 2024-09-16T14:57:15.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1800a2170 con 0x7fe180138ff0 2024-09-16T14:57:15.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 362224204 0 0) 0x7fe18009a240 con 0x7fe180132650 2024-09-16T14:57:15.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 >> v1:172.21.15.189:6789/0 conn(0x7fe180138ff0 legacy=0x7fe1801d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 >> v1:172.21.15.26:6789/0 conn(0x7fe18009cb80 legacy=0x7fe18009a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient: _finish_hunting 0 2024-09-16T14:57:15.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 monclient: found mon.c 2024-09-16T14:57:15.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18013c5b0 con 0x7fe180132650 2024-09-16T14:57:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient: _finish_auth 0 2024-09-16T14:57:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient: _check_auth_tickets 2024-09-16T14:57:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe185ef3640 5 monclient: authenticate success, global_id 284288 2024-09-16T14:57:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe185ef3640 10 monclient: _renew_subs 2024-09-16T14:57:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe185ef3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe185ef3640 1 -- 172.21.15.26:0/147369201 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1800713d0 con 0x7fe180132650 2024-09-16T14:57:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe185ef3640 10 monclient: _renew_subs 2024-09-16T14:57:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe185ef3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe185ef3640 1 -- 172.21.15.26:0/147369201 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe18006da40 con 0x7fe180132650 2024-09-16T14:57:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1740047f0 con 0x7fe180132650 2024-09-16T14:57:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe174004b80 con 0x7fe180132650 2024-09-16T14:57:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe17d7fa640 4 set_mon_vals no callback set 2024-09-16T14:57:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient: dump: 2024-09-16T14:57:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.822 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe185ef3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe185ef3640 1 -- 172.21.15.26:0/147369201 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe160006590 con 0x7fe160002830 2024-09-16T14:57:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe174004d60 con 0x7fe180132650 2024-09-16T14:57:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fe174043de0 con 0x7fe180132650 2024-09-16T14:57:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154770810 0 0) 0x7fe160006590 con 0x7fe160002830 2024-09-16T14:57:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient(hunting): my global_id is 284291 2024-09-16T14:57:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe18009a240 con 0x7fe160002830 2024-09-16T14:57:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2440895770 0 0) 0x7fe18009a240 con 0x7fe160002830 2024-09-16T14:57:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe160006590 con 0x7fe160002830 2024-09-16T14:57:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe170003a10 con 0x7fe160002830 2024-09-16T14:57:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient: dump: 2024-09-16T14:57:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.824 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe17ffff640 10 monclient: get_auth_request con 0x7fe16c0363b0 auth_method 0 2024-09-16T14:57:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 899511484 0 0) 0x7fe160006590 con 0x7fe160002830 2024-09-16T14:57:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe16000a0e0 con 0x7fe160002830 2024-09-16T14:57:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe170005790 con 0x7fe160002830 2024-09-16T14:57:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 >> v1:172.21.15.26:6790/0 conn(0x7fe160002830 legacy=0x7fe160004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.804+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 >> v1:172.21.15.26:6790/0 conn(0x7fe160002830 legacy=0x7fe160004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe185ef3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:57:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe185ef3640 1 -- 172.21.15.26:0/147369201 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe160009da0 con 0x7fe1600090c0 2024-09-16T14:57:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982783074 0 0) 0x7fe160009da0 con 0x7fe1600090c0 2024-09-16T14:57:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 10 monclient(hunting): my global_id is 284294 2024-09-16T14:57:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe160006590 con 0x7fe1600090c0 2024-09-16T14:57:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 553189476 0 0) 0x7fe160006590 con 0x7fe1600090c0 2024-09-16T14:57:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe160009da0 con 0x7fe1600090c0 2024-09-16T14:57:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe170004110 con 0x7fe1600090c0 2024-09-16T14:57:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 10 monclient: dump: 2024-09-16T14:57:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.828 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1305854237 0 0) 0x7fe160009da0 con 0x7fe1600090c0 2024-09-16T14:57:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe16000c290 con 0x7fe1600090c0 2024-09-16T14:57:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 262739162) 0x7fe1700043d0 con 0x7fe1600090c0 2024-09-16T14:57:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:57:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 >> v1:172.21.15.26:6790/0 conn(0x7fe1600090c0 legacy=0x7fe160009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe167fff640 1 -- 172.21.15.26:0/147369201 >> v1:172.21.15.26:6790/0 conn(0x7fe1600090c0 legacy=0x7fe160009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:15.828 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:57:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:57:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 138, 2024-09-16T14:57:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:57:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:57:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:57:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:57:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:57:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:15.829 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:57:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:57:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:57:15.830 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:57:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.831 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:57:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:57:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:57:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:57:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:57:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:57:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:57:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:57:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:57:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:57:15.832 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:57:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:57:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:57:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:57:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:15.833 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:57:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:57:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:15.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:57:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:57:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:57:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:15.835 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:57:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:57:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:57:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:15.836 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:57:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:15.837 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:57:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:57:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:57:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:57:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:57:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:57:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:57:15.840 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:57:15.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe185ef3640 1 -- 172.21.15.26:0/147369201 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe16c0363b0 legacy=0x7fe16c038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe185ef3640 10 monclient: shutdown 2024-09-16T14:57:15.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe185ef3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:15.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe185ef3640 1 -- 172.21.15.26:0/147369201 >> v1:172.21.15.26:6790/0 conn(0x7fe180132650 legacy=0x7fe18009ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe17ffff640 1 -- 172.21.15.26:0/147369201 reap_dead start 2024-09-16T14:57:15.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe17ffff640 1 -- 172.21.15.26:0/147369201 reap_dead start 2024-09-16T14:57:15.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe185ef3640 1 -- 172.21.15.26:0/147369201 shutdown_connections 2024-09-16T14:57:15.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe185ef3640 1 -- 172.21.15.26:0/147369201 >> 172.21.15.26:0/147369201 conn(0x7fe18009fb50 msgr2=0x7fe1800a0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:15.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe185ef3640 1 -- 172.21.15.26:0/147369201 shutdown_connections 2024-09-16T14:57:15.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.812+0000 7fe185ef3640 1 -- 172.21.15.26:0/147369201 wait complete. 2024-09-16T14:57:15.842 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:57:15.842 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:57:15.842 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:57:15.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb03d64f640 10 monclient: get_monmap_and_config 2024-09-16T14:57:15.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb03d64f640 10 monclient: build_initial_monmap 2024-09-16T14:57:15.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb03d64f640 10 monclient: monmap: 2024-09-16T14:57:15.961 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:15.961 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.961 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:15.963340+0000 2024-09-16T14:57:15.961 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:15.963340+0000 2024-09-16T14:57:15.961 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:15.962 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb03d64f640 1 Processor -- start 2024-09-16T14:57:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb03d64f640 1 -- start start 2024-09-16T14:57:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb03d64f640 10 monclient: init 2024-09-16T14:57:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb03d64f640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb03d64f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:57:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb03d64f640 10 monclient(hunting): picked mon.noname-c con 0x7fb0380a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb03d64f640 10 monclient(hunting): picked mon.noname-b con 0x7fb0380a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb03d64f640 10 monclient(hunting): picked mon.noname-a con 0x7fb0380a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb03d64f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0380a3250 con 0x7fb0380a2800 2024-09-16T14:57:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb03d64f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03809f600 con 0x7fb0380a1d10 2024-09-16T14:57:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb03d64f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0380a3830 con 0x7fb0380a0000 2024-09-16T14:57:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb03d64f640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb03d64f640 10 monclient(hunting): authenticate will time out at 1860.083471s 2024-09-16T14:57:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb0367fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb0380a0000 0x7fb038139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55638/0 (socket says 172.21.15.26:55638) 2024-09-16T14:57:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb0367fc640 1 -- 172.21.15.26:0/4023423564 learned_addr learned my addr 172.21.15.26:0/4023423564 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.960+0000 7fb0377fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb0380a2800 0x7fb03813d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39620/0 (socket says 172.21.15.26:39620) 2024-09-16T14:57:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 1 -- 172.21.15.26:0/4023423564 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951974823 0 0) 0x7fb0380a3830 con 0x7fb0380a0000 2024-09-16T14:57:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient(hunting): my global_id is 284297 2024-09-16T14:57:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 1 -- 172.21.15.26:0/4023423564 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb024003530 con 0x7fb0380a0000 2024-09-16T14:57:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 1 -- 172.21.15.26:0/4023423564 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465191931 0 0) 0x7fb0380a3250 con 0x7fb0380a2800 2024-09-16T14:57:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient(hunting): my global_id is 274386 2024-09-16T14:57:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 1 -- 172.21.15.26:0/4023423564 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0380a3830 con 0x7fb0380a2800 2024-09-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 1 -- 172.21.15.26:0/4023423564 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3044823961 0 0) 0x7fb03809f600 con 0x7fb0380a1d10 2024-09-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient(hunting): my global_id is 294142 2024-09-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 1 -- 172.21.15.26:0/4023423564 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0380a3250 con 0x7fb0380a1d10 2024-09-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 1 -- 172.21.15.26:0/4023423564 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3601458842 0 0) 0x7fb024003530 con 0x7fb0380a0000 2024-09-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 1 -- 172.21.15.26:0/4023423564 >> v1:172.21.15.189:6789/0 conn(0x7fb0380a1d10 legacy=0x7fb0380a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 1 -- 172.21.15.26:0/4023423564 >> v1:172.21.15.26:6789/0 conn(0x7fb0380a2800 legacy=0x7fb03813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 1 monclient: found mon.noname-b 2024-09-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:57:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 1 -- 172.21.15.26:0/4023423564 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03809ed60 con 0x7fb0380a0000 2024-09-16T14:57:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient: _finish_auth 0 2024-09-16T14:57:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:57:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 1 -- 172.21.15.26:0/4023423564 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb020001620 con 0x7fb0380a0000 2024-09-16T14:57:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:57:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient: dump: 2024-09-16T14:57:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.969 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 5 monclient: authenticate success, global_id 284297 2024-09-16T14:57:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 1 -- 172.21.15.26:0/4023423564 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb020001840 con 0x7fb0380a0000 2024-09-16T14:57:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 1 -- 172.21.15.26:0/4023423564 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb020001c10 con 0x7fb0380a0000 2024-09-16T14:57:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 4 set_mon_vals no callback set 2024-09-16T14:57:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb035ffb640 10 monclient: dump: 2024-09-16T14:57:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.970 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 10 monclient: shutdown 2024-09-16T14:57:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 1 -- 172.21.15.26:0/4023423564 >> v1:172.21.15.26:6790/0 conn(0x7fb0380a0000 legacy=0x7fb038139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 1 -- 172.21.15.26:0/4023423564 shutdown_connections 2024-09-16T14:57:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 1 -- 172.21.15.26:0/4023423564 >> 172.21.15.26:0/4023423564 conn(0x7fb0380a7f30 msgr2=0x7fb0380a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 1 -- 172.21.15.26:0/4023423564 shutdown_connections 2024-09-16T14:57:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 1 -- 172.21.15.26:0/4023423564 wait complete. 2024-09-16T14:57:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 10 monclient: build_initial_monmap 2024-09-16T14:57:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 10 monclient: monmap: 2024-09-16T14:57:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:15.972 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 1 Processor -- start 2024-09-16T14:57:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 1 -- start start 2024-09-16T14:57:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 10 monclient: init 2024-09-16T14:57:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:57:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 10 monclient(hunting): picked mon.noname-a con 0x7fb0380a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 10 monclient(hunting): picked mon.noname-b con 0x7fb0380a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 10 monclient(hunting): picked mon.noname-c con 0x7fb0380a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03813ccc0 con 0x7fb0380a0000 2024-09-16T14:57:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0380a6530 con 0x7fb0380a1d10 2024-09-16T14:57:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03809e500 con 0x7fb0380a2800 2024-09-16T14:57:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 10 monclient(hunting): authenticate will time out at 1860.086788s 2024-09-16T14:57:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb036ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb0380a0000 0x7fb03813d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39626/0 (socket says 172.21.15.26:39626) 2024-09-16T14:57:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb0377fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb0380a2800 0x7fb0381d10b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55642/0 (socket says 172.21.15.26:55642) 2024-09-16T14:57:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb036ffd640 1 -- 172.21.15.26:0/4015100702 learned_addr learned my addr 172.21.15.26:0/4015100702 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268385002 0 0) 0x7fb0380a3250 con 0x7fb0380a0000 2024-09-16T14:57:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): my global_id is 274389 2024-09-16T14:57:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb018003530 con 0x7fb0380a0000 2024-09-16T14:57:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162678985 0 0) 0x7fb03809e500 con 0x7fb0380a2800 2024-09-16T14:57:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): my global_id is 284300 2024-09-16T14:57:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0380a3250 con 0x7fb0380a2800 2024-09-16T14:57:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3891574151 0 0) 0x7fb018003530 con 0x7fb0380a0000 2024-09-16T14:57:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb03809e500 con 0x7fb0380a0000 2024-09-16T14:57:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb030001b80 con 0x7fb0380a0000 2024-09-16T14:57:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): dump: 2024-09-16T14:57:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.980 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 982029443 0 0) 0x7fb0380a3250 con 0x7fb0380a2800 2024-09-16T14:57:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb018003530 con 0x7fb0380a2800 2024-09-16T14:57:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb0280019e0 con 0x7fb0380a2800 2024-09-16T14:57:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): dump: 2024-09-16T14:57:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.981 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741863686 0 0) 0x7fb0380a6530 con 0x7fb0380a1d10 2024-09-16T14:57:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient(hunting): my global_id is 294145 2024-09-16T14:57:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0380a3250 con 0x7fb0380a1d10 2024-09-16T14:57:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1934458101 0 0) 0x7fb03809e500 con 0x7fb0380a0000 2024-09-16T14:57:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 >> v1:172.21.15.26:6790/0 conn(0x7fb0380a2800 legacy=0x7fb0381d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 >> v1:172.21.15.189:6789/0 conn(0x7fb0380a1d10 legacy=0x7fb0381cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:57:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 monclient: found mon.a 2024-09-16T14:57:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb038134a20 con 0x7fb0380a0000 2024-09-16T14:57:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:57:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:57:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 5 monclient: authenticate success, global_id 274389 2024-09-16T14:57:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 10 monclient: _renew_subs 2024-09-16T14:57:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 1 -- 172.21.15.26:0/4015100702 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb038136700 con 0x7fb0380a0000 2024-09-16T14:57:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 10 monclient: _renew_subs 2024-09-16T14:57:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb03d64f640 1 -- 172.21.15.26:0/4015100702 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0381355f0 con 0x7fb0380a0000 2024-09-16T14:57:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0300039a0 con 0x7fb0380a0000 2024-09-16T14:57:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.964+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb030003d30 con 0x7fb0380a0000 2024-09-16T14:57:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb0357fa640 4 set_mon_vals no callback set 2024-09-16T14:57:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 10 monclient: dump: 2024-09-16T14:57:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.984 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb03d64f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb03d64f640 1 -- 172.21.15.26:0/4015100702 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb01c006590 con 0x7fb01c002830 2024-09-16T14:57:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb030003e90 con 0x7fb0380a0000 2024-09-16T14:57:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fb030043e70 con 0x7fb0380a0000 2024-09-16T14:57:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815154120 0 0) 0x7fb01c006590 con 0x7fb01c002830 2024-09-16T14:57:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 10 monclient(hunting): my global_id is 294148 2024-09-16T14:57:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb03809e500 con 0x7fb01c002830 2024-09-16T14:57:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb0377fe640 10 monclient: get_auth_request con 0x7fb018036420 auth_method 0 2024-09-16T14:57:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2573692901 0 0) 0x7fb03809e500 con 0x7fb01c002830 2024-09-16T14:57:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb01c006590 con 0x7fb01c002830 2024-09-16T14:57:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb020001230 con 0x7fb01c002830 2024-09-16T14:57:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:57:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 10 monclient: dump: 2024-09-16T14:57:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.989 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2320199663 0 0) 0x7fb01c006590 con 0x7fb01c002830 2024-09-16T14:57:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb01c00a0e0 con 0x7fb01c002830 2024-09-16T14:57:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fb020001400 con 0x7fb01c002830 2024-09-16T14:57:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 >> v1:172.21.15.189:6789/0 conn(0x7fb01c002830 legacy=0x7fb01c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.968+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 >> v1:172.21.15.189:6789/0 conn(0x7fb01c002830 legacy=0x7fb01c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb03d64f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:57:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb03d64f640 1 -- 172.21.15.26:0/4015100702 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb01c009da0 con 0x7fb01c0090c0 2024-09-16T14:57:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787673914 0 0) 0x7fb01c009da0 con 0x7fb01c0090c0 2024-09-16T14:57:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:57:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 10 monclient(hunting): my global_id is 294151 2024-09-16T14:57:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb01c006590 con 0x7fb01c0090c0 2024-09-16T14:57:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 217022773 0 0) 0x7fb01c006590 con 0x7fb01c0090c0 2024-09-16T14:57:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb01c009da0 con 0x7fb01c0090c0 2024-09-16T14:57:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb020004a10 con 0x7fb01c0090c0 2024-09-16T14:57:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:57:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 10 monclient: dump: 2024-09-16T14:57:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:15.992 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1363423038 0 0) 0x7fb01c009da0 con 0x7fb01c0090c0 2024-09-16T14:57:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb01c00c290 con 0x7fb01c0090c0 2024-09-16T14:57:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3067852984) 0x7fb020005f30 con 0x7fb01c0090c0 2024-09-16T14:57:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:57:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 >> v1:172.21.15.189:6789/0 conn(0x7fb01c0090c0 legacy=0x7fb01c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb02f7fe640 1 -- 172.21.15.26:0/4015100702 >> v1:172.21.15.189:6789/0 conn(0x7fb01c0090c0 legacy=0x7fb01c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:15.993 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:57:15.993 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:15.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:15.994 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:57:15.994 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 138, 2024-09-16T14:57:15.994 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:57:15.994 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:57:15.994 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:57:15.994 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:57:15.994 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:15.994 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:57:15.994 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:15.994 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:57:15.994 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:57:15.995 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:15.995 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:15.995 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:15.995 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:15.995 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:15.995 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:15.995 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:15.995 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:15.995 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:15.995 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:15.995 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:15.995 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:57:15.996 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:57:15.996 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:15.996 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:15.996 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:15.996 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:15.996 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:15.996 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:15.996 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:15.996 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:15.996 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:15.996 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:15.996 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:15.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.997 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:57:15.997 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:57:15.997 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:57:15.997 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:57:15.997 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:57:15.997 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:57:15.997 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:15.997 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:15.997 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:57:15.997 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:57:15.997 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:57:15.998 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:57:15.998 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:57:15.998 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:57:15.998 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:57:15.998 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:15.998 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:57:15.998 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:15.998 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:15.998 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:15.998 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:15.998 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:15.998 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:15.998 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:57:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:57:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:57:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:57:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:57:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:57:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:57:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:57:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:57:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:57:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:57:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:57:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:57:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:57:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:16.005 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:57:16.005 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:57:16.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb03d64f640 1 -- 172.21.15.26:0/4015100702 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb018036420 legacy=0x7fb0180388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:16.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb03d64f640 10 monclient: shutdown 2024-09-16T14:57:16.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb03d64f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:16.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb03d64f640 1 -- 172.21.15.26:0/4015100702 >> v1:172.21.15.26:6789/0 conn(0x7fb0380a0000 legacy=0x7fb03813d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:16.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb0377fe640 1 -- 172.21.15.26:0/4015100702 reap_dead start 2024-09-16T14:57:16.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb0377fe640 1 -- 172.21.15.26:0/4015100702 reap_dead start 2024-09-16T14:57:16.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb03d64f640 1 -- 172.21.15.26:0/4015100702 shutdown_connections 2024-09-16T14:57:16.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb03d64f640 1 -- 172.21.15.26:0/4015100702 >> 172.21.15.26:0/4015100702 conn(0x7fb0380a7f30 msgr2=0x7fb03806b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:16.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb03d64f640 1 -- 172.21.15.26:0/4015100702 shutdown_connections 2024-09-16T14:57:16.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:15.976+0000 7fb03d64f640 1 -- 172.21.15.26:0/4015100702 wait complete. 2024-09-16T14:57:16.007 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:57:16.007 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T14:57:16.007 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:57:16.007 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T14:57:16.007 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:57:16.008 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T14:57:16.008 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:57:16.026 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T14:57:16.026 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:57:16.026 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:57:16.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.136+0000 7f8906ed3640 10 monclient: get_monmap_and_config 2024-09-16T14:57:16.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.136+0000 7f8906ed3640 10 monclient: build_initial_monmap 2024-09-16T14:57:16.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.136+0000 7f8906ed3640 10 monclient: monmap: 2024-09-16T14:57:16.138 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:16.139 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:16.139 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:16.140703+0000 2024-09-16T14:57:16.139 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:16.140703+0000 2024-09-16T14:57:16.139 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:16.139 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:16.139 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:16.139 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:16.139 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:16.139 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:16.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 1 Processor -- start 2024-09-16T14:57:16.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 1 -- start start 2024-09-16T14:57:16.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 10 monclient: init 2024-09-16T14:57:16.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:16.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:57:16.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 10 monclient(hunting): picked mon.noname-c con 0x7f8900070e70 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:16.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 10 monclient(hunting): picked mon.noname-a con 0x7f8900134b80 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:16.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 10 monclient(hunting): picked mon.noname-b con 0x7f890013b0b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:16.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f890009d910 con 0x7f8900134b80 2024-09-16T14:57:16.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8900071960 con 0x7f8900070e70 2024-09-16T14:57:16.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8900135030 con 0x7f890013b0b0 2024-09-16T14:57:16.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:16.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 10 monclient(hunting): authenticate will time out at 1860.261394s 2024-09-16T14:57:16.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8904c6f640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f8900070e70 legacy=0x7f8900071250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:16.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88fffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8900134b80 0x7f8900139990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39632/0 (socket says 172.21.15.26:39632) 2024-09-16T14:57:16.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8905470640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f890013b0b0 0x7f890013d4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55652/0 (socket says 172.21.15.26:55652) 2024-09-16T14:57:16.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88fffff640 1 -- 172.21.15.26:0/2677080741 learned_addr learned my addr 172.21.15.26:0/2677080741 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:16.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 1 -- 172.21.15.26:0/2677080741 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746515786 0 0) 0x7f8900135030 con 0x7f890013b0b0 2024-09-16T14:57:16.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:57:16.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:16.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:16.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient(hunting): my global_id is 284303 2024-09-16T14:57:16.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 1 -- 172.21.15.26:0/2677080741 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88f0003530 con 0x7f890013b0b0 2024-09-16T14:57:16.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 1 -- 172.21.15.26:0/2677080741 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411627226 0 0) 0x7f890009d910 con 0x7f8900134b80 2024-09-16T14:57:16.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:57:16.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:16.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:16.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient(hunting): my global_id is 274392 2024-09-16T14:57:16.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 1 -- 172.21.15.26:0/2677080741 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8900135030 con 0x7f8900134b80 2024-09-16T14:57:16.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 1 -- 172.21.15.26:0/2677080741 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1173167417 0 0) 0x7f88f0003530 con 0x7f890013b0b0 2024-09-16T14:57:16.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 1 -- 172.21.15.26:0/2677080741 >> v1:172.21.15.189:6789/0 conn(0x7f8900070e70 legacy=0x7f8900071250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:57:16.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 1 -- 172.21.15.26:0/2677080741 >> v1:172.21.15.26:6789/0 conn(0x7f8900134b80 legacy=0x7f8900139990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:16.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:57:16.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 1 monclient: found mon.noname-b 2024-09-16T14:57:16.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:16.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:57:16.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 1 -- 172.21.15.26:0/2677080741 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890009d2d0 con 0x7f890013b0b0 2024-09-16T14:57:16.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient: _finish_auth 0 2024-09-16T14:57:16.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:57:16.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:16.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 1 -- 172.21.15.26:0/2677080741 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f88f80018d0 con 0x7f890013b0b0 2024-09-16T14:57:16.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 5 monclient: authenticate success, global_id 284303 2024-09-16T14:57:16.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:16.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:16.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:57:16.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient: dump: 2024-09-16T14:57:16.146 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:16.146 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:16.146 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:16.147 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:16.147 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:16.147 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:16.147 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:16.147 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:16.147 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:16.147 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:16.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:16.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 1 -- 172.21.15.26:0/2677080741 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88f8001ac0 con 0x7f890013b0b0 2024-09-16T14:57:16.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:16.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:16.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 4 set_mon_vals no callback set 2024-09-16T14:57:16.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 1 -- 172.21.15.26:0/2677080741 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f88f8001e90 con 0x7f890013b0b0 2024-09-16T14:57:16.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:16.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:16.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f88ff7fe640 10 monclient: dump: 2024-09-16T14:57:16.148 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:16.148 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:16.148 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:16.148 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:16.148 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:16.148 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:16.148 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:16.148 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:16.148 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:16.148 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:16.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 10 monclient: shutdown 2024-09-16T14:57:16.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:16.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 1 -- 172.21.15.26:0/2677080741 >> v1:172.21.15.26:6790/0 conn(0x7f890013b0b0 legacy=0x7f890013d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:16.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 1 -- 172.21.15.26:0/2677080741 shutdown_connections 2024-09-16T14:57:16.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 1 -- 172.21.15.26:0/2677080741 >> 172.21.15.26:0/2677080741 conn(0x7f89000a03d0 msgr2=0x7f89000a07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:16.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 1 -- 172.21.15.26:0/2677080741 shutdown_connections 2024-09-16T14:57:16.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 1 -- 172.21.15.26:0/2677080741 wait complete. 2024-09-16T14:57:16.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 10 monclient: build_initial_monmap 2024-09-16T14:57:16.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 10 monclient: monmap: 2024-09-16T14:57:16.149 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:16.149 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:16.149 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:16.149 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:16.149 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:16.150 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:16.150 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:16.150 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:16.150 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:16.150 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:16.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 1 Processor -- start 2024-09-16T14:57:16.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 1 -- start start 2024-09-16T14:57:16.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.140+0000 7f8906ed3640 10 monclient: init 2024-09-16T14:57:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:57:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 10 monclient(hunting): picked mon.noname-c con 0x7f8900070e70 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 10 monclient(hunting): picked mon.noname-a con 0x7f8900134b80 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 10 monclient(hunting): picked mon.noname-b con 0x7f890013b0b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f890013a5c0 con 0x7f8900134b80 2024-09-16T14:57:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89000a17d0 con 0x7f890013b0b0 2024-09-16T14:57:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f890013ab40 con 0x7f8900070e70 2024-09-16T14:57:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 10 monclient(hunting): authenticate will time out at 1860.264499s 2024-09-16T14:57:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8904c6f640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8900070e70 0x7f89001c9490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55660/0 (socket says 172.21.15.26:55660) 2024-09-16T14:57:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8904c6f640 1 -- 172.21.15.26:0/1194400434 learned_addr learned my addr 172.21.15.26:0/1194400434 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8905470640 1 -- 172.21.15.26:0/1194400434 >> v1:172.21.15.189:6789/0 conn(0x7f890013b0b0 legacy=0x7f89001d1c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283822519 0 0) 0x7f890013ab40 con 0x7f8900070e70 2024-09-16T14:57:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:57:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient(hunting): my global_id is 284306 2024-09-16T14:57:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e4003560 con 0x7f8900070e70 2024-09-16T14:57:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475173072 0 0) 0x7f890013a5c0 con 0x7f8900134b80 2024-09-16T14:57:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient(hunting): _negotiate 2024-09-16T14:57:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient(hunting): my global_id is 274395 2024-09-16T14:57:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f890013ab40 con 0x7f8900134b80 2024-09-16T14:57:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1870168808 0 0) 0x7f88e4003560 con 0x7f8900070e70 2024-09-16T14:57:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f890013a5c0 con 0x7f8900070e70 2024-09-16T14:57:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f88ec0041e0 con 0x7f8900070e70 2024-09-16T14:57:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient(hunting): dump: 2024-09-16T14:57:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:16.156 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1191724603 0 0) 0x7f890013ab40 con 0x7f8900134b80 2024-09-16T14:57:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88e4003560 con 0x7f8900134b80 2024-09-16T14:57:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f88f4001380 con 0x7f8900134b80 2024-09-16T14:57:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient(hunting): dump: 2024-09-16T14:57:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:16.158 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1451515706 0 0) 0x7f88e4003560 con 0x7f8900134b80 2024-09-16T14:57:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 >> v1:172.21.15.26:6790/0 conn(0x7f8900070e70 legacy=0x7f89001c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 >> v1:172.21.15.189:6789/0 conn(0x7f890013b0b0 legacy=0x7f89001d1c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:57:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient: _finish_hunting 0 2024-09-16T14:57:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 monclient: found mon.a 2024-09-16T14:57:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890011b3d0 con 0x7f8900134b80 2024-09-16T14:57:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient: _finish_auth 0 2024-09-16T14:57:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T14:57:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 5 monclient: authenticate success, global_id 274395 2024-09-16T14:57:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 10 monclient: _renew_subs 2024-09-16T14:57:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 1 -- 172.21.15.26:0/1194400434 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890011b6b0 con 0x7f8900134b80 2024-09-16T14:57:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 10 monclient: _renew_subs 2024-09-16T14:57:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 1 -- 172.21.15.26:0/1194400434 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8900139030 con 0x7f8900134b80 2024-09-16T14:57:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88f40015a0 con 0x7f8900134b80 2024-09-16T14:57:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f88f4004c40 con 0x7f8900134b80 2024-09-16T14:57:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 10 monclient: dump: 2024-09-16T14:57:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:16.160 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88feffd640 4 set_mon_vals no callback set 2024-09-16T14:57:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8906ed3640 1 -- 172.21.15.26:0/1194400434 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e0005050 con 0x7f8900134b80 2024-09-16T14:57:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f88f400f6b0 con 0x7f8900134b80 2024-09-16T14:57:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f88f4043cf0 con 0x7f8900134b80 2024-09-16T14:57:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.144+0000 7f8904c6f640 10 monclient: get_auth_request con 0x7f88e40362e0 auth_method 0 2024-09-16T14:57:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.148+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f88f40143a0 con 0x7f8900134b80 2024-09-16T14:57:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.148+0000 7f88fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.148+0000 7f88fd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:16.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.368+0000 7f8906ed3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:57:16.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.368+0000 7f8906ed3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:16.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.368+0000 7f8906ed3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:16.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.368+0000 7f8906ed3640 1 -- 172.21.15.26:0/1194400434 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f88e0005880 con 0x7f8900134b80 2024-09-16T14:57:16.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.368+0000 7f88fd7fa640 1 -- 172.21.15.26:0/1194400434 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2937780276) 0x7f88f4012a40 con 0x7f8900134b80 2024-09-16T14:57:16.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.368+0000 7f88fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:16.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.368+0000 7f88fd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:16.371 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:57:16.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.372+0000 7f8906ed3640 1 -- 172.21.15.26:0/1194400434 >> v1:172.21.15.189:6812/3196161582 conn(0x7f88e40362e0 legacy=0x7f88e40387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:16.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.372+0000 7f8906ed3640 10 monclient: shutdown 2024-09-16T14:57:16.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.372+0000 7f8906ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:16.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.372+0000 7f8906ed3640 1 -- 172.21.15.26:0/1194400434 >> v1:172.21.15.26:6789/0 conn(0x7f8900134b80 legacy=0x7f89001ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:16.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.372+0000 7f8906ed3640 1 -- 172.21.15.26:0/1194400434 shutdown_connections 2024-09-16T14:57:16.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.372+0000 7f8906ed3640 1 -- 172.21.15.26:0/1194400434 >> 172.21.15.26:0/1194400434 conn(0x7f89000a03d0 msgr2=0x7f89000a1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:16.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.372+0000 7f8906ed3640 1 -- 172.21.15.26:0/1194400434 shutdown_connections 2024-09-16T14:57:16.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:16.372+0000 7f8906ed3640 1 -- 172.21.15.26:0/1194400434 wait complete. 2024-09-16T14:57:19.390 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 10 monclient: get_monmap_and_config 2024-09-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 10 monclient: build_initial_monmap 2024-09-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 10 monclient: monmap: 2024-09-16T14:57:19.524 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:19.526496+0000 2024-09-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:19.526496+0000 2024-09-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:19.525 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 1 Processor -- start 2024-09-16T14:57:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 1 -- start start 2024-09-16T14:57:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 10 monclient: init 2024-09-16T14:57:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:57:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 10 monclient(hunting): picked mon.noname-c con 0x7f5c280a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 10 monclient(hunting): picked mon.noname-a con 0x7f5c280a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 10 monclient(hunting): picked mon.noname-b con 0x7f5c280a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c280a3250 con 0x7f5c280a0000 2024-09-16T14:57:19.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2809f600 con 0x7f5c280a1d10 2024-09-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c280a3830 con 0x7f5c280a2800 2024-09-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 10 monclient(hunting): authenticate will time out at 1863.646692s 2024-09-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2d36f640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f5c280a1d10 legacy=0x7f5c280a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2cb6e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5c280a0000 0x7f5c28139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:49660/0 (socket says 172.21.15.26:49660) 2024-09-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2db70640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5c280a2800 0x7f5c2813d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:52038/0 (socket says 172.21.15.26:52038) 2024-09-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2cb6e640 1 -- 172.21.15.26:0/569075461 learned_addr learned my addr 172.21.15.26:0/569075461 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 1 -- 172.21.15.26:0/569075461 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251034802 0 0) 0x7f5c280a3830 con 0x7f5c280a2800 2024-09-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 10 monclient(hunting): my global_id is 284309 2024-09-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 1 -- 172.21.15.26:0/569075461 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c14003530 con 0x7f5c280a2800 2024-09-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 1 -- 172.21.15.26:0/569075461 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3787142291 0 0) 0x7f5c14003530 con 0x7f5c280a2800 2024-09-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 1 -- 172.21.15.26:0/569075461 >> v1:172.21.15.189:6789/0 conn(0x7f5c280a1d10 legacy=0x7f5c280a20f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 1 -- 172.21.15.26:0/569075461 >> v1:172.21.15.26:6789/0 conn(0x7f5c280a0000 legacy=0x7f5c28139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 1 monclient: found mon.noname-b 2024-09-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 1 -- 172.21.15.26:0/569075461 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2809ed60 con 0x7f5c280a2800 2024-09-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 10 monclient: _finish_auth 0 2024-09-16T14:57:19.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 1 -- 172.21.15.26:0/569075461 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5c200019a0 con 0x7f5c280a2800 2024-09-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 5 monclient: authenticate success, global_id 284309 2024-09-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 10 monclient: dump: 2024-09-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:19.531 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:19.532 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 1 -- 172.21.15.26:0/569075461 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c20001cc0 con 0x7f5c280a2800 2024-09-16T14:57:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 1 -- 172.21.15.26:0/569075461 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5c20002090 con 0x7f5c280a2800 2024-09-16T14:57:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c1ffff640 10 monclient: dump: 2024-09-16T14:57:19.532 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:19.533 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.524+0000 7f5c2f5d3640 4 set_mon_vals no callback set 2024-09-16T14:57:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 monclient: shutdown 2024-09-16T14:57:19.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 1 -- 172.21.15.26:0/569075461 >> v1:172.21.15.26:6790/0 conn(0x7f5c280a2800 legacy=0x7f5c2813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 1 -- 172.21.15.26:0/569075461 shutdown_connections 2024-09-16T14:57:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 1 -- 172.21.15.26:0/569075461 >> 172.21.15.26:0/569075461 conn(0x7f5c280a7f30 msgr2=0x7f5c280a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 1 -- 172.21.15.26:0/569075461 shutdown_connections 2024-09-16T14:57:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 1 -- 172.21.15.26:0/569075461 wait complete. 2024-09-16T14:57:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 monclient: build_initial_monmap 2024-09-16T14:57:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 monclient: monmap: 2024-09-16T14:57:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:19.534 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:19.535 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:19.535 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:19.535 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:19.535 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:19.535 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:19.535 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:19.535 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:19.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 1 Processor -- start 2024-09-16T14:57:19.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 1 -- start start 2024-09-16T14:57:19.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 monclient: init 2024-09-16T14:57:19.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:19.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:57:19.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 monclient(hunting): picked mon.noname-b con 0x7f5c280a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:19.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 monclient(hunting): picked mon.noname-a con 0x7f5c280a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:19.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 monclient(hunting): picked mon.noname-c con 0x7f5c280a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:19.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2809e500 con 0x7f5c280a1d10 2024-09-16T14:57:19.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c280a6650 con 0x7f5c280a0000 2024-09-16T14:57:19.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c28132350 con 0x7f5c280a2800 2024-09-16T14:57:19.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:19.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 monclient(hunting): authenticate will time out at 1863.650079s 2024-09-16T14:57:19.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2db70640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5c280a2800 0x7f5c281d10b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:52050/0 (socket says 172.21.15.26:52050) 2024-09-16T14:57:19.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2db70640 1 -- 172.21.15.26:0/1135074254 learned_addr learned my addr 172.21.15.26:0/1135074254 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:19.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2d36f640 1 -- 172.21.15.26:0/1135074254 >> v1:172.21.15.189:6789/0 conn(0x7f5c280a0000 legacy=0x7f5c28134310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 1 -- 172.21.15.26:0/1135074254 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087232265 0 0) 0x7f5c28132350 con 0x7f5c280a2800 2024-09-16T14:57:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 10 monclient(hunting): my global_id is 284312 2024-09-16T14:57:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 1 -- 172.21.15.26:0/1135074254 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c0c003530 con 0x7f5c280a2800 2024-09-16T14:57:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 1 -- 172.21.15.26:0/1135074254 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 513858127 0 0) 0x7f5c0c003530 con 0x7f5c280a2800 2024-09-16T14:57:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 1 -- 172.21.15.26:0/1135074254 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c28132350 con 0x7f5c280a2800 2024-09-16T14:57:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 1 -- 172.21.15.26:0/1135074254 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5c20001960 con 0x7f5c280a2800 2024-09-16T14:57:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:19.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 10 monclient(hunting): dump: 2024-09-16T14:57:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:19.541 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:19.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 1 -- 172.21.15.26:0/1135074254 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3449310561 0 0) 0x7f5c28132350 con 0x7f5c280a2800 2024-09-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 1 -- 172.21.15.26:0/1135074254 >> v1:172.21.15.189:6789/0 conn(0x7f5c280a0000 legacy=0x7f5c28134310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 1 -- 172.21.15.26:0/1135074254 >> v1:172.21.15.26:6789/0 conn(0x7f5c280a1d10 legacy=0x7f5c281cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 10 monclient: _finish_hunting 0 2024-09-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 1 monclient: found mon.c 2024-09-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 1 -- 172.21.15.26:0/1135074254 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c28133ef0 con 0x7f5c280a2800 2024-09-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 10 monclient: _finish_auth 0 2024-09-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 10 monclient: _check_auth_tickets 2024-09-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 5 monclient: authenticate success, global_id 284312 2024-09-16T14:57:19.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 monclient: _renew_subs 2024-09-16T14:57:19.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:19.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 1 -- 172.21.15.26:0/1135074254 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c28131b40 con 0x7f5c280a2800 2024-09-16T14:57:19.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 monclient: _renew_subs 2024-09-16T14:57:19.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:19.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 1 -- 172.21.15.26:0/1135074254 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c281d27d0 con 0x7f5c280a2800 2024-09-16T14:57:19.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 1 -- 172.21.15.26:0/1135074254 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c20004ec0 con 0x7f5c280a2800 2024-09-16T14:57:19.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:19.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 1 -- 172.21.15.26:0/1135074254 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5c20005250 con 0x7f5c280a2800 2024-09-16T14:57:19.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:19.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:19.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1effd640 4 set_mon_vals no callback set 2024-09-16T14:57:19.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c1dffb640 10 monclient: dump: 2024-09-16T14:57:19.543 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:19.543 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:19.544 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:19.544 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:19.544 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:19.544 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:19.544 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:19.544 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:19.544 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:19.544 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:19.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:19.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.528+0000 7f5c2f5d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:19.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.532+0000 7f5c2f5d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:19.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.532+0000 7f5c2f5d3640 1 -- 172.21.15.26:0/1135074254 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c08005050 con 0x7f5c280a2800 2024-09-16T14:57:19.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.532+0000 7f5c1dffb640 1 -- 172.21.15.26:0/1135074254 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5c200053b0 con 0x7f5c280a2800 2024-09-16T14:57:19.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.532+0000 7f5c1dffb640 1 -- 172.21.15.26:0/1135074254 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f5c20044320 con 0x7f5c280a2800 2024-09-16T14:57:19.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.532+0000 7f5c2d36f640 10 monclient: get_auth_request con 0x7f5c0c036050 auth_method 0 2024-09-16T14:57:19.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.536+0000 7f5c1dffb640 1 -- 172.21.15.26:0/1135074254 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5c20006950 con 0x7f5c280a2800 2024-09-16T14:57:19.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.536+0000 7f5c1dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:19.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.536+0000 7f5c1dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:19.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.752+0000 7f5c2f5d3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:57:19.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.752+0000 7f5c2f5d3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:19.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.752+0000 7f5c2f5d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:19.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.752+0000 7f5c2f5d3640 1 -- 172.21.15.26:0/1135074254 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5c08005880 con 0x7f5c280a2800 2024-09-16T14:57:19.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.752+0000 7f5c1dffb640 1 -- 172.21.15.26:0/1135074254 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 4294325781) 0x7f5c20013190 con 0x7f5c280a2800 2024-09-16T14:57:19.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.752+0000 7f5c1dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:19.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.752+0000 7f5c1dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:19.754 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:57:19.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.752+0000 7f5c2f5d3640 1 -- 172.21.15.26:0/1135074254 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5c0c036050 legacy=0x7f5c0c038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:19.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.752+0000 7f5c2f5d3640 10 monclient: shutdown 2024-09-16T14:57:19.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.752+0000 7f5c2f5d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:19.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.752+0000 7f5c2f5d3640 1 -- 172.21.15.26:0/1135074254 >> v1:172.21.15.26:6790/0 conn(0x7f5c280a2800 legacy=0x7f5c281d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:19.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.752+0000 7f5c2f5d3640 1 -- 172.21.15.26:0/1135074254 shutdown_connections 2024-09-16T14:57:19.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.752+0000 7f5c2f5d3640 1 -- 172.21.15.26:0/1135074254 >> 172.21.15.26:0/1135074254 conn(0x7f5c280a7f30 msgr2=0x7f5c2813e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:19.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.756+0000 7f5c2f5d3640 1 -- 172.21.15.26:0/1135074254 shutdown_connections 2024-09-16T14:57:19.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:19.756+0000 7f5c2f5d3640 1 -- 172.21.15.26:0/1135074254 wait complete. 2024-09-16T14:57:22.771 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:57:22.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb77e7640 10 monclient: get_monmap_and_config 2024-09-16T14:57:22.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb77e7640 10 monclient: build_initial_monmap 2024-09-16T14:57:22.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb77e7640 10 monclient: monmap: 2024-09-16T14:57:22.905 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:22.905 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:22.905 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:22.906998+0000 2024-09-16T14:57:22.905 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:22.906998+0000 2024-09-16T14:57:22.905 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:22.905 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:22.905 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:22.905 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:22.906 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:22.906 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:22.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb77e7640 1 Processor -- start 2024-09-16T14:57:22.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb77e7640 1 -- start start 2024-09-16T14:57:22.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb77e7640 10 monclient: init 2024-09-16T14:57:22.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb77e7640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:22.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb77e7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:57:22.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb77e7640 10 monclient(hunting): picked mon.noname-a con 0x7f0eb009cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:22.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb77e7640 10 monclient(hunting): picked mon.noname-b con 0x7f0eb0132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:22.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb77e7640 10 monclient(hunting): picked mon.noname-c con 0x7f0eb0138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:22.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb77e7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb009d260 con 0x7f0eb009cb80 2024-09-16T14:57:22.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb77e7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb0132a80 con 0x7f0eb0138ff0 2024-09-16T14:57:22.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb77e7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb009cf60 con 0x7f0eb0132650 2024-09-16T14:57:22.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb77e7640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:22.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb77e7640 10 monclient(hunting): authenticate will time out at 1867.027080s 2024-09-16T14:57:22.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb5583640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0eb009cb80 0x7f0eb00717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:49676/0 (socket says 172.21.15.26:49676) 2024-09-16T14:57:22.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb5583640 1 -- 172.21.15.26:0/1381815720 learned_addr learned my addr 172.21.15.26:0/1381815720 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:22.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:22.904+0000 7f0eb5d84640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.189:6789/0 conn(0x7f0eb0138ff0 legacy=0x7f0eb013b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:23.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:23.108+0000 7f0eb5d84640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.189:6789/0 conn(0x7f0eb0138ff0 legacy=0x7f0eb013b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:23.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:23.512+0000 7f0eb5d84640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.189:6789/0 conn(0x7f0eb0138ff0 legacy=0x7f0eb013b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:24.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:24.316+0000 7f0eb5d84640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.189:6789/0 conn(0x7f0eb0138ff0 legacy=0x7f0eb013b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:25.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:25.904+0000 7f0eaeffd640 10 monclient(hunting): tick 2024-09-16T14:57:25.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:25.904+0000 7f0eaeffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:57:25.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:25.904+0000 7f0eaeffd640 1 monclient(hunting): continuing hunt 2024-09-16T14:57:25.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:25.904+0000 7f0eaeffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:57:25.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:25.904+0000 7f0eaeffd640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.26:6790/0 conn(0x7f0eb0132650 legacy=0x7f0eb01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:25.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:25.904+0000 7f0eaeffd640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.189:6789/0 conn(0x7f0eb0138ff0 legacy=0x7f0eb013b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:57:25.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:25.904+0000 7f0eaeffd640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.26:6789/0 conn(0x7f0eb009cb80 legacy=0x7f0eb00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:25.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:25.904+0000 7f0eaeffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:57:25.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:25.904+0000 7f0eaeffd640 10 monclient(hunting): picked mon.noname-c con 0x7f0e9c0036d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:25.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:25.904+0000 7f0eaeffd640 10 monclient(hunting): picked mon.noname-b con 0x7f0e9c007360 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:25.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:25.904+0000 7f0eaeffd640 10 monclient(hunting): picked mon.noname-a con 0x7f0e9c00b090 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:25.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:25.904+0000 7f0eaeffd640 1 -- 172.21.15.26:0/1381815720 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb009f2d0 con 0x7f0e9c00b090 2024-09-16T14:57:25.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:25.904+0000 7f0eaeffd640 1 -- 172.21.15.26:0/1381815720 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb0132a80 con 0x7f0e9c0036d0 2024-09-16T14:57:25.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:25.904+0000 7f0eaeffd640 1 -- 172.21.15.26:0/1381815720 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e9c010dd0 con 0x7f0e9c007360 2024-09-16T14:57:25.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:25.904+0000 7f0eaeffd640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:25.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:25.904+0000 7f0eb5583640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.189:6789/0 conn(0x7f0e9c0036d0 legacy=0x7f0e9c005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:26.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:26.108+0000 7f0eb5583640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.189:6789/0 conn(0x7f0e9c0036d0 legacy=0x7f0e9c005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:26.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:26.512+0000 7f0eb5583640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.189:6789/0 conn(0x7f0e9c0036d0 legacy=0x7f0e9c005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:27.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:27.316+0000 7f0eb5583640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.189:6789/0 conn(0x7f0e9c0036d0 legacy=0x7f0e9c005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:28.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:28.904+0000 7f0eaeffd640 10 monclient(hunting): tick 2024-09-16T14:57:28.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:28.904+0000 7f0eaeffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:57:28.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:28.904+0000 7f0eaeffd640 1 monclient(hunting): continuing hunt 2024-09-16T14:57:28.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:28.904+0000 7f0eaeffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:57:28.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:28.904+0000 7f0eaeffd640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.26:6790/0 conn(0x7f0e9c007360 legacy=0x7f0e9c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:28.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:28.904+0000 7f0eaeffd640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.189:6789/0 conn(0x7f0e9c0036d0 legacy=0x7f0e9c005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:57:28.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:28.904+0000 7f0eaeffd640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.26:6789/0 conn(0x7f0e9c00b090 legacy=0x7f0e9c00d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:28.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:28.904+0000 7f0eaeffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:57:28.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:28.904+0000 7f0eaeffd640 10 monclient(hunting): picked mon.noname-c con 0x7f0e9c00ee90 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:28.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:28.904+0000 7f0eaeffd640 10 monclient(hunting): picked mon.noname-b con 0x7f0e9c011790 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:28.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:28.904+0000 7f0eaeffd640 10 monclient(hunting): picked mon.noname-a con 0x7f0e9c0102e0 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:28.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:28.904+0000 7f0eaeffd640 1 -- 172.21.15.26:0/1381815720 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e9c012020 con 0x7f0e9c0102e0 2024-09-16T14:57:28.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:28.904+0000 7f0eaeffd640 1 -- 172.21.15.26:0/1381815720 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb0132a80 con 0x7f0e9c00ee90 2024-09-16T14:57:28.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:28.904+0000 7f0eaeffd640 1 -- 172.21.15.26:0/1381815720 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e9c010760 con 0x7f0e9c011790 2024-09-16T14:57:28.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:28.904+0000 7f0eaeffd640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:28.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:28.908+0000 7f0eb5583640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.189:6789/0 conn(0x7f0e9c00ee90 legacy=0x7f0e9c010f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:29.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:29.112+0000 7f0eb5583640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.189:6789/0 conn(0x7f0e9c00ee90 legacy=0x7f0e9c010f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:29.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:29.516+0000 7f0eb5583640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.189:6789/0 conn(0x7f0e9c00ee90 legacy=0x7f0e9c010f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:30.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.320+0000 7f0eb5583640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.189:6789/0 conn(0x7f0e9c00ee90 legacy=0x7f0e9c010f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:30.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 1 -- 172.21.15.26:0/1381815720 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398753730 0 0) 0x7f0e9c010760 con 0x7f0e9c011790 2024-09-16T14:57:30.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:30.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:30.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:30.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 10 monclient(hunting): my global_id is 284315 2024-09-16T14:57:30.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 1 -- 172.21.15.26:0/1381815720 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e98003530 con 0x7f0e9c011790 2024-09-16T14:57:30.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 1 -- 172.21.15.26:0/1381815720 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1129994599 0 0) 0x7f0e98003530 con 0x7f0e9c011790 2024-09-16T14:57:30.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.189:6789/0 conn(0x7f0e9c00ee90 legacy=0x7f0e9c010f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:57:30.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.26:6789/0 conn(0x7f0e9c0102e0 legacy=0x7f0e9c016330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:30.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 10 monclient: _finish_hunting 0 2024-09-16T14:57:30.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 1 monclient: found mon.noname-b 2024-09-16T14:57:30.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:30.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:57:30.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 1 -- 172.21.15.26:0/1381815720 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9c0108f0 con 0x7f0e9c011790 2024-09-16T14:57:30.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 10 monclient: _finish_auth 0 2024-09-16T14:57:30.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 10 monclient: _check_auth_tickets 2024-09-16T14:57:30.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:30.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 1 -- 172.21.15.26:0/1381815720 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0ea40014e0 con 0x7f0e9c011790 2024-09-16T14:57:30.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eb77e7640 5 monclient: authenticate success, global_id 284315 2024-09-16T14:57:30.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eb77e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:30.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:30.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:57:30.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.448+0000 7f0eaffff640 10 monclient: dump: 2024-09-16T14:57:30.452 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:30.452 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:30.452 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.452 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.452 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:30.453 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:30.453 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:30.453 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:30.453 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:30.453 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:30.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.452+0000 7f0eb77e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:30.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.452+0000 7f0eaffff640 1 -- 172.21.15.26:0/1381815720 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ea4001730 con 0x7f0e9c011790 2024-09-16T14:57:30.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.452+0000 7f0eaffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:30.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.452+0000 7f0eaffff640 1 -- 172.21.15.26:0/1381815720 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0ea4001b00 con 0x7f0e9c011790 2024-09-16T14:57:30.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.452+0000 7f0eaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:30.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.452+0000 7f0eaffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:30.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.452+0000 7f0eaffff640 10 monclient: dump: 2024-09-16T14:57:30.454 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:30.454 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:30.454 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.454 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.454 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:30.454 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:30.454 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:30.454 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:30.454 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:30.454 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:30.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.452+0000 7f0eb77e7640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:30.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.452+0000 7f0eb77e7640 4 set_mon_vals no callback set 2024-09-16T14:57:30.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.452+0000 7f0eb77e7640 10 monclient: shutdown 2024-09-16T14:57:30.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.452+0000 7f0eb77e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:30.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.452+0000 7f0eb77e7640 1 -- 172.21.15.26:0/1381815720 >> v1:172.21.15.26:6790/0 conn(0x7f0e9c011790 legacy=0x7f0e9c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:30.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.452+0000 7f0eb77e7640 1 -- 172.21.15.26:0/1381815720 shutdown_connections 2024-09-16T14:57:30.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.452+0000 7f0eb77e7640 1 -- 172.21.15.26:0/1381815720 >> 172.21.15.26:0/1381815720 conn(0x7f0eb009fb50 msgr2=0x7f0eb009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:30.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.452+0000 7f0eb77e7640 1 -- 172.21.15.26:0/1381815720 shutdown_connections 2024-09-16T14:57:30.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.456+0000 7f0eb77e7640 1 -- 172.21.15.26:0/1381815720 wait complete. 2024-09-16T14:57:30.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.456+0000 7f0eb77e7640 10 monclient: build_initial_monmap 2024-09-16T14:57:30.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.456+0000 7f0eb77e7640 10 monclient: monmap: 2024-09-16T14:57:30.457 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:30.457 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:30.457 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:30.457 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:30.457 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:30.458 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:30.458 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:30.458 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:30.458 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:30.458 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:30.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.456+0000 7f0eb77e7640 1 Processor -- start 2024-09-16T14:57:30.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.456+0000 7f0eb77e7640 1 -- start start 2024-09-16T14:57:30.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.456+0000 7f0eb77e7640 10 monclient: init 2024-09-16T14:57:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.460+0000 7f0eb77e7640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.460+0000 7f0eb77e7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:57:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.460+0000 7f0eb77e7640 10 monclient(hunting): picked mon.noname-a con 0x7f0e9c00ee90 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.460+0000 7f0eb77e7640 10 monclient(hunting): picked mon.noname-c con 0x7f0e9c0102e0 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.460+0000 7f0eb77e7640 10 monclient(hunting): picked mon.noname-b con 0x7f0e9c011790 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.460+0000 7f0eb77e7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb0134350 con 0x7f0e9c00ee90 2024-09-16T14:57:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.460+0000 7f0eb77e7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb0138a40 con 0x7f0e9c011790 2024-09-16T14:57:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.460+0000 7f0eb77e7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb006f3e0 con 0x7f0e9c0102e0 2024-09-16T14:57:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.460+0000 7f0eb77e7640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:30.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.460+0000 7f0eb77e7640 10 monclient(hunting): authenticate will time out at 1874.581657s 2024-09-16T14:57:30.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.460+0000 7f0eb5d84640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f0e9c011790 legacy=0x7f0eb01cf5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:30.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.460+0000 7f0eb4d82640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0e9c0102e0 0x7f0eb006ecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48796/0 (socket says 172.21.15.26:48796) 2024-09-16T14:57:30.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.460+0000 7f0eb4d82640 1 -- 172.21.15.26:0/3846422144 learned_addr learned my addr 172.21.15.26:0/3846422144 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:30.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 1 -- 172.21.15.26:0/3846422144 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298938823 0 0) 0x7f0eb006f3e0 con 0x7f0e9c0102e0 2024-09-16T14:57:30.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:30.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:30.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:30.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 10 monclient(hunting): my global_id is 284318 2024-09-16T14:57:30.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 1 -- 172.21.15.26:0/3846422144 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e9c0106c0 con 0x7f0e9c0102e0 2024-09-16T14:57:30.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 1 -- 172.21.15.26:0/3846422144 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2895510205 0 0) 0x7f0e9c0106c0 con 0x7f0e9c0102e0 2024-09-16T14:57:30.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 1 -- 172.21.15.26:0/3846422144 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0eb006f3e0 con 0x7f0e9c0102e0 2024-09-16T14:57:30.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 1 -- 172.21.15.26:0/3846422144 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0ea4001b00 con 0x7f0e9c0102e0 2024-09-16T14:57:30.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:30.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:30.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 10 monclient(hunting): dump: 2024-09-16T14:57:30.465 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:30.465 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:30.465 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.465 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.465 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:30.465 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:30.465 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:30.465 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:30.465 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:30.466 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:30.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:30.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 1 -- 172.21.15.26:0/3846422144 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1303269888 0 0) 0x7f0eb006f3e0 con 0x7f0e9c0102e0 2024-09-16T14:57:30.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 1 -- 172.21.15.26:0/3846422144 >> v1:172.21.15.189:6789/0 conn(0x7f0e9c011790 legacy=0x7f0eb01cf5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:57:30.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 1 -- 172.21.15.26:0/3846422144 >> v1:172.21.15.26:6789/0 conn(0x7f0e9c00ee90 legacy=0x7f0eb006e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:30.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 10 monclient: _finish_hunting 0 2024-09-16T14:57:30.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 1 monclient: found mon.c 2024-09-16T14:57:30.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:30.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:30.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 1 -- 172.21.15.26:0/3846422144 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb0133700 con 0x7f0e9c0102e0 2024-09-16T14:57:30.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 10 monclient: _finish_auth 0 2024-09-16T14:57:30.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 10 monclient: _check_auth_tickets 2024-09-16T14:57:30.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:30.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eb77e7640 5 monclient: authenticate success, global_id 284318 2024-09-16T14:57:30.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eb77e7640 10 monclient: _renew_subs 2024-09-16T14:57:30.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eb77e7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:30.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eb77e7640 1 -- 172.21.15.26:0/3846422144 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb009a240 con 0x7f0e9c0102e0 2024-09-16T14:57:30.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eb77e7640 10 monclient: _renew_subs 2024-09-16T14:57:30.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eb77e7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:30.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eb77e7640 1 -- 172.21.15.26:0/3846422144 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb0133ab0 con 0x7f0e9c0102e0 2024-09-16T14:57:30.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 1 -- 172.21.15.26:0/3846422144 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ea40038b0 con 0x7f0e9c0102e0 2024-09-16T14:57:30.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:30.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 1 -- 172.21.15.26:0/3846422144 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0ea40011c0 con 0x7f0e9c0102e0 2024-09-16T14:57:30.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:30.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:30.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 10 monclient: dump: 2024-09-16T14:57:30.467 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:30.468 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:30.468 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.468 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.468 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:30.468 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:30.468 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:30.468 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:30.468 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:30.468 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:30.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eaf7fe640 4 set_mon_vals no callback set 2024-09-16T14:57:30.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eb77e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:30.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 1 -- 172.21.15.26:0/3846422144 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0ea4003bc0 con 0x7f0e9c0102e0 2024-09-16T14:57:30.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eadffb640 1 -- 172.21.15.26:0/3846422144 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f0ea4043d90 con 0x7f0e9c0102e0 2024-09-16T14:57:30.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eb77e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:30.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eb77e7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:30.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eb77e7640 1 -- 172.21.15.26:0/3846422144 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e90005050 con 0x7f0e9c0102e0 2024-09-16T14:57:30.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.464+0000 7f0eb5583640 10 monclient: get_auth_request con 0x7f0e9c0383c0 auth_method 0 2024-09-16T14:57:30.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.476+0000 7f0eadffb640 1 -- 172.21.15.26:0/3846422144 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0ea4014500 con 0x7f0e9c0102e0 2024-09-16T14:57:30.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.476+0000 7f0eadffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:30.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.476+0000 7f0eadffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:30.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.708+0000 7f0eb77e7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:57:30.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.708+0000 7f0eb77e7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:30.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.708+0000 7f0eb77e7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:30.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.708+0000 7f0eb77e7640 1 -- 172.21.15.26:0/3846422144 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0e90005880 con 0x7f0e9c0102e0 2024-09-16T14:57:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.712+0000 7f0eadffb640 1 -- 172.21.15.26:0/3846422144 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 3765367394) 0x7f0ea4012ba0 con 0x7f0e9c0102e0 2024-09-16T14:57:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.712+0000 7f0eadffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:30.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.712+0000 7f0eadffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:30.712 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":142,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:57:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.712+0000 7f0eb77e7640 1 -- 172.21.15.26:0/3846422144 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0e9c0383c0 legacy=0x7f0e9c03a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.712+0000 7f0eb77e7640 10 monclient: shutdown 2024-09-16T14:57:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.712+0000 7f0eb77e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.712+0000 7f0eb77e7640 1 -- 172.21.15.26:0/3846422144 >> v1:172.21.15.26:6790/0 conn(0x7f0e9c0102e0 legacy=0x7f0eb006ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.712+0000 7f0eb77e7640 1 -- 172.21.15.26:0/3846422144 shutdown_connections 2024-09-16T14:57:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.712+0000 7f0eb77e7640 1 -- 172.21.15.26:0/3846422144 >> 172.21.15.26:0/3846422144 conn(0x7f0eb009fb50 msgr2=0x7f0eb0131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.712+0000 7f0eb77e7640 1 -- 172.21.15.26:0/3846422144 shutdown_connections 2024-09-16T14:57:30.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.712+0000 7f0eb77e7640 1 -- 172.21.15.26:0/3846422144 wait complete. 2024-09-16T14:57:30.731 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:57:30.731 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T14:57:30.731 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:57:30.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 10 monclient: get_monmap_and_config 2024-09-16T14:57:30.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 10 monclient: build_initial_monmap 2024-09-16T14:57:30.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 10 monclient: monmap: 2024-09-16T14:57:30.856 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:30.857 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:30.857 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:30.858473+0000 2024-09-16T14:57:30.857 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:30.858473+0000 2024-09-16T14:57:30.857 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:30.857 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:30.857 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:30.857 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:30.857 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:30.857 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:30.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 1 Processor -- start 2024-09-16T14:57:30.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 1 -- start start 2024-09-16T14:57:30.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 10 monclient: init 2024-09-16T14:57:30.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:30.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:57:30.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 10 monclient(hunting): picked mon.noname-a con 0x7f0c8409a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 10 monclient(hunting): picked mon.noname-b con 0x7f0c8409d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 10 monclient(hunting): picked mon.noname-c con 0x7f0c8409b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c8409dc60 con 0x7f0c8409a550 2024-09-16T14:57:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c84099ae0 con 0x7f0c8409b040 2024-09-16T14:57:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c8409b8f0 con 0x7f0c8409d1b0 2024-09-16T14:57:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 10 monclient(hunting): authenticate will time out at 1874.978505s 2024-09-16T14:57:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c88b3d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0c8409d1b0 0x7f0c84136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48804/0 (socket says 172.21.15.26:48804) 2024-09-16T14:57:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c88b3d640 1 -- 172.21.15.26:0/1951007032 learned_addr learned my addr 172.21.15.26:0/1951007032 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8933e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0c8409a550 0x7f0c8409a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41962/0 (socket says 172.21.15.26:41962) 2024-09-16T14:57:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c89b3f640 1 -- 172.21.15.26:0/1951007032 >> v1:172.21.15.189:6789/0 conn(0x7f0c8409b040 legacy=0x7f0c8413a190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:30.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 1 -- 172.21.15.26:0/1951007032 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487904433 0 0) 0x7f0c8409dc60 con 0x7f0c8409a550 2024-09-16T14:57:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient(hunting): my global_id is 274401 2024-09-16T14:57:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 1 -- 172.21.15.26:0/1951007032 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c6c003530 con 0x7f0c8409a550 2024-09-16T14:57:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 1 -- 172.21.15.26:0/1951007032 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989168360 0 0) 0x7f0c8409b8f0 con 0x7f0c8409d1b0 2024-09-16T14:57:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient(hunting): my global_id is 284321 2024-09-16T14:57:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 1 -- 172.21.15.26:0/1951007032 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c8409dc60 con 0x7f0c8409d1b0 2024-09-16T14:57:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 1 -- 172.21.15.26:0/1951007032 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2174858251 0 0) 0x7f0c6c003530 con 0x7f0c8409a550 2024-09-16T14:57:30.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 1 -- 172.21.15.26:0/1951007032 >> v1:172.21.15.26:6790/0 conn(0x7f0c8409d1b0 legacy=0x7f0c84136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:30.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 1 -- 172.21.15.26:0/1951007032 >> v1:172.21.15.189:6789/0 conn(0x7f0c8409b040 legacy=0x7f0c8413a190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:57:30.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient: _finish_hunting 0 2024-09-16T14:57:30.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 1 monclient: found mon.noname-a 2024-09-16T14:57:30.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:30.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:57:30.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 1 -- 172.21.15.26:0/1951007032 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8409bbd0 con 0x7f0c8409a550 2024-09-16T14:57:30.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient: _finish_auth 0 2024-09-16T14:57:30.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient: _check_auth_tickets 2024-09-16T14:57:30.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:30.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 1 -- 172.21.15.26:0/1951007032 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0c7c0019a0 con 0x7f0c8409a550 2024-09-16T14:57:30.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 5 monclient: authenticate success, global_id 274401 2024-09-16T14:57:30.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:30.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient: dump: 2024-09-16T14:57:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:30.865 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:30.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 1 -- 172.21.15.26:0/1951007032 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c7c001c80 con 0x7f0c8409a550 2024-09-16T14:57:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 1 -- 172.21.15.26:0/1951007032 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0c7c002010 con 0x7f0c8409a550 2024-09-16T14:57:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 4 set_mon_vals no callback set 2024-09-16T14:57:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c83fff640 10 monclient: dump: 2024-09-16T14:57:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:30.866 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:30.867 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 10 monclient: shutdown 2024-09-16T14:57:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.856+0000 7f0c8b5a2640 1 -- 172.21.15.26:0/1951007032 >> v1:172.21.15.26:6789/0 conn(0x7f0c8409a550 legacy=0x7f0c8409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 1 -- 172.21.15.26:0/1951007032 shutdown_connections 2024-09-16T14:57:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 1 -- 172.21.15.26:0/1951007032 >> 172.21.15.26:0/1951007032 conn(0x7f0c8409fb50 msgr2=0x7f0c8409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:30.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 1 -- 172.21.15.26:0/1951007032 shutdown_connections 2024-09-16T14:57:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 1 -- 172.21.15.26:0/1951007032 wait complete. 2024-09-16T14:57:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 10 monclient: build_initial_monmap 2024-09-16T14:57:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 10 monclient: monmap: 2024-09-16T14:57:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:30.868 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 1 Processor -- start 2024-09-16T14:57:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 1 -- start start 2024-09-16T14:57:30.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 10 monclient: init 2024-09-16T14:57:30.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:30.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:57:30.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 10 monclient(hunting): picked mon.noname-c con 0x7f0c8409a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:30.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 10 monclient(hunting): picked mon.noname-b con 0x7f0c8409b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:30.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 10 monclient(hunting): picked mon.noname-a con 0x7f0c8409d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:30.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c841c7c40 con 0x7f0c8409d1b0 2024-09-16T14:57:30.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c8409a240 con 0x7f0c8409b040 2024-09-16T14:57:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c8413aba0 con 0x7f0c8409a550 2024-09-16T14:57:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 10 monclient(hunting): authenticate will time out at 1874.981942s 2024-09-16T14:57:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8933e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0c8409a550 0x7f0c84130ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48814/0 (socket says 172.21.15.26:48814) 2024-09-16T14:57:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8933e640 1 -- 172.21.15.26:0/3093795925 learned_addr learned my addr 172.21.15.26:0/3093795925 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c89b3f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0c8409d1b0 0x7f0c841cf740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41976/0 (socket says 172.21.15.26:41976) 2024-09-16T14:57:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c88b3d640 1 -- 172.21.15.26:0/3093795925 >> v1:172.21.15.189:6789/0 conn(0x7f0c8409b040 legacy=0x7f0c841cc010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699884558 0 0) 0x7f0c8413aba0 con 0x7f0c8409a550 2024-09-16T14:57:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient(hunting): my global_id is 284324 2024-09-16T14:57:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c70003530 con 0x7f0c8409a550 2024-09-16T14:57:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018368189 0 0) 0x7f0c74003a00 con 0x7f0c8409d1b0 2024-09-16T14:57:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient(hunting): my global_id is 274404 2024-09-16T14:57:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c8413aba0 con 0x7f0c8409d1b0 2024-09-16T14:57:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1269777505 0 0) 0x7f0c8413aba0 con 0x7f0c8409d1b0 2024-09-16T14:57:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c74003a00 con 0x7f0c8409d1b0 2024-09-16T14:57:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4115400969 0 0) 0x7f0c70003530 con 0x7f0c8409a550 2024-09-16T14:57:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c8413aba0 con 0x7f0c8409a550 2024-09-16T14:57:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0c74004030 con 0x7f0c8409d1b0 2024-09-16T14:57:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient(hunting): dump: 2024-09-16T14:57:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:30.877 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:30.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0c7c001960 con 0x7f0c8409a550 2024-09-16T14:57:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient(hunting): dump: 2024-09-16T14:57:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:30.878 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:30.879 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2158104986 0 0) 0x7f0c74003a00 con 0x7f0c8409d1b0 2024-09-16T14:57:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 >> v1:172.21.15.26:6790/0 conn(0x7f0c8409a550 legacy=0x7f0c84130ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 >> v1:172.21.15.189:6789/0 conn(0x7f0c8409b040 legacy=0x7f0c841cc010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:57:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:57:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 monclient: found mon.a 2024-09-16T14:57:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c84139d10 con 0x7f0c8409d1b0 2024-09-16T14:57:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient: _finish_auth 0 2024-09-16T14:57:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:57:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 5 monclient: authenticate success, global_id 274404 2024-09-16T14:57:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 10 monclient: _renew_subs 2024-09-16T14:57:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 1 -- 172.21.15.26:0/3093795925 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8413a8c0 con 0x7f0c8409d1b0 2024-09-16T14:57:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 10 monclient: _renew_subs 2024-09-16T14:57:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 1 -- 172.21.15.26:0/3093795925 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8409e540 con 0x7f0c8409d1b0 2024-09-16T14:57:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c74004290 con 0x7f0c8409d1b0 2024-09-16T14:57:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0c74004620 con 0x7f0c8409d1b0 2024-09-16T14:57:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 10 monclient: dump: 2024-09-16T14:57:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:30.881 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c837fe640 4 set_mon_vals no callback set 2024-09-16T14:57:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c8b5a2640 1 -- 172.21.15.26:0/3093795925 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0c64006590 con 0x7f0c64002830 2024-09-16T14:57:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.860+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0c74004800 con 0x7f0c8409d1b0 2024-09-16T14:57:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f0c74043c00 con 0x7f0c8409d1b0 2024-09-16T14:57:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191279365 0 0) 0x7f0c64006590 con 0x7f0c64002830 2024-09-16T14:57:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 10 monclient(hunting): my global_id is 274407 2024-09-16T14:57:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c74003a00 con 0x7f0c64002830 2024-09-16T14:57:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1597924041 0 0) 0x7f0c74003a00 con 0x7f0c64002830 2024-09-16T14:57:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c64006590 con 0x7f0c64002830 2024-09-16T14:57:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0c7c004fa0 con 0x7f0c64002830 2024-09-16T14:57:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 10 monclient: dump: 2024-09-16T14:57:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:30.883 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c88b3d640 10 monclient: get_auth_request con 0x7f0c70036250 auth_method 0 2024-09-16T14:57:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2784805933 0 0) 0x7f0c64006590 con 0x7f0c64002830 2024-09-16T14:57:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c6400a0e0 con 0x7f0c64002830 2024-09-16T14:57:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0c7c005100 con 0x7f0c64002830 2024-09-16T14:57:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 >> v1:172.21.15.26:6789/0 conn(0x7f0c64002830 legacy=0x7f0c64004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.864+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 >> v1:172.21.15.26:6789/0 conn(0x7f0c64002830 legacy=0x7f0c64004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.868+0000 7f0c8b5a2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:57:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.868+0000 7f0c8b5a2640 1 -- 172.21.15.26:0/3093795925 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0c64009da0 con 0x7f0c640090c0 2024-09-16T14:57:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.868+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683965648 0 0) 0x7f0c64009da0 con 0x7f0c640090c0 2024-09-16T14:57:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.868+0000 7f0c81ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.868+0000 7f0c81ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.868+0000 7f0c81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.868+0000 7f0c81ffb640 10 monclient(hunting): my global_id is 274410 2024-09-16T14:57:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.868+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c64006590 con 0x7f0c640090c0 2024-09-16T14:57:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.868+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2009971991 0 0) 0x7f0c64006590 con 0x7f0c640090c0 2024-09-16T14:57:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.868+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c64009da0 con 0x7f0c640090c0 2024-09-16T14:57:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.868+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0c7c005820 con 0x7f0c640090c0 2024-09-16T14:57:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.868+0000 7f0c81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.868+0000 7f0c81ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.868+0000 7f0c81ffb640 10 monclient: dump: 2024-09-16T14:57:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:30.886 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:30.886 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:30.886 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:30.886 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:30.886 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:30.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.868+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2141369651 0 0) 0x7f0c64009da0 con 0x7f0c640090c0 2024-09-16T14:57:30.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.868+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0c6400c290 con 0x7f0c640090c0 2024-09-16T14:57:30.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.872+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4201 (unknown 0 0 970942659) 0x7f0c7c007af0 con 0x7f0c640090c0 2024-09-16T14:57:30.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.872+0000 7f0c81ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:57:30.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.872+0000 7f0c81ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:30.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.872+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 >> v1:172.21.15.26:6789/0 conn(0x7f0c640090c0 legacy=0x7f0c64009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:30.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.872+0000 7f0c81ffb640 1 -- 172.21.15.26:0/3093795925 >> v1:172.21.15.26:6789/0 conn(0x7f0c640090c0 legacy=0x7f0c64009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:30.887 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:57:30.887 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:30.887 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:30.887 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:57:30.887 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 142, 2024-09-16T14:57:30.887 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:57:30.887 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:57:30.887 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:57:30.888 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:30.888 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:57:30.888 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:30.888 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:57:30.888 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:57:30.888 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:30.888 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:30.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:30.888 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:30.888 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:30.889 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:30.889 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:30.889 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:30.889 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:30.889 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:30.889 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:30.889 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:57:30.889 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:57:30.889 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:30.889 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:30.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:30.889 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:30.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:30.889 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:30.890 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:30.890 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:30.890 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:30.890 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:30.890 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:30.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:30.890 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:57:30.890 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:57:30.890 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:57:30.890 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:57:30.890 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:57:30.890 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:57:30.890 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:30.891 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:30.891 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:57:30.891 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:57:30.891 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:57:30.891 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:57:30.891 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:57:30.891 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:57:30.891 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:57:30.891 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:30.891 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:57:30.891 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:30.891 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:30.891 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:30.891 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:30.892 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:30.892 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:30.892 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:30.892 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:30.892 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:30.892 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:30.892 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:30.892 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:57:30.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:30.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:57:30.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:30.893 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:30.893 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:30.893 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:30.893 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:30.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:30.893 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:30.893 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:57:30.893 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:30.893 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:30.893 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:30.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:30.893 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:57:30.893 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:57:30.894 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:30.894 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:30.894 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:30.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:30.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:30.894 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:30.894 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:30.894 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:30.894 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:30.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:30.895 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:30.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:57:30.895 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:30.895 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:30.895 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:30.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:30.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:57:30.895 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:57:30.895 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:30.895 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:30.895 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:30.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:30.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:30.896 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:30.896 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:30.896 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:30.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:30.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:30.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:30.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:57:30.896 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:30.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:30.896 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:30.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:30.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:57:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:57:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:57:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:57:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:30.897 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:57:30.899 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:57:30.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.872+0000 7f0c8b5a2640 1 -- 172.21.15.26:0/3093795925 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0c70036250 legacy=0x7f0c70038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:30.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.872+0000 7f0c8b5a2640 10 monclient: shutdown 2024-09-16T14:57:30.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.872+0000 7f0c8b5a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:30.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.872+0000 7f0c8b5a2640 1 -- 172.21.15.26:0/3093795925 >> v1:172.21.15.26:6789/0 conn(0x7f0c8409d1b0 legacy=0x7f0c841cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:30.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.872+0000 7f0c89b3f640 1 -- 172.21.15.26:0/3093795925 reap_dead start 2024-09-16T14:57:30.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.872+0000 7f0c89b3f640 1 -- 172.21.15.26:0/3093795925 reap_dead start 2024-09-16T14:57:30.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.872+0000 7f0c8b5a2640 1 -- 172.21.15.26:0/3093795925 shutdown_connections 2024-09-16T14:57:30.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.872+0000 7f0c8b5a2640 1 -- 172.21.15.26:0/3093795925 >> 172.21.15.26:0/3093795925 conn(0x7f0c8409fb50 msgr2=0x7f0c841345c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:30.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.872+0000 7f0c8b5a2640 1 -- 172.21.15.26:0/3093795925 shutdown_connections 2024-09-16T14:57:30.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:30.872+0000 7f0c8b5a2640 1 -- 172.21.15.26:0/3093795925 wait complete. 2024-09-16T14:57:30.902 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:57:30.902 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:57:30.902 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:57:31.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 10 monclient: get_monmap_and_config 2024-09-16T14:57:31.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 10 monclient: build_initial_monmap 2024-09-16T14:57:31.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 10 monclient: monmap: 2024-09-16T14:57:31.020 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:31.020 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:31.020 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:31.021801+0000 2024-09-16T14:57:31.020 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:31.021801+0000 2024-09-16T14:57:31.020 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:31.020 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:31.020 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:31.020 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:31.020 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:31.020 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:31.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 1 Processor -- start 2024-09-16T14:57:31.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 1 -- start start 2024-09-16T14:57:31.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 10 monclient: init 2024-09-16T14:57:31.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:31.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:57:31.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 10 monclient(hunting): picked mon.noname-a con 0x7f022409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:31.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 10 monclient(hunting): picked mon.noname-b con 0x7f0224132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:31.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 10 monclient(hunting): picked mon.noname-c con 0x7f0224138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:31.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022409d260 con 0x7f022409cb80 2024-09-16T14:57:31.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0224132a80 con 0x7f0224138ff0 2024-09-16T14:57:31.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022409cf60 con 0x7f0224132650 2024-09-16T14:57:31.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:31.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 10 monclient(hunting): authenticate will time out at 1875.141882s 2024-09-16T14:57:31.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02291d4640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0224132650 0x7f02241378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48824/0 (socket says 172.21.15.26:48824) 2024-09-16T14:57:31.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02299d5640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f022409cb80 0x7f02240717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42008/0 (socket says 172.21.15.26:42008) 2024-09-16T14:57:31.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02291d4640 1 -- 172.21.15.26:0/3049369976 learned_addr learned my addr 172.21.15.26:0/3049369976 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022a1d6640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f0224138ff0 legacy=0x7f022413b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 1 -- 172.21.15.26:0/3049369976 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868604823 0 0) 0x7f022409d260 con 0x7f022409cb80 2024-09-16T14:57:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient(hunting): _negotiate 2024-09-16T14:57:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient(hunting): my global_id is 274413 2024-09-16T14:57:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 1 -- 172.21.15.26:0/3049369976 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f020c003530 con 0x7f022409cb80 2024-09-16T14:57:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 1 -- 172.21.15.26:0/3049369976 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499991618 0 0) 0x7f022409cf60 con 0x7f0224132650 2024-09-16T14:57:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient(hunting): _negotiate 2024-09-16T14:57:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient(hunting): my global_id is 284327 2024-09-16T14:57:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 1 -- 172.21.15.26:0/3049369976 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f022409d260 con 0x7f0224132650 2024-09-16T14:57:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 1 -- 172.21.15.26:0/3049369976 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1394373074 0 0) 0x7f020c003530 con 0x7f022409cb80 2024-09-16T14:57:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 1 -- 172.21.15.26:0/3049369976 >> v1:172.21.15.26:6790/0 conn(0x7f0224132650 legacy=0x7f02241378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 1 -- 172.21.15.26:0/3049369976 >> v1:172.21.15.189:6789/0 conn(0x7f0224138ff0 legacy=0x7f022413b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:57:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient: _finish_hunting 0 2024-09-16T14:57:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 1 monclient: found mon.noname-a 2024-09-16T14:57:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:57:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 1 -- 172.21.15.26:0/3049369976 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022409f2d0 con 0x7f022409cb80 2024-09-16T14:57:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient: _finish_auth 0 2024-09-16T14:57:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient: _check_auth_tickets 2024-09-16T14:57:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 1 -- 172.21.15.26:0/3049369976 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f02140019a0 con 0x7f022409cb80 2024-09-16T14:57:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 5 monclient: authenticate success, global_id 274413 2024-09-16T14:57:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient: dump: 2024-09-16T14:57:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:31.032 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 1 -- 172.21.15.26:0/3049369976 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0214001c10 con 0x7f022409cb80 2024-09-16T14:57:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 4 set_mon_vals no callback set 2024-09-16T14:57:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 1 -- 172.21.15.26:0/3049369976 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0214001fa0 con 0x7f022409cb80 2024-09-16T14:57:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f02289d3640 10 monclient: dump: 2024-09-16T14:57:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:31.034 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 10 monclient: shutdown 2024-09-16T14:57:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 1 -- 172.21.15.26:0/3049369976 >> v1:172.21.15.26:6789/0 conn(0x7f022409cb80 legacy=0x7f02240717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 1 -- 172.21.15.26:0/3049369976 shutdown_connections 2024-09-16T14:57:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 1 -- 172.21.15.26:0/3049369976 >> 172.21.15.26:0/3049369976 conn(0x7f022409fb50 msgr2=0x7f022409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 1 -- 172.21.15.26:0/3049369976 shutdown_connections 2024-09-16T14:57:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 1 -- 172.21.15.26:0/3049369976 wait complete. 2024-09-16T14:57:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 10 monclient: build_initial_monmap 2024-09-16T14:57:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.020+0000 7f022bc39640 10 monclient: monmap: 2024-09-16T14:57:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:31.035 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 1 Processor -- start 2024-09-16T14:57:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 1 -- start start 2024-09-16T14:57:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 10 monclient: init 2024-09-16T14:57:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:57:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 10 monclient(hunting): picked mon.noname-c con 0x7f022409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 10 monclient(hunting): picked mon.noname-a con 0x7f0224132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 10 monclient(hunting): picked mon.noname-b con 0x7f0224138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022409c290 con 0x7f0224132650 2024-09-16T14:57:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022406dfa0 con 0x7f0224138ff0 2024-09-16T14:57:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022409b370 con 0x7f022409cb80 2024-09-16T14:57:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 10 monclient(hunting): authenticate will time out at 1875.144855s 2024-09-16T14:57:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02299d5640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f022409cb80 0x7f022409a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48836/0 (socket says 172.21.15.26:48836) 2024-09-16T14:57:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02299d5640 1 -- 172.21.15.26:0/4070946517 learned_addr learned my addr 172.21.15.26:0/4070946517 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022a1d6640 1 -- 172.21.15.26:0/4070946517 >> v1:172.21.15.189:6789/0 conn(0x7f0224138ff0 legacy=0x7f02241d1b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T14:57:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524579306 0 0) 0x7f022409b370 con 0x7f022409cb80 2024-09-16T14:57:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): my global_id is 284330 2024-09-16T14:57:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0210003530 con 0x7f022409cb80 2024-09-16T14:57:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546420440 0 0) 0x7f022409c290 con 0x7f0224132650 2024-09-16T14:57:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): my global_id is 274416 2024-09-16T14:57:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f022409b370 con 0x7f0224132650 2024-09-16T14:57:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1546268215 0 0) 0x7f0210003530 con 0x7f022409cb80 2024-09-16T14:57:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f022409c290 con 0x7f022409cb80 2024-09-16T14:57:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 184749677 0 0) 0x7f022409b370 con 0x7f0224132650 2024-09-16T14:57:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0210003530 con 0x7f0224132650 2024-09-16T14:57:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0214001a20 con 0x7f022409cb80 2024-09-16T14:57:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): dump: 2024-09-16T14:57:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:31.039 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0218001880 con 0x7f0224132650 2024-09-16T14:57:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): dump: 2024-09-16T14:57:31.040 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:31.040 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:31.040 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:31.040 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:31.040 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:31.040 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:31.040 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:31.040 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:31.040 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:31.040 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:31.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4244673318 0 0) 0x7f022409c290 con 0x7f022409cb80 2024-09-16T14:57:31.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 >> v1:172.21.15.189:6789/0 conn(0x7f0224138ff0 legacy=0x7f02241d1b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:57:31.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 >> v1:172.21.15.26:6789/0 conn(0x7f0224132650 legacy=0x7f022409ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:31.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient: _finish_hunting 0 2024-09-16T14:57:31.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 monclient: found mon.c 2024-09-16T14:57:31.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:31.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:31.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022409a240 con 0x7f022409cb80 2024-09-16T14:57:31.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient: _finish_auth 0 2024-09-16T14:57:31.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient: _check_auth_tickets 2024-09-16T14:57:31.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:31.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 5 monclient: authenticate success, global_id 284330 2024-09-16T14:57:31.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 10 monclient: _renew_subs 2024-09-16T14:57:31.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:31.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 1 -- 172.21.15.26:0/4070946517 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f022413c400 con 0x7f022409cb80 2024-09-16T14:57:31.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 10 monclient: _renew_subs 2024-09-16T14:57:31.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:31.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 1 -- 172.21.15.26:0/4070946517 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02240a2460 con 0x7f022409cb80 2024-09-16T14:57:31.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0214004ee0 con 0x7f022409cb80 2024-09-16T14:57:31.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:31.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0214005270 con 0x7f022409cb80 2024-09-16T14:57:31.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:31.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:31.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient: dump: 2024-09-16T14:57:31.042 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:31.042 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:31.042 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:31.042 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:31.043 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:31.043 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:31.043 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:31.043 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:31.043 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:31.043 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:31.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f0223fff640 4 set_mon_vals no callback set 2024-09-16T14:57:31.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:31.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022bc39640 1 -- 172.21.15.26:0/4070946517 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0204006590 con 0x7f0204002830 2024-09-16T14:57:31.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0214005450 con 0x7f022409cb80 2024-09-16T14:57:31.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f0214044470 con 0x7f022409cb80 2024-09-16T14:57:31.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209755319 0 0) 0x7f0204006590 con 0x7f0204002830 2024-09-16T14:57:31.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:31.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:31.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:31.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient(hunting): my global_id is 284333 2024-09-16T14:57:31.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f022409c290 con 0x7f0204002830 2024-09-16T14:57:31.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 455014637 0 0) 0x7f022409c290 con 0x7f0204002830 2024-09-16T14:57:31.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0204006590 con 0x7f0204002830 2024-09-16T14:57:31.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f02180049e0 con 0x7f0204002830 2024-09-16T14:57:31.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:31.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:31.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 10 monclient: dump: 2024-09-16T14:57:31.044 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:31.044 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:31.044 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:31.045 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f022a1d6640 10 monclient: get_auth_request con 0x7f0210036210 auth_method 0 2024-09-16T14:57:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3111214821 0 0) 0x7f0204006590 con 0x7f0204002830 2024-09-16T14:57:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.024+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f020400a0e0 con 0x7f0204002830 2024-09-16T14:57:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.028+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0218004d20 con 0x7f0204002830 2024-09-16T14:57:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.028+0000 7f02227fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.028+0000 7f02227fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:31.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.028+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 >> v1:172.21.15.26:6790/0 conn(0x7f0204002830 legacy=0x7f0204004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:31.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.028+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 >> v1:172.21.15.26:6790/0 conn(0x7f0204002830 legacy=0x7f0204004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f022bc39640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:57:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f022bc39640 1 -- 172.21.15.26:0/4070946517 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0204009d80 con 0x7f02040090a0 2024-09-16T14:57:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762407738 0 0) 0x7f0204009d80 con 0x7f02040090a0 2024-09-16T14:57:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 10 monclient(hunting): my global_id is 284336 2024-09-16T14:57:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0204006590 con 0x7f02040090a0 2024-09-16T14:57:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3066801137 0 0) 0x7f0204006590 con 0x7f02040090a0 2024-09-16T14:57:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0204009d80 con 0x7f02040090a0 2024-09-16T14:57:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0218005380 con 0x7f02040090a0 2024-09-16T14:57:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 10 monclient: dump: 2024-09-16T14:57:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:31.049 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3814289366 0 0) 0x7f0204009d80 con 0x7f02040090a0 2024-09-16T14:57:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f020400c290 con 0x7f02040090a0 2024-09-16T14:57:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3871 (unknown 0 0 1084547250) 0x7f0218006730 con 0x7f02040090a0 2024-09-16T14:57:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:57:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 >> v1:172.21.15.26:6790/0 conn(0x7f02040090a0 legacy=0x7f0204009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f02227fc640 1 -- 172.21.15.26:0/4070946517 >> v1:172.21.15.26:6790/0 conn(0x7f02040090a0 legacy=0x7f0204009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:31.050 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:57:31.050 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:31.050 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:31.050 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:57:31.050 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 142, 2024-09-16T14:57:31.050 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:57:31.051 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:57:31.051 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:57:31.051 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:31.051 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:57:31.051 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:31.051 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:57:31.051 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:57:31.051 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:31.051 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:31.051 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:31.051 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:31.051 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:31.051 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:31.052 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:31.052 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:31.052 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:31.052 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:31.052 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:31.052 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:57:31.052 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:57:31.052 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:31.052 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:31.052 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:31.052 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:31.052 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:31.052 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:31.053 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:31.053 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:31.053 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:31.053 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:31.053 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:31.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:31.053 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:57:31.053 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:57:31.053 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:57:31.053 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:57:31.053 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:57:31.053 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:57:31.054 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:31.054 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:31.054 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:57:31.054 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:57:31.054 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:57:31.054 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:57:31.054 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:57:31.054 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:57:31.054 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:57:31.054 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:31.054 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:57:31.054 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:31.054 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:31.054 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:31.055 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:31.055 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:31.055 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:31.055 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:31.055 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:31.055 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:31.055 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:31.055 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:31.055 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:57:31.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:31.055 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:57:31.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:31.055 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:31.056 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:31.056 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:31.056 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:31.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:31.056 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:31.056 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:57:31.056 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:31.056 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:31.056 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:31.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:31.056 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:57:31.056 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:57:31.056 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:31.057 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:31.057 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:31.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:31.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:31.057 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:31.057 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:31.057 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:31.057 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:31.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:31.057 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:31.057 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:57:31.057 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:31.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:31.058 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:31.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:31.058 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:57:31.058 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:57:31.058 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:31.058 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:31.058 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:31.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:31.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:31.058 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:31.058 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:31.058 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:31.058 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:31.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:31.059 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:31.059 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:57:31.059 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:31.059 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:31.059 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:31.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:31.059 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:57:31.059 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:57:31.059 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:31.059 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:31.059 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:31.059 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:31.060 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:31.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:31.060 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:57:31.060 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:57:31.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:31.060 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:31.060 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:31.060 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:31.060 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:31.060 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:31.060 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:57:31.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:31.060 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:31.061 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:31.061 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T14:57:31.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:31.061 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:31.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:31.061 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:57:31.061 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:57:31.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f022bc39640 1 -- 172.21.15.26:0/4070946517 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0210036210 legacy=0x7f02100386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:31.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f022bc39640 10 monclient: shutdown 2024-09-16T14:57:31.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f022bc39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:31.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f022bc39640 1 -- 172.21.15.26:0/4070946517 >> v1:172.21.15.26:6790/0 conn(0x7f022409cb80 legacy=0x7f022409a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:31.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f022a1d6640 1 -- 172.21.15.26:0/4070946517 reap_dead start 2024-09-16T14:57:31.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f022a1d6640 1 -- 172.21.15.26:0/4070946517 reap_dead start 2024-09-16T14:57:31.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f022bc39640 1 -- 172.21.15.26:0/4070946517 shutdown_connections 2024-09-16T14:57:31.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f022bc39640 1 -- 172.21.15.26:0/4070946517 >> 172.21.15.26:0/4070946517 conn(0x7f022409fb50 msgr2=0x7f02241304d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:31.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f022bc39640 1 -- 172.21.15.26:0/4070946517 shutdown_connections 2024-09-16T14:57:31.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:31.032+0000 7f022bc39640 1 -- 172.21.15.26:0/4070946517 wait complete. 2024-09-16T14:57:31.063 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:57:31.063 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:57:51.065 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T14:57:51.065 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T14:57:51.065 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T14:57:51.065 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T14:57:51.068 INFO:tasks.ceph.mon.b:Started 2024-09-16T14:57:51.069 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:57:51.069 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:57:51.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c4742640 10 monclient: get_monmap_and_config 2024-09-16T14:57:51.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c4742640 10 monclient: build_initial_monmap 2024-09-16T14:57:51.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c4742640 10 monclient: monmap: 2024-09-16T14:57:51.205 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:51.205 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.205 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:51.206754+0000 2024-09-16T14:57:51.205 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:51.206754+0000 2024-09-16T14:57:51.205 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:51.205 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.205 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:51.205 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:51.205 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:51.205 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c4742640 1 Processor -- start 2024-09-16T14:57:51.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c4742640 1 -- start start 2024-09-16T14:57:51.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c4742640 10 monclient: init 2024-09-16T14:57:51.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c4742640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:51.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c4742640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:57:51.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c4742640 10 monclient(hunting): picked mon.noname-c con 0x7f11bc09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:51.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c4742640 10 monclient(hunting): picked mon.noname-b con 0x7f11bc132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:51.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c4742640 10 monclient(hunting): picked mon.noname-a con 0x7f11bc138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:51.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c4742640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc09d260 con 0x7f11bc138ff0 2024-09-16T14:57:51.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c4742640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc132a80 con 0x7f11bc09cb80 2024-09-16T14:57:51.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c4742640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc09cf60 con 0x7f11bc132650 2024-09-16T14:57:51.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c4742640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:51.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c4742640 10 monclient(hunting): authenticate will time out at 1895.326881s 2024-09-16T14:57:51.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c1cdd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f11bc132650 0x7f11bc1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53492/0 (socket says 172.21.15.26:53492) 2024-09-16T14:57:51.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c1cdd640 1 -- 172.21.15.26:0/1230405733 learned_addr learned my addr 172.21.15.26:0/1230405733 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:51.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.204+0000 7f11c2cdf640 1 --1- 172.21.15.26:0/1230405733 >> v1:172.21.15.26:6789/0 conn(0x7f11bc138ff0 0x7f11bc13b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36866/0 (socket says 172.21.15.26:36866) 2024-09-16T14:57:51.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 1 -- 172.21.15.26:0/1230405733 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141340270 0 0) 0x7f11bc09d260 con 0x7f11bc138ff0 2024-09-16T14:57:51.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 10 monclient(hunting): my global_id is 274419 2024-09-16T14:57:51.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 1 -- 172.21.15.26:0/1230405733 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11a4003530 con 0x7f11bc138ff0 2024-09-16T14:57:51.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 1 -- 172.21.15.26:0/1230405733 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3887998949 0 0) 0x7f11a4003530 con 0x7f11bc138ff0 2024-09-16T14:57:51.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 1 -- 172.21.15.26:0/1230405733 >> v1:172.21.15.26:6790/0 conn(0x7f11bc132650 legacy=0x7f11bc1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 1 -- 172.21.15.26:0/1230405733 >> v1:172.21.15.189:6789/0 conn(0x7f11bc09cb80 legacy=0x7f11bc0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 10 monclient: _finish_hunting 0 2024-09-16T14:57:51.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 1 monclient: found mon.noname-a 2024-09-16T14:57:51.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:51.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:57:51.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 1 -- 172.21.15.26:0/1230405733 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc09f2d0 con 0x7f11bc138ff0 2024-09-16T14:57:51.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 10 monclient: _finish_auth 0 2024-09-16T14:57:51.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 10 monclient: _check_auth_tickets 2024-09-16T14:57:51.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:51.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 1 -- 172.21.15.26:0/1230405733 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f11b4001550 con 0x7f11bc138ff0 2024-09-16T14:57:51.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:51.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c14dc640 10 monclient: dump: 2024-09-16T14:57:51.213 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.213 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.213 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.213 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.214 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.214 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.214 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.214 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.214 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.214 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c4742640 5 monclient: authenticate success, global_id 274419 2024-09-16T14:57:51.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.208+0000 7f11c4742640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:51.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c14dc640 1 -- 172.21.15.26:0/1230405733 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11b4001800 con 0x7f11bc138ff0 2024-09-16T14:57:51.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c14dc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:51.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:51.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 4 set_mon_vals no callback set 2024-09-16T14:57:51.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c14dc640 1 -- 172.21.15.26:0/1230405733 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f11b4001b90 con 0x7f11bc138ff0 2024-09-16T14:57:51.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c14dc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c14dc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:51.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c14dc640 10 monclient: dump: 2024-09-16T14:57:51.215 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.215 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.215 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.215 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.215 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.215 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.215 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.215 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.215 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.215 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 10 monclient: shutdown 2024-09-16T14:57:51.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 1 -- 172.21.15.26:0/1230405733 >> v1:172.21.15.26:6789/0 conn(0x7f11bc138ff0 legacy=0x7f11bc13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 1 -- 172.21.15.26:0/1230405733 shutdown_connections 2024-09-16T14:57:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 1 -- 172.21.15.26:0/1230405733 >> 172.21.15.26:0/1230405733 conn(0x7f11bc09fb50 msgr2=0x7f11bc09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 1 -- 172.21.15.26:0/1230405733 shutdown_connections 2024-09-16T14:57:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 1 -- 172.21.15.26:0/1230405733 wait complete. 2024-09-16T14:57:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 10 monclient: build_initial_monmap 2024-09-16T14:57:51.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 10 monclient: monmap: 2024-09-16T14:57:51.220 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:51.220 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:51.220 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:51.220 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:51.220 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:51.220 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:51.220 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:51.220 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:51.220 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:51.220 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 1 Processor -- start 2024-09-16T14:57:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 1 -- start start 2024-09-16T14:57:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 10 monclient: init 2024-09-16T14:57:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:57:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 10 monclient(hunting): picked mon.noname-b con 0x7f11bc09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 10 monclient(hunting): picked mon.noname-a con 0x7f11bc132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 10 monclient(hunting): picked mon.noname-c con 0x7f11bc138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc09a240 con 0x7f11bc132650 2024-09-16T14:57:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc131ff0 con 0x7f11bc09cb80 2024-09-16T14:57:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc071200 con 0x7f11bc138ff0 2024-09-16T14:57:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:51.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 10 monclient(hunting): authenticate will time out at 1895.334531s 2024-09-16T14:57:51.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c1cdd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f11bc132650 0x7f11bc134d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36874/0 (socket says 172.21.15.26:36874) 2024-09-16T14:57:51.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c1cdd640 1 -- 172.21.15.26:0/1800587435 learned_addr learned my addr 172.21.15.26:0/1800587435 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:51.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 1 -- 172.21.15.26:0/1800587435 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396444715 0 0) 0x7f11bc071200 con 0x7f11bc138ff0 2024-09-16T14:57:51.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 10 monclient(hunting): my global_id is 284339 2024-09-16T14:57:51.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 1 -- 172.21.15.26:0/1800587435 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11a8003530 con 0x7f11bc138ff0 2024-09-16T14:57:51.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 1 -- 172.21.15.26:0/1800587435 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3559610635 0 0) 0x7f11a8003530 con 0x7f11bc138ff0 2024-09-16T14:57:51.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 1 -- 172.21.15.26:0/1800587435 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11bc071200 con 0x7f11bc138ff0 2024-09-16T14:57:51.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 1 -- 172.21.15.26:0/1800587435 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f11b4001510 con 0x7f11bc138ff0 2024-09-16T14:57:51.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:51.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 10 monclient(hunting): dump: 2024-09-16T14:57:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.223 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 1 -- 172.21.15.26:0/1800587435 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2332503068 0 0) 0x7f11bc071200 con 0x7f11bc138ff0 2024-09-16T14:57:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 1 -- 172.21.15.26:0/1800587435 >> v1:172.21.15.189:6789/0 conn(0x7f11bc09cb80 legacy=0x7f11bc134610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 1 -- 172.21.15.26:0/1800587435 >> v1:172.21.15.26:6789/0 conn(0x7f11bc132650 legacy=0x7f11bc134d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 10 monclient: _finish_hunting 0 2024-09-16T14:57:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 1 monclient: found mon.c 2024-09-16T14:57:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 1 -- 172.21.15.26:0/1800587435 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc1369a0 con 0x7f11bc138ff0 2024-09-16T14:57:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 10 monclient: _finish_auth 0 2024-09-16T14:57:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 10 monclient: _check_auth_tickets 2024-09-16T14:57:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 5 monclient: authenticate success, global_id 284339 2024-09-16T14:57:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 10 monclient: _renew_subs 2024-09-16T14:57:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 1 -- 172.21.15.26:0/1800587435 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11bc1374d0 con 0x7f11bc138ff0 2024-09-16T14:57:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 10 monclient: _renew_subs 2024-09-16T14:57:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c4742640 1 -- 172.21.15.26:0/1800587435 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11bc135600 con 0x7f11bc138ff0 2024-09-16T14:57:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 1 -- 172.21.15.26:0/1800587435 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11b4001170 con 0x7f11bc138ff0 2024-09-16T14:57:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 1 -- 172.21.15.26:0/1800587435 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f11b4004c40 con 0x7f11bc138ff0 2024-09-16T14:57:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11c0cdb640 4 set_mon_vals no callback set 2024-09-16T14:57:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.212+0000 7f11baffd640 10 monclient: dump: 2024-09-16T14:57:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.226 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.216+0000 7f11c4742640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.216+0000 7f11c4742640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.216+0000 7f11c4742640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.216+0000 7f11c4742640 1 -- 172.21.15.26:0/1800587435 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f119c005050 con 0x7f11bc138ff0 2024-09-16T14:57:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.216+0000 7f11baffd640 1 -- 172.21.15.26:0/1800587435 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f11b4004da0 con 0x7f11bc138ff0 2024-09-16T14:57:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.216+0000 7f11baffd640 1 -- 172.21.15.26:0/1800587435 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f11b4043de0 con 0x7f11bc138ff0 2024-09-16T14:57:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.216+0000 7f11c24de640 10 monclient: get_auth_request con 0x7f11a8036150 auth_method 0 2024-09-16T14:57:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.220+0000 7f11baffd640 1 -- 172.21.15.26:0/1800587435 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f11b4014660 con 0x7f11bc138ff0 2024-09-16T14:57:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.220+0000 7f11baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.220+0000 7f11baffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:51.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.440+0000 7f11c4742640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:57:51.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.440+0000 7f11c4742640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:51.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.440+0000 7f11c4742640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:51.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.440+0000 7f11c4742640 1 -- 172.21.15.26:0/1800587435 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f119c005880 con 0x7f11bc138ff0 2024-09-16T14:57:51.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.440+0000 7f11baffd640 1 -- 172.21.15.26:0/1800587435 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 204128848) 0x7f11b4012d00 con 0x7f11bc138ff0 2024-09-16T14:57:51.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.440+0000 7f11baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:51.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.440+0000 7f11baffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:51.442 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:57:51.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.444+0000 7f11c4742640 1 -- 172.21.15.26:0/1800587435 >> v1:172.21.15.189:6812/3196161582 conn(0x7f11a8036150 legacy=0x7f11a8038610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.444+0000 7f11c4742640 10 monclient: shutdown 2024-09-16T14:57:51.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.444+0000 7f11c4742640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:51.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.444+0000 7f11c4742640 1 -- 172.21.15.26:0/1800587435 >> v1:172.21.15.26:6790/0 conn(0x7f11bc138ff0 legacy=0x7f11bc1cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.444+0000 7f11c4742640 1 -- 172.21.15.26:0/1800587435 shutdown_connections 2024-09-16T14:57:51.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.444+0000 7f11c4742640 1 -- 172.21.15.26:0/1800587435 >> 172.21.15.26:0/1800587435 conn(0x7f11bc09fb50 msgr2=0x7f11bc130820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:51.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.444+0000 7f11c4742640 1 -- 172.21.15.26:0/1800587435 shutdown_connections 2024-09-16T14:57:51.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.444+0000 7f11c4742640 1 -- 172.21.15.26:0/1800587435 wait complete. 2024-09-16T14:57:51.458 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:57:51.458 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:57:51.458 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:57:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.572+0000 7fd1d6a53640 10 monclient: get_monmap_and_config 2024-09-16T14:57:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.572+0000 7fd1d6a53640 10 monclient: build_initial_monmap 2024-09-16T14:57:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.572+0000 7fd1d6a53640 10 monclient: monmap: 2024-09-16T14:57:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:51.575711+0000 2024-09-16T14:57:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:51.575711+0000 2024-09-16T14:57:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:51.574 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.572+0000 7fd1d6a53640 1 Processor -- start 2024-09-16T14:57:51.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.572+0000 7fd1d6a53640 1 -- start start 2024-09-16T14:57:51.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.572+0000 7fd1d6a53640 10 monclient: init 2024-09-16T14:57:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.572+0000 7fd1d6a53640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.572+0000 7fd1d6a53640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:57:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.572+0000 7fd1d6a53640 10 monclient(hunting): picked mon.noname-a con 0x7fd1d00a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.572+0000 7fd1d6a53640 10 monclient(hunting): picked mon.noname-b con 0x7fd1d00a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.572+0000 7fd1d6a53640 10 monclient(hunting): picked mon.noname-c con 0x7fd1d00a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:51.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.572+0000 7fd1d6a53640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d00a3250 con 0x7fd1d00a1d10 2024-09-16T14:57:51.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.572+0000 7fd1d6a53640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d009f600 con 0x7fd1d00a2800 2024-09-16T14:57:51.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.572+0000 7fd1d6a53640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d00a3830 con 0x7fd1d00a0000 2024-09-16T14:57:51.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.572+0000 7fd1d6a53640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:51.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.572+0000 7fd1d6a53640 10 monclient(hunting): authenticate will time out at 1895.695971s 2024-09-16T14:57:51.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1cf7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd1d00a0000 0x7fd1d0139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53514/0 (socket says 172.21.15.26:53514) 2024-09-16T14:57:51.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1cf7fe640 1 -- 172.21.15.26:0/1480978704 learned_addr learned my addr 172.21.15.26:0/1480978704 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:51.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 1 -- 172.21.15.26:0/1480978704 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080909926 0 0) 0x7fd1d00a3830 con 0x7fd1d00a0000 2024-09-16T14:57:51.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient(hunting): my global_id is 284342 2024-09-16T14:57:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 1 -- 172.21.15.26:0/1480978704 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1b8003530 con 0x7fd1d00a0000 2024-09-16T14:57:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 1 -- 172.21.15.26:0/1480978704 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055135330 0 0) 0x7fd1d00a3250 con 0x7fd1d00a1d10 2024-09-16T14:57:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient(hunting): my global_id is 274422 2024-09-16T14:57:51.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 1 -- 172.21.15.26:0/1480978704 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1d00a3830 con 0x7fd1d00a1d10 2024-09-16T14:57:51.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 1 -- 172.21.15.26:0/1480978704 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3168640140 0 0) 0x7fd1b8003530 con 0x7fd1d00a0000 2024-09-16T14:57:51.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 1 -- 172.21.15.26:0/1480978704 >> v1:172.21.15.189:6789/0 conn(0x7fd1d00a2800 legacy=0x7fd1d013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 1 -- 172.21.15.26:0/1480978704 >> v1:172.21.15.26:6789/0 conn(0x7fd1d00a1d10 legacy=0x7fd1d00a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient: _finish_hunting 0 2024-09-16T14:57:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 1 monclient: found mon.noname-b 2024-09-16T14:57:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:57:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 1 -- 172.21.15.26:0/1480978704 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d009ed60 con 0x7fd1d00a0000 2024-09-16T14:57:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient: _finish_auth 0 2024-09-16T14:57:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient: _check_auth_tickets 2024-09-16T14:57:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 1 -- 172.21.15.26:0/1480978704 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd1c4001620 con 0x7fd1d00a0000 2024-09-16T14:57:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:57:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient: dump: 2024-09-16T14:57:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.585 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.585 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.585 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.585 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.585 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.586 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 5 monclient: authenticate success, global_id 284342 2024-09-16T14:57:51.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:51.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 1 -- 172.21.15.26:0/1480978704 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1c4001840 con 0x7fd1d00a0000 2024-09-16T14:57:51.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:51.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 1 -- 172.21.15.26:0/1480978704 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd1c4001c10 con 0x7fd1d00a0000 2024-09-16T14:57:51.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:51.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 4 set_mon_vals no callback set 2024-09-16T14:57:51.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:51.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1ceffd640 10 monclient: dump: 2024-09-16T14:57:51.601 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.601 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.601 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.601 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.601 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.601 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.602 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.602 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.602 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.602 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 10 monclient: shutdown 2024-09-16T14:57:51.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:51.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 1 -- 172.21.15.26:0/1480978704 >> v1:172.21.15.26:6790/0 conn(0x7fd1d00a0000 legacy=0x7fd1d0139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 1 -- 172.21.15.26:0/1480978704 shutdown_connections 2024-09-16T14:57:51.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 1 -- 172.21.15.26:0/1480978704 >> 172.21.15.26:0/1480978704 conn(0x7fd1d00a7f30 msgr2=0x7fd1d00a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:51.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 1 -- 172.21.15.26:0/1480978704 shutdown_connections 2024-09-16T14:57:51.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 1 -- 172.21.15.26:0/1480978704 wait complete. 2024-09-16T14:57:51.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 10 monclient: build_initial_monmap 2024-09-16T14:57:51.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 10 monclient: monmap: 2024-09-16T14:57:51.603 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:51.603 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:51.603 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:51.603 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:51.603 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:51.603 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:51.603 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:51.603 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:51.604 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:51.604 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 1 Processor -- start 2024-09-16T14:57:51.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 1 -- start start 2024-09-16T14:57:51.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 10 monclient: init 2024-09-16T14:57:51.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:51.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:57:51.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 10 monclient(hunting): picked mon.noname-c con 0x7fd1d00a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:51.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 10 monclient(hunting): picked mon.noname-a con 0x7fd1d00a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:51.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 10 monclient(hunting): picked mon.noname-b con 0x7fd1d00a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:51.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.576+0000 7fd1d6a53640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d0132d40 con 0x7fd1d00a1d10 2024-09-16T14:57:51.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1d6a53640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d009e500 con 0x7fd1d00a2800 2024-09-16T14:57:51.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1d6a53640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d00a6920 con 0x7fd1d00a0000 2024-09-16T14:57:51.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1d6a53640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:51.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1d6a53640 10 monclient(hunting): authenticate will time out at 1895.700037s 2024-09-16T14:57:51.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1cf7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd1d00a1d10 0x7fd1d0132630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36898/0 (socket says 172.21.15.26:36898) 2024-09-16T14:57:51.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1cf7fe640 1 -- 172.21.15.26:0/523662871 learned_addr learned my addr 172.21.15.26:0/523662871 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:51.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473971282 0 0) 0x7fd1d00a6920 con 0x7fd1d00a0000 2024-09-16T14:57:51.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): my global_id is 284345 2024-09-16T14:57:51.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1bc003530 con 0x7fd1d00a0000 2024-09-16T14:57:51.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566964853 0 0) 0x7fd1d0132d40 con 0x7fd1d00a1d10 2024-09-16T14:57:51.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): my global_id is 274425 2024-09-16T14:57:51.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1d00a6920 con 0x7fd1d00a1d10 2024-09-16T14:57:51.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1465230946 0 0) 0x7fd1bc003530 con 0x7fd1d00a0000 2024-09-16T14:57:51.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1d0132d40 con 0x7fd1d00a0000 2024-09-16T14:57:51.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd1c0001a00 con 0x7fd1d00a0000 2024-09-16T14:57:51.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:51.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): dump: 2024-09-16T14:57:51.606 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.606 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.606 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.607 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.607 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.607 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.607 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.607 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.607 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.607 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:51.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2494251720 0 0) 0x7fd1d00a6920 con 0x7fd1d00a1d10 2024-09-16T14:57:51.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1bc003530 con 0x7fd1d00a1d10 2024-09-16T14:57:51.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd1c4001b50 con 0x7fd1d00a1d10 2024-09-16T14:57:51.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:51.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): dump: 2024-09-16T14:57:51.608 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.608 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.608 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.608 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.608 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.608 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.608 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.608 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.608 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.608 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1853035840 0 0) 0x7fd1d0132d40 con 0x7fd1d00a0000 2024-09-16T14:57:51.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 >> v1:172.21.15.189:6789/0 conn(0x7fd1d00a2800 legacy=0x7fd1d01d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 >> v1:172.21.15.26:6789/0 conn(0x7fd1d00a1d10 legacy=0x7fd1d0132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient: _finish_hunting 0 2024-09-16T14:57:51.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 monclient: found mon.c 2024-09-16T14:57:51.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:51.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:51.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d00a63f0 con 0x7fd1d00a0000 2024-09-16T14:57:51.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient: _finish_auth 0 2024-09-16T14:57:51.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient: _check_auth_tickets 2024-09-16T14:57:51.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:51.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1d6a53640 5 monclient: authenticate success, global_id 284345 2024-09-16T14:57:51.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1d6a53640 10 monclient: _renew_subs 2024-09-16T14:57:51.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1d6a53640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:51.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1d6a53640 1 -- 172.21.15.26:0/523662871 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d0136070 con 0x7fd1d00a0000 2024-09-16T14:57:51.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1d6a53640 10 monclient: _renew_subs 2024-09-16T14:57:51.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1d6a53640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:51.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1d6a53640 1 -- 172.21.15.26:0/523662871 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d0135d90 con 0x7fd1d00a0000 2024-09-16T14:57:51.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1c00015f0 con 0x7fd1d00a0000 2024-09-16T14:57:51.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:51.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd1c00038b0 con 0x7fd1d00a0000 2024-09-16T14:57:51.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:51.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ce7fc640 4 set_mon_vals no callback set 2024-09-16T14:57:51.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient: dump: 2024-09-16T14:57:51.610 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.610 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.610 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.610 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.611 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.611 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.611 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.611 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.611 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.611 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1d6a53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:51.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd1c000e320 con 0x7fd1d00a0000 2024-09-16T14:57:51.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1d6a53640 1 -- 172.21.15.26:0/523662871 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd1b0006590 con 0x7fd1b0002830 2024-09-16T14:57:51.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fd1c0042b60 con 0x7fd1d00a0000 2024-09-16T14:57:51.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905461720 0 0) 0x7fd1b0006590 con 0x7fd1b0002830 2024-09-16T14:57:51.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient(hunting): my global_id is 274428 2024-09-16T14:57:51.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1d0132d40 con 0x7fd1b0002830 2024-09-16T14:57:51.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3012365812 0 0) 0x7fd1d0132d40 con 0x7fd1b0002830 2024-09-16T14:57:51.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1b0006590 con 0x7fd1b0002830 2024-09-16T14:57:51.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd1c4004b30 con 0x7fd1b0002830 2024-09-16T14:57:51.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1d4ff0640 10 monclient: get_auth_request con 0x7fd1bc036250 auth_method 0 2024-09-16T14:57:51.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:51.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient: dump: 2024-09-16T14:57:51.612 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.612 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.612 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.613 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.613 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.613 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.613 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.613 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.613 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.613 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1323844399 0 0) 0x7fd1b0006590 con 0x7fd1b0002830 2024-09-16T14:57:51.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1b000a0e0 con 0x7fd1b0002830 2024-09-16T14:57:51.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fd1c4004e90 con 0x7fd1b0002830 2024-09-16T14:57:51.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:51.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:51.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 >> v1:172.21.15.26:6789/0 conn(0x7fd1b0002830 legacy=0x7fd1b0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.580+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 >> v1:172.21.15.26:6789/0 conn(0x7fd1b0002830 legacy=0x7fd1b0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:51.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1d6a53640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:57:51.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1d6a53640 1 -- 172.21.15.26:0/523662871 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd1b0009d80 con 0x7fd1b00090a0 2024-09-16T14:57:51.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823209079 0 0) 0x7fd1b0009d80 con 0x7fd1b00090a0 2024-09-16T14:57:51.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 10 monclient(hunting): my global_id is 274431 2024-09-16T14:57:51.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1b0006590 con 0x7fd1b00090a0 2024-09-16T14:57:51.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2990493654 0 0) 0x7fd1b0006590 con 0x7fd1b00090a0 2024-09-16T14:57:51.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1b0009d80 con 0x7fd1b00090a0 2024-09-16T14:57:51.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd1c40054f0 con 0x7fd1b00090a0 2024-09-16T14:57:51.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:51.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 10 monclient: dump: 2024-09-16T14:57:51.615 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.615 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.615 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.615 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.615 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.615 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.615 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.615 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.615 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.615 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3592319305 0 0) 0x7fd1b0009d80 con 0x7fd1b00090a0 2024-09-16T14:57:51.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd1b000c290 con 0x7fd1b00090a0 2024-09-16T14:57:51.616 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:57:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:57:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 144, 2024-09-16T14:57:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:57:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:57:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:57:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:57:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:57:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:57:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:57:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:51.618 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:51.618 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:51.618 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:51.618 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:57:51.618 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:57:51.618 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:51.618 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:51.618 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:51.618 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:51.618 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:51.618 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:51.618 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:51.618 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:51.618 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:51.619 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:51.619 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.619 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:57:51.619 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:57:51.619 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:57:51.619 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:57:51.619 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:57:51.619 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:57:51.619 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:51.619 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:51.619 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:57:51.619 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:57:51.620 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:57:51.620 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:57:51.620 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:57:51.620 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:57:51.620 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:57:51.620 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:51.620 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:57:51.620 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:51.620 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:51.620 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:51.620 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:51.620 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:51.620 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:51.620 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:51.621 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:51.621 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:51.621 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:51.621 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:51.621 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:57:51.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.621 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:57:51.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.621 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:51.621 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:51.621 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:51.621 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:51.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.621 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:51.622 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:57:51.622 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:51.622 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.622 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.622 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:57:51.622 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:57:51.622 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:51.622 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:51.622 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:51.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.622 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:51.623 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:51.623 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:51.623 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:51.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.623 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:51.623 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:57:51.623 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:51.623 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.623 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.623 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:57:51.623 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:57:51.623 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:51.624 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:51.624 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:51.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.624 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:51.624 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:51.624 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:51.624 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:51.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.624 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:51.624 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:57:51.624 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:51.624 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.625 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:57:51.625 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:57:51.625 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:51.625 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:51.625 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:51.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.625 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:57:51.625 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:57:51.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.626 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:51.626 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:51.626 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:51.626 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.626 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:51.626 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:57:51.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.626 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:51.626 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:51.626 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:57:51.626 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.626 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:51.626 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:57:51.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.627 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:51.627 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:51.627 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:57:51.627 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.627 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:51.627 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:57:51.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.627 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:51.627 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:51.627 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:51.627 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.627 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.627 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:57:51.628 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:57:51.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 2061481539) 0x7fd1c4006870 con 0x7fd1b00090a0 2024-09-16T14:57:51.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:57:51.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:51.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 >> v1:172.21.15.26:6789/0 conn(0x7fd1b00090a0 legacy=0x7fd1b0009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1ccff9640 1 -- 172.21.15.26:0/523662871 >> v1:172.21.15.26:6789/0 conn(0x7fd1b00090a0 legacy=0x7fd1b0009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:51.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1d6a53640 1 -- 172.21.15.26:0/523662871 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd1bc036250 legacy=0x7fd1bc038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1d6a53640 10 monclient: shutdown 2024-09-16T14:57:51.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1d6a53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:51.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1d6a53640 1 -- 172.21.15.26:0/523662871 >> v1:172.21.15.26:6790/0 conn(0x7fd1d00a0000 legacy=0x7fd1d0131f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1d4ff0640 1 -- 172.21.15.26:0/523662871 reap_dead start 2024-09-16T14:57:51.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1d4ff0640 1 -- 172.21.15.26:0/523662871 reap_dead start 2024-09-16T14:57:51.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1d6a53640 1 -- 172.21.15.26:0/523662871 shutdown_connections 2024-09-16T14:57:51.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1d6a53640 1 -- 172.21.15.26:0/523662871 >> 172.21.15.26:0/523662871 conn(0x7fd1d00a7f30 msgr2=0x7fd1d006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:51.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1d6a53640 1 -- 172.21.15.26:0/523662871 shutdown_connections 2024-09-16T14:57:51.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.588+0000 7fd1d6a53640 1 -- 172.21.15.26:0/523662871 wait complete. 2024-09-16T14:57:51.630 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:57:51.630 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:57:51.631 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:57:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.740+0000 7f75ac2a4640 10 monclient: get_monmap_and_config 2024-09-16T14:57:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.740+0000 7f75ac2a4640 10 monclient: build_initial_monmap 2024-09-16T14:57:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.740+0000 7f75ac2a4640 10 monclient: monmap: 2024-09-16T14:57:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:51.745357+0000 2024-09-16T14:57:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:51.745357+0000 2024-09-16T14:57:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:51.744 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 1 Processor -- start 2024-09-16T14:57:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 1 -- start start 2024-09-16T14:57:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 10 monclient: init 2024-09-16T14:57:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:57:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 10 monclient(hunting): picked mon.noname-b con 0x7f75a4134d00 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 10 monclient(hunting): picked mon.noname-a con 0x7f75a409a1c0 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 10 monclient(hunting): picked mon.noname-c con 0x7f75a41357f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a406ca10 con 0x7f75a409a1c0 2024-09-16T14:57:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a40998e0 con 0x7f75a41357f0 2024-09-16T14:57:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a4099040 con 0x7f75a4134d00 2024-09-16T14:57:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 10 monclient(hunting): authenticate will time out at 1895.865484s 2024-09-16T14:57:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75aa040640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f75a4134d00 0x7f75a41350e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53526/0 (socket says 172.21.15.26:53526) 2024-09-16T14:57:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a983f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f75a409a1c0 0x7f75a413a230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36930/0 (socket says 172.21.15.26:36930) 2024-09-16T14:57:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75aa040640 1 -- 172.21.15.26:0/2626748767 learned_addr learned my addr 172.21.15.26:0/2626748767 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 1 -- 172.21.15.26:0/2626748767 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298845623 0 0) 0x7f75a406ca10 con 0x7f75a409a1c0 2024-09-16T14:57:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient(hunting): my global_id is 274434 2024-09-16T14:57:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 1 -- 172.21.15.26:0/2626748767 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7590003530 con 0x7f75a409a1c0 2024-09-16T14:57:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 1 -- 172.21.15.26:0/2626748767 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905896567 0 0) 0x7f75a4099040 con 0x7f75a4134d00 2024-09-16T14:57:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient(hunting): my global_id is 284348 2024-09-16T14:57:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 1 -- 172.21.15.26:0/2626748767 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a406ca10 con 0x7f75a4134d00 2024-09-16T14:57:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 1 -- 172.21.15.26:0/2626748767 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 888147759 0 0) 0x7f7590003530 con 0x7f75a409a1c0 2024-09-16T14:57:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 1 -- 172.21.15.26:0/2626748767 >> v1:172.21.15.26:6790/0 conn(0x7f75a4134d00 legacy=0x7f75a41350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 1 -- 172.21.15.26:0/2626748767 >> v1:172.21.15.189:6789/0 conn(0x7f75a41357f0 legacy=0x7f75a413d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient: _finish_hunting 0 2024-09-16T14:57:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 1 monclient: found mon.noname-a 2024-09-16T14:57:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:57:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 1 -- 172.21.15.26:0/2626748767 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a4099320 con 0x7f75a409a1c0 2024-09-16T14:57:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient: _finish_auth 0 2024-09-16T14:57:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient: _check_auth_tickets 2024-09-16T14:57:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 5 monclient: authenticate success, global_id 274434 2024-09-16T14:57:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 1 -- 172.21.15.26:0/2626748767 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7594001620 con 0x7f75a409a1c0 2024-09-16T14:57:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient: dump: 2024-09-16T14:57:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.755 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.755 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:51.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 1 -- 172.21.15.26:0/2626748767 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7594001840 con 0x7f75a409a1c0 2024-09-16T14:57:51.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:51.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 1 -- 172.21.15.26:0/2626748767 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7594001bd0 con 0x7f75a409a1c0 2024-09-16T14:57:51.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:51.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75a903e640 10 monclient: dump: 2024-09-16T14:57:51.755 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.755 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.755 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.756 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 4 set_mon_vals no callback set 2024-09-16T14:57:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 10 monclient: shutdown 2024-09-16T14:57:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 1 -- 172.21.15.26:0/2626748767 >> v1:172.21.15.26:6789/0 conn(0x7f75a409a1c0 legacy=0x7f75a413a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 1 -- 172.21.15.26:0/2626748767 shutdown_connections 2024-09-16T14:57:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 1 -- 172.21.15.26:0/2626748767 >> 172.21.15.26:0/2626748767 conn(0x7f75a40a0da0 msgr2=0x7f75a40a11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 1 -- 172.21.15.26:0/2626748767 shutdown_connections 2024-09-16T14:57:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 1 -- 172.21.15.26:0/2626748767 wait complete. 2024-09-16T14:57:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 10 monclient: build_initial_monmap 2024-09-16T14:57:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 10 monclient: monmap: 2024-09-16T14:57:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:51.758 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 1 Processor -- start 2024-09-16T14:57:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 1 -- start start 2024-09-16T14:57:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.744+0000 7f75ac2a4640 10 monclient: init 2024-09-16T14:57:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:57:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 10 monclient(hunting): picked mon.noname-c con 0x7f75a409a1c0 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 10 monclient(hunting): picked mon.noname-a con 0x7f75a4134d00 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 10 monclient(hunting): picked mon.noname-b con 0x7f75a41357f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a41ca620 con 0x7f75a4134d00 2024-09-16T14:57:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a41208c0 con 0x7f75a41357f0 2024-09-16T14:57:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a409b2c0 con 0x7f75a409a1c0 2024-09-16T14:57:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 10 monclient(hunting): authenticate will time out at 1895.868465s 2024-09-16T14:57:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75aa040640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f75a409a1c0 0x7f75a4133150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53532/0 (socket says 172.21.15.26:53532) 2024-09-16T14:57:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75aa040640 1 -- 172.21.15.26:0/3942708592 learned_addr learned my addr 172.21.15.26:0/3942708592 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230105457 0 0) 0x7f75a409b2c0 con 0x7f75a409a1c0 2024-09-16T14:57:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): my global_id is 284351 2024-09-16T14:57:51.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7588003530 con 0x7f75a409a1c0 2024-09-16T14:57:51.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058688988 0 0) 0x7f75a41ca620 con 0x7f75a4134d00 2024-09-16T14:57:51.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): my global_id is 274437 2024-09-16T14:57:51.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a409b2c0 con 0x7f75a4134d00 2024-09-16T14:57:51.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2529032741 0 0) 0x7f7588003530 con 0x7f75a409a1c0 2024-09-16T14:57:51.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75a41ca620 con 0x7f75a409a1c0 2024-09-16T14:57:51.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2465172226 0 0) 0x7f75a409b2c0 con 0x7f75a4134d00 2024-09-16T14:57:51.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7588003530 con 0x7f75a4134d00 2024-09-16T14:57:51.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f75a0001a00 con 0x7f75a409a1c0 2024-09-16T14:57:51.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:51.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): dump: 2024-09-16T14:57:51.761 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.761 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.761 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.761 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.761 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.761 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.761 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.761 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.761 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.761 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:51.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7594001b50 con 0x7f75a4134d00 2024-09-16T14:57:51.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:51.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): dump: 2024-09-16T14:57:51.762 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.762 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.762 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.762 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.762 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.762 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.762 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.762 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.762 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.763 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2760832387 0 0) 0x7f7588003530 con 0x7f75a4134d00 2024-09-16T14:57:51.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 >> v1:172.21.15.26:6790/0 conn(0x7f75a409a1c0 legacy=0x7f75a4133150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 >> v1:172.21.15.189:6789/0 conn(0x7f75a41357f0 legacy=0x7f75a41d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient: _finish_hunting 0 2024-09-16T14:57:51.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 monclient: found mon.a 2024-09-16T14:57:51.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:51.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:51.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a40a36e0 con 0x7f75a4134d00 2024-09-16T14:57:51.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient: _finish_auth 0 2024-09-16T14:57:51.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient: _check_auth_tickets 2024-09-16T14:57:51.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:51.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 5 monclient: authenticate success, global_id 274437 2024-09-16T14:57:51.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 10 monclient: _renew_subs 2024-09-16T14:57:51.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:51.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 1 -- 172.21.15.26:0/3942708592 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a40a3190 con 0x7f75a4134d00 2024-09-16T14:57:51.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 10 monclient: _renew_subs 2024-09-16T14:57:51.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:51.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 1 -- 172.21.15.26:0/3942708592 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a4120270 con 0x7f75a4134d00 2024-09-16T14:57:51.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75940011e0 con 0x7f75a4134d00 2024-09-16T14:57:51.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:51.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7594004c40 con 0x7f75a4134d00 2024-09-16T14:57:51.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:51.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75a883d640 4 set_mon_vals no callback set 2024-09-16T14:57:51.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient: dump: 2024-09-16T14:57:51.765 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.765 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.765 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.765 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.765 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.765 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.765 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.765 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.765 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.765 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:51.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75ac2a4640 1 -- 172.21.15.26:0/3942708592 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f758c006590 con 0x7f758c002830 2024-09-16T14:57:51.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f759400f6b0 con 0x7f75a4134d00 2024-09-16T14:57:51.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f7594043cd0 con 0x7f75a4134d00 2024-09-16T14:57:51.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110413538 0 0) 0x7f758c006590 con 0x7f758c002830 2024-09-16T14:57:51.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient(hunting): my global_id is 284354 2024-09-16T14:57:51.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7588003530 con 0x7f758c002830 2024-09-16T14:57:51.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1285858536 0 0) 0x7f7588003530 con 0x7f758c002830 2024-09-16T14:57:51.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f758c006590 con 0x7f758c002830 2024-09-16T14:57:51.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f75a0004c40 con 0x7f758c002830 2024-09-16T14:57:51.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:51.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient: dump: 2024-09-16T14:57:51.766 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f75aa841640 10 monclient: get_auth_request con 0x7f75880361d0 auth_method 0 2024-09-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 202785529 0 0) 0x7f758c006590 con 0x7f758c002830 2024-09-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f758c00a0e0 con 0x7f758c002830 2024-09-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f75a0007010 con 0x7f758c002830 2024-09-16T14:57:51.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:51.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:51.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 >> v1:172.21.15.26:6790/0 conn(0x7f758c002830 legacy=0x7f758c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.748+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 >> v1:172.21.15.26:6790/0 conn(0x7f758c002830 legacy=0x7f758c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:51.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f75ac2a4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:57:51.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f75ac2a4640 1 -- 172.21.15.26:0/3942708592 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f758c009d80 con 0x7f758c0090a0 2024-09-16T14:57:51.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905266588 0 0) 0x7f758c009d80 con 0x7f758c0090a0 2024-09-16T14:57:51.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 10 monclient(hunting): my global_id is 284357 2024-09-16T14:57:51.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f758c006590 con 0x7f758c0090a0 2024-09-16T14:57:51.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 151254100 0 0) 0x7f758c006590 con 0x7f758c0090a0 2024-09-16T14:57:51.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f758c009d80 con 0x7f758c0090a0 2024-09-16T14:57:51.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f75a00053c0 con 0x7f758c0090a0 2024-09-16T14:57:51.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:51.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 10 monclient: dump: 2024-09-16T14:57:51.770 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.770 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.770 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.771 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.771 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.771 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.771 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.771 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.771 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.771 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 932667867 0 0) 0x7f758c009d80 con 0x7f758c0090a0 2024-09-16T14:57:51.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f758c00c290 con 0x7f758c0090a0 2024-09-16T14:57:51.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 702423034) 0x7f75a0006810 con 0x7f758c0090a0 2024-09-16T14:57:51.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:57:51.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:51.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 >> v1:172.21.15.26:6790/0 conn(0x7f758c0090a0 legacy=0x7f758c009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f759affd640 1 -- 172.21.15.26:0/3942708592 >> v1:172.21.15.26:6790/0 conn(0x7f758c0090a0 legacy=0x7f758c009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:51.772 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:57:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:57:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 144, 2024-09-16T14:57:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:57:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:57:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:57:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:57:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:57:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:57:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:57:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:57:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:57:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:57:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:57:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:57:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:57:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:57:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:57:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:57:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:57:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:57:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:57:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:57:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:57:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:57:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:57:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:57:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:57:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:57:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:57:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:57:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:51.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.780 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:51.780 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:57:51.780 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:51.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.780 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.780 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:57:51.780 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:57:51.780 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:51.780 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:51.780 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:51.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.781 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:51.781 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:51.781 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:51.781 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:51.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.781 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:51.781 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:57:51.781 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:51.781 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.781 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.781 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:57:51.781 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:57:51.782 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:51.782 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:51.782 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:51.782 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.782 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.782 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:57:51.782 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:57:51.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.782 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:51.782 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:51.782 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:51.783 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.783 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:51.783 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:57:51.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.783 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:51.783 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:51.783 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:57:51.783 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.783 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.784 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:57:51.784 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:57:51.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f75ac2a4640 1 -- 172.21.15.26:0/3942708592 >> v1:172.21.15.189:6812/3196161582 conn(0x7f75880361d0 legacy=0x7f7588038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f75ac2a4640 10 monclient: shutdown 2024-09-16T14:57:51.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f75ac2a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:51.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f75ac2a4640 1 -- 172.21.15.26:0/3942708592 >> v1:172.21.15.26:6789/0 conn(0x7f75a4134d00 legacy=0x7f75a41cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f75aa841640 1 -- 172.21.15.26:0/3942708592 reap_dead start 2024-09-16T14:57:51.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f75aa841640 1 -- 172.21.15.26:0/3942708592 reap_dead start 2024-09-16T14:57:51.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f75ac2a4640 1 -- 172.21.15.26:0/3942708592 shutdown_connections 2024-09-16T14:57:51.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f75ac2a4640 1 -- 172.21.15.26:0/3942708592 >> 172.21.15.26:0/3942708592 conn(0x7f75a40a0da0 msgr2=0x7f75a40a1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:51.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f75ac2a4640 1 -- 172.21.15.26:0/3942708592 shutdown_connections 2024-09-16T14:57:51.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.756+0000 7f75ac2a4640 1 -- 172.21.15.26:0/3942708592 wait complete. 2024-09-16T14:57:51.786 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:57:51.786 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:57:51.786 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:57:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 10 monclient: get_monmap_and_config 2024-09-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 10 monclient: build_initial_monmap 2024-09-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 10 monclient: monmap: 2024-09-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:51.898030+0000 2024-09-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:51.898030+0000 2024-09-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:51.896 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 1 Processor -- start 2024-09-16T14:57:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 1 -- start start 2024-09-16T14:57:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 10 monclient: init 2024-09-16T14:57:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:57:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 10 monclient(hunting): picked mon.noname-a con 0x7fe7940a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 10 monclient(hunting): picked mon.noname-c con 0x7fe7940a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 10 monclient(hunting): picked mon.noname-b con 0x7fe7940a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7940a3250 con 0x7fe7940a1d10 2024-09-16T14:57:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79409f600 con 0x7fe7940a0000 2024-09-16T14:57:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7940a3830 con 0x7fe7940a2800 2024-09-16T14:57:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 10 monclient(hunting): authenticate will time out at 1896.017947s 2024-09-16T14:57:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe798db3640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe7940a1d10 0x7fe7940a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36948/0 (socket says 172.21.15.26:36948) 2024-09-16T14:57:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe798db3640 1 -- 172.21.15.26:0/3731852079 learned_addr learned my addr 172.21.15.26:0/3731852079 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7995b4640 1 --1- 172.21.15.26:0/3731852079 >> v1:172.21.15.26:6790/0 conn(0x7fe7940a2800 0x7fe79413d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53560/0 (socket says 172.21.15.26:53560) 2024-09-16T14:57:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 1 -- 172.21.15.26:0/3731852079 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965998921 0 0) 0x7fe7940a3250 con 0x7fe7940a1d10 2024-09-16T14:57:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient(hunting): my global_id is 274440 2024-09-16T14:57:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 1 -- 172.21.15.26:0/3731852079 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe77c003530 con 0x7fe7940a1d10 2024-09-16T14:57:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 1 -- 172.21.15.26:0/3731852079 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545706046 0 0) 0x7fe7940a3830 con 0x7fe7940a2800 2024-09-16T14:57:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient(hunting): my global_id is 284360 2024-09-16T14:57:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 1 -- 172.21.15.26:0/3731852079 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7940a3250 con 0x7fe7940a2800 2024-09-16T14:57:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 1 -- 172.21.15.26:0/3731852079 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3686983551 0 0) 0x7fe77c003530 con 0x7fe7940a1d10 2024-09-16T14:57:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 1 -- 172.21.15.26:0/3731852079 >> v1:172.21.15.26:6790/0 conn(0x7fe7940a2800 legacy=0x7fe79413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 1 -- 172.21.15.26:0/3731852079 >> v1:172.21.15.189:6789/0 conn(0x7fe7940a0000 legacy=0x7fe794139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient: _finish_hunting 0 2024-09-16T14:57:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 1 monclient: found mon.noname-a 2024-09-16T14:57:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:57:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 1 -- 172.21.15.26:0/3731852079 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79409ed60 con 0x7fe7940a1d10 2024-09-16T14:57:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient: _finish_auth 0 2024-09-16T14:57:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient: _check_auth_tickets 2024-09-16T14:57:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 1 -- 172.21.15.26:0/3731852079 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe784001620 con 0x7fe7940a1d10 2024-09-16T14:57:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 5 monclient: authenticate success, global_id 274440 2024-09-16T14:57:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient: dump: 2024-09-16T14:57:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.908 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 1 -- 172.21.15.26:0/3731852079 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7840018c0 con 0x7fe7940a1d10 2024-09-16T14:57:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 1 -- 172.21.15.26:0/3731852079 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe784001c50 con 0x7fe7940a1d10 2024-09-16T14:57:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe7937fe640 10 monclient: dump: 2024-09-16T14:57:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.909 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 4 set_mon_vals no callback set 2024-09-16T14:57:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 10 monclient: shutdown 2024-09-16T14:57:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 1 -- 172.21.15.26:0/3731852079 >> v1:172.21.15.26:6789/0 conn(0x7fe7940a1d10 legacy=0x7fe7940a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 1 -- 172.21.15.26:0/3731852079 shutdown_connections 2024-09-16T14:57:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 1 -- 172.21.15.26:0/3731852079 >> 172.21.15.26:0/3731852079 conn(0x7fe7940a7f30 msgr2=0x7fe7940a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 1 -- 172.21.15.26:0/3731852079 shutdown_connections 2024-09-16T14:57:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 1 -- 172.21.15.26:0/3731852079 wait complete. 2024-09-16T14:57:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 10 monclient: build_initial_monmap 2024-09-16T14:57:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.896+0000 7fe79b017640 10 monclient: monmap: 2024-09-16T14:57:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:51.911 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 1 Processor -- start 2024-09-16T14:57:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 1 -- start start 2024-09-16T14:57:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 10 monclient: init 2024-09-16T14:57:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:57:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 10 monclient(hunting): picked mon.noname-b con 0x7fe7940a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 10 monclient(hunting): picked mon.noname-a con 0x7fe7940a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 10 monclient(hunting): picked mon.noname-c con 0x7fe7940a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7940a6630 con 0x7fe7940a1d10 2024-09-16T14:57:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79409e500 con 0x7fe7940a0000 2024-09-16T14:57:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe794133ae0 con 0x7fe7940a2800 2024-09-16T14:57:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 10 monclient(hunting): authenticate will time out at 1896.020924s 2024-09-16T14:57:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7995b4640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe7940a2800 0x7fe794132d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53564/0 (socket says 172.21.15.26:53564) 2024-09-16T14:57:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe793fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe7940a1d10 0x7fe794132680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36962/0 (socket says 172.21.15.26:36962) 2024-09-16T14:57:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7995b4640 1 -- 172.21.15.26:0/2679549224 learned_addr learned my addr 172.21.15.26:0/2679549224 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121581039 0 0) 0x7fe794133ae0 con 0x7fe7940a2800 2024-09-16T14:57:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): my global_id is 284363 2024-09-16T14:57:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe780003530 con 0x7fe7940a2800 2024-09-16T14:57:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710120371 0 0) 0x7fe7940a6630 con 0x7fe7940a1d10 2024-09-16T14:57:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): my global_id is 274443 2024-09-16T14:57:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe794133ae0 con 0x7fe7940a1d10 2024-09-16T14:57:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1577345942 0 0) 0x7fe780003530 con 0x7fe7940a2800 2024-09-16T14:57:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7940a6630 con 0x7fe7940a2800 2024-09-16T14:57:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe78c001c60 con 0x7fe7940a2800 2024-09-16T14:57:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): dump: 2024-09-16T14:57:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.915 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1796800345 0 0) 0x7fe794133ae0 con 0x7fe7940a1d10 2024-09-16T14:57:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe780003530 con 0x7fe7940a1d10 2024-09-16T14:57:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe788001840 con 0x7fe7940a1d10 2024-09-16T14:57:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): dump: 2024-09-16T14:57:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.916 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873261556 0 0) 0x7fe79409e500 con 0x7fe7940a0000 2024-09-16T14:57:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): my global_id is 304108 2024-09-16T14:57:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe794133ae0 con 0x7fe7940a0000 2024-09-16T14:57:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2580954356 0 0) 0x7fe7940a6630 con 0x7fe7940a2800 2024-09-16T14:57:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 >> v1:172.21.15.189:6789/0 conn(0x7fe7940a0000 legacy=0x7fe794131f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 >> v1:172.21.15.26:6789/0 conn(0x7fe7940a1d10 legacy=0x7fe794132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient: _finish_hunting 0 2024-09-16T14:57:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 monclient: found mon.c 2024-09-16T14:57:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe794133fc0 con 0x7fe7940a2800 2024-09-16T14:57:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient: _finish_auth 0 2024-09-16T14:57:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient: _check_auth_tickets 2024-09-16T14:57:51.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:51.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 5 monclient: authenticate success, global_id 284363 2024-09-16T14:57:51.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 10 monclient: _renew_subs 2024-09-16T14:57:51.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:51.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 1 -- 172.21.15.26:0/2679549224 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe794134430 con 0x7fe7940a2800 2024-09-16T14:57:51.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 10 monclient: _renew_subs 2024-09-16T14:57:51.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:51.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 1 -- 172.21.15.26:0/2679549224 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe79413da70 con 0x7fe7940a2800 2024-09-16T14:57:51.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe78c004b70 con 0x7fe7940a2800 2024-09-16T14:57:51.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:51.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe78c004e80 con 0x7fe7940a2800 2024-09-16T14:57:51.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient: dump: 2024-09-16T14:57:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.919 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe792ffd640 4 set_mon_vals no callback set 2024-09-16T14:57:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe79b017640 1 -- 172.21.15.26:0/2679549224 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe774006590 con 0x7fe774002830 2024-09-16T14:57:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe78c00f8f0 con 0x7fe7940a2800 2024-09-16T14:57:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fe78c044080 con 0x7fe7940a2800 2024-09-16T14:57:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe793fff640 10 monclient: get_auth_request con 0x7fe780036490 auth_method 0 2024-09-16T14:57:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936069518 0 0) 0x7fe774006590 con 0x7fe774002830 2024-09-16T14:57:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient(hunting): my global_id is 304111 2024-09-16T14:57:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7940a6630 con 0x7fe774002830 2024-09-16T14:57:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3008202637 0 0) 0x7fe7940a6630 con 0x7fe774002830 2024-09-16T14:57:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe774006590 con 0x7fe774002830 2024-09-16T14:57:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe784001190 con 0x7fe774002830 2024-09-16T14:57:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:57:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.900+0000 7fe7917fa640 10 monclient: dump: 2024-09-16T14:57:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.921 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.904+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3945582855 0 0) 0x7fe774006590 con 0x7fe774002830 2024-09-16T14:57:51.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.904+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe77400a0e0 con 0x7fe774002830 2024-09-16T14:57:51.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.904+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe784001c40 con 0x7fe774002830 2024-09-16T14:57:51.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.904+0000 7fe7917fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:51.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.904+0000 7fe7917fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:51.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.904+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 >> v1:172.21.15.189:6789/0 conn(0x7fe774002830 legacy=0x7fe774004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.904+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 >> v1:172.21.15.189:6789/0 conn(0x7fe774002830 legacy=0x7fe774004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:51.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.908+0000 7fe79b017640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:57:51.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.908+0000 7fe79b017640 1 -- 172.21.15.26:0/2679549224 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe774009d80 con 0x7fe7740090a0 2024-09-16T14:57:51.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.908+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990581725 0 0) 0x7fe774009d80 con 0x7fe7740090a0 2024-09-16T14:57:51.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.908+0000 7fe7917fa640 10 monclient(hunting): _negotiate 2024-09-16T14:57:51.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.908+0000 7fe7917fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:51.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.908+0000 7fe7917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:51.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.908+0000 7fe7917fa640 10 monclient(hunting): my global_id is 304114 2024-09-16T14:57:51.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.908+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe774006590 con 0x7fe7740090a0 2024-09-16T14:57:51.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3958083923 0 0) 0x7fe774006590 con 0x7fe7740090a0 2024-09-16T14:57:51.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe774009d80 con 0x7fe7740090a0 2024-09-16T14:57:51.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe784004ce0 con 0x7fe7740090a0 2024-09-16T14:57:51.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe7917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:51.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe7917fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:57:51.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe7917fa640 10 monclient: dump: 2024-09-16T14:57:51.924 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:51.924 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:51.925 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.925 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:51.925 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:51.925 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:51.925 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:51.925 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:51.925 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:51.925 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:51.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2797075574 0 0) 0x7fe774009d80 con 0x7fe7740090a0 2024-09-16T14:57:51.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe77400c290 con 0x7fe7740090a0 2024-09-16T14:57:51.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 1484093908) 0x7fe784005f30 con 0x7fe7740090a0 2024-09-16T14:57:51.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe7917fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:57:51.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe7917fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:51.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 >> v1:172.21.15.189:6789/0 conn(0x7fe7740090a0 legacy=0x7fe774009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe7917fa640 1 -- 172.21.15.26:0/2679549224 >> v1:172.21.15.189:6789/0 conn(0x7fe7740090a0 legacy=0x7fe774009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:51.926 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:57:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:57:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 144, 2024-09-16T14:57:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:57:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:57:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:57:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:57:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:57:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:57:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:57:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:57:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:57:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:57:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:57:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:57:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:57:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:57:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:57:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:57:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:57:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:57:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:57:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:57:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:57:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:57:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:57:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:57:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:57:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:57:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:57:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:57:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:57:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:57:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:57:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:57:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:57:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:57:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:57:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:57:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:57:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:57:51.937 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:57:51.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe79b017640 1 -- 172.21.15.26:0/2679549224 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe780036490 legacy=0x7fe780038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe79b017640 10 monclient: shutdown 2024-09-16T14:57:51.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe79b017640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:51.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe79b017640 1 -- 172.21.15.26:0/2679549224 >> v1:172.21.15.26:6790/0 conn(0x7fe7940a2800 legacy=0x7fe794132d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:51.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe7995b4640 1 -- 172.21.15.26:0/2679549224 reap_dead start 2024-09-16T14:57:51.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe7995b4640 1 -- 172.21.15.26:0/2679549224 reap_dead start 2024-09-16T14:57:51.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe79b017640 1 -- 172.21.15.26:0/2679549224 shutdown_connections 2024-09-16T14:57:51.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe79b017640 1 -- 172.21.15.26:0/2679549224 >> 172.21.15.26:0/2679549224 conn(0x7fe7940a7f30 msgr2=0x7fe79406b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:51.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe79b017640 1 -- 172.21.15.26:0/2679549224 shutdown_connections 2024-09-16T14:57:51.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:51.912+0000 7fe79b017640 1 -- 172.21.15.26:0/2679549224 wait complete. 2024-09-16T14:57:51.940 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:57:51.940 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:57:51.941 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:57:52.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.056+0000 7f1c4c0ae640 10 monclient: get_monmap_and_config 2024-09-16T14:57:52.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.056+0000 7f1c4c0ae640 10 monclient: build_initial_monmap 2024-09-16T14:57:52.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.056+0000 7f1c4c0ae640 10 monclient: monmap: 2024-09-16T14:57:52.058 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:52.058 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:52.058 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:52.059942+0000 2024-09-16T14:57:52.058 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:52.059942+0000 2024-09-16T14:57:52.058 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:52.058 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:52.058 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:52.058 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:52.058 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:52.058 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.056+0000 7f1c4c0ae640 1 Processor -- start 2024-09-16T14:57:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.056+0000 7f1c4c0ae640 1 -- start start 2024-09-16T14:57:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.056+0000 7f1c4c0ae640 10 monclient: init 2024-09-16T14:57:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:57:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient(hunting): picked mon.noname-b con 0x7f1c440a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient(hunting): picked mon.noname-c con 0x7f1c440a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient(hunting): picked mon.noname-a con 0x7f1c440a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c440a3250 con 0x7f1c440a2800 2024-09-16T14:57:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4409f600 con 0x7f1c440a0000 2024-09-16T14:57:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c440a3830 con 0x7f1c440a1d10 2024-09-16T14:57:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient(hunting): authenticate will time out at 1896.180465s 2024-09-16T14:57:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c49e4a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1c440a1d10 0x7f1c440a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53570/0 (socket says 172.21.15.26:53570) 2024-09-16T14:57:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c49e4a640 1 -- 172.21.15.26:0/869054797 learned_addr learned my addr 172.21.15.26:0/869054797 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4a64b640 1 --1- 172.21.15.26:0/869054797 >> v1:172.21.15.26:6789/0 conn(0x7f1c440a2800 0x7f1c4413d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36968/0 (socket says 172.21.15.26:36968) 2024-09-16T14:57:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 1 -- 172.21.15.26:0/869054797 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202708677 0 0) 0x7f1c440a3830 con 0x7f1c440a1d10 2024-09-16T14:57:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient(hunting): _negotiate 2024-09-16T14:57:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient(hunting): my global_id is 284366 2024-09-16T14:57:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 1 -- 172.21.15.26:0/869054797 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c30003530 con 0x7f1c440a1d10 2024-09-16T14:57:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 1 -- 172.21.15.26:0/869054797 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076848973 0 0) 0x7f1c440a3250 con 0x7f1c440a2800 2024-09-16T14:57:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient(hunting): _negotiate 2024-09-16T14:57:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient(hunting): my global_id is 274446 2024-09-16T14:57:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 1 -- 172.21.15.26:0/869054797 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c440a3830 con 0x7f1c440a2800 2024-09-16T14:57:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 1 -- 172.21.15.26:0/869054797 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 333509893 0 0) 0x7f1c30003530 con 0x7f1c440a1d10 2024-09-16T14:57:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 1 -- 172.21.15.26:0/869054797 >> v1:172.21.15.189:6789/0 conn(0x7f1c440a0000 legacy=0x7f1c44139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 1 -- 172.21.15.26:0/869054797 >> v1:172.21.15.26:6789/0 conn(0x7f1c440a2800 legacy=0x7f1c4413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient: _finish_hunting 0 2024-09-16T14:57:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 1 monclient: found mon.noname-b 2024-09-16T14:57:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:57:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 1 -- 172.21.15.26:0/869054797 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4409ed60 con 0x7f1c440a1d10 2024-09-16T14:57:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient: _finish_auth 0 2024-09-16T14:57:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient: _check_auth_tickets 2024-09-16T14:57:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 1 -- 172.21.15.26:0/869054797 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1c40001620 con 0x7f1c440a1d10 2024-09-16T14:57:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:57:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient: dump: 2024-09-16T14:57:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:52.066 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 5 monclient: authenticate success, global_id 284366 2024-09-16T14:57:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 1 -- 172.21.15.26:0/869054797 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c40001900 con 0x7f1c440a1d10 2024-09-16T14:57:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 1 -- 172.21.15.26:0/869054797 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1c40001c90 con 0x7f1c440a1d10 2024-09-16T14:57:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 4 set_mon_vals no callback set 2024-09-16T14:57:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c48e48640 10 monclient: dump: 2024-09-16T14:57:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:52.067 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient: shutdown 2024-09-16T14:57:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 1 -- 172.21.15.26:0/869054797 >> v1:172.21.15.26:6790/0 conn(0x7f1c440a1d10 legacy=0x7f1c440a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 1 -- 172.21.15.26:0/869054797 shutdown_connections 2024-09-16T14:57:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 1 -- 172.21.15.26:0/869054797 >> 172.21.15.26:0/869054797 conn(0x7f1c440a7f30 msgr2=0x7f1c440a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 1 -- 172.21.15.26:0/869054797 shutdown_connections 2024-09-16T14:57:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 1 -- 172.21.15.26:0/869054797 wait complete. 2024-09-16T14:57:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient: build_initial_monmap 2024-09-16T14:57:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient: monmap: 2024-09-16T14:57:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:52.069 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 1 Processor -- start 2024-09-16T14:57:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 1 -- start start 2024-09-16T14:57:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient: init 2024-09-16T14:57:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:57:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient(hunting): picked mon.noname-a con 0x7f1c440a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient(hunting): picked mon.noname-b con 0x7f1c440a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient(hunting): picked mon.noname-c con 0x7f1c440a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c44133ec0 con 0x7f1c440a0000 2024-09-16T14:57:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c440a6750 con 0x7f1c440a1d10 2024-09-16T14:57:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c441315b0 con 0x7f1c440a2800 2024-09-16T14:57:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c4c0ae640 10 monclient(hunting): authenticate will time out at 1896.183572s 2024-09-16T14:57:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c49e4a640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1c440a0000 0x7f1c441301e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36982/0 (socket says 172.21.15.26:36982) 2024-09-16T14:57:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c49e4a640 1 -- 172.21.15.26:0/1246969993 learned_addr learned my addr 172.21.15.26:0/1246969993 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254627130 0 0) 0x7f1c44133ec0 con 0x7f1c440a0000 2024-09-16T14:57:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c3a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c3a7fc640 10 monclient(hunting): my global_id is 274449 2024-09-16T14:57:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c28003530 con 0x7f1c440a0000 2024-09-16T14:57:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543502769 0 0) 0x7f1c441315b0 con 0x7f1c440a2800 2024-09-16T14:57:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c3a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:57:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c3a7fc640 10 monclient(hunting): my global_id is 284369 2024-09-16T14:57:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.060+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c44133ec0 con 0x7f1c440a2800 2024-09-16T14:57:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3642421279 0 0) 0x7f1c28003530 con 0x7f1c440a0000 2024-09-16T14:57:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c441315b0 con 0x7f1c440a0000 2024-09-16T14:57:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1c400015e0 con 0x7f1c440a0000 2024-09-16T14:57:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 10 monclient(hunting): dump: 2024-09-16T14:57:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:52.075 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 976065128 0 0) 0x7f1c44133ec0 con 0x7f1c440a2800 2024-09-16T14:57:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c28003530 con 0x7f1c440a2800 2024-09-16T14:57:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1c3c001c40 con 0x7f1c440a2800 2024-09-16T14:57:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 10 monclient(hunting): dump: 2024-09-16T14:57:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:52.077 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4284883467 0 0) 0x7f1c441315b0 con 0x7f1c440a0000 2024-09-16T14:57:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 >> v1:172.21.15.26:6790/0 conn(0x7f1c440a2800 legacy=0x7f1c441d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 >> v1:172.21.15.189:6789/0 conn(0x7f1c440a1d10 legacy=0x7f1c441308f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:57:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 1 monclient: found mon.a 2024-09-16T14:57:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c44131000 con 0x7f1c440a0000 2024-09-16T14:57:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:57:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:57:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c4c0ae640 5 monclient: authenticate success, global_id 274449 2024-09-16T14:57:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c4c0ae640 10 monclient: _renew_subs 2024-09-16T14:57:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c4c0ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c4c0ae640 1 -- 172.21.15.26:0/1246969993 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c44132070 con 0x7f1c440a0000 2024-09-16T14:57:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c4c0ae640 10 monclient: _renew_subs 2024-09-16T14:57:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c4c0ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c4c0ae640 1 -- 172.21.15.26:0/1246969993 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c44131b60 con 0x7f1c440a0000 2024-09-16T14:57:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c40001190 con 0x7f1c440a0000 2024-09-16T14:57:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1c40004da0 con 0x7f1c440a0000 2024-09-16T14:57:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3bfff640 4 set_mon_vals no callback set 2024-09-16T14:57:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 10 monclient: dump: 2024-09-16T14:57:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:52.079 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c4c0ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c4c0ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c4c0ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c4c0ae640 1 -- 172.21.15.26:0/1246969993 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2c005050 con 0x7f1c440a0000 2024-09-16T14:57:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1c4000f810 con 0x7f1c440a0000 2024-09-16T14:57:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f1c40043db0 con 0x7f1c440a0000 2024-09-16T14:57:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.064+0000 7f1c49649640 10 monclient: get_auth_request con 0x7f1c28036270 auth_method 0 2024-09-16T14:57:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.068+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1c40014670 con 0x7f1c440a0000 2024-09-16T14:57:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.068+0000 7f1c3a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.068+0000 7f1c3a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:52.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.280+0000 7f1c4c0ae640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:57:52.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.280+0000 7f1c4c0ae640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:57:52.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.280+0000 7f1c4c0ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:52.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.280+0000 7f1c4c0ae640 1 -- 172.21.15.26:0/1246969993 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f1c2c005b10 con 0x7f1c440a0000 2024-09-16T14:57:52.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.280+0000 7f1c3a7fc640 1 -- 172.21.15.26:0/1246969993 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f1c4001ce20 con 0x7f1c440a0000 2024-09-16T14:57:52.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.280+0000 7f1c3a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:57:52.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.280+0000 7f1c3a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:52.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.280+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.280+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=11,kv=10,logm=51} crc {auth=1623430971,config=692619927,health=1726845097,kv=3455456366,logm=104732795}) 2024-09-16T14:57:52.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.284+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=10,kv=10,logm=53} crc {auth=837128673,config=692619927,health=2005095319,kv=17738942,logm=2001372550}) 2024-09-16T14:57:52.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.284+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.284 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.284+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=11,kv=10,logm=51} crc {auth=1623430971,config=692619927,health=1726845097,kv=3455456366,logm=104732795}) 2024-09-16T14:57:52.284 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.284+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=9,logm=52} crc {auth=3838329989,config=692619927,health=1596331942,kv=655417215,logm=2223953727}) 2024-09-16T14:57:52.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.284+0000 7f1c4c0ae640 1 -- 172.21.15.26:0/1246969993 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1c28036270 legacy=0x7f1c28038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:52.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.284+0000 7f1c4c0ae640 10 monclient: shutdown 2024-09-16T14:57:52.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.284+0000 7f1c4c0ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:52.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.284+0000 7f1c4c0ae640 1 -- 172.21.15.26:0/1246969993 >> v1:172.21.15.26:6789/0 conn(0x7f1c440a0000 legacy=0x7f1c441301e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:52.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.284+0000 7f1c4c0ae640 1 -- 172.21.15.26:0/1246969993 shutdown_connections 2024-09-16T14:57:52.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.284+0000 7f1c4c0ae640 1 -- 172.21.15.26:0/1246969993 >> 172.21.15.26:0/1246969993 conn(0x7f1c440a7f30 msgr2=0x7f1c4406b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:52.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.284+0000 7f1c4c0ae640 1 -- 172.21.15.26:0/1246969993 shutdown_connections 2024-09-16T14:57:52.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:52.284+0000 7f1c4c0ae640 1 -- 172.21.15.26:0/1246969993 wait complete. 2024-09-16T14:57:52.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=895824967}) 2024-09-16T14:57:52.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1394790161}) 2024-09-16T14:57:52.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=895824967}) 2024-09-16T14:57:52.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.288+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2385388529}) 2024-09-16T14:57:52.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1528287259}) 2024-09-16T14:57:52.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4164327683}) 2024-09-16T14:57:52.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1528287259}) 2024-09-16T14:57:52.292 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.292+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1438459599}) 2024-09-16T14:57:52.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.296+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.296+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1984490973}) 2024-09-16T14:57:52.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.296+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752884473}) 2024-09-16T14:57:52.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.296+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.296+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1984490973}) 2024-09-16T14:57:52.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.296+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2581001245}) 2024-09-16T14:57:52.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.296+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.296+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969661534}) 2024-09-16T14:57:52.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.296+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2847865007}) 2024-09-16T14:57:52.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.296+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.296+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969661534}) 2024-09-16T14:57:52.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.296+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1820569268}) 2024-09-16T14:57:52.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.300+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.300+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3558795120}) 2024-09-16T14:57:52.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.300+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554778178}) 2024-09-16T14:57:52.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.300+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.300+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3558795120}) 2024-09-16T14:57:52.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.300+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=346863099}) 2024-09-16T14:57:52.301 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:57:52.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.300+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.300+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2132949479}) 2024-09-16T14:57:52.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.304+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3017329294}) 2024-09-16T14:57:52.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.304+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.304+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2132949479}) 2024-09-16T14:57:52.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.304+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=71298669}) 2024-09-16T14:57:52.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.304+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.304+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1421546368}) 2024-09-16T14:57:52.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.304+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=269522781}) 2024-09-16T14:57:52.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.304+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.306 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.304+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1421546368}) 2024-09-16T14:57:52.306 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.304+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=664682289}) 2024-09-16T14:57:52.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=865512465}) 2024-09-16T14:57:52.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144704904}) 2024-09-16T14:57:52.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=865512465}) 2024-09-16T14:57:52.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1958022700}) 2024-09-16T14:57:52.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2441041996}) 2024-09-16T14:57:52.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=754779099}) 2024-09-16T14:57:52.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2441041996}) 2024-09-16T14:57:52.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.308+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669259570}) 2024-09-16T14:57:52.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.312+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.312+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3319628152}) 2024-09-16T14:57:52.312 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.312+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1198647175}) 2024-09-16T14:57:52.312 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.312+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.312 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.312+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3319628152}) 2024-09-16T14:57:52.312 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.312+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1572026181}) 2024-09-16T14:57:52.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.312+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.314 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.312+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=351745274}) 2024-09-16T14:57:52.314 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.312+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=999125118}) 2024-09-16T14:57:52.314 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.312+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.314 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.312+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=351745274}) 2024-09-16T14:57:52.314 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.312+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4029623952}) 2024-09-16T14:57:52.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.316+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.316+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1864515995}) 2024-09-16T14:57:52.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.316+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4092340268}) 2024-09-16T14:57:52.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.316+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.316+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1864515995}) 2024-09-16T14:57:52.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.316+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3077985397}) 2024-09-16T14:57:52.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.316+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.316+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=453531066}) 2024-09-16T14:57:52.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.316+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3458128087}) 2024-09-16T14:57:52.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.316+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.316+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=453531066}) 2024-09-16T14:57:52.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.316+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1701482179}) 2024-09-16T14:57:52.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.320+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.320+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2491042274}) 2024-09-16T14:57:52.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.320+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4245752041}) 2024-09-16T14:57:52.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.320+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.320+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2491042274}) 2024-09-16T14:57:52.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.320+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1787855940}) 2024-09-16T14:57:52.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.320+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.320+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=960075967}) 2024-09-16T14:57:52.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.320+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286600387}) 2024-09-16T14:57:52.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.320+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.320+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=960075967}) 2024-09-16T14:57:52.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.320+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1044684045}) 2024-09-16T14:57:52.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.324+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.324+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=977075788}) 2024-09-16T14:57:52.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.324+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3899009519}) 2024-09-16T14:57:52.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.324+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.324+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=977075788}) 2024-09-16T14:57:52.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.324+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4200750830}) 2024-09-16T14:57:52.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.324+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.324+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3837487476}) 2024-09-16T14:57:52.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.324+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2770024396}) 2024-09-16T14:57:52.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.324+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.324+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3837487476}) 2024-09-16T14:57:52.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.324+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1252029894}) 2024-09-16T14:57:52.328 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.328+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.328+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=258754242}) 2024-09-16T14:57:52.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.328+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586621635}) 2024-09-16T14:57:52.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.328+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.328+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=258754242}) 2024-09-16T14:57:52.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.328+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4146835371}) 2024-09-16T14:57:52.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.328+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.328+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663520967}) 2024-09-16T14:57:52.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.328+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4147617489}) 2024-09-16T14:57:52.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.328+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.328+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663520967}) 2024-09-16T14:57:52.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.328+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3814916090}) 2024-09-16T14:57:52.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.332+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.332+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1224348537}) 2024-09-16T14:57:52.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.332+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1406190766}) 2024-09-16T14:57:52.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.332+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.332+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1224348537}) 2024-09-16T14:57:52.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.332+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4191947470}) 2024-09-16T14:57:52.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.332+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.332+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=714078967}) 2024-09-16T14:57:52.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.332+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1847035435}) 2024-09-16T14:57:52.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.332+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.332+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=714078967}) 2024-09-16T14:57:52.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.332+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1326521510}) 2024-09-16T14:57:52.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.336+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.336+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1333278803}) 2024-09-16T14:57:52.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.336+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4173548789}) 2024-09-16T14:57:52.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.336+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.336+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1333278803}) 2024-09-16T14:57:52.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.336+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=944620936}) 2024-09-16T14:57:52.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.340+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.340+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2269408474}) 2024-09-16T14:57:52.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.340+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810104604}) 2024-09-16T14:57:52.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.340+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.340+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2269408474}) 2024-09-16T14:57:52.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.340+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2309043914}) 2024-09-16T14:57:52.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.340+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.340+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3626834304}) 2024-09-16T14:57:52.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.340+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912601981}) 2024-09-16T14:57:52.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.340+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.340+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3626834304}) 2024-09-16T14:57:52.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.340+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3997951223}) 2024-09-16T14:57:52.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.344+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.344+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2157829384}) 2024-09-16T14:57:52.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.344+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1912769122}) 2024-09-16T14:57:52.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.344+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.344+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2157829384}) 2024-09-16T14:57:52.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.344+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3761889540}) 2024-09-16T14:57:52.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.344+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.344+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4232857870}) 2024-09-16T14:57:52.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.344+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2685333292}) 2024-09-16T14:57:52.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.344+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.344+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4232857870}) 2024-09-16T14:57:52.346 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.344+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=346696062}) 2024-09-16T14:57:52.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.348+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.348+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1874482896}) 2024-09-16T14:57:52.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.348+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2928282728}) 2024-09-16T14:57:52.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.348+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.348+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1874482896}) 2024-09-16T14:57:52.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.348+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1815610276}) 2024-09-16T14:57:52.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.348+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.348+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=917181364}) 2024-09-16T14:57:52.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.348+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2482858133}) 2024-09-16T14:57:52.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.348+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.348+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=917181364}) 2024-09-16T14:57:52.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.348+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3097106729}) 2024-09-16T14:57:52.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.352+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.352+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149875925}) 2024-09-16T14:57:52.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.352+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2717349376}) 2024-09-16T14:57:52.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.352+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.352+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149875925}) 2024-09-16T14:57:52.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.352+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2702253843}) 2024-09-16T14:57:52.353 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.352+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.353 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.352+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1854098949}) 2024-09-16T14:57:52.353 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.352+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1458812832}) 2024-09-16T14:57:52.353 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.352+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.353 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.352+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1854098949}) 2024-09-16T14:57:52.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.352+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3306434461}) 2024-09-16T14:57:52.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.356+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.356+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=145371542}) 2024-09-16T14:57:52.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.356+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676145199}) 2024-09-16T14:57:52.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.356+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.356+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=145371542}) 2024-09-16T14:57:52.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.356+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2569266852}) 2024-09-16T14:57:52.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.356+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.356+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3055352446}) 2024-09-16T14:57:52.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.356+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3179840030}) 2024-09-16T14:57:52.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.356+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.356+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3055352446}) 2024-09-16T14:57:52.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.356+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1023213676}) 2024-09-16T14:57:52.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.360+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.360+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2711842378}) 2024-09-16T14:57:52.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.360+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=328806068}) 2024-09-16T14:57:52.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.360+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.360+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2711842378}) 2024-09-16T14:57:52.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.360+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612082393}) 2024-09-16T14:57:52.361 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.360+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.361 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.360+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1134187970}) 2024-09-16T14:57:52.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.360+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2969328293}) 2024-09-16T14:57:52.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.360+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.360+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1134187970}) 2024-09-16T14:57:52.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.360+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1518648799}) 2024-09-16T14:57:52.363 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.364+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.363 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.364+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=797235802}) 2024-09-16T14:57:52.363 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.364+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=283521069}) 2024-09-16T14:57:52.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.364+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.364+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=797235802}) 2024-09-16T14:57:52.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.364+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1489398690}) 2024-09-16T14:57:52.365 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.364+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.365 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.364+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829270702}) 2024-09-16T14:57:52.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.364+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3387116126}) 2024-09-16T14:57:52.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.364+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.364+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829270702}) 2024-09-16T14:57:52.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.364+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=488739933}) 2024-09-16T14:57:52.367 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.368+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.367 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.368+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1553838629}) 2024-09-16T14:57:52.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.368+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=959037957}) 2024-09-16T14:57:52.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.368+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.368+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1553838629}) 2024-09-16T14:57:52.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.368+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1393338467}) 2024-09-16T14:57:52.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.368+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.368+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2099228765}) 2024-09-16T14:57:52.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.368+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3964519333}) 2024-09-16T14:57:52.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.368+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.368+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2099228765}) 2024-09-16T14:57:52.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.368+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1884488494}) 2024-09-16T14:57:52.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.372+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.372 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.372+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3286471675}) 2024-09-16T14:57:52.372 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.372+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3463998859}) 2024-09-16T14:57:52.372 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.372+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.372 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.372+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3286471675}) 2024-09-16T14:57:52.372 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.372+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3374655711}) 2024-09-16T14:57:52.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.372+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.372+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3707488479}) 2024-09-16T14:57:52.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.372+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3603151905}) 2024-09-16T14:57:52.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.372+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.374 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.372+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3707488479}) 2024-09-16T14:57:52.374 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.372+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=505169832}) 2024-09-16T14:57:52.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.376+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.376+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=503850123}) 2024-09-16T14:57:52.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.376+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661827717}) 2024-09-16T14:57:52.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.376+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.376+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=503850123}) 2024-09-16T14:57:52.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.376+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3748477378}) 2024-09-16T14:57:52.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.376+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.376+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4217797062}) 2024-09-16T14:57:52.378 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.376+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1263568071}) 2024-09-16T14:57:52.378 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.376+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.378 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.376+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4217797062}) 2024-09-16T14:57:52.378 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.376+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1302703977}) 2024-09-16T14:57:52.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=726417019}) 2024-09-16T14:57:52.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1570385574}) 2024-09-16T14:57:52.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=726417019}) 2024-09-16T14:57:52.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1100994997}) 2024-09-16T14:57:52.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=224163493}) 2024-09-16T14:57:52.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709006242}) 2024-09-16T14:57:52.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=224163493}) 2024-09-16T14:57:52.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157539423}) 2024-09-16T14:57:52.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2823529}) 2024-09-16T14:57:52.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=326176637}) 2024-09-16T14:57:52.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2823529}) 2024-09-16T14:57:52.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.380+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2702239064}) 2024-09-16T14:57:52.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.384+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.384+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3293380013}) 2024-09-16T14:57:52.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.384+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3682105582}) 2024-09-16T14:57:52.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.384+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.384+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3293380013}) 2024-09-16T14:57:52.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.384+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1399945688}) 2024-09-16T14:57:52.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.384+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.384+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2513100944}) 2024-09-16T14:57:52.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.384+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2264638840}) 2024-09-16T14:57:52.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.384+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.384+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2513100944}) 2024-09-16T14:57:52.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.384+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2327743706}) 2024-09-16T14:57:52.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.388+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.388+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2771332103}) 2024-09-16T14:57:52.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.388+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2041138217}) 2024-09-16T14:57:52.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.388+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.388+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2771332103}) 2024-09-16T14:57:52.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.388+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=901618454}) 2024-09-16T14:57:52.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.388+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.391 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.388+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3462943724}) 2024-09-16T14:57:52.391 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.388+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390044475}) 2024-09-16T14:57:52.391 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.388+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.391 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.388+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3462943724}) 2024-09-16T14:57:52.391 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.388+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=60639802}) 2024-09-16T14:57:52.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.392+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.392+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1749291938}) 2024-09-16T14:57:52.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.392+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1473609122}) 2024-09-16T14:57:52.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.392+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.393 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.392+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1749291938}) 2024-09-16T14:57:52.393 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.392+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1692833980}) 2024-09-16T14:57:52.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.392+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.392+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1184076180}) 2024-09-16T14:57:52.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.392+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546960709}) 2024-09-16T14:57:52.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.392+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.392+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1184076180}) 2024-09-16T14:57:52.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.392+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264408462}) 2024-09-16T14:57:52.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.396+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.396+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3412765197}) 2024-09-16T14:57:52.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.396+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2287577124}) 2024-09-16T14:57:52.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.396+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.396+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3412765197}) 2024-09-16T14:57:52.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.396+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2324345980}) 2024-09-16T14:57:52.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.396+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.396+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=385671430}) 2024-09-16T14:57:52.398 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.396+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2792138979}) 2024-09-16T14:57:52.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.396+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.396+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=385671430}) 2024-09-16T14:57:52.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.396+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=172980585}) 2024-09-16T14:57:52.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.400+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.400+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2945648055}) 2024-09-16T14:57:52.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.400+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706417235}) 2024-09-16T14:57:52.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.400+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.400+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2945648055}) 2024-09-16T14:57:52.400 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.400+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=993465224}) 2024-09-16T14:57:52.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.400+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.400+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1239019766}) 2024-09-16T14:57:52.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.400+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2190510071}) 2024-09-16T14:57:52.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.400+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.400+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1239019766}) 2024-09-16T14:57:52.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.400+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1198371821}) 2024-09-16T14:57:52.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.404+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.404+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507212614}) 2024-09-16T14:57:52.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.404+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2437365323}) 2024-09-16T14:57:52.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.404+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.404+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507212614}) 2024-09-16T14:57:52.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.404+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3186086433}) 2024-09-16T14:57:52.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.404+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.404+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1253583352}) 2024-09-16T14:57:52.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.404+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3371900505}) 2024-09-16T14:57:52.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.404+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.404+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1253583352}) 2024-09-16T14:57:52.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.404+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2636464715}) 2024-09-16T14:57:52.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.408+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.408+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3380127672}) 2024-09-16T14:57:52.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.408+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2232269825}) 2024-09-16T14:57:52.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.408+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.408+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3380127672}) 2024-09-16T14:57:52.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.408+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1128302532}) 2024-09-16T14:57:52.409 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.408+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.409 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.408+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3575496612}) 2024-09-16T14:57:52.409 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.408+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2078949658}) 2024-09-16T14:57:52.409 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.408+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.408+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3575496612}) 2024-09-16T14:57:52.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.408+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3659770154}) 2024-09-16T14:57:52.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.412+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.412+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1466510692}) 2024-09-16T14:57:52.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.412+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119701812}) 2024-09-16T14:57:52.412 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.412+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.412+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1466510692}) 2024-09-16T14:57:52.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.412+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2824404205}) 2024-09-16T14:57:52.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.412+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.414 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.412+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3325094719}) 2024-09-16T14:57:52.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.412+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778941057}) 2024-09-16T14:57:52.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.412+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.412+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3325094719}) 2024-09-16T14:57:52.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.412+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3171122571}) 2024-09-16T14:57:52.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.412+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=415582299}) 2024-09-16T14:57:52.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=609866284}) 2024-09-16T14:57:52.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=415582299}) 2024-09-16T14:57:52.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3932174480}) 2024-09-16T14:57:52.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2973348070}) 2024-09-16T14:57:52.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3302783116}) 2024-09-16T14:57:52.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2973348070}) 2024-09-16T14:57:52.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=465065618}) 2024-09-16T14:57:52.419 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.419 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=513355432}) 2024-09-16T14:57:52.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946512712}) 2024-09-16T14:57:52.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=513355432}) 2024-09-16T14:57:52.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.416+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3372563320}) 2024-09-16T14:57:52.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.420+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.420+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1599949248}) 2024-09-16T14:57:52.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.420+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1555223053}) 2024-09-16T14:57:52.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.420+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.420+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1599949248}) 2024-09-16T14:57:52.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.420+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1237593789}) 2024-09-16T14:57:52.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.420+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.420+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2955223350}) 2024-09-16T14:57:52.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.420+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=464255866}) 2024-09-16T14:57:52.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.420+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.420+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2955223350}) 2024-09-16T14:57:52.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.420+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3319315858}) 2024-09-16T14:57:52.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.424+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.424+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3618097874}) 2024-09-16T14:57:52.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.424+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=291602122}) 2024-09-16T14:57:52.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.424+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.424+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3618097874}) 2024-09-16T14:57:52.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.424+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3219648242}) 2024-09-16T14:57:52.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.424+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.424+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1306867727}) 2024-09-16T14:57:52.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.424+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268875922}) 2024-09-16T14:57:52.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.424+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.424+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1306867727}) 2024-09-16T14:57:52.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.424+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=229696264}) 2024-09-16T14:57:52.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.428+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.428+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3291410068}) 2024-09-16T14:57:52.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.428+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2654549551}) 2024-09-16T14:57:52.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.428+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.428+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3291410068}) 2024-09-16T14:57:52.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.428+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=884090371}) 2024-09-16T14:57:52.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.428+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.428+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4111319317}) 2024-09-16T14:57:52.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.428+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034525406}) 2024-09-16T14:57:52.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.428+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.428+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4111319317}) 2024-09-16T14:57:52.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.428+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86163633}) 2024-09-16T14:57:52.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3972884763}) 2024-09-16T14:57:52.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3105722403}) 2024-09-16T14:57:52.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3972884763}) 2024-09-16T14:57:52.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=915873237}) 2024-09-16T14:57:52.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=717870531}) 2024-09-16T14:57:52.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1571296408}) 2024-09-16T14:57:52.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=717870531}) 2024-09-16T14:57:52.434 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4254724369}) 2024-09-16T14:57:52.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4224979950}) 2024-09-16T14:57:52.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3431450306}) 2024-09-16T14:57:52.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4224979950}) 2024-09-16T14:57:52.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.432+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=328805256}) 2024-09-16T14:57:52.436 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.436+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.436+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=765613453}) 2024-09-16T14:57:52.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.436+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1917472683}) 2024-09-16T14:57:52.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.436+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.436+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=765613453}) 2024-09-16T14:57:52.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.436+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2917540071}) 2024-09-16T14:57:52.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.436+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.436+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1464069037}) 2024-09-16T14:57:52.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.436+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3458506877}) 2024-09-16T14:57:52.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.436+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.436+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1464069037}) 2024-09-16T14:57:52.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.436+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=111471410}) 2024-09-16T14:57:52.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.440+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.440 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.440+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3428838016}) 2024-09-16T14:57:52.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.440+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358768524}) 2024-09-16T14:57:52.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.440+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.440+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3428838016}) 2024-09-16T14:57:52.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.440+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2236605284}) 2024-09-16T14:57:52.442 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.440+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.442 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.440+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1070271503}) 2024-09-16T14:57:52.442 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.440+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090965772}) 2024-09-16T14:57:52.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.440+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.440+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1070271503}) 2024-09-16T14:57:52.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.440+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3041091812}) 2024-09-16T14:57:52.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.444+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.444+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=545612286}) 2024-09-16T14:57:52.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.444+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954346087}) 2024-09-16T14:57:52.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.444+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.444 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.444+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=545612286}) 2024-09-16T14:57:52.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.444+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=527020373}) 2024-09-16T14:57:52.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.444+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.444+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1851075450}) 2024-09-16T14:57:52.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.444+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046284647}) 2024-09-16T14:57:52.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.444+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.444+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1851075450}) 2024-09-16T14:57:52.446 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.444+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1755780634}) 2024-09-16T14:57:52.448 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.448+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.448 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.448+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3898759746}) 2024-09-16T14:57:52.448 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.448+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308752028}) 2024-09-16T14:57:52.448 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.448+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.448 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.448+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3898759746}) 2024-09-16T14:57:52.448 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.448+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2198824776}) 2024-09-16T14:57:52.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.448+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.448+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3588229427}) 2024-09-16T14:57:52.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.448+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1919135692}) 2024-09-16T14:57:52.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.448+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.448+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3588229427}) 2024-09-16T14:57:52.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.448+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=327007123}) 2024-09-16T14:57:52.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.452+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.452+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=144851466}) 2024-09-16T14:57:52.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.452+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2336972698}) 2024-09-16T14:57:52.452 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.452+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.452 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.452+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=144851466}) 2024-09-16T14:57:52.452 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.452+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2615457538}) 2024-09-16T14:57:52.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.452+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.452+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3774430415}) 2024-09-16T14:57:52.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.452+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3343732092}) 2024-09-16T14:57:52.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.452+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.452+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3774430415}) 2024-09-16T14:57:52.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.452+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4155509815}) 2024-09-16T14:57:52.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2733678517}) 2024-09-16T14:57:52.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2021430446}) 2024-09-16T14:57:52.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.456 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2733678517}) 2024-09-16T14:57:52.456 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3375890021}) 2024-09-16T14:57:52.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4059342613}) 2024-09-16T14:57:52.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3894464919}) 2024-09-16T14:57:52.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4059342613}) 2024-09-16T14:57:52.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=261517188}) 2024-09-16T14:57:52.458 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4179725710}) 2024-09-16T14:57:52.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850777689}) 2024-09-16T14:57:52.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4179725710}) 2024-09-16T14:57:52.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.456+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2681370283}) 2024-09-16T14:57:52.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.460+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.460+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2657003132}) 2024-09-16T14:57:52.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.460+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249744326}) 2024-09-16T14:57:52.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.460+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.460+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2657003132}) 2024-09-16T14:57:52.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.460+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=648041958}) 2024-09-16T14:57:52.467 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.468+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.468+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=889139071}) 2024-09-16T14:57:52.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.468+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3259966236}) 2024-09-16T14:57:52.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.468+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.468+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=889139071}) 2024-09-16T14:57:52.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.468+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3253067626}) 2024-09-16T14:57:52.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.472+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.472+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=307693163}) 2024-09-16T14:57:52.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.472+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=484569376}) 2024-09-16T14:57:52.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.472+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.472+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=307693163}) 2024-09-16T14:57:52.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.472+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2539957295}) 2024-09-16T14:57:52.499 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.496+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.499 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.500+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2169912498}) 2024-09-16T14:57:52.499 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.500+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265761769}) 2024-09-16T14:57:52.499 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.500+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.499 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.500+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2169912498}) 2024-09-16T14:57:52.499 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.500+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1445704945}) 2024-09-16T14:57:52.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.500+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.500+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2955091841}) 2024-09-16T14:57:52.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.500+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225466446}) 2024-09-16T14:57:52.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.500+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.500+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2955091841}) 2024-09-16T14:57:52.501 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.500+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1072056295}) 2024-09-16T14:57:52.504 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.504+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.504 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.504+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897686624}) 2024-09-16T14:57:52.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.504+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329526043}) 2024-09-16T14:57:52.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.504+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.504+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897686624}) 2024-09-16T14:57:52.505 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.504+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3797483887}) 2024-09-16T14:57:52.506 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.504+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.504+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1619570250}) 2024-09-16T14:57:52.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.504+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1200697222}) 2024-09-16T14:57:52.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.504+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.504+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1619570250}) 2024-09-16T14:57:52.507 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.504+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4165155801}) 2024-09-16T14:57:52.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.508+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.508+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=205829597}) 2024-09-16T14:57:52.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.508+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2849014913}) 2024-09-16T14:57:52.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.508+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.508+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=205829597}) 2024-09-16T14:57:52.509 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.508+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2090912680}) 2024-09-16T14:57:52.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.508+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.510 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.508+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3375101357}) 2024-09-16T14:57:52.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.508+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=920565211}) 2024-09-16T14:57:52.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.508+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.508+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3375101357}) 2024-09-16T14:57:52.511 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.508+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3730605840}) 2024-09-16T14:57:52.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.512+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.512 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.512+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182399248}) 2024-09-16T14:57:52.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.512+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2875545599}) 2024-09-16T14:57:52.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.512+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.512+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182399248}) 2024-09-16T14:57:52.513 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.512+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2288956621}) 2024-09-16T14:57:52.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.512+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.512+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3611553185}) 2024-09-16T14:57:52.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.512+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=373622671}) 2024-09-16T14:57:52.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.512+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.512+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3611553185}) 2024-09-16T14:57:52.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.512+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3407987536}) 2024-09-16T14:57:52.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1933520425}) 2024-09-16T14:57:52.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4064001493}) 2024-09-16T14:57:52.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1933520425}) 2024-09-16T14:57:52.516 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=28209749}) 2024-09-16T14:57:52.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3402846259}) 2024-09-16T14:57:52.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217141615}) 2024-09-16T14:57:52.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3402846259}) 2024-09-16T14:57:52.517 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2905856323}) 2024-09-16T14:57:52.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1682926836}) 2024-09-16T14:57:52.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1909795716}) 2024-09-16T14:57:52.518 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1682926836}) 2024-09-16T14:57:52.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.516+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2756266044}) 2024-09-16T14:57:52.519 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.520+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.520+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3601483035}) 2024-09-16T14:57:52.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.520+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551795762}) 2024-09-16T14:57:52.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.520+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.520+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3601483035}) 2024-09-16T14:57:52.520 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.520+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651931457}) 2024-09-16T14:57:52.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.520+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.520+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2736636055}) 2024-09-16T14:57:52.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.520+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=811279443}) 2024-09-16T14:57:52.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.520+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.520+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2736636055}) 2024-09-16T14:57:52.521 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.520+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=351866183}) 2024-09-16T14:57:52.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.524+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.524+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=13,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=2567296877,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=404653546,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1008926401}) 2024-09-16T14:57:52.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.524+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=14,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=40} crc {logm=1025455746,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3413992839,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3251277988}) 2024-09-16T14:57:52.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.524+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.524 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.524+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=13,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=2567296877,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=404653546,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1008926401}) 2024-09-16T14:57:52.525 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.524+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=13,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=9,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=44} crc {logm=319774459,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=141933469,mon_config_key=3475477691,monmap=970648238,osd_metadata=2172984177,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=2780412337}) 2024-09-16T14:57:52.535 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.536+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:57:52.535 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.536+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=9} crc {osdmap=1682338693}) 2024-09-16T14:57:52.535 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:57:52.536+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=7} crc {osdmap=3831351364}) 2024-09-16T14:57:53.301 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:57:53.302 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:57:53.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 10 monclient: get_monmap_and_config 2024-09-16T14:57:53.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 10 monclient: build_initial_monmap 2024-09-16T14:57:53.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 10 monclient: monmap: 2024-09-16T14:57:53.436 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:53.436 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.436 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:53.438362+0000 2024-09-16T14:57:53.436 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:53.438362+0000 2024-09-16T14:57:53.436 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:53.437 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.437 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:53.437 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:53.437 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:53.437 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 1 Processor -- start 2024-09-16T14:57:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 1 -- start start 2024-09-16T14:57:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 10 monclient: init 2024-09-16T14:57:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:57:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 10 monclient(hunting): picked mon.noname-c con 0x7f4bdc09a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 10 monclient(hunting): picked mon.noname-a con 0x7f4bdc09d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 10 monclient(hunting): picked mon.noname-b con 0x7f4bdc09b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc09dc60 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc099ae0 con 0x7f4bdc09a550 2024-09-16T14:57:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc09b8f0 con 0x7f4bdc09b040 2024-09-16T14:57:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 10 monclient(hunting): authenticate will time out at 1897.558353s 2024-09-16T14:57:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1c48640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4bdc09d1b0 0x7f4bdc136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36996/0 (socket says 172.21.15.26:36996) 2024-09-16T14:57:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1c48640 1 -- 172.21.15.26:0/4023694023 learned_addr learned my addr 172.21.15.26:0/4023694023 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 1 -- 172.21.15.26:0/4023694023 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028616721 0 0) 0x7f4bdc09dc60 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient(hunting): my global_id is 274452 2024-09-16T14:57:53.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 1 -- 172.21.15.26:0/4023694023 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc8003530 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 1 -- 172.21.15.26:0/4023694023 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011207223 0 0) 0x7f4bdc09b8f0 con 0x7f4bdc09b040 2024-09-16T14:57:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient(hunting): my global_id is 284372 2024-09-16T14:57:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 1 -- 172.21.15.26:0/4023694023 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bdc09dc60 con 0x7f4bdc09b040 2024-09-16T14:57:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 1 -- 172.21.15.26:0/4023694023 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3062835733 0 0) 0x7f4bc8003530 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 1 -- 172.21.15.26:0/4023694023 >> v1:172.21.15.26:6790/0 conn(0x7f4bdc09b040 legacy=0x7f4bdc13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 1 -- 172.21.15.26:0/4023694023 >> v1:172.21.15.189:6789/0 conn(0x7f4bdc09a550 legacy=0x7f4bdc09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient: _finish_hunting 0 2024-09-16T14:57:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 1 monclient: found mon.noname-a 2024-09-16T14:57:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:53.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:57:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 1 -- 172.21.15.26:0/4023694023 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc09bbd0 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient: _finish_auth 0 2024-09-16T14:57:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient: _check_auth_tickets 2024-09-16T14:57:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 1 -- 172.21.15.26:0/4023694023 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4bcc001620 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient: dump: 2024-09-16T14:57:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.451 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.452 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 5 monclient: authenticate success, global_id 274452 2024-09-16T14:57:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 1 -- 172.21.15.26:0/4023694023 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bcc001900 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 1 -- 172.21.15.26:0/4023694023 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4bcc001cd0 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be1447640 10 monclient: dump: 2024-09-16T14:57:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.454 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.436+0000 7f4be46ad640 4 set_mon_vals no callback set 2024-09-16T14:57:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient: shutdown 2024-09-16T14:57:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 1 -- 172.21.15.26:0/4023694023 >> v1:172.21.15.26:6789/0 conn(0x7f4bdc09d1b0 legacy=0x7f4bdc136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 1 -- 172.21.15.26:0/4023694023 shutdown_connections 2024-09-16T14:57:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 1 -- 172.21.15.26:0/4023694023 >> 172.21.15.26:0/4023694023 conn(0x7f4bdc09fb50 msgr2=0x7f4bdc09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 1 -- 172.21.15.26:0/4023694023 shutdown_connections 2024-09-16T14:57:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 1 -- 172.21.15.26:0/4023694023 wait complete. 2024-09-16T14:57:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient: build_initial_monmap 2024-09-16T14:57:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient: monmap: 2024-09-16T14:57:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:53.456 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 1 Processor -- start 2024-09-16T14:57:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 1 -- start start 2024-09-16T14:57:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient: init 2024-09-16T14:57:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:57:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient(hunting): picked mon.noname-a con 0x7f4bdc09a550 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient(hunting): picked mon.noname-b con 0x7f4bdc09b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:53.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient(hunting): picked mon.noname-c con 0x7f4bdc09d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:53.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc1364f0 con 0x7f4bdc09a550 2024-09-16T14:57:53.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc0a23a0 con 0x7f4bdc09b040 2024-09-16T14:57:53.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc12f680 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:53.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient(hunting): authenticate will time out at 1897.561704s 2024-09-16T14:57:53.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be2449640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4bdc09a550 0x7f4bdc12e860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37006/0 (socket says 172.21.15.26:37006) 2024-09-16T14:57:53.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be2449640 1 -- 172.21.15.26:0/802959568 learned_addr learned my addr 172.21.15.26:0/802959568 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:53.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983509680 0 0) 0x7f4bdc12f680 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): my global_id is 284375 2024-09-16T14:57:53.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc0003530 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910298936 0 0) 0x7f4bdc1364f0 con 0x7f4bdc09a550 2024-09-16T14:57:53.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): my global_id is 274455 2024-09-16T14:57:53.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bdc12f680 con 0x7f4bdc09a550 2024-09-16T14:57:53.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 961691974 0 0) 0x7f4bc0003530 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bdc1364f0 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4bd4001c40 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:53.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): dump: 2024-09-16T14:57:53.459 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.459 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.459 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.459 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.459 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.459 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.459 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.460 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.460 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.460 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:53.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 14686131 0 0) 0x7f4bdc12f680 con 0x7f4bdc09a550 2024-09-16T14:57:53.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bc0003530 con 0x7f4bdc09a550 2024-09-16T14:57:53.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4bd8001920 con 0x7f4bdc09a550 2024-09-16T14:57:53.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:53.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): dump: 2024-09-16T14:57:53.460 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.460 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.460 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.461 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687603371 0 0) 0x7f4bdc0a23a0 con 0x7f4bdc09b040 2024-09-16T14:57:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient(hunting): my global_id is 304126 2024-09-16T14:57:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bdc12f680 con 0x7f4bdc09b040 2024-09-16T14:57:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 266142208 0 0) 0x7f4bdc1364f0 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 >> v1:172.21.15.189:6789/0 conn(0x7f4bdc09b040 legacy=0x7f4bdc12ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 >> v1:172.21.15.26:6789/0 conn(0x7f4bdc09a550 legacy=0x7f4bdc12e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:57:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 monclient: found mon.c 2024-09-16T14:57:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc135af0 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:57:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:57:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 5 monclient: authenticate success, global_id 284375 2024-09-16T14:57:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient: _renew_subs 2024-09-16T14:57:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 1 -- 172.21.15.26:0/802959568 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bdc09a2a0 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient: _renew_subs 2024-09-16T14:57:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 1 -- 172.21.15.26:0/802959568 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bdc136850 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bd40039a0 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4bd4003cb0 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be0c46640 4 set_mon_vals no callback set 2024-09-16T14:57:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 10 monclient: dump: 2024-09-16T14:57:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.466 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4be46ad640 1 -- 172.21.15.26:0/802959568 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc4005050 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4bd400e720 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.440+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f4bd4042d50 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.444+0000 7f4be2449640 10 monclient: get_auth_request con 0x7f4bc0036490 auth_method 0 2024-09-16T14:57:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.448+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4bd4013530 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.448+0000 7f4bd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.448+0000 7f4bd2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:53.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.668+0000 7f4be46ad640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:57:53.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.668+0000 7f4be46ad640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:53.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.668+0000 7f4be46ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:53.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.668+0000 7f4be46ad640 1 -- 172.21.15.26:0/802959568 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4bc4005880 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.668+0000 7f4bd2ffd640 1 -- 172.21.15.26:0/802959568 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2015627999) 0x7f4bd4011bd0 con 0x7f4bdc09d1b0 2024-09-16T14:57:53.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.668+0000 7f4bd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:53.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.668+0000 7f4bd2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:53.670 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:57:53.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.672+0000 7f4be46ad640 1 -- 172.21.15.26:0/802959568 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4bc0036490 legacy=0x7f4bc0038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.672+0000 7f4be46ad640 10 monclient: shutdown 2024-09-16T14:57:53.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.672+0000 7f4be46ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:53.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.672+0000 7f4be46ad640 1 -- 172.21.15.26:0/802959568 >> v1:172.21.15.26:6790/0 conn(0x7f4bdc09d1b0 legacy=0x7f4bdc1cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.672+0000 7f4be46ad640 1 -- 172.21.15.26:0/802959568 shutdown_connections 2024-09-16T14:57:53.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.672+0000 7f4be46ad640 1 -- 172.21.15.26:0/802959568 >> 172.21.15.26:0/802959568 conn(0x7f4bdc09fb50 msgr2=0x7f4bdc132a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:53.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.672+0000 7f4be46ad640 1 -- 172.21.15.26:0/802959568 shutdown_connections 2024-09-16T14:57:53.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.672+0000 7f4be46ad640 1 -- 172.21.15.26:0/802959568 wait complete. 2024-09-16T14:57:53.688 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:57:53.688 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:57:53.689 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:57:53.689 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:57:53.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.804+0000 7fdbd5109640 10 monclient: get_monmap_and_config 2024-09-16T14:57:53.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.804+0000 7fdbd5109640 10 monclient: build_initial_monmap 2024-09-16T14:57:53.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.804+0000 7fdbd5109640 10 monclient: monmap: 2024-09-16T14:57:53.806 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:53.806 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.806 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:53.808760+0000 2024-09-16T14:57:53.807 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:53.808760+0000 2024-09-16T14:57:53.807 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:53.807 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.807 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:53.807 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:53.807 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:53.807 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.804+0000 7fdbd5109640 1 Processor -- start 2024-09-16T14:57:53.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 1 -- start start 2024-09-16T14:57:53.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient: init 2024-09-16T14:57:53.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:53.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:57:53.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient(hunting): picked mon.noname-b con 0x7fdbd006c5a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:53.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient(hunting): picked mon.noname-a con 0x7fdbd00a23b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:53.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient(hunting): picked mon.noname-c con 0x7fdbd00a27c0 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:53.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd006cee0 con 0x7fdbd00a23b0 2024-09-16T14:57:53.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd006c980 con 0x7fdbd00a27c0 2024-09-16T14:57:53.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd00a15e0 con 0x7fdbd006c5a0 2024-09-16T14:57:53.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:53.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient(hunting): authenticate will time out at 1897.928770s 2024-09-16T14:57:53.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbce59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdbd00a23b0 0x7fdbd0139eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37022/0 (socket says 172.21.15.26:37022) 2024-09-16T14:57:53.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbced9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fdbd006c5a0 0x7fdbd00a1ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53612/0 (socket says 172.21.15.26:53612) 2024-09-16T14:57:53.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbce59c640 1 -- 172.21.15.26:0/3998488000 learned_addr learned my addr 172.21.15.26:0/3998488000 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:53.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 1 -- 172.21.15.26:0/3998488000 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842706524 0 0) 0x7fdbd006cee0 con 0x7fdbd00a23b0 2024-09-16T14:57:53.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient(hunting): my global_id is 274458 2024-09-16T14:57:53.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 1 -- 172.21.15.26:0/3998488000 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbb8003530 con 0x7fdbd00a23b0 2024-09-16T14:57:53.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 1 -- 172.21.15.26:0/3998488000 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111225710 0 0) 0x7fdbd00a15e0 con 0x7fdbd006c5a0 2024-09-16T14:57:53.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient(hunting): my global_id is 284378 2024-09-16T14:57:53.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 1 -- 172.21.15.26:0/3998488000 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd006cee0 con 0x7fdbd006c5a0 2024-09-16T14:57:53.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 1 -- 172.21.15.26:0/3998488000 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2128211288 0 0) 0x7fdbb8003530 con 0x7fdbd00a23b0 2024-09-16T14:57:53.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 1 -- 172.21.15.26:0/3998488000 >> v1:172.21.15.26:6790/0 conn(0x7fdbd006c5a0 legacy=0x7fdbd00a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 1 -- 172.21.15.26:0/3998488000 >> v1:172.21.15.189:6789/0 conn(0x7fdbd00a27c0 legacy=0x7fdbd013d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient: _finish_hunting 0 2024-09-16T14:57:53.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 1 monclient: found mon.noname-a 2024-09-16T14:57:53.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:53.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:57:53.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 1 -- 172.21.15.26:0/3998488000 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd009edd0 con 0x7fdbd00a23b0 2024-09-16T14:57:53.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient: _finish_auth 0 2024-09-16T14:57:53.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient: _check_auth_tickets 2024-09-16T14:57:53.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:53.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 1 -- 172.21.15.26:0/3998488000 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdbc4001620 con 0x7fdbd00a23b0 2024-09-16T14:57:53.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:53.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient: dump: 2024-09-16T14:57:53.818 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.818 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.818 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.818 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.818 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.818 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.818 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.818 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.818 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.818 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 5 monclient: authenticate success, global_id 274458 2024-09-16T14:57:53.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:53.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 1 -- 172.21.15.26:0/3998488000 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbc40018c0 con 0x7fdbd00a23b0 2024-09-16T14:57:53.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:53.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 1 -- 172.21.15.26:0/3998488000 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdbc4001c50 con 0x7fdbd00a23b0 2024-09-16T14:57:53.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:53.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbcdd9b640 10 monclient: dump: 2024-09-16T14:57:53.819 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.819 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.819 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.819 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.819 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.819 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.820 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.820 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.820 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.820 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:53.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 4 set_mon_vals no callback set 2024-09-16T14:57:53.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient: shutdown 2024-09-16T14:57:53.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:53.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 1 -- 172.21.15.26:0/3998488000 >> v1:172.21.15.26:6789/0 conn(0x7fdbd00a23b0 legacy=0x7fdbd0139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 1 -- 172.21.15.26:0/3998488000 shutdown_connections 2024-09-16T14:57:53.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 1 -- 172.21.15.26:0/3998488000 >> 172.21.15.26:0/3998488000 conn(0x7fdbd00a3da0 msgr2=0x7fdbd00a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:53.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 1 -- 172.21.15.26:0/3998488000 shutdown_connections 2024-09-16T14:57:53.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 1 -- 172.21.15.26:0/3998488000 wait complete. 2024-09-16T14:57:53.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient: build_initial_monmap 2024-09-16T14:57:53.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient: monmap: 2024-09-16T14:57:53.821 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:53.821 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:53.821 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:53.821 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:53.821 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:53.821 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:53.821 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:53.821 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:53.821 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:53.822 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 1 Processor -- start 2024-09-16T14:57:53.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 1 -- start start 2024-09-16T14:57:53.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient: init 2024-09-16T14:57:53.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:53.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:57:53.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient(hunting): picked mon.noname-a con 0x7fdbd006c5a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:53.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient(hunting): picked mon.noname-b con 0x7fdbd00a23b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:53.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient(hunting): picked mon.noname-c con 0x7fdbd00a27c0 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:53.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd00a64d0 con 0x7fdbd006c5a0 2024-09-16T14:57:53.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd009e490 con 0x7fdbd00a23b0 2024-09-16T14:57:53.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd0133710 con 0x7fdbd00a27c0 2024-09-16T14:57:53.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:53.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.808+0000 7fdbd5109640 10 monclient(hunting): authenticate will time out at 1897.931932s 2024-09-16T14:57:53.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbced9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdbd006c5a0 0x7fdbd01343d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37032/0 (socket says 172.21.15.26:37032) 2024-09-16T14:57:53.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbced9d640 1 -- 172.21.15.26:0/3099439235 learned_addr learned my addr 172.21.15.26:0/3099439235 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:53.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010480743 0 0) 0x7fdbd00a64d0 con 0x7fdbd006c5a0 2024-09-16T14:57:53.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): my global_id is 274461 2024-09-16T14:57:53.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbbc003530 con 0x7fdbd006c5a0 2024-09-16T14:57:53.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428963248 0 0) 0x7fdbd0133710 con 0x7fdbd00a27c0 2024-09-16T14:57:53.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): my global_id is 284381 2024-09-16T14:57:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd00a64d0 con 0x7fdbd00a27c0 2024-09-16T14:57:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3387222754 0 0) 0x7fdbbc003530 con 0x7fdbd006c5a0 2024-09-16T14:57:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbd0133710 con 0x7fdbd006c5a0 2024-09-16T14:57:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdbc0001b70 con 0x7fdbd006c5a0 2024-09-16T14:57:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): dump: 2024-09-16T14:57:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.824 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.825 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.825 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.825 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.825 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:53.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454619309 0 0) 0x7fdbd009e490 con 0x7fdbd00a23b0 2024-09-16T14:57:53.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): my global_id is 304132 2024-09-16T14:57:53.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbbc003530 con 0x7fdbd00a23b0 2024-09-16T14:57:53.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3536516711 0 0) 0x7fdbd00a64d0 con 0x7fdbd00a27c0 2024-09-16T14:57:53.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbd009e490 con 0x7fdbd00a27c0 2024-09-16T14:57:53.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdbc8001800 con 0x7fdbd00a27c0 2024-09-16T14:57:53.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:53.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): dump: 2024-09-16T14:57:53.826 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.826 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.826 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.826 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.826 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.826 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.826 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.826 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.826 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.827 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2003366387 0 0) 0x7fdbd0133710 con 0x7fdbd006c5a0 2024-09-16T14:57:53.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 >> v1:172.21.15.26:6790/0 conn(0x7fdbd00a27c0 legacy=0x7fdbd01d14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 >> v1:172.21.15.189:6789/0 conn(0x7fdbd00a23b0 legacy=0x7fdbd01cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient: _finish_hunting 0 2024-09-16T14:57:53.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 monclient: found mon.a 2024-09-16T14:57:53.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:53.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:53.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd0131c00 con 0x7fdbd006c5a0 2024-09-16T14:57:53.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient: _finish_auth 0 2024-09-16T14:57:53.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient: _check_auth_tickets 2024-09-16T14:57:53.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:53.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbd5109640 5 monclient: authenticate success, global_id 274461 2024-09-16T14:57:53.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbd5109640 10 monclient: _renew_subs 2024-09-16T14:57:53.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbd5109640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:53.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbd5109640 1 -- 172.21.15.26:0/3099439235 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd0134ae0 con 0x7fdbd006c5a0 2024-09-16T14:57:53.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbd5109640 10 monclient: _renew_subs 2024-09-16T14:57:53.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbd5109640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:53.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbd5109640 1 -- 172.21.15.26:0/3099439235 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd01d2db0 con 0x7fdbd006c5a0 2024-09-16T14:57:53.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbc00020e0 con 0x7fdbd006c5a0 2024-09-16T14:57:53.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:53.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdbc0004d10 con 0x7fdbd006c5a0 2024-09-16T14:57:53.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:53.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient: dump: 2024-09-16T14:57:53.829 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.829 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.829 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.829 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.829 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.829 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.829 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.829 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.829 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.829 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbcd59a640 4 set_mon_vals no callback set 2024-09-16T14:57:53.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbd5109640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:53.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbd5109640 1 -- 172.21.15.26:0/3099439235 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdbb0006590 con 0x7fdbb0002830 2024-09-16T14:57:53.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fdbc000f780 con 0x7fdbd006c5a0 2024-09-16T14:57:53.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fdbc0043f80 con 0x7fdbd006c5a0 2024-09-16T14:57:53.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114855752 0 0) 0x7fdbb0006590 con 0x7fdbb0002830 2024-09-16T14:57:53.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient(hunting): my global_id is 274464 2024-09-16T14:57:53.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd0133710 con 0x7fdbb0002830 2024-09-16T14:57:53.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1748181055 0 0) 0x7fdbd0133710 con 0x7fdbb0002830 2024-09-16T14:57:53.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbb0006590 con 0x7fdbb0002830 2024-09-16T14:57:53.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdbc4001190 con 0x7fdbb0002830 2024-09-16T14:57:53.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbcf59e640 10 monclient: get_auth_request con 0x7fdbbc036490 auth_method 0 2024-09-16T14:57:53.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:53.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient: dump: 2024-09-16T14:57:53.831 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.831 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.831 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.831 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.831 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.831 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.831 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.831 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.831 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.831 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 365350339 0 0) 0x7fdbb0006590 con 0x7fdbb0002830 2024-09-16T14:57:53.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdbb000a0e0 con 0x7fdbb0002830 2024-09-16T14:57:53.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fdbc4001b50 con 0x7fdbb0002830 2024-09-16T14:57:53.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:53.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:53.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 >> v1:172.21.15.26:6789/0 conn(0x7fdbb0002830 legacy=0x7fdbb0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.812+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 >> v1:172.21.15.26:6789/0 conn(0x7fdbb0002830 legacy=0x7fdbb0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:53.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbd5109640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:57:53.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbd5109640 1 -- 172.21.15.26:0/3099439235 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdbb0009d80 con 0x7fdbb00090a0 2024-09-16T14:57:53.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773645429 0 0) 0x7fdbb0009d80 con 0x7fdbb00090a0 2024-09-16T14:57:53.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 10 monclient(hunting): my global_id is 274467 2024-09-16T14:57:53.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbb0006590 con 0x7fdbb00090a0 2024-09-16T14:57:53.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1616708009 0 0) 0x7fdbb0006590 con 0x7fdbb00090a0 2024-09-16T14:57:53.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbb0009d80 con 0x7fdbb00090a0 2024-09-16T14:57:53.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdbc4004ce0 con 0x7fdbb00090a0 2024-09-16T14:57:53.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:53.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 10 monclient: dump: 2024-09-16T14:57:53.835 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.835 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.835 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.835 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.835 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.835 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.835 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.835 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.835 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.835 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 588941901 0 0) 0x7fdbb0009d80 con 0x7fdbb00090a0 2024-09-16T14:57:53.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdbb000c290 con 0x7fdbb00090a0 2024-09-16T14:57:53.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 3478143798) 0x7fdbc4004ef0 con 0x7fdbb00090a0 2024-09-16T14:57:53.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:57:53.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:53.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 >> v1:172.21.15.26:6789/0 conn(0x7fdbb00090a0 legacy=0x7fdbb0009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbb77fe640 1 -- 172.21.15.26:0/3099439235 >> v1:172.21.15.26:6789/0 conn(0x7fdbb00090a0 legacy=0x7fdbb0009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:53.836 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:57:53.836 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:53.836 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:53.836 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:57:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 144, 2024-09-16T14:57:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:57:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:57:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:57:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:57:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:57:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:57:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:57:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:53.838 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:53.838 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:53.838 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:53.838 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:53.838 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:53.838 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:53.838 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:53.838 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:53.838 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:57:53.838 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:57:53.838 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:53.838 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:57:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:57:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:57:53.839 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:57:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:57:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:57:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:57:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:57:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:57:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:57:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:57:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:57:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:57:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:53.840 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:57:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:57:53.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:57:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:57:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:53.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:57:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:57:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:53.843 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:57:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:57:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:57:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:53.844 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:57:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:57:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:57:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:53.845 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:57:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:57:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:53.846 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:57:53.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:53.847 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:53.847 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:53.847 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:57:53.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:53.847 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:53.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:57:53.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:53.847 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:53.847 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:53.847 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:57:53.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:53.848 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:53.848 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:57:53.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:53.848 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:53.848 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:53.848 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:53.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:53.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:53.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:53.848 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:57:53.848 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:57:53.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbd5109640 1 -- 172.21.15.26:0/3099439235 >> v1:172.21.15.189:6812/3196161582 conn(0x7fdbbc036490 legacy=0x7fdbbc038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbd5109640 10 monclient: shutdown 2024-09-16T14:57:53.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbd5109640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:53.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbd5109640 1 -- 172.21.15.26:0/3099439235 >> v1:172.21.15.26:6789/0 conn(0x7fdbd006c5a0 legacy=0x7fdbd01343d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbcf59e640 1 -- 172.21.15.26:0/3099439235 reap_dead start 2024-09-16T14:57:53.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbcf59e640 1 -- 172.21.15.26:0/3099439235 reap_dead start 2024-09-16T14:57:53.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbd5109640 1 -- 172.21.15.26:0/3099439235 shutdown_connections 2024-09-16T14:57:53.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbd5109640 1 -- 172.21.15.26:0/3099439235 >> 172.21.15.26:0/3099439235 conn(0x7fdbd00a3da0 msgr2=0x7fdbd00a51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:53.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbd5109640 1 -- 172.21.15.26:0/3099439235 shutdown_connections 2024-09-16T14:57:53.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.820+0000 7fdbd5109640 1 -- 172.21.15.26:0/3099439235 wait complete. 2024-09-16T14:57:53.851 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:57:53.851 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:57:53.851 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:57:53.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 10 monclient: get_monmap_and_config 2024-09-16T14:57:53.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 10 monclient: build_initial_monmap 2024-09-16T14:57:53.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 10 monclient: monmap: 2024-09-16T14:57:53.968 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:53.968 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.968 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:53.969678+0000 2024-09-16T14:57:53.969 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:53.969678+0000 2024-09-16T14:57:53.969 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:53.969 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.969 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:53.969 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:53.969 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:53.969 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 1 Processor -- start 2024-09-16T14:57:53.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 1 -- start start 2024-09-16T14:57:53.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 10 monclient: init 2024-09-16T14:57:53.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:53.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:57:53.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 10 monclient(hunting): picked mon.noname-a con 0x7f1edc0a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:53.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 10 monclient(hunting): picked mon.noname-b con 0x7f1edc0a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:53.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 10 monclient(hunting): picked mon.noname-c con 0x7f1edc0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:53.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc0a3250 con 0x7f1edc0a1d10 2024-09-16T14:57:53.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc09f600 con 0x7f1edc0a2800 2024-09-16T14:57:53.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc0a3830 con 0x7f1edc0a0000 2024-09-16T14:57:53.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:53.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 10 monclient(hunting): authenticate will time out at 1898.089768s 2024-09-16T14:57:53.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1eda59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1edc0a1d10 0x7f1edc0a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37068/0 (socket says 172.21.15.26:37068) 2024-09-16T14:57:53.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1eda59c640 1 -- 172.21.15.26:0/1446727664 learned_addr learned my addr 172.21.15.26:0/1446727664 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:53.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 1 -- 172.21.15.26:0/1446727664 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909866721 0 0) 0x7f1edc0a3830 con 0x7f1edc0a0000 2024-09-16T14:57:53.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient(hunting): my global_id is 284384 2024-09-16T14:57:53.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 1 -- 172.21.15.26:0/1446727664 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ec8003530 con 0x7f1edc0a0000 2024-09-16T14:57:53.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 1 -- 172.21.15.26:0/1446727664 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062797474 0 0) 0x7f1edc0a3250 con 0x7f1edc0a1d10 2024-09-16T14:57:53.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient(hunting): my global_id is 274470 2024-09-16T14:57:53.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 1 -- 172.21.15.26:0/1446727664 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1edc0a3830 con 0x7f1edc0a1d10 2024-09-16T14:57:53.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 1 -- 172.21.15.26:0/1446727664 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2209077357 0 0) 0x7f1ec8003530 con 0x7f1edc0a0000 2024-09-16T14:57:53.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 1 -- 172.21.15.26:0/1446727664 >> v1:172.21.15.189:6789/0 conn(0x7f1edc0a2800 legacy=0x7f1edc13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 1 -- 172.21.15.26:0/1446727664 >> v1:172.21.15.26:6789/0 conn(0x7f1edc0a1d10 legacy=0x7f1edc0a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient: _finish_hunting 0 2024-09-16T14:57:53.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 1 monclient: found mon.noname-b 2024-09-16T14:57:53.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:53.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:57:53.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 1 -- 172.21.15.26:0/1446727664 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1edc09ed60 con 0x7f1edc0a0000 2024-09-16T14:57:53.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient: _finish_auth 0 2024-09-16T14:57:53.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient: _check_auth_tickets 2024-09-16T14:57:53.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:53.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 1 -- 172.21.15.26:0/1446727664 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1ec4001550 con 0x7f1edc0a0000 2024-09-16T14:57:53.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 5 monclient: authenticate success, global_id 284384 2024-09-16T14:57:53.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:53.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:57:53.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient: dump: 2024-09-16T14:57:53.982 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.982 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.982 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.982 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.982 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.982 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.982 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.982 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.982 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.983 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:53.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 1 -- 172.21.15.26:0/1446727664 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ec40017b0 con 0x7f1edc0a0000 2024-09-16T14:57:53.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:53.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:53.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 4 set_mon_vals no callback set 2024-09-16T14:57:53.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 1 -- 172.21.15.26:0/1446727664 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1ec4001b80 con 0x7f1edc0a0000 2024-09-16T14:57:53.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:53.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ed959a640 10 monclient: dump: 2024-09-16T14:57:53.983 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.983 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.983 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.983 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.984 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.984 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.984 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.984 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.984 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.984 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 10 monclient: shutdown 2024-09-16T14:57:53.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:53.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 1 -- 172.21.15.26:0/1446727664 >> v1:172.21.15.26:6790/0 conn(0x7f1edc0a0000 legacy=0x7f1edc139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 1 -- 172.21.15.26:0/1446727664 shutdown_connections 2024-09-16T14:57:53.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 1 -- 172.21.15.26:0/1446727664 >> 172.21.15.26:0/1446727664 conn(0x7f1edc0a7f30 msgr2=0x7f1edc0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:53.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 1 -- 172.21.15.26:0/1446727664 shutdown_connections 2024-09-16T14:57:53.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.968+0000 7f1ee0b43640 1 -- 172.21.15.26:0/1446727664 wait complete. 2024-09-16T14:57:53.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 10 monclient: build_initial_monmap 2024-09-16T14:57:53.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 10 monclient: monmap: 2024-09-16T14:57:53.985 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:53.985 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:53.985 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:53.985 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:53.985 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:53.985 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:53.985 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:53.985 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:53.985 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:53.985 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 1 Processor -- start 2024-09-16T14:57:53.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 1 -- start start 2024-09-16T14:57:53.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 10 monclient: init 2024-09-16T14:57:53.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:53.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:57:53.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 10 monclient(hunting): picked mon.noname-b con 0x7f1edc0a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:53.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 10 monclient(hunting): picked mon.noname-a con 0x7f1edc0a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:53.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 10 monclient(hunting): picked mon.noname-c con 0x7f1edc0a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:53.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc13cd10 con 0x7f1edc0a1d10 2024-09-16T14:57:53.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc132cc0 con 0x7f1edc0a0000 2024-09-16T14:57:53.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc0a62c0 con 0x7f1edc0a2800 2024-09-16T14:57:53.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:53.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 10 monclient(hunting): authenticate will time out at 1898.093065s 2024-09-16T14:57:53.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1edad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1edc0a2800 0x7f1edc1d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53636/0 (socket says 172.21.15.26:53636) 2024-09-16T14:57:53.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1edad9d640 1 -- 172.21.15.26:0/898446623 learned_addr learned my addr 172.21.15.26:0/898446623 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:53.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443995151 0 0) 0x7f1edc0a62c0 con 0x7f1edc0a2800 2024-09-16T14:57:53.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): my global_id is 284387 2024-09-16T14:57:53.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ebc003530 con 0x7f1edc0a2800 2024-09-16T14:57:53.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952191060 0 0) 0x7f1edc13cd10 con 0x7f1edc0a1d10 2024-09-16T14:57:53.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): my global_id is 274473 2024-09-16T14:57:53.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1edc0a62c0 con 0x7f1edc0a1d10 2024-09-16T14:57:53.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2056820261 0 0) 0x7f1edc0a62c0 con 0x7f1edc0a1d10 2024-09-16T14:57:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1edc13cd10 con 0x7f1edc0a1d10 2024-09-16T14:57:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 342433436 0 0) 0x7f1ebc003530 con 0x7f1edc0a2800 2024-09-16T14:57:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1edc0a62c0 con 0x7f1edc0a2800 2024-09-16T14:57:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1ec4001ac0 con 0x7f1edc0a1d10 2024-09-16T14:57:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:57:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): dump: 2024-09-16T14:57:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.988 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.989 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.989 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.989 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:53.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1ecc0018c0 con 0x7f1edc0a2800 2024-09-16T14:57:53.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:53.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): dump: 2024-09-16T14:57:53.990 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.990 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.990 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.990 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.990 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.990 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.990 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.990 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.990 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.990 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107938287 0 0) 0x7f1edc132cc0 con 0x7f1edc0a0000 2024-09-16T14:57:53.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): my global_id is 304138 2024-09-16T14:57:53.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ebc003530 con 0x7f1edc0a0000 2024-09-16T14:57:53.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 855030798 0 0) 0x7f1edc13cd10 con 0x7f1edc0a1d10 2024-09-16T14:57:53.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 >> v1:172.21.15.26:6790/0 conn(0x7f1edc0a2800 legacy=0x7f1edc1d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 >> v1:172.21.15.189:6789/0 conn(0x7f1edc0a0000 legacy=0x7f1edc131ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:57:53.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 monclient: found mon.a 2024-09-16T14:57:53.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:53.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:53.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1edc09e500 con 0x7f1edc0a1d10 2024-09-16T14:57:53.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:57:53.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:57:53.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:53.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 5 monclient: authenticate success, global_id 274473 2024-09-16T14:57:53.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 10 monclient: _renew_subs 2024-09-16T14:57:53.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:53.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 1 -- 172.21.15.26:0/898446623 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1edc0a6810 con 0x7f1edc0a1d10 2024-09-16T14:57:53.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 10 monclient: _renew_subs 2024-09-16T14:57:53.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:57:53.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 1 -- 172.21.15.26:0/898446623 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1edc1343f0 con 0x7f1edc0a1d10 2024-09-16T14:57:53.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ec4003920 con 0x7f1edc0a1d10 2024-09-16T14:57:53.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:53.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1ec4003cb0 con 0x7f1edc0a1d10 2024-09-16T14:57:53.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:53.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed8d99640 4 set_mon_vals no callback set 2024-09-16T14:57:53.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient: dump: 2024-09-16T14:57:53.993 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.993 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.993 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.993 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.993 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.993 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.993 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.993 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.993 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.993 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:53.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ee0b43640 1 -- 172.21.15.26:0/898446623 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1ec0006590 con 0x7f1ec0002830 2024-09-16T14:57:53.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1ec4003ed0 con 0x7f1edc0a1d10 2024-09-16T14:57:53.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f1ec4042c70 con 0x7f1edc0a1d10 2024-09-16T14:57:53.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085520738 0 0) 0x7f1ec0006590 con 0x7f1ec0002830 2024-09-16T14:57:53.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient(hunting): my global_id is 284390 2024-09-16T14:57:53.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1edc13cd10 con 0x7f1ec0002830 2024-09-16T14:57:53.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1563955796 0 0) 0x7f1edc13cd10 con 0x7f1ec0002830 2024-09-16T14:57:53.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ec0006590 con 0x7f1ec0002830 2024-09-16T14:57:53.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1ed4001920 con 0x7f1ec0002830 2024-09-16T14:57:53.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1edad9d640 10 monclient: get_auth_request con 0x7f1ebc0363c0 auth_method 0 2024-09-16T14:57:53.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:53.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 10 monclient: dump: 2024-09-16T14:57:53.995 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.995 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.995 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.995 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:53.995 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:53.995 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:53.995 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:53.995 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:53.995 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:53.995 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:53.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2461373194 0 0) 0x7f1ec0006590 con 0x7f1ec0002830 2024-09-16T14:57:53.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.972+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ec000a0e0 con 0x7f1ec0002830 2024-09-16T14:57:53.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.976+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1ed4001e60 con 0x7f1ec0002830 2024-09-16T14:57:53.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.976+0000 7f1ed2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:53.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.976+0000 7f1ed2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:53.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.976+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 >> v1:172.21.15.26:6790/0 conn(0x7f1ec0002830 legacy=0x7f1ec0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:53.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.976+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 >> v1:172.21.15.26:6790/0 conn(0x7f1ec0002830 legacy=0x7f1ec0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:53.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ee0b43640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:57:53.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ee0b43640 1 -- 172.21.15.26:0/898446623 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1ec0009da0 con 0x7f1ec00090c0 2024-09-16T14:57:53.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546206167 0 0) 0x7f1ec0009da0 con 0x7f1ec00090c0 2024-09-16T14:57:53.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:53.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:53.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:53.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 10 monclient(hunting): my global_id is 284393 2024-09-16T14:57:53.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ec0006590 con 0x7f1ec00090c0 2024-09-16T14:57:53.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3133050389 0 0) 0x7f1ec0006590 con 0x7f1ec00090c0 2024-09-16T14:57:53.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ec0009da0 con 0x7f1ec00090c0 2024-09-16T14:57:53.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1ed4004d10 con 0x7f1ec00090c0 2024-09-16T14:57:53.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:53.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:53.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 10 monclient: dump: 2024-09-16T14:57:53.999 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:53.999 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:53.999 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.000 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.000 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:54.000 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:54.000 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:54.000 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:54.000 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:54.000 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:54.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3192682039 0 0) 0x7f1ec0009da0 con 0x7f1ec00090c0 2024-09-16T14:57:54.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1ec000c290 con 0x7f1ec00090c0 2024-09-16T14:57:54.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 2999637059) 0x7f1ed4004ff0 con 0x7f1ec00090c0 2024-09-16T14:57:54.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:57:54.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:54.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 >> v1:172.21.15.26:6790/0 conn(0x7f1ec00090c0 legacy=0x7f1ec0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ed2ffd640 1 -- 172.21.15.26:0/898446623 >> v1:172.21.15.26:6790/0 conn(0x7f1ec00090c0 legacy=0x7f1ec0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:54.001 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:57:54.001 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:54.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:54.001 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:57:54.001 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 144, 2024-09-16T14:57:54.001 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:57:54.002 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:57:54.002 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:57:54.002 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:57:54.002 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:54.002 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:57:54.002 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:54.002 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:57:54.002 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:57:54.002 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:54.002 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:54.002 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:54.002 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:54.003 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:54.003 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:54.003 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:54.003 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:54.003 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:54.003 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:54.003 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:54.003 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:57:54.003 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:57:54.003 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:54.003 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:54.003 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:54.003 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:54.004 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:54.004 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:54.004 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:54.004 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:54.004 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:54.004 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:54.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:54.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.004 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:57:54.004 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:57:54.004 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:57:54.004 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:57:54.004 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:57:54.004 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:57:54.005 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:54.005 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:54.005 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:57:54.005 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:57:54.005 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:57:54.005 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:57:54.005 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:57:54.005 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:57:54.005 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:57:54.005 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:54.005 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:57:54.006 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:54.006 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:54.006 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:54.006 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:54.006 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:54.006 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:54.006 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:54.006 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:54.006 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:54.006 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:54.006 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:54.006 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:57:54.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.007 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:57:54.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:54.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:54.007 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:54.007 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:54.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:54.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:54.007 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:54.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:57:54.007 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:54.007 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:54.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:54.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:57:54.008 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:57:54.008 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:54.008 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:54.008 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:54.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:54.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:54.008 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:54.008 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:54.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:54.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:54.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:54.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:57:54.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:54.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:54.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:54.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:57:54.010 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:57:54.010 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:54.010 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:54.010 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:54.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:54.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:54.010 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:54.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:54.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:54.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:54.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:54.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:57:54.011 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:54.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:54.011 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:54.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:57:54.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:57:54.012 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:54.012 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:54.012 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:54.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:54.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:54.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.013 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:57:54.013 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:57:54.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:54.013 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:54.013 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:54.013 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:54.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:54.013 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:54.013 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:57:54.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:54.014 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:54.014 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:54.014 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:57:54.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:54.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:54.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.014 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:57:54.015 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:57:54.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ee0b43640 1 -- 172.21.15.26:0/898446623 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1ebc0363c0 legacy=0x7f1ebc038880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ee0b43640 10 monclient: shutdown 2024-09-16T14:57:54.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ee0b43640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:54.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ee0b43640 1 -- 172.21.15.26:0/898446623 >> v1:172.21.15.26:6789/0 conn(0x7f1edc0a1d10 legacy=0x7f1edc1325b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1edad9d640 1 -- 172.21.15.26:0/898446623 reap_dead start 2024-09-16T14:57:54.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1edad9d640 1 -- 172.21.15.26:0/898446623 reap_dead start 2024-09-16T14:57:54.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ee0b43640 1 -- 172.21.15.26:0/898446623 shutdown_connections 2024-09-16T14:57:54.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ee0b43640 1 -- 172.21.15.26:0/898446623 >> 172.21.15.26:0/898446623 conn(0x7f1edc0a7f30 msgr2=0x7f1edc06b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:54.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ee0b43640 1 -- 172.21.15.26:0/898446623 shutdown_connections 2024-09-16T14:57:54.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:53.980+0000 7f1ee0b43640 1 -- 172.21.15.26:0/898446623 wait complete. 2024-09-16T14:57:54.018 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:57:54.018 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:57:54.018 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:57:54.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f51176a2640 10 monclient: get_monmap_and_config 2024-09-16T14:57:54.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f51176a2640 10 monclient: build_initial_monmap 2024-09-16T14:57:54.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f51176a2640 10 monclient: monmap: 2024-09-16T14:57:54.137 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:54.137 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:54.137 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:54.139483+0000 2024-09-16T14:57:54.137 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:54.139483+0000 2024-09-16T14:57:54.138 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:54.138 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:54.138 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:54.138 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:54.138 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:54.138 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:54.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f51176a2640 1 Processor -- start 2024-09-16T14:57:54.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f51176a2640 1 -- start start 2024-09-16T14:57:54.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f51176a2640 10 monclient: init 2024-09-16T14:57:54.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f51176a2640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:54.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f51176a2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:57:54.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f51176a2640 10 monclient(hunting): picked mon.noname-c con 0x7f511009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:54.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f51176a2640 10 monclient(hunting): picked mon.noname-b con 0x7f5110132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:54.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f51176a2640 10 monclient(hunting): picked mon.noname-a con 0x7f5110138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:54.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f51176a2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511009d260 con 0x7f5110138ff0 2024-09-16T14:57:54.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f51176a2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5110132a80 con 0x7f511009cb80 2024-09-16T14:57:54.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f51176a2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511009cf60 con 0x7f5110132650 2024-09-16T14:57:54.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f51176a2640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:54.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f51176a2640 10 monclient(hunting): authenticate will time out at 1898.259683s 2024-09-16T14:57:54.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f5114c3d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5110132650 0x7f51101378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53648/0 (socket says 172.21.15.26:53648) 2024-09-16T14:57:54.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.136+0000 7f5114c3d640 1 -- 172.21.15.26:0/640222109 learned_addr learned my addr 172.21.15.26:0/640222109 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:54.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 1 -- 172.21.15.26:0/640222109 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526696332 0 0) 0x7f511009cf60 con 0x7f5110132650 2024-09-16T14:57:54.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:54.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:54.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:54.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient(hunting): my global_id is 284396 2024-09-16T14:57:54.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 1 -- 172.21.15.26:0/640222109 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50fc003530 con 0x7f5110132650 2024-09-16T14:57:54.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 1 -- 172.21.15.26:0/640222109 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506788659 0 0) 0x7f511009d260 con 0x7f5110138ff0 2024-09-16T14:57:54.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:54.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:54.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:54.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient(hunting): my global_id is 274476 2024-09-16T14:57:54.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 1 -- 172.21.15.26:0/640222109 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f511009cf60 con 0x7f5110138ff0 2024-09-16T14:57:54.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 1 -- 172.21.15.26:0/640222109 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992705778 0 0) 0x7f5110132a80 con 0x7f511009cb80 2024-09-16T14:57:54.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient(hunting): _negotiate 2024-09-16T14:57:54.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:54.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:54.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient(hunting): my global_id is 304141 2024-09-16T14:57:54.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 1 -- 172.21.15.26:0/640222109 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f511009d260 con 0x7f511009cb80 2024-09-16T14:57:54.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 1 -- 172.21.15.26:0/640222109 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1858156643 0 0) 0x7f50fc003530 con 0x7f5110132650 2024-09-16T14:57:54.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 1 -- 172.21.15.26:0/640222109 >> v1:172.21.15.189:6789/0 conn(0x7f511009cb80 legacy=0x7f51100717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 1 -- 172.21.15.26:0/640222109 >> v1:172.21.15.26:6789/0 conn(0x7f5110138ff0 legacy=0x7f511013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient: _finish_hunting 0 2024-09-16T14:57:54.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 1 monclient: found mon.noname-b 2024-09-16T14:57:54.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:54.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:57:54.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 1 -- 172.21.15.26:0/640222109 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511009f2d0 con 0x7f5110132650 2024-09-16T14:57:54.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient: _finish_auth 0 2024-09-16T14:57:54.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient: _check_auth_tickets 2024-09-16T14:57:54.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:54.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 1 -- 172.21.15.26:0/640222109 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5104001660 con 0x7f5110132650 2024-09-16T14:57:54.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:54.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:57:54.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient: dump: 2024-09-16T14:57:54.152 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:54.152 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:54.152 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.152 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.152 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:54.152 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:54.152 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:54.152 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:54.152 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:54.152 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:54.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 5 monclient: authenticate success, global_id 284396 2024-09-16T14:57:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 1 -- 172.21.15.26:0/640222109 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5104001880 con 0x7f5110132650 2024-09-16T14:57:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 1 -- 172.21.15.26:0/640222109 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5104001bd0 con 0x7f5110132650 2024-09-16T14:57:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 4 set_mon_vals no callback set 2024-09-16T14:57:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5103fff640 10 monclient: dump: 2024-09-16T14:57:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.153 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.154 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:54.154 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:54.154 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:54.154 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:54.154 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:54.154 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:54.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 10 monclient: shutdown 2024-09-16T14:57:54.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:54.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 1 -- 172.21.15.26:0/640222109 >> v1:172.21.15.26:6790/0 conn(0x7f5110132650 legacy=0x7f51101378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 1 -- 172.21.15.26:0/640222109 shutdown_connections 2024-09-16T14:57:54.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 1 -- 172.21.15.26:0/640222109 >> 172.21.15.26:0/640222109 conn(0x7f511009fb50 msgr2=0x7f511009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:54.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 1 -- 172.21.15.26:0/640222109 shutdown_connections 2024-09-16T14:57:54.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 1 -- 172.21.15.26:0/640222109 wait complete. 2024-09-16T14:57:54.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 10 monclient: build_initial_monmap 2024-09-16T14:57:54.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 10 monclient: monmap: 2024-09-16T14:57:54.155 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:54.155 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:54.155 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:54.155 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:54.155 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:54.155 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:54.155 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:54.155 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:54.155 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:54.155 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:54.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 1 Processor -- start 2024-09-16T14:57:54.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 1 -- start start 2024-09-16T14:57:54.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 10 monclient: init 2024-09-16T14:57:54.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:54.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:57:54.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 10 monclient(hunting): picked mon.noname-b con 0x7f511009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:54.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 10 monclient(hunting): picked mon.noname-c con 0x7f5110132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:54.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 10 monclient(hunting): picked mon.noname-a con 0x7f5110138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:54.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51100a2940 con 0x7f5110138ff0 2024-09-16T14:57:54.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5110071a30 con 0x7f511009cb80 2024-09-16T14:57:54.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511009a240 con 0x7f5110132650 2024-09-16T14:57:54.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:54.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f51176a2640 10 monclient(hunting): authenticate will time out at 1898.263373s 2024-09-16T14:57:54.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5114c3d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5110132650 0x7f511006de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53654/0 (socket says 172.21.15.26:53654) 2024-09-16T14:57:54.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5114c3d640 1 -- 172.21.15.26:0/840257658 learned_addr learned my addr 172.21.15.26:0/840257658 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:54.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5115c3f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5110138ff0 0x7f511006e560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37094/0 (socket says 172.21.15.26:37094) 2024-09-16T14:57:54.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694716770 0 0) 0x7f511009a240 con 0x7f5110132650 2024-09-16T14:57:54.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5101ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:54.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:54.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:54.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5101ffb640 10 monclient(hunting): my global_id is 284399 2024-09-16T14:57:54.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50f4003530 con 0x7f5110132650 2024-09-16T14:57:54.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257364876 0 0) 0x7f51100a2940 con 0x7f5110138ff0 2024-09-16T14:57:54.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5101ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:54.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:54.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:54.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5101ffb640 10 monclient(hunting): my global_id is 274479 2024-09-16T14:57:54.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.140+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f511009a240 con 0x7f5110138ff0 2024-09-16T14:57:54.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 490815933 0 0) 0x7f50f4003530 con 0x7f5110132650 2024-09-16T14:57:54.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51100a2940 con 0x7f5110132650 2024-09-16T14:57:54.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5104001560 con 0x7f5110132650 2024-09-16T14:57:54.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:54.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:54.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient(hunting): dump: 2024-09-16T14:57:54.158 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:54.158 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:54.158 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.158 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.158 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:54.158 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:54.158 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:54.158 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:54.159 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:54.159 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:54.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:54.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3315565191 0 0) 0x7f511009a240 con 0x7f5110138ff0 2024-09-16T14:57:54.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50f4003530 con 0x7f5110138ff0 2024-09-16T14:57:54.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5108001d60 con 0x7f5110138ff0 2024-09-16T14:57:54.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:54.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:57:54.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient(hunting): dump: 2024-09-16T14:57:54.159 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:54.159 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:54.160 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.160 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.160 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:54.160 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:54.160 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:54.160 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:54.160 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:54.160 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:54.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463407922 0 0) 0x7f5110071a30 con 0x7f511009cb80 2024-09-16T14:57:54.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:54.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:54.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:54.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient(hunting): my global_id is 304144 2024-09-16T14:57:54.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f511009a240 con 0x7f511009cb80 2024-09-16T14:57:54.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 453505469 0 0) 0x7f51100a2940 con 0x7f5110132650 2024-09-16T14:57:54.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 >> v1:172.21.15.189:6789/0 conn(0x7f511009cb80 legacy=0x7f511006d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 >> v1:172.21.15.26:6789/0 conn(0x7f5110138ff0 legacy=0x7f511006e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:57:54.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 monclient: found mon.c 2024-09-16T14:57:54.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:54.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:54.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5110070240 con 0x7f5110132650 2024-09-16T14:57:54.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient: _finish_auth 0 2024-09-16T14:57:54.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:57:54.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:54.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f51176a2640 5 monclient: authenticate success, global_id 284399 2024-09-16T14:57:54.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f51176a2640 10 monclient: _renew_subs 2024-09-16T14:57:54.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f51176a2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:54.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f51176a2640 1 -- 172.21.15.26:0/840257658 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511006f340 con 0x7f5110132650 2024-09-16T14:57:54.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f51176a2640 10 monclient: _renew_subs 2024-09-16T14:57:54.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f51176a2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:54.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f51176a2640 1 -- 172.21.15.26:0/840257658 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51100716a0 con 0x7f5110132650 2024-09-16T14:57:54.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51040010f0 con 0x7f5110132650 2024-09-16T14:57:54.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:54.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5104004c40 con 0x7f5110132650 2024-09-16T14:57:54.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:54.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:54.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f51037fe640 4 set_mon_vals no callback set 2024-09-16T14:57:54.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient: dump: 2024-09-16T14:57:54.163 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:54.163 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:54.163 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.163 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.163 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:54.163 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:54.163 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:54.163 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:54.164 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:54.164 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:54.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f51176a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:54.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f51176a2640 1 -- 172.21.15.26:0/840257658 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f50f0006590 con 0x7f50f0002830 2024-09-16T14:57:54.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f510400f6b0 con 0x7f5110132650 2024-09-16T14:57:54.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f5104043e20 con 0x7f5110132650 2024-09-16T14:57:54.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5115c3f640 10 monclient: get_auth_request con 0x7f50f4036420 auth_method 0 2024-09-16T14:57:54.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704932386 0 0) 0x7f50f0006590 con 0x7f50f0002830 2024-09-16T14:57:54.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:54.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:54.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:54.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient(hunting): my global_id is 304147 2024-09-16T14:57:54.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51100a2940 con 0x7f50f0002830 2024-09-16T14:57:54.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1121748322 0 0) 0x7f51100a2940 con 0x7f50f0002830 2024-09-16T14:57:54.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50f0006590 con 0x7f50f0002830 2024-09-16T14:57:54.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f510c0016f0 con 0x7f50f0002830 2024-09-16T14:57:54.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:54.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:57:54.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient: dump: 2024-09-16T14:57:54.167 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:54.167 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:54.167 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.167 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.167 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:54.167 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:54.168 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:54.168 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:54.168 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:54.168 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:54.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 338537531 0 0) 0x7f50f0006590 con 0x7f50f0002830 2024-09-16T14:57:54.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50f000a0e0 con 0x7f50f0002830 2024-09-16T14:57:54.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f510c0019d0 con 0x7f50f0002830 2024-09-16T14:57:54.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:54.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:54.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 >> v1:172.21.15.189:6789/0 conn(0x7f50f0002830 legacy=0x7f50f0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.144+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 >> v1:172.21.15.189:6789/0 conn(0x7f50f0002830 legacy=0x7f50f0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:54.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.152+0000 7f51176a2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:57:54.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.152+0000 7f51176a2640 1 -- 172.21.15.26:0/840257658 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f50f0009d80 con 0x7f50f00090a0 2024-09-16T14:57:54.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.152+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420667196 0 0) 0x7f50f0009d80 con 0x7f50f00090a0 2024-09-16T14:57:54.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.152+0000 7f5101ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:57:54.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.152+0000 7f5101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:54.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.152+0000 7f5101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:54.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.152+0000 7f5101ffb640 10 monclient(hunting): my global_id is 304150 2024-09-16T14:57:54.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.152+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50f0006590 con 0x7f50f00090a0 2024-09-16T14:57:54.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.152+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1810719512 0 0) 0x7f50f0006590 con 0x7f50f00090a0 2024-09-16T14:57:54.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.152+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50f0009d80 con 0x7f50f00090a0 2024-09-16T14:57:54.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.152+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f510c0037a0 con 0x7f50f00090a0 2024-09-16T14:57:54.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.152+0000 7f5101ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:54.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.152+0000 7f5101ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:57:54.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.152+0000 7f5101ffb640 10 monclient: dump: 2024-09-16T14:57:54.174 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:54.174 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:54.174 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.174 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.175 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:54.175 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:54.175 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:54.175 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:54.175 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:54.175 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:54.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.152+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4143084431 0 0) 0x7f50f0009d80 con 0x7f50f00090a0 2024-09-16T14:57:54.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.152+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f50f000c290 con 0x7f50f00090a0 2024-09-16T14:57:54.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.156+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3277959789) 0x7f510c003a10 con 0x7f50f00090a0 2024-09-16T14:57:54.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.156+0000 7f5101ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:57:54.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.156+0000 7f5101ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:54.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.156+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 >> v1:172.21.15.189:6789/0 conn(0x7f50f00090a0 legacy=0x7f50f0009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.156+0000 7f5101ffb640 1 -- 172.21.15.26:0/840257658 >> v1:172.21.15.189:6789/0 conn(0x7f50f00090a0 legacy=0x7f50f0009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:57:54.177 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:57:54.177 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:54.178 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:54.178 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:57:54.178 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 144, 2024-09-16T14:57:54.178 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:57:54.178 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:57:54.179 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:57:54.179 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:57:54.179 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:54.179 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:57:54.179 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:54.179 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:57:54.180 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:57:54.180 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:54.180 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:54.180 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:54.180 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:54.180 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:54.180 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:54.181 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:54.181 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:54.181 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:54.181 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:54.181 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:54.181 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:57:54.182 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:57:54.182 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:54.182 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:54.182 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:54.182 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:54.182 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:54.182 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:54.182 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:57:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:57:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:57:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:57:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:57:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:57:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:57:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:57:54.184 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:57:54.184 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:57:54.184 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:57:54.184 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:57:54.184 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:57:54.184 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:57:54.184 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:57:54.184 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:57:54.184 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:57:54.184 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:57:54.185 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:57:54.185 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:57:54.185 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:57:54.185 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:57:54.185 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:57:54.185 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:57:54.185 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:57:54.185 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:57:54.185 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:54.185 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:57:54.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.185 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:57:54.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:54.186 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:57:54.186 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:57:54.186 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:54.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:54.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:54.186 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:54.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:57:54.186 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:54.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:54.186 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:54.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:57:54.186 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:57:54.187 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:54.187 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:54.187 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:54.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:54.187 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:57:54.187 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:57:54.187 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:54.187 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:54.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:54.187 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:54.187 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:57:54.187 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:54.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:54.188 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:54.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.188 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:57:54.188 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:57:54.188 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:54.188 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:54.188 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:54.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:54.188 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:57:54.188 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:57:54.188 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:57:54.189 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:57:54.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:54.189 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:57:54.189 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:57:54.189 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:57:54.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:54.189 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:54.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.189 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:57:54.189 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:57:54.189 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:57:54.189 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:57:54.189 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:57:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:57:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:57:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:57:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:57:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:57:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:57:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:57:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:57:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:57:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:57:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:57:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:57:54.191 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:57:54.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.156+0000 7f51176a2640 1 -- 172.21.15.26:0/840257658 >> v1:172.21.15.189:6812/3196161582 conn(0x7f50f4036420 legacy=0x7f50f40388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.156+0000 7f51176a2640 10 monclient: shutdown 2024-09-16T14:57:54.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.156+0000 7f51176a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:54.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.156+0000 7f51176a2640 1 -- 172.21.15.26:0/840257658 >> v1:172.21.15.26:6790/0 conn(0x7f5110132650 legacy=0x7f511006de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.156+0000 7f5115c3f640 1 -- 172.21.15.26:0/840257658 reap_dead start 2024-09-16T14:57:54.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.156+0000 7f5115c3f640 1 -- 172.21.15.26:0/840257658 reap_dead start 2024-09-16T14:57:54.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.156+0000 7f51176a2640 1 -- 172.21.15.26:0/840257658 shutdown_connections 2024-09-16T14:57:54.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.156+0000 7f51176a2640 1 -- 172.21.15.26:0/840257658 >> 172.21.15.26:0/840257658 conn(0x7f511009fb50 msgr2=0x7f51100a1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:54.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.156+0000 7f51176a2640 1 -- 172.21.15.26:0/840257658 shutdown_connections 2024-09-16T14:57:54.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.156+0000 7f51176a2640 1 -- 172.21.15.26:0/840257658 wait complete. 2024-09-16T14:57:54.193 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:57:54.193 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T14:57:54.194 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:57:54.194 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T14:57:54.194 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:57:54.194 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T14:57:54.194 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:57:54.216 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T14:57:54.216 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:57:54.217 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:57:54.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadf027e640 10 monclient: get_monmap_and_config 2024-09-16T14:57:54.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadf027e640 10 monclient: build_initial_monmap 2024-09-16T14:57:54.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadf027e640 10 monclient: monmap: 2024-09-16T14:57:54.381 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:54.381 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:54.381 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:54.382696+0000 2024-09-16T14:57:54.382 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:54.382696+0000 2024-09-16T14:57:54.382 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:54.382 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:54.382 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:54.382 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:54.382 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:54.383 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:54.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadf027e640 1 Processor -- start 2024-09-16T14:57:54.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadf027e640 1 -- start start 2024-09-16T14:57:54.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadf027e640 10 monclient: init 2024-09-16T14:57:54.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadf027e640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:54.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadf027e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:57:54.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadf027e640 10 monclient(hunting): picked mon.noname-c con 0x7fade80a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:54.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadf027e640 10 monclient(hunting): picked mon.noname-b con 0x7fade80a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:54.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadf027e640 10 monclient(hunting): picked mon.noname-a con 0x7fade80a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:54.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadf027e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade80a3250 con 0x7fade80a2800 2024-09-16T14:57:54.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadf027e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade809f600 con 0x7fade80a1d10 2024-09-16T14:57:54.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadf027e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade80a3830 con 0x7fade80a0000 2024-09-16T14:57:54.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadf027e640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:54.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadf027e640 10 monclient(hunting): authenticate will time out at 1898.503435s 2024-09-16T14:57:54.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7fadee81b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fade80a2800 legacy=0x7fade813d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:57:54.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7faded819640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fade80a0000 0x7fade8139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53698/0 (socket says 172.21.15.26:53698) 2024-09-16T14:57:54.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7faded819640 1 -- 172.21.15.26:0/1573909908 learned_addr learned my addr 172.21.15.26:0/1573909908 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:54.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7faded018640 1 -- 172.21.15.26:0/1573909908 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970896347 0 0) 0x7fade80a3830 con 0x7fade80a0000 2024-09-16T14:57:54.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7faded018640 10 monclient(hunting): _negotiate 2024-09-16T14:57:54.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7faded018640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:54.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7faded018640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:54.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7faded018640 10 monclient(hunting): my global_id is 284402 2024-09-16T14:57:54.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.380+0000 7faded018640 1 -- 172.21.15.26:0/1573909908 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd4003530 con 0x7fade80a0000 2024-09-16T14:57:54.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 1 -- 172.21.15.26:0/1573909908 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1692840177 0 0) 0x7fadd4003530 con 0x7fade80a0000 2024-09-16T14:57:54.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 1 -- 172.21.15.26:0/1573909908 >> v1:172.21.15.189:6789/0 conn(0x7fade80a1d10 legacy=0x7fade80a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 1 -- 172.21.15.26:0/1573909908 >> v1:172.21.15.26:6789/0 conn(0x7fade80a2800 legacy=0x7fade813d440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:57:54.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 10 monclient: _finish_hunting 0 2024-09-16T14:57:54.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 1 monclient: found mon.noname-b 2024-09-16T14:57:54.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:54.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:57:54.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 1 -- 172.21.15.26:0/1573909908 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade809ed60 con 0x7fade80a0000 2024-09-16T14:57:54.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 10 monclient: _finish_auth 0 2024-09-16T14:57:54.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 10 monclient: _check_auth_tickets 2024-09-16T14:57:54.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:54.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 1 -- 172.21.15.26:0/1573909908 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fadd8001620 con 0x7fade80a0000 2024-09-16T14:57:54.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 5 monclient: authenticate success, global_id 284402 2024-09-16T14:57:54.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:54.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:54.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:57:54.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 10 monclient: dump: 2024-09-16T14:57:54.395 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:54.396 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:54.396 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.396 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.396 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:54.396 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:54.396 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:54.396 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:54.396 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:54.396 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:54.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:54.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 1 -- 172.21.15.26:0/1573909908 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadd8001940 con 0x7fade80a0000 2024-09-16T14:57:54.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:54.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 1 -- 172.21.15.26:0/1573909908 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fadd8001d10 con 0x7fade80a0000 2024-09-16T14:57:54.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:54.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:54.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded018640 10 monclient: dump: 2024-09-16T14:57:54.397 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:54.397 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:54.397 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.397 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.397 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:54.397 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:54.397 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:54.397 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:54.397 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:54.397 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:54.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:54.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 4 set_mon_vals no callback set 2024-09-16T14:57:54.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 10 monclient: shutdown 2024-09-16T14:57:54.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:54.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 1 -- 172.21.15.26:0/1573909908 >> v1:172.21.15.26:6790/0 conn(0x7fade80a0000 legacy=0x7fade8139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 1 -- 172.21.15.26:0/1573909908 shutdown_connections 2024-09-16T14:57:54.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 1 -- 172.21.15.26:0/1573909908 >> 172.21.15.26:0/1573909908 conn(0x7fade80a7f30 msgr2=0x7fade80a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:54.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 1 -- 172.21.15.26:0/1573909908 shutdown_connections 2024-09-16T14:57:54.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 1 -- 172.21.15.26:0/1573909908 wait complete. 2024-09-16T14:57:54.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 10 monclient: build_initial_monmap 2024-09-16T14:57:54.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 10 monclient: monmap: 2024-09-16T14:57:54.398 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:54.398 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:54.398 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:54.399 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:54.399 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:54.399 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:54.399 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:54.399 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:54.399 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:54.399 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:54.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 1 Processor -- start 2024-09-16T14:57:54.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 1 -- start start 2024-09-16T14:57:54.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 10 monclient: init 2024-09-16T14:57:54.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:54.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:57:54.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 10 monclient(hunting): picked mon.noname-b con 0x7fade80a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:54.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 10 monclient(hunting): picked mon.noname-a con 0x7fade80a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:54.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 10 monclient(hunting): picked mon.noname-c con 0x7fade80a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:54.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade813d0c0 con 0x7fade80a1d10 2024-09-16T14:57:54.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade809e500 con 0x7fade80a0000 2024-09-16T14:57:54.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade8139f30 con 0x7fade80a2800 2024-09-16T14:57:54.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:54.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 10 monclient(hunting): authenticate will time out at 1898.506672s 2024-09-16T14:57:54.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faded819640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fade80a1d10 legacy=0x7fade8139820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:57:54.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadee81b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fade80a2800 0x7fade81d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53704/0 (socket says 172.21.15.26:53704) 2024-09-16T14:57:54.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadee81b640 1 -- 172.21.15.26:0/1827632829 learned_addr learned my addr 172.21.15.26:0/1827632829 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:54.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200676252 0 0) 0x7fade8139f30 con 0x7fade80a2800 2024-09-16T14:57:54.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:54.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient(hunting): my global_id is 284405 2024-09-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadcc003530 con 0x7fade80a2800 2024-09-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4092241303 0 0) 0x7fadcc003530 con 0x7fade80a2800 2024-09-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fade8139f30 con 0x7fade80a2800 2024-09-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fade00043c0 con 0x7fade80a2800 2024-09-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient(hunting): dump: 2024-09-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.401 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837301103 0 0) 0x7fade809e500 con 0x7fade80a0000 2024-09-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient(hunting): _negotiate 2024-09-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient(hunting): my global_id is 304156 2024-09-16T14:57:54.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadcc003530 con 0x7fade80a0000 2024-09-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1502749107 0 0) 0x7fade8139f30 con 0x7fade80a2800 2024-09-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 >> v1:172.21.15.189:6789/0 conn(0x7fade80a0000 legacy=0x7fade8139110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 >> v1:172.21.15.26:6789/0 conn(0x7fade80a1d10 legacy=0x7fade8139820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient: _finish_hunting 0 2024-09-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 1 monclient: found mon.c 2024-09-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade80a6740 con 0x7fade80a2800 2024-09-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient: _finish_auth 0 2024-09-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient: _check_auth_tickets 2024-09-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 5 monclient: authenticate success, global_id 284405 2024-09-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 10 monclient: _renew_subs 2024-09-16T14:57:54.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 1 -- 172.21.15.26:0/1827632829 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade813b6b0 con 0x7fade80a2800 2024-09-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 10 monclient: _renew_subs 2024-09-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadf027e640 1 -- 172.21.15.26:0/1827632829 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fade813bb50 con 0x7fade80a2800 2024-09-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fade00045e0 con 0x7fade80a2800 2024-09-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fade0004970 con 0x7fade80a2800 2024-09-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7fadec817640 4 set_mon_vals no callback set 2024-09-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.384+0000 7faddeffd640 10 monclient: dump: 2024-09-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:54.404 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:54.405 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.405 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:54.405 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:54.405 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:54.405 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:54.405 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:54.405 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:54.405 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:54.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.388+0000 7fadf027e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:54.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.388+0000 7fadf027e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:54.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.388+0000 7fadf027e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:54.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.388+0000 7fadf027e640 1 -- 172.21.15.26:0/1827632829 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd0005050 con 0x7fade80a2800 2024-09-16T14:57:54.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.388+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fade0004b50 con 0x7fade80a2800 2024-09-16T14:57:54.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.388+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fade0043fa0 con 0x7fade80a2800 2024-09-16T14:57:54.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.388+0000 7fadee01a640 10 monclient: get_auth_request con 0x7fadcc036180 auth_method 0 2024-09-16T14:57:54.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.392+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fade0005350 con 0x7fade80a2800 2024-09-16T14:57:54.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.392+0000 7faddeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:54.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.392+0000 7faddeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:54.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.608+0000 7fadf027e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:57:54.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.608+0000 7fadf027e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:54.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.608+0000 7fadf027e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:54.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.608+0000 7fadf027e640 1 -- 172.21.15.26:0/1827632829 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fadd0005880 con 0x7fade80a2800 2024-09-16T14:57:54.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.608+0000 7faddeffd640 1 -- 172.21.15.26:0/1827632829 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 3235206624) 0x7fade0012d90 con 0x7fade80a2800 2024-09-16T14:57:54.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.608+0000 7faddeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:54.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.608+0000 7faddeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:54.611 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:57:54.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.612+0000 7fadf027e640 1 -- 172.21.15.26:0/1827632829 >> v1:172.21.15.189:6812/3196161582 conn(0x7fadcc036180 legacy=0x7fadcc038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.612+0000 7fadf027e640 10 monclient: shutdown 2024-09-16T14:57:54.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.612+0000 7fadf027e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:54.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.612+0000 7fadf027e640 1 -- 172.21.15.26:0/1827632829 >> v1:172.21.15.26:6790/0 conn(0x7fade80a2800 legacy=0x7fade81d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:54.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.612+0000 7fadf027e640 1 -- 172.21.15.26:0/1827632829 shutdown_connections 2024-09-16T14:57:54.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.612+0000 7fadf027e640 1 -- 172.21.15.26:0/1827632829 >> 172.21.15.26:0/1827632829 conn(0x7fade80a7f30 msgr2=0x7fade809b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:54.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.612+0000 7fadf027e640 1 -- 172.21.15.26:0/1827632829 shutdown_connections 2024-09-16T14:57:54.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:54.612+0000 7fadf027e640 1 -- 172.21.15.26:0/1827632829 wait complete. 2024-09-16T14:57:57.631 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:57:57.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb90ad59640 10 monclient: get_monmap_and_config 2024-09-16T14:57:57.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb90ad59640 10 monclient: build_initial_monmap 2024-09-16T14:57:57.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb90ad59640 10 monclient: monmap: 2024-09-16T14:57:57.761 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:57.761 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:57.762 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:57:57.763661+0000 2024-09-16T14:57:57.762 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:57:57.763661+0000 2024-09-16T14:57:57.762 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:57.762 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:57.762 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:57.762 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:57:57.762 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:57:57.762 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:57.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb90ad59640 1 Processor -- start 2024-09-16T14:57:57.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb90ad59640 1 -- start start 2024-09-16T14:57:57.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb90ad59640 10 monclient: init 2024-09-16T14:57:57.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb90ad59640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:57.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb90ad59640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:57:57.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb90ad59640 10 monclient(hunting): picked mon.noname-a con 0x7fb904134e40 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:57.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb90ad59640 10 monclient(hunting): picked mon.noname-b con 0x7fb904099a70 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:57.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb90ad59640 10 monclient(hunting): picked mon.noname-c con 0x7fb904135220 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:57.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb90ad59640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb904099e80 con 0x7fb904134e40 2024-09-16T14:57:57.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb90ad59640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9040719d0 con 0x7fb904135220 2024-09-16T14:57:57.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb90ad59640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90409e1e0 con 0x7fb904099a70 2024-09-16T14:57:57.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb90ad59640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:57.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb90ad59640 10 monclient(hunting): authenticate will time out at 1901.883706s 2024-09-16T14:57:57.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb908af5640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fb904134e40 legacy=0x7fb904070bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:57:57.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb903fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb904099a70 0x7fb9040712c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39376/0 (socket says 172.21.15.26:39376) 2024-09-16T14:57:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.760+0000 7fb903fff640 1 -- 172.21.15.26:0/3832208465 learned_addr learned my addr 172.21.15.26:0/3832208465 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 1 -- 172.21.15.26:0/3832208465 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4032265391 0 0) 0x7fb90409e1e0 con 0x7fb904099a70 2024-09-16T14:57:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 10 monclient(hunting): _negotiate 2024-09-16T14:57:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 10 monclient(hunting): my global_id is 284408 2024-09-16T14:57:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 1 -- 172.21.15.26:0/3832208465 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f0003530 con 0x7fb904099a70 2024-09-16T14:57:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 1 -- 172.21.15.26:0/3832208465 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1804693170 0 0) 0x7fb8f0003530 con 0x7fb904099a70 2024-09-16T14:57:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 1 -- 172.21.15.26:0/3832208465 >> v1:172.21.15.189:6789/0 conn(0x7fb904135220 legacy=0x7fb90413db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 1 -- 172.21.15.26:0/3832208465 >> v1:172.21.15.26:6789/0 conn(0x7fb904134e40 legacy=0x7fb904070bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:57:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 10 monclient: _finish_hunting 0 2024-09-16T14:57:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 1 monclient: found mon.noname-b 2024-09-16T14:57:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:57:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 1 -- 172.21.15.26:0/3832208465 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90409e7d0 con 0x7fb904099a70 2024-09-16T14:57:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 10 monclient: _finish_auth 0 2024-09-16T14:57:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 10 monclient: _check_auth_tickets 2024-09-16T14:57:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 1 -- 172.21.15.26:0/3832208465 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb8f4001620 con 0x7fb904099a70 2024-09-16T14:57:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:57:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 10 monclient: dump: 2024-09-16T14:57:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:57.772 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 5 monclient: authenticate success, global_id 284408 2024-09-16T14:57:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:57:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 1 -- 172.21.15.26:0/3832208465 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8f40018d0 con 0x7fb904099a70 2024-09-16T14:57:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 1 -- 172.21.15.26:0/3832208465 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb8f4001c60 con 0x7fb904099a70 2024-09-16T14:57:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 10 monclient: get_monmap_and_config success 2024-09-16T14:57:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 4 set_mon_vals no callback set 2024-09-16T14:57:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9037fe640 10 monclient: dump: 2024-09-16T14:57:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:57.773 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 10 monclient: shutdown 2024-09-16T14:57:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 1 -- 172.21.15.26:0/3832208465 >> v1:172.21.15.26:6790/0 conn(0x7fb904099a70 legacy=0x7fb9040712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 1 -- 172.21.15.26:0/3832208465 shutdown_connections 2024-09-16T14:57:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 1 -- 172.21.15.26:0/3832208465 >> 172.21.15.26:0/3832208465 conn(0x7fb90409fb50 msgr2=0x7fb90409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 1 -- 172.21.15.26:0/3832208465 shutdown_connections 2024-09-16T14:57:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 1 -- 172.21.15.26:0/3832208465 wait complete. 2024-09-16T14:57:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 10 monclient: build_initial_monmap 2024-09-16T14:57:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 10 monclient: monmap: 2024-09-16T14:57:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:57:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:57:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:57:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:57:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:57:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:57:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:57:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:57:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:57:57.775 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 1 Processor -- start 2024-09-16T14:57:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 1 -- start start 2024-09-16T14:57:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 10 monclient: init 2024-09-16T14:57:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 10 monclient: _reopen_session rank -1 2024-09-16T14:57:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:57:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 10 monclient(hunting): picked mon.noname-a con 0x7fb904099a70 addr v1:172.21.15.26:6789/0 2024-09-16T14:57:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 10 monclient(hunting): picked mon.noname-c con 0x7fb904134e40 addr v1:172.21.15.26:6790/0 2024-09-16T14:57:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 10 monclient(hunting): picked mon.noname-b con 0x7fb904135220 addr v1:172.21.15.189:6789/0 2024-09-16T14:57:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb904124450 con 0x7fb904099a70 2024-09-16T14:57:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb904122cc0 con 0x7fb904135220 2024-09-16T14:57:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90409c200 con 0x7fb904134e40 2024-09-16T14:57:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 10 monclient(hunting): _renew_subs 2024-09-16T14:57:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 10 monclient(hunting): authenticate will time out at 1901.886776s 2024-09-16T14:57:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb908af5640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fb904099a70 legacy=0x7fb9041c9a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:57:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb903fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb904134e40 0x7fb9041cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39378/0 (socket says 172.21.15.26:39378) 2024-09-16T14:57:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb903fff640 1 -- 172.21.15.26:0/3539144857 learned_addr learned my addr 172.21.15.26:0/3539144857 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:57:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 1 -- 172.21.15.26:0/3539144857 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156143962 0 0) 0x7fb90409c200 con 0x7fb904134e40 2024-09-16T14:57:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 10 monclient(hunting): _negotiate 2024-09-16T14:57:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:57:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:57:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 10 monclient(hunting): my global_id is 284411 2024-09-16T14:57:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 1 -- 172.21.15.26:0/3539144857 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e4003530 con 0x7fb904134e40 2024-09-16T14:57:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 1 -- 172.21.15.26:0/3539144857 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2073082849 0 0) 0x7fb8e4003530 con 0x7fb904134e40 2024-09-16T14:57:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 1 -- 172.21.15.26:0/3539144857 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb90409c200 con 0x7fb904134e40 2024-09-16T14:57:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 1 -- 172.21.15.26:0/3539144857 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb8f4001be0 con 0x7fb904134e40 2024-09-16T14:57:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:57:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 10 monclient(hunting): dump: 2024-09-16T14:57:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:57.778 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:57:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 1 -- 172.21.15.26:0/3539144857 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 225174905 0 0) 0x7fb90409c200 con 0x7fb904134e40 2024-09-16T14:57:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 1 -- 172.21.15.26:0/3539144857 >> v1:172.21.15.189:6789/0 conn(0x7fb904135220 legacy=0x7fb9041d2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 1 -- 172.21.15.26:0/3539144857 >> v1:172.21.15.26:6789/0 conn(0x7fb904099a70 legacy=0x7fb9041c9a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:57:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 10 monclient: _finish_hunting 0 2024-09-16T14:57:57.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 1 monclient: found mon.c 2024-09-16T14:57:57.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:57:57.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:57.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 1 -- 172.21.15.26:0/3539144857 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9041340c0 con 0x7fb904134e40 2024-09-16T14:57:57.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 10 monclient: _finish_auth 0 2024-09-16T14:57:57.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 10 monclient: _check_auth_tickets 2024-09-16T14:57:57.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb9017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:57:57.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 5 monclient: authenticate success, global_id 284411 2024-09-16T14:57:57.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 10 monclient: _renew_subs 2024-09-16T14:57:57.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:57.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 1 -- 172.21.15.26:0/3539144857 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb904124170 con 0x7fb904134e40 2024-09-16T14:57:57.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 10 monclient: _renew_subs 2024-09-16T14:57:57.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:57.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.764+0000 7fb90ad59640 1 -- 172.21.15.26:0/3539144857 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb904121460 con 0x7fb904134e40 2024-09-16T14:57:57.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.768+0000 7fb9017fa640 1 -- 172.21.15.26:0/3539144857 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8f4004aa0 con 0x7fb904134e40 2024-09-16T14:57:57.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.768+0000 7fb9017fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:57:57.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.768+0000 7fb9017fa640 1 -- 172.21.15.26:0/3539144857 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb8f4001220 con 0x7fb904134e40 2024-09-16T14:57:57.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.768+0000 7fb9017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:57:57.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.768+0000 7fb9017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:57:57.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.768+0000 7fb902ffd640 4 set_mon_vals no callback set 2024-09-16T14:57:57.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.768+0000 7fb9017fa640 10 monclient: dump: 2024-09-16T14:57:57.781 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:57:57.781 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:57:57.781 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:57.781 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:57:57.781 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:57:57.781 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:57:57.781 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:57:57.781 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:57:57.781 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:57:57.782 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:57:57.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.768+0000 7fb9017fa640 1 -- 172.21.15.26:0/3539144857 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb8f4004e30 con 0x7fb904134e40 2024-09-16T14:57:57.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.768+0000 7fb90ad59640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:57.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.768+0000 7fb90ad59640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:57.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.768+0000 7fb90ad59640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:57.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.768+0000 7fb90ad59640 1 -- 172.21.15.26:0/3539144857 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e8005050 con 0x7fb904134e40 2024-09-16T14:57:57.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.768+0000 7fb9017fa640 1 -- 172.21.15.26:0/3539144857 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fb8f4043f80 con 0x7fb904134e40 2024-09-16T14:57:57.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.768+0000 7fb908af5640 10 monclient: get_auth_request con 0x7fb8e4036080 auth_method 0 2024-09-16T14:57:57.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.772+0000 7fb9017fa640 1 -- 172.21.15.26:0/3539144857 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb8f40064a0 con 0x7fb904134e40 2024-09-16T14:57:57.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.772+0000 7fb9017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:57:57.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.772+0000 7fb9017fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:57:57.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.996+0000 7fb90ad59640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:57:57.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.996+0000 7fb90ad59640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:57.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.996+0000 7fb90ad59640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:57:57.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.996+0000 7fb90ad59640 1 -- 172.21.15.26:0/3539144857 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb8e8005880 con 0x7fb904134e40 2024-09-16T14:57:57.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.996+0000 7fb9017fa640 1 -- 172.21.15.26:0/3539144857 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2419694529) 0x7fb8f4012df0 con 0x7fb904134e40 2024-09-16T14:57:57.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.996+0000 7fb9017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:57:57.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:57.996+0000 7fb9017fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:57:57.997 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:57:57.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:58.000+0000 7fb90ad59640 1 -- 172.21.15.26:0/3539144857 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb8e4036080 legacy=0x7fb8e4038540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:57.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:58.000+0000 7fb90ad59640 10 monclient: shutdown 2024-09-16T14:57:57.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:58.000+0000 7fb90ad59640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:57:57.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:58.000+0000 7fb90ad59640 1 -- 172.21.15.26:0/3539144857 >> v1:172.21.15.26:6790/0 conn(0x7fb904134e40 legacy=0x7fb9041cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:57:57.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:58.000+0000 7fb90ad59640 1 -- 172.21.15.26:0/3539144857 shutdown_connections 2024-09-16T14:57:58.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:58.000+0000 7fb90ad59640 1 -- 172.21.15.26:0/3539144857 >> 172.21.15.26:0/3539144857 conn(0x7fb90409fb50 msgr2=0x7fb9040a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:57:58.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:58.000+0000 7fb90ad59640 1 -- 172.21.15.26:0/3539144857 shutdown_connections 2024-09-16T14:57:58.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:57:58.000+0000 7fb90ad59640 1 -- 172.21.15.26:0/3539144857 wait complete. 2024-09-16T14:58:01.018 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:58:01.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f7201226640 10 monclient: get_monmap_and_config 2024-09-16T14:58:01.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f7201226640 10 monclient: build_initial_monmap 2024-09-16T14:58:01.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f7201226640 10 monclient: monmap: 2024-09-16T14:58:01.153 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:01.153 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:01.153 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:58:01.155159+0000 2024-09-16T14:58:01.153 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:58:01.155159+0000 2024-09-16T14:58:01.153 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:01.153 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:01.153 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:01.154 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:58:01.154 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:58:01.154 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:01.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f7201226640 1 Processor -- start 2024-09-16T14:58:01.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f7201226640 1 -- start start 2024-09-16T14:58:01.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f7201226640 10 monclient: init 2024-09-16T14:58:01.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f7201226640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:01.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f7201226640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:58:01.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f7201226640 10 monclient(hunting): picked mon.noname-b con 0x7f71fc09cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:01.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f7201226640 10 monclient(hunting): picked mon.noname-a con 0x7f71fc09e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:01.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f7201226640 10 monclient(hunting): picked mon.noname-c con 0x7f71fc09d1a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:01.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f7201226640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc09d5b0 con 0x7f71fc09e7a0 2024-09-16T14:58:01.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f7201226640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc131a90 con 0x7f71fc09d1a0 2024-09-16T14:58:01.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f7201226640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc09aff0 con 0x7f71fc09cdc0 2024-09-16T14:58:01.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f7201226640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:01.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f7201226640 10 monclient(hunting): authenticate will time out at 1905.275278s 2024-09-16T14:58:01.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f71fa59c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e7a0 legacy=0x7f71fc131380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:01.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f71fad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f71fc09cdc0 0x7f71fc130c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39384/0 (socket says 172.21.15.26:39384) 2024-09-16T14:58:01.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.152+0000 7f71fad9d640 1 -- 172.21.15.26:0/1037338825 learned_addr learned my addr 172.21.15.26:0/1037338825 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:01.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.356+0000 7f71fa59c640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e7a0 legacy=0x7f71fc131380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:01.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:01.760+0000 7f71fa59c640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e7a0 legacy=0x7f71fc131380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:02.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:02.564+0000 7f71fa59c640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e7a0 legacy=0x7f71fc131380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.152+0000 7f71f8d99640 10 monclient(hunting): tick 2024-09-16T14:58:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.152+0000 7f71f8d99640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:58:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.152+0000 7f71f8d99640 1 monclient(hunting): continuing hunt 2024-09-16T14:58:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.152+0000 7f71f8d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:58:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.152+0000 7f71f8d99640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.26:6790/0 conn(0x7f71fc09cdc0 legacy=0x7f71fc130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.152+0000 7f71f8d99640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.189:6789/0 conn(0x7f71fc09d1a0 legacy=0x7f71fc139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.152+0000 7f71f8d99640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e7a0 legacy=0x7f71fc131380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:58:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.152+0000 7f71f8d99640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:58:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.152+0000 7f71f8d99640 10 monclient(hunting): picked mon.noname-c con 0x7f71dc0047e0 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.152+0000 7f71f8d99640 10 monclient(hunting): picked mon.noname-b con 0x7f71dc0083d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.152+0000 7f71f8d99640 10 monclient(hunting): picked mon.noname-a con 0x7f71fc09e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.152+0000 7f71f8d99640 1 -- 172.21.15.26:0/1037338825 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc09b2d0 con 0x7f71fc09e7a0 2024-09-16T14:58:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.152+0000 7f71f8d99640 1 -- 172.21.15.26:0/1037338825 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc09d5b0 con 0x7f71dc0047e0 2024-09-16T14:58:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.152+0000 7f71f8d99640 1 -- 172.21.15.26:0/1037338825 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc0119b0 con 0x7f71dc0083d0 2024-09-16T14:58:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.152+0000 7f71f8d99640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.152+0000 7f71fb59e640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e7a0 legacy=0x7f71dc00e190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:04.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.356+0000 7f71fb59e640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e7a0 legacy=0x7f71dc00e190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:04.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:04.760+0000 7f71fb59e640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e7a0 legacy=0x7f71dc00e190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:05.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:05.564+0000 7f71fb59e640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e7a0 legacy=0x7f71dc00e190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:07.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.152+0000 7f71f8d99640 10 monclient(hunting): tick 2024-09-16T14:58:07.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.152+0000 7f71f8d99640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:58:07.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.152+0000 7f71f8d99640 1 monclient(hunting): continuing hunt 2024-09-16T14:58:07.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.152+0000 7f71f8d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:58:07.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.152+0000 7f71f8d99640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.26:6790/0 conn(0x7f71dc0083d0 legacy=0x7f71dc00a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:07.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.152+0000 7f71f8d99640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.189:6789/0 conn(0x7f71dc0047e0 legacy=0x7f71dc006c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:07.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.152+0000 7f71f8d99640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e7a0 legacy=0x7f71dc00e190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:58:07.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.152+0000 7f71f8d99640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:58:07.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.152+0000 7f71f8d99640 10 monclient(hunting): picked mon.noname-b con 0x7f71dc010a80 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:07.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.152+0000 7f71f8d99640 10 monclient(hunting): picked mon.noname-c con 0x7f71dc011b40 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:07.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.152+0000 7f71f8d99640 10 monclient(hunting): picked mon.noname-a con 0x7f71fc09e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:07.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.152+0000 7f71f8d99640 1 -- 172.21.15.26:0/1037338825 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc012c20 con 0x7f71fc09e7a0 2024-09-16T14:58:07.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.152+0000 7f71f8d99640 1 -- 172.21.15.26:0/1037338825 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc09b2d0 con 0x7f71dc011b40 2024-09-16T14:58:07.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.152+0000 7f71f8d99640 1 -- 172.21.15.26:0/1037338825 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc0126d0 con 0x7f71dc010a80 2024-09-16T14:58:07.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.152+0000 7f71f8d99640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:07.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.156+0000 7f71fb59e640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e7a0 legacy=0x7f71dc00e190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:07.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.360+0000 7f71fb59e640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e7a0 legacy=0x7f71dc00e190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:07.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:07.764+0000 7f71fb59e640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e7a0 legacy=0x7f71dc00e190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:08.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.536+0000 7f71f9d9b640 1 -- 172.21.15.26:0/1037338825 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600350136 0 0) 0x7f71fc09b2d0 con 0x7f71dc011b40 2024-09-16T14:58:08.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.536+0000 7f71f9d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:58:08.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.536+0000 7f71f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:08.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.536+0000 7f71f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:08.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.536+0000 7f71f9d9b640 10 monclient(hunting): my global_id is 304159 2024-09-16T14:58:08.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.536+0000 7f71f9d9b640 1 -- 172.21.15.26:0/1037338825 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71e8003530 con 0x7f71dc011b40 2024-09-16T14:58:08.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 1 -- 172.21.15.26:0/1037338825 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606187543 0 0) 0x7f71dc0126d0 con 0x7f71dc010a80 2024-09-16T14:58:08.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:58:08.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:08.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:08.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 10 monclient(hunting): my global_id is 284414 2024-09-16T14:58:08.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 1 -- 172.21.15.26:0/1037338825 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71fc09b2d0 con 0x7f71dc010a80 2024-09-16T14:58:08.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 1 -- 172.21.15.26:0/1037338825 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2638843097 0 0) 0x7f71fc09b2d0 con 0x7f71dc010a80 2024-09-16T14:58:08.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.189:6789/0 conn(0x7f71dc011b40 legacy=0x7f71dc011fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:08.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e7a0 legacy=0x7f71dc00e190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:58:08.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:58:08.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 1 monclient: found mon.noname-b 2024-09-16T14:58:08.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:08.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:58:08.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 1 -- 172.21.15.26:0/1037338825 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc012860 con 0x7f71dc010a80 2024-09-16T14:58:08.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 10 monclient: _finish_auth 0 2024-09-16T14:58:08.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:58:08.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:08.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 1 -- 172.21.15.26:0/1037338825 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f71e40015a0 con 0x7f71dc010a80 2024-09-16T14:58:08.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f7201226640 5 monclient: authenticate success, global_id 284414 2024-09-16T14:58:08.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f7201226640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:08.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:08.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:58:08.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f71f9d9b640 10 monclient: dump: 2024-09-16T14:58:08.546 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:08.546 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:08.546 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.546 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.546 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:08.546 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:08.546 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:08.546 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:08.546 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:08.546 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:08.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.544+0000 7f7201226640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:08.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f71f9d9b640 1 -- 172.21.15.26:0/1037338825 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71e40017c0 con 0x7f71dc010a80 2024-09-16T14:58:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f71f9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f71f9d9b640 1 -- 172.21.15.26:0/1037338825 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f71e4001b90 con 0x7f71dc010a80 2024-09-16T14:58:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f7201226640 10 monclient: get_monmap_and_config success 2024-09-16T14:58:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f7201226640 4 set_mon_vals no callback set 2024-09-16T14:58:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f71f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f71f9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f71f9d9b640 10 monclient: dump: 2024-09-16T14:58:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:08.549 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:08.549 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:08.549 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:08.549 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:08.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f7201226640 10 monclient: shutdown 2024-09-16T14:58:08.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f7201226640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:08.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f7201226640 1 -- 172.21.15.26:0/1037338825 >> v1:172.21.15.26:6790/0 conn(0x7f71dc010a80 legacy=0x7f71dc010ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:08.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f7201226640 1 -- 172.21.15.26:0/1037338825 shutdown_connections 2024-09-16T14:58:08.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f7201226640 1 -- 172.21.15.26:0/1037338825 >> 172.21.15.26:0/1037338825 conn(0x7f71fc09fb50 msgr2=0x7f71fc09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:08.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f7201226640 1 -- 172.21.15.26:0/1037338825 shutdown_connections 2024-09-16T14:58:08.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f7201226640 1 -- 172.21.15.26:0/1037338825 wait complete. 2024-09-16T14:58:08.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f7201226640 10 monclient: build_initial_monmap 2024-09-16T14:58:08.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.548+0000 7f7201226640 10 monclient: monmap: 2024-09-16T14:58:08.550 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:08.550 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:58:08.551 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:58:08.551 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:58:08.551 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:08.551 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:58:08.551 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:08.551 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:58:08.551 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:58:08.551 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:08.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 1 Processor -- start 2024-09-16T14:58:08.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 1 -- start start 2024-09-16T14:58:08.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 10 monclient: init 2024-09-16T14:58:08.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:08.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:58:08.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 10 monclient(hunting): picked mon.noname-b con 0x7f71dc010a80 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:08.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 10 monclient(hunting): picked mon.noname-c con 0x7f71dc011b40 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:08.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 10 monclient(hunting): picked mon.noname-a con 0x7f71fc09e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:08.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc1247c0 con 0x7f71fc09e7a0 2024-09-16T14:58:08.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc1251a0 con 0x7f71dc010a80 2024-09-16T14:58:08.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc124d80 con 0x7f71dc011b40 2024-09-16T14:58:08.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:08.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 10 monclient(hunting): authenticate will time out at 1912.673349s 2024-09-16T14:58:08.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71fb59e640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e7a0 legacy=0x7f71fc1d1bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:08.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71fa59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f71dc011b40 0x7f71fc121830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48006/0 (socket says 172.21.15.26:48006) 2024-09-16T14:58:08.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71fa59c640 1 -- 172.21.15.26:0/1241957548 learned_addr learned my addr 172.21.15.26:0/1241957548 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:08.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272144134 0 0) 0x7f71fc124d80 con 0x7f71dc011b40 2024-09-16T14:58:08.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:58:08.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:08.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:08.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient(hunting): my global_id is 284417 2024-09-16T14:58:08.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71dc012bc0 con 0x7f71dc011b40 2024-09-16T14:58:08.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863149923 0 0) 0x7f71fc1251a0 con 0x7f71dc010a80 2024-09-16T14:58:08.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:58:08.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:08.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:08.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient(hunting): my global_id is 304162 2024-09-16T14:58:08.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71fc124d80 con 0x7f71dc010a80 2024-09-16T14:58:08.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3234132445 0 0) 0x7f71dc012bc0 con 0x7f71dc011b40 2024-09-16T14:58:08.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71fc1251a0 con 0x7f71dc011b40 2024-09-16T14:58:08.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f71ec0041f0 con 0x7f71dc011b40 2024-09-16T14:58:08.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:08.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:58:08.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient(hunting): dump: 2024-09-16T14:58:08.557 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:08.557 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:08.557 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.557 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.557 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:08.557 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:08.557 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:08.557 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:08.557 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:08.557 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:58:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2132451995 0 0) 0x7f71fc1251a0 con 0x7f71dc011b40 2024-09-16T14:58:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 >> v1:172.21.15.189:6789/0 conn(0x7f71dc010a80 legacy=0x7f71fc121120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 >> v1:172.21.15.26:6789/0 conn(0x7f71fc09e7a0 legacy=0x7f71fc1d1bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:58:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:58:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 1 monclient: found mon.c 2024-09-16T14:58:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71fc0a1890 con 0x7f71dc011b40 2024-09-16T14:58:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient: _finish_auth 0 2024-09-16T14:58:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:58:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 5 monclient: authenticate success, global_id 284417 2024-09-16T14:58:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 10 monclient: _renew_subs 2024-09-16T14:58:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 1 -- 172.21.15.26:0/1241957548 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71fc09a240 con 0x7f71dc011b40 2024-09-16T14:58:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 10 monclient: _renew_subs 2024-09-16T14:58:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 1 -- 172.21.15.26:0/1241957548 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71fc124540 con 0x7f71dc011b40 2024-09-16T14:58:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71ec004600 con 0x7f71dc011b40 2024-09-16T14:58:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f959a640 4 set_mon_vals no callback set 2024-09-16T14:58:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f71ec004910 con 0x7f71dc011b40 2024-09-16T14:58:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f71f37fe640 10 monclient: dump: 2024-09-16T14:58:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:08.560 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:08.560 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.560 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.560 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:08.560 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:08.560 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:08.560 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:08.560 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:08.560 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:08.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:08.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:08.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.552+0000 7f7201226640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:08.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.556+0000 7f7201226640 1 -- 172.21.15.26:0/1241957548 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e0005050 con 0x7f71dc011b40 2024-09-16T14:58:08.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.556+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f71ec004b30 con 0x7f71dc011b40 2024-09-16T14:58:08.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.556+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f71ec043d10 con 0x7f71dc011b40 2024-09-16T14:58:08.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.556+0000 7f71fad9d640 10 monclient: get_auth_request con 0x7f71dc037990 auth_method 0 2024-09-16T14:58:08.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.560+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f71ec0145f0 con 0x7f71dc011b40 2024-09-16T14:58:08.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.560+0000 7f71f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:08.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.560+0000 7f71f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:58:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.784+0000 7f7201226640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:58:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.784+0000 7f7201226640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:58:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.784+0000 7f7201226640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.784+0000 7f7201226640 1 -- 172.21.15.26:0/1241957548 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f71e0005880 con 0x7f71dc011b40 2024-09-16T14:58:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.784+0000 7f71f37fe640 1 -- 172.21.15.26:0/1241957548 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 2518907284) 0x7f71ec012a60 con 0x7f71dc011b40 2024-09-16T14:58:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.784+0000 7f71f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:58:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.784+0000 7f71f37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:58:08.785 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":148,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:58:08.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.784+0000 7f7201226640 1 -- 172.21.15.26:0/1241957548 >> v1:172.21.15.189:6812/3196161582 conn(0x7f71dc037990 legacy=0x7f71dc039e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:08.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.784+0000 7f7201226640 10 monclient: shutdown 2024-09-16T14:58:08.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.784+0000 7f7201226640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:08.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.784+0000 7f7201226640 1 -- 172.21.15.26:0/1241957548 >> v1:172.21.15.26:6790/0 conn(0x7f71dc011b40 legacy=0x7f71fc121830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:08.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.784+0000 7f7201226640 1 -- 172.21.15.26:0/1241957548 shutdown_connections 2024-09-16T14:58:08.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.784+0000 7f7201226640 1 -- 172.21.15.26:0/1241957548 >> 172.21.15.26:0/1241957548 conn(0x7f71fc09fb50 msgr2=0x7f71fc09c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:08.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.788+0000 7f7201226640 1 -- 172.21.15.26:0/1241957548 shutdown_connections 2024-09-16T14:58:08.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.788+0000 7f7201226640 1 -- 172.21.15.26:0/1241957548 wait complete. 2024-09-16T14:58:08.804 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:58:08.804 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:58:08.804 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:58:08.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 10 monclient: get_monmap_and_config 2024-09-16T14:58:08.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 10 monclient: build_initial_monmap 2024-09-16T14:58:08.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 10 monclient: monmap: 2024-09-16T14:58:08.923 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:08.923 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:08.923 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:58:08.925588+0000 2024-09-16T14:58:08.924 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:58:08.925588+0000 2024-09-16T14:58:08.924 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:08.924 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:08.924 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:08.924 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:58:08.924 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:58:08.924 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 1 Processor -- start 2024-09-16T14:58:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 1 -- start start 2024-09-16T14:58:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 10 monclient: init 2024-09-16T14:58:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:58:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 10 monclient(hunting): picked mon.noname-b con 0x7f57d409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 10 monclient(hunting): picked mon.noname-a con 0x7f57d4132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 10 monclient(hunting): picked mon.noname-c con 0x7f57d4138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d409d260 con 0x7f57d4132650 2024-09-16T14:58:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d4132a80 con 0x7f57d4138ff0 2024-09-16T14:58:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d409cf60 con 0x7f57d409cb80 2024-09-16T14:58:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 10 monclient(hunting): authenticate will time out at 1913.045690s 2024-09-16T14:58:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d1d9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f57d4132650 legacy=0x7f57d41378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d259c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f57d409cb80 0x7f57d40717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48008/0 (socket says 172.21.15.26:48008) 2024-09-16T14:58:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d259c640 1 -- 172.21.15.26:0/3918929845 learned_addr learned my addr 172.21.15.26:0/3918929845 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 1 -- 172.21.15.26:0/3918929845 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139779253 0 0) 0x7f57d409cf60 con 0x7f57d409cb80 2024-09-16T14:58:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 10 monclient(hunting): _negotiate 2024-09-16T14:58:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 10 monclient(hunting): my global_id is 284420 2024-09-16T14:58:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 1 -- 172.21.15.26:0/3918929845 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57bc003530 con 0x7f57d409cb80 2024-09-16T14:58:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 1 -- 172.21.15.26:0/3918929845 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1734843096 0 0) 0x7f57bc003530 con 0x7f57d409cb80 2024-09-16T14:58:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 1 -- 172.21.15.26:0/3918929845 >> v1:172.21.15.189:6789/0 conn(0x7f57d4138ff0 legacy=0x7f57d413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 1 -- 172.21.15.26:0/3918929845 >> v1:172.21.15.26:6789/0 conn(0x7f57d4132650 legacy=0x7f57d41378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:58:08.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 10 monclient: _finish_hunting 0 2024-09-16T14:58:08.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 1 monclient: found mon.noname-b 2024-09-16T14:58:08.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:08.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:58:08.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 1 -- 172.21.15.26:0/3918929845 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d409f2d0 con 0x7f57d409cb80 2024-09-16T14:58:08.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 10 monclient: _finish_auth 0 2024-09-16T14:58:08.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 10 monclient: _check_auth_tickets 2024-09-16T14:58:08.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:08.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 1 -- 172.21.15.26:0/3918929845 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f57c8001660 con 0x7f57d409cb80 2024-09-16T14:58:08.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:08.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:58:08.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 10 monclient: dump: 2024-09-16T14:58:08.932 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:08.932 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:08.932 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.933 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.933 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:08.933 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:08.933 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:08.933 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:08.933 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:08.933 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:08.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 5 monclient: authenticate success, global_id 284420 2024-09-16T14:58:08.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:08.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 1 -- 172.21.15.26:0/3918929845 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57c80018c0 con 0x7f57d409cb80 2024-09-16T14:58:08.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:08.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 10 monclient: get_monmap_and_config success 2024-09-16T14:58:08.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 4 set_mon_vals no callback set 2024-09-16T14:58:08.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 1 -- 172.21.15.26:0/3918929845 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f57c8001c90 con 0x7f57d409cb80 2024-09-16T14:58:08.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:08.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:08.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d159a640 10 monclient: dump: 2024-09-16T14:58:08.934 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:08.934 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:08.934 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.934 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.934 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:08.934 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:08.934 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:08.934 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:08.934 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:08.934 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:08.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 10 monclient: shutdown 2024-09-16T14:58:08.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:08.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 1 -- 172.21.15.26:0/3918929845 >> v1:172.21.15.26:6790/0 conn(0x7f57d409cb80 legacy=0x7f57d40717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 1 -- 172.21.15.26:0/3918929845 shutdown_connections 2024-09-16T14:58:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 1 -- 172.21.15.26:0/3918929845 >> 172.21.15.26:0/3918929845 conn(0x7f57d409fb50 msgr2=0x7f57d409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 1 -- 172.21.15.26:0/3918929845 shutdown_connections 2024-09-16T14:58:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 1 -- 172.21.15.26:0/3918929845 wait complete. 2024-09-16T14:58:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 10 monclient: build_initial_monmap 2024-09-16T14:58:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.924+0000 7f57d8ebc640 10 monclient: monmap: 2024-09-16T14:58:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:58:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:58:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:58:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:58:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:58:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:58:08.936 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 1 Processor -- start 2024-09-16T14:58:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 1 -- start start 2024-09-16T14:58:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 10 monclient: init 2024-09-16T14:58:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:58:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 10 monclient(hunting): picked mon.noname-a con 0x7f57d409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 10 monclient(hunting): picked mon.noname-b con 0x7f57d4132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 10 monclient(hunting): picked mon.noname-c con 0x7f57d4138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d41381a0 con 0x7f57d409cb80 2024-09-16T14:58:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d406e480 con 0x7f57d4132650 2024-09-16T14:58:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d259c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f57d409cb80 legacy=0x7f57d406d660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d406fb60 con 0x7f57d4138ff0 2024-09-16T14:58:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 10 monclient(hunting): authenticate will time out at 1913.048818s 2024-09-16T14:58:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d2d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f57d4138ff0 0x7f57d41cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48024/0 (socket says 172.21.15.26:48024) 2024-09-16T14:58:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d2d9d640 1 -- 172.21.15.26:0/1984576961 learned_addr learned my addr 172.21.15.26:0/1984576961 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448102181 0 0) 0x7f57d406fb60 con 0x7f57d4138ff0 2024-09-16T14:58:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient(hunting): my global_id is 284423 2024-09-16T14:58:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b4003530 con 0x7f57d4138ff0 2024-09-16T14:58:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2794470820 0 0) 0x7f57b4003530 con 0x7f57d4138ff0 2024-09-16T14:58:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57d406fb60 con 0x7f57d4138ff0 2024-09-16T14:58:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f57cc001c20 con 0x7f57d4138ff0 2024-09-16T14:58:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:58:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient(hunting): dump: 2024-09-16T14:58:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:08.939 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:58:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4190508520 0 0) 0x7f57d406fb60 con 0x7f57d4138ff0 2024-09-16T14:58:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 >> v1:172.21.15.189:6789/0 conn(0x7f57d4132650 legacy=0x7f57d406dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 >> v1:172.21.15.26:6789/0 conn(0x7f57d409cb80 legacy=0x7f57d406d660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:58:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:58:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 monclient: found mon.c 2024-09-16T14:58:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d40a29a0 con 0x7f57d4138ff0 2024-09-16T14:58:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:58:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:58:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 5 monclient: authenticate success, global_id 284423 2024-09-16T14:58:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 10 monclient: _renew_subs 2024-09-16T14:58:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 1 -- 172.21.15.26:0/1984576961 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d409a240 con 0x7f57d4138ff0 2024-09-16T14:58:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 10 monclient: _renew_subs 2024-09-16T14:58:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 1 -- 172.21.15.26:0/1984576961 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d413c3b0 con 0x7f57d4138ff0 2024-09-16T14:58:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57cc001ff0 con 0x7f57d4138ff0 2024-09-16T14:58:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f57cc003df0 con 0x7f57d4138ff0 2024-09-16T14:58:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d0d99640 4 set_mon_vals no callback set 2024-09-16T14:58:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient: dump: 2024-09-16T14:58:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:08.941 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d8ebc640 1 -- 172.21.15.26:0/1984576961 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f57b8006590 con 0x7f57b8002830 2024-09-16T14:58:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f57cc003fd0 con 0x7f57d4138ff0 2024-09-16T14:58:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f57cc044160 con 0x7f57d4138ff0 2024-09-16T14:58:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266699901 0 0) 0x7f57b8006590 con 0x7f57b8002830 2024-09-16T14:58:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient(hunting): my global_id is 284426 2024-09-16T14:58:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d406fb60 con 0x7f57b8002830 2024-09-16T14:58:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57d1d9b640 10 monclient: get_auth_request con 0x7f57b4035fe0 auth_method 0 2024-09-16T14:58:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 934770892 0 0) 0x7f57d406fb60 con 0x7f57b8002830 2024-09-16T14:58:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57b8006590 con 0x7f57b8002830 2024-09-16T14:58:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f57c8001260 con 0x7f57b8002830 2024-09-16T14:58:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient: dump: 2024-09-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 683888347 0 0) 0x7f57b8006590 con 0x7f57b8002830 2024-09-16T14:58:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57b800a0e0 con 0x7f57b8002830 2024-09-16T14:58:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f57c80016c0 con 0x7f57b8002830 2024-09-16T14:58:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:58:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 >> v1:172.21.15.26:6790/0 conn(0x7f57b8002830 legacy=0x7f57b8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.928+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 >> v1:172.21.15.26:6790/0 conn(0x7f57b8002830 legacy=0x7f57b8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57d8ebc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:58:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57d8ebc640 1 -- 172.21.15.26:0/1984576961 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f57b8009d80 con 0x7f57b80090a0 2024-09-16T14:58:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031818363 0 0) 0x7f57b8009d80 con 0x7f57b80090a0 2024-09-16T14:58:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 10 monclient(hunting): my global_id is 284429 2024-09-16T14:58:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b8006590 con 0x7f57b80090a0 2024-09-16T14:58:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3072450270 0 0) 0x7f57b8006590 con 0x7f57b80090a0 2024-09-16T14:58:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57b8009d80 con 0x7f57b80090a0 2024-09-16T14:58:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f57c8004c80 con 0x7f57b80090a0 2024-09-16T14:58:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 10 monclient: dump: 2024-09-16T14:58:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:08.947 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3245859618 0 0) 0x7f57b8009d80 con 0x7f57b80090a0 2024-09-16T14:58:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f57b800c290 con 0x7f57b80090a0 2024-09-16T14:58:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4203 (unknown 0 0 3905672017) 0x7f57c8004fd0 con 0x7f57b80090a0 2024-09-16T14:58:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:58:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:58:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 >> v1:172.21.15.26:6790/0 conn(0x7f57b80090a0 legacy=0x7f57b8009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57c2ffd640 1 -- 172.21.15.26:0/1984576961 >> v1:172.21.15.26:6790/0 conn(0x7f57b80090a0 legacy=0x7f57b8009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:08.948 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:58:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:58:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:58:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:58:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 148, 2024-09-16T14:58:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:58:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:58:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:58:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:58:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:58:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:58:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:58:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:58:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:58:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:58:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:58:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:58:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:58:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:58:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:58:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:58:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:58:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:58:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:58:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:58:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:58:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:58:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:58:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:58:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:58:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:58:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:58:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:58:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:58:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:58:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:58:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:58:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:58:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:58:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:58:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:58:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:58:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:58:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:58:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:58:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:58:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:58:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:58:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:58:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:58:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:58:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:58:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:58:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:58:08.959 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:58:08.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57d8ebc640 1 -- 172.21.15.26:0/1984576961 >> v1:172.21.15.189:6812/3196161582 conn(0x7f57b4035fe0 legacy=0x7f57b40384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:08.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57d8ebc640 10 monclient: shutdown 2024-09-16T14:58:08.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57d8ebc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:08.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57d8ebc640 1 -- 172.21.15.26:0/1984576961 >> v1:172.21.15.26:6790/0 conn(0x7f57d4138ff0 legacy=0x7f57d41cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:08.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57d2d9d640 1 -- 172.21.15.26:0/1984576961 reap_dead start 2024-09-16T14:58:08.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57d2d9d640 1 -- 172.21.15.26:0/1984576961 reap_dead start 2024-09-16T14:58:08.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57d8ebc640 1 -- 172.21.15.26:0/1984576961 shutdown_connections 2024-09-16T14:58:08.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57d8ebc640 1 -- 172.21.15.26:0/1984576961 >> 172.21.15.26:0/1984576961 conn(0x7f57d409fb50 msgr2=0x7f57d4131ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:08.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57d8ebc640 1 -- 172.21.15.26:0/1984576961 shutdown_connections 2024-09-16T14:58:08.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:08.936+0000 7f57d8ebc640 1 -- 172.21.15.26:0/1984576961 wait complete. 2024-09-16T14:58:08.962 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:58:08.962 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T14:58:08.962 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:58:09.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.076+0000 7fe6744d2640 10 monclient: get_monmap_and_config 2024-09-16T14:58:09.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.076+0000 7fe6744d2640 10 monclient: build_initial_monmap 2024-09-16T14:58:09.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.076+0000 7fe6744d2640 10 monclient: monmap: 2024-09-16T14:58:09.079 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:09.079 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:09.079 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:58:09.081252+0000 2024-09-16T14:58:09.079 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:58:09.081252+0000 2024-09-16T14:58:09.079 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:09.079 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:09.080 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:09.080 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:58:09.080 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:58:09.080 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:09.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 1 Processor -- start 2024-09-16T14:58:09.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 1 -- start start 2024-09-16T14:58:09.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 10 monclient: init 2024-09-16T14:58:09.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:09.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:58:09.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 10 monclient(hunting): picked mon.noname-c con 0x7fe66c09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:09.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 10 monclient(hunting): picked mon.noname-b con 0x7fe66c132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:09.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 10 monclient(hunting): picked mon.noname-a con 0x7fe66c138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:09.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66c09d260 con 0x7fe66c138ff0 2024-09-16T14:58:09.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66c132a80 con 0x7fe66c09cb80 2024-09-16T14:58:09.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66c09cf60 con 0x7fe66c132650 2024-09-16T14:58:09.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:09.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 10 monclient(hunting): authenticate will time out at 1913.201327s 2024-09-16T14:58:09.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe672a6f640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fe66c138ff0 legacy=0x7fe66c13b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:09.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe671a6d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe66c132650 0x7fe66c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48042/0 (socket says 172.21.15.26:48042) 2024-09-16T14:58:09.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe671a6d640 1 -- 172.21.15.26:0/2685532998 learned_addr learned my addr 172.21.15.26:0/2685532998 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:09.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 1 -- 172.21.15.26:0/2685532998 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192581140 0 0) 0x7fe66c09cf60 con 0x7fe66c132650 2024-09-16T14:58:09.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 10 monclient(hunting): _negotiate 2024-09-16T14:58:09.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:09.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:09.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 10 monclient(hunting): my global_id is 284432 2024-09-16T14:58:09.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 1 -- 172.21.15.26:0/2685532998 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe658003530 con 0x7fe66c132650 2024-09-16T14:58:09.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 1 -- 172.21.15.26:0/2685532998 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3030002402 0 0) 0x7fe658003530 con 0x7fe66c132650 2024-09-16T14:58:09.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 1 -- 172.21.15.26:0/2685532998 >> v1:172.21.15.189:6789/0 conn(0x7fe66c09cb80 legacy=0x7fe66c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:09.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 1 -- 172.21.15.26:0/2685532998 >> v1:172.21.15.26:6789/0 conn(0x7fe66c138ff0 legacy=0x7fe66c13b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:58:09.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 10 monclient: _finish_hunting 0 2024-09-16T14:58:09.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 1 monclient: found mon.noname-b 2024-09-16T14:58:09.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:09.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:58:09.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 1 -- 172.21.15.26:0/2685532998 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66c09f2d0 con 0x7fe66c132650 2024-09-16T14:58:09.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 10 monclient: _finish_auth 0 2024-09-16T14:58:09.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 10 monclient: _check_auth_tickets 2024-09-16T14:58:09.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:09.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 5 monclient: authenticate success, global_id 284432 2024-09-16T14:58:09.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 1 -- 172.21.15.26:0/2685532998 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe668001660 con 0x7fe66c132650 2024-09-16T14:58:09.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:09.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:09.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:58:09.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 10 monclient: dump: 2024-09-16T14:58:09.089 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:09.089 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:09.089 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:09.089 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:09.089 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:09.089 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:09.089 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:09.090 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:09.090 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:09.090 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:09.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:09.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 1 -- 172.21.15.26:0/2685532998 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6680018c0 con 0x7fe66c132650 2024-09-16T14:58:09.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:09.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 1 -- 172.21.15.26:0/2685532998 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe668001c90 con 0x7fe66c132650 2024-09-16T14:58:09.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 10 monclient: get_monmap_and_config success 2024-09-16T14:58:09.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 4 set_mon_vals no callback set 2024-09-16T14:58:09.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:09.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:09.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe67126c640 10 monclient: dump: 2024-09-16T14:58:09.090 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:09.090 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:09.091 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:09.091 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:09.091 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:09.091 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:09.091 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:09.091 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:09.091 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:09.091 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:09.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 10 monclient: shutdown 2024-09-16T14:58:09.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:09.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 1 -- 172.21.15.26:0/2685532998 >> v1:172.21.15.26:6790/0 conn(0x7fe66c132650 legacy=0x7fe66c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:09.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 1 -- 172.21.15.26:0/2685532998 shutdown_connections 2024-09-16T14:58:09.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 1 -- 172.21.15.26:0/2685532998 >> 172.21.15.26:0/2685532998 conn(0x7fe66c09fb50 msgr2=0x7fe66c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:09.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 1 -- 172.21.15.26:0/2685532998 shutdown_connections 2024-09-16T14:58:09.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 1 -- 172.21.15.26:0/2685532998 wait complete. 2024-09-16T14:58:09.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 10 monclient: build_initial_monmap 2024-09-16T14:58:09.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 10 monclient: monmap: 2024-09-16T14:58:09.092 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:09.092 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:58:09.092 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:58:09.092 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:58:09.092 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:09.092 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:58:09.092 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:09.092 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:58:09.093 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:58:09.093 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:09.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 1 Processor -- start 2024-09-16T14:58:09.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 1 -- start start 2024-09-16T14:58:09.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.080+0000 7fe6744d2640 10 monclient: init 2024-09-16T14:58:09.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:09.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:58:09.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 10 monclient(hunting): picked mon.noname-a con 0x7fe66c09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:09.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 10 monclient(hunting): picked mon.noname-c con 0x7fe66c132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:09.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 10 monclient(hunting): picked mon.noname-b con 0x7fe66c138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:09.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66c06fec0 con 0x7fe66c09cb80 2024-09-16T14:58:09.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66c13c590 con 0x7fe66c138ff0 2024-09-16T14:58:09.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66c131e30 con 0x7fe66c132650 2024-09-16T14:58:09.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe67226e640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fe66c09cb80 legacy=0x7fe66c06d6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:09.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:09.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 10 monclient(hunting): authenticate will time out at 1913.204302s 2024-09-16T14:58:09.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe671a6d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe66c132650 0x7fe66c06ddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48052/0 (socket says 172.21.15.26:48052) 2024-09-16T14:58:09.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe671a6d640 1 -- 172.21.15.26:0/1997044417 learned_addr learned my addr 172.21.15.26:0/1997044417 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:09.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472736038 0 0) 0x7fe66c131e30 con 0x7fe66c132650 2024-09-16T14:58:09.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:09.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:09.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:09.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient(hunting): my global_id is 284435 2024-09-16T14:58:09.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe650003530 con 0x7fe66c132650 2024-09-16T14:58:09.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4144148432 0 0) 0x7fe650003530 con 0x7fe66c132650 2024-09-16T14:58:09.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe66c131e30 con 0x7fe66c132650 2024-09-16T14:58:09.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe668001620 con 0x7fe66c132650 2024-09-16T14:58:09.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:09.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:58:09.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient(hunting): dump: 2024-09-16T14:58:09.095 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:09.095 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:09.095 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:09.095 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:09.095 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:09.095 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:09.095 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:09.095 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:09.095 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:09.095 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:09.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:58:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3888882427 0 0) 0x7fe66c131e30 con 0x7fe66c132650 2024-09-16T14:58:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 >> v1:172.21.15.189:6789/0 conn(0x7fe66c138ff0 legacy=0x7fe66c06e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 >> v1:172.21.15.26:6789/0 conn(0x7fe66c09cb80 legacy=0x7fe66c06d6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:58:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:58:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 monclient: found mon.c 2024-09-16T14:58:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66c071300 con 0x7fe66c132650 2024-09-16T14:58:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient: _finish_auth 0 2024-09-16T14:58:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:58:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 5 monclient: authenticate success, global_id 284435 2024-09-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 10 monclient: _renew_subs 2024-09-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 1 -- 172.21.15.26:0/1997044417 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66c070070 con 0x7fe66c132650 2024-09-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 10 monclient: _renew_subs 2024-09-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 1 -- 172.21.15.26:0/1997044417 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66c071020 con 0x7fe66c132650 2024-09-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6680011d0 con 0x7fe66c132650 2024-09-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe668004da0 con 0x7fe66c132650 2024-09-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe663fff640 4 set_mon_vals no callback set 2024-09-16T14:58:09.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient: dump: 2024-09-16T14:58:09.098 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:09.098 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:09.098 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:09.098 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:09.098 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:09.098 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:09.098 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:09.098 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:09.098 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:09.098 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:09.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:09.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe6744d2640 1 -- 172.21.15.26:0/1997044417 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe654006590 con 0x7fe654002830 2024-09-16T14:58:09.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe66800f810 con 0x7fe66c132650 2024-09-16T14:58:09.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fe668043ff0 con 0x7fe66c132650 2024-09-16T14:58:09.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe672a6f640 10 monclient: get_auth_request con 0x7fe650036080 auth_method 0 2024-09-16T14:58:09.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501973518 0 0) 0x7fe654006590 con 0x7fe654002830 2024-09-16T14:58:09.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:09.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:09.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:09.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient(hunting): my global_id is 304177 2024-09-16T14:58:09.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe66c131e30 con 0x7fe654002830 2024-09-16T14:58:09.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3563333358 0 0) 0x7fe66c131e30 con 0x7fe654002830 2024-09-16T14:58:09.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe654006590 con 0x7fe654002830 2024-09-16T14:58:09.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe65c0016a0 con 0x7fe654002830 2024-09-16T14:58:09.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:09.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:58:09.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 10 monclient: dump: 2024-09-16T14:58:09.102 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:09.102 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:09.102 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:09.102 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:09.102 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:09.102 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:09.102 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:09.102 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:09.102 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:09.102 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:09.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1944055451 0 0) 0x7fe654006590 con 0x7fe654002830 2024-09-16T14:58:09.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.084+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe65400a0e0 con 0x7fe654002830 2024-09-16T14:58:09.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.088+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe65c001a90 con 0x7fe654002830 2024-09-16T14:58:09.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.088+0000 7fe662ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:09.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.088+0000 7fe662ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:58:09.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.088+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 >> v1:172.21.15.189:6789/0 conn(0x7fe654002830 legacy=0x7fe654004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:09.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.088+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 >> v1:172.21.15.189:6789/0 conn(0x7fe654002830 legacy=0x7fe654004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:09.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.092+0000 7fe6744d2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:58:09.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.092+0000 7fe6744d2640 1 -- 172.21.15.26:0/1997044417 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe654009d80 con 0x7fe6540090a0 2024-09-16T14:58:09.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.092+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187605204 0 0) 0x7fe654009d80 con 0x7fe6540090a0 2024-09-16T14:58:09.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.092+0000 7fe662ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:09.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.092+0000 7fe662ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:09.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.092+0000 7fe662ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:09.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.092+0000 7fe662ffd640 10 monclient(hunting): my global_id is 304180 2024-09-16T14:58:09.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.092+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe654006590 con 0x7fe6540090a0 2024-09-16T14:58:09.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1481972246 0 0) 0x7fe654006590 con 0x7fe6540090a0 2024-09-16T14:58:09.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe654009d80 con 0x7fe6540090a0 2024-09-16T14:58:09.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe65c0037a0 con 0x7fe6540090a0 2024-09-16T14:58:09.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe662ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:09.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe662ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:58:09.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe662ffd640 10 monclient: dump: 2024-09-16T14:58:09.104 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:09.104 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:09.104 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:09.104 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:09.104 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:09.105 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:09.105 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:09.105 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:09.105 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:09.105 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:09.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1310705601 0 0) 0x7fe654009d80 con 0x7fe6540090a0 2024-09-16T14:58:09.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe65400c290 con 0x7fe6540090a0 2024-09-16T14:58:09.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3869 (unknown 0 0 2180536156) 0x7fe65c0057d0 con 0x7fe6540090a0 2024-09-16T14:58:09.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe662ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:58:09.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe662ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:58:09.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 >> v1:172.21.15.189:6789/0 conn(0x7fe6540090a0 legacy=0x7fe654009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:09.106 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:58:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:58:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:58:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:58:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 148, 2024-09-16T14:58:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:58:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:58:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:58:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:58:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:58:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:58:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:58:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:58:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:58:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:58:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:58:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:58:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:58:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:58:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:58:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:58:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:58:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:58:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:58:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:58:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:58:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:58:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:58:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:58:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:58:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:58:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:58:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:58:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:58:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:58:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:58:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:58:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:58:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:58:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:58:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:58:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:58:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:58:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:58:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:58:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:58:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:58:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:58:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:58:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:58:09.116 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:58:09.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe662ffd640 1 -- 172.21.15.26:0/1997044417 >> v1:172.21.15.189:6789/0 conn(0x7fe6540090a0 legacy=0x7fe654009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:09.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe6744d2640 1 -- 172.21.15.26:0/1997044417 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe650036080 legacy=0x7fe650038540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:09.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe6744d2640 10 monclient: shutdown 2024-09-16T14:58:09.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe6744d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:09.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe6744d2640 1 -- 172.21.15.26:0/1997044417 >> v1:172.21.15.26:6790/0 conn(0x7fe66c132650 legacy=0x7fe66c06ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:09.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe672a6f640 1 -- 172.21.15.26:0/1997044417 reap_dead start 2024-09-16T14:58:09.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe672a6f640 1 -- 172.21.15.26:0/1997044417 reap_dead start 2024-09-16T14:58:09.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe6744d2640 1 -- 172.21.15.26:0/1997044417 shutdown_connections 2024-09-16T14:58:09.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe6744d2640 1 -- 172.21.15.26:0/1997044417 >> 172.21.15.26:0/1997044417 conn(0x7fe66c09fb50 msgr2=0x7fe66c0a2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:09.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe6744d2640 1 -- 172.21.15.26:0/1997044417 shutdown_connections 2024-09-16T14:58:09.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:09.096+0000 7fe6744d2640 1 -- 172.21.15.26:0/1997044417 wait complete. 2024-09-16T14:58:09.118 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:58:09.118 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:58:29.120 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:58:29.120 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T14:58:29.121 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T14:58:29.121 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T14:58:29.124 INFO:tasks.ceph.mon.a:Started 2024-09-16T14:58:29.125 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:58:29.125 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:58:29.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.251+0000 7fb79d7ea640 10 monclient: get_monmap_and_config 2024-09-16T14:58:29.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.251+0000 7fb79d7ea640 10 monclient: build_initial_monmap 2024-09-16T14:58:29.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.251+0000 7fb79d7ea640 10 monclient: monmap: 2024-09-16T14:58:29.254 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:29.254 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.254 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:58:29.256097+0000 2024-09-16T14:58:29.254 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:58:29.256097+0000 2024-09-16T14:58:29.254 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:29.254 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.254 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:29.254 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:58:29.254 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:58:29.254 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.251+0000 7fb79d7ea640 1 Processor -- start 2024-09-16T14:58:29.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.251+0000 7fb79d7ea640 1 -- start start 2024-09-16T14:58:29.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.251+0000 7fb79d7ea640 10 monclient: init 2024-09-16T14:58:29.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.251+0000 7fb79d7ea640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:29.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.251+0000 7fb79d7ea640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:58:29.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.251+0000 7fb79d7ea640 10 monclient(hunting): picked mon.noname-c con 0x7fb7980a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:29.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.251+0000 7fb79d7ea640 10 monclient(hunting): picked mon.noname-a con 0x7fb7980a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:29.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.251+0000 7fb79d7ea640 10 monclient(hunting): picked mon.noname-b con 0x7fb7980a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:29.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.251+0000 7fb79d7ea640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7980a3250 con 0x7fb7980a0000 2024-09-16T14:58:29.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.251+0000 7fb79d7ea640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb79809f600 con 0x7fb7980a1d10 2024-09-16T14:58:29.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.251+0000 7fb79d7ea640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7980a3830 con 0x7fb7980a2800 2024-09-16T14:58:29.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.251+0000 7fb79d7ea640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:29.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.251+0000 7fb79d7ea640 10 monclient(hunting): authenticate will time out at 1933.375932s 2024-09-16T14:58:29.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7977fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb7980a2800 0x7fb79813d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60680/0 (socket says 172.21.15.26:60680) 2024-09-16T14:58:29.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb78e7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb7980a0000 0x7fb798139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60922/0 (socket says 172.21.15.26:60922) 2024-09-16T14:58:29.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7977fe640 1 -- 172.21.15.26:0/1767350180 learned_addr learned my addr 172.21.15.26:0/1767350180 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:29.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 1 -- 172.21.15.26:0/1767350180 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616097588 0 0) 0x7fb79809f600 con 0x7fb7980a1d10 2024-09-16T14:58:29.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 10 monclient(hunting): my global_id is 304183 2024-09-16T14:58:29.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 1 -- 172.21.15.26:0/1767350180 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb784003530 con 0x7fb7980a1d10 2024-09-16T14:58:29.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 1 -- 172.21.15.26:0/1767350180 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 111645926 0 0) 0x7fb784003530 con 0x7fb7980a1d10 2024-09-16T14:58:29.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 1 -- 172.21.15.26:0/1767350180 >> v1:172.21.15.26:6790/0 conn(0x7fb7980a2800 legacy=0x7fb79813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 1 -- 172.21.15.26:0/1767350180 >> v1:172.21.15.26:6789/0 conn(0x7fb7980a0000 legacy=0x7fb798139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 10 monclient: _finish_hunting 0 2024-09-16T14:58:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 1 monclient: found mon.noname-c 2024-09-16T14:58:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T14:58:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 1 -- 172.21.15.26:0/1767350180 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79809ed60 con 0x7fb7980a1d10 2024-09-16T14:58:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 10 monclient: _finish_auth 0 2024-09-16T14:58:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 10 monclient: _check_auth_tickets 2024-09-16T14:58:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 1 -- 172.21.15.26:0/1767350180 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb788001620 con 0x7fb7980a1d10 2024-09-16T14:58:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:58:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb7967fc640 10 monclient: dump: 2024-09-16T14:58:29.262 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.262 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.262 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.262 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.262 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.262 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.262 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.262 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.262 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.262 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb79d7ea640 5 monclient: authenticate success, global_id 304183 2024-09-16T14:58:29.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.255+0000 7fb79d7ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:29.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb7967fc640 1 -- 172.21.15.26:0/1767350180 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb788001840 con 0x7fb7980a1d10 2024-09-16T14:58:29.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb7967fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:29.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb7967fc640 1 -- 172.21.15.26:0/1767350180 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb788001c50 con 0x7fb7980a1d10 2024-09-16T14:58:29.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 10 monclient: get_monmap_and_config success 2024-09-16T14:58:29.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 4 set_mon_vals no callback set 2024-09-16T14:58:29.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb7967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb7967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:58:29.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb7967fc640 10 monclient: dump: 2024-09-16T14:58:29.263 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.263 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.263 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.263 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.263 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.263 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.263 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.263 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.264 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.264 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 10 monclient: shutdown 2024-09-16T14:58:29.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:29.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 1 -- 172.21.15.26:0/1767350180 >> v1:172.21.15.189:6789/0 conn(0x7fb7980a1d10 legacy=0x7fb7980a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 1 -- 172.21.15.26:0/1767350180 shutdown_connections 2024-09-16T14:58:29.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 1 -- 172.21.15.26:0/1767350180 >> 172.21.15.26:0/1767350180 conn(0x7fb7980a7f30 msgr2=0x7fb7980a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:29.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 1 -- 172.21.15.26:0/1767350180 shutdown_connections 2024-09-16T14:58:29.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 1 -- 172.21.15.26:0/1767350180 wait complete. 2024-09-16T14:58:29.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 10 monclient: build_initial_monmap 2024-09-16T14:58:29.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 10 monclient: monmap: 2024-09-16T14:58:29.264 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:29.264 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:58:29.265 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:58:29.265 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:58:29.265 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:29.265 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:58:29.265 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:29.265 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:58:29.265 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:58:29.265 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 1 Processor -- start 2024-09-16T14:58:29.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 1 -- start start 2024-09-16T14:58:29.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 10 monclient: init 2024-09-16T14:58:29.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:29.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:58:29.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 10 monclient(hunting): picked mon.noname-c con 0x7fb7980a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:29.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 10 monclient(hunting): picked mon.noname-b con 0x7fb7980a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:29.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 10 monclient(hunting): picked mon.noname-a con 0x7fb7980a2800 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:29.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7980a68c0 con 0x7fb7980a2800 2024-09-16T14:58:29.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7981340b0 con 0x7fb7980a1d10 2024-09-16T14:58:29.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7981312c0 con 0x7fb7980a0000 2024-09-16T14:58:29.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:29.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 10 monclient(hunting): authenticate will time out at 1933.382721s 2024-09-16T14:58:29.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb796ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb7980a0000 0x7fb7981304a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60684/0 (socket says 172.21.15.26:60684) 2024-09-16T14:58:29.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb796ffd640 1 -- 172.21.15.26:0/1313992547 learned_addr learned my addr 172.21.15.26:0/1313992547 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:29.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 1 -- 172.21.15.26:0/1313992547 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545206886 0 0) 0x7fb7981312c0 con 0x7fb7980a0000 2024-09-16T14:58:29.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 10 monclient(hunting): my global_id is 284438 2024-09-16T14:58:29.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 1 -- 172.21.15.26:0/1313992547 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb778003530 con 0x7fb7980a0000 2024-09-16T14:58:29.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 1 -- 172.21.15.26:0/1313992547 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2334879693 0 0) 0x7fb778003530 con 0x7fb7980a0000 2024-09-16T14:58:29.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 1 -- 172.21.15.26:0/1313992547 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7981312c0 con 0x7fb7980a0000 2024-09-16T14:58:29.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 1 -- 172.21.15.26:0/1313992547 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb788001c10 con 0x7fb7980a0000 2024-09-16T14:58:29.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:58:29.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 10 monclient(hunting): dump: 2024-09-16T14:58:29.270 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.270 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.270 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.270 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.270 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.271 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.271 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.271 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.271 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.271 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:58:29.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 1 -- 172.21.15.26:0/1313992547 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2789567965 0 0) 0x7fb7981312c0 con 0x7fb7980a0000 2024-09-16T14:58:29.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 1 -- 172.21.15.26:0/1313992547 >> v1:172.21.15.189:6789/0 conn(0x7fb7980a1d10 legacy=0x7fb798130bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 1 -- 172.21.15.26:0/1313992547 >> v1:172.21.15.26:6789/0 conn(0x7fb7980a2800 legacy=0x7fb7981d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:58:29.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 1 monclient: found mon.c 2024-09-16T14:58:29.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:29.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:29.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 1 -- 172.21.15.26:0/1313992547 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79809e500 con 0x7fb7980a0000 2024-09-16T14:58:29.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 10 monclient: _finish_auth 0 2024-09-16T14:58:29.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:58:29.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb78ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:29.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 5 monclient: authenticate success, global_id 284438 2024-09-16T14:58:29.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 10 monclient: _renew_subs 2024-09-16T14:58:29.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:29.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.259+0000 7fb79d7ea640 1 -- 172.21.15.26:0/1313992547 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb798133810 con 0x7fb7980a0000 2024-09-16T14:58:29.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb79d7ea640 10 monclient: _renew_subs 2024-09-16T14:58:29.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb79d7ea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:29.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb79d7ea640 1 -- 172.21.15.26:0/1313992547 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7981357a0 con 0x7fb7980a0000 2024-09-16T14:58:29.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb78ffff640 1 -- 172.21.15.26:0/1313992547 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb788003970 con 0x7fb7980a0000 2024-09-16T14:58:29.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb78ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:29.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb78ffff640 1 -- 172.21.15.26:0/1313992547 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb788003c80 con 0x7fb7980a0000 2024-09-16T14:58:29.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb78ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb78ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:29.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb78ffff640 10 monclient: dump: 2024-09-16T14:58:29.273 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.273 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.273 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.273 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.273 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.273 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.273 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.273 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.274 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.274 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb795ffb640 4 set_mon_vals no callback set 2024-09-16T14:58:29.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb78ffff640 1 -- 172.21.15.26:0/1313992547 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb788003de0 con 0x7fb7980a0000 2024-09-16T14:58:29.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb79d7ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:29.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb79d7ea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:29.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb79d7ea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:29.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb79d7ea640 1 -- 172.21.15.26:0/1313992547 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb77c005050 con 0x7fb7980a0000 2024-09-16T14:58:29.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb78ffff640 1 -- 172.21.15.26:0/1313992547 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fb788043f50 con 0x7fb7980a0000 2024-09-16T14:58:29.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.263+0000 7fb78e7fc640 10 monclient: get_auth_request con 0x7fb778036100 auth_method 0 2024-09-16T14:58:29.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.267+0000 7fb78ffff640 1 -- 172.21.15.26:0/1313992547 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb788014900 con 0x7fb7980a0000 2024-09-16T14:58:29.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.267+0000 7fb78ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:29.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.267+0000 7fb78ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:58:29.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.483+0000 7fb79d7ea640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:58:29.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.483+0000 7fb79d7ea640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:58:29.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.483+0000 7fb79d7ea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:29.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.483+0000 7fb79d7ea640 1 -- 172.21.15.26:0/1313992547 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb77c005880 con 0x7fb7980a0000 2024-09-16T14:58:29.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.483+0000 7fb78ffff640 1 -- 172.21.15.26:0/1313992547 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1069234958) 0x7fb7880146d0 con 0x7fb7980a0000 2024-09-16T14:58:29.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.483+0000 7fb78ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:58:29.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.483+0000 7fb78ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:58:29.485 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:58:29.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.483+0000 7fb79d7ea640 1 -- 172.21.15.26:0/1313992547 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb778036100 legacy=0x7fb7780385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.483+0000 7fb79d7ea640 10 monclient: shutdown 2024-09-16T14:58:29.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.483+0000 7fb79d7ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:29.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.483+0000 7fb79d7ea640 1 -- 172.21.15.26:0/1313992547 >> v1:172.21.15.26:6790/0 conn(0x7fb7980a0000 legacy=0x7fb7981304a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.483+0000 7fb79d7ea640 1 -- 172.21.15.26:0/1313992547 shutdown_connections 2024-09-16T14:58:29.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.483+0000 7fb79d7ea640 1 -- 172.21.15.26:0/1313992547 >> 172.21.15.26:0/1313992547 conn(0x7fb7980a7f30 msgr2=0x7fb79806b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:29.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.483+0000 7fb79d7ea640 1 -- 172.21.15.26:0/1313992547 shutdown_connections 2024-09-16T14:58:29.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.483+0000 7fb79d7ea640 1 -- 172.21.15.26:0/1313992547 wait complete. 2024-09-16T14:58:29.501 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:58:29.501 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:58:29.501 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:58:29.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.607+0000 7ff14df7e640 10 monclient: get_monmap_and_config 2024-09-16T14:58:29.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.607+0000 7ff14df7e640 10 monclient: build_initial_monmap 2024-09-16T14:58:29.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.607+0000 7ff14df7e640 10 monclient: monmap: 2024-09-16T14:58:29.611 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:29.611 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.611 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:58:29.613443+0000 2024-09-16T14:58:29.611 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:58:29.613443+0000 2024-09-16T14:58:29.611 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:29.611 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.611 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:29.612 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:58:29.612 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:58:29.612 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 1 Processor -- start 2024-09-16T14:58:29.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 1 -- start start 2024-09-16T14:58:29.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 10 monclient: init 2024-09-16T14:58:29.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:29.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:58:29.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 10 monclient(hunting): picked mon.noname-b con 0x7ff14809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:29.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 10 monclient(hunting): picked mon.noname-c con 0x7ff148132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:29.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 10 monclient(hunting): picked mon.noname-a con 0x7ff148138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:29.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14809d260 con 0x7ff148138ff0 2024-09-16T14:58:29.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff148132a80 con 0x7ff148132650 2024-09-16T14:58:29.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14809cf60 con 0x7ff14809cb80 2024-09-16T14:58:29.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:29.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 10 monclient(hunting): authenticate will time out at 1933.733410s 2024-09-16T14:58:29.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1477fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff14809cb80 0x7ff1480717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60686/0 (socket says 172.21.15.26:60686) 2024-09-16T14:58:29.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1477fe640 1 -- 172.21.15.26:0/1813123034 learned_addr learned my addr 172.21.15.26:0/1813123034 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:29.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff147fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff148138ff0 0x7ff14813b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60950/0 (socket says 172.21.15.26:60950) 2024-09-16T14:58:29.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 1 -- 172.21.15.26:0/1813123034 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894690236 0 0) 0x7ff14809d260 con 0x7ff148138ff0 2024-09-16T14:58:29.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient(hunting): my global_id is 314097 2024-09-16T14:58:29.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 1 -- 172.21.15.26:0/1813123034 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff130003530 con 0x7ff148138ff0 2024-09-16T14:58:29.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 1 -- 172.21.15.26:0/1813123034 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760144126 0 0) 0x7ff14809cf60 con 0x7ff14809cb80 2024-09-16T14:58:29.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient(hunting): my global_id is 284441 2024-09-16T14:58:29.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 1 -- 172.21.15.26:0/1813123034 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff14809d260 con 0x7ff14809cb80 2024-09-16T14:58:29.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 1 -- 172.21.15.26:0/1813123034 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 388600548 0 0) 0x7ff130003530 con 0x7ff148138ff0 2024-09-16T14:58:29.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 1 -- 172.21.15.26:0/1813123034 >> v1:172.21.15.26:6790/0 conn(0x7ff14809cb80 legacy=0x7ff1480717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 1 -- 172.21.15.26:0/1813123034 >> v1:172.21.15.189:6789/0 conn(0x7ff148132650 legacy=0x7ff1481378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient: _finish_hunting 0 2024-09-16T14:58:29.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 1 monclient: found mon.noname-a 2024-09-16T14:58:29.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:29.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:58:29.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 1 -- 172.21.15.26:0/1813123034 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14809f2d0 con 0x7ff148138ff0 2024-09-16T14:58:29.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient: _finish_auth 0 2024-09-16T14:58:29.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient: _check_auth_tickets 2024-09-16T14:58:29.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:29.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 1 -- 172.21.15.26:0/1813123034 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff1400019a0 con 0x7ff148138ff0 2024-09-16T14:58:29.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:58:29.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient: dump: 2024-09-16T14:58:29.623 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.623 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.623 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.623 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.623 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.623 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.623 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.623 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.623 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.623 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 5 monclient: authenticate success, global_id 314097 2024-09-16T14:58:29.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:29.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 1 -- 172.21.15.26:0/1813123034 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff140001c10 con 0x7ff148138ff0 2024-09-16T14:58:29.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:29.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 1 -- 172.21.15.26:0/1813123034 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff140001fa0 con 0x7ff148138ff0 2024-09-16T14:58:29.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 10 monclient: get_monmap_and_config success 2024-09-16T14:58:29.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 4 set_mon_vals no callback set 2024-09-16T14:58:29.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:58:29.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff1467fc640 10 monclient: dump: 2024-09-16T14:58:29.624 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.624 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.624 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.624 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.624 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.624 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.625 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 10 monclient: shutdown 2024-09-16T14:58:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 1 -- 172.21.15.26:0/1813123034 >> v1:172.21.15.26:6789/0 conn(0x7ff148138ff0 legacy=0x7ff14813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 1 -- 172.21.15.26:0/1813123034 shutdown_connections 2024-09-16T14:58:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 1 -- 172.21.15.26:0/1813123034 >> 172.21.15.26:0/1813123034 conn(0x7ff14809fb50 msgr2=0x7ff14809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 1 -- 172.21.15.26:0/1813123034 shutdown_connections 2024-09-16T14:58:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 1 -- 172.21.15.26:0/1813123034 wait complete. 2024-09-16T14:58:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 10 monclient: build_initial_monmap 2024-09-16T14:58:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.611+0000 7ff14df7e640 10 monclient: monmap: 2024-09-16T14:58:29.625 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:29.626 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:58:29.626 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:58:29.626 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:58:29.626 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:29.626 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:58:29.626 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:29.626 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:58:29.626 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:58:29.626 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 1 Processor -- start 2024-09-16T14:58:29.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 1 -- start start 2024-09-16T14:58:29.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 10 monclient: init 2024-09-16T14:58:29.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:29.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:58:29.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 10 monclient(hunting): picked mon.noname-b con 0x7ff14809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:29.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 10 monclient(hunting): picked mon.noname-a con 0x7ff148132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:29.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 10 monclient(hunting): picked mon.noname-c con 0x7ff148138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:29.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14813c500 con 0x7ff148132650 2024-09-16T14:58:29.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff148133700 con 0x7ff14809cb80 2024-09-16T14:58:29.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff148070520 con 0x7ff148138ff0 2024-09-16T14:58:29.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:29.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 10 monclient(hunting): authenticate will time out at 1933.736932s 2024-09-16T14:58:29.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff146ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff148132650 0x7ff1481ccd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60962/0 (socket says 172.21.15.26:60962) 2024-09-16T14:58:29.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff146ffd640 1 -- 172.21.15.26:0/3760618075 learned_addr learned my addr 172.21.15.26:0/3760618075 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:29.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162847493 0 0) 0x7ff14813c500 con 0x7ff148132650 2024-09-16T14:58:29.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): my global_id is 314100 2024-09-16T14:58:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff134003530 con 0x7ff148132650 2024-09-16T14:58:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368309620 0 0) 0x7ff148070520 con 0x7ff148138ff0 2024-09-16T14:58:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): my global_id is 284444 2024-09-16T14:58:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff14813c500 con 0x7ff148138ff0 2024-09-16T14:58:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1191596363 0 0) 0x7ff134003530 con 0x7ff148132650 2024-09-16T14:58:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff148070520 con 0x7ff148132650 2024-09-16T14:58:29.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff13c001920 con 0x7ff148132650 2024-09-16T14:58:29.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:58:29.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): dump: 2024-09-16T14:58:29.629 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.629 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.629 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.629 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.629 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.629 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.629 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.629 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.629 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.629 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:58:29.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858490023 0 0) 0x7ff148133700 con 0x7ff14809cb80 2024-09-16T14:58:29.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): my global_id is 304189 2024-09-16T14:58:29.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff134003530 con 0x7ff14809cb80 2024-09-16T14:58:29.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 265615814 0 0) 0x7ff14813c500 con 0x7ff148138ff0 2024-09-16T14:58:29.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff148133700 con 0x7ff148138ff0 2024-09-16T14:58:29.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff140001960 con 0x7ff148138ff0 2024-09-16T14:58:29.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:29.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): dump: 2024-09-16T14:58:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.631 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3722277055 0 0) 0x7ff148070520 con 0x7ff148132650 2024-09-16T14:58:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 >> v1:172.21.15.26:6790/0 conn(0x7ff148138ff0 legacy=0x7ff1481d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 >> v1:172.21.15.189:6789/0 conn(0x7ff14809cb80 legacy=0x7ff1481c7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient: _finish_hunting 0 2024-09-16T14:58:29.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 monclient: found mon.a 2024-09-16T14:58:29.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:29.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:29.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14809a240 con 0x7ff148132650 2024-09-16T14:58:29.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient: _finish_auth 0 2024-09-16T14:58:29.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient: _check_auth_tickets 2024-09-16T14:58:29.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:29.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 5 monclient: authenticate success, global_id 314100 2024-09-16T14:58:29.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 10 monclient: _renew_subs 2024-09-16T14:58:29.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:29.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 1 -- 172.21.15.26:0/3760618075 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff148132100 con 0x7ff148132650 2024-09-16T14:58:29.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 10 monclient: _renew_subs 2024-09-16T14:58:29.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 1 -- 172.21.15.26:0/3760618075 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff148071410 con 0x7ff148132650 2024-09-16T14:58:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff13c001c70 con 0x7ff148132650 2024-09-16T14:58:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff13c003a70 con 0x7ff148132650 2024-09-16T14:58:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff145ffb640 4 set_mon_vals no callback set 2024-09-16T14:58:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:58:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient: dump: 2024-09-16T14:58:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.633 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.634 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.634 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.634 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.634 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:29.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff14df7e640 1 -- 172.21.15.26:0/3760618075 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff128006590 con 0x7ff128002830 2024-09-16T14:58:29.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff13c003cd0 con 0x7ff148132650 2024-09-16T14:58:29.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7ff13c043cb0 con 0x7ff148132650 2024-09-16T14:58:29.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2705636866 0 0) 0x7ff128006590 con 0x7ff128002830 2024-09-16T14:58:29.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient(hunting): my global_id is 314103 2024-09-16T14:58:29.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff148070520 con 0x7ff128002830 2024-09-16T14:58:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1951532194 0 0) 0x7ff148070520 con 0x7ff128002830 2024-09-16T14:58:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff128006590 con 0x7ff128002830 2024-09-16T14:58:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff138001730 con 0x7ff128002830 2024-09-16T14:58:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:58:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient: dump: 2024-09-16T14:58:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.636 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.636 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.636 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff147fff640 10 monclient: get_auth_request con 0x7ff1340363a0 auth_method 0 2024-09-16T14:58:29.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1761412133 0 0) 0x7ff128006590 con 0x7ff128002830 2024-09-16T14:58:29.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff12800a0e0 con 0x7ff128002830 2024-09-16T14:58:29.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ff138001aa0 con 0x7ff128002830 2024-09-16T14:58:29.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:29.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:58:29.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 >> v1:172.21.15.26:6789/0 conn(0x7ff128002830 legacy=0x7ff128004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.615+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 >> v1:172.21.15.26:6789/0 conn(0x7ff128002830 legacy=0x7ff128004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:29.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff14df7e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:58:29.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff14df7e640 1 -- 172.21.15.26:0/3760618075 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff128009da0 con 0x7ff1280090c0 2024-09-16T14:58:29.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858764620 0 0) 0x7ff128009da0 con 0x7ff1280090c0 2024-09-16T14:58:29.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 10 monclient(hunting): my global_id is 314106 2024-09-16T14:58:29.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff128006590 con 0x7ff1280090c0 2024-09-16T14:58:29.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1788172197 0 0) 0x7ff128006590 con 0x7ff1280090c0 2024-09-16T14:58:29.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff128009da0 con 0x7ff1280090c0 2024-09-16T14:58:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff138004990 con 0x7ff1280090c0 2024-09-16T14:58:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:58:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 10 monclient: dump: 2024-09-16T14:58:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.639 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.640 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1472276190 0 0) 0x7ff128009da0 con 0x7ff1280090c0 2024-09-16T14:58:29.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff12800c290 con 0x7ff1280090c0 2024-09-16T14:58:29.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3892336873) 0x7ff138005f30 con 0x7ff1280090c0 2024-09-16T14:58:29.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:58:29.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:58:29.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 >> v1:172.21.15.26:6789/0 conn(0x7ff1280090c0 legacy=0x7ff128009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff12ffff640 1 -- 172.21.15.26:0/3760618075 >> v1:172.21.15.26:6789/0 conn(0x7ff1280090c0 legacy=0x7ff128009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:29.640 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:58:29.640 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:58:29.641 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:58:29.641 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:58:29.641 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 150, 2024-09-16T14:58:29.641 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:58:29.641 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:58:29.641 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:58:29.641 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:58:29.641 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:29.641 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:58:29.641 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:29.641 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:58:29.641 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:58:29.641 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:29.642 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:29.642 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:29.642 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:29.642 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:29.642 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:29.642 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:29.642 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:29.642 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:29.642 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:29.642 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:29.642 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:58:29.642 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:58:29.642 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:29.642 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:29.642 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:29.643 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:29.643 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:29.643 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:29.643 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:29.643 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:29.643 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:29.643 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:29.643 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.643 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:58:29.643 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:58:29.643 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:58:29.643 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:58:29.644 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:58:29.644 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:58:29.644 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:29.644 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:29.644 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:58:29.644 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:58:29.644 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:58:29.644 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:58:29.644 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:58:29.644 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:58:29.645 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:58:29.645 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:29.645 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:58:29.645 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:29.645 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:29.645 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:29.645 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:29.645 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:29.645 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:29.645 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:29.645 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:29.645 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:29.645 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:29.645 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:29.646 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:58:29.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.646 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:58:29.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.646 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:58:29.646 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:58:29.646 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:29.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:29.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:29.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:58:29.647 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:29.647 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.647 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:58:29.647 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:58:29.647 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:29.647 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:29.647 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:29.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.648 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:58:29.648 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:58:29.648 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:29.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:29.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:29.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:58:29.648 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:29.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.648 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:58:29.649 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:58:29.649 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:29.649 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:29.649 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:29.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.649 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:58:29.649 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:58:29.649 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:29.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:29.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:29.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:58:29.649 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:29.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:58:29.650 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:58:29.650 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:29.650 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:29.650 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:29.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.650 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:58:29.650 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:58:29.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.651 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:29.651 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:29.651 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:58:29.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.651 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:29.651 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:58:29.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.651 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:29.651 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:29.651 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:58:29.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.651 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.652 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:58:29.652 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:58:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff14df7e640 1 -- 172.21.15.26:0/3760618075 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff1340363a0 legacy=0x7ff134038860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff14df7e640 10 monclient: shutdown 2024-09-16T14:58:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff14df7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff14df7e640 1 -- 172.21.15.26:0/3760618075 >> v1:172.21.15.26:6789/0 conn(0x7ff148132650 legacy=0x7ff1481ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff147fff640 1 -- 172.21.15.26:0/3760618075 reap_dead start 2024-09-16T14:58:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff147fff640 1 -- 172.21.15.26:0/3760618075 reap_dead start 2024-09-16T14:58:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff14df7e640 1 -- 172.21.15.26:0/3760618075 shutdown_connections 2024-09-16T14:58:29.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff14df7e640 1 -- 172.21.15.26:0/3760618075 >> 172.21.15.26:0/3760618075 conn(0x7ff14809fb50 msgr2=0x7ff148130090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:29.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff14df7e640 1 -- 172.21.15.26:0/3760618075 shutdown_connections 2024-09-16T14:58:29.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.623+0000 7ff14df7e640 1 -- 172.21.15.26:0/3760618075 wait complete. 2024-09-16T14:58:29.654 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:58:29.654 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:58:29.654 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:58:29.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 10 monclient: get_monmap_and_config 2024-09-16T14:58:29.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 10 monclient: build_initial_monmap 2024-09-16T14:58:29.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 10 monclient: monmap: 2024-09-16T14:58:29.764 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:29.764 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.764 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:58:29.766334+0000 2024-09-16T14:58:29.764 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:58:29.766334+0000 2024-09-16T14:58:29.764 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:29.764 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.764 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:29.764 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:58:29.765 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:58:29.765 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 1 Processor -- start 2024-09-16T14:58:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 1 -- start start 2024-09-16T14:58:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 10 monclient: init 2024-09-16T14:58:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:58:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 10 monclient(hunting): picked mon.noname-b con 0x7f69b409a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 10 monclient(hunting): picked mon.noname-a con 0x7f69b409d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 10 monclient(hunting): picked mon.noname-c con 0x7f69b409b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b409dc60 con 0x7f69b409d1b0 2024-09-16T14:58:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b4099ae0 con 0x7f69b409b040 2024-09-16T14:58:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b409b8f0 con 0x7f69b409a550 2024-09-16T14:58:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:29.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 10 monclient(hunting): authenticate will time out at 1933.886421s 2024-09-16T14:58:29.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b37fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f69b409d1b0 0x7f69b4136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60990/0 (socket says 172.21.15.26:60990) 2024-09-16T14:58:29.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b37fe640 1 -- 172.21.15.26:0/3777912187 learned_addr learned my addr 172.21.15.26:0/3777912187 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:29.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 1 -- 172.21.15.26:0/3777912187 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170143879 0 0) 0x7f69b409b8f0 con 0x7f69b409a550 2024-09-16T14:58:29.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient(hunting): my global_id is 284447 2024-09-16T14:58:29.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 1 -- 172.21.15.26:0/3777912187 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f699c003530 con 0x7f69b409a550 2024-09-16T14:58:29.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 1 -- 172.21.15.26:0/3777912187 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862502382 0 0) 0x7f69b409dc60 con 0x7f69b409d1b0 2024-09-16T14:58:29.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient(hunting): my global_id is 314109 2024-09-16T14:58:29.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 1 -- 172.21.15.26:0/3777912187 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b409b8f0 con 0x7f69b409d1b0 2024-09-16T14:58:29.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 1 -- 172.21.15.26:0/3777912187 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 617901489 0 0) 0x7f699c003530 con 0x7f69b409a550 2024-09-16T14:58:29.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 1 -- 172.21.15.26:0/3777912187 >> v1:172.21.15.189:6789/0 conn(0x7f69b409b040 legacy=0x7f69b413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 1 -- 172.21.15.26:0/3777912187 >> v1:172.21.15.26:6789/0 conn(0x7f69b409d1b0 legacy=0x7f69b4136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:58:29.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 1 monclient: found mon.noname-b 2024-09-16T14:58:29.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:29.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:58:29.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 1 -- 172.21.15.26:0/3777912187 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b409bbd0 con 0x7f69b409a550 2024-09-16T14:58:29.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:58:29.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:58:29.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:29.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 5 monclient: authenticate success, global_id 284447 2024-09-16T14:58:29.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:29.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 1 -- 172.21.15.26:0/3777912187 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f69a4001620 con 0x7f69b409a550 2024-09-16T14:58:29.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:58:29.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient: dump: 2024-09-16T14:58:29.775 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.775 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.775 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.775 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.775 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.775 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.775 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.775 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.775 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.776 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:29.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 1 -- 172.21.15.26:0/3777912187 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69a4001840 con 0x7f69b409a550 2024-09-16T14:58:29.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:29.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 1 -- 172.21.15.26:0/3777912187 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f69a4001c10 con 0x7f69b409a550 2024-09-16T14:58:29.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 10 monclient: get_monmap_and_config success 2024-09-16T14:58:29.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 4 set_mon_vals no callback set 2024-09-16T14:58:29.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:29.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69b2ffd640 10 monclient: dump: 2024-09-16T14:58:29.776 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.776 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.776 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.777 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.777 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.777 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.777 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.777 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.777 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.777 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 10 monclient: shutdown 2024-09-16T14:58:29.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:29.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 1 -- 172.21.15.26:0/3777912187 >> v1:172.21.15.26:6790/0 conn(0x7f69b409a550 legacy=0x7f69b409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 1 -- 172.21.15.26:0/3777912187 shutdown_connections 2024-09-16T14:58:29.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 1 -- 172.21.15.26:0/3777912187 >> 172.21.15.26:0/3777912187 conn(0x7f69b409fb50 msgr2=0x7f69b409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:29.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 1 -- 172.21.15.26:0/3777912187 shutdown_connections 2024-09-16T14:58:29.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.763+0000 7f69baa60640 1 -- 172.21.15.26:0/3777912187 wait complete. 2024-09-16T14:58:29.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 10 monclient: build_initial_monmap 2024-09-16T14:58:29.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 10 monclient: monmap: 2024-09-16T14:58:29.778 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:29.778 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:58:29.778 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:58:29.778 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:58:29.778 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:29.778 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:58:29.778 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:29.778 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:58:29.778 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:58:29.778 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 1 Processor -- start 2024-09-16T14:58:29.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 1 -- start start 2024-09-16T14:58:29.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 10 monclient: init 2024-09-16T14:58:29.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:29.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:58:29.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 10 monclient(hunting): picked mon.noname-c con 0x7f69b409a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:29.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 10 monclient(hunting): picked mon.noname-b con 0x7f69b409b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:29.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 10 monclient(hunting): picked mon.noname-a con 0x7f69b409d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:29.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b41198f0 con 0x7f69b409d1b0 2024-09-16T14:58:29.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b409a240 con 0x7f69b409b040 2024-09-16T14:58:29.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b406ca50 con 0x7f69b409a550 2024-09-16T14:58:29.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:29.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 10 monclient(hunting): authenticate will time out at 1933.889485s 2024-09-16T14:58:29.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b3fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f69b409a550 0x7f69b4138290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60706/0 (socket says 172.21.15.26:60706) 2024-09-16T14:58:29.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b3fff640 1 -- 172.21.15.26:0/2085247596 learned_addr learned my addr 172.21.15.26:0/2085247596 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:29.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956449767 0 0) 0x7f69b406ca50 con 0x7f69b409a550 2024-09-16T14:58:29.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): my global_id is 284450 2024-09-16T14:58:29.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69a0003530 con 0x7f69b409a550 2024-09-16T14:58:29.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765785602 0 0) 0x7f69b41198f0 con 0x7f69b409d1b0 2024-09-16T14:58:29.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): my global_id is 314112 2024-09-16T14:58:29.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b406ca50 con 0x7f69b409d1b0 2024-09-16T14:58:29.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4065750149 0 0) 0x7f69a0003530 con 0x7f69b409a550 2024-09-16T14:58:29.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69b41198f0 con 0x7f69b409a550 2024-09-16T14:58:29.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f69a40015e0 con 0x7f69b409a550 2024-09-16T14:58:29.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:58:29.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): dump: 2024-09-16T14:58:29.781 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.781 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.782 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.782 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.782 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.782 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.782 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.782 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.782 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.782 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:58:29.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 47198118 0 0) 0x7f69b406ca50 con 0x7f69b409d1b0 2024-09-16T14:58:29.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69a0003530 con 0x7f69b409d1b0 2024-09-16T14:58:29.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934626123 0 0) 0x7f69b409a240 con 0x7f69b409b040 2024-09-16T14:58:29.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): my global_id is 304195 2024-09-16T14:58:29.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b406ca50 con 0x7f69b409b040 2024-09-16T14:58:29.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f69ac001b80 con 0x7f69b409d1b0 2024-09-16T14:58:29.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:58:29.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): dump: 2024-09-16T14:58:29.783 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.783 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.783 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.783 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.783 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.783 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.783 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.784 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.784 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.784 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3673591889 0 0) 0x7f69b41198f0 con 0x7f69b409a550 2024-09-16T14:58:29.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 >> v1:172.21.15.189:6789/0 conn(0x7f69b409b040 legacy=0x7f69b41389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 >> v1:172.21.15.26:6789/0 conn(0x7f69b409d1b0 legacy=0x7f69b41cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:58:29.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 monclient: found mon.c 2024-09-16T14:58:29.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:29.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:29.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b41390b0 con 0x7f69b409a550 2024-09-16T14:58:29.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient: _finish_auth 0 2024-09-16T14:58:29.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:58:29.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:29.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 5 monclient: authenticate success, global_id 284450 2024-09-16T14:58:29.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 10 monclient: _renew_subs 2024-09-16T14:58:29.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:29.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 1 -- 172.21.15.26:0/2085247596 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b41399d0 con 0x7f69b409a550 2024-09-16T14:58:29.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 10 monclient: _renew_subs 2024-09-16T14:58:29.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:29.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 1 -- 172.21.15.26:0/2085247596 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b4139710 con 0x7f69b409a550 2024-09-16T14:58:29.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69a4003970 con 0x7f69b409a550 2024-09-16T14:58:29.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:29.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f69a4003d00 con 0x7f69b409a550 2024-09-16T14:58:29.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:29.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient: dump: 2024-09-16T14:58:29.786 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.786 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.786 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.786 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.786 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.786 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.786 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.786 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.786 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.786 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b27fc640 4 set_mon_vals no callback set 2024-09-16T14:58:29.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:29.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69baa60640 1 -- 172.21.15.26:0/2085247596 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6994006590 con 0x7f6994002830 2024-09-16T14:58:29.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f69a4003e60 con 0x7f69b409a550 2024-09-16T14:58:29.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f69a4043f40 con 0x7f69b409a550 2024-09-16T14:58:29.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383383609 0 0) 0x7f6994006590 con 0x7f6994002830 2024-09-16T14:58:29.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient(hunting): my global_id is 284453 2024-09-16T14:58:29.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b41198f0 con 0x7f6994002830 2024-09-16T14:58:29.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2896511808 0 0) 0x7f69b41198f0 con 0x7f6994002830 2024-09-16T14:58:29.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6994006590 con 0x7f6994002830 2024-09-16T14:58:29.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f69a8001140 con 0x7f6994002830 2024-09-16T14:58:29.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b8ffd640 10 monclient: get_auth_request con 0x7f69a0036440 auth_method 0 2024-09-16T14:58:29.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:29.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 10 monclient: dump: 2024-09-16T14:58:29.788 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.788 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.788 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.788 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.788 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.788 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.788 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.788 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.788 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.788 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2650004998 0 0) 0x7f6994006590 con 0x7f6994002830 2024-09-16T14:58:29.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.767+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f699400a0e0 con 0x7f6994002830 2024-09-16T14:58:29.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.771+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f69a80013f0 con 0x7f6994002830 2024-09-16T14:58:29.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.771+0000 7f69b0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:29.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.771+0000 7f69b0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:58:29.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.771+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 >> v1:172.21.15.26:6790/0 conn(0x7f6994002830 legacy=0x7f6994004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.771+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 >> v1:172.21.15.26:6790/0 conn(0x7f6994002830 legacy=0x7f6994004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:29.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69baa60640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:58:29.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69baa60640 1 -- 172.21.15.26:0/2085247596 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6994009da0 con 0x7f69940090c0 2024-09-16T14:58:29.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419074724 0 0) 0x7f6994009da0 con 0x7f69940090c0 2024-09-16T14:58:29.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 10 monclient(hunting): my global_id is 284456 2024-09-16T14:58:29.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6994006590 con 0x7f69940090c0 2024-09-16T14:58:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3994201668 0 0) 0x7f6994006590 con 0x7f69940090c0 2024-09-16T14:58:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6994009da0 con 0x7f69940090c0 2024-09-16T14:58:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f69a8004990 con 0x7f69940090c0 2024-09-16T14:58:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 10 monclient: dump: 2024-09-16T14:58:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.791 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.792 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.792 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 944301747 0 0) 0x7f6994009da0 con 0x7f69940090c0 2024-09-16T14:58:29.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f699400c290 con 0x7f69940090c0 2024-09-16T14:58:29.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 3328857222) 0x7f69a8004c40 con 0x7f69940090c0 2024-09-16T14:58:29.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:58:29.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:58:29.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 >> v1:172.21.15.26:6790/0 conn(0x7f69940090c0 legacy=0x7f6994009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b0ff9640 1 -- 172.21.15.26:0/2085247596 >> v1:172.21.15.26:6790/0 conn(0x7f69940090c0 legacy=0x7f6994009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:29.792 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:58:29.793 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:58:29.793 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:58:29.793 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:58:29.793 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 150, 2024-09-16T14:58:29.793 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:58:29.793 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:58:29.793 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:58:29.793 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:58:29.793 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:29.793 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:58:29.793 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:29.793 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:58:29.793 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:58:29.794 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:29.794 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:29.794 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:29.794 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:29.794 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:29.794 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:29.794 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:29.794 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:29.794 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:29.794 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:29.794 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:29.794 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:58:29.794 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:58:29.794 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:29.794 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:29.795 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:29.795 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:29.795 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:29.795 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:29.795 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:29.795 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:29.795 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:29.795 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:29.795 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.795 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:58:29.795 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:58:29.795 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:58:29.795 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:58:29.796 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:58:29.796 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:58:29.796 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:29.796 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:29.796 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:58:29.796 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:58:29.796 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:58:29.796 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:58:29.796 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:58:29.796 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:58:29.796 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:58:29.796 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:29.796 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:58:29.797 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:29.797 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:29.797 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:29.797 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:29.797 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:29.797 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:29.797 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:29.797 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:29.797 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:29.797 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:29.797 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:29.797 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:58:29.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.798 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:58:29.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.798 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:58:29.798 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:58:29.798 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:29.798 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:29.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.798 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:29.798 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:58:29.798 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:29.798 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.798 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.799 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:58:29.799 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:58:29.799 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:29.799 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:29.799 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:29.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.799 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:58:29.799 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:58:29.799 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:29.800 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:29.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.800 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:29.800 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:58:29.800 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:29.800 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.800 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.800 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:58:29.800 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:58:29.800 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:29.800 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:29.800 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:29.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.801 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:58:29.801 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:58:29.801 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:29.801 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:29.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.801 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:29.801 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:58:29.801 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:29.801 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.801 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.801 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:58:29.802 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:58:29.802 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:29.802 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:29.802 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:29.802 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.802 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.802 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:58:29.802 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:58:29.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.802 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:29.802 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:29.802 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:58:29.803 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.803 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:29.803 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:58:29.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.803 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:29.803 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:29.803 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:58:29.803 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.803 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:29.803 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:58:29.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.803 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:29.803 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:29.803 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:58:29.804 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.804 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:29.804 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:58:29.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.804 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:29.804 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:29.804 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:58:29.804 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.804 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.805 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:58:29.805 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:58:29.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69baa60640 1 -- 172.21.15.26:0/2085247596 >> v1:172.21.15.189:6812/3196161582 conn(0x7f69a0036440 legacy=0x7f69a0038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69baa60640 10 monclient: shutdown 2024-09-16T14:58:29.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69baa60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69baa60640 1 -- 172.21.15.26:0/2085247596 >> v1:172.21.15.26:6790/0 conn(0x7f69b409a550 legacy=0x7f69b4138290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b8ffd640 1 -- 172.21.15.26:0/2085247596 reap_dead start 2024-09-16T14:58:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69b8ffd640 1 -- 172.21.15.26:0/2085247596 reap_dead start 2024-09-16T14:58:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69baa60640 1 -- 172.21.15.26:0/2085247596 shutdown_connections 2024-09-16T14:58:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69baa60640 1 -- 172.21.15.26:0/2085247596 >> 172.21.15.26:0/2085247596 conn(0x7f69b409fb50 msgr2=0x7f69b4133a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69baa60640 1 -- 172.21.15.26:0/2085247596 shutdown_connections 2024-09-16T14:58:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.775+0000 7f69baa60640 1 -- 172.21.15.26:0/2085247596 wait complete. 2024-09-16T14:58:29.807 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:58:29.807 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:58:29.808 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:58:29.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 10 monclient: get_monmap_and_config 2024-09-16T14:58:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 10 monclient: build_initial_monmap 2024-09-16T14:58:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 10 monclient: monmap: 2024-09-16T14:58:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:58:29.922299+0000 2024-09-16T14:58:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:58:29.922299+0000 2024-09-16T14:58:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:58:29.920 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:58:29.920 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 1 Processor -- start 2024-09-16T14:58:29.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 1 -- start start 2024-09-16T14:58:29.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 10 monclient: init 2024-09-16T14:58:29.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:29.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:58:29.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 10 monclient(hunting): picked mon.noname-c con 0x7fecf409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:29.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 10 monclient(hunting): picked mon.noname-a con 0x7fecf4132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:29.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 10 monclient(hunting): picked mon.noname-b con 0x7fecf4138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:29.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf409d260 con 0x7fecf4132650 2024-09-16T14:58:29.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf4132a80 con 0x7fecf409cb80 2024-09-16T14:58:29.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf409cf60 con 0x7fecf4138ff0 2024-09-16T14:58:29.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 10 monclient(hunting): authenticate will time out at 1934.042317s 2024-09-16T14:58:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf8f0d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fecf4138ff0 0x7fecf413b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60742/0 (socket says 172.21.15.26:60742) 2024-09-16T14:58:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf8f0d640 1 -- 172.21.15.26:0/3145396106 learned_addr learned my addr 172.21.15.26:0/3145396106 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf37fe640 1 --1- 172.21.15.26:0/3145396106 >> v1:172.21.15.26:6789/0 conn(0x7fecf4132650 0x7fecf41378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:32780/0 (socket says 172.21.15.26:32780) 2024-09-16T14:58:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 1 -- 172.21.15.26:0/3145396106 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867545823 0 0) 0x7fecf409d260 con 0x7fecf4132650 2024-09-16T14:58:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient(hunting): my global_id is 314115 2024-09-16T14:58:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 1 -- 172.21.15.26:0/3145396106 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecdc003530 con 0x7fecf4132650 2024-09-16T14:58:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 1 -- 172.21.15.26:0/3145396106 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714214741 0 0) 0x7fecf409cf60 con 0x7fecf4138ff0 2024-09-16T14:58:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient(hunting): my global_id is 284459 2024-09-16T14:58:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 1 -- 172.21.15.26:0/3145396106 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf409d260 con 0x7fecf4138ff0 2024-09-16T14:58:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 1 -- 172.21.15.26:0/3145396106 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3486565705 0 0) 0x7fecdc003530 con 0x7fecf4132650 2024-09-16T14:58:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 1 -- 172.21.15.26:0/3145396106 >> v1:172.21.15.26:6790/0 conn(0x7fecf4138ff0 legacy=0x7fecf413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 1 -- 172.21.15.26:0/3145396106 >> v1:172.21.15.189:6789/0 conn(0x7fecf409cb80 legacy=0x7fecf40717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:58:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 1 monclient: found mon.noname-a 2024-09-16T14:58:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:58:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 1 -- 172.21.15.26:0/3145396106 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf409f2d0 con 0x7fecf4132650 2024-09-16T14:58:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient: _finish_auth 0 2024-09-16T14:58:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:58:29.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 1 -- 172.21.15.26:0/3145396106 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fece8001620 con 0x7fecf4132650 2024-09-16T14:58:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 5 monclient: authenticate success, global_id 314115 2024-09-16T14:58:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:58:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient: dump: 2024-09-16T14:58:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.932 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.933 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 1 -- 172.21.15.26:0/3145396106 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fece8001890 con 0x7fecf4132650 2024-09-16T14:58:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 1 -- 172.21.15.26:0/3145396106 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fece8001c60 con 0x7fecf4132650 2024-09-16T14:58:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:58:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecf2ffd640 10 monclient: dump: 2024-09-16T14:58:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.933 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.934 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 10 monclient: get_monmap_and_config success 2024-09-16T14:58:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 4 set_mon_vals no callback set 2024-09-16T14:58:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 10 monclient: shutdown 2024-09-16T14:58:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 1 -- 172.21.15.26:0/3145396106 >> v1:172.21.15.26:6789/0 conn(0x7fecf4132650 legacy=0x7fecf41378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 1 -- 172.21.15.26:0/3145396106 shutdown_connections 2024-09-16T14:58:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 1 -- 172.21.15.26:0/3145396106 >> 172.21.15.26:0/3145396106 conn(0x7fecf409fb50 msgr2=0x7fecf409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 1 -- 172.21.15.26:0/3145396106 shutdown_connections 2024-09-16T14:58:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.919+0000 7fecfa970640 1 -- 172.21.15.26:0/3145396106 wait complete. 2024-09-16T14:58:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 10 monclient: build_initial_monmap 2024-09-16T14:58:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 10 monclient: monmap: 2024-09-16T14:58:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:58:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:58:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:58:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:58:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:58:29.935 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:58:29.935 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 1 Processor -- start 2024-09-16T14:58:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 1 -- start start 2024-09-16T14:58:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 10 monclient: init 2024-09-16T14:58:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:58:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 10 monclient(hunting): picked mon.noname-b con 0x7fecf409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 10 monclient(hunting): picked mon.noname-c con 0x7fecf4132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 10 monclient(hunting): picked mon.noname-a con 0x7fecf4138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf4133770 con 0x7fecf4138ff0 2024-09-16T14:58:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf4138080 con 0x7fecf409cb80 2024-09-16T14:58:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf409a240 con 0x7fecf4132650 2024-09-16T14:58:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:29.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 10 monclient(hunting): authenticate will time out at 1934.045273s 2024-09-16T14:58:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf37fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fecf4132650 0x7fecf406dc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60752/0 (socket says 172.21.15.26:60752) 2024-09-16T14:58:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf8f0d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fecf4138ff0 0x7fecf41cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:32790/0 (socket says 172.21.15.26:32790) 2024-09-16T14:58:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf37fe640 1 -- 172.21.15.26:0/1499967271 learned_addr learned my addr 172.21.15.26:0/1499967271 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556742053 0 0) 0x7fecf4133770 con 0x7fecf4138ff0 2024-09-16T14:58:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): my global_id is 314118 2024-09-16T14:58:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fece0003530 con 0x7fecf4138ff0 2024-09-16T14:58:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658498740 0 0) 0x7fecf409a240 con 0x7fecf4132650 2024-09-16T14:58:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): my global_id is 284462 2024-09-16T14:58:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf4133770 con 0x7fecf4132650 2024-09-16T14:58:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3930332983 0 0) 0x7fece0003530 con 0x7fecf4138ff0 2024-09-16T14:58:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecf409a240 con 0x7fecf4138ff0 2024-09-16T14:58:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1321907633 0 0) 0x7fecf4133770 con 0x7fecf4132650 2024-09-16T14:58:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fece0003530 con 0x7fecf4132650 2024-09-16T14:58:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fecec001c80 con 0x7fecf4138ff0 2024-09-16T14:58:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:58:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): dump: 2024-09-16T14:58:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.938 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.939 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:58:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176032441 0 0) 0x7fece4001120 con 0x7fecf409cb80 2024-09-16T14:58:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): my global_id is 304201 2024-09-16T14:58:29.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf4133770 con 0x7fecf409cb80 2024-09-16T14:58:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fece8001ba0 con 0x7fecf4132650 2024-09-16T14:58:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): dump: 2024-09-16T14:58:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.940 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.941 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1253195026 0 0) 0x7fecf409a240 con 0x7fecf4138ff0 2024-09-16T14:58:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 >> v1:172.21.15.26:6790/0 conn(0x7fecf4132650 legacy=0x7fecf406dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 >> v1:172.21.15.189:6789/0 conn(0x7fecf409cb80 legacy=0x7fecf406d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:58:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 monclient: found mon.a 2024-09-16T14:58:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf406e330 con 0x7fecf4138ff0 2024-09-16T14:58:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient: _finish_auth 0 2024-09-16T14:58:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:58:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 5 monclient: authenticate success, global_id 314118 2024-09-16T14:58:29.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 10 monclient: _renew_subs 2024-09-16T14:58:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 1 -- 172.21.15.26:0/1499967271 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf40a2100 con 0x7fecf4138ff0 2024-09-16T14:58:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 10 monclient: _renew_subs 2024-09-16T14:58:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 1 -- 172.21.15.26:0/1499967271 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf413c5f0 con 0x7fecf4138ff0 2024-09-16T14:58:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecec004b70 con 0x7fecf4138ff0 2024-09-16T14:58:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fecec004e80 con 0x7fecf4138ff0 2024-09-16T14:58:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:58:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf27fc640 4 set_mon_vals no callback set 2024-09-16T14:58:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient: dump: 2024-09-16T14:58:29.942 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.943 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.943 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.943 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.943 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.943 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.943 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.943 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.943 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.943 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:29.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecfa970640 1 -- 172.21.15.26:0/1499967271 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fecd4006590 con 0x7fecd4002830 2024-09-16T14:58:29.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fecec00f8f0 con 0x7fecf4138ff0 2024-09-16T14:58:29.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fecec043ef0 con 0x7fecf4138ff0 2024-09-16T14:58:29.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf37fe640 10 monclient: get_auth_request con 0x7fece0036470 auth_method 0 2024-09-16T14:58:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027681251 0 0) 0x7fecd4006590 con 0x7fecd4002830 2024-09-16T14:58:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 10 monclient(hunting): my global_id is 304204 2024-09-16T14:58:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.923+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf409a240 con 0x7fecd4002830 2024-09-16T14:58:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.927+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3503058686 0 0) 0x7fecf409a240 con 0x7fecd4002830 2024-09-16T14:58:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.927+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecd4006590 con 0x7fecd4002830 2024-09-16T14:58:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.927+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fece4001630 con 0x7fecd4002830 2024-09-16T14:58:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.927+0000 7fecf0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.927+0000 7fecf0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:58:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.927+0000 7fecf0ff9640 10 monclient: dump: 2024-09-16T14:58:29.944 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.945 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.927+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3472937048 0 0) 0x7fecd4006590 con 0x7fecd4002830 2024-09-16T14:58:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.927+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecd400a0e0 con 0x7fecd4002830 2024-09-16T14:58:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.927+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fece4001c30 con 0x7fecd4002830 2024-09-16T14:58:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.927+0000 7fecf0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.927+0000 7fecf0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:58:29.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.927+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 >> v1:172.21.15.189:6789/0 conn(0x7fecd4002830 legacy=0x7fecd4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.927+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 >> v1:172.21.15.189:6789/0 conn(0x7fecd4002830 legacy=0x7fecd4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.931+0000 7fecfa970640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:58:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.931+0000 7fecfa970640 1 -- 172.21.15.26:0/1499967271 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fecd4009d80 con 0x7fecd40090a0 2024-09-16T14:58:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052829764 0 0) 0x7fecd4009d80 con 0x7fecd40090a0 2024-09-16T14:58:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:58:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 10 monclient(hunting): my global_id is 304207 2024-09-16T14:58:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecd4006590 con 0x7fecd40090a0 2024-09-16T14:58:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 187065096 0 0) 0x7fecd4006590 con 0x7fecd40090a0 2024-09-16T14:58:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecd4009d80 con 0x7fecd40090a0 2024-09-16T14:58:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fece4003990 con 0x7fecd40090a0 2024-09-16T14:58:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:58:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 10 monclient: dump: 2024-09-16T14:58:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:29.949 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 274349174 0 0) 0x7fecd4009d80 con 0x7fecd40090a0 2024-09-16T14:58:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fecd400c290 con 0x7fecd40090a0 2024-09-16T14:58:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 2129112320) 0x7fece4003be0 con 0x7fecd40090a0 2024-09-16T14:58:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:58:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:58:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 >> v1:172.21.15.189:6789/0 conn(0x7fecd40090a0 legacy=0x7fecd4009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf0ff9640 1 -- 172.21.15.26:0/1499967271 >> v1:172.21.15.189:6789/0 conn(0x7fecd40090a0 legacy=0x7fecd4009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:29.949 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:58:29.950 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:58:29.950 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:58:29.950 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:58:29.950 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 150, 2024-09-16T14:58:29.950 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:58:29.950 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:58:29.950 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:58:29.950 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:58:29.950 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:29.950 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:58:29.950 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:29.950 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:58:29.950 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:58:29.951 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:29.951 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:29.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:29.951 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:29.951 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:29.951 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:29.951 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:29.951 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:29.951 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:29.951 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:29.951 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:29.951 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:58:29.951 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:58:29.951 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:29.952 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:29.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:29.952 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:29.952 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:29.952 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:29.952 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:29.952 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:29.952 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:29.952 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:29.952 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.952 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:58:29.952 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:58:29.952 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:58:29.953 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:58:29.953 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:58:29.953 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:58:29.953 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:29.953 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:29.953 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:58:29.953 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:58:29.953 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:58:29.953 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:58:29.953 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:58:29.953 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:58:29.953 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:58:29.953 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:29.953 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:58:29.954 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:29.954 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:29.954 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:29.954 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:29.954 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:29.954 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:29.954 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:29.954 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:29.954 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:29.954 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:29.954 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:29.954 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:58:29.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.954 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:58:29.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.955 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:58:29.955 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:58:29.955 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:29.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:29.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.955 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:29.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:58:29.955 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:29.955 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.955 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:58:29.955 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:58:29.956 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:29.956 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:29.956 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:29.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.956 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:58:29.956 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:58:29.956 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:29.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:29.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.956 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:29.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:58:29.956 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:29.956 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.957 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:58:29.957 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:58:29.957 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:29.957 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:29.957 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:29.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.957 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:58:29.957 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:58:29.957 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:29.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:29.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.958 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:29.958 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:58:29.958 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:29.958 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.958 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.958 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:58:29.958 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:58:29.958 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:29.958 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:29.958 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:29.958 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.958 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.959 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:58:29.959 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:58:29.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.959 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:29.959 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:29.959 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:58:29.959 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.959 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:29.959 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:58:29.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:29.959 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:29.959 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:29.959 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:58:29.959 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:29.960 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:29.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:29.960 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:58:29.960 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:58:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecfa970640 1 -- 172.21.15.26:0/1499967271 >> v1:172.21.15.189:6812/3196161582 conn(0x7fece0036470 legacy=0x7fece0038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecfa970640 10 monclient: shutdown 2024-09-16T14:58:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecfa970640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecfa970640 1 -- 172.21.15.26:0/1499967271 >> v1:172.21.15.26:6789/0 conn(0x7fecf4138ff0 legacy=0x7fecf41cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf8f0d640 1 -- 172.21.15.26:0/1499967271 reap_dead start 2024-09-16T14:58:29.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecf8f0d640 1 -- 172.21.15.26:0/1499967271 reap_dead start 2024-09-16T14:58:29.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecfa970640 1 -- 172.21.15.26:0/1499967271 shutdown_connections 2024-09-16T14:58:29.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecfa970640 1 -- 172.21.15.26:0/1499967271 >> 172.21.15.26:0/1499967271 conn(0x7fecf409fb50 msgr2=0x7fecf4131da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:29.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecfa970640 1 -- 172.21.15.26:0/1499967271 shutdown_connections 2024-09-16T14:58:29.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:29.935+0000 7fecfa970640 1 -- 172.21.15.26:0/1499967271 wait complete. 2024-09-16T14:58:29.962 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:58:29.962 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:58:29.962 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:58:30.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 10 monclient: get_monmap_and_config 2024-09-16T14:58:30.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 10 monclient: build_initial_monmap 2024-09-16T14:58:30.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 10 monclient: monmap: 2024-09-16T14:58:30.071 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:30.071 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:30.071 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:58:30.073642+0000 2024-09-16T14:58:30.071 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:58:30.073642+0000 2024-09-16T14:58:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:58:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:58:30.072 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 1 Processor -- start 2024-09-16T14:58:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 1 -- start start 2024-09-16T14:58:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 10 monclient: init 2024-09-16T14:58:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:58:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 10 monclient(hunting): picked mon.noname-c con 0x7f494c0a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 10 monclient(hunting): picked mon.noname-a con 0x7f494c099b60 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 10 monclient(hunting): picked mon.noname-b con 0x7f494c0a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494c098ea0 con 0x7f494c099b60 2024-09-16T14:58:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494c0997b0 con 0x7f494c0a1c00 2024-09-16T14:58:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494c099460 con 0x7f494c0a26f0 2024-09-16T14:58:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 10 monclient(hunting): authenticate will time out at 1934.193670s 2024-09-16T14:58:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4951044640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f494c099b60 0x7f494c139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:32796/0 (socket says 172.21.15.26:32796) 2024-09-16T14:58:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4951044640 1 -- 172.21.15.26:0/446563359 learned_addr learned my addr 172.21.15.26:0/446563359 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 1 -- 172.21.15.26:0/446563359 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738205944 0 0) 0x7f494c098ea0 con 0x7f494c099b60 2024-09-16T14:58:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient(hunting): _negotiate 2024-09-16T14:58:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient(hunting): my global_id is 314121 2024-09-16T14:58:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 1 -- 172.21.15.26:0/446563359 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4934003530 con 0x7f494c099b60 2024-09-16T14:58:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 1 -- 172.21.15.26:0/446563359 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877842587 0 0) 0x7f494c099460 con 0x7f494c0a26f0 2024-09-16T14:58:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient(hunting): _negotiate 2024-09-16T14:58:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient(hunting): my global_id is 284465 2024-09-16T14:58:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 1 -- 172.21.15.26:0/446563359 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f494c098ea0 con 0x7f494c0a26f0 2024-09-16T14:58:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 1 -- 172.21.15.26:0/446563359 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3229015 0 0) 0x7f4934003530 con 0x7f494c099b60 2024-09-16T14:58:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 1 -- 172.21.15.26:0/446563359 >> v1:172.21.15.26:6790/0 conn(0x7f494c0a26f0 legacy=0x7f494c13d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 1 -- 172.21.15.26:0/446563359 >> v1:172.21.15.189:6789/0 conn(0x7f494c0a1c00 legacy=0x7f494c0a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:30.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient: _finish_hunting 0 2024-09-16T14:58:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 1 monclient: found mon.noname-a 2024-09-16T14:58:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:58:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 1 -- 172.21.15.26:0/446563359 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c0a34b0 con 0x7f494c099b60 2024-09-16T14:58:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient: _finish_auth 0 2024-09-16T14:58:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient: _check_auth_tickets 2024-09-16T14:58:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 1 -- 172.21.15.26:0/446563359 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f49400019a0 con 0x7f494c099b60 2024-09-16T14:58:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 5 monclient: authenticate success, global_id 314121 2024-09-16T14:58:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:58:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient: dump: 2024-09-16T14:58:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:30.082 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 1 -- 172.21.15.26:0/446563359 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4940001c80 con 0x7f494c099b60 2024-09-16T14:58:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 1 -- 172.21.15.26:0/446563359 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4940002050 con 0x7f494c099b60 2024-09-16T14:58:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 10 monclient: get_monmap_and_config success 2024-09-16T14:58:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 4 set_mon_vals no callback set 2024-09-16T14:58:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:58:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4950843640 10 monclient: dump: 2024-09-16T14:58:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:30.084 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 10 monclient: shutdown 2024-09-16T14:58:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 1 -- 172.21.15.26:0/446563359 >> v1:172.21.15.26:6789/0 conn(0x7f494c099b60 legacy=0x7f494c139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 1 -- 172.21.15.26:0/446563359 shutdown_connections 2024-09-16T14:58:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 1 -- 172.21.15.26:0/446563359 >> 172.21.15.26:0/446563359 conn(0x7f494c0a3d00 msgr2=0x7f494c0a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 1 -- 172.21.15.26:0/446563359 shutdown_connections 2024-09-16T14:58:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 1 -- 172.21.15.26:0/446563359 wait complete. 2024-09-16T14:58:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 10 monclient: build_initial_monmap 2024-09-16T14:58:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.071+0000 7f4953aa9640 10 monclient: monmap: 2024-09-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 1 Processor -- start 2024-09-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 1 -- start start 2024-09-16T14:58:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 10 monclient: init 2024-09-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 10 monclient(hunting): picked mon.noname-c con 0x7f494c099b60 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 10 monclient(hunting): picked mon.noname-a con 0x7f494c0a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 10 monclient(hunting): picked mon.noname-b con 0x7f494c0a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494c11fc40 con 0x7f494c0a1c00 2024-09-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494c09f080 con 0x7f494c0a26f0 2024-09-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494c09eb30 con 0x7f494c099b60 2024-09-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 10 monclient(hunting): authenticate will time out at 1934.197000s 2024-09-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4951845640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f494c099b60 0x7f494c13cee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60772/0 (socket says 172.21.15.26:60772) 2024-09-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4951845640 1 -- 172.21.15.26:0/2676796812 learned_addr learned my addr 172.21.15.26:0/2676796812 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634401034 0 0) 0x7f494c11fc40 con 0x7f494c0a1c00 2024-09-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): my global_id is 314124 2024-09-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4938003530 con 0x7f494c0a1c00 2024-09-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495404662 0 0) 0x7f494c09eb30 con 0x7f494c099b60 2024-09-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): my global_id is 284468 2024-09-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f494c11fc40 con 0x7f494c099b60 2024-09-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 472957358 0 0) 0x7f4938003530 con 0x7f494c0a1c00 2024-09-16T14:58:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f494c09eb30 con 0x7f494c0a1c00 2024-09-16T14:58:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4940001df0 con 0x7f494c0a1c00 2024-09-16T14:58:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:58:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): dump: 2024-09-16T14:58:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:30.089 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:30.089 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:30.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:58:30.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4037420484 0 0) 0x7f494c11fc40 con 0x7f494c099b60 2024-09-16T14:58:30.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4938003530 con 0x7f494c099b60 2024-09-16T14:58:30.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4944001920 con 0x7f494c099b60 2024-09-16T14:58:30.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:30.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:30.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): dump: 2024-09-16T14:58:30.089 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:30.089 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:30.089 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:30.089 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204632216 0 0) 0x7f494c09f080 con 0x7f494c0a26f0 2024-09-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient(hunting): my global_id is 304213 2024-09-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f494c11fc40 con 0x7f494c0a26f0 2024-09-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1225707794 0 0) 0x7f494c09eb30 con 0x7f494c0a1c00 2024-09-16T14:58:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 >> v1:172.21.15.26:6790/0 conn(0x7f494c099b60 legacy=0x7f494c13cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 >> v1:172.21.15.189:6789/0 conn(0x7f494c0a26f0 legacy=0x7f494c13dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 monclient: found mon.a 2024-09-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c09e3f0 con 0x7f494c0a1c00 2024-09-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 5 monclient: authenticate success, global_id 314124 2024-09-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 10 monclient: _renew_subs 2024-09-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 1 -- 172.21.15.26:0/2676796812 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494c11f960 con 0x7f494c0a1c00 2024-09-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 10 monclient: _renew_subs 2024-09-16T14:58:30.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 1 -- 172.21.15.26:0/2676796812 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f494c1d2940 con 0x7f494c0a1c00 2024-09-16T14:58:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4940002090 con 0x7f494c0a1c00 2024-09-16T14:58:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4940005120 con 0x7f494c0a1c00 2024-09-16T14:58:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:58:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 10 monclient: dump: 2024-09-16T14:58:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:30.092 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494bfff640 4 set_mon_vals no callback set 2024-09-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4953aa9640 1 -- 172.21.15.26:0/2676796812 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f492c005050 con 0x7f494c0a1c00 2024-09-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f494000fb90 con 0x7f494c0a1c00 2024-09-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f4940044120 con 0x7f494c0a1c00 2024-09-16T14:58:30.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.075+0000 7f4951845640 10 monclient: get_auth_request con 0x7f4938036680 auth_method 0 2024-09-16T14:58:30.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.079+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4940014950 con 0x7f494c0a1c00 2024-09-16T14:58:30.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.079+0000 7f494a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:30.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.079+0000 7f494a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:58:30.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.299+0000 7f4953aa9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:58:30.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.299+0000 7f4953aa9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:58:30.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.299+0000 7f4953aa9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:30.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.299+0000 7f4953aa9640 1 -- 172.21.15.26:0/2676796812 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f492c005b10 con 0x7f494c0a1c00 2024-09-16T14:58:30.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f494a7fc640 1 -- 172.21.15.26:0/2676796812 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f4940014770 con 0x7f494c0a1c00 2024-09-16T14:58:30.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f494a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:58:30.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f494a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:58:30.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=10,kv=11,logm=48} crc {auth=2364982500,config=692619927,health=495137188,kv=1867599010,logm=2304607265}) 2024-09-16T14:58:30.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=11,kv=9,logm=49} crc {auth=703101386,config=692619927,health=2667511041,kv=1879805014,logm=1944557278}) 2024-09-16T14:58:30.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=10,kv=11,logm=48} crc {auth=2364982500,config=692619927,health=495137188,kv=1867599010,logm=2304607265}) 2024-09-16T14:58:30.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=12,kv=10,logm=48} crc {auth=1014689143,config=692619927,health=3559181088,kv=2418557040,logm=774204438}) 2024-09-16T14:58:30.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f4953aa9640 1 -- 172.21.15.26:0/2676796812 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4938036680 legacy=0x7f4938038b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:30.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f4953aa9640 10 monclient: shutdown 2024-09-16T14:58:30.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f4953aa9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:30.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f4953aa9640 1 -- 172.21.15.26:0/2676796812 >> v1:172.21.15.26:6789/0 conn(0x7f494c0a1c00 legacy=0x7f494c13d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:30.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f4953aa9640 1 -- 172.21.15.26:0/2676796812 shutdown_connections 2024-09-16T14:58:30.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f4953aa9640 1 -- 172.21.15.26:0/2676796812 >> 172.21.15.26:0/2676796812 conn(0x7f494c0a3d00 msgr2=0x7f494c0a0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:30.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f4953aa9640 1 -- 172.21.15.26:0/2676796812 shutdown_connections 2024-09-16T14:58:30.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:30.303+0000 7f4953aa9640 1 -- 172.21.15.26:0/2676796812 wait complete. 2024-09-16T14:58:30.307 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.307+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.307+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2497026378}) 2024-09-16T14:58:30.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.307+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1778396116}) 2024-09-16T14:58:30.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.307+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.307+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2497026378}) 2024-09-16T14:58:30.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.307+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3650836975}) 2024-09-16T14:58:30.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.307+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.309 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.307+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3152839733}) 2024-09-16T14:58:30.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.307+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2997712635}) 2024-09-16T14:58:30.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.307+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.307+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3152839733}) 2024-09-16T14:58:30.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.307+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1205052042}) 2024-09-16T14:58:30.312 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.311+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.312 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.311+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1243116838}) 2024-09-16T14:58:30.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.311+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=778692545}) 2024-09-16T14:58:30.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.311+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.311+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1243116838}) 2024-09-16T14:58:30.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.311+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=101492264}) 2024-09-16T14:58:30.315 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.315+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.315 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.315+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2873879196}) 2024-09-16T14:58:30.315 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.315+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=381326400}) 2024-09-16T14:58:30.315 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.315+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.315 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.315+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2873879196}) 2024-09-16T14:58:30.315 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.315+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=386843462}) 2024-09-16T14:58:30.317 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.315+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.317 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.315+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=932709291}) 2024-09-16T14:58:30.317 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.315+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649252270}) 2024-09-16T14:58:30.317 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.315+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.317 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.315+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=932709291}) 2024-09-16T14:58:30.317 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.315+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=753226729}) 2024-09-16T14:58:30.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2003051612}) 2024-09-16T14:58:30.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691923644}) 2024-09-16T14:58:30.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2003051612}) 2024-09-16T14:58:30.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3200366911}) 2024-09-16T14:58:30.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394271035}) 2024-09-16T14:58:30.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3336468281}) 2024-09-16T14:58:30.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394271035}) 2024-09-16T14:58:30.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3365057358}) 2024-09-16T14:58:30.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2241921272}) 2024-09-16T14:58:30.322 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773549912}) 2024-09-16T14:58:30.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2241921272}) 2024-09-16T14:58:30.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.319+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=848306566}) 2024-09-16T14:58:30.323 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:58:30.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.323+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.323+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4191190141}) 2024-09-16T14:58:30.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.323+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3245909232}) 2024-09-16T14:58:30.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.323+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.323+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4191190141}) 2024-09-16T14:58:30.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.323+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3284942116}) 2024-09-16T14:58:30.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.323+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.323+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829407372}) 2024-09-16T14:58:30.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.323+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3154708593}) 2024-09-16T14:58:30.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.323+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.323+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829407372}) 2024-09-16T14:58:30.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.323+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086207682}) 2024-09-16T14:58:30.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.327+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.328 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.327+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4071005947}) 2024-09-16T14:58:30.328 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.327+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3542051609}) 2024-09-16T14:58:30.328 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.327+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.328 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.327+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4071005947}) 2024-09-16T14:58:30.328 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.327+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3325942570}) 2024-09-16T14:58:30.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.327+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.327+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3398835080}) 2024-09-16T14:58:30.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.327+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191373149}) 2024-09-16T14:58:30.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.327+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.327+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3398835080}) 2024-09-16T14:58:30.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.327+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3211613394}) 2024-09-16T14:58:30.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=825039208}) 2024-09-16T14:58:30.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1909518616}) 2024-09-16T14:58:30.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=825039208}) 2024-09-16T14:58:30.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159121419}) 2024-09-16T14:58:30.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=843300568}) 2024-09-16T14:58:30.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338089424}) 2024-09-16T14:58:30.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=843300568}) 2024-09-16T14:58:30.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3968699718}) 2024-09-16T14:58:30.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3465382860}) 2024-09-16T14:58:30.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=121601295}) 2024-09-16T14:58:30.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3465382860}) 2024-09-16T14:58:30.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.331+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1000259017}) 2024-09-16T14:58:30.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.335+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.335+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=317679122}) 2024-09-16T14:58:30.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.335+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105574997}) 2024-09-16T14:58:30.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.335+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.335+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=317679122}) 2024-09-16T14:58:30.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.335+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=670165785}) 2024-09-16T14:58:30.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.335+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.335+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1198387761}) 2024-09-16T14:58:30.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.335+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4255670470}) 2024-09-16T14:58:30.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.335+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.335+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1198387761}) 2024-09-16T14:58:30.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.335+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3160257229}) 2024-09-16T14:58:30.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.339+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.339+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4070705948}) 2024-09-16T14:58:30.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.339+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2790932405}) 2024-09-16T14:58:30.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.339+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.339+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4070705948}) 2024-09-16T14:58:30.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.339+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3784242782}) 2024-09-16T14:58:30.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.339+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.339+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=61134206}) 2024-09-16T14:58:30.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.339+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1139592248}) 2024-09-16T14:58:30.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.339+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.339+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=61134206}) 2024-09-16T14:58:30.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.339+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1318573948}) 2024-09-16T14:58:30.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.343+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.343+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3447109910}) 2024-09-16T14:58:30.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.343+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2091444568}) 2024-09-16T14:58:30.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.343+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.343+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3447109910}) 2024-09-16T14:58:30.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.343+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2705032098}) 2024-09-16T14:58:30.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.343+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.343+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1592932219}) 2024-09-16T14:58:30.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.343+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=977855148}) 2024-09-16T14:58:30.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.343+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.343+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1592932219}) 2024-09-16T14:58:30.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.343+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=667847600}) 2024-09-16T14:58:30.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.343+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972781570}) 2024-09-16T14:58:30.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1886795642}) 2024-09-16T14:58:30.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972781570}) 2024-09-16T14:58:30.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=795516863}) 2024-09-16T14:58:30.348 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3060477135}) 2024-09-16T14:58:30.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=405672285}) 2024-09-16T14:58:30.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3060477135}) 2024-09-16T14:58:30.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1128829543}) 2024-09-16T14:58:30.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=362407964}) 2024-09-16T14:58:30.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3375157902}) 2024-09-16T14:58:30.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.350 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=362407964}) 2024-09-16T14:58:30.351 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.347+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1327387219}) 2024-09-16T14:58:30.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.351+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.351+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3800929522}) 2024-09-16T14:58:30.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.351+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628103119}) 2024-09-16T14:58:30.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.351+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.351+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3800929522}) 2024-09-16T14:58:30.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.351+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2927991908}) 2024-09-16T14:58:30.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.351+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.351+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1549072758}) 2024-09-16T14:58:30.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.351+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=435031932}) 2024-09-16T14:58:30.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.351+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.351+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1549072758}) 2024-09-16T14:58:30.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.351+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2475201154}) 2024-09-16T14:58:30.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.355+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.355+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=146160364}) 2024-09-16T14:58:30.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.355+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2336093870}) 2024-09-16T14:58:30.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.355+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.355+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=146160364}) 2024-09-16T14:58:30.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.355+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3698408039}) 2024-09-16T14:58:30.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.355+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.355+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1311518183}) 2024-09-16T14:58:30.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.355+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2440269922}) 2024-09-16T14:58:30.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.355+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.355+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1311518183}) 2024-09-16T14:58:30.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.355+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=31659505}) 2024-09-16T14:58:30.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3225093760}) 2024-09-16T14:58:30.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3452253722}) 2024-09-16T14:58:30.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3225093760}) 2024-09-16T14:58:30.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1893028892}) 2024-09-16T14:58:30.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.361 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=326306897}) 2024-09-16T14:58:30.361 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1764041572}) 2024-09-16T14:58:30.361 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.361 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=326306897}) 2024-09-16T14:58:30.361 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4199256672}) 2024-09-16T14:58:30.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2271847669}) 2024-09-16T14:58:30.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2748224341}) 2024-09-16T14:58:30.363 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.363 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2271847669}) 2024-09-16T14:58:30.363 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.359+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2300719685}) 2024-09-16T14:58:30.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.363+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.363+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3356817229}) 2024-09-16T14:58:30.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.363+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1243743224}) 2024-09-16T14:58:30.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.363+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.365 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.363+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3356817229}) 2024-09-16T14:58:30.365 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.363+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=439112281}) 2024-09-16T14:58:30.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.363+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.363+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=740306985}) 2024-09-16T14:58:30.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.363+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3505721257}) 2024-09-16T14:58:30.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.363+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.363+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=740306985}) 2024-09-16T14:58:30.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.363+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=685869896}) 2024-09-16T14:58:30.367 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.367+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.367 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.367+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=545546845}) 2024-09-16T14:58:30.367 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.367+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4137055084}) 2024-09-16T14:58:30.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.367+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.367+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=545546845}) 2024-09-16T14:58:30.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.367+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1811117170}) 2024-09-16T14:58:30.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.367+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.367+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=366462541}) 2024-09-16T14:58:30.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.367+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3686352748}) 2024-09-16T14:58:30.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.367+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.367+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=366462541}) 2024-09-16T14:58:30.369 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.367+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=55001166}) 2024-09-16T14:58:30.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=942473112}) 2024-09-16T14:58:30.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1854565889}) 2024-09-16T14:58:30.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=942473112}) 2024-09-16T14:58:30.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1766250711}) 2024-09-16T14:58:30.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2162224091}) 2024-09-16T14:58:30.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2969915305}) 2024-09-16T14:58:30.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2162224091}) 2024-09-16T14:58:30.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3650222072}) 2024-09-16T14:58:30.374 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=403648648}) 2024-09-16T14:58:30.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484352339}) 2024-09-16T14:58:30.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=403648648}) 2024-09-16T14:58:30.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.371+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=482008222}) 2024-09-16T14:58:30.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.375+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.375+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2927865026}) 2024-09-16T14:58:30.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.375+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=331251438}) 2024-09-16T14:58:30.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.375+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.375+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2927865026}) 2024-09-16T14:58:30.376 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.375+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3837959690}) 2024-09-16T14:58:30.378 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.375+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.378 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.375+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=985996460}) 2024-09-16T14:58:30.378 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.375+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1171576869}) 2024-09-16T14:58:30.378 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.375+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.378 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.375+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=985996460}) 2024-09-16T14:58:30.378 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.375+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2557403174}) 2024-09-16T14:58:30.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.379+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.380 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.379+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2019934686}) 2024-09-16T14:58:30.380 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.379+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333241698}) 2024-09-16T14:58:30.380 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.379+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.380 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.379+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2019934686}) 2024-09-16T14:58:30.380 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.379+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=596362540}) 2024-09-16T14:58:30.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.379+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.379+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2786184794}) 2024-09-16T14:58:30.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.379+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584452418}) 2024-09-16T14:58:30.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.379+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.382 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.379+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2786184794}) 2024-09-16T14:58:30.382 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.379+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2892938115}) 2024-09-16T14:58:30.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.383+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.383+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2565397054}) 2024-09-16T14:58:30.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.383+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=44036192}) 2024-09-16T14:58:30.384 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.383+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.384 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.383+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2565397054}) 2024-09-16T14:58:30.384 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.383+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1197942655}) 2024-09-16T14:58:30.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.383+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.383+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2312889733}) 2024-09-16T14:58:30.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.383+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1029266180}) 2024-09-16T14:58:30.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.383+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.383+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2312889733}) 2024-09-16T14:58:30.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.383+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3061150937}) 2024-09-16T14:58:30.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.383+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.383+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=648923450}) 2024-09-16T14:58:30.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.383+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=648908218}) 2024-09-16T14:58:30.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.383+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.387+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=648923450}) 2024-09-16T14:58:30.387 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.387+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1263802327}) 2024-09-16T14:58:30.388 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.387+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.388 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.387+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2930105428}) 2024-09-16T14:58:30.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.387+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=887143602}) 2024-09-16T14:58:30.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.387+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.387+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2930105428}) 2024-09-16T14:58:30.389 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.387+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=551181052}) 2024-09-16T14:58:30.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.387+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.387+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=193869335}) 2024-09-16T14:58:30.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.387+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2971867636}) 2024-09-16T14:58:30.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.387+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.387+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=193869335}) 2024-09-16T14:58:30.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.387+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4028148942}) 2024-09-16T14:58:30.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.391+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.391+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199849300}) 2024-09-16T14:58:30.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.391+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2153348543}) 2024-09-16T14:58:30.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.391+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.391+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199849300}) 2024-09-16T14:58:30.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.391+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782337976}) 2024-09-16T14:58:30.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.391+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.391+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3237259638}) 2024-09-16T14:58:30.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.391+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=15462532}) 2024-09-16T14:58:30.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.391+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.391+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3237259638}) 2024-09-16T14:58:30.394 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.391+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996524003}) 2024-09-16T14:58:30.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.395+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.395+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1337426765}) 2024-09-16T14:58:30.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.395+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918436932}) 2024-09-16T14:58:30.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.395+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.395+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1337426765}) 2024-09-16T14:58:30.396 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.395+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1179106756}) 2024-09-16T14:58:30.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.395+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.395+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027069316}) 2024-09-16T14:58:30.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.395+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=963500492}) 2024-09-16T14:58:30.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.395+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.395+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027069316}) 2024-09-16T14:58:30.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.395+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=942606632}) 2024-09-16T14:58:30.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1167678873}) 2024-09-16T14:58:30.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703121125}) 2024-09-16T14:58:30.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1167678873}) 2024-09-16T14:58:30.399 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3096012015}) 2024-09-16T14:58:30.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3525781097}) 2024-09-16T14:58:30.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605925306}) 2024-09-16T14:58:30.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3525781097}) 2024-09-16T14:58:30.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1168054149}) 2024-09-16T14:58:30.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3610746902}) 2024-09-16T14:58:30.402 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023733142}) 2024-09-16T14:58:30.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3610746902}) 2024-09-16T14:58:30.403 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.399+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4187424497}) 2024-09-16T14:58:30.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.403+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.403+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001354495}) 2024-09-16T14:58:30.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.403+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3890159552}) 2024-09-16T14:58:30.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.403+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.403+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001354495}) 2024-09-16T14:58:30.404 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.403+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2517520381}) 2024-09-16T14:58:30.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.403+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.403+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3594690288}) 2024-09-16T14:58:30.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.403+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1888884981}) 2024-09-16T14:58:30.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.403+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.403+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3594690288}) 2024-09-16T14:58:30.406 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.403+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1643698278}) 2024-09-16T14:58:30.407 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.407+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.407+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2440545360}) 2024-09-16T14:58:30.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.407+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321519628}) 2024-09-16T14:58:30.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.407+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.407+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2440545360}) 2024-09-16T14:58:30.408 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.407+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3336262806}) 2024-09-16T14:58:30.409 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.407+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.409 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.407+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2661693399}) 2024-09-16T14:58:30.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.407+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605673883}) 2024-09-16T14:58:30.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.407+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.407+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2661693399}) 2024-09-16T14:58:30.410 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.407+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2196866798}) 2024-09-16T14:58:30.411 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.411+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.411 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.411+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=127802414}) 2024-09-16T14:58:30.411 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.411+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1564304874}) 2024-09-16T14:58:30.411 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.411+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.411 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.411+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=127802414}) 2024-09-16T14:58:30.411 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.411+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3380554954}) 2024-09-16T14:58:30.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.411+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.411+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1630956283}) 2024-09-16T14:58:30.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.411+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1807252104}) 2024-09-16T14:58:30.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.411+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.411+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1630956283}) 2024-09-16T14:58:30.413 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.411+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4246856685}) 2024-09-16T14:58:30.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2446230759}) 2024-09-16T14:58:30.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1303202800}) 2024-09-16T14:58:30.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2446230759}) 2024-09-16T14:58:30.415 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2915182238}) 2024-09-16T14:58:30.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1502187720}) 2024-09-16T14:58:30.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1076218126}) 2024-09-16T14:58:30.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1502187720}) 2024-09-16T14:58:30.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1561020538}) 2024-09-16T14:58:30.418 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.419 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=136664001}) 2024-09-16T14:58:30.419 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=370250458}) 2024-09-16T14:58:30.419 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.419 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=136664001}) 2024-09-16T14:58:30.419 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.415+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1810188774}) 2024-09-16T14:58:30.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.419+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.419+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1608603143}) 2024-09-16T14:58:30.420 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.419+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=430007532}) 2024-09-16T14:58:30.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.419+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.419+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1608603143}) 2024-09-16T14:58:30.421 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.419+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2292040356}) 2024-09-16T14:58:30.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.419+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.419+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1983649829}) 2024-09-16T14:58:30.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.419+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3284924409}) 2024-09-16T14:58:30.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.419+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.419+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1983649829}) 2024-09-16T14:58:30.422 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.419+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1718866981}) 2024-09-16T14:58:30.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.423+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.423+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4219188356}) 2024-09-16T14:58:30.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.423+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972090214}) 2024-09-16T14:58:30.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.423+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.423+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4219188356}) 2024-09-16T14:58:30.424 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.423+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2300622624}) 2024-09-16T14:58:30.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.423+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.423+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1024262827}) 2024-09-16T14:58:30.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.423+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1281435638}) 2024-09-16T14:58:30.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.423+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.423+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1024262827}) 2024-09-16T14:58:30.426 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.423+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2100535209}) 2024-09-16T14:58:30.427 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.427+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.427+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=886872252}) 2024-09-16T14:58:30.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.427+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1153807195}) 2024-09-16T14:58:30.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.427+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.427+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=886872252}) 2024-09-16T14:58:30.428 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.427+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=454693673}) 2024-09-16T14:58:30.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.427+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.427+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3474792465}) 2024-09-16T14:58:30.429 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.427+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048490181}) 2024-09-16T14:58:30.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.427+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.427+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3474792465}) 2024-09-16T14:58:30.430 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.427+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4253459392}) 2024-09-16T14:58:30.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.431+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.431+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605420245}) 2024-09-16T14:58:30.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.431+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824144590}) 2024-09-16T14:58:30.431 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.431+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.431+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605420245}) 2024-09-16T14:58:30.432 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.431+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2561506614}) 2024-09-16T14:58:30.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.431+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.431+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3954403911}) 2024-09-16T14:58:30.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.431+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1172175044}) 2024-09-16T14:58:30.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.431+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.431+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3954403911}) 2024-09-16T14:58:30.433 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.431+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2162664020}) 2024-09-16T14:58:30.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=357851247}) 2024-09-16T14:58:30.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3306783776}) 2024-09-16T14:58:30.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=357851247}) 2024-09-16T14:58:30.435 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1081627138}) 2024-09-16T14:58:30.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1648274777}) 2024-09-16T14:58:30.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2454868114}) 2024-09-16T14:58:30.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1648274777}) 2024-09-16T14:58:30.437 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1884819331}) 2024-09-16T14:58:30.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.438 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4104239790}) 2024-09-16T14:58:30.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773854452}) 2024-09-16T14:58:30.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4104239790}) 2024-09-16T14:58:30.439 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.435+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2875358231}) 2024-09-16T14:58:30.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.439+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.439+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2861295821}) 2024-09-16T14:58:30.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.439+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1179849038}) 2024-09-16T14:58:30.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.439+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.439+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2861295821}) 2024-09-16T14:58:30.441 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.439+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=283073824}) 2024-09-16T14:58:30.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.443+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.443+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2461830367}) 2024-09-16T14:58:30.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.443+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3047844553}) 2024-09-16T14:58:30.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.443+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.443+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2461830367}) 2024-09-16T14:58:30.443 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.443+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1136875424}) 2024-09-16T14:58:30.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.443+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.443+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1563644908}) 2024-09-16T14:58:30.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.443+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3081980728}) 2024-09-16T14:58:30.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.443+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.443+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1563644908}) 2024-09-16T14:58:30.445 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.443+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3040188037}) 2024-09-16T14:58:30.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.447+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.447+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3369082711}) 2024-09-16T14:58:30.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.447+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=998774452}) 2024-09-16T14:58:30.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.447+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.447+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3369082711}) 2024-09-16T14:58:30.447 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.447+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692741345}) 2024-09-16T14:58:30.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.447+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.447+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=952246916}) 2024-09-16T14:58:30.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.447+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=841909289}) 2024-09-16T14:58:30.449 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.447+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.447+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=952246916}) 2024-09-16T14:58:30.450 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.447+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3102529753}) 2024-09-16T14:58:30.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.451+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.451+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=684762681}) 2024-09-16T14:58:30.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.451+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=640700824}) 2024-09-16T14:58:30.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.451+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.451+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=684762681}) 2024-09-16T14:58:30.451 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.451+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3596748331}) 2024-09-16T14:58:30.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.451+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.451+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=82501230}) 2024-09-16T14:58:30.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.451+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3339431962}) 2024-09-16T14:58:30.453 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.451+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.454 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.451+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=82501230}) 2024-09-16T14:58:30.454 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.451+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1252008414}) 2024-09-16T14:58:30.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.455+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.455+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3246974991}) 2024-09-16T14:58:30.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.455+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108893703}) 2024-09-16T14:58:30.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.455+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.455 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.455+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3246974991}) 2024-09-16T14:58:30.456 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.455+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1342592356}) 2024-09-16T14:58:30.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.455+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.455+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3215270753}) 2024-09-16T14:58:30.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.455+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=825515125}) 2024-09-16T14:58:30.457 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.455+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.458 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.455+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3215270753}) 2024-09-16T14:58:30.458 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.455+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3511072884}) 2024-09-16T14:58:30.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.459+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.459+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3552043808}) 2024-09-16T14:58:30.459 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.459+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1804097469}) 2024-09-16T14:58:30.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.459+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.459+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3552043808}) 2024-09-16T14:58:30.460 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.459+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3837432738}) 2024-09-16T14:58:30.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.459+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.459+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=756993376}) 2024-09-16T14:58:30.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.459+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1975060957}) 2024-09-16T14:58:30.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.459+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.461 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.459+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=756993376}) 2024-09-16T14:58:30.462 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.459+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=338884633}) 2024-09-16T14:58:30.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.463+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.463+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4084025272}) 2024-09-16T14:58:30.463 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.463+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3575426778}) 2024-09-16T14:58:30.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.463+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.463+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4084025272}) 2024-09-16T14:58:30.464 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.463+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1055775378}) 2024-09-16T14:58:30.465 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.463+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.463+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=504710317}) 2024-09-16T14:58:30.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.463+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636491651}) 2024-09-16T14:58:30.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.463+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.463+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=504710317}) 2024-09-16T14:58:30.466 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.463+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2385452092}) 2024-09-16T14:58:30.467 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.467+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.467+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3938064257}) 2024-09-16T14:58:30.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.467+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1084070223}) 2024-09-16T14:58:30.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.467+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.467+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3938064257}) 2024-09-16T14:58:30.468 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.467+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3257450987}) 2024-09-16T14:58:30.469 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.467+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.467+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=444656029}) 2024-09-16T14:58:30.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.467+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2856180006}) 2024-09-16T14:58:30.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.467+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.467+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=444656029}) 2024-09-16T14:58:30.470 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.467+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2111480680}) 2024-09-16T14:58:30.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.471+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.471 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.471+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1941499682}) 2024-09-16T14:58:30.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.471+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3346438291}) 2024-09-16T14:58:30.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.471+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.471+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1941499682}) 2024-09-16T14:58:30.472 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.471+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=599251846}) 2024-09-16T14:58:30.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.471+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.471+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598624658}) 2024-09-16T14:58:30.473 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.471+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1019888189}) 2024-09-16T14:58:30.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.471+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.471+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598624658}) 2024-09-16T14:58:30.474 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.471+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3465174124}) 2024-09-16T14:58:30.475 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.475+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.475 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.475+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1290706736}) 2024-09-16T14:58:30.475 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.475+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809741959}) 2024-09-16T14:58:30.475 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.475+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.475+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1290706736}) 2024-09-16T14:58:30.476 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.475+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1817340439}) 2024-09-16T14:58:30.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.475+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.475+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3360846747}) 2024-09-16T14:58:30.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.475+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2695328618}) 2024-09-16T14:58:30.477 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.475+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.475+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3360846747}) 2024-09-16T14:58:30.478 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.475+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3394939653}) 2024-09-16T14:58:30.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.479+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.479+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3670126045}) 2024-09-16T14:58:30.479 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.479+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902785034}) 2024-09-16T14:58:30.480 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.479+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.480 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.479+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3670126045}) 2024-09-16T14:58:30.480 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.479+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2771491028}) 2024-09-16T14:58:30.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.479+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.479+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4119136328}) 2024-09-16T14:58:30.481 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.479+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2219664648}) 2024-09-16T14:58:30.482 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.479+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.482 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.479+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4119136328}) 2024-09-16T14:58:30.482 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.479+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758637292}) 2024-09-16T14:58:30.483 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.483+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.483+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4158362582}) 2024-09-16T14:58:30.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.483+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3277425225}) 2024-09-16T14:58:30.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.483+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.483+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4158362582}) 2024-09-16T14:58:30.484 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.483+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=396610068}) 2024-09-16T14:58:30.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.483+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.483+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1663313033}) 2024-09-16T14:58:30.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.483+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=489227250}) 2024-09-16T14:58:30.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.483+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.483+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1663313033}) 2024-09-16T14:58:30.486 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.483+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2442279136}) 2024-09-16T14:58:30.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.487+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.487+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3753625292}) 2024-09-16T14:58:30.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.487+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412042397}) 2024-09-16T14:58:30.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.487+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.487+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3753625292}) 2024-09-16T14:58:30.488 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.487+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2242546343}) 2024-09-16T14:58:30.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.487+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.487+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=210848009}) 2024-09-16T14:58:30.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.487+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000495340}) 2024-09-16T14:58:30.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.487+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.487+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=210848009}) 2024-09-16T14:58:30.490 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.487+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3880235874}) 2024-09-16T14:58:30.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.491+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.491+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4293504831}) 2024-09-16T14:58:30.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.491+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1504891058}) 2024-09-16T14:58:30.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.491+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.491+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4293504831}) 2024-09-16T14:58:30.492 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.491+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=494603777}) 2024-09-16T14:58:30.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.491+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.491+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4018446749}) 2024-09-16T14:58:30.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.491+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3231055051}) 2024-09-16T14:58:30.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.491+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.491+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4018446749}) 2024-09-16T14:58:30.494 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.491+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3074196205}) 2024-09-16T14:58:30.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.495+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.495+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1244533970}) 2024-09-16T14:58:30.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.495+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3108146564}) 2024-09-16T14:58:30.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.495+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.495+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1244533970}) 2024-09-16T14:58:30.496 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.495+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1898946511}) 2024-09-16T14:58:30.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.495+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.495+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=594971638}) 2024-09-16T14:58:30.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.495+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=31496368}) 2024-09-16T14:58:30.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.495+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.495+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=594971638}) 2024-09-16T14:58:30.498 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.495+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4220760915}) 2024-09-16T14:58:30.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.499+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.499+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=244111111,mdsmap=1531151339,mgr=3836259515}) 2024-09-16T14:58:30.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.499+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=3109146599,mdsmap=1531151339,mgr=209975238}) 2024-09-16T14:58:30.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.499+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.499+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=244111111,mdsmap=1531151339,mgr=3836259515}) 2024-09-16T14:58:30.500 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.499+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=98,mdsmap=2} crc {logm=1041515955,mdsmap=3585761405}) 2024-09-16T14:58:30.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.511+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.511+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {mgr=542399043,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=398608506,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=458083565}) 2024-09-16T14:58:30.514 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.511+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=9,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {mgr=620920336,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=398608506,mon_config_key=1081754342,monmap=3140514009,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=2012053637}) 2024-09-16T14:58:30.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.511+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:58:30.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.511+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {mgr=542399043,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=398608506,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=458083565}) 2024-09-16T14:58:30.515 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:58:30.511+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {mgr=620920336,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=398608506,mon_config_key=1518680965,monmap=970648238,osd_metadata=1033174197,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3728135073}) 2024-09-16T14:58:31.324 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:58:31.325 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:58:31.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 10 monclient: get_monmap_and_config 2024-09-16T14:58:31.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 10 monclient: build_initial_monmap 2024-09-16T14:58:31.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 10 monclient: monmap: 2024-09-16T14:58:31.457 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:31.457 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.457 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:58:31.458880+0000 2024-09-16T14:58:31.457 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:58:31.458880+0000 2024-09-16T14:58:31.457 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:31.457 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.457 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:31.457 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:58:31.457 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:58:31.457 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 1 Processor -- start 2024-09-16T14:58:31.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 1 -- start start 2024-09-16T14:58:31.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 10 monclient: init 2024-09-16T14:58:31.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:31.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:58:31.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 10 monclient(hunting): picked mon.noname-c con 0x7f9f2809cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:31.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 10 monclient(hunting): picked mon.noname-b con 0x7f9f2809e7a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:31.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 10 monclient(hunting): picked mon.noname-a con 0x7f9f2809d1a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:31.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2809d5b0 con 0x7f9f2809d1a0 2024-09-16T14:58:31.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f28131a90 con 0x7f9f2809cdc0 2024-09-16T14:58:31.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2809aff0 con 0x7f9f2809e7a0 2024-09-16T14:58:31.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:31.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 10 monclient(hunting): authenticate will time out at 1935.579014s 2024-09-16T14:58:31.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f27fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9f2809e7a0 0x7f9f28131380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60786/0 (socket says 172.21.15.26:60786) 2024-09-16T14:58:31.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f27fff640 1 -- 172.21.15.26:0/3753386239 learned_addr learned my addr 172.21.15.26:0/3753386239 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:31.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 1 -- 172.21.15.26:0/3753386239 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214186842 0 0) 0x7f9f2809aff0 con 0x7f9f2809e7a0 2024-09-16T14:58:31.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 10 monclient(hunting): my global_id is 284471 2024-09-16T14:58:31.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 1 -- 172.21.15.26:0/3753386239 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f10003530 con 0x7f9f2809e7a0 2024-09-16T14:58:31.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 1 -- 172.21.15.26:0/3753386239 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829503583 0 0) 0x7f9f2809d5b0 con 0x7f9f2809d1a0 2024-09-16T14:58:31.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 10 monclient(hunting): my global_id is 314127 2024-09-16T14:58:31.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 1 -- 172.21.15.26:0/3753386239 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f2809aff0 con 0x7f9f2809d1a0 2024-09-16T14:58:31.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 1 -- 172.21.15.26:0/3753386239 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1147102974 0 0) 0x7f9f2809aff0 con 0x7f9f2809d1a0 2024-09-16T14:58:31.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 1 -- 172.21.15.26:0/3753386239 >> v1:172.21.15.26:6790/0 conn(0x7f9f2809e7a0 legacy=0x7f9f28131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 1 -- 172.21.15.26:0/3753386239 >> v1:172.21.15.189:6789/0 conn(0x7f9f2809cdc0 legacy=0x7f9f28130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 10 monclient: _finish_hunting 0 2024-09-16T14:58:31.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 1 monclient: found mon.noname-a 2024-09-16T14:58:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:58:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 1 -- 172.21.15.26:0/3753386239 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2809b2d0 con 0x7f9f2809d1a0 2024-09-16T14:58:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 10 monclient: _finish_auth 0 2024-09-16T14:58:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 10 monclient: _check_auth_tickets 2024-09-16T14:58:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 1 -- 172.21.15.26:0/3753386239 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f200019a0 con 0x7f9f2809d1a0 2024-09-16T14:58:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:58:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f277fe640 10 monclient: dump: 2024-09-16T14:58:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.464 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.464 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.464 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.464 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.464 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.464 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.464 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.464 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 5 monclient: authenticate success, global_id 314127 2024-09-16T14:58:31.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.455+0000 7f9f2ef4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:31.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f277fe640 1 -- 172.21.15.26:0/3753386239 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f20001bd0 con 0x7f9f2809d1a0 2024-09-16T14:58:31.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f277fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:31.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f277fe640 1 -- 172.21.15.26:0/3753386239 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f20001fa0 con 0x7f9f2809d1a0 2024-09-16T14:58:31.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:58:31.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f277fe640 10 monclient: dump: 2024-09-16T14:58:31.465 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.465 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.465 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.465 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.465 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.465 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.465 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.465 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.465 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.465 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 10 monclient: get_monmap_and_config success 2024-09-16T14:58:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 4 set_mon_vals no callback set 2024-09-16T14:58:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 10 monclient: shutdown 2024-09-16T14:58:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 1 -- 172.21.15.26:0/3753386239 >> v1:172.21.15.26:6789/0 conn(0x7f9f2809d1a0 legacy=0x7f9f28139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 1 -- 172.21.15.26:0/3753386239 shutdown_connections 2024-09-16T14:58:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 1 -- 172.21.15.26:0/3753386239 >> 172.21.15.26:0/3753386239 conn(0x7f9f2809fb50 msgr2=0x7f9f2809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 1 -- 172.21.15.26:0/3753386239 shutdown_connections 2024-09-16T14:58:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 1 -- 172.21.15.26:0/3753386239 wait complete. 2024-09-16T14:58:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 10 monclient: build_initial_monmap 2024-09-16T14:58:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 10 monclient: monmap: 2024-09-16T14:58:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:58:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:58:31.467 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:58:31.467 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:31.467 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:58:31.467 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:31.467 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:58:31.467 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:58:31.467 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 1 Processor -- start 2024-09-16T14:58:31.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 1 -- start start 2024-09-16T14:58:31.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 10 monclient: init 2024-09-16T14:58:31.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:31.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:58:31.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 10 monclient(hunting): picked mon.noname-a con 0x7f9f2809cdc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:31.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 10 monclient(hunting): picked mon.noname-b con 0x7f9f2809d1a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:31.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 10 monclient(hunting): picked mon.noname-c con 0x7f9f2809e7a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:31.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f280a2110 con 0x7f9f2809cdc0 2024-09-16T14:58:31.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2809a240 con 0x7f9f2809d1a0 2024-09-16T14:58:31.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2811e060 con 0x7f9f2809e7a0 2024-09-16T14:58:31.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:31.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 10 monclient(hunting): authenticate will time out at 1935.582224s 2024-09-16T14:58:31.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2cce8640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9f2809cdc0 0x7f9f2812cf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:32824/0 (socket says 172.21.15.26:32824) 2024-09-16T14:58:31.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2cce8640 1 -- 172.21.15.26:0/2952221211 learned_addr learned my addr 172.21.15.26:0/2952221211 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:31.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796800733 0 0) 0x7f9f28131a90 con 0x7f9f2809cdc0 2024-09-16T14:58:31.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): my global_id is 314130 2024-09-16T14:58:31.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f14003560 con 0x7f9f2809cdc0 2024-09-16T14:58:31.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110252164 0 0) 0x7f9f2811e060 con 0x7f9f2809e7a0 2024-09-16T14:58:31.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): my global_id is 284474 2024-09-16T14:58:31.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f28131a90 con 0x7f9f2809e7a0 2024-09-16T14:58:31.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 534808399 0 0) 0x7f9f14003560 con 0x7f9f2809cdc0 2024-09-16T14:58:31.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f2811e060 con 0x7f9f2809cdc0 2024-09-16T14:58:31.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f180017a0 con 0x7f9f2809cdc0 2024-09-16T14:58:31.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:58:31.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): dump: 2024-09-16T14:58:31.473 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.473 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.473 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.473 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.473 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.473 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.474 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:58:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3101165561 0 0) 0x7f9f28131a90 con 0x7f9f2809e7a0 2024-09-16T14:58:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f14003560 con 0x7f9f2809e7a0 2024-09-16T14:58:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f20001ee0 con 0x7f9f2809e7a0 2024-09-16T14:58:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): dump: 2024-09-16T14:58:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.474 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.475 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.475 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.475 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.475 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.475 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.475 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.475 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.475 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649363784 0 0) 0x7f9f2809a240 con 0x7f9f2809d1a0 2024-09-16T14:58:31.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient(hunting): my global_id is 304219 2024-09-16T14:58:31.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f28131a90 con 0x7f9f2809d1a0 2024-09-16T14:58:31.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1735681783 0 0) 0x7f9f2811e060 con 0x7f9f2809cdc0 2024-09-16T14:58:31.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 >> v1:172.21.15.26:6790/0 conn(0x7f9f2809e7a0 legacy=0x7f9f281d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 >> v1:172.21.15.189:6789/0 conn(0x7f9f2809d1a0 legacy=0x7f9f281ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient: _finish_hunting 0 2024-09-16T14:58:31.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 monclient: found mon.a 2024-09-16T14:58:31.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:31.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:31.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f280701f0 con 0x7f9f2809cdc0 2024-09-16T14:58:31.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient: _finish_auth 0 2024-09-16T14:58:31.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient: _check_auth_tickets 2024-09-16T14:58:31.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:31.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 5 monclient: authenticate success, global_id 314130 2024-09-16T14:58:31.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 10 monclient: _renew_subs 2024-09-16T14:58:31.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:31.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 1 -- 172.21.15.26:0/2952221211 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f2812d610 con 0x7f9f2809cdc0 2024-09-16T14:58:31.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 10 monclient: _renew_subs 2024-09-16T14:58:31.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:31.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f2ef4c640 1 -- 172.21.15.26:0/2952221211 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f28070f90 con 0x7f9f2809cdc0 2024-09-16T14:58:31.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f18001d50 con 0x7f9f2809cdc0 2024-09-16T14:58:31.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:31.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9f180037c0 con 0x7f9f2809cdc0 2024-09-16T14:58:31.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:58:31.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f26ffd640 4 set_mon_vals no callback set 2024-09-16T14:58:31.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 10 monclient: dump: 2024-09-16T14:58:31.478 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.478 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.478 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.478 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.478 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.478 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.478 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.478 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.478 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.479 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.459+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9f18003920 con 0x7f9f2809cdc0 2024-09-16T14:58:31.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.463+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f9f18043bf0 con 0x7f9f2809cdc0 2024-09-16T14:58:31.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.463+0000 7f9f2ef4c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:31.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.463+0000 7f9f2ef4c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:31.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.463+0000 7f9f2ef4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:31.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.463+0000 7f9f2ef4c640 1 -- 172.21.15.26:0/2952221211 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f08005050 con 0x7f9f2809cdc0 2024-09-16T14:58:31.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.463+0000 7f9f27fff640 10 monclient: get_auth_request con 0x7f9f14036400 auth_method 0 2024-09-16T14:58:31.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.467+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9f180040f0 con 0x7f9f2809cdc0 2024-09-16T14:58:31.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.467+0000 7f9f257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:31.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.467+0000 7f9f257fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:58:31.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.687+0000 7f9f2ef4c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:58:31.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.687+0000 7f9f2ef4c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:58:31.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.687+0000 7f9f2ef4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:31.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.687+0000 7f9f2ef4c640 1 -- 172.21.15.26:0/2952221211 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9f08005880 con 0x7f9f2809cdc0 2024-09-16T14:58:31.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.687+0000 7f9f257fa640 1 -- 172.21.15.26:0/2952221211 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1270215553) 0x7f9f18012940 con 0x7f9f2809cdc0 2024-09-16T14:58:31.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.687+0000 7f9f257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:58:31.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.687+0000 7f9f257fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:58:31.690 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:58:31.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.691+0000 7f9f2ef4c640 1 -- 172.21.15.26:0/2952221211 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9f14036400 legacy=0x7f9f140388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.691+0000 7f9f2ef4c640 10 monclient: shutdown 2024-09-16T14:58:31.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.691+0000 7f9f2ef4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:31.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.691+0000 7f9f2ef4c640 1 -- 172.21.15.26:0/2952221211 >> v1:172.21.15.26:6789/0 conn(0x7f9f2809cdc0 legacy=0x7f9f2812cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.691+0000 7f9f2ef4c640 1 -- 172.21.15.26:0/2952221211 shutdown_connections 2024-09-16T14:58:31.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.691+0000 7f9f2ef4c640 1 -- 172.21.15.26:0/2952221211 >> 172.21.15.26:0/2952221211 conn(0x7f9f2809fb50 msgr2=0x7f9f2809bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:31.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.691+0000 7f9f2ef4c640 1 -- 172.21.15.26:0/2952221211 shutdown_connections 2024-09-16T14:58:31.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.691+0000 7f9f2ef4c640 1 -- 172.21.15.26:0/2952221211 wait complete. 2024-09-16T14:58:31.706 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:58:31.706 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:58:31.706 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:58:31.707 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:58:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.815+0000 7f2ba6772640 10 monclient: get_monmap_and_config 2024-09-16T14:58:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.815+0000 7f2ba6772640 10 monclient: build_initial_monmap 2024-09-16T14:58:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.815+0000 7f2ba6772640 10 monclient: monmap: 2024-09-16T14:58:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:58:31.821395+0000 2024-09-16T14:58:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:58:31.821395+0000 2024-09-16T14:58:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:58:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:58:31.820 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 1 Processor -- start 2024-09-16T14:58:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 1 -- start start 2024-09-16T14:58:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 10 monclient: init 2024-09-16T14:58:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:58:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 10 monclient(hunting): picked mon.noname-a con 0x7f2ba009cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 10 monclient(hunting): picked mon.noname-b con 0x7f2ba0132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 10 monclient(hunting): picked mon.noname-c con 0x7f2ba0138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba009d260 con 0x7f2ba009cb80 2024-09-16T14:58:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba0132a80 con 0x7f2ba0138ff0 2024-09-16T14:58:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba009cf60 con 0x7f2ba0132650 2024-09-16T14:58:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 10 monclient(hunting): authenticate will time out at 1935.941337s 2024-09-16T14:58:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9f7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f2ba0132650 0x7f2ba01378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60808/0 (socket says 172.21.15.26:60808) 2024-09-16T14:58:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9f7fe640 1 -- 172.21.15.26:0/3834487239 learned_addr learned my addr 172.21.15.26:0/3834487239 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 1 -- 172.21.15.26:0/3834487239 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159616012 0 0) 0x7f2ba009cf60 con 0x7f2ba0132650 2024-09-16T14:58:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient(hunting): my global_id is 284477 2024-09-16T14:58:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 1 -- 172.21.15.26:0/3834487239 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b88003530 con 0x7f2ba0132650 2024-09-16T14:58:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 1 -- 172.21.15.26:0/3834487239 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661167540 0 0) 0x7f2ba009d260 con 0x7f2ba009cb80 2024-09-16T14:58:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient(hunting): my global_id is 314133 2024-09-16T14:58:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 1 -- 172.21.15.26:0/3834487239 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba009cf60 con 0x7f2ba009cb80 2024-09-16T14:58:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 1 -- 172.21.15.26:0/3834487239 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 366312807 0 0) 0x7f2b88003530 con 0x7f2ba0132650 2024-09-16T14:58:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 1 -- 172.21.15.26:0/3834487239 >> v1:172.21.15.189:6789/0 conn(0x7f2ba0138ff0 legacy=0x7f2ba013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 1 -- 172.21.15.26:0/3834487239 >> v1:172.21.15.26:6789/0 conn(0x7f2ba009cb80 legacy=0x7f2ba00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient: _finish_hunting 0 2024-09-16T14:58:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 1 monclient: found mon.noname-b 2024-09-16T14:58:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:58:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 1 -- 172.21.15.26:0/3834487239 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba009f2d0 con 0x7f2ba0132650 2024-09-16T14:58:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient: _finish_auth 0 2024-09-16T14:58:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient: _check_auth_tickets 2024-09-16T14:58:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 1 -- 172.21.15.26:0/3834487239 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b94001590 con 0x7f2ba0132650 2024-09-16T14:58:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:58:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient: dump: 2024-09-16T14:58:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.830 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.830 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 5 monclient: authenticate success, global_id 284477 2024-09-16T14:58:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 1 -- 172.21.15.26:0/3834487239 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b940017b0 con 0x7f2ba0132650 2024-09-16T14:58:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 10 monclient: get_monmap_and_config success 2024-09-16T14:58:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 1 -- 172.21.15.26:0/3834487239 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b94001ac0 con 0x7f2ba0132650 2024-09-16T14:58:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 4 set_mon_vals no callback set 2024-09-16T14:58:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2b9effd640 10 monclient: dump: 2024-09-16T14:58:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.831 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 10 monclient: shutdown 2024-09-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 1 -- 172.21.15.26:0/3834487239 >> v1:172.21.15.26:6790/0 conn(0x7f2ba0132650 legacy=0x7f2ba01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 1 -- 172.21.15.26:0/3834487239 shutdown_connections 2024-09-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 1 -- 172.21.15.26:0/3834487239 >> 172.21.15.26:0/3834487239 conn(0x7f2ba009fb50 msgr2=0x7f2ba009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 1 -- 172.21.15.26:0/3834487239 shutdown_connections 2024-09-16T14:58:31.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 1 -- 172.21.15.26:0/3834487239 wait complete. 2024-09-16T14:58:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 10 monclient: build_initial_monmap 2024-09-16T14:58:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 10 monclient: monmap: 2024-09-16T14:58:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:58:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:58:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:58:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:58:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:58:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:58:31.833 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 1 Processor -- start 2024-09-16T14:58:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 1 -- start start 2024-09-16T14:58:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.819+0000 7f2ba6772640 10 monclient: init 2024-09-16T14:58:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:58:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 10 monclient(hunting): picked mon.noname-b con 0x7f2ba009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 10 monclient(hunting): picked mon.noname-c con 0x7f2ba0132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 10 monclient(hunting): picked mon.noname-a con 0x7f2ba0138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba009b370 con 0x7f2ba0138ff0 2024-09-16T14:58:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba013c400 con 0x7f2ba009cb80 2024-09-16T14:58:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba00a2a10 con 0x7f2ba0132650 2024-09-16T14:58:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 10 monclient(hunting): authenticate will time out at 1935.944262s 2024-09-16T14:58:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9f7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f2ba0132650 0x7f2ba009ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60816/0 (socket says 172.21.15.26:60816) 2024-09-16T14:58:31.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9f7fe640 1 -- 172.21.15.26:0/1322730844 learned_addr learned my addr 172.21.15.26:0/1322730844 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba4d0f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2ba0138ff0 0x7f2ba01d1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:32844/0 (socket says 172.21.15.26:32844) 2024-09-16T14:58:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777168698 0 0) 0x7f2ba009b370 con 0x7f2ba0138ff0 2024-09-16T14:58:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): my global_id is 314136 2024-09-16T14:58:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b8c003530 con 0x7f2ba0138ff0 2024-09-16T14:58:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346038073 0 0) 0x7f2ba00a2a10 con 0x7f2ba0132650 2024-09-16T14:58:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): my global_id is 284480 2024-09-16T14:58:31.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba009b370 con 0x7f2ba0132650 2024-09-16T14:58:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3052212309 0 0) 0x7f2b8c003530 con 0x7f2ba0138ff0 2024-09-16T14:58:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ba00a2a10 con 0x7f2ba0138ff0 2024-09-16T14:58:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b98001bc0 con 0x7f2ba0138ff0 2024-09-16T14:58:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:58:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): dump: 2024-09-16T14:58:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.837 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:58:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3687469897 0 0) 0x7f2ba009b370 con 0x7f2ba0132650 2024-09-16T14:58:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b8c003530 con 0x7f2ba0132650 2024-09-16T14:58:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b94001490 con 0x7f2ba0132650 2024-09-16T14:58:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): dump: 2024-09-16T14:58:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.837 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.838 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894427042 0 0) 0x7f2ba013c400 con 0x7f2ba009cb80 2024-09-16T14:58:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): my global_id is 304225 2024-09-16T14:58:31.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba009b370 con 0x7f2ba009cb80 2024-09-16T14:58:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2469309175 0 0) 0x7f2ba00a2a10 con 0x7f2ba0138ff0 2024-09-16T14:58:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 >> v1:172.21.15.26:6790/0 conn(0x7f2ba0132650 legacy=0x7f2ba009ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 >> v1:172.21.15.189:6789/0 conn(0x7f2ba009cb80 legacy=0x7f2ba009a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:58:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 monclient: found mon.a 2024-09-16T14:58:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba009a240 con 0x7f2ba0138ff0 2024-09-16T14:58:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient: _finish_auth 0 2024-09-16T14:58:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:58:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 5 monclient: authenticate success, global_id 314136 2024-09-16T14:58:31.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 10 monclient: _renew_subs 2024-09-16T14:58:31.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:31.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 1 -- 172.21.15.26:0/1322730844 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba006e120 con 0x7f2ba0138ff0 2024-09-16T14:58:31.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 10 monclient: _renew_subs 2024-09-16T14:58:31.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:31.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 1 -- 172.21.15.26:0/1322730844 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba01386e0 con 0x7f2ba0138ff0 2024-09-16T14:58:31.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b980039a0 con 0x7f2ba0138ff0 2024-09-16T14:58:31.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:31.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b98003d30 con 0x7f2ba0138ff0 2024-09-16T14:58:31.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:58:31.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient: dump: 2024-09-16T14:58:31.840 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.840 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.841 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9e7fc640 4 set_mon_vals no callback set 2024-09-16T14:58:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2ba6772640 1 -- 172.21.15.26:0/1322730844 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2b80006590 con 0x7f2b80002830 2024-09-16T14:58:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2b98003f10 con 0x7f2ba0138ff0 2024-09-16T14:58:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f2b980441c0 con 0x7f2ba0138ff0 2024-09-16T14:58:31.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278342809 0 0) 0x7f2b80006590 con 0x7f2b80002830 2024-09-16T14:58:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient(hunting): my global_id is 314139 2024-09-16T14:58:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba00a2a10 con 0x7f2b80002830 2024-09-16T14:58:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3617546241 0 0) 0x7f2ba00a2a10 con 0x7f2b80002830 2024-09-16T14:58:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b80006590 con 0x7f2b80002830 2024-09-16T14:58:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b900015a0 con 0x7f2b80002830 2024-09-16T14:58:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:58:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient: dump: 2024-09-16T14:58:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.842 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.843 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9f7fe640 10 monclient: get_auth_request con 0x7f2b8c0363f0 auth_method 0 2024-09-16T14:58:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 42442425 0 0) 0x7f2b80006590 con 0x7f2b80002830 2024-09-16T14:58:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2b8000a0e0 con 0x7f2b80002830 2024-09-16T14:58:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2b900019d0 con 0x7f2b80002830 2024-09-16T14:58:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:58:31.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 >> v1:172.21.15.26:6789/0 conn(0x7f2b80002830 legacy=0x7f2b80004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.823+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 >> v1:172.21.15.26:6789/0 conn(0x7f2b80002830 legacy=0x7f2b80004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:31.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2ba6772640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:58:31.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2ba6772640 1 -- 172.21.15.26:0/1322730844 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2b80009da0 con 0x7f2b800090c0 2024-09-16T14:58:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581585301 0 0) 0x7f2b80009da0 con 0x7f2b800090c0 2024-09-16T14:58:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 10 monclient(hunting): my global_id is 314142 2024-09-16T14:58:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b80006590 con 0x7f2b800090c0 2024-09-16T14:58:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2092485950 0 0) 0x7f2b80006590 con 0x7f2b800090c0 2024-09-16T14:58:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b80009da0 con 0x7f2b800090c0 2024-09-16T14:58:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b900037a0 con 0x7f2b800090c0 2024-09-16T14:58:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:58:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 10 monclient: dump: 2024-09-16T14:58:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.847 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1946137679 0 0) 0x7f2b80009da0 con 0x7f2b800090c0 2024-09-16T14:58:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2b8000c290 con 0x7f2b800090c0 2024-09-16T14:58:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 1930728272) 0x7f2b900059a0 con 0x7f2b800090c0 2024-09-16T14:58:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:58:31.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:58:31.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 >> v1:172.21.15.26:6789/0 conn(0x7f2b800090c0 legacy=0x7f2b80009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2b9cff9640 1 -- 172.21.15.26:0/1322730844 >> v1:172.21.15.26:6789/0 conn(0x7f2b800090c0 legacy=0x7f2b80009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:31.848 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:58:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:58:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:58:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:58:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 150, 2024-09-16T14:58:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:58:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:58:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:58:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:58:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:58:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:58:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:58:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:58:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:58:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:58:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:58:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:58:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:58:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:58:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:58:31.851 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:58:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:58:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:58:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:58:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:58:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:58:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:58:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:58:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:31.852 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:31.853 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:31.853 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:31.853 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:31.853 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:31.853 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:31.853 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:31.853 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:31.853 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:58:31.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:31.853 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:58:31.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:31.853 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:58:31.853 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:58:31.854 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:31.854 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:31.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:31.854 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:31.854 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:58:31.854 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:31.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:31.854 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:31.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:31.854 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:58:31.854 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:58:31.854 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:31.854 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:31.855 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:31.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:31.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:31.855 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:58:31.855 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:58:31.855 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:31.855 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:31.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:31.855 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:31.855 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:58:31.855 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:31.855 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:31.855 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:31.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:31.856 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:58:31.856 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:58:31.856 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:31.856 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:31.856 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:31.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:31.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:31.856 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:58:31.856 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:58:31.856 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:31.856 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:31.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:31.856 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:31.857 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:58:31.857 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:31.857 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:31.857 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:31.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:31.857 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:58:31.857 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:58:31.857 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:31.857 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:31.857 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:31.857 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:31.857 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:31.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:31.858 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:58:31.858 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:58:31.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:31.858 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:31.858 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:31.858 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:58:31.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:31.858 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:31.858 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:58:31.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:31.858 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:31.858 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:31.858 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:58:31.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:31.858 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:31.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:31.859 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:58:31.859 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:58:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2ba6772640 1 -- 172.21.15.26:0/1322730844 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2b8c0363f0 legacy=0x7f2b8c0388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2ba6772640 10 monclient: shutdown 2024-09-16T14:58:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2ba6772640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2ba6772640 1 -- 172.21.15.26:0/1322730844 >> v1:172.21.15.26:6789/0 conn(0x7f2ba0138ff0 legacy=0x7f2ba01d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2ba4d0f640 1 -- 172.21.15.26:0/1322730844 reap_dead start 2024-09-16T14:58:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2ba4d0f640 1 -- 172.21.15.26:0/1322730844 reap_dead start 2024-09-16T14:58:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2ba6772640 1 -- 172.21.15.26:0/1322730844 shutdown_connections 2024-09-16T14:58:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2ba6772640 1 -- 172.21.15.26:0/1322730844 >> 172.21.15.26:0/1322730844 conn(0x7f2ba009fb50 msgr2=0x7f2ba0131d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2ba6772640 1 -- 172.21.15.26:0/1322730844 shutdown_connections 2024-09-16T14:58:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.831+0000 7f2ba6772640 1 -- 172.21.15.26:0/1322730844 wait complete. 2024-09-16T14:58:31.861 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:58:31.861 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:58:31.861 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:58:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.967+0000 7f668b11a640 10 monclient: get_monmap_and_config 2024-09-16T14:58:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.967+0000 7f668b11a640 10 monclient: build_initial_monmap 2024-09-16T14:58:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.967+0000 7f668b11a640 10 monclient: monmap: 2024-09-16T14:58:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:58:31.972988+0000 2024-09-16T14:58:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:58:31.972988+0000 2024-09-16T14:58:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:58:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:58:31.971 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 1 Processor -- start 2024-09-16T14:58:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 1 -- start start 2024-09-16T14:58:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 10 monclient: init 2024-09-16T14:58:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:58:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 10 monclient(hunting): picked mon.noname-b con 0x7f668409a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 10 monclient(hunting): picked mon.noname-a con 0x7f668409d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 10 monclient(hunting): picked mon.noname-c con 0x7f668409b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668409dc60 con 0x7f668409d1b0 2024-09-16T14:58:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6684099ae0 con 0x7f668409b040 2024-09-16T14:58:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668409b8f0 con 0x7f668409a550 2024-09-16T14:58:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 10 monclient(hunting): authenticate will time out at 1936.093282s 2024-09-16T14:58:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f6688eb6640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f668409a550 0x7f668409a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60820/0 (socket says 172.21.15.26:60820) 2024-09-16T14:58:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f6688eb6640 1 -- 172.21.15.26:0/3644717437 learned_addr learned my addr 172.21.15.26:0/3644717437 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 1 -- 172.21.15.26:0/3644717437 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503234444 0 0) 0x7f668409dc60 con 0x7f668409d1b0 2024-09-16T14:58:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient(hunting): my global_id is 314145 2024-09-16T14:58:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 1 -- 172.21.15.26:0/3644717437 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6674003530 con 0x7f668409d1b0 2024-09-16T14:58:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 1 -- 172.21.15.26:0/3644717437 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135394667 0 0) 0x7f668409b8f0 con 0x7f668409a550 2024-09-16T14:58:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient(hunting): my global_id is 284483 2024-09-16T14:58:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 1 -- 172.21.15.26:0/3644717437 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f668409dc60 con 0x7f668409a550 2024-09-16T14:58:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 1 -- 172.21.15.26:0/3644717437 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 326526254 0 0) 0x7f668409dc60 con 0x7f668409a550 2024-09-16T14:58:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 1 -- 172.21.15.26:0/3644717437 >> v1:172.21.15.189:6789/0 conn(0x7f668409b040 legacy=0x7f668413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 1 -- 172.21.15.26:0/3644717437 >> v1:172.21.15.26:6789/0 conn(0x7f668409d1b0 legacy=0x7f6684136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient: _finish_hunting 0 2024-09-16T14:58:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 1 monclient: found mon.noname-b 2024-09-16T14:58:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:58:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 1 -- 172.21.15.26:0/3644717437 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668409bbd0 con 0x7f668409a550 2024-09-16T14:58:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient: _finish_auth 0 2024-09-16T14:58:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient: _check_auth_tickets 2024-09-16T14:58:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 1 -- 172.21.15.26:0/3644717437 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6678001620 con 0x7f668409a550 2024-09-16T14:58:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:58:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient: dump: 2024-09-16T14:58:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.983 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 5 monclient: authenticate success, global_id 284483 2024-09-16T14:58:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 1 -- 172.21.15.26:0/3644717437 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66780017d0 con 0x7f668409a550 2024-09-16T14:58:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 1 -- 172.21.15.26:0/3644717437 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6678001ba0 con 0x7f668409a550 2024-09-16T14:58:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f66837fe640 10 monclient: dump: 2024-09-16T14:58:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.985 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 10 monclient: get_monmap_and_config success 2024-09-16T14:58:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 4 set_mon_vals no callback set 2024-09-16T14:58:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 10 monclient: shutdown 2024-09-16T14:58:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 1 -- 172.21.15.26:0/3644717437 >> v1:172.21.15.26:6790/0 conn(0x7f668409a550 legacy=0x7f668409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 1 -- 172.21.15.26:0/3644717437 shutdown_connections 2024-09-16T14:58:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 1 -- 172.21.15.26:0/3644717437 >> 172.21.15.26:0/3644717437 conn(0x7f668409fb50 msgr2=0x7f668409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 1 -- 172.21.15.26:0/3644717437 shutdown_connections 2024-09-16T14:58:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 1 -- 172.21.15.26:0/3644717437 wait complete. 2024-09-16T14:58:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 10 monclient: build_initial_monmap 2024-09-16T14:58:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 10 monclient: monmap: 2024-09-16T14:58:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:31.986 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:58:31.986 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:58:31.986 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:58:31.986 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:31.986 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:58:31.986 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:31.986 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:58:31.986 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:58:31.986 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 1 Processor -- start 2024-09-16T14:58:31.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 1 -- start start 2024-09-16T14:58:31.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.971+0000 7f668b11a640 10 monclient: init 2024-09-16T14:58:31.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:31.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:58:31.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 10 monclient(hunting): picked mon.noname-c con 0x7f668409a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:31.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 10 monclient(hunting): picked mon.noname-a con 0x7f668409b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:31.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 10 monclient(hunting): picked mon.noname-b con 0x7f668409d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:31.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668413d550 con 0x7f668409b040 2024-09-16T14:58:31.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6684139c40 con 0x7f668409d1b0 2024-09-16T14:58:31.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6684139960 con 0x7f668409a550 2024-09-16T14:58:31.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:31.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 10 monclient(hunting): authenticate will time out at 1936.096329s 2024-09-16T14:58:31.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f6683fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f668409b040 0x7f668413bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:32868/0 (socket says 172.21.15.26:32868) 2024-09-16T14:58:31.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f6683fff640 1 -- 172.21.15.26:0/4209413461 learned_addr learned my addr 172.21.15.26:0/4209413461 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:31.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679007936 0 0) 0x7f668413d550 con 0x7f668409b040 2024-09-16T14:58:31.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): my global_id is 314148 2024-09-16T14:58:31.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6668003530 con 0x7f668409b040 2024-09-16T14:58:31.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660685202 0 0) 0x7f6684139960 con 0x7f668409a550 2024-09-16T14:58:31.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): my global_id is 284486 2024-09-16T14:58:31.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f668413d550 con 0x7f668409a550 2024-09-16T14:58:31.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3480807985 0 0) 0x7f6668003530 con 0x7f668409b040 2024-09-16T14:58:31.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6684139960 con 0x7f668409b040 2024-09-16T14:58:31.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6670001a00 con 0x7f668409b040 2024-09-16T14:58:31.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:58:31.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): dump: 2024-09-16T14:58:31.989 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.989 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.989 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.989 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.989 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.989 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.989 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.989 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.989 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.989 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:58:31.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1862701188 0 0) 0x7f668413d550 con 0x7f668409a550 2024-09-16T14:58:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6668003530 con 0x7f668409a550 2024-09-16T14:58:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6678001ae0 con 0x7f668409a550 2024-09-16T14:58:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): dump: 2024-09-16T14:58:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.991 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498510729 0 0) 0x7f6684139c40 con 0x7f668409d1b0 2024-09-16T14:58:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): my global_id is 304231 2024-09-16T14:58:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f668413d550 con 0x7f668409d1b0 2024-09-16T14:58:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3472439278 0 0) 0x7f6668003530 con 0x7f668409a550 2024-09-16T14:58:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 >> v1:172.21.15.189:6789/0 conn(0x7f668409d1b0 legacy=0x7f66841d1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 >> v1:172.21.15.26:6789/0 conn(0x7f668409b040 legacy=0x7f668413bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient: _finish_hunting 0 2024-09-16T14:58:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 monclient: found mon.c 2024-09-16T14:58:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6684139f20 con 0x7f668409a550 2024-09-16T14:58:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient: _finish_auth 0 2024-09-16T14:58:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient: _check_auth_tickets 2024-09-16T14:58:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 5 monclient: authenticate success, global_id 284486 2024-09-16T14:58:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 10 monclient: _renew_subs 2024-09-16T14:58:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 1 -- 172.21.15.26:0/4209413461 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6684138510 con 0x7f668409a550 2024-09-16T14:58:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 10 monclient: _renew_subs 2024-09-16T14:58:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 1 -- 172.21.15.26:0/4209413461 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6684130500 con 0x7f668409a550 2024-09-16T14:58:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6678004aa0 con 0x7f668409a550 2024-09-16T14:58:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6678004db0 con 0x7f668409a550 2024-09-16T14:58:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f6682ffd640 4 set_mon_vals no callback set 2024-09-16T14:58:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient: dump: 2024-09-16T14:58:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.994 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f668b11a640 1 -- 172.21.15.26:0/4209413461 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6664006590 con 0x7f6664002830 2024-09-16T14:58:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f667800f820 con 0x7f668409a550 2024-09-16T14:58:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f6678043e60 con 0x7f668409a550 2024-09-16T14:58:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879454188 0 0) 0x7f6664006590 con 0x7f6664002830 2024-09-16T14:58:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient(hunting): my global_id is 284489 2024-09-16T14:58:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6668003530 con 0x7f6664002830 2024-09-16T14:58:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 580770941 0 0) 0x7f6668003530 con 0x7f6664002830 2024-09-16T14:58:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6664006590 con 0x7f6664002830 2024-09-16T14:58:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6670003a50 con 0x7f6664002830 2024-09-16T14:58:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66896b7640 10 monclient: get_auth_request con 0x7f66680363f0 auth_method 0 2024-09-16T14:58:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient: dump: 2024-09-16T14:58:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.996 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.996 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.996 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 174868015 0 0) 0x7f6664006590 con 0x7f6664002830 2024-09-16T14:58:31.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f666400a0e0 con 0x7f6664002830 2024-09-16T14:58:31.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6670003e50 con 0x7f6664002830 2024-09-16T14:58:31.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:31.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:58:31.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 >> v1:172.21.15.26:6790/0 conn(0x7f6664002830 legacy=0x7f6664004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:31.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.975+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 >> v1:172.21.15.26:6790/0 conn(0x7f6664002830 legacy=0x7f6664004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:31.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f668b11a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:58:31.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f668b11a640 1 -- 172.21.15.26:0/4209413461 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6664009d80 con 0x7f66640090a0 2024-09-16T14:58:31.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472185907 0 0) 0x7f6664009d80 con 0x7f66640090a0 2024-09-16T14:58:31.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 10 monclient(hunting): _negotiate 2024-09-16T14:58:31.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:31.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:31.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 10 monclient(hunting): my global_id is 284492 2024-09-16T14:58:31.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6664006590 con 0x7f66640090a0 2024-09-16T14:58:31.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 525857092 0 0) 0x7f6664006590 con 0x7f66640090a0 2024-09-16T14:58:31.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6664009d80 con 0x7f66640090a0 2024-09-16T14:58:31.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f66700043d0 con 0x7f66640090a0 2024-09-16T14:58:31.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:31.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:31.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 10 monclient: dump: 2024-09-16T14:58:31.999 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:31.999 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:31.999 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.999 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:31.999 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:31.999 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:31.999 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:31.999 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:31.999 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:31.999 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:31.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3091904373 0 0) 0x7f6664009d80 con 0x7f66640090a0 2024-09-16T14:58:32.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f666400c290 con 0x7f66640090a0 2024-09-16T14:58:32.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 2480106292) 0x7f6670007870 con 0x7f66640090a0 2024-09-16T14:58:32.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:58:32.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:58:32.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 >> v1:172.21.15.26:6790/0 conn(0x7f66640090a0 legacy=0x7f6664009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:32.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66817fa640 1 -- 172.21.15.26:0/4209413461 >> v1:172.21.15.26:6790/0 conn(0x7f66640090a0 legacy=0x7f6664009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:32.000 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:58:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:58:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:58:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:58:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 150, 2024-09-16T14:58:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:58:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:58:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:58:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:58:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:58:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:58:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:58:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:58:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:58:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:58:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:58:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:58:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:58:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:58:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:58:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:58:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:58:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:58:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:58:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:58:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:58:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:58:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:58:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:58:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:58:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:58:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:58:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:58:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:58:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:58:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:58:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:58:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:58:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:58:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:58:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:58:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:58:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:58:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:58:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:58:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:58:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:58:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:58:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:58:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:58:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:58:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:58:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:58:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:58:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:58:32.012 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:58:32.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f668b11a640 1 -- 172.21.15.26:0/4209413461 >> v1:172.21.15.189:6812/3196161582 conn(0x7f66680363f0 legacy=0x7f66680388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:32.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f668b11a640 10 monclient: shutdown 2024-09-16T14:58:32.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f668b11a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:32.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f668b11a640 1 -- 172.21.15.26:0/4209413461 >> v1:172.21.15.26:6790/0 conn(0x7f668409a550 legacy=0x7f668413b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:32.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66896b7640 1 -- 172.21.15.26:0/4209413461 reap_dead start 2024-09-16T14:58:32.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f66896b7640 1 -- 172.21.15.26:0/4209413461 reap_dead start 2024-09-16T14:58:32.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f668b11a640 1 -- 172.21.15.26:0/4209413461 shutdown_connections 2024-09-16T14:58:32.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f668b11a640 1 -- 172.21.15.26:0/4209413461 >> 172.21.15.26:0/4209413461 conn(0x7f668409fb50 msgr2=0x7f66840a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:32.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f668b11a640 1 -- 172.21.15.26:0/4209413461 shutdown_connections 2024-09-16T14:58:32.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:31.983+0000 7f668b11a640 1 -- 172.21.15.26:0/4209413461 wait complete. 2024-09-16T14:58:32.014 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:58:32.015 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:58:32.015 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:58:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.127+0000 7efe5ce6b640 10 monclient: get_monmap_and_config 2024-09-16T14:58:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.127+0000 7efe5ce6b640 10 monclient: build_initial_monmap 2024-09-16T14:58:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.127+0000 7efe5ce6b640 10 monclient: monmap: 2024-09-16T14:58:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:58:32.133176+0000 2024-09-16T14:58:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:58:32.133176+0000 2024-09-16T14:58:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:58:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:58:32.131 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 1 Processor -- start 2024-09-16T14:58:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 1 -- start start 2024-09-16T14:58:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 10 monclient: init 2024-09-16T14:58:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:58:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 10 monclient(hunting): picked mon.noname-c con 0x7efe5809cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 10 monclient(hunting): picked mon.noname-a con 0x7efe5809e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 10 monclient(hunting): picked mon.noname-b con 0x7efe5809d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5809d5b0 con 0x7efe5809e7a0 2024-09-16T14:58:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe58131a90 con 0x7efe5809cdc0 2024-09-16T14:58:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5809aff0 con 0x7efe5809d1a0 2024-09-16T14:58:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 10 monclient(hunting): authenticate will time out at 1936.253174s 2024-09-16T14:58:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe56d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7efe5809d1a0 0x7efe58139fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60854/0 (socket says 172.21.15.26:60854) 2024-09-16T14:58:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe55d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7efe5809e7a0 0x7efe58131380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:32878/0 (socket says 172.21.15.26:32878) 2024-09-16T14:58:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe56d9d640 1 -- 172.21.15.26:0/1381451689 learned_addr learned my addr 172.21.15.26:0/1381451689 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 1 -- 172.21.15.26:0/1381451689 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328072834 0 0) 0x7efe5809d5b0 con 0x7efe5809e7a0 2024-09-16T14:58:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient(hunting): _negotiate 2024-09-16T14:58:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient(hunting): my global_id is 314151 2024-09-16T14:58:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 1 -- 172.21.15.26:0/1381451689 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe44003530 con 0x7efe5809e7a0 2024-09-16T14:58:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 1 -- 172.21.15.26:0/1381451689 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799707380 0 0) 0x7efe5809aff0 con 0x7efe5809d1a0 2024-09-16T14:58:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient(hunting): _negotiate 2024-09-16T14:58:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient(hunting): my global_id is 284495 2024-09-16T14:58:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 1 -- 172.21.15.26:0/1381451689 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe5809d5b0 con 0x7efe5809d1a0 2024-09-16T14:58:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 1 -- 172.21.15.26:0/1381451689 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256454639 0 0) 0x7efe58131a90 con 0x7efe5809cdc0 2024-09-16T14:58:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient(hunting): _negotiate 2024-09-16T14:58:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient(hunting): my global_id is 304234 2024-09-16T14:58:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 1 -- 172.21.15.26:0/1381451689 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe5809aff0 con 0x7efe5809cdc0 2024-09-16T14:58:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 1 -- 172.21.15.26:0/1381451689 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 76585204 0 0) 0x7efe44003530 con 0x7efe5809e7a0 2024-09-16T14:58:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 1 -- 172.21.15.26:0/1381451689 >> v1:172.21.15.26:6790/0 conn(0x7efe5809d1a0 legacy=0x7efe58139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 1 -- 172.21.15.26:0/1381451689 >> v1:172.21.15.189:6789/0 conn(0x7efe5809cdc0 legacy=0x7efe58130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient: _finish_hunting 0 2024-09-16T14:58:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 1 monclient: found mon.noname-a 2024-09-16T14:58:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:58:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 1 -- 172.21.15.26:0/1381451689 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5809b2d0 con 0x7efe5809e7a0 2024-09-16T14:58:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient: _finish_auth 0 2024-09-16T14:58:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient: _check_auth_tickets 2024-09-16T14:58:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 1 -- 172.21.15.26:0/1381451689 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efe40001620 con 0x7efe5809e7a0 2024-09-16T14:58:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 5 monclient: authenticate success, global_id 314151 2024-09-16T14:58:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:58:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient: dump: 2024-09-16T14:58:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:32.141 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:32.142 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 1 -- 172.21.15.26:0/1381451689 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe40001890 con 0x7efe5809e7a0 2024-09-16T14:58:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 10 monclient: get_monmap_and_config success 2024-09-16T14:58:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 1 -- 172.21.15.26:0/1381451689 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efe40001c60 con 0x7efe5809e7a0 2024-09-16T14:58:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 4 set_mon_vals no callback set 2024-09-16T14:58:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:58:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5559a640 10 monclient: dump: 2024-09-16T14:58:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:32.142 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:32.143 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 10 monclient: shutdown 2024-09-16T14:58:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 1 -- 172.21.15.26:0/1381451689 >> v1:172.21.15.26:6789/0 conn(0x7efe5809e7a0 legacy=0x7efe58131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 1 -- 172.21.15.26:0/1381451689 shutdown_connections 2024-09-16T14:58:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 1 -- 172.21.15.26:0/1381451689 >> 172.21.15.26:0/1381451689 conn(0x7efe5809fb50 msgr2=0x7efe5809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 1 -- 172.21.15.26:0/1381451689 shutdown_connections 2024-09-16T14:58:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 1 -- 172.21.15.26:0/1381451689 wait complete. 2024-09-16T14:58:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 10 monclient: build_initial_monmap 2024-09-16T14:58:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 10 monclient: monmap: 2024-09-16T14:58:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:58:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:58:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:58:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:58:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:58:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:58:32.144 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.131+0000 7efe5ce6b640 1 Processor -- start 2024-09-16T14:58:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 1 -- start start 2024-09-16T14:58:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 10 monclient: init 2024-09-16T14:58:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:58:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 10 monclient(hunting): picked mon.noname-b con 0x7efe5809cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 10 monclient(hunting): picked mon.noname-c con 0x7efe5809d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 10 monclient(hunting): picked mon.noname-a con 0x7efe5809e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5811f2e0 con 0x7efe5809e7a0 2024-09-16T14:58:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe580a1620 con 0x7efe5809cdc0 2024-09-16T14:58:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5809a240 con 0x7efe5809d1a0 2024-09-16T14:58:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 10 monclient(hunting): authenticate will time out at 1936.256554s 2024-09-16T14:58:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe56d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7efe5809e7a0 0x7efe581d1c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:32894/0 (socket says 172.21.15.26:32894) 2024-09-16T14:58:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe55d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7efe5809d1a0 0x7efe581ce520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60864/0 (socket says 172.21.15.26:60864) 2024-09-16T14:58:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe56d9d640 1 -- 172.21.15.26:0/3882892068 learned_addr learned my addr 172.21.15.26:0/3882892068 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54592167 0 0) 0x7efe5811f2e0 con 0x7efe5809e7a0 2024-09-16T14:58:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): my global_id is 314154 2024-09-16T14:58:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe38003530 con 0x7efe5809e7a0 2024-09-16T14:58:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656287557 0 0) 0x7efe5809a240 con 0x7efe5809d1a0 2024-09-16T14:58:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): my global_id is 284498 2024-09-16T14:58:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe5811f2e0 con 0x7efe5809d1a0 2024-09-16T14:58:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128563946 0 0) 0x7efe580a1620 con 0x7efe5809cdc0 2024-09-16T14:58:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): my global_id is 304237 2024-09-16T14:58:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe5809a240 con 0x7efe5809cdc0 2024-09-16T14:58:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3640411064 0 0) 0x7efe38003530 con 0x7efe5809e7a0 2024-09-16T14:58:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe580a1620 con 0x7efe5809e7a0 2024-09-16T14:58:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efe50001d60 con 0x7efe5809e7a0 2024-09-16T14:58:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:58:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): dump: 2024-09-16T14:58:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:32.148 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:58:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3059311688 0 0) 0x7efe5811f2e0 con 0x7efe5809d1a0 2024-09-16T14:58:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe38003530 con 0x7efe5809d1a0 2024-09-16T14:58:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efe400015e0 con 0x7efe5809d1a0 2024-09-16T14:58:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): dump: 2024-09-16T14:58:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:32.149 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1012878158 0 0) 0x7efe580a1620 con 0x7efe5809e7a0 2024-09-16T14:58:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 >> v1:172.21.15.26:6790/0 conn(0x7efe5809d1a0 legacy=0x7efe581ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 >> v1:172.21.15.189:6789/0 conn(0x7efe5809cdc0 legacy=0x7efe58130bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient: _finish_hunting 0 2024-09-16T14:58:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 monclient: found mon.a 2024-09-16T14:58:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe581b9aa0 con 0x7efe5809e7a0 2024-09-16T14:58:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient: _finish_auth 0 2024-09-16T14:58:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient: _check_auth_tickets 2024-09-16T14:58:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 5 monclient: authenticate success, global_id 314154 2024-09-16T14:58:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 10 monclient: _renew_subs 2024-09-16T14:58:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 1 -- 172.21.15.26:0/3882892068 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5811dbb0 con 0x7efe5809e7a0 2024-09-16T14:58:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 10 monclient: _renew_subs 2024-09-16T14:58:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:58:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 1 -- 172.21.15.26:0/3882892068 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe58131300 con 0x7efe5809e7a0 2024-09-16T14:58:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe50002010 con 0x7efe5809e7a0 2024-09-16T14:58:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efe50004fc0 con 0x7efe5809e7a0 2024-09-16T14:58:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:58:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe54d99640 4 set_mon_vals no callback set 2024-09-16T14:58:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient: dump: 2024-09-16T14:58:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:32.152 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe5ce6b640 1 -- 172.21.15.26:0/3882892068 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efe3c006590 con 0x7efe3c002830 2024-09-16T14:58:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7efe5000fa30 con 0x7efe5809e7a0 2024-09-16T14:58:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7efe50044140 con 0x7efe5809e7a0 2024-09-16T14:58:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe55d9b640 10 monclient: get_auth_request con 0x7efe380363d0 auth_method 0 2024-09-16T14:58:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568401221 0 0) 0x7efe3c006590 con 0x7efe3c002830 2024-09-16T14:58:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient(hunting): my global_id is 304240 2024-09-16T14:58:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe580a1620 con 0x7efe3c002830 2024-09-16T14:58:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3423093661 0 0) 0x7efe580a1620 con 0x7efe3c002830 2024-09-16T14:58:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe3c006590 con 0x7efe3c002830 2024-09-16T14:58:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efe48001860 con 0x7efe3c002830 2024-09-16T14:58:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:58:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.135+0000 7efe4effd640 10 monclient: dump: 2024-09-16T14:58:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:32.156 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.139+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 944612391 0 0) 0x7efe3c006590 con 0x7efe3c002830 2024-09-16T14:58:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.139+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efe3c00a0e0 con 0x7efe3c002830 2024-09-16T14:58:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.139+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7efe48001b70 con 0x7efe3c002830 2024-09-16T14:58:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.139+0000 7efe4effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.139+0000 7efe4effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:58:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.139+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 >> v1:172.21.15.189:6789/0 conn(0x7efe3c002830 legacy=0x7efe3c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.139+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 >> v1:172.21.15.189:6789/0 conn(0x7efe3c002830 legacy=0x7efe3c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.143+0000 7efe5ce6b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:58:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.143+0000 7efe5ce6b640 1 -- 172.21.15.26:0/3882892068 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efe3c009da0 con 0x7efe3c0090c0 2024-09-16T14:58:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752415182 0 0) 0x7efe3c009da0 con 0x7efe3c0090c0 2024-09-16T14:58:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 10 monclient(hunting): _negotiate 2024-09-16T14:58:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 10 monclient(hunting): my global_id is 304243 2024-09-16T14:58:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe3c006590 con 0x7efe3c0090c0 2024-09-16T14:58:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3370951681 0 0) 0x7efe3c006590 con 0x7efe3c0090c0 2024-09-16T14:58:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe3c009da0 con 0x7efe3c0090c0 2024-09-16T14:58:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7efe48003840 con 0x7efe3c0090c0 2024-09-16T14:58:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:58:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 10 monclient: dump: 2024-09-16T14:58:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:32.159 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 880060213 0 0) 0x7efe3c009da0 con 0x7efe3c0090c0 2024-09-16T14:58:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efe3c00c290 con 0x7efe3c0090c0 2024-09-16T14:58:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3857934009) 0x7efe48003b70 con 0x7efe3c0090c0 2024-09-16T14:58:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:58:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:58:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 >> v1:172.21.15.189:6789/0 conn(0x7efe3c0090c0 legacy=0x7efe3c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe4effd640 1 -- 172.21.15.26:0/3882892068 >> v1:172.21.15.189:6789/0 conn(0x7efe3c0090c0 legacy=0x7efe3c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:32.160 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:58:32.160 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:58:32.160 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:58:32.160 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:58:32.160 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 150, 2024-09-16T14:58:32.160 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:58:32.160 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:58:32.160 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:58:32.160 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:58:32.160 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:32.160 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:58:32.160 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:32.161 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:58:32.161 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:58:32.161 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:32.161 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:32.161 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:32.161 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:32.161 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:32.161 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:32.161 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:32.161 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:32.161 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:32.161 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:32.161 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:32.161 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:58:32.162 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:58:32.162 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:32.162 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:32.162 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:32.162 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:32.162 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:32.162 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:32.162 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:32.162 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:32.162 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:32.162 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:32.162 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:32.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:58:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:58:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:58:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:58:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:58:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:58:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:58:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:58:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:58:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:58:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:58:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:58:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:58:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:58:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:58:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:58:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:58:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:58:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:58:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:58:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:58:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:58:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:58:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:58:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:58:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:58:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:58:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:58:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:58:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:58:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:58:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:58:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:58:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:58:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:58:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:58:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:58:32.170 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:58:32.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe5ce6b640 1 -- 172.21.15.26:0/3882892068 >> v1:172.21.15.189:6812/3196161582 conn(0x7efe380363d0 legacy=0x7efe38038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:32.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe5ce6b640 10 monclient: shutdown 2024-09-16T14:58:32.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe5ce6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:32.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe5ce6b640 1 -- 172.21.15.26:0/3882892068 >> v1:172.21.15.26:6789/0 conn(0x7efe5809e7a0 legacy=0x7efe581d1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:32.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe56d9d640 1 -- 172.21.15.26:0/3882892068 reap_dead start 2024-09-16T14:58:32.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe56d9d640 1 -- 172.21.15.26:0/3882892068 reap_dead start 2024-09-16T14:58:32.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe5ce6b640 1 -- 172.21.15.26:0/3882892068 shutdown_connections 2024-09-16T14:58:32.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe5ce6b640 1 -- 172.21.15.26:0/3882892068 >> 172.21.15.26:0/3882892068 conn(0x7efe5809fb50 msgr2=0x7efe581bc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:32.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe5ce6b640 1 -- 172.21.15.26:0/3882892068 shutdown_connections 2024-09-16T14:58:32.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:32.147+0000 7efe5ce6b640 1 -- 172.21.15.26:0/3882892068 wait complete. 2024-09-16T14:58:32.172 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:58:32.173 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T14:58:32.173 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-16T14:58:32.173 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T14:58:32.173 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:58:32.173 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T14:58:32.173 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:58:32.184 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T14:58:32.184 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T14:58:32.184 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T14:58:32.185 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T14:58:47.185 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T14:58:47.186 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T14:58:47.186 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:58:47.186 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:58:47.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce2c1b640 10 monclient: get_monmap_and_config 2024-09-16T14:58:47.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce2c1b640 10 monclient: build_initial_monmap 2024-09-16T14:58:47.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce2c1b640 10 monclient: monmap: 2024-09-16T14:58:47.320 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:47.320 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:47.320 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:58:47.322429+0000 2024-09-16T14:58:47.320 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:58:47.322429+0000 2024-09-16T14:58:47.321 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:47.321 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:47.321 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:47.321 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:58:47.321 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:58:47.321 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:47.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce2c1b640 1 Processor -- start 2024-09-16T14:58:47.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce2c1b640 1 -- start start 2024-09-16T14:58:47.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce2c1b640 10 monclient: init 2024-09-16T14:58:47.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce2c1b640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:47.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce2c1b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:58:47.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce2c1b640 10 monclient(hunting): picked mon.noname-b con 0x7f6cdc09a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:47.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce2c1b640 10 monclient(hunting): picked mon.noname-c con 0x7f6cdc09d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:47.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce2c1b640 10 monclient(hunting): picked mon.noname-a con 0x7f6cdc09b040 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:47.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce2c1b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc09dc60 con 0x7f6cdc09b040 2024-09-16T14:58:47.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce2c1b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc099ae0 con 0x7f6cdc09d1b0 2024-09-16T14:58:47.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce2c1b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc09b8f0 con 0x7f6cdc09a550 2024-09-16T14:58:47.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce2c1b640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:47.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce2c1b640 10 monclient(hunting): authenticate will time out at 1951.442650s 2024-09-16T14:58:47.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce11b8640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f6cdc09b040 legacy=0x7f6cdc13a190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:47.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce09b7640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6cdc09a550 0x7f6cdc09a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60916/0 (socket says 172.21.15.26:60916) 2024-09-16T14:58:47.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.319+0000 7f6ce09b7640 1 -- 172.21.15.26:0/440400144 learned_addr learned my addr 172.21.15.26:0/440400144 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:47.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.523+0000 7f6ce11b8640 1 -- 172.21.15.26:0/440400144 >> v1:172.21.15.26:6789/0 conn(0x7f6cdc09b040 legacy=0x7f6cdc13a190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:47.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:47.927+0000 7f6ce11b8640 1 -- 172.21.15.26:0/440400144 >> v1:172.21.15.26:6789/0 conn(0x7f6cdc09b040 legacy=0x7f6cdc13a190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:48.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:48.731+0000 7f6ce11b8640 1 -- 172.21.15.26:0/440400144 >> v1:172.21.15.26:6789/0 conn(0x7f6cdc09b040 legacy=0x7f6cdc13a190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:50.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.319+0000 7f6cda7fc640 10 monclient(hunting): tick 2024-09-16T14:58:50.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.319+0000 7f6cda7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:58:50.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.319+0000 7f6cda7fc640 1 monclient(hunting): continuing hunt 2024-09-16T14:58:50.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.319+0000 7f6cda7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:58:50.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.319+0000 7f6cda7fc640 1 -- 172.21.15.26:0/440400144 >> v1:172.21.15.26:6790/0 conn(0x7f6cdc09a550 legacy=0x7f6cdc09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:50.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.319+0000 7f6cda7fc640 1 -- 172.21.15.26:0/440400144 >> v1:172.21.15.189:6789/0 conn(0x7f6cdc09d1b0 legacy=0x7f6cdc136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:50.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.319+0000 7f6cda7fc640 1 -- 172.21.15.26:0/440400144 >> v1:172.21.15.26:6789/0 conn(0x7f6cdc09b040 legacy=0x7f6cdc13a190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:58:50.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.319+0000 7f6cda7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:58:50.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.319+0000 7f6cda7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f6cbc0036d0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:50.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.319+0000 7f6cda7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f6cbc007390 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:50.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.319+0000 7f6cda7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f6cbc00b0c0 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:50.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.319+0000 7f6cda7fc640 1 -- 172.21.15.26:0/440400144 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc09bbd0 con 0x7f6cbc0036d0 2024-09-16T14:58:50.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.319+0000 7f6cda7fc640 1 -- 172.21.15.26:0/440400144 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc09dc60 con 0x7f6cbc007390 2024-09-16T14:58:50.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.319+0000 7f6cda7fc640 1 -- 172.21.15.26:0/440400144 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cbc013010 con 0x7f6cbc00b0c0 2024-09-16T14:58:50.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.319+0000 7f6cda7fc640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:50.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.319+0000 7f6ce09b7640 1 -- 172.21.15.26:0/440400144 >> v1:172.21.15.26:6789/0 conn(0x7f6cbc0036d0 legacy=0x7f6cbc005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:50.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.523+0000 7f6ce09b7640 1 -- 172.21.15.26:0/440400144 >> v1:172.21.15.26:6789/0 conn(0x7f6cbc0036d0 legacy=0x7f6cbc005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:50.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:50.927+0000 7f6ce09b7640 1 -- 172.21.15.26:0/440400144 >> v1:172.21.15.26:6789/0 conn(0x7f6cbc0036d0 legacy=0x7f6cbc005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:51.731+0000 7f6ce09b7640 1 -- 172.21.15.26:0/440400144 >> v1:172.21.15.26:6789/0 conn(0x7f6cbc0036d0 legacy=0x7f6cbc005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:52.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.239+0000 7f6cdb7fe640 1 -- 172.21.15.26:0/440400144 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901878732 0 0) 0x7f6cdc09dc60 con 0x7f6cbc007390 2024-09-16T14:58:52.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.239+0000 7f6cdb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:58:52.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.239+0000 7f6cdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:52.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.239+0000 7f6cdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:52.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.239+0000 7f6cdb7fe640 10 monclient(hunting): my global_id is 304246 2024-09-16T14:58:52.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.239+0000 7f6cdb7fe640 1 -- 172.21.15.26:0/440400144 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc8003530 con 0x7f6cbc007390 2024-09-16T14:58:52.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 1 -- 172.21.15.26:0/440400144 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640807508 0 0) 0x7f6cbc013010 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:58:52.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:52.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:52.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 10 monclient(hunting): my global_id is 284501 2024-09-16T14:58:52.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 1 -- 172.21.15.26:0/440400144 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cdc09dc60 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 1 -- 172.21.15.26:0/440400144 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2944433283 0 0) 0x7f6cdc09dc60 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 1 -- 172.21.15.26:0/440400144 >> v1:172.21.15.189:6789/0 conn(0x7f6cbc007390 legacy=0x7f6cbc009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 1 -- 172.21.15.26:0/440400144 >> v1:172.21.15.26:6789/0 conn(0x7f6cbc0036d0 legacy=0x7f6cbc005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:58:52.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:58:52.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 1 monclient: found mon.noname-b 2024-09-16T14:58:52.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:52.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:58:52.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 1 -- 172.21.15.26:0/440400144 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cbc014220 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 10 monclient: _finish_auth 0 2024-09-16T14:58:52.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:58:52.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:52.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 1 -- 172.21.15.26:0/440400144 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6ccc004230 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:52.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:58:52.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6cdb7fe640 10 monclient: dump: 2024-09-16T14:58:52.254 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:52.254 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.255 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.255 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.255 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:52.255 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.255 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:52.255 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:52.255 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:52.255 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6ce2c1b640 5 monclient: authenticate success, global_id 284501 2024-09-16T14:58:52.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.251+0000 7f6ce2c1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:52.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6cdb7fe640 1 -- 172.21.15.26:0/440400144 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ccc004490 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6cdb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:52.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6cdb7fe640 1 -- 172.21.15.26:0/440400144 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6ccc004860 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6ce2c1b640 10 monclient: get_monmap_and_config success 2024-09-16T14:58:52.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6ce2c1b640 4 set_mon_vals no callback set 2024-09-16T14:58:52.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6cdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:52.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6cdb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:52.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6cdb7fe640 10 monclient: dump: 2024-09-16T14:58:52.259 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:52.259 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.259 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.259 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.259 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:52.259 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.260 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:52.260 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:52.260 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:52.260 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6ce2c1b640 10 monclient: shutdown 2024-09-16T14:58:52.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6ce2c1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:52.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6ce2c1b640 1 -- 172.21.15.26:0/440400144 >> v1:172.21.15.26:6790/0 conn(0x7f6cbc00b0c0 legacy=0x7f6cbc00d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6ce2c1b640 1 -- 172.21.15.26:0/440400144 shutdown_connections 2024-09-16T14:58:52.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6ce2c1b640 1 -- 172.21.15.26:0/440400144 >> 172.21.15.26:0/440400144 conn(0x7f6cdc09fb50 msgr2=0x7f6cdc09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:52.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6ce2c1b640 1 -- 172.21.15.26:0/440400144 shutdown_connections 2024-09-16T14:58:52.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6ce2c1b640 1 -- 172.21.15.26:0/440400144 wait complete. 2024-09-16T14:58:52.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6ce2c1b640 10 monclient: build_initial_monmap 2024-09-16T14:58:52.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.255+0000 7f6ce2c1b640 10 monclient: monmap: 2024-09-16T14:58:52.268 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:52.268 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:58:52.268 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:58:52.268 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:58:52.268 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:58:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:58:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:58:52.269 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 1 Processor -- start 2024-09-16T14:58:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 1 -- start start 2024-09-16T14:58:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 10 monclient: init 2024-09-16T14:58:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:58:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 10 monclient(hunting): picked mon.noname-b con 0x7f6cbc0036d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 10 monclient(hunting): picked mon.noname-a con 0x7f6cbc007390 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:52.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 10 monclient(hunting): picked mon.noname-c con 0x7f6cbc00b0c0 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc09a240 con 0x7f6cbc007390 2024-09-16T14:58:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc1b5500 con 0x7f6cbc0036d0 2024-09-16T14:58:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cdbfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f6cbc007390 legacy=0x7f6cdc1ce180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc1a1a40 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 10 monclient(hunting): authenticate will time out at 1956.381894s 2024-09-16T14:58:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce11b8640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6cbc00b0c0 0x7f6cdc1d18b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60944/0 (socket says 172.21.15.26:60944) 2024-09-16T14:58:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce11b8640 1 -- 172.21.15.26:0/2508748587 learned_addr learned my addr 172.21.15.26:0/2508748587 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557624100 0 0) 0x7f6cdc1a1a40 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient(hunting): _negotiate 2024-09-16T14:58:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:52.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient(hunting): my global_id is 284504 2024-09-16T14:58:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cbc0062d0 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2731190640 0 0) 0x7f6cbc0062d0 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cdc1a1a40 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674183546 0 0) 0x7f6cdc1b5500 con 0x7f6cbc0036d0 2024-09-16T14:58:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient(hunting): _negotiate 2024-09-16T14:58:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient(hunting): my global_id is 304249 2024-09-16T14:58:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cbc0062d0 con 0x7f6cbc0036d0 2024-09-16T14:58:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6ccc004600 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:58:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient(hunting): dump: 2024-09-16T14:58:52.272 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:52.272 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.272 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.272 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.272 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:52.272 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.272 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:52.272 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:52.272 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:52.272 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:58:52.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 637159836 0 0) 0x7f6cdc1a1a40 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 >> v1:172.21.15.189:6789/0 conn(0x7f6cbc0036d0 legacy=0x7f6cdc1b4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 >> v1:172.21.15.26:6789/0 conn(0x7f6cbc007390 legacy=0x7f6cdc1ce180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:58:52.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient: _finish_hunting 0 2024-09-16T14:58:52.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 1 monclient: found mon.c 2024-09-16T14:58:52.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:52.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:52.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc1a1d20 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient: _finish_auth 0 2024-09-16T14:58:52.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient: _check_auth_tickets 2024-09-16T14:58:52.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:52.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 5 monclient: authenticate success, global_id 284504 2024-09-16T14:58:52.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 10 monclient: _renew_subs 2024-09-16T14:58:52.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:52.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 1 -- 172.21.15.26:0/2508748587 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cdc0a2390 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 10 monclient: _renew_subs 2024-09-16T14:58:52.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:52.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6ce2c1b640 1 -- 172.21.15.26:0/2508748587 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cdc19fc30 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ccc004cc0 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:52.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6ccc004fd0 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:52.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:52.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cd97fa640 10 monclient: dump: 2024-09-16T14:58:52.274 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:52.274 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.274 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.274 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.274 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:52.275 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.275 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:52.275 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:52.275 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:52.275 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.259+0000 7f6cdaffd640 4 set_mon_vals no callback set 2024-09-16T14:58:52.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.263+0000 7f6ce2c1b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:52.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.263+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f6ccc00fa40 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.263+0000 7f6ce2c1b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:52.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.263+0000 7f6ce2c1b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:52.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.263+0000 7f6ce2c1b640 1 -- 172.21.15.26:0/2508748587 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc0005050 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.263+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f6ccc043d80 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.263+0000 7f6ce09b7640 10 monclient: get_auth_request con 0x7f6cbc037310 auth_method 0 2024-09-16T14:58:52.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.267+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6ccc071db0 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.267+0000 7f6cd97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:52.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.267+0000 7f6cd97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:58:52.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.491+0000 7f6ce2c1b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:58:52.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.491+0000 7f6ce2c1b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:58:52.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.491+0000 7f6ce2c1b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:52.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.491+0000 7f6ce2c1b640 1 -- 172.21.15.26:0/2508748587 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6cc0005880 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.491+0000 7f6cd97fa640 1 -- 172.21.15.26:0/2508748587 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 4089408612) 0x7f6ccc0145a0 con 0x7f6cbc00b0c0 2024-09-16T14:58:52.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.491+0000 7f6cd97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:58:52.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.491+0000 7f6cd97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:58:52.492 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":152,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:58:52.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.491+0000 7f6ce2c1b640 1 -- 172.21.15.26:0/2508748587 >> v1:172.21.15.189:6812/3196161582 conn(0x7f6cbc037310 legacy=0x7f6cbc0397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.491+0000 7f6ce2c1b640 10 monclient: shutdown 2024-09-16T14:58:52.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.491+0000 7f6ce2c1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:52.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.491+0000 7f6ce2c1b640 1 -- 172.21.15.26:0/2508748587 >> v1:172.21.15.26:6790/0 conn(0x7f6cbc00b0c0 legacy=0x7f6cdc1d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.491+0000 7f6ce2c1b640 1 -- 172.21.15.26:0/2508748587 shutdown_connections 2024-09-16T14:58:52.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.491+0000 7f6ce2c1b640 1 -- 172.21.15.26:0/2508748587 >> 172.21.15.26:0/2508748587 conn(0x7f6cdc09fb50 msgr2=0x7f6cdc071150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:52.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.491+0000 7f6ce2c1b640 1 -- 172.21.15.26:0/2508748587 shutdown_connections 2024-09-16T14:58:52.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.491+0000 7f6ce2c1b640 1 -- 172.21.15.26:0/2508748587 wait complete. 2024-09-16T14:58:52.511 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:58:52.511 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:58:52.511 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:58:52.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 10 monclient: get_monmap_and_config 2024-09-16T14:58:52.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 10 monclient: build_initial_monmap 2024-09-16T14:58:52.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 10 monclient: monmap: 2024-09-16T14:58:52.628 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:52.628 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.628 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:58:52.630779+0000 2024-09-16T14:58:52.628 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:58:52.630779+0000 2024-09-16T14:58:52.629 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:52.629 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.629 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:52.629 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:58:52.629 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:58:52.629 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 1 Processor -- start 2024-09-16T14:58:52.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 1 -- start start 2024-09-16T14:58:52.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 10 monclient: init 2024-09-16T14:58:52.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:52.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:58:52.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 10 monclient(hunting): picked mon.noname-a con 0x7f196809fe40 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:52.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 10 monclient(hunting): picked mon.noname-c con 0x7f19680a0930 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:52.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 10 monclient(hunting): picked mon.noname-b con 0x7f196809b4a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:52.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19680993e0 con 0x7f196809fe40 2024-09-16T14:58:52.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f196809ad20 con 0x7f19680a0930 2024-09-16T14:58:52.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19680a1920 con 0x7f196809b4a0 2024-09-16T14:58:52.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:52.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1966d9d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f196809fe40 legacy=0x7f19680a0220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:52.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 10 monclient(hunting): authenticate will time out at 1956.751002s 2024-09-16T14:58:52.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196759e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f196809b4a0 0x7f196813b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60960/0 (socket says 172.21.15.26:60960) 2024-09-16T14:58:52.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196759e640 1 -- 172.21.15.26:0/1104894630 learned_addr learned my addr 172.21.15.26:0/1104894630 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:52.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 1 -- 172.21.15.26:0/1104894630 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1612795338 0 0) 0x7f19680a1920 con 0x7f196809b4a0 2024-09-16T14:58:52.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:58:52.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:52.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:52.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 10 monclient(hunting): my global_id is 284507 2024-09-16T14:58:52.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 1 -- 172.21.15.26:0/1104894630 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1954003530 con 0x7f196809b4a0 2024-09-16T14:58:52.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 1 -- 172.21.15.26:0/1104894630 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2219848292 0 0) 0x7f1954003530 con 0x7f196809b4a0 2024-09-16T14:58:52.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 1 -- 172.21.15.26:0/1104894630 >> v1:172.21.15.189:6789/0 conn(0x7f19680a0930 legacy=0x7f1968137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 1 -- 172.21.15.26:0/1104894630 >> v1:172.21.15.26:6789/0 conn(0x7f196809fe40 legacy=0x7f19680a0220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:58:52.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:58:52.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 1 monclient: found mon.noname-b 2024-09-16T14:58:52.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:52.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:58:52.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 1 -- 172.21.15.26:0/1104894630 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f196809a480 con 0x7f196809b4a0 2024-09-16T14:58:52.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 10 monclient: _finish_auth 0 2024-09-16T14:58:52.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:58:52.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:52.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 1 -- 172.21.15.26:0/1104894630 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1960001620 con 0x7f196809b4a0 2024-09-16T14:58:52.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:52.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:58:52.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 10 monclient: dump: 2024-09-16T14:58:52.635 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:52.635 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.635 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.635 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.635 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:52.635 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.635 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:52.635 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:52.636 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:52.636 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 5 monclient: authenticate success, global_id 284507 2024-09-16T14:58:52.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:52.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 1 -- 172.21.15.26:0/1104894630 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1960001940 con 0x7f196809b4a0 2024-09-16T14:58:52.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:52.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 1 -- 172.21.15.26:0/1104894630 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1960001d10 con 0x7f196809b4a0 2024-09-16T14:58:52.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 10 monclient: get_monmap_and_config success 2024-09-16T14:58:52.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f196d0f5640 4 set_mon_vals no callback set 2024-09-16T14:58:52.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:52.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:52.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.627+0000 7f1965d9b640 10 monclient: dump: 2024-09-16T14:58:52.636 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:52.637 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.637 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.637 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.637 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:52.637 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.637 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:52.637 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:52.637 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:52.637 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 10 monclient: shutdown 2024-09-16T14:58:52.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:52.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 1 -- 172.21.15.26:0/1104894630 >> v1:172.21.15.26:6790/0 conn(0x7f196809b4a0 legacy=0x7f196813b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 1 -- 172.21.15.26:0/1104894630 shutdown_connections 2024-09-16T14:58:52.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 1 -- 172.21.15.26:0/1104894630 >> 172.21.15.26:0/1104894630 conn(0x7f19680a5540 msgr2=0x7f19680a5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:52.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 1 -- 172.21.15.26:0/1104894630 shutdown_connections 2024-09-16T14:58:52.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 1 -- 172.21.15.26:0/1104894630 wait complete. 2024-09-16T14:58:52.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 10 monclient: build_initial_monmap 2024-09-16T14:58:52.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 10 monclient: monmap: 2024-09-16T14:58:52.638 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:52.638 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:58:52.638 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:58:52.638 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:58:52.638 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:52.638 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:58:52.638 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:52.638 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:58:52.638 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:58:52.638 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 1 Processor -- start 2024-09-16T14:58:52.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 1 -- start start 2024-09-16T14:58:52.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 10 monclient: init 2024-09-16T14:58:52.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:52.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:58:52.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 10 monclient(hunting): picked mon.noname-b con 0x7f196809b4a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:52.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 10 monclient(hunting): picked mon.noname-a con 0x7f196809fe40 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:52.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 10 monclient(hunting): picked mon.noname-c con 0x7f19680a0930 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:52.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19681c6740 con 0x7f196809fe40 2024-09-16T14:58:52.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f196806c530 con 0x7f196809b4a0 2024-09-16T14:58:52.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f196809c360 con 0x7f19680a0930 2024-09-16T14:58:52.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:52.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 10 monclient(hunting): authenticate will time out at 1956.753938s 2024-09-16T14:58:52.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196659c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f196809fe40 legacy=0x7f19681ce650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:52.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196759e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f19680a0930 0x7f19681d1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60974/0 (socket says 172.21.15.26:60974) 2024-09-16T14:58:52.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196759e640 1 -- 172.21.15.26:0/2870452933 learned_addr learned my addr 172.21.15.26:0/2870452933 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:52.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150055432 0 0) 0x7f196809c360 con 0x7f19680a0930 2024-09-16T14:58:52.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:58:52.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:52.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:52.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 10 monclient(hunting): my global_id is 284510 2024-09-16T14:58:52.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1948003530 con 0x7f19680a0930 2024-09-16T14:58:52.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 350546643 0 0) 0x7f1948003530 con 0x7f19680a0930 2024-09-16T14:58:52.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f196809c360 con 0x7f19680a0930 2024-09-16T14:58:52.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f19600015e0 con 0x7f19680a0930 2024-09-16T14:58:52.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:52.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:58:52.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 10 monclient(hunting): dump: 2024-09-16T14:58:52.646 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:52.646 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.646 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.646 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.646 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:52.646 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.646 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:52.646 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:52.646 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:52.646 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:58:52.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3830471672 0 0) 0x7f196809c360 con 0x7f19680a0930 2024-09-16T14:58:52.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 >> v1:172.21.15.189:6789/0 conn(0x7f196809b4a0 legacy=0x7f19681b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 >> v1:172.21.15.26:6789/0 conn(0x7f196809fe40 legacy=0x7f19681ce650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:58:52.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:58:52.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 1 monclient: found mon.c 2024-09-16T14:58:52.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:52.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:52.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1968099ae0 con 0x7f19680a0930 2024-09-16T14:58:52.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 10 monclient: _finish_auth 0 2024-09-16T14:58:52.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:58:52.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:52.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 5 monclient: authenticate success, global_id 284510 2024-09-16T14:58:52.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 10 monclient: _renew_subs 2024-09-16T14:58:52.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:52.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 1 -- 172.21.15.26:0/2870452933 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19681a7600 con 0x7f19680a0930 2024-09-16T14:58:52.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 10 monclient: _renew_subs 2024-09-16T14:58:52.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:52.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 1 -- 172.21.15.26:0/2870452933 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f196806cd10 con 0x7f19680a0930 2024-09-16T14:58:52.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1960004c00 con 0x7f19680a0930 2024-09-16T14:58:52.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:52.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1960004f90 con 0x7f19680a0930 2024-09-16T14:58:52.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:52.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:52.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 10 monclient: dump: 2024-09-16T14:58:52.648 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:52.648 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.648 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.648 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.648 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:52.649 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.649 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:52.649 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:52.649 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:52.649 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196559a640 4 set_mon_vals no callback set 2024-09-16T14:58:52.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:52.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f196d0f5640 1 -- 172.21.15.26:0/2870452933 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f194c006590 con 0x7f194c002830 2024-09-16T14:58:52.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.631+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1960005170 con 0x7f19680a0930 2024-09-16T14:58:52.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f19600440b0 con 0x7f19680a0930 2024-09-16T14:58:52.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002539845 0 0) 0x7f194c006590 con 0x7f194c002830 2024-09-16T14:58:52.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:58:52.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:52.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:52.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 10 monclient(hunting): my global_id is 284513 2024-09-16T14:58:52.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f196809c360 con 0x7f194c002830 2024-09-16T14:58:52.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3372526102 0 0) 0x7f196809c360 con 0x7f194c002830 2024-09-16T14:58:52.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f194c006590 con 0x7f194c002830 2024-09-16T14:58:52.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1958004330 con 0x7f194c002830 2024-09-16T14:58:52.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:52.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:52.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 10 monclient: dump: 2024-09-16T14:58:52.650 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:52.650 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.650 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.650 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.650 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:52.651 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.651 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:52.651 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:52.651 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:52.651 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f196659c640 10 monclient: get_auth_request con 0x7f1948035fe0 auth_method 0 2024-09-16T14:58:52.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3099443449 0 0) 0x7f194c006590 con 0x7f194c002830 2024-09-16T14:58:52.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f194c00a0e0 con 0x7f194c002830 2024-09-16T14:58:52.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1958006670 con 0x7f194c002830 2024-09-16T14:58:52.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:52.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:58:52.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 >> v1:172.21.15.26:6790/0 conn(0x7f194c002830 legacy=0x7f194c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.635+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 >> v1:172.21.15.26:6790/0 conn(0x7f194c002830 legacy=0x7f194c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:52.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.639+0000 7f196d0f5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:58:52.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.639+0000 7f196d0f5640 1 -- 172.21.15.26:0/2870452933 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f194c009da0 con 0x7f194c0090c0 2024-09-16T14:58:52.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.639+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610462833 0 0) 0x7f194c009da0 con 0x7f194c0090c0 2024-09-16T14:58:52.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.639+0000 7f195f7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:58:52.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.639+0000 7f195f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:52.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.639+0000 7f195f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:52.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.639+0000 7f195f7fe640 10 monclient(hunting): my global_id is 284516 2024-09-16T14:58:52.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.639+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f194c006590 con 0x7f194c0090c0 2024-09-16T14:58:52.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.639+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1497235572 0 0) 0x7f194c006590 con 0x7f194c0090c0 2024-09-16T14:58:52.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.639+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f194c009da0 con 0x7f194c0090c0 2024-09-16T14:58:52.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.639+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1958004b30 con 0x7f194c0090c0 2024-09-16T14:58:52.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.639+0000 7f195f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:52.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.639+0000 7f195f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:52.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.639+0000 7f195f7fe640 10 monclient: dump: 2024-09-16T14:58:52.653 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:52.653 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.653 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.653 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.653 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:52.653 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.653 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:52.653 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:52.653 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:52.653 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.639+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 563588240 0 0) 0x7f194c009da0 con 0x7f194c0090c0 2024-09-16T14:58:52.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.639+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f194c00c290 con 0x7f194c0090c0 2024-09-16T14:58:52.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.643+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4203 (unknown 0 0 3407806782) 0x7f1958005fe0 con 0x7f194c0090c0 2024-09-16T14:58:52.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.643+0000 7f195f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:58:52.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.643+0000 7f195f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:58:52.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.643+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 >> v1:172.21.15.26:6790/0 conn(0x7f194c0090c0 legacy=0x7f194c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.643+0000 7f195f7fe640 1 -- 172.21.15.26:0/2870452933 >> v1:172.21.15.26:6790/0 conn(0x7f194c0090c0 legacy=0x7f194c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:52.654 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:58:52.654 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:58:52.654 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:58:52.654 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:58:52.655 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 152, 2024-09-16T14:58:52.655 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:58:52.655 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:58:52.655 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:58:52.655 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:52.655 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:58:52.655 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:52.655 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:58:52.655 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:58:52.655 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:52.655 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:52.655 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:52.655 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:52.656 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:52.656 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:52.656 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:52.656 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:52.656 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:52.656 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:52.656 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:52.656 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:58:52.656 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:58:52.656 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:52.656 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:52.656 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:52.656 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:52.657 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:52.657 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:52.657 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:52.657 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:52.657 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:52.657 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:52.657 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:52.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.657 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:58:52.657 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:58:52.657 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:58:52.657 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:58:52.657 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:58:52.658 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:58:52.658 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:52.658 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:52.658 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:58:52.658 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:58:52.658 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:58:52.658 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:58:52.658 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:58:52.658 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:58:52.658 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:58:52.658 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:52.658 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:58:52.658 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:52.658 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:52.659 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:52.659 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:52.659 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:52.659 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:52.659 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:52.659 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:52.659 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:52.659 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:52.659 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:52.659 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:58:52.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.659 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:58:52.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.660 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:58:52.660 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:58:52.660 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:52.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:52.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.660 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:52.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:58:52.660 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:52.660 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:52.660 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:52.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:58:52.660 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:58:52.660 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:52.661 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:52.661 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:52.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.661 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:58:52.661 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:58:52.661 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:52.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:52.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.661 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:52.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:58:52.661 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:52.661 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:52.661 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:52.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.662 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:58:52.662 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:58:52.662 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:52.662 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:52.662 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:52.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.662 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:58:52.662 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:58:52.662 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:52.662 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:52.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.662 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:52.663 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:58:52.663 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:52.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:52.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:52.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.663 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:58:52.663 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:58:52.663 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:52.663 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:52.663 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:52.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:52.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:52.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.663 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:58:52.664 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:58:52.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.664 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:52.664 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:52.664 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:58:52.664 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:52.664 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:52.664 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:58:52.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.664 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:52.664 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:52.664 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:58:52.664 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:52.665 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:52.665 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:58:52.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.665 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:52.665 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:52.665 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:58:52.665 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:52.665 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:52.665 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:58:52.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.665 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:52.665 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:52.665 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:58:52.665 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:52.666 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:52.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.666 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:58:52.666 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:58:52.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.643+0000 7f196d0f5640 1 -- 172.21.15.26:0/2870452933 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1948035fe0 legacy=0x7f19480384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.643+0000 7f196d0f5640 10 monclient: shutdown 2024-09-16T14:58:52.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.643+0000 7f196d0f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:52.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.643+0000 7f196d0f5640 1 -- 172.21.15.26:0/2870452933 >> v1:172.21.15.26:6790/0 conn(0x7f19680a0930 legacy=0x7f19681d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.643+0000 7f196759e640 1 -- 172.21.15.26:0/2870452933 reap_dead start 2024-09-16T14:58:52.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.643+0000 7f196759e640 1 -- 172.21.15.26:0/2870452933 reap_dead start 2024-09-16T14:58:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.643+0000 7f196d0f5640 1 -- 172.21.15.26:0/2870452933 shutdown_connections 2024-09-16T14:58:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.643+0000 7f196d0f5640 1 -- 172.21.15.26:0/2870452933 >> 172.21.15.26:0/2870452933 conn(0x7f19680a5540 msgr2=0x7f1968071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.643+0000 7f196d0f5640 1 -- 172.21.15.26:0/2870452933 shutdown_connections 2024-09-16T14:58:52.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.643+0000 7f196d0f5640 1 -- 172.21.15.26:0/2870452933 wait complete. 2024-09-16T14:58:52.668 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:58:52.668 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T14:58:52.668 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:58:52.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.779+0000 7fec4be0b640 10 monclient: get_monmap_and_config 2024-09-16T14:58:52.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.779+0000 7fec4be0b640 10 monclient: build_initial_monmap 2024-09-16T14:58:52.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.779+0000 7fec4be0b640 10 monclient: monmap: 2024-09-16T14:58:52.783 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:52.783 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.783 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:58:52.784900+0000 2024-09-16T14:58:52.783 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:58:52.784900+0000 2024-09-16T14:58:52.783 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:52.783 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.783 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:52.783 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:58:52.783 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:58:52.783 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 1 Processor -- start 2024-09-16T14:58:52.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 1 -- start start 2024-09-16T14:58:52.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 10 monclient: init 2024-09-16T14:58:52.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:52.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:58:52.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 10 monclient(hunting): picked mon.noname-c con 0x7fec4409a550 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:52.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 10 monclient(hunting): picked mon.noname-a con 0x7fec4409d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:52.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 10 monclient(hunting): picked mon.noname-b con 0x7fec4409b040 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:52.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec4409dc60 con 0x7fec4409d1b0 2024-09-16T14:58:52.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec44099ae0 con 0x7fec4409a550 2024-09-16T14:58:52.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec4409b8f0 con 0x7fec4409b040 2024-09-16T14:58:52.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:52.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec493a6640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fec4409d1b0 legacy=0x7fec44136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:52.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 10 monclient(hunting): authenticate will time out at 1956.905122s 2024-09-16T14:58:52.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4a3a8640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fec4409b040 0x7fec4413a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60996/0 (socket says 172.21.15.26:60996) 2024-09-16T14:58:52.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4a3a8640 1 -- 172.21.15.26:0/2762446272 learned_addr learned my addr 172.21.15.26:0/2762446272 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:52.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 1 -- 172.21.15.26:0/2762446272 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751015035 0 0) 0x7fec4409b8f0 con 0x7fec4409b040 2024-09-16T14:58:52.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 10 monclient(hunting): _negotiate 2024-09-16T14:58:52.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:52.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:52.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 10 monclient(hunting): my global_id is 284519 2024-09-16T14:58:52.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 1 -- 172.21.15.26:0/2762446272 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec30003530 con 0x7fec4409b040 2024-09-16T14:58:52.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 1 -- 172.21.15.26:0/2762446272 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1894317446 0 0) 0x7fec30003530 con 0x7fec4409b040 2024-09-16T14:58:52.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 1 -- 172.21.15.26:0/2762446272 >> v1:172.21.15.189:6789/0 conn(0x7fec4409a550 legacy=0x7fec4409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 1 -- 172.21.15.26:0/2762446272 >> v1:172.21.15.26:6789/0 conn(0x7fec4409d1b0 legacy=0x7fec44136a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:58:52.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 10 monclient: _finish_hunting 0 2024-09-16T14:58:52.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 1 monclient: found mon.noname-b 2024-09-16T14:58:52.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:52.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:58:52.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 1 -- 172.21.15.26:0/2762446272 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4409bbd0 con 0x7fec4409b040 2024-09-16T14:58:52.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 10 monclient: _finish_auth 0 2024-09-16T14:58:52.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 10 monclient: _check_auth_tickets 2024-09-16T14:58:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 1 -- 172.21.15.26:0/2762446272 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec3c001620 con 0x7fec4409b040 2024-09-16T14:58:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 5 monclient: authenticate success, global_id 284519 2024-09-16T14:58:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:58:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 10 monclient: dump: 2024-09-16T14:58:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:52.789 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:52.790 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:52.790 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:58:52.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 1 -- 172.21.15.26:0/2762446272 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec3c001940 con 0x7fec4409b040 2024-09-16T14:58:52.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:52.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 1 -- 172.21.15.26:0/2762446272 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec3c001cd0 con 0x7fec4409b040 2024-09-16T14:58:52.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 10 monclient: get_monmap_and_config success 2024-09-16T14:58:52.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 4 set_mon_vals no callback set 2024-09-16T14:58:52.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:52.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:52.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec48ba5640 10 monclient: dump: 2024-09-16T14:58:52.790 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:52.790 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.790 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.791 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.791 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:52.791 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.791 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:52.791 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:52.791 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:52.791 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 10 monclient: shutdown 2024-09-16T14:58:52.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:52.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 1 -- 172.21.15.26:0/2762446272 >> v1:172.21.15.26:6790/0 conn(0x7fec4409b040 legacy=0x7fec4413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 1 -- 172.21.15.26:0/2762446272 shutdown_connections 2024-09-16T14:58:52.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 1 -- 172.21.15.26:0/2762446272 >> 172.21.15.26:0/2762446272 conn(0x7fec4409fb50 msgr2=0x7fec4409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:52.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 1 -- 172.21.15.26:0/2762446272 shutdown_connections 2024-09-16T14:58:52.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 1 -- 172.21.15.26:0/2762446272 wait complete. 2024-09-16T14:58:52.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 10 monclient: build_initial_monmap 2024-09-16T14:58:52.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 10 monclient: monmap: 2024-09-16T14:58:52.792 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:58:52.792 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:58:52.792 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:58:52.792 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:58:52.792 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:58:52.792 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:58:52.792 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:58:52.792 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:58:52.792 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:58:52.792 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.783+0000 7fec4be0b640 1 Processor -- start 2024-09-16T14:58:52.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 1 -- start start 2024-09-16T14:58:52.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 10 monclient: init 2024-09-16T14:58:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 10 monclient: _reopen_session rank -1 2024-09-16T14:58:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:58:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 10 monclient(hunting): picked mon.noname-c con 0x7fec4409a550 addr v1:172.21.15.26:6790/0 2024-09-16T14:58:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 10 monclient(hunting): picked mon.noname-b con 0x7fec4409b040 addr v1:172.21.15.189:6789/0 2024-09-16T14:58:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 10 monclient(hunting): picked mon.noname-a con 0x7fec4409d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T14:58:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec441a5da0 con 0x7fec4409d1b0 2024-09-16T14:58:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec441a6070 con 0x7fec4409b040 2024-09-16T14:58:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4a3a8640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fec4409d1b0 legacy=0x7fec441d19b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:58:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec441a51d0 con 0x7fec4409a550 2024-09-16T14:58:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 10 monclient(hunting): _renew_subs 2024-09-16T14:58:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 10 monclient(hunting): authenticate will time out at 1956.908768s 2024-09-16T14:58:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec49ba7640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fec4409a550 0x7fec441b4b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:32774/0 (socket says 172.21.15.26:32774) 2024-09-16T14:58:52.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec49ba7640 1 -- 172.21.15.26:0/288219010 learned_addr learned my addr 172.21.15.26:0/288219010 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:58:52.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919134215 0 0) 0x7fec441a51d0 con 0x7fec4409a550 2024-09-16T14:58:52.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:58:52.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:52.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:52.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient(hunting): my global_id is 284522 2024-09-16T14:58:52.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec28003530 con 0x7fec4409a550 2024-09-16T14:58:52.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2069018837 0 0) 0x7fec28003530 con 0x7fec4409a550 2024-09-16T14:58:52.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec441a51d0 con 0x7fec4409a550 2024-09-16T14:58:52.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec40001cc0 con 0x7fec4409a550 2024-09-16T14:58:52.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:52.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:58:52.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient(hunting): dump: 2024-09-16T14:58:52.798 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:52.799 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.799 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.799 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.799 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:52.799 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.799 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:52.799 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:52.799 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:52.799 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:58:52.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135660531 0 0) 0x7fec441a6070 con 0x7fec4409b040 2024-09-16T14:58:52.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:58:52.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:52.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:52.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient(hunting): my global_id is 304261 2024-09-16T14:58:52.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec28003530 con 0x7fec4409b040 2024-09-16T14:58:52.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3871536518 0 0) 0x7fec441a51d0 con 0x7fec4409a550 2024-09-16T14:58:52.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 >> v1:172.21.15.189:6789/0 conn(0x7fec4409b040 legacy=0x7fec441ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 >> v1:172.21.15.26:6789/0 conn(0x7fec4409d1b0 legacy=0x7fec441d19b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:58:52.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:58:52.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 monclient: found mon.c 2024-09-16T14:58:52.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:58:52.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:52.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec441a6c90 con 0x7fec4409a550 2024-09-16T14:58:52.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient: _finish_auth 0 2024-09-16T14:58:52.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:58:52.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:58:52.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 5 monclient: authenticate success, global_id 284522 2024-09-16T14:58:52.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 10 monclient: _renew_subs 2024-09-16T14:58:52.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:52.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 1 -- 172.21.15.26:0/288219010 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec441a6f70 con 0x7fec4409a550 2024-09-16T14:58:52.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 10 monclient: _renew_subs 2024-09-16T14:58:52.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:58:52.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 1 -- 172.21.15.26:0/288219010 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec441b5230 con 0x7fec4409a550 2024-09-16T14:58:52.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec40002090 con 0x7fec4409a550 2024-09-16T14:58:52.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:58:52.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec40003e70 con 0x7fec4409a550 2024-09-16T14:58:52.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:52.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3b7fe640 4 set_mon_vals no callback set 2024-09-16T14:58:52.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:58:52.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient: dump: 2024-09-16T14:58:52.802 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:52.802 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.802 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.802 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.802 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:52.802 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.802 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:52.802 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:52.802 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:52.802 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:52.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4be0b640 1 -- 172.21.15.26:0/288219010 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fec2c006590 con 0x7fec2c002830 2024-09-16T14:58:52.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fec40003fd0 con 0x7fec4409a550 2024-09-16T14:58:52.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fec40044260 con 0x7fec4409a550 2024-09-16T14:58:52.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec4a3a8640 10 monclient: get_auth_request con 0x7fec28036130 auth_method 0 2024-09-16T14:58:52.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49739763 0 0) 0x7fec2c006590 con 0x7fec2c002830 2024-09-16T14:58:52.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:58:52.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:52.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:52.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient(hunting): my global_id is 304264 2024-09-16T14:58:52.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec441a51d0 con 0x7fec2c002830 2024-09-16T14:58:52.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 428658832 0 0) 0x7fec441a51d0 con 0x7fec2c002830 2024-09-16T14:58:52.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec2c006590 con 0x7fec2c002830 2024-09-16T14:58:52.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec34004070 con 0x7fec2c002830 2024-09-16T14:58:52.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:52.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:58:52.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.787+0000 7fec3a7fc640 10 monclient: dump: 2024-09-16T14:58:52.804 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:52.804 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.804 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.804 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.804 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:52.804 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.804 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:52.804 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:52.804 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:52.804 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.791+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3462776469 0 0) 0x7fec2c006590 con 0x7fec2c002830 2024-09-16T14:58:52.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.791+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec2c00a0e0 con 0x7fec2c002830 2024-09-16T14:58:52.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.791+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fec34006370 con 0x7fec2c002830 2024-09-16T14:58:52.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.791+0000 7fec3a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:58:52.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.791+0000 7fec3a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:58:52.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.791+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 >> v1:172.21.15.189:6789/0 conn(0x7fec2c002830 legacy=0x7fec2c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.791+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 >> v1:172.21.15.189:6789/0 conn(0x7fec2c002830 legacy=0x7fec2c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:52.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.795+0000 7fec4be0b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:58:52.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.795+0000 7fec4be0b640 1 -- 172.21.15.26:0/288219010 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fec2c009d80 con 0x7fec2c0090a0 2024-09-16T14:58:52.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089296641 0 0) 0x7fec2c009d80 con 0x7fec2c0090a0 2024-09-16T14:58:52.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:58:52.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:58:52.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:58:52.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 10 monclient(hunting): my global_id is 304267 2024-09-16T14:58:52.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec2c006590 con 0x7fec2c0090a0 2024-09-16T14:58:52.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3017773454 0 0) 0x7fec2c006590 con 0x7fec2c0090a0 2024-09-16T14:58:52.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec2c009d80 con 0x7fec2c0090a0 2024-09-16T14:58:52.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fec34004860 con 0x7fec2c0090a0 2024-09-16T14:58:52.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:58:52.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:58:52.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 10 monclient: dump: 2024-09-16T14:58:52.807 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:58:52.807 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:58:52.807 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.807 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:58:52.807 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:58:52.807 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:58:52.808 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:58:52.808 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:58:52.808 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:58:52.808 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:58:52.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1661162029 0 0) 0x7fec2c009d80 con 0x7fec2c0090a0 2024-09-16T14:58:52.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fec2c00c290 con 0x7fec2c0090a0 2024-09-16T14:58:52.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3869 (unknown 0 0 1819261116) 0x7fec34005c80 con 0x7fec2c0090a0 2024-09-16T14:58:52.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:58:52.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:58:52.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 >> v1:172.21.15.189:6789/0 conn(0x7fec2c0090a0 legacy=0x7fec2c009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec3a7fc640 1 -- 172.21.15.26:0/288219010 >> v1:172.21.15.189:6789/0 conn(0x7fec2c0090a0 legacy=0x7fec2c009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:58:52.809 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:58:52.809 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:58:52.809 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:58:52.809 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:58:52.809 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 152, 2024-09-16T14:58:52.809 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:58:52.809 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:58:52.809 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:58:52.809 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:52.809 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:58:52.809 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:52.809 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:58:52.810 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:58:52.810 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:52.810 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:52.810 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:52.810 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:52.810 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:52.810 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:52.810 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:52.810 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:52.810 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:52.810 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:52.810 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:52.810 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:58:52.810 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:58:52.811 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:52.811 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:52.811 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:52.811 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:52.811 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:52.811 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:52.811 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:52.811 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:52.811 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:52.811 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:52.811 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:52.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.811 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:58:52.812 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:58:52.812 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:58:52.812 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:58:52.812 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:58:52.812 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:58:52.812 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:52.812 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:58:52.812 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:58:52.812 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:58:52.812 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:58:52.812 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:58:52.812 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:58:52.812 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:58:52.812 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:58:52.813 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:58:52.813 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:58:52.813 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:58:52.813 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:58:52.813 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:58:52.813 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:58:52.813 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:58:52.813 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:58:52.813 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:58:52.813 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:58:52.813 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:58:52.813 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:58:52.813 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:52.814 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:58:52.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.814 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:58:52.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.814 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:58:52.814 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:58:52.814 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:52.814 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:52.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.814 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:52.814 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:58:52.814 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:52.814 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:52.814 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:52.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:58:52.815 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:58:52.815 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:52.815 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:52.815 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:52.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.815 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:58:52.815 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:58:52.815 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:52.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:52.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:52.816 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:58:52.816 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:52.816 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:52.816 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:52.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.816 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:58:52.816 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:58:52.816 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:52.816 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:52.816 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:52.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.816 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:58:52.817 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:58:52.817 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:58:52.817 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:58:52.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.817 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:58:52.817 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:58:52.817 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:58:52.817 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:52.817 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:52.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.817 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:58:52.817 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:58:52.817 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:58:52.817 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:58:52.818 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:58:52.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:52.818 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:52.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.818 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:58:52.818 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:58:52.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.818 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:52.818 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:52.818 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:58:52.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:52.818 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:58:52.818 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:58:52.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:58:52.819 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:58:52.819 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:58:52.819 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:58:52.819 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:58:52.819 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:58:52.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:58:52.819 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:58:52.819 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:58:52.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec4be0b640 1 -- 172.21.15.26:0/288219010 >> v1:172.21.15.189:6812/3196161582 conn(0x7fec28036130 legacy=0x7fec280385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec4be0b640 10 monclient: shutdown 2024-09-16T14:58:52.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec4be0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:58:52.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec4be0b640 1 -- 172.21.15.26:0/288219010 >> v1:172.21.15.26:6790/0 conn(0x7fec4409a550 legacy=0x7fec441b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:58:52.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec4a3a8640 1 -- 172.21.15.26:0/288219010 reap_dead start 2024-09-16T14:58:52.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec4a3a8640 1 -- 172.21.15.26:0/288219010 reap_dead start 2024-09-16T14:58:52.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec4be0b640 1 -- 172.21.15.26:0/288219010 shutdown_connections 2024-09-16T14:58:52.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec4be0b640 1 -- 172.21.15.26:0/288219010 >> 172.21.15.26:0/288219010 conn(0x7fec4409fb50 msgr2=0x7fec441b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:58:52.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec4be0b640 1 -- 172.21.15.26:0/288219010 shutdown_connections 2024-09-16T14:58:52.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:58:52.799+0000 7fec4be0b640 1 -- 172.21.15.26:0/288219010 wait complete. 2024-09-16T14:58:52.821 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:58:52.821 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T14:59:12.822 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:59:12.823 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T14:59:12.823 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T14:59:12.823 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T14:59:12.827 INFO:tasks.ceph.mon.a:Started 2024-09-16T14:59:12.827 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T14:59:12.827 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T14:59:12.827 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T14:59:27.828 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T14:59:27.829 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T14:59:27.829 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:59:27.829 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:59:27.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a4798640 10 monclient: get_monmap_and_config 2024-09-16T14:59:27.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a4798640 10 monclient: build_initial_monmap 2024-09-16T14:59:27.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a4798640 10 monclient: monmap: 2024-09-16T14:59:27.961 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:27.961 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:27.961 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:59:27.963409+0000 2024-09-16T14:59:27.961 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:59:27.963409+0000 2024-09-16T14:59:27.961 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:27.961 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:27.962 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:27.962 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:59:27.962 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:59:27.962 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:27.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a4798640 1 Processor -- start 2024-09-16T14:59:27.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a4798640 1 -- start start 2024-09-16T14:59:27.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a4798640 10 monclient: init 2024-09-16T14:59:27.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a4798640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:27.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a4798640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:59:27.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a4798640 10 monclient(hunting): picked mon.noname-c con 0x7fe39c09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:27.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a4798640 10 monclient(hunting): picked mon.noname-a con 0x7fe39c132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:27.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a4798640 10 monclient(hunting): picked mon.noname-b con 0x7fe39c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:27.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a4798640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39c09d260 con 0x7fe39c132650 2024-09-16T14:59:27.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a4798640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39c132a80 con 0x7fe39c09cb80 2024-09-16T14:59:27.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a4798640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39c09cf60 con 0x7fe39c138ff0 2024-09-16T14:59:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a4798640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a4798640 10 monclient(hunting): authenticate will time out at 1992.083423s 2024-09-16T14:59:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a1d33640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe39c132650 0x7fe39c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34546/0 (socket says 172.21.15.26:34546) 2024-09-16T14:59:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a1d33640 1 -- 172.21.15.26:0/3581135434 learned_addr learned my addr 172.21.15.26:0/3581135434 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a1d33640 1 -- 172.21.15.26:0/3581135434 >> v1:172.21.15.26:6789/0 conn(0x7fe39c132650 legacy=0x7fe39c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T14:59:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a1d33640 1 -- 172.21.15.26:0/3581135434 >> v1:172.21.15.26:6789/0 conn(0x7fe39c132650 legacy=0x7fe39c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:59:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a1d33640 1 --1- 172.21.15.26:0/3581135434 >> v1:172.21.15.26:6789/0 conn(0x7fe39c132650 0x7fe39c1378d0 :-1 s=OPENED pgs=1109 cs=1 l=1).handle_message read tag failed 2024-09-16T14:59:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a1d33640 1 --1- 172.21.15.26:0/3581135434 >> v1:172.21.15.26:6789/0 conn(0x7fe39c132650 0x7fe39c1378d0 :-1 s=OPENED pgs=1109 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:59:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a1532640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6789/0 2024-09-16T14:59:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.958+0000 7fe3a1532640 1 -- 172.21.15.26:0/3581135434 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855311065 0 0) 0x7fe39c09cf60 con 0x7fe39c138ff0 2024-09-16T14:59:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 10 monclient(hunting): _negotiate 2024-09-16T14:59:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 10 monclient(hunting): my global_id is 284525 2024-09-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 1 -- 172.21.15.26:0/3581135434 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe388003530 con 0x7fe39c138ff0 2024-09-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 1 -- 172.21.15.26:0/3581135434 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2798932304 0 0) 0x7fe388003530 con 0x7fe39c138ff0 2024-09-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 1 -- 172.21.15.26:0/3581135434 >> v1:172.21.15.189:6789/0 conn(0x7fe39c09cb80 legacy=0x7fe39c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 1 -- 172.21.15.26:0/3581135434 >> v1:172.21.15.26:6789/0 conn(0x7fe39c132650 legacy=0x7fe39c1378d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 10 monclient: _finish_hunting 0 2024-09-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 1 monclient: found mon.noname-b 2024-09-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 1 -- 172.21.15.26:0/3581135434 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39c09f2d0 con 0x7fe39c138ff0 2024-09-16T14:59:27.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 10 monclient: _finish_auth 0 2024-09-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 10 monclient: _check_auth_tickets 2024-09-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 1 -- 172.21.15.26:0/3581135434 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe3940019e0 con 0x7fe39c138ff0 2024-09-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 5 monclient: authenticate success, global_id 284525 2024-09-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 10 monclient: dump: 2024-09-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:27.971 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:27.972 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:27.972 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:27.972 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:27.972 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:27.972 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:27.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:27.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 1 -- 172.21.15.26:0/3581135434 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe394001c00 con 0x7fe39c138ff0 2024-09-16T14:59:27.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:27.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 10 monclient: get_monmap_and_config success 2024-09-16T14:59:27.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 1 -- 172.21.15.26:0/3581135434 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe394001fd0 con 0x7fe39c138ff0 2024-09-16T14:59:27.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:27.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 4 set_mon_vals no callback set 2024-09-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a1532640 10 monclient: dump: 2024-09-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:27.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 10 monclient: shutdown 2024-09-16T14:59:27.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:27.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 1 -- 172.21.15.26:0/3581135434 >> v1:172.21.15.26:6790/0 conn(0x7fe39c138ff0 legacy=0x7fe39c13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:27.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 1 -- 172.21.15.26:0/3581135434 shutdown_connections 2024-09-16T14:59:27.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 1 -- 172.21.15.26:0/3581135434 >> 172.21.15.26:0/3581135434 conn(0x7fe39c09fb50 msgr2=0x7fe39c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:27.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 1 -- 172.21.15.26:0/3581135434 shutdown_connections 2024-09-16T14:59:27.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 1 -- 172.21.15.26:0/3581135434 wait complete. 2024-09-16T14:59:27.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 10 monclient: build_initial_monmap 2024-09-16T14:59:27.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 10 monclient: monmap: 2024-09-16T14:59:27.974 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:27.974 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:59:27.974 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:59:27.974 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:59:27.974 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:27.975 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:59:27.975 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:27.975 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:59:27.975 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:59:27.975 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:27.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 1 Processor -- start 2024-09-16T14:59:27.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 1 -- start start 2024-09-16T14:59:27.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 10 monclient: init 2024-09-16T14:59:27.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:27.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:59:27.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 10 monclient(hunting): picked mon.noname-a con 0x7fe39c09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:27.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 10 monclient(hunting): picked mon.noname-c con 0x7fe39c132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:27.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 10 monclient(hunting): picked mon.noname-b con 0x7fe39c138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:27.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39c06e400 con 0x7fe39c09cb80 2024-09-16T14:59:27.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39c09a240 con 0x7fe39c138ff0 2024-09-16T14:59:27.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39c13c570 con 0x7fe39c132650 2024-09-16T14:59:27.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:27.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 10 monclient(hunting): authenticate will time out at 1992.086746s 2024-09-16T14:59:27.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a2534640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe39c09cb80 0x7fe39c06d5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34554/0 (socket says 172.21.15.26:34554) 2024-09-16T14:59:27.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a2534640 1 -- 172.21.15.26:0/353914291 learned_addr learned my addr 172.21.15.26:0/353914291 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:27.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824331374 0 0) 0x7fe39c13c570 con 0x7fe39c132650 2024-09-16T14:59:27.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:59:27.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:27.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:27.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): my global_id is 284528 2024-09-16T14:59:27.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe380003530 con 0x7fe39c132650 2024-09-16T14:59:27.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053054591 0 0) 0x7fe39c06e400 con 0x7fe39c09cb80 2024-09-16T14:59:27.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:59:27.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:27.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:27.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): my global_id is 324099 2024-09-16T14:59:27.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe39c13c570 con 0x7fe39c09cb80 2024-09-16T14:59:27.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3899971847 0 0) 0x7fe39c13c570 con 0x7fe39c09cb80 2024-09-16T14:59:27.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe39c06e400 con 0x7fe39c09cb80 2024-09-16T14:59:27.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe38c001880 con 0x7fe39c09cb80 2024-09-16T14:59:27.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:27.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:59:27.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): dump: 2024-09-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1458665226 0 0) 0x7fe380003530 con 0x7fe39c132650 2024-09-16T14:59:27.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe39c13c570 con 0x7fe39c132650 2024-09-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe3980018c0 con 0x7fe39c132650 2024-09-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): dump: 2024-09-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:27.979 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:27.980 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:27.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861640863 0 0) 0x7fe39c09a240 con 0x7fe39c138ff0 2024-09-16T14:59:27.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:59:27.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:27.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:27.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient(hunting): my global_id is 304273 2024-09-16T14:59:27.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe380003530 con 0x7fe39c138ff0 2024-09-16T14:59:27.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2209501194 0 0) 0x7fe39c06e400 con 0x7fe39c09cb80 2024-09-16T14:59:27.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 >> v1:172.21.15.26:6790/0 conn(0x7fe39c132650 legacy=0x7fe39c06dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:27.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 >> v1:172.21.15.189:6789/0 conn(0x7fe39c138ff0 legacy=0x7fe39c1cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:27.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:59:27.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 monclient: found mon.a 2024-09-16T14:59:27.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39c071560 con 0x7fe39c09cb80 2024-09-16T14:59:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient: _finish_auth 0 2024-09-16T14:59:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:59:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 5 monclient: authenticate success, global_id 324099 2024-09-16T14:59:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 10 monclient: _renew_subs 2024-09-16T14:59:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 1 -- 172.21.15.26:0/353914291 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39c138030 con 0x7fe39c09cb80 2024-09-16T14:59:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 10 monclient: _renew_subs 2024-09-16T14:59:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a4798640 1 -- 172.21.15.26:0/353914291 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39c06ff80 con 0x7fe39c09cb80 2024-09-16T14:59:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe38c003620 con 0x7fe39c09cb80 2024-09-16T14:59:27.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:27.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe38c0039b0 con 0x7fe39c09cb80 2024-09-16T14:59:27.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:27.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe3a0d31640 4 set_mon_vals no callback set 2024-09-16T14:59:27.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:27.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.962+0000 7fe392ffd640 10 monclient: dump: 2024-09-16T14:59:27.982 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:27.982 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:27.982 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:27.982 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:27.982 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:27.982 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:27.982 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:27.983 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:27.983 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:27.983 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:27.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.966+0000 7fe3a4798640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:27.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.966+0000 7fe3a4798640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:27.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.966+0000 7fe3a4798640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:27.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.966+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe38c003b10 con 0x7fe39c09cb80 2024-09-16T14:59:27.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.966+0000 7fe3a4798640 1 -- 172.21.15.26:0/353914291 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe384005050 con 0x7fe39c09cb80 2024-09-16T14:59:27.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.966+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fe38c043c70 con 0x7fe39c09cb80 2024-09-16T14:59:27.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.966+0000 7fe3a1d33640 10 monclient: get_auth_request con 0x7fe3800363b0 auth_method 0 2024-09-16T14:59:27.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.970+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe38c0145b0 con 0x7fe39c09cb80 2024-09-16T14:59:27.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.970+0000 7fe392ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:27.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:27.970+0000 7fe392ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:59:28.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.190+0000 7fe3a4798640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:59:28.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.190+0000 7fe3a4798640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:59:28.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.190+0000 7fe3a4798640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:28.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.190+0000 7fe3a4798640 1 -- 172.21.15.26:0/353914291 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe384005880 con 0x7fe39c09cb80 2024-09-16T14:59:28.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.190+0000 7fe392ffd640 1 -- 172.21.15.26:0/353914291 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1563033096) 0x7fe38c012a20 con 0x7fe39c09cb80 2024-09-16T14:59:28.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.190+0000 7fe392ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:59:28.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.190+0000 7fe392ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:59:28.195 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:59:28.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.194+0000 7fe3a4798640 1 -- 172.21.15.26:0/353914291 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe3800363b0 legacy=0x7fe380038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.194+0000 7fe3a4798640 10 monclient: shutdown 2024-09-16T14:59:28.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.194+0000 7fe3a4798640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:28.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.194+0000 7fe3a4798640 1 -- 172.21.15.26:0/353914291 >> v1:172.21.15.26:6789/0 conn(0x7fe39c09cb80 legacy=0x7fe39c06d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.194+0000 7fe3a4798640 1 -- 172.21.15.26:0/353914291 shutdown_connections 2024-09-16T14:59:28.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.194+0000 7fe3a4798640 1 -- 172.21.15.26:0/353914291 >> 172.21.15.26:0/353914291 conn(0x7fe39c09fb50 msgr2=0x7fe39c12eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:28.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.194+0000 7fe3a4798640 1 -- 172.21.15.26:0/353914291 shutdown_connections 2024-09-16T14:59:28.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.194+0000 7fe3a4798640 1 -- 172.21.15.26:0/353914291 wait complete. 2024-09-16T14:59:28.212 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:59:28.212 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:59:28.212 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:59:28.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 10 monclient: get_monmap_and_config 2024-09-16T14:59:28.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 10 monclient: build_initial_monmap 2024-09-16T14:59:28.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 10 monclient: monmap: 2024-09-16T14:59:28.332 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:28.332 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.332 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:59:28.334230+0000 2024-09-16T14:59:28.332 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:59:28.334230+0000 2024-09-16T14:59:28.332 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:28.333 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.333 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:28.333 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:59:28.333 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:59:28.333 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 1 Processor -- start 2024-09-16T14:59:28.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 1 -- start start 2024-09-16T14:59:28.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 10 monclient: init 2024-09-16T14:59:28.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:28.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:59:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 10 monclient(hunting): picked mon.noname-c con 0x7f184c070dc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 10 monclient(hunting): picked mon.noname-b con 0x7f184c1348d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 10 monclient(hunting): picked mon.noname-a con 0x7f184c134d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c06d6d0 con 0x7f184c134d10 2024-09-16T14:59:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c0a10d0 con 0x7f184c070dc0 2024-09-16T14:59:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c06d080 con 0x7f184c1348d0 2024-09-16T14:59:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 10 monclient(hunting): authenticate will time out at 1992.454535s 2024-09-16T14:59:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f184a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f184c1348d0 0x7f184c139a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48012/0 (socket says 172.21.15.26:48012) 2024-09-16T14:59:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f184a59c640 1 -- 172.21.15.26:0/67507930 learned_addr learned my addr 172.21.15.26:0/67507930 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 1 -- 172.21.15.26:0/67507930 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222216642 0 0) 0x7f184c06d080 con 0x7f184c1348d0 2024-09-16T14:59:28.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient(hunting): my global_id is 284531 2024-09-16T14:59:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 1 -- 172.21.15.26:0/67507930 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1834003530 con 0x7f184c1348d0 2024-09-16T14:59:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 1 -- 172.21.15.26:0/67507930 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515151491 0 0) 0x7f184c06d6d0 con 0x7f184c134d10 2024-09-16T14:59:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient(hunting): my global_id is 324102 2024-09-16T14:59:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 1 -- 172.21.15.26:0/67507930 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f184c06d080 con 0x7f184c134d10 2024-09-16T14:59:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 1 -- 172.21.15.26:0/67507930 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149678578 0 0) 0x7f184c0a10d0 con 0x7f184c070dc0 2024-09-16T14:59:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient(hunting): my global_id is 304276 2024-09-16T14:59:28.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 1 -- 172.21.15.26:0/67507930 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f184c06d6d0 con 0x7f184c070dc0 2024-09-16T14:59:28.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 1 -- 172.21.15.26:0/67507930 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 746720487 0 0) 0x7f184c06d080 con 0x7f184c134d10 2024-09-16T14:59:28.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 1 -- 172.21.15.26:0/67507930 >> v1:172.21.15.26:6790/0 conn(0x7f184c1348d0 legacy=0x7f184c139a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 1 -- 172.21.15.26:0/67507930 >> v1:172.21.15.189:6789/0 conn(0x7f184c070dc0 legacy=0x7f184c0711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient: _finish_hunting 0 2024-09-16T14:59:28.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 1 monclient: found mon.noname-a 2024-09-16T14:59:28.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:28.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:59:28.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 1 -- 172.21.15.26:0/67507930 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c06d360 con 0x7f184c134d10 2024-09-16T14:59:28.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient: _finish_auth 0 2024-09-16T14:59:28.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient: _check_auth_tickets 2024-09-16T14:59:28.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:28.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 1 -- 172.21.15.26:0/67507930 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f183c001600 con 0x7f184c134d10 2024-09-16T14:59:28.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 5 monclient: authenticate success, global_id 324102 2024-09-16T14:59:28.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:28.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:59:28.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient: dump: 2024-09-16T14:59:28.341 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.342 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.342 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.342 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.342 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.342 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.342 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.342 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.342 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.342 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:28.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 1 -- 172.21.15.26:0/67507930 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f183c001820 con 0x7f184c134d10 2024-09-16T14:59:28.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:28.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 1 -- 172.21.15.26:0/67507930 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f183c001b30 con 0x7f184c134d10 2024-09-16T14:59:28.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 10 monclient: get_monmap_and_config success 2024-09-16T14:59:28.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1851044640 4 set_mon_vals no callback set 2024-09-16T14:59:28.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:28.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.330+0000 7f1849d9b640 10 monclient: dump: 2024-09-16T14:59:28.343 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.343 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.343 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.343 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.343 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.343 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.343 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.343 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.344 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.344 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 10 monclient: shutdown 2024-09-16T14:59:28.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:28.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 1 -- 172.21.15.26:0/67507930 >> v1:172.21.15.26:6789/0 conn(0x7f184c134d10 legacy=0x7f184c13d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 1 -- 172.21.15.26:0/67507930 shutdown_connections 2024-09-16T14:59:28.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 1 -- 172.21.15.26:0/67507930 >> 172.21.15.26:0/67507930 conn(0x7f184c0a3e30 msgr2=0x7f184c0a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:28.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 1 -- 172.21.15.26:0/67507930 shutdown_connections 2024-09-16T14:59:28.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 1 -- 172.21.15.26:0/67507930 wait complete. 2024-09-16T14:59:28.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 10 monclient: build_initial_monmap 2024-09-16T14:59:28.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 10 monclient: monmap: 2024-09-16T14:59:28.344 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:28.344 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:59:28.345 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:59:28.345 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:59:28.345 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:28.345 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:59:28.345 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:28.345 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:59:28.345 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:59:28.345 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 1 Processor -- start 2024-09-16T14:59:28.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 1 -- start start 2024-09-16T14:59:28.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 10 monclient: init 2024-09-16T14:59:28.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:28.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:59:28.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 10 monclient(hunting): picked mon.noname-b con 0x7f184c070dc0 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:28.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 10 monclient(hunting): picked mon.noname-c con 0x7f184c1348d0 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:28.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 10 monclient(hunting): picked mon.noname-a con 0x7f184c134d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:28.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c139540 con 0x7f184c134d10 2024-09-16T14:59:28.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c13aab0 con 0x7f184c070dc0 2024-09-16T14:59:28.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c137970 con 0x7f184c1348d0 2024-09-16T14:59:28.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:28.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 10 monclient(hunting): authenticate will time out at 1992.458382s 2024-09-16T14:59:28.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f184a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f184c1348d0 0x7f184c137260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48016/0 (socket says 172.21.15.26:48016) 2024-09-16T14:59:28.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f184a59c640 1 -- 172.21.15.26:0/3526847897 learned_addr learned my addr 172.21.15.26:0/3526847897 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:28.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1143114451 0 0) 0x7f184c139540 con 0x7f184c134d10 2024-09-16T14:59:28.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): my global_id is 324105 2024-09-16T14:59:28.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1838003530 con 0x7f184c134d10 2024-09-16T14:59:28.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995102365 0 0) 0x7f184c137970 con 0x7f184c1348d0 2024-09-16T14:59:28.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): my global_id is 284534 2024-09-16T14:59:28.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f184c139540 con 0x7f184c1348d0 2024-09-16T14:59:28.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 14910609 0 0) 0x7f1838003530 con 0x7f184c134d10 2024-09-16T14:59:28.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f184c137970 con 0x7f184c134d10 2024-09-16T14:59:28.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f183c001500 con 0x7f184c134d10 2024-09-16T14:59:28.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:59:28.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): dump: 2024-09-16T14:59:28.353 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.353 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.353 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.353 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.353 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.353 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.353 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.353 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.353 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.354 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:59:28.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2328376153 0 0) 0x7f184c139540 con 0x7f184c1348d0 2024-09-16T14:59:28.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1838003530 con 0x7f184c1348d0 2024-09-16T14:59:28.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 108038210 0 0) 0x7f184c13aab0 con 0x7f184c070dc0 2024-09-16T14:59:28.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): my global_id is 304279 2024-09-16T14:59:28.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f184c139540 con 0x7f184c070dc0 2024-09-16T14:59:28.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1840001a00 con 0x7f184c1348d0 2024-09-16T14:59:28.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:28.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient(hunting): dump: 2024-09-16T14:59:28.355 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.355 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.355 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.355 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.355 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.355 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.355 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.355 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.355 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.355 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2244534473 0 0) 0x7f184c137970 con 0x7f184c134d10 2024-09-16T14:59:28.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 >> v1:172.21.15.26:6790/0 conn(0x7f184c1348d0 legacy=0x7f184c137260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 >> v1:172.21.15.189:6789/0 conn(0x7f184c070dc0 legacy=0x7f184c136b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient: _finish_hunting 0 2024-09-16T14:59:28.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 monclient: found mon.a 2024-09-16T14:59:28.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:28.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:28.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c09fca0 con 0x7f184c134d10 2024-09-16T14:59:28.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient: _finish_auth 0 2024-09-16T14:59:28.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient: _check_auth_tickets 2024-09-16T14:59:28.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:28.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 5 monclient: authenticate success, global_id 324105 2024-09-16T14:59:28.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 10 monclient: _renew_subs 2024-09-16T14:59:28.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:28.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 1 -- 172.21.15.26:0/3526847897 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f184c133fe0 con 0x7f184c134d10 2024-09-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 10 monclient: _renew_subs 2024-09-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f1851044640 1 -- 172.21.15.26:0/3526847897 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f184c1390a0 con 0x7f184c134d10 2024-09-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f183c0038b0 con 0x7f184c134d10 2024-09-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f183c003bc0 con 0x7f184c134d10 2024-09-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f184959a640 4 set_mon_vals no callback set 2024-09-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.334+0000 7f18337fe640 10 monclient: dump: 2024-09-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.357 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.358 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.358 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.358 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.358 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.358 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.358 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f1851044640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:28.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f1851044640 1 -- 172.21.15.26:0/3526847897 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f182c006590 con 0x7f182c002830 2024-09-16T14:59:28.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f183c00e630 con 0x7f184c134d10 2024-09-16T14:59:28.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f183c042bb0 con 0x7f184c134d10 2024-09-16T14:59:28.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689061548 0 0) 0x7f182c006590 con 0x7f182c002830 2024-09-16T14:59:28.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 10 monclient(hunting): my global_id is 324108 2024-09-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f184c137970 con 0x7f182c002830 2024-09-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2009461822 0 0) 0x7f184c137970 con 0x7f182c002830 2024-09-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f182c006590 con 0x7f182c002830 2024-09-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1844001a90 con 0x7f182c002830 2024-09-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 10 monclient: dump: 2024-09-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.359 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.360 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.360 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.360 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.360 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.360 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f184a59c640 10 monclient: get_auth_request con 0x7f18380363e0 auth_method 0 2024-09-16T14:59:28.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1997018838 0 0) 0x7f182c006590 con 0x7f182c002830 2024-09-16T14:59:28.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f182c00a0e0 con 0x7f182c002830 2024-09-16T14:59:28.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1844001f60 con 0x7f182c002830 2024-09-16T14:59:28.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:28.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:59:28.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 >> v1:172.21.15.26:6789/0 conn(0x7f182c002830 legacy=0x7f182c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.338+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 >> v1:172.21.15.26:6789/0 conn(0x7f182c002830 legacy=0x7f182c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.342+0000 7f1851044640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.342+0000 7f1851044640 1 -- 172.21.15.26:0/3526847897 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f182c009d80 con 0x7f182c0090a0 2024-09-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.342+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714508937 0 0) 0x7f182c009d80 con 0x7f182c0090a0 2024-09-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.342+0000 7f18337fe640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.342+0000 7f18337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.342+0000 7f18337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.342+0000 7f18337fe640 10 monclient(hunting): my global_id is 324111 2024-09-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.342+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f182c006590 con 0x7f182c0090a0 2024-09-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.342+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3729065801 0 0) 0x7f182c006590 con 0x7f182c0090a0 2024-09-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.342+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f182c009d80 con 0x7f182c0090a0 2024-09-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.342+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1844003c10 con 0x7f182c0090a0 2024-09-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.342+0000 7f18337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.342+0000 7f18337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.342+0000 7f18337fe640 10 monclient: dump: 2024-09-16T14:59:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.362 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3829693577 0 0) 0x7f182c009d80 con 0x7f182c0090a0 2024-09-16T14:59:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f182c00c290 con 0x7f182c0090a0 2024-09-16T14:59:28.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 1292033709) 0x7f18440049d0 con 0x7f182c0090a0 2024-09-16T14:59:28.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f18337fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:59:28.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f18337fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:59:28.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 >> v1:172.21.15.26:6789/0 conn(0x7f182c0090a0 legacy=0x7f182c009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f18337fe640 1 -- 172.21.15.26:0/3526847897 >> v1:172.21.15.26:6789/0 conn(0x7f182c0090a0 legacy=0x7f182c009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:28.363 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:59:28.363 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:59:28.363 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:59:28.363 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:59:28.363 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 156, 2024-09-16T14:59:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:59:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:59:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:59:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:59:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:59:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:59:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:59:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:59:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:28.364 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:59:28.366 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:59:28.367 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:59:28.367 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:59:28.367 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:59:28.367 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:59:28.367 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:59:28.367 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:59:28.367 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:59:28.367 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:59:28.367 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:59:28.367 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:59:28.367 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:59:28.367 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:28.367 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:28.368 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:28.368 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:28.368 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:28.368 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:28.368 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:28.368 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:28.368 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:28.368 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:28.368 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:28.368 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:59:28.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.368 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:59:28.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.369 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:59:28.369 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:59:28.369 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:28.369 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:28.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.369 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:28.369 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:59:28.369 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:28.369 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.369 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.369 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:59:28.369 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:59:28.369 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:28.370 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:28.370 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:28.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.370 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:59:28.370 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:59:28.370 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:28.370 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:28.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.370 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:28.370 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:59:28.370 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:28.370 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.370 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.371 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:59:28.371 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:59:28.371 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:28.371 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:28.371 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:28.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.371 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:59:28.372 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:59:28.372 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:28.372 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:28.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.372 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:28.372 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:59:28.372 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:28.372 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.372 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.372 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:59:28.372 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:59:28.373 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:28.373 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:28.373 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:28.373 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.373 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.373 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:59:28.373 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:59:28.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.373 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:28.373 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:28.373 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:59:28.373 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.373 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:28.374 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:59:28.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.374 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:28.374 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:28.374 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:59:28.374 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.374 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.374 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:59:28.374 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:59:28.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f1851044640 1 -- 172.21.15.26:0/3526847897 >> v1:172.21.15.189:6812/3196161582 conn(0x7f18380363e0 legacy=0x7f18380388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f1851044640 10 monclient: shutdown 2024-09-16T14:59:28.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f1851044640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:28.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f1851044640 1 -- 172.21.15.26:0/3526847897 >> v1:172.21.15.26:6789/0 conn(0x7f184c134d10 legacy=0x7f184c1d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f184b59e640 1 -- 172.21.15.26:0/3526847897 reap_dead start 2024-09-16T14:59:28.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f184b59e640 1 -- 172.21.15.26:0/3526847897 reap_dead start 2024-09-16T14:59:28.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f1851044640 1 -- 172.21.15.26:0/3526847897 shutdown_connections 2024-09-16T14:59:28.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f1851044640 1 -- 172.21.15.26:0/3526847897 >> 172.21.15.26:0/3526847897 conn(0x7f184c0a3e30 msgr2=0x7f184c0a0080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:28.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f1851044640 1 -- 172.21.15.26:0/3526847897 shutdown_connections 2024-09-16T14:59:28.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.346+0000 7f1851044640 1 -- 172.21.15.26:0/3526847897 wait complete. 2024-09-16T14:59:28.377 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:59:28.377 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:59:28.377 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:59:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.494+0000 7f4483fc6640 10 monclient: get_monmap_and_config 2024-09-16T14:59:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.494+0000 7f4483fc6640 10 monclient: build_initial_monmap 2024-09-16T14:59:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.494+0000 7f4483fc6640 10 monclient: monmap: 2024-09-16T14:59:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:59:28.499934+0000 2024-09-16T14:59:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:59:28.499934+0000 2024-09-16T14:59:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:59:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:59:28.498 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.494+0000 7f4483fc6640 1 Processor -- start 2024-09-16T14:59:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.494+0000 7f4483fc6640 1 -- start start 2024-09-16T14:59:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.494+0000 7f4483fc6640 10 monclient: init 2024-09-16T14:59:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.494+0000 7f4483fc6640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.494+0000 7f4483fc6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:59:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.494+0000 7f4483fc6640 10 monclient(hunting): picked mon.noname-a con 0x7f447c0a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.494+0000 7f4483fc6640 10 monclient(hunting): picked mon.noname-b con 0x7f447c0a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.494+0000 7f4483fc6640 10 monclient(hunting): picked mon.noname-c con 0x7f447c0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f447c0a3250 con 0x7f447c0a1d10 2024-09-16T14:59:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f447c09f600 con 0x7f447c0a2800 2024-09-16T14:59:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f447c0a3830 con 0x7f447c0a0000 2024-09-16T14:59:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 10 monclient(hunting): authenticate will time out at 1992.620064s 2024-09-16T14:59:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4481d62640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f447c0a1d10 0x7f447c0a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34592/0 (socket says 172.21.15.26:34592) 2024-09-16T14:59:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4481d62640 1 -- 172.21.15.26:0/1548518878 learned_addr learned my addr 172.21.15.26:0/1548518878 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 1 -- 172.21.15.26:0/1548518878 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222002186 0 0) 0x7f447c0a3250 con 0x7f447c0a1d10 2024-09-16T14:59:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient(hunting): my global_id is 324114 2024-09-16T14:59:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 1 -- 172.21.15.26:0/1548518878 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4468003530 con 0x7f447c0a1d10 2024-09-16T14:59:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 1 -- 172.21.15.26:0/1548518878 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507346371 0 0) 0x7f447c0a3830 con 0x7f447c0a0000 2024-09-16T14:59:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient(hunting): my global_id is 284537 2024-09-16T14:59:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 1 -- 172.21.15.26:0/1548518878 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f447c0a3250 con 0x7f447c0a0000 2024-09-16T14:59:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 1 -- 172.21.15.26:0/1548518878 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206966261 0 0) 0x7f447c09f600 con 0x7f447c0a2800 2024-09-16T14:59:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient(hunting): my global_id is 304282 2024-09-16T14:59:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 1 -- 172.21.15.26:0/1548518878 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f447c0a3830 con 0x7f447c0a2800 2024-09-16T14:59:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 1 -- 172.21.15.26:0/1548518878 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 742030355 0 0) 0x7f447c0a3250 con 0x7f447c0a0000 2024-09-16T14:59:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 1 -- 172.21.15.26:0/1548518878 >> v1:172.21.15.189:6789/0 conn(0x7f447c0a2800 legacy=0x7f447c13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 1 -- 172.21.15.26:0/1548518878 >> v1:172.21.15.26:6789/0 conn(0x7f447c0a1d10 legacy=0x7f447c0a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient: _finish_hunting 0 2024-09-16T14:59:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 1 monclient: found mon.noname-b 2024-09-16T14:59:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:59:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 1 -- 172.21.15.26:0/1548518878 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c09ed60 con 0x7f447c0a0000 2024-09-16T14:59:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient: _finish_auth 0 2024-09-16T14:59:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient: _check_auth_tickets 2024-09-16T14:59:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 1 -- 172.21.15.26:0/1548518878 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f446c001580 con 0x7f447c0a0000 2024-09-16T14:59:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:59:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient: dump: 2024-09-16T14:59:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.511 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.511 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.511 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.511 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.511 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.511 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.511 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.511 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.511 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 5 monclient: authenticate success, global_id 284537 2024-09-16T14:59:28.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:28.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 1 -- 172.21.15.26:0/1548518878 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f446c0017a0 con 0x7f447c0a0000 2024-09-16T14:59:28.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:28.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 1 -- 172.21.15.26:0/1548518878 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f446c001bb0 con 0x7f447c0a0000 2024-09-16T14:59:28.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:28.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4480d60640 10 monclient: dump: 2024-09-16T14:59:28.512 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.512 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.512 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.512 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.512 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.512 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.512 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.512 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.512 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.513 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 10 monclient: get_monmap_and_config success 2024-09-16T14:59:28.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 4 set_mon_vals no callback set 2024-09-16T14:59:28.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 10 monclient: shutdown 2024-09-16T14:59:28.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:28.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 1 -- 172.21.15.26:0/1548518878 >> v1:172.21.15.26:6790/0 conn(0x7f447c0a0000 legacy=0x7f447c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 1 -- 172.21.15.26:0/1548518878 shutdown_connections 2024-09-16T14:59:28.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 1 -- 172.21.15.26:0/1548518878 >> 172.21.15.26:0/1548518878 conn(0x7f447c0a7f30 msgr2=0x7f447c0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:28.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 1 -- 172.21.15.26:0/1548518878 shutdown_connections 2024-09-16T14:59:28.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 1 -- 172.21.15.26:0/1548518878 wait complete. 2024-09-16T14:59:28.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 10 monclient: build_initial_monmap 2024-09-16T14:59:28.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 10 monclient: monmap: 2024-09-16T14:59:28.514 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:28.514 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:59:28.514 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:59:28.514 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:59:28.514 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:28.514 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:59:28.514 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:28.514 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:59:28.514 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:59:28.514 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 1 Processor -- start 2024-09-16T14:59:28.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 1 -- start start 2024-09-16T14:59:28.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 10 monclient: init 2024-09-16T14:59:28.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:28.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:59:28.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 10 monclient(hunting): picked mon.noname-b con 0x7f447c0a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:28.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 10 monclient(hunting): picked mon.noname-a con 0x7f447c0a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:28.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 10 monclient(hunting): picked mon.noname-c con 0x7f447c0a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:28.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f447c13dde0 con 0x7f447c0a1d10 2024-09-16T14:59:28.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f447c13c8e0 con 0x7f447c0a0000 2024-09-16T14:59:28.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f447c13a020 con 0x7f447c0a2800 2024-09-16T14:59:28.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:28.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4483fc6640 10 monclient(hunting): authenticate will time out at 1992.623447s 2024-09-16T14:59:28.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4481561640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f447c0a1d10 0x7f447c139910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34606/0 (socket says 172.21.15.26:34606) 2024-09-16T14:59:28.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4481561640 1 -- 172.21.15.26:0/3059354181 learned_addr learned my addr 172.21.15.26:0/3059354181 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:28.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f4482563640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f447c0a2800 0x7f447c1d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48038/0 (socket says 172.21.15.26:48038) 2024-09-16T14:59:28.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360911471 0 0) 0x7f447c09ed60 con 0x7f447c0a1d10 2024-09-16T14:59:28.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f44727fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f44727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f44727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f44727fc640 10 monclient(hunting): my global_id is 324117 2024-09-16T14:59:28.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4460003530 con 0x7f447c0a1d10 2024-09-16T14:59:28.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881322840 0 0) 0x7f447c13a020 con 0x7f447c0a2800 2024-09-16T14:59:28.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f44727fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f44727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f44727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f44727fc640 10 monclient(hunting): my global_id is 284540 2024-09-16T14:59:28.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.498+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f447c09ed60 con 0x7f447c0a2800 2024-09-16T14:59:28.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3525262229 0 0) 0x7f447c09ed60 con 0x7f447c0a2800 2024-09-16T14:59:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f447c13a020 con 0x7f447c0a2800 2024-09-16T14:59:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3834094913 0 0) 0x7f4460003530 con 0x7f447c0a1d10 2024-09-16T14:59:28.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f447c09ed60 con 0x7f447c0a1d10 2024-09-16T14:59:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4474001840 con 0x7f447c0a2800 2024-09-16T14:59:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:59:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient(hunting): dump: 2024-09-16T14:59:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.520 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.521 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:59:28.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f446c001910 con 0x7f447c0a1d10 2024-09-16T14:59:28.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:28.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient(hunting): dump: 2024-09-16T14:59:28.521 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.521 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.521 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.521 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.522 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.522 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.522 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.522 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.522 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.522 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126678537 0 0) 0x7f447c13c8e0 con 0x7f447c0a0000 2024-09-16T14:59:28.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient(hunting): my global_id is 304285 2024-09-16T14:59:28.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4460003530 con 0x7f447c0a0000 2024-09-16T14:59:28.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3628488135 0 0) 0x7f447c13a020 con 0x7f447c0a2800 2024-09-16T14:59:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f4481d62640 1 -- 172.21.15.26:0/3059354181 >> v1:172.21.15.189:6789/0 conn(0x7f447c0a0000 legacy=0x7f447c139200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T14:59:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f4481d62640 1 -- 172.21.15.26:0/3059354181 >> v1:172.21.15.189:6789/0 conn(0x7f447c0a0000 legacy=0x7f447c139200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T14:59:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f4481d62640 1 --1- 172.21.15.26:0/3059354181 >> v1:172.21.15.189:6789/0 conn(0x7f447c0a0000 0x7f447c139200 :-1 s=OPENED pgs=1431 cs=1 l=1).handle_message read tag failed 2024-09-16T14:59:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f4481d62640 1 --1- 172.21.15.26:0/3059354181 >> v1:172.21.15.189:6789/0 conn(0x7f447c0a0000 0x7f447c139200 :-1 s=OPENED pgs=1431 cs=1 l=1).fault on lossy channel, failing 2024-09-16T14:59:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 >> v1:172.21.15.189:6789/0 conn(0x7f447c0a0000 legacy=0x7f447c139200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 >> v1:172.21.15.26:6789/0 conn(0x7f447c0a1d10 legacy=0x7f447c139910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient: _finish_hunting 0 2024-09-16T14:59:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 monclient: found mon.c 2024-09-16T14:59:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c0a65f0 con 0x7f447c0a2800 2024-09-16T14:59:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient: _finish_auth 0 2024-09-16T14:59:28.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient: _check_auth_tickets 2024-09-16T14:59:28.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:28.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient: ms_handle_reset stray mon v1:172.21.15.189:6789/0 2024-09-16T14:59:28.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f4483fc6640 5 monclient: authenticate success, global_id 284540 2024-09-16T14:59:28.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f4483fc6640 10 monclient: _renew_subs 2024-09-16T14:59:28.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f4483fc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:28.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f4483fc6640 1 -- 172.21.15.26:0/3059354181 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f447c09e500 con 0x7f447c0a2800 2024-09-16T14:59:28.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f4483fc6640 10 monclient: _renew_subs 2024-09-16T14:59:28.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f4483fc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:28.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f4483fc6640 1 -- 172.21.15.26:0/3059354181 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f447c135850 con 0x7f447c0a2800 2024-09-16T14:59:28.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4474003620 con 0x7f447c0a2800 2024-09-16T14:59:28.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:28.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4474001d90 con 0x7f447c0a2800 2024-09-16T14:59:28.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:28.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient: dump: 2024-09-16T14:59:28.525 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.525 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.525 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.525 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.525 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.525 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.525 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.525 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.525 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.525 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f4473fff640 4 set_mon_vals no callback set 2024-09-16T14:59:28.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f4483fc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:28.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f4483fc6640 1 -- 172.21.15.26:0/3059354181 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4464006590 con 0x7f4464002830 2024-09-16T14:59:28.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4474003a30 con 0x7f447c0a2800 2024-09-16T14:59:28.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f4474043cd0 con 0x7f447c0a2800 2024-09-16T14:59:28.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 391406705 0 0) 0x7f4464006590 con 0x7f4464002830 2024-09-16T14:59:28.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient(hunting): my global_id is 284543 2024-09-16T14:59:28.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f447c13a020 con 0x7f4464002830 2024-09-16T14:59:28.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 80759154 0 0) 0x7f447c13a020 con 0x7f4464002830 2024-09-16T14:59:28.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4464006590 con 0x7f4464002830 2024-09-16T14:59:28.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4478001420 con 0x7f4464002830 2024-09-16T14:59:28.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:28.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient: dump: 2024-09-16T14:59:28.527 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.527 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.527 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.527 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.527 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.527 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.527 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.527 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.527 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.527 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f4481561640 10 monclient: get_auth_request con 0x7f44600363e0 auth_method 0 2024-09-16T14:59:28.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4180735719 0 0) 0x7f4464006590 con 0x7f4464002830 2024-09-16T14:59:28.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f446400a0e0 con 0x7f4464002830 2024-09-16T14:59:28.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f44780017c0 con 0x7f4464002830 2024-09-16T14:59:28.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:28.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:59:28.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 >> v1:172.21.15.26:6790/0 conn(0x7f4464002830 legacy=0x7f4464004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.502+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 >> v1:172.21.15.26:6790/0 conn(0x7f4464002830 legacy=0x7f4464004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:28.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f4483fc6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:59:28.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f4483fc6640 1 -- 172.21.15.26:0/3059354181 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4464009da0 con 0x7f44640090c0 2024-09-16T14:59:28.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656720441 0 0) 0x7f4464009da0 con 0x7f44640090c0 2024-09-16T14:59:28.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 10 monclient(hunting): my global_id is 284546 2024-09-16T14:59:28.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4464006590 con 0x7f44640090c0 2024-09-16T14:59:28.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1432602148 0 0) 0x7f4464006590 con 0x7f44640090c0 2024-09-16T14:59:28.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4464009da0 con 0x7f44640090c0 2024-09-16T14:59:28.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4478004f00 con 0x7f44640090c0 2024-09-16T14:59:28.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:28.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 10 monclient: dump: 2024-09-16T14:59:28.530 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.530 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.531 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.531 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.531 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.531 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.531 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.531 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.531 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.531 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1899750118 0 0) 0x7f4464009da0 con 0x7f44640090c0 2024-09-16T14:59:28.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f446400c290 con 0x7f44640090c0 2024-09-16T14:59:28.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 3755999708) 0x7f4478009080 con 0x7f44640090c0 2024-09-16T14:59:28.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:59:28.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:59:28.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 >> v1:172.21.15.26:6790/0 conn(0x7f44640090c0 legacy=0x7f4464009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f44727fc640 1 -- 172.21.15.26:0/3059354181 >> v1:172.21.15.26:6790/0 conn(0x7f44640090c0 legacy=0x7f4464009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:28.532 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:59:28.532 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:59:28.532 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:59:28.532 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:59:28.532 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 156, 2024-09-16T14:59:28.532 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:59:28.533 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:59:28.533 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:59:28.533 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:59:28.533 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:28.533 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:59:28.533 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:59:28.533 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:59:28.533 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:59:28.533 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:28.533 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:28.533 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:28.533 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:28.533 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:28.534 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:28.534 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:28.534 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:28.534 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:28.534 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:28.534 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:28.534 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:59:28.534 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:59:28.534 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:28.534 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:28.534 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:28.534 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:28.534 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:28.535 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:28.535 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:28.535 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:28.535 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:28.535 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:28.535 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.535 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:59:28.535 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:59:28.535 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:59:28.535 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:59:28.535 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:59:28.535 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:59:28.536 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:59:28.536 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:59:28.536 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:59:28.536 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:59:28.536 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:59:28.536 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:59:28.536 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:59:28.536 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:59:28.536 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:59:28.536 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:59:28.536 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:59:28.536 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:28.536 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:28.537 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:28.537 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:28.537 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:28.537 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:28.537 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:28.537 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:28.537 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:28.537 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:28.537 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:28.537 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:59:28.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.537 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:59:28.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.538 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:59:28.538 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:59:28.538 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:28.538 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:28.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.538 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:28.538 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:59:28.538 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:28.538 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.538 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.538 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:59:28.538 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:59:28.539 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:28.539 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:28.539 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:28.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.539 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:59:28.539 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:59:28.539 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:28.539 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:28.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.539 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:28.539 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:59:28.539 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:28.540 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.540 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.540 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:59:28.540 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:59:28.540 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:28.540 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:28.540 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:28.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.540 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:59:28.540 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:59:28.540 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:28.541 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:28.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:28.541 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:59:28.541 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:28.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.541 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.541 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:59:28.541 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:59:28.541 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:28.541 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:28.541 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:28.542 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.542 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.542 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:59:28.542 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:59:28.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.542 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:28.542 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:28.542 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:59:28.542 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.542 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:28.542 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:59:28.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.543 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:28.543 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:28.543 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:59:28.543 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.543 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:28.543 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:59:28.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.543 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:28.543 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:28.543 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:59:28.543 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.543 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:28.543 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:59:28.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.544 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:28.544 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:28.544 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:59:28.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.544 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.544 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:59:28.544 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:59:28.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f4483fc6640 1 -- 172.21.15.26:0/3059354181 >> v1:172.21.15.189:6812/3196161582 conn(0x7f44600363e0 legacy=0x7f44600388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f4483fc6640 10 monclient: shutdown 2024-09-16T14:59:28.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f4483fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:28.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f4483fc6640 1 -- 172.21.15.26:0/3059354181 >> v1:172.21.15.26:6790/0 conn(0x7f447c0a2800 legacy=0x7f447c1d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f4482563640 1 -- 172.21.15.26:0/3059354181 reap_dead start 2024-09-16T14:59:28.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f4482563640 1 -- 172.21.15.26:0/3059354181 reap_dead start 2024-09-16T14:59:28.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f4483fc6640 1 -- 172.21.15.26:0/3059354181 shutdown_connections 2024-09-16T14:59:28.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f4483fc6640 1 -- 172.21.15.26:0/3059354181 >> 172.21.15.26:0/3059354181 conn(0x7f447c0a7f30 msgr2=0x7f447c13e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:28.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f4483fc6640 1 -- 172.21.15.26:0/3059354181 shutdown_connections 2024-09-16T14:59:28.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.510+0000 7f4483fc6640 1 -- 172.21.15.26:0/3059354181 wait complete. 2024-09-16T14:59:28.546 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:59:28.546 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:59:28.547 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:59:28.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 10 monclient: get_monmap_and_config 2024-09-16T14:59:28.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 10 monclient: build_initial_monmap 2024-09-16T14:59:28.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 10 monclient: monmap: 2024-09-16T14:59:28.656 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:28.656 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.656 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:59:28.658349+0000 2024-09-16T14:59:28.656 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:59:28.658349+0000 2024-09-16T14:59:28.656 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:28.656 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.657 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:28.657 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:59:28.657 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:59:28.657 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 1 Processor -- start 2024-09-16T14:59:28.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 1 -- start start 2024-09-16T14:59:28.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 10 monclient: init 2024-09-16T14:59:28.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:28.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:59:28.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 10 monclient(hunting): picked mon.noname-a con 0x7f912009cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:28.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 10 monclient(hunting): picked mon.noname-c con 0x7f9120132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:28.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 10 monclient(hunting): picked mon.noname-b con 0x7f9120138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:28.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912009d260 con 0x7f912009cb80 2024-09-16T14:59:28.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9120132a80 con 0x7f9120132650 2024-09-16T14:59:28.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912009cf60 con 0x7f9120138ff0 2024-09-16T14:59:28.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:28.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 10 monclient(hunting): authenticate will time out at 1992.778389s 2024-09-16T14:59:28.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911ed9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9120138ff0 0x7f912013b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48074/0 (socket says 172.21.15.26:48074) 2024-09-16T14:59:28.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911ed9d640 1 -- 172.21.15.26:0/738233817 learned_addr learned my addr 172.21.15.26:0/738233817 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:28.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911e59c640 1 --1- 172.21.15.26:0/738233817 >> v1:172.21.15.26:6789/0 conn(0x7f912009cb80 0x7f91200717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34622/0 (socket says 172.21.15.26:34622) 2024-09-16T14:59:28.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 1 -- 172.21.15.26:0/738233817 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953926306 0 0) 0x7f912009d260 con 0x7f912009cb80 2024-09-16T14:59:28.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient(hunting): my global_id is 324120 2024-09-16T14:59:28.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 1 -- 172.21.15.26:0/738233817 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f910c003530 con 0x7f912009cb80 2024-09-16T14:59:28.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 1 -- 172.21.15.26:0/738233817 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345764841 0 0) 0x7f912009cf60 con 0x7f9120138ff0 2024-09-16T14:59:28.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient(hunting): my global_id is 284549 2024-09-16T14:59:28.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 1 -- 172.21.15.26:0/738233817 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f912009d260 con 0x7f9120138ff0 2024-09-16T14:59:28.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 1 -- 172.21.15.26:0/738233817 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 751158662 0 0) 0x7f912009d260 con 0x7f9120138ff0 2024-09-16T14:59:28.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 1 -- 172.21.15.26:0/738233817 >> v1:172.21.15.189:6789/0 conn(0x7f9120132650 legacy=0x7f91201378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 1 -- 172.21.15.26:0/738233817 >> v1:172.21.15.26:6789/0 conn(0x7f912009cb80 legacy=0x7f91200717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient: _finish_hunting 0 2024-09-16T14:59:28.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 1 monclient: found mon.noname-b 2024-09-16T14:59:28.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:28.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:59:28.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 1 -- 172.21.15.26:0/738233817 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912009f2d0 con 0x7f9120138ff0 2024-09-16T14:59:28.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient: _finish_auth 0 2024-09-16T14:59:28.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient: _check_auth_tickets 2024-09-16T14:59:28.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:28.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 1 -- 172.21.15.26:0/738233817 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f91180019e0 con 0x7f9120138ff0 2024-09-16T14:59:28.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:59:28.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient: dump: 2024-09-16T14:59:28.667 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.667 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.667 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.667 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.667 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.667 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.668 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.668 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.668 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.668 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 5 monclient: authenticate success, global_id 284549 2024-09-16T14:59:28.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:28.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 1 -- 172.21.15.26:0/738233817 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9118001b40 con 0x7f9120138ff0 2024-09-16T14:59:28.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:28.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 1 -- 172.21.15.26:0/738233817 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9118001e90 con 0x7f9120138ff0 2024-09-16T14:59:28.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:28.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f911dd9b640 10 monclient: dump: 2024-09-16T14:59:28.669 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.669 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.669 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.669 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.669 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.669 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.669 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.669 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.669 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.669 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 10 monclient: get_monmap_and_config success 2024-09-16T14:59:28.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 4 set_mon_vals no callback set 2024-09-16T14:59:28.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 10 monclient: shutdown 2024-09-16T14:59:28.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:28.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 1 -- 172.21.15.26:0/738233817 >> v1:172.21.15.26:6790/0 conn(0x7f9120138ff0 legacy=0x7f912013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 1 -- 172.21.15.26:0/738233817 shutdown_connections 2024-09-16T14:59:28.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 1 -- 172.21.15.26:0/738233817 >> 172.21.15.26:0/738233817 conn(0x7f912009fb50 msgr2=0x7f912009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:28.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 1 -- 172.21.15.26:0/738233817 shutdown_connections 2024-09-16T14:59:28.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.654+0000 7f9124b7d640 1 -- 172.21.15.26:0/738233817 wait complete. 2024-09-16T14:59:28.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 10 monclient: build_initial_monmap 2024-09-16T14:59:28.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 10 monclient: monmap: 2024-09-16T14:59:28.670 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:28.671 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:59:28.671 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:59:28.671 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:59:28.671 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:28.671 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:59:28.671 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:28.671 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:59:28.671 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:59:28.671 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 1 Processor -- start 2024-09-16T14:59:28.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 1 -- start start 2024-09-16T14:59:28.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 10 monclient: init 2024-09-16T14:59:28.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:28.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:59:28.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 10 monclient(hunting): picked mon.noname-b con 0x7f912009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:28.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 10 monclient(hunting): picked mon.noname-c con 0x7f9120132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:28.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 10 monclient(hunting): picked mon.noname-a con 0x7f9120138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:28.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912009b370 con 0x7f9120138ff0 2024-09-16T14:59:28.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912009a240 con 0x7f912009cb80 2024-09-16T14:59:28.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91200a2940 con 0x7f9120132650 2024-09-16T14:59:28.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:28.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 10 monclient(hunting): authenticate will time out at 1992.781306s 2024-09-16T14:59:28.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f911ed9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9120138ff0 0x7f91201d1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34638/0 (socket says 172.21.15.26:34638) 2024-09-16T14:59:28.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9117fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9120132650 0x7f912009ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48088/0 (socket says 172.21.15.26:48088) 2024-09-16T14:59:28.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f911ed9d640 1 -- 172.21.15.26:0/3422017290 learned_addr learned my addr 172.21.15.26:0/3422017290 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:28.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252086754 0 0) 0x7f912009b370 con 0x7f9120138ff0 2024-09-16T14:59:28.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): my global_id is 324123 2024-09-16T14:59:28.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9100003530 con 0x7f9120138ff0 2024-09-16T14:59:28.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617568413 0 0) 0x7f91200a2940 con 0x7f9120132650 2024-09-16T14:59:28.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): my global_id is 284552 2024-09-16T14:59:28.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f912009b370 con 0x7f9120132650 2024-09-16T14:59:28.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3863822998 0 0) 0x7f912009b370 con 0x7f9120132650 2024-09-16T14:59:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91200a2940 con 0x7f9120132650 2024-09-16T14:59:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2661625209 0 0) 0x7f9100003530 con 0x7f9120138ff0 2024-09-16T14:59:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f912009b370 con 0x7f9120138ff0 2024-09-16T14:59:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f91100018c0 con 0x7f9120132650 2024-09-16T14:59:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:59:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): dump: 2024-09-16T14:59:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.674 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.675 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.675 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.675 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:59:28.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9118001800 con 0x7f9120138ff0 2024-09-16T14:59:28.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:28.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): dump: 2024-09-16T14:59:28.675 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.675 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.676 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.676 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.676 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.676 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.676 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.676 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.676 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.676 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902896640 0 0) 0x7f912009a240 con 0x7f912009cb80 2024-09-16T14:59:28.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): my global_id is 304291 2024-09-16T14:59:28.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9100003530 con 0x7f912009cb80 2024-09-16T14:59:28.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4087212617 0 0) 0x7f91200a2940 con 0x7f9120132650 2024-09-16T14:59:28.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 >> v1:172.21.15.189:6789/0 conn(0x7f912009cb80 legacy=0x7f912009a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 >> v1:172.21.15.26:6789/0 conn(0x7f9120138ff0 legacy=0x7f91201d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:59:28.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 monclient: found mon.c 2024-09-16T14:59:28.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:28.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:28.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9120071a30 con 0x7f9120132650 2024-09-16T14:59:28.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient: _finish_auth 0 2024-09-16T14:59:28.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:59:28.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:28.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 5 monclient: authenticate success, global_id 284552 2024-09-16T14:59:28.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 10 monclient: _renew_subs 2024-09-16T14:59:28.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:28.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 1 -- 172.21.15.26:0/3422017290 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f912006db90 con 0x7f9120132650 2024-09-16T14:59:28.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 10 monclient: _renew_subs 2024-09-16T14:59:28.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:28.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 1 -- 172.21.15.26:0/3422017290 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9120138640 con 0x7f9120132650 2024-09-16T14:59:28.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9110003620 con 0x7f9120132650 2024-09-16T14:59:28.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:28.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f91100039b0 con 0x7f9120132650 2024-09-16T14:59:28.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:28.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f911d59a640 4 set_mon_vals no callback set 2024-09-16T14:59:28.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient: dump: 2024-09-16T14:59:28.679 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.679 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.679 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.679 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.679 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.679 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.679 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.679 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.679 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.680 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:28.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9124b7d640 1 -- 172.21.15.26:0/3422017290 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9104006590 con 0x7f9104002830 2024-09-16T14:59:28.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9110003b90 con 0x7f9120132650 2024-09-16T14:59:28.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f9110043e20 con 0x7f9120132650 2024-09-16T14:59:28.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f911ed9d640 10 monclient: get_auth_request con 0x7f91000363d0 auth_method 0 2024-09-16T14:59:28.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186469168 0 0) 0x7f9104006590 con 0x7f9104002830 2024-09-16T14:59:28.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 10 monclient(hunting): my global_id is 304294 2024-09-16T14:59:28.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.658+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91200a2940 con 0x7f9104002830 2024-09-16T14:59:28.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.662+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4239656511 0 0) 0x7f91200a2940 con 0x7f9104002830 2024-09-16T14:59:28.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.662+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9104006590 con 0x7f9104002830 2024-09-16T14:59:28.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.662+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9108001600 con 0x7f9104002830 2024-09-16T14:59:28.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.662+0000 7f9116ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.662+0000 7f9116ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:59:28.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.662+0000 7f9116ffd640 10 monclient: dump: 2024-09-16T14:59:28.681 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.681 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.681 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.681 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.681 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.681 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.681 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.681 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.682 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.682 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.662+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2379492553 0 0) 0x7f9104006590 con 0x7f9104002830 2024-09-16T14:59:28.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.662+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f910400a0e0 con 0x7f9104002830 2024-09-16T14:59:28.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.662+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f9108001950 con 0x7f9104002830 2024-09-16T14:59:28.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.662+0000 7f9116ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:28.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.662+0000 7f9116ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:59:28.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.662+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 >> v1:172.21.15.189:6789/0 conn(0x7f9104002830 legacy=0x7f9104004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.662+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 >> v1:172.21.15.189:6789/0 conn(0x7f9104002830 legacy=0x7f9104004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:28.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.666+0000 7f9124b7d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:59:28.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.666+0000 7f9124b7d640 1 -- 172.21.15.26:0/3422017290 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9104009d80 con 0x7f91040090a0 2024-09-16T14:59:28.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361420743 0 0) 0x7f9104009d80 con 0x7f91040090a0 2024-09-16T14:59:28.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 10 monclient(hunting): my global_id is 304297 2024-09-16T14:59:28.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9104006590 con 0x7f91040090a0 2024-09-16T14:59:28.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2898967684 0 0) 0x7f9104006590 con 0x7f91040090a0 2024-09-16T14:59:28.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9104009d80 con 0x7f91040090a0 2024-09-16T14:59:28.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9108004990 con 0x7f91040090a0 2024-09-16T14:59:28.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:59:28.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 10 monclient: dump: 2024-09-16T14:59:28.687 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.687 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.687 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.687 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.687 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.687 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.687 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.687 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.687 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.687 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 177066627 0 0) 0x7f9104009d80 con 0x7f91040090a0 2024-09-16T14:59:28.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f910400c290 con 0x7f91040090a0 2024-09-16T14:59:28.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3689229124) 0x7f9108004ba0 con 0x7f91040090a0 2024-09-16T14:59:28.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:59:28.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:59:28.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 >> v1:172.21.15.189:6789/0 conn(0x7f91040090a0 legacy=0x7f9104009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9116ffd640 1 -- 172.21.15.26:0/3422017290 >> v1:172.21.15.189:6789/0 conn(0x7f91040090a0 legacy=0x7f9104009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:28.688 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:59:28.689 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:59:28.689 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:59:28.689 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:59:28.689 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 156, 2024-09-16T14:59:28.689 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:59:28.689 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:59:28.689 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:59:28.689 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:59:28.689 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:28.689 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:59:28.689 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:59:28.690 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:59:28.690 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:59:28.690 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:28.690 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:28.690 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:28.690 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:28.690 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:28.690 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:28.690 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:28.690 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:28.690 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:28.690 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:28.690 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:28.691 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:59:28.691 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:59:28.691 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:28.691 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:28.691 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:28.691 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:28.691 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:28.691 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:28.691 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:28.691 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:28.691 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:28.691 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:28.691 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.692 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:59:28.692 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:59:28.692 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:59:28.692 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:59:28.692 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:59:28.692 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:59:28.692 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:59:28.692 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:59:28.692 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:59:28.692 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:59:28.692 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:59:28.692 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:59:28.692 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:59:28.693 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:59:28.693 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:59:28.693 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:59:28.693 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:59:28.693 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:28.693 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:28.693 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:28.693 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:28.693 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:28.693 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:28.693 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:28.693 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:28.693 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:28.694 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:28.694 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:28.694 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:59:28.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.694 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:59:28.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.694 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:59:28.694 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:59:28.694 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:28.694 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:28.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.694 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:28.694 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:59:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:59:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:59:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:59:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:59:28.695 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:59:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:59:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:59:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:28.696 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:59:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:59:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:59:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.698 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:59:28.698 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:59:28.698 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:28.698 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:28.698 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:28.698 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.698 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.698 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:59:28.698 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:59:28.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.698 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:28.698 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:28.698 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:59:28.699 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.699 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:28.699 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:59:28.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:28.699 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:28.699 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:28.699 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:59:28.699 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:28.699 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:28.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:28.700 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:59:28.700 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:59:28.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9124b7d640 1 -- 172.21.15.26:0/3422017290 >> v1:172.21.15.189:6812/3196161582 conn(0x7f91000363d0 legacy=0x7f9100038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9124b7d640 10 monclient: shutdown 2024-09-16T14:59:28.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9124b7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:28.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9124b7d640 1 -- 172.21.15.26:0/3422017290 >> v1:172.21.15.26:6790/0 conn(0x7f9120132650 legacy=0x7f912009ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f911ed9d640 1 -- 172.21.15.26:0/3422017290 reap_dead start 2024-09-16T14:59:28.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f911ed9d640 1 -- 172.21.15.26:0/3422017290 reap_dead start 2024-09-16T14:59:28.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9124b7d640 1 -- 172.21.15.26:0/3422017290 shutdown_connections 2024-09-16T14:59:28.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9124b7d640 1 -- 172.21.15.26:0/3422017290 >> 172.21.15.26:0/3422017290 conn(0x7f912009fb50 msgr2=0x7f91200a1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:28.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9124b7d640 1 -- 172.21.15.26:0/3422017290 shutdown_connections 2024-09-16T14:59:28.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.670+0000 7f9124b7d640 1 -- 172.21.15.26:0/3422017290 wait complete. 2024-09-16T14:59:28.702 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:59:28.702 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T14:59:28.702 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T14:59:28.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f331b788640 10 monclient: get_monmap_and_config 2024-09-16T14:59:28.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f331b788640 10 monclient: build_initial_monmap 2024-09-16T14:59:28.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f331b788640 10 monclient: monmap: 2024-09-16T14:59:28.821 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:28.821 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.821 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:59:28.823606+0000 2024-09-16T14:59:28.821 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:59:28.823606+0000 2024-09-16T14:59:28.821 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:28.822 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.822 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:28.822 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:59:28.822 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:59:28.822 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f331b788640 1 Processor -- start 2024-09-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f331b788640 1 -- start start 2024-09-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f331b788640 10 monclient: init 2024-09-16T14:59:28.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f331b788640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f331b788640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f331b788640 10 monclient(hunting): picked mon.noname-b con 0x7f331409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f331b788640 10 monclient(hunting): picked mon.noname-c con 0x7f3314132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f331b788640 10 monclient(hunting): picked mon.noname-a con 0x7f3314138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f331b788640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331409d260 con 0x7f3314138ff0 2024-09-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f331b788640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3314132a80 con 0x7f3314132650 2024-09-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f331b788640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331409cf60 con 0x7f331409cb80 2024-09-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f331b788640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f331b788640 10 monclient(hunting): authenticate will time out at 1992.943615s 2024-09-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f3319524640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f331409cb80 0x7f33140717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48094/0 (socket says 172.21.15.26:48094) 2024-09-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.818+0000 7f3319524640 1 -- 172.21.15.26:0/1752005088 learned_addr learned my addr 172.21.15.26:0/1752005088 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 1 -- 172.21.15.26:0/1752005088 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858771794 0 0) 0x7f331409cf60 con 0x7f331409cb80 2024-09-16T14:59:28.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient(hunting): my global_id is 284555 2024-09-16T14:59:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 1 -- 172.21.15.26:0/1752005088 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3300003530 con 0x7f331409cb80 2024-09-16T14:59:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 1 -- 172.21.15.26:0/1752005088 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292046438 0 0) 0x7f331409d260 con 0x7f3314138ff0 2024-09-16T14:59:28.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient(hunting): my global_id is 324126 2024-09-16T14:59:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 1 -- 172.21.15.26:0/1752005088 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f331409cf60 con 0x7f3314138ff0 2024-09-16T14:59:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 1 -- 172.21.15.26:0/1752005088 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1410425634 0 0) 0x7f331409cf60 con 0x7f3314138ff0 2024-09-16T14:59:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 1 -- 172.21.15.26:0/1752005088 >> v1:172.21.15.26:6790/0 conn(0x7f331409cb80 legacy=0x7f33140717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 1 -- 172.21.15.26:0/1752005088 >> v1:172.21.15.189:6789/0 conn(0x7f3314132650 legacy=0x7f33141378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient: _finish_hunting 0 2024-09-16T14:59:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 1 monclient: found mon.noname-a 2024-09-16T14:59:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:59:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 1 -- 172.21.15.26:0/1752005088 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331409f2d0 con 0x7f3314138ff0 2024-09-16T14:59:28.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient: _finish_auth 0 2024-09-16T14:59:28.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient: _check_auth_tickets 2024-09-16T14:59:28.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:28.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 1 -- 172.21.15.26:0/1752005088 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f330c0019a0 con 0x7f3314138ff0 2024-09-16T14:59:28.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 5 monclient: authenticate success, global_id 324126 2024-09-16T14:59:28.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:28.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:59:28.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient: dump: 2024-09-16T14:59:28.833 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.833 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.833 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.833 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.833 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.834 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.834 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.834 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.834 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.834 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:28.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 1 -- 172.21.15.26:0/1752005088 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f330c001b50 con 0x7f3314138ff0 2024-09-16T14:59:28.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:28.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 1 -- 172.21.15.26:0/1752005088 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f330c001f20 con 0x7f3314138ff0 2024-09-16T14:59:28.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 10 monclient: get_monmap_and_config success 2024-09-16T14:59:28.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 4 set_mon_vals no callback set 2024-09-16T14:59:28.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:28.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330bfff640 10 monclient: dump: 2024-09-16T14:59:28.835 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.835 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.835 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.835 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.835 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.835 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.835 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.835 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.835 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.835 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 10 monclient: shutdown 2024-09-16T14:59:28.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:28.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 1 -- 172.21.15.26:0/1752005088 >> v1:172.21.15.26:6789/0 conn(0x7f3314138ff0 legacy=0x7f331413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 1 -- 172.21.15.26:0/1752005088 shutdown_connections 2024-09-16T14:59:28.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 1 -- 172.21.15.26:0/1752005088 >> 172.21.15.26:0/1752005088 conn(0x7f331409fb50 msgr2=0x7f331409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:28.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 1 -- 172.21.15.26:0/1752005088 shutdown_connections 2024-09-16T14:59:28.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 1 -- 172.21.15.26:0/1752005088 wait complete. 2024-09-16T14:59:28.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 10 monclient: build_initial_monmap 2024-09-16T14:59:28.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 10 monclient: monmap: 2024-09-16T14:59:28.836 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:28.836 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:59:28.836 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:59:28.836 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:59:28.836 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:28.836 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:59:28.837 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:28.837 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:59:28.837 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:59:28.837 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 1 Processor -- start 2024-09-16T14:59:28.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 1 -- start start 2024-09-16T14:59:28.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 10 monclient: init 2024-09-16T14:59:28.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:28.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:59:28.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 10 monclient(hunting): picked mon.noname-a con 0x7f331409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:28.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 10 monclient(hunting): picked mon.noname-c con 0x7f3314132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:28.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 10 monclient(hunting): picked mon.noname-b con 0x7f3314138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:28.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33140a2080 con 0x7f331409cb80 2024-09-16T14:59:28.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3314071b50 con 0x7f3314138ff0 2024-09-16T14:59:28.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331409a240 con 0x7f3314132650 2024-09-16T14:59:28.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:28.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 10 monclient(hunting): authenticate will time out at 1992.946850s 2024-09-16T14:59:28.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3318d23640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3314132650 0x7f33141cbb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48108/0 (socket says 172.21.15.26:48108) 2024-09-16T14:59:28.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3318d23640 1 -- 172.21.15.26:0/3637663350 learned_addr learned my addr 172.21.15.26:0/3637663350 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:28.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3319524640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f331409cb80 0x7f3314138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34656/0 (socket says 172.21.15.26:34656) 2024-09-16T14:59:28.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948884654 0 0) 0x7f33140a2080 con 0x7f331409cb80 2024-09-16T14:59:28.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): my global_id is 324129 2024-09-16T14:59:28.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32f8003530 con 0x7f331409cb80 2024-09-16T14:59:28.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425922320 0 0) 0x7f331409a240 con 0x7f3314132650 2024-09-16T14:59:28.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): my global_id is 284558 2024-09-16T14:59:28.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33140a2080 con 0x7f3314132650 2024-09-16T14:59:28.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1204729019 0 0) 0x7f32f8003530 con 0x7f331409cb80 2024-09-16T14:59:28.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f331409a240 con 0x7f331409cb80 2024-09-16T14:59:28.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f33040016f0 con 0x7f331409cb80 2024-09-16T14:59:28.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:59:28.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): dump: 2024-09-16T14:59:28.840 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.840 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.840 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.840 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.840 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.840 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.840 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.840 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.840 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.840 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:59:28.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 179290807 0 0) 0x7f33140a2080 con 0x7f3314132650 2024-09-16T14:59:28.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32f8003530 con 0x7f3314132650 2024-09-16T14:59:28.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f33100019d0 con 0x7f3314132650 2024-09-16T14:59:28.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:28.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): dump: 2024-09-16T14:59:28.841 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.841 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.841 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.841 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.841 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.841 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.841 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.841 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.842 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.842 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659951856 0 0) 0x7f3314071b50 con 0x7f3314138ff0 2024-09-16T14:59:28.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): _negotiate 2024-09-16T14:59:28.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:28.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:28.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient(hunting): my global_id is 304303 2024-09-16T14:59:28.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33140a2080 con 0x7f3314138ff0 2024-09-16T14:59:28.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4080730794 0 0) 0x7f331409a240 con 0x7f331409cb80 2024-09-16T14:59:28.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 >> v1:172.21.15.26:6790/0 conn(0x7f3314132650 legacy=0x7f33141cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 >> v1:172.21.15.189:6789/0 conn(0x7f3314138ff0 legacy=0x7f33141cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:28.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient: _finish_hunting 0 2024-09-16T14:59:28.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 monclient: found mon.a 2024-09-16T14:59:28.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:28.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:28.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331406f130 con 0x7f331409cb80 2024-09-16T14:59:28.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient: _finish_auth 0 2024-09-16T14:59:28.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient: _check_auth_tickets 2024-09-16T14:59:28.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:28.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 5 monclient: authenticate success, global_id 324129 2024-09-16T14:59:28.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 10 monclient: _renew_subs 2024-09-16T14:59:28.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:28.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 1 -- 172.21.15.26:0/3637663350 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33140711b0 con 0x7f331409cb80 2024-09-16T14:59:28.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 10 monclient: _renew_subs 2024-09-16T14:59:28.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:28.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f331b788640 1 -- 172.21.15.26:0/3637663350 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3314137e00 con 0x7f331409cb80 2024-09-16T14:59:28.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3304001c40 con 0x7f331409cb80 2024-09-16T14:59:28.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:28.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3304004990 con 0x7f331409cb80 2024-09-16T14:59:28.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:28.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:28.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f330b7fe640 4 set_mon_vals no callback set 2024-09-16T14:59:28.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.822+0000 7f3309ffb640 10 monclient: dump: 2024-09-16T14:59:28.844 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:28.844 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:28.844 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.844 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:28.844 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:28.844 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:28.845 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:28.845 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:28.845 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:28.845 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:28.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.826+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f330400f400 con 0x7f331409cb80 2024-09-16T14:59:28.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.826+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f3304043c40 con 0x7f331409cb80 2024-09-16T14:59:28.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.826+0000 7f331b788640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:28.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.826+0000 7f331b788640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:28.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.826+0000 7f331b788640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:28.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.826+0000 7f331b788640 1 -- 172.21.15.26:0/3637663350 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f4005050 con 0x7f331409cb80 2024-09-16T14:59:28.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.826+0000 7f3318d23640 10 monclient: get_auth_request con 0x7f32f80363f0 auth_method 0 2024-09-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.830+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3304014580 con 0x7f331409cb80 2024-09-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.830+0000 7f3309ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:28.830+0000 7f3309ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:59:29.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:29.050+0000 7f331b788640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T14:59:29.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:29.050+0000 7f331b788640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T14:59:29.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:29.050+0000 7f331b788640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:29.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:29.050+0000 7f331b788640 1 -- 172.21.15.26:0/3637663350 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f32f4005b10 con 0x7f331409cb80 2024-09-16T14:59:29.054 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,config=2,health=11,kv=11,logm=55} crc {auth=2727625593,config=692619927,health=1458390088,kv=1867599010,logm=2848244918}) 2024-09-16T14:59:29.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=1977112250,config=692619927,health=2313729962,kv=1867599010,logm=2653411703}) 2024-09-16T14:59:29.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,config=2,health=11,kv=11,logm=55} crc {auth=2727625593,config=692619927,health=1458390088,kv=1867599010,logm=2848244918}) 2024-09-16T14:59:29.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=11,kv=11,logm=51} crc {auth=813910348,config=692619927,health=2851302572,kv=1867599010,logm=2357628483}) 2024-09-16T14:59:29.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:29.050+0000 7f3309ffb640 1 -- 172.21.15.26:0/3637663350 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f33040129f0 con 0x7f331409cb80 2024-09-16T14:59:29.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:29.050+0000 7f3309ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T14:59:29.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:29.050+0000 7f3309ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:59:29.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:29.054+0000 7f331b788640 1 -- 172.21.15.26:0/3637663350 >> v1:172.21.15.189:6812/3196161582 conn(0x7f32f80363f0 legacy=0x7f32f80388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:29.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:29.054+0000 7f331b788640 10 monclient: shutdown 2024-09-16T14:59:29.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:29.054+0000 7f331b788640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:29.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:29.054+0000 7f331b788640 1 -- 172.21.15.26:0/3637663350 >> v1:172.21.15.26:6789/0 conn(0x7f331409cb80 legacy=0x7f3314138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:29.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:29.054+0000 7f331b788640 1 -- 172.21.15.26:0/3637663350 shutdown_connections 2024-09-16T14:59:29.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:29.054+0000 7f331b788640 1 -- 172.21.15.26:0/3637663350 >> 172.21.15.26:0/3637663350 conn(0x7f331409fb50 msgr2=0x7f3314131770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:29.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:29.054+0000 7f331b788640 1 -- 172.21.15.26:0/3637663350 shutdown_connections 2024-09-16T14:59:29.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:29.054+0000 7f331b788640 1 -- 172.21.15.26:0/3637663350 wait complete. 2024-09-16T14:59:29.058 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.058 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2293474331}) 2024-09-16T14:59:29.058 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1275160982}) 2024-09-16T14:59:29.058 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.058 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2293474331}) 2024-09-16T14:59:29.058 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=271920572}) 2024-09-16T14:59:29.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=205783138}) 2024-09-16T14:59:29.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384381150}) 2024-09-16T14:59:29.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=205783138}) 2024-09-16T14:59:29.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4208183524}) 2024-09-16T14:59:29.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.062+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.062+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4150304805}) 2024-09-16T14:59:29.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.062+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151915939}) 2024-09-16T14:59:29.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.062+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.062+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4150304805}) 2024-09-16T14:59:29.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.062+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3893295615}) 2024-09-16T14:59:29.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2058687482}) 2024-09-16T14:59:29.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1970377601}) 2024-09-16T14:59:29.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2058687482}) 2024-09-16T14:59:29.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1854385419}) 2024-09-16T14:59:29.073 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T14:59:29.074 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.074 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1681176781}) 2024-09-16T14:59:29.074 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652450752}) 2024-09-16T14:59:29.074 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.074 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1681176781}) 2024-09-16T14:59:29.074 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=396502948}) 2024-09-16T14:59:29.076 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.076 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1399148737}) 2024-09-16T14:59:29.076 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1314658260}) 2024-09-16T14:59:29.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1399148737}) 2024-09-16T14:59:29.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3835852538}) 2024-09-16T14:59:29.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3973923004}) 2024-09-16T14:59:29.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=389037989}) 2024-09-16T14:59:29.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3973923004}) 2024-09-16T14:59:29.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2809928609}) 2024-09-16T14:59:29.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.078+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.078+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948204744}) 2024-09-16T14:59:29.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.078+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740844504}) 2024-09-16T14:59:29.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.078+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.078+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948204744}) 2024-09-16T14:59:29.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.078+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2847392867}) 2024-09-16T14:59:29.083 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.083 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3139258297}) 2024-09-16T14:59:29.083 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=530485018}) 2024-09-16T14:59:29.083 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.083 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3139258297}) 2024-09-16T14:59:29.083 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1693849918}) 2024-09-16T14:59:29.085 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.085 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2442082001}) 2024-09-16T14:59:29.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=894116636}) 2024-09-16T14:59:29.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2442082001}) 2024-09-16T14:59:29.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1298013800}) 2024-09-16T14:59:29.087 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255094831}) 2024-09-16T14:59:29.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413287434}) 2024-09-16T14:59:29.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255094831}) 2024-09-16T14:59:29.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2703343377}) 2024-09-16T14:59:29.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2876078893}) 2024-09-16T14:59:29.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287109765}) 2024-09-16T14:59:29.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2876078893}) 2024-09-16T14:59:29.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1918725076}) 2024-09-16T14:59:29.092 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.092 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948336379}) 2024-09-16T14:59:29.092 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1904485436}) 2024-09-16T14:59:29.092 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.092 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948336379}) 2024-09-16T14:59:29.092 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1808811403}) 2024-09-16T14:59:29.094 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.094 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=801439401}) 2024-09-16T14:59:29.094 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=676493992}) 2024-09-16T14:59:29.094 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.094 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=801439401}) 2024-09-16T14:59:29.094 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4132013418}) 2024-09-16T14:59:29.096 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.096 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3770667038}) 2024-09-16T14:59:29.096 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1458733147}) 2024-09-16T14:59:29.097 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.097 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3770667038}) 2024-09-16T14:59:29.097 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781830776}) 2024-09-16T14:59:29.098 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.099 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1892837803}) 2024-09-16T14:59:29.099 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3167775667}) 2024-09-16T14:59:29.099 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.099 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1892837803}) 2024-09-16T14:59:29.099 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3203507019}) 2024-09-16T14:59:29.101 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.101 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1324753613}) 2024-09-16T14:59:29.101 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3274345718}) 2024-09-16T14:59:29.101 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.101 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1324753613}) 2024-09-16T14:59:29.101 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1604067921}) 2024-09-16T14:59:29.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2671437471}) 2024-09-16T14:59:29.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=161351228}) 2024-09-16T14:59:29.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2671437471}) 2024-09-16T14:59:29.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2483956397}) 2024-09-16T14:59:29.105 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.105 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2795872052}) 2024-09-16T14:59:29.105 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=570171218}) 2024-09-16T14:59:29.105 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.105 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2795872052}) 2024-09-16T14:59:29.105 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=490741526}) 2024-09-16T14:59:29.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=787590782}) 2024-09-16T14:59:29.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516767426}) 2024-09-16T14:59:29.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=787590782}) 2024-09-16T14:59:29.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86478433}) 2024-09-16T14:59:29.109 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.109 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3752331529}) 2024-09-16T14:59:29.109 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2971917855}) 2024-09-16T14:59:29.110 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.110 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3752331529}) 2024-09-16T14:59:29.110 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1313279930}) 2024-09-16T14:59:29.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3484341208}) 2024-09-16T14:59:29.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3664174595}) 2024-09-16T14:59:29.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.112 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3484341208}) 2024-09-16T14:59:29.112 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3621726607}) 2024-09-16T14:59:29.113 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=216740674}) 2024-09-16T14:59:29.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1831902630}) 2024-09-16T14:59:29.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=216740674}) 2024-09-16T14:59:29.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=805201024}) 2024-09-16T14:59:29.115 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3950032850}) 2024-09-16T14:59:29.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3503715097}) 2024-09-16T14:59:29.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3950032850}) 2024-09-16T14:59:29.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4097475394}) 2024-09-16T14:59:29.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4291586536}) 2024-09-16T14:59:29.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822002382}) 2024-09-16T14:59:29.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4291586536}) 2024-09-16T14:59:29.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=614213510}) 2024-09-16T14:59:29.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1312313596}) 2024-09-16T14:59:29.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2937483767}) 2024-09-16T14:59:29.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1312313596}) 2024-09-16T14:59:29.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1656061059}) 2024-09-16T14:59:29.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2596571848}) 2024-09-16T14:59:29.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3728800022}) 2024-09-16T14:59:29.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2596571848}) 2024-09-16T14:59:29.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2628256070}) 2024-09-16T14:59:29.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1366907318}) 2024-09-16T14:59:29.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918634682}) 2024-09-16T14:59:29.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1366907318}) 2024-09-16T14:59:29.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1401045206}) 2024-09-16T14:59:29.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=913564231}) 2024-09-16T14:59:29.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461710284}) 2024-09-16T14:59:29.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=913564231}) 2024-09-16T14:59:29.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2122649509}) 2024-09-16T14:59:29.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3958366975}) 2024-09-16T14:59:29.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4134106423}) 2024-09-16T14:59:29.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3958366975}) 2024-09-16T14:59:29.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2132826336}) 2024-09-16T14:59:29.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2698117317}) 2024-09-16T14:59:29.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014305348}) 2024-09-16T14:59:29.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2698117317}) 2024-09-16T14:59:29.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1691759025}) 2024-09-16T14:59:29.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3221385370}) 2024-09-16T14:59:29.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3352040497}) 2024-09-16T14:59:29.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3221385370}) 2024-09-16T14:59:29.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1216640747}) 2024-09-16T14:59:29.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1276309378}) 2024-09-16T14:59:29.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2262238304}) 2024-09-16T14:59:29.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1276309378}) 2024-09-16T14:59:29.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=291203013}) 2024-09-16T14:59:29.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3972983699}) 2024-09-16T14:59:29.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3129804097}) 2024-09-16T14:59:29.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3972983699}) 2024-09-16T14:59:29.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2747656132}) 2024-09-16T14:59:29.139 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.139 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1569856237}) 2024-09-16T14:59:29.139 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832510736}) 2024-09-16T14:59:29.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1569856237}) 2024-09-16T14:59:29.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2629599368}) 2024-09-16T14:59:29.143 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.143 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1130391165}) 2024-09-16T14:59:29.143 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259414004}) 2024-09-16T14:59:29.143 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.143 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1130391165}) 2024-09-16T14:59:29.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3147479993}) 2024-09-16T14:59:29.152 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.152 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2918764733}) 2024-09-16T14:59:29.152 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775475873}) 2024-09-16T14:59:29.152 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.152 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2918764733}) 2024-09-16T14:59:29.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4243179743}) 2024-09-16T14:59:29.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3173400056}) 2024-09-16T14:59:29.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3541193763}) 2024-09-16T14:59:29.158 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.158 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3173400056}) 2024-09-16T14:59:29.158 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1158427539}) 2024-09-16T14:59:29.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2934449451}) 2024-09-16T14:59:29.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64480885}) 2024-09-16T14:59:29.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.160 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2934449451}) 2024-09-16T14:59:29.160 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2944718011}) 2024-09-16T14:59:29.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1766675310}) 2024-09-16T14:59:29.162 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408362477}) 2024-09-16T14:59:29.162 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.162 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1766675310}) 2024-09-16T14:59:29.162 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=103579028}) 2024-09-16T14:59:29.163 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.163 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1169516260}) 2024-09-16T14:59:29.163 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46354282}) 2024-09-16T14:59:29.163 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.164 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1169516260}) 2024-09-16T14:59:29.164 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=696616272}) 2024-09-16T14:59:29.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3064148348}) 2024-09-16T14:59:29.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340943683}) 2024-09-16T14:59:29.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3064148348}) 2024-09-16T14:59:29.166 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1009198524}) 2024-09-16T14:59:29.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2392839109}) 2024-09-16T14:59:29.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020898910}) 2024-09-16T14:59:29.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2392839109}) 2024-09-16T14:59:29.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1901388197}) 2024-09-16T14:59:29.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=764273440}) 2024-09-16T14:59:29.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333646112}) 2024-09-16T14:59:29.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=764273440}) 2024-09-16T14:59:29.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2665391946}) 2024-09-16T14:59:29.171 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.171 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4081645077}) 2024-09-16T14:59:29.171 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2633864277}) 2024-09-16T14:59:29.171 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.171 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4081645077}) 2024-09-16T14:59:29.171 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3412135747}) 2024-09-16T14:59:29.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.173 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=958597137}) 2024-09-16T14:59:29.173 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=224037784}) 2024-09-16T14:59:29.173 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.173 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=958597137}) 2024-09-16T14:59:29.173 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=739809301}) 2024-09-16T14:59:29.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1990732584}) 2024-09-16T14:59:29.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3310194713}) 2024-09-16T14:59:29.175 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.175 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1990732584}) 2024-09-16T14:59:29.175 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3360523113}) 2024-09-16T14:59:29.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2626259160}) 2024-09-16T14:59:29.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2687396796}) 2024-09-16T14:59:29.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2626259160}) 2024-09-16T14:59:29.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1380752079}) 2024-09-16T14:59:29.178 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.178 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4190968610}) 2024-09-16T14:59:29.178 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1840924033}) 2024-09-16T14:59:29.178 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.178 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4190968610}) 2024-09-16T14:59:29.178 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1553307542}) 2024-09-16T14:59:29.180 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.180 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2805636899}) 2024-09-16T14:59:29.180 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=576030127}) 2024-09-16T14:59:29.180 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.180 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2805636899}) 2024-09-16T14:59:29.180 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=284343582}) 2024-09-16T14:59:29.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=408683499}) 2024-09-16T14:59:29.182 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=990490708}) 2024-09-16T14:59:29.182 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.182 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=408683499}) 2024-09-16T14:59:29.182 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2168175178}) 2024-09-16T14:59:29.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=178644600}) 2024-09-16T14:59:29.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472785643}) 2024-09-16T14:59:29.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=178644600}) 2024-09-16T14:59:29.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3768594746}) 2024-09-16T14:59:29.185 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.185 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2574322652}) 2024-09-16T14:59:29.185 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097190642}) 2024-09-16T14:59:29.185 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2574322652}) 2024-09-16T14:59:29.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3147615749}) 2024-09-16T14:59:29.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3097012147}) 2024-09-16T14:59:29.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=862835285}) 2024-09-16T14:59:29.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3097012147}) 2024-09-16T14:59:29.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3334409435}) 2024-09-16T14:59:29.189 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.189 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=218131775}) 2024-09-16T14:59:29.189 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1303343138}) 2024-09-16T14:59:29.189 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.189 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=218131775}) 2024-09-16T14:59:29.189 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2069728103}) 2024-09-16T14:59:29.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3077317982}) 2024-09-16T14:59:29.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=119650343}) 2024-09-16T14:59:29.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3077317982}) 2024-09-16T14:59:29.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1124437290}) 2024-09-16T14:59:29.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=977697639}) 2024-09-16T14:59:29.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3871007730}) 2024-09-16T14:59:29.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=977697639}) 2024-09-16T14:59:29.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3853495102}) 2024-09-16T14:59:29.194 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2294403771}) 2024-09-16T14:59:29.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=813138426}) 2024-09-16T14:59:29.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2294403771}) 2024-09-16T14:59:29.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=548991779}) 2024-09-16T14:59:29.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1627291184}) 2024-09-16T14:59:29.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=110882264}) 2024-09-16T14:59:29.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1627291184}) 2024-09-16T14:59:29.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1511896011}) 2024-09-16T14:59:29.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2951488403}) 2024-09-16T14:59:29.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1249796088}) 2024-09-16T14:59:29.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2951488403}) 2024-09-16T14:59:29.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2975331471}) 2024-09-16T14:59:29.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1094071926}) 2024-09-16T14:59:29.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3780849692}) 2024-09-16T14:59:29.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1094071926}) 2024-09-16T14:59:29.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2856521890}) 2024-09-16T14:59:29.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3630838627}) 2024-09-16T14:59:29.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1917995483}) 2024-09-16T14:59:29.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3630838627}) 2024-09-16T14:59:29.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3674863496}) 2024-09-16T14:59:29.203 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.203 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705983224}) 2024-09-16T14:59:29.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=925412017}) 2024-09-16T14:59:29.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705983224}) 2024-09-16T14:59:29.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=353848623}) 2024-09-16T14:59:29.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690119042}) 2024-09-16T14:59:29.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137397181}) 2024-09-16T14:59:29.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690119042}) 2024-09-16T14:59:29.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2047886642}) 2024-09-16T14:59:29.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3281820102}) 2024-09-16T14:59:29.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1854820810}) 2024-09-16T14:59:29.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.208 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3281820102}) 2024-09-16T14:59:29.208 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=30349055}) 2024-09-16T14:59:29.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1285081799}) 2024-09-16T14:59:29.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=369621226}) 2024-09-16T14:59:29.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1285081799}) 2024-09-16T14:59:29.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1421565149}) 2024-09-16T14:59:29.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3988875614}) 2024-09-16T14:59:29.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1814406870}) 2024-09-16T14:59:29.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3988875614}) 2024-09-16T14:59:29.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2128170191}) 2024-09-16T14:59:29.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=242932868}) 2024-09-16T14:59:29.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1275848838}) 2024-09-16T14:59:29.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=242932868}) 2024-09-16T14:59:29.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1283264621}) 2024-09-16T14:59:29.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3324679877}) 2024-09-16T14:59:29.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286288959}) 2024-09-16T14:59:29.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3324679877}) 2024-09-16T14:59:29.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3541590031}) 2024-09-16T14:59:29.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=768158043}) 2024-09-16T14:59:29.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2769398901}) 2024-09-16T14:59:29.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=768158043}) 2024-09-16T14:59:29.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3564139139}) 2024-09-16T14:59:29.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4042462434}) 2024-09-16T14:59:29.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3927288874}) 2024-09-16T14:59:29.220 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.220 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4042462434}) 2024-09-16T14:59:29.220 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=234763874}) 2024-09-16T14:59:29.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605900731}) 2024-09-16T14:59:29.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553625142}) 2024-09-16T14:59:29.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605900731}) 2024-09-16T14:59:29.222 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2693635940}) 2024-09-16T14:59:29.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=555850640}) 2024-09-16T14:59:29.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388622722}) 2024-09-16T14:59:29.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=555850640}) 2024-09-16T14:59:29.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1546630831}) 2024-09-16T14:59:29.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1335258138}) 2024-09-16T14:59:29.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1873079531}) 2024-09-16T14:59:29.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1335258138}) 2024-09-16T14:59:29.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=854170479}) 2024-09-16T14:59:29.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3835983976}) 2024-09-16T14:59:29.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1012628483}) 2024-09-16T14:59:29.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3835983976}) 2024-09-16T14:59:29.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4223408600}) 2024-09-16T14:59:29.228 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.228 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1469588947}) 2024-09-16T14:59:29.228 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=429233519}) 2024-09-16T14:59:29.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1469588947}) 2024-09-16T14:59:29.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3867483430}) 2024-09-16T14:59:29.230 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.230 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=760960692}) 2024-09-16T14:59:29.230 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2118521589}) 2024-09-16T14:59:29.230 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.230 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=760960692}) 2024-09-16T14:59:29.230 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2956208450}) 2024-09-16T14:59:29.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1903770532}) 2024-09-16T14:59:29.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2278797057}) 2024-09-16T14:59:29.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1903770532}) 2024-09-16T14:59:29.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3825232438}) 2024-09-16T14:59:29.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2990170605}) 2024-09-16T14:59:29.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2829159714}) 2024-09-16T14:59:29.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.235 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2990170605}) 2024-09-16T14:59:29.235 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=42989570}) 2024-09-16T14:59:29.235 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4160165191}) 2024-09-16T14:59:29.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3335759064}) 2024-09-16T14:59:29.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4160165191}) 2024-09-16T14:59:29.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=976262246}) 2024-09-16T14:59:29.237 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.237 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=941055994}) 2024-09-16T14:59:29.237 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=805346806}) 2024-09-16T14:59:29.237 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.237 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=941055994}) 2024-09-16T14:59:29.237 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=526871856}) 2024-09-16T14:59:29.239 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.239 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=893090888}) 2024-09-16T14:59:29.239 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067538588}) 2024-09-16T14:59:29.239 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.239 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=893090888}) 2024-09-16T14:59:29.239 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2583411661}) 2024-09-16T14:59:29.241 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.241 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3098420392}) 2024-09-16T14:59:29.241 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1386199410}) 2024-09-16T14:59:29.241 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.241 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3098420392}) 2024-09-16T14:59:29.241 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3105770791}) 2024-09-16T14:59:29.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.243 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2506276756}) 2024-09-16T14:59:29.243 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713774204}) 2024-09-16T14:59:29.243 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.243 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2506276756}) 2024-09-16T14:59:29.243 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=325262047}) 2024-09-16T14:59:29.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2429350819}) 2024-09-16T14:59:29.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1401761700}) 2024-09-16T14:59:29.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.245 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2429350819}) 2024-09-16T14:59:29.245 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3441840599}) 2024-09-16T14:59:29.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835458815}) 2024-09-16T14:59:29.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3620111997}) 2024-09-16T14:59:29.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835458815}) 2024-09-16T14:59:29.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1552140944}) 2024-09-16T14:59:29.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1857325579}) 2024-09-16T14:59:29.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3696915794}) 2024-09-16T14:59:29.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1857325579}) 2024-09-16T14:59:29.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=531273039}) 2024-09-16T14:59:29.249 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.249 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247521261}) 2024-09-16T14:59:29.249 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2614112107}) 2024-09-16T14:59:29.250 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.250 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247521261}) 2024-09-16T14:59:29.250 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4185297383}) 2024-09-16T14:59:29.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4195225793}) 2024-09-16T14:59:29.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1413629387}) 2024-09-16T14:59:29.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4195225793}) 2024-09-16T14:59:29.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2667021422}) 2024-09-16T14:59:29.253 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.253 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3643876384}) 2024-09-16T14:59:29.253 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3343345100}) 2024-09-16T14:59:29.253 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.253 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3643876384}) 2024-09-16T14:59:29.253 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1917003367}) 2024-09-16T14:59:29.255 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.255 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3822273487}) 2024-09-16T14:59:29.255 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2622342859}) 2024-09-16T14:59:29.255 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.255 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3822273487}) 2024-09-16T14:59:29.255 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3976050533}) 2024-09-16T14:59:29.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=440493389}) 2024-09-16T14:59:29.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033802357}) 2024-09-16T14:59:29.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=440493389}) 2024-09-16T14:59:29.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3310688052}) 2024-09-16T14:59:29.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.258 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2243265635}) 2024-09-16T14:59:29.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2794198298}) 2024-09-16T14:59:29.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2243265635}) 2024-09-16T14:59:29.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1726493649}) 2024-09-16T14:59:29.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516130342}) 2024-09-16T14:59:29.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2993441744}) 2024-09-16T14:59:29.260 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516130342}) 2024-09-16T14:59:29.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=275538951}) 2024-09-16T14:59:29.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1190726178}) 2024-09-16T14:59:29.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3751670100}) 2024-09-16T14:59:29.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1190726178}) 2024-09-16T14:59:29.262 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1985389487}) 2024-09-16T14:59:29.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3123268989}) 2024-09-16T14:59:29.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=229313418}) 2024-09-16T14:59:29.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3123268989}) 2024-09-16T14:59:29.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417364502}) 2024-09-16T14:59:29.265 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.265 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=648002067}) 2024-09-16T14:59:29.265 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1072631442}) 2024-09-16T14:59:29.265 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=648002067}) 2024-09-16T14:59:29.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=536133376}) 2024-09-16T14:59:29.267 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.267 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2613265892}) 2024-09-16T14:59:29.267 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145451380}) 2024-09-16T14:59:29.267 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.267 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2613265892}) 2024-09-16T14:59:29.267 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3872166218}) 2024-09-16T14:59:29.269 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.269 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=801291198}) 2024-09-16T14:59:29.269 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=395884280}) 2024-09-16T14:59:29.269 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.269 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=801291198}) 2024-09-16T14:59:29.269 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2109732813}) 2024-09-16T14:59:29.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.270 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2039445110}) 2024-09-16T14:59:29.271 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2513012761}) 2024-09-16T14:59:29.271 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.271 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2039445110}) 2024-09-16T14:59:29.271 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.266+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1992978023}) 2024-09-16T14:59:29.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.272 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3008727017}) 2024-09-16T14:59:29.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2219922446}) 2024-09-16T14:59:29.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3008727017}) 2024-09-16T14:59:29.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3552469819}) 2024-09-16T14:59:29.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1193931991}) 2024-09-16T14:59:29.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=211499204}) 2024-09-16T14:59:29.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1193931991}) 2024-09-16T14:59:29.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2038674161}) 2024-09-16T14:59:29.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.274+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.274+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=523902833}) 2024-09-16T14:59:29.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.274+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390348390}) 2024-09-16T14:59:29.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.274+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.274+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=523902833}) 2024-09-16T14:59:29.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.274+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=251355538}) 2024-09-16T14:59:29.277 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.274+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.274+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679506083}) 2024-09-16T14:59:29.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.274+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=484665599}) 2024-09-16T14:59:29.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.274+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.274+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679506083}) 2024-09-16T14:59:29.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.274+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1717342538}) 2024-09-16T14:59:29.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.278+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.278+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2400425100}) 2024-09-16T14:59:29.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.278+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=414286063}) 2024-09-16T14:59:29.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.278+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.278+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2400425100}) 2024-09-16T14:59:29.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.278+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1268130707}) 2024-09-16T14:59:29.281 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.278+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.281 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.278+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3091934800}) 2024-09-16T14:59:29.281 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.278+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624608180}) 2024-09-16T14:59:29.281 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.278+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.282 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.278+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3091934800}) 2024-09-16T14:59:29.282 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.278+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1641359504}) 2024-09-16T14:59:29.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.282+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.282+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2081822496}) 2024-09-16T14:59:29.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.282+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1786302677}) 2024-09-16T14:59:29.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.282+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.282+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2081822496}) 2024-09-16T14:59:29.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.282+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181295119}) 2024-09-16T14:59:29.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.282+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.282+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=69152834}) 2024-09-16T14:59:29.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.282+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010024856}) 2024-09-16T14:59:29.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.282+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.282+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=69152834}) 2024-09-16T14:59:29.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.282+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782288327}) 2024-09-16T14:59:29.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.286+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.286+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4066240304}) 2024-09-16T14:59:29.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.286+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606857198}) 2024-09-16T14:59:29.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.286+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.286+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4066240304}) 2024-09-16T14:59:29.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.286+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648622813}) 2024-09-16T14:59:29.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.286+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.286+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=5976218}) 2024-09-16T14:59:29.290 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.286+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=795652002}) 2024-09-16T14:59:29.290 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.286+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.290 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.286+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=5976218}) 2024-09-16T14:59:29.290 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.286+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603456126}) 2024-09-16T14:59:29.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1877640135}) 2024-09-16T14:59:29.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177823332}) 2024-09-16T14:59:29.292 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.292 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1877640135}) 2024-09-16T14:59:29.292 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1699724870}) 2024-09-16T14:59:29.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646212365}) 2024-09-16T14:59:29.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2103007525}) 2024-09-16T14:59:29.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646212365}) 2024-09-16T14:59:29.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3598466603}) 2024-09-16T14:59:29.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2486119250}) 2024-09-16T14:59:29.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=829750157}) 2024-09-16T14:59:29.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2486119250}) 2024-09-16T14:59:29.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.290+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=398488854}) 2024-09-16T14:59:29.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.294+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.294+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=23,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=31} crc {logm=3432895397,mdsmap=3585761405,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3417497542,mon_config_key=859304116,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=309386618}) 2024-09-16T14:59:29.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.294+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=21,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=36} crc {logm=2678620362,mdsmap=1531151339,mgr=460219048,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2730239896,mon_config_key=285498709,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=642700999}) 2024-09-16T14:59:29.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.294+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.294+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=23,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=31} crc {logm=3432895397,mdsmap=3585761405,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3417497542,mon_config_key=859304116,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=309386618}) 2024-09-16T14:59:29.298 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.294+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=21,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=32} crc {logm=4204335150,mdsmap=1531151339,mgr=2683218624,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3610620226,mon_config_key=1518680965,monmap=970648238,osd_metadata=1524398128,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1525799898}) 2024-09-16T14:59:29.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.306+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.306+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=19} crc {osdmap=1745805975}) 2024-09-16T14:59:29.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.306+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=18} crc {osdmap=593714857}) 2024-09-16T14:59:29.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.306+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T14:59:29.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.306+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=19} crc {osdmap=1745805975}) 2024-09-16T14:59:29.311 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T14:59:29.306+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=15} crc {osdmap=774565323}) 2024-09-16T14:59:30.074 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T14:59:30.074 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:59:30.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.206+0000 7f7beae33640 10 monclient: get_monmap_and_config 2024-09-16T14:59:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.206+0000 7f7beae33640 10 monclient: build_initial_monmap 2024-09-16T14:59:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.206+0000 7f7beae33640 10 monclient: monmap: 2024-09-16T14:59:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:59:30.212448+0000 2024-09-16T14:59:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:59:30.212448+0000 2024-09-16T14:59:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:59:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:59:30.211 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.206+0000 7f7beae33640 1 Processor -- start 2024-09-16T14:59:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.206+0000 7f7beae33640 1 -- start start 2024-09-16T14:59:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.206+0000 7f7beae33640 10 monclient: init 2024-09-16T14:59:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:59:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient(hunting): picked mon.noname-a con 0x7f7be40a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient(hunting): picked mon.noname-c con 0x7f7be40a0000 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient(hunting): picked mon.noname-b con 0x7f7be40a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7be40a3250 con 0x7f7be40a1d10 2024-09-16T14:59:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7be409f600 con 0x7f7be40a0000 2024-09-16T14:59:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7be40a3830 con 0x7f7be40a2800 2024-09-16T14:59:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient(hunting): authenticate will time out at 1994.332723s 2024-09-16T14:59:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be8bcf640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7be40a1d10 0x7f7be40a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34668/0 (socket says 172.21.15.26:34668) 2024-09-16T14:59:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be8bcf640 1 -- 172.21.15.26:0/126376695 learned_addr learned my addr 172.21.15.26:0/126376695 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be93d0640 1 --1- 172.21.15.26:0/126376695 >> v1:172.21.15.26:6790/0 conn(0x7f7be40a2800 0x7f7be413d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48134/0 (socket says 172.21.15.26:48134) 2024-09-16T14:59:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 1 -- 172.21.15.26:0/126376695 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255122788 0 0) 0x7f7be40a3250 con 0x7f7be40a1d10 2024-09-16T14:59:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient(hunting): my global_id is 324132 2024-09-16T14:59:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 1 -- 172.21.15.26:0/126376695 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bd0003530 con 0x7f7be40a1d10 2024-09-16T14:59:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 1 -- 172.21.15.26:0/126376695 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844963620 0 0) 0x7f7be40a3830 con 0x7f7be40a2800 2024-09-16T14:59:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient(hunting): my global_id is 284561 2024-09-16T14:59:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 1 -- 172.21.15.26:0/126376695 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7be40a3250 con 0x7f7be40a2800 2024-09-16T14:59:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 1 -- 172.21.15.26:0/126376695 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3840700025 0 0) 0x7f7bd0003530 con 0x7f7be40a1d10 2024-09-16T14:59:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 1 -- 172.21.15.26:0/126376695 >> v1:172.21.15.26:6790/0 conn(0x7f7be40a2800 legacy=0x7f7be413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 1 -- 172.21.15.26:0/126376695 >> v1:172.21.15.189:6789/0 conn(0x7f7be40a0000 legacy=0x7f7be4139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient: _finish_hunting 0 2024-09-16T14:59:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 1 monclient: found mon.noname-a 2024-09-16T14:59:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:59:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 1 -- 172.21.15.26:0/126376695 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be409ed60 con 0x7f7be40a1d10 2024-09-16T14:59:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient: _finish_auth 0 2024-09-16T14:59:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient: _check_auth_tickets 2024-09-16T14:59:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 1 -- 172.21.15.26:0/126376695 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7bdc0019a0 con 0x7f7be40a1d10 2024-09-16T14:59:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:59:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient: dump: 2024-09-16T14:59:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.218 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 5 monclient: authenticate success, global_id 324132 2024-09-16T14:59:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 1 -- 172.21.15.26:0/126376695 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bdc001c40 con 0x7f7be40a1d10 2024-09-16T14:59:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient: get_monmap_and_config success 2024-09-16T14:59:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 1 -- 172.21.15.26:0/126376695 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7bdc001fd0 con 0x7f7be40a1d10 2024-09-16T14:59:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 4 set_mon_vals no callback set 2024-09-16T14:59:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be37fe640 10 monclient: dump: 2024-09-16T14:59:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.219 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient: shutdown 2024-09-16T14:59:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 1 -- 172.21.15.26:0/126376695 >> v1:172.21.15.26:6789/0 conn(0x7f7be40a1d10 legacy=0x7f7be40a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 1 -- 172.21.15.26:0/126376695 shutdown_connections 2024-09-16T14:59:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 1 -- 172.21.15.26:0/126376695 >> 172.21.15.26:0/126376695 conn(0x7f7be40a7f30 msgr2=0x7f7be40a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 1 -- 172.21.15.26:0/126376695 shutdown_connections 2024-09-16T14:59:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 1 -- 172.21.15.26:0/126376695 wait complete. 2024-09-16T14:59:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient: build_initial_monmap 2024-09-16T14:59:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient: monmap: 2024-09-16T14:59:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:59:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:59:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:59:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:59:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:59:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:59:30.221 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 1 Processor -- start 2024-09-16T14:59:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 1 -- start start 2024-09-16T14:59:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient: init 2024-09-16T14:59:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:59:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient(hunting): picked mon.noname-c con 0x7f7be40a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient(hunting): picked mon.noname-a con 0x7f7be40a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient(hunting): picked mon.noname-b con 0x7f7be40a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7be409e500 con 0x7f7be40a1d10 2024-09-16T14:59:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7be413d2a0 con 0x7f7be40a2800 2024-09-16T14:59:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7be413ca00 con 0x7f7be40a0000 2024-09-16T14:59:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7beae33640 10 monclient(hunting): authenticate will time out at 1994.335853s 2024-09-16T14:59:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be8bcf640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7be40a0000 0x7f7be41393c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48146/0 (socket says 172.21.15.26:48146) 2024-09-16T14:59:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.210+0000 7f7be8bcf640 1 -- 172.21.15.26:0/3361653379 learned_addr learned my addr 172.21.15.26:0/3361653379 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968494951 0 0) 0x7f7be409e500 con 0x7f7be40a1d10 2024-09-16T14:59:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): my global_id is 324135 2024-09-16T14:59:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bc4003530 con 0x7f7be40a1d10 2024-09-16T14:59:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191509646 0 0) 0x7f7be413ca00 con 0x7f7be40a0000 2024-09-16T14:59:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): my global_id is 284564 2024-09-16T14:59:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7be409e500 con 0x7f7be40a0000 2024-09-16T14:59:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3754556830 0 0) 0x7f7bc4003530 con 0x7f7be40a1d10 2024-09-16T14:59:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7be413ca00 con 0x7f7be40a1d10 2024-09-16T14:59:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7bcc001880 con 0x7f7be40a1d10 2024-09-16T14:59:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:59:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): dump: 2024-09-16T14:59:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.228 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:59:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2229658884 0 0) 0x7f7be409e500 con 0x7f7be40a0000 2024-09-16T14:59:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bc4003530 con 0x7f7be40a0000 2024-09-16T14:59:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7bdc001f50 con 0x7f7be40a0000 2024-09-16T14:59:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): dump: 2024-09-16T14:59:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.229 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275222044 0 0) 0x7f7be413d2a0 con 0x7f7be40a2800 2024-09-16T14:59:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient(hunting): my global_id is 304309 2024-09-16T14:59:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7be409e500 con 0x7f7be40a2800 2024-09-16T14:59:30.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2686958399 0 0) 0x7f7be413ca00 con 0x7f7be40a1d10 2024-09-16T14:59:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 >> v1:172.21.15.26:6790/0 conn(0x7f7be40a0000 legacy=0x7f7be41393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 >> v1:172.21.15.189:6789/0 conn(0x7f7be40a2800 legacy=0x7f7be41d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient: _finish_hunting 0 2024-09-16T14:59:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 monclient: found mon.a 2024-09-16T14:59:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be413aed0 con 0x7f7be40a1d10 2024-09-16T14:59:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient: _finish_auth 0 2024-09-16T14:59:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient: _check_auth_tickets 2024-09-16T14:59:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7beae33640 5 monclient: authenticate success, global_id 324135 2024-09-16T14:59:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7beae33640 10 monclient: _renew_subs 2024-09-16T14:59:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7beae33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7beae33640 1 -- 172.21.15.26:0/3361653379 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be413a1e0 con 0x7f7be40a1d10 2024-09-16T14:59:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7beae33640 10 monclient: _renew_subs 2024-09-16T14:59:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7beae33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7beae33640 1 -- 172.21.15.26:0/3361653379 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be413da20 con 0x7f7be40a1d10 2024-09-16T14:59:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bcc003660 con 0x7f7be40a1d10 2024-09-16T14:59:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f7bcc003970 con 0x7f7be40a1d10 2024-09-16T14:59:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 10 monclient: dump: 2024-09-16T14:59:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.231 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.232 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.232 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.232 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.232 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.232 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.232 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.232 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be2ffd640 4 set_mon_vals no callback set 2024-09-16T14:59:30.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f7bcc003ad0 con 0x7f7be40a1d10 2024-09-16T14:59:30.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7beae33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:30.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7beae33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:30.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7beae33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:30.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f7bcc042c00 con 0x7f7be40a1d10 2024-09-16T14:59:30.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7beae33640 1 -- 172.21.15.26:0/3361653379 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc8005050 con 0x7f7be40a1d10 2024-09-16T14:59:30.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.214+0000 7f7be8bcf640 10 monclient: get_auth_request con 0x7f7bc4036440 auth_method 0 2024-09-16T14:59:30.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.218+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7bcc013560 con 0x7f7be40a1d10 2024-09-16T14:59:30.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.218+0000 7f7be17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:30.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.218+0000 7f7be17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:59:30.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.426+0000 7f7beae33640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:59:30.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.426+0000 7f7beae33640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:59:30.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.426+0000 7f7beae33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:30.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.426+0000 7f7beae33640 1 -- 172.21.15.26:0/3361653379 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7bc8005880 con 0x7f7be40a1d10 2024-09-16T14:59:30.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.430+0000 7f7be17fa640 1 -- 172.21.15.26:0/3361653379 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 690435719) 0x7f7bcc01bb50 con 0x7f7be40a1d10 2024-09-16T14:59:30.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.430+0000 7f7be17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:59:30.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.430+0000 7f7be17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:59:30.432 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:59:30.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.430+0000 7f7beae33640 1 -- 172.21.15.26:0/3361653379 >> v1:172.21.15.189:6812/3196161582 conn(0x7f7bc4036440 legacy=0x7f7bc4038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.430+0000 7f7beae33640 10 monclient: shutdown 2024-09-16T14:59:30.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.430+0000 7f7beae33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:30.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.430+0000 7f7beae33640 1 -- 172.21.15.26:0/3361653379 >> v1:172.21.15.26:6789/0 conn(0x7f7be40a1d10 legacy=0x7f7be4139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.430+0000 7f7beae33640 1 -- 172.21.15.26:0/3361653379 shutdown_connections 2024-09-16T14:59:30.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.430+0000 7f7beae33640 1 -- 172.21.15.26:0/3361653379 >> 172.21.15.26:0/3361653379 conn(0x7f7be40a7f30 msgr2=0x7f7be406bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:30.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.430+0000 7f7beae33640 1 -- 172.21.15.26:0/3361653379 shutdown_connections 2024-09-16T14:59:30.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.430+0000 7f7beae33640 1 -- 172.21.15.26:0/3361653379 wait complete. 2024-09-16T14:59:30.450 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T14:59:30.450 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T14:59:30.450 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T14:59:30.450 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T14:59:30.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 10 monclient: get_monmap_and_config 2024-09-16T14:59:30.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 10 monclient: build_initial_monmap 2024-09-16T14:59:30.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 10 monclient: monmap: 2024-09-16T14:59:30.559 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:30.559 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.559 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:59:30.561759+0000 2024-09-16T14:59:30.559 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:59:30.561759+0000 2024-09-16T14:59:30.560 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:30.560 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.560 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:30.560 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:59:30.560 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:59:30.560 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 1 Processor -- start 2024-09-16T14:59:30.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 1 -- start start 2024-09-16T14:59:30.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 10 monclient: init 2024-09-16T14:59:30.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:30.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:59:30.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 10 monclient(hunting): picked mon.noname-c con 0x7f8bfc0a1ca0 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:30.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 10 monclient(hunting): picked mon.noname-b con 0x7f8bfc0a2790 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 10 monclient(hunting): picked mon.noname-a con 0x7f8bfc09ff50 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc09f520 con 0x7f8bfc09ff50 2024-09-16T14:59:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc0a0360 con 0x7f8bfc0a1ca0 2024-09-16T14:59:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc09e7a0 con 0x7f8bfc0a2790 2024-09-16T14:59:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 10 monclient(hunting): authenticate will time out at 1994.681762s 2024-09-16T14:59:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfaffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8bfc0a2790 0x7f8bfc139e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48158/0 (socket says 172.21.15.26:48158) 2024-09-16T14:59:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfbfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8bfc09ff50 0x7f8bfc13d590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34684/0 (socket says 172.21.15.26:34684) 2024-09-16T14:59:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfbfff640 1 -- 172.21.15.26:0/2419139711 learned_addr learned my addr 172.21.15.26:0/2419139711 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 1 -- 172.21.15.26:0/2419139711 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549444275 0 0) 0x7f8bfc09f520 con 0x7f8bfc09ff50 2024-09-16T14:59:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient(hunting): my global_id is 324138 2024-09-16T14:59:30.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 1 -- 172.21.15.26:0/2419139711 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8be8003530 con 0x7f8bfc09ff50 2024-09-16T14:59:30.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 1 -- 172.21.15.26:0/2419139711 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084515780 0 0) 0x7f8bfc09e7a0 con 0x7f8bfc0a2790 2024-09-16T14:59:30.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient(hunting): my global_id is 284567 2024-09-16T14:59:30.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 1 -- 172.21.15.26:0/2419139711 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bfc09f520 con 0x7f8bfc0a2790 2024-09-16T14:59:30.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 1 -- 172.21.15.26:0/2419139711 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173949268 0 0) 0x7f8bfc0a0360 con 0x7f8bfc0a1ca0 2024-09-16T14:59:30.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient(hunting): my global_id is 304312 2024-09-16T14:59:30.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 1 -- 172.21.15.26:0/2419139711 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bfc09e7a0 con 0x7f8bfc0a1ca0 2024-09-16T14:59:30.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 1 -- 172.21.15.26:0/2419139711 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3578660522 0 0) 0x7f8be8003530 con 0x7f8bfc09ff50 2024-09-16T14:59:30.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 1 -- 172.21.15.26:0/2419139711 >> v1:172.21.15.26:6790/0 conn(0x7f8bfc0a2790 legacy=0x7f8bfc139e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 1 -- 172.21.15.26:0/2419139711 >> v1:172.21.15.189:6789/0 conn(0x7f8bfc0a1ca0 legacy=0x7f8bfc0a2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:59:30.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 1 monclient: found mon.noname-a 2024-09-16T14:59:30.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:30.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:59:30.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 1 -- 172.21.15.26:0/2419139711 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc09ec80 con 0x7f8bfc09ff50 2024-09-16T14:59:30.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient: _finish_auth 0 2024-09-16T14:59:30.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:59:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 1 -- 172.21.15.26:0/2419139711 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8bec001620 con 0x7f8bfc09ff50 2024-09-16T14:59:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:59:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient: dump: 2024-09-16T14:59:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.574 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.575 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.575 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 5 monclient: authenticate success, global_id 324138 2024-09-16T14:59:30.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:30.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 1 -- 172.21.15.26:0/2419139711 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bec001900 con 0x7f8bfc09ff50 2024-09-16T14:59:30.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:30.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 10 monclient: get_monmap_and_config success 2024-09-16T14:59:30.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 4 set_mon_vals no callback set 2024-09-16T14:59:30.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 1 -- 172.21.15.26:0/2419139711 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8bec001cd0 con 0x7f8bfc09ff50 2024-09-16T14:59:30.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:30.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8bfa7fc640 10 monclient: dump: 2024-09-16T14:59:30.575 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.576 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 10 monclient: shutdown 2024-09-16T14:59:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 1 -- 172.21.15.26:0/2419139711 >> v1:172.21.15.26:6789/0 conn(0x7f8bfc09ff50 legacy=0x7f8bfc13d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 1 -- 172.21.15.26:0/2419139711 shutdown_connections 2024-09-16T14:59:30.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 1 -- 172.21.15.26:0/2419139711 >> 172.21.15.26:0/2419139711 conn(0x7f8bfc0a3da0 msgr2=0x7f8bfc0a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:30.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 1 -- 172.21.15.26:0/2419139711 shutdown_connections 2024-09-16T14:59:30.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.558+0000 7f8c01a7d640 1 -- 172.21.15.26:0/2419139711 wait complete. 2024-09-16T14:59:30.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 10 monclient: build_initial_monmap 2024-09-16T14:59:30.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 10 monclient: monmap: 2024-09-16T14:59:30.577 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:30.577 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:59:30.578 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:59:30.578 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:59:30.578 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:30.578 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:59:30.578 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:30.578 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:59:30.578 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:59:30.578 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 1 Processor -- start 2024-09-16T14:59:30.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 1 -- start start 2024-09-16T14:59:30.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 10 monclient: init 2024-09-16T14:59:30.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:30.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:59:30.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 10 monclient(hunting): picked mon.noname-b con 0x7f8bfc09ff50 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:30.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 10 monclient(hunting): picked mon.noname-a con 0x7f8bfc0a1ca0 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:30.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 10 monclient(hunting): picked mon.noname-c con 0x7f8bfc0a2790 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:30.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc135380 con 0x7f8bfc0a1ca0 2024-09-16T14:59:30.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc09e490 con 0x7f8bfc09ff50 2024-09-16T14:59:30.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc0a5680 con 0x7f8bfc0a2790 2024-09-16T14:59:30.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:30.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 10 monclient(hunting): authenticate will time out at 1994.684974s 2024-09-16T14:59:30.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bfaffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8bfc0a1ca0 0x7f8bfc1ce3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34694/0 (socket says 172.21.15.26:34694) 2024-09-16T14:59:30.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bfaffd640 1 -- 172.21.15.26:0/925296456 learned_addr learned my addr 172.21.15.26:0/925296456 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:30.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876904958 0 0) 0x7f8bfc0a5680 con 0x7f8bfc0a2790 2024-09-16T14:59:30.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): my global_id is 284570 2024-09-16T14:59:30.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bdc003530 con 0x7f8bfc0a2790 2024-09-16T14:59:30.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250746096 0 0) 0x7f8bfc135380 con 0x7f8bfc0a1ca0 2024-09-16T14:59:30.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): my global_id is 324141 2024-09-16T14:59:30.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bfc0a5680 con 0x7f8bfc0a1ca0 2024-09-16T14:59:30.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3517345064 0 0) 0x7f8bdc003530 con 0x7f8bfc0a2790 2024-09-16T14:59:30.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bfc135380 con 0x7f8bfc0a2790 2024-09-16T14:59:30.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8bec001c10 con 0x7f8bfc0a2790 2024-09-16T14:59:30.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:59:30.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): dump: 2024-09-16T14:59:30.581 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.581 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.581 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.581 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.581 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.581 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.581 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.581 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.581 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.582 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:59:30.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1501569693 0 0) 0x7f8bfc0a5680 con 0x7f8bfc0a1ca0 2024-09-16T14:59:30.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bdc003530 con 0x7f8bfc0a1ca0 2024-09-16T14:59:30.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8be4001890 con 0x7f8bfc0a1ca0 2024-09-16T14:59:30.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:30.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): dump: 2024-09-16T14:59:30.582 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.582 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.582 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.582 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.582 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.583 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526918018 0 0) 0x7f8bfc09e490 con 0x7f8bfc09ff50 2024-09-16T14:59:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): my global_id is 304315 2024-09-16T14:59:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bfc0a5680 con 0x7f8bfc09ff50 2024-09-16T14:59:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2010116198 0 0) 0x7f8bfc135380 con 0x7f8bfc0a2790 2024-09-16T14:59:30.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 >> v1:172.21.15.189:6789/0 conn(0x7f8bfc09ff50 legacy=0x7f8bfc13d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 >> v1:172.21.15.26:6789/0 conn(0x7f8bfc0a1ca0 legacy=0x7f8bfc1ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient: _finish_hunting 0 2024-09-16T14:59:30.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 monclient: found mon.c 2024-09-16T14:59:30.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:30.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:30.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc0a5130 con 0x7f8bfc0a2790 2024-09-16T14:59:30.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient: _finish_auth 0 2024-09-16T14:59:30.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient: _check_auth_tickets 2024-09-16T14:59:30.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:30.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 5 monclient: authenticate success, global_id 284570 2024-09-16T14:59:30.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 10 monclient: _renew_subs 2024-09-16T14:59:30.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:30.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 1 -- 172.21.15.26:0/925296456 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bfc136cf0 con 0x7f8bfc0a2790 2024-09-16T14:59:30.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 10 monclient: _renew_subs 2024-09-16T14:59:30.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:30.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 1 -- 172.21.15.26:0/925296456 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bfc134980 con 0x7f8bfc0a2790 2024-09-16T14:59:30.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bec001110 con 0x7f8bfc0a2790 2024-09-16T14:59:30.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:30.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8bec004da0 con 0x7f8bfc0a2790 2024-09-16T14:59:30.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:30.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf9ffb640 4 set_mon_vals no callback set 2024-09-16T14:59:30.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient: dump: 2024-09-16T14:59:30.585 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.585 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.585 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.586 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.586 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.586 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.586 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.586 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.586 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.586 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:30.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8c01a7d640 1 -- 172.21.15.26:0/925296456 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8be0006590 con 0x7f8be0002830 2024-09-16T14:59:30.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890935049 0 0) 0x7f8be0006590 con 0x7f8be0002830 2024-09-16T14:59:30.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient(hunting): my global_id is 324144 2024-09-16T14:59:30.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bfc135380 con 0x7f8be0002830 2024-09-16T14:59:30.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8bec00f810 con 0x7f8bfc0a2790 2024-09-16T14:59:30.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f8bec043e00 con 0x7f8bfc0a2790 2024-09-16T14:59:30.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2793698195 0 0) 0x7f8bfc135380 con 0x7f8be0002830 2024-09-16T14:59:30.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8be0006590 con 0x7f8be0002830 2024-09-16T14:59:30.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8bf4001ad0 con 0x7f8be0002830 2024-09-16T14:59:30.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:30.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient: dump: 2024-09-16T14:59:30.587 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.587 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.587 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.588 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.588 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.588 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.588 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.588 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.588 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.588 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2808206387 0 0) 0x7f8be0006590 con 0x7f8be0002830 2024-09-16T14:59:30.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8be000a0e0 con 0x7f8be0002830 2024-09-16T14:59:30.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bfaffd640 10 monclient: get_auth_request con 0x7f8bdc0363c0 auth_method 0 2024-09-16T14:59:30.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8bf4001e90 con 0x7f8be0002830 2024-09-16T14:59:30.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:30.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:59:30.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 >> v1:172.21.15.26:6789/0 conn(0x7f8be0002830 legacy=0x7f8be0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.562+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 >> v1:172.21.15.26:6789/0 conn(0x7f8be0002830 legacy=0x7f8be0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:30.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8c01a7d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:59:30.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8c01a7d640 1 -- 172.21.15.26:0/925296456 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8be0009d80 con 0x7f8be00090a0 2024-09-16T14:59:30.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146873165 0 0) 0x7f8be0009d80 con 0x7f8be00090a0 2024-09-16T14:59:30.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 10 monclient(hunting): my global_id is 324147 2024-09-16T14:59:30.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8be0006590 con 0x7f8be00090a0 2024-09-16T14:59:30.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 273745082 0 0) 0x7f8be0006590 con 0x7f8be00090a0 2024-09-16T14:59:30.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8be0009d80 con 0x7f8be00090a0 2024-09-16T14:59:30.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8bf4003be0 con 0x7f8be00090a0 2024-09-16T14:59:30.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:30.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 10 monclient: dump: 2024-09-16T14:59:30.591 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.591 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.591 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.591 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.591 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.591 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.591 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.592 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.592 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.592 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2485775663 0 0) 0x7f8be0009d80 con 0x7f8be00090a0 2024-09-16T14:59:30.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8be000c290 con 0x7f8be00090a0 2024-09-16T14:59:30.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 2324708657) 0x7f8bf40049d0 con 0x7f8be00090a0 2024-09-16T14:59:30.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:59:30.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:59:30.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 >> v1:172.21.15.26:6789/0 conn(0x7f8be00090a0 legacy=0x7f8be0009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bf3fff640 1 -- 172.21.15.26:0/925296456 >> v1:172.21.15.26:6789/0 conn(0x7f8be00090a0 legacy=0x7f8be0009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:30.593 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:59:30.593 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:59:30.593 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:59:30.593 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:59:30.593 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 156, 2024-09-16T14:59:30.593 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:59:30.593 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:59:30.593 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:59:30.593 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:59:30.593 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:30.594 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:59:30.594 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:59:30.594 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:59:30.594 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:59:30.594 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:30.594 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:30.594 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:30.594 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:30.594 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:30.594 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:30.594 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:30.594 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:30.594 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:30.594 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:30.595 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:30.595 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:59:30.595 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:59:30.595 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:30.595 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:30.595 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:30.595 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:30.595 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:30.595 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:30.595 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:30.595 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:30.595 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:30.595 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:30.596 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.596 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:59:30.596 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:59:30.596 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:59:30.596 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:59:30.596 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:59:30.596 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:59:30.596 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:59:30.596 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:59:30.596 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:59:30.596 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:59:30.596 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:59:30.597 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:59:30.597 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:59:30.597 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:59:30.597 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:59:30.597 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:59:30.597 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:59:30.597 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:30.597 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:30.597 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:30.597 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:30.597 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:30.597 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:30.597 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:30.598 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:30.598 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:30.598 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:30.598 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:30.598 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:59:30.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.598 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:59:30.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.598 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:59:30.598 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:59:30.598 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:30.598 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:30.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.599 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:30.599 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:59:30.599 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:30.599 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.599 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.599 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:59:30.599 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:59:30.599 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:30.599 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:30.599 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:30.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.600 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:59:30.600 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:59:30.600 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:30.600 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:30.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.600 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:30.600 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:59:30.600 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:30.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.600 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.600 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:59:30.600 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:59:30.600 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:30.601 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:30.601 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:30.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.601 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:59:30.601 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:59:30.601 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:30.601 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:30.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.601 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:30.601 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:59:30.601 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:30.601 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.602 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.602 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:59:30.602 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:59:30.602 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:30.602 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:30.602 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:30.602 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.602 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.603 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:59:30.603 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:59:30.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.603 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:30.603 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:30.603 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:59:30.603 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.603 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:30.603 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:59:30.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.603 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:30.603 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:30.603 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:59:30.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.604 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.604 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:59:30.604 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:59:30.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8c01a7d640 1 -- 172.21.15.26:0/925296456 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8bdc0363c0 legacy=0x7f8bdc038880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8c01a7d640 10 monclient: shutdown 2024-09-16T14:59:30.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8c01a7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:30.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8c01a7d640 1 -- 172.21.15.26:0/925296456 >> v1:172.21.15.26:6790/0 conn(0x7f8bfc0a2790 legacy=0x7f8bfc1d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bfbfff640 1 -- 172.21.15.26:0/925296456 reap_dead start 2024-09-16T14:59:30.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8bfbfff640 1 -- 172.21.15.26:0/925296456 reap_dead start 2024-09-16T14:59:30.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8c01a7d640 1 -- 172.21.15.26:0/925296456 shutdown_connections 2024-09-16T14:59:30.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8c01a7d640 1 -- 172.21.15.26:0/925296456 >> 172.21.15.26:0/925296456 conn(0x7f8bfc0a3da0 msgr2=0x7f8bfc13e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:30.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8c01a7d640 1 -- 172.21.15.26:0/925296456 shutdown_connections 2024-09-16T14:59:30.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.570+0000 7f8c01a7d640 1 -- 172.21.15.26:0/925296456 wait complete. 2024-09-16T14:59:30.606 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:59:30.606 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T14:59:30.607 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:59:30.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 10 monclient: get_monmap_and_config 2024-09-16T14:59:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 10 monclient: build_initial_monmap 2024-09-16T14:59:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 10 monclient: monmap: 2024-09-16T14:59:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:59:30.722145+0000 2024-09-16T14:59:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:59:30.722145+0000 2024-09-16T14:59:30.720 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:59:30.721 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:59:30.721 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 1 Processor -- start 2024-09-16T14:59:30.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 1 -- start start 2024-09-16T14:59:30.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 10 monclient: init 2024-09-16T14:59:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:59:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 10 monclient(hunting): picked mon.noname-c con 0x7ff5a409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 10 monclient(hunting): picked mon.noname-a con 0x7ff5a4132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 10 monclient(hunting): picked mon.noname-b con 0x7ff5a4138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a409d260 con 0x7ff5a4132650 2024-09-16T14:59:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a4132a80 con 0x7ff5a409cb80 2024-09-16T14:59:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a409cf60 con 0x7ff5a4138ff0 2024-09-16T14:59:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 10 monclient(hunting): authenticate will time out at 1994.842819s 2024-09-16T14:59:30.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59bfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff5a4132650 0x7ff5a41378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34728/0 (socket says 172.21.15.26:34728) 2024-09-16T14:59:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59bfff640 1 -- 172.21.15.26:0/2733961244 learned_addr learned my addr 172.21.15.26:0/2733961244 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5a944e640 1 --1- 172.21.15.26:0/2733961244 >> v1:172.21.15.26:6790/0 conn(0x7ff5a4138ff0 0x7ff5a413b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48174/0 (socket says 172.21.15.26:48174) 2024-09-16T14:59:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 1 -- 172.21.15.26:0/2733961244 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305107105 0 0) 0x7ff5a409d260 con 0x7ff5a4132650 2024-09-16T14:59:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient(hunting): my global_id is 324150 2024-09-16T14:59:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 1 -- 172.21.15.26:0/2733961244 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff58c003530 con 0x7ff5a4132650 2024-09-16T14:59:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 1 -- 172.21.15.26:0/2733961244 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161380201 0 0) 0x7ff5a409cf60 con 0x7ff5a4138ff0 2024-09-16T14:59:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient(hunting): my global_id is 284573 2024-09-16T14:59:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 1 -- 172.21.15.26:0/2733961244 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a409d260 con 0x7ff5a4138ff0 2024-09-16T14:59:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 1 -- 172.21.15.26:0/2733961244 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642825846 0 0) 0x7ff5a4132a80 con 0x7ff5a409cb80 2024-09-16T14:59:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient(hunting): my global_id is 304318 2024-09-16T14:59:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 1 -- 172.21.15.26:0/2733961244 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a409cf60 con 0x7ff5a409cb80 2024-09-16T14:59:30.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 1 -- 172.21.15.26:0/2733961244 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 776236037 0 0) 0x7ff58c003530 con 0x7ff5a4132650 2024-09-16T14:59:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 1 -- 172.21.15.26:0/2733961244 >> v1:172.21.15.26:6790/0 conn(0x7ff5a4138ff0 legacy=0x7ff5a413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 1 -- 172.21.15.26:0/2733961244 >> v1:172.21.15.189:6789/0 conn(0x7ff5a409cb80 legacy=0x7ff5a40717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient: _finish_hunting 0 2024-09-16T14:59:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 1 monclient: found mon.noname-a 2024-09-16T14:59:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T14:59:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 1 -- 172.21.15.26:0/2733961244 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a409f2d0 con 0x7ff5a4132650 2024-09-16T14:59:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient: _finish_auth 0 2024-09-16T14:59:30.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient: _check_auth_tickets 2024-09-16T14:59:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 1 -- 172.21.15.26:0/2733961244 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff594001620 con 0x7ff5a4132650 2024-09-16T14:59:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:59:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient: dump: 2024-09-16T14:59:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.735 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.736 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.736 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.736 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.736 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.736 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.736 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 5 monclient: authenticate success, global_id 324150 2024-09-16T14:59:30.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:30.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 1 -- 172.21.15.26:0/2733961244 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff594001890 con 0x7ff5a4132650 2024-09-16T14:59:30.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:30.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 1 -- 172.21.15.26:0/2733961244 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff594001c60 con 0x7ff5a4132650 2024-09-16T14:59:30.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 10 monclient: get_monmap_and_config success 2024-09-16T14:59:30.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff5aaeb1640 4 set_mon_vals no callback set 2024-09-16T14:59:30.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:30.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.718+0000 7ff59b7fe640 10 monclient: dump: 2024-09-16T14:59:30.737 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.737 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.737 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.737 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.737 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.738 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.738 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.738 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.738 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.738 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 10 monclient: shutdown 2024-09-16T14:59:30.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:30.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 1 -- 172.21.15.26:0/2733961244 >> v1:172.21.15.26:6789/0 conn(0x7ff5a4132650 legacy=0x7ff5a41378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 1 -- 172.21.15.26:0/2733961244 shutdown_connections 2024-09-16T14:59:30.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 1 -- 172.21.15.26:0/2733961244 >> 172.21.15.26:0/2733961244 conn(0x7ff5a409fb50 msgr2=0x7ff5a409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:30.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 1 -- 172.21.15.26:0/2733961244 shutdown_connections 2024-09-16T14:59:30.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 1 -- 172.21.15.26:0/2733961244 wait complete. 2024-09-16T14:59:30.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 10 monclient: build_initial_monmap 2024-09-16T14:59:30.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 10 monclient: monmap: 2024-09-16T14:59:30.739 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:30.739 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:59:30.739 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:59:30.739 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:59:30.739 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:30.739 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:59:30.740 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:30.740 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:59:30.740 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:59:30.740 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 1 Processor -- start 2024-09-16T14:59:30.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 1 -- start start 2024-09-16T14:59:30.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 10 monclient: init 2024-09-16T14:59:30.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:30.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:59:30.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 10 monclient(hunting): picked mon.noname-c con 0x7ff5a409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:30.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 10 monclient(hunting): picked mon.noname-b con 0x7ff5a4132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:30.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 10 monclient(hunting): picked mon.noname-a con 0x7ff5a4138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:30.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a409b680 con 0x7ff5a4138ff0 2024-09-16T14:59:30.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a409a240 con 0x7ff5a4132650 2024-09-16T14:59:30.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a40719c0 con 0x7ff5a409cb80 2024-09-16T14:59:30.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:30.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 10 monclient(hunting): authenticate will time out at 1994.846076s 2024-09-16T14:59:30.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5a944e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff5a4138ff0 0x7ff5a41d1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34742/0 (socket says 172.21.15.26:34742) 2024-09-16T14:59:30.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5a944e640 1 -- 172.21.15.26:0/2786097808 learned_addr learned my addr 172.21.15.26:0/2786097808 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:30.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366844272 0 0) 0x7ff5a40719c0 con 0x7ff5a409cb80 2024-09-16T14:59:30.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): my global_id is 284576 2024-09-16T14:59:30.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff590003530 con 0x7ff5a409cb80 2024-09-16T14:59:30.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2759541730 0 0) 0x7ff5a409b680 con 0x7ff5a4138ff0 2024-09-16T14:59:30.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): my global_id is 324153 2024-09-16T14:59:30.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a40719c0 con 0x7ff5a4138ff0 2024-09-16T14:59:30.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2324784879 0 0) 0x7ff590003530 con 0x7ff5a409cb80 2024-09-16T14:59:30.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5a409b680 con 0x7ff5a409cb80 2024-09-16T14:59:30.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff5a0001840 con 0x7ff5a409cb80 2024-09-16T14:59:30.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:59:30.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): dump: 2024-09-16T14:59:30.743 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.744 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.744 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.744 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.744 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.744 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.744 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.744 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.744 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.745 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:59:30.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2804686560 0 0) 0x7ff5a40719c0 con 0x7ff5a4138ff0 2024-09-16T14:59:30.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff590003530 con 0x7ff5a4138ff0 2024-09-16T14:59:30.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff59c001cd0 con 0x7ff5a4138ff0 2024-09-16T14:59:30.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:30.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): dump: 2024-09-16T14:59:30.745 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.745 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.745 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.746 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.746 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.746 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.746 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.746 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.746 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.746 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959231508 0 0) 0x7ff5a409a240 con 0x7ff5a4132650 2024-09-16T14:59:30.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient(hunting): my global_id is 304321 2024-09-16T14:59:30.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a40719c0 con 0x7ff5a4132650 2024-09-16T14:59:30.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3741278861 0 0) 0x7ff590003530 con 0x7ff5a4138ff0 2024-09-16T14:59:30.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 >> v1:172.21.15.26:6790/0 conn(0x7ff5a409cb80 legacy=0x7ff5a409a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 >> v1:172.21.15.189:6789/0 conn(0x7ff5a4132650 legacy=0x7ff5a409ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient: _finish_hunting 0 2024-09-16T14:59:30.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 monclient: found mon.a 2024-09-16T14:59:30.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:30.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:30.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a409b370 con 0x7ff5a4138ff0 2024-09-16T14:59:30.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient: _finish_auth 0 2024-09-16T14:59:30.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient: _check_auth_tickets 2024-09-16T14:59:30.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:30.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 5 monclient: authenticate success, global_id 324153 2024-09-16T14:59:30.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 10 monclient: _renew_subs 2024-09-16T14:59:30.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:30.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 1 -- 172.21.15.26:0/2786097808 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a40a2940 con 0x7ff5a4138ff0 2024-09-16T14:59:30.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 10 monclient: _renew_subs 2024-09-16T14:59:30.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:30.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 1 -- 172.21.15.26:0/2786097808 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a406d570 con 0x7ff5a4138ff0 2024-09-16T14:59:30.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff59c004b70 con 0x7ff5a4138ff0 2024-09-16T14:59:30.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:30.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff59c004e80 con 0x7ff5a4138ff0 2024-09-16T14:59:30.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:30.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5997fa640 10 monclient: dump: 2024-09-16T14:59:30.749 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.749 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.749 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.749 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.749 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.749 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.749 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.749 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.750 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.750 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff59affd640 4 set_mon_vals no callback set 2024-09-16T14:59:30.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.722+0000 7ff5aaeb1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:30.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5aaeb1640 1 -- 172.21.15.26:0/2786097808 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff584006590 con 0x7ff584002830 2024-09-16T14:59:30.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff59c00f8f0 con 0x7ff5a4138ff0 2024-09-16T14:59:30.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7ff59c044010 con 0x7ff5a4138ff0 2024-09-16T14:59:30.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017768614 0 0) 0x7ff584006590 con 0x7ff584002830 2024-09-16T14:59:30.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 10 monclient(hunting): my global_id is 284579 2024-09-16T14:59:30.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff590003530 con 0x7ff584002830 2024-09-16T14:59:30.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 961291751 0 0) 0x7ff590003530 con 0x7ff584002830 2024-09-16T14:59:30.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff584006590 con 0x7ff584002830 2024-09-16T14:59:30.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff5a00039b0 con 0x7ff584002830 2024-09-16T14:59:30.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff59bfff640 10 monclient: get_auth_request con 0x7ff5900362c0 auth_method 0 2024-09-16T14:59:30.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:30.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 10 monclient: dump: 2024-09-16T14:59:30.752 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.752 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.752 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.752 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.752 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.753 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.753 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.753 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.753 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.754 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2205448485 0 0) 0x7ff584006590 con 0x7ff584002830 2024-09-16T14:59:30.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff58400a0e0 con 0x7ff584002830 2024-09-16T14:59:30.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ff5a0003dc0 con 0x7ff584002830 2024-09-16T14:59:30.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:30.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:59:30.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 >> v1:172.21.15.26:6790/0 conn(0x7ff584002830 legacy=0x7ff584004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.726+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 >> v1:172.21.15.26:6790/0 conn(0x7ff584002830 legacy=0x7ff584004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:30.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.730+0000 7ff5aaeb1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:59:30.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.730+0000 7ff5aaeb1640 1 -- 172.21.15.26:0/2786097808 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff584009d80 con 0x7ff5840090a0 2024-09-16T14:59:30.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.730+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957174300 0 0) 0x7ff584009d80 con 0x7ff5840090a0 2024-09-16T14:59:30.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.730+0000 7ff5997fa640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.730+0000 7ff5997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.730+0000 7ff5997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.730+0000 7ff5997fa640 10 monclient(hunting): my global_id is 284582 2024-09-16T14:59:30.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.730+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff584006590 con 0x7ff5840090a0 2024-09-16T14:59:30.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2581226816 0 0) 0x7ff584006590 con 0x7ff5840090a0 2024-09-16T14:59:30.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff584009d80 con 0x7ff5840090a0 2024-09-16T14:59:30.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff5a0004340 con 0x7ff5840090a0 2024-09-16T14:59:30.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:30.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5997fa640 10 monclient: dump: 2024-09-16T14:59:30.762 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.762 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.763 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.763 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.763 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.763 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.763 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.763 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.763 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.763 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2426383271 0 0) 0x7ff584009d80 con 0x7ff5840090a0 2024-09-16T14:59:30.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff58400c290 con 0x7ff5840090a0 2024-09-16T14:59:30.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 1785722025) 0x7ff5a00057e0 con 0x7ff5840090a0 2024-09-16T14:59:30.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5997fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:59:30.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5997fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:59:30.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 >> v1:172.21.15.26:6790/0 conn(0x7ff5840090a0 legacy=0x7ff584009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5997fa640 1 -- 172.21.15.26:0/2786097808 >> v1:172.21.15.26:6790/0 conn(0x7ff5840090a0 legacy=0x7ff584009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:30.765 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:59:30.765 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:59:30.765 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:59:30.765 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:59:30.765 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 156, 2024-09-16T14:59:30.765 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:59:30.766 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:59:30.766 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:59:30.766 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:59:30.766 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:30.766 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T14:59:30.766 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:59:30.766 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:59:30.766 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:59:30.766 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:30.766 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:30.766 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:30.766 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:30.766 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:30.767 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:30.767 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:30.767 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:30.767 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:30.767 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:30.767 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:30.767 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:59:30.767 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:59:30.767 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:30.767 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:30.767 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:30.767 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:30.768 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:30.768 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:30.768 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:30.768 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:30.768 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:30.768 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:30.768 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.768 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:59:30.768 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:59:30.768 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:59:30.768 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:59:30.768 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:59:30.769 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:59:30.769 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:59:30.769 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:59:30.769 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:59:30.769 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:59:30.769 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:59:30.769 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:59:30.769 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:59:30.769 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:59:30.769 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:59:30.769 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:59:30.769 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:59:30.769 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:30.770 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:30.770 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:30.770 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:30.770 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:30.770 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:30.770 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:30.770 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:30.770 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:30.770 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:30.770 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:30.770 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:59:30.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.770 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:59:30.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.771 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:59:30.771 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:59:30.771 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:30.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:30.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.771 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:30.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:59:30.771 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:30.771 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.771 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:59:30.771 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:59:30.772 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:30.772 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:30.772 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:30.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.772 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:59:30.772 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:59:30.772 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:30.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:30.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.772 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:30.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:59:30.772 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:30.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.773 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:59:30.773 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:59:30.773 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:30.773 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:30.773 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:30.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.773 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:59:30.773 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:59:30.773 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:30.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:30.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.774 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:30.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:59:30.774 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:30.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.774 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:59:30.774 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:59:30.774 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:30.774 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:30.774 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:30.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.774 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.775 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:59:30.775 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:59:30.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.775 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:30.775 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:30.775 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:59:30.775 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.775 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:30.775 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:59:30.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.775 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:30.775 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:30.776 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:59:30.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.776 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:30.776 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:59:30.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.776 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:30.776 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:30.776 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T14:59:30.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.776 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:30.776 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:59:30.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.776 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:30.776 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:30.777 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:59:30.777 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.777 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.777 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:59:30.777 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:59:30.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5aaeb1640 1 -- 172.21.15.26:0/2786097808 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff5900362c0 legacy=0x7ff590038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5aaeb1640 10 monclient: shutdown 2024-09-16T14:59:30.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5aaeb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:30.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5aaeb1640 1 -- 172.21.15.26:0/2786097808 >> v1:172.21.15.26:6789/0 conn(0x7ff5a4138ff0 legacy=0x7ff5a41d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5a944e640 1 -- 172.21.15.26:0/2786097808 reap_dead start 2024-09-16T14:59:30.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5a944e640 1 -- 172.21.15.26:0/2786097808 reap_dead start 2024-09-16T14:59:30.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5aaeb1640 1 -- 172.21.15.26:0/2786097808 shutdown_connections 2024-09-16T14:59:30.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5aaeb1640 1 -- 172.21.15.26:0/2786097808 >> 172.21.15.26:0/2786097808 conn(0x7ff5a409fb50 msgr2=0x7ff5a40a1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:30.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5aaeb1640 1 -- 172.21.15.26:0/2786097808 shutdown_connections 2024-09-16T14:59:30.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.734+0000 7ff5aaeb1640 1 -- 172.21.15.26:0/2786097808 wait complete. 2024-09-16T14:59:30.780 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:59:30.780 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T14:59:30.780 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:59:30.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 10 monclient: get_monmap_and_config 2024-09-16T14:59:30.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 10 monclient: build_initial_monmap 2024-09-16T14:59:30.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 10 monclient: monmap: 2024-09-16T14:59:30.904 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:30.904 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.904 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:59:30.906306+0000 2024-09-16T14:59:30.904 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:59:30.906306+0000 2024-09-16T14:59:30.905 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:30.905 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.905 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:30.905 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:59:30.905 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:59:30.905 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 1 Processor -- start 2024-09-16T14:59:30.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 1 -- start start 2024-09-16T14:59:30.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 10 monclient: init 2024-09-16T14:59:30.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:30.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:59:30.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 10 monclient(hunting): picked mon.noname-b con 0x7fe2fc0a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:30.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 10 monclient(hunting): picked mon.noname-a con 0x7fe2fc0a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:30.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 10 monclient(hunting): picked mon.noname-c con 0x7fe2fc0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:30.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2fc0a3250 con 0x7fe2fc0a0000 2024-09-16T14:59:30.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2fc09f600 con 0x7fe2fc0a2800 2024-09-16T14:59:30.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2fc0a3830 con 0x7fe2fc0a1d10 2024-09-16T14:59:30.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:30.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 10 monclient(hunting): authenticate will time out at 1995.026560s 2024-09-16T14:59:30.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2fbfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe2fc0a1d10 0x7fe2fc0a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48232/0 (socket says 172.21.15.26:48232) 2024-09-16T14:59:30.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2fbfff640 1 -- 172.21.15.26:0/1419836449 learned_addr learned my addr 172.21.15.26:0/1419836449 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:30.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2fb7fe640 1 --1- 172.21.15.26:0/1419836449 >> v1:172.21.15.26:6789/0 conn(0x7fe2fc0a0000 0x7fe2fc139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34750/0 (socket says 172.21.15.26:34750) 2024-09-16T14:59:30.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 1 -- 172.21.15.26:0/1419836449 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72911355 0 0) 0x7fe2fc0a3830 con 0x7fe2fc0a1d10 2024-09-16T14:59:30.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient(hunting): my global_id is 284585 2024-09-16T14:59:30.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 1 -- 172.21.15.26:0/1419836449 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2e8003530 con 0x7fe2fc0a1d10 2024-09-16T14:59:30.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 1 -- 172.21.15.26:0/1419836449 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153419440 0 0) 0x7fe2fc0a3250 con 0x7fe2fc0a0000 2024-09-16T14:59:30.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient(hunting): my global_id is 324156 2024-09-16T14:59:30.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 1 -- 172.21.15.26:0/1419836449 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2fc0a3830 con 0x7fe2fc0a0000 2024-09-16T14:59:30.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 1 -- 172.21.15.26:0/1419836449 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1418968717 0 0) 0x7fe2e8003530 con 0x7fe2fc0a1d10 2024-09-16T14:59:30.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 1 -- 172.21.15.26:0/1419836449 >> v1:172.21.15.189:6789/0 conn(0x7fe2fc0a2800 legacy=0x7fe2fc13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 1 -- 172.21.15.26:0/1419836449 >> v1:172.21.15.26:6789/0 conn(0x7fe2fc0a0000 legacy=0x7fe2fc139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient: _finish_hunting 0 2024-09-16T14:59:30.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 1 monclient: found mon.noname-b 2024-09-16T14:59:30.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:30.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:59:30.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 1 -- 172.21.15.26:0/1419836449 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc09ed60 con 0x7fe2fc0a1d10 2024-09-16T14:59:30.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient: _finish_auth 0 2024-09-16T14:59:30.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient: _check_auth_tickets 2024-09-16T14:59:30.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:30.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 1 -- 172.21.15.26:0/1419836449 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2e4001620 con 0x7fe2fc0a1d10 2024-09-16T14:59:30.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:59:30.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient: dump: 2024-09-16T14:59:30.915 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.915 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.915 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.915 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.915 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.915 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.915 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.915 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.915 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.915 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 5 monclient: authenticate success, global_id 284585 2024-09-16T14:59:30.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:30.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 1 -- 172.21.15.26:0/1419836449 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2e40018c0 con 0x7fe2fc0a1d10 2024-09-16T14:59:30.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:30.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 1 -- 172.21.15.26:0/1419836449 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2e4001c90 con 0x7fe2fc0a1d10 2024-09-16T14:59:30.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 10 monclient: get_monmap_and_config success 2024-09-16T14:59:30.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe302440640 4 set_mon_vals no callback set 2024-09-16T14:59:30.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:30.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.902+0000 7fe2faffd640 10 monclient: dump: 2024-09-16T14:59:30.916 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.916 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.916 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.917 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.917 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.917 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.917 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.917 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.917 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.917 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 10 monclient: shutdown 2024-09-16T14:59:30.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:30.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 1 -- 172.21.15.26:0/1419836449 >> v1:172.21.15.26:6790/0 conn(0x7fe2fc0a1d10 legacy=0x7fe2fc0a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 1 -- 172.21.15.26:0/1419836449 shutdown_connections 2024-09-16T14:59:30.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 1 -- 172.21.15.26:0/1419836449 >> 172.21.15.26:0/1419836449 conn(0x7fe2fc0a7f30 msgr2=0x7fe2fc0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:30.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 1 -- 172.21.15.26:0/1419836449 shutdown_connections 2024-09-16T14:59:30.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 1 -- 172.21.15.26:0/1419836449 wait complete. 2024-09-16T14:59:30.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 10 monclient: build_initial_monmap 2024-09-16T14:59:30.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 10 monclient: monmap: 2024-09-16T14:59:30.918 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:30.918 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:59:30.918 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:59:30.918 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:59:30.918 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:30.918 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:59:30.918 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:30.918 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:59:30.918 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:59:30.918 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 1 Processor -- start 2024-09-16T14:59:30.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 1 -- start start 2024-09-16T14:59:30.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 10 monclient: init 2024-09-16T14:59:30.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:30.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:59:30.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 10 monclient(hunting): picked mon.noname-a con 0x7fe2fc0a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:30.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 10 monclient(hunting): picked mon.noname-c con 0x7fe2fc0a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:30.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 10 monclient(hunting): picked mon.noname-b con 0x7fe2fc0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:30.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2fc09e500 con 0x7fe2fc0a0000 2024-09-16T14:59:30.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2fc1341d0 con 0x7fe2fc0a2800 2024-09-16T14:59:30.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2fc1348e0 con 0x7fe2fc0a1d10 2024-09-16T14:59:30.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 10 monclient(hunting): authenticate will time out at 1995.029621s 2024-09-16T14:59:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2fbfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe2fc0a0000 0x7fe2fc13d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34760/0 (socket says 172.21.15.26:34760) 2024-09-16T14:59:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2fb7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe2fc0a1d10 0x7fe2fc1c8e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48246/0 (socket says 172.21.15.26:48246) 2024-09-16T14:59:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2fbfff640 1 -- 172.21.15.26:0/4149845410 learned_addr learned my addr 172.21.15.26:0/4149845410 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60380024 0 0) 0x7fe2fc1348e0 con 0x7fe2fc0a1d10 2024-09-16T14:59:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): my global_id is 284588 2024-09-16T14:59:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2dc003530 con 0x7fe2fc0a1d10 2024-09-16T14:59:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057687832 0 0) 0x7fe2fc09e500 con 0x7fe2fc0a0000 2024-09-16T14:59:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): my global_id is 324159 2024-09-16T14:59:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2fc1348e0 con 0x7fe2fc0a0000 2024-09-16T14:59:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785885336 0 0) 0x7fe2fc1341d0 con 0x7fe2fc0a2800 2024-09-16T14:59:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): my global_id is 304327 2024-09-16T14:59:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2fc09e500 con 0x7fe2fc0a2800 2024-09-16T14:59:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 268418096 0 0) 0x7fe2fc1348e0 con 0x7fe2fc0a0000 2024-09-16T14:59:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2fc1341d0 con 0x7fe2fc0a0000 2024-09-16T14:59:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1023717830 0 0) 0x7fe2dc003530 con 0x7fe2fc0a1d10 2024-09-16T14:59:30.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2fc1348e0 con 0x7fe2fc0a1d10 2024-09-16T14:59:30.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2e4001bd0 con 0x7fe2fc0a0000 2024-09-16T14:59:30.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T14:59:30.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): dump: 2024-09-16T14:59:30.922 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.922 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.922 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.922 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.922 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.922 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.922 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.923 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.923 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.923 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:59:30.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2ec001be0 con 0x7fe2fc0a1d10 2024-09-16T14:59:30.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:30.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient(hunting): dump: 2024-09-16T14:59:30.923 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.923 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.923 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.923 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.923 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.924 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.924 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.924 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.924 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.924 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1439141055 0 0) 0x7fe2fc1341d0 con 0x7fe2fc0a0000 2024-09-16T14:59:30.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 >> v1:172.21.15.26:6790/0 conn(0x7fe2fc0a1d10 legacy=0x7fe2fc1c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 >> v1:172.21.15.189:6789/0 conn(0x7fe2fc0a2800 legacy=0x7fe2fc1d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:59:30.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 monclient: found mon.a 2024-09-16T14:59:30.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:30.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:30.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc136650 con 0x7fe2fc0a0000 2024-09-16T14:59:30.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient: _finish_auth 0 2024-09-16T14:59:30.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:59:30.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:30.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 5 monclient: authenticate success, global_id 324159 2024-09-16T14:59:30.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 10 monclient: _renew_subs 2024-09-16T14:59:30.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:30.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 1 -- 172.21.15.26:0/4149845410 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2fc136930 con 0x7fe2fc0a0000 2024-09-16T14:59:30.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 10 monclient: _renew_subs 2024-09-16T14:59:30.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T14:59:30.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 1 -- 172.21.15.26:0/4149845410 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2fc1c95a0 con 0x7fe2fc0a0000 2024-09-16T14:59:30.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2e4004c70 con 0x7fe2fc0a0000 2024-09-16T14:59:30.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:30.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2e4005000 con 0x7fe2fc0a0000 2024-09-16T14:59:30.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T14:59:30.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2fa7fc640 4 set_mon_vals no callback set 2024-09-16T14:59:30.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 10 monclient: dump: 2024-09-16T14:59:30.926 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.926 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.926 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.926 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.926 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.926 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.926 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.927 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.927 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.927 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe2e40051e0 con 0x7fe2fc0a0000 2024-09-16T14:59:30.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:30.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe302440640 1 -- 172.21.15.26:0/4149845410 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe2e0006590 con 0x7fe2e0002830 2024-09-16T14:59:30.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.906+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fe2e4044050 con 0x7fe2fc0a0000 2024-09-16T14:59:30.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe3009dd640 10 monclient: get_auth_request con 0x7fe2dc036370 auth_method 0 2024-09-16T14:59:30.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604052500 0 0) 0x7fe2e0006590 con 0x7fe2e0002830 2024-09-16T14:59:30.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 10 monclient(hunting): my global_id is 304330 2024-09-16T14:59:30.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2fc1341d0 con 0x7fe2e0002830 2024-09-16T14:59:30.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2368978557 0 0) 0x7fe2fc1341d0 con 0x7fe2e0002830 2024-09-16T14:59:30.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2e0006590 con 0x7fe2e0002830 2024-09-16T14:59:30.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2ec004f00 con 0x7fe2e0002830 2024-09-16T14:59:30.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:59:30.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 10 monclient: dump: 2024-09-16T14:59:30.930 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.930 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.930 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.930 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.930 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.930 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.930 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.930 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.930 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.931 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2424482855 0 0) 0x7fe2e0006590 con 0x7fe2e0002830 2024-09-16T14:59:30.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2e000a0e0 con 0x7fe2e0002830 2024-09-16T14:59:30.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe2ec0071b0 con 0x7fe2e0002830 2024-09-16T14:59:30.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:30.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:59:30.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 >> v1:172.21.15.189:6789/0 conn(0x7fe2e0002830 legacy=0x7fe2e0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.910+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 >> v1:172.21.15.189:6789/0 conn(0x7fe2e0002830 legacy=0x7fe2e0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:30.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe302440640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:59:30.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe302440640 1 -- 172.21.15.26:0/4149845410 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe2e0009da0 con 0x7fe2e00090c0 2024-09-16T14:59:30.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746452023 0 0) 0x7fe2e0009da0 con 0x7fe2e00090c0 2024-09-16T14:59:30.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:59:30.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:30.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:30.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 10 monclient(hunting): my global_id is 304333 2024-09-16T14:59:30.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2e0006590 con 0x7fe2e00090c0 2024-09-16T14:59:30.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1318215826 0 0) 0x7fe2e0006590 con 0x7fe2e00090c0 2024-09-16T14:59:30.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2e0009da0 con 0x7fe2e00090c0 2024-09-16T14:59:30.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe2ec005600 con 0x7fe2e00090c0 2024-09-16T14:59:30.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:30.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:59:30.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 10 monclient: dump: 2024-09-16T14:59:30.932 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:30.932 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:30.932 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.932 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:30.932 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:30.933 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:30.933 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:30.933 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:30.933 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:30.933 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:30.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4122918267 0 0) 0x7fe2e0009da0 con 0x7fe2e00090c0 2024-09-16T14:59:30.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe2e000c290 con 0x7fe2e00090c0 2024-09-16T14:59:30.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 2408431193) 0x7fe2ec007320 con 0x7fe2e00090c0 2024-09-16T14:59:30.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:59:30.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:59:30.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 >> v1:172.21.15.189:6789/0 conn(0x7fe2e00090c0 legacy=0x7fe2e0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe2f8ff9640 1 -- 172.21.15.26:0/4149845410 >> v1:172.21.15.189:6789/0 conn(0x7fe2e00090c0 legacy=0x7fe2e0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:30.934 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:59:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:59:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:59:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:59:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 156, 2024-09-16T14:59:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:59:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T14:59:30.934 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:59:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:59:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-09-16T14:59:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:59:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:59:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:59:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:30.935 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:59:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:59:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:30.936 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:59:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:59:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:59:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:59:30.937 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:59:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:59:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:59:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:59:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:59:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:59:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:59:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:59:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:59:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:59:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:59:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:59:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:59:30.938 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:59:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:59:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:59:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:59:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:59:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:59:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:59:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:59:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:59:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:59:30.941 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:59:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:59:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:59:30.942 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:59:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:59:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:59:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:59:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:30.943 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:59:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:59:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:59:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:30.944 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:59:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:59:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:30.945 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:59:30.945 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:59:30.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe302440640 1 -- 172.21.15.26:0/4149845410 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe2dc036370 legacy=0x7fe2dc038830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe302440640 10 monclient: shutdown 2024-09-16T14:59:30.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe302440640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:30.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe302440640 1 -- 172.21.15.26:0/4149845410 >> v1:172.21.15.26:6789/0 conn(0x7fe2fc0a0000 legacy=0x7fe2fc13d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:30.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe3009dd640 1 -- 172.21.15.26:0/4149845410 reap_dead start 2024-09-16T14:59:30.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe3009dd640 1 -- 172.21.15.26:0/4149845410 reap_dead start 2024-09-16T14:59:30.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe302440640 1 -- 172.21.15.26:0/4149845410 shutdown_connections 2024-09-16T14:59:30.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe302440640 1 -- 172.21.15.26:0/4149845410 >> 172.21.15.26:0/4149845410 conn(0x7fe2fc0a7f30 msgr2=0x7fe2fc13dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:30.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe302440640 1 -- 172.21.15.26:0/4149845410 shutdown_connections 2024-09-16T14:59:30.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:30.918+0000 7fe302440640 1 -- 172.21.15.26:0/4149845410 wait complete. 2024-09-16T14:59:30.947 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T14:59:30.948 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T14:59:30.948 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T14:59:30.948 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T14:59:30.948 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T14:59:30.948 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T14:59:30.948 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T14:59:30.959 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T14:59:30.959 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T14:59:30.959 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:59:31.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f098ea56640 10 monclient: get_monmap_and_config 2024-09-16T14:59:31.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f098ea56640 10 monclient: build_initial_monmap 2024-09-16T14:59:31.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f098ea56640 10 monclient: monmap: 2024-09-16T14:59:31.117 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:31.117 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:31.117 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:59:31.119107+0000 2024-09-16T14:59:31.117 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:59:31.119107+0000 2024-09-16T14:59:31.117 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:31.117 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:31.117 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:31.117 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:59:31.117 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:59:31.117 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:31.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f098ea56640 1 Processor -- start 2024-09-16T14:59:31.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f098ea56640 1 -- start start 2024-09-16T14:59:31.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f098ea56640 10 monclient: init 2024-09-16T14:59:31.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f098ea56640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:31.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f098ea56640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:59:31.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f098ea56640 10 monclient(hunting): picked mon.noname-b con 0x7f09880a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:31.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f098ea56640 10 monclient(hunting): picked mon.noname-a con 0x7f09880a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:31.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f098ea56640 10 monclient(hunting): picked mon.noname-c con 0x7f09880a2800 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:31.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f098ea56640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09880a3250 con 0x7f09880a0000 2024-09-16T14:59:31.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f098ea56640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f098809f600 con 0x7f09880a2800 2024-09-16T14:59:31.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f098ea56640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09880a3830 con 0x7f09880a1d10 2024-09-16T14:59:31.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f098ea56640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:31.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f098ea56640 10 monclient(hunting): authenticate will time out at 1995.239236s 2024-09-16T14:59:31.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f09877fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f09880a0000 legacy=0x7f0988139d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:31.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f0987fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f09880a1d10 0x7f09880a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48254/0 (socket says 172.21.15.26:48254) 2024-09-16T14:59:31.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f0987fff640 1 -- 172.21.15.26:0/2733832298 learned_addr learned my addr 172.21.15.26:0/2733832298 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:31.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f0986ffd640 1 -- 172.21.15.26:0/2733832298 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513150410 0 0) 0x7f09880a3830 con 0x7f09880a1d10 2024-09-16T14:59:31.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f0986ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:59:31.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f0986ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:31.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f0986ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:31.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f0986ffd640 10 monclient(hunting): my global_id is 284591 2024-09-16T14:59:31.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.114+0000 7f0986ffd640 1 -- 172.21.15.26:0/2733832298 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0974003530 con 0x7f09880a1d10 2024-09-16T14:59:31.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 1 -- 172.21.15.26:0/2733832298 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216794187 0 0) 0x7f098809f600 con 0x7f09880a2800 2024-09-16T14:59:31.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 10 monclient(hunting): _negotiate 2024-09-16T14:59:31.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:31.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:31.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 10 monclient(hunting): my global_id is 304336 2024-09-16T14:59:31.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 1 -- 172.21.15.26:0/2733832298 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09880a3830 con 0x7f09880a2800 2024-09-16T14:59:31.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 1 -- 172.21.15.26:0/2733832298 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2044161199 0 0) 0x7f0974003530 con 0x7f09880a1d10 2024-09-16T14:59:31.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 1 -- 172.21.15.26:0/2733832298 >> v1:172.21.15.189:6789/0 conn(0x7f09880a2800 legacy=0x7f098813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:31.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 1 -- 172.21.15.26:0/2733832298 >> v1:172.21.15.26:6789/0 conn(0x7f09880a0000 legacy=0x7f0988139d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:59:31.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 10 monclient: _finish_hunting 0 2024-09-16T14:59:31.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 1 monclient: found mon.noname-b 2024-09-16T14:59:31.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:31.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:59:31.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 1 -- 172.21.15.26:0/2733832298 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098809ed60 con 0x7f09880a1d10 2024-09-16T14:59:31.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 10 monclient: _finish_auth 0 2024-09-16T14:59:31.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 10 monclient: _check_auth_tickets 2024-09-16T14:59:31.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:31.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 1 -- 172.21.15.26:0/2733832298 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0978001620 con 0x7f09880a1d10 2024-09-16T14:59:31.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:31.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:59:31.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 10 monclient: dump: 2024-09-16T14:59:31.123 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:31.124 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:31.124 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:31.124 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:31.124 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:31.124 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:31.124 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:31.124 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:31.124 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:31.124 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:31.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 5 monclient: authenticate success, global_id 284591 2024-09-16T14:59:31.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:31.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 1 -- 172.21.15.26:0/2733832298 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0978001900 con 0x7f09880a1d10 2024-09-16T14:59:31.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:31.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 1 -- 172.21.15.26:0/2733832298 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0978001c90 con 0x7f09880a1d10 2024-09-16T14:59:31.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:31.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:31.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0986ffd640 10 monclient: dump: 2024-09-16T14:59:31.125 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:31.125 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:31.125 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:31.125 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:31.125 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:31.125 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:31.125 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:31.125 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:31.125 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:31.125 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:31.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 10 monclient: get_monmap_and_config success 2024-09-16T14:59:31.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 4 set_mon_vals no callback set 2024-09-16T14:59:31.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 10 monclient: shutdown 2024-09-16T14:59:31.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:31.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 1 -- 172.21.15.26:0/2733832298 >> v1:172.21.15.26:6790/0 conn(0x7f09880a1d10 legacy=0x7f09880a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:31.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 1 -- 172.21.15.26:0/2733832298 shutdown_connections 2024-09-16T14:59:31.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 1 -- 172.21.15.26:0/2733832298 >> 172.21.15.26:0/2733832298 conn(0x7f09880a7f30 msgr2=0x7f09880a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:31.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 1 -- 172.21.15.26:0/2733832298 shutdown_connections 2024-09-16T14:59:31.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 1 -- 172.21.15.26:0/2733832298 wait complete. 2024-09-16T14:59:31.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 10 monclient: build_initial_monmap 2024-09-16T14:59:31.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 10 monclient: monmap: 2024-09-16T14:59:31.130 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:31.130 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:59:31.130 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:59:31.130 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:59:31.130 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:31.130 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:59:31.130 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:31.130 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:59:31.130 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:59:31.130 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:31.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 1 Processor -- start 2024-09-16T14:59:31.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 1 -- start start 2024-09-16T14:59:31.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 10 monclient: init 2024-09-16T14:59:31.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:31.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:59:31.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 10 monclient(hunting): picked mon.noname-a con 0x7f09880a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:31.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 10 monclient(hunting): picked mon.noname-b con 0x7f09880a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:31.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 10 monclient(hunting): picked mon.noname-c con 0x7f09880a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:31.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0988136fa0 con 0x7f09880a0000 2024-09-16T14:59:31.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09880a6480 con 0x7f09880a1d10 2024-09-16T14:59:31.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f098809e500 con 0x7f09880a2800 2024-09-16T14:59:31.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:31.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 10 monclient(hunting): authenticate will time out at 1995.242972s 2024-09-16T14:59:31.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0987fff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f09880a0000 legacy=0x7f098813d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:31.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098cff3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f09880a2800 0x7f09881d10b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48264/0 (socket says 172.21.15.26:48264) 2024-09-16T14:59:31.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098cff3640 1 -- 172.21.15.26:0/1344216918 learned_addr learned my addr 172.21.15.26:0/1344216918 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:31.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911069400 0 0) 0x7f098809e500 con 0x7f09880a2800 2024-09-16T14:59:31.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:59:31.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:31.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:31.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 10 monclient(hunting): my global_id is 284594 2024-09-16T14:59:31.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0968003530 con 0x7f09880a2800 2024-09-16T14:59:31.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 57153161 0 0) 0x7f0968003530 con 0x7f09880a2800 2024-09-16T14:59:31.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f098809e500 con 0x7f09880a2800 2024-09-16T14:59:31.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0980001c60 con 0x7f09880a2800 2024-09-16T14:59:31.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:31.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:59:31.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 10 monclient(hunting): dump: 2024-09-16T14:59:31.133 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:31.133 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:31.133 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:31.133 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:31.133 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:31.133 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:31.133 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:31.133 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:31.133 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:31.133 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:31.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:59:31.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746912972 0 0) 0x7f09880a6480 con 0x7f09880a1d10 2024-09-16T14:59:31.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 10 monclient(hunting): _negotiate 2024-09-16T14:59:31.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:31.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:31.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 10 monclient(hunting): my global_id is 304339 2024-09-16T14:59:31.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0968003530 con 0x7f09880a1d10 2024-09-16T14:59:31.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2397955396 0 0) 0x7f098809e500 con 0x7f09880a2800 2024-09-16T14:59:31.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 >> v1:172.21.15.189:6789/0 conn(0x7f09880a1d10 legacy=0x7f09881cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:31.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 >> v1:172.21.15.26:6789/0 conn(0x7f09880a0000 legacy=0x7f098813d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:59:31.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 10 monclient: _finish_hunting 0 2024-09-16T14:59:31.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 1 monclient: found mon.c 2024-09-16T14:59:31.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:31.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:31.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09881349e0 con 0x7f09880a2800 2024-09-16T14:59:31.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 10 monclient: _finish_auth 0 2024-09-16T14:59:31.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 10 monclient: _check_auth_tickets 2024-09-16T14:59:31.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f0984ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:31.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 5 monclient: authenticate success, global_id 284594 2024-09-16T14:59:31.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 10 monclient: _renew_subs 2024-09-16T14:59:31.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:31.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.118+0000 7f098ea56640 1 -- 172.21.15.26:0/1344216918 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0988136700 con 0x7f09880a2800 2024-09-16T14:59:31.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f098ea56640 10 monclient: _renew_subs 2024-09-16T14:59:31.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f098ea56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:31.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f098ea56640 1 -- 172.21.15.26:0/1344216918 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f098813cc50 con 0x7f09880a2800 2024-09-16T14:59:31.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0980002030 con 0x7f09880a2800 2024-09-16T14:59:31.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f0984ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:31.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0980003e70 con 0x7f09880a2800 2024-09-16T14:59:31.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f0984ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:31.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f0984ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:31.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f09867fc640 4 set_mon_vals no callback set 2024-09-16T14:59:31.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f0984ff9640 10 monclient: dump: 2024-09-16T14:59:31.136 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:31.136 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:31.136 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:31.136 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:31.136 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:31.136 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:31.136 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:31.136 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:31.136 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:31.136 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:31.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f098ea56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:31.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f098ea56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:31.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f098ea56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:31.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0980004050 con 0x7f09880a2800 2024-09-16T14:59:31.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f098ea56640 1 -- 172.21.15.26:0/1344216918 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096c005050 con 0x7f09880a2800 2024-09-16T14:59:31.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f09800442d0 con 0x7f09880a2800 2024-09-16T14:59:31.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.122+0000 7f0987fff640 10 monclient: get_auth_request con 0x7f09680360e0 auth_method 0 2024-09-16T14:59:31.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.126+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f09800149e0 con 0x7f09880a2800 2024-09-16T14:59:31.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.126+0000 7f0984ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:31.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.126+0000 7f0984ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:59:31.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.350+0000 7f098ea56640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:59:31.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.350+0000 7f098ea56640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:59:31.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.350+0000 7f098ea56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:31.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.350+0000 7f098ea56640 1 -- 172.21.15.26:0/1344216918 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f096c005880 con 0x7f09880a2800 2024-09-16T14:59:31.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.350+0000 7f0984ff9640 1 -- 172.21.15.26:0/1344216918 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2446748088) 0x7f0980013080 con 0x7f09880a2800 2024-09-16T14:59:31.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.350+0000 7f0984ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:59:31.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.350+0000 7f0984ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:59:31.354 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:59:31.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.354+0000 7f098ea56640 1 -- 172.21.15.26:0/1344216918 >> v1:172.21.15.189:6812/3196161582 conn(0x7f09680360e0 legacy=0x7f09680385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:31.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.354+0000 7f098ea56640 10 monclient: shutdown 2024-09-16T14:59:31.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.354+0000 7f098ea56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:31.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.354+0000 7f098ea56640 1 -- 172.21.15.26:0/1344216918 >> v1:172.21.15.26:6790/0 conn(0x7f09880a2800 legacy=0x7f09881d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:31.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.354+0000 7f098ea56640 1 -- 172.21.15.26:0/1344216918 shutdown_connections 2024-09-16T14:59:31.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.354+0000 7f098ea56640 1 -- 172.21.15.26:0/1344216918 >> 172.21.15.26:0/1344216918 conn(0x7f09880a7f30 msgr2=0x7f098806b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:31.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.354+0000 7f098ea56640 1 -- 172.21.15.26:0/1344216918 shutdown_connections 2024-09-16T14:59:31.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:31.354+0000 7f098ea56640 1 -- 172.21.15.26:0/1344216918 wait complete. 2024-09-16T14:59:34.371 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:59:34.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 10 monclient: get_monmap_and_config 2024-09-16T14:59:34.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 10 monclient: build_initial_monmap 2024-09-16T14:59:34.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 10 monclient: monmap: 2024-09-16T14:59:34.504 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:34.504 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:34.504 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:59:34.506354+0000 2024-09-16T14:59:34.504 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:59:34.506354+0000 2024-09-16T14:59:34.504 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:34.504 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:34.504 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:34.505 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:59:34.505 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:59:34.505 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:34.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 1 Processor -- start 2024-09-16T14:59:34.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 1 -- start start 2024-09-16T14:59:34.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 10 monclient: init 2024-09-16T14:59:34.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:34.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T14:59:34.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 10 monclient(hunting): picked mon.noname-c con 0x7fb63009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:34.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 10 monclient(hunting): picked mon.noname-b con 0x7fb630132650 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:34.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 10 monclient(hunting): picked mon.noname-a con 0x7fb630138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:34.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb63009d260 con 0x7fb630138ff0 2024-09-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb630132a80 con 0x7fb63009cb80 2024-09-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb63009cf60 con 0x7fb630132650 2024-09-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 10 monclient(hunting): authenticate will time out at 1998.626384s 2024-09-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb634c73640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fb630138ff0 legacy=0x7fb63013b3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62f7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb630132650 0x7fb6301378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48278/0 (socket says 172.21.15.26:48278) 2024-09-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62f7fe640 1 -- 172.21.15.26:0/431627083 learned_addr learned my addr 172.21.15.26:0/431627083 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 1 -- 172.21.15.26:0/431627083 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058232897 0 0) 0x7fb63009cf60 con 0x7fb630132650 2024-09-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 10 monclient(hunting): _negotiate 2024-09-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:34.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:34.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 10 monclient(hunting): my global_id is 284597 2024-09-16T14:59:34.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 1 -- 172.21.15.26:0/431627083 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb61c003530 con 0x7fb630132650 2024-09-16T14:59:34.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 1 -- 172.21.15.26:0/431627083 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3295011302 0 0) 0x7fb61c003530 con 0x7fb630132650 2024-09-16T14:59:34.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 1 -- 172.21.15.26:0/431627083 >> v1:172.21.15.189:6789/0 conn(0x7fb63009cb80 legacy=0x7fb6300717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:34.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 1 -- 172.21.15.26:0/431627083 >> v1:172.21.15.26:6789/0 conn(0x7fb630138ff0 legacy=0x7fb63013b3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:59:34.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 10 monclient: _finish_hunting 0 2024-09-16T14:59:34.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 1 monclient: found mon.noname-b 2024-09-16T14:59:34.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:34.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:59:34.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 1 -- 172.21.15.26:0/431627083 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63009f2d0 con 0x7fb630132650 2024-09-16T14:59:34.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 10 monclient: _finish_auth 0 2024-09-16T14:59:34.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 10 monclient: _check_auth_tickets 2024-09-16T14:59:34.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 1 -- 172.21.15.26:0/431627083 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb620001660 con 0x7fb630132650 2024-09-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 10 monclient: dump: 2024-09-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:34.511 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 5 monclient: authenticate success, global_id 284597 2024-09-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 1 -- 172.21.15.26:0/431627083 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6200018c0 con 0x7fb630132650 2024-09-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 10 monclient: get_monmap_and_config success 2024-09-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb6366d6640 4 set_mon_vals no callback set 2024-09-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 1 -- 172.21.15.26:0/431627083 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb620001c50 con 0x7fb630132650 2024-09-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.502+0000 7fb62effd640 10 monclient: dump: 2024-09-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:34.512 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:34.513 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:34.513 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:34.513 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:34.513 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:34.513 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:34.513 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:34.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient: shutdown 2024-09-16T14:59:34.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:34.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 1 -- 172.21.15.26:0/431627083 >> v1:172.21.15.26:6790/0 conn(0x7fb630132650 legacy=0x7fb6301378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:34.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 1 -- 172.21.15.26:0/431627083 shutdown_connections 2024-09-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 1 -- 172.21.15.26:0/431627083 >> 172.21.15.26:0/431627083 conn(0x7fb63009fb50 msgr2=0x7fb63009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 1 -- 172.21.15.26:0/431627083 shutdown_connections 2024-09-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 1 -- 172.21.15.26:0/431627083 wait complete. 2024-09-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient: build_initial_monmap 2024-09-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient: monmap: 2024-09-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:59:34.514 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:34.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 1 Processor -- start 2024-09-16T14:59:34.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 1 -- start start 2024-09-16T14:59:34.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient: init 2024-09-16T14:59:34.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:34.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient(hunting): picked mon.noname-c con 0x7fb63009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient(hunting): picked mon.noname-b con 0x7fb630132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient(hunting): picked mon.noname-a con 0x7fb630138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb63006dfa0 con 0x7fb630138ff0 2024-09-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb63009b370 con 0x7fb630132650 2024-09-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb63009a240 con 0x7fb63009cb80 2024-09-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient(hunting): authenticate will time out at 1998.629727s 2024-09-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb634c73640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fb630138ff0 legacy=0x7fb6301d1d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb63009cb80 0x7fb63009a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48290/0 (socket says 172.21.15.26:48290) 2024-09-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62ffff640 1 -- 172.21.15.26:0/308038809 learned_addr learned my addr 172.21.15.26:0/308038809 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:34.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 1 -- 172.21.15.26:0/308038809 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169856098 0 0) 0x7fb63009a240 con 0x7fb63009cb80 2024-09-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 10 monclient(hunting): _negotiate 2024-09-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 10 monclient(hunting): my global_id is 284600 2024-09-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 1 -- 172.21.15.26:0/308038809 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb610003530 con 0x7fb63009cb80 2024-09-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 1 -- 172.21.15.26:0/308038809 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3201416227 0 0) 0x7fb610003530 con 0x7fb63009cb80 2024-09-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 1 -- 172.21.15.26:0/308038809 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb63009a240 con 0x7fb63009cb80 2024-09-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 1 -- 172.21.15.26:0/308038809 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb618001820 con 0x7fb63009cb80 2024-09-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 10 monclient(hunting): dump: 2024-09-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:34.519 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:34.520 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:34.520 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:34.520 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:34.520 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:34.520 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:34.520 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:34.520 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:34.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:59:34.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 1 -- 172.21.15.26:0/308038809 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 909508428 0 0) 0x7fb63009a240 con 0x7fb63009cb80 2024-09-16T14:59:34.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 1 -- 172.21.15.26:0/308038809 >> v1:172.21.15.189:6789/0 conn(0x7fb630132650 legacy=0x7fb63009ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:34.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 1 -- 172.21.15.26:0/308038809 >> v1:172.21.15.26:6789/0 conn(0x7fb630138ff0 legacy=0x7fb6301d1d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:59:34.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 10 monclient: _finish_hunting 0 2024-09-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 1 monclient: found mon.c 2024-09-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 1 -- 172.21.15.26:0/308038809 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63013c400 con 0x7fb63009cb80 2024-09-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 10 monclient: _finish_auth 0 2024-09-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 10 monclient: _check_auth_tickets 2024-09-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 5 monclient: authenticate success, global_id 284600 2024-09-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient: _renew_subs 2024-09-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 1 -- 172.21.15.26:0/308038809 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6300a21e0 con 0x7fb63009cb80 2024-09-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient: _renew_subs 2024-09-16T14:59:34.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 1 -- 172.21.15.26:0/308038809 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6301386e0 con 0x7fb63009cb80 2024-09-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 1 -- 172.21.15.26:0/308038809 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb618003620 con 0x7fb63009cb80 2024-09-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 1 -- 172.21.15.26:0/308038809 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb6180039b0 con 0x7fb63009cb80 2024-09-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 10 monclient: dump: 2024-09-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:34.522 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62e7fc640 4 set_mon_vals no callback set 2024-09-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb6366d6640 1 -- 172.21.15.26:0/308038809 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb614005050 con 0x7fb63009cb80 2024-09-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 1 -- 172.21.15.26:0/308038809 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb618003b90 con 0x7fb63009cb80 2024-09-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.506+0000 7fb62cff9640 1 -- 172.21.15.26:0/308038809 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fb618043e00 con 0x7fb63009cb80 2024-09-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.510+0000 7fb62f7fe640 10 monclient: get_auth_request con 0x7fb610036000 auth_method 0 2024-09-16T14:59:34.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.510+0000 7fb62cff9640 1 -- 172.21.15.26:0/308038809 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb6180146e0 con 0x7fb63009cb80 2024-09-16T14:59:34.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.510+0000 7fb62cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:34.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.510+0000 7fb62cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:59:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.730+0000 7fb6366d6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:59:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.730+0000 7fb6366d6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:59:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.730+0000 7fb6366d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.730+0000 7fb6366d6640 1 -- 172.21.15.26:0/308038809 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb614005880 con 0x7fb63009cb80 2024-09-16T14:59:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.730+0000 7fb62cff9640 1 -- 172.21.15.26:0/308038809 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 3241846681) 0x7fb618012b50 con 0x7fb63009cb80 2024-09-16T14:59:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.730+0000 7fb62cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:59:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.730+0000 7fb62cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:59:34.732 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:59:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.730+0000 7fb6366d6640 1 -- 172.21.15.26:0/308038809 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb610036000 legacy=0x7fb6100384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.730+0000 7fb6366d6640 10 monclient: shutdown 2024-09-16T14:59:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.730+0000 7fb6366d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.730+0000 7fb6366d6640 1 -- 172.21.15.26:0/308038809 >> v1:172.21.15.26:6790/0 conn(0x7fb63009cb80 legacy=0x7fb63009a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.730+0000 7fb6366d6640 1 -- 172.21.15.26:0/308038809 shutdown_connections 2024-09-16T14:59:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.730+0000 7fb6366d6640 1 -- 172.21.15.26:0/308038809 >> 172.21.15.26:0/308038809 conn(0x7fb63009fb50 msgr2=0x7fb630130750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.730+0000 7fb6366d6640 1 -- 172.21.15.26:0/308038809 shutdown_connections 2024-09-16T14:59:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:34.730+0000 7fb6366d6640 1 -- 172.21.15.26:0/308038809 wait complete. 2024-09-16T14:59:37.750 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T14:59:37.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d2003a640 10 monclient: get_monmap_and_config 2024-09-16T14:59:37.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d2003a640 10 monclient: build_initial_monmap 2024-09-16T14:59:37.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d2003a640 10 monclient: monmap: 2024-09-16T14:59:37.881 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:37.881 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:37.881 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:59:37.883382+0000 2024-09-16T14:59:37.881 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:59:37.883382+0000 2024-09-16T14:59:37.881 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:37.881 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:37.881 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:37.882 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:59:37.882 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:59:37.882 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:37.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d2003a640 1 Processor -- start 2024-09-16T14:59:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d2003a640 1 -- start start 2024-09-16T14:59:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d2003a640 10 monclient: init 2024-09-16T14:59:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d2003a640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d2003a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:59:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d2003a640 10 monclient(hunting): picked mon.noname-c con 0x7f4d180a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d2003a640 10 monclient(hunting): picked mon.noname-a con 0x7f4d180a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d2003a640 10 monclient(hunting): picked mon.noname-b con 0x7f4d180a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d2003a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d180a3250 con 0x7f4d180a0000 2024-09-16T14:59:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d2003a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1809f600 con 0x7f4d180a1d10 2024-09-16T14:59:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d2003a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d180a3830 con 0x7f4d180a2800 2024-09-16T14:59:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d2003a640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d2003a640 10 monclient(hunting): authenticate will time out at 2002.003442s 2024-09-16T14:59:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d1e5d7640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4d180a2800 0x7f4d1813d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:42500/0 (socket says 172.21.15.26:42500) 2024-09-16T14:59:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d1d5d5640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f4d180a0000 legacy=0x7f4d18139d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:37.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:37.878+0000 7f4d1e5d7640 1 -- 172.21.15.26:0/2526359696 learned_addr learned my addr 172.21.15.26:0/2526359696 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:38.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:38.082+0000 7f4d1d5d5640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.26:6789/0 conn(0x7f4d180a0000 legacy=0x7f4d18139d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:38.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:38.486+0000 7f4d1d5d5640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.26:6789/0 conn(0x7f4d180a0000 legacy=0x7f4d18139d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:39.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:39.290+0000 7f4d1d5d5640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.26:6789/0 conn(0x7f4d180a0000 legacy=0x7f4d18139d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:40.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:40.882+0000 7f4d0f7fe640 10 monclient(hunting): tick 2024-09-16T14:59:40.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:40.882+0000 7f4d0f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:59:40.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:40.882+0000 7f4d0f7fe640 1 monclient(hunting): continuing hunt 2024-09-16T14:59:40.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:40.882+0000 7f4d0f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:59:40.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:40.882+0000 7f4d0f7fe640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.26:6790/0 conn(0x7f4d180a2800 legacy=0x7f4d1813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:40.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:40.882+0000 7f4d0f7fe640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.189:6789/0 conn(0x7f4d180a1d10 legacy=0x7f4d180a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:40.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:40.882+0000 7f4d0f7fe640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.26:6789/0 conn(0x7f4d180a0000 legacy=0x7f4d18139d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:59:40.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:40.882+0000 7f4d0f7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:59:40.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:40.882+0000 7f4d0f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f4cfc0036d0 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:40.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:40.882+0000 7f4d0f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f4d180a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:40.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:40.882+0000 7f4d0f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f4d180a2800 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:40.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:40.882+0000 7f4d0f7fe640 1 -- 172.21.15.26:0/2526359696 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1809ed60 con 0x7f4d180a0000 2024-09-16T14:59:40.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:40.882+0000 7f4d0f7fe640 1 -- 172.21.15.26:0/2526359696 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d180a3250 con 0x7f4cfc0036d0 2024-09-16T14:59:40.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:40.882+0000 7f4d0f7fe640 1 -- 172.21.15.26:0/2526359696 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc010380 con 0x7f4d180a2800 2024-09-16T14:59:40.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:40.882+0000 7f4d0f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:40.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:40.882+0000 7f4d1d5d5640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.26:6789/0 conn(0x7f4d180a0000 legacy=0x7f4cfc0093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:41.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:41.086+0000 7f4d1d5d5640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.26:6789/0 conn(0x7f4d180a0000 legacy=0x7f4cfc0093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:41.490+0000 7f4d1d5d5640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.26:6789/0 conn(0x7f4d180a0000 legacy=0x7f4cfc0093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:42.294+0000 7f4d1d5d5640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.26:6789/0 conn(0x7f4d180a0000 legacy=0x7f4cfc0093b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:43.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:43.882+0000 7f4d0f7fe640 10 monclient(hunting): tick 2024-09-16T14:59:43.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:43.882+0000 7f4d0f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T14:59:43.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:43.882+0000 7f4d0f7fe640 1 monclient(hunting): continuing hunt 2024-09-16T14:59:43.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:43.882+0000 7f4d0f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T14:59:43.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:43.882+0000 7f4d0f7fe640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.26:6790/0 conn(0x7f4d180a2800 legacy=0x7f4cfc00cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:43.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:43.882+0000 7f4d0f7fe640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.189:6789/0 conn(0x7f4cfc0036d0 legacy=0x7f4cfc005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:43.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:43.882+0000 7f4d0f7fe640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.26:6789/0 conn(0x7f4d180a0000 legacy=0x7f4cfc0093b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:59:43.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:43.882+0000 7f4d0f7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:59:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:43.882+0000 7f4d0f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f4cfc010610 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:43.882+0000 7f4d0f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f4cfc012890 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:43.882+0000 7f4d0f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f4d180a0000 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:43.882+0000 7f4d0f7fe640 1 -- 172.21.15.26:0/2526359696 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc011590 con 0x7f4d180a0000 2024-09-16T14:59:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:43.882+0000 7f4d0f7fe640 1 -- 172.21.15.26:0/2526359696 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1809ed60 con 0x7f4cfc012890 2024-09-16T14:59:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:43.882+0000 7f4d0f7fe640 1 -- 172.21.15.26:0/2526359696 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc0073a0 con 0x7f4cfc010610 2024-09-16T14:59:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:43.882+0000 7f4d0f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:43.882+0000 7f4d1e5d7640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.26:6789/0 conn(0x7f4d180a0000 legacy=0x7f4cfc005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:44.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:44.086+0000 7f4d1e5d7640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.26:6789/0 conn(0x7f4d180a0000 legacy=0x7f4cfc005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:44.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:44.490+0000 7f4d1e5d7640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.26:6789/0 conn(0x7f4d180a0000 legacy=0x7f4cfc005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:45.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.234+0000 7f4d1cdd4640 1 -- 172.21.15.26:0/2526359696 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290109642 0 0) 0x7f4d1809ed60 con 0x7f4cfc012890 2024-09-16T14:59:45.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.234+0000 7f4d1cdd4640 10 monclient(hunting): _negotiate 2024-09-16T14:59:45.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.234+0000 7f4d1cdd4640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:45.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.234+0000 7f4d1cdd4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:45.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.234+0000 7f4d1cdd4640 10 monclient(hunting): my global_id is 304342 2024-09-16T14:59:45.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.234+0000 7f4d1cdd4640 1 -- 172.21.15.26:0/2526359696 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d04003530 con 0x7f4cfc012890 2024-09-16T14:59:45.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 1 -- 172.21.15.26:0/2526359696 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67680671 0 0) 0x7f4cfc0073a0 con 0x7f4cfc010610 2024-09-16T14:59:45.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 10 monclient(hunting): _negotiate 2024-09-16T14:59:45.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:45.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:45.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 10 monclient(hunting): my global_id is 284603 2024-09-16T14:59:45.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 1 -- 172.21.15.26:0/2526359696 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d1809ed60 con 0x7f4cfc010610 2024-09-16T14:59:45.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 1 -- 172.21.15.26:0/2526359696 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3135022364 0 0) 0x7f4d1809ed60 con 0x7f4cfc010610 2024-09-16T14:59:45.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.189:6789/0 conn(0x7f4cfc012890 legacy=0x7f4cfc014c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.26:6789/0 conn(0x7f4d180a0000 legacy=0x7f4cfc005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:59:45.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 10 monclient: _finish_hunting 0 2024-09-16T14:59:45.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 1 monclient: found mon.noname-b 2024-09-16T14:59:45.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:45.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:59:45.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 1 -- 172.21.15.26:0/2526359696 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cfc008540 con 0x7f4cfc010610 2024-09-16T14:59:45.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 10 monclient: _finish_auth 0 2024-09-16T14:59:45.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 10 monclient: _check_auth_tickets 2024-09-16T14:59:45.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:45.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 1 -- 172.21.15.26:0/2526359696 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4d140015a0 con 0x7f4cfc010610 2024-09-16T14:59:45.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:59:45.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 10 monclient: dump: 2024-09-16T14:59:45.246 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.246 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.246 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.246 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.246 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.246 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.246 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.246 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.246 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.246 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d2003a640 5 monclient: authenticate success, global_id 284603 2024-09-16T14:59:45.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d2003a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:45.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 1 -- 172.21.15.26:0/2526359696 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d140017b0 con 0x7f4cfc010610 2024-09-16T14:59:45.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:45.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 1 -- 172.21.15.26:0/2526359696 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4d14001b80 con 0x7f4cfc010610 2024-09-16T14:59:45.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d2003a640 10 monclient: get_monmap_and_config success 2024-09-16T14:59:45.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d2003a640 4 set_mon_vals no callback set 2024-09-16T14:59:45.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:45.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.242+0000 7f4d1cdd4640 10 monclient: dump: 2024-09-16T14:59:45.248 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.248 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.248 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.248 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.248 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.248 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.249 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.249 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.249 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.249 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 10 monclient: shutdown 2024-09-16T14:59:45.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:45.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 1 -- 172.21.15.26:0/2526359696 >> v1:172.21.15.26:6790/0 conn(0x7f4cfc010610 legacy=0x7f4cfc0109f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 1 -- 172.21.15.26:0/2526359696 shutdown_connections 2024-09-16T14:59:45.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 1 -- 172.21.15.26:0/2526359696 >> 172.21.15.26:0/2526359696 conn(0x7f4d180a7f30 msgr2=0x7f4d180a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:45.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 1 -- 172.21.15.26:0/2526359696 shutdown_connections 2024-09-16T14:59:45.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 1 -- 172.21.15.26:0/2526359696 wait complete. 2024-09-16T14:59:45.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 10 monclient: build_initial_monmap 2024-09-16T14:59:45.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 10 monclient: monmap: 2024-09-16T14:59:45.251 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:45.251 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:59:45.251 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:59:45.251 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:59:45.251 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:45.251 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:59:45.252 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:45.252 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:59:45.252 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:59:45.252 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 1 Processor -- start 2024-09-16T14:59:45.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 1 -- start start 2024-09-16T14:59:45.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 10 monclient: init 2024-09-16T14:59:45.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:45.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T14:59:45.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 10 monclient(hunting): picked mon.noname-b con 0x7f4cfc010610 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:45.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 10 monclient(hunting): picked mon.noname-a con 0x7f4cfc012890 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:45.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 10 monclient(hunting): picked mon.noname-c con 0x7f4d180a0000 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:45.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1809e500 con 0x7f4cfc012890 2024-09-16T14:59:45.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d180a66b0 con 0x7f4cfc010610 2024-09-16T14:59:45.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d1d5d5640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f4cfc012890 legacy=0x7f4d181ce460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:45.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d18121400 con 0x7f4d180a0000 2024-09-16T14:59:45.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:45.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d2003a640 10 monclient(hunting): authenticate will time out at 2009.371757s 2024-09-16T14:59:45.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d1e5d7640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4d180a0000 0x7f4d181d1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:42530/0 (socket says 172.21.15.26:42530) 2024-09-16T14:59:45.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.246+0000 7f4d1e5d7640 1 -- 172.21.15.26:0/1166071304 learned_addr learned my addr 172.21.15.26:0/1166071304 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:45.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526215215 0 0) 0x7f4d18121400 con 0x7f4d180a0000 2024-09-16T14:59:45.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:45.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:45.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:45.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient(hunting): my global_id is 284606 2024-09-16T14:59:45.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cfc011580 con 0x7f4d180a0000 2024-09-16T14:59:45.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551730527 0 0) 0x7f4d180a66b0 con 0x7f4cfc010610 2024-09-16T14:59:45.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:45.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:45.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:45.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient(hunting): my global_id is 304345 2024-09-16T14:59:45.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d18121400 con 0x7f4cfc010610 2024-09-16T14:59:45.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1159155840 0 0) 0x7f4cfc011580 con 0x7f4d180a0000 2024-09-16T14:59:45.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d180a66b0 con 0x7f4d180a0000 2024-09-16T14:59:45.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4d10001b30 con 0x7f4d180a0000 2024-09-16T14:59:45.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:59:45.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient(hunting): dump: 2024-09-16T14:59:45.257 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.257 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.257 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.257 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.257 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.258 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.258 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.258 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.258 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.258 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:59:45.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3643913450 0 0) 0x7f4d18121400 con 0x7f4cfc010610 2024-09-16T14:59:45.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cfc011580 con 0x7f4cfc010610 2024-09-16T14:59:45.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4d14001ac0 con 0x7f4cfc010610 2024-09-16T14:59:45.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T14:59:45.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient(hunting): dump: 2024-09-16T14:59:45.259 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.259 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.259 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.259 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.259 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.259 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.259 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.259 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.259 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.259 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1346356408 0 0) 0x7f4d180a66b0 con 0x7f4d180a0000 2024-09-16T14:59:45.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 >> v1:172.21.15.189:6789/0 conn(0x7f4cfc010610 legacy=0x7f4d18133d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 >> v1:172.21.15.26:6789/0 conn(0x7f4cfc012890 legacy=0x7f4d181ce460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:59:45.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:59:45.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 monclient: found mon.c 2024-09-16T14:59:45.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:45.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:45.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d181216e0 con 0x7f4d180a0000 2024-09-16T14:59:45.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient: _finish_auth 0 2024-09-16T14:59:45.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:59:45.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:45.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d2003a640 5 monclient: authenticate success, global_id 284606 2024-09-16T14:59:45.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d2003a640 10 monclient: _renew_subs 2024-09-16T14:59:45.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d2003a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:45.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d2003a640 1 -- 172.21.15.26:0/1166071304 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1811fcd0 con 0x7f4d180a0000 2024-09-16T14:59:45.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d2003a640 10 monclient: _renew_subs 2024-09-16T14:59:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d2003a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d2003a640 1 -- 172.21.15.26:0/1166071304 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1811f380 con 0x7f4d180a0000 2024-09-16T14:59:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d10002040 con 0x7f4d180a0000 2024-09-16T14:59:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4d10004d10 con 0x7f4d180a0000 2024-09-16T14:59:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 10 monclient: dump: 2024-09-16T14:59:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.262 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.262 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.262 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.262 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.262 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0ffff640 4 set_mon_vals no callback set 2024-09-16T14:59:45.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4d1000f780 con 0x7f4d180a0000 2024-09-16T14:59:45.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d2003a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:45.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d2003a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:45.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d2003a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:45.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d2003a640 1 -- 172.21.15.26:0/1166071304 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d00005050 con 0x7f4d180a0000 2024-09-16T14:59:45.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f4d10044040 con 0x7f4d180a0000 2024-09-16T14:59:45.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.250+0000 7f4d1ddd6640 10 monclient: get_auth_request con 0x7f4cfc037600 auth_method 0 2024-09-16T14:59:45.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.258+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4d10014710 con 0x7f4d180a0000 2024-09-16T14:59:45.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.258+0000 7f4d0e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:45.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.258+0000 7f4d0e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:59:45.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.474+0000 7f4d2003a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T14:59:45.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.474+0000 7f4d2003a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T14:59:45.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.474+0000 7f4d2003a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:45.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.474+0000 7f4d2003a640 1 -- 172.21.15.26:0/1166071304 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4d00005880 con 0x7f4d180a0000 2024-09-16T14:59:45.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.474+0000 7f4d0e7fc640 1 -- 172.21.15.26:0/1166071304 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 1627873948) 0x7f4d10012db0 con 0x7f4d180a0000 2024-09-16T14:59:45.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.474+0000 7f4d0e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T14:59:45.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.474+0000 7f4d0e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:59:45.479 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":160,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T14:59:45.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.478+0000 7f4d2003a640 1 -- 172.21.15.26:0/1166071304 >> v1:172.21.15.189:6812/3196161582 conn(0x7f4cfc037600 legacy=0x7f4cfc039ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.478+0000 7f4d2003a640 10 monclient: shutdown 2024-09-16T14:59:45.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.478+0000 7f4d2003a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:45.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.478+0000 7f4d2003a640 1 -- 172.21.15.26:0/1166071304 >> v1:172.21.15.26:6790/0 conn(0x7f4d180a0000 legacy=0x7f4d181d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.478+0000 7f4d2003a640 1 -- 172.21.15.26:0/1166071304 shutdown_connections 2024-09-16T14:59:45.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.478+0000 7f4d2003a640 1 -- 172.21.15.26:0/1166071304 >> 172.21.15.26:0/1166071304 conn(0x7f4d180a7f30 msgr2=0x7f4d1806b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:45.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.478+0000 7f4d2003a640 1 -- 172.21.15.26:0/1166071304 shutdown_connections 2024-09-16T14:59:45.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.478+0000 7f4d2003a640 1 -- 172.21.15.26:0/1166071304 wait complete. 2024-09-16T14:59:45.496 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T14:59:45.496 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T14:59:45.497 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T14:59:45.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 10 monclient: get_monmap_and_config 2024-09-16T14:59:45.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 10 monclient: build_initial_monmap 2024-09-16T14:59:45.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 10 monclient: monmap: 2024-09-16T14:59:45.608 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:45.608 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.608 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:59:45.609837+0000 2024-09-16T14:59:45.608 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:59:45.609837+0000 2024-09-16T14:59:45.608 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:45.608 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.608 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:45.608 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:59:45.608 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:59:45.608 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 1 Processor -- start 2024-09-16T14:59:45.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 1 -- start start 2024-09-16T14:59:45.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 10 monclient: init 2024-09-16T14:59:45.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:45.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T14:59:45.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 10 monclient(hunting): picked mon.noname-a con 0x7f110409cdc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:45.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 10 monclient(hunting): picked mon.noname-b con 0x7f110409e7a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:45.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 10 monclient(hunting): picked mon.noname-c con 0x7f110409d1a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:45.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110409d5b0 con 0x7f110409cdc0 2024-09-16T14:59:45.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1104131a90 con 0x7f110409d1a0 2024-09-16T14:59:45.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110409aff0 con 0x7f110409e7a0 2024-09-16T14:59:45.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:45.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 10 monclient(hunting): authenticate will time out at 2009.729773s 2024-09-16T14:59:45.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1109b06640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f110409cdc0 legacy=0x7f1104130c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:45.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1109305640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f110409e7a0 0x7f1104131380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:42536/0 (socket says 172.21.15.26:42536) 2024-09-16T14:59:45.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1109305640 1 -- 172.21.15.26:0/2039905599 learned_addr learned my addr 172.21.15.26:0/2039905599 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:45.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 1 -- 172.21.15.26:0/2039905599 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130724911 0 0) 0x7f110409aff0 con 0x7f110409e7a0 2024-09-16T14:59:45.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 10 monclient(hunting): _negotiate 2024-09-16T14:59:45.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:45.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:45.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 10 monclient(hunting): my global_id is 284609 2024-09-16T14:59:45.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 1 -- 172.21.15.26:0/2039905599 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f0003450 con 0x7f110409e7a0 2024-09-16T14:59:45.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 1 -- 172.21.15.26:0/2039905599 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2707555664 0 0) 0x7f10f0003450 con 0x7f110409e7a0 2024-09-16T14:59:45.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 1 -- 172.21.15.26:0/2039905599 >> v1:172.21.15.189:6789/0 conn(0x7f110409d1a0 legacy=0x7f1104139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 1 -- 172.21.15.26:0/2039905599 >> v1:172.21.15.26:6789/0 conn(0x7f110409cdc0 legacy=0x7f1104130c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:59:45.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 10 monclient: _finish_hunting 0 2024-09-16T14:59:45.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 1 monclient: found mon.noname-b 2024-09-16T14:59:45.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:45.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:59:45.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 1 -- 172.21.15.26:0/2039905599 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110409b2d0 con 0x7f110409e7a0 2024-09-16T14:59:45.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 10 monclient: _finish_auth 0 2024-09-16T14:59:45.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 10 monclient: _check_auth_tickets 2024-09-16T14:59:45.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:45.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 1 -- 172.21.15.26:0/2039905599 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10f4001620 con 0x7f110409e7a0 2024-09-16T14:59:45.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:59:45.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 10 monclient: dump: 2024-09-16T14:59:45.618 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.618 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.619 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.619 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.619 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.619 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.619 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.619 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.619 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.619 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 5 monclient: authenticate success, global_id 284609 2024-09-16T14:59:45.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:45.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 1 -- 172.21.15.26:0/2039905599 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10f40018d0 con 0x7f110409e7a0 2024-09-16T14:59:45.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:45.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 10 monclient: get_monmap_and_config success 2024-09-16T14:59:45.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 1 -- 172.21.15.26:0/2039905599 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10f4001ca0 con 0x7f110409e7a0 2024-09-16T14:59:45.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 4 set_mon_vals no callback set 2024-09-16T14:59:45.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:45.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f1108b04640 10 monclient: dump: 2024-09-16T14:59:45.620 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.620 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.620 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.620 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.620 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.620 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.620 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.620 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.621 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.621 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 10 monclient: shutdown 2024-09-16T14:59:45.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:45.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 1 -- 172.21.15.26:0/2039905599 >> v1:172.21.15.26:6790/0 conn(0x7f110409e7a0 legacy=0x7f1104131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 1 -- 172.21.15.26:0/2039905599 shutdown_connections 2024-09-16T14:59:45.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 1 -- 172.21.15.26:0/2039905599 >> 172.21.15.26:0/2039905599 conn(0x7f110409fb50 msgr2=0x7f110409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:45.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 1 -- 172.21.15.26:0/2039905599 shutdown_connections 2024-09-16T14:59:45.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 1 -- 172.21.15.26:0/2039905599 wait complete. 2024-09-16T14:59:45.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 10 monclient: build_initial_monmap 2024-09-16T14:59:45.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 10 monclient: monmap: 2024-09-16T14:59:45.622 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:45.622 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:59:45.622 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:59:45.622 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:59:45.622 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:45.622 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:59:45.622 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:45.622 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:59:45.622 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:59:45.622 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.606+0000 7f110bd6a640 1 Processor -- start 2024-09-16T14:59:45.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 1 -- start start 2024-09-16T14:59:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 10 monclient: init 2024-09-16T14:59:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T14:59:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 10 monclient(hunting): picked mon.noname-a con 0x7f110409cdc0 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 10 monclient(hunting): picked mon.noname-b con 0x7f110409d1a0 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 10 monclient(hunting): picked mon.noname-c con 0x7f110409e7a0 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1104130b00 con 0x7f110409cdc0 2024-09-16T14:59:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f1109b06640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f110409cdc0 legacy=0x7f110412fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11040a1640 con 0x7f110409d1a0 2024-09-16T14:59:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110411d880 con 0x7f110409e7a0 2024-09-16T14:59:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 10 monclient(hunting): authenticate will time out at 2009.732620s 2024-09-16T14:59:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110a307640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f110409e7a0 0x7f11041d1f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:42542/0 (socket says 172.21.15.26:42542) 2024-09-16T14:59:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110a307640 1 -- 172.21.15.26:0/3916567862 learned_addr learned my addr 172.21.15.26:0/3916567862 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401564857 0 0) 0x7f110411d880 con 0x7f110409e7a0 2024-09-16T14:59:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient(hunting): my global_id is 284612 2024-09-16T14:59:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e8003560 con 0x7f110409e7a0 2024-09-16T14:59:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2512622375 0 0) 0x7f10e8003560 con 0x7f110409e7a0 2024-09-16T14:59:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f110411d880 con 0x7f110409e7a0 2024-09-16T14:59:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10fc001c80 con 0x7f110409e7a0 2024-09-16T14:59:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:59:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient(hunting): dump: 2024-09-16T14:59:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.625 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:59:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3384942068 0 0) 0x7f110411d880 con 0x7f110409e7a0 2024-09-16T14:59:45.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 >> v1:172.21.15.189:6789/0 conn(0x7f110409d1a0 legacy=0x7f11041ce830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 >> v1:172.21.15.26:6789/0 conn(0x7f110409cdc0 legacy=0x7f110412fea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:59:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:59:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 monclient: found mon.c 2024-09-16T14:59:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110411be70 con 0x7f110409e7a0 2024-09-16T14:59:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient: _finish_auth 0 2024-09-16T14:59:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:59:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 5 monclient: authenticate success, global_id 284612 2024-09-16T14:59:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 10 monclient: _renew_subs 2024-09-16T14:59:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 1 -- 172.21.15.26:0/3916567862 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f110411d5a0 con 0x7f110409e7a0 2024-09-16T14:59:45.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 10 monclient: _renew_subs 2024-09-16T14:59:45.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:45.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 1 -- 172.21.15.26:0/3916567862 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11041305b0 con 0x7f110409e7a0 2024-09-16T14:59:45.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10fc002050 con 0x7f110409e7a0 2024-09-16T14:59:45.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:45.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f10fc003df0 con 0x7f110409e7a0 2024-09-16T14:59:45.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:45.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient: dump: 2024-09-16T14:59:45.627 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.627 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.627 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.627 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.627 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.628 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.628 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.628 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.628 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.628 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fbfff640 4 set_mon_vals no callback set 2024-09-16T14:59:45.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:45.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f110bd6a640 1 -- 172.21.15.26:0/3916567862 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f10ec006590 con 0x7f10ec002830 2024-09-16T14:59:45.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f10fc003fd0 con 0x7f110409e7a0 2024-09-16T14:59:45.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f10fc044060 con 0x7f110409e7a0 2024-09-16T14:59:45.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994855285 0 0) 0x7f10ec006590 con 0x7f10ec002830 2024-09-16T14:59:45.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:45.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient(hunting): my global_id is 284615 2024-09-16T14:59:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f110411d880 con 0x7f10ec002830 2024-09-16T14:59:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3761345596 0 0) 0x7f110411d880 con 0x7f10ec002830 2024-09-16T14:59:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10ec006590 con 0x7f10ec002830 2024-09-16T14:59:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f1109305640 10 monclient: get_auth_request con 0x7f10e8036040 auth_method 0 2024-09-16T14:59:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1100003ef0 con 0x7f10ec002830 2024-09-16T14:59:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient: dump: 2024-09-16T14:59:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.630 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2550830520 0 0) 0x7f10ec006590 con 0x7f10ec002830 2024-09-16T14:59:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10ec00a0e0 con 0x7f10ec002830 2024-09-16T14:59:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1100006300 con 0x7f10ec002830 2024-09-16T14:59:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:45.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:59:45.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 >> v1:172.21.15.26:6790/0 conn(0x7f10ec002830 legacy=0x7f10ec004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.610+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 >> v1:172.21.15.26:6790/0 conn(0x7f10ec002830 legacy=0x7f10ec004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f110bd6a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:59:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f110bd6a640 1 -- 172.21.15.26:0/3916567862 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f10ec009da0 con 0x7f10ec0090c0 2024-09-16T14:59:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421626043 0 0) 0x7f10ec009da0 con 0x7f10ec0090c0 2024-09-16T14:59:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 10 monclient(hunting): my global_id is 284618 2024-09-16T14:59:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10ec006590 con 0x7f10ec0090c0 2024-09-16T14:59:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 986104133 0 0) 0x7f10ec006590 con 0x7f10ec0090c0 2024-09-16T14:59:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10ec009da0 con 0x7f10ec0090c0 2024-09-16T14:59:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1100004780 con 0x7f10ec0090c0 2024-09-16T14:59:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 10 monclient: dump: 2024-09-16T14:59:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.635 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1943120784 0 0) 0x7f10ec009da0 con 0x7f10ec0090c0 2024-09-16T14:59:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f10ec00c290 con 0x7f10ec0090c0 2024-09-16T14:59:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4203 (unknown 0 0 1317682924) 0x7f1100005c20 con 0x7f10ec0090c0 2024-09-16T14:59:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:59:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:59:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 >> v1:172.21.15.26:6790/0 conn(0x7f10ec0090c0 legacy=0x7f10ec009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f10fa7fc640 1 -- 172.21.15.26:0/3916567862 >> v1:172.21.15.26:6790/0 conn(0x7f10ec0090c0 legacy=0x7f10ec009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:45.636 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:59:45.636 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:59:45.636 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:59:45.636 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T14:59:45.636 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 160, 2024-09-16T14:59:45.636 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:59:45.636 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:59:45.636 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:59:45.636 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:45.637 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:59:45.637 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:59:45.637 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:59:45.637 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:59:45.637 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:45.637 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:45.637 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:45.637 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:45.637 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:45.637 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:45.637 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:45.637 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:45.637 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:45.637 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:45.638 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:45.638 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:59:45.638 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:59:45.638 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:45.638 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:45.638 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:45.638 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:45.638 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:45.638 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:45.638 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:45.638 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:45.638 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:45.638 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:45.639 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:45.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.639 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:59:45.639 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:59:45.639 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:59:45.639 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:59:45.639 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:59:45.639 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:59:45.639 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:59:45.639 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:59:45.639 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:59:45.639 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:59:45.639 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:59:45.640 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:59:45.640 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:59:45.640 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:59:45.640 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:59:45.640 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:59:45.640 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:59:45.640 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:45.640 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:45.640 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:45.640 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:45.640 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:45.640 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:45.640 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:45.641 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:45.641 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:45.641 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:45.641 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:45.641 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:59:45.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.641 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:59:45.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.641 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:59:45.641 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:59:45.641 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:45.641 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:45.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.641 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:45.642 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:59:45.642 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:45.642 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:45.642 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:45.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.642 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:59:45.642 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:59:45.642 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:45.642 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:45.642 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:45.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.642 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:59:45.642 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:59:45.643 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:45.643 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:45.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.643 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:45.643 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:59:45.643 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:45.643 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:45.643 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:45.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.643 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:59:45.643 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:59:45.643 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:45.643 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:45.644 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:45.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.644 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:59:45.644 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:59:45.644 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:45.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:45.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.644 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:45.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:59:45.644 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:45.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:45.644 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:45.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:59:45.645 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:59:45.645 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:45.645 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:45.645 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:45.645 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:45.645 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:45.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.645 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:59:45.645 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:59:45.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.645 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:45.645 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:59:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T14:59:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:59:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:59:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:45.646 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:45.647 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T14:59:45.647 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:45.647 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:45.647 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T14:59:45.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.647 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:45.647 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:45.647 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:59:45.647 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:45.647 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:45.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.647 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:59:45.647 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:59:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f110bd6a640 1 -- 172.21.15.26:0/3916567862 >> v1:172.21.15.189:6812/3196161582 conn(0x7f10e8036040 legacy=0x7f10e8038500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f110bd6a640 10 monclient: shutdown 2024-09-16T14:59:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f110bd6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f110bd6a640 1 -- 172.21.15.26:0/3916567862 >> v1:172.21.15.26:6790/0 conn(0x7f110409e7a0 legacy=0x7f11041d1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f110a307640 1 -- 172.21.15.26:0/3916567862 reap_dead start 2024-09-16T14:59:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f110a307640 1 -- 172.21.15.26:0/3916567862 reap_dead start 2024-09-16T14:59:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f110bd6a640 1 -- 172.21.15.26:0/3916567862 shutdown_connections 2024-09-16T14:59:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f110bd6a640 1 -- 172.21.15.26:0/3916567862 >> 172.21.15.26:0/3916567862 conn(0x7f110409fb50 msgr2=0x7f1104130ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f110bd6a640 1 -- 172.21.15.26:0/3916567862 shutdown_connections 2024-09-16T14:59:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.618+0000 7f110bd6a640 1 -- 172.21.15.26:0/3916567862 wait complete. 2024-09-16T14:59:45.650 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:59:45.650 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T14:59:45.650 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T14:59:45.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 10 monclient: get_monmap_and_config 2024-09-16T14:59:45.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 10 monclient: build_initial_monmap 2024-09-16T14:59:45.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 10 monclient: monmap: 2024-09-16T14:59:45.757 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:45.757 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.757 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:59:45.758920+0000 2024-09-16T14:59:45.757 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:59:45.758920+0000 2024-09-16T14:59:45.757 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:45.757 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.757 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:45.757 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T14:59:45.757 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T14:59:45.757 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 1 Processor -- start 2024-09-16T14:59:45.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 1 -- start start 2024-09-16T14:59:45.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 10 monclient: init 2024-09-16T14:59:45.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:45.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T14:59:45.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 10 monclient(hunting): picked mon.noname-b con 0x7fdbb409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:45.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 10 monclient(hunting): picked mon.noname-a con 0x7fdbb4132650 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:45.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 10 monclient(hunting): picked mon.noname-c con 0x7fdbb4138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:45.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb409d260 con 0x7fdbb4132650 2024-09-16T14:59:45.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb4132a80 con 0x7fdbb4138ff0 2024-09-16T14:59:45.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb409cf60 con 0x7fdbb409cb80 2024-09-16T14:59:45.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:45.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 10 monclient(hunting): authenticate will time out at 2009.878769s 2024-09-16T14:59:45.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb97c4640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fdbb4132650 legacy=0x7fdbb41378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:45.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb9fc5640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fdbb409cb80 0x7fdbb40717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:42582/0 (socket says 172.21.15.26:42582) 2024-09-16T14:59:45.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb9fc5640 1 -- 172.21.15.26:0/3198453433 learned_addr learned my addr 172.21.15.26:0/3198453433 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:45.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 1 -- 172.21.15.26:0/3198453433 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068139333 0 0) 0x7fdbb409cf60 con 0x7fdbb409cb80 2024-09-16T14:59:45.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 10 monclient(hunting): _negotiate 2024-09-16T14:59:45.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 10 monclient(hunting): my global_id is 284621 2024-09-16T14:59:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 1 -- 172.21.15.26:0/3198453433 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba0003530 con 0x7fdbb409cb80 2024-09-16T14:59:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 1 -- 172.21.15.26:0/3198453433 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2482613627 0 0) 0x7fdba0003530 con 0x7fdbb409cb80 2024-09-16T14:59:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 1 -- 172.21.15.26:0/3198453433 >> v1:172.21.15.189:6789/0 conn(0x7fdbb4138ff0 legacy=0x7fdbb413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 1 -- 172.21.15.26:0/3198453433 >> v1:172.21.15.26:6789/0 conn(0x7fdbb4132650 legacy=0x7fdbb41378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T14:59:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 10 monclient: _finish_hunting 0 2024-09-16T14:59:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 1 monclient: found mon.noname-b 2024-09-16T14:59:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T14:59:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 1 -- 172.21.15.26:0/3198453433 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb409f2d0 con 0x7fdbb409cb80 2024-09-16T14:59:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 10 monclient: _finish_auth 0 2024-09-16T14:59:45.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 10 monclient: _check_auth_tickets 2024-09-16T14:59:45.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:45.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 1 -- 172.21.15.26:0/3198453433 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdbac0019e0 con 0x7fdbb409cb80 2024-09-16T14:59:45.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 5 monclient: authenticate success, global_id 284621 2024-09-16T14:59:45.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:45.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T14:59:45.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 10 monclient: dump: 2024-09-16T14:59:45.767 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.767 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.767 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.767 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.768 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.768 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.768 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.768 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.768 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.768 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T14:59:45.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 1 -- 172.21.15.26:0/3198453433 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbac001c40 con 0x7fdbb409cb80 2024-09-16T14:59:45.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:45.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 10 monclient: get_monmap_and_config success 2024-09-16T14:59:45.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbbc229640 4 set_mon_vals no callback set 2024-09-16T14:59:45.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 1 -- 172.21.15.26:0/3198453433 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdbac002010 con 0x7fdbb409cb80 2024-09-16T14:59:45.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.754+0000 7fdbb8fc3640 10 monclient: dump: 2024-09-16T14:59:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.769 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 10 monclient: shutdown 2024-09-16T14:59:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 1 -- 172.21.15.26:0/3198453433 >> v1:172.21.15.26:6790/0 conn(0x7fdbb409cb80 legacy=0x7fdbb40717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 1 -- 172.21.15.26:0/3198453433 shutdown_connections 2024-09-16T14:59:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 1 -- 172.21.15.26:0/3198453433 >> 172.21.15.26:0/3198453433 conn(0x7fdbb409fb50 msgr2=0x7fdbb409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 1 -- 172.21.15.26:0/3198453433 shutdown_connections 2024-09-16T14:59:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 1 -- 172.21.15.26:0/3198453433 wait complete. 2024-09-16T14:59:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 10 monclient: build_initial_monmap 2024-09-16T14:59:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 10 monclient: monmap: 2024-09-16T14:59:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T14:59:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T14:59:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T14:59:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T14:59:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T14:59:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T14:59:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T14:59:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T14:59:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T14:59:45.771 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 1 Processor -- start 2024-09-16T14:59:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 1 -- start start 2024-09-16T14:59:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 10 monclient: init 2024-09-16T14:59:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 10 monclient: _reopen_session rank -1 2024-09-16T14:59:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T14:59:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 10 monclient(hunting): picked mon.noname-c con 0x7fdbb409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T14:59:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 10 monclient(hunting): picked mon.noname-b con 0x7fdbb4132650 addr v1:172.21.15.189:6789/0 2024-09-16T14:59:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 10 monclient(hunting): picked mon.noname-a con 0x7fdbb4138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T14:59:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb41381a0 con 0x7fdbb4138ff0 2024-09-16T14:59:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb406e480 con 0x7fdbb4132650 2024-09-16T14:59:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbba7c6640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fdbb4138ff0 legacy=0x7fdbb41cfb60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-09-16T14:59:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb406fb60 con 0x7fdbb409cb80 2024-09-16T14:59:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 10 monclient(hunting): _renew_subs 2024-09-16T14:59:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 10 monclient(hunting): authenticate will time out at 2009.881621s 2024-09-16T14:59:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbb9fc5640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fdbb409cb80 0x7fdbb406d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:42598/0 (socket says 172.21.15.26:42598) 2024-09-16T14:59:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbb9fc5640 1 -- 172.21.15.26:0/3078837201 learned_addr learned my addr 172.21.15.26:0/3078837201 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T14:59:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067679293 0 0) 0x7fdbb406fb60 con 0x7fdbb409cb80 2024-09-16T14:59:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient(hunting): my global_id is 284624 2024-09-16T14:59:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb98003530 con 0x7fdbb409cb80 2024-09-16T14:59:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3921986999 0 0) 0x7fdb98003530 con 0x7fdbb409cb80 2024-09-16T14:59:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbb406fb60 con 0x7fdbb409cb80 2024-09-16T14:59:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdbac0019a0 con 0x7fdbb409cb80 2024-09-16T14:59:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T14:59:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient(hunting): dump: 2024-09-16T14:59:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.774 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T14:59:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162801626 0 0) 0x7fdbb406e480 con 0x7fdbb4132650 2024-09-16T14:59:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient(hunting): my global_id is 304357 2024-09-16T14:59:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb98003530 con 0x7fdbb4132650 2024-09-16T14:59:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1216897171 0 0) 0x7fdbb406fb60 con 0x7fdbb409cb80 2024-09-16T14:59:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 >> v1:172.21.15.189:6789/0 conn(0x7fdbb4132650 legacy=0x7fdbb406dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 >> v1:172.21.15.26:6789/0 conn(0x7fdbb4138ff0 legacy=0x7fdbb41cfb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T14:59:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient: _finish_hunting 0 2024-09-16T14:59:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 monclient: found mon.c 2024-09-16T14:59:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T14:59:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb40a29a0 con 0x7fdbb409cb80 2024-09-16T14:59:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient: _finish_auth 0 2024-09-16T14:59:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient: _check_auth_tickets 2024-09-16T14:59:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T14:59:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 5 monclient: authenticate success, global_id 284624 2024-09-16T14:59:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 10 monclient: _renew_subs 2024-09-16T14:59:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:45.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 1 -- 172.21.15.26:0/3078837201 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb409a240 con 0x7fdbb409cb80 2024-09-16T14:59:45.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 10 monclient: _renew_subs 2024-09-16T14:59:45.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T14:59:45.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 1 -- 172.21.15.26:0/3078837201 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb413c3b0 con 0x7fdbb409cb80 2024-09-16T14:59:45.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbac004e20 con 0x7fdbb409cb80 2024-09-16T14:59:45.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T14:59:45.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdbac005130 con 0x7fdbb409cb80 2024-09-16T14:59:45.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T14:59:45.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbabfff640 4 set_mon_vals no callback set 2024-09-16T14:59:45.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient: dump: 2024-09-16T14:59:45.776 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.777 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.777 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.777 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.777 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.777 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.777 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.777 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.777 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.777 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:45.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbbc229640 1 -- 172.21.15.26:0/3078837201 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdb9c006590 con 0x7fdb9c002830 2024-09-16T14:59:45.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fdbac00fba0 con 0x7fdbb409cb80 2024-09-16T14:59:45.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fdbac044360 con 0x7fdbb409cb80 2024-09-16T14:59:45.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbba7c6640 10 monclient: get_auth_request con 0x7fdb98036130 auth_method 0 2024-09-16T14:59:45.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917849321 0 0) 0x7fdb9c006590 con 0x7fdb9c002830 2024-09-16T14:59:45.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:45.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:45.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:45.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 10 monclient(hunting): my global_id is 304360 2024-09-16T14:59:45.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.758+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbb406fb60 con 0x7fdb9c002830 2024-09-16T14:59:45.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.762+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1919298387 0 0) 0x7fdbb406fb60 con 0x7fdb9c002830 2024-09-16T14:59:45.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.762+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb9c006590 con 0x7fdb9c002830 2024-09-16T14:59:45.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.762+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdba4004070 con 0x7fdb9c002830 2024-09-16T14:59:45.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.762+0000 7fdbaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.762+0000 7fdbaa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:59:45.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.762+0000 7fdbaa7fc640 10 monclient: dump: 2024-09-16T14:59:45.779 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.779 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.779 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.779 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.779 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.779 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.779 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.779 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.779 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.779 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.762+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3527452204 0 0) 0x7fdb9c006590 con 0x7fdb9c002830 2024-09-16T14:59:45.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.762+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb9c00a0e0 con 0x7fdb9c002830 2024-09-16T14:59:45.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.762+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fdba4006440 con 0x7fdb9c002830 2024-09-16T14:59:45.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.762+0000 7fdbaa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T14:59:45.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.762+0000 7fdbaa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T14:59:45.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.762+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 >> v1:172.21.15.189:6789/0 conn(0x7fdb9c002830 legacy=0x7fdb9c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.762+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 >> v1:172.21.15.189:6789/0 conn(0x7fdb9c002830 legacy=0x7fdb9c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.766+0000 7fdbbc229640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T14:59:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.766+0000 7fdbbc229640 1 -- 172.21.15.26:0/3078837201 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdb9c009d80 con 0x7fdb9c0090a0 2024-09-16T14:59:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.766+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198184486 0 0) 0x7fdb9c009d80 con 0x7fdb9c0090a0 2024-09-16T14:59:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.766+0000 7fdbaa7fc640 10 monclient(hunting): _negotiate 2024-09-16T14:59:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.766+0000 7fdbaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T14:59:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.766+0000 7fdbaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T14:59:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.766+0000 7fdbaa7fc640 10 monclient(hunting): my global_id is 304363 2024-09-16T14:59:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.766+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb9c006590 con 0x7fdb9c0090a0 2024-09-16T14:59:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 304742646 0 0) 0x7fdb9c006590 con 0x7fdb9c0090a0 2024-09-16T14:59:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb9c009d80 con 0x7fdb9c0090a0 2024-09-16T14:59:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fdba40048b0 con 0x7fdb9c0090a0 2024-09-16T14:59:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T14:59:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbaa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T14:59:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbaa7fc640 10 monclient: dump: 2024-09-16T14:59:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T14:59:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T14:59:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T14:59:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T14:59:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T14:59:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T14:59:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T14:59:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T14:59:45.783 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T14:59:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4037547701 0 0) 0x7fdb9c009d80 con 0x7fdb9c0090a0 2024-09-16T14:59:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdb9c00c290 con 0x7fdb9c0090a0 2024-09-16T14:59:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3869 (unknown 0 0 3176487378) 0x7fdba4005be0 con 0x7fdb9c0090a0 2024-09-16T14:59:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbaa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T14:59:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbaa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T14:59:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 >> v1:172.21.15.189:6789/0 conn(0x7fdb9c0090a0 legacy=0x7fdb9c009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbaa7fc640 1 -- 172.21.15.26:0/3078837201 >> v1:172.21.15.189:6789/0 conn(0x7fdb9c0090a0 legacy=0x7fdb9c009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T14:59:45.783 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T14:59:45.784 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:59:45.784 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:59:45.784 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T14:59:45.784 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 160, 2024-09-16T14:59:45.784 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T14:59:45.784 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T14:59:45.784 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T14:59:45.784 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:45.784 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T14:59:45.784 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:59:45.784 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T14:59:45.784 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T14:59:45.785 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:45.785 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:45.785 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:45.785 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:45.785 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:45.785 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:45.785 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:45.785 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:45.785 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:45.785 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:45.785 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:45.785 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T14:59:45.785 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T14:59:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.786 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T14:59:45.787 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T14:59:45.787 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T14:59:45.787 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T14:59:45.787 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T14:59:45.787 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T14:59:45.787 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:59:45.787 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T14:59:45.787 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T14:59:45.787 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T14:59:45.787 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T14:59:45.787 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T14:59:45.787 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T14:59:45.788 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T14:59:45.788 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T14:59:45.788 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T14:59:45.788 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T14:59:45.788 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T14:59:45.788 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T14:59:45.788 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T14:59:45.788 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T14:59:45.789 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T14:59:45.789 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T14:59:45.789 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T14:59:45.789 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T14:59:45.789 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T14:59:45.789 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T14:59:45.789 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:45.789 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T14:59:45.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.789 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T14:59:45.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.789 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T14:59:45.790 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T14:59:45.790 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:45.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:45.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.790 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:45.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T14:59:45.790 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:45.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:45.790 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:45.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T14:59:45.790 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T14:59:45.790 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:45.791 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:45.791 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:45.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.791 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T14:59:45.791 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T14:59:45.791 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:45.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:45.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.791 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:45.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T14:59:45.791 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:45.791 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:45.792 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:45.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.792 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T14:59:45.792 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T14:59:45.792 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:45.792 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:45.792 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:45.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.792 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T14:59:45.792 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T14:59:45.792 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T14:59:45.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T14:59:45.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.793 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T14:59:45.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T14:59:45.793 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T14:59:45.793 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:45.793 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:45.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T14:59:45.793 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T14:59:45.793 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T14:59:45.793 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T14:59:45.793 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T14:59:45.794 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:45.794 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:45.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.794 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T14:59:45.794 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T14:59:45.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.794 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:45.794 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:45.794 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T14:59:45.794 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:45.794 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T14:59:45.794 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T14:59:45.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T14:59:45.795 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T14:59:45.795 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T14:59:45.795 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T14:59:45.795 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T14:59:45.795 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T14:59:45.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T14:59:45.795 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T14:59:45.795 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T14:59:45.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbbc229640 1 -- 172.21.15.26:0/3078837201 >> v1:172.21.15.189:6812/3196161582 conn(0x7fdb98036130 legacy=0x7fdb980385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbbc229640 10 monclient: shutdown 2024-09-16T14:59:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbbc229640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T14:59:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbbc229640 1 -- 172.21.15.26:0/3078837201 >> v1:172.21.15.26:6790/0 conn(0x7fdbb409cb80 legacy=0x7fdbb406d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T14:59:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbba7c6640 1 -- 172.21.15.26:0/3078837201 reap_dead start 2024-09-16T14:59:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbba7c6640 1 -- 172.21.15.26:0/3078837201 reap_dead start 2024-09-16T14:59:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbbc229640 1 -- 172.21.15.26:0/3078837201 shutdown_connections 2024-09-16T14:59:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbbc229640 1 -- 172.21.15.26:0/3078837201 >> 172.21.15.26:0/3078837201 conn(0x7fdbb409fb50 msgr2=0x7fdbb4131ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T14:59:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbbc229640 1 -- 172.21.15.26:0/3078837201 shutdown_connections 2024-09-16T14:59:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T14:59:45.770+0000 7fdbbc229640 1 -- 172.21.15.26:0/3078837201 wait complete. 2024-09-16T14:59:45.797 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T14:59:45.798 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T15:00:05.798 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T15:00:05.799 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T15:00:05.799 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T15:00:05.800 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T15:00:05.803 INFO:tasks.ceph.mon.a:Started 2024-09-16T15:00:05.803 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T15:00:05.803 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T15:00:05.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.930+0000 7faef65a9640 10 monclient: get_monmap_and_config 2024-09-16T15:00:05.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.930+0000 7faef65a9640 10 monclient: build_initial_monmap 2024-09-16T15:00:05.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.930+0000 7faef65a9640 10 monclient: monmap: 2024-09-16T15:00:05.934 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:05.934 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:05.934 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:00:05.936318+0000 2024-09-16T15:00:05.934 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:00:05.936318+0000 2024-09-16T15:00:05.934 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:05.934 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:05.935 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:05.935 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:00:05.935 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:00:05.935 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:05.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.930+0000 7faef65a9640 1 Processor -- start 2024-09-16T15:00:05.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.930+0000 7faef65a9640 1 -- start start 2024-09-16T15:00:05.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.930+0000 7faef65a9640 10 monclient: init 2024-09-16T15:00:05.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:05.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:00:05.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient(hunting): picked mon.noname-a con 0x7faef00a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:05.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient(hunting): picked mon.noname-c con 0x7faef0099b60 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:05.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient(hunting): picked mon.noname-b con 0x7faef00a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:05.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef0098ea0 con 0x7faef00a1c00 2024-09-16T15:00:05.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef00997b0 con 0x7faef0099b60 2024-09-16T15:00:05.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef0099460 con 0x7faef00a26f0 2024-09-16T15:00:05.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:05.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient(hunting): authenticate will time out at 2030.056158s 2024-09-16T15:00:05.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef4b46640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7faef00a26f0 0x7faef013d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59744/0 (socket says 172.21.15.26:59744) 2024-09-16T15:00:05.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef4b46640 1 -- 172.21.15.26:0/2322410555 learned_addr learned my addr 172.21.15.26:0/2322410555 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:05.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 1 -- 172.21.15.26:0/2322410555 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643194868 0 0) 0x7faef0099460 con 0x7faef00a26f0 2024-09-16T15:00:05.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:00:05.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:05.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:05.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 10 monclient(hunting): my global_id is 284627 2024-09-16T15:00:05.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 1 -- 172.21.15.26:0/2322410555 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faedc003530 con 0x7faef00a26f0 2024-09-16T15:00:05.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 1 -- 172.21.15.26:0/2322410555 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4136021538 0 0) 0x7faedc003530 con 0x7faef00a26f0 2024-09-16T15:00:05.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 1 -- 172.21.15.26:0/2322410555 >> v1:172.21.15.189:6789/0 conn(0x7faef0099b60 legacy=0x7faef0139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:05.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 1 -- 172.21.15.26:0/2322410555 >> v1:172.21.15.26:6789/0 conn(0x7faef00a1c00 legacy=0x7faef00a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:05.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 10 monclient: _finish_hunting 0 2024-09-16T15:00:05.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 1 monclient: found mon.noname-b 2024-09-16T15:00:05.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:05.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:00:05.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 1 -- 172.21.15.26:0/2322410555 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef00a34b0 con 0x7faef00a26f0 2024-09-16T15:00:05.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 10 monclient: _finish_auth 0 2024-09-16T15:00:05.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 10 monclient: _check_auth_tickets 2024-09-16T15:00:05.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:05.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 1 -- 172.21.15.26:0/2322410555 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faee80019a0 con 0x7faef00a26f0 2024-09-16T15:00:05.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 5 monclient: authenticate success, global_id 284627 2024-09-16T15:00:05.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:05.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:05.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:00:05.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 10 monclient: dump: 2024-09-16T15:00:05.941 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:05.941 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:05.941 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:05.941 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:05.941 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:05.941 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:05.941 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:05.942 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:05.942 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:05.942 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:05.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:05.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 1 -- 172.21.15.26:0/2322410555 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faee8001c50 con 0x7faef00a26f0 2024-09-16T15:00:05.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:05.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 1 -- 172.21.15.26:0/2322410555 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faee8002020 con 0x7faef00a26f0 2024-09-16T15:00:05.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:05.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:05.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faeef7fe640 10 monclient: dump: 2024-09-16T15:00:05.942 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:05.943 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:05.943 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:05.943 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:05.943 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:05.943 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:05.943 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:05.943 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:05.943 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:05.943 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:05.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient: get_monmap_and_config success 2024-09-16T15:00:05.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 4 set_mon_vals no callback set 2024-09-16T15:00:05.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient: shutdown 2024-09-16T15:00:05.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:05.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 1 -- 172.21.15.26:0/2322410555 >> v1:172.21.15.26:6790/0 conn(0x7faef00a26f0 legacy=0x7faef013d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:05.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 1 -- 172.21.15.26:0/2322410555 shutdown_connections 2024-09-16T15:00:05.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 1 -- 172.21.15.26:0/2322410555 >> 172.21.15.26:0/2322410555 conn(0x7faef00a3d00 msgr2=0x7faef00a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:05.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 1 -- 172.21.15.26:0/2322410555 shutdown_connections 2024-09-16T15:00:05.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 1 -- 172.21.15.26:0/2322410555 wait complete. 2024-09-16T15:00:05.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient: build_initial_monmap 2024-09-16T15:00:05.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient: monmap: 2024-09-16T15:00:05.944 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:05.944 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:00:05.944 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:00:05.944 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:00:05.944 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:05.945 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:00:05.945 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:05.945 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:00:05.945 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:00:05.945 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 1 Processor -- start 2024-09-16T15:00:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 1 -- start start 2024-09-16T15:00:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient: init 2024-09-16T15:00:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:00:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient(hunting): picked mon.noname-c con 0x7faef0099b60 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient(hunting): picked mon.noname-b con 0x7faef00a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.934+0000 7faef65a9640 10 monclient(hunting): picked mon.noname-a con 0x7faef00a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faef65a9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef011e900 con 0x7faef00a26f0 2024-09-16T15:00:05.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faef65a9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef013df80 con 0x7faef00a1c00 2024-09-16T15:00:05.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faef65a9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef009ed90 con 0x7faef0099b60 2024-09-16T15:00:05.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeeffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7faef0099b60 0x7faef013c0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59756/0 (socket says 172.21.15.26:59756) 2024-09-16T15:00:05.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeeffff640 1 -- 172.21.15.26:0/703535541 learned_addr learned my addr 172.21.15.26:0/703535541 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:05.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faef65a9640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:05.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faef65a9640 10 monclient(hunting): authenticate will time out at 2030.060133s 2024-09-16T15:00:05.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903926080 0 0) 0x7faef009ed90 con 0x7faef0099b60 2024-09-16T15:00:05.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient(hunting): _negotiate 2024-09-16T15:00:05.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:05.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:05.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient(hunting): my global_id is 284630 2024-09-16T15:00:05.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faed0003530 con 0x7faef0099b60 2024-09-16T15:00:05.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627020650 0 0) 0x7faef011e900 con 0x7faef00a26f0 2024-09-16T15:00:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient(hunting): _negotiate 2024-09-16T15:00:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient(hunting): my global_id is 334098 2024-09-16T15:00:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faef009ed90 con 0x7faef00a26f0 2024-09-16T15:00:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2811348720 0 0) 0x7faef009ed90 con 0x7faef00a26f0 2024-09-16T15:00:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faef011e900 con 0x7faef00a26f0 2024-09-16T15:00:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faee8001960 con 0x7faef00a26f0 2024-09-16T15:00:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:00:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient(hunting): dump: 2024-09-16T15:00:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:05.952 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:05.952 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:05.952 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:05.952 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:05.952 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:05.952 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:05.952 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:05.952 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:05.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:00:05.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 700212772 0 0) 0x7faed0003530 con 0x7faef0099b60 2024-09-16T15:00:05.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faef009ed90 con 0x7faef0099b60 2024-09-16T15:00:05.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faee00018c0 con 0x7faef0099b60 2024-09-16T15:00:05.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:05.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:05.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient(hunting): dump: 2024-09-16T15:00:05.953 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:05.953 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:05.953 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:05.953 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:05.953 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:05.953 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:05.953 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:05.953 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:05.953 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:05.953 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:05.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3694464185 0 0) 0x7faef011e900 con 0x7faef00a26f0 2024-09-16T15:00:05.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 >> v1:172.21.15.26:6790/0 conn(0x7faef0099b60 legacy=0x7faef013c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:05.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 >> v1:172.21.15.189:6789/0 conn(0x7faef00a1c00 legacy=0x7faef013c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:05.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient: _finish_hunting 0 2024-09-16T15:00:05.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 monclient: found mon.a 2024-09-16T15:00:05.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:05.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:05.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef009f2e0 con 0x7faef00a26f0 2024-09-16T15:00:05.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient: _finish_auth 0 2024-09-16T15:00:05.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient: _check_auth_tickets 2024-09-16T15:00:05.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:05.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faef65a9640 5 monclient: authenticate success, global_id 334098 2024-09-16T15:00:05.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faef65a9640 10 monclient: _renew_subs 2024-09-16T15:00:05.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faef65a9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:05.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faef65a9640 1 -- 172.21.15.26:0/703535541 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef011e620 con 0x7faef00a26f0 2024-09-16T15:00:05.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faef65a9640 10 monclient: _renew_subs 2024-09-16T15:00:05.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faef65a9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:05.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faef65a9640 1 -- 172.21.15.26:0/703535541 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faef013e350 con 0x7faef00a26f0 2024-09-16T15:00:05.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faee8001510 con 0x7faef00a26f0 2024-09-16T15:00:05.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:05.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faee8005120 con 0x7faef00a26f0 2024-09-16T15:00:05.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:05.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:05.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeeeffd640 4 set_mon_vals no callback set 2024-09-16T15:00:05.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 10 monclient: dump: 2024-09-16T15:00:05.955 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:05.955 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:05.956 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7faee800fb90 con 0x7faef00a26f0 2024-09-16T15:00:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7faee80441e0 con 0x7faef00a26f0 2024-09-16T15:00:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faef65a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faef65a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faef65a9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faef65a9640 1 -- 172.21.15.26:0/703535541 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed4005050 con 0x7faef00a26f0 2024-09-16T15:00:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.938+0000 7faeeffff640 10 monclient: get_auth_request con 0x7faed0036200 auth_method 0 2024-09-16T15:00:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.942+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7faee8014950 con 0x7faef00a26f0 2024-09-16T15:00:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.942+0000 7faeed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:05.942+0000 7faeed7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:00:06.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.166+0000 7faef65a9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T15:00:06.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.166+0000 7faef65a9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T15:00:06.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.166+0000 7faef65a9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:06.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.166+0000 7faef65a9640 1 -- 172.21.15.26:0/703535541 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7faed4005880 con 0x7faef00a26f0 2024-09-16T15:00:06.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.166+0000 7faeed7fa640 1 -- 172.21.15.26:0/703535541 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1929827405) 0x7faee8012ff0 con 0x7faef00a26f0 2024-09-16T15:00:06.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.166+0000 7faeed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T15:00:06.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.166+0000 7faeed7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:00:06.168 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T15:00:06.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.166+0000 7faef65a9640 1 -- 172.21.15.26:0/703535541 >> v1:172.21.15.189:6812/3196161582 conn(0x7faed0036200 legacy=0x7faed00386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.166+0000 7faef65a9640 10 monclient: shutdown 2024-09-16T15:00:06.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.166+0000 7faef65a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:06.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.166+0000 7faef65a9640 1 -- 172.21.15.26:0/703535541 >> v1:172.21.15.26:6789/0 conn(0x7faef00a26f0 legacy=0x7faef01d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.166+0000 7faef65a9640 1 -- 172.21.15.26:0/703535541 shutdown_connections 2024-09-16T15:00:06.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.166+0000 7faef65a9640 1 -- 172.21.15.26:0/703535541 >> 172.21.15.26:0/703535541 conn(0x7faef00a3d00 msgr2=0x7faef006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:06.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.166+0000 7faef65a9640 1 -- 172.21.15.26:0/703535541 shutdown_connections 2024-09-16T15:00:06.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.166+0000 7faef65a9640 1 -- 172.21.15.26:0/703535541 wait complete. 2024-09-16T15:00:06.186 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T15:00:06.187 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T15:00:06.187 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T15:00:06.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.314+0000 7f0872ef4640 10 monclient: get_monmap_and_config 2024-09-16T15:00:06.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.314+0000 7f0872ef4640 10 monclient: build_initial_monmap 2024-09-16T15:00:06.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.314+0000 7f0872ef4640 10 monclient: monmap: 2024-09-16T15:00:06.319 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:06.319 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.319 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:00:06.320915+0000 2024-09-16T15:00:06.319 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:00:06.320915+0000 2024-09-16T15:00:06.319 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:06.319 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.319 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:06.319 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:00:06.319 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:00:06.319 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 1 Processor -- start 2024-09-16T15:00:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 1 -- start start 2024-09-16T15:00:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 10 monclient: init 2024-09-16T15:00:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:00:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 10 monclient(hunting): picked mon.noname-c con 0x7f086c09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 10 monclient(hunting): picked mon.noname-b con 0x7f086c132650 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 10 monclient(hunting): picked mon.noname-a con 0x7f086c138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086c09d260 con 0x7f086c138ff0 2024-09-16T15:00:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086c132a80 con 0x7f086c09cb80 2024-09-16T15:00:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086c09cf60 con 0x7f086c132650 2024-09-16T15:00:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:06.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 10 monclient(hunting): authenticate will time out at 2030.441239s 2024-09-16T15:00:06.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086bfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f086c132650 0x7f086c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59772/0 (socket says 172.21.15.26:59772) 2024-09-16T15:00:06.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0871491640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f086c138ff0 0x7f086c13b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50030/0 (socket says 172.21.15.26:50030) 2024-09-16T15:00:06.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086bfff640 1 -- 172.21.15.26:0/2367222047 learned_addr learned my addr 172.21.15.26:0/2367222047 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:06.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 1 -- 172.21.15.26:0/2367222047 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102494774 0 0) 0x7f086c09cf60 con 0x7f086c132650 2024-09-16T15:00:06.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient(hunting): my global_id is 284633 2024-09-16T15:00:06.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 1 -- 172.21.15.26:0/2367222047 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0854003530 con 0x7f086c132650 2024-09-16T15:00:06.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 1 -- 172.21.15.26:0/2367222047 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155468858 0 0) 0x7f086c09d260 con 0x7f086c138ff0 2024-09-16T15:00:06.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient(hunting): my global_id is 334101 2024-09-16T15:00:06.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 1 -- 172.21.15.26:0/2367222047 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f086c09cf60 con 0x7f086c138ff0 2024-09-16T15:00:06.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 1 -- 172.21.15.26:0/2367222047 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543691009 0 0) 0x7f086c132a80 con 0x7f086c09cb80 2024-09-16T15:00:06.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient(hunting): my global_id is 304366 2024-09-16T15:00:06.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 1 -- 172.21.15.26:0/2367222047 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f086c09d260 con 0x7f086c09cb80 2024-09-16T15:00:06.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 1 -- 172.21.15.26:0/2367222047 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3776475976 0 0) 0x7f0854003530 con 0x7f086c132650 2024-09-16T15:00:06.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 1 -- 172.21.15.26:0/2367222047 >> v1:172.21.15.189:6789/0 conn(0x7f086c09cb80 legacy=0x7f086c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 1 -- 172.21.15.26:0/2367222047 >> v1:172.21.15.26:6789/0 conn(0x7f086c138ff0 legacy=0x7f086c13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient: _finish_hunting 0 2024-09-16T15:00:06.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 1 monclient: found mon.noname-b 2024-09-16T15:00:06.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:06.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:00:06.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 1 -- 172.21.15.26:0/2367222047 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c09f2d0 con 0x7f086c132650 2024-09-16T15:00:06.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient: _finish_auth 0 2024-09-16T15:00:06.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient: _check_auth_tickets 2024-09-16T15:00:06.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:06.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 1 -- 172.21.15.26:0/2367222047 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0860001660 con 0x7f086c132650 2024-09-16T15:00:06.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:00:06.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient: dump: 2024-09-16T15:00:06.326 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.326 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.326 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.327 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.327 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.327 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.327 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.327 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.327 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.327 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 5 monclient: authenticate success, global_id 284633 2024-09-16T15:00:06.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:06.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 1 -- 172.21.15.26:0/2367222047 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0860001800 con 0x7f086c132650 2024-09-16T15:00:06.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:06.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 1 -- 172.21.15.26:0/2367222047 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0860001bd0 con 0x7f086c132650 2024-09-16T15:00:06.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:06.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f086b7fe640 10 monclient: dump: 2024-09-16T15:00:06.328 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.328 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.328 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.328 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.328 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.328 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.328 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.328 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.328 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.328 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 10 monclient: get_monmap_and_config success 2024-09-16T15:00:06.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 4 set_mon_vals no callback set 2024-09-16T15:00:06.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 10 monclient: shutdown 2024-09-16T15:00:06.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:06.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 1 -- 172.21.15.26:0/2367222047 >> v1:172.21.15.26:6790/0 conn(0x7f086c132650 legacy=0x7f086c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 1 -- 172.21.15.26:0/2367222047 shutdown_connections 2024-09-16T15:00:06.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 1 -- 172.21.15.26:0/2367222047 >> 172.21.15.26:0/2367222047 conn(0x7f086c09fb50 msgr2=0x7f086c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:06.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 1 -- 172.21.15.26:0/2367222047 shutdown_connections 2024-09-16T15:00:06.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 1 -- 172.21.15.26:0/2367222047 wait complete. 2024-09-16T15:00:06.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 10 monclient: build_initial_monmap 2024-09-16T15:00:06.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.318+0000 7f0872ef4640 10 monclient: monmap: 2024-09-16T15:00:06.329 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:06.330 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:00:06.330 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:00:06.330 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:00:06.330 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:06.330 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:00:06.330 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:06.330 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:00:06.330 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:00:06.330 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 1 Processor -- start 2024-09-16T15:00:06.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 1 -- start start 2024-09-16T15:00:06.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 10 monclient: init 2024-09-16T15:00:06.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:06.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:00:06.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 10 monclient(hunting): picked mon.noname-a con 0x7f086c09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:06.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 10 monclient(hunting): picked mon.noname-c con 0x7f086c132650 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:06.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 10 monclient(hunting): picked mon.noname-b con 0x7f086c138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:06.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086c0a28f0 con 0x7f086c09cb80 2024-09-16T15:00:06.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086c071480 con 0x7f086c138ff0 2024-09-16T15:00:06.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086c06f400 con 0x7f086c132650 2024-09-16T15:00:06.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:06.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 10 monclient(hunting): authenticate will time out at 2030.444608s 2024-09-16T15:00:06.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f086bfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f086c132650 0x7f086c06dec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59778/0 (socket says 172.21.15.26:59778) 2024-09-16T15:00:06.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f086bfff640 1 -- 172.21.15.26:0/2665851493 learned_addr learned my addr 172.21.15.26:0/2665851493 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:06.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152923949 0 0) 0x7f086c0a28f0 con 0x7f086c09cb80 2024-09-16T15:00:06.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): my global_id is 334104 2024-09-16T15:00:06.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0858003530 con 0x7f086c09cb80 2024-09-16T15:00:06.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824699267 0 0) 0x7f086c06f400 con 0x7f086c132650 2024-09-16T15:00:06.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): my global_id is 284636 2024-09-16T15:00:06.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f086c0a28f0 con 0x7f086c132650 2024-09-16T15:00:06.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3523616664 0 0) 0x7f0858003530 con 0x7f086c09cb80 2024-09-16T15:00:06.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f086c06f400 con 0x7f086c09cb80 2024-09-16T15:00:06.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f085c001820 con 0x7f086c09cb80 2024-09-16T15:00:06.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:00:06.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): dump: 2024-09-16T15:00:06.338 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.338 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.338 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.338 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.338 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.338 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.338 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.338 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.339 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:00:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3293109627 0 0) 0x7f086c0a28f0 con 0x7f086c132650 2024-09-16T15:00:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0858003530 con 0x7f086c132650 2024-09-16T15:00:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0860001b10 con 0x7f086c132650 2024-09-16T15:00:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): dump: 2024-09-16T15:00:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.340 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236505134 0 0) 0x7f086c071480 con 0x7f086c138ff0 2024-09-16T15:00:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): my global_id is 304369 2024-09-16T15:00:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f086c0a28f0 con 0x7f086c138ff0 2024-09-16T15:00:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2962328282 0 0) 0x7f086c06f400 con 0x7f086c09cb80 2024-09-16T15:00:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 >> v1:172.21.15.26:6790/0 conn(0x7f086c132650 legacy=0x7f086c06dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 >> v1:172.21.15.189:6789/0 conn(0x7f086c138ff0 legacy=0x7f086c06e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient: _finish_hunting 0 2024-09-16T15:00:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 monclient: found mon.a 2024-09-16T15:00:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c13c400 con 0x7f086c09cb80 2024-09-16T15:00:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient: _finish_auth 0 2024-09-16T15:00:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient: _check_auth_tickets 2024-09-16T15:00:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 5 monclient: authenticate success, global_id 334104 2024-09-16T15:00:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 10 monclient: _renew_subs 2024-09-16T15:00:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:06.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 1 -- 172.21.15.26:0/2665851493 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f086c09a240 con 0x7f086c09cb80 2024-09-16T15:00:06.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 10 monclient: _renew_subs 2024-09-16T15:00:06.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:06.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 1 -- 172.21.15.26:0/2665851493 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f086c0701f0 con 0x7f086c09cb80 2024-09-16T15:00:06.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f085c003620 con 0x7f086c09cb80 2024-09-16T15:00:06.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:06.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f085c003930 con 0x7f086c09cb80 2024-09-16T15:00:06.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:06.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient: dump: 2024-09-16T15:00:06.342 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.342 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.342 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.342 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.343 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.343 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.343 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.343 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.343 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.343 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f086affd640 4 set_mon_vals no callback set 2024-09-16T15:00:06.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f085c00e3a0 con 0x7f086c09cb80 2024-09-16T15:00:06.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:06.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0872ef4640 1 -- 172.21.15.26:0/2665851493 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f084c006590 con 0x7f084c002830 2024-09-16T15:00:06.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f085c042980 con 0x7f086c09cb80 2024-09-16T15:00:06.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108587224 0 0) 0x7f084c006590 con 0x7f084c002830 2024-09-16T15:00:06.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient(hunting): my global_id is 334107 2024-09-16T15:00:06.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f086c06f400 con 0x7f084c002830 2024-09-16T15:00:06.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1989679370 0 0) 0x7f086c06f400 con 0x7f084c002830 2024-09-16T15:00:06.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f084c006590 con 0x7f084c002830 2024-09-16T15:00:06.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0860004f30 con 0x7f084c002830 2024-09-16T15:00:06.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f0871491640 10 monclient: get_auth_request con 0x7f0858036390 auth_method 0 2024-09-16T15:00:06.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:06.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient: dump: 2024-09-16T15:00:06.344 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.345 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.345 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.345 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.345 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.345 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.345 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.345 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.345 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.345 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3127685081 0 0) 0x7f084c006590 con 0x7f084c002830 2024-09-16T15:00:06.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f084c00a0e0 con 0x7f084c002830 2024-09-16T15:00:06.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0860007240 con 0x7f084c002830 2024-09-16T15:00:06.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:06.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:00:06.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 >> v1:172.21.15.26:6789/0 conn(0x7f084c002830 legacy=0x7f084c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.322+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 >> v1:172.21.15.26:6789/0 conn(0x7f084c002830 legacy=0x7f084c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:00:06.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f0872ef4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:00:06.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f0872ef4640 1 -- 172.21.15.26:0/2665851493 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f084c009da0 con 0x7f084c0090c0 2024-09-16T15:00:06.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973750113 0 0) 0x7f084c009da0 con 0x7f084c0090c0 2024-09-16T15:00:06.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 10 monclient(hunting): my global_id is 334110 2024-09-16T15:00:06.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f084c006590 con 0x7f084c0090c0 2024-09-16T15:00:06.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3650575244 0 0) 0x7f084c006590 con 0x7f084c0090c0 2024-09-16T15:00:06.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f084c009da0 con 0x7f084c0090c0 2024-09-16T15:00:06.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0860005730 con 0x7f084c0090c0 2024-09-16T15:00:06.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:06.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 10 monclient: dump: 2024-09-16T15:00:06.347 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.347 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.347 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.347 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.347 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.347 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.347 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.347 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.347 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.347 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2451955225 0 0) 0x7f084c009da0 con 0x7f084c0090c0 2024-09-16T15:00:06.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f084c00c290 con 0x7f084c0090c0 2024-09-16T15:00:06.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 1255515583) 0x7f0860006a20 con 0x7f084c0090c0 2024-09-16T15:00:06.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:00:06.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:00:06.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 >> v1:172.21.15.26:6789/0 conn(0x7f084c0090c0 legacy=0x7f084c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f08697fa640 1 -- 172.21.15.26:0/2665851493 >> v1:172.21.15.26:6789/0 conn(0x7f084c0090c0 legacy=0x7f084c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:00:06.349 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:00:06.349 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:00:06.349 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:00:06.349 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:00:06.349 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 162, 2024-09-16T15:00:06.349 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:00:06.349 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:00:06.349 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:00:06.349 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:00:06.349 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:06.349 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:00:06.349 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:00:06.350 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:00:06.350 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:00:06.350 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:06.350 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:06.350 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:06.350 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:06.350 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:06.350 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:06.350 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:06.350 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:06.350 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:06.350 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:06.350 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:06.350 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:00:06.351 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:00:06.351 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:06.351 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:06.351 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:06.351 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:06.351 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:06.351 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:06.351 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:06.351 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:06.351 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:06.351 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:06.351 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:00:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:00:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:00:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:00:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:00:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:00:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:00:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:00:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:00:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:00:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:00:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:00:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:00:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:00:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:00:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:00:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:00:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:00:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:00:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:00:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:00:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:00:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:00:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:00:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:00:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:00:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:00:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:00:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:00:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:00:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:00:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:00:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:00:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:00:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:00:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:00:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:00:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:00:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T15:00:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:00:06.359 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:00:06.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f0872ef4640 1 -- 172.21.15.26:0/2665851493 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0858036390 legacy=0x7f0858038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f0872ef4640 10 monclient: shutdown 2024-09-16T15:00:06.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f0872ef4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:06.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f0872ef4640 1 -- 172.21.15.26:0/2665851493 >> v1:172.21.15.26:6789/0 conn(0x7f086c09cb80 legacy=0x7f086c06d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f0871491640 1 -- 172.21.15.26:0/2665851493 reap_dead start 2024-09-16T15:00:06.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f0871491640 1 -- 172.21.15.26:0/2665851493 reap_dead start 2024-09-16T15:00:06.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f0872ef4640 1 -- 172.21.15.26:0/2665851493 shutdown_connections 2024-09-16T15:00:06.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f0872ef4640 1 -- 172.21.15.26:0/2665851493 >> 172.21.15.26:0/2665851493 conn(0x7f086c09fb50 msgr2=0x7f086c131bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:06.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f0872ef4640 1 -- 172.21.15.26:0/2665851493 shutdown_connections 2024-09-16T15:00:06.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.330+0000 7f0872ef4640 1 -- 172.21.15.26:0/2665851493 wait complete. 2024-09-16T15:00:06.361 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:00:06.362 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T15:00:06.362 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T15:00:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.474+0000 7fb9d703b640 10 monclient: get_monmap_and_config 2024-09-16T15:00:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.474+0000 7fb9d703b640 10 monclient: build_initial_monmap 2024-09-16T15:00:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.474+0000 7fb9d703b640 10 monclient: monmap: 2024-09-16T15:00:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:00:06.481372+0000 2024-09-16T15:00:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:00:06.481372+0000 2024-09-16T15:00:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.480 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:06.480 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:00:06.480 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:00:06.480 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 1 Processor -- start 2024-09-16T15:00:06.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 1 -- start start 2024-09-16T15:00:06.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 10 monclient: init 2024-09-16T15:00:06.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:06.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:00:06.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 10 monclient(hunting): picked mon.noname-a con 0x7fb9d009a550 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:06.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 10 monclient(hunting): picked mon.noname-c con 0x7fb9d009d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:06.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 10 monclient(hunting): picked mon.noname-b con 0x7fb9d009b040 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:06.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d009dc60 con 0x7fb9d009a550 2024-09-16T15:00:06.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d0099ae0 con 0x7fb9d009d1b0 2024-09-16T15:00:06.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d009b8f0 con 0x7fb9d009b040 2024-09-16T15:00:06.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:06.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 10 monclient(hunting): authenticate will time out at 2030.601376s 2024-09-16T15:00:06.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d4dd7640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb9d009a550 0x7fb9d009a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50082/0 (socket says 172.21.15.26:50082) 2024-09-16T15:00:06.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d4dd7640 1 -- 172.21.15.26:0/2437722743 learned_addr learned my addr 172.21.15.26:0/2437722743 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:06.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 1 -- 172.21.15.26:0/2437722743 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153902477 0 0) 0x7fb9d009dc60 con 0x7fb9d009a550 2024-09-16T15:00:06.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient(hunting): my global_id is 334113 2024-09-16T15:00:06.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 1 -- 172.21.15.26:0/2437722743 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b8003530 con 0x7fb9d009a550 2024-09-16T15:00:06.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 1 -- 172.21.15.26:0/2437722743 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843039129 0 0) 0x7fb9d009b8f0 con 0x7fb9d009b040 2024-09-16T15:00:06.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient(hunting): my global_id is 284639 2024-09-16T15:00:06.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 1 -- 172.21.15.26:0/2437722743 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d009dc60 con 0x7fb9d009b040 2024-09-16T15:00:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 1 -- 172.21.15.26:0/2437722743 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 861086119 0 0) 0x7fb9b8003530 con 0x7fb9d009a550 2024-09-16T15:00:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 1 -- 172.21.15.26:0/2437722743 >> v1:172.21.15.26:6790/0 conn(0x7fb9d009b040 legacy=0x7fb9d013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 1 -- 172.21.15.26:0/2437722743 >> v1:172.21.15.189:6789/0 conn(0x7fb9d009d1b0 legacy=0x7fb9d0136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient: _finish_hunting 0 2024-09-16T15:00:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 1 monclient: found mon.noname-a 2024-09-16T15:00:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:00:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 1 -- 172.21.15.26:0/2437722743 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d009bbd0 con 0x7fb9d009a550 2024-09-16T15:00:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient: _finish_auth 0 2024-09-16T15:00:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient: _check_auth_tickets 2024-09-16T15:00:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 1 -- 172.21.15.26:0/2437722743 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9c0001620 con 0x7fb9d009a550 2024-09-16T15:00:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 5 monclient: authenticate success, global_id 334113 2024-09-16T15:00:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:00:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient: dump: 2024-09-16T15:00:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.487 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 1 -- 172.21.15.26:0/2437722743 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9c0001900 con 0x7fb9d009a550 2024-09-16T15:00:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 1 -- 172.21.15.26:0/2437722743 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9c0001cd0 con 0x7fb9d009a550 2024-09-16T15:00:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9cf7fe640 10 monclient: dump: 2024-09-16T15:00:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.488 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 10 monclient: get_monmap_and_config success 2024-09-16T15:00:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 4 set_mon_vals no callback set 2024-09-16T15:00:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 10 monclient: shutdown 2024-09-16T15:00:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 1 -- 172.21.15.26:0/2437722743 >> v1:172.21.15.26:6789/0 conn(0x7fb9d009a550 legacy=0x7fb9d009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 1 -- 172.21.15.26:0/2437722743 shutdown_connections 2024-09-16T15:00:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 1 -- 172.21.15.26:0/2437722743 >> 172.21.15.26:0/2437722743 conn(0x7fb9d009fb50 msgr2=0x7fb9d009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 1 -- 172.21.15.26:0/2437722743 shutdown_connections 2024-09-16T15:00:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 1 -- 172.21.15.26:0/2437722743 wait complete. 2024-09-16T15:00:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 10 monclient: build_initial_monmap 2024-09-16T15:00:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.478+0000 7fb9d703b640 10 monclient: monmap: 2024-09-16T15:00:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:00:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:00:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:00:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:00:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:00:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:00:06.490 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 1 Processor -- start 2024-09-16T15:00:06.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 1 -- start start 2024-09-16T15:00:06.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 10 monclient: init 2024-09-16T15:00:06.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:06.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:00:06.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 10 monclient(hunting): picked mon.noname-a con 0x7fb9d009a550 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:06.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 10 monclient(hunting): picked mon.noname-b con 0x7fb9d009b040 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:06.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 10 monclient(hunting): picked mon.noname-c con 0x7fb9d009d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:06.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d00a2060 con 0x7fb9d009a550 2024-09-16T15:00:06.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d009a240 con 0x7fb9d009b040 2024-09-16T15:00:06.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d013dcd0 con 0x7fb9d009d1b0 2024-09-16T15:00:06.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:06.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 10 monclient(hunting): authenticate will time out at 2030.604869s 2024-09-16T15:00:06.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d55d8640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb9d009d1b0 0x7fb9d01d1800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59794/0 (socket says 172.21.15.26:59794) 2024-09-16T15:00:06.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d55d8640 1 -- 172.21.15.26:0/4093499092 learned_addr learned my addr 172.21.15.26:0/4093499092 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:06.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50369250 0 0) 0x7fb9d013dcd0 con 0x7fb9d009d1b0 2024-09-16T15:00:06.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): my global_id is 284642 2024-09-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9bc003530 con 0x7fb9d009d1b0 2024-09-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397022284 0 0) 0x7fb9d00a2060 con 0x7fb9d009a550 2024-09-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): my global_id is 334116 2024-09-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d013dcd0 con 0x7fb9d009a550 2024-09-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2146412277 0 0) 0x7fb9bc003530 con 0x7fb9d009d1b0 2024-09-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9d00a2060 con 0x7fb9d009d1b0 2024-09-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9c8001d60 con 0x7fb9d009d1b0 2024-09-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:00:06.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): dump: 2024-09-16T15:00:06.499 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.499 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.499 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.499 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.499 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.499 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.499 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.499 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.499 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.499 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:00:06.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3296547990 0 0) 0x7fb9d013dcd0 con 0x7fb9d009a550 2024-09-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9bc003530 con 0x7fb9d009a550 2024-09-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9c0001c10 con 0x7fb9d009a550 2024-09-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): dump: 2024-09-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.500 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.501 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.501 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258525710 0 0) 0x7fb9d009a240 con 0x7fb9d009b040 2024-09-16T15:00:06.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): my global_id is 304375 2024-09-16T15:00:06.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d013dcd0 con 0x7fb9d009b040 2024-09-16T15:00:06.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 522695816 0 0) 0x7fb9d00a2060 con 0x7fb9d009d1b0 2024-09-16T15:00:06.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 >> v1:172.21.15.189:6789/0 conn(0x7fb9d009b040 legacy=0x7fb9d01ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 >> v1:172.21.15.26:6789/0 conn(0x7fb9d009a550 legacy=0x7fb9d0140190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient: _finish_hunting 0 2024-09-16T15:00:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 monclient: found mon.c 2024-09-16T15:00:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:00:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d01408a0 con 0x7fb9d009d1b0 2024-09-16T15:00:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient: _finish_auth 0 2024-09-16T15:00:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient: _check_auth_tickets 2024-09-16T15:00:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 5 monclient: authenticate success, global_id 284642 2024-09-16T15:00:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 10 monclient: _renew_subs 2024-09-16T15:00:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:00:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 1 -- 172.21.15.26:0/4093499092 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d013e3c0 con 0x7fb9d009d1b0 2024-09-16T15:00:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 10 monclient: _renew_subs 2024-09-16T15:00:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:00:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 1 -- 172.21.15.26:0/4093499092 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d013d4d0 con 0x7fb9d009d1b0 2024-09-16T15:00:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9c8003980 con 0x7fb9d009d1b0 2024-09-16T15:00:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9c8003c90 con 0x7fb9d009d1b0 2024-09-16T15:00:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9ceffd640 4 set_mon_vals no callback set 2024-09-16T15:00:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient: dump: 2024-09-16T15:00:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9d703b640 1 -- 172.21.15.26:0/4093499092 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb9b0006590 con 0x7fb9b0002830 2024-09-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb9c8003df0 con 0x7fb9d009d1b0 2024-09-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299730674 0 0) 0x7fb9b0006590 con 0x7fb9b0002830 2024-09-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient(hunting): my global_id is 284645 2024-09-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d00a2060 con 0x7fb9b0002830 2024-09-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fb9c8043f60 con 0x7fb9d009d1b0 2024-09-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2441937037 0 0) 0x7fb9d00a2060 con 0x7fb9b0002830 2024-09-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9b0006590 con 0x7fb9b0002830 2024-09-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9c0004af0 con 0x7fb9b0002830 2024-09-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 10 monclient: dump: 2024-09-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cffff640 10 monclient: get_auth_request con 0x7fb9bc0363f0 auth_method 0 2024-09-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 111891846 0 0) 0x7fb9b0006590 con 0x7fb9b0002830 2024-09-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.482+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9b000a0e0 con 0x7fb9b0002830 2024-09-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.486+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fb9c0004d90 con 0x7fb9b0002830 2024-09-16T15:00:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.486+0000 7fb9cd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.486+0000 7fb9cd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:00:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.486+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 >> v1:172.21.15.26:6790/0 conn(0x7fb9b0002830 legacy=0x7fb9b0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.486+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 >> v1:172.21.15.26:6790/0 conn(0x7fb9b0002830 legacy=0x7fb9b0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:00:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9d703b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:00:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9d703b640 1 -- 172.21.15.26:0/4093499092 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb9b0009da0 con 0x7fb9b00090c0 2024-09-16T15:00:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803084548 0 0) 0x7fb9b0009da0 con 0x7fb9b00090c0 2024-09-16T15:00:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 10 monclient(hunting): my global_id is 284648 2024-09-16T15:00:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b0006590 con 0x7fb9b00090c0 2024-09-16T15:00:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2798264340 0 0) 0x7fb9b0006590 con 0x7fb9b00090c0 2024-09-16T15:00:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9b0009da0 con 0x7fb9b00090c0 2024-09-16T15:00:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb9c00053a0 con 0x7fb9b00090c0 2024-09-16T15:00:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 10 monclient: dump: 2024-09-16T15:00:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.509 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2877258265 0 0) 0x7fb9b0009da0 con 0x7fb9b00090c0 2024-09-16T15:00:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb9b000c290 con 0x7fb9b00090c0 2024-09-16T15:00:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 791115328) 0x7fb9c00067c0 con 0x7fb9b00090c0 2024-09-16T15:00:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:00:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:00:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 >> v1:172.21.15.26:6790/0 conn(0x7fb9b00090c0 legacy=0x7fb9b0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9cd7fa640 1 -- 172.21.15.26:0/4093499092 >> v1:172.21.15.26:6790/0 conn(0x7fb9b00090c0 legacy=0x7fb9b0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:00:06.510 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:00:06.510 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:00:06.510 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:00:06.510 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T15:00:06.510 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 162, 2024-09-16T15:00:06.510 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:00:06.510 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:00:06.510 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:00:06.510 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:00:06.510 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:06.510 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:00:06.511 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:00:06.511 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:00:06.511 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:00:06.511 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:06.511 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:06.511 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:06.511 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:06.511 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:06.511 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:06.511 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:06.511 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:06.511 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:06.511 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:06.512 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:06.512 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:00:06.512 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:00:06.512 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:06.512 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:06.512 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:06.512 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:06.512 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:06.512 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:06.512 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:06.512 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:06.512 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:06.512 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:06.513 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.513 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:00:06.513 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:00:06.513 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:00:06.513 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:00:06.513 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:00:06.513 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:00:06.513 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:00:06.513 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:00:06.513 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:00:06.513 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:00:06.513 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:00:06.513 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:00:06.514 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:00:06.514 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:00:06.514 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:00:06.514 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:00:06.514 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:00:06.514 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:06.514 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:06.514 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:06.514 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:06.514 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:06.514 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:06.514 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.515 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:06.516 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:00:06.516 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:06.516 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.516 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.516 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:00:06.516 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:00:06.516 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:06.516 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:06.516 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:06.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.516 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:00:06.517 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:00:06.517 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:06.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:06.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.517 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:06.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:00:06.517 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:06.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.517 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:00:06.517 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:00:06.517 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.518 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.519 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:00:06.519 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:00:06.519 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:06.519 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:06.519 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:06.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.519 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.519 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:00:06.519 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:00:06.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.519 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:00:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T15:00:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T15:00:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:00:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T15:00:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T15:00:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:00:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:00:06.522 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:00:06.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9d703b640 1 -- 172.21.15.26:0/4093499092 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb9bc0363f0 legacy=0x7fb9bc0388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9d703b640 10 monclient: shutdown 2024-09-16T15:00:06.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9d703b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:06.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9d703b640 1 -- 172.21.15.26:0/4093499092 >> v1:172.21.15.26:6790/0 conn(0x7fb9d009d1b0 legacy=0x7fb9d01d1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9d55d8640 1 -- 172.21.15.26:0/4093499092 reap_dead start 2024-09-16T15:00:06.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9d55d8640 1 -- 172.21.15.26:0/4093499092 reap_dead start 2024-09-16T15:00:06.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9d703b640 1 -- 172.21.15.26:0/4093499092 shutdown_connections 2024-09-16T15:00:06.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9d703b640 1 -- 172.21.15.26:0/4093499092 >> 172.21.15.26:0/4093499092 conn(0x7fb9d009fb50 msgr2=0x7fb9d0133130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:06.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9d703b640 1 -- 172.21.15.26:0/4093499092 shutdown_connections 2024-09-16T15:00:06.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.490+0000 7fb9d703b640 1 -- 172.21.15.26:0/4093499092 wait complete. 2024-09-16T15:00:06.524 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:00:06.524 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T15:00:06.524 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T15:00:06.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.634+0000 7fadf8347640 10 monclient: get_monmap_and_config 2024-09-16T15:00:06.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.634+0000 7fadf8347640 10 monclient: build_initial_monmap 2024-09-16T15:00:06.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.634+0000 7fadf8347640 10 monclient: monmap: 2024-09-16T15:00:06.638 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:06.638 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.638 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:00:06.640137+0000 2024-09-16T15:00:06.638 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:00:06.640137+0000 2024-09-16T15:00:06.638 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:06.638 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.638 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:06.638 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:00:06.639 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:00:06.639 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.634+0000 7fadf8347640 1 Processor -- start 2024-09-16T15:00:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.634+0000 7fadf8347640 1 -- start start 2024-09-16T15:00:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.634+0000 7fadf8347640 10 monclient: init 2024-09-16T15:00:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:00:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient(hunting): picked mon.noname-c con 0x7fadf009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient(hunting): picked mon.noname-b con 0x7fadf0132650 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient(hunting): picked mon.noname-a con 0x7fadf0138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf009d260 con 0x7fadf0138ff0 2024-09-16T15:00:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf0132a80 con 0x7fadf009cb80 2024-09-16T15:00:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf009cf60 con 0x7fadf0132650 2024-09-16T15:00:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient(hunting): authenticate will time out at 2030.760247s 2024-09-16T15:00:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf58e2640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fadf0132650 0x7fadf01378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59816/0 (socket says 172.21.15.26:59816) 2024-09-16T15:00:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf58e2640 1 -- 172.21.15.26:0/4184016632 learned_addr learned my addr 172.21.15.26:0/4184016632 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 1 -- 172.21.15.26:0/4184016632 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853246924 0 0) 0x7fadf009cf60 con 0x7fadf0132650 2024-09-16T15:00:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient(hunting): my global_id is 284651 2024-09-16T15:00:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 1 -- 172.21.15.26:0/4184016632 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faddc003530 con 0x7fadf0132650 2024-09-16T15:00:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 1 -- 172.21.15.26:0/4184016632 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20540981 0 0) 0x7fadf009d260 con 0x7fadf0138ff0 2024-09-16T15:00:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient(hunting): my global_id is 334119 2024-09-16T15:00:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 1 -- 172.21.15.26:0/4184016632 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadf009cf60 con 0x7fadf0138ff0 2024-09-16T15:00:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 1 -- 172.21.15.26:0/4184016632 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3310707418 0 0) 0x7faddc003530 con 0x7fadf0132650 2024-09-16T15:00:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 1 -- 172.21.15.26:0/4184016632 >> v1:172.21.15.189:6789/0 conn(0x7fadf009cb80 legacy=0x7fadf00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 1 -- 172.21.15.26:0/4184016632 >> v1:172.21.15.26:6789/0 conn(0x7fadf0138ff0 legacy=0x7fadf013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient: _finish_hunting 0 2024-09-16T15:00:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 1 monclient: found mon.noname-b 2024-09-16T15:00:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:00:06.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 1 -- 172.21.15.26:0/4184016632 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf009f2d0 con 0x7fadf0132650 2024-09-16T15:00:06.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient: _finish_auth 0 2024-09-16T15:00:06.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient: _check_auth_tickets 2024-09-16T15:00:06.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:06.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 1 -- 172.21.15.26:0/4184016632 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fade0001660 con 0x7fadf0132650 2024-09-16T15:00:06.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 5 monclient: authenticate success, global_id 284651 2024-09-16T15:00:06.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:00:06.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient: dump: 2024-09-16T15:00:06.650 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.650 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.650 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.650 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.651 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.651 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.651 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.651 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.651 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.651 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:06.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 1 -- 172.21.15.26:0/4184016632 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fade0001800 con 0x7fadf0132650 2024-09-16T15:00:06.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:06.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient: get_monmap_and_config success 2024-09-16T15:00:06.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 4 set_mon_vals no callback set 2024-09-16T15:00:06.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 1 -- 172.21.15.26:0/4184016632 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fade0001bd0 con 0x7fadf0132650 2024-09-16T15:00:06.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:06.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf50e1640 10 monclient: dump: 2024-09-16T15:00:06.652 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.652 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.652 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.652 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.652 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.652 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.652 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.652 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.652 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.652 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient: shutdown 2024-09-16T15:00:06.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:06.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 1 -- 172.21.15.26:0/4184016632 >> v1:172.21.15.26:6790/0 conn(0x7fadf0132650 legacy=0x7fadf01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 1 -- 172.21.15.26:0/4184016632 shutdown_connections 2024-09-16T15:00:06.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 1 -- 172.21.15.26:0/4184016632 >> 172.21.15.26:0/4184016632 conn(0x7fadf009fb50 msgr2=0x7fadf009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:06.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 1 -- 172.21.15.26:0/4184016632 shutdown_connections 2024-09-16T15:00:06.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 1 -- 172.21.15.26:0/4184016632 wait complete. 2024-09-16T15:00:06.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient: build_initial_monmap 2024-09-16T15:00:06.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient: monmap: 2024-09-16T15:00:06.653 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:06.653 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:00:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:00:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:00:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:00:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:00:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:00:06.654 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 1 Processor -- start 2024-09-16T15:00:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 1 -- start start 2024-09-16T15:00:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient: init 2024-09-16T15:00:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:00:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient(hunting): picked mon.noname-b con 0x7fadf009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient(hunting): picked mon.noname-a con 0x7fadf0132650 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient(hunting): picked mon.noname-c con 0x7fadf0138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf0138840 con 0x7fadf0132650 2024-09-16T15:00:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf0134bf0 con 0x7fadf009cb80 2024-09-16T15:00:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf012e460 con 0x7fadf0138ff0 2024-09-16T15:00:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf8347640 10 monclient(hunting): authenticate will time out at 2030.763485s 2024-09-16T15:00:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf58e2640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fadf0132650 0x7fadf012dd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50120/0 (socket says 172.21.15.26:50120) 2024-09-16T15:00:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fadf58e2640 1 -- 172.21.15.26:0/2725836398 learned_addr learned my addr 172.21.15.26:0/2725836398 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837465158 0 0) 0x7fadf012e460 con 0x7fadf0138ff0 2024-09-16T15:00:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fade6ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fade6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fade6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fade6ffd640 10 monclient(hunting): my global_id is 284654 2024-09-16T15:00:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd4003530 con 0x7fadf0138ff0 2024-09-16T15:00:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102076408 0 0) 0x7fadf0138840 con 0x7fadf0132650 2024-09-16T15:00:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fade6ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fade6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fade6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fade6ffd640 10 monclient(hunting): my global_id is 334122 2024-09-16T15:00:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.638+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadf012e460 con 0x7fadf0132650 2024-09-16T15:00:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3860861831 0 0) 0x7fadf012e460 con 0x7fadf0132650 2024-09-16T15:00:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadf0138840 con 0x7fadf0132650 2024-09-16T15:00:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3577116097 0 0) 0x7fadd4003530 con 0x7fadf0138ff0 2024-09-16T15:00:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadf012e460 con 0x7fadf0138ff0 2024-09-16T15:00:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fade0001560 con 0x7fadf0132650 2024-09-16T15:00:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:00:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient(hunting): dump: 2024-09-16T15:00:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.658 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:00:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fade8001840 con 0x7fadf0138ff0 2024-09-16T15:00:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient(hunting): dump: 2024-09-16T15:00:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.659 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709287843 0 0) 0x7fadf0134bf0 con 0x7fadf009cb80 2024-09-16T15:00:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient(hunting): my global_id is 304381 2024-09-16T15:00:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd4003530 con 0x7fadf009cb80 2024-09-16T15:00:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4090346602 0 0) 0x7fadf0138840 con 0x7fadf0132650 2024-09-16T15:00:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 >> v1:172.21.15.26:6790/0 conn(0x7fadf0138ff0 legacy=0x7fadf01d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 >> v1:172.21.15.189:6789/0 conn(0x7fadf009cb80 legacy=0x7fadf012d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient: _finish_hunting 0 2024-09-16T15:00:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 monclient: found mon.a 2024-09-16T15:00:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf013c700 con 0x7fadf0132650 2024-09-16T15:00:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient: _finish_auth 0 2024-09-16T15:00:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient: _check_auth_tickets 2024-09-16T15:00:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fadf8347640 5 monclient: authenticate success, global_id 334122 2024-09-16T15:00:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fadf8347640 10 monclient: _renew_subs 2024-09-16T15:00:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fadf8347640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fadf8347640 1 -- 172.21.15.26:0/2725836398 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf009a240 con 0x7fadf0132650 2024-09-16T15:00:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fadf8347640 10 monclient: _renew_subs 2024-09-16T15:00:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fadf8347640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fadf8347640 1 -- 172.21.15.26:0/2725836398 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf0134910 con 0x7fadf0132650 2024-09-16T15:00:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fade0004aa0 con 0x7fadf0132650 2024-09-16T15:00:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fade0001260 con 0x7fadf0132650 2024-09-16T15:00:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fadf48e0640 4 set_mon_vals no callback set 2024-09-16T15:00:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient: dump: 2024-09-16T15:00:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.662 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fadf8347640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fadf8347640 1 -- 172.21.15.26:0/2725836398 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fadd8006590 con 0x7fadd8002830 2024-09-16T15:00:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fade000f6c0 con 0x7fadf0132650 2024-09-16T15:00:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fade0043ee0 con 0x7fadf0132650 2024-09-16T15:00:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fadf68e4640 10 monclient: get_auth_request con 0x7fadd4036450 auth_method 0 2024-09-16T15:00:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885702892 0 0) 0x7fadd8006590 con 0x7fadd8002830 2024-09-16T15:00:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient(hunting): my global_id is 304384 2024-09-16T15:00:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadf0138840 con 0x7fadd8002830 2024-09-16T15:00:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 864456771 0 0) 0x7fadf0138840 con 0x7fadd8002830 2024-09-16T15:00:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadd8006590 con 0x7fadd8002830 2024-09-16T15:00:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fadec001a80 con 0x7fadd8002830 2024-09-16T15:00:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:00:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient: dump: 2024-09-16T15:00:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.664 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.664 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.664 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.664 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.664 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2883843953 0 0) 0x7fadd8006590 con 0x7fadd8002830 2024-09-16T15:00:06.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadd800a0e0 con 0x7fadd8002830 2024-09-16T15:00:06.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fadec001e30 con 0x7fadd8002830 2024-09-16T15:00:06.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:06.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:00:06.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 >> v1:172.21.15.189:6789/0 conn(0x7fadd8002830 legacy=0x7fadd8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.642+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 >> v1:172.21.15.189:6789/0 conn(0x7fadd8002830 legacy=0x7fadd8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:00:06.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.650+0000 7fadf8347640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:00:06.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.650+0000 7fadf8347640 1 -- 172.21.15.26:0/2725836398 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fadd8009d80 con 0x7fadd80090a0 2024-09-16T15:00:06.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.650+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437067361 0 0) 0x7fadd8009d80 con 0x7fadd80090a0 2024-09-16T15:00:06.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.650+0000 7fade6ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.650+0000 7fade6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.650+0000 7fade6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.650+0000 7fade6ffd640 10 monclient(hunting): my global_id is 304387 2024-09-16T15:00:06.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.650+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd8006590 con 0x7fadd80090a0 2024-09-16T15:00:06.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.650+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1617236922 0 0) 0x7fadd8006590 con 0x7fadd80090a0 2024-09-16T15:00:06.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.650+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadd8009d80 con 0x7fadd80090a0 2024-09-16T15:00:06.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.650+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fadec003b20 con 0x7fadd80090a0 2024-09-16T15:00:06.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.650+0000 7fade6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.650+0000 7fade6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:00:06.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.650+0000 7fade6ffd640 10 monclient: dump: 2024-09-16T15:00:06.667 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.667 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.667 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.667 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.667 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.667 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.667 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.668 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.668 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.668 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.650+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3543667466 0 0) 0x7fadd8009d80 con 0x7fadd80090a0 2024-09-16T15:00:06.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.650+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fadd800c290 con 0x7fadd80090a0 2024-09-16T15:00:06.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.654+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3694284886) 0x7fadec004f30 con 0x7fadd80090a0 2024-09-16T15:00:06.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.654+0000 7fade6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:00:06.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.654+0000 7fade6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:00:06.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.654+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 >> v1:172.21.15.189:6789/0 conn(0x7fadd80090a0 legacy=0x7fadd8009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.654+0000 7fade6ffd640 1 -- 172.21.15.26:0/2725836398 >> v1:172.21.15.189:6789/0 conn(0x7fadd80090a0 legacy=0x7fadd8009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:00:06.668 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:00:06.669 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:00:06.669 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:00:06.669 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:00:06.669 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 162, 2024-09-16T15:00:06.669 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:00:06.669 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:00:06.669 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:00:06.669 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:00:06.669 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:06.669 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:00:06.669 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:00:06.669 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:00:06.670 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:00:06.670 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:06.670 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:06.670 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:06.670 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:06.670 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:06.670 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:06.670 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:06.670 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:06.670 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:06.670 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:06.670 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:06.670 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:00:06.671 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:00:06.671 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:06.671 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:06.671 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:06.671 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:06.671 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:06.671 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:06.671 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:06.671 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:06.672 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:06.672 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:06.672 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.672 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:00:06.672 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:00:06.672 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:00:06.672 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:00:06.672 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:00:06.672 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:00:06.672 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:00:06.672 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:00:06.672 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:00:06.673 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:00:06.673 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:00:06.673 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:00:06.673 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:00:06.673 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:00:06.673 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:00:06.673 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:00:06.673 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:00:06.673 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:06.673 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:06.673 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:06.673 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:06.674 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:06.674 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:06.674 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:06.674 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:06.674 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:06.674 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:06.674 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:06.674 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:00:06.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.674 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:00:06.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.674 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:00:06.674 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:00:06.675 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:06.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:06.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.675 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:06.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:00:06.675 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:06.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.675 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:00:06.675 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:00:06.675 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:06.675 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:06.675 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:06.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.676 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:00:06.676 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:00:06.676 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:06.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:06.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:06.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:00:06.676 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:06.676 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.676 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.677 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:00:06.677 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:00:06.677 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:06.677 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:06.677 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:06.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.677 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:00:06.678 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:00:06.678 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:06.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:06.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.678 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:06.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:00:06.678 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:06.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:00:06.678 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:00:06.678 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:06.678 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:00:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:00:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:00:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:06.679 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:00:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T15:00:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:06.680 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:00:06.680 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:00:06.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.654+0000 7fadf8347640 1 -- 172.21.15.26:0/2725836398 >> v1:172.21.15.189:6812/3196161582 conn(0x7fadd4036450 legacy=0x7fadd4038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.654+0000 7fadf8347640 10 monclient: shutdown 2024-09-16T15:00:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.654+0000 7fadf8347640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.654+0000 7fadf8347640 1 -- 172.21.15.26:0/2725836398 >> v1:172.21.15.26:6789/0 conn(0x7fadf0132650 legacy=0x7fadf012dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.654+0000 7fadf68e4640 1 -- 172.21.15.26:0/2725836398 reap_dead start 2024-09-16T15:00:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.654+0000 7fadf68e4640 1 -- 172.21.15.26:0/2725836398 reap_dead start 2024-09-16T15:00:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.654+0000 7fadf8347640 1 -- 172.21.15.26:0/2725836398 shutdown_connections 2024-09-16T15:00:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.654+0000 7fadf8347640 1 -- 172.21.15.26:0/2725836398 >> 172.21.15.26:0/2725836398 conn(0x7fadf009fb50 msgr2=0x7fadf009bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.654+0000 7fadf8347640 1 -- 172.21.15.26:0/2725836398 shutdown_connections 2024-09-16T15:00:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.654+0000 7fadf8347640 1 -- 172.21.15.26:0/2725836398 wait complete. 2024-09-16T15:00:06.683 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:00:06.683 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T15:00:06.683 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T15:00:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.794+0000 7f869083e640 10 monclient: get_monmap_and_config 2024-09-16T15:00:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.794+0000 7f869083e640 10 monclient: build_initial_monmap 2024-09-16T15:00:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.794+0000 7f869083e640 10 monclient: monmap: 2024-09-16T15:00:06.798 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:06.799 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.799 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:00:06.800727+0000 2024-09-16T15:00:06.799 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:00:06.800727+0000 2024-09-16T15:00:06.799 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:06.799 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.799 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:06.799 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:00:06.799 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:00:06.799 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 1 Processor -- start 2024-09-16T15:00:06.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 1 -- start start 2024-09-16T15:00:06.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient: init 2024-09-16T15:00:06.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:06.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:00:06.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient(hunting): picked mon.noname-c con 0x7f868c09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:06.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient(hunting): picked mon.noname-a con 0x7f868c132650 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:06.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient(hunting): picked mon.noname-b con 0x7f868c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:06.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f868c09d260 con 0x7f868c132650 2024-09-16T15:00:06.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f868c132a80 con 0x7f868c09cb80 2024-09-16T15:00:06.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f868c09cf60 con 0x7f868c138ff0 2024-09-16T15:00:06.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:06.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient(hunting): authenticate will time out at 2030.920823s 2024-09-16T15:00:06.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868ad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f868c138ff0 0x7f868c13b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59826/0 (socket says 172.21.15.26:59826) 2024-09-16T15:00:06.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868ad9d640 1 -- 172.21.15.26:0/1224956125 learned_addr learned my addr 172.21.15.26:0/1224956125 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:06.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 1 -- 172.21.15.26:0/1224956125 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153261093 0 0) 0x7f868c09d260 con 0x7f868c132650 2024-09-16T15:00:06.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient(hunting): my global_id is 334125 2024-09-16T15:00:06.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 1 -- 172.21.15.26:0/1224956125 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f866c003530 con 0x7f868c132650 2024-09-16T15:00:06.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 1 -- 172.21.15.26:0/1224956125 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236616368 0 0) 0x7f868c09cf60 con 0x7f868c138ff0 2024-09-16T15:00:06.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient(hunting): my global_id is 284657 2024-09-16T15:00:06.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 1 -- 172.21.15.26:0/1224956125 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f868c09d260 con 0x7f868c138ff0 2024-09-16T15:00:06.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 1 -- 172.21.15.26:0/1224956125 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2869298299 0 0) 0x7f868c09d260 con 0x7f868c138ff0 2024-09-16T15:00:06.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 1 -- 172.21.15.26:0/1224956125 >> v1:172.21.15.189:6789/0 conn(0x7f868c09cb80 legacy=0x7f868c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 1 -- 172.21.15.26:0/1224956125 >> v1:172.21.15.26:6789/0 conn(0x7f868c132650 legacy=0x7f868c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient: _finish_hunting 0 2024-09-16T15:00:06.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 1 monclient: found mon.noname-b 2024-09-16T15:00:06.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:06.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:00:06.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 1 -- 172.21.15.26:0/1224956125 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868c09f2d0 con 0x7f868c138ff0 2024-09-16T15:00:06.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient: _finish_auth 0 2024-09-16T15:00:06.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient: _check_auth_tickets 2024-09-16T15:00:06.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:06.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 1 -- 172.21.15.26:0/1224956125 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8684001640 con 0x7f868c138ff0 2024-09-16T15:00:06.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:00:06.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient: dump: 2024-09-16T15:00:06.809 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.809 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.809 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.809 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.809 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.809 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.809 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.810 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.810 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.810 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 5 monclient: authenticate success, global_id 284657 2024-09-16T15:00:06.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:06.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 1 -- 172.21.15.26:0/1224956125 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86840017e0 con 0x7f868c138ff0 2024-09-16T15:00:06.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:06.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient: get_monmap_and_config success 2024-09-16T15:00:06.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 4 set_mon_vals no callback set 2024-09-16T15:00:06.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 1 -- 172.21.15.26:0/1224956125 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8684001b30 con 0x7f868c138ff0 2024-09-16T15:00:06.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:06.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f868959a640 10 monclient: dump: 2024-09-16T15:00:06.811 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.811 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.811 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.811 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.811 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.811 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.811 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.811 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.811 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.812 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient: shutdown 2024-09-16T15:00:06.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:06.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 1 -- 172.21.15.26:0/1224956125 >> v1:172.21.15.26:6790/0 conn(0x7f868c138ff0 legacy=0x7f868c13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 1 -- 172.21.15.26:0/1224956125 shutdown_connections 2024-09-16T15:00:06.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 1 -- 172.21.15.26:0/1224956125 >> 172.21.15.26:0/1224956125 conn(0x7f868c09fb50 msgr2=0x7f868c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:06.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 1 -- 172.21.15.26:0/1224956125 shutdown_connections 2024-09-16T15:00:06.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 1 -- 172.21.15.26:0/1224956125 wait complete. 2024-09-16T15:00:06.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient: build_initial_monmap 2024-09-16T15:00:06.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient: monmap: 2024-09-16T15:00:06.812 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:06.812 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:00:06.812 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:00:06.813 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:00:06.813 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:06.813 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:00:06.813 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:06.813 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:00:06.813 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:00:06.813 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 1 Processor -- start 2024-09-16T15:00:06.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 1 -- start start 2024-09-16T15:00:06.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient: init 2024-09-16T15:00:06.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:06.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:00:06.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient(hunting): picked mon.noname-b con 0x7f868c09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient(hunting): picked mon.noname-a con 0x7f868c132650 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient(hunting): picked mon.noname-c con 0x7f868c138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f868c0718a0 con 0x7f868c132650 2024-09-16T15:00:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f868c13c6b0 con 0x7f868c09cb80 2024-09-16T15:00:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f868c09a240 con 0x7f868c138ff0 2024-09-16T15:00:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.798+0000 7f869083e640 10 monclient(hunting): authenticate will time out at 2030.923912s 2024-09-16T15:00:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f868ad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f868c138ff0 0x7f868c1cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59838/0 (socket says 172.21.15.26:59838) 2024-09-16T15:00:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f868ad9d640 1 -- 172.21.15.26:0/2038716136 learned_addr learned my addr 172.21.15.26:0/2038716136 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104894426 0 0) 0x7f868c09a240 con 0x7f868c138ff0 2024-09-16T15:00:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): my global_id is 284660 2024-09-16T15:00:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8670003530 con 0x7f868c138ff0 2024-09-16T15:00:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51657359 0 0) 0x7f868c0718a0 con 0x7f868c132650 2024-09-16T15:00:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): my global_id is 334128 2024-09-16T15:00:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f868c09a240 con 0x7f868c132650 2024-09-16T15:00:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689043205 0 0) 0x7f868c13c6b0 con 0x7f868c09cb80 2024-09-16T15:00:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): my global_id is 304393 2024-09-16T15:00:06.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f868c0718a0 con 0x7f868c09cb80 2024-09-16T15:00:06.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4070451516 0 0) 0x7f868c09a240 con 0x7f868c132650 2024-09-16T15:00:06.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f868c13c6b0 con 0x7f868c132650 2024-09-16T15:00:06.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8674001500 con 0x7f868c132650 2024-09-16T15:00:06.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:00:06.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): dump: 2024-09-16T15:00:06.816 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.816 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.816 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.816 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.816 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.816 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.816 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.817 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:00:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 639918592 0 0) 0x7f8670003530 con 0x7f868c138ff0 2024-09-16T15:00:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f868c09a240 con 0x7f868c138ff0 2024-09-16T15:00:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8684001af0 con 0x7f868c138ff0 2024-09-16T15:00:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:06.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient(hunting): dump: 2024-09-16T15:00:06.818 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.818 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.818 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.818 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.818 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.818 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.818 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.818 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.818 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.818 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 570493613 0 0) 0x7f868c13c6b0 con 0x7f868c132650 2024-09-16T15:00:06.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 >> v1:172.21.15.26:6790/0 conn(0x7f868c138ff0 legacy=0x7f868c1cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 >> v1:172.21.15.189:6789/0 conn(0x7f868c09cb80 legacy=0x7f868c06da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:06.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient: _finish_hunting 0 2024-09-16T15:00:06.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 monclient: found mon.a 2024-09-16T15:00:06.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:06.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:06.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868c06f490 con 0x7f868c132650 2024-09-16T15:00:06.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient: _finish_auth 0 2024-09-16T15:00:06.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient: _check_auth_tickets 2024-09-16T15:00:06.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:06.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f869083e640 5 monclient: authenticate success, global_id 334128 2024-09-16T15:00:06.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f869083e640 10 monclient: _renew_subs 2024-09-16T15:00:06.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f869083e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:06.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f869083e640 1 -- 172.21.15.26:0/2038716136 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868c06fde0 con 0x7f868c132650 2024-09-16T15:00:06.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f869083e640 10 monclient: _renew_subs 2024-09-16T15:00:06.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f869083e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:06.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f869083e640 1 -- 172.21.15.26:0/2038716136 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f868c070280 con 0x7f868c132650 2024-09-16T15:00:06.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8674001660 con 0x7f868c132650 2024-09-16T15:00:06.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:06.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8674003b20 con 0x7f868c132650 2024-09-16T15:00:06.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:06.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8688d99640 4 set_mon_vals no callback set 2024-09-16T15:00:06.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:06.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 10 monclient: dump: 2024-09-16T15:00:06.820 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:06.820 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:06.820 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.820 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:06.821 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:06.821 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:06.821 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:06.821 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:06.821 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:06.821 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:06.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8674003c80 con 0x7f868c132650 2024-09-16T15:00:06.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f869083e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:06.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f869083e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:06.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f869083e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:06.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f869083e640 1 -- 172.21.15.26:0/2038716136 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8664005050 con 0x7f868c132650 2024-09-16T15:00:06.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f8674042e60 con 0x7f868c132650 2024-09-16T15:00:06.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.802+0000 7f868a59c640 10 monclient: get_auth_request con 0x7f8670036340 auth_method 0 2024-09-16T15:00:06.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.806+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8674013510 con 0x7f868c132650 2024-09-16T15:00:06.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.806+0000 7f8682ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:06.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:06.806+0000 7f8682ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:00:07.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:07.030+0000 7f869083e640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T15:00:07.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:07.030+0000 7f869083e640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T15:00:07.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:07.030+0000 7f869083e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:07.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:07.030+0000 7f869083e640 1 -- 172.21.15.26:0/2038716136 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f8664005b10 con 0x7f868c132650 2024-09-16T15:00:07.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:07.030+0000 7f8682ffd640 1 -- 172.21.15.26:0/2038716136 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f8674013330 con 0x7f868c132650 2024-09-16T15:00:07.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:07.030+0000 7f8682ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T15:00:07.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:07.030+0000 7f8682ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:00:07.034 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.030+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.034 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.030+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=11,kv=11,logm=51} crc {auth=2835817426,config=692619927,health=3386755592,kv=1867599010,logm=3538922658}) 2024-09-16T15:00:07.035 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.030+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=1,health=11,kv=9,logm=52} crc {auth=3367628517,config=1228700967,health=1991956587,kv=3455456366,logm=1997650305}) 2024-09-16T15:00:07.035 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.030+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.035 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.030+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=11,kv=11,logm=51} crc {auth=2835817426,config=692619927,health=3386755592,kv=1867599010,logm=3538922658}) 2024-09-16T15:00:07.035 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.030+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=12,kv=11,logm=51} crc {auth=962922531,config=692619927,health=3132210607,kv=1867599010,logm=3274576530}) 2024-09-16T15:00:07.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:07.034+0000 7f869083e640 1 -- 172.21.15.26:0/2038716136 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8670036340 legacy=0x7f8670038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:07.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:07.034+0000 7f869083e640 10 monclient: shutdown 2024-09-16T15:00:07.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:07.034+0000 7f869083e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:07.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:07.034+0000 7f869083e640 1 -- 172.21.15.26:0/2038716136 >> v1:172.21.15.26:6789/0 conn(0x7f868c132650 legacy=0x7f868c06e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:07.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:07.034+0000 7f869083e640 1 -- 172.21.15.26:0/2038716136 shutdown_connections 2024-09-16T15:00:07.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:07.034+0000 7f869083e640 1 -- 172.21.15.26:0/2038716136 >> 172.21.15.26:0/2038716136 conn(0x7f868c09fb50 msgr2=0x7f868c131420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:07.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:07.034+0000 7f869083e640 1 -- 172.21.15.26:0/2038716136 shutdown_connections 2024-09-16T15:00:07.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:07.034+0000 7f869083e640 1 -- 172.21.15.26:0/2038716136 wait complete. 2024-09-16T15:00:07.038 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.034+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.038 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.034+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=991881959}) 2024-09-16T15:00:07.038 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.034+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4007015848}) 2024-09-16T15:00:07.038 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.034+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.038 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.034+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=991881959}) 2024-09-16T15:00:07.038 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.034+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1577950396}) 2024-09-16T15:00:07.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.038+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.038+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1665128362}) 2024-09-16T15:00:07.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.038+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896210664}) 2024-09-16T15:00:07.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.038+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.038+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1665128362}) 2024-09-16T15:00:07.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.038+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1397482093}) 2024-09-16T15:00:07.046 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.042+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.046 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.042+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3687061783}) 2024-09-16T15:00:07.046 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.042+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900168322}) 2024-09-16T15:00:07.046 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.042+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.047 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.042+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3687061783}) 2024-09-16T15:00:07.047 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.042+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2593411296}) 2024-09-16T15:00:07.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.046+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.046+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1381025304}) 2024-09-16T15:00:07.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.046+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2139157148}) 2024-09-16T15:00:07.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.046+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.046+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1381025304}) 2024-09-16T15:00:07.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.046+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2732666036}) 2024-09-16T15:00:07.050 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.046+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.050 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.046+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1678906819}) 2024-09-16T15:00:07.050 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.046+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=762693773}) 2024-09-16T15:00:07.050 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.046+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.050 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.046+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1678906819}) 2024-09-16T15:00:07.050 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.046+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1446047435}) 2024-09-16T15:00:07.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1342836818}) 2024-09-16T15:00:07.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2030804795}) 2024-09-16T15:00:07.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1342836818}) 2024-09-16T15:00:07.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3330739007}) 2024-09-16T15:00:07.053 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T15:00:07.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=166740132}) 2024-09-16T15:00:07.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=705504752}) 2024-09-16T15:00:07.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=166740132}) 2024-09-16T15:00:07.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.050+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565428708}) 2024-09-16T15:00:07.056 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.056 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=560460104}) 2024-09-16T15:00:07.056 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1024139364}) 2024-09-16T15:00:07.057 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.057 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=560460104}) 2024-09-16T15:00:07.057 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408240961}) 2024-09-16T15:00:07.058 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.058 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1429391086}) 2024-09-16T15:00:07.058 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2731888770}) 2024-09-16T15:00:07.059 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.059 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1429391086}) 2024-09-16T15:00:07.059 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.054+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=318047744}) 2024-09-16T15:00:07.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1285469171}) 2024-09-16T15:00:07.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=805077371}) 2024-09-16T15:00:07.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1285469171}) 2024-09-16T15:00:07.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3854410557}) 2024-09-16T15:00:07.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3829341491}) 2024-09-16T15:00:07.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=827417006}) 2024-09-16T15:00:07.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.058+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3829341491}) 2024-09-16T15:00:07.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.062+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=747646943}) 2024-09-16T15:00:07.065 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.062+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.065 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.062+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2514673526}) 2024-09-16T15:00:07.065 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.062+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=569114308}) 2024-09-16T15:00:07.065 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.062+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.065 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.062+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2514673526}) 2024-09-16T15:00:07.065 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.062+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3059033971}) 2024-09-16T15:00:07.067 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.067 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1580548251}) 2024-09-16T15:00:07.067 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883043154}) 2024-09-16T15:00:07.067 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.067 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1580548251}) 2024-09-16T15:00:07.068 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3442394610}) 2024-09-16T15:00:07.069 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3874381244}) 2024-09-16T15:00:07.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1126920228}) 2024-09-16T15:00:07.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3874381244}) 2024-09-16T15:00:07.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.066+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3735889275}) 2024-09-16T15:00:07.072 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.072 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3117014475}) 2024-09-16T15:00:07.072 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1588180138}) 2024-09-16T15:00:07.072 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.072 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3117014475}) 2024-09-16T15:00:07.072 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=307958298}) 2024-09-16T15:00:07.074 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.074 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2784157545}) 2024-09-16T15:00:07.074 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1699074085}) 2024-09-16T15:00:07.074 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.074 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2784157545}) 2024-09-16T15:00:07.074 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.070+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3248839583}) 2024-09-16T15:00:07.076 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.076 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2860333585}) 2024-09-16T15:00:07.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=851268995}) 2024-09-16T15:00:07.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2860333585}) 2024-09-16T15:00:07.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1385944163}) 2024-09-16T15:00:07.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3360168511}) 2024-09-16T15:00:07.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=357182356}) 2024-09-16T15:00:07.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3360168511}) 2024-09-16T15:00:07.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.074+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3384976696}) 2024-09-16T15:00:07.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.078+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.078+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192241096}) 2024-09-16T15:00:07.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.078+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2540055528}) 2024-09-16T15:00:07.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.078+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.078+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192241096}) 2024-09-16T15:00:07.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.078+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1852600943}) 2024-09-16T15:00:07.083 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.084 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3410728111}) 2024-09-16T15:00:07.084 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=922786956}) 2024-09-16T15:00:07.084 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.084 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3410728111}) 2024-09-16T15:00:07.084 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=803365926}) 2024-09-16T15:00:07.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=397257557}) 2024-09-16T15:00:07.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1130834864}) 2024-09-16T15:00:07.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=397257557}) 2024-09-16T15:00:07.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.082+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=34569619}) 2024-09-16T15:00:07.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1826755469}) 2024-09-16T15:00:07.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053248089}) 2024-09-16T15:00:07.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1826755469}) 2024-09-16T15:00:07.089 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1210383396}) 2024-09-16T15:00:07.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2805734631}) 2024-09-16T15:00:07.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1893134886}) 2024-09-16T15:00:07.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.091 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2805734631}) 2024-09-16T15:00:07.091 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.086+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2333408754}) 2024-09-16T15:00:07.093 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.093 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3752375815}) 2024-09-16T15:00:07.093 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1558841000}) 2024-09-16T15:00:07.094 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.094 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3752375815}) 2024-09-16T15:00:07.094 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1748041012}) 2024-09-16T15:00:07.096 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.096 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=318013947}) 2024-09-16T15:00:07.096 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2579692582}) 2024-09-16T15:00:07.096 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.097 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=318013947}) 2024-09-16T15:00:07.097 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.090+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=395540807}) 2024-09-16T15:00:07.098 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.098 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3775246365}) 2024-09-16T15:00:07.099 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248357045}) 2024-09-16T15:00:07.099 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.099 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3775246365}) 2024-09-16T15:00:07.099 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.094+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1747144119}) 2024-09-16T15:00:07.102 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.102 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=643076837}) 2024-09-16T15:00:07.102 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2187687144}) 2024-09-16T15:00:07.102 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.102 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=643076837}) 2024-09-16T15:00:07.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2472602554}) 2024-09-16T15:00:07.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4267505382}) 2024-09-16T15:00:07.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840655195}) 2024-09-16T15:00:07.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4267505382}) 2024-09-16T15:00:07.104 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.098+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3286485839}) 2024-09-16T15:00:07.106 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.106 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3558299775}) 2024-09-16T15:00:07.106 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=191171506}) 2024-09-16T15:00:07.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3558299775}) 2024-09-16T15:00:07.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3471718098}) 2024-09-16T15:00:07.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=211876882}) 2024-09-16T15:00:07.108 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494612946}) 2024-09-16T15:00:07.108 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.108 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=211876882}) 2024-09-16T15:00:07.108 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.102+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2026630597}) 2024-09-16T15:00:07.110 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.110 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3999011810}) 2024-09-16T15:00:07.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=339662578}) 2024-09-16T15:00:07.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3999011810}) 2024-09-16T15:00:07.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3830999412}) 2024-09-16T15:00:07.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.112 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2826584608}) 2024-09-16T15:00:07.112 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354724452}) 2024-09-16T15:00:07.112 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.112 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2826584608}) 2024-09-16T15:00:07.112 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.106+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1138630976}) 2024-09-16T15:00:07.113 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.113 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1429534225}) 2024-09-16T15:00:07.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953102286}) 2024-09-16T15:00:07.115 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.115 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1429534225}) 2024-09-16T15:00:07.115 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2973796786}) 2024-09-16T15:00:07.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=6956019}) 2024-09-16T15:00:07.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2961527071}) 2024-09-16T15:00:07.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.118 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=6956019}) 2024-09-16T15:00:07.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.110+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3421864093}) 2024-09-16T15:00:07.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=172059515}) 2024-09-16T15:00:07.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963534343}) 2024-09-16T15:00:07.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=172059515}) 2024-09-16T15:00:07.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3134956848}) 2024-09-16T15:00:07.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.114+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3067699225}) 2024-09-16T15:00:07.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388156445}) 2024-09-16T15:00:07.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3067699225}) 2024-09-16T15:00:07.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4178852557}) 2024-09-16T15:00:07.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=924295060}) 2024-09-16T15:00:07.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834272280}) 2024-09-16T15:00:07.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=924295060}) 2024-09-16T15:00:07.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.118+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=754058388}) 2024-09-16T15:00:07.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1332406222}) 2024-09-16T15:00:07.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1325417555}) 2024-09-16T15:00:07.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1332406222}) 2024-09-16T15:00:07.124 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2675857672}) 2024-09-16T15:00:07.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=994882646}) 2024-09-16T15:00:07.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2040753614}) 2024-09-16T15:00:07.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=994882646}) 2024-09-16T15:00:07.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.122+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2917347870}) 2024-09-16T15:00:07.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1200920340}) 2024-09-16T15:00:07.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553397829}) 2024-09-16T15:00:07.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1200920340}) 2024-09-16T15:00:07.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3309364257}) 2024-09-16T15:00:07.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2923446282}) 2024-09-16T15:00:07.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=785999058}) 2024-09-16T15:00:07.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2923446282}) 2024-09-16T15:00:07.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.126+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3508768487}) 2024-09-16T15:00:07.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2226409882}) 2024-09-16T15:00:07.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436520682}) 2024-09-16T15:00:07.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2226409882}) 2024-09-16T15:00:07.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3638419823}) 2024-09-16T15:00:07.134 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.134 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1241499431}) 2024-09-16T15:00:07.134 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=43434861}) 2024-09-16T15:00:07.135 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.135 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1241499431}) 2024-09-16T15:00:07.135 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.130+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3060870047}) 2024-09-16T15:00:07.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3680475370}) 2024-09-16T15:00:07.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1659206719}) 2024-09-16T15:00:07.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3680475370}) 2024-09-16T15:00:07.137 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=798940277}) 2024-09-16T15:00:07.139 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.139 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2399256300}) 2024-09-16T15:00:07.139 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2794796277}) 2024-09-16T15:00:07.139 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.139 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2399256300}) 2024-09-16T15:00:07.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.134+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=5827082}) 2024-09-16T15:00:07.141 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.141 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1252756468}) 2024-09-16T15:00:07.141 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=567487366}) 2024-09-16T15:00:07.141 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.141 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1252756468}) 2024-09-16T15:00:07.141 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1375418688}) 2024-09-16T15:00:07.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=38618136}) 2024-09-16T15:00:07.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1625495403}) 2024-09-16T15:00:07.143 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.143 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=38618136}) 2024-09-16T15:00:07.143 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.138+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=614534615}) 2024-09-16T15:00:07.143 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3542218860}) 2024-09-16T15:00:07.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1008980859}) 2024-09-16T15:00:07.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3542218860}) 2024-09-16T15:00:07.144 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2730792918}) 2024-09-16T15:00:07.146 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.146 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2865941274}) 2024-09-16T15:00:07.146 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3460097980}) 2024-09-16T15:00:07.146 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.146 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2865941274}) 2024-09-16T15:00:07.146 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.142+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2410995699}) 2024-09-16T15:00:07.148 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.146+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.148 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.146+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1402462148}) 2024-09-16T15:00:07.148 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.146+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4035852756}) 2024-09-16T15:00:07.148 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.146+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.148 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.146+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1402462148}) 2024-09-16T15:00:07.148 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.146+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2269423747}) 2024-09-16T15:00:07.150 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.146+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.150 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.146+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1898385072}) 2024-09-16T15:00:07.150 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.146+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385377273}) 2024-09-16T15:00:07.150 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.146+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.150 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.146+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1898385072}) 2024-09-16T15:00:07.150 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.146+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3447322274}) 2024-09-16T15:00:07.152 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.152 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1957516575}) 2024-09-16T15:00:07.152 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=65493684}) 2024-09-16T15:00:07.152 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.152 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1957516575}) 2024-09-16T15:00:07.152 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1034866139}) 2024-09-16T15:00:07.154 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.154 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2199721428}) 2024-09-16T15:00:07.154 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198002402}) 2024-09-16T15:00:07.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2199721428}) 2024-09-16T15:00:07.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.150+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2065784841}) 2024-09-16T15:00:07.156 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2944232979}) 2024-09-16T15:00:07.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624589624}) 2024-09-16T15:00:07.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2944232979}) 2024-09-16T15:00:07.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2869479603}) 2024-09-16T15:00:07.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=743513846}) 2024-09-16T15:00:07.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1388094198}) 2024-09-16T15:00:07.159 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.160 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=743513846}) 2024-09-16T15:00:07.160 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.154+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3334323704}) 2024-09-16T15:00:07.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2392472156}) 2024-09-16T15:00:07.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115061956}) 2024-09-16T15:00:07.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2392472156}) 2024-09-16T15:00:07.162 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2529298509}) 2024-09-16T15:00:07.163 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.163 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3943566348}) 2024-09-16T15:00:07.163 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.158+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3718108106}) 2024-09-16T15:00:07.163 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.164 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3943566348}) 2024-09-16T15:00:07.164 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1713688806}) 2024-09-16T15:00:07.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2571515942}) 2024-09-16T15:00:07.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1971497168}) 2024-09-16T15:00:07.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2571515942}) 2024-09-16T15:00:07.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.162+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3257931677}) 2024-09-16T15:00:07.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3270271811}) 2024-09-16T15:00:07.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3820924741}) 2024-09-16T15:00:07.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3270271811}) 2024-09-16T15:00:07.168 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=208589605}) 2024-09-16T15:00:07.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2853479502}) 2024-09-16T15:00:07.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906120895}) 2024-09-16T15:00:07.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2853479502}) 2024-09-16T15:00:07.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.166+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3740465532}) 2024-09-16T15:00:07.171 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.171 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=564662518}) 2024-09-16T15:00:07.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890973956}) 2024-09-16T15:00:07.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=564662518}) 2024-09-16T15:00:07.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3830458713}) 2024-09-16T15:00:07.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1051009902}) 2024-09-16T15:00:07.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=647158792}) 2024-09-16T15:00:07.175 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.175 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1051009902}) 2024-09-16T15:00:07.175 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.170+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1448709576}) 2024-09-16T15:00:07.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2124281640}) 2024-09-16T15:00:07.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643978019}) 2024-09-16T15:00:07.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2124281640}) 2024-09-16T15:00:07.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2074064679}) 2024-09-16T15:00:07.178 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=73395131}) 2024-09-16T15:00:07.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458993586}) 2024-09-16T15:00:07.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=73395131}) 2024-09-16T15:00:07.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.174+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2230455384}) 2024-09-16T15:00:07.180 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.180 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=840885569}) 2024-09-16T15:00:07.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983283105}) 2024-09-16T15:00:07.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=840885569}) 2024-09-16T15:00:07.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2463500454}) 2024-09-16T15:00:07.182 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.182 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3933161402}) 2024-09-16T15:00:07.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1129822924}) 2024-09-16T15:00:07.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3933161402}) 2024-09-16T15:00:07.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.178+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3550861599}) 2024-09-16T15:00:07.185 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.185 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=865898217}) 2024-09-16T15:00:07.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2055379644}) 2024-09-16T15:00:07.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=865898217}) 2024-09-16T15:00:07.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2551401428}) 2024-09-16T15:00:07.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2966014604}) 2024-09-16T15:00:07.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3431402528}) 2024-09-16T15:00:07.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2966014604}) 2024-09-16T15:00:07.187 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.182+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3460065417}) 2024-09-16T15:00:07.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057228103}) 2024-09-16T15:00:07.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=216922103}) 2024-09-16T15:00:07.189 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.189 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057228103}) 2024-09-16T15:00:07.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3592215966}) 2024-09-16T15:00:07.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1808763012}) 2024-09-16T15:00:07.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=393816366}) 2024-09-16T15:00:07.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1808763012}) 2024-09-16T15:00:07.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.186+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4291339049}) 2024-09-16T15:00:07.192 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.192 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2154303024}) 2024-09-16T15:00:07.192 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2791778949}) 2024-09-16T15:00:07.192 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2154303024}) 2024-09-16T15:00:07.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2332640213}) 2024-09-16T15:00:07.194 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3201477654}) 2024-09-16T15:00:07.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2518100169}) 2024-09-16T15:00:07.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3201477654}) 2024-09-16T15:00:07.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.190+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2252097695}) 2024-09-16T15:00:07.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2648802226}) 2024-09-16T15:00:07.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3064197743}) 2024-09-16T15:00:07.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2648802226}) 2024-09-16T15:00:07.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3281683712}) 2024-09-16T15:00:07.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1114230153}) 2024-09-16T15:00:07.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955318763}) 2024-09-16T15:00:07.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1114230153}) 2024-09-16T15:00:07.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.194+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327965368}) 2024-09-16T15:00:07.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4048942720}) 2024-09-16T15:00:07.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2136152618}) 2024-09-16T15:00:07.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4048942720}) 2024-09-16T15:00:07.201 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4220205802}) 2024-09-16T15:00:07.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1785411444}) 2024-09-16T15:00:07.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=74036897}) 2024-09-16T15:00:07.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1785411444}) 2024-09-16T15:00:07.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.198+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4270721117}) 2024-09-16T15:00:07.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2441066831}) 2024-09-16T15:00:07.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=540024508}) 2024-09-16T15:00:07.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2441066831}) 2024-09-16T15:00:07.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3193844789}) 2024-09-16T15:00:07.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=289319698}) 2024-09-16T15:00:07.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=609596616}) 2024-09-16T15:00:07.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=289319698}) 2024-09-16T15:00:07.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.202+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1124907066}) 2024-09-16T15:00:07.208 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.208 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2022352165}) 2024-09-16T15:00:07.208 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449303267}) 2024-09-16T15:00:07.208 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.208 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2022352165}) 2024-09-16T15:00:07.208 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1778154761}) 2024-09-16T15:00:07.210 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.210 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3281268534}) 2024-09-16T15:00:07.210 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870634712}) 2024-09-16T15:00:07.210 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.210 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3281268534}) 2024-09-16T15:00:07.210 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.206+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=247039016}) 2024-09-16T15:00:07.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3352198602}) 2024-09-16T15:00:07.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=351321555}) 2024-09-16T15:00:07.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3352198602}) 2024-09-16T15:00:07.215 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.210+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=682496922}) 2024-09-16T15:00:07.215 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3241614091}) 2024-09-16T15:00:07.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2915214047}) 2024-09-16T15:00:07.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3241614091}) 2024-09-16T15:00:07.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=455667074}) 2024-09-16T15:00:07.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2671642739}) 2024-09-16T15:00:07.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1779566236}) 2024-09-16T15:00:07.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2671642739}) 2024-09-16T15:00:07.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.214+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2791976309}) 2024-09-16T15:00:07.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4074247743}) 2024-09-16T15:00:07.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200029811}) 2024-09-16T15:00:07.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.220 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4074247743}) 2024-09-16T15:00:07.220 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=181125086}) 2024-09-16T15:00:07.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1056016508}) 2024-09-16T15:00:07.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629022131}) 2024-09-16T15:00:07.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1056016508}) 2024-09-16T15:00:07.222 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.218+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1399549146}) 2024-09-16T15:00:07.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1250936653}) 2024-09-16T15:00:07.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=359003512}) 2024-09-16T15:00:07.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1250936653}) 2024-09-16T15:00:07.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1892142609}) 2024-09-16T15:00:07.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4024987272}) 2024-09-16T15:00:07.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4025876600}) 2024-09-16T15:00:07.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4024987272}) 2024-09-16T15:00:07.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.222+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1922963032}) 2024-09-16T15:00:07.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=350062033}) 2024-09-16T15:00:07.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314342128}) 2024-09-16T15:00:07.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=350062033}) 2024-09-16T15:00:07.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3064309156}) 2024-09-16T15:00:07.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2336641215}) 2024-09-16T15:00:07.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=89099415}) 2024-09-16T15:00:07.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2336641215}) 2024-09-16T15:00:07.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086875948}) 2024-09-16T15:00:07.231 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.231 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2362915217}) 2024-09-16T15:00:07.231 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1787351507}) 2024-09-16T15:00:07.231 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.231 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2362915217}) 2024-09-16T15:00:07.231 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.226+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=342557901}) 2024-09-16T15:00:07.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394552275}) 2024-09-16T15:00:07.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000446319}) 2024-09-16T15:00:07.233 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.233 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394552275}) 2024-09-16T15:00:07.233 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=577792899}) 2024-09-16T15:00:07.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1594740511}) 2024-09-16T15:00:07.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806653781}) 2024-09-16T15:00:07.235 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.235 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1594740511}) 2024-09-16T15:00:07.235 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.230+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=412547524}) 2024-09-16T15:00:07.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3261313770}) 2024-09-16T15:00:07.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623216293}) 2024-09-16T15:00:07.237 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.237 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3261313770}) 2024-09-16T15:00:07.237 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=966719280}) 2024-09-16T15:00:07.239 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3587749967}) 2024-09-16T15:00:07.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2644044969}) 2024-09-16T15:00:07.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3587749967}) 2024-09-16T15:00:07.241 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.234+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1044747822}) 2024-09-16T15:00:07.241 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.241 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=651383567}) 2024-09-16T15:00:07.241 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=185539918}) 2024-09-16T15:00:07.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=651383567}) 2024-09-16T15:00:07.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2611446413}) 2024-09-16T15:00:07.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1776916251}) 2024-09-16T15:00:07.243 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364732629}) 2024-09-16T15:00:07.243 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.243 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1776916251}) 2024-09-16T15:00:07.243 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.238+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=98071369}) 2024-09-16T15:00:07.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=381131571}) 2024-09-16T15:00:07.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2928978284}) 2024-09-16T15:00:07.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=381131571}) 2024-09-16T15:00:07.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3768700464}) 2024-09-16T15:00:07.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2464303112}) 2024-09-16T15:00:07.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=157473522}) 2024-09-16T15:00:07.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2464303112}) 2024-09-16T15:00:07.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.242+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=505927260}) 2024-09-16T15:00:07.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=434948234}) 2024-09-16T15:00:07.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4067805635}) 2024-09-16T15:00:07.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=434948234}) 2024-09-16T15:00:07.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3312433204}) 2024-09-16T15:00:07.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3129224436}) 2024-09-16T15:00:07.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2890051972}) 2024-09-16T15:00:07.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3129224436}) 2024-09-16T15:00:07.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.246+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=953684993}) 2024-09-16T15:00:07.253 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.253 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3561721927}) 2024-09-16T15:00:07.253 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1506395609}) 2024-09-16T15:00:07.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3561721927}) 2024-09-16T15:00:07.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4240936194}) 2024-09-16T15:00:07.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2524280598}) 2024-09-16T15:00:07.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3985120846}) 2024-09-16T15:00:07.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2524280598}) 2024-09-16T15:00:07.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.250+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1640463697}) 2024-09-16T15:00:07.255 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.255 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2059014755}) 2024-09-16T15:00:07.255 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=66322328}) 2024-09-16T15:00:07.255 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.255 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2059014755}) 2024-09-16T15:00:07.255 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3714585969}) 2024-09-16T15:00:07.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=564403039}) 2024-09-16T15:00:07.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1928562273}) 2024-09-16T15:00:07.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=564403039}) 2024-09-16T15:00:07.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.254+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=941994542}) 2024-09-16T15:00:07.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598852569}) 2024-09-16T15:00:07.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1271899053}) 2024-09-16T15:00:07.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598852569}) 2024-09-16T15:00:07.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=104715889}) 2024-09-16T15:00:07.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4136543440}) 2024-09-16T15:00:07.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=542298800}) 2024-09-16T15:00:07.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4136543440}) 2024-09-16T15:00:07.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.258+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2429424063}) 2024-09-16T15:00:07.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2256943847}) 2024-09-16T15:00:07.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=344100997}) 2024-09-16T15:00:07.267 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.267 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2256943847}) 2024-09-16T15:00:07.267 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.262+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1574575407}) 2024-09-16T15:00:07.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2400379241}) 2024-09-16T15:00:07.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624790013}) 2024-09-16T15:00:07.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2400379241}) 2024-09-16T15:00:07.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.270+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3276467656}) 2024-09-16T15:00:07.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.274+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.274+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3942582097}) 2024-09-16T15:00:07.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.274+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=996864314}) 2024-09-16T15:00:07.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.274+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.278+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3942582097}) 2024-09-16T15:00:07.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.278+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=391306169}) 2024-09-16T15:00:07.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.314+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.314+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1193654419}) 2024-09-16T15:00:07.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.314+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3416684832}) 2024-09-16T15:00:07.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.314+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.314+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1193654419}) 2024-09-16T15:00:07.316 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.314+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3426736930}) 2024-09-16T15:00:07.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.314+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.314+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3574973459}) 2024-09-16T15:00:07.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.314+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1235858357}) 2024-09-16T15:00:07.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.314+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.314+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3574973459}) 2024-09-16T15:00:07.318 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.314+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1710104390}) 2024-09-16T15:00:07.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.318+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.320 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.318+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1088950259}) 2024-09-16T15:00:07.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.318+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725846025}) 2024-09-16T15:00:07.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.318+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.318+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1088950259}) 2024-09-16T15:00:07.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.318+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3494023157}) 2024-09-16T15:00:07.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.318+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.318+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=46237060}) 2024-09-16T15:00:07.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.318+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=402836421}) 2024-09-16T15:00:07.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.318+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.318+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=46237060}) 2024-09-16T15:00:07.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.318+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4269050745}) 2024-09-16T15:00:07.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.322+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.322+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=509754916}) 2024-09-16T15:00:07.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.322+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3093620871}) 2024-09-16T15:00:07.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.322+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.322+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=509754916}) 2024-09-16T15:00:07.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.322+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3897339604}) 2024-09-16T15:00:07.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.326+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.326+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3022210600}) 2024-09-16T15:00:07.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.326+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3982407192}) 2024-09-16T15:00:07.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.326+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.326+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3022210600}) 2024-09-16T15:00:07.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.326+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4143956339}) 2024-09-16T15:00:07.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.326+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.326+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1047805247}) 2024-09-16T15:00:07.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.326+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=187016301}) 2024-09-16T15:00:07.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.326+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.326+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1047805247}) 2024-09-16T15:00:07.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.326+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3297748728}) 2024-09-16T15:00:07.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.330+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.330+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=923411814}) 2024-09-16T15:00:07.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.330+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2162997390}) 2024-09-16T15:00:07.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.330+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.330+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=923411814}) 2024-09-16T15:00:07.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.330+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1142628264}) 2024-09-16T15:00:07.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.330+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.330+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3434280890}) 2024-09-16T15:00:07.333 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.330+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=831004806}) 2024-09-16T15:00:07.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.330+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.330+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3434280890}) 2024-09-16T15:00:07.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.330+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159594093}) 2024-09-16T15:00:07.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1551353814}) 2024-09-16T15:00:07.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1930520952}) 2024-09-16T15:00:07.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1551353814}) 2024-09-16T15:00:07.335 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2291976830}) 2024-09-16T15:00:07.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=843567918}) 2024-09-16T15:00:07.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3838087331}) 2024-09-16T15:00:07.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=843567918}) 2024-09-16T15:00:07.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4114392410}) 2024-09-16T15:00:07.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348336192}) 2024-09-16T15:00:07.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=342789977}) 2024-09-16T15:00:07.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348336192}) 2024-09-16T15:00:07.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.334+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3767793445}) 2024-09-16T15:00:07.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3714087581}) 2024-09-16T15:00:07.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2273456911}) 2024-09-16T15:00:07.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3714087581}) 2024-09-16T15:00:07.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2182326860}) 2024-09-16T15:00:07.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=46048656}) 2024-09-16T15:00:07.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1931680808}) 2024-09-16T15:00:07.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=46048656}) 2024-09-16T15:00:07.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=684673795}) 2024-09-16T15:00:07.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=191874476}) 2024-09-16T15:00:07.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2818283120}) 2024-09-16T15:00:07.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=191874476}) 2024-09-16T15:00:07.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.338+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4082440872}) 2024-09-16T15:00:07.346 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.342+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.346 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.342+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=43} crc {logm=2115265916,mdsmap=2147943554,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=1041356535,mon_config_key=1766424633,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=2973260763}) 2024-09-16T15:00:07.346 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.342+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=9,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=49} crc {logm=4023014761,mdsmap=1531151339,mgr=273342918,mgr_metadata=2109538692,mgrstat=3577788595,mon_config_key=1518680965,monmap=970648238,osd_metadata=612206294,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3678135256}) 2024-09-16T15:00:07.346 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.342+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.346 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.342+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=43} crc {logm=2115265916,mdsmap=2147943554,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=1041356535,mon_config_key=1766424633,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=2973260763}) 2024-09-16T15:00:07.346 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.342+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=46} crc {logm=2115265916,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3577788595,mon_config_key=3840829532,monmap=970648238,osd_metadata=2172984177,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=2410268095}) 2024-09-16T15:00:07.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.358+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:00:07.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.358+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=9} crc {osdmap=1682338693}) 2024-09-16T15:00:07.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:00:07.358+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=10} crc {osdmap=4071426683}) 2024-09-16T15:00:08.054 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T15:00:08.055 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T15:00:08.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 10 monclient: get_monmap_and_config 2024-09-16T15:00:08.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 10 monclient: build_initial_monmap 2024-09-16T15:00:08.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 10 monclient: monmap: 2024-09-16T15:00:08.185 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:08.185 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.185 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:00:08.186810+0000 2024-09-16T15:00:08.185 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:00:08.186810+0000 2024-09-16T15:00:08.185 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:08.185 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.185 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:08.185 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:00:08.185 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:00:08.185 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 1 Processor -- start 2024-09-16T15:00:08.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 1 -- start start 2024-09-16T15:00:08.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 10 monclient: init 2024-09-16T15:00:08.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:08.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:00:08.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 10 monclient(hunting): picked mon.noname-b con 0x7f25b009a550 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:08.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 10 monclient(hunting): picked mon.noname-c con 0x7f25b009d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:08.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 10 monclient(hunting): picked mon.noname-a con 0x7f25b009b040 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:08.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b009dc60 con 0x7f25b009b040 2024-09-16T15:00:08.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b0099ae0 con 0x7f25b009d1b0 2024-09-16T15:00:08.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b009b8f0 con 0x7f25b009a550 2024-09-16T15:00:08.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:08.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 10 monclient(hunting): authenticate will time out at 2032.306855s 2024-09-16T15:00:08.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ae59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f25b009a550 0x7f25b009a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43202/0 (socket says 172.21.15.26:43202) 2024-09-16T15:00:08.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ae59c640 1 -- 172.21.15.26:0/2084586754 learned_addr learned my addr 172.21.15.26:0/2084586754 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:08.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 1 -- 172.21.15.26:0/2084586754 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893473852 0 0) 0x7f25b009dc60 con 0x7f25b009b040 2024-09-16T15:00:08.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient(hunting): my global_id is 334131 2024-09-16T15:00:08.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 1 -- 172.21.15.26:0/2084586754 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2598003530 con 0x7f25b009b040 2024-09-16T15:00:08.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 1 -- 172.21.15.26:0/2084586754 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373702721 0 0) 0x7f25b009b8f0 con 0x7f25b009a550 2024-09-16T15:00:08.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient(hunting): my global_id is 284663 2024-09-16T15:00:08.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 1 -- 172.21.15.26:0/2084586754 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25b009dc60 con 0x7f25b009a550 2024-09-16T15:00:08.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 1 -- 172.21.15.26:0/2084586754 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3591875772 0 0) 0x7f25b009dc60 con 0x7f25b009a550 2024-09-16T15:00:08.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 1 -- 172.21.15.26:0/2084586754 >> v1:172.21.15.189:6789/0 conn(0x7f25b009d1b0 legacy=0x7f25b0136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 1 -- 172.21.15.26:0/2084586754 >> v1:172.21.15.26:6789/0 conn(0x7f25b009b040 legacy=0x7f25b013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient: _finish_hunting 0 2024-09-16T15:00:08.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 1 monclient: found mon.noname-b 2024-09-16T15:00:08.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:08.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:00:08.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 1 -- 172.21.15.26:0/2084586754 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b009bbd0 con 0x7f25b009a550 2024-09-16T15:00:08.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient: _finish_auth 0 2024-09-16T15:00:08.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient: _check_auth_tickets 2024-09-16T15:00:08.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:08.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 1 -- 172.21.15.26:0/2084586754 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f25a00015a0 con 0x7f25b009a550 2024-09-16T15:00:08.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:00:08.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient: dump: 2024-09-16T15:00:08.195 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.195 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.195 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.195 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.195 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.195 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.195 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.195 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.195 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.196 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 5 monclient: authenticate success, global_id 284663 2024-09-16T15:00:08.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:08.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 1 -- 172.21.15.26:0/2084586754 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25a0001750 con 0x7f25b009a550 2024-09-16T15:00:08.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:08.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 1 -- 172.21.15.26:0/2084586754 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f25a0001b20 con 0x7f25b009a550 2024-09-16T15:00:08.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:08.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25ad59a640 10 monclient: dump: 2024-09-16T15:00:08.196 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.196 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.196 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.197 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.197 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.197 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.197 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.197 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.197 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.197 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 10 monclient: get_monmap_and_config success 2024-09-16T15:00:08.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.182+0000 7f25b49ad640 4 set_mon_vals no callback set 2024-09-16T15:00:08.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient: shutdown 2024-09-16T15:00:08.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:08.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 1 -- 172.21.15.26:0/2084586754 >> v1:172.21.15.26:6790/0 conn(0x7f25b009a550 legacy=0x7f25b009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 1 -- 172.21.15.26:0/2084586754 shutdown_connections 2024-09-16T15:00:08.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 1 -- 172.21.15.26:0/2084586754 >> 172.21.15.26:0/2084586754 conn(0x7f25b009fb50 msgr2=0x7f25b009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:08.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 1 -- 172.21.15.26:0/2084586754 shutdown_connections 2024-09-16T15:00:08.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 1 -- 172.21.15.26:0/2084586754 wait complete. 2024-09-16T15:00:08.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient: build_initial_monmap 2024-09-16T15:00:08.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient: monmap: 2024-09-16T15:00:08.198 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:08.198 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:00:08.198 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:00:08.198 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:00:08.198 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:08.198 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:00:08.198 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:08.198 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:00:08.199 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:00:08.199 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 1 Processor -- start 2024-09-16T15:00:08.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 1 -- start start 2024-09-16T15:00:08.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient: init 2024-09-16T15:00:08.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:08.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:00:08.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient(hunting): picked mon.noname-b con 0x7f25b009a550 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:08.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient(hunting): picked mon.noname-c con 0x7f25b009b040 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:08.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient(hunting): picked mon.noname-a con 0x7f25b009d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:08.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b01390d0 con 0x7f25b009d1b0 2024-09-16T15:00:08.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b01397c0 con 0x7f25b009a550 2024-09-16T15:00:08.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b0139aa0 con 0x7f25b009b040 2024-09-16T15:00:08.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:08.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient(hunting): authenticate will time out at 2032.310103s 2024-09-16T15:00:08.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25add9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f25b009b040 0x7f25b01308a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43218/0 (socket says 172.21.15.26:43218) 2024-09-16T15:00:08.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25aed9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f25b009d1b0 0x7f25b0130fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42284/0 (socket says 172.21.15.26:42284) 2024-09-16T15:00:08.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25add9b640 1 -- 172.21.15.26:0/1359040412 learned_addr learned my addr 172.21.15.26:0/1359040412 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:08.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444642773 0 0) 0x7f25b01390d0 con 0x7f25b009d1b0 2024-09-16T15:00:08.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): my global_id is 334134 2024-09-16T15:00:08.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2590005740 con 0x7f25b009d1b0 2024-09-16T15:00:08.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330005704 0 0) 0x7f25b0139aa0 con 0x7f25b009b040 2024-09-16T15:00:08.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): my global_id is 284666 2024-09-16T15:00:08.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25b01390d0 con 0x7f25b009b040 2024-09-16T15:00:08.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1985240856 0 0) 0x7f2590005740 con 0x7f25b009d1b0 2024-09-16T15:00:08.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25b0139aa0 con 0x7f25b009d1b0 2024-09-16T15:00:08.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 185217469 0 0) 0x7f25b01390d0 con 0x7f25b009b040 2024-09-16T15:00:08.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2590005740 con 0x7f25b009b040 2024-09-16T15:00:08.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f25a8001c60 con 0x7f25b009d1b0 2024-09-16T15:00:08.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:00:08.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): dump: 2024-09-16T15:00:08.202 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.202 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.202 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.202 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.202 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.202 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.202 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.202 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.202 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.202 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:00:08.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530555043 0 0) 0x7f25b01397c0 con 0x7f25b009a550 2024-09-16T15:00:08.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): my global_id is 304399 2024-09-16T15:00:08.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25b01390d0 con 0x7f25b009a550 2024-09-16T15:00:08.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f25a4001940 con 0x7f25b009b040 2024-09-16T15:00:08.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:08.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient(hunting): dump: 2024-09-16T15:00:08.203 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.203 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.203 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.203 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.203 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.204 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.204 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.204 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.204 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.204 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 679796403 0 0) 0x7f25b0139aa0 con 0x7f25b009d1b0 2024-09-16T15:00:08.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 >> v1:172.21.15.26:6790/0 conn(0x7f25b009b040 legacy=0x7f25b01308a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 >> v1:172.21.15.189:6789/0 conn(0x7f25b009a550 legacy=0x7f25b013a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient: _finish_hunting 0 2024-09-16T15:00:08.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 monclient: found mon.a 2024-09-16T15:00:08.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:08.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:08.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b0139d80 con 0x7f25b009d1b0 2024-09-16T15:00:08.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient: _finish_auth 0 2024-09-16T15:00:08.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient: _check_auth_tickets 2024-09-16T15:00:08.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:08.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 5 monclient: authenticate success, global_id 334134 2024-09-16T15:00:08.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient: _renew_subs 2024-09-16T15:00:08.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:08.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 1 -- 172.21.15.26:0/1359040412 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b01388d0 con 0x7f25b009d1b0 2024-09-16T15:00:08.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient: _renew_subs 2024-09-16T15:00:08.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:08.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 1 -- 172.21.15.26:0/1359040412 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b009e2e0 con 0x7f25b009d1b0 2024-09-16T15:00:08.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25a8003980 con 0x7f25b009d1b0 2024-09-16T15:00:08.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:08.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f25a8003d10 con 0x7f25b009d1b0 2024-09-16T15:00:08.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:08.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259ffff640 4 set_mon_vals no callback set 2024-09-16T15:00:08.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f259effd640 10 monclient: dump: 2024-09-16T15:00:08.206 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.206 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.206 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.206 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.206 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.206 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.206 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.206 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.206 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.206 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:08.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:08.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:08.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.186+0000 7f25b49ad640 1 -- 172.21.15.26:0/1359040412 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2594005050 con 0x7f25b009d1b0 2024-09-16T15:00:08.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.190+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f25a8003e70 con 0x7f25b009d1b0 2024-09-16T15:00:08.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.190+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f25a8043f30 con 0x7f25b009d1b0 2024-09-16T15:00:08.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.190+0000 7f25ae59c640 10 monclient: get_auth_request con 0x7f2590038610 auth_method 0 2024-09-16T15:00:08.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.194+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f25a8004620 con 0x7f25b009d1b0 2024-09-16T15:00:08.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.194+0000 7f259effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:08.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.194+0000 7f259effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:00:08.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.406+0000 7f25b49ad640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T15:00:08.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.406+0000 7f25b49ad640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T15:00:08.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.406+0000 7f25b49ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:08.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.406+0000 7f25b49ad640 1 -- 172.21.15.26:0/1359040412 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2594005880 con 0x7f25b009d1b0 2024-09-16T15:00:08.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.406+0000 7f259effd640 1 -- 172.21.15.26:0/1359040412 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 117972162) 0x7f25a801ce60 con 0x7f25b009d1b0 2024-09-16T15:00:08.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.406+0000 7f259effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T15:00:08.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.406+0000 7f259effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:00:08.409 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T15:00:08.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.406+0000 7f25b49ad640 1 -- 172.21.15.26:0/1359040412 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2590038610 legacy=0x7f259003aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.406+0000 7f25b49ad640 10 monclient: shutdown 2024-09-16T15:00:08.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.406+0000 7f25b49ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:08.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.406+0000 7f25b49ad640 1 -- 172.21.15.26:0/1359040412 >> v1:172.21.15.26:6789/0 conn(0x7f25b009d1b0 legacy=0x7f25b0130fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.406+0000 7f25b49ad640 1 -- 172.21.15.26:0/1359040412 shutdown_connections 2024-09-16T15:00:08.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.406+0000 7f25b49ad640 1 -- 172.21.15.26:0/1359040412 >> 172.21.15.26:0/1359040412 conn(0x7f25b009fb50 msgr2=0x7f25b0133a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:08.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.406+0000 7f25b49ad640 1 -- 172.21.15.26:0/1359040412 shutdown_connections 2024-09-16T15:00:08.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.406+0000 7f25b49ad640 1 -- 172.21.15.26:0/1359040412 wait complete. 2024-09-16T15:00:08.425 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T15:00:08.425 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T15:00:08.425 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T15:00:08.426 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T15:00:08.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.542+0000 7fb1904b5640 10 monclient: get_monmap_and_config 2024-09-16T15:00:08.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.542+0000 7fb1904b5640 10 monclient: build_initial_monmap 2024-09-16T15:00:08.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.542+0000 7fb1904b5640 10 monclient: monmap: 2024-09-16T15:00:08.547 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:08.547 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.547 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:00:08.548935+0000 2024-09-16T15:00:08.547 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:00:08.548935+0000 2024-09-16T15:00:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:00:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:00:08.548 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 1 Processor -- start 2024-09-16T15:00:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 1 -- start start 2024-09-16T15:00:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 10 monclient: init 2024-09-16T15:00:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:00:08.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 10 monclient(hunting): picked mon.noname-b con 0x7fb18809a550 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 10 monclient(hunting): picked mon.noname-a con 0x7fb18809d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 10 monclient(hunting): picked mon.noname-c con 0x7fb18809b040 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb18809dc60 con 0x7fb18809d1b0 2024-09-16T15:00:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb188099ae0 con 0x7fb18809b040 2024-09-16T15:00:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb18809b8f0 con 0x7fb18809a550 2024-09-16T15:00:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 10 monclient(hunting): authenticate will time out at 2032.668775s 2024-09-16T15:00:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18e251640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb18809a550 0x7fb18809a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43226/0 (socket says 172.21.15.26:43226) 2024-09-16T15:00:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18e251640 1 -- 172.21.15.26:0/191433643 learned_addr learned my addr 172.21.15.26:0/191433643 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 1 -- 172.21.15.26:0/191433643 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645039213 0 0) 0x7fb18809dc60 con 0x7fb18809d1b0 2024-09-16T15:00:08.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient(hunting): my global_id is 334137 2024-09-16T15:00:08.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 1 -- 172.21.15.26:0/191433643 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb170003530 con 0x7fb18809d1b0 2024-09-16T15:00:08.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 1 -- 172.21.15.26:0/191433643 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267302229 0 0) 0x7fb18809b8f0 con 0x7fb18809a550 2024-09-16T15:00:08.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient(hunting): my global_id is 284669 2024-09-16T15:00:08.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 1 -- 172.21.15.26:0/191433643 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb18809dc60 con 0x7fb18809a550 2024-09-16T15:00:08.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 1 -- 172.21.15.26:0/191433643 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2269452741 0 0) 0x7fb170003530 con 0x7fb18809d1b0 2024-09-16T15:00:08.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 1 -- 172.21.15.26:0/191433643 >> v1:172.21.15.26:6790/0 conn(0x7fb18809a550 legacy=0x7fb18809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 1 -- 172.21.15.26:0/191433643 >> v1:172.21.15.189:6789/0 conn(0x7fb18809b040 legacy=0x7fb18813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient: _finish_hunting 0 2024-09-16T15:00:08.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 1 monclient: found mon.noname-a 2024-09-16T15:00:08.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:08.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:00:08.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 1 -- 172.21.15.26:0/191433643 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18809bbd0 con 0x7fb18809d1b0 2024-09-16T15:00:08.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient: _finish_auth 0 2024-09-16T15:00:08.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient: _check_auth_tickets 2024-09-16T15:00:08.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:08.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 1 -- 172.21.15.26:0/191433643 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb17c001620 con 0x7fb18809d1b0 2024-09-16T15:00:08.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:00:08.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient: dump: 2024-09-16T15:00:08.564 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.565 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.565 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.565 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.565 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.565 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.565 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.566 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.566 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.566 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 5 monclient: authenticate success, global_id 334137 2024-09-16T15:00:08.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:08.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 1 -- 172.21.15.26:0/191433643 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb17c0018c0 con 0x7fb18809d1b0 2024-09-16T15:00:08.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:08.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 1 -- 172.21.15.26:0/191433643 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb17c001c50 con 0x7fb18809d1b0 2024-09-16T15:00:08.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:08.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb18d24f640 10 monclient: dump: 2024-09-16T15:00:08.568 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.568 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.568 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.568 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.568 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.568 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.568 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.569 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.569 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.569 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 10 monclient: get_monmap_and_config success 2024-09-16T15:00:08.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 4 set_mon_vals no callback set 2024-09-16T15:00:08.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 10 monclient: shutdown 2024-09-16T15:00:08.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:08.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 1 -- 172.21.15.26:0/191433643 >> v1:172.21.15.26:6789/0 conn(0x7fb18809d1b0 legacy=0x7fb188136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 1 -- 172.21.15.26:0/191433643 shutdown_connections 2024-09-16T15:00:08.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 1 -- 172.21.15.26:0/191433643 >> 172.21.15.26:0/191433643 conn(0x7fb18809fb50 msgr2=0x7fb18809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:08.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 1 -- 172.21.15.26:0/191433643 shutdown_connections 2024-09-16T15:00:08.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 1 -- 172.21.15.26:0/191433643 wait complete. 2024-09-16T15:00:08.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 10 monclient: build_initial_monmap 2024-09-16T15:00:08.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 10 monclient: monmap: 2024-09-16T15:00:08.571 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:08.571 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:00:08.572 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:00:08.572 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:00:08.572 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:08.572 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:00:08.572 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:08.573 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:00:08.573 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:00:08.573 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 1 Processor -- start 2024-09-16T15:00:08.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 1 -- start start 2024-09-16T15:00:08.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.546+0000 7fb1904b5640 10 monclient: init 2024-09-16T15:00:08.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:08.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:00:08.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 10 monclient(hunting): picked mon.noname-c con 0x7fb18809a550 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:08.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 10 monclient(hunting): picked mon.noname-a con 0x7fb18809b040 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:08.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 10 monclient(hunting): picked mon.noname-b con 0x7fb18809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:08.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb18806c8b0 con 0x7fb18809b040 2024-09-16T15:00:08.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb18806d480 con 0x7fb18809d1b0 2024-09-16T15:00:08.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1880709a0 con 0x7fb18809a550 2024-09-16T15:00:08.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:08.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 10 monclient(hunting): authenticate will time out at 2032.672475s 2024-09-16T15:00:08.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb18e251640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb18809a550 0x7fb1881c9940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43238/0 (socket says 172.21.15.26:43238) 2024-09-16T15:00:08.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb18da50640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb18809b040 0x7fb1881ce920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42304/0 (socket says 172.21.15.26:42304) 2024-09-16T15:00:08.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb18e251640 1 -- 172.21.15.26:0/4218413881 learned_addr learned my addr 172.21.15.26:0/4218413881 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:08.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159947910 0 0) 0x7fb1880709a0 con 0x7fb18809a550 2024-09-16T15:00:08.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): my global_id is 284672 2024-09-16T15:00:08.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb174003530 con 0x7fb18809a550 2024-09-16T15:00:08.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679912061 0 0) 0x7fb18806c8b0 con 0x7fb18809b040 2024-09-16T15:00:08.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): my global_id is 334140 2024-09-16T15:00:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1880709a0 con 0x7fb18809b040 2024-09-16T15:00:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2086568907 0 0) 0x7fb174003530 con 0x7fb18809a550 2024-09-16T15:00:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb18806c8b0 con 0x7fb18809a550 2024-09-16T15:00:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb180001c60 con 0x7fb18809a550 2024-09-16T15:00:08.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:00:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): dump: 2024-09-16T15:00:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.579 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.581 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:00:08.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2547423209 0 0) 0x7fb1880709a0 con 0x7fb18809b040 2024-09-16T15:00:08.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb174003530 con 0x7fb18809b040 2024-09-16T15:00:08.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb17c001bd0 con 0x7fb18809b040 2024-09-16T15:00:08.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:08.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): dump: 2024-09-16T15:00:08.582 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.582 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.582 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.582 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.583 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.583 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.583 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.583 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.583 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.584 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161354858 0 0) 0x7fb18806d480 con 0x7fb18809d1b0 2024-09-16T15:00:08.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): my global_id is 304405 2024-09-16T15:00:08.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1880709a0 con 0x7fb18809d1b0 2024-09-16T15:00:08.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 412712417 0 0) 0x7fb174003530 con 0x7fb18809b040 2024-09-16T15:00:08.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 >> v1:172.21.15.26:6790/0 conn(0x7fb18809a550 legacy=0x7fb1881c9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 >> v1:172.21.15.189:6789/0 conn(0x7fb18809d1b0 legacy=0x7fb1881d2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient: _finish_hunting 0 2024-09-16T15:00:08.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 monclient: found mon.a 2024-09-16T15:00:08.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:08.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:08.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18806e370 con 0x7fb18809b040 2024-09-16T15:00:08.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient: _finish_auth 0 2024-09-16T15:00:08.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient: _check_auth_tickets 2024-09-16T15:00:08.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:08.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 5 monclient: authenticate success, global_id 334140 2024-09-16T15:00:08.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 10 monclient: _renew_subs 2024-09-16T15:00:08.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:08.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 1 -- 172.21.15.26:0/4218413881 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb18811ddf0 con 0x7fb18809b040 2024-09-16T15:00:08.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 10 monclient: _renew_subs 2024-09-16T15:00:08.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:08.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 1 -- 172.21.15.26:0/4218413881 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb18806cfc0 con 0x7fb18809b040 2024-09-16T15:00:08.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb17c001170 con 0x7fb18809b040 2024-09-16T15:00:08.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:08.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb17c003b60 con 0x7fb18809b040 2024-09-16T15:00:08.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:08.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb18ca4e640 4 set_mon_vals no callback set 2024-09-16T15:00:08.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient: dump: 2024-09-16T15:00:08.590 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.590 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.590 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.590 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.590 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.590 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.591 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.591 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.591 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.591 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:08.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb1904b5640 1 -- 172.21.15.26:0/4218413881 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb168006590 con 0x7fb168002830 2024-09-16T15:00:08.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb17c003cc0 con 0x7fb18809b040 2024-09-16T15:00:08.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fb17c043cc0 con 0x7fb18809b040 2024-09-16T15:00:08.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365808170 0 0) 0x7fb168006590 con 0x7fb168002830 2024-09-16T15:00:08.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient(hunting): my global_id is 334143 2024-09-16T15:00:08.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb174003530 con 0x7fb168002830 2024-09-16T15:00:08.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1960434375 0 0) 0x7fb174003530 con 0x7fb168002830 2024-09-16T15:00:08.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb168006590 con 0x7fb168002830 2024-09-16T15:00:08.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb180004f00 con 0x7fb168002830 2024-09-16T15:00:08.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb18ea52640 10 monclient: get_auth_request con 0x7fb174036390 auth_method 0 2024-09-16T15:00:08.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:08.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient: dump: 2024-09-16T15:00:08.594 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.594 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.596 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.596 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1653183874 0 0) 0x7fb168006590 con 0x7fb168002830 2024-09-16T15:00:08.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb16800a0e0 con 0x7fb168002830 2024-09-16T15:00:08.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fb180007290 con 0x7fb168002830 2024-09-16T15:00:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:00:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 >> v1:172.21.15.26:6789/0 conn(0x7fb168002830 legacy=0x7fb168004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.550+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 >> v1:172.21.15.26:6789/0 conn(0x7fb168002830 legacy=0x7fb168004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:00:08.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb1904b5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:00:08.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb1904b5640 1 -- 172.21.15.26:0/4218413881 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb168009d80 con 0x7fb1680090a0 2024-09-16T15:00:08.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677298215 0 0) 0x7fb168009d80 con 0x7fb1680090a0 2024-09-16T15:00:08.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 10 monclient(hunting): my global_id is 334146 2024-09-16T15:00:08.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb168006590 con 0x7fb1680090a0 2024-09-16T15:00:08.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1956428976 0 0) 0x7fb168006590 con 0x7fb1680090a0 2024-09-16T15:00:08.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb168009d80 con 0x7fb1680090a0 2024-09-16T15:00:08.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb180005650 con 0x7fb1680090a0 2024-09-16T15:00:08.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:08.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 10 monclient: dump: 2024-09-16T15:00:08.602 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.602 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.603 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.603 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.603 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.603 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.603 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.604 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3857088414 0 0) 0x7fb168009d80 con 0x7fb1680090a0 2024-09-16T15:00:08.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb16800c290 con 0x7fb1680090a0 2024-09-16T15:00:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3519114758) 0x7fb180006b40 con 0x7fb1680090a0 2024-09-16T15:00:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:00:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:00:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 >> v1:172.21.15.26:6789/0 conn(0x7fb1680090a0 legacy=0x7fb168009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb186ffd640 1 -- 172.21.15.26:0/4218413881 >> v1:172.21.15.26:6789/0 conn(0x7fb1680090a0 legacy=0x7fb168009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:00:08.606 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:00:08.606 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:00:08.606 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:00:08.606 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:00:08.607 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 162, 2024-09-16T15:00:08.607 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:00:08.607 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:00:08.607 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:00:08.607 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:00:08.607 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:08.608 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T15:00:08.608 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:00:08.608 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:00:08.608 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:00:08.608 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:08.608 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:08.609 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:08.609 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:08.609 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:08.609 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:08.609 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:00:08.610 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:00:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:08.611 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:08.612 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:00:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:00:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:00:08.613 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:00:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:00:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:00:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:00:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:00:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:00:08.614 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:00:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:00:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:00:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:00:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:00:08.615 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:00:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:00:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:00:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:08.616 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:08.617 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:00:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:00:08.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:00:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:00:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:00:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:00:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:00:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:08.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:00:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:00:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:08.622 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:00:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:00:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:00:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:00:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:00:08.624 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:00:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:00:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:00:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:08.625 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:00:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:00:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:00:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:00:08.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T15:00:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.627 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:00:08.627 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:00:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb1904b5640 1 -- 172.21.15.26:0/4218413881 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb174036390 legacy=0x7fb174038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb1904b5640 10 monclient: shutdown 2024-09-16T15:00:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb1904b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb1904b5640 1 -- 172.21.15.26:0/4218413881 >> v1:172.21.15.26:6789/0 conn(0x7fb18809b040 legacy=0x7fb1881ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb18ea52640 1 -- 172.21.15.26:0/4218413881 reap_dead start 2024-09-16T15:00:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb18ea52640 1 -- 172.21.15.26:0/4218413881 reap_dead start 2024-09-16T15:00:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb1904b5640 1 -- 172.21.15.26:0/4218413881 shutdown_connections 2024-09-16T15:00:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb1904b5640 1 -- 172.21.15.26:0/4218413881 >> 172.21.15.26:0/4218413881 conn(0x7fb18809fb50 msgr2=0x7fb18809e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb1904b5640 1 -- 172.21.15.26:0/4218413881 shutdown_connections 2024-09-16T15:00:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.558+0000 7fb1904b5640 1 -- 172.21.15.26:0/4218413881 wait complete. 2024-09-16T15:00:08.629 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:00:08.629 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T15:00:08.630 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T15:00:08.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 10 monclient: get_monmap_and_config 2024-09-16T15:00:08.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 10 monclient: build_initial_monmap 2024-09-16T15:00:08.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 10 monclient: monmap: 2024-09-16T15:00:08.751 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:08.752 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.752 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:00:08.753804+0000 2024-09-16T15:00:08.752 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:00:08.753804+0000 2024-09-16T15:00:08.752 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:08.752 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.752 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:08.752 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:00:08.752 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:00:08.752 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 1 Processor -- start 2024-09-16T15:00:08.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 1 -- start start 2024-09-16T15:00:08.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 10 monclient: init 2024-09-16T15:00:08.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:08.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:00:08.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 10 monclient(hunting): picked mon.noname-c con 0x7fe9ac09a550 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:08.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 10 monclient(hunting): picked mon.noname-b con 0x7fe9ac09d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:08.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 10 monclient(hunting): picked mon.noname-a con 0x7fe9ac09b040 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:08.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac09dc60 con 0x7fe9ac09b040 2024-09-16T15:00:08.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac099ae0 con 0x7fe9ac09a550 2024-09-16T15:00:08.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac09b8f0 con 0x7fe9ac09d1b0 2024-09-16T15:00:08.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:08.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 10 monclient(hunting): authenticate will time out at 2032.873800s 2024-09-16T15:00:08.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a9d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe9ac09d1b0 0x7fe9ac136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43246/0 (socket says 172.21.15.26:43246) 2024-09-16T15:00:08.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a9d9b640 1 -- 172.21.15.26:0/692933412 learned_addr learned my addr 172.21.15.26:0/692933412 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:08.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 1 -- 172.21.15.26:0/692933412 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233698677 0 0) 0x7fe9ac09b8f0 con 0x7fe9ac09d1b0 2024-09-16T15:00:08.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient(hunting): my global_id is 284675 2024-09-16T15:00:08.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 1 -- 172.21.15.26:0/692933412 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe994003530 con 0x7fe9ac09d1b0 2024-09-16T15:00:08.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 1 -- 172.21.15.26:0/692933412 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394377000 0 0) 0x7fe9ac09dc60 con 0x7fe9ac09b040 2024-09-16T15:00:08.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient(hunting): my global_id is 334149 2024-09-16T15:00:08.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 1 -- 172.21.15.26:0/692933412 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9ac09b8f0 con 0x7fe9ac09b040 2024-09-16T15:00:08.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 1 -- 172.21.15.26:0/692933412 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2672251077 0 0) 0x7fe994003530 con 0x7fe9ac09d1b0 2024-09-16T15:00:08.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 1 -- 172.21.15.26:0/692933412 >> v1:172.21.15.189:6789/0 conn(0x7fe9ac09a550 legacy=0x7fe9ac09a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 1 -- 172.21.15.26:0/692933412 >> v1:172.21.15.26:6789/0 conn(0x7fe9ac09b040 legacy=0x7fe9ac13a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient: _finish_hunting 0 2024-09-16T15:00:08.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 1 monclient: found mon.noname-b 2024-09-16T15:00:08.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:00:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 1 -- 172.21.15.26:0/692933412 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ac09bbd0 con 0x7fe9ac09d1b0 2024-09-16T15:00:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient: _finish_auth 0 2024-09-16T15:00:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient: _check_auth_tickets 2024-09-16T15:00:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 1 -- 172.21.15.26:0/692933412 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe9a0001620 con 0x7fe9ac09d1b0 2024-09-16T15:00:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 5 monclient: authenticate success, global_id 284675 2024-09-16T15:00:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:00:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient: dump: 2024-09-16T15:00:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.762 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.763 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 1 -- 172.21.15.26:0/692933412 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9a0001840 con 0x7fe9ac09d1b0 2024-09-16T15:00:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 1 -- 172.21.15.26:0/692933412 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe9a0001bd0 con 0x7fe9ac09d1b0 2024-09-16T15:00:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:08.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9a959a640 10 monclient: dump: 2024-09-16T15:00:08.764 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.764 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.764 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.764 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.764 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.764 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.764 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.764 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.764 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.764 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 10 monclient: get_monmap_and_config success 2024-09-16T15:00:08.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 4 set_mon_vals no callback set 2024-09-16T15:00:08.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 10 monclient: shutdown 2024-09-16T15:00:08.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:08.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 1 -- 172.21.15.26:0/692933412 >> v1:172.21.15.26:6790/0 conn(0x7fe9ac09d1b0 legacy=0x7fe9ac136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 1 -- 172.21.15.26:0/692933412 shutdown_connections 2024-09-16T15:00:08.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 1 -- 172.21.15.26:0/692933412 >> 172.21.15.26:0/692933412 conn(0x7fe9ac09fb50 msgr2=0x7fe9ac09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:08.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 1 -- 172.21.15.26:0/692933412 shutdown_connections 2024-09-16T15:00:08.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.750+0000 7fe9b0c62640 1 -- 172.21.15.26:0/692933412 wait complete. 2024-09-16T15:00:08.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 10 monclient: build_initial_monmap 2024-09-16T15:00:08.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 10 monclient: monmap: 2024-09-16T15:00:08.765 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:08.765 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:00:08.765 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:00:08.765 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:00:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:00:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:00:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:00:08.766 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 1 Processor -- start 2024-09-16T15:00:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 1 -- start start 2024-09-16T15:00:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 10 monclient: init 2024-09-16T15:00:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:00:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 10 monclient(hunting): picked mon.noname-c con 0x7fe9ac09a550 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 10 monclient(hunting): picked mon.noname-a con 0x7fe9ac09b040 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:08.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 10 monclient(hunting): picked mon.noname-b con 0x7fe9ac09d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:08.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac09ee90 con 0x7fe9ac09b040 2024-09-16T15:00:08.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac13b710 con 0x7fe9ac09d1b0 2024-09-16T15:00:08.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac137930 con 0x7fe9ac09a550 2024-09-16T15:00:08.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:08.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 10 monclient(hunting): authenticate will time out at 2032.877038s 2024-09-16T15:00:08.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9aa59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe9ac09a550 0x7fe9ac136b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43260/0 (socket says 172.21.15.26:43260) 2024-09-16T15:00:08.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9aa59c640 1 -- 172.21.15.26:0/1122603648 learned_addr learned my addr 172.21.15.26:0/1122603648 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:08.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009286161 0 0) 0x7fe9ac137930 con 0x7fe9ac09a550 2024-09-16T15:00:08.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): my global_id is 284678 2024-09-16T15:00:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe98c003530 con 0x7fe9ac09a550 2024-09-16T15:00:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129384783 0 0) 0x7fe9ac09ee90 con 0x7fe9ac09b040 2024-09-16T15:00:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): my global_id is 334152 2024-09-16T15:00:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9ac137930 con 0x7fe9ac09b040 2024-09-16T15:00:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2517857834 0 0) 0x7fe98c003530 con 0x7fe9ac09a550 2024-09-16T15:00:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9ac09ee90 con 0x7fe9ac09a550 2024-09-16T15:00:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe99c001980 con 0x7fe9ac09a550 2024-09-16T15:00:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:00:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): dump: 2024-09-16T15:00:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.769 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:00:08.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144732215 0 0) 0x7fe9a40015c0 con 0x7fe9ac09d1b0 2024-09-16T15:00:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): my global_id is 304411 2024-09-16T15:00:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe98c003530 con 0x7fe9ac09d1b0 2024-09-16T15:00:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4147632347 0 0) 0x7fe9ac09ee90 con 0x7fe9ac09a550 2024-09-16T15:00:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 >> v1:172.21.15.189:6789/0 conn(0x7fe9ac09d1b0 legacy=0x7fe9ac1d1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 >> v1:172.21.15.26:6789/0 conn(0x7fe9ac09b040 legacy=0x7fe9ac137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient: _finish_hunting 0 2024-09-16T15:00:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 monclient: found mon.c 2024-09-16T15:00:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:00:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ac139110 con 0x7fe9ac09a550 2024-09-16T15:00:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient: _finish_auth 0 2024-09-16T15:00:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient: _check_auth_tickets 2024-09-16T15:00:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 5 monclient: authenticate success, global_id 284678 2024-09-16T15:00:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 10 monclient: _renew_subs 2024-09-16T15:00:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:00:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 1 -- 172.21.15.26:0/1122603648 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ac1393f0 con 0x7fe9ac09a550 2024-09-16T15:00:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 10 monclient: _renew_subs 2024-09-16T15:00:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:00:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 1 -- 172.21.15.26:0/1122603648 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ac139dd0 con 0x7fe9ac09a550 2024-09-16T15:00:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe99c003660 con 0x7fe9ac09a550 2024-09-16T15:00:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe99c0039f0 con 0x7fe9ac09a550 2024-09-16T15:00:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9a8d99640 4 set_mon_vals no callback set 2024-09-16T15:00:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient: dump: 2024-09-16T15:00:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.772 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.773 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9b0c62640 1 -- 172.21.15.26:0/1122603648 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe990006590 con 0x7fe990002830 2024-09-16T15:00:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fe99c003bd0 con 0x7fe9ac09a550 2024-09-16T15:00:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fe99c043e70 con 0x7fe9ac09a550 2024-09-16T15:00:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904879821 0 0) 0x7fe990006590 con 0x7fe990002830 2024-09-16T15:00:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient(hunting): my global_id is 284681 2024-09-16T15:00:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9ac09ee90 con 0x7fe990002830 2024-09-16T15:00:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1869133227 0 0) 0x7fe9ac09ee90 con 0x7fe990002830 2024-09-16T15:00:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe990006590 con 0x7fe990002830 2024-09-16T15:00:08.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe9a0004990 con 0x7fe990002830 2024-09-16T15:00:08.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe9aad9d640 10 monclient: get_auth_request con 0x7fe98c0360a0 auth_method 0 2024-09-16T15:00:08.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:08.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 10 monclient: dump: 2024-09-16T15:00:08.774 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.774 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.774 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.774 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.774 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.774 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.774 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.774 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.775 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.775 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3928977060 0 0) 0x7fe990006590 con 0x7fe990002830 2024-09-16T15:00:08.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.754+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe99000a0e0 con 0x7fe990002830 2024-09-16T15:00:08.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.758+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe9a0004d00 con 0x7fe990002830 2024-09-16T15:00:08.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.758+0000 7fe99affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:08.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.758+0000 7fe99affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:00:08.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.758+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 >> v1:172.21.15.26:6790/0 conn(0x7fe990002830 legacy=0x7fe990004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.758+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 >> v1:172.21.15.26:6790/0 conn(0x7fe990002830 legacy=0x7fe990004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:00:08.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe9b0c62640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:00:08.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe9b0c62640 1 -- 172.21.15.26:0/1122603648 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe990009da0 con 0x7fe9900090c0 2024-09-16T15:00:08.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561057134 0 0) 0x7fe990009da0 con 0x7fe9900090c0 2024-09-16T15:00:08.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 10 monclient(hunting): my global_id is 284684 2024-09-16T15:00:08.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe990006590 con 0x7fe9900090c0 2024-09-16T15:00:08.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1487037338 0 0) 0x7fe990006590 con 0x7fe9900090c0 2024-09-16T15:00:08.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe990009da0 con 0x7fe9900090c0 2024-09-16T15:00:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fe9a0005280 con 0x7fe9900090c0 2024-09-16T15:00:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 10 monclient: dump: 2024-09-16T15:00:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.779 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3112465565 0 0) 0x7fe990009da0 con 0x7fe9900090c0 2024-09-16T15:00:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe99000c290 con 0x7fe9900090c0 2024-09-16T15:00:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 2057193970) 0x7fe9a0006710 con 0x7fe9900090c0 2024-09-16T15:00:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:00:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:00:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 >> v1:172.21.15.26:6790/0 conn(0x7fe9900090c0 legacy=0x7fe990009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe99affd640 1 -- 172.21.15.26:0/1122603648 >> v1:172.21.15.26:6790/0 conn(0x7fe9900090c0 legacy=0x7fe990009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:00:08.779 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:00:08.779 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:00:08.780 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:00:08.780 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T15:00:08.780 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 162, 2024-09-16T15:00:08.780 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:00:08.780 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:00:08.780 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:00:08.780 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:00:08.780 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:08.780 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T15:00:08.780 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:00:08.780 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:00:08.780 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:00:08.780 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:08.781 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:08.781 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:08.781 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:08.781 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:08.781 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:08.781 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:08.781 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:08.781 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:08.781 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:08.781 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:08.781 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:00:08.781 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:00:08.781 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:08.782 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:08.782 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:08.782 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:08.782 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:08.782 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:08.782 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:08.782 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:08.782 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:08.782 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:08.782 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.782 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:00:08.782 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:00:08.782 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:00:08.783 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:00:08.783 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:00:08.783 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:00:08.783 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:00:08.783 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:00:08.783 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:00:08.783 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:00:08.783 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:00:08.783 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:00:08.783 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:00:08.783 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:00:08.783 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:00:08.783 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:00:08.784 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:00:08.784 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:08.784 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:08.784 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:08.784 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:08.784 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:08.784 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:08.784 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:08.784 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:08.784 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:08.784 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:08.784 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:08.784 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:00:08.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.785 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:00:08.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.785 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:00:08.785 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:00:08.785 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:08.785 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:08.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.785 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:08.785 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:00:08.785 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:08.785 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.785 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.786 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:00:08.786 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:00:08.786 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:08.786 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:08.786 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:08.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.786 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:00:08.786 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:00:08.786 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:08.786 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:08.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.786 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:08.787 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:00:08.787 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:08.787 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.787 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.787 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:00:08.787 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:00:08.787 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:08.787 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:08.787 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:08.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.787 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:00:08.788 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:00:08.788 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:08.788 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:08.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.788 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:08.788 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:00:08.788 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:08.788 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.788 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.788 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:00:08.788 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:00:08.788 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:00:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:00:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:00:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:08.789 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T15:00:08.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.790 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:08.790 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:08.790 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T15:00:08.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.790 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:08.790 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:00:08.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.790 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:08.790 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:08.790 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T15:00:08.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.790 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T15:00:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:00:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.791 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:00:08.791 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:00:08.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe9b0c62640 1 -- 172.21.15.26:0/1122603648 >> v1:172.21.15.189:6812/3196161582 conn(0x7fe98c0360a0 legacy=0x7fe98c038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe9b0c62640 10 monclient: shutdown 2024-09-16T15:00:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe9b0c62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe9b0c62640 1 -- 172.21.15.26:0/1122603648 >> v1:172.21.15.26:6790/0 conn(0x7fe9ac09a550 legacy=0x7fe9ac136b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe9aad9d640 1 -- 172.21.15.26:0/1122603648 reap_dead start 2024-09-16T15:00:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe9aad9d640 1 -- 172.21.15.26:0/1122603648 reap_dead start 2024-09-16T15:00:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe9b0c62640 1 -- 172.21.15.26:0/1122603648 shutdown_connections 2024-09-16T15:00:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe9b0c62640 1 -- 172.21.15.26:0/1122603648 >> 172.21.15.26:0/1122603648 conn(0x7fe9ac09fb50 msgr2=0x7fe9ac1340f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe9b0c62640 1 -- 172.21.15.26:0/1122603648 shutdown_connections 2024-09-16T15:00:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.762+0000 7fe9b0c62640 1 -- 172.21.15.26:0/1122603648 wait complete. 2024-09-16T15:00:08.794 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:00:08.794 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T15:00:08.794 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T15:00:08.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.902+0000 7f3f2a238640 10 monclient: get_monmap_and_config 2024-09-16T15:00:08.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.902+0000 7f3f2a238640 10 monclient: build_initial_monmap 2024-09-16T15:00:08.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.902+0000 7f3f2a238640 10 monclient: monmap: 2024-09-16T15:00:08.907 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:08.907 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.907 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:00:08.909185+0000 2024-09-16T15:00:08.907 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:00:08.909185+0000 2024-09-16T15:00:08.907 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:08.907 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.907 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:08.907 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:00:08.908 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:00:08.908 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 1 Processor -- start 2024-09-16T15:00:08.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 1 -- start start 2024-09-16T15:00:08.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 10 monclient: init 2024-09-16T15:00:08.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:08.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:00:08.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 10 monclient(hunting): picked mon.noname-b con 0x7f3f240a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:08.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 10 monclient(hunting): picked mon.noname-a con 0x7f3f240a0000 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:08.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 10 monclient(hunting): picked mon.noname-c con 0x7f3f240a2800 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:08.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f240a3250 con 0x7f3f240a0000 2024-09-16T15:00:08.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2409f600 con 0x7f3f240a2800 2024-09-16T15:00:08.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f240a3830 con 0x7f3f240a1d10 2024-09-16T15:00:08.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:08.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 10 monclient(hunting): authenticate will time out at 2033.029178s 2024-09-16T15:00:08.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f237fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3f240a1d10 0x7f3f240a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43288/0 (socket says 172.21.15.26:43288) 2024-09-16T15:00:08.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f237fe640 1 -- 172.21.15.26:0/2226033471 learned_addr learned my addr 172.21.15.26:0/2226033471 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:08.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 1 -- 172.21.15.26:0/2226033471 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192901206 0 0) 0x7f3f240a3830 con 0x7f3f240a1d10 2024-09-16T15:00:08.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient(hunting): my global_id is 284687 2024-09-16T15:00:08.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 1 -- 172.21.15.26:0/2226033471 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f0c003530 con 0x7f3f240a1d10 2024-09-16T15:00:08.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 1 -- 172.21.15.26:0/2226033471 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273708823 0 0) 0x7f3f240a3250 con 0x7f3f240a0000 2024-09-16T15:00:08.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient(hunting): my global_id is 334155 2024-09-16T15:00:08.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 1 -- 172.21.15.26:0/2226033471 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f240a3830 con 0x7f3f240a0000 2024-09-16T15:00:08.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 1 -- 172.21.15.26:0/2226033471 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3122972599 0 0) 0x7f3f0c003530 con 0x7f3f240a1d10 2024-09-16T15:00:08.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 1 -- 172.21.15.26:0/2226033471 >> v1:172.21.15.189:6789/0 conn(0x7f3f240a2800 legacy=0x7f3f2413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 1 -- 172.21.15.26:0/2226033471 >> v1:172.21.15.26:6789/0 conn(0x7f3f240a0000 legacy=0x7f3f24139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient: _finish_hunting 0 2024-09-16T15:00:08.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 1 monclient: found mon.noname-b 2024-09-16T15:00:08.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:08.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:00:08.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 1 -- 172.21.15.26:0/2226033471 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2409ed60 con 0x7f3f240a1d10 2024-09-16T15:00:08.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient: _finish_auth 0 2024-09-16T15:00:08.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient: _check_auth_tickets 2024-09-16T15:00:08.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:08.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 1 -- 172.21.15.26:0/2226033471 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f14001550 con 0x7f3f240a1d10 2024-09-16T15:00:08.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 5 monclient: authenticate success, global_id 284687 2024-09-16T15:00:08.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:08.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:00:08.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient: dump: 2024-09-16T15:00:08.914 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.915 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.915 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.915 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.915 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.915 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.915 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.916 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.916 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.916 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:08.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 1 -- 172.21.15.26:0/2226033471 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f140017f0 con 0x7f3f240a1d10 2024-09-16T15:00:08.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:08.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 1 -- 172.21.15.26:0/2226033471 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f14001b80 con 0x7f3f240a1d10 2024-09-16T15:00:08.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:08.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f227fc640 10 monclient: dump: 2024-09-16T15:00:08.918 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.918 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.918 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.918 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.918 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.919 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.919 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.919 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.919 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.919 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 10 monclient: get_monmap_and_config success 2024-09-16T15:00:08.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 4 set_mon_vals no callback set 2024-09-16T15:00:08.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 10 monclient: shutdown 2024-09-16T15:00:08.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:08.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 1 -- 172.21.15.26:0/2226033471 >> v1:172.21.15.26:6790/0 conn(0x7f3f240a1d10 legacy=0x7f3f240a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 1 -- 172.21.15.26:0/2226033471 shutdown_connections 2024-09-16T15:00:08.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 1 -- 172.21.15.26:0/2226033471 >> 172.21.15.26:0/2226033471 conn(0x7f3f240a7f30 msgr2=0x7f3f240a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:08.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 1 -- 172.21.15.26:0/2226033471 shutdown_connections 2024-09-16T15:00:08.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 1 -- 172.21.15.26:0/2226033471 wait complete. 2024-09-16T15:00:08.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 10 monclient: build_initial_monmap 2024-09-16T15:00:08.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 10 monclient: monmap: 2024-09-16T15:00:08.922 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:08.922 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:00:08.922 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:00:08.922 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:00:08.922 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:08.922 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:00:08.923 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:08.923 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:00:08.923 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:00:08.923 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 1 Processor -- start 2024-09-16T15:00:08.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 1 -- start start 2024-09-16T15:00:08.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.906+0000 7f3f2a238640 10 monclient: init 2024-09-16T15:00:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:00:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 10 monclient(hunting): picked mon.noname-c con 0x7f3f240a0000 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 10 monclient(hunting): picked mon.noname-a con 0x7f3f240a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:08.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 10 monclient(hunting): picked mon.noname-b con 0x7f3f240a2800 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2413ae60 con 0x7f3f240a1d10 2024-09-16T15:00:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2413cee0 con 0x7f3f240a2800 2024-09-16T15:00:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2413a070 con 0x7f3f240a0000 2024-09-16T15:00:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 10 monclient(hunting): authenticate will time out at 2033.032560s 2024-09-16T15:00:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f237fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3f240a0000 0x7f3f24139250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:43302/0 (socket says 172.21.15.26:43302) 2024-09-16T15:00:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f22ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3f240a1d10 0x7f3f24139960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42352/0 (socket says 172.21.15.26:42352) 2024-09-16T15:00:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f237fe640 1 -- 172.21.15.26:0/3578046285 learned_addr learned my addr 172.21.15.26:0/3578046285 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480686574 0 0) 0x7f3f2413a070 con 0x7f3f240a0000 2024-09-16T15:00:08.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): my global_id is 284690 2024-09-16T15:00:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f10003530 con 0x7f3f240a0000 2024-09-16T15:00:08.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810680126 0 0) 0x7f3f2413ae60 con 0x7f3f240a1d10 2024-09-16T15:00:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): my global_id is 334158 2024-09-16T15:00:08.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f2413a070 con 0x7f3f240a1d10 2024-09-16T15:00:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3303957142 0 0) 0x7f3f10003530 con 0x7f3f240a0000 2024-09-16T15:00:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f2413ae60 con 0x7f3f240a0000 2024-09-16T15:00:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429926285 0 0) 0x7f3f2413cee0 con 0x7f3f240a2800 2024-09-16T15:00:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): my global_id is 304417 2024-09-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f10003530 con 0x7f3f240a2800 2024-09-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f14001bc0 con 0x7f3f240a0000 2024-09-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): dump: 2024-09-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.940 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.941 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.942 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:00:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3648301184 0 0) 0x7f3f2413a070 con 0x7f3f240a1d10 2024-09-16T15:00:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f2413cee0 con 0x7f3f240a1d10 2024-09-16T15:00:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f18001860 con 0x7f3f240a1d10 2024-09-16T15:00:08.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): dump: 2024-09-16T15:00:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.943 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.944 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.944 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.944 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.944 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.944 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.944 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.945 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2439726944 0 0) 0x7f3f2413ae60 con 0x7f3f240a0000 2024-09-16T15:00:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 >> v1:172.21.15.189:6789/0 conn(0x7f3f240a2800 legacy=0x7f3f241d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 >> v1:172.21.15.26:6789/0 conn(0x7f3f240a1d10 legacy=0x7f3f24139960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient: _finish_hunting 0 2024-09-16T15:00:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 monclient: found mon.c 2024-09-16T15:00:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:00:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2413ba30 con 0x7f3f240a0000 2024-09-16T15:00:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient: _finish_auth 0 2024-09-16T15:00:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient: _check_auth_tickets 2024-09-16T15:00:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 5 monclient: authenticate success, global_id 284690 2024-09-16T15:00:08.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 10 monclient: _renew_subs 2024-09-16T15:00:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:00:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 1 -- 172.21.15.26:0/3578046285 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2413d1c0 con 0x7f3f240a0000 2024-09-16T15:00:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 10 monclient: _renew_subs 2024-09-16T15:00:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:00:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 1 -- 172.21.15.26:0/3578046285 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2413cc00 con 0x7f3f240a0000 2024-09-16T15:00:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f14004aa0 con 0x7f3f240a0000 2024-09-16T15:00:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f14004e30 con 0x7f3f240a0000 2024-09-16T15:00:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f21ffb640 4 set_mon_vals no callback set 2024-09-16T15:00:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient: dump: 2024-09-16T15:00:08.948 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.948 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.948 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.948 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.948 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.948 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.948 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.948 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.948 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.948 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:08.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f2a238640 1 -- 172.21.15.26:0/3578046285 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3f04006590 con 0x7f3f04002830 2024-09-16T15:00:08.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3f14004f90 con 0x7f3f240a0000 2024-09-16T15:00:08.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f3f14043ec0 con 0x7f3f240a0000 2024-09-16T15:00:08.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f23fff640 10 monclient: get_auth_request con 0x7f3f100363d0 auth_method 0 2024-09-16T15:00:08.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779909456 0 0) 0x7f3f04006590 con 0x7f3f04002830 2024-09-16T15:00:08.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient(hunting): my global_id is 304420 2024-09-16T15:00:08.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f2413ae60 con 0x7f3f04002830 2024-09-16T15:00:08.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2494887402 0 0) 0x7f3f2413ae60 con 0x7f3f04002830 2024-09-16T15:00:08.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f04006590 con 0x7f3f04002830 2024-09-16T15:00:08.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f18003a30 con 0x7f3f04002830 2024-09-16T15:00:08.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:00:08.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 10 monclient: dump: 2024-09-16T15:00:08.950 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.950 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.950 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.950 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.950 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.950 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.950 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.950 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.950 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.950 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 932179015 0 0) 0x7f3f04006590 con 0x7f3f04002830 2024-09-16T15:00:08.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.910+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f0400a0e0 con 0x7f3f04002830 2024-09-16T15:00:08.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.914+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f3f180057d0 con 0x7f3f04002830 2024-09-16T15:00:08.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.914+0000 7f3f0bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:08.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.914+0000 7f3f0bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:00:08.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.914+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 >> v1:172.21.15.189:6789/0 conn(0x7f3f04002830 legacy=0x7f3f04004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.914+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 >> v1:172.21.15.189:6789/0 conn(0x7f3f04002830 legacy=0x7f3f04004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:00:08.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.918+0000 7f3f2a238640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:00:08.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.918+0000 7f3f2a238640 1 -- 172.21.15.26:0/3578046285 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3f04009d80 con 0x7f3f040090a0 2024-09-16T15:00:08.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.918+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4856462 0 0) 0x7f3f04009d80 con 0x7f3f040090a0 2024-09-16T15:00:08.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.918+0000 7f3f0bfff640 10 monclient(hunting): _negotiate 2024-09-16T15:00:08.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.918+0000 7f3f0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:08.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.918+0000 7f3f0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:08.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.918+0000 7f3f0bfff640 10 monclient(hunting): my global_id is 304423 2024-09-16T15:00:08.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.918+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f04006590 con 0x7f3f040090a0 2024-09-16T15:00:08.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1257128113 0 0) 0x7f3f04006590 con 0x7f3f040090a0 2024-09-16T15:00:08.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f04009d80 con 0x7f3f040090a0 2024-09-16T15:00:08.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3f18004020 con 0x7f3f040090a0 2024-09-16T15:00:08.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:08.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f0bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:00:08.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f0bfff640 10 monclient: dump: 2024-09-16T15:00:08.952 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:08.952 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:08.952 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.952 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:08.952 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:08.952 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:08.953 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:08.953 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:08.953 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:08.953 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:08.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2370017283 0 0) 0x7f3f04009d80 con 0x7f3f040090a0 2024-09-16T15:00:08.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3f0400c290 con 0x7f3f040090a0 2024-09-16T15:00:08.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 1193712623) 0x7f3f18004330 con 0x7f3f040090a0 2024-09-16T15:00:08.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f0bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:00:08.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f0bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:00:08.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 >> v1:172.21.15.189:6789/0 conn(0x7f3f040090a0 legacy=0x7f3f04009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f0bfff640 1 -- 172.21.15.26:0/3578046285 >> v1:172.21.15.189:6789/0 conn(0x7f3f040090a0 legacy=0x7f3f04009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:00:08.954 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:00:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:00:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:00:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:00:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 162, 2024-09-16T15:00:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:00:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:00:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:00:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:00:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T15:00:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:00:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:00:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:00:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:00:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:00:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:00:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:00:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:00:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:00:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:00:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:00:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:00:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:00:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:00:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:00:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:00:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:00:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:00:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:00:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:00:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:00:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:00:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:00:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:00:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:00:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:00:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:00:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:00:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:00:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:00:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:00:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:00:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:00:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:00:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:00:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:00:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:00:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:00:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:00:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:08.964 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:08.964 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.964 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.964 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:00:08.964 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:00:08.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.964 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:08.964 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:08.964 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:00:08.964 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.964 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:08.964 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:00:08.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:08.965 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:08.965 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:08.965 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T15:00:08.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:08.965 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:08.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:08.965 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:00:08.965 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:00:08.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f2a238640 1 -- 172.21.15.26:0/3578046285 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3f100363d0 legacy=0x7f3f10038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f2a238640 10 monclient: shutdown 2024-09-16T15:00:08.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f2a238640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:08.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f2a238640 1 -- 172.21.15.26:0/3578046285 >> v1:172.21.15.26:6790/0 conn(0x7f3f240a0000 legacy=0x7f3f24139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:08.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f23fff640 1 -- 172.21.15.26:0/3578046285 reap_dead start 2024-09-16T15:00:08.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f23fff640 1 -- 172.21.15.26:0/3578046285 reap_dead start 2024-09-16T15:00:08.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f2a238640 1 -- 172.21.15.26:0/3578046285 shutdown_connections 2024-09-16T15:00:08.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f2a238640 1 -- 172.21.15.26:0/3578046285 >> 172.21.15.26:0/3578046285 conn(0x7f3f240a7f30 msgr2=0x7f3f2409b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:08.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f2a238640 1 -- 172.21.15.26:0/3578046285 shutdown_connections 2024-09-16T15:00:08.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:08.922+0000 7f3f2a238640 1 -- 172.21.15.26:0/3578046285 wait complete. 2024-09-16T15:00:08.968 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:00:08.968 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T15:00:08.968 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-16T15:00:08.968 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T15:00:08.968 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T15:00:08.968 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T15:00:08.968 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:00:08.996 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T15:00:08.996 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T15:00:08.997 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T15:00:08.997 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T15:00:23.998 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T15:00:23.998 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T15:00:23.999 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T15:00:23.999 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T15:00:24.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff0804f6640 10 monclient: get_monmap_and_config 2024-09-16T15:00:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff0804f6640 10 monclient: build_initial_monmap 2024-09-16T15:00:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff0804f6640 10 monclient: monmap: 2024-09-16T15:00:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:00:24.134839+0000 2024-09-16T15:00:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:00:24.134839+0000 2024-09-16T15:00:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:00:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:00:24.134 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:24.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff0804f6640 1 Processor -- start 2024-09-16T15:00:24.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff0804f6640 1 -- start start 2024-09-16T15:00:24.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff0804f6640 10 monclient: init 2024-09-16T15:00:24.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff0804f6640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:24.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff0804f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:00:24.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff0804f6640 10 monclient(hunting): picked mon.noname-a con 0x7ff07809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:24.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff0804f6640 10 monclient(hunting): picked mon.noname-c con 0x7ff078132650 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:24.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff0804f6640 10 monclient(hunting): picked mon.noname-b con 0x7ff078138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:24.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff0804f6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07809d260 con 0x7ff07809cb80 2024-09-16T15:00:24.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff0804f6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff078132a80 con 0x7ff078132650 2024-09-16T15:00:24.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff0804f6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07809cf60 con 0x7ff078138ff0 2024-09-16T15:00:24.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff0804f6640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:24.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff0804f6640 10 monclient(hunting): authenticate will time out at 2048.254853s 2024-09-16T15:00:24.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff07e292640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff07809cb80 0x7ff0780717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33936/0 (socket says 172.21.15.26:33936) 2024-09-16T15:00:24.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff07da91640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7ff078132650 legacy=0x7ff0781378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:00:24.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.129+0000 7ff07e292640 1 -- 172.21.15.26:0/570838226 learned_addr learned my addr 172.21.15.26:0/570838226 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:24.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.333+0000 7ff07da91640 1 -- 172.21.15.26:0/570838226 >> v1:172.21.15.189:6789/0 conn(0x7ff078132650 legacy=0x7ff0781378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:00:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:24.737+0000 7ff07da91640 1 -- 172.21.15.26:0/570838226 >> v1:172.21.15.189:6789/0 conn(0x7ff078132650 legacy=0x7ff0781378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:00:25.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:25.541+0000 7ff07da91640 1 -- 172.21.15.26:0/570838226 >> v1:172.21.15.189:6789/0 conn(0x7ff078132650 legacy=0x7ff0781378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:00:27.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.133+0000 7ff06bfff640 10 monclient(hunting): tick 2024-09-16T15:00:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.133+0000 7ff06bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T15:00:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.133+0000 7ff06bfff640 1 monclient(hunting): continuing hunt 2024-09-16T15:00:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.133+0000 7ff06bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T15:00:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.133+0000 7ff06bfff640 1 -- 172.21.15.26:0/570838226 >> v1:172.21.15.26:6790/0 conn(0x7ff078138ff0 legacy=0x7ff07813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.133+0000 7ff06bfff640 1 -- 172.21.15.26:0/570838226 >> v1:172.21.15.189:6789/0 conn(0x7ff078132650 legacy=0x7ff0781378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:00:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.133+0000 7ff06bfff640 1 -- 172.21.15.26:0/570838226 >> v1:172.21.15.26:6789/0 conn(0x7ff07809cb80 legacy=0x7ff0780717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.133+0000 7ff06bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:00:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.133+0000 7ff06bfff640 10 monclient(hunting): picked mon.noname-a con 0x7ff05c0036d0 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.133+0000 7ff06bfff640 10 monclient(hunting): picked mon.noname-b con 0x7ff078138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.133+0000 7ff06bfff640 10 monclient(hunting): picked mon.noname-c con 0x7ff078132650 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.133+0000 7ff06bfff640 1 -- 172.21.15.26:0/570838226 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07809f2d0 con 0x7ff05c0036d0 2024-09-16T15:00:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.133+0000 7ff06bfff640 1 -- 172.21.15.26:0/570838226 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff078132a80 con 0x7ff078132650 2024-09-16T15:00:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.133+0000 7ff06bfff640 1 -- 172.21.15.26:0/570838226 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05c0103b0 con 0x7ff078138ff0 2024-09-16T15:00:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.133+0000 7ff06bfff640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.133+0000 7ff07ea93640 1 -- 172.21.15.26:0/570838226 >> v1:172.21.15.189:6789/0 conn(0x7ff078132650 legacy=0x7ff05c00cbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:00:27.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.337+0000 7ff07ea93640 1 -- 172.21.15.26:0/570838226 >> v1:172.21.15.189:6789/0 conn(0x7ff078132650 legacy=0x7ff05c00cbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:00:27.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:27.741+0000 7ff07ea93640 1 -- 172.21.15.26:0/570838226 >> v1:172.21.15.189:6789/0 conn(0x7ff078132650 legacy=0x7ff05c00cbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:00:28.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:28.545+0000 7ff07ea93640 1 -- 172.21.15.26:0/570838226 >> v1:172.21.15.189:6789/0 conn(0x7ff078132650 legacy=0x7ff05c00cbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:00:29.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 1 -- 172.21.15.26:0/570838226 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854707889 0 0) 0x7ff07809f2d0 con 0x7ff05c0036d0 2024-09-16T15:00:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 10 monclient(hunting): _negotiate 2024-09-16T15:00:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 10 monclient(hunting): my global_id is 334161 2024-09-16T15:00:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 1 -- 172.21.15.26:0/570838226 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff064003530 con 0x7ff05c0036d0 2024-09-16T15:00:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 1 -- 172.21.15.26:0/570838226 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 745083241 0 0) 0x7ff064003530 con 0x7ff05c0036d0 2024-09-16T15:00:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 1 -- 172.21.15.26:0/570838226 >> v1:172.21.15.26:6790/0 conn(0x7ff078138ff0 legacy=0x7ff05c0093d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 1 -- 172.21.15.26:0/570838226 >> v1:172.21.15.189:6789/0 conn(0x7ff078132650 legacy=0x7ff05c00cbd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:00:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 10 monclient: _finish_hunting 0 2024-09-16T15:00:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 1 monclient: found mon.noname-a 2024-09-16T15:00:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:00:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 1 -- 172.21.15.26:0/570838226 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c011620 con 0x7ff05c0036d0 2024-09-16T15:00:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 10 monclient: _finish_auth 0 2024-09-16T15:00:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 10 monclient: _check_auth_tickets 2024-09-16T15:00:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 1 -- 172.21.15.26:0/570838226 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff0740015a0 con 0x7ff05c0036d0 2024-09-16T15:00:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff0804f6640 5 monclient: authenticate success, global_id 334161 2024-09-16T15:00:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff0804f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:00:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 10 monclient: dump: 2024-09-16T15:00:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.016 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.016 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.016 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.016 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.016 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.016 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.016 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff0804f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:29.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 1 -- 172.21.15.26:0/570838226 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff074001840 con 0x7ff05c0036d0 2024-09-16T15:00:29.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:29.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 1 -- 172.21.15.26:0/570838226 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff074001c10 con 0x7ff05c0036d0 2024-09-16T15:00:29.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff0804f6640 10 monclient: get_monmap_and_config success 2024-09-16T15:00:29.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff0804f6640 4 set_mon_vals no callback set 2024-09-16T15:00:29.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:29.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.009+0000 7ff07d290640 10 monclient: dump: 2024-09-16T15:00:29.019 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.019 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.019 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.019 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.019 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.019 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.019 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.019 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.019 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.020 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 10 monclient: shutdown 2024-09-16T15:00:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 1 -- 172.21.15.26:0/570838226 >> v1:172.21.15.26:6789/0 conn(0x7ff05c0036d0 legacy=0x7ff05c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 1 -- 172.21.15.26:0/570838226 shutdown_connections 2024-09-16T15:00:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 1 -- 172.21.15.26:0/570838226 >> 172.21.15.26:0/570838226 conn(0x7ff07809fb50 msgr2=0x7ff07809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 1 -- 172.21.15.26:0/570838226 shutdown_connections 2024-09-16T15:00:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 1 -- 172.21.15.26:0/570838226 wait complete. 2024-09-16T15:00:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 10 monclient: build_initial_monmap 2024-09-16T15:00:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 10 monclient: monmap: 2024-09-16T15:00:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:00:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:00:29.020 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:00:29.021 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:29.021 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:00:29.021 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:29.021 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:00:29.021 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:00:29.021 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 1 Processor -- start 2024-09-16T15:00:29.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 1 -- start start 2024-09-16T15:00:29.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 10 monclient: init 2024-09-16T15:00:29.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:29.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:00:29.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 10 monclient(hunting): picked mon.noname-c con 0x7ff05c0036d0 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:29.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 10 monclient(hunting): picked mon.noname-a con 0x7ff078132650 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:29.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 10 monclient(hunting): picked mon.noname-b con 0x7ff078138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff078070fb0 con 0x7ff078132650 2024-09-16T15:00:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0780714e0 con 0x7ff078138ff0 2024-09-16T15:00:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07809a240 con 0x7ff05c0036d0 2024-09-16T15:00:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff0804f6640 10 monclient(hunting): authenticate will time out at 2053.139147s 2024-09-16T15:00:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff07e292640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff05c0036d0 0x7ff07812baa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46756/0 (socket says 172.21.15.26:46756) 2024-09-16T15:00:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff07e292640 1 -- 172.21.15.26:0/2457449855 learned_addr learned my addr 172.21.15.26:0/2457449855 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff07ea93640 1 -- 172.21.15.26:0/2457449855 >> v1:172.21.15.189:6789/0 conn(0x7ff078138ff0 legacy=0x7ff0781d1d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:00:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff06affd640 1 -- 172.21.15.26:0/2457449855 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814448636 0 0) 0x7ff078070fb0 con 0x7ff078132650 2024-09-16T15:00:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff06affd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff06affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff06affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff06affd640 10 monclient(hunting): my global_id is 334164 2024-09-16T15:00:29.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff06affd640 1 -- 172.21.15.26:0/2457449855 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff05c00fa50 con 0x7ff078132650 2024-09-16T15:00:29.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.013+0000 7ff06affd640 1 -- 172.21.15.26:0/2457449855 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2238000358 0 0) 0x7ff05c00fa50 con 0x7ff078132650 2024-09-16T15:00:29.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 1 -- 172.21.15.26:0/2457449855 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff078070fb0 con 0x7ff078132650 2024-09-16T15:00:29.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 1 -- 172.21.15.26:0/2457449855 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff06c0040e0 con 0x7ff078132650 2024-09-16T15:00:29.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:00:29.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 10 monclient(hunting): dump: 2024-09-16T15:00:29.025 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.025 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.025 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.025 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.025 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.025 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.025 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.025 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.025 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.025 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:00:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 1 -- 172.21.15.26:0/2457449855 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2269845432 0 0) 0x7ff078070fb0 con 0x7ff078132650 2024-09-16T15:00:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 1 -- 172.21.15.26:0/2457449855 >> v1:172.21.15.26:6790/0 conn(0x7ff05c0036d0 legacy=0x7ff07812baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 1 -- 172.21.15.26:0/2457449855 >> v1:172.21.15.189:6789/0 conn(0x7ff078138ff0 legacy=0x7ff0781d1d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:00:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 10 monclient: _finish_hunting 0 2024-09-16T15:00:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 1 monclient: found mon.a 2024-09-16T15:00:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 1 -- 172.21.15.26:0/2457449855 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff078138280 con 0x7ff078132650 2024-09-16T15:00:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 10 monclient: _finish_auth 0 2024-09-16T15:00:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 10 monclient: _check_auth_tickets 2024-09-16T15:00:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff0804f6640 5 monclient: authenticate success, global_id 334164 2024-09-16T15:00:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff0804f6640 10 monclient: _renew_subs 2024-09-16T15:00:29.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff0804f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff0804f6640 1 -- 172.21.15.26:0/2457449855 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff078136ae0 con 0x7ff078132650 2024-09-16T15:00:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff0804f6640 10 monclient: _renew_subs 2024-09-16T15:00:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff0804f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff0804f6640 1 -- 172.21.15.26:0/2457449855 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07812c1b0 con 0x7ff078132650 2024-09-16T15:00:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 1 -- 172.21.15.26:0/2457449855 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff06c004530 con 0x7ff078132650 2024-09-16T15:00:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 1 -- 172.21.15.26:0/2457449855 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff06c004900 con 0x7ff078132650 2024-09-16T15:00:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 10 monclient: dump: 2024-09-16T15:00:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.027 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.028 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.028 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.028 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.028 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.028 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.028 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 1 -- 172.21.15.26:0/2457449855 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff06c004b20 con 0x7ff078132650 2024-09-16T15:00:29.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff0804f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:29.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff07ca8f640 4 set_mon_vals no callback set 2024-09-16T15:00:29.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff06affd640 1 -- 172.21.15.26:0/2457449855 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7ff06c043bb0 con 0x7ff078132650 2024-09-16T15:00:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff0804f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff0804f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff0804f6640 1 -- 172.21.15.26:0/2457449855 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff078138080 con 0x7ff078132650 2024-09-16T15:00:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.017+0000 7ff07e292640 10 monclient: get_auth_request con 0x7ff05c036bd0 auth_method 0 2024-09-16T15:00:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.025+0000 7ff06affd640 1 -- 172.21.15.26:0/2457449855 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff06c0145c0 con 0x7ff078132650 2024-09-16T15:00:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.025+0000 7ff06affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:29.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.025+0000 7ff06affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:00:29.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.253+0000 7ff0804f6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T15:00:29.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.253+0000 7ff0804f6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T15:00:29.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.253+0000 7ff0804f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:00:29.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.253+0000 7ff0804f6640 1 -- 172.21.15.26:0/2457449855 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff07809c960 con 0x7ff078132650 2024-09-16T15:00:29.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.253+0000 7ff06affd640 1 -- 172.21.15.26:0/2457449855 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 155513118) 0x7ff06c0141b0 con 0x7ff078132650 2024-09-16T15:00:29.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.253+0000 7ff06affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T15:00:29.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.253+0000 7ff06affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:00:29.258 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":164,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T15:00:29.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.257+0000 7ff0804f6640 1 -- 172.21.15.26:0/2457449855 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff05c036bd0 legacy=0x7ff05c039090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.257+0000 7ff0804f6640 10 monclient: shutdown 2024-09-16T15:00:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.257+0000 7ff0804f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.257+0000 7ff0804f6640 1 -- 172.21.15.26:0/2457449855 >> v1:172.21.15.26:6789/0 conn(0x7ff078132650 legacy=0x7ff0781ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.257+0000 7ff0804f6640 1 -- 172.21.15.26:0/2457449855 shutdown_connections 2024-09-16T15:00:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.257+0000 7ff0804f6640 1 -- 172.21.15.26:0/2457449855 >> 172.21.15.26:0/2457449855 conn(0x7ff07809fb50 msgr2=0x7ff0780a1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.257+0000 7ff0804f6640 1 -- 172.21.15.26:0/2457449855 shutdown_connections 2024-09-16T15:00:29.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.257+0000 7ff0804f6640 1 -- 172.21.15.26:0/2457449855 wait complete. 2024-09-16T15:00:29.277 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T15:00:29.277 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T15:00:29.277 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T15:00:29.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 10 monclient: get_monmap_and_config 2024-09-16T15:00:29.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 10 monclient: build_initial_monmap 2024-09-16T15:00:29.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 10 monclient: monmap: 2024-09-16T15:00:29.396 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:29.396 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.396 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:00:29.397733+0000 2024-09-16T15:00:29.396 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:00:29.397733+0000 2024-09-16T15:00:29.396 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:29.396 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.396 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:29.396 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:00:29.396 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:00:29.397 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 1 Processor -- start 2024-09-16T15:00:29.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 1 -- start start 2024-09-16T15:00:29.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 10 monclient: init 2024-09-16T15:00:29.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:29.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:00:29.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 10 monclient(hunting): picked mon.noname-c con 0x7f13300a1c00 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:29.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 10 monclient(hunting): picked mon.noname-b con 0x7f1330099b60 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:29.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 10 monclient(hunting): picked mon.noname-a con 0x7f13300a26f0 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:29.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1330098ea0 con 0x7f13300a26f0 2024-09-16T15:00:29.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13300997b0 con 0x7f13300a1c00 2024-09-16T15:00:29.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1330099460 con 0x7f1330099b60 2024-09-16T15:00:29.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:29.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 10 monclient(hunting): authenticate will time out at 2053.517850s 2024-09-16T15:00:29.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1335759640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1330099b60 0x7f1330139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46762/0 (socket says 172.21.15.26:46762) 2024-09-16T15:00:29.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1335f5a640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f13300a1c00 legacy=0x7f13300a1fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:00:29.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f133675b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f13300a26f0 0x7f133013d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60856/0 (socket says 172.21.15.26:60856) 2024-09-16T15:00:29.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1335759640 1 -- 172.21.15.26:0/1428780367 learned_addr learned my addr 172.21.15.26:0/1428780367 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:29.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 1 -- 172.21.15.26:0/1428780367 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068986856 0 0) 0x7f1330098ea0 con 0x7f13300a26f0 2024-09-16T15:00:29.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient(hunting): _negotiate 2024-09-16T15:00:29.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:29.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:29.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient(hunting): my global_id is 334167 2024-09-16T15:00:29.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 1 -- 172.21.15.26:0/1428780367 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1318003530 con 0x7f13300a26f0 2024-09-16T15:00:29.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 1 -- 172.21.15.26:0/1428780367 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329905639 0 0) 0x7f1330099460 con 0x7f1330099b60 2024-09-16T15:00:29.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient(hunting): _negotiate 2024-09-16T15:00:29.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:29.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:29.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient(hunting): my global_id is 284693 2024-09-16T15:00:29.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 1 -- 172.21.15.26:0/1428780367 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1330098ea0 con 0x7f1330099b60 2024-09-16T15:00:29.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 1 -- 172.21.15.26:0/1428780367 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3272228555 0 0) 0x7f1318003530 con 0x7f13300a26f0 2024-09-16T15:00:29.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 1 -- 172.21.15.26:0/1428780367 >> v1:172.21.15.26:6790/0 conn(0x7f1330099b60 legacy=0x7f1330139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 1 -- 172.21.15.26:0/1428780367 >> v1:172.21.15.189:6789/0 conn(0x7f13300a1c00 legacy=0x7f13300a1fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:00:29.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient: _finish_hunting 0 2024-09-16T15:00:29.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 1 monclient: found mon.noname-a 2024-09-16T15:00:29.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:29.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:00:29.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 1 -- 172.21.15.26:0/1428780367 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13300a34b0 con 0x7f13300a26f0 2024-09-16T15:00:29.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient: _finish_auth 0 2024-09-16T15:00:29.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient: _check_auth_tickets 2024-09-16T15:00:29.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:29.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 5 monclient: authenticate success, global_id 334167 2024-09-16T15:00:29.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:29.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 1 -- 172.21.15.26:0/1428780367 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1320001550 con 0x7f13300a26f0 2024-09-16T15:00:29.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:00:29.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient: dump: 2024-09-16T15:00:29.410 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.411 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.411 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.411 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.411 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.411 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.411 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.411 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.411 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.411 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:29.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 1 -- 172.21.15.26:0/1428780367 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13200017b0 con 0x7f13300a26f0 2024-09-16T15:00:29.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:29.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 10 monclient: get_monmap_and_config success 2024-09-16T15:00:29.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 1 -- 172.21.15.26:0/1428780367 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1320001b80 con 0x7f13300a26f0 2024-09-16T15:00:29.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 4 set_mon_vals no callback set 2024-09-16T15:00:29.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:29.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f1334f58640 10 monclient: dump: 2024-09-16T15:00:29.412 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.412 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.412 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.412 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.412 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.412 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.412 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.413 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.413 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.413 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 10 monclient: shutdown 2024-09-16T15:00:29.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:29.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 1 -- 172.21.15.26:0/1428780367 >> v1:172.21.15.26:6789/0 conn(0x7f13300a26f0 legacy=0x7f133013d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 1 -- 172.21.15.26:0/1428780367 shutdown_connections 2024-09-16T15:00:29.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 1 -- 172.21.15.26:0/1428780367 >> 172.21.15.26:0/1428780367 conn(0x7f13300a3d00 msgr2=0x7f13300a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:29.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 1 -- 172.21.15.26:0/1428780367 shutdown_connections 2024-09-16T15:00:29.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 1 -- 172.21.15.26:0/1428780367 wait complete. 2024-09-16T15:00:29.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 10 monclient: build_initial_monmap 2024-09-16T15:00:29.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.393+0000 7f13381be640 10 monclient: monmap: 2024-09-16T15:00:29.414 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:29.414 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:00:29.414 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:00:29.414 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:00:29.414 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:29.415 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:00:29.415 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:29.415 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:00:29.415 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:00:29.415 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 1 Processor -- start 2024-09-16T15:00:29.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 1 -- start start 2024-09-16T15:00:29.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 10 monclient: init 2024-09-16T15:00:29.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:29.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:00:29.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 10 monclient(hunting): picked mon.noname-c con 0x7f1330099b60 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:29.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 10 monclient(hunting): picked mon.noname-a con 0x7f13300a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:29.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 10 monclient(hunting): picked mon.noname-b con 0x7f13300a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:29.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133011e5e0 con 0x7f13300a1c00 2024-09-16T15:00:29.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133009f010 con 0x7f13300a26f0 2024-09-16T15:00:29.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133009eae0 con 0x7f1330099b60 2024-09-16T15:00:29.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:29.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 10 monclient(hunting): authenticate will time out at 2053.520935s 2024-09-16T15:00:29.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f1335759640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f13300a1c00 0x7f133013d500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60858/0 (socket says 172.21.15.26:60858) 2024-09-16T15:00:29.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f1335759640 1 -- 172.21.15.26:0/2140868545 learned_addr learned my addr 172.21.15.26:0/2140868545 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:29.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f133675b640 1 -- 172.21.15.26:0/2140868545 >> v1:172.21.15.189:6789/0 conn(0x7f13300a26f0 legacy=0x7f133013dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:00:29.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172942292 0 0) 0x7f133011e5e0 con 0x7f13300a1c00 2024-09-16T15:00:29.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:00:29.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:29.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:29.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): my global_id is 334170 2024-09-16T15:00:29.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f131c003530 con 0x7f13300a1c00 2024-09-16T15:00:29.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005766065 0 0) 0x7f133009eae0 con 0x7f1330099b60 2024-09-16T15:00:29.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:00:29.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:29.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:29.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): my global_id is 284696 2024-09-16T15:00:29.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f133011e5e0 con 0x7f1330099b60 2024-09-16T15:00:29.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3821621397 0 0) 0x7f133011e5e0 con 0x7f1330099b60 2024-09-16T15:00:29.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f133009eae0 con 0x7f1330099b60 2024-09-16T15:00:29.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f13280044d0 con 0x7f1330099b60 2024-09-16T15:00:29.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:00:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): dump: 2024-09-16T15:00:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.418 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.418 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:00:29.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1039155136 0 0) 0x7f131c003530 con 0x7f13300a1c00 2024-09-16T15:00:29.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f133011e5e0 con 0x7f13300a1c00 2024-09-16T15:00:29.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1324001380 con 0x7f13300a1c00 2024-09-16T15:00:29.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:29.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): dump: 2024-09-16T15:00:29.419 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.419 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.419 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.419 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.419 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.419 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.419 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.420 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.420 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.420 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 329545858 0 0) 0x7f133009eae0 con 0x7f1330099b60 2024-09-16T15:00:29.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 >> v1:172.21.15.189:6789/0 conn(0x7f13300a26f0 legacy=0x7f133013dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:00:29.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 >> v1:172.21.15.26:6789/0 conn(0x7f13300a1c00 legacy=0x7f133013d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient: _finish_hunting 0 2024-09-16T15:00:29.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 monclient: found mon.c 2024-09-16T15:00:29.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:29.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:00:29.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133011f600 con 0x7f1330099b60 2024-09-16T15:00:29.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient: _finish_auth 0 2024-09-16T15:00:29.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient: _check_auth_tickets 2024-09-16T15:00:29.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:29.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 5 monclient: authenticate success, global_id 284696 2024-09-16T15:00:29.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 10 monclient: _renew_subs 2024-09-16T15:00:29.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:00:29.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 1 -- 172.21.15.26:0/2140868545 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133011f8e0 con 0x7f1330099b60 2024-09-16T15:00:29.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 10 monclient: _renew_subs 2024-09-16T15:00:29.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:00:29.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 1 -- 172.21.15.26:0/2140868545 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13301d2940 con 0x7f1330099b60 2024-09-16T15:00:29.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13280049a0 con 0x7f1330099b60 2024-09-16T15:00:29.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:29.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1328004d30 con 0x7f1330099b60 2024-09-16T15:00:29.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:29.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132f7fe640 4 set_mon_vals no callback set 2024-09-16T15:00:29.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient: dump: 2024-09-16T15:00:29.422 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.422 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.422 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.422 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.422 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.422 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.422 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.422 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.422 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.422 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:29.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f13381be640 1 -- 172.21.15.26:0/2140868545 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1310006590 con 0x7f1310002830 2024-09-16T15:00:29.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f1328004e90 con 0x7f1330099b60 2024-09-16T15:00:29.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f1328043fe0 con 0x7f1330099b60 2024-09-16T15:00:29.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838651433 0 0) 0x7f1310006590 con 0x7f1310002830 2024-09-16T15:00:29.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:00:29.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:29.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:29.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient(hunting): my global_id is 334173 2024-09-16T15:00:29.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f133009eae0 con 0x7f1310002830 2024-09-16T15:00:29.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1597573607 0 0) 0x7f133009eae0 con 0x7f1310002830 2024-09-16T15:00:29.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1310006590 con 0x7f1310002830 2024-09-16T15:00:29.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1324004c40 con 0x7f1310002830 2024-09-16T15:00:29.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:29.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient: dump: 2024-09-16T15:00:29.424 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.424 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.424 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.424 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.424 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.424 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.424 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.424 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.424 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.424 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f133675b640 10 monclient: get_auth_request con 0x7f131c036280 auth_method 0 2024-09-16T15:00:29.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3401354610 0 0) 0x7f1310006590 con 0x7f1310002830 2024-09-16T15:00:29.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f131000a0e0 con 0x7f1310002830 2024-09-16T15:00:29.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1324004f20 con 0x7f1310002830 2024-09-16T15:00:29.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:29.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:00:29.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 >> v1:172.21.15.26:6789/0 conn(0x7f1310002830 legacy=0x7f1310004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.397+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 >> v1:172.21.15.26:6789/0 conn(0x7f1310002830 legacy=0x7f1310004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:00:29.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f13381be640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:00:29.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f13381be640 1 -- 172.21.15.26:0/2140868545 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1310009da0 con 0x7f13100090c0 2024-09-16T15:00:29.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957750419 0 0) 0x7f1310009da0 con 0x7f13100090c0 2024-09-16T15:00:29.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:00:29.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:29.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:29.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 10 monclient(hunting): my global_id is 334176 2024-09-16T15:00:29.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1310006590 con 0x7f13100090c0 2024-09-16T15:00:29.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2961629652 0 0) 0x7f1310006590 con 0x7f13100090c0 2024-09-16T15:00:29.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1310009da0 con 0x7f13100090c0 2024-09-16T15:00:29.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f13240055e0 con 0x7f13100090c0 2024-09-16T15:00:29.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:29.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 10 monclient: dump: 2024-09-16T15:00:29.428 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.428 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.428 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.428 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.428 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.428 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.428 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.428 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.428 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.428 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3625173608 0 0) 0x7f1310009da0 con 0x7f13100090c0 2024-09-16T15:00:29.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f131000c290 con 0x7f13100090c0 2024-09-16T15:00:29.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3869 (unknown 0 0 4244214873) 0x7f13240069b0 con 0x7f13100090c0 2024-09-16T15:00:29.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:00:29.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:00:29.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 >> v1:172.21.15.26:6789/0 conn(0x7f13100090c0 legacy=0x7f1310009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f132e7fc640 1 -- 172.21.15.26:0/2140868545 >> v1:172.21.15.26:6789/0 conn(0x7f13100090c0 legacy=0x7f1310009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:00:29.429 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:00:29.429 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:00:29.429 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:00:29.429 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:00:29.430 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 164, 2024-09-16T15:00:29.430 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:00:29.430 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:00:29.430 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:00:29.430 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:29.430 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:00:29.430 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:00:29.430 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:00:29.430 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:00:29.430 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:29.430 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:29.430 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:29.431 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:29.431 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:29.431 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:29.431 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:29.431 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:29.431 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:29.431 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:29.431 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:29.431 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:00:29.431 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:00:29.431 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:29.432 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:29.432 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:29.432 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:29.432 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:29.432 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:29.432 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:29.432 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:29.432 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:29.432 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:29.432 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:29.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.432 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:00:29.433 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:00:29.433 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:00:29.433 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:00:29.433 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:00:29.433 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:00:29.433 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:00:29.433 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:00:29.433 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:00:29.433 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:00:29.433 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:00:29.433 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:00:29.434 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:00:29.435 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:00:29.435 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:00:29.435 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:00:29.435 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:00:29.435 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:29.435 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:29.435 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:29.436 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:29.436 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:29.436 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:29.436 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:29.436 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:29.436 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:29.436 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:29.436 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:29.436 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:00:29.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.436 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:00:29.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.437 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:00:29.437 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:00:29.437 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:29.437 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:29.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.437 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:29.437 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:00:29.437 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:29.437 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:29.437 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:29.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.437 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:00:29.437 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:00:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:00:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:00:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:29.438 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:00:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:00:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:00:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:00:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:00:29.439 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:00:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:00:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:00:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:29.440 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:29.441 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:29.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.441 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:00:29.441 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:00:29.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.441 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:29.441 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:29.441 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:00:29.441 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:29.441 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:29.441 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:00:29.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.442 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:29.442 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:29.442 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T15:00:29.442 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:29.442 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:29.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.442 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:00:29.442 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:00:29.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f13381be640 1 -- 172.21.15.26:0/2140868545 >> v1:172.21.15.189:6812/3196161582 conn(0x7f131c036280 legacy=0x7f131c038740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f13381be640 10 monclient: shutdown 2024-09-16T15:00:29.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f13381be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:29.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f13381be640 1 -- 172.21.15.26:0/2140868545 >> v1:172.21.15.26:6790/0 conn(0x7f1330099b60 legacy=0x7f133013cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f133675b640 1 -- 172.21.15.26:0/2140868545 reap_dead start 2024-09-16T15:00:29.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f133675b640 1 -- 172.21.15.26:0/2140868545 reap_dead start 2024-09-16T15:00:29.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f13381be640 1 -- 172.21.15.26:0/2140868545 shutdown_connections 2024-09-16T15:00:29.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f13381be640 1 -- 172.21.15.26:0/2140868545 >> 172.21.15.26:0/2140868545 conn(0x7f13300a3d00 msgr2=0x7f13300a0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:29.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f13381be640 1 -- 172.21.15.26:0/2140868545 shutdown_connections 2024-09-16T15:00:29.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.405+0000 7f13381be640 1 -- 172.21.15.26:0/2140868545 wait complete. 2024-09-16T15:00:29.445 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T15:00:29.445 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T15:00:29.445 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T15:00:29.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 10 monclient: get_monmap_and_config 2024-09-16T15:00:29.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 10 monclient: build_initial_monmap 2024-09-16T15:00:29.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 10 monclient: monmap: 2024-09-16T15:00:29.560 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:29.560 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.560 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:00:29.562062+0000 2024-09-16T15:00:29.560 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:00:29.562062+0000 2024-09-16T15:00:29.560 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:29.560 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.560 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:29.560 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:00:29.560 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:00:29.561 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 1 Processor -- start 2024-09-16T15:00:29.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 1 -- start start 2024-09-16T15:00:29.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 10 monclient: init 2024-09-16T15:00:29.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:29.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:00:29.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 10 monclient(hunting): picked mon.noname-b con 0x7fa9d809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:29.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 10 monclient(hunting): picked mon.noname-c con 0x7fa9d8132650 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:29.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 10 monclient(hunting): picked mon.noname-a con 0x7fa9d8138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:29.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d809d260 con 0x7fa9d8138ff0 2024-09-16T15:00:29.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d8132a80 con 0x7fa9d8132650 2024-09-16T15:00:29.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d809cf60 con 0x7fa9d809cb80 2024-09-16T15:00:29.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:29.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 10 monclient(hunting): authenticate will time out at 2053.682134s 2024-09-16T15:00:29.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d6d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa9d8138ff0 0x7fa9d813b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60892/0 (socket says 172.21.15.26:60892) 2024-09-16T15:00:29.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d6d9d640 1 -- 172.21.15.26:0/2154194291 learned_addr learned my addr 172.21.15.26:0/2154194291 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:29.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d5d9b640 1 -- 172.21.15.26:0/2154194291 >> v1:172.21.15.189:6789/0 conn(0x7fa9d8132650 legacy=0x7fa9d81378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:00:29.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 1 -- 172.21.15.26:0/2154194291 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395580644 0 0) 0x7fa9d809d260 con 0x7fa9d8138ff0 2024-09-16T15:00:29.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient(hunting): _negotiate 2024-09-16T15:00:29.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:29.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:29.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient(hunting): my global_id is 334179 2024-09-16T15:00:29.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 1 -- 172.21.15.26:0/2154194291 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c0003530 con 0x7fa9d8138ff0 2024-09-16T15:00:29.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 1 -- 172.21.15.26:0/2154194291 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490976872 0 0) 0x7fa9d809cf60 con 0x7fa9d809cb80 2024-09-16T15:00:29.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient(hunting): _negotiate 2024-09-16T15:00:29.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:29.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:29.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient(hunting): my global_id is 284699 2024-09-16T15:00:29.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 1 -- 172.21.15.26:0/2154194291 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d809d260 con 0x7fa9d809cb80 2024-09-16T15:00:29.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 1 -- 172.21.15.26:0/2154194291 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1988094117 0 0) 0x7fa9c0003530 con 0x7fa9d8138ff0 2024-09-16T15:00:29.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 1 -- 172.21.15.26:0/2154194291 >> v1:172.21.15.26:6790/0 conn(0x7fa9d809cb80 legacy=0x7fa9d80717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 1 -- 172.21.15.26:0/2154194291 >> v1:172.21.15.189:6789/0 conn(0x7fa9d8132650 legacy=0x7fa9d81378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:00:29.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient: _finish_hunting 0 2024-09-16T15:00:29.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 1 monclient: found mon.noname-a 2024-09-16T15:00:29.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:29.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:00:29.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 1 -- 172.21.15.26:0/2154194291 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d809f2d0 con 0x7fa9d8138ff0 2024-09-16T15:00:29.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient: _finish_auth 0 2024-09-16T15:00:29.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient: _check_auth_tickets 2024-09-16T15:00:29.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:29.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 1 -- 172.21.15.26:0/2154194291 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa9d00019a0 con 0x7fa9d8138ff0 2024-09-16T15:00:29.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 5 monclient: authenticate success, global_id 334179 2024-09-16T15:00:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:00:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient: dump: 2024-09-16T15:00:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.571 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:00:29.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 1 -- 172.21.15.26:0/2154194291 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9d0001bc0 con 0x7fa9d8138ff0 2024-09-16T15:00:29.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:29.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 10 monclient: get_monmap_and_config success 2024-09-16T15:00:29.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 4 set_mon_vals no callback set 2024-09-16T15:00:29.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 1 -- 172.21.15.26:0/2154194291 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa9d0001ed0 con 0x7fa9d8138ff0 2024-09-16T15:00:29.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:29.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9d559a640 10 monclient: dump: 2024-09-16T15:00:29.572 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.572 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.572 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.572 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.573 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.573 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.573 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.573 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.573 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.573 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 10 monclient: shutdown 2024-09-16T15:00:29.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:29.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 1 -- 172.21.15.26:0/2154194291 >> v1:172.21.15.26:6789/0 conn(0x7fa9d8138ff0 legacy=0x7fa9d813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 1 -- 172.21.15.26:0/2154194291 shutdown_connections 2024-09-16T15:00:29.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 1 -- 172.21.15.26:0/2154194291 >> 172.21.15.26:0/2154194291 conn(0x7fa9d809fb50 msgr2=0x7fa9d809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:29.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 1 -- 172.21.15.26:0/2154194291 shutdown_connections 2024-09-16T15:00:29.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.557+0000 7fa9dcb86640 1 -- 172.21.15.26:0/2154194291 wait complete. 2024-09-16T15:00:29.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 10 monclient: build_initial_monmap 2024-09-16T15:00:29.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 10 monclient: monmap: 2024-09-16T15:00:29.574 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:00:29.574 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:00:29.574 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:00:29.574 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:00:29.574 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:00:29.574 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:00:29.574 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:00:29.574 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:00:29.574 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:00:29.574 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 1 Processor -- start 2024-09-16T15:00:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 1 -- start start 2024-09-16T15:00:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 10 monclient: init 2024-09-16T15:00:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 10 monclient: _reopen_session rank -1 2024-09-16T15:00:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:00:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 10 monclient(hunting): picked mon.noname-b con 0x7fa9d809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:00:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 10 monclient(hunting): picked mon.noname-a con 0x7fa9d8132650 addr v1:172.21.15.26:6789/0 2024-09-16T15:00:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 10 monclient(hunting): picked mon.noname-c con 0x7fa9d8138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:00:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d8070eb0 con 0x7fa9d8132650 2024-09-16T15:00:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d809a240 con 0x7fa9d809cb80 2024-09-16T15:00:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d8133660 con 0x7fa9d8138ff0 2024-09-16T15:00:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 10 monclient(hunting): _renew_subs 2024-09-16T15:00:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 10 monclient(hunting): authenticate will time out at 2053.685228s 2024-09-16T15:00:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9d6d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa9d8138ff0 0x7fa9d806e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46802/0 (socket says 172.21.15.26:46802) 2024-09-16T15:00:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9d6d9d640 1 -- 172.21.15.26:0/69833840 learned_addr learned my addr 172.21.15.26:0/69833840 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:00:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9d5d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa9d8132650 0x7fa9d806deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60906/0 (socket says 172.21.15.26:60906) 2024-09-16T15:00:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9d659c640 1 -- 172.21.15.26:0/69833840 >> v1:172.21.15.189:6789/0 conn(0x7fa9d809cb80 legacy=0x7fa9d806d7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:00:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3107188111 0 0) 0x7fa9d8133660 con 0x7fa9d8138ff0 2024-09-16T15:00:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): my global_id is 284702 2024-09-16T15:00:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b8003530 con 0x7fa9d8138ff0 2024-09-16T15:00:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689316595 0 0) 0x7fa9d8070eb0 con 0x7fa9d8132650 2024-09-16T15:00:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): my global_id is 334182 2024-09-16T15:00:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d8133660 con 0x7fa9d8132650 2024-09-16T15:00:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2921510470 0 0) 0x7fa9b8003530 con 0x7fa9d8138ff0 2024-09-16T15:00:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9d8070eb0 con 0x7fa9d8138ff0 2024-09-16T15:00:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa9d0001960 con 0x7fa9d8138ff0 2024-09-16T15:00:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:00:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): dump: 2024-09-16T15:00:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.578 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:00:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3946169330 0 0) 0x7fa9d8133660 con 0x7fa9d8132650 2024-09-16T15:00:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9b8003530 con 0x7fa9d8132650 2024-09-16T15:00:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa9cc004080 con 0x7fa9d8132650 2024-09-16T15:00:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:00:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): dump: 2024-09-16T15:00:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.579 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4246960386 0 0) 0x7fa9d8070eb0 con 0x7fa9d8138ff0 2024-09-16T15:00:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 >> v1:172.21.15.189:6789/0 conn(0x7fa9d809cb80 legacy=0x7fa9d806d7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:00:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 >> v1:172.21.15.26:6789/0 conn(0x7fa9d8132650 legacy=0x7fa9d806deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient: _finish_hunting 0 2024-09-16T15:00:29.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 monclient: found mon.c 2024-09-16T15:00:29.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:00:29.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:00:29.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d80701e0 con 0x7fa9d8138ff0 2024-09-16T15:00:29.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient: _finish_auth 0 2024-09-16T15:00:29.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient: _check_auth_tickets 2024-09-16T15:00:29.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:00:29.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 5 monclient: authenticate success, global_id 284702 2024-09-16T15:00:29.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 10 monclient: _renew_subs 2024-09-16T15:00:29.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:00:29.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 1 -- 172.21.15.26:0/69833840 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d80a2940 con 0x7fa9d8138ff0 2024-09-16T15:00:29.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 10 monclient: _renew_subs 2024-09-16T15:00:29.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:00:29.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 1 -- 172.21.15.26:0/69833840 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d8071470 con 0x7fa9d8138ff0 2024-09-16T15:00:29.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9d0004e20 con 0x7fa9d8138ff0 2024-09-16T15:00:29.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:00:29.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa9d0005130 con 0x7fa9d8138ff0 2024-09-16T15:00:29.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:29.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9d4d99640 4 set_mon_vals no callback set 2024-09-16T15:00:29.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient: dump: 2024-09-16T15:00:29.581 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.581 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.581 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.581 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.582 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.582 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.582 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.582 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.582 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.582 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:29.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9dcb86640 1 -- 172.21.15.26:0/69833840 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa9bc006590 con 0x7fa9bc002830 2024-09-16T15:00:29.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa9d0005310 con 0x7fa9d8138ff0 2024-09-16T15:00:29.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fa9d0044360 con 0x7fa9d8138ff0 2024-09-16T15:00:29.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430243892 0 0) 0x7fa9bc006590 con 0x7fa9bc002830 2024-09-16T15:00:29.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:29.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient(hunting): my global_id is 284705 2024-09-16T15:00:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d8070eb0 con 0x7fa9bc002830 2024-09-16T15:00:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2644052651 0 0) 0x7fa9d8070eb0 con 0x7fa9bc002830 2024-09-16T15:00:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9bc006590 con 0x7fa9bc002830 2024-09-16T15:00:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa9c80016b0 con 0x7fa9bc002830 2024-09-16T15:00:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 10 monclient: dump: 2024-09-16T15:00:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.584 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.584 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.584 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.584 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.584 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9d5d9b640 10 monclient: get_auth_request con 0x7fa9b8036200 auth_method 0 2024-09-16T15:00:29.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2601733522 0 0) 0x7fa9bc006590 con 0x7fa9bc002830 2024-09-16T15:00:29.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.561+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9bc00a0e0 con 0x7fa9bc002830 2024-09-16T15:00:29.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.565+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa9c8001ac0 con 0x7fa9bc002830 2024-09-16T15:00:29.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.565+0000 7fa9c6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:00:29.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.565+0000 7fa9c6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:00:29.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.565+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 >> v1:172.21.15.26:6790/0 conn(0x7fa9bc002830 legacy=0x7fa9bc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.565+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 >> v1:172.21.15.26:6790/0 conn(0x7fa9bc002830 legacy=0x7fa9bc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:00:29.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9dcb86640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:00:29.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9dcb86640 1 -- 172.21.15.26:0/69833840 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa9bc009d80 con 0x7fa9bc0090a0 2024-09-16T15:00:29.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029945553 0 0) 0x7fa9bc009d80 con 0x7fa9bc0090a0 2024-09-16T15:00:29.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:00:29.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:00:29.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:00:29.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 10 monclient(hunting): my global_id is 284708 2024-09-16T15:00:29.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9bc006590 con 0x7fa9bc0090a0 2024-09-16T15:00:29.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1747754881 0 0) 0x7fa9bc006590 con 0x7fa9bc0090a0 2024-09-16T15:00:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9bc009d80 con 0x7fa9bc0090a0 2024-09-16T15:00:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa9c80037c0 con 0x7fa9bc0090a0 2024-09-16T15:00:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:00:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:00:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 10 monclient: dump: 2024-09-16T15:00:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:00:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:00:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:00:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:00:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:00:29.587 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:00:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:00:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:00:29.588 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:00:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 533601127 0 0) 0x7fa9bc009d80 con 0x7fa9bc0090a0 2024-09-16T15:00:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa9bc00c290 con 0x7fa9bc0090a0 2024-09-16T15:00:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4203 (unknown 0 0 4150891725) 0x7fa9c80057d0 con 0x7fa9bc0090a0 2024-09-16T15:00:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:00:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:00:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 >> v1:172.21.15.26:6790/0 conn(0x7fa9bc0090a0 legacy=0x7fa9bc009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9c6ffd640 1 -- 172.21.15.26:0/69833840 >> v1:172.21.15.26:6790/0 conn(0x7fa9bc0090a0 legacy=0x7fa9bc009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:00:29.588 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:00:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:00:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:00:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T15:00:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 164, 2024-09-16T15:00:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:00:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:00:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:00:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:00:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:00:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:00:29.589 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:00:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:00:29.590 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:00:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.591 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:00:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:00:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:00:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:00:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:00:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:00:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:00:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:00:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:00:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:00:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:00:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:00:29.592 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:00:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:00:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:00:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:00:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:00:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:00:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:00:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:00:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:00:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:00:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:00:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:00:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:00:29.593 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:00:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:00:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:00:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:00:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:00:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:00:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:29.594 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:00:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:00:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:00:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.595 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:00:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:00:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:00:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:00:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:00:29.596 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:00:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:00:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:00:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:00:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:00:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:00:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:00:29.597 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:00:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:00:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:00:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:00:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:00:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:00:29.598 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:00:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:00:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T15:00:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T15:00:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:29.599 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:00:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T15:00:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:00:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T15:00:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:00:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:00:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:00:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:00:29.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:00:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:00:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:00:29.601 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:00:29.601 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:00:29.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9dcb86640 1 -- 172.21.15.26:0/69833840 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa9b8036200 legacy=0x7fa9b80386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9dcb86640 10 monclient: shutdown 2024-09-16T15:00:29.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9dcb86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:00:29.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9dcb86640 1 -- 172.21.15.26:0/69833840 >> v1:172.21.15.26:6790/0 conn(0x7fa9d8138ff0 legacy=0x7fa9d806e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:00:29.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9d6d9d640 1 -- 172.21.15.26:0/69833840 reap_dead start 2024-09-16T15:00:29.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9d6d9d640 1 -- 172.21.15.26:0/69833840 reap_dead start 2024-09-16T15:00:29.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.569+0000 7fa9dcb86640 1 -- 172.21.15.26:0/69833840 shutdown_connections 2024-09-16T15:00:29.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.573+0000 7fa9dcb86640 1 -- 172.21.15.26:0/69833840 >> 172.21.15.26:0/69833840 conn(0x7fa9d809fb50 msgr2=0x7fa9d80a12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:00:29.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.573+0000 7fa9dcb86640 1 -- 172.21.15.26:0/69833840 shutdown_connections 2024-09-16T15:00:29.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:00:29.573+0000 7fa9dcb86640 1 -- 172.21.15.26:0/69833840 wait complete. 2024-09-16T15:00:29.603 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T15:00:29.603 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T15:00:49.604 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T15:00:49.604 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T15:00:49.604 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T15:00:49.605 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T15:00:49.608 INFO:tasks.ceph.mon.b:Started 2024-09-16T15:00:49.608 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T15:00:49.608 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T15:00:49.608 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T15:01:04.609 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T15:01:04.610 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T15:01:04.610 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T15:01:04.611 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T15:01:04.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd9962f5640 10 monclient: get_monmap_and_config 2024-09-16T15:01:04.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd9962f5640 10 monclient: build_initial_monmap 2024-09-16T15:01:04.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd9962f5640 10 monclient: monmap: 2024-09-16T15:01:04.745 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:04.745 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:04.745 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:04.747467+0000 2024-09-16T15:01:04.746 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:04.747467+0000 2024-09-16T15:01:04.746 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:04.746 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:04.746 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:04.746 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:04.746 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:04.746 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:04.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd9962f5640 1 Processor -- start 2024-09-16T15:01:04.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd9962f5640 1 -- start start 2024-09-16T15:01:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd9962f5640 10 monclient: init 2024-09-16T15:01:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd9962f5640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd9962f5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:01:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd9962f5640 10 monclient(hunting): picked mon.noname-b con 0x7fd99009cb80 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd9962f5640 10 monclient(hunting): picked mon.noname-c con 0x7fd990132650 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd9962f5640 10 monclient(hunting): picked mon.noname-a con 0x7fd990138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd9962f5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd99009d260 con 0x7fd990138ff0 2024-09-16T15:01:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd9962f5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd990132a80 con 0x7fd990132650 2024-09-16T15:01:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd9962f5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd99009cf60 con 0x7fd99009cb80 2024-09-16T15:01:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd9962f5640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd9962f5640 10 monclient(hunting): authenticate will time out at 2088.867651s 2024-09-16T15:01:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd98ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd99009cb80 0x7fd9900717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:45538/0 (socket says 172.21.15.26:45538) 2024-09-16T15:01:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd98ffff640 1 -- 172.21.15.26:0/2491572798 learned_addr learned my addr 172.21.15.26:0/2491572798 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.745+0000 7fd994892640 1 --1- 172.21.15.26:0/2491572798 >> v1:172.21.15.26:6789/0 conn(0x7fd990138ff0 0x7fd99013b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36720/0 (socket says 172.21.15.26:36720) 2024-09-16T15:01:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 1 -- 172.21.15.26:0/2491572798 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4792200 0 0) 0x7fd99009d260 con 0x7fd990138ff0 2024-09-16T15:01:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient(hunting): my global_id is 334185 2024-09-16T15:01:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 1 -- 172.21.15.26:0/2491572798 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd978003530 con 0x7fd990138ff0 2024-09-16T15:01:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 1 -- 172.21.15.26:0/2491572798 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281154145 0 0) 0x7fd99009cf60 con 0x7fd99009cb80 2024-09-16T15:01:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient(hunting): my global_id is 284711 2024-09-16T15:01:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 1 -- 172.21.15.26:0/2491572798 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd99009d260 con 0x7fd99009cb80 2024-09-16T15:01:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 1 -- 172.21.15.26:0/2491572798 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479481177 0 0) 0x7fd990132a80 con 0x7fd990132650 2024-09-16T15:01:04.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:04.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:04.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:04.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient(hunting): my global_id is 344098 2024-09-16T15:01:04.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 1 -- 172.21.15.26:0/2491572798 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd99009cf60 con 0x7fd990132650 2024-09-16T15:01:04.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 1 -- 172.21.15.26:0/2491572798 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1261620695 0 0) 0x7fd978003530 con 0x7fd990138ff0 2024-09-16T15:01:04.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 1 -- 172.21.15.26:0/2491572798 >> v1:172.21.15.26:6790/0 conn(0x7fd99009cb80 legacy=0x7fd9900717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:04.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 1 -- 172.21.15.26:0/2491572798 >> v1:172.21.15.189:6789/0 conn(0x7fd990132650 legacy=0x7fd9901378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:04.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient: _finish_hunting 0 2024-09-16T15:01:04.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 1 monclient: found mon.noname-a 2024-09-16T15:01:04.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:04.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:01:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 1 -- 172.21.15.26:0/2491572798 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99009f2d0 con 0x7fd990138ff0 2024-09-16T15:01:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient: _finish_auth 0 2024-09-16T15:01:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient: _check_auth_tickets 2024-09-16T15:01:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 1 -- 172.21.15.26:0/2491572798 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd9880019a0 con 0x7fd990138ff0 2024-09-16T15:01:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient: dump: 2024-09-16T15:01:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:04.758 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 5 monclient: authenticate success, global_id 334185 2024-09-16T15:01:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 1 -- 172.21.15.26:0/2491572798 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd988001bc0 con 0x7fd990138ff0 2024-09-16T15:01:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 1 -- 172.21.15.26:0/2491572798 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd988001f10 con 0x7fd990138ff0 2024-09-16T15:01:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 4 set_mon_vals no callback set 2024-09-16T15:01:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:04.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:04.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98effd640 10 monclient: dump: 2024-09-16T15:01:04.759 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:04.759 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:04.759 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:04.759 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:04.759 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:04.759 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:04.759 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:04.759 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:04.759 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:04.759 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:04.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 10 monclient: shutdown 2024-09-16T15:01:04.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:04.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 1 -- 172.21.15.26:0/2491572798 >> v1:172.21.15.26:6789/0 conn(0x7fd990138ff0 legacy=0x7fd99013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:04.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 1 -- 172.21.15.26:0/2491572798 shutdown_connections 2024-09-16T15:01:04.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 1 -- 172.21.15.26:0/2491572798 >> 172.21.15.26:0/2491572798 conn(0x7fd99009fb50 msgr2=0x7fd99009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:04.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 1 -- 172.21.15.26:0/2491572798 shutdown_connections 2024-09-16T15:01:04.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 1 -- 172.21.15.26:0/2491572798 wait complete. 2024-09-16T15:01:04.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 10 monclient: build_initial_monmap 2024-09-16T15:01:04.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 10 monclient: monmap: 2024-09-16T15:01:04.760 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:04.760 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:04.760 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:04.760 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:04.760 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:04.760 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:04.761 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:04.761 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:04.761 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:04.761 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:04.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 1 Processor -- start 2024-09-16T15:01:04.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 1 -- start start 2024-09-16T15:01:04.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 10 monclient: init 2024-09-16T15:01:04.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:04.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:01:04.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 10 monclient(hunting): picked mon.noname-b con 0x7fd99009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:04.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 10 monclient(hunting): picked mon.noname-c con 0x7fd990132650 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:04.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 10 monclient(hunting): picked mon.noname-a con 0x7fd990138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:04.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd990138860 con 0x7fd990138ff0 2024-09-16T15:01:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd990133780 con 0x7fd99009cb80 2024-09-16T15:01:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd99013c720 con 0x7fd990132650 2024-09-16T15:01:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd9962f5640 10 monclient(hunting): authenticate will time out at 2088.871075s 2024-09-16T15:01:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd994892640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd990138ff0 0x7fd99012e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36722/0 (socket says 172.21.15.26:36722) 2024-09-16T15:01:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd994892640 1 -- 172.21.15.26:0/2634309454 learned_addr learned my addr 172.21.15.26:0/2634309454 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98f7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd990132650 0x7fd99012e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:45548/0 (socket says 172.21.15.26:45548) 2024-09-16T15:01:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415577423 0 0) 0x7fd990138860 con 0x7fd990138ff0 2024-09-16T15:01:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): my global_id is 334188 2024-09-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd97c003530 con 0x7fd990138ff0 2024-09-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811726443 0 0) 0x7fd99013c720 con 0x7fd990132650 2024-09-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): my global_id is 284714 2024-09-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd990138860 con 0x7fd990132650 2024-09-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4090598958 0 0) 0x7fd97c003530 con 0x7fd990138ff0 2024-09-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd99013c720 con 0x7fd990138ff0 2024-09-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd9880018a0 con 0x7fd990138ff0 2024-09-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): dump: 2024-09-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425241548 0 0) 0x7fd990133780 con 0x7fd99009cb80 2024-09-16T15:01:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): my global_id is 344101 2024-09-16T15:01:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd97c003530 con 0x7fd99009cb80 2024-09-16T15:01:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 686203404 0 0) 0x7fd990138860 con 0x7fd990132650 2024-09-16T15:01:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd990133780 con 0x7fd990132650 2024-09-16T15:01:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd980001860 con 0x7fd990132650 2024-09-16T15:01:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.749+0000 7fd98cff9640 10 monclient(hunting): dump: 2024-09-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3897817225 0 0) 0x7fd99013c720 con 0x7fd990138ff0 2024-09-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 >> v1:172.21.15.26:6790/0 conn(0x7fd990132650 legacy=0x7fd99012e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 >> v1:172.21.15.189:6789/0 conn(0x7fd99009cb80 legacy=0x7fd99012da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 10 monclient: _finish_hunting 0 2024-09-16T15:01:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 1 monclient: found mon.a 2024-09-16T15:01:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99009a240 con 0x7fd990138ff0 2024-09-16T15:01:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 10 monclient: _finish_auth 0 2024-09-16T15:01:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 10 monclient: _check_auth_tickets 2024-09-16T15:01:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd9962f5640 5 monclient: authenticate success, global_id 334188 2024-09-16T15:01:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd9962f5640 10 monclient: _renew_subs 2024-09-16T15:01:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd9962f5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd9962f5640 1 -- 172.21.15.26:0/2634309454 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd990134ca0 con 0x7fd990138ff0 2024-09-16T15:01:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd9962f5640 10 monclient: _renew_subs 2024-09-16T15:01:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd9962f5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd9962f5640 1 -- 172.21.15.26:0/2634309454 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9901d2b90 con 0x7fd990138ff0 2024-09-16T15:01:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd988004e20 con 0x7fd990138ff0 2024-09-16T15:01:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fd988005130 con 0x7fd990138ff0 2024-09-16T15:01:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 10 monclient: dump: 2024-09-16T15:01:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:04.769 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98e7fc640 4 set_mon_vals no callback set 2024-09-16T15:01:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fd98800fba0 con 0x7fd990138ff0 2024-09-16T15:01:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd9962f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd9962f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd9962f5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd9962f5640 1 -- 172.21.15.26:0/2634309454 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd970005050 con 0x7fd990138ff0 2024-09-16T15:01:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fd988044150 con 0x7fd990138ff0 2024-09-16T15:01:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.753+0000 7fd98ffff640 10 monclient: get_auth_request con 0x7fd97c0363b0 auth_method 0 2024-09-16T15:01:04.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.757+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd988014910 con 0x7fd990138ff0 2024-09-16T15:01:04.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.757+0000 7fd98cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:04.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.757+0000 7fd98cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:04.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.981+0000 7fd9962f5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T15:01:04.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.981+0000 7fd9962f5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:04.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.981+0000 7fd9962f5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:04.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.981+0000 7fd9962f5640 1 -- 172.21.15.26:0/2634309454 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd970005880 con 0x7fd990138ff0 2024-09-16T15:01:04.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.981+0000 7fd98cff9640 1 -- 172.21.15.26:0/2634309454 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 3568409415) 0x7fd988012fb0 con 0x7fd990138ff0 2024-09-16T15:01:04.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.981+0000 7fd98cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:04.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.981+0000 7fd98cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:04.983 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":168,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T15:01:04.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.985+0000 7fd9962f5640 1 -- 172.21.15.26:0/2634309454 >> v1:172.21.15.189:6812/3196161582 conn(0x7fd97c0363b0 legacy=0x7fd97c038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:04.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.985+0000 7fd9962f5640 10 monclient: shutdown 2024-09-16T15:01:04.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.985+0000 7fd9962f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:04.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.985+0000 7fd9962f5640 1 -- 172.21.15.26:0/2634309454 >> v1:172.21.15.26:6789/0 conn(0x7fd990138ff0 legacy=0x7fd99012e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:04.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.985+0000 7fd9962f5640 1 -- 172.21.15.26:0/2634309454 shutdown_connections 2024-09-16T15:01:04.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.985+0000 7fd9962f5640 1 -- 172.21.15.26:0/2634309454 >> 172.21.15.26:0/2634309454 conn(0x7fd99009fb50 msgr2=0x7fd9900a13a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:04.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.985+0000 7fd9962f5640 1 -- 172.21.15.26:0/2634309454 shutdown_connections 2024-09-16T15:01:04.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:04.985+0000 7fd9962f5640 1 -- 172.21.15.26:0/2634309454 wait complete. 2024-09-16T15:01:04.999 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T15:01:05.000 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T15:01:05.000 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T15:01:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 10 monclient: get_monmap_and_config 2024-09-16T15:01:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 10 monclient: build_initial_monmap 2024-09-16T15:01:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 10 monclient: monmap: 2024-09-16T15:01:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.112 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:05.114554+0000 2024-09-16T15:01:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:05.114554+0000 2024-09-16T15:01:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:05.113 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:05.113 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 1 Processor -- start 2024-09-16T15:01:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 1 -- start start 2024-09-16T15:01:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 10 monclient: init 2024-09-16T15:01:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:01:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 10 monclient(hunting): picked mon.noname-b con 0x7f65ac09cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 10 monclient(hunting): picked mon.noname-a con 0x7f65ac09e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 10 monclient(hunting): picked mon.noname-c con 0x7f65ac09d1a0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac09d5b0 con 0x7f65ac09e7a0 2024-09-16T15:01:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac131a90 con 0x7f65ac09d1a0 2024-09-16T15:01:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac09aff0 con 0x7f65ac09cdc0 2024-09-16T15:01:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 10 monclient(hunting): authenticate will time out at 2089.234598s 2024-09-16T15:01:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65abfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f65ac09cdc0 0x7f65ac130c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:45564/0 (socket says 172.21.15.26:45564) 2024-09-16T15:01:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65ab7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f65ac09e7a0 0x7f65ac131380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36736/0 (socket says 172.21.15.26:36736) 2024-09-16T15:01:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65abfff640 1 -- 172.21.15.26:0/1094774176 learned_addr learned my addr 172.21.15.26:0/1094774176 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 1 -- 172.21.15.26:0/1094774176 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631735148 0 0) 0x7f65ac09d5b0 con 0x7f65ac09e7a0 2024-09-16T15:01:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient(hunting): my global_id is 334191 2024-09-16T15:01:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 1 -- 172.21.15.26:0/1094774176 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6598003530 con 0x7f65ac09e7a0 2024-09-16T15:01:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 1 -- 172.21.15.26:0/1094774176 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592755520 0 0) 0x7f65ac09aff0 con 0x7f65ac09cdc0 2024-09-16T15:01:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient(hunting): my global_id is 284717 2024-09-16T15:01:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 1 -- 172.21.15.26:0/1094774176 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65ac09d5b0 con 0x7f65ac09cdc0 2024-09-16T15:01:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 1 -- 172.21.15.26:0/1094774176 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3953311933 0 0) 0x7f6598003530 con 0x7f65ac09e7a0 2024-09-16T15:01:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 1 -- 172.21.15.26:0/1094774176 >> v1:172.21.15.26:6790/0 conn(0x7f65ac09cdc0 legacy=0x7f65ac130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 1 -- 172.21.15.26:0/1094774176 >> v1:172.21.15.189:6789/0 conn(0x7f65ac09d1a0 legacy=0x7f65ac139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient: _finish_hunting 0 2024-09-16T15:01:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 1 monclient: found mon.noname-a 2024-09-16T15:01:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:01:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 1 -- 172.21.15.26:0/1094774176 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ac09b2d0 con 0x7f65ac09e7a0 2024-09-16T15:01:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient: _finish_auth 0 2024-09-16T15:01:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient: _check_auth_tickets 2024-09-16T15:01:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 1 -- 172.21.15.26:0/1094774176 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f65940018d0 con 0x7f65ac09e7a0 2024-09-16T15:01:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient: dump: 2024-09-16T15:01:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.124 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 5 monclient: authenticate success, global_id 334191 2024-09-16T15:01:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 1 -- 172.21.15.26:0/1094774176 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6594001af0 con 0x7f65ac09e7a0 2024-09-16T15:01:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 1 -- 172.21.15.26:0/1094774176 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f6594001e00 con 0x7f65ac09e7a0 2024-09-16T15:01:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65b277a640 4 set_mon_vals no callback set 2024-09-16T15:01:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.113+0000 7f65aaffd640 10 monclient: dump: 2024-09-16T15:01:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.125 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 10 monclient: shutdown 2024-09-16T15:01:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 1 -- 172.21.15.26:0/1094774176 >> v1:172.21.15.26:6789/0 conn(0x7f65ac09e7a0 legacy=0x7f65ac131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 1 -- 172.21.15.26:0/1094774176 shutdown_connections 2024-09-16T15:01:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 1 -- 172.21.15.26:0/1094774176 >> 172.21.15.26:0/1094774176 conn(0x7f65ac09fb50 msgr2=0x7f65ac09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 1 -- 172.21.15.26:0/1094774176 shutdown_connections 2024-09-16T15:01:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 1 -- 172.21.15.26:0/1094774176 wait complete. 2024-09-16T15:01:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 10 monclient: build_initial_monmap 2024-09-16T15:01:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 10 monclient: monmap: 2024-09-16T15:01:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:05.127 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 1 Processor -- start 2024-09-16T15:01:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 1 -- start start 2024-09-16T15:01:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 10 monclient: init 2024-09-16T15:01:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:01:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 10 monclient(hunting): picked mon.noname-b con 0x7f65ac09cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 10 monclient(hunting): picked mon.noname-c con 0x7f65ac09d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 10 monclient(hunting): picked mon.noname-a con 0x7f65ac09e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac121570 con 0x7f65ac09e7a0 2024-09-16T15:01:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac120030 con 0x7f65ac09cdc0 2024-09-16T15:01:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac120f20 con 0x7f65ac09d1a0 2024-09-16T15:01:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 10 monclient(hunting): authenticate will time out at 2089.237750s 2024-09-16T15:01:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65ab7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f65ac09d1a0 0x7f65ac1ce670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:45580/0 (socket says 172.21.15.26:45580) 2024-09-16T15:01:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65ab7fe640 1 -- 172.21.15.26:0/3694672714 learned_addr learned my addr 172.21.15.26:0/3694672714 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822538632 0 0) 0x7f65ac121570 con 0x7f65ac09e7a0 2024-09-16T15:01:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): my global_id is 334194 2024-09-16T15:01:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f658c003530 con 0x7f65ac09e7a0 2024-09-16T15:01:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028585422 0 0) 0x7f65ac120f20 con 0x7f65ac09d1a0 2024-09-16T15:01:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): my global_id is 284720 2024-09-16T15:01:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65ac121570 con 0x7f65ac09d1a0 2024-09-16T15:01:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 663222157 0 0) 0x7f658c003530 con 0x7f65ac09e7a0 2024-09-16T15:01:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65ac120f20 con 0x7f65ac09e7a0 2024-09-16T15:01:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f65a4001800 con 0x7f65ac09e7a0 2024-09-16T15:01:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): dump: 2024-09-16T15:01:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.131 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2886264336 0 0) 0x7f65ac121570 con 0x7f65ac09d1a0 2024-09-16T15:01:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f658c003530 con 0x7f65ac09d1a0 2024-09-16T15:01:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f65940017d0 con 0x7f65ac09d1a0 2024-09-16T15:01:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): dump: 2024-09-16T15:01:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.132 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.132 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.132 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.132 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109510703 0 0) 0x7f65ac120030 con 0x7f65ac09cdc0 2024-09-16T15:01:05.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): my global_id is 344107 2024-09-16T15:01:05.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65ac121570 con 0x7f65ac09cdc0 2024-09-16T15:01:05.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4073245712 0 0) 0x7f65ac120f20 con 0x7f65ac09e7a0 2024-09-16T15:01:05.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 >> v1:172.21.15.26:6790/0 conn(0x7f65ac09d1a0 legacy=0x7f65ac1ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 >> v1:172.21.15.189:6789/0 conn(0x7f65ac09cdc0 legacy=0x7f65ac12dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient: _finish_hunting 0 2024-09-16T15:01:05.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 monclient: found mon.a 2024-09-16T15:01:05.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:05.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:05.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ac121200 con 0x7f65ac09e7a0 2024-09-16T15:01:05.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient: _finish_auth 0 2024-09-16T15:01:05.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient: _check_auth_tickets 2024-09-16T15:01:05.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:05.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 5 monclient: authenticate success, global_id 334194 2024-09-16T15:01:05.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 10 monclient: _renew_subs 2024-09-16T15:01:05.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:05.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 1 -- 172.21.15.26:0/3694672714 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65ac130650 con 0x7f65ac09e7a0 2024-09-16T15:01:05.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 10 monclient: _renew_subs 2024-09-16T15:01:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 1 -- 172.21.15.26:0/3694672714 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65ac1c95b0 con 0x7f65ac09e7a0 2024-09-16T15:01:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65a4003600 con 0x7f65ac09e7a0 2024-09-16T15:01:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f65a4003910 con 0x7f65ac09e7a0 2024-09-16T15:01:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65aa7fc640 4 set_mon_vals no callback set 2024-09-16T15:01:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient: dump: 2024-09-16T15:01:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.135 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65b277a640 1 -- 172.21.15.26:0/3694672714 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6590006590 con 0x7f6590002830 2024-09-16T15:01:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273830828 0 0) 0x7f6590006590 con 0x7f6590002830 2024-09-16T15:01:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 10 monclient(hunting): my global_id is 334197 2024-09-16T15:01:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65ac120f20 con 0x7f6590002830 2024-09-16T15:01:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f65a4003a70 con 0x7f65ac09e7a0 2024-09-16T15:01:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f65a4042a40 con 0x7f65ac09e7a0 2024-09-16T15:01:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2551580810 0 0) 0x7f65ac120f20 con 0x7f6590002830 2024-09-16T15:01:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.117+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6590006590 con 0x7f6590002830 2024-09-16T15:01:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.121+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f659c0015e0 con 0x7f6590002830 2024-09-16T15:01:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.121+0000 7f65a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.121+0000 7f65a8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.121+0000 7f65a8ff9640 10 monclient: dump: 2024-09-16T15:01:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.137 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.137 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.137 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.137 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.137 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.121+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4199608811 0 0) 0x7f6590006590 con 0x7f6590002830 2024-09-16T15:01:05.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.121+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f659000a0e0 con 0x7f6590002830 2024-09-16T15:01:05.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.121+0000 7f65ab7fe640 10 monclient: get_auth_request con 0x7f658c0363b0 auth_method 0 2024-09-16T15:01:05.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.121+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f659c0019d0 con 0x7f6590002830 2024-09-16T15:01:05.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.121+0000 7f65a8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:05.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.121+0000 7f65a8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:05.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.121+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 >> v1:172.21.15.26:6789/0 conn(0x7f6590002830 legacy=0x7f6590004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.121+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 >> v1:172.21.15.26:6789/0 conn(0x7f6590002830 legacy=0x7f6590004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:05.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65b277a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:01:05.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65b277a640 1 -- 172.21.15.26:0/3694672714 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6590009da0 con 0x7f65900090c0 2024-09-16T15:01:05.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165218586 0 0) 0x7f6590009da0 con 0x7f65900090c0 2024-09-16T15:01:05.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 10 monclient(hunting): my global_id is 334200 2024-09-16T15:01:05.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6590006590 con 0x7f65900090c0 2024-09-16T15:01:05.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2619518503 0 0) 0x7f6590006590 con 0x7f65900090c0 2024-09-16T15:01:05.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6590009da0 con 0x7f65900090c0 2024-09-16T15:01:05.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f659c004990 con 0x7f65900090c0 2024-09-16T15:01:05.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:05.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 10 monclient: dump: 2024-09-16T15:01:05.140 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.140 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.140 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.140 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.140 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.140 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.140 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.140 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.140 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.140 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3797107077 0 0) 0x7f6590009da0 con 0x7f65900090c0 2024-09-16T15:01:05.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f659000c290 con 0x7f65900090c0 2024-09-16T15:01:05.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 1760007118) 0x7f659c005f30 con 0x7f65900090c0 2024-09-16T15:01:05.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:01:05.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:05.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 >> v1:172.21.15.26:6789/0 conn(0x7f65900090c0 legacy=0x7f6590009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65a8ff9640 1 -- 172.21.15.26:0/3694672714 >> v1:172.21.15.26:6789/0 conn(0x7f65900090c0 legacy=0x7f6590009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:05.141 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:01:05.141 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:05.141 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:01:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 168, 2024-09-16T15:01:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:01:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:01:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:01:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:01:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:01:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:01:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:01:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:01:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:01:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:01:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:01:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:01:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:01:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:01:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:01:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:01:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:01:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:01:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:01:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:01:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:01:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:01:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:01:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:01:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:01:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:01:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:01:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:01:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:01:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:01:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:01:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:01:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:01:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:01:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:01:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:01:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:01:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T15:01:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:01:05.152 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:01:05.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65b277a640 1 -- 172.21.15.26:0/3694672714 >> v1:172.21.15.189:6812/3196161582 conn(0x7f658c0363b0 legacy=0x7f658c038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65b277a640 10 monclient: shutdown 2024-09-16T15:01:05.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65b277a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:05.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65b277a640 1 -- 172.21.15.26:0/3694672714 >> v1:172.21.15.26:6789/0 conn(0x7f65ac09e7a0 legacy=0x7f65ac1d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65b0d17640 1 -- 172.21.15.26:0/3694672714 reap_dead start 2024-09-16T15:01:05.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65b0d17640 1 -- 172.21.15.26:0/3694672714 reap_dead start 2024-09-16T15:01:05.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65b277a640 1 -- 172.21.15.26:0/3694672714 shutdown_connections 2024-09-16T15:01:05.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65b277a640 1 -- 172.21.15.26:0/3694672714 >> 172.21.15.26:0/3694672714 conn(0x7f65ac09fb50 msgr2=0x7f65ac09c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:05.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65b277a640 1 -- 172.21.15.26:0/3694672714 shutdown_connections 2024-09-16T15:01:05.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.125+0000 7f65b277a640 1 -- 172.21.15.26:0/3694672714 wait complete. 2024-09-16T15:01:05.154 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:01:05.154 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T15:01:05.154 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T15:01:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 10 monclient: get_monmap_and_config 2024-09-16T15:01:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 10 monclient: build_initial_monmap 2024-09-16T15:01:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 10 monclient: monmap: 2024-09-16T15:01:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:05.263 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:05.265727+0000 2024-09-16T15:01:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:05.265727+0000 2024-09-16T15:01:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:05.264 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:05.264 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 1 Processor -- start 2024-09-16T15:01:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 1 -- start start 2024-09-16T15:01:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 10 monclient: init 2024-09-16T15:01:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:01:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 10 monclient(hunting): picked mon.noname-b con 0x7f221c0a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 10 monclient(hunting): picked mon.noname-a con 0x7f221c099b60 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 10 monclient(hunting): picked mon.noname-c con 0x7f221c0a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221c098ea0 con 0x7f221c099b60 2024-09-16T15:01:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221c0997b0 con 0x7f221c0a26f0 2024-09-16T15:01:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221c099460 con 0x7f221c0a1c00 2024-09-16T15:01:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 10 monclient(hunting): authenticate will time out at 2089.385965s 2024-09-16T15:01:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f221c0a1c00 0x7f221c0a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:45594/0 (socket says 172.21.15.26:45594) 2024-09-16T15:01:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221a59c640 1 -- 172.21.15.26:0/1109288435 learned_addr learned my addr 172.21.15.26:0/1109288435 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 1 -- 172.21.15.26:0/1109288435 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501926562 0 0) 0x7f221c099460 con 0x7f221c0a1c00 2024-09-16T15:01:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient(hunting): my global_id is 284723 2024-09-16T15:01:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 1 -- 172.21.15.26:0/1109288435 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2204003530 con 0x7f221c0a1c00 2024-09-16T15:01:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 1 -- 172.21.15.26:0/1109288435 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124025473 0 0) 0x7f221c098ea0 con 0x7f221c099b60 2024-09-16T15:01:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient(hunting): my global_id is 334203 2024-09-16T15:01:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 1 -- 172.21.15.26:0/1109288435 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f221c099460 con 0x7f221c099b60 2024-09-16T15:01:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 1 -- 172.21.15.26:0/1109288435 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2693102503 0 0) 0x7f2204003530 con 0x7f221c0a1c00 2024-09-16T15:01:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 1 -- 172.21.15.26:0/1109288435 >> v1:172.21.15.189:6789/0 conn(0x7f221c0a26f0 legacy=0x7f221c13d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 1 -- 172.21.15.26:0/1109288435 >> v1:172.21.15.26:6789/0 conn(0x7f221c099b60 legacy=0x7f221c139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient: _finish_hunting 0 2024-09-16T15:01:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 1 monclient: found mon.noname-b 2024-09-16T15:01:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:01:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 1 -- 172.21.15.26:0/1109288435 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221c0a34b0 con 0x7f221c0a1c00 2024-09-16T15:01:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient: _finish_auth 0 2024-09-16T15:01:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient: _check_auth_tickets 2024-09-16T15:01:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 1 -- 172.21.15.26:0/1109288435 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22140019a0 con 0x7f221c0a1c00 2024-09-16T15:01:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 5 monclient: authenticate success, global_id 284723 2024-09-16T15:01:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:01:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient: dump: 2024-09-16T15:01:05.270 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.271 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 1 -- 172.21.15.26:0/1109288435 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2214001c80 con 0x7f221c0a1c00 2024-09-16T15:01:05.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 1 -- 172.21.15.26:0/1109288435 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2214002050 con 0x7f221c0a1c00 2024-09-16T15:01:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 4 set_mon_vals no callback set 2024-09-16T15:01:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f221959a640 10 monclient: dump: 2024-09-16T15:01:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.272 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.273 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 10 monclient: shutdown 2024-09-16T15:01:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 1 -- 172.21.15.26:0/1109288435 >> v1:172.21.15.26:6790/0 conn(0x7f221c0a1c00 legacy=0x7f221c0a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 1 -- 172.21.15.26:0/1109288435 shutdown_connections 2024-09-16T15:01:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 1 -- 172.21.15.26:0/1109288435 >> 172.21.15.26:0/1109288435 conn(0x7f221c0a3d00 msgr2=0x7f221c0a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 1 -- 172.21.15.26:0/1109288435 shutdown_connections 2024-09-16T15:01:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.265+0000 7f2220f9f640 1 -- 172.21.15.26:0/1109288435 wait complete. 2024-09-16T15:01:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 10 monclient: build_initial_monmap 2024-09-16T15:01:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 10 monclient: monmap: 2024-09-16T15:01:05.273 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:05.274 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 1 Processor -- start 2024-09-16T15:01:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 1 -- start start 2024-09-16T15:01:05.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 10 monclient: init 2024-09-16T15:01:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:01:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 10 monclient(hunting): picked mon.noname-b con 0x7f221c099b60 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 10 monclient(hunting): picked mon.noname-a con 0x7f221c0a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 10 monclient(hunting): picked mon.noname-c con 0x7f221c0a26f0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221c11e850 con 0x7f221c0a1c00 2024-09-16T15:01:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221c09ea90 con 0x7f221c099b60 2024-09-16T15:01:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221c13ccb0 con 0x7f221c0a26f0 2024-09-16T15:01:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 10 monclient(hunting): authenticate will time out at 2089.389001s 2024-09-16T15:01:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2219d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f221c0a1c00 0x7f221c13c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36774/0 (socket says 172.21.15.26:36774) 2024-09-16T15:01:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2219d9b640 1 -- 172.21.15.26:0/2150316920 learned_addr learned my addr 172.21.15.26:0/2150316920 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388643789 0 0) 0x7f221c13ccb0 con 0x7f221c0a26f0 2024-09-16T15:01:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): my global_id is 284726 2024-09-16T15:01:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21fc003530 con 0x7f221c0a26f0 2024-09-16T15:01:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021964663 0 0) 0x7f221c11e850 con 0x7f221c0a1c00 2024-09-16T15:01:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): my global_id is 334206 2024-09-16T15:01:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f221c13ccb0 con 0x7f221c0a1c00 2024-09-16T15:01:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2885794192 0 0) 0x7f21fc003530 con 0x7f221c0a26f0 2024-09-16T15:01:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f221c11e850 con 0x7f221c0a26f0 2024-09-16T15:01:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2306038708 0 0) 0x7f221c13ccb0 con 0x7f221c0a1c00 2024-09-16T15:01:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21fc003530 con 0x7f221c0a1c00 2024-09-16T15:01:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f220c001840 con 0x7f221c0a26f0 2024-09-16T15:01:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:01:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): dump: 2024-09-16T15:01:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.283 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2210001910 con 0x7f221c0a1c00 2024-09-16T15:01:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): dump: 2024-09-16T15:01:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.284 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964237876 0 0) 0x7f221c09ea90 con 0x7f221c099b60 2024-09-16T15:01:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): my global_id is 344113 2024-09-16T15:01:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f221c13ccb0 con 0x7f221c099b60 2024-09-16T15:01:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1632179687 0 0) 0x7f221c11e850 con 0x7f221c0a26f0 2024-09-16T15:01:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 >> v1:172.21.15.189:6789/0 conn(0x7f221c099b60 legacy=0x7f221c13be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 >> v1:172.21.15.26:6789/0 conn(0x7f221c0a1c00 legacy=0x7f221c13c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient: _finish_hunting 0 2024-09-16T15:01:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 monclient: found mon.c 2024-09-16T15:01:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221c09efe0 con 0x7f221c0a26f0 2024-09-16T15:01:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient: _finish_auth 0 2024-09-16T15:01:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient: _check_auth_tickets 2024-09-16T15:01:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 5 monclient: authenticate success, global_id 284726 2024-09-16T15:01:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 10 monclient: _renew_subs 2024-09-16T15:01:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 1 -- 172.21.15.26:0/2150316920 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f221c11eb30 con 0x7f221c0a26f0 2024-09-16T15:01:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 10 monclient: _renew_subs 2024-09-16T15:01:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 1 -- 172.21.15.26:0/2150316920 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f221c13d9d0 con 0x7f221c0a26f0 2024-09-16T15:01:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f220c003620 con 0x7f221c0a26f0 2024-09-16T15:01:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f220c003930 con 0x7f221c0a26f0 2024-09-16T15:01:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2218d99640 4 set_mon_vals no callback set 2024-09-16T15:01:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient: dump: 2024-09-16T15:01:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.287 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2220f9f640 1 -- 172.21.15.26:0/2150316920 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2200006590 con 0x7f2200002830 2024-09-16T15:01:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f220c00e3a0 con 0x7f221c0a26f0 2024-09-16T15:01:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f220c042a50 con 0x7f221c0a26f0 2024-09-16T15:01:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4680908 0 0) 0x7f2200006590 con 0x7f2200002830 2024-09-16T15:01:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient(hunting): my global_id is 284729 2024-09-16T15:01:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f221c11e850 con 0x7f2200002830 2024-09-16T15:01:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f2219d9b640 10 monclient: get_auth_request con 0x7f21fc0363c0 auth_method 0 2024-09-16T15:01:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 982294473 0 0) 0x7f221c11e850 con 0x7f2200002830 2024-09-16T15:01:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2200006590 con 0x7f2200002830 2024-09-16T15:01:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2214001960 con 0x7f2200002830 2024-09-16T15:01:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient: dump: 2024-09-16T15:01:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.289 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 139711620 0 0) 0x7f2200006590 con 0x7f2200002830 2024-09-16T15:01:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f220000a0e0 con 0x7f2200002830 2024-09-16T15:01:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2214001ee0 con 0x7f2200002830 2024-09-16T15:01:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 >> v1:172.21.15.26:6790/0 conn(0x7f2200002830 legacy=0x7f2200004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.269+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 >> v1:172.21.15.26:6790/0 conn(0x7f2200002830 legacy=0x7f2200004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f2220f9f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:01:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f2220f9f640 1 -- 172.21.15.26:0/2150316920 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2200009da0 con 0x7f22000090c0 2024-09-16T15:01:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208642937 0 0) 0x7f2200009da0 con 0x7f22000090c0 2024-09-16T15:01:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 10 monclient(hunting): my global_id is 284732 2024-09-16T15:01:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2200006590 con 0x7f22000090c0 2024-09-16T15:01:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 635649489 0 0) 0x7f2200006590 con 0x7f22000090c0 2024-09-16T15:01:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2200009da0 con 0x7f22000090c0 2024-09-16T15:01:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22140051e0 con 0x7f22000090c0 2024-09-16T15:01:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 10 monclient: dump: 2024-09-16T15:01:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.292 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3943736140 0 0) 0x7f2200009da0 con 0x7f22000090c0 2024-09-16T15:01:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f220000c290 con 0x7f22000090c0 2024-09-16T15:01:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 96045486) 0x7f2214007560 con 0x7f22000090c0 2024-09-16T15:01:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:01:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 >> v1:172.21.15.26:6790/0 conn(0x7f22000090c0 legacy=0x7f2200009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f220affd640 1 -- 172.21.15.26:0/2150316920 >> v1:172.21.15.26:6790/0 conn(0x7f22000090c0 legacy=0x7f2200009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:05.293 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:01:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T15:01:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 168, 2024-09-16T15:01:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:01:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:01:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:01:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:01:05.293 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:01:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:01:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:01:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:05.294 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:01:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:01:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:05.295 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:01:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:01:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:01:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:01:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:01:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:01:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:05.296 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:01:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:01:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:01:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:01:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:01:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:01:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:01:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:01:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:05.297 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:01:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:01:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:05.298 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:05.299 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:05.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.299 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:05.299 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:01:05.299 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:05.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.299 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.299 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:01:05.299 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:01:05.299 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:05.299 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:05.299 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:05.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.300 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:05.300 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:05.300 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:05.300 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:05.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.300 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:05.300 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:01:05.300 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:05.300 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.300 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.300 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:01:05.301 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:01:05.301 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:05.301 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:05.301 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:05.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.301 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:05.301 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:05.301 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:05.301 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:05.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.301 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:05.301 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:01:05.302 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:05.302 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.302 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.302 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:01:05.302 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:01:05.302 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:05.302 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:05.302 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:05.302 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.302 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.302 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:01:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:01:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T15:01:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T15:01:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:05.303 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:01:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T15:01:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T15:01:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.305 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:01:05.305 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:01:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f2220f9f640 1 -- 172.21.15.26:0/2150316920 >> v1:172.21.15.189:6812/3196161582 conn(0x7f21fc0363c0 legacy=0x7f21fc038880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f2220f9f640 10 monclient: shutdown 2024-09-16T15:01:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f2220f9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f2220f9f640 1 -- 172.21.15.26:0/2150316920 >> v1:172.21.15.26:6790/0 conn(0x7f221c0a26f0 legacy=0x7f221c1d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f221ad9d640 1 -- 172.21.15.26:0/2150316920 reap_dead start 2024-09-16T15:01:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f221ad9d640 1 -- 172.21.15.26:0/2150316920 reap_dead start 2024-09-16T15:01:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f2220f9f640 1 -- 172.21.15.26:0/2150316920 shutdown_connections 2024-09-16T15:01:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f2220f9f640 1 -- 172.21.15.26:0/2150316920 >> 172.21.15.26:0/2150316920 conn(0x7f221c0a3d00 msgr2=0x7f221c0a0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f2220f9f640 1 -- 172.21.15.26:0/2150316920 shutdown_connections 2024-09-16T15:01:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.277+0000 7f2220f9f640 1 -- 172.21.15.26:0/2150316920 wait complete. 2024-09-16T15:01:05.307 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:01:05.307 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T15:01:05.307 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T15:01:05.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 10 monclient: get_monmap_and_config 2024-09-16T15:01:05.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 10 monclient: build_initial_monmap 2024-09-16T15:01:05.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 10 monclient: monmap: 2024-09-16T15:01:05.420 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:05.420 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.420 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:05.422171+0000 2024-09-16T15:01:05.420 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:05.422171+0000 2024-09-16T15:01:05.420 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:05.420 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.420 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:05.421 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:05.421 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:05.421 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 1 Processor -- start 2024-09-16T15:01:05.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 1 -- start start 2024-09-16T15:01:05.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 10 monclient: init 2024-09-16T15:01:05.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:05.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:01:05.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 10 monclient(hunting): picked mon.noname-a con 0x7f34f4134e40 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:05.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 10 monclient(hunting): picked mon.noname-c con 0x7f34f4099a70 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:05.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 10 monclient(hunting): picked mon.noname-b con 0x7f34f4135220 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:05.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f4099e80 con 0x7f34f4134e40 2024-09-16T15:01:05.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f40719d0 con 0x7f34f4099a70 2024-09-16T15:01:05.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f409e1e0 con 0x7f34f4135220 2024-09-16T15:01:05.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:05.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 10 monclient(hunting): authenticate will time out at 2089.542188s 2024-09-16T15:01:05.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34f9999640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f34f4135220 0x7f34f413db30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:45616/0 (socket says 172.21.15.26:45616) 2024-09-16T15:01:05.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34f9198640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f34f4134e40 0x7f34f4070bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36784/0 (socket says 172.21.15.26:36784) 2024-09-16T15:01:05.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34f9999640 1 -- 172.21.15.26:0/2715562736 learned_addr learned my addr 172.21.15.26:0/2715562736 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:05.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 1 -- 172.21.15.26:0/2715562736 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1613978601 0 0) 0x7f34e4001120 con 0x7f34f4134e40 2024-09-16T15:01:05.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient(hunting): my global_id is 334209 2024-09-16T15:01:05.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 1 -- 172.21.15.26:0/2715562736 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34e0003530 con 0x7f34f4134e40 2024-09-16T15:01:05.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 1 -- 172.21.15.26:0/2715562736 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088874044 0 0) 0x7f34f409e1e0 con 0x7f34f4135220 2024-09-16T15:01:05.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient(hunting): my global_id is 284735 2024-09-16T15:01:05.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 1 -- 172.21.15.26:0/2715562736 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34e4001120 con 0x7f34f4135220 2024-09-16T15:01:05.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 1 -- 172.21.15.26:0/2715562736 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1379596384 0 0) 0x7f34e0003530 con 0x7f34f4134e40 2024-09-16T15:01:05.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 1 -- 172.21.15.26:0/2715562736 >> v1:172.21.15.26:6790/0 conn(0x7f34f4135220 legacy=0x7f34f413db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 1 -- 172.21.15.26:0/2715562736 >> v1:172.21.15.189:6789/0 conn(0x7f34f4099a70 legacy=0x7f34f40712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient: _finish_hunting 0 2024-09-16T15:01:05.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 1 monclient: found mon.noname-a 2024-09-16T15:01:05.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:05.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:01:05.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 1 -- 172.21.15.26:0/2715562736 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f409e7d0 con 0x7f34f4134e40 2024-09-16T15:01:05.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient: _finish_auth 0 2024-09-16T15:01:05.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient: _check_auth_tickets 2024-09-16T15:01:05.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:05.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 1 -- 172.21.15.26:0/2715562736 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34e4001610 con 0x7f34f4134e40 2024-09-16T15:01:05.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 5 monclient: authenticate success, global_id 334209 2024-09-16T15:01:05.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:05.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:05.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient: dump: 2024-09-16T15:01:05.432 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.432 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.432 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.432 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.432 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.432 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.432 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.432 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.432 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.432 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:05.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 1 -- 172.21.15.26:0/2715562736 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34e4001840 con 0x7f34f4134e40 2024-09-16T15:01:05.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:05.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 1 -- 172.21.15.26:0/2715562736 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34e4001c10 con 0x7f34f4134e40 2024-09-16T15:01:05.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:05.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 4 set_mon_vals no callback set 2024-09-16T15:01:05.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:05.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34ebfff640 10 monclient: dump: 2024-09-16T15:01:05.433 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.433 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.433 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.433 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.434 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.434 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.434 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.434 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.434 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.434 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 10 monclient: shutdown 2024-09-16T15:01:05.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:05.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 1 -- 172.21.15.26:0/2715562736 >> v1:172.21.15.26:6789/0 conn(0x7f34f4134e40 legacy=0x7f34f4070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 1 -- 172.21.15.26:0/2715562736 shutdown_connections 2024-09-16T15:01:05.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 1 -- 172.21.15.26:0/2715562736 >> 172.21.15.26:0/2715562736 conn(0x7f34f409fb50 msgr2=0x7f34f409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:05.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 1 -- 172.21.15.26:0/2715562736 shutdown_connections 2024-09-16T15:01:05.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.421+0000 7f34fb3fc640 1 -- 172.21.15.26:0/2715562736 wait complete. 2024-09-16T15:01:05.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 10 monclient: build_initial_monmap 2024-09-16T15:01:05.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 10 monclient: monmap: 2024-09-16T15:01:05.435 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:05.435 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:05.435 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:05.435 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:05.435 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:05.435 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:05.435 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:05.435 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:05.435 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:05.436 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 1 Processor -- start 2024-09-16T15:01:05.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 1 -- start start 2024-09-16T15:01:05.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 10 monclient: init 2024-09-16T15:01:05.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:05.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:01:05.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 10 monclient(hunting): picked mon.noname-b con 0x7f34f4099a70 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:05.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 10 monclient(hunting): picked mon.noname-a con 0x7f34f4134e40 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:05.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 10 monclient(hunting): picked mon.noname-c con 0x7f34f4135220 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:05.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f4124330 con 0x7f34f4134e40 2024-09-16T15:01:05.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f41224c0 con 0x7f34f4099a70 2024-09-16T15:01:05.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f409a240 con 0x7f34f4135220 2024-09-16T15:01:05.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:05.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 10 monclient(hunting): authenticate will time out at 2089.545268s 2024-09-16T15:01:05.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34f9999640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f34f4135220 0x7f34f41d2250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:45618/0 (socket says 172.21.15.26:45618) 2024-09-16T15:01:05.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34f9999640 1 -- 172.21.15.26:0/1164954107 learned_addr learned my addr 172.21.15.26:0/1164954107 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:05.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699593195 0 0) 0x7f34f4124330 con 0x7f34f4134e40 2024-09-16T15:01:05.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): my global_id is 334212 2024-09-16T15:01:05.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d8003530 con 0x7f34f4134e40 2024-09-16T15:01:05.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056905621 0 0) 0x7f34f409a240 con 0x7f34f4135220 2024-09-16T15:01:05.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): my global_id is 284738 2024-09-16T15:01:05.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f4124330 con 0x7f34f4135220 2024-09-16T15:01:05.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 460139500 0 0) 0x7f34d8003530 con 0x7f34f4134e40 2024-09-16T15:01:05.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34f409a240 con 0x7f34f4134e40 2024-09-16T15:01:05.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34f00017a0 con 0x7f34f4134e40 2024-09-16T15:01:05.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:05.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): dump: 2024-09-16T15:01:05.438 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.438 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.438 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.438 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.438 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.439 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.439 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.439 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.439 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.439 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:05.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3190471188 0 0) 0x7f34f4124330 con 0x7f34f4135220 2024-09-16T15:01:05.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34d8003530 con 0x7f34f4135220 2024-09-16T15:01:05.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34ec001d50 con 0x7f34f4135220 2024-09-16T15:01:05.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:05.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): dump: 2024-09-16T15:01:05.439 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.440 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.440 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.440 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.440 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.440 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.440 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.440 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.440 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.440 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414415975 0 0) 0x7f34f41224c0 con 0x7f34f4099a70 2024-09-16T15:01:05.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): my global_id is 344119 2024-09-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f4124330 con 0x7f34f4099a70 2024-09-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 128263350 0 0) 0x7f34f409a240 con 0x7f34f4134e40 2024-09-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 >> v1:172.21.15.26:6790/0 conn(0x7f34f4135220 legacy=0x7f34f41d2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 >> v1:172.21.15.189:6789/0 conn(0x7f34f4099a70 legacy=0x7f34f41c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient: _finish_hunting 0 2024-09-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 monclient: found mon.a 2024-09-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f4123d70 con 0x7f34f4134e40 2024-09-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient: _finish_auth 0 2024-09-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 5 monclient: authenticate success, global_id 334212 2024-09-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 10 monclient: _renew_subs 2024-09-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 1 -- 172.21.15.26:0/1164954107 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f4121f60 con 0x7f34f4134e40 2024-09-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 10 monclient: _renew_subs 2024-09-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 1 -- 172.21.15.26:0/1164954107 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f409c210 con 0x7f34f4134e40 2024-09-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34f0001d50 con 0x7f34f4134e40 2024-09-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34f00037a0 con 0x7f34f4134e40 2024-09-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient: dump: 2024-09-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34eb7fe640 4 set_mon_vals no callback set 2024-09-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:05.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34fb3fc640 1 -- 172.21.15.26:0/1164954107 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f34d4006590 con 0x7f34d4002830 2024-09-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f34f0003900 con 0x7f34f4134e40 2024-09-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f34f0042af0 con 0x7f34f4134e40 2024-09-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34f9999640 10 monclient: get_auth_request con 0x7f34d8036440 auth_method 0 2024-09-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363562358 0 0) 0x7f34d4006590 con 0x7f34d4002830 2024-09-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 10 monclient(hunting): my global_id is 344122 2024-09-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.425+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f409a240 con 0x7f34d4002830 2024-09-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.429+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1660733488 0 0) 0x7f34f409a240 con 0x7f34d4002830 2024-09-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.429+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34d4006590 con 0x7f34d4002830 2024-09-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.429+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34e4001190 con 0x7f34d4002830 2024-09-16T15:01:05.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.429+0000 7f34e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.429+0000 7f34e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.429+0000 7f34e9ffb640 10 monclient: dump: 2024-09-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.429+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1038964609 0 0) 0x7f34d4006590 con 0x7f34d4002830 2024-09-16T15:01:05.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.429+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34d400a0e0 con 0x7f34d4002830 2024-09-16T15:01:05.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.429+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f34e4001c60 con 0x7f34d4002830 2024-09-16T15:01:05.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.429+0000 7f34e9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:05.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.429+0000 7f34e9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:05.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.429+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 >> v1:172.21.15.189:6789/0 conn(0x7f34d4002830 legacy=0x7f34d4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.429+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 >> v1:172.21.15.189:6789/0 conn(0x7f34d4002830 legacy=0x7f34d4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:05.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.433+0000 7f34fb3fc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:01:05.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.433+0000 7f34fb3fc640 1 -- 172.21.15.26:0/1164954107 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f34d4009da0 con 0x7f34d40090c0 2024-09-16T15:01:05.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495296647 0 0) 0x7f34d4009da0 con 0x7f34d40090c0 2024-09-16T15:01:05.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 10 monclient(hunting): my global_id is 344125 2024-09-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d4006590 con 0x7f34d40090c0 2024-09-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1817943368 0 0) 0x7f34d4006590 con 0x7f34d40090c0 2024-09-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34d4009da0 con 0x7f34d40090c0 2024-09-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34e4004c20 con 0x7f34d40090c0 2024-09-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 10 monclient: dump: 2024-09-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1577644243 0 0) 0x7f34d4009da0 con 0x7f34d40090c0 2024-09-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f34d400c290 con 0x7f34d40090c0 2024-09-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 1784644686) 0x7f34e4005f30 con 0x7f34d40090c0 2024-09-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 >> v1:172.21.15.189:6789/0 conn(0x7f34d40090c0 legacy=0x7f34d4009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34e9ffb640 1 -- 172.21.15.26:0/1164954107 >> v1:172.21.15.189:6789/0 conn(0x7f34d40090c0 legacy=0x7f34d4009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 168, 2024-09-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:01:05.450 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:01:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:01:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:01:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:05.451 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:01:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:01:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:05.452 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:01:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:01:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:01:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:01:05.453 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:01:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:01:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:01:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:01:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:01:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:01:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:01:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:01:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:01:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:01:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:01:05.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:01:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:01:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:01:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:01:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.457 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:01:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:01:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:01:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.458 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:01:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:01:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:01:05.459 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:01:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:01:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.460 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:01:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:05.461 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:01:05.461 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:01:05.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34fb3fc640 1 -- 172.21.15.26:0/1164954107 >> v1:172.21.15.189:6812/3196161582 conn(0x7f34d8036440 legacy=0x7f34d8038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34fb3fc640 10 monclient: shutdown 2024-09-16T15:01:05.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34fb3fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:05.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34fb3fc640 1 -- 172.21.15.26:0/1164954107 >> v1:172.21.15.26:6789/0 conn(0x7f34f4134e40 legacy=0x7f34f41ceb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34f9999640 1 -- 172.21.15.26:0/1164954107 reap_dead start 2024-09-16T15:01:05.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34f9999640 1 -- 172.21.15.26:0/1164954107 reap_dead start 2024-09-16T15:01:05.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34fb3fc640 1 -- 172.21.15.26:0/1164954107 shutdown_connections 2024-09-16T15:01:05.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34fb3fc640 1 -- 172.21.15.26:0/1164954107 >> 172.21.15.26:0/1164954107 conn(0x7f34f409fb50 msgr2=0x7f34f40a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:05.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34fb3fc640 1 -- 172.21.15.26:0/1164954107 shutdown_connections 2024-09-16T15:01:05.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.437+0000 7f34fb3fc640 1 -- 172.21.15.26:0/1164954107 wait complete. 2024-09-16T15:01:05.463 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:01:05.464 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T15:01:05.464 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T15:01:05.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.573+0000 7f22f0ba9640 10 monclient: get_monmap_and_config 2024-09-16T15:01:05.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.573+0000 7f22f0ba9640 10 monclient: build_initial_monmap 2024-09-16T15:01:05.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.573+0000 7f22f0ba9640 10 monclient: monmap: 2024-09-16T15:01:05.575 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:05.575 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.575 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:05.576920+0000 2024-09-16T15:01:05.575 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:05.576920+0000 2024-09-16T15:01:05.575 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:05.575 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.575 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:05.575 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:05.575 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:05.575 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 1 Processor -- start 2024-09-16T15:01:05.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 1 -- start start 2024-09-16T15:01:05.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 10 monclient: init 2024-09-16T15:01:05.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:05.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:01:05.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 10 monclient(hunting): picked mon.noname-a con 0x7f22ec09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:05.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 10 monclient(hunting): picked mon.noname-b con 0x7f22ec132650 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:05.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 10 monclient(hunting): picked mon.noname-c con 0x7f22ec138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:05.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec09d260 con 0x7f22ec09cb80 2024-09-16T15:01:05.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec132a80 con 0x7f22ec138ff0 2024-09-16T15:01:05.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec09cf60 con 0x7f22ec132650 2024-09-16T15:01:05.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:05.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 10 monclient(hunting): authenticate will time out at 2089.696985s 2024-09-16T15:01:05.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22ea59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f22ec09cb80 0x7f22ec0717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36788/0 (socket says 172.21.15.26:36788) 2024-09-16T15:01:05.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22ea59c640 1 -- 172.21.15.26:0/961373416 learned_addr learned my addr 172.21.15.26:0/961373416 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:05.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 1 -- 172.21.15.26:0/961373416 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206802499 0 0) 0x7f22ec09d260 con 0x7f22ec09cb80 2024-09-16T15:01:05.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient(hunting): my global_id is 334215 2024-09-16T15:01:05.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 1 -- 172.21.15.26:0/961373416 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d4003530 con 0x7f22ec09cb80 2024-09-16T15:01:05.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 1 -- 172.21.15.26:0/961373416 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069187132 0 0) 0x7f22ec09cf60 con 0x7f22ec132650 2024-09-16T15:01:05.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient(hunting): my global_id is 284741 2024-09-16T15:01:05.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 1 -- 172.21.15.26:0/961373416 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22ec09d260 con 0x7f22ec132650 2024-09-16T15:01:05.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 1 -- 172.21.15.26:0/961373416 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3977416350 0 0) 0x7f22d4003530 con 0x7f22ec09cb80 2024-09-16T15:01:05.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 1 -- 172.21.15.26:0/961373416 >> v1:172.21.15.26:6790/0 conn(0x7f22ec132650 legacy=0x7f22ec1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 1 -- 172.21.15.26:0/961373416 >> v1:172.21.15.189:6789/0 conn(0x7f22ec138ff0 legacy=0x7f22ec13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient: _finish_hunting 0 2024-09-16T15:01:05.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 1 monclient: found mon.noname-a 2024-09-16T15:01:05.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:05.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:01:05.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 1 -- 172.21.15.26:0/961373416 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec09f2d0 con 0x7f22ec09cb80 2024-09-16T15:01:05.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient: _finish_auth 0 2024-09-16T15:01:05.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient: _check_auth_tickets 2024-09-16T15:01:05.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:05.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 1 -- 172.21.15.26:0/961373416 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22dc001620 con 0x7f22ec09cb80 2024-09-16T15:01:05.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:05.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient: dump: 2024-09-16T15:01:05.581 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.581 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.582 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.582 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.582 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.582 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.582 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.582 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.582 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.582 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 5 monclient: authenticate success, global_id 334215 2024-09-16T15:01:05.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:05.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 1 -- 172.21.15.26:0/961373416 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22dc001810 con 0x7f22ec09cb80 2024-09-16T15:01:05.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:05.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 1 -- 172.21.15.26:0/961373416 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22dc001be0 con 0x7f22ec09cb80 2024-09-16T15:01:05.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:05.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 4 set_mon_vals no callback set 2024-09-16T15:01:05.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:05.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22e959a640 10 monclient: dump: 2024-09-16T15:01:05.583 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.583 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.583 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.583 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.583 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.583 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.583 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.583 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.583 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.584 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 10 monclient: shutdown 2024-09-16T15:01:05.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:05.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 1 -- 172.21.15.26:0/961373416 >> v1:172.21.15.26:6789/0 conn(0x7f22ec09cb80 legacy=0x7f22ec0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 1 -- 172.21.15.26:0/961373416 shutdown_connections 2024-09-16T15:01:05.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 1 -- 172.21.15.26:0/961373416 >> 172.21.15.26:0/961373416 conn(0x7f22ec09fb50 msgr2=0x7f22ec09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:05.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 1 -- 172.21.15.26:0/961373416 shutdown_connections 2024-09-16T15:01:05.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 1 -- 172.21.15.26:0/961373416 wait complete. 2024-09-16T15:01:05.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 10 monclient: build_initial_monmap 2024-09-16T15:01:05.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 10 monclient: monmap: 2024-09-16T15:01:05.584 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:05.584 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:05.584 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:05.585 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:05.585 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:05.585 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:05.585 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:05.585 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:05.585 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:05.585 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 1 Processor -- start 2024-09-16T15:01:05.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 1 -- start start 2024-09-16T15:01:05.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.577+0000 7f22f0ba9640 10 monclient: init 2024-09-16T15:01:05.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:05.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:01:05.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 10 monclient(hunting): picked mon.noname-c con 0x7f22ec09cb80 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:05.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 10 monclient(hunting): picked mon.noname-a con 0x7f22ec132650 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:05.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 10 monclient(hunting): picked mon.noname-b con 0x7f22ec138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:05.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec06e2b0 con 0x7f22ec132650 2024-09-16T15:01:05.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec06e7e0 con 0x7f22ec138ff0 2024-09-16T15:01:05.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec06eac0 con 0x7f22ec09cb80 2024-09-16T15:01:05.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:05.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 10 monclient(hunting): authenticate will time out at 2089.700224s 2024-09-16T15:01:05.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22ea59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f22ec09cb80 0x7f22ec1cddd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:45636/0 (socket says 172.21.15.26:45636) 2024-09-16T15:01:05.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22ea59c640 1 -- 172.21.15.26:0/3129275507 learned_addr learned my addr 172.21.15.26:0/3129275507 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:05.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551288245 0 0) 0x7f22ec06e2b0 con 0x7f22ec132650 2024-09-16T15:01:05.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): my global_id is 334218 2024-09-16T15:01:05.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22cc003530 con 0x7f22ec132650 2024-09-16T15:01:05.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709942876 0 0) 0x7f22ec06eac0 con 0x7f22ec09cb80 2024-09-16T15:01:05.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): my global_id is 284744 2024-09-16T15:01:05.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22ec06e2b0 con 0x7f22ec09cb80 2024-09-16T15:01:05.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2138048357 0 0) 0x7f22ec06e2b0 con 0x7f22ec09cb80 2024-09-16T15:01:05.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22ec06eac0 con 0x7f22ec09cb80 2024-09-16T15:01:05.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22dc001b20 con 0x7f22ec09cb80 2024-09-16T15:01:05.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:01:05.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): dump: 2024-09-16T15:01:05.591 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.591 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.591 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.591 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.591 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.591 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.591 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.591 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.592 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.592 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:05.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4042730871 0 0) 0x7f22cc003530 con 0x7f22ec132650 2024-09-16T15:01:05.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22ec06e2b0 con 0x7f22ec132650 2024-09-16T15:01:05.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22e00013c0 con 0x7f22ec132650 2024-09-16T15:01:05.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:05.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): dump: 2024-09-16T15:01:05.592 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.592 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.592 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.592 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.593 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.593 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.593 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.593 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.593 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.593 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012817689 0 0) 0x7f22ec06e7e0 con 0x7f22ec138ff0 2024-09-16T15:01:05.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:05.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:05.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:05.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient(hunting): my global_id is 344131 2024-09-16T15:01:05.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22cc003530 con 0x7f22ec138ff0 2024-09-16T15:01:05.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 897676322 0 0) 0x7f22ec06eac0 con 0x7f22ec09cb80 2024-09-16T15:01:05.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 >> v1:172.21.15.189:6789/0 conn(0x7f22ec138ff0 legacy=0x7f22ec1d64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 >> v1:172.21.15.26:6789/0 conn(0x7f22ec132650 legacy=0x7f22ec1d2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient: _finish_hunting 0 2024-09-16T15:01:05.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 monclient: found mon.c 2024-09-16T15:01:05.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:05.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:05.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec09c010 con 0x7f22ec09cb80 2024-09-16T15:01:05.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient: _finish_auth 0 2024-09-16T15:01:05.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient: _check_auth_tickets 2024-09-16T15:01:05.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:05.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 5 monclient: authenticate success, global_id 284744 2024-09-16T15:01:05.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 10 monclient: _renew_subs 2024-09-16T15:01:05.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:05.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 1 -- 172.21.15.26:0/3129275507 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22ec09a240 con 0x7f22ec09cb80 2024-09-16T15:01:05.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 10 monclient: _renew_subs 2024-09-16T15:01:05.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:05.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 1 -- 172.21.15.26:0/3129275507 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22ec12c8b0 con 0x7f22ec09cb80 2024-09-16T15:01:05.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22dc004b20 con 0x7f22ec09cb80 2024-09-16T15:01:05.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:05.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22dc004e30 con 0x7f22ec09cb80 2024-09-16T15:01:05.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:05.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:05.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22e8d99640 4 set_mon_vals no callback set 2024-09-16T15:01:05.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 10 monclient: dump: 2024-09-16T15:01:05.595 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:05.595 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:05.595 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.595 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:05.596 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:05.596 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:05.596 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:05.596 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:05.596 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:05.596 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:05.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:05.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f22dc004f90 con 0x7f22ec09cb80 2024-09-16T15:01:05.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:05.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:05.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22f0ba9640 1 -- 172.21.15.26:0/3129275507 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22d0005050 con 0x7f22ec09cb80 2024-09-16T15:01:05.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f22dc043f30 con 0x7f22ec09cb80 2024-09-16T15:01:05.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.581+0000 7f22e9d9b640 10 monclient: get_auth_request con 0x7f22cc0363d0 auth_method 0 2024-09-16T15:01:05.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.585+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f22dc0148d0 con 0x7f22ec09cb80 2024-09-16T15:01:05.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.585+0000 7f22daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:05.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.585+0000 7f22daffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:05.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.805+0000 7f22f0ba9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T15:01:05.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.805+0000 7f22f0ba9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T15:01:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.805+0000 7f22f0ba9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.805+0000 7f22f0ba9640 1 -- 172.21.15.26:0/3129275507 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f22d0005b10 con 0x7f22ec09cb80 2024-09-16T15:01:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.805+0000 7f22daffd640 1 -- 172.21.15.26:0/3129275507 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f22dc012d40 con 0x7f22ec09cb80 2024-09-16T15:01:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.805+0000 7f22daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T15:01:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.805+0000 7f22daffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:05.806 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.805+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.806 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.805+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=11,kv=11,logm=49} crc {auth=3504238702,config=692619927,health=3809114041,kv=1867599010,logm=2082363983}) 2024-09-16T15:01:05.807 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.805+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=2165166245,config=692619927,health=1372882202,kv=3088479044,logm=3708904690}) 2024-09-16T15:01:05.807 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.805+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.807 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.805+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=11,kv=11,logm=49} crc {auth=3504238702,config=692619927,health=3809114041,kv=1867599010,logm=2082363983}) 2024-09-16T15:01:05.807 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.805+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=12,kv=11,logm=48} crc {auth=2383541524,config=692619927,health=406396989,kv=1867599010,logm=2558442237}) 2024-09-16T15:01:05.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.809+0000 7f22f0ba9640 1 -- 172.21.15.26:0/3129275507 >> v1:172.21.15.189:6812/3196161582 conn(0x7f22cc0363d0 legacy=0x7f22cc038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.809+0000 7f22f0ba9640 10 monclient: shutdown 2024-09-16T15:01:05.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.809+0000 7f22f0ba9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:05.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.809+0000 7f22f0ba9640 1 -- 172.21.15.26:0/3129275507 >> v1:172.21.15.26:6790/0 conn(0x7f22ec09cb80 legacy=0x7f22ec1cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.809+0000 7f22f0ba9640 1 -- 172.21.15.26:0/3129275507 shutdown_connections 2024-09-16T15:01:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.809+0000 7f22f0ba9640 1 -- 172.21.15.26:0/3129275507 >> 172.21.15.26:0/3129275507 conn(0x7f22ec09fb50 msgr2=0x7f22ec0a2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.809+0000 7f22f0ba9640 1 -- 172.21.15.26:0/3129275507 shutdown_connections 2024-09-16T15:01:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:05.809+0000 7f22f0ba9640 1 -- 172.21.15.26:0/3129275507 wait complete. 2024-09-16T15:01:05.810 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.809+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.810 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.809+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2024539555}) 2024-09-16T15:01:05.810 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.809+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2944463276}) 2024-09-16T15:01:05.810 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.809+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.810 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.809+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2024539555}) 2024-09-16T15:01:05.810 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.809+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=119810741}) 2024-09-16T15:01:05.812 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.813+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.812 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.813+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1494245866}) 2024-09-16T15:01:05.812 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.813+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3453813336}) 2024-09-16T15:01:05.812 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.813+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.812 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.813+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1494245866}) 2024-09-16T15:01:05.812 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.813+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2093738505}) 2024-09-16T15:01:05.815 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.813+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.815 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.813+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014118974}) 2024-09-16T15:01:05.815 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.813+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2830267017}) 2024-09-16T15:01:05.815 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.813+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.815 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.813+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014118974}) 2024-09-16T15:01:05.815 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.817+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4249753940}) 2024-09-16T15:01:05.822 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.817+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.822 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.817+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2053785568}) 2024-09-16T15:01:05.822 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.817+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3027762569}) 2024-09-16T15:01:05.822 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.817+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.822 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.817+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2053785568}) 2024-09-16T15:01:05.822 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.817+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=393967211}) 2024-09-16T15:01:05.822 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.821+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.822 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.821+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1331506995}) 2024-09-16T15:01:05.822 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.821+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3595049130}) 2024-09-16T15:01:05.822 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.821+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.822 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.821+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1331506995}) 2024-09-16T15:01:05.822 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.821+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1971398185}) 2024-09-16T15:01:05.823 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.821+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.823 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.821+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2004618765}) 2024-09-16T15:01:05.823 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.821+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2615375765}) 2024-09-16T15:01:05.824 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.821+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.824 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.821+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2004618765}) 2024-09-16T15:01:05.824 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.821+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4230801801}) 2024-09-16T15:01:05.825 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T15:01:05.825 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.825+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.825 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.825+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3032151916}) 2024-09-16T15:01:05.825 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.825+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2092378113}) 2024-09-16T15:01:05.825 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.825+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.826 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.825+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3032151916}) 2024-09-16T15:01:05.826 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.825+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=352739501}) 2024-09-16T15:01:05.826 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.825+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.826 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.825+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4080619561}) 2024-09-16T15:01:05.826 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.825+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3520552281}) 2024-09-16T15:01:05.827 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.825+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.827 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.825+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4080619561}) 2024-09-16T15:01:05.827 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.825+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2443131905}) 2024-09-16T15:01:05.828 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.829+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.828 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.829+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1913880101}) 2024-09-16T15:01:05.828 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.829+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2997053016}) 2024-09-16T15:01:05.828 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.829+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.829 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.829+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1913880101}) 2024-09-16T15:01:05.829 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.829+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2257899504}) 2024-09-16T15:01:05.830 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.829+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.830 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.829+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2151404562}) 2024-09-16T15:01:05.830 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.829+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544754659}) 2024-09-16T15:01:05.830 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.829+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.830 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.829+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2151404562}) 2024-09-16T15:01:05.830 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.829+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634183430}) 2024-09-16T15:01:05.832 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.833+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.832 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.833+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3037523102}) 2024-09-16T15:01:05.832 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.833+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2717749355}) 2024-09-16T15:01:05.832 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.833+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.832 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.833+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3037523102}) 2024-09-16T15:01:05.832 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.833+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1519546158}) 2024-09-16T15:01:05.834 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.833+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.834 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.833+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=943410928}) 2024-09-16T15:01:05.834 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.833+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=465940315}) 2024-09-16T15:01:05.834 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.833+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.834 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.833+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=943410928}) 2024-09-16T15:01:05.835 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.833+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2731419673}) 2024-09-16T15:01:05.836 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.837+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.836 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.837+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3349032174}) 2024-09-16T15:01:05.836 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.837+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=751975727}) 2024-09-16T15:01:05.836 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.837+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.837 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.837+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3349032174}) 2024-09-16T15:01:05.837 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.837+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1059496384}) 2024-09-16T15:01:05.838 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.837+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.838 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.837+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1545187446}) 2024-09-16T15:01:05.838 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.837+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022422471}) 2024-09-16T15:01:05.838 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.837+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.838 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.837+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1545187446}) 2024-09-16T15:01:05.839 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.837+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=291814627}) 2024-09-16T15:01:05.840 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.841+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.840 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.841+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=821960098}) 2024-09-16T15:01:05.841 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.841+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=276444988}) 2024-09-16T15:01:05.841 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.841+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.841 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.841+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=821960098}) 2024-09-16T15:01:05.841 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.841+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=450940393}) 2024-09-16T15:01:05.842 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.841+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.842 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.841+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4271035884}) 2024-09-16T15:01:05.843 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.841+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=993752915}) 2024-09-16T15:01:05.843 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.841+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.843 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.841+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4271035884}) 2024-09-16T15:01:05.843 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.841+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3771759159}) 2024-09-16T15:01:05.844 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.845+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.844 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.845+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=368674487}) 2024-09-16T15:01:05.844 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.845+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2688122360}) 2024-09-16T15:01:05.845 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.845+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.845 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.845+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=368674487}) 2024-09-16T15:01:05.845 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.845+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1976136006}) 2024-09-16T15:01:05.847 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.845+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.847 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.845+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4264318134}) 2024-09-16T15:01:05.847 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.845+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749586257}) 2024-09-16T15:01:05.847 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.845+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.847 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.845+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4264318134}) 2024-09-16T15:01:05.847 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.845+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3109631159}) 2024-09-16T15:01:05.848 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.849+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.849 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.849+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=206469546}) 2024-09-16T15:01:05.849 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.849+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955266068}) 2024-09-16T15:01:05.849 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.849+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.849 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.849+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=206469546}) 2024-09-16T15:01:05.849 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.849+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2968275497}) 2024-09-16T15:01:05.851 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.849+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.851 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.849+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2131655224}) 2024-09-16T15:01:05.851 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.849+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159998290}) 2024-09-16T15:01:05.852 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.849+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.852 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.849+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2131655224}) 2024-09-16T15:01:05.852 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.849+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086303838}) 2024-09-16T15:01:05.853 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.853+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.853 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.853+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1123822409}) 2024-09-16T15:01:05.853 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.853+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2960510458}) 2024-09-16T15:01:05.854 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.853+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.854 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.853+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1123822409}) 2024-09-16T15:01:05.854 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.853+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2182012644}) 2024-09-16T15:01:05.855 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.853+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.856 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.853+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2899784291}) 2024-09-16T15:01:05.856 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.853+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1671223436}) 2024-09-16T15:01:05.856 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.853+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.856 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.853+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2899784291}) 2024-09-16T15:01:05.856 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.853+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2992909043}) 2024-09-16T15:01:05.859 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.857+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.859 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.857+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3114379649}) 2024-09-16T15:01:05.859 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.857+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=919942686}) 2024-09-16T15:01:05.859 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.857+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.859 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.857+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3114379649}) 2024-09-16T15:01:05.859 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.857+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1230989164}) 2024-09-16T15:01:05.860 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.857+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.860 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.857+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1988666827}) 2024-09-16T15:01:05.860 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.857+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1088866084}) 2024-09-16T15:01:05.860 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.857+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.860 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.857+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1988666827}) 2024-09-16T15:01:05.860 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.857+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=914891456}) 2024-09-16T15:01:05.862 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.861+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.862 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.861+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=156005283}) 2024-09-16T15:01:05.862 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.861+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3491624148}) 2024-09-16T15:01:05.863 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.861+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.863 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.861+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=156005283}) 2024-09-16T15:01:05.863 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.861+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2502534182}) 2024-09-16T15:01:05.863 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.861+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.863 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.861+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=772090890}) 2024-09-16T15:01:05.863 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.861+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3690382738}) 2024-09-16T15:01:05.863 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.861+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.863 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.861+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=772090890}) 2024-09-16T15:01:05.863 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.861+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2485587813}) 2024-09-16T15:01:05.864 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.865+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.864 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.865+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3710435817}) 2024-09-16T15:01:05.864 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.865+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2244082366}) 2024-09-16T15:01:05.864 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.865+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.864 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.865+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3710435817}) 2024-09-16T15:01:05.864 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.865+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=218170306}) 2024-09-16T15:01:05.866 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.865+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.866 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.865+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2069441125}) 2024-09-16T15:01:05.866 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.865+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2394253539}) 2024-09-16T15:01:05.866 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.865+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.866 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.865+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2069441125}) 2024-09-16T15:01:05.866 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.865+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1067913677}) 2024-09-16T15:01:05.867 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.869+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.867 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.869+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3116182676}) 2024-09-16T15:01:05.868 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.869+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1573294955}) 2024-09-16T15:01:05.868 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.869+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.868 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.869+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3116182676}) 2024-09-16T15:01:05.868 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.869+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1966293170}) 2024-09-16T15:01:05.869 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.869+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.869 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.869+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1780529832}) 2024-09-16T15:01:05.869 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.869+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=399307014}) 2024-09-16T15:01:05.870 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.869+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.870 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.869+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1780529832}) 2024-09-16T15:01:05.870 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.869+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1816612864}) 2024-09-16T15:01:05.871 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.873+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.871 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.873+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2312847494}) 2024-09-16T15:01:05.871 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.873+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2992135311}) 2024-09-16T15:01:05.871 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.873+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.871 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.873+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2312847494}) 2024-09-16T15:01:05.871 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.873+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=161299034}) 2024-09-16T15:01:05.873 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.873+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.873 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.873+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1557398774}) 2024-09-16T15:01:05.873 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.873+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457950710}) 2024-09-16T15:01:05.874 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.873+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.874 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.873+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1557398774}) 2024-09-16T15:01:05.874 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.873+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3845216729}) 2024-09-16T15:01:05.875 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.877+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.875 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.877+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3779535087}) 2024-09-16T15:01:05.875 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.877+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622136811}) 2024-09-16T15:01:05.875 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.877+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.875 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.877+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3779535087}) 2024-09-16T15:01:05.876 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.877+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2750181129}) 2024-09-16T15:01:05.877 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.877+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.877 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.877+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2860216582}) 2024-09-16T15:01:05.877 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.877+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=247123326}) 2024-09-16T15:01:05.877 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.877+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.877 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.877+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2860216582}) 2024-09-16T15:01:05.877 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.877+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2839024563}) 2024-09-16T15:01:05.879 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.877+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.879 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.877+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1573420760}) 2024-09-16T15:01:05.879 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.877+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1290373850}) 2024-09-16T15:01:05.879 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.877+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.879 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.881+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1573420760}) 2024-09-16T15:01:05.879 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.881+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=801595231}) 2024-09-16T15:01:05.881 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.881+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.881 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.881+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=317146685}) 2024-09-16T15:01:05.881 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.881+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2021082583}) 2024-09-16T15:01:05.881 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.881+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.881 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.881+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=317146685}) 2024-09-16T15:01:05.881 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.881+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3566684445}) 2024-09-16T15:01:05.883 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.881+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.883 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.881+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=447701909}) 2024-09-16T15:01:05.883 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.881+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906122988}) 2024-09-16T15:01:05.883 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.881+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.884 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.881+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=447701909}) 2024-09-16T15:01:05.884 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.881+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=552381132}) 2024-09-16T15:01:05.885 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.885+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.885 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.885+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4192800818}) 2024-09-16T15:01:05.885 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.885+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3746433712}) 2024-09-16T15:01:05.885 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.885+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.886 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.885+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4192800818}) 2024-09-16T15:01:05.886 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.885+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=899757463}) 2024-09-16T15:01:05.887 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.885+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.887 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.885+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2615491538}) 2024-09-16T15:01:05.887 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.885+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=116070417}) 2024-09-16T15:01:05.887 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.885+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.887 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.885+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2615491538}) 2024-09-16T15:01:05.888 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.885+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782523474}) 2024-09-16T15:01:05.889 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.889+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.889 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.889+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4271430206}) 2024-09-16T15:01:05.889 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.889+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2453036831}) 2024-09-16T15:01:05.889 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.889+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.889 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.889+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4271430206}) 2024-09-16T15:01:05.889 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.889+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2266926392}) 2024-09-16T15:01:05.890 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.889+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.890 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.889+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2387718448}) 2024-09-16T15:01:05.890 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.889+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480279398}) 2024-09-16T15:01:05.890 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.889+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.891 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.889+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2387718448}) 2024-09-16T15:01:05.891 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.889+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2335124220}) 2024-09-16T15:01:05.892 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.893+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.892 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.893+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=50738339}) 2024-09-16T15:01:05.892 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.893+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415485097}) 2024-09-16T15:01:05.892 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.893+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.892 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.893+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=50738339}) 2024-09-16T15:01:05.893 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.893+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=88170010}) 2024-09-16T15:01:05.894 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.893+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.894 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.893+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=619640807}) 2024-09-16T15:01:05.894 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.893+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2717368710}) 2024-09-16T15:01:05.894 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.893+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.894 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.893+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=619640807}) 2024-09-16T15:01:05.894 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.893+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=575994630}) 2024-09-16T15:01:05.896 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.897+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.896 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.897+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1544297401}) 2024-09-16T15:01:05.896 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.897+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2929847723}) 2024-09-16T15:01:05.896 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.897+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.896 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.897+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1544297401}) 2024-09-16T15:01:05.896 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.897+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=789528613}) 2024-09-16T15:01:05.897 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.897+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.897 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.897+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1775641539}) 2024-09-16T15:01:05.898 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.897+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2821317711}) 2024-09-16T15:01:05.898 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.897+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.898 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.897+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1775641539}) 2024-09-16T15:01:05.898 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.897+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1448313759}) 2024-09-16T15:01:05.899 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.901+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.899 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.901+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3281233758}) 2024-09-16T15:01:05.899 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.901+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4043843947}) 2024-09-16T15:01:05.899 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.901+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.900 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.901+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3281233758}) 2024-09-16T15:01:05.900 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.901+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2672849321}) 2024-09-16T15:01:05.901 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.901+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.901 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.901+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3579869968}) 2024-09-16T15:01:05.901 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.901+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770396619}) 2024-09-16T15:01:05.902 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.901+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.902 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.901+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3579869968}) 2024-09-16T15:01:05.902 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.901+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1825192557}) 2024-09-16T15:01:05.903 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.903 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4050095139}) 2024-09-16T15:01:05.903 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714072001}) 2024-09-16T15:01:05.903 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.903 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4050095139}) 2024-09-16T15:01:05.904 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1457778497}) 2024-09-16T15:01:05.905 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.905 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4262679930}) 2024-09-16T15:01:05.905 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1429064500}) 2024-09-16T15:01:05.905 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.905 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4262679930}) 2024-09-16T15:01:05.905 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1168262853}) 2024-09-16T15:01:05.907 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.907 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3089690189}) 2024-09-16T15:01:05.907 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859431168}) 2024-09-16T15:01:05.907 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.907 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3089690189}) 2024-09-16T15:01:05.907 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.905+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2147774574}) 2024-09-16T15:01:05.908 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.909+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.909 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.909+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2259066720}) 2024-09-16T15:01:05.909 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.909+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1244727868}) 2024-09-16T15:01:05.909 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.909+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.909 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.909+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2259066720}) 2024-09-16T15:01:05.909 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.909+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=88639589}) 2024-09-16T15:01:05.910 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.909+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.910 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.909+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2773164370}) 2024-09-16T15:01:05.910 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.909+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=817708464}) 2024-09-16T15:01:05.910 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.909+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.910 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.909+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2773164370}) 2024-09-16T15:01:05.910 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.909+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2152440145}) 2024-09-16T15:01:05.912 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.913+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.912 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.913+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=483034304}) 2024-09-16T15:01:05.912 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.913+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2186415460}) 2024-09-16T15:01:05.912 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.913+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.912 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.913+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=483034304}) 2024-09-16T15:01:05.912 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.913+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2110447216}) 2024-09-16T15:01:05.913 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.913+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.914 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.913+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1118332794}) 2024-09-16T15:01:05.914 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.913+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=641094660}) 2024-09-16T15:01:05.914 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.913+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.914 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.913+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1118332794}) 2024-09-16T15:01:05.914 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.913+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=820985215}) 2024-09-16T15:01:05.915 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.915 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=861201678}) 2024-09-16T15:01:05.915 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3474484758}) 2024-09-16T15:01:05.915 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.915 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=861201678}) 2024-09-16T15:01:05.915 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3020052870}) 2024-09-16T15:01:05.917 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.917 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2214636242}) 2024-09-16T15:01:05.917 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1819175836}) 2024-09-16T15:01:05.917 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.917 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2214636242}) 2024-09-16T15:01:05.918 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545076604}) 2024-09-16T15:01:05.919 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.919 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1725341094}) 2024-09-16T15:01:05.919 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=214940394}) 2024-09-16T15:01:05.919 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.919 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1725341094}) 2024-09-16T15:01:05.919 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.917+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2660189092}) 2024-09-16T15:01:05.920 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.921+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.920 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.921+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=849493039}) 2024-09-16T15:01:05.921 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.921+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1276622300}) 2024-09-16T15:01:05.921 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.921+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.921 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.921+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=849493039}) 2024-09-16T15:01:05.921 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.921+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2834463349}) 2024-09-16T15:01:05.922 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.921+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.922 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.921+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1890898911}) 2024-09-16T15:01:05.923 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.921+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2466359388}) 2024-09-16T15:01:05.923 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.921+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.923 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.921+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1890898911}) 2024-09-16T15:01:05.923 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.921+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4247836017}) 2024-09-16T15:01:05.924 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.925+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.924 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.925+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=944623247}) 2024-09-16T15:01:05.925 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.925+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=242401871}) 2024-09-16T15:01:05.925 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.925+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.925 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.925+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=944623247}) 2024-09-16T15:01:05.925 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.925+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=16014620}) 2024-09-16T15:01:05.926 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.925+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.926 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.925+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2701565938}) 2024-09-16T15:01:05.926 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.925+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2549378488}) 2024-09-16T15:01:05.926 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.925+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.926 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.925+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2701565938}) 2024-09-16T15:01:05.926 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.925+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2259935710}) 2024-09-16T15:01:05.928 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.929+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.928 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.929+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3107943427}) 2024-09-16T15:01:05.928 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.929+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501422908}) 2024-09-16T15:01:05.928 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.929+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.928 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.929+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3107943427}) 2024-09-16T15:01:05.928 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.929+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2114549183}) 2024-09-16T15:01:05.930 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.929+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.930 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.929+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2664525389}) 2024-09-16T15:01:05.930 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.929+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=823568004}) 2024-09-16T15:01:05.930 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.929+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.930 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.929+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2664525389}) 2024-09-16T15:01:05.930 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.929+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3598402589}) 2024-09-16T15:01:05.931 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.933+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.932 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.933+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2726072593}) 2024-09-16T15:01:05.932 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.933+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1602045055}) 2024-09-16T15:01:05.932 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.933+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.932 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.933+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2726072593}) 2024-09-16T15:01:05.932 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.933+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4141951067}) 2024-09-16T15:01:05.933 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.933+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.933 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.933+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2200964795}) 2024-09-16T15:01:05.934 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.933+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3515504377}) 2024-09-16T15:01:05.934 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.933+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.934 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.933+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2200964795}) 2024-09-16T15:01:05.934 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.933+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=925438896}) 2024-09-16T15:01:05.935 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.935 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=796529226}) 2024-09-16T15:01:05.935 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299800281}) 2024-09-16T15:01:05.935 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.936 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=796529226}) 2024-09-16T15:01:05.936 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1881842211}) 2024-09-16T15:01:05.937 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.937 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007469191}) 2024-09-16T15:01:05.937 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=499707366}) 2024-09-16T15:01:05.937 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.937 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007469191}) 2024-09-16T15:01:05.937 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=181347204}) 2024-09-16T15:01:05.939 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.939 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4146054508}) 2024-09-16T15:01:05.939 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597024673}) 2024-09-16T15:01:05.939 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.939 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4146054508}) 2024-09-16T15:01:05.939 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.937+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2614882685}) 2024-09-16T15:01:05.940 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.941+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.940 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.941+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1762516075}) 2024-09-16T15:01:05.940 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.941+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2727583803}) 2024-09-16T15:01:05.941 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.941+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.941 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.941+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1762516075}) 2024-09-16T15:01:05.941 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.941+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3340813619}) 2024-09-16T15:01:05.942 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.941+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.942 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.941+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1630441719}) 2024-09-16T15:01:05.942 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.941+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2587405256}) 2024-09-16T15:01:05.942 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.941+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.942 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.941+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1630441719}) 2024-09-16T15:01:05.942 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.941+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=457828664}) 2024-09-16T15:01:05.943 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.943 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1348667529}) 2024-09-16T15:01:05.944 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090675026}) 2024-09-16T15:01:05.944 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.944 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1348667529}) 2024-09-16T15:01:05.944 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=722513570}) 2024-09-16T15:01:05.945 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.945 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1874369417}) 2024-09-16T15:01:05.945 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3387988078}) 2024-09-16T15:01:05.945 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.945 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1874369417}) 2024-09-16T15:01:05.946 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1797741280}) 2024-09-16T15:01:05.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4044156934}) 2024-09-16T15:01:05.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4073225636}) 2024-09-16T15:01:05.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4044156934}) 2024-09-16T15:01:05.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.945+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1192852945}) 2024-09-16T15:01:05.948 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.949+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.948 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.949+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=174510923}) 2024-09-16T15:01:05.948 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.949+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1440191746}) 2024-09-16T15:01:05.949 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.949+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.949 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.949+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=174510923}) 2024-09-16T15:01:05.949 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.949+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4079386222}) 2024-09-16T15:01:05.950 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.949+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.950 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.949+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126924317}) 2024-09-16T15:01:05.950 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.949+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307552611}) 2024-09-16T15:01:05.950 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.949+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.950 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.949+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126924317}) 2024-09-16T15:01:05.950 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.949+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4261448228}) 2024-09-16T15:01:05.952 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.953+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.952 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.953+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3729504094}) 2024-09-16T15:01:05.952 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.953+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=130031651}) 2024-09-16T15:01:05.952 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.953+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.952 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.953+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3729504094}) 2024-09-16T15:01:05.952 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.953+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1705131378}) 2024-09-16T15:01:05.953 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.953+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.953 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.953+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3396444132}) 2024-09-16T15:01:05.953 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.953+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2726611889}) 2024-09-16T15:01:05.954 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.953+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.954 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.953+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3396444132}) 2024-09-16T15:01:05.954 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.953+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2650629597}) 2024-09-16T15:01:05.955 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.955 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3989926195}) 2024-09-16T15:01:05.955 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2582077975}) 2024-09-16T15:01:05.956 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.956 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3989926195}) 2024-09-16T15:01:05.956 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3844811369}) 2024-09-16T15:01:05.957 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.957 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1278967676}) 2024-09-16T15:01:05.957 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820086531}) 2024-09-16T15:01:05.957 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.957 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1278967676}) 2024-09-16T15:01:05.957 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=361851719}) 2024-09-16T15:01:05.959 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.959 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2790424644}) 2024-09-16T15:01:05.959 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234425012}) 2024-09-16T15:01:05.959 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.959 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2790424644}) 2024-09-16T15:01:05.959 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.957+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=827737218}) 2024-09-16T15:01:05.960 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.961+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.960 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.961+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181040139}) 2024-09-16T15:01:05.960 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.961+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1371214383}) 2024-09-16T15:01:05.961 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.961+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.961 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.961+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181040139}) 2024-09-16T15:01:05.961 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.961+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3849112290}) 2024-09-16T15:01:05.962 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.961+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.962 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.961+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1166864235}) 2024-09-16T15:01:05.962 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.961+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=576366448}) 2024-09-16T15:01:05.962 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.961+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.962 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.961+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1166864235}) 2024-09-16T15:01:05.962 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.961+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2065090138}) 2024-09-16T15:01:05.964 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.965+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.964 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.965+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3498077973}) 2024-09-16T15:01:05.964 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.965+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=865079886}) 2024-09-16T15:01:05.964 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.965+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.964 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.965+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3498077973}) 2024-09-16T15:01:05.964 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.965+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3547713674}) 2024-09-16T15:01:05.966 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.965+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.966 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.965+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1807280934}) 2024-09-16T15:01:05.966 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.965+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=38825204}) 2024-09-16T15:01:05.966 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.965+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.966 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.965+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1807280934}) 2024-09-16T15:01:05.966 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.965+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2703222627}) 2024-09-16T15:01:05.967 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.969+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.968 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.969+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1032651849}) 2024-09-16T15:01:05.968 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.969+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2353150103}) 2024-09-16T15:01:05.968 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.969+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.968 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.969+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1032651849}) 2024-09-16T15:01:05.968 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.969+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3777939509}) 2024-09-16T15:01:05.969 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.969+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.969 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.969+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=279330931}) 2024-09-16T15:01:05.969 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.969+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1666771086}) 2024-09-16T15:01:05.969 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.969+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.970 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.969+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=279330931}) 2024-09-16T15:01:05.970 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.969+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3336128287}) 2024-09-16T15:01:05.971 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.973+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.971 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.973+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3079280642}) 2024-09-16T15:01:05.971 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.973+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1387028557}) 2024-09-16T15:01:05.971 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.973+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.971 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.973+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3079280642}) 2024-09-16T15:01:05.972 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.973+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3160173614}) 2024-09-16T15:01:05.973 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.973+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.973 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.973+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=235158901}) 2024-09-16T15:01:05.973 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.973+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1342093164}) 2024-09-16T15:01:05.973 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.973+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.973 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.973+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=235158901}) 2024-09-16T15:01:05.973 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.973+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1413946586}) 2024-09-16T15:01:05.975 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.975 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3571805505}) 2024-09-16T15:01:05.975 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1797792649}) 2024-09-16T15:01:05.975 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.975 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3571805505}) 2024-09-16T15:01:05.975 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2513507845}) 2024-09-16T15:01:05.977 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.977 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=618974750}) 2024-09-16T15:01:05.977 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2686515942}) 2024-09-16T15:01:05.977 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.978 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=618974750}) 2024-09-16T15:01:05.978 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=425470224}) 2024-09-16T15:01:05.979 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.979 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574314124}) 2024-09-16T15:01:05.979 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053185585}) 2024-09-16T15:01:05.979 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.980 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574314124}) 2024-09-16T15:01:05.980 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.977+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3189985529}) 2024-09-16T15:01:05.981 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.981+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.981 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.981+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2277193144}) 2024-09-16T15:01:05.981 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.981+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4015244484}) 2024-09-16T15:01:05.982 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.981+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.982 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.981+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2277193144}) 2024-09-16T15:01:05.982 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.981+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3757197501}) 2024-09-16T15:01:05.983 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.981+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.983 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.981+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3921185041}) 2024-09-16T15:01:05.983 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.981+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=928850902}) 2024-09-16T15:01:05.983 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.981+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.983 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.981+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3921185041}) 2024-09-16T15:01:05.983 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.981+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2944393323}) 2024-09-16T15:01:05.985 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.985+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.985 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.985+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1064099954}) 2024-09-16T15:01:05.985 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.985+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2033437710}) 2024-09-16T15:01:05.985 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.985+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.985 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.985+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1064099954}) 2024-09-16T15:01:05.985 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.985+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3223113355}) 2024-09-16T15:01:05.986 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.985+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.987 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.985+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2227542790}) 2024-09-16T15:01:05.987 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.985+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2786911924}) 2024-09-16T15:01:05.987 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.985+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.987 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.985+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2227542790}) 2024-09-16T15:01:05.987 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.985+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3395037457}) 2024-09-16T15:01:05.988 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.989+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.988 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.989+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3011038666}) 2024-09-16T15:01:05.988 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.989+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491703414}) 2024-09-16T15:01:05.988 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.989+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.988 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.989+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3011038666}) 2024-09-16T15:01:05.988 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.989+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=154489930}) 2024-09-16T15:01:05.990 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.989+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.990 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.989+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3294502198}) 2024-09-16T15:01:05.990 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.989+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4037829281}) 2024-09-16T15:01:05.990 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.989+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.990 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.989+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3294502198}) 2024-09-16T15:01:05.990 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.989+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=280761103}) 2024-09-16T15:01:05.992 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.993+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.992 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.993+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2859875881}) 2024-09-16T15:01:05.992 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.993+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3584241724}) 2024-09-16T15:01:05.992 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.993+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.992 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.993+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2859875881}) 2024-09-16T15:01:05.992 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.993+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=136855365}) 2024-09-16T15:01:05.994 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.993+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.994 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.993+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2578777511}) 2024-09-16T15:01:05.994 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.993+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1840468261}) 2024-09-16T15:01:05.994 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.993+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.994 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.993+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2578777511}) 2024-09-16T15:01:05.994 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.993+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3474926909}) 2024-09-16T15:01:05.995 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.997+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.995 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.997+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4178459077}) 2024-09-16T15:01:05.996 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.997+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=879131563}) 2024-09-16T15:01:05.996 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.997+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.996 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.997+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4178459077}) 2024-09-16T15:01:05.996 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.997+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2607073516}) 2024-09-16T15:01:05.998 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.997+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.998 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.997+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295642623}) 2024-09-16T15:01:05.998 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.997+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3861126181}) 2024-09-16T15:01:05.998 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.997+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:05.998 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.997+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295642623}) 2024-09-16T15:01:05.998 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:05.997+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589249261}) 2024-09-16T15:01:06.000 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.001+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.000 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.001+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3791558387}) 2024-09-16T15:01:06.000 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.001+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2112885930}) 2024-09-16T15:01:06.000 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.001+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.000 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.001+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3791558387}) 2024-09-16T15:01:06.000 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.001+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2170190666}) 2024-09-16T15:01:06.002 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.001+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.002 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.001+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=640582745}) 2024-09-16T15:01:06.002 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.001+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2230514173}) 2024-09-16T15:01:06.002 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.001+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.002 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.001+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=640582745}) 2024-09-16T15:01:06.002 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.001+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3188225642}) 2024-09-16T15:01:06.003 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.005+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.003 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.005+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3050357033}) 2024-09-16T15:01:06.004 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.005+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4282749322}) 2024-09-16T15:01:06.004 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.005+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.004 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.005+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3050357033}) 2024-09-16T15:01:06.004 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.005+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3803173389}) 2024-09-16T15:01:06.005 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.005+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.005 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.005+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1511594100}) 2024-09-16T15:01:06.005 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.005+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597998970}) 2024-09-16T15:01:06.005 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.005+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.005 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.005+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1511594100}) 2024-09-16T15:01:06.006 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.005+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=837851306}) 2024-09-16T15:01:06.007 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.007 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3081651312}) 2024-09-16T15:01:06.007 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1202859069}) 2024-09-16T15:01:06.007 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.007 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3081651312}) 2024-09-16T15:01:06.007 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=944089308}) 2024-09-16T15:01:06.009 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.009 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1627340556}) 2024-09-16T15:01:06.009 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=495623844}) 2024-09-16T15:01:06.009 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.009 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1627340556}) 2024-09-16T15:01:06.009 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1443112804}) 2024-09-16T15:01:06.011 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.011 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3399278661}) 2024-09-16T15:01:06.011 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2761922168}) 2024-09-16T15:01:06.011 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.011 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3399278661}) 2024-09-16T15:01:06.011 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.009+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3238003695}) 2024-09-16T15:01:06.012 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.013+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.013 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.013+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=135348933}) 2024-09-16T15:01:06.013 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.013+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90210773}) 2024-09-16T15:01:06.013 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.013+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.013 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.013+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=135348933}) 2024-09-16T15:01:06.013 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.013+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2328356169}) 2024-09-16T15:01:06.014 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.013+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.014 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.013+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2945360971}) 2024-09-16T15:01:06.014 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.013+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=823642331}) 2024-09-16T15:01:06.014 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.013+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.014 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.013+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2945360971}) 2024-09-16T15:01:06.014 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.013+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1522532246}) 2024-09-16T15:01:06.016 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.017+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.016 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.017+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2537004854}) 2024-09-16T15:01:06.016 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.017+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2373084034}) 2024-09-16T15:01:06.017 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.017+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.017 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.017+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2537004854}) 2024-09-16T15:01:06.017 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.017+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=194227800}) 2024-09-16T15:01:06.018 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.017+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.018 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.017+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467092515}) 2024-09-16T15:01:06.018 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.017+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=865756194}) 2024-09-16T15:01:06.018 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.017+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.018 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.017+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467092515}) 2024-09-16T15:01:06.019 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.017+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=747745945}) 2024-09-16T15:01:06.020 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.021+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.021 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.021+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=851768869}) 2024-09-16T15:01:06.021 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.021+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4068903902}) 2024-09-16T15:01:06.021 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.021+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.021 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.021+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=851768869}) 2024-09-16T15:01:06.021 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.021+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=949909979}) 2024-09-16T15:01:06.022 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.021+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.023 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.021+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2293162234}) 2024-09-16T15:01:06.023 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.021+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820673162}) 2024-09-16T15:01:06.023 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.021+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.023 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.021+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2293162234}) 2024-09-16T15:01:06.023 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.021+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1480316551}) 2024-09-16T15:01:06.025 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.025+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.025 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.025+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3518721800}) 2024-09-16T15:01:06.025 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.025+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2851556881}) 2024-09-16T15:01:06.025 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.025+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.025 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.025+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3518721800}) 2024-09-16T15:01:06.025 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.025+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3399889090}) 2024-09-16T15:01:06.029 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.025+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.029 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.025+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197621075}) 2024-09-16T15:01:06.029 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.025+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4074154678}) 2024-09-16T15:01:06.029 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.025+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.029 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.025+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197621075}) 2024-09-16T15:01:06.029 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.025+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2088649734}) 2024-09-16T15:01:06.029 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.029+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.030 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.029+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=324032445}) 2024-09-16T15:01:06.030 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.029+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335348061}) 2024-09-16T15:01:06.030 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.029+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.030 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.029+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=324032445}) 2024-09-16T15:01:06.030 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.029+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1362484422}) 2024-09-16T15:01:06.030 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.029+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.030 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.029+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3988294568}) 2024-09-16T15:01:06.031 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.029+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3429001605}) 2024-09-16T15:01:06.031 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.029+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.031 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.029+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3988294568}) 2024-09-16T15:01:06.031 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.029+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=166974338}) 2024-09-16T15:01:06.032 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.033+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.032 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.033+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3653209461}) 2024-09-16T15:01:06.032 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.033+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2789591522}) 2024-09-16T15:01:06.032 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.033+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.033 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.033+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3653209461}) 2024-09-16T15:01:06.033 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.033+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1322450254}) 2024-09-16T15:01:06.034 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.033+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.034 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.033+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309918224}) 2024-09-16T15:01:06.034 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.033+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1453939760}) 2024-09-16T15:01:06.034 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.033+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.034 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.033+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309918224}) 2024-09-16T15:01:06.034 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.033+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4102266399}) 2024-09-16T15:01:06.036 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.037+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.036 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.037+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4053376527}) 2024-09-16T15:01:06.036 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.037+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=319415526}) 2024-09-16T15:01:06.036 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.037+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.036 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.037+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4053376527}) 2024-09-16T15:01:06.036 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.037+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2830819433}) 2024-09-16T15:01:06.037 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.037+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.037 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.037+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2169345086}) 2024-09-16T15:01:06.037 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.037+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3374840467}) 2024-09-16T15:01:06.038 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.037+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.038 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.037+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2169345086}) 2024-09-16T15:01:06.038 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.037+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3859519010}) 2024-09-16T15:01:06.039 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.039 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1150977819}) 2024-09-16T15:01:06.039 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3924893185}) 2024-09-16T15:01:06.039 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.040 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1150977819}) 2024-09-16T15:01:06.040 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=799692057}) 2024-09-16T15:01:06.041 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.041 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3115143466}) 2024-09-16T15:01:06.041 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3626047059}) 2024-09-16T15:01:06.041 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.041 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3115143466}) 2024-09-16T15:01:06.041 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2686535382}) 2024-09-16T15:01:06.042 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.043 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2911515089}) 2024-09-16T15:01:06.043 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=560385798}) 2024-09-16T15:01:06.043 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.043 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2911515089}) 2024-09-16T15:01:06.043 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.041+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=272200853}) 2024-09-16T15:01:06.044 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.045+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.045+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975611087}) 2024-09-16T15:01:06.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.045+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606569945}) 2024-09-16T15:01:06.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.045+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.045+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975611087}) 2024-09-16T15:01:06.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.045+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4043172237}) 2024-09-16T15:01:06.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.049+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.049+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=9,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=29} crc {logm=2155747424,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=502331882,mon_config_key=3475477691,monmap=970648238,osd_metadata=1173325520,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=845135638}) 2024-09-16T15:01:06.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.049+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=26} crc {logm=2655097500,mdsmap=1531151339,mgr=3587032506,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=502331882,mon_config_key=1518680965,monmap=2765269217,osd_metadata=2172984177,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=2472646211}) 2024-09-16T15:01:06.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.049+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.048 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.049+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=9,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=29} crc {logm=2155747424,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=502331882,mon_config_key=3475477691,monmap=970648238,osd_metadata=1173325520,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=845135638}) 2024-09-16T15:01:06.049 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.049+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=27,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=26} crc {logm=2223254766,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=502331882,mon_config_key=1518680965,monmap=970648238,osd_metadata=3749276052,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3968680119}) 2024-09-16T15:01:06.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.061+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.061+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=25} crc {osdmap=2049356209}) 2024-09-16T15:01:06.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.061+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=25} crc {osdmap=756857738}) 2024-09-16T15:01:06.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.061+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:06.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.061+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=25} crc {osdmap=2049356209}) 2024-09-16T15:01:06.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:06.061+0000 7f6935033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=25} crc {osdmap=51131823}) 2024-09-16T15:01:06.825 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T15:01:06.826 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T15:01:06.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34b1a50640 10 monclient: get_monmap_and_config 2024-09-16T15:01:06.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34b1a50640 10 monclient: build_initial_monmap 2024-09-16T15:01:06.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34b1a50640 10 monclient: monmap: 2024-09-16T15:01:06.957 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:06.957 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:06.957 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:06.959001+0000 2024-09-16T15:01:06.957 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:06.959001+0000 2024-09-16T15:01:06.957 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:06.957 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:06.957 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:06.957 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:06.957 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:06.957 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:06.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34b1a50640 1 Processor -- start 2024-09-16T15:01:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34b1a50640 1 -- start start 2024-09-16T15:01:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34b1a50640 10 monclient: init 2024-09-16T15:01:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34b1a50640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34b1a50640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:01:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34b1a50640 10 monclient(hunting): picked mon.noname-a con 0x7f34ac06c5a0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34b1a50640 10 monclient(hunting): picked mon.noname-b con 0x7f34ac0a23b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34b1a50640 10 monclient(hunting): picked mon.noname-c con 0x7f34ac0a2840 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34b1a50640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ac09f390 con 0x7f34ac06c5a0 2024-09-16T15:01:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34b1a50640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ac06c980 con 0x7f34ac0a2840 2024-09-16T15:01:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34b1a50640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ac09edd0 con 0x7f34ac0a23b0 2024-09-16T15:01:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34b1a50640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34b1a50640 10 monclient(hunting): authenticate will time out at 2091.079693s 2024-09-16T15:01:06.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34a27fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f34ac0a23b0 0x7f34ac139eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59874/0 (socket says 172.21.15.26:59874) 2024-09-16T15:01:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.957+0000 7f34a27fc640 1 -- 172.21.15.26:0/1927336456 learned_addr learned my addr 172.21.15.26:0/1927336456 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 1 -- 172.21.15.26:0/1927336456 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228590780 0 0) 0x7f34ac09edd0 con 0x7f34ac0a23b0 2024-09-16T15:01:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient(hunting): my global_id is 284747 2024-09-16T15:01:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 1 -- 172.21.15.26:0/1927336456 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3498003530 con 0x7f34ac0a23b0 2024-09-16T15:01:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 1 -- 172.21.15.26:0/1927336456 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283202381 0 0) 0x7f34ac09f390 con 0x7f34ac06c5a0 2024-09-16T15:01:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient(hunting): my global_id is 334221 2024-09-16T15:01:06.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 1 -- 172.21.15.26:0/1927336456 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34ac09edd0 con 0x7f34ac06c5a0 2024-09-16T15:01:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 1 -- 172.21.15.26:0/1927336456 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1495388497 0 0) 0x7f3498003530 con 0x7f34ac0a23b0 2024-09-16T15:01:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 1 -- 172.21.15.26:0/1927336456 >> v1:172.21.15.189:6789/0 conn(0x7f34ac0a2840 legacy=0x7f34ac13d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 1 -- 172.21.15.26:0/1927336456 >> v1:172.21.15.26:6789/0 conn(0x7f34ac06c5a0 legacy=0x7f34ac0a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient: _finish_hunting 0 2024-09-16T15:01:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 1 monclient: found mon.noname-b 2024-09-16T15:01:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:01:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 1 -- 172.21.15.26:0/1927336456 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac09f0b0 con 0x7f34ac0a23b0 2024-09-16T15:01:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient: _finish_auth 0 2024-09-16T15:01:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient: _check_auth_tickets 2024-09-16T15:01:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 1 -- 172.21.15.26:0/1927336456 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3494001620 con 0x7f34ac0a23b0 2024-09-16T15:01:06.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:01:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient: dump: 2024-09-16T15:01:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:06.964 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:06.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 5 monclient: authenticate success, global_id 284747 2024-09-16T15:01:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 1 -- 172.21.15.26:0/1927336456 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3494001840 con 0x7f34ac0a23b0 2024-09-16T15:01:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 4 set_mon_vals no callback set 2024-09-16T15:01:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 1 -- 172.21.15.26:0/1927336456 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3494001c10 con 0x7f34ac0a23b0 2024-09-16T15:01:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34aa7fc640 10 monclient: dump: 2024-09-16T15:01:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:06.966 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:06.966 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:06.966 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:06.966 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:06.966 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:06.966 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:06.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 10 monclient: shutdown 2024-09-16T15:01:06.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:06.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 1 -- 172.21.15.26:0/1927336456 >> v1:172.21.15.26:6790/0 conn(0x7f34ac0a23b0 legacy=0x7f34ac139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:06.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 1 -- 172.21.15.26:0/1927336456 shutdown_connections 2024-09-16T15:01:06.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 1 -- 172.21.15.26:0/1927336456 >> 172.21.15.26:0/1927336456 conn(0x7f34ac0a3da0 msgr2=0x7f34ac0a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:06.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 1 -- 172.21.15.26:0/1927336456 shutdown_connections 2024-09-16T15:01:06.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 1 -- 172.21.15.26:0/1927336456 wait complete. 2024-09-16T15:01:06.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 10 monclient: build_initial_monmap 2024-09-16T15:01:06.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 10 monclient: monmap: 2024-09-16T15:01:06.970 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:06.970 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:06.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:06.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:06.971 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:06.971 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:06.971 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:06.971 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:06.971 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:06.971 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:06.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 1 Processor -- start 2024-09-16T15:01:06.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 1 -- start start 2024-09-16T15:01:06.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 10 monclient: init 2024-09-16T15:01:06.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:06.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:01:06.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 10 monclient(hunting): picked mon.noname-b con 0x7f34ac06c5a0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:06.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 10 monclient(hunting): picked mon.noname-a con 0x7f34ac0a23b0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:06.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 10 monclient(hunting): picked mon.noname-c con 0x7f34ac0a2840 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:06.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ac09e490 con 0x7f34ac0a23b0 2024-09-16T15:01:06.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ac11fe50 con 0x7f34ac06c5a0 2024-09-16T15:01:06.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ac0a61d0 con 0x7f34ac0a2840 2024-09-16T15:01:06.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:06.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34b1a50640 10 monclient(hunting): authenticate will time out at 2091.083270s 2024-09-16T15:01:06.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a27fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f34ac0a23b0 0x7f34ac1ce320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53708/0 (socket says 172.21.15.26:53708) 2024-09-16T15:01:06.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a27fc640 1 -- 172.21.15.26:0/1175312476 learned_addr learned my addr 172.21.15.26:0/1175312476 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:06.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116770772 0 0) 0x7f34ac09e490 con 0x7f34ac0a23b0 2024-09-16T15:01:06.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:06.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:06.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:06.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 10 monclient(hunting): my global_id is 334224 2024-09-16T15:01:06.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f348c003530 con 0x7f34ac0a23b0 2024-09-16T15:01:06.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940978217 0 0) 0x7f34ac0a61d0 con 0x7f34ac0a2840 2024-09-16T15:01:06.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:06.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:06.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:06.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 10 monclient(hunting): my global_id is 284750 2024-09-16T15:01:06.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34ac09e490 con 0x7f34ac0a2840 2024-09-16T15:01:06.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3922356723 0 0) 0x7f348c003530 con 0x7f34ac0a23b0 2024-09-16T15:01:06.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34ac0a61d0 con 0x7f34ac0a23b0 2024-09-16T15:01:06.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3494001440 con 0x7f34ac0a23b0 2024-09-16T15:01:06.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:06.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:06.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 10 monclient(hunting): dump: 2024-09-16T15:01:06.974 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:06.974 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:06.974 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:06.974 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:06.974 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:06.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:06.974 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:06.974 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:06.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:06.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:06.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:06.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2942962372 0 0) 0x7f34ac09e490 con 0x7f34ac0a2840 2024-09-16T15:01:06.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.961+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f348c003530 con 0x7f34ac0a2840 2024-09-16T15:01:06.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3057391855 0 0) 0x7f34ac11fe50 con 0x7f34ac06c5a0 2024-09-16T15:01:06.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:06.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:06.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:06.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 10 monclient(hunting): my global_id is 344137 2024-09-16T15:01:06.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34ac09e490 con 0x7f34ac06c5a0 2024-09-16T15:01:06.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f34a4001c40 con 0x7f34ac0a2840 2024-09-16T15:01:06.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:06.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:06.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 10 monclient(hunting): dump: 2024-09-16T15:01:06.976 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:06.976 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:06.976 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:06.976 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:06.976 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:06.976 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:06.976 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:06.976 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:06.976 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:06.976 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:06.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2734394824 0 0) 0x7f34ac0a61d0 con 0x7f34ac0a23b0 2024-09-16T15:01:06.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 >> v1:172.21.15.26:6790/0 conn(0x7f34ac0a2840 legacy=0x7f34ac1d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:06.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 >> v1:172.21.15.189:6789/0 conn(0x7f34ac06c5a0 legacy=0x7f34ac133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:06.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 10 monclient: _finish_hunting 0 2024-09-16T15:01:06.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 1 monclient: found mon.a 2024-09-16T15:01:06.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:06.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:06.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac121370 con 0x7f34ac0a23b0 2024-09-16T15:01:06.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 10 monclient: _finish_auth 0 2024-09-16T15:01:06.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 10 monclient: _check_auth_tickets 2024-09-16T15:01:06.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:06.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34b1a50640 5 monclient: authenticate success, global_id 334224 2024-09-16T15:01:06.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34b1a50640 10 monclient: _renew_subs 2024-09-16T15:01:06.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34b1a50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:06.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34b1a50640 1 -- 172.21.15.26:0/1175312476 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34ac11fbd0 con 0x7f34ac0a23b0 2024-09-16T15:01:06.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34b1a50640 10 monclient: _renew_subs 2024-09-16T15:01:06.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34b1a50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:06.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34b1a50640 1 -- 172.21.15.26:0/1175312476 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34ac134a10 con 0x7f34ac0a23b0 2024-09-16T15:01:06.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3494004c00 con 0x7f34ac0a23b0 2024-09-16T15:01:06.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:06.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3494004f90 con 0x7f34ac0a23b0 2024-09-16T15:01:06.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:06.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:06.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a9ffb640 4 set_mon_vals no callback set 2024-09-16T15:01:06.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 10 monclient: dump: 2024-09-16T15:01:06.978 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:06.978 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:06.978 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:06.978 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:06.979 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:06.979 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:06.979 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:06.979 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:06.979 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:06.979 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:06.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f34940051b0 con 0x7f34ac0a23b0 2024-09-16T15:01:06.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34b1a50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:06.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34b1a50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:06.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34b1a50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:06.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34b1a50640 1 -- 172.21.15.26:0/1175312476 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3490005050 con 0x7f34ac0a23b0 2024-09-16T15:01:06.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f34940440d0 con 0x7f34ac0a23b0 2024-09-16T15:01:06.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.965+0000 7f34aaffd640 10 monclient: get_auth_request con 0x7f348c0363b0 auth_method 0 2024-09-16T15:01:06.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.969+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f34940148a0 con 0x7f34ac0a23b0 2024-09-16T15:01:06.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.969+0000 7f34a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:06.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:06.969+0000 7f34a8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:07.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.185+0000 7f34b1a50640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T15:01:07.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.185+0000 7f34b1a50640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:07.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.185+0000 7f34b1a50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:07.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.185+0000 7f34b1a50640 1 -- 172.21.15.26:0/1175312476 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3490005880 con 0x7f34ac0a23b0 2024-09-16T15:01:07.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.185+0000 7f34a8ff9640 1 -- 172.21.15.26:0/1175312476 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2696893384) 0x7f34940146c0 con 0x7f34ac0a23b0 2024-09-16T15:01:07.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.185+0000 7f34a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:07.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.185+0000 7f34a8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:07.186 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":168,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T15:01:07.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.189+0000 7f34b1a50640 1 -- 172.21.15.26:0/1175312476 >> v1:172.21.15.189:6812/3196161582 conn(0x7f348c0363b0 legacy=0x7f348c038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.189+0000 7f34b1a50640 10 monclient: shutdown 2024-09-16T15:01:07.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.189+0000 7f34b1a50640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:07.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.189+0000 7f34b1a50640 1 -- 172.21.15.26:0/1175312476 >> v1:172.21.15.26:6789/0 conn(0x7f34ac0a23b0 legacy=0x7f34ac1ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.189+0000 7f34b1a50640 1 -- 172.21.15.26:0/1175312476 shutdown_connections 2024-09-16T15:01:07.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.189+0000 7f34b1a50640 1 -- 172.21.15.26:0/1175312476 >> 172.21.15.26:0/1175312476 conn(0x7f34ac0a3da0 msgr2=0x7f34ac1354f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:07.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.189+0000 7f34b1a50640 1 -- 172.21.15.26:0/1175312476 shutdown_connections 2024-09-16T15:01:07.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.189+0000 7f34b1a50640 1 -- 172.21.15.26:0/1175312476 wait complete. 2024-09-16T15:01:07.202 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T15:01:07.202 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T15:01:07.202 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T15:01:07.202 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T15:01:07.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.321+0000 7f032b67d640 10 monclient: get_monmap_and_config 2024-09-16T15:01:07.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.321+0000 7f032b67d640 10 monclient: build_initial_monmap 2024-09-16T15:01:07.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.321+0000 7f032b67d640 10 monclient: monmap: 2024-09-16T15:01:07.322 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:07.322 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.322 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:07.324410+0000 2024-09-16T15:01:07.322 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:07.324410+0000 2024-09-16T15:01:07.322 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:07.323 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.323 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:07.323 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:07.323 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:07.323 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.321+0000 7f032b67d640 1 Processor -- start 2024-09-16T15:01:07.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.321+0000 7f032b67d640 1 -- start start 2024-09-16T15:01:07.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.321+0000 7f032b67d640 10 monclient: init 2024-09-16T15:01:07.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:07.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:01:07.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient(hunting): picked mon.noname-b con 0x7f03240a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:07.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient(hunting): picked mon.noname-a con 0x7f03240a0000 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:07.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient(hunting): picked mon.noname-c con 0x7f03240a2800 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:07.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03240a3250 con 0x7f03240a0000 2024-09-16T15:01:07.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032409f600 con 0x7f03240a2800 2024-09-16T15:01:07.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03240a3830 con 0x7f03240a1d10 2024-09-16T15:01:07.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:07.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient(hunting): authenticate will time out at 2091.444459s 2024-09-16T15:01:07.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0328c18640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f03240a0000 0x7f0324139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53724/0 (socket says 172.21.15.26:53724) 2024-09-16T15:01:07.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0328c18640 1 -- 172.21.15.26:0/362211387 learned_addr learned my addr 172.21.15.26:0/362211387 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:07.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 1 -- 172.21.15.26:0/362211387 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563831095 0 0) 0x7f03240a3250 con 0x7f03240a0000 2024-09-16T15:01:07.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient(hunting): my global_id is 334227 2024-09-16T15:01:07.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 1 -- 172.21.15.26:0/362211387 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0310003530 con 0x7f03240a0000 2024-09-16T15:01:07.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 1 -- 172.21.15.26:0/362211387 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706694266 0 0) 0x7f03240a3830 con 0x7f03240a1d10 2024-09-16T15:01:07.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient(hunting): my global_id is 284753 2024-09-16T15:01:07.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 1 -- 172.21.15.26:0/362211387 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03240a3250 con 0x7f03240a1d10 2024-09-16T15:01:07.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 1 -- 172.21.15.26:0/362211387 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1608892570 0 0) 0x7f0310003530 con 0x7f03240a0000 2024-09-16T15:01:07.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 1 -- 172.21.15.26:0/362211387 >> v1:172.21.15.26:6790/0 conn(0x7f03240a1d10 legacy=0x7f03240a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 1 -- 172.21.15.26:0/362211387 >> v1:172.21.15.189:6789/0 conn(0x7f03240a2800 legacy=0x7f032413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient: _finish_hunting 0 2024-09-16T15:01:07.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 1 monclient: found mon.noname-a 2024-09-16T15:01:07.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:07.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:01:07.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 1 -- 172.21.15.26:0/362211387 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032409ed60 con 0x7f03240a0000 2024-09-16T15:01:07.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient: _finish_auth 0 2024-09-16T15:01:07.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient: _check_auth_tickets 2024-09-16T15:01:07.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:07.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 1 -- 172.21.15.26:0/362211387 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0314001620 con 0x7f03240a0000 2024-09-16T15:01:07.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:07.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient: dump: 2024-09-16T15:01:07.334 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.334 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.334 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.334 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.334 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.334 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.334 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.334 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.334 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.334 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 5 monclient: authenticate success, global_id 334227 2024-09-16T15:01:07.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:07.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 1 -- 172.21.15.26:0/362211387 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03140018c0 con 0x7f03240a0000 2024-09-16T15:01:07.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:07.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 1 -- 172.21.15.26:0/362211387 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0314001c90 con 0x7f03240a0000 2024-09-16T15:01:07.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:07.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f031bfff640 10 monclient: dump: 2024-09-16T15:01:07.335 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.335 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.335 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.335 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.335 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.335 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.335 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.335 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.335 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.335 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:07.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 4 set_mon_vals no callback set 2024-09-16T15:01:07.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient: shutdown 2024-09-16T15:01:07.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:07.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 1 -- 172.21.15.26:0/362211387 >> v1:172.21.15.26:6789/0 conn(0x7f03240a0000 legacy=0x7f0324139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 1 -- 172.21.15.26:0/362211387 shutdown_connections 2024-09-16T15:01:07.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 1 -- 172.21.15.26:0/362211387 >> 172.21.15.26:0/362211387 conn(0x7f03240a7f30 msgr2=0x7f03240a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:07.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 1 -- 172.21.15.26:0/362211387 shutdown_connections 2024-09-16T15:01:07.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 1 -- 172.21.15.26:0/362211387 wait complete. 2024-09-16T15:01:07.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient: build_initial_monmap 2024-09-16T15:01:07.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient: monmap: 2024-09-16T15:01:07.336 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:07.337 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:07.337 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:07.337 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:07.337 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:07.337 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:07.337 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:07.337 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:07.337 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:07.337 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 1 Processor -- start 2024-09-16T15:01:07.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 1 -- start start 2024-09-16T15:01:07.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient: init 2024-09-16T15:01:07.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:01:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient(hunting): picked mon.noname-a con 0x7f03240a0000 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient(hunting): picked mon.noname-c con 0x7f03240a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient(hunting): picked mon.noname-b con 0x7f03240a2800 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0324134b00 con 0x7f03240a0000 2024-09-16T15:01:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0324133d20 con 0x7f03240a2800 2024-09-16T15:01:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03240a6890 con 0x7f03240a1d10 2024-09-16T15:01:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f032b67d640 10 monclient(hunting): authenticate will time out at 2091.447437s 2024-09-16T15:01:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0329419640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f03240a0000 0x7f0324132f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53730/0 (socket says 172.21.15.26:53730) 2024-09-16T15:01:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0329419640 1 -- 172.21.15.26:0/2144945879 learned_addr learned my addr 172.21.15.26:0/2144945879 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:07.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0328c18640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f03240a1d10 0x7f0324133610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59902/0 (socket says 172.21.15.26:59902) 2024-09-16T15:01:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923276975 0 0) 0x7f03240a6890 con 0x7f03240a1d10 2024-09-16T15:01:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0319ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0319ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0319ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0319ffb640 10 monclient(hunting): my global_id is 284756 2024-09-16T15:01:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0308003530 con 0x7f03240a1d10 2024-09-16T15:01:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626406475 0 0) 0x7f0324134b00 con 0x7f03240a0000 2024-09-16T15:01:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0319ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0319ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0319ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0319ffb640 10 monclient(hunting): my global_id is 334230 2024-09-16T15:01:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.325+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03240a6890 con 0x7f03240a0000 2024-09-16T15:01:07.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 22289089 0 0) 0x7f03240a6890 con 0x7f03240a0000 2024-09-16T15:01:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0324134b00 con 0x7f03240a0000 2024-09-16T15:01:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0320001a00 con 0x7f03240a0000 2024-09-16T15:01:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient(hunting): dump: 2024-09-16T15:01:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.340 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.341 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1791931537 0 0) 0x7f0308003530 con 0x7f03240a1d10 2024-09-16T15:01:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03240a6890 con 0x7f03240a1d10 2024-09-16T15:01:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0314001bd0 con 0x7f03240a1d10 2024-09-16T15:01:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient(hunting): dump: 2024-09-16T15:01:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.341 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.342 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.342 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.342 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.342 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.342 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.342 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894335540 0 0) 0x7f0324133d20 con 0x7f03240a2800 2024-09-16T15:01:07.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient(hunting): my global_id is 344143 2024-09-16T15:01:07.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0308003530 con 0x7f03240a2800 2024-09-16T15:01:07.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3209161267 0 0) 0x7f0324134b00 con 0x7f03240a0000 2024-09-16T15:01:07.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 >> v1:172.21.15.26:6790/0 conn(0x7f03240a1d10 legacy=0x7f0324133610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 >> v1:172.21.15.189:6789/0 conn(0x7f03240a2800 legacy=0x7f03241d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient: _finish_hunting 0 2024-09-16T15:01:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 monclient: found mon.a 2024-09-16T15:01:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0324136ac0 con 0x7f03240a0000 2024-09-16T15:01:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient: _finish_auth 0 2024-09-16T15:01:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f032b67d640 5 monclient: authenticate success, global_id 334230 2024-09-16T15:01:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f032b67d640 10 monclient: _renew_subs 2024-09-16T15:01:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f032b67d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:07.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f032b67d640 1 -- 172.21.15.26:0/2144945879 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0324136da0 con 0x7f03240a0000 2024-09-16T15:01:07.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f032b67d640 10 monclient: _renew_subs 2024-09-16T15:01:07.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f032b67d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:07.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f032b67d640 1 -- 172.21.15.26:0/2144945879 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0324135210 con 0x7f03240a0000 2024-09-16T15:01:07.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0320003640 con 0x7f03240a0000 2024-09-16T15:01:07.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:07.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f03200039d0 con 0x7f03240a0000 2024-09-16T15:01:07.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:07.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f031b7fe640 4 set_mon_vals no callback set 2024-09-16T15:01:07.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient: dump: 2024-09-16T15:01:07.344 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.344 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.344 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.345 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f032b67d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f032b67d640 1 -- 172.21.15.26:0/2144945879 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0304006590 con 0x7f0304002830 2024-09-16T15:01:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0320003b30 con 0x7f03240a0000 2024-09-16T15:01:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f0320043d00 con 0x7f03240a0000 2024-09-16T15:01:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378568656 0 0) 0x7f0304006590 con 0x7f0304002830 2024-09-16T15:01:07.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient(hunting): my global_id is 334233 2024-09-16T15:01:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0324134b00 con 0x7f0304002830 2024-09-16T15:01:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1755401150 0 0) 0x7f0324134b00 con 0x7f0304002830 2024-09-16T15:01:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0304006590 con 0x7f0304002830 2024-09-16T15:01:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0314004da0 con 0x7f0304002830 2024-09-16T15:01:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient: dump: 2024-09-16T15:01:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.347 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.347 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.347 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.347 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.347 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.347 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.347 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0329c1a640 10 monclient: get_auth_request con 0x7f03080363b0 auth_method 0 2024-09-16T15:01:07.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2252859608 0 0) 0x7f0304006590 con 0x7f0304002830 2024-09-16T15:01:07.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f030400a0e0 con 0x7f0304002830 2024-09-16T15:01:07.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f031400a480 con 0x7f0304002830 2024-09-16T15:01:07.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:07.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:07.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 >> v1:172.21.15.26:6789/0 conn(0x7f0304002830 legacy=0x7f0304004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.329+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 >> v1:172.21.15.26:6789/0 conn(0x7f0304002830 legacy=0x7f0304004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:07.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f032b67d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:01:07.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f032b67d640 1 -- 172.21.15.26:0/2144945879 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0304009da0 con 0x7f03040090c0 2024-09-16T15:01:07.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645686651 0 0) 0x7f0304009da0 con 0x7f03040090c0 2024-09-16T15:01:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 10 monclient(hunting): my global_id is 334236 2024-09-16T15:01:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0304006590 con 0x7f03040090c0 2024-09-16T15:01:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 574679305 0 0) 0x7f0304006590 con 0x7f03040090c0 2024-09-16T15:01:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0304009da0 con 0x7f03040090c0 2024-09-16T15:01:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0314005580 con 0x7f03040090c0 2024-09-16T15:01:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 10 monclient: dump: 2024-09-16T15:01:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.350 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.351 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3327933942 0 0) 0x7f0304009da0 con 0x7f03040090c0 2024-09-16T15:01:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f030400c290 con 0x7f03040090c0 2024-09-16T15:01:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 4092811383) 0x7f03140058f0 con 0x7f03040090c0 2024-09-16T15:01:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:01:07.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 >> v1:172.21.15.26:6789/0 conn(0x7f03040090c0 legacy=0x7f0304009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0319ffb640 1 -- 172.21.15.26:0/2144945879 >> v1:172.21.15.26:6789/0 conn(0x7f03040090c0 legacy=0x7f0304009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:07.352 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:01:07.352 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:07.352 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:07.352 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:01:07.352 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 168, 2024-09-16T15:01:07.352 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:01:07.352 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:01:07.353 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:01:07.353 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:01:07.353 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:07.353 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T15:01:07.353 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:07.353 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:01:07.353 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:01:07.353 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:07.353 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:07.353 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:07.353 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:07.353 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:07.353 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:07.353 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:07.354 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:07.354 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:07.354 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:07.354 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:07.354 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:01:07.354 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:01:07.354 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:07.354 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:07.354 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:07.354 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:07.354 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:07.354 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:07.354 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:07.355 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:07.355 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:07.355 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:07.355 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.355 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:01:07.355 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:01:07.355 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:01:07.355 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:01:07.355 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:01:07.355 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:01:07.355 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:07.355 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:07.356 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:01:07.356 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:01:07.356 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:01:07.356 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:01:07.356 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:01:07.356 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:01:07.356 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:01:07.356 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:07.356 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:01:07.356 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:07.356 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:07.356 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:07.356 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:07.356 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:07.357 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:07.357 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:07.357 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:07.357 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:07.357 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:07.357 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:07.357 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:01:07.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.357 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:01:07.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.357 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:07.357 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:07.357 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:07.357 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:07.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.358 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:07.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:01:07.358 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:07.358 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.358 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:01:07.358 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:01:07.358 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:07.358 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:07.358 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:07.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.359 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:07.359 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:07.359 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:07.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:07.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.359 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:07.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:01:07.359 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:07.359 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.359 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:01:07.360 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:01:07.360 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:07.360 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:07.360 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:07.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.360 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:07.360 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:07.360 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:07.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:07.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.360 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:07.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:01:07.361 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:07.361 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.361 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.361 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:01:07.361 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:01:07.361 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:07.361 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:07.361 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:07.361 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.361 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.361 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:01:07.362 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:01:07.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.362 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:07.362 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:07.362 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:07.362 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.362 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:07.362 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:01:07.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.362 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:07.362 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:07.362 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-09-16T15:01:07.362 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.362 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.363 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:01:07.363 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:01:07.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f032b67d640 1 -- 172.21.15.26:0/2144945879 >> v1:172.21.15.189:6812/3196161582 conn(0x7f03080363b0 legacy=0x7f0308038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f032b67d640 10 monclient: shutdown 2024-09-16T15:01:07.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f032b67d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:07.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f032b67d640 1 -- 172.21.15.26:0/2144945879 >> v1:172.21.15.26:6789/0 conn(0x7f03240a0000 legacy=0x7f0324132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0329c1a640 1 -- 172.21.15.26:0/2144945879 reap_dead start 2024-09-16T15:01:07.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f0329c1a640 1 -- 172.21.15.26:0/2144945879 reap_dead start 2024-09-16T15:01:07.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f032b67d640 1 -- 172.21.15.26:0/2144945879 shutdown_connections 2024-09-16T15:01:07.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f032b67d640 1 -- 172.21.15.26:0/2144945879 >> 172.21.15.26:0/2144945879 conn(0x7f03240a7f30 msgr2=0x7f032406b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:07.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f032b67d640 1 -- 172.21.15.26:0/2144945879 shutdown_connections 2024-09-16T15:01:07.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.337+0000 7f032b67d640 1 -- 172.21.15.26:0/2144945879 wait complete. 2024-09-16T15:01:07.365 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:01:07.365 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T15:01:07.365 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T15:01:07.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 10 monclient: get_monmap_and_config 2024-09-16T15:01:07.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 10 monclient: build_initial_monmap 2024-09-16T15:01:07.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 10 monclient: monmap: 2024-09-16T15:01:07.483 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:07.484 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.484 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:07.485775+0000 2024-09-16T15:01:07.484 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:07.485775+0000 2024-09-16T15:01:07.484 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:07.484 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.484 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:07.484 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:07.484 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:07.484 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 1 Processor -- start 2024-09-16T15:01:07.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 1 -- start start 2024-09-16T15:01:07.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 10 monclient: init 2024-09-16T15:01:07.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:07.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:01:07.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 10 monclient(hunting): picked mon.noname-a con 0x7ffa981325e0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:07.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 10 monclient(hunting): picked mon.noname-c con 0x7ffa981330d0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:07.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 10 monclient(hunting): picked mon.noname-b con 0x7ffa98139150 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:07.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9809e810 con 0x7ffa981325e0 2024-09-16T15:01:07.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9809ed10 con 0x7ffa981330d0 2024-09-16T15:01:07.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9809f2d0 con 0x7ffa98139150 2024-09-16T15:01:07.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:07.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 10 monclient(hunting): authenticate will time out at 2091.605830s 2024-09-16T15:01:07.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9d3bf640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ffa981325e0 0x7ffa981329c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53770/0 (socket says 172.21.15.26:53770) 2024-09-16T15:01:07.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9d3bf640 1 -- 172.21.15.26:0/690544625 learned_addr learned my addr 172.21.15.26:0/690544625 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:07.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9dbc0640 1 --1- 172.21.15.26:0/690544625 >> v1:172.21.15.26:6790/0 conn(0x7ffa98139150 0x7ffa9813b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59916/0 (socket says 172.21.15.26:59916) 2024-09-16T15:01:07.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 1 -- 172.21.15.26:0/690544625 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416839281 0 0) 0x7ffa9809f2d0 con 0x7ffa98139150 2024-09-16T15:01:07.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient(hunting): my global_id is 284759 2024-09-16T15:01:07.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 1 -- 172.21.15.26:0/690544625 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa84003530 con 0x7ffa98139150 2024-09-16T15:01:07.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 1 -- 172.21.15.26:0/690544625 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823958620 0 0) 0x7ffa9809e810 con 0x7ffa981325e0 2024-09-16T15:01:07.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient(hunting): my global_id is 334239 2024-09-16T15:01:07.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 1 -- 172.21.15.26:0/690544625 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa9809f2d0 con 0x7ffa981325e0 2024-09-16T15:01:07.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 1 -- 172.21.15.26:0/690544625 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233916179 0 0) 0x7ffa9809ed10 con 0x7ffa981330d0 2024-09-16T15:01:07.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient(hunting): my global_id is 344146 2024-09-16T15:01:07.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 1 -- 172.21.15.26:0/690544625 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa9809e810 con 0x7ffa981330d0 2024-09-16T15:01:07.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 1 -- 172.21.15.26:0/690544625 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2512690344 0 0) 0x7ffa84003530 con 0x7ffa98139150 2024-09-16T15:01:07.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 1 -- 172.21.15.26:0/690544625 >> v1:172.21.15.189:6789/0 conn(0x7ffa981330d0 legacy=0x7ffa98137a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 1 -- 172.21.15.26:0/690544625 >> v1:172.21.15.26:6789/0 conn(0x7ffa981325e0 legacy=0x7ffa981329c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient: _finish_hunting 0 2024-09-16T15:01:07.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 1 monclient: found mon.noname-b 2024-09-16T15:01:07.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:07.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:01:07.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 1 -- 172.21.15.26:0/690544625 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9809c6a0 con 0x7ffa98139150 2024-09-16T15:01:07.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient: _finish_auth 0 2024-09-16T15:01:07.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient: _check_auth_tickets 2024-09-16T15:01:07.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:07.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 1 -- 172.21.15.26:0/690544625 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffa900018d0 con 0x7ffa98139150 2024-09-16T15:01:07.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 5 monclient: authenticate success, global_id 284759 2024-09-16T15:01:07.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:07.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:01:07.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient: dump: 2024-09-16T15:01:07.496 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.496 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.496 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.496 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.497 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.497 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.497 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.497 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.497 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.497 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:07.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 1 -- 172.21.15.26:0/690544625 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa90001af0 con 0x7ffa98139150 2024-09-16T15:01:07.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:07.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 1 -- 172.21.15.26:0/690544625 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffa90001e80 con 0x7ffa98139150 2024-09-16T15:01:07.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:07.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa8ffff640 10 monclient: dump: 2024-09-16T15:01:07.497 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.498 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.498 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.498 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.498 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.498 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.498 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.498 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.498 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.498 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:07.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 4 set_mon_vals no callback set 2024-09-16T15:01:07.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 10 monclient: shutdown 2024-09-16T15:01:07.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 1 -- 172.21.15.26:0/690544625 >> v1:172.21.15.26:6790/0 conn(0x7ffa98139150 legacy=0x7ffa9813b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 1 -- 172.21.15.26:0/690544625 shutdown_connections 2024-09-16T15:01:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 1 -- 172.21.15.26:0/690544625 >> 172.21.15.26:0/690544625 conn(0x7ffa9809f900 msgr2=0x7ffa9809fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 1 -- 172.21.15.26:0/690544625 shutdown_connections 2024-09-16T15:01:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.485+0000 7ffa9f623640 1 -- 172.21.15.26:0/690544625 wait complete. 2024-09-16T15:01:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 10 monclient: build_initial_monmap 2024-09-16T15:01:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 10 monclient: monmap: 2024-09-16T15:01:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:07.500 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:07.500 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:07.500 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:07.500 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 1 Processor -- start 2024-09-16T15:01:07.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 1 -- start start 2024-09-16T15:01:07.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 10 monclient: init 2024-09-16T15:01:07.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:07.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:01:07.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 10 monclient(hunting): picked mon.noname-a con 0x7ffa981325e0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:07.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 10 monclient(hunting): picked mon.noname-b con 0x7ffa981330d0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:07.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 10 monclient(hunting): picked mon.noname-c con 0x7ffa98139150 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:07.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa98138c80 con 0x7ffa981325e0 2024-09-16T15:01:07.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9811e910 con 0x7ffa981330d0 2024-09-16T15:01:07.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa980a1ca0 con 0x7ffa98139150 2024-09-16T15:01:07.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:07.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 10 monclient(hunting): authenticate will time out at 2091.609058s 2024-09-16T15:01:07.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9dbc0640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ffa98139150 0x7ffa981d1c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59922/0 (socket says 172.21.15.26:59922) 2024-09-16T15:01:07.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9d3bf640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ffa981325e0 0x7ffa98131510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53776/0 (socket says 172.21.15.26:53776) 2024-09-16T15:01:07.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9dbc0640 1 -- 172.21.15.26:0/134701993 learned_addr learned my addr 172.21.15.26:0/134701993 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:07.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262410736 0 0) 0x7ffa980a1ca0 con 0x7ffa98139150 2024-09-16T15:01:07.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): my global_id is 284762 2024-09-16T15:01:07.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa7c003530 con 0x7ffa98139150 2024-09-16T15:01:07.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970817305 0 0) 0x7ffa98138c80 con 0x7ffa981325e0 2024-09-16T15:01:07.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): my global_id is 334242 2024-09-16T15:01:07.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa980a1ca0 con 0x7ffa981325e0 2024-09-16T15:01:07.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3611792828 0 0) 0x7ffa980a1ca0 con 0x7ffa981325e0 2024-09-16T15:01:07.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa98138c80 con 0x7ffa981325e0 2024-09-16T15:01:07.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffa940018c0 con 0x7ffa981325e0 2024-09-16T15:01:07.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:07.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): dump: 2024-09-16T15:01:07.503 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.503 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.503 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.503 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.503 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.503 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.503 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.503 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.503 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.503 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:07.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2678338963 0 0) 0x7ffa7c003530 con 0x7ffa98139150 2024-09-16T15:01:07.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa980a1ca0 con 0x7ffa98139150 2024-09-16T15:01:07.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffa90001e00 con 0x7ffa98139150 2024-09-16T15:01:07.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:07.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): dump: 2024-09-16T15:01:07.504 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.504 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.504 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.504 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.504 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.504 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.504 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.504 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.505 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.505 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196378611 0 0) 0x7ffa9811e910 con 0x7ffa981330d0 2024-09-16T15:01:07.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): my global_id is 344149 2024-09-16T15:01:07.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa7c003530 con 0x7ffa981330d0 2024-09-16T15:01:07.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2016034610 0 0) 0x7ffa98138c80 con 0x7ffa981325e0 2024-09-16T15:01:07.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 >> v1:172.21.15.26:6790/0 conn(0x7ffa98139150 legacy=0x7ffa981d1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 >> v1:172.21.15.189:6789/0 conn(0x7ffa981330d0 legacy=0x7ffa981ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient: _finish_hunting 0 2024-09-16T15:01:07.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 monclient: found mon.a 2024-09-16T15:01:07.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:07.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:07.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9811eed0 con 0x7ffa981325e0 2024-09-16T15:01:07.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient: _finish_auth 0 2024-09-16T15:01:07.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:07.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:07.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 5 monclient: authenticate success, global_id 334242 2024-09-16T15:01:07.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 10 monclient: _renew_subs 2024-09-16T15:01:07.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:07.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 1 -- 172.21.15.26:0/134701993 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa9811e630 con 0x7ffa981325e0 2024-09-16T15:01:07.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 10 monclient: _renew_subs 2024-09-16T15:01:07.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:07.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 1 -- 172.21.15.26:0/134701993 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9809b100 con 0x7ffa981325e0 2024-09-16T15:01:07.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa940047f0 con 0x7ffa981325e0 2024-09-16T15:01:07.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:07.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffa94004b00 con 0x7ffa981325e0 2024-09-16T15:01:07.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:07.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8f7fe640 4 set_mon_vals no callback set 2024-09-16T15:01:07.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient: dump: 2024-09-16T15:01:07.507 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.507 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.508 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.508 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.508 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.508 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.508 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.508 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.508 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.508 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:07.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9f623640 1 -- 172.21.15.26:0/134701993 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ffa78006590 con 0x7ffa78002830 2024-09-16T15:01:07.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ffa9400f570 con 0x7ffa981325e0 2024-09-16T15:01:07.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7ffa94043d30 con 0x7ffa981325e0 2024-09-16T15:01:07.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239807653 0 0) 0x7ffa78006590 con 0x7ffa78002830 2024-09-16T15:01:07.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient(hunting): my global_id is 284765 2024-09-16T15:01:07.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa98138c80 con 0x7ffa78002830 2024-09-16T15:01:07.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1541362537 0 0) 0x7ffa98138c80 con 0x7ffa78002830 2024-09-16T15:01:07.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa78006590 con 0x7ffa78002830 2024-09-16T15:01:07.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffa88001680 con 0x7ffa78002830 2024-09-16T15:01:07.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:07.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 10 monclient: dump: 2024-09-16T15:01:07.509 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.509 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.510 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.510 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.510 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.510 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.510 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.510 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.510 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.510 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa9dbc0640 10 monclient: get_auth_request con 0x7ffa7c0363b0 auth_method 0 2024-09-16T15:01:07.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 83724700 0 0) 0x7ffa78006590 con 0x7ffa78002830 2024-09-16T15:01:07.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.489+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa7800a0e0 con 0x7ffa78002830 2024-09-16T15:01:07.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.493+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ffa88001ad0 con 0x7ffa78002830 2024-09-16T15:01:07.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.493+0000 7ffa8dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:07.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.493+0000 7ffa8dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:07.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.493+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 >> v1:172.21.15.26:6790/0 conn(0x7ffa78002830 legacy=0x7ffa78004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.493+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 >> v1:172.21.15.26:6790/0 conn(0x7ffa78002830 legacy=0x7ffa78004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:07.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa9f623640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:01:07.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa9f623640 1 -- 172.21.15.26:0/134701993 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ffa78009d80 con 0x7ffa780090a0 2024-09-16T15:01:07.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397353847 0 0) 0x7ffa78009d80 con 0x7ffa780090a0 2024-09-16T15:01:07.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 10 monclient(hunting): my global_id is 284768 2024-09-16T15:01:07.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa78006590 con 0x7ffa780090a0 2024-09-16T15:01:07.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3854511240 0 0) 0x7ffa78006590 con 0x7ffa780090a0 2024-09-16T15:01:07.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa78009d80 con 0x7ffa780090a0 2024-09-16T15:01:07.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ffa88003840 con 0x7ffa780090a0 2024-09-16T15:01:07.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:07.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 10 monclient: dump: 2024-09-16T15:01:07.513 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.513 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.514 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.514 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.514 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.514 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.514 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.514 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.514 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.514 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 777485447 0 0) 0x7ffa78009d80 con 0x7ffa780090a0 2024-09-16T15:01:07.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ffa7800c290 con 0x7ffa780090a0 2024-09-16T15:01:07.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 2956360923) 0x7ffa88005b50 con 0x7ffa780090a0 2024-09-16T15:01:07.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:01:07.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:07.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 >> v1:172.21.15.26:6790/0 conn(0x7ffa780090a0 legacy=0x7ffa78009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa8dffb640 1 -- 172.21.15.26:0/134701993 >> v1:172.21.15.26:6790/0 conn(0x7ffa780090a0 legacy=0x7ffa78009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:07.515 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:01:07.515 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:07.515 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:07.515 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T15:01:07.515 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 168, 2024-09-16T15:01:07.515 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:01:07.515 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:01:07.515 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:01:07.516 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:01:07.516 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:07.516 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T15:01:07.516 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:07.516 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:01:07.516 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:01:07.516 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:07.516 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:07.516 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:07.516 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:07.516 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:07.516 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:07.516 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:07.517 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:07.517 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:07.517 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:07.517 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:07.517 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:01:07.517 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:01:07.517 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:07.517 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:07.517 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:07.517 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:07.517 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:07.517 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:07.517 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:07.517 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:07.518 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:07.518 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:07.518 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.518 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:01:07.518 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:01:07.518 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:01:07.518 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:01:07.518 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:01:07.518 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:01:07.518 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:07.518 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:07.518 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:01:07.519 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:01:07.519 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:01:07.519 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:01:07.519 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:01:07.519 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:01:07.519 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:01:07.519 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:07.519 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:01:07.519 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:07.519 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:07.519 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:07.519 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:07.519 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:07.520 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:07.520 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:07.520 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:07.520 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:07.520 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:07.520 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:07.520 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:01:07.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.520 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:01:07.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.520 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:07.521 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:07.521 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:07.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:07.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.521 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:07.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:01:07.521 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:07.521 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.521 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.522 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:01:07.522 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:01:07.522 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:07.522 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:07.522 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:07.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.522 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:07.522 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:07.522 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:07.522 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:07.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.522 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:07.523 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:01:07.523 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:07.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.523 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.523 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:01:07.523 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:01:07.523 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:07.523 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:07.523 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:07.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.523 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:07.523 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:07.524 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:07.524 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:07.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.524 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:07.524 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:01:07.524 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:07.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.524 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.524 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:01:07.524 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:01:07.524 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:07.524 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:01:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:01:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:07.525 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T15:01:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T15:01:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:01:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T15:01:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.526 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:07.527 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T15:01:07.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.527 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:07.527 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:07.527 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:07.527 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.527 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.527 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:01:07.527 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:01:07.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa9f623640 1 -- 172.21.15.26:0/134701993 >> v1:172.21.15.189:6812/3196161582 conn(0x7ffa7c0363b0 legacy=0x7ffa7c038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa9f623640 10 monclient: shutdown 2024-09-16T15:01:07.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa9f623640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:07.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa9f623640 1 -- 172.21.15.26:0/134701993 >> v1:172.21.15.26:6789/0 conn(0x7ffa981325e0 legacy=0x7ffa98131510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa9dbc0640 1 -- 172.21.15.26:0/134701993 reap_dead start 2024-09-16T15:01:07.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa9dbc0640 1 -- 172.21.15.26:0/134701993 reap_dead start 2024-09-16T15:01:07.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa9f623640 1 -- 172.21.15.26:0/134701993 shutdown_connections 2024-09-16T15:01:07.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa9f623640 1 -- 172.21.15.26:0/134701993 >> 172.21.15.26:0/134701993 conn(0x7ffa9809f900 msgr2=0x7ffa980704a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:07.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa9f623640 1 -- 172.21.15.26:0/134701993 shutdown_connections 2024-09-16T15:01:07.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.497+0000 7ffa9f623640 1 -- 172.21.15.26:0/134701993 wait complete. 2024-09-16T15:01:07.530 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:01:07.530 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T15:01:07.530 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T15:01:07.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 10 monclient: get_monmap_and_config 2024-09-16T15:01:07.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 10 monclient: build_initial_monmap 2024-09-16T15:01:07.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 10 monclient: monmap: 2024-09-16T15:01:07.648 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:07.648 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.648 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:07.650299+0000 2024-09-16T15:01:07.648 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:07.650299+0000 2024-09-16T15:01:07.649 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:07.649 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.649 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:07.649 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:07.649 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:07.649 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 1 Processor -- start 2024-09-16T15:01:07.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 1 -- start start 2024-09-16T15:01:07.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 10 monclient: init 2024-09-16T15:01:07.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:07.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:01:07.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 10 monclient(hunting): picked mon.noname-c con 0x7f2b7809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:07.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 10 monclient(hunting): picked mon.noname-a con 0x7f2b78132650 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:07.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 10 monclient(hunting): picked mon.noname-b con 0x7f2b78138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:07.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b7809d260 con 0x7f2b78132650 2024-09-16T15:01:07.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b78132a80 con 0x7f2b7809cb80 2024-09-16T15:01:07.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b7809cf60 con 0x7f2b78138ff0 2024-09-16T15:01:07.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:07.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 10 monclient(hunting): authenticate will time out at 2091.770096s 2024-09-16T15:01:07.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b77fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2b78132650 0x7f2b781378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53782/0 (socket says 172.21.15.26:53782) 2024-09-16T15:01:07.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b77fff640 1 -- 172.21.15.26:0/1799261410 learned_addr learned my addr 172.21.15.26:0/1799261410 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:07.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 1 -- 172.21.15.26:0/1799261410 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198039387 0 0) 0x7f2b7809cf60 con 0x7f2b78138ff0 2024-09-16T15:01:07.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient(hunting): my global_id is 284771 2024-09-16T15:01:07.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 1 -- 172.21.15.26:0/1799261410 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b60003530 con 0x7f2b78138ff0 2024-09-16T15:01:07.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 1 -- 172.21.15.26:0/1799261410 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171866868 0 0) 0x7f2b7809d260 con 0x7f2b78132650 2024-09-16T15:01:07.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient(hunting): my global_id is 334245 2024-09-16T15:01:07.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 1 -- 172.21.15.26:0/1799261410 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b7809cf60 con 0x7f2b78132650 2024-09-16T15:01:07.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 1 -- 172.21.15.26:0/1799261410 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2138315675 0 0) 0x7f2b60003530 con 0x7f2b78138ff0 2024-09-16T15:01:07.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 1 -- 172.21.15.26:0/1799261410 >> v1:172.21.15.189:6789/0 conn(0x7f2b7809cb80 legacy=0x7f2b780717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 1 -- 172.21.15.26:0/1799261410 >> v1:172.21.15.26:6789/0 conn(0x7f2b78132650 legacy=0x7f2b781378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient: _finish_hunting 0 2024-09-16T15:01:07.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 1 monclient: found mon.noname-b 2024-09-16T15:01:07.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:07.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:01:07.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 1 -- 172.21.15.26:0/1799261410 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7809f2d0 con 0x7f2b78138ff0 2024-09-16T15:01:07.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient: _finish_auth 0 2024-09-16T15:01:07.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient: _check_auth_tickets 2024-09-16T15:01:07.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:07.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 5 monclient: authenticate success, global_id 284771 2024-09-16T15:01:07.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 1 -- 172.21.15.26:0/1799261410 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b700019e0 con 0x7f2b78138ff0 2024-09-16T15:01:07.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:01:07.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient: dump: 2024-09-16T15:01:07.661 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.661 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.661 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.661 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.661 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.661 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.662 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.662 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.662 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.662 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:07.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 1 -- 172.21.15.26:0/1799261410 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b70001bc0 con 0x7f2b78138ff0 2024-09-16T15:01:07.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:07.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:07.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 4 set_mon_vals no callback set 2024-09-16T15:01:07.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 1 -- 172.21.15.26:0/1799261410 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b70001f50 con 0x7f2b78138ff0 2024-09-16T15:01:07.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:07.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b777fe640 10 monclient: dump: 2024-09-16T15:01:07.663 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.663 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.663 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.663 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.663 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.663 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.663 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.663 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.663 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.663 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 10 monclient: shutdown 2024-09-16T15:01:07.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:07.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 1 -- 172.21.15.26:0/1799261410 >> v1:172.21.15.26:6790/0 conn(0x7f2b78138ff0 legacy=0x7f2b7813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 1 -- 172.21.15.26:0/1799261410 shutdown_connections 2024-09-16T15:01:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 1 -- 172.21.15.26:0/1799261410 >> 172.21.15.26:0/1799261410 conn(0x7f2b7809fb50 msgr2=0x7f2b7809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 1 -- 172.21.15.26:0/1799261410 shutdown_connections 2024-09-16T15:01:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.649+0000 7f2b7f25e640 1 -- 172.21.15.26:0/1799261410 wait complete. 2024-09-16T15:01:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 10 monclient: build_initial_monmap 2024-09-16T15:01:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 10 monclient: monmap: 2024-09-16T15:01:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:07.664 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:07.665 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:07.665 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:07.665 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 1 Processor -- start 2024-09-16T15:01:07.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 1 -- start start 2024-09-16T15:01:07.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 10 monclient: init 2024-09-16T15:01:07.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:07.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:01:07.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 10 monclient(hunting): picked mon.noname-b con 0x7f2b7809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:07.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 10 monclient(hunting): picked mon.noname-a con 0x7f2b78132650 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:07.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 10 monclient(hunting): picked mon.noname-c con 0x7f2b78138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:07.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b78070130 con 0x7f2b78132650 2024-09-16T15:01:07.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b78131e60 con 0x7f2b7809cb80 2024-09-16T15:01:07.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b78130330 con 0x7f2b78138ff0 2024-09-16T15:01:07.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:07.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 10 monclient(hunting): authenticate will time out at 2091.773388s 2024-09-16T15:01:07.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b77fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2b78132650 0x7f2b7806e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53794/0 (socket says 172.21.15.26:53794) 2024-09-16T15:01:07.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b77fff640 1 -- 172.21.15.26:0/2468154104 learned_addr learned my addr 172.21.15.26:0/2468154104 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:07.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393693209 0 0) 0x7f2b78130330 con 0x7f2b78138ff0 2024-09-16T15:01:07.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): my global_id is 284774 2024-09-16T15:01:07.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b64003530 con 0x7f2b78138ff0 2024-09-16T15:01:07.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467693834 0 0) 0x7f2b78070130 con 0x7f2b78132650 2024-09-16T15:01:07.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): my global_id is 334248 2024-09-16T15:01:07.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b78130330 con 0x7f2b78132650 2024-09-16T15:01:07.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2169471897 0 0) 0x7f2b78130330 con 0x7f2b78132650 2024-09-16T15:01:07.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b78070130 con 0x7f2b78132650 2024-09-16T15:01:07.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 112123475 0 0) 0x7f2b64003530 con 0x7f2b78138ff0 2024-09-16T15:01:07.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b78130330 con 0x7f2b78138ff0 2024-09-16T15:01:07.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348552751 0 0) 0x7f2b78131e60 con 0x7f2b7809cb80 2024-09-16T15:01:07.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): my global_id is 344155 2024-09-16T15:01:07.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b64003530 con 0x7f2b7809cb80 2024-09-16T15:01:07.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b6c0018e0 con 0x7f2b78132650 2024-09-16T15:01:07.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:07.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): dump: 2024-09-16T15:01:07.668 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.668 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.669 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.669 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.669 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.669 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.669 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.669 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.669 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.669 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:07.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b70001d30 con 0x7f2b78138ff0 2024-09-16T15:01:07.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:07.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): dump: 2024-09-16T15:01:07.670 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.670 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.670 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.670 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.670 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.670 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.670 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.670 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.670 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.670 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1209588939 0 0) 0x7f2b78070130 con 0x7f2b78132650 2024-09-16T15:01:07.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 >> v1:172.21.15.26:6790/0 conn(0x7f2b78138ff0 legacy=0x7f2b7806e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 >> v1:172.21.15.189:6789/0 conn(0x7f2b7809cb80 legacy=0x7f2b7806da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient: _finish_hunting 0 2024-09-16T15:01:07.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 monclient: found mon.a 2024-09-16T15:01:07.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:07.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:07.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b78070c50 con 0x7f2b78132650 2024-09-16T15:01:07.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient: _finish_auth 0 2024-09-16T15:01:07.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient: _check_auth_tickets 2024-09-16T15:01:07.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:07.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 5 monclient: authenticate success, global_id 334248 2024-09-16T15:01:07.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 10 monclient: _renew_subs 2024-09-16T15:01:07.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:07.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 1 -- 172.21.15.26:0/2468154104 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b78071300 con 0x7f2b78132650 2024-09-16T15:01:07.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 10 monclient: _renew_subs 2024-09-16T15:01:07.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:07.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 1 -- 172.21.15.26:0/2468154104 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7806f560 con 0x7f2b78132650 2024-09-16T15:01:07.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b6c0047f0 con 0x7f2b78132650 2024-09-16T15:01:07.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:07.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b6c004b80 con 0x7f2b78132650 2024-09-16T15:01:07.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:07.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b76ffd640 4 set_mon_vals no callback set 2024-09-16T15:01:07.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient: dump: 2024-09-16T15:01:07.672 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.672 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.672 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.672 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.673 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.673 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.673 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.673 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.673 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.673 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:07.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7f25e640 1 -- 172.21.15.26:0/2468154104 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2b58006590 con 0x7f2b58002830 2024-09-16T15:01:07.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2b6c004d60 con 0x7f2b78132650 2024-09-16T15:01:07.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f2b6c043e00 con 0x7f2b78132650 2024-09-16T15:01:07.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b7d7fb640 10 monclient: get_auth_request con 0x7f2b64036610 auth_method 0 2024-09-16T15:01:07.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866034307 0 0) 0x7f2b58006590 con 0x7f2b58002830 2024-09-16T15:01:07.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 10 monclient(hunting): my global_id is 344158 2024-09-16T15:01:07.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.653+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b78070130 con 0x7f2b58002830 2024-09-16T15:01:07.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.657+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3500308654 0 0) 0x7f2b78070130 con 0x7f2b58002830 2024-09-16T15:01:07.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.657+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b58006590 con 0x7f2b58002830 2024-09-16T15:01:07.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.657+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b68001720 con 0x7f2b58002830 2024-09-16T15:01:07.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.657+0000 7f2b757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.657+0000 7f2b757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:07.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.657+0000 7f2b757fa640 10 monclient: dump: 2024-09-16T15:01:07.675 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.675 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.675 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.675 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.675 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.675 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.675 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.675 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.675 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.675 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.657+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4054742363 0 0) 0x7f2b58006590 con 0x7f2b58002830 2024-09-16T15:01:07.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.657+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2b5800a0e0 con 0x7f2b58002830 2024-09-16T15:01:07.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.657+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2b68001c10 con 0x7f2b58002830 2024-09-16T15:01:07.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.657+0000 7f2b757fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:07.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.657+0000 7f2b757fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:07.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.657+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 >> v1:172.21.15.189:6789/0 conn(0x7f2b58002830 legacy=0x7f2b58004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.657+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 >> v1:172.21.15.189:6789/0 conn(0x7f2b58002830 legacy=0x7f2b58004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:07.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b7f25e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:01:07.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b7f25e640 1 -- 172.21.15.26:0/2468154104 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2b58009da0 con 0x7f2b580090c0 2024-09-16T15:01:07.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169061741 0 0) 0x7f2b58009da0 con 0x7f2b580090c0 2024-09-16T15:01:07.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 10 monclient(hunting): my global_id is 344161 2024-09-16T15:01:07.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b58006590 con 0x7f2b580090c0 2024-09-16T15:01:07.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2473624368 0 0) 0x7f2b58006590 con 0x7f2b580090c0 2024-09-16T15:01:07.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b58009da0 con 0x7f2b580090c0 2024-09-16T15:01:07.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2b680039f0 con 0x7f2b580090c0 2024-09-16T15:01:07.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:07.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 10 monclient: dump: 2024-09-16T15:01:07.679 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.679 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.679 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.679 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.679 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.679 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.679 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.679 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.680 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.680 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2888053808 0 0) 0x7f2b58009da0 con 0x7f2b580090c0 2024-09-16T15:01:07.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2b5800c290 con 0x7f2b580090c0 2024-09-16T15:01:07.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 1043796307) 0x7f2b68003810 con 0x7f2b580090c0 2024-09-16T15:01:07.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:01:07.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:07.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 >> v1:172.21.15.189:6789/0 conn(0x7f2b580090c0 legacy=0x7f2b58009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b757fa640 1 -- 172.21.15.26:0/2468154104 >> v1:172.21.15.189:6789/0 conn(0x7f2b580090c0 legacy=0x7f2b58009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:07.681 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:01:07.681 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:07.681 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:07.681 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:01:07.681 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 168, 2024-09-16T15:01:07.681 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:01:07.681 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:01:07.681 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:01:07.682 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:01:07.682 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:07.682 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-09-16T15:01:07.682 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:07.682 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:01:07.682 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:01:07.682 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:07.682 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:07.682 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:07.682 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:07.682 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:07.682 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:07.683 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:07.683 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:07.683 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:07.683 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:07.683 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:07.683 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:01:07.683 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:01:07.683 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:07.683 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:07.684 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:07.684 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:07.684 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:07.684 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:07.684 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:07.684 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:07.684 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:07.684 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:07.684 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.685 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:01:07.685 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:01:07.685 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:01:07.685 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:01:07.685 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:01:07.685 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:01:07.685 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:07.685 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:07.685 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:01:07.685 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:01:07.685 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:01:07.686 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:01:07.686 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:01:07.686 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:01:07.686 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:01:07.686 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:07.686 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:01:07.686 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:07.686 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:07.686 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:07.686 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:07.686 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:07.686 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:07.686 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:07.687 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:07.687 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:07.687 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:07.687 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:07.687 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:01:07.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.687 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:01:07.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.687 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:07.687 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:07.687 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:07.688 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:07.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.688 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:07.688 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:01:07.688 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:07.688 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.688 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.688 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:01:07.688 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:01:07.688 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:07.688 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:07.688 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:07.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.689 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:07.689 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:07.689 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:07.689 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:07.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.689 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:07.689 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:01:07.689 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:07.690 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.690 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.690 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:01:07.690 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:01:07.690 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:07.690 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:07.690 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:07.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.690 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:07.690 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:07.690 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:07.691 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:07.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.691 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:07.691 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:01:07.691 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:07.691 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.691 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.691 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:01:07.691 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:01:07.691 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:07.691 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:07.691 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:07.691 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.692 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.692 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:01:07.692 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:01:07.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.692 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:07.692 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:07.692 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:07.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.692 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:07.692 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:01:07.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:07.693 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:07.693 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:07.693 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:07.693 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:07.693 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:07.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:07.693 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:01:07.693 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:01:07.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b7f25e640 1 -- 172.21.15.26:0/2468154104 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2b64036610 legacy=0x7f2b64038ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b7f25e640 10 monclient: shutdown 2024-09-16T15:01:07.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b7f25e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:07.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b7f25e640 1 -- 172.21.15.26:0/2468154104 >> v1:172.21.15.26:6789/0 conn(0x7f2b78132650 legacy=0x7f2b7806e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b7d7fb640 1 -- 172.21.15.26:0/2468154104 reap_dead start 2024-09-16T15:01:07.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b7d7fb640 1 -- 172.21.15.26:0/2468154104 reap_dead start 2024-09-16T15:01:07.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b7f25e640 1 -- 172.21.15.26:0/2468154104 shutdown_connections 2024-09-16T15:01:07.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b7f25e640 1 -- 172.21.15.26:0/2468154104 >> 172.21.15.26:0/2468154104 conn(0x7f2b7809fb50 msgr2=0x7f2b78131610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:07.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b7f25e640 1 -- 172.21.15.26:0/2468154104 shutdown_connections 2024-09-16T15:01:07.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.665+0000 7f2b7f25e640 1 -- 172.21.15.26:0/2468154104 wait complete. 2024-09-16T15:01:07.695 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:01:07.696 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T15:01:07.696 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T15:01:07.696 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T15:01:07.696 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T15:01:07.696 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T15:01:07.696 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:01:07.714 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T15:01:07.714 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T15:01:07.714 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T15:01:07.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535f0cb640 10 monclient: get_monmap_and_config 2024-09-16T15:01:07.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535f0cb640 10 monclient: build_initial_monmap 2024-09-16T15:01:07.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535f0cb640 10 monclient: monmap: 2024-09-16T15:01:07.881 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:07.881 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.881 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:07.882868+0000 2024-09-16T15:01:07.882 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:07.882868+0000 2024-09-16T15:01:07.882 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:07.882 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.882 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:07.882 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:07.882 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:07.882 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535f0cb640 1 Processor -- start 2024-09-16T15:01:07.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535f0cb640 1 -- start start 2024-09-16T15:01:07.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535f0cb640 10 monclient: init 2024-09-16T15:01:07.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535f0cb640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:07.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535f0cb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:01:07.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535f0cb640 10 monclient(hunting): picked mon.noname-a con 0x7f535809a550 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:07.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535f0cb640 10 monclient(hunting): picked mon.noname-c con 0x7f535809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:07.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535f0cb640 10 monclient(hunting): picked mon.noname-b con 0x7f535809b040 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:07.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535f0cb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535809dc60 con 0x7f535809a550 2024-09-16T15:01:07.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535f0cb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5358099ae0 con 0x7f535809d1b0 2024-09-16T15:01:07.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535f0cb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535809b8f0 con 0x7f535809b040 2024-09-16T15:01:07.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535f0cb640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:07.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535f0cb640 10 monclient(hunting): authenticate will time out at 2092.003448s 2024-09-16T15:01:07.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535d668640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f535809b040 legacy=0x7f535813a190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:07.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535ce67640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f535809a550 0x7f535809a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53826/0 (socket says 172.21.15.26:53826) 2024-09-16T15:01:07.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f535ce67640 1 -- 172.21.15.26:0/3812943141 learned_addr learned my addr 172.21.15.26:0/3812943141 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:07.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f53577fe640 1 -- 172.21.15.26:0/3812943141 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886695443 0 0) 0x7f535809dc60 con 0x7f535809a550 2024-09-16T15:01:07.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f53577fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f53577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f53577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f53577fe640 10 monclient(hunting): my global_id is 334251 2024-09-16T15:01:07.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.881+0000 7f53577fe640 1 -- 172.21.15.26:0/3812943141 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5344005740 con 0x7f535809a550 2024-09-16T15:01:07.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 1 -- 172.21.15.26:0/3812943141 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2557797995 0 0) 0x7f5344005740 con 0x7f535809a550 2024-09-16T15:01:07.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 1 -- 172.21.15.26:0/3812943141 >> v1:172.21.15.26:6790/0 conn(0x7f535809b040 legacy=0x7f535813a190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:01:07.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 1 -- 172.21.15.26:0/3812943141 >> v1:172.21.15.189:6789/0 conn(0x7f535809d1b0 legacy=0x7f5358136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 10 monclient: _finish_hunting 0 2024-09-16T15:01:07.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 1 monclient: found mon.noname-a 2024-09-16T15:01:07.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:07.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:01:07.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 1 -- 172.21.15.26:0/3812943141 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535809bbd0 con 0x7f535809a550 2024-09-16T15:01:07.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 10 monclient: _finish_auth 0 2024-09-16T15:01:07.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 10 monclient: _check_auth_tickets 2024-09-16T15:01:07.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:07.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 1 -- 172.21.15.26:0/3812943141 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5348001620 con 0x7f535809a550 2024-09-16T15:01:07.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 5 monclient: authenticate success, global_id 334251 2024-09-16T15:01:07.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:07.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:07.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 10 monclient: dump: 2024-09-16T15:01:07.902 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.902 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.902 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.902 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.903 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.903 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.903 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.903 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.903 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.903 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:07.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 1 -- 172.21.15.26:0/3812943141 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5348001940 con 0x7f535809a550 2024-09-16T15:01:07.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:07.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 1 -- 172.21.15.26:0/3812943141 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5348001cd0 con 0x7f535809a550 2024-09-16T15:01:07.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:07.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 4 set_mon_vals no callback set 2024-09-16T15:01:07.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:07.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53577fe640 10 monclient: dump: 2024-09-16T15:01:07.905 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.906 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.906 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.906 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.906 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.906 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.907 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.907 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.907 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.907 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 10 monclient: shutdown 2024-09-16T15:01:07.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:07.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 1 -- 172.21.15.26:0/3812943141 >> v1:172.21.15.26:6789/0 conn(0x7f535809a550 legacy=0x7f535809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 1 -- 172.21.15.26:0/3812943141 shutdown_connections 2024-09-16T15:01:07.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 1 -- 172.21.15.26:0/3812943141 >> 172.21.15.26:0/3812943141 conn(0x7f535809fb50 msgr2=0x7f535809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:07.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 1 -- 172.21.15.26:0/3812943141 shutdown_connections 2024-09-16T15:01:07.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 1 -- 172.21.15.26:0/3812943141 wait complete. 2024-09-16T15:01:07.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 10 monclient: build_initial_monmap 2024-09-16T15:01:07.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 10 monclient: monmap: 2024-09-16T15:01:07.909 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:07.909 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:07.910 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:07.910 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:07.910 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:07.910 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:07.910 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:07.911 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:07.911 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:07.911 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 1 Processor -- start 2024-09-16T15:01:07.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 1 -- start start 2024-09-16T15:01:07.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 10 monclient: init 2024-09-16T15:01:07.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:07.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:01:07.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 10 monclient(hunting): picked mon.noname-a con 0x7f535809a550 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:07.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 10 monclient(hunting): picked mon.noname-b con 0x7f535809b040 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:07.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 10 monclient(hunting): picked mon.noname-c con 0x7f535809d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:07.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53581a3100 con 0x7f535809a550 2024-09-16T15:01:07.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535809a240 con 0x7f535809b040 2024-09-16T15:01:07.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535819ffa0 con 0x7f535809d1b0 2024-09-16T15:01:07.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 10 monclient(hunting): authenticate will time out at 2092.006585s 2024-09-16T15:01:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535d668640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f535809d1b0 legacy=0x7f53581d5bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535ce67640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f535809a550 0x7f535819f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53836/0 (socket says 172.21.15.26:53836) 2024-09-16T15:01:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535ce67640 1 -- 172.21.15.26:0/1024267788 learned_addr learned my addr 172.21.15.26:0/1024267788 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294842099 0 0) 0x7f53581a3100 con 0x7f535809a550 2024-09-16T15:01:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient(hunting): my global_id is 334254 2024-09-16T15:01:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5338003530 con 0x7f535809a550 2024-09-16T15:01:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 31294683 0 0) 0x7f5338003530 con 0x7f535809a550 2024-09-16T15:01:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53581a3100 con 0x7f535809a550 2024-09-16T15:01:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f53480015e0 con 0x7f535809a550 2024-09-16T15:01:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient(hunting): dump: 2024-09-16T15:01:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.916 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.917 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:07.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158213982 0 0) 0x7f535809a240 con 0x7f535809b040 2024-09-16T15:01:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient(hunting): _negotiate 2024-09-16T15:01:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient(hunting): my global_id is 344167 2024-09-16T15:01:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5338003530 con 0x7f535809b040 2024-09-16T15:01:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1762258777 0 0) 0x7f53581a3100 con 0x7f535809a550 2024-09-16T15:01:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 >> v1:172.21.15.26:6790/0 conn(0x7f535809d1b0 legacy=0x7f53581d5bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:01:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 >> v1:172.21.15.189:6789/0 conn(0x7f535809b040 legacy=0x7f535819f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient: _finish_hunting 0 2024-09-16T15:01:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 1 monclient: found mon.a 2024-09-16T15:01:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:07.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53580a2580 con 0x7f535809a550 2024-09-16T15:01:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient: _finish_auth 0 2024-09-16T15:01:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient: _check_auth_tickets 2024-09-16T15:01:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 5 monclient: authenticate success, global_id 334254 2024-09-16T15:01:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 10 monclient: _renew_subs 2024-09-16T15:01:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 1 -- 172.21.15.26:0/1024267788 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53581a1680 con 0x7f535809a550 2024-09-16T15:01:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 10 monclient: _renew_subs 2024-09-16T15:01:07.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f535f0cb640 1 -- 172.21.15.26:0/1024267788 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53581a0d30 con 0x7f535809a550 2024-09-16T15:01:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5348001190 con 0x7f535809a550 2024-09-16T15:01:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5348004da0 con 0x7f535809a550 2024-09-16T15:01:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53557fa640 10 monclient: dump: 2024-09-16T15:01:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:07.920 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:07.921 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.885+0000 7f53567fc640 4 set_mon_vals no callback set 2024-09-16T15:01:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.889+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f534800f810 con 0x7f535809a550 2024-09-16T15:01:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.889+0000 7f535f0cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.889+0000 7f535f0cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.889+0000 7f535f0cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.889+0000 7f535f0cb640 1 -- 172.21.15.26:0/1024267788 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f533c005050 con 0x7f535809a550 2024-09-16T15:01:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.889+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f5348043f30 con 0x7f535809a550 2024-09-16T15:01:07.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.889+0000 7f5357fff640 10 monclient: get_auth_request con 0x7f5338036180 auth_method 0 2024-09-16T15:01:07.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.893+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5348014720 con 0x7f535809a550 2024-09-16T15:01:07.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.893+0000 7f53557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:07.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:07.893+0000 7f53557fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:08.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:08.117+0000 7f535f0cb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T15:01:08.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:08.117+0000 7f535f0cb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:08.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:08.117+0000 7f535f0cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:08.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:08.117+0000 7f535f0cb640 1 -- 172.21.15.26:0/1024267788 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f533c005880 con 0x7f535809a550 2024-09-16T15:01:08.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:08.117+0000 7f53557fa640 1 -- 172.21.15.26:0/1024267788 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 407768311) 0x7f5348012dc0 con 0x7f535809a550 2024-09-16T15:01:08.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:08.117+0000 7f53557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:08.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:08.117+0000 7f53557fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:08.221 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":168,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T15:01:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:08.117+0000 7f535f0cb640 1 -- 172.21.15.26:0/1024267788 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5338036180 legacy=0x7f5338038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:08.117+0000 7f535f0cb640 10 monclient: shutdown 2024-09-16T15:01:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:08.117+0000 7f535f0cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:08.117+0000 7f535f0cb640 1 -- 172.21.15.26:0/1024267788 >> v1:172.21.15.26:6789/0 conn(0x7f535809a550 legacy=0x7f535819f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:08.117+0000 7f535f0cb640 1 -- 172.21.15.26:0/1024267788 shutdown_connections 2024-09-16T15:01:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:08.117+0000 7f535f0cb640 1 -- 172.21.15.26:0/1024267788 >> 172.21.15.26:0/1024267788 conn(0x7f535809fb50 msgr2=0x7f53581b7740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:08.117+0000 7f535f0cb640 1 -- 172.21.15.26:0/1024267788 shutdown_connections 2024-09-16T15:01:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:08.117+0000 7f535f0cb640 1 -- 172.21.15.26:0/1024267788 wait complete. 2024-09-16T15:01:11.224 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T15:01:11.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f182d6f9640 10 monclient: get_monmap_and_config 2024-09-16T15:01:11.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f182d6f9640 10 monclient: build_initial_monmap 2024-09-16T15:01:11.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f182d6f9640 10 monclient: monmap: 2024-09-16T15:01:11.357 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:11.359293+0000 2024-09-16T15:01:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:11.359293+0000 2024-09-16T15:01:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:11.358 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:11.358 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:11.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f182d6f9640 1 Processor -- start 2024-09-16T15:01:11.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f182d6f9640 1 -- start start 2024-09-16T15:01:11.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f182d6f9640 10 monclient: init 2024-09-16T15:01:11.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f182d6f9640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:11.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f182d6f9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:01:11.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f182d6f9640 10 monclient(hunting): picked mon.noname-b con 0x7f18280a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:11.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f182d6f9640 10 monclient(hunting): picked mon.noname-c con 0x7f18280a0000 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:11.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f182d6f9640 10 monclient(hunting): picked mon.noname-a con 0x7f18280a2800 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:11.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f182d6f9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18280a3250 con 0x7f18280a2800 2024-09-16T15:01:11.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f182d6f9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f182809f600 con 0x7f18280a0000 2024-09-16T15:01:11.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f182d6f9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18280a3830 con 0x7f18280a1d10 2024-09-16T15:01:11.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f182d6f9640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:11.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f182d6f9640 10 monclient(hunting): authenticate will time out at 2095.479363s 2024-09-16T15:01:11.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f1826ffd640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f18280a1d10 legacy=0x7f18280a20f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:11.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f18277fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f18280a2800 0x7f182813d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53838/0 (socket says 172.21.15.26:53838) 2024-09-16T15:01:11.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f18277fe640 1 -- 172.21.15.26:0/3438180645 learned_addr learned my addr 172.21.15.26:0/3438180645 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:11.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f1825ffb640 1 -- 172.21.15.26:0/3438180645 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249683677 0 0) 0x7f18280a3250 con 0x7f18280a2800 2024-09-16T15:01:11.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f1825ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:11.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f1825ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:11.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f1825ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:11.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f1825ffb640 10 monclient(hunting): my global_id is 334257 2024-09-16T15:01:11.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.357+0000 7f1825ffb640 1 -- 172.21.15.26:0/3438180645 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1810003530 con 0x7f18280a2800 2024-09-16T15:01:11.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 1 -- 172.21.15.26:0/3438180645 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2980405986 0 0) 0x7f1810003530 con 0x7f18280a2800 2024-09-16T15:01:11.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 1 -- 172.21.15.26:0/3438180645 >> v1:172.21.15.26:6790/0 conn(0x7f18280a1d10 legacy=0x7f18280a20f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:01:11.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 1 -- 172.21.15.26:0/3438180645 >> v1:172.21.15.189:6789/0 conn(0x7f18280a0000 legacy=0x7f1828139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:11.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 10 monclient: _finish_hunting 0 2024-09-16T15:01:11.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 1 monclient: found mon.noname-a 2024-09-16T15:01:11.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:11.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:01:11.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 1 -- 172.21.15.26:0/3438180645 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182809ed60 con 0x7f18280a2800 2024-09-16T15:01:11.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 10 monclient: _finish_auth 0 2024-09-16T15:01:11.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 1 -- 172.21.15.26:0/3438180645 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f18200019a0 con 0x7f18280a2800 2024-09-16T15:01:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 10 monclient: dump: 2024-09-16T15:01:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:11.367 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:11.368 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 5 monclient: authenticate success, global_id 334257 2024-09-16T15:01:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 1 -- 172.21.15.26:0/3438180645 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1820001c40 con 0x7f18280a2800 2024-09-16T15:01:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 1 -- 172.21.15.26:0/3438180645 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1820002010 con 0x7f18280a2800 2024-09-16T15:01:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1825ffb640 10 monclient: dump: 2024-09-16T15:01:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:11.368 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:11.369 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 4 set_mon_vals no callback set 2024-09-16T15:01:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 10 monclient: shutdown 2024-09-16T15:01:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 1 -- 172.21.15.26:0/3438180645 >> v1:172.21.15.26:6789/0 conn(0x7f18280a2800 legacy=0x7f182813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 1 -- 172.21.15.26:0/3438180645 shutdown_connections 2024-09-16T15:01:11.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 1 -- 172.21.15.26:0/3438180645 >> 172.21.15.26:0/3438180645 conn(0x7f18280a7f30 msgr2=0x7f18280a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 1 -- 172.21.15.26:0/3438180645 shutdown_connections 2024-09-16T15:01:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 1 -- 172.21.15.26:0/3438180645 wait complete. 2024-09-16T15:01:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 10 monclient: build_initial_monmap 2024-09-16T15:01:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 10 monclient: monmap: 2024-09-16T15:01:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:11.370 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:11.371 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 1 Processor -- start 2024-09-16T15:01:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 1 -- start start 2024-09-16T15:01:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 10 monclient: init 2024-09-16T15:01:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:01:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 10 monclient(hunting): picked mon.noname-c con 0x7f18280a0000 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 10 monclient(hunting): picked mon.noname-b con 0x7f18280a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 10 monclient(hunting): picked mon.noname-a con 0x7f18280a2800 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1828134bb0 con 0x7f18280a2800 2024-09-16T15:01:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1828136e40 con 0x7f18280a1d10 2024-09-16T15:01:11.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1828133dd0 con 0x7f18280a0000 2024-09-16T15:01:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 10 monclient(hunting): authenticate will time out at 2095.482591s 2024-09-16T15:01:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f1826ffd640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f18280a0000 legacy=0x7f1828132fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f18277fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f18280a2800 0x7f18281d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53846/0 (socket says 172.21.15.26:53846) 2024-09-16T15:01:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f18277fe640 1 -- 172.21.15.26:0/2644130994 learned_addr learned my addr 172.21.15.26:0/2644130994 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278839431 0 0) 0x7f1828134bb0 con 0x7f18280a2800 2024-09-16T15:01:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient(hunting): my global_id is 334260 2024-09-16T15:01:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1814003530 con 0x7f18280a2800 2024-09-16T15:01:11.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 735633149 0 0) 0x7f1814003530 con 0x7f18280a2800 2024-09-16T15:01:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1828134bb0 con 0x7f18280a2800 2024-09-16T15:01:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f18200018a0 con 0x7f18280a2800 2024-09-16T15:01:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient(hunting): dump: 2024-09-16T15:01:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:11.373 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:11.374 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367298169 0 0) 0x7f1828136e40 con 0x7f18280a1d10 2024-09-16T15:01:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:11.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient(hunting): my global_id is 344173 2024-09-16T15:01:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1814003530 con 0x7f18280a1d10 2024-09-16T15:01:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3212276933 0 0) 0x7f1828134bb0 con 0x7f18280a2800 2024-09-16T15:01:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 >> v1:172.21.15.26:6790/0 conn(0x7f18280a0000 legacy=0x7f1828132fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:01:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 >> v1:172.21.15.189:6789/0 conn(0x7f18280a1d10 legacy=0x7f18281336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient: _finish_hunting 0 2024-09-16T15:01:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 1 monclient: found mon.a 2024-09-16T15:01:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1828137120 con 0x7f18280a2800 2024-09-16T15:01:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient: _finish_auth 0 2024-09-16T15:01:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient: _check_auth_tickets 2024-09-16T15:01:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:11.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 5 monclient: authenticate success, global_id 334260 2024-09-16T15:01:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 10 monclient: _renew_subs 2024-09-16T15:01:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 1 -- 172.21.15.26:0/2644130994 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18280a67c0 con 0x7f18280a2800 2024-09-16T15:01:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 10 monclient: _renew_subs 2024-09-16T15:01:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f182d6f9640 1 -- 172.21.15.26:0/2644130994 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f182809e500 con 0x7f18280a2800 2024-09-16T15:01:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1820004f80 con 0x7f18280a2800 2024-09-16T15:01:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f1820005290 con 0x7f18280a2800 2024-09-16T15:01:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f18257fa640 4 set_mon_vals no callback set 2024-09-16T15:01:11.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.361+0000 7f180f7fe640 10 monclient: dump: 2024-09-16T15:01:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:11.377 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.365+0000 7f182d6f9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:11.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.365+0000 7f182d6f9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.365+0000 7f182d6f9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.365+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f182000fd00 con 0x7f18280a2800 2024-09-16T15:01:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.365+0000 7f182d6f9640 1 -- 172.21.15.26:0/2644130994 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1808005050 con 0x7f18280a2800 2024-09-16T15:01:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.365+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f1820044290 con 0x7f18280a2800 2024-09-16T15:01:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.365+0000 7f1826ffd640 10 monclient: get_auth_request con 0x7f1814036220 auth_method 0 2024-09-16T15:01:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.369+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1820014a60 con 0x7f18280a2800 2024-09-16T15:01:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.369+0000 7f180f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:11.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.369+0000 7f180f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.585+0000 7f182d6f9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T15:01:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.585+0000 7f182d6f9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.585+0000 7f182d6f9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.585+0000 7f182d6f9640 1 -- 172.21.15.26:0/2644130994 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1808005880 con 0x7f18280a2800 2024-09-16T15:01:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.585+0000 7f180f7fe640 1 -- 172.21.15.26:0/2644130994 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1218626262) 0x7f1820013100 con 0x7f18280a2800 2024-09-16T15:01:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.585+0000 7f180f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:11.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.585+0000 7f180f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:11.587 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":168,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T15:01:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.589+0000 7f182d6f9640 1 -- 172.21.15.26:0/2644130994 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1814036220 legacy=0x7f18140386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.589+0000 7f182d6f9640 10 monclient: shutdown 2024-09-16T15:01:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.589+0000 7f182d6f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.589+0000 7f182d6f9640 1 -- 172.21.15.26:0/2644130994 >> v1:172.21.15.26:6789/0 conn(0x7f18280a2800 legacy=0x7f18281d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.589+0000 7f182d6f9640 1 -- 172.21.15.26:0/2644130994 shutdown_connections 2024-09-16T15:01:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.589+0000 7f182d6f9640 1 -- 172.21.15.26:0/2644130994 >> 172.21.15.26:0/2644130994 conn(0x7f18280a7f30 msgr2=0x7f18280a5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.589+0000 7f182d6f9640 1 -- 172.21.15.26:0/2644130994 shutdown_connections 2024-09-16T15:01:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:11.589+0000 7f182d6f9640 1 -- 172.21.15.26:0/2644130994 wait complete. 2024-09-16T15:01:14.604 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T15:01:14.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2ff44640 10 monclient: get_monmap_and_config 2024-09-16T15:01:14.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2ff44640 10 monclient: build_initial_monmap 2024-09-16T15:01:14.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2ff44640 10 monclient: monmap: 2024-09-16T15:01:14.737 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:14.737 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:14.737 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:14.738930+0000 2024-09-16T15:01:14.737 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:14.738930+0000 2024-09-16T15:01:14.737 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:14.737 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:14.737 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:14.738 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:14.738 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:14.738 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:14.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2ff44640 1 Processor -- start 2024-09-16T15:01:14.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2ff44640 1 -- start start 2024-09-16T15:01:14.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2ff44640 10 monclient: init 2024-09-16T15:01:14.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2ff44640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:14.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2ff44640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:01:14.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2ff44640 10 monclient(hunting): picked mon.noname-c con 0x7f0e2809a550 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:14.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2ff44640 10 monclient(hunting): picked mon.noname-b con 0x7f0e2809d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:14.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2ff44640 10 monclient(hunting): picked mon.noname-a con 0x7f0e2809b040 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:14.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2ff44640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2809dc60 con 0x7f0e2809b040 2024-09-16T15:01:14.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2ff44640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e28099ae0 con 0x7f0e2809a550 2024-09-16T15:01:14.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2ff44640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2809b8f0 con 0x7f0e2809d1b0 2024-09-16T15:01:14.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2ff44640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:14.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2ff44640 10 monclient(hunting): authenticate will time out at 2098.859204s 2024-09-16T15:01:14.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2d4df640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e28136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:14.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2e4e1640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0e2809b040 0x7f0e2813a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53862/0 (socket says 172.21.15.26:53862) 2024-09-16T15:01:14.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.737+0000 7f0e2e4e1640 1 -- 172.21.15.26:0/3932895812 learned_addr learned my addr 172.21.15.26:0/3932895812 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:14.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:14.941+0000 7f0e2d4df640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e28136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:15.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:15.345+0000 7f0e2d4df640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e28136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:16.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:16.149+0000 7f0e2d4df640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e28136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:17.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.737+0000 7f0e277fe640 10 monclient(hunting): tick 2024-09-16T15:01:17.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.737+0000 7f0e277fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T15:01:17.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.737+0000 7f0e277fe640 1 monclient(hunting): continuing hunt 2024-09-16T15:01:17.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.737+0000 7f0e277fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T15:01:17.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.737+0000 7f0e277fe640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e28136a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:01:17.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.737+0000 7f0e277fe640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.189:6789/0 conn(0x7f0e2809a550 legacy=0x7f0e2809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:17.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.737+0000 7f0e277fe640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6789/0 conn(0x7f0e2809b040 legacy=0x7f0e2813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:17.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.737+0000 7f0e277fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:01:17.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.737+0000 7f0e277fe640 10 monclient(hunting): picked mon.noname-c con 0x7f0e140058e0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:17.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.737+0000 7f0e277fe640 10 monclient(hunting): picked mon.noname-a con 0x7f0e14009570 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:17.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.737+0000 7f0e277fe640 10 monclient(hunting): picked mon.noname-b con 0x7f0e2809d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:17.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.737+0000 7f0e277fe640 1 -- 172.21.15.26:0/3932895812 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2809bbd0 con 0x7f0e14009570 2024-09-16T15:01:17.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.737+0000 7f0e277fe640 1 -- 172.21.15.26:0/3932895812 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2809b8f0 con 0x7f0e140058e0 2024-09-16T15:01:17.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.737+0000 7f0e277fe640 1 -- 172.21.15.26:0/3932895812 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e14012b10 con 0x7f0e2809d1b0 2024-09-16T15:01:17.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.737+0000 7f0e277fe640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:17.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.737+0000 7f0e2e4e1640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e1400f360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:17.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:17.941+0000 7f0e2e4e1640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e1400f360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:18.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:18.345+0000 7f0e2e4e1640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e1400f360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:19.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:19.149+0000 7f0e2e4e1640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e1400f360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:20.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.737+0000 7f0e277fe640 10 monclient(hunting): tick 2024-09-16T15:01:20.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.737+0000 7f0e277fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T15:01:20.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.737+0000 7f0e277fe640 1 monclient(hunting): continuing hunt 2024-09-16T15:01:20.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.737+0000 7f0e277fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T15:01:20.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.737+0000 7f0e277fe640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e1400f360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:01:20.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.737+0000 7f0e277fe640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.189:6789/0 conn(0x7f0e140058e0 legacy=0x7f0e14007d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:20.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.737+0000 7f0e277fe640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6789/0 conn(0x7f0e14009570 legacy=0x7f0e1400ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:20.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.737+0000 7f0e277fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:01:20.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.737+0000 7f0e277fe640 10 monclient(hunting): picked mon.noname-c con 0x7f0e14012dd0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:20.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.737+0000 7f0e277fe640 10 monclient(hunting): picked mon.noname-a con 0x7f0e14014f80 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:20.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.737+0000 7f0e277fe640 10 monclient(hunting): picked mon.noname-b con 0x7f0e2809d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:20.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.737+0000 7f0e277fe640 1 -- 172.21.15.26:0/3932895812 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e14013d00 con 0x7f0e14014f80 2024-09-16T15:01:20.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.737+0000 7f0e277fe640 1 -- 172.21.15.26:0/3932895812 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e14012b10 con 0x7f0e14012dd0 2024-09-16T15:01:20.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.737+0000 7f0e277fe640 1 -- 172.21.15.26:0/3932895812 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e1400c1d0 con 0x7f0e2809d1b0 2024-09-16T15:01:20.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.737+0000 7f0e277fe640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:20.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.737+0000 7f0e2e4e1640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e1400f360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:20.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:20.945+0000 7f0e2e4e1640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e1400f360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:21.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:21.349+0000 7f0e2e4e1640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e1400f360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:22.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.153+0000 7f0e2e4e1640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e1400f360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:22.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.225+0000 7f0e2ccde640 1 -- 172.21.15.26:0/3932895812 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828589955 0 0) 0x7f0e14013d00 con 0x7f0e14014f80 2024-09-16T15:01:22.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.225+0000 7f0e2ccde640 10 monclient(hunting): _negotiate 2024-09-16T15:01:22.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.225+0000 7f0e2ccde640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:22.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.225+0000 7f0e2ccde640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:22.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.225+0000 7f0e2ccde640 10 monclient(hunting): my global_id is 334263 2024-09-16T15:01:22.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.225+0000 7f0e2ccde640 1 -- 172.21.15.26:0/3932895812 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e10003530 con 0x7f0e14014f80 2024-09-16T15:01:22.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.229+0000 7f0e2ccde640 1 -- 172.21.15.26:0/3932895812 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291419551 0 0) 0x7f0e14012b10 con 0x7f0e14012dd0 2024-09-16T15:01:22.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.229+0000 7f0e2ccde640 10 monclient(hunting): _negotiate 2024-09-16T15:01:22.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.229+0000 7f0e2ccde640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:22.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.229+0000 7f0e2ccde640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:22.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.229+0000 7f0e2ccde640 10 monclient(hunting): my global_id is 344176 2024-09-16T15:01:22.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.229+0000 7f0e2ccde640 1 -- 172.21.15.26:0/3932895812 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e14013d00 con 0x7f0e14012dd0 2024-09-16T15:01:22.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 1 -- 172.21.15.26:0/3932895812 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1929165287 0 0) 0x7f0e14013d00 con 0x7f0e14012dd0 2024-09-16T15:01:22.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e1400f360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:01:22.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.26:6789/0 conn(0x7f0e14014f80 legacy=0x7f0e14017370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 10 monclient: _finish_hunting 0 2024-09-16T15:01:22.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 1 monclient: found mon.noname-c 2024-09-16T15:01:22.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:22.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T15:01:22.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 1 -- 172.21.15.26:0/3932895812 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1400c3b0 con 0x7f0e14012dd0 2024-09-16T15:01:22.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 10 monclient: _finish_auth 0 2024-09-16T15:01:22.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 10 monclient: _check_auth_tickets 2024-09-16T15:01:22.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:22.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 1 -- 172.21.15.26:0/3932895812 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0e20001920 con 0x7f0e14012dd0 2024-09-16T15:01:22.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:01:22.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 10 monclient: dump: 2024-09-16T15:01:22.233 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.233 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.233 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.233 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.234 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.234 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.234 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.234 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.234 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.234 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ff44640 5 monclient: authenticate success, global_id 344176 2024-09-16T15:01:22.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ff44640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:22.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 1 -- 172.21.15.26:0/3932895812 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e20001b40 con 0x7f0e14012dd0 2024-09-16T15:01:22.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:22.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 1 -- 172.21.15.26:0/3932895812 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0e20001f10 con 0x7f0e14012dd0 2024-09-16T15:01:22.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:22.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ccde640 10 monclient: dump: 2024-09-16T15:01:22.235 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.235 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.235 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.235 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.235 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.235 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.235 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.236 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.236 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.236 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ff44640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:22.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.233+0000 7f0e2ff44640 4 set_mon_vals no callback set 2024-09-16T15:01:22.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 10 monclient: shutdown 2024-09-16T15:01:22.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:22.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 1 -- 172.21.15.26:0/3932895812 >> v1:172.21.15.189:6789/0 conn(0x7f0e14012dd0 legacy=0x7f0e14013210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 1 -- 172.21.15.26:0/3932895812 shutdown_connections 2024-09-16T15:01:22.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 1 -- 172.21.15.26:0/3932895812 >> 172.21.15.26:0/3932895812 conn(0x7f0e2809fb50 msgr2=0x7f0e2809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:22.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 1 -- 172.21.15.26:0/3932895812 shutdown_connections 2024-09-16T15:01:22.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 1 -- 172.21.15.26:0/3932895812 wait complete. 2024-09-16T15:01:22.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 10 monclient: build_initial_monmap 2024-09-16T15:01:22.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 10 monclient: monmap: 2024-09-16T15:01:22.238 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:22.238 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:22.238 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:22.238 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:22.238 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:22.238 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:22.238 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:22.238 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:22.238 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:22.238 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 1 Processor -- start 2024-09-16T15:01:22.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 1 -- start start 2024-09-16T15:01:22.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 10 monclient: init 2024-09-16T15:01:22.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:22.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:01:22.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 10 monclient(hunting): picked mon.noname-b con 0x7f0e14012dd0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:22.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 10 monclient(hunting): picked mon.noname-a con 0x7f0e14014f80 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 10 monclient(hunting): picked mon.noname-c con 0x7f0e2809d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e281a72e0 con 0x7f0e14014f80 2024-09-16T15:01:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e281a4f10 con 0x7f0e14012dd0 2024-09-16T15:01:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e281b9a70 con 0x7f0e2809d1b0 2024-09-16T15:01:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2ff44640 10 monclient(hunting): authenticate will time out at 2106.359825s 2024-09-16T15:01:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.237+0000 7f0e2e4e1640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e281d1da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e2dce0640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0e14012dd0 0x7f0e281b8910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.26:60266/0 (socket says 172.21.15.26:60266) 2024-09-16T15:01:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e2dce0640 1 -- 172.21.15.26:0/1828962430 learned_addr learned my addr 172.21.15.26:0/1828962430 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079948779 0 0) 0x7f0e281a72e0 con 0x7f0e14014f80 2024-09-16T15:01:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:22.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient(hunting): my global_id is 334266 2024-09-16T15:01:22.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e1400c1d0 con 0x7f0e14014f80 2024-09-16T15:01:22.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 387974986 0 0) 0x7f0e281a4f10 con 0x7f0e14012dd0 2024-09-16T15:01:22.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:22.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:22.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:22.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient(hunting): my global_id is 344179 2024-09-16T15:01:22.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e281a72e0 con 0x7f0e14012dd0 2024-09-16T15:01:22.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 764803302 0 0) 0x7f0e1400c1d0 con 0x7f0e14014f80 2024-09-16T15:01:22.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e281a4f10 con 0x7f0e14014f80 2024-09-16T15:01:22.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0e1c0041f0 con 0x7f0e14014f80 2024-09-16T15:01:22.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:22.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient(hunting): dump: 2024-09-16T15:01:22.245 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.245 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.245 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.245 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.245 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.245 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.245 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.245 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.245 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.245 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:22.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1730257143 0 0) 0x7f0e281a72e0 con 0x7f0e14012dd0 2024-09-16T15:01:22.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e1400c1d0 con 0x7f0e14012dd0 2024-09-16T15:01:22.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0e20001f10 con 0x7f0e14012dd0 2024-09-16T15:01:22.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:22.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient(hunting): dump: 2024-09-16T15:01:22.246 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.246 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.246 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.246 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.246 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.246 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.246 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.246 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.247 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.247 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2292682659 0 0) 0x7f0e281a4f10 con 0x7f0e14014f80 2024-09-16T15:01:22.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 >> v1:172.21.15.26:6790/0 conn(0x7f0e2809d1b0 legacy=0x7f0e281d1da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:01:22.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 >> v1:172.21.15.189:6789/0 conn(0x7f0e14012dd0 legacy=0x7f0e281b8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient: _finish_hunting 0 2024-09-16T15:01:22.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 monclient: found mon.a 2024-09-16T15:01:22.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:22.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:22.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e280712e0 con 0x7f0e14014f80 2024-09-16T15:01:22.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient: _finish_auth 0 2024-09-16T15:01:22.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient: _check_auth_tickets 2024-09-16T15:01:22.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:22.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e2ff44640 5 monclient: authenticate success, global_id 334266 2024-09-16T15:01:22.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e2ff44640 10 monclient: _renew_subs 2024-09-16T15:01:22.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e2ff44640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:22.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e2ff44640 1 -- 172.21.15.26:0/1828962430 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2809a240 con 0x7f0e14014f80 2024-09-16T15:01:22.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e2ff44640 10 monclient: _renew_subs 2024-09-16T15:01:22.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e2ff44640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:22.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e2ff44640 1 -- 172.21.15.26:0/1828962430 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e281a6a40 con 0x7f0e14014f80 2024-09-16T15:01:22.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e1c004690 con 0x7f0e14014f80 2024-09-16T15:01:22.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:22.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0e1c004a20 con 0x7f0e14014f80 2024-09-16T15:01:22.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:22.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e27fff640 4 set_mon_vals no callback set 2024-09-16T15:01:22.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 10 monclient: dump: 2024-09-16T15:01:22.249 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.249 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.249 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.249 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.249 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.249 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.249 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.249 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.249 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.249 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e2ff44640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:22.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0e1c004c00 con 0x7f0e14014f80 2024-09-16T15:01:22.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e2ff44640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:22.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e2ff44640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:22.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e2ff44640 1 -- 172.21.15.26:0/1828962430 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e08005050 con 0x7f0e14014f80 2024-09-16T15:01:22.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f0e1c043d70 con 0x7f0e14014f80 2024-09-16T15:01:22.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.241+0000 7f0e2dce0640 10 monclient: get_auth_request con 0x7f0e14037870 auth_method 0 2024-09-16T15:01:22.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.249+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0e1c014660 con 0x7f0e14014f80 2024-09-16T15:01:22.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.249+0000 7f0e267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:22.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.249+0000 7f0e267fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:22.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.480+0000 7f0e2ff44640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T15:01:22.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.480+0000 7f0e2ff44640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:22.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.480+0000 7f0e2ff44640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:22.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.480+0000 7f0e2ff44640 1 -- 172.21.15.26:0/1828962430 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0e08005880 con 0x7f0e14014f80 2024-09-16T15:01:22.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.480+0000 7f0e267fc640 1 -- 172.21.15.26:0/1828962430 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 3993594072) 0x7f0e1c012ad0 con 0x7f0e14014f80 2024-09-16T15:01:22.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.480+0000 7f0e267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:22.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.480+0000 7f0e267fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:22.482 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":170,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T15:01:22.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.484+0000 7f0e2ff44640 1 -- 172.21.15.26:0/1828962430 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0e14037870 legacy=0x7f0e14039d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.484+0000 7f0e2ff44640 10 monclient: shutdown 2024-09-16T15:01:22.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.484+0000 7f0e2ff44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:22.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.484+0000 7f0e2ff44640 1 -- 172.21.15.26:0/1828962430 >> v1:172.21.15.26:6789/0 conn(0x7f0e14014f80 legacy=0x7f0e281ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.484+0000 7f0e2ff44640 1 -- 172.21.15.26:0/1828962430 shutdown_connections 2024-09-16T15:01:22.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.484+0000 7f0e2ff44640 1 -- 172.21.15.26:0/1828962430 >> 172.21.15.26:0/1828962430 conn(0x7f0e2809fb50 msgr2=0x7f0e280a0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:22.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.484+0000 7f0e2ff44640 1 -- 172.21.15.26:0/1828962430 shutdown_connections 2024-09-16T15:01:22.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.484+0000 7f0e2ff44640 1 -- 172.21.15.26:0/1828962430 wait complete. 2024-09-16T15:01:22.502 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T15:01:22.502 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T15:01:22.503 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T15:01:22.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 10 monclient: get_monmap_and_config 2024-09-16T15:01:22.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 10 monclient: build_initial_monmap 2024-09-16T15:01:22.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 10 monclient: monmap: 2024-09-16T15:01:22.624 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:22.624 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.624 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:22.626100+0000 2024-09-16T15:01:22.624 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:22.626100+0000 2024-09-16T15:01:22.624 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:22.624 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.624 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:22.625 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:22.625 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:22.625 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 1 Processor -- start 2024-09-16T15:01:22.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 1 -- start start 2024-09-16T15:01:22.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 10 monclient: init 2024-09-16T15:01:22.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:22.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:01:22.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 10 monclient(hunting): picked mon.noname-a con 0x7f79f809a550 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:22.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 10 monclient(hunting): picked mon.noname-b con 0x7f79f809d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:22.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 10 monclient(hunting): picked mon.noname-c con 0x7f79f809b040 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:22.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f809dc60 con 0x7f79f809a550 2024-09-16T15:01:22.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f8099ae0 con 0x7f79f809b040 2024-09-16T15:01:22.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f809b8f0 con 0x7f79f809d1b0 2024-09-16T15:01:22.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:22.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f6ffd640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f79f809d1b0 legacy=0x7f79f8136a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:22.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 10 monclient(hunting): authenticate will time out at 2106.746531s 2024-09-16T15:01:22.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f77fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f79f809a550 0x7f79f809a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53632/0 (socket says 172.21.15.26:53632) 2024-09-16T15:01:22.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f77fe640 1 -- 172.21.15.26:0/3273053550 learned_addr learned my addr 172.21.15.26:0/3273053550 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:22.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 1 -- 172.21.15.26:0/3273053550 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671070365 0 0) 0x7f79f809dc60 con 0x7f79f809a550 2024-09-16T15:01:22.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:22.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:22.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:22.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient(hunting): my global_id is 334269 2024-09-16T15:01:22.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 1 -- 172.21.15.26:0/3273053550 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79e0003530 con 0x7f79f809a550 2024-09-16T15:01:22.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 1 -- 172.21.15.26:0/3273053550 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014773136 0 0) 0x7f79f8099ae0 con 0x7f79f809b040 2024-09-16T15:01:22.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:22.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:22.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:22.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient(hunting): my global_id is 344182 2024-09-16T15:01:22.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 1 -- 172.21.15.26:0/3273053550 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79f809dc60 con 0x7f79f809b040 2024-09-16T15:01:22.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 1 -- 172.21.15.26:0/3273053550 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1822864750 0 0) 0x7f79e0003530 con 0x7f79f809a550 2024-09-16T15:01:22.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 1 -- 172.21.15.26:0/3273053550 >> v1:172.21.15.26:6790/0 conn(0x7f79f809d1b0 legacy=0x7f79f8136a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:01:22.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 1 -- 172.21.15.26:0/3273053550 >> v1:172.21.15.189:6789/0 conn(0x7f79f809b040 legacy=0x7f79f813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient: _finish_hunting 0 2024-09-16T15:01:22.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 1 monclient: found mon.noname-a 2024-09-16T15:01:22.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:22.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:01:22.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 1 -- 172.21.15.26:0/3273053550 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f809bbd0 con 0x7f79f809a550 2024-09-16T15:01:22.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient: _finish_auth 0 2024-09-16T15:01:22.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient: _check_auth_tickets 2024-09-16T15:01:22.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:22.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 1 -- 172.21.15.26:0/3273053550 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f79e8001620 con 0x7f79f809a550 2024-09-16T15:01:22.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:22.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient: dump: 2024-09-16T15:01:22.631 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.631 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.631 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.631 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.631 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.631 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.631 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.631 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.631 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.632 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 5 monclient: authenticate success, global_id 334269 2024-09-16T15:01:22.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:22.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 1 -- 172.21.15.26:0/3273053550 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79e8001900 con 0x7f79f809a550 2024-09-16T15:01:22.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:22.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 1 -- 172.21.15.26:0/3273053550 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f79e8001c90 con 0x7f79f809a550 2024-09-16T15:01:22.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:22.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79fdd89640 4 set_mon_vals no callback set 2024-09-16T15:01:22.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:22.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.624+0000 7f79f67fc640 10 monclient: dump: 2024-09-16T15:01:22.632 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.632 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.633 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.633 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.633 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.633 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.633 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.633 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.633 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.633 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 10 monclient: shutdown 2024-09-16T15:01:22.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:22.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 1 -- 172.21.15.26:0/3273053550 >> v1:172.21.15.26:6789/0 conn(0x7f79f809a550 legacy=0x7f79f809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 1 -- 172.21.15.26:0/3273053550 shutdown_connections 2024-09-16T15:01:22.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 1 -- 172.21.15.26:0/3273053550 >> 172.21.15.26:0/3273053550 conn(0x7f79f809fb50 msgr2=0x7f79f809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:22.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 1 -- 172.21.15.26:0/3273053550 shutdown_connections 2024-09-16T15:01:22.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 1 -- 172.21.15.26:0/3273053550 wait complete. 2024-09-16T15:01:22.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 10 monclient: build_initial_monmap 2024-09-16T15:01:22.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 10 monclient: monmap: 2024-09-16T15:01:22.637 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:22.637 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:22.638 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:22.638 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:22.638 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:22.638 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:22.638 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:22.638 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:22.638 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:22.638 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 1 Processor -- start 2024-09-16T15:01:22.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 1 -- start start 2024-09-16T15:01:22.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 10 monclient: init 2024-09-16T15:01:22.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:22.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:01:22.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 10 monclient(hunting): picked mon.noname-c con 0x7f79f809a550 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:22.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 10 monclient(hunting): picked mon.noname-a con 0x7f79f809b040 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:22.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 10 monclient(hunting): picked mon.noname-b con 0x7f79f809d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:22.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f809a240 con 0x7f79f809b040 2024-09-16T15:01:22.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79f77fe640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f79f809a550 legacy=0x7f79f81c9070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:22.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f81a72c0 con 0x7f79f809d1b0 2024-09-16T15:01:22.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f81b9740 con 0x7f79f809a550 2024-09-16T15:01:22.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:22.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 10 monclient(hunting): authenticate will time out at 2106.750701s 2024-09-16T15:01:22.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79f6ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f79f809b040 0x7f79f81b88d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53646/0 (socket says 172.21.15.26:53646) 2024-09-16T15:01:22.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79f6ffd640 1 -- 172.21.15.26:0/4075255571 learned_addr learned my addr 172.21.15.26:0/4075255571 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:22.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541992076 0 0) 0x7f79f809a240 con 0x7f79f809b040 2024-09-16T15:01:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient(hunting): my global_id is 334272 2024-09-16T15:01:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79e4003530 con 0x7f79f809b040 2024-09-16T15:01:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 198091912 0 0) 0x7f79e4003530 con 0x7f79f809b040 2024-09-16T15:01:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79f809a240 con 0x7f79f809b040 2024-09-16T15:01:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f79ec004160 con 0x7f79f809b040 2024-09-16T15:01:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient(hunting): dump: 2024-09-16T15:01:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.640 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.641 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362135576 0 0) 0x7f79f81a72c0 con 0x7f79f809d1b0 2024-09-16T15:01:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:22.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient(hunting): my global_id is 344185 2024-09-16T15:01:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79e4003530 con 0x7f79f809d1b0 2024-09-16T15:01:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3386445608 0 0) 0x7f79f809a240 con 0x7f79f809b040 2024-09-16T15:01:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 >> v1:172.21.15.26:6790/0 conn(0x7f79f809a550 legacy=0x7f79f81c9070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:01:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 >> v1:172.21.15.189:6789/0 conn(0x7f79f809d1b0 legacy=0x7f79f81d13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient: _finish_hunting 0 2024-09-16T15:01:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 1 monclient: found mon.a 2024-09-16T15:01:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f8070ff0 con 0x7f79f809b040 2024-09-16T15:01:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient: _finish_auth 0 2024-09-16T15:01:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient: _check_auth_tickets 2024-09-16T15:01:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:22.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 5 monclient: authenticate success, global_id 334272 2024-09-16T15:01:22.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 10 monclient: _renew_subs 2024-09-16T15:01:22.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:22.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 1 -- 172.21.15.26:0/4075255571 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f81a6fe0 con 0x7f79f809b040 2024-09-16T15:01:22.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 10 monclient: _renew_subs 2024-09-16T15:01:22.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:22.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79fdd89640 1 -- 172.21.15.26:0/4075255571 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f81a4f60 con 0x7f79f809b040 2024-09-16T15:01:22.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79ec004380 con 0x7f79f809b040 2024-09-16T15:01:22.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:22.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f79ec004710 con 0x7f79f809b040 2024-09-16T15:01:22.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:22.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79f5ffb640 4 set_mon_vals no callback set 2024-09-16T15:01:22.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.628+0000 7f79dffff640 10 monclient: dump: 2024-09-16T15:01:22.644 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.644 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.644 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.644 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.644 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.644 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.644 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.644 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.644 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.645 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79fdd89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:22.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79fdd89640 1 -- 172.21.15.26:0/4075255571 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f79d8006590 con 0x7f79d8002830 2024-09-16T15:01:22.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f79ec004870 con 0x7f79f809b040 2024-09-16T15:01:22.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f79ec043d80 con 0x7f79f809b040 2024-09-16T15:01:22.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852001784 0 0) 0x7f79d8006590 con 0x7f79d8002830 2024-09-16T15:01:22.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:22.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:22.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:22.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 10 monclient(hunting): my global_id is 334275 2024-09-16T15:01:22.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79f809a240 con 0x7f79d8002830 2024-09-16T15:01:22.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79f7fff640 10 monclient: get_auth_request con 0x7f79e4036180 auth_method 0 2024-09-16T15:01:22.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2140925044 0 0) 0x7f79f809a240 con 0x7f79d8002830 2024-09-16T15:01:22.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79d8006590 con 0x7f79d8002830 2024-09-16T15:01:22.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f79e8001080 con 0x7f79d8002830 2024-09-16T15:01:22.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:22.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 10 monclient: dump: 2024-09-16T15:01:22.646 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.646 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.646 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.646 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.646 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.646 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.646 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.646 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.646 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.647 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3337026693 0 0) 0x7f79d8006590 con 0x7f79d8002830 2024-09-16T15:01:22.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f79d800a0e0 con 0x7f79d8002830 2024-09-16T15:01:22.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f79e8001a90 con 0x7f79d8002830 2024-09-16T15:01:22.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:22.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:22.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 >> v1:172.21.15.26:6789/0 conn(0x7f79d8002830 legacy=0x7f79d8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.632+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 >> v1:172.21.15.26:6789/0 conn(0x7f79d8002830 legacy=0x7f79d8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:22.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.636+0000 7f79fdd89640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:01:22.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.636+0000 7f79fdd89640 1 -- 172.21.15.26:0/4075255571 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f79d8009da0 con 0x7f79d80090c0 2024-09-16T15:01:22.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790986116 0 0) 0x7f79d8009da0 con 0x7f79d80090c0 2024-09-16T15:01:22.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:22.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:22.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:22.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 10 monclient(hunting): my global_id is 334278 2024-09-16T15:01:22.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79d8006590 con 0x7f79d80090c0 2024-09-16T15:01:22.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3565538240 0 0) 0x7f79d8006590 con 0x7f79d80090c0 2024-09-16T15:01:22.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79d8009da0 con 0x7f79d80090c0 2024-09-16T15:01:22.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f79e8004ce0 con 0x7f79d80090c0 2024-09-16T15:01:22.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:22.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 10 monclient: dump: 2024-09-16T15:01:22.649 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.649 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.650 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.650 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.650 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.650 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.650 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.650 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.650 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.650 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 480127020 0 0) 0x7f79d8009da0 con 0x7f79d80090c0 2024-09-16T15:01:22.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f79d800c290 con 0x7f79d80090c0 2024-09-16T15:01:22.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4035 (unknown 0 0 89592233) 0x7f79e8005fd0 con 0x7f79d80090c0 2024-09-16T15:01:22.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:01:22.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:22.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 >> v1:172.21.15.26:6789/0 conn(0x7f79d80090c0 legacy=0x7f79d8009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79dffff640 1 -- 172.21.15.26:0/4075255571 >> v1:172.21.15.26:6789/0 conn(0x7f79d80090c0 legacy=0x7f79d8009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:22.651 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:01:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:01:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 170, 2024-09-16T15:01:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:01:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:01:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: 1 2024-09-16T15:01:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:01:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:01:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:01:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:01:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:01:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:01:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:01:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:01:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:01:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:01:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:01:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:01:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:01:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:01:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:01:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:01:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:01:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:01:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:01:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:01:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:01:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:01:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:01:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:01:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:01:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:01:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:01:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:01:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:01:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:01:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:01:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:01:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T15:01:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T15:01:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:01:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 7 2024-09-16T15:01:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:01:22.662 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:01:22.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79fdd89640 1 -- 172.21.15.26:0/4075255571 >> v1:172.21.15.189:6812/3196161582 conn(0x7f79e4036180 legacy=0x7f79e4038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79fdd89640 10 monclient: shutdown 2024-09-16T15:01:22.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79fdd89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:22.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79fdd89640 1 -- 172.21.15.26:0/4075255571 >> v1:172.21.15.26:6789/0 conn(0x7f79f809b040 legacy=0x7f79f81b88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79f7fff640 1 -- 172.21.15.26:0/4075255571 reap_dead start 2024-09-16T15:01:22.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79f7fff640 1 -- 172.21.15.26:0/4075255571 reap_dead start 2024-09-16T15:01:22.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79fdd89640 1 -- 172.21.15.26:0/4075255571 shutdown_connections 2024-09-16T15:01:22.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79fdd89640 1 -- 172.21.15.26:0/4075255571 >> 172.21.15.26:0/4075255571 conn(0x7f79f809fb50 msgr2=0x7f79f80a1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:22.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79fdd89640 1 -- 172.21.15.26:0/4075255571 shutdown_connections 2024-09-16T15:01:22.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.640+0000 7f79fdd89640 1 -- 172.21.15.26:0/4075255571 wait complete. 2024-09-16T15:01:22.664 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T15:01:22.664 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T15:01:22.665 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T15:01:22.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.776+0000 7f2c2e1c6640 10 monclient: get_monmap_and_config 2024-09-16T15:01:22.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.776+0000 7f2c2e1c6640 10 monclient: build_initial_monmap 2024-09-16T15:01:22.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.776+0000 7f2c2e1c6640 10 monclient: monmap: 2024-09-16T15:01:22.779 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:22.779 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.779 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:22.781240+0000 2024-09-16T15:01:22.779 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:22.781240+0000 2024-09-16T15:01:22.779 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:22.779 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.780 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:22.780 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:22.780 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:22.780 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 1 Processor -- start 2024-09-16T15:01:22.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 1 -- start start 2024-09-16T15:01:22.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 10 monclient: init 2024-09-16T15:01:22.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:22.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:01:22.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 10 monclient(hunting): picked mon.noname-b con 0x7f2c2809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:22.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 10 monclient(hunting): picked mon.noname-c con 0x7f2c28132650 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:22.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 10 monclient(hunting): picked mon.noname-a con 0x7f2c28138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:22.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2809d260 con 0x7f2c28138ff0 2024-09-16T15:01:22.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c28132a80 con 0x7f2c28132650 2024-09-16T15:01:22.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2809cf60 con 0x7f2c2809cb80 2024-09-16T15:01:22.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c277fe640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2c2809cb80 legacy=0x7f2c280717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:22.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:22.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 10 monclient(hunting): authenticate will time out at 2106.901285s 2024-09-16T15:01:22.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c27fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2c28138ff0 0x7f2c2813b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53672/0 (socket says 172.21.15.26:53672) 2024-09-16T15:01:22.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c27fff640 1 -- 172.21.15.26:0/1164995891 learned_addr learned my addr 172.21.15.26:0/1164995891 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:22.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 1 -- 172.21.15.26:0/1164995891 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769296023 0 0) 0x7f2c2809d260 con 0x7f2c28138ff0 2024-09-16T15:01:22.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:22.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:22.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:22.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient(hunting): my global_id is 334281 2024-09-16T15:01:22.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 1 -- 172.21.15.26:0/1164995891 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c14003530 con 0x7f2c28138ff0 2024-09-16T15:01:22.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 1 -- 172.21.15.26:0/1164995891 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113199714 0 0) 0x7f2c28132a80 con 0x7f2c28132650 2024-09-16T15:01:22.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:22.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:22.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:22.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient(hunting): my global_id is 344188 2024-09-16T15:01:22.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 1 -- 172.21.15.26:0/1164995891 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c2809d260 con 0x7f2c28132650 2024-09-16T15:01:22.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 1 -- 172.21.15.26:0/1164995891 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 216954907 0 0) 0x7f2c14003530 con 0x7f2c28138ff0 2024-09-16T15:01:22.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 1 -- 172.21.15.26:0/1164995891 >> v1:172.21.15.26:6790/0 conn(0x7f2c2809cb80 legacy=0x7f2c280717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:01:22.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 1 -- 172.21.15.26:0/1164995891 >> v1:172.21.15.189:6789/0 conn(0x7f2c28132650 legacy=0x7f2c281378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient: _finish_hunting 0 2024-09-16T15:01:22.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 1 monclient: found mon.noname-a 2024-09-16T15:01:22.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:22.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:01:22.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 1 -- 172.21.15.26:0/1164995891 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2809f2d0 con 0x7f2c28138ff0 2024-09-16T15:01:22.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient: _finish_auth 0 2024-09-16T15:01:22.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient: _check_auth_tickets 2024-09-16T15:01:22.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 1 -- 172.21.15.26:0/1164995891 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2c200019a0 con 0x7f2c28138ff0 2024-09-16T15:01:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient: dump: 2024-09-16T15:01:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.791 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 5 monclient: authenticate success, global_id 334281 2024-09-16T15:01:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 1 -- 172.21.15.26:0/1164995891 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c20001bc0 con 0x7f2c28138ff0 2024-09-16T15:01:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 4 set_mon_vals no callback set 2024-09-16T15:01:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 1 -- 172.21.15.26:0/1164995891 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2c20001ed0 con 0x7f2c28138ff0 2024-09-16T15:01:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c267fc640 10 monclient: dump: 2024-09-16T15:01:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.791 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.792 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.792 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.792 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.792 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.792 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 10 monclient: shutdown 2024-09-16T15:01:22.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:22.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 1 -- 172.21.15.26:0/1164995891 >> v1:172.21.15.26:6789/0 conn(0x7f2c28138ff0 legacy=0x7f2c2813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 1 -- 172.21.15.26:0/1164995891 shutdown_connections 2024-09-16T15:01:22.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 1 -- 172.21.15.26:0/1164995891 >> 172.21.15.26:0/1164995891 conn(0x7f2c2809fb50 msgr2=0x7f2c2809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:22.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 1 -- 172.21.15.26:0/1164995891 shutdown_connections 2024-09-16T15:01:22.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 1 -- 172.21.15.26:0/1164995891 wait complete. 2024-09-16T15:01:22.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 10 monclient: build_initial_monmap 2024-09-16T15:01:22.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 10 monclient: monmap: 2024-09-16T15:01:22.793 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:22.793 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:22.793 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:22.793 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:22.793 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:22.793 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:22.793 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:22.793 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:22.793 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:22.793 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 1 Processor -- start 2024-09-16T15:01:22.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 1 -- start start 2024-09-16T15:01:22.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.780+0000 7f2c2e1c6640 10 monclient: init 2024-09-16T15:01:22.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:22.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:01:22.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 10 monclient(hunting): picked mon.noname-c con 0x7f2c2809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:22.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 10 monclient(hunting): picked mon.noname-b con 0x7f2c28132650 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:22.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 10 monclient(hunting): picked mon.noname-a con 0x7f2c28138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:22.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c28133770 con 0x7f2c28138ff0 2024-09-16T15:01:22.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c28138080 con 0x7f2c28132650 2024-09-16T15:01:22.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c277fe640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2c2809cb80 legacy=0x7f2c2806d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:22.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2806e470 con 0x7f2c2809cb80 2024-09-16T15:01:22.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:22.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 10 monclient(hunting): authenticate will time out at 2106.904499s 2024-09-16T15:01:22.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c27fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2c28138ff0 0x7f2c281cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53686/0 (socket says 172.21.15.26:53686) 2024-09-16T15:01:22.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c27fff640 1 -- 172.21.15.26:0/3299616574 learned_addr learned my addr 172.21.15.26:0/3299616574 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:22.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120739144 0 0) 0x7f2c28133770 con 0x7f2c28138ff0 2024-09-16T15:01:22.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:22.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:22.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:22.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient(hunting): my global_id is 334284 2024-09-16T15:01:22.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c08003530 con 0x7f2c28138ff0 2024-09-16T15:01:22.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3352240837 0 0) 0x7f2c08003530 con 0x7f2c28138ff0 2024-09-16T15:01:22.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c28133770 con 0x7f2c28138ff0 2024-09-16T15:01:22.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2c20001960 con 0x7f2c28138ff0 2024-09-16T15:01:22.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:22.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient(hunting): dump: 2024-09-16T15:01:22.796 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.796 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.796 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.796 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.796 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.796 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.796 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.796 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.796 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.796 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:22.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092830636 0 0) 0x7f2c2809d260 con 0x7f2c28132650 2024-09-16T15:01:22.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:22.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:22.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:22.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient(hunting): my global_id is 344191 2024-09-16T15:01:22.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c08003530 con 0x7f2c28132650 2024-09-16T15:01:22.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2124104379 0 0) 0x7f2c28133770 con 0x7f2c28138ff0 2024-09-16T15:01:22.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 >> v1:172.21.15.26:6790/0 conn(0x7f2c2809cb80 legacy=0x7f2c2806d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:01:22.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 >> v1:172.21.15.189:6789/0 conn(0x7f2c28132650 legacy=0x7f2c2806dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient: _finish_hunting 0 2024-09-16T15:01:22.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 monclient: found mon.a 2024-09-16T15:01:22.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:22.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:22.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2806f290 con 0x7f2c28138ff0 2024-09-16T15:01:22.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient: _finish_auth 0 2024-09-16T15:01:22.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient: _check_auth_tickets 2024-09-16T15:01:22.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:22.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 5 monclient: authenticate success, global_id 334284 2024-09-16T15:01:22.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 10 monclient: _renew_subs 2024-09-16T15:01:22.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:22.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 1 -- 172.21.15.26:0/3299616574 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2809a240 con 0x7f2c28138ff0 2024-09-16T15:01:22.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 10 monclient: _renew_subs 2024-09-16T15:01:22.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:22.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 1 -- 172.21.15.26:0/3299616574 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2806fb60 con 0x7f2c28138ff0 2024-09-16T15:01:22.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c20004f10 con 0x7f2c28138ff0 2024-09-16T15:01:22.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:22.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2c20005220 con 0x7f2c28138ff0 2024-09-16T15:01:22.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c25ffb640 4 set_mon_vals no callback set 2024-09-16T15:01:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient: dump: 2024-09-16T15:01:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.799 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c2e1c6640 1 -- 172.21.15.26:0/3299616574 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2c0c006590 con 0x7f2c0c002830 2024-09-16T15:01:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f2c20005400 con 0x7f2c28138ff0 2024-09-16T15:01:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f2c200442f0 con 0x7f2c28138ff0 2024-09-16T15:01:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c26ffd640 10 monclient: get_auth_request con 0x7f2c080360e0 auth_method 0 2024-09-16T15:01:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831814182 0 0) 0x7f2c0c006590 con 0x7f2c0c002830 2024-09-16T15:01:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient(hunting): my global_id is 344194 2024-09-16T15:01:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c28133770 con 0x7f2c0c002830 2024-09-16T15:01:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1422295514 0 0) 0x7f2c28133770 con 0x7f2c0c002830 2024-09-16T15:01:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c0c006590 con 0x7f2c0c002830 2024-09-16T15:01:22.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2c10003f70 con 0x7f2c0c002830 2024-09-16T15:01:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 10 monclient: dump: 2024-09-16T15:01:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.801 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1335393562 0 0) 0x7f2c0c006590 con 0x7f2c0c002830 2024-09-16T15:01:22.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.784+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c0c00a0e0 con 0x7f2c0c002830 2024-09-16T15:01:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.788+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2c10006340 con 0x7f2c0c002830 2024-09-16T15:01:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.788+0000 7f2c1ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.788+0000 7f2c1ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.788+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 >> v1:172.21.15.189:6789/0 conn(0x7f2c0c002830 legacy=0x7f2c0c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.788+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 >> v1:172.21.15.189:6789/0 conn(0x7f2c0c002830 legacy=0x7f2c0c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.792+0000 7f2c2e1c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:01:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.792+0000 7f2c2e1c6640 1 -- 172.21.15.26:0/3299616574 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2c0c009da0 con 0x7f2c0c0090c0 2024-09-16T15:01:22.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.792+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530347339 0 0) 0x7f2c0c009da0 con 0x7f2c0c0090c0 2024-09-16T15:01:22.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.792+0000 7f2c1ffff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:22.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.792+0000 7f2c1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:22.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.792+0000 7f2c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:22.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.792+0000 7f2c1ffff640 10 monclient(hunting): my global_id is 344197 2024-09-16T15:01:22.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.792+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c0c006590 con 0x7f2c0c0090c0 2024-09-16T15:01:22.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1643623995 0 0) 0x7f2c0c006590 con 0x7f2c0c0090c0 2024-09-16T15:01:22.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c0c009da0 con 0x7f2c0c0090c0 2024-09-16T15:01:22.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f2c10004770 con 0x7f2c0c0090c0 2024-09-16T15:01:22.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:22.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c1ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:22.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c1ffff640 10 monclient: dump: 2024-09-16T15:01:22.804 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:22.805 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3100886568 0 0) 0x7f2c0c009da0 con 0x7f2c0c0090c0 2024-09-16T15:01:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2c0c00c290 con 0x7f2c0c0090c0 2024-09-16T15:01:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4037 (unknown 0 0 3032848798) 0x7f2c10005b10 con 0x7f2c0c0090c0 2024-09-16T15:01:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c1ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:01:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c1ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 >> v1:172.21.15.189:6789/0 conn(0x7f2c0c0090c0 legacy=0x7f2c0c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c1ffff640 1 -- 172.21.15.26:0/3299616574 >> v1:172.21.15.189:6789/0 conn(0x7f2c0c0090c0 legacy=0x7f2c0c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:22.806 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:01:22.806 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:22.806 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:22.806 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T15:01:22.806 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 170, 2024-09-16T15:01:22.806 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:01:22.807 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:01:22.807 INFO:teuthology.orchestra.run.smithi026.stdout: 1 2024-09-16T15:01:22.807 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:22.807 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:01:22.807 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:22.807 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:01:22.807 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:01:22.807 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:22.807 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:22.807 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:22.807 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:22.807 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:22.807 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:22.808 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:22.808 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:22.808 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:22.808 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:22.808 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:22.808 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:01:22.808 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:01:22.808 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:22.808 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:22.808 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:22.808 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:22.808 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:22.808 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:22.809 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:22.809 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:22.809 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:22.809 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:22.809 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:22.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.809 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:01:22.809 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:01:22.809 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:01:22.809 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:01:22.809 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:01:22.809 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:01:22.809 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:22.810 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:22.810 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:01:22.810 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:01:22.810 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:01:22.810 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:01:22.810 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:01:22.810 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:01:22.810 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:01:22.810 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:22.810 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:01:22.810 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:22.810 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:22.810 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:22.811 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:22.811 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:22.811 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:22.811 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:22.811 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:22.811 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:22.811 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:22.811 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:22.811 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:01:22.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.811 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:01:22.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.811 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:22.811 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:22.812 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:22.812 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:22.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.812 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:22.812 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:01:22.812 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:22.812 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:22.812 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:22.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.812 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:01:22.812 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:01:22.812 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:22.813 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:22.813 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:22.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.813 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:22.813 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:22.813 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:22.813 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:22.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.813 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:22.813 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:01:22.813 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:22.814 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:22.814 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:22.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.814 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:01:22.814 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:01:22.814 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:22.814 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:22.814 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:22.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.814 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:22.814 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:22.815 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:22.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:22.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:22.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:01:22.815 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:22.815 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:22.815 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:22.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:01:22.815 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:01:22.816 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:22.816 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:22.816 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:22.816 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:22.816 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:22.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.816 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:01:22.816 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:01:22.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.816 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:22.816 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:22.817 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:22.817 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:22.817 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:22.817 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:01:22.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.817 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:22.817 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:22.817 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:22.817 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:22.817 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:22.817 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T15:01:22.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:22.818 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:22.818 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:22.818 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:22.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:22.818 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:22.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:22.818 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:01:22.818 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:01:22.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c2e1c6640 1 -- 172.21.15.26:0/3299616574 >> v1:172.21.15.189:6812/3196161582 conn(0x7f2c080360e0 legacy=0x7f2c080385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c2e1c6640 10 monclient: shutdown 2024-09-16T15:01:22.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c2e1c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:22.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c2e1c6640 1 -- 172.21.15.26:0/3299616574 >> v1:172.21.15.26:6789/0 conn(0x7f2c28138ff0 legacy=0x7f2c281cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:22.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c27fff640 1 -- 172.21.15.26:0/3299616574 reap_dead start 2024-09-16T15:01:22.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c27fff640 1 -- 172.21.15.26:0/3299616574 reap_dead start 2024-09-16T15:01:22.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c2e1c6640 1 -- 172.21.15.26:0/3299616574 shutdown_connections 2024-09-16T15:01:22.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c2e1c6640 1 -- 172.21.15.26:0/3299616574 >> 172.21.15.26:0/3299616574 conn(0x7f2c2809fb50 msgr2=0x7f2c281314f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:22.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c2e1c6640 1 -- 172.21.15.26:0/3299616574 shutdown_connections 2024-09-16T15:01:22.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:22.796+0000 7f2c2e1c6640 1 -- 172.21.15.26:0/3299616574 wait complete. 2024-09-16T15:01:22.821 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T15:01:22.821 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T15:01:37.865 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=110 2024-09-16T15:01:37.865 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 110 -ne 110 ]] 2024-09-16T15:01:37.866 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T15:01:37.866 INFO:tasks.workunit.client.0.smithi189.stderr:+ export CEPH_ARGS= 2024-09-16T15:01:37.866 INFO:tasks.workunit.client.0.smithi189.stderr:+ CEPH_ARGS= 2024-09-16T15:01:37.866 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph auth del client.bar 2024-09-16T15:01:37.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 10 monclient: get_monmap_and_config 2024-09-16T15:01:37.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 10 monclient: build_initial_monmap 2024-09-16T15:01:37.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 10 monclient: monmap: 2024-09-16T15:01:37.978 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T15:01:37.978 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:37.978 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T15:01:37.982539+0000 2024-09-16T15:01:37.978 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T15:01:37.982539+0000 2024-09-16T15:01:37.978 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:37.978 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:37.979 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:37.979 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:37.979 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:37.979 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:37.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 1 Processor -- start 2024-09-16T15:01:37.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 1 -- start start 2024-09-16T15:01:37.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 10 monclient: init 2024-09-16T15:01:37.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:37.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:01:37.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 10 monclient(hunting): picked mon.noname-b con 0x7f52f00a38d0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:37.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 10 monclient(hunting): picked mon.noname-a con 0x7f52f0131da0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:37.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 10 monclient(hunting): picked mon.noname-c con 0x7f52f01321d0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:37.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f00a3cb0 con 0x7f52f0131da0 2024-09-16T15:01:37.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f009cc60 con 0x7f52f01321d0 2024-09-16T15:01:37.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f009eaf0 con 0x7f52f00a38d0 2024-09-16T15:01:37.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:37.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 10 monclient(hunting): authenticate will time out at 2080.098151s 2024-09-16T15:01:37.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52ef7fe640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f52f00a38d0 legacy=0x7f52f0131690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:37.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52effff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f52f01321d0 0x7f52f013a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33354/0 (socket says 172.21.15.189:33354) 2024-09-16T15:01:37.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52effff640 1 -- 172.21.15.189:0/3448374932 learned_addr learned my addr 172.21.15.189:0/3448374932 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T15:01:37.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 1 -- 172.21.15.189:0/3448374932 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916020024 0 0) 0x7f52f009cc60 con 0x7f52f01321d0 2024-09-16T15:01:37.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:37.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:37.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:37.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 10 monclient(hunting): my global_id is 344200 2024-09-16T15:01:37.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 1 -- 172.21.15.189:0/3448374932 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52dc003530 con 0x7f52f01321d0 2024-09-16T15:01:37.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 1 -- 172.21.15.189:0/3448374932 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3501695672 0 0) 0x7f52dc003530 con 0x7f52f01321d0 2024-09-16T15:01:37.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 1 -- 172.21.15.189:0/3448374932 >> v1:172.21.15.26:6790/0 conn(0x7f52f00a38d0 legacy=0x7f52f0131690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:01:37.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 1 -- 172.21.15.189:0/3448374932 >> v1:172.21.15.26:6789/0 conn(0x7f52f0131da0 legacy=0x7f52f0136ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:37.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 10 monclient: _finish_hunting 0 2024-09-16T15:01:37.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 1 monclient: found mon.noname-c 2024-09-16T15:01:37.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:37.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T15:01:37.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 1 -- 172.21.15.189:0/3448374932 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f00a05f0 con 0x7f52f01321d0 2024-09-16T15:01:37.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 10 monclient: _finish_auth 0 2024-09-16T15:01:37.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 10 monclient: _check_auth_tickets 2024-09-16T15:01:37.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:37.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 1 -- 172.21.15.189:0/3448374932 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52e80019a0 con 0x7f52f01321d0 2024-09-16T15:01:37.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 5 monclient: authenticate success, global_id 344200 2024-09-16T15:01:37.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:37.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:37.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:01:37.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 10 monclient: dump: 2024-09-16T15:01:37.990 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:37.990 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:37.990 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:37.990 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:37.990 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:37.991 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:37.991 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:37.991 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:37.991 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:37.991 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:37.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:37.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 1 -- 172.21.15.189:0/3448374932 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52e8001cc0 con 0x7f52f01321d0 2024-09-16T15:01:37.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:37.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:37.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 4 set_mon_vals no callback set 2024-09-16T15:01:37.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 1 -- 172.21.15.189:0/3448374932 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52e8002050 con 0x7f52f01321d0 2024-09-16T15:01:37.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:37.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:37.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52eeffd640 10 monclient: dump: 2024-09-16T15:01:37.992 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:37.992 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:37.992 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:37.992 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:37.992 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:37.992 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:37.992 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:37.993 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:37.993 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:37.993 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:37.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 10 monclient: shutdown 2024-09-16T15:01:37.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:37.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 1 -- 172.21.15.189:0/3448374932 >> v1:172.21.15.189:6789/0 conn(0x7f52f01321d0 legacy=0x7f52f013a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:37.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 1 -- 172.21.15.189:0/3448374932 shutdown_connections 2024-09-16T15:01:37.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 1 -- 172.21.15.189:0/3448374932 >> 172.21.15.189:0/3448374932 conn(0x7f52f00a34c0 msgr2=0x7f52f009f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:37.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 1 -- 172.21.15.189:0/3448374932 shutdown_connections 2024-09-16T15:01:37.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.977+0000 7f52f5d8a640 1 -- 172.21.15.189:0/3448374932 wait complete. 2024-09-16T15:01:37.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 monclient: build_initial_monmap 2024-09-16T15:01:37.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 monclient: monmap: 2024-09-16T15:01:37.994 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T15:01:37.994 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:37.994 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T15:01:37.994 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T15:01:37.994 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:37.994 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T15:01:37.994 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:37.994 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:37.994 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:37.994 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:37.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 1 Processor -- start 2024-09-16T15:01:37.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 1 -- start start 2024-09-16T15:01:37.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 monclient: init 2024-09-16T15:01:37.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:37.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:01:37.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 monclient(hunting): picked mon.noname-a con 0x7f52f00a38d0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:37.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 monclient(hunting): picked mon.noname-b con 0x7f52f0131da0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:37.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 monclient(hunting): picked mon.noname-c con 0x7f52f01321d0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:37.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f0120530 con 0x7f52f00a38d0 2024-09-16T15:01:37.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f0121aa0 con 0x7f52f0131da0 2024-09-16T15:01:37.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f011f760 con 0x7f52f01321d0 2024-09-16T15:01:37.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:37.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 monclient(hunting): authenticate will time out at 2080.101311s 2024-09-16T15:01:37.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52e7fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f52f0131da0 0x7f52f011e660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33364/0 (socket says 172.21.15.189:33364) 2024-09-16T15:01:37.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52e7fff640 1 -- 172.21.15.189:0/1070076546 learned_addr learned my addr 172.21.15.189:0/1070076546 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T15:01:37.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52effff640 1 -- 172.21.15.189:0/1070076546 >> v1:172.21.15.26:6790/0 conn(0x7f52f01321d0 legacy=0x7f52f01d1570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6790/0 2024-09-16T15:01:37.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209039378 0 0) 0x7f52f0121aa0 con 0x7f52f0131da0 2024-09-16T15:01:37.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:37.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:37.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:37.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient(hunting): my global_id is 344203 2024-09-16T15:01:37.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d0003530 con 0x7f52f0131da0 2024-09-16T15:01:37.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4066742010 0 0) 0x7f52d0003530 con 0x7f52f0131da0 2024-09-16T15:01:37.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52f0121aa0 con 0x7f52f0131da0 2024-09-16T15:01:37.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52d80019d0 con 0x7f52f0131da0 2024-09-16T15:01:37.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:37.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:01:37.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient(hunting): dump: 2024-09-16T15:01:37.997 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:37.997 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:37.997 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:37.997 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:37.998 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:37.998 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:37.998 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:37.998 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:37.998 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:37.998 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:37.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:37.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252610544 0 0) 0x7f52f0120530 con 0x7f52f00a38d0 2024-09-16T15:01:37.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:37.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:37.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:37.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient(hunting): my global_id is 334290 2024-09-16T15:01:37.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d0003530 con 0x7f52f00a38d0 2024-09-16T15:01:37.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2289960844 0 0) 0x7f52f0121aa0 con 0x7f52f0131da0 2024-09-16T15:01:37.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 >> v1:172.21.15.26:6790/0 conn(0x7f52f01321d0 legacy=0x7f52f01d1570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:01:37.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 >> v1:172.21.15.26:6789/0 conn(0x7f52f00a38d0 legacy=0x7f52f011df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:37.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient: _finish_hunting 0 2024-09-16T15:01:37.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 1 monclient: found mon.b 2024-09-16T15:01:37.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:37.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:38.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f012f700 con 0x7f52f0131da0 2024-09-16T15:01:38.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient: _finish_auth 0 2024-09-16T15:01:38.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient: _check_auth_tickets 2024-09-16T15:01:38.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:38.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 5 monclient: authenticate success, global_id 344203 2024-09-16T15:01:38.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 monclient: _renew_subs 2024-09-16T15:01:38.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:38.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 1 -- 172.21.15.189:0/1070076546 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f00a12c0 con 0x7f52f0131da0 2024-09-16T15:01:38.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 monclient: _renew_subs 2024-09-16T15:01:38.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:38.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 1 -- 172.21.15.189:0/1070076546 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f01217c0 con 0x7f52f0131da0 2024-09-16T15:01:38.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52d8001cb0 con 0x7f52f0131da0 2024-09-16T15:01:38.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:38.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52d8003af0 con 0x7f52f0131da0 2024-09-16T15:01:38.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:38.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:38.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ee7fc640 4 set_mon_vals no callback set 2024-09-16T15:01:38.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 10 monclient: dump: 2024-09-16T15:01:38.001 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:38.001 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:38.001 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:38.001 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:38.002 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:38.002 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:38.002 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:38.002 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:38.002 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:38.002 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:38.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f52d8003c50 con 0x7f52f0131da0 2024-09-16T15:01:38.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:38.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:38.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:38.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52f5d8a640 1 -- 172.21.15.189:0/1070076546 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d4005050 con 0x7f52f0131da0 2024-09-16T15:01:38.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f52d8043e50 con 0x7f52f0131da0 2024-09-16T15:01:38.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.981+0000 7f52ef7fe640 10 monclient: get_auth_request con 0x7f52d0036220 auth_method 0 2024-09-16T15:01:38.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.985+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f52d8014790 con 0x7f52f0131da0 2024-09-16T15:01:38.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.985+0000 7f52ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:38.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:37.985+0000 7f52ecff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:38.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.201+0000 7f52f5d8a640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T15:01:38.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.201+0000 7f52f5d8a640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T15:01:38.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.201+0000 7f52f5d8a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:38.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.201+0000 7f52f5d8a640 1 -- 172.21.15.189:0/1070076546 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.bar"} v 0) v1 -- 0x7f52d4005300 con 0x7f52f0131da0 2024-09-16T15:01:38.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.209+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bar"}]=0 v45) v1 ==== 80+0+0 (unknown 526246777 0 0) 0x7f52d8012c00 con 0x7f52f0131da0 2024-09-16T15:01:38.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.209+0000 7f52ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T15:01:38.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.209+0000 7f52ecff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:38.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.349+0000 7f52f5d8a640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T15:01:38.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.349+0000 7f52f5d8a640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T15:01:38.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.349+0000 7f52f5d8a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:38.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.349+0000 7f52f5d8a640 1 -- 172.21.15.189:0/1070076546 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.bar"} v 0) v1 -- 0x7f52d4005850 con 0x7f52f0131da0 2024-09-16T15:01:38.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.349+0000 7f52ecff9640 1 -- 172.21.15.189:0/1070076546 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bar"}]=0 v45) v1 ==== 80+0+0 (unknown 526246777 0 0) 0x7f52d8012c00 con 0x7f52f0131da0 2024-09-16T15:01:38.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.349+0000 7f52ecff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T15:01:38.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.349+0000 7f52ecff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T15:01:38.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.353+0000 7f52f5d8a640 1 -- 172.21.15.189:0/1070076546 >> v1:172.21.15.189:6812/3196161582 conn(0x7f52d0036220 legacy=0x7f52d00386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:38.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.353+0000 7f52f5d8a640 10 monclient: shutdown 2024-09-16T15:01:38.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.353+0000 7f52f5d8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:38.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.353+0000 7f52f5d8a640 1 -- 172.21.15.189:0/1070076546 >> v1:172.21.15.189:6789/0 conn(0x7f52f0131da0 legacy=0x7f52f011e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:38.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.353+0000 7f52f5d8a640 1 -- 172.21.15.189:0/1070076546 shutdown_connections 2024-09-16T15:01:38.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.353+0000 7f52f5d8a640 1 -- 172.21.15.189:0/1070076546 >> 172.21.15.189:0/1070076546 conn(0x7f52f00a34c0 msgr2=0x7f52f00a1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:38.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.353+0000 7f52f5d8a640 1 -- 172.21.15.189:0/1070076546 shutdown_connections 2024-09-16T15:01:38.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:38.353+0000 7f52f5d8a640 1 -- 172.21.15.189:0/1070076546 wait complete. 2024-09-16T15:01:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 13 2024-09-16T15:01:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 2024-09-16T15:01:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T15:01:38.368 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T15:01:38.368 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T15:01:38.369 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T15:01:42.822 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T15:01:42.822 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T15:01:42.822 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T15:01:42.823 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T15:01:42.825 INFO:tasks.ceph.mon.c:Started 2024-09-16T15:01:42.825 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T15:01:42.826 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T15:01:42.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.960+0000 7fa9064fb640 10 monclient: get_monmap_and_config 2024-09-16T15:01:42.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.960+0000 7fa9064fb640 10 monclient: build_initial_monmap 2024-09-16T15:01:42.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.960+0000 7fa9064fb640 10 monclient: monmap: 2024-09-16T15:01:42.962 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:42.962 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:42.962 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:42.964270+0000 2024-09-16T15:01:42.962 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:42.964270+0000 2024-09-16T15:01:42.962 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:42.962 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:42.962 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:42.962 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:42.962 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:42.963 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:42.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.960+0000 7fa9064fb640 1 Processor -- start 2024-09-16T15:01:42.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.960+0000 7fa9064fb640 1 -- start start 2024-09-16T15:01:42.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.960+0000 7fa9064fb640 10 monclient: init 2024-09-16T15:01:42.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.960+0000 7fa9064fb640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:42.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:01:42.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 10 monclient(hunting): picked mon.noname-c con 0x7fa9000a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:42.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 10 monclient(hunting): picked mon.noname-b con 0x7fa9000a0000 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:42.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 10 monclient(hunting): picked mon.noname-a con 0x7fa9000a2800 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:42.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9000a3250 con 0x7fa9000a2800 2024-09-16T15:01:42.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa904a98640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa9000a2800 0x7fa90013d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39862/0 (socket says 172.21.15.26:39862) 2024-09-16T15:01:42.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa904a98640 1 -- 172.21.15.26:0/2805998843 learned_addr learned my addr 172.21.15.26:0/2805998843 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:42.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 1 -- 172.21.15.26:0/2805998843 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa90009f600 con 0x7fa9000a1d10 2024-09-16T15:01:42.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 1 -- 172.21.15.26:0/2805998843 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9000a3830 con 0x7fa9000a0000 2024-09-16T15:01:42.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:42.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 10 monclient(hunting): authenticate will time out at 2127.084329s 2024-09-16T15:01:42.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 1 -- 172.21.15.26:0/2805998843 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798349668 0 0) 0x7fa9000a3830 con 0x7fa9000a0000 2024-09-16T15:01:42.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:42.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:42.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:42.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 10 monclient(hunting): my global_id is 354101 2024-09-16T15:01:42.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 1 -- 172.21.15.26:0/2805998843 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e8003530 con 0x7fa9000a0000 2024-09-16T15:01:42.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 1 -- 172.21.15.26:0/2805998843 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 979282601 0 0) 0x7fa8e8003530 con 0x7fa9000a0000 2024-09-16T15:01:42.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 1 -- 172.21.15.26:0/2805998843 >> v1:172.21.15.189:6789/0 conn(0x7fa9000a1d10 legacy=0x7fa9000a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:42.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 1 -- 172.21.15.26:0/2805998843 >> v1:172.21.15.26:6789/0 conn(0x7fa9000a2800 legacy=0x7fa90013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:42.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 10 monclient: _finish_hunting 0 2024-09-16T15:01:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 1 monclient: found mon.noname-b 2024-09-16T15:01:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:01:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 1 -- 172.21.15.26:0/2805998843 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90009ed60 con 0x7fa9000a0000 2024-09-16T15:01:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 10 monclient: _finish_auth 0 2024-09-16T15:01:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 10 monclient: _check_auth_tickets 2024-09-16T15:01:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 1 -- 172.21.15.26:0/2805998843 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa8f4001620 con 0x7fa9000a0000 2024-09-16T15:01:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 5 monclient: authenticate success, global_id 354101 2024-09-16T15:01:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:42.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:01:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 10 monclient: dump: 2024-09-16T15:01:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:42.970 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:42.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:42.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 1 -- 172.21.15.26:0/2805998843 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8f40018d0 con 0x7fa9000a0000 2024-09-16T15:01:42.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:42.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 1 -- 172.21.15.26:0/2805998843 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa8f4001ca0 con 0x7fa9000a0000 2024-09-16T15:01:42.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:42.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 4 set_mon_vals no callback set 2024-09-16T15:01:42.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:42.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:42.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa8feffd640 10 monclient: dump: 2024-09-16T15:01:42.971 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:42.971 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:42.972 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:42.972 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:42.972 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:42.972 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:42.972 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:42.972 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:42.972 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:42.972 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:42.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 10 monclient: shutdown 2024-09-16T15:01:42.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:42.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 1 -- 172.21.15.26:0/2805998843 >> v1:172.21.15.26:6790/0 conn(0x7fa9000a0000 legacy=0x7fa900139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:42.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 1 -- 172.21.15.26:0/2805998843 shutdown_connections 2024-09-16T15:01:42.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 1 -- 172.21.15.26:0/2805998843 >> 172.21.15.26:0/2805998843 conn(0x7fa9000a7f30 msgr2=0x7fa9000a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:42.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 1 -- 172.21.15.26:0/2805998843 shutdown_connections 2024-09-16T15:01:42.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 1 -- 172.21.15.26:0/2805998843 wait complete. 2024-09-16T15:01:42.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 10 monclient: build_initial_monmap 2024-09-16T15:01:42.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.964+0000 7fa9064fb640 10 monclient: monmap: 2024-09-16T15:01:42.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:42.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:42.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:42.973 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:42.973 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:42.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 1 Processor -- start 2024-09-16T15:01:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 1 -- start start 2024-09-16T15:01:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 10 monclient: init 2024-09-16T15:01:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:01:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 10 monclient(hunting): picked mon.noname-c con 0x7fa9000a0000 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 10 monclient(hunting): picked mon.noname-b con 0x7fa9000a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 10 monclient(hunting): picked mon.noname-a con 0x7fa9000a2800 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:42.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa90013d130 con 0x7fa9000a2800 2024-09-16T15:01:42.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9000a67d0 con 0x7fa9000a1d10 2024-09-16T15:01:42.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa900139f50 con 0x7fa9000a0000 2024-09-16T15:01:42.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:42.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 10 monclient(hunting): authenticate will time out at 2127.088713s 2024-09-16T15:01:42.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa904a98640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa9000a2800 0x7fa9001d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39868/0 (socket says 172.21.15.26:39868) 2024-09-16T15:01:42.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa9000a0000 0x7fa900139130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:56624/0 (socket says 172.21.15.26:56624) 2024-09-16T15:01:42.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa904a98640 1 -- 172.21.15.26:0/666307326 learned_addr learned my addr 172.21.15.26:0/666307326 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:42.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615385853 0 0) 0x7fa900139f50 con 0x7fa9000a0000 2024-09-16T15:01:42.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:42.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:42.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:42.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient(hunting): my global_id is 354104 2024-09-16T15:01:42.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8ec003530 con 0x7fa9000a0000 2024-09-16T15:01:42.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 305354985 0 0) 0x7fa8ec003530 con 0x7fa9000a0000 2024-09-16T15:01:42.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa900139f50 con 0x7fa9000a0000 2024-09-16T15:01:42.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa8f0001870 con 0x7fa9000a0000 2024-09-16T15:01:42.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:42.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:01:42.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient(hunting): dump: 2024-09-16T15:01:42.978 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:42.978 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:42.979 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:42.979 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:42.979 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:42.979 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:42.979 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:42.979 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:42.979 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:42.979 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:42.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:42.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3432719182 0 0) 0x7fa90013d130 con 0x7fa9000a2800 2024-09-16T15:01:42.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:42.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:42.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:42.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient(hunting): my global_id is 334299 2024-09-16T15:01:42.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8ec003530 con 0x7fa9000a2800 2024-09-16T15:01:42.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 845354056 0 0) 0x7fa900139f50 con 0x7fa9000a0000 2024-09-16T15:01:42.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 >> v1:172.21.15.189:6789/0 conn(0x7fa9000a1d10 legacy=0x7fa900139840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:42.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 >> v1:172.21.15.26:6789/0 conn(0x7fa9000a2800 legacy=0x7fa9001d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:42.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient: _finish_hunting 0 2024-09-16T15:01:42.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 1 monclient: found mon.c 2024-09-16T15:01:42.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:42.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:42.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90013bc60 con 0x7fa9000a0000 2024-09-16T15:01:42.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient: _finish_auth 0 2024-09-16T15:01:42.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient: _check_auth_tickets 2024-09-16T15:01:42.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:42.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 5 monclient: authenticate success, global_id 354104 2024-09-16T15:01:42.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 10 monclient: _renew_subs 2024-09-16T15:01:42.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:42.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 1 -- 172.21.15.26:0/666307326 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90009e500 con 0x7fa9000a0000 2024-09-16T15:01:42.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 10 monclient: _renew_subs 2024-09-16T15:01:42.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:42.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 1 -- 172.21.15.26:0/666307326 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90013c890 con 0x7fa9000a0000 2024-09-16T15:01:42.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8f0001c60 con 0x7fa9000a0000 2024-09-16T15:01:42.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:42.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa8f0003af0 con 0x7fa9000a0000 2024-09-16T15:01:42.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:42.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:42.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fe7fc640 4 set_mon_vals no callback set 2024-09-16T15:01:42.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 10 monclient: dump: 2024-09-16T15:01:42.982 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:42.982 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:42.983 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:42.983 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:42.983 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:42.983 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:42.983 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:42.983 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:42.984 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:42.984 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:42.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa8f0003cd0 con 0x7fa9000a0000 2024-09-16T15:01:42.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:42.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:42.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:42.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.968+0000 7fa9064fb640 1 -- 172.21.15.26:0/666307326 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e0005050 con 0x7fa9000a0000 2024-09-16T15:01:42.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.972+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fa8f0043ea0 con 0x7fa9000a0000 2024-09-16T15:01:42.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.972+0000 7fa8ff7fe640 10 monclient: get_auth_request con 0x7fa8ec036180 auth_method 0 2024-09-16T15:01:42.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.976+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa8f0014550 con 0x7fa9000a0000 2024-09-16T15:01:42.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.976+0000 7fa8fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:42.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:42.976+0000 7fa8fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:43.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.200+0000 7fa9064fb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T15:01:43.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.200+0000 7fa9064fb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:43.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.200+0000 7fa9064fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:43.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.200+0000 7fa9064fb640 1 -- 172.21.15.26:0/666307326 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa8e0005880 con 0x7fa9000a0000 2024-09-16T15:01:43.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.200+0000 7fa8fcff9640 1 -- 172.21.15.26:0/666307326 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2646426558) 0x7fa8f001cdd0 con 0x7fa9000a0000 2024-09-16T15:01:43.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.200+0000 7fa8fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:43.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.200+0000 7fa8fcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:43.204 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":174,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T15:01:43.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.204+0000 7fa9064fb640 1 -- 172.21.15.26:0/666307326 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa8ec036180 legacy=0x7fa8ec038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.204+0000 7fa9064fb640 10 monclient: shutdown 2024-09-16T15:01:43.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.204+0000 7fa9064fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:43.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.204+0000 7fa9064fb640 1 -- 172.21.15.26:0/666307326 >> v1:172.21.15.26:6790/0 conn(0x7fa9000a0000 legacy=0x7fa900139130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.204+0000 7fa9064fb640 1 -- 172.21.15.26:0/666307326 shutdown_connections 2024-09-16T15:01:43.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.204+0000 7fa9064fb640 1 -- 172.21.15.26:0/666307326 >> 172.21.15.26:0/666307326 conn(0x7fa9000a7f30 msgr2=0x7fa90009b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:43.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.204+0000 7fa9064fb640 1 -- 172.21.15.26:0/666307326 shutdown_connections 2024-09-16T15:01:43.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.204+0000 7fa9064fb640 1 -- 172.21.15.26:0/666307326 wait complete. 2024-09-16T15:01:43.222 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T15:01:43.222 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T15:01:43.222 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T15:01:43.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.348+0000 7f52dfbb4640 10 monclient: get_monmap_and_config 2024-09-16T15:01:43.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.348+0000 7f52dfbb4640 10 monclient: build_initial_monmap 2024-09-16T15:01:43.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.348+0000 7f52dfbb4640 10 monclient: monmap: 2024-09-16T15:01:43.350 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:43.350 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.350 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:43.352026+0000 2024-09-16T15:01:43.350 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:43.352026+0000 2024-09-16T15:01:43.350 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:43.350 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.350 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:43.351 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:43.351 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:43.351 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.348+0000 7f52dfbb4640 1 Processor -- start 2024-09-16T15:01:43.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.348+0000 7f52dfbb4640 1 -- start start 2024-09-16T15:01:43.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.348+0000 7f52dfbb4640 10 monclient: init 2024-09-16T15:01:43.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.348+0000 7f52dfbb4640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:43.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:01:43.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient(hunting): picked mon.noname-b con 0x7f52d809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:43.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient(hunting): picked mon.noname-a con 0x7f52d8132650 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:43.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient(hunting): picked mon.noname-c con 0x7f52d8138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:43.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d809d260 con 0x7f52d8132650 2024-09-16T15:01:43.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d8132a80 con 0x7f52d8138ff0 2024-09-16T15:01:43.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d809cf60 con 0x7f52d809cb80 2024-09-16T15:01:43.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:43.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient(hunting): authenticate will time out at 2127.472138s 2024-09-16T15:01:43.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dd14f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f52d8132650 0x7f52d81378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39882/0 (socket says 172.21.15.26:39882) 2024-09-16T15:01:43.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dd14f640 1 -- 172.21.15.26:0/95249393 learned_addr learned my addr 172.21.15.26:0/95249393 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:43.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 1 -- 172.21.15.26:0/95249393 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512092839 0 0) 0x7f52d809d260 con 0x7f52d8132650 2024-09-16T15:01:43.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient(hunting): my global_id is 334302 2024-09-16T15:01:43.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 1 -- 172.21.15.26:0/95249393 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52c4003530 con 0x7f52d8132650 2024-09-16T15:01:43.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 1 -- 172.21.15.26:0/95249393 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177918568 0 0) 0x7f52d809cf60 con 0x7f52d809cb80 2024-09-16T15:01:43.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient(hunting): my global_id is 354107 2024-09-16T15:01:43.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 1 -- 172.21.15.26:0/95249393 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d809d260 con 0x7f52d809cb80 2024-09-16T15:01:43.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 1 -- 172.21.15.26:0/95249393 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 902506800 0 0) 0x7f52c4003530 con 0x7f52d8132650 2024-09-16T15:01:43.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 1 -- 172.21.15.26:0/95249393 >> v1:172.21.15.26:6790/0 conn(0x7f52d809cb80 legacy=0x7f52d80717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 1 -- 172.21.15.26:0/95249393 >> v1:172.21.15.189:6789/0 conn(0x7f52d8138ff0 legacy=0x7f52d813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient: _finish_hunting 0 2024-09-16T15:01:43.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 1 monclient: found mon.noname-a 2024-09-16T15:01:43.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:43.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:01:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 1 -- 172.21.15.26:0/95249393 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d809f2d0 con 0x7f52d8132650 2024-09-16T15:01:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient: _finish_auth 0 2024-09-16T15:01:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient: _check_auth_tickets 2024-09-16T15:01:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 1 -- 172.21.15.26:0/95249393 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52c8001620 con 0x7f52d8132650 2024-09-16T15:01:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient: dump: 2024-09-16T15:01:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.361 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.362 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 5 monclient: authenticate success, global_id 334302 2024-09-16T15:01:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 1 -- 172.21.15.26:0/95249393 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52c8001810 con 0x7f52d8132650 2024-09-16T15:01:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 1 -- 172.21.15.26:0/95249393 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52c8001be0 con 0x7f52d8132650 2024-09-16T15:01:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 4 set_mon_vals no callback set 2024-09-16T15:01:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dc94e640 10 monclient: dump: 2024-09-16T15:01:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.363 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient: shutdown 2024-09-16T15:01:43.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 1 -- 172.21.15.26:0/95249393 >> v1:172.21.15.26:6789/0 conn(0x7f52d8132650 legacy=0x7f52d81378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 1 -- 172.21.15.26:0/95249393 shutdown_connections 2024-09-16T15:01:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 1 -- 172.21.15.26:0/95249393 >> 172.21.15.26:0/95249393 conn(0x7f52d809fb50 msgr2=0x7f52d809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 1 -- 172.21.15.26:0/95249393 shutdown_connections 2024-09-16T15:01:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 1 -- 172.21.15.26:0/95249393 wait complete. 2024-09-16T15:01:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient: build_initial_monmap 2024-09-16T15:01:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient: monmap: 2024-09-16T15:01:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:43.364 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:43.365 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 1 Processor -- start 2024-09-16T15:01:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 1 -- start start 2024-09-16T15:01:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient: init 2024-09-16T15:01:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:01:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient(hunting): picked mon.noname-b con 0x7f52d809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient(hunting): picked mon.noname-a con 0x7f52d8132650 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient(hunting): picked mon.noname-c con 0x7f52d8138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d8134c60 con 0x7f52d8132650 2024-09-16T15:01:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d8138840 con 0x7f52d809cb80 2024-09-16T15:01:43.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d813c700 con 0x7f52d8138ff0 2024-09-16T15:01:43.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:43.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dfbb4640 10 monclient(hunting): authenticate will time out at 2127.475337s 2024-09-16T15:01:43.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dd14f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f52d8132650 0x7f52d812dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39884/0 (socket says 172.21.15.26:39884) 2024-09-16T15:01:43.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52dd14f640 1 -- 172.21.15.26:0/3716420441 learned_addr learned my addr 172.21.15.26:0/3716420441 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:43.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579183005 0 0) 0x7f52d8134c60 con 0x7f52d8132650 2024-09-16T15:01:43.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52ce7fc640 10 monclient(hunting): my global_id is 334305 2024-09-16T15:01:43.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52bc003560 con 0x7f52d8132650 2024-09-16T15:01:43.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117413546 0 0) 0x7f52d813c700 con 0x7f52d8138ff0 2024-09-16T15:01:43.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52ce7fc640 10 monclient(hunting): my global_id is 354110 2024-09-16T15:01:43.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d8134c60 con 0x7f52d8138ff0 2024-09-16T15:01:43.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3414882516 0 0) 0x7f52bc003560 con 0x7f52d8132650 2024-09-16T15:01:43.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52d813c700 con 0x7f52d8132650 2024-09-16T15:01:43.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52c8001520 con 0x7f52d8132650 2024-09-16T15:01:43.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.352+0000 7f52ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:43.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient(hunting): dump: 2024-09-16T15:01:43.367 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.367 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.367 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.367 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.368 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.368 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.368 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.368 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.368 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.368 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:43.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1047326762 0 0) 0x7f52d8134c60 con 0x7f52d8138ff0 2024-09-16T15:01:43.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52bc003560 con 0x7f52d8138ff0 2024-09-16T15:01:43.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52d40017e0 con 0x7f52d8138ff0 2024-09-16T15:01:43.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:43.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient(hunting): dump: 2024-09-16T15:01:43.369 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.369 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.369 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.369 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.369 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.369 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.369 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.369 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.369 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.369 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282331434 0 0) 0x7f52d8138840 con 0x7f52d809cb80 2024-09-16T15:01:43.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient(hunting): my global_id is 344215 2024-09-16T15:01:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d8134c60 con 0x7f52d809cb80 2024-09-16T15:01:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2989419272 0 0) 0x7f52d813c700 con 0x7f52d8132650 2024-09-16T15:01:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 >> v1:172.21.15.26:6790/0 conn(0x7f52d8138ff0 legacy=0x7f52d812e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 >> v1:172.21.15.189:6789/0 conn(0x7f52d809cb80 legacy=0x7f52d812d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient: _finish_hunting 0 2024-09-16T15:01:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 monclient: found mon.a 2024-09-16T15:01:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d809a240 con 0x7f52d8132650 2024-09-16T15:01:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient: _finish_auth 0 2024-09-16T15:01:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient: _check_auth_tickets 2024-09-16T15:01:43.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52dfbb4640 5 monclient: authenticate success, global_id 334305 2024-09-16T15:01:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52dfbb4640 10 monclient: _renew_subs 2024-09-16T15:01:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52dfbb4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52dfbb4640 1 -- 172.21.15.26:0/3716420441 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d8134980 con 0x7f52d8132650 2024-09-16T15:01:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52dfbb4640 10 monclient: _renew_subs 2024-09-16T15:01:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52dfbb4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52dfbb4640 1 -- 172.21.15.26:0/3716420441 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d812f2b0 con 0x7f52d8132650 2024-09-16T15:01:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52c8001180 con 0x7f52d8132650 2024-09-16T15:01:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52c8004cc0 con 0x7f52d8132650 2024-09-16T15:01:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52cffff640 4 set_mon_vals no callback set 2024-09-16T15:01:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient: dump: 2024-09-16T15:01:43.371 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.372 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.372 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.372 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.372 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.372 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.372 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.372 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.372 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.372 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52dfbb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:43.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52dfbb4640 1 -- 172.21.15.26:0/3716420441 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f52c0006590 con 0x7f52c0002830 2024-09-16T15:01:43.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f52c8004e20 con 0x7f52d8132650 2024-09-16T15:01:43.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f52c8043ee0 con 0x7f52d8132650 2024-09-16T15:01:43.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914754029 0 0) 0x7f52c0006590 con 0x7f52c0002830 2024-09-16T15:01:43.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient(hunting): my global_id is 334308 2024-09-16T15:01:43.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d813c700 con 0x7f52c0002830 2024-09-16T15:01:43.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 933715357 0 0) 0x7f52d813c700 con 0x7f52c0002830 2024-09-16T15:01:43.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52c0006590 con 0x7f52c0002830 2024-09-16T15:01:43.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52d0001a30 con 0x7f52c0002830 2024-09-16T15:01:43.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52de151640 10 monclient: get_auth_request con 0x7f52bc036470 auth_method 0 2024-09-16T15:01:43.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:43.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient: dump: 2024-09-16T15:01:43.373 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.374 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1930961212 0 0) 0x7f52c0006590 con 0x7f52c0002830 2024-09-16T15:01:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52c000a0e0 con 0x7f52c0002830 2024-09-16T15:01:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f52d0001d40 con 0x7f52c0002830 2024-09-16T15:01:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:43.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:43.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 >> v1:172.21.15.26:6789/0 conn(0x7f52c0002830 legacy=0x7f52c0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.356+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 >> v1:172.21.15.26:6789/0 conn(0x7f52c0002830 legacy=0x7f52c0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:43.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52dfbb4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:01:43.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52dfbb4640 1 -- 172.21.15.26:0/3716420441 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f52c0009da0 con 0x7f52c00090c0 2024-09-16T15:01:43.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029508334 0 0) 0x7f52c0009da0 con 0x7f52c00090c0 2024-09-16T15:01:43.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 10 monclient(hunting): my global_id is 334311 2024-09-16T15:01:43.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52c0006590 con 0x7f52c00090c0 2024-09-16T15:01:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 264498897 0 0) 0x7f52c0006590 con 0x7f52c00090c0 2024-09-16T15:01:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52c0009da0 con 0x7f52c00090c0 2024-09-16T15:01:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f52d0003b20 con 0x7f52c00090c0 2024-09-16T15:01:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 10 monclient: dump: 2024-09-16T15:01:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.377 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.378 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.378 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3888113749 0 0) 0x7f52c0009da0 con 0x7f52c00090c0 2024-09-16T15:01:43.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f52c000c290 con 0x7f52c00090c0 2024-09-16T15:01:43.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 323968611) 0x7f52d0004fe0 con 0x7f52c00090c0 2024-09-16T15:01:43.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:01:43.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:43.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 >> v1:172.21.15.26:6789/0 conn(0x7f52c00090c0 legacy=0x7f52c0009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52ce7fc640 1 -- 172.21.15.26:0/3716420441 >> v1:172.21.15.26:6789/0 conn(0x7f52c00090c0 legacy=0x7f52c0009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:43.378 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:01:43.379 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:43.379 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:43.379 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:01:43.379 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 174, 2024-09-16T15:01:43.379 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:01:43.379 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:01:43.379 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:01:43.379 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:01:43.379 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:43.379 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:01:43.379 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:43.379 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:01:43.379 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:01:43.380 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:43.380 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:43.380 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:43.380 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:43.380 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:43.380 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:43.380 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:43.380 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:43.380 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:43.380 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:43.380 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:43.380 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:01:43.380 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:01:43.380 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:43.381 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:43.381 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:43.381 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:43.381 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:43.381 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:43.381 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:43.381 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:43.381 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:43.381 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:43.381 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.382 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:01:43.382 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:01:43.382 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:01:43.382 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:01:43.382 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:01:43.382 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:01:43.382 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:43.382 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:43.382 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:01:43.382 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:01:43.382 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:01:43.382 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:01:43.383 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:01:43.383 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:01:43.383 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:01:43.383 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:43.383 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:01:43.383 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:43.383 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:43.383 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:43.384 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:43.384 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:43.384 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:43.384 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:43.384 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:43.384 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:43.384 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:43.384 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:43.384 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:01:43.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.384 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:01:43.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.385 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:43.385 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:43.385 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:43.385 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:43.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.385 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:43.385 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:01:43.385 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:43.385 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.385 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.386 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:01:43.386 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:01:43.386 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:43.386 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:43.386 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:43.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.386 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:43.386 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:43.386 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:43.386 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:43.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.386 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:43.387 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:01:43.387 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:43.387 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.387 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.387 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:01:43.387 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:01:43.387 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:43.387 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:43.387 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:43.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.387 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:43.388 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:43.388 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:43.388 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:43.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.388 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:43.388 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:01:43.388 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:43.388 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.388 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.388 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:01:43.388 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:01:43.388 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:43.388 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:43.389 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:43.389 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.389 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.389 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:01:43.389 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:01:43.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.389 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:43.389 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:43.389 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:43.389 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.389 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:43.389 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T15:01:43.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.389 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:43.390 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:43.390 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T15:01:43.390 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.390 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:43.390 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:01:43.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.390 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:43.390 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:43.390 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T15:01:43.390 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.390 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.391 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:01:43.391 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:01:43.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52dfbb4640 1 -- 172.21.15.26:0/3716420441 >> v1:172.21.15.189:6812/3196161582 conn(0x7f52bc036470 legacy=0x7f52bc038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52dfbb4640 10 monclient: shutdown 2024-09-16T15:01:43.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52dfbb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:43.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52dfbb4640 1 -- 172.21.15.26:0/3716420441 >> v1:172.21.15.26:6789/0 conn(0x7f52d8132650 legacy=0x7f52d812dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52de151640 1 -- 172.21.15.26:0/3716420441 reap_dead start 2024-09-16T15:01:43.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52de151640 1 -- 172.21.15.26:0/3716420441 reap_dead start 2024-09-16T15:01:43.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52dfbb4640 1 -- 172.21.15.26:0/3716420441 shutdown_connections 2024-09-16T15:01:43.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52dfbb4640 1 -- 172.21.15.26:0/3716420441 >> 172.21.15.26:0/3716420441 conn(0x7f52d809fb50 msgr2=0x7f52d809bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:43.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52dfbb4640 1 -- 172.21.15.26:0/3716420441 shutdown_connections 2024-09-16T15:01:43.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.364+0000 7f52dfbb4640 1 -- 172.21.15.26:0/3716420441 wait complete. 2024-09-16T15:01:43.393 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:01:43.393 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T15:01:43.393 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T15:01:43.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f95638b2640 10 monclient: get_monmap_and_config 2024-09-16T15:01:43.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f95638b2640 10 monclient: build_initial_monmap 2024-09-16T15:01:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f95638b2640 10 monclient: monmap: 2024-09-16T15:01:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:43.511788+0000 2024-09-16T15:01:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:43.511788+0000 2024-09-16T15:01:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:43.510 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f95638b2640 1 Processor -- start 2024-09-16T15:01:43.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f95638b2640 1 -- start start 2024-09-16T15:01:43.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f95638b2640 10 monclient: init 2024-09-16T15:01:43.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f95638b2640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:43.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f95638b2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:01:43.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f95638b2640 10 monclient(hunting): picked mon.noname-a con 0x7f955c09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:43.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f95638b2640 10 monclient(hunting): picked mon.noname-b con 0x7f955c132650 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:43.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f95638b2640 10 monclient(hunting): picked mon.noname-c con 0x7f955c138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:43.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f95638b2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955c09d260 con 0x7f955c09cb80 2024-09-16T15:01:43.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f95638b2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955c132a80 con 0x7f955c138ff0 2024-09-16T15:01:43.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f95638b2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955c09cf60 con 0x7f955c132650 2024-09-16T15:01:43.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f95638b2640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:43.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f95638b2640 10 monclient(hunting): authenticate will time out at 2127.631896s 2024-09-16T15:01:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.508+0000 7f956164e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f955c09cb80 0x7f955c0717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39912/0 (socket says 172.21.15.26:39912) 2024-09-16T15:01:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f956164e640 1 -- 172.21.15.26:0/3019654251 learned_addr learned my addr 172.21.15.26:0/3019654251 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 1 -- 172.21.15.26:0/3019654251 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258828084 0 0) 0x7f955c09d260 con 0x7f955c09cb80 2024-09-16T15:01:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient(hunting): my global_id is 334314 2024-09-16T15:01:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 1 -- 172.21.15.26:0/3019654251 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9548003530 con 0x7f955c09cb80 2024-09-16T15:01:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 1 -- 172.21.15.26:0/3019654251 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775872183 0 0) 0x7f955c09cf60 con 0x7f955c132650 2024-09-16T15:01:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient(hunting): my global_id is 354113 2024-09-16T15:01:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 1 -- 172.21.15.26:0/3019654251 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f955c09d260 con 0x7f955c132650 2024-09-16T15:01:43.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 1 -- 172.21.15.26:0/3019654251 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3375901451 0 0) 0x7f9548003530 con 0x7f955c09cb80 2024-09-16T15:01:43.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 1 -- 172.21.15.26:0/3019654251 >> v1:172.21.15.26:6790/0 conn(0x7f955c132650 legacy=0x7f955c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 1 -- 172.21.15.26:0/3019654251 >> v1:172.21.15.189:6789/0 conn(0x7f955c138ff0 legacy=0x7f955c13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient: _finish_hunting 0 2024-09-16T15:01:43.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 1 monclient: found mon.noname-a 2024-09-16T15:01:43.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:43.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:01:43.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 1 -- 172.21.15.26:0/3019654251 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955c09f2d0 con 0x7f955c09cb80 2024-09-16T15:01:43.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient: _finish_auth 0 2024-09-16T15:01:43.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient: _check_auth_tickets 2024-09-16T15:01:43.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:43.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 1 -- 172.21.15.26:0/3019654251 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9558001620 con 0x7f955c09cb80 2024-09-16T15:01:43.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 5 monclient: authenticate success, global_id 334314 2024-09-16T15:01:43.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:43.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:43.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient: dump: 2024-09-16T15:01:43.522 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.522 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.522 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.522 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.522 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.522 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.522 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.522 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.523 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.523 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:43.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 1 -- 172.21.15.26:0/3019654251 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9558001840 con 0x7f955c09cb80 2024-09-16T15:01:43.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:43.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 1 -- 172.21.15.26:0/3019654251 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9558001b90 con 0x7f955c09cb80 2024-09-16T15:01:43.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:43.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 4 set_mon_vals no callback set 2024-09-16T15:01:43.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:43.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9553fff640 10 monclient: dump: 2024-09-16T15:01:43.523 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.524 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.524 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.524 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.524 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.524 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.524 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.524 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.524 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.524 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 10 monclient: shutdown 2024-09-16T15:01:43.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:43.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 1 -- 172.21.15.26:0/3019654251 >> v1:172.21.15.26:6789/0 conn(0x7f955c09cb80 legacy=0x7f955c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 1 -- 172.21.15.26:0/3019654251 shutdown_connections 2024-09-16T15:01:43.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 1 -- 172.21.15.26:0/3019654251 >> 172.21.15.26:0/3019654251 conn(0x7f955c09fb50 msgr2=0x7f955c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:43.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 1 -- 172.21.15.26:0/3019654251 shutdown_connections 2024-09-16T15:01:43.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 1 -- 172.21.15.26:0/3019654251 wait complete. 2024-09-16T15:01:43.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 10 monclient: build_initial_monmap 2024-09-16T15:01:43.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 10 monclient: monmap: 2024-09-16T15:01:43.525 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:43.525 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:43.525 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:43.526 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:43.526 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:43.526 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:43.526 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:43.526 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:43.526 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:43.526 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 1 Processor -- start 2024-09-16T15:01:43.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 1 -- start start 2024-09-16T15:01:43.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 10 monclient: init 2024-09-16T15:01:43.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:43.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:01:43.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 10 monclient(hunting): picked mon.noname-a con 0x7f955c09cb80 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 10 monclient(hunting): picked mon.noname-c con 0x7f955c132650 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 10 monclient(hunting): picked mon.noname-b con 0x7f955c138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955c071460 con 0x7f955c09cb80 2024-09-16T15:01:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955c071a30 con 0x7f955c138ff0 2024-09-16T15:01:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955c071180 con 0x7f955c132650 2024-09-16T15:01:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 10 monclient(hunting): authenticate will time out at 2127.634918s 2024-09-16T15:01:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f956164e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f955c09cb80 0x7f955c06d500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39918/0 (socket says 172.21.15.26:39918) 2024-09-16T15:01:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f956164e640 1 -- 172.21.15.26:0/4150652524 learned_addr learned my addr 172.21.15.26:0/4150652524 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9560e4d640 1 --1- 172.21.15.26:0/4150652524 >> v1:172.21.15.26:6790/0 conn(0x7f955c132650 0x7f955c06dc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:56672/0 (socket says 172.21.15.26:56672) 2024-09-16T15:01:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437441522 0 0) 0x7f955c071180 con 0x7f955c132650 2024-09-16T15:01:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): my global_id is 354116 2024-09-16T15:01:43.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9540003530 con 0x7f955c132650 2024-09-16T15:01:43.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770074308 0 0) 0x7f955c071460 con 0x7f955c09cb80 2024-09-16T15:01:43.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): my global_id is 334317 2024-09-16T15:01:43.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f955c071180 con 0x7f955c09cb80 2024-09-16T15:01:43.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3659516675 0 0) 0x7f9540003530 con 0x7f955c132650 2024-09-16T15:01:43.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f955c071460 con 0x7f955c132650 2024-09-16T15:01:43.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f954c001680 con 0x7f955c132650 2024-09-16T15:01:43.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:01:43.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): dump: 2024-09-16T15:01:43.529 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.529 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.529 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.529 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.529 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.529 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.529 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.529 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.529 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.530 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:43.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 456477763 0 0) 0x7f955c071180 con 0x7f955c09cb80 2024-09-16T15:01:43.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9540003530 con 0x7f955c09cb80 2024-09-16T15:01:43.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9558001520 con 0x7f955c09cb80 2024-09-16T15:01:43.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:43.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): dump: 2024-09-16T15:01:43.530 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.530 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.530 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.530 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.531 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.531 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.531 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.531 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.531 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.531 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737896339 0 0) 0x7f955c071a30 con 0x7f955c138ff0 2024-09-16T15:01:43.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient(hunting): my global_id is 344221 2024-09-16T15:01:43.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f955c071180 con 0x7f955c138ff0 2024-09-16T15:01:43.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 586417686 0 0) 0x7f955c071460 con 0x7f955c132650 2024-09-16T15:01:43.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 >> v1:172.21.15.189:6789/0 conn(0x7f955c138ff0 legacy=0x7f955c1cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 >> v1:172.21.15.26:6789/0 conn(0x7f955c09cb80 legacy=0x7f955c06d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient: _finish_hunting 0 2024-09-16T15:01:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 monclient: found mon.c 2024-09-16T15:01:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955c06e320 con 0x7f955c132650 2024-09-16T15:01:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient: _finish_auth 0 2024-09-16T15:01:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f9551ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 5 monclient: authenticate success, global_id 354116 2024-09-16T15:01:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 10 monclient: _renew_subs 2024-09-16T15:01:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 1 -- 172.21.15.26:0/4150652524 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955c06f100 con 0x7f955c132650 2024-09-16T15:01:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 10 monclient: _renew_subs 2024-09-16T15:01:43.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:43.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.512+0000 7f95638b2640 1 -- 172.21.15.26:0/4150652524 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f955c0a2040 con 0x7f955c132650 2024-09-16T15:01:43.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f954c003600 con 0x7f955c132650 2024-09-16T15:01:43.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:43.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f954c003990 con 0x7f955c132650 2024-09-16T15:01:43.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:43.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 10 monclient: dump: 2024-09-16T15:01:43.534 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.534 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.534 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.535 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.535 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.535 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.535 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.535 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.535 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.536 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f95537fe640 4 set_mon_vals no callback set 2024-09-16T15:01:43.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f954c003b70 con 0x7f955c132650 2024-09-16T15:01:43.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f95638b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:43.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f95638b2640 1 -- 172.21.15.26:0/4150652524 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f953c006590 con 0x7f953c002830 2024-09-16T15:01:43.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f954c043cf0 con 0x7f955c132650 2024-09-16T15:01:43.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301329971 0 0) 0x7f953c006590 con 0x7f953c002830 2024-09-16T15:01:43.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 10 monclient(hunting): my global_id is 354119 2024-09-16T15:01:43.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f955c071460 con 0x7f953c002830 2024-09-16T15:01:43.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9561e4f640 10 monclient: get_auth_request con 0x7f9540036340 auth_method 0 2024-09-16T15:01:43.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2526725343 0 0) 0x7f955c071460 con 0x7f953c002830 2024-09-16T15:01:43.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f953c006590 con 0x7f953c002830 2024-09-16T15:01:43.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9558000fe0 con 0x7f953c002830 2024-09-16T15:01:43.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:43.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 10 monclient: dump: 2024-09-16T15:01:43.541 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.541 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.541 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.541 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.541 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.541 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.541 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.541 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.541 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.541 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3391353675 0 0) 0x7f953c006590 con 0x7f953c002830 2024-09-16T15:01:43.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f953c00a0e0 con 0x7f953c002830 2024-09-16T15:01:43.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f95580070b0 con 0x7f953c002830 2024-09-16T15:01:43.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:43.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:43.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 >> v1:172.21.15.26:6790/0 conn(0x7f953c002830 legacy=0x7f953c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.516+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 >> v1:172.21.15.26:6790/0 conn(0x7f953c002830 legacy=0x7f953c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:43.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f95638b2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:01:43.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f95638b2640 1 -- 172.21.15.26:0/4150652524 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f953c009da0 con 0x7f953c0090c0 2024-09-16T15:01:43.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100643029 0 0) 0x7f953c009da0 con 0x7f953c0090c0 2024-09-16T15:01:43.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 10 monclient(hunting): my global_id is 354122 2024-09-16T15:01:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f953c006590 con 0x7f953c0090c0 2024-09-16T15:01:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2235746685 0 0) 0x7f953c006590 con 0x7f953c0090c0 2024-09-16T15:01:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f953c009da0 con 0x7f953c0090c0 2024-09-16T15:01:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9558005530 con 0x7f953c0090c0 2024-09-16T15:01:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 10 monclient: dump: 2024-09-16T15:01:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.544 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.545 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.545 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.545 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.545 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.545 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.545 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.545 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.545 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2526791094 0 0) 0x7f953c009da0 con 0x7f953c0090c0 2024-09-16T15:01:43.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f953c00c290 con 0x7f953c0090c0 2024-09-16T15:01:43.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 3892994192) 0x7f95580068a0 con 0x7f953c0090c0 2024-09-16T15:01:43.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:01:43.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:43.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 >> v1:172.21.15.26:6790/0 conn(0x7f953c0090c0 legacy=0x7f953c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9551ffb640 1 -- 172.21.15.26:0/4150652524 >> v1:172.21.15.26:6790/0 conn(0x7f953c0090c0 legacy=0x7f953c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:43.546 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:01:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T15:01:43.546 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 174, 2024-09-16T15:01:43.547 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:01:43.547 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:01:43.547 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:01:43.547 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:01:43.547 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:43.547 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:01:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:01:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:01:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:43.548 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:01:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:01:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.549 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:01:43.550 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:01:43.550 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:01:43.550 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:01:43.550 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:01:43.550 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:01:43.550 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:43.550 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:43.550 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:01:43.550 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:01:43.550 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:01:43.550 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:01:43.550 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:01:43.550 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:01:43.550 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:01:43.551 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:43.551 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:01:43.551 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:43.551 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:43.551 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:43.551 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:43.551 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:43.551 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:43.551 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:43.551 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:43.551 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:43.551 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:43.552 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:43.552 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:01:43.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.552 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:01:43.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.552 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:43.552 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:43.552 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:43.552 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:43.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.552 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:43.552 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:01:43.553 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:43.553 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.553 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.553 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:01:43.553 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:01:43.553 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:43.553 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:43.553 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:43.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.553 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:43.553 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:43.554 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:43.554 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:43.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.554 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:43.554 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:01:43.554 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:43.554 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.554 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.554 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:01:43.554 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:01:43.554 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:43.554 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:43.554 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:43.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.555 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:43.555 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:43.555 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:43.555 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:43.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.555 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:43.555 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:01:43.555 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:43.555 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.555 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.555 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:01:43.556 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:01:43.556 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:43.556 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:43.557 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:43.558 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.558 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.558 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:01:43.558 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:01:43.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.559 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:43.559 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:43.559 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:43.559 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.559 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:43.559 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:01:43.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.560 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:43.560 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:43.560 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T15:01:43.560 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.560 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.561 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:01:43.561 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:01:43.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f95638b2640 1 -- 172.21.15.26:0/4150652524 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9540036340 legacy=0x7f9540038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f95638b2640 10 monclient: shutdown 2024-09-16T15:01:43.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f95638b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:43.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f95638b2640 1 -- 172.21.15.26:0/4150652524 >> v1:172.21.15.26:6790/0 conn(0x7f955c132650 legacy=0x7f955c06dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9561e4f640 1 -- 172.21.15.26:0/4150652524 reap_dead start 2024-09-16T15:01:43.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f9561e4f640 1 -- 172.21.15.26:0/4150652524 reap_dead start 2024-09-16T15:01:43.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f95638b2640 1 -- 172.21.15.26:0/4150652524 shutdown_connections 2024-09-16T15:01:43.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f95638b2640 1 -- 172.21.15.26:0/4150652524 >> 172.21.15.26:0/4150652524 conn(0x7f955c09fb50 msgr2=0x7f955c131a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:43.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f95638b2640 1 -- 172.21.15.26:0/4150652524 shutdown_connections 2024-09-16T15:01:43.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.524+0000 7f95638b2640 1 -- 172.21.15.26:0/4150652524 wait complete. 2024-09-16T15:01:43.563 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:01:43.563 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T15:01:43.563 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T15:01:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0683da0640 10 monclient: get_monmap_and_config 2024-09-16T15:01:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0683da0640 10 monclient: build_initial_monmap 2024-09-16T15:01:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0683da0640 10 monclient: monmap: 2024-09-16T15:01:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:43.682942+0000 2024-09-16T15:01:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:43.682942+0000 2024-09-16T15:01:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:43.684 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0683da0640 1 Processor -- start 2024-09-16T15:01:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0683da0640 1 -- start start 2024-09-16T15:01:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0683da0640 10 monclient: init 2024-09-16T15:01:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0683da0640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0683da0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:01:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0683da0640 10 monclient(hunting): picked mon.noname-b con 0x7f067c09cb80 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0683da0640 10 monclient(hunting): picked mon.noname-c con 0x7f067c132650 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0683da0640 10 monclient(hunting): picked mon.noname-a con 0x7f067c138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0683da0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c09d260 con 0x7f067c138ff0 2024-09-16T15:01:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0683da0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c132a80 con 0x7f067c132650 2024-09-16T15:01:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0683da0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c09cf60 con 0x7f067c09cb80 2024-09-16T15:01:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0683da0640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0683da0640 10 monclient(hunting): authenticate will time out at 2127.803063s 2024-09-16T15:01:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f068233d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f067c138ff0 0x7f067c13b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39922/0 (socket says 172.21.15.26:39922) 2024-09-16T15:01:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f068233d640 1 -- 172.21.15.26:0/2453642573 learned_addr learned my addr 172.21.15.26:0/2453642573 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 1 -- 172.21.15.26:0/2453642573 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899269329 0 0) 0x7f067c09cf60 con 0x7f067c09cb80 2024-09-16T15:01:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 10 monclient(hunting): my global_id is 354125 2024-09-16T15:01:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 1 -- 172.21.15.26:0/2453642573 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0668003530 con 0x7f067c09cb80 2024-09-16T15:01:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 1 -- 172.21.15.26:0/2453642573 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159195097 0 0) 0x7f067c09d260 con 0x7f067c138ff0 2024-09-16T15:01:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 10 monclient(hunting): my global_id is 334320 2024-09-16T15:01:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 1 -- 172.21.15.26:0/2453642573 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f067c09cf60 con 0x7f067c138ff0 2024-09-16T15:01:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 1 -- 172.21.15.26:0/2453642573 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1580057367 0 0) 0x7f067c09cf60 con 0x7f067c138ff0 2024-09-16T15:01:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 1 -- 172.21.15.26:0/2453642573 >> v1:172.21.15.26:6790/0 conn(0x7f067c09cb80 legacy=0x7f067c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 1 -- 172.21.15.26:0/2453642573 >> v1:172.21.15.189:6789/0 conn(0x7f067c132650 legacy=0x7f067c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 10 monclient: _finish_hunting 0 2024-09-16T15:01:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 1 monclient: found mon.noname-a 2024-09-16T15:01:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:01:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 1 -- 172.21.15.26:0/2453642573 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c09f2d0 con 0x7f067c138ff0 2024-09-16T15:01:43.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 10 monclient: _finish_auth 0 2024-09-16T15:01:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 10 monclient: _check_auth_tickets 2024-09-16T15:01:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 1 -- 172.21.15.26:0/2453642573 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0674001980 con 0x7f067c138ff0 2024-09-16T15:01:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.680+0000 7f0680b3a640 10 monclient: dump: 2024-09-16T15:01:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.709 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.710 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.711 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 5 monclient: authenticate success, global_id 334320 2024-09-16T15:01:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:43.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0680b3a640 1 -- 172.21.15.26:0/2453642573 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0674001bb0 con 0x7f067c138ff0 2024-09-16T15:01:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0680b3a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0680b3a640 1 -- 172.21.15.26:0/2453642573 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0674001f80 con 0x7f067c138ff0 2024-09-16T15:01:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 4 set_mon_vals no callback set 2024-09-16T15:01:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0680b3a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0680b3a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:43.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0680b3a640 10 monclient: dump: 2024-09-16T15:01:43.713 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.713 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.713 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.714 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.715 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.715 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 monclient: shutdown 2024-09-16T15:01:43.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:43.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 1 -- 172.21.15.26:0/2453642573 >> v1:172.21.15.26:6789/0 conn(0x7f067c138ff0 legacy=0x7f067c13b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 1 -- 172.21.15.26:0/2453642573 shutdown_connections 2024-09-16T15:01:43.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 1 -- 172.21.15.26:0/2453642573 >> 172.21.15.26:0/2453642573 conn(0x7f067c09fb50 msgr2=0x7f067c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:43.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 1 -- 172.21.15.26:0/2453642573 shutdown_connections 2024-09-16T15:01:43.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 1 -- 172.21.15.26:0/2453642573 wait complete. 2024-09-16T15:01:43.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 monclient: build_initial_monmap 2024-09-16T15:01:43.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 monclient: monmap: 2024-09-16T15:01:43.717 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:43.717 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:43.717 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:43.718 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:43.718 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:43.718 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:43.718 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:43.718 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:43.718 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:43.719 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 1 Processor -- start 2024-09-16T15:01:43.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 1 -- start start 2024-09-16T15:01:43.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 monclient: init 2024-09-16T15:01:43.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:43.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:01:43.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 monclient(hunting): picked mon.noname-b con 0x7f067c09cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:43.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 monclient(hunting): picked mon.noname-c con 0x7f067c132650 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:43.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 monclient(hunting): picked mon.noname-a con 0x7f067c138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:43.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c12e2d0 con 0x7f067c138ff0 2024-09-16T15:01:43.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c13c720 con 0x7f067c09cb80 2024-09-16T15:01:43.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c09a240 con 0x7f067c132650 2024-09-16T15:01:43.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:43.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 monclient(hunting): authenticate will time out at 2127.806120s 2024-09-16T15:01:43.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f068233d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f067c138ff0 0x7f067c1d1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39930/0 (socket says 172.21.15.26:39930) 2024-09-16T15:01:43.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f068233d640 1 -- 172.21.15.26:0/1361242817 learned_addr learned my addr 172.21.15.26:0/1361242817 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:43.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f068133b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f067c132650 0x7f067c12dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:56714/0 (socket says 172.21.15.26:56714) 2024-09-16T15:01:43.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772474262 0 0) 0x7f067c12e2d0 con 0x7f067c138ff0 2024-09-16T15:01:43.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient(hunting): my global_id is 334323 2024-09-16T15:01:43.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0660003530 con 0x7f067c138ff0 2024-09-16T15:01:43.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937647434 0 0) 0x7f067c09a240 con 0x7f067c132650 2024-09-16T15:01:43.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient(hunting): my global_id is 354128 2024-09-16T15:01:43.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f067c12e2d0 con 0x7f067c132650 2024-09-16T15:01:43.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2997723651 0 0) 0x7f0660003530 con 0x7f067c138ff0 2024-09-16T15:01:43.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f067c09a240 con 0x7f067c138ff0 2024-09-16T15:01:43.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f06740018c0 con 0x7f067c138ff0 2024-09-16T15:01:43.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:43.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient(hunting): dump: 2024-09-16T15:01:43.726 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.726 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.726 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.726 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.726 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.727 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.727 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.727 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.727 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.727 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:43.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3815208281 0 0) 0x7f067c12e2d0 con 0x7f067c132650 2024-09-16T15:01:43.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0660003530 con 0x7f067c132650 2024-09-16T15:01:43.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f066c0018a0 con 0x7f067c132650 2024-09-16T15:01:43.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:43.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient(hunting): dump: 2024-09-16T15:01:43.729 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.729 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.729 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.729 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.730 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.730 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.730 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.730 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.730 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.731 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2167401279 0 0) 0x7f067c09a240 con 0x7f067c138ff0 2024-09-16T15:01:43.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 >> v1:172.21.15.26:6790/0 conn(0x7f067c132650 legacy=0x7f067c12dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 >> v1:172.21.15.189:6789/0 conn(0x7f067c09cb80 legacy=0x7f067c12d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient: _finish_hunting 0 2024-09-16T15:01:43.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 monclient: found mon.a 2024-09-16T15:01:43.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:43.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:43.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c138860 con 0x7f067c138ff0 2024-09-16T15:01:43.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient: _finish_auth 0 2024-09-16T15:01:43.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient: _check_auth_tickets 2024-09-16T15:01:43.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:43.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 5 monclient: authenticate success, global_id 334323 2024-09-16T15:01:43.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 monclient: _renew_subs 2024-09-16T15:01:43.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:43.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 1 -- 172.21.15.26:0/1361242817 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067c134bb0 con 0x7f067c138ff0 2024-09-16T15:01:43.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 monclient: _renew_subs 2024-09-16T15:01:43.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:43.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 1 -- 172.21.15.26:0/1361242817 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f067c1348d0 con 0x7f067c138ff0 2024-09-16T15:01:43.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06740015a0 con 0x7f067c138ff0 2024-09-16T15:01:43.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:43.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0674005090 con 0x7f067c138ff0 2024-09-16T15:01:43.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:43.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0673fff640 4 set_mon_vals no callback set 2024-09-16T15:01:43.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f06727fc640 10 monclient: dump: 2024-09-16T15:01:43.735 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.736 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.736 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.736 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.736 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.736 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.737 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.737 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.737 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.737 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:43.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.684+0000 7f0683da0640 1 -- 172.21.15.26:0/1361242817 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0664006590 con 0x7f0664002830 2024-09-16T15:01:43.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f06740051f0 con 0x7f067c138ff0 2024-09-16T15:01:43.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f0674044260 con 0x7f067c138ff0 2024-09-16T15:01:43.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623823616 0 0) 0x7f0664006590 con 0x7f0664002830 2024-09-16T15:01:43.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 10 monclient(hunting): my global_id is 344230 2024-09-16T15:01:43.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f067c09a240 con 0x7f0664002830 2024-09-16T15:01:43.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f068133b640 10 monclient: get_auth_request con 0x7f06600361c0 auth_method 0 2024-09-16T15:01:43.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2213677270 0 0) 0x7f067c09a240 con 0x7f0664002830 2024-09-16T15:01:43.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0664006590 con 0x7f0664002830 2024-09-16T15:01:43.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f06780015a0 con 0x7f0664002830 2024-09-16T15:01:43.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:43.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 10 monclient: dump: 2024-09-16T15:01:43.740 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.740 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.740 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.740 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.740 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.740 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.740 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.740 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.741 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.741 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3397466007 0 0) 0x7f0664006590 con 0x7f0664002830 2024-09-16T15:01:43.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f066400a0e0 con 0x7f0664002830 2024-09-16T15:01:43.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0678001910 con 0x7f0664002830 2024-09-16T15:01:43.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:43.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:43.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 >> v1:172.21.15.189:6789/0 conn(0x7f0664002830 legacy=0x7f0664004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.688+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 >> v1:172.21.15.189:6789/0 conn(0x7f0664002830 legacy=0x7f0664004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:43.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f0683da0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:01:43.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f0683da0640 1 -- 172.21.15.26:0/1361242817 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0664009da0 con 0x7f06640090c0 2024-09-16T15:01:43.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093692240 0 0) 0x7f0664009da0 con 0x7f06640090c0 2024-09-16T15:01:43.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 10 monclient(hunting): my global_id is 344233 2024-09-16T15:01:43.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0664006590 con 0x7f06640090c0 2024-09-16T15:01:43.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3536801874 0 0) 0x7f0664006590 con 0x7f06640090c0 2024-09-16T15:01:43.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0664009da0 con 0x7f06640090c0 2024-09-16T15:01:43.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f06780037a0 con 0x7f06640090c0 2024-09-16T15:01:43.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:43.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 10 monclient: dump: 2024-09-16T15:01:43.742 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.742 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.743 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.743 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.743 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.743 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.743 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.743 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.743 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.743 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1799614881 0 0) 0x7f0664009da0 con 0x7f06640090c0 2024-09-16T15:01:43.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f066400c290 con 0x7f06640090c0 2024-09-16T15:01:43.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 760146139) 0x7f0678005960 con 0x7f06640090c0 2024-09-16T15:01:43.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:01:43.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:43.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 >> v1:172.21.15.189:6789/0 conn(0x7f06640090c0 legacy=0x7f0664009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f06727fc640 1 -- 172.21.15.26:0/1361242817 >> v1:172.21.15.189:6789/0 conn(0x7f06640090c0 legacy=0x7f0664009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:43.744 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:01:43.744 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:43.744 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:43.744 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:01:43.744 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 174, 2024-09-16T15:01:43.744 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:01:43.744 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:01:43.745 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:01:43.745 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:01:43.745 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:43.745 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:01:43.745 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:43.745 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:01:43.745 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:01:43.745 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:43.745 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:43.745 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:43.745 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:43.745 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:43.746 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:43.746 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:43.746 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:43.746 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:43.746 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:43.746 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:43.746 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:01:43.746 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:01:43.746 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:43.746 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:43.746 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:43.747 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:43.747 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:43.747 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:43.747 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:43.747 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:43.747 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:43.747 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:43.747 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.747 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:01:43.747 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:01:43.747 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:01:43.748 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:01:43.748 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:01:43.748 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:01:43.748 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:43.748 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:43.748 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:01:43.748 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:01:43.748 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:01:43.748 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:01:43.748 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:01:43.748 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:01:43.748 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:01:43.748 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:43.748 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:01:43.749 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:43.749 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:43.749 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:43.749 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:43.749 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:43.749 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:43.749 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:43.749 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:43.749 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:43.749 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:43.749 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:43.749 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:01:43.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.750 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:01:43.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.750 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:43.750 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:43.750 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:43.750 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:43.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.750 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:43.750 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:01:43.750 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:43.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.750 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.750 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:01:43.751 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:01:43.751 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:43.751 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:43.751 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:43.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.751 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:43.751 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:43.751 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:43.751 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:43.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.751 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:43.752 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:01:43.752 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:43.752 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.752 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.752 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:01:43.752 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:01:43.752 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:43.752 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:43.752 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:43.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.752 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:43.752 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:43.753 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:43.753 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:43.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.753 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:43.753 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:01:43.753 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:43.753 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.753 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.753 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:01:43.753 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:01:43.753 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:43.753 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:43.754 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:43.754 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.754 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.754 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:01:43.754 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:01:43.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.754 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:43.754 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:43.754 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:43.754 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.754 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:43.754 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:01:43.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.755 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:43.755 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:43.755 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:43.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.755 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:43.755 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T15:01:43.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:43.755 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:43.755 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:43.755 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:43.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:43.755 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:43.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:43.755 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:01:43.756 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:01:43.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f0683da0640 1 -- 172.21.15.26:0/1361242817 >> v1:172.21.15.189:6812/3196161582 conn(0x7f06600361c0 legacy=0x7f0660038680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f0683da0640 10 monclient: shutdown 2024-09-16T15:01:43.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f0683da0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:43.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f0683da0640 1 -- 172.21.15.26:0/1361242817 >> v1:172.21.15.26:6789/0 conn(0x7f067c138ff0 legacy=0x7f067c1d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f068233d640 1 -- 172.21.15.26:0/1361242817 reap_dead start 2024-09-16T15:01:43.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f068233d640 1 -- 172.21.15.26:0/1361242817 reap_dead start 2024-09-16T15:01:43.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f0683da0640 1 -- 172.21.15.26:0/1361242817 shutdown_connections 2024-09-16T15:01:43.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f0683da0640 1 -- 172.21.15.26:0/1361242817 >> 172.21.15.26:0/1361242817 conn(0x7f067c09fb50 msgr2=0x7f067c09bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:43.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f0683da0640 1 -- 172.21.15.26:0/1361242817 shutdown_connections 2024-09-16T15:01:43.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.696+0000 7f0683da0640 1 -- 172.21.15.26:0/1361242817 wait complete. 2024-09-16T15:01:43.758 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:01:43.758 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T15:01:43.758 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T15:01:43.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.876+0000 7fbf5a7c4640 10 monclient: get_monmap_and_config 2024-09-16T15:01:43.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.876+0000 7fbf5a7c4640 10 monclient: build_initial_monmap 2024-09-16T15:01:43.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.876+0000 7fbf5a7c4640 10 monclient: monmap: 2024-09-16T15:01:43.878 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:43.878 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.878 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:43.879858+0000 2024-09-16T15:01:43.878 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:43.879858+0000 2024-09-16T15:01:43.878 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:43.878 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.878 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:43.878 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:43.878 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:43.878 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.876+0000 7fbf5a7c4640 1 Processor -- start 2024-09-16T15:01:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.876+0000 7fbf5a7c4640 1 -- start start 2024-09-16T15:01:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.876+0000 7fbf5a7c4640 10 monclient: init 2024-09-16T15:01:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.876+0000 7fbf5a7c4640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.876+0000 7fbf5a7c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:01:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.876+0000 7fbf5a7c4640 10 monclient(hunting): picked mon.noname-b con 0x7fbf5409a550 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.876+0000 7fbf5a7c4640 10 monclient(hunting): picked mon.noname-c con 0x7fbf5409d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.876+0000 7fbf5a7c4640 10 monclient(hunting): picked mon.noname-a con 0x7fbf5409b040 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5409dc60 con 0x7fbf5409b040 2024-09-16T15:01:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf54099ae0 con 0x7fbf5409d1b0 2024-09-16T15:01:43.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5409b8f0 con 0x7fbf5409a550 2024-09-16T15:01:43.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:43.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 10 monclient(hunting): authenticate will time out at 2128.000134s 2024-09-16T15:01:43.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf53fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbf5409a550 0x7fbf5409a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:56718/0 (socket says 172.21.15.26:56718) 2024-09-16T15:01:43.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf53fff640 1 -- 172.21.15.26:0/1215533159 learned_addr learned my addr 172.21.15.26:0/1215533159 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:43.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 1 -- 172.21.15.26:0/1215533159 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737568026 0 0) 0x7fbf5409b8f0 con 0x7fbf5409a550 2024-09-16T15:01:43.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient(hunting): my global_id is 354131 2024-09-16T15:01:43.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 1 -- 172.21.15.26:0/1215533159 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf34003530 con 0x7fbf5409a550 2024-09-16T15:01:43.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 1 -- 172.21.15.26:0/1215533159 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91793877 0 0) 0x7fbf5409dc60 con 0x7fbf5409b040 2024-09-16T15:01:43.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient(hunting): my global_id is 334326 2024-09-16T15:01:43.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 1 -- 172.21.15.26:0/1215533159 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5409b8f0 con 0x7fbf5409b040 2024-09-16T15:01:43.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 1 -- 172.21.15.26:0/1215533159 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510337095 0 0) 0x7fbf54099ae0 con 0x7fbf5409d1b0 2024-09-16T15:01:43.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient(hunting): my global_id is 344236 2024-09-16T15:01:43.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 1 -- 172.21.15.26:0/1215533159 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5409dc60 con 0x7fbf5409d1b0 2024-09-16T15:01:43.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 1 -- 172.21.15.26:0/1215533159 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2502981817 0 0) 0x7fbf34003530 con 0x7fbf5409a550 2024-09-16T15:01:43.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 1 -- 172.21.15.26:0/1215533159 >> v1:172.21.15.189:6789/0 conn(0x7fbf5409d1b0 legacy=0x7fbf54136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 1 -- 172.21.15.26:0/1215533159 >> v1:172.21.15.26:6789/0 conn(0x7fbf5409b040 legacy=0x7fbf5413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient: _finish_hunting 0 2024-09-16T15:01:43.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 1 monclient: found mon.noname-b 2024-09-16T15:01:43.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:43.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:01:43.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 1 -- 172.21.15.26:0/1215533159 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5409bbd0 con 0x7fbf5409a550 2024-09-16T15:01:43.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient: _finish_auth 0 2024-09-16T15:01:43.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient: _check_auth_tickets 2024-09-16T15:01:43.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:43.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 1 -- 172.21.15.26:0/1215533159 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf3c0018d0 con 0x7fbf5409a550 2024-09-16T15:01:43.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 5 monclient: authenticate success, global_id 354131 2024-09-16T15:01:43.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:43.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:01:43.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient: dump: 2024-09-16T15:01:43.891 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.892 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.892 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.892 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.892 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.892 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.893 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.893 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.893 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.893 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:43.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 1 -- 172.21.15.26:0/1215533159 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf3c001bb0 con 0x7fbf5409a550 2024-09-16T15:01:43.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:43.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:43.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 4 set_mon_vals no callback set 2024-09-16T15:01:43.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 1 -- 172.21.15.26:0/1215533159 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf3c001f80 con 0x7fbf5409a550 2024-09-16T15:01:43.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:43.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf52ffd640 10 monclient: dump: 2024-09-16T15:01:43.894 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.894 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.894 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.895 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 10 monclient: shutdown 2024-09-16T15:01:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 1 -- 172.21.15.26:0/1215533159 >> v1:172.21.15.26:6790/0 conn(0x7fbf5409a550 legacy=0x7fbf5409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 1 -- 172.21.15.26:0/1215533159 shutdown_connections 2024-09-16T15:01:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 1 -- 172.21.15.26:0/1215533159 >> 172.21.15.26:0/1215533159 conn(0x7fbf5409fb50 msgr2=0x7fbf5409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:43.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 1 -- 172.21.15.26:0/1215533159 shutdown_connections 2024-09-16T15:01:43.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 1 -- 172.21.15.26:0/1215533159 wait complete. 2024-09-16T15:01:43.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 10 monclient: build_initial_monmap 2024-09-16T15:01:43.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 10 monclient: monmap: 2024-09-16T15:01:43.896 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:43.896 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:43.896 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:43.896 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:43.896 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:43.896 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:43.896 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:43.896 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:43.896 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:43.896 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 1 Processor -- start 2024-09-16T15:01:43.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 1 -- start start 2024-09-16T15:01:43.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 10 monclient: init 2024-09-16T15:01:43.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:43.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:01:43.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 10 monclient(hunting): picked mon.noname-b con 0x7fbf5409a550 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:43.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 10 monclient(hunting): picked mon.noname-a con 0x7fbf5409b040 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:43.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 10 monclient(hunting): picked mon.noname-c con 0x7fbf5409d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:43.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf54120660 con 0x7fbf5409b040 2024-09-16T15:01:43.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf540a26c0 con 0x7fbf5409a550 2024-09-16T15:01:43.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5409a240 con 0x7fbf5409d1b0 2024-09-16T15:01:43.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:43.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf5a7c4640 10 monclient(hunting): authenticate will time out at 2128.003274s 2024-09-16T15:01:43.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf58d61640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbf5409d1b0 0x7fbf5411ee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:56726/0 (socket says 172.21.15.26:56726) 2024-09-16T15:01:43.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf58d61640 1 -- 172.21.15.26:0/3723158776 learned_addr learned my addr 172.21.15.26:0/3723158776 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:43.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf58d61640 1 -- 172.21.15.26:0/3723158776 >> v1:172.21.15.26:6790/0 conn(0x7fbf5409d1b0 legacy=0x7fbf5411ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T15:01:43.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf58d61640 1 -- 172.21.15.26:0/3723158776 >> v1:172.21.15.26:6790/0 conn(0x7fbf5409d1b0 legacy=0x7fbf5411ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T15:01:43.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf58d61640 1 --1- 172.21.15.26:0/3723158776 >> v1:172.21.15.26:6790/0 conn(0x7fbf5409d1b0 0x7fbf5411ee80 :-1 s=OPENED pgs=1741 cs=1 l=1).handle_message read tag failed 2024-09-16T15:01:43.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf58d61640 1 --1- 172.21.15.26:0/3723158776 >> v1:172.21.15.26:6790/0 conn(0x7fbf5409d1b0 0x7fbf5411ee80 :-1 s=OPENED pgs=1741 cs=1 l=1).fault on lossy channel, failing 2024-09-16T15:01:43.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf50ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6790/0 2024-09-16T15:01:43.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254726509 0 0) 0x7fbf54120660 con 0x7fbf5409b040 2024-09-16T15:01:43.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf50ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf50ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf50ff9640 10 monclient(hunting): my global_id is 334329 2024-09-16T15:01:43.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf38003530 con 0x7fbf5409b040 2024-09-16T15:01:43.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.880+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 62794948 0 0) 0x7fbf38003530 con 0x7fbf5409b040 2024-09-16T15:01:43.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf54120660 con 0x7fbf5409b040 2024-09-16T15:01:43.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565659667 0 0) 0x7fbf540a26c0 con 0x7fbf5409a550 2024-09-16T15:01:43.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:01:43.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:43.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:43.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 10 monclient(hunting): my global_id is 344239 2024-09-16T15:01:43.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf38003530 con 0x7fbf5409a550 2024-09-16T15:01:43.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf440018c0 con 0x7fbf5409b040 2024-09-16T15:01:43.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:43.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 10 monclient(hunting): dump: 2024-09-16T15:01:43.900 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.900 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.900 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.900 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.900 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.900 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.900 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.900 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.900 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.901 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:43.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2760745630 0 0) 0x7fbf54120660 con 0x7fbf5409b040 2024-09-16T15:01:43.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 >> v1:172.21.15.26:6790/0 conn(0x7fbf5409d1b0 legacy=0x7fbf5411ee80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:43.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 >> v1:172.21.15.189:6789/0 conn(0x7fbf5409a550 legacy=0x7fbf5411e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:43.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 10 monclient: _finish_hunting 0 2024-09-16T15:01:43.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 1 monclient: found mon.a 2024-09-16T15:01:43.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:43.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:43.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf54121fc0 con 0x7fbf5409b040 2024-09-16T15:01:43.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 10 monclient: _finish_auth 0 2024-09-16T15:01:43.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 10 monclient: _check_auth_tickets 2024-09-16T15:01:43.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:43.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf5a7c4640 5 monclient: authenticate success, global_id 334329 2024-09-16T15:01:43.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf5a7c4640 10 monclient: _renew_subs 2024-09-16T15:01:43.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf5a7c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:43.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf5a7c4640 1 -- 172.21.15.26:0/3723158776 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf54121720 con 0x7fbf5409b040 2024-09-16T15:01:43.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf5a7c4640 10 monclient: _renew_subs 2024-09-16T15:01:43.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf5a7c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:43.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf5a7c4640 1 -- 172.21.15.26:0/3723158776 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf541d2740 con 0x7fbf5409b040 2024-09-16T15:01:43.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf44001c50 con 0x7fbf5409b040 2024-09-16T15:01:43.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:43.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fbf44003a70 con 0x7fbf5409b040 2024-09-16T15:01:43.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:43.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:43.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf527fc640 4 set_mon_vals no callback set 2024-09-16T15:01:43.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 10 monclient: dump: 2024-09-16T15:01:43.902 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:43.903 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:43.903 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.903 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:43.903 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:43.903 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:43.903 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:43.903 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:43.903 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:43.903 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:43.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf5a7c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:43.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf5a7c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:43.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf5a7c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:43.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf5a7c4640 1 -- 172.21.15.26:0/3723158776 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf2c005050 con 0x7fbf5409b040 2024-09-16T15:01:43.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fbf44003bd0 con 0x7fbf5409b040 2024-09-16T15:01:43.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fbf44042b60 con 0x7fbf5409b040 2024-09-16T15:01:43.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.884+0000 7fbf53fff640 10 monclient: get_auth_request con 0x7fbf38036080 auth_method 0 2024-09-16T15:01:43.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.888+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbf440134a0 con 0x7fbf5409b040 2024-09-16T15:01:43.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.888+0000 7fbf50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:43.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:43.888+0000 7fbf50ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:44.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:44.108+0000 7fbf5a7c4640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T15:01:44.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:44.108+0000 7fbf5a7c4640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T15:01:44.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:44.108+0000 7fbf5a7c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:44.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:44.108+0000 7fbf5a7c4640 1 -- 172.21.15.26:0/3723158776 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fbf2c005b10 con 0x7fbf5409b040 2024-09-16T15:01:44.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:44.108+0000 7fbf50ff9640 1 -- 172.21.15.26:0/3723158776 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fbf44011910 con 0x7fbf5409b040 2024-09-16T15:01:44.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:44.108+0000 7fbf50ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T15:01:44.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:44.108+0000 7fbf50ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:44.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.108+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.108+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=4260265912,config=692619927,health=2950445267,kv=1867599010,logm=159420501}) 2024-09-16T15:01:44.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.108+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=3968713238,config=692619927,health=3610035298,kv=2773850397,logm=2109994126}) 2024-09-16T15:01:44.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.108+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.108+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=4260265912,config=692619927,health=2950445267,kv=1867599010,logm=159420501}) 2024-09-16T15:01:44.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.112+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=11,logm=46} crc {auth=3050809925,config=692619927,health=151922238,kv=4148237855,logm=1335451932}) 2024-09-16T15:01:44.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:44.112+0000 7fbf5a7c4640 1 -- 172.21.15.26:0/3723158776 >> v1:172.21.15.189:6812/3196161582 conn(0x7fbf38036080 legacy=0x7fbf38038540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:44.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:44.112+0000 7fbf5a7c4640 10 monclient: shutdown 2024-09-16T15:01:44.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:44.112+0000 7fbf5a7c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:44.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:44.112+0000 7fbf5a7c4640 1 -- 172.21.15.26:0/3723158776 >> v1:172.21.15.26:6789/0 conn(0x7fbf5409b040 legacy=0x7fbf5411e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:44.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:44.112+0000 7fbf5a7c4640 1 -- 172.21.15.26:0/3723158776 shutdown_connections 2024-09-16T15:01:44.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:44.112+0000 7fbf5a7c4640 1 -- 172.21.15.26:0/3723158776 >> 172.21.15.26:0/3723158776 conn(0x7fbf5409fb50 msgr2=0x7fbf54139f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:44.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:44.112+0000 7fbf5a7c4640 1 -- 172.21.15.26:0/3723158776 shutdown_connections 2024-09-16T15:01:44.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:44.112+0000 7fbf5a7c4640 1 -- 172.21.15.26:0/3723158776 wait complete. 2024-09-16T15:01:44.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.112+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.112+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=261818460}) 2024-09-16T15:01:44.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.112+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=437415966}) 2024-09-16T15:01:44.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.112+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.112+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=261818460}) 2024-09-16T15:01:44.114 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.112+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=144042219}) 2024-09-16T15:01:44.117 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.116+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.117 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.116+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3842977331}) 2024-09-16T15:01:44.117 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.116+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249584962}) 2024-09-16T15:01:44.117 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.116+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.117 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.116+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3842977331}) 2024-09-16T15:01:44.117 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.116+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2723271576}) 2024-09-16T15:01:44.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.120+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.120+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1342203271}) 2024-09-16T15:01:44.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.120+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3161646226}) 2024-09-16T15:01:44.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.120+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.120+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1342203271}) 2024-09-16T15:01:44.120 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.120+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3031242893}) 2024-09-16T15:01:44.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.120+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.124+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862654847}) 2024-09-16T15:01:44.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.124+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826837137}) 2024-09-16T15:01:44.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.124+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.124+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862654847}) 2024-09-16T15:01:44.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.124+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2303837506}) 2024-09-16T15:01:44.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.124+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.124+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=513775343}) 2024-09-16T15:01:44.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.124+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537985791}) 2024-09-16T15:01:44.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.124+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.124+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=513775343}) 2024-09-16T15:01:44.126 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.124+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=704455652}) 2024-09-16T15:01:44.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.128+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.128+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1617162200}) 2024-09-16T15:01:44.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.128+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876312386}) 2024-09-16T15:01:44.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.128+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.128+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1617162200}) 2024-09-16T15:01:44.129 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.128+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2315438773}) 2024-09-16T15:01:44.130 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T15:01:44.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.128+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.128+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3915950790}) 2024-09-16T15:01:44.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.128+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661315212}) 2024-09-16T15:01:44.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.128+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.128+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3915950790}) 2024-09-16T15:01:44.131 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.128+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1394165750}) 2024-09-16T15:01:44.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.132+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.132+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3849461550}) 2024-09-16T15:01:44.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.132+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090932883}) 2024-09-16T15:01:44.133 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.132+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.133 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.132+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3849461550}) 2024-09-16T15:01:44.133 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.132+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3549376856}) 2024-09-16T15:01:44.135 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.136+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.136+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1650949334}) 2024-09-16T15:01:44.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.136+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=530005146}) 2024-09-16T15:01:44.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.136+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.136+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1650949334}) 2024-09-16T15:01:44.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.136+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=229032310}) 2024-09-16T15:01:44.139 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.136+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.139 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.136+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=609618277}) 2024-09-16T15:01:44.139 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.136+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4051915692}) 2024-09-16T15:01:44.139 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.136+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.139 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.136+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=609618277}) 2024-09-16T15:01:44.139 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.136+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=738082315}) 2024-09-16T15:01:44.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.140+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.140+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1918789703}) 2024-09-16T15:01:44.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.140+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3478236615}) 2024-09-16T15:01:44.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.140+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.140+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1918789703}) 2024-09-16T15:01:44.142 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.140+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3584445387}) 2024-09-16T15:01:44.145 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.144+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.145 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.144+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=378574293}) 2024-09-16T15:01:44.145 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.144+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674612608}) 2024-09-16T15:01:44.145 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.144+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.145 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.144+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=378574293}) 2024-09-16T15:01:44.145 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.144+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=579275136}) 2024-09-16T15:01:44.148 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.148+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.148 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.148+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3510724017}) 2024-09-16T15:01:44.148 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.148+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2719943495}) 2024-09-16T15:01:44.149 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.148+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.149 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.148+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3510724017}) 2024-09-16T15:01:44.149 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.148+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2467919486}) 2024-09-16T15:01:44.150 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.148+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.151 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.148+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3050537792}) 2024-09-16T15:01:44.151 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.148+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2093240597}) 2024-09-16T15:01:44.151 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.148+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.151 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.148+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3050537792}) 2024-09-16T15:01:44.151 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.148+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=806279867}) 2024-09-16T15:01:44.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.152+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.152+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2144130696}) 2024-09-16T15:01:44.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.152+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528224455}) 2024-09-16T15:01:44.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.152+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.152+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2144130696}) 2024-09-16T15:01:44.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.152+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2154608021}) 2024-09-16T15:01:44.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.152+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.156+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2411863376}) 2024-09-16T15:01:44.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.156+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1609836142}) 2024-09-16T15:01:44.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.156+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.156+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2411863376}) 2024-09-16T15:01:44.155 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.156+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=620515617}) 2024-09-16T15:01:44.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.156+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.156+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2334009100}) 2024-09-16T15:01:44.157 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.156+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=133694150}) 2024-09-16T15:01:44.158 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.156+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.158 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.156+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2334009100}) 2024-09-16T15:01:44.158 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.156+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3436275088}) 2024-09-16T15:01:44.160 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.160+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.160 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.160+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1538858101}) 2024-09-16T15:01:44.160 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.160+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=464440744}) 2024-09-16T15:01:44.160 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.160+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.160+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1538858101}) 2024-09-16T15:01:44.161 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.160+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2489808225}) 2024-09-16T15:01:44.162 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.160+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.162 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.160+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=341247826}) 2024-09-16T15:01:44.162 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.160+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2007147584}) 2024-09-16T15:01:44.162 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.160+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.162 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.160+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=341247826}) 2024-09-16T15:01:44.163 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.160+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=513220838}) 2024-09-16T15:01:44.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.164+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.164+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4077817910}) 2024-09-16T15:01:44.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.164+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=238362922}) 2024-09-16T15:01:44.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.164+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.164+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4077817910}) 2024-09-16T15:01:44.165 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.164+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3717946688}) 2024-09-16T15:01:44.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.168+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.168+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=712155676}) 2024-09-16T15:01:44.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.168+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1924585380}) 2024-09-16T15:01:44.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.168+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.168+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=712155676}) 2024-09-16T15:01:44.167 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.168+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3689842394}) 2024-09-16T15:01:44.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.168+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.169 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.168+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3194749351}) 2024-09-16T15:01:44.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.168+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335623626}) 2024-09-16T15:01:44.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.168+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.168+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3194749351}) 2024-09-16T15:01:44.170 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.168+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1877368822}) 2024-09-16T15:01:44.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.172+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.172+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1392087598}) 2024-09-16T15:01:44.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.172+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2793028713}) 2024-09-16T15:01:44.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.172+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.172+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1392087598}) 2024-09-16T15:01:44.172 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.172+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4026074544}) 2024-09-16T15:01:44.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.172+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.172+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1824329099}) 2024-09-16T15:01:44.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.172+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1143828042}) 2024-09-16T15:01:44.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.172+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.172+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1824329099}) 2024-09-16T15:01:44.174 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.172+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=812276592}) 2024-09-16T15:01:44.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.176+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.176+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1147410365}) 2024-09-16T15:01:44.176 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.176+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056485533}) 2024-09-16T15:01:44.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.176+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.176+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1147410365}) 2024-09-16T15:01:44.177 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.176+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=343789661}) 2024-09-16T15:01:44.178 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.176+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.178 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.176+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3761908056}) 2024-09-16T15:01:44.178 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.176+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=484120137}) 2024-09-16T15:01:44.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.176+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.176+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3761908056}) 2024-09-16T15:01:44.179 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.176+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3228667184}) 2024-09-16T15:01:44.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.180+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.180+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4064807550}) 2024-09-16T15:01:44.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.180+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820496795}) 2024-09-16T15:01:44.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.180+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.180+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4064807550}) 2024-09-16T15:01:44.181 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.180+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741959867}) 2024-09-16T15:01:44.183 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.184+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.184+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2311805217}) 2024-09-16T15:01:44.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.184+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=801478995}) 2024-09-16T15:01:44.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.184+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.184+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2311805217}) 2024-09-16T15:01:44.184 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.184+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2941175953}) 2024-09-16T15:01:44.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.184+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.184+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2814983805}) 2024-09-16T15:01:44.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.184+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2698029685}) 2024-09-16T15:01:44.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.184+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.184+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2814983805}) 2024-09-16T15:01:44.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.184+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3731919289}) 2024-09-16T15:01:44.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.188+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.188+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=244802235}) 2024-09-16T15:01:44.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.188+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1762307720}) 2024-09-16T15:01:44.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.188+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.188+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=244802235}) 2024-09-16T15:01:44.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.188+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2085430212}) 2024-09-16T15:01:44.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.188+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.188+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1515103197}) 2024-09-16T15:01:44.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.188+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2030345591}) 2024-09-16T15:01:44.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.188+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.188+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1515103197}) 2024-09-16T15:01:44.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.188+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4154199559}) 2024-09-16T15:01:44.192 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.192+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.192 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.192+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=967670857}) 2024-09-16T15:01:44.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.192+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3975527283}) 2024-09-16T15:01:44.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.192+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.192+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=967670857}) 2024-09-16T15:01:44.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.192+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1009727242}) 2024-09-16T15:01:44.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.192+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.192+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3106804513}) 2024-09-16T15:01:44.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.196+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3988581674}) 2024-09-16T15:01:44.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.196+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.196+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3106804513}) 2024-09-16T15:01:44.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.196+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=235683918}) 2024-09-16T15:01:44.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.196+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.196+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3424493735}) 2024-09-16T15:01:44.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.196+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2818556511}) 2024-09-16T15:01:44.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.196+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.196+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3424493735}) 2024-09-16T15:01:44.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.196+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3012739928}) 2024-09-16T15:01:44.199 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.200+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.199 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.200+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2668427207}) 2024-09-16T15:01:44.199 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.200+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2579985119}) 2024-09-16T15:01:44.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.200+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.200+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2668427207}) 2024-09-16T15:01:44.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.200+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598026450}) 2024-09-16T15:01:44.201 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.200+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.201 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.200+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2221362474}) 2024-09-16T15:01:44.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.200+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1547182001}) 2024-09-16T15:01:44.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.200+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.200+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2221362474}) 2024-09-16T15:01:44.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.200+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3126210386}) 2024-09-16T15:01:44.203 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.204+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.204+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3370342406}) 2024-09-16T15:01:44.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.204+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685352479}) 2024-09-16T15:01:44.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.204+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.204+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3370342406}) 2024-09-16T15:01:44.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.204+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3376312574}) 2024-09-16T15:01:44.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.204+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.204+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1990035534}) 2024-09-16T15:01:44.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.204+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3830469712}) 2024-09-16T15:01:44.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.204+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.204+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1990035534}) 2024-09-16T15:01:44.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.204+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2174369565}) 2024-09-16T15:01:44.208 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.208+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.208 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.208+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2909702221}) 2024-09-16T15:01:44.208 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.208+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1142278550}) 2024-09-16T15:01:44.208 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.208+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.208 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.208+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2909702221}) 2024-09-16T15:01:44.208 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.208+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4262842212}) 2024-09-16T15:01:44.210 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.208+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.210 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.208+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3497541993}) 2024-09-16T15:01:44.210 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.208+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1328165972}) 2024-09-16T15:01:44.210 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.208+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.210 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.208+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3497541993}) 2024-09-16T15:01:44.210 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.208+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2292784926}) 2024-09-16T15:01:44.212 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.212+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.212 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.212+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1614694515}) 2024-09-16T15:01:44.212 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.212+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1147847265}) 2024-09-16T15:01:44.212 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.212+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.212 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.212+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1614694515}) 2024-09-16T15:01:44.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.212+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3389554466}) 2024-09-16T15:01:44.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.212+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.212+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4039929755}) 2024-09-16T15:01:44.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.212+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1532335604}) 2024-09-16T15:01:44.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.212+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.212+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4039929755}) 2024-09-16T15:01:44.215 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.212+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=890901515}) 2024-09-16T15:01:44.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.216+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.216+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2053483724}) 2024-09-16T15:01:44.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.216+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4110715444}) 2024-09-16T15:01:44.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.216+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.217 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.216+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2053483724}) 2024-09-16T15:01:44.217 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.216+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1166781936}) 2024-09-16T15:01:44.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.216+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.216+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4167544018}) 2024-09-16T15:01:44.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.216+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924633261}) 2024-09-16T15:01:44.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.216+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.216+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4167544018}) 2024-09-16T15:01:44.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.216+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3794278487}) 2024-09-16T15:01:44.220 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.220+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.220 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.220+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1417477335}) 2024-09-16T15:01:44.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.220+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3506911439}) 2024-09-16T15:01:44.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.220+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.220+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1417477335}) 2024-09-16T15:01:44.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.220+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=773400050}) 2024-09-16T15:01:44.222 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.220+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.220+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3615860901}) 2024-09-16T15:01:44.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.220+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2604194685}) 2024-09-16T15:01:44.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.220+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.220+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3615860901}) 2024-09-16T15:01:44.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.220+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2680118461}) 2024-09-16T15:01:44.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.224+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.224+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3965817342}) 2024-09-16T15:01:44.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.224+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3505462822}) 2024-09-16T15:01:44.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.224+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.224+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3965817342}) 2024-09-16T15:01:44.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.224+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3512241950}) 2024-09-16T15:01:44.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.224+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.224+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=422157793}) 2024-09-16T15:01:44.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.224+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575151354}) 2024-09-16T15:01:44.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.224+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.224+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=422157793}) 2024-09-16T15:01:44.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.228+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248541230}) 2024-09-16T15:01:44.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.228+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.228+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1096821610}) 2024-09-16T15:01:44.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.228+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995217095}) 2024-09-16T15:01:44.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.228+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.228+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1096821610}) 2024-09-16T15:01:44.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.228+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1430856461}) 2024-09-16T15:01:44.231 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.228+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.231 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.232+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3647265838}) 2024-09-16T15:01:44.231 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.232+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884978210}) 2024-09-16T15:01:44.231 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.232+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.232+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3647265838}) 2024-09-16T15:01:44.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.232+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3082319210}) 2024-09-16T15:01:44.233 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.232+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.233 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.232+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3157611572}) 2024-09-16T15:01:44.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.232+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1592716047}) 2024-09-16T15:01:44.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.232+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.232+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3157611572}) 2024-09-16T15:01:44.234 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.232+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3898597742}) 2024-09-16T15:01:44.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.236+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.236+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=950820998}) 2024-09-16T15:01:44.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.236+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1659661518}) 2024-09-16T15:01:44.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.236+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.236+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=950820998}) 2024-09-16T15:01:44.236 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.236+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2353483603}) 2024-09-16T15:01:44.237 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.236+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.236+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=706647760}) 2024-09-16T15:01:44.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.236+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1238787075}) 2024-09-16T15:01:44.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.236+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.236+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=706647760}) 2024-09-16T15:01:44.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.236+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=178048978}) 2024-09-16T15:01:44.239 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.240+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.240+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3962407247}) 2024-09-16T15:01:44.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.240+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1926932376}) 2024-09-16T15:01:44.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.240+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.240+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3962407247}) 2024-09-16T15:01:44.240 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.240+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4114127597}) 2024-09-16T15:01:44.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.240+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.240+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2749225317}) 2024-09-16T15:01:44.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.240+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3932911085}) 2024-09-16T15:01:44.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.240+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.240+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2749225317}) 2024-09-16T15:01:44.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.240+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1695983187}) 2024-09-16T15:01:44.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.244+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.244+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2376081260}) 2024-09-16T15:01:44.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.244+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443637781}) 2024-09-16T15:01:44.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.244+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.244+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2376081260}) 2024-09-16T15:01:44.244 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.244+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=957588253}) 2024-09-16T15:01:44.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.244+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.244+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352312838}) 2024-09-16T15:01:44.246 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.244+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289054314}) 2024-09-16T15:01:44.247 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.244+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.247 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.244+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352312838}) 2024-09-16T15:01:44.247 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.244+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=386630519}) 2024-09-16T15:01:44.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.248+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.248 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.248+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=992966447}) 2024-09-16T15:01:44.249 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.248+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165604816}) 2024-09-16T15:01:44.249 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.248+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.249 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.248+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=992966447}) 2024-09-16T15:01:44.249 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.248+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2052198534}) 2024-09-16T15:01:44.250 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.248+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.248+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=139423250}) 2024-09-16T15:01:44.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.248+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800259054}) 2024-09-16T15:01:44.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.248+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.248+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=139423250}) 2024-09-16T15:01:44.251 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.248+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=129627733}) 2024-09-16T15:01:44.252 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.252+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.253 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.252+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2143471448}) 2024-09-16T15:01:44.253 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.252+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2310987162}) 2024-09-16T15:01:44.253 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.252+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.253 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.252+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2143471448}) 2024-09-16T15:01:44.253 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.252+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3026741}) 2024-09-16T15:01:44.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.252+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.252+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4277314062}) 2024-09-16T15:01:44.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.252+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1173569413}) 2024-09-16T15:01:44.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.252+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.252+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4277314062}) 2024-09-16T15:01:44.254 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.252+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1494038553}) 2024-09-16T15:01:44.256 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.256+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.256+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2615725209}) 2024-09-16T15:01:44.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.256+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=596215846}) 2024-09-16T15:01:44.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.256+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.256+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2615725209}) 2024-09-16T15:01:44.257 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.256+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1719191152}) 2024-09-16T15:01:44.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.256+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.256+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=518620957}) 2024-09-16T15:01:44.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.256+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126246586}) 2024-09-16T15:01:44.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.256+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.256+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=518620957}) 2024-09-16T15:01:44.259 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.256+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=50967997}) 2024-09-16T15:01:44.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.260+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.260+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021866936}) 2024-09-16T15:01:44.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.260+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=91256724}) 2024-09-16T15:01:44.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.260+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.260+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021866936}) 2024-09-16T15:01:44.261 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.260+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=75425793}) 2024-09-16T15:01:44.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.264+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.264+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251647867}) 2024-09-16T15:01:44.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.264+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2228192069}) 2024-09-16T15:01:44.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.264+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.264+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251647867}) 2024-09-16T15:01:44.264 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.264+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2414499854}) 2024-09-16T15:01:44.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.264+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.264+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=683228249}) 2024-09-16T15:01:44.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.264+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3444770241}) 2024-09-16T15:01:44.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.264+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.264+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=683228249}) 2024-09-16T15:01:44.266 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.264+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=358326663}) 2024-09-16T15:01:44.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.268+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.268+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758769526}) 2024-09-16T15:01:44.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.268+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3631160637}) 2024-09-16T15:01:44.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.268+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.268+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758769526}) 2024-09-16T15:01:44.268 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.268+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2077373224}) 2024-09-16T15:01:44.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.268+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.268+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3472275159}) 2024-09-16T15:01:44.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.268+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=971685075}) 2024-09-16T15:01:44.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.268+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.268+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3472275159}) 2024-09-16T15:01:44.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.268+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2005097972}) 2024-09-16T15:01:44.273 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.272+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.272+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1773947041}) 2024-09-16T15:01:44.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.272+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1030700123}) 2024-09-16T15:01:44.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.272+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.272+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1773947041}) 2024-09-16T15:01:44.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.272+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2479100772}) 2024-09-16T15:01:44.274 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.272+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.272+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1630593804}) 2024-09-16T15:01:44.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.272+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1929854209}) 2024-09-16T15:01:44.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.272+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.272+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1630593804}) 2024-09-16T15:01:44.275 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.272+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1412772329}) 2024-09-16T15:01:44.276 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.276+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.277 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.276+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228031240}) 2024-09-16T15:01:44.277 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.276+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=142507379}) 2024-09-16T15:01:44.277 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.276+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.277 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.276+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228031240}) 2024-09-16T15:01:44.277 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.276+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1605263369}) 2024-09-16T15:01:44.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.276+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.278 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.276+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=194521663}) 2024-09-16T15:01:44.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.276+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613467573}) 2024-09-16T15:01:44.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.276+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.276+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=194521663}) 2024-09-16T15:01:44.279 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.276+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3177708646}) 2024-09-16T15:01:44.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.280+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.280+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3491746189}) 2024-09-16T15:01:44.280 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.280+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=77434911}) 2024-09-16T15:01:44.281 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.280+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.281 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.280+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3491746189}) 2024-09-16T15:01:44.281 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.280+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=877546279}) 2024-09-16T15:01:44.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.280+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.280+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3491714236}) 2024-09-16T15:01:44.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.280+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3616932147}) 2024-09-16T15:01:44.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.280+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.280+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3491714236}) 2024-09-16T15:01:44.283 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.280+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4215361535}) 2024-09-16T15:01:44.284 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.284+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.284+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3239630703}) 2024-09-16T15:01:44.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.284+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3256059610}) 2024-09-16T15:01:44.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.284+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.284+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3239630703}) 2024-09-16T15:01:44.285 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.284+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=676559804}) 2024-09-16T15:01:44.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.284+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.286 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.284+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3634364713}) 2024-09-16T15:01:44.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.284+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1754757797}) 2024-09-16T15:01:44.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.284+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.284+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3634364713}) 2024-09-16T15:01:44.287 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.284+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3608683530}) 2024-09-16T15:01:44.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.288+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.288 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.288+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1020367600}) 2024-09-16T15:01:44.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.288+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1739820029}) 2024-09-16T15:01:44.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.288+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.288+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1020367600}) 2024-09-16T15:01:44.289 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.288+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2799870494}) 2024-09-16T15:01:44.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.288+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.288+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1278514172}) 2024-09-16T15:01:44.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.288+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191008269}) 2024-09-16T15:01:44.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.288+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.288+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1278514172}) 2024-09-16T15:01:44.291 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.288+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=903513067}) 2024-09-16T15:01:44.292 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.292+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.292+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1533257009}) 2024-09-16T15:01:44.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.292+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=359519114}) 2024-09-16T15:01:44.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.292+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.292+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1533257009}) 2024-09-16T15:01:44.293 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.292+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=579752932}) 2024-09-16T15:01:44.294 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.292+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.292+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2498095144}) 2024-09-16T15:01:44.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.292+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1717872463}) 2024-09-16T15:01:44.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.292+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.292+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2498095144}) 2024-09-16T15:01:44.295 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.292+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=962810697}) 2024-09-16T15:01:44.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.296+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.296+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=589715621}) 2024-09-16T15:01:44.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.296+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063885801}) 2024-09-16T15:01:44.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.296+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.296+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=589715621}) 2024-09-16T15:01:44.297 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.296+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1796993175}) 2024-09-16T15:01:44.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.296+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.296+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426184996}) 2024-09-16T15:01:44.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.296+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477851125}) 2024-09-16T15:01:44.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.296+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.296+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426184996}) 2024-09-16T15:01:44.300 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.296+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2624215282}) 2024-09-16T15:01:44.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.300+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.300+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1827628748}) 2024-09-16T15:01:44.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.300+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3588062315}) 2024-09-16T15:01:44.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.300+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.300+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1827628748}) 2024-09-16T15:01:44.301 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.300+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2706071316}) 2024-09-16T15:01:44.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.304+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.304+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=424786808}) 2024-09-16T15:01:44.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.304+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3473585972}) 2024-09-16T15:01:44.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.304+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.303 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.304+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=424786808}) 2024-09-16T15:01:44.304 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.304+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3059579348}) 2024-09-16T15:01:44.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.304+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.304+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4186260296}) 2024-09-16T15:01:44.305 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.304+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572860927}) 2024-09-16T15:01:44.306 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.304+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.306 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.304+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4186260296}) 2024-09-16T15:01:44.306 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.304+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3673809522}) 2024-09-16T15:01:44.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.308+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.308+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4181638567}) 2024-09-16T15:01:44.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.308+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2163502342}) 2024-09-16T15:01:44.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.308+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.308+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4181638567}) 2024-09-16T15:01:44.308 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.308+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2744198350}) 2024-09-16T15:01:44.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.308+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.308+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3769262613}) 2024-09-16T15:01:44.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.308+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3975999529}) 2024-09-16T15:01:44.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.308+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.308+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3769262613}) 2024-09-16T15:01:44.310 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.308+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2568450304}) 2024-09-16T15:01:44.312 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.312+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.312 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.312+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3667255544}) 2024-09-16T15:01:44.312 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.312+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3928747239}) 2024-09-16T15:01:44.312 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.312+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.312 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.312+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3667255544}) 2024-09-16T15:01:44.313 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.312+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1173213918}) 2024-09-16T15:01:44.314 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.312+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.314 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.312+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1461974431}) 2024-09-16T15:01:44.314 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.312+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1630543940}) 2024-09-16T15:01:44.315 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.312+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.315 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.312+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1461974431}) 2024-09-16T15:01:44.315 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.312+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3216797428}) 2024-09-16T15:01:44.317 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.316+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.317 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.316+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2826081938}) 2024-09-16T15:01:44.317 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.316+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680397616}) 2024-09-16T15:01:44.317 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.316+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.317 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.316+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2826081938}) 2024-09-16T15:01:44.317 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.316+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2614696122}) 2024-09-16T15:01:44.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.316+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.316+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1792480429}) 2024-09-16T15:01:44.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.316+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=760865317}) 2024-09-16T15:01:44.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.316+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.316+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1792480429}) 2024-09-16T15:01:44.319 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.316+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1572517217}) 2024-09-16T15:01:44.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.320+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.320+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=133228632}) 2024-09-16T15:01:44.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.320+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3310267252}) 2024-09-16T15:01:44.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.320+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.320+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=133228632}) 2024-09-16T15:01:44.321 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.320+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2585230153}) 2024-09-16T15:01:44.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.320+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.320+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4090266811}) 2024-09-16T15:01:44.323 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.324+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2449781922}) 2024-09-16T15:01:44.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.324+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.324+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4090266811}) 2024-09-16T15:01:44.324 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.324+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2427297584}) 2024-09-16T15:01:44.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.324+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.324+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=495334959}) 2024-09-16T15:01:44.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.324+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3361195235}) 2024-09-16T15:01:44.325 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.324+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.324+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=495334959}) 2024-09-16T15:01:44.326 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.324+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1297175056}) 2024-09-16T15:01:44.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.328+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.328+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1621202644}) 2024-09-16T15:01:44.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.328+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3965303124}) 2024-09-16T15:01:44.327 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.328+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.328 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.328+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1621202644}) 2024-09-16T15:01:44.328 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.328+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2846673276}) 2024-09-16T15:01:44.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.328+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.329 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.328+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2359153043}) 2024-09-16T15:01:44.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.328+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=54536634}) 2024-09-16T15:01:44.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.328+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.328+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2359153043}) 2024-09-16T15:01:44.330 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.328+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3217129733}) 2024-09-16T15:01:44.331 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.332+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.332+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2982454648}) 2024-09-16T15:01:44.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.332+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3871156004}) 2024-09-16T15:01:44.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.332+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.332+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2982454648}) 2024-09-16T15:01:44.332 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.332+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3109737915}) 2024-09-16T15:01:44.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.332+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.332+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1259603436}) 2024-09-16T15:01:44.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.332+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75246026}) 2024-09-16T15:01:44.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.332+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.332+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1259603436}) 2024-09-16T15:01:44.334 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.332+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2232935811}) 2024-09-16T15:01:44.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.336+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.336+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=369239323}) 2024-09-16T15:01:44.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.336+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1375735667}) 2024-09-16T15:01:44.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.336+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.336 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.336+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=369239323}) 2024-09-16T15:01:44.337 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.336+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1377511677}) 2024-09-16T15:01:44.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.336+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.336+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972042103}) 2024-09-16T15:01:44.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.336+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558265582}) 2024-09-16T15:01:44.338 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.336+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.336+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972042103}) 2024-09-16T15:01:44.339 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.336+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2878656781}) 2024-09-16T15:01:44.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.340+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.340+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2706571048}) 2024-09-16T15:01:44.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.340+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3289929956}) 2024-09-16T15:01:44.340 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.340+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.340+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2706571048}) 2024-09-16T15:01:44.341 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.340+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1730724593}) 2024-09-16T15:01:44.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.340+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.340+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2757494498}) 2024-09-16T15:01:44.342 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.340+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3910839936}) 2024-09-16T15:01:44.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.340+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.340+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2757494498}) 2024-09-16T15:01:44.343 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.340+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=428705188}) 2024-09-16T15:01:44.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.344+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.344+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2600022046}) 2024-09-16T15:01:44.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.344+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516647695}) 2024-09-16T15:01:44.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.344+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.344+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2600022046}) 2024-09-16T15:01:44.345 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.344+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1028671988}) 2024-09-16T15:01:44.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.344+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.344+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2691268568}) 2024-09-16T15:01:44.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.344+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=335142759}) 2024-09-16T15:01:44.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.344+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.344+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2691268568}) 2024-09-16T15:01:44.347 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.344+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1479916111}) 2024-09-16T15:01:44.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.348+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.348+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3530229721}) 2024-09-16T15:01:44.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.348+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1804375523}) 2024-09-16T15:01:44.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.348+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.348+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3530229721}) 2024-09-16T15:01:44.349 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.348+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1802258093}) 2024-09-16T15:01:44.351 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.352+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.351 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.352+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3094599758}) 2024-09-16T15:01:44.351 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.352+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1360974920}) 2024-09-16T15:01:44.351 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.352+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.352+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3094599758}) 2024-09-16T15:01:44.352 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.352+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=517832566}) 2024-09-16T15:01:44.353 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.352+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.353 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.352+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1523620018}) 2024-09-16T15:01:44.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.352+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2315103082}) 2024-09-16T15:01:44.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.352+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.352+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1523620018}) 2024-09-16T15:01:44.354 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.352+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3428369306}) 2024-09-16T15:01:44.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.356+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.355 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.356+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2916245151}) 2024-09-16T15:01:44.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.356+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046452317}) 2024-09-16T15:01:44.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.356+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.356+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2916245151}) 2024-09-16T15:01:44.356 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.356+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1773047719}) 2024-09-16T15:01:44.357 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.356+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.356+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2975282704}) 2024-09-16T15:01:44.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.356+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344490126}) 2024-09-16T15:01:44.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.356+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.356+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2975282704}) 2024-09-16T15:01:44.358 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.356+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1143051310}) 2024-09-16T15:01:44.359 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.360+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.360+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=720511913}) 2024-09-16T15:01:44.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.360+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4261077016}) 2024-09-16T15:01:44.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.360+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.360+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=720511913}) 2024-09-16T15:01:44.360 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.360+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3046420983}) 2024-09-16T15:01:44.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.360+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.360+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3450388537}) 2024-09-16T15:01:44.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.360+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=121409624}) 2024-09-16T15:01:44.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.360+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.360+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3450388537}) 2024-09-16T15:01:44.362 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.360+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3000694236}) 2024-09-16T15:01:44.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.364+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.364+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=964721957}) 2024-09-16T15:01:44.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.364+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=927134235}) 2024-09-16T15:01:44.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.364+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.364+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=964721957}) 2024-09-16T15:01:44.364 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.364+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1190772403}) 2024-09-16T15:01:44.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.364+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.364+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=825590139}) 2024-09-16T15:01:44.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.364+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3881433061}) 2024-09-16T15:01:44.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.364+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.364+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=825590139}) 2024-09-16T15:01:44.366 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.364+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3816413056}) 2024-09-16T15:01:44.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.368+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.368+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022187797}) 2024-09-16T15:01:44.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.368+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2588373270}) 2024-09-16T15:01:44.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.368+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.368+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022187797}) 2024-09-16T15:01:44.368 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.368+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1357488351}) 2024-09-16T15:01:44.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.368+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.368+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4193368170}) 2024-09-16T15:01:44.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.368+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=250641290}) 2024-09-16T15:01:44.370 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.368+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.368+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4193368170}) 2024-09-16T15:01:44.371 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.368+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2960219861}) 2024-09-16T15:01:44.372 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.372+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.372 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.372+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3466405415}) 2024-09-16T15:01:44.372 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.372+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3721590738}) 2024-09-16T15:01:44.372 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.372+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.372+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3466405415}) 2024-09-16T15:01:44.373 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.372+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3178515209}) 2024-09-16T15:01:44.374 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.372+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.374 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.372+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1409764883}) 2024-09-16T15:01:44.374 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.372+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2422342759}) 2024-09-16T15:01:44.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.372+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.372+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1409764883}) 2024-09-16T15:01:44.375 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.372+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1428798927}) 2024-09-16T15:01:44.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.376+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.376+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1982635990}) 2024-09-16T15:01:44.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.376+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494830488}) 2024-09-16T15:01:44.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.376+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.376+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1982635990}) 2024-09-16T15:01:44.377 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.376+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4150925871}) 2024-09-16T15:01:44.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.376+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.376+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3151905177}) 2024-09-16T15:01:44.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.376+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4280761462}) 2024-09-16T15:01:44.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.376+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.376+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3151905177}) 2024-09-16T15:01:44.379 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.376+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2648381929}) 2024-09-16T15:01:44.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.380+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.380+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3543651431}) 2024-09-16T15:01:44.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.380+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2793715239}) 2024-09-16T15:01:44.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.380+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.380+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3543651431}) 2024-09-16T15:01:44.381 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.380+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1139187481}) 2024-09-16T15:01:44.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.384+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.384+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3007771952}) 2024-09-16T15:01:44.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.384+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511958768}) 2024-09-16T15:01:44.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.384+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.384+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3007771952}) 2024-09-16T15:01:44.383 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.384+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3721457275}) 2024-09-16T15:01:44.385 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.384+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.386 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.384+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1579954646}) 2024-09-16T15:01:44.386 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.384+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1433176854}) 2024-09-16T15:01:44.386 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.384+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.386 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.384+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1579954646}) 2024-09-16T15:01:44.386 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.384+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4039433394}) 2024-09-16T15:01:44.388 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.388+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.388 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.388+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4177358553}) 2024-09-16T15:01:44.388 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.388+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1409492147}) 2024-09-16T15:01:44.388 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.388+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.388 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.388+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4177358553}) 2024-09-16T15:01:44.388 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.388+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=632867495}) 2024-09-16T15:01:44.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.388+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.388+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1046621254}) 2024-09-16T15:01:44.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.388+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1631856041}) 2024-09-16T15:01:44.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.388+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.388+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1046621254}) 2024-09-16T15:01:44.390 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.388+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3799828612}) 2024-09-16T15:01:44.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.392+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.392+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=48133830}) 2024-09-16T15:01:44.392 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.392+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2113790092}) 2024-09-16T15:01:44.393 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.392+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.393 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.392+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=48133830}) 2024-09-16T15:01:44.393 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.392+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=173528378}) 2024-09-16T15:01:44.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.392+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.392+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1902203549}) 2024-09-16T15:01:44.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.392+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335174458}) 2024-09-16T15:01:44.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.392+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.392+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1902203549}) 2024-09-16T15:01:44.395 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.392+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1509346648}) 2024-09-16T15:01:44.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.396+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.396+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3392718603}) 2024-09-16T15:01:44.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.396+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2559967978}) 2024-09-16T15:01:44.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.396+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.396+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3392718603}) 2024-09-16T15:01:44.397 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.396+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3145882913}) 2024-09-16T15:01:44.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.400+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.400+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=16,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=38} crc {logm=2077948063,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=492275907,mon_config_key=3576433448,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3312862165}) 2024-09-16T15:01:44.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.400+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=16,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=37} crc {logm=2077948063,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2278984543,mon_config_key=3840829532,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=2193850397}) 2024-09-16T15:01:44.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.400+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.400+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=16,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=38} crc {logm=2077948063,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=492275907,mon_config_key=3576433448,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3312862165}) 2024-09-16T15:01:44.401 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.400+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=13,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=40} crc {logm=3988124143,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=2278984543,mon_config_key=2512756450,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1817018960}) 2024-09-16T15:01:44.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.416+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.416+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=11} crc {osdmap=4203315398}) 2024-09-16T15:01:44.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.416+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=13} crc {osdmap=506183497}) 2024-09-16T15:01:44.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.416+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:01:44.416 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.416+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=11} crc {osdmap=4203315398}) 2024-09-16T15:01:44.417 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.416+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=12} crc {osdmap=2721513729}) 2024-09-16T15:01:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T15:01:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T15:01:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T15:01:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:+ rm /tmp/cephtest-mon-caps-madness.bazar.keyring /tmp/cephtest-mon-caps-madness.foo.keyring /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-16T15:01:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:+ cat 2024-09-16T15:01:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph auth import -i - 2024-09-16T15:01:44.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.593+0000 7f3047049640 10 monclient: get_monmap_and_config 2024-09-16T15:01:44.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.593+0000 7f3047049640 10 monclient: build_initial_monmap 2024-09-16T15:01:44.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.593+0000 7f3047049640 10 monclient: monmap: 2024-09-16T15:01:44.593 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T15:01:44.593 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:44.593 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T15:01:44.597449+0000 2024-09-16T15:01:44.594 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T15:01:44.597449+0000 2024-09-16T15:01:44.594 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:44.594 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:44.594 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:44.594 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:44.594 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:44.594 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:44.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 1 Processor -- start 2024-09-16T15:01:44.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 1 -- start start 2024-09-16T15:01:44.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 10 monclient: init 2024-09-16T15:01:44.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:44.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:01:44.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 10 monclient(hunting): picked mon.noname-c con 0x7f3040134cf0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:44.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 10 monclient(hunting): picked mon.noname-a con 0x7f304009f4b0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:44.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 10 monclient(hunting): picked mon.noname-b con 0x7f304013b170 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:44.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304009f8c0 con 0x7f304009f4b0 2024-09-16T15:01:44.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304009e7f0 con 0x7f3040134cf0 2024-09-16T15:01:44.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304009ead0 con 0x7f304013b170 2024-09-16T15:01:44.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:44.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 10 monclient(hunting): authenticate will time out at 2086.713157s 2024-09-16T15:01:44.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3044de5640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3040134cf0 0x7f30401350d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36270/0 (socket says 172.21.15.189:36270) 2024-09-16T15:01:44.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3044de5640 1 -- 172.21.15.189:0/1710997369 learned_addr learned my addr 172.21.15.189:0/1710997369 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T15:01:44.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 1 -- 172.21.15.189:0/1710997369 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858014194 0 0) 0x7f304009e7f0 con 0x7f3040134cf0 2024-09-16T15:01:44.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:44.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:44.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:44.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 10 monclient(hunting): my global_id is 344245 2024-09-16T15:01:44.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 1 -- 172.21.15.189:0/1710997369 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3028003530 con 0x7f3040134cf0 2024-09-16T15:01:44.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 1 -- 172.21.15.189:0/1710997369 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1041581616 0 0) 0x7f3028003530 con 0x7f3040134cf0 2024-09-16T15:01:44.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 1 -- 172.21.15.189:0/1710997369 >> v1:172.21.15.26:6790/0 conn(0x7f304013b170 legacy=0x7f304013d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:44.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 1 -- 172.21.15.189:0/1710997369 >> v1:172.21.15.26:6789/0 conn(0x7f304009f4b0 legacy=0x7f3040139a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:44.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 10 monclient: _finish_hunting 0 2024-09-16T15:01:44.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 1 monclient: found mon.noname-c 2024-09-16T15:01:44.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:44.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T15:01:44.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 1 -- 172.21.15.189:0/1710997369 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304009edb0 con 0x7f3040134cf0 2024-09-16T15:01:44.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 10 monclient: _finish_auth 0 2024-09-16T15:01:44.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 10 monclient: _check_auth_tickets 2024-09-16T15:01:44.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:44.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 1 -- 172.21.15.189:0/1710997369 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3034001620 con 0x7f3040134cf0 2024-09-16T15:01:44.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:44.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:01:44.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 10 monclient: dump: 2024-09-16T15:01:44.603 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:44.603 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:44.603 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:44.604 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:44.604 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:44.604 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:44.604 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:44.604 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:44.604 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:44.604 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:44.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 5 monclient: authenticate success, global_id 344245 2024-09-16T15:01:44.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:44.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 1 -- 172.21.15.189:0/1710997369 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3034001940 con 0x7f3040134cf0 2024-09-16T15:01:44.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:44.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 1 -- 172.21.15.189:0/1710997369 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3034001cd0 con 0x7f3040134cf0 2024-09-16T15:01:44.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:44.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:44.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f303f7fe640 10 monclient: dump: 2024-09-16T15:01:44.605 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:44.605 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:44.605 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:44.605 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:44.605 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:44.605 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:44.605 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:44.605 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:44.605 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:44.605 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:44.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:44.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 4 set_mon_vals no callback set 2024-09-16T15:01:44.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 10 monclient: shutdown 2024-09-16T15:01:44.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:44.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 1 -- 172.21.15.189:0/1710997369 >> v1:172.21.15.189:6789/0 conn(0x7f3040134cf0 legacy=0x7f30401350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:44.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 1 -- 172.21.15.189:0/1710997369 shutdown_connections 2024-09-16T15:01:44.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 1 -- 172.21.15.189:0/1710997369 >> 172.21.15.189:0/1710997369 conn(0x7f304009faa0 msgr2=0x7f304009feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:44.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 1 -- 172.21.15.189:0/1710997369 shutdown_connections 2024-09-16T15:01:44.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 1 -- 172.21.15.189:0/1710997369 wait complete. 2024-09-16T15:01:44.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 10 monclient: build_initial_monmap 2024-09-16T15:01:44.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 10 monclient: monmap: 2024-09-16T15:01:44.606 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T15:01:44.606 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:44.606 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T15:01:44.607 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T15:01:44.607 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:44.607 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T15:01:44.607 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:44.607 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:44.607 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:44.607 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:44.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 1 Processor -- start 2024-09-16T15:01:44.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 1 -- start start 2024-09-16T15:01:44.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.597+0000 7f3047049640 10 monclient: init 2024-09-16T15:01:44.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:44.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:01:44.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 10 monclient(hunting): picked mon.noname-b con 0x7f304009f4b0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:44.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 10 monclient(hunting): picked mon.noname-c con 0x7f3040134cf0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:44.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 10 monclient(hunting): picked mon.noname-a con 0x7f304013b170 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:44.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304011bdf0 con 0x7f304013b170 2024-09-16T15:01:44.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304011d230 con 0x7f304009f4b0 2024-09-16T15:01:44.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304011d510 con 0x7f3040134cf0 2024-09-16T15:01:44.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:44.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 10 monclient(hunting): authenticate will time out at 2086.716282s 2024-09-16T15:01:44.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3044de5640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f304009f4b0 0x7f3040131b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36272/0 (socket says 172.21.15.189:36272) 2024-09-16T15:01:44.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3044de5640 1 -- 172.21.15.189:0/2075415413 learned_addr learned my addr 172.21.15.189:0/2075415413 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T15:01:44.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087032163 0 0) 0x7f304011d230 con 0x7f304009f4b0 2024-09-16T15:01:44.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): _negotiate 2024-09-16T15:01:44.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): my global_id is 344248 2024-09-16T15:01:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f302c003530 con 0x7f304009f4b0 2024-09-16T15:01:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1188500939 0 0) 0x7f302c003530 con 0x7f304009f4b0 2024-09-16T15:01:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f304011d230 con 0x7f304009f4b0 2024-09-16T15:01:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f30340015e0 con 0x7f304009f4b0 2024-09-16T15:01:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:01:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): dump: 2024-09-16T15:01:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:44.610 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:44.610 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:44.610 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:44.610 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:44.610 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:44.610 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:44.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:44.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3044de5640 1 -- 172.21.15.189:0/2075415413 >> v1:172.21.15.189:6789/0 conn(0x7f304009f4b0 legacy=0x7f3040131b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T15:01:44.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3044de5640 1 -- 172.21.15.189:0/2075415413 >> v1:172.21.15.189:6789/0 conn(0x7f304009f4b0 legacy=0x7f3040131b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T15:01:44.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3044de5640 1 --1- 172.21.15.189:0/2075415413 >> v1:172.21.15.189:6789/0 conn(0x7f304009f4b0 0x7f3040131b70 :-1 s=OPENED pgs=1575 cs=1 l=1).handle_message read tag failed 2024-09-16T15:01:44.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3044de5640 1 --1- 172.21.15.189:0/2075415413 >> v1:172.21.15.189:6789/0 conn(0x7f304009f4b0 0x7f3040131b70 :-1 s=OPENED pgs=1575 cs=1 l=1).fault on lossy channel, failing 2024-09-16T15:01:44.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.189:6789/0 2024-09-16T15:01:44.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975779630 0 0) 0x7f304011d510 con 0x7f3040134cf0 2024-09-16T15:01:44.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): _negotiate 2024-09-16T15:01:44.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:44.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:44.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): my global_id is 354140 2024-09-16T15:01:44.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f302c003530 con 0x7f3040134cf0 2024-09-16T15:01:44.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985193409 0 0) 0x7f304011bdf0 con 0x7f304013b170 2024-09-16T15:01:44.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): _negotiate 2024-09-16T15:01:44.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:44.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:44.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): my global_id is 334338 2024-09-16T15:01:44.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f304011d510 con 0x7f304013b170 2024-09-16T15:01:44.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1421257158 0 0) 0x7f302c003530 con 0x7f3040134cf0 2024-09-16T15:01:44.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f304011bdf0 con 0x7f3040134cf0 2024-09-16T15:01:44.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f30300017c0 con 0x7f3040134cf0 2024-09-16T15:01:44.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:44.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:44.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): dump: 2024-09-16T15:01:44.612 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:44.612 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:44.612 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:44.612 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:44.612 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:44.612 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:44.612 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:44.613 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:44.613 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:44.613 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:44.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1487205810 0 0) 0x7f304011d510 con 0x7f304013b170 2024-09-16T15:01:44.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f302c003530 con 0x7f304013b170 2024-09-16T15:01:44.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3038001a70 con 0x7f304013b170 2024-09-16T15:01:44.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:44.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:44.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient(hunting): dump: 2024-09-16T15:01:44.613 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:44.613 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:44.613 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:44.614 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:44.614 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:44.614 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:44.614 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:44.614 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:44.614 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:44.614 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:44.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1443928113 0 0) 0x7f304011bdf0 con 0x7f3040134cf0 2024-09-16T15:01:44.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 >> v1:172.21.15.189:6789/0 conn(0x7f304009f4b0 legacy=0x7f3040131b70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:44.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 >> v1:172.21.15.26:6789/0 conn(0x7f304013b170 legacy=0x7f30401d1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:44.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient: _finish_hunting 0 2024-09-16T15:01:44.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 monclient: found mon.c 2024-09-16T15:01:44.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:44.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:44.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30400a2040 con 0x7f3040134cf0 2024-09-16T15:01:44.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient: _finish_auth 0 2024-09-16T15:01:44.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient: _check_auth_tickets 2024-09-16T15:01:44.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:44.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 5 monclient: authenticate success, global_id 354140 2024-09-16T15:01:44.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 10 monclient: _renew_subs 2024-09-16T15:01:44.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:44.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 1 -- 172.21.15.189:0/2075415413 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f304013ad50 con 0x7f3040134cf0 2024-09-16T15:01:44.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 10 monclient: _renew_subs 2024-09-16T15:01:44.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:44.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 1 -- 172.21.15.189:0/2075415413 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f304011cf50 con 0x7f3040134cf0 2024-09-16T15:01:44.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3030003600 con 0x7f3040134cf0 2024-09-16T15:01:44.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:44.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3030003910 con 0x7f3040134cf0 2024-09-16T15:01:44.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:44.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:44.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303effd640 4 set_mon_vals no callback set 2024-09-16T15:01:44.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 10 monclient: dump: 2024-09-16T15:01:44.616 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:44.616 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:44.616 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:44.616 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:44.616 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:44.616 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:44.617 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:44.617 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:44.617 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:44.617 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:44.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:44.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:44.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:44.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3047049640 1 -- 172.21.15.189:0/2075415413 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3020005050 con 0x7f3040134cf0 2024-09-16T15:01:44.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f303000e380 con 0x7f3040134cf0 2024-09-16T15:01:44.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f30300428e0 con 0x7f3040134cf0 2024-09-16T15:01:44.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.601+0000 7f3044de5640 10 monclient: get_auth_request con 0x7f302c0364c0 auth_method 0 2024-09-16T15:01:44.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.617+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3030013130 con 0x7f3040134cf0 2024-09-16T15:01:44.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.617+0000 7f303d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:44.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.617+0000 7f303d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:44.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.829+0000 7f3047049640 10 start_mon_command cmd=[{"prefix": "auth import"}] 2024-09-16T15:01:44.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.829+0000 7f3047049640 10 monclient: _send_command 2 [{"prefix": "auth import"}] 2024-09-16T15:01:44.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.829+0000 7f3047049640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:44.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.829+0000 7f3047049640 1 -- 172.21.15.189:0/2075415413 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "auth import"} v 0) v1 -- 0x7f3020002400 con 0x7f3040134cf0 2024-09-16T15:01:44.826 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:01:44.824+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 auth entities have invalid capabilities (AUTH_BAD_CAPS) 2024-09-16T15:01:44.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.833+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "auth import"}]=0 v47) v1 ==== 59+0+0 (unknown 2235118774 0 0) 0x7f3030012f50 con 0x7f3040134cf0 2024-09-16T15:01:44.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.833+0000 7f303d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth import"}] 2024-09-16T15:01:44.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.833+0000 7f303d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:44.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.977+0000 7f3047049640 10 start_mon_command cmd=[{"prefix": "auth import"}] 2024-09-16T15:01:44.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.977+0000 7f3047049640 10 monclient: _send_command 3 [{"prefix": "auth import"}] 2024-09-16T15:01:44.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.977+0000 7f3047049640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:44.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.977+0000 7f3047049640 1 -- 172.21.15.189:0/2075415413 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "auth import"} v 0) v1 -- 0x7f30200020b0 con 0x7f3040134cf0 2024-09-16T15:01:44.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.985+0000 7f303d7fa640 1 -- 172.21.15.189:0/2075415413 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "auth import"}]=0 v48) v1 ==== 59+0+0 (unknown 3845246904 0 0) 0x7f3030012f50 con 0x7f3040134cf0 2024-09-16T15:01:44.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.985+0000 7f303d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth import"}] 2024-09-16T15:01:44.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.985+0000 7f303d7fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T15:01:44.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.985+0000 7f3047049640 1 -- 172.21.15.189:0/2075415413 >> v1:172.21.15.189:6812/3196161582 conn(0x7f302c0364c0 legacy=0x7f302c038980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:44.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.985+0000 7f3047049640 10 monclient: shutdown 2024-09-16T15:01:44.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.985+0000 7f3047049640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:44.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.985+0000 7f3047049640 1 -- 172.21.15.189:0/2075415413 >> v1:172.21.15.26:6790/0 conn(0x7f3040134cf0 legacy=0x7f30401ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:44.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.985+0000 7f3047049640 1 -- 172.21.15.189:0/2075415413 shutdown_connections 2024-09-16T15:01:44.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.985+0000 7f3047049640 1 -- 172.21.15.189:0/2075415413 >> 172.21.15.189:0/2075415413 conn(0x7f304009faa0 msgr2=0x7f3040133980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:44.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.985+0000 7f3047049640 1 -- 172.21.15.189:0/2075415413 shutdown_connections 2024-09-16T15:01:44.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:44.985+0000 7f3047049640 1 -- 172.21.15.189:0/2075415413 wait complete. 2024-09-16T15:01:44.997 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph health 2024-09-16T15:01:44.997 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep AUTH_BAD_CAP 2024-09-16T15:01:45.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 10 monclient: get_monmap_and_config 2024-09-16T15:01:45.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 10 monclient: build_initial_monmap 2024-09-16T15:01:45.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 10 monclient: monmap: 2024-09-16T15:01:45.103 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T15:01:45.103 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.103 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T15:01:45.107519+0000 2024-09-16T15:01:45.103 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T15:01:45.107519+0000 2024-09-16T15:01:45.103 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:45.103 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:45.103 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:45.103 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:45.104 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:45.104 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:45.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 1 Processor -- start 2024-09-16T15:01:45.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 1 -- start start 2024-09-16T15:01:45.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 10 monclient: init 2024-09-16T15:01:45.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:45.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:01:45.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 10 monclient(hunting): picked mon.noname-b con 0x7fb64409d9a0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:45.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 10 monclient(hunting): picked mon.noname-c con 0x7fb64409e490 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:45.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 10 monclient(hunting): picked mon.noname-a con 0x7fb644139360 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:45.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6440712d0 con 0x7fb644139360 2024-09-16T15:01:45.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64409ed30 con 0x7fb64409e490 2024-09-16T15:01:45.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb644099ef0 con 0x7fb64409d9a0 2024-09-16T15:01:45.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:45.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 10 monclient(hunting): authenticate will time out at 2087.223155s 2024-09-16T15:01:45.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb643fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb64409e490 0x7fb644137c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36274/0 (socket says 172.21.15.189:36274) 2024-09-16T15:01:45.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb643fff640 1 -- 172.21.15.189:0/2795353368 learned_addr learned my addr 172.21.15.189:0/2795353368 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T15:01:45.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 1 -- 172.21.15.189:0/2795353368 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169346442 0 0) 0x7fb64409ed30 con 0x7fb64409e490 2024-09-16T15:01:45.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 10 monclient(hunting): my global_id is 344251 2024-09-16T15:01:45.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 1 -- 172.21.15.189:0/2795353368 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb62c003530 con 0x7fb64409e490 2024-09-16T15:01:45.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 1 -- 172.21.15.189:0/2795353368 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2319002785 0 0) 0x7fb62c003530 con 0x7fb64409e490 2024-09-16T15:01:45.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 1 -- 172.21.15.189:0/2795353368 >> v1:172.21.15.26:6790/0 conn(0x7fb64409d9a0 legacy=0x7fb64409dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 1 -- 172.21.15.189:0/2795353368 >> v1:172.21.15.26:6789/0 conn(0x7fb644139360 legacy=0x7fb64413b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 10 monclient: _finish_hunting 0 2024-09-16T15:01:45.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 1 monclient: found mon.noname-c 2024-09-16T15:01:45.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:45.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T15:01:45.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 1 -- 172.21.15.189:0/2795353368 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64409c760 con 0x7fb64409e490 2024-09-16T15:01:45.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 10 monclient: _finish_auth 0 2024-09-16T15:01:45.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 10 monclient: _check_auth_tickets 2024-09-16T15:01:45.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:45.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 1 -- 172.21.15.189:0/2795353368 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb634001620 con 0x7fb64409e490 2024-09-16T15:01:45.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 5 monclient: authenticate success, global_id 344251 2024-09-16T15:01:45.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:45.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:01:45.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb6437fe640 10 monclient: dump: 2024-09-16T15:01:45.113 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:45.113 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.113 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.113 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.113 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.113 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:45.113 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.113 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.113 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.113 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:45.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.105+0000 7fb64ac66640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:45.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6437fe640 1 -- 172.21.15.189:0/2795353368 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6340018d0 con 0x7fb64409e490 2024-09-16T15:01:45.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6437fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6437fe640 1 -- 172.21.15.189:0/2795353368 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb634001ca0 con 0x7fb64409e490 2024-09-16T15:01:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 4 set_mon_vals no callback set 2024-09-16T15:01:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6437fe640 10 monclient: dump: 2024-09-16T15:01:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.115 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.115 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:45.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient: shutdown 2024-09-16T15:01:45.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:45.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 1 -- 172.21.15.189:0/2795353368 >> v1:172.21.15.189:6789/0 conn(0x7fb64409e490 legacy=0x7fb644137c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 1 -- 172.21.15.189:0/2795353368 shutdown_connections 2024-09-16T15:01:45.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 1 -- 172.21.15.189:0/2795353368 >> 172.21.15.189:0/2795353368 conn(0x7fb64409faa0 msgr2=0x7fb64409feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:45.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 1 -- 172.21.15.189:0/2795353368 shutdown_connections 2024-09-16T15:01:45.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 1 -- 172.21.15.189:0/2795353368 wait complete. 2024-09-16T15:01:45.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient: build_initial_monmap 2024-09-16T15:01:45.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient: monmap: 2024-09-16T15:01:45.115 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T15:01:45.115 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:45.115 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T15:01:45.115 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T15:01:45.116 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:45.116 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T15:01:45.116 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:45.116 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:45.116 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:45.116 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:45.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 1 Processor -- start 2024-09-16T15:01:45.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 1 -- start start 2024-09-16T15:01:45.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient: init 2024-09-16T15:01:45.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:45.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:01:45.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient(hunting): picked mon.noname-a con 0x7fb64409d9a0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:45.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient(hunting): picked mon.noname-b con 0x7fb64409e490 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:45.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient(hunting): picked mon.noname-c con 0x7fb644139360 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:45.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb644139030 con 0x7fb64409d9a0 2024-09-16T15:01:45.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb644138c70 con 0x7fb64409e490 2024-09-16T15:01:45.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64409a190 con 0x7fb644139360 2024-09-16T15:01:45.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:45.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient(hunting): authenticate will time out at 2087.226029s 2024-09-16T15:01:45.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb643fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb64409e490 0x7fb6441ce740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36282/0 (socket says 172.21.15.189:36282) 2024-09-16T15:01:45.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb643fff640 1 -- 172.21.15.189:0/3791169008 learned_addr learned my addr 172.21.15.189:0/3791169008 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T15:01:45.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 1 -- 172.21.15.189:0/3791169008 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274267408 0 0) 0x7fb644138c70 con 0x7fb64409e490 2024-09-16T15:01:45.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 10 monclient(hunting): my global_id is 344254 2024-09-16T15:01:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 1 -- 172.21.15.189:0/3791169008 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb630003530 con 0x7fb64409e490 2024-09-16T15:01:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 1 -- 172.21.15.189:0/3791169008 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 608375454 0 0) 0x7fb630003530 con 0x7fb64409e490 2024-09-16T15:01:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 1 -- 172.21.15.189:0/3791169008 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb644138c70 con 0x7fb64409e490 2024-09-16T15:01:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 1 -- 172.21.15.189:0/3791169008 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb6340015e0 con 0x7fb64409e490 2024-09-16T15:01:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:01:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 10 monclient(hunting): dump: 2024-09-16T15:01:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:45.119 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.119 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.119 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.119 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:45.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:45.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 1 -- 172.21.15.189:0/3791169008 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3074421884 0 0) 0x7fb644138c70 con 0x7fb64409e490 2024-09-16T15:01:45.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 1 -- 172.21.15.189:0/3791169008 >> v1:172.21.15.26:6790/0 conn(0x7fb644139360 legacy=0x7fb6441d1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 1 -- 172.21.15.189:0/3791169008 >> v1:172.21.15.26:6789/0 conn(0x7fb64409d9a0 legacy=0x7fb6441c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 10 monclient: _finish_hunting 0 2024-09-16T15:01:45.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 1 monclient: found mon.b 2024-09-16T15:01:45.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:45.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:45.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 1 -- 172.21.15.189:0/3791169008 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64411fe00 con 0x7fb64409e490 2024-09-16T15:01:45.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 10 monclient: _finish_auth 0 2024-09-16T15:01:45.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 10 monclient: _check_auth_tickets 2024-09-16T15:01:45.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:45.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 5 monclient: authenticate success, global_id 344254 2024-09-16T15:01:45.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient: _renew_subs 2024-09-16T15:01:45.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:45.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 1 -- 172.21.15.189:0/3791169008 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6441200e0 con 0x7fb64409e490 2024-09-16T15:01:45.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient: _renew_subs 2024-09-16T15:01:45.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:45.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 1 -- 172.21.15.189:0/3791169008 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64409b4b0 con 0x7fb64409e490 2024-09-16T15:01:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 1 -- 172.21.15.189:0/3791169008 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb634004c50 con 0x7fb64409e490 2024-09-16T15:01:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 1 -- 172.21.15.189:0/3791169008 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fb634004fe0 con 0x7fb64409e490 2024-09-16T15:01:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb642ffd640 4 set_mon_vals no callback set 2024-09-16T15:01:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 10 monclient: dump: 2024-09-16T15:01:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:45.122 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.122 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.122 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.122 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:45.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:45.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:45.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:45.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb64ac66640 1 -- 172.21.15.189:0/3791169008 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb624005050 con 0x7fb64409e490 2024-09-16T15:01:45.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 1 -- 172.21.15.189:0/3791169008 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fb634005140 con 0x7fb64409e490 2024-09-16T15:01:45.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.109+0000 7fb6417fa640 1 -- 172.21.15.189:0/3791169008 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fb6340440d0 con 0x7fb64409e490 2024-09-16T15:01:45.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.113+0000 7fb648a02640 10 monclient: get_auth_request con 0x7fb630036010 auth_method 0 2024-09-16T15:01:45.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.117+0000 7fb6417fa640 1 -- 172.21.15.189:0/3791169008 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb634006730 con 0x7fb64409e490 2024-09-16T15:01:45.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.117+0000 7fb6417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:45.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.117+0000 7fb6417fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:45.130 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T15:01:45.131 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T15:01:45.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 10 monclient: get_monmap_and_config 2024-09-16T15:01:45.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 10 monclient: build_initial_monmap 2024-09-16T15:01:45.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 10 monclient: monmap: 2024-09-16T15:01:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:45.262794+0000 2024-09-16T15:01:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:45.262794+0000 2024-09-16T15:01:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:45.261 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:45.261 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 1 Processor -- start 2024-09-16T15:01:45.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 1 -- start start 2024-09-16T15:01:45.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 10 monclient: init 2024-09-16T15:01:45.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:45.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:01:45.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 10 monclient(hunting): picked mon.noname-a con 0x7fc3c009cb80 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:45.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 10 monclient(hunting): picked mon.noname-c con 0x7fc3c0132650 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:45.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 10 monclient(hunting): picked mon.noname-b con 0x7fc3c0138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:45.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c009d260 con 0x7fc3c009cb80 2024-09-16T15:01:45.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c0132a80 con 0x7fc3c0132650 2024-09-16T15:01:45.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c009cf60 con 0x7fc3c0138ff0 2024-09-16T15:01:45.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:45.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 10 monclient(hunting): authenticate will time out at 2129.382913s 2024-09-16T15:01:45.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3beffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc3c009cb80 0x7fc3c00717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39962/0 (socket says 172.21.15.26:39962) 2024-09-16T15:01:45.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3beffd640 1 -- 172.21.15.26:0/3301851982 learned_addr learned my addr 172.21.15.26:0/3301851982 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:45.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 1 -- 172.21.15.26:0/3301851982 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913149734 0 0) 0x7fc3c009d260 con 0x7fc3c009cb80 2024-09-16T15:01:45.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 10 monclient(hunting): my global_id is 334347 2024-09-16T15:01:45.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 1 -- 172.21.15.26:0/3301851982 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ac003530 con 0x7fc3c009cb80 2024-09-16T15:01:45.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 1 -- 172.21.15.26:0/3301851982 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811243117 0 0) 0x7fc3c009cf60 con 0x7fc3c0138ff0 2024-09-16T15:01:45.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 10 monclient(hunting): my global_id is 354149 2024-09-16T15:01:45.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 1 -- 172.21.15.26:0/3301851982 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c009d260 con 0x7fc3c0138ff0 2024-09-16T15:01:45.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 1 -- 172.21.15.26:0/3301851982 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2639471717 0 0) 0x7fc3ac003530 con 0x7fc3c009cb80 2024-09-16T15:01:45.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 1 -- 172.21.15.26:0/3301851982 >> v1:172.21.15.26:6790/0 conn(0x7fc3c0138ff0 legacy=0x7fc3c013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 1 -- 172.21.15.26:0/3301851982 >> v1:172.21.15.189:6789/0 conn(0x7fc3c0132650 legacy=0x7fc3c01378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 10 monclient: _finish_hunting 0 2024-09-16T15:01:45.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 1 monclient: found mon.noname-a 2024-09-16T15:01:45.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:45.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:01:45.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 1 -- 172.21.15.26:0/3301851982 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c009f2d0 con 0x7fc3c009cb80 2024-09-16T15:01:45.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 10 monclient: _finish_auth 0 2024-09-16T15:01:45.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:45.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:45.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 5 monclient: authenticate success, global_id 334347 2024-09-16T15:01:45.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:45.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 1 -- 172.21.15.26:0/3301851982 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc3a8001620 con 0x7fc3c009cb80 2024-09-16T15:01:45.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:45.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 10 monclient: dump: 2024-09-16T15:01:45.271 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.272 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.272 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.272 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.272 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.272 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.272 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.272 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.272 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.272 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3c59a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:45.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.260+0000 7fc3bdffb640 1 -- 172.21.15.26:0/3301851982 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3a8001850 con 0x7fc3c009cb80 2024-09-16T15:01:45.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3bdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:45.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3bdffb640 1 -- 172.21.15.26:0/3301851982 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc3a8001be0 con 0x7fc3c009cb80 2024-09-16T15:01:45.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:45.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3bdffb640 10 monclient: dump: 2024-09-16T15:01:45.273 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.273 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.273 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.273 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.273 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.273 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.273 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.273 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.273 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.273 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:45.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 4 set_mon_vals no callback set 2024-09-16T15:01:45.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 10 monclient: shutdown 2024-09-16T15:01:45.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:45.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 1 -- 172.21.15.26:0/3301851982 >> v1:172.21.15.26:6789/0 conn(0x7fc3c009cb80 legacy=0x7fc3c00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 1 -- 172.21.15.26:0/3301851982 shutdown_connections 2024-09-16T15:01:45.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 1 -- 172.21.15.26:0/3301851982 >> 172.21.15.26:0/3301851982 conn(0x7fc3c009fb50 msgr2=0x7fc3c009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:45.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 1 -- 172.21.15.26:0/3301851982 shutdown_connections 2024-09-16T15:01:45.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 1 -- 172.21.15.26:0/3301851982 wait complete. 2024-09-16T15:01:45.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 10 monclient: build_initial_monmap 2024-09-16T15:01:45.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 10 monclient: monmap: 2024-09-16T15:01:45.274 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:45.274 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:45.274 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:45.274 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:45.274 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:45.275 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:45.275 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:45.275 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:45.275 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:45.275 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 1 Processor -- start 2024-09-16T15:01:45.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 1 -- start start 2024-09-16T15:01:45.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 10 monclient: init 2024-09-16T15:01:45.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:45.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:01:45.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 10 monclient(hunting): picked mon.noname-a con 0x7fc3c009cb80 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:45.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 10 monclient(hunting): picked mon.noname-c con 0x7fc3c0132650 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:45.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 10 monclient(hunting): picked mon.noname-b con 0x7fc3c0138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:45.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c006e010 con 0x7fc3c009cb80 2024-09-16T15:01:45.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c009b370 con 0x7fc3c0138ff0 2024-09-16T15:01:45.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c0138b50 con 0x7fc3c0132650 2024-09-16T15:01:45.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:45.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 10 monclient(hunting): authenticate will time out at 2129.386274s 2024-09-16T15:01:45.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3be7fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc3c0132650 0x7fc3c009ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:56748/0 (socket says 172.21.15.26:56748) 2024-09-16T15:01:45.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3be7fc640 1 -- 172.21.15.26:0/1962528233 learned_addr learned my addr 172.21.15.26:0/1962528233 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:45.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353892913 0 0) 0x7fc3c0138b50 con 0x7fc3c0132650 2024-09-16T15:01:45.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): my global_id is 354152 2024-09-16T15:01:45.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3a0003530 con 0x7fc3c0132650 2024-09-16T15:01:45.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562233405 0 0) 0x7fc3c006e010 con 0x7fc3c009cb80 2024-09-16T15:01:45.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): my global_id is 334350 2024-09-16T15:01:45.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c0138b50 con 0x7fc3c009cb80 2024-09-16T15:01:45.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130730581 0 0) 0x7fc3c009b370 con 0x7fc3c0138ff0 2024-09-16T15:01:45.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): my global_id is 344260 2024-09-16T15:01:45.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c006e010 con 0x7fc3c0138ff0 2024-09-16T15:01:45.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3704674080 0 0) 0x7fc3a0003530 con 0x7fc3c0132650 2024-09-16T15:01:45.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3c009b370 con 0x7fc3c0132650 2024-09-16T15:01:45.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc3b0001d00 con 0x7fc3c0132650 2024-09-16T15:01:45.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:01:45.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): dump: 2024-09-16T15:01:45.278 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.278 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.278 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.278 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.278 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.278 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.278 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.278 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.278 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.278 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:45.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1202774790 0 0) 0x7fc3c0138b50 con 0x7fc3c009cb80 2024-09-16T15:01:45.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3a0003530 con 0x7fc3c009cb80 2024-09-16T15:01:45.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc3a8001b60 con 0x7fc3c009cb80 2024-09-16T15:01:45.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:45.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient(hunting): dump: 2024-09-16T15:01:45.279 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.279 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.279 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.279 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.279 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.280 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.280 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.280 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.280 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.280 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4214802849 0 0) 0x7fc3c009b370 con 0x7fc3c0132650 2024-09-16T15:01:45.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 >> v1:172.21.15.189:6789/0 conn(0x7fc3c0138ff0 legacy=0x7fc3c01d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 >> v1:172.21.15.26:6789/0 conn(0x7fc3c009cb80 legacy=0x7fc3c009a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient: _finish_hunting 0 2024-09-16T15:01:45.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 monclient: found mon.c 2024-09-16T15:01:45.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:45.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:45.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c009c3c0 con 0x7fc3c0132650 2024-09-16T15:01:45.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient: _finish_auth 0 2024-09-16T15:01:45.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient: _check_auth_tickets 2024-09-16T15:01:45.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:45.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 5 monclient: authenticate success, global_id 354152 2024-09-16T15:01:45.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 10 monclient: _renew_subs 2024-09-16T15:01:45.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:45.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 1 -- 172.21.15.26:0/1962528233 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c013c470 con 0x7fc3c0132650 2024-09-16T15:01:45.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 10 monclient: _renew_subs 2024-09-16T15:01:45.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:45.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3c59a7640 1 -- 172.21.15.26:0/1962528233 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c0138640 con 0x7fc3c0132650 2024-09-16T15:01:45.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3b00039a0 con 0x7fc3c0132650 2024-09-16T15:01:45.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:45.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc3b0003cb0 con 0x7fc3c0132650 2024-09-16T15:01:45.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:45.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3bd7fa640 4 set_mon_vals no callback set 2024-09-16T15:01:45.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.264+0000 7fc3b7fff640 10 monclient: dump: 2024-09-16T15:01:45.282 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.282 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.282 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.282 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.282 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.282 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.282 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.282 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.283 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.283 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.268+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc3b0003e10 con 0x7fc3c0132650 2024-09-16T15:01:45.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.268+0000 7fc3c59a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:45.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.268+0000 7fc3c59a7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:45.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.268+0000 7fc3c59a7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:45.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.268+0000 7fc3c59a7640 1 -- 172.21.15.26:0/1962528233 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a4005050 con 0x7fc3c0132650 2024-09-16T15:01:45.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.268+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fc3b0044160 con 0x7fc3c0132650 2024-09-16T15:01:45.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.268+0000 7fc3beffd640 10 monclient: get_auth_request con 0x7fc3a00363d0 auth_method 0 2024-09-16T15:01:45.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.272+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc3b0004610 con 0x7fc3c0132650 2024-09-16T15:01:45.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.272+0000 7fc3b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:45.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.272+0000 7fc3b7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:45.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.293+0000 7fb64ac66640 10 start_mon_command cmd=[{"prefix": "health"}] 2024-09-16T15:01:45.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.293+0000 7fb64ac66640 10 monclient: _send_command 2 [{"prefix": "health"}] 2024-09-16T15:01:45.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.293+0000 7fb64ac66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:45.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.293+0000 7fb64ac66640 1 -- 172.21.15.189:0/3791169008 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "health"} v 0) v1 -- 0x7fb624005b10 con 0x7fb64409e490 2024-09-16T15:01:45.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.293+0000 7fb6417fa640 1 -- 172.21.15.189:0/3791169008 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+99 (unknown 2555570491 0 746166484) 0x7fb634012fa0 con 0x7fb64409e490 2024-09-16T15:01:45.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.293+0000 7fb6417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health"}] 2024-09-16T15:01:45.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.293+0000 7fb6417fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:45.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.413+0000 7fb64ac66640 10 start_mon_command cmd=[{"prefix": "health"}] 2024-09-16T15:01:45.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.413+0000 7fb64ac66640 10 monclient: _send_command 3 [{"prefix": "health"}] 2024-09-16T15:01:45.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.413+0000 7fb64ac66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:45.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.413+0000 7fb64ac66640 1 -- 172.21.15.189:0/3791169008 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "health"} v 0) v1 -- 0x7fb6240055a0 con 0x7fb64409e490 2024-09-16T15:01:45.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.413+0000 7fb6417fa640 1 -- 172.21.15.189:0/3791169008 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+99 (unknown 2555570491 0 746166484) 0x7fb634014720 con 0x7fb64409e490 2024-09-16T15:01:45.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.413+0000 7fb6417fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "health"}] 2024-09-16T15:01:45.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.413+0000 7fb6417fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T15:01:45.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.413+0000 7fb64ac66640 1 -- 172.21.15.189:0/3791169008 >> v1:172.21.15.189:6812/3196161582 conn(0x7fb630036010 legacy=0x7fb6300384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.413+0000 7fb64ac66640 10 monclient: shutdown 2024-09-16T15:01:45.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.413+0000 7fb64ac66640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:45.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.413+0000 7fb64ac66640 1 -- 172.21.15.189:0/3791169008 >> v1:172.21.15.189:6789/0 conn(0x7fb64409e490 legacy=0x7fb6441ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.413+0000 7fb64ac66640 1 -- 172.21.15.189:0/3791169008 shutdown_connections 2024-09-16T15:01:45.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.413+0000 7fb64ac66640 1 -- 172.21.15.189:0/3791169008 >> 172.21.15.189:0/3791169008 conn(0x7fb64409faa0 msgr2=0x7fb6440a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:45.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.413+0000 7fb64ac66640 1 -- 172.21.15.189:0/3791169008 shutdown_connections 2024-09-16T15:01:45.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.413+0000 7fb64ac66640 1 -- 172.21.15.189:0/3791169008 wait complete. 2024-09-16T15:01:45.427 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph health detail 2024-09-16T15:01:45.427 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep client.bad 2024-09-16T15:01:45.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.488+0000 7fc3c59a7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T15:01:45.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.488+0000 7fc3c59a7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:45.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.488+0000 7fc3c59a7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:45.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.492+0000 7fc3c59a7640 1 -- 172.21.15.26:0/1962528233 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc3a4005880 con 0x7fc3c0132650 2024-09-16T15:01:45.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.492+0000 7fc3b7fff640 1 -- 172.21.15.26:0/1962528233 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 3920857905) 0x7fc3b0012f10 con 0x7fc3c0132650 2024-09-16T15:01:45.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.492+0000 7fc3b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:45.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.492+0000 7fc3b7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:45.492 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":174,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T15:01:45.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.492+0000 7fc3c59a7640 1 -- 172.21.15.26:0/1962528233 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc3a00363d0 legacy=0x7fc3a0038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.492+0000 7fc3c59a7640 10 monclient: shutdown 2024-09-16T15:01:45.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.492+0000 7fc3c59a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:45.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.492+0000 7fc3c59a7640 1 -- 172.21.15.26:0/1962528233 >> v1:172.21.15.26:6790/0 conn(0x7fc3c0132650 legacy=0x7fc3c009ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.492+0000 7fc3c59a7640 1 -- 172.21.15.26:0/1962528233 shutdown_connections 2024-09-16T15:01:45.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.492+0000 7fc3c59a7640 1 -- 172.21.15.26:0/1962528233 >> 172.21.15.26:0/1962528233 conn(0x7fc3c009fb50 msgr2=0x7fc3c00a2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:45.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.492+0000 7fc3c59a7640 1 -- 172.21.15.26:0/1962528233 shutdown_connections 2024-09-16T15:01:45.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.492+0000 7fc3c59a7640 1 -- 172.21.15.26:0/1962528233 wait complete. 2024-09-16T15:01:45.510 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T15:01:45.510 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T15:01:45.510 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T15:01:45.511 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T15:01:45.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.533+0000 7f3205f12640 10 monclient: get_monmap_and_config 2024-09-16T15:01:45.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.533+0000 7f3205f12640 10 monclient: build_initial_monmap 2024-09-16T15:01:45.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.533+0000 7f3205f12640 10 monclient: monmap: 2024-09-16T15:01:45.533 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T15:01:45.533 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.533 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T15:01:45.537582+0000 2024-09-16T15:01:45.533 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T15:01:45.537582+0000 2024-09-16T15:01:45.533 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:45.533 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:45.533 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:45.533 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:45.533 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:45.534 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:45.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 1 Processor -- start 2024-09-16T15:01:45.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 1 -- start start 2024-09-16T15:01:45.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 10 monclient: init 2024-09-16T15:01:45.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:45.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:01:45.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 10 monclient(hunting): picked mon.noname-c con 0x7f320009d9a0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 10 monclient(hunting): picked mon.noname-b con 0x7f320009e490 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 10 monclient(hunting): picked mon.noname-a con 0x7f3200139360 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32000712d0 con 0x7f3200139360 2024-09-16T15:01:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f320009ed30 con 0x7f320009d9a0 2024-09-16T15:01:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3200099ef0 con 0x7f320009e490 2024-09-16T15:01:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 10 monclient(hunting): authenticate will time out at 2087.653399s 2024-09-16T15:01:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31ff7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f320009d9a0 0x7f320009dd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36296/0 (socket says 172.21.15.189:36296) 2024-09-16T15:01:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31ff7fe640 1 -- 172.21.15.189:0/339231334 learned_addr learned my addr 172.21.15.189:0/339231334 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T15:01:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 1 -- 172.21.15.189:0/339231334 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553082446 0 0) 0x7f320009ed30 con 0x7f320009d9a0 2024-09-16T15:01:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient(hunting): my global_id is 344263 2024-09-16T15:01:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 1 -- 172.21.15.189:0/339231334 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31e8003530 con 0x7f320009d9a0 2024-09-16T15:01:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 1 -- 172.21.15.189:0/339231334 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674204700 0 0) 0x7f32000712d0 con 0x7f3200139360 2024-09-16T15:01:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient(hunting): my global_id is 334353 2024-09-16T15:01:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 1 -- 172.21.15.189:0/339231334 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f320009ed30 con 0x7f3200139360 2024-09-16T15:01:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 1 -- 172.21.15.189:0/339231334 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 675073984 0 0) 0x7f31e8003530 con 0x7f320009d9a0 2024-09-16T15:01:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 1 -- 172.21.15.189:0/339231334 >> v1:172.21.15.26:6790/0 conn(0x7f320009e490 legacy=0x7f3200137c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 1 -- 172.21.15.189:0/339231334 >> v1:172.21.15.26:6789/0 conn(0x7f3200139360 legacy=0x7f320013b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient: _finish_hunting 0 2024-09-16T15:01:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 1 monclient: found mon.noname-c 2024-09-16T15:01:45.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:45.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T15:01:45.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 1 -- 172.21.15.189:0/339231334 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320009c760 con 0x7f320009d9a0 2024-09-16T15:01:45.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient: _finish_auth 0 2024-09-16T15:01:45.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient: _check_auth_tickets 2024-09-16T15:01:45.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:45.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 1 -- 172.21.15.189:0/339231334 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f31f0001620 con 0x7f320009d9a0 2024-09-16T15:01:45.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 5 monclient: authenticate success, global_id 344263 2024-09-16T15:01:45.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:45.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:01:45.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient: dump: 2024-09-16T15:01:45.543 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:45.544 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.544 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.544 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.544 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.544 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:45.544 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.544 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.544 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.544 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:45.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:45.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 1 -- 172.21.15.189:0/339231334 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31f0001900 con 0x7f320009d9a0 2024-09-16T15:01:45.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:45.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 1 -- 172.21.15.189:0/339231334 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f31f0001cd0 con 0x7f320009d9a0 2024-09-16T15:01:45.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:45.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 4 set_mon_vals no callback set 2024-09-16T15:01:45.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:45.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f31fe7fc640 10 monclient: dump: 2024-09-16T15:01:45.545 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:45.545 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.545 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.545 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.545 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.545 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:45.545 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.545 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.546 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.546 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:45.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 10 monclient: shutdown 2024-09-16T15:01:45.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:45.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 1 -- 172.21.15.189:0/339231334 >> v1:172.21.15.189:6789/0 conn(0x7f320009d9a0 legacy=0x7f320009dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 1 -- 172.21.15.189:0/339231334 shutdown_connections 2024-09-16T15:01:45.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 1 -- 172.21.15.189:0/339231334 >> 172.21.15.189:0/339231334 conn(0x7f320009faa0 msgr2=0x7f320009feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:45.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 1 -- 172.21.15.189:0/339231334 shutdown_connections 2024-09-16T15:01:45.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 1 -- 172.21.15.189:0/339231334 wait complete. 2024-09-16T15:01:45.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 10 monclient: build_initial_monmap 2024-09-16T15:01:45.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 10 monclient: monmap: 2024-09-16T15:01:45.546 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T15:01:45.546 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:45.547 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T15:01:45.547 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T15:01:45.547 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:45.547 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T15:01:45.547 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:45.547 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:45.547 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:45.547 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:45.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 1 Processor -- start 2024-09-16T15:01:45.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 1 -- start start 2024-09-16T15:01:45.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.537+0000 7f3205f12640 10 monclient: init 2024-09-16T15:01:45.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:45.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:01:45.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 10 monclient(hunting): picked mon.noname-c con 0x7f320009d9a0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:45.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 10 monclient(hunting): picked mon.noname-b con 0x7f320009e490 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:45.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 10 monclient(hunting): picked mon.noname-a con 0x7f3200139360 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:45.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f320011ff50 con 0x7f3200139360 2024-09-16T15:01:45.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32001ca320 con 0x7f320009e490 2024-09-16T15:01:45.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f320011e860 con 0x7f320009d9a0 2024-09-16T15:01:45.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:45.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 10 monclient(hunting): authenticate will time out at 2087.656478s 2024-09-16T15:01:45.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31feffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f320009e490 0x7f32001c9c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36304/0 (socket says 172.21.15.189:36304) 2024-09-16T15:01:45.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31feffd640 1 -- 172.21.15.189:0/3586545967 learned_addr learned my addr 172.21.15.189:0/3586545967 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T15:01:45.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979039227 0 0) 0x7f32001ca320 con 0x7f320009e490 2024-09-16T15:01:45.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): my global_id is 344266 2024-09-16T15:01:45.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31ec003530 con 0x7f320009e490 2024-09-16T15:01:45.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513336817 0 0) 0x7f320011ff50 con 0x7f3200139360 2024-09-16T15:01:45.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): my global_id is 334356 2024-09-16T15:01:45.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32001ca320 con 0x7f3200139360 2024-09-16T15:01:45.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149205332 0 0) 0x7f320011e860 con 0x7f320009d9a0 2024-09-16T15:01:45.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): my global_id is 354158 2024-09-16T15:01:45.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f320011ff50 con 0x7f320009d9a0 2024-09-16T15:01:45.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1368072213 0 0) 0x7f31ec003530 con 0x7f320009e490 2024-09-16T15:01:45.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f320011e860 con 0x7f320009e490 2024-09-16T15:01:45.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f31f4001920 con 0x7f320009e490 2024-09-16T15:01:45.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:01:45.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): dump: 2024-09-16T15:01:45.550 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:45.550 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.550 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.550 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.550 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.551 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:45.551 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.551 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.551 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.551 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:45.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:45.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4046481577 0 0) 0x7f32001ca320 con 0x7f3200139360 2024-09-16T15:01:45.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31ec003530 con 0x7f3200139360 2024-09-16T15:01:45.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f31f8001c40 con 0x7f3200139360 2024-09-16T15:01:45.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:45.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient(hunting): dump: 2024-09-16T15:01:45.552 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:45.552 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.552 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.552 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.552 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.552 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:45.552 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.552 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.552 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.552 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:45.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1873594277 0 0) 0x7f320011e860 con 0x7f320009e490 2024-09-16T15:01:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 >> v1:172.21.15.26:6790/0 conn(0x7f320009d9a0 legacy=0x7f32001c9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 >> v1:172.21.15.26:6789/0 conn(0x7f3200139360 legacy=0x7f32001d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient: _finish_hunting 0 2024-09-16T15:01:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 monclient: found mon.b 2024-09-16T15:01:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320009a190 con 0x7f320009e490 2024-09-16T15:01:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient: _finish_auth 0 2024-09-16T15:01:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient: _check_auth_tickets 2024-09-16T15:01:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 5 monclient: authenticate success, global_id 344266 2024-09-16T15:01:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 10 monclient: _renew_subs 2024-09-16T15:01:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 1 -- 172.21.15.189:0/3586545967 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3200070480 con 0x7f320009e490 2024-09-16T15:01:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 10 monclient: _renew_subs 2024-09-16T15:01:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 1 -- 172.21.15.189:0/3586545967 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f320009c120 con 0x7f320009e490 2024-09-16T15:01:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31f4001d00 con 0x7f320009e490 2024-09-16T15:01:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f31f4003a70 con 0x7f320009e490 2024-09-16T15:01:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31fdffb640 4 set_mon_vals no callback set 2024-09-16T15:01:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 10 monclient: dump: 2024-09-16T15:01:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.555 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.555 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.555 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.555 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:45.555 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.555 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.555 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.555 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:45.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:45.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:45.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:45.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f3205f12640 1 -- 172.21.15.189:0/3586545967 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e0005050 con 0x7f320009e490 2024-09-16T15:01:45.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f31f4003c90 con 0x7f320009e490 2024-09-16T15:01:45.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f31f4043e20 con 0x7f320009e490 2024-09-16T15:01:45.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.541+0000 7f31ff7fe640 10 monclient: get_auth_request con 0x7f31ec036390 auth_method 0 2024-09-16T15:01:45.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.545+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f31f40145f0 con 0x7f320009e490 2024-09-16T15:01:45.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.545+0000 7f31e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:45.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.545+0000 7f31e7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.632+0000 7f5655271640 10 monclient: get_monmap_and_config 2024-09-16T15:01:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.632+0000 7f5655271640 10 monclient: build_initial_monmap 2024-09-16T15:01:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.632+0000 7f5655271640 10 monclient: monmap: 2024-09-16T15:01:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:45.636181+0000 2024-09-16T15:01:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:45.636181+0000 2024-09-16T15:01:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:45.635 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.632+0000 7f5655271640 1 Processor -- start 2024-09-16T15:01:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.632+0000 7f5655271640 1 -- start start 2024-09-16T15:01:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.632+0000 7f5655271640 10 monclient: init 2024-09-16T15:01:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:01:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient(hunting): picked mon.noname-b con 0x7f565009a550 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient(hunting): picked mon.noname-c con 0x7f565009d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient(hunting): picked mon.noname-a con 0x7f565009b040 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565009dc60 con 0x7f565009b040 2024-09-16T15:01:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5650099ae0 con 0x7f565009d1b0 2024-09-16T15:01:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565009b8f0 con 0x7f565009a550 2024-09-16T15:01:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient(hunting): authenticate will time out at 2129.756230s 2024-09-16T15:01:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564effd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f565009a550 0x7f565009a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:56750/0 (socket says 172.21.15.26:56750) 2024-09-16T15:01:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564f7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f565009b040 0x7f565013a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39978/0 (socket says 172.21.15.26:39978) 2024-09-16T15:01:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564effd640 1 -- 172.21.15.26:0/4215134754 learned_addr learned my addr 172.21.15.26:0/4215134754 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 1 -- 172.21.15.26:0/4215134754 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677861061 0 0) 0x7f565009b8f0 con 0x7f565009a550 2024-09-16T15:01:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient(hunting): my global_id is 354161 2024-09-16T15:01:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 1 -- 172.21.15.26:0/4215134754 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5638005740 con 0x7f565009a550 2024-09-16T15:01:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 1 -- 172.21.15.26:0/4215134754 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495759588 0 0) 0x7f565009dc60 con 0x7f565009b040 2024-09-16T15:01:45.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient(hunting): my global_id is 334359 2024-09-16T15:01:45.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 1 -- 172.21.15.26:0/4215134754 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f565009b8f0 con 0x7f565009b040 2024-09-16T15:01:45.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 1 -- 172.21.15.26:0/4215134754 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1223104215 0 0) 0x7f5638005740 con 0x7f565009a550 2024-09-16T15:01:45.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 1 -- 172.21.15.26:0/4215134754 >> v1:172.21.15.189:6789/0 conn(0x7f565009d1b0 legacy=0x7f5650136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 1 -- 172.21.15.26:0/4215134754 >> v1:172.21.15.26:6789/0 conn(0x7f565009b040 legacy=0x7f565013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient: _finish_hunting 0 2024-09-16T15:01:45.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 1 monclient: found mon.noname-b 2024-09-16T15:01:45.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:45.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:01:45.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 1 -- 172.21.15.26:0/4215134754 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565009bbd0 con 0x7f565009a550 2024-09-16T15:01:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient: _finish_auth 0 2024-09-16T15:01:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 1 -- 172.21.15.26:0/4215134754 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5640001620 con 0x7f565009a550 2024-09-16T15:01:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 5 monclient: authenticate success, global_id 354161 2024-09-16T15:01:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:01:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient: dump: 2024-09-16T15:01:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.641 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.642 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.642 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.642 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.642 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:45.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 1 -- 172.21.15.26:0/4215134754 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5640001840 con 0x7f565009a550 2024-09-16T15:01:45.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:45.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 1 -- 172.21.15.26:0/4215134754 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5640001bd0 con 0x7f565009a550 2024-09-16T15:01:45.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:45.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564dffb640 10 monclient: dump: 2024-09-16T15:01:45.642 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.642 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.642 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.643 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 4 set_mon_vals no callback set 2024-09-16T15:01:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient: shutdown 2024-09-16T15:01:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 1 -- 172.21.15.26:0/4215134754 >> v1:172.21.15.26:6790/0 conn(0x7f565009a550 legacy=0x7f565009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 1 -- 172.21.15.26:0/4215134754 shutdown_connections 2024-09-16T15:01:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 1 -- 172.21.15.26:0/4215134754 >> 172.21.15.26:0/4215134754 conn(0x7f565009fb50 msgr2=0x7f565009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:45.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 1 -- 172.21.15.26:0/4215134754 shutdown_connections 2024-09-16T15:01:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 1 -- 172.21.15.26:0/4215134754 wait complete. 2024-09-16T15:01:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient: build_initial_monmap 2024-09-16T15:01:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient: monmap: 2024-09-16T15:01:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:45.644 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 1 Processor -- start 2024-09-16T15:01:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 1 -- start start 2024-09-16T15:01:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient: init 2024-09-16T15:01:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:01:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient(hunting): picked mon.noname-b con 0x7f565009a550 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient(hunting): picked mon.noname-c con 0x7f565009b040 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient(hunting): picked mon.noname-a con 0x7f565009d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56501a3240 con 0x7f565009d1b0 2024-09-16T15:01:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56501a3770 con 0x7f565009a550 2024-09-16T15:01:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56500a2530 con 0x7f565009b040 2024-09-16T15:01:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f5655271640 10 monclient(hunting): authenticate will time out at 2129.759520s 2024-09-16T15:01:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564f7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f565009d1b0 0x7f56501cd590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39994/0 (socket says 172.21.15.26:39994) 2024-09-16T15:01:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564f7fe640 1 -- 172.21.15.26:0/3965993312 learned_addr learned my addr 172.21.15.26:0/3965993312 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f564e7fc640 1 --1- 172.21.15.26:0/3965993312 >> v1:172.21.15.26:6790/0 conn(0x7f565009b040 0x7f565019ffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:56766/0 (socket says 172.21.15.26:56766) 2024-09-16T15:01:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595951397 0 0) 0x7f56500a2530 con 0x7f565009b040 2024-09-16T15:01:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f56377fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f56377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f56377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f56377fe640 10 monclient(hunting): my global_id is 354164 2024-09-16T15:01:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f563c003530 con 0x7f565009b040 2024-09-16T15:01:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540830591 0 0) 0x7f56501a3240 con 0x7f565009d1b0 2024-09-16T15:01:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f56377fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f56377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f56377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f56377fe640 10 monclient(hunting): my global_id is 334362 2024-09-16T15:01:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.636+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56500a2530 con 0x7f565009d1b0 2024-09-16T15:01:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1364114818 0 0) 0x7f563c003530 con 0x7f565009b040 2024-09-16T15:01:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56501a3240 con 0x7f565009b040 2024-09-16T15:01:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f56440018c0 con 0x7f565009b040 2024-09-16T15:01:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:01:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient(hunting): dump: 2024-09-16T15:01:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.653 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1468953311 0 0) 0x7f56500a2530 con 0x7f565009d1b0 2024-09-16T15:01:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f563c003530 con 0x7f565009d1b0 2024-09-16T15:01:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894710479 0 0) 0x7f56501a3770 con 0x7f565009a550 2024-09-16T15:01:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient(hunting): my global_id is 344272 2024-09-16T15:01:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56500a2530 con 0x7f565009a550 2024-09-16T15:01:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5648001cb0 con 0x7f565009d1b0 2024-09-16T15:01:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient(hunting): dump: 2024-09-16T15:01:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.654 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1421517204 0 0) 0x7f56501a3240 con 0x7f565009b040 2024-09-16T15:01:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 >> v1:172.21.15.189:6789/0 conn(0x7f565009a550 legacy=0x7f565019f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 >> v1:172.21.15.26:6789/0 conn(0x7f565009d1b0 legacy=0x7f56501cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient: _finish_hunting 0 2024-09-16T15:01:45.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 monclient: found mon.c 2024-09-16T15:01:45.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:45.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:45.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565009a240 con 0x7f565009b040 2024-09-16T15:01:45.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient: _finish_auth 0 2024-09-16T15:01:45.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient: _check_auth_tickets 2024-09-16T15:01:45.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:45.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f5655271640 5 monclient: authenticate success, global_id 354164 2024-09-16T15:01:45.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f5655271640 10 monclient: _renew_subs 2024-09-16T15:01:45.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f5655271640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:45.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f5655271640 1 -- 172.21.15.26:0/3965993312 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56501a2ed0 con 0x7f565009b040 2024-09-16T15:01:45.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f5655271640 10 monclient: _renew_subs 2024-09-16T15:01:45.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f5655271640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:45.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f5655271640 1 -- 172.21.15.26:0/3965993312 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56501b4ff0 con 0x7f565009b040 2024-09-16T15:01:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5644003620 con 0x7f565009b040 2024-09-16T15:01:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f56440039b0 con 0x7f565009b040 2024-09-16T15:01:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient: dump: 2024-09-16T15:01:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.657 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f564d7fa640 4 set_mon_vals no callback set 2024-09-16T15:01:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f5655271640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f5655271640 1 -- 172.21.15.26:0/3965993312 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5630006590 con 0x7f5630002830 2024-09-16T15:01:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5644003b10 con 0x7f565009b040 2024-09-16T15:01:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f5644043db0 con 0x7f565009b040 2024-09-16T15:01:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097290949 0 0) 0x7f5630006590 con 0x7f5630002830 2024-09-16T15:01:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient(hunting): my global_id is 334365 2024-09-16T15:01:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56501a3240 con 0x7f5630002830 2024-09-16T15:01:45.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f564f7fe640 10 monclient: get_auth_request con 0x7f563c0363b0 auth_method 0 2024-09-16T15:01:45.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3402635083 0 0) 0x7f56501a3240 con 0x7f5630002830 2024-09-16T15:01:45.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5630006590 con 0x7f5630002830 2024-09-16T15:01:45.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5640001130 con 0x7f5630002830 2024-09-16T15:01:45.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:45.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient: dump: 2024-09-16T15:01:45.658 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.658 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.658 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.658 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.658 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.658 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.658 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.659 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.659 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.659 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2096271296 0 0) 0x7f5630006590 con 0x7f5630002830 2024-09-16T15:01:45.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f563000a0e0 con 0x7f5630002830 2024-09-16T15:01:45.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5640001390 con 0x7f5630002830 2024-09-16T15:01:45.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:45.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:45.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 >> v1:172.21.15.26:6789/0 conn(0x7f5630002830 legacy=0x7f5630004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.640+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 >> v1:172.21.15.26:6789/0 conn(0x7f5630002830 legacy=0x7f5630004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:45.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f5655271640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:01:45.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f5655271640 1 -- 172.21.15.26:0/3965993312 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5630009da0 con 0x7f56300090c0 2024-09-16T15:01:45.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607884689 0 0) 0x7f5630009da0 con 0x7f56300090c0 2024-09-16T15:01:45.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 10 monclient(hunting): my global_id is 334368 2024-09-16T15:01:45.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5630006590 con 0x7f56300090c0 2024-09-16T15:01:45.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3966721410 0 0) 0x7f5630006590 con 0x7f56300090c0 2024-09-16T15:01:45.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5630009da0 con 0x7f56300090c0 2024-09-16T15:01:45.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5640004990 con 0x7f56300090c0 2024-09-16T15:01:45.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:45.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 10 monclient: dump: 2024-09-16T15:01:45.660 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.661 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.661 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.661 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.661 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.661 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.661 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.661 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.661 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.661 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 276307338 0 0) 0x7f5630009da0 con 0x7f56300090c0 2024-09-16T15:01:45.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f563000c290 con 0x7f56300090c0 2024-09-16T15:01:45.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 3673318249) 0x7f5640005fe0 con 0x7f56300090c0 2024-09-16T15:01:45.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:01:45.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:45.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 >> v1:172.21.15.26:6789/0 conn(0x7f56300090c0 legacy=0x7f5630009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f56377fe640 1 -- 172.21.15.26:0/3965993312 >> v1:172.21.15.26:6789/0 conn(0x7f56300090c0 legacy=0x7f5630009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:45.662 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:01:45.662 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:45.662 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:45.662 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:01:45.662 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 174, 2024-09-16T15:01:45.663 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:01:45.663 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:01:45.663 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:01:45.663 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:01:45.663 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:45.663 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T15:01:45.663 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:45.663 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:01:45.663 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:01:45.663 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:45.663 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:45.663 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:45.663 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:45.664 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:45.664 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:45.664 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:45.664 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:45.664 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:45.664 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:45.664 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:45.664 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:01:45.664 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:01:45.664 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:45.664 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:45.664 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:45.664 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:45.664 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:45.665 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:45.665 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:45.665 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:45.665 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:45.665 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:45.665 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.665 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:01:45.665 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:01:45.665 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:01:45.665 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:01:45.665 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:01:45.665 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:01:45.666 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:45.666 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:45.666 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:01:45.666 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:01:45.666 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:01:45.666 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:01:45.666 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:01:45.666 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:01:45.666 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:01:45.666 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:45.666 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:01:45.666 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:45.666 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:45.666 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:45.666 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:45.667 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:45.667 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:45.667 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:45.667 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:45.667 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:45.667 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:45.667 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:45.667 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:01:45.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.667 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:01:45.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.667 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:45.667 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:45.667 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:45.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:45.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.668 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:45.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:01:45.668 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:45.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.668 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:01:45.668 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:01:45.668 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:45.668 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:45.668 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:45.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.669 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:45.669 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:45.669 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:45.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:45.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:45.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:01:45.669 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:45.669 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.669 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:01:45.670 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:01:45.670 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:45.670 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:45.670 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:45.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.670 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:45.670 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:45.670 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:45.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:45.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.670 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:45.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:01:45.671 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:45.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.671 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:01:45.671 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:01:45.671 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:45.671 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:45.671 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:45.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.671 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.671 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:01:45.671 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:01:45.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.672 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:45.672 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:45.672 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:45.672 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.672 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:45.672 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T15:01:45.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.672 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:45.672 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:45.672 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T15:01:45.672 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.672 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:45.672 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:01:45.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.673 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:45.673 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:45.673 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T15:01:45.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.673 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.673 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:01:45.673 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:01:45.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f5655271640 1 -- 172.21.15.26:0/3965993312 >> v1:172.21.15.189:6812/3196161582 conn(0x7f563c0363b0 legacy=0x7f563c038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f5655271640 10 monclient: shutdown 2024-09-16T15:01:45.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f5655271640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:45.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f5655271640 1 -- 172.21.15.26:0/3965993312 >> v1:172.21.15.26:6790/0 conn(0x7f565009b040 legacy=0x7f565019ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f564f7fe640 1 -- 172.21.15.26:0/3965993312 reap_dead start 2024-09-16T15:01:45.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f564f7fe640 1 -- 172.21.15.26:0/3965993312 reap_dead start 2024-09-16T15:01:45.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f5655271640 1 -- 172.21.15.26:0/3965993312 shutdown_connections 2024-09-16T15:01:45.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f5655271640 1 -- 172.21.15.26:0/3965993312 >> 172.21.15.26:0/3965993312 conn(0x7f565009fb50 msgr2=0x7f56501b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:45.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f5655271640 1 -- 172.21.15.26:0/3965993312 shutdown_connections 2024-09-16T15:01:45.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.648+0000 7f5655271640 1 -- 172.21.15.26:0/3965993312 wait complete. 2024-09-16T15:01:45.675 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:01:45.675 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T15:01:45.676 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T15:01:45.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.749+0000 7f3205f12640 10 start_mon_command cmd=[{"prefix": "health", "detail": "detail"}] 2024-09-16T15:01:45.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.749+0000 7f3205f12640 10 monclient: _send_command 2 [{"prefix": "health", "detail": "detail"}] 2024-09-16T15:01:45.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.749+0000 7f3205f12640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:45.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.749+0000 7f3205f12640 1 -- 172.21.15.189:0/3586545967 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f31e0005300 con 0x7f320009e490 2024-09-16T15:01:45.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.749+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1041 (unknown 501213350 0 2456344155) 0x7f31f4014410 con 0x7f320009e490 2024-09-16T15:01:45.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.749+0000 7f31e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "detail": "detail"}] 2024-09-16T15:01:45.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.749+0000 7f31e7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.788+0000 7f5bb7aed640 10 monclient: get_monmap_and_config 2024-09-16T15:01:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.788+0000 7f5bb7aed640 10 monclient: build_initial_monmap 2024-09-16T15:01:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.788+0000 7f5bb7aed640 10 monclient: monmap: 2024-09-16T15:01:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:45.792803+0000 2024-09-16T15:01:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:45.792803+0000 2024-09-16T15:01:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:45.791 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 1 Processor -- start 2024-09-16T15:01:45.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 1 -- start start 2024-09-16T15:01:45.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 10 monclient: init 2024-09-16T15:01:45.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:45.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:01:45.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 10 monclient(hunting): picked mon.noname-a con 0x7f5bb009a550 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:45.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 10 monclient(hunting): picked mon.noname-b con 0x7f5bb009d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:45.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 10 monclient(hunting): picked mon.noname-c con 0x7f5bb009b040 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:45.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb009dc60 con 0x7f5bb009a550 2024-09-16T15:01:45.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb0099ae0 con 0x7f5bb009b040 2024-09-16T15:01:45.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb009b8f0 con 0x7f5bb009d1b0 2024-09-16T15:01:45.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:45.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 10 monclient(hunting): authenticate will time out at 2129.913329s 2024-09-16T15:01:45.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb5088640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5bb009d1b0 0x7f5bb0136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:56776/0 (socket says 172.21.15.26:56776) 2024-09-16T15:01:45.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb5088640 1 -- 172.21.15.26:0/3024798498 learned_addr learned my addr 172.21.15.26:0/3024798498 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:45.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb5889640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5bb009a550 0x7f5bb009a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40022/0 (socket says 172.21.15.26:40022) 2024-09-16T15:01:45.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 1 -- 172.21.15.26:0/3024798498 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347225695 0 0) 0x7f5bb009dc60 con 0x7f5bb009a550 2024-09-16T15:01:45.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient(hunting): my global_id is 334371 2024-09-16T15:01:45.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 1 -- 172.21.15.26:0/3024798498 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b9c005740 con 0x7f5bb009a550 2024-09-16T15:01:45.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 1 -- 172.21.15.26:0/3024798498 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242253853 0 0) 0x7f5bb009b8f0 con 0x7f5bb009d1b0 2024-09-16T15:01:45.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient(hunting): my global_id is 354167 2024-09-16T15:01:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 1 -- 172.21.15.26:0/3024798498 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb009dc60 con 0x7f5bb009d1b0 2024-09-16T15:01:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 1 -- 172.21.15.26:0/3024798498 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491226849 0 0) 0x7f5bb0099ae0 con 0x7f5bb009b040 2024-09-16T15:01:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient(hunting): my global_id is 344275 2024-09-16T15:01:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 1 -- 172.21.15.26:0/3024798498 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb009b8f0 con 0x7f5bb009b040 2024-09-16T15:01:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 1 -- 172.21.15.26:0/3024798498 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1206853241 0 0) 0x7f5bb009dc60 con 0x7f5bb009d1b0 2024-09-16T15:01:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 1 -- 172.21.15.26:0/3024798498 >> v1:172.21.15.189:6789/0 conn(0x7f5bb009b040 legacy=0x7f5bb013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 1 -- 172.21.15.26:0/3024798498 >> v1:172.21.15.26:6789/0 conn(0x7f5bb009a550 legacy=0x7f5bb009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient: _finish_hunting 0 2024-09-16T15:01:45.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 1 monclient: found mon.noname-b 2024-09-16T15:01:45.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:45.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:01:45.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 1 -- 172.21.15.26:0/3024798498 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb009bbd0 con 0x7f5bb009d1b0 2024-09-16T15:01:45.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient: _finish_auth 0 2024-09-16T15:01:45.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient: _check_auth_tickets 2024-09-16T15:01:45.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:45.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 1 -- 172.21.15.26:0/3024798498 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5ba0001580 con 0x7f5bb009d1b0 2024-09-16T15:01:45.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:01:45.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient: dump: 2024-09-16T15:01:45.798 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.798 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.798 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.799 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.799 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.799 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.799 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.799 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.799 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.799 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 5 monclient: authenticate success, global_id 354167 2024-09-16T15:01:45.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:45.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 1 -- 172.21.15.26:0/3024798498 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ba0001820 con 0x7f5bb009d1b0 2024-09-16T15:01:45.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:45.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:45.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 1 -- 172.21.15.26:0/3024798498 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5ba0001bf0 con 0x7f5bb009d1b0 2024-09-16T15:01:45.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 4 set_mon_vals no callback set 2024-09-16T15:01:45.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:45.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb4887640 10 monclient: dump: 2024-09-16T15:01:45.800 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.800 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.800 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.800 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.800 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.800 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.800 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.800 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.801 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.801 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 10 monclient: shutdown 2024-09-16T15:01:45.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:45.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 1 -- 172.21.15.26:0/3024798498 >> v1:172.21.15.26:6790/0 conn(0x7f5bb009d1b0 legacy=0x7f5bb0136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 1 -- 172.21.15.26:0/3024798498 shutdown_connections 2024-09-16T15:01:45.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 1 -- 172.21.15.26:0/3024798498 >> 172.21.15.26:0/3024798498 conn(0x7f5bb009fb50 msgr2=0x7f5bb009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:45.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 1 -- 172.21.15.26:0/3024798498 shutdown_connections 2024-09-16T15:01:45.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 1 -- 172.21.15.26:0/3024798498 wait complete. 2024-09-16T15:01:45.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 10 monclient: build_initial_monmap 2024-09-16T15:01:45.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.792+0000 7f5bb7aed640 10 monclient: monmap: 2024-09-16T15:01:45.801 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:45.802 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:45.802 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:45.802 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:45.802 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:45.802 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:45.802 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:45.802 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:45.802 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:45.802 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 1 Processor -- start 2024-09-16T15:01:45.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 1 -- start start 2024-09-16T15:01:45.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 10 monclient: init 2024-09-16T15:01:45.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:45.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:01:45.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 10 monclient(hunting): picked mon.noname-c con 0x7f5bb009a550 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:45.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 10 monclient(hunting): picked mon.noname-a con 0x7f5bb009b040 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:45.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 10 monclient(hunting): picked mon.noname-b con 0x7f5bb009d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:45.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb00a2330 con 0x7f5bb009b040 2024-09-16T15:01:45.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb01a1a30 con 0x7f5bb009d1b0 2024-09-16T15:01:45.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb01a17b0 con 0x7f5bb009a550 2024-09-16T15:01:45.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:45.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 10 monclient(hunting): authenticate will time out at 2129.916626s 2024-09-16T15:01:45.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb5088640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5bb009b040 0x7f5bb01c9f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40030/0 (socket says 172.21.15.26:40030) 2024-09-16T15:01:45.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb5889640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5bb009a550 0x7f5bb01b47f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:56784/0 (socket says 172.21.15.26:56784) 2024-09-16T15:01:45.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb5088640 1 -- 172.21.15.26:0/2983985666 learned_addr learned my addr 172.21.15.26:0/2983985666 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:45.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403398176 0 0) 0x7f5bb01a17b0 con 0x7f5bb009a550 2024-09-16T15:01:45.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): my global_id is 354170 2024-09-16T15:01:45.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b94003530 con 0x7f5bb009a550 2024-09-16T15:01:45.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818646909 0 0) 0x7f5bb00a2330 con 0x7f5bb009b040 2024-09-16T15:01:45.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): my global_id is 334374 2024-09-16T15:01:45.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb01a17b0 con 0x7f5bb009b040 2024-09-16T15:01:45.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795210315 0 0) 0x7f5bb01a1a30 con 0x7f5bb009d1b0 2024-09-16T15:01:45.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): my global_id is 344278 2024-09-16T15:01:45.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb00a2330 con 0x7f5bb009d1b0 2024-09-16T15:01:45.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2196838568 0 0) 0x7f5b94003530 con 0x7f5bb009a550 2024-09-16T15:01:45.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bb01a1a30 con 0x7f5bb009a550 2024-09-16T15:01:45.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5bac0011a0 con 0x7f5bb009a550 2024-09-16T15:01:45.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:01:45.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): dump: 2024-09-16T15:01:45.811 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.811 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.811 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.811 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.811 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.812 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.812 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.812 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.812 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.812 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:45.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1224233801 0 0) 0x7f5bb01a17b0 con 0x7f5bb009b040 2024-09-16T15:01:45.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b94003530 con 0x7f5bb009b040 2024-09-16T15:01:45.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5ba0001b30 con 0x7f5bb009b040 2024-09-16T15:01:45.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:45.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): dump: 2024-09-16T15:01:45.812 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.813 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.813 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.813 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.813 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.813 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.813 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.813 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.813 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.813 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1238173152 0 0) 0x7f5bb01a1a30 con 0x7f5bb009a550 2024-09-16T15:01:45.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 >> v1:172.21.15.189:6789/0 conn(0x7f5bb009d1b0 legacy=0x7f5bb01cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 >> v1:172.21.15.26:6789/0 conn(0x7f5bb009b040 legacy=0x7f5bb01c9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient: _finish_hunting 0 2024-09-16T15:01:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 monclient: found mon.c 2024-09-16T15:01:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb01a2ee0 con 0x7f5bb009a550 2024-09-16T15:01:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient: _finish_auth 0 2024-09-16T15:01:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient: _check_auth_tickets 2024-09-16T15:01:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 5 monclient: authenticate success, global_id 354170 2024-09-16T15:01:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 10 monclient: _renew_subs 2024-09-16T15:01:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 1 -- 172.21.15.26:0/2983985666 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb01a0e60 con 0x7f5bb009a550 2024-09-16T15:01:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 10 monclient: _renew_subs 2024-09-16T15:01:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 1 -- 172.21.15.26:0/2983985666 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb01b4f00 con 0x7f5bb009a550 2024-09-16T15:01:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bac0047f0 con 0x7f5bb009a550 2024-09-16T15:01:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5bac004b00 con 0x7f5bb009a550 2024-09-16T15:01:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba7fff640 4 set_mon_vals no callback set 2024-09-16T15:01:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient: dump: 2024-09-16T15:01:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.816 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb7aed640 1 -- 172.21.15.26:0/2983985666 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5b98006590 con 0x7f5b98002830 2024-09-16T15:01:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5bac00f570 con 0x7f5bb009a550 2024-09-16T15:01:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f5bac043d90 con 0x7f5bb009a550 2024-09-16T15:01:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886003160 0 0) 0x7f5b98006590 con 0x7f5b98002830 2024-09-16T15:01:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient(hunting): my global_id is 354173 2024-09-16T15:01:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb01a1a30 con 0x7f5b98002830 2024-09-16T15:01:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5bb608a640 10 monclient: get_auth_request con 0x7f5b94036530 auth_method 0 2024-09-16T15:01:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3965631689 0 0) 0x7f5bb01a1a30 con 0x7f5b98002830 2024-09-16T15:01:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b98006590 con 0x7f5b98002830 2024-09-16T15:01:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5ba0004b80 con 0x7f5b98002830 2024-09-16T15:01:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient: dump: 2024-09-16T15:01:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.818 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2675961917 0 0) 0x7f5b98006590 con 0x7f5b98002830 2024-09-16T15:01:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b9800a0e0 con 0x7f5b98002830 2024-09-16T15:01:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5ba0004e20 con 0x7f5b98002830 2024-09-16T15:01:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 >> v1:172.21.15.26:6790/0 conn(0x7f5b98002830 legacy=0x7f5b98004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.796+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 >> v1:172.21.15.26:6790/0 conn(0x7f5b98002830 legacy=0x7f5b98004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5bb7aed640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:01:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5bb7aed640 1 -- 172.21.15.26:0/2983985666 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5b98009da0 con 0x7f5b980090c0 2024-09-16T15:01:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381778011 0 0) 0x7f5b98009da0 con 0x7f5b980090c0 2024-09-16T15:01:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 10 monclient(hunting): my global_id is 354176 2024-09-16T15:01:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b98006590 con 0x7f5b980090c0 2024-09-16T15:01:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1993131844 0 0) 0x7f5b98006590 con 0x7f5b980090c0 2024-09-16T15:01:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b98009da0 con 0x7f5b980090c0 2024-09-16T15:01:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5ba0005430 con 0x7f5b980090c0 2024-09-16T15:01:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 10 monclient: dump: 2024-09-16T15:01:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.821 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2250975942 0 0) 0x7f5b98009da0 con 0x7f5b980090c0 2024-09-16T15:01:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5b9800c290 con 0x7f5b980090c0 2024-09-16T15:01:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 1931389737) 0x7f5ba0006910 con 0x7f5b980090c0 2024-09-16T15:01:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:01:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 >> v1:172.21.15.26:6790/0 conn(0x7f5b980090c0 legacy=0x7f5b98009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5ba67fc640 1 -- 172.21.15.26:0/2983985666 >> v1:172.21.15.26:6790/0 conn(0x7f5b980090c0 legacy=0x7f5b98009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:45.822 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:01:45.822 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:45.822 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:45.822 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T15:01:45.822 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 174, 2024-09-16T15:01:45.822 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:01:45.822 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:01:45.822 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:01:45.822 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:01:45.823 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:45.823 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-09-16T15:01:45.823 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:45.823 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:01:45.823 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:01:45.823 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:45.823 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:45.823 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:45.824 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:45.824 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:45.824 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:45.824 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:45.824 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:45.824 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:45.824 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:45.824 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:45.824 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:01:45.824 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:01:45.824 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:45.824 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:45.824 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:45.825 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:45.825 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:45.825 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:45.825 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:45.825 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:45.825 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:45.825 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:45.825 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.825 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:01:45.825 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:01:45.826 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:01:45.826 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:01:45.826 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:01:45.826 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:01:45.826 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:45.826 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:45.826 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:01:45.826 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:01:45.826 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:01:45.826 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:01:45.826 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:01:45.826 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:01:45.827 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:01:45.827 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:45.827 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:01:45.827 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:45.827 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:45.827 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:45.827 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:45.827 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:45.827 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:45.827 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:45.827 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:45.827 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:45.827 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:45.828 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:45.828 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:01:45.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.828 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:01:45.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.828 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:45.828 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:45.828 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:45.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:45.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.828 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:45.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:01:45.829 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:45.829 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.829 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:01:45.829 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:01:45.829 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:45.829 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:45.829 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:45.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.830 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:45.830 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:45.830 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:45.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:45.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.830 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:45.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:01:45.830 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:45.830 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.830 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:01:45.830 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:01:45.831 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:45.831 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:45.831 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:45.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.831 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:45.831 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:45.831 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:45.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:45.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.831 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:45.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:01:45.832 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:45.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:01:45.832 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:01:45.832 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:45.832 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:45.832 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:45.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.833 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:01:45.833 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:01:45.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.833 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:45.833 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:45.833 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:45.833 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.833 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:45.833 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:01:45.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.833 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:45.833 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:45.833 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T15:01:45.834 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.834 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.834 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:01:45.834 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:01:45.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5bb7aed640 1 -- 172.21.15.26:0/2983985666 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5b94036530 legacy=0x7f5b940389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5bb7aed640 10 monclient: shutdown 2024-09-16T15:01:45.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5bb7aed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:45.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5bb7aed640 1 -- 172.21.15.26:0/2983985666 >> v1:172.21.15.26:6790/0 conn(0x7f5bb009a550 legacy=0x7f5bb01b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5bb608a640 1 -- 172.21.15.26:0/2983985666 reap_dead start 2024-09-16T15:01:45.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5bb608a640 1 -- 172.21.15.26:0/2983985666 reap_dead start 2024-09-16T15:01:45.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5bb7aed640 1 -- 172.21.15.26:0/2983985666 shutdown_connections 2024-09-16T15:01:45.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5bb7aed640 1 -- 172.21.15.26:0/2983985666 >> 172.21.15.26:0/2983985666 conn(0x7f5bb009fb50 msgr2=0x7f5bb01b7370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:45.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5bb7aed640 1 -- 172.21.15.26:0/2983985666 shutdown_connections 2024-09-16T15:01:45.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.804+0000 7f5bb7aed640 1 -- 172.21.15.26:0/2983985666 wait complete. 2024-09-16T15:01:45.836 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:01:45.836 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T15:01:45.837 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T15:01:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.901+0000 7f3205f12640 10 start_mon_command cmd=[{"prefix": "health", "detail": "detail"}] 2024-09-16T15:01:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.901+0000 7f3205f12640 10 monclient: _send_command 3 [{"prefix": "health", "detail": "detail"}] 2024-09-16T15:01:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.901+0000 7f3205f12640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T15:01:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.901+0000 7f3205f12640 1 -- 172.21.15.189:0/3586545967 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f31e0005850 con 0x7f320009e490 2024-09-16T15:01:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.901+0000 7f31e7fff640 1 -- 172.21.15.189:0/3586545967 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1041 (unknown 501213350 0 2456344155) 0x7f31f4014410 con 0x7f320009e490 2024-09-16T15:01:45.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.901+0000 7f31e7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "health", "detail": "detail"}] 2024-09-16T15:01:45.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.901+0000 7f31e7fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T15:01:45.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.901+0000 7f3205f12640 1 -- 172.21.15.189:0/3586545967 >> v1:172.21.15.189:6812/3196161582 conn(0x7f31ec036390 legacy=0x7f31ec038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.901+0000 7f3205f12640 10 monclient: shutdown 2024-09-16T15:01:45.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.901+0000 7f3205f12640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:45.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.901+0000 7f3205f12640 1 -- 172.21.15.189:0/3586545967 >> v1:172.21.15.189:6789/0 conn(0x7f320009e490 legacy=0x7f32001c9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.901+0000 7f3205f12640 1 -- 172.21.15.189:0/3586545967 shutdown_connections 2024-09-16T15:01:45.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.901+0000 7f3205f12640 1 -- 172.21.15.189:0/3586545967 >> 172.21.15.189:0/3586545967 conn(0x7f320009faa0 msgr2=0x7f320006ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:45.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.901+0000 7f3205f12640 1 -- 172.21.15.189:0/3586545967 shutdown_connections 2024-09-16T15:01:45.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:45.901+0000 7f3205f12640 1 -- 172.21.15.189:0/3586545967 wait complete. 2024-09-16T15:01:45.915 INFO:tasks.workunit.client.0.smithi189.stdout: client.bad mds capability parse failed, stopped at 'also does not parse' of 'also does not parse' 2024-09-16T15:01:45.915 INFO:tasks.workunit.client.0.smithi189.stdout: client.bad mon capability parse failed, stopped at 'this is wrong' of 'this is wrong' 2024-09-16T15:01:45.915 INFO:tasks.workunit.client.0.smithi189.stdout: client.bad osd capability parse failed, stopped at 'does not parse' of 'does not parse' 2024-09-16T15:01:45.915 INFO:tasks.workunit.client.0.smithi189.stdout: client.bad mds capability parse failed, stopped at 'also does not parse' of 'also does not parse' 2024-09-16T15:01:45.915 INFO:tasks.workunit.client.0.smithi189.stdout: client.bad mon capability parse failed, stopped at 'this is wrong' of 'this is wrong' 2024-09-16T15:01:45.915 INFO:tasks.workunit.client.0.smithi189.stdout: client.bad osd capability parse failed, stopped at 'does not parse' of 'does not parse' 2024-09-16T15:01:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph auth rm client.bad 2024-09-16T15:01:45.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 10 monclient: get_monmap_and_config 2024-09-16T15:01:45.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 10 monclient: build_initial_monmap 2024-09-16T15:01:45.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 10 monclient: monmap: 2024-09-16T15:01:45.952 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:45.952 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.952 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:45.953873+0000 2024-09-16T15:01:45.952 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:45.953873+0000 2024-09-16T15:01:45.952 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:45.952 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.952 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:45.952 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:45.953 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:45.953 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 1 Processor -- start 2024-09-16T15:01:45.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 1 -- start start 2024-09-16T15:01:45.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 10 monclient: init 2024-09-16T15:01:45.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:45.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:01:45.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 10 monclient(hunting): picked mon.noname-c con 0x7f23b809cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:45.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 10 monclient(hunting): picked mon.noname-a con 0x7f23b809e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:45.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 10 monclient(hunting): picked mon.noname-b con 0x7f23b809d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:45.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b809d5b0 con 0x7f23b809e7a0 2024-09-16T15:01:45.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b8131a90 con 0x7f23b809cdc0 2024-09-16T15:01:45.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b809aff0 con 0x7f23b809d1a0 2024-09-16T15:01:45.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:45.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 10 monclient(hunting): authenticate will time out at 2130.073939s 2024-09-16T15:01:45.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b67fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f23b809e7a0 0x7f23b8131380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40034/0 (socket says 172.21.15.26:40034) 2024-09-16T15:01:45.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b67fc640 1 -- 172.21.15.26:0/1984072247 learned_addr learned my addr 172.21.15.26:0/1984072247 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:45.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 1 -- 172.21.15.26:0/1984072247 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660230036 0 0) 0x7f23b809aff0 con 0x7f23b809d1a0 2024-09-16T15:01:45.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient(hunting): my global_id is 354179 2024-09-16T15:01:45.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 1 -- 172.21.15.26:0/1984072247 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a0003530 con 0x7f23b809d1a0 2024-09-16T15:01:45.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 1 -- 172.21.15.26:0/1984072247 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719981803 0 0) 0x7f23b809d5b0 con 0x7f23b809e7a0 2024-09-16T15:01:45.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient(hunting): my global_id is 334377 2024-09-16T15:01:45.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 1 -- 172.21.15.26:0/1984072247 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b809aff0 con 0x7f23b809e7a0 2024-09-16T15:01:45.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 1 -- 172.21.15.26:0/1984072247 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205613099 0 0) 0x7f23b8131a90 con 0x7f23b809cdc0 2024-09-16T15:01:45.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient(hunting): my global_id is 344281 2024-09-16T15:01:45.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 1 -- 172.21.15.26:0/1984072247 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b809d5b0 con 0x7f23b809cdc0 2024-09-16T15:01:45.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 1 -- 172.21.15.26:0/1984072247 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4080579349 0 0) 0x7f23a0003530 con 0x7f23b809d1a0 2024-09-16T15:01:45.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 1 -- 172.21.15.26:0/1984072247 >> v1:172.21.15.189:6789/0 conn(0x7f23b809cdc0 legacy=0x7f23b8130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 1 -- 172.21.15.26:0/1984072247 >> v1:172.21.15.26:6789/0 conn(0x7f23b809e7a0 legacy=0x7f23b8131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient: _finish_hunting 0 2024-09-16T15:01:45.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 1 monclient: found mon.noname-b 2024-09-16T15:01:45.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:45.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:01:45.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 1 -- 172.21.15.26:0/1984072247 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b809b2d0 con 0x7f23b809d1a0 2024-09-16T15:01:45.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient: _finish_auth 0 2024-09-16T15:01:45.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:45.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:45.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 1 -- 172.21.15.26:0/1984072247 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f23b00019a0 con 0x7f23b809d1a0 2024-09-16T15:01:45.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 5 monclient: authenticate success, global_id 354179 2024-09-16T15:01:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:01:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient: dump: 2024-09-16T15:01:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.964 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 1 -- 172.21.15.26:0/1984072247 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23b0001c10 con 0x7f23b809d1a0 2024-09-16T15:01:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 1 -- 172.21.15.26:0/1984072247 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f23b0001fe0 con 0x7f23b809d1a0 2024-09-16T15:01:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 4 set_mon_vals no callback set 2024-09-16T15:01:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23b5ffb640 10 monclient: dump: 2024-09-16T15:01:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.965 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.965 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.965 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.965 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.965 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.965 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.965 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 10 monclient: shutdown 2024-09-16T15:01:45.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:45.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 1 -- 172.21.15.26:0/1984072247 >> v1:172.21.15.26:6790/0 conn(0x7f23b809d1a0 legacy=0x7f23b8139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 1 -- 172.21.15.26:0/1984072247 shutdown_connections 2024-09-16T15:01:45.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 1 -- 172.21.15.26:0/1984072247 >> 172.21.15.26:0/1984072247 conn(0x7f23b809fb50 msgr2=0x7f23b809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:45.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 1 -- 172.21.15.26:0/1984072247 shutdown_connections 2024-09-16T15:01:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.952+0000 7f23bd409640 1 -- 172.21.15.26:0/1984072247 wait complete. 2024-09-16T15:01:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 10 monclient: build_initial_monmap 2024-09-16T15:01:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 10 monclient: monmap: 2024-09-16T15:01:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:45.966 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 1 Processor -- start 2024-09-16T15:01:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 1 -- start start 2024-09-16T15:01:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 10 monclient: init 2024-09-16T15:01:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:01:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 10 monclient(hunting): picked mon.noname-c con 0x7f23b809cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 10 monclient(hunting): picked mon.noname-b con 0x7f23b809d1a0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 10 monclient(hunting): picked mon.noname-a con 0x7f23b809e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b81ca110 con 0x7f23b809e7a0 2024-09-16T15:01:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b8120f90 con 0x7f23b809d1a0 2024-09-16T15:01:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b8121270 con 0x7f23b809cdc0 2024-09-16T15:01:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 10 monclient(hunting): authenticate will time out at 2130.077066s 2024-09-16T15:01:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23b77fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f23b809e7a0 0x7f23b81d1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40038/0 (socket says 172.21.15.26:40038) 2024-09-16T15:01:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23b77fe640 1 -- 172.21.15.26:0/350944565 learned_addr learned my addr 172.21.15.26:0/350944565 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279411571 0 0) 0x7f23b81ca110 con 0x7f23b809e7a0 2024-09-16T15:01:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): my global_id is 334380 2024-09-16T15:01:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a4003530 con 0x7f23b809e7a0 2024-09-16T15:01:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005956324 0 0) 0x7f23b8121270 con 0x7f23b809cdc0 2024-09-16T15:01:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): my global_id is 354182 2024-09-16T15:01:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b81ca110 con 0x7f23b809cdc0 2024-09-16T15:01:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 749141370 0 0) 0x7f23a4003530 con 0x7f23b809e7a0 2024-09-16T15:01:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23b8121270 con 0x7f23b809e7a0 2024-09-16T15:01:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f23b0001f20 con 0x7f23b809e7a0 2024-09-16T15:01:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:01:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): dump: 2024-09-16T15:01:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.969 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.970 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2621285752 0 0) 0x7f23b81ca110 con 0x7f23b809cdc0 2024-09-16T15:01:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23a4003530 con 0x7f23b809cdc0 2024-09-16T15:01:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f23a80018c0 con 0x7f23b809cdc0 2024-09-16T15:01:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): dump: 2024-09-16T15:01:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.970 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.971 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359606909 0 0) 0x7f23b8120f90 con 0x7f23b809d1a0 2024-09-16T15:01:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): my global_id is 344284 2024-09-16T15:01:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b81ca110 con 0x7f23b809d1a0 2024-09-16T15:01:45.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 290421847 0 0) 0x7f23b8121270 con 0x7f23b809e7a0 2024-09-16T15:01:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 >> v1:172.21.15.26:6790/0 conn(0x7f23b809cdc0 legacy=0x7f23b812de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 >> v1:172.21.15.189:6789/0 conn(0x7f23b809d1a0 legacy=0x7f23b81ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient: _finish_hunting 0 2024-09-16T15:01:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 monclient: found mon.a 2024-09-16T15:01:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b8121550 con 0x7f23b809e7a0 2024-09-16T15:01:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient: _finish_auth 0 2024-09-16T15:01:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient: _check_auth_tickets 2024-09-16T15:01:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 5 monclient: authenticate success, global_id 334380 2024-09-16T15:01:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 10 monclient: _renew_subs 2024-09-16T15:01:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:45.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 1 -- 172.21.15.26:0/350944565 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b8121830 con 0x7f23b809e7a0 2024-09-16T15:01:45.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 10 monclient: _renew_subs 2024-09-16T15:01:45.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:01:45.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 1 -- 172.21.15.26:0/350944565 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b812e520 con 0x7f23b809e7a0 2024-09-16T15:01:45.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23b0003d70 con 0x7f23b809e7a0 2024-09-16T15:01:45.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:45.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f23b0004100 con 0x7f23b809e7a0 2024-09-16T15:01:45.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:45.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient: dump: 2024-09-16T15:01:45.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23b57fa640 4 set_mon_vals no callback set 2024-09-16T15:01:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f23b0004260 con 0x7f23b809e7a0 2024-09-16T15:01:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23bd409640 1 -- 172.21.15.26:0/350944565 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2398006590 con 0x7f2398002830 2024-09-16T15:01:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f23b0044240 con 0x7f23b809e7a0 2024-09-16T15:01:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f23b67fc640 10 monclient: get_auth_request con 0x7f23a40363d0 auth_method 0 2024-09-16T15:01:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18072015 0 0) 0x7f2398006590 con 0x7f2398002830 2024-09-16T15:01:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 10 monclient(hunting): my global_id is 344287 2024-09-16T15:01:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b8121270 con 0x7f2398002830 2024-09-16T15:01:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.956+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4210716204 0 0) 0x7f23b8121270 con 0x7f2398002830 2024-09-16T15:01:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.960+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2398006590 con 0x7f2398002830 2024-09-16T15:01:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.960+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f23a80037c0 con 0x7f2398002830 2024-09-16T15:01:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.960+0000 7f239f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.960+0000 7f239f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.960+0000 7f239f7fe640 10 monclient: dump: 2024-09-16T15:01:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.975 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.976 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.976 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.976 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.976 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.976 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.976 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.976 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.976 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.960+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1196273013 0 0) 0x7f2398006590 con 0x7f2398002830 2024-09-16T15:01:45.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.960+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f239800a0e0 con 0x7f2398002830 2024-09-16T15:01:45.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.960+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f23a8004650 con 0x7f2398002830 2024-09-16T15:01:45.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.960+0000 7f239f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:45.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.960+0000 7f239f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:45.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.960+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 >> v1:172.21.15.189:6789/0 conn(0x7f2398002830 legacy=0x7f2398004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.960+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 >> v1:172.21.15.189:6789/0 conn(0x7f2398002830 legacy=0x7f2398004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:45.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.964+0000 7f23bd409640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:01:45.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.964+0000 7f23bd409640 1 -- 172.21.15.26:0/350944565 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2398009da0 con 0x7f23980090c0 2024-09-16T15:01:45.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.964+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460306431 0 0) 0x7f2398009da0 con 0x7f23980090c0 2024-09-16T15:01:45.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.964+0000 7f239f7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:45.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.964+0000 7f239f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:45.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.964+0000 7f239f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.964+0000 7f239f7fe640 10 monclient(hunting): my global_id is 344290 2024-09-16T15:01:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.964+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2398006590 con 0x7f23980090c0 2024-09-16T15:01:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.964+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2332134749 0 0) 0x7f2398006590 con 0x7f23980090c0 2024-09-16T15:01:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.964+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2398009da0 con 0x7f23980090c0 2024-09-16T15:01:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.964+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f23a8003e40 con 0x7f23980090c0 2024-09-16T15:01:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.964+0000 7f239f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.964+0000 7f239f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:01:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.964+0000 7f239f7fe640 10 monclient: dump: 2024-09-16T15:01:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:45.979 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:45.980 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:45.980 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:45.980 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:45.980 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:45.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3320770094 0 0) 0x7f2398009da0 con 0x7f23980090c0 2024-09-16T15:01:45.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f239800c290 con 0x7f23980090c0 2024-09-16T15:01:45.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 1881033755) 0x7f23a80040e0 con 0x7f23980090c0 2024-09-16T15:01:45.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f239f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:01:45.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f239f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:45.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 >> v1:172.21.15.189:6789/0 conn(0x7f23980090c0 legacy=0x7f2398009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f239f7fe640 1 -- 172.21.15.26:0/350944565 >> v1:172.21.15.189:6789/0 conn(0x7f23980090c0 legacy=0x7f2398009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:01:45.980 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:01:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:01:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 174, 2024-09-16T15:01:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:01:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:01:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:01:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:01:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-09-16T15:01:45.981 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:01:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:01:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:45.982 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:01:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:01:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.983 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:01:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:01:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:01:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:01:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:01:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:01:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:01:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:01:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:01:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:01:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:01:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:01:45.984 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:01:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:01:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:01:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:01:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:01:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:01:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:01:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:01:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:01:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:01:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:01:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:01:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:01:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:01:45.985 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:01:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:01:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:01:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:01:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:01:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:01:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:01:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:01:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:01:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:45.987 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:01:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:01:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:01:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:01:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:01:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:01:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:01:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:01:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:01:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:01:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:01:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:01:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:01:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:01:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:01:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:01:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:01:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:01:45.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:01:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T15:01:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:01:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:01:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:01:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:01:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:01:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:01:45.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:01:45.992 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:01:45.992 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:01:45.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f23bd409640 1 -- 172.21.15.26:0/350944565 >> v1:172.21.15.189:6812/3196161582 conn(0x7f23a40363d0 legacy=0x7f23a4038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f23bd409640 10 monclient: shutdown 2024-09-16T15:01:45.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f23bd409640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:45.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f23bd409640 1 -- 172.21.15.26:0/350944565 >> v1:172.21.15.26:6789/0 conn(0x7f23b809e7a0 legacy=0x7f23b81d1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:45.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f23b77fe640 1 -- 172.21.15.26:0/350944565 reap_dead start 2024-09-16T15:01:45.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f23b77fe640 1 -- 172.21.15.26:0/350944565 reap_dead start 2024-09-16T15:01:45.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f23bd409640 1 -- 172.21.15.26:0/350944565 shutdown_connections 2024-09-16T15:01:45.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f23bd409640 1 -- 172.21.15.26:0/350944565 >> 172.21.15.26:0/350944565 conn(0x7f23b809fb50 msgr2=0x7f23b809bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:45.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f23bd409640 1 -- 172.21.15.26:0/350944565 shutdown_connections 2024-09-16T15:01:45.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:45.968+0000 7f23bd409640 1 -- 172.21.15.26:0/350944565 wait complete. 2024-09-16T15:01:45.994 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:01:45.994 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T15:01:45.994 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T15:01:45.994 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T15:01:45.994 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T15:01:45.995 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T15:01:45.995 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:01:46.012 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T15:01:46.012 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T15:01:46.012 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T15:01:46.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.017+0000 7fde10bf1640 10 monclient: get_monmap_and_config 2024-09-16T15:01:46.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.017+0000 7fde10bf1640 10 monclient: build_initial_monmap 2024-09-16T15:01:46.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.017+0000 7fde10bf1640 10 monclient: monmap: 2024-09-16T15:01:46.017 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T15:01:46.017 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:46.017 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T15:01:46.021593+0000 2024-09-16T15:01:46.017 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T15:01:46.021593+0000 2024-09-16T15:01:46.017 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:46.017 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:46.017 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:46.017 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:46.017 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:46.017 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:46.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 1 Processor -- start 2024-09-16T15:01:46.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 1 -- start start 2024-09-16T15:01:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient: init 2024-09-16T15:01:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:01:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient(hunting): picked mon.noname-b con 0x7fde0c134c60 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient(hunting): picked mon.noname-a con 0x7fde0c0a2c10 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient(hunting): picked mon.noname-c con 0x7fde0c13abd0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0c098a20 con 0x7fde0c0a2c10 2024-09-16T15:01:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0c135750 con 0x7fde0c13abd0 2024-09-16T15:01:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0c0a4900 con 0x7fde0c134c60 2024-09-16T15:01:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient(hunting): authenticate will time out at 2088.137107s 2024-09-16T15:01:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0ad9d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fde0c13abd0 legacy=0x7fde0c13cfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:01:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fde0c134c60 0x7fde0c099bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:44208/0 (socket says 172.21.15.189:44208) 2024-09-16T15:01:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0a59c640 1 -- 172.21.15.189:0/735210945 learned_addr learned my addr 172.21.15.189:0/735210945 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T15:01:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 1 -- 172.21.15.189:0/735210945 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433483663 0 0) 0x7fde0c0a4900 con 0x7fde0c134c60 2024-09-16T15:01:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient(hunting): _negotiate 2024-09-16T15:01:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient(hunting): my global_id is 354185 2024-09-16T15:01:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 1 -- 172.21.15.189:0/735210945 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddf4003530 con 0x7fde0c134c60 2024-09-16T15:01:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 1 -- 172.21.15.189:0/735210945 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002819227 0 0) 0x7fde0c098a20 con 0x7fde0c0a2c10 2024-09-16T15:01:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient(hunting): _negotiate 2024-09-16T15:01:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:46.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient(hunting): my global_id is 334383 2024-09-16T15:01:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 1 -- 172.21.15.189:0/735210945 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde0c0a4900 con 0x7fde0c0a2c10 2024-09-16T15:01:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 1 -- 172.21.15.189:0/735210945 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2604813341 0 0) 0x7fddf4003530 con 0x7fde0c134c60 2024-09-16T15:01:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 1 -- 172.21.15.189:0/735210945 >> v1:172.21.15.189:6789/0 conn(0x7fde0c13abd0 legacy=0x7fde0c13cfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:01:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 1 -- 172.21.15.189:0/735210945 >> v1:172.21.15.26:6789/0 conn(0x7fde0c0a2c10 legacy=0x7fde0c135040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient: _finish_hunting 0 2024-09-16T15:01:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 1 monclient: found mon.noname-b 2024-09-16T15:01:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:01:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 1 -- 172.21.15.189:0/735210945 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c0a4a90 con 0x7fde0c134c60 2024-09-16T15:01:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient: _finish_auth 0 2024-09-16T15:01:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient: _check_auth_tickets 2024-09-16T15:01:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 1 -- 172.21.15.189:0/735210945 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fddfc001620 con 0x7fde0c134c60 2024-09-16T15:01:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 5 monclient: authenticate success, global_id 354185 2024-09-16T15:01:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:01:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient: dump: 2024-09-16T15:01:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:46.029 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:46.029 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:46.029 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:46.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:46.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 1 -- 172.21.15.189:0/735210945 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddfc001840 con 0x7fde0c134c60 2024-09-16T15:01:46.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:46.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:46.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 4 set_mon_vals no callback set 2024-09-16T15:01:46.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 1 -- 172.21.15.189:0/735210945 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fddfc001c10 con 0x7fde0c134c60 2024-09-16T15:01:46.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:46.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:46.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0959a640 10 monclient: dump: 2024-09-16T15:01:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:46.030 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient: shutdown 2024-09-16T15:01:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:46.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 1 -- 172.21.15.189:0/735210945 >> v1:172.21.15.26:6790/0 conn(0x7fde0c134c60 legacy=0x7fde0c099bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 1 -- 172.21.15.189:0/735210945 shutdown_connections 2024-09-16T15:01:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 1 -- 172.21.15.189:0/735210945 >> 172.21.15.189:0/735210945 conn(0x7fde0c0a7b70 msgr2=0x7fde0c0a3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 1 -- 172.21.15.189:0/735210945 shutdown_connections 2024-09-16T15:01:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 1 -- 172.21.15.189:0/735210945 wait complete. 2024-09-16T15:01:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient: build_initial_monmap 2024-09-16T15:01:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient: monmap: 2024-09-16T15:01:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T15:01:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T15:01:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T15:01:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T15:01:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:46.032 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 1 Processor -- start 2024-09-16T15:01:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 1 -- start start 2024-09-16T15:01:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient: init 2024-09-16T15:01:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:01:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient(hunting): picked mon.noname-a con 0x7fde0c0a2c10 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient(hunting): picked mon.noname-c con 0x7fde0c134c60 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient(hunting): picked mon.noname-b con 0x7fde0c13abd0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0c0a1240 con 0x7fde0c0a2c10 2024-09-16T15:01:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0c09f160 con 0x7fde0c13abd0 2024-09-16T15:01:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0c1347b0 con 0x7fde0c134c60 2024-09-16T15:01:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde10bf1640 10 monclient(hunting): authenticate will time out at 2088.139808s 2024-09-16T15:01:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde0ad9d640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fde0c13abd0 legacy=0x7fde0c1d1950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:01:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde09d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fde0c134c60 0x7fde0c09ea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.189:44218/0 (socket says 172.21.15.189:44218) 2024-09-16T15:01:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.021+0000 7fde09d9b640 1 -- 172.21.15.189:0/37163962 learned_addr learned my addr 172.21.15.189:0/37163962 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T15:01:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077455977 0 0) 0x7fde0c1347b0 con 0x7fde0c134c60 2024-09-16T15:01:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient(hunting): my global_id is 354188 2024-09-16T15:01:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddec003530 con 0x7fde0c134c60 2024-09-16T15:01:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402806393 0 0) 0x7fde0c0a1240 con 0x7fde0c0a2c10 2024-09-16T15:01:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient(hunting): _negotiate 2024-09-16T15:01:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient(hunting): my global_id is 334386 2024-09-16T15:01:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde0c1347b0 con 0x7fde0c0a2c10 2024-09-16T15:01:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3325761262 0 0) 0x7fddec003530 con 0x7fde0c134c60 2024-09-16T15:01:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde0c0a1240 con 0x7fde0c134c60 2024-09-16T15:01:46.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fde000019e0 con 0x7fde0c134c60 2024-09-16T15:01:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:01:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient(hunting): dump: 2024-09-16T15:01:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:46.036 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 479659174 0 0) 0x7fde0c1347b0 con 0x7fde0c0a2c10 2024-09-16T15:01:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddec003530 con 0x7fde0c0a2c10 2024-09-16T15:01:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fddfc001b50 con 0x7fde0c0a2c10 2024-09-16T15:01:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:01:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient(hunting): dump: 2024-09-16T15:01:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:46.037 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1810772810 0 0) 0x7fde0c0a1240 con 0x7fde0c134c60 2024-09-16T15:01:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 >> v1:172.21.15.189:6789/0 conn(0x7fde0c13abd0 legacy=0x7fde0c1d1950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:01:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 >> v1:172.21.15.26:6789/0 conn(0x7fde0c0a2c10 legacy=0x7fde0c09e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient: _finish_hunting 0 2024-09-16T15:01:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 monclient: found mon.c 2024-09-16T15:01:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c0a0c80 con 0x7fde0c134c60 2024-09-16T15:01:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient: _finish_auth 0 2024-09-16T15:01:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient: _check_auth_tickets 2024-09-16T15:01:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fde10bf1640 5 monclient: authenticate success, global_id 354188 2024-09-16T15:01:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fde10bf1640 10 monclient: _renew_subs 2024-09-16T15:01:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fde10bf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fde10bf1640 1 -- 172.21.15.189:0/37163962 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0c0a0f60 con 0x7fde0c134c60 2024-09-16T15:01:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fde10bf1640 10 monclient: _renew_subs 2024-09-16T15:01:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fde10bf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fde10bf1640 1 -- 172.21.15.189:0/37163962 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde0c13dbd0 con 0x7fde0c134c60 2024-09-16T15:01:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde00001d20 con 0x7fde0c134c60 2024-09-16T15:01:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fde00003a50 con 0x7fde0c134c60 2024-09-16T15:01:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fde08d99640 4 set_mon_vals no callback set 2024-09-16T15:01:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 10 monclient: dump: 2024-09-16T15:01:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T15:01:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T15:01:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:46.040 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T15:01:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fde10bf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fde10bf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fde10bf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fde10bf1640 1 -- 172.21.15.189:0/37163962 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf0005050 con 0x7fde0c134c60 2024-09-16T15:01:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fde00003bb0 con 0x7fde0c134c60 2024-09-16T15:01:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fde00043dc0 con 0x7fde0c134c60 2024-09-16T15:01:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.025+0000 7fde0a59c640 10 monclient: get_auth_request con 0x7fddec036270 auth_method 0 2024-09-16T15:01:46.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.029+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fde00014760 con 0x7fde0c134c60 2024-09-16T15:01:46.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.029+0000 7fddfaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:46.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.029+0000 7fddfaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:46.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 10 monclient: get_monmap_and_config 2024-09-16T15:01:46.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 10 monclient: build_initial_monmap 2024-09-16T15:01:46.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 10 monclient: monmap: 2024-09-16T15:01:46.136 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:46.136 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:46.136 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:46.138440+0000 2024-09-16T15:01:46.137 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:46.138440+0000 2024-09-16T15:01:46.137 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:46.137 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:46.137 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:46.137 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:46.137 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:46.137 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:46.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 1 Processor -- start 2024-09-16T15:01:46.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 1 -- start start 2024-09-16T15:01:46.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 10 monclient: init 2024-09-16T15:01:46.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:46.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:01:46.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 10 monclient(hunting): picked mon.noname-c con 0x7f486009cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 10 monclient(hunting): picked mon.noname-a con 0x7f4860132650 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 10 monclient(hunting): picked mon.noname-b con 0x7f4860138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f486009d260 con 0x7f4860132650 2024-09-16T15:01:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4860132a80 con 0x7f486009cb80 2024-09-16T15:01:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f486009cf60 con 0x7f4860138ff0 2024-09-16T15:01:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 10 monclient(hunting): authenticate will time out at 2130.258483s 2024-09-16T15:01:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4865503640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4860138ff0 0x7f486013b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:56820/0 (socket says 172.21.15.26:56820) 2024-09-16T15:01:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4864d02640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f486009cb80 legacy=0x7f48600717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:01:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4865503640 1 -- 172.21.15.26:0/3903119494 learned_addr learned my addr 172.21.15.26:0/3903119494 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 1 -- 172.21.15.26:0/3903119494 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439317008 0 0) 0x7f486009cf60 con 0x7f4860138ff0 2024-09-16T15:01:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:01:46.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 10 monclient(hunting): my global_id is 354191 2024-09-16T15:01:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 1 -- 172.21.15.26:0/3903119494 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4848003530 con 0x7f4860138ff0 2024-09-16T15:01:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 1 -- 172.21.15.26:0/3903119494 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 357528220 0 0) 0x7f4848003530 con 0x7f4860138ff0 2024-09-16T15:01:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 1 -- 172.21.15.26:0/3903119494 >> v1:172.21.15.189:6789/0 conn(0x7f486009cb80 legacy=0x7f48600717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:01:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 1 -- 172.21.15.26:0/3903119494 >> v1:172.21.15.26:6789/0 conn(0x7f4860132650 legacy=0x7f48601378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 10 monclient: _finish_hunting 0 2024-09-16T15:01:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 1 monclient: found mon.noname-b 2024-09-16T15:01:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:01:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 1 -- 172.21.15.26:0/3903119494 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486009f2d0 con 0x7f4860138ff0 2024-09-16T15:01:46.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 10 monclient: _finish_auth 0 2024-09-16T15:01:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 10 monclient: _check_auth_tickets 2024-09-16T15:01:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 1 -- 172.21.15.26:0/3903119494 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f48580019e0 con 0x7f4860138ff0 2024-09-16T15:01:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 5 monclient: authenticate success, global_id 354191 2024-09-16T15:01:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:01:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 10 monclient: dump: 2024-09-16T15:01:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.147 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.148 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:46.148 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:46.148 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:46.148 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:46.148 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:46.148 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:46.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:46.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 1 -- 172.21.15.26:0/3903119494 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4858001c40 con 0x7f4860138ff0 2024-09-16T15:01:46.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:46.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:46.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 1 -- 172.21.15.26:0/3903119494 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4858001fd0 con 0x7f4860138ff0 2024-09-16T15:01:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 4 set_mon_vals no callback set 2024-09-16T15:01:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f485f7fe640 10 monclient: dump: 2024-09-16T15:01:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:46.149 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:46.149 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 10 monclient: shutdown 2024-09-16T15:01:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 1 -- 172.21.15.26:0/3903119494 >> v1:172.21.15.26:6790/0 conn(0x7f4860138ff0 legacy=0x7f486013b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 1 -- 172.21.15.26:0/3903119494 shutdown_connections 2024-09-16T15:01:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.136+0000 7f4866f66640 1 -- 172.21.15.26:0/3903119494 >> 172.21.15.26:0/3903119494 conn(0x7f486009fb50 msgr2=0x7f486009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 1 -- 172.21.15.26:0/3903119494 shutdown_connections 2024-09-16T15:01:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 1 -- 172.21.15.26:0/3903119494 wait complete. 2024-09-16T15:01:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 monclient: build_initial_monmap 2024-09-16T15:01:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 monclient: monmap: 2024-09-16T15:01:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:46.150 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:46.151 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 1 Processor -- start 2024-09-16T15:01:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 1 -- start start 2024-09-16T15:01:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 monclient: init 2024-09-16T15:01:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:01:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 monclient(hunting): picked mon.noname-a con 0x7f486009cb80 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:46.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 monclient(hunting): picked mon.noname-c con 0x7f4860132650 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 monclient(hunting): picked mon.noname-b con 0x7f4860138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f486006ddd0 con 0x7f486009cb80 2024-09-16T15:01:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f486009b370 con 0x7f4860138ff0 2024-09-16T15:01:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f486013c600 con 0x7f4860132650 2024-09-16T15:01:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 monclient(hunting): authenticate will time out at 2130.261458s 2024-09-16T15:01:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4864d02640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f486009cb80 0x7f486009a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40046/0 (socket says 172.21.15.26:40046) 2024-09-16T15:01:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4860132650 0x7f486009ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:56832/0 (socket says 172.21.15.26:56832) 2024-09-16T15:01:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4864d02640 1 -- 172.21.15.26:0/683703850 learned_addr learned my addr 172.21.15.26:0/683703850 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4865503640 1 -- 172.21.15.26:0/683703850 >> v1:172.21.15.189:6789/0 conn(0x7f4860138ff0 legacy=0x7f48601d1b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:01:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 1 -- 172.21.15.26:0/683703850 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949318611 0 0) 0x7f486013c600 con 0x7f4860132650 2024-09-16T15:01:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 10 monclient(hunting): _negotiate 2024-09-16T15:01:46.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 10 monclient(hunting): my global_id is 354194 2024-09-16T15:01:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 1 -- 172.21.15.26:0/683703850 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f484c003530 con 0x7f4860132650 2024-09-16T15:01:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 1 -- 172.21.15.26:0/683703850 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3380480565 0 0) 0x7f484c003530 con 0x7f4860132650 2024-09-16T15:01:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 1 -- 172.21.15.26:0/683703850 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f486013c600 con 0x7f4860132650 2024-09-16T15:01:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 1 -- 172.21.15.26:0/683703850 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4854001830 con 0x7f4860132650 2024-09-16T15:01:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:01:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 10 monclient(hunting): dump: 2024-09-16T15:01:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.153 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:46.154 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 1 -- 172.21.15.26:0/683703850 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3224301335 0 0) 0x7f486013c600 con 0x7f4860132650 2024-09-16T15:01:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 1 -- 172.21.15.26:0/683703850 >> v1:172.21.15.189:6789/0 conn(0x7f4860138ff0 legacy=0x7f48601d1b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:01:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 1 -- 172.21.15.26:0/683703850 >> v1:172.21.15.26:6789/0 conn(0x7f486009cb80 legacy=0x7f486009a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 10 monclient: _finish_hunting 0 2024-09-16T15:01:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 1 monclient: found mon.c 2024-09-16T15:01:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 1 -- 172.21.15.26:0/683703850 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486009a240 con 0x7f4860132650 2024-09-16T15:01:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 10 monclient: _finish_auth 0 2024-09-16T15:01:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 10 monclient: _check_auth_tickets 2024-09-16T15:01:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 5 monclient: authenticate success, global_id 354194 2024-09-16T15:01:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 monclient: _renew_subs 2024-09-16T15:01:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 1 -- 172.21.15.26:0/683703850 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f486006daf0 con 0x7f4860132650 2024-09-16T15:01:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 monclient: _renew_subs 2024-09-16T15:01:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 1 -- 172.21.15.26:0/683703850 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4860138b90 con 0x7f4860132650 2024-09-16T15:01:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 1 -- 172.21.15.26:0/683703850 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4854003620 con 0x7f4860132650 2024-09-16T15:01:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 1 -- 172.21.15.26:0/683703850 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f48540039b0 con 0x7f4860132650 2024-09-16T15:01:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485effd640 4 set_mon_vals no callback set 2024-09-16T15:01:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 10 monclient: dump: 2024-09-16T15:01:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:46.157 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4866f66640 1 -- 172.21.15.26:0/683703850 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4840005050 con 0x7f4860132650 2024-09-16T15:01:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 1 -- 172.21.15.26:0/683703850 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f4854003b90 con 0x7f4860132650 2024-09-16T15:01:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f485d7fa640 1 -- 172.21.15.26:0/683703850 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f4854043df0 con 0x7f4860132650 2024-09-16T15:01:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.140+0000 7f4864d02640 10 monclient: get_auth_request con 0x7f484c036000 auth_method 0 2024-09-16T15:01:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.144+0000 7f485d7fa640 1 -- 172.21.15.26:0/683703850 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4854014730 con 0x7f4860132650 2024-09-16T15:01:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.144+0000 7f485d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.144+0000 7f485d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.237+0000 7fde10bf1640 10 start_mon_command cmd=[{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T15:01:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.237+0000 7fde10bf1640 10 monclient: _send_command 2 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T15:01:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.237+0000 7fde10bf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:46.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:46.237+0000 7fde10bf1640 1 -- 172.21.15.189:0/37163962 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "auth rm", "entity": "client.bad"} v 0) v1 -- 0x7fddf0005300 con 0x7fde0c134c60 2024-09-16T15:01:46.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.368+0000 7f4866f66640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T15:01:46.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.368+0000 7f4866f66640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:46.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.368+0000 7f4866f66640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:46.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.368+0000 7f4866f66640 1 -- 172.21.15.26:0/683703850 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4840005880 con 0x7f4860132650 2024-09-16T15:01:46.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.368+0000 7f485d7fa640 1 -- 172.21.15.26:0/683703850 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1363334158) 0x7f485401cd20 con 0x7f4860132650 2024-09-16T15:01:46.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.368+0000 7f485d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:46.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.368+0000 7f485d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:46.371 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":174,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T15:01:46.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.372+0000 7f4866f66640 1 -- 172.21.15.26:0/683703850 >> v1:172.21.15.189:6812/3196161582 conn(0x7f484c036000 legacy=0x7f484c0384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:46.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.372+0000 7f4866f66640 10 monclient: shutdown 2024-09-16T15:01:46.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.372+0000 7f4866f66640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:46.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.372+0000 7f4866f66640 1 -- 172.21.15.26:0/683703850 >> v1:172.21.15.26:6790/0 conn(0x7f4860132650 legacy=0x7f486009ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:46.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.372+0000 7f4866f66640 1 -- 172.21.15.26:0/683703850 shutdown_connections 2024-09-16T15:01:46.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.372+0000 7f4866f66640 1 -- 172.21.15.26:0/683703850 >> 172.21.15.26:0/683703850 conn(0x7f486009fb50 msgr2=0x7f48600a13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:46.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.372+0000 7f4866f66640 1 -- 172.21.15.26:0/683703850 shutdown_connections 2024-09-16T15:01:46.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:46.372+0000 7f4866f66640 1 -- 172.21.15.26:0/683703850 wait complete. 2024-09-16T15:01:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:49.021+0000 7fddf9ffb640 10 monclient: tick 2024-09-16T15:01:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:49.021+0000 7fddf9ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:49.021+0000 7fddf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:49.025+0000 7fddf9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:49.390 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T15:01:49.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.520+0000 7f22fda7e640 10 monclient: get_monmap_and_config 2024-09-16T15:01:49.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.520+0000 7f22fda7e640 10 monclient: build_initial_monmap 2024-09-16T15:01:49.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.520+0000 7f22fda7e640 10 monclient: monmap: 2024-09-16T15:01:49.521 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:49.521 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:49.522 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:49.523715+0000 2024-09-16T15:01:49.522 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:49.523715+0000 2024-09-16T15:01:49.522 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:49.522 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:49.522 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:49.522 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:49.522 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:49.522 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:49.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.520+0000 7f22fda7e640 1 Processor -- start 2024-09-16T15:01:49.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.520+0000 7f22fda7e640 1 -- start start 2024-09-16T15:01:49.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.520+0000 7f22fda7e640 10 monclient: init 2024-09-16T15:01:49.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.520+0000 7f22fda7e640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:49.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.520+0000 7f22fda7e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:01:49.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.520+0000 7f22fda7e640 10 monclient(hunting): picked mon.noname-b con 0x7f22f80a0050 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:49.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 10 monclient(hunting): picked mon.noname-c con 0x7f22f80a0b40 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:49.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 10 monclient(hunting): picked mon.noname-a con 0x7f22f80a0f50 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:49.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f806cb40 con 0x7f22f80a0f50 2024-09-16T15:01:49.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f806c210 con 0x7f22f80a0b40 2024-09-16T15:01:49.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f806c4f0 con 0x7f22f80a0050 2024-09-16T15:01:49.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:49.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f6ffd640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f22f80a0b40 legacy=0x7f22f8139d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:01:49.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 10 monclient(hunting): authenticate will time out at 2133.644200s 2024-09-16T15:01:49.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f77fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f22f80a0050 0x7f22f80a0430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:44580/0 (socket says 172.21.15.26:44580) 2024-09-16T15:01:49.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f77fe640 1 -- 172.21.15.26:0/2587344765 learned_addr learned my addr 172.21.15.26:0/2587344765 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:49.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 1 -- 172.21.15.26:0/2587344765 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524036121 0 0) 0x7f22f806c4f0 con 0x7f22f80a0050 2024-09-16T15:01:49.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 10 monclient(hunting): _negotiate 2024-09-16T15:01:49.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:49.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:49.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 10 monclient(hunting): my global_id is 354197 2024-09-16T15:01:49.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 1 -- 172.21.15.26:0/2587344765 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e0003530 con 0x7f22f80a0050 2024-09-16T15:01:49.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 1 -- 172.21.15.26:0/2587344765 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2823905352 0 0) 0x7f22e0003530 con 0x7f22f80a0050 2024-09-16T15:01:49.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 1 -- 172.21.15.26:0/2587344765 >> v1:172.21.15.189:6789/0 conn(0x7f22f80a0b40 legacy=0x7f22f8139d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:01:49.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 1 -- 172.21.15.26:0/2587344765 >> v1:172.21.15.26:6789/0 conn(0x7f22f80a0f50 legacy=0x7f22f813d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:49.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 10 monclient: _finish_hunting 0 2024-09-16T15:01:49.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 1 monclient: found mon.noname-b 2024-09-16T15:01:49.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:49.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:01:49.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 1 -- 172.21.15.26:0/2587344765 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f806c7d0 con 0x7f22f80a0050 2024-09-16T15:01:49.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 10 monclient: _finish_auth 0 2024-09-16T15:01:49.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 10 monclient: _check_auth_tickets 2024-09-16T15:01:49.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:49.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 5 monclient: authenticate success, global_id 354197 2024-09-16T15:01:49.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:49.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 1 -- 172.21.15.26:0/2587344765 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22e8001620 con 0x7f22f80a0050 2024-09-16T15:01:49.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:49.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:01:49.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 10 monclient: dump: 2024-09-16T15:01:49.531 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:49.531 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:49.531 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:49.531 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:49.531 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:49.531 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:49.531 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:49.531 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:49.531 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:49.531 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:49.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:01:49.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 1 -- 172.21.15.26:0/2587344765 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22e80018d0 con 0x7f22f80a0050 2024-09-16T15:01:49.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:49.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 1 -- 172.21.15.26:0/2587344765 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22e8001c60 con 0x7f22f80a0050 2024-09-16T15:01:49.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:49.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:49.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f67fc640 10 monclient: dump: 2024-09-16T15:01:49.532 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:49.532 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:49.532 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:49.532 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:49.532 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:49.533 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:49.533 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:49.533 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:49.533 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:49.533 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:49.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 10 monclient: get_monmap_and_config success 2024-09-16T15:01:49.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 4 set_mon_vals no callback set 2024-09-16T15:01:49.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 10 monclient: shutdown 2024-09-16T15:01:49.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:49.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 1 -- 172.21.15.26:0/2587344765 >> v1:172.21.15.26:6790/0 conn(0x7f22f80a0050 legacy=0x7f22f80a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:49.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 1 -- 172.21.15.26:0/2587344765 shutdown_connections 2024-09-16T15:01:49.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 1 -- 172.21.15.26:0/2587344765 >> 172.21.15.26:0/2587344765 conn(0x7f22f80a5540 msgr2=0x7f22f80a5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:49.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 1 -- 172.21.15.26:0/2587344765 shutdown_connections 2024-09-16T15:01:49.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 1 -- 172.21.15.26:0/2587344765 wait complete. 2024-09-16T15:01:49.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 10 monclient: build_initial_monmap 2024-09-16T15:01:49.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 10 monclient: monmap: 2024-09-16T15:01:49.534 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:49.534 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:01:49.534 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:01:49.534 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:01:49.534 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:49.534 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:01:49.534 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:49.534 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:01:49.534 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:01:49.534 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:49.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 1 Processor -- start 2024-09-16T15:01:49.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 1 -- start start 2024-09-16T15:01:49.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 10 monclient: init 2024-09-16T15:01:49.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:49.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:01:49.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 10 monclient(hunting): picked mon.noname-c con 0x7f22f80a0050 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:49.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 10 monclient(hunting): picked mon.noname-a con 0x7f22f80a0b40 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:49.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 10 monclient(hunting): picked mon.noname-b con 0x7f22f80a0f50 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:49.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f813dd70 con 0x7f22f80a0b40 2024-09-16T15:01:49.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f809c360 con 0x7f22f80a0f50 2024-09-16T15:01:49.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f806b540 con 0x7f22f80a0050 2024-09-16T15:01:49.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:49.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22fda7e640 10 monclient(hunting): authenticate will time out at 2133.647224s 2024-09-16T15:01:49.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f77fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f22f80a0050 0x7f22f813a0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:44584/0 (socket says 172.21.15.26:44584) 2024-09-16T15:01:49.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f77fe640 1 -- 172.21.15.26:0/1764047379 learned_addr learned my addr 172.21.15.26:0/1764047379 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:49.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22f7fff640 1 -- 172.21.15.26:0/1764047379 >> v1:172.21.15.189:6789/0 conn(0x7f22f80a0f50 legacy=0x7f22f81d1a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:01:49.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22dffff640 1 -- 172.21.15.26:0/1764047379 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779539510 0 0) 0x7f22f806b540 con 0x7f22f80a0050 2024-09-16T15:01:49.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22dffff640 10 monclient(hunting): _negotiate 2024-09-16T15:01:49.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:01:49.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:01:49.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22dffff640 10 monclient(hunting): my global_id is 354200 2024-09-16T15:01:49.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22dffff640 1 -- 172.21.15.26:0/1764047379 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e4003530 con 0x7f22f80a0050 2024-09-16T15:01:49.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22dffff640 1 -- 172.21.15.26:0/1764047379 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 801293724 0 0) 0x7f22e4003530 con 0x7f22f80a0050 2024-09-16T15:01:49.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22dffff640 1 -- 172.21.15.26:0/1764047379 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22f806b540 con 0x7f22f80a0050 2024-09-16T15:01:49.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22dffff640 1 -- 172.21.15.26:0/1764047379 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22e8001c20 con 0x7f22f80a0050 2024-09-16T15:01:49.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:49.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22dffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:01:49.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22dffff640 10 monclient(hunting): dump: 2024-09-16T15:01:49.537 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:49.537 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:49.537 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:49.537 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:49.537 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:49.537 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:49.537 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:49.537 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:49.537 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:49.537 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:49.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22dffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:01:49.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.524+0000 7f22dffff640 1 -- 172.21.15.26:0/1764047379 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 642339380 0 0) 0x7f22f806b540 con 0x7f22f80a0050 2024-09-16T15:01:49.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 1 -- 172.21.15.26:0/1764047379 >> v1:172.21.15.189:6789/0 conn(0x7f22f80a0f50 legacy=0x7f22f81d1a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:01:49.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 1 -- 172.21.15.26:0/1764047379 >> v1:172.21.15.26:6789/0 conn(0x7f22f80a0b40 legacy=0x7f22f813a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:49.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 10 monclient: _finish_hunting 0 2024-09-16T15:01:49.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 1 monclient: found mon.c 2024-09-16T15:01:49.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:49.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:49.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 1 -- 172.21.15.26:0/1764047379 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f813aec0 con 0x7f22f80a0050 2024-09-16T15:01:49.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 10 monclient: _finish_auth 0 2024-09-16T15:01:49.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 10 monclient: _check_auth_tickets 2024-09-16T15:01:49.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:49.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22fda7e640 5 monclient: authenticate success, global_id 354200 2024-09-16T15:01:49.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22fda7e640 10 monclient: _renew_subs 2024-09-16T15:01:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22fda7e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22fda7e640 1 -- 172.21.15.26:0/1764047379 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f80994b0 con 0x7f22f80a0050 2024-09-16T15:01:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22fda7e640 10 monclient: _renew_subs 2024-09-16T15:01:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22fda7e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22fda7e640 1 -- 172.21.15.26:0/1764047379 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f813c640 con 0x7f22f80a0050 2024-09-16T15:01:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 1 -- 172.21.15.26:0/1764047379 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22e8004aa0 con 0x7f22f80a0050 2024-09-16T15:01:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:01:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 1 -- 172.21.15.26:0/1764047379 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f22e8004db0 con 0x7f22f80a0050 2024-09-16T15:01:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:01:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22f5ffb640 4 set_mon_vals no callback set 2024-09-16T15:01:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:01:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 10 monclient: dump: 2024-09-16T15:01:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:01:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:49.540 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:01:49.540 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:01:49.540 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:49.540 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:01:49.540 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:01:49.540 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:01:49.540 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:49.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 1 -- 172.21.15.26:0/1764047379 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f22e8004f90 con 0x7f22f80a0050 2024-09-16T15:01:49.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22fda7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:49.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22fda7e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:49.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22fda7e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:49.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22fda7e640 1 -- 172.21.15.26:0/1764047379 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22d8005050 con 0x7f22f80a0050 2024-09-16T15:01:49.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22dffff640 1 -- 172.21.15.26:0/1764047379 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f22e8043fe0 con 0x7f22f80a0050 2024-09-16T15:01:49.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.528+0000 7f22f6ffd640 10 monclient: get_auth_request con 0x7f22e4036050 auth_method 0 2024-09-16T15:01:49.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.532+0000 7f22dffff640 1 -- 172.21.15.26:0/1764047379 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f22e8014980 con 0x7f22f80a0050 2024-09-16T15:01:49.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.532+0000 7f22dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:01:49.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.532+0000 7f22dffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:01:49.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.752+0000 7f22fda7e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T15:01:49.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.752+0000 7f22fda7e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:49.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.752+0000 7f22fda7e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:01:49.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.752+0000 7f22fda7e640 1 -- 172.21.15.26:0/1764047379 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f22d8005880 con 0x7f22f80a0050 2024-09-16T15:01:49.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.752+0000 7f22dffff640 1 -- 172.21.15.26:0/1764047379 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 28187183) 0x7f22e8014570 con 0x7f22f80a0050 2024-09-16T15:01:49.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.752+0000 7f22dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T15:01:49.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.752+0000 7f22dffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:01:49.754 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":174,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T15:01:49.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.756+0000 7f22fda7e640 1 -- 172.21.15.26:0/1764047379 >> v1:172.21.15.189:6812/3196161582 conn(0x7f22e4036050 legacy=0x7f22e4038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:49.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.756+0000 7f22fda7e640 10 monclient: shutdown 2024-09-16T15:01:49.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.756+0000 7f22fda7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:01:49.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.756+0000 7f22fda7e640 1 -- 172.21.15.26:0/1764047379 >> v1:172.21.15.26:6790/0 conn(0x7f22f80a0050 legacy=0x7f22f813a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:49.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.756+0000 7f22fda7e640 1 -- 172.21.15.26:0/1764047379 shutdown_connections 2024-09-16T15:01:49.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.756+0000 7f22fda7e640 1 -- 172.21.15.26:0/1764047379 >> 172.21.15.26:0/1764047379 conn(0x7f22f80a5540 msgr2=0x7f22f80714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:01:49.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.756+0000 7f22fda7e640 1 -- 172.21.15.26:0/1764047379 shutdown_connections 2024-09-16T15:01:49.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:49.756+0000 7f22fda7e640 1 -- 172.21.15.26:0/1764047379 wait complete. 2024-09-16T15:01:50.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:50.025+0000 7fddf9ffb640 10 monclient: tick 2024-09-16T15:01:50.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:50.025+0000 7fddf9ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:50.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:50.025+0000 7fddf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:51.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:51.025+0000 7fddf9ffb640 10 monclient: tick 2024-09-16T15:01:51.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:51.025+0000 7fddf9ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:51.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:51.025+0000 7fddf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:52.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:52.025+0000 7fddf9ffb640 10 monclient: tick 2024-09-16T15:01:52.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:52.025+0000 7fddf9ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:52.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:52.025+0000 7fddf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:52.776 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T15:01:52.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f025106f640 10 monclient: get_monmap_and_config 2024-09-16T15:01:52.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f025106f640 10 monclient: build_initial_monmap 2024-09-16T15:01:52.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f025106f640 10 monclient: monmap: 2024-09-16T15:01:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:01:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:01:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:01:52.910648+0000 2024-09-16T15:01:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:01:52.910648+0000 2024-09-16T15:01:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:01:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:01:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:01:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:01:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:01:52.909 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:01:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f025106f640 1 Processor -- start 2024-09-16T15:01:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f025106f640 1 -- start start 2024-09-16T15:01:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f025106f640 10 monclient: init 2024-09-16T15:01:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f025106f640 10 monclient: _reopen_session rank -1 2024-09-16T15:01:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f025106f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:01:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f025106f640 10 monclient(hunting): picked mon.noname-b con 0x7f024c0a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f025106f640 10 monclient(hunting): picked mon.noname-a con 0x7f024c0a0000 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f025106f640 10 monclient(hunting): picked mon.noname-c con 0x7f024c0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f025106f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c0a3250 con 0x7f024c0a0000 2024-09-16T15:01:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f025106f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c09f600 con 0x7f024c0a2800 2024-09-16T15:01:52.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f025106f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c0a3830 con 0x7f024c0a1d10 2024-09-16T15:01:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f025106f640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f025106f640 10 monclient(hunting): authenticate will time out at 2137.030833s 2024-09-16T15:01:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f024ad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f024c0a1d10 0x7f024c0a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:44586/0 (socket says 172.21.15.26:44586) 2024-09-16T15:01:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f024ad9d640 1 -- 172.21.15.26:0/73320976 learned_addr learned my addr 172.21.15.26:0/73320976 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:01:52.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:52.908+0000 7f024b59e640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.189:6789/0 conn(0x7f024c0a2800 legacy=0x7f024c13d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:01:53.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:53.025+0000 7fddf9ffb640 10 monclient: tick 2024-09-16T15:01:53.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:53.025+0000 7fddf9ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:53.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:53.025+0000 7fddf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:53.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:53.112+0000 7f024b59e640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.189:6789/0 conn(0x7f024c0a2800 legacy=0x7f024c13d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:01:53.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:53.516+0000 7f024b59e640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.189:6789/0 conn(0x7f024c0a2800 legacy=0x7f024c13d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:01:54.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:54.025+0000 7fddf9ffb640 10 monclient: tick 2024-09-16T15:01:54.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:54.025+0000 7fddf9ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:54.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:54.025+0000 7fddf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:54.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:54.320+0000 7f024b59e640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.189:6789/0 conn(0x7f024c0a2800 legacy=0x7f024c13d440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:01:55.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:55.025+0000 7fddf9ffb640 10 monclient: tick 2024-09-16T15:01:55.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:55.025+0000 7fddf9ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:55.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:55.025+0000 7fddf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:55.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:55.908+0000 7f0248d99640 10 monclient(hunting): tick 2024-09-16T15:01:55.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:55.908+0000 7f0248d99640 10 monclient(hunting): _check_auth_tickets 2024-09-16T15:01:55.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:55.908+0000 7f0248d99640 1 monclient(hunting): continuing hunt 2024-09-16T15:01:55.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:55.908+0000 7f0248d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T15:01:55.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:55.908+0000 7f0248d99640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.26:6790/0 conn(0x7f024c0a1d10 legacy=0x7f024c0a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:55.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:55.908+0000 7f0248d99640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.189:6789/0 conn(0x7f024c0a2800 legacy=0x7f024c13d440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:01:55.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:55.908+0000 7f0248d99640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.26:6789/0 conn(0x7f024c0a0000 legacy=0x7f024c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:55.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:55.908+0000 7f0248d99640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:01:55.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:55.908+0000 7f0248d99640 10 monclient(hunting): picked mon.noname-c con 0x7f02300036d0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:55.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:55.908+0000 7f0248d99640 10 monclient(hunting): picked mon.noname-a con 0x7f0230007360 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:55.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:55.908+0000 7f0248d99640 10 monclient(hunting): picked mon.noname-b con 0x7f023000b090 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:55.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:55.908+0000 7f0248d99640 1 -- 172.21.15.26:0/73320976 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c09ed60 con 0x7f0230007360 2024-09-16T15:01:55.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:55.908+0000 7f0248d99640 1 -- 172.21.15.26:0/73320976 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c09f600 con 0x7f02300036d0 2024-09-16T15:01:55.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:55.908+0000 7f0248d99640 1 -- 172.21.15.26:0/73320976 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0230010dd0 con 0x7f023000b090 2024-09-16T15:01:55.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:55.908+0000 7f0248d99640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:55.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:55.908+0000 7f024ad9d640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.189:6789/0 conn(0x7f02300036d0 legacy=0x7f0230005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:01:56.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:56.025+0000 7fddf9ffb640 10 monclient: tick 2024-09-16T15:01:56.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:56.025+0000 7fddf9ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:56.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:56.025+0000 7fddf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:56.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:56.112+0000 7f024ad9d640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.189:6789/0 conn(0x7f02300036d0 legacy=0x7f0230005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:01:56.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:56.516+0000 7f024ad9d640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.189:6789/0 conn(0x7f02300036d0 legacy=0x7f0230005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:01:57.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:57.025+0000 7fddf9ffb640 10 monclient: tick 2024-09-16T15:01:57.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:57.025+0000 7fddf9ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:57.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:57.025+0000 7fddf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:57.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:57.320+0000 7f024ad9d640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.189:6789/0 conn(0x7f02300036d0 legacy=0x7f0230005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:01:58.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:58.025+0000 7fddf9ffb640 10 monclient: tick 2024-09-16T15:01:58.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:58.025+0000 7fddf9ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:58.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:58.025+0000 7fddf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:58.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:58.912+0000 7f0248d99640 10 monclient(hunting): tick 2024-09-16T15:01:58.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:58.912+0000 7f0248d99640 10 monclient(hunting): _check_auth_tickets 2024-09-16T15:01:58.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:58.912+0000 7f0248d99640 1 monclient(hunting): continuing hunt 2024-09-16T15:01:58.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:58.912+0000 7f0248d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T15:01:58.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:58.912+0000 7f0248d99640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.26:6790/0 conn(0x7f023000b090 legacy=0x7f023000d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:58.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:58.912+0000 7f0248d99640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.189:6789/0 conn(0x7f02300036d0 legacy=0x7f0230005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:01:58.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:58.912+0000 7f0248d99640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.26:6789/0 conn(0x7f0230007360 legacy=0x7f02300097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:01:58.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:58.912+0000 7f0248d99640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:01:58.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:58.912+0000 7f0248d99640 10 monclient(hunting): picked mon.noname-b con 0x7f0230011090 addr v1:172.21.15.26:6790/0 2024-09-16T15:01:58.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:58.912+0000 7f0248d99640 10 monclient(hunting): picked mon.noname-c con 0x7f02300036d0 addr v1:172.21.15.189:6789/0 2024-09-16T15:01:58.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:58.912+0000 7f0248d99640 10 monclient(hunting): picked mon.noname-a con 0x7f02300132c0 addr v1:172.21.15.26:6789/0 2024-09-16T15:01:58.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:58.912+0000 7f0248d99640 1 -- 172.21.15.26:0/73320976 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0230011fc0 con 0x7f02300132c0 2024-09-16T15:01:58.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:58.912+0000 7f0248d99640 1 -- 172.21.15.26:0/73320976 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c09f600 con 0x7f02300036d0 2024-09-16T15:01:58.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:58.912+0000 7f0248d99640 1 -- 172.21.15.26:0/73320976 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023000dcf0 con 0x7f0230011090 2024-09-16T15:01:58.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:58.912+0000 7f0248d99640 10 monclient(hunting): _renew_subs 2024-09-16T15:01:58.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:58.912+0000 7f024a59c640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.189:6789/0 conn(0x7f02300036d0 legacy=0x7f0230005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:01:59.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:59.025+0000 7fddf9ffb640 10 monclient: tick 2024-09-16T15:01:59.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:59.025+0000 7fddf9ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:01:59.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:59.025+0000 7fddf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:01:59.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:01:59.025+0000 7fddf9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:01:59.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:59.116+0000 7f024a59c640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.189:6789/0 conn(0x7f02300036d0 legacy=0x7f0230005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:01:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:01:59.520+0000 7f024a59c640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.189:6789/0 conn(0x7f02300036d0 legacy=0x7f0230005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:02:00.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.025+0000 7fddf9ffb640 10 monclient: tick 2024-09-16T15:02:00.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.025+0000 7fddf9ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:02:00.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.025+0000 7fddf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 1 -- 172.21.15.26:0/73320976 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332171255 0 0) 0x7f0230011fc0 con 0x7f02300132c0 2024-09-16T15:02:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 10 monclient(hunting): _negotiate 2024-09-16T15:02:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 10 monclient(hunting): my global_id is 334389 2024-09-16T15:02:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 1 -- 172.21.15.26:0/73320976 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f022c003530 con 0x7f02300132c0 2024-09-16T15:02:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 1 -- 172.21.15.26:0/73320976 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 735132070 0 0) 0x7f022c003530 con 0x7f02300132c0 2024-09-16T15:02:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.26:6790/0 conn(0x7f0230011090 legacy=0x7f02300114d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.189:6789/0 conn(0x7f02300036d0 legacy=0x7f0230005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:02:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 10 monclient: _finish_hunting 0 2024-09-16T15:02:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 1 monclient: found mon.noname-a 2024-09-16T15:02:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:02:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 1 -- 172.21.15.26:0/73320976 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023000ded0 con 0x7f02300132c0 2024-09-16T15:02:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 10 monclient: _finish_auth 0 2024-09-16T15:02:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 10 monclient: _check_auth_tickets 2024-09-16T15:02:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 1 -- 172.21.15.26:0/73320976 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f02440044f0 con 0x7f02300132c0 2024-09-16T15:02:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 10 monclient: dump: 2024-09-16T15:02:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.101 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f025106f640 5 monclient: authenticate success, global_id 334389 2024-09-16T15:02:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f025106f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 1 -- 172.21.15.26:0/73320976 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0244004710 con 0x7f02300132c0 2024-09-16T15:02:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f0249d9b640 1 -- 172.21.15.26:0/73320976 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0244004b20 con 0x7f02300132c0 2024-09-16T15:02:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f025106f640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.096+0000 7f025106f640 4 set_mon_vals no callback set 2024-09-16T15:02:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.100+0000 7f0249d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.100+0000 7f0249d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.100+0000 7f0249d9b640 10 monclient: dump: 2024-09-16T15:02:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.103 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.100+0000 7f025106f640 10 monclient: shutdown 2024-09-16T15:02:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.100+0000 7f025106f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.100+0000 7f025106f640 1 -- 172.21.15.26:0/73320976 >> v1:172.21.15.26:6789/0 conn(0x7f02300132c0 legacy=0x7f02300156b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.100+0000 7f025106f640 1 -- 172.21.15.26:0/73320976 shutdown_connections 2024-09-16T15:02:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.100+0000 7f025106f640 1 -- 172.21.15.26:0/73320976 >> 172.21.15.26:0/73320976 conn(0x7f024c0a7f30 msgr2=0x7f024c0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.100+0000 7f025106f640 1 -- 172.21.15.26:0/73320976 shutdown_connections 2024-09-16T15:02:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.100+0000 7f025106f640 1 -- 172.21.15.26:0/73320976 wait complete. 2024-09-16T15:02:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.100+0000 7f025106f640 10 monclient: build_initial_monmap 2024-09-16T15:02:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.100+0000 7f025106f640 10 monclient: monmap: 2024-09-16T15:02:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:00.108 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.100+0000 7f025106f640 1 Processor -- start 2024-09-16T15:02:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.100+0000 7f025106f640 1 -- start start 2024-09-16T15:02:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.100+0000 7f025106f640 10 monclient: init 2024-09-16T15:02:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:02:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 10 monclient(hunting): picked mon.noname-a con 0x7f02300036d0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 10 monclient(hunting): picked mon.noname-b con 0x7f0230011090 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 10 monclient(hunting): picked mon.noname-c con 0x7f02300132c0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c1212f0 con 0x7f02300036d0 2024-09-16T15:02:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c134b40 con 0x7f0230011090 2024-09-16T15:02:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c09e500 con 0x7f02300132c0 2024-09-16T15:02:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 10 monclient(hunting): authenticate will time out at 2144.224819s 2024-09-16T15:02:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f024a59c640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f0230011090 legacy=0x7f024c1ce7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:02:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f024ad9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f02300036d0 0x7f024c1341b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43132/0 (socket says 172.21.15.26:43132) 2024-09-16T15:02:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f024ad9d640 1 -- 172.21.15.26:0/2015475635 learned_addr learned my addr 172.21.15.26:0/2015475635 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 1 -- 172.21.15.26:0/2015475635 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618720530 0 0) 0x7f024c1212f0 con 0x7f02300036d0 2024-09-16T15:02:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 10 monclient(hunting): my global_id is 334392 2024-09-16T15:02:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 1 -- 172.21.15.26:0/2015475635 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0230011fc0 con 0x7f02300036d0 2024-09-16T15:02:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 1 -- 172.21.15.26:0/2015475635 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2616135163 0 0) 0x7f0230011fc0 con 0x7f02300036d0 2024-09-16T15:02:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 1 -- 172.21.15.26:0/2015475635 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f024c1212f0 con 0x7f02300036d0 2024-09-16T15:02:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 1 -- 172.21.15.26:0/2015475635 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f023c001840 con 0x7f02300036d0 2024-09-16T15:02:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 10 monclient(hunting): dump: 2024-09-16T15:02:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.111 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 1 -- 172.21.15.26:0/2015475635 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 311918042 0 0) 0x7f024c1212f0 con 0x7f02300036d0 2024-09-16T15:02:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 1 -- 172.21.15.26:0/2015475635 >> v1:172.21.15.26:6790/0 conn(0x7f02300132c0 legacy=0x7f024c1d1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 1 -- 172.21.15.26:0/2015475635 >> v1:172.21.15.189:6789/0 conn(0x7f0230011090 legacy=0x7f024c1ce7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:02:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 10 monclient: _finish_hunting 0 2024-09-16T15:02:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 1 monclient: found mon.a 2024-09-16T15:02:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 1 -- 172.21.15.26:0/2015475635 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c1348c0 con 0x7f02300036d0 2024-09-16T15:02:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 10 monclient: _finish_auth 0 2024-09-16T15:02:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 10 monclient: _check_auth_tickets 2024-09-16T15:02:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 5 monclient: authenticate success, global_id 334392 2024-09-16T15:02:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 10 monclient: _renew_subs 2024-09-16T15:02:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 1 -- 172.21.15.26:0/2015475635 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f024c0a6830 con 0x7f02300036d0 2024-09-16T15:02:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 10 monclient: _renew_subs 2024-09-16T15:02:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 1 -- 172.21.15.26:0/2015475635 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f024c11f200 con 0x7f02300036d0 2024-09-16T15:02:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 1 -- 172.21.15.26:0/2015475635 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f023c0047f0 con 0x7f02300036d0 2024-09-16T15:02:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 1 -- 172.21.15.26:0/2015475635 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f023c004b80 con 0x7f02300036d0 2024-09-16T15:02:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:00.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f024959a640 4 set_mon_vals no callback set 2024-09-16T15:02:00.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 10 monclient: dump: 2024-09-16T15:02:00.114 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.114 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.114 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.114 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.114 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.114 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.114 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.114 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.114 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.115 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 1 -- 172.21.15.26:0/2015475635 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f023c004d60 con 0x7f02300036d0 2024-09-16T15:02:00.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:00.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:00.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:00.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f025106f640 1 -- 172.21.15.26:0/2015475635 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0224005050 con 0x7f02300036d0 2024-09-16T15:02:00.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f02437fe640 1 -- 172.21.15.26:0/2015475635 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f023c043e00 con 0x7f02300036d0 2024-09-16T15:02:00.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.104+0000 7f024a59c640 10 monclient: get_auth_request con 0x7f0230037e70 auth_method 0 2024-09-16T15:02:00.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.112+0000 7f02437fe640 1 -- 172.21.15.26:0/2015475635 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f023c014740 con 0x7f02300036d0 2024-09-16T15:02:00.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.112+0000 7f02437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:00.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.112+0000 7f02437fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:00.116 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:00.116+0000 7f15e8d50640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 auth entities have invalid capabilities; 1/3 mons down, quorum a,c; 2 pool(s) do not have an application enabled 2024-09-16T15:02:00.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.121+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.bad"}]=0 v49) v1 ==== 79+0+0 (unknown 1768221316 0 0) 0x7fde00012bd0 con 0x7fde0c134c60 2024-09-16T15:02:00.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.121+0000 7fddfaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T15:02:00.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.121+0000 7fddfaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:02:00.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.269+0000 7fde10bf1640 10 start_mon_command cmd=[{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T15:02:00.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.269+0000 7fde10bf1640 10 monclient: _send_command 3 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T15:02:00.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.269+0000 7fde10bf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:00.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.269+0000 7fde10bf1640 1 -- 172.21.15.189:0/37163962 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "auth rm", "entity": "client.bad"} v 0) v1 -- 0x7fddf0005850 con 0x7fde0c134c60 2024-09-16T15:02:00.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.269+0000 7fddfaffd640 1 -- 172.21.15.189:0/37163962 <== mon.2 v1:172.21.15.26:6790/0 11 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.bad"}]=0 v49) v1 ==== 79+0+0 (unknown 1768221316 0 0) 0x7fde00014350 con 0x7fde0c134c60 2024-09-16T15:02:00.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.269+0000 7fddfaffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T15:02:00.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.269+0000 7fddfaffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T15:02:00.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.273+0000 7fde10bf1640 1 -- 172.21.15.189:0/37163962 >> v1:172.21.15.189:6812/3196161582 conn(0x7fddec036270 legacy=0x7fddec038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.273+0000 7fde10bf1640 10 monclient: shutdown 2024-09-16T15:02:00.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.273+0000 7fde10bf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:00.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.273+0000 7fde10bf1640 1 -- 172.21.15.189:0/37163962 >> v1:172.21.15.26:6790/0 conn(0x7fde0c134c60 legacy=0x7fde0c09ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.273+0000 7fde10bf1640 1 -- 172.21.15.189:0/37163962 shutdown_connections 2024-09-16T15:02:00.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.273+0000 7fde10bf1640 1 -- 172.21.15.189:0/37163962 >> 172.21.15.189:0/37163962 conn(0x7fde0c0a7b70 msgr2=0x7fde0c12daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:00.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.273+0000 7fde10bf1640 1 -- 172.21.15.189:0/37163962 shutdown_connections 2024-09-16T15:02:00.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T15:02:00.273+0000 7fde10bf1640 1 -- 172.21.15.189:0/37163962 wait complete. 2024-09-16T15:02:00.284 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph auth health | grep AUTH_BAD_CAP' 1 2024-09-16T15:02:00.284 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph auth health | grep AUTH_BAD_CAP' 2024-09-16T15:02:00.284 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=1 2024-09-16T15:02:00.284 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph auth health '|' grep AUTH_BAD_CAP 2024-09-16T15:02:00.285 INFO:tasks.workunit.client.0.smithi189.stdout:ceph auth health | grep AUTH_BAD_CAP 2024-09-16T15:02:00.285 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph auth health '|' grep AUTH_BAD_CAP 2024-09-16T15:02:00.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.348+0000 7f025106f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T15:02:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.348+0000 7f025106f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T15:02:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.348+0000 7f025106f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.348+0000 7f025106f640 1 -- 172.21.15.26:0/2015475635 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0224005880 con 0x7f02300036d0 2024-09-16T15:02:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.352+0000 7f02437fe640 1 -- 172.21.15.26:0/2015475635 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 234627105) 0x7f023c014330 con 0x7f02300036d0 2024-09-16T15:02:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.352+0000 7f02437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T15:02:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.352+0000 7f02437fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:02:00.352 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":178,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T15:02:00.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.352+0000 7f025106f640 1 -- 172.21.15.26:0/2015475635 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0230037e70 legacy=0x7f023003a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.352+0000 7f025106f640 10 monclient: shutdown 2024-09-16T15:02:00.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.352+0000 7f025106f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:00.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.352+0000 7f025106f640 1 -- 172.21.15.26:0/2015475635 >> v1:172.21.15.26:6789/0 conn(0x7f02300036d0 legacy=0x7f024c1341b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.352+0000 7f025106f640 1 -- 172.21.15.26:0/2015475635 shutdown_connections 2024-09-16T15:02:00.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.352+0000 7f025106f640 1 -- 172.21.15.26:0/2015475635 >> 172.21.15.26:0/2015475635 conn(0x7f024c0a7f30 msgr2=0x7f024c06b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:00.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.352+0000 7f025106f640 1 -- 172.21.15.26:0/2015475635 shutdown_connections 2024-09-16T15:02:00.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.352+0000 7f025106f640 1 -- 172.21.15.26:0/2015475635 wait complete. 2024-09-16T15:02:00.370 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T15:02:00.370 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T15:02:00.371 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T15:02:00.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 10 monclient: get_monmap_and_config 2024-09-16T15:02:00.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 10 monclient: build_initial_monmap 2024-09-16T15:02:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 10 monclient: monmap: 2024-09-16T15:02:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:00.489850+0000 2024-09-16T15:02:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:00.489850+0000 2024-09-16T15:02:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:00.488 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 1 Processor -- start 2024-09-16T15:02:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 1 -- start start 2024-09-16T15:02:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 10 monclient: init 2024-09-16T15:02:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:02:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 10 monclient(hunting): picked mon.noname-a con 0x7f41d809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 10 monclient(hunting): picked mon.noname-c con 0x7f41d8132650 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 10 monclient(hunting): picked mon.noname-b con 0x7f41d8138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d809d260 con 0x7f41d809cb80 2024-09-16T15:02:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d8132a80 con 0x7f41d8132650 2024-09-16T15:02:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d809cf60 con 0x7f41d8138ff0 2024-09-16T15:02:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 10 monclient(hunting): authenticate will time out at 2144.609850s 2024-09-16T15:02:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41dd8f1640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f41d8138ff0 0x7f41d813b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40334/0 (socket says 172.21.15.26:40334) 2024-09-16T15:02:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41dd0f0640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f41d809cb80 0x7f41d80717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43148/0 (socket says 172.21.15.26:43148) 2024-09-16T15:02:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41dd8f1640 1 -- 172.21.15.26:0/3504009123 learned_addr learned my addr 172.21.15.26:0/3504009123 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41dc8ef640 1 -- 172.21.15.26:0/3504009123 >> v1:172.21.15.189:6789/0 conn(0x7f41d8132650 legacy=0x7f41d81378d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:02:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 1 -- 172.21.15.26:0/3504009123 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504414748 0 0) 0x7f41d809d260 con 0x7f41d809cb80 2024-09-16T15:02:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient(hunting): my global_id is 334401 2024-09-16T15:02:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 1 -- 172.21.15.26:0/3504009123 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41c4003530 con 0x7f41d809cb80 2024-09-16T15:02:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 1 -- 172.21.15.26:0/3504009123 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291783464 0 0) 0x7f41d809cf60 con 0x7f41d8138ff0 2024-09-16T15:02:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:00.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:00.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient(hunting): my global_id is 354209 2024-09-16T15:02:00.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 1 -- 172.21.15.26:0/3504009123 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41d809d260 con 0x7f41d8138ff0 2024-09-16T15:02:00.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 1 -- 172.21.15.26:0/3504009123 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1557492642 0 0) 0x7f41c4003530 con 0x7f41d809cb80 2024-09-16T15:02:00.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 1 -- 172.21.15.26:0/3504009123 >> v1:172.21.15.26:6790/0 conn(0x7f41d8138ff0 legacy=0x7f41d813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 1 -- 172.21.15.26:0/3504009123 >> v1:172.21.15.189:6789/0 conn(0x7f41d8132650 legacy=0x7f41d81378d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:02:00.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient: _finish_hunting 0 2024-09-16T15:02:00.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 1 monclient: found mon.noname-a 2024-09-16T15:02:00.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:00.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:02:00.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 1 -- 172.21.15.26:0/3504009123 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d809f2d0 con 0x7f41d809cb80 2024-09-16T15:02:00.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient: _finish_auth 0 2024-09-16T15:02:00.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient: _check_auth_tickets 2024-09-16T15:02:00.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:00.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 5 monclient: authenticate success, global_id 334401 2024-09-16T15:02:00.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:00.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 1 -- 172.21.15.26:0/3504009123 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f41d00019a0 con 0x7f41d809cb80 2024-09-16T15:02:00.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:00.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient: dump: 2024-09-16T15:02:00.499 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.499 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.499 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.499 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.499 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.500 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.500 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.500 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.500 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.500 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:00.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 1 -- 172.21.15.26:0/3504009123 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41d0001c10 con 0x7f41d809cb80 2024-09-16T15:02:00.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:00.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:00.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 1 -- 172.21.15.26:0/3504009123 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f41d0001fe0 con 0x7f41d809cb80 2024-09-16T15:02:00.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 4 set_mon_vals no callback set 2024-09-16T15:02:00.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:00.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41cffff640 10 monclient: dump: 2024-09-16T15:02:00.501 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.501 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.501 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.501 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.501 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.501 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.501 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.501 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.501 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.501 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 10 monclient: shutdown 2024-09-16T15:02:00.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:00.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 1 -- 172.21.15.26:0/3504009123 >> v1:172.21.15.26:6789/0 conn(0x7f41d809cb80 legacy=0x7f41d80717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 1 -- 172.21.15.26:0/3504009123 shutdown_connections 2024-09-16T15:02:00.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 1 -- 172.21.15.26:0/3504009123 >> 172.21.15.26:0/3504009123 conn(0x7f41d809fb50 msgr2=0x7f41d809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:00.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 1 -- 172.21.15.26:0/3504009123 shutdown_connections 2024-09-16T15:02:00.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.488+0000 7f41df354640 1 -- 172.21.15.26:0/3504009123 wait complete. 2024-09-16T15:02:00.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 10 monclient: build_initial_monmap 2024-09-16T15:02:00.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 10 monclient: monmap: 2024-09-16T15:02:00.502 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:00.502 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:00.502 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:00.502 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:00.503 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:00.503 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:00.503 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:00.503 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:00.503 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:00.503 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 1 Processor -- start 2024-09-16T15:02:00.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 1 -- start start 2024-09-16T15:02:00.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 10 monclient: init 2024-09-16T15:02:00.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:00.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:02:00.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 10 monclient(hunting): picked mon.noname-c con 0x7f41d809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:00.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 10 monclient(hunting): picked mon.noname-a con 0x7f41d8132650 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:00.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 10 monclient(hunting): picked mon.noname-b con 0x7f41d8138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:00.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d809b370 con 0x7f41d8132650 2024-09-16T15:02:00.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d8138bf0 con 0x7f41d8138ff0 2024-09-16T15:02:00.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d813c5e0 con 0x7f41d809cb80 2024-09-16T15:02:00.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:00.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 10 monclient(hunting): authenticate will time out at 2144.613027s 2024-09-16T15:02:00.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41dc8ef640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f41d8132650 0x7f41d809ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43154/0 (socket says 172.21.15.26:43154) 2024-09-16T15:02:00.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41dd0f0640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f41d809cb80 0x7f41d809a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40336/0 (socket says 172.21.15.26:40336) 2024-09-16T15:02:00.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41dc8ef640 1 -- 172.21.15.26:0/149279234 learned_addr learned my addr 172.21.15.26:0/149279234 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:00.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41dd8f1640 1 -- 172.21.15.26:0/149279234 >> v1:172.21.15.189:6789/0 conn(0x7f41d8138ff0 legacy=0x7f41d81d1b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:02:00.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415660103 0 0) 0x7f41d813c5e0 con 0x7f41d809cb80 2024-09-16T15:02:00.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): _negotiate 2024-09-16T15:02:00.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:00.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:00.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): my global_id is 354212 2024-09-16T15:02:00.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41bc003560 con 0x7f41d809cb80 2024-09-16T15:02:00.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720937327 0 0) 0x7f41d809b370 con 0x7f41d8132650 2024-09-16T15:02:00.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): _negotiate 2024-09-16T15:02:00.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:00.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:00.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): my global_id is 334404 2024-09-16T15:02:00.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41d813c5e0 con 0x7f41d8132650 2024-09-16T15:02:00.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2780442700 0 0) 0x7f41bc003560 con 0x7f41d809cb80 2024-09-16T15:02:00.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41d809b370 con 0x7f41d809cb80 2024-09-16T15:02:00.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2500162183 0 0) 0x7f41d813c5e0 con 0x7f41d8132650 2024-09-16T15:02:00.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41bc003560 con 0x7f41d8132650 2024-09-16T15:02:00.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f41d0001f20 con 0x7f41d809cb80 2024-09-16T15:02:00.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:02:00.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): dump: 2024-09-16T15:02:00.506 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.506 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.506 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.506 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.506 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.506 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.506 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.506 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.506 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.507 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:00.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f41c8004110 con 0x7f41d8132650 2024-09-16T15:02:00.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:00.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): dump: 2024-09-16T15:02:00.507 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.507 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.507 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.507 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.507 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.507 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.507 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.508 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2377245914 0 0) 0x7f41d809b370 con 0x7f41d809cb80 2024-09-16T15:02:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 >> v1:172.21.15.189:6789/0 conn(0x7f41d8138ff0 legacy=0x7f41d81d1b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:02:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 >> v1:172.21.15.26:6789/0 conn(0x7f41d8132650 legacy=0x7f41d809ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient: _finish_hunting 0 2024-09-16T15:02:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 monclient: found mon.c 2024-09-16T15:02:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d809a240 con 0x7f41d809cb80 2024-09-16T15:02:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient: _finish_auth 0 2024-09-16T15:02:00.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient: _check_auth_tickets 2024-09-16T15:02:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 5 monclient: authenticate success, global_id 354212 2024-09-16T15:02:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 10 monclient: _renew_subs 2024-09-16T15:02:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 1 -- 172.21.15.26:0/149279234 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d806dd50 con 0x7f41d809cb80 2024-09-16T15:02:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 10 monclient: _renew_subs 2024-09-16T15:02:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 1 -- 172.21.15.26:0/149279234 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d81386e0 con 0x7f41d809cb80 2024-09-16T15:02:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41d0004e20 con 0x7f41d809cb80 2024-09-16T15:02:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f41d0005130 con 0x7f41d809cb80 2024-09-16T15:02:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cf7fe640 4 set_mon_vals no callback set 2024-09-16T15:02:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient: dump: 2024-09-16T15:02:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.510 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f41d0005350 con 0x7f41d809cb80 2024-09-16T15:02:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41df354640 1 -- 172.21.15.26:0/149279234 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f41b8006590 con 0x7f41b8002830 2024-09-16T15:02:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f41d0044220 con 0x7f41d809cb80 2024-09-16T15:02:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115588660 0 0) 0x7f41b8006590 con 0x7f41b8002830 2024-09-16T15:02:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): _negotiate 2024-09-16T15:02:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient(hunting): my global_id is 334407 2024-09-16T15:02:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41d809b370 con 0x7f41b8002830 2024-09-16T15:02:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2509140183 0 0) 0x7f41d809b370 con 0x7f41b8002830 2024-09-16T15:02:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41b8006590 con 0x7f41b8002830 2024-09-16T15:02:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41dd8f1640 10 monclient: get_auth_request con 0x7f41bc036240 auth_method 0 2024-09-16T15:02:00.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f41c8004ad0 con 0x7f41b8002830 2024-09-16T15:02:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 10 monclient: dump: 2024-09-16T15:02:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.512 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.513 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2479137268 0 0) 0x7f41b8006590 con 0x7f41b8002830 2024-09-16T15:02:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.492+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f41b800a0e0 con 0x7f41b8002830 2024-09-16T15:02:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.496+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f41c8004ec0 con 0x7f41b8002830 2024-09-16T15:02:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.496+0000 7f41cdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.496+0000 7f41cdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.496+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 >> v1:172.21.15.26:6789/0 conn(0x7f41b8002830 legacy=0x7f41b8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.496+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 >> v1:172.21.15.26:6789/0 conn(0x7f41b8002830 legacy=0x7f41b8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:02:00.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41df354640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:02:00.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41df354640 1 -- 172.21.15.26:0/149279234 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f41b8009d80 con 0x7f41b80090a0 2024-09-16T15:02:00.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603273417 0 0) 0x7f41b8009d80 con 0x7f41b80090a0 2024-09-16T15:02:00.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 10 monclient(hunting): _negotiate 2024-09-16T15:02:00.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:00.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:00.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 10 monclient(hunting): my global_id is 334410 2024-09-16T15:02:00.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41b8006590 con 0x7f41b80090a0 2024-09-16T15:02:00.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 354863049 0 0) 0x7f41b8006590 con 0x7f41b80090a0 2024-09-16T15:02:00.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41b8009d80 con 0x7f41b80090a0 2024-09-16T15:02:00.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f41c80054d0 con 0x7f41b80090a0 2024-09-16T15:02:00.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:00.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 10 monclient: dump: 2024-09-16T15:02:00.515 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.515 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.516 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 178939474 0 0) 0x7f41b8009d80 con 0x7f41b80090a0 2024-09-16T15:02:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f41b800c290 con 0x7f41b80090a0 2024-09-16T15:02:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4201 (unknown 0 0 1890477831) 0x7f41c8006910 con 0x7f41b80090a0 2024-09-16T15:02:00.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:02:00.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:02:00.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 >> v1:172.21.15.26:6789/0 conn(0x7f41b80090a0 legacy=0x7f41b8009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41cdffb640 1 -- 172.21.15.26:0/149279234 >> v1:172.21.15.26:6789/0 conn(0x7f41b80090a0 legacy=0x7f41b8009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:02:00.517 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:02:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:02:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:02:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:02:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 178, 2024-09-16T15:02:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:02:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:02:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:02:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:02:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:02:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:02:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:02:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:02:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:02:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:02:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:02:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:02:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:02:00.519 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:02:00.519 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:02:00.519 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:02:00.519 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:02:00.519 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:00.519 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:02:00.519 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:02:00.519 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:02:00.519 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:02:00.519 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:02:00.519 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:02:00.519 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:02:00.519 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:02:00.519 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:02:00.520 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:02:00.520 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:02:00.520 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:02:00.520 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:00.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.520 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:02:00.520 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:02:00.520 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:02:00.520 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:02:00.520 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:02:00.520 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:02:00.521 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:02:00.521 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:02:00.521 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:02:00.521 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:02:00.521 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:02:00.521 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:02:00.521 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:02:00.521 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:02:00.521 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:02:00.521 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:02:00.521 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:02:00.521 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:02:00.521 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:02:00.522 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:02:00.522 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:02:00.522 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:02:00.522 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:02:00.522 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:02:00.522 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:02:00.522 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:02:00.522 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:02:00.522 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:00.522 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:02:00.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.522 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:02:00.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.522 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:02:00.522 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:02:00.523 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:02:00.523 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:02:00.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.523 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:02:00.523 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:02:00.523 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:02:00.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:00.523 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:00.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.523 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:02:00.523 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:02:00.523 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:02:00.523 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:02:00.523 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:02:00.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.524 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:02:00.524 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:02:00.524 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:02:00.524 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:02:00.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.524 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:02:00.524 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:02:00.524 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:02:00.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:00.524 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:00.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.525 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:02:00.525 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:02:00.525 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:02:00.525 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:02:00.525 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:02:00.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.525 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:02:00.525 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:02:00.525 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:02:00.525 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:02:00.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.525 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:02:00.525 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:02:00.526 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:02:00.526 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:00.526 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:00.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.526 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:02:00.526 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:02:00.526 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:02:00.526 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:02:00.526 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:02:00.526 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:00.526 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:00.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.526 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:02:00.526 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:02:00.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.527 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:02:00.527 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:02:00.527 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:02:00.527 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:00.527 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:00.527 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T15:02:00.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.527 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:02:00.527 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:02:00.527 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T15:02:00.527 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:00.527 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:00.527 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:02:00.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.528 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:02:00.528 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:02:00.528 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T15:02:00.528 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:00.528 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:00.528 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T15:02:00.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.528 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:02:00.528 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:02:00.528 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:02:00.528 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:00.529 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:00.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.529 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:02:00.529 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:02:00.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41df354640 1 -- 172.21.15.26:0/149279234 >> v1:172.21.15.189:6812/3196161582 conn(0x7f41bc036240 legacy=0x7f41bc038700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41df354640 10 monclient: shutdown 2024-09-16T15:02:00.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41df354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:00.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41df354640 1 -- 172.21.15.26:0/149279234 >> v1:172.21.15.26:6790/0 conn(0x7f41d809cb80 legacy=0x7f41d809a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41dd8f1640 1 -- 172.21.15.26:0/149279234 reap_dead start 2024-09-16T15:02:00.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41dd8f1640 1 -- 172.21.15.26:0/149279234 reap_dead start 2024-09-16T15:02:00.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41df354640 1 -- 172.21.15.26:0/149279234 shutdown_connections 2024-09-16T15:02:00.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41df354640 1 -- 172.21.15.26:0/149279234 >> 172.21.15.26:0/149279234 conn(0x7f41d809fb50 msgr2=0x7f41d81314f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:00.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41df354640 1 -- 172.21.15.26:0/149279234 shutdown_connections 2024-09-16T15:02:00.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.500+0000 7f41df354640 1 -- 172.21.15.26:0/149279234 wait complete. 2024-09-16T15:02:00.531 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T15:02:00.531 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T15:02:00.531 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T15:02:00.646 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=1 2024-09-16T15:02:00.646 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 1 -ne 1 ]] 2024-09-16T15:02:00.646 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T15:02:00.646 INFO:tasks.workunit.client.0.smithi189.stdout:OK 2024-09-16T15:02:00.647 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo OK 2024-09-16T15:02:00.647 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T15:02:00.647 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T15:02:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 10 monclient: get_monmap_and_config 2024-09-16T15:02:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 10 monclient: build_initial_monmap 2024-09-16T15:02:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 10 monclient: monmap: 2024-09-16T15:02:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:00.649691+0000 2024-09-16T15:02:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:00.649691+0000 2024-09-16T15:02:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:00.652 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:00.652 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 1 Processor -- start 2024-09-16T15:02:00.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 1 -- start start 2024-09-16T15:02:00.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 10 monclient: init 2024-09-16T15:02:00.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:00.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:02:00.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 10 monclient(hunting): picked mon.noname-b con 0x7ff4600a1c00 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:00.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 10 monclient(hunting): picked mon.noname-a con 0x7ff460099b60 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:00.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 10 monclient(hunting): picked mon.noname-c con 0x7ff4600a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:00.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff460098ea0 con 0x7ff460099b60 2024-09-16T15:02:00.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4600997b0 con 0x7ff4600a26f0 2024-09-16T15:02:00.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff460099460 con 0x7ff4600a1c00 2024-09-16T15:02:00.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:00.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 10 monclient(hunting): authenticate will time out at 2144.769975s 2024-09-16T15:02:00.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff4600a1c00 0x7ff4600a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40342/0 (socket says 172.21.15.26:40342) 2024-09-16T15:02:00.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45ffff640 1 -- 172.21.15.26:0/2442063641 learned_addr learned my addr 172.21.15.26:0/2442063641 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:00.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff464d82640 1 -- 172.21.15.26:0/2442063641 >> v1:172.21.15.189:6789/0 conn(0x7ff4600a26f0 legacy=0x7ff46013d330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:02:00.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 1 -- 172.21.15.26:0/2442063641 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311755201 0 0) 0x7ff460099460 con 0x7ff4600a1c00 2024-09-16T15:02:00.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:00.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:00.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:00.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient(hunting): my global_id is 354215 2024-09-16T15:02:00.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 1 -- 172.21.15.26:0/2442063641 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff44c003530 con 0x7ff4600a1c00 2024-09-16T15:02:00.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 1 -- 172.21.15.26:0/2442063641 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2270332870 0 0) 0x7ff460098ea0 con 0x7ff460099b60 2024-09-16T15:02:00.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:00.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:00.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:00.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient(hunting): my global_id is 334413 2024-09-16T15:02:00.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 1 -- 172.21.15.26:0/2442063641 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff460099460 con 0x7ff460099b60 2024-09-16T15:02:00.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 1 -- 172.21.15.26:0/2442063641 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3883253488 0 0) 0x7ff44c003530 con 0x7ff4600a1c00 2024-09-16T15:02:00.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 1 -- 172.21.15.26:0/2442063641 >> v1:172.21.15.189:6789/0 conn(0x7ff4600a26f0 legacy=0x7ff46013d330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T15:02:00.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 1 -- 172.21.15.26:0/2442063641 >> v1:172.21.15.26:6789/0 conn(0x7ff460099b60 legacy=0x7ff460139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient: _finish_hunting 0 2024-09-16T15:02:00.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 1 monclient: found mon.noname-b 2024-09-16T15:02:00.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:02:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 1 -- 172.21.15.26:0/2442063641 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4600a34b0 con 0x7ff4600a1c00 2024-09-16T15:02:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient: _finish_auth 0 2024-09-16T15:02:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient: _check_auth_tickets 2024-09-16T15:02:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 1 -- 172.21.15.26:0/2442063641 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff4580019a0 con 0x7ff4600a1c00 2024-09-16T15:02:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:02:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient: dump: 2024-09-16T15:02:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.655 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.656 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 5 monclient: authenticate success, global_id 354215 2024-09-16T15:02:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 1 -- 172.21.15.26:0/2442063641 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff458001bc0 con 0x7ff4600a1c00 2024-09-16T15:02:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 1 -- 172.21.15.26:0/2442063641 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff458001f90 con 0x7ff4600a1c00 2024-09-16T15:02:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 4 set_mon_vals no callback set 2024-09-16T15:02:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff45effd640 10 monclient: dump: 2024-09-16T15:02:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.660 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.660 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 10 monclient: shutdown 2024-09-16T15:02:00.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:00.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 1 -- 172.21.15.26:0/2442063641 >> v1:172.21.15.26:6790/0 conn(0x7ff4600a1c00 legacy=0x7ff4600a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 1 -- 172.21.15.26:0/2442063641 shutdown_connections 2024-09-16T15:02:00.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 1 -- 172.21.15.26:0/2442063641 >> 172.21.15.26:0/2442063641 conn(0x7ff4600a3d00 msgr2=0x7ff4600a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:00.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 1 -- 172.21.15.26:0/2442063641 shutdown_connections 2024-09-16T15:02:00.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.648+0000 7ff4667e5640 1 -- 172.21.15.26:0/2442063641 wait complete. 2024-09-16T15:02:00.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 10 monclient: build_initial_monmap 2024-09-16T15:02:00.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 10 monclient: monmap: 2024-09-16T15:02:00.661 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:00.661 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:00.661 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:00.661 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:00.661 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:00.661 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:00.662 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:00.662 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:00.662 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:00.662 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 1 Processor -- start 2024-09-16T15:02:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 1 -- start start 2024-09-16T15:02:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 10 monclient: init 2024-09-16T15:02:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:02:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 10 monclient(hunting): picked mon.noname-c con 0x7ff460099b60 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 10 monclient(hunting): picked mon.noname-a con 0x7ff4600a1c00 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 10 monclient(hunting): picked mon.noname-b con 0x7ff4600a26f0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff46009f070 con 0x7ff4600a1c00 2024-09-16T15:02:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff46013e270 con 0x7ff4600a26f0 2024-09-16T15:02:00.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff46011fb50 con 0x7ff460099b60 2024-09-16T15:02:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 10 monclient(hunting): authenticate will time out at 2144.773100s 2024-09-16T15:02:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff460099b60 0x7ff4601381f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40350/0 (socket says 172.21.15.26:40350) 2024-09-16T15:02:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45ffff640 1 -- 172.21.15.26:0/2124599430 learned_addr learned my addr 172.21.15.26:0/2124599430 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff464d82640 1 -- 172.21.15.26:0/2124599430 >> v1:172.21.15.189:6789/0 conn(0x7ff4600a26f0 legacy=0x7ff4601d1960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T15:02:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625230529 0 0) 0x7ff46011fb50 con 0x7ff460099b60 2024-09-16T15:02:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): _negotiate 2024-09-16T15:02:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): my global_id is 354218 2024-09-16T15:02:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff440003530 con 0x7ff460099b60 2024-09-16T15:02:00.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555523089 0 0) 0x7ff46009f070 con 0x7ff4600a1c00 2024-09-16T15:02:00.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): _negotiate 2024-09-16T15:02:00.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:00.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:00.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): my global_id is 334416 2024-09-16T15:02:00.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff46011fb50 con 0x7ff4600a1c00 2024-09-16T15:02:00.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2852416395 0 0) 0x7ff440003530 con 0x7ff460099b60 2024-09-16T15:02:00.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff46009f070 con 0x7ff460099b60 2024-09-16T15:02:00.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff458001f10 con 0x7ff460099b60 2024-09-16T15:02:00.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:02:00.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): dump: 2024-09-16T15:02:00.670 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.670 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.671 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.671 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.671 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.671 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.671 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.671 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.671 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.671 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:00.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3434619357 0 0) 0x7ff46011fb50 con 0x7ff4600a1c00 2024-09-16T15:02:00.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff440003530 con 0x7ff4600a1c00 2024-09-16T15:02:00.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff4480019e0 con 0x7ff4600a1c00 2024-09-16T15:02:00.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:00.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): dump: 2024-09-16T15:02:00.672 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.672 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.672 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.672 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.672 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.672 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.672 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.672 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.673 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.673 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1370263488 0 0) 0x7ff46009f070 con 0x7ff460099b60 2024-09-16T15:02:00.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 >> v1:172.21.15.189:6789/0 conn(0x7ff4600a26f0 legacy=0x7ff4601d1960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:02:00.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 >> v1:172.21.15.26:6789/0 conn(0x7ff4600a1c00 legacy=0x7ff4601ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient: _finish_hunting 0 2024-09-16T15:02:00.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 monclient: found mon.c 2024-09-16T15:02:00.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:00.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:00.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff460138900 con 0x7ff460099b60 2024-09-16T15:02:00.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient: _finish_auth 0 2024-09-16T15:02:00.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient: _check_auth_tickets 2024-09-16T15:02:00.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:00.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 5 monclient: authenticate success, global_id 354218 2024-09-16T15:02:00.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 10 monclient: _renew_subs 2024-09-16T15:02:00.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:00.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 1 -- 172.21.15.26:0/2124599430 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff46009eb20 con 0x7ff460099b60 2024-09-16T15:02:00.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 10 monclient: _renew_subs 2024-09-16T15:02:00.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:00.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 1 -- 172.21.15.26:0/2124599430 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff46011f870 con 0x7ff460099b60 2024-09-16T15:02:00.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff458004e20 con 0x7ff460099b60 2024-09-16T15:02:00.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:00.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff4580051b0 con 0x7ff460099b60 2024-09-16T15:02:00.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:00.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45e7fc640 4 set_mon_vals no callback set 2024-09-16T15:02:00.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient: dump: 2024-09-16T15:02:00.675 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.675 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.675 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.675 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.675 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.675 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.675 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.675 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.675 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.676 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:00.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff4667e5640 1 -- 172.21.15.26:0/2124599430 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff444006590 con 0x7ff444002830 2024-09-16T15:02:00.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7ff458005390 con 0x7ff460099b60 2024-09-16T15:02:00.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7ff458044270 con 0x7ff460099b60 2024-09-16T15:02:00.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203349708 0 0) 0x7ff444006590 con 0x7ff444002830 2024-09-16T15:02:00.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): _negotiate 2024-09-16T15:02:00.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:00.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:00.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient(hunting): my global_id is 354221 2024-09-16T15:02:00.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff46009f070 con 0x7ff444002830 2024-09-16T15:02:00.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2256733871 0 0) 0x7ff46009f070 con 0x7ff444002830 2024-09-16T15:02:00.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff444006590 con 0x7ff444002830 2024-09-16T15:02:00.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff4480037a0 con 0x7ff444002830 2024-09-16T15:02:00.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:00.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 10 monclient: dump: 2024-09-16T15:02:00.677 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.677 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.677 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.677 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.677 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.677 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.677 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.677 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.678 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.678 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff464d82640 10 monclient: get_auth_request con 0x7ff440036220 auth_method 0 2024-09-16T15:02:00.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2917262067 0 0) 0x7ff444006590 con 0x7ff444002830 2024-09-16T15:02:00.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.652+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff44400a0e0 con 0x7ff444002830 2024-09-16T15:02:00.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.656+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ff4480059e0 con 0x7ff444002830 2024-09-16T15:02:00.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.656+0000 7ff45cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:00.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.656+0000 7ff45cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:00.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.656+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 >> v1:172.21.15.26:6790/0 conn(0x7ff444002830 legacy=0x7ff444004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.656+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 >> v1:172.21.15.26:6790/0 conn(0x7ff444002830 legacy=0x7ff444004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:02:00.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff4667e5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:02:00.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff4667e5640 1 -- 172.21.15.26:0/2124599430 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff444009da0 con 0x7ff4440090c0 2024-09-16T15:02:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920577661 0 0) 0x7ff444009da0 con 0x7ff4440090c0 2024-09-16T15:02:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 10 monclient(hunting): _negotiate 2024-09-16T15:02:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 10 monclient(hunting): my global_id is 354224 2024-09-16T15:02:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff444006590 con 0x7ff4440090c0 2024-09-16T15:02:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 562368978 0 0) 0x7ff444006590 con 0x7ff4440090c0 2024-09-16T15:02:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff444009da0 con 0x7ff4440090c0 2024-09-16T15:02:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7ff448003e10 con 0x7ff4440090c0 2024-09-16T15:02:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 10 monclient: dump: 2024-09-16T15:02:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:00.680 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3296631640 0 0) 0x7ff444009da0 con 0x7ff4440090c0 2024-09-16T15:02:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff44400c290 con 0x7ff4440090c0 2024-09-16T15:02:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3871 (unknown 0 0 416862850) 0x7ff448005b50 con 0x7ff4440090c0 2024-09-16T15:02:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:02:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:02:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 >> v1:172.21.15.26:6790/0 conn(0x7ff4440090c0 legacy=0x7ff444009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff45cff9640 1 -- 172.21.15.26:0/2124599430 >> v1:172.21.15.26:6790/0 conn(0x7ff4440090c0 legacy=0x7ff444009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:02:00.681 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:02:00.681 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:02:00.681 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:02:00.681 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T15:02:00.681 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 178, 2024-09-16T15:02:00.681 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:02:00.681 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:02:00.681 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:02:00.682 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:00.682 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:02:00.682 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:02:00.682 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:02:00.682 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:02:00.682 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:02:00.682 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:02:00.682 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:02:00.682 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:02:00.682 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:02:00.682 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:02:00.682 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:02:00.682 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:02:00.682 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:02:00.683 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:02:00.683 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:00.683 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:02:00.683 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:02:00.683 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:02:00.683 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:02:00.683 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:02:00.683 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:02:00.683 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:02:00.683 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:02:00.683 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:02:00.683 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:02:00.684 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:02:00.684 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:02:00.684 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:00.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.684 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:02:00.684 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:02:00.684 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:02:00.684 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:02:00.684 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:02:00.684 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:02:00.684 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:02:00.684 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:02:00.684 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:02:00.684 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:02:00.685 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:02:00.685 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:02:00.685 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:02:00.685 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:02:00.685 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:02:00.685 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:02:00.685 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:02:00.685 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:02:00.685 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:02:00.685 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:02:00.685 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:02:00.685 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:02:00.685 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:02:00.686 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:02:00.686 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:02:00.686 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:02:00.686 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:02:00.686 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:00.686 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:02:00.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.686 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:02:00.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.686 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:02:00.686 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:02:00.686 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:02:00.687 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:02:00.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:02:00.687 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:02:00.687 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:02:00.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:00.687 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:00.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.687 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:02:00.687 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:02:00.687 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:02:00.687 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:02:00.688 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:02:00.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.688 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:02:00.688 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:02:00.688 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:02:00.688 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:02:00.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.688 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:02:00.688 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:02:00.688 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:02:00.688 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:00.688 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:00.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.689 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:02:00.689 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:02:00.689 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:02:00.689 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:02:00.689 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:02:00.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.689 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:02:00.689 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:02:00.689 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:02:00.689 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:02:00.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.690 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:02:00.690 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:02:00.690 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:02:00.690 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:00.690 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:00.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.690 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:02:00.690 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:02:00.690 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:02:00.690 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:02:00.690 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:02:00.690 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:00.690 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:00.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.691 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:02:00.691 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:02:00.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.691 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:02:00.691 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:02:00.691 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:02:00.691 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:00.691 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:00.691 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:02:00.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:00.691 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:02:00.691 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:02:00.692 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T15:02:00.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:00.692 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:00.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:00.692 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:02:00.692 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:02:00.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff4667e5640 1 -- 172.21.15.26:0/2124599430 >> v1:172.21.15.189:6812/3196161582 conn(0x7ff440036220 legacy=0x7ff4400386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff4667e5640 10 monclient: shutdown 2024-09-16T15:02:00.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff4667e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:00.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff4667e5640 1 -- 172.21.15.26:0/2124599430 >> v1:172.21.15.26:6790/0 conn(0x7ff460099b60 legacy=0x7ff4601381f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:00.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff464d82640 1 -- 172.21.15.26:0/2124599430 reap_dead start 2024-09-16T15:02:00.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff464d82640 1 -- 172.21.15.26:0/2124599430 reap_dead start 2024-09-16T15:02:00.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff4667e5640 1 -- 172.21.15.26:0/2124599430 shutdown_connections 2024-09-16T15:02:00.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff4667e5640 1 -- 172.21.15.26:0/2124599430 >> 172.21.15.26:0/2124599430 conn(0x7ff4600a3d00 msgr2=0x7ff4600a0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:00.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff4667e5640 1 -- 172.21.15.26:0/2124599430 shutdown_connections 2024-09-16T15:02:00.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:00.660+0000 7ff4667e5640 1 -- 172.21.15.26:0/2124599430 wait complete. 2024-09-16T15:02:00.695 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T15:02:00.695 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T15:02:00.707 INFO:tasks.workunit:Stopping ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh'] on client.0... 2024-09-16T15:02:00.707 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T15:02:01.062 DEBUG:teuthology.parallel:result is None 2024-09-16T15:02:01.062 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T15:02:01.074 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T15:02:01.074 DEBUG:teuthology.orchestra.run.smithi189:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T15:02:01.203 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T15:02:01.203 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-16T15:02:01.213 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-16T15:02:20.696 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T15:02:20.696 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T15:02:20.696 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T15:02:20.697 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T15:02:20.700 INFO:tasks.ceph.mon.b:Started 2024-09-16T15:02:20.700 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T15:02:20.700 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T15:02:20.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e82dd9640 10 monclient: get_monmap_and_config 2024-09-16T15:02:20.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e82dd9640 10 monclient: build_initial_monmap 2024-09-16T15:02:20.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e82dd9640 10 monclient: monmap: 2024-09-16T15:02:20.833 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:20.833 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:20.834 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:20.835305+0000 2024-09-16T15:02:20.834 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:20.835305+0000 2024-09-16T15:02:20.834 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:20.834 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:20.834 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:20.834 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:20.834 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:20.834 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:20.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e82dd9640 1 Processor -- start 2024-09-16T15:02:20.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e82dd9640 1 -- start start 2024-09-16T15:02:20.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e82dd9640 10 monclient: init 2024-09-16T15:02:20.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e82dd9640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:20.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e82dd9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:02:20.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e82dd9640 10 monclient(hunting): picked mon.noname-b con 0x7f9e7c0a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:20.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e82dd9640 10 monclient(hunting): picked mon.noname-c con 0x7f9e7c0a0000 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:20.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e82dd9640 10 monclient(hunting): picked mon.noname-a con 0x7f9e7c0a2800 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:20.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e82dd9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c0a3250 con 0x7f9e7c0a2800 2024-09-16T15:02:20.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e82dd9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c09f600 con 0x7f9e7c0a0000 2024-09-16T15:02:20.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e82dd9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c0a3830 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e82dd9640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:20.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e82dd9640 10 monclient(hunting): authenticate will time out at 2164.955375s 2024-09-16T15:02:20.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e80b75640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9e7c0a1d10 0x7f9e7c0a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47768/0 (socket says 172.21.15.26:47768) 2024-09-16T15:02:20.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.831+0000 7f9e80b75640 1 -- 172.21.15.26:0/1146740607 learned_addr learned my addr 172.21.15.26:0/1146740607 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:20.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 1 -- 172.21.15.26:0/1146740607 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240984792 0 0) 0x7f9e7c0a3830 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 10 monclient(hunting): my global_id is 354227 2024-09-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 1 -- 172.21.15.26:0/1146740607 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e64003530 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 1 -- 172.21.15.26:0/1146740607 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3628148176 0 0) 0x7f9e64003530 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 1 -- 172.21.15.26:0/1146740607 >> v1:172.21.15.189:6789/0 conn(0x7f9e7c0a0000 legacy=0x7f9e7c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 1 -- 172.21.15.26:0/1146740607 >> v1:172.21.15.26:6789/0 conn(0x7f9e7c0a2800 legacy=0x7f9e7c13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 10 monclient: _finish_hunting 0 2024-09-16T15:02:20.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 1 monclient: found mon.noname-b 2024-09-16T15:02:20.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:20.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:02:20.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 1 -- 172.21.15.26:0/1146740607 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c09ed60 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 10 monclient: _finish_auth 0 2024-09-16T15:02:20.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 10 monclient: _check_auth_tickets 2024-09-16T15:02:20.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:20.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 1 -- 172.21.15.26:0/1146740607 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e740018d0 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:20.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:02:20.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 10 monclient: dump: 2024-09-16T15:02:20.848 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:20.848 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:20.849 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:20.849 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:20.849 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:20.849 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:20.849 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:20.849 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:20.849 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:20.849 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:20.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 5 monclient: authenticate success, global_id 354227 2024-09-16T15:02:20.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:20.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 1 -- 172.21.15.26:0/1146740607 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e74001bf0 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 1 -- 172.21.15.26:0/1146740607 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e74001f80 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7b7fe640 10 monclient: dump: 2024-09-16T15:02:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:20.850 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:20.851 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:20.851 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:20.851 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:20.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:20.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 4 set_mon_vals no callback set 2024-09-16T15:02:20.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 10 monclient: shutdown 2024-09-16T15:02:20.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:20.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 1 -- 172.21.15.26:0/1146740607 >> v1:172.21.15.26:6790/0 conn(0x7f9e7c0a1d10 legacy=0x7f9e7c0a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:20.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 1 -- 172.21.15.26:0/1146740607 shutdown_connections 2024-09-16T15:02:20.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 1 -- 172.21.15.26:0/1146740607 >> 172.21.15.26:0/1146740607 conn(0x7f9e7c0a7f30 msgr2=0x7f9e7c0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 1 -- 172.21.15.26:0/1146740607 shutdown_connections 2024-09-16T15:02:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 1 -- 172.21.15.26:0/1146740607 wait complete. 2024-09-16T15:02:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 10 monclient: build_initial_monmap 2024-09-16T15:02:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 10 monclient: monmap: 2024-09-16T15:02:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:20.852 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:20.853 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 1 Processor -- start 2024-09-16T15:02:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 1 -- start start 2024-09-16T15:02:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 10 monclient: init 2024-09-16T15:02:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:02:20.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 10 monclient(hunting): picked mon.noname-a con 0x7f9e7c0a0000 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 10 monclient(hunting): picked mon.noname-c con 0x7f9e7c0a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 10 monclient(hunting): picked mon.noname-b con 0x7f9e7c0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c13d120 con 0x7f9e7c0a0000 2024-09-16T15:02:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c09e500 con 0x7f9e7c0a2800 2024-09-16T15:02:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c139f90 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e82dd9640 10 monclient(hunting): authenticate will time out at 2164.963379s 2024-09-16T15:02:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e80b75640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9e7c0a0000 0x7f9e7c139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42042/0 (socket says 172.21.15.26:42042) 2024-09-16T15:02:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e80b75640 1 -- 172.21.15.26:0/532336433 learned_addr learned my addr 172.21.15.26:0/532336433 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:20.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e7bfff640 1 --1- 172.21.15.26:0/532336433 >> v1:172.21.15.26:6790/0 conn(0x7f9e7c0a1d10 0x7f9e7c139880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47778/0 (socket says 172.21.15.26:47778) 2024-09-16T15:02:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e797fa640 1 -- 172.21.15.26:0/532336433 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975007142 0 0) 0x7f9e7c139f90 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e797fa640 10 monclient(hunting): _negotiate 2024-09-16T15:02:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e797fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e797fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e797fa640 10 monclient(hunting): my global_id is 354230 2024-09-16T15:02:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.839+0000 7f9e797fa640 1 -- 172.21.15.26:0/532336433 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e68003530 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 1 -- 172.21.15.26:0/532336433 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3736738229 0 0) 0x7f9e68003530 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 1 -- 172.21.15.26:0/532336433 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e7c139f90 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 1 -- 172.21.15.26:0/532336433 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e70001870 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:02:20.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 10 monclient(hunting): dump: 2024-09-16T15:02:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:20.856 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:20.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:20.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 1 -- 172.21.15.26:0/532336433 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 752100574 0 0) 0x7f9e7c139f90 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 1 -- 172.21.15.26:0/532336433 >> v1:172.21.15.189:6789/0 conn(0x7f9e7c0a2800 legacy=0x7f9e7c1d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:20.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 1 -- 172.21.15.26:0/532336433 >> v1:172.21.15.26:6789/0 conn(0x7f9e7c0a0000 legacy=0x7f9e7c139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:20.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 10 monclient: _finish_hunting 0 2024-09-16T15:02:20.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 1 monclient: found mon.c 2024-09-16T15:02:20.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:20.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:20.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 1 -- 172.21.15.26:0/532336433 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c13bc50 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 10 monclient: _finish_auth 0 2024-09-16T15:02:20.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 10 monclient: _check_auth_tickets 2024-09-16T15:02:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e82dd9640 5 monclient: authenticate success, global_id 354230 2024-09-16T15:02:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e82dd9640 10 monclient: _renew_subs 2024-09-16T15:02:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e82dd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e82dd9640 1 -- 172.21.15.26:0/532336433 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7c0a67d0 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e82dd9640 10 monclient: _renew_subs 2024-09-16T15:02:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e82dd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e82dd9640 1 -- 172.21.15.26:0/532336433 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7c13c880 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 1 -- 172.21.15.26:0/532336433 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e70003600 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:20.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 1 -- 172.21.15.26:0/532336433 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9e70003910 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:20.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:20.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e7affd640 4 set_mon_vals no callback set 2024-09-16T15:02:20.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 10 monclient: dump: 2024-09-16T15:02:20.859 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:20.859 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:20.859 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:20.859 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:20.859 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:20.859 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:20.859 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:20.860 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 1 -- 172.21.15.26:0/532336433 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9e70003af0 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e82dd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e82dd9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e82dd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e82dd9640 1 -- 172.21.15.26:0/532336433 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e5c005050 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e797fa640 1 -- 172.21.15.26:0/532336433 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f9e70043cf0 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.843+0000 7f9e80b75640 10 monclient: get_auth_request con 0x7f9e680360b0 auth_method 0 2024-09-16T15:02:20.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.847+0000 7f9e797fa640 1 -- 172.21.15.26:0/532336433 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9e700143a0 con 0x7f9e7c0a1d10 2024-09-16T15:02:20.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.847+0000 7f9e797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:20.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:20.847+0000 7f9e797fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:21.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.071+0000 7f9e82dd9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T15:02:21.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.071+0000 7f9e82dd9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T15:02:21.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.071+0000 7f9e82dd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:21.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.071+0000 7f9e82dd9640 1 -- 172.21.15.26:0/532336433 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9e5c005880 con 0x7f9e7c0a1d10 2024-09-16T15:02:21.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.071+0000 7f9e797fa640 1 -- 172.21.15.26:0/532336433 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 4262757281) 0x7f9e700141c0 con 0x7f9e7c0a1d10 2024-09-16T15:02:21.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.071+0000 7f9e797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T15:02:21.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.071+0000 7f9e797fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:02:21.075 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":180,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T15:02:21.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.075+0000 7f9e82dd9640 1 -- 172.21.15.26:0/532336433 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9e680360b0 legacy=0x7f9e68038570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.075+0000 7f9e82dd9640 10 monclient: shutdown 2024-09-16T15:02:21.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.075+0000 7f9e82dd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:21.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.075+0000 7f9e82dd9640 1 -- 172.21.15.26:0/532336433 >> v1:172.21.15.26:6790/0 conn(0x7f9e7c0a1d10 legacy=0x7f9e7c139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.075+0000 7f9e82dd9640 1 -- 172.21.15.26:0/532336433 shutdown_connections 2024-09-16T15:02:21.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.075+0000 7f9e82dd9640 1 -- 172.21.15.26:0/532336433 >> 172.21.15.26:0/532336433 conn(0x7f9e7c0a7f30 msgr2=0x7f9e7c09b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:21.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.075+0000 7f9e82dd9640 1 -- 172.21.15.26:0/532336433 shutdown_connections 2024-09-16T15:02:21.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.075+0000 7f9e82dd9640 1 -- 172.21.15.26:0/532336433 wait complete. 2024-09-16T15:02:21.092 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T15:02:21.092 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T15:02:21.092 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T15:02:21.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc4630ca640 10 monclient: get_monmap_and_config 2024-09-16T15:02:21.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc4630ca640 10 monclient: build_initial_monmap 2024-09-16T15:02:21.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc4630ca640 10 monclient: monmap: 2024-09-16T15:02:21.217 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:21.217 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.217 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:21.218733+0000 2024-09-16T15:02:21.217 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:21.218733+0000 2024-09-16T15:02:21.217 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:21.217 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.217 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:21.217 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:21.217 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:21.217 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc4630ca640 1 Processor -- start 2024-09-16T15:02:21.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc4630ca640 1 -- start start 2024-09-16T15:02:21.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc4630ca640 10 monclient: init 2024-09-16T15:02:21.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc4630ca640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:21.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc4630ca640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:02:21.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc4630ca640 10 monclient(hunting): picked mon.noname-c con 0x7fc45c09cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:21.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc4630ca640 10 monclient(hunting): picked mon.noname-a con 0x7fc45c09e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:21.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc4630ca640 10 monclient(hunting): picked mon.noname-b con 0x7fc45c09d1a0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:21.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc4630ca640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c09d5b0 con 0x7fc45c09e7a0 2024-09-16T15:02:21.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc4630ca640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c131a90 con 0x7fc45c09cdc0 2024-09-16T15:02:21.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc4630ca640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c09aff0 con 0x7fc45c09d1a0 2024-09-16T15:02:21.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc4630ca640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:21.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc4630ca640 10 monclient(hunting): authenticate will time out at 2165.339571s 2024-09-16T15:02:21.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc453fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc45c09e7a0 0x7fc45c131380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42058/0 (socket says 172.21.15.26:42058) 2024-09-16T15:02:21.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.215+0000 7fc453fff640 1 -- 172.21.15.26:0/2391028002 learned_addr learned my addr 172.21.15.26:0/2391028002 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:21.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 1 -- 172.21.15.26:0/2391028002 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232153085 0 0) 0x7fc45c09d5b0 con 0x7fc45c09e7a0 2024-09-16T15:02:21.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient(hunting): my global_id is 334419 2024-09-16T15:02:21.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 1 -- 172.21.15.26:0/2391028002 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc444003530 con 0x7fc45c09e7a0 2024-09-16T15:02:21.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 1 -- 172.21.15.26:0/2391028002 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893195961 0 0) 0x7fc45c09aff0 con 0x7fc45c09d1a0 2024-09-16T15:02:21.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient(hunting): my global_id is 354233 2024-09-16T15:02:21.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 1 -- 172.21.15.26:0/2391028002 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc45c09d5b0 con 0x7fc45c09d1a0 2024-09-16T15:02:21.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 1 -- 172.21.15.26:0/2391028002 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2178855812 0 0) 0x7fc444003530 con 0x7fc45c09e7a0 2024-09-16T15:02:21.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 1 -- 172.21.15.26:0/2391028002 >> v1:172.21.15.26:6790/0 conn(0x7fc45c09d1a0 legacy=0x7fc45c139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 1 -- 172.21.15.26:0/2391028002 >> v1:172.21.15.189:6789/0 conn(0x7fc45c09cdc0 legacy=0x7fc45c130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient: _finish_hunting 0 2024-09-16T15:02:21.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 1 monclient: found mon.noname-a 2024-09-16T15:02:21.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:21.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:02:21.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 1 -- 172.21.15.26:0/2391028002 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c09b2d0 con 0x7fc45c09e7a0 2024-09-16T15:02:21.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient: _finish_auth 0 2024-09-16T15:02:21.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient: _check_auth_tickets 2024-09-16T15:02:21.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:21.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 1 -- 172.21.15.26:0/2391028002 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc44c001620 con 0x7fc45c09e7a0 2024-09-16T15:02:21.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:21.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient: dump: 2024-09-16T15:02:21.223 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.223 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.223 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.223 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.223 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.223 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.223 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.223 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.223 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.223 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 5 monclient: authenticate success, global_id 334419 2024-09-16T15:02:21.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:21.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 1 -- 172.21.15.26:0/2391028002 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc44c0017d0 con 0x7fc45c09e7a0 2024-09-16T15:02:21.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:21.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 1 -- 172.21.15.26:0/2391028002 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc44c001ba0 con 0x7fc45c09e7a0 2024-09-16T15:02:21.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:21.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4537fe640 10 monclient: dump: 2024-09-16T15:02:21.224 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.224 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.224 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.224 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.224 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.224 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.224 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.224 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.224 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.224 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:21.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 4 set_mon_vals no callback set 2024-09-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 10 monclient: shutdown 2024-09-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 1 -- 172.21.15.26:0/2391028002 >> v1:172.21.15.26:6789/0 conn(0x7fc45c09e7a0 legacy=0x7fc45c131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 1 -- 172.21.15.26:0/2391028002 shutdown_connections 2024-09-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 1 -- 172.21.15.26:0/2391028002 >> 172.21.15.26:0/2391028002 conn(0x7fc45c09fb50 msgr2=0x7fc45c09ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 1 -- 172.21.15.26:0/2391028002 shutdown_connections 2024-09-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 1 -- 172.21.15.26:0/2391028002 wait complete. 2024-09-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 10 monclient: build_initial_monmap 2024-09-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 10 monclient: monmap: 2024-09-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 1 Processor -- start 2024-09-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 1 -- start start 2024-09-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 10 monclient: init 2024-09-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 10 monclient(hunting): picked mon.noname-b con 0x7fc45c09cdc0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 10 monclient(hunting): picked mon.noname-a con 0x7fc45c09d1a0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 10 monclient(hunting): picked mon.noname-c con 0x7fc45c09e7a0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c11e560 con 0x7fc45c09d1a0 2024-09-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c0a1de0 con 0x7fc45c09cdc0 2024-09-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c09a240 con 0x7fc45c09e7a0 2024-09-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 10 monclient(hunting): authenticate will time out at 2165.342849s 2024-09-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc461667640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc45c09e7a0 0x7fc45c1cd1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47786/0 (socket says 172.21.15.26:47786) 2024-09-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc461667640 1 -- 172.21.15.26:0/1206328734 learned_addr learned my addr 172.21.15.26:0/1206328734 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891567312 0 0) 0x7fc45c09a240 con 0x7fc45c09e7a0 2024-09-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): my global_id is 354236 2024-09-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc448003560 con 0x7fc45c09e7a0 2024-09-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581127645 0 0) 0x7fc45c11e560 con 0x7fc45c09d1a0 2024-09-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): my global_id is 334422 2024-09-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc45c09a240 con 0x7fc45c09d1a0 2024-09-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1502526697 0 0) 0x7fc448003560 con 0x7fc45c09e7a0 2024-09-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc45c11e560 con 0x7fc45c09e7a0 2024-09-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc454001c60 con 0x7fc45c09e7a0 2024-09-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): dump: 2024-09-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1783157321 0 0) 0x7fc45c09a240 con 0x7fc45c09d1a0 2024-09-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc448003560 con 0x7fc45c09d1a0 2024-09-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc44c001440 con 0x7fc45c09d1a0 2024-09-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): dump: 2024-09-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621272289 0 0) 0x7fc45c0a1de0 con 0x7fc45c09cdc0 2024-09-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient(hunting): my global_id is 364102 2024-09-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc45c09a240 con 0x7fc45c09cdc0 2024-09-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1097933301 0 0) 0x7fc45c11e560 con 0x7fc45c09e7a0 2024-09-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 >> v1:172.21.15.189:6789/0 conn(0x7fc45c09cdc0 legacy=0x7fc45c1c4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 >> v1:172.21.15.26:6789/0 conn(0x7fc45c09d1a0 legacy=0x7fc45c12db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient: _finish_hunting 0 2024-09-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 monclient: found mon.c 2024-09-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c0a2310 con 0x7fc45c09e7a0 2024-09-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient: _finish_auth 0 2024-09-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 10 monclient: _check_auth_tickets 2024-09-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 5 monclient: authenticate success, global_id 354236 2024-09-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 10 monclient: _renew_subs 2024-09-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 1 -- 172.21.15.26:0/1206328734 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45c11dcc0 con 0x7fc45c09e7a0 2024-09-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 10 monclient: _renew_subs 2024-09-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.219+0000 7fc4630ca640 1 -- 172.21.15.26:0/1206328734 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45c12e280 con 0x7fc45c09e7a0 2024-09-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc454003980 con 0x7fc45c09e7a0 2024-09-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc454003c90 con 0x7fc45c09e7a0 2024-09-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc452ffd640 4 set_mon_vals no callback set 2024-09-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 10 monclient: dump: 2024-09-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4630ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4630ca640 1 -- 172.21.15.26:0/1206328734 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc43c006590 con 0x7fc43c002830 2024-09-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc454003e70 con 0x7fc45c09e7a0 2024-09-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fc454044090 con 0x7fc45c09e7a0 2024-09-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250253509 0 0) 0x7fc43c006590 con 0x7fc43c002830 2024-09-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 10 monclient(hunting): my global_id is 334425 2024-09-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc45c11e560 con 0x7fc43c002830 2024-09-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc453fff640 10 monclient: get_auth_request con 0x7fc448036610 auth_method 0 2024-09-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1306350753 0 0) 0x7fc45c11e560 con 0x7fc43c002830 2024-09-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc43c006590 con 0x7fc43c002830 2024-09-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc458001720 con 0x7fc43c002830 2024-09-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 10 monclient: dump: 2024-09-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.240 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.240 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.240 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.240 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.240 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.240 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.240 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.240 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2564275031 0 0) 0x7fc43c006590 con 0x7fc43c002830 2024-09-16T15:02:21.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc43c00a0e0 con 0x7fc43c002830 2024-09-16T15:02:21.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc458001a40 con 0x7fc43c002830 2024-09-16T15:02:21.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:21.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:21.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 >> v1:172.21.15.26:6789/0 conn(0x7fc43c002830 legacy=0x7fc43c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.223+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 >> v1:172.21.15.26:6789/0 conn(0x7fc43c002830 legacy=0x7fc43c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:02:21.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.227+0000 7fc4630ca640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:02:21.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.227+0000 7fc4630ca640 1 -- 172.21.15.26:0/1206328734 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc43c009d80 con 0x7fc43c0090a0 2024-09-16T15:02:21.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411361766 0 0) 0x7fc43c009d80 con 0x7fc43c0090a0 2024-09-16T15:02:21.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 10 monclient(hunting): my global_id is 334428 2024-09-16T15:02:21.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc43c006590 con 0x7fc43c0090a0 2024-09-16T15:02:21.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 222592776 0 0) 0x7fc43c006590 con 0x7fc43c0090a0 2024-09-16T15:02:21.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc43c009d80 con 0x7fc43c0090a0 2024-09-16T15:02:21.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc4580037a0 con 0x7fc43c0090a0 2024-09-16T15:02:21.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:21.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 10 monclient: dump: 2024-09-16T15:02:21.243 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.243 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.243 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.243 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.243 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.243 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.243 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.243 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.243 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.244 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3668342835 0 0) 0x7fc43c009d80 con 0x7fc43c0090a0 2024-09-16T15:02:21.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc43c00c290 con 0x7fc43c0090a0 2024-09-16T15:02:21.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 3804957948) 0x7fc458003a40 con 0x7fc43c0090a0 2024-09-16T15:02:21.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:02:21.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:02:21.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 >> v1:172.21.15.26:6789/0 conn(0x7fc43c0090a0 legacy=0x7fc43c009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4517fa640 1 -- 172.21.15.26:0/1206328734 >> v1:172.21.15.26:6789/0 conn(0x7fc43c0090a0 legacy=0x7fc43c009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:02:21.244 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:02:21.244 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:02:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:02:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:02:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 180, 2024-09-16T15:02:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:02:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:02:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:02:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:02:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:02:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:02:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:02:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:02:21.245 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:02:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:02:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:02:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:02:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:02:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:02:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:02:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:02:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:02:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:02:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:02:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:02:21.246 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:02:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:02:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:02:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:02:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:02:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:02:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:02:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:02:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:02:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:02:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:02:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:02:21.247 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:02:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:02:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:02:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:02:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:02:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:02:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:02:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:02:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:02:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:02:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:02:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:02:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:02:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:02:21.248 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:02:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:02:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:02:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:02:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:02:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:02:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:02:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:02:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:02:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:02:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:02:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:02:21.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:02:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:02:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:02:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:02:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:02:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:02:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:02:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:02:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:02:21.250 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:02:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:02:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:02:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:02:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:02:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:02:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:02:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:02:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:02:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:02:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:02:21.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:02:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:02:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:02:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:02:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:02:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:02:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:02:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:02:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:02:21.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:02:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:02:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:02:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:02:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:02:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:02:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:02:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:02:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.253 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:02:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:02:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:02:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:02:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:02:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-09-16T15:02:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:02:21.254 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:02:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-09-16T15:02:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:02:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:02:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:02:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-09-16T15:02:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-09-16T15:02:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:02:21.255 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:02:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:02:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.256 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:02:21.256 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:02:21.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4630ca640 1 -- 172.21.15.26:0/1206328734 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc448036610 legacy=0x7fc448038ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4630ca640 10 monclient: shutdown 2024-09-16T15:02:21.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4630ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:21.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4630ca640 1 -- 172.21.15.26:0/1206328734 >> v1:172.21.15.26:6790/0 conn(0x7fc45c09e7a0 legacy=0x7fc45c1cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc461667640 1 -- 172.21.15.26:0/1206328734 reap_dead start 2024-09-16T15:02:21.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc461667640 1 -- 172.21.15.26:0/1206328734 reap_dead start 2024-09-16T15:02:21.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4630ca640 1 -- 172.21.15.26:0/1206328734 shutdown_connections 2024-09-16T15:02:21.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4630ca640 1 -- 172.21.15.26:0/1206328734 >> 172.21.15.26:0/1206328734 conn(0x7fc45c09fb50 msgr2=0x7fc45c09c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:21.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4630ca640 1 -- 172.21.15.26:0/1206328734 shutdown_connections 2024-09-16T15:02:21.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.231+0000 7fc4630ca640 1 -- 172.21.15.26:0/1206328734 wait complete. 2024-09-16T15:02:21.258 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:02:21.259 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T15:02:21.259 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T15:02:21.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.367+0000 7feb85e16640 10 monclient: get_monmap_and_config 2024-09-16T15:02:21.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.367+0000 7feb85e16640 10 monclient: build_initial_monmap 2024-09-16T15:02:21.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.367+0000 7feb85e16640 10 monclient: monmap: 2024-09-16T15:02:21.371 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:21.371 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.371 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:21.372800+0000 2024-09-16T15:02:21.371 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:21.372800+0000 2024-09-16T15:02:21.371 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:21.371 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.371 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:21.371 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:21.371 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:21.372 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 1 Processor -- start 2024-09-16T15:02:21.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 1 -- start start 2024-09-16T15:02:21.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient: init 2024-09-16T15:02:21.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:21.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:02:21.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient(hunting): picked mon.noname-b con 0x7feb8009a550 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:21.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient(hunting): picked mon.noname-c con 0x7feb8009d1b0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:21.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient(hunting): picked mon.noname-a con 0x7feb8009b040 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:21.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8009dc60 con 0x7feb8009b040 2024-09-16T15:02:21.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb80099ae0 con 0x7feb8009d1b0 2024-09-16T15:02:21.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8009b8f0 con 0x7feb8009a550 2024-09-16T15:02:21.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:21.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient(hunting): authenticate will time out at 2165.492877s 2024-09-16T15:02:21.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7f7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7feb8009a550 0x7feb8009a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47796/0 (socket says 172.21.15.26:47796) 2024-09-16T15:02:21.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7f7fe640 1 -- 172.21.15.26:0/3941468291 learned_addr learned my addr 172.21.15.26:0/3941468291 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:21.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 1 -- 172.21.15.26:0/3941468291 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301738519 0 0) 0x7feb8009b8f0 con 0x7feb8009a550 2024-09-16T15:02:21.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient(hunting): my global_id is 354239 2024-09-16T15:02:21.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 1 -- 172.21.15.26:0/3941468291 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb6c005740 con 0x7feb8009a550 2024-09-16T15:02:21.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 1 -- 172.21.15.26:0/3941468291 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124278998 0 0) 0x7feb8009dc60 con 0x7feb8009b040 2024-09-16T15:02:21.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient(hunting): my global_id is 334431 2024-09-16T15:02:21.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 1 -- 172.21.15.26:0/3941468291 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb8009b8f0 con 0x7feb8009b040 2024-09-16T15:02:21.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 1 -- 172.21.15.26:0/3941468291 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 119184227 0 0) 0x7feb6c005740 con 0x7feb8009a550 2024-09-16T15:02:21.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 1 -- 172.21.15.26:0/3941468291 >> v1:172.21.15.189:6789/0 conn(0x7feb8009d1b0 legacy=0x7feb80136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 1 -- 172.21.15.26:0/3941468291 >> v1:172.21.15.26:6789/0 conn(0x7feb8009b040 legacy=0x7feb8013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient: _finish_hunting 0 2024-09-16T15:02:21.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 1 monclient: found mon.noname-b 2024-09-16T15:02:21.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:21.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:02:21.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 1 -- 172.21.15.26:0/3941468291 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8009bbd0 con 0x7feb8009a550 2024-09-16T15:02:21.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient: _finish_auth 0 2024-09-16T15:02:21.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient: _check_auth_tickets 2024-09-16T15:02:21.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:21.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 1 -- 172.21.15.26:0/3941468291 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb68001620 con 0x7feb8009a550 2024-09-16T15:02:21.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 5 monclient: authenticate success, global_id 354239 2024-09-16T15:02:21.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:21.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:02:21.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient: dump: 2024-09-16T15:02:21.382 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.382 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.382 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.383 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.383 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.383 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.383 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.383 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.383 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.383 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:21.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 1 -- 172.21.15.26:0/3941468291 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb68001840 con 0x7feb8009a550 2024-09-16T15:02:21.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:21.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:21.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 4 set_mon_vals no callback set 2024-09-16T15:02:21.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 1 -- 172.21.15.26:0/3941468291 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb68001bd0 con 0x7feb8009a550 2024-09-16T15:02:21.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:21.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7e7fc640 10 monclient: dump: 2024-09-16T15:02:21.384 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.384 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.384 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.384 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.384 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.384 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.385 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.385 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.385 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.385 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient: shutdown 2024-09-16T15:02:21.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:21.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 1 -- 172.21.15.26:0/3941468291 >> v1:172.21.15.26:6790/0 conn(0x7feb8009a550 legacy=0x7feb8009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 1 -- 172.21.15.26:0/3941468291 shutdown_connections 2024-09-16T15:02:21.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 1 -- 172.21.15.26:0/3941468291 >> 172.21.15.26:0/3941468291 conn(0x7feb8009fb50 msgr2=0x7feb8009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:21.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 1 -- 172.21.15.26:0/3941468291 shutdown_connections 2024-09-16T15:02:21.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 1 -- 172.21.15.26:0/3941468291 wait complete. 2024-09-16T15:02:21.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient: build_initial_monmap 2024-09-16T15:02:21.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient: monmap: 2024-09-16T15:02:21.386 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:21.386 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:21.386 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:21.386 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:21.386 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:21.386 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:21.386 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:21.386 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:21.386 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:21.386 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 1 Processor -- start 2024-09-16T15:02:21.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 1 -- start start 2024-09-16T15:02:21.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient: init 2024-09-16T15:02:21.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:21.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:02:21.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient(hunting): picked mon.noname-a con 0x7feb8009a550 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:21.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient(hunting): picked mon.noname-b con 0x7feb8009b040 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:21.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient(hunting): picked mon.noname-c con 0x7feb8009d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:21.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb801a4a90 con 0x7feb8009a550 2024-09-16T15:02:21.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb800709f0 con 0x7feb8009b040 2024-09-16T15:02:21.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb800a27a0 con 0x7feb8009d1b0 2024-09-16T15:02:21.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:21.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb85e16640 10 monclient(hunting): authenticate will time out at 2165.495828s 2024-09-16T15:02:21.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7f7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7feb8009a550 0x7feb801b5bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42088/0 (socket says 172.21.15.26:42088) 2024-09-16T15:02:21.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.371+0000 7feb7f7fe640 1 -- 172.21.15.26:0/2618766300 learned_addr learned my addr 172.21.15.26:0/2618766300 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:21.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784498489 0 0) 0x7feb800a27a0 con 0x7feb8009d1b0 2024-09-16T15:02:21.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): my global_id is 354242 2024-09-16T15:02:21.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb60003530 con 0x7feb8009d1b0 2024-09-16T15:02:21.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197438997 0 0) 0x7feb801a4a90 con 0x7feb8009a550 2024-09-16T15:02:21.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): my global_id is 334434 2024-09-16T15:02:21.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb800a27a0 con 0x7feb8009a550 2024-09-16T15:02:21.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2583516378 0 0) 0x7feb60003530 con 0x7feb8009d1b0 2024-09-16T15:02:21.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb801a4a90 con 0x7feb8009d1b0 2024-09-16T15:02:21.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb78001c60 con 0x7feb8009d1b0 2024-09-16T15:02:21.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:02:21.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): dump: 2024-09-16T15:02:21.389 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.390 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.390 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.390 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.390 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.390 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.390 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.390 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.390 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.390 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:21.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2200943884 0 0) 0x7feb800a27a0 con 0x7feb8009a550 2024-09-16T15:02:21.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb60003530 con 0x7feb8009a550 2024-09-16T15:02:21.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644492510 0 0) 0x7feb800709f0 con 0x7feb8009b040 2024-09-16T15:02:21.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): my global_id is 364108 2024-09-16T15:02:21.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb800a27a0 con 0x7feb8009b040 2024-09-16T15:02:21.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb68001380 con 0x7feb8009a550 2024-09-16T15:02:21.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:21.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): dump: 2024-09-16T15:02:21.391 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.391 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.391 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.391 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.392 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.392 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.392 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.392 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.392 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.392 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3161331408 0 0) 0x7feb801a4a90 con 0x7feb8009d1b0 2024-09-16T15:02:21.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 >> v1:172.21.15.189:6789/0 conn(0x7feb8009b040 legacy=0x7feb801ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 >> v1:172.21.15.26:6789/0 conn(0x7feb8009a550 legacy=0x7feb801b5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient: _finish_hunting 0 2024-09-16T15:02:21.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 monclient: found mon.c 2024-09-16T15:02:21.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:21.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:21.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8009a240 con 0x7feb8009d1b0 2024-09-16T15:02:21.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient: _finish_auth 0 2024-09-16T15:02:21.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient: _check_auth_tickets 2024-09-16T15:02:21.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:21.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb85e16640 5 monclient: authenticate success, global_id 354242 2024-09-16T15:02:21.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb85e16640 10 monclient: _renew_subs 2024-09-16T15:02:21.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb85e16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:21.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb85e16640 1 -- 172.21.15.26:0/2618766300 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb801b6300 con 0x7feb8009d1b0 2024-09-16T15:02:21.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb85e16640 10 monclient: _renew_subs 2024-09-16T15:02:21.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb85e16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:21.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb85e16640 1 -- 172.21.15.26:0/2618766300 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb801a5290 con 0x7feb8009d1b0 2024-09-16T15:02:21.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb780039a0 con 0x7feb8009d1b0 2024-09-16T15:02:21.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:21.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb78003cb0 con 0x7feb8009d1b0 2024-09-16T15:02:21.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:21.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb7dffb640 4 set_mon_vals no callback set 2024-09-16T15:02:21.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient: dump: 2024-09-16T15:02:21.394 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.394 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.394 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.394 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.394 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.395 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.395 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.395 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.395 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.395 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb85e16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:21.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb85e16640 1 -- 172.21.15.26:0/2618766300 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7feb64006590 con 0x7feb64002830 2024-09-16T15:02:21.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7feb7800e720 con 0x7feb8009d1b0 2024-09-16T15:02:21.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7feb78042df0 con 0x7feb8009d1b0 2024-09-16T15:02:21.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912859297 0 0) 0x7feb64006590 con 0x7feb64002830 2024-09-16T15:02:21.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient(hunting): my global_id is 354245 2024-09-16T15:02:21.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb801a4a90 con 0x7feb64002830 2024-09-16T15:02:21.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4260254219 0 0) 0x7feb801a4a90 con 0x7feb64002830 2024-09-16T15:02:21.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb64006590 con 0x7feb64002830 2024-09-16T15:02:21.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb68004c90 con 0x7feb64002830 2024-09-16T15:02:21.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb7effd640 10 monclient: get_auth_request con 0x7feb600365c0 auth_method 0 2024-09-16T15:02:21.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:21.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient: dump: 2024-09-16T15:02:21.396 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.397 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.397 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.397 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.398 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.398 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.398 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.398 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.399 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.401 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 78039698 0 0) 0x7feb64006590 con 0x7feb64002830 2024-09-16T15:02:21.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb6400a0e0 con 0x7feb64002830 2024-09-16T15:02:21.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7feb68007040 con 0x7feb64002830 2024-09-16T15:02:21.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:21.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:21.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 >> v1:172.21.15.26:6790/0 conn(0x7feb64002830 legacy=0x7feb64004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.375+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 >> v1:172.21.15.26:6790/0 conn(0x7feb64002830 legacy=0x7feb64004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:02:21.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb85e16640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:02:21.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb85e16640 1 -- 172.21.15.26:0/2618766300 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7feb64009d80 con 0x7feb640090a0 2024-09-16T15:02:21.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436945467 0 0) 0x7feb64009d80 con 0x7feb640090a0 2024-09-16T15:02:21.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 10 monclient(hunting): my global_id is 354248 2024-09-16T15:02:21.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb64006590 con 0x7feb640090a0 2024-09-16T15:02:21.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2331808718 0 0) 0x7feb64006590 con 0x7feb640090a0 2024-09-16T15:02:21.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb64009d80 con 0x7feb640090a0 2024-09-16T15:02:21.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb68005520 con 0x7feb640090a0 2024-09-16T15:02:21.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:21.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 10 monclient: dump: 2024-09-16T15:02:21.405 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.405 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.405 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.405 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.405 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.405 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.405 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.405 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.405 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.405 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 848834755 0 0) 0x7feb64009d80 con 0x7feb640090a0 2024-09-16T15:02:21.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7feb6400c290 con 0x7feb640090a0 2024-09-16T15:02:21.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 3870850361) 0x7feb680067d0 con 0x7feb640090a0 2024-09-16T15:02:21.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:02:21.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:02:21.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 >> v1:172.21.15.26:6790/0 conn(0x7feb640090a0 legacy=0x7feb64009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb77fff640 1 -- 172.21.15.26:0/2618766300 >> v1:172.21.15.26:6790/0 conn(0x7feb640090a0 legacy=0x7feb64009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:02:21.406 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:02:21.406 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:02:21.407 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:02:21.407 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-09-16T15:02:21.407 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 180, 2024-09-16T15:02:21.407 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:02:21.407 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:02:21.407 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:02:21.408 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:02:21.409 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:21.409 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:02:21.410 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:02:21.411 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:02:21.411 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:02:21.411 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:02:21.411 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:02:21.411 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:02:21.412 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:02:21.412 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:02:21.412 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:02:21.412 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:02:21.412 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:02:21.412 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:02:21.412 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:02:21.412 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:21.412 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:02:21.412 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:02:21.412 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:02:21.412 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:02:21.413 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:02:21.413 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:02:21.413 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:02:21.413 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:02:21.413 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:02:21.413 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:02:21.413 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:02:21.413 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:02:21.413 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.413 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:02:21.413 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:02:21.413 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:02:21.414 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:02:21.414 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:02:21.414 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:02:21.414 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:02:21.414 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:02:21.414 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:02:21.414 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:02:21.414 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:02:21.414 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:02:21.414 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:02:21.414 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:02:21.414 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:02:21.414 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:02:21.415 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:02:21.415 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:02:21.415 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:02:21.415 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:02:21.415 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:02:21.415 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:02:21.415 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:02:21.415 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:02:21.415 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:02:21.415 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:02:21.415 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:02:21.415 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:21.415 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:02:21.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.416 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:02:21.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.416 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:02:21.416 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:02:21.416 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:02:21.416 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:02:21.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.416 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:02:21.416 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:02:21.416 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:02:21.416 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.416 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.417 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:02:21.417 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:02:21.417 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:02:21.417 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:02:21.417 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:02:21.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.417 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:02:21.417 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:02:21.417 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:02:21.417 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:02:21.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.417 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:02:21.418 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:02:21.418 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:02:21.418 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.418 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.418 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:02:21.418 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:02:21.418 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:02:21.418 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:02:21.418 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:02:21.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.418 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:02:21.418 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:02:21.419 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:02:21.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:02:21.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.419 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:02:21.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:02:21.419 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:02:21.419 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.419 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:02:21.419 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:02:21.419 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:02:21.419 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:02:21.420 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:02:21.420 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.420 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.420 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:02:21.420 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:02:21.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.420 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:02:21.420 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:02:21.420 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:02:21.420 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.420 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:21.420 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:02:21.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.421 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:02:21.421 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:02:21.421 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-09-16T15:02:21.421 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.421 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.421 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:02:21.421 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:02:21.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb85e16640 1 -- 172.21.15.26:0/2618766300 >> v1:172.21.15.189:6812/3196161582 conn(0x7feb600365c0 legacy=0x7feb60038a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb85e16640 10 monclient: shutdown 2024-09-16T15:02:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb85e16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb85e16640 1 -- 172.21.15.26:0/2618766300 >> v1:172.21.15.26:6790/0 conn(0x7feb8009d1b0 legacy=0x7feb801d1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb7ffff640 1 -- 172.21.15.26:0/2618766300 reap_dead start 2024-09-16T15:02:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb7ffff640 1 -- 172.21.15.26:0/2618766300 reap_dead start 2024-09-16T15:02:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb85e16640 1 -- 172.21.15.26:0/2618766300 shutdown_connections 2024-09-16T15:02:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb85e16640 1 -- 172.21.15.26:0/2618766300 >> 172.21.15.26:0/2618766300 conn(0x7feb8009fb50 msgr2=0x7feb801b8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb85e16640 1 -- 172.21.15.26:0/2618766300 shutdown_connections 2024-09-16T15:02:21.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.383+0000 7feb85e16640 1 -- 172.21.15.26:0/2618766300 wait complete. 2024-09-16T15:02:21.423 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:02:21.424 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T15:02:21.424 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T15:02:21.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.531+0000 7f83b68cb640 10 monclient: get_monmap_and_config 2024-09-16T15:02:21.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.531+0000 7f83b68cb640 10 monclient: build_initial_monmap 2024-09-16T15:02:21.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.531+0000 7f83b68cb640 10 monclient: monmap: 2024-09-16T15:02:21.535 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:21.535 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.535 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:21.536836+0000 2024-09-16T15:02:21.535 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:21.536836+0000 2024-09-16T15:02:21.535 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:21.535 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.535 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:21.535 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:21.535 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:21.535 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 1 Processor -- start 2024-09-16T15:02:21.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 1 -- start start 2024-09-16T15:02:21.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient: init 2024-09-16T15:02:21.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:21.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:02:21.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient(hunting): picked mon.noname-b con 0x7f83b009cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:21.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient(hunting): picked mon.noname-a con 0x7f83b009e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:21.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient(hunting): picked mon.noname-c con 0x7f83b009d1a0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:21.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b009d5b0 con 0x7f83b009e7a0 2024-09-16T15:02:21.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b0131a90 con 0x7f83b009d1a0 2024-09-16T15:02:21.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b009aff0 con 0x7f83b009cdc0 2024-09-16T15:02:21.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:21.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient(hunting): authenticate will time out at 2165.656840s 2024-09-16T15:02:21.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83affff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f83b009cdc0 0x7f83b0130c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47834/0 (socket says 172.21.15.26:47834) 2024-09-16T15:02:21.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83affff640 1 -- 172.21.15.26:0/226578052 learned_addr learned my addr 172.21.15.26:0/226578052 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:21.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83af7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f83b009e7a0 0x7f83b0131380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42096/0 (socket says 172.21.15.26:42096) 2024-09-16T15:02:21.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 1 -- 172.21.15.26:0/226578052 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349379356 0 0) 0x7f83b009d5b0 con 0x7f83b009e7a0 2024-09-16T15:02:21.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient(hunting): my global_id is 334437 2024-09-16T15:02:21.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 1 -- 172.21.15.26:0/226578052 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8398003530 con 0x7f83b009e7a0 2024-09-16T15:02:21.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 1 -- 172.21.15.26:0/226578052 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246336999 0 0) 0x7f83b009aff0 con 0x7f83b009cdc0 2024-09-16T15:02:21.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient(hunting): my global_id is 354251 2024-09-16T15:02:21.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 1 -- 172.21.15.26:0/226578052 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b009d5b0 con 0x7f83b009cdc0 2024-09-16T15:02:21.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 1 -- 172.21.15.26:0/226578052 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1303068755 0 0) 0x7f8398003530 con 0x7f83b009e7a0 2024-09-16T15:02:21.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 1 -- 172.21.15.26:0/226578052 >> v1:172.21.15.26:6790/0 conn(0x7f83b009cdc0 legacy=0x7f83b0130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 1 -- 172.21.15.26:0/226578052 >> v1:172.21.15.189:6789/0 conn(0x7f83b009d1a0 legacy=0x7f83b0139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient: _finish_hunting 0 2024-09-16T15:02:21.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 1 monclient: found mon.noname-a 2024-09-16T15:02:21.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:21.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:02:21.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 1 -- 172.21.15.26:0/226578052 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b009b2d0 con 0x7f83b009e7a0 2024-09-16T15:02:21.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient: _finish_auth 0 2024-09-16T15:02:21.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient: _check_auth_tickets 2024-09-16T15:02:21.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:21.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 5 monclient: authenticate success, global_id 334437 2024-09-16T15:02:21.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:21.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 1 -- 172.21.15.26:0/226578052 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f83a0001620 con 0x7f83b009e7a0 2024-09-16T15:02:21.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:21.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient: dump: 2024-09-16T15:02:21.547 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.548 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.548 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.548 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.548 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.548 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.548 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.548 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.548 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.548 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:21.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 1 -- 172.21.15.26:0/226578052 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83a0001850 con 0x7f83b009e7a0 2024-09-16T15:02:21.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:21.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:21.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 4 set_mon_vals no callback set 2024-09-16T15:02:21.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 1 -- 172.21.15.26:0/226578052 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f83a0001be0 con 0x7f83b009e7a0 2024-09-16T15:02:21.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:21.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83aeffd640 10 monclient: dump: 2024-09-16T15:02:21.549 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.549 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.549 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.549 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.549 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.549 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.549 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.549 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.549 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.549 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient: shutdown 2024-09-16T15:02:21.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:21.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 1 -- 172.21.15.26:0/226578052 >> v1:172.21.15.26:6789/0 conn(0x7f83b009e7a0 legacy=0x7f83b0131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 1 -- 172.21.15.26:0/226578052 shutdown_connections 2024-09-16T15:02:21.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 1 -- 172.21.15.26:0/226578052 >> 172.21.15.26:0/226578052 conn(0x7f83b009fb50 msgr2=0x7f83b009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:21.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 1 -- 172.21.15.26:0/226578052 shutdown_connections 2024-09-16T15:02:21.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 1 -- 172.21.15.26:0/226578052 wait complete. 2024-09-16T15:02:21.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient: build_initial_monmap 2024-09-16T15:02:21.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient: monmap: 2024-09-16T15:02:21.550 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:21.550 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:21.550 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:21.550 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:21.550 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:21.551 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:21.551 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:21.551 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:21.551 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:21.551 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 1 Processor -- start 2024-09-16T15:02:21.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 1 -- start start 2024-09-16T15:02:21.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient: init 2024-09-16T15:02:21.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:21.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:02:21.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient(hunting): picked mon.noname-c con 0x7f83b009cdc0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:21.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient(hunting): picked mon.noname-b con 0x7f83b009d1a0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:21.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient(hunting): picked mon.noname-a con 0x7f83b009e7a0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:21.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b00a2500 con 0x7f83b009e7a0 2024-09-16T15:02:21.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b012d7c0 con 0x7f83b009d1a0 2024-09-16T15:02:21.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b009a240 con 0x7f83b009cdc0 2024-09-16T15:02:21.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:21.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83b68cb640 10 monclient(hunting): authenticate will time out at 2165.659910s 2024-09-16T15:02:21.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83affff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f83b009cdc0 0x7f83b012cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47838/0 (socket says 172.21.15.26:47838) 2024-09-16T15:02:21.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.535+0000 7f83affff640 1 -- 172.21.15.26:0/966692050 learned_addr learned my addr 172.21.15.26:0/966692050 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:21.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96101670 0 0) 0x7f83b0131a90 con 0x7f83b009e7a0 2024-09-16T15:02:21.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): my global_id is 334440 2024-09-16T15:02:21.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f839c003530 con 0x7f83b009e7a0 2024-09-16T15:02:21.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727780560 0 0) 0x7f83b009a240 con 0x7f83b009cdc0 2024-09-16T15:02:21.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): my global_id is 354254 2024-09-16T15:02:21.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b0131a90 con 0x7f83b009cdc0 2024-09-16T15:02:21.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3254526922 0 0) 0x7f839c003530 con 0x7f83b009e7a0 2024-09-16T15:02:21.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83b009a240 con 0x7f83b009e7a0 2024-09-16T15:02:21.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f83a80019e0 con 0x7f83b009e7a0 2024-09-16T15:02:21.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:21.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): dump: 2024-09-16T15:02:21.553 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.554 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.554 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.554 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.554 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.554 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.554 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.554 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.554 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.554 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:21.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3986595608 0 0) 0x7f83b0131a90 con 0x7f83b009cdc0 2024-09-16T15:02:21.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f839c003530 con 0x7f83b009cdc0 2024-09-16T15:02:21.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f83a40018e0 con 0x7f83b009cdc0 2024-09-16T15:02:21.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:21.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): dump: 2024-09-16T15:02:21.555 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.555 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.555 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.555 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.555 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.555 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.555 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.555 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.555 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.555 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639029296 0 0) 0x7f83b012d7c0 con 0x7f83b009d1a0 2024-09-16T15:02:21.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): my global_id is 364114 2024-09-16T15:02:21.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b0131a90 con 0x7f83b009d1a0 2024-09-16T15:02:21.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2751387818 0 0) 0x7f83b009a240 con 0x7f83b009e7a0 2024-09-16T15:02:21.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 >> v1:172.21.15.26:6790/0 conn(0x7f83b009cdc0 legacy=0x7f83b012cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 >> v1:172.21.15.189:6789/0 conn(0x7f83b009d1a0 legacy=0x7f83b01ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient: _finish_hunting 0 2024-09-16T15:02:21.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 monclient: found mon.a 2024-09-16T15:02:21.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:21.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:21.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b011a5f0 con 0x7f83b009e7a0 2024-09-16T15:02:21.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient: _finish_auth 0 2024-09-16T15:02:21.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient: _check_auth_tickets 2024-09-16T15:02:21.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:21.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83b68cb640 5 monclient: authenticate success, global_id 334440 2024-09-16T15:02:21.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83b68cb640 10 monclient: _renew_subs 2024-09-16T15:02:21.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83b68cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:21.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83b68cb640 1 -- 172.21.15.26:0/966692050 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b0119d50 con 0x7f83b009e7a0 2024-09-16T15:02:21.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83b68cb640 10 monclient: _renew_subs 2024-09-16T15:02:21.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83b68cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:21.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83b68cb640 1 -- 172.21.15.26:0/966692050 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b00702e0 con 0x7f83b009e7a0 2024-09-16T15:02:21.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83a8002060 con 0x7f83b009e7a0 2024-09-16T15:02:21.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:21.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f83a8003b20 con 0x7f83b009e7a0 2024-09-16T15:02:21.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83ae7fc640 4 set_mon_vals no callback set 2024-09-16T15:02:21.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:21.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient: dump: 2024-09-16T15:02:21.558 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.558 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.558 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.558 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.558 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.558 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.558 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.558 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.558 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.558 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83b68cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:21.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83b68cb640 1 -- 172.21.15.26:0/966692050 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8390006590 con 0x7f8390002830 2024-09-16T15:02:21.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f83a8003d40 con 0x7f83b009e7a0 2024-09-16T15:02:21.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f83a8042cc0 con 0x7f83b009e7a0 2024-09-16T15:02:21.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83af7fe640 10 monclient: get_auth_request con 0x7f839c0363e0 auth_method 0 2024-09-16T15:02:21.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571283830 0 0) 0x7f8390006590 con 0x7f8390002830 2024-09-16T15:02:21.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient(hunting): my global_id is 364117 2024-09-16T15:02:21.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b009a240 con 0x7f8390002830 2024-09-16T15:02:21.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1297473878 0 0) 0x7f83b009a240 con 0x7f8390002830 2024-09-16T15:02:21.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8390006590 con 0x7f8390002830 2024-09-16T15:02:21.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f83a4004990 con 0x7f8390002830 2024-09-16T15:02:21.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:02:21.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 10 monclient: dump: 2024-09-16T15:02:21.560 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.560 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.560 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.560 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.560 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.560 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.560 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.560 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.560 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.560 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3530315523 0 0) 0x7f8390006590 con 0x7f8390002830 2024-09-16T15:02:21.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.539+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f839000a0e0 con 0x7f8390002830 2024-09-16T15:02:21.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.543+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f83a40058e0 con 0x7f8390002830 2024-09-16T15:02:21.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.543+0000 7f83acff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:21.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.543+0000 7f83acff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:21.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.543+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 >> v1:172.21.15.189:6789/0 conn(0x7f8390002830 legacy=0x7f8390004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.543+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 >> v1:172.21.15.189:6789/0 conn(0x7f8390002830 legacy=0x7f8390004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:02:21.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.547+0000 7f83b68cb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T15:02:21.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.547+0000 7f83b68cb640 1 -- 172.21.15.26:0/966692050 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8390009da0 con 0x7f83900090c0 2024-09-16T15:02:21.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.547+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319069979 0 0) 0x7f8390009da0 con 0x7f83900090c0 2024-09-16T15:02:21.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.547+0000 7f83acff9640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.547+0000 7f83acff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.547+0000 7f83acff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.547+0000 7f83acff9640 10 monclient(hunting): my global_id is 364120 2024-09-16T15:02:21.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.547+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8390006590 con 0x7f83900090c0 2024-09-16T15:02:21.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.547+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1809263421 0 0) 0x7f8390006590 con 0x7f83900090c0 2024-09-16T15:02:21.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.547+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8390009da0 con 0x7f83900090c0 2024-09-16T15:02:21.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.547+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f83a4005050 con 0x7f83900090c0 2024-09-16T15:02:21.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.547+0000 7f83acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.547+0000 7f83acff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T15:02:21.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.547+0000 7f83acff9640 10 monclient: dump: 2024-09-16T15:02:21.563 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.563 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.563 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.563 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.563 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.563 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.563 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.563 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.563 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.564 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.547+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2871304570 0 0) 0x7f8390009da0 con 0x7f83900090c0 2024-09-16T15:02:21.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.547+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f839000c290 con 0x7f83900090c0 2024-09-16T15:02:21.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.551+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 2534611735) 0x7f83a4005a50 con 0x7f83900090c0 2024-09-16T15:02:21.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.551+0000 7f83acff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T15:02:21.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.551+0000 7f83acff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:02:21.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.551+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 >> v1:172.21.15.189:6789/0 conn(0x7f83900090c0 legacy=0x7f8390009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.551+0000 7f83acff9640 1 -- 172.21.15.26:0/966692050 >> v1:172.21.15.189:6789/0 conn(0x7f83900090c0 legacy=0x7f8390009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T15:02:21.564 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:02:21.564 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:02:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:02:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-09-16T15:02:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 180, 2024-09-16T15:02:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-09-16T15:02:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-09-16T15:02:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-09-16T15:02:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-09-16T15:02:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-09-16T15:02:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:02:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2449958758054445060", 2024-09-16T15:02:21.565 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-09-16T15:02:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:02:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:02:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:02:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:02:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:02:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:02:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:02:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:02:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:02:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:02:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540138322906710015", 2024-09-16T15:02:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-09-16T15:02:21.566 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:02:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:02:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:02:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:02:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:02:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:02:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:02:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:02:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:02:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:02:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-09-16T15:02:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-09-16T15:02:21.567 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-09-16T15:02:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-09-16T15:02:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-09-16T15:02:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "febeb2df-62e5-4150-ac3f-7a627f391606", 2024-09-16T15:02:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:02:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-09-16T14:41:47.153956Z", 2024-09-16T15:02:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 18, 2024-09-16T15:02:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "reef", 2024-09-16T15:02:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 3, 2024-09-16T15:02:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-09-16T15:02:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-09-16T15:02:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-09-16T15:02:21.568 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-09-16T15:02:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-09-16T15:02:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-09-16T15:02:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-09-16T15:02:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-09-16T15:02:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-09-16T15:02:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-09-16T15:02:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-09-16T15:02:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-09-16T15:02:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-09-16T15:02:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-09-16T15:02:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-09-16T15:02:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "reef" 2024-09-16T15:02:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:21.569 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-09-16T15:02:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-09-16T15:02:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-09-16T15:02:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-09-16T15:02:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:02:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:02:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:02:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-09-16T15:02:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:02:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-09-16T15:02:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-09-16T15:02:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:02:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:02:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:02:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-09-16T15:02:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-09-16T15:02:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:02:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:02:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.571 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:02:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789", 2024-09-16T15:02:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:02:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T15:02:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T15:02:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:02:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:02:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:02:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.572 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-09-16T15:02:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-09-16T15:02:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-09-16T15:02:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-09-16T15:02:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-09-16T15:02:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-09-16T15:02:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-09-16T15:02:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-09-16T15:02:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-09-16T15:02:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-09-16T15:02:21.573 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-09-16T15:02:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-09-16T15:02:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-09-16T15:02:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-09-16T15:02:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:02:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:02:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:02:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-09-16T15:02:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-09-16T15:02:21.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-09-16T15:02:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T15:02:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "reef", 2024-09-16T15:02:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-09-16T15:02:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-09-16T15:02:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-09-16T15:02:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-09-16T15:02:21.575 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-09-16T15:02:21.575 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:02:21.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.551+0000 7f83b68cb640 1 -- 172.21.15.26:0/966692050 >> v1:172.21.15.189:6812/3196161582 conn(0x7f839c0363e0 legacy=0x7f839c0388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.551+0000 7f83b68cb640 10 monclient: shutdown 2024-09-16T15:02:21.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.551+0000 7f83b68cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:21.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.551+0000 7f83b68cb640 1 -- 172.21.15.26:0/966692050 >> v1:172.21.15.26:6789/0 conn(0x7f83b009e7a0 legacy=0x7f83b01cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.551+0000 7f83b4e68640 1 -- 172.21.15.26:0/966692050 reap_dead start 2024-09-16T15:02:21.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.551+0000 7f83b4e68640 1 -- 172.21.15.26:0/966692050 reap_dead start 2024-09-16T15:02:21.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.551+0000 7f83b68cb640 1 -- 172.21.15.26:0/966692050 shutdown_connections 2024-09-16T15:02:21.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.551+0000 7f83b68cb640 1 -- 172.21.15.26:0/966692050 >> 172.21.15.26:0/966692050 conn(0x7f83b009fb50 msgr2=0x7f83b009bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:21.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.551+0000 7f83b68cb640 1 -- 172.21.15.26:0/966692050 shutdown_connections 2024-09-16T15:02:21.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.551+0000 7f83b68cb640 1 -- 172.21.15.26:0/966692050 wait complete. 2024-09-16T15:02:21.577 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T15:02:21.578 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T15:02:21.578 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T15:02:21.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.687+0000 7f3ef2506640 10 monclient: get_monmap_and_config 2024-09-16T15:02:21.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.687+0000 7f3ef2506640 10 monclient: build_initial_monmap 2024-09-16T15:02:21.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.687+0000 7f3ef2506640 10 monclient: monmap: 2024-09-16T15:02:21.690 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:21.691 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.691 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:21.692715+0000 2024-09-16T15:02:21.691 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:21.692715+0000 2024-09-16T15:02:21.691 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:21.691 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.691 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:21.691 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:21.691 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:21.691 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.687+0000 7f3ef2506640 1 Processor -- start 2024-09-16T15:02:21.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 1 -- start start 2024-09-16T15:02:21.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient: init 2024-09-16T15:02:21.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:21.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:02:21.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient(hunting): picked mon.noname-b con 0x7f3eec0a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:21.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient(hunting): picked mon.noname-a con 0x7f3eec0a0000 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:21.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient(hunting): picked mon.noname-c con 0x7f3eec0a2800 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:21.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eec0a3250 con 0x7f3eec0a0000 2024-09-16T15:02:21.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eec09f600 con 0x7f3eec0a2800 2024-09-16T15:02:21.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eec0a3830 con 0x7f3eec0a1d10 2024-09-16T15:02:21.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:21.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient(hunting): authenticate will time out at 2165.812740s 2024-09-16T15:02:21.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeb7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3eec0a0000 0x7f3eec139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42120/0 (socket says 172.21.15.26:42120) 2024-09-16T15:02:21.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeb7fe640 1 -- 172.21.15.26:0/2509696940 learned_addr learned my addr 172.21.15.26:0/2509696940 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:21.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eebfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3eec0a1d10 0x7f3eec0a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47842/0 (socket says 172.21.15.26:47842) 2024-09-16T15:02:21.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 1 -- 172.21.15.26:0/2509696940 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784830440 0 0) 0x7f3eec0a3830 con 0x7f3eec0a1d10 2024-09-16T15:02:21.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient(hunting): my global_id is 354257 2024-09-16T15:02:21.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 1 -- 172.21.15.26:0/2509696940 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed4003530 con 0x7f3eec0a1d10 2024-09-16T15:02:21.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 1 -- 172.21.15.26:0/2509696940 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566283739 0 0) 0x7f3eec0a3250 con 0x7f3eec0a0000 2024-09-16T15:02:21.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient(hunting): my global_id is 334443 2024-09-16T15:02:21.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 1 -- 172.21.15.26:0/2509696940 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eec0a3830 con 0x7f3eec0a0000 2024-09-16T15:02:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 1 -- 172.21.15.26:0/2509696940 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1268202789 0 0) 0x7f3ed4003530 con 0x7f3eec0a1d10 2024-09-16T15:02:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 1 -- 172.21.15.26:0/2509696940 >> v1:172.21.15.189:6789/0 conn(0x7f3eec0a2800 legacy=0x7f3eec13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 1 -- 172.21.15.26:0/2509696940 >> v1:172.21.15.26:6789/0 conn(0x7f3eec0a0000 legacy=0x7f3eec139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient: _finish_hunting 0 2024-09-16T15:02:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 1 monclient: found mon.noname-b 2024-09-16T15:02:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:02:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 1 -- 172.21.15.26:0/2509696940 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eec09ed60 con 0x7f3eec0a1d10 2024-09-16T15:02:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient: _finish_auth 0 2024-09-16T15:02:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient: _check_auth_tickets 2024-09-16T15:02:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 1 -- 172.21.15.26:0/2509696940 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3edc001620 con 0x7f3eec0a1d10 2024-09-16T15:02:21.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 5 monclient: authenticate success, global_id 354257 2024-09-16T15:02:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:02:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient: dump: 2024-09-16T15:02:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.701 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.702 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 1 -- 172.21.15.26:0/2509696940 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3edc001840 con 0x7f3eec0a1d10 2024-09-16T15:02:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 1 -- 172.21.15.26:0/2509696940 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3edc001bd0 con 0x7f3eec0a1d10 2024-09-16T15:02:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 4 set_mon_vals no callback set 2024-09-16T15:02:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:21.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeaffd640 10 monclient: dump: 2024-09-16T15:02:21.703 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.703 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.703 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.703 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.703 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.703 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.703 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.703 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.703 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.703 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient: shutdown 2024-09-16T15:02:21.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:21.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 1 -- 172.21.15.26:0/2509696940 >> v1:172.21.15.26:6790/0 conn(0x7f3eec0a1d10 legacy=0x7f3eec0a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 1 -- 172.21.15.26:0/2509696940 shutdown_connections 2024-09-16T15:02:21.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 1 -- 172.21.15.26:0/2509696940 >> 172.21.15.26:0/2509696940 conn(0x7f3eec0a7f30 msgr2=0x7f3eec0a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:21.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 1 -- 172.21.15.26:0/2509696940 shutdown_connections 2024-09-16T15:02:21.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 1 -- 172.21.15.26:0/2509696940 wait complete. 2024-09-16T15:02:21.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient: build_initial_monmap 2024-09-16T15:02:21.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient: monmap: 2024-09-16T15:02:21.704 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:21.704 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:21.704 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:21.704 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:21.704 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:21.704 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:21.705 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:21.705 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:21.705 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:21.705 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 1 Processor -- start 2024-09-16T15:02:21.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 1 -- start start 2024-09-16T15:02:21.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient: init 2024-09-16T15:02:21.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:21.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:02:21.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient(hunting): picked mon.noname-b con 0x7f3eec0a0000 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:21.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient(hunting): picked mon.noname-c con 0x7f3eec0a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:21.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient(hunting): picked mon.noname-a con 0x7f3eec0a2800 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:21.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eec134810 con 0x7f3eec0a2800 2024-09-16T15:02:21.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eec1358d0 con 0x7f3eec0a0000 2024-09-16T15:02:21.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eec0a6590 con 0x7f3eec0a1d10 2024-09-16T15:02:21.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:21.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3ef2506640 10 monclient(hunting): authenticate will time out at 2165.815731s 2024-09-16T15:02:21.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeb7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3eec0a1d10 0x7f3eec132c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47844/0 (socket says 172.21.15.26:47844) 2024-09-16T15:02:21.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.691+0000 7f3eeb7fe640 1 -- 172.21.15.26:0/3112516275 learned_addr learned my addr 172.21.15.26:0/3112516275 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:21.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424136691 0 0) 0x7f3eec134810 con 0x7f3eec0a2800 2024-09-16T15:02:21.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): my global_id is 334446 2024-09-16T15:02:21.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed8003530 con 0x7f3eec0a2800 2024-09-16T15:02:21.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348828025 0 0) 0x7f3eec0a6590 con 0x7f3eec0a1d10 2024-09-16T15:02:21.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): my global_id is 354260 2024-09-16T15:02:21.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eec134810 con 0x7f3eec0a1d10 2024-09-16T15:02:21.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2895772218 0 0) 0x7f3ed8003530 con 0x7f3eec0a2800 2024-09-16T15:02:21.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3eec0a6590 con 0x7f3eec0a2800 2024-09-16T15:02:21.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1189039270 0 0) 0x7f3eec134810 con 0x7f3eec0a1d10 2024-09-16T15:02:21.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ed8003530 con 0x7f3eec0a1d10 2024-09-16T15:02:21.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ee4001b80 con 0x7f3eec0a2800 2024-09-16T15:02:21.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:21.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): dump: 2024-09-16T15:02:21.707 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.708 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.708 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.708 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.708 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.708 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.708 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.708 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.708 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.708 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:21.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65447387 0 0) 0x7f3eec1358d0 con 0x7f3eec0a0000 2024-09-16T15:02:21.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:02:21.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:21.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:21.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): my global_id is 364126 2024-09-16T15:02:21.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eec134810 con 0x7f3eec0a0000 2024-09-16T15:02:21.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ee0001980 con 0x7f3eec0a1d10 2024-09-16T15:02:21.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:21.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient(hunting): dump: 2024-09-16T15:02:21.709 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.709 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.709 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.709 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.709 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.710 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.710 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.710 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.710 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.710 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3647679383 0 0) 0x7f3eec0a6590 con 0x7f3eec0a2800 2024-09-16T15:02:21.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 >> v1:172.21.15.26:6790/0 conn(0x7f3eec0a1d10 legacy=0x7f3eec132c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 >> v1:172.21.15.189:6789/0 conn(0x7f3eec0a0000 legacy=0x7f3eec1324f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient: _finish_hunting 0 2024-09-16T15:02:21.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 monclient: found mon.a 2024-09-16T15:02:21.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:21.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:21.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eec1363e0 con 0x7f3eec0a2800 2024-09-16T15:02:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient: _finish_auth 0 2024-09-16T15:02:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient: _check_auth_tickets 2024-09-16T15:02:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ef2506640 5 monclient: authenticate success, global_id 334446 2024-09-16T15:02:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ef2506640 10 monclient: _renew_subs 2024-09-16T15:02:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ef2506640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ef2506640 1 -- 172.21.15.26:0/3112516275 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eec133e10 con 0x7f3eec0a2800 2024-09-16T15:02:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ef2506640 10 monclient: _renew_subs 2024-09-16T15:02:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ef2506640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ef2506640 1 -- 172.21.15.26:0/3112516275 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eec133310 con 0x7f3eec0a2800 2024-09-16T15:02:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ee4001d70 con 0x7f3eec0a2800 2024-09-16T15:02:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f3ee4003b20 con 0x7f3eec0a2800 2024-09-16T15:02:21.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:21.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:21.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3eea7fc640 4 set_mon_vals no callback set 2024-09-16T15:02:21.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 10 monclient: dump: 2024-09-16T15:02:21.712 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:21.712 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:21.712 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.712 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:21.712 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:21.712 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:21.712 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:21.712 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:21.712 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:21.712 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:21.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ef2506640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:21.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ef2506640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:21.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ef2506640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:21.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f3ee4003c80 con 0x7f3eec0a2800 2024-09-16T15:02:21.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ef2506640 1 -- 172.21.15.26:0/3112516275 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ecc005050 con 0x7f3eec0a2800 2024-09-16T15:02:21.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f3ee4042e00 con 0x7f3eec0a2800 2024-09-16T15:02:21.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.695+0000 7f3eebfff640 10 monclient: get_auth_request con 0x7f3ed8036400 auth_method 0 2024-09-16T15:02:21.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.699+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3ee4004450 con 0x7f3eec0a2800 2024-09-16T15:02:21.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.699+0000 7f3ee8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:21.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.699+0000 7f3ee8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:21.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.919+0000 7f3ef2506640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T15:02:21.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.919+0000 7f3ef2506640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T15:02:21.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.919+0000 7f3ef2506640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:21.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.919+0000 7f3ef2506640 1 -- 172.21.15.26:0/3112516275 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f3ecc005b10 con 0x7f3eec0a2800 2024-09-16T15:02:21.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f3ee8ff9640 1 -- 172.21.15.26:0/3112516275 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f3ee4011c30 con 0x7f3eec0a2800 2024-09-16T15:02:21.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f3ee8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T15:02:21.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f3ee8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:02:21.925 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.925 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=11,kv=11,logm=52} crc {auth=1718199875,config=692619927,health=1139636889,kv=1867599010,logm=289562351}) 2024-09-16T15:02:21.925 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=1,health=10,kv=10,logm=55} crc {auth=320997900,config=1228700967,health=720087267,kv=3455456366,logm=2899265233}) 2024-09-16T15:02:21.925 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.925 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=11,kv=11,logm=52} crc {auth=1718199875,config=692619927,health=1139636889,kv=1867599010,logm=289562351}) 2024-09-16T15:02:21.925 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=1,health=9,kv=11,logm=54} crc {auth=1877003231,config=3678701417,health=192518597,kv=1867599010,logm=3294200097}) 2024-09-16T15:02:21.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f3ef2506640 1 -- 172.21.15.26:0/3112516275 >> v1:172.21.15.189:6812/3196161582 conn(0x7f3ed8036400 legacy=0x7f3ed80388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f3ef2506640 10 monclient: shutdown 2024-09-16T15:02:21.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f3ef2506640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:21.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f3ef2506640 1 -- 172.21.15.26:0/3112516275 >> v1:172.21.15.26:6789/0 conn(0x7f3eec0a2800 legacy=0x7f3eec1d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:21.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f3ef2506640 1 -- 172.21.15.26:0/3112516275 shutdown_connections 2024-09-16T15:02:21.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f3ef2506640 1 -- 172.21.15.26:0/3112516275 >> 172.21.15.26:0/3112516275 conn(0x7f3eec0a7f30 msgr2=0x7f3eec06b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:21.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f3ef2506640 1 -- 172.21.15.26:0/3112516275 shutdown_connections 2024-09-16T15:02:21.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:21.923+0000 7f3ef2506640 1 -- 172.21.15.26:0/3112516275 wait complete. 2024-09-16T15:02:21.928 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.927+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.928 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.927+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1029635783}) 2024-09-16T15:02:21.928 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.927+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=806779214}) 2024-09-16T15:02:21.929 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.927+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.929 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.927+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1029635783}) 2024-09-16T15:02:21.929 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.927+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1529152398}) 2024-09-16T15:02:21.931 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.931+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.931 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.931+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4280765808}) 2024-09-16T15:02:21.931 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.931+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1603690711}) 2024-09-16T15:02:21.931 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.931+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.932 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.931+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4280765808}) 2024-09-16T15:02:21.932 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.931+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1857791749}) 2024-09-16T15:02:21.935 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.935+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.935 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.935+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=831372612}) 2024-09-16T15:02:21.935 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.935+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829860230}) 2024-09-16T15:02:21.935 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.935+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.935 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.935+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=831372612}) 2024-09-16T15:02:21.936 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.935+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2105923827}) 2024-09-16T15:02:21.938 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.935+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.938 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.935+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2913890852}) 2024-09-16T15:02:21.938 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.935+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320802836}) 2024-09-16T15:02:21.939 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.935+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.939 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.935+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2913890852}) 2024-09-16T15:02:21.939 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.935+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1982199086}) 2024-09-16T15:02:21.941 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.939+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.941 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.939+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3603708559}) 2024-09-16T15:02:21.941 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.939+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4019146369}) 2024-09-16T15:02:21.941 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.939+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.941 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.939+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3603708559}) 2024-09-16T15:02:21.941 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.939+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1804778533}) 2024-09-16T15:02:21.943 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.939+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.943 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.939+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=477786671}) 2024-09-16T15:02:21.943 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.939+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3568260459}) 2024-09-16T15:02:21.943 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.939+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.943 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.939+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=477786671}) 2024-09-16T15:02:21.943 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.939+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4242651996}) 2024-09-16T15:02:21.945 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T15:02:21.945 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.943+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.946 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.943+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1821942038}) 2024-09-16T15:02:21.946 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.943+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1471763803}) 2024-09-16T15:02:21.946 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.943+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.946 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.943+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1821942038}) 2024-09-16T15:02:21.946 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.943+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3284784984}) 2024-09-16T15:02:21.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.943+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.943+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2634596667}) 2024-09-16T15:02:21.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.943+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2569863587}) 2024-09-16T15:02:21.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.943+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.943+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2634596667}) 2024-09-16T15:02:21.947 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.943+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1288800978}) 2024-09-16T15:02:21.949 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.947+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.949 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.947+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4194707733}) 2024-09-16T15:02:21.949 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.947+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2430160282}) 2024-09-16T15:02:21.949 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.947+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.949 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.947+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4194707733}) 2024-09-16T15:02:21.949 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.947+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=918412329}) 2024-09-16T15:02:21.951 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.947+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.951 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.947+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2723392721}) 2024-09-16T15:02:21.951 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.947+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909569131}) 2024-09-16T15:02:21.951 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.947+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.951 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.947+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2723392721}) 2024-09-16T15:02:21.951 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.947+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1075938114}) 2024-09-16T15:02:21.953 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.951+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.953 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.951+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3862663991}) 2024-09-16T15:02:21.953 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.951+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2092706717}) 2024-09-16T15:02:21.953 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.951+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.953 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.951+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3862663991}) 2024-09-16T15:02:21.953 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.951+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2431323418}) 2024-09-16T15:02:21.955 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.951+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.955 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.951+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=964532221}) 2024-09-16T15:02:21.955 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.951+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386149676}) 2024-09-16T15:02:21.955 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.951+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.955 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.951+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=964532221}) 2024-09-16T15:02:21.955 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.951+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3690911009}) 2024-09-16T15:02:21.956 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.955+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.957 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.955+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=822707358}) 2024-09-16T15:02:21.957 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.955+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3598114104}) 2024-09-16T15:02:21.957 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.955+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.957 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.955+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=822707358}) 2024-09-16T15:02:21.957 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.955+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2701102708}) 2024-09-16T15:02:21.959 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.955+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.959 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.955+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2677243909}) 2024-09-16T15:02:21.959 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.955+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260435533}) 2024-09-16T15:02:21.959 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.955+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.959 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.955+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2677243909}) 2024-09-16T15:02:21.959 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.955+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2909197404}) 2024-09-16T15:02:21.961 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.959+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.961 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.959+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3621653504}) 2024-09-16T15:02:21.961 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.959+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=84920537}) 2024-09-16T15:02:21.961 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.959+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.963 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.959+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3621653504}) 2024-09-16T15:02:21.963 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.959+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3640269738}) 2024-09-16T15:02:21.964 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.959+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.964 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.959+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1584724325}) 2024-09-16T15:02:21.964 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.959+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2506369252}) 2024-09-16T15:02:21.964 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.959+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.964 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.959+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1584724325}) 2024-09-16T15:02:21.964 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.959+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2352927945}) 2024-09-16T15:02:21.965 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.963+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.965 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.963+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=226159344}) 2024-09-16T15:02:21.965 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.963+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575523034}) 2024-09-16T15:02:21.965 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.963+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.966 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.963+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=226159344}) 2024-09-16T15:02:21.966 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.963+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2865178804}) 2024-09-16T15:02:21.967 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.963+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.967 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.963+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2323460163}) 2024-09-16T15:02:21.967 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.963+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215847047}) 2024-09-16T15:02:21.967 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.963+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.967 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.963+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2323460163}) 2024-09-16T15:02:21.968 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.963+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4102912165}) 2024-09-16T15:02:21.969 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.967+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.969 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.967+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2442777626}) 2024-09-16T15:02:21.969 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.967+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1852156437}) 2024-09-16T15:02:21.969 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.967+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.969 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.967+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2442777626}) 2024-09-16T15:02:21.969 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.967+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=53239357}) 2024-09-16T15:02:21.971 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.967+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.971 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.967+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=104547473}) 2024-09-16T15:02:21.971 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.967+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=723133524}) 2024-09-16T15:02:21.971 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.967+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.971 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.967+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=104547473}) 2024-09-16T15:02:21.971 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.967+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1454486900}) 2024-09-16T15:02:21.973 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.971+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.973 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.971+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1940961384}) 2024-09-16T15:02:21.973 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.971+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1487539692}) 2024-09-16T15:02:21.973 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.971+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.973 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.971+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1940961384}) 2024-09-16T15:02:21.973 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.971+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2494808208}) 2024-09-16T15:02:21.975 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.971+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.975 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.971+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4032376811}) 2024-09-16T15:02:21.975 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.971+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407366714}) 2024-09-16T15:02:21.975 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.971+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.975 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.971+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4032376811}) 2024-09-16T15:02:21.975 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.971+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3249096123}) 2024-09-16T15:02:21.976 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.975+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.977 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.975+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1540047872}) 2024-09-16T15:02:21.977 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.975+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=245642461}) 2024-09-16T15:02:21.977 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.975+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.977 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.975+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1540047872}) 2024-09-16T15:02:21.977 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.975+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=599624575}) 2024-09-16T15:02:21.979 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.975+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.979 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.975+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=637645189}) 2024-09-16T15:02:21.979 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.975+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2570504641}) 2024-09-16T15:02:21.979 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.975+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.979 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.975+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=637645189}) 2024-09-16T15:02:21.979 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.975+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1469545661}) 2024-09-16T15:02:21.980 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.979+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.981 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.979+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=781514440}) 2024-09-16T15:02:21.981 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.979+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345532055}) 2024-09-16T15:02:21.981 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.979+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.981 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.979+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=781514440}) 2024-09-16T15:02:21.981 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.979+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=398042849}) 2024-09-16T15:02:21.982 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.979+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.983 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.979+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3375667790}) 2024-09-16T15:02:21.983 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.979+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498993700}) 2024-09-16T15:02:21.983 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.979+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.983 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.979+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3375667790}) 2024-09-16T15:02:21.983 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.979+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=146680885}) 2024-09-16T15:02:21.985 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.983+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.985 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.983+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4041101850}) 2024-09-16T15:02:21.985 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.983+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=104852413}) 2024-09-16T15:02:21.985 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.983+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.985 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.983+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4041101850}) 2024-09-16T15:02:21.985 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.983+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3869939001}) 2024-09-16T15:02:21.987 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.983+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.987 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.983+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3106374277}) 2024-09-16T15:02:21.987 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.983+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2570358774}) 2024-09-16T15:02:21.987 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.983+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.987 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.983+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3106374277}) 2024-09-16T15:02:21.987 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.983+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3084173950}) 2024-09-16T15:02:21.989 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.987+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.989 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.987+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2724087801}) 2024-09-16T15:02:21.989 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.987+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2941538624}) 2024-09-16T15:02:21.989 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.987+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.989 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.987+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2724087801}) 2024-09-16T15:02:21.989 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.987+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1222843150}) 2024-09-16T15:02:21.991 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.987+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.991 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.987+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2421057395}) 2024-09-16T15:02:21.991 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.987+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1415568911}) 2024-09-16T15:02:21.991 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.987+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.991 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.987+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2421057395}) 2024-09-16T15:02:21.991 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.987+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2788005265}) 2024-09-16T15:02:21.993 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.991+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.993 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.991+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=596176893}) 2024-09-16T15:02:21.993 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.991+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2895469397}) 2024-09-16T15:02:21.993 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.991+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.993 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.991+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=596176893}) 2024-09-16T15:02:21.993 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.991+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1708413087}) 2024-09-16T15:02:21.994 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.991+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.995 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.991+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1641369539}) 2024-09-16T15:02:21.995 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.991+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1079063723}) 2024-09-16T15:02:21.995 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.991+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.995 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.991+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1641369539}) 2024-09-16T15:02:21.995 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.991+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787732223}) 2024-09-16T15:02:21.996 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.995+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.996 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.995+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3814310801}) 2024-09-16T15:02:21.997 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.995+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2907316692}) 2024-09-16T15:02:21.997 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.995+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.997 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.995+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3814310801}) 2024-09-16T15:02:21.997 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.995+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=778059783}) 2024-09-16T15:02:21.998 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.995+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.998 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.995+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2917672418}) 2024-09-16T15:02:21.998 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.995+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139171012}) 2024-09-16T15:02:21.998 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.995+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:21.999 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.995+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2917672418}) 2024-09-16T15:02:21.999 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.995+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987675769}) 2024-09-16T15:02:22.000 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.999+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.000 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.999+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=915107694}) 2024-09-16T15:02:22.000 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.999+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2758095974}) 2024-09-16T15:02:22.000 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.999+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.000 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.999+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=915107694}) 2024-09-16T15:02:22.000 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.999+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1668030887}) 2024-09-16T15:02:22.002 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.999+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.002 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.999+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2278280344}) 2024-09-16T15:02:22.002 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.999+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=812587370}) 2024-09-16T15:02:22.002 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.999+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.002 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.999+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2278280344}) 2024-09-16T15:02:22.002 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:21.999+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2349446157}) 2024-09-16T15:02:22.004 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.003+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.004 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.003+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2572432594}) 2024-09-16T15:02:22.004 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.003+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1604618054}) 2024-09-16T15:02:22.004 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.003+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.004 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.003+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2572432594}) 2024-09-16T15:02:22.004 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.003+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=682220758}) 2024-09-16T15:02:22.005 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.003+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.005 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.003+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3729620857}) 2024-09-16T15:02:22.005 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.003+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1270121713}) 2024-09-16T15:02:22.006 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.003+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.006 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.003+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3729620857}) 2024-09-16T15:02:22.006 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.003+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4188180470}) 2024-09-16T15:02:22.007 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.007 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=140177569}) 2024-09-16T15:02:22.007 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1008934567}) 2024-09-16T15:02:22.007 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.007 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=140177569}) 2024-09-16T15:02:22.007 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2755433110}) 2024-09-16T15:02:22.009 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.009 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=822310342}) 2024-09-16T15:02:22.009 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=56068082}) 2024-09-16T15:02:22.009 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.009 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=822310342}) 2024-09-16T15:02:22.009 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=937977465}) 2024-09-16T15:02:22.011 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.011 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=525691801}) 2024-09-16T15:02:22.011 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3732227248}) 2024-09-16T15:02:22.011 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.011 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=525691801}) 2024-09-16T15:02:22.011 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.007+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2179742799}) 2024-09-16T15:02:22.012 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.011+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.013 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.011+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1595912320}) 2024-09-16T15:02:22.013 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.011+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2844774040}) 2024-09-16T15:02:22.013 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.011+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.013 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.011+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1595912320}) 2024-09-16T15:02:22.013 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.011+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3298281923}) 2024-09-16T15:02:22.014 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.011+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.015 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.011+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4015678079}) 2024-09-16T15:02:22.015 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.011+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=695972353}) 2024-09-16T15:02:22.015 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.011+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.015 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.011+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4015678079}) 2024-09-16T15:02:22.015 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.011+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=319622386}) 2024-09-16T15:02:22.016 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.015+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.017 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.015+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=613647749}) 2024-09-16T15:02:22.017 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.015+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2274436547}) 2024-09-16T15:02:22.017 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.015+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.017 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.015+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=613647749}) 2024-09-16T15:02:22.017 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.015+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2752790544}) 2024-09-16T15:02:22.018 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.015+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.018 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.015+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=468839097}) 2024-09-16T15:02:22.018 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.015+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3253512765}) 2024-09-16T15:02:22.019 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.015+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.019 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.015+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=468839097}) 2024-09-16T15:02:22.019 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.015+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=411559959}) 2024-09-16T15:02:22.020 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.019+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.020 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.019+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=208162523}) 2024-09-16T15:02:22.020 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.019+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178121483}) 2024-09-16T15:02:22.020 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.019+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.020 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.019+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=208162523}) 2024-09-16T15:02:22.020 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.019+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3283607265}) 2024-09-16T15:02:22.022 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.019+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.022 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.019+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3349895782}) 2024-09-16T15:02:22.022 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.019+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2657380368}) 2024-09-16T15:02:22.022 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.019+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.022 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.019+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3349895782}) 2024-09-16T15:02:22.022 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.019+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2864155242}) 2024-09-16T15:02:22.024 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.023+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.024 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.023+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2157222334}) 2024-09-16T15:02:22.024 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.023+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3542028551}) 2024-09-16T15:02:22.024 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.023+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.024 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.023+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2157222334}) 2024-09-16T15:02:22.025 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.023+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=606861330}) 2024-09-16T15:02:22.026 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.023+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.026 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.023+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=24320665}) 2024-09-16T15:02:22.026 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.023+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1644075250}) 2024-09-16T15:02:22.026 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.023+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.027 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.023+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=24320665}) 2024-09-16T15:02:22.027 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.023+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1775783679}) 2024-09-16T15:02:22.028 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.027+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.028 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.027+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2121570426}) 2024-09-16T15:02:22.028 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.027+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=912131406}) 2024-09-16T15:02:22.028 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.027+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.028 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.027+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2121570426}) 2024-09-16T15:02:22.029 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.027+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1436999714}) 2024-09-16T15:02:22.031 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.027+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.031 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.027+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1614496973}) 2024-09-16T15:02:22.031 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.027+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297110223}) 2024-09-16T15:02:22.031 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.027+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.031 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.027+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1614496973}) 2024-09-16T15:02:22.032 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.027+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3754621137}) 2024-09-16T15:02:22.033 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.031+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.033 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.031+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2560359091}) 2024-09-16T15:02:22.033 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.031+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954951928}) 2024-09-16T15:02:22.033 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.031+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.033 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.031+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2560359091}) 2024-09-16T15:02:22.033 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.031+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3665986757}) 2024-09-16T15:02:22.035 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.031+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.035 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.031+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=663862883}) 2024-09-16T15:02:22.035 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.031+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130960793}) 2024-09-16T15:02:22.035 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.031+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.035 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.031+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=663862883}) 2024-09-16T15:02:22.035 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.031+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3137571915}) 2024-09-16T15:02:22.037 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.035+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.037 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.035+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742343835}) 2024-09-16T15:02:22.037 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.035+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365690656}) 2024-09-16T15:02:22.037 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.035+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.037 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.035+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742343835}) 2024-09-16T15:02:22.037 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.035+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1238651561}) 2024-09-16T15:02:22.039 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.035+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.039 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.039+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1962165613}) 2024-09-16T15:02:22.039 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.039+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167765568}) 2024-09-16T15:02:22.040 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.039+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.040 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.039+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1962165613}) 2024-09-16T15:02:22.040 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.039+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3332597278}) 2024-09-16T15:02:22.041 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.039+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.041 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.039+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3525402599}) 2024-09-16T15:02:22.041 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.039+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=594464078}) 2024-09-16T15:02:22.041 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.039+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.041 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.039+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3525402599}) 2024-09-16T15:02:22.042 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.039+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3010762316}) 2024-09-16T15:02:22.043 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.043 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4129445438}) 2024-09-16T15:02:22.043 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1611027168}) 2024-09-16T15:02:22.043 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.044 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4129445438}) 2024-09-16T15:02:22.044 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1063389586}) 2024-09-16T15:02:22.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3505783697}) 2024-09-16T15:02:22.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345526653}) 2024-09-16T15:02:22.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3505783697}) 2024-09-16T15:02:22.045 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=843695463}) 2024-09-16T15:02:22.047 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.047 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2981052648}) 2024-09-16T15:02:22.047 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4274843259}) 2024-09-16T15:02:22.047 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.047 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.043+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2981052648}) 2024-09-16T15:02:22.047 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.047+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2952582945}) 2024-09-16T15:02:22.049 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.047+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.049 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.047+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3480613081}) 2024-09-16T15:02:22.049 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.047+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2970533080}) 2024-09-16T15:02:22.049 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.047+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.049 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.047+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3480613081}) 2024-09-16T15:02:22.049 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.047+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=624269270}) 2024-09-16T15:02:22.051 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.047+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.051 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.047+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2587048741}) 2024-09-16T15:02:22.051 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.047+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3552632025}) 2024-09-16T15:02:22.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.047+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.047+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2587048741}) 2024-09-16T15:02:22.052 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.047+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3424443075}) 2024-09-16T15:02:22.053 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.051+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.053 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.051+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1469951231}) 2024-09-16T15:02:22.053 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.051+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=483804637}) 2024-09-16T15:02:22.053 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.051+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.053 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.051+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1469951231}) 2024-09-16T15:02:22.053 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.051+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3653197290}) 2024-09-16T15:02:22.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.051+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.051+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2581029377}) 2024-09-16T15:02:22.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.051+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575917249}) 2024-09-16T15:02:22.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.051+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.051+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2581029377}) 2024-09-16T15:02:22.055 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.051+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3121871177}) 2024-09-16T15:02:22.057 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.055+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.057 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.055+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=191896956}) 2024-09-16T15:02:22.057 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.055+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517518887}) 2024-09-16T15:02:22.057 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.055+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.057 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.055+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=191896956}) 2024-09-16T15:02:22.057 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.055+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3068216955}) 2024-09-16T15:02:22.059 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.055+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.059 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.055+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=68012643}) 2024-09-16T15:02:22.059 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.055+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2669172816}) 2024-09-16T15:02:22.059 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.055+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.059 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.055+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=68012643}) 2024-09-16T15:02:22.059 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.055+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3261358842}) 2024-09-16T15:02:22.060 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.059+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.059+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2633166384}) 2024-09-16T15:02:22.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.059+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2984112819}) 2024-09-16T15:02:22.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.059+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.059+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2633166384}) 2024-09-16T15:02:22.061 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.059+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=110036893}) 2024-09-16T15:02:22.062 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.059+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.059+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1653934998}) 2024-09-16T15:02:22.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.059+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2471775487}) 2024-09-16T15:02:22.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.059+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.059+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1653934998}) 2024-09-16T15:02:22.063 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.059+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3311013469}) 2024-09-16T15:02:22.064 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.063+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.064 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.063+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=908755804}) 2024-09-16T15:02:22.065 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.063+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185636080}) 2024-09-16T15:02:22.065 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.063+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.065 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.063+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=908755804}) 2024-09-16T15:02:22.065 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.063+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1903838622}) 2024-09-16T15:02:22.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.063+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.063+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794211828}) 2024-09-16T15:02:22.066 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.063+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2594367983}) 2024-09-16T15:02:22.067 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.063+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.067 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.063+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794211828}) 2024-09-16T15:02:22.067 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.063+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2169782952}) 2024-09-16T15:02:22.068 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.067+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.068 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.067+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1832821257}) 2024-09-16T15:02:22.068 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.067+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1620201020}) 2024-09-16T15:02:22.069 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.067+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.069 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.067+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1832821257}) 2024-09-16T15:02:22.069 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.067+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1253127178}) 2024-09-16T15:02:22.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.067+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.067+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2840884080}) 2024-09-16T15:02:22.070 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.067+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3399130554}) 2024-09-16T15:02:22.071 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.067+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.071 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.067+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2840884080}) 2024-09-16T15:02:22.071 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.067+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1800571601}) 2024-09-16T15:02:22.072 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.071+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.073 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.071+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3771444378}) 2024-09-16T15:02:22.073 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.071+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3512018310}) 2024-09-16T15:02:22.073 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.071+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.073 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.071+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3771444378}) 2024-09-16T15:02:22.073 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.071+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1725101182}) 2024-09-16T15:02:22.075 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.071+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.075 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.071+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3415971465}) 2024-09-16T15:02:22.075 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.071+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4038107010}) 2024-09-16T15:02:22.075 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.071+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.076 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.071+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3415971465}) 2024-09-16T15:02:22.076 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.071+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2716700934}) 2024-09-16T15:02:22.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.075+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.075+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3910080262}) 2024-09-16T15:02:22.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.075+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1037783846}) 2024-09-16T15:02:22.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.075+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.075+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3910080262}) 2024-09-16T15:02:22.077 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.075+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2676133741}) 2024-09-16T15:02:22.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.075+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.075+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2184090549}) 2024-09-16T15:02:22.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.075+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=747817538}) 2024-09-16T15:02:22.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.075+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.079 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.079+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2184090549}) 2024-09-16T15:02:22.080 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.079+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=343682738}) 2024-09-16T15:02:22.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.079+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.079+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3665545319}) 2024-09-16T15:02:22.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.079+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691735248}) 2024-09-16T15:02:22.081 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.079+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.082 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.079+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3665545319}) 2024-09-16T15:02:22.082 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.079+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4259981894}) 2024-09-16T15:02:22.083 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.083+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.083 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.083+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2025372480}) 2024-09-16T15:02:22.084 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.083+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2881925719}) 2024-09-16T15:02:22.084 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.083+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.084 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.083+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2025372480}) 2024-09-16T15:02:22.084 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.083+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3179858431}) 2024-09-16T15:02:22.085 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.083+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.085 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.083+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3261843959}) 2024-09-16T15:02:22.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.083+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492744317}) 2024-09-16T15:02:22.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.083+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.083+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3261843959}) 2024-09-16T15:02:22.086 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.083+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=478154375}) 2024-09-16T15:02:22.087 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.087+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.087+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3805939420}) 2024-09-16T15:02:22.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.087+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860938982}) 2024-09-16T15:02:22.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.087+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.087+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3805939420}) 2024-09-16T15:02:22.088 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.087+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3367148348}) 2024-09-16T15:02:22.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.087+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.087+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=501008627}) 2024-09-16T15:02:22.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.087+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=216143868}) 2024-09-16T15:02:22.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.087+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.087+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=501008627}) 2024-09-16T15:02:22.090 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.087+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2528992872}) 2024-09-16T15:02:22.092 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.091+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.092 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.091+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=740797523}) 2024-09-16T15:02:22.092 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.091+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2848730904}) 2024-09-16T15:02:22.092 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.091+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.092 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.091+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=740797523}) 2024-09-16T15:02:22.092 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.091+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4141796343}) 2024-09-16T15:02:22.094 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.091+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.094 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.091+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4074274323}) 2024-09-16T15:02:22.094 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.091+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3860783890}) 2024-09-16T15:02:22.094 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.091+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.094 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.091+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4074274323}) 2024-09-16T15:02:22.094 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.091+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833663545}) 2024-09-16T15:02:22.096 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.095+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.096 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.095+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2000458863}) 2024-09-16T15:02:22.096 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.095+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2614717176}) 2024-09-16T15:02:22.097 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.095+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.097 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.095+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2000458863}) 2024-09-16T15:02:22.097 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.095+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2058288627}) 2024-09-16T15:02:22.098 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.095+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.098 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.095+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4165041871}) 2024-09-16T15:02:22.098 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.095+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387696944}) 2024-09-16T15:02:22.099 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.095+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.099 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.095+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4165041871}) 2024-09-16T15:02:22.099 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.095+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3511842688}) 2024-09-16T15:02:22.100 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.099+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.101 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.099+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2095675687}) 2024-09-16T15:02:22.101 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.099+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3926301424}) 2024-09-16T15:02:22.101 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.099+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.101 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.099+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2095675687}) 2024-09-16T15:02:22.101 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.099+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3058812391}) 2024-09-16T15:02:22.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.099+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.099+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4080716671}) 2024-09-16T15:02:22.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.099+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511615932}) 2024-09-16T15:02:22.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.099+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.099+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4080716671}) 2024-09-16T15:02:22.103 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.099+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=505200494}) 2024-09-16T15:02:22.105 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.103+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.105 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.103+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3439650509}) 2024-09-16T15:02:22.105 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.103+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2775328370}) 2024-09-16T15:02:22.105 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.103+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.105 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.103+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3439650509}) 2024-09-16T15:02:22.105 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.103+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2659943817}) 2024-09-16T15:02:22.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.103+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.103+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=510873186}) 2024-09-16T15:02:22.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.103+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1727725572}) 2024-09-16T15:02:22.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.103+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.103+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=510873186}) 2024-09-16T15:02:22.107 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.103+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2077683361}) 2024-09-16T15:02:22.109 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.107+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.109 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.107+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=701185733}) 2024-09-16T15:02:22.109 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.107+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3263373152}) 2024-09-16T15:02:22.109 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.107+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.109 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.107+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=701185733}) 2024-09-16T15:02:22.109 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.107+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=436021321}) 2024-09-16T15:02:22.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.107+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.107+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2458229531}) 2024-09-16T15:02:22.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.107+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=507073943}) 2024-09-16T15:02:22.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.107+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.107+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2458229531}) 2024-09-16T15:02:22.111 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.107+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983288551}) 2024-09-16T15:02:22.113 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.111+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.113 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.111+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1733796708}) 2024-09-16T15:02:22.113 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.111+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950325735}) 2024-09-16T15:02:22.113 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.111+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.113 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.111+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1733796708}) 2024-09-16T15:02:22.113 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.111+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159557510}) 2024-09-16T15:02:22.115 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.111+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.115 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.111+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1536592362}) 2024-09-16T15:02:22.115 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.111+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2259878417}) 2024-09-16T15:02:22.115 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.111+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.115 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.111+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1536592362}) 2024-09-16T15:02:22.115 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.111+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3444098672}) 2024-09-16T15:02:22.117 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.115+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.117 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.115+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=805025790}) 2024-09-16T15:02:22.117 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.115+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3145797940}) 2024-09-16T15:02:22.117 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.115+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.117 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.115+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=805025790}) 2024-09-16T15:02:22.117 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.115+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=436733955}) 2024-09-16T15:02:22.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.115+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.115+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4265379497}) 2024-09-16T15:02:22.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.115+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2394902940}) 2024-09-16T15:02:22.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.115+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.115+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4265379497}) 2024-09-16T15:02:22.119 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.115+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=311713040}) 2024-09-16T15:02:22.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.119+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.119+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=221000724}) 2024-09-16T15:02:22.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.119+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=497337935}) 2024-09-16T15:02:22.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.119+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.119+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=221000724}) 2024-09-16T15:02:22.121 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.119+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=698687387}) 2024-09-16T15:02:22.122 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.119+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.119+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=177335057}) 2024-09-16T15:02:22.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.119+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=248204559}) 2024-09-16T15:02:22.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.119+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.119+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=177335057}) 2024-09-16T15:02:22.123 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.119+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3539143987}) 2024-09-16T15:02:22.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.123+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.123+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1002141626}) 2024-09-16T15:02:22.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.123+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665736830}) 2024-09-16T15:02:22.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.123+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.123+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1002141626}) 2024-09-16T15:02:22.125 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.123+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3879082597}) 2024-09-16T15:02:22.127 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.123+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.127 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.123+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2387817366}) 2024-09-16T15:02:22.127 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.123+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3345209448}) 2024-09-16T15:02:22.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.123+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.123+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2387817366}) 2024-09-16T15:02:22.128 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.123+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=545064078}) 2024-09-16T15:02:22.129 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.127+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.129 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.127+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1393880295}) 2024-09-16T15:02:22.129 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.127+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1249878543}) 2024-09-16T15:02:22.129 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.127+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.127+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1393880295}) 2024-09-16T15:02:22.130 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.127+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2739953658}) 2024-09-16T15:02:22.131 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.131+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.131 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.131+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2249605525}) 2024-09-16T15:02:22.131 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.131+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4275331520}) 2024-09-16T15:02:22.131 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.131+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.131+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2249605525}) 2024-09-16T15:02:22.132 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.131+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3417947144}) 2024-09-16T15:02:22.134 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.131+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.134 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.131+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1861743967}) 2024-09-16T15:02:22.134 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.131+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631081823}) 2024-09-16T15:02:22.134 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.131+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.134 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.131+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1861743967}) 2024-09-16T15:02:22.134 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.131+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4244484369}) 2024-09-16T15:02:22.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.135+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.135+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=120007639}) 2024-09-16T15:02:22.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.135+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3266706173}) 2024-09-16T15:02:22.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.135+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.135+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=120007639}) 2024-09-16T15:02:22.136 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.135+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=808817335}) 2024-09-16T15:02:22.138 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.135+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.138 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.135+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3040108092}) 2024-09-16T15:02:22.138 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.135+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3514240368}) 2024-09-16T15:02:22.138 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.135+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.138 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.135+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3040108092}) 2024-09-16T15:02:22.138 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.135+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1559321699}) 2024-09-16T15:02:22.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.139+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.139+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=874086262}) 2024-09-16T15:02:22.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.139+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3686311885}) 2024-09-16T15:02:22.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.139+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.139+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=874086262}) 2024-09-16T15:02:22.140 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.139+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=563576254}) 2024-09-16T15:02:22.148 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.147+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.148 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.147+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3971049678}) 2024-09-16T15:02:22.149 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.147+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2763437150}) 2024-09-16T15:02:22.149 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.147+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.149 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.147+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3971049678}) 2024-09-16T15:02:22.149 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.147+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024697846}) 2024-09-16T15:02:22.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.151+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.151+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2772494938}) 2024-09-16T15:02:22.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.151+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3077324106}) 2024-09-16T15:02:22.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.151+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.151+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2772494938}) 2024-09-16T15:02:22.153 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.151+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=80439908}) 2024-09-16T15:02:22.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.183+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.183+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4281631989}) 2024-09-16T15:02:22.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.183+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2228464415}) 2024-09-16T15:02:22.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.183+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.183+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4281631989}) 2024-09-16T15:02:22.186 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.183+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3037620512}) 2024-09-16T15:02:22.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.187+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.187+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2964125618}) 2024-09-16T15:02:22.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.187+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2965364335}) 2024-09-16T15:02:22.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.187+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.187+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2964125618}) 2024-09-16T15:02:22.188 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.187+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2508455193}) 2024-09-16T15:02:22.189 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.187+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.187+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=796941143}) 2024-09-16T15:02:22.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.187+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4128047603}) 2024-09-16T15:02:22.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.187+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.187+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=796941143}) 2024-09-16T15:02:22.190 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.187+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1763580439}) 2024-09-16T15:02:22.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.187+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2744829043}) 2024-09-16T15:02:22.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1065622910}) 2024-09-16T15:02:22.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2744829043}) 2024-09-16T15:02:22.191 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2408447958}) 2024-09-16T15:02:22.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1786869032}) 2024-09-16T15:02:22.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=655581511}) 2024-09-16T15:02:22.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1786869032}) 2024-09-16T15:02:22.193 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3401459021}) 2024-09-16T15:02:22.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1524064506}) 2024-09-16T15:02:22.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2882939033}) 2024-09-16T15:02:22.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1524064506}) 2024-09-16T15:02:22.195 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.191+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1130821993}) 2024-09-16T15:02:22.196 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.195+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.195+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3382836779}) 2024-09-16T15:02:22.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.195+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1093270383}) 2024-09-16T15:02:22.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.195+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.195+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3382836779}) 2024-09-16T15:02:22.197 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.195+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2026435456}) 2024-09-16T15:02:22.198 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.195+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.199 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.195+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1467489511}) 2024-09-16T15:02:22.199 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.195+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=28347910}) 2024-09-16T15:02:22.199 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.195+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.199 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.195+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1467489511}) 2024-09-16T15:02:22.199 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.195+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2620919243}) 2024-09-16T15:02:22.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.199+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.199+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4038235440}) 2024-09-16T15:02:22.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.199+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553724760}) 2024-09-16T15:02:22.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.199+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.199+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4038235440}) 2024-09-16T15:02:22.200 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.199+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1996913514}) 2024-09-16T15:02:22.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.199+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.199+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3156490694}) 2024-09-16T15:02:22.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.199+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622258917}) 2024-09-16T15:02:22.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.199+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.199+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3156490694}) 2024-09-16T15:02:22.202 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.199+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=412131522}) 2024-09-16T15:02:22.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.203+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.203+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=390328874}) 2024-09-16T15:02:22.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.203+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=96888679}) 2024-09-16T15:02:22.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.203+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.203+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=390328874}) 2024-09-16T15:02:22.204 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.203+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3594887850}) 2024-09-16T15:02:22.205 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.203+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.205 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.203+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3412839619}) 2024-09-16T15:02:22.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.203+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3081212349}) 2024-09-16T15:02:22.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.203+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.203+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3412839619}) 2024-09-16T15:02:22.206 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.203+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1673753081}) 2024-09-16T15:02:22.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1786438564}) 2024-09-16T15:02:22.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3508164071}) 2024-09-16T15:02:22.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.207 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1786438564}) 2024-09-16T15:02:22.208 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2040946882}) 2024-09-16T15:02:22.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3031678428}) 2024-09-16T15:02:22.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1506483073}) 2024-09-16T15:02:22.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3031678428}) 2024-09-16T15:02:22.209 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3744406208}) 2024-09-16T15:02:22.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=399448490}) 2024-09-16T15:02:22.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239600640}) 2024-09-16T15:02:22.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=399448490}) 2024-09-16T15:02:22.211 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.207+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2445470643}) 2024-09-16T15:02:22.212 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.211+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.212 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.211+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3181519861}) 2024-09-16T15:02:22.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.211+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1371254998}) 2024-09-16T15:02:22.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.211+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.211+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3181519861}) 2024-09-16T15:02:22.213 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.211+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1986773555}) 2024-09-16T15:02:22.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.211+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.211+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3847248924}) 2024-09-16T15:02:22.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.211+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1545234433}) 2024-09-16T15:02:22.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.211+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.211+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3847248924}) 2024-09-16T15:02:22.214 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.211+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=56147624}) 2024-09-16T15:02:22.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.215+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.215+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1204405434}) 2024-09-16T15:02:22.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.215+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662846491}) 2024-09-16T15:02:22.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.215+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.215+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1204405434}) 2024-09-16T15:02:22.216 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.215+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=974031899}) 2024-09-16T15:02:22.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.215+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.215+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3469532034}) 2024-09-16T15:02:22.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.215+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19780615}) 2024-09-16T15:02:22.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.215+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.215+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3469532034}) 2024-09-16T15:02:22.218 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.215+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=69337644}) 2024-09-16T15:02:22.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.219 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2583023619}) 2024-09-16T15:02:22.220 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3206825365}) 2024-09-16T15:02:22.220 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.220 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2583023619}) 2024-09-16T15:02:22.220 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2866058825}) 2024-09-16T15:02:22.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3942849231}) 2024-09-16T15:02:22.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1720573212}) 2024-09-16T15:02:22.221 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.222 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3942849231}) 2024-09-16T15:02:22.222 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=894725881}) 2024-09-16T15:02:22.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2013701451}) 2024-09-16T15:02:22.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1512684407}) 2024-09-16T15:02:22.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2013701451}) 2024-09-16T15:02:22.223 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.219+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2855949961}) 2024-09-16T15:02:22.224 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.224 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2903309224}) 2024-09-16T15:02:22.224 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2187387037}) 2024-09-16T15:02:22.224 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.224 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2903309224}) 2024-09-16T15:02:22.224 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2911474303}) 2024-09-16T15:02:22.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1846370550}) 2024-09-16T15:02:22.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1882938646}) 2024-09-16T15:02:22.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.225 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1846370550}) 2024-09-16T15:02:22.226 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1226612914}) 2024-09-16T15:02:22.226 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3137774567}) 2024-09-16T15:02:22.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=74553955}) 2024-09-16T15:02:22.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3137774567}) 2024-09-16T15:02:22.227 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.223+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2399904845}) 2024-09-16T15:02:22.228 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.227+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.228 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.227+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1243896796}) 2024-09-16T15:02:22.228 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.227+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2457129879}) 2024-09-16T15:02:22.228 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.227+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.228 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.227+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1243896796}) 2024-09-16T15:02:22.228 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.227+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4030809670}) 2024-09-16T15:02:22.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.227+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.227+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2617444840}) 2024-09-16T15:02:22.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.227+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4049289123}) 2024-09-16T15:02:22.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.227+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.229 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.227+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2617444840}) 2024-09-16T15:02:22.230 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.227+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3917197793}) 2024-09-16T15:02:22.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.231+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.231+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=26} crc {logm=4142567286,mdsmap=1531151339,mgr=1843570494,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3528158512,mon_config_key=2512756450,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=201405530}) 2024-09-16T15:02:22.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.231+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=23} crc {logm=2004527249,mdsmap=1531151339,mgr=2120425906,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3528158512,mon_config_key=1518680965,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=1176474358}) 2024-09-16T15:02:22.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.231+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.231+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=26} crc {logm=4142567286,mdsmap=1531151339,mgr=1843570494,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3528158512,mon_config_key=2512756450,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=201405530}) 2024-09-16T15:02:22.232 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.231+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=29,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=26} crc {logm=654241994,mdsmap=1531151339,mgr=3484265733,mgr_command_descs=794974419,mgr_metadata=2109538692,mgrstat=3528158512,mon_config_key=270382713,monmap=970648238,osd_metadata=3424249799,osd_pg_creating=1354844604,osd_snap=3794734670,osdmap=3968680119}) 2024-09-16T15:02:22.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.239+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.239+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=27} crc {osdmap=2622570999}) 2024-09-16T15:02:22.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.239+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=27} crc {osdmap=3608984321}) 2024-09-16T15:02:22.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.239+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T15:02:22.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.239+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=27} crc {osdmap=2622570999}) 2024-09-16T15:02:22.242 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-09-16T15:02:22.239+0000 7f15eb555640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=27} crc {osdmap=976652768}) 2024-09-16T15:02:22.946 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T15:02:22.946 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T15:02:23.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 10 monclient: get_monmap_and_config 2024-09-16T15:02:23.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 10 monclient: build_initial_monmap 2024-09-16T15:02:23.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 10 monclient: monmap: 2024-09-16T15:02:23.077 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:23.077 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.077 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:23.078673+0000 2024-09-16T15:02:23.077 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:23.078673+0000 2024-09-16T15:02:23.077 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:23.077 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.077 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:23.077 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:23.077 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:23.077 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 1 Processor -- start 2024-09-16T15:02:23.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 1 -- start start 2024-09-16T15:02:23.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 10 monclient: init 2024-09-16T15:02:23.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:23.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:02:23.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 10 monclient(hunting): picked mon.noname-c con 0x7fa32409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:23.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 10 monclient(hunting): picked mon.noname-b con 0x7fa324132650 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:23.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 10 monclient(hunting): picked mon.noname-a con 0x7fa324138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:23.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32409d260 con 0x7fa324138ff0 2024-09-16T15:02:23.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa324132a80 con 0x7fa32409cb80 2024-09-16T15:02:23.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32409cf60 con 0x7fa324132650 2024-09-16T15:02:23.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:23.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 10 monclient(hunting): authenticate will time out at 2167.198750s 2024-09-16T15:02:23.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32a1ad640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa324138ff0 0x7fa32413b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42134/0 (socket says 172.21.15.26:42134) 2024-09-16T15:02:23.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32a1ad640 1 -- 172.21.15.26:0/2676580239 learned_addr learned my addr 172.21.15.26:0/2676580239 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:23.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3291ab640 1 --1- 172.21.15.26:0/2676580239 >> v1:172.21.15.26:6790/0 conn(0x7fa324132650 0x7fa3241378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47860/0 (socket says 172.21.15.26:47860) 2024-09-16T15:02:23.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 1 -- 172.21.15.26:0/2676580239 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474529056 0 0) 0x7fa32409d260 con 0x7fa324138ff0 2024-09-16T15:02:23.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient(hunting): _negotiate 2024-09-16T15:02:23.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:23.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:23.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient(hunting): my global_id is 334449 2024-09-16T15:02:23.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 1 -- 172.21.15.26:0/2676580239 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa310003530 con 0x7fa324138ff0 2024-09-16T15:02:23.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 1 -- 172.21.15.26:0/2676580239 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440849749 0 0) 0x7fa32409cf60 con 0x7fa324132650 2024-09-16T15:02:23.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient(hunting): _negotiate 2024-09-16T15:02:23.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:23.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:23.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient(hunting): my global_id is 354263 2024-09-16T15:02:23.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 1 -- 172.21.15.26:0/2676580239 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa32409d260 con 0x7fa324132650 2024-09-16T15:02:23.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 1 -- 172.21.15.26:0/2676580239 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 908688460 0 0) 0x7fa310003530 con 0x7fa324138ff0 2024-09-16T15:02:23.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 1 -- 172.21.15.26:0/2676580239 >> v1:172.21.15.26:6790/0 conn(0x7fa324132650 legacy=0x7fa3241378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 1 -- 172.21.15.26:0/2676580239 >> v1:172.21.15.189:6789/0 conn(0x7fa32409cb80 legacy=0x7fa3240717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient: _finish_hunting 0 2024-09-16T15:02:23.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 1 monclient: found mon.noname-a 2024-09-16T15:02:23.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:23.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:02:23.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 1 -- 172.21.15.26:0/2676580239 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32409f2d0 con 0x7fa324138ff0 2024-09-16T15:02:23.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient: _finish_auth 0 2024-09-16T15:02:23.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient: _check_auth_tickets 2024-09-16T15:02:23.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:23.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 1 -- 172.21.15.26:0/2676580239 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa31c001620 con 0x7fa324138ff0 2024-09-16T15:02:23.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 5 monclient: authenticate success, global_id 334449 2024-09-16T15:02:23.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:23.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:23.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:23.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient: dump: 2024-09-16T15:02:23.084 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:23.084 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.084 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.084 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.084 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:23.084 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.084 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:23.084 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:23.084 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:23.085 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:23.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 1 -- 172.21.15.26:0/2676580239 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa31c001850 con 0x7fa324138ff0 2024-09-16T15:02:23.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:23.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:23.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 1 -- 172.21.15.26:0/2676580239 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa31c001c20 con 0x7fa324138ff0 2024-09-16T15:02:23.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa32bc10640 4 set_mon_vals no callback set 2024-09-16T15:02:23.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:23.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:23.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.075+0000 7fa3289aa640 10 monclient: dump: 2024-09-16T15:02:23.085 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:23.085 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.085 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.086 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.086 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:23.086 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.086 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:23.086 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:23.086 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:23.086 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 10 monclient: shutdown 2024-09-16T15:02:23.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:23.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 1 -- 172.21.15.26:0/2676580239 >> v1:172.21.15.26:6789/0 conn(0x7fa324138ff0 legacy=0x7fa32413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 1 -- 172.21.15.26:0/2676580239 shutdown_connections 2024-09-16T15:02:23.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 1 -- 172.21.15.26:0/2676580239 >> 172.21.15.26:0/2676580239 conn(0x7fa32409fb50 msgr2=0x7fa32409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:23.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 1 -- 172.21.15.26:0/2676580239 shutdown_connections 2024-09-16T15:02:23.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 1 -- 172.21.15.26:0/2676580239 wait complete. 2024-09-16T15:02:23.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 10 monclient: build_initial_monmap 2024-09-16T15:02:23.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 10 monclient: monmap: 2024-09-16T15:02:23.087 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:23.087 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:23.087 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:23.087 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:23.087 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:23.087 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:23.087 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:23.087 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:23.087 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:23.087 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 1 Processor -- start 2024-09-16T15:02:23.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 1 -- start start 2024-09-16T15:02:23.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 10 monclient: init 2024-09-16T15:02:23.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:23.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:02:23.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 10 monclient(hunting): picked mon.noname-a con 0x7fa32409cb80 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:23.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 10 monclient(hunting): picked mon.noname-b con 0x7fa324132650 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:23.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 10 monclient(hunting): picked mon.noname-c con 0x7fa324138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:23.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa324070cf0 con 0x7fa32409cb80 2024-09-16T15:02:23.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa324138150 con 0x7fa324132650 2024-09-16T15:02:23.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3240a2910 con 0x7fa324138ff0 2024-09-16T15:02:23.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:23.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 10 monclient(hunting): authenticate will time out at 2167.202288s 2024-09-16T15:02:23.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa3299ac640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa32409cb80 0x7fa32406d5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42144/0 (socket says 172.21.15.26:42144) 2024-09-16T15:02:23.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa3299ac640 1 -- 172.21.15.26:0/1151854853 learned_addr learned my addr 172.21.15.26:0/1151854853 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:23.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116504794 0 0) 0x7fa324070cf0 con 0x7fa32409cb80 2024-09-16T15:02:23.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:02:23.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:23.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:23.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): my global_id is 334452 2024-09-16T15:02:23.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa308003530 con 0x7fa32409cb80 2024-09-16T15:02:23.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664918125 0 0) 0x7fa3240a2910 con 0x7fa324138ff0 2024-09-16T15:02:23.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:02:23.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:23.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:23.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): my global_id is 354266 2024-09-16T15:02:23.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa324070cf0 con 0x7fa324138ff0 2024-09-16T15:02:23.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1180251713 0 0) 0x7fa324070cf0 con 0x7fa324138ff0 2024-09-16T15:02:23.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3240a2910 con 0x7fa324138ff0 2024-09-16T15:02:23.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 23037375 0 0) 0x7fa308003530 con 0x7fa32409cb80 2024-09-16T15:02:23.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa324070cf0 con 0x7fa32409cb80 2024-09-16T15:02:23.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688014673 0 0) 0x7fa314000ff0 con 0x7fa324132650 2024-09-16T15:02:23.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:02:23.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:23.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:23.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): my global_id is 364132 2024-09-16T15:02:23.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa308003530 con 0x7fa324132650 2024-09-16T15:02:23.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa31c0019c0 con 0x7fa324138ff0 2024-09-16T15:02:23.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:23.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:02:23.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): dump: 2024-09-16T15:02:23.094 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:23.094 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.094 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.094 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.094 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:23.094 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.094 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:23.095 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:23.095 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:23.095 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:23.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa320001c60 con 0x7fa32409cb80 2024-09-16T15:02:23.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:23.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:23.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient(hunting): dump: 2024-09-16T15:02:23.095 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:23.095 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.096 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.096 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.096 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:23.096 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.096 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:23.096 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:23.096 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:23.096 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3407087644 0 0) 0x7fa3240a2910 con 0x7fa324138ff0 2024-09-16T15:02:23.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 >> v1:172.21.15.189:6789/0 conn(0x7fa324132650 legacy=0x7fa32406dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 >> v1:172.21.15.26:6789/0 conn(0x7fa32409cb80 legacy=0x7fa32406d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient: _finish_hunting 0 2024-09-16T15:02:23.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 monclient: found mon.c 2024-09-16T15:02:23.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:23.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:23.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32406e400 con 0x7fa324138ff0 2024-09-16T15:02:23.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient: _finish_auth 0 2024-09-16T15:02:23.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient: _check_auth_tickets 2024-09-16T15:02:23.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:23.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 5 monclient: authenticate success, global_id 354266 2024-09-16T15:02:23.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 10 monclient: _renew_subs 2024-09-16T15:02:23.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:23.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 1 -- 172.21.15.26:0/1151854853 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32406fbe0 con 0x7fa324138ff0 2024-09-16T15:02:23.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 10 monclient: _renew_subs 2024-09-16T15:02:23.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:23.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa32bc10640 1 -- 172.21.15.26:0/1151854853 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3240712b0 con 0x7fa324138ff0 2024-09-16T15:02:23.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa31c001cc0 con 0x7fa324138ff0 2024-09-16T15:02:23.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:23.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fa31c004da0 con 0x7fa324138ff0 2024-09-16T15:02:23.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:23.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:23.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31bfff640 4 set_mon_vals no callback set 2024-09-16T15:02:23.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.079+0000 7fa31a7fc640 10 monclient: dump: 2024-09-16T15:02:23.098 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:23.098 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.098 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.098 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.098 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:23.098 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.098 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:23.099 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:23.099 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:23.099 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.083+0000 7fa32bc10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:23.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.083+0000 7fa32bc10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:23.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.083+0000 7fa32bc10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:23.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.083+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fa31c00f810 con 0x7fa324138ff0 2024-09-16T15:02:23.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.083+0000 7fa32bc10640 1 -- 172.21.15.26:0/1151854853 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa30c005050 con 0x7fa324138ff0 2024-09-16T15:02:23.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.083+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fa31c043e00 con 0x7fa324138ff0 2024-09-16T15:02:23.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.083+0000 7fa3299ac640 10 monclient: get_auth_request con 0x7fa308036440 auth_method 0 2024-09-16T15:02:23.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.087+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa31c014620 con 0x7fa324138ff0 2024-09-16T15:02:23.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.087+0000 7fa31a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:23.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.087+0000 7fa31a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:23.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.311+0000 7fa32bc10640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T15:02:23.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.311+0000 7fa32bc10640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T15:02:23.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.311+0000 7fa32bc10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:23.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.311+0000 7fa32bc10640 1 -- 172.21.15.26:0/1151854853 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa30c005880 con 0x7fa324138ff0 2024-09-16T15:02:23.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.311+0000 7fa31a7fc640 1 -- 172.21.15.26:0/1151854853 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2316978990) 0x7fa31c012cc0 con 0x7fa324138ff0 2024-09-16T15:02:23.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.311+0000 7fa31a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T15:02:23.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.311+0000 7fa31a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:02:23.313 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":180,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","modified":"2024-09-16T14:41:47.153956Z","created":"2024-09-16T14:41:47.153956Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T15:02:23.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.311+0000 7fa32bc10640 1 -- 172.21.15.26:0/1151854853 >> v1:172.21.15.189:6812/3196161582 conn(0x7fa308036440 legacy=0x7fa308038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.311+0000 7fa32bc10640 10 monclient: shutdown 2024-09-16T15:02:23.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.311+0000 7fa32bc10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:23.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.311+0000 7fa32bc10640 1 -- 172.21.15.26:0/1151854853 >> v1:172.21.15.26:6790/0 conn(0x7fa324138ff0 legacy=0x7fa3241cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.311+0000 7fa32bc10640 1 -- 172.21.15.26:0/1151854853 shutdown_connections 2024-09-16T15:02:23.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.311+0000 7fa32bc10640 1 -- 172.21.15.26:0/1151854853 >> 172.21.15.26:0/1151854853 conn(0x7fa32409fb50 msgr2=0x7fa3241317c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:23.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.311+0000 7fa32bc10640 1 -- 172.21.15.26:0/1151854853 shutdown_connections 2024-09-16T15:02:23.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.311+0000 7fa32bc10640 1 -- 172.21.15.26:0/1151854853 wait complete. 2024-09-16T15:02:23.331 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T15:02:23.331 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T15:02:23.343 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T15:02:23.343 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:02:23.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.463+0000 7f589bd00640 10 monclient: get_monmap_and_config 2024-09-16T15:02:23.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.463+0000 7f589bd00640 10 monclient: build_initial_monmap 2024-09-16T15:02:23.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.463+0000 7f589bd00640 10 monclient: monmap: 2024-09-16T15:02:23.466 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:23.466 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.467 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:23.468440+0000 2024-09-16T15:02:23.467 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:23.468440+0000 2024-09-16T15:02:23.467 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:23.467 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.467 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:23.467 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:23.467 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:23.467 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.463+0000 7f589bd00640 1 Processor -- start 2024-09-16T15:02:23.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.463+0000 7f589bd00640 1 -- start start 2024-09-16T15:02:23.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.463+0000 7f589bd00640 10 monclient: init 2024-09-16T15:02:23.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:23.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:02:23.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient(hunting): picked mon.noname-c con 0x7f589409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:23.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient(hunting): picked mon.noname-a con 0x7f5894132650 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:23.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient(hunting): picked mon.noname-b con 0x7f5894138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:23.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589409d260 con 0x7f5894132650 2024-09-16T15:02:23.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5894132a80 con 0x7f589409cb80 2024-09-16T15:02:23.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589409cf60 con 0x7f5894138ff0 2024-09-16T15:02:23.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:23.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient(hunting): authenticate will time out at 2167.588563s 2024-09-16T15:02:23.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589929b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5894132650 0x7f58941378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42146/0 (socket says 172.21.15.26:42146) 2024-09-16T15:02:23.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589929b640 1 -- 172.21.15.26:0/427108058 learned_addr learned my addr 172.21.15.26:0/427108058 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:23.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589a29d640 1 --1- 172.21.15.26:0/427108058 >> v1:172.21.15.26:6790/0 conn(0x7f5894138ff0 0x7f589413b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47874/0 (socket says 172.21.15.26:47874) 2024-09-16T15:02:23.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 1 -- 172.21.15.26:0/427108058 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128282861 0 0) 0x7f589409cf60 con 0x7f5894138ff0 2024-09-16T15:02:23.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient(hunting): _negotiate 2024-09-16T15:02:23.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:23.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:23.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient(hunting): my global_id is 354269 2024-09-16T15:02:23.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 1 -- 172.21.15.26:0/427108058 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5880003530 con 0x7f5894138ff0 2024-09-16T15:02:23.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 1 -- 172.21.15.26:0/427108058 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211981195 0 0) 0x7f589409d260 con 0x7f5894132650 2024-09-16T15:02:23.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient(hunting): _negotiate 2024-09-16T15:02:23.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:23.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:23.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient(hunting): my global_id is 334455 2024-09-16T15:02:23.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 1 -- 172.21.15.26:0/427108058 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f589409cf60 con 0x7f5894132650 2024-09-16T15:02:23.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 1 -- 172.21.15.26:0/427108058 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2824436462 0 0) 0x7f589409cf60 con 0x7f5894132650 2024-09-16T15:02:23.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 1 -- 172.21.15.26:0/427108058 >> v1:172.21.15.26:6790/0 conn(0x7f5894138ff0 legacy=0x7f589413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 1 -- 172.21.15.26:0/427108058 >> v1:172.21.15.189:6789/0 conn(0x7f589409cb80 legacy=0x7f58940717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient: _finish_hunting 0 2024-09-16T15:02:23.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 1 monclient: found mon.noname-a 2024-09-16T15:02:23.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:23.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:02:23.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 1 -- 172.21.15.26:0/427108058 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589409f2d0 con 0x7f5894132650 2024-09-16T15:02:23.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient: _finish_auth 0 2024-09-16T15:02:23.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient: _check_auth_tickets 2024-09-16T15:02:23.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:23.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 1 -- 172.21.15.26:0/427108058 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5884001620 con 0x7f5894132650 2024-09-16T15:02:23.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 5 monclient: authenticate success, global_id 334455 2024-09-16T15:02:23.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:23.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:23.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:23.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient: dump: 2024-09-16T15:02:23.476 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:23.477 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.477 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.477 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.477 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:23.477 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.477 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:23.477 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:23.477 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:23.477 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:23.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 1 -- 172.21.15.26:0/427108058 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5884001840 con 0x7f5894132650 2024-09-16T15:02:23.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:23.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 1 -- 172.21.15.26:0/427108058 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5884001b50 con 0x7f5894132650 2024-09-16T15:02:23.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:23.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 4 set_mon_vals no callback set 2024-09-16T15:02:23.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:23.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:23.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f5898a9a640 10 monclient: dump: 2024-09-16T15:02:23.478 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:23.478 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.478 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.478 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.478 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:23.478 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.478 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:23.478 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:23.478 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:23.479 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient: shutdown 2024-09-16T15:02:23.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:23.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 1 -- 172.21.15.26:0/427108058 >> v1:172.21.15.26:6789/0 conn(0x7f5894132650 legacy=0x7f58941378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 1 -- 172.21.15.26:0/427108058 shutdown_connections 2024-09-16T15:02:23.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 1 -- 172.21.15.26:0/427108058 >> 172.21.15.26:0/427108058 conn(0x7f589409fb50 msgr2=0x7f589409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:23.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 1 -- 172.21.15.26:0/427108058 shutdown_connections 2024-09-16T15:02:23.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 1 -- 172.21.15.26:0/427108058 wait complete. 2024-09-16T15:02:23.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient: build_initial_monmap 2024-09-16T15:02:23.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient: monmap: 2024-09-16T15:02:23.479 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:23.479 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:23.479 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:23.480 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:23.480 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:23.480 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:23.480 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:23.480 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:23.480 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:23.480 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 1 Processor -- start 2024-09-16T15:02:23.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 1 -- start start 2024-09-16T15:02:23.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient: init 2024-09-16T15:02:23.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:23.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:02:23.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient(hunting): picked mon.noname-b con 0x7f589409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:23.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient(hunting): picked mon.noname-a con 0x7f5894132650 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:23.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient(hunting): picked mon.noname-c con 0x7f5894138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:23.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5894071540 con 0x7f5894132650 2024-09-16T15:02:23.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589409b370 con 0x7f589409cb80 2024-09-16T15:02:23.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589409a240 con 0x7f5894138ff0 2024-09-16T15:02:23.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:23.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589bd00640 10 monclient(hunting): authenticate will time out at 2167.591735s 2024-09-16T15:02:23.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589929b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5894132650 0x7f589409ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42152/0 (socket says 172.21.15.26:42152) 2024-09-16T15:02:23.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.467+0000 7f589929b640 1 -- 172.21.15.26:0/523026099 learned_addr learned my addr 172.21.15.26:0/523026099 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:23.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553291630 0 0) 0x7f5894071540 con 0x7f5894132650 2024-09-16T15:02:23.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:02:23.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:23.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:23.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): my global_id is 334458 2024-09-16T15:02:23.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5878003530 con 0x7f5894132650 2024-09-16T15:02:23.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506772130 0 0) 0x7f589409a240 con 0x7f5894138ff0 2024-09-16T15:02:23.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:02:23.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:23.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:23.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): my global_id is 354272 2024-09-16T15:02:23.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5894071540 con 0x7f5894138ff0 2024-09-16T15:02:23.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3940925058 0 0) 0x7f5878003530 con 0x7f5894132650 2024-09-16T15:02:23.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f589409a240 con 0x7f5894132650 2024-09-16T15:02:23.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5884001520 con 0x7f5894132650 2024-09-16T15:02:23.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:23.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:23.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): dump: 2024-09-16T15:02:23.483 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:23.483 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.483 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.483 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.483 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:23.483 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.483 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:23.483 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:23.483 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:23.483 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:23.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2294740234 0 0) 0x7f5894071540 con 0x7f5894138ff0 2024-09-16T15:02:23.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5878003530 con 0x7f5894138ff0 2024-09-16T15:02:23.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5890001100 con 0x7f5894138ff0 2024-09-16T15:02:23.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:23.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:23.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): dump: 2024-09-16T15:02:23.484 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:23.484 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.484 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.484 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.484 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:23.484 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.484 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:23.484 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:23.484 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:23.484 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2141509158 0 0) 0x7f589409b370 con 0x7f589409cb80 2024-09-16T15:02:23.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): _negotiate 2024-09-16T15:02:23.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:23.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:23.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient(hunting): my global_id is 364138 2024-09-16T15:02:23.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5894071540 con 0x7f589409cb80 2024-09-16T15:02:23.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3804950651 0 0) 0x7f589409a240 con 0x7f5894132650 2024-09-16T15:02:23.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 >> v1:172.21.15.26:6790/0 conn(0x7f5894138ff0 legacy=0x7f58941d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 >> v1:172.21.15.189:6789/0 conn(0x7f589409cb80 legacy=0x7f589409a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient: _finish_hunting 0 2024-09-16T15:02:23.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 monclient: found mon.a 2024-09-16T15:02:23.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:23.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:23.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589413c590 con 0x7f5894132650 2024-09-16T15:02:23.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient: _finish_auth 0 2024-09-16T15:02:23.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient: _check_auth_tickets 2024-09-16T15:02:23.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:23.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f589bd00640 5 monclient: authenticate success, global_id 334458 2024-09-16T15:02:23.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f589bd00640 10 monclient: _renew_subs 2024-09-16T15:02:23.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f589bd00640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:23.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f589bd00640 1 -- 172.21.15.26:0/523026099 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5894138600 con 0x7f5894132650 2024-09-16T15:02:23.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f589bd00640 10 monclient: _renew_subs 2024-09-16T15:02:23.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f589bd00640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:23.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f589bd00640 1 -- 172.21.15.26:0/523026099 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5894137cd0 con 0x7f5894132650 2024-09-16T15:02:23.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5884004aa0 con 0x7f5894132650 2024-09-16T15:02:23.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:23.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5884004db0 con 0x7f5894132650 2024-09-16T15:02:23.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:23.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:23.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 10 monclient: dump: 2024-09-16T15:02:23.487 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:23.487 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.487 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.487 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.487 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:23.487 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.487 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:23.487 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:23.487 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:23.487 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588bfff640 4 set_mon_vals no callback set 2024-09-16T15:02:23.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f589bd00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:23.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f589bd00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:23.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f589bd00640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:23.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f588400f820 con 0x7f5894132650 2024-09-16T15:02:23.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f589bd00640 1 -- 172.21.15.26:0/523026099 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f587c005050 con 0x7f5894132650 2024-09-16T15:02:23.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f5884043ed0 con 0x7f5894132650 2024-09-16T15:02:23.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.471+0000 7f5899a9c640 10 monclient: get_auth_request con 0x7f5878036440 auth_method 0 2024-09-16T15:02:23.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.475+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5884010c80 con 0x7f5894132650 2024-09-16T15:02:23.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.475+0000 7f588a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:23.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.475+0000 7f588a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:23.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.631+0000 7f589bd00640 1 -- 172.21.15.26:0/523026099 --> v1:172.21.15.189:6812/3196161582 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f587c000ba0 con 0x7f5878036440 2024-09-16T15:02:23.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.639+0000 7f588a7fc640 1 -- 172.21.15.26:0/523026099 <== mgr.4105 v1:172.21.15.189:6812/3196161582 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+97880 (unknown 2965378022 0 4189929426) 0x7f587c000ba0 con 0x7f5878036440 2024-09-16T15:02:23.644 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T15:02:23.644 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T15:02:23.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.643+0000 7f589bd00640 1 -- 172.21.15.26:0/523026099 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5878036440 legacy=0x7f5878038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.643+0000 7f589bd00640 10 monclient: shutdown 2024-09-16T15:02:23.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.643+0000 7f589bd00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:23.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.643+0000 7f589bd00640 1 -- 172.21.15.26:0/523026099 >> v1:172.21.15.26:6789/0 conn(0x7f5894132650 legacy=0x7f589409ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.643+0000 7f589bd00640 1 -- 172.21.15.26:0/523026099 shutdown_connections 2024-09-16T15:02:23.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.643+0000 7f589bd00640 1 -- 172.21.15.26:0/523026099 >> 172.21.15.26:0/523026099 conn(0x7f589409fb50 msgr2=0x7f58940a0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:23.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.643+0000 7f589bd00640 1 -- 172.21.15.26:0/523026099 shutdown_connections 2024-09-16T15:02:23.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.643+0000 7f589bd00640 1 -- 172.21.15.26:0/523026099 wait complete. 2024-09-16T15:02:23.661 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":841,"stamp":"2024-09-16T15:02:22.725242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1057913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":199188,"kb_used_data":8396,"kb_used_omap":9,"kb_used_meta":190710,"kb_avail":566031852,"statfs":{"total":579820584960,"available":579616616448,"internally_reserved":0,"allocated":8597504,"data_stored":8794066,"data_compressed":1510597,"data_compressed_allocated":2170880,"data_compressed_original":4796416,"omap_allocated":9559,"internal_metadata":195287721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003445"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":164,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:02.474374+0000","last_change":"2024-09-16T14:48:52.143014+0000","last_active":"2024-09-16T15:00:02.474374+0000","last_peered":"2024-09-16T15:00:02.474374+0000","last_clean":"2024-09-16T15:00:02.474374+0000","last_became_active":"2024-09-16T14:48:52.141236+0000","last_became_peered":"2024-09-16T14:48:52.141236+0000","last_unstale":"2024-09-16T15:00:02.474374+0000","last_undegraded":"2024-09-16T15:00:02.474374+0000","last_fullsized":"2024-09-16T15:00:02.474374+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:18:35.458388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":89,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:42.604638+0000","last_change":"2024-09-16T14:48:52.126720+0000","last_active":"2024-09-16T14:59:42.604638+0000","last_peered":"2024-09-16T14:59:42.604638+0000","last_clean":"2024-09-16T14:59:42.604638+0000","last_became_active":"2024-09-16T14:48:52.126325+0000","last_became_peered":"2024-09-16T14:48:52.126325+0000","last_unstale":"2024-09-16T14:59:42.604638+0000","last_undegraded":"2024-09-16T14:59:42.604638+0000","last_fullsized":"2024-09-16T14:59:42.604638+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:23.845897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":156,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:07.474741+0000","last_change":"2024-09-16T14:48:52.138778+0000","last_active":"2024-09-16T15:00:07.474741+0000","last_peered":"2024-09-16T15:00:07.474741+0000","last_clean":"2024-09-16T15:00:07.474741+0000","last_became_active":"2024-09-16T14:48:52.137680+0000","last_became_peered":"2024-09-16T14:48:52.137680+0000","last_unstale":"2024-09-16T15:00:07.474741+0000","last_undegraded":"2024-09-16T15:00:07.474741+0000","last_fullsized":"2024-09-16T15:00:07.474741+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:57.689750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":89,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:42.604588+0000","last_change":"2024-09-16T14:48:52.133563+0000","last_active":"2024-09-16T14:59:42.604588+0000","last_peered":"2024-09-16T14:59:42.604588+0000","last_clean":"2024-09-16T14:59:42.604588+0000","last_became_active":"2024-09-16T14:48:52.133190+0000","last_became_peered":"2024-09-16T14:48:52.133190+0000","last_unstale":"2024-09-16T14:59:42.604588+0000","last_undegraded":"2024-09-16T14:59:42.604588+0000","last_fullsized":"2024-09-16T14:59:42.604588+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:53:36.816994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":124,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:58:22.592610+0000","last_change":"2024-09-16T14:48:53.142970+0000","last_active":"2024-09-16T14:58:22.592610+0000","last_peered":"2024-09-16T14:58:22.592610+0000","last_clean":"2024-09-16T14:58:22.592610+0000","last_became_active":"2024-09-16T14:48:53.142066+0000","last_became_peered":"2024-09-16T14:48:53.142066+0000","last_unstale":"2024-09-16T14:58:22.592610+0000","last_undegraded":"2024-09-16T14:58:22.592610+0000","last_fullsized":"2024-09-16T14:58:22.592610+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:29:40.181990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":143,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:07.475174+0000","last_change":"2024-09-16T14:48:52.128011+0000","last_active":"2024-09-16T15:00:07.475174+0000","last_peered":"2024-09-16T15:00:07.475174+0000","last_clean":"2024-09-16T15:00:07.475174+0000","last_became_active":"2024-09-16T14:48:52.127250+0000","last_became_peered":"2024-09-16T14:48:52.127250+0000","last_unstale":"2024-09-16T15:00:07.475174+0000","last_undegraded":"2024-09-16T15:00:07.475174+0000","last_fullsized":"2024-09-16T15:00:07.475174+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:03:30.725414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":91,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:42.474316+0000","last_change":"2024-09-16T14:48:52.133060+0000","last_active":"2024-09-16T14:59:42.474316+0000","last_peered":"2024-09-16T14:59:42.474316+0000","last_clean":"2024-09-16T14:59:42.474316+0000","last_became_active":"2024-09-16T14:48:52.132725+0000","last_became_peered":"2024-09-16T14:48:52.132725+0000","last_unstale":"2024-09-16T14:59:42.474316+0000","last_undegraded":"2024-09-16T14:59:42.474316+0000","last_fullsized":"2024-09-16T14:59:42.474316+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:55:32.475641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":81,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468949+0000","last_change":"2024-09-16T14:48:52.129972+0000","last_active":"2024-09-16T14:59:17.468949+0000","last_peered":"2024-09-16T14:59:17.468949+0000","last_clean":"2024-09-16T14:59:17.468949+0000","last_became_active":"2024-09-16T14:48:52.128799+0000","last_became_peered":"2024-09-16T14:48:52.128799+0000","last_unstale":"2024-09-16T14:59:17.468949+0000","last_undegraded":"2024-09-16T14:59:17.468949+0000","last_fullsized":"2024-09-16T14:59:17.468949+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:48:37.984512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":111,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:02.607606+0000","last_change":"2024-09-16T14:48:52.143178+0000","last_active":"2024-09-16T15:00:02.607606+0000","last_peered":"2024-09-16T15:00:02.607606+0000","last_clean":"2024-09-16T15:00:02.607606+0000","last_became_active":"2024-09-16T14:48:52.142697+0000","last_became_peered":"2024-09-16T14:48:52.142697+0000","last_unstale":"2024-09-16T15:00:02.607606+0000","last_undegraded":"2024-09-16T15:00:02.607606+0000","last_fullsized":"2024-09-16T15:00:02.607606+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:43:37.007895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":112,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:12.478953+0000","last_change":"2024-09-16T14:48:52.144132+0000","last_active":"2024-09-16T15:00:12.478953+0000","last_peered":"2024-09-16T15:00:12.478953+0000","last_clean":"2024-09-16T15:00:12.478953+0000","last_became_active":"2024-09-16T14:48:52.143756+0000","last_became_peered":"2024-09-16T14:48:52.143756+0000","last_unstale":"2024-09-16T15:00:12.478953+0000","last_undegraded":"2024-09-16T15:00:12.478953+0000","last_fullsized":"2024-09-16T15:00:12.478953+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:15.504836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":126,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:02.600738+0000","last_change":"2024-09-16T14:48:52.139682+0000","last_active":"2024-09-16T15:00:02.600738+0000","last_peered":"2024-09-16T15:00:02.600738+0000","last_clean":"2024-09-16T15:00:02.600738+0000","last_became_active":"2024-09-16T14:48:52.139205+0000","last_became_peered":"2024-09-16T14:48:52.139205+0000","last_unstale":"2024-09-16T15:00:02.600738+0000","last_undegraded":"2024-09-16T15:00:02.600738+0000","last_fullsized":"2024-09-16T15:00:02.600738+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:56.518201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":81,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468831+0000","last_change":"2024-09-16T14:48:52.131936+0000","last_active":"2024-09-16T14:59:17.468831+0000","last_peered":"2024-09-16T14:59:17.468831+0000","last_clean":"2024-09-16T14:59:17.468831+0000","last_became_active":"2024-09-16T14:48:52.131089+0000","last_became_peered":"2024-09-16T14:48:52.131089+0000","last_unstale":"2024-09-16T14:59:17.468831+0000","last_undegraded":"2024-09-16T14:59:17.468831+0000","last_fullsized":"2024-09-16T14:59:17.468831+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:04:08.492279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":82,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468863+0000","last_change":"2024-09-16T14:48:52.138889+0000","last_active":"2024-09-16T14:59:17.468863+0000","last_peered":"2024-09-16T14:59:17.468863+0000","last_clean":"2024-09-16T14:59:17.468863+0000","last_became_active":"2024-09-16T14:48:52.137830+0000","last_became_peered":"2024-09-16T14:48:52.137830+0000","last_unstale":"2024-09-16T14:59:17.468863+0000","last_undegraded":"2024-09-16T14:59:17.468863+0000","last_fullsized":"2024-09-16T14:59:17.468863+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:00:48.737166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":169,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:02.474335+0000","last_change":"2024-09-16T14:48:52.145132+0000","last_active":"2024-09-16T15:00:02.474335+0000","last_peered":"2024-09-16T15:00:02.474335+0000","last_clean":"2024-09-16T15:00:02.474335+0000","last_became_active":"2024-09-16T14:48:52.144280+0000","last_became_peered":"2024-09-16T14:48:52.144280+0000","last_unstale":"2024-09-16T15:00:02.474335+0000","last_undegraded":"2024-09-16T15:00:02.474335+0000","last_fullsized":"2024-09-16T15:00:02.474335+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:37:18.905007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":90,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.467831+0000","last_change":"2024-09-16T14:48:53.564947+0000","last_active":"2024-09-16T14:59:17.467831+0000","last_peered":"2024-09-16T14:59:17.467831+0000","last_clean":"2024-09-16T14:59:17.467831+0000","last_became_active":"2024-09-16T14:48:53.563655+0000","last_became_peered":"2024-09-16T14:48:53.563655+0000","last_unstale":"2024-09-16T14:59:17.467831+0000","last_undegraded":"2024-09-16T14:59:17.467831+0000","last_fullsized":"2024-09-16T14:59:17.467831+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:32:48.022677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":81,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:58:57.465826+0000","last_change":"2024-09-16T14:48:52.138403+0000","last_active":"2024-09-16T14:58:57.465826+0000","last_peered":"2024-09-16T14:58:57.465826+0000","last_clean":"2024-09-16T14:58:57.465826+0000","last_became_active":"2024-09-16T14:48:52.137512+0000","last_became_peered":"2024-09-16T14:48:52.137512+0000","last_unstale":"2024-09-16T14:58:57.465826+0000","last_undegraded":"2024-09-16T14:58:57.465826+0000","last_fullsized":"2024-09-16T14:58:57.465826+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:00:15.913134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":229,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:58:57.465176+0000","last_change":"2024-09-16T14:48:52.131578+0000","last_active":"2024-09-16T14:58:57.465176+0000","last_peered":"2024-09-16T14:58:57.465176+0000","last_clean":"2024-09-16T14:58:57.465176+0000","last_became_active":"2024-09-16T14:48:52.131259+0000","last_became_peered":"2024-09-16T14:48:52.131259+0000","last_unstale":"2024-09-16T14:58:57.465176+0000","last_undegraded":"2024-09-16T14:58:57.465176+0000","last_fullsized":"2024-09-16T14:58:57.465176+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:02:29.950099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067836399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":220,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.600970+0000","last_change":"2024-09-16T14:48:53.141414+0000","last_active":"2024-09-16T14:59:17.600970+0000","last_peered":"2024-09-16T14:59:17.600970+0000","last_clean":"2024-09-16T14:59:17.600970+0000","last_became_active":"2024-09-16T14:48:53.140914+0000","last_became_peered":"2024-09-16T14:48:53.140914+0000","last_unstale":"2024-09-16T14:59:17.600970+0000","last_undegraded":"2024-09-16T14:59:17.600970+0000","last_fullsized":"2024-09-16T14:59:17.600970+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:53:48.881111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097689999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.594663+0000","last_change":"2024-09-16T14:48:52.149868+0000","last_active":"2024-09-16T14:59:17.594663+0000","last_peered":"2024-09-16T14:59:17.594663+0000","last_clean":"2024-09-16T14:59:17.594663+0000","last_became_active":"2024-09-16T14:48:52.142312+0000","last_became_peered":"2024-09-16T14:48:52.142312+0000","last_unstale":"2024-09-16T14:59:17.594663+0000","last_undegraded":"2024-09-16T14:59:17.594663+0000","last_fullsized":"2024-09-16T14:59:17.594663+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:31:20.443534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":87,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468897+0000","last_change":"2024-09-16T14:48:52.151300+0000","last_active":"2024-09-16T14:59:17.468897+0000","last_peered":"2024-09-16T14:59:17.468897+0000","last_clean":"2024-09-16T14:59:17.468897+0000","last_became_active":"2024-09-16T14:48:52.137048+0000","last_became_peered":"2024-09-16T14:48:52.137048+0000","last_unstale":"2024-09-16T14:59:17.468897+0000","last_undegraded":"2024-09-16T14:59:17.468897+0000","last_fullsized":"2024-09-16T14:59:17.468897+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:17:13.952991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042440799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":428,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:58:37.462283+0000","last_change":"2024-09-16T14:48:55.155094+0000","last_active":"2024-09-16T14:58:37.462283+0000","last_peered":"2024-09-16T14:58:37.462283+0000","last_clean":"2024-09-16T14:58:37.462283+0000","last_became_active":"2024-09-16T14:48:55.154511+0000","last_became_peered":"2024-09-16T14:48:55.154511+0000","last_unstale":"2024-09-16T14:58:37.462283+0000","last_undegraded":"2024-09-16T14:58:37.462283+0000","last_fullsized":"2024-09-16T14:58:37.462283+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:28.462531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:28.462531+0000","last_clean_scrub_stamp":"2024-09-16T14:42:28.462531+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:06.351384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":81,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468924+0000","last_change":"2024-09-16T14:48:52.129414+0000","last_active":"2024-09-16T14:59:17.468924+0000","last_peered":"2024-09-16T14:59:17.468924+0000","last_clean":"2024-09-16T14:59:17.468924+0000","last_became_active":"2024-09-16T14:48:52.128554+0000","last_became_peered":"2024-09-16T14:48:52.128554+0000","last_unstale":"2024-09-16T14:59:17.468924+0000","last_undegraded":"2024-09-16T14:59:17.468924+0000","last_fullsized":"2024-09-16T14:59:17.468924+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:38:09.329918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039796999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":89,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.594692+0000","last_change":"2024-09-16T14:48:52.144047+0000","last_active":"2024-09-16T14:59:17.594692+0000","last_peered":"2024-09-16T14:59:17.594692+0000","last_clean":"2024-09-16T14:59:17.594692+0000","last_became_active":"2024-09-16T14:48:52.143705+0000","last_became_peered":"2024-09-16T14:48:52.143705+0000","last_unstale":"2024-09-16T14:59:17.594692+0000","last_undegraded":"2024-09-16T14:59:17.594692+0000","last_fullsized":"2024-09-16T14:59:17.594692+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:14:59.892564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061478100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":90,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:42.474287+0000","last_change":"2024-09-16T14:48:52.131363+0000","last_active":"2024-09-16T14:59:42.474287+0000","last_peered":"2024-09-16T14:59:42.474287+0000","last_clean":"2024-09-16T14:59:42.474287+0000","last_became_active":"2024-09-16T14:48:52.131160+0000","last_became_peered":"2024-09-16T14:48:52.131160+0000","last_unstale":"2024-09-16T14:59:42.474287+0000","last_undegraded":"2024-09-16T14:59:42.474287+0000","last_fullsized":"2024-09-16T14:59:42.474287+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:12.859518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065779100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":226,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:52.606162+0000","last_change":"2024-09-16T14:48:52.133533+0000","last_active":"2024-09-16T14:59:52.606162+0000","last_peered":"2024-09-16T14:59:52.606162+0000","last_clean":"2024-09-16T14:59:52.606162+0000","last_became_active":"2024-09-16T14:48:52.133197+0000","last_became_peered":"2024-09-16T14:48:52.133197+0000","last_unstale":"2024-09-16T14:59:52.606162+0000","last_undegraded":"2024-09-16T14:59:52.606162+0000","last_fullsized":"2024-09-16T14:59:52.606162+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:37:19.055419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085482099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":94400,"data_compressed_allocated":122880,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640499,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1413120,"data_stored":1289362,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.622,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.92600000000000005,"5min":1.042,"15min":1.2},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.81499999999999995,"15min":0.79100000000000004},"min":{"1min":0.627,"5min":0.51500000000000001,"15min":0.47899999999999998},"max":{"1min":0.95599999999999996,"5min":1.0369999999999999,"15min":1.3300000000000001},"last":0.95599999999999996}]},{"osd":1,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.81799999999999995,"15min":0.81000000000000005},"min":{"1min":0.61899999999999999,"5min":0.44900000000000001,"15min":0.33900000000000002},"max":{"1min":1.0589999999999999,"5min":1.073,"15min":1.2010000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.74399999999999999,"15min":0.752},"min":{"1min":0.60999999999999999,"5min":0.54000000000000004,"15min":0.438},"max":{"1min":0.90800000000000003,"5min":0.94299999999999995,"15min":1.089},"last":0.77300000000000002}]},{"osd":2,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.81799999999999995,"15min":0.82899999999999996},"min":{"1min":0.628,"5min":0.50800000000000001,"15min":0.44900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.272},"last":0.75},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.90600000000000003,"15min":0.91300000000000003},"min":{"1min":0.81000000000000005,"5min":0.52900000000000003,"15min":0.42899999999999999},"max":{"1min":1.0289999999999999,"5min":1.056,"15min":1.3540000000000001},"last":0.93000000000000005}]},{"osd":3,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69699999999999995,"15min":0.67600000000000005},"min":{"1min":0.52300000000000002,"5min":0.46600000000000003,"15min":0.26000000000000001},"max":{"1min":0.96799999999999997,"5min":0.96999999999999997,"15min":1.1020000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.68200000000000005,"15min":0.69099999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33500000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":1.153},"last":0.90000000000000002}]},{"osd":4,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72899999999999998,"15min":0.749},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.27500000000000002},"max":{"1min":0.95299999999999996,"5min":1.05,"15min":1.236},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.80000000000000004,"15min":0.77200000000000002},"min":{"1min":0.48799999999999999,"5min":0.44800000000000001,"15min":0.26300000000000001},"max":{"1min":1.038,"5min":1.0529999999999999,"15min":1.2949999999999999},"last":1.01}]}]},{"osd":4,"up_from":11,"seq":47244640499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1417216,"data_stored":1289381,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78200000000000003,"15min":0.77300000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.437},"max":{"1min":0.96699999999999997,"5min":1.0800000000000001,"15min":1.1579999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73899999999999999,"15min":0.79000000000000004},"min":{"1min":0.52600000000000002,"5min":0.48899999999999999,"15min":0.48499999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.1679999999999999},"last":0.73999999999999999}]},{"osd":1,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.85899999999999999,"15min":0.81599999999999995},"min":{"1min":0.71199999999999997,"5min":0.52800000000000002,"15min":0.42399999999999999},"max":{"1min":1.0429999999999999,"5min":1.0780000000000001,"15min":1.4630000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.79600000000000004,"15min":0.83399999999999996},"min":{"1min":0.65700000000000003,"5min":0.53700000000000003,"15min":0.50800000000000001},"max":{"1min":1.0149999999999999,"5min":1.024,"15min":1.2729999999999999},"last":0.92300000000000004}]},{"osd":2,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.84999999999999998,"15min":0.84899999999999998},"min":{"1min":0.68000000000000005,"5min":0.65500000000000003,"15min":0.47599999999999998},"max":{"1min":1.02,"5min":1.0449999999999999,"15min":1.536},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84799999999999998,"15min":0.85899999999999999},"min":{"1min":0.63500000000000001,"5min":0.56599999999999995,"15min":0.51200000000000001},"max":{"1min":0.97099999999999997,"5min":1.0720000000000001,"15min":1.29},"last":0.79300000000000004}]},{"osd":3,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.70999999999999996,"15min":0.68899999999999995},"min":{"1min":0.53700000000000003,"5min":0.51300000000000001,"15min":0.27700000000000002},"max":{"1min":0.83299999999999996,"5min":1.0720000000000001,"15min":1.2669999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72899999999999998,"15min":0.72499999999999998},"min":{"1min":0.55700000000000005,"5min":0.50600000000000001,"15min":0.25700000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.198},"last":0.69799999999999995}]},{"osd":5,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.75600000000000001,"15min":0.73099999999999998},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.28999999999999998},"max":{"1min":0.95799999999999996,"5min":1.0600000000000001,"15min":1.1140000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.94699999999999995,"15min":0.91700000000000004},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.24299999999999999},"max":{"1min":1.0700000000000001,"5min":1.169,"15min":1.3879999999999999},"last":0.96599999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640498,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32556,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339284,"statfs":{"total":96636764160,"available":96603426816,"internally_reserved":0,"allocated":1474560,"data_stored":1818290,"data_compressed":271867,"data_compressed_allocated":389120,"data_compressed_original":1138688,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.45200000000000001},"max":{"1min":0.998,"5min":0.998,"15min":1.2729999999999999},"last":0.622},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74099999999999999,"15min":0.753},"min":{"1min":0.60799999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90500000000000003,"5min":0.94199999999999995,"15min":1.306},"last":0.83499999999999996}]},{"osd":1,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90200000000000002,"15min":0.86599999999999999},"min":{"1min":0.55800000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.069,"5min":1.1259999999999999,"15min":1.294},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.86199999999999999,"15min":0.86599999999999999},"min":{"1min":0.67900000000000005,"5min":0.56200000000000006,"15min":0.41199999999999998},"max":{"1min":0.93999999999999995,"5min":1.046,"15min":1.9219999999999999},"last":0.72899999999999998}]},{"osd":2,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81799999999999995,"15min":0.81699999999999995},"min":{"1min":0.71299999999999997,"5min":0.61199999999999999,"15min":0.45900000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.181},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79900000000000004,"15min":0.80600000000000005},"min":{"1min":0.627,"5min":0.627,"15min":0.38800000000000001},"max":{"1min":0.90300000000000002,"5min":0.96899999999999997,"15min":2.444},"last":0.78000000000000003}]},{"osd":4,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.624,"15min":0.623},"min":{"1min":0.49299999999999999,"5min":0.47699999999999998,"15min":0.28199999999999997},"max":{"1min":0.89800000000000002,"5min":0.98799999999999999,"15min":1.0700000000000001},"last":0.75},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.67900000000000005,"5min":0.628,"15min":0.317},"max":{"1min":0.99299999999999999,"5min":1.0740000000000001,"15min":1.2290000000000001},"last":0.92000000000000004}]},{"osd":5,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69299999999999995,"15min":0.70699999999999996},"min":{"1min":0.55500000000000005,"5min":0.52400000000000002,"15min":0.29899999999999999},"max":{"1min":0.92200000000000004,"5min":1.4279999999999999,"15min":2.5409999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.83499999999999996,"15min":0.80900000000000005},"min":{"1min":0.63700000000000001,"5min":0.60999999999999999,"15min":0.33100000000000002},"max":{"1min":1.081,"5min":1.514,"15min":1.514},"last":0.871}]}]},{"osd":2,"up_from":11,"seq":47244640498,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1413120,"data_stored":1289362,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56200000000000006,"15min":0.58099999999999996},"min":{"1min":0.252,"5min":0.252,"15min":0.249},"max":{"1min":0.749,"5min":0.94699999999999995,"15min":1.819},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.77600000000000002,"15min":0.75700000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.312},"max":{"1min":1.147,"5min":1.147,"15min":1.444},"last":0.91900000000000004}]},{"osd":1,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.39200000000000002,"5min":0.379,"15min":0.28299999999999997},"max":{"1min":0.95499999999999996,"5min":0.95999999999999996,"15min":1.27},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.62,"15min":0.623},"min":{"1min":0.313,"5min":0.313,"15min":0.26700000000000002},"max":{"1min":1.0129999999999999,"5min":1.0189999999999999,"15min":1.383},"last":0.68000000000000005}]},{"osd":3,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.84199999999999997,"15min":0.82299999999999995},"min":{"1min":0.58599999999999997,"5min":0.56000000000000005,"15min":0.36899999999999999},"max":{"1min":1.0720000000000001,"5min":1.169,"15min":1.2110000000000001},"last":1.034},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.871,"15min":0.86299999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.44700000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.335},"last":0.72399999999999998}]},{"osd":4,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.83599999999999997,"15min":0.83499999999999996},"min":{"1min":0.53300000000000003,"5min":0.50600000000000001,"15min":0.45500000000000002},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.874},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.84999999999999998,"15min":0.85499999999999998},"min":{"1min":0.62,"5min":0.58599999999999997,"15min":0.40999999999999998},"max":{"1min":0.97099999999999997,"5min":1.0489999999999999,"15min":1.367},"last":0.84299999999999997}]},{"osd":5,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.88200000000000001,"15min":0.89100000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.53700000000000003},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.911},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.95399999999999996,"15min":0.95499999999999996},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.121,"5min":1.1990000000000001,"15min":2.0499999999999998},"last":0.90800000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640497,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1462272,"data_stored":1818290,"data_compressed":273302,"data_compressed_allocated":389120,"data_compressed_original":1150976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56000000000000005,"15min":0.57099999999999995},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.251},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":1.0720000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.628,"15min":0.63800000000000001},"min":{"1min":0.44500000000000001,"5min":0.36199999999999999,"15min":0.26100000000000001},"max":{"1min":0.87,"5min":0.87,"15min":1.0589999999999999},"last":0.81499999999999995}]},{"osd":2,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.66700000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.28699999999999998},"max":{"1min":0.82599999999999996,"5min":0.88200000000000001,"15min":1.052},"last":0.748},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.628,"15min":0.66700000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32100000000000001},"max":{"1min":0.81899999999999995,"5min":0.879,"15min":1.403},"last":0.83799999999999997}]},{"osd":3,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85699999999999998,"15min":0.88800000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.503},"max":{"1min":1.052,"5min":1.117,"15min":1.359},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.86299999999999999,"15min":0.877},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1459999999999999,"5min":1.147,"15min":1.284},"last":1.1719999999999999}]},{"osd":4,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.88900000000000001,"15min":0.88200000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.46600000000000003},"max":{"1min":1.0900000000000001,"5min":1.097,"15min":1.341},"last":1.111},{"interface":"front","average":{"1min":0.88,"5min":0.86199999999999999,"15min":0.86899999999999999},"min":{"1min":0.67600000000000005,"5min":0.66200000000000003,"15min":0.46999999999999997},"max":{"1min":1.056,"5min":1.0800000000000001,"15min":1.399},"last":1.0269999999999999}]},{"osd":5,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.622,"5min":0.622,"15min":0.50900000000000001},"max":{"1min":1,"5min":1.01,"15min":1.258},"last":1.054},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90100000000000002,"15min":0.90400000000000003},"min":{"1min":0.69499999999999995,"5min":0.629,"15min":0.47299999999999998},"max":{"1min":1.1519999999999999,"5min":1.411,"15min":1.411},"last":1.087}]}]},{"osd":0,"up_from":11,"seq":47244640498,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36596,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335244,"statfs":{"total":96636764160,"available":96599289856,"internally_reserved":0,"allocated":1417216,"data_stored":1289381,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57499999999999996,"15min":0.57999999999999996},"min":{"1min":0.36099999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.74299999999999999,"5min":0.77700000000000002,"15min":1.1739999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.61399999999999999,"15min":0.63800000000000001},"min":{"1min":0.38200000000000001,"5min":0.28299999999999997,"15min":0.27400000000000002},"max":{"1min":0.67800000000000005,"5min":1.032,"15min":1.21},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68700000000000006,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.29899999999999999,"15min":0.25900000000000001},"max":{"1min":0.85799999999999998,"5min":0.997,"15min":1.103},"last":0.495},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.62,"15min":0.67200000000000004},"min":{"1min":0.47999999999999998,"5min":0.36099999999999999,"15min":0.26900000000000002},"max":{"1min":0.69099999999999995,"5min":1.008,"15min":1.137},"last":0.44}]},{"osd":3,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.82599999999999996},"min":{"1min":0.67100000000000004,"5min":0.59599999999999997,"15min":0.52400000000000002},"max":{"1min":1.014,"5min":1.1040000000000001,"15min":1.4650000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.81799999999999995,"15min":0.83999999999999997},"min":{"1min":0.69299999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.99299999999999999,"5min":1.101,"15min":1.4950000000000001},"last":0.59199999999999997}]},{"osd":4,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.84599999999999997,"15min":0.84699999999999998},"min":{"1min":0.68899999999999995,"5min":0.57999999999999996,"15min":0.48999999999999999},"max":{"1min":0.98499999999999999,"5min":1.1559999999999999,"15min":1.3300000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.85099999999999998},"min":{"1min":0.63700000000000001,"5min":0.55700000000000005,"15min":0.51600000000000001},"max":{"1min":0.96099999999999997,"5min":1.095,"15min":1.4179999999999999},"last":0.46999999999999997}]},{"osd":5,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.91200000000000003,"15min":0.91400000000000003},"min":{"1min":0.70899999999999996,"5min":0.66400000000000003,"15min":0.57999999999999996},"max":{"1min":1.1080000000000001,"5min":1.304,"15min":1.6499999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95799999999999996,"15min":0.94799999999999995},"min":{"1min":0.76900000000000002,"5min":0.69999999999999996,"15min":0.51900000000000002},"max":{"1min":1.0589999999999999,"5min":1.2030000000000001,"15min":1.7290000000000001},"last":0.754}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:02:23.663 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:02:23.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.783+0000 7fab0c399640 10 monclient: get_monmap_and_config 2024-09-16T15:02:23.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.783+0000 7fab0c399640 10 monclient: build_initial_monmap 2024-09-16T15:02:23.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.783+0000 7fab0c399640 10 monclient: monmap: 2024-09-16T15:02:23.786 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:23.786 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.787 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:23.788296+0000 2024-09-16T15:02:23.787 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:23.788296+0000 2024-09-16T15:02:23.787 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:23.787 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.787 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:23.787 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:23.787 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:23.787 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.783+0000 7fab0c399640 1 Processor -- start 2024-09-16T15:02:23.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.783+0000 7fab0c399640 1 -- start start 2024-09-16T15:02:23.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.783+0000 7fab0c399640 10 monclient: init 2024-09-16T15:02:23.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:23.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:02:23.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient(hunting): picked mon.noname-c con 0x7fab040a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient(hunting): picked mon.noname-b con 0x7fab040a0000 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient(hunting): picked mon.noname-a con 0x7fab040a2800 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab040a3250 con 0x7fab040a2800 2024-09-16T15:02:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab0409f600 con 0x7fab040a1d10 2024-09-16T15:02:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab040a3830 con 0x7fab040a0000 2024-09-16T15:02:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient(hunting): authenticate will time out at 2167.908373s 2024-09-16T15:02:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0a936640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fab040a2800 0x7fab0413d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42164/0 (socket says 172.21.15.26:42164) 2024-09-16T15:02:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09934640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fab040a0000 0x7fab04139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47894/0 (socket says 172.21.15.26:47894) 2024-09-16T15:02:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0a936640 1 -- 172.21.15.26:0/2046744045 learned_addr learned my addr 172.21.15.26:0/2046744045 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 1 -- 172.21.15.26:0/2046744045 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881889211 0 0) 0x7fab040a3250 con 0x7fab040a2800 2024-09-16T15:02:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient(hunting): _negotiate 2024-09-16T15:02:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient(hunting): my global_id is 334461 2024-09-16T15:02:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 1 -- 172.21.15.26:0/2046744045 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaf0003530 con 0x7fab040a2800 2024-09-16T15:02:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 1 -- 172.21.15.26:0/2046744045 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387704251 0 0) 0x7fab040a3830 con 0x7fab040a0000 2024-09-16T15:02:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient(hunting): _negotiate 2024-09-16T15:02:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient(hunting): my global_id is 354275 2024-09-16T15:02:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 1 -- 172.21.15.26:0/2046744045 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab040a3250 con 0x7fab040a0000 2024-09-16T15:02:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 1 -- 172.21.15.26:0/2046744045 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2496665313 0 0) 0x7faaf0003530 con 0x7fab040a2800 2024-09-16T15:02:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 1 -- 172.21.15.26:0/2046744045 >> v1:172.21.15.26:6790/0 conn(0x7fab040a0000 legacy=0x7fab04139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 1 -- 172.21.15.26:0/2046744045 >> v1:172.21.15.189:6789/0 conn(0x7fab040a1d10 legacy=0x7fab040a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient: _finish_hunting 0 2024-09-16T15:02:23.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 1 monclient: found mon.noname-a 2024-09-16T15:02:23.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:23.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:02:23.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 1 -- 172.21.15.26:0/2046744045 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0409ed60 con 0x7fab040a2800 2024-09-16T15:02:23.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient: _finish_auth 0 2024-09-16T15:02:23.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient: _check_auth_tickets 2024-09-16T15:02:23.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:23.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 1 -- 172.21.15.26:0/2046744045 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faafc0019a0 con 0x7fab040a2800 2024-09-16T15:02:23.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:23.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:23.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient: dump: 2024-09-16T15:02:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:23.798 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 5 monclient: authenticate success, global_id 334461 2024-09-16T15:02:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 1 -- 172.21.15.26:0/2046744045 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faafc001bc0 con 0x7fab040a2800 2024-09-16T15:02:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 1 -- 172.21.15.26:0/2046744045 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faafc001f50 con 0x7fab040a2800 2024-09-16T15:02:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab09133640 10 monclient: dump: 2024-09-16T15:02:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:23.799 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:23.800 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:23.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 4 set_mon_vals no callback set 2024-09-16T15:02:23.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient: shutdown 2024-09-16T15:02:23.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:23.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 1 -- 172.21.15.26:0/2046744045 >> v1:172.21.15.26:6789/0 conn(0x7fab040a2800 legacy=0x7fab0413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 1 -- 172.21.15.26:0/2046744045 shutdown_connections 2024-09-16T15:02:23.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 1 -- 172.21.15.26:0/2046744045 >> 172.21.15.26:0/2046744045 conn(0x7fab040a7f30 msgr2=0x7fab040a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:23.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 1 -- 172.21.15.26:0/2046744045 shutdown_connections 2024-09-16T15:02:23.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 1 -- 172.21.15.26:0/2046744045 wait complete. 2024-09-16T15:02:23.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient: build_initial_monmap 2024-09-16T15:02:23.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient: monmap: 2024-09-16T15:02:23.801 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:23.801 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:23.801 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:23.801 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:23.801 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:23.801 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:23.801 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:23.801 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:23.801 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:23.801 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 1 Processor -- start 2024-09-16T15:02:23.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 1 -- start start 2024-09-16T15:02:23.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient: init 2024-09-16T15:02:23.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:23.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:02:23.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient(hunting): picked mon.noname-a con 0x7fab040a0000 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:23.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient(hunting): picked mon.noname-c con 0x7fab040a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:23.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient(hunting): picked mon.noname-b con 0x7fab040a2800 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:23.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab040a6210 con 0x7fab040a0000 2024-09-16T15:02:23.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab040a6760 con 0x7fab040a2800 2024-09-16T15:02:23.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab0409e500 con 0x7fab040a1d10 2024-09-16T15:02:23.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:23.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0c399640 10 monclient(hunting): authenticate will time out at 2167.911500s 2024-09-16T15:02:23.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0a135640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fab040a0000 0x7fab0413d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42178/0 (socket says 172.21.15.26:42178) 2024-09-16T15:02:23.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7fab0a135640 1 -- 172.21.15.26:0/3616403162 learned_addr learned my addr 172.21.15.26:0/3616403162 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:23.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26402635 0 0) 0x7fab0409e500 con 0x7fab040a1d10 2024-09-16T15:02:23.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7faafaffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:23.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7faafaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:23.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7faafaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:23.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7faafaffd640 10 monclient(hunting): my global_id is 354278 2024-09-16T15:02:23.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faae8003530 con 0x7fab040a1d10 2024-09-16T15:02:23.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182232488 0 0) 0x7fab040a6210 con 0x7fab040a0000 2024-09-16T15:02:23.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7faafaffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:23.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7faafaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:23.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7faafaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:23.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7faafaffd640 10 monclient(hunting): my global_id is 334464 2024-09-16T15:02:23.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.787+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab0409e500 con 0x7fab040a0000 2024-09-16T15:02:23.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2638413090 0 0) 0x7faae8003530 con 0x7fab040a1d10 2024-09-16T15:02:23.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab040a6210 con 0x7fab040a1d10 2024-09-16T15:02:23.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faaf40017e0 con 0x7fab040a1d10 2024-09-16T15:02:23.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:23.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:02:23.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient(hunting): dump: 2024-09-16T15:02:23.804 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:23.804 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.805 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.805 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.805 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:23.805 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.805 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:23.805 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:23.805 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:23.805 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:23.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931610421 0 0) 0x7fab040a6760 con 0x7fab040a2800 2024-09-16T15:02:23.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:23.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:23.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:23.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient(hunting): my global_id is 364144 2024-09-16T15:02:23.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faae8003530 con 0x7fab040a2800 2024-09-16T15:02:23.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1140440596 0 0) 0x7fab0409e500 con 0x7fab040a0000 2024-09-16T15:02:23.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab040a6760 con 0x7fab040a0000 2024-09-16T15:02:23.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fab00001950 con 0x7fab040a0000 2024-09-16T15:02:23.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:23.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:23.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient(hunting): dump: 2024-09-16T15:02:23.806 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:23.806 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.807 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.807 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.807 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:23.807 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.807 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:23.807 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:23.807 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:23.807 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 376993030 0 0) 0x7fab040a6210 con 0x7fab040a1d10 2024-09-16T15:02:23.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 >> v1:172.21.15.189:6789/0 conn(0x7fab040a2800 legacy=0x7fab041d18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 >> v1:172.21.15.26:6789/0 conn(0x7fab040a0000 legacy=0x7fab0413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient: _finish_hunting 0 2024-09-16T15:02:23.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 monclient: found mon.c 2024-09-16T15:02:23.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:23.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:23.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab04136b80 con 0x7fab040a1d10 2024-09-16T15:02:23.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient: _finish_auth 0 2024-09-16T15:02:23.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient: _check_auth_tickets 2024-09-16T15:02:23.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:23.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7fab0c399640 5 monclient: authenticate success, global_id 354278 2024-09-16T15:02:23.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7fab0c399640 10 monclient: _renew_subs 2024-09-16T15:02:23.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7fab0c399640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:23.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7fab0c399640 1 -- 172.21.15.26:0/3616403162 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab04135650 con 0x7fab040a1d10 2024-09-16T15:02:23.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7fab0c399640 10 monclient: _renew_subs 2024-09-16T15:02:23.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7fab0c399640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7fab0c399640 1 -- 172.21.15.26:0/3616403162 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab0413daf0 con 0x7fab040a1d10 2024-09-16T15:02:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faaf4001d10 con 0x7fab040a1d10 2024-09-16T15:02:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faaf4004990 con 0x7fab040a1d10 2024-09-16T15:02:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7fab08932640 4 set_mon_vals no callback set 2024-09-16T15:02:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 10 monclient: dump: 2024-09-16T15:02:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:23.809 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:23.810 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:23.810 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:23.810 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:23.810 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:23.810 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:23.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7fab0c399640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:23.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7fab0c399640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:23.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7fab0c399640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:23.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7fab0c399640 1 -- 172.21.15.26:0/3616403162 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaec005050 con 0x7fab040a1d10 2024-09-16T15:02:23.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7faaf400f400 con 0x7fab040a1d10 2024-09-16T15:02:23.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7faaf4043bf0 con 0x7fab040a1d10 2024-09-16T15:02:23.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.791+0000 7fab0a135640 10 monclient: get_auth_request con 0x7faae8036490 auth_method 0 2024-09-16T15:02:23.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.795+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7faaf40109a0 con 0x7fab040a1d10 2024-09-16T15:02:23.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.795+0000 7faafaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:23.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.795+0000 7faafaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:23.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.951+0000 7fab0c399640 1 -- 172.21.15.26:0/3616403162 --> v1:172.21.15.189:6812/3196161582 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faaec000ba0 con 0x7faae8036490 2024-09-16T15:02:23.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.959+0000 7faafaffd640 1 -- 172.21.15.26:0/3616403162 <== mgr.4105 v1:172.21.15.189:6812/3196161582 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+97880 (unknown 2965378022 0 4189929426) 0x7faaec000ba0 con 0x7faae8036490 2024-09-16T15:02:23.963 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T15:02:23.963 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T15:02:23.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.963+0000 7fab0c399640 1 -- 172.21.15.26:0/3616403162 >> v1:172.21.15.189:6812/3196161582 conn(0x7faae8036490 legacy=0x7faae8038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.963+0000 7fab0c399640 10 monclient: shutdown 2024-09-16T15:02:23.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.963+0000 7fab0c399640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:23.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.963+0000 7fab0c399640 1 -- 172.21.15.26:0/3616403162 >> v1:172.21.15.26:6790/0 conn(0x7fab040a1d10 legacy=0x7fab041ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:23.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.963+0000 7fab0c399640 1 -- 172.21.15.26:0/3616403162 shutdown_connections 2024-09-16T15:02:23.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.963+0000 7fab0c399640 1 -- 172.21.15.26:0/3616403162 >> 172.21.15.26:0/3616403162 conn(0x7fab040a7f30 msgr2=0x7fab0406b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:23.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.963+0000 7fab0c399640 1 -- 172.21.15.26:0/3616403162 shutdown_connections 2024-09-16T15:02:23.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:23.963+0000 7fab0c399640 1 -- 172.21.15.26:0/3616403162 wait complete. 2024-09-16T15:02:23.981 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":841,"stamp":"2024-09-16T15:02:22.725242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1057913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":199188,"kb_used_data":8396,"kb_used_omap":9,"kb_used_meta":190710,"kb_avail":566031852,"statfs":{"total":579820584960,"available":579616616448,"internally_reserved":0,"allocated":8597504,"data_stored":8794066,"data_compressed":1510597,"data_compressed_allocated":2170880,"data_compressed_original":4796416,"omap_allocated":9559,"internal_metadata":195287721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003445"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":164,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:02.474374+0000","last_change":"2024-09-16T14:48:52.143014+0000","last_active":"2024-09-16T15:00:02.474374+0000","last_peered":"2024-09-16T15:00:02.474374+0000","last_clean":"2024-09-16T15:00:02.474374+0000","last_became_active":"2024-09-16T14:48:52.141236+0000","last_became_peered":"2024-09-16T14:48:52.141236+0000","last_unstale":"2024-09-16T15:00:02.474374+0000","last_undegraded":"2024-09-16T15:00:02.474374+0000","last_fullsized":"2024-09-16T15:00:02.474374+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:18:35.458388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":89,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:42.604638+0000","last_change":"2024-09-16T14:48:52.126720+0000","last_active":"2024-09-16T14:59:42.604638+0000","last_peered":"2024-09-16T14:59:42.604638+0000","last_clean":"2024-09-16T14:59:42.604638+0000","last_became_active":"2024-09-16T14:48:52.126325+0000","last_became_peered":"2024-09-16T14:48:52.126325+0000","last_unstale":"2024-09-16T14:59:42.604638+0000","last_undegraded":"2024-09-16T14:59:42.604638+0000","last_fullsized":"2024-09-16T14:59:42.604638+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:23.845897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":156,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:07.474741+0000","last_change":"2024-09-16T14:48:52.138778+0000","last_active":"2024-09-16T15:00:07.474741+0000","last_peered":"2024-09-16T15:00:07.474741+0000","last_clean":"2024-09-16T15:00:07.474741+0000","last_became_active":"2024-09-16T14:48:52.137680+0000","last_became_peered":"2024-09-16T14:48:52.137680+0000","last_unstale":"2024-09-16T15:00:07.474741+0000","last_undegraded":"2024-09-16T15:00:07.474741+0000","last_fullsized":"2024-09-16T15:00:07.474741+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:57.689750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":89,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:42.604588+0000","last_change":"2024-09-16T14:48:52.133563+0000","last_active":"2024-09-16T14:59:42.604588+0000","last_peered":"2024-09-16T14:59:42.604588+0000","last_clean":"2024-09-16T14:59:42.604588+0000","last_became_active":"2024-09-16T14:48:52.133190+0000","last_became_peered":"2024-09-16T14:48:52.133190+0000","last_unstale":"2024-09-16T14:59:42.604588+0000","last_undegraded":"2024-09-16T14:59:42.604588+0000","last_fullsized":"2024-09-16T14:59:42.604588+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:53:36.816994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":124,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:58:22.592610+0000","last_change":"2024-09-16T14:48:53.142970+0000","last_active":"2024-09-16T14:58:22.592610+0000","last_peered":"2024-09-16T14:58:22.592610+0000","last_clean":"2024-09-16T14:58:22.592610+0000","last_became_active":"2024-09-16T14:48:53.142066+0000","last_became_peered":"2024-09-16T14:48:53.142066+0000","last_unstale":"2024-09-16T14:58:22.592610+0000","last_undegraded":"2024-09-16T14:58:22.592610+0000","last_fullsized":"2024-09-16T14:58:22.592610+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:29:40.181990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":143,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:07.475174+0000","last_change":"2024-09-16T14:48:52.128011+0000","last_active":"2024-09-16T15:00:07.475174+0000","last_peered":"2024-09-16T15:00:07.475174+0000","last_clean":"2024-09-16T15:00:07.475174+0000","last_became_active":"2024-09-16T14:48:52.127250+0000","last_became_peered":"2024-09-16T14:48:52.127250+0000","last_unstale":"2024-09-16T15:00:07.475174+0000","last_undegraded":"2024-09-16T15:00:07.475174+0000","last_fullsized":"2024-09-16T15:00:07.475174+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:03:30.725414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":91,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:42.474316+0000","last_change":"2024-09-16T14:48:52.133060+0000","last_active":"2024-09-16T14:59:42.474316+0000","last_peered":"2024-09-16T14:59:42.474316+0000","last_clean":"2024-09-16T14:59:42.474316+0000","last_became_active":"2024-09-16T14:48:52.132725+0000","last_became_peered":"2024-09-16T14:48:52.132725+0000","last_unstale":"2024-09-16T14:59:42.474316+0000","last_undegraded":"2024-09-16T14:59:42.474316+0000","last_fullsized":"2024-09-16T14:59:42.474316+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:55:32.475641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":81,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468949+0000","last_change":"2024-09-16T14:48:52.129972+0000","last_active":"2024-09-16T14:59:17.468949+0000","last_peered":"2024-09-16T14:59:17.468949+0000","last_clean":"2024-09-16T14:59:17.468949+0000","last_became_active":"2024-09-16T14:48:52.128799+0000","last_became_peered":"2024-09-16T14:48:52.128799+0000","last_unstale":"2024-09-16T14:59:17.468949+0000","last_undegraded":"2024-09-16T14:59:17.468949+0000","last_fullsized":"2024-09-16T14:59:17.468949+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:48:37.984512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":111,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:02.607606+0000","last_change":"2024-09-16T14:48:52.143178+0000","last_active":"2024-09-16T15:00:02.607606+0000","last_peered":"2024-09-16T15:00:02.607606+0000","last_clean":"2024-09-16T15:00:02.607606+0000","last_became_active":"2024-09-16T14:48:52.142697+0000","last_became_peered":"2024-09-16T14:48:52.142697+0000","last_unstale":"2024-09-16T15:00:02.607606+0000","last_undegraded":"2024-09-16T15:00:02.607606+0000","last_fullsized":"2024-09-16T15:00:02.607606+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:43:37.007895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":112,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:12.478953+0000","last_change":"2024-09-16T14:48:52.144132+0000","last_active":"2024-09-16T15:00:12.478953+0000","last_peered":"2024-09-16T15:00:12.478953+0000","last_clean":"2024-09-16T15:00:12.478953+0000","last_became_active":"2024-09-16T14:48:52.143756+0000","last_became_peered":"2024-09-16T14:48:52.143756+0000","last_unstale":"2024-09-16T15:00:12.478953+0000","last_undegraded":"2024-09-16T15:00:12.478953+0000","last_fullsized":"2024-09-16T15:00:12.478953+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:15.504836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":126,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:02.600738+0000","last_change":"2024-09-16T14:48:52.139682+0000","last_active":"2024-09-16T15:00:02.600738+0000","last_peered":"2024-09-16T15:00:02.600738+0000","last_clean":"2024-09-16T15:00:02.600738+0000","last_became_active":"2024-09-16T14:48:52.139205+0000","last_became_peered":"2024-09-16T14:48:52.139205+0000","last_unstale":"2024-09-16T15:00:02.600738+0000","last_undegraded":"2024-09-16T15:00:02.600738+0000","last_fullsized":"2024-09-16T15:00:02.600738+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:56.518201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":81,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468831+0000","last_change":"2024-09-16T14:48:52.131936+0000","last_active":"2024-09-16T14:59:17.468831+0000","last_peered":"2024-09-16T14:59:17.468831+0000","last_clean":"2024-09-16T14:59:17.468831+0000","last_became_active":"2024-09-16T14:48:52.131089+0000","last_became_peered":"2024-09-16T14:48:52.131089+0000","last_unstale":"2024-09-16T14:59:17.468831+0000","last_undegraded":"2024-09-16T14:59:17.468831+0000","last_fullsized":"2024-09-16T14:59:17.468831+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:04:08.492279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":82,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468863+0000","last_change":"2024-09-16T14:48:52.138889+0000","last_active":"2024-09-16T14:59:17.468863+0000","last_peered":"2024-09-16T14:59:17.468863+0000","last_clean":"2024-09-16T14:59:17.468863+0000","last_became_active":"2024-09-16T14:48:52.137830+0000","last_became_peered":"2024-09-16T14:48:52.137830+0000","last_unstale":"2024-09-16T14:59:17.468863+0000","last_undegraded":"2024-09-16T14:59:17.468863+0000","last_fullsized":"2024-09-16T14:59:17.468863+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:00:48.737166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":169,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:02.474335+0000","last_change":"2024-09-16T14:48:52.145132+0000","last_active":"2024-09-16T15:00:02.474335+0000","last_peered":"2024-09-16T15:00:02.474335+0000","last_clean":"2024-09-16T15:00:02.474335+0000","last_became_active":"2024-09-16T14:48:52.144280+0000","last_became_peered":"2024-09-16T14:48:52.144280+0000","last_unstale":"2024-09-16T15:00:02.474335+0000","last_undegraded":"2024-09-16T15:00:02.474335+0000","last_fullsized":"2024-09-16T15:00:02.474335+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:37:18.905007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":90,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.467831+0000","last_change":"2024-09-16T14:48:53.564947+0000","last_active":"2024-09-16T14:59:17.467831+0000","last_peered":"2024-09-16T14:59:17.467831+0000","last_clean":"2024-09-16T14:59:17.467831+0000","last_became_active":"2024-09-16T14:48:53.563655+0000","last_became_peered":"2024-09-16T14:48:53.563655+0000","last_unstale":"2024-09-16T14:59:17.467831+0000","last_undegraded":"2024-09-16T14:59:17.467831+0000","last_fullsized":"2024-09-16T14:59:17.467831+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:32:48.022677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":81,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:58:57.465826+0000","last_change":"2024-09-16T14:48:52.138403+0000","last_active":"2024-09-16T14:58:57.465826+0000","last_peered":"2024-09-16T14:58:57.465826+0000","last_clean":"2024-09-16T14:58:57.465826+0000","last_became_active":"2024-09-16T14:48:52.137512+0000","last_became_peered":"2024-09-16T14:48:52.137512+0000","last_unstale":"2024-09-16T14:58:57.465826+0000","last_undegraded":"2024-09-16T14:58:57.465826+0000","last_fullsized":"2024-09-16T14:58:57.465826+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:00:15.913134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":229,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:58:57.465176+0000","last_change":"2024-09-16T14:48:52.131578+0000","last_active":"2024-09-16T14:58:57.465176+0000","last_peered":"2024-09-16T14:58:57.465176+0000","last_clean":"2024-09-16T14:58:57.465176+0000","last_became_active":"2024-09-16T14:48:52.131259+0000","last_became_peered":"2024-09-16T14:48:52.131259+0000","last_unstale":"2024-09-16T14:58:57.465176+0000","last_undegraded":"2024-09-16T14:58:57.465176+0000","last_fullsized":"2024-09-16T14:58:57.465176+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:02:29.950099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067836399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":220,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.600970+0000","last_change":"2024-09-16T14:48:53.141414+0000","last_active":"2024-09-16T14:59:17.600970+0000","last_peered":"2024-09-16T14:59:17.600970+0000","last_clean":"2024-09-16T14:59:17.600970+0000","last_became_active":"2024-09-16T14:48:53.140914+0000","last_became_peered":"2024-09-16T14:48:53.140914+0000","last_unstale":"2024-09-16T14:59:17.600970+0000","last_undegraded":"2024-09-16T14:59:17.600970+0000","last_fullsized":"2024-09-16T14:59:17.600970+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:53:48.881111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097689999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.594663+0000","last_change":"2024-09-16T14:48:52.149868+0000","last_active":"2024-09-16T14:59:17.594663+0000","last_peered":"2024-09-16T14:59:17.594663+0000","last_clean":"2024-09-16T14:59:17.594663+0000","last_became_active":"2024-09-16T14:48:52.142312+0000","last_became_peered":"2024-09-16T14:48:52.142312+0000","last_unstale":"2024-09-16T14:59:17.594663+0000","last_undegraded":"2024-09-16T14:59:17.594663+0000","last_fullsized":"2024-09-16T14:59:17.594663+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:31:20.443534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":87,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468897+0000","last_change":"2024-09-16T14:48:52.151300+0000","last_active":"2024-09-16T14:59:17.468897+0000","last_peered":"2024-09-16T14:59:17.468897+0000","last_clean":"2024-09-16T14:59:17.468897+0000","last_became_active":"2024-09-16T14:48:52.137048+0000","last_became_peered":"2024-09-16T14:48:52.137048+0000","last_unstale":"2024-09-16T14:59:17.468897+0000","last_undegraded":"2024-09-16T14:59:17.468897+0000","last_fullsized":"2024-09-16T14:59:17.468897+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:17:13.952991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042440799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":428,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:58:37.462283+0000","last_change":"2024-09-16T14:48:55.155094+0000","last_active":"2024-09-16T14:58:37.462283+0000","last_peered":"2024-09-16T14:58:37.462283+0000","last_clean":"2024-09-16T14:58:37.462283+0000","last_became_active":"2024-09-16T14:48:55.154511+0000","last_became_peered":"2024-09-16T14:48:55.154511+0000","last_unstale":"2024-09-16T14:58:37.462283+0000","last_undegraded":"2024-09-16T14:58:37.462283+0000","last_fullsized":"2024-09-16T14:58:37.462283+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:28.462531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:28.462531+0000","last_clean_scrub_stamp":"2024-09-16T14:42:28.462531+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:06.351384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":81,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468924+0000","last_change":"2024-09-16T14:48:52.129414+0000","last_active":"2024-09-16T14:59:17.468924+0000","last_peered":"2024-09-16T14:59:17.468924+0000","last_clean":"2024-09-16T14:59:17.468924+0000","last_became_active":"2024-09-16T14:48:52.128554+0000","last_became_peered":"2024-09-16T14:48:52.128554+0000","last_unstale":"2024-09-16T14:59:17.468924+0000","last_undegraded":"2024-09-16T14:59:17.468924+0000","last_fullsized":"2024-09-16T14:59:17.468924+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:38:09.329918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039796999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":89,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.594692+0000","last_change":"2024-09-16T14:48:52.144047+0000","last_active":"2024-09-16T14:59:17.594692+0000","last_peered":"2024-09-16T14:59:17.594692+0000","last_clean":"2024-09-16T14:59:17.594692+0000","last_became_active":"2024-09-16T14:48:52.143705+0000","last_became_peered":"2024-09-16T14:48:52.143705+0000","last_unstale":"2024-09-16T14:59:17.594692+0000","last_undegraded":"2024-09-16T14:59:17.594692+0000","last_fullsized":"2024-09-16T14:59:17.594692+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:14:59.892564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061478100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":90,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:42.474287+0000","last_change":"2024-09-16T14:48:52.131363+0000","last_active":"2024-09-16T14:59:42.474287+0000","last_peered":"2024-09-16T14:59:42.474287+0000","last_clean":"2024-09-16T14:59:42.474287+0000","last_became_active":"2024-09-16T14:48:52.131160+0000","last_became_peered":"2024-09-16T14:48:52.131160+0000","last_unstale":"2024-09-16T14:59:42.474287+0000","last_undegraded":"2024-09-16T14:59:42.474287+0000","last_fullsized":"2024-09-16T14:59:42.474287+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:12.859518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065779100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":226,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:52.606162+0000","last_change":"2024-09-16T14:48:52.133533+0000","last_active":"2024-09-16T14:59:52.606162+0000","last_peered":"2024-09-16T14:59:52.606162+0000","last_clean":"2024-09-16T14:59:52.606162+0000","last_became_active":"2024-09-16T14:48:52.133197+0000","last_became_peered":"2024-09-16T14:48:52.133197+0000","last_unstale":"2024-09-16T14:59:52.606162+0000","last_undegraded":"2024-09-16T14:59:52.606162+0000","last_fullsized":"2024-09-16T14:59:52.606162+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:37:19.055419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085482099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":94400,"data_compressed_allocated":122880,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640499,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1413120,"data_stored":1289362,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.622,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.92600000000000005,"5min":1.042,"15min":1.2},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.81499999999999995,"15min":0.79100000000000004},"min":{"1min":0.627,"5min":0.51500000000000001,"15min":0.47899999999999998},"max":{"1min":0.95599999999999996,"5min":1.0369999999999999,"15min":1.3300000000000001},"last":0.95599999999999996}]},{"osd":1,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.81799999999999995,"15min":0.81000000000000005},"min":{"1min":0.61899999999999999,"5min":0.44900000000000001,"15min":0.33900000000000002},"max":{"1min":1.0589999999999999,"5min":1.073,"15min":1.2010000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.74399999999999999,"15min":0.752},"min":{"1min":0.60999999999999999,"5min":0.54000000000000004,"15min":0.438},"max":{"1min":0.90800000000000003,"5min":0.94299999999999995,"15min":1.089},"last":0.77300000000000002}]},{"osd":2,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.81799999999999995,"15min":0.82899999999999996},"min":{"1min":0.628,"5min":0.50800000000000001,"15min":0.44900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.272},"last":0.75},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.90600000000000003,"15min":0.91300000000000003},"min":{"1min":0.81000000000000005,"5min":0.52900000000000003,"15min":0.42899999999999999},"max":{"1min":1.0289999999999999,"5min":1.056,"15min":1.3540000000000001},"last":0.93000000000000005}]},{"osd":3,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69699999999999995,"15min":0.67600000000000005},"min":{"1min":0.52300000000000002,"5min":0.46600000000000003,"15min":0.26000000000000001},"max":{"1min":0.96799999999999997,"5min":0.96999999999999997,"15min":1.1020000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.68200000000000005,"15min":0.69099999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33500000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":1.153},"last":0.90000000000000002}]},{"osd":4,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72899999999999998,"15min":0.749},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.27500000000000002},"max":{"1min":0.95299999999999996,"5min":1.05,"15min":1.236},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.80000000000000004,"15min":0.77200000000000002},"min":{"1min":0.48799999999999999,"5min":0.44800000000000001,"15min":0.26300000000000001},"max":{"1min":1.038,"5min":1.0529999999999999,"15min":1.2949999999999999},"last":1.01}]}]},{"osd":4,"up_from":11,"seq":47244640499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1417216,"data_stored":1289381,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78200000000000003,"15min":0.77300000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.437},"max":{"1min":0.96699999999999997,"5min":1.0800000000000001,"15min":1.1579999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73899999999999999,"15min":0.79000000000000004},"min":{"1min":0.52600000000000002,"5min":0.48899999999999999,"15min":0.48499999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.1679999999999999},"last":0.73999999999999999}]},{"osd":1,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.85899999999999999,"15min":0.81599999999999995},"min":{"1min":0.71199999999999997,"5min":0.52800000000000002,"15min":0.42399999999999999},"max":{"1min":1.0429999999999999,"5min":1.0780000000000001,"15min":1.4630000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.79600000000000004,"15min":0.83399999999999996},"min":{"1min":0.65700000000000003,"5min":0.53700000000000003,"15min":0.50800000000000001},"max":{"1min":1.0149999999999999,"5min":1.024,"15min":1.2729999999999999},"last":0.92300000000000004}]},{"osd":2,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.84999999999999998,"15min":0.84899999999999998},"min":{"1min":0.68000000000000005,"5min":0.65500000000000003,"15min":0.47599999999999998},"max":{"1min":1.02,"5min":1.0449999999999999,"15min":1.536},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84799999999999998,"15min":0.85899999999999999},"min":{"1min":0.63500000000000001,"5min":0.56599999999999995,"15min":0.51200000000000001},"max":{"1min":0.97099999999999997,"5min":1.0720000000000001,"15min":1.29},"last":0.79300000000000004}]},{"osd":3,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.70999999999999996,"15min":0.68899999999999995},"min":{"1min":0.53700000000000003,"5min":0.51300000000000001,"15min":0.27700000000000002},"max":{"1min":0.83299999999999996,"5min":1.0720000000000001,"15min":1.2669999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72899999999999998,"15min":0.72499999999999998},"min":{"1min":0.55700000000000005,"5min":0.50600000000000001,"15min":0.25700000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.198},"last":0.69799999999999995}]},{"osd":5,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.75600000000000001,"15min":0.73099999999999998},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.28999999999999998},"max":{"1min":0.95799999999999996,"5min":1.0600000000000001,"15min":1.1140000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.94699999999999995,"15min":0.91700000000000004},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.24299999999999999},"max":{"1min":1.0700000000000001,"5min":1.169,"15min":1.3879999999999999},"last":0.96599999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640498,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32556,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339284,"statfs":{"total":96636764160,"available":96603426816,"internally_reserved":0,"allocated":1474560,"data_stored":1818290,"data_compressed":271867,"data_compressed_allocated":389120,"data_compressed_original":1138688,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.45200000000000001},"max":{"1min":0.998,"5min":0.998,"15min":1.2729999999999999},"last":0.622},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74099999999999999,"15min":0.753},"min":{"1min":0.60799999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90500000000000003,"5min":0.94199999999999995,"15min":1.306},"last":0.83499999999999996}]},{"osd":1,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90200000000000002,"15min":0.86599999999999999},"min":{"1min":0.55800000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.069,"5min":1.1259999999999999,"15min":1.294},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.86199999999999999,"15min":0.86599999999999999},"min":{"1min":0.67900000000000005,"5min":0.56200000000000006,"15min":0.41199999999999998},"max":{"1min":0.93999999999999995,"5min":1.046,"15min":1.9219999999999999},"last":0.72899999999999998}]},{"osd":2,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81799999999999995,"15min":0.81699999999999995},"min":{"1min":0.71299999999999997,"5min":0.61199999999999999,"15min":0.45900000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.181},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79900000000000004,"15min":0.80600000000000005},"min":{"1min":0.627,"5min":0.627,"15min":0.38800000000000001},"max":{"1min":0.90300000000000002,"5min":0.96899999999999997,"15min":2.444},"last":0.78000000000000003}]},{"osd":4,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.624,"15min":0.623},"min":{"1min":0.49299999999999999,"5min":0.47699999999999998,"15min":0.28199999999999997},"max":{"1min":0.89800000000000002,"5min":0.98799999999999999,"15min":1.0700000000000001},"last":0.75},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.67900000000000005,"5min":0.628,"15min":0.317},"max":{"1min":0.99299999999999999,"5min":1.0740000000000001,"15min":1.2290000000000001},"last":0.92000000000000004}]},{"osd":5,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69299999999999995,"15min":0.70699999999999996},"min":{"1min":0.55500000000000005,"5min":0.52400000000000002,"15min":0.29899999999999999},"max":{"1min":0.92200000000000004,"5min":1.4279999999999999,"15min":2.5409999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.83499999999999996,"15min":0.80900000000000005},"min":{"1min":0.63700000000000001,"5min":0.60999999999999999,"15min":0.33100000000000002},"max":{"1min":1.081,"5min":1.514,"15min":1.514},"last":0.871}]}]},{"osd":2,"up_from":11,"seq":47244640498,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1413120,"data_stored":1289362,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56200000000000006,"15min":0.58099999999999996},"min":{"1min":0.252,"5min":0.252,"15min":0.249},"max":{"1min":0.749,"5min":0.94699999999999995,"15min":1.819},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.77600000000000002,"15min":0.75700000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.312},"max":{"1min":1.147,"5min":1.147,"15min":1.444},"last":0.91900000000000004}]},{"osd":1,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.39200000000000002,"5min":0.379,"15min":0.28299999999999997},"max":{"1min":0.95499999999999996,"5min":0.95999999999999996,"15min":1.27},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.62,"15min":0.623},"min":{"1min":0.313,"5min":0.313,"15min":0.26700000000000002},"max":{"1min":1.0129999999999999,"5min":1.0189999999999999,"15min":1.383},"last":0.68000000000000005}]},{"osd":3,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.84199999999999997,"15min":0.82299999999999995},"min":{"1min":0.58599999999999997,"5min":0.56000000000000005,"15min":0.36899999999999999},"max":{"1min":1.0720000000000001,"5min":1.169,"15min":1.2110000000000001},"last":1.034},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.871,"15min":0.86299999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.44700000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.335},"last":0.72399999999999998}]},{"osd":4,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.83599999999999997,"15min":0.83499999999999996},"min":{"1min":0.53300000000000003,"5min":0.50600000000000001,"15min":0.45500000000000002},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.874},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.84999999999999998,"15min":0.85499999999999998},"min":{"1min":0.62,"5min":0.58599999999999997,"15min":0.40999999999999998},"max":{"1min":0.97099999999999997,"5min":1.0489999999999999,"15min":1.367},"last":0.84299999999999997}]},{"osd":5,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.88200000000000001,"15min":0.89100000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.53700000000000003},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.911},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.95399999999999996,"15min":0.95499999999999996},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.121,"5min":1.1990000000000001,"15min":2.0499999999999998},"last":0.90800000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640497,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1462272,"data_stored":1818290,"data_compressed":273302,"data_compressed_allocated":389120,"data_compressed_original":1150976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56000000000000005,"15min":0.57099999999999995},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.251},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":1.0720000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.628,"15min":0.63800000000000001},"min":{"1min":0.44500000000000001,"5min":0.36199999999999999,"15min":0.26100000000000001},"max":{"1min":0.87,"5min":0.87,"15min":1.0589999999999999},"last":0.81499999999999995}]},{"osd":2,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.66700000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.28699999999999998},"max":{"1min":0.82599999999999996,"5min":0.88200000000000001,"15min":1.052},"last":0.748},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.628,"15min":0.66700000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32100000000000001},"max":{"1min":0.81899999999999995,"5min":0.879,"15min":1.403},"last":0.83799999999999997}]},{"osd":3,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85699999999999998,"15min":0.88800000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.503},"max":{"1min":1.052,"5min":1.117,"15min":1.359},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.86299999999999999,"15min":0.877},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1459999999999999,"5min":1.147,"15min":1.284},"last":1.1719999999999999}]},{"osd":4,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.88900000000000001,"15min":0.88200000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.46600000000000003},"max":{"1min":1.0900000000000001,"5min":1.097,"15min":1.341},"last":1.111},{"interface":"front","average":{"1min":0.88,"5min":0.86199999999999999,"15min":0.86899999999999999},"min":{"1min":0.67600000000000005,"5min":0.66200000000000003,"15min":0.46999999999999997},"max":{"1min":1.056,"5min":1.0800000000000001,"15min":1.399},"last":1.0269999999999999}]},{"osd":5,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.622,"5min":0.622,"15min":0.50900000000000001},"max":{"1min":1,"5min":1.01,"15min":1.258},"last":1.054},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90100000000000002,"15min":0.90400000000000003},"min":{"1min":0.69499999999999995,"5min":0.629,"15min":0.47299999999999998},"max":{"1min":1.1519999999999999,"5min":1.411,"15min":1.411},"last":1.087}]}]},{"osd":0,"up_from":11,"seq":47244640498,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36596,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335244,"statfs":{"total":96636764160,"available":96599289856,"internally_reserved":0,"allocated":1417216,"data_stored":1289381,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57499999999999996,"15min":0.57999999999999996},"min":{"1min":0.36099999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.74299999999999999,"5min":0.77700000000000002,"15min":1.1739999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.61399999999999999,"15min":0.63800000000000001},"min":{"1min":0.38200000000000001,"5min":0.28299999999999997,"15min":0.27400000000000002},"max":{"1min":0.67800000000000005,"5min":1.032,"15min":1.21},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68700000000000006,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.29899999999999999,"15min":0.25900000000000001},"max":{"1min":0.85799999999999998,"5min":0.997,"15min":1.103},"last":0.495},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.62,"15min":0.67200000000000004},"min":{"1min":0.47999999999999998,"5min":0.36099999999999999,"15min":0.26900000000000002},"max":{"1min":0.69099999999999995,"5min":1.008,"15min":1.137},"last":0.44}]},{"osd":3,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.82599999999999996},"min":{"1min":0.67100000000000004,"5min":0.59599999999999997,"15min":0.52400000000000002},"max":{"1min":1.014,"5min":1.1040000000000001,"15min":1.4650000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.81799999999999995,"15min":0.83999999999999997},"min":{"1min":0.69299999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.99299999999999999,"5min":1.101,"15min":1.4950000000000001},"last":0.59199999999999997}]},{"osd":4,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.84599999999999997,"15min":0.84699999999999998},"min":{"1min":0.68899999999999995,"5min":0.57999999999999996,"15min":0.48999999999999999},"max":{"1min":0.98499999999999999,"5min":1.1559999999999999,"15min":1.3300000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.85099999999999998},"min":{"1min":0.63700000000000001,"5min":0.55700000000000005,"15min":0.51600000000000001},"max":{"1min":0.96099999999999997,"5min":1.095,"15min":1.4179999999999999},"last":0.46999999999999997}]},{"osd":5,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.91200000000000003,"15min":0.91400000000000003},"min":{"1min":0.70899999999999996,"5min":0.66400000000000003,"15min":0.57999999999999996},"max":{"1min":1.1080000000000001,"5min":1.304,"15min":1.6499999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95799999999999996,"15min":0.94799999999999995},"min":{"1min":0.76900000000000002,"5min":0.69999999999999996,"15min":0.51900000000000002},"max":{"1min":1.0589999999999999,"5min":1.2030000000000001,"15min":1.7290000000000001},"last":0.754}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:02:23.984 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T15:02:23.984 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:02:24.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.107+0000 7f044906d640 10 monclient: get_monmap_and_config 2024-09-16T15:02:24.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.107+0000 7f044906d640 10 monclient: build_initial_monmap 2024-09-16T15:02:24.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.107+0000 7f044906d640 10 monclient: monmap: 2024-09-16T15:02:24.110 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:24.110 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:24.110 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:24.111767+0000 2024-09-16T15:02:24.110 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:24.111767+0000 2024-09-16T15:02:24.110 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:24.110 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:24.110 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:24.110 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:24.110 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:24.110 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:24.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.107+0000 7f044906d640 1 Processor -- start 2024-09-16T15:02:24.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.107+0000 7f044906d640 1 -- start start 2024-09-16T15:02:24.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.107+0000 7f044906d640 10 monclient: init 2024-09-16T15:02:24.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.107+0000 7f044906d640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:24.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.107+0000 7f044906d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:02:24.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.107+0000 7f044906d640 10 monclient(hunting): picked mon.noname-b con 0x7f044409cb80 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:24.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.107+0000 7f044906d640 10 monclient(hunting): picked mon.noname-c con 0x7f0444132650 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:24.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.107+0000 7f044906d640 10 monclient(hunting): picked mon.noname-a con 0x7f0444138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:24.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.107+0000 7f044906d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044409d260 con 0x7f0444138ff0 2024-09-16T15:02:24.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.107+0000 7f044906d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0444132a80 con 0x7f0444132650 2024-09-16T15:02:24.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.107+0000 7f044906d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044409cf60 con 0x7f044409cb80 2024-09-16T15:02:24.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.107+0000 7f044906d640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:24.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.107+0000 7f044906d640 10 monclient(hunting): authenticate will time out at 2168.231851s 2024-09-16T15:02:24.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044359e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0444138ff0 0x7f044413b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42182/0 (socket says 172.21.15.26:42182) 2024-09-16T15:02:24.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0442d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f044409cb80 0x7f04440717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47906/0 (socket says 172.21.15.26:47906) 2024-09-16T15:02:24.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044359e640 1 -- 172.21.15.26:0/1401260530 learned_addr learned my addr 172.21.15.26:0/1401260530 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:24.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 1 -- 172.21.15.26:0/1401260530 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064969120 0 0) 0x7f044409d260 con 0x7f0444138ff0 2024-09-16T15:02:24.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient(hunting): _negotiate 2024-09-16T15:02:24.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:24.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:24.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient(hunting): my global_id is 334467 2024-09-16T15:02:24.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 1 -- 172.21.15.26:0/1401260530 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f042c003530 con 0x7f0444138ff0 2024-09-16T15:02:24.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 1 -- 172.21.15.26:0/1401260530 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230218270 0 0) 0x7f044409cf60 con 0x7f044409cb80 2024-09-16T15:02:24.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient(hunting): _negotiate 2024-09-16T15:02:24.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:24.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:24.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient(hunting): my global_id is 354281 2024-09-16T15:02:24.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 1 -- 172.21.15.26:0/1401260530 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f044409d260 con 0x7f044409cb80 2024-09-16T15:02:24.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 1 -- 172.21.15.26:0/1401260530 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404563806 0 0) 0x7f0444132a80 con 0x7f0444132650 2024-09-16T15:02:24.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient(hunting): _negotiate 2024-09-16T15:02:24.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:24.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:24.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient(hunting): my global_id is 364147 2024-09-16T15:02:24.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 1 -- 172.21.15.26:0/1401260530 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f044409cf60 con 0x7f0444132650 2024-09-16T15:02:24.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 1 -- 172.21.15.26:0/1401260530 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 249803863 0 0) 0x7f044409d260 con 0x7f044409cb80 2024-09-16T15:02:24.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 1 -- 172.21.15.26:0/1401260530 >> v1:172.21.15.189:6789/0 conn(0x7f0444132650 legacy=0x7f04441378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:24.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 1 -- 172.21.15.26:0/1401260530 >> v1:172.21.15.26:6789/0 conn(0x7f0444138ff0 legacy=0x7f044413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:24.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient: _finish_hunting 0 2024-09-16T15:02:24.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 1 monclient: found mon.noname-b 2024-09-16T15:02:24.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:24.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:02:24.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 1 -- 172.21.15.26:0/1401260530 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044409f2d0 con 0x7f044409cb80 2024-09-16T15:02:24.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient: _finish_auth 0 2024-09-16T15:02:24.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient: _check_auth_tickets 2024-09-16T15:02:24.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:24.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 1 -- 172.21.15.26:0/1401260530 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0434001640 con 0x7f044409cb80 2024-09-16T15:02:24.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 5 monclient: authenticate success, global_id 354281 2024-09-16T15:02:24.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:24.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:24.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:02:24.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient: dump: 2024-09-16T15:02:24.121 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:24.121 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:24.121 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.121 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.121 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:24.121 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:24.121 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:24.122 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:24.122 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:24.122 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:24.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:24.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 1 -- 172.21.15.26:0/1401260530 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04340017e0 con 0x7f044409cb80 2024-09-16T15:02:24.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:24.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 1 -- 172.21.15.26:0/1401260530 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0434001b70 con 0x7f044409cb80 2024-09-16T15:02:24.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:24.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 4 set_mon_vals no callback set 2024-09-16T15:02:24.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:24.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:24.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f0441d9b640 10 monclient: dump: 2024-09-16T15:02:24.123 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:24.123 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:24.123 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.123 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.123 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:24.123 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:24.123 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:24.123 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:24.123 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:24.123 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:24.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 10 monclient: shutdown 2024-09-16T15:02:24.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:24.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 1 -- 172.21.15.26:0/1401260530 >> v1:172.21.15.26:6790/0 conn(0x7f044409cb80 legacy=0x7f04440717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:24.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 1 -- 172.21.15.26:0/1401260530 shutdown_connections 2024-09-16T15:02:24.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 1 -- 172.21.15.26:0/1401260530 >> 172.21.15.26:0/1401260530 conn(0x7f044409fb50 msgr2=0x7f044409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:24.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 1 -- 172.21.15.26:0/1401260530 shutdown_connections 2024-09-16T15:02:24.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 1 -- 172.21.15.26:0/1401260530 wait complete. 2024-09-16T15:02:24.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 10 monclient: build_initial_monmap 2024-09-16T15:02:24.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 10 monclient: monmap: 2024-09-16T15:02:24.124 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:24.125 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:24.125 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:24.125 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:24.125 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:24.125 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:24.125 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:24.125 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:24.125 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:24.125 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:24.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 1 Processor -- start 2024-09-16T15:02:24.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 1 -- start start 2024-09-16T15:02:24.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 10 monclient: init 2024-09-16T15:02:24.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:24.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:02:24.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 10 monclient(hunting): picked mon.noname-b con 0x7f044409cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:24.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 10 monclient(hunting): picked mon.noname-a con 0x7f0444132650 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:24.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 10 monclient(hunting): picked mon.noname-c con 0x7f0444138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:24.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044406e3e0 con 0x7f0444132650 2024-09-16T15:02:24.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044413c590 con 0x7f044409cb80 2024-09-16T15:02:24.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044409a240 con 0x7f0444138ff0 2024-09-16T15:02:24.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:24.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044906d640 10 monclient(hunting): authenticate will time out at 2168.235184s 2024-09-16T15:02:24.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044259c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0444132650 0x7f044406dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42194/0 (socket says 172.21.15.26:42194) 2024-09-16T15:02:24.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f044259c640 1 -- 172.21.15.26:0/4114048660 learned_addr learned my addr 172.21.15.26:0/4114048660 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:24.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139491517 0 0) 0x7f044406e3e0 con 0x7f0444132650 2024-09-16T15:02:24.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:24.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:24.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:24.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 10 monclient(hunting): my global_id is 334470 2024-09-16T15:02:24.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0430003530 con 0x7f0444132650 2024-09-16T15:02:24.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726000424 0 0) 0x7f044409a240 con 0x7f0444138ff0 2024-09-16T15:02:24.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:24.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:24.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:24.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 10 monclient(hunting): my global_id is 354284 2024-09-16T15:02:24.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f044406e3e0 con 0x7f0444138ff0 2024-09-16T15:02:24.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4269152414 0 0) 0x7f0430003530 con 0x7f0444132650 2024-09-16T15:02:24.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f044409a240 con 0x7f0444132650 2024-09-16T15:02:24.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0438001940 con 0x7f0444132650 2024-09-16T15:02:24.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:24.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:24.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 10 monclient(hunting): dump: 2024-09-16T15:02:24.128 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:24.128 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:24.128 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.128 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.128 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:24.128 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:24.128 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:24.128 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:24.128 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:24.129 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:24.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:24.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909633045 0 0) 0x7f044413c590 con 0x7f044409cb80 2024-09-16T15:02:24.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:24.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:24.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:24.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.111+0000 7f042b7fe640 10 monclient(hunting): my global_id is 364150 2024-09-16T15:02:24.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0430003530 con 0x7f044409cb80 2024-09-16T15:02:24.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1120830043 0 0) 0x7f044406e3e0 con 0x7f0444138ff0 2024-09-16T15:02:24.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f044413c590 con 0x7f0444138ff0 2024-09-16T15:02:24.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f043c0014a0 con 0x7f0444138ff0 2024-09-16T15:02:24.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:24.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:24.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 10 monclient(hunting): dump: 2024-09-16T15:02:24.130 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:24.130 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:24.130 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.130 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.130 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:24.130 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:24.130 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:24.130 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:24.130 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:24.130 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:24.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1690028788 0 0) 0x7f044409a240 con 0x7f0444132650 2024-09-16T15:02:24.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 >> v1:172.21.15.26:6790/0 conn(0x7f0444138ff0 legacy=0x7f04441cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:24.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 >> v1:172.21.15.189:6789/0 conn(0x7f044409cb80 legacy=0x7f044406d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:24.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 10 monclient: _finish_hunting 0 2024-09-16T15:02:24.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 1 monclient: found mon.a 2024-09-16T15:02:24.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:24.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:24.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044406fb10 con 0x7f0444132650 2024-09-16T15:02:24.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 10 monclient: _finish_auth 0 2024-09-16T15:02:24.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 10 monclient: _check_auth_tickets 2024-09-16T15:02:24.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:24.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f044906d640 5 monclient: authenticate success, global_id 334470 2024-09-16T15:02:24.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f044906d640 10 monclient: _renew_subs 2024-09-16T15:02:24.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f044906d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:24.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f044906d640 1 -- 172.21.15.26:0/4114048660 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0444070ce0 con 0x7f0444132650 2024-09-16T15:02:24.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0438003600 con 0x7f0444132650 2024-09-16T15:02:24.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:24.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0438003910 con 0x7f0444132650 2024-09-16T15:02:24.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:24.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:24.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 10 monclient: dump: 2024-09-16T15:02:24.132 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:24.132 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:24.132 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.132 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.132 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:24.132 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:24.132 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:24.132 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:24.132 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:24.133 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f044906d640 10 monclient: _renew_subs 2024-09-16T15:02:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f044159a640 4 set_mon_vals no callback set 2024-09-16T15:02:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f044906d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f044906d640 1 -- 172.21.15.26:0/4114048660 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0444071520 con 0x7f0444132650 2024-09-16T15:02:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f044906d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f044906d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f044906d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f044906d640 1 -- 172.21.15.26:0/4114048660 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044409c960 con 0x7f0444132650 2024-09-16T15:02:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f0438003a70 con 0x7f0444132650 2024-09-16T15:02:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f0438043d40 con 0x7f0444132650 2024-09-16T15:02:24.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.115+0000 7f0442d9d640 10 monclient: get_auth_request con 0x7f0430036440 auth_method 0 2024-09-16T15:02:24.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.119+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0438004270 con 0x7f0444132650 2024-09-16T15:02:24.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.119+0000 7f042b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:24.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.119+0000 7f042b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:24.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.275+0000 7f044906d640 1 -- 172.21.15.26:0/4114048660 --> v1:172.21.15.189:6812/3196161582 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f044409f5b0 con 0x7f0430036440 2024-09-16T15:02:24.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.287+0000 7f042b7fe640 1 -- 172.21.15.26:0/4114048660 <== mgr.4105 v1:172.21.15.189:6812/3196161582 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+97880 (unknown 2965378022 0 4189929426) 0x7f0434001220 con 0x7f0430036440 2024-09-16T15:02:24.287 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T15:02:24.287 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T15:02:24.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.287+0000 7f044906d640 1 -- 172.21.15.26:0/4114048660 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0430036440 legacy=0x7f0430038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:24.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.287+0000 7f044906d640 10 monclient: shutdown 2024-09-16T15:02:24.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.287+0000 7f044906d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:24.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.287+0000 7f044906d640 1 -- 172.21.15.26:0/4114048660 >> v1:172.21.15.26:6789/0 conn(0x7f0444132650 legacy=0x7f044406dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:24.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.287+0000 7f044906d640 1 -- 172.21.15.26:0/4114048660 shutdown_connections 2024-09-16T15:02:24.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.287+0000 7f044906d640 1 -- 172.21.15.26:0/4114048660 >> 172.21.15.26:0/4114048660 conn(0x7f044409fb50 msgr2=0x7f0444131540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:24.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.287+0000 7f044906d640 1 -- 172.21.15.26:0/4114048660 shutdown_connections 2024-09-16T15:02:24.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.287+0000 7f044906d640 1 -- 172.21.15.26:0/4114048660 wait complete. 2024-09-16T15:02:24.305 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":841,"stamp":"2024-09-16T15:02:22.725242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1057913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":199188,"kb_used_data":8396,"kb_used_omap":9,"kb_used_meta":190710,"kb_avail":566031852,"statfs":{"total":579820584960,"available":579616616448,"internally_reserved":0,"allocated":8597504,"data_stored":8794066,"data_compressed":1510597,"data_compressed_allocated":2170880,"data_compressed_original":4796416,"omap_allocated":9559,"internal_metadata":195287721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003445"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":164,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:02.474374+0000","last_change":"2024-09-16T14:48:52.143014+0000","last_active":"2024-09-16T15:00:02.474374+0000","last_peered":"2024-09-16T15:00:02.474374+0000","last_clean":"2024-09-16T15:00:02.474374+0000","last_became_active":"2024-09-16T14:48:52.141236+0000","last_became_peered":"2024-09-16T14:48:52.141236+0000","last_unstale":"2024-09-16T15:00:02.474374+0000","last_undegraded":"2024-09-16T15:00:02.474374+0000","last_fullsized":"2024-09-16T15:00:02.474374+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:18:35.458388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":89,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:42.604638+0000","last_change":"2024-09-16T14:48:52.126720+0000","last_active":"2024-09-16T14:59:42.604638+0000","last_peered":"2024-09-16T14:59:42.604638+0000","last_clean":"2024-09-16T14:59:42.604638+0000","last_became_active":"2024-09-16T14:48:52.126325+0000","last_became_peered":"2024-09-16T14:48:52.126325+0000","last_unstale":"2024-09-16T14:59:42.604638+0000","last_undegraded":"2024-09-16T14:59:42.604638+0000","last_fullsized":"2024-09-16T14:59:42.604638+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:23.845897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":156,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:07.474741+0000","last_change":"2024-09-16T14:48:52.138778+0000","last_active":"2024-09-16T15:00:07.474741+0000","last_peered":"2024-09-16T15:00:07.474741+0000","last_clean":"2024-09-16T15:00:07.474741+0000","last_became_active":"2024-09-16T14:48:52.137680+0000","last_became_peered":"2024-09-16T14:48:52.137680+0000","last_unstale":"2024-09-16T15:00:07.474741+0000","last_undegraded":"2024-09-16T15:00:07.474741+0000","last_fullsized":"2024-09-16T15:00:07.474741+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:57.689750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":89,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:42.604588+0000","last_change":"2024-09-16T14:48:52.133563+0000","last_active":"2024-09-16T14:59:42.604588+0000","last_peered":"2024-09-16T14:59:42.604588+0000","last_clean":"2024-09-16T14:59:42.604588+0000","last_became_active":"2024-09-16T14:48:52.133190+0000","last_became_peered":"2024-09-16T14:48:52.133190+0000","last_unstale":"2024-09-16T14:59:42.604588+0000","last_undegraded":"2024-09-16T14:59:42.604588+0000","last_fullsized":"2024-09-16T14:59:42.604588+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:53:36.816994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":124,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:58:22.592610+0000","last_change":"2024-09-16T14:48:53.142970+0000","last_active":"2024-09-16T14:58:22.592610+0000","last_peered":"2024-09-16T14:58:22.592610+0000","last_clean":"2024-09-16T14:58:22.592610+0000","last_became_active":"2024-09-16T14:48:53.142066+0000","last_became_peered":"2024-09-16T14:48:53.142066+0000","last_unstale":"2024-09-16T14:58:22.592610+0000","last_undegraded":"2024-09-16T14:58:22.592610+0000","last_fullsized":"2024-09-16T14:58:22.592610+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:29:40.181990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":143,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:07.475174+0000","last_change":"2024-09-16T14:48:52.128011+0000","last_active":"2024-09-16T15:00:07.475174+0000","last_peered":"2024-09-16T15:00:07.475174+0000","last_clean":"2024-09-16T15:00:07.475174+0000","last_became_active":"2024-09-16T14:48:52.127250+0000","last_became_peered":"2024-09-16T14:48:52.127250+0000","last_unstale":"2024-09-16T15:00:07.475174+0000","last_undegraded":"2024-09-16T15:00:07.475174+0000","last_fullsized":"2024-09-16T15:00:07.475174+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:03:30.725414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":91,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:42.474316+0000","last_change":"2024-09-16T14:48:52.133060+0000","last_active":"2024-09-16T14:59:42.474316+0000","last_peered":"2024-09-16T14:59:42.474316+0000","last_clean":"2024-09-16T14:59:42.474316+0000","last_became_active":"2024-09-16T14:48:52.132725+0000","last_became_peered":"2024-09-16T14:48:52.132725+0000","last_unstale":"2024-09-16T14:59:42.474316+0000","last_undegraded":"2024-09-16T14:59:42.474316+0000","last_fullsized":"2024-09-16T14:59:42.474316+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:55:32.475641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":81,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468949+0000","last_change":"2024-09-16T14:48:52.129972+0000","last_active":"2024-09-16T14:59:17.468949+0000","last_peered":"2024-09-16T14:59:17.468949+0000","last_clean":"2024-09-16T14:59:17.468949+0000","last_became_active":"2024-09-16T14:48:52.128799+0000","last_became_peered":"2024-09-16T14:48:52.128799+0000","last_unstale":"2024-09-16T14:59:17.468949+0000","last_undegraded":"2024-09-16T14:59:17.468949+0000","last_fullsized":"2024-09-16T14:59:17.468949+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:48:37.984512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":111,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:02.607606+0000","last_change":"2024-09-16T14:48:52.143178+0000","last_active":"2024-09-16T15:00:02.607606+0000","last_peered":"2024-09-16T15:00:02.607606+0000","last_clean":"2024-09-16T15:00:02.607606+0000","last_became_active":"2024-09-16T14:48:52.142697+0000","last_became_peered":"2024-09-16T14:48:52.142697+0000","last_unstale":"2024-09-16T15:00:02.607606+0000","last_undegraded":"2024-09-16T15:00:02.607606+0000","last_fullsized":"2024-09-16T15:00:02.607606+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:43:37.007895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":112,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:12.478953+0000","last_change":"2024-09-16T14:48:52.144132+0000","last_active":"2024-09-16T15:00:12.478953+0000","last_peered":"2024-09-16T15:00:12.478953+0000","last_clean":"2024-09-16T15:00:12.478953+0000","last_became_active":"2024-09-16T14:48:52.143756+0000","last_became_peered":"2024-09-16T14:48:52.143756+0000","last_unstale":"2024-09-16T15:00:12.478953+0000","last_undegraded":"2024-09-16T15:00:12.478953+0000","last_fullsized":"2024-09-16T15:00:12.478953+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:15.504836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":126,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:02.600738+0000","last_change":"2024-09-16T14:48:52.139682+0000","last_active":"2024-09-16T15:00:02.600738+0000","last_peered":"2024-09-16T15:00:02.600738+0000","last_clean":"2024-09-16T15:00:02.600738+0000","last_became_active":"2024-09-16T14:48:52.139205+0000","last_became_peered":"2024-09-16T14:48:52.139205+0000","last_unstale":"2024-09-16T15:00:02.600738+0000","last_undegraded":"2024-09-16T15:00:02.600738+0000","last_fullsized":"2024-09-16T15:00:02.600738+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:56.518201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":81,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468831+0000","last_change":"2024-09-16T14:48:52.131936+0000","last_active":"2024-09-16T14:59:17.468831+0000","last_peered":"2024-09-16T14:59:17.468831+0000","last_clean":"2024-09-16T14:59:17.468831+0000","last_became_active":"2024-09-16T14:48:52.131089+0000","last_became_peered":"2024-09-16T14:48:52.131089+0000","last_unstale":"2024-09-16T14:59:17.468831+0000","last_undegraded":"2024-09-16T14:59:17.468831+0000","last_fullsized":"2024-09-16T14:59:17.468831+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:04:08.492279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":82,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468863+0000","last_change":"2024-09-16T14:48:52.138889+0000","last_active":"2024-09-16T14:59:17.468863+0000","last_peered":"2024-09-16T14:59:17.468863+0000","last_clean":"2024-09-16T14:59:17.468863+0000","last_became_active":"2024-09-16T14:48:52.137830+0000","last_became_peered":"2024-09-16T14:48:52.137830+0000","last_unstale":"2024-09-16T14:59:17.468863+0000","last_undegraded":"2024-09-16T14:59:17.468863+0000","last_fullsized":"2024-09-16T14:59:17.468863+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:00:48.737166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":169,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:02.474335+0000","last_change":"2024-09-16T14:48:52.145132+0000","last_active":"2024-09-16T15:00:02.474335+0000","last_peered":"2024-09-16T15:00:02.474335+0000","last_clean":"2024-09-16T15:00:02.474335+0000","last_became_active":"2024-09-16T14:48:52.144280+0000","last_became_peered":"2024-09-16T14:48:52.144280+0000","last_unstale":"2024-09-16T15:00:02.474335+0000","last_undegraded":"2024-09-16T15:00:02.474335+0000","last_fullsized":"2024-09-16T15:00:02.474335+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:37:18.905007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":90,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.467831+0000","last_change":"2024-09-16T14:48:53.564947+0000","last_active":"2024-09-16T14:59:17.467831+0000","last_peered":"2024-09-16T14:59:17.467831+0000","last_clean":"2024-09-16T14:59:17.467831+0000","last_became_active":"2024-09-16T14:48:53.563655+0000","last_became_peered":"2024-09-16T14:48:53.563655+0000","last_unstale":"2024-09-16T14:59:17.467831+0000","last_undegraded":"2024-09-16T14:59:17.467831+0000","last_fullsized":"2024-09-16T14:59:17.467831+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:32:48.022677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":81,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:58:57.465826+0000","last_change":"2024-09-16T14:48:52.138403+0000","last_active":"2024-09-16T14:58:57.465826+0000","last_peered":"2024-09-16T14:58:57.465826+0000","last_clean":"2024-09-16T14:58:57.465826+0000","last_became_active":"2024-09-16T14:48:52.137512+0000","last_became_peered":"2024-09-16T14:48:52.137512+0000","last_unstale":"2024-09-16T14:58:57.465826+0000","last_undegraded":"2024-09-16T14:58:57.465826+0000","last_fullsized":"2024-09-16T14:58:57.465826+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:00:15.913134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":229,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:58:57.465176+0000","last_change":"2024-09-16T14:48:52.131578+0000","last_active":"2024-09-16T14:58:57.465176+0000","last_peered":"2024-09-16T14:58:57.465176+0000","last_clean":"2024-09-16T14:58:57.465176+0000","last_became_active":"2024-09-16T14:48:52.131259+0000","last_became_peered":"2024-09-16T14:48:52.131259+0000","last_unstale":"2024-09-16T14:58:57.465176+0000","last_undegraded":"2024-09-16T14:58:57.465176+0000","last_fullsized":"2024-09-16T14:58:57.465176+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:02:29.950099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067836399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":220,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.600970+0000","last_change":"2024-09-16T14:48:53.141414+0000","last_active":"2024-09-16T14:59:17.600970+0000","last_peered":"2024-09-16T14:59:17.600970+0000","last_clean":"2024-09-16T14:59:17.600970+0000","last_became_active":"2024-09-16T14:48:53.140914+0000","last_became_peered":"2024-09-16T14:48:53.140914+0000","last_unstale":"2024-09-16T14:59:17.600970+0000","last_undegraded":"2024-09-16T14:59:17.600970+0000","last_fullsized":"2024-09-16T14:59:17.600970+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:53:48.881111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097689999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.594663+0000","last_change":"2024-09-16T14:48:52.149868+0000","last_active":"2024-09-16T14:59:17.594663+0000","last_peered":"2024-09-16T14:59:17.594663+0000","last_clean":"2024-09-16T14:59:17.594663+0000","last_became_active":"2024-09-16T14:48:52.142312+0000","last_became_peered":"2024-09-16T14:48:52.142312+0000","last_unstale":"2024-09-16T14:59:17.594663+0000","last_undegraded":"2024-09-16T14:59:17.594663+0000","last_fullsized":"2024-09-16T14:59:17.594663+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:31:20.443534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":87,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468897+0000","last_change":"2024-09-16T14:48:52.151300+0000","last_active":"2024-09-16T14:59:17.468897+0000","last_peered":"2024-09-16T14:59:17.468897+0000","last_clean":"2024-09-16T14:59:17.468897+0000","last_became_active":"2024-09-16T14:48:52.137048+0000","last_became_peered":"2024-09-16T14:48:52.137048+0000","last_unstale":"2024-09-16T14:59:17.468897+0000","last_undegraded":"2024-09-16T14:59:17.468897+0000","last_fullsized":"2024-09-16T14:59:17.468897+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:17:13.952991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042440799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":428,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:58:37.462283+0000","last_change":"2024-09-16T14:48:55.155094+0000","last_active":"2024-09-16T14:58:37.462283+0000","last_peered":"2024-09-16T14:58:37.462283+0000","last_clean":"2024-09-16T14:58:37.462283+0000","last_became_active":"2024-09-16T14:48:55.154511+0000","last_became_peered":"2024-09-16T14:48:55.154511+0000","last_unstale":"2024-09-16T14:58:37.462283+0000","last_undegraded":"2024-09-16T14:58:37.462283+0000","last_fullsized":"2024-09-16T14:58:37.462283+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:28.462531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:28.462531+0000","last_clean_scrub_stamp":"2024-09-16T14:42:28.462531+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:06.351384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":81,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468924+0000","last_change":"2024-09-16T14:48:52.129414+0000","last_active":"2024-09-16T14:59:17.468924+0000","last_peered":"2024-09-16T14:59:17.468924+0000","last_clean":"2024-09-16T14:59:17.468924+0000","last_became_active":"2024-09-16T14:48:52.128554+0000","last_became_peered":"2024-09-16T14:48:52.128554+0000","last_unstale":"2024-09-16T14:59:17.468924+0000","last_undegraded":"2024-09-16T14:59:17.468924+0000","last_fullsized":"2024-09-16T14:59:17.468924+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:38:09.329918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039796999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":89,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.594692+0000","last_change":"2024-09-16T14:48:52.144047+0000","last_active":"2024-09-16T14:59:17.594692+0000","last_peered":"2024-09-16T14:59:17.594692+0000","last_clean":"2024-09-16T14:59:17.594692+0000","last_became_active":"2024-09-16T14:48:52.143705+0000","last_became_peered":"2024-09-16T14:48:52.143705+0000","last_unstale":"2024-09-16T14:59:17.594692+0000","last_undegraded":"2024-09-16T14:59:17.594692+0000","last_fullsized":"2024-09-16T14:59:17.594692+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:14:59.892564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061478100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":90,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:42.474287+0000","last_change":"2024-09-16T14:48:52.131363+0000","last_active":"2024-09-16T14:59:42.474287+0000","last_peered":"2024-09-16T14:59:42.474287+0000","last_clean":"2024-09-16T14:59:42.474287+0000","last_became_active":"2024-09-16T14:48:52.131160+0000","last_became_peered":"2024-09-16T14:48:52.131160+0000","last_unstale":"2024-09-16T14:59:42.474287+0000","last_undegraded":"2024-09-16T14:59:42.474287+0000","last_fullsized":"2024-09-16T14:59:42.474287+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:12.859518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065779100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":226,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:52.606162+0000","last_change":"2024-09-16T14:48:52.133533+0000","last_active":"2024-09-16T14:59:52.606162+0000","last_peered":"2024-09-16T14:59:52.606162+0000","last_clean":"2024-09-16T14:59:52.606162+0000","last_became_active":"2024-09-16T14:48:52.133197+0000","last_became_peered":"2024-09-16T14:48:52.133197+0000","last_unstale":"2024-09-16T14:59:52.606162+0000","last_undegraded":"2024-09-16T14:59:52.606162+0000","last_fullsized":"2024-09-16T14:59:52.606162+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:37:19.055419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085482099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":94400,"data_compressed_allocated":122880,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640499,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1413120,"data_stored":1289362,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.622,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.92600000000000005,"5min":1.042,"15min":1.2},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.81499999999999995,"15min":0.79100000000000004},"min":{"1min":0.627,"5min":0.51500000000000001,"15min":0.47899999999999998},"max":{"1min":0.95599999999999996,"5min":1.0369999999999999,"15min":1.3300000000000001},"last":0.95599999999999996}]},{"osd":1,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.81799999999999995,"15min":0.81000000000000005},"min":{"1min":0.61899999999999999,"5min":0.44900000000000001,"15min":0.33900000000000002},"max":{"1min":1.0589999999999999,"5min":1.073,"15min":1.2010000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.74399999999999999,"15min":0.752},"min":{"1min":0.60999999999999999,"5min":0.54000000000000004,"15min":0.438},"max":{"1min":0.90800000000000003,"5min":0.94299999999999995,"15min":1.089},"last":0.77300000000000002}]},{"osd":2,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.81799999999999995,"15min":0.82899999999999996},"min":{"1min":0.628,"5min":0.50800000000000001,"15min":0.44900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.272},"last":0.75},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.90600000000000003,"15min":0.91300000000000003},"min":{"1min":0.81000000000000005,"5min":0.52900000000000003,"15min":0.42899999999999999},"max":{"1min":1.0289999999999999,"5min":1.056,"15min":1.3540000000000001},"last":0.93000000000000005}]},{"osd":3,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69699999999999995,"15min":0.67600000000000005},"min":{"1min":0.52300000000000002,"5min":0.46600000000000003,"15min":0.26000000000000001},"max":{"1min":0.96799999999999997,"5min":0.96999999999999997,"15min":1.1020000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.68200000000000005,"15min":0.69099999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33500000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":1.153},"last":0.90000000000000002}]},{"osd":4,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72899999999999998,"15min":0.749},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.27500000000000002},"max":{"1min":0.95299999999999996,"5min":1.05,"15min":1.236},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.80000000000000004,"15min":0.77200000000000002},"min":{"1min":0.48799999999999999,"5min":0.44800000000000001,"15min":0.26300000000000001},"max":{"1min":1.038,"5min":1.0529999999999999,"15min":1.2949999999999999},"last":1.01}]}]},{"osd":4,"up_from":11,"seq":47244640499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1417216,"data_stored":1289381,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78200000000000003,"15min":0.77300000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.437},"max":{"1min":0.96699999999999997,"5min":1.0800000000000001,"15min":1.1579999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73899999999999999,"15min":0.79000000000000004},"min":{"1min":0.52600000000000002,"5min":0.48899999999999999,"15min":0.48499999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.1679999999999999},"last":0.73999999999999999}]},{"osd":1,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.85899999999999999,"15min":0.81599999999999995},"min":{"1min":0.71199999999999997,"5min":0.52800000000000002,"15min":0.42399999999999999},"max":{"1min":1.0429999999999999,"5min":1.0780000000000001,"15min":1.4630000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.79600000000000004,"15min":0.83399999999999996},"min":{"1min":0.65700000000000003,"5min":0.53700000000000003,"15min":0.50800000000000001},"max":{"1min":1.0149999999999999,"5min":1.024,"15min":1.2729999999999999},"last":0.92300000000000004}]},{"osd":2,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.84999999999999998,"15min":0.84899999999999998},"min":{"1min":0.68000000000000005,"5min":0.65500000000000003,"15min":0.47599999999999998},"max":{"1min":1.02,"5min":1.0449999999999999,"15min":1.536},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84799999999999998,"15min":0.85899999999999999},"min":{"1min":0.63500000000000001,"5min":0.56599999999999995,"15min":0.51200000000000001},"max":{"1min":0.97099999999999997,"5min":1.0720000000000001,"15min":1.29},"last":0.79300000000000004}]},{"osd":3,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.70999999999999996,"15min":0.68899999999999995},"min":{"1min":0.53700000000000003,"5min":0.51300000000000001,"15min":0.27700000000000002},"max":{"1min":0.83299999999999996,"5min":1.0720000000000001,"15min":1.2669999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72899999999999998,"15min":0.72499999999999998},"min":{"1min":0.55700000000000005,"5min":0.50600000000000001,"15min":0.25700000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.198},"last":0.69799999999999995}]},{"osd":5,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.75600000000000001,"15min":0.73099999999999998},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.28999999999999998},"max":{"1min":0.95799999999999996,"5min":1.0600000000000001,"15min":1.1140000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.94699999999999995,"15min":0.91700000000000004},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.24299999999999999},"max":{"1min":1.0700000000000001,"5min":1.169,"15min":1.3879999999999999},"last":0.96599999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640498,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32556,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339284,"statfs":{"total":96636764160,"available":96603426816,"internally_reserved":0,"allocated":1474560,"data_stored":1818290,"data_compressed":271867,"data_compressed_allocated":389120,"data_compressed_original":1138688,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.45200000000000001},"max":{"1min":0.998,"5min":0.998,"15min":1.2729999999999999},"last":0.622},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74099999999999999,"15min":0.753},"min":{"1min":0.60799999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90500000000000003,"5min":0.94199999999999995,"15min":1.306},"last":0.83499999999999996}]},{"osd":1,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90200000000000002,"15min":0.86599999999999999},"min":{"1min":0.55800000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.069,"5min":1.1259999999999999,"15min":1.294},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.86199999999999999,"15min":0.86599999999999999},"min":{"1min":0.67900000000000005,"5min":0.56200000000000006,"15min":0.41199999999999998},"max":{"1min":0.93999999999999995,"5min":1.046,"15min":1.9219999999999999},"last":0.72899999999999998}]},{"osd":2,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81799999999999995,"15min":0.81699999999999995},"min":{"1min":0.71299999999999997,"5min":0.61199999999999999,"15min":0.45900000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.181},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79900000000000004,"15min":0.80600000000000005},"min":{"1min":0.627,"5min":0.627,"15min":0.38800000000000001},"max":{"1min":0.90300000000000002,"5min":0.96899999999999997,"15min":2.444},"last":0.78000000000000003}]},{"osd":4,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.624,"15min":0.623},"min":{"1min":0.49299999999999999,"5min":0.47699999999999998,"15min":0.28199999999999997},"max":{"1min":0.89800000000000002,"5min":0.98799999999999999,"15min":1.0700000000000001},"last":0.75},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.67900000000000005,"5min":0.628,"15min":0.317},"max":{"1min":0.99299999999999999,"5min":1.0740000000000001,"15min":1.2290000000000001},"last":0.92000000000000004}]},{"osd":5,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69299999999999995,"15min":0.70699999999999996},"min":{"1min":0.55500000000000005,"5min":0.52400000000000002,"15min":0.29899999999999999},"max":{"1min":0.92200000000000004,"5min":1.4279999999999999,"15min":2.5409999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.83499999999999996,"15min":0.80900000000000005},"min":{"1min":0.63700000000000001,"5min":0.60999999999999999,"15min":0.33100000000000002},"max":{"1min":1.081,"5min":1.514,"15min":1.514},"last":0.871}]}]},{"osd":2,"up_from":11,"seq":47244640498,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1413120,"data_stored":1289362,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56200000000000006,"15min":0.58099999999999996},"min":{"1min":0.252,"5min":0.252,"15min":0.249},"max":{"1min":0.749,"5min":0.94699999999999995,"15min":1.819},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.77600000000000002,"15min":0.75700000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.312},"max":{"1min":1.147,"5min":1.147,"15min":1.444},"last":0.91900000000000004}]},{"osd":1,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.39200000000000002,"5min":0.379,"15min":0.28299999999999997},"max":{"1min":0.95499999999999996,"5min":0.95999999999999996,"15min":1.27},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.62,"15min":0.623},"min":{"1min":0.313,"5min":0.313,"15min":0.26700000000000002},"max":{"1min":1.0129999999999999,"5min":1.0189999999999999,"15min":1.383},"last":0.68000000000000005}]},{"osd":3,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.84199999999999997,"15min":0.82299999999999995},"min":{"1min":0.58599999999999997,"5min":0.56000000000000005,"15min":0.36899999999999999},"max":{"1min":1.0720000000000001,"5min":1.169,"15min":1.2110000000000001},"last":1.034},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.871,"15min":0.86299999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.44700000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.335},"last":0.72399999999999998}]},{"osd":4,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.83599999999999997,"15min":0.83499999999999996},"min":{"1min":0.53300000000000003,"5min":0.50600000000000001,"15min":0.45500000000000002},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.874},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.84999999999999998,"15min":0.85499999999999998},"min":{"1min":0.62,"5min":0.58599999999999997,"15min":0.40999999999999998},"max":{"1min":0.97099999999999997,"5min":1.0489999999999999,"15min":1.367},"last":0.84299999999999997}]},{"osd":5,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.88200000000000001,"15min":0.89100000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.53700000000000003},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.911},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.95399999999999996,"15min":0.95499999999999996},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.121,"5min":1.1990000000000001,"15min":2.0499999999999998},"last":0.90800000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640497,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1462272,"data_stored":1818290,"data_compressed":273302,"data_compressed_allocated":389120,"data_compressed_original":1150976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56000000000000005,"15min":0.57099999999999995},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.251},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":1.0720000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.628,"15min":0.63800000000000001},"min":{"1min":0.44500000000000001,"5min":0.36199999999999999,"15min":0.26100000000000001},"max":{"1min":0.87,"5min":0.87,"15min":1.0589999999999999},"last":0.81499999999999995}]},{"osd":2,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.66700000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.28699999999999998},"max":{"1min":0.82599999999999996,"5min":0.88200000000000001,"15min":1.052},"last":0.748},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.628,"15min":0.66700000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32100000000000001},"max":{"1min":0.81899999999999995,"5min":0.879,"15min":1.403},"last":0.83799999999999997}]},{"osd":3,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85699999999999998,"15min":0.88800000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.503},"max":{"1min":1.052,"5min":1.117,"15min":1.359},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.86299999999999999,"15min":0.877},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1459999999999999,"5min":1.147,"15min":1.284},"last":1.1719999999999999}]},{"osd":4,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.88900000000000001,"15min":0.88200000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.46600000000000003},"max":{"1min":1.0900000000000001,"5min":1.097,"15min":1.341},"last":1.111},{"interface":"front","average":{"1min":0.88,"5min":0.86199999999999999,"15min":0.86899999999999999},"min":{"1min":0.67600000000000005,"5min":0.66200000000000003,"15min":0.46999999999999997},"max":{"1min":1.056,"5min":1.0800000000000001,"15min":1.399},"last":1.0269999999999999}]},{"osd":5,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.622,"5min":0.622,"15min":0.50900000000000001},"max":{"1min":1,"5min":1.01,"15min":1.258},"last":1.054},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90100000000000002,"15min":0.90400000000000003},"min":{"1min":0.69499999999999995,"5min":0.629,"15min":0.47299999999999998},"max":{"1min":1.1519999999999999,"5min":1.411,"15min":1.411},"last":1.087}]}]},{"osd":0,"up_from":11,"seq":47244640498,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36596,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335244,"statfs":{"total":96636764160,"available":96599289856,"internally_reserved":0,"allocated":1417216,"data_stored":1289381,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57499999999999996,"15min":0.57999999999999996},"min":{"1min":0.36099999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.74299999999999999,"5min":0.77700000000000002,"15min":1.1739999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.61399999999999999,"15min":0.63800000000000001},"min":{"1min":0.38200000000000001,"5min":0.28299999999999997,"15min":0.27400000000000002},"max":{"1min":0.67800000000000005,"5min":1.032,"15min":1.21},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68700000000000006,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.29899999999999999,"15min":0.25900000000000001},"max":{"1min":0.85799999999999998,"5min":0.997,"15min":1.103},"last":0.495},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.62,"15min":0.67200000000000004},"min":{"1min":0.47999999999999998,"5min":0.36099999999999999,"15min":0.26900000000000002},"max":{"1min":0.69099999999999995,"5min":1.008,"15min":1.137},"last":0.44}]},{"osd":3,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.82599999999999996},"min":{"1min":0.67100000000000004,"5min":0.59599999999999997,"15min":0.52400000000000002},"max":{"1min":1.014,"5min":1.1040000000000001,"15min":1.4650000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.81799999999999995,"15min":0.83999999999999997},"min":{"1min":0.69299999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.99299999999999999,"5min":1.101,"15min":1.4950000000000001},"last":0.59199999999999997}]},{"osd":4,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.84599999999999997,"15min":0.84699999999999998},"min":{"1min":0.68899999999999995,"5min":0.57999999999999996,"15min":0.48999999999999999},"max":{"1min":0.98499999999999999,"5min":1.1559999999999999,"15min":1.3300000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.85099999999999998},"min":{"1min":0.63700000000000001,"5min":0.55700000000000005,"15min":0.51600000000000001},"max":{"1min":0.96099999999999997,"5min":1.095,"15min":1.4179999999999999},"last":0.46999999999999997}]},{"osd":5,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.91200000000000003,"15min":0.91400000000000003},"min":{"1min":0.70899999999999996,"5min":0.66400000000000003,"15min":0.57999999999999996},"max":{"1min":1.1080000000000001,"5min":1.304,"15min":1.6499999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95799999999999996,"15min":0.94799999999999995},"min":{"1min":0.76900000000000002,"5min":0.69999999999999996,"15min":0.51900000000000002},"max":{"1min":1.0589999999999999,"5min":1.2030000000000001,"15min":1.7290000000000001},"last":0.754}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:02:24.307 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:02:24.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.427+0000 7f558ec9b640 10 monclient: get_monmap_and_config 2024-09-16T15:02:24.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.427+0000 7f558ec9b640 10 monclient: build_initial_monmap 2024-09-16T15:02:24.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.427+0000 7f558ec9b640 10 monclient: monmap: 2024-09-16T15:02:24.431 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:24.431 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:24.431 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:24.432510+0000 2024-09-16T15:02:24.431 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:24.432510+0000 2024-09-16T15:02:24.431 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:24.431 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:24.431 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:24.431 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:24.431 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:24.431 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.427+0000 7f558ec9b640 1 Processor -- start 2024-09-16T15:02:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.427+0000 7f558ec9b640 1 -- start start 2024-09-16T15:02:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.427+0000 7f558ec9b640 10 monclient: init 2024-09-16T15:02:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:02:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 10 monclient(hunting): picked mon.noname-a con 0x7f558809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 10 monclient(hunting): picked mon.noname-b con 0x7f5588132650 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 10 monclient(hunting): picked mon.noname-c con 0x7f5588138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558809d260 con 0x7f558809cb80 2024-09-16T15:02:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5588132a80 con 0x7f5588138ff0 2024-09-16T15:02:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558809cf60 con 0x7f5588132650 2024-09-16T15:02:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 10 monclient(hunting): authenticate will time out at 2168.552528s 2024-09-16T15:02:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ca37640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f558809cb80 0x7f55880717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42198/0 (socket says 172.21.15.26:42198) 2024-09-16T15:02:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f5587fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5588132650 0x7f55881378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47926/0 (socket says 172.21.15.26:47926) 2024-09-16T15:02:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ca37640 1 -- 172.21.15.26:0/2455814089 learned_addr learned my addr 172.21.15.26:0/2455814089 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 1 -- 172.21.15.26:0/2455814089 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478625593 0 0) 0x7f558809cf60 con 0x7f5588132650 2024-09-16T15:02:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient(hunting): my global_id is 354287 2024-09-16T15:02:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 1 -- 172.21.15.26:0/2455814089 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5574003530 con 0x7f5588132650 2024-09-16T15:02:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 1 -- 172.21.15.26:0/2455814089 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719138689 0 0) 0x7f558809d260 con 0x7f558809cb80 2024-09-16T15:02:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient(hunting): my global_id is 334473 2024-09-16T15:02:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 1 -- 172.21.15.26:0/2455814089 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f558809cf60 con 0x7f558809cb80 2024-09-16T15:02:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 1 -- 172.21.15.26:0/2455814089 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 605660653 0 0) 0x7f5574003530 con 0x7f5588132650 2024-09-16T15:02:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 1 -- 172.21.15.26:0/2455814089 >> v1:172.21.15.189:6789/0 conn(0x7f5588138ff0 legacy=0x7f558813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 1 -- 172.21.15.26:0/2455814089 >> v1:172.21.15.26:6789/0 conn(0x7f558809cb80 legacy=0x7f55880717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient: _finish_hunting 0 2024-09-16T15:02:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 1 monclient: found mon.noname-b 2024-09-16T15:02:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:02:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 1 -- 172.21.15.26:0/2455814089 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558809f2d0 con 0x7f5588132650 2024-09-16T15:02:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient: _finish_auth 0 2024-09-16T15:02:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient: _check_auth_tickets 2024-09-16T15:02:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 1 -- 172.21.15.26:0/2455814089 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5578001660 con 0x7f5588132650 2024-09-16T15:02:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:02:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient: dump: 2024-09-16T15:02:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:24.442 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 5 monclient: authenticate success, global_id 354287 2024-09-16T15:02:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 1 -- 172.21.15.26:0/2455814089 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5578001880 con 0x7f5588132650 2024-09-16T15:02:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 1 -- 172.21.15.26:0/2455814089 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5578001b90 con 0x7f5588132650 2024-09-16T15:02:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 4 set_mon_vals no callback set 2024-09-16T15:02:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f55877fe640 10 monclient: dump: 2024-09-16T15:02:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:24.443 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 10 monclient: shutdown 2024-09-16T15:02:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 1 -- 172.21.15.26:0/2455814089 >> v1:172.21.15.26:6790/0 conn(0x7f5588132650 legacy=0x7f55881378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 1 -- 172.21.15.26:0/2455814089 shutdown_connections 2024-09-16T15:02:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 1 -- 172.21.15.26:0/2455814089 >> 172.21.15.26:0/2455814089 conn(0x7f558809fb50 msgr2=0x7f558809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 1 -- 172.21.15.26:0/2455814089 shutdown_connections 2024-09-16T15:02:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 1 -- 172.21.15.26:0/2455814089 wait complete. 2024-09-16T15:02:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 10 monclient: build_initial_monmap 2024-09-16T15:02:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 10 monclient: monmap: 2024-09-16T15:02:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:24.445 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 1 Processor -- start 2024-09-16T15:02:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 1 -- start start 2024-09-16T15:02:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.431+0000 7f558ec9b640 10 monclient: init 2024-09-16T15:02:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:02:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 10 monclient(hunting): picked mon.noname-a con 0x7f558809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 10 monclient(hunting): picked mon.noname-b con 0x7f5588132650 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 10 monclient(hunting): picked mon.noname-c con 0x7f5588138ff0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5588138b80 con 0x7f558809cb80 2024-09-16T15:02:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55880a2350 con 0x7f5588132650 2024-09-16T15:02:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558809b370 con 0x7f5588138ff0 2024-09-16T15:02:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 10 monclient(hunting): authenticate will time out at 2168.556182s 2024-09-16T15:02:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ca37640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f558809cb80 0x7f558809a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42202/0 (socket says 172.21.15.26:42202) 2024-09-16T15:02:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ca37640 1 -- 172.21.15.26:0/888830145 learned_addr learned my addr 172.21.15.26:0/888830145 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350377526 0 0) 0x7f558809b370 con 0x7f5588138ff0 2024-09-16T15:02:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient(hunting): _negotiate 2024-09-16T15:02:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient(hunting): my global_id is 354290 2024-09-16T15:02:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5568003530 con 0x7f5588138ff0 2024-09-16T15:02:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903506580 0 0) 0x7f5588138b80 con 0x7f558809cb80 2024-09-16T15:02:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient(hunting): _negotiate 2024-09-16T15:02:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient(hunting): my global_id is 334476 2024-09-16T15:02:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f558809b370 con 0x7f558809cb80 2024-09-16T15:02:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3562838908 0 0) 0x7f5568003530 con 0x7f5588138ff0 2024-09-16T15:02:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5588138b80 con 0x7f5588138ff0 2024-09-16T15:02:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5580001b80 con 0x7f5588138ff0 2024-09-16T15:02:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:02:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient(hunting): dump: 2024-09-16T15:02:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:24.448 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4180151231 0 0) 0x7f558809b370 con 0x7f558809cb80 2024-09-16T15:02:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5568003530 con 0x7f558809cb80 2024-09-16T15:02:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5570001880 con 0x7f558809cb80 2024-09-16T15:02:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient(hunting): dump: 2024-09-16T15:02:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:24.449 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1629626929 0 0) 0x7f5588138b80 con 0x7f5588138ff0 2024-09-16T15:02:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 >> v1:172.21.15.189:6789/0 conn(0x7f5588132650 legacy=0x7f558809ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 >> v1:172.21.15.26:6789/0 conn(0x7f558809cb80 legacy=0x7f558809a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient: _finish_hunting 0 2024-09-16T15:02:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 monclient: found mon.c 2024-09-16T15:02:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5588130330 con 0x7f5588138ff0 2024-09-16T15:02:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient: _finish_auth 0 2024-09-16T15:02:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient: _check_auth_tickets 2024-09-16T15:02:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 5 monclient: authenticate success, global_id 354290 2024-09-16T15:02:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 10 monclient: _renew_subs 2024-09-16T15:02:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 1 -- 172.21.15.26:0/888830145 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558809a240 con 0x7f5588138ff0 2024-09-16T15:02:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 10 monclient: _renew_subs 2024-09-16T15:02:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55800020b0 con 0x7f5588138ff0 2024-09-16T15:02:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 1 -- 172.21.15.26:0/888830145 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f558806dd70 con 0x7f5588138ff0 2024-09-16T15:02:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5580003b20 con 0x7f5588138ff0 2024-09-16T15:02:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 10 monclient: dump: 2024-09-16T15:02:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:24.452 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f5586ffd640 4 set_mon_vals no callback set 2024-09-16T15:02:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5580003ca0 con 0x7f5588138ff0 2024-09-16T15:02:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f5580042c20 con 0x7f5588138ff0 2024-09-16T15:02:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ec9b640 1 -- 172.21.15.26:0/888830145 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558809c960 con 0x7f5588138ff0 2024-09-16T15:02:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.435+0000 7f558ca37640 10 monclient: get_auth_request con 0x7f5568036270 auth_method 0 2024-09-16T15:02:24.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.439+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5580004470 con 0x7f5588138ff0 2024-09-16T15:02:24.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.439+0000 7f55857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:24.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.439+0000 7f55857fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:24.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.599+0000 7f558ec9b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T15:02:24.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.599+0000 7f558ec9b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T15:02:24.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.599+0000 7f558ec9b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:24.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.599+0000 7f558ec9b640 1 -- 172.21.15.26:0/888830145 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55880008e0 con 0x7f5588138ff0 2024-09-16T15:02:24.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.603+0000 7f55857fa640 1 -- 172.21.15.26:0/888830145 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) v1 ==== 74+0+20250 (unknown 2895976921 0 2895439187) 0x7f5580011be0 con 0x7f5588138ff0 2024-09-16T15:02:24.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.603+0000 7f55857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T15:02:24.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.603+0000 7f55857fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:02:24.605 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T15:02:24.605 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":257,"fsid":"febeb2df-62e5-4150-ac3f-7a627f391606","created":"2024-09-16T14:42:17.361142+0000","modified":"2024-09-16T14:51:21.293645+0000","last_up_change":"2024-09-16T14:42:26.444878+0000","last_in_change":"2024-09-16T14:47:00.709191+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":182,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":15,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-16T14:42:27.380911+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-16T14:42:28.450753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"fooset","create_time":"2024-09-16T14:47:29.657891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"barset","create_time":"2024-09-16T14:47:30.650918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a4578188-6e8b-430f-b70d-1c3480eb7bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6808","nonce":3182580250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6809","nonce":3182580250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6811","nonce":3182580250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6810","nonce":3182580250}]},"public_addr":"172.21.15.26:6808/3182580250","cluster_addr":"172.21.15.26:6809/3182580250","heartbeat_back_addr":"172.21.15.26:6811/3182580250","heartbeat_front_addr":"172.21.15.26:6810/3182580250","state":["exists","up"]},{"osd":1,"uuid":"1a96367f-73f1-4a99-9b85-47bf8e3d4cdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6800","nonce":3032704211}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6801","nonce":3032704211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6805","nonce":3032704211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6803","nonce":3032704211}]},"public_addr":"172.21.15.26:6800/3032704211","cluster_addr":"172.21.15.26:6801/3032704211","heartbeat_back_addr":"172.21.15.26:6805/3032704211","heartbeat_front_addr":"172.21.15.26:6803/3032704211","state":["exists","up"]},{"osd":2,"uuid":"3030f43d-f848-4189-be1e-6ea367941218","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6802","nonce":1811454603}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6804","nonce":1811454603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6807","nonce":1811454603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6806","nonce":1811454603}]},"public_addr":"172.21.15.26:6802/1811454603","cluster_addr":"172.21.15.26:6804/1811454603","heartbeat_back_addr":"172.21.15.26:6807/1811454603","heartbeat_front_addr":"172.21.15.26:6806/1811454603","state":["exists","up"]},{"osd":3,"uuid":"78731cf9-4ba7-4ce9-a483-080aa1136b65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6808","nonce":3651837298}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6809","nonce":3651837298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6811","nonce":3651837298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6810","nonce":3651837298}]},"public_addr":"172.21.15.189:6808/3651837298","cluster_addr":"172.21.15.189:6809/3651837298","heartbeat_back_addr":"172.21.15.189:6811/3651837298","heartbeat_front_addr":"172.21.15.189:6810/3651837298","state":["exists","up"]},{"osd":4,"uuid":"d74156df-be8d-4ad7-93dc-89ac28df70dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6800","nonce":1782891661}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6801","nonce":1782891661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6803","nonce":1782891661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6802","nonce":1782891661}]},"public_addr":"172.21.15.189:6800/1782891661","cluster_addr":"172.21.15.189:6801/1782891661","heartbeat_back_addr":"172.21.15.189:6803/1782891661","heartbeat_front_addr":"172.21.15.189:6802/1782891661","state":["exists","up"]},{"osd":5,"uuid":"ccad9c1e-247f-4ed4-8437-6aa5c8b515d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6804","nonce":3793314587}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6805","nonce":3793314587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6807","nonce":3793314587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6806","nonce":3793314587}]},"public_addr":"172.21.15.189:6804/3793314587","cluster_addr":"172.21.15.189:6805/3793314587","heartbeat_back_addr":"172.21.15.189:6807/3793314587","heartbeat_front_addr":"172.21.15.189:6806/3793314587","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"0689a78d-9a3c-48ff-a75a-01eedf6472fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.603081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.576634+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.472343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.602340+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.465228+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T14:42:24.494237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:02:24.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.603+0000 7f558ec9b640 1 -- 172.21.15.26:0/888830145 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5568036270 legacy=0x7f5568038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:24.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.603+0000 7f558ec9b640 10 monclient: shutdown 2024-09-16T15:02:24.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.603+0000 7f558ec9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:24.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.603+0000 7f558ec9b640 1 -- 172.21.15.26:0/888830145 >> v1:172.21.15.26:6790/0 conn(0x7f5588138ff0 legacy=0x7f55881d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:24.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.603+0000 7f558ec9b640 1 -- 172.21.15.26:0/888830145 shutdown_connections 2024-09-16T15:02:24.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.603+0000 7f558ec9b640 1 -- 172.21.15.26:0/888830145 >> 172.21.15.26:0/888830145 conn(0x7f558809fb50 msgr2=0x7f55881317e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:24.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.607+0000 7f558ec9b640 1 -- 172.21.15.26:0/888830145 shutdown_connections 2024-09-16T15:02:24.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:24.607+0000 7f558ec9b640 1 -- 172.21.15.26:0/888830145 wait complete. 2024-09-16T15:02:25.624 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T15:02:25.625 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T15:02:25.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 10 monclient: get_monmap_and_config 2024-09-16T15:02:25.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 10 monclient: build_initial_monmap 2024-09-16T15:02:25.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 10 monclient: monmap: 2024-09-16T15:02:25.764 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:25.764 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:25.765 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:25.766478+0000 2024-09-16T15:02:25.765 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:25.766478+0000 2024-09-16T15:02:25.765 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:25.765 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:25.765 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:25.765 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:25.765 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:25.765 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:25.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 1 Processor -- start 2024-09-16T15:02:25.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 1 -- start start 2024-09-16T15:02:25.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 10 monclient: init 2024-09-16T15:02:25.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:25.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:02:25.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 10 monclient(hunting): picked mon.noname-c con 0x7f38d809d9e0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:25.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 10 monclient(hunting): picked mon.noname-b con 0x7f38d809d080 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:25.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 10 monclient(hunting): picked mon.noname-a con 0x7f38d809e4d0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:25.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d809f5b0 con 0x7f38d809e4d0 2024-09-16T15:02:25.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d80a06c0 con 0x7f38d809d9e0 2024-09-16T15:02:25.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d80a0900 con 0x7f38d809d080 2024-09-16T15:02:25.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:25.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 10 monclient(hunting): authenticate will time out at 2169.886495s 2024-09-16T15:02:25.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d6d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f38d809e4d0 0x7f38d813b130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42208/0 (socket says 172.21.15.26:42208) 2024-09-16T15:02:25.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d5d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f38d809d080 0x7f38d8137a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47946/0 (socket says 172.21.15.26:47946) 2024-09-16T15:02:25.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d6d9d640 1 -- 172.21.15.26:0/214093015 learned_addr learned my addr 172.21.15.26:0/214093015 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:25.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 1 -- 172.21.15.26:0/214093015 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382438606 0 0) 0x7f38d809f5b0 con 0x7f38d809e4d0 2024-09-16T15:02:25.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient(hunting): _negotiate 2024-09-16T15:02:25.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:25.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:25.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient(hunting): my global_id is 334479 2024-09-16T15:02:25.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 1 -- 172.21.15.26:0/214093015 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38c0003530 con 0x7f38d809e4d0 2024-09-16T15:02:25.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 1 -- 172.21.15.26:0/214093015 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110839873 0 0) 0x7f38d80a0900 con 0x7f38d809d080 2024-09-16T15:02:25.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient(hunting): _negotiate 2024-09-16T15:02:25.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:25.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:25.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient(hunting): my global_id is 354293 2024-09-16T15:02:25.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 1 -- 172.21.15.26:0/214093015 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38d809f5b0 con 0x7f38d809d080 2024-09-16T15:02:25.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 1 -- 172.21.15.26:0/214093015 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841988031 0 0) 0x7f38d80a06c0 con 0x7f38d809d9e0 2024-09-16T15:02:25.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient(hunting): _negotiate 2024-09-16T15:02:25.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:25.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:25.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient(hunting): my global_id is 364159 2024-09-16T15:02:25.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 1 -- 172.21.15.26:0/214093015 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38d80a0900 con 0x7f38d809d9e0 2024-09-16T15:02:25.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 1 -- 172.21.15.26:0/214093015 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3882550485 0 0) 0x7f38c0003530 con 0x7f38d809e4d0 2024-09-16T15:02:25.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 1 -- 172.21.15.26:0/214093015 >> v1:172.21.15.26:6790/0 conn(0x7f38d809d080 legacy=0x7f38d8137a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:25.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 1 -- 172.21.15.26:0/214093015 >> v1:172.21.15.189:6789/0 conn(0x7f38d809d9e0 legacy=0x7f38d809ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:25.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient: _finish_hunting 0 2024-09-16T15:02:25.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 1 monclient: found mon.noname-a 2024-09-16T15:02:25.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:25.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:02:25.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 1 -- 172.21.15.26:0/214093015 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d809ed10 con 0x7f38d809e4d0 2024-09-16T15:02:25.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient: _finish_auth 0 2024-09-16T15:02:25.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient: _check_auth_tickets 2024-09-16T15:02:25.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:25.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 1 -- 172.21.15.26:0/214093015 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f38d00019a0 con 0x7f38d809e4d0 2024-09-16T15:02:25.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:25.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:25.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient: dump: 2024-09-16T15:02:25.772 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:25.772 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:25.772 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.772 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.772 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:25.772 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:25.772 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:25.772 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:25.772 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:25.772 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:25.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 5 monclient: authenticate success, global_id 334479 2024-09-16T15:02:25.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:25.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 1 -- 172.21.15.26:0/214093015 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38d0001c10 con 0x7f38d809e4d0 2024-09-16T15:02:25.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:25.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 1 -- 172.21.15.26:0/214093015 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f38d0001fe0 con 0x7f38d809e4d0 2024-09-16T15:02:25.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:25.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38dcc4f640 4 set_mon_vals no callback set 2024-09-16T15:02:25.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:25.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:25.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.763+0000 7f38d559a640 10 monclient: dump: 2024-09-16T15:02:25.773 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:25.773 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:25.773 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.773 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.773 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:25.773 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:25.774 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:25.774 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:25.774 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:25.774 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:25.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 10 monclient: shutdown 2024-09-16T15:02:25.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:25.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 1 -- 172.21.15.26:0/214093015 >> v1:172.21.15.26:6789/0 conn(0x7f38d809e4d0 legacy=0x7f38d813b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:25.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 1 -- 172.21.15.26:0/214093015 shutdown_connections 2024-09-16T15:02:25.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 1 -- 172.21.15.26:0/214093015 >> 172.21.15.26:0/214093015 conn(0x7f38d80a35e0 msgr2=0x7f38d809fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:25.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 1 -- 172.21.15.26:0/214093015 shutdown_connections 2024-09-16T15:02:25.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 1 -- 172.21.15.26:0/214093015 wait complete. 2024-09-16T15:02:25.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 10 monclient: build_initial_monmap 2024-09-16T15:02:25.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 10 monclient: monmap: 2024-09-16T15:02:25.775 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:25.775 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:25.775 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:25.775 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:25.775 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:25.775 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:25.775 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:25.775 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:25.775 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:25.776 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:25.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 1 Processor -- start 2024-09-16T15:02:25.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 1 -- start start 2024-09-16T15:02:25.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 10 monclient: init 2024-09-16T15:02:25.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:25.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:02:25.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 10 monclient(hunting): picked mon.noname-c con 0x7f38d809d080 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:25.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 10 monclient(hunting): picked mon.noname-b con 0x7f38d809d9e0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:25.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 10 monclient(hunting): picked mon.noname-a con 0x7f38d809e4d0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:25.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d813a220 con 0x7f38d809e4d0 2024-09-16T15:02:25.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d809b330 con 0x7f38d809d9e0 2024-09-16T15:02:25.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d8139f40 con 0x7f38d809d080 2024-09-16T15:02:25.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:25.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 10 monclient(hunting): authenticate will time out at 2169.890159s 2024-09-16T15:02:25.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38d6d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f38d809e4d0 0x7f38d81d17e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42210/0 (socket says 172.21.15.26:42210) 2024-09-16T15:02:25.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38d6d9d640 1 -- 172.21.15.26:0/145106418 learned_addr learned my addr 172.21.15.26:0/145106418 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:25.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38d659c640 1 --1- 172.21.15.26:0/145106418 >> v1:172.21.15.26:6790/0 conn(0x7f38d809d080 0x7f38d809a510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47956/0 (socket says 172.21.15.26:47956) 2024-09-16T15:02:25.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301490356 0 0) 0x7f38d813a220 con 0x7f38d809e4d0 2024-09-16T15:02:25.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:25.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:25.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:25.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): my global_id is 334482 2024-09-16T15:02:25.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b8003530 con 0x7f38d809e4d0 2024-09-16T15:02:25.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229505384 0 0) 0x7f38d8139f40 con 0x7f38d809d080 2024-09-16T15:02:25.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:25.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:25.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:25.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): my global_id is 354296 2024-09-16T15:02:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38d813a220 con 0x7f38d809d080 2024-09-16T15:02:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 24717843 0 0) 0x7f38d813a220 con 0x7f38d809d080 2024-09-16T15:02:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38d8139f40 con 0x7f38d809d080 2024-09-16T15:02:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f38cc0018c0 con 0x7f38d809d080 2024-09-16T15:02:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:02:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): dump: 2024-09-16T15:02:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:25.783 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 115154045 0 0) 0x7f38b8003530 con 0x7f38d809e4d0 2024-09-16T15:02:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38d813a220 con 0x7f38d809e4d0 2024-09-16T15:02:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f38d0001d80 con 0x7f38d809e4d0 2024-09-16T15:02:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): dump: 2024-09-16T15:02:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:25.784 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.784 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.784 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:25.784 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:25.784 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:25.784 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:25.784 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:25.784 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:25.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824769829 0 0) 0x7f38d809b330 con 0x7f38d809d9e0 2024-09-16T15:02:25.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:25.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:25.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:25.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient(hunting): my global_id is 364162 2024-09-16T15:02:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b8003530 con 0x7f38d809d9e0 2024-09-16T15:02:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3989325148 0 0) 0x7f38d8139f40 con 0x7f38d809d080 2024-09-16T15:02:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 >> v1:172.21.15.189:6789/0 conn(0x7f38d809d9e0 legacy=0x7f38d809ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 >> v1:172.21.15.26:6789/0 conn(0x7f38d809e4d0 legacy=0x7f38d81d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient: _finish_hunting 0 2024-09-16T15:02:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 monclient: found mon.c 2024-09-16T15:02:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d813a500 con 0x7f38d809d080 2024-09-16T15:02:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient: _finish_auth 0 2024-09-16T15:02:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient: _check_auth_tickets 2024-09-16T15:02:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 5 monclient: authenticate success, global_id 354296 2024-09-16T15:02:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 10 monclient: _renew_subs 2024-09-16T15:02:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 1 -- 172.21.15.26:0/145106418 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d813ac00 con 0x7f38d809d080 2024-09-16T15:02:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 10 monclient: _renew_subs 2024-09-16T15:02:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 1 -- 172.21.15.26:0/145106418 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d81389d0 con 0x7f38d809d080 2024-09-16T15:02:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38cc003620 con 0x7f38d809d080 2024-09-16T15:02:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f38cc0039b0 con 0x7f38d809d080 2024-09-16T15:02:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38d4d99640 4 set_mon_vals no callback set 2024-09-16T15:02:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38c6ffd640 10 monclient: dump: 2024-09-16T15:02:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:25.787 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.767+0000 7f38dcc4f640 1 -- 172.21.15.26:0/145106418 --> v1:172.21.15.26:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f38bc000f20 con 0x7f38d809d080 2024-09-16T15:02:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.771+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f38cc003b90 con 0x7f38d809d080 2024-09-16T15:02:25.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.771+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f38cc043dd0 con 0x7f38d809d080 2024-09-16T15:02:25.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.771+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 --> v1:172.21.15.26:6808/3182580250 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f38b8043c70 con 0x7f38b8040040 2024-09-16T15:02:25.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.771+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_get_version_reply(handle=1 version=257) v2 ==== 24+0+0 (unknown 1324813924 0 0) 0x7f38cc044210 con 0x7f38d809d080 2024-09-16T15:02:25.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.771+0000 7f38c6ffd640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-16T15:02:25.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.771+0000 7f38d6d9d640 10 monclient: get_auth_request con 0x7f38b8040040 auth_method 0 2024-09-16T15:02:25.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.771+0000 7f38d5d9b640 10 monclient: get_auth_request con 0x7f38b8036390 auth_method 0 2024-09-16T15:02:25.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.771+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 <== osd.0 v1:172.21.15.26:6808/3182580250 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f38b8043c70 con 0x7f38b8040040 2024-09-16T15:02:25.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.783+0000 7f38dcc4f640 1 -- 172.21.15.26:0/145106418 --> v1:172.21.15.26:6808/3182580250 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f38bc002b60 con 0x7f38b8040040 2024-09-16T15:02:25.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.783+0000 7f38c6ffd640 1 -- 172.21.15.26:0/145106418 <== osd.0 v1:172.21.15.26:6808/3182580250 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f38bc002b60 con 0x7f38b8040040 2024-09-16T15:02:25.789 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:02:25.789 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T15:02:25.789 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:02:25.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.783+0000 7f38dcc4f640 1 -- 172.21.15.26:0/145106418 >> v1:172.21.15.26:6808/3182580250 conn(0x7f38b8040040 legacy=0x7f38b80424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:25.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.783+0000 7f38dcc4f640 1 -- 172.21.15.26:0/145106418 >> v1:172.21.15.189:6812/3196161582 conn(0x7f38b8036390 legacy=0x7f38b8038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:25.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.783+0000 7f38dcc4f640 10 monclient: shutdown 2024-09-16T15:02:25.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.783+0000 7f38dcc4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:25.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.783+0000 7f38dcc4f640 1 -- 172.21.15.26:0/145106418 >> v1:172.21.15.26:6790/0 conn(0x7f38d809d080 legacy=0x7f38d809a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.783+0000 7f38d6d9d640 1 -- 172.21.15.26:0/145106418 reap_dead start 2024-09-16T15:02:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.783+0000 7f38dcc4f640 1 -- 172.21.15.26:0/145106418 shutdown_connections 2024-09-16T15:02:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.783+0000 7f38dcc4f640 1 -- 172.21.15.26:0/145106418 >> 172.21.15.26:0/145106418 conn(0x7f38d80a35e0 msgr2=0x7f38d81351b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.783+0000 7f38dcc4f640 1 -- 172.21.15.26:0/145106418 shutdown_connections 2024-09-16T15:02:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.783+0000 7f38dcc4f640 1 -- 172.21.15.26:0/145106418 wait complete. 2024-09-16T15:02:25.803 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T15:02:25.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28f85f640 10 monclient: get_monmap_and_config 2024-09-16T15:02:25.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28f85f640 10 monclient: build_initial_monmap 2024-09-16T15:02:25.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28f85f640 10 monclient: monmap: 2024-09-16T15:02:25.917 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:25.917 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:25.917 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:25.919150+0000 2024-09-16T15:02:25.917 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:25.919150+0000 2024-09-16T15:02:25.917 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:25.917 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:25.918 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:25.918 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:25.918 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:25.918 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28f85f640 1 Processor -- start 2024-09-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28f85f640 1 -- start start 2024-09-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28f85f640 10 monclient: init 2024-09-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28f85f640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28f85f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28f85f640 10 monclient(hunting): picked mon.noname-c con 0x7fc2880a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28f85f640 10 monclient(hunting): picked mon.noname-a con 0x7fc2880a0000 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28f85f640 10 monclient(hunting): picked mon.noname-b con 0x7fc2880a2800 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28f85f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2880a3250 con 0x7fc2880a0000 2024-09-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28f85f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28809f600 con 0x7fc2880a1d10 2024-09-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28f85f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2880a3830 con 0x7fc2880a2800 2024-09-16T15:02:25.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28f85f640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:25.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28f85f640 10 monclient(hunting): authenticate will time out at 2170.039808s 2024-09-16T15:02:25.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28ddfc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc2880a2800 0x7fc28813d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47962/0 (socket says 172.21.15.26:47962) 2024-09-16T15:02:25.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28cdfa640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc2880a0000 0x7fc288139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42222/0 (socket says 172.21.15.26:42222) 2024-09-16T15:02:25.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.915+0000 7fc28ddfc640 1 -- 172.21.15.26:0/2496034093 learned_addr learned my addr 172.21.15.26:0/2496034093 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:25.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 1 -- 172.21.15.26:0/2496034093 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784610730 0 0) 0x7fc2880a3250 con 0x7fc2880a0000 2024-09-16T15:02:25.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:25.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:25.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:25.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient(hunting): my global_id is 334485 2024-09-16T15:02:25.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 1 -- 172.21.15.26:0/2496034093 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc274003530 con 0x7fc2880a0000 2024-09-16T15:02:25.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 1 -- 172.21.15.26:0/2496034093 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696911237 0 0) 0x7fc2880a3830 con 0x7fc2880a2800 2024-09-16T15:02:25.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:25.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:25.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:25.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient(hunting): my global_id is 354299 2024-09-16T15:02:25.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 1 -- 172.21.15.26:0/2496034093 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2880a3250 con 0x7fc2880a2800 2024-09-16T15:02:25.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 1 -- 172.21.15.26:0/2496034093 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2223051232 0 0) 0x7fc2880a3250 con 0x7fc2880a2800 2024-09-16T15:02:25.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 1 -- 172.21.15.26:0/2496034093 >> v1:172.21.15.189:6789/0 conn(0x7fc2880a1d10 legacy=0x7fc2880a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:25.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 1 -- 172.21.15.26:0/2496034093 >> v1:172.21.15.26:6789/0 conn(0x7fc2880a0000 legacy=0x7fc288139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:25.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient: _finish_hunting 0 2024-09-16T15:02:25.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 1 monclient: found mon.noname-b 2024-09-16T15:02:25.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:25.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:02:25.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 1 -- 172.21.15.26:0/2496034093 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28809ed60 con 0x7fc2880a2800 2024-09-16T15:02:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient: _finish_auth 0 2024-09-16T15:02:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient: _check_auth_tickets 2024-09-16T15:02:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 1 -- 172.21.15.26:0/2496034093 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc2800019a0 con 0x7fc2880a2800 2024-09-16T15:02:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 5 monclient: authenticate success, global_id 354299 2024-09-16T15:02:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:02:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient: dump: 2024-09-16T15:02:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:25.927 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:25.928 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:25.928 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:25.928 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:25.928 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:25.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:25.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 1 -- 172.21.15.26:0/2496034093 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc280001b50 con 0x7fc2880a2800 2024-09-16T15:02:25.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:25.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 1 -- 172.21.15.26:0/2496034093 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc280001f20 con 0x7fc2880a2800 2024-09-16T15:02:25.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:25.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 4 set_mon_vals no callback set 2024-09-16T15:02:25.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:25.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:25.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27ffff640 10 monclient: dump: 2024-09-16T15:02:25.928 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:25.929 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:25.929 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.929 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.929 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:25.929 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:25.929 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:25.929 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:25.929 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:25.929 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:25.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 10 monclient: shutdown 2024-09-16T15:02:25.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:25.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 1 -- 172.21.15.26:0/2496034093 >> v1:172.21.15.26:6790/0 conn(0x7fc2880a2800 legacy=0x7fc28813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:25.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 1 -- 172.21.15.26:0/2496034093 shutdown_connections 2024-09-16T15:02:25.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 1 -- 172.21.15.26:0/2496034093 >> 172.21.15.26:0/2496034093 conn(0x7fc2880a7f30 msgr2=0x7fc2880a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:25.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 1 -- 172.21.15.26:0/2496034093 shutdown_connections 2024-09-16T15:02:25.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 1 -- 172.21.15.26:0/2496034093 wait complete. 2024-09-16T15:02:25.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 10 monclient: build_initial_monmap 2024-09-16T15:02:25.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 10 monclient: monmap: 2024-09-16T15:02:25.930 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:25.930 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:25.930 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:25.930 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:25.930 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:25.930 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:25.931 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:25.931 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:25.931 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:25.931 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:25.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 1 Processor -- start 2024-09-16T15:02:25.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 1 -- start start 2024-09-16T15:02:25.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 10 monclient: init 2024-09-16T15:02:25.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:25.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:02:25.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 10 monclient(hunting): picked mon.noname-b con 0x7fc2880a0000 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:25.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 10 monclient(hunting): picked mon.noname-a con 0x7fc2880a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:25.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 10 monclient(hunting): picked mon.noname-c con 0x7fc2880a2800 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:25.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28813cfd0 con 0x7fc2880a1d10 2024-09-16T15:02:25.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc288139ed0 con 0x7fc2880a0000 2024-09-16T15:02:25.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2880a6590 con 0x7fc2880a2800 2024-09-16T15:02:25.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:25.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 10 monclient(hunting): authenticate will time out at 2170.042762s 2024-09-16T15:02:25.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28cdfa640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc2880a1d10 0x7fc2881397c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42232/0 (socket says 172.21.15.26:42232) 2024-09-16T15:02:25.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28cdfa640 1 -- 172.21.15.26:0/3735320152 learned_addr learned my addr 172.21.15.26:0/3735320152 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:25.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28ddfc640 1 --1- 172.21.15.26:0/3735320152 >> v1:172.21.15.26:6790/0 conn(0x7fc2880a2800 0x7fc2881d19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47970/0 (socket says 172.21.15.26:47970) 2024-09-16T15:02:25.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162293723 0 0) 0x7fc28813cfd0 con 0x7fc2880a1d10 2024-09-16T15:02:25.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): _negotiate 2024-09-16T15:02:25.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:25.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:25.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): my global_id is 334488 2024-09-16T15:02:25.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc26c003530 con 0x7fc2880a1d10 2024-09-16T15:02:25.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224372590 0 0) 0x7fc2880a6590 con 0x7fc2880a2800 2024-09-16T15:02:25.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): _negotiate 2024-09-16T15:02:25.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:25.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:25.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): my global_id is 354302 2024-09-16T15:02:25.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc28813cfd0 con 0x7fc2880a2800 2024-09-16T15:02:25.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 888528814 0 0) 0x7fc28813cfd0 con 0x7fc2880a2800 2024-09-16T15:02:25.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2880a6590 con 0x7fc2880a2800 2024-09-16T15:02:25.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc2800017c0 con 0x7fc2880a2800 2024-09-16T15:02:25.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:25.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:02:25.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): dump: 2024-09-16T15:02:25.933 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:25.933 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:25.934 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.934 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.934 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:25.934 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:25.934 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:25.934 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:25.934 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:25.934 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:25.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:25.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4225230576 0 0) 0x7fc26c003530 con 0x7fc2880a1d10 2024-09-16T15:02:25.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc28813cfd0 con 0x7fc2880a1d10 2024-09-16T15:02:25.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc278001370 con 0x7fc2880a1d10 2024-09-16T15:02:25.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:25.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): dump: 2024-09-16T15:02:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:25.935 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396192588 0 0) 0x7fc288139ed0 con 0x7fc2880a0000 2024-09-16T15:02:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): _negotiate 2024-09-16T15:02:25.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient(hunting): my global_id is 364168 2024-09-16T15:02:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc26c003530 con 0x7fc2880a0000 2024-09-16T15:02:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1794131707 0 0) 0x7fc2880a6590 con 0x7fc2880a2800 2024-09-16T15:02:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 >> v1:172.21.15.189:6789/0 conn(0x7fc2880a0000 legacy=0x7fc2881390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 >> v1:172.21.15.26:6789/0 conn(0x7fc2880a1d10 legacy=0x7fc2881397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient: _finish_hunting 0 2024-09-16T15:02:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 monclient: found mon.c 2024-09-16T15:02:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28809e500 con 0x7fc2880a2800 2024-09-16T15:02:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient: _finish_auth 0 2024-09-16T15:02:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient: _check_auth_tickets 2024-09-16T15:02:25.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:25.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 5 monclient: authenticate success, global_id 354302 2024-09-16T15:02:25.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 10 monclient: _renew_subs 2024-09-16T15:02:25.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:25.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 1 -- 172.21.15.26:0/3735320152 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28813b5c0 con 0x7fc2880a2800 2024-09-16T15:02:25.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 10 monclient: _renew_subs 2024-09-16T15:02:25.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:25.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc28f85f640 1 -- 172.21.15.26:0/3735320152 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28813ba60 con 0x7fc2880a2800 2024-09-16T15:02:25.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc280001920 con 0x7fc2880a2800 2024-09-16T15:02:25.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:25.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc280004fc0 con 0x7fc2880a2800 2024-09-16T15:02:25.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:25.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:25.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27f7fe640 4 set_mon_vals no callback set 2024-09-16T15:02:25.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.919+0000 7fc27dffb640 10 monclient: dump: 2024-09-16T15:02:25.938 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:25.938 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:25.938 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.938 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:25.938 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:25.938 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:25.938 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:25.938 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:25.938 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:25.938 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:25.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.923+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc28000fa30 con 0x7fc2880a2800 2024-09-16T15:02:25.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.923+0000 7fc28f85f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:25.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.923+0000 7fc28f85f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:25.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.923+0000 7fc28f85f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:25.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.923+0000 7fc28f85f640 1 -- 172.21.15.26:0/3735320152 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc268005050 con 0x7fc2880a2800 2024-09-16T15:02:25.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.923+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fc2800440a0 con 0x7fc2880a2800 2024-09-16T15:02:25.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.923+0000 7fc28d5fb640 10 monclient: get_auth_request con 0x7fc26c0365c0 auth_method 0 2024-09-16T15:02:25.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.927+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc280014820 con 0x7fc2880a2800 2024-09-16T15:02:25.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.927+0000 7fc27dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:25.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:25.927+0000 7fc27dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:26.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.079+0000 7fc28f85f640 1 -- 172.21.15.26:0/3735320152 --> v1:172.21.15.189:6812/3196161582 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fc268000ba0 con 0x7fc26c0365c0 2024-09-16T15:02:26.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.079+0000 7fc27dffb640 1 -- 172.21.15.26:0/3735320152 <== mgr.4105 v1:172.21.15.189:6812/3196161582 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (unknown 2057319476 0 0) 0x7fc268000ba0 con 0x7fc26c0365c0 2024-09-16T15:02:26.082 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T15:02:26.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.083+0000 7fc28f85f640 1 -- 172.21.15.26:0/3735320152 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc26c0365c0 legacy=0x7fc26c038a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.083+0000 7fc28f85f640 10 monclient: shutdown 2024-09-16T15:02:26.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.083+0000 7fc28f85f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:26.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.083+0000 7fc28f85f640 1 -- 172.21.15.26:0/3735320152 >> v1:172.21.15.26:6790/0 conn(0x7fc2880a2800 legacy=0x7fc2881d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.083+0000 7fc28f85f640 1 -- 172.21.15.26:0/3735320152 shutdown_connections 2024-09-16T15:02:26.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.083+0000 7fc28f85f640 1 -- 172.21.15.26:0/3735320152 >> 172.21.15.26:0/3735320152 conn(0x7fc2880a7f30 msgr2=0x7fc28813e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:26.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.083+0000 7fc28f85f640 1 -- 172.21.15.26:0/3735320152 shutdown_connections 2024-09-16T15:02:26.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.083+0000 7fc28f85f640 1 -- 172.21.15.26:0/3735320152 wait complete. 2024-09-16T15:02:26.100 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T15:02:26.100 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T15:02:26.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 10 monclient: get_monmap_and_config 2024-09-16T15:02:26.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 10 monclient: build_initial_monmap 2024-09-16T15:02:26.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 10 monclient: monmap: 2024-09-16T15:02:26.215 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:26.217721+0000 2024-09-16T15:02:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:26.217721+0000 2024-09-16T15:02:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:26.216 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:26.216 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 1 Processor -- start 2024-09-16T15:02:26.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 1 -- start start 2024-09-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 10 monclient: init 2024-09-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 10 monclient(hunting): picked mon.noname-a con 0x7f056009a4e0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 10 monclient(hunting): picked mon.noname-c con 0x7f056009afd0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 10 monclient(hunting): picked mon.noname-b con 0x7f0560138110 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056009b810 con 0x7f056009a4e0 2024-09-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056009ecf0 con 0x7f056009afd0 2024-09-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056009baf0 con 0x7f0560138110 2024-09-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 10 monclient(hunting): authenticate will time out at 2170.337700s 2024-09-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f05657b6640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f056009a4e0 0x7f056009a8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42236/0 (socket says 172.21.15.26:42236) 2024-09-16T15:02:26.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f05657b6640 1 -- 172.21.15.26:0/3493249723 learned_addr learned my addr 172.21.15.26:0/3493249723 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 1 -- 172.21.15.26:0/3493249723 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598716765 0 0) 0x7f056009baf0 con 0x7f0560138110 2024-09-16T15:02:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient(hunting): my global_id is 354305 2024-09-16T15:02:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 1 -- 172.21.15.26:0/3493249723 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f054c003530 con 0x7f0560138110 2024-09-16T15:02:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 1 -- 172.21.15.26:0/3493249723 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493353320 0 0) 0x7f056009b810 con 0x7f056009a4e0 2024-09-16T15:02:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient(hunting): my global_id is 334491 2024-09-16T15:02:26.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 1 -- 172.21.15.26:0/3493249723 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f056009baf0 con 0x7f056009a4e0 2024-09-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 1 -- 172.21.15.26:0/3493249723 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3188197215 0 0) 0x7f054c003530 con 0x7f0560138110 2024-09-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 1 -- 172.21.15.26:0/3493249723 >> v1:172.21.15.189:6789/0 conn(0x7f056009afd0 legacy=0x7f05601369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 1 -- 172.21.15.26:0/3493249723 >> v1:172.21.15.26:6789/0 conn(0x7f056009a4e0 legacy=0x7f056009a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient: _finish_hunting 0 2024-09-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 1 monclient: found mon.noname-b 2024-09-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 1 -- 172.21.15.26:0/3493249723 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056009bdd0 con 0x7f0560138110 2024-09-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient: _finish_auth 0 2024-09-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient: _check_auth_tickets 2024-09-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 5 monclient: authenticate success, global_id 354305 2024-09-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 1 -- 172.21.15.26:0/3493249723 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f05580019a0 con 0x7f0560138110 2024-09-16T15:02:26.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient: dump: 2024-09-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:26.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 1 -- 172.21.15.26:0/3493249723 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0558001bd0 con 0x7f0560138110 2024-09-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 1 -- 172.21.15.26:0/3493249723 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0558001fa0 con 0x7f0560138110 2024-09-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 4 set_mon_vals no callback set 2024-09-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0557fff640 10 monclient: dump: 2024-09-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.224 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 10 monclient: shutdown 2024-09-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 1 -- 172.21.15.26:0/3493249723 >> v1:172.21.15.26:6790/0 conn(0x7f0560138110 legacy=0x7f056013a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 1 -- 172.21.15.26:0/3493249723 shutdown_connections 2024-09-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 1 -- 172.21.15.26:0/3493249723 >> 172.21.15.26:0/3493249723 conn(0x7f05600a35e0 msgr2=0x7f056009fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 1 -- 172.21.15.26:0/3493249723 shutdown_connections 2024-09-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 1 -- 172.21.15.26:0/3493249723 wait complete. 2024-09-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 10 monclient: build_initial_monmap 2024-09-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.215+0000 7f0567a1a640 10 monclient: monmap: 2024-09-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:26.225 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:26.226 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 1 Processor -- start 2024-09-16T15:02:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 1 -- start start 2024-09-16T15:02:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 10 monclient: init 2024-09-16T15:02:26.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:02:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 10 monclient(hunting): picked mon.noname-c con 0x7f056009a4e0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 10 monclient(hunting): picked mon.noname-b con 0x7f056009afd0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 10 monclient(hunting): picked mon.noname-a con 0x7f0560138110 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056006c9c0 con 0x7f0560138110 2024-09-16T15:02:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056011ce20 con 0x7f056009afd0 2024-09-16T15:02:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f056011a450 con 0x7f056009a4e0 2024-09-16T15:02:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 10 monclient(hunting): authenticate will time out at 2170.341299s 2024-09-16T15:02:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0565fb7640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0560138110 0x7f05601ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42246/0 (socket says 172.21.15.26:42246) 2024-09-16T15:02:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f05657b6640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f056009a4e0 0x7f0560119630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47982/0 (socket says 172.21.15.26:47982) 2024-09-16T15:02:26.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0565fb7640 1 -- 172.21.15.26:0/1402512468 learned_addr learned my addr 172.21.15.26:0/1402512468 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511477232 0 0) 0x7f056011a450 con 0x7f056009a4e0 2024-09-16T15:02:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): my global_id is 354308 2024-09-16T15:02:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0544003530 con 0x7f056009a4e0 2024-09-16T15:02:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098273236 0 0) 0x7f056006c9c0 con 0x7f0560138110 2024-09-16T15:02:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): my global_id is 334494 2024-09-16T15:02:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f056011a450 con 0x7f0560138110 2024-09-16T15:02:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3718652097 0 0) 0x7f0544003530 con 0x7f056009a4e0 2024-09-16T15:02:26.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f056006c9c0 con 0x7f056009a4e0 2024-09-16T15:02:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f05500017e0 con 0x7f056009a4e0 2024-09-16T15:02:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:02:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): dump: 2024-09-16T15:02:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.232 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.232 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 675953818 0 0) 0x7f056011a450 con 0x7f0560138110 2024-09-16T15:02:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0544003530 con 0x7f0560138110 2024-09-16T15:02:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0558001ee0 con 0x7f0560138110 2024-09-16T15:02:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): dump: 2024-09-16T15:02:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.234 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.234 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.234 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.234 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212278025 0 0) 0x7f056011ce20 con 0x7f056009afd0 2024-09-16T15:02:26.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient(hunting): my global_id is 364174 2024-09-16T15:02:26.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f056011a450 con 0x7f056009afd0 2024-09-16T15:02:26.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 604512338 0 0) 0x7f056006c9c0 con 0x7f056009a4e0 2024-09-16T15:02:26.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 >> v1:172.21.15.189:6789/0 conn(0x7f056009afd0 legacy=0x7f0560119d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 >> v1:172.21.15.26:6789/0 conn(0x7f0560138110 legacy=0x7f05601ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient: _finish_hunting 0 2024-09-16T15:02:26.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 monclient: found mon.c 2024-09-16T15:02:26.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:26.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:26.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05600a2520 con 0x7f056009a4e0 2024-09-16T15:02:26.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient: _finish_auth 0 2024-09-16T15:02:26.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:02:26.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:26.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 5 monclient: authenticate success, global_id 354308 2024-09-16T15:02:26.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 10 monclient: _renew_subs 2024-09-16T15:02:26.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:26.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 1 -- 172.21.15.26:0/1402512468 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056009a1d0 con 0x7f056009a4e0 2024-09-16T15:02:26.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 10 monclient: _renew_subs 2024-09-16T15:02:26.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:26.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 1 -- 172.21.15.26:0/1402512468 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f056011cb40 con 0x7f056009a4e0 2024-09-16T15:02:26.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0550001cb0 con 0x7f056009a4e0 2024-09-16T15:02:26.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:26.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f0550004990 con 0x7f056009a4e0 2024-09-16T15:02:26.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:26.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f05577fe640 4 set_mon_vals no callback set 2024-09-16T15:02:26.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient: dump: 2024-09-16T15:02:26.236 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.236 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.236 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.236 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.236 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.236 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.237 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.237 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.237 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.237 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:26.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0567a1a640 1 -- 172.21.15.26:0/1402512468 --> v1:172.21.15.26:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0540000f20 con 0x7f056009a4e0 2024-09-16T15:02:26.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f055000f400 con 0x7f056009a4e0 2024-09-16T15:02:26.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f0550043c70 con 0x7f056009a4e0 2024-09-16T15:02:26.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 --> v1:172.21.15.26:6800/3032704211 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0544046000 con 0x7f0544042330 2024-09-16T15:02:26.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_get_version_reply(handle=1 version=257) v2 ==== 24+0+0 (unknown 1324813924 0 0) 0x7f0550044040 con 0x7f056009a4e0 2024-09-16T15:02:26.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0555ffb640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-16T15:02:26.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0565fb7640 10 monclient: get_auth_request con 0x7f0544042330 auth_method 0 2024-09-16T15:02:26.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.219+0000 7f0564fb5640 10 monclient: get_auth_request con 0x7f0544038680 auth_method 0 2024-09-16T15:02:26.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.223+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 <== osd.1 v1:172.21.15.26:6800/3032704211 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f0544046000 con 0x7f0544042330 2024-09-16T15:02:26.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.235+0000 7f0567a1a640 1 -- 172.21.15.26:0/1402512468 --> v1:172.21.15.26:6800/3032704211 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0540002b60 con 0x7f0544042330 2024-09-16T15:02:26.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.235+0000 7f0555ffb640 1 -- 172.21.15.26:0/1402512468 <== osd.1 v1:172.21.15.26:6800/3032704211 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f0540002b60 con 0x7f0544042330 2024-09-16T15:02:26.240 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:02:26.240 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T15:02:26.240 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:02:26.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.235+0000 7f0567a1a640 1 -- 172.21.15.26:0/1402512468 >> v1:172.21.15.26:6800/3032704211 conn(0x7f0544042330 legacy=0x7f0544044790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.235+0000 7f0567a1a640 1 -- 172.21.15.26:0/1402512468 >> v1:172.21.15.189:6812/3196161582 conn(0x7f0544038680 legacy=0x7f054403ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.235+0000 7f0567a1a640 10 monclient: shutdown 2024-09-16T15:02:26.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.235+0000 7f0567a1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:26.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.235+0000 7f0567a1a640 1 -- 172.21.15.26:0/1402512468 >> v1:172.21.15.26:6790/0 conn(0x7f056009a4e0 legacy=0x7f0560119630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.235+0000 7f0565fb7640 1 -- 172.21.15.26:0/1402512468 reap_dead start 2024-09-16T15:02:26.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.235+0000 7f0567a1a640 1 -- 172.21.15.26:0/1402512468 shutdown_connections 2024-09-16T15:02:26.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.235+0000 7f0567a1a640 1 -- 172.21.15.26:0/1402512468 >> 172.21.15.26:0/1402512468 conn(0x7f05600a35e0 msgr2=0x7f05600a1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:26.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.235+0000 7f0567a1a640 1 -- 172.21.15.26:0/1402512468 shutdown_connections 2024-09-16T15:02:26.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.235+0000 7f0567a1a640 1 -- 172.21.15.26:0/1402512468 wait complete. 2024-09-16T15:02:26.251 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T15:02:26.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.359+0000 7fadedc3a640 10 monclient: get_monmap_and_config 2024-09-16T15:02:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.359+0000 7fadedc3a640 10 monclient: build_initial_monmap 2024-09-16T15:02:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.359+0000 7fadedc3a640 10 monclient: monmap: 2024-09-16T15:02:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:26.363848+0000 2024-09-16T15:02:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:26.363848+0000 2024-09-16T15:02:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:26.362 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.359+0000 7fadedc3a640 1 Processor -- start 2024-09-16T15:02:26.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.359+0000 7fadedc3a640 1 -- start start 2024-09-16T15:02:26.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.359+0000 7fadedc3a640 10 monclient: init 2024-09-16T15:02:26.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.359+0000 7fadedc3a640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:26.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.359+0000 7fadedc3a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:02:26.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.359+0000 7fadedc3a640 10 monclient(hunting): picked mon.noname-b con 0x7fade80a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:26.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.359+0000 7fadedc3a640 10 monclient(hunting): picked mon.noname-c con 0x7fade80a0000 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:26.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.359+0000 7fadedc3a640 10 monclient(hunting): picked mon.noname-a con 0x7fade80a2800 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:26.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.359+0000 7fadedc3a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade80a3250 con 0x7fade80a2800 2024-09-16T15:02:26.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade809f600 con 0x7fade80a0000 2024-09-16T15:02:26.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade80a3830 con 0x7fade80a1d10 2024-09-16T15:02:26.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:26.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 10 monclient(hunting): authenticate will time out at 2170.484044s 2024-09-16T15:02:26.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade77fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fade80a1d10 0x7fade80a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47998/0 (socket says 172.21.15.26:47998) 2024-09-16T15:02:26.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade77fe640 1 -- 172.21.15.26:0/535614801 learned_addr learned my addr 172.21.15.26:0/535614801 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:26.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 1 -- 172.21.15.26:0/535614801 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984356502 0 0) 0x7fade80a3830 con 0x7fade80a1d10 2024-09-16T15:02:26.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient(hunting): my global_id is 354311 2024-09-16T15:02:26.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 1 -- 172.21.15.26:0/535614801 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd0003530 con 0x7fade80a1d10 2024-09-16T15:02:26.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 1 -- 172.21.15.26:0/535614801 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664837483 0 0) 0x7fade80a3250 con 0x7fade80a2800 2024-09-16T15:02:26.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient(hunting): my global_id is 334497 2024-09-16T15:02:26.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 1 -- 172.21.15.26:0/535614801 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade80a3830 con 0x7fade80a2800 2024-09-16T15:02:26.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 1 -- 172.21.15.26:0/535614801 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3081941654 0 0) 0x7fade80a3830 con 0x7fade80a2800 2024-09-16T15:02:26.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 1 -- 172.21.15.26:0/535614801 >> v1:172.21.15.26:6790/0 conn(0x7fade80a1d10 legacy=0x7fade80a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 1 -- 172.21.15.26:0/535614801 >> v1:172.21.15.189:6789/0 conn(0x7fade80a0000 legacy=0x7fade8139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient: _finish_hunting 0 2024-09-16T15:02:26.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 1 monclient: found mon.noname-a 2024-09-16T15:02:26.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:26.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:02:26.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 1 -- 172.21.15.26:0/535614801 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade809ed60 con 0x7fade80a2800 2024-09-16T15:02:26.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient: _finish_auth 0 2024-09-16T15:02:26.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient: _check_auth_tickets 2024-09-16T15:02:26.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:26.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 5 monclient: authenticate success, global_id 334497 2024-09-16T15:02:26.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 1 -- 172.21.15.26:0/535614801 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fade00019a0 con 0x7fade80a2800 2024-09-16T15:02:26.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:26.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:26.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient: dump: 2024-09-16T15:02:26.369 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.369 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.369 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.369 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.369 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.369 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.369 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.369 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.369 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.369 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:26.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 1 -- 172.21.15.26:0/535614801 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fade0001c40 con 0x7fade80a2800 2024-09-16T15:02:26.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:26.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:26.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 4 set_mon_vals no callback set 2024-09-16T15:02:26.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 1 -- 172.21.15.26:0/535614801 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fade0002010 con 0x7fade80a2800 2024-09-16T15:02:26.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:26.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade67fc640 10 monclient: dump: 2024-09-16T15:02:26.370 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.370 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.370 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.370 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.371 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.371 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.371 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.371 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.371 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.371 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 10 monclient: shutdown 2024-09-16T15:02:26.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:26.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 1 -- 172.21.15.26:0/535614801 >> v1:172.21.15.26:6789/0 conn(0x7fade80a2800 legacy=0x7fade813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 1 -- 172.21.15.26:0/535614801 shutdown_connections 2024-09-16T15:02:26.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 1 -- 172.21.15.26:0/535614801 >> 172.21.15.26:0/535614801 conn(0x7fade80a7f30 msgr2=0x7fade80a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:26.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 1 -- 172.21.15.26:0/535614801 shutdown_connections 2024-09-16T15:02:26.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 1 -- 172.21.15.26:0/535614801 wait complete. 2024-09-16T15:02:26.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 10 monclient: build_initial_monmap 2024-09-16T15:02:26.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 10 monclient: monmap: 2024-09-16T15:02:26.372 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:26.372 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:26.372 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:26.372 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:26.372 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:26.372 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:26.372 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:26.372 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:26.372 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:26.372 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 1 Processor -- start 2024-09-16T15:02:26.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 1 -- start start 2024-09-16T15:02:26.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 10 monclient: init 2024-09-16T15:02:26.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:26.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:02:26.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 10 monclient(hunting): picked mon.noname-c con 0x7fade80a0000 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:26.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 10 monclient(hunting): picked mon.noname-a con 0x7fade80a1d10 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:26.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 10 monclient(hunting): picked mon.noname-b con 0x7fade80a2800 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:26.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade8134ce0 con 0x7fade80a1d10 2024-09-16T15:02:26.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade80a6460 con 0x7fade80a2800 2024-09-16T15:02:26.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade809e500 con 0x7fade80a0000 2024-09-16T15:02:26.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:26.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadedc3a640 10 monclient(hunting): authenticate will time out at 2170.487254s 2024-09-16T15:02:26.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade77fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fade80a0000 0x7fade8133090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48008/0 (socket says 172.21.15.26:48008) 2024-09-16T15:02:26.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade77fe640 1 -- 172.21.15.26:0/3133531738 learned_addr learned my addr 172.21.15.26:0/3133531738 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:26.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fade6ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fade80a1d10 0x7fade81337a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42262/0 (socket says 172.21.15.26:42262) 2024-09-16T15:02:26.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612111560 0 0) 0x7fade8134ce0 con 0x7fade80a1d10 2024-09-16T15:02:26.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 10 monclient(hunting): my global_id is 334500 2024-09-16T15:02:26.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd4003530 con 0x7fade80a1d10 2024-09-16T15:02:26.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325306849 0 0) 0x7fade809e500 con 0x7fade80a0000 2024-09-16T15:02:26.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 10 monclient(hunting): my global_id is 354314 2024-09-16T15:02:26.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade8134ce0 con 0x7fade80a0000 2024-09-16T15:02:26.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1006059097 0 0) 0x7fade8134ce0 con 0x7fade80a0000 2024-09-16T15:02:26.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fade809e500 con 0x7fade80a0000 2024-09-16T15:02:26.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fadd8001a00 con 0x7fade80a0000 2024-09-16T15:02:26.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:02:26.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 10 monclient(hunting): dump: 2024-09-16T15:02:26.378 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.378 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.378 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.379 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.379 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.379 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.379 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.379 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.379 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.379 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:26.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2762128233 0 0) 0x7fadd4003530 con 0x7fade80a1d10 2024-09-16T15:02:26.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fade8134ce0 con 0x7fade80a1d10 2024-09-16T15:02:26.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7faddc001840 con 0x7fade80a1d10 2024-09-16T15:02:26.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:26.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.363+0000 7fadcffff640 10 monclient(hunting): dump: 2024-09-16T15:02:26.380 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.380 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.380 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.380 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.380 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.380 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.380 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.380 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.380 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.380 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036467727 0 0) 0x7fade80a6460 con 0x7fade80a2800 2024-09-16T15:02:26.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 10 monclient(hunting): my global_id is 364180 2024-09-16T15:02:26.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd4003530 con 0x7fade80a2800 2024-09-16T15:02:26.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2862298765 0 0) 0x7fade809e500 con 0x7fade80a0000 2024-09-16T15:02:26.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 >> v1:172.21.15.189:6789/0 conn(0x7fade80a2800 legacy=0x7fade8133eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 >> v1:172.21.15.26:6789/0 conn(0x7fade80a1d10 legacy=0x7fade81337a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 10 monclient: _finish_hunting 0 2024-09-16T15:02:26.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 1 monclient: found mon.c 2024-09-16T15:02:26.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:26.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:26.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade80a6990 con 0x7fade80a0000 2024-09-16T15:02:26.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 10 monclient: _finish_auth 0 2024-09-16T15:02:26.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 10 monclient: _check_auth_tickets 2024-09-16T15:02:26.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:26.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadedc3a640 5 monclient: authenticate success, global_id 354314 2024-09-16T15:02:26.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadedc3a640 10 monclient: _renew_subs 2024-09-16T15:02:26.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadedc3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:26.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadedc3a640 1 -- 172.21.15.26:0/3133531738 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade8136cc0 con 0x7fade80a0000 2024-09-16T15:02:26.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadedc3a640 10 monclient: _renew_subs 2024-09-16T15:02:26.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadedc3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:26.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadedc3a640 1 -- 172.21.15.26:0/3133531738 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fade8135e30 con 0x7fade80a0000 2024-09-16T15:02:26.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadd80047f0 con 0x7fade80a0000 2024-09-16T15:02:26.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:26.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fadd8004b00 con 0x7fade80a0000 2024-09-16T15:02:26.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:26.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fade5ffb640 4 set_mon_vals no callback set 2024-09-16T15:02:26.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 10 monclient: dump: 2024-09-16T15:02:26.383 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.383 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.383 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.383 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.383 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.383 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.383 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.383 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.383 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.383 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadedc3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:26.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadedc3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:26.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadedc3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:26.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadedc3a640 1 -- 172.21.15.26:0/3133531738 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc8005050 con 0x7fade80a0000 2024-09-16T15:02:26.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fadd800f570 con 0x7fade80a0000 2024-09-16T15:02:26.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fadd8043d00 con 0x7fade80a0000 2024-09-16T15:02:26.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.367+0000 7fade6ffd640 10 monclient: get_auth_request con 0x7fadd4036400 auth_method 0 2024-09-16T15:02:26.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.371+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fadd8014450 con 0x7fade80a0000 2024-09-16T15:02:26.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.371+0000 7fadcffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:26.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.371+0000 7fadcffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:26.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.531+0000 7fadedc3a640 1 -- 172.21.15.26:0/3133531738 --> v1:172.21.15.189:6812/3196161582 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fadc8000ba0 con 0x7fadd4036400 2024-09-16T15:02:26.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.531+0000 7fadcffff640 1 -- 172.21.15.26:0/3133531738 <== mgr.4105 v1:172.21.15.189:6812/3196161582 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (unknown 489605429 0 0) 0x7fadc8000ba0 con 0x7fadd4036400 2024-09-16T15:02:26.533 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T15:02:26.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.535+0000 7fadedc3a640 1 -- 172.21.15.26:0/3133531738 >> v1:172.21.15.189:6812/3196161582 conn(0x7fadd4036400 legacy=0x7fadd40388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.535+0000 7fadedc3a640 10 monclient: shutdown 2024-09-16T15:02:26.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.535+0000 7fadedc3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:26.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.535+0000 7fadedc3a640 1 -- 172.21.15.26:0/3133531738 >> v1:172.21.15.26:6790/0 conn(0x7fade80a0000 legacy=0x7fade8133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.535+0000 7fadedc3a640 1 -- 172.21.15.26:0/3133531738 shutdown_connections 2024-09-16T15:02:26.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.535+0000 7fadedc3a640 1 -- 172.21.15.26:0/3133531738 >> 172.21.15.26:0/3133531738 conn(0x7fade80a7f30 msgr2=0x7fade806b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:26.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.535+0000 7fadedc3a640 1 -- 172.21.15.26:0/3133531738 shutdown_connections 2024-09-16T15:02:26.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.535+0000 7fadedc3a640 1 -- 172.21.15.26:0/3133531738 wait complete. 2024-09-16T15:02:26.551 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T15:02:26.552 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T15:02:26.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 10 monclient: get_monmap_and_config 2024-09-16T15:02:26.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 10 monclient: build_initial_monmap 2024-09-16T15:02:26.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 10 monclient: monmap: 2024-09-16T15:02:26.668 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:26.668 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.668 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:26.670170+0000 2024-09-16T15:02:26.668 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:26.670170+0000 2024-09-16T15:02:26.668 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:26.669 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.669 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:26.669 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:26.669 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:26.669 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 1 Processor -- start 2024-09-16T15:02:26.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 1 -- start start 2024-09-16T15:02:26.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 10 monclient: init 2024-09-16T15:02:26.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:26.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:02:26.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 10 monclient(hunting): picked mon.noname-c con 0x7f21b009a4e0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:26.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 10 monclient(hunting): picked mon.noname-a con 0x7f21b009afd0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:26.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 10 monclient(hunting): picked mon.noname-b con 0x7f21b0138110 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:26.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b009b810 con 0x7f21b009afd0 2024-09-16T15:02:26.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b009ecf0 con 0x7f21b009a4e0 2024-09-16T15:02:26.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b009baf0 con 0x7f21b0138110 2024-09-16T15:02:26.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:26.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 10 monclient(hunting): authenticate will time out at 2170.790292s 2024-09-16T15:02:26.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21affff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f21b009afd0 0x7f21b01369f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42268/0 (socket says 172.21.15.26:42268) 2024-09-16T15:02:26.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21affff640 1 -- 172.21.15.26:0/2118940335 learned_addr learned my addr 172.21.15.26:0/2118940335 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:26.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b5601640 1 --1- 172.21.15.26:0/2118940335 >> v1:172.21.15.26:6790/0 conn(0x7f21b0138110 0x7f21b013a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48012/0 (socket says 172.21.15.26:48012) 2024-09-16T15:02:26.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 1 -- 172.21.15.26:0/2118940335 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543682213 0 0) 0x7f21b009b810 con 0x7f21b009afd0 2024-09-16T15:02:26.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient(hunting): my global_id is 334503 2024-09-16T15:02:26.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 1 -- 172.21.15.26:0/2118940335 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2198003530 con 0x7f21b009afd0 2024-09-16T15:02:26.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 1 -- 172.21.15.26:0/2118940335 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905979400 0 0) 0x7f21b009baf0 con 0x7f21b0138110 2024-09-16T15:02:26.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient(hunting): my global_id is 354317 2024-09-16T15:02:26.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 1 -- 172.21.15.26:0/2118940335 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b009b810 con 0x7f21b0138110 2024-09-16T15:02:26.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 1 -- 172.21.15.26:0/2118940335 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3346000566 0 0) 0x7f21b009b810 con 0x7f21b0138110 2024-09-16T15:02:26.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 1 -- 172.21.15.26:0/2118940335 >> v1:172.21.15.189:6789/0 conn(0x7f21b009a4e0 legacy=0x7f21b009a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 1 -- 172.21.15.26:0/2118940335 >> v1:172.21.15.26:6789/0 conn(0x7f21b009afd0 legacy=0x7f21b01369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient: _finish_hunting 0 2024-09-16T15:02:26.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 1 monclient: found mon.noname-b 2024-09-16T15:02:26.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:26.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:02:26.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 1 -- 172.21.15.26:0/2118940335 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b009bdd0 con 0x7f21b0138110 2024-09-16T15:02:26.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient: _finish_auth 0 2024-09-16T15:02:26.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient: _check_auth_tickets 2024-09-16T15:02:26.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:26.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 1 -- 172.21.15.26:0/2118940335 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f21a8001600 con 0x7f21b0138110 2024-09-16T15:02:26.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 5 monclient: authenticate success, global_id 354317 2024-09-16T15:02:26.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:26.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:02:26.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient: dump: 2024-09-16T15:02:26.679 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.679 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.679 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.679 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.679 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.679 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.680 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.680 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.680 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.680 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:26.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 1 -- 172.21.15.26:0/2118940335 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21a80018b0 con 0x7f21b0138110 2024-09-16T15:02:26.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:26.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:26.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 1 -- 172.21.15.26:0/2118940335 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f21a8001c80 con 0x7f21b0138110 2024-09-16T15:02:26.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 4 set_mon_vals no callback set 2024-09-16T15:02:26.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:26.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21af7fe640 10 monclient: dump: 2024-09-16T15:02:26.681 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.681 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.681 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.681 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.681 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.681 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.681 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.681 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.681 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.681 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 10 monclient: shutdown 2024-09-16T15:02:26.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:26.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 1 -- 172.21.15.26:0/2118940335 >> v1:172.21.15.26:6790/0 conn(0x7f21b0138110 legacy=0x7f21b013a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 1 -- 172.21.15.26:0/2118940335 shutdown_connections 2024-09-16T15:02:26.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 1 -- 172.21.15.26:0/2118940335 >> 172.21.15.26:0/2118940335 conn(0x7f21b00a35e0 msgr2=0x7f21b009fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:26.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 1 -- 172.21.15.26:0/2118940335 shutdown_connections 2024-09-16T15:02:26.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.667+0000 7f21b7064640 1 -- 172.21.15.26:0/2118940335 wait complete. 2024-09-16T15:02:26.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 10 monclient: build_initial_monmap 2024-09-16T15:02:26.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 10 monclient: monmap: 2024-09-16T15:02:26.682 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:26.682 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:26.682 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:26.682 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:26.682 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:26.682 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:26.683 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:26.683 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:26.683 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:26.683 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 1 Processor -- start 2024-09-16T15:02:26.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 1 -- start start 2024-09-16T15:02:26.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 10 monclient: init 2024-09-16T15:02:26.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:26.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:02:26.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 10 monclient(hunting): picked mon.noname-a con 0x7f21b009a4e0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:26.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 10 monclient(hunting): picked mon.noname-b con 0x7f21b009afd0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:26.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 10 monclient(hunting): picked mon.noname-c con 0x7f21b0138110 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:26.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b00a1850 con 0x7f21b009a4e0 2024-09-16T15:02:26.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b009a1d0 con 0x7f21b009afd0 2024-09-16T15:02:26.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b011c3d0 con 0x7f21b0138110 2024-09-16T15:02:26.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:26.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 10 monclient(hunting): authenticate will time out at 2170.793259s 2024-09-16T15:02:26.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b5601640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f21b0138110 0x7f21b01cfc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48028/0 (socket says 172.21.15.26:48028) 2024-09-16T15:02:26.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b5601640 1 -- 172.21.15.26:0/2685405510 learned_addr learned my addr 172.21.15.26:0/2685405510 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:26.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553835006 0 0) 0x7f21b011c3d0 con 0x7f21b0138110 2024-09-16T15:02:26.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): my global_id is 354320 2024-09-16T15:02:26.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f219c003530 con 0x7f21b0138110 2024-09-16T15:02:26.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919997138 0 0) 0x7f21b00a1850 con 0x7f21b009a4e0 2024-09-16T15:02:26.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): my global_id is 334506 2024-09-16T15:02:26.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b011c3d0 con 0x7f21b009a4e0 2024-09-16T15:02:26.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1004374966 0 0) 0x7f219c003530 con 0x7f21b0138110 2024-09-16T15:02:26.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21b00a1850 con 0x7f21b0138110 2024-09-16T15:02:26.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f21a8001bc0 con 0x7f21b0138110 2024-09-16T15:02:26.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:02:26.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): dump: 2024-09-16T15:02:26.685 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.686 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.686 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.686 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.686 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.686 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.686 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.686 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.686 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.686 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:26.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2609271530 0 0) 0x7f21b011c3d0 con 0x7f21b009a4e0 2024-09-16T15:02:26.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f219c003530 con 0x7f21b009a4e0 2024-09-16T15:02:26.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f21a0001be0 con 0x7f21b009a4e0 2024-09-16T15:02:26.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:26.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): dump: 2024-09-16T15:02:26.687 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.687 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.687 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.687 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.687 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.687 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.687 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.687 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.687 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.687 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624694090 0 0) 0x7f21b009a1d0 con 0x7f21b009afd0 2024-09-16T15:02:26.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient(hunting): my global_id is 364186 2024-09-16T15:02:26.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b011c3d0 con 0x7f21b009afd0 2024-09-16T15:02:26.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4243179772 0 0) 0x7f219c003530 con 0x7f21b009a4e0 2024-09-16T15:02:26.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 >> v1:172.21.15.26:6790/0 conn(0x7f21b0138110 legacy=0x7f21b01cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 >> v1:172.21.15.189:6789/0 conn(0x7f21b009afd0 legacy=0x7f21b01cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient: _finish_hunting 0 2024-09-16T15:02:26.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 monclient: found mon.a 2024-09-16T15:02:26.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:26.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:26.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b011d540 con 0x7f21b009a4e0 2024-09-16T15:02:26.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient: _finish_auth 0 2024-09-16T15:02:26.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient: _check_auth_tickets 2024-09-16T15:02:26.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:26.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 5 monclient: authenticate success, global_id 334506 2024-09-16T15:02:26.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 10 monclient: _renew_subs 2024-09-16T15:02:26.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:26.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 1 -- 172.21.15.26:0/2685405510 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b011dde0 con 0x7f21b009a4e0 2024-09-16T15:02:26.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 10 monclient: _renew_subs 2024-09-16T15:02:26.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:26.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 1 -- 172.21.15.26:0/2685405510 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b011ba80 con 0x7f21b009a4e0 2024-09-16T15:02:26.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21a00039a0 con 0x7f21b009a4e0 2024-09-16T15:02:26.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:26.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f21a0003cb0 con 0x7f21b009a4e0 2024-09-16T15:02:26.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:26.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21aeffd640 4 set_mon_vals no callback set 2024-09-16T15:02:26.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient: dump: 2024-09-16T15:02:26.691 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.691 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.691 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.691 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.691 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.691 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.691 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.692 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.692 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.692 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:26.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b7064640 1 -- 172.21.15.26:0/2685405510 --> v1:172.21.15.26:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2190000f20 con 0x7f21b009a4e0 2024-09-16T15:02:26.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f21a000e720 con 0x7f21b009a4e0 2024-09-16T15:02:26.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f21a0042f70 con 0x7f21b009a4e0 2024-09-16T15:02:26.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 --> v1:172.21.15.26:6802/1811454603 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f219c043cb0 con 0x7f219c03ffe0 2024-09-16T15:02:26.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_get_version_reply(handle=1 version=257) v2 ==== 24+0+0 (unknown 1324813924 0 0) 0x7f21a0043310 con 0x7f21b009a4e0 2024-09-16T15:02:26.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21ad7fa640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-16T15:02:26.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21b5601640 10 monclient: get_auth_request con 0x7f219c03ffe0 auth_method 0 2024-09-16T15:02:26.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.671+0000 7f21affff640 10 monclient: get_auth_request con 0x7f219c036330 auth_method 0 2024-09-16T15:02:26.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.675+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 <== osd.2 v1:172.21.15.26:6802/1811454603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f219c043cb0 con 0x7f219c03ffe0 2024-09-16T15:02:26.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.687+0000 7f21b7064640 1 -- 172.21.15.26:0/2685405510 --> v1:172.21.15.26:6802/1811454603 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2190002b60 con 0x7f219c03ffe0 2024-09-16T15:02:26.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.687+0000 7f21ad7fa640 1 -- 172.21.15.26:0/2685405510 <== osd.2 v1:172.21.15.26:6802/1811454603 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f2190002b60 con 0x7f219c03ffe0 2024-09-16T15:02:26.694 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:02:26.694 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T15:02:26.694 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:02:26.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.687+0000 7f21b7064640 1 -- 172.21.15.26:0/2685405510 >> v1:172.21.15.26:6802/1811454603 conn(0x7f219c03ffe0 legacy=0x7f219c042440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.687+0000 7f21b7064640 1 -- 172.21.15.26:0/2685405510 >> v1:172.21.15.189:6812/3196161582 conn(0x7f219c036330 legacy=0x7f219c0387f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.687+0000 7f21b7064640 10 monclient: shutdown 2024-09-16T15:02:26.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.687+0000 7f21b7064640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:26.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.687+0000 7f21b7064640 1 -- 172.21.15.26:0/2685405510 >> v1:172.21.15.26:6789/0 conn(0x7f21b009a4e0 legacy=0x7f21b01c7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.687+0000 7f21b5601640 1 -- 172.21.15.26:0/2685405510 reap_dead start 2024-09-16T15:02:26.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.687+0000 7f21b7064640 1 -- 172.21.15.26:0/2685405510 shutdown_connections 2024-09-16T15:02:26.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.687+0000 7f21b7064640 1 -- 172.21.15.26:0/2685405510 >> 172.21.15.26:0/2685405510 conn(0x7f21b00a35e0 msgr2=0x7f21b009d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:26.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.687+0000 7f21b7064640 1 -- 172.21.15.26:0/2685405510 shutdown_connections 2024-09-16T15:02:26.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.687+0000 7f21b7064640 1 -- 172.21.15.26:0/2685405510 wait complete. 2024-09-16T15:02:26.702 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T15:02:26.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc97532f640 10 monclient: get_monmap_and_config 2024-09-16T15:02:26.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc97532f640 10 monclient: build_initial_monmap 2024-09-16T15:02:26.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc97532f640 10 monclient: monmap: 2024-09-16T15:02:26.809 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:26.809 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.809 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:26.811196+0000 2024-09-16T15:02:26.809 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:26.811196+0000 2024-09-16T15:02:26.809 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:26.809 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.810 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:26.810 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:26.810 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:26.810 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc97532f640 1 Processor -- start 2024-09-16T15:02:26.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc97532f640 1 -- start start 2024-09-16T15:02:26.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc97532f640 10 monclient: init 2024-09-16T15:02:26.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc97532f640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:26.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc97532f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:02:26.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc97532f640 10 monclient(hunting): picked mon.noname-c con 0x7fc97009a550 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:26.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc97532f640 10 monclient(hunting): picked mon.noname-b con 0x7fc97009d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:26.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc97532f640 10 monclient(hunting): picked mon.noname-a con 0x7fc97009b040 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:26.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc97532f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97009dc60 con 0x7fc97009b040 2024-09-16T15:02:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc97532f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc970099ae0 con 0x7fc97009a550 2024-09-16T15:02:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc97532f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97009b8f0 con 0x7fc97009d1b0 2024-09-16T15:02:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc97532f640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc97532f640 10 monclient(hunting): authenticate will time out at 2170.931718s 2024-09-16T15:02:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc96f7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc97009b040 0x7fc97013a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42282/0 (socket says 172.21.15.26:42282) 2024-09-16T15:02:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.807+0000 7fc96f7fe640 1 -- 172.21.15.26:0/3223436751 learned_addr learned my addr 172.21.15.26:0/3223436751 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 1 -- 172.21.15.26:0/3223436751 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177429445 0 0) 0x7fc97009b8f0 con 0x7fc97009d1b0 2024-09-16T15:02:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient(hunting): my global_id is 354323 2024-09-16T15:02:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 1 -- 172.21.15.26:0/3223436751 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc958003530 con 0x7fc97009d1b0 2024-09-16T15:02:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 1 -- 172.21.15.26:0/3223436751 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340038748 0 0) 0x7fc97009dc60 con 0x7fc97009b040 2024-09-16T15:02:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient(hunting): my global_id is 334509 2024-09-16T15:02:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 1 -- 172.21.15.26:0/3223436751 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc97009b8f0 con 0x7fc97009b040 2024-09-16T15:02:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 1 -- 172.21.15.26:0/3223436751 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3433547079 0 0) 0x7fc958003530 con 0x7fc97009d1b0 2024-09-16T15:02:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 1 -- 172.21.15.26:0/3223436751 >> v1:172.21.15.189:6789/0 conn(0x7fc97009a550 legacy=0x7fc97009a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 1 -- 172.21.15.26:0/3223436751 >> v1:172.21.15.26:6789/0 conn(0x7fc97009b040 legacy=0x7fc97013a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient: _finish_hunting 0 2024-09-16T15:02:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 1 monclient: found mon.noname-b 2024-09-16T15:02:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:02:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 1 -- 172.21.15.26:0/3223436751 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97009bbd0 con 0x7fc97009d1b0 2024-09-16T15:02:26.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient: _finish_auth 0 2024-09-16T15:02:26.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient: _check_auth_tickets 2024-09-16T15:02:26.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:26.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 5 monclient: authenticate success, global_id 354323 2024-09-16T15:02:26.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 1 -- 172.21.15.26:0/3223436751 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc964001620 con 0x7fc97009d1b0 2024-09-16T15:02:26.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:02:26.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient: dump: 2024-09-16T15:02:26.816 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.816 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.816 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.816 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.816 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.817 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.817 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.817 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.817 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.817 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:26.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 1 -- 172.21.15.26:0/3223436751 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc964001840 con 0x7fc97009d1b0 2024-09-16T15:02:26.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:26.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 1 -- 172.21.15.26:0/3223436751 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc964001c10 con 0x7fc97009d1b0 2024-09-16T15:02:26.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:26.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96dffb640 10 monclient: dump: 2024-09-16T15:02:26.817 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.818 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 4 set_mon_vals no callback set 2024-09-16T15:02:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 10 monclient: shutdown 2024-09-16T15:02:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:26.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 1 -- 172.21.15.26:0/3223436751 >> v1:172.21.15.26:6790/0 conn(0x7fc97009d1b0 legacy=0x7fc970136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 1 -- 172.21.15.26:0/3223436751 shutdown_connections 2024-09-16T15:02:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 1 -- 172.21.15.26:0/3223436751 >> 172.21.15.26:0/3223436751 conn(0x7fc97009fb50 msgr2=0x7fc97009ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 1 -- 172.21.15.26:0/3223436751 shutdown_connections 2024-09-16T15:02:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 1 -- 172.21.15.26:0/3223436751 wait complete. 2024-09-16T15:02:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 10 monclient: build_initial_monmap 2024-09-16T15:02:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 10 monclient: monmap: 2024-09-16T15:02:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:26.819 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:26.820 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:26.820 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 1 Processor -- start 2024-09-16T15:02:26.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 1 -- start start 2024-09-16T15:02:26.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 10 monclient: init 2024-09-16T15:02:26.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:26.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:02:26.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 10 monclient(hunting): picked mon.noname-a con 0x7fc97009a550 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:26.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 10 monclient(hunting): picked mon.noname-b con 0x7fc97009b040 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:26.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 10 monclient(hunting): picked mon.noname-c con 0x7fc97009d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:26.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9700a2470 con 0x7fc97009a550 2024-09-16T15:02:26.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc970071c40 con 0x7fc97009b040 2024-09-16T15:02:26.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97009a240 con 0x7fc97009d1b0 2024-09-16T15:02:26.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:26.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 10 monclient(hunting): authenticate will time out at 2170.934539s 2024-09-16T15:02:26.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96f7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc97009d1b0 0x7fc9701d1730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48058/0 (socket says 172.21.15.26:48058) 2024-09-16T15:02:26.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96effd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc97009a550 0x7fc97006f670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42296/0 (socket says 172.21.15.26:42296) 2024-09-16T15:02:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96f7fe640 1 -- 172.21.15.26:0/3227684138 learned_addr learned my addr 172.21.15.26:0/3227684138 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740738599 0 0) 0x7fc9700a2470 con 0x7fc97009a550 2024-09-16T15:02:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): my global_id is 334512 2024-09-16T15:02:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc95c003560 con 0x7fc97009a550 2024-09-16T15:02:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631091877 0 0) 0x7fc97009a240 con 0x7fc97009d1b0 2024-09-16T15:02:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): my global_id is 354326 2024-09-16T15:02:26.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9700a2470 con 0x7fc97009d1b0 2024-09-16T15:02:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3524384095 0 0) 0x7fc9700a2470 con 0x7fc97009d1b0 2024-09-16T15:02:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc97009a240 con 0x7fc97009d1b0 2024-09-16T15:02:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc968001980 con 0x7fc97009d1b0 2024-09-16T15:02:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:02:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): dump: 2024-09-16T15:02:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.827 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.827 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:26.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1591160785 0 0) 0x7fc95c003560 con 0x7fc97009a550 2024-09-16T15:02:26.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9700a2470 con 0x7fc97009a550 2024-09-16T15:02:26.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc960001bc0 con 0x7fc97009a550 2024-09-16T15:02:26.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:26.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): dump: 2024-09-16T15:02:26.827 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.827 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.827 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.827 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.828 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.828 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.828 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.828 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.828 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.828 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466577707 0 0) 0x7fc970071c40 con 0x7fc97009b040 2024-09-16T15:02:26.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:26.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:26.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:26.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient(hunting): my global_id is 364192 2024-09-16T15:02:26.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc95c003560 con 0x7fc97009b040 2024-09-16T15:02:26.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1067648258 0 0) 0x7fc97009a240 con 0x7fc97009d1b0 2024-09-16T15:02:26.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 >> v1:172.21.15.189:6789/0 conn(0x7fc97009b040 legacy=0x7fc97006fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 >> v1:172.21.15.26:6789/0 conn(0x7fc97009a550 legacy=0x7fc97006f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient: _finish_hunting 0 2024-09-16T15:02:26.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 monclient: found mon.c 2024-09-16T15:02:26.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:26.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:26.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97011f020 con 0x7fc97009d1b0 2024-09-16T15:02:26.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient: _finish_auth 0 2024-09-16T15:02:26.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient: _check_auth_tickets 2024-09-16T15:02:26.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:26.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 5 monclient: authenticate success, global_id 354326 2024-09-16T15:02:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 10 monclient: _renew_subs 2024-09-16T15:02:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 1 -- 172.21.15.26:0/3227684138 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97011e780 con 0x7fc97009d1b0 2024-09-16T15:02:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 10 monclient: _renew_subs 2024-09-16T15:02:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc97532f640 1 -- 172.21.15.26:0/3227684138 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97011dab0 con 0x7fc97009d1b0 2024-09-16T15:02:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9680047f0 con 0x7fc97009d1b0 2024-09-16T15:02:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fc968004b00 con 0x7fc97009d1b0 2024-09-16T15:02:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc96d7fa640 4 set_mon_vals no callback set 2024-09-16T15:02:26.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.811+0000 7fc9577fe640 10 monclient: dump: 2024-09-16T15:02:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:26.831 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.815+0000 7fc97532f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.815+0000 7fc97532f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.815+0000 7fc97532f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:26.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.815+0000 7fc97532f640 1 -- 172.21.15.26:0/3227684138 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc950005050 con 0x7fc97009d1b0 2024-09-16T15:02:26.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.815+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fc96800f570 con 0x7fc97009d1b0 2024-09-16T15:02:26.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.815+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fc968043d00 con 0x7fc97009d1b0 2024-09-16T15:02:26.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.815+0000 7fc96effd640 10 monclient: get_auth_request con 0x7fc95c036410 auth_method 0 2024-09-16T15:02:26.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.819+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc968011010 con 0x7fc97009d1b0 2024-09-16T15:02:26.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.819+0000 7fc9577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:26.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.819+0000 7fc9577fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:26.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.975+0000 7fc97532f640 1 -- 172.21.15.26:0/3227684138 --> v1:172.21.15.189:6812/3196161582 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc950000ba0 con 0x7fc95c036410 2024-09-16T15:02:26.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.979+0000 7fc9577fe640 1 -- 172.21.15.26:0/3227684138 <== mgr.4105 v1:172.21.15.189:6812/3196161582 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (unknown 3049115190 0 0) 0x7fc950000ba0 con 0x7fc95c036410 2024-09-16T15:02:26.980 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T15:02:26.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.979+0000 7fc97532f640 1 -- 172.21.15.26:0/3227684138 >> v1:172.21.15.189:6812/3196161582 conn(0x7fc95c036410 legacy=0x7fc95c0388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.979+0000 7fc97532f640 10 monclient: shutdown 2024-09-16T15:02:26.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.979+0000 7fc97532f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.979+0000 7fc97532f640 1 -- 172.21.15.26:0/3227684138 >> v1:172.21.15.26:6790/0 conn(0x7fc97009d1b0 legacy=0x7fc9701d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.979+0000 7fc97532f640 1 -- 172.21.15.26:0/3227684138 shutdown_connections 2024-09-16T15:02:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.979+0000 7fc97532f640 1 -- 172.21.15.26:0/3227684138 >> 172.21.15.26:0/3227684138 conn(0x7fc97009fb50 msgr2=0x7fc97013a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.979+0000 7fc97532f640 1 -- 172.21.15.26:0/3227684138 shutdown_connections 2024-09-16T15:02:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:26.979+0000 7fc97532f640 1 -- 172.21.15.26:0/3227684138 wait complete. 2024-09-16T15:02:26.999 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T15:02:26.999 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T15:02:27.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f41114b0640 10 monclient: get_monmap_and_config 2024-09-16T15:02:27.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f41114b0640 10 monclient: build_initial_monmap 2024-09-16T15:02:27.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f41114b0640 10 monclient: monmap: 2024-09-16T15:02:27.129 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:27.129 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.129 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:27.131332+0000 2024-09-16T15:02:27.129 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:27.131332+0000 2024-09-16T15:02:27.130 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:27.130 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.130 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:27.130 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:27.130 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:27.130 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f41114b0640 1 Processor -- start 2024-09-16T15:02:27.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f41114b0640 1 -- start start 2024-09-16T15:02:27.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f41114b0640 10 monclient: init 2024-09-16T15:02:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f41114b0640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f41114b0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:02:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f41114b0640 10 monclient(hunting): picked mon.noname-a con 0x7f410c09d9e0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f41114b0640 10 monclient(hunting): picked mon.noname-b con 0x7f410c09d080 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f41114b0640 10 monclient(hunting): picked mon.noname-c con 0x7f410c09e4d0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f41114b0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c09f5b0 con 0x7f410c09d9e0 2024-09-16T15:02:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f41114b0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c0a06c0 con 0x7f410c09e4d0 2024-09-16T15:02:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f41114b0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c0a0900 con 0x7f410c09d080 2024-09-16T15:02:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f41114b0640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f41114b0640 10 monclient(hunting): authenticate will time out at 2171.251364s 2024-09-16T15:02:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f410affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f410c09d9e0 0x7f410c09ddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38746/0 (socket says 172.21.15.26:38746) 2024-09-16T15:02:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f410a7fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f410c09d080 0x7f410c137a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51512/0 (socket says 172.21.15.26:51512) 2024-09-16T15:02:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f410affd640 1 -- 172.21.15.26:0/3560981881 learned_addr learned my addr 172.21.15.26:0/3560981881 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:27.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f4109ffb640 1 -- 172.21.15.26:0/3560981881 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522827181 0 0) 0x7f410c09f5b0 con 0x7f410c09d9e0 2024-09-16T15:02:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f4109ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f4109ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f4109ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f4109ffb640 10 monclient(hunting): my global_id is 334515 2024-09-16T15:02:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f4109ffb640 1 -- 172.21.15.26:0/3560981881 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f4003530 con 0x7f410c09d9e0 2024-09-16T15:02:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f4109ffb640 1 -- 172.21.15.26:0/3560981881 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665746425 0 0) 0x7f410c0a0900 con 0x7f410c09d080 2024-09-16T15:02:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f4109ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f4109ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f4109ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f4109ffb640 10 monclient(hunting): my global_id is 354329 2024-09-16T15:02:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.127+0000 7f4109ffb640 1 -- 172.21.15.26:0/3560981881 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f410c09f5b0 con 0x7f410c09d080 2024-09-16T15:02:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 1 -- 172.21.15.26:0/3560981881 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 839904969 0 0) 0x7f40f4003530 con 0x7f410c09d9e0 2024-09-16T15:02:27.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 1 -- 172.21.15.26:0/3560981881 >> v1:172.21.15.26:6790/0 conn(0x7f410c09d080 legacy=0x7f410c137a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 1 -- 172.21.15.26:0/3560981881 >> v1:172.21.15.189:6789/0 conn(0x7f410c09e4d0 legacy=0x7f410c13b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 10 monclient: _finish_hunting 0 2024-09-16T15:02:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 1 monclient: found mon.noname-a 2024-09-16T15:02:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:02:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 1 -- 172.21.15.26:0/3560981881 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c09ed10 con 0x7f410c09d9e0 2024-09-16T15:02:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 10 monclient: _finish_auth 0 2024-09-16T15:02:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:02:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 1 -- 172.21.15.26:0/3560981881 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4100001620 con 0x7f410c09d9e0 2024-09-16T15:02:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 5 monclient: authenticate success, global_id 334515 2024-09-16T15:02:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 10 monclient: dump: 2024-09-16T15:02:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.141 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:27.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 1 -- 172.21.15.26:0/3560981881 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41000017d0 con 0x7f410c09d9e0 2024-09-16T15:02:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 4 set_mon_vals no callback set 2024-09-16T15:02:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 1 -- 172.21.15.26:0/3560981881 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4100001ba0 con 0x7f410c09d9e0 2024-09-16T15:02:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f4109ffb640 10 monclient: dump: 2024-09-16T15:02:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.142 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.143 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 10 monclient: shutdown 2024-09-16T15:02:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 1 -- 172.21.15.26:0/3560981881 >> v1:172.21.15.26:6789/0 conn(0x7f410c09d9e0 legacy=0x7f410c09ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 1 -- 172.21.15.26:0/3560981881 shutdown_connections 2024-09-16T15:02:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 1 -- 172.21.15.26:0/3560981881 >> 172.21.15.26:0/3560981881 conn(0x7f410c0a35e0 msgr2=0x7f410c09fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 1 -- 172.21.15.26:0/3560981881 shutdown_connections 2024-09-16T15:02:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 1 -- 172.21.15.26:0/3560981881 wait complete. 2024-09-16T15:02:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 10 monclient: build_initial_monmap 2024-09-16T15:02:27.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 10 monclient: monmap: 2024-09-16T15:02:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:27.144 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 1 Processor -- start 2024-09-16T15:02:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 1 -- start start 2024-09-16T15:02:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 10 monclient: init 2024-09-16T15:02:27.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T15:02:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 10 monclient(hunting): picked mon.noname-b con 0x7f410c09d080 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 10 monclient(hunting): picked mon.noname-a con 0x7f410c09d9e0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 10 monclient(hunting): picked mon.noname-c con 0x7f410c09e4d0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c09a1d0 con 0x7f410c09d9e0 2024-09-16T15:02:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c139a20 con 0x7f410c09d080 2024-09-16T15:02:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c13a400 con 0x7f410c09e4d0 2024-09-16T15:02:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 10 monclient(hunting): authenticate will time out at 2171.254483s 2024-09-16T15:02:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f410a7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f410c09d9e0 0x7f410c09ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38752/0 (socket says 172.21.15.26:38752) 2024-09-16T15:02:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f410b7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f410c09e4d0 0x7f410c09b330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51520/0 (socket says 172.21.15.26:51520) 2024-09-16T15:02:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f410a7fc640 1 -- 172.21.15.26:0/480287645 learned_addr learned my addr 172.21.15.26:0/480287645 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:27.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892962151 0 0) 0x7f410c09a1d0 con 0x7f410c09d9e0 2024-09-16T15:02:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): my global_id is 334518 2024-09-16T15:02:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f8003530 con 0x7f410c09d9e0 2024-09-16T15:02:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809796724 0 0) 0x7f410c13a400 con 0x7f410c09e4d0 2024-09-16T15:02:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): my global_id is 354332 2024-09-16T15:02:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f410c09a1d0 con 0x7f410c09e4d0 2024-09-16T15:02:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2470760976 0 0) 0x7f40f8003530 con 0x7f410c09d9e0 2024-09-16T15:02:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f410c13a400 con 0x7f410c09d9e0 2024-09-16T15:02:27.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f40fc001970 con 0x7f410c09d9e0 2024-09-16T15:02:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): dump: 2024-09-16T15:02:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.147 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1221881122 0 0) 0x7f410c09a1d0 con 0x7f410c09e4d0 2024-09-16T15:02:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40f8003530 con 0x7f410c09e4d0 2024-09-16T15:02:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f4104001bc0 con 0x7f410c09e4d0 2024-09-16T15:02:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): dump: 2024-09-16T15:02:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.148 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.149 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170628745 0 0) 0x7f410c139a20 con 0x7f410c09d080 2024-09-16T15:02:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient(hunting): my global_id is 364198 2024-09-16T15:02:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f410c09a1d0 con 0x7f410c09d080 2024-09-16T15:02:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2874205460 0 0) 0x7f410c13a400 con 0x7f410c09d9e0 2024-09-16T15:02:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 >> v1:172.21.15.26:6790/0 conn(0x7f410c09e4d0 legacy=0x7f410c09b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 >> v1:172.21.15.189:6789/0 conn(0x7f410c09d080 legacy=0x7f410c09a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient: _finish_hunting 0 2024-09-16T15:02:27.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 monclient: found mon.a 2024-09-16T15:02:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c139fe0 con 0x7f410c09d9e0 2024-09-16T15:02:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient: _finish_auth 0 2024-09-16T15:02:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient: _check_auth_tickets 2024-09-16T15:02:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 5 monclient: authenticate success, global_id 334518 2024-09-16T15:02:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 10 monclient: _renew_subs 2024-09-16T15:02:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 1 -- 172.21.15.26:0/480287645 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410c139740 con 0x7f410c09d9e0 2024-09-16T15:02:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 10 monclient: _renew_subs 2024-09-16T15:02:27.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41114b0640 1 -- 172.21.15.26:0/480287645 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f410c09c110 con 0x7f410c09d9e0 2024-09-16T15:02:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40fc001cc0 con 0x7f410c09d9e0 2024-09-16T15:02:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f40fc004c40 con 0x7f410c09d9e0 2024-09-16T15:02:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f41097fa640 4 set_mon_vals no callback set 2024-09-16T15:02:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.131+0000 7f40f37fe640 10 monclient: dump: 2024-09-16T15:02:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.151 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.152 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.135+0000 7f41114b0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.135+0000 7f41114b0640 1 -- 172.21.15.26:0/480287645 --> v1:172.21.15.26:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f40ec000f20 con 0x7f410c09d9e0 2024-09-16T15:02:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.135+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f40fc00f6b0 con 0x7f410c09d9e0 2024-09-16T15:02:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.135+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f40fc043e40 con 0x7f410c09d9e0 2024-09-16T15:02:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.135+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 --> v1:172.21.15.189:6808/3651837298 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f40f8043ce0 con 0x7f40f8040010 2024-09-16T15:02:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.135+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_get_version_reply(handle=1 version=257) v2 ==== 24+0+0 (unknown 1324813924 0 0) 0x7f40fc044260 con 0x7f410c09d9e0 2024-09-16T15:02:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.135+0000 7f40f37fe640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-16T15:02:27.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.135+0000 7f410b7fe640 10 monclient: get_auth_request con 0x7f40f8040010 auth_method 0 2024-09-16T15:02:27.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.135+0000 7f410affd640 10 monclient: get_auth_request con 0x7f40f8036360 auth_method 0 2024-09-16T15:02:27.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.135+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 <== osd.3 v1:172.21.15.189:6808/3651837298 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f40f8043ce0 con 0x7f40f8040010 2024-09-16T15:02:27.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.147+0000 7f41114b0640 1 -- 172.21.15.26:0/480287645 --> v1:172.21.15.189:6808/3651837298 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f40ec002b60 con 0x7f40f8040010 2024-09-16T15:02:27.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.151+0000 7f40f37fe640 1 -- 172.21.15.26:0/480287645 <== osd.3 v1:172.21.15.189:6808/3651837298 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f40ec002b60 con 0x7f40f8040010 2024-09-16T15:02:27.154 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:02:27.154 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T15:02:27.154 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:02:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.151+0000 7f41114b0640 1 -- 172.21.15.26:0/480287645 >> v1:172.21.15.189:6808/3651837298 conn(0x7f40f8040010 legacy=0x7f40f8042470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.151+0000 7f41114b0640 1 -- 172.21.15.26:0/480287645 >> v1:172.21.15.189:6812/3196161582 conn(0x7f40f8036360 legacy=0x7f40f8038820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.151+0000 7f41114b0640 10 monclient: shutdown 2024-09-16T15:02:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.151+0000 7f41114b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.151+0000 7f41114b0640 1 -- 172.21.15.26:0/480287645 >> v1:172.21.15.26:6789/0 conn(0x7f410c09d9e0 legacy=0x7f410c09ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.151+0000 7f410b7fe640 1 -- 172.21.15.26:0/480287645 reap_dead start 2024-09-16T15:02:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.151+0000 7f41114b0640 1 -- 172.21.15.26:0/480287645 shutdown_connections 2024-09-16T15:02:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.151+0000 7f41114b0640 1 -- 172.21.15.26:0/480287645 >> 172.21.15.26:0/480287645 conn(0x7f410c0a35e0 msgr2=0x7f410c135100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.151+0000 7f41114b0640 1 -- 172.21.15.26:0/480287645 shutdown_connections 2024-09-16T15:02:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.151+0000 7f41114b0640 1 -- 172.21.15.26:0/480287645 wait complete. 2024-09-16T15:02:27.166 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T15:02:27.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.267+0000 7feb47946640 10 monclient: get_monmap_and_config 2024-09-16T15:02:27.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.267+0000 7feb47946640 10 monclient: build_initial_monmap 2024-09-16T15:02:27.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.267+0000 7feb47946640 10 monclient: monmap: 2024-09-16T15:02:27.271 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:27.271 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.271 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:27.273149+0000 2024-09-16T15:02:27.271 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:27.273149+0000 2024-09-16T15:02:27.271 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:27.271 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.271 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:27.271 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:27.272 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:27.272 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 1 Processor -- start 2024-09-16T15:02:27.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 1 -- start start 2024-09-16T15:02:27.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient: init 2024-09-16T15:02:27.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:27.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T15:02:27.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient(hunting): picked mon.noname-b con 0x7feb400a1d10 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:27.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient(hunting): picked mon.noname-a con 0x7feb400a0000 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:27.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient(hunting): picked mon.noname-c con 0x7feb400a2800 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:27.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb400a3250 con 0x7feb400a0000 2024-09-16T15:02:27.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4009f600 con 0x7feb400a2800 2024-09-16T15:02:27.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb400a3830 con 0x7feb400a1d10 2024-09-16T15:02:27.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:27.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient(hunting): authenticate will time out at 2171.392972s 2024-09-16T15:02:27.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb456e2640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7feb400a1d10 0x7feb400a20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51522/0 (socket says 172.21.15.26:51522) 2024-09-16T15:02:27.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb44ee1640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7feb400a0000 0x7feb40139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38768/0 (socket says 172.21.15.26:38768) 2024-09-16T15:02:27.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb456e2640 1 -- 172.21.15.26:0/1334896410 learned_addr learned my addr 172.21.15.26:0/1334896410 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:27.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 1 -- 172.21.15.26:0/1334896410 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669351734 0 0) 0x7feb400a3250 con 0x7feb400a0000 2024-09-16T15:02:27.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient(hunting): my global_id is 334521 2024-09-16T15:02:27.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 1 -- 172.21.15.26:0/1334896410 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb2c003530 con 0x7feb400a0000 2024-09-16T15:02:27.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 1 -- 172.21.15.26:0/1334896410 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752339168 0 0) 0x7feb400a3830 con 0x7feb400a1d10 2024-09-16T15:02:27.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient(hunting): my global_id is 354335 2024-09-16T15:02:27.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 1 -- 172.21.15.26:0/1334896410 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb400a3250 con 0x7feb400a1d10 2024-09-16T15:02:27.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 1 -- 172.21.15.26:0/1334896410 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3280796712 0 0) 0x7feb400a3250 con 0x7feb400a1d10 2024-09-16T15:02:27.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 1 -- 172.21.15.26:0/1334896410 >> v1:172.21.15.189:6789/0 conn(0x7feb400a2800 legacy=0x7feb4013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 1 -- 172.21.15.26:0/1334896410 >> v1:172.21.15.26:6789/0 conn(0x7feb400a0000 legacy=0x7feb40139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient: _finish_hunting 0 2024-09-16T15:02:27.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 1 monclient: found mon.noname-b 2024-09-16T15:02:27.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:27.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:02:27.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 1 -- 172.21.15.26:0/1334896410 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4009ed60 con 0x7feb400a1d10 2024-09-16T15:02:27.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient: _finish_auth 0 2024-09-16T15:02:27.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient: _check_auth_tickets 2024-09-16T15:02:27.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:27.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 1 -- 172.21.15.26:0/1334896410 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb30001550 con 0x7feb400a1d10 2024-09-16T15:02:27.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:02:27.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient: dump: 2024-09-16T15:02:27.281 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.281 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.281 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.281 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.281 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.281 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.281 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.282 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.282 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.282 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 5 monclient: authenticate success, global_id 354335 2024-09-16T15:02:27.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:27.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 1 -- 172.21.15.26:0/1334896410 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb30001780 con 0x7feb400a1d10 2024-09-16T15:02:27.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:27.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 1 -- 172.21.15.26:0/1334896410 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb30001b50 con 0x7feb400a1d10 2024-09-16T15:02:27.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:27.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb37fff640 10 monclient: dump: 2024-09-16T15:02:27.282 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.282 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.283 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.283 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.283 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.283 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.283 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.283 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.283 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.283 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:27.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 4 set_mon_vals no callback set 2024-09-16T15:02:27.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient: shutdown 2024-09-16T15:02:27.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:27.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 1 -- 172.21.15.26:0/1334896410 >> v1:172.21.15.26:6790/0 conn(0x7feb400a1d10 legacy=0x7feb400a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 1 -- 172.21.15.26:0/1334896410 shutdown_connections 2024-09-16T15:02:27.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 1 -- 172.21.15.26:0/1334896410 >> 172.21.15.26:0/1334896410 conn(0x7feb400a7f30 msgr2=0x7feb400a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:27.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 1 -- 172.21.15.26:0/1334896410 shutdown_connections 2024-09-16T15:02:27.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 1 -- 172.21.15.26:0/1334896410 wait complete. 2024-09-16T15:02:27.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient: build_initial_monmap 2024-09-16T15:02:27.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient: monmap: 2024-09-16T15:02:27.284 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:27.284 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:27.284 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:27.284 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:27.284 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:27.284 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:27.284 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:27.285 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:27.285 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:27.285 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 1 Processor -- start 2024-09-16T15:02:27.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 1 -- start start 2024-09-16T15:02:27.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient: init 2024-09-16T15:02:27.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:27.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:02:27.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient(hunting): picked mon.noname-a con 0x7feb400a0000 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:27.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient(hunting): picked mon.noname-b con 0x7feb400a1d10 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:27.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient(hunting): picked mon.noname-c con 0x7feb400a2800 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:27.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4009e500 con 0x7feb400a0000 2024-09-16T15:02:27.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb400a6400 con 0x7feb400a1d10 2024-09-16T15:02:27.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb40134a30 con 0x7feb400a2800 2024-09-16T15:02:27.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:27.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.271+0000 7feb47946640 10 monclient(hunting): authenticate will time out at 2171.395990s 2024-09-16T15:02:27.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb456e2640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7feb400a0000 0x7feb4013d770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38780/0 (socket says 172.21.15.26:38780) 2024-09-16T15:02:27.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb456e2640 1 -- 172.21.15.26:0/374940499 learned_addr learned my addr 172.21.15.26:0/374940499 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:27.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669740573 0 0) 0x7feb40134a30 con 0x7feb400a2800 2024-09-16T15:02:27.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient(hunting): my global_id is 354338 2024-09-16T15:02:27.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb24003530 con 0x7feb400a2800 2024-09-16T15:02:27.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286621240 0 0) 0x7feb4009e500 con 0x7feb400a0000 2024-09-16T15:02:27.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient(hunting): my global_id is 334524 2024-09-16T15:02:27.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb40134a30 con 0x7feb400a0000 2024-09-16T15:02:27.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1493563103 0 0) 0x7feb24003530 con 0x7feb400a2800 2024-09-16T15:02:27.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb4009e500 con 0x7feb400a2800 2024-09-16T15:02:27.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb38001b80 con 0x7feb400a2800 2024-09-16T15:02:27.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:02:27.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient(hunting): dump: 2024-09-16T15:02:27.288 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.288 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.288 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.288 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.288 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.288 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.288 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.288 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.288 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.288 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:27.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2086018219 0 0) 0x7feb40134a30 con 0x7feb400a0000 2024-09-16T15:02:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb24003530 con 0x7feb400a0000 2024-09-16T15:02:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb30001b50 con 0x7feb400a0000 2024-09-16T15:02:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient(hunting): dump: 2024-09-16T15:02:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.289 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.290 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.290 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 629582270 0 0) 0x7feb4009e500 con 0x7feb400a2800 2024-09-16T15:02:27.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 >> v1:172.21.15.189:6789/0 conn(0x7feb400a1d10 legacy=0x7feb401cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 >> v1:172.21.15.26:6789/0 conn(0x7feb400a0000 legacy=0x7feb4013d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient: _finish_hunting 0 2024-09-16T15:02:27.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 monclient: found mon.c 2024-09-16T15:02:27.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:27.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:27.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40136700 con 0x7feb400a2800 2024-09-16T15:02:27.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient: _finish_auth 0 2024-09-16T15:02:27.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient: _check_auth_tickets 2024-09-16T15:02:27.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:27.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb47946640 5 monclient: authenticate success, global_id 354338 2024-09-16T15:02:27.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb47946640 10 monclient: _renew_subs 2024-09-16T15:02:27.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb47946640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:27.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb47946640 1 -- 172.21.15.26:0/374940499 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb401369e0 con 0x7feb400a2800 2024-09-16T15:02:27.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb47946640 10 monclient: _renew_subs 2024-09-16T15:02:27.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb47946640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:27.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb47946640 1 -- 172.21.15.26:0/374940499 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb40135690 con 0x7feb400a2800 2024-09-16T15:02:27.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb380020b0 con 0x7feb400a2800 2024-09-16T15:02:27.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:27.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7feb38003b20 con 0x7feb400a2800 2024-09-16T15:02:27.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:27.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb377fe640 4 set_mon_vals no callback set 2024-09-16T15:02:27.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 10 monclient: dump: 2024-09-16T15:02:27.292 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.292 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.292 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.292 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.292 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.292 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.292 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.292 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.292 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.292 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb47946640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:27.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb47946640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:27.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb47946640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-09-16T15:02:27.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7feb38003c80 con 0x7feb400a2800 2024-09-16T15:02:27.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb47946640 1 -- 172.21.15.26:0/374940499 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb20005050 con 0x7feb400a2800 2024-09-16T15:02:27.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7feb38042dc0 con 0x7feb400a2800 2024-09-16T15:02:27.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.275+0000 7feb456e2640 10 monclient: get_auth_request con 0x7feb240362c0 auth_method 0 2024-09-16T15:02:27.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.279+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7feb38004430 con 0x7feb400a2800 2024-09-16T15:02:27.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.279+0000 7feb35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:27.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.279+0000 7feb35ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:27.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.435+0000 7feb47946640 1 -- 172.21.15.26:0/374940499 --> v1:172.21.15.189:6812/3196161582 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7feb20000ba0 con 0x7feb240362c0 2024-09-16T15:02:27.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.435+0000 7feb35ffb640 1 -- 172.21.15.26:0/374940499 <== mgr.4105 v1:172.21.15.189:6812/3196161582 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (unknown 3526570807 0 0) 0x7feb20000ba0 con 0x7feb240362c0 2024-09-16T15:02:27.437 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T15:02:27.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.439+0000 7feb47946640 1 -- 172.21.15.26:0/374940499 >> v1:172.21.15.189:6812/3196161582 conn(0x7feb240362c0 legacy=0x7feb24038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.439+0000 7feb47946640 10 monclient: shutdown 2024-09-16T15:02:27.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.439+0000 7feb47946640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:27.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.439+0000 7feb47946640 1 -- 172.21.15.26:0/374940499 >> v1:172.21.15.26:6790/0 conn(0x7feb400a2800 legacy=0x7feb401d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.439+0000 7feb47946640 1 -- 172.21.15.26:0/374940499 shutdown_connections 2024-09-16T15:02:27.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.439+0000 7feb47946640 1 -- 172.21.15.26:0/374940499 >> 172.21.15.26:0/374940499 conn(0x7feb400a7f30 msgr2=0x7feb400a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:27.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.439+0000 7feb47946640 1 -- 172.21.15.26:0/374940499 shutdown_connections 2024-09-16T15:02:27.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.439+0000 7feb47946640 1 -- 172.21.15.26:0/374940499 wait complete. 2024-09-16T15:02:27.454 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T15:02:27.454 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T15:02:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 10 monclient: get_monmap_and_config 2024-09-16T15:02:27.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 10 monclient: build_initial_monmap 2024-09-16T15:02:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 10 monclient: monmap: 2024-09-16T15:02:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:27.582661+0000 2024-09-16T15:02:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:27.582661+0000 2024-09-16T15:02:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:27.581 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:27.582 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:27.582 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 1 Processor -- start 2024-09-16T15:02:27.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 1 -- start start 2024-09-16T15:02:27.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 10 monclient: init 2024-09-16T15:02:27.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:27.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:02:27.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 10 monclient(hunting): picked mon.noname-b con 0x7f8c18134ba0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:27.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 10 monclient(hunting): picked mon.noname-c con 0x7f8c1809d9e0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:27.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 10 monclient(hunting): picked mon.noname-a con 0x7f8c180a03a0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:27.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c1809de30 con 0x7f8c180a03a0 2024-09-16T15:02:27.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c18135690 con 0x7f8c1809d9e0 2024-09-16T15:02:27.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c1809efc0 con 0x7f8c18134ba0 2024-09-16T15:02:27.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:27.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 10 monclient(hunting): authenticate will time out at 2171.702949s 2024-09-16T15:02:27.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1c83d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8c180a03a0 0x7f8c1813d1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38786/0 (socket says 172.21.15.26:38786) 2024-09-16T15:02:27.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c17fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8c18134ba0 0x7f8c18134f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51530/0 (socket says 172.21.15.26:51530) 2024-09-16T15:02:27.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1c83d640 1 -- 172.21.15.26:0/4081718820 learned_addr learned my addr 172.21.15.26:0/4081718820 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:27.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 1 -- 172.21.15.26:0/4081718820 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570861673 0 0) 0x7f8c1809efc0 con 0x7f8c18134ba0 2024-09-16T15:02:27.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient(hunting): my global_id is 354341 2024-09-16T15:02:27.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 1 -- 172.21.15.26:0/4081718820 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c00003530 con 0x7f8c18134ba0 2024-09-16T15:02:27.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 1 -- 172.21.15.26:0/4081718820 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919933950 0 0) 0x7f8c1809de30 con 0x7f8c180a03a0 2024-09-16T15:02:27.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient(hunting): my global_id is 334527 2024-09-16T15:02:27.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 1 -- 172.21.15.26:0/4081718820 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c1809efc0 con 0x7f8c180a03a0 2024-09-16T15:02:27.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 1 -- 172.21.15.26:0/4081718820 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2654363332 0 0) 0x7f8c00003530 con 0x7f8c18134ba0 2024-09-16T15:02:27.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 1 -- 172.21.15.26:0/4081718820 >> v1:172.21.15.189:6789/0 conn(0x7f8c1809d9e0 legacy=0x7f8c18139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 1 -- 172.21.15.26:0/4081718820 >> v1:172.21.15.26:6789/0 conn(0x7f8c180a03a0 legacy=0x7f8c1813d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient: _finish_hunting 0 2024-09-16T15:02:27.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 1 monclient: found mon.noname-b 2024-09-16T15:02:27.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:27.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:02:27.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 1 -- 172.21.15.26:0/4081718820 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c180a0870 con 0x7f8c18134ba0 2024-09-16T15:02:27.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient: _finish_auth 0 2024-09-16T15:02:27.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient: _check_auth_tickets 2024-09-16T15:02:27.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:27.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 1 -- 172.21.15.26:0/4081718820 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8c08001620 con 0x7f8c18134ba0 2024-09-16T15:02:27.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:02:27.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient: dump: 2024-09-16T15:02:27.592 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.592 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.592 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.592 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.592 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.592 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.592 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.592 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.592 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.592 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 5 monclient: authenticate success, global_id 354341 2024-09-16T15:02:27.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:27.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 1 -- 172.21.15.26:0/4081718820 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c080018c0 con 0x7f8c18134ba0 2024-09-16T15:02:27.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:27.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 1 -- 172.21.15.26:0/4081718820 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8c08001c90 con 0x7f8c18134ba0 2024-09-16T15:02:27.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:27.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c16ffd640 10 monclient: dump: 2024-09-16T15:02:27.593 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.593 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.593 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.593 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.593 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.593 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.593 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.593 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.593 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.593 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:27.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.579+0000 7f8c1e2a0640 4 set_mon_vals no callback set 2024-09-16T15:02:27.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 10 monclient: shutdown 2024-09-16T15:02:27.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:27.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 1 -- 172.21.15.26:0/4081718820 >> v1:172.21.15.26:6790/0 conn(0x7f8c18134ba0 legacy=0x7f8c18134f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 1 -- 172.21.15.26:0/4081718820 shutdown_connections 2024-09-16T15:02:27.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 1 -- 172.21.15.26:0/4081718820 >> 172.21.15.26:0/4081718820 conn(0x7f8c180a35e0 msgr2=0x7f8c1809fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:27.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 1 -- 172.21.15.26:0/4081718820 shutdown_connections 2024-09-16T15:02:27.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 1 -- 172.21.15.26:0/4081718820 wait complete. 2024-09-16T15:02:27.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 10 monclient: build_initial_monmap 2024-09-16T15:02:27.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 10 monclient: monmap: 2024-09-16T15:02:27.594 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:27.594 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:27.594 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:27.594 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:27.595 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:27.595 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:27.595 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:27.595 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:27.595 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:27.595 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 1 Processor -- start 2024-09-16T15:02:27.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 1 -- start start 2024-09-16T15:02:27.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 10 monclient: init 2024-09-16T15:02:27.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:27.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:02:27.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 10 monclient(hunting): picked mon.noname-a con 0x7f8c1809d9e0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:27.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 10 monclient(hunting): picked mon.noname-c con 0x7f8c180a03a0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:27.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 10 monclient(hunting): picked mon.noname-b con 0x7f8c18134ba0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:27.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c18138b00 con 0x7f8c1809d9e0 2024-09-16T15:02:27.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c1813c800 con 0x7f8c18134ba0 2024-09-16T15:02:27.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c18136680 con 0x7f8c180a03a0 2024-09-16T15:02:27.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:27.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 10 monclient(hunting): authenticate will time out at 2171.706093s 2024-09-16T15:02:27.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c17fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8c1809d9e0 0x7f8c18135860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38802/0 (socket says 172.21.15.26:38802) 2024-09-16T15:02:27.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c17fff640 1 -- 172.21.15.26:0/2256550599 learned_addr learned my addr 172.21.15.26:0/2256550599 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:27.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1179865255 0 0) 0x7f8c18138b00 con 0x7f8c1809d9e0 2024-09-16T15:02:27.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient(hunting): my global_id is 334530 2024-09-16T15:02:27.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c04003530 con 0x7f8c1809d9e0 2024-09-16T15:02:27.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739202985 0 0) 0x7f8c18136680 con 0x7f8c180a03a0 2024-09-16T15:02:27.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient(hunting): my global_id is 354344 2024-09-16T15:02:27.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c18138b00 con 0x7f8c180a03a0 2024-09-16T15:02:27.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4229544953 0 0) 0x7f8c04003530 con 0x7f8c1809d9e0 2024-09-16T15:02:27.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c18136680 con 0x7f8c1809d9e0 2024-09-16T15:02:27.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8c08001c10 con 0x7f8c1809d9e0 2024-09-16T15:02:27.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:27.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient(hunting): dump: 2024-09-16T15:02:27.598 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.598 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.598 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.598 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.598 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.598 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.598 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.598 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.598 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.598 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:27.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3031143502 0 0) 0x7f8c18138b00 con 0x7f8c180a03a0 2024-09-16T15:02:27.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c04003530 con 0x7f8c180a03a0 2024-09-16T15:02:27.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8c0c001880 con 0x7f8c180a03a0 2024-09-16T15:02:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient(hunting): dump: 2024-09-16T15:02:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.599 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3643337107 0 0) 0x7f8c18136680 con 0x7f8c1809d9e0 2024-09-16T15:02:27.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 >> v1:172.21.15.26:6790/0 conn(0x7f8c180a03a0 legacy=0x7f8c18135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 >> v1:172.21.15.189:6789/0 conn(0x7f8c18134ba0 legacy=0x7f8c181d1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient: _finish_hunting 0 2024-09-16T15:02:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 monclient: found mon.a 2024-09-16T15:02:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1809b840 con 0x7f8c1809d9e0 2024-09-16T15:02:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient: _finish_auth 0 2024-09-16T15:02:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient: _check_auth_tickets 2024-09-16T15:02:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 5 monclient: authenticate success, global_id 334530 2024-09-16T15:02:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 10 monclient: _renew_subs 2024-09-16T15:02:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:27.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 1 -- 172.21.15.26:0/2256550599 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c1809a230 con 0x7f8c1809d9e0 2024-09-16T15:02:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 10 monclient: _renew_subs 2024-09-16T15:02:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 1 -- 172.21.15.26:0/2256550599 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c1813d2b0 con 0x7f8c1809d9e0 2024-09-16T15:02:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c08004b40 con 0x7f8c1809d9e0 2024-09-16T15:02:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8c08004e50 con 0x7f8c1809d9e0 2024-09-16T15:02:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c167fc640 4 set_mon_vals no callback set 2024-09-16T15:02:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 10 monclient: dump: 2024-09-16T15:02:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.601 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.602 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c1e2a0640 1 -- 172.21.15.26:0/2256550599 --> v1:172.21.15.26:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8bf8000f20 con 0x7f8c1809d9e0 2024-09-16T15:02:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.583+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8c0800f8c0 con 0x7f8c1809d9e0 2024-09-16T15:02:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.587+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f8c08044050 con 0x7f8c1809d9e0 2024-09-16T15:02:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.587+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 --> v1:172.21.15.189:6800/1782891661 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c04043bf0 con 0x7f8c0403ff20 2024-09-16T15:02:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.587+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_get_version_reply(handle=1 version=257) v2 ==== 24+0+0 (unknown 1324813924 0 0) 0x7f8c080443a0 con 0x7f8c1809d9e0 2024-09-16T15:02:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.587+0000 7f8c14ff9640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-16T15:02:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.587+0000 7f8c177fe640 10 monclient: get_auth_request con 0x7f8c04036270 auth_method 0 2024-09-16T15:02:27.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.587+0000 7f8c1c83d640 10 monclient: get_auth_request con 0x7f8c0403ff20 auth_method 0 2024-09-16T15:02:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.587+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 <== osd.4 v1:172.21.15.189:6800/1782891661 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f8c04043bf0 con 0x7f8c0403ff20 2024-09-16T15:02:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.599+0000 7f8c1e2a0640 1 -- 172.21.15.26:0/2256550599 --> v1:172.21.15.189:6800/1782891661 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8bf8002b60 con 0x7f8c0403ff20 2024-09-16T15:02:27.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.603+0000 7f8c14ff9640 1 -- 172.21.15.26:0/2256550599 <== osd.4 v1:172.21.15.189:6800/1782891661 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f8bf8002b60 con 0x7f8c0403ff20 2024-09-16T15:02:27.604 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:02:27.604 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T15:02:27.604 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:02:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.603+0000 7f8c1e2a0640 1 -- 172.21.15.26:0/2256550599 >> v1:172.21.15.189:6800/1782891661 conn(0x7f8c0403ff20 legacy=0x7f8c04042380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.603+0000 7f8c1e2a0640 1 -- 172.21.15.26:0/2256550599 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8c04036270 legacy=0x7f8c04038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.603+0000 7f8c1e2a0640 10 monclient: shutdown 2024-09-16T15:02:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.603+0000 7f8c1e2a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.603+0000 7f8c1e2a0640 1 -- 172.21.15.26:0/2256550599 >> v1:172.21.15.26:6789/0 conn(0x7f8c1809d9e0 legacy=0x7f8c18135860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.603+0000 7f8c1c83d640 1 -- 172.21.15.26:0/2256550599 reap_dead start 2024-09-16T15:02:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.603+0000 7f8c1e2a0640 1 -- 172.21.15.26:0/2256550599 shutdown_connections 2024-09-16T15:02:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.603+0000 7f8c1e2a0640 1 -- 172.21.15.26:0/2256550599 >> 172.21.15.26:0/2256550599 conn(0x7f8c180a35e0 msgr2=0x7f8c1809a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.603+0000 7f8c1e2a0640 1 -- 172.21.15.26:0/2256550599 shutdown_connections 2024-09-16T15:02:27.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.603+0000 7f8c1e2a0640 1 -- 172.21.15.26:0/2256550599 wait complete. 2024-09-16T15:02:27.619 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T15:02:27.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 10 monclient: get_monmap_and_config 2024-09-16T15:02:27.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 10 monclient: build_initial_monmap 2024-09-16T15:02:27.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 10 monclient: monmap: 2024-09-16T15:02:27.728 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:27.728 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.728 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:27.730220+0000 2024-09-16T15:02:27.728 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:27.730220+0000 2024-09-16T15:02:27.728 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:27.729 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.729 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:27.729 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:27.729 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:27.729 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 1 Processor -- start 2024-09-16T15:02:27.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 1 -- start start 2024-09-16T15:02:27.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 10 monclient: init 2024-09-16T15:02:27.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:27.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:02:27.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 10 monclient(hunting): picked mon.noname-a con 0x7f963809a550 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:27.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 10 monclient(hunting): picked mon.noname-b con 0x7f963809d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:27.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 10 monclient(hunting): picked mon.noname-c con 0x7f963809b040 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:27.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963809dc60 con 0x7f963809a550 2024-09-16T15:02:27.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9638099ae0 con 0x7f963809b040 2024-09-16T15:02:27.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963809b8f0 con 0x7f963809d1b0 2024-09-16T15:02:27.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:27.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 10 monclient(hunting): authenticate will time out at 2171.850795s 2024-09-16T15:02:27.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f963809a550 0x7f963809a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38818/0 (socket says 172.21.15.26:38818) 2024-09-16T15:02:27.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963659c640 1 -- 172.21.15.26:0/2202712642 learned_addr learned my addr 172.21.15.26:0/2202712642 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:27.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 1 -- 172.21.15.26:0/2202712642 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091662530 0 0) 0x7f963809dc60 con 0x7f963809a550 2024-09-16T15:02:27.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient(hunting): my global_id is 334533 2024-09-16T15:02:27.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 1 -- 172.21.15.26:0/2202712642 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9624003530 con 0x7f963809a550 2024-09-16T15:02:27.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 1 -- 172.21.15.26:0/2202712642 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780681851 0 0) 0x7f963809b8f0 con 0x7f963809d1b0 2024-09-16T15:02:27.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient(hunting): my global_id is 354347 2024-09-16T15:02:27.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 1 -- 172.21.15.26:0/2202712642 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963809dc60 con 0x7f963809d1b0 2024-09-16T15:02:27.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 1 -- 172.21.15.26:0/2202712642 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2652202829 0 0) 0x7f963809dc60 con 0x7f963809d1b0 2024-09-16T15:02:27.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 1 -- 172.21.15.26:0/2202712642 >> v1:172.21.15.189:6789/0 conn(0x7f963809b040 legacy=0x7f963813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 1 -- 172.21.15.26:0/2202712642 >> v1:172.21.15.26:6789/0 conn(0x7f963809a550 legacy=0x7f963809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient: _finish_hunting 0 2024-09-16T15:02:27.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 1 monclient: found mon.noname-b 2024-09-16T15:02:27.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:27.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:02:27.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 1 -- 172.21.15.26:0/2202712642 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963809bbd0 con 0x7f963809d1b0 2024-09-16T15:02:27.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient: _finish_auth 0 2024-09-16T15:02:27.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient: _check_auth_tickets 2024-09-16T15:02:27.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:27.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 1 -- 172.21.15.26:0/2202712642 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9620001620 con 0x7f963809d1b0 2024-09-16T15:02:27.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 5 monclient: authenticate success, global_id 354347 2024-09-16T15:02:27.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:27.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:02:27.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient: dump: 2024-09-16T15:02:27.735 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.735 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.735 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.735 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.735 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.735 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.736 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.736 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.736 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.736 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:27.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 1 -- 172.21.15.26:0/2202712642 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9620001840 con 0x7f963809d1b0 2024-09-16T15:02:27.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:27.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 1 -- 172.21.15.26:0/2202712642 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9620001b50 con 0x7f963809d1b0 2024-09-16T15:02:27.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:27.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f963c8c1640 4 set_mon_vals no callback set 2024-09-16T15:02:27.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:27.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.727+0000 7f9635d9b640 10 monclient: dump: 2024-09-16T15:02:27.736 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.737 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.737 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.737 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.737 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.737 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.737 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.737 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.737 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.737 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient: shutdown 2024-09-16T15:02:27.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:27.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 1 -- 172.21.15.26:0/2202712642 >> v1:172.21.15.26:6790/0 conn(0x7f963809d1b0 legacy=0x7f9638136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 1 -- 172.21.15.26:0/2202712642 shutdown_connections 2024-09-16T15:02:27.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 1 -- 172.21.15.26:0/2202712642 >> 172.21.15.26:0/2202712642 conn(0x7f963809fb50 msgr2=0x7f963809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:27.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 1 -- 172.21.15.26:0/2202712642 shutdown_connections 2024-09-16T15:02:27.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 1 -- 172.21.15.26:0/2202712642 wait complete. 2024-09-16T15:02:27.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient: build_initial_monmap 2024-09-16T15:02:27.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient: monmap: 2024-09-16T15:02:27.738 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:27.738 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:27.738 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:27.738 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:27.738 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:27.738 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:27.738 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:27.738 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:27.739 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:27.739 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 1 Processor -- start 2024-09-16T15:02:27.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 1 -- start start 2024-09-16T15:02:27.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient: init 2024-09-16T15:02:27.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:27.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:02:27.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient(hunting): picked mon.noname-b con 0x7f963809a550 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:27.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient(hunting): picked mon.noname-c con 0x7f963809b040 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:27.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient(hunting): picked mon.noname-a con 0x7f963809d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:27.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96381394e0 con 0x7f963809d1b0 2024-09-16T15:02:27.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963813a0b0 con 0x7f963809a550 2024-09-16T15:02:27.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9638119e00 con 0x7f963809b040 2024-09-16T15:02:27.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:27.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient(hunting): authenticate will time out at 2171.853669s 2024-09-16T15:02:27.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f9636d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f963809d1b0 0x7f96381d0030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38834/0 (socket says 172.21.15.26:38834) 2024-09-16T15:02:27.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f963809b040 0x7f96381cc900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51546/0 (socket says 172.21.15.26:51546) 2024-09-16T15:02:27.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f9636d9d640 1 -- 172.21.15.26:0/584643026 learned_addr learned my addr 172.21.15.26:0/584643026 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:27.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014883752 0 0) 0x7f96381394e0 con 0x7f963809d1b0 2024-09-16T15:02:27.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): my global_id is 334536 2024-09-16T15:02:27.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9618003530 con 0x7f963809d1b0 2024-09-16T15:02:27.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424776526 0 0) 0x7f9638119e00 con 0x7f963809b040 2024-09-16T15:02:27.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): my global_id is 354350 2024-09-16T15:02:27.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96381394e0 con 0x7f963809b040 2024-09-16T15:02:27.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 115542579 0 0) 0x7f9618003530 con 0x7f963809d1b0 2024-09-16T15:02:27.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9638119e00 con 0x7f963809d1b0 2024-09-16T15:02:27.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9630001c80 con 0x7f963809d1b0 2024-09-16T15:02:27.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:27.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): dump: 2024-09-16T15:02:27.744 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.744 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.744 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.744 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.744 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.744 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.745 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.745 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.745 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.745 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:27.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3119047525 0 0) 0x7f96381394e0 con 0x7f963809b040 2024-09-16T15:02:27.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9618003530 con 0x7f963809b040 2024-09-16T15:02:27.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9620001520 con 0x7f963809b040 2024-09-16T15:02:27.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:27.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): dump: 2024-09-16T15:02:27.745 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.746 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.746 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.746 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.746 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.746 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.746 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.746 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.746 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.746 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549370847 0 0) 0x7f963813a0b0 con 0x7f963809a550 2024-09-16T15:02:27.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:27.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:27.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:27.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient(hunting): my global_id is 364216 2024-09-16T15:02:27.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96381394e0 con 0x7f963809a550 2024-09-16T15:02:27.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3017170979 0 0) 0x7f9638119e00 con 0x7f963809d1b0 2024-09-16T15:02:27.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 >> v1:172.21.15.26:6790/0 conn(0x7f963809b040 legacy=0x7f96381cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 >> v1:172.21.15.189:6789/0 conn(0x7f963809a550 legacy=0x7f96381c7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient: _finish_hunting 0 2024-09-16T15:02:27.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 monclient: found mon.a 2024-09-16T15:02:27.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:27.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:27.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9638139e30 con 0x7f963809d1b0 2024-09-16T15:02:27.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient: _finish_auth 0 2024-09-16T15:02:27.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient: _check_auth_tickets 2024-09-16T15:02:27.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:27.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 5 monclient: authenticate success, global_id 334536 2024-09-16T15:02:27.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient: _renew_subs 2024-09-16T15:02:27.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:27.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 1 -- 172.21.15.26:0/584643026 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9638131940 con 0x7f963809d1b0 2024-09-16T15:02:27.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient: _renew_subs 2024-09-16T15:02:27.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:27.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 1 -- 172.21.15.26:0/584643026 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9638119b20 con 0x7f963809d1b0 2024-09-16T15:02:27.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9630001fd0 con 0x7f963809d1b0 2024-09-16T15:02:27.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:27.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f9630003e70 con 0x7f963809d1b0 2024-09-16T15:02:27.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:27.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:27.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 10 monclient: dump: 2024-09-16T15:02:27.748 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:27.749 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:27.749 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.749 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:27.749 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:27.749 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:27.749 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:27.749 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:27.749 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:27.749 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:27.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963559a640 4 set_mon_vals no callback set 2024-09-16T15:02:27.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:27.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:27.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:27.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f963c8c1640 1 -- 172.21.15.26:0/584643026 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961c005050 con 0x7f963809d1b0 2024-09-16T15:02:27.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f9630003fd0 con 0x7f963809d1b0 2024-09-16T15:02:27.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.731+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f9630043fb0 con 0x7f963809d1b0 2024-09-16T15:02:27.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.735+0000 7f963659c640 10 monclient: get_auth_request con 0x7f9618036370 auth_method 0 2024-09-16T15:02:27.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.735+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9630014890 con 0x7f963809d1b0 2024-09-16T15:02:27.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.735+0000 7f962effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:27.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.735+0000 7f962effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:27.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.891+0000 7f963c8c1640 1 -- 172.21.15.26:0/584643026 --> v1:172.21.15.189:6812/3196161582 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f961c000ba0 con 0x7f9618036370 2024-09-16T15:02:27.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.895+0000 7f962effd640 1 -- 172.21.15.26:0/584643026 <== mgr.4105 v1:172.21.15.189:6812/3196161582 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (unknown 3782719169 0 0) 0x7f961c000ba0 con 0x7f9618036370 2024-09-16T15:02:27.895 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T15:02:27.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.895+0000 7f963c8c1640 1 -- 172.21.15.26:0/584643026 >> v1:172.21.15.189:6812/3196161582 conn(0x7f9618036370 legacy=0x7f9618038830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.895+0000 7f963c8c1640 10 monclient: shutdown 2024-09-16T15:02:27.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.895+0000 7f963c8c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:27.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.895+0000 7f963c8c1640 1 -- 172.21.15.26:0/584643026 >> v1:172.21.15.26:6789/0 conn(0x7f963809d1b0 legacy=0x7f96381d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:27.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.895+0000 7f963c8c1640 1 -- 172.21.15.26:0/584643026 shutdown_connections 2024-09-16T15:02:27.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.895+0000 7f963c8c1640 1 -- 172.21.15.26:0/584643026 >> 172.21.15.26:0/584643026 conn(0x7f963809fb50 msgr2=0x7f963809e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:27.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.895+0000 7f963c8c1640 1 -- 172.21.15.26:0/584643026 shutdown_connections 2024-09-16T15:02:27.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:27.895+0000 7f963c8c1640 1 -- 172.21.15.26:0/584643026 wait complete. 2024-09-16T15:02:27.912 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T15:02:27.913 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T15:02:28.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.039+0000 7f58f4ab6640 10 monclient: get_monmap_and_config 2024-09-16T15:02:28.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.039+0000 7f58f4ab6640 10 monclient: build_initial_monmap 2024-09-16T15:02:28.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.039+0000 7f58f4ab6640 10 monclient: monmap: 2024-09-16T15:02:28.042 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:28.042 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.043 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:28.044443+0000 2024-09-16T15:02:28.043 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:28.044443+0000 2024-09-16T15:02:28.043 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:28.043 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.043 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:28.043 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:28.043 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:28.043 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.039+0000 7f58f4ab6640 1 Processor -- start 2024-09-16T15:02:28.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.039+0000 7f58f4ab6640 1 -- start start 2024-09-16T15:02:28.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.039+0000 7f58f4ab6640 10 monclient: init 2024-09-16T15:02:28.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:28.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:02:28.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient(hunting): picked mon.noname-b con 0x7f58f009a4e0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:28.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient(hunting): picked mon.noname-c con 0x7f58f009afd0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:28.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient(hunting): picked mon.noname-a con 0x7f58f0138110 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:28.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f009b810 con 0x7f58f0138110 2024-09-16T15:02:28.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f009ecf0 con 0x7f58f009afd0 2024-09-16T15:02:28.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f009baf0 con 0x7f58f009a4e0 2024-09-16T15:02:28.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:28.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient(hunting): authenticate will time out at 2172.164495s 2024-09-16T15:02:28.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58eed9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f58f0138110 0x7f58f013a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38842/0 (socket says 172.21.15.26:38842) 2024-09-16T15:02:28.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58eed9d640 1 -- 172.21.15.26:0/1809462722 learned_addr learned my addr 172.21.15.26:0/1809462722 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:28.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ee59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f58f009a4e0 0x7f58f009a8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51560/0 (socket says 172.21.15.26:51560) 2024-09-16T15:02:28.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 1 -- 172.21.15.26:0/1809462722 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071408687 0 0) 0x7f58f009b810 con 0x7f58f0138110 2024-09-16T15:02:28.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient(hunting): _negotiate 2024-09-16T15:02:28.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:28.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:28.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient(hunting): my global_id is 334539 2024-09-16T15:02:28.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 1 -- 172.21.15.26:0/1809462722 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58dc003530 con 0x7f58f0138110 2024-09-16T15:02:28.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 1 -- 172.21.15.26:0/1809462722 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225392225 0 0) 0x7f58f009baf0 con 0x7f58f009a4e0 2024-09-16T15:02:28.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient(hunting): _negotiate 2024-09-16T15:02:28.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:28.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:28.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient(hunting): my global_id is 354353 2024-09-16T15:02:28.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 1 -- 172.21.15.26:0/1809462722 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58f009b810 con 0x7f58f009a4e0 2024-09-16T15:02:28.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 1 -- 172.21.15.26:0/1809462722 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 909337767 0 0) 0x7f58dc003530 con 0x7f58f0138110 2024-09-16T15:02:28.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 1 -- 172.21.15.26:0/1809462722 >> v1:172.21.15.26:6790/0 conn(0x7f58f009a4e0 legacy=0x7f58f009a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 1 -- 172.21.15.26:0/1809462722 >> v1:172.21.15.189:6789/0 conn(0x7f58f009afd0 legacy=0x7f58f01369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient: _finish_hunting 0 2024-09-16T15:02:28.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 1 monclient: found mon.noname-a 2024-09-16T15:02:28.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:28.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:02:28.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 1 -- 172.21.15.26:0/1809462722 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f009bdd0 con 0x7f58f0138110 2024-09-16T15:02:28.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient: _finish_auth 0 2024-09-16T15:02:28.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient: _check_auth_tickets 2024-09-16T15:02:28.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:28.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 1 -- 172.21.15.26:0/1809462722 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58e80019a0 con 0x7f58f0138110 2024-09-16T15:02:28.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 5 monclient: authenticate success, global_id 334539 2024-09-16T15:02:28.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:28.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:28.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:28.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient: dump: 2024-09-16T15:02:28.053 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:28.053 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.053 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.054 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.054 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:28.054 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.054 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:28.054 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:28.054 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:28.054 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:28.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 1 -- 172.21.15.26:0/1809462722 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58e8001bd0 con 0x7f58f0138110 2024-09-16T15:02:28.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:28.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 1 -- 172.21.15.26:0/1809462722 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58e8001f60 con 0x7f58f0138110 2024-09-16T15:02:28.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:28.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:28.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ed59a640 10 monclient: dump: 2024-09-16T15:02:28.054 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:28.055 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.055 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.055 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.055 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:28.055 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.055 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:28.055 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:28.055 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:28.055 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:28.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 4 set_mon_vals no callback set 2024-09-16T15:02:28.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient: shutdown 2024-09-16T15:02:28.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:28.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 1 -- 172.21.15.26:0/1809462722 >> v1:172.21.15.26:6789/0 conn(0x7f58f0138110 legacy=0x7f58f013a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 1 -- 172.21.15.26:0/1809462722 shutdown_connections 2024-09-16T15:02:28.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 1 -- 172.21.15.26:0/1809462722 >> 172.21.15.26:0/1809462722 conn(0x7f58f00a35e0 msgr2=0x7f58f009fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:28.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 1 -- 172.21.15.26:0/1809462722 shutdown_connections 2024-09-16T15:02:28.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 1 -- 172.21.15.26:0/1809462722 wait complete. 2024-09-16T15:02:28.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient: build_initial_monmap 2024-09-16T15:02:28.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient: monmap: 2024-09-16T15:02:28.056 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:28.056 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:28.056 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:28.056 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:28.056 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:28.057 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:28.057 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:28.057 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:28.057 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:28.057 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 1 Processor -- start 2024-09-16T15:02:28.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 1 -- start start 2024-09-16T15:02:28.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient: init 2024-09-16T15:02:28.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:28.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:02:28.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient(hunting): picked mon.noname-c con 0x7f58f009a4e0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:28.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient(hunting): picked mon.noname-b con 0x7f58f009afd0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:28.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient(hunting): picked mon.noname-a con 0x7f58f0138110 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:28.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f011ff10 con 0x7f58f0138110 2024-09-16T15:02:28.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f009a1d0 con 0x7f58f009afd0 2024-09-16T15:02:28.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f0137be0 con 0x7f58f009a4e0 2024-09-16T15:02:28.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:28.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58f4ab6640 10 monclient(hunting): authenticate will time out at 2172.167578s 2024-09-16T15:02:28.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ee59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f58f009a4e0 0x7f58f0131820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51576/0 (socket says 172.21.15.26:51576) 2024-09-16T15:02:28.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58ee59c640 1 -- 172.21.15.26:0/1551840424 learned_addr learned my addr 172.21.15.26:0/1551840424 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:28.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661118025 0 0) 0x7f58f0137be0 con 0x7f58f009a4e0 2024-09-16T15:02:28.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58e6ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:28.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:28.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:28.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58e6ffd640 10 monclient(hunting): my global_id is 354356 2024-09-16T15:02:28.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d0003530 con 0x7f58f009a4e0 2024-09-16T15:02:28.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034319855 0 0) 0x7f58f011ff10 con 0x7f58f0138110 2024-09-16T15:02:28.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58e6ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:28.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:28.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:28.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58e6ffd640 10 monclient(hunting): my global_id is 334542 2024-09-16T15:02:28.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.043+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58f0137be0 con 0x7f58f0138110 2024-09-16T15:02:28.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2339746251 0 0) 0x7f58d0003530 con 0x7f58f009a4e0 2024-09-16T15:02:28.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58f011ff10 con 0x7f58f009a4e0 2024-09-16T15:02:28.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58e00018e0 con 0x7f58f009a4e0 2024-09-16T15:02:28.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:28.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T15:02:28.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient(hunting): dump: 2024-09-16T15:02:28.059 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:28.059 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.059 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.059 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.059 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:28.059 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.060 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:28.060 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:28.060 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:28.060 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:28.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1018180367 0 0) 0x7f58f0137be0 con 0x7f58f0138110 2024-09-16T15:02:28.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58d0003530 con 0x7f58f0138110 2024-09-16T15:02:28.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58e8001ee0 con 0x7f58f0138110 2024-09-16T15:02:28.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:28.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:28.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient(hunting): dump: 2024-09-16T15:02:28.060 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:28.060 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.060 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.061 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.061 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:28.061 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.061 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:28.061 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:28.061 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:28.061 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336174555 0 0) 0x7f58f009a1d0 con 0x7f58f009afd0 2024-09-16T15:02:28.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:28.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:28.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:28.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient(hunting): my global_id is 364219 2024-09-16T15:02:28.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58f0137be0 con 0x7f58f009afd0 2024-09-16T15:02:28.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1661356871 0 0) 0x7f58d0003530 con 0x7f58f0138110 2024-09-16T15:02:28.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 >> v1:172.21.15.26:6790/0 conn(0x7f58f009a4e0 legacy=0x7f58f0131820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 >> v1:172.21.15.189:6789/0 conn(0x7f58f009afd0 legacy=0x7f58f01ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient: _finish_hunting 0 2024-09-16T15:02:28.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 monclient: found mon.a 2024-09-16T15:02:28.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:28.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:28.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f011e9f0 con 0x7f58f0138110 2024-09-16T15:02:28.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient: _finish_auth 0 2024-09-16T15:02:28.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient: _check_auth_tickets 2024-09-16T15:02:28.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:28.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58f4ab6640 5 monclient: authenticate success, global_id 334542 2024-09-16T15:02:28.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58f4ab6640 10 monclient: _renew_subs 2024-09-16T15:02:28.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58f4ab6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:28.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58f4ab6640 1 -- 172.21.15.26:0/1551840424 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f00a1be0 con 0x7f58f0138110 2024-09-16T15:02:28.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58f4ab6640 10 monclient: _renew_subs 2024-09-16T15:02:28.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58f4ab6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:28.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58f4ab6640 1 -- 172.21.15.26:0/1551840424 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f0131f30 con 0x7f58f0138110 2024-09-16T15:02:28.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58e8001430 con 0x7f58f0138110 2024-09-16T15:02:28.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:28.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f58e8005000 con 0x7f58f0138110 2024-09-16T15:02:28.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:28.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:28.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient: dump: 2024-09-16T15:02:28.063 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:28.063 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.063 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.063 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.064 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:28.064 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.064 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:28.064 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:28.064 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:28.064 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58ecd99640 4 set_mon_vals no callback set 2024-09-16T15:02:28.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58f4ab6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:28.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58f4ab6640 1 -- 172.21.15.26:0/1551840424 --> v1:172.21.15.26:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f58f011fca0 con 0x7f58f0138110 2024-09-16T15:02:28.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f58e800fa70 con 0x7f58f0138110 2024-09-16T15:02:28.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f58e8044060 con 0x7f58f0138110 2024-09-16T15:02:28.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 --> v1:172.21.15.189:6804/3793314587 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58d0043d90 con 0x7f58d00400c0 2024-09-16T15:02:28.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_get_version_reply(handle=1 version=257) v2 ==== 24+0+0 (unknown 1324813924 0 0) 0x7f58e8044430 con 0x7f58f0138110 2024-09-16T15:02:28.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58e6ffd640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-16T15:02:28.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58ee59c640 10 monclient: get_auth_request con 0x7f58d0036410 auth_method 0 2024-09-16T15:02:28.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.047+0000 7f58edd9b640 10 monclient: get_auth_request con 0x7f58d00400c0 auth_method 0 2024-09-16T15:02:28.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.051+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 <== osd.5 v1:172.21.15.189:6804/3793314587 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f58d0043d90 con 0x7f58d00400c0 2024-09-16T15:02:28.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.063+0000 7f58f4ab6640 1 -- 172.21.15.26:0/1551840424 --> v1:172.21.15.189:6804/3793314587 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f58f009cae0 con 0x7f58d00400c0 2024-09-16T15:02:28.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.063+0000 7f58e6ffd640 1 -- 172.21.15.26:0/1551840424 <== osd.5 v1:172.21.15.189:6804/3793314587 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f58d8001820 con 0x7f58d00400c0 2024-09-16T15:02:28.066 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-09-16T15:02:28.066 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T15:02:28.066 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-09-16T15:02:28.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.063+0000 7f58f4ab6640 1 -- 172.21.15.26:0/1551840424 >> v1:172.21.15.189:6804/3793314587 conn(0x7f58d00400c0 legacy=0x7f58d0042520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.063+0000 7f58f4ab6640 1 -- 172.21.15.26:0/1551840424 >> v1:172.21.15.189:6812/3196161582 conn(0x7f58d0036410 legacy=0x7f58d00388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.063+0000 7f58f4ab6640 10 monclient: shutdown 2024-09-16T15:02:28.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.063+0000 7f58f4ab6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:28.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.063+0000 7f58f4ab6640 1 -- 172.21.15.26:0/1551840424 >> v1:172.21.15.26:6789/0 conn(0x7f58f0138110 legacy=0x7f58f01d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.063+0000 7f58eed9d640 1 -- 172.21.15.26:0/1551840424 reap_dead start 2024-09-16T15:02:28.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.063+0000 7f58f4ab6640 1 -- 172.21.15.26:0/1551840424 shutdown_connections 2024-09-16T15:02:28.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.063+0000 7f58f4ab6640 1 -- 172.21.15.26:0/1551840424 >> 172.21.15.26:0/1551840424 conn(0x7f58f00a35e0 msgr2=0x7f58f0071340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:28.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.063+0000 7f58f4ab6640 1 -- 172.21.15.26:0/1551840424 shutdown_connections 2024-09-16T15:02:28.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.063+0000 7f58f4ab6640 1 -- 172.21.15.26:0/1551840424 wait complete. 2024-09-16T15:02:28.079 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T15:02:28.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d4dbc3640 10 monclient: get_monmap_and_config 2024-09-16T15:02:28.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d4dbc3640 10 monclient: build_initial_monmap 2024-09-16T15:02:28.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d4dbc3640 10 monclient: monmap: 2024-09-16T15:02:28.193 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:28.193 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.193 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:28.195358+0000 2024-09-16T15:02:28.194 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:28.195358+0000 2024-09-16T15:02:28.194 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:28.194 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.194 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:28.194 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:28.194 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:28.194 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d4dbc3640 1 Processor -- start 2024-09-16T15:02:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d4dbc3640 1 -- start start 2024-09-16T15:02:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d4dbc3640 10 monclient: init 2024-09-16T15:02:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d4dbc3640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d4dbc3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:02:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d4dbc3640 10 monclient(hunting): picked mon.noname-a con 0x7f8d4809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d4dbc3640 10 monclient(hunting): picked mon.noname-b con 0x7f8d48132650 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:28.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d4dbc3640 10 monclient(hunting): picked mon.noname-c con 0x7f8d48138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d4dbc3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4809d260 con 0x7f8d4809cb80 2024-09-16T15:02:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d4dbc3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d48132a80 con 0x7f8d48138ff0 2024-09-16T15:02:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d4dbc3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4809cf60 con 0x7f8d48132650 2024-09-16T15:02:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d4dbc3640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d4dbc3640 10 monclient(hunting): authenticate will time out at 2172.315426s 2024-09-16T15:02:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d477fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8d4809cb80 0x7f8d480717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38864/0 (socket says 172.21.15.26:38864) 2024-09-16T15:02:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d477fe640 1 -- 172.21.15.26:0/1995067006 learned_addr learned my addr 172.21.15.26:0/1995067006 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d467fc640 1 -- 172.21.15.26:0/1995067006 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490554406 0 0) 0x7f8d4809d260 con 0x7f8d4809cb80 2024-09-16T15:02:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d467fc640 10 monclient(hunting): _negotiate 2024-09-16T15:02:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:28.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d467fc640 10 monclient(hunting): my global_id is 334545 2024-09-16T15:02:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d467fc640 1 -- 172.21.15.26:0/1995067006 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d30003530 con 0x7f8d4809cb80 2024-09-16T15:02:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d467fc640 1 -- 172.21.15.26:0/1995067006 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599987065 0 0) 0x7f8d4809cf60 con 0x7f8d48132650 2024-09-16T15:02:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d467fc640 10 monclient(hunting): _negotiate 2024-09-16T15:02:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d467fc640 10 monclient(hunting): my global_id is 354359 2024-09-16T15:02:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.191+0000 7f8d467fc640 1 -- 172.21.15.26:0/1995067006 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d4809d260 con 0x7f8d48132650 2024-09-16T15:02:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 1 -- 172.21.15.26:0/1995067006 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1381917796 0 0) 0x7f8d30003530 con 0x7f8d4809cb80 2024-09-16T15:02:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 1 -- 172.21.15.26:0/1995067006 >> v1:172.21.15.26:6790/0 conn(0x7f8d48132650 legacy=0x7f8d481378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 1 -- 172.21.15.26:0/1995067006 >> v1:172.21.15.189:6789/0 conn(0x7f8d48138ff0 legacy=0x7f8d4813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 10 monclient: _finish_hunting 0 2024-09-16T15:02:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 1 monclient: found mon.noname-a 2024-09-16T15:02:28.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:02:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 1 -- 172.21.15.26:0/1995067006 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4809f2d0 con 0x7f8d4809cb80 2024-09-16T15:02:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 10 monclient: _finish_auth 0 2024-09-16T15:02:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 10 monclient: _check_auth_tickets 2024-09-16T15:02:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 1 -- 172.21.15.26:0/1995067006 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8d400019a0 con 0x7f8d4809cb80 2024-09-16T15:02:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 10 monclient: dump: 2024-09-16T15:02:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.204 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:28.205 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 5 monclient: authenticate success, global_id 334545 2024-09-16T15:02:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 1 -- 172.21.15.26:0/1995067006 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d40001b50 con 0x7f8d4809cb80 2024-09-16T15:02:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 1 -- 172.21.15.26:0/1995067006 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8d40001f20 con 0x7f8d4809cb80 2024-09-16T15:02:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 4 set_mon_vals no callback set 2024-09-16T15:02:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d467fc640 10 monclient: dump: 2024-09-16T15:02:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:28.206 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 10 monclient: shutdown 2024-09-16T15:02:28.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 1 -- 172.21.15.26:0/1995067006 >> v1:172.21.15.26:6789/0 conn(0x7f8d4809cb80 legacy=0x7f8d480717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 1 -- 172.21.15.26:0/1995067006 shutdown_connections 2024-09-16T15:02:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 1 -- 172.21.15.26:0/1995067006 >> 172.21.15.26:0/1995067006 conn(0x7f8d4809fb50 msgr2=0x7f8d4809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 1 -- 172.21.15.26:0/1995067006 shutdown_connections 2024-09-16T15:02:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 1 -- 172.21.15.26:0/1995067006 wait complete. 2024-09-16T15:02:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 10 monclient: build_initial_monmap 2024-09-16T15:02:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 10 monclient: monmap: 2024-09-16T15:02:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:28.207 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:28.208 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:28.208 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:28.208 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 1 Processor -- start 2024-09-16T15:02:28.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 1 -- start start 2024-09-16T15:02:28.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 10 monclient: init 2024-09-16T15:02:28.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:28.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:02:28.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 10 monclient(hunting): picked mon.noname-c con 0x7f8d4809cb80 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:28.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 10 monclient(hunting): picked mon.noname-b con 0x7f8d48132650 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:28.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 10 monclient(hunting): picked mon.noname-a con 0x7f8d48138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:28.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4809a240 con 0x7f8d48138ff0 2024-09-16T15:02:28.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4806e440 con 0x7f8d48132650 2024-09-16T15:02:28.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4813c5e0 con 0x7f8d4809cb80 2024-09-16T15:02:28.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:28.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 10 monclient(hunting): authenticate will time out at 2172.318311s 2024-09-16T15:02:28.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d477fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f8d4809cb80 0x7f8d4806d620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51598/0 (socket says 172.21.15.26:51598) 2024-09-16T15:02:28.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d477fe640 1 -- 172.21.15.26:0/2238073560 learned_addr learned my addr 172.21.15.26:0/2238073560 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:28.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d47fff640 1 --1- 172.21.15.26:0/2238073560 >> v1:172.21.15.26:6789/0 conn(0x7f8d48138ff0 0x7f8d481cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38880/0 (socket says 172.21.15.26:38880) 2024-09-16T15:02:28.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115036314 0 0) 0x7f8d4809a240 con 0x7f8d48138ff0 2024-09-16T15:02:28.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:28.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:28.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:28.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): my global_id is 334548 2024-09-16T15:02:28.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d34003530 con 0x7f8d48138ff0 2024-09-16T15:02:28.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636672467 0 0) 0x7f8d4813c5e0 con 0x7f8d4809cb80 2024-09-16T15:02:28.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:28.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:28.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:28.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): my global_id is 354362 2024-09-16T15:02:28.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d4809a240 con 0x7f8d4809cb80 2024-09-16T15:02:28.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1828207203 0 0) 0x7f8d34003530 con 0x7f8d48138ff0 2024-09-16T15:02:28.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d4813c5e0 con 0x7f8d48138ff0 2024-09-16T15:02:28.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8d38001840 con 0x7f8d48138ff0 2024-09-16T15:02:28.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:28.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:28.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): dump: 2024-09-16T15:02:28.210 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:28.210 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.210 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.211 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.211 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:28.211 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.211 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:28.211 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:28.211 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:28.211 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:28.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 543397134 0 0) 0x7f8d4809a240 con 0x7f8d4809cb80 2024-09-16T15:02:28.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d34003530 con 0x7f8d4809cb80 2024-09-16T15:02:28.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8d40001e60 con 0x7f8d4809cb80 2024-09-16T15:02:28.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:28.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:28.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): dump: 2024-09-16T15:02:28.212 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:28.212 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.212 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.212 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.212 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:28.212 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.212 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:28.212 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:28.212 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:28.212 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365435954 0 0) 0x7f8d4806e440 con 0x7f8d48132650 2024-09-16T15:02:28.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:28.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:28.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:28.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient(hunting): my global_id is 364225 2024-09-16T15:02:28.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d4809a240 con 0x7f8d48132650 2024-09-16T15:02:28.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4046383249 0 0) 0x7f8d4813c5e0 con 0x7f8d48138ff0 2024-09-16T15:02:28.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 >> v1:172.21.15.26:6790/0 conn(0x7f8d4809cb80 legacy=0x7f8d4806d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 >> v1:172.21.15.189:6789/0 conn(0x7f8d48132650 legacy=0x7f8d4806dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient: _finish_hunting 0 2024-09-16T15:02:28.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 monclient: found mon.a 2024-09-16T15:02:28.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:28.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:28.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d48071580 con 0x7f8d48138ff0 2024-09-16T15:02:28.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient: _finish_auth 0 2024-09-16T15:02:28.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient: _check_auth_tickets 2024-09-16T15:02:28.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:28.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 5 monclient: authenticate success, global_id 334548 2024-09-16T15:02:28.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 10 monclient: _renew_subs 2024-09-16T15:02:28.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:28.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 1 -- 172.21.15.26:0/2238073560 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d481380c0 con 0x7f8d48138ff0 2024-09-16T15:02:28.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 10 monclient: _renew_subs 2024-09-16T15:02:28.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:28.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d4dbc3640 1 -- 172.21.15.26:0/2238073560 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4806fb70 con 0x7f8d48138ff0 2024-09-16T15:02:28.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d38003620 con 0x7f8d48138ff0 2024-09-16T15:02:28.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:28.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f8d380039b0 con 0x7f8d48138ff0 2024-09-16T15:02:28.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:28.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:28.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d45ffb640 4 set_mon_vals no callback set 2024-09-16T15:02:28.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.195+0000 7f8d2ffff640 10 monclient: dump: 2024-09-16T15:02:28.215 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:28.215 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.215 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.215 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.215 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:28.215 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.215 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:28.215 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:28.215 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:28.215 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.199+0000 7f8d4dbc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:28.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.199+0000 7f8d4dbc3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:28.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.199+0000 7f8d4dbc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:28.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.199+0000 7f8d4dbc3640 1 -- 172.21.15.26:0/2238073560 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d28005050 con 0x7f8d48138ff0 2024-09-16T15:02:28.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.199+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f8d38003b10 con 0x7f8d48138ff0 2024-09-16T15:02:28.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.199+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f8d38043c10 con 0x7f8d48138ff0 2024-09-16T15:02:28.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.199+0000 7f8d477fe640 10 monclient: get_auth_request con 0x7f8d340363a0 auth_method 0 2024-09-16T15:02:28.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.203+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8d38014620 con 0x7f8d48138ff0 2024-09-16T15:02:28.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.203+0000 7f8d2ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:28.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.203+0000 7f8d2ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:28.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.355+0000 7f8d4dbc3640 1 -- 172.21.15.26:0/2238073560 --> v1:172.21.15.189:6812/3196161582 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d28000ba0 con 0x7f8d340363a0 2024-09-16T15:02:28.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.359+0000 7f8d2ffff640 1 -- 172.21.15.26:0/2238073560 <== mgr.4105 v1:172.21.15.189:6812/3196161582 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (unknown 2264484800 0 0) 0x7f8d28000ba0 con 0x7f8d340363a0 2024-09-16T15:02:28.359 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T15:02:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.359+0000 7f8d4dbc3640 1 -- 172.21.15.26:0/2238073560 >> v1:172.21.15.189:6812/3196161582 conn(0x7f8d340363a0 legacy=0x7f8d34038860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.359+0000 7f8d4dbc3640 10 monclient: shutdown 2024-09-16T15:02:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.359+0000 7f8d4dbc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.359+0000 7f8d4dbc3640 1 -- 172.21.15.26:0/2238073560 >> v1:172.21.15.26:6789/0 conn(0x7f8d48138ff0 legacy=0x7f8d481cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.359+0000 7f8d4dbc3640 1 -- 172.21.15.26:0/2238073560 shutdown_connections 2024-09-16T15:02:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.359+0000 7f8d4dbc3640 1 -- 172.21.15.26:0/2238073560 >> 172.21.15.26:0/2238073560 conn(0x7f8d4809fb50 msgr2=0x7f8d480a0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.359+0000 7f8d4dbc3640 1 -- 172.21.15.26:0/2238073560 shutdown_connections 2024-09-16T15:02:28.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.359+0000 7f8d4dbc3640 1 -- 172.21.15.26:0/2238073560 wait complete. 2024-09-16T15:02:28.377 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:02:28.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.483+0000 7fed9c55f640 10 monclient: get_monmap_and_config 2024-09-16T15:02:28.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.483+0000 7fed9c55f640 10 monclient: build_initial_monmap 2024-09-16T15:02:28.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.483+0000 7fed9c55f640 10 monclient: monmap: 2024-09-16T15:02:28.486 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:28.486 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.487 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:28.488504+0000 2024-09-16T15:02:28.487 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:28.488504+0000 2024-09-16T15:02:28.487 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:28.487 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.487 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:28.487 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:28.487 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:28.487 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.483+0000 7fed9c55f640 1 Processor -- start 2024-09-16T15:02:28.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.483+0000 7fed9c55f640 1 -- start start 2024-09-16T15:02:28.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.483+0000 7fed9c55f640 10 monclient: init 2024-09-16T15:02:28.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:28.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:02:28.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient(hunting): picked mon.noname-c con 0x7fed9409a550 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:28.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient(hunting): picked mon.noname-b con 0x7fed9409d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:28.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient(hunting): picked mon.noname-a con 0x7fed9409b040 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:28.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9409dc60 con 0x7fed9409b040 2024-09-16T15:02:28.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed94099ae0 con 0x7fed9409a550 2024-09-16T15:02:28.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9409b8f0 con 0x7fed9409d1b0 2024-09-16T15:02:28.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:28.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient(hunting): authenticate will time out at 2172.608455s 2024-09-16T15:02:28.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9aafc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fed9409b040 0x7fed9413a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38896/0 (socket says 172.21.15.26:38896) 2024-09-16T15:02:28.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed99afa640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fed9409d1b0 0x7fed94136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51602/0 (socket says 172.21.15.26:51602) 2024-09-16T15:02:28.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9aafc640 1 -- 172.21.15.26:0/2218362883 learned_addr learned my addr 172.21.15.26:0/2218362883 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:28.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 1 -- 172.21.15.26:0/2218362883 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646751836 0 0) 0x7fed9409b8f0 con 0x7fed9409d1b0 2024-09-16T15:02:28.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient(hunting): _negotiate 2024-09-16T15:02:28.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:28.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:28.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient(hunting): my global_id is 354365 2024-09-16T15:02:28.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 1 -- 172.21.15.26:0/2218362883 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed7c003530 con 0x7fed9409d1b0 2024-09-16T15:02:28.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 1 -- 172.21.15.26:0/2218362883 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589644714 0 0) 0x7fed9409dc60 con 0x7fed9409b040 2024-09-16T15:02:28.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient(hunting): _negotiate 2024-09-16T15:02:28.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient(hunting): my global_id is 334551 2024-09-16T15:02:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 1 -- 172.21.15.26:0/2218362883 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed9409b8f0 con 0x7fed9409b040 2024-09-16T15:02:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 1 -- 172.21.15.26:0/2218362883 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 326488614 0 0) 0x7fed7c003530 con 0x7fed9409d1b0 2024-09-16T15:02:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 1 -- 172.21.15.26:0/2218362883 >> v1:172.21.15.189:6789/0 conn(0x7fed9409a550 legacy=0x7fed9409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 1 -- 172.21.15.26:0/2218362883 >> v1:172.21.15.26:6789/0 conn(0x7fed9409b040 legacy=0x7fed9413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient: _finish_hunting 0 2024-09-16T15:02:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 1 monclient: found mon.noname-b 2024-09-16T15:02:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:02:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 1 -- 172.21.15.26:0/2218362883 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9409bbd0 con 0x7fed9409d1b0 2024-09-16T15:02:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient: _finish_auth 0 2024-09-16T15:02:28.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient: _check_auth_tickets 2024-09-16T15:02:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 1 -- 172.21.15.26:0/2218362883 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fed88001620 con 0x7fed9409d1b0 2024-09-16T15:02:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 5 monclient: authenticate success, global_id 354365 2024-09-16T15:02:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:02:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient: dump: 2024-09-16T15:02:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:28.498 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.499 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.499 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.499 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:28.499 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.499 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:28.499 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:28.499 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:28.499 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:28.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 1 -- 172.21.15.26:0/2218362883 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed880018c0 con 0x7fed9409d1b0 2024-09-16T15:02:28.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:28.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:28.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 4 set_mon_vals no callback set 2024-09-16T15:02:28.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 1 -- 172.21.15.26:0/2218362883 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fed88001c90 con 0x7fed9409d1b0 2024-09-16T15:02:28.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:28.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:28.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed992f9640 10 monclient: dump: 2024-09-16T15:02:28.500 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:28.500 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.500 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.500 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.500 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:28.500 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.500 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:28.500 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:28.500 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:28.500 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient: shutdown 2024-09-16T15:02:28.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:28.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 1 -- 172.21.15.26:0/2218362883 >> v1:172.21.15.26:6790/0 conn(0x7fed9409d1b0 legacy=0x7fed94136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 1 -- 172.21.15.26:0/2218362883 shutdown_connections 2024-09-16T15:02:28.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 1 -- 172.21.15.26:0/2218362883 >> 172.21.15.26:0/2218362883 conn(0x7fed9409fb50 msgr2=0x7fed9409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:28.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 1 -- 172.21.15.26:0/2218362883 shutdown_connections 2024-09-16T15:02:28.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 1 -- 172.21.15.26:0/2218362883 wait complete. 2024-09-16T15:02:28.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient: build_initial_monmap 2024-09-16T15:02:28.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient: monmap: 2024-09-16T15:02:28.501 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:28.501 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:28.501 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:28.501 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:28.502 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:28.502 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:28.502 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:28.502 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:28.502 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:28.502 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 1 Processor -- start 2024-09-16T15:02:28.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 1 -- start start 2024-09-16T15:02:28.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient: init 2024-09-16T15:02:28.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:28.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:02:28.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient(hunting): picked mon.noname-b con 0x7fed9409a550 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:28.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient(hunting): picked mon.noname-c con 0x7fed9409b040 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:28.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient(hunting): picked mon.noname-a con 0x7fed9409d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:28.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9409a240 con 0x7fed9409d1b0 2024-09-16T15:02:28.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9406fee0 con 0x7fed9409a550 2024-09-16T15:02:28.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed940a2220 con 0x7fed9409b040 2024-09-16T15:02:28.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:28.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed9c55f640 10 monclient(hunting): authenticate will time out at 2172.611408s 2024-09-16T15:02:28.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed99afa640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fed9409b040 0x7fed9406f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51614/0 (socket says 172.21.15.26:51614) 2024-09-16T15:02:28.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed99afa640 1 -- 172.21.15.26:0/3062167375 learned_addr learned my addr 172.21.15.26:0/3062167375 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:28.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839508046 0 0) 0x7fed9409a240 con 0x7fed9409d1b0 2024-09-16T15:02:28.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed92ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:28.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:28.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:28.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed92ffd640 10 monclient(hunting): my global_id is 334554 2024-09-16T15:02:28.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed80003530 con 0x7fed9409d1b0 2024-09-16T15:02:28.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215855331 0 0) 0x7fed940a2220 con 0x7fed9409b040 2024-09-16T15:02:28.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed92ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:02:28.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:28.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:28.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed92ffd640 10 monclient(hunting): my global_id is 354368 2024-09-16T15:02:28.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed9409a240 con 0x7fed9409b040 2024-09-16T15:02:28.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2943521707 0 0) 0x7fed80003530 con 0x7fed9409d1b0 2024-09-16T15:02:28.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed940a2220 con 0x7fed9409d1b0 2024-09-16T15:02:28.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fed8c001100 con 0x7fed9409d1b0 2024-09-16T15:02:28.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.487+0000 7fed92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:28.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:28.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 10 monclient(hunting): dump: 2024-09-16T15:02:28.505 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:28.505 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.505 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.505 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.505 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:28.505 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.505 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:28.505 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:28.505 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:28.505 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:28.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1098980564 0 0) 0x7fed9409a240 con 0x7fed9409b040 2024-09-16T15:02:28.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed80003530 con 0x7fed9409b040 2024-09-16T15:02:28.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fed88001c10 con 0x7fed9409b040 2024-09-16T15:02:28.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:28.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:28.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 10 monclient(hunting): dump: 2024-09-16T15:02:28.506 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:28.506 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.506 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.506 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.506 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:28.506 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.506 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:28.506 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:28.507 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1122208911 0 0) 0x7fed940a2220 con 0x7fed9409d1b0 2024-09-16T15:02:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 >> v1:172.21.15.26:6790/0 conn(0x7fed9409b040 legacy=0x7fed9406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 >> v1:172.21.15.189:6789/0 conn(0x7fed9409a550 legacy=0x7fed9406f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 10 monclient: _finish_hunting 0 2024-09-16T15:02:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 1 monclient: found mon.a 2024-09-16T15:02:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9411ed60 con 0x7fed9409d1b0 2024-09-16T15:02:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 10 monclient: _finish_auth 0 2024-09-16T15:02:28.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 10 monclient: _check_auth_tickets 2024-09-16T15:02:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed9c55f640 5 monclient: authenticate success, global_id 334554 2024-09-16T15:02:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed9c55f640 10 monclient: _renew_subs 2024-09-16T15:02:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed9c55f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed9c55f640 1 -- 172.21.15.26:0/3062167375 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed940716b0 con 0x7fed9409d1b0 2024-09-16T15:02:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed9c55f640 10 monclient: _renew_subs 2024-09-16T15:02:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed9c55f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed9c55f640 1 -- 172.21.15.26:0/3062167375 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed94134600 con 0x7fed9409d1b0 2024-09-16T15:02:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed8c0047f0 con 0x7fed9409d1b0 2024-09-16T15:02:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7fed8c004b00 con 0x7fed9409d1b0 2024-09-16T15:02:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:28.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed98af8640 4 set_mon_vals no callback set 2024-09-16T15:02:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 10 monclient: dump: 2024-09-16T15:02:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:28.509 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed9c55f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:28.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed9c55f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed9c55f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed9c55f640 1 -- 172.21.15.26:0/3062167375 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed74005050 con 0x7fed9409d1b0 2024-09-16T15:02:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7fed8c00f570 con 0x7fed9409d1b0 2024-09-16T15:02:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7fed8c043c50 con 0x7fed9409d1b0 2024-09-16T15:02:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.491+0000 7fed9a2fb640 10 monclient: get_auth_request con 0x7fed800362c0 auth_method 0 2024-09-16T15:02:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.495+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fed8c014410 con 0x7fed9409d1b0 2024-09-16T15:02:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.495+0000 7fed92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:28.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.495+0000 7fed92ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:28.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.647+0000 7fed9c55f640 1 -- 172.21.15.26:0/3062167375 --> v1:172.21.15.189:6812/3196161582 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fed74000ba0 con 0x7fed800362c0 2024-09-16T15:02:28.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.655+0000 7fed92ffd640 1 -- 172.21.15.26:0/3062167375 <== mgr.4105 v1:172.21.15.189:6812/3196161582 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+97880 (unknown 2965378022 0 1373311433) 0x7fed74000ba0 con 0x7fed800362c0 2024-09-16T15:02:28.658 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T15:02:28.658 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T15:02:28.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.655+0000 7fed9c55f640 1 -- 172.21.15.26:0/3062167375 >> v1:172.21.15.189:6812/3196161582 conn(0x7fed800362c0 legacy=0x7fed80038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.655+0000 7fed9c55f640 10 monclient: shutdown 2024-09-16T15:02:28.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.655+0000 7fed9c55f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:28.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.655+0000 7fed9c55f640 1 -- 172.21.15.26:0/3062167375 >> v1:172.21.15.26:6789/0 conn(0x7fed9409d1b0 legacy=0x7fed941d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:28.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.659+0000 7fed9c55f640 1 -- 172.21.15.26:0/3062167375 shutdown_connections 2024-09-16T15:02:28.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.659+0000 7fed9c55f640 1 -- 172.21.15.26:0/3062167375 >> 172.21.15.26:0/3062167375 conn(0x7fed9409fb50 msgr2=0x7fed940a2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:28.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.659+0000 7fed9c55f640 1 -- 172.21.15.26:0/3062167375 shutdown_connections 2024-09-16T15:02:28.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:28.659+0000 7fed9c55f640 1 -- 172.21.15.26:0/3062167375 wait complete. 2024-09-16T15:02:28.674 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":843,"stamp":"2024-09-16T15:02:26.726443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1057913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":199188,"kb_used_data":8396,"kb_used_omap":9,"kb_used_meta":190710,"kb_avail":566031852,"statfs":{"total":579820584960,"available":579616616448,"internally_reserved":0,"allocated":8597504,"data_stored":8794066,"data_compressed":1510597,"data_compressed_allocated":2170880,"data_compressed_original":4796416,"omap_allocated":9559,"internal_metadata":195287721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003451"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":164,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:02.474374+0000","last_change":"2024-09-16T14:48:52.143014+0000","last_active":"2024-09-16T15:00:02.474374+0000","last_peered":"2024-09-16T15:00:02.474374+0000","last_clean":"2024-09-16T15:00:02.474374+0000","last_became_active":"2024-09-16T14:48:52.141236+0000","last_became_peered":"2024-09-16T14:48:52.141236+0000","last_unstale":"2024-09-16T15:00:02.474374+0000","last_undegraded":"2024-09-16T15:00:02.474374+0000","last_fullsized":"2024-09-16T15:00:02.474374+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:18:35.458388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":89,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:42.604638+0000","last_change":"2024-09-16T14:48:52.126720+0000","last_active":"2024-09-16T14:59:42.604638+0000","last_peered":"2024-09-16T14:59:42.604638+0000","last_clean":"2024-09-16T14:59:42.604638+0000","last_became_active":"2024-09-16T14:48:52.126325+0000","last_became_peered":"2024-09-16T14:48:52.126325+0000","last_unstale":"2024-09-16T14:59:42.604638+0000","last_undegraded":"2024-09-16T14:59:42.604638+0000","last_fullsized":"2024-09-16T14:59:42.604638+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:23.845897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":156,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:07.474741+0000","last_change":"2024-09-16T14:48:52.138778+0000","last_active":"2024-09-16T15:00:07.474741+0000","last_peered":"2024-09-16T15:00:07.474741+0000","last_clean":"2024-09-16T15:00:07.474741+0000","last_became_active":"2024-09-16T14:48:52.137680+0000","last_became_peered":"2024-09-16T14:48:52.137680+0000","last_unstale":"2024-09-16T15:00:07.474741+0000","last_undegraded":"2024-09-16T15:00:07.474741+0000","last_fullsized":"2024-09-16T15:00:07.474741+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:57.689750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":89,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:42.604588+0000","last_change":"2024-09-16T14:48:52.133563+0000","last_active":"2024-09-16T14:59:42.604588+0000","last_peered":"2024-09-16T14:59:42.604588+0000","last_clean":"2024-09-16T14:59:42.604588+0000","last_became_active":"2024-09-16T14:48:52.133190+0000","last_became_peered":"2024-09-16T14:48:52.133190+0000","last_unstale":"2024-09-16T14:59:42.604588+0000","last_undegraded":"2024-09-16T14:59:42.604588+0000","last_fullsized":"2024-09-16T14:59:42.604588+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:53:36.816994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":124,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:58:22.592610+0000","last_change":"2024-09-16T14:48:53.142970+0000","last_active":"2024-09-16T14:58:22.592610+0000","last_peered":"2024-09-16T14:58:22.592610+0000","last_clean":"2024-09-16T14:58:22.592610+0000","last_became_active":"2024-09-16T14:48:53.142066+0000","last_became_peered":"2024-09-16T14:48:53.142066+0000","last_unstale":"2024-09-16T14:58:22.592610+0000","last_undegraded":"2024-09-16T14:58:22.592610+0000","last_fullsized":"2024-09-16T14:58:22.592610+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:29:40.181990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":143,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:07.475174+0000","last_change":"2024-09-16T14:48:52.128011+0000","last_active":"2024-09-16T15:00:07.475174+0000","last_peered":"2024-09-16T15:00:07.475174+0000","last_clean":"2024-09-16T15:00:07.475174+0000","last_became_active":"2024-09-16T14:48:52.127250+0000","last_became_peered":"2024-09-16T14:48:52.127250+0000","last_unstale":"2024-09-16T15:00:07.475174+0000","last_undegraded":"2024-09-16T15:00:07.475174+0000","last_fullsized":"2024-09-16T15:00:07.475174+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:03:30.725414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":91,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:42.474316+0000","last_change":"2024-09-16T14:48:52.133060+0000","last_active":"2024-09-16T14:59:42.474316+0000","last_peered":"2024-09-16T14:59:42.474316+0000","last_clean":"2024-09-16T14:59:42.474316+0000","last_became_active":"2024-09-16T14:48:52.132725+0000","last_became_peered":"2024-09-16T14:48:52.132725+0000","last_unstale":"2024-09-16T14:59:42.474316+0000","last_undegraded":"2024-09-16T14:59:42.474316+0000","last_fullsized":"2024-09-16T14:59:42.474316+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:55:32.475641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":81,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468949+0000","last_change":"2024-09-16T14:48:52.129972+0000","last_active":"2024-09-16T14:59:17.468949+0000","last_peered":"2024-09-16T14:59:17.468949+0000","last_clean":"2024-09-16T14:59:17.468949+0000","last_became_active":"2024-09-16T14:48:52.128799+0000","last_became_peered":"2024-09-16T14:48:52.128799+0000","last_unstale":"2024-09-16T14:59:17.468949+0000","last_undegraded":"2024-09-16T14:59:17.468949+0000","last_fullsized":"2024-09-16T14:59:17.468949+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:48:37.984512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":111,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:02.607606+0000","last_change":"2024-09-16T14:48:52.143178+0000","last_active":"2024-09-16T15:00:02.607606+0000","last_peered":"2024-09-16T15:00:02.607606+0000","last_clean":"2024-09-16T15:00:02.607606+0000","last_became_active":"2024-09-16T14:48:52.142697+0000","last_became_peered":"2024-09-16T14:48:52.142697+0000","last_unstale":"2024-09-16T15:00:02.607606+0000","last_undegraded":"2024-09-16T15:00:02.607606+0000","last_fullsized":"2024-09-16T15:00:02.607606+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:43:37.007895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":112,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:12.478953+0000","last_change":"2024-09-16T14:48:52.144132+0000","last_active":"2024-09-16T15:00:12.478953+0000","last_peered":"2024-09-16T15:00:12.478953+0000","last_clean":"2024-09-16T15:00:12.478953+0000","last_became_active":"2024-09-16T14:48:52.143756+0000","last_became_peered":"2024-09-16T14:48:52.143756+0000","last_unstale":"2024-09-16T15:00:12.478953+0000","last_undegraded":"2024-09-16T15:00:12.478953+0000","last_fullsized":"2024-09-16T15:00:12.478953+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:15.504836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":126,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:02.600738+0000","last_change":"2024-09-16T14:48:52.139682+0000","last_active":"2024-09-16T15:00:02.600738+0000","last_peered":"2024-09-16T15:00:02.600738+0000","last_clean":"2024-09-16T15:00:02.600738+0000","last_became_active":"2024-09-16T14:48:52.139205+0000","last_became_peered":"2024-09-16T14:48:52.139205+0000","last_unstale":"2024-09-16T15:00:02.600738+0000","last_undegraded":"2024-09-16T15:00:02.600738+0000","last_fullsized":"2024-09-16T15:00:02.600738+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:56.518201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":81,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468831+0000","last_change":"2024-09-16T14:48:52.131936+0000","last_active":"2024-09-16T14:59:17.468831+0000","last_peered":"2024-09-16T14:59:17.468831+0000","last_clean":"2024-09-16T14:59:17.468831+0000","last_became_active":"2024-09-16T14:48:52.131089+0000","last_became_peered":"2024-09-16T14:48:52.131089+0000","last_unstale":"2024-09-16T14:59:17.468831+0000","last_undegraded":"2024-09-16T14:59:17.468831+0000","last_fullsized":"2024-09-16T14:59:17.468831+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:04:08.492279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":82,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468863+0000","last_change":"2024-09-16T14:48:52.138889+0000","last_active":"2024-09-16T14:59:17.468863+0000","last_peered":"2024-09-16T14:59:17.468863+0000","last_clean":"2024-09-16T14:59:17.468863+0000","last_became_active":"2024-09-16T14:48:52.137830+0000","last_became_peered":"2024-09-16T14:48:52.137830+0000","last_unstale":"2024-09-16T14:59:17.468863+0000","last_undegraded":"2024-09-16T14:59:17.468863+0000","last_fullsized":"2024-09-16T14:59:17.468863+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:00:48.737166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":169,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:00:02.474335+0000","last_change":"2024-09-16T14:48:52.145132+0000","last_active":"2024-09-16T15:00:02.474335+0000","last_peered":"2024-09-16T15:00:02.474335+0000","last_clean":"2024-09-16T15:00:02.474335+0000","last_became_active":"2024-09-16T14:48:52.144280+0000","last_became_peered":"2024-09-16T14:48:52.144280+0000","last_unstale":"2024-09-16T15:00:02.474335+0000","last_undegraded":"2024-09-16T15:00:02.474335+0000","last_fullsized":"2024-09-16T15:00:02.474335+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:37:18.905007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":90,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.467831+0000","last_change":"2024-09-16T14:48:53.564947+0000","last_active":"2024-09-16T14:59:17.467831+0000","last_peered":"2024-09-16T14:59:17.467831+0000","last_clean":"2024-09-16T14:59:17.467831+0000","last_became_active":"2024-09-16T14:48:53.563655+0000","last_became_peered":"2024-09-16T14:48:53.563655+0000","last_unstale":"2024-09-16T14:59:17.467831+0000","last_undegraded":"2024-09-16T14:59:17.467831+0000","last_fullsized":"2024-09-16T14:59:17.467831+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:32:48.022677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":81,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:58:57.465826+0000","last_change":"2024-09-16T14:48:52.138403+0000","last_active":"2024-09-16T14:58:57.465826+0000","last_peered":"2024-09-16T14:58:57.465826+0000","last_clean":"2024-09-16T14:58:57.465826+0000","last_became_active":"2024-09-16T14:48:52.137512+0000","last_became_peered":"2024-09-16T14:48:52.137512+0000","last_unstale":"2024-09-16T14:58:57.465826+0000","last_undegraded":"2024-09-16T14:58:57.465826+0000","last_fullsized":"2024-09-16T14:58:57.465826+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:30.193610+0000","last_clean_scrub_stamp":"2024-09-16T14:47:30.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:00:15.913134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":229,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:58:57.465176+0000","last_change":"2024-09-16T14:48:52.131578+0000","last_active":"2024-09-16T14:58:57.465176+0000","last_peered":"2024-09-16T14:58:57.465176+0000","last_clean":"2024-09-16T14:58:57.465176+0000","last_became_active":"2024-09-16T14:48:52.131259+0000","last_became_peered":"2024-09-16T14:48:52.131259+0000","last_unstale":"2024-09-16T14:58:57.465176+0000","last_undegraded":"2024-09-16T14:58:57.465176+0000","last_fullsized":"2024-09-16T14:58:57.465176+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:02:29.950099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067836399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":220,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.600970+0000","last_change":"2024-09-16T14:48:53.141414+0000","last_active":"2024-09-16T14:59:17.600970+0000","last_peered":"2024-09-16T14:59:17.600970+0000","last_clean":"2024-09-16T14:59:17.600970+0000","last_became_active":"2024-09-16T14:48:53.140914+0000","last_became_peered":"2024-09-16T14:48:53.140914+0000","last_unstale":"2024-09-16T14:59:17.600970+0000","last_undegraded":"2024-09-16T14:59:17.600970+0000","last_fullsized":"2024-09-16T14:59:17.600970+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:53:48.881111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097689999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.594663+0000","last_change":"2024-09-16T14:48:52.149868+0000","last_active":"2024-09-16T14:59:17.594663+0000","last_peered":"2024-09-16T14:59:17.594663+0000","last_clean":"2024-09-16T14:59:17.594663+0000","last_became_active":"2024-09-16T14:48:52.142312+0000","last_became_peered":"2024-09-16T14:48:52.142312+0000","last_unstale":"2024-09-16T14:59:17.594663+0000","last_undegraded":"2024-09-16T14:59:17.594663+0000","last_fullsized":"2024-09-16T14:59:17.594663+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:31:20.443534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":87,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468897+0000","last_change":"2024-09-16T14:48:52.151300+0000","last_active":"2024-09-16T14:59:17.468897+0000","last_peered":"2024-09-16T14:59:17.468897+0000","last_clean":"2024-09-16T14:59:17.468897+0000","last_became_active":"2024-09-16T14:48:52.137048+0000","last_became_peered":"2024-09-16T14:48:52.137048+0000","last_unstale":"2024-09-16T14:59:17.468897+0000","last_undegraded":"2024-09-16T14:59:17.468897+0000","last_fullsized":"2024-09-16T14:59:17.468897+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:17:13.952991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042440799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":428,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:58:37.462283+0000","last_change":"2024-09-16T14:48:55.155094+0000","last_active":"2024-09-16T14:58:37.462283+0000","last_peered":"2024-09-16T14:58:37.462283+0000","last_clean":"2024-09-16T14:58:37.462283+0000","last_became_active":"2024-09-16T14:48:55.154511+0000","last_became_peered":"2024-09-16T14:48:55.154511+0000","last_unstale":"2024-09-16T14:58:37.462283+0000","last_undegraded":"2024-09-16T14:58:37.462283+0000","last_fullsized":"2024-09-16T14:58:37.462283+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:28.462531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:28.462531+0000","last_clean_scrub_stamp":"2024-09-16T14:42:28.462531+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:06.351384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":81,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.468924+0000","last_change":"2024-09-16T14:48:52.129414+0000","last_active":"2024-09-16T14:59:17.468924+0000","last_peered":"2024-09-16T14:59:17.468924+0000","last_clean":"2024-09-16T14:59:17.468924+0000","last_became_active":"2024-09-16T14:48:52.128554+0000","last_became_peered":"2024-09-16T14:48:52.128554+0000","last_unstale":"2024-09-16T14:59:17.468924+0000","last_undegraded":"2024-09-16T14:59:17.468924+0000","last_fullsized":"2024-09-16T14:59:17.468924+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:38:09.329918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039796999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":89,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:17.594692+0000","last_change":"2024-09-16T14:48:52.144047+0000","last_active":"2024-09-16T14:59:17.594692+0000","last_peered":"2024-09-16T14:59:17.594692+0000","last_clean":"2024-09-16T14:59:17.594692+0000","last_became_active":"2024-09-16T14:48:52.143705+0000","last_became_peered":"2024-09-16T14:48:52.143705+0000","last_unstale":"2024-09-16T14:59:17.594692+0000","last_undegraded":"2024-09-16T14:59:17.594692+0000","last_fullsized":"2024-09-16T14:59:17.594692+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:14:59.892564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061478100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":90,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:42.474287+0000","last_change":"2024-09-16T14:48:52.131363+0000","last_active":"2024-09-16T14:59:42.474287+0000","last_peered":"2024-09-16T14:59:42.474287+0000","last_clean":"2024-09-16T14:59:42.474287+0000","last_became_active":"2024-09-16T14:48:52.131160+0000","last_became_peered":"2024-09-16T14:48:52.131160+0000","last_unstale":"2024-09-16T14:59:42.474287+0000","last_undegraded":"2024-09-16T14:59:42.474287+0000","last_fullsized":"2024-09-16T14:59:42.474287+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:12.859518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065779100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":226,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T14:59:52.606162+0000","last_change":"2024-09-16T14:48:52.133533+0000","last_active":"2024-09-16T14:59:52.606162+0000","last_peered":"2024-09-16T14:59:52.606162+0000","last_clean":"2024-09-16T14:59:52.606162+0000","last_became_active":"2024-09-16T14:48:52.133197+0000","last_became_peered":"2024-09-16T14:48:52.133197+0000","last_unstale":"2024-09-16T14:59:52.606162+0000","last_undegraded":"2024-09-16T14:59:52.606162+0000","last_fullsized":"2024-09-16T14:59:52.606162+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:37:19.055419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085482099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":94400,"data_compressed_allocated":122880,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640499,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1413120,"data_stored":1289362,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.622,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.92600000000000005,"5min":1.042,"15min":1.2},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.81499999999999995,"15min":0.79100000000000004},"min":{"1min":0.627,"5min":0.51500000000000001,"15min":0.47899999999999998},"max":{"1min":0.95599999999999996,"5min":1.0369999999999999,"15min":1.3300000000000001},"last":0.95599999999999996}]},{"osd":1,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.81799999999999995,"15min":0.81000000000000005},"min":{"1min":0.61899999999999999,"5min":0.44900000000000001,"15min":0.33900000000000002},"max":{"1min":1.0589999999999999,"5min":1.073,"15min":1.2010000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.74399999999999999,"15min":0.752},"min":{"1min":0.60999999999999999,"5min":0.54000000000000004,"15min":0.438},"max":{"1min":0.90800000000000003,"5min":0.94299999999999995,"15min":1.089},"last":0.77300000000000002}]},{"osd":2,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.81799999999999995,"15min":0.82899999999999996},"min":{"1min":0.628,"5min":0.50800000000000001,"15min":0.44900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.272},"last":0.75},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.90600000000000003,"15min":0.91300000000000003},"min":{"1min":0.81000000000000005,"5min":0.52900000000000003,"15min":0.42899999999999999},"max":{"1min":1.0289999999999999,"5min":1.056,"15min":1.3540000000000001},"last":0.93000000000000005}]},{"osd":3,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69699999999999995,"15min":0.67600000000000005},"min":{"1min":0.52300000000000002,"5min":0.46600000000000003,"15min":0.26000000000000001},"max":{"1min":0.96799999999999997,"5min":0.96999999999999997,"15min":1.1020000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.68200000000000005,"15min":0.69099999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33500000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":1.153},"last":0.90000000000000002}]},{"osd":4,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72899999999999998,"15min":0.749},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.27500000000000002},"max":{"1min":0.95299999999999996,"5min":1.05,"15min":1.236},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.80000000000000004,"15min":0.77200000000000002},"min":{"1min":0.48799999999999999,"5min":0.44800000000000001,"15min":0.26300000000000001},"max":{"1min":1.038,"5min":1.0529999999999999,"15min":1.2949999999999999},"last":1.01}]}]},{"osd":4,"up_from":11,"seq":47244640499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1417216,"data_stored":1289381,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78200000000000003,"15min":0.77300000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.437},"max":{"1min":0.96699999999999997,"5min":1.0800000000000001,"15min":1.1579999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73899999999999999,"15min":0.79000000000000004},"min":{"1min":0.52600000000000002,"5min":0.48899999999999999,"15min":0.48499999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.1679999999999999},"last":0.73999999999999999}]},{"osd":1,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.85899999999999999,"15min":0.81599999999999995},"min":{"1min":0.71199999999999997,"5min":0.52800000000000002,"15min":0.42399999999999999},"max":{"1min":1.0429999999999999,"5min":1.0780000000000001,"15min":1.4630000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.79600000000000004,"15min":0.83399999999999996},"min":{"1min":0.65700000000000003,"5min":0.53700000000000003,"15min":0.50800000000000001},"max":{"1min":1.0149999999999999,"5min":1.024,"15min":1.2729999999999999},"last":0.92300000000000004}]},{"osd":2,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.84999999999999998,"15min":0.84899999999999998},"min":{"1min":0.68000000000000005,"5min":0.65500000000000003,"15min":0.47599999999999998},"max":{"1min":1.02,"5min":1.0449999999999999,"15min":1.536},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84799999999999998,"15min":0.85899999999999999},"min":{"1min":0.63500000000000001,"5min":0.56599999999999995,"15min":0.51200000000000001},"max":{"1min":0.97099999999999997,"5min":1.0720000000000001,"15min":1.29},"last":0.79300000000000004}]},{"osd":3,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.70999999999999996,"15min":0.68899999999999995},"min":{"1min":0.53700000000000003,"5min":0.51300000000000001,"15min":0.27700000000000002},"max":{"1min":0.83299999999999996,"5min":1.0720000000000001,"15min":1.2669999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72899999999999998,"15min":0.72499999999999998},"min":{"1min":0.55700000000000005,"5min":0.50600000000000001,"15min":0.25700000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.198},"last":0.69799999999999995}]},{"osd":5,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.75600000000000001,"15min":0.73099999999999998},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.28999999999999998},"max":{"1min":0.95799999999999996,"5min":1.0600000000000001,"15min":1.1140000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.94699999999999995,"15min":0.91700000000000004},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.24299999999999999},"max":{"1min":1.0700000000000001,"5min":1.169,"15min":1.3879999999999999},"last":0.96599999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640498,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32556,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339284,"statfs":{"total":96636764160,"available":96603426816,"internally_reserved":0,"allocated":1474560,"data_stored":1818290,"data_compressed":271867,"data_compressed_allocated":389120,"data_compressed_original":1138688,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.45200000000000001},"max":{"1min":0.998,"5min":0.998,"15min":1.2729999999999999},"last":0.622},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74099999999999999,"15min":0.753},"min":{"1min":0.60799999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90500000000000003,"5min":0.94199999999999995,"15min":1.306},"last":0.83499999999999996}]},{"osd":1,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90200000000000002,"15min":0.86599999999999999},"min":{"1min":0.55800000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.069,"5min":1.1259999999999999,"15min":1.294},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.86199999999999999,"15min":0.86599999999999999},"min":{"1min":0.67900000000000005,"5min":0.56200000000000006,"15min":0.41199999999999998},"max":{"1min":0.93999999999999995,"5min":1.046,"15min":1.9219999999999999},"last":0.72899999999999998}]},{"osd":2,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81799999999999995,"15min":0.81699999999999995},"min":{"1min":0.71299999999999997,"5min":0.61199999999999999,"15min":0.45900000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.181},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79900000000000004,"15min":0.80600000000000005},"min":{"1min":0.627,"5min":0.627,"15min":0.38800000000000001},"max":{"1min":0.90300000000000002,"5min":0.96899999999999997,"15min":2.444},"last":0.78000000000000003}]},{"osd":4,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.624,"15min":0.623},"min":{"1min":0.49299999999999999,"5min":0.47699999999999998,"15min":0.28199999999999997},"max":{"1min":0.89800000000000002,"5min":0.98799999999999999,"15min":1.0700000000000001},"last":0.75},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.67900000000000005,"5min":0.628,"15min":0.317},"max":{"1min":0.99299999999999999,"5min":1.0740000000000001,"15min":1.2290000000000001},"last":0.92000000000000004}]},{"osd":5,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69299999999999995,"15min":0.70699999999999996},"min":{"1min":0.55500000000000005,"5min":0.52400000000000002,"15min":0.29899999999999999},"max":{"1min":0.92200000000000004,"5min":1.4279999999999999,"15min":2.5409999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.83499999999999996,"15min":0.80900000000000005},"min":{"1min":0.63700000000000001,"5min":0.60999999999999999,"15min":0.33100000000000002},"max":{"1min":1.081,"5min":1.514,"15min":1.514},"last":0.871}]}]},{"osd":2,"up_from":11,"seq":47244640498,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1413120,"data_stored":1289362,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56200000000000006,"15min":0.58099999999999996},"min":{"1min":0.252,"5min":0.252,"15min":0.249},"max":{"1min":0.749,"5min":0.94699999999999995,"15min":1.819},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.77600000000000002,"15min":0.75700000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.312},"max":{"1min":1.147,"5min":1.147,"15min":1.444},"last":0.91900000000000004}]},{"osd":1,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.39200000000000002,"5min":0.379,"15min":0.28299999999999997},"max":{"1min":0.95499999999999996,"5min":0.95999999999999996,"15min":1.27},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.62,"15min":0.623},"min":{"1min":0.313,"5min":0.313,"15min":0.26700000000000002},"max":{"1min":1.0129999999999999,"5min":1.0189999999999999,"15min":1.383},"last":0.68000000000000005}]},{"osd":3,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.84199999999999997,"15min":0.82299999999999995},"min":{"1min":0.58599999999999997,"5min":0.56000000000000005,"15min":0.36899999999999999},"max":{"1min":1.0720000000000001,"5min":1.169,"15min":1.2110000000000001},"last":1.034},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.871,"15min":0.86299999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.44700000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.335},"last":0.72399999999999998}]},{"osd":4,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.83599999999999997,"15min":0.83499999999999996},"min":{"1min":0.53300000000000003,"5min":0.50600000000000001,"15min":0.45500000000000002},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.874},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.84999999999999998,"15min":0.85499999999999998},"min":{"1min":0.62,"5min":0.58599999999999997,"15min":0.40999999999999998},"max":{"1min":0.97099999999999997,"5min":1.0489999999999999,"15min":1.367},"last":0.84299999999999997}]},{"osd":5,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.88200000000000001,"15min":0.89100000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.53700000000000003},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.911},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.95399999999999996,"15min":0.95499999999999996},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.121,"5min":1.1990000000000001,"15min":2.0499999999999998},"last":0.90800000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640497,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1462272,"data_stored":1818290,"data_compressed":273302,"data_compressed_allocated":389120,"data_compressed_original":1150976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56000000000000005,"15min":0.57099999999999995},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.251},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":1.0720000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.628,"15min":0.63800000000000001},"min":{"1min":0.44500000000000001,"5min":0.36199999999999999,"15min":0.26100000000000001},"max":{"1min":0.87,"5min":0.87,"15min":1.0589999999999999},"last":0.81499999999999995}]},{"osd":2,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.66700000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.28699999999999998},"max":{"1min":0.82599999999999996,"5min":0.88200000000000001,"15min":1.052},"last":0.748},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.628,"15min":0.66700000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32100000000000001},"max":{"1min":0.81899999999999995,"5min":0.879,"15min":1.403},"last":0.83799999999999997}]},{"osd":3,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85699999999999998,"15min":0.88800000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.503},"max":{"1min":1.052,"5min":1.117,"15min":1.359},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.86299999999999999,"15min":0.877},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1459999999999999,"5min":1.147,"15min":1.284},"last":1.1719999999999999}]},{"osd":4,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.88900000000000001,"15min":0.88200000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.46600000000000003},"max":{"1min":1.0900000000000001,"5min":1.097,"15min":1.341},"last":1.111},{"interface":"front","average":{"1min":0.88,"5min":0.86199999999999999,"15min":0.86899999999999999},"min":{"1min":0.67600000000000005,"5min":0.66200000000000003,"15min":0.46999999999999997},"max":{"1min":1.056,"5min":1.0800000000000001,"15min":1.399},"last":1.0269999999999999}]},{"osd":5,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.622,"5min":0.622,"15min":0.50900000000000001},"max":{"1min":1,"5min":1.01,"15min":1.258},"last":1.054},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90100000000000002,"15min":0.90400000000000003},"min":{"1min":0.69499999999999995,"5min":0.629,"15min":0.47299999999999998},"max":{"1min":1.1519999999999999,"5min":1.411,"15min":1.411},"last":1.087}]}]},{"osd":0,"up_from":11,"seq":47244640498,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36596,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335244,"statfs":{"total":96636764160,"available":96599289856,"internally_reserved":0,"allocated":1417216,"data_stored":1289381,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57499999999999996,"15min":0.57999999999999996},"min":{"1min":0.36099999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.74299999999999999,"5min":0.77700000000000002,"15min":1.1739999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.61399999999999999,"15min":0.63800000000000001},"min":{"1min":0.38200000000000001,"5min":0.28299999999999997,"15min":0.27400000000000002},"max":{"1min":0.67800000000000005,"5min":1.032,"15min":1.21},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68700000000000006,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.29899999999999999,"15min":0.25900000000000001},"max":{"1min":0.85799999999999998,"5min":0.997,"15min":1.103},"last":0.495},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.62,"15min":0.67200000000000004},"min":{"1min":0.47999999999999998,"5min":0.36099999999999999,"15min":0.26900000000000002},"max":{"1min":0.69099999999999995,"5min":1.008,"15min":1.137},"last":0.44}]},{"osd":3,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.82599999999999996},"min":{"1min":0.67100000000000004,"5min":0.59599999999999997,"15min":0.52400000000000002},"max":{"1min":1.014,"5min":1.1040000000000001,"15min":1.4650000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.81799999999999995,"15min":0.83999999999999997},"min":{"1min":0.69299999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.99299999999999999,"5min":1.101,"15min":1.4950000000000001},"last":0.59199999999999997}]},{"osd":4,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.84599999999999997,"15min":0.84699999999999998},"min":{"1min":0.68899999999999995,"5min":0.57999999999999996,"15min":0.48999999999999999},"max":{"1min":0.98499999999999999,"5min":1.1559999999999999,"15min":1.3300000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.85099999999999998},"min":{"1min":0.63700000000000001,"5min":0.55700000000000005,"15min":0.51600000000000001},"max":{"1min":0.96099999999999997,"5min":1.095,"15min":1.4179999999999999},"last":0.46999999999999997}]},{"osd":5,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.91200000000000003,"15min":0.91400000000000003},"min":{"1min":0.70899999999999996,"5min":0.66400000000000003,"15min":0.57999999999999996},"max":{"1min":1.1080000000000001,"5min":1.304,"15min":1.6499999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95799999999999996,"15min":0.94799999999999995},"min":{"1min":0.76900000000000002,"5min":0.69999999999999996,"15min":0.51900000000000002},"max":{"1min":1.0589999999999999,"5min":1.2030000000000001,"15min":1.7290000000000001},"last":0.754}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:02:28.680 INFO:tasks.ceph:pgid 10.4 last_scrub_stamp 2024-09-16T14:47:31.170011+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.680 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2024-09-16T14:47:30.193610+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.680 INFO:tasks.ceph:pgid 10.5 last_scrub_stamp 2024-09-16T14:47:31.170011+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.680 INFO:tasks.ceph:pgid 9.6 last_scrub_stamp 2024-09-16T14:47:30.193610+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.680 INFO:tasks.ceph:pgid 10.6 last_scrub_stamp 2024-09-16T14:47:31.170011+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.680 INFO:tasks.ceph:pgid 9.5 last_scrub_stamp 2024-09-16T14:47:30.193610+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.680 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2024-09-16T14:47:31.170011+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.681 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2024-09-16T14:47:30.193610+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.681 INFO:tasks.ceph:pgid 10.0 last_scrub_stamp 2024-09-16T14:47:31.170011+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.681 INFO:tasks.ceph:pgid 9.3 last_scrub_stamp 2024-09-16T14:47:30.193610+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.681 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2024-09-16T14:47:31.170011+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.681 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2024-09-16T14:47:30.193610+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.681 INFO:tasks.ceph:pgid 10.2 last_scrub_stamp 2024-09-16T14:47:31.170011+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.681 INFO:tasks.ceph:pgid 9.1 last_scrub_stamp 2024-09-16T14:47:30.193610+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.681 INFO:tasks.ceph:pgid 10.3 last_scrub_stamp 2024-09-16T14:47:31.170011+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.681 INFO:tasks.ceph:pgid 9.0 last_scrub_stamp 2024-09-16T14:47:30.193610+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.682 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T14:42:27.455102+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.682 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-16T14:42:27.455102+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.682 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-16T14:42:27.455102+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.682 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-16T14:42:27.455102+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.682 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T14:42:28.462531+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=42, tm_sec=28, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.682 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-16T14:42:27.455102+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.682 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-16T14:42:27.455102+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.683 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-16T14:42:27.455102+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.683 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-16T14:42:27.455102+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:28.683 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T15:02:48.684 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:02:48.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.815+0000 7f980deda640 10 monclient: get_monmap_and_config 2024-09-16T15:02:48.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.815+0000 7f980deda640 10 monclient: build_initial_monmap 2024-09-16T15:02:48.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.815+0000 7f980deda640 10 monclient: monmap: 2024-09-16T15:02:48.818 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:48.819 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:48.819 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:02:48.820394+0000 2024-09-16T15:02:48.819 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:02:48.820394+0000 2024-09-16T15:02:48.819 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:48.819 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:48.819 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:48.819 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:02:48.819 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:02:48.819 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:48.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.815+0000 7f980deda640 1 Processor -- start 2024-09-16T15:02:48.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.815+0000 7f980deda640 1 -- start start 2024-09-16T15:02:48.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.815+0000 7f980deda640 10 monclient: init 2024-09-16T15:02:48.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:48.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:02:48.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient(hunting): picked mon.noname-c con 0x7f980809cb80 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:48.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient(hunting): picked mon.noname-b con 0x7f9808132650 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:48.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient(hunting): picked mon.noname-a con 0x7f9808138ff0 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:48.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980809d260 con 0x7f9808138ff0 2024-09-16T15:02:48.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9808132a80 con 0x7f980809cb80 2024-09-16T15:02:48.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980809cf60 con 0x7f9808132650 2024-09-16T15:02:48.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:48.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient(hunting): authenticate will time out at 2192.940462s 2024-09-16T15:02:48.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f9807fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9808138ff0 0x7f980813b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35322/0 (socket says 172.21.15.26:35322) 2024-09-16T15:02:48.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f9807fff640 1 -- 172.21.15.26:0/2158169170 learned_addr learned my addr 172.21.15.26:0/2158169170 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:48.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f9806ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9808132650 0x7f98081378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:41114/0 (socket says 172.21.15.26:41114) 2024-09-16T15:02:48.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 1 -- 172.21.15.26:0/2158169170 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854994430 0 0) 0x7f980809cf60 con 0x7f9808132650 2024-09-16T15:02:48.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient(hunting): _negotiate 2024-09-16T15:02:48.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:48.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:48.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient(hunting): my global_id is 354371 2024-09-16T15:02:48.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 1 -- 172.21.15.26:0/2158169170 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97f0003530 con 0x7f9808132650 2024-09-16T15:02:48.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 1 -- 172.21.15.26:0/2158169170 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290516470 0 0) 0x7f980809d260 con 0x7f9808138ff0 2024-09-16T15:02:48.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient(hunting): _negotiate 2024-09-16T15:02:48.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:48.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:48.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient(hunting): my global_id is 334557 2024-09-16T15:02:48.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 1 -- 172.21.15.26:0/2158169170 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f980809cf60 con 0x7f9808138ff0 2024-09-16T15:02:48.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 1 -- 172.21.15.26:0/2158169170 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3835664073 0 0) 0x7f97f0003530 con 0x7f9808132650 2024-09-16T15:02:48.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 1 -- 172.21.15.26:0/2158169170 >> v1:172.21.15.189:6789/0 conn(0x7f980809cb80 legacy=0x7f98080717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:48.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 1 -- 172.21.15.26:0/2158169170 >> v1:172.21.15.26:6789/0 conn(0x7f9808138ff0 legacy=0x7f980813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:48.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient: _finish_hunting 0 2024-09-16T15:02:48.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 1 monclient: found mon.noname-b 2024-09-16T15:02:48.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:48.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:02:48.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 1 -- 172.21.15.26:0/2158169170 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980809f2d0 con 0x7f9808132650 2024-09-16T15:02:48.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient: _finish_auth 0 2024-09-16T15:02:48.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient: _check_auth_tickets 2024-09-16T15:02:48.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:48.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 1 -- 172.21.15.26:0/2158169170 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f97fc001660 con 0x7f9808132650 2024-09-16T15:02:48.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:48.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:02:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient: dump: 2024-09-16T15:02:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:48.826 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 5 monclient: authenticate success, global_id 354371 2024-09-16T15:02:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:02:48.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 1 -- 172.21.15.26:0/2158169170 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97fc001880 con 0x7f9808132650 2024-09-16T15:02:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 1 -- 172.21.15.26:0/2158169170 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f97fc001bd0 con 0x7f9808132650 2024-09-16T15:02:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98067fc640 10 monclient: dump: 2024-09-16T15:02:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:48.827 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:48.827 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:48.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient: get_monmap_and_config success 2024-09-16T15:02:48.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 4 set_mon_vals no callback set 2024-09-16T15:02:48.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient: shutdown 2024-09-16T15:02:48.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:48.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 1 -- 172.21.15.26:0/2158169170 >> v1:172.21.15.26:6790/0 conn(0x7f9808132650 legacy=0x7f98081378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:48.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 1 -- 172.21.15.26:0/2158169170 shutdown_connections 2024-09-16T15:02:48.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 1 -- 172.21.15.26:0/2158169170 >> 172.21.15.26:0/2158169170 conn(0x7f980809fb50 msgr2=0x7f980809ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:48.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 1 -- 172.21.15.26:0/2158169170 shutdown_connections 2024-09-16T15:02:48.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 1 -- 172.21.15.26:0/2158169170 wait complete. 2024-09-16T15:02:48.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient: build_initial_monmap 2024-09-16T15:02:48.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient: monmap: 2024-09-16T15:02:48.829 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:02:48.829 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:02:48.829 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:02:48.829 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:02:48.829 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:02:48.829 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:02:48.829 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:02:48.829 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:02:48.829 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:02:48.829 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:48.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 1 Processor -- start 2024-09-16T15:02:48.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 1 -- start start 2024-09-16T15:02:48.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient: init 2024-09-16T15:02:48.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient: _reopen_session rank -1 2024-09-16T15:02:48.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T15:02:48.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient(hunting): picked mon.noname-a con 0x7f980809cb80 addr v1:172.21.15.26:6789/0 2024-09-16T15:02:48.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient(hunting): picked mon.noname-c con 0x7f9808132650 addr v1:172.21.15.26:6790/0 2024-09-16T15:02:48.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient(hunting): picked mon.noname-b con 0x7f9808138ff0 addr v1:172.21.15.189:6789/0 2024-09-16T15:02:48.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98080700f0 con 0x7f980809cb80 2024-09-16T15:02:48.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980806ff40 con 0x7f9808138ff0 2024-09-16T15:02:48.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980806e5c0 con 0x7f9808132650 2024-09-16T15:02:48.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient(hunting): _renew_subs 2024-09-16T15:02:48.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f980deda640 10 monclient(hunting): authenticate will time out at 2192.943578s 2024-09-16T15:02:48.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98077fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f980809cb80 0x7f980806d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35336/0 (socket says 172.21.15.26:35336) 2024-09-16T15:02:48.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f98077fe640 1 -- 172.21.15.26:0/1059105042 learned_addr learned my addr 172.21.15.26:0/1059105042 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:02:48.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295601754 0 0) 0x7f98080700f0 con 0x7f980809cb80 2024-09-16T15:02:48.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f97effff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:48.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f97effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:48.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f97effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:48.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f97effff640 10 monclient(hunting): my global_id is 334560 2024-09-16T15:02:48.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97f4003530 con 0x7f980809cb80 2024-09-16T15:02:48.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181979211 0 0) 0x7f980806e5c0 con 0x7f9808132650 2024-09-16T15:02:48.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f97effff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:48.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f97effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:48.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f97effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:48.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f97effff640 10 monclient(hunting): my global_id is 354374 2024-09-16T15:02:48.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.819+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98080700f0 con 0x7f9808132650 2024-09-16T15:02:48.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1490750809 0 0) 0x7f97f4003530 con 0x7f980809cb80 2024-09-16T15:02:48.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f980806e5c0 con 0x7f980809cb80 2024-09-16T15:02:48.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f97f80018e0 con 0x7f980809cb80 2024-09-16T15:02:48.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:48.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:02:48.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient(hunting): dump: 2024-09-16T15:02:48.835 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:48.835 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:48.835 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:48.835 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:48.835 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:48.835 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:48.835 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:48.835 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:48.835 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:48.836 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:48.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:02:48.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1166366725 0 0) 0x7f98080700f0 con 0x7f9808132650 2024-09-16T15:02:48.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97f4003530 con 0x7f9808132650 2024-09-16T15:02:48.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f97fc001560 con 0x7f9808132650 2024-09-16T15:02:48.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:48.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:02:48.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient(hunting): dump: 2024-09-16T15:02:48.836 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:48.836 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:48.836 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:48.836 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:48.836 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:48.837 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:48.837 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:48.837 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:48.837 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:48.837 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:48.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62265762 0 0) 0x7f980806ff40 con 0x7f9808138ff0 2024-09-16T15:02:48.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient(hunting): _negotiate 2024-09-16T15:02:48.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:02:48.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:02:48.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient(hunting): my global_id is 364237 2024-09-16T15:02:48.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98080700f0 con 0x7f9808138ff0 2024-09-16T15:02:48.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3528486569 0 0) 0x7f980806e5c0 con 0x7f980809cb80 2024-09-16T15:02:48.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 >> v1:172.21.15.26:6790/0 conn(0x7f9808132650 legacy=0x7f980806deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:48.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 >> v1:172.21.15.189:6789/0 conn(0x7f9808138ff0 legacy=0x7f98081cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:48.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient: _finish_hunting 0 2024-09-16T15:02:48.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 monclient: found mon.a 2024-09-16T15:02:48.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:02:48.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:48.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980813c400 con 0x7f980809cb80 2024-09-16T15:02:48.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient: _finish_auth 0 2024-09-16T15:02:48.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient: _check_auth_tickets 2024-09-16T15:02:48.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:02:48.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f980deda640 5 monclient: authenticate success, global_id 334560 2024-09-16T15:02:48.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f980deda640 10 monclient: _renew_subs 2024-09-16T15:02:48.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f980deda640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:48.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f980deda640 1 -- 172.21.15.26:0/1059105042 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98080a2960 con 0x7f980809cb80 2024-09-16T15:02:48.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f980deda640 10 monclient: _renew_subs 2024-09-16T15:02:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f980deda640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f980deda640 1 -- 172.21.15.26:0/1059105042 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98080713f0 con 0x7f980809cb80 2024-09-16T15:02:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97f8003620 con 0x7f980809cb80 2024-09-16T15:02:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:02:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f97f8003930 con 0x7f980809cb80 2024-09-16T15:02:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:02:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:02:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f9805ffb640 4 set_mon_vals no callback set 2024-09-16T15:02:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 10 monclient: dump: 2024-09-16T15:02:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:02:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:02:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:48.839 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:02:48.840 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:02:48.840 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:02:48.840 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:02:48.840 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:02:48.840 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:02:48.840 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:02:48.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f980deda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:48.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f980deda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:48.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f980deda640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:02:48.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f97f8003a90 con 0x7f980809cb80 2024-09-16T15:02:48.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f980deda640 1 -- 172.21.15.26:0/1059105042 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e8005050 con 0x7f980809cb80 2024-09-16T15:02:48.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f97f8043d50 con 0x7f980809cb80 2024-09-16T15:02:48.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.823+0000 7f9806ffd640 10 monclient: get_auth_request con 0x7f97f40363b0 auth_method 0 2024-09-16T15:02:48.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.827+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f97f8004290 con 0x7f980809cb80 2024-09-16T15:02:48.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.827+0000 7f97effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:02:48.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.827+0000 7f97effff640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:02:48.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.987+0000 7f980deda640 1 -- 172.21.15.26:0/1059105042 --> v1:172.21.15.189:6812/3196161582 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f97e8000ba0 con 0x7f97f40363b0 2024-09-16T15:02:48.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.995+0000 7f97effff640 1 -- 172.21.15.26:0/1059105042 <== mgr.4105 v1:172.21.15.189:6812/3196161582 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98260 (unknown 2965378022 0 573919836) 0x7f97e8000ba0 con 0x7f97f40363b0 2024-09-16T15:02:48.998 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T15:02:48.999 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T15:02:49.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.999+0000 7f980deda640 1 -- 172.21.15.26:0/1059105042 >> v1:172.21.15.189:6812/3196161582 conn(0x7f97f40363b0 legacy=0x7f97f4038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:49.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.999+0000 7f980deda640 10 monclient: shutdown 2024-09-16T15:02:49.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.999+0000 7f980deda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:02:49.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.999+0000 7f980deda640 1 -- 172.21.15.26:0/1059105042 >> v1:172.21.15.26:6789/0 conn(0x7f980809cb80 legacy=0x7f980806d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:02:49.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.999+0000 7f980deda640 1 -- 172.21.15.26:0/1059105042 shutdown_connections 2024-09-16T15:02:49.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.999+0000 7f980deda640 1 -- 172.21.15.26:0/1059105042 >> 172.21.15.26:0/1059105042 conn(0x7f980809fb50 msgr2=0x7f98080a0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:02:49.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.999+0000 7f980deda640 1 -- 172.21.15.26:0/1059105042 shutdown_connections 2024-09-16T15:02:49.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:02:48.999+0000 7f980deda640 1 -- 172.21.15.26:0/1059105042 wait complete. 2024-09-16T15:02:49.015 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":854,"stamp":"2024-09-16T15:02:48.732936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1057913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":199164,"kb_used_data":8396,"kb_used_omap":9,"kb_used_meta":190710,"kb_avail":566031876,"statfs":{"total":579820584960,"available":579616641024,"internally_reserved":0,"allocated":8597504,"data_stored":8794066,"data_compressed":1510597,"data_compressed_allocated":2170880,"data_compressed_original":4796416,"omap_allocated":9559,"internal_metadata":195287721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003542"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":176,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:37.776268+0000","last_change":"2024-09-16T15:02:37.776508+0000","last_active":"2024-09-16T15:02:37.776268+0000","last_peered":"2024-09-16T15:02:37.776268+0000","last_clean":"2024-09-16T15:02:37.776268+0000","last_became_active":"2024-09-16T14:48:52.141236+0000","last_became_peered":"2024-09-16T14:48:52.141236+0000","last_unstale":"2024-09-16T15:02:37.776268+0000","last_undegraded":"2024-09-16T15:02:37.776268+0000","last_fullsized":"2024-09-16T15:02:37.776268+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:37.776218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:37.776218+0000","last_clean_scrub_stamp":"2024-09-16T15:02:37.776218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:34:06.261458+0000","scrub_duration":0.0032363639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":101,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:41.591081+0000","last_change":"2024-09-16T15:02:41.591268+0000","last_active":"2024-09-16T15:02:41.591081+0000","last_peered":"2024-09-16T15:02:41.591081+0000","last_clean":"2024-09-16T15:02:41.591081+0000","last_became_active":"2024-09-16T14:48:52.126325+0000","last_became_peered":"2024-09-16T14:48:52.126325+0000","last_unstale":"2024-09-16T15:02:41.591081+0000","last_undegraded":"2024-09-16T15:02:41.591081+0000","last_fullsized":"2024-09-16T15:02:41.591081+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:41.591034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:41.591034+0000","last_clean_scrub_stamp":"2024-09-16T15:02:41.591034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:53:02.777154+0000","scrub_duration":0.002805526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":168,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:29.648090+0000","last_change":"2024-09-16T15:02:29.648373+0000","last_active":"2024-09-16T15:02:29.648090+0000","last_peered":"2024-09-16T15:02:29.648090+0000","last_clean":"2024-09-16T15:02:29.648090+0000","last_became_active":"2024-09-16T14:48:52.137680+0000","last_became_peered":"2024-09-16T14:48:52.137680+0000","last_unstale":"2024-09-16T15:02:29.648090+0000","last_undegraded":"2024-09-16T15:02:29.648090+0000","last_fullsized":"2024-09-16T15:02:29.648090+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:29.648041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:29.648041+0000","last_clean_scrub_stamp":"2024-09-16T15:02:29.648041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:30:06.027936+0000","scrub_duration":0.0031323079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":101,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:28.447794+0000","last_change":"2024-09-16T15:02:28.447979+0000","last_active":"2024-09-16T15:02:28.447794+0000","last_peered":"2024-09-16T15:02:28.447794+0000","last_clean":"2024-09-16T15:02:28.447794+0000","last_became_active":"2024-09-16T14:48:52.133190+0000","last_became_peered":"2024-09-16T14:48:52.133190+0000","last_unstale":"2024-09-16T15:02:28.447794+0000","last_undegraded":"2024-09-16T15:02:28.447794+0000","last_fullsized":"2024-09-16T15:02:28.447794+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:28.447742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:28.447742+0000","last_clean_scrub_stamp":"2024-09-16T15:02:28.447742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:06:55.629405+0000","scrub_duration":0.0028722219999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":126,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:28.364536+0000","last_change":"2024-09-16T14:48:53.142970+0000","last_active":"2024-09-16T15:02:28.364536+0000","last_peered":"2024-09-16T15:02:28.364536+0000","last_clean":"2024-09-16T15:02:28.364536+0000","last_became_active":"2024-09-16T14:48:53.142066+0000","last_became_peered":"2024-09-16T14:48:53.142066+0000","last_unstale":"2024-09-16T15:02:28.364536+0000","last_undegraded":"2024-09-16T15:02:28.364536+0000","last_fullsized":"2024-09-16T15:02:28.364536+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:47:31.170011+0000","last_clean_scrub_stamp":"2024-09-16T14:47:31.170011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":155,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:28.776962+0000","last_change":"2024-09-16T15:02:28.777162+0000","last_active":"2024-09-16T15:02:28.776962+0000","last_peered":"2024-09-16T15:02:28.776962+0000","last_clean":"2024-09-16T15:02:28.776962+0000","last_became_active":"2024-09-16T14:48:52.127250+0000","last_became_peered":"2024-09-16T14:48:52.127250+0000","last_unstale":"2024-09-16T15:02:28.776962+0000","last_undegraded":"2024-09-16T15:02:28.776962+0000","last_fullsized":"2024-09-16T15:02:28.776962+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:28.776902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:28.776902+0000","last_clean_scrub_stamp":"2024-09-16T15:02:28.776902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:02.012508+0000","scrub_duration":0.0027473609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":103,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:29.897041+0000","last_change":"2024-09-16T15:02:29.897364+0000","last_active":"2024-09-16T15:02:29.897041+0000","last_peered":"2024-09-16T15:02:29.897041+0000","last_clean":"2024-09-16T15:02:29.897041+0000","last_became_active":"2024-09-16T14:48:52.132725+0000","last_became_peered":"2024-09-16T14:48:52.132725+0000","last_unstale":"2024-09-16T15:02:29.897041+0000","last_undegraded":"2024-09-16T15:02:29.897041+0000","last_fullsized":"2024-09-16T15:02:29.897041+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:29.896944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:29.896944+0000","last_clean_scrub_stamp":"2024-09-16T15:02:29.896944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:46:23.086939+0000","scrub_duration":0.0038215739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:32.166812+0000","last_change":"2024-09-16T15:02:32.167001+0000","last_active":"2024-09-16T15:02:32.166812+0000","last_peered":"2024-09-16T15:02:32.166812+0000","last_clean":"2024-09-16T15:02:32.166812+0000","last_became_active":"2024-09-16T14:48:52.128799+0000","last_became_peered":"2024-09-16T14:48:52.128799+0000","last_unstale":"2024-09-16T15:02:32.166812+0000","last_undegraded":"2024-09-16T15:02:32.166812+0000","last_fullsized":"2024-09-16T15:02:32.166812+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:32.166762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:32.166762+0000","last_clean_scrub_stamp":"2024-09-16T15:02:32.166762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:59:47.625216+0000","scrub_duration":0.003161428,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":123,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:34.489168+0000","last_change":"2024-09-16T15:02:34.489374+0000","last_active":"2024-09-16T15:02:34.489168+0000","last_peered":"2024-09-16T15:02:34.489168+0000","last_clean":"2024-09-16T15:02:34.489168+0000","last_became_active":"2024-09-16T14:48:52.142697+0000","last_became_peered":"2024-09-16T14:48:52.142697+0000","last_unstale":"2024-09-16T15:02:34.489168+0000","last_undegraded":"2024-09-16T15:02:34.489168+0000","last_fullsized":"2024-09-16T15:02:34.489168+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:34.489116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:34.489116+0000","last_clean_scrub_stamp":"2024-09-16T15:02:34.489116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:16:04.696794+0000","scrub_duration":0.0030410989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":124,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:33.922190+0000","last_change":"2024-09-16T15:02:33.922392+0000","last_active":"2024-09-16T15:02:33.922190+0000","last_peered":"2024-09-16T15:02:33.922190+0000","last_clean":"2024-09-16T15:02:33.922190+0000","last_became_active":"2024-09-16T14:48:52.143756+0000","last_became_peered":"2024-09-16T14:48:52.143756+0000","last_unstale":"2024-09-16T15:02:33.922190+0000","last_undegraded":"2024-09-16T15:02:33.922190+0000","last_fullsized":"2024-09-16T15:02:33.922190+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:33.922135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:33.922135+0000","last_clean_scrub_stamp":"2024-09-16T15:02:33.922135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:51:34.609549+0000","scrub_duration":0.0033949929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":138,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:42.037368+0000","last_change":"2024-09-16T15:02:42.037571+0000","last_active":"2024-09-16T15:02:42.037368+0000","last_peered":"2024-09-16T15:02:42.037368+0000","last_clean":"2024-09-16T15:02:42.037368+0000","last_became_active":"2024-09-16T14:48:52.139205+0000","last_became_peered":"2024-09-16T14:48:52.139205+0000","last_unstale":"2024-09-16T15:02:42.037368+0000","last_undegraded":"2024-09-16T15:02:42.037368+0000","last_fullsized":"2024-09-16T15:02:42.037368+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:42.037320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:42.037320+0000","last_clean_scrub_stamp":"2024-09-16T15:02:42.037320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:45:24.558371+0000","scrub_duration":0.0029991079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:38.135573+0000","last_change":"2024-09-16T15:02:38.135771+0000","last_active":"2024-09-16T15:02:38.135573+0000","last_peered":"2024-09-16T15:02:38.135573+0000","last_clean":"2024-09-16T15:02:38.135573+0000","last_became_active":"2024-09-16T14:48:52.131089+0000","last_became_peered":"2024-09-16T14:48:52.131089+0000","last_unstale":"2024-09-16T15:02:38.135573+0000","last_undegraded":"2024-09-16T15:02:38.135573+0000","last_fullsized":"2024-09-16T15:02:38.135573+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:38.135519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:38.135519+0000","last_clean_scrub_stamp":"2024-09-16T15:02:38.135519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:56:54.906391+0000","scrub_duration":0.0029639390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":94,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:28.132349+0000","last_change":"2024-09-16T15:02:28.132437+0000","last_active":"2024-09-16T15:02:28.132349+0000","last_peered":"2024-09-16T15:02:28.132349+0000","last_clean":"2024-09-16T15:02:28.132349+0000","last_became_active":"2024-09-16T14:48:52.137830+0000","last_became_peered":"2024-09-16T14:48:52.137830+0000","last_unstale":"2024-09-16T15:02:28.132349+0000","last_undegraded":"2024-09-16T15:02:28.132349+0000","last_fullsized":"2024-09-16T15:02:28.132349+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:28.132329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:28.132329+0000","last_clean_scrub_stamp":"2024-09-16T15:02:28.132329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:51:24.543120+0000","scrub_duration":0.0021857640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":181,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:34.802560+0000","last_change":"2024-09-16T15:02:34.802748+0000","last_active":"2024-09-16T15:02:34.802560+0000","last_peered":"2024-09-16T15:02:34.802560+0000","last_clean":"2024-09-16T15:02:34.802560+0000","last_became_active":"2024-09-16T14:48:52.144280+0000","last_became_peered":"2024-09-16T14:48:52.144280+0000","last_unstale":"2024-09-16T15:02:34.802560+0000","last_undegraded":"2024-09-16T15:02:34.802560+0000","last_fullsized":"2024-09-16T15:02:34.802560+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:34.802512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:34.802512+0000","last_clean_scrub_stamp":"2024-09-16T15:02:34.802512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:34:25.730014+0000","scrub_duration":0.002951122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":102,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:30.679985+0000","last_change":"2024-09-16T15:02:30.680193+0000","last_active":"2024-09-16T15:02:30.679985+0000","last_peered":"2024-09-16T15:02:30.679985+0000","last_clean":"2024-09-16T15:02:30.679985+0000","last_became_active":"2024-09-16T14:48:53.563655+0000","last_became_peered":"2024-09-16T14:48:53.563655+0000","last_unstale":"2024-09-16T15:02:30.679985+0000","last_undegraded":"2024-09-16T15:02:30.679985+0000","last_fullsized":"2024-09-16T15:02:30.679985+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:30.679931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:30.679931+0000","last_clean_scrub_stamp":"2024-09-16T15:02:30.679931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:47:03.923625+0000","scrub_duration":0.003081332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:27.090225+0000","last_change":"2024-09-16T15:02:27.090415+0000","last_active":"2024-09-16T15:02:27.090225+0000","last_peered":"2024-09-16T15:02:27.090225+0000","last_clean":"2024-09-16T15:02:27.090225+0000","last_became_active":"2024-09-16T14:48:52.137512+0000","last_became_peered":"2024-09-16T14:48:52.137512+0000","last_unstale":"2024-09-16T15:02:27.090225+0000","last_undegraded":"2024-09-16T15:02:27.090225+0000","last_fullsized":"2024-09-16T15:02:27.090225+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:27.090164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:27.090164+0000","last_clean_scrub_stamp":"2024-09-16T15:02:27.090164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:52:02.084535+0000","scrub_duration":0.0025678739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":241,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:37.875103+0000","last_change":"2024-09-16T15:02:37.875319+0000","last_active":"2024-09-16T15:02:37.875103+0000","last_peered":"2024-09-16T15:02:37.875103+0000","last_clean":"2024-09-16T15:02:37.875103+0000","last_became_active":"2024-09-16T14:48:52.131259+0000","last_became_peered":"2024-09-16T14:48:52.131259+0000","last_unstale":"2024-09-16T15:02:37.875103+0000","last_undegraded":"2024-09-16T15:02:37.875103+0000","last_fullsized":"2024-09-16T15:02:37.875103+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:37.875053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:37.875053+0000","last_clean_scrub_stamp":"2024-09-16T15:02:37.875053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:34:43.558682+0000","scrub_duration":0.0032437389999999998,"objects_trimmed":0,"snaptrim_duration":0.00067836399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":221,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:01:22.620194+0000","last_change":"2024-09-16T14:48:53.141414+0000","last_active":"2024-09-16T15:01:22.620194+0000","last_peered":"2024-09-16T15:01:22.620194+0000","last_clean":"2024-09-16T15:01:22.620194+0000","last_became_active":"2024-09-16T14:48:53.140914+0000","last_became_peered":"2024-09-16T14:48:53.140914+0000","last_unstale":"2024-09-16T15:01:22.620194+0000","last_undegraded":"2024-09-16T15:01:22.620194+0000","last_fullsized":"2024-09-16T15:01:22.620194+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097689999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"257'3","reported_seq":106,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:33.054187+0000","last_change":"2024-09-16T15:02:33.054406+0000","last_active":"2024-09-16T15:02:33.054187+0000","last_peered":"2024-09-16T15:02:33.054187+0000","last_clean":"2024-09-16T15:02:33.054187+0000","last_became_active":"2024-09-16T14:48:52.142312+0000","last_became_peered":"2024-09-16T14:48:52.142312+0000","last_unstale":"2024-09-16T15:02:33.054187+0000","last_undegraded":"2024-09-16T15:02:33.054187+0000","last_fullsized":"2024-09-16T15:02:33.054187+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'3","last_scrub_stamp":"2024-09-16T15:02:33.054129+0000","last_deep_scrub":"257'3","last_deep_scrub_stamp":"2024-09-16T15:02:33.054129+0000","last_clean_scrub_stamp":"2024-09-16T15:02:33.054129+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:21:26.267265+0000","scrub_duration":0.0070024249999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"257'2","reported_seq":100,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:29.147037+0000","last_change":"2024-09-16T15:02:29.147233+0000","last_active":"2024-09-16T15:02:29.147037+0000","last_peered":"2024-09-16T15:02:29.147037+0000","last_clean":"2024-09-16T15:02:29.147037+0000","last_became_active":"2024-09-16T14:48:52.137048+0000","last_became_peered":"2024-09-16T14:48:52.137048+0000","last_unstale":"2024-09-16T15:02:29.147037+0000","last_undegraded":"2024-09-16T15:02:29.147037+0000","last_fullsized":"2024-09-16T15:02:29.147037+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'2","last_scrub_stamp":"2024-09-16T15:02:29.146986+0000","last_deep_scrub":"257'2","last_deep_scrub_stamp":"2024-09-16T15:02:29.146986+0000","last_clean_scrub_stamp":"2024-09-16T15:02:29.146986+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:20:50.027805+0000","scrub_duration":0.0055780450000000002,"objects_trimmed":0,"snaptrim_duration":0.00042440799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"257'119","reported_seq":443,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:28.888589+0000","last_change":"2024-09-16T15:02:28.888589+0000","last_active":"2024-09-16T15:02:28.888589+0000","last_peered":"2024-09-16T15:02:28.888589+0000","last_clean":"2024-09-16T15:02:28.888589+0000","last_became_active":"2024-09-16T14:48:55.154511+0000","last_became_peered":"2024-09-16T14:48:55.154511+0000","last_unstale":"2024-09-16T15:02:28.888589+0000","last_undegraded":"2024-09-16T15:02:28.888589+0000","last_fullsized":"2024-09-16T15:02:28.888589+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'119","last_scrub_stamp":"2024-09-16T15:02:28.888323+0000","last_deep_scrub":"257'119","last_deep_scrub_stamp":"2024-09-16T15:02:28.888323+0000","last_clean_scrub_stamp":"2024-09-16T15:02:28.888323+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:13:08.558812+0000","scrub_duration":0.0090316540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:34.234230+0000","last_change":"2024-09-16T15:02:34.234420+0000","last_active":"2024-09-16T15:02:34.234230+0000","last_peered":"2024-09-16T15:02:34.234230+0000","last_clean":"2024-09-16T15:02:34.234230+0000","last_became_active":"2024-09-16T14:48:52.128554+0000","last_became_peered":"2024-09-16T14:48:52.128554+0000","last_unstale":"2024-09-16T15:02:34.234230+0000","last_undegraded":"2024-09-16T15:02:34.234230+0000","last_fullsized":"2024-09-16T15:02:34.234230+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:34.234184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:34.234184+0000","last_clean_scrub_stamp":"2024-09-16T15:02:34.234184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:26:34.155051+0000","scrub_duration":0.0032743960000000002,"objects_trimmed":0,"snaptrim_duration":0.00039796999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":90,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:01:22.612202+0000","last_change":"2024-09-16T14:48:52.144047+0000","last_active":"2024-09-16T15:01:22.612202+0000","last_peered":"2024-09-16T15:01:22.612202+0000","last_clean":"2024-09-16T15:01:22.612202+0000","last_became_active":"2024-09-16T14:48:52.143705+0000","last_became_peered":"2024-09-16T14:48:52.143705+0000","last_unstale":"2024-09-16T15:01:22.612202+0000","last_undegraded":"2024-09-16T15:01:22.612202+0000","last_fullsized":"2024-09-16T15:01:22.612202+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T14:42:27.455102+0000","last_clean_scrub_stamp":"2024-09-16T14:42:27.455102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061478100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":102,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:40.833723+0000","last_change":"2024-09-16T15:02:40.833971+0000","last_active":"2024-09-16T15:02:40.833723+0000","last_peered":"2024-09-16T15:02:40.833723+0000","last_clean":"2024-09-16T15:02:40.833723+0000","last_became_active":"2024-09-16T14:48:52.131160+0000","last_became_peered":"2024-09-16T14:48:52.131160+0000","last_unstale":"2024-09-16T15:02:40.833723+0000","last_undegraded":"2024-09-16T15:02:40.833723+0000","last_fullsized":"2024-09-16T15:02:40.833723+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:40.833671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:40.833671+0000","last_clean_scrub_stamp":"2024-09-16T15:02:40.833671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:23:14.783075+0000","scrub_duration":0.003363787,"objects_trimmed":0,"snaptrim_duration":0.00065779100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":238,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:39.587162+0000","last_change":"2024-09-16T15:02:39.587358+0000","last_active":"2024-09-16T15:02:39.587162+0000","last_peered":"2024-09-16T15:02:39.587162+0000","last_clean":"2024-09-16T15:02:39.587162+0000","last_became_active":"2024-09-16T14:48:52.133197+0000","last_became_peered":"2024-09-16T14:48:52.133197+0000","last_unstale":"2024-09-16T15:02:39.587162+0000","last_undegraded":"2024-09-16T15:02:39.587162+0000","last_fullsized":"2024-09-16T15:02:39.587162+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:39.587113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:39.587113+0000","last_clean_scrub_stamp":"2024-09-16T15:02:39.587113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:32:20.211569+0000","scrub_duration":0.0032142170000000001,"objects_trimmed":0,"snaptrim_duration":0.00085482099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":94400,"data_compressed_allocated":122880,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640504,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32492,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339348,"statfs":{"total":96636764160,"available":96603492352,"internally_reserved":0,"allocated":1413120,"data_stored":1289362,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.622,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.92600000000000005,"5min":1.042,"15min":1.2},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.81499999999999995,"15min":0.79100000000000004},"min":{"1min":0.627,"5min":0.51500000000000001,"15min":0.47899999999999998},"max":{"1min":0.95599999999999996,"5min":1.0369999999999999,"15min":1.3300000000000001},"last":0.70999999999999996}]},{"osd":1,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.81799999999999995,"15min":0.81000000000000005},"min":{"1min":0.61899999999999999,"5min":0.44900000000000001,"15min":0.33900000000000002},"max":{"1min":1.0589999999999999,"5min":1.073,"15min":1.2010000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.752,"5min":0.74399999999999999,"15min":0.752},"min":{"1min":0.60999999999999999,"5min":0.54000000000000004,"15min":0.438},"max":{"1min":0.90800000000000003,"5min":0.94299999999999995,"15min":1.089},"last":0.67300000000000004}]},{"osd":2,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.81799999999999995,"15min":0.82899999999999996},"min":{"1min":0.628,"5min":0.50800000000000001,"15min":0.44900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.272},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.90600000000000003,"15min":0.91300000000000003},"min":{"1min":0.81000000000000005,"5min":0.52900000000000003,"15min":0.42899999999999999},"max":{"1min":1.0289999999999999,"5min":1.056,"15min":1.3540000000000001},"last":0.80100000000000005}]},{"osd":3,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69699999999999995,"15min":0.67600000000000005},"min":{"1min":0.52300000000000002,"5min":0.46600000000000003,"15min":0.26000000000000001},"max":{"1min":0.96799999999999997,"5min":0.96999999999999997,"15min":1.1020000000000001},"last":0.621},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.68200000000000005,"15min":0.69099999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33500000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":1.153},"last":0.57899999999999996}]},{"osd":4,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72899999999999998,"15min":0.749},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.27500000000000002},"max":{"1min":0.95299999999999996,"5min":1.05,"15min":1.236},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.80000000000000004,"15min":0.77200000000000002},"min":{"1min":0.48799999999999999,"5min":0.44800000000000001,"15min":0.26300000000000001},"max":{"1min":1.038,"5min":1.0529999999999999,"15min":1.2949999999999999},"last":0.76200000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640504,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1417216,"data_stored":1289381,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78200000000000003,"15min":0.77300000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.437},"max":{"1min":0.96699999999999997,"5min":1.0800000000000001,"15min":1.1579999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73899999999999999,"15min":0.79000000000000004},"min":{"1min":0.52600000000000002,"5min":0.48899999999999999,"15min":0.48499999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.1679999999999999},"last":0.81699999999999995}]},{"osd":1,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.85899999999999999,"15min":0.81599999999999995},"min":{"1min":0.71199999999999997,"5min":0.52800000000000002,"15min":0.42399999999999999},"max":{"1min":1.0429999999999999,"5min":1.0780000000000001,"15min":1.4630000000000001},"last":0.88},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.79600000000000004,"15min":0.83399999999999996},"min":{"1min":0.65700000000000003,"5min":0.53700000000000003,"15min":0.50800000000000001},"max":{"1min":1.0149999999999999,"5min":1.024,"15min":1.2729999999999999},"last":0.90300000000000002}]},{"osd":2,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.84999999999999998,"15min":0.84899999999999998},"min":{"1min":0.68000000000000005,"5min":0.65500000000000003,"15min":0.47599999999999998},"max":{"1min":1.02,"5min":1.0449999999999999,"15min":1.536},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84799999999999998,"15min":0.85899999999999999},"min":{"1min":0.63500000000000001,"5min":0.56599999999999995,"15min":0.51200000000000001},"max":{"1min":0.97099999999999997,"5min":1.0720000000000001,"15min":1.29},"last":0.85799999999999998}]},{"osd":3,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.70999999999999996,"15min":0.68899999999999995},"min":{"1min":0.53700000000000003,"5min":0.51300000000000001,"15min":0.27700000000000002},"max":{"1min":0.83299999999999996,"5min":1.0720000000000001,"15min":1.2669999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72899999999999998,"15min":0.72499999999999998},"min":{"1min":0.55700000000000005,"5min":0.50600000000000001,"15min":0.25700000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.198},"last":0.74099999999999999}]},{"osd":5,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.75600000000000001,"15min":0.73099999999999998},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.28999999999999998},"max":{"1min":0.95799999999999996,"5min":1.0600000000000001,"15min":1.1140000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.94699999999999995,"15min":0.91700000000000004},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.24299999999999999},"max":{"1min":1.0700000000000001,"5min":1.169,"15min":1.3879999999999999},"last":0.98999999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640503,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32552,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339288,"statfs":{"total":96636764160,"available":96603430912,"internally_reserved":0,"allocated":1474560,"data_stored":1818290,"data_compressed":271867,"data_compressed_allocated":389120,"data_compressed_original":1138688,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.45200000000000001},"max":{"1min":0.998,"5min":0.998,"15min":1.2729999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74099999999999999,"15min":0.753},"min":{"1min":0.60799999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90500000000000003,"5min":0.94199999999999995,"15min":1.306},"last":0.79100000000000004}]},{"osd":1,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90200000000000002,"15min":0.86599999999999999},"min":{"1min":0.55800000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.069,"5min":1.1259999999999999,"15min":1.294},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.86199999999999999,"15min":0.86599999999999999},"min":{"1min":0.67900000000000005,"5min":0.56200000000000006,"15min":0.41199999999999998},"max":{"1min":0.93999999999999995,"5min":1.046,"15min":1.9219999999999999},"last":0.82399999999999995}]},{"osd":2,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81799999999999995,"15min":0.81699999999999995},"min":{"1min":0.71299999999999997,"5min":0.61199999999999999,"15min":0.45900000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.181},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79900000000000004,"15min":0.80600000000000005},"min":{"1min":0.627,"5min":0.627,"15min":0.38800000000000001},"max":{"1min":0.90300000000000002,"5min":0.96899999999999997,"15min":2.444},"last":0.69299999999999995}]},{"osd":4,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.624,"15min":0.623},"min":{"1min":0.49299999999999999,"5min":0.47699999999999998,"15min":0.28199999999999997},"max":{"1min":0.89800000000000002,"5min":0.98799999999999999,"15min":1.0700000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.67900000000000005,"5min":0.628,"15min":0.317},"max":{"1min":0.99299999999999999,"5min":1.0740000000000001,"15min":1.2290000000000001},"last":0.70599999999999996}]},{"osd":5,"last update":"Mon Sep 16 15:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69299999999999995,"15min":0.70699999999999996},"min":{"1min":0.55500000000000005,"5min":0.52400000000000002,"15min":0.29899999999999999},"max":{"1min":0.92200000000000004,"5min":1.4279999999999999,"15min":2.5409999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.83499999999999996,"15min":0.80900000000000005},"min":{"1min":0.63700000000000001,"5min":0.60999999999999999,"15min":0.33100000000000002},"max":{"1min":1.081,"5min":1.514,"15min":1.514},"last":0.746}]}]},{"osd":2,"up_from":11,"seq":47244640503,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32492,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339348,"statfs":{"total":96636764160,"available":96603492352,"internally_reserved":0,"allocated":1413120,"data_stored":1289362,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56200000000000006,"15min":0.58099999999999996},"min":{"1min":0.252,"5min":0.252,"15min":0.249},"max":{"1min":0.749,"5min":0.94699999999999995,"15min":1.819},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.77600000000000002,"15min":0.75700000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.312},"max":{"1min":1.147,"5min":1.147,"15min":1.444},"last":0.72199999999999998}]},{"osd":1,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.39200000000000002,"5min":0.379,"15min":0.28299999999999997},"max":{"1min":0.95499999999999996,"5min":0.95999999999999996,"15min":1.27},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.62,"15min":0.623},"min":{"1min":0.313,"5min":0.313,"15min":0.26700000000000002},"max":{"1min":1.0129999999999999,"5min":1.0189999999999999,"15min":1.383},"last":0.65300000000000002}]},{"osd":3,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.84199999999999997,"15min":0.82299999999999995},"min":{"1min":0.58599999999999997,"5min":0.56000000000000005,"15min":0.36899999999999999},"max":{"1min":1.0720000000000001,"5min":1.169,"15min":1.2110000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.871,"15min":0.86299999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.44700000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.335},"last":0.997}]},{"osd":4,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.83599999999999997,"15min":0.83499999999999996},"min":{"1min":0.53300000000000003,"5min":0.50600000000000001,"15min":0.45500000000000002},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.84999999999999998,"15min":0.85499999999999998},"min":{"1min":0.62,"5min":0.58599999999999997,"15min":0.40999999999999998},"max":{"1min":0.97099999999999997,"5min":1.0489999999999999,"15min":1.367},"last":0.81299999999999994}]},{"osd":5,"last update":"Mon Sep 16 15:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.88200000000000001,"15min":0.89100000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.53700000000000003},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.911},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.95399999999999996,"15min":0.95499999999999996},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.121,"5min":1.1990000000000001,"15min":2.0499999999999998},"last":0.91800000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640502,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32540,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339300,"statfs":{"total":96636764160,"available":96603443200,"internally_reserved":0,"allocated":1462272,"data_stored":1818290,"data_compressed":273302,"data_compressed_allocated":389120,"data_compressed_original":1150976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56000000000000005,"15min":0.57099999999999995},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.251},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":1.0720000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.628,"15min":0.63800000000000001},"min":{"1min":0.44500000000000001,"5min":0.36199999999999999,"15min":0.26100000000000001},"max":{"1min":0.87,"5min":0.87,"15min":1.0589999999999999},"last":0.58999999999999997}]},{"osd":2,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.66700000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.28699999999999998},"max":{"1min":0.82599999999999996,"5min":0.88200000000000001,"15min":1.052},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.628,"15min":0.66700000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32100000000000001},"max":{"1min":0.81899999999999995,"5min":0.879,"15min":1.403},"last":0.624}]},{"osd":3,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85699999999999998,"15min":0.88800000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.503},"max":{"1min":1.052,"5min":1.117,"15min":1.359},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.86299999999999999,"15min":0.877},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1459999999999999,"5min":1.147,"15min":1.284},"last":0.88200000000000001}]},{"osd":4,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.88900000000000001,"15min":0.88200000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.46600000000000003},"max":{"1min":1.0900000000000001,"5min":1.097,"15min":1.341},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.88,"5min":0.86199999999999999,"15min":0.86899999999999999},"min":{"1min":0.67600000000000005,"5min":0.66200000000000003,"15min":0.46999999999999997},"max":{"1min":1.056,"5min":1.0800000000000001,"15min":1.399},"last":0.85699999999999998}]},{"osd":5,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.622,"5min":0.622,"15min":0.50900000000000001},"max":{"1min":1,"5min":1.01,"15min":1.258},"last":0.749},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90100000000000002,"15min":0.90400000000000003},"min":{"1min":0.69499999999999995,"5min":0.629,"15min":0.47299999999999998},"max":{"1min":1.1519999999999999,"5min":1.411,"15min":1.411},"last":0.78300000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640503,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36592,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335248,"statfs":{"total":96636764160,"available":96599293952,"internally_reserved":0,"allocated":1417216,"data_stored":1289381,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57499999999999996,"15min":0.57999999999999996},"min":{"1min":0.36099999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.74299999999999999,"5min":0.77700000000000002,"15min":1.1739999999999999},"last":0.504},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.61399999999999999,"15min":0.63800000000000001},"min":{"1min":0.38200000000000001,"5min":0.28299999999999997,"15min":0.27400000000000002},"max":{"1min":0.67800000000000005,"5min":1.032,"15min":1.21},"last":0.60299999999999998}]},{"osd":2,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68700000000000006,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.29899999999999999,"15min":0.25900000000000001},"max":{"1min":0.85799999999999998,"5min":0.997,"15min":1.103},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.62,"15min":0.67200000000000004},"min":{"1min":0.47999999999999998,"5min":0.36099999999999999,"15min":0.26900000000000002},"max":{"1min":0.69099999999999995,"5min":1.008,"15min":1.137},"last":0.56599999999999995}]},{"osd":3,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.82599999999999996},"min":{"1min":0.67100000000000004,"5min":0.59599999999999997,"15min":0.52400000000000002},"max":{"1min":1.014,"5min":1.1040000000000001,"15min":1.4650000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.81799999999999995,"15min":0.83999999999999997},"min":{"1min":0.69299999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.99299999999999999,"5min":1.101,"15min":1.4950000000000001},"last":0.78500000000000003}]},{"osd":4,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.84599999999999997,"15min":0.84699999999999998},"min":{"1min":0.68899999999999995,"5min":0.57999999999999996,"15min":0.48999999999999999},"max":{"1min":0.98499999999999999,"5min":1.1559999999999999,"15min":1.3300000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.85099999999999998},"min":{"1min":0.63700000000000001,"5min":0.55700000000000005,"15min":0.51600000000000001},"max":{"1min":0.96099999999999997,"5min":1.095,"15min":1.4179999999999999},"last":0.749}]},{"osd":5,"last update":"Mon Sep 16 15:01:58 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.91200000000000003,"15min":0.91400000000000003},"min":{"1min":0.70899999999999996,"5min":0.66400000000000003,"15min":0.57999999999999996},"max":{"1min":1.1080000000000001,"5min":1.304,"15min":1.6499999999999999},"last":1.004},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95799999999999996,"15min":0.94799999999999995},"min":{"1min":0.76900000000000002,"5min":0.69999999999999996,"15min":0.51900000000000002},"max":{"1min":1.0589999999999999,"5min":1.2030000000000001,"15min":1.7290000000000001},"last":0.93400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:02:49.017 INFO:tasks.ceph:pgid 10.6 last_scrub_stamp 2024-09-16T14:47:31.170011+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=47, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:49.018 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-16T14:42:27.455102+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:49.018 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-16T14:42:27.455102+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=14, tm_min=42, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=2, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:02:49.018 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T15:03:09.019 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:03:09.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.150+0000 7f5b2a5ed640 10 monclient: get_monmap_and_config 2024-09-16T15:03:09.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.150+0000 7f5b2a5ed640 10 monclient: build_initial_monmap 2024-09-16T15:03:09.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.150+0000 7f5b2a5ed640 10 monclient: monmap: 2024-09-16T15:03:09.154 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:03:09.154 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:03:09.154 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:03:09.155892+0000 2024-09-16T15:03:09.154 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:03:09.155892+0000 2024-09-16T15:03:09.154 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:03:09.154 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:03:09.155 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:03:09.155 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:03:09.155 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:03:09.155 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:03:09.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.150+0000 7f5b2a5ed640 1 Processor -- start 2024-09-16T15:03:09.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.150+0000 7f5b2a5ed640 1 -- start start 2024-09-16T15:03:09.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.150+0000 7f5b2a5ed640 10 monclient: init 2024-09-16T15:03:09.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.150+0000 7f5b2a5ed640 10 monclient: _reopen_session rank -1 2024-09-16T15:03:09.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.150+0000 7f5b2a5ed640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T15:03:09.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.150+0000 7f5b2a5ed640 10 monclient(hunting): picked mon.noname-a con 0x7f5b24134e40 addr v1:172.21.15.26:6789/0 2024-09-16T15:03:09.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.150+0000 7f5b2a5ed640 10 monclient(hunting): picked mon.noname-c con 0x7f5b24099a70 addr v1:172.21.15.189:6789/0 2024-09-16T15:03:09.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.150+0000 7f5b2a5ed640 10 monclient(hunting): picked mon.noname-b con 0x7f5b24135220 addr v1:172.21.15.26:6790/0 2024-09-16T15:03:09.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.150+0000 7f5b2a5ed640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b24099e80 con 0x7f5b24134e40 2024-09-16T15:03:09.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.150+0000 7f5b2a5ed640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240719d0 con 0x7f5b24099a70 2024-09-16T15:03:09.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.150+0000 7f5b2a5ed640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2409e1e0 con 0x7f5b24135220 2024-09-16T15:03:09.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.150+0000 7f5b2a5ed640 10 monclient(hunting): _renew_subs 2024-09-16T15:03:09.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.150+0000 7f5b2a5ed640 10 monclient(hunting): authenticate will time out at 2213.275940s 2024-09-16T15:03:09.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b28b8a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5b24135220 0x7f5b2413db30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35756/0 (socket says 172.21.15.26:35756) 2024-09-16T15:03:09.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b28b8a640 1 -- 172.21.15.26:0/193892942 learned_addr learned my addr 172.21.15.26:0/193892942 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:03:09.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 1 -- 172.21.15.26:0/193892942 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505364779 0 0) 0x7f5b2409e1e0 con 0x7f5b24135220 2024-09-16T15:03:09.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:03:09.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:03:09.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:03:09.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient(hunting): my global_id is 354377 2024-09-16T15:03:09.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 1 -- 172.21.15.26:0/193892942 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b0c003530 con 0x7f5b24135220 2024-09-16T15:03:09.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 1 -- 172.21.15.26:0/193892942 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139211911 0 0) 0x7f5b1c0014a0 con 0x7f5b24134e40 2024-09-16T15:03:09.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:03:09.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:03:09.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:03:09.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient(hunting): my global_id is 334563 2024-09-16T15:03:09.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 1 -- 172.21.15.26:0/193892942 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b2409e1e0 con 0x7f5b24134e40 2024-09-16T15:03:09.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 1 -- 172.21.15.26:0/193892942 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3999491064 0 0) 0x7f5b0c003530 con 0x7f5b24135220 2024-09-16T15:03:09.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 1 -- 172.21.15.26:0/193892942 >> v1:172.21.15.189:6789/0 conn(0x7f5b24099a70 legacy=0x7f5b240712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:03:09.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 1 -- 172.21.15.26:0/193892942 >> v1:172.21.15.26:6789/0 conn(0x7f5b24134e40 legacy=0x7f5b24070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:03:09.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient: _finish_hunting 0 2024-09-16T15:03:09.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 1 monclient: found mon.noname-b 2024-09-16T15:03:09.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:03:09.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-09-16T15:03:09.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 1 -- 172.21.15.26:0/193892942 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2409e7d0 con 0x7f5b24135220 2024-09-16T15:03:09.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient: _finish_auth 0 2024-09-16T15:03:09.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient: _check_auth_tickets 2024-09-16T15:03:09.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:03:09.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 1 -- 172.21.15.26:0/193892942 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5b14001620 con 0x7f5b24135220 2024-09-16T15:03:09.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:03:09.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T15:03:09.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient: dump: 2024-09-16T15:03:09.161 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:03:09.161 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:03:09.161 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.161 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.161 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:03:09.161 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:03:09.161 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:03:09.161 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:03:09.161 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:03:09.162 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:03:09.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 5 monclient: authenticate success, global_id 354377 2024-09-16T15:03:09.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:03:09.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 1 -- 172.21.15.26:0/193892942 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b14001850 con 0x7f5b24135220 2024-09-16T15:03:09.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:03:09.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 1 -- 172.21.15.26:0/193892942 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5b14001c20 con 0x7f5b24135220 2024-09-16T15:03:09.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:03:09.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T15:03:09.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b22ffd640 10 monclient: dump: 2024-09-16T15:03:09.162 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:03:09.162 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:03:09.163 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.163 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.163 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:03:09.163 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:03:09.163 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:03:09.163 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:03:09.163 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:03:09.163 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:03:09.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 10 monclient: get_monmap_and_config success 2024-09-16T15:03:09.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 4 set_mon_vals no callback set 2024-09-16T15:03:09.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 10 monclient: shutdown 2024-09-16T15:03:09.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:03:09.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 1 -- 172.21.15.26:0/193892942 >> v1:172.21.15.26:6790/0 conn(0x7f5b24135220 legacy=0x7f5b2413db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:03:09.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 1 -- 172.21.15.26:0/193892942 shutdown_connections 2024-09-16T15:03:09.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 1 -- 172.21.15.26:0/193892942 >> 172.21.15.26:0/193892942 conn(0x7f5b2409fb50 msgr2=0x7f5b2409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:03:09.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 1 -- 172.21.15.26:0/193892942 shutdown_connections 2024-09-16T15:03:09.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 1 -- 172.21.15.26:0/193892942 wait complete. 2024-09-16T15:03:09.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 10 monclient: build_initial_monmap 2024-09-16T15:03:09.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 10 monclient: monmap: 2024-09-16T15:03:09.164 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:03:09.164 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:03:09.164 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:03:09.164 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:03:09.164 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:03:09.164 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:03:09.164 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:03:09.164 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:03:09.164 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:03:09.164 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:03:09.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 1 Processor -- start 2024-09-16T15:03:09.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 1 -- start start 2024-09-16T15:03:09.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 10 monclient: init 2024-09-16T15:03:09.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 10 monclient: _reopen_session rank -1 2024-09-16T15:03:09.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T15:03:09.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 10 monclient(hunting): picked mon.noname-c con 0x7f5b24099a70 addr v1:172.21.15.26:6790/0 2024-09-16T15:03:09.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 10 monclient(hunting): picked mon.noname-b con 0x7f5b24134e40 addr v1:172.21.15.189:6789/0 2024-09-16T15:03:09.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 10 monclient(hunting): picked mon.noname-a con 0x7f5b24135220 addr v1:172.21.15.26:6789/0 2024-09-16T15:03:09.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b241241b0 con 0x7f5b24135220 2024-09-16T15:03:09.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b24122d00 con 0x7f5b24134e40 2024-09-16T15:03:09.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b241343b0 con 0x7f5b24099a70 2024-09-16T15:03:09.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 10 monclient(hunting): _renew_subs 2024-09-16T15:03:09.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b2a5ed640 10 monclient(hunting): authenticate will time out at 2213.279464s 2024-09-16T15:03:09.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b28b8a640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5b24135220 0x7f5b241d2250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33340/0 (socket says 172.21.15.26:33340) 2024-09-16T15:03:09.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b28b8a640 1 -- 172.21.15.26:0/286101270 learned_addr learned my addr 172.21.15.26:0/286101270 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:03:09.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250083105 0 0) 0x7f5b241343b0 con 0x7f5b24099a70 2024-09-16T15:03:09.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b20ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:03:09.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b20ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:03:09.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:03:09.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b20ff9640 10 monclient(hunting): my global_id is 354380 2024-09-16T15:03:09.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b10003530 con 0x7f5b24099a70 2024-09-16T15:03:09.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161812966 0 0) 0x7f5b241241b0 con 0x7f5b24135220 2024-09-16T15:03:09.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b20ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:03:09.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b20ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:03:09.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:03:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b20ff9640 10 monclient(hunting): my global_id is 334566 2024-09-16T15:03:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.154+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b241343b0 con 0x7f5b24135220 2024-09-16T15:03:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054484744 0 0) 0x7f5b24122d00 con 0x7f5b24134e40 2024-09-16T15:03:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:03:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:03:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:03:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient(hunting): my global_id is 364243 2024-09-16T15:03:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b241241b0 con 0x7f5b24134e40 2024-09-16T15:03:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2928387383 0 0) 0x7f5b241343b0 con 0x7f5b24135220 2024-09-16T15:03:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b24122d00 con 0x7f5b24135220 2024-09-16T15:03:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3050340951 0 0) 0x7f5b10003530 con 0x7f5b24099a70 2024-09-16T15:03:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b241343b0 con 0x7f5b24099a70 2024-09-16T15:03:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5b14001b60 con 0x7f5b24135220 2024-09-16T15:03:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:03:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:03:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient(hunting): dump: 2024-09-16T15:03:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:03:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:03:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:03:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:03:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:03:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:03:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:03:09.174 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:03:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:03:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5b1c001bd0 con 0x7f5b24099a70 2024-09-16T15:03:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:03:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:03:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient(hunting): dump: 2024-09-16T15:03:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:03:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:03:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:03:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:03:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:03:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:03:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:03:09.176 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:03:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1130891249 0 0) 0x7f5b24122d00 con 0x7f5b24135220 2024-09-16T15:03:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 >> v1:172.21.15.26:6790/0 conn(0x7f5b24099a70 legacy=0x7f5b241c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:03:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 >> v1:172.21.15.189:6789/0 conn(0x7f5b24134e40 legacy=0x7f5b241ceb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:03:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient: _finish_hunting 0 2024-09-16T15:03:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 monclient: found mon.a 2024-09-16T15:03:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:03:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:03:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2409bb90 con 0x7f5b24135220 2024-09-16T15:03:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient: _finish_auth 0 2024-09-16T15:03:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient: _check_auth_tickets 2024-09-16T15:03:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:03:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b2a5ed640 5 monclient: authenticate success, global_id 334566 2024-09-16T15:03:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b2a5ed640 10 monclient: _renew_subs 2024-09-16T15:03:09.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b2a5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:03:09.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b2a5ed640 1 -- 172.21.15.26:0/286101270 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b24124490 con 0x7f5b24135220 2024-09-16T15:03:09.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b2a5ed640 10 monclient: _renew_subs 2024-09-16T15:03:09.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b2a5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:03:09.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b2a5ed640 1 -- 172.21.15.26:0/286101270 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b24121540 con 0x7f5b24135220 2024-09-16T15:03:09.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b14004aa0 con 0x7f5b24135220 2024-09-16T15:03:09.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:03:09.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f5b14004db0 con 0x7f5b24135220 2024-09-16T15:03:09.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:03:09.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:03:09.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 10 monclient: dump: 2024-09-16T15:03:09.177 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:03:09.177 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:03:09.177 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.178 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.178 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:03:09.178 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:03:09.178 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:03:09.178 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:03:09.178 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:03:09.178 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:03:09.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b227fc640 4 set_mon_vals no callback set 2024-09-16T15:03:09.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b2a5ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:03:09.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b2a5ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:03:09.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b2a5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:03:09.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f5b14004f90 con 0x7f5b24135220 2024-09-16T15:03:09.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b2a5ed640 1 -- 172.21.15.26:0/286101270 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b04005050 con 0x7f5b24135220 2024-09-16T15:03:09.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f5b14043e00 con 0x7f5b24135220 2024-09-16T15:03:09.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.158+0000 7f5b23fff640 10 monclient: get_auth_request con 0x7f5b10036450 auth_method 0 2024-09-16T15:03:09.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.162+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5b14006430 con 0x7f5b24135220 2024-09-16T15:03:09.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.162+0000 7f5b20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:03:09.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.162+0000 7f5b20ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:03:09.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.318+0000 7f5b2a5ed640 1 -- 172.21.15.26:0/286101270 --> v1:172.21.15.189:6812/3196161582 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5b04000ba0 con 0x7f5b10036450 2024-09-16T15:03:09.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.326+0000 7f5b20ff9640 1 -- 172.21.15.26:0/286101270 <== mgr.4105 v1:172.21.15.189:6812/3196161582 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98404 (unknown 2965378022 0 3230314990) 0x7f5b04000ba0 con 0x7f5b10036450 2024-09-16T15:03:09.330 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-09-16T15:03:09.330 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-09-16T15:03:09.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.326+0000 7f5b2a5ed640 1 -- 172.21.15.26:0/286101270 >> v1:172.21.15.189:6812/3196161582 conn(0x7f5b10036450 legacy=0x7f5b10038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:03:09.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.326+0000 7f5b2a5ed640 10 monclient: shutdown 2024-09-16T15:03:09.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.326+0000 7f5b2a5ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:03:09.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.326+0000 7f5b2a5ed640 1 -- 172.21.15.26:0/286101270 >> v1:172.21.15.26:6789/0 conn(0x7f5b24135220 legacy=0x7f5b241d2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:03:09.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.330+0000 7f5b2a5ed640 1 -- 172.21.15.26:0/286101270 shutdown_connections 2024-09-16T15:03:09.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.330+0000 7f5b2a5ed640 1 -- 172.21.15.26:0/286101270 >> 172.21.15.26:0/286101270 conn(0x7f5b2409fb50 msgr2=0x7f5b240a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:03:09.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.330+0000 7f5b2a5ed640 1 -- 172.21.15.26:0/286101270 shutdown_connections 2024-09-16T15:03:09.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.330+0000 7f5b2a5ed640 1 -- 172.21.15.26:0/286101270 wait complete. 2024-09-16T15:03:09.348 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":864,"stamp":"2024-09-16T15:03:08.738883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1057913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":199164,"kb_used_data":8396,"kb_used_omap":9,"kb_used_meta":190710,"kb_avail":566031876,"statfs":{"total":579820584960,"available":579616641024,"internally_reserved":0,"allocated":8597504,"data_stored":8794066,"data_compressed":1510597,"data_compressed_allocated":2170880,"data_compressed_original":4796416,"omap_allocated":9559,"internal_metadata":195287721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003532"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":176,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:37.776268+0000","last_change":"2024-09-16T15:02:37.776508+0000","last_active":"2024-09-16T15:02:37.776268+0000","last_peered":"2024-09-16T15:02:37.776268+0000","last_clean":"2024-09-16T15:02:37.776268+0000","last_became_active":"2024-09-16T14:48:52.141236+0000","last_became_peered":"2024-09-16T14:48:52.141236+0000","last_unstale":"2024-09-16T15:02:37.776268+0000","last_undegraded":"2024-09-16T15:02:37.776268+0000","last_fullsized":"2024-09-16T15:02:37.776268+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:37.776218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:37.776218+0000","last_clean_scrub_stamp":"2024-09-16T15:02:37.776218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:34:06.261458+0000","scrub_duration":0.0032363639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":101,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:41.591081+0000","last_change":"2024-09-16T15:02:41.591268+0000","last_active":"2024-09-16T15:02:41.591081+0000","last_peered":"2024-09-16T15:02:41.591081+0000","last_clean":"2024-09-16T15:02:41.591081+0000","last_became_active":"2024-09-16T14:48:52.126325+0000","last_became_peered":"2024-09-16T14:48:52.126325+0000","last_unstale":"2024-09-16T15:02:41.591081+0000","last_undegraded":"2024-09-16T15:02:41.591081+0000","last_fullsized":"2024-09-16T15:02:41.591081+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:41.591034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:41.591034+0000","last_clean_scrub_stamp":"2024-09-16T15:02:41.591034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:53:02.777154+0000","scrub_duration":0.002805526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":168,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:29.648090+0000","last_change":"2024-09-16T15:02:29.648373+0000","last_active":"2024-09-16T15:02:29.648090+0000","last_peered":"2024-09-16T15:02:29.648090+0000","last_clean":"2024-09-16T15:02:29.648090+0000","last_became_active":"2024-09-16T14:48:52.137680+0000","last_became_peered":"2024-09-16T14:48:52.137680+0000","last_unstale":"2024-09-16T15:02:29.648090+0000","last_undegraded":"2024-09-16T15:02:29.648090+0000","last_fullsized":"2024-09-16T15:02:29.648090+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:29.648041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:29.648041+0000","last_clean_scrub_stamp":"2024-09-16T15:02:29.648041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:30:06.027936+0000","scrub_duration":0.0031323079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":101,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:28.447794+0000","last_change":"2024-09-16T15:02:28.447979+0000","last_active":"2024-09-16T15:02:28.447794+0000","last_peered":"2024-09-16T15:02:28.447794+0000","last_clean":"2024-09-16T15:02:28.447794+0000","last_became_active":"2024-09-16T14:48:52.133190+0000","last_became_peered":"2024-09-16T14:48:52.133190+0000","last_unstale":"2024-09-16T15:02:28.447794+0000","last_undegraded":"2024-09-16T15:02:28.447794+0000","last_fullsized":"2024-09-16T15:02:28.447794+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:28.447742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:28.447742+0000","last_clean_scrub_stamp":"2024-09-16T15:02:28.447742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:06:55.629405+0000","scrub_duration":0.0028722219999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":137,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:50.468409+0000","last_change":"2024-09-16T15:02:50.468613+0000","last_active":"2024-09-16T15:02:50.468409+0000","last_peered":"2024-09-16T15:02:50.468409+0000","last_clean":"2024-09-16T15:02:50.468409+0000","last_became_active":"2024-09-16T14:48:53.142066+0000","last_became_peered":"2024-09-16T14:48:53.142066+0000","last_unstale":"2024-09-16T15:02:50.468409+0000","last_undegraded":"2024-09-16T15:02:50.468409+0000","last_fullsized":"2024-09-16T15:02:50.468409+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:50.468360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:50.468360+0000","last_clean_scrub_stamp":"2024-09-16T15:02:50.468360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:47:44.734633+0000","scrub_duration":0.0032666739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":155,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:28.776962+0000","last_change":"2024-09-16T15:02:28.777162+0000","last_active":"2024-09-16T15:02:28.776962+0000","last_peered":"2024-09-16T15:02:28.776962+0000","last_clean":"2024-09-16T15:02:28.776962+0000","last_became_active":"2024-09-16T14:48:52.127250+0000","last_became_peered":"2024-09-16T14:48:52.127250+0000","last_unstale":"2024-09-16T15:02:28.776962+0000","last_undegraded":"2024-09-16T15:02:28.776962+0000","last_fullsized":"2024-09-16T15:02:28.776962+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:28.776902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:28.776902+0000","last_clean_scrub_stamp":"2024-09-16T15:02:28.776902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:02.012508+0000","scrub_duration":0.0027473609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":103,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:29.897041+0000","last_change":"2024-09-16T15:02:29.897364+0000","last_active":"2024-09-16T15:02:29.897041+0000","last_peered":"2024-09-16T15:02:29.897041+0000","last_clean":"2024-09-16T15:02:29.897041+0000","last_became_active":"2024-09-16T14:48:52.132725+0000","last_became_peered":"2024-09-16T14:48:52.132725+0000","last_unstale":"2024-09-16T15:02:29.897041+0000","last_undegraded":"2024-09-16T15:02:29.897041+0000","last_fullsized":"2024-09-16T15:02:29.897041+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:29.896944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:29.896944+0000","last_clean_scrub_stamp":"2024-09-16T15:02:29.896944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:46:23.086939+0000","scrub_duration":0.0038215739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:32.166812+0000","last_change":"2024-09-16T15:02:32.167001+0000","last_active":"2024-09-16T15:02:32.166812+0000","last_peered":"2024-09-16T15:02:32.166812+0000","last_clean":"2024-09-16T15:02:32.166812+0000","last_became_active":"2024-09-16T14:48:52.128799+0000","last_became_peered":"2024-09-16T14:48:52.128799+0000","last_unstale":"2024-09-16T15:02:32.166812+0000","last_undegraded":"2024-09-16T15:02:32.166812+0000","last_fullsized":"2024-09-16T15:02:32.166812+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:32.166762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:32.166762+0000","last_clean_scrub_stamp":"2024-09-16T15:02:32.166762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:59:47.625216+0000","scrub_duration":0.003161428,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":123,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:34.489168+0000","last_change":"2024-09-16T15:02:34.489374+0000","last_active":"2024-09-16T15:02:34.489168+0000","last_peered":"2024-09-16T15:02:34.489168+0000","last_clean":"2024-09-16T15:02:34.489168+0000","last_became_active":"2024-09-16T14:48:52.142697+0000","last_became_peered":"2024-09-16T14:48:52.142697+0000","last_unstale":"2024-09-16T15:02:34.489168+0000","last_undegraded":"2024-09-16T15:02:34.489168+0000","last_fullsized":"2024-09-16T15:02:34.489168+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:34.489116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:34.489116+0000","last_clean_scrub_stamp":"2024-09-16T15:02:34.489116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:16:04.696794+0000","scrub_duration":0.0030410989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":124,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:33.922190+0000","last_change":"2024-09-16T15:02:33.922392+0000","last_active":"2024-09-16T15:02:33.922190+0000","last_peered":"2024-09-16T15:02:33.922190+0000","last_clean":"2024-09-16T15:02:33.922190+0000","last_became_active":"2024-09-16T14:48:52.143756+0000","last_became_peered":"2024-09-16T14:48:52.143756+0000","last_unstale":"2024-09-16T15:02:33.922190+0000","last_undegraded":"2024-09-16T15:02:33.922190+0000","last_fullsized":"2024-09-16T15:02:33.922190+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:33.922135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:33.922135+0000","last_clean_scrub_stamp":"2024-09-16T15:02:33.922135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:51:34.609549+0000","scrub_duration":0.0033949929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":138,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:42.037368+0000","last_change":"2024-09-16T15:02:42.037571+0000","last_active":"2024-09-16T15:02:42.037368+0000","last_peered":"2024-09-16T15:02:42.037368+0000","last_clean":"2024-09-16T15:02:42.037368+0000","last_became_active":"2024-09-16T14:48:52.139205+0000","last_became_peered":"2024-09-16T14:48:52.139205+0000","last_unstale":"2024-09-16T15:02:42.037368+0000","last_undegraded":"2024-09-16T15:02:42.037368+0000","last_fullsized":"2024-09-16T15:02:42.037368+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:42.037320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:42.037320+0000","last_clean_scrub_stamp":"2024-09-16T15:02:42.037320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:45:24.558371+0000","scrub_duration":0.0029991079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:38.135573+0000","last_change":"2024-09-16T15:02:38.135771+0000","last_active":"2024-09-16T15:02:38.135573+0000","last_peered":"2024-09-16T15:02:38.135573+0000","last_clean":"2024-09-16T15:02:38.135573+0000","last_became_active":"2024-09-16T14:48:52.131089+0000","last_became_peered":"2024-09-16T14:48:52.131089+0000","last_unstale":"2024-09-16T15:02:38.135573+0000","last_undegraded":"2024-09-16T15:02:38.135573+0000","last_fullsized":"2024-09-16T15:02:38.135573+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:38.135519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:38.135519+0000","last_clean_scrub_stamp":"2024-09-16T15:02:38.135519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:56:54.906391+0000","scrub_duration":0.0029639390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":94,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:28.132349+0000","last_change":"2024-09-16T15:02:28.132437+0000","last_active":"2024-09-16T15:02:28.132349+0000","last_peered":"2024-09-16T15:02:28.132349+0000","last_clean":"2024-09-16T15:02:28.132349+0000","last_became_active":"2024-09-16T14:48:52.137830+0000","last_became_peered":"2024-09-16T14:48:52.137830+0000","last_unstale":"2024-09-16T15:02:28.132349+0000","last_undegraded":"2024-09-16T15:02:28.132349+0000","last_fullsized":"2024-09-16T15:02:28.132349+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:28.132329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:28.132329+0000","last_clean_scrub_stamp":"2024-09-16T15:02:28.132329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:51:24.543120+0000","scrub_duration":0.0021857640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":181,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:34.802560+0000","last_change":"2024-09-16T15:02:34.802748+0000","last_active":"2024-09-16T15:02:34.802560+0000","last_peered":"2024-09-16T15:02:34.802560+0000","last_clean":"2024-09-16T15:02:34.802560+0000","last_became_active":"2024-09-16T14:48:52.144280+0000","last_became_peered":"2024-09-16T14:48:52.144280+0000","last_unstale":"2024-09-16T15:02:34.802560+0000","last_undegraded":"2024-09-16T15:02:34.802560+0000","last_fullsized":"2024-09-16T15:02:34.802560+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:34.802512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:34.802512+0000","last_clean_scrub_stamp":"2024-09-16T15:02:34.802512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:34:25.730014+0000","scrub_duration":0.002951122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":102,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:30.679985+0000","last_change":"2024-09-16T15:02:30.680193+0000","last_active":"2024-09-16T15:02:30.679985+0000","last_peered":"2024-09-16T15:02:30.679985+0000","last_clean":"2024-09-16T15:02:30.679985+0000","last_became_active":"2024-09-16T14:48:53.563655+0000","last_became_peered":"2024-09-16T14:48:53.563655+0000","last_unstale":"2024-09-16T15:02:30.679985+0000","last_undegraded":"2024-09-16T15:02:30.679985+0000","last_fullsized":"2024-09-16T15:02:30.679985+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:30.679931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:30.679931+0000","last_clean_scrub_stamp":"2024-09-16T15:02:30.679931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:47:03.923625+0000","scrub_duration":0.003081332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":94,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:27.090415+0000","last_change":"2024-09-16T15:02:27.090415+0000","last_active":"2024-09-16T15:02:27.090415+0000","last_peered":"2024-09-16T15:02:27.090415+0000","last_clean":"2024-09-16T15:02:27.090415+0000","last_became_active":"2024-09-16T14:48:52.137512+0000","last_became_peered":"2024-09-16T14:48:52.137512+0000","last_unstale":"2024-09-16T15:02:27.090415+0000","last_undegraded":"2024-09-16T15:02:27.090415+0000","last_fullsized":"2024-09-16T15:02:27.090415+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:27.090164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:27.090164+0000","last_clean_scrub_stamp":"2024-09-16T15:02:27.090164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:52:02.084535+0000","scrub_duration":0.0025678739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":242,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:37.875319+0000","last_change":"2024-09-16T15:02:37.875319+0000","last_active":"2024-09-16T15:02:37.875319+0000","last_peered":"2024-09-16T15:02:37.875319+0000","last_clean":"2024-09-16T15:02:37.875319+0000","last_became_active":"2024-09-16T14:48:52.131259+0000","last_became_peered":"2024-09-16T14:48:52.131259+0000","last_unstale":"2024-09-16T15:02:37.875319+0000","last_undegraded":"2024-09-16T15:02:37.875319+0000","last_fullsized":"2024-09-16T15:02:37.875319+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:37.875053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:37.875053+0000","last_clean_scrub_stamp":"2024-09-16T15:02:37.875053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:34:43.558682+0000","scrub_duration":0.0032437389999999998,"objects_trimmed":0,"snaptrim_duration":0.00067836399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":232,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:48.520863+0000","last_change":"2024-09-16T15:02:48.521064+0000","last_active":"2024-09-16T15:02:48.520863+0000","last_peered":"2024-09-16T15:02:48.520863+0000","last_clean":"2024-09-16T15:02:48.520863+0000","last_became_active":"2024-09-16T14:48:53.140914+0000","last_became_peered":"2024-09-16T14:48:53.140914+0000","last_unstale":"2024-09-16T15:02:48.520863+0000","last_undegraded":"2024-09-16T15:02:48.520863+0000","last_fullsized":"2024-09-16T15:02:48.520863+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:48.520812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:48.520812+0000","last_clean_scrub_stamp":"2024-09-16T15:02:48.520812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:07:34.307593+0000","scrub_duration":0.003302794,"objects_trimmed":0,"snaptrim_duration":0.00097689999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"257'3","reported_seq":106,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:33.054187+0000","last_change":"2024-09-16T15:02:33.054406+0000","last_active":"2024-09-16T15:02:33.054187+0000","last_peered":"2024-09-16T15:02:33.054187+0000","last_clean":"2024-09-16T15:02:33.054187+0000","last_became_active":"2024-09-16T14:48:52.142312+0000","last_became_peered":"2024-09-16T14:48:52.142312+0000","last_unstale":"2024-09-16T15:02:33.054187+0000","last_undegraded":"2024-09-16T15:02:33.054187+0000","last_fullsized":"2024-09-16T15:02:33.054187+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'3","last_scrub_stamp":"2024-09-16T15:02:33.054129+0000","last_deep_scrub":"257'3","last_deep_scrub_stamp":"2024-09-16T15:02:33.054129+0000","last_clean_scrub_stamp":"2024-09-16T15:02:33.054129+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:21:26.267265+0000","scrub_duration":0.0070024249999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"257'2","reported_seq":100,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:29.147037+0000","last_change":"2024-09-16T15:02:29.147233+0000","last_active":"2024-09-16T15:02:29.147037+0000","last_peered":"2024-09-16T15:02:29.147037+0000","last_clean":"2024-09-16T15:02:29.147037+0000","last_became_active":"2024-09-16T14:48:52.137048+0000","last_became_peered":"2024-09-16T14:48:52.137048+0000","last_unstale":"2024-09-16T15:02:29.147037+0000","last_undegraded":"2024-09-16T15:02:29.147037+0000","last_fullsized":"2024-09-16T15:02:29.147037+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'2","last_scrub_stamp":"2024-09-16T15:02:29.146986+0000","last_deep_scrub":"257'2","last_deep_scrub_stamp":"2024-09-16T15:02:29.146986+0000","last_clean_scrub_stamp":"2024-09-16T15:02:29.146986+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:20:50.027805+0000","scrub_duration":0.0055780450000000002,"objects_trimmed":0,"snaptrim_duration":0.00042440799999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"257'119","reported_seq":443,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:28.888589+0000","last_change":"2024-09-16T15:02:28.888589+0000","last_active":"2024-09-16T15:02:28.888589+0000","last_peered":"2024-09-16T15:02:28.888589+0000","last_clean":"2024-09-16T15:02:28.888589+0000","last_became_active":"2024-09-16T14:48:55.154511+0000","last_became_peered":"2024-09-16T14:48:55.154511+0000","last_unstale":"2024-09-16T15:02:28.888589+0000","last_undegraded":"2024-09-16T15:02:28.888589+0000","last_fullsized":"2024-09-16T15:02:28.888589+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'119","last_scrub_stamp":"2024-09-16T15:02:28.888323+0000","last_deep_scrub":"257'119","last_deep_scrub_stamp":"2024-09-16T15:02:28.888323+0000","last_clean_scrub_stamp":"2024-09-16T15:02:28.888323+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:13:08.558812+0000","scrub_duration":0.0090316540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":93,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:34.234230+0000","last_change":"2024-09-16T15:02:34.234420+0000","last_active":"2024-09-16T15:02:34.234230+0000","last_peered":"2024-09-16T15:02:34.234230+0000","last_clean":"2024-09-16T15:02:34.234230+0000","last_became_active":"2024-09-16T14:48:52.128554+0000","last_became_peered":"2024-09-16T14:48:52.128554+0000","last_unstale":"2024-09-16T15:02:34.234230+0000","last_undegraded":"2024-09-16T15:02:34.234230+0000","last_fullsized":"2024-09-16T15:02:34.234230+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:34.234184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:34.234184+0000","last_clean_scrub_stamp":"2024-09-16T15:02:34.234184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:26:34.155051+0000","scrub_duration":0.0032743960000000002,"objects_trimmed":0,"snaptrim_duration":0.00039796999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":101,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:03:01.865797+0000","last_change":"2024-09-16T15:03:01.866014+0000","last_active":"2024-09-16T15:03:01.865797+0000","last_peered":"2024-09-16T15:03:01.865797+0000","last_clean":"2024-09-16T15:03:01.865797+0000","last_became_active":"2024-09-16T14:48:52.143705+0000","last_became_peered":"2024-09-16T14:48:52.143705+0000","last_unstale":"2024-09-16T15:03:01.865797+0000","last_undegraded":"2024-09-16T15:03:01.865797+0000","last_fullsized":"2024-09-16T15:03:01.865797+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:03:01.865725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:03:01.865725+0000","last_clean_scrub_stamp":"2024-09-16T15:03:01.865725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:01:46.247150+0000","scrub_duration":0.0031700579999999999,"objects_trimmed":0,"snaptrim_duration":0.00061478100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":102,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:40.833723+0000","last_change":"2024-09-16T15:02:40.833971+0000","last_active":"2024-09-16T15:02:40.833723+0000","last_peered":"2024-09-16T15:02:40.833723+0000","last_clean":"2024-09-16T15:02:40.833723+0000","last_became_active":"2024-09-16T14:48:52.131160+0000","last_became_peered":"2024-09-16T14:48:52.131160+0000","last_unstale":"2024-09-16T15:02:40.833723+0000","last_undegraded":"2024-09-16T15:02:40.833723+0000","last_fullsized":"2024-09-16T15:02:40.833723+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:40.833671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:40.833671+0000","last_clean_scrub_stamp":"2024-09-16T15:02:40.833671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:23:14.783075+0000","scrub_duration":0.003363787,"objects_trimmed":0,"snaptrim_duration":0.00065779100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":238,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T15:02:39.587162+0000","last_change":"2024-09-16T15:02:39.587358+0000","last_active":"2024-09-16T15:02:39.587162+0000","last_peered":"2024-09-16T15:02:39.587162+0000","last_clean":"2024-09-16T15:02:39.587162+0000","last_became_active":"2024-09-16T14:48:52.133197+0000","last_became_peered":"2024-09-16T14:48:52.133197+0000","last_unstale":"2024-09-16T15:02:39.587162+0000","last_undegraded":"2024-09-16T15:02:39.587162+0000","last_fullsized":"2024-09-16T15:02:39.587162+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:02:39.587113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:02:39.587113+0000","last_clean_scrub_stamp":"2024-09-16T15:02:39.587113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:32:20.211569+0000","scrub_duration":0.0032142170000000001,"objects_trimmed":0,"snaptrim_duration":0.00085482099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":94400,"data_compressed_allocated":122880,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640508,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32492,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339348,"statfs":{"total":96636764160,"available":96603492352,"internally_reserved":0,"allocated":1413120,"data_stored":1289362,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.622,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.92600000000000005,"5min":1.042,"15min":1.2},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.81499999999999995,"15min":0.79100000000000004},"min":{"1min":0.627,"5min":0.51500000000000001,"15min":0.47899999999999998},"max":{"1min":0.95599999999999996,"5min":1.0369999999999999,"15min":1.3300000000000001},"last":0.98599999999999999}]},{"osd":1,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.81799999999999995,"15min":0.81000000000000005},"min":{"1min":0.61899999999999999,"5min":0.44900000000000001,"15min":0.33900000000000002},"max":{"1min":1.0589999999999999,"5min":1.073,"15min":1.2010000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.752,"5min":0.74399999999999999,"15min":0.752},"min":{"1min":0.60999999999999999,"5min":0.54000000000000004,"15min":0.438},"max":{"1min":0.90800000000000003,"5min":0.94299999999999995,"15min":1.089},"last":0.80000000000000004}]},{"osd":2,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.81799999999999995,"15min":0.82899999999999996},"min":{"1min":0.628,"5min":0.50800000000000001,"15min":0.44900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.272},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.90600000000000003,"15min":0.91300000000000003},"min":{"1min":0.81000000000000005,"5min":0.52900000000000003,"15min":0.42899999999999999},"max":{"1min":1.0289999999999999,"5min":1.056,"15min":1.3540000000000001},"last":0.95499999999999996}]},{"osd":3,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69699999999999995,"15min":0.67600000000000005},"min":{"1min":0.52300000000000002,"5min":0.46600000000000003,"15min":0.26000000000000001},"max":{"1min":0.96799999999999997,"5min":0.96999999999999997,"15min":1.1020000000000001},"last":0.87},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.68200000000000005,"15min":0.69099999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33500000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":1.153},"last":0.69199999999999995}]},{"osd":4,"last update":"Mon Sep 16 15:02:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72899999999999998,"15min":0.749},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.27500000000000002},"max":{"1min":0.95299999999999996,"5min":1.05,"15min":1.236},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.80000000000000004,"15min":0.77200000000000002},"min":{"1min":0.48799999999999999,"5min":0.44800000000000001,"15min":0.26300000000000001},"max":{"1min":1.038,"5min":1.0529999999999999,"15min":1.2949999999999999},"last":0.63800000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640508,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1417216,"data_stored":1289381,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78200000000000003,"15min":0.77300000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.437},"max":{"1min":0.96699999999999997,"5min":1.0800000000000001,"15min":1.1579999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73899999999999999,"15min":0.79000000000000004},"min":{"1min":0.52600000000000002,"5min":0.48899999999999999,"15min":0.48499999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.1679999999999999},"last":0.64900000000000002}]},{"osd":1,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.85899999999999999,"15min":0.81599999999999995},"min":{"1min":0.71199999999999997,"5min":0.52800000000000002,"15min":0.42399999999999999},"max":{"1min":1.0429999999999999,"5min":1.0780000000000001,"15min":1.4630000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.79600000000000004,"15min":0.83399999999999996},"min":{"1min":0.65700000000000003,"5min":0.53700000000000003,"15min":0.50800000000000001},"max":{"1min":1.0149999999999999,"5min":1.024,"15min":1.2729999999999999},"last":0.78800000000000003}]},{"osd":2,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.84999999999999998,"15min":0.84899999999999998},"min":{"1min":0.68000000000000005,"5min":0.65500000000000003,"15min":0.47599999999999998},"max":{"1min":1.02,"5min":1.0449999999999999,"15min":1.536},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84799999999999998,"15min":0.85899999999999999},"min":{"1min":0.63500000000000001,"5min":0.56599999999999995,"15min":0.51200000000000001},"max":{"1min":0.97099999999999997,"5min":1.0720000000000001,"15min":1.29},"last":0.68999999999999995}]},{"osd":3,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.70999999999999996,"15min":0.68899999999999995},"min":{"1min":0.53700000000000003,"5min":0.51300000000000001,"15min":0.27700000000000002},"max":{"1min":0.83299999999999996,"5min":1.0720000000000001,"15min":1.2669999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72899999999999998,"15min":0.72499999999999998},"min":{"1min":0.55700000000000005,"5min":0.50600000000000001,"15min":0.25700000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.198},"last":0.88500000000000001}]},{"osd":5,"last update":"Mon Sep 16 15:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.75600000000000001,"15min":0.73099999999999998},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.28999999999999998},"max":{"1min":0.95799999999999996,"5min":1.0600000000000001,"15min":1.1140000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.94699999999999995,"15min":0.91700000000000004},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.24299999999999999},"max":{"1min":1.0700000000000001,"5min":1.169,"15min":1.3879999999999999},"last":1.0900000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640507,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32552,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339288,"statfs":{"total":96636764160,"available":96603430912,"internally_reserved":0,"allocated":1474560,"data_stored":1818290,"data_compressed":271867,"data_compressed_allocated":389120,"data_compressed_original":1138688,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.73699999999999999,"15min":0.74299999999999999},"min":{"1min":0.60999999999999999,"5min":0.53100000000000003,"15min":0.45200000000000001},"max":{"1min":0.84599999999999997,"5min":0.998,"15min":1.2729999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73299999999999998,"15min":0.75},"min":{"1min":0.47699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.86199999999999999,"5min":0.91400000000000003,"15min":1.306},"last":0.90100000000000002}]},{"osd":1,"last update":"Mon Sep 16 15:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.90300000000000002,"15min":0.879},"min":{"1min":0.50600000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.018,"5min":1.123,"15min":1.294},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.86099999999999999,"15min":0.86799999999999999},"min":{"1min":0.70299999999999996,"5min":0.57299999999999995,"15min":0.41199999999999998},"max":{"1min":0.99299999999999999,"5min":1.046,"15min":1.9219999999999999},"last":0.96899999999999997}]},{"osd":2,"last update":"Mon Sep 16 15:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.81399999999999995,"15min":0.82099999999999995},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.45900000000000002},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.181},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.80800000000000005,"15min":0.80700000000000005},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.38800000000000001},"max":{"1min":0.96299999999999997,"5min":0.96899999999999997,"15min":2.444},"last":0.874}]},{"osd":4,"last update":"Mon Sep 16 15:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.628,"15min":0.621},"min":{"1min":0.502,"5min":0.47699999999999998,"15min":0.28199999999999997},"max":{"1min":0.878,"5min":0.98799999999999999,"15min":1.016},"last":0.622},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.81899999999999995,"15min":0.82099999999999995},"min":{"1min":0.68200000000000005,"5min":0.629,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":1.0740000000000001,"15min":1.2290000000000001},"last":0.78900000000000003}]},{"osd":5,"last update":"Mon Sep 16 15:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.70499999999999996,"15min":0.71199999999999997},"min":{"1min":0.60099999999999998,"5min":0.52400000000000002,"15min":0.29899999999999999},"max":{"1min":0.92300000000000004,"5min":1.4279999999999999,"15min":2.5409999999999999},"last":0.747},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83599999999999997,"15min":0.82199999999999995},"min":{"1min":0.65600000000000003,"5min":0.60999999999999999,"15min":0.33100000000000002},"max":{"1min":1.0329999999999999,"5min":1.514,"15min":1.514},"last":0.83399999999999996}]}]},{"osd":2,"up_from":11,"seq":47244640507,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32492,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339348,"statfs":{"total":96636764160,"available":96603492352,"internally_reserved":0,"allocated":1413120,"data_stored":1289362,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:03:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55200000000000005,"15min":0.57599999999999996},"min":{"1min":0.40600000000000003,"5min":0.252,"15min":0.249},"max":{"1min":0.72999999999999998,"5min":0.94699999999999995,"15min":1.819},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.76500000000000001,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.34499999999999997,"15min":0.312},"max":{"1min":0.91900000000000004,"5min":1.147,"15min":1.444},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Sep 16 15:03:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.28299999999999997},"max":{"1min":0.89300000000000002,"5min":0.95999999999999996,"15min":1.264},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61799999999999999,"15min":0.61899999999999999},"min":{"1min":0.375,"5min":0.313,"15min":0.26700000000000002},"max":{"1min":0.95699999999999996,"5min":1.0129999999999999,"15min":1.383},"last":0.61599999999999999}]},{"osd":3,"last update":"Mon Sep 16 15:03:05 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.84199999999999997,"15min":0.82999999999999996},"min":{"1min":0.71499999999999997,"5min":0.56000000000000005,"15min":0.36899999999999999},"max":{"1min":1.034,"5min":1.0720000000000001,"15min":1.2050000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.876,"15min":0.86599999999999999},"min":{"1min":0.72199999999999998,"5min":0.57999999999999996,"15min":0.44700000000000001},"max":{"1min":1.0469999999999999,"5min":1.21,"15min":1.335},"last":1.02}]},{"osd":4,"last update":"Mon Sep 16 15:03:05 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.82999999999999996,"15min":0.83399999999999996},"min":{"1min":0.65300000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"max":{"1min":0.98499999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.84499999999999997,"15min":0.85599999999999998},"min":{"1min":0.69099999999999995,"5min":0.58599999999999997,"15min":0.40999999999999998},"max":{"1min":1.012,"5min":1.0269999999999999,"15min":1.367},"last":0.876}]},{"osd":5,"last update":"Mon Sep 16 15:03:05 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.88300000000000001,"15min":0.88900000000000001},"min":{"1min":0.69999999999999996,"5min":0.57499999999999996,"15min":0.53700000000000003},"max":{"1min":1.0429999999999999,"5min":1.3149999999999999,"15min":1.911},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.95299999999999996,"15min":0.96099999999999997},"min":{"1min":0.73299999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1359999999999999,"5min":1.1990000000000001,"15min":2.0499999999999998},"last":0.96399999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640506,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32540,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339300,"statfs":{"total":96636764160,"available":96603443200,"internally_reserved":0,"allocated":1462272,"data_stored":1818290,"data_compressed":273302,"data_compressed_allocated":389120,"data_compressed_original":1150976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56000000000000005,"15min":0.57099999999999995},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.251},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":1.0720000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.628,"15min":0.63800000000000001},"min":{"1min":0.44500000000000001,"5min":0.36199999999999999,"15min":0.26100000000000001},"max":{"1min":0.87,"5min":0.87,"15min":1.0589999999999999},"last":0.70499999999999996}]},{"osd":2,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.66700000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.28699999999999998},"max":{"1min":0.82599999999999996,"5min":0.88200000000000001,"15min":1.052},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.628,"15min":0.66700000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32100000000000001},"max":{"1min":0.81899999999999995,"5min":0.879,"15min":1.403},"last":0.58099999999999996}]},{"osd":3,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85699999999999998,"15min":0.88800000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.503},"max":{"1min":1.052,"5min":1.117,"15min":1.359},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.86299999999999999,"15min":0.877},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1459999999999999,"5min":1.147,"15min":1.284},"last":0.84999999999999998}]},{"osd":4,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.88900000000000001,"15min":0.88200000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.46600000000000003},"max":{"1min":1.0900000000000001,"5min":1.097,"15min":1.341},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.88,"5min":0.86199999999999999,"15min":0.86899999999999999},"min":{"1min":0.67600000000000005,"5min":0.66200000000000003,"15min":0.46999999999999997},"max":{"1min":1.056,"5min":1.0800000000000001,"15min":1.399},"last":1.022}]},{"osd":5,"last update":"Mon Sep 16 15:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.622,"5min":0.622,"15min":0.50900000000000001},"max":{"1min":1,"5min":1.01,"15min":1.258},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90100000000000002,"15min":0.90400000000000003},"min":{"1min":0.69499999999999995,"5min":0.629,"15min":0.47299999999999998},"max":{"1min":1.1519999999999999,"5min":1.411,"15min":1.411},"last":0.95299999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640507,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36592,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335248,"statfs":{"total":96636764160,"available":96599293952,"internally_reserved":0,"allocated":1417216,"data_stored":1289381,"data_compressed":241357,"data_compressed_allocated":348160,"data_compressed_original":626688,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 15:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.36399999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0840000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.621,"15min":0.63500000000000001},"min":{"1min":0.32700000000000001,"5min":0.28299999999999997,"15min":0.27400000000000002},"max":{"1min":0.84499999999999997,"5min":0.91100000000000003,"15min":1.21},"last":0.873}]},{"osd":2,"last update":"Mon Sep 16 15:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68799999999999994,"15min":0.65400000000000003},"min":{"1min":0.39400000000000002,"5min":0.29899999999999999,"15min":0.25900000000000001},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.103},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.61099999999999999,"15min":0.66200000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.26900000000000002},"max":{"1min":0.90900000000000003,"5min":1.008,"15min":1.137},"last":0.71199999999999997}]},{"osd":3,"last update":"Mon Sep 16 15:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.84299999999999997,"15min":0.83299999999999996},"min":{"1min":0.71199999999999997,"5min":0.59599999999999997,"15min":0.52400000000000002},"max":{"1min":1.198,"5min":1.198,"15min":1.4650000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.82399999999999995,"15min":0.83799999999999997},"min":{"1min":0.56699999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.4019999999999999},"last":0.94499999999999995}]},{"osd":4,"last update":"Mon Sep 16 15:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.84399999999999997,"15min":0.84799999999999998},"min":{"1min":0.59399999999999997,"5min":0.57999999999999996,"15min":0.48999999999999999},"max":{"1min":1.25,"5min":1.25,"15min":1.3300000000000001},"last":1.014},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81699999999999995,"15min":0.84699999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.196,"5min":1.196,"15min":1.339},"last":0.91600000000000004}]},{"osd":5,"last update":"Mon Sep 16 15:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.91700000000000004,"15min":0.91600000000000004},"min":{"1min":0.747,"5min":0.66400000000000003,"15min":0.57999999999999996},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.6499999999999999},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.95899999999999996,"15min":0.95099999999999996},"min":{"1min":0.72799999999999998,"5min":0.72099999999999997,"15min":0.51900000000000002},"max":{"1min":1.331,"5min":1.331,"15min":1.7290000000000001},"last":0.83499999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":31945,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30510,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:03:09.351 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T15:03:09.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.466+0000 7f143a2ad640 10 monclient: get_monmap_and_config 2024-09-16T15:03:09.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.466+0000 7f143a2ad640 10 monclient: build_initial_monmap 2024-09-16T15:03:09.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.466+0000 7f143a2ad640 10 monclient: monmap: 2024-09-16T15:03:09.471 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:03:09.471 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:03:09.471 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T15:03:09.472548+0000 2024-09-16T15:03:09.471 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T15:03:09.472548+0000 2024-09-16T15:03:09.471 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:03:09.471 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:03:09.471 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:03:09.471 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T15:03:09.471 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-09-16T15:03:09.472 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:03:09.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.466+0000 7f143a2ad640 1 Processor -- start 2024-09-16T15:03:09.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.466+0000 7f143a2ad640 1 -- start start 2024-09-16T15:03:09.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.466+0000 7f143a2ad640 10 monclient: init 2024-09-16T15:03:09.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient: _reopen_session rank -1 2024-09-16T15:03:09.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:03:09.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient(hunting): picked mon.noname-c con 0x7f143409a550 addr v1:172.21.15.189:6789/0 2024-09-16T15:03:09.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient(hunting): picked mon.noname-b con 0x7f143409d1b0 addr v1:172.21.15.26:6790/0 2024-09-16T15:03:09.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient(hunting): picked mon.noname-a con 0x7f143409b040 addr v1:172.21.15.26:6789/0 2024-09-16T15:03:09.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f143409dc60 con 0x7f143409b040 2024-09-16T15:03:09.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1434099ae0 con 0x7f143409a550 2024-09-16T15:03:09.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f143409b8f0 con 0x7f143409d1b0 2024-09-16T15:03:09.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient(hunting): _renew_subs 2024-09-16T15:03:09.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient(hunting): authenticate will time out at 2213.592554s 2024-09-16T15:03:09.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f14337fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f143409d1b0 0x7f1434136a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35768/0 (socket says 172.21.15.26:35768) 2024-09-16T15:03:09.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f14337fe640 1 -- 172.21.15.26:0/1066316440 learned_addr learned my addr 172.21.15.26:0/1066316440 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:03:09.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 1 -- 172.21.15.26:0/1066316440 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606730198 0 0) 0x7f143409dc60 con 0x7f143409b040 2024-09-16T15:03:09.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:03:09.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:03:09.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:03:09.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient(hunting): my global_id is 334569 2024-09-16T15:03:09.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 1 -- 172.21.15.26:0/1066316440 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f141c005740 con 0x7f143409b040 2024-09-16T15:03:09.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 1 -- 172.21.15.26:0/1066316440 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126810216 0 0) 0x7f143409b8f0 con 0x7f143409d1b0 2024-09-16T15:03:09.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:03:09.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:03:09.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:03:09.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient(hunting): my global_id is 354383 2024-09-16T15:03:09.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 1 -- 172.21.15.26:0/1066316440 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f143409dc60 con 0x7f143409d1b0 2024-09-16T15:03:09.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 1 -- 172.21.15.26:0/1066316440 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750424063 0 0) 0x7f1434099ae0 con 0x7f143409a550 2024-09-16T15:03:09.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient(hunting): _negotiate 2024-09-16T15:03:09.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:03:09.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:03:09.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient(hunting): my global_id is 364246 2024-09-16T15:03:09.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 1 -- 172.21.15.26:0/1066316440 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f143409b8f0 con 0x7f143409a550 2024-09-16T15:03:09.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 1 -- 172.21.15.26:0/1066316440 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1240933527 0 0) 0x7f141c005740 con 0x7f143409b040 2024-09-16T15:03:09.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 1 -- 172.21.15.26:0/1066316440 >> v1:172.21.15.26:6790/0 conn(0x7f143409d1b0 legacy=0x7f1434136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:03:09.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 1 -- 172.21.15.26:0/1066316440 >> v1:172.21.15.189:6789/0 conn(0x7f143409a550 legacy=0x7f143409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:03:09.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient: _finish_hunting 0 2024-09-16T15:03:09.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 1 monclient: found mon.noname-a 2024-09-16T15:03:09.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:03:09.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-09-16T15:03:09.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 1 -- 172.21.15.26:0/1066316440 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143409bbd0 con 0x7f143409b040 2024-09-16T15:03:09.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient: _finish_auth 0 2024-09-16T15:03:09.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient: _check_auth_tickets 2024-09-16T15:03:09.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:03:09.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 1 -- 172.21.15.26:0/1066316440 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f142c001920 con 0x7f143409b040 2024-09-16T15:03:09.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:03:09.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:03:09.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient: dump: 2024-09-16T15:03:09.482 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:03:09.482 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:03:09.482 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:03:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:03:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:03:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:03:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:03:09.483 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:03:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 5 monclient: authenticate success, global_id 334569 2024-09-16T15:03:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T15:03:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 1 -- 172.21.15.26:0/1066316440 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f142c001b40 con 0x7f143409b040 2024-09-16T15:03:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:03:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 1 -- 172.21.15.26:0/1066316440 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f142c001f10 con 0x7f143409b040 2024-09-16T15:03:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:03:09.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:03:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f1432ffd640 10 monclient: dump: 2024-09-16T15:03:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:03:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:03:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:03:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:03:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:03:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:03:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:03:09.484 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:03:09.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient: get_monmap_and_config success 2024-09-16T15:03:09.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 4 set_mon_vals no callback set 2024-09-16T15:03:09.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient: shutdown 2024-09-16T15:03:09.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:03:09.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 1 -- 172.21.15.26:0/1066316440 >> v1:172.21.15.26:6789/0 conn(0x7f143409b040 legacy=0x7f143413a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:03:09.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 1 -- 172.21.15.26:0/1066316440 shutdown_connections 2024-09-16T15:03:09.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 1 -- 172.21.15.26:0/1066316440 >> 172.21.15.26:0/1066316440 conn(0x7f143409fb50 msgr2=0x7f143409ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:03:09.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 1 -- 172.21.15.26:0/1066316440 shutdown_connections 2024-09-16T15:03:09.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 1 -- 172.21.15.26:0/1066316440 wait complete. 2024-09-16T15:03:09.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient: build_initial_monmap 2024-09-16T15:03:09.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient: monmap: 2024-09-16T15:03:09.487 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-09-16T15:03:09.488 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T15:03:09.488 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-09-16T15:03:09.488 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-09-16T15:03:09.488 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-09-16T15:03:09.488 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-09-16T15:03:09.488 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-09-16T15:03:09.489 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T15:03:09.489 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-09-16T15:03:09.489 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:03:09.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 1 Processor -- start 2024-09-16T15:03:09.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 1 -- start start 2024-09-16T15:03:09.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient: init 2024-09-16T15:03:09.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient: _reopen_session rank -1 2024-09-16T15:03:09.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T15:03:09.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient(hunting): picked mon.noname-b con 0x7f143409a550 addr v1:172.21.15.189:6789/0 2024-09-16T15:03:09.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient(hunting): picked mon.noname-c con 0x7f143409b040 addr v1:172.21.15.26:6790/0 2024-09-16T15:03:09.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient(hunting): picked mon.noname-a con 0x7f143409d1b0 addr v1:172.21.15.26:6789/0 2024-09-16T15:03:09.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14341b54c0 con 0x7f143409d1b0 2024-09-16T15:03:09.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14341a3350 con 0x7f143409a550 2024-09-16T15:03:09.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14341a1820 con 0x7f143409b040 2024-09-16T15:03:09.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient(hunting): _renew_subs 2024-09-16T15:03:09.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143a2ad640 10 monclient(hunting): authenticate will time out at 2213.595697s 2024-09-16T15:03:09.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143884a640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f143409d1b0 0x7f14341cd380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33350/0 (socket says 172.21.15.26:33350) 2024-09-16T15:03:09.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f143884a640 1 -- 172.21.15.26:0/2471912520 learned_addr learned my addr 172.21.15.26:0/2471912520 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-09-16T15:03:09.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.470+0000 7f14337fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f143409b040 0x7f14341c5830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:35770/0 (socket says 172.21.15.26:35770) 2024-09-16T15:03:09.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031557646 0 0) 0x7f14341b54c0 con 0x7f143409d1b0 2024-09-16T15:03:09.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:03:09.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:03:09.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:03:09.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): my global_id is 334572 2024-09-16T15:03:09.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1420003530 con 0x7f143409d1b0 2024-09-16T15:03:09.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301621181 0 0) 0x7f14341a1820 con 0x7f143409b040 2024-09-16T15:03:09.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:03:09.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:03:09.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:03:09.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): my global_id is 354386 2024-09-16T15:03:09.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14341b54c0 con 0x7f143409b040 2024-09-16T15:03:09.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 882247434 0 0) 0x7f1420003530 con 0x7f143409d1b0 2024-09-16T15:03:09.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14341a1820 con 0x7f143409d1b0 2024-09-16T15:03:09.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f142c001e50 con 0x7f143409d1b0 2024-09-16T15:03:09.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:03:09.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T15:03:09.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): dump: 2024-09-16T15:03:09.492 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:03:09.492 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:03:09.493 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.493 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.493 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:03:09.493 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:03:09.493 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:03:09.493 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:03:09.493 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:03:09.493 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:03:09.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T15:03:09.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4156775088 0 0) 0x7f14341b54c0 con 0x7f143409b040 2024-09-16T15:03:09.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1420003530 con 0x7f143409b040 2024-09-16T15:03:09.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f14280019e0 con 0x7f143409b040 2024-09-16T15:03:09.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T15:03:09.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T15:03:09.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): dump: 2024-09-16T15:03:09.494 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:03:09.494 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:03:09.494 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.494 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.495 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:03:09.495 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:03:09.495 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:03:09.495 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:03:09.495 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:03:09.496 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:03:09.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351673585 0 0) 0x7f14341a3350 con 0x7f143409a550 2024-09-16T15:03:09.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): _negotiate 2024-09-16T15:03:09.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T15:03:09.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T15:03:09.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient(hunting): my global_id is 364249 2024-09-16T15:03:09.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14341b54c0 con 0x7f143409a550 2024-09-16T15:03:09.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3537508736 0 0) 0x7f14341a1820 con 0x7f143409d1b0 2024-09-16T15:03:09.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 >> v1:172.21.15.26:6790/0 conn(0x7f143409b040 legacy=0x7f14341c5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:03:09.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 >> v1:172.21.15.189:6789/0 conn(0x7f143409a550 legacy=0x7f14341c5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:03:09.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient: _finish_hunting 0 2024-09-16T15:03:09.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 monclient: found mon.a 2024-09-16T15:03:09.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T15:03:09.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:03:09.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14341a3910 con 0x7f143409d1b0 2024-09-16T15:03:09.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient: _finish_auth 0 2024-09-16T15:03:09.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient: _check_auth_tickets 2024-09-16T15:03:09.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T15:03:09.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f143a2ad640 5 monclient: authenticate success, global_id 334572 2024-09-16T15:03:09.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f143a2ad640 10 monclient: _renew_subs 2024-09-16T15:03:09.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f143a2ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:03:09.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f143a2ad640 1 -- 172.21.15.26:0/2471912520 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14341a3c50 con 0x7f143409d1b0 2024-09-16T15:03:09.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f143a2ad640 10 monclient: _renew_subs 2024-09-16T15:03:09.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f143a2ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:03:09.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f143a2ad640 1 -- 172.21.15.26:0/2471912520 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f143409a240 con 0x7f143409d1b0 2024-09-16T15:03:09.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f142c001580 con 0x7f143409d1b0 2024-09-16T15:03:09.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-16T15:03:09.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3194916077 0 0) 0x7f142c003ed0 con 0x7f143409d1b0 2024-09-16T15:03:09.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T15:03:09.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T15:03:09.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 10 monclient: dump: 2024-09-16T15:03:09.498 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-09-16T15:03:09.498 INFO:teuthology.orchestra.run.smithi026.stderr:fsid febeb2df-62e5-4150-ac3f-7a627f391606 2024-09-16T15:03:09.498 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.498 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-09-16T14:41:47.153956+0000 2024-09-16T15:03:09.498 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 18 (reef) 2024-09-16T15:03:09.498 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-09-16T15:03:09.499 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-09-16T15:03:09.499 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T15:03:09.499 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-09-16T15:03:09.499 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-09-16T15:03:09.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f14327fc640 4 set_mon_vals no callback set 2024-09-16T15:03:09.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3807003579 0 0) 0x7f142c004030 con 0x7f143409d1b0 2024-09-16T15:03:09.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f143a2ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T15:03:09.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f143a2ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:03:09.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f143a2ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:03:09.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f143a2ad640 1 -- 172.21.15.26:0/2471912520 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1414005050 con 0x7f143409d1b0 2024-09-16T15:03:09.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(257..257 src has 232..257) v4 ==== 6745+0+0 (unknown 1661979738 0 0) 0x7f142c0441a0 con 0x7f143409d1b0 2024-09-16T15:03:09.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.474+0000 7f1433fff640 10 monclient: get_auth_request con 0x7f1420036370 auth_method 0 2024-09-16T15:03:09.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.478+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f142c014c50 con 0x7f143409d1b0 2024-09-16T15:03:09.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.478+0000 7f1430ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:03:09.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.478+0000 7f1430ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T15:03:09.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.638+0000 7f143a2ad640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T15:03:09.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.638+0000 7f143a2ad640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T15:03:09.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.638+0000 7f143a2ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-09-16T15:03:09.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.638+0000 7f143a2ad640 1 -- 172.21.15.26:0/2471912520 --> v1:172.21.15.26:6789/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f1414005880 con 0x7f143409d1b0 2024-09-16T15:03:09.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.646+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 <== mon.0 v1:172.21.15.26:6789/0 10 ==== config(1 keys) v1 ==== 35+0+0 (unknown 2555425225 0 0) 0x7f142c014c50 con 0x7f143409d1b0 2024-09-16T15:03:09.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.646+0000 7f1430ff9640 10 monclient: handle_config config(1 keys) v1 2024-09-16T15:03:09.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.646+0000 7f14327fc640 4 set_mon_vals no callback set 2024-09-16T15:03:09.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.646+0000 7f14327fc640 10 set_mon_vals mon_health_to_clog = false 2024-09-16T15:03:09.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.646+0000 7f1430ff9640 1 -- 172.21.15.26:0/2471912520 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v4)=0 v4) v1 ==== 123+0+0 (unknown 1766731237 0 0) 0x7f142c0130c0 con 0x7f143409d1b0 2024-09-16T15:03:09.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.646+0000 7f1430ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T15:03:09.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.646+0000 7f1430ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T15:03:09.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.646+0000 7f143a2ad640 1 -- 172.21.15.26:0/2471912520 >> v1:172.21.15.189:6812/3196161582 conn(0x7f1420036370 legacy=0x7f1420038830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:03:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.646+0000 7f143a2ad640 10 monclient: shutdown 2024-09-16T15:03:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.646+0000 7f143a2ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T15:03:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.646+0000 7f143a2ad640 1 -- 172.21.15.26:0/2471912520 >> v1:172.21.15.26:6789/0 conn(0x7f143409d1b0 legacy=0x7f14341cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:03:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.650+0000 7f143a2ad640 1 -- 172.21.15.26:0/2471912520 shutdown_connections 2024-09-16T15:03:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.650+0000 7f143a2ad640 1 -- 172.21.15.26:0/2471912520 >> 172.21.15.26:0/2471912520 conn(0x7f143409fb50 msgr2=0x7f14340a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:03:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.650+0000 7f143a2ad640 1 -- 172.21.15.26:0/2471912520 shutdown_connections 2024-09-16T15:03:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-09-16T15:03:09.650+0000 7f143a2ad640 1 -- 172.21.15.26:0/2471912520 wait complete. 2024-09-16T15:03:09.667 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T15:03:09.667 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T15:03:09.667 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T15:03:09.668 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:03:09.757 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T15:03:09.757 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T15:03:09.757 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:03:09.834 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T15:03:09.834 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T15:03:09.835 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:03:09.906 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T15:03:09.906 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T15:03:09.906 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:03:10.004 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T15:03:10.005 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T15:03:10.005 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:03:10.096 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T15:03:10.097 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T15:03:10.097 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:03:10.164 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T15:03:10.164 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T15:03:10.164 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T15:03:10.164 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:03:10.194 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T15:03:10.194 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T15:03:10.194 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T15:03:10.194 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:03:10.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-09-16T15:03:10.222 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T15:03:10.223 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T15:03:10.223 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:03:10.288 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T15:03:10.289 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T15:03:10.289 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:03:10.312 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T15:03:10.313 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T15:03:10.313 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2024-09-16T15:03:10.355 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T15:03:10.355 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T15:03:10.593 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T15:03:10.593 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T15:03:10.754 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T15:03:10.754 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T15:03:10.871 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T15:03:10.871 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T15:03:11.047 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T15:03:11.047 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T15:03:11.229 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T15:03:11.229 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T15:03:11.401 INFO:tasks.ceph:Archiving mon data... 2024-09-16T15:03:11.403 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906750/data/mon.a.tgz 2024-09-16T15:03:11.403 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-09-16T15:03:11.408 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.M8wIMBRClg 2024-09-16T15:03:11.409 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.M8wIMBRClg 2024-09-16T15:03:11.616 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.M8wIMBRClg 2024-09-16T15:03:11.680 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.M8wIMBRClg is 1012KB 2024-09-16T15:03:11.750 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.M8wIMBRClg 2024-09-16T15:03:11.755 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906750/data/mon.c.tgz 2024-09-16T15:03:11.755 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-09-16T15:03:11.805 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.ucgxY7flZa 2024-09-16T15:03:11.806 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.ucgxY7flZa 2024-09-16T15:03:12.015 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.ucgxY7flZa 2024-09-16T15:03:12.084 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.ucgxY7flZa is 1012KB 2024-09-16T15:03:12.150 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.ucgxY7flZa 2024-09-16T15:03:12.155 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906750/data/mon.b.tgz 2024-09-16T15:03:12.155 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-09-16T15:03:12.161 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.25ASnoUkAk 2024-09-16T15:03:12.162 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.25ASnoUkAk 2024-09-16T15:03:12.366 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.25ASnoUkAk 2024-09-16T15:03:12.429 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.25ASnoUkAk is 1015KB 2024-09-16T15:03:12.496 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.25ASnoUkAk 2024-09-16T15:03:12.501 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T15:03:12.501 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T15:03:12.504 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T15:03:12.557 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-16T15:03:12.557 DEBUG:teuthology.orchestra.run.smithi026:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T15:03:12.561 DEBUG:teuthology.orchestra.run.smithi189:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T15:03:12.583 INFO:teuthology.orchestra.run.smithi026.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T15:03:12.624 INFO:teuthology.orchestra.run.smithi189.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T15:03:12.628 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T15:03:12.630 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906750/remote/smithi026/crash 2024-09-16T15:03:12.630 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T15:03:12.644 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906750/remote/smithi189/crash 2024-09-16T15:03:12.645 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T15:03:12.678 INFO:tasks.ceph:Compressing logs... 2024-09-16T15:03:12.679 DEBUG:teuthology.orchestra.run.smithi026:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T15:03:12.689 DEBUG:teuthology.orchestra.run.smithi189:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T15:03:15.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-09-16T15:03:19.078 INFO:tasks.ceph:Archiving logs... 2024-09-16T15:03:19.079 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906750/remote/smithi026/log 2024-09-16T15:03:19.080 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T15:03:20.107 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906750/remote/smithi189/log 2024-09-16T15:03:20.108 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T15:03:20.881 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T15:03:20.892 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T15:03:20.892 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T15:03:20.895 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T15:03:20.959 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T15:03:20.967 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T15:03:20.968 DEBUG:teuthology.orchestra.run.smithi026:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T15:03:20.970 DEBUG:teuthology.orchestra.run.smithi189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T15:03:20.981 INFO:teuthology.orchestra.run.smithi026.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T15:03:20.981 INFO:teuthology.orchestra.run.smithi026.stdout:============================================================================== 2024-09-16T15:03:20.982 INFO:teuthology.orchestra.run.smithi026.stdout:*hv01.front.sepi 67.205.162.81 3 u 24 64 377 0.075 -2.602 0.351 2024-09-16T15:03:20.982 INFO:teuthology.orchestra.run.smithi026.stdout:+hv02.front.sepi 63.231.80.2 3 u 24 64 377 0.097 -2.333 0.266 2024-09-16T15:03:20.982 INFO:teuthology.orchestra.run.smithi026.stdout:+hv03.front.sepi 74.6.168.72 3 u 9 64 377 0.078 -3.305 0.498 2024-09-16T15:03:20.982 INFO:teuthology.orchestra.run.smithi026.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-09-16T15:03:20.983 INFO:teuthology.orchestra.run.smithi189.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T15:03:20.983 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================== 2024-09-16T15:03:20.984 INFO:teuthology.orchestra.run.smithi189.stdout:+hv01.front.sepi 67.205.162.81 3 u 6 64 377 0.099 -5.901 0.457 2024-09-16T15:03:20.984 INFO:teuthology.orchestra.run.smithi189.stdout:*hv02.front.sepi 63.231.80.2 3 u 19 64 377 0.088 -5.328 0.436 2024-09-16T15:03:20.984 INFO:teuthology.orchestra.run.smithi189.stdout:+hv03.front.sepi 74.6.168.72 3 u 24 64 377 0.108 -6.316 0.686 2024-09-16T15:03:20.984 INFO:teuthology.orchestra.run.smithi189.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-09-16T15:03:20.984 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T15:03:20.992 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T15:03:20.992 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T15:03:21.000 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T15:03:21.008 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T15:03:21.016 INFO:teuthology.task.internal:Duration was 1858.117796 seconds 2024-09-16T15:03:21.017 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T15:03:21.025 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T15:03:21.026 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T15:03:21.029 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T15:03:21.067 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T15:03:21.067 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi026.front.sepia.ceph.com 2024-09-16T15:03:21.068 DEBUG:teuthology.orchestra.run.smithi026:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T15:03:21.116 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T15:03:21.117 DEBUG:teuthology.orchestra.run.smithi189:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T15:03:21.126 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T15:03:21.127 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T15:03:21.165 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T15:03:21.352 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T15:03:21.353 DEBUG:teuthology.orchestra.run.smithi026:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T15:03:21.365 DEBUG:teuthology.orchestra.run.smithi189:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T15:03:21.459 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T15:03:21.470 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T15:03:21.470 DEBUG:teuthology.orchestra.run.smithi026:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T15:03:21.484 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T15:03:21.498 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T15:03:21.507 DEBUG:teuthology.orchestra.run.smithi026:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:03:21.528 DEBUG:teuthology.orchestra.run.smithi189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:03:21.538 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = core 2024-09-16T15:03:21.549 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = core 2024-09-16T15:03:21.562 DEBUG:teuthology.orchestra.run.smithi026:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:03:21.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:03:21.604 DEBUG:teuthology.orchestra.run.smithi189:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:03:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:03:21.608 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T15:03:21.617 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T15:03:21.618 DEBUG:teuthology.misc:Transferring archived files from smithi026:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906750/remote/smithi026 2024-09-16T15:03:21.618 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T15:03:21.668 DEBUG:teuthology.misc:Transferring archived files from smithi189:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906750/remote/smithi189 2024-09-16T15:03:21.668 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T15:03:21.689 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T15:03:21.689 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T15:03:21.709 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T15:03:21.733 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T15:03:21.743 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T15:03:21.743 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T15:03:21.752 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T15:03:21.752 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T15:03:21.757 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T15:03:21.760 INFO:teuthology.orchestra.run.smithi026.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 15:03 /home/ubuntu/cephtest 2024-09-16T15:03:21.782 INFO:teuthology.orchestra.run.smithi189.stdout: 655429 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 15:03 /home/ubuntu/cephtest 2024-09-16T15:03:21.783 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T15:03:21.793 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T15:03:21.815 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_mon_workunits} duration: 1858.1177961826324 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T15:03:21.815 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T15:03:21.909 INFO:teuthology.run:pass